2021-03-24T21:34:47.711 INFO:root:teuthology version: 1.1.0-6b3150e9 2021-03-24T21:34:47.713 DEBUG:teuthology.run:Teuthology command: teuthology --description rados:cephadm/thrash/{0-distro/ubuntu_20.04_kubic_stable 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v1only root} --name sage-2021-03-24_21:26:22-rados:cephadm-wip-sage2-testing-2021-03-24-1043-pacific-distro-basic-smithi --owner scheduled_sage@teuthology --verbose --archive /home/teuthworker/archive/sage-2021-03-24_21:26:22-rados:cephadm-wip-sage2-testing-2021-03-24-1043-pacific-distro-basic-smithi/5995072 -- /home/teuthworker/archive/sage-2021-03-24_21:26:22-rados:cephadm-wip-sage2-testing-2021-03-24-1043-pacific-distro-basic-smithi/5995072/orig.config.yaml 2021-03-24T21:34:47.751 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-03-24T21:34:47.802 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/sage-2021-03-24_21:26:22-rados:cephadm-wip-sage2-testing-2021-03-24-1043-pacific-distro-basic-smithi/5995072 branch: wip-sage2-testing-2021-03-24-1043-pacific description: rados:cephadm/thrash/{0-distro/ubuntu_20.04_kubic_stable 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v1only root} email: null first_in_suite: false job_id: '5995072' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: sage-2021-03-24_21:26:22-rados:cephadm-wip-sage2-testing-2021-03-24-1043-pacific-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-sage2-testing-2021-03-24-1043-pacific ceph: conf: global: ms bind msgr2: false ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug ms: 1 debug osd: 20 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) mon_bind_msgr2: false sha1: d71e96d216190f28f757f409532464ab0eca4ea9 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 cephadm: cephadm_mode: root install: ceph: sha1: d71e96d216190f28f757f409532464ab0eca4ea9 workunit: branch: wip-sage2-testing-2021-03-24-1043-pacific sha1: d71e96d216190f28f757f409532464ab0eca4ea9 owner: scheduled_sage@teuthology priority: 70 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a sha1: d71e96d216190f28f757f409532464ab0eca4ea9 sleep_before_teardown: 0 suite: rados:cephadm suite_branch: wip-sage2-testing-2021-03-24-1043-pacific suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_d71e96d216190f28f757f409532464ab0eca4ea9/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: d71e96d216190f28f757f409532464ab0eca4ea9 targets: smithi125.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQNNfhXPTo23DE1+cYieHVWwxZsOikZOwImR5JijRn/ULhNz2FFKO4CZ2LlXehqtC7jcFGDcOtwMxgLsUFW92/BHB+Cz3ESE9P2AwkEGliq4Ic794slaTHZuc4tIxLYF51H/UxmSBCYtNZy2MB+oAjJdWM/eUtpC2/F6HcEpqTsndZrJ8v5b1ryIRT60oEZJDm330IPEENDGGkobGZ2Vvg5iF6SsYi00q8x6ArC3V/WOQUS5fK0FHMOvmI9c0zRMBjnl4GJuDF9bJmhRPf9WYq7A9uxVGTgw8+xJ2kWDpVM2kipzdyvn66nwZwik979MIMIOCuTAEjG/EtxyuITpmaM/UjXe9y2D/LG4NCEMLqYN+bZ2PbqsqgktI+wOVvl0lOwZJ6eG6kmtfwE+cg2lF5kJ0UO+sJYel07hEqF01oCUTXGwWh+Y33kvH6ND3H87D2u4hc9cT3hTQzVHqNcr27AnCEl1FyAIE/Gv9arp4+62ofNYGlmJS9EO9Q/4A6EW0= smithi142.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHHqjuTv+a+sA8U+1+Pm0wUm+3xl2J2k+Fxyni5EJVogS40EUCMNI0otmUi4bSpwHnZBdVhel7e2wXY6uOeQTDbzjk0cVA8rba7+f6A6RR8XhMTJ3GQnKmzdJumK26FCCf+hqOO8vEoVQ8CzDg0yC86vCmrvXm1w79/G1VMh+3Abfc8NUr6yIqTAUYtyLx3e+WviirjFFFF6G5PL9NU6N1UlsbqotF0RYsbmwP7GGR7L+05JeWCYk4OuBtQsfRJIknb4tB7upXQpGzEDPI+UStNcWQj6G9TIjr5tu+481DaC6YVbNc5MMmw7XEXmnj2Y7lr8lOG69pkkgU9AiHd3KkUb5pyk9s5o/b3icXS1GlUnENlqGbYYllwUyrM1yat16iERTs0NYayrMstlpM+oeZeK+bdRftBFjSvWqEsURgJKXStLid5ltDKiFMVH9nqm3HYQdmReaN40JACoM7yVnV5MQk/5c9lOkqtl0kemPJ2kMmMEtlI5jjplzs1cy8Da8= tasks: - pexec: all: - curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04/Release.key | sudo apt-key add - - echo "deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04/ /" | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list - sudo apt update - sudo apt -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install podman containernetworking-plugins - echo -e "[[registry]]\nlocation = 'docker.io'\n\n[[registry.mirror]]\nlocation='docker-mirror.front.sepia.ceph.com:5000'\n" | sudo tee /etc/containers/registries.conf - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - rados: clients: - client.0 objects: 50 op_weights: copy_from: 50 delete: 50 read: 100 rollback: 50 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 teuthology_branch: master teuthology_sha1: 6b3150e9e0aa7ca432e26f31d87920ebd77f3708 timestamp: 2021-03-24_21:26:22 tube: smithi user: sage verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.9855 2021-03-24T21:34:47.803 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_d71e96d216190f28f757f409532464ab0eca4ea9/qa; will attempt to use it 2021-03-24T21:34:47.804 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_d71e96d216190f28f757f409532464ab0eca4ea9/qa/tasks 2021-03-24T21:34:47.804 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-03-24T21:34:47.806 INFO:teuthology.task.internal:Checking packages... 2021-03-24T21:34:47.828 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'basic' and ceph hash 'd71e96d216190f28f757f409532464ab0eca4ea9' 2021-03-24T21:34:47.829 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-03-24T21:34:47.830 INFO:teuthology.packaging:ref: None 2021-03-24T21:34:47.830 INFO:teuthology.packaging:tag: None 2021-03-24T21:34:47.830 INFO:teuthology.packaging:branch: wip-sage2-testing-2021-03-24-1043-pacific 2021-03-24T21:34:47.831 INFO:teuthology.packaging:sha1: d71e96d216190f28f757f409532464ab0eca4ea9 2021-03-24T21:34:47.831 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-sage2-testing-2021-03-24-1043-pacific 2021-03-24T21:34:47.969 INFO:teuthology.task.internal:Found packages for ceph version 16.1.0-1166-gd71e96d2-1focal 2021-03-24T21:34:47.970 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-03-24T21:34:47.978 INFO:teuthology.task.internal:no buildpackages task found 2021-03-24T21:34:47.978 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-03-24T21:34:48.028 INFO:teuthology.task.internal:Saving configuration 2021-03-24T21:34:48.042 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-03-24T21:34:48.064 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-24T21:34:48.084 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-03-24 21:28:23.583327', 'description': '/home/teuthworker/archive/sage-2021-03-24_21:26:22-rados:cephadm-wip-sage2-testing-2021-03-24-1043-pacific-distro-basic-smithi/5995072', 'locked_by': 'scheduled_sage@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMYz8S3PXso6DZyzuiPvbHYIEVL17N2X1cydBpxrXJ3T', 'os_version': '20.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi125.front.sepia.ceph.com'} 2021-03-24T21:34:48.100 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-03-24 21:28:23.583349', 'description': '/home/teuthworker/archive/sage-2021-03-24_21:26:22-rados:cephadm-wip-sage2-testing-2021-03-24-1043-pacific-distro-basic-smithi/5995072', 'locked_by': 'scheduled_sage@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILGxDy2WONydMFj8cHkpb/UKOVKskhkDmN3ISAtvjwl9', 'os_version': '20.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi142.front.sepia.ceph.com'} 2021-03-24T21:34:48.101 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-03-24T21:34:48.110 INFO:teuthology.task.internal:roles: ubuntu@smithi125.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2021-03-24T21:34:48.111 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2021-03-24T21:34:48.111 INFO:teuthology.run_tasks:Running task console_log... 2021-03-24T21:34:48.188 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f9120ec78c8>, signals=[15]) 2021-03-24T21:34:48.190 INFO:teuthology.run_tasks:Running task internal.connect... 2021-03-24T21:34:48.201 INFO:teuthology.task.internal:Opening connections... 2021-03-24T21:34:48.201 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi125.front.sepia.ceph.com 2021-03-24T21:34:48.203 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi125.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-24T21:34:48.254 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi142.front.sepia.ceph.com 2021-03-24T21:34:48.255 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-24T21:34:48.304 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-03-24T21:34:48.315 DEBUG:teuthology.orchestra.run.smithi125:> uname -m 2021-03-24T21:34:48.333 INFO:teuthology.orchestra.run.smithi125.stdout:x86_64 2021-03-24T21:34:48.334 DEBUG:teuthology.orchestra.run.smithi125:> cat /etc/os-release 2021-03-24T21:34:48.383 INFO:teuthology.orchestra.run.smithi125.stdout:NAME="Ubuntu" 2021-03-24T21:34:48.384 INFO:teuthology.orchestra.run.smithi125.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-03-24T21:34:48.384 INFO:teuthology.orchestra.run.smithi125.stdout:ID=ubuntu 2021-03-24T21:34:48.385 INFO:teuthology.orchestra.run.smithi125.stdout:ID_LIKE=debian 2021-03-24T21:34:48.385 INFO:teuthology.orchestra.run.smithi125.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-03-24T21:34:48.385 INFO:teuthology.orchestra.run.smithi125.stdout:VERSION_ID="20.04" 2021-03-24T21:34:48.385 INFO:teuthology.orchestra.run.smithi125.stdout:HOME_URL="https://www.ubuntu.com/" 2021-03-24T21:34:48.385 INFO:teuthology.orchestra.run.smithi125.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-03-24T21:34:48.386 INFO:teuthology.orchestra.run.smithi125.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-03-24T21:34:48.386 INFO:teuthology.orchestra.run.smithi125.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-03-24T21:34:48.386 INFO:teuthology.orchestra.run.smithi125.stdout:VERSION_CODENAME=focal 2021-03-24T21:34:48.386 INFO:teuthology.orchestra.run.smithi125.stdout:UBUNTU_CODENAME=focal 2021-03-24T21:34:48.387 INFO:teuthology.lock.ops:Updating smithi125.front.sepia.ceph.com on lock server 2021-03-24T21:34:48.404 DEBUG:teuthology.orchestra.run.smithi142:> uname -m 2021-03-24T21:34:48.411 INFO:teuthology.orchestra.run.smithi142.stdout:x86_64 2021-03-24T21:34:48.411 DEBUG:teuthology.orchestra.run.smithi142:> cat /etc/os-release 2021-03-24T21:34:48.461 INFO:teuthology.orchestra.run.smithi142.stdout:NAME="Ubuntu" 2021-03-24T21:34:48.461 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-03-24T21:34:48.461 INFO:teuthology.orchestra.run.smithi142.stdout:ID=ubuntu 2021-03-24T21:34:48.462 INFO:teuthology.orchestra.run.smithi142.stdout:ID_LIKE=debian 2021-03-24T21:34:48.462 INFO:teuthology.orchestra.run.smithi142.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-03-24T21:34:48.462 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION_ID="20.04" 2021-03-24T21:34:48.462 INFO:teuthology.orchestra.run.smithi142.stdout:HOME_URL="https://www.ubuntu.com/" 2021-03-24T21:34:48.463 INFO:teuthology.orchestra.run.smithi142.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-03-24T21:34:48.463 INFO:teuthology.orchestra.run.smithi142.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-03-24T21:34:48.463 INFO:teuthology.orchestra.run.smithi142.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-03-24T21:34:48.464 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION_CODENAME=focal 2021-03-24T21:34:48.464 INFO:teuthology.orchestra.run.smithi142.stdout:UBUNTU_CODENAME=focal 2021-03-24T21:34:48.464 INFO:teuthology.lock.ops:Updating smithi142.front.sepia.ceph.com on lock server 2021-03-24T21:34:48.485 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-03-24T21:34:48.501 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-03-24T21:34:48.531 INFO:teuthology.task.internal:Checking for old test directory... 2021-03-24T21:34:48.531 DEBUG:teuthology.orchestra.run.smithi125:> test '!' -e /home/ubuntu/cephtest 2021-03-24T21:34:48.534 DEBUG:teuthology.orchestra.run.smithi142:> test '!' -e /home/ubuntu/cephtest 2021-03-24T21:34:48.540 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-03-24T21:34:48.564 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-03-24T21:34:48.564 DEBUG:teuthology.orchestra.run.smithi125:> test -z $(ls -A /var/lib/ceph) 2021-03-24T21:34:48.567 DEBUG:teuthology.orchestra.run.smithi142:> test -z $(ls -A /var/lib/ceph) 2021-03-24T21:34:48.590 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-03-24T21:34:48.637 INFO:teuthology.run_tasks:Running task kernel... 2021-03-24T21:34:48.655 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}, 'mgr.y': {'kdb': True, 'sha1': 'distro'}, 'osd.0': {'kdb': True, 'sha1': 'distro'}, 'osd.1': {'kdb': True, 'sha1': 'distro'}, 'osd.2': {'kdb': True, 'sha1': 'distro'}, 'osd.3': {'kdb': True, 'sha1': 'distro'}, 'client.0': {'kdb': True, 'sha1': 'distro'}, 'ceph.rgw.foo.a': {'kdb': True, 'sha1': 'distro'}, 'node-exporter.a': {'kdb': True, 'sha1': 'distro'}, 'alertmanager.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}, 'osd.4': {'kdb': True, 'sha1': 'distro'}, 'osd.5': {'kdb': True, 'sha1': 'distro'}, 'osd.6': {'kdb': True, 'sha1': 'distro'}, 'osd.7': {'kdb': True, 'sha1': 'distro'}, 'client.1': {'kdb': True, 'sha1': 'distro'}, 'prometheus.a': {'kdb': True, 'sha1': 'distro'}, 'grafana.a': {'kdb': True, 'sha1': 'distro'}, 'node-exporter.b': {'kdb': True, 'sha1': 'distro'}, 'ceph.iscsi.iscsi.a': {'kdb': True, 'sha1': 'distro'}} 2021-03-24T21:34:48.656 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-03-24T21:34:48.657 DEBUG:teuthology.orchestra.run.smithi125:> uname -r 2021-03-24T21:34:48.664 INFO:teuthology.orchestra.run.smithi125.stdout:5.4.0-66-generic 2021-03-24T21:34:48.664 INFO:teuthology.task.kernel:Running kernel on smithi125: 5.4.0-66-generic 2021-03-24T21:34:48.665 DEBUG:teuthology.orchestra.run.smithi125:> sudo apt-get clean 2021-03-24T21:34:48.758 DEBUG:teuthology.orchestra.run.smithi125:> sudo apt-get update 2021-03-24T21:34:49.043 INFO:teuthology.orchestra.run.smithi125.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2021-03-24T21:34:49.043 INFO:teuthology.orchestra.run.smithi125.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [109 kB] 2021-03-24T21:34:49.103 INFO:teuthology.orchestra.run.smithi125.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2021-03-24T21:34:49.431 INFO:teuthology.orchestra.run.smithi125.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [101 kB] 2021-03-24T21:34:49.504 INFO:teuthology.orchestra.run.smithi125.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [574 kB] 2021-03-24T21:34:49.637 INFO:teuthology.orchestra.run.smithi125.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [889 kB] 2021-03-24T21:34:49.817 INFO:teuthology.orchestra.run.smithi125.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [210 kB] 2021-03-24T21:34:49.828 INFO:teuthology.orchestra.run.smithi125.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [120 kB] 2021-03-24T21:34:49.833 INFO:teuthology.orchestra.run.smithi125.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [7,376 B] 2021-03-24T21:34:49.834 INFO:teuthology.orchestra.run.smithi125.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [148 kB] 2021-03-24T21:34:49.847 INFO:teuthology.orchestra.run.smithi125.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [21.6 kB] 2021-03-24T21:34:49.848 INFO:teuthology.orchestra.run.smithi125.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [550 kB] 2021-03-24T21:34:49.914 INFO:teuthology.orchestra.run.smithi125.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [443 kB] 2021-03-24T21:34:49.931 INFO:teuthology.orchestra.run.smithi125.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [436 kB] 2021-03-24T21:34:49.937 INFO:teuthology.orchestra.run.smithi125.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [207 kB] 2021-03-24T21:34:49.949 INFO:teuthology.orchestra.run.smithi125.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [12.9 kB] 2021-03-24T21:34:49.950 INFO:teuthology.orchestra.run.smithi125.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [171 kB] 2021-03-24T21:34:49.953 INFO:teuthology.orchestra.run.smithi125.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [81.2 kB] 2021-03-24T21:34:49.956 INFO:teuthology.orchestra.run.smithi125.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [10.6 kB] 2021-03-24T21:34:49.960 INFO:teuthology.orchestra.run.smithi125.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [25.3 kB] 2021-03-24T21:34:49.961 INFO:teuthology.orchestra.run.smithi125.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [560 kB] 2021-03-24T21:34:50.015 INFO:teuthology.orchestra.run.smithi125.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [755 kB] 2021-03-24T21:34:50.050 INFO:teuthology.orchestra.run.smithi125.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [159 kB] 2021-03-24T21:34:50.057 INFO:teuthology.orchestra.run.smithi125.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [16.4 kB] 2021-03-24T21:34:50.058 INFO:teuthology.orchestra.run.smithi125.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [2,932 B] 2021-03-24T21:34:50.059 INFO:teuthology.orchestra.run.smithi125.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [4,032 B] 2021-03-24T21:34:50.100 INFO:teuthology.orchestra.run.smithi125.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [224 B] 2021-03-24T21:34:53.104 INFO:teuthology.orchestra.run.smithi125.stdout:Fetched 5,728 kB in 2s (3,712 kB/s) 2021-03-24T21:34:54.209 INFO:teuthology.orchestra.run.smithi125.stdout:Reading package lists... 2021-03-24T21:34:54.230 DEBUG:teuthology.orchestra.run.smithi125:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2021-03-24T21:34:54.295 INFO:teuthology.orchestra.run.smithi125.stdout:Reading package lists... 2021-03-24T21:34:54.446 INFO:teuthology.orchestra.run.smithi125.stdout:Building dependency tree... 2021-03-24T21:34:54.447 INFO:teuthology.orchestra.run.smithi125.stdout:Reading state information... 2021-03-24T21:34:54.524 INFO:teuthology.orchestra.run.smithi125.stderr:E: Unable to locate package linux-image-current-generic 2021-03-24T21:34:54.525 DEBUG:teuthology.orchestra.run:got remote process result: 100 2021-03-24T21:34:54.527 DEBUG:teuthology.orchestra.run.smithi125:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2021-03-24T21:34:54.593 INFO:teuthology.orchestra.run.smithi125.stdout:Reading package lists... 2021-03-24T21:34:54.745 INFO:teuthology.orchestra.run.smithi125.stdout:Building dependency tree... 2021-03-24T21:34:54.745 INFO:teuthology.orchestra.run.smithi125.stdout:Reading state information... 2021-03-24T21:34:54.877 INFO:teuthology.orchestra.run.smithi125.stdout:The following additional packages will be installed: 2021-03-24T21:34:54.878 INFO:teuthology.orchestra.run.smithi125.stdout: linux-generic linux-headers-5.4.0-70 linux-headers-5.4.0-70-generic 2021-03-24T21:34:54.878 INFO:teuthology.orchestra.run.smithi125.stdout: linux-headers-generic linux-image-5.4.0-70-generic 2021-03-24T21:34:54.879 INFO:teuthology.orchestra.run.smithi125.stdout: linux-modules-5.4.0-70-generic linux-modules-extra-5.4.0-70-generic 2021-03-24T21:34:54.879 INFO:teuthology.orchestra.run.smithi125.stdout:Suggested packages: 2021-03-24T21:34:54.880 INFO:teuthology.orchestra.run.smithi125.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2021-03-24T21:34:54.880 INFO:teuthology.orchestra.run.smithi125.stdout:Recommended packages: 2021-03-24T21:34:54.880 INFO:teuthology.orchestra.run.smithi125.stdout: thermald 2021-03-24T21:34:54.908 INFO:teuthology.orchestra.run.smithi125.stdout:The following NEW packages will be installed: 2021-03-24T21:34:54.908 INFO:teuthology.orchestra.run.smithi125.stdout: linux-headers-5.4.0-70 linux-headers-5.4.0-70-generic 2021-03-24T21:34:54.909 INFO:teuthology.orchestra.run.smithi125.stdout: linux-image-5.4.0-70-generic linux-modules-5.4.0-70-generic 2021-03-24T21:34:54.909 INFO:teuthology.orchestra.run.smithi125.stdout: linux-modules-extra-5.4.0-70-generic 2021-03-24T21:34:54.910 INFO:teuthology.orchestra.run.smithi125.stdout:The following packages will be upgraded: 2021-03-24T21:34:54.910 INFO:teuthology.orchestra.run.smithi125.stdout: linux-generic linux-headers-generic linux-image-generic 2021-03-24T21:34:55.084 INFO:teuthology.orchestra.run.smithi125.stdout:3 upgraded, 5 newly installed, 0 to remove and 35 not upgraded. 2021-03-24T21:34:55.084 INFO:teuthology.orchestra.run.smithi125.stdout:Need to get 75.5 MB of archives. 2021-03-24T21:34:55.085 INFO:teuthology.orchestra.run.smithi125.stdout:After this operation, 377 MB of additional disk space will be used. 2021-03-24T21:34:55.085 INFO:teuthology.orchestra.run.smithi125.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-70-generic amd64 5.4.0-70.78 [14.8 MB] 2021-03-24T21:34:56.183 INFO:teuthology.orchestra.run.smithi125.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-70-generic amd64 5.4.0-70.78 [8,971 kB] 2021-03-24T21:34:56.433 INFO:teuthology.orchestra.run.smithi125.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-70-generic amd64 5.4.0-70.78 [39.4 MB] 2021-03-24T21:34:57.644 INFO:teuthology.orchestra.run.smithi125.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.70.73 [1,896 B] 2021-03-24T21:34:57.644 INFO:teuthology.orchestra.run.smithi125.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.70.73 [2,560 B] 2021-03-24T21:34:57.645 INFO:teuthology.orchestra.run.smithi125.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-70 all 5.4.0-70.78 [11.0 MB] 2021-03-24T21:34:57.976 INFO:teuthology.orchestra.run.smithi125.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-70-generic amd64 5.4.0-70.78 [1,400 kB] 2021-03-24T21:34:57.995 INFO:teuthology.orchestra.run.smithi125.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.70.73 [2,428 B] 2021-03-24T21:34:58.426 INFO:teuthology.orchestra.run.smithi125.stdout:Fetched 75.5 MB in 3s (24.5 MB/s) 2021-03-24T21:34:58.631 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package linux-modules-5.4.0-70-generic. 2021-03-24T21:35:00.033 INFO:teuthology.orchestra.run.smithi125.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 101691 files and directories currently installed.) 2021-03-24T21:35:00.038 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../0-linux-modules-5.4.0-70-generic_5.4.0-70.78_amd64.deb ... 2021-03-24T21:35:00.119 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking linux-modules-5.4.0-70-generic (5.4.0-70.78) ... 2021-03-24T21:35:02.084 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package linux-image-5.4.0-70-generic. 2021-03-24T21:35:02.100 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../1-linux-image-5.4.0-70-generic_5.4.0-70.78_amd64.deb ... 2021-03-24T21:35:02.196 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking linux-image-5.4.0-70-generic (5.4.0-70.78) ... 2021-03-24T21:35:03.209 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-70-generic. 2021-03-24T21:35:03.222 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-70-generic_5.4.0-70.78_amd64.deb ... 2021-03-24T21:35:03.267 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking linux-modules-extra-5.4.0-70-generic (5.4.0-70.78) ... 2021-03-24T21:35:08.484 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../3-linux-generic_5.4.0.70.73_amd64.deb ... 2021-03-24T21:35:08.604 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking linux-generic (5.4.0.70.73) over (5.4.0.66.69) ... 2021-03-24T21:35:08.935 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.70.73_amd64.deb ... 2021-03-24T21:35:09.076 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking linux-image-generic (5.4.0.70.73) over (5.4.0.66.69) ... 2021-03-24T21:35:09.406 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package linux-headers-5.4.0-70. 2021-03-24T21:35:09.421 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../5-linux-headers-5.4.0-70_5.4.0-70.78_all.deb ... 2021-03-24T21:35:09.458 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking linux-headers-5.4.0-70 (5.4.0-70.78) ... 2021-03-24T21:35:15.620 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package linux-headers-5.4.0-70-generic. 2021-03-24T21:35:15.642 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../6-linux-headers-5.4.0-70-generic_5.4.0-70.78_amd64.deb ... 2021-03-24T21:35:15.687 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking linux-headers-5.4.0-70-generic (5.4.0-70.78) ... 2021-03-24T21:35:18.226 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.70.73_amd64.deb ... 2021-03-24T21:35:18.379 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking linux-headers-generic (5.4.0.70.73) over (5.4.0.66.69) ... 2021-03-24T21:35:18.712 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up linux-headers-5.4.0-70 (5.4.0-70.78) ... 2021-03-24T21:35:18.849 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up linux-modules-5.4.0-70-generic (5.4.0-70.78) ... 2021-03-24T21:35:20.052 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up linux-image-5.4.0-70-generic (5.4.0-70.78) ... 2021-03-24T21:35:21.166 INFO:teuthology.orchestra.run.smithi125.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-70-generic 2021-03-24T21:35:21.167 INFO:teuthology.orchestra.run.smithi125.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-70-generic 2021-03-24T21:35:21.296 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up linux-headers-5.4.0-70-generic (5.4.0-70.78) ... 2021-03-24T21:35:21.422 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up linux-headers-generic (5.4.0.70.73) ... 2021-03-24T21:35:21.548 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up linux-modules-extra-5.4.0-70-generic (5.4.0-70.78) ... 2021-03-24T21:35:23.024 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up linux-image-generic (5.4.0.70.73) ... 2021-03-24T21:35:23.150 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up linux-generic (5.4.0.70.73) ... 2021-03-24T21:35:23.302 INFO:teuthology.orchestra.run.smithi125.stdout:Processing triggers for linux-image-5.4.0-70-generic (5.4.0-70.78) ... 2021-03-24T21:35:23.492 INFO:teuthology.orchestra.run.smithi125.stdout:/etc/kernel/postinst.d/initramfs-tools: 2021-03-24T21:35:23.492 INFO:teuthology.orchestra.run.smithi125.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-70-generic 2021-03-24T21:35:45.282 INFO:teuthology.orchestra.run.smithi125.stdout:/etc/kernel/postinst.d/zz-update-grub: 2021-03-24T21:35:45.283 INFO:teuthology.orchestra.run.smithi125.stdout:Sourcing file `/etc/default/grub' 2021-03-24T21:35:45.296 INFO:teuthology.orchestra.run.smithi125.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2021-03-24T21:35:45.315 INFO:teuthology.orchestra.run.smithi125.stdout:Generating grub configuration file ... 2021-03-24T21:35:45.629 INFO:teuthology.orchestra.run.smithi125.stdout:Found linux image: /boot/vmlinuz-5.4.0-70-generic 2021-03-24T21:35:45.655 INFO:teuthology.orchestra.run.smithi125.stdout:Found initrd image: /boot/initrd.img-5.4.0-70-generic 2021-03-24T21:35:46.021 INFO:teuthology.orchestra.run.smithi125.stdout:Found linux image: /boot/vmlinuz-5.4.0-66-generic 2021-03-24T21:35:46.032 INFO:teuthology.orchestra.run.smithi125.stdout:Found initrd image: /boot/initrd.img-5.4.0-66-generic 2021-03-24T21:35:47.765 INFO:teuthology.orchestra.run.smithi125.stdout:done 2021-03-24T21:35:48.129 DEBUG:teuthology.orchestra.run.smithi125:> dpkg -s linux-image-generic 2021-03-24T21:35:48.154 INFO:teuthology.orchestra.run.smithi125.stdout:Package: linux-image-generic 2021-03-24T21:35:48.154 INFO:teuthology.orchestra.run.smithi125.stdout:Status: install ok installed 2021-03-24T21:35:48.155 INFO:teuthology.orchestra.run.smithi125.stdout:Priority: optional 2021-03-24T21:35:48.155 INFO:teuthology.orchestra.run.smithi125.stdout:Section: kernel 2021-03-24T21:35:48.155 INFO:teuthology.orchestra.run.smithi125.stdout:Installed-Size: 18 2021-03-24T21:35:48.155 INFO:teuthology.orchestra.run.smithi125.stdout:Maintainer: Ubuntu Kernel Team 2021-03-24T21:35:48.156 INFO:teuthology.orchestra.run.smithi125.stdout:Architecture: amd64 2021-03-24T21:35:48.156 INFO:teuthology.orchestra.run.smithi125.stdout:Source: linux-meta 2021-03-24T21:35:48.156 INFO:teuthology.orchestra.run.smithi125.stdout:Version: 5.4.0.70.73 2021-03-24T21:35:48.156 INFO:teuthology.orchestra.run.smithi125.stdout:Provides: virtualbox-guest-modules (= 6.1.16-dfsg-6~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.6) 2021-03-24T21:35:48.157 INFO:teuthology.orchestra.run.smithi125.stdout:Depends: linux-image-5.4.0-70-generic, linux-modules-extra-5.4.0-70-generic, linux-firmware, intel-microcode, amd64-microcode 2021-03-24T21:35:48.157 INFO:teuthology.orchestra.run.smithi125.stdout:Recommends: thermald 2021-03-24T21:35:48.157 INFO:teuthology.orchestra.run.smithi125.stdout:Description: Generic Linux kernel image 2021-03-24T21:35:48.157 INFO:teuthology.orchestra.run.smithi125.stdout: This package will always depend on the latest generic kernel image 2021-03-24T21:35:48.158 INFO:teuthology.orchestra.run.smithi125.stdout: available. 2021-03-24T21:35:48.158 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-66-generic Expected: 5.4.0-70-generic 2021-03-24T21:35:48.159 DEBUG:teuthology.orchestra.run.smithi142:> uname -r 2021-03-24T21:35:48.166 INFO:teuthology.orchestra.run.smithi142.stdout:5.4.0-66-generic 2021-03-24T21:35:48.167 INFO:teuthology.task.kernel:Running kernel on smithi142: 5.4.0-66-generic 2021-03-24T21:35:48.167 DEBUG:teuthology.orchestra.run.smithi142:> sudo apt-get clean 2021-03-24T21:35:48.272 DEBUG:teuthology.orchestra.run.smithi142:> sudo apt-get update 2021-03-24T21:35:48.417 INFO:teuthology.orchestra.run.smithi142.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [109 kB] 2021-03-24T21:35:48.532 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2021-03-24T21:35:48.613 INFO:teuthology.orchestra.run.smithi142.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2021-03-24T21:35:48.706 INFO:teuthology.orchestra.run.smithi142.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [574 kB] 2021-03-24T21:35:48.780 INFO:teuthology.orchestra.run.smithi142.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [210 kB] 2021-03-24T21:35:48.784 INFO:teuthology.orchestra.run.smithi142.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [120 kB] 2021-03-24T21:35:48.787 INFO:teuthology.orchestra.run.smithi142.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [7,376 B] 2021-03-24T21:35:48.788 INFO:teuthology.orchestra.run.smithi142.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [148 kB] 2021-03-24T21:35:48.790 INFO:teuthology.orchestra.run.smithi142.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [21.6 kB] 2021-03-24T21:35:48.791 INFO:teuthology.orchestra.run.smithi142.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [550 kB] 2021-03-24T21:35:48.800 INFO:teuthology.orchestra.run.smithi142.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [436 kB] 2021-03-24T21:35:48.804 INFO:teuthology.orchestra.run.smithi142.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [81.2 kB] 2021-03-24T21:35:48.806 INFO:teuthology.orchestra.run.smithi142.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [10.6 kB] 2021-03-24T21:35:48.948 INFO:teuthology.orchestra.run.smithi142.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-backports InRelease [101 kB] 2021-03-24T21:35:49.073 INFO:teuthology.orchestra.run.smithi142.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [443 kB] 2021-03-24T21:35:49.283 INFO:teuthology.orchestra.run.smithi142.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [889 kB] 2021-03-24T21:35:49.391 INFO:teuthology.orchestra.run.smithi142.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [207 kB] 2021-03-24T21:35:49.413 INFO:teuthology.orchestra.run.smithi142.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [12.9 kB] 2021-03-24T21:35:49.413 INFO:teuthology.orchestra.run.smithi142.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [171 kB] 2021-03-24T21:35:49.422 INFO:teuthology.orchestra.run.smithi142.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [25.3 kB] 2021-03-24T21:35:49.423 INFO:teuthology.orchestra.run.smithi142.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [560 kB] 2021-03-24T21:35:49.457 INFO:teuthology.orchestra.run.smithi142.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [755 kB] 2021-03-24T21:35:49.496 INFO:teuthology.orchestra.run.smithi142.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [159 kB] 2021-03-24T21:35:49.502 INFO:teuthology.orchestra.run.smithi142.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [16.4 kB] 2021-03-24T21:35:49.503 INFO:teuthology.orchestra.run.smithi142.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [2,932 B] 2021-03-24T21:35:49.503 INFO:teuthology.orchestra.run.smithi142.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [4,032 B] 2021-03-24T21:35:49.552 INFO:teuthology.orchestra.run.smithi142.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [224 B] 2021-03-24T21:35:52.413 INFO:teuthology.orchestra.run.smithi142.stdout:Fetched 5,728 kB in 1s (4,241 kB/s) 2021-03-24T21:35:53.574 INFO:teuthology.orchestra.run.smithi142.stdout:Reading package lists... 2021-03-24T21:35:53.598 DEBUG:teuthology.orchestra.run.smithi142:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2021-03-24T21:35:53.669 INFO:teuthology.orchestra.run.smithi142.stdout:Reading package lists... 2021-03-24T21:35:53.824 INFO:teuthology.orchestra.run.smithi142.stdout:Building dependency tree... 2021-03-24T21:35:53.824 INFO:teuthology.orchestra.run.smithi142.stdout:Reading state information... 2021-03-24T21:35:53.899 INFO:teuthology.orchestra.run.smithi142.stderr:E: Unable to locate package linux-image-current-generic 2021-03-24T21:35:53.901 DEBUG:teuthology.orchestra.run:got remote process result: 100 2021-03-24T21:35:53.902 DEBUG:teuthology.orchestra.run.smithi142:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2021-03-24T21:35:53.966 INFO:teuthology.orchestra.run.smithi142.stdout:Reading package lists... 2021-03-24T21:35:54.117 INFO:teuthology.orchestra.run.smithi142.stdout:Building dependency tree... 2021-03-24T21:35:54.118 INFO:teuthology.orchestra.run.smithi142.stdout:Reading state information... 2021-03-24T21:35:54.250 INFO:teuthology.orchestra.run.smithi142.stdout:The following additional packages will be installed: 2021-03-24T21:35:54.251 INFO:teuthology.orchestra.run.smithi142.stdout: linux-generic linux-headers-5.4.0-70 linux-headers-5.4.0-70-generic 2021-03-24T21:35:54.251 INFO:teuthology.orchestra.run.smithi142.stdout: linux-headers-generic linux-image-5.4.0-70-generic 2021-03-24T21:35:54.251 INFO:teuthology.orchestra.run.smithi142.stdout: linux-modules-5.4.0-70-generic linux-modules-extra-5.4.0-70-generic 2021-03-24T21:35:54.252 INFO:teuthology.orchestra.run.smithi142.stdout:Suggested packages: 2021-03-24T21:35:54.252 INFO:teuthology.orchestra.run.smithi142.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2021-03-24T21:35:54.252 INFO:teuthology.orchestra.run.smithi142.stdout:Recommended packages: 2021-03-24T21:35:54.252 INFO:teuthology.orchestra.run.smithi142.stdout: thermald 2021-03-24T21:35:54.281 INFO:teuthology.orchestra.run.smithi142.stdout:The following NEW packages will be installed: 2021-03-24T21:35:54.281 INFO:teuthology.orchestra.run.smithi142.stdout: linux-headers-5.4.0-70 linux-headers-5.4.0-70-generic 2021-03-24T21:35:54.281 INFO:teuthology.orchestra.run.smithi142.stdout: linux-image-5.4.0-70-generic linux-modules-5.4.0-70-generic 2021-03-24T21:35:54.281 INFO:teuthology.orchestra.run.smithi142.stdout: linux-modules-extra-5.4.0-70-generic 2021-03-24T21:35:54.282 INFO:teuthology.orchestra.run.smithi142.stdout:The following packages will be upgraded: 2021-03-24T21:35:54.282 INFO:teuthology.orchestra.run.smithi142.stdout: linux-generic linux-headers-generic linux-image-generic 2021-03-24T21:35:54.459 INFO:teuthology.orchestra.run.smithi142.stdout:3 upgraded, 5 newly installed, 0 to remove and 35 not upgraded. 2021-03-24T21:35:54.459 INFO:teuthology.orchestra.run.smithi142.stdout:Need to get 75.5 MB of archives. 2021-03-24T21:35:54.460 INFO:teuthology.orchestra.run.smithi142.stdout:After this operation, 377 MB of additional disk space will be used. 2021-03-24T21:35:54.460 INFO:teuthology.orchestra.run.smithi142.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-70-generic amd64 5.4.0-70.78 [14.8 MB] 2021-03-24T21:35:55.526 INFO:teuthology.orchestra.run.smithi142.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-70-generic amd64 5.4.0-70.78 [8,971 kB] 2021-03-24T21:35:55.780 INFO:teuthology.orchestra.run.smithi142.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-70-generic amd64 5.4.0-70.78 [39.4 MB] 2021-03-24T21:35:57.040 INFO:teuthology.orchestra.run.smithi142.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.70.73 [1,896 B] 2021-03-24T21:35:57.041 INFO:teuthology.orchestra.run.smithi142.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.70.73 [2,560 B] 2021-03-24T21:35:57.041 INFO:teuthology.orchestra.run.smithi142.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-70 all 5.4.0-70.78 [11.0 MB] 2021-03-24T21:35:57.686 INFO:teuthology.orchestra.run.smithi142.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-70-generic amd64 5.4.0-70.78 [1,400 kB] 2021-03-24T21:35:57.749 INFO:teuthology.orchestra.run.smithi142.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.70.73 [2,428 B] 2021-03-24T21:35:58.195 INFO:teuthology.orchestra.run.smithi142.stdout:Fetched 75.5 MB in 3s (21.8 MB/s) 2021-03-24T21:35:58.408 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package linux-modules-5.4.0-70-generic. 2021-03-24T21:35:59.820 INFO:teuthology.orchestra.run.smithi142.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 101691 files and directories currently installed.) 2021-03-24T21:35:59.824 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../0-linux-modules-5.4.0-70-generic_5.4.0-70.78_amd64.deb ... 2021-03-24T21:35:59.979 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking linux-modules-5.4.0-70-generic (5.4.0-70.78) ... 2021-03-24T21:36:02.041 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package linux-image-5.4.0-70-generic. 2021-03-24T21:36:02.057 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../1-linux-image-5.4.0-70-generic_5.4.0-70.78_amd64.deb ... 2021-03-24T21:36:02.176 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking linux-image-5.4.0-70-generic (5.4.0-70.78) ... 2021-03-24T21:36:03.196 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-70-generic. 2021-03-24T21:36:03.211 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-70-generic_5.4.0-70.78_amd64.deb ... 2021-03-24T21:36:03.246 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking linux-modules-extra-5.4.0-70-generic (5.4.0-70.78) ... 2021-03-24T21:36:08.503 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../3-linux-generic_5.4.0.70.73_amd64.deb ... 2021-03-24T21:36:08.624 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking linux-generic (5.4.0.70.73) over (5.4.0.66.69) ... 2021-03-24T21:36:08.991 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.70.73_amd64.deb ... 2021-03-24T21:36:09.140 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking linux-image-generic (5.4.0.70.73) over (5.4.0.66.69) ... 2021-03-24T21:36:09.543 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package linux-headers-5.4.0-70. 2021-03-24T21:36:09.560 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../5-linux-headers-5.4.0-70_5.4.0-70.78_all.deb ... 2021-03-24T21:36:09.599 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking linux-headers-5.4.0-70 (5.4.0-70.78) ... 2021-03-24T21:36:16.055 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package linux-headers-5.4.0-70-generic. 2021-03-24T21:36:16.078 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../6-linux-headers-5.4.0-70-generic_5.4.0-70.78_amd64.deb ... 2021-03-24T21:36:16.121 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking linux-headers-5.4.0-70-generic (5.4.0-70.78) ... 2021-03-24T21:36:19.066 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.70.73_amd64.deb ... 2021-03-24T21:36:19.193 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking linux-headers-generic (5.4.0.70.73) over (5.4.0.66.69) ... 2021-03-24T21:36:19.564 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up linux-headers-5.4.0-70 (5.4.0-70.78) ... 2021-03-24T21:36:19.708 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up linux-modules-5.4.0-70-generic (5.4.0-70.78) ... 2021-03-24T21:36:20.960 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up linux-image-5.4.0-70-generic (5.4.0-70.78) ... 2021-03-24T21:36:22.190 INFO:teuthology.orchestra.run.smithi142.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-70-generic 2021-03-24T21:36:22.190 INFO:teuthology.orchestra.run.smithi142.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-70-generic 2021-03-24T21:36:22.354 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up linux-headers-5.4.0-70-generic (5.4.0-70.78) ... 2021-03-24T21:36:22.455 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up linux-headers-generic (5.4.0.70.73) ... 2021-03-24T21:36:22.606 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up linux-modules-extra-5.4.0-70-generic (5.4.0-70.78) ... 2021-03-24T21:36:23.815 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up linux-image-generic (5.4.0.70.73) ... 2021-03-24T21:36:23.934 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up linux-generic (5.4.0.70.73) ... 2021-03-24T21:36:24.068 INFO:teuthology.orchestra.run.smithi142.stdout:Processing triggers for linux-image-5.4.0-70-generic (5.4.0-70.78) ... 2021-03-24T21:36:24.224 INFO:teuthology.orchestra.run.smithi142.stdout:/etc/kernel/postinst.d/initramfs-tools: 2021-03-24T21:36:24.225 INFO:teuthology.orchestra.run.smithi142.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-70-generic 2021-03-24T21:36:46.063 INFO:teuthology.orchestra.run.smithi142.stdout:/etc/kernel/postinst.d/zz-update-grub: 2021-03-24T21:36:46.063 INFO:teuthology.orchestra.run.smithi142.stdout:Sourcing file `/etc/default/grub' 2021-03-24T21:36:46.070 INFO:teuthology.orchestra.run.smithi142.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2021-03-24T21:36:46.090 INFO:teuthology.orchestra.run.smithi142.stdout:Generating grub configuration file ... 2021-03-24T21:36:46.453 INFO:teuthology.orchestra.run.smithi142.stdout:Found linux image: /boot/vmlinuz-5.4.0-70-generic 2021-03-24T21:36:46.484 INFO:teuthology.orchestra.run.smithi142.stdout:Found initrd image: /boot/initrd.img-5.4.0-70-generic 2021-03-24T21:36:46.942 INFO:teuthology.orchestra.run.smithi142.stdout:Found linux image: /boot/vmlinuz-5.4.0-66-generic 2021-03-24T21:36:46.954 INFO:teuthology.orchestra.run.smithi142.stdout:Found initrd image: /boot/initrd.img-5.4.0-66-generic 2021-03-24T21:36:49.557 INFO:teuthology.orchestra.run.smithi142.stdout:done 2021-03-24T21:36:49.955 DEBUG:teuthology.orchestra.run.smithi142:> dpkg -s linux-image-generic 2021-03-24T21:36:49.992 INFO:teuthology.orchestra.run.smithi142.stdout:Package: linux-image-generic 2021-03-24T21:36:49.992 INFO:teuthology.orchestra.run.smithi142.stdout:Status: install ok installed 2021-03-24T21:36:49.993 INFO:teuthology.orchestra.run.smithi142.stdout:Priority: optional 2021-03-24T21:36:49.993 INFO:teuthology.orchestra.run.smithi142.stdout:Section: kernel 2021-03-24T21:36:49.993 INFO:teuthology.orchestra.run.smithi142.stdout:Installed-Size: 18 2021-03-24T21:36:49.993 INFO:teuthology.orchestra.run.smithi142.stdout:Maintainer: Ubuntu Kernel Team 2021-03-24T21:36:49.993 INFO:teuthology.orchestra.run.smithi142.stdout:Architecture: amd64 2021-03-24T21:36:49.994 INFO:teuthology.orchestra.run.smithi142.stdout:Source: linux-meta 2021-03-24T21:36:49.994 INFO:teuthology.orchestra.run.smithi142.stdout:Version: 5.4.0.70.73 2021-03-24T21:36:49.994 INFO:teuthology.orchestra.run.smithi142.stdout:Provides: virtualbox-guest-modules (= 6.1.16-dfsg-6~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.6) 2021-03-24T21:36:49.994 INFO:teuthology.orchestra.run.smithi142.stdout:Depends: linux-image-5.4.0-70-generic, linux-modules-extra-5.4.0-70-generic, linux-firmware, intel-microcode, amd64-microcode 2021-03-24T21:36:49.995 INFO:teuthology.orchestra.run.smithi142.stdout:Recommends: thermald 2021-03-24T21:36:49.995 INFO:teuthology.orchestra.run.smithi142.stdout:Description: Generic Linux kernel image 2021-03-24T21:36:49.995 INFO:teuthology.orchestra.run.smithi142.stdout: This package will always depend on the latest generic kernel image 2021-03-24T21:36:49.995 INFO:teuthology.orchestra.run.smithi142.stdout: available. 2021-03-24T21:36:49.996 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-66-generic Expected: 5.4.0-70-generic 2021-03-24T21:36:49.996 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2021-03-24T21:36:49.996 DEBUG:teuthology.task.kernel:src is distro, skipping download 2021-03-24T21:36:49.997 DEBUG:teuthology.task.kernel:src is distro, skipping download 2021-03-24T21:36:49.997 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2021-03-24T21:36:49.997 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi125.front.sepia.ceph.com, path=None, version=distro) 2021-03-24T21:36:49.998 DEBUG:teuthology.orchestra.run.smithi125:> sudo apt-get clean 2021-03-24T21:36:50.119 DEBUG:teuthology.orchestra.run.smithi125:> sudo apt-get update 2021-03-24T21:36:50.255 INFO:teuthology.orchestra.run.smithi125.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2021-03-24T21:36:50.368 INFO:teuthology.orchestra.run.smithi125.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2021-03-24T21:36:50.448 INFO:teuthology.orchestra.run.smithi125.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2021-03-24T21:36:50.528 INFO:teuthology.orchestra.run.smithi125.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2021-03-24T21:36:51.708 INFO:teuthology.orchestra.run.smithi125.stdout:Reading package lists... 2021-03-24T21:36:51.730 DEBUG:teuthology.orchestra.run.smithi125:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2021-03-24T21:36:51.804 INFO:teuthology.orchestra.run.smithi125.stdout:Reading package lists... 2021-03-24T21:36:51.955 INFO:teuthology.orchestra.run.smithi125.stdout:Building dependency tree... 2021-03-24T21:36:51.956 INFO:teuthology.orchestra.run.smithi125.stdout:Reading state information... 2021-03-24T21:36:52.033 INFO:teuthology.orchestra.run.smithi125.stderr:E: Unable to locate package linux-image-current-generic 2021-03-24T21:36:52.034 DEBUG:teuthology.orchestra.run:got remote process result: 100 2021-03-24T21:36:52.035 DEBUG:teuthology.orchestra.run.smithi125:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2021-03-24T21:36:52.102 INFO:teuthology.orchestra.run.smithi125.stdout:Reading package lists... 2021-03-24T21:36:52.255 INFO:teuthology.orchestra.run.smithi125.stdout:Building dependency tree... 2021-03-24T21:36:52.255 INFO:teuthology.orchestra.run.smithi125.stdout:Reading state information... 2021-03-24T21:36:52.403 INFO:teuthology.orchestra.run.smithi125.stdout:linux-image-generic is already the newest version (5.4.0.70.73). 2021-03-24T21:36:52.403 INFO:teuthology.orchestra.run.smithi125.stdout:0 upgraded, 0 newly installed, 0 to remove and 35 not upgraded. 2021-03-24T21:36:52.405 DEBUG:teuthology.orchestra.run.smithi125:> dpkg -s linux-image-generic 2021-03-24T21:36:52.420 INFO:teuthology.orchestra.run.smithi125.stdout:Package: linux-image-generic 2021-03-24T21:36:52.420 INFO:teuthology.orchestra.run.smithi125.stdout:Status: install ok installed 2021-03-24T21:36:52.421 INFO:teuthology.orchestra.run.smithi125.stdout:Priority: optional 2021-03-24T21:36:52.421 INFO:teuthology.orchestra.run.smithi125.stdout:Section: kernel 2021-03-24T21:36:52.421 INFO:teuthology.orchestra.run.smithi125.stdout:Installed-Size: 18 2021-03-24T21:36:52.421 INFO:teuthology.orchestra.run.smithi125.stdout:Maintainer: Ubuntu Kernel Team 2021-03-24T21:36:52.421 INFO:teuthology.orchestra.run.smithi125.stdout:Architecture: amd64 2021-03-24T21:36:52.422 INFO:teuthology.orchestra.run.smithi125.stdout:Source: linux-meta 2021-03-24T21:36:52.422 INFO:teuthology.orchestra.run.smithi125.stdout:Version: 5.4.0.70.73 2021-03-24T21:36:52.422 INFO:teuthology.orchestra.run.smithi125.stdout:Provides: virtualbox-guest-modules (= 6.1.16-dfsg-6~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.6) 2021-03-24T21:36:52.422 INFO:teuthology.orchestra.run.smithi125.stdout:Depends: linux-image-5.4.0-70-generic, linux-modules-extra-5.4.0-70-generic, linux-firmware, intel-microcode, amd64-microcode 2021-03-24T21:36:52.422 INFO:teuthology.orchestra.run.smithi125.stdout:Recommends: thermald 2021-03-24T21:36:52.423 INFO:teuthology.orchestra.run.smithi125.stdout:Description: Generic Linux kernel image 2021-03-24T21:36:52.423 INFO:teuthology.orchestra.run.smithi125.stdout: This package will always depend on the latest generic kernel image 2021-03-24T21:36:52.423 INFO:teuthology.orchestra.run.smithi125.stdout: available. 2021-03-24T21:36:52.424 DEBUG:teuthology.orchestra.run.smithi125:> mktemp 2021-03-24T21:36:52.431 INFO:teuthology.orchestra.run.smithi125.stdout:/tmp/tmp.Apy8YQE02u 2021-03-24T21:36:52.432 DEBUG:teuthology.orchestra.run.smithi125:> sudo cp /boot/grub/grub.cfg /tmp/tmp.Apy8YQE02u 2021-03-24T21:36:52.488 DEBUG:teuthology.orchestra.run.smithi125:> sudo chmod 0666 /tmp/tmp.Apy8YQE02u 2021-03-24T21:36:52.531 DEBUG:teuthology.orchestra.remote:smithi125:/tmp/tmp.Apy8YQE02u is 8KB 2021-03-24T21:36:52.545 DEBUG:teuthology.orchestra.run.smithi125:> rm -fr /tmp/tmp.Apy8YQE02u 2021-03-24T21:36:52.552 DEBUG:teuthology.orchestra.run.smithi125:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2021-03-24T21:36:52.609 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2021-03-24T21:36:52.609 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd of=/etc/grub.d/01_ceph_kernel 2021-03-24T21:36:52.609 DEBUG:teuthology.orchestra.run.smithi125:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2021-03-24T21:36:52.663 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-70-generic 2021-03-24T21:36:52.663 DEBUG:teuthology.orchestra.run.smithi125:> sudo update-grub 2021-03-24T21:36:53.706 INFO:teuthology.orchestra.run.smithi125.stderr:Sourcing file `/etc/default/grub' 2021-03-24T21:36:53.708 INFO:teuthology.orchestra.run.smithi125.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2021-03-24T21:36:53.720 INFO:teuthology.orchestra.run.smithi125.stderr:Generating grub configuration file ... 2021-03-24T21:36:54.024 INFO:teuthology.orchestra.run.smithi125.stderr:Found linux image: /boot/vmlinuz-5.4.0-70-generic 2021-03-24T21:36:54.036 INFO:teuthology.orchestra.run.smithi125.stderr:Found initrd image: /boot/initrd.img-5.4.0-70-generic 2021-03-24T21:36:54.380 INFO:teuthology.orchestra.run.smithi125.stderr:Found linux image: /boot/vmlinuz-5.4.0-66-generic 2021-03-24T21:36:54.391 INFO:teuthology.orchestra.run.smithi125.stderr:Found initrd image: /boot/initrd.img-5.4.0-66-generic 2021-03-24T21:36:55.132 INFO:teuthology.orchestra.run.smithi125.stderr:done 2021-03-24T21:36:55.133 DEBUG:teuthology.orchestra.run.smithi125:> sudo shutdown -r now 2021-03-24T21:36:55.136 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2021-03-24T21:36:55.137 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi142.front.sepia.ceph.com, path=None, version=distro) 2021-03-24T21:36:55.137 DEBUG:teuthology.orchestra.run.smithi142:> sudo apt-get clean 2021-03-24T21:36:55.205 DEBUG:teuthology.orchestra.run.smithi142:> sudo apt-get update 2021-03-24T21:36:55.462 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2021-03-24T21:36:55.462 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2021-03-24T21:36:55.530 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2021-03-24T21:36:55.610 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2021-03-24T21:36:56.882 INFO:teuthology.orchestra.run.smithi142.stdout:Reading package lists... 2021-03-24T21:36:56.903 DEBUG:teuthology.orchestra.run.smithi142:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2021-03-24T21:36:56.970 INFO:teuthology.orchestra.run.smithi142.stdout:Reading package lists... 2021-03-24T21:36:57.121 INFO:teuthology.orchestra.run.smithi142.stdout:Building dependency tree... 2021-03-24T21:36:57.122 INFO:teuthology.orchestra.run.smithi142.stdout:Reading state information... 2021-03-24T21:36:57.199 INFO:teuthology.orchestra.run.smithi142.stderr:E: Unable to locate package linux-image-current-generic 2021-03-24T21:36:57.200 DEBUG:teuthology.orchestra.run:got remote process result: 100 2021-03-24T21:36:57.201 DEBUG:teuthology.orchestra.run.smithi142:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2021-03-24T21:36:57.268 INFO:teuthology.orchestra.run.smithi142.stdout:Reading package lists... 2021-03-24T21:36:57.417 INFO:teuthology.orchestra.run.smithi142.stdout:Building dependency tree... 2021-03-24T21:36:57.418 INFO:teuthology.orchestra.run.smithi142.stdout:Reading state information... 2021-03-24T21:36:57.554 INFO:teuthology.orchestra.run.smithi142.stdout:linux-image-generic is already the newest version (5.4.0.70.73). 2021-03-24T21:36:57.554 INFO:teuthology.orchestra.run.smithi142.stdout:0 upgraded, 0 newly installed, 0 to remove and 35 not upgraded. 2021-03-24T21:36:57.556 DEBUG:teuthology.orchestra.run.smithi142:> dpkg -s linux-image-generic 2021-03-24T21:36:57.577 INFO:teuthology.orchestra.run.smithi142.stdout:Package: linux-image-generic 2021-03-24T21:36:57.578 INFO:teuthology.orchestra.run.smithi142.stdout:Status: install ok installed 2021-03-24T21:36:57.578 INFO:teuthology.orchestra.run.smithi142.stdout:Priority: optional 2021-03-24T21:36:57.578 INFO:teuthology.orchestra.run.smithi142.stdout:Section: kernel 2021-03-24T21:36:57.578 INFO:teuthology.orchestra.run.smithi142.stdout:Installed-Size: 18 2021-03-24T21:36:57.579 INFO:teuthology.orchestra.run.smithi142.stdout:Maintainer: Ubuntu Kernel Team 2021-03-24T21:36:57.579 INFO:teuthology.orchestra.run.smithi142.stdout:Architecture: amd64 2021-03-24T21:36:57.579 INFO:teuthology.orchestra.run.smithi142.stdout:Source: linux-meta 2021-03-24T21:36:57.579 INFO:teuthology.orchestra.run.smithi142.stdout:Version: 5.4.0.70.73 2021-03-24T21:36:57.579 INFO:teuthology.orchestra.run.smithi142.stdout:Provides: virtualbox-guest-modules (= 6.1.16-dfsg-6~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.6) 2021-03-24T21:36:57.580 INFO:teuthology.orchestra.run.smithi142.stdout:Depends: linux-image-5.4.0-70-generic, linux-modules-extra-5.4.0-70-generic, linux-firmware, intel-microcode, amd64-microcode 2021-03-24T21:36:57.580 INFO:teuthology.orchestra.run.smithi142.stdout:Recommends: thermald 2021-03-24T21:36:57.580 INFO:teuthology.orchestra.run.smithi142.stdout:Description: Generic Linux kernel image 2021-03-24T21:36:57.580 INFO:teuthology.orchestra.run.smithi142.stdout: This package will always depend on the latest generic kernel image 2021-03-24T21:36:57.581 INFO:teuthology.orchestra.run.smithi142.stdout: available. 2021-03-24T21:36:57.581 DEBUG:teuthology.orchestra.run.smithi142:> mktemp 2021-03-24T21:36:57.626 INFO:teuthology.orchestra.run.smithi142.stdout:/tmp/tmp.xZqDE7Np8g 2021-03-24T21:36:57.627 DEBUG:teuthology.orchestra.run.smithi142:> sudo cp /boot/grub/grub.cfg /tmp/tmp.xZqDE7Np8g 2021-03-24T21:36:57.640 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod 0666 /tmp/tmp.xZqDE7Np8g 2021-03-24T21:36:57.718 DEBUG:teuthology.orchestra.remote:smithi142:/tmp/tmp.xZqDE7Np8g is 8KB 2021-03-24T21:36:57.733 DEBUG:teuthology.orchestra.run.smithi142:> rm -fr /tmp/tmp.xZqDE7Np8g 2021-03-24T21:36:57.741 DEBUG:teuthology.orchestra.run.smithi142:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2021-03-24T21:36:57.800 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2021-03-24T21:36:57.800 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/etc/grub.d/01_ceph_kernel 2021-03-24T21:36:57.801 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2021-03-24T21:36:57.841 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-70-generic 2021-03-24T21:36:57.841 DEBUG:teuthology.orchestra.run.smithi142:> sudo update-grub 2021-03-24T21:36:59.147 INFO:teuthology.orchestra.run.smithi142.stderr:Sourcing file `/etc/default/grub' 2021-03-24T21:36:59.149 INFO:teuthology.orchestra.run.smithi142.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2021-03-24T21:36:59.171 INFO:teuthology.orchestra.run.smithi142.stderr:Generating grub configuration file ... 2021-03-24T21:36:59.449 INFO:teuthology.orchestra.run.smithi142.stderr:Found linux image: /boot/vmlinuz-5.4.0-70-generic 2021-03-24T21:36:59.460 INFO:teuthology.orchestra.run.smithi142.stderr:Found initrd image: /boot/initrd.img-5.4.0-70-generic 2021-03-24T21:36:59.858 INFO:teuthology.orchestra.run.smithi142.stderr:Found linux image: /boot/vmlinuz-5.4.0-66-generic 2021-03-24T21:36:59.869 INFO:teuthology.orchestra.run.smithi142.stderr:Found initrd image: /boot/initrd.img-5.4.0-66-generic 2021-03-24T21:37:01.532 INFO:teuthology.orchestra.run.smithi142.stderr:done 2021-03-24T21:37:01.534 DEBUG:teuthology.orchestra.run.smithi142:> sudo shutdown -r now 2021-03-24T21:37:31.566 INFO:teuthology.misc:Re-opening connections... 2021-03-24T21:37:31.567 INFO:teuthology.misc:trying to connect to ubuntu@smithi125.front.sepia.ceph.com 2021-03-24T21:37:31.568 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-24T21:37:31.569 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi125.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-24T21:37:50.019 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.125 2021-03-24T21:37:50.024 INFO:teuthology.misc:trying to connect to ubuntu@smithi142.front.sepia.ceph.com 2021-03-24T21:37:50.025 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-24T21:37:50.026 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-24T21:38:50.087 DEBUG:teuthology.orchestra.remote:timed out 2021-03-24T21:38:50.089 DEBUG:teuthology.misc:waited 78.52180004119873 2021-03-24T21:38:51.090 INFO:teuthology.misc:trying to connect to ubuntu@smithi125.front.sepia.ceph.com 2021-03-24T21:38:51.091 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-24T21:38:51.092 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi125.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-24T21:38:51.272 DEBUG:teuthology.orchestra.run.smithi125:> true 2021-03-24T21:38:51.856 DEBUG:teuthology.misc:waited 80.2888536453247 2021-03-24T21:38:52.857 INFO:teuthology.misc:trying to connect to ubuntu@smithi142.front.sepia.ceph.com 2021-03-24T21:38:52.858 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-24T21:38:52.859 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-24T21:38:53.038 DEBUG:teuthology.orchestra.run.smithi142:> true 2021-03-24T21:38:53.593 DEBUG:teuthology.misc:waited 82.02543807029724 2021-03-24T21:38:54.594 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2021-03-24T21:38:54.594 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.4.0-70-generic"... 2021-03-24T21:38:54.595 DEBUG:teuthology.orchestra.run.smithi125:> uname -r 2021-03-24T21:38:54.603 INFO:teuthology.orchestra.run.smithi125.stdout:5.4.0-70-generic 2021-03-24T21:38:54.603 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-70-generic vs 5.4.0-70-generic 2021-03-24T21:38:54.604 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2021-03-24T21:38:54.604 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2021-03-24T21:38:54.605 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2021-03-24T21:38:54.605 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.4.0-70-generic"... 2021-03-24T21:38:54.605 DEBUG:teuthology.orchestra.run.smithi142:> uname -r 2021-03-24T21:38:54.612 INFO:teuthology.orchestra.run.smithi142.stdout:5.4.0-70-generic 2021-03-24T21:38:54.612 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-70-generic vs 5.4.0-70-generic 2021-03-24T21:38:54.613 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2021-03-24T21:38:54.613 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2021-03-24T21:38:55.614 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2021-03-24T21:38:55.614 DEBUG:teuthology.orchestra.run.smithi125:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-03-24T21:38:55.670 INFO:teuthology.orchestra.run.smithi125.stdout:ttyS1 2021-03-24T21:38:55.709 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2021-03-24T21:38:55.709 DEBUG:teuthology.orchestra.run.smithi142:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-03-24T21:38:55.765 INFO:teuthology.orchestra.run.smithi142.stdout:ttyS1 2021-03-24T21:38:55.783 INFO:teuthology.run_tasks:Running task internal.base... 2021-03-24T21:38:55.792 INFO:teuthology.task.internal:Creating test directory... 2021-03-24T21:38:55.793 DEBUG:teuthology.orchestra.run.smithi125:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-03-24T21:38:55.796 DEBUG:teuthology.orchestra.run.smithi142:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-03-24T21:38:55.803 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2021-03-24T21:38:55.897 INFO:teuthology.run_tasks:Running task internal.archive... 2021-03-24T21:38:55.980 INFO:teuthology.task.internal:Creating archive directory... 2021-03-24T21:38:55.981 DEBUG:teuthology.orchestra.run.smithi125:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-03-24T21:38:55.984 DEBUG:teuthology.orchestra.run.smithi142:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-03-24T21:38:56.033 INFO:teuthology.run_tasks:Running task internal.coredump... 2021-03-24T21:38:56.122 INFO:teuthology.task.internal:Enabling coredump saving... 2021-03-24T21:38:56.123 DEBUG:teuthology.orchestra.run.smithi125:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-03-24T21:38:56.126 DEBUG:teuthology.orchestra.run.smithi142:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-03-24T21:38:56.154 INFO:teuthology.orchestra.run.smithi142.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-03-24T21:38:56.161 INFO:teuthology.orchestra.run.smithi125.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-03-24T21:38:56.165 INFO:teuthology.orchestra.run.smithi142.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-03-24T21:38:56.172 INFO:teuthology.orchestra.run.smithi125.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-03-24T21:38:56.173 INFO:teuthology.run_tasks:Running task internal.sudo... 2021-03-24T21:38:56.260 INFO:teuthology.task.internal:Configuring sudo... 2021-03-24T21:38:56.261 DEBUG:teuthology.orchestra.run.smithi125:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-03-24T21:38:56.264 DEBUG:teuthology.orchestra.run.smithi142:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-03-24T21:38:56.281 INFO:teuthology.run_tasks:Running task internal.syslog... 2021-03-24T21:38:56.404 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2021-03-24T21:38:56.405 DEBUG:teuthology.orchestra.run.smithi125:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-03-24T21:38:56.410 DEBUG:teuthology.orchestra.run.smithi142:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-03-24T21:38:56.417 DEBUG:teuthology.orchestra.run.smithi125:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-03-24T21:38:56.467 DEBUG:teuthology.orchestra.run.smithi125:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-03-24T21:38:56.475 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2021-03-24T21:38:56.475 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-03-24T21:38:56.506 DEBUG:teuthology.orchestra.run.smithi142:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-03-24T21:38:56.514 DEBUG:teuthology.orchestra.run.smithi142:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-03-24T21:38:56.563 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2021-03-24T21:38:56.564 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-03-24T21:38:56.594 DEBUG:teuthology.orchestra.run.smithi125:> sudo service rsyslog restart 2021-03-24T21:38:56.597 DEBUG:teuthology.orchestra.run.smithi142:> sudo service rsyslog restart 2021-03-24T21:38:56.673 INFO:teuthology.run_tasks:Running task internal.timer... 2021-03-24T21:38:56.685 INFO:teuthology.task.internal:Starting timer... 2021-03-24T21:38:56.685 INFO:teuthology.run_tasks:Running task pcp... 2021-03-24T21:38:56.782 INFO:teuthology.run_tasks:Running task selinux... 2021-03-24T21:38:56.878 DEBUG:teuthology.task.selinux:Excluding smithi125: OS 'ubuntu' does not support SELinux 2021-03-24T21:38:56.878 DEBUG:teuthology.task.selinux:Excluding smithi142: OS 'ubuntu' does not support SELinux 2021-03-24T21:38:56.879 DEBUG:teuthology.task.selinux:Getting current SELinux state 2021-03-24T21:38:56.879 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2021-03-24T21:38:56.879 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2021-03-24T21:38:56.879 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2021-03-24T21:38:56.976 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2021-03-24T21:38:56.978 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2021-03-24T21:38:56.997 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2021-03-24T21:38:57.000 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi125.front.sepia.ceph.com,smithi142.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2021-03-24T21:38:57.529 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2021-03-24T21:38:58.745 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2021-03-24T21:39:01.726 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2021-03-24T21:39:01.760 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:39:01 +0000 (0:00:00.064) 0:00:00.064 ******* 2021-03-24T21:39:01.774 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:39:01 +0000 (0:00:00.011) 0:00:00.076 ******* 2021-03-24T21:39:04.146 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2021-03-24T21:39:04.148 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Distribution Ubuntu 20.04 on host smithi125.front.sepia.ceph.com should use /usr/bin/python3, but is using /usr/bin/python for backward compatibility with prior Ansible releases. A future Ansible release will default to using the discovered platform python for this host. See https://docs.ansible.com/ansible/2.9/reference_appendices/inter preter_discovery.html for more information. This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. ok: [smithi125.front.sepia.ceph.com] 2021-03-24T21:39:04.216 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:39:04 +0000 (0:00:02.444) 0:00:02.521 ******* 2021-03-24T21:39:04.311 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Distribution Ubuntu 20.04 on host smithi142.front.sepia.ceph.com should use /usr/bin/python3, but is using /usr/bin/python for backward compatibility with prior Ansible releases. A future Ansible release will default to using the discovered platform python for this host. See https://docs.ansible.com/ansible/2.9/reference_appendices/inter preter_discovery.html for more information. This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. ok: [smithi142.front.sepia.ceph.com] 2021-03-24T21:39:04.370 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:39:04 +0000 (0:00:00.153) 0:00:02.675 ******* 2021-03-24T21:39:04.546 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2021-03-24T21:39:04.548 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2021-03-24T21:39:04.569 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:39:04 +0000 (0:00:00.199) 0:00:02.874 ******* 2021-03-24T21:39:04.593 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:39:04 +0000 (0:00:00.023) 0:00:02.897 ******* 2021-03-24T21:39:05.302 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** changed: [smithi142.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-03-24T21:39:05.335 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:39:05 +0000 (0:00:00.742) 0:00:03.640 ******* 2021-03-24T21:39:05.343 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-03-24T21:39:05.368 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:39:05 +0000 (0:00:00.033) 0:00:03.673 ******* 2021-03-24T21:39:05.825 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2021-03-24T21:39:05.826 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.061198", "end": "2021-03-24 21:39:06.146308", "rc": 0, "start": "2021-03-24 21:39:06.085110", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2021-03-24T21:39:05.864 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:39:05 +0000 (0:00:00.495) 0:00:04.169 ******* 2021-03-24T21:39:05.883 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.072949", "end": "2021-03-24 21:39:06.018217", "rc": 0, "start": "2021-03-24 21:39:05.945268", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2021-03-24T21:39:05.927 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:39:05 +0000 (0:00:00.063) 0:00:04.232 ******* 2021-03-24T21:39:06.261 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** ok: [smithi125.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-24T21:39:06.285 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:39:06 +0000 (0:00:00.357) 0:00:04.590 ******* 2021-03-24T21:39:06.298 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-24T21:39:06.324 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:39:06 +0000 (0:00:00.039) 0:00:04.629 ******* 2021-03-24T21:39:07.632 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2021-03-24T21:39:07.634 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1616621946.3636389-6954-137196810241183/source", "state": "file", "uid": 0} 2021-03-24T21:39:07.682 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:39:07 +0000 (0:00:01.357) 0:00:05.986 ******* 2021-03-24T21:39:07.706 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1616621946.4164186-6960-116083127302402/source", "state": "file", "uid": 0} 2021-03-24T21:39:07.734 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:39:07 +0000 (0:00:00.051) 0:00:06.038 ******* 2021-03-24T21:39:08.316 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2021-03-24T21:39:08.318 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-03-24T21:39:08.320 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-03-24T21:39:08.374 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-03-24T21:39:08.375 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-03-24T21:39:08.450 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2021-03-24T21:39:08.453 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-03-24T21:39:08.702 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:39:08 +0000 (0:00:00.968) 0:00:07.007 ******* 2021-03-24T21:39:08.734 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:39:08 +0000 (0:00:00.031) 0:00:07.038 ******* 2021-03-24T21:39:09.622 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi125.front.sepia.ceph.com] 2021-03-24T21:39:09.662 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] 2021-03-24T21:39:09.704 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2021-03-24T21:39:09.706 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:39:09 +0000 (0:00:00.971) 0:00:08.009 ******* 2021-03-24T21:39:09.780 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:39:09 +0000 (0:00:00.075) 0:00:08.084 ******* 2021-03-24T21:39:10.289 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2021-03-24T21:39:10.293 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.205018", "end": "2021-03-24 21:39:10.242231", "rc": 0, "start": "2021-03-24 21:39:10.037213", "stderr": "", "stderr_lines": [], "stdout": "c31004c4720a55ac12a95d66c32e64f592877604", "stdout_lines": ["c31004c4720a55ac12a95d66c32e64f592877604"]} 2021-03-24T21:39:10.314 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:39:10 +0000 (0:00:00.534) 0:00:08.619 ******* 2021-03-24T21:39:10.394 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.210780", "end": "2021-03-24 21:39:10.347063", "rc": 0, "start": "2021-03-24 21:39:10.136283", "stderr": "", "stderr_lines": [], "stdout": "c31004c4720a55ac12a95d66c32e64f592877604", "stdout_lines": ["c31004c4720a55ac12a95d66c32e64f592877604"]} 2021-03-24T21:39:10.423 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:39:10 +0000 (0:00:00.107) 0:00:08.726 ******* 2021-03-24T21:39:10.467 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2021-03-24T21:39:10.468 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.011382", "end": "2021-03-24 21:39:10.803060", "failed_when_result": false, "rc": 0, "start": "2021-03-24 21:39:10.791678", "stderr": "", "stderr_lines": [], "stdout": "df8946f08e870e2d2565b79099289ca48bd825f3", "stdout_lines": ["df8946f08e870e2d2565b79099289ca48bd825f3"]} 2021-03-24T21:39:10.493 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:39:10 +0000 (0:00:00.071) 0:00:08.798 ******* 2021-03-24T21:39:10.585 INFO:teuthology.task.ansible.out: TASK [Determine if we can skip users and pubkeys updates] ********************** 2021-03-24T21:39:10.587 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:39:10.613 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:39:10 +0000 (0:00:00.119) 0:00:08.917 ******* 2021-03-24T21:39:10.620 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2021-03-24T21:39:10.622 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.014922", "end": "2021-03-24 21:39:10.767768", "failed_when_result": false, "rc": 0, "start": "2021-03-24 21:39:10.752846", "stderr": "", "stderr_lines": [], "stdout": "df8946f08e870e2d2565b79099289ca48bd825f3", "stdout_lines": ["df8946f08e870e2d2565b79099289ca48bd825f3"]} 2021-03-24T21:39:10.658 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:39:10 +0000 (0:00:00.045) 0:00:08.963 ******* 2021-03-24T21:39:10.741 INFO:teuthology.task.ansible.out: TASK [Merge extra_admin_users into managed_admin_users] ************************ 2021-03-24T21:39:10.742 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:39:10.776 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:39:10 +0000 (0:00:00.117) 0:00:09.081 ******* 2021-03-24T21:39:10.786 INFO:teuthology.task.ansible.out: TASK [Determine if we can skip users and pubkeys updates] ********************** 2021-03-24T21:39:10.788 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:39:10.827 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:39:10 +0000 (0:00:00.050) 0:00:09.132 ******* 2021-03-24T21:39:10.878 INFO:teuthology.task.ansible.out: TASK [Remove managed_admin_users from managed_users] *************************** 2021-03-24T21:39:10.880 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:39:10.928 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:39:10 +0000 (0:00:00.095) 0:00:09.227 ******* 2021-03-24T21:39:10.949 INFO:teuthology.task.ansible.out: TASK [Merge extra_admin_users into managed_admin_users] ************************ skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:39:10.985 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:39:10 +0000 (0:00:00.062) 0:00:09.290 ******* 2021-03-24T21:39:11.055 INFO:teuthology.task.ansible.out: TASK [Filter the managed_users list] ******************************************* 2021-03-24T21:39:11.057 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:39:11.086 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:39:11 +0000 (0:00:00.100) 0:00:09.391 ******* 2021-03-24T21:39:11.103 INFO:teuthology.task.ansible.out: TASK [Remove managed_admin_users from managed_users] *************************** skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:39:11.136 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:39:11 +0000 (0:00:00.050) 0:00:09.441 ******* 2021-03-24T21:39:11.202 INFO:teuthology.task.ansible.out: TASK [Filter the managed_admin_users list] ************************************* 2021-03-24T21:39:11.204 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:39:11.234 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:39:11 +0000 (0:00:00.097) 0:00:09.539 ******* 2021-03-24T21:39:11.258 INFO:teuthology.task.ansible.out: TASK [Filter the managed_users list] ******************************************* 2021-03-24T21:39:11.259 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:39:11.292 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:39:11 +0000 (0:00:00.053) 0:00:09.592 ******* 2021-03-24T21:39:11.418 INFO:teuthology.task.ansible.out: TASK [Filter the managed_admin_users list] ************************************* 2021-03-24T21:39:11.419 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:39:11.442 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:39:11 +0000 (0:00:00.155) 0:00:09.747 ******* 2021-03-24T21:39:11.546 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2021-03-24T21:39:11.547 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-24T21:39:11.572 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:39:11 +0000 (0:00:00.129) 0:00:09.877 ******* 2021-03-24T21:39:11.701 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-24T21:39:11.741 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:39:11 +0000 (0:00:00.168) 0:00:10.045 ******* 2021-03-24T21:39:11.977 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-03-24T21:39:11.979 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-03-24T21:39:12.138 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-03-24T21:39:12.150 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-03-24T21:39:12.321 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-03-24T21:39:12.344 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-03-24T21:39:12.508 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-03-24T21:39:12.534 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-03-24T21:39:12.717 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-03-24T21:39:12.727 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-03-24T21:39:12.907 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-03-24T21:39:12.919 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-03-24T21:39:13.058 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-03-24T21:39:13.091 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-03-24T21:39:13.226 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-03-24T21:39:13.259 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-03-24T21:39:13.424 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-03-24T21:39:13.572 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-03-24T21:39:13.584 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-03-24T21:39:13.747 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-03-24T21:39:13.791 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-03-24T21:39:13.911 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-03-24T21:39:13.912 INFO:teuthology.task.ansible.out: 2021-03-24T21:39:13.959 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-03-24T21:39:14.075 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-03-24T21:39:14.127 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-03-24T21:39:14.219 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-03-24T21:39:14.307 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-03-24T21:39:14.384 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-03-24T21:39:14.482 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-03-24T21:39:14.578 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-03-24T21:39:14.674 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-03-24T21:39:14.749 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-03-24T21:39:14.859 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-03-24T21:39:14.940 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-03-24T21:39:15.025 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-03-24T21:39:15.121 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-03-24T21:39:15.225 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-03-24T21:39:15.314 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-03-24T21:39:15.391 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-03-24T21:39:15.467 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-03-24T21:39:15.542 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-03-24T21:39:15.638 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-03-24T21:39:15.724 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-03-24T21:39:15.810 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-03-24T21:39:15.919 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-03-24T21:39:15.973 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-03-24T21:39:16.082 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-03-24T21:39:16.146 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-03-24T21:39:16.243 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-03-24T21:39:16.340 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-03-24T21:39:16.415 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-03-24T21:39:16.514 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-03-24T21:39:16.590 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-03-24T21:39:16.688 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-03-24T21:39:16.784 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-03-24T21:39:16.872 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-03-24T21:39:16.968 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-03-24T21:39:17.035 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-03-24T21:39:17.124 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-03-24T21:39:17.243 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-03-24T21:39:17.318 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-03-24T21:39:17.434 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-03-24T21:39:17.469 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-03-24T21:39:17.640 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-03-24T21:39:17.651 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-03-24T21:39:17.792 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-03-24T21:39:17.825 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-03-24T21:39:17.962 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-03-24T21:39:18.024 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-03-24T21:39:18.158 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-03-24T21:39:18.187 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-03-24T21:39:18.308 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-03-24T21:39:18.353 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-03-24T21:39:18.497 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-03-24T21:39:18.498 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-03-24T21:39:18.682 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-03-24T21:39:18.693 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-03-24T21:39:18.877 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-03-24T21:39:18.901 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-03-24T21:39:19.069 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-03-24T21:39:19.091 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-03-24T21:39:19.247 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-03-24T21:39:19.255 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-03-24T21:39:19.457 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-03-24T21:39:19.467 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-03-24T21:39:19.619 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-03-24T21:39:19.641 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-03-24T21:39:19.791 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-03-24T21:39:19.878 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-03-24T21:39:20.018 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-03-24T21:39:20.050 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-03-24T21:39:20.182 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-03-24T21:39:20.204 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-03-24T21:39:20.338 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-03-24T21:39:20.381 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-03-24T21:39:20.501 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-03-24T21:39:20.555 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-03-24T21:39:20.687 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-03-24T21:39:20.710 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-03-24T21:39:20.886 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-03-24T21:39:20.909 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-03-24T21:39:21.070 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-03-24T21:39:21.267 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-03-24T21:39:21.295 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-03-24T21:39:21.459 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-03-24T21:39:21.491 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-03-24T21:39:21.703 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-03-24T21:39:21.704 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-03-24T21:39:21.884 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-03-24T21:39:21.896 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-03-24T21:39:22.055 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-03-24T21:39:22.080 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-03-24T21:39:22.248 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-03-24T21:39:22.259 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-03-24T21:39:22.448 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-03-24T21:39:22.526 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-03-24T21:39:22.637 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-03-24T21:39:22.715 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-03-24T21:39:22.867 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-03-24T21:39:22.879 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-03-24T21:39:23.039 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-03-24T21:39:23.051 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-03-24T21:39:23.205 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-03-24T21:39:23.289 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-03-24T21:39:23.407 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-03-24T21:39:23.521 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-03-24T21:39:23.564 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-03-24T21:39:23.723 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-03-24T21:39:23.725 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-03-24T21:39:23.876 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-03-24T21:39:23.877 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-03-24T21:39:24.024 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-03-24T21:39:24.046 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-03-24T21:39:24.200 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-03-24T21:39:24.224 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-03-24T21:39:24.353 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-03-24T21:39:24.416 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-03-24T21:39:24.543 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-03-24T21:39:24.636 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-03-24T21:39:24.758 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-03-24T21:39:24.759 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-03-24T21:39:24.954 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-03-24T21:39:24.975 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-03-24T21:39:25.158 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-03-24T21:39:25.186 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-03-24T21:39:25.368 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-03-24T21:39:25.416 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-03-24T21:39:25.566 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-03-24T21:39:25.631 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-03-24T21:39:25.741 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-03-24T21:39:25.791 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-03-24T21:39:25.921 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-03-24T21:39:25.982 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-03-24T21:39:25.984 INFO:teuthology.task.ansible.out: 2021-03-24T21:39:26.084 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-03-24T21:39:26.161 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-03-24T21:39:26.280 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-03-24T21:39:26.323 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-03-24T21:39:26.521 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-03-24T21:39:26.541 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-03-24T21:39:26.729 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-03-24T21:39:26.740 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-03-24T21:39:26.912 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-03-24T21:39:26.915 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-03-24T21:39:27.066 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-03-24T21:39:27.100 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-03-24T21:39:27.264 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-03-24T21:39:27.298 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-03-24T21:39:27.457 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-03-24T21:39:27.480 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-03-24T21:39:27.649 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-03-24T21:39:27.661 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-03-24T21:39:27.857 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-03-24T21:39:27.858 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-03-24T21:39:28.065 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-03-24T21:39:28.100 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-03-24T21:39:28.217 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-03-24T21:39:28.281 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-03-24T21:39:28.391 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-03-24T21:39:28.462 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-03-24T21:39:28.560 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-03-24T21:39:28.635 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-03-24T21:39:28.717 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-03-24T21:39:28.812 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-03-24T21:39:28.911 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-03-24T21:39:29.023 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-03-24T21:39:29.100 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-03-24T21:39:29.207 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-03-24T21:39:29.261 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-03-24T21:39:29.367 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-03-24T21:39:29.432 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-03-24T21:39:29.538 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-03-24T21:39:29.603 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-03-24T21:39:29.802 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-03-24T21:39:29.942 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-03-24T21:39:29.995 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-03-24T21:39:30.143 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-03-24T21:39:30.207 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-03-24T21:39:30.314 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-03-24T21:39:30.378 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-03-24T21:39:30.558 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-03-24T21:39:30.559 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-03-24T21:39:30.728 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-03-24T21:39:30.755 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-03-24T21:39:30.902 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-03-24T21:39:30.936 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-03-24T21:39:31.091 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-03-24T21:39:31.092 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-03-24T21:39:31.261 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-03-24T21:39:31.262 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-03-24T21:39:31.410 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-03-24T21:39:31.412 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-03-24T21:39:31.567 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-03-24T21:39:31.568 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-03-24T21:39:31.727 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-03-24T21:39:31.728 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-03-24T21:39:31.886 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-03-24T21:39:31.887 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-03-24T21:39:32.037 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-03-24T21:39:32.049 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-03-24T21:39:32.210 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-03-24T21:39:32.245 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-03-24T21:39:32.355 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-03-24T21:39:32.388 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-03-24T21:39:32.509 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-03-24T21:39:32.543 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-03-24T21:39:32.676 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-03-24T21:39:32.740 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-03-24T21:39:32.836 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-03-24T21:39:32.879 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-03-24T21:39:32.985 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-03-24T21:39:33.023 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-03-24T21:39:33.120 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-03-24T21:39:33.174 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-03-24T21:39:33.176 INFO:teuthology.task.ansible.out: 2021-03-24T21:39:33.272 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-03-24T21:39:33.317 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-03-24T21:39:33.433 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-03-24T21:39:33.472 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-03-24T21:39:33.589 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-03-24T21:39:33.611 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-03-24T21:39:33.739 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-03-24T21:39:33.761 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-03-24T21:39:33.880 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-03-24T21:39:33.914 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-03-24T21:39:34.031 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-03-24T21:39:34.074 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-03-24T21:39:34.181 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-03-24T21:39:34.225 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-03-24T21:39:34.355 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-03-24T21:39:34.356 INFO:teuthology.task.ansible.out: 2021-03-24T21:39:34.377 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-03-24T21:39:34.506 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-03-24T21:39:34.530 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-03-24T21:39:34.657 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-03-24T21:39:34.691 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-03-24T21:39:34.811 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-03-24T21:39:34.856 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-03-24T21:39:34.960 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-03-24T21:39:35.014 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-03-24T21:39:35.111 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-03-24T21:39:35.176 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-03-24T21:39:35.262 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-03-24T21:39:35.326 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-03-24T21:39:35.416 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-03-24T21:39:35.470 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-03-24T21:39:35.569 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-03-24T21:39:35.621 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-03-24T21:39:35.718 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-03-24T21:39:35.772 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-03-24T21:39:35.868 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-03-24T21:39:35.922 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-03-24T21:39:36.019 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-03-24T21:39:36.072 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-03-24T21:39:36.158 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-03-24T21:39:36.223 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-03-24T21:39:36.319 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-03-24T21:39:36.383 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-03-24T21:39:36.624 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-03-24T21:39:36.626 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-03-24T21:39:36.648 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:39:36 +0000 (0:00:24.907) 0:00:34.952 ******* 2021-03-24T21:39:36.717 INFO:teuthology.task.ansible.out: TASK [Create all users without sudo access.] *********************************** 2021-03-24T21:39:36.743 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:39:36 +0000 (0:00:00.092) 0:00:35.045 ******* 2021-03-24T21:39:36.781 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-03-24T21:39:36.783 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-03-24T21:39:36.896 INFO:teuthology.task.ansible.out: TASK [Merge managed_users and managed_admin users] ***************************** 2021-03-24T21:39:36.897 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-03-24T21:39:36.899 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-03-24T21:39:36.901 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop K 2021-03-24T21:39:36.902 INFO:teuthology.task.ansible.out:D0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9 2021-03-24T21:39:36.904 INFO:teuthology.task.ansible.out:df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 2021-03-24T21:39:36.906 INFO:teuthology.task.ansible.out: 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc9795 2021-03-24T21:39:36.907 INFO:teuthology.task.ansible.out:3cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a 2021-03-24T21:39:36.908 INFO:teuthology.task.ansible.out:4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e 2021-03-24T21:39:36.910 INFO:teuthology.task.ansible.out:7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef14 2021-03-24T21:39:36.911 INFO:teuthology.task.ansible.out:80c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d301 2021-03-24T21:39:36.913 INFO:teuthology.task.ansible.out:7b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214 2021-03-24T21:39:36.914 INFO:teuthology.task.ansible.out:bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelope 2021-03-24T21:39:36.916 INFO:teuthology.task.ansible.out:z@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed 2021-03-24T21:39:36.918 INFO:teuthology.task.ansible.out:19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}]}, "changed": false} 2021-03-24T21:39:36.940 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:39:36 +0000 (0:00:00.199) 0:00:35.245 ******* 2021-03-24T21:39:36.967 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:39:36 +0000 (0:00:00.026) 0:00:35.271 ******* 2021-03-24T21:39:37.031 INFO:teuthology.task.ansible.out: TASK [Create all users without sudo access.] *********************************** 2021-03-24T21:39:37.054 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:39:37 +0000 (0:00:00.087) 0:00:35.359 ******* 2021-03-24T21:39:37.195 INFO:teuthology.task.ansible.out: TASK [Merge managed_users and managed_admin users] ***************************** 2021-03-24T21:39:37.201 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-03-24T21:39:37.202 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-03-24T21:39:37.204 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop K 2021-03-24T21:39:37.205 INFO:teuthology.task.ansible.out:D0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9 2021-03-24T21:39:37.207 INFO:teuthology.task.ansible.out:df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 2021-03-24T21:39:37.209 INFO:teuthology.task.ansible.out: 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc9795 2021-03-24T21:39:37.210 INFO:teuthology.task.ansible.out:3cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a 2021-03-24T21:39:37.212 INFO:teuthology.task.ansible.out:4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"na 2021-03-24T21:39:37.213 INFO:teuthology.task.ansible.out:me": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9 2021-03-24T21:39:37.215 INFO:teuthology.task.ansible.out:e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c6116863 2021-03-24T21:39:37.217 INFO:teuthology.task.ansible.out:3ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61 2021-03-24T21:39:37.219 INFO:teuthology.task.ansible.out:ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ES 2021-03-24T21:39:37.220 INFO:teuthology.task.ansible.out:Yy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6 2021-03-24T21:39:37.222 INFO:teuthology.task.ansible.out:611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174f 2021-03-24T21:39:37.222 INFO:teuthology.task.ansible.out:e16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}]}, "changed": false} 2021-03-24T21:39:37.252 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:39:37 +0000 (0:00:00.197) 0:00:35.556 ******* 2021-03-24T21:39:38.090 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-03-24T21:39:38.092 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"after": "c31004c4720a55ac12a95d66c32e64f592877604", "attempts": 1, "before": "c31004c4720a55ac12a95d66c32e64f592877604", "changed": false, "remote_url_changed": false} 2021-03-24T21:39:38.124 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:39:38 +0000 (0:00:00.872) 0:00:36.428 ******* 2021-03-24T21:39:38.125 INFO:teuthology.task.ansible.out: 2021-03-24T21:39:38.298 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"after": "c31004c4720a55ac12a95d66c32e64f592877604", "attempts": 1, "before": "c31004c4720a55ac12a95d66c32e64f592877604", "changed": false, "remote_url_changed": false} 2021-03-24T21:39:38.325 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:39:38 +0000 (0:00:00.201) 0:00:36.630 ******* 2021-03-24T21:39:38.409 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T21:39:38.411 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-03-24T21:39:38.559 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-03-24T21:39:38.655 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-03-24T21:39:38.719 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-03-24T21:39:38.792 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-03-24T21:39:38.856 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-03-24T21:39:38.858 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-03-24T21:39:38.933 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-03-24T21:39:38.988 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-03-24T21:39:39.072 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-03-24T21:39:39.074 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-03-24T21:39:39.129 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-03-24T21:39:39.132 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-03-24T21:39:39.221 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-03-24T21:39:39.317 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-03-24T21:39:39.398 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-03-24T21:39:39.400 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-03-24T21:39:39.502 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-03-24T21:39:39.556 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-03-24T21:39:39.666 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-03-24T21:39:39.668 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:39:39.731 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-03-24T21:39:39.818 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-03-24T21:39:39.820 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-03-24T21:39:39.880 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-03-24T21:39:39.891 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:39:39.976 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-03-24T21:39:39.978 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-03-24T21:39:39.990 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:39:40.002 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:39:40.023 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-03-24T21:39:40.025 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-03-24T21:39:40.160 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-03-24T21:39:40.183 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-03-24T21:39:40.184 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-03-24T21:39:40.194 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:39:40.196 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:39:40.321 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-03-24T21:39:40.322 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-03-24T21:39:40.342 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-03-24T21:39:40.469 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-03-24T21:39:40.502 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-03-24T21:39:40.504 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-03-24T21:39:40.640 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": 2021-03-24T21:39:40.642 INFO:teuthology.task.ansible.out:null, "state": "present", "user": "davidz", "validate_certs": true} 2021-03-24T21:39:40.662 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-03-24T21:39:40.811 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-03-24T21:39:40.844 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": 2021-03-24T21:39:40.846 INFO:teuthology.task.ansible.out:null, "state": "present", "user": "davidz", "validate_certs": true} 2021-03-24T21:39:40.983 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-03-24T21:39:41.003 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-03-24T21:39:41.180 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-03-24T21:39:41.195 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-03-24T21:39:41.380 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-03-24T21:39:41.381 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-03-24T21:39:41.407 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-03-24T21:39:41.622 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-03-24T21:39:41.624 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-03-24T21:39:41.644 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-03-24T21:39:41.646 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-03-24T21:39:41.787 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-03-24T21:39:41.790 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-03-24T21:39:41.798 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-03-24T21:39:41.967 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-03-24T21:39:42.012 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-03-24T21:39:42.153 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-03-24T21:39:42.164 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46 2021-03-24T21:39:42.166 INFO:teuthology.task.ansible.out:TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-03-24T21:39:42.358 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46 2021-03-24T21:39:42.359 INFO:teuthology.task.ansible.out:TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-03-24T21:39:42.369 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-03-24T21:39:42.551 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-03-24T21:39:42.564 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-03-24T21:39:42.718 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-03-24T21:39:42.742 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-03-24T21:39:42.881 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-03-24T21:39:42.883 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-03-24T21:39:43.037 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-03-24T21:39:43.047 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-03-24T21:39:43.172 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-03-24T21:39:43.174 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-03-24T21:39:43.194 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-03-24T21:39:43.343 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-03-24T21:39:43.364 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-03-24T21:39:43.366 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-03-24T21:39:43.501 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-03-24T21:39:43.554 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-03-24T21:39:43.642 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-03-24T21:39:43.752 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-03-24T21:39:43.826 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-03-24T21:39:43.918 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-03-24T21:39:44.000 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2021-03-24T21:39:44.082 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-03-24T21:39:44.149 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-03-24T21:39:44.276 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2021-03-24T21:39:44.319 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-03-24T21:39:44.439 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-03-24T21:39:44.534 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-03-24T21:39:44.598 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-03-24T21:39:44.702 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-03-24T21:39:44.776 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-03-24T21:39:44.858 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-03-24T21:39:44.872 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-03-24T21:39:44.960 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-03-24T21:39:45.035 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-03-24T21:39:45.038 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-03-24T21:39:45.165 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-03-24T21:39:45.167 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-03-24T21:39:45.187 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-03-24T21:39:45.375 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-03-24T21:39:45.376 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-03-24T21:39:45.398 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-03-24T21:39:45.534 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-03-24T21:39:45.558 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-03-24T21:39:45.845 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nu 2021-03-24T21:39:45.847 INFO:teuthology.task.ansible.out:ll, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-03-24T21:39:46.019 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-03-24T21:39:46.051 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-03-24T21:39:46.053 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-03-24T21:39:46.054 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-03-24T21:39:46.176 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nu 2021-03-24T21:39:46.178 INFO:teuthology.task.ansible.out:ll, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-03-24T21:39:46.240 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-03-24T21:39:46.358 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-03-24T21:39:46.360 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-03-24T21:39:46.361 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-03-24T21:39:46.390 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-03-24T21:39:46.530 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-03-24T21:39:46.588 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-03-24T21:39:46.689 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-03-24T21:39:46.774 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-03-24T21:39:46.872 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-03-24T21:39:46.995 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-03-24T21:39:47.060 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-03-24T21:39:47.161 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-03-24T21:39:47.163 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-03-24T21:39:47.240 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-03-24T21:39:47.359 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-03-24T21:39:47.447 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-03-24T21:39:47.449 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-03-24T21:39:47.522 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-03-24T21:39:47.678 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-03-24T21:39:47.707 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-03-24T21:39:47.709 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-03-24T21:39:47.860 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-03-24T21:39:47.862 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-03-24T21:39:48.037 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-03-24T21:39:48.038 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-03-24T21:39:48.069 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-03-24T21:39:48.265 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-03-24T21:39:48.267 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-03-24T21:39:48.269 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-03-24T21:39:48.431 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-03-24T21:39:48.443 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-03-24T21:39:48.597 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-03-24T21:39:48.599 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-03-24T21:39:48.661 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-03-24T21:39:48.749 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-03-24T21:39:48.812 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-03-24T21:39:48.927 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-03-24T21:39:48.970 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-03-24T21:39:49.089 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-03-24T21:39:49.195 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-03-24T21:39:49.270 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-03-24T21:39:49.450 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-03-24T21:39:49.452 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-03-24T21:39:49.676 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-03-24T21:39:49.711 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-03-24T21:39:49.902 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-03-24T21:39:49.904 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-03-24T21:39:50.080 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-03-24T21:39:50.112 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-03-24T21:39:50.291 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-03-24T21:39:50.297 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-03-24T21:39:50.472 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-03-24T21:39:50.474 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-03-24T21:39:50.634 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-03-24T21:39:50.677 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-03-24T21:39:50.856 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-03-24T21:39:50.876 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-03-24T21:39:51.034 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-03-24T21:39:51.056 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-03-24T21:39:51.225 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-03-24T21:39:51.237 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-03-24T21:39:51.401 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-03-24T21:39:51.444 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-03-24T21:39:51.560 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-03-24T21:39:51.708 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-03-24T21:39:51.751 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-03-24T21:39:51.898 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-03-24T21:39:51.941 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-03-24T21:39:52.079 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-03-24T21:39:52.100 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-03-24T21:39:52.231 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-03-24T21:39:52.306 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-03-24T21:39:52.391 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-03-24T21:39:52.393 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-03-24T21:39:52.535 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-03-24T21:39:52.537 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-03-24T21:39:52.609 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-03-24T21:39:52.737 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-03-24T21:39:52.868 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-03-24T21:39:52.931 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-03-24T21:39:53.028 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-03-24T21:39:53.099 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-03-24T21:39:53.231 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-03-24T21:39:53.326 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-03-24T21:39:53.436 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2021-03-24T21:39:53.438 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2021-03-24T21:39:53.525 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2021-03-24T21:39:53.527 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2021-03-24T21:39:53.647 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-03-24T21:39:53.649 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-03-24T21:39:53.684 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-03-24T21:39:53.686 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-03-24T21:39:53.825 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-03-24T21:39:53.899 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-03-24T21:39:54.016 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-03-24T21:39:54.101 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-03-24T21:39:54.197 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-03-24T21:39:54.313 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-03-24T21:39:54.485 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-03-24T21:39:54.494 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-03-24T21:39:54.704 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-03-24T21:39:54.706 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-03-24T21:39:54.887 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-03-24T21:39:54.943 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-03-24T21:39:55.070 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-03-24T21:39:55.138 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-03-24T21:39:55.239 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-03-24T21:39:55.241 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-03-24T21:39:55.362 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-03-24T21:39:55.364 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-03-24T21:39:55.447 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-03-24T21:39:55.570 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-03-24T21:39:55.624 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-03-24T21:39:55.750 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-03-24T21:39:55.804 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-03-24T21:39:56.084 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-03-24T21:39:56.085 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-03-24T21:39:56.267 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-03-24T21:39:56.363 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-03-24T21:39:56.471 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-03-24T21:39:56.567 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-03-24T21:39:56.709 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", 2021-03-24T21:39:56.711 INFO:teuthology.task.ansible.out:"manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-03-24T21:39:56.826 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-03-24T21:39:56.917 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-03-24T21:39:57.002 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", 2021-03-24T21:39:57.004 INFO:teuthology.task.ansible.out:"manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-03-24T21:39:57.050 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-03-24T21:39:57.204 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-03-24T21:39:57.289 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-03-24T21:39:57.400 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-03-24T21:39:57.506 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-03-24T21:39:57.570 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-03-24T21:39:57.752 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-03-24T21:39:57.764 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-03-24T21:39:57.979 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-03-24T21:39:57.981 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-03-24T21:39:58.214 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-03-24T21:39:58.237 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-03-24T21:39:58.419 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@loca 2021-03-24T21:39:58.420 INFO:teuthology.task.ansible.out:lhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-03-24T21:39:58.589 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-03-24T21:39:58.601 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-03-24T21:39:58.809 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-03-24T21:39:58.831 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-03-24T21:39:58.836 INFO:teuthology.task.ansible.out: 2021-03-24T21:39:58.982 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-03-24T21:39:59.056 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-03-24T21:39:59.204 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-03-24T21:39:59.279 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-03-24T21:39:59.395 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-03-24T21:39:59.501 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-03-24T21:39:59.555 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-03-24T21:39:59.667 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-03-24T21:39:59.731 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-03-24T21:39:59.838 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-03-24T21:39:59.903 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-03-24T21:40:00.037 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-03-24T21:40:00.059 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-03-24T21:40:00.263 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-03-24T21:40:00.276 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-03-24T21:40:00.278 INFO:teuthology.task.ansible.out: 2021-03-24T21:40:00.459 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-03-24T21:40:00.468 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-03-24T21:40:00.470 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-03-24T21:40:00.659 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-03-24T21:40:00.692 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-03-24T21:40:00.869 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-03-24T21:40:00.871 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-03-24T21:40:01.025 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-03-24T21:40:01.037 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-03-24T21:40:01.206 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-03-24T21:40:01.249 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-03-24T21:40:01.427 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-03-24T21:40:01.506 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-03-24T21:40:01.613 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-03-24T21:40:01.696 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-03-24T21:40:01.778 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-03-24T21:40:01.857 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-03-24T21:40:02.004 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-03-24T21:40:02.059 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-03-24T21:40:02.164 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-03-24T21:40:02.223 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-03-24T21:40:02.348 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-03-24T21:40:02.425 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-03-24T21:40:02.496 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-03-24T21:40:02.611 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-03-24T21:40:02.692 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-03-24T21:40:02.809 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-03-24T21:40:02.841 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-03-24T21:40:02.996 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-03-24T21:40:03.043 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-03-24T21:40:03.187 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-03-24T21:40:03.198 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-03-24T21:40:03.366 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-03-24T21:40:03.378 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-03-24T21:40:03.542 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-03-24T21:40:03.588 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-03-24T21:40:03.737 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", " 2021-03-24T21:40:03.739 INFO:teuthology.task.ansible.out:key_options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-03-24T21:40:03.769 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-03-24T21:40:03.932 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", " 2021-03-24T21:40:03.934 INFO:teuthology.task.ansible.out:key_options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-03-24T21:40:03.961 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-03-24T21:40:04.118 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:04 +0000 (0:00:25.793) 0:01:02.423 ******* 2021-03-24T21:40:04.168 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-03-24T21:40:04.333 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:04 +0000 (0:00:00.214) 0:01:02.637 ******* 2021-03-24T21:40:04.478 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:04.493 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:04.515 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:04.537 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:04.548 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:04.562 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:04.576 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:04.586 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:04.604 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:04.606 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:04.608 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:04.618 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:04.629 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:04.684 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:04.696 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:04.708 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:04.719 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:04.747 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:04.801 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-03-24T21:40:04.838 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:04.844 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:04.938 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-03-24T21:40:04.950 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:04.971 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.050 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-03-24T21:40:05.175 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-03-24T21:40:05.272 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfad 2021-03-24T21:40:05.273 INFO:teuthology.task.ansible.out:a28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.314 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.347 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-03-24T21:40:05.358 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.370 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.372 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.402 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.414 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.425 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.436 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.447 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.459 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.481 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.492 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.516 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.519 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.539 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.554 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.565 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.567 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.577 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.578 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.599 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.631 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.633 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.644 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.647 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.656 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.667 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.689 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.711 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.722 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.724 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.734 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.735 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.766 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.768 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.778 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.779 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.779 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.791 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.792 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.813 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.815 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.838 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.840 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.860 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.882 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.884 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.906 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.908 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.918 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.939 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.941 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.961 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.963 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.973 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.984 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:05.995 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.017 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.019 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.039 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.051 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.063 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.065 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.078 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.090 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.105 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.129 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.130 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.140 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.152 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.153 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.165 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.167 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.181 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.181 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.193 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.194 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.217 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.219 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.232 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.234 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.244 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.267 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.269 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.290 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.301 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.313 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.325 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.336 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.338 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.348 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.360 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.362 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.372 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.384 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.386 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.396 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.398 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.411 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.413 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.424 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.425 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.441 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.443 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.458 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.480 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.481 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.503 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.505 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.518 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.529 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.550 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.552 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.567 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.579 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.625 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.627 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.642 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.644 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.654 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.656 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.666 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.667 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.678 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.689 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.690 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.701 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.712 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.713 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.723 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.737 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.746 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.757 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.770 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.781 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.783 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.809 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.812 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.823 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.835 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.836 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.847 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.858 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.871 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.880 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.882 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.892 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.894 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.914 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.916 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.936 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.958 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.971 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.973 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.982 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.984 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:06.994 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.009 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.010 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.021 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.033 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.044 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.046 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.057 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.064 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.068 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.070 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.092 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.103 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.115 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.117 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.137 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.139 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.150 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.151 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.162 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.164 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.177 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.179 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.180 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.180 INFO:teuthology.task.ansible.out: 2021-03-24T21:40:07.181 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.192 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.194 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.205 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.207 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.217 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.219 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.229 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.231 INFO:teuthology.task.ansible.out: 2021-03-24T21:40:07.247 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.249 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.259 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.261 INFO:teuthology.task.ansible.out: 2021-03-24T21:40:07.271 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.273 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.283 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.295 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.306 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.322 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.329 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.330 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.342 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.355 INFO:teuthology.task.ansible.out: 2021-03-24T21:40:07.358 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.364 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.375 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.377 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.378 INFO:teuthology.task.ansible.out: 2021-03-24T21:40:07.379 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.381 INFO:teuthology.task.ansible.out: 2021-03-24T21:40:07.397 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.403 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.418 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.420 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.430 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.432 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.442 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.444 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.470 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.478 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.479 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.532 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:07 +0000 (0:00:03.199) 0:01:05.837 ******* 2021-03-24T21:40:07.544 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.546 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.557 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.559 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.569 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.580 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.582 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.676 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:07 +0000 (0:00:00.143) 0:01:05.980 ******* 2021-03-24T21:40:07.714 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* 2021-03-24T21:40:07.716 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.740 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:07 +0000 (0:00:00.064) 0:01:06.045 ******* 2021-03-24T21:40:07.812 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:07.839 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:07 +0000 (0:00:00.096) 0:01:06.141 ******* 2021-03-24T21:40:08.076 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-24T21:40:08.078 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-03-24T21:40:08.215 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-03-24T21:40:08.232 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-03-24T21:40:08.376 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-03-24T21:40:08.387 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-03-24T21:40:08.538 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-03-24T21:40:08.539 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-03-24T21:40:08.704 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-03-24T21:40:08.715 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-03-24T21:40:08.863 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-03-24T21:40:08.911 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-03-24T21:40:09.046 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-03-24T21:40:09.050 INFO:teuthology.task.ansible.out: 2021-03-24T21:40:09.079 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-03-24T21:40:09.186 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-03-24T21:40:09.230 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-03-24T21:40:09.363 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-03-24T21:40:09.370 INFO:teuthology.task.ansible.out: 2021-03-24T21:40:09.404 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-03-24T21:40:09.526 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-03-24T21:40:09.585 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-03-24T21:40:09.698 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-03-24T21:40:09.730 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-03-24T21:40:09.871 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-03-24T21:40:09.903 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-03-24T21:40:10.046 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-03-24T21:40:10.048 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-03-24T21:40:10.187 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-03-24T21:40:10.209 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-03-24T21:40:10.348 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-03-24T21:40:10.359 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-03-24T21:40:10.517 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-03-24T21:40:10.529 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-03-24T21:40:10.675 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-03-24T21:40:10.677 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-03-24T21:40:10.845 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-03-24T21:40:10.887 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-03-24T21:40:11.005 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-03-24T21:40:11.063 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-03-24T21:40:11.152 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-03-24T21:40:11.248 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-03-24T21:40:11.336 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-03-24T21:40:11.400 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-03-24T21:40:11.489 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2021-03-24T21:40:11.585 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-03-24T21:40:11.649 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-03-24T21:40:11.727 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2021-03-24T21:40:11.814 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-03-24T21:40:11.816 INFO:teuthology.task.ansible.out: 2021-03-24T21:40:11.924 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-03-24T21:40:11.990 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-03-24T21:40:12.065 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-03-24T21:40:12.164 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-03-24T21:40:12.207 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-03-24T21:40:12.303 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-03-24T21:40:12.358 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-03-24T21:40:12.457 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-03-24T21:40:12.503 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-03-24T21:40:12.587 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-03-24T21:40:12.654 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-03-24T21:40:12.752 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-03-24T21:40:12.817 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-03-24T21:40:12.892 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-03-24T21:40:12.976 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-03-24T21:40:13.051 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-03-24T21:40:13.106 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-03-24T21:40:13.201 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-03-24T21:40:13.254 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-03-24T21:40:13.340 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-03-24T21:40:13.395 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-03-24T21:40:13.483 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-03-24T21:40:13.525 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-03-24T21:40:13.652 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-03-24T21:40:13.674 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-03-24T21:40:13.782 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-03-24T21:40:13.804 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-03-24T21:40:13.932 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-03-24T21:40:13.943 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-03-24T21:40:14.072 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} 2021-03-24T21:40:14.074 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-03-24T21:40:14.218 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} ok: [smithi125.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-03-24T21:40:14.363 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-03-24T21:40:14.375 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-03-24T21:40:14.506 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-03-24T21:40:14.528 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-03-24T21:40:14.657 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-03-24T21:40:14.682 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-03-24T21:40:14.829 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-03-24T21:40:14.831 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-03-24T21:40:14.960 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-03-24T21:40:14.962 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-03-24T21:40:15.121 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-03-24T21:40:15.133 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-03-24T21:40:15.271 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-03-24T21:40:15.274 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-03-24T21:40:15.420 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-03-24T21:40:15.422 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-03-24T21:40:15.550 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-03-24T21:40:15.561 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-03-24T21:40:15.632 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:15 +0000 (0:00:07.795) 0:01:13.937 ******* 2021-03-24T21:40:15.698 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-03-24T21:40:15.777 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:15 +0000 (0:00:00.124) 0:01:14.061 ******* 2021-03-24T21:40:16.129 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-03-24T21:40:16.131 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => {"changed": true, "checksum": "1d9711ff879c231c67cb70d00ca0088797a377d0", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "80a55def4b505d08aff3687f81f74dc2", "mode": "0644", "owner": "root", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1616622015.7319868-29512-38613985587106/source", "state": "file", "uid": 0} 2021-03-24T21:40:16.157 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:16 +0000 (0:00:00.398) 0:01:14.460 ******* 2021-03-24T21:40:16.197 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "checksum": "1d9711ff879c231c67cb70d00ca0088797a377d0", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "80a55def4b505d08aff3687f81f74dc2", "mode": "0644", "owner": "root", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1616622015.8462362-29547-143014580576484/source", "state": "file", "uid": 0} 2021-03-24T21:40:16.248 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:16 +0000 (0:00:00.091) 0:01:14.552 ******* 2021-03-24T21:40:16.265 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-03-24T21:40:16.267 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => { "msg": "Host smithi125.front.sepia.ceph.com is running Ubuntu 20.04 (focal)" } 2021-03-24T21:40:16.293 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:16 +0000 (0:00:00.044) 0:01:14.597 ******* 2021-03-24T21:40:16.332 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => { "msg": "Host smithi142.front.sepia.ceph.com is running Ubuntu 20.04 (focal)" } 2021-03-24T21:40:16.357 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:16 +0000 (0:00:00.064) 0:01:14.661 ******* 2021-03-24T21:40:16.379 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-03-24T21:40:16.380 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2021-03-24T21:40:16.406 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:16 +0000 (0:00:00.048) 0:01:14.710 ******* 2021-03-24T21:40:16.489 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2021-03-24T21:40:16.490 INFO:teuthology.task.ansible.out: 2021-03-24T21:40:16.492 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:16.520 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:16 +0000 (0:00:00.114) 0:01:14.824 ******* 2021-03-24T21:40:16.571 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:16 +0000 (0:00:00.051) 0:01:14.876 ******* 2021-03-24T21:40:16.612 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-03-24T21:40:16.614 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:16.640 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:16 +0000 (0:00:00.068) 0:01:14.945 ******* 2021-03-24T21:40:16.662 INFO:teuthology.task.ansible.out: 2021-03-24T21:40:16.663 INFO:teuthology.task.ansible.out:TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-03-24T21:40:16.665 INFO:teuthology.task.ansible.out: skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:16.704 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:16 +0000 (0:00:00.064) 0:01:15.009 ******* 2021-03-24T21:40:16.776 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-03-24T21:40:16.778 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:16.814 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:16 +0000 (0:00:00.109) 0:01:15.119 ******* 2021-03-24T21:40:16.833 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:16.861 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:16 +0000 (0:00:00.046) 0:01:15.165 ******* 2021-03-24T21:40:16.953 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-03-24T21:40:16.955 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:16.979 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:16 +0000 (0:00:00.118) 0:01:15.284 ******* 2021-03-24T21:40:16.992 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-03-24T21:40:16.994 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:17.018 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:17 +0000 (0:00:00.038) 0:01:15.322 ******* 2021-03-24T21:40:17.145 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:17.184 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:17 +0000 (0:00:00.166) 0:01:15.489 ******* 2021-03-24T21:40:17.247 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:17 +0000 (0:00:00.062) 0:01:15.552 ******* 2021-03-24T21:40:17.250 INFO:teuthology.task.ansible.out: 2021-03-24T21:40:17.346 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:17.389 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:17 +0000 (0:00:00.141) 0:01:15.694 ******* 2021-03-24T21:40:17.411 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-03-24T21:40:17.414 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:17.438 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:17 +0000 (0:00:00.049) 0:01:15.743 ******* 2021-03-24T21:40:17.500 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-03-24T21:40:17.505 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:17.524 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:17 +0000 (0:00:00.086) 0:01:15.829 ******* 2021-03-24T21:40:17.531 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-03-24T21:40:17.533 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:17.570 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:17 +0000 (0:00:00.045) 0:01:15.874 ******* 2021-03-24T21:40:17.620 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-03-24T21:40:17.622 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:17.647 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:17 +0000 (0:00:00.077) 0:01:15.952 ******* 2021-03-24T21:40:17.655 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:17.684 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:17 +0000 (0:00:00.036) 0:01:15.988 ******* 2021-03-24T21:40:17.723 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-03-24T21:40:17.725 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-24T21:40:17.726 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-24T21:40:17.754 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:17 +0000 (0:00:00.070) 0:01:16.058 ******* 2021-03-24T21:40:17.763 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-03-24T21:40:17.764 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:17.792 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:17 +0000 (0:00:00.038) 0:01:16.097 ******* 2021-03-24T21:40:17.833 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2021-03-24T21:40:17.834 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:17.863 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:17 +0000 (0:00:00.071) 0:01:16.168 ******* 2021-03-24T21:40:17.871 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-03-24T21:40:17.872 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi142.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-24T21:40:17.900 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:17 +0000 (0:00:00.036) 0:01:16.204 ******* 2021-03-24T21:40:17.902 INFO:teuthology.task.ansible.out: 2021-03-24T21:40:17.939 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:17.965 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:17 +0000 (0:00:00.063) 0:01:16.268 ******* 2021-03-24T21:40:17.981 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:18.015 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:18 +0000 (0:00:00.051) 0:01:16.320 ******* 2021-03-24T21:40:18.046 INFO:teuthology.task.ansible.out: 2021-03-24T21:40:18.047 INFO:teuthology.task.ansible.out:TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-03-24T21:40:18.047 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-24T21:40:18.079 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:18 +0000 (0:00:00.062) 0:01:16.383 ******* 2021-03-24T21:40:18.099 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-03-24T21:40:18.101 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:18.129 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:18 +0000 (0:00:00.049) 0:01:16.433 ******* 2021-03-24T21:40:18.169 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:18.193 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:18 +0000 (0:00:00.065) 0:01:16.498 ******* 2021-03-24T21:40:18.195 INFO:teuthology.task.ansible.out: 2021-03-24T21:40:18.214 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-03-24T21:40:18.216 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-24T21:40:18.247 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:18 +0000 (0:00:00.054) 0:01:16.552 ******* 2021-03-24T21:40:18.287 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-03-24T21:40:18.290 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-24T21:40:18.314 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:18 +0000 (0:00:00.066) 0:01:16.618 ******* 2021-03-24T21:40:18.332 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:18.357 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:18 +0000 (0:00:00.043) 0:01:16.662 ******* 2021-03-24T21:40:18.397 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-03-24T21:40:18.398 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:18.426 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:18 +0000 (0:00:00.068) 0:01:16.731 ******* 2021-03-24T21:40:18.433 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** skipping: [smithi142.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-24T21:40:18.435 INFO:teuthology.task.ansible.out: 2021-03-24T21:40:18.463 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:18 +0000 (0:00:00.036) 0:01:16.767 ******* 2021-03-24T21:40:18.493 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:18.523 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:18 +0000 (0:00:00.059) 0:01:16.827 ******* 2021-03-24T21:40:18.541 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:18.567 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:18 +0000 (0:00:00.044) 0:01:16.872 ******* 2021-03-24T21:40:18.597 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:18.623 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:18 +0000 (0:00:00.054) 0:01:16.927 ******* 2021-03-24T21:40:18.650 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-03-24T21:40:18.652 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:18.679 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:18 +0000 (0:00:00.056) 0:01:16.984 ******* 2021-03-24T21:40:18.708 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-03-24T21:40:18.710 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:18.733 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:18 +0000 (0:00:00.053) 0:01:17.038 ******* 2021-03-24T21:40:18.765 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-03-24T21:40:18.767 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:18.792 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:18 +0000 (0:00:00.058) 0:01:17.096 ******* 2021-03-24T21:40:18.819 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-03-24T21:40:18.821 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-24T21:40:18.844 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:18 +0000 (0:00:00.051) 0:01:17.148 ******* 2021-03-24T21:40:18.861 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-03-24T21:40:18.863 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:18.891 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:18 +0000 (0:00:00.047) 0:01:17.196 ******* 2021-03-24T21:40:18.921 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-03-24T21:40:18.923 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:18.954 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:18 +0000 (0:00:00.062) 0:01:17.259 ******* 2021-03-24T21:40:18.963 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-03-24T21:40:18.964 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-24T21:40:18.991 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:18 +0000 (0:00:00.036) 0:01:17.295 ******* 2021-03-24T21:40:19.025 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:19.050 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:19 +0000 (0:00:00.059) 0:01:17.354 ******* 2021-03-24T21:40:19.068 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-03-24T21:40:19.070 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:19.094 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:19 +0000 (0:00:00.044) 0:01:17.399 ******* 2021-03-24T21:40:19.122 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-03-24T21:40:19.124 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:19.148 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:19 +0000 (0:00:00.053) 0:01:17.452 ******* 2021-03-24T21:40:19.166 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:19.191 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:19 +0000 (0:00:00.043) 0:01:17.495 ******* 2021-03-24T21:40:19.218 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-03-24T21:40:19.220 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:19.244 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:19 +0000 (0:00:00.052) 0:01:17.548 ******* 2021-03-24T21:40:19.261 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-03-24T21:40:19.263 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:19.289 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:19 +0000 (0:00:00.045) 0:01:17.594 ******* 2021-03-24T21:40:19.317 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-03-24T21:40:19.319 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:19.343 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:19 +0000 (0:00:00.053) 0:01:17.647 ******* 2021-03-24T21:40:19.368 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-03-24T21:40:19.370 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:19.396 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:19 +0000 (0:00:00.052) 0:01:17.700 ******* 2021-03-24T21:40:19.423 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-03-24T21:40:19.425 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:19.451 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:19 +0000 (0:00:00.055) 0:01:17.756 ******* 2021-03-24T21:40:19.480 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-03-24T21:40:19.481 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:19.515 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:19 +0000 (0:00:00.063) 0:01:17.819 ******* 2021-03-24T21:40:19.558 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-03-24T21:40:19.560 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-24T21:40:19.589 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:19 +0000 (0:00:00.074) 0:01:17.893 ******* 2021-03-24T21:40:19.597 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-03-24T21:40:19.598 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:19.628 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:19 +0000 (0:00:00.038) 0:01:17.932 ******* 2021-03-24T21:40:19.659 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-03-24T21:40:19.661 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:19.689 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:19 +0000 (0:00:00.061) 0:01:17.994 ******* 2021-03-24T21:40:19.708 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** skipping: [smithi142.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-24T21:40:19.736 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:19 +0000 (0:00:00.046) 0:01:18.041 ******* 2021-03-24T21:40:19.764 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-03-24T21:40:19.766 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:19.794 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:19 +0000 (0:00:00.056) 0:01:18.098 ******* 2021-03-24T21:40:19.810 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-03-24T21:40:19.812 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:19.840 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:19 +0000 (0:00:00.046) 0:01:18.144 ******* 2021-03-24T21:40:19.867 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-03-24T21:40:19.869 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:19.894 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:19 +0000 (0:00:00.054) 0:01:18.198 ******* 2021-03-24T21:40:19.917 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-03-24T21:40:19.919 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:19.944 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:19 +0000 (0:00:00.048) 0:01:18.247 ******* 2021-03-24T21:40:19.997 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-03-24T21:40:19.999 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:20.010 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:20.036 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:20 +0000 (0:00:00.093) 0:01:18.341 ******* 2021-03-24T21:40:20.044 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-03-24T21:40:20.046 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:20.072 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:20 +0000 (0:00:00.036) 0:01:18.377 ******* 2021-03-24T21:40:20.124 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-03-24T21:40:20.126 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} 2021-03-24T21:40:20.137 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2021-03-24T21:40:20.163 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:20 +0000 (0:00:00.090) 0:01:18.468 ******* 2021-03-24T21:40:20.170 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-03-24T21:40:20.172 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:20.173 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:20.214 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:20 +0000 (0:00:00.050) 0:01:18.518 ******* 2021-03-24T21:40:20.232 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-03-24T21:40:20.234 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:20.259 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:20 +0000 (0:00:00.045) 0:01:18.563 ******* 2021-03-24T21:40:20.291 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-03-24T21:40:20.292 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} 2021-03-24T21:40:20.293 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2021-03-24T21:40:20.319 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:20 +0000 (0:00:00.059) 0:01:18.623 ******* 2021-03-24T21:40:20.388 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-03-24T21:40:20.390 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:20.413 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:20 +0000 (0:00:00.094) 0:01:18.718 ******* 2021-03-24T21:40:20.462 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-03-24T21:40:20.464 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.012052", "end": "2021-03-24 21:40:20.790870", "rc": 0, "start": "2021-03-24 21:40:20.778818", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-03-24T21:40:20.488 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:20 +0000 (0:00:00.074) 0:01:18.792 ******* 2021-03-24T21:40:20.568 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-03-24T21:40:20.570 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:20.597 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:20 +0000 (0:00:00.108) 0:01:18.901 ******* 2021-03-24T21:40:20.624 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-03-24T21:40:20.626 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.013682", "end": "2021-03-24 21:40:20.795675", "rc": 0, "start": "2021-03-24 21:40:20.781993", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-03-24T21:40:20.651 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:20 +0000 (0:00:00.054) 0:01:18.955 ******* 2021-03-24T21:40:20.678 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-03-24T21:40:20.680 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:20.703 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:20 +0000 (0:00:00.052) 0:01:19.008 ******* 2021-03-24T21:40:20.726 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-03-24T21:40:20.728 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:20.757 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:20 +0000 (0:00:00.053) 0:01:19.061 ******* 2021-03-24T21:40:20.850 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-03-24T21:40:20.852 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:20.876 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:20 +0000 (0:00:00.119) 0:01:19.180 ******* 2021-03-24T21:40:21.415 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-03-24T21:40:21.417 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "name": "sleep.target", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "AllowIsolate": "no", "AssertResult": "no", "AssertTimestampMonotonic": "0", "CanIsolate": "no", "CanReload": "no", "CanStart": "no", "CanStop": "yes", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "DefaultDependencies": "yes", "Description": "sleep.target", "FailureAction": "none", "FragmentPath": "/dev/null", "Id": "sleep.target", "IgnoreOnIsolate": "no", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "LoadError": "org.freedesktop.systemd1.UnitMasked \"Unit sleep.target is masked.\"", "LoadState": "masked", "Names": "sleep.target", "NeedDaemonReload": "no", "OnFailureJobMode": "replace", "Perpetual": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StateChangeTimestampMonotonic": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "Transient": "no", "UnitFileState": "masked"}} ok: [smithi142.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "name": "sleep.target", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "AllowIsolate": "no", "AssertResult": "no", "AssertTimestampMonotonic": "0", "CanIsolate": "no", "CanReload": "no", "CanStart": "no", "CanStop": "yes", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "DefaultDependencies": "yes", "Description": "sleep.target", "FailureAction": "none", "FragmentPat 2021-03-24T21:40:21.419 INFO:teuthology.task.ansible.out:h": "/dev/null", "Id": "sleep.target", "IgnoreOnIsolate": "no", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "LoadError": "org.freedesktop.systemd1.UnitMasked \"Unit sleep.target is masked.\"", "LoadState": "masked", "Names": "sleep.target", "NeedDaemonReload": "no", "OnFailureJobMode": "replace", "Perpetual": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StateChangeTimestampMonotonic": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "Transient": "no", "UnitFileState": "masked"}} 2021-03-24T21:40:21.598 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "name": "suspend.target", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "AllowIsolate": "no", "AssertResult": "no", "AssertTimestampMonotonic": "0", "CanIsolate": "no", "CanReload": "no", "CanStart": "no", "CanStop": "yes", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "DefaultDependencies": "yes", "Description": "suspend.target", "FailureAction": "none", "FragmentPath": "/dev/null", "Id": "suspend.target", "IgnoreOnIsolate": "no", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "LoadError": "org.freedesktop.systemd1.UnitMasked \"Unit suspend.target is masked.\"", "LoadState": "masked", "Names": "suspend.target", "NeedDaemonReload": "no", "OnFailureJobMode": "replace", "Perpetual": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StateChangeTimestampMonotonic": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "Transient": "no", "UnitFileState": "masked"}} 2021-03-24T21:40:21.599 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "name": "suspend.target", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "AllowIsolate": "no", "AssertResult": "no", "AssertTimestampMonotonic": "0", "CanIsolate": "no", "CanReload": "no", "CanStart": "no", "CanStop": "yes", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "DefaultDependencies": "yes", "Description": "suspend.target", "FailureAction": "none", "FragmentPath": "/dev/null", "Id": "suspend.target", "IgnoreOnIsolate": "no", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "LoadError": "org.freedesktop.systemd1.UnitMasked \"Unit suspend.target is masked.\"", "LoadState": "masked", "Names": "suspend.target", "NeedDaemonReload": "no", "OnFailureJobMode": "replace", "Perpetual": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StateChangeTimestampMonotonic": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "Transient": "no", "UnitFileState": "masked"}} 2021-03-24T21:40:21.791 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "name": "hibernate.target", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "AllowIsolate": "no", "AssertResult": "no", "AssertTimestampMonotonic": "0", "CanIsolate": "no", "CanReload": "no", "CanStart": "no", "CanStop": "yes", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "DefaultDependencies": "yes", "Description": "hibernate.target", "FailureAction": "none", "FragmentPath": "/dev/null", "Id": "hibernate.target", "IgnoreOnIsolate": "no", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "LoadError": "org.freedesktop.systemd1.UnitMasked \"Unit hibernate.target is masked.\"", "LoadState": "masked", "Names": "hibernate.target", "NeedDaemonReload": "no", "OnFailureJobMode": "replace", "Perpetual": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StateChangeTimestampMonotonic": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "Transient": "no", "UnitFileState": "masked"}} 2021-03-24T21:40:21.801 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "name": "hibernate.target", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "AllowIsolate": "no", "AssertResult": "no", "AssertTimestampMonotonic": "0", "CanIsolate": "no", "CanReload": "no", "CanStart": "no", "CanStop": "yes", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "DefaultDependencies": "yes", "Description": "hibernate.target", "FailureAction": "none", "FragmentPath": "/dev/null", "Id": "hibernate.target", "IgnoreOnIsolate": "no", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "LoadError": "org.freedesktop.systemd1.UnitMasked \"Unit hibernate.target is masked.\"", "LoadState": "masked", "Names": "hibernate.target", "NeedDaemonReload": "no", "OnFailureJobMode": "replace", "Perpetual": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StateChangeTimestampMonotonic": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "Transient": "no", "UnitFileState": "masked"}} 2021-03-24T21:40:21.983 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "name": "hybrid-sleep.target", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "AllowIsolate": "no", "AssertResult": "no", "AssertTimestampMonotonic": "0", "CanIsolate": "no", "CanReload": "no", "CanStart": "no", "CanStop": "yes", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "DefaultDependencies": "yes", "Description": "hybrid-sleep.target", "FailureAction": "none", "FragmentPath": "/dev/null", "Id": "hybrid-sleep.target", "IgnoreOnIsolate": "no", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "LoadError": "org.freedesktop.systemd1.UnitMasked \"Unit hybrid-sleep.target is masked.\"", "LoadState": "masked", "Names": "hybrid-sleep.target", "NeedDaemonReload": "no", "OnFailureJobMode": "replace", "Perpetual": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StateChangeTimestampMonotonic": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "Transient": "no", "UnitFileState": "masked"}} 2021-03-24T21:40:21.986 INFO:teuthology.task.ansible.out: 2021-03-24T21:40:22.014 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:22 +0000 (0:00:01.138) 0:01:20.319 ******* 2021-03-24T21:40:22.018 INFO:teuthology.task.ansible.out: 2021-03-24T21:40:22.026 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "name": "hybrid-sleep.target", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "AllowIsolate": "no", "AssertResult": "no", "AssertTimestampMonotonic": "0", "CanIsolate": "no", "CanReload": "no", "CanStart": "no", "CanStop": "yes", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "DefaultDependencies": "yes", "Description": "hybrid-sleep.target", "FailureAction": "none", "FragmentPath": "/dev/null", "Id": "hybrid-sleep.target", "IgnoreOnIsolate": "no", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "LoadError": "org.freedesktop.systemd1.UnitMasked \"Unit hybrid-sleep.target is masked.\"", "LoadState": "masked", "Names": "hybrid-sleep.target", "NeedDaemonReload": "no", "OnFailureJobMode": "replace", "Perpetual": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StateChangeTimestampMonotonic": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "Transient": "no", "UnitFileState": "masked"}} 2021-03-24T21:40:22.027 INFO:teuthology.task.ansible.out: 2021-03-24T21:40:22.082 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:22 +0000 (0:00:00.066) 0:01:20.385 ******* 2021-03-24T21:40:22.103 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-03-24T21:40:22.104 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:22.133 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:22 +0000 (0:00:00.051) 0:01:20.438 ******* 2021-03-24T21:40:22.197 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:22.236 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:22 +0000 (0:00:00.102) 0:01:20.541 ******* 2021-03-24T21:40:22.251 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-03-24T21:40:22.253 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:22.283 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:22 +0000 (0:00:00.047) 0:01:20.588 ******* 2021-03-24T21:40:22.334 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:22.336 INFO:teuthology.task.ansible.out: 2021-03-24T21:40:22.385 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:22 +0000 (0:00:00.101) 0:01:20.689 ******* 2021-03-24T21:40:22.449 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-03-24T21:40:22.452 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:22.486 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:22 +0000 (0:00:00.101) 0:01:20.791 ******* 2021-03-24T21:40:22.526 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:22.565 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:22 +0000 (0:00:00.078) 0:01:20.870 ******* 2021-03-24T21:40:22.663 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-03-24T21:40:22.665 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:22.689 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:22 +0000 (0:00:00.123) 0:01:20.993 ******* 2021-03-24T21:40:22.707 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:22.750 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:22 +0000 (0:00:00.060) 0:01:21.054 ******* 2021-03-24T21:40:22.836 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-03-24T21:40:22.838 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:22.863 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:22 +0000 (0:00:00.111) 0:01:21.166 ******* 2021-03-24T21:40:22.869 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:22.908 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:22 +0000 (0:00:00.046) 0:01:21.213 ******* 2021-03-24T21:40:22.913 INFO:teuthology.task.ansible.out: 2021-03-24T21:40:23.188 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-03-24T21:40:23.190 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.113991", "end": "2021-03-24 21:40:23.508972", "rc": 0, "start": "2021-03-24 21:40:23.394981", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-24T21:40:23.213 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:23 +0000 (0:00:00.305) 0:01:21.518 ******* 2021-03-24T21:40:23.318 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-03-24T21:40:23.323 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:23.344 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:23 +0000 (0:00:00.130) 0:01:21.649 ******* 2021-03-24T21:40:23.355 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-03-24T21:40:23.357 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.280077", "end": "2021-03-24 21:40:23.517194", "rc": 0, "start": "2021-03-24 21:40:23.237117", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-24T21:40:23.382 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:23 +0000 (0:00:00.037) 0:01:21.686 ******* 2021-03-24T21:40:23.507 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-03-24T21:40:23.509 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:23.530 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:23 +0000 (0:00:00.148) 0:01:21.835 ******* 2021-03-24T21:40:23.731 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-03-24T21:40:23.733 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the apt module rather than running 'apt-get'. If you need to use command because apt is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi125.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.130154", "end": "2021-03-24 21:40:24.055499", "rc": 0, "start": "2021-03-24 21:40:23.925345", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-24T21:40:23.759 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:23 +0000 (0:00:00.228) 0:01:22.063 ******* 2021-03-24T21:40:23.883 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.120024", "end": "2021-03-24 21:40:24.056432", "rc": 0, "start": "2021-03-24 21:40:23.936408", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-24T21:40:23.911 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:23 +0000 (0:00:00.152) 0:01:22.216 ******* 2021-03-24T21:40:27.640 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-03-24T21:40:27.641 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1616621749, "cache_updated": false, "changed": false} 2021-03-24T21:40:27.666 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:27 +0000 (0:00:03.755) 0:01:25.971 ******* 2021-03-24T21:40:27.867 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1616621690, "cache_updated": false, "changed": false} 2021-03-24T21:40:27.903 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:27 +0000 (0:00:00.236) 0:01:26.207 ******* 2021-03-24T21:40:28.471 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-03-24T21:40:28.473 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"cache_update_time": 1616621749, "cache_updated": false, "changed": false} 2021-03-24T21:40:28.497 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:28 +0000 (0:00:00.594) 0:01:26.802 ******* 2021-03-24T21:40:28.578 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-03-24T21:40:28.580 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:28.602 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:28 +0000 (0:00:00.104) 0:01:26.906 ******* 2021-03-24T21:40:28.779 INFO:teuthology.task.ansible.out: 2021-03-24T21:40:28.781 INFO:teuthology.task.ansible.out:TASK [common : Install Kerberos Packages (Debian)] ***************************** ok: [smithi125.front.sepia.ceph.com] => {"cache_update_time": 1616621690, "cache_updated": false, "changed": false} 2021-03-24T21:40:28.809 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:28 +0000 (0:00:00.203) 0:01:27.110 ******* 2021-03-24T21:40:28.811 INFO:teuthology.task.ansible.out: 2021-03-24T21:40:28.906 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-03-24T21:40:28.907 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:28.932 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:28 +0000 (0:00:00.126) 0:01:27.236 ******* 2021-03-24T21:40:29.352 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-03-24T21:40:29.354 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2021-03-24T21:40:29.377 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:29 +0000 (0:00:00.445) 0:01:27.682 ******* 2021-03-24T21:40:29.461 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2021-03-24T21:40:29.486 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:29 +0000 (0:00:00.108) 0:01:27.791 ******* 2021-03-24T21:40:30.097 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-03-24T21:40:30.099 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"cache_update_time": 1616621749, "cache_updated": false, "changed": false} 2021-03-24T21:40:30.134 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:30 +0000 (0:00:00.642) 0:01:28.433 ******* 2021-03-24T21:40:30.338 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"cache_update_time": 1616621690, "cache_updated": false, "changed": false} 2021-03-24T21:40:30.365 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:30 +0000 (0:00:00.236) 0:01:28.670 ******* 2021-03-24T21:40:30.808 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-03-24T21:40:30.810 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2021-03-24T21:40:31.064 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2021-03-24T21:40:31.253 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2021-03-24T21:40:31.408 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2021-03-24T21:40:31.600 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2021-03-24T21:40:31.633 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:31 +0000 (0:00:01.267) 0:01:29.937 ******* 2021-03-24T21:40:31.729 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2021-03-24T21:40:31.758 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:31 +0000 (0:00:00.125) 0:01:30.063 ******* 2021-03-24T21:40:31.779 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2021-03-24T21:40:31.807 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:31 +0000 (0:00:00.048) 0:01:30.112 ******* 2021-03-24T21:40:31.917 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2021-03-24T21:40:31.933 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:31 +0000 (0:00:00.125) 0:01:30.238 ******* 2021-03-24T21:40:32.416 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-03-24T21:40:32.418 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 13355, "state": "file", "uid": 0} 2021-03-24T21:40:32.601 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 13355, "state": "file", "uid": 0} 2021-03-24T21:40:32.771 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2021-03-24T21:40:32.927 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2021-03-24T21:40:33.112 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2021-03-24T21:40:33.138 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:33 +0000 (0:00:01.204) 0:01:31.443 ******* 2021-03-24T21:40:33.241 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2021-03-24T21:40:33.270 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:33 +0000 (0:00:00.130) 0:01:31.573 ******* 2021-03-24T21:40:33.447 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-03-24T21:40:33.448 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2021-03-24T21:40:33.641 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:33 +0000 (0:00:00.370) 0:01:31.946 ******* 2021-03-24T21:40:33.649 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2021-03-24T21:40:33.687 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:33 +0000 (0:00:00.046) 0:01:31.992 ******* 2021-03-24T21:40:33.731 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2021-03-24T21:40:33.733 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:33.764 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:33 +0000 (0:00:00.076) 0:01:32.068 ******* 2021-03-24T21:40:33.793 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:33.819 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:33 +0000 (0:00:00.054) 0:01:32.123 ******* 2021-03-24T21:40:33.862 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* skipping: [smithi142.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-24T21:40:33.888 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:33 +0000 (0:00:00.069) 0:01:32.193 ******* 2021-03-24T21:40:33.928 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-24T21:40:33.955 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:33 +0000 (0:00:00.066) 0:01:32.260 ******* 2021-03-24T21:40:34.015 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** skipping: [smithi142.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-24T21:40:34.017 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-24T21:40:34.027 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-24T21:40:34.029 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-24T21:40:34.069 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:34 +0000 (0:00:00.114) 0:01:32.374 ******* 2021-03-24T21:40:34.079 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi125.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-24T21:40:34.093 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-24T21:40:34.095 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-24T21:40:34.130 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:34 +0000 (0:00:00.060) 0:01:32.434 ******* 2021-03-24T21:40:34.195 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-03-24T21:40:34.196 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:34.233 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:34 +0000 (0:00:00.103) 0:01:32.538 ******* 2021-03-24T21:40:34.252 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:34.280 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:34 +0000 (0:00:00.045) 0:01:32.583 ******* 2021-03-24T21:40:34.990 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** ok: [smithi142.front.sepia.ceph.com] => {"cache_update_time": 1616621749, "cache_updated": false, "changed": false} 2021-03-24T21:40:35.020 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:35 +0000 (0:00:00.741) 0:01:33.324 ******* 2021-03-24T21:40:35.047 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"cache_update_time": 1616621690, "cache_updated": false, "changed": false} 2021-03-24T21:40:35.083 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:35 +0000 (0:00:00.063) 0:01:33.388 ******* 2021-03-24T21:40:35.542 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-03-24T21:40:35.544 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2021-03-24T21:40:35.568 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:35 +0000 (0:00:00.484) 0:01:33.872 ******* 2021-03-24T21:40:35.585 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2021-03-24T21:40:35.612 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:35 +0000 (0:00:00.044) 0:01:33.917 ******* 2021-03-24T21:40:36.056 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-03-24T21:40:36.058 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2021-03-24T21:40:36.092 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:36 +0000 (0:00:00.479) 0:01:34.397 ******* 2021-03-24T21:40:36.123 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:36 +0000 (0:00:00.030) 0:01:34.427 ******* 2021-03-24T21:40:36.361 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-03-24T21:40:36.363 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2021-03-24T21:40:36.376 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2021-03-24T21:40:36.493 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2021-03-24T21:40:36.535 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:36 +0000 (0:00:00.412) 0:01:34.840 ******* 2021-03-24T21:40:36.548 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2021-03-24T21:40:36.607 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:36 +0000 (0:00:00.071) 0:01:34.911 ******* 2021-03-24T21:40:36.678 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:36.707 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:36 +0000 (0:00:00.100) 0:01:35.011 ******* 2021-03-24T21:40:36.736 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:36.765 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:36 +0000 (0:00:00.057) 0:01:35.069 ******* 2021-03-24T21:40:36.883 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** skipping: [smithi125.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-24T21:40:36.916 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:36 +0000 (0:00:00.149) 0:01:35.220 ******* 2021-03-24T21:40:36.925 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-24T21:40:36.952 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:36 +0000 (0:00:00.036) 0:01:35.257 ******* 2021-03-24T21:40:36.992 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:37.025 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:37 +0000 (0:00:00.072) 0:01:35.330 ******* 2021-03-24T21:40:37.047 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:37.077 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:37 +0000 (0:00:00.052) 0:01:35.382 ******* 2021-03-24T21:40:37.473 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-03-24T21:40:37.475 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2021-03-24T21:40:37.498 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:37 +0000 (0:00:00.420) 0:01:35.803 ******* 2021-03-24T21:40:37.526 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2021-03-24T21:40:37.551 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:37 +0000 (0:00:00.052) 0:01:35.856 ******* 2021-03-24T21:40:37.792 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-03-24T21:40:37.794 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2021-03-24 21:38:29 UTC", "ActiveEnterTimestampMonotonic": "14841564", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "network.target remote-fs.target var-run.mount basic.target systemd-tmpfiles-setup.service sysinit.target nss-lookup.target -.mount time-sync.target systemd-journald.socket local-fs.target system.slice", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2021-03-24 21:38:29 UTC", "AssertTimestampMonotonic": "14839770", "Before": "shutdown.target xdm.service multi-user.target plymouth-quit.service getty@tty1.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read", "CleanResult": "success", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2021-03-24 21:38:29 UTC", "Co 2021-03-24T21:40:37.796 INFO:teuthology.task.ansible.out:nditionTimestampMonotonic": "14839769", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nagios-nrpe-server.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "889", "ExecMainStartTimestamp": "Wed 2021-03-24 21:38:29 UTC", "ExecMainStartTimestampMonotonic": "14841349", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecReloadEx": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[Wed 2021-03-24 21:38:29 UTC] ; stop_time=[n/a] ; pid=889 ; code=(null) ; status=0/0 }", "ExecStartEx": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; flags= ; start_time=[Wed 2021-03-24 21:38:29 UTC] ; stop_time=[n/a] ; pid=889 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPostEx": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nagios/nrpe.pid ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FinalKillSignal": "9", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GI 2021-03-24T21:40:37.797 INFO:teuthology.task.ansible.out:D": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOReadBytes": "18446744073709551615", "IOReadOperations": "18446744073709551615", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IOWriteBytes": "18446744073709551615", "IOWriteOperations": "18446744073709551615", "IPAccounting": "no", "IPEgressBytes": "[no data]", "IPEgressPackets": "[no data]", "IPIngressBytes": "[no data]", "IPIngressPackets": "[no data]", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2021-03-24 21:38:29 UTC", "InactiveExitTimestampMonotonic": "14841564", "InvocationID": "78775b73e3594d9a9ad54bdcc616d76a", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "524288", "LimitNOFILESoft": "1024", "LimitNPROC": "127777", "LimitNPROCSoft": "127777", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127777", "LimitSIGPENDINGSoft": "127777", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "889", "MemoryAccounting": "yes", "MemoryCurrent": "1318912", "MemoryD 2021-03-24T21:40:37.799 INFO:teuthology.task.ansible.out:enyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMPolicy": "stop", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/run/nagios/nrpe.pid", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectHostname": "no", "ProtectKernelLogs": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "ReloadResult": "success", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice -.mount", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartKillSignal": "15", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2021-03-24 21:38:29 UTC", "StateChangeTimestampMonotonic": "14841564", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no" 2021-03-24T21:40:37.801 INFO:teuthology.task.ansible.out:, "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "38333", "TimeoutAbortUSec": "1min", "TimeoutCleanUSec": "infinity", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "110", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogSignal": "6", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-03-24T21:40:37.819 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:37 +0000 (0:00:00.267) 0:01:36.124 ******* 2021-03-24T21:40:37.860 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2021-03-24 21:38:35 UTC", "ActiveEnterTimestampMonotonic": "15623859", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "-.mount var-run.mount remote-fs.target sysinit.target systemd-journald.socket nss-lookup.target system.slice basic.target time-sync.target network.target systemd-tmpfiles-setup.service local-fs.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2021-03-24 21:38:35 UTC", "AssertTimestampMonotonic": "15611013", "Before": "plymouth-quit.service xdm.service shutdown.target multi-user.target getty@tty1.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read", "CleanResult": "success", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2021-03-24 21:38:35 UTC", "Co 2021-03-24T21:40:37.862 INFO:teuthology.task.ansible.out:nditionTimestampMonotonic": "15611012", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nagios-nrpe-server.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "898", "ExecMainStartTimestamp": "Wed 2021-03-24 21:38:35 UTC", "ExecMainStartTimestampMonotonic": "15623690", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecReloadEx": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[Wed 2021-03-24 21:38:35 UTC] ; stop_time=[n/a] ; pid=898 ; code=(null) ; status=0/0 }", "ExecStartEx": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; flags= ; start_time=[Wed 2021-03-24 21:38:35 UTC] ; stop_time=[n/a] ; pid=898 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPostEx": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nagios/nrpe.pid ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FinalKillSignal": "9", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GI 2021-03-24T21:40:37.864 INFO:teuthology.task.ansible.out:D": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOReadBytes": "18446744073709551615", "IOReadOperations": "18446744073709551615", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IOWriteBytes": "18446744073709551615", "IOWriteOperations": "18446744073709551615", "IPAccounting": "no", "IPEgressBytes": "[no data]", "IPEgressPackets": "[no data]", "IPIngressBytes": "[no data]", "IPIngressPackets": "[no data]", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2021-03-24 21:38:35 UTC", "InactiveExitTimestampMonotonic": "15623859", "InvocationID": "0c6ce922b6734b028e30387645eabc85", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "524288", "LimitNOFILESoft": "1024", "LimitNPROC": "127777", "LimitNPROCSoft": "127777", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127777", "LimitSIGPENDINGSoft": "127777", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "898", "MemoryAccounting": "yes", "MemoryCurrent": "1310720", "MemoryD 2021-03-24T21:40:37.865 INFO:teuthology.task.ansible.out:enyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMPolicy": "stop", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/run/nagios/nrpe.pid", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectHostname": "no", "ProtectKernelLogs": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "ReloadResult": "success", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice -.mount", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartKillSignal": "15", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2021-03-24 21:38:35 UTC", "StateChangeTimestampMonotonic": "15623859", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no" 2021-03-24T21:40:37.867 INFO:teuthology.task.ansible.out:, "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "38333", "TimeoutAbortUSec": "1min", "TimeoutCleanUSec": "infinity", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "110", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogSignal": "6", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-03-24T21:40:37.888 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:37 +0000 (0:00:00.068) 0:01:36.192 ******* 2021-03-24T21:40:37.908 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-03-24T21:40:37.909 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:37.937 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:37 +0000 (0:00:00.049) 0:01:36.242 ******* 2021-03-24T21:40:37.977 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:38.004 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:38 +0000 (0:00:00.067) 0:01:36.309 ******* 2021-03-24T21:40:38.011 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-03-24T21:40:38.013 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:38.039 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:38 +0000 (0:00:00.034) 0:01:36.343 ******* 2021-03-24T21:40:38.078 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:38.105 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:38 +0000 (0:00:00.066) 0:01:36.410 ******* 2021-03-24T21:40:38.124 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-03-24T21:40:38.126 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:38.150 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:38 +0000 (0:00:00.044) 0:01:36.455 ******* 2021-03-24T21:40:38.192 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:38.219 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:38 +0000 (0:00:00.068) 0:01:36.523 ******* 2021-03-24T21:40:38.236 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-03-24T21:40:38.238 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:38.264 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:38 +0000 (0:00:00.045) 0:01:36.569 ******* 2021-03-24T21:40:38.304 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:38.332 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:38 +0000 (0:00:00.067) 0:01:36.636 ******* 2021-03-24T21:40:38.381 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-03-24T21:40:38.382 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:38.406 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:38 +0000 (0:00:00.074) 0:01:36.711 ******* 2021-03-24T21:40:38.426 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:38.454 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:38 +0000 (0:00:00.047) 0:01:36.758 ******* 2021-03-24T21:40:38.508 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-03-24T21:40:38.512 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:38.534 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:38 +0000 (0:00:00.079) 0:01:36.838 ******* 2021-03-24T21:40:38.566 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:38.592 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:38 +0000 (0:00:00.057) 0:01:36.896 ******* 2021-03-24T21:40:38.621 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:38.653 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:38 +0000 (0:00:00.062) 0:01:36.958 ******* 2021-03-24T21:40:38.696 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:38.721 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:38 +0000 (0:00:00.067) 0:01:37.025 ******* 2021-03-24T21:40:38.739 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:38.769 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:38 +0000 (0:00:00.048) 0:01:37.074 ******* 2021-03-24T21:40:38.802 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:38.805 INFO:teuthology.task.ansible.out: 2021-03-24T21:40:38.837 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:38 +0000 (0:00:00.067) 0:01:37.142 ******* 2021-03-24T21:40:38.871 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:38.898 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:38 +0000 (0:00:00.059) 0:01:37.202 ******* 2021-03-24T21:40:38.900 INFO:teuthology.task.ansible.out: 2021-03-24T21:40:38.939 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:38.941 INFO:teuthology.task.ansible.out: 2021-03-24T21:40:38.976 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:38 +0000 (0:00:00.079) 0:01:37.281 ******* 2021-03-24T21:40:38.999 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-03-24T21:40:39.001 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:39.040 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:39 +0000 (0:00:00.063) 0:01:37.344 ******* 2021-03-24T21:40:39.107 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:39.148 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:39 +0000 (0:00:00.108) 0:01:37.452 ******* 2021-03-24T21:40:39.177 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-03-24T21:40:39.179 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2021-03-24T21:40:39.208 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:39 +0000 (0:00:00.058) 0:01:37.511 ******* 2021-03-24T21:40:39.267 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2021-03-24T21:40:39.296 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:39 +0000 (0:00:00.089) 0:01:37.600 ******* 2021-03-24T21:40:39.324 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-03-24T21:40:39.325 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "no_recommended_packages": ["collectl"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2021-03-24T21:40:39.354 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:39 +0000 (0:00:00.057) 0:01:37.658 ******* 2021-03-24T21:40:39.405 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "no_recommended_packages": ["collectl"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2021-03-24T21:40:39.432 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:39 +0000 (0:00:00.078) 0:01:37.737 ******* 2021-03-24T21:40:39.494 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** ok: [smithi125.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_20.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "lttng-tools", "libtool-bin", "docker.io", "python3-nose", "python3-virtualenv", "python3-configobj", "python3-gevent", "python3-numpy", "python3-matplotlib", "python3-setuptools"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_20.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_20.yml"} 2021-03-24T21:40:39.527 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:39 +0000 (0:00:00.095) 0:01:37.832 ******* 2021-03-24T21:40:39.559 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_20.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "lttng-tools", "libtool-bin", "docker.io", "python3-nose", "python3-virtualenv", "python3-configobj", "python3-gevent", "python3-numpy", "python3-matplotlib", "python3-setuptools"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_20.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_20.yml"} 2021-03-24T21:40:39.587 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:39 +0000 (0:00:00.060) 0:01:37.892 ******* 2021-03-24T21:40:39.637 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-03-24T21:40:39.639 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-03-24T21:40:39.673 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:39 +0000 (0:00:00.083) 0:01:37.976 ******* 2021-03-24T21:40:39.704 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-03-24T21:40:39.733 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:39 +0000 (0:00:00.060) 0:01:38.037 ******* 2021-03-24T21:40:39.880 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2021-03-24T21:40:39.907 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:39 +0000 (0:00:00.174) 0:01:38.212 ******* 2021-03-24T21:40:39.917 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2021-03-24T21:40:39.957 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:39 +0000 (0:00:00.049) 0:01:38.262 ******* 2021-03-24T21:40:40.126 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-03-24T21:40:40.128 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-03-24T21:40:40.162 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:40 +0000 (0:00:00.204) 0:01:38.467 ******* 2021-03-24T21:40:40.192 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-03-24T21:40:40.238 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:40 +0000 (0:00:00.075) 0:01:38.543 ******* 2021-03-24T21:40:40.277 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-03-24T21:40:40.279 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:40.308 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:40 +0000 (0:00:00.070) 0:01:38.613 ******* 2021-03-24T21:40:40.383 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:40.417 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:40 +0000 (0:00:00.108) 0:01:38.721 ******* 2021-03-24T21:40:40.461 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-03-24T21:40:40.462 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:40.497 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:40 +0000 (0:00:00.075) 0:01:38.797 ******* 2021-03-24T21:40:40.522 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:40.579 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:40 +0000 (0:00:00.083) 0:01:38.881 ******* 2021-03-24T21:40:40.797 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-03-24T21:40:40.799 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2021-03-24T21:40:40.833 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:40 +0000 (0:00:00.256) 0:01:39.137 ******* 2021-03-24T21:40:40.857 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2021-03-24T21:40:40.903 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:40 +0000 (0:00:00.065) 0:01:39.203 ******* 2021-03-24T21:40:41.496 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-03-24T21:40:41.498 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2021-03-24T21:40:41.523 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:41 +0000 (0:00:00.624) 0:01:39.827 ******* 2021-03-24T21:40:41.661 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* ok: [smithi125.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi125"}, "changed": false} 2021-03-24T21:40:41.685 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:41 +0000 (0:00:00.162) 0:01:39.989 ******* 2021-03-24T21:40:41.757 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-03-24T21:40:41.759 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2021-03-24T21:40:41.793 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:41 +0000 (0:00:00.108) 0:01:40.098 ******* 2021-03-24T21:40:41.922 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-03-24T21:40:41.924 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi142"}, "changed": false} 2021-03-24T21:40:41.955 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:41 +0000 (0:00:00.161) 0:01:40.260 ******* 2021-03-24T21:40:42.783 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-03-24T21:40:42.785 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi142.front.sepia.ceph.com", "ansible_hostname": "smithi142", "ansible_nodename": "smithi142"}, "changed": true, "name": "smithi142"} 2021-03-24T21:40:42.808 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:42 +0000 (0:00:00.853) 0:01:41.113 ******* 2021-03-24T21:40:42.826 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi125.front.sepia.ceph.com", "ansible_hostname": "smithi125", "ansible_nodename": "smithi125"}, "changed": true, "name": "smithi125"} 2021-03-24T21:40:42.858 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:42 +0000 (0:00:00.050) 0:01:41.163 ******* 2021-03-24T21:40:43.399 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2021-03-24T21:40:43.402 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2021-03-24T21:40:43.447 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:43 +0000 (0:00:00.585) 0:01:41.749 ******* 2021-03-24T21:40:43.464 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2021-03-24T21:40:43.500 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:43 +0000 (0:00:00.055) 0:01:41.804 ******* 2021-03-24T21:40:43.969 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2021-03-24T21:40:43.998 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:43 +0000 (0:00:00.498) 0:01:42.303 ******* 2021-03-24T21:40:44.027 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2021-03-24T21:40:44.056 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:44 +0000 (0:00:00.058) 0:01:42.361 ******* 2021-03-24T21:40:44.634 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-03-24T21:40:44.638 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-03-24T21:40:44.664 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:44 +0000 (0:00:00.604) 0:01:42.966 ******* 2021-03-24T21:40:44.734 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-03-24T21:40:44.763 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:44 +0000 (0:00:00.098) 0:01:43.065 ******* 2021-03-24T21:40:44.782 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-03-24T21:40:44.784 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:44.838 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:44 +0000 (0:00:00.077) 0:01:43.142 ******* 2021-03-24T21:40:44.928 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:44.955 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:44 +0000 (0:00:00.117) 0:01:43.259 ******* 2021-03-24T21:40:45.032 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-03-24T21:40:45.034 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:45.057 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:45 +0000 (0:00:00.102) 0:01:43.362 ******* 2021-03-24T21:40:45.084 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:45.119 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:45 +0000 (0:00:00.061) 0:01:43.424 ******* 2021-03-24T21:40:45.179 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-03-24T21:40:45.181 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:45.223 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:45 +0000 (0:00:00.104) 0:01:43.528 ******* 2021-03-24T21:40:45.252 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:45.282 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:45 +0000 (0:00:00.058) 0:01:43.587 ******* 2021-03-24T21:40:45.311 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-03-24T21:40:45.313 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:45.343 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:45 +0000 (0:00:00.060) 0:01:43.647 ******* 2021-03-24T21:40:45.362 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:45.388 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:45 +0000 (0:00:00.045) 0:01:43.692 ******* 2021-03-24T21:40:45.473 INFO:teuthology.task.ansible.out: 2021-03-24T21:40:45.481 INFO:teuthology.task.ansible.out:TASK [testnode : Configure /etc/sudoers.] ************************************** skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:45.503 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:45 +0000 (0:00:00.115) 0:01:43.808 ******* 2021-03-24T21:40:45.513 INFO:teuthology.task.ansible.out: 2021-03-24T21:40:45.535 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:45.572 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:45 +0000 (0:00:00.068) 0:01:43.877 ******* 2021-03-24T21:40:45.634 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:45.673 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:45 +0000 (0:00:00.100) 0:01:43.978 ******* 2021-03-24T21:40:45.776 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:45.802 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:45 +0000 (0:00:00.128) 0:01:44.106 ******* 2021-03-24T21:40:45.839 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:45.866 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:45 +0000 (0:00:00.065) 0:01:44.171 ******* 2021-03-24T21:40:45.992 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:46.026 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:46 +0000 (0:00:00.159) 0:01:44.330 ******* 2021-03-24T21:40:46.049 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-03-24T21:40:46.051 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:46.110 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:46 +0000 (0:00:00.084) 0:01:44.415 ******* 2021-03-24T21:40:46.221 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-03-24T21:40:46.223 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:46.283 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:46 +0000 (0:00:00.168) 0:01:44.584 ******* 2021-03-24T21:40:46.314 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:46 +0000 (0:00:00.031) 0:01:44.615 ******* 2021-03-24T21:40:46.404 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-03-24T21:40:46.428 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:46 +0000 (0:00:00.117) 0:01:44.733 ******* 2021-03-24T21:40:46.437 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-03-24T21:40:46.469 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:46 +0000 (0:00:00.040) 0:01:44.774 ******* 2021-03-24T21:40:46.542 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-03-24T21:40:46.586 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:46 +0000 (0:00:00.117) 0:01:44.891 ******* 2021-03-24T21:40:46.693 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-03-24T21:40:46.721 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:46 +0000 (0:00:00.134) 0:01:45.026 ******* 2021-03-24T21:40:46.742 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-03-24T21:40:46.743 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:46.769 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:46 +0000 (0:00:00.048) 0:01:45.074 ******* 2021-03-24T21:40:46.874 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-03-24T21:40:46.908 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:46 +0000 (0:00:00.138) 0:01:45.213 ******* 2021-03-24T21:40:46.994 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-03-24T21:40:46.996 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-03-24T21:40:47.023 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:47 +0000 (0:00:00.115) 0:01:45.328 ******* 2021-03-24T21:40:47.087 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:47.114 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:47 +0000 (0:00:00.090) 0:01:45.418 ******* 2021-03-24T21:40:47.316 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-03-24T21:40:47.318 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:47.341 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:47 +0000 (0:00:00.227) 0:01:45.646 ******* 2021-03-24T21:40:47.351 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-03-24T21:40:47.353 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi125.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-03-24T21:40:47.392 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:47 +0000 (0:00:00.050) 0:01:45.696 ******* 2021-03-24T21:40:47.486 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-24T21:40:47.488 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:47.522 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:47 +0000 (0:00:00.130) 0:01:45.827 ******* 2021-03-24T21:40:47.569 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-03-24T21:40:47.571 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:47.600 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:47 +0000 (0:00:00.077) 0:01:45.904 ******* 2021-03-24T21:40:47.622 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-03-24T21:40:47.623 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:47.677 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:47 +0000 (0:00:00.070) 0:01:45.975 ******* 2021-03-24T21:40:47.703 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:47.745 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:47 +0000 (0:00:00.074) 0:01:46.049 ******* 2021-03-24T21:40:47.799 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:47.831 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:47 +0000 (0:00:00.085) 0:01:46.135 ******* 2021-03-24T21:40:47.924 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-03-24T21:40:47.931 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:47.951 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:47 +0000 (0:00:00.120) 0:01:46.256 ******* 2021-03-24T21:40:47.991 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-03-24T21:40:47.992 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:48.020 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:48 +0000 (0:00:00.068) 0:01:46.324 ******* 2021-03-24T21:40:48.090 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-03-24T21:40:48.091 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:48.126 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:48 +0000 (0:00:00.106) 0:01:46.431 ******* 2021-03-24T21:40:48.187 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-03-24T21:40:48.189 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:48.219 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:48 +0000 (0:00:00.092) 0:01:46.524 ******* 2021-03-24T21:40:48.319 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-03-24T21:40:48.321 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:48.348 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:48 +0000 (0:00:00.128) 0:01:46.653 ******* 2021-03-24T21:40:48.357 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:48.406 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:48 +0000 (0:00:00.058) 0:01:46.711 ******* 2021-03-24T21:40:48.501 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-03-24T21:40:48.504 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:48.530 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:48 +0000 (0:00:00.123) 0:01:46.835 ******* 2021-03-24T21:40:48.542 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-03-24T21:40:48.543 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:48.593 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:48 +0000 (0:00:00.062) 0:01:46.897 ******* 2021-03-24T21:40:48.657 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:48.689 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:48 +0000 (0:00:00.095) 0:01:46.994 ******* 2021-03-24T21:40:48.752 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-03-24T21:40:48.753 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:48.787 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:48 +0000 (0:00:00.096) 0:01:47.090 ******* 2021-03-24T21:40:48.817 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-03-24T21:40:48.818 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:48.852 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:48 +0000 (0:00:00.066) 0:01:47.157 ******* 2021-03-24T21:40:48.926 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-03-24T21:40:48.928 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:48.953 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:48 +0000 (0:00:00.100) 0:01:47.258 ******* 2021-03-24T21:40:49.006 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-03-24T21:40:49.008 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:49.044 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:49 +0000 (0:00:00.090) 0:01:47.348 ******* 2021-03-24T21:40:49.096 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-03-24T21:40:49.098 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:49.124 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:49 +0000 (0:00:00.080) 0:01:47.429 ******* 2021-03-24T21:40:49.174 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:49.207 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:49 +0000 (0:00:00.083) 0:01:47.512 ******* 2021-03-24T21:40:49.232 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-03-24T21:40:49.234 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:49.263 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:49 +0000 (0:00:00.055) 0:01:47.568 ******* 2021-03-24T21:40:49.308 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-03-24T21:40:49.310 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:49.333 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:49 +0000 (0:00:00.069) 0:01:47.637 ******* 2021-03-24T21:40:49.350 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-03-24T21:40:49.352 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:49.377 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:49 +0000 (0:00:00.044) 0:01:47.682 ******* 2021-03-24T21:40:49.416 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-03-24T21:40:49.418 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:49.442 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:49 +0000 (0:00:00.065) 0:01:47.747 ******* 2021-03-24T21:40:49.512 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-03-24T21:40:49.514 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:49.546 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:49 +0000 (0:00:00.104) 0:01:47.851 ******* 2021-03-24T21:40:49.572 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-03-24T21:40:49.574 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:49.617 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:49 +0000 (0:00:00.070) 0:01:47.922 ******* 2021-03-24T21:40:49.650 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-03-24T21:40:49.652 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:49.678 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:49 +0000 (0:00:00.059) 0:01:47.982 ******* 2021-03-24T21:40:49.722 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-03-24T21:40:49.724 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:49.754 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:49 +0000 (0:00:00.076) 0:01:48.058 ******* 2021-03-24T21:40:49.783 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:49.808 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:49 +0000 (0:00:00.054) 0:01:48.113 ******* 2021-03-24T21:40:49.836 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-03-24T21:40:49.838 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:49.864 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:49 +0000 (0:00:00.055) 0:01:48.168 ******* 2021-03-24T21:40:49.923 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-03-24T21:40:49.925 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:49.949 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:49 +0000 (0:00:00.084) 0:01:48.253 ******* 2021-03-24T21:40:49.966 INFO:teuthology.task.ansible.out: 2021-03-24T21:40:49.966 INFO:teuthology.task.ansible.out:TASK [testnode : Install abrt] ************************************************* 2021-03-24T21:40:49.967 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:49.967 INFO:teuthology.task.ansible.out: 2021-03-24T21:40:49.997 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:49 +0000 (0:00:00.047) 0:01:48.301 ******* 2021-03-24T21:40:49.998 INFO:teuthology.task.ansible.out: 2021-03-24T21:40:50.075 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:50.106 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:50 +0000 (0:00:00.102) 0:01:48.404 ******* 2021-03-24T21:40:50.107 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:50.150 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:50 +0000 (0:00:00.051) 0:01:48.455 ******* 2021-03-24T21:40:50.217 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-03-24T21:40:50.219 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:50.246 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:50 +0000 (0:00:00.095) 0:01:48.550 ******* 2021-03-24T21:40:50.264 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-03-24T21:40:50.266 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:50.306 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:50 +0000 (0:00:00.057) 0:01:48.607 ******* 2021-03-24T21:40:50.367 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:50.408 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:50 +0000 (0:00:00.104) 0:01:48.712 ******* 2021-03-24T21:40:50.418 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-03-24T21:40:50.420 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:50.454 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:50 +0000 (0:00:00.046) 0:01:48.758 ******* 2021-03-24T21:40:50.620 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-03-24T21:40:50.622 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:50.644 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:50 +0000 (0:00:00.190) 0:01:48.949 ******* 2021-03-24T21:40:50.663 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-03-24T21:40:50.665 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.005508", "end": "2021-03-24 21:40:50.797467", "rc": 0, "start": "2021-03-24 21:40:50.791959", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-24T21:40:50.706 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:50 +0000 (0:00:00.062) 0:01:49.011 ******* 2021-03-24T21:40:50.818 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-03-24T21:40:50.849 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:50 +0000 (0:00:00.143) 0:01:49.154 ******* 2021-03-24T21:40:50.878 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-03-24T21:40:50.880 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.005402", "end": "2021-03-24 21:40:51.201404", "rc": 0, "start": "2021-03-24 21:40:51.196002", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-24T21:40:50.910 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:50 +0000 (0:00:00.060) 0:01:49.215 ******* 2021-03-24T21:40:51.002 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-03-24T21:40:51.026 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:51 +0000 (0:00:00.116) 0:01:49.331 ******* 2021-03-24T21:40:51.381 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-03-24T21:40:51.383 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2021-03-24T21:40:51.418 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:51 +0000 (0:00:00.391) 0:01:49.723 ******* 2021-03-24T21:40:51.568 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2021-03-24T21:40:51.599 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:51 +0000 (0:00:00.180) 0:01:49.903 ******* 2021-03-24T21:40:51.613 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-03-24T21:40:51.614 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:51.647 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:51 +0000 (0:00:00.047) 0:01:49.951 ******* 2021-03-24T21:40:51.698 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:40:51.737 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:51 +0000 (0:00:00.090) 0:01:50.042 ******* 2021-03-24T21:40:53.105 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-03-24T21:40:53.106 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} ok: [smithi125.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2021-03-24T21:40:54.034 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2021-03-24T21:40:54.059 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:54 +0000 (0:00:02.321) 0:01:52.363 ******* 2021-03-24T21:40:54.069 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2021-03-24T21:40:54.099 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:54 +0000 (0:00:00.039) 0:01:52.403 ******* 2021-03-24T21:40:54.767 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-03-24T21:40:54.769 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"cache_update_time": 1616621690, "cache_updated": false, "changed": false} 2021-03-24T21:40:54.800 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:54 +0000 (0:00:00.701) 0:01:53.105 ******* 2021-03-24T21:40:54.824 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"cache_update_time": 1616621749, "cache_updated": false, "changed": false} 2021-03-24T21:40:54.851 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:54 +0000 (0:00:00.050) 0:01:53.155 ******* 2021-03-24T21:40:54.880 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-03-24T21:40:54.908 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:54 +0000 (0:00:00.057) 0:01:53.213 ******* 2021-03-24T21:40:54.963 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:54 +0000 (0:00:00.055) 0:01:53.268 ******* 2021-03-24T21:40:55.164 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-03-24T21:40:55.166 INFO:teuthology.task.ansible.out:[WARNING]: Consider using 'become', 'become_method', and 'become_user' rather than running sudo changed: [smithi125.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.031642", "end": "2021-03-24 21:40:55.489669", "rc": 0, "start": "2021-03-24 21:40:55.458027", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-24T21:40:55.197 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:55 +0000 (0:00:00.233) 0:01:53.501 ******* 2021-03-24T21:40:55.226 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.032764", "end": "2021-03-24 21:40:55.399732", "rc": 0, "start": "2021-03-24 21:40:55.366968", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-24T21:40:55.253 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:55 +0000 (0:00:00.056) 0:01:53.558 ******* 2021-03-24T21:40:57.310 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-03-24T21:40:57.312 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"cache_update_time": 1616621690, "cache_updated": false, "changed": false} 2021-03-24T21:40:57.338 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:57 +0000 (0:00:02.084) 0:01:55.642 ******* 2021-03-24T21:40:57.388 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"cache_update_time": 1616621749, "cache_updated": false, "changed": false} 2021-03-24T21:40:57.413 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:40:57 +0000 (0:00:00.075) 0:01:55.718 ******* 2021-03-24T21:41:08.674 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-03-24T21:41:08.676 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1\n libboost-iostreams1.71.0 libboost-thread1.71.0 libbrlapi0.7 libcacard0\n libcdparanoia0 libdv4 libgbm1 libgstreamer-plugins-base1.0-0\n libgstreamer-plugins-good1.0-0 libgudev-1.0-0 libiscsi7 liborc-0.4-0\n libpmem1 libshout3 libslirp0 libspice-server1 libtag1v5 libtag1v5-vanilla\n libusbredirparser1 libv4l-0 libv4lconvert0 libvirglrenderer1 libvisual-0.4-0\n libvte-2.91-0 libvte-2.91-common libwayland-server0 msr-tools ovmf\n qemu-system-data qemu-system-gui seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 35 not upgraded.\nAfter this operation, 62.7 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 138107 files and directories currently installed.)\r\ 2021-03-24T21:41:08.677 INFO:teuthology.task.ansible.out:nRemoving qemu-utils (1:4.2-3ubuntu6.14) ...\r\nRemoving qemu-system-x86 (1:4.2-3ubuntu6.14) ...\r\nRemoving qemu-system-common (1:4.2-3ubuntu6.14) ...\r\nRemoving qemu-block-extra:amd64 (1:4.2-3ubuntu6.14) ...\r\nRemoving librbd1 (15.2.8-0ubuntu0.20.04.1) ...\r\nRemoving librados2 (15.2.8-0ubuntu0.20.04.1) ...\r\nProcessing triggers for man-db (2.9.1-1) ...\r\nProcessing triggers for libc-bin (2.31-0ubuntu9.2) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1", " libboost-iostreams1.71.0 libboost-thread1.71.0 libbrlapi0.7 libcacard0", " libcdparanoia0 libdv4 libgbm1 libgstreamer-plugins-base1.0-0", " libgstreamer-plugins-good1.0-0 libgudev-1.0-0 libiscsi7 liborc-0.4-0", " libpmem1 libshout3 libslirp0 libspice-server1 libtag1v5 libtag1v5-vanilla", " libusbredirparser1 libv4l-0 libv4lconvert0 libvirglrenderer1 libvisual-0.4-0", " libvte-2.91-0 libvte-2.91-common libwayland-server0 msr-tools ovmf", " qemu-system-data qemu-system-gui seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 35 not upgraded.", "After this operation, 62.7 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Read 2021-03-24T21:41:08.679 INFO:teuthology.task.ansible.out:ing database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 138107 files and directories currently installed.)", "Removing qemu-utils (1:4.2-3ubuntu6.14) ...", "Removing qemu-system-x86 (1:4.2-3ubuntu6.14) ...", "Removing qemu-system-common (1:4.2-3ubuntu6.14) ...", "Removing qemu-block-extra:amd64 (1:4.2-3ubuntu6.14) ...", "Removing librbd1 (15.2.8-0ubuntu0.20.04.1) ...", "Removing librados2 (15.2.8-0ubuntu0.20.04.1) ...", "Processing triggers for man-db (2.9.1-1) ...", "Processing triggers for libc-bin (2.31-0ubuntu9.2) ..."]} 2021-03-24T21:41:08.702 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:08 +0000 (0:00:11.288) 0:02:07.006 ******* 2021-03-24T21:41:08.851 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1\n libboost-iostreams1.71.0 libboost-thread1.71.0 libbrlapi0.7 libcacard0\n libcdparanoia0 libdv4 libgbm1 libgstreamer-plugins-base1.0-0\n libgstreamer-plugins-good1.0-0 libgudev-1.0-0 libiscsi7 liborc-0.4-0\n libpmem1 libshout3 libslirp0 libspice-server1 libtag1v5 libtag1v5-vanilla\n libusbredirparser1 libv4l-0 libv4lconvert0 libvirglrenderer1 libvisual-0.4-0\n libvte-2.91-0 libvte-2.91-common libwayland-server0 msr-tools ovmf\n qemu-system-data qemu-system-gui seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 35 not upgraded.\nAfter this operation, 62.7 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 138107 files and directories currently installed.)\r\ 2021-03-24T21:41:08.853 INFO:teuthology.task.ansible.out:nRemoving qemu-utils (1:4.2-3ubuntu6.14) ...\r\nRemoving qemu-system-x86 (1:4.2-3ubuntu6.14) ...\r\nRemoving qemu-system-common (1:4.2-3ubuntu6.14) ...\r\nRemoving qemu-block-extra:amd64 (1:4.2-3ubuntu6.14) ...\r\nRemoving librbd1 (15.2.8-0ubuntu0.20.04.1) ...\r\nRemoving librados2 (15.2.8-0ubuntu0.20.04.1) ...\r\nProcessing triggers for man-db (2.9.1-1) ...\r\nProcessing triggers for libc-bin (2.31-0ubuntu9.2) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1", " libboost-iostreams1.71.0 libboost-thread1.71.0 libbrlapi0.7 libcacard0", " libcdparanoia0 libdv4 libgbm1 libgstreamer-plugins-base1.0-0", " libgstreamer-plugins-good1.0-0 libgudev-1.0-0 libiscsi7 liborc-0.4-0", " libpmem1 libshout3 libslirp0 libspice-server1 libtag1v5 libtag1v5-vanilla", " libusbredirparser1 libv4l-0 libv4lconvert0 libvirglrenderer1 libvisual-0.4-0", " libvte-2.91-0 libvte-2.91-common libwayland-server0 msr-tools ovmf", " qemu-system-data qemu-system-gui seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 35 not upgraded.", "After this operation, 62.7 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Read 2021-03-24T21:41:08.855 INFO:teuthology.task.ansible.out:ing database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 138107 files and directories currently installed.)", "Removing qemu-utils (1:4.2-3ubuntu6.14) ...", "Removing qemu-system-x86 (1:4.2-3ubuntu6.14) ...", "Removing qemu-system-common (1:4.2-3ubuntu6.14) ...", "Removing qemu-block-extra:amd64 (1:4.2-3ubuntu6.14) ...", "Removing librbd1 (15.2.8-0ubuntu0.20.04.1) ...", "Removing librados2 (15.2.8-0ubuntu0.20.04.1) ...", "Processing triggers for man-db (2.9.1-1) ...", "Processing triggers for libc-bin (2.31-0ubuntu9.2) ..."]} 2021-03-24T21:41:08.882 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:08 +0000 (0:00:00.180) 0:02:07.186 ******* 2021-03-24T21:41:09.416 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-03-24T21:41:09.418 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"cache_update_time": 1616621690, "cache_updated": false, "changed": false} 2021-03-24T21:41:09.443 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:09 +0000 (0:00:00.560) 0:02:07.747 ******* 2021-03-24T21:41:09.527 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-03-24T21:41:09.529 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:09.552 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:09 +0000 (0:00:00.108) 0:02:07.856 ******* 2021-03-24T21:41:09.569 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-03-24T21:41:09.571 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"cache_update_time": 1616621749, "cache_updated": false, "changed": false} 2021-03-24T21:41:09.596 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:09 +0000 (0:00:00.044) 0:02:07.900 ******* 2021-03-24T21:41:09.678 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-03-24T21:41:09.680 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:09.702 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:09 +0000 (0:00:00.106) 0:02:08.007 ******* 2021-03-24T21:41:20.849 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-03-24T21:41:20.851 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => {"cache_update_time": 1616621690, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n git-daemon-run | git-daemon-sysvinit git-doc git-el git-email git-gui gitk\n gitweb git-cvs git-mediawiki git-svn samba vde2 debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\nThe following packages will be upgraded:\n git\n1 upgraded, 6 newly installed, 0 to remove and 34 not upgraded.\nNeed to get 18.2 MB of archives.\nAfter this operation, 62.7 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 git amd64 1:2.25.1-1ubuntu3.1 [4557 kB]\nGet:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 librados2 amd64 15.2.8-0ubuntu0.20.04.1 [3209 kB]\nGet:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 librbd1 amd64 15.2.8-0ubuntu0.20.04.1 [1615 kB]\nGet:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-block-extra amd64 1:4.2-3ubuntu6.14 [53.1 kB]\nGet:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-common amd64 1:4.2-3ubuntu6.14 [1055 kB]\nGet:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-x86 amd64 1:4.2-3ubuntu6.14 [6726 kB]\nGet:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-utils amd64 1:4.2-3ubuntu6.14 [975 kB]\nFetched 18.2 MB in 1s (13.7 MB/s)\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading datab 2021-03-24T21:41:20.853 INFO:teuthology.task.ansible.out:ase ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 138035 files and directories currently installed.)\r\nPreparing to unpack .../0-git_1%3a2.25.1-1ubuntu3.1_amd64.deb ...\r\nUnpacking git (1:2.25.1-1ubuntu3.1) over (1:2.25.1-1ubuntu3) ...\r\nSelecting previously unselected package librados2.\r\nPreparing to unpack .../1-librados2_15.2.8-0ubuntu0.20.04.1_amd64.deb ...\r\nUnpacking librados2 (15.2.8-0ubuntu0.20.04.1) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../2-librbd1_15.2.8-0ubuntu0.20.04.1_amd64.deb ...\r\nUnpacking librbd1 (15.2.8-0ubuntu0.20.04.1) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../3-qemu-block-extra_1%3a4.2-3ubuntu6.14_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:4.2-3ubuntu6.14) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../4-qemu-system-common_1%3a4.2-3ubuntu6.14_amd64.deb ...\r\nUnpacking qemu-system-common (1:4.2-3ubuntu6.14) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../5-qemu-system-x86_1%3a4.2-3ubuntu6.14_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:4.2-3ubuntu6.14) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../6-qemu-utils_1%3a4.2-3ubuntu6.14_amd64.deb ...\r\nUnpacking qemu-utils (1:4.2-3ubuntu6.14) ...\r\nSetting up librados2 (15.2.8-0ubuntu0.20.04.1) ...\r\nSetting up git (1:2.25.1-1ubuntu3.1) ...\r\nSetting up librbd1 (15.2.8-0ubuntu0.20.04.1) ...\r\nSetting up qemu-block-extra:amd64 (1:4.2-3ubuntu6.14 2021-03-24T21:41:20.855 INFO:teuthology.task.ansible.out:) ...\r\nSetting up qemu-system-common (1:4.2-3ubuntu6.14) ...\r\nSetting up qemu-system-x86 (1:4.2-3ubuntu6.14) ...\r\nSetting up qemu-utils (1:4.2-3ubuntu6.14) ...\r\nProcessing triggers for man-db (2.9.1-1) ...\r\nProcessing triggers for libc-bin (2.31-0ubuntu9.2) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " git-daemon-run | git-daemon-sysvinit git-doc git-el git-email git-gui gitk", " gitweb git-cvs git-mediawiki git-svn samba vde2 debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "The following packages will be upgraded:", " git", "1 upgraded, 6 newly installed, 0 to remove and 34 not upgraded.", "Need to get 18.2 MB of archives.", "After this operation, 62.7 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 git amd64 1:2.25.1-1ubuntu3.1 [4557 kB]", "Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 librados2 amd64 15.2.8-0ubuntu0.20.04.1 [3209 kB]", "Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 librbd1 amd64 15.2.8-0ubuntu0.20.04.1 [1615 kB]", "Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-block-extra amd64 1:4.2-3ubuntu6.14 [53.1 kB]", "Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-common amd64 1:4.2-3ubuntu6.14 [1055 kB]", "Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-x86 amd64 1:4.2-3ubuntu6.14 [6726 kB]", "Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-utils amd64 1:4.2-3ubuntu6.14 [975 kB]", "Fetched 18.2 MB in 1s (13.7 MB/s)", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%" 2021-03-24T21:41:20.856 INFO:teuthology.task.ansible.out:, "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 138035 files and directories currently installed.)", "Preparing to unpack .../0-git_1%3a2.25.1-1ubuntu3.1_amd64.deb ...", "Unpacking git (1:2.25.1-1ubuntu3.1) over (1:2.25.1-1ubuntu3) ...", "Selecting previously unselected package librados2.", "Preparing to unpack .../1-librados2_15.2.8-0ubuntu0.20.04.1_amd64.deb ...", "Unpacking librados2 (15.2.8-0ubuntu0.20.04.1) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../2-librbd1_15.2.8-0ubuntu0.20.04.1_amd64.deb ...", "Unpacking librbd1 (15.2.8-0ubuntu0.20.04.1) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../3-qemu-block-extra_1%3a4.2-3ubuntu6.14_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:4.2-3ubuntu6.14) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../4-qemu-system-common_1%3a4.2-3ubuntu6.14_amd64.deb ...", "Unpacking qemu-system-common (1:4.2-3ubuntu6.14) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../5-qemu-system-x86_1%3a4.2-3ubuntu6.14_amd64.deb ...", "Unpacking qemu-system-x86 (1:4.2-3ubuntu6.14) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../6-qemu-utils_1%3a4.2-3ubuntu6.14_amd64.deb ...", "Unpacking qemu-utils (1:4.2-3ubuntu6.14) ...", "Setting up librados2 (15.2.8-0ubuntu0.20.04.1) ...", "Setting up git (1:2.25.1-1ubuntu3.1) ...", "Setting up librbd1 (15.2.8-0ubuntu0.20.04.1) ...", "S 2021-03-24T21:41:20.858 INFO:teuthology.task.ansible.out:etting up qemu-block-extra:amd64 (1:4.2-3ubuntu6.14) ...", "Setting up qemu-system-common (1:4.2-3ubuntu6.14) ...", "Setting up qemu-system-x86 (1:4.2-3ubuntu6.14) ...", "Setting up qemu-utils (1:4.2-3ubuntu6.14) ...", "Processing triggers for man-db (2.9.1-1) ...", "Processing triggers for libc-bin (2.31-0ubuntu9.2) ..."]} 2021-03-24T21:41:20.876 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:20 +0000 (0:00:11.173) 0:02:19.180 ******* 2021-03-24T21:41:21.271 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"cache_update_time": 1616621749, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n git-daemon-run | git-daemon-sysvinit git-doc git-el git-email git-gui gitk\n gitweb git-cvs git-mediawiki git-svn samba vde2 debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\nThe following packages will be upgraded:\n git\n1 upgraded, 6 newly installed, 0 to remove and 34 not upgraded.\nNeed to get 18.2 MB of archives.\nAfter this operation, 62.7 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 git amd64 1:2.25.1-1ubuntu3.1 [4557 kB]\nGet:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 librados2 amd64 15.2.8-0ubuntu0.20.04.1 [3209 kB]\nGet:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 librbd1 amd64 15.2.8-0ubuntu0.20.04.1 [1615 kB]\nGet:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-block-extra amd64 1:4.2-3ubuntu6.14 [53.1 kB]\nGet:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-common amd64 1:4.2-3ubuntu6.14 [1055 kB]\nGet:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-x86 amd64 1:4.2-3ubuntu6.14 [6726 kB]\nGet:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-utils amd64 1:4.2-3ubuntu6.14 [975 kB]\nFetched 18.2 MB in 1s (14.0 MB/s)\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading datab 2021-03-24T21:41:21.273 INFO:teuthology.task.ansible.out:ase ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 138035 files and directories currently installed.)\r\nPreparing to unpack .../0-git_1%3a2.25.1-1ubuntu3.1_amd64.deb ...\r\nUnpacking git (1:2.25.1-1ubuntu3.1) over (1:2.25.1-1ubuntu3) ...\r\nSelecting previously unselected package librados2.\r\nPreparing to unpack .../1-librados2_15.2.8-0ubuntu0.20.04.1_amd64.deb ...\r\nUnpacking librados2 (15.2.8-0ubuntu0.20.04.1) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../2-librbd1_15.2.8-0ubuntu0.20.04.1_amd64.deb ...\r\nUnpacking librbd1 (15.2.8-0ubuntu0.20.04.1) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../3-qemu-block-extra_1%3a4.2-3ubuntu6.14_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:4.2-3ubuntu6.14) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../4-qemu-system-common_1%3a4.2-3ubuntu6.14_amd64.deb ...\r\nUnpacking qemu-system-common (1:4.2-3ubuntu6.14) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../5-qemu-system-x86_1%3a4.2-3ubuntu6.14_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:4.2-3ubuntu6.14) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../6-qemu-utils_1%3a4.2-3ubuntu6.14_amd64.deb ...\r\nUnpacking qemu-utils (1:4.2-3ubuntu6.14) ...\r\nSetting up librados2 (15.2.8-0ubuntu0.20.04.1) ...\r\nSetting up git (1:2.25.1-1ubuntu3.1) ...\r\nSetting up librbd1 (15.2.8-0ubuntu0.20.04.1) ...\r\nSetting up qemu-block-extra:amd64 (1:4.2-3ubuntu6.14 2021-03-24T21:41:21.275 INFO:teuthology.task.ansible.out:) ...\r\nSetting up qemu-system-common (1:4.2-3ubuntu6.14) ...\r\nSetting up qemu-system-x86 (1:4.2-3ubuntu6.14) ...\r\nSetting up qemu-utils (1:4.2-3ubuntu6.14) ...\r\nProcessing triggers for man-db (2.9.1-1) ...\r\nProcessing triggers for libc-bin (2.31-0ubuntu9.2) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " git-daemon-run | git-daemon-sysvinit git-doc git-el git-email git-gui gitk", " gitweb git-cvs git-mediawiki git-svn samba vde2 debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "The following packages will be upgraded:", " git", "1 upgraded, 6 newly installed, 0 to remove and 34 not upgraded.", "Need to get 18.2 MB of archives.", "After this operation, 62.7 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 git amd64 1:2.25.1-1ubuntu3.1 [4557 kB]", "Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 librados2 amd64 15.2.8-0ubuntu0.20.04.1 [3209 kB]", "Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 librbd1 amd64 15.2.8-0ubuntu0.20.04.1 [1615 kB]", "Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-block-extra amd64 1:4.2-3ubuntu6.14 [53.1 kB]", "Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-common amd64 1:4.2-3ubuntu6.14 [1055 kB]", "Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-x86 amd64 1:4.2-3ubuntu6.14 [6726 kB]", "Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-utils amd64 1:4.2-3ubuntu6.14 [975 kB]", "Fetched 18.2 MB in 1s (14.0 MB/s)", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%" 2021-03-24T21:41:21.276 INFO:teuthology.task.ansible.out:, "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 138035 files and directories currently installed.)", "Preparing to unpack .../0-git_1%3a2.25.1-1ubuntu3.1_amd64.deb ...", "Unpacking git (1:2.25.1-1ubuntu3.1) over (1:2.25.1-1ubuntu3) ...", "Selecting previously unselected package librados2.", "Preparing to unpack .../1-librados2_15.2.8-0ubuntu0.20.04.1_amd64.deb ...", "Unpacking librados2 (15.2.8-0ubuntu0.20.04.1) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../2-librbd1_15.2.8-0ubuntu0.20.04.1_amd64.deb ...", "Unpacking librbd1 (15.2.8-0ubuntu0.20.04.1) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../3-qemu-block-extra_1%3a4.2-3ubuntu6.14_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:4.2-3ubuntu6.14) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../4-qemu-system-common_1%3a4.2-3ubuntu6.14_amd64.deb ...", "Unpacking qemu-system-common (1:4.2-3ubuntu6.14) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../5-qemu-system-x86_1%3a4.2-3ubuntu6.14_amd64.deb ...", "Unpacking qemu-system-x86 (1:4.2-3ubuntu6.14) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../6-qemu-utils_1%3a4.2-3ubuntu6.14_amd64.deb ...", "Unpacking qemu-utils (1:4.2-3ubuntu6.14) ...", "Setting up librados2 (15.2.8-0ubuntu0.20.04.1) ...", "Setting up git (1:2.25.1-1ubuntu3.1) ...", "Setting up librbd1 (15.2.8-0ubuntu0.20.04.1) ...", "S 2021-03-24T21:41:21.278 INFO:teuthology.task.ansible.out:etting up qemu-block-extra:amd64 (1:4.2-3ubuntu6.14) ...", "Setting up qemu-system-common (1:4.2-3ubuntu6.14) ...", "Setting up qemu-system-x86 (1:4.2-3ubuntu6.14) ...", "Setting up qemu-utils 2021-03-24T21:41:21.278 INFO:teuthology.task.ansible.out: (1:4.2-3ubuntu6.14) ...", "Processing triggers for man-db (2.9.1-1) ...", "Processing triggers for libc-bin (2.31-0ubuntu9.2) ..."]} 2021-03-24T21:41:21.305 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:21 +0000 (0:00:00.429) 0:02:19.610 ******* 2021-03-24T21:41:21.562 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-03-24T21:41:21.564 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"cache_update_time": 1616621690, "cache_updated": false, "changed": false} 2021-03-24T21:41:21.589 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:21 +0000 (0:00:00.283) 0:02:19.893 ******* 2021-03-24T21:41:22.001 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"cache_update_time": 1616621749, "cache_updated": false, "changed": false} 2021-03-24T21:41:22.026 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:22 +0000 (0:00:00.437) 0:02:20.331 ******* 2021-03-24T21:41:22.287 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages with --no-install-recommends] **************** 2021-03-24T21:41:22.289 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"cache_update_time": 1616621690, "cache_updated": false, "changed": false} 2021-03-24T21:41:22.312 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:22 +0000 (0:00:00.285) 0:02:20.617 ******* 2021-03-24T21:41:22.499 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-03-24T21:41:22.501 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2021-03-24T21:41:22.526 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:22 +0000 (0:00:00.213) 0:02:20.831 ******* 2021-03-24T21:41:22.597 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:22.623 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:22 +0000 (0:00:00.096) 0:02:20.927 ******* 2021-03-24T21:41:22.726 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages with --no-install-recommends] **************** 2021-03-24T21:41:22.728 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"cache_update_time": 1616621749, "cache_updated": false, "changed": false} 2021-03-24T21:41:22.752 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:22 +0000 (0:00:00.129) 0:02:21.057 ******* 2021-03-24T21:41:22.847 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-03-24T21:41:22.848 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2021-03-24T21:41:22.872 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:22 +0000 (0:00:00.119) 0:02:21.177 ******* 2021-03-24T21:41:22.957 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-03-24T21:41:22.959 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2021-03-24T21:41:22.986 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:22 +0000 (0:00:00.113) 0:02:21.290 ******* 2021-03-24T21:41:23.076 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-03-24T21:41:23.078 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:23.101 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:23 +0000 (0:00:00.114) 0:02:21.405 ******* 2021-03-24T21:41:23.318 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-03-24T21:41:23.320 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2021-03-24T21:41:23.346 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:23 +0000 (0:00:00.245) 0:02:21.651 ******* 2021-03-24T21:41:23.354 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-03-24T21:41:23.356 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2021-03-24T21:41:23.384 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:23 +0000 (0:00:00.037) 0:02:21.688 ******* 2021-03-24T21:41:23.582 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-03-24T21:41:23.584 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-03-24T21:41:23.608 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:23 +0000 (0:00:00.223) 0:02:21.912 ******* 2021-03-24T21:41:23.793 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-03-24T21:41:23.795 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-24T21:41:23.818 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:23 +0000 (0:00:00.210) 0:02:22.123 ******* 2021-03-24T21:41:23.826 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-03-24T21:41:23.827 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2021-03-24T21:41:23.854 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:23 +0000 (0:00:00.035) 0:02:22.159 ******* 2021-03-24T21:41:24.077 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-03-24T21:41:24.079 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-03-24T21:41:24.102 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:24 +0000 (0:00:00.247) 0:02:22.406 ******* 2021-03-24T21:41:24.142 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-03-24T21:41:24.143 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Wed 2021-03-24 21:38:29 UTC", "ActiveEnterTimestampMonotonic": "15270663", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "basic.target -.mount system.slice systemd-journald.socket systemd-tmpfiles-setup.service sysinit.target remote-fs.target nss-lookup.target network.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2021-03-24 21:38:29 UTC", "AssertTimestampMonotonic": "14759662", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read", "CleanResult": "success", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2021-03-24 21:38:29 UTC", "ConditionTimestampMonotonic": "14759661", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup" 2021-03-24T21:41:24.144 INFO:teuthology.task.ansible.out:: "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", "Documentation": "https://httpd.apache.org/docs/2.4/", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1008", "ExecMainStartTimestamp": "Wed 2021-03-24 21:38:29 UTC", "ExecMainStartTimestampMonotonic": "15270648", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecReloadEx": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartEx": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopEx": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FinalKillSignal": "9", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOReadBytes": "18446744073709551615", "IOReadOperations": "18446744073709551615", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IOWriteBytes": "1844674407370 2021-03-24T21:41:24.146 INFO:teuthology.task.ansible.out:9551615", "IOWriteOperations": "18446744073709551615", "IPAccounting": "no", "IPEgressBytes": "[no data]", "IPEgressPackets": "[no data]", "IPIngressBytes": "[no data]", "IPIngressPackets": "[no data]", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2021-03-24 21:38:29 UTC", "InactiveExitTimestampMonotonic": "14760815", "InvocationID": "7be236f5386346ff88a43f6607a768cc", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "524288", "LimitNOFILESoft": "1024", "LimitNPROC": "127777", "LimitNPROCSoft": "127777", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127777", "LimitSIGPENDINGSoft": "127777", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "1008", "MemoryAccounting": "yes", "MemoryCurrent": "10145792", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPol 2021-03-24T21:41:24.148 INFO:teuthology.task.ansible.out:icy": "n/a", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMPolicy": "stop", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectHostname": "no", "ProtectKernelLogs": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "ReloadResult": "success", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice -.mount", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartKillSignal": "15", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2021-03-24 21:38:29 UTC", "StateChangeTimestampMonotonic": "15270663", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "38333", "TimeoutAbortUSec" 2021-03-24T21:41:24.149 INFO:teuthology.task.ansible.out:: "1min 30s", "TimeoutCleanUSec": "infinity", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogSignal": "6", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-03-24T21:41:24.169 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:24 +0000 (0:00:00.067) 0:02:22.473 ******* 2021-03-24T21:41:24.252 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:24.277 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:24 +0000 (0:00:00.107) 0:02:22.582 ******* 2021-03-24T21:41:24.297 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-03-24T21:41:24.299 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-24T21:41:24.325 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:24 +0000 (0:00:00.048) 0:02:22.630 ******* 2021-03-24T21:41:24.353 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-03-24T21:41:24.355 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:24.382 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:24 +0000 (0:00:00.056) 0:02:22.686 ******* 2021-03-24T21:41:24.452 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-03-24T21:41:24.453 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:24.481 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:24 +0000 (0:00:00.099) 0:02:22.786 ******* 2021-03-24T21:41:24.558 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:24.583 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:24 +0000 (0:00:00.101) 0:02:22.888 ******* 2021-03-24T21:41:24.646 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Wed 2021-03-24 21:38:36 UTC", "ActiveEnterTimestampMonotonic": "15958056", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "sysinit.target remote-fs.target network.target system.slice systemd-journald.socket systemd-tmpfiles-setup.service basic.target -.mount nss-lookup.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2021-03-24 21:38:35 UTC", "AssertTimestampMonotonic": "15538942", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read", "CleanResult": "success", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2021-03-24 21:38:35 UTC", "ConditionTimestampMonotonic": "15538942", 2021-03-24T21:41:24.648 INFO:teuthology.task.ansible.out:"ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", "Documentation": "https://httpd.apache.org/docs/2.4/", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1029", "ExecMainStartTimestamp": "Wed 2021-03-24 21:38:36 UTC", "ExecMainStartTimestampMonotonic": "15958020", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecReloadEx": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartEx": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopEx": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FinalKillSignal": "9", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOReadBytes": "18446744073709551615", "IOReadOperations": "18446744073709551615", "IOSchedulingClass": "0", 2021-03-24T21:41:24.650 INFO:teuthology.task.ansible.out:"IOSchedulingPriority": "0", "IOWeight": "[not set]", "IOWriteBytes": "18446744073709551615", "IOWriteOperations": "18446744073709551615", "IPAccounting": "no", "IPEgressBytes": "[no data]", "IPEgressPackets": "[no data]", "IPIngressBytes": "[no data]", "IPIngressPackets": "[no data]", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2021-03-24 21:38:35 UTC", "InactiveExitTimestampMonotonic": "15540016", "InvocationID": "2f3487d1c11648d79f78e658c1e97c30", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "524288", "LimitNOFILESoft": "1024", "LimitNPROC": "127777", "LimitNPROCSoft": "127777", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127777", "LimitSIGPENDINGSoft": "127777", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "1029", "MemoryAccounting": "yes", "MemoryCurrent": "10133504", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "Moun 2021-03-24T21:41:24.651 INFO:teuthology.task.ansible.out:tFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMPolicy": "stop", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectHostname": "no", "ProtectKernelLogs": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "ReloadResult": "success", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartKillSignal": "15", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2021-03-24 21:38:36 UTC", "StateChangeTimestampMonotonic": "15958056", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "Ta 2021-03-24T21:41:24.653 INFO:teuthology.task.ansible.out:sksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "38333", "TimeoutAbortUSec": "1min 30s", "TimeoutCleanUSec": "infinity", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogSignal": "6", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-03-24T21:41:24.672 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:24 +0000 (0:00:00.088) 0:02:22.977 ******* 2021-03-24T21:41:24.681 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-03-24T21:41:24.682 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:24.707 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:24 +0000 (0:00:00.034) 0:02:23.012 ******* 2021-03-24T21:41:24.747 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-03-24T21:41:24.748 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:24.773 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:24 +0000 (0:00:00.066) 0:02:23.078 ******* 2021-03-24T21:41:24.791 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-03-24T21:41:24.792 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:24.816 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:24 +0000 (0:00:00.042) 0:02:23.121 ******* 2021-03-24T21:41:24.856 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:24.882 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:24 +0000 (0:00:00.065) 0:02:23.186 ******* 2021-03-24T21:41:24.900 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:24.926 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:24 +0000 (0:00:00.044) 0:02:23.231 ******* 2021-03-24T21:41:24.966 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:24.992 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:24 +0000 (0:00:00.066) 0:02:23.297 ******* 2021-03-24T21:41:25.000 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-03-24T21:41:25.002 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:25.027 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:25 +0000 (0:00:00.035) 0:02:23.332 ******* 2021-03-24T21:41:25.067 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-03-24T21:41:25.069 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:25.095 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:25 +0000 (0:00:00.067) 0:02:23.400 ******* 2021-03-24T21:41:25.112 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-24T21:41:25.115 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:25.142 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:25 +0000 (0:00:00.046) 0:02:23.446 ******* 2021-03-24T21:41:25.174 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-03-24T21:41:25.176 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:25.200 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:25 +0000 (0:00:00.058) 0:02:23.505 ******* 2021-03-24T21:41:25.218 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-03-24T21:41:25.219 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:25.248 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:25 +0000 (0:00:00.047) 0:02:23.553 ******* 2021-03-24T21:41:25.277 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-03-24T21:41:25.279 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:25.307 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:25 +0000 (0:00:00.058) 0:02:23.612 ******* 2021-03-24T21:41:25.324 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-03-24T21:41:25.325 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:25.350 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:25 +0000 (0:00:00.042) 0:02:23.654 ******* 2021-03-24T21:41:25.378 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-03-24T21:41:25.380 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:25.404 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:25 +0000 (0:00:00.053) 0:02:23.708 ******* 2021-03-24T21:41:25.405 INFO:teuthology.task.ansible.out: 2021-03-24T21:41:25.433 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-03-24T21:41:25.434 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:25.459 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:25 +0000 (0:00:00.055) 0:02:23.763 ******* 2021-03-24T21:41:25.487 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-03-24T21:41:25.489 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:25.513 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:25 +0000 (0:00:00.054) 0:02:23.818 ******* 2021-03-24T21:41:25.545 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:25.571 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:25 +0000 (0:00:00.057) 0:02:23.875 ******* 2021-03-24T21:41:25.599 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-24T21:41:25.601 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:25.624 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:25 +0000 (0:00:00.053) 0:02:23.929 ******* 2021-03-24T21:41:25.662 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-03-24T21:41:25.664 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:25.691 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:25 +0000 (0:00:00.066) 0:02:23.995 ******* 2021-03-24T21:41:25.698 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-03-24T21:41:25.701 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:25.725 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:25 +0000 (0:00:00.033) 0:02:24.029 ******* 2021-03-24T21:41:25.774 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-03-24T21:41:25.775 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:25.803 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:25 +0000 (0:00:00.078) 0:02:24.108 ******* 2021-03-24T21:41:25.811 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-03-24T21:41:25.813 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:25.838 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:25 +0000 (0:00:00.034) 0:02:24.142 ******* 2021-03-24T21:41:25.877 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-03-24T21:41:25.879 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:25.902 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:25 +0000 (0:00:00.064) 0:02:24.207 ******* 2021-03-24T21:41:25.919 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-03-24T21:41:25.921 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:25.948 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:25 +0000 (0:00:00.045) 0:02:24.253 ******* 2021-03-24T21:41:25.976 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:26.010 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:26 +0000 (0:00:00.062) 0:02:24.315 ******* 2021-03-24T21:41:26.017 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-03-24T21:41:26.020 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:26.048 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:26 +0000 (0:00:00.037) 0:02:24.352 ******* 2021-03-24T21:41:26.084 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-03-24T21:41:26.086 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:26.109 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:26 +0000 (0:00:00.061) 0:02:24.413 ******* 2021-03-24T21:41:26.127 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-03-24T21:41:26.129 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:26.154 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:26 +0000 (0:00:00.043) 0:02:24.457 ******* 2021-03-24T21:41:26.182 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-03-24T21:41:26.184 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:26.209 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:26 +0000 (0:00:00.055) 0:02:24.513 ******* 2021-03-24T21:41:26.236 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:26.268 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:26 +0000 (0:00:00.059) 0:02:24.572 ******* 2021-03-24T21:41:26.296 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-03-24T21:41:26.297 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:26.329 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:26 +0000 (0:00:00.061) 0:02:24.634 ******* 2021-03-24T21:41:26.351 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-03-24T21:41:26.353 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:26.376 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:26 +0000 (0:00:00.046) 0:02:24.681 ******* 2021-03-24T21:41:26.414 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-03-24T21:41:26.416 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:26.446 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:26 +0000 (0:00:00.069) 0:02:24.750 ******* 2021-03-24T21:41:26.464 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-03-24T21:41:26.466 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:26.491 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:26 +0000 (0:00:00.045) 0:02:24.796 ******* 2021-03-24T21:41:26.553 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:26.577 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:26 +0000 (0:00:00.085) 0:02:24.881 ******* 2021-03-24T21:41:26.585 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-03-24T21:41:26.587 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:26.614 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:26 +0000 (0:00:00.037) 0:02:24.919 ******* 2021-03-24T21:41:26.653 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:26.686 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:26 +0000 (0:00:00.071) 0:02:24.990 ******* 2021-03-24T21:41:26.696 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-03-24T21:41:26.698 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:26.727 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:26 +0000 (0:00:00.041) 0:02:25.031 ******* 2021-03-24T21:41:26.777 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-03-24T21:41:26.779 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:26.809 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:26 +0000 (0:00:00.081) 0:02:25.113 ******* 2021-03-24T21:41:26.826 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-03-24T21:41:26.828 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:26.854 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:26 +0000 (0:00:00.045) 0:02:25.159 ******* 2021-03-24T21:41:26.903 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-03-24T21:41:26.905 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:26.928 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:26 +0000 (0:00:00.073) 0:02:25.232 ******* 2021-03-24T21:41:26.945 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-03-24T21:41:26.946 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:26.972 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:26 +0000 (0:00:00.044) 0:02:25.277 ******* 2021-03-24T21:41:27.011 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-03-24T21:41:27.013 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:27.043 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:27 +0000 (0:00:00.071) 0:02:25.348 ******* 2021-03-24T21:41:27.053 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-03-24T21:41:27.056 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:27.082 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:27 +0000 (0:00:00.038) 0:02:25.386 ******* 2021-03-24T21:41:27.174 INFO:teuthology.task.ansible.out: 2021-03-24T21:41:27.175 INFO:teuthology.task.ansible.out:TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:27.202 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:27 +0000 (0:00:00.120) 0:02:25.506 ******* 2021-03-24T21:41:27.252 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-03-24T21:41:27.254 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2021-03-24T21:41:27.277 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:27 +0000 (0:00:00.075) 0:02:25.582 ******* 2021-03-24T21:41:27.285 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-03-24T21:41:27.287 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:27.313 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:27 +0000 (0:00:00.035) 0:02:25.618 ******* 2021-03-24T21:41:27.394 INFO:teuthology.task.ansible.out: 2021-03-24T21:41:27.396 INFO:teuthology.task.ansible.out:TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:27.419 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:27 +0000 (0:00:00.106) 0:02:25.724 ******* 2021-03-24T21:41:27.489 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-03-24T21:41:27.490 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:27.513 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:27 +0000 (0:00:00.093) 0:02:25.817 ******* 2021-03-24T21:41:27.655 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-03-24T21:41:27.657 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2021-03-24T21:41:27.680 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:27 +0000 (0:00:00.167) 0:02:25.985 ******* 2021-03-24T21:41:27.698 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-03-24T21:41:27.699 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2021-03-24T21:41:27.728 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:27 +0000 (0:00:00.047) 0:02:26.033 ******* 2021-03-24T21:41:27.756 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-03-24T21:41:27.758 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:27.785 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:27 +0000 (0:00:00.056) 0:02:26.089 ******* 2021-03-24T21:41:28.129 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-03-24T21:41:28.131 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2021-03-24T21:41:28.154 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:28 +0000 (0:00:00.369) 0:02:26.458 ******* 2021-03-24T21:41:28.193 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-03-24T21:41:28.194 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2021-03-24T21:41:28.194 INFO:teuthology.task.ansible.out: 2021-03-24T21:41:28.220 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:28 +0000 (0:00:00.066) 0:02:26.525 ******* 2021-03-24T21:41:28.228 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-03-24T21:41:28.230 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:28.257 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:28 +0000 (0:00:00.036) 0:02:26.561 ******* 2021-03-24T21:41:28.258 INFO:teuthology.task.ansible.out: 2021-03-24T21:41:28.610 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-03-24T21:41:28.612 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2021-03-24T21:41:28.635 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:28 +0000 (0:00:00.378) 0:02:26.940 ******* 2021-03-24T21:41:28.664 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-03-24T21:41:28.666 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2021-03-24T21:41:28.693 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:28 +0000 (0:00:00.058) 0:02:26.998 ******* 2021-03-24T21:41:29.007 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-03-24T21:41:29.009 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2021-03-24T21:41:29.034 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:29 +0000 (0:00:00.340) 0:02:27.338 ******* 2021-03-24T21:41:29.076 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-03-24T21:41:29.077 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2021-03-24T21:41:29.106 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:29 +0000 (0:00:00.072) 0:02:27.411 ******* 2021-03-24T21:41:29.228 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-03-24T21:41:29.230 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-24T21:41:29.260 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:29 +0000 (0:00:00.153) 0:02:27.564 ******* 2021-03-24T21:41:29.331 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:29.356 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:29 +0000 (0:00:00.096) 0:02:27.660 ******* 2021-03-24T21:41:29.425 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-03-24T21:41:29.426 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:29.449 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:29 +0000 (0:00:00.093) 0:02:27.754 ******* 2021-03-24T21:41:29.488 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-03-24T21:41:29.490 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2021-03-24T21:41:29.517 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:29 +0000 (0:00:00.066) 0:02:27.821 ******* 2021-03-24T21:41:29.534 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-03-24T21:41:29.536 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:29.560 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:29 +0000 (0:00:00.043) 0:02:27.864 ******* 2021-03-24T21:41:29.641 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-03-24T21:41:29.642 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:29.664 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:29 +0000 (0:00:00.104) 0:02:27.969 ******* 2021-03-24T21:41:29.704 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-03-24T21:41:29.706 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-24T21:41:29.733 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:29 +0000 (0:00:00.068) 0:02:28.038 ******* 2021-03-24T21:41:29.740 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-03-24T21:41:29.741 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:29.768 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:29 +0000 (0:00:00.035) 0:02:28.073 ******* 2021-03-24T21:41:29.797 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-03-24T21:41:29.799 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:29.824 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:29 +0000 (0:00:00.055) 0:02:28.129 ******* 2021-03-24T21:41:29.852 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-24T21:41:29.854 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-03-24T21:41:29.878 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:29 +0000 (0:00:00.053) 0:02:28.182 ******* 2021-03-24T21:41:29.897 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-03-24T21:41:29.899 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:29.924 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:29 +0000 (0:00:00.046) 0:02:28.229 ******* 2021-03-24T21:41:29.974 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-03-24T21:41:29.975 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} 2021-03-24T21:41:29.986 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-03-24T21:41:29.988 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2021-03-24T21:41:29.997 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2021-03-24T21:41:29.999 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi125.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-03-24T21:41:30.026 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:30 +0000 (0:00:00.101) 0:02:28.330 ******* 2021-03-24T21:41:30.035 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-03-24T21:41:30.037 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} skipping: [smithi125.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} skipping: [smithi125.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2021-03-24T21:41:30.037 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-03-24T21:41:30.037 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2021-03-24T21:41:30.049 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2021-03-24T21:41:30.050 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-03-24T21:41:30.061 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2021-03-24T21:41:30.062 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-03-24T21:41:30.092 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:30 +0000 (0:00:00.066) 0:02:28.396 ******* 2021-03-24T21:41:30.098 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-03-24T21:41:30.099 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:30.126 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:30 +0000 (0:00:00.033) 0:02:28.430 ******* 2021-03-24T21:41:30.198 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-03-24T21:41:30.200 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:30.225 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:30 +0000 (0:00:00.098) 0:02:28.529 ******* 2021-03-24T21:41:30.306 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-24T21:41:30.308 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-03-24T21:41:30.330 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:30 +0000 (0:00:00.105) 0:02:28.635 ******* 2021-03-24T21:41:30.338 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-03-24T21:41:30.340 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.004013", "end": "2021-03-24 21:41:30.302199", "rc": 0, "start": "2021-03-24 21:41:30.298186", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/virtualenv/bin/teuthology-lock"]} 2021-03-24T21:41:30.367 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:30 +0000 (0:00:00.037) 0:02:28.672 ******* 2021-03-24T21:41:30.418 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-03-24T21:41:30.419 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} 2021-03-24T21:41:30.429 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-03-24T21:41:30.431 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2021-03-24T21:41:30.431 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2021-03-24T21:41:30.442 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-03-24T21:41:30.444 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi142.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-03-24T21:41:30.474 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:30 +0000 (0:00:00.106) 0:02:28.778 ******* 2021-03-24T21:41:30.482 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-03-24T21:41:30.483 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2021-03-24T21:41:30.485 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2021-03-24T21:41:30.496 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-03-24T21:41:30.497 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2021-03-24T21:41:30.508 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-03-24T21:41:30.542 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:30 +0000 (0:00:00.068) 0:02:28.847 ******* 2021-03-24T21:41:30.560 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-03-24T21:41:30.561 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:30.585 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:30 +0000 (0:00:00.043) 0:02:28.890 ******* 2021-03-24T21:41:30.655 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-03-24T21:41:30.656 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:30.681 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:30 +0000 (0:00:00.095) 0:02:28.985 ******* 2021-03-24T21:41:30.792 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-03-24T21:41:30.794 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003169", "end": "2021-03-24 21:41:30.764759", "rc": 0, "start": "2021-03-24 21:41:30.761590", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/virtualenv/bin/teuthology-lock"]} 2021-03-24T21:41:30.820 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:30 +0000 (0:00:00.139) 0:02:29.124 ******* 2021-03-24T21:41:30.888 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-03-24T21:41:30.890 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:30.915 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:30 +0000 (0:00:00.095) 0:02:29.220 ******* 2021-03-24T21:41:30.986 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-03-24T21:41:30.990 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:31.011 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:31 +0000 (0:00:00.095) 0:02:29.316 ******* 2021-03-24T21:41:31.251 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-03-24T21:41:31.253 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:31.277 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:31 +0000 (0:00:00.265) 0:02:29.581 ******* 2021-03-24T21:41:31.368 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-03-24T21:41:31.369 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"cache_update_time": 1616621690, "cache_updated": false, "changed": false} 2021-03-24T21:41:31.403 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:31 +0000 (0:00:00.125) 0:02:29.707 ******* 2021-03-24T21:41:31.475 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-03-24T21:41:31.477 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:31.498 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:31 +0000 (0:00:00.095) 0:02:29.803 ******* 2021-03-24T21:41:31.589 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-03-24T21:41:31.591 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'block_used': 13852083, 'uuid': '765c9454-acf9-4ab1-8c0e-444ef075a2c6', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226243025, 'size_available': 926691430400, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro', 'device': '/dev/sda1', 'inode_used': 150105, 'block_size': 4096, 'inode_available': 60904871}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226243025, "block_size": 4096, "block_total": 240095108, "block_used": 13852083, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60904871, "inode_total": 61054976, "inode_used": 150105, "mount": "/", "options": "rw,relatime,errors=remount-ro", "size_available": 926691430400, "size_total": 983429562368, "uuid": "765c9454-acf9-4ab1-8c0e-444ef075a2c6"}} 2021-03-24T21:41:31.613 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:31 +0000 (0:00:00.114) 0:02:29.918 ******* 2021-03-24T21:41:31.808 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-03-24T21:41:31.810 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.009225", "end": "2021-03-24 21:41:32.141320", "rc": 0, "start": "2021-03-24 21:41:32.132095", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-24T21:41:31.838 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:31 +0000 (0:00:00.224) 0:02:30.143 ******* 2021-03-24T21:41:31.919 INFO:teuthology.task.ansible.out: 2021-03-24T21:41:31.921 INFO:teuthology.task.ansible.out:TASK [testnode : Unmount any non-root mountpoints] ***************************** skipping: [smithi125.front.sepia.ceph.com] => (item={'block_used': 13852083, 'uuid': '765c9454-acf9-4ab1-8c0e-444ef075a2c6', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226243025, 'size_available': 926691430400, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro', 'device': '/dev/sda1', 'inode_used': 150105, 'block_size': 4096, 'inode_available': 60904871}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226243025, "block_size": 4096, "block_total": 240095108, "block_used": 13852083, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60904871, "inode_total": 61054976, "inode_used": 150105, "mount": "/", "options": "rw,relatime,errors=remount-ro", "size_available": 926691430400, "size_total": 983429562368, "uuid": "765c9454-acf9-4ab1-8c0e-444ef075a2c6"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:31.945 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:31 +0000 (0:00:00.106) 0:02:30.249 ******* 2021-03-24T21:41:31.972 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-03-24T21:41:31.974 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"cache_update_time": 1616621749, "cache_updated": false, "changed": false} 2021-03-24T21:41:32.002 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:31 +0000 (0:00:00.056) 0:02:30.306 ******* 2021-03-24T21:41:32.019 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-03-24T21:41:32.044 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:32 +0000 (0:00:00.042) 0:02:30.349 ******* 2021-03-24T21:41:32.073 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-03-24T21:41:32.074 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:32.100 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:32 +0000 (0:00:00.055) 0:02:30.404 ******* 2021-03-24T21:41:32.139 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-24T21:41:32.141 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:32.184 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-03-24T21:41:32.186 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'block_used': 13852087, 'uuid': '765c9454-acf9-4ab1-8c0e-444ef075a2c6', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226243021, 'size_available': 926691414016, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro', 'device': '/dev/sda1', 'inode_used': 150105, 'block_size': 4096, 'inode_available': 60904871}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226243021, "block_size": 4096, "block_total": 240095108, "block_used": 13852087, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60904871, "inode_total": 61054976, "inode_used": 150105, "mount": "/", "options": "rw,relatime,errors=remount-ro", "size_available": 926691414016, "size_total": 983429562368, "uuid": "765c9454-acf9-4ab1-8c0e-444ef075a2c6"}} 2021-03-24T21:41:32.209 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:32 +0000 (0:00:00.109) 0:02:30.513 ******* 2021-03-24T21:41:32.394 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-03-24T21:41:32.396 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.009405", "end": "2021-03-24 21:41:32.575552", "rc": 0, "start": "2021-03-24 21:41:32.566147", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-24T21:41:32.419 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:32 +0000 (0:00:00.209) 0:02:30.723 ******* 2021-03-24T21:41:32.646 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** skipping: [smithi142.front.sepia.ceph.com] => (item={'block_used': 13852087, 'uuid': '765c9454-acf9-4ab1-8c0e-444ef075a2c6', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226243021, 'size_available': 926691414016, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro', 'device': '/dev/sda1', 'inode_used': 150105, 'block_size': 4096, 'inode_available': 60904871}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226243021, "block_size": 4096, "block_total": 240095108, "block_used": 13852087, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60904871, "inode_total": 61054976, "inode_used": 150105, "mount": "/", "options": "rw,relatime,errors=remount-ro", "size_available": 926691414016, "size_total": 983429562368, "uuid": "765c9454-acf9-4ab1-8c0e-444ef075a2c6"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:32.774 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:32 +0000 (0:00:00.252) 0:02:30.976 ******* 2021-03-24T21:41:32.886 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-03-24T21:41:32.910 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:32 +0000 (0:00:00.239) 0:02:31.215 ******* 2021-03-24T21:41:33.136 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-24T21:41:33.137 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:34.056 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-SXkXEdF4Js7S9ZmkRjUiqKTCZctZT11220LL7rS3BYZGXB3ahqn20YFMfTNjMdfX'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.632176", "end": "2021-03-24 21:41:34.378135", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-SXkXEdF4Js7S9ZmkRjUiqKTCZctZT11220LL7rS3BYZGXB3ahqn20YFMfTNjMdfX"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-24 21:41:32.745959", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries in memory.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries in memory.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-24T21:41:34.057 INFO:teuthology.task.ansible.out: 2021-03-24T21:41:34.068 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:34.069 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:34.805 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-P9CAM2gYonr6HY8FNX70SA0xNBy0rKS7rFV1uDTSk148a1aOFKB01ILJsu785PEk'], 'uuids': ['92b60d3a-c02f-48bb-bf66-aa9983e742e9']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.537116", "end": "2021-03-24 21:41:34.970827", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-P9CAM2gYonr6HY8FNX70SA0xNBy0rKS7rFV1uDTSk148a1aOFKB01ILJsu785PEk"], "labels": [], "masters": [], "uuids": ["92b60d3a-c02f-48bb-bf66-aa9983e742e9"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-24 21:41:33.433711", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries in memory.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries in memory.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-24T21:41:34.818 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:34.819 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:35.338 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-SXkXEdF4Js7S9ZmkRjUiqKTCZctZT112eolep4nDWE30AiKj41zbmFFWv25cOzo5'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.135546", "end": "2021-03-24 21:41:35.661102", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-SXkXEdF4Js7S9ZmkRjUiqKTCZctZT112eolep4nDWE30AiKj41zbmFFWv25cOzo5"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-24 21:41:34.525556", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries in memory.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries in memory.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-24T21:41:35.350 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QHY2', 'wwn-0x5000c50091e1abc3'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '765c9454-acf9-4ab1-8c0e-444ef075a2c6', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QHY2-part1', 'wwn-0x5000c50091e1abc3-part1'], 'uuids': ['765c9454-acf9-4ab1-8c0e-444ef075a2c6']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c50091e1abc3', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5QHY2', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QHY2", "wwn-0x5000c50091e1abc3"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QHY2-part1", "wwn-0x5000c50091e1abc3-part1"], "labels": [], "masters": [], "uuids": ["765c9454-acf9-4ab1-8c0e-444ef075a2c6"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "765c9454-acf9-4ab1-8c0e-444ef075a2c6"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5QHY2", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50091e1abc3"}}, 2021-03-24T21:41:35.352 INFO:teuthology.task.ansible.out: "skip_reason": "Conditional result was False"} 2021-03-24T21:41:36.123 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-P9CAM2gYonr6HY8FNX70SA0xNBy0rKS7bDNBLeIdmlsGPFSWK6TLlnN27V3Xyj5Z'], 'uuids': ['52185abc-3307-4163-a56f-4aa637808291']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.173728", "end": "2021-03-24 21:41:36.289768", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-P9CAM2gYonr6HY8FNX70SA0xNBy0rKS7bDNBLeIdmlsGPFSWK6TLlnN27V3Xyj5Z"], "labels": [], "masters": [], "uuids": ["52185abc-3307-4163-a56f-4aa637808291"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-24 21:41:35.116040", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries in memory.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries in memory.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-24T21:41:36.135 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFVQ', 'wwn-0x5000c5009292426c'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '765c9454-acf9-4ab1-8c0e-444ef075a2c6', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFVQ-part1', 'wwn-0x5000c5009292426c-part1'], 'uuids': ['765c9454-acf9-4ab1-8c0e-444ef075a2c6']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c5009292426c', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5XFVQ', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFVQ", "wwn-0x5000c5009292426c"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFVQ-part1", "wwn-0x5000c5009292426c-part1"], "labels": [], "masters": [], "uuids": ["765c9454-acf9-4ab1-8c0e-444ef075a2c6"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "765c9454-acf9-4ab1-8c0e-444ef075a2c6"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XFVQ", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c5009292426c"}}, 2021-03-24T21:41:36.137 INFO:teuthology.task.ansible.out: "skip_reason": "Conditional result was False"} 2021-03-24T21:41:36.670 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-aHn4g2-hKZw-8wc8-KfIp-jczc-GW0d-3oRtiL', 'nvme-INTEL_SSDPEDMD400G4_PHFT62040123400BGN', 'nvme-nvme.8086-50484654363230343031323334303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.197509", "end": "2021-03-24 21:41:36.994157", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-aHn4g2-hKZw-8wc8-KfIp-jczc-GW0d-3oRtiL", "nvme-INTEL_SSDPEDMD400G4_PHFT62040123400BGN", "nvme-nvme.8086-50484654363230343031323334303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-24 21:41:35.796648", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries in memory.\nGPT data structures d 2021-03-24T21:41:36.672 INFO:teuthology.task.ansible.out:estroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries in memory.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-24T21:41:36.682 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:36.683 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:37.443 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-V4WU43-WfWt-xq5p-tHkS-Y8lr-REOb-rs4vVb', 'nvme-INTEL_SSDPEDMD400G4_CVFT6303009Y400BGN', 'nvme-nvme.8086-43564654363330333030395934303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.180855", "end": "2021-03-24 21:41:37.609139", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-V4WU43-WfWt-xq5p-tHkS-Y8lr-REOb-rs4vVb", "nvme-INTEL_SSDPEDMD400G4_CVFT6303009Y400BGN", "nvme-nvme.8086-43564654363330333030395934303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-24 21:41:36.428284", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries in memory.\nGPT data structures d 2021-03-24T21:41:37.445 INFO:teuthology.task.ansible.out:estroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries in memory.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-24T21:41:37.455 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:37.456 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:37.980 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-SXkXEdF4Js7S9ZmkRjUiqKTCZctZT1120XPOqf1gqKrFzJdLoHhdAtNtbNFsN45o'], 'uuids': ['5f8ef52e-921b-4907-b9d0-062814c62c89']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.169268", "end": "2021-03-24 21:41:38.306501", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-SXkXEdF4Js7S9ZmkRjUiqKTCZctZT1120XPOqf1gqKrFzJdLoHhdAtNtbNFsN45o"], "labels": [], "masters": [], "uuids": ["5f8ef52e-921b-4907-b9d0-062814c62c89"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-24 21:41:37.137233", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries in memory.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries in memory.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-24T21:41:37.992 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:38.004 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:38.719 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-P9CAM2gYonr6HY8FNX70SA0xNBy0rKS72hIOIDBlZaVeCd3jwwKhghRVoEIdT26P'], 'uuids': ['697b65b0-e63c-4089-a019-3f091c63e887']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.130667", "end": "2021-03-24 21:41:38.888873", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-P9CAM2gYonr6HY8FNX70SA0xNBy0rKS72hIOIDBlZaVeCd3jwwKhghRVoEIdT26P"], "labels": [], "masters": [], "uuids": ["697b65b0-e63c-4089-a019-3f091c63e887"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-24 21:41:37.758206", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries in memory.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries in memory.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-24T21:41:38.731 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:38.743 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:39.278 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-SXkXEdF4Js7S9ZmkRjUiqKTCZctZT112hlTODlA4tjtYEN65nwKBNF2sIyJj3wDq'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.148131", "end": "2021-03-24 21:41:39.597673", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-SXkXEdF4Js7S9ZmkRjUiqKTCZctZT112hlTODlA4tjtYEN65nwKBNF2sIyJj3wDq"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-24 21:41:38.449542", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries in memory.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries in memory.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-24T21:41:39.290 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:40.033 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-P9CAM2gYonr6HY8FNX70SA0xNBy0rKS77xLINRfci0aJ9I8K60cQoZVfa5MDR1eI'], 'uuids': ['fc91bd17-74ee-4355-950b-c89bf0228fdf']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.159303", "end": "2021-03-24 21:41:40.199693", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-P9CAM2gYonr6HY8FNX70SA0xNBy0rKS77xLINRfci0aJ9I8K60cQoZVfa5MDR1eI"], "labels": [], "masters": [], "uuids": ["fc91bd17-74ee-4355-950b-c89bf0228fdf"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-24 21:41:39.040390", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries in memory.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries in memory.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-24T21:41:40.034 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:40.555 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-SXkXEdF4Js7S9ZmkRjUiqKTCZctZT1125htgjJE6j7sSwyj6wrEfcYdCFdTL6eF8'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.133666", "end": "2021-03-24 21:41:40.883463", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-SXkXEdF4Js7S9ZmkRjUiqKTCZctZT1125htgjJE6j7sSwyj6wrEfcYdCFdTL6eF8"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-24 21:41:39.749797", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries in memory.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries in memory.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-24T21:41:40.607 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:40 +0000 (0:00:07.696) 0:02:38.912 ******* 2021-03-24T21:41:40.886 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-03-24T21:41:40.888 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:41.185 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-SXkXEdF4Js7S9ZmkRjUiqKTCZctZT11220LL7rS3BYZGXB3ahqn20YFMfTNjMdfX'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.165712", "end": "2021-03-24 21:41:41.505303", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-SXkXEdF4Js7S9ZmkRjUiqKTCZctZT11220LL7rS3BYZGXB3ahqn20YFMfTNjMdfX"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-24 21:41:41.339591", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.160709 s, 718 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.160709 s, 718 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-24T21:41:41.186 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:41.198 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:41.556 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-24T21:41:41.558 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-P9CAM2gYonr6HY8FNX70SA0xNBy0rKS7tvEmMTTsnippdE8yR5hzEN65iGhe3C2n'], 'uuids': ['21a896f2-c1a9-4cef-a4c7-c4d8f740506e']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.267105", "end": "2021-03-24 21:41:41.601902", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-P9CAM2gYonr6HY8FNX70SA0xNBy0rKS7tvEmMTTsnippdE8yR5hzEN65iGhe3C2n"], "labels": [], "masters": [], "uuids": ["21a896f2-c1a9-4cef-a4c7-c4d8f740506e"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-24 21:41:40.334797", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries in memory.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries in memory.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-24T21:41:41.580 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-03-24T21:41:41.582 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-SXkXEdF4Js7S9ZmkRjUiqKTCZctZT112eolep4nDWE30AiKj41zbmFFWv25cOzo5'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.168038", "end": "2021-03-24 21:41:41.821278", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-SXkXEdF4Js7S9ZmkRjUiqKTCZctZT112eolep4nDWE30AiKj41zbmFFWv25cOzo5"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-24 21:41:41.653240", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.16308 s, 707 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.16308 s, 707 MB/s"], "stdout": "", "stdout_lines": []} skipping: [smithi125.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QHY2', 'wwn-0x5000c50091e1abc3'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '765c9454-acf9-4ab1-8c0e-444ef075a2c6', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QHY2-part1', 'wwn-0x5000c500 2021-03-24T21:41:41.584 INFO:teuthology.task.ansible.out:91e1abc3-part1'], 'uuids': ['765c9454-acf9-4ab1-8c0e-444ef075a2c6']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c50091e1abc3', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5QHY2', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QHY2", "wwn-0x5000c50091e1abc3"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QHY2-part1", "wwn-0x5000c50091e1abc3-part1"], "labels": [], "masters": [], "uuids": ["765c9454-acf9-4ab1-8c0e-444ef075a2c6"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "765c9454-acf9-4ab1-8c0e-444ef075a2c6"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5QHY2", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50091e1abc3"}}, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:41.608 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:41 +0000 (0:00:01.000) 0:02:39.913 ******* 2021-03-24T21:41:41.692 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-aHn4g2-hKZw-8wc8-KfIp-jczc-GW0d-3oRtiL', 'nvme-INTEL_SSDPEDMD400G4_PHFT62040123400BGN', 'nvme-nvme.8086-50484654363230343031323334303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.057443", "end": "2021-03-24 21:41:42.018008", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-aHn4g2-hKZw-8wc8-KfIp-jczc-GW0d-3oRtiL", "nvme-INTEL_SSDPEDMD400G4_PHFT62040123400BGN", "nvme-nvme.8086-50484654363230343031323334303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-24 21:41:41.960565", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0526325 s, 2.2 GB/s", "std 2021-03-24T21:41:41.694 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0526325 s, 2.2 GB/s"], "stdout": "", "stdout_lines": []} skipping: [smithi125.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:41.696 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:41.707 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:41.983 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-SXkXEdF4Js7S9ZmkRjUiqKTCZctZT1120XPOqf1gqKrFzJdLoHhdAtNtbNFsN45o'], 'uuids': ['5f8ef52e-921b-4907-b9d0-062814c62c89']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.150873", "end": "2021-03-24 21:41:42.310080", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-SXkXEdF4Js7S9ZmkRjUiqKTCZctZT1120XPOqf1gqKrFzJdLoHhdAtNtbNFsN45o"], "labels": [], "masters": [], "uuids": ["5f8ef52e-921b-4907-b9d0-062814c62c89"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-24 21:41:42.159207", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.14706 s, 784 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.14706 s, 784 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-24T21:41:41.994 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:41.996 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-P9CAM2gYonr6HY8FNX70SA0xNBy0rKS7rFV1uDTSk148a1aOFKB01ILJsu785PEk'], 'uuids': ['92b60d3a-c02f-48bb-bf66-aa9983e742e9']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.163628", "end": "2021-03-24 21:41:42.166963", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-P9CAM2gYonr6HY8FNX70SA0xNBy0rKS7rFV1uDTSk148a1aOFKB01ILJsu785PEk"], "labels": [], "masters": [], "uuids": ["92b60d3a-c02f-48bb-bf66-aa9983e742e9"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-24 21:41:42.003335", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.158691 s, 727 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.158691 s, 727 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-24T21:41:41.997 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:42.009 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:42.021 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:42.023 INFO:teuthology.task.ansible.out: 2021-03-24T21:41:42.295 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-SXkXEdF4Js7S9ZmkRjUiqKTCZctZT112hlTODlA4tjtYEN65nwKBNF2sIyJj3wDq'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.156275", "end": "2021-03-24 21:41:42.614363", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-SXkXEdF4Js7S9ZmkRjUiqKTCZctZT112hlTODlA4tjtYEN65nwKBNF2sIyJj3wDq"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-24 21:41:42.458088", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.152912 s, 754 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.152912 s, 754 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-24T21:41:42.296 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:42.308 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-P9CAM2gYonr6HY8FNX70SA0xNBy0rKS7bDNBLeIdmlsGPFSWK6TLlnN27V3Xyj5Z'], 'uuids': ['52185abc-3307-4163-a56f-4aa637808291']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.164490", "end": "2021-03-24 21:41:42.475531", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-P9CAM2gYonr6HY8FNX70SA0xNBy0rKS7bDNBLeIdmlsGPFSWK6TLlnN27V3Xyj5Z"], "labels": [], "masters": [], "uuids": ["52185abc-3307-4163-a56f-4aa637808291"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-24 21:41:42.311041", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.159562 s, 723 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.159562 s, 723 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-24T21:41:42.320 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFVQ', 'wwn-0x5000c5009292426c'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '765c9454-acf9-4ab1-8c0e-444ef075a2c6', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFVQ-part1', 'wwn-0x5000c5009292426c-part1'], 'uuids': ['765c9454-acf9-4ab1-8c0e-444ef075a2c6']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c5009292426c', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5XFVQ', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFVQ", "wwn-0x5000c5009292426c"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFVQ-part1", "wwn-0x5000c5009292426c-part1"], "labels": [], "masters": [], "uuids": ["765c9454-acf9-4ab1-8c0e-444ef075a2c6"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "765c9454-acf9-4ab1-8c0e-444ef075a2c6"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XFVQ", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c5009292426c"}}, 2021-03-24T21:41:42.321 INFO:teuthology.task.ansible.out: "skip_reason": "Conditional result was False"} 2021-03-24T21:41:42.509 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-V4WU43-WfWt-xq5p-tHkS-Y8lr-REOb-rs4vVb', 'nvme-INTEL_SSDPEDMD400G4_CVFT6303009Y400BGN', 'nvme-nvme.8086-43564654363330333030395934303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.058413", "end": "2021-03-24 21:41:42.676215", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-V4WU43-WfWt-xq5p-tHkS-Y8lr-REOb-rs4vVb", "nvme-INTEL_SSDPEDMD400G4_CVFT6303009Y400BGN", "nvme-nvme.8086-43564654363330333030395934303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-24 21:41:42.617802", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0535767 s, 2.2 GB/s", "std 2021-03-24T21:41:42.511 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0535767 s, 2.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-03-24T21:41:42.521 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:42.522 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:42.596 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-SXkXEdF4Js7S9ZmkRjUiqKTCZctZT1125htgjJE6j7sSwyj6wrEfcYdCFdTL6eF8'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.164880", "end": "2021-03-24 21:41:42.919293", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-SXkXEdF4Js7S9ZmkRjUiqKTCZctZT1125htgjJE6j7sSwyj6wrEfcYdCFdTL6eF8"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-24 21:41:42.754413", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.161654 s, 714 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.161654 s, 714 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-24T21:41:42.631 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:42 +0000 (0:00:01.022) 0:02:40.936 ******* 2021-03-24T21:41:42.817 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-P9CAM2gYonr6HY8FNX70SA0xNBy0rKS72hIOIDBlZaVeCd3jwwKhghRVoEIdT26P'], 'uuids': ['697b65b0-e63c-4089-a019-3f091c63e887']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.159316", "end": "2021-03-24 21:41:42.982837", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-P9CAM2gYonr6HY8FNX70SA0xNBy0rKS72hIOIDBlZaVeCd3jwwKhghRVoEIdT26P"], "labels": [], "masters": [], "uuids": ["697b65b0-e63c-4089-a019-3f091c63e887"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-24 21:41:42.823521", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.155603 s, 741 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.155603 s, 741 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-24T21:41:42.818 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:42.830 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:42.966 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-03-24T21:41:42.968 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.151536", "end": "2021-03-24 21:41:43.250217", "rc": 0, "start": "2021-03-24 21:41:43.098681", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-24T21:41:42.990 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:42 +0000 (0:00:00.359) 0:02:41.295 ******* 2021-03-24T21:41:43.112 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-03-24T21:41:43.114 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-P9CAM2gYonr6HY8FNX70SA0xNBy0rKS77xLINRfci0aJ9I8K60cQoZVfa5MDR1eI'], 'uuids': ['fc91bd17-74ee-4355-950b-c89bf0228fdf']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.164093", "end": "2021-03-24 21:41:43.286446", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-P9CAM2gYonr6HY8FNX70SA0xNBy0rKS77xLINRfci0aJ9I8K60cQoZVfa5MDR1eI"], "labels": [], "masters": [], "uuids": ["fc91bd17-74ee-4355-950b-c89bf0228fdf"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-24 21:41:43.122353", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.15912 s, 725 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.15912 s, 725 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-24T21:41:43.124 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:43.208 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-03-24T21:41:43.210 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.082684", "end": "2021-03-24 21:41:43.539029", "rc": 0, "start": "2021-03-24 21:41:43.456345", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-24T21:41:43.232 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:43 +0000 (0:00:00.241) 0:02:41.536 ******* 2021-03-24T21:41:43.312 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-03-24T21:41:43.382 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:43 +0000 (0:00:00.150) 0:02:41.687 ******* 2021-03-24T21:41:43.410 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-03-24T21:41:43.412 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-P9CAM2gYonr6HY8FNX70SA0xNBy0rKS7tvEmMTTsnippdE8yR5hzEN65iGhe3C2n'], 'uuids': ['21a896f2-c1a9-4cef-a4c7-c4d8f740506e']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.164593", "end": "2021-03-24 21:41:43.581802", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-P9CAM2gYonr6HY8FNX70SA0xNBy0rKS7tvEmMTTsnippdE8yR5hzEN65iGhe3C2n"], "labels": [], "masters": [], "uuids": ["21a896f2-c1a9-4cef-a4c7-c4d8f740506e"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-24 21:41:43.417209", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.159594 s, 723 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.159594 s, 723 MB/s"], "stdout": "", "stdout_lines": []} TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:43.442 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:43 +0000 (0:00:00.059) 0:02:41.747 ******* 2021-03-24T21:41:43.454 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-03-24T21:41:43.487 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:43 +0000 (0:00:00.045) 0:02:41.792 ******* 2021-03-24T21:41:43.497 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-03-24T21:41:43.499 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:43.525 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:43 +0000 (0:00:00.038) 0:02:41.830 ******* 2021-03-24T21:41:43.554 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:43.582 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:43 +0000 (0:00:00.056) 0:02:41.886 ******* 2021-03-24T21:41:43.610 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-03-24T21:41:43.612 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:43.634 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:43 +0000 (0:00:00.052) 0:02:41.939 ******* 2021-03-24T21:41:43.737 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-03-24T21:41:43.738 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.101162", "end": "2021-03-24 21:41:43.901093", "rc": 0, "start": "2021-03-24 21:41:43.799931", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-24T21:41:43.770 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:43 +0000 (0:00:00.135) 0:02:42.074 ******* 2021-03-24T21:41:43.778 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-03-24T21:41:43.780 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-24T21:41:43.806 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:43 +0000 (0:00:00.036) 0:02:42.111 ******* 2021-03-24T21:41:43.845 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-03-24T21:41:43.847 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'block_used': 13852083, 'uuid': '765c9454-acf9-4ab1-8c0e-444ef075a2c6', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226243025, 'size_available': 926691430400, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro', 'device': '/dev/sda1', 'inode_used': 150105, 'block_size': 4096, 'inode_available': 60904871}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226243025, "block_size": 4096, "block_total": 240095108, "block_used": 13852083, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60904871, "inode_total": 61054976, "inode_used": 150105, "mount": "/", "options": "rw,relatime,errors=remount-ro", "size_available": 926691430400, "size_total": 983429562368, "uuid": "765c9454-acf9-4ab1-8c0e-444ef075a2c6"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:43.871 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:43 +0000 (0:00:00.064) 0:02:42.175 ******* 2021-03-24T21:41:43.899 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:43.922 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:43 +0000 (0:00:00.051) 0:02:42.227 ******* 2021-03-24T21:41:43.950 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-24T21:41:43.952 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:43.973 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:43 +0000 (0:00:00.051) 0:02:42.278 ******* 2021-03-24T21:41:43.991 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-03-24T21:41:43.993 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.082219", "end": "2021-03-24 21:41:44.169710", "rc": 0, "start": "2021-03-24 21:41:44.087491", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-24T21:41:44.015 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:44 +0000 (0:00:00.042) 0:02:42.320 ******* 2021-03-24T21:41:44.022 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-24T21:41:44.024 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:44.052 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:44 +0000 (0:00:00.035) 0:02:42.356 ******* 2021-03-24T21:41:44.058 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-03-24T21:41:44.090 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:44 +0000 (0:00:00.038) 0:02:42.394 ******* 2021-03-24T21:41:44.096 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-03-24T21:41:44.098 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:44.122 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:44 +0000 (0:00:00.031) 0:02:42.426 ******* 2021-03-24T21:41:44.129 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-03-24T21:41:44.131 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:44.157 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:44 +0000 (0:00:00.035) 0:02:42.461 ******* 2021-03-24T21:41:44.163 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-03-24T21:41:44.165 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:44.188 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:44 +0000 (0:00:00.031) 0:02:42.493 ******* 2021-03-24T21:41:44.195 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-03-24T21:41:44.197 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:44.223 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:44 +0000 (0:00:00.034) 0:02:42.527 ******* 2021-03-24T21:41:44.229 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-03-24T21:41:44.231 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:44.256 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:44 +0000 (0:00:00.032) 0:02:42.560 ******* 2021-03-24T21:41:44.264 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-03-24T21:41:44.265 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:44.293 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:44 +0000 (0:00:00.037) 0:02:42.598 ******* 2021-03-24T21:41:44.327 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-03-24T21:41:44.328 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:44.350 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:44 +0000 (0:00:00.057) 0:02:42.655 ******* 2021-03-24T21:41:44.495 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-03-24T21:41:44.497 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-24T21:41:44.531 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:44 +0000 (0:00:00.180) 0:02:42.835 ******* 2021-03-24T21:41:44.579 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-03-24T21:41:44.581 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'block_used': 13852087, 'uuid': '765c9454-acf9-4ab1-8c0e-444ef075a2c6', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226243021, 'size_available': 926691414016, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro', 'device': '/dev/sda1', 'inode_used': 150105, 'block_size': 4096, 'inode_available': 60904871}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226243021, "block_size": 4096, "block_total": 240095108, "block_used": 13852087, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60904871, "inode_total": 61054976, "inode_used": 150105, "mount": "/", "options": "rw,relatime,errors=remount-ro", "size_available": 926691414016, "size_total": 983429562368, "uuid": "765c9454-acf9-4ab1-8c0e-444ef075a2c6"}, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:44.608 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:44 +0000 (0:00:00.077) 0:02:42.913 ******* 2021-03-24T21:41:44.638 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-03-24T21:41:44.639 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:44.664 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:44 +0000 (0:00:00.055) 0:02:42.969 ******* 2021-03-24T21:41:44.692 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-24T21:41:44.694 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:44.722 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:44 +0000 (0:00:00.057) 0:02:43.027 ******* 2021-03-24T21:41:44.749 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-24T21:41:44.751 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:44.773 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:44 +0000 (0:00:00.050) 0:02:43.078 ******* 2021-03-24T21:41:44.800 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-03-24T21:41:44.802 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:44.824 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:44 +0000 (0:00:00.051) 0:02:43.129 ******* 2021-03-24T21:41:44.863 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-03-24T21:41:44.865 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:44.889 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:44 +0000 (0:00:00.064) 0:02:43.194 ******* 2021-03-24T21:41:44.928 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-03-24T21:41:44.930 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:44.954 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:44 +0000 (0:00:00.064) 0:02:43.259 ******* 2021-03-24T21:41:44.961 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-03-24T21:41:44.963 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-03-24T21:41:44.989 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:44 +0000 (0:00:00.034) 0:02:43.294 ******* 2021-03-24T21:41:45.517 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-03-24T21:41:45.542 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:45 +0000 (0:00:00.553) 0:02:43.847 ******* 2021-03-24T21:41:45.570 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-03-24T21:41:45.572 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-24T21:41:45.975 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-24T21:41:45.987 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-24T21:41:46.366 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-24T21:41:46.368 INFO:teuthology.task.ansible.out: 2021-03-24T21:41:46.422 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-24T21:41:46.855 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-24T21:41:46.857 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-24T21:41:47.273 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-24T21:41:47.300 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:47 +0000 (0:00:01.757) 0:02:45.605 ******* 2021-03-24T21:41:47.309 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-24T21:41:47.448 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-03-24T21:41:47.449 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-03-24T21:41:47.472 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:47 +0000 (0:00:00.171) 0:02:45.777 ******* 2021-03-24T21:41:47.627 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-03-24T21:41:47.629 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-24T21:41:47.682 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-03-24T21:41:47.683 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-24T21:41:47.708 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:47 +0000 (0:00:00.235) 0:02:46.012 ******* 2021-03-24T21:41:47.759 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-03-24T21:41:47.761 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-24T21:41:47.856 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-03-24T21:41:47.857 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-03-24T21:41:47.883 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:47 +0000 (0:00:00.175) 0:02:46.188 ******* 2021-03-24T21:41:47.893 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-03-24T21:41:47.894 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-24T21:41:48.013 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-24T21:41:48.025 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:48.056 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:48 +0000 (0:00:00.171) 0:02:46.359 ******* 2021-03-24T21:41:48.064 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-24T21:41:48.097 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-03-24T21:41:48.099 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:48.121 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:48 +0000 (0:00:00.066) 0:02:46.425 ******* 2021-03-24T21:41:48.149 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-03-24T21:41:48.150 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:48.172 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:48 +0000 (0:00:00.051) 0:02:46.477 ******* 2021-03-24T21:41:48.180 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-03-24T21:41:48.182 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-24T21:41:48.288 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-24T21:41:48.321 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2021-03-24T21:41:48.345 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:48 +0000 (0:00:00.172) 0:02:46.650 ******* 2021-03-24T21:41:48.385 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-03-24T21:41:48.387 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-03-24T21:41:48.422 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:48 +0000 (0:00:00.076) 0:02:46.726 ******* 2021-03-24T21:41:48.431 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-03-24T21:41:48.433 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} skipping: [smithi142.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:48.469 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:48 +0000 (0:00:00.047) 0:02:46.773 ******* 2021-03-24T21:41:48.499 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-03-24T21:41:48.499 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:48.525 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:48 +0000 (0:00:00.056) 0:02:46.830 ******* 2021-03-24T21:41:48.563 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-03-24T21:41:48.565 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:48.587 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:48 +0000 (0:00:00.062) 0:02:46.892 ******* 2021-03-24T21:41:48.736 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-03-24T21:41:48.738 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2021-03-24T21:41:48.762 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:48 +0000 (0:00:00.174) 0:02:47.067 ******* 2021-03-24T21:41:48.799 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-03-24T21:41:48.801 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-03-24T21:41:48.825 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:48 +0000 (0:00:00.062) 0:02:47.129 ******* 2021-03-24T21:41:48.969 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2021-03-24T21:41:48.970 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => {"changed": true} 2021-03-24T21:41:48.996 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:48 +0000 (0:00:00.171) 0:02:47.301 ******* 2021-03-24T21:41:49.077 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"changed": true} 2021-03-24T21:41:49.102 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:49 +0000 (0:00:00.106) 0:02:47.407 ******* 2021-03-24T21:41:49.508 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-03-24T21:41:49.510 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} changed: [smithi125.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-03-24T21:41:49.534 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:49 +0000 (0:00:00.431) 0:02:47.839 ******* 2021-03-24T21:41:49.558 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:49 +0000 (0:00:00.023) 0:02:47.862 ******* 2021-03-24T21:41:49.577 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-03-24T21:41:49.578 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:49.602 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:49 +0000 (0:00:00.044) 0:02:47.907 ******* 2021-03-24T21:41:49.611 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:49.642 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:49 +0000 (0:00:00.039) 0:02:47.946 ******* 2021-03-24T21:41:49.648 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-03-24T21:41:49.650 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:49.674 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:49 +0000 (0:00:00.032) 0:02:47.979 ******* 2021-03-24T21:41:49.681 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:49.707 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:49 +0000 (0:00:00.033) 0:02:48.012 ******* 2021-03-24T21:41:49.713 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-03-24T21:41:49.715 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:49.739 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:49 +0000 (0:00:00.031) 0:02:48.043 ******* 2021-03-24T21:41:49.746 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:49.772 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:49 +0000 (0:00:00.033) 0:02:48.077 ******* 2021-03-24T21:41:49.779 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-03-24T21:41:49.781 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:49.806 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:49 +0000 (0:00:00.033) 0:02:48.111 ******* 2021-03-24T21:41:49.815 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:49.845 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:49 +0000 (0:00:00.038) 0:02:48.149 ******* 2021-03-24T21:41:49.980 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-03-24T21:41:49.981 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2021-03-24T21:41:50.006 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:50 +0000 (0:00:00.161) 0:02:48.311 ******* 2021-03-24T21:41:50.015 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2021-03-24T21:41:50.044 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:50 +0000 (0:00:00.037) 0:02:48.349 ******* 2021-03-24T21:41:50.157 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2021-03-24T21:41:50.159 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2021-03-24T21:41:50.184 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:50 +0000 (0:00:00.140) 0:02:48.489 ******* 2021-03-24T21:41:50.193 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2021-03-24T21:41:50.220 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:50 +0000 (0:00:00.035) 0:02:48.525 ******* 2021-03-24T21:41:50.567 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-03-24T21:41:50.569 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2021-03-24T21:41:50.604 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:50 +0000 (0:00:00.383) 0:02:48.908 ******* 2021-03-24T21:41:50.613 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2021-03-24T21:41:50.643 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:50 +0000 (0:00:00.036) 0:02:48.945 ******* 2021-03-24T21:41:50.988 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-03-24T21:41:50.990 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2021-03-24T21:41:51.014 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:51 +0000 (0:00:00.374) 0:02:49.319 ******* 2021-03-24T21:41:51.023 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2021-03-24T21:41:51.049 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:51 +0000 (0:00:00.034) 0:02:49.354 ******* 2021-03-24T21:41:51.051 INFO:teuthology.task.ansible.out: 2021-03-24T21:41:51.669 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-03-24T21:41:51.671 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"cache_update_time": 1616621690, "cache_updated": false, "changed": false} 2021-03-24T21:41:51.696 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:51 +0000 (0:00:00.646) 0:02:50.001 ******* 2021-03-24T21:41:51.714 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"cache_update_time": 1616621749, "cache_updated": false, "changed": false} 2021-03-24T21:41:51.740 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:51 +0000 (0:00:00.044) 0:02:50.045 ******* 2021-03-24T21:41:51.970 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-03-24T21:41:51.972 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.124970", "end": "2021-03-24 21:41:51.941869", "rc": 0, "start": "2021-03-24 21:41:51.816899", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.30.0/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.30.0/Amazon/S3.pm"]} 2021-03-24T21:41:51.998 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:51 +0000 (0:00:00.257) 0:02:50.302 ******* 2021-03-24T21:41:52.017 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.121763", "end": "2021-03-24 21:41:51.991714", "rc": 0, "start": "2021-03-24 21:41:51.869951", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.30.0/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.30.0/Amazon/S3.pm"]} 2021-03-24T21:41:52.042 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:52 +0000 (0:00:00.044) 0:02:50.347 ******* 2021-03-24T21:41:52.049 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-03-24T21:41:52.051 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:52.076 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:52 +0000 (0:00:00.033) 0:02:50.381 ******* 2021-03-24T21:41:52.084 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:52.112 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:52 +0000 (0:00:00.036) 0:02:50.417 ******* 2021-03-24T21:41:52.120 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2021-03-24T21:41:52.121 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:52.151 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:52 +0000 (0:00:00.038) 0:02:50.455 ******* 2021-03-24T21:41:52.159 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:52.190 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:52 +0000 (0:00:00.038) 0:02:50.494 ******* 2021-03-24T21:41:52.807 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-03-24T21:41:52.809 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"cache_update_time": 1616621690, "cache_updated": false, "changed": false} 2021-03-24T21:41:52.833 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:52 +0000 (0:00:00.643) 0:02:51.138 ******* 2021-03-24T21:41:52.853 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"cache_update_time": 1616621749, "cache_updated": false, "changed": false} 2021-03-24T21:41:52.854 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:52.879 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:52 +0000 (0:00:00.046) 0:02:51.184 ******* 2021-03-24T21:41:52.902 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:52 +0000 (0:00:00.022) 0:02:51.206 ******* 2021-03-24T21:41:52.930 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:52.956 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:52 +0000 (0:00:00.054) 0:02:51.261 ******* 2021-03-24T21:41:53.215 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-03-24T21:41:53.217 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2021-03-24T21:41:53.240 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:53 +0000 (0:00:00.283) 0:02:51.545 ******* 2021-03-24T21:41:53.268 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-03-24T21:41:53.270 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:53.294 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:53 +0000 (0:00:00.053) 0:02:51.599 ******* 2021-03-24T21:41:53.300 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-03-24T21:41:53.301 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2021-03-24T21:41:53.333 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:53 +0000 (0:00:00.039) 0:02:51.638 ******* 2021-03-24T21:41:53.362 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-03-24T21:41:53.364 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:53.388 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:53 +0000 (0:00:00.054) 0:02:51.692 ******* 2021-03-24T21:41:53.511 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-03-24T21:41:53.512 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2021-03-24 21:38:27 UTC", "ActiveEnterTimestampMonotonic": "13396039", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "sysinit.target systemd-tmpfiles-setup.service system.slice systemd-journald.socket -.mount basic.target network.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2021-03-24 21:38:27 UTC", "AssertTimestampMonotonic": "13295680", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read", "CleanResult": "success", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2021-03-24 21:38:27 UTC", "ConditionTimestampMonotonic": "13295679", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.service shutdown.target", "ControlGroup": 2021-03-24T21:41:53.514 INFO:teuthology.task.ansible.out: "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documentation": "man:ntpd(8)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "819", "ExecMainStartTimestamp": "Wed 2021-03-24 21:38:27 UTC", "ExecMainStartTimestampMonotonic": "13396026", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartEx": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FinalKillSignal": "9", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOReadBytes": "18446744073709551615", "IOReadOperations": "18446744073709551615", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IOWriteBytes": "18446744073709551615", "IOWriteOperations": "18446744073709551615", "IPAccounting": "no", "IPEgressBytes": "[no data]", "IPEgressPackets": "[no data]", "IPIngressBytes": "[no data]", "IPIngressPackets": "[no data]", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2021-03-24 21:38:27 UTC", "InactiveExitTimestampMonotonic": "13296925", "InvocationID": "401f3d003a6640bba3f4c977bb2bbf9e", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "Lim 2021-03-24T21:41:53.516 INFO:teuthology.task.ansible.out:itCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "524288", "LimitNOFILESoft": "1024", "LimitNPROC": "127777", "LimitNPROCSoft": "127777", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127777", "LimitSIGPENDINGSoft": "127777", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "819", "MemoryAccounting": "yes", "MemoryCurrent": "3936256", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMPolicy": "stop", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectHostname": "no", "ProtectKernelLogs": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "ReloadResult": "success", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "Requi 2021-03-24T21:41:53.517 INFO:teuthology.task.ansible.out:resMountsFor": "/var/tmp /tmp", "Restart": "no", "RestartKillSignal": "15", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2021-03-24 21:38:27 UTC", "StateChangeTimestampMonotonic": "13396039", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "38333", "TimeoutAbortUSec": "1min 30s", "TimeoutCleanUSec": "infinity", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogSignal": "6", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-03-24T21:41:53.546 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:53 +0000 (0:00:00.158) 0:02:51.851 ******* 2021-03-24T21:41:53.575 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-24T21:41:53.577 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-03-24T21:41:53.601 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:53 +0000 (0:00:00.054) 0:02:51.905 ******* 2021-03-24T21:41:53.610 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-03-24T21:41:53.612 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2021-03-24 21:38:33 UTC", "ActiveEnterTimestampMonotonic": "13487112", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "network.target -.mount systemd-tmpfiles-setup.service systemd-journald.socket basic.target system.slice sysinit.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2021-03-24 21:38:33 UTC", "AssertTimestampMonotonic": "13362810", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read", "CleanResult": "success", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2021-03-24 21:38:33 UTC", "ConditionTimestampMonotonic": "13362809", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.service shutdown.target", "ControlGroup": 2021-03-24T21:41:53.613 INFO:teuthology.task.ansible.out: "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documentation": "man:ntpd(8)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "830", "ExecMainStartTimestamp": "Wed 2021-03-24 21:38:33 UTC", "ExecMainStartTimestampMonotonic": "13487100", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartEx": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FinalKillSignal": "9", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOReadBytes": "18446744073709551615", "IOReadOperations": "18446744073709551615", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IOWriteBytes": "18446744073709551615", "IOWriteOperations": "18446744073709551615", "IPAccounting": "no", "IPEgressBytes": "[no data]", "IPEgressPackets": "[no data]", "IPIngressBytes": "[no data]", "IPIngressPackets": "[no data]", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2021-03-24 21:38:33 UTC", "InactiveExitTimestampMonotonic": "13363735", "InvocationID": "39470ecdf23e407d9931c6e36136f99b", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "Lim 2021-03-24T21:41:53.615 INFO:teuthology.task.ansible.out:itCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "524288", "LimitNOFILESoft": "1024", "LimitNPROC": "127777", "LimitNPROCSoft": "127777", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127777", "LimitSIGPENDINGSoft": "127777", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "830", "MemoryAccounting": "yes", "MemoryCurrent": "3371008", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMPolicy": "stop", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectHostname": "no", "ProtectKernelLogs": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "ReloadResult": "success", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "Requi 2021-03-24T21:41:53.616 INFO:teuthology.task.ansible.out:resMountsFor": "/tmp /var/tmp", "Restart": "no", "RestartKillSignal": "15", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2021-03-24 21:38:33 UTC", "StateChangeTimestampMonotonic": "13487112", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "38333", "TimeoutAbortUSec": "1min 30s", "TimeoutCleanUSec": "infinity", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogSignal": "6", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-03-24T21:41:53.639 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:53 +0000 (0:00:00.038) 0:02:51.944 ******* 2021-03-24T21:41:53.646 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi125.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2021-03-24T21:41:53.671 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:53 +0000 (0:00:00.032) 0:02:51.976 ******* 2021-03-24T21:41:53.678 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-24T21:41:53.680 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-03-24T21:41:53.705 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:53 +0000 (0:00:00.033) 0:02:52.010 ******* 2021-03-24T21:41:53.712 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-03-24T21:41:53.714 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:53.739 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:53 +0000 (0:00:00.034) 0:02:52.044 ******* 2021-03-24T21:41:53.747 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-24T21:41:53.749 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2021-03-24T21:41:53.774 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:53 +0000 (0:00:00.034) 0:02:52.079 ******* 2021-03-24T21:41:53.808 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-03-24T21:41:53.810 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:53.831 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:53 +0000 (0:00:00.056) 0:02:52.136 ******* 2021-03-24T21:41:54.844 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-03-24T21:41:54.846 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"cache_update_time": 1616621690, "cache_updated": false, "changed": false} 2021-03-24T21:41:54.869 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:54 +0000 (0:00:01.037) 0:02:53.173 ******* 2021-03-24T21:41:54.877 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"cache_update_time": 1616621749, "cache_updated": false, "changed": false} 2021-03-24T21:41:54.902 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:54 +0000 (0:00:00.033) 0:02:53.207 ******* 2021-03-24T21:41:54.909 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-03-24T21:41:54.911 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:54.940 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:54 +0000 (0:00:00.037) 0:02:53.245 ******* 2021-03-24T21:41:54.949 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:54.978 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:54 +0000 (0:00:00.037) 0:02:53.283 ******* 2021-03-24T21:41:55.122 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2021-03-24T21:41:55.124 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2021-03-24T21:41:55.150 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:55 +0000 (0:00:00.172) 0:02:53.455 ******* 2021-03-24T21:41:55.168 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2021-03-24T21:41:55.199 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:55 +0000 (0:00:00.048) 0:02:53.503 ******* 2021-03-24T21:41:55.608 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2021-03-24T21:41:55.610 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2021-03-24T21:41:55.633 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:55 +0000 (0:00:00.434) 0:02:53.938 ******* 2021-03-24T21:41:55.768 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2021-03-24T21:41:55.792 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:55 +0000 (0:00:00.158) 0:02:54.096 ******* 2021-03-24T21:41:57.928 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-03-24T21:41:57.930 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.8/dist-packages (1.2.0)\nRequirement already satisfied: execnet in /usr/local/lib/python3.8/dist-packages (from remoto>=0.0.35) (1.8.0)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.8/dist-packages (from execnet->remoto>=0.0.35) (1.5)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.8/dist-packages (1.2.0)", "Requirement already satisfied: execnet in /usr/local/lib/python3.8/dist-packages (from remoto>=0.0.35) (1.8.0)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.8/dist-packages (from execnet->remoto>=0.0.35) (1.5)"], "version": null, "virtualenv": null} 2021-03-24T21:41:57.960 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:57 +0000 (0:00:02.168) 0:02:56.265 ******* 2021-03-24T21:41:58.081 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.8/dist-packages (1.2.0)\nRequirement already satisfied: execnet in /usr/local/lib/python3.8/dist-packages (from remoto>=0.0.35) (1.8.0)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.8/dist-packages (from execnet->remoto>=0.0.35) (1.5)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.8/dist-packages (1.2.0)", "Requirement already satisfied: execnet in /usr/local/lib/python3.8/dist-packages (from remoto>=0.0.35) (1.8.0)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.8/dist-packages (from execnet->remoto>=0.0.35) (1.5)"], "version": null, "virtualenv": null} 2021-03-24T21:41:58.109 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:58 +0000 (0:00:00.148) 0:02:56.413 ******* 2021-03-24T21:41:58.484 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-03-24T21:41:58.486 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false} 2021-03-24T21:41:58.517 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:58 +0000 (0:00:00.408) 0:02:56.822 ******* 2021-03-24T21:41:58.560 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-03-24T21:41:58.561 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2021-03-24T21:41:58.585 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:58 +0000 (0:00:00.068) 0:02:56.890 ******* 2021-03-24T21:41:58.587 INFO:teuthology.task.ansible.out: 2021-03-24T21:41:58.628 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-03-24T21:41:58.630 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false} 2021-03-24T21:41:58.655 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:58 +0000 (0:00:00.069) 0:02:56.959 ******* 2021-03-24T21:41:58.693 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-03-24T21:41:58.695 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2021-03-24T21:41:58.720 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:58 +0000 (0:00:00.065) 0:02:57.025 ******* 2021-03-24T21:41:58.780 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-03-24T21:41:58.782 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1616621673.14209, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1616621673.1380901, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 105181, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1616621673.1380901, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 71, "uid": 0, "version": "1956422440", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2021-03-24T21:41:58.807 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:58 +0000 (0:00:00.087) 0:02:57.112 ******* 2021-03-24T21:41:58.901 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1616621658.914377, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1616621658.9103768, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 105181, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1616621658.9103768, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 71, "uid": 0, "version": "1956422440", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2021-03-24T21:41:58.927 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:58 +0000 (0:00:00.119) 0:02:57.232 ******* 2021-03-24T21:41:59.260 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-03-24T21:41:59.262 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "msg": ""} ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2021-03-24T21:41:59.284 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:59 +0000 (0:00:00.357) 0:02:57.589 ******* 2021-03-24T21:41:59.306 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:59 +0000 (0:00:00.022) 0:02:57.611 ******* 2021-03-24T21:41:59.324 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2021-03-24T21:41:59.326 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-03-24T21:41:59.349 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:59 +0000 (0:00:00.042) 0:02:57.654 ******* 2021-03-24T21:41:59.356 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-03-24T21:41:59.382 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:59 +0000 (0:00:00.032) 0:02:57.686 ******* 2021-03-24T21:41:59.388 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2021-03-24T21:41:59.390 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:59.413 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:59 +0000 (0:00:00.031) 0:02:57.718 ******* 2021-03-24T21:41:59.421 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:59.446 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:59 +0000 (0:00:00.032) 0:02:57.751 ******* 2021-03-24T21:41:59.593 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** changed: [smithi125.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-03-24T21:41:59.617 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:59 +0000 (0:00:00.171) 0:02:57.922 ******* 2021-03-24T21:41:59.626 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-03-24T21:41:59.651 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:59 +0000 (0:00:00.033) 0:02:57.956 ******* 2021-03-24T21:41:59.784 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-03-24T21:41:59.786 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-03-24T21:41:59.817 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:59 +0000 (0:00:00.166) 0:02:58.122 ******* 2021-03-24T21:41:59.825 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-03-24T21:41:59.851 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:59 +0000 (0:00:00.033) 0:02:58.155 ******* 2021-03-24T21:41:59.857 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-03-24T21:41:59.859 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:59.884 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:59 +0000 (0:00:00.033) 0:02:58.189 ******* 2021-03-24T21:41:59.891 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:41:59.918 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:59 +0000 (0:00:00.033) 0:02:58.222 ******* 2021-03-24T21:41:59.925 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-03-24T21:41:59.926 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2021-03-24T21:41:59.927 INFO:teuthology.task.ansible.out: skipping: [smithi125.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} skipping: [smithi125.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-03-24T21:41:59.929 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2021-03-24T21:41:59.940 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2021-03-24T21:41:59.942 INFO:teuthology.task.ansible.out: 2021-03-24T21:41:59.969 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:41:59 +0000 (0:00:00.051) 0:02:58.274 ******* 2021-03-24T21:41:59.971 INFO:teuthology.task.ansible.out: 2021-03-24T21:41:59.978 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2021-03-24T21:41:59.980 INFO:teuthology.task.ansible.out: 2021-03-24T21:41:59.980 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2021-03-24T21:41:59.981 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-03-24T21:41:59.982 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-03-24T21:42:00.010 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:42:00 +0000 (0:00:00.040) 0:02:58.314 ******* 2021-03-24T21:42:00.017 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-03-24T21:42:00.018 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-24T21:42:00.020 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi125.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-24T21:42:00.020 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-24T21:42:00.032 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-24T21:42:00.060 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:42:00 +0000 (0:00:00.049) 0:02:58.364 ******* 2021-03-24T21:42:00.068 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-24T21:42:00.068 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-24T21:42:00.070 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-24T21:42:00.071 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-24T21:42:00.097 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:42:00 +0000 (0:00:00.036) 0:02:58.401 ******* 2021-03-24T21:42:00.104 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-24T21:42:00.106 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:42:00.135 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:42:00 +0000 (0:00:00.038) 0:02:58.440 ******* 2021-03-24T21:42:00.143 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:42:00.173 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:42:00 +0000 (0:00:00.037) 0:02:58.478 ******* 2021-03-24T21:42:00.180 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-03-24T21:42:00.182 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:42:00.205 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:42:00 +0000 (0:00:00.031) 0:02:58.510 ******* 2021-03-24T21:42:00.212 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:42:00.237 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:42:00 +0000 (0:00:00.031) 0:02:58.541 ******* 2021-03-24T21:42:00.244 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-03-24T21:42:00.246 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:42:00.275 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:42:00 +0000 (0:00:00.037) 0:02:58.579 ******* 2021-03-24T21:42:00.282 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:42:00.311 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:42:00 +0000 (0:00:00.036) 0:02:58.615 ******* 2021-03-24T21:42:00.318 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:42:00.344 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:42:00 +0000 (0:00:00.033) 0:02:58.649 ******* 2021-03-24T21:42:00.352 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:42:00.377 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:42:00 +0000 (0:00:00.033) 0:02:58.682 ******* 2021-03-24T21:42:00.385 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-03-24T21:42:00.387 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:42:00.415 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:42:00 +0000 (0:00:00.037) 0:02:58.720 ******* 2021-03-24T21:42:00.422 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:42:00.450 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:42:00 +0000 (0:00:00.034) 0:02:58.755 ******* 2021-03-24T21:42:00.457 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-03-24T21:42:00.458 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:42:00.485 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:42:00 +0000 (0:00:00.034) 0:02:58.789 ******* 2021-03-24T21:42:00.492 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:42:00.520 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:42:00 +0000 (0:00:00.035) 0:02:58.825 ******* 2021-03-24T21:42:00.527 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-03-24T21:42:00.527 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:42:00.554 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:42:00 +0000 (0:00:00.034) 0:02:58.859 ******* 2021-03-24T21:42:00.561 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:42:00.589 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:42:00 +0000 (0:00:00.034) 0:02:58.894 ******* 2021-03-24T21:42:00.596 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-03-24T21:42:00.598 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:42:00.622 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:42:00 +0000 (0:00:00.033) 0:02:58.927 ******* 2021-03-24T21:42:00.631 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:42:00.662 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:42:00 +0000 (0:00:00.039) 0:02:58.967 ******* 2021-03-24T21:42:00.835 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-03-24T21:42:00.837 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.010311", "end": "2021-03-24 21:42:00.800015", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-03-24 21:42:00.789704", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-24T21:42:00.868 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.010289", "end": "2021-03-24 21:42:00.834539", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-03-24 21:42:00.824250", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-24T21:42:00.992 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.006080", "end": "2021-03-24 21:42:00.960808", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-03-24 21:42:00.954728", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-24T21:42:01.024 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.006079", "end": "2021-03-24 21:42:00.998640", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-03-24 21:42:00.992561", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-24T21:42:01.149 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005476", "end": "2021-03-24 21:42:01.117459", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-03-24 21:42:01.111983", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2021-03-24T21:42:01.182 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005452", "end": "2021-03-24 21:42:01.154730", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-03-24 21:42:01.149278", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2021-03-24T21:42:01.349 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.032121", "end": "2021-03-24 21:42:01.310376", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-03-24 21:42:01.278255", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-24T21:42:01.375 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:42:01 +0000 (0:00:00.712) 0:02:59.679 ******* 2021-03-24T21:42:01.382 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.027164", "end": "2021-03-24 21:42:01.331329", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-03-24 21:42:01.304165", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-24T21:42:01.409 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:42:01 +0000 (0:00:00.033) 0:02:59.713 ******* 2021-03-24T21:42:01.415 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-03-24T21:42:01.441 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:42:01 +0000 (0:00:00.031) 0:02:59.745 ******* 2021-03-24T21:42:01.472 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:42:01 +0000 (0:00:00.031) 0:02:59.777 ******* 2021-03-24T21:42:01.619 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-03-24T21:42:01.621 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2021-03-24T21:42:01.645 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2021-03-24T21:42:01.727 INFO:teuthology.task.ansible.out: PLAY [testnodes,senta,vossi,jenkins_slaves] ************************************ 2021-03-24T21:42:01.745 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:42:01 +0000 (0:00:00.273) 0:03:00.050 ******* 2021-03-24T21:42:01.771 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:42:01 +0000 (0:00:00.025) 0:03:00.076 ******* 2021-03-24T21:42:02.759 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi125.front.sepia.ceph.com] 2021-03-24T21:42:02.815 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:42:02 +0000 (0:00:01.043) 0:03:01.120 ******* 2021-03-24T21:42:02.828 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] 2021-03-24T21:42:02.858 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-24T21:42:02.861 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "apt"}, "changed": false} 2021-03-24T21:42:02.884 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:42:02 +0000 (0:00:00.068) 0:03:01.189 ******* 2021-03-24T21:42:02.906 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:42:02 +0000 (0:00:00.021) 0:03:01.211 ******* 2021-03-24T21:42:02.914 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-24T21:42:02.916 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:42:02.939 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:42:02 +0000 (0:00:00.033) 0:03:01.244 ******* 2021-03-24T21:42:02.946 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-24T21:42:02.948 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "apt"}, "changed": false} 2021-03-24T21:42:02.973 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:42:02 +0000 (0:00:00.033) 0:03:01.277 ******* 2021-03-24T21:42:02.979 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-03-24T21:42:02.981 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml) => {"ansible_facts": {"container_packages": ["docker.io"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"} 2021-03-24T21:42:03.005 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:42:03 +0000 (0:00:00.032) 0:03:01.310 ******* 2021-03-24T21:42:03.007 INFO:teuthology.task.ansible.out: 2021-03-24T21:42:03.014 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-24T21:42:03.016 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:42:03.040 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:42:03 +0000 (0:00:00.035) 0:03:01.345 ******* 2021-03-24T21:42:03.078 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-03-24T21:42:03.080 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml) => {"ansible_facts": {"container_packages": ["docker.io"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"} 2021-03-24T21:42:03.103 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:42:03 +0000 (0:00:00.062) 0:03:01.407 ******* 2021-03-24T21:42:03.744 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-03-24T21:42:03.745 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"cache_update_time": 1616621690, "cache_updated": false, "changed": false} 2021-03-24T21:42:03.769 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:42:03 +0000 (0:00:00.666) 0:03:02.074 ******* 2021-03-24T21:42:03.797 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-24T21:42:03.799 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:42:03.822 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:42:03 +0000 (0:00:00.052) 0:03:02.126 ******* 2021-03-24T21:42:03.849 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-03-24T21:42:03.851 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"cache_update_time": 1616621749, "cache_updated": false, "changed": false} TASK [container-host : set_fact] *********************************************** ok: [smithi125.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/docker/daemon.json"}, "changed": false} 2021-03-24T21:42:03.875 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:42:03 +0000 (0:00:00.053) 0:03:02.180 ******* 2021-03-24T21:42:03.905 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:42:03 +0000 (0:00:00.029) 0:03:02.210 ******* 2021-03-24T21:42:03.935 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-24T21:42:03.937 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T21:42:03.960 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:42:03 +0000 (0:00:00.054) 0:03:02.264 ******* 2021-03-24T21:42:03.999 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** ok: [smithi142.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/docker/daemon.json"}, "changed": false} 2021-03-24T21:42:04.022 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:42:04 +0000 (0:00:00.062) 0:03:02.327 ******* 2021-03-24T21:42:04.091 INFO:teuthology.task.ansible.out: 2021-03-24T21:42:04.093 INFO:teuthology.task.ansible.out:TASK [container-host : Create container_mirror_cert_paths] ********************* changed: [smithi125.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-03-24T21:42:04.241 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-03-24T21:42:04.242 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-03-24T21:42:04.286 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:42:04 +0000 (0:00:00.263) 0:03:02.591 ******* 2021-03-24T21:42:04.399 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-03-24T21:42:04.429 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:42:04 +0000 (0:00:00.142) 0:03:02.733 ******* 2021-03-24T21:42:04.732 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-03-24T21:42:04.734 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1616622124.345006-27756-272531949302135/source", "state": "file", "uid": 0} 2021-03-24T21:42:04.891 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1616622124.4902039-27792-186044606773408/source", "state": "file", "uid": 0} 2021-03-24T21:42:05.145 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1616622124.7555237-27756-104213763235771/source", "state": "file", "uid": 0} 2021-03-24T21:42:05.172 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:42:05 +0000 (0:00:00.742) 0:03:03.477 ******* 2021-03-24T21:42:05.308 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1616622124.9234838-27792-177956431726775/source", "state": "file", "uid": 0} 2021-03-24T21:42:05.334 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:42:05 +0000 (0:00:00.161) 0:03:03.639 ******* 2021-03-24T21:42:09.883 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-03-24T21:42:09.885 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => {"changed": true, "cmd": ["/usr/bin/pip3", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": " Running command git clone -q https://github.com/sebastian-philipp/registries-conf-ctl /tmp/pip-req-build-qx7dctvi\n", "stderr_lines": [" Running command git clone -q https://github.com/sebastian-philipp/registries-conf-ctl /tmp/pip-req-build-qx7dctvi"], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-req-build-qx7dctvi\nCollecting docopt\n Downloading docopt-0.6.2.tar.gz (25 kB)\nCollecting toml\n Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB)\nBuilding wheels for collected packages: registries-conf-ctl, docopt\n Building wheel for registries-conf-ctl (setup.py): started\n Building wheel for registries-conf-ctl (setup.py): finished with status 'done'\n Created wheel for registries-conf-ctl: filename=registries_conf_ctl-1.0-py3-none-any.whl size=6035 sha256=22b2e7f8fb502ab73a6fd363d01fea3b8814ca7fe7691dff66174f97451bc091\n Stored in directory: /tmp/pip-ephem-wheel-cache-kdhw49k9/wheels/66/49/be/0ca4a7322f6cd17532451655790e478b4f82776818b44d6d42\n Building wheel for docopt (setup.py): started\n Building wheel for docopt (setup.py): finished with status 'done'\n Created wheel for docopt: filename=docopt-0.6.2-py2.py3-none-any.whl size=13704 sha256=c2c233bb8c157d300dff13f60ab72a9505e749f8bd107b0f65c72b5ea9fa2af2\n Stored in directory: /root/.cache/pip/wheels/56/ea/58/ead137b087d9e326852a851351d1debf4ada529b6ac0ec4e8c\nSuccessfully built registries-conf-ctl docopt\nInstalling collected packages: docopt, toml, registries-conf-ctl\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2\n", "stdout_lines": ["Collecting git+https://github. 2021-03-24T21:42:09.887 INFO:teuthology.task.ansible.out:com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-req-build-qx7dctvi", "Collecting docopt", " Downloading docopt-0.6.2.tar.gz (25 kB)", "Collecting toml", " Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB)", "Building wheels for collected packages: registries-conf-ctl, docopt", " Building wheel for registries-conf-ctl (setup.py): started", " Building wheel for registries-conf-ctl (setup.py): finished with status 'done'", " Created wheel for registries-conf-ctl: filename=registries_conf_ctl-1.0-py3-none-any.whl size=6035 sha256=22b2e7f8fb502ab73a6fd363d01fea3b8814ca7fe7691dff66174f97451bc091", " Stored in directory: /tmp/pip-ephem-wheel-cache-kdhw49k9/wheels/66/49/be/0ca4a7322f6cd17532451655790e478b4f82776818b44d6d42", " Building wheel for docopt (setup.py): started", " Building wheel for docopt (setup.py): finished with status 'done'", " Created wheel for docopt: filename=docopt-0.6.2-py2.py3-none-any.whl size=13704 sha256=c2c233bb8c157d300dff13f60ab72a9505e749f8bd107b0f65c72b5ea9fa2af2", " Stored in directory: /root/.cache/pip/wheels/56/ea/58/ead137b087d9e326852a851351d1debf4ada529b6ac0ec4e8c", "Successfully built registries-conf-ctl docopt", "Installing collected packages: docopt, toml, registries-conf-ctl", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2"], "version": null, "virtualenv": null} 2021-03-24T21:42:09.911 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:42:09 +0000 (0:00:04.576) 0:03:08.216 ******* 2021-03-24T21:42:10.079 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "cmd": ["/usr/bin/pip3", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": " Running command git clone -q https://github.com/sebastian-philipp/registries-conf-ctl /tmp/pip-req-build-4o4k_f_w\n", "stderr_lines": [" Running command git clone -q https://github.com/sebastian-philipp/registries-conf-ctl /tmp/pip-req-build-4o4k_f_w"], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-req-build-4o4k_f_w\nCollecting docopt\n Downloading docopt-0.6.2.tar.gz (25 kB)\nCollecting toml\n Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB)\nBuilding wheels for collected packages: registries-conf-ctl, docopt\n Building wheel for registries-conf-ctl (setup.py): started\n Building wheel for registries-conf-ctl (setup.py): finished with status 'done'\n Created wheel for registries-conf-ctl: filename=registries_conf_ctl-1.0-py3-none-any.whl size=6035 sha256=22b2e7f8fb502ab73a6fd363d01fea3b8814ca7fe7691dff66174f97451bc091\n Stored in directory: /tmp/pip-ephem-wheel-cache-bgojlyd8/wheels/66/49/be/0ca4a7322f6cd17532451655790e478b4f82776818b44d6d42\n Building wheel for docopt (setup.py): started\n Building wheel for docopt (setup.py): finished with status 'done'\n Created wheel for docopt: filename=docopt-0.6.2-py2.py3-none-any.whl size=13704 sha256=c2c233bb8c157d300dff13f60ab72a9505e749f8bd107b0f65c72b5ea9fa2af2\n Stored in directory: /root/.cache/pip/wheels/56/ea/58/ead137b087d9e326852a851351d1debf4ada529b6ac0ec4e8c\nSuccessfully built registries-conf-ctl docopt\nInstalling collected packages: docopt, toml, registries-conf-ctl\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2\n", "stdout_lines": ["Collecting git+https://github. 2021-03-24T21:42:10.080 INFO:teuthology.task.ansible.out:com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-req-build-4o4k_f_w", "Collecting docopt", " Downloading docopt-0.6.2.tar.gz (25 kB)", "Collecting toml", " Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB)", "Building wheels for collected packages: registries-conf-ctl, docopt", " Building wheel for registries-conf-ctl (setup.py): started", " Building wheel for registries-conf-ctl (setup.py): finished with status 'done'", " Created wheel for registries-conf-ctl: filename=registries_conf_ctl-1.0-py3-none-any.whl size=6035 sha256=22b2e7f8fb502ab73a6fd363d01fea3b8814ca7fe7691dff66174f97451bc091", " Stored in directory: /tmp/pip-ephem-wheel-cache-bgojlyd8/wheels/66/49/be/0ca4a7322f6cd17532451655790e478b4f82776818b44d6d42", " Building wheel for docopt (setup.py): started", " Building wheel for docopt (setup.py): finished with status 'done'", " Created wheel for docopt: filename=docopt-0.6.2-py2.py3-none-any.whl size=13704 sha256=c2c233bb8c157d300dff13f60ab72a9505e749f8bd107b0f65c72b5ea9fa2af2", " Stored in directory: /root/.cache/pip/wheels/56/ea/58/ead137b087d9e326852a851351d1debf4ada529b6ac0ec4e8c", "Successfully built registries-conf-ctl docopt", "Installing collected packages: docopt, toml, registries-conf-ctl", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2"], "version": null, "virtualenv": null} 2021-03-24T21:42:10.106 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:42:10 +0000 (0:00:00.194) 0:03:08.410 ******* 2021-03-24T21:42:10.113 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-03-24T21:42:10.116 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-03-24T21:42:10.148 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:42:10 +0000 (0:00:00.042) 0:03:08.453 ******* 2021-03-24T21:42:10.282 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-03-24T21:42:10.310 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:42:10 +0000 (0:00:00.162) 0:03:08.615 ******* 2021-03-24T21:42:10.592 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/docker/daemon.json if necessary] ************ 2021-03-24T21:42:10.593 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => {"changed": true, "checksum": "bf21a9e8fbc5a3846fb05b4fa0859e0917b2202f", "dest": "/etc/docker/daemon.json", "gid": 0, "group": "root", "md5sum": "99914b932bd37a50b983c5e7c90ae93b", "mode": "0644", "owner": "root", "size": 2, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1616622130.2066658-28853-190720587077470/source", "state": "file", "uid": 0} 2021-03-24T21:42:10.618 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:42:10 +0000 (0:00:00.307) 0:03:08.922 ******* 2021-03-24T21:42:10.751 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "checksum": "bf21a9e8fbc5a3846fb05b4fa0859e0917b2202f", "dest": "/etc/docker/daemon.json", "gid": 0, "group": "root", "md5sum": "99914b932bd37a50b983c5e7c90ae93b", "mode": "0644", "owner": "root", "size": 2, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1616622130.3683128-28882-246644594884863/source", "state": "file", "uid": 0} 2021-03-24T21:42:10.776 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:42:10 +0000 (0:00:00.158) 0:03:09.081 ******* 2021-03-24T21:42:10.837 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-03-24T21:42:10.838 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.038645", "end": "2021-03-24 21:42:10.814852", "rc": 0, "start": "2021-03-24 21:42:10.776207", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-24T21:42:10.863 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:42:10 +0000 (0:00:00.086) 0:03:09.168 ******* 2021-03-24T21:42:10.986 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.035451", "end": "2021-03-24 21:42:10.966297", "rc": 0, "start": "2021-03-24 21:42:10.930846", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-24T21:42:11.011 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 21:42:11 +0000 (0:00:00.147) 0:03:09.316 ******* 2021-03-24T21:42:14.880 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-03-24T21:42:14.881 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => {"changed": true, "name": "docker", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "sysinit.target system.slice docker.socket containerd.service firewalld.service basic.target network-online.target systemd-journald.socket", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read", "CleanResult": "success", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ConsistsOf": "docker.socket", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "yes", "DelegateControllers": "cpu cpuacct cpuset io blkio memory devices pids bpf-fir 2021-03-24T21:42:14.883 INFO:teuthology.task.ansible.out:ewall bpf-devices", "Description": "Docker Application Container Engine", "DevicePolicy": "auto", "Documentation": "https://docs.docker.com", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecReloadEx": "{ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartEx": "{ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FinalKillSignal": "9", "FragmentPath": "/lib/systemd/system/docker.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOReadBytes": "18446744073709551615", "IOReadOperations": "18446744073709551615", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IOWriteBytes": "18446744073709551615", "IOWriteOperations": "18446744073709551615", "IPAccounting": "no", "IPEgressBytes": "[no data]", "IPEgressPackets": "[no data]", "IPIngressBytes": "[no data]", "IPIngressPackets": "[no data]", "Id": "docker.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "process", "KillSignal": "15", 2021-03-24T21:42:14.885 INFO:teuthology.task.ansible.out:"LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "infinity", "LimitNOFILESoft": "infinity", "LimitNPROC": "infinity", "LimitNPROCSoft": "infinity", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127777", "LimitSIGPENDINGSoft": "127777", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "docker.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "main", "OOMPolicy": "continue", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectHostname": "no", "ProtectKernelLogs": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "ReloadResult": "success", 2021-03-24T21:42:14.886 INFO:teuthology.task.ansible.out: "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice docker.socket", "Restart": "always", "RestartKillSignal": "15", "RestartUSec": "2s", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "3", "StartLimitIntervalUSec": "1min", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2021-03-24 21:41:02 UTC", "StateChangeTimestampMonotonic": "167853661", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "infinity", "TimeoutAbortUSec": "infinity", "TimeoutCleanUSec": "infinity", "TimeoutStartUSec": "infinity", "TimeoutStopUSec": "infinity", "TimerSlackNSec": "50000", "Transient": "no", "TriggeredBy": "docker.socket", "Type": "notify", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "containerd.service network-online.target", "WatchdogSignal": "6", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-03-24T21:42:15.211 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "name": "docker", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "system.slice docker.socket sysinit.target systemd-journald.socket containerd.service network-online.target basic.target firewalld.service", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read", "CleanResult": "success", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ConsistsOf": "docker.socket", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "yes", "DelegateControllers": "cpu cpuacct cpuset io blkio memory devices pids bpf-fir 2021-03-24T21:42:15.213 INFO:teuthology.task.ansible.out:ewall bpf-devices", "Description": "Docker Application Container Engine", "DevicePolicy": "auto", "Documentation": "https://docs.docker.com", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecReloadEx": "{ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartEx": "{ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FinalKillSignal": "9", "FragmentPath": "/lib/systemd/system/docker.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOReadBytes": "18446744073709551615", "IOReadOperations": "18446744073709551615", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IOWriteBytes": "18446744073709551615", "IOWriteOperations": "18446744073709551615", "IPAccounting": "no", "IPEgressBytes": "[no data]", "IPEgressPackets": "[no data]", "IPIngressBytes": "[no data]", "IPIngressPackets": "[no data]", "Id": "docker.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "process", "KillSignal": "15", 2021-03-24T21:42:15.215 INFO:teuthology.task.ansible.out:"LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "infinity", "LimitNOFILESoft": "infinity", "LimitNPROC": "infinity", "LimitNPROCSoft": "infinity", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127777", "LimitSIGPENDINGSoft": "127777", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "docker.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "main", "OOMPolicy": "continue", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectHostname": "no", "ProtectKernelLogs": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "ReloadResult": "success", 2021-03-24T21:42:15.216 INFO:teuthology.task.ansible.out: "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "docker.socket system.slice sysinit.target", "Restart": "always", "RestartKillSignal": "15", "RestartUSec": "2s", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "3", "StartLimitIntervalUSec": "1min", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2021-03-24 21:41:02 UTC", "StateChangeTimestampMonotonic": "161796284", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "infinity", "TimeoutAbortUSec": "infinity", "TimeoutCleanUSec": "infinity", "TimeoutStartUSec": "infinity", "TimeoutStopUSec": "infinity", "TimerSlackNSec": "50000", "Transient": "no", "TriggeredBy": "docker.socket", "Type": "notify", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-online.target containerd.service", "WatchdogSignal": "6", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-03-24T21:42:15.282 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2021-03-24T21:42:15.283 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2021-03-24T21:42:15.285 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched PLAY RECAP ********************************************************************* smithi125.front.sepia.ceph.com : ok=129 changed=36 unreachable=0 failed=0 skipped=182 rescued=0 ignored=0 smithi142.front.sepia.ceph.com : ok=129 changed=36 unreachable=0 failed=0 skipped=182 rescued=0 ignored=0 Wednesday 24 March 2021 21:42:15 +0000 (0:00:04.273) 0:03:13.589 ******* =============================================================================== 2021-03-24T21:42:15.285 INFO:teuthology.task.ansible.out:users : Update authorized_keys using the keys repo --------------------- 25.80s Create all admin users with sudo access. ------------------------------- 24.91s testnode : Ensure packages are not present. ---------------------------- 11.29s testnode : Install packages -------------------------------------------- 11.17s Remove revoked users ---------------------------------------------------- 7.80s testnode : Zap all non-root disks --------------------------------------- 7.70s container-host : Install registries-conf-ctl ---------------------------- 4.58s container-host : Restart docker service --------------------------------- 4.27s 2021-03-24T21:42:15.286 INFO:teuthology.task.ansible.out:common : Update apt cache ----------------------------------------------- 3.76s users : Update authorized_keys for each user with literal keys ---------- 3.20s Gathering Facts --------------------------------------------------------- 2.44s testnode : Install apt keys --------------------------------------------- 2.32s testnode : Install packages via pip ------------------------------------- 2.17s testnode : Update apt cache. -------------------------------------------- 2.08s testnode : Create logical volume(s) ------------------------------------- 1.76s ansible-managed : Create the cephlab_sudo sudoers.d file. --------------- 1.36s common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/. --- 1.27s com 2021-03-24T21:42:15.287 INFO:teuthology.task.ansible.out:mon : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/. --- 1.20s common : Mask sleep units ----------------------------------------------- 1.14s Gathering Facts --------------------------------------------------------- 1.04s 2021-03-24T21:42:15.649 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi125.front.sepia.ceph.com'), Remote(name='ubuntu@smithi142.front.sepia.ceph.com')] 2021-03-24T21:42:15.651 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-24T21:42:15.653 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi125.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-24T21:42:15.714 DEBUG:teuthology.orchestra.run.smithi125:> true 2021-03-24T21:42:15.832 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-24T21:42:15.833 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-24T21:42:15.885 DEBUG:teuthology.orchestra.run.smithi142:> true 2021-03-24T21:42:16.000 INFO:teuthology.run_tasks:Running task clock... 2021-03-24T21:42:16.011 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2021-03-24T21:42:16.012 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-03-24T21:42:16.014 DEBUG:teuthology.orchestra.run.smithi125:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-03-24T21:42:16.047 INFO:teuthology.orchestra.run.smithi125.stdout:24 Mar 21:42:16 ntpd[7344]: ntpd 4.2.8p12@1.3728-o (1): Starting 2021-03-24T21:42:16.048 INFO:teuthology.orchestra.run.smithi125.stdout:24 Mar 21:42:16 ntpd[7344]: Command line: ntpd -gq 2021-03-24T21:42:16.048 INFO:teuthology.orchestra.run.smithi125.stdout:24 Mar 21:42:16 ntpd[7344]: proto: precision = 0.121 usec (-23) 2021-03-24T21:42:16.049 INFO:teuthology.orchestra.run.smithi125.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-03-24T21:42:16.049 INFO:teuthology.orchestra.run.smithi125.stdout:24 Mar 21:42:16 ntpd[7344]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-03-24T21:42:16.050 INFO:teuthology.orchestra.run.smithi125.stdout:24 Mar 21:42:16 ntpd[7344]: restrict ::: KOD does nothing without LIMITED. 2021-03-24T21:42:16.050 INFO:teuthology.orchestra.run.smithi125.stderr:restrict ::: KOD does nothing without LIMITED. 2021-03-24T21:42:16.051 INFO:teuthology.orchestra.run.smithi125.stdout:24 Mar 21:42:16 ntpd[7344]: Listen and drop on 0 v6wildcard [::]:123 2021-03-24T21:42:16.051 INFO:teuthology.orchestra.run.smithi125.stdout:24 Mar 21:42:16 ntpd[7344]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2021-03-24T21:42:16.052 INFO:teuthology.orchestra.run.smithi125.stdout:24 Mar 21:42:16 ntpd[7344]: Listen normally on 2 lo 127.0.0.1:123 2021-03-24T21:42:16.052 INFO:teuthology.orchestra.run.smithi125.stdout:24 Mar 21:42:16 ntpd[7344]: Listen normally on 3 enp3s0f1 172.21.15.125:123 2021-03-24T21:42:16.052 INFO:teuthology.orchestra.run.smithi125.stdout:24 Mar 21:42:16 ntpd[7344]: Listen normally on 4 lo [::1]:123 2021-03-24T21:42:16.052 INFO:teuthology.orchestra.run.smithi125.stdout:24 Mar 21:42:16 ntpd[7344]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d739%5]:123 2021-03-24T21:42:16.053 INFO:teuthology.orchestra.run.smithi125.stdout:24 Mar 21:42:16 ntpd[7344]: Listening on routing socket on fd #22 for interface updates 2021-03-24T21:42:17.049 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:17 ntpd[7344]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T21:42:17.049 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:17 ntpd[7344]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T21:42:17.050 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:17 ntpd[7344]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T21:42:17.050 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:17 ntpd[7344]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T21:42:17.050 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:17 ntpd[7344]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T21:42:17.050 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:17 ntpd[7344]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T21:42:17.051 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:17 ntpd[7344]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-24T21:42:17.051 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:17 ntpd[7344]: can't open /var/log/ntpstats/peerstats.20210324: Permission denied 2021-03-24T21:42:17.051 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:17 ntpd[7344]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T21:42:17.051 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:17 ntpd[7344]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T21:42:17.052 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:17 ntpd[7344]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-24T21:42:17.052 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:17 ntpd[7344]: can't open /var/log/ntpstats/peerstats.20210324: Permission denied 2021-03-24T21:42:18.049 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:18 ntpd[7344]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T21:42:18.049 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:18 ntpd[7344]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T21:42:18.050 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:18 ntpd[7344]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T21:42:18.050 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:18 ntpd[7344]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T21:42:18.050 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:18 ntpd[7344]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T21:42:18.050 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:18 ntpd[7344]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T21:42:18.051 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:18 ntpd[7344]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-24T21:42:18.051 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:18 ntpd[7344]: can't open /var/log/ntpstats/peerstats.20210324: Permission denied 2021-03-24T21:42:18.051 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:18 ntpd[7344]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T21:42:18.051 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:18 ntpd[7344]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T21:42:18.052 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:18 ntpd[7344]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-24T21:42:18.052 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:18 ntpd[7344]: can't open /var/log/ntpstats/peerstats.20210324: Permission denied 2021-03-24T21:42:19.049 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:19 ntpd[7344]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T21:42:19.049 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:19 ntpd[7344]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T21:42:19.049 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:19 ntpd[7344]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T21:42:19.050 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:19 ntpd[7344]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T21:42:19.050 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:19 ntpd[7344]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T21:42:19.050 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:19 ntpd[7344]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T21:42:19.050 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:19 ntpd[7344]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-24T21:42:19.051 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:19 ntpd[7344]: can't open /var/log/ntpstats/peerstats.20210324: Permission denied 2021-03-24T21:42:19.051 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:19 ntpd[7344]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T21:42:19.051 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:19 ntpd[7344]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T21:42:19.051 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:19 ntpd[7344]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-24T21:42:19.051 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:19 ntpd[7344]: can't open /var/log/ntpstats/peerstats.20210324: Permission denied 2021-03-24T21:42:20.051 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:20 ntpd[7344]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T21:42:20.051 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:20 ntpd[7344]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T21:42:20.051 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:20 ntpd[7344]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T21:42:20.052 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:20 ntpd[7344]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T21:42:20.052 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:20 ntpd[7344]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T21:42:20.052 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:20 ntpd[7344]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T21:42:20.052 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:20 ntpd[7344]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-24T21:42:20.053 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:20 ntpd[7344]: can't open /var/log/ntpstats/peerstats.20210324: Permission denied 2021-03-24T21:42:20.053 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:20 ntpd[7344]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T21:42:20.053 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:20 ntpd[7344]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T21:42:20.053 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:20 ntpd[7344]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-24T21:42:20.053 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:20 ntpd[7344]: can't open /var/log/ntpstats/peerstats.20210324: Permission denied 2021-03-24T21:42:21.049 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:21 ntpd[7344]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T21:42:21.049 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:21 ntpd[7344]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T21:42:21.049 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:21 ntpd[7344]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T21:42:21.050 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:21 ntpd[7344]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T21:42:21.050 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:21 ntpd[7344]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T21:42:21.050 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:21 ntpd[7344]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T21:42:21.050 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:21 ntpd[7344]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-24T21:42:21.051 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:21 ntpd[7344]: can't open /var/log/ntpstats/peerstats.20210324: Permission denied 2021-03-24T21:42:21.051 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:21 ntpd[7344]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T21:42:21.051 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:21 ntpd[7344]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T21:42:21.051 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:21 ntpd[7344]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-24T21:42:21.051 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:21 ntpd[7344]: can't open /var/log/ntpstats/peerstats.20210324: Permission denied 2021-03-24T21:42:22.049 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:22 ntpd[7344]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T21:42:22.049 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:22 ntpd[7344]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T21:42:22.050 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:22 ntpd[7344]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T21:42:22.050 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:22 ntpd[7344]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T21:42:22.050 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:22 ntpd[7344]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T21:42:22.050 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:22 ntpd[7344]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T21:42:22.050 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:22 ntpd[7344]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-24T21:42:22.051 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:22 ntpd[7344]: can't open /var/log/ntpstats/peerstats.20210324: Permission denied 2021-03-24T21:42:22.051 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:22 ntpd[7344]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T21:42:22.051 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:22 ntpd[7344]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T21:42:22.051 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:22 ntpd[7344]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-24T21:42:22.052 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:22 ntpd[7344]: can't open /var/log/ntpstats/peerstats.20210324: Permission denied 2021-03-24T21:42:23.048 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:23 ntpd[7344]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T21:42:23.049 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:23 ntpd[7344]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T21:42:23.049 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:23 ntpd[7344]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T21:42:23.049 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:23 ntpd[7344]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T21:42:23.050 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:23 ntpd[7344]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T21:42:23.050 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:23 ntpd[7344]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T21:42:23.050 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:23 ntpd[7344]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-24T21:42:23.050 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:23 ntpd[7344]: can't open /var/log/ntpstats/peerstats.20210324: Permission denied 2021-03-24T21:42:23.051 INFO:teuthology.orchestra.run.smithi125.stdout:24 Mar 21:42:23 ntpd[7344]: ntpd: time slew -0.000620 s 2021-03-24T21:42:23.051 INFO:teuthology.orchestra.run.smithi125.stdout:ntpd: time slew -0.000620s 2021-03-24T21:42:23.052 INFO:teuthology.orchestra.run.smithi125.stderr:24 Mar 21:42:23 ntpd[7344]: can't open /var/log/ntpstats/loopstats.20210324: Permission denied 2021-03-24T21:42:23.109 INFO:teuthology.orchestra.run.smithi125.stdout: remote refid st t when poll reach delay offset jitter 2021-03-24T21:42:23.109 INFO:teuthology.orchestra.run.smithi125.stdout:============================================================================== 2021-03-24T21:42:23.109 INFO:teuthology.orchestra.run.smithi125.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-24T21:42:23.110 INFO:teuthology.orchestra.run.smithi125.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-24T21:42:23.110 INFO:teuthology.orchestra.run.smithi125.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-24T21:42:23.110 INFO:teuthology.orchestra.run.smithi125.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-24T21:42:23.110 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-03-24T21:42:23.111 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-03-24T21:42:23.140 INFO:teuthology.orchestra.run.smithi142.stdout:24 Mar 21:42:23 ntpd[7371]: ntpd 4.2.8p12@1.3728-o (1): Starting 2021-03-24T21:42:23.141 INFO:teuthology.orchestra.run.smithi142.stdout:24 Mar 21:42:23 ntpd[7371]: Command line: ntpd -gq 2021-03-24T21:42:23.142 INFO:teuthology.orchestra.run.smithi142.stdout:24 Mar 21:42:23 ntpd[7371]: proto: precision = 0.080 usec (-23) 2021-03-24T21:42:23.142 INFO:teuthology.orchestra.run.smithi142.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-03-24T21:42:23.143 INFO:teuthology.orchestra.run.smithi142.stdout:24 Mar 21:42:23 ntpd[7371]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-03-24T21:42:23.143 INFO:teuthology.orchestra.run.smithi142.stderr:restrict ::: KOD does nothing without LIMITED. 2021-03-24T21:42:23.144 INFO:teuthology.orchestra.run.smithi142.stdout:24 Mar 21:42:23 ntpd[7371]: restrict ::: KOD does nothing without LIMITED. 2021-03-24T21:42:23.145 INFO:teuthology.orchestra.run.smithi142.stdout:24 Mar 21:42:23 ntpd[7371]: Listen and drop on 0 v6wildcard [::]:123 2021-03-24T21:42:23.145 INFO:teuthology.orchestra.run.smithi142.stdout:24 Mar 21:42:23 ntpd[7371]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2021-03-24T21:42:23.145 INFO:teuthology.orchestra.run.smithi142.stdout:24 Mar 21:42:23 ntpd[7371]: Listen normally on 2 lo 127.0.0.1:123 2021-03-24T21:42:23.145 INFO:teuthology.orchestra.run.smithi142.stdout:24 Mar 21:42:23 ntpd[7371]: Listen normally on 3 enp3s0f1 172.21.15.142:123 2021-03-24T21:42:23.145 INFO:teuthology.orchestra.run.smithi142.stdout:24 Mar 21:42:23 ntpd[7371]: Listen normally on 4 lo [::1]:123 2021-03-24T21:42:23.146 INFO:teuthology.orchestra.run.smithi142.stdout:24 Mar 21:42:23 ntpd[7371]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6e0d%5]:123 2021-03-24T21:42:23.146 INFO:teuthology.orchestra.run.smithi142.stdout:24 Mar 21:42:23 ntpd[7371]: Listening on routing socket on fd #22 for interface updates 2021-03-24T21:42:24.141 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:24 ntpd[7371]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T21:42:24.142 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:24 ntpd[7371]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T21:42:24.142 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:24 ntpd[7371]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T21:42:24.142 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:24 ntpd[7371]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T21:42:24.143 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:24 ntpd[7371]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-24T21:42:24.143 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:24 ntpd[7371]: can't open /var/log/ntpstats/peerstats.20210324: Permission denied 2021-03-24T21:42:25.141 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:25 ntpd[7371]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T21:42:25.142 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:25 ntpd[7371]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T21:42:25.142 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:25 ntpd[7371]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T21:42:25.142 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:25 ntpd[7371]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T21:42:25.143 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:25 ntpd[7371]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-24T21:42:25.143 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:25 ntpd[7371]: can't open /var/log/ntpstats/peerstats.20210324: Permission denied 2021-03-24T21:42:26.142 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:26 ntpd[7371]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T21:42:26.143 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:26 ntpd[7371]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T21:42:26.143 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:26 ntpd[7371]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T21:42:26.143 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:26 ntpd[7371]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T21:42:26.143 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:26 ntpd[7371]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T21:42:26.144 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:26 ntpd[7371]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T21:42:26.144 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:26 ntpd[7371]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-24T21:42:26.144 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:26 ntpd[7371]: can't open /var/log/ntpstats/peerstats.20210324: Permission denied 2021-03-24T21:42:26.144 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:26 ntpd[7371]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T21:42:26.145 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:26 ntpd[7371]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T21:42:26.145 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:26 ntpd[7371]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-24T21:42:26.145 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:26 ntpd[7371]: can't open /var/log/ntpstats/peerstats.20210324: Permission denied 2021-03-24T21:42:27.141 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:27 ntpd[7371]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T21:42:27.142 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:27 ntpd[7371]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T21:42:27.142 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:27 ntpd[7371]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T21:42:27.142 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:27 ntpd[7371]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T21:42:27.142 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:27 ntpd[7371]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-24T21:42:27.143 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:27 ntpd[7371]: can't open /var/log/ntpstats/peerstats.20210324: Permission denied 2021-03-24T21:42:28.143 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:28 ntpd[7371]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T21:42:28.143 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:28 ntpd[7371]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T21:42:28.144 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:28 ntpd[7371]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T21:42:28.144 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:28 ntpd[7371]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T21:42:28.144 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:28 ntpd[7371]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T21:42:28.144 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:28 ntpd[7371]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T21:42:28.144 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:28 ntpd[7371]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T21:42:28.145 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:28 ntpd[7371]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T21:42:28.145 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:28 ntpd[7371]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-24T21:42:28.145 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:28 ntpd[7371]: can't open /var/log/ntpstats/peerstats.20210324: Permission denied 2021-03-24T21:42:28.145 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:28 ntpd[7371]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T21:42:28.146 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:28 ntpd[7371]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T21:42:28.146 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:28 ntpd[7371]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-24T21:42:28.146 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:28 ntpd[7371]: can't open /var/log/ntpstats/peerstats.20210324: Permission denied 2021-03-24T21:42:28.146 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:28 ntpd[7371]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T21:42:28.146 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:28 ntpd[7371]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T21:42:28.147 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:28 ntpd[7371]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-24T21:42:28.147 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:28 ntpd[7371]: can't open /var/log/ntpstats/peerstats.20210324: Permission denied 2021-03-24T21:42:29.141 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:29 ntpd[7371]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T21:42:29.142 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:29 ntpd[7371]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T21:42:29.142 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:29 ntpd[7371]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T21:42:29.142 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:29 ntpd[7371]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T21:42:29.142 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:29 ntpd[7371]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-24T21:42:29.143 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:29 ntpd[7371]: can't open /var/log/ntpstats/peerstats.20210324: Permission denied 2021-03-24T21:42:30.142 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:30 ntpd[7371]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T21:42:30.142 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:30 ntpd[7371]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T21:42:30.143 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:30 ntpd[7371]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T21:42:30.143 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:30 ntpd[7371]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T21:42:30.143 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:30 ntpd[7371]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T21:42:30.143 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:30 ntpd[7371]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T21:42:30.144 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:30 ntpd[7371]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T21:42:30.144 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:30 ntpd[7371]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T21:42:30.144 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:30 ntpd[7371]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-24T21:42:30.144 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:30 ntpd[7371]: can't open /var/log/ntpstats/peerstats.20210324: Permission denied 2021-03-24T21:42:30.145 INFO:teuthology.orchestra.run.smithi142.stdout:24 Mar 21:42:30 ntpd[7371]: ntpd: time slew -0.003902 s 2021-03-24T21:42:30.145 INFO:teuthology.orchestra.run.smithi142.stdout:ntpd: time slew -0.003902s 2021-03-24T21:42:30.145 INFO:teuthology.orchestra.run.smithi142.stderr:24 Mar 21:42:30 ntpd[7371]: can't open /var/log/ntpstats/loopstats.20210324: Permission denied 2021-03-24T21:42:30.200 INFO:teuthology.orchestra.run.smithi142.stdout: remote refid st t when poll reach delay offset jitter 2021-03-24T21:42:30.201 INFO:teuthology.orchestra.run.smithi142.stdout:============================================================================== 2021-03-24T21:42:30.201 INFO:teuthology.orchestra.run.smithi142.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-24T21:42:30.201 INFO:teuthology.orchestra.run.smithi142.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-24T21:42:30.201 INFO:teuthology.orchestra.run.smithi142.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-24T21:42:30.202 INFO:teuthology.orchestra.run.smithi142.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-24T21:42:30.203 INFO:teuthology.run_tasks:Running task pexec... 2021-03-24T21:42:30.215 INFO:teuthology.task.pexec:Executing custom commands... 2021-03-24T21:42:30.216 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi125.front.sepia.ceph.com 2021-03-24T21:42:30.217 DEBUG:teuthology.orchestra.run.smithi125:> TESTDIR=/home/ubuntu/cephtest bash -s 2021-03-24T21:42:30.217 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi142.front.sepia.ceph.com 2021-03-24T21:42:30.218 DEBUG:teuthology.orchestra.run.smithi142:> TESTDIR=/home/ubuntu/cephtest bash -s 2021-03-24T21:42:30.251 INFO:teuthology.orchestra.run.smithi142.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-24T21:42:30.251 INFO:teuthology.orchestra.run.smithi142.stderr: Dload Upload Total Spent Left Speed 2021-03-24T21:42:30.292 INFO:teuthology.orchestra.run.smithi125.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-24T21:42:30.292 INFO:teuthology.orchestra.run.smithi125.stderr: Dload Upload Total Spent Left Speed 2021-03-24T21:42:30.313 INFO:teuthology.orchestra.run.smithi142.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: apt-key output should not be parsed (stdout is not a terminal) 2021-03-24T21:42:30.354 INFO:teuthology.orchestra.run.smithi125.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: apt-key output should not be parsed (stdout is not a terminal) 2021-03-24T21:42:30.578 INFO:teuthology.orchestra.run.smithi142.stderr: 100 1093 100 1093 0 0 3322 0 --:--:-- --:--:-- --:--:-- 3332 2021-03-24T21:42:30.636 INFO:teuthology.orchestra.run.smithi125.stderr: 100 1093 100 1093 0 0 3186 0 --:--:-- --:--:-- --:--:-- 3186 2021-03-24T21:42:30.809 INFO:teuthology.orchestra.run.smithi142.stdout:OK 2021-03-24T21:42:30.827 INFO:teuthology.orchestra.run.smithi142.stdout:deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04/ / 2021-03-24T21:42:30.849 INFO:teuthology.orchestra.run.smithi142.stderr: 2021-03-24T21:42:30.850 INFO:teuthology.orchestra.run.smithi142.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2021-03-24T21:42:30.850 INFO:teuthology.orchestra.run.smithi142.stderr: 2021-03-24T21:42:30.873 INFO:teuthology.orchestra.run.smithi125.stdout:OK 2021-03-24T21:42:30.894 INFO:teuthology.orchestra.run.smithi125.stdout:deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04/ / 2021-03-24T21:42:30.912 INFO:teuthology.orchestra.run.smithi125.stderr: 2021-03-24T21:42:30.912 INFO:teuthology.orchestra.run.smithi125.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2021-03-24T21:42:30.912 INFO:teuthology.orchestra.run.smithi125.stderr: 2021-03-24T21:42:30.970 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2021-03-24T21:42:31.085 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2021-03-24T21:42:31.160 INFO:teuthology.orchestra.run.smithi125.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2021-03-24T21:42:31.160 INFO:teuthology.orchestra.run.smithi125.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2021-03-24T21:42:31.165 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2021-03-24T21:42:31.229 INFO:teuthology.orchestra.run.smithi125.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2021-03-24T21:42:31.245 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2021-03-24T21:42:31.309 INFO:teuthology.orchestra.run.smithi125.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2021-03-24T21:42:31.355 INFO:teuthology.orchestra.run.smithi142.stdout:Get:5 https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04 InRelease [1,642 B] 2021-03-24T21:42:31.428 INFO:teuthology.orchestra.run.smithi125.stdout:Get:5 https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04 InRelease [1,642 B] 2021-03-24T21:42:31.756 INFO:teuthology.orchestra.run.smithi142.stdout:Get:6 https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04 Packages [12.5 kB] 2021-03-24T21:42:31.873 INFO:teuthology.orchestra.run.smithi142.stdout:Fetched 14.1 kB in 1s (16.0 kB/s) 2021-03-24T21:42:31.952 INFO:teuthology.orchestra.run.smithi125.stdout:Get:6 https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04 Packages [12.5 kB] 2021-03-24T21:42:32.072 INFO:teuthology.orchestra.run.smithi125.stdout:Fetched 14.1 kB in 1s (13.8 kB/s) 2021-03-24T21:42:32.603 INFO:teuthology.orchestra.run.smithi142.stdout:Reading package lists... 2021-03-24T21:42:32.756 INFO:teuthology.orchestra.run.smithi142.stdout:Building dependency tree... 2021-03-24T21:42:32.756 INFO:teuthology.orchestra.run.smithi142.stdout:Reading state information... 2021-03-24T21:42:32.768 INFO:teuthology.orchestra.run.smithi142.stdout:34 packages can be upgraded. Run 'apt list --upgradable' to see them. 2021-03-24T21:42:32.787 INFO:teuthology.orchestra.run.smithi125.stdout:Reading package lists... 2021-03-24T21:42:32.787 INFO:teuthology.orchestra.run.smithi142.stderr: 2021-03-24T21:42:32.788 INFO:teuthology.orchestra.run.smithi142.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2021-03-24T21:42:32.788 INFO:teuthology.orchestra.run.smithi142.stderr: 2021-03-24T21:42:32.833 INFO:teuthology.orchestra.run.smithi142.stdout:Reading package lists... 2021-03-24T21:42:32.934 INFO:teuthology.orchestra.run.smithi125.stdout:Building dependency tree... 2021-03-24T21:42:32.934 INFO:teuthology.orchestra.run.smithi125.stdout:Reading state information... 2021-03-24T21:42:32.946 INFO:teuthology.orchestra.run.smithi125.stdout:34 packages can be upgraded. Run 'apt list --upgradable' to see them. 2021-03-24T21:42:32.959 INFO:teuthology.orchestra.run.smithi125.stderr: 2021-03-24T21:42:32.960 INFO:teuthology.orchestra.run.smithi125.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2021-03-24T21:42:32.960 INFO:teuthology.orchestra.run.smithi125.stderr: 2021-03-24T21:42:32.986 INFO:teuthology.orchestra.run.smithi142.stdout:Building dependency tree... 2021-03-24T21:42:32.987 INFO:teuthology.orchestra.run.smithi142.stdout:Reading state information... 2021-03-24T21:42:33.011 INFO:teuthology.orchestra.run.smithi125.stdout:Reading package lists... 2021-03-24T21:42:33.116 INFO:teuthology.orchestra.run.smithi142.stdout:The following additional packages will be installed: 2021-03-24T21:42:33.116 INFO:teuthology.orchestra.run.smithi142.stdout: catatonit conmon containers-common criu crun fuse-overlayfs fuse3 libfuse3-3 2021-03-24T21:42:33.117 INFO:teuthology.orchestra.run.smithi142.stdout: libgpgme11 libnet1 libprotobuf-c1 libprotobuf17 libyajl2 podman-plugins 2021-03-24T21:42:33.117 INFO:teuthology.orchestra.run.smithi142.stdout: python3-protobuf slirp4netns uidmap 2021-03-24T21:42:33.161 INFO:teuthology.orchestra.run.smithi125.stdout:Building dependency tree... 2021-03-24T21:42:33.162 INFO:teuthology.orchestra.run.smithi125.stdout:Reading state information... 2021-03-24T21:42:33.165 INFO:teuthology.orchestra.run.smithi142.stdout:The following packages will be REMOVED: 2021-03-24T21:42:33.166 INFO:teuthology.orchestra.run.smithi142.stdout: fuse 2021-03-24T21:42:33.167 INFO:teuthology.orchestra.run.smithi142.stdout:The following NEW packages will be installed: 2021-03-24T21:42:33.167 INFO:teuthology.orchestra.run.smithi142.stdout: catatonit conmon containernetworking-plugins containers-common criu crun 2021-03-24T21:42:33.168 INFO:teuthology.orchestra.run.smithi142.stdout: fuse-overlayfs fuse3 libfuse3-3 libgpgme11 libnet1 libprotobuf-c1 2021-03-24T21:42:33.168 INFO:teuthology.orchestra.run.smithi142.stdout: libprotobuf17 libyajl2 podman podman-plugins python3-protobuf slirp4netns 2021-03-24T21:42:33.168 INFO:teuthology.orchestra.run.smithi142.stdout: uidmap 2021-03-24T21:42:33.293 INFO:teuthology.orchestra.run.smithi125.stdout:The following additional packages will be installed: 2021-03-24T21:42:33.293 INFO:teuthology.orchestra.run.smithi125.stdout: catatonit conmon containers-common criu crun fuse-overlayfs fuse3 libfuse3-3 2021-03-24T21:42:33.294 INFO:teuthology.orchestra.run.smithi125.stdout: libgpgme11 libnet1 libprotobuf-c1 libprotobuf17 libyajl2 podman-plugins 2021-03-24T21:42:33.294 INFO:teuthology.orchestra.run.smithi125.stdout: python3-protobuf slirp4netns uidmap 2021-03-24T21:42:33.344 INFO:teuthology.orchestra.run.smithi125.stdout:The following packages will be REMOVED: 2021-03-24T21:42:33.345 INFO:teuthology.orchestra.run.smithi125.stdout: fuse 2021-03-24T21:42:33.345 INFO:teuthology.orchestra.run.smithi125.stdout:The following NEW packages will be installed: 2021-03-24T21:42:33.345 INFO:teuthology.orchestra.run.smithi125.stdout: catatonit conmon containernetworking-plugins containers-common criu crun 2021-03-24T21:42:33.346 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-overlayfs fuse3 libfuse3-3 libgpgme11 libnet1 libprotobuf-c1 2021-03-24T21:42:33.346 INFO:teuthology.orchestra.run.smithi125.stdout: libprotobuf17 libyajl2 podman podman-plugins python3-protobuf slirp4netns 2021-03-24T21:42:33.346 INFO:teuthology.orchestra.run.smithi125.stdout: uidmap 2021-03-24T21:42:33.352 INFO:teuthology.orchestra.run.smithi142.stdout:0 upgraded, 19 newly installed, 1 to remove and 34 not upgraded. 2021-03-24T21:42:33.352 INFO:teuthology.orchestra.run.smithi142.stdout:Need to get 30.3 MB of archives. 2021-03-24T21:42:33.352 INFO:teuthology.orchestra.run.smithi142.stdout:After this operation, 155 MB of additional disk space will be used. 2021-03-24T21:42:33.353 INFO:teuthology.orchestra.run.smithi142.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/universe amd64 libfuse3-3 amd64 3.9.0-2 [74.0 kB] 2021-03-24T21:42:33.532 INFO:teuthology.orchestra.run.smithi125.stdout:0 upgraded, 19 newly installed, 1 to remove and 34 not upgraded. 2021-03-24T21:42:33.532 INFO:teuthology.orchestra.run.smithi125.stdout:Need to get 30.3 MB of archives. 2021-03-24T21:42:33.532 INFO:teuthology.orchestra.run.smithi125.stdout:After this operation, 155 MB of additional disk space will be used. 2021-03-24T21:42:33.532 INFO:teuthology.orchestra.run.smithi125.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/universe amd64 libfuse3-3 amd64 3.9.0-2 [74.0 kB] 2021-03-24T21:42:33.610 INFO:teuthology.orchestra.run.smithi142.stdout:Get:2 https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04 catatonit 0.1.5~1 [275 kB] 2021-03-24T21:42:33.639 INFO:teuthology.orchestra.run.smithi142.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/universe amd64 fuse3 amd64 3.9.0-2 [24.7 kB] 2021-03-24T21:42:33.663 INFO:teuthology.orchestra.run.smithi142.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libprotobuf17 amd64 3.6.1.3-2ubuntu5 [798 kB] 2021-03-24T21:42:33.807 INFO:teuthology.orchestra.run.smithi125.stdout:Get:2 https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04 catatonit 0.1.5~1 [275 kB] 2021-03-24T21:42:33.814 INFO:teuthology.orchestra.run.smithi125.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/universe amd64 fuse3 amd64 3.9.0-2 [24.7 kB] 2021-03-24T21:42:33.815 INFO:teuthology.orchestra.run.smithi125.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libprotobuf17 amd64 3.6.1.3-2ubuntu5 [798 kB] 2021-03-24T21:42:33.942 INFO:teuthology.orchestra.run.smithi142.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-protobuf amd64 3.6.1.3-2ubuntu5 [298 kB] 2021-03-24T21:42:34.099 INFO:teuthology.orchestra.run.smithi142.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 libnet1 amd64 1.1.6+dfsg-3.1build1 [43.3 kB] 2021-03-24T21:42:34.100 INFO:teuthology.orchestra.run.smithi142.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/universe amd64 libprotobuf-c1 amd64 1.3.3-1 [18.9 kB] 2021-03-24T21:42:34.100 INFO:teuthology.orchestra.run.smithi142.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 libyajl2 amd64 2.1.0-3 [20.1 kB] 2021-03-24T21:42:34.100 INFO:teuthology.orchestra.run.smithi142.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 libgpgme11 amd64 1.13.1-7ubuntu2 [121 kB] 2021-03-24T21:42:34.101 INFO:teuthology.orchestra.run.smithi142.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 uidmap amd64 1:4.8.1-1ubuntu5.20.04 [26.3 kB] 2021-03-24T21:42:34.101 INFO:teuthology.orchestra.run.smithi142.stdout:Get:11 https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04 conmon 100:2.0.27-1 [31.5 kB] 2021-03-24T21:42:34.101 INFO:teuthology.orchestra.run.smithi142.stdout:Get:12 https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04 containers-common 100:1-14 [49.0 kB] 2021-03-24T21:42:34.101 INFO:teuthology.orchestra.run.smithi142.stdout:Get:13 https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04 criu 3.15-1 [635 kB] 2021-03-24T21:42:34.120 INFO:teuthology.orchestra.run.smithi125.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-protobuf amd64 3.6.1.3-2ubuntu5 [298 kB] 2021-03-24T21:42:34.155 INFO:teuthology.orchestra.run.smithi125.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 libnet1 amd64 1.1.6+dfsg-3.1build1 [43.3 kB] 2021-03-24T21:42:34.159 INFO:teuthology.orchestra.run.smithi125.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/universe amd64 libprotobuf-c1 amd64 1.3.3-1 [18.9 kB] 2021-03-24T21:42:34.161 INFO:teuthology.orchestra.run.smithi125.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 libyajl2 amd64 2.1.0-3 [20.1 kB] 2021-03-24T21:42:34.162 INFO:teuthology.orchestra.run.smithi125.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 libgpgme11 amd64 1.13.1-7ubuntu2 [121 kB] 2021-03-24T21:42:34.173 INFO:teuthology.orchestra.run.smithi125.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 uidmap amd64 1:4.8.1-1ubuntu5.20.04 [26.3 kB] 2021-03-24T21:42:34.198 INFO:teuthology.orchestra.run.smithi142.stdout:Get:14 https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04 crun 100:0.18-2 [242 kB] 2021-03-24T21:42:34.289 INFO:teuthology.orchestra.run.smithi142.stdout:Get:15 https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04 fuse-overlayfs 1.4.0~1 [41.4 kB] 2021-03-24T21:42:34.290 INFO:teuthology.orchestra.run.smithi142.stdout:Get:16 https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04 podman-plugins 100:1.1.1-5 [1,035 kB] 2021-03-24T21:42:34.310 INFO:teuthology.orchestra.run.smithi125.stdout:Get:11 https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04 conmon 100:2.0.27-1 [31.5 kB] 2021-03-24T21:42:34.312 INFO:teuthology.orchestra.run.smithi125.stdout:Get:12 https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04 containers-common 100:1-14 [49.0 kB] 2021-03-24T21:42:34.313 INFO:teuthology.orchestra.run.smithi125.stdout:Get:13 https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04 criu 3.15-1 [635 kB] 2021-03-24T21:42:34.385 INFO:teuthology.orchestra.run.smithi142.stdout:Get:17 https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04 podman 100:3.0.1-2 [18.5 MB] 2021-03-24T21:42:34.421 INFO:teuthology.orchestra.run.smithi125.stdout:Get:14 https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04 crun 100:0.18-2 [242 kB] 2021-03-24T21:42:34.516 INFO:teuthology.orchestra.run.smithi125.stdout:Get:15 https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04 fuse-overlayfs 1.4.0~1 [41.4 kB] 2021-03-24T21:42:34.516 INFO:teuthology.orchestra.run.smithi125.stdout:Get:16 https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04 podman-plugins 100:1.1.1-5 [1,035 kB] 2021-03-24T21:42:34.617 INFO:teuthology.orchestra.run.smithi125.stdout:Get:17 https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04 podman 100:3.0.1-2 [18.5 MB] 2021-03-24T21:42:36.059 INFO:teuthology.orchestra.run.smithi142.stdout:Get:18 https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04 slirp4netns 100:1.1.8-3 [707 kB] 2021-03-24T21:42:36.072 INFO:teuthology.orchestra.run.smithi142.stdout:Get:19 https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04 containernetworking-plugins 100:0.9.1-1 [7,342 kB] 2021-03-24T21:42:36.136 INFO:teuthology.orchestra.run.smithi125.stdout:Get:18 https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04 slirp4netns 100:1.1.8-3 [707 kB] 2021-03-24T21:42:36.144 INFO:teuthology.orchestra.run.smithi125.stdout:Get:19 https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04 containernetworking-plugins 100:0.9.1-1 [7,342 kB] 2021-03-24T21:42:36.964 INFO:teuthology.orchestra.run.smithi125.stderr:debconf: unable to initialize frontend: Dialog 2021-03-24T21:42:36.964 INFO:teuthology.orchestra.run.smithi125.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2021-03-24T21:42:36.964 INFO:teuthology.orchestra.run.smithi125.stderr:debconf: falling back to frontend: Readline 2021-03-24T21:42:36.988 INFO:teuthology.orchestra.run.smithi142.stderr:debconf: unable to initialize frontend: Dialog 2021-03-24T21:42:36.988 INFO:teuthology.orchestra.run.smithi142.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2021-03-24T21:42:36.988 INFO:teuthology.orchestra.run.smithi142.stderr:debconf: falling back to frontend: Readline 2021-03-24T21:42:36.989 INFO:teuthology.orchestra.run.smithi125.stderr:debconf: unable to initialize frontend: Readline 2021-03-24T21:42:36.989 INFO:teuthology.orchestra.run.smithi125.stderr:debconf: (This frontend requires a controlling tty.) 2021-03-24T21:42:36.989 INFO:teuthology.orchestra.run.smithi125.stderr:debconf: falling back to frontend: Teletype 2021-03-24T21:42:36.993 INFO:teuthology.orchestra.run.smithi125.stderr:dpkg-preconfigure: unable to re-open stdin: 2021-03-24T21:42:37.012 INFO:teuthology.orchestra.run.smithi142.stderr:debconf: unable to initialize frontend: Readline 2021-03-24T21:42:37.012 INFO:teuthology.orchestra.run.smithi142.stderr:debconf: (This frontend requires a controlling tty.) 2021-03-24T21:42:37.012 INFO:teuthology.orchestra.run.smithi142.stderr:debconf: falling back to frontend: Teletype 2021-03-24T21:42:37.018 INFO:teuthology.orchestra.run.smithi142.stderr:dpkg-preconfigure: unable to re-open stdin: 2021-03-24T21:42:37.137 INFO:teuthology.orchestra.run.smithi125.stdout:Fetched 30.3 MB in 3s (8,959 kB/s) 2021-03-24T21:42:37.193 INFO:teuthology.orchestra.run.smithi142.stdout:Fetched 30.3 MB in 4s (8,469 kB/s) 2021-03-24T21:42:37.288 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libfuse3-3:amd64. 2021-03-24T21:42:37.344 INFO:teuthology.orchestra.run.smithi125.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 138107 files and directories currently installed.) 2021-03-24T21:42:37.347 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../libfuse3-3_3.9.0-2_amd64.deb ... 2021-03-24T21:42:37.357 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libfuse3-3:amd64. 2021-03-24T21:42:37.411 INFO:teuthology.orchestra.run.smithi142.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 138107 files and directories currently installed.) 2021-03-24T21:42:37.415 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../libfuse3-3_3.9.0-2_amd64.deb ... 2021-03-24T21:42:37.429 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libfuse3-3:amd64 (3.9.0-2) ... 2021-03-24T21:42:37.524 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libfuse3-3:amd64 (3.9.0-2) ... 2021-03-24T21:42:38.395 INFO:teuthology.orchestra.run.smithi125.stdout:dpkg: fuse: dependency problems, but removing anyway as you requested: 2021-03-24T21:42:38.395 INFO:teuthology.orchestra.run.smithi125.stdout: ntfs-3g depends on fuse. 2021-03-24T21:42:38.395 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-24T21:42:38.451 INFO:teuthology.orchestra.run.smithi125.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 138115 files and directories currently installed.) 2021-03-24T21:42:38.455 INFO:teuthology.orchestra.run.smithi125.stdout:Removing fuse (2.9.9-3) ... 2021-03-24T21:42:38.544 INFO:teuthology.orchestra.run.smithi142.stdout:dpkg: fuse: dependency problems, but removing anyway as you requested: 2021-03-24T21:42:38.544 INFO:teuthology.orchestra.run.smithi142.stdout: ntfs-3g depends on fuse. 2021-03-24T21:42:38.544 INFO:teuthology.orchestra.run.smithi142.stdout: 2021-03-24T21:42:38.600 INFO:teuthology.orchestra.run.smithi142.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 138115 files and directories currently installed.) 2021-03-24T21:42:38.604 INFO:teuthology.orchestra.run.smithi142.stdout:Removing fuse (2.9.9-3) ... 2021-03-24T21:42:38.704 INFO:teuthology.orchestra.run.smithi125.stdout:update-initramfs: deferring update (trigger activated) 2021-03-24T21:42:38.879 INFO:teuthology.orchestra.run.smithi142.stdout:update-initramfs: deferring update (trigger activated) 2021-03-24T21:42:39.078 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package fuse3. 2021-03-24T21:42:39.133 INFO:teuthology.orchestra.run.smithi125.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 138105 files and directories currently installed.) 2021-03-24T21:42:39.137 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../00-fuse3_3.9.0-2_amd64.deb ... 2021-03-24T21:42:39.186 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking fuse3 (3.9.0-2) ... 2021-03-24T21:42:39.349 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package fuse3. 2021-03-24T21:42:39.403 INFO:teuthology.orchestra.run.smithi142.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 138105 files and directories currently installed.) 2021-03-24T21:42:39.407 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../00-fuse3_3.9.0-2_amd64.deb ... 2021-03-24T21:42:39.440 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking fuse3 (3.9.0-2) ... 2021-03-24T21:42:39.476 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package catatonit. 2021-03-24T21:42:39.495 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../01-catatonit_0.1.5~1_amd64.deb ... 2021-03-24T21:42:39.537 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking catatonit (0.1.5~1) ... 2021-03-24T21:42:39.763 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package catatonit. 2021-03-24T21:42:39.778 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../01-catatonit_0.1.5~1_amd64.deb ... 2021-03-24T21:42:39.813 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking catatonit (0.1.5~1) ... 2021-03-24T21:42:39.813 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package conmon. 2021-03-24T21:42:39.830 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../02-conmon_100%3a2.0.27-1_amd64.deb ... 2021-03-24T21:42:39.871 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking conmon (100:2.0.27-1) ... 2021-03-24T21:42:40.122 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package conmon. 2021-03-24T21:42:40.141 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../02-conmon_100%3a2.0.27-1_amd64.deb ... 2021-03-24T21:42:40.173 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking conmon (100:2.0.27-1) ... 2021-03-24T21:42:40.191 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package containers-common. 2021-03-24T21:42:40.206 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../03-containers-common_100%3a1-14_all.deb ... 2021-03-24T21:42:40.243 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking containers-common (100:1-14) ... 2021-03-24T21:42:40.479 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package containers-common. 2021-03-24T21:42:40.495 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../03-containers-common_100%3a1-14_all.deb ... 2021-03-24T21:42:40.557 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking containers-common (100:1-14) ... 2021-03-24T21:42:40.576 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libprotobuf17:amd64. 2021-03-24T21:42:40.591 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../04-libprotobuf17_3.6.1.3-2ubuntu5_amd64.deb ... 2021-03-24T21:42:40.634 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libprotobuf17:amd64 (3.6.1.3-2ubuntu5) ... 2021-03-24T21:42:40.895 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libprotobuf17:amd64. 2021-03-24T21:42:40.913 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../04-libprotobuf17_3.6.1.3-2ubuntu5_amd64.deb ... 2021-03-24T21:42:40.954 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libprotobuf17:amd64 (3.6.1.3-2ubuntu5) ... 2021-03-24T21:42:41.038 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-protobuf. 2021-03-24T21:42:41.054 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../05-python3-protobuf_3.6.1.3-2ubuntu5_amd64.deb ... 2021-03-24T21:42:41.107 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-protobuf (3.6.1.3-2ubuntu5) ... 2021-03-24T21:42:41.377 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-protobuf. 2021-03-24T21:42:41.392 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../05-python3-protobuf_3.6.1.3-2ubuntu5_amd64.deb ... 2021-03-24T21:42:41.436 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-protobuf (3.6.1.3-2ubuntu5) ... 2021-03-24T21:42:41.490 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libnet1:amd64. 2021-03-24T21:42:41.509 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../06-libnet1_1.1.6+dfsg-3.1build1_amd64.deb ... 2021-03-24T21:42:41.549 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libnet1:amd64 (1.1.6+dfsg-3.1build1) ... 2021-03-24T21:42:41.855 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libprotobuf-c1:amd64. 2021-03-24T21:42:41.871 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../07-libprotobuf-c1_1.3.3-1_amd64.deb ... 2021-03-24T21:42:41.905 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libprotobuf-c1:amd64 (1.3.3-1) ... 2021-03-24T21:42:41.916 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libnet1:amd64. 2021-03-24T21:42:41.932 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../06-libnet1_1.1.6+dfsg-3.1build1_amd64.deb ... 2021-03-24T21:42:41.968 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libnet1:amd64 (1.1.6+dfsg-3.1build1) ... 2021-03-24T21:42:42.300 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package criu. 2021-03-24T21:42:42.312 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libprotobuf-c1:amd64. 2021-03-24T21:42:42.320 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../08-criu_3.15-1_amd64.deb ... 2021-03-24T21:42:42.328 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../07-libprotobuf-c1_1.3.3-1_amd64.deb ... 2021-03-24T21:42:42.359 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking criu (3.15-1) ... 2021-03-24T21:42:42.371 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libprotobuf-c1:amd64 (1.3.3-1) ... 2021-03-24T21:42:42.777 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package criu. 2021-03-24T21:42:42.797 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../08-criu_3.15-1_amd64.deb ... 2021-03-24T21:42:42.828 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking criu (3.15-1) ... 2021-03-24T21:42:42.874 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libyajl2:amd64. 2021-03-24T21:42:42.893 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../09-libyajl2_2.1.0-3_amd64.deb ... 2021-03-24T21:42:42.943 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libyajl2:amd64 (2.1.0-3) ... 2021-03-24T21:42:43.184 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package crun. 2021-03-24T21:42:43.201 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../10-crun_100%3a0.18-2_amd64.deb ... 2021-03-24T21:42:43.246 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking crun (100:0.18-2) ... 2021-03-24T21:42:43.320 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libyajl2:amd64. 2021-03-24T21:42:43.336 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../09-libyajl2_2.1.0-3_amd64.deb ... 2021-03-24T21:42:43.404 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libyajl2:amd64 (2.1.0-3) ... 2021-03-24T21:42:43.553 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package fuse-overlayfs. 2021-03-24T21:42:43.569 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../11-fuse-overlayfs_1.4.0~1_amd64.deb ... 2021-03-24T21:42:43.611 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking fuse-overlayfs (1.4.0~1) ... 2021-03-24T21:42:43.695 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package crun. 2021-03-24T21:42:43.714 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../10-crun_100%3a0.18-2_amd64.deb ... 2021-03-24T21:42:43.763 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking crun (100:0.18-2) ... 2021-03-24T21:42:43.969 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libgpgme11:amd64. 2021-03-24T21:42:43.984 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../12-libgpgme11_1.13.1-7ubuntu2_amd64.deb ... 2021-03-24T21:42:44.045 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libgpgme11:amd64 (1.13.1-7ubuntu2) ... 2021-03-24T21:42:44.085 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package fuse-overlayfs. 2021-03-24T21:42:44.103 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../11-fuse-overlayfs_1.4.0~1_amd64.deb ... 2021-03-24T21:42:44.144 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking fuse-overlayfs (1.4.0~1) ... 2021-03-24T21:42:44.328 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package podman-plugins. 2021-03-24T21:42:44.346 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../13-podman-plugins_100%3a1.1.1-5_amd64.deb ... 2021-03-24T21:42:44.388 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking podman-plugins (100:1.1.1-5) ... 2021-03-24T21:42:44.494 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libgpgme11:amd64. 2021-03-24T21:42:44.514 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../12-libgpgme11_1.13.1-7ubuntu2_amd64.deb ... 2021-03-24T21:42:44.578 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libgpgme11:amd64 (1.13.1-7ubuntu2) ... 2021-03-24T21:42:44.836 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package podman-plugins. 2021-03-24T21:42:44.846 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package podman. 2021-03-24T21:42:44.854 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../13-podman-plugins_100%3a1.1.1-5_amd64.deb ... 2021-03-24T21:42:44.862 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../14-podman_100%3a3.0.1-2_amd64.deb ... 2021-03-24T21:42:44.909 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking podman (100:3.0.1-2) ... 2021-03-24T21:42:44.920 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking podman-plugins (100:1.1.1-5) ... 2021-03-24T21:42:45.371 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package podman. 2021-03-24T21:42:45.387 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../14-podman_100%3a3.0.1-2_amd64.deb ... 2021-03-24T21:42:45.470 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking podman (100:3.0.1-2) ... 2021-03-24T21:42:47.009 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package slirp4netns. 2021-03-24T21:42:47.028 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../15-slirp4netns_100%3a1.1.8-3_amd64.deb ... 2021-03-24T21:42:47.084 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking slirp4netns (100:1.1.8-3) ... 2021-03-24T21:42:47.416 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package uidmap. 2021-03-24T21:42:47.433 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../16-uidmap_1%3a4.8.1-1ubuntu5.20.04_amd64.deb ... 2021-03-24T21:42:47.477 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking uidmap (1:4.8.1-1ubuntu5.20.04) ... 2021-03-24T21:42:47.612 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package slirp4netns. 2021-03-24T21:42:47.628 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../15-slirp4netns_100%3a1.1.8-3_amd64.deb ... 2021-03-24T21:42:47.671 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking slirp4netns (100:1.1.8-3) ... 2021-03-24T21:42:47.754 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package containernetworking-plugins. 2021-03-24T21:42:47.770 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../17-containernetworking-plugins_100%3a0.9.1-1_amd64.deb ... 2021-03-24T21:42:47.813 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking containernetworking-plugins (100:0.9.1-1) ... 2021-03-24T21:42:48.018 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package uidmap. 2021-03-24T21:42:48.037 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../16-uidmap_1%3a4.8.1-1ubuntu5.20.04_amd64.deb ... 2021-03-24T21:42:48.079 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking uidmap (1:4.8.1-1ubuntu5.20.04) ... 2021-03-24T21:42:48.356 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package containernetworking-plugins. 2021-03-24T21:42:48.375 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../17-containernetworking-plugins_100%3a0.9.1-1_amd64.deb ... 2021-03-24T21:42:48.415 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking containernetworking-plugins (100:0.9.1-1) ... 2021-03-24T21:42:48.744 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up slirp4netns (100:1.1.8-3) ... 2021-03-24T21:42:48.888 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up uidmap (1:4.8.1-1ubuntu5.20.04) ... 2021-03-24T21:42:49.001 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libyajl2:amd64 (2.1.0-3) ... 2021-03-24T21:42:49.138 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libnet1:amd64 (1.1.6+dfsg-3.1build1) ... 2021-03-24T21:42:49.251 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libprotobuf-c1:amd64 (1.3.3-1) ... 2021-03-24T21:42:49.377 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up slirp4netns (100:1.1.8-3) ... 2021-03-24T21:42:49.405 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libgpgme11:amd64 (1.13.1-7ubuntu2) ... 2021-03-24T21:42:49.497 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up uidmap (1:4.8.1-1ubuntu5.20.04) ... 2021-03-24T21:42:49.519 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up conmon (100:2.0.27-1) ... 2021-03-24T21:42:49.634 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libyajl2:amd64 (2.1.0-3) ... 2021-03-24T21:42:49.648 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up containernetworking-plugins (100:0.9.1-1) ... 2021-03-24T21:42:49.751 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up catatonit (0.1.5~1) ... 2021-03-24T21:42:49.763 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libnet1:amd64 (1.1.6+dfsg-3.1build1) ... 2021-03-24T21:42:49.870 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up containers-common (100:1-14) ... 2021-03-24T21:42:49.926 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libprotobuf-c1:amd64 (1.3.3-1) ... 2021-03-24T21:42:50.054 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libgpgme11:amd64 (1.13.1-7ubuntu2) ... 2021-03-24T21:42:50.197 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up conmon (100:2.0.27-1) ... 2021-03-24T21:42:50.225 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up podman-plugins (100:1.1.1-5) ... 2021-03-24T21:42:50.313 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up containernetworking-plugins (100:0.9.1-1) ... 2021-03-24T21:42:50.338 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libfuse3-3:amd64 (3.9.0-2) ... 2021-03-24T21:42:50.424 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up catatonit (0.1.5~1) ... 2021-03-24T21:42:50.452 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libprotobuf17:amd64 (3.6.1.3-2ubuntu5) ... 2021-03-24T21:42:50.585 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up containers-common (100:1-14) ... 2021-03-24T21:42:50.606 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-protobuf (3.6.1.3-2ubuntu5) ... 2021-03-24T21:42:50.925 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up podman-plugins (100:1.1.1-5) ... 2021-03-24T21:42:51.047 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libfuse3-3:amd64 (3.9.0-2) ... 2021-03-24T21:42:51.162 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up fuse3 (3.9.0-2) ... 2021-03-24T21:42:51.192 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libprotobuf17:amd64 (3.6.1.3-2ubuntu5) ... 2021-03-24T21:42:51.205 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-24T21:42:51.205 INFO:teuthology.orchestra.run.smithi125.stdout:Configuration file '/etc/fuse.conf' 2021-03-24T21:42:51.206 INFO:teuthology.orchestra.run.smithi125.stdout: ==> Modified (by you or by a script) since installation. 2021-03-24T21:42:51.206 INFO:teuthology.orchestra.run.smithi125.stdout: ==> Package distributor has shipped an updated version. 2021-03-24T21:42:51.206 INFO:teuthology.orchestra.run.smithi125.stdout: ==> Keeping old config file as default. 2021-03-24T21:42:51.322 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-protobuf (3.6.1.3-2ubuntu5) ... 2021-03-24T21:42:51.339 INFO:teuthology.orchestra.run.smithi125.stdout:update-initramfs: deferring update (trigger activated) 2021-03-24T21:42:51.398 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up criu (3.15-1) ... 2021-03-24T21:42:51.775 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up fuse-overlayfs (1.4.0~1) ... 2021-03-24T21:42:51.871 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up fuse3 (3.9.0-2) ... 2021-03-24T21:42:51.885 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up crun (100:0.18-2) ... 2021-03-24T21:42:51.906 INFO:teuthology.orchestra.run.smithi142.stdout: 2021-03-24T21:42:51.906 INFO:teuthology.orchestra.run.smithi142.stdout:Configuration file '/etc/fuse.conf' 2021-03-24T21:42:51.907 INFO:teuthology.orchestra.run.smithi142.stdout: ==> Modified (by you or by a script) since installation. 2021-03-24T21:42:51.907 INFO:teuthology.orchestra.run.smithi142.stdout: ==> Package distributor has shipped an updated version. 2021-03-24T21:42:51.907 INFO:teuthology.orchestra.run.smithi142.stdout: ==> Keeping old config file as default. 2021-03-24T21:42:52.002 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up podman (100:3.0.1-2) ... 2021-03-24T21:42:52.054 INFO:teuthology.orchestra.run.smithi142.stdout:update-initramfs: deferring update (trigger activated) 2021-03-24T21:42:52.118 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up criu (3.15-1) ... 2021-03-24T21:42:52.162 INFO:teuthology.orchestra.run.smithi125.stdout:Processing triggers for initramfs-tools (0.136ubuntu6.3) ... 2021-03-24T21:42:52.282 INFO:teuthology.orchestra.run.smithi125.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-70-generic 2021-03-24T21:42:52.469 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up fuse-overlayfs (1.4.0~1) ... 2021-03-24T21:42:52.591 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up crun (100:0.18-2) ... 2021-03-24T21:42:52.743 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up podman (100:3.0.1-2) ... 2021-03-24T21:42:52.924 INFO:teuthology.orchestra.run.smithi142.stdout:Processing triggers for initramfs-tools (0.136ubuntu6.3) ... 2021-03-24T21:42:53.040 INFO:teuthology.orchestra.run.smithi142.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-70-generic 2021-03-24T21:43:17.159 INFO:teuthology.orchestra.run.smithi125.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2021-03-24T21:43:17.291 INFO:teuthology.orchestra.run.smithi125.stdout:Processing triggers for man-db (2.9.1-1) ... 2021-03-24T21:43:18.090 INFO:teuthology.orchestra.run.smithi142.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2021-03-24T21:43:18.237 INFO:teuthology.orchestra.run.smithi142.stdout:Processing triggers for man-db (2.9.1-1) ... 2021-03-24T21:43:20.156 INFO:teuthology.orchestra.run.smithi125.stdout:[[registry]] 2021-03-24T21:43:20.156 INFO:teuthology.orchestra.run.smithi125.stdout:location = 'docker.io' 2021-03-24T21:43:20.157 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-24T21:43:20.157 INFO:teuthology.orchestra.run.smithi125.stdout:[[registry.mirror]] 2021-03-24T21:43:20.157 INFO:teuthology.orchestra.run.smithi125.stdout:location='docker-mirror.front.sepia.ceph.com:5000' 2021-03-24T21:43:20.157 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-24T21:43:20.159 DEBUG:teuthology.parallel:result is None 2021-03-24T21:43:21.185 INFO:teuthology.orchestra.run.smithi142.stdout:[[registry]] 2021-03-24T21:43:21.185 INFO:teuthology.orchestra.run.smithi142.stdout:location = 'docker.io' 2021-03-24T21:43:21.185 INFO:teuthology.orchestra.run.smithi142.stdout: 2021-03-24T21:43:21.185 INFO:teuthology.orchestra.run.smithi142.stdout:[[registry.mirror]] 2021-03-24T21:43:21.186 INFO:teuthology.orchestra.run.smithi142.stdout:location='docker-mirror.front.sepia.ceph.com:5000' 2021-03-24T21:43:21.186 INFO:teuthology.orchestra.run.smithi142.stdout: 2021-03-24T21:43:21.187 DEBUG:teuthology.parallel:result is None 2021-03-24T21:43:21.187 INFO:teuthology.run_tasks:Running task install... 2021-03-24T21:43:21.198 DEBUG:teuthology.task.install:project ceph 2021-03-24T21:43:21.199 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': 'd71e96d216190f28f757f409532464ab0eca4ea9'}} 2021-03-24T21:43:21.200 DEBUG:teuthology.task.install:config {'sha1': 'd71e96d216190f28f757f409532464ab0eca4ea9'} 2021-03-24T21:43:21.200 INFO:teuthology.task.install:Using flavor: basic 2021-03-24T21:43:21.207 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2021-03-24T21:43:21.208 INFO:teuthology.task.install:extra packages: [] 2021-03-24T21:43:21.209 DEBUG:teuthology.orchestra.run.smithi125:> sudo lsb_release -is 2021-03-24T21:43:21.257 INFO:teuthology.orchestra.run.smithi125.stdout:Ubuntu 2021-03-24T21:43:21.262 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-03-24T21:43:21.263 DEBUG:teuthology.orchestra.run.smithi142:> sudo lsb_release -is 2021-03-24T21:43:21.264 DEBUG:teuthology.orchestra.run.smithi125:> sudo apt-key list | grep Ceph 2021-03-24T21:43:21.310 INFO:teuthology.orchestra.run.smithi142.stdout:Ubuntu 2021-03-24T21:43:21.315 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-03-24T21:43:21.315 DEBUG:teuthology.orchestra.run.smithi142:> sudo apt-key list | grep Ceph 2021-03-24T21:43:21.358 INFO:teuthology.orchestra.run.smithi125.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2021-03-24T21:43:21.411 INFO:teuthology.orchestra.run.smithi142.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2021-03-24T21:43:21.414 INFO:teuthology.orchestra.run.smithi125.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2021-03-24T21:43:21.415 INFO:teuthology.orchestra.run.smithi125.stdout:uid [ unknown] Ceph.com (release key) 2021-03-24T21:43:21.416 DEBUG:teuthology.orchestra.run.smithi125:> sudo lsb_release -is 2021-03-24T21:43:21.465 INFO:teuthology.orchestra.run.smithi125.stdout:Ubuntu 2021-03-24T21:43:21.470 INFO:teuthology.orchestra.run.smithi142.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2021-03-24T21:43:21.470 INFO:teuthology.orchestra.run.smithi142.stdout:uid [ unknown] Ceph.com (release key) 2021-03-24T21:43:21.472 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-03-24T21:43:21.473 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2021-03-24T21:43:21.473 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=d71e96d216190f28f757f409532464ab0eca4ea9 2021-03-24T21:43:21.477 DEBUG:teuthology.orchestra.run.smithi142:> sudo lsb_release -is 2021-03-24T21:43:21.526 INFO:teuthology.orchestra.run.smithi142.stdout:Ubuntu 2021-03-24T21:43:21.532 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-03-24T21:43:21.532 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2021-03-24T21:43:21.533 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=d71e96d216190f28f757f409532464ab0eca4ea9 2021-03-24T21:43:21.622 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default/ 2021-03-24T21:43:21.622 INFO:teuthology.task.install.deb:Package version is 16.1.0-1166-gd71e96d2-1focal 2021-03-24T21:43:21.670 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default/ 2021-03-24T21:43:21.670 INFO:teuthology.task.install.deb:Package version is 16.1.0-1166-gd71e96d2-1focal 2021-03-24T21:43:21.740 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2021-03-24T21:43:21.741 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2021-03-24T21:43:21.758 DEBUG:teuthology.orchestra.run.smithi125:> sudo apt-get update 2021-03-24T21:43:21.779 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2021-03-24T21:43:21.779 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2021-03-24T21:43:21.796 DEBUG:teuthology.orchestra.run.smithi142:> sudo apt-get update 2021-03-24T21:43:21.948 INFO:teuthology.orchestra.run.smithi125.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2021-03-24T21:43:22.061 INFO:teuthology.orchestra.run.smithi125.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2021-03-24T21:43:22.066 INFO:teuthology.orchestra.run.smithi125.stdout:Ign:3 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal InRelease 2021-03-24T21:43:22.075 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2021-03-24T21:43:22.075 INFO:teuthology.orchestra.run.smithi142.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [109 kB] 2021-03-24T21:43:22.075 INFO:teuthology.orchestra.run.smithi142.stdout:Ign:3 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal InRelease 2021-03-24T21:43:22.087 INFO:teuthology.orchestra.run.smithi125.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal Release [7,680 B] 2021-03-24T21:43:22.096 INFO:teuthology.orchestra.run.smithi142.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal Release [7,680 B] 2021-03-24T21:43:22.110 INFO:teuthology.orchestra.run.smithi125.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal Release.gpg 2021-03-24T21:43:22.118 INFO:teuthology.orchestra.run.smithi142.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal Release.gpg 2021-03-24T21:43:22.130 INFO:teuthology.orchestra.run.smithi125.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 Packages [16.6 kB] 2021-03-24T21:43:22.140 INFO:teuthology.orchestra.run.smithi142.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main i386 Packages [3,095 B] 2021-03-24T21:43:22.141 INFO:teuthology.orchestra.run.smithi125.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2021-03-24T21:43:22.151 INFO:teuthology.orchestra.run.smithi142.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2021-03-24T21:43:22.160 INFO:teuthology.orchestra.run.smithi125.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main i386 Packages [3,095 B] 2021-03-24T21:43:22.172 INFO:teuthology.orchestra.run.smithi142.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 Packages [16.6 kB] 2021-03-24T21:43:22.317 INFO:teuthology.orchestra.run.smithi125.stdout:Hit:9 https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04 InRelease 2021-03-24T21:43:22.319 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:9 https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04 InRelease 2021-03-24T21:43:22.471 INFO:teuthology.orchestra.run.smithi125.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-backports InRelease [101 kB] 2021-03-24T21:43:22.484 INFO:teuthology.orchestra.run.smithi142.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-backports InRelease [101 kB] 2021-03-24T21:43:22.656 INFO:teuthology.orchestra.run.smithi125.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [16.4 kB] 2021-03-24T21:43:22.800 INFO:teuthology.orchestra.run.smithi142.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [16.4 kB] 2021-03-24T21:43:25.206 INFO:teuthology.orchestra.run.smithi125.stdout:Fetched 258 kB in 1s (306 kB/s) 2021-03-24T21:43:25.473 INFO:teuthology.orchestra.run.smithi142.stdout:Fetched 367 kB in 1s (370 kB/s) 2021-03-24T21:43:25.917 INFO:teuthology.orchestra.run.smithi125.stdout:Reading package lists... 2021-03-24T21:43:25.938 DEBUG:teuthology.orchestra.run.smithi125:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=16.1.0-1166-gd71e96d2-1focal cephadm=16.1.0-1166-gd71e96d2-1focal ceph-mds=16.1.0-1166-gd71e96d2-1focal ceph-mgr=16.1.0-1166-gd71e96d2-1focal ceph-common=16.1.0-1166-gd71e96d2-1focal ceph-fuse=16.1.0-1166-gd71e96d2-1focal ceph-test=16.1.0-1166-gd71e96d2-1focal radosgw=16.1.0-1166-gd71e96d2-1focal python3-rados=16.1.0-1166-gd71e96d2-1focal python3-rgw=16.1.0-1166-gd71e96d2-1focal python3-cephfs=16.1.0-1166-gd71e96d2-1focal python3-rbd=16.1.0-1166-gd71e96d2-1focal libcephfs2=16.1.0-1166-gd71e96d2-1focal libcephfs-dev=16.1.0-1166-gd71e96d2-1focal librados2=16.1.0-1166-gd71e96d2-1focal librbd1=16.1.0-1166-gd71e96d2-1focal rbd-fuse=16.1.0-1166-gd71e96d2-1focal 2021-03-24T21:43:26.003 INFO:teuthology.orchestra.run.smithi125.stdout:Reading package lists... 2021-03-24T21:43:26.159 INFO:teuthology.orchestra.run.smithi125.stdout:Building dependency tree... 2021-03-24T21:43:26.160 INFO:teuthology.orchestra.run.smithi125.stdout:Reading state information... 2021-03-24T21:43:26.184 INFO:teuthology.orchestra.run.smithi142.stdout:Reading package lists... 2021-03-24T21:43:26.206 DEBUG:teuthology.orchestra.run.smithi142:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=16.1.0-1166-gd71e96d2-1focal cephadm=16.1.0-1166-gd71e96d2-1focal ceph-mds=16.1.0-1166-gd71e96d2-1focal ceph-mgr=16.1.0-1166-gd71e96d2-1focal ceph-common=16.1.0-1166-gd71e96d2-1focal ceph-fuse=16.1.0-1166-gd71e96d2-1focal ceph-test=16.1.0-1166-gd71e96d2-1focal radosgw=16.1.0-1166-gd71e96d2-1focal python3-rados=16.1.0-1166-gd71e96d2-1focal python3-rgw=16.1.0-1166-gd71e96d2-1focal python3-cephfs=16.1.0-1166-gd71e96d2-1focal python3-rbd=16.1.0-1166-gd71e96d2-1focal libcephfs2=16.1.0-1166-gd71e96d2-1focal libcephfs-dev=16.1.0-1166-gd71e96d2-1focal librados2=16.1.0-1166-gd71e96d2-1focal librbd1=16.1.0-1166-gd71e96d2-1focal rbd-fuse=16.1.0-1166-gd71e96d2-1focal 2021-03-24T21:43:26.271 INFO:teuthology.orchestra.run.smithi142.stdout:Reading package lists... 2021-03-24T21:43:26.285 INFO:teuthology.orchestra.run.smithi125.stdout:The following packages were automatically installed and are no longer required: 2021-03-24T21:43:26.286 INFO:teuthology.orchestra.run.smithi125.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2021-03-24T21:43:26.287 INFO:teuthology.orchestra.run.smithi125.stdout:Use 'sudo apt autoremove' to remove them. 2021-03-24T21:43:26.287 INFO:teuthology.orchestra.run.smithi125.stdout:The following additional packages will be installed: 2021-03-24T21:43:26.287 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2021-03-24T21:43:26.287 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd jq 2021-03-24T21:43:26.288 INFO:teuthology.orchestra.run.smithi125.stdout: libjaeger libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 libleveldb1d 2021-03-24T21:43:26.288 INFO:teuthology.orchestra.run.smithi125.stdout: liblttng-ust0 liblua5.3-0 liboath0 libonig5 librabbitmq4 libradosstriper1 2021-03-24T21:43:26.288 INFO:teuthology.orchestra.run.smithi125.stdout: librdkafka1 librgw2 libxslt1.1 nvme-cli python-pastedeploy-tpl 2021-03-24T21:43:26.288 INFO:teuthology.orchestra.run.smithi125.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2021-03-24T21:43:26.289 INFO:teuthology.orchestra.run.smithi125.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common 2021-03-24T21:43:26.289 INFO:teuthology.orchestra.run.smithi125.stdout: python3-cherrypy3 python3-cryptography python3-decorator python3-google-auth 2021-03-24T21:43:26.289 INFO:teuthology.orchestra.run.smithi125.stdout: python3-html5lib python3-jinja2 python3-joblib python3-json-pointer 2021-03-24T21:43:26.289 INFO:teuthology.orchestra.run.smithi125.stdout: python3-jsonpatch python3-jwt python3-kubernetes python3-logutils 2021-03-24T21:43:26.289 INFO:teuthology.orchestra.run.smithi125.stdout: python3-lxml python3-mako python3-markupsafe python3-oauthlib 2021-03-24T21:43:26.290 INFO:teuthology.orchestra.run.smithi125.stdout: python3-openssl python3-packaging python3-paste python3-pastedeploy 2021-03-24T21:43:26.290 INFO:teuthology.orchestra.run.smithi125.stdout: python3-pastescript python3-pecan python3-pluggy python3-prettytable 2021-03-24T21:43:26.290 INFO:teuthology.orchestra.run.smithi125.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2021-03-24T21:43:26.290 INFO:teuthology.orchestra.run.smithi125.stdout: python3-pyinotify python3-pytest python3-repoze.lru 2021-03-24T21:43:26.291 INFO:teuthology.orchestra.run.smithi125.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2021-03-24T21:43:26.291 INFO:teuthology.orchestra.run.smithi125.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2021-03-24T21:43:26.291 INFO:teuthology.orchestra.run.smithi125.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2021-03-24T21:43:26.291 INFO:teuthology.orchestra.run.smithi125.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2021-03-24T21:43:26.291 INFO:teuthology.orchestra.run.smithi125.stdout: python3-websocket python3-webtest python3-werkzeug socat xmlstarlet 2021-03-24T21:43:26.292 INFO:teuthology.orchestra.run.smithi125.stdout:Suggested packages: 2021-03-24T21:43:26.292 INFO:teuthology.orchestra.run.smithi125.stdout: python3-influxdb python-attr-doc python-blinker-doc python-cryptography-doc 2021-03-24T21:43:26.292 INFO:teuthology.orchestra.run.smithi125.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2021-03-24T21:43:26.293 INFO:teuthology.orchestra.run.smithi125.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2021-03-24T21:43:26.293 INFO:teuthology.orchestra.run.smithi125.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2021-03-24T21:43:26.293 INFO:teuthology.orchestra.run.smithi125.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2021-03-24T21:43:26.293 INFO:teuthology.orchestra.run.smithi125.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2021-03-24T21:43:26.293 INFO:teuthology.orchestra.run.smithi125.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2021-03-24T21:43:26.294 INFO:teuthology.orchestra.run.smithi125.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2021-03-24T21:43:26.294 INFO:teuthology.orchestra.run.smithi125.stdout:Recommended packages: 2021-03-24T21:43:26.294 INFO:teuthology.orchestra.run.smithi125.stdout: btrfs-tools 2021-03-24T21:43:26.380 INFO:teuthology.orchestra.run.smithi125.stdout:The following NEW packages will be installed: 2021-03-24T21:43:26.380 INFO:teuthology.orchestra.run.smithi125.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2021-03-24T21:43:26.380 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2021-03-24T21:43:26.381 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq 2021-03-24T21:43:26.381 INFO:teuthology.orchestra.run.smithi125.stdout: libcephfs-dev libcephfs2 libjaeger libjq1 libjs-sphinxdoc libjs-underscore 2021-03-24T21:43:26.381 INFO:teuthology.orchestra.run.smithi125.stdout: liblbfgsb0 libleveldb1d liblttng-ust0 liblua5.3-0 liboath0 libonig5 2021-03-24T21:43:26.382 INFO:teuthology.orchestra.run.smithi125.stdout: librabbitmq4 libradosstriper1 librdkafka1 librgw2 libxslt1.1 nvme-cli 2021-03-24T21:43:26.382 INFO:teuthology.orchestra.run.smithi125.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2021-03-24T21:43:26.383 INFO:teuthology.orchestra.run.smithi125.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2021-03-24T21:43:26.383 INFO:teuthology.orchestra.run.smithi125.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2021-03-24T21:43:26.383 INFO:teuthology.orchestra.run.smithi125.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2021-03-24T21:43:26.383 INFO:teuthology.orchestra.run.smithi125.stdout: python3-joblib python3-json-pointer python3-jsonpatch python3-jwt 2021-03-24T21:43:26.384 INFO:teuthology.orchestra.run.smithi125.stdout: python3-kubernetes python3-logutils python3-lxml python3-mako 2021-03-24T21:43:26.384 INFO:teuthology.orchestra.run.smithi125.stdout: python3-markupsafe python3-oauthlib python3-openssl python3-packaging 2021-03-24T21:43:26.384 INFO:teuthology.orchestra.run.smithi125.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2021-03-24T21:43:26.384 INFO:teuthology.orchestra.run.smithi125.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2021-03-24T21:43:26.384 INFO:teuthology.orchestra.run.smithi125.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-rados 2021-03-24T21:43:26.385 INFO:teuthology.orchestra.run.smithi125.stdout: python3-rbd python3-repoze.lru python3-requests-oauthlib python3-rgw 2021-03-24T21:43:26.385 INFO:teuthology.orchestra.run.smithi125.stdout: python3-routes python3-rsa python3-scipy python3-simplegeneric 2021-03-24T21:43:26.385 INFO:teuthology.orchestra.run.smithi125.stdout: python3-simplejson python3-singledispatch python3-sklearn 2021-03-24T21:43:26.385 INFO:teuthology.orchestra.run.smithi125.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2021-03-24T21:43:26.386 INFO:teuthology.orchestra.run.smithi125.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2021-03-24T21:43:26.386 INFO:teuthology.orchestra.run.smithi125.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2021-03-24T21:43:26.386 INFO:teuthology.orchestra.run.smithi125.stdout:The following packages will be upgraded: 2021-03-24T21:43:26.386 INFO:teuthology.orchestra.run.smithi125.stdout: librados2 librbd1 2021-03-24T21:43:26.423 INFO:teuthology.orchestra.run.smithi142.stdout:Building dependency tree... 2021-03-24T21:43:26.424 INFO:teuthology.orchestra.run.smithi142.stdout:Reading state information... 2021-03-24T21:43:26.506 INFO:teuthology.orchestra.run.smithi125.stdout:2 upgraded, 101 newly installed, 0 to remove and 34 not upgraded. 2021-03-24T21:43:26.507 INFO:teuthology.orchestra.run.smithi125.stdout:Need to get 171 MB of archives. 2021-03-24T21:43:26.507 INFO:teuthology.orchestra.run.smithi125.stdout:After this operation, 788 MB of additional disk space will be used. 2021-03-24T21:43:26.507 INFO:teuthology.orchestra.run.smithi125.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 libjaeger amd64 16.1.0-1166-gd71e96d2-1focal [3,728 B] 2021-03-24T21:43:26.549 INFO:teuthology.orchestra.run.smithi125.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 16.1.0-1166-gd71e96d2-1focal [3,281 kB] 2021-03-24T21:43:26.554 INFO:teuthology.orchestra.run.smithi142.stdout:The following packages were automatically installed and are no longer required: 2021-03-24T21:43:26.556 INFO:teuthology.orchestra.run.smithi142.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2021-03-24T21:43:26.556 INFO:teuthology.orchestra.run.smithi142.stdout:Use 'sudo apt autoremove' to remove them. 2021-03-24T21:43:26.557 INFO:teuthology.orchestra.run.smithi142.stdout:The following additional packages will be installed: 2021-03-24T21:43:26.557 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2021-03-24T21:43:26.557 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd jq 2021-03-24T21:43:26.557 INFO:teuthology.orchestra.run.smithi142.stdout: libjaeger libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 libleveldb1d 2021-03-24T21:43:26.558 INFO:teuthology.orchestra.run.smithi142.stdout: liblttng-ust0 liblua5.3-0 liboath0 libonig5 librabbitmq4 libradosstriper1 2021-03-24T21:43:26.558 INFO:teuthology.orchestra.run.smithi142.stdout: librdkafka1 librgw2 libxslt1.1 nvme-cli python-pastedeploy-tpl 2021-03-24T21:43:26.558 INFO:teuthology.orchestra.run.smithi142.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2021-03-24T21:43:26.558 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common 2021-03-24T21:43:26.558 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cherrypy3 python3-cryptography python3-decorator python3-google-auth 2021-03-24T21:43:26.559 INFO:teuthology.orchestra.run.smithi142.stdout: python3-html5lib python3-jinja2 python3-joblib python3-json-pointer 2021-03-24T21:43:26.559 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jsonpatch python3-jwt python3-kubernetes python3-logutils 2021-03-24T21:43:26.559 INFO:teuthology.orchestra.run.smithi142.stdout: python3-lxml python3-mako python3-markupsafe python3-oauthlib 2021-03-24T21:43:26.559 INFO:teuthology.orchestra.run.smithi142.stdout: python3-openssl python3-packaging python3-paste python3-pastedeploy 2021-03-24T21:43:26.560 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pastescript python3-pecan python3-pluggy python3-prettytable 2021-03-24T21:43:26.560 INFO:teuthology.orchestra.run.smithi142.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2021-03-24T21:43:26.560 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pyinotify python3-pytest python3-repoze.lru 2021-03-24T21:43:26.560 INFO:teuthology.orchestra.run.smithi142.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2021-03-24T21:43:26.560 INFO:teuthology.orchestra.run.smithi142.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2021-03-24T21:43:26.561 INFO:teuthology.orchestra.run.smithi142.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2021-03-24T21:43:26.561 INFO:teuthology.orchestra.run.smithi142.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2021-03-24T21:43:26.561 INFO:teuthology.orchestra.run.smithi142.stdout: python3-websocket python3-webtest python3-werkzeug socat xmlstarlet 2021-03-24T21:43:26.561 INFO:teuthology.orchestra.run.smithi142.stdout:Suggested packages: 2021-03-24T21:43:26.561 INFO:teuthology.orchestra.run.smithi142.stdout: python3-influxdb python-attr-doc python-blinker-doc python-cryptography-doc 2021-03-24T21:43:26.562 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2021-03-24T21:43:26.562 INFO:teuthology.orchestra.run.smithi142.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2021-03-24T21:43:26.562 INFO:teuthology.orchestra.run.smithi142.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2021-03-24T21:43:26.562 INFO:teuthology.orchestra.run.smithi142.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2021-03-24T21:43:26.563 INFO:teuthology.orchestra.run.smithi142.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2021-03-24T21:43:26.563 INFO:teuthology.orchestra.run.smithi142.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2021-03-24T21:43:26.563 INFO:teuthology.orchestra.run.smithi142.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2021-03-24T21:43:26.563 INFO:teuthology.orchestra.run.smithi142.stdout:Recommended packages: 2021-03-24T21:43:26.563 INFO:teuthology.orchestra.run.smithi142.stdout: btrfs-tools 2021-03-24T21:43:26.568 INFO:teuthology.orchestra.run.smithi125.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2021-03-24T21:43:26.651 INFO:teuthology.orchestra.run.smithi142.stdout:The following NEW packages will be installed: 2021-03-24T21:43:26.651 INFO:teuthology.orchestra.run.smithi142.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2021-03-24T21:43:26.651 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2021-03-24T21:43:26.652 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq 2021-03-24T21:43:26.652 INFO:teuthology.orchestra.run.smithi142.stdout: libcephfs-dev libcephfs2 libjaeger libjq1 libjs-sphinxdoc libjs-underscore 2021-03-24T21:43:26.652 INFO:teuthology.orchestra.run.smithi142.stdout: liblbfgsb0 libleveldb1d liblttng-ust0 liblua5.3-0 liboath0 libonig5 2021-03-24T21:43:26.652 INFO:teuthology.orchestra.run.smithi142.stdout: librabbitmq4 libradosstriper1 librdkafka1 librgw2 libxslt1.1 nvme-cli 2021-03-24T21:43:26.653 INFO:teuthology.orchestra.run.smithi142.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2021-03-24T21:43:26.653 INFO:teuthology.orchestra.run.smithi142.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2021-03-24T21:43:26.653 INFO:teuthology.orchestra.run.smithi142.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2021-03-24T21:43:26.653 INFO:teuthology.orchestra.run.smithi142.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2021-03-24T21:43:26.654 INFO:teuthology.orchestra.run.smithi142.stdout: python3-joblib python3-json-pointer python3-jsonpatch python3-jwt 2021-03-24T21:43:26.654 INFO:teuthology.orchestra.run.smithi142.stdout: python3-kubernetes python3-logutils python3-lxml python3-mako 2021-03-24T21:43:26.654 INFO:teuthology.orchestra.run.smithi142.stdout: python3-markupsafe python3-oauthlib python3-openssl python3-packaging 2021-03-24T21:43:26.654 INFO:teuthology.orchestra.run.smithi142.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2021-03-24T21:43:26.654 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2021-03-24T21:43:26.655 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-rados 2021-03-24T21:43:26.655 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rbd python3-repoze.lru python3-requests-oauthlib python3-rgw 2021-03-24T21:43:26.655 INFO:teuthology.orchestra.run.smithi142.stdout: python3-routes python3-rsa python3-scipy python3-simplegeneric 2021-03-24T21:43:26.655 INFO:teuthology.orchestra.run.smithi142.stdout: python3-simplejson python3-singledispatch python3-sklearn 2021-03-24T21:43:26.656 INFO:teuthology.orchestra.run.smithi142.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2021-03-24T21:43:26.656 INFO:teuthology.orchestra.run.smithi142.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2021-03-24T21:43:26.656 INFO:teuthology.orchestra.run.smithi142.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2021-03-24T21:43:26.656 INFO:teuthology.orchestra.run.smithi142.stdout:The following packages will be upgraded: 2021-03-24T21:43:26.656 INFO:teuthology.orchestra.run.smithi142.stdout: librados2 librbd1 2021-03-24T21:43:26.725 INFO:teuthology.orchestra.run.smithi125.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 16.1.0-1166-gd71e96d2-1focal [3,353 kB] 2021-03-24T21:43:26.799 INFO:teuthology.orchestra.run.smithi142.stdout:2 upgraded, 101 newly installed, 0 to remove and 34 not upgraded. 2021-03-24T21:43:26.799 INFO:teuthology.orchestra.run.smithi142.stdout:Need to get 171 MB of archives. 2021-03-24T21:43:26.800 INFO:teuthology.orchestra.run.smithi142.stdout:After this operation, 788 MB of additional disk space will be used. 2021-03-24T21:43:26.800 INFO:teuthology.orchestra.run.smithi142.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 libjaeger amd64 16.1.0-1166-gd71e96d2-1focal [3,728 B] 2021-03-24T21:43:26.834 INFO:teuthology.orchestra.run.smithi125.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 16.1.0-1166-gd71e96d2-1focal [706 kB] 2021-03-24T21:43:26.857 INFO:teuthology.orchestra.run.smithi125.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 16.1.0-1166-gd71e96d2-1focal [409 kB] 2021-03-24T21:43:26.864 INFO:teuthology.orchestra.run.smithi142.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 16.1.0-1166-gd71e96d2-1focal [3,281 kB] 2021-03-24T21:43:26.871 INFO:teuthology.orchestra.run.smithi125.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 16.1.0-1166-gd71e96d2-1focal [21.7 kB] 2021-03-24T21:43:26.871 INFO:teuthology.orchestra.run.smithi125.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 16.1.0-1166-gd71e96d2-1focal [210 kB] 2021-03-24T21:43:26.878 INFO:teuthology.orchestra.run.smithi125.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 16.1.0-1166-gd71e96d2-1focal [30.0 kB] 2021-03-24T21:43:26.879 INFO:teuthology.orchestra.run.smithi125.stdout:Get:10 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 16.1.0-1166-gd71e96d2-1focal [384 kB] 2021-03-24T21:43:26.892 INFO:teuthology.orchestra.run.smithi125.stdout:Get:11 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 16.1.0-1166-gd71e96d2-1focal [3,545 kB] 2021-03-24T21:43:26.893 INFO:teuthology.orchestra.run.smithi142.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2021-03-24T21:43:26.986 INFO:teuthology.orchestra.run.smithi125.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2021-03-24T21:43:26.987 INFO:teuthology.orchestra.run.smithi125.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2021-03-24T21:43:26.993 INFO:teuthology.orchestra.run.smithi125.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2021-03-24T21:43:26.994 INFO:teuthology.orchestra.run.smithi125.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2021-03-24T21:43:27.014 INFO:teuthology.orchestra.run.smithi125.stdout:Get:16 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 16.1.0-1166-gd71e96d2-1focal [116 kB] 2021-03-24T21:43:27.020 INFO:teuthology.orchestra.run.smithi125.stdout:Get:17 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 16.1.0-1166-gd71e96d2-1focal [446 kB] 2021-03-24T21:43:27.034 INFO:teuthology.orchestra.run.smithi125.stdout:Get:18 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 16.1.0-1166-gd71e96d2-1focal [22.1 MB] 2021-03-24T21:43:27.161 INFO:teuthology.orchestra.run.smithi125.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 libleveldb1d amd64 1.22-3ubuntu2 [139 kB] 2021-03-24T21:43:27.183 INFO:teuthology.orchestra.run.smithi125.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2021-03-24T21:43:27.190 INFO:teuthology.orchestra.run.smithi125.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2021-03-24T21:43:27.225 INFO:teuthology.orchestra.run.smithi125.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2021-03-24T21:43:27.230 INFO:teuthology.orchestra.run.smithi125.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2021-03-24T21:43:27.232 INFO:teuthology.orchestra.run.smithi125.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2021-03-24T21:43:27.245 INFO:teuthology.orchestra.run.smithi125.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jwt all 1.7.1-2ubuntu2 [17.4 kB] 2021-03-24T21:43:27.247 INFO:teuthology.orchestra.run.smithi125.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2021-03-24T21:43:27.248 INFO:teuthology.orchestra.run.smithi125.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2021-03-24T21:43:27.249 INFO:teuthology.orchestra.run.smithi125.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2 [59.1 kB] 2021-03-24T21:43:27.268 INFO:teuthology.orchestra.run.smithi142.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2021-03-24T21:43:27.275 INFO:teuthology.orchestra.run.smithi142.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2021-03-24T21:43:27.290 INFO:teuthology.orchestra.run.smithi142.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2021-03-24T21:43:27.299 INFO:teuthology.orchestra.run.smithi142.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2021-03-24T21:43:27.323 INFO:teuthology.orchestra.run.smithi125.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2021-03-24T21:43:27.324 INFO:teuthology.orchestra.run.smithi125.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2021-03-24T21:43:27.324 INFO:teuthology.orchestra.run.smithi125.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2021-03-24T21:43:27.328 INFO:teuthology.orchestra.run.smithi125.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2021-03-24T21:43:27.329 INFO:teuthology.orchestra.run.smithi125.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2021-03-24T21:43:27.415 INFO:teuthology.orchestra.run.smithi125.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-waitress all 1.4.1-1 [80.6 kB] 2021-03-24T21:43:27.418 INFO:teuthology.orchestra.run.smithi125.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2021-03-24T21:43:27.419 INFO:teuthology.orchestra.run.smithi125.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2021-03-24T21:43:27.439 INFO:teuthology.orchestra.run.smithi125.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2021-03-24T21:43:27.439 INFO:teuthology.orchestra.run.smithi125.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2021-03-24T21:43:27.443 INFO:teuthology.orchestra.run.smithi142.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 libleveldb1d amd64 1.22-3ubuntu2 [139 kB] 2021-03-24T21:43:27.465 INFO:teuthology.orchestra.run.smithi142.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2021-03-24T21:43:27.472 INFO:teuthology.orchestra.run.smithi142.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2021-03-24T21:43:27.495 INFO:teuthology.orchestra.run.smithi125.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2021-03-24T21:43:27.496 INFO:teuthology.orchestra.run.smithi125.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2021-03-24T21:43:27.499 INFO:teuthology.orchestra.run.smithi125.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-werkzeug all 0.16.1+dfsg1-2 [183 kB] 2021-03-24T21:43:27.501 INFO:teuthology.orchestra.run.smithi142.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2021-03-24T21:43:27.507 INFO:teuthology.orchestra.run.smithi142.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2021-03-24T21:43:27.507 INFO:teuthology.orchestra.run.smithi125.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2021-03-24T21:43:27.509 INFO:teuthology.orchestra.run.smithi142.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2021-03-24T21:43:27.511 INFO:teuthology.orchestra.run.smithi125.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2021-03-24T21:43:27.523 INFO:teuthology.orchestra.run.smithi142.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jwt all 1.7.1-2ubuntu2 [17.4 kB] 2021-03-24T21:43:27.525 INFO:teuthology.orchestra.run.smithi142.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2021-03-24T21:43:27.526 INFO:teuthology.orchestra.run.smithi142.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2021-03-24T21:43:27.527 INFO:teuthology.orchestra.run.smithi142.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2 [59.1 kB] 2021-03-24T21:43:27.575 INFO:teuthology.orchestra.run.smithi125.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-underscore all 1.9.1~dfsg-1 [98.6 kB] 2021-03-24T21:43:27.579 INFO:teuthology.orchestra.run.smithi125.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2021-03-24T21:43:27.583 INFO:teuthology.orchestra.run.smithi125.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2021-03-24T21:43:27.587 INFO:teuthology.orchestra.run.smithi125.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2021-03-24T21:43:27.587 INFO:teuthology.orchestra.run.smithi125.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2021-03-24T21:43:27.607 INFO:teuthology.orchestra.run.smithi142.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2021-03-24T21:43:27.607 INFO:teuthology.orchestra.run.smithi142.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2021-03-24T21:43:27.608 INFO:teuthology.orchestra.run.smithi142.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2021-03-24T21:43:27.611 INFO:teuthology.orchestra.run.smithi142.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2021-03-24T21:43:27.612 INFO:teuthology.orchestra.run.smithi142.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2021-03-24T21:43:27.615 INFO:teuthology.orchestra.run.smithi142.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-waitress all 1.4.1-1 [80.6 kB] 2021-03-24T21:43:27.618 INFO:teuthology.orchestra.run.smithi142.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2021-03-24T21:43:27.619 INFO:teuthology.orchestra.run.smithi142.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2021-03-24T21:43:27.655 INFO:teuthology.orchestra.run.smithi125.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2021-03-24T21:43:27.686 INFO:teuthology.orchestra.run.smithi142.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2021-03-24T21:43:27.687 INFO:teuthology.orchestra.run.smithi142.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2021-03-24T21:43:27.766 INFO:teuthology.orchestra.run.smithi142.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2021-03-24T21:43:27.767 INFO:teuthology.orchestra.run.smithi142.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2021-03-24T21:43:27.770 INFO:teuthology.orchestra.run.smithi142.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-werkzeug all 0.16.1+dfsg1-2 [183 kB] 2021-03-24T21:43:27.774 INFO:teuthology.orchestra.run.smithi142.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2021-03-24T21:43:27.776 INFO:teuthology.orchestra.run.smithi142.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2021-03-24T21:43:27.778 INFO:teuthology.orchestra.run.smithi142.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-underscore all 1.9.1~dfsg-1 [98.6 kB] 2021-03-24T21:43:27.779 INFO:teuthology.orchestra.run.smithi142.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2021-03-24T21:43:27.782 INFO:teuthology.orchestra.run.smithi142.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2021-03-24T21:43:27.846 INFO:teuthology.orchestra.run.smithi142.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2021-03-24T21:43:27.846 INFO:teuthology.orchestra.run.smithi142.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2021-03-24T21:43:27.883 INFO:teuthology.orchestra.run.smithi125.stdout:Get:50 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 16.1.0-1166-gd71e96d2-1focal [5,868 kB] 2021-03-24T21:43:27.926 INFO:teuthology.orchestra.run.smithi142.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2021-03-24T21:43:28.016 INFO:teuthology.orchestra.run.smithi125.stdout:Get:51 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 16.1.0-1166-gd71e96d2-1focal [180 kB] 2021-03-24T21:43:28.021 INFO:teuthology.orchestra.run.smithi125.stdout:Get:52 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 16.1.0-1166-gd71e96d2-1focal [1,492 kB] 2021-03-24T21:43:28.108 INFO:teuthology.orchestra.run.smithi125.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2021-03-24T21:43:28.109 INFO:teuthology.orchestra.run.smithi125.stdout:Get:54 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 16.1.0-1166-gd71e96d2-1focal [6,925 kB] 2021-03-24T21:43:28.132 INFO:teuthology.orchestra.run.smithi125.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2021-03-24T21:43:28.134 INFO:teuthology.orchestra.run.smithi125.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2021-03-24T21:43:28.211 INFO:teuthology.orchestra.run.smithi125.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2021-03-24T21:43:28.211 INFO:teuthology.orchestra.run.smithi125.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2021-03-24T21:43:28.212 INFO:teuthology.orchestra.run.smithi125.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2021-03-24T21:43:28.214 INFO:teuthology.orchestra.run.smithi125.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2021-03-24T21:43:28.215 INFO:teuthology.orchestra.run.smithi125.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2021-03-24T21:43:28.215 INFO:teuthology.orchestra.run.smithi125.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2021-03-24T21:43:28.216 INFO:teuthology.orchestra.run.smithi125.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2021-03-24T21:43:28.270 INFO:teuthology.orchestra.run.smithi125.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2021-03-24T21:43:28.270 INFO:teuthology.orchestra.run.smithi125.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2021-03-24T21:43:28.276 INFO:teuthology.orchestra.run.smithi142.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2021-03-24T21:43:28.324 INFO:teuthology.orchestra.run.smithi142.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2021-03-24T21:43:28.326 INFO:teuthology.orchestra.run.smithi142.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2021-03-24T21:43:28.349 INFO:teuthology.orchestra.run.smithi125.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2021-03-24T21:43:28.355 INFO:teuthology.orchestra.run.smithi125.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-json-pointer all 2.0-0ubuntu1 [8,320 B] 2021-03-24T21:43:28.356 INFO:teuthology.orchestra.run.smithi125.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jsonpatch all 1.23-3 [12.0 kB] 2021-03-24T21:43:28.356 INFO:teuthology.orchestra.run.smithi125.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2021-03-24T21:43:28.360 INFO:teuthology.orchestra.run.smithi125.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2021-03-24T21:43:28.363 INFO:teuthology.orchestra.run.smithi125.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2021-03-24T21:43:28.363 INFO:teuthology.orchestra.run.smithi125.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2021-03-24T21:43:28.366 INFO:teuthology.orchestra.run.smithi125.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/main amd64 libxslt1.1 amd64 1.1.34-4 [152 kB] 2021-03-24T21:43:28.376 INFO:teuthology.orchestra.run.smithi142.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2021-03-24T21:43:28.376 INFO:teuthology.orchestra.run.smithi142.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2021-03-24T21:43:28.377 INFO:teuthology.orchestra.run.smithi142.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2021-03-24T21:43:28.378 INFO:teuthology.orchestra.run.smithi142.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2021-03-24T21:43:28.378 INFO:teuthology.orchestra.run.smithi142.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2021-03-24T21:43:28.379 INFO:teuthology.orchestra.run.smithi142.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2021-03-24T21:43:28.379 INFO:teuthology.orchestra.run.smithi142.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2021-03-24T21:43:28.430 INFO:teuthology.orchestra.run.smithi125.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2021-03-24T21:43:28.435 INFO:teuthology.orchestra.run.smithi125.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2021-03-24T21:43:28.437 INFO:teuthology.orchestra.run.smithi142.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2021-03-24T21:43:28.438 INFO:teuthology.orchestra.run.smithi142.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2021-03-24T21:43:28.478 INFO:teuthology.orchestra.run.smithi125.stdout:Get:76 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 16.1.0-1166-gd71e96d2-1focal [26.0 MB] 2021-03-24T21:43:28.510 INFO:teuthology.orchestra.run.smithi125.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2021-03-24T21:43:28.510 INFO:teuthology.orchestra.run.smithi125.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2021-03-24T21:43:28.511 INFO:teuthology.orchestra.run.smithi125.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2021-03-24T21:43:28.512 INFO:teuthology.orchestra.run.smithi125.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2021-03-24T21:43:28.514 INFO:teuthology.orchestra.run.smithi125.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.2 [1,382 kB] 2021-03-24T21:43:28.517 INFO:teuthology.orchestra.run.smithi142.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2021-03-24T21:43:28.522 INFO:teuthology.orchestra.run.smithi142.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-json-pointer all 2.0-0ubuntu1 [8,320 B] 2021-03-24T21:43:28.523 INFO:teuthology.orchestra.run.smithi142.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jsonpatch all 1.23-3 [12.0 kB] 2021-03-24T21:43:28.524 INFO:teuthology.orchestra.run.smithi142.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2021-03-24T21:43:28.527 INFO:teuthology.orchestra.run.smithi142.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2021-03-24T21:43:28.530 INFO:teuthology.orchestra.run.smithi142.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2021-03-24T21:43:28.531 INFO:teuthology.orchestra.run.smithi125.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2021-03-24T21:43:28.531 INFO:teuthology.orchestra.run.smithi125.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2021-03-24T21:43:28.532 INFO:teuthology.orchestra.run.smithi142.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2021-03-24T21:43:28.532 INFO:teuthology.orchestra.run.smithi125.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2021-03-24T21:43:28.534 INFO:teuthology.orchestra.run.smithi142.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/main amd64 libxslt1.1 amd64 1.1.34-4 [152 kB] 2021-03-24T21:43:28.590 INFO:teuthology.orchestra.run.smithi125.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2021-03-24T21:43:28.592 INFO:teuthology.orchestra.run.smithi125.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-py all 1.8.1-1 [65.4 kB] 2021-03-24T21:43:28.597 INFO:teuthology.orchestra.run.smithi142.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2021-03-24T21:43:28.602 INFO:teuthology.orchestra.run.smithi142.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2021-03-24T21:43:28.670 INFO:teuthology.orchestra.run.smithi125.stdout:Get:87 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2021-03-24T21:43:28.671 INFO:teuthology.orchestra.run.smithi125.stdout:Get:88 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2021-03-24T21:43:28.671 INFO:teuthology.orchestra.run.smithi125.stdout:Get:89 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2021-03-24T21:43:28.675 INFO:teuthology.orchestra.run.smithi125.stdout:Get:90 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2021-03-24T21:43:28.676 INFO:teuthology.orchestra.run.smithi142.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2021-03-24T21:43:28.677 INFO:teuthology.orchestra.run.smithi142.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2021-03-24T21:43:28.678 INFO:teuthology.orchestra.run.smithi142.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2021-03-24T21:43:28.679 INFO:teuthology.orchestra.run.smithi142.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2021-03-24T21:43:28.681 INFO:teuthology.orchestra.run.smithi142.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.2 [1,382 kB] 2021-03-24T21:43:28.700 INFO:teuthology.orchestra.run.smithi142.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2021-03-24T21:43:28.700 INFO:teuthology.orchestra.run.smithi142.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2021-03-24T21:43:28.701 INFO:teuthology.orchestra.run.smithi142.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2021-03-24T21:43:28.756 INFO:teuthology.orchestra.run.smithi142.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2021-03-24T21:43:28.759 INFO:teuthology.orchestra.run.smithi142.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-py all 1.8.1-1 [65.4 kB] 2021-03-24T21:43:28.836 INFO:teuthology.orchestra.run.smithi142.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2021-03-24T21:43:28.837 INFO:teuthology.orchestra.run.smithi142.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2021-03-24T21:43:28.837 INFO:teuthology.orchestra.run.smithi142.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2021-03-24T21:43:28.846 INFO:teuthology.orchestra.run.smithi142.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2021-03-24T21:43:29.320 INFO:teuthology.orchestra.run.smithi142.stdout:Get:75 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 16.1.0-1166-gd71e96d2-1focal [3,353 kB] 2021-03-24T21:43:29.968 INFO:teuthology.orchestra.run.smithi125.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 16.1.0-1166-gd71e96d2-1focal [3,820 B] 2021-03-24T21:43:29.970 INFO:teuthology.orchestra.run.smithi125.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 16.1.0-1166-gd71e96d2-1focal [810 kB] 2021-03-24T21:43:29.983 INFO:teuthology.orchestra.run.smithi125.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 16.1.0-1166-gd71e96d2-1focal [2,302 kB] 2021-03-24T21:43:30.015 INFO:teuthology.orchestra.run.smithi125.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 16.1.0-1166-gd71e96d2-1focal [63.3 kB] 2021-03-24T21:43:30.017 INFO:teuthology.orchestra.run.smithi125.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 16.1.0-1166-gd71e96d2-1focal [75.3 kB] 2021-03-24T21:43:30.019 INFO:teuthology.orchestra.run.smithi125.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 16.1.0-1166-gd71e96d2-1focal [2,335 kB] 2021-03-24T21:43:30.146 INFO:teuthology.orchestra.run.smithi125.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 16.1.0-1166-gd71e96d2-1focal [7,876 kB] 2021-03-24T21:43:30.582 INFO:teuthology.orchestra.run.smithi125.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 16.1.0-1166-gd71e96d2-1focal [3,940 B] 2021-03-24T21:43:30.582 INFO:teuthology.orchestra.run.smithi125.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-rook all 16.1.0-1166-gd71e96d2-1focal [22.1 kB] 2021-03-24T21:43:30.583 INFO:teuthology.orchestra.run.smithi125.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 16.1.0-1166-gd71e96d2-1focal [48.5 MB] 2021-03-24T21:43:32.013 INFO:teuthology.orchestra.run.smithi142.stdout:Get:76 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 16.1.0-1166-gd71e96d2-1focal [706 kB] 2021-03-24T21:43:32.384 INFO:teuthology.orchestra.run.smithi142.stdout:Get:77 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 16.1.0-1166-gd71e96d2-1focal [409 kB] 2021-03-24T21:43:32.566 INFO:teuthology.orchestra.run.smithi142.stdout:Get:78 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 16.1.0-1166-gd71e96d2-1focal [21.7 kB] 2021-03-24T21:43:32.589 INFO:teuthology.orchestra.run.smithi142.stdout:Get:79 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 16.1.0-1166-gd71e96d2-1focal [210 kB] 2021-03-24T21:43:32.669 INFO:teuthology.orchestra.run.smithi142.stdout:Get:80 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 16.1.0-1166-gd71e96d2-1focal [30.0 kB] 2021-03-24T21:43:32.695 INFO:teuthology.orchestra.run.smithi142.stdout:Get:81 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 16.1.0-1166-gd71e96d2-1focal [384 kB] 2021-03-24T21:43:32.839 INFO:teuthology.orchestra.run.smithi142.stdout:Get:82 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 16.1.0-1166-gd71e96d2-1focal [3,545 kB] 2021-03-24T21:43:32.996 INFO:teuthology.orchestra.run.smithi125.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 16.1.0-1166-gd71e96d2-1focal [19.8 kB] 2021-03-24T21:43:32.997 INFO:teuthology.orchestra.run.smithi125.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 16.1.0-1166-gd71e96d2-1focal [10.9 MB] 2021-03-24T21:43:33.295 INFO:teuthology.orchestra.run.smithi125.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 16.1.0-1166-gd71e96d2-1focal [77.9 kB] 2021-03-24T21:43:33.629 INFO:teuthology.orchestra.run.smithi142.stdout:Get:83 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 16.1.0-1166-gd71e96d2-1focal [116 kB] 2021-03-24T21:43:33.648 INFO:teuthology.orchestra.run.smithi142.stdout:Get:84 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 16.1.0-1166-gd71e96d2-1focal [446 kB] 2021-03-24T21:43:33.711 INFO:teuthology.orchestra.run.smithi142.stdout:Get:85 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 16.1.0-1166-gd71e96d2-1focal [22.1 MB] 2021-03-24T21:43:33.860 INFO:teuthology.orchestra.run.smithi125.stdout:Fetched 171 MB in 7s (24.7 MB/s) 2021-03-24T21:43:33.973 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libjaeger. 2021-03-24T21:43:34.028 INFO:teuthology.orchestra.run.smithi125.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 138636 files and directories currently installed.) 2021-03-24T21:43:34.032 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../000-libjaeger_16.1.0-1166-gd71e96d2-1focal_amd64.deb ... 2021-03-24T21:43:34.064 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libjaeger (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:43:34.405 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2021-03-24T21:43:34.421 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../001-liblttng-ust0_2.11.0-1_amd64.deb ... 2021-03-24T21:43:34.497 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2021-03-24T21:43:34.841 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../002-librbd1_16.1.0-1166-gd71e96d2-1focal_amd64.deb ... 2021-03-24T21:43:34.942 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking librbd1 (16.1.0-1166-gd71e96d2-1focal) over (15.2.8-0ubuntu0.20.04.1) ... 2021-03-24T21:43:35.633 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../003-librados2_16.1.0-1166-gd71e96d2-1focal_amd64.deb ... 2021-03-24T21:43:35.744 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking librados2 (16.1.0-1166-gd71e96d2-1focal) over (15.2.8-0ubuntu0.20.04.1) ... 2021-03-24T21:43:36.277 INFO:teuthology.orchestra.run.smithi142.stdout:Get:86 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 16.1.0-1166-gd71e96d2-1focal [5,868 kB] 2021-03-24T21:43:36.403 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libcephfs2. 2021-03-24T21:43:36.418 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../004-libcephfs2_16.1.0-1166-gd71e96d2-1focal_amd64.deb ... 2021-03-24T21:43:36.477 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libcephfs2 (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:43:36.748 INFO:teuthology.orchestra.run.smithi142.stdout:Get:87 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 16.1.0-1166-gd71e96d2-1focal [180 kB] 2021-03-24T21:43:36.764 INFO:teuthology.orchestra.run.smithi142.stdout:Get:88 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 16.1.0-1166-gd71e96d2-1focal [1,492 kB] 2021-03-24T21:43:36.803 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-rados. 2021-03-24T21:43:36.821 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../005-python3-rados_16.1.0-1166-gd71e96d2-1focal_amd64.deb ... 2021-03-24T21:43:36.862 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-rados (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:43:36.887 INFO:teuthology.orchestra.run.smithi142.stdout:Get:89 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 16.1.0-1166-gd71e96d2-1focal [6,925 kB] 2021-03-24T21:43:37.208 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-ceph-argparse. 2021-03-24T21:43:37.225 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../006-python3-ceph-argparse_16.1.0-1166-gd71e96d2-1focal_all.deb ... 2021-03-24T21:43:37.258 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-ceph-argparse (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:43:37.436 INFO:teuthology.orchestra.run.smithi142.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 16.1.0-1166-gd71e96d2-1focal [26.0 MB] 2021-03-24T21:43:37.534 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-cephfs. 2021-03-24T21:43:37.550 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../007-python3-cephfs_16.1.0-1166-gd71e96d2-1focal_amd64.deb ... 2021-03-24T21:43:37.610 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-cephfs (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:43:37.919 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-ceph-common. 2021-03-24T21:43:37.936 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../008-python3-ceph-common_16.1.0-1166-gd71e96d2-1focal_all.deb ... 2021-03-24T21:43:37.969 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-ceph-common (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:43:38.268 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-prettytable. 2021-03-24T21:43:38.284 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../009-python3-prettytable_0.7.2-5_all.deb ... 2021-03-24T21:43:38.318 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2021-03-24T21:43:38.551 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-rbd. 2021-03-24T21:43:38.569 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../010-python3-rbd_16.1.0-1166-gd71e96d2-1focal_amd64.deb ... 2021-03-24T21:43:38.609 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-rbd (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:43:38.956 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2021-03-24T21:43:38.973 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../011-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2021-03-24T21:43:39.014 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2021-03-24T21:43:39.352 INFO:teuthology.orchestra.run.smithi142.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 16.1.0-1166-gd71e96d2-1focal [3,820 B] 2021-03-24T21:43:39.353 INFO:teuthology.orchestra.run.smithi142.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 16.1.0-1166-gd71e96d2-1focal [810 kB] 2021-03-24T21:43:39.365 INFO:teuthology.orchestra.run.smithi142.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 16.1.0-1166-gd71e96d2-1focal [2,302 kB] 2021-03-24T21:43:39.382 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package librabbitmq4:amd64. 2021-03-24T21:43:39.398 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../012-librabbitmq4_0.10.0-1_amd64.deb ... 2021-03-24T21:43:39.434 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2021-03-24T21:43:39.495 INFO:teuthology.orchestra.run.smithi142.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 16.1.0-1166-gd71e96d2-1focal [63.3 kB] 2021-03-24T21:43:39.496 INFO:teuthology.orchestra.run.smithi142.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 16.1.0-1166-gd71e96d2-1focal [75.3 kB] 2021-03-24T21:43:39.497 INFO:teuthology.orchestra.run.smithi142.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 16.1.0-1166-gd71e96d2-1focal [2,335 kB] 2021-03-24T21:43:39.640 INFO:teuthology.orchestra.run.smithi142.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 16.1.0-1166-gd71e96d2-1focal [7,876 kB] 2021-03-24T21:43:39.727 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package librdkafka1:amd64. 2021-03-24T21:43:39.747 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../013-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2021-03-24T21:43:39.778 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2021-03-24T21:43:40.104 INFO:teuthology.orchestra.run.smithi142.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 16.1.0-1166-gd71e96d2-1focal [3,940 B] 2021-03-24T21:43:40.104 INFO:teuthology.orchestra.run.smithi142.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-rook all 16.1.0-1166-gd71e96d2-1focal [22.1 kB] 2021-03-24T21:43:40.105 INFO:teuthology.orchestra.run.smithi142.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 16.1.0-1166-gd71e96d2-1focal [48.5 MB] 2021-03-24T21:43:40.130 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package librgw2. 2021-03-24T21:43:40.146 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../014-librgw2_16.1.0-1166-gd71e96d2-1focal_amd64.deb ... 2021-03-24T21:43:40.180 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking librgw2 (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:43:40.766 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-rgw. 2021-03-24T21:43:40.781 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../015-python3-rgw_16.1.0-1166-gd71e96d2-1focal_amd64.deb ... 2021-03-24T21:43:40.824 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-rgw (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:43:41.169 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libleveldb1d:amd64. 2021-03-24T21:43:41.185 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../016-libleveldb1d_1.22-3ubuntu2_amd64.deb ... 2021-03-24T21:43:41.218 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libleveldb1d:amd64 (1.22-3ubuntu2) ... 2021-03-24T21:43:41.480 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package liboath0. 2021-03-24T21:43:41.495 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../017-liboath0_2.6.1-1.3_amd64.deb ... 2021-03-24T21:43:41.529 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2021-03-24T21:43:41.840 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libradosstriper1. 2021-03-24T21:43:41.857 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../018-libradosstriper1_16.1.0-1166-gd71e96d2-1focal_amd64.deb ... 2021-03-24T21:43:41.898 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libradosstriper1 (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:43:42.402 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package ceph-common. 2021-03-24T21:43:42.418 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../019-ceph-common_16.1.0-1166-gd71e96d2-1focal_amd64.deb ... 2021-03-24T21:43:42.460 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking ceph-common (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:43:42.830 INFO:teuthology.orchestra.run.smithi142.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 16.1.0-1166-gd71e96d2-1focal [19.8 kB] 2021-03-24T21:43:42.830 INFO:teuthology.orchestra.run.smithi142.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 16.1.0-1166-gd71e96d2-1focal [10.9 MB] 2021-03-24T21:43:43.223 INFO:teuthology.orchestra.run.smithi142.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-03-24-1043-pacific/d71e96d216190f28f757f409532464ab0eca4ea9/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 16.1.0-1166-gd71e96d2-1focal [77.9 kB] 2021-03-24T21:43:43.778 INFO:teuthology.orchestra.run.smithi142.stdout:Fetched 171 MB in 17s (10.3 MB/s) 2021-03-24T21:43:43.884 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libjaeger. 2021-03-24T21:43:43.937 INFO:teuthology.orchestra.run.smithi142.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 138636 files and directories currently installed.) 2021-03-24T21:43:43.941 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../000-libjaeger_16.1.0-1166-gd71e96d2-1focal_amd64.deb ... 2021-03-24T21:43:43.991 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libjaeger (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:43:44.324 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2021-03-24T21:43:44.344 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../001-liblttng-ust0_2.11.0-1_amd64.deb ... 2021-03-24T21:43:44.436 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2021-03-24T21:43:44.565 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package ceph-base. 2021-03-24T21:43:44.584 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../020-ceph-base_16.1.0-1166-gd71e96d2-1focal_amd64.deb ... 2021-03-24T21:43:44.642 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking ceph-base (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:43:44.799 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../002-librbd1_16.1.0-1166-gd71e96d2-1focal_amd64.deb ... 2021-03-24T21:43:44.912 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking librbd1 (16.1.0-1166-gd71e96d2-1focal) over (15.2.8-0ubuntu0.20.04.1) ... 2021-03-24T21:43:45.547 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-cryptography. 2021-03-24T21:43:45.564 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../021-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2021-03-24T21:43:45.596 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2021-03-24T21:43:45.634 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../003-librados2_16.1.0-1166-gd71e96d2-1focal_amd64.deb ... 2021-03-24T21:43:45.741 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking librados2 (16.1.0-1166-gd71e96d2-1focal) over (15.2.8-0ubuntu0.20.04.1) ... 2021-03-24T21:43:46.000 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-openssl. 2021-03-24T21:43:46.017 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../022-python3-openssl_19.0.0-1build1_all.deb ... 2021-03-24T21:43:46.058 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2021-03-24T21:43:46.369 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2021-03-24T21:43:46.385 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../023-ceph-mgr-modules-core_16.1.0-1166-gd71e96d2-1focal_all.deb ... 2021-03-24T21:43:46.424 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libcephfs2. 2021-03-24T21:43:46.436 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking ceph-mgr-modules-core (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:43:46.444 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../004-libcephfs2_16.1.0-1166-gd71e96d2-1focal_amd64.deb ... 2021-03-24T21:43:46.516 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libcephfs2 (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:43:46.782 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-bcrypt. 2021-03-24T21:43:46.800 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../024-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2021-03-24T21:43:46.840 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2021-03-24T21:43:46.841 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-rados. 2021-03-24T21:43:46.858 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../005-python3-rados_16.1.0-1166-gd71e96d2-1focal_amd64.deb ... 2021-03-24T21:43:46.896 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-rados (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:43:47.185 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-cherrypy3. 2021-03-24T21:43:47.200 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../025-python3-cherrypy3_8.9.1-6_all.deb ... 2021-03-24T21:43:47.251 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2021-03-24T21:43:47.272 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-ceph-argparse. 2021-03-24T21:43:47.287 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../006-python3-ceph-argparse_16.1.0-1166-gd71e96d2-1focal_all.deb ... 2021-03-24T21:43:47.331 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-ceph-argparse (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:43:47.613 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-jwt. 2021-03-24T21:43:47.624 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-cephfs. 2021-03-24T21:43:47.629 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../026-python3-jwt_1.7.1-2ubuntu2_all.deb ... 2021-03-24T21:43:47.641 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../007-python3-cephfs_16.1.0-1166-gd71e96d2-1focal_amd64.deb ... 2021-03-24T21:43:47.696 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2) ... 2021-03-24T21:43:47.698 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-cephfs (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:43:48.007 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-logutils. 2021-03-24T21:43:48.025 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../027-python3-logutils_0.3.3-7_all.deb ... 2021-03-24T21:43:48.041 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-ceph-common. 2021-03-24T21:43:48.058 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../008-python3-ceph-common_16.1.0-1166-gd71e96d2-1focal_all.deb ... 2021-03-24T21:43:48.089 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-logutils (0.3.3-7) ... 2021-03-24T21:43:48.100 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-ceph-common (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:43:48.393 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-markupsafe. 2021-03-24T21:43:48.410 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../028-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2021-03-24T21:43:48.441 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-prettytable. 2021-03-24T21:43:48.455 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2021-03-24T21:43:48.457 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../009-python3-prettytable_0.7.2-5_all.deb ... 2021-03-24T21:43:48.493 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2021-03-24T21:43:48.762 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-rbd. 2021-03-24T21:43:48.779 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../010-python3-rbd_16.1.0-1166-gd71e96d2-1focal_amd64.deb ... 2021-03-24T21:43:48.788 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-mako. 2021-03-24T21:43:48.804 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../029-python3-mako_1.1.0+ds1-1ubuntu2_all.deb ... 2021-03-24T21:43:48.821 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-rbd (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:43:48.846 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2) ... 2021-03-24T21:43:49.175 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-simplegeneric. 2021-03-24T21:43:49.186 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2021-03-24T21:43:49.192 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../030-python3-simplegeneric_0.8.1-3_all.deb ... 2021-03-24T21:43:49.205 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../011-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2021-03-24T21:43:49.258 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2021-03-24T21:43:49.273 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2021-03-24T21:43:49.560 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-singledispatch. 2021-03-24T21:43:49.576 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../031-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2021-03-24T21:43:49.618 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2021-03-24T21:43:49.658 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package librabbitmq4:amd64. 2021-03-24T21:43:49.675 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../012-librabbitmq4_0.10.0-1_amd64.deb ... 2021-03-24T21:43:49.709 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2021-03-24T21:43:49.947 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-webob. 2021-03-24T21:43:49.964 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../032-python3-webob_1%3a1.8.5-2_all.deb ... 2021-03-24T21:43:50.004 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2021-03-24T21:43:50.054 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package librdkafka1:amd64. 2021-03-24T21:43:50.070 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../013-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2021-03-24T21:43:50.138 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2021-03-24T21:43:50.333 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-soupsieve. 2021-03-24T21:43:50.352 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../033-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2021-03-24T21:43:50.415 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2021-03-24T21:43:50.517 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package librgw2. 2021-03-24T21:43:50.533 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../014-librgw2_16.1.0-1166-gd71e96d2-1focal_amd64.deb ... 2021-03-24T21:43:50.576 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking librgw2 (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:43:50.710 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-bs4. 2021-03-24T21:43:50.725 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../034-python3-bs4_4.8.2-1_all.deb ... 2021-03-24T21:43:50.768 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2021-03-24T21:43:51.130 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-waitress. 2021-03-24T21:43:51.148 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../035-python3-waitress_1.4.1-1_all.deb ... 2021-03-24T21:43:51.191 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-waitress (1.4.1-1) ... 2021-03-24T21:43:51.194 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-rgw. 2021-03-24T21:43:51.211 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../015-python3-rgw_16.1.0-1166-gd71e96d2-1focal_amd64.deb ... 2021-03-24T21:43:51.253 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-rgw (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:43:51.516 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-tempita. 2021-03-24T21:43:51.533 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../036-python3-tempita_0.5.2-6_all.deb ... 2021-03-24T21:43:51.607 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-tempita (0.5.2-6) ... 2021-03-24T21:43:51.622 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libleveldb1d:amd64. 2021-03-24T21:43:51.639 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../016-libleveldb1d_1.22-3ubuntu2_amd64.deb ... 2021-03-24T21:43:51.714 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libleveldb1d:amd64 (1.22-3ubuntu2) ... 2021-03-24T21:43:51.927 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-paste. 2021-03-24T21:43:51.942 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../037-python3-paste_3.4.0+dfsg1-1_all.deb ... 2021-03-24T21:43:51.984 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2021-03-24T21:43:52.100 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package liboath0. 2021-03-24T21:43:52.115 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../017-liboath0_2.6.1-1.3_amd64.deb ... 2021-03-24T21:43:52.159 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2021-03-24T21:43:52.338 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2021-03-24T21:43:52.354 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../038-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2021-03-24T21:43:52.396 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2021-03-24T21:43:52.478 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libradosstriper1. 2021-03-24T21:43:52.497 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../018-libradosstriper1_16.1.0-1166-gd71e96d2-1focal_amd64.deb ... 2021-03-24T21:43:52.536 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libradosstriper1 (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:43:52.699 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-pastedeploy. 2021-03-24T21:43:52.716 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../039-python3-pastedeploy_2.1.0-1_all.deb ... 2021-03-24T21:43:52.796 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2021-03-24T21:43:53.039 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-common. 2021-03-24T21:43:53.057 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../019-ceph-common_16.1.0-1166-gd71e96d2-1focal_amd64.deb ... 2021-03-24T21:43:53.089 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-common (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:43:53.152 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-webtest. 2021-03-24T21:43:53.171 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../040-python3-webtest_2.0.32-2_all.deb ... 2021-03-24T21:43:53.210 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-webtest (2.0.32-2) ... 2021-03-24T21:43:53.563 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-pecan. 2021-03-24T21:43:53.578 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../041-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2021-03-24T21:43:53.621 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2021-03-24T21:43:53.984 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-werkzeug. 2021-03-24T21:43:54.001 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../042-python3-werkzeug_0.16.1+dfsg1-2_all.deb ... 2021-03-24T21:43:54.042 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2) ... 2021-03-24T21:43:54.429 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package ceph-mgr. 2021-03-24T21:43:54.445 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../043-ceph-mgr_16.1.0-1166-gd71e96d2-1focal_amd64.deb ... 2021-03-24T21:43:54.495 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking ceph-mgr (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:43:54.965 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package ceph-mon. 2021-03-24T21:43:54.981 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../044-ceph-mon_16.1.0-1166-gd71e96d2-1focal_amd64.deb ... 2021-03-24T21:43:55.057 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking ceph-mon (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:43:55.428 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-base. 2021-03-24T21:43:55.446 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../020-ceph-base_16.1.0-1166-gd71e96d2-1focal_amd64.deb ... 2021-03-24T21:43:55.497 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-base (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:43:56.052 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package ceph-osd. 2021-03-24T21:43:56.069 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../045-ceph-osd_16.1.0-1166-gd71e96d2-1focal_amd64.deb ... 2021-03-24T21:43:56.102 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking ceph-osd (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:43:56.542 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-cryptography. 2021-03-24T21:43:56.560 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../021-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2021-03-24T21:43:56.624 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2021-03-24T21:43:57.145 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-openssl. 2021-03-24T21:43:57.162 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../022-python3-openssl_19.0.0-1build1_all.deb ... 2021-03-24T21:43:57.237 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2021-03-24T21:43:57.517 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2021-03-24T21:43:57.537 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../023-ceph-mgr-modules-core_16.1.0-1166-gd71e96d2-1focal_all.deb ... 2021-03-24T21:43:57.581 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-mgr-modules-core (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:43:57.961 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-bcrypt. 2021-03-24T21:43:57.980 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../024-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2021-03-24T21:43:58.027 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2021-03-24T21:43:58.330 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-cherrypy3. 2021-03-24T21:43:58.349 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../025-python3-cherrypy3_8.9.1-6_all.deb ... 2021-03-24T21:43:58.380 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2021-03-24T21:43:58.932 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package ceph. 2021-03-24T21:43:58.947 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../046-ceph_16.1.0-1166-gd71e96d2-1focal_amd64.deb ... 2021-03-24T21:43:58.990 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking ceph (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:43:59.317 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-jwt. 2021-03-24T21:43:59.334 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package ceph-fuse. 2021-03-24T21:43:59.335 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../026-python3-jwt_1.7.1-2ubuntu2_all.deb ... 2021-03-24T21:43:59.349 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../047-ceph-fuse_16.1.0-1166-gd71e96d2-1focal_amd64.deb ... 2021-03-24T21:43:59.367 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2) ... 2021-03-24T21:43:59.392 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking ceph-fuse (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:43:59.628 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-logutils. 2021-03-24T21:43:59.646 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../027-python3-logutils_0.3.3-7_all.deb ... 2021-03-24T21:43:59.678 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-logutils (0.3.3-7) ... 2021-03-24T21:43:59.871 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package ceph-mds. 2021-03-24T21:43:59.886 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../048-ceph-mds_16.1.0-1166-gd71e96d2-1focal_amd64.deb ... 2021-03-24T21:43:59.929 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking ceph-mds (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:43:59.997 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-markupsafe. 2021-03-24T21:44:00.014 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../028-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2021-03-24T21:44:00.055 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2021-03-24T21:44:00.392 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-mako. 2021-03-24T21:44:00.408 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../029-python3-mako_1.1.0+ds1-1ubuntu2_all.deb ... 2021-03-24T21:44:00.450 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2) ... 2021-03-24T21:44:00.465 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package cephadm. 2021-03-24T21:44:00.481 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../049-cephadm_16.1.0-1166-gd71e96d2-1focal_amd64.deb ... 2021-03-24T21:44:00.515 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking cephadm (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:00.768 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-jinja2. 2021-03-24T21:44:00.787 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../050-python3-jinja2_2.10.1-2_all.deb ... 2021-03-24T21:44:00.818 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2021-03-24T21:44:00.820 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-simplegeneric. 2021-03-24T21:44:00.838 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../030-python3-simplegeneric_0.8.1-3_all.deb ... 2021-03-24T21:44:00.869 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2021-03-24T21:44:01.104 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2021-03-24T21:44:01.120 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../051-ceph-mgr-cephadm_16.1.0-1166-gd71e96d2-1focal_all.deb ... 2021-03-24T21:44:01.139 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-singledispatch. 2021-03-24T21:44:01.154 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking ceph-mgr-cephadm (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:01.155 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../031-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2021-03-24T21:44:01.189 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2021-03-24T21:44:01.441 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-repoze.lru. 2021-03-24T21:44:01.457 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../052-python3-repoze.lru_0.7-2_all.deb ... 2021-03-24T21:44:01.508 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-webob. 2021-03-24T21:44:01.524 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2021-03-24T21:44:01.525 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../032-python3-webob_1%3a1.8.5-2_all.deb ... 2021-03-24T21:44:01.567 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2021-03-24T21:44:01.776 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libjs-underscore. 2021-03-24T21:44:01.793 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../053-libjs-underscore_1.9.1~dfsg-1_all.deb ... 2021-03-24T21:44:01.835 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1) ... 2021-03-24T21:44:01.928 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-soupsieve. 2021-03-24T21:44:01.948 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../033-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2021-03-24T21:44:01.994 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2021-03-24T21:44:02.146 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libjs-sphinxdoc. 2021-03-24T21:44:02.161 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../054-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2021-03-24T21:44:02.204 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2021-03-24T21:44:02.381 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-bs4. 2021-03-24T21:44:02.398 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../034-python3-bs4_4.8.2-1_all.deb ... 2021-03-24T21:44:02.439 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2021-03-24T21:44:02.515 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-routes. 2021-03-24T21:44:02.533 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../055-python3-routes_2.4.1-2_all.deb ... 2021-03-24T21:44:02.573 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-routes (2.4.1-2) ... 2021-03-24T21:44:02.834 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-waitress. 2021-03-24T21:44:02.850 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../035-python3-waitress_1.4.1-1_all.deb ... 2021-03-24T21:44:02.895 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-waitress (1.4.1-1) ... 2021-03-24T21:44:02.926 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2021-03-24T21:44:02.942 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../056-ceph-mgr-dashboard_16.1.0-1166-gd71e96d2-1focal_all.deb ... 2021-03-24T21:44:02.984 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking ceph-mgr-dashboard (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:03.274 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-tempita. 2021-03-24T21:44:03.293 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../036-python3-tempita_0.5.2-6_all.deb ... 2021-03-24T21:44:03.337 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-tempita (0.5.2-6) ... 2021-03-24T21:44:03.673 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-paste. 2021-03-24T21:44:03.690 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../037-python3-paste_3.4.0+dfsg1-1_all.deb ... 2021-03-24T21:44:03.739 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2021-03-24T21:44:03.967 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-decorator. 2021-03-24T21:44:03.985 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../057-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2021-03-24T21:44:04.025 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2021-03-24T21:44:04.093 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2021-03-24T21:44:04.108 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../038-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2021-03-24T21:44:04.173 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2021-03-24T21:44:04.328 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2021-03-24T21:44:04.345 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../058-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2021-03-24T21:44:04.403 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2021-03-24T21:44:04.512 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-pastedeploy. 2021-03-24T21:44:04.532 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../039-python3-pastedeploy_2.1.0-1_all.deb ... 2021-03-24T21:44:04.570 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2021-03-24T21:44:04.739 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-scipy. 2021-03-24T21:44:04.754 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../059-python3-scipy_1.3.3-3build1_amd64.deb ... 2021-03-24T21:44:04.797 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2021-03-24T21:44:04.898 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-webtest. 2021-03-24T21:44:04.914 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../040-python3-webtest_2.0.32-2_all.deb ... 2021-03-24T21:44:04.956 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-webtest (2.0.32-2) ... 2021-03-24T21:44:05.351 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-pecan. 2021-03-24T21:44:05.367 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../041-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2021-03-24T21:44:05.426 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2021-03-24T21:44:05.797 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-werkzeug. 2021-03-24T21:44:05.814 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../042-python3-werkzeug_0.16.1+dfsg1-2_all.deb ... 2021-03-24T21:44:05.864 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2) ... 2021-03-24T21:44:06.124 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-sklearn-lib. 2021-03-24T21:44:06.141 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../060-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2021-03-24T21:44:06.207 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2021-03-24T21:44:06.326 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-mgr. 2021-03-24T21:44:06.345 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../043-ceph-mgr_16.1.0-1166-gd71e96d2-1focal_amd64.deb ... 2021-03-24T21:44:06.392 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-mgr (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:06.785 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-joblib. 2021-03-24T21:44:06.801 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../061-python3-joblib_0.14.0-3_all.deb ... 2021-03-24T21:44:06.843 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-joblib (0.14.0-3) ... 2021-03-24T21:44:06.945 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-mon. 2021-03-24T21:44:06.964 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../044-ceph-mon_16.1.0-1166-gd71e96d2-1focal_amd64.deb ... 2021-03-24T21:44:07.003 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-mon (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:07.205 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-sklearn. 2021-03-24T21:44:07.221 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../062-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2021-03-24T21:44:07.263 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2021-03-24T21:44:07.980 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2021-03-24T21:44:08.000 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../063-ceph-mgr-diskprediction-local_16.1.0-1166-gd71e96d2-1focal_all.deb ... 2021-03-24T21:44:08.038 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking ceph-mgr-diskprediction-local (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:08.124 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-osd. 2021-03-24T21:44:08.142 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../045-ceph-osd_16.1.0-1166-gd71e96d2-1focal_amd64.deb ... 2021-03-24T21:44:08.182 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-osd (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:09.133 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-cachetools. 2021-03-24T21:44:09.151 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../064-python3-cachetools_4.0.0-1_all.deb ... 2021-03-24T21:44:09.191 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2021-03-24T21:44:09.493 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-pyasn1. 2021-03-24T21:44:09.510 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../065-python3-pyasn1_0.4.2-3build1_all.deb ... 2021-03-24T21:44:09.552 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2021-03-24T21:44:09.939 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-pyasn1-modules. 2021-03-24T21:44:09.956 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../066-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2021-03-24T21:44:10.005 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2021-03-24T21:44:10.358 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-rsa. 2021-03-24T21:44:10.375 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../067-python3-rsa_4.0-3_all.deb ... 2021-03-24T21:44:10.416 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-rsa (4.0-3) ... 2021-03-24T21:44:10.727 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-google-auth. 2021-03-24T21:44:10.745 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../068-python3-google-auth_1.5.1-2_all.deb ... 2021-03-24T21:44:10.770 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph. 2021-03-24T21:44:10.787 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../046-ceph_16.1.0-1166-gd71e96d2-1focal_amd64.deb ... 2021-03-24T21:44:10.807 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2021-03-24T21:44:10.829 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:11.131 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-blinker. 2021-03-24T21:44:11.148 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../069-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2021-03-24T21:44:11.181 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-fuse. 2021-03-24T21:44:11.188 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2021-03-24T21:44:11.198 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../047-ceph-fuse_16.1.0-1166-gd71e96d2-1focal_amd64.deb ... 2021-03-24T21:44:11.239 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-fuse (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:11.550 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-oauthlib. 2021-03-24T21:44:11.566 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../070-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2021-03-24T21:44:11.608 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2021-03-24T21:44:11.743 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-mds. 2021-03-24T21:44:11.759 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../048-ceph-mds_16.1.0-1166-gd71e96d2-1focal_amd64.deb ... 2021-03-24T21:44:11.801 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-mds (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:11.928 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-requests-oauthlib. 2021-03-24T21:44:11.948 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../071-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2021-03-24T21:44:12.011 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2021-03-24T21:44:12.338 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package cephadm. 2021-03-24T21:44:12.339 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-websocket. 2021-03-24T21:44:12.355 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../072-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2021-03-24T21:44:12.357 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../049-cephadm_16.1.0-1166-gd71e96d2-1focal_amd64.deb ... 2021-03-24T21:44:12.396 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking cephadm (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:12.397 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2021-03-24T21:44:12.717 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-kubernetes. 2021-03-24T21:44:12.734 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../073-python3-kubernetes_7.0.0~a1-2_all.deb ... 2021-03-24T21:44:12.749 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-jinja2. 2021-03-24T21:44:12.766 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../050-python3-jinja2_2.10.1-2_all.deb ... 2021-03-24T21:44:12.775 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2021-03-24T21:44:12.832 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2021-03-24T21:44:13.185 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2021-03-24T21:44:13.202 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../051-ceph-mgr-cephadm_16.1.0-1166-gd71e96d2-1focal_all.deb ... 2021-03-24T21:44:13.243 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-mgr-cephadm (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:13.355 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2021-03-24T21:44:13.371 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../074-ceph-mgr-k8sevents_16.1.0-1166-gd71e96d2-1focal_all.deb ... 2021-03-24T21:44:13.413 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking ceph-mgr-k8sevents (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:13.638 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-repoze.lru. 2021-03-24T21:44:13.655 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../052-python3-repoze.lru_0.7-2_all.deb ... 2021-03-24T21:44:13.705 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2021-03-24T21:44:13.766 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-json-pointer. 2021-03-24T21:44:13.782 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../075-python3-json-pointer_2.0-0ubuntu1_all.deb ... 2021-03-24T21:44:13.824 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-json-pointer (2.0-0ubuntu1) ... 2021-03-24T21:44:14.008 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libjs-underscore. 2021-03-24T21:44:14.026 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../053-libjs-underscore_1.9.1~dfsg-1_all.deb ... 2021-03-24T21:44:14.066 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1) ... 2021-03-24T21:44:14.085 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-jsonpatch. 2021-03-24T21:44:14.104 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../076-python3-jsonpatch_1.23-3_all.deb ... 2021-03-24T21:44:14.135 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-jsonpatch (1.23-3) ... 2021-03-24T21:44:14.360 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libjs-sphinxdoc. 2021-03-24T21:44:14.378 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../054-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2021-03-24T21:44:14.443 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2021-03-24T21:44:14.455 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package ceph-mgr-rook. 2021-03-24T21:44:14.471 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../077-ceph-mgr-rook_16.1.0-1166-gd71e96d2-1focal_all.deb ... 2021-03-24T21:44:14.513 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking ceph-mgr-rook (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:14.813 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-routes. 2021-03-24T21:44:14.830 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../055-python3-routes_2.4.1-2_all.deb ... 2021-03-24T21:44:14.841 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libonig5:amd64. 2021-03-24T21:44:14.858 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../078-libonig5_6.9.4-1_amd64.deb ... 2021-03-24T21:44:14.871 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-routes (2.4.1-2) ... 2021-03-24T21:44:14.915 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2021-03-24T21:44:15.242 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2021-03-24T21:44:15.261 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../056-ceph-mgr-dashboard_16.1.0-1166-gd71e96d2-1focal_all.deb ... 2021-03-24T21:44:15.268 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libjq1:amd64. 2021-03-24T21:44:15.285 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../079-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2021-03-24T21:44:15.299 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-mgr-dashboard (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:15.326 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2021-03-24T21:44:15.621 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package jq. 2021-03-24T21:44:15.638 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../080-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2021-03-24T21:44:15.679 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2021-03-24T21:44:15.983 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package socat. 2021-03-24T21:44:16.001 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../081-socat_1.7.3.3-2_amd64.deb ... 2021-03-24T21:44:16.058 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking socat (1.7.3.3-2) ... 2021-03-24T21:44:16.232 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-decorator. 2021-03-24T21:44:16.252 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../057-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2021-03-24T21:44:16.290 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2021-03-24T21:44:16.420 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libxslt1.1:amd64. 2021-03-24T21:44:16.437 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../082-libxslt1.1_1.1.34-4_amd64.deb ... 2021-03-24T21:44:16.478 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libxslt1.1:amd64 (1.1.34-4) ... 2021-03-24T21:44:16.635 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2021-03-24T21:44:16.651 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../058-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2021-03-24T21:44:16.727 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2021-03-24T21:44:16.790 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package xmlstarlet. 2021-03-24T21:44:16.807 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../083-xmlstarlet_1.6.1-2build1_amd64.deb ... 2021-03-24T21:44:16.848 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2021-03-24T21:44:17.080 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-scipy. 2021-03-24T21:44:17.095 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../059-python3-scipy_1.3.3-3build1_amd64.deb ... 2021-03-24T21:44:17.138 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2021-03-24T21:44:17.227 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package ceph-test. 2021-03-24T21:44:17.243 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../084-ceph-test_16.1.0-1166-gd71e96d2-1focal_amd64.deb ... 2021-03-24T21:44:17.285 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking ceph-test (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:18.523 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-sklearn-lib. 2021-03-24T21:44:18.539 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../060-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2021-03-24T21:44:18.581 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2021-03-24T21:44:19.193 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-joblib. 2021-03-24T21:44:19.210 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../061-python3-joblib_0.14.0-3_all.deb ... 2021-03-24T21:44:19.251 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-joblib (0.14.0-3) ... 2021-03-24T21:44:19.613 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-sklearn. 2021-03-24T21:44:19.630 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../062-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2021-03-24T21:44:19.871 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2021-03-24T21:44:20.938 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2021-03-24T21:44:20.957 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../063-ceph-mgr-diskprediction-local_16.1.0-1166-gd71e96d2-1focal_all.deb ... 2021-03-24T21:44:21.004 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-mgr-diskprediction-local (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:21.388 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libcephfs-dev. 2021-03-24T21:44:21.405 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../085-libcephfs-dev_16.1.0-1166-gd71e96d2-1focal_amd64.deb ... 2021-03-24T21:44:21.446 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libcephfs-dev (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:21.850 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package nvme-cli. 2021-03-24T21:44:21.865 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../086-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2021-03-24T21:44:21.925 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2021-03-24T21:44:22.166 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-cachetools. 2021-03-24T21:44:22.182 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../064-python3-cachetools_4.0.0-1_all.deb ... 2021-03-24T21:44:22.224 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2021-03-24T21:44:22.230 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-atomicwrites. 2021-03-24T21:44:22.249 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../087-python3-atomicwrites_1.1.5-2build1_all.deb ... 2021-03-24T21:44:22.280 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2021-03-24T21:44:22.560 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-pyasn1. 2021-03-24T21:44:22.576 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../065-python3-pyasn1_0.4.2-3build1_all.deb ... 2021-03-24T21:44:22.591 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-attr. 2021-03-24T21:44:22.611 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../088-python3-attr_19.3.0-2_all.deb ... 2021-03-24T21:44:22.618 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2021-03-24T21:44:22.649 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-attr (19.3.0-2) ... 2021-03-24T21:44:22.972 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-pyasn1-modules. 2021-03-24T21:44:22.985 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-webencodings. 2021-03-24T21:44:22.992 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../066-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2021-03-24T21:44:23.002 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../089-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2021-03-24T21:44:23.043 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2021-03-24T21:44:23.050 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2021-03-24T21:44:23.330 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-html5lib. 2021-03-24T21:44:23.346 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../090-python3-html5lib_1.0.1-2_all.deb ... 2021-03-24T21:44:23.374 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-rsa. 2021-03-24T21:44:23.379 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2021-03-24T21:44:23.393 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../067-python3-rsa_4.0-3_all.deb ... 2021-03-24T21:44:23.432 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-rsa (4.0-3) ... 2021-03-24T21:44:23.700 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-lxml:amd64. 2021-03-24T21:44:23.719 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../091-python3-lxml_4.5.0-1ubuntu0.2_amd64.deb ... 2021-03-24T21:44:23.749 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.2) ... 2021-03-24T21:44:23.786 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-google-auth. 2021-03-24T21:44:23.803 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../068-python3-google-auth_1.5.1-2_all.deb ... 2021-03-24T21:44:23.844 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2021-03-24T21:44:24.189 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-blinker. 2021-03-24T21:44:24.206 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../069-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2021-03-24T21:44:24.211 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-packaging. 2021-03-24T21:44:24.231 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../092-python3-packaging_20.3-1_all.deb ... 2021-03-24T21:44:24.247 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2021-03-24T21:44:24.260 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-packaging (20.3-1) ... 2021-03-24T21:44:24.556 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-pastescript. 2021-03-24T21:44:24.572 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../093-python3-pastescript_2.0.2-4_all.deb ... 2021-03-24T21:44:24.592 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-oauthlib. 2021-03-24T21:44:24.610 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../070-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2021-03-24T21:44:24.613 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2021-03-24T21:44:24.661 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2021-03-24T21:44:24.959 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-pluggy. 2021-03-24T21:44:24.976 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../094-python3-pluggy_0.13.0-2_all.deb ... 2021-03-24T21:44:25.014 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-requests-oauthlib. 2021-03-24T21:44:25.016 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2021-03-24T21:44:25.031 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../071-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2021-03-24T21:44:25.072 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2021-03-24T21:44:25.345 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-psutil. 2021-03-24T21:44:25.361 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../095-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2021-03-24T21:44:25.403 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2021-03-24T21:44:25.408 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-websocket. 2021-03-24T21:44:25.424 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../072-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2021-03-24T21:44:25.466 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2021-03-24T21:44:25.731 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-py. 2021-03-24T21:44:25.747 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../096-python3-py_1.8.1-1_all.deb ... 2021-03-24T21:44:25.789 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-py (1.8.1-1) ... 2021-03-24T21:44:25.836 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-kubernetes. 2021-03-24T21:44:25.854 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../073-python3-kubernetes_7.0.0~a1-2_all.deb ... 2021-03-24T21:44:25.903 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2021-03-24T21:44:26.092 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-pyinotify. 2021-03-24T21:44:26.112 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../097-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2021-03-24T21:44:26.142 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2021-03-24T21:44:26.420 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-wcwidth. 2021-03-24T21:44:26.439 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../098-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2021-03-24T21:44:26.511 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2021-03-24T21:44:26.549 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2021-03-24T21:44:26.567 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../074-ceph-mgr-k8sevents_16.1.0-1166-gd71e96d2-1focal_all.deb ... 2021-03-24T21:44:26.607 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-mgr-k8sevents (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:26.823 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-pytest. 2021-03-24T21:44:26.842 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../099-python3-pytest_4.6.9-1_all.deb ... 2021-03-24T21:44:26.880 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-pytest (4.6.9-1) ... 2021-03-24T21:44:26.960 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-json-pointer. 2021-03-24T21:44:26.976 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../075-python3-json-pointer_2.0-0ubuntu1_all.deb ... 2021-03-24T21:44:27.018 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-json-pointer (2.0-0ubuntu1) ... 2021-03-24T21:44:27.251 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-simplejson. 2021-03-24T21:44:27.271 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../100-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2021-03-24T21:44:27.308 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2021-03-24T21:44:27.379 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-jsonpatch. 2021-03-24T21:44:27.400 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../076-python3-jsonpatch_1.23-3_all.deb ... 2021-03-24T21:44:27.437 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-jsonpatch (1.23-3) ... 2021-03-24T21:44:27.762 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package radosgw. 2021-03-24T21:44:27.774 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-mgr-rook. 2021-03-24T21:44:27.778 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../101-radosgw_16.1.0-1166-gd71e96d2-1focal_amd64.deb ... 2021-03-24T21:44:27.790 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../077-ceph-mgr-rook_16.1.0-1166-gd71e96d2-1focal_all.deb ... 2021-03-24T21:44:27.822 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking radosgw (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:27.832 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-mgr-rook (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:28.185 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libonig5:amd64. 2021-03-24T21:44:28.202 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../078-libonig5_6.9.4-1_amd64.deb ... 2021-03-24T21:44:28.293 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2021-03-24T21:44:28.605 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libjq1:amd64. 2021-03-24T21:44:28.624 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../079-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2021-03-24T21:44:28.654 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2021-03-24T21:44:28.943 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package rbd-fuse. 2021-03-24T21:44:28.957 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package jq. 2021-03-24T21:44:28.960 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../102-rbd-fuse_16.1.0-1166-gd71e96d2-1focal_amd64.deb ... 2021-03-24T21:44:28.973 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../080-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2021-03-24T21:44:29.001 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking rbd-fuse (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:29.015 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2021-03-24T21:44:29.296 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-attr (19.3.0-2) ... 2021-03-24T21:44:29.379 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package socat. 2021-03-24T21:44:29.395 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../081-socat_1.7.3.3-2_amd64.deb ... 2021-03-24T21:44:29.461 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking socat (1.7.3.3-2) ... 2021-03-24T21:44:29.621 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libleveldb1d:amd64 (1.22-3ubuntu2) ... 2021-03-24T21:44:29.722 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2021-03-24T21:44:29.807 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libxslt1.1:amd64. 2021-03-24T21:44:29.825 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../082-libxslt1.1_1.1.34-4_amd64.deb ... 2021-03-24T21:44:29.848 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2021-03-24T21:44:29.856 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libxslt1.1:amd64 (1.1.34-4) ... 2021-03-24T21:44:30.066 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up cephadm (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:30.143 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package xmlstarlet. 2021-03-24T21:44:30.159 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../083-xmlstarlet_1.6.1-2build1_amd64.deb ... 2021-03-24T21:44:30.201 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2021-03-24T21:44:30.597 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-test. 2021-03-24T21:44:30.614 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../084-ceph-test_16.1.0-1166-gd71e96d2-1focal_amd64.deb ... 2021-03-24T21:44:30.655 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-test (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:30.719 INFO:teuthology.orchestra.run.smithi125.stdout:Adding system user cephadm....done 2021-03-24T21:44:30.897 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-waitress (1.4.1-1) ... 2021-03-24T21:44:31.232 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-repoze.lru (0.7-2) ... 2021-03-24T21:44:31.492 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up liboath0 (2.6.1-1.3) ... 2021-03-24T21:44:31.592 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-py (1.8.1-1) ... 2021-03-24T21:44:31.944 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-joblib (0.14.0-3) ... 2021-03-24T21:44:32.354 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-jwt (1.7.1-2ubuntu2) ... 2021-03-24T21:44:32.655 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-cachetools (4.0.0-1) ... 2021-03-24T21:44:32.948 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2021-03-24T21:44:33.199 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2021-03-24T21:44:33.467 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-ceph-argparse (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:33.743 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2021-03-24T21:44:33.869 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2021-03-24T21:44:34.145 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2021-03-24T21:44:34.438 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2021-03-24T21:44:34.806 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2021-03-24T21:44:35.008 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libcephfs-dev. 2021-03-24T21:44:35.026 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../085-libcephfs-dev_16.1.0-1166-gd71e96d2-1focal_amd64.deb ... 2021-03-24T21:44:35.073 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2021-03-24T21:44:35.091 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libcephfs-dev (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:35.400 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2021-03-24T21:44:35.511 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package nvme-cli. 2021-03-24T21:44:35.531 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../086-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2021-03-24T21:44:35.587 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2021-03-24T21:44:35.667 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libjaeger (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:35.818 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2021-03-24T21:44:36.018 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-atomicwrites. 2021-03-24T21:44:36.036 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../087-python3-atomicwrites_1.1.5-2build1_all.deb ... 2021-03-24T21:44:36.075 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2021-03-24T21:44:36.153 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-tempita (0.5.2-6) ... 2021-03-24T21:44:36.387 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-attr. 2021-03-24T21:44:36.406 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../088-python3-attr_19.3.0-2_all.deb ... 2021-03-24T21:44:36.437 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-attr (19.3.0-2) ... 2021-03-24T21:44:36.437 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-packaging (20.3-1) ... 2021-03-24T21:44:36.713 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2021-03-24T21:44:36.731 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-webencodings. 2021-03-24T21:44:36.750 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../089-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2021-03-24T21:44:36.781 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2021-03-24T21:44:36.839 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2021-03-24T21:44:37.100 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-html5lib. 2021-03-24T21:44:37.117 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../090-python3-html5lib_1.0.1-2_all.deb ... 2021-03-24T21:44:37.124 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2021-03-24T21:44:37.158 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2021-03-24T21:44:37.500 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2) ... 2021-03-24T21:44:37.563 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-lxml:amd64. 2021-03-24T21:44:37.584 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../091-python3-lxml_4.5.0-1ubuntu0.2_amd64.deb ... 2021-03-24T21:44:37.628 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.2) ... 2021-03-24T21:44:37.852 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2021-03-24T21:44:38.132 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-packaging. 2021-03-24T21:44:38.147 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../092-python3-packaging_20.3-1_all.deb ... 2021-03-24T21:44:38.215 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-packaging (20.3-1) ... 2021-03-24T21:44:38.220 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up socat (1.7.3.3-2) ... 2021-03-24T21:44:38.346 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-ceph-common (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:38.543 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-pastescript. 2021-03-24T21:44:38.561 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../093-python3-pastescript_2.0.2-4_all.deb ... 2021-03-24T21:44:38.601 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2021-03-24T21:44:38.656 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-html5lib (1.0.1-2) ... 2021-03-24T21:44:38.971 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-pluggy. 2021-03-24T21:44:38.987 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../094-python3-pluggy_0.13.0-2_all.deb ... 2021-03-24T21:44:39.016 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2021-03-24T21:44:39.029 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2021-03-24T21:44:39.117 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-pluggy (0.13.0-2) ... 2021-03-24T21:44:39.316 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-psutil. 2021-03-24T21:44:39.331 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../095-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2021-03-24T21:44:39.365 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2021-03-24T21:44:39.418 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-json-pointer (2.0-0ubuntu1) ... 2021-03-24T21:44:39.694 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2021-03-24T21:44:39.727 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-py. 2021-03-24T21:44:39.743 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../096-python3-py_1.8.1-1_all.deb ... 2021-03-24T21:44:39.801 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-py (1.8.1-1) ... 2021-03-24T21:44:39.845 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libxslt1.1:amd64 (1.1.34-4) ... 2021-03-24T21:44:39.971 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2021-03-24T21:44:40.122 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-pyinotify. 2021-03-24T21:44:40.141 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../097-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2021-03-24T21:44:40.171 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2021-03-24T21:44:40.289 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2021-03-24T21:44:40.524 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-wcwidth. 2021-03-24T21:44:40.541 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../098-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2021-03-24T21:44:40.582 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2021-03-24T21:44:40.607 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-logutils (0.3.3-7) ... 2021-03-24T21:44:40.918 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-pytest. 2021-03-24T21:44:40.924 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2021-03-24T21:44:40.934 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../099-python3-pytest_4.6.9-1_all.deb ... 2021-03-24T21:44:41.001 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-pytest (4.6.9-1) ... 2021-03-24T21:44:41.025 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2021-03-24T21:44:41.302 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-prettytable (0.7.2-5) ... 2021-03-24T21:44:41.347 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-simplejson. 2021-03-24T21:44:41.365 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../100-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2021-03-24T21:44:41.405 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2021-03-24T21:44:41.577 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2021-03-24T21:44:41.876 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package radosgw. 2021-03-24T21:44:41.878 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2021-03-24T21:44:41.892 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../101-radosgw_16.1.0-1166-gd71e96d2-1focal_amd64.deb ... 2021-03-24T21:44:41.959 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking radosgw (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:42.225 INFO:teuthology.orchestra.run.smithi125.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2021-03-24T21:44:42.490 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2021-03-24T21:44:42.615 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2) ... 2021-03-24T21:44:42.950 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2021-03-24T21:44:43.211 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package rbd-fuse. 2021-03-24T21:44:43.229 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../102-rbd-fuse_16.1.0-1166-gd71e96d2-1focal_amd64.deb ... 2021-03-24T21:44:43.285 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libjs-underscore (1.9.1~dfsg-1) ... 2021-03-24T21:44:43.302 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking rbd-fuse (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:43.396 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2021-03-24T21:44:43.522 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-webob (1:1.8.5-2) ... 2021-03-24T21:44:43.644 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-attr (19.3.0-2) ... 2021-03-24T21:44:43.857 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2021-03-24T21:44:43.915 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libleveldb1d:amd64 (1.22-3ubuntu2) ... 2021-03-24T21:44:44.033 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2021-03-24T21:44:44.141 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up librados2 (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:44.192 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2021-03-24T21:44:44.242 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2021-03-24T21:44:44.394 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up cephadm (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:45.147 INFO:teuthology.orchestra.run.smithi142.stdout:Adding system user cephadm....done 2021-03-24T21:44:45.309 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-waitress (1.4.1-1) ... 2021-03-24T21:44:45.660 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-repoze.lru (0.7-2) ... 2021-03-24T21:44:45.703 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up librgw2 (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:45.848 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2021-03-24T21:44:45.959 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-pytest (4.6.9-1) ... 2021-03-24T21:44:45.978 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up liboath0 (2.6.1-1.3) ... 2021-03-24T21:44:46.129 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-py (1.8.1-1) ... 2021-03-24T21:44:46.325 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2021-03-24T21:44:46.439 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-joblib (0.14.0-3) ... 2021-03-24T21:44:46.611 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-jsonpatch (1.23-3) ... 2021-03-24T21:44:46.713 INFO:teuthology.orchestra.run.smithi125.stdout:update-alternatives: using /usr/bin/jsonpatch-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2021-03-24T21:44:46.840 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-jwt (1.7.1-2ubuntu2) ... 2021-03-24T21:44:46.942 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2021-03-24T21:44:47.108 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-cachetools (4.0.0-1) ... 2021-03-24T21:44:47.218 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-bs4 (4.8.2-1) ... 2021-03-24T21:44:47.409 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2021-03-24T21:44:47.569 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libcephfs2 (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:47.694 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2021-03-24T21:44:47.695 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2021-03-24T21:44:47.987 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-ceph-argparse (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:48.022 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libradosstriper1 (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:48.148 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2021-03-24T21:44:48.279 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2021-03-24T21:44:48.380 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2021-03-24T21:44:48.491 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2021-03-24T21:44:48.623 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2021-03-24T21:44:48.882 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2021-03-24T21:44:49.250 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2021-03-24T21:44:49.524 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up librbd1 (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:49.526 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2021-03-24T21:44:49.650 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2021-03-24T21:44:49.751 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2021-03-24T21:44:49.861 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.2) ... 2021-03-24T21:44:49.861 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2021-03-24T21:44:50.154 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libjaeger (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:50.170 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2021-03-24T21:44:50.296 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-rsa (4.0-3) ... 2021-03-24T21:44:50.297 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2021-03-24T21:44:50.589 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-webtest (2.0.32-2) ... 2021-03-24T21:44:50.707 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-tempita (0.5.2-6) ... 2021-03-24T21:44:50.899 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up ceph-mgr-modules-core (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:51.000 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up ceph-fuse (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:51.207 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-03-24T21:44:51.207 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-03-24T21:44:51.215 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-packaging (20.3-1) ... 2021-03-24T21:44:51.575 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2021-03-24T21:44:51.660 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libcephfs-dev (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:51.709 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2021-03-24T21:44:51.778 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2021-03-24T21:44:51.994 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2021-03-24T21:44:52.096 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-pastescript (2.0.2-4) ... 2021-03-24T21:44:52.395 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2) ... 2021-03-24T21:44:52.406 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2021-03-24T21:44:52.789 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2021-03-24T21:44:52.800 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-rados (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:52.901 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-rbd (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:53.027 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up rbd-fuse (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:53.166 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up socat (1.7.3.3-2) ... 2021-03-24T21:44:53.221 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-routes (2.4.1-2) ... 2021-03-24T21:44:53.350 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-ceph-common (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:53.522 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-google-auth (1.5.1-2) ... 2021-03-24T21:44:53.676 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-html5lib (1.0.1-2) ... 2021-03-24T21:44:53.857 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-rgw (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:53.975 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-cephfs (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:54.020 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2021-03-24T21:44:54.146 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-pluggy (0.13.0-2) ... 2021-03-24T21:44:54.235 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2021-03-24T21:44:54.455 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-json-pointer (2.0-0ubuntu1) ... 2021-03-24T21:44:54.764 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2021-03-24T21:44:54.915 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libxslt1.1:amd64 (1.1.34-4) ... 2021-03-24T21:44:55.058 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2021-03-24T21:44:55.385 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2021-03-24T21:44:55.546 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up ceph-common (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:55.702 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-logutils (0.3.3-7) ... 2021-03-24T21:44:56.029 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2021-03-24T21:44:56.097 INFO:teuthology.orchestra.run.smithi125.stdout:Adding group ceph....done 2021-03-24T21:44:56.188 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2021-03-24T21:44:56.473 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-prettytable (0.7.2-5) ... 2021-03-24T21:44:56.487 INFO:teuthology.orchestra.run.smithi125.stdout:Adding system user ceph....done 2021-03-24T21:44:56.688 INFO:teuthology.orchestra.run.smithi125.stdout:Setting system user ceph properties....done 2021-03-24T21:44:56.704 INFO:teuthology.orchestra.run.smithi125.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2021-03-24T21:44:56.782 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2021-03-24T21:44:56.855 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2021-03-24T21:44:57.077 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2021-03-24T21:44:57.174 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2021-03-24T21:44:57.448 INFO:teuthology.orchestra.run.smithi142.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2021-03-24T21:44:57.609 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up ceph-test (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:57.730 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2021-03-24T21:44:57.735 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up radosgw (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:57.889 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2) ... 2021-03-24T21:44:58.039 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-03-24T21:44:58.039 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-03-24T21:44:58.223 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2021-03-24T21:44:58.575 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libjs-underscore (1.9.1~dfsg-1) ... 2021-03-24T21:44:58.721 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up ceph-base (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:58.726 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2021-03-24T21:44:58.877 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-webob (1:1.8.5-2) ... 2021-03-24T21:44:59.055 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2021-03-24T21:44:59.215 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2021-03-24T21:44:59.506 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up ceph-mds (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:59.508 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up librados2 (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:44:59.646 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2021-03-24T21:44:59.785 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-03-24T21:44:59.786 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-03-24T21:45:00.199 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up ceph-mgr (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:45:00.425 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-03-24T21:45:00.425 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-03-24T21:45:00.859 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up ceph-mgr-rook (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:45:01.001 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up ceph-osd (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:45:01.115 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up librgw2 (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:45:01.261 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2021-03-24T21:45:01.401 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-pytest (4.6.9-1) ... 2021-03-24T21:45:01.572 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-03-24T21:45:01.572 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-03-24T21:45:01.769 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2021-03-24T21:45:02.012 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up ceph-mgr-k8sevents (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:45:02.096 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-jsonpatch (1.23-3) ... 2021-03-24T21:45:02.138 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up ceph-mgr-diskprediction-local (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:45:02.210 INFO:teuthology.orchestra.run.smithi142.stdout:update-alternatives: using /usr/bin/jsonpatch-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2021-03-24T21:45:02.280 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up ceph-mon (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:45:02.439 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2021-03-24T21:45:02.505 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-03-24T21:45:02.505 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-03-24T21:45:02.764 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-bs4 (4.8.2-1) ... 2021-03-24T21:45:02.949 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up ceph-mgr-cephadm (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:45:03.092 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up ceph (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:45:03.144 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libcephfs2 (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:45:03.218 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up ceph-mgr-dashboard (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:45:03.270 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2021-03-24T21:45:03.361 INFO:teuthology.orchestra.run.smithi125.stdout:Processing triggers for systemd (245.4-4ubuntu3.4) ... 2021-03-24T21:45:03.632 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libradosstriper1 (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:45:03.670 INFO:teuthology.orchestra.run.smithi125.stdout:Processing triggers for man-db (2.9.1-1) ... 2021-03-24T21:45:03.792 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2021-03-24T21:45:04.135 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2021-03-24T21:45:05.166 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up librbd1 (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:45:05.333 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2021-03-24T21:45:05.494 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2021-03-24T21:45:05.620 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.2) ... 2021-03-24T21:45:05.938 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2021-03-24T21:45:06.106 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-rsa (4.0-3) ... 2021-03-24T21:45:06.424 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-webtest (2.0.32-2) ... 2021-03-24T21:45:06.733 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-mgr-modules-core (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:45:06.782 INFO:teuthology.orchestra.run.smithi125.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2021-03-24T21:45:06.884 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-fuse (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:45:07.129 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-03-24T21:45:07.130 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-03-24T21:45:07.175 INFO:teuthology.orchestra.run.smithi125.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2021-03-24T21:45:07.179 DEBUG:teuthology.parallel:result is None 2021-03-24T21:45:07.602 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libcephfs-dev (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:45:07.736 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2021-03-24T21:45:08.021 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-pastescript (2.0.2-4) ... 2021-03-24T21:45:08.340 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2021-03-24T21:45:08.701 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-rados (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:45:08.860 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-rbd (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:45:08.986 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up rbd-fuse (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:45:09.222 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-routes (2.4.1-2) ... 2021-03-24T21:45:09.525 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-google-auth (1.5.1-2) ... 2021-03-24T21:45:09.828 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-rgw (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:45:09.954 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-cephfs (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:45:10.280 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2021-03-24T21:45:11.267 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-common (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:45:11.852 INFO:teuthology.orchestra.run.smithi142.stdout:Adding group ceph....done 2021-03-24T21:45:12.271 INFO:teuthology.orchestra.run.smithi142.stdout:Adding system user ceph....done 2021-03-24T21:45:12.497 INFO:teuthology.orchestra.run.smithi142.stdout:Setting system user ceph properties....done 2021-03-24T21:45:12.512 INFO:teuthology.orchestra.run.smithi142.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2021-03-24T21:45:12.664 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2021-03-24T21:45:12.986 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2021-03-24T21:45:13.435 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-test (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:45:13.560 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up radosgw (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:45:13.860 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-03-24T21:45:13.860 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-03-24T21:45:14.513 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-base (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:45:14.883 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2021-03-24T21:45:15.323 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-mds (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:45:15.555 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-03-24T21:45:15.555 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-03-24T21:45:15.966 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-mgr (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:45:16.197 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-03-24T21:45:16.197 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-03-24T21:45:16.676 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-mgr-rook (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:45:16.844 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-osd (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:45:17.392 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-03-24T21:45:17.392 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-03-24T21:45:17.838 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-mgr-k8sevents (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:45:17.997 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-mgr-diskprediction-local (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:45:18.140 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-mon (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:45:18.367 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-03-24T21:45:18.367 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-03-24T21:45:18.808 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-mgr-cephadm (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:45:18.985 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:45:19.111 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-mgr-dashboard (16.1.0-1166-gd71e96d2-1focal) ... 2021-03-24T21:45:19.279 INFO:teuthology.orchestra.run.smithi142.stdout:Processing triggers for systemd (245.4-4ubuntu3.4) ... 2021-03-24T21:45:19.572 INFO:teuthology.orchestra.run.smithi142.stdout:Processing triggers for man-db (2.9.1-1) ... 2021-03-24T21:45:22.659 INFO:teuthology.orchestra.run.smithi142.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2021-03-24T21:45:23.083 INFO:teuthology.orchestra.run.smithi142.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2021-03-24T21:45:23.086 DEBUG:teuthology.parallel:result is None 2021-03-24T21:45:23.087 DEBUG:teuthology.orchestra.run.smithi125:> sudo lsb_release -is 2021-03-24T21:45:23.139 INFO:teuthology.orchestra.run.smithi125.stdout:Ubuntu 2021-03-24T21:45:23.145 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-03-24T21:45:23.146 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=d71e96d216190f28f757f409532464ab0eca4ea9 2021-03-24T21:45:23.287 DEBUG:teuthology.orchestra.run.smithi125:> dpkg-query -W -f '${Version}' ceph 2021-03-24T21:45:23.312 INFO:teuthology.orchestra.run.smithi125.stdout:16.1.0-1166-gd71e96d2-1focal 2021-03-24T21:45:23.313 INFO:teuthology.packaging:The installed version of ceph is 16.1.0-1166-gd71e96d2-1focal 2021-03-24T21:45:23.313 INFO:teuthology.task.install:The correct ceph version 16.1.0-1166-gd71e96d2-1focal is installed. 2021-03-24T21:45:23.314 DEBUG:teuthology.orchestra.run.smithi142:> sudo lsb_release -is 2021-03-24T21:45:23.364 INFO:teuthology.orchestra.run.smithi142.stdout:Ubuntu 2021-03-24T21:45:23.369 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-03-24T21:45:23.370 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=d71e96d216190f28f757f409532464ab0eca4ea9 2021-03-24T21:45:23.502 DEBUG:teuthology.orchestra.run.smithi142:> dpkg-query -W -f '${Version}' ceph 2021-03-24T21:45:23.525 INFO:teuthology.orchestra.run.smithi142.stdout:16.1.0-1166-gd71e96d2-1focal 2021-03-24T21:45:23.526 INFO:teuthology.packaging:The installed version of ceph is 16.1.0-1166-gd71e96d2-1focal 2021-03-24T21:45:23.526 INFO:teuthology.task.install:The correct ceph version 16.1.0-1166-gd71e96d2-1focal is installed. 2021-03-24T21:45:23.527 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2021-03-24T21:45:23.527 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2021-03-24T21:45:23.528 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-03-24T21:45:23.545 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2021-03-24T21:45:23.545 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-03-24T21:45:23.559 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2021-03-24T21:45:23.560 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2021-03-24T21:45:23.560 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd of=/usr/bin/daemon-helper 2021-03-24T21:45:23.576 DEBUG:teuthology.orchestra.run.smithi125:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-03-24T21:45:23.635 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2021-03-24T21:45:23.636 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/usr/bin/daemon-helper 2021-03-24T21:45:23.652 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-03-24T21:45:23.664 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2021-03-24T21:45:23.664 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2021-03-24T21:45:23.664 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd of=/usr/bin/adjust-ulimits 2021-03-24T21:45:23.681 DEBUG:teuthology.orchestra.run.smithi125:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-03-24T21:45:23.691 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2021-03-24T21:45:23.691 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/usr/bin/adjust-ulimits 2021-03-24T21:45:23.721 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-03-24T21:45:23.732 INFO:teuthology.run_tasks:Running task cephadm... 2021-03-24T21:45:23.818 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'global': {'ms bind msgr2': False, 'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'mon_bind_msgr2': False, 'sha1': 'd71e96d216190f28f757f409532464ab0eca4ea9', 'cephadm_mode': 'root'} 2021-03-24T21:45:23.819 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 2021-03-24T21:45:23.819 INFO:tasks.cephadm:Cluster fsid is 3cfa339a-8cea-11eb-90b2-001a4aab830c 2021-03-24T21:45:23.820 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2021-03-24T21:45:23.820 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '[v1:172.21.15.125:6789]', 'mon.c': '[v1:172.21.15.125:6790]', 'mon.b': '[v1:172.21.15.142:6789]'} 2021-03-24T21:45:23.821 INFO:tasks.cephadm:First mon is mon.a on smithi125 2021-03-24T21:45:23.821 INFO:tasks.cephadm:First mgr is y 2021-03-24T21:45:23.821 INFO:tasks.cephadm:Normalizing hostnames... 2021-03-24T21:45:23.822 DEBUG:teuthology.orchestra.run.smithi125:> sudo hostname $(hostname -s) 2021-03-24T21:45:23.839 DEBUG:teuthology.orchestra.run.smithi142:> sudo hostname $(hostname -s) 2021-03-24T21:45:23.856 INFO:tasks.cephadm:Downloading cephadm (repo git://git.ceph.com/ceph-ci.git ref d71e96d216190f28f757f409532464ab0eca4ea9)... 2021-03-24T21:45:23.857 DEBUG:teuthology.orchestra.run.smithi125:> git archive --remote=git://git.ceph.com/ceph-ci.git d71e96d216190f28f757f409532464ab0eca4ea9 src/cephadm/cephadm | tar -xO src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm 2021-03-24T21:45:24.028 DEBUG:teuthology.orchestra.run.smithi142:> git archive --remote=git://git.ceph.com/ceph-ci.git d71e96d216190f28f757f409532464ab0eca4ea9 src/cephadm/cephadm | tar -xO src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm 2021-03-24T21:45:24.182 DEBUG:teuthology.orchestra.run.smithi125:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2021-03-24T21:45:24.191 DEBUG:teuthology.orchestra.run.smithi142:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2021-03-24T21:45:24.212 DEBUG:teuthology.orchestra.run.smithi125:> sudo mkdir -p /etc/ceph 2021-03-24T21:45:24.228 DEBUG:teuthology.orchestra.run.smithi142:> sudo mkdir -p /etc/ceph 2021-03-24T21:45:24.244 DEBUG:teuthology.orchestra.run.smithi125:> sudo chmod 777 /etc/ceph 2021-03-24T21:45:24.258 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod 777 /etc/ceph 2021-03-24T21:45:24.301 INFO:tasks.cephadm:Writing seed config... 2021-03-24T21:45:24.302 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2021-03-24T21:45:24.303 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2021-03-24T21:45:24.303 INFO:tasks.cephadm: override: [global] ms bind msgr2 = False 2021-03-24T21:45:24.303 INFO:tasks.cephadm: override: [global] ms type = async 2021-03-24T21:45:24.303 INFO:tasks.cephadm: override: [mon] debug mon = 20 2021-03-24T21:45:24.304 INFO:tasks.cephadm: override: [mon] debug ms = 1 2021-03-24T21:45:24.304 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2021-03-24T21:45:24.304 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2021-03-24T21:45:24.305 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2021-03-24T21:45:24.305 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2021-03-24T21:45:24.305 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2021-03-24T21:45:24.306 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2021-03-24T21:45:24.306 INFO:tasks.cephadm: override: [osd] debug ms = 1 2021-03-24T21:45:24.306 INFO:tasks.cephadm: override: [osd] debug osd = 20 2021-03-24T21:45:24.307 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2021-03-24T21:45:24.307 INFO:tasks.cephadm: override: [osd] osd delete sleep = 1 2021-03-24T21:45:24.307 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2021-03-24T21:45:24.307 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2021-03-24T21:45:24.308 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2021-03-24T21:45:24.308 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2021-03-24T21:45:24.308 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2021-03-24T21:45:24.309 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2021-03-24T21:45:24.309 DEBUG:teuthology.orchestra.run.smithi125:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2021-03-24T21:45:24.317 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 3cfa339a-8cea-11eb-90b2-001a4aab830c ms bind msgr2 = False ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd debug reject backfill probability = 0.3 osd delete sleep = 1 osd max backfills = 3 osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2021-03-24T21:45:24.317 DEBUG:teuthology.orchestra.run.smithi125:mon.a> sudo journalctl -f -n 0 -u ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c@mon.a.service 2021-03-24T21:45:24.320 DEBUG:teuthology.orchestra.run.smithi125:mgr.y> sudo journalctl -f -n 0 -u ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c@mgr.y.service 2021-03-24T21:45:24.357 INFO:journalctl@ceph.mon.a.smithi125.stdout:-- Logs begin at Wed 2021-03-03 23:51:07 UTC. -- 2021-03-24T21:45:24.358 INFO:tasks.cephadm:Bootstrapping... 2021-03-24T21:45:24.359 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 -v bootstrap --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-addrv '[v1:172.21.15.125:6789]' && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2021-03-24T21:45:24.392 INFO:journalctl@ceph.mgr.y.smithi125.stdout:-- Logs begin at Wed 2021-03-03 23:51:07 UTC. -- 2021-03-24T21:45:24.680 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: /usr/bin/podman --version 2021-03-24T21:45:24.735 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/podman: podman version 3.0.1 2021-03-24T21:45:24.738 INFO:teuthology.orchestra.run.smithi125.stderr:Verifying podman|docker is present... 2021-03-24T21:45:24.739 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: /usr/bin/podman --version 2021-03-24T21:45:24.787 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/podman: podman version 3.0.1 2021-03-24T21:45:24.793 INFO:teuthology.orchestra.run.smithi125.stderr:Verifying lvm2 is present... 2021-03-24T21:45:24.793 INFO:teuthology.orchestra.run.smithi125.stderr:Verifying time synchronization is in place... 2021-03-24T21:45:24.793 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: systemctl is-enabled chrony.service 2021-03-24T21:45:24.799 INFO:teuthology.orchestra.run.smithi125.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2021-03-24T21:45:24.799 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: systemctl is-active chrony.service 2021-03-24T21:45:24.804 INFO:teuthology.orchestra.run.smithi125.stderr:systemctl: inactive 2021-03-24T21:45:24.805 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: systemctl is-enabled chronyd.service 2021-03-24T21:45:24.811 INFO:teuthology.orchestra.run.smithi125.stderr:systemctl: Failed to get unit file state for chronyd.service: No such file or directory 2021-03-24T21:45:24.812 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: systemctl is-active chronyd.service 2021-03-24T21:45:24.817 INFO:teuthology.orchestra.run.smithi125.stderr:systemctl: inactive 2021-03-24T21:45:24.818 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: systemctl is-enabled systemd-timesyncd.service 2021-03-24T21:45:24.823 INFO:teuthology.orchestra.run.smithi125.stderr:systemctl: masked 2021-03-24T21:45:24.824 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: systemctl is-active systemd-timesyncd.service 2021-03-24T21:45:24.829 INFO:teuthology.orchestra.run.smithi125.stderr:systemctl: inactive 2021-03-24T21:45:24.830 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: systemctl is-enabled ntpd.service 2021-03-24T21:45:24.836 INFO:teuthology.orchestra.run.smithi125.stderr:systemctl: Failed to get unit file state for ntpd.service: No such file or directory 2021-03-24T21:45:24.837 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: systemctl is-active ntpd.service 2021-03-24T21:45:24.842 INFO:teuthology.orchestra.run.smithi125.stderr:systemctl: inactive 2021-03-24T21:45:24.843 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: systemctl is-enabled ntp.service 2021-03-24T21:45:24.849 INFO:teuthology.orchestra.run.smithi125.stderr:systemctl: enabled 2021-03-24T21:45:24.850 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: systemctl is-active ntp.service 2021-03-24T21:45:24.856 INFO:teuthology.orchestra.run.smithi125.stderr:systemctl: active 2021-03-24T21:45:24.857 INFO:teuthology.orchestra.run.smithi125.stderr:Unit ntp.service is enabled and running 2021-03-24T21:45:24.857 INFO:teuthology.orchestra.run.smithi125.stderr:Repeating the final host check... 2021-03-24T21:45:24.857 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: /usr/bin/podman --version 2021-03-24T21:45:24.910 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/podman: podman version 3.0.1 2021-03-24T21:45:24.913 INFO:teuthology.orchestra.run.smithi125.stderr:podman|docker (/usr/bin/podman) is present 2021-03-24T21:45:24.914 INFO:teuthology.orchestra.run.smithi125.stderr:systemctl is present 2021-03-24T21:45:24.914 INFO:teuthology.orchestra.run.smithi125.stderr:lvcreate is present 2021-03-24T21:45:24.914 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: systemctl is-enabled chrony.service 2021-03-24T21:45:24.920 INFO:teuthology.orchestra.run.smithi125.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2021-03-24T21:45:24.921 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: systemctl is-active chrony.service 2021-03-24T21:45:24.927 INFO:teuthology.orchestra.run.smithi125.stderr:systemctl: inactive 2021-03-24T21:45:24.928 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: systemctl is-enabled chronyd.service 2021-03-24T21:45:24.934 INFO:teuthology.orchestra.run.smithi125.stderr:systemctl: Failed to get unit file state for chronyd.service: No such file or directory 2021-03-24T21:45:24.935 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: systemctl is-active chronyd.service 2021-03-24T21:45:24.940 INFO:teuthology.orchestra.run.smithi125.stderr:systemctl: inactive 2021-03-24T21:45:24.941 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: systemctl is-enabled systemd-timesyncd.service 2021-03-24T21:45:24.946 INFO:teuthology.orchestra.run.smithi125.stderr:systemctl: masked 2021-03-24T21:45:24.947 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: systemctl is-active systemd-timesyncd.service 2021-03-24T21:45:24.952 INFO:teuthology.orchestra.run.smithi125.stderr:systemctl: inactive 2021-03-24T21:45:24.953 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: systemctl is-enabled ntpd.service 2021-03-24T21:45:24.959 INFO:teuthology.orchestra.run.smithi125.stderr:systemctl: Failed to get unit file state for ntpd.service: No such file or directory 2021-03-24T21:45:24.960 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: systemctl is-active ntpd.service 2021-03-24T21:45:24.967 INFO:teuthology.orchestra.run.smithi125.stderr:systemctl: inactive 2021-03-24T21:45:24.967 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: systemctl is-enabled ntp.service 2021-03-24T21:45:24.974 INFO:teuthology.orchestra.run.smithi125.stderr:systemctl: enabled 2021-03-24T21:45:24.975 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: systemctl is-active ntp.service 2021-03-24T21:45:24.980 INFO:teuthology.orchestra.run.smithi125.stderr:systemctl: active 2021-03-24T21:45:24.982 INFO:teuthology.orchestra.run.smithi125.stderr:Unit ntp.service is enabled and running 2021-03-24T21:45:24.982 INFO:teuthology.orchestra.run.smithi125.stderr:Host looks OK 2021-03-24T21:45:24.982 INFO:teuthology.orchestra.run.smithi125.stderr:Cluster fsid: 3cfa339a-8cea-11eb-90b2-001a4aab830c 2021-03-24T21:45:24.983 INFO:teuthology.orchestra.run.smithi125.stderr:Acquiring lock 140164061439264 on /run/cephadm/3cfa339a-8cea-11eb-90b2-001a4aab830c.lock 2021-03-24T21:45:24.983 INFO:teuthology.orchestra.run.smithi125.stderr:Lock 140164061439264 acquired on /run/cephadm/3cfa339a-8cea-11eb-90b2-001a4aab830c.lock 2021-03-24T21:45:24.983 INFO:teuthology.orchestra.run.smithi125.stderr:Verifying IP 172.21.15.125 port 6789 ... 2021-03-24T21:45:24.984 INFO:teuthology.orchestra.run.smithi125.stderr:Base mon IP is 172.21.15.125, final addrv is [v1:172.21.15.125:6789] 2021-03-24T21:45:24.984 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: /usr/sbin/ip route ls 2021-03-24T21:45:24.986 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/sbin/ip: default via 172.21.15.254 dev enp3s0f1 2021-03-24T21:45:24.987 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.125 metric 100 2021-03-24T21:45:24.987 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/sbin/ip: 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 2021-03-24T21:45:24.987 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.125 2021-03-24T21:45:24.987 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/sbin/ip: 172.21.15.254 dev enp3s0f1 proto dhcp scope link src 172.21.15.125 metric 100 2021-03-24T21:45:24.988 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: /usr/sbin/ip -6 route ls 2021-03-24T21:45:24.990 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2021-03-24T21:45:24.990 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2021-03-24T21:45:24.990 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 expires 1791sec pref medium 2021-03-24T21:45:24.991 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: /usr/sbin/ip -6 addr ls 2021-03-24T21:45:24.994 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2021-03-24T21:45:24.994 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/sbin/ip: inet6 ::1/128 scope host 2021-03-24T21:45:24.994 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/sbin/ip: valid_lft forever preferred_lft forever 2021-03-24T21:45:24.995 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2021-03-24T21:45:24.995 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/sbin/ip: inet6 fe80::ec4:7aff:fe8f:d739/64 scope link 2021-03-24T21:45:24.996 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/sbin/ip: valid_lft forever preferred_lft forever 2021-03-24T21:45:24.996 INFO:teuthology.orchestra.run.smithi125.stderr:Mon IP 172.21.15.125 is in CIDR network 172.21.0.0/20 2021-03-24T21:45:24.996 INFO:teuthology.orchestra.run.smithi125.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2021-03-24T21:45:24.997 INFO:teuthology.orchestra.run.smithi125.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9... 2021-03-24T21:45:24.997 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: /usr/bin/podman pull quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 2021-03-24T21:45:26.367 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/podman: Trying to pull quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9... 2021-03-24T21:45:26.492 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/podman: Getting image source signatures 2021-03-24T21:45:26.611 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/podman: Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 2021-03-24T21:45:26.686 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/podman: Copying blob sha256:889524391d554cc9bd269d2e47820d562c512d9d60fd7d4da881803a833083d7 2021-03-24T21:45:26.703 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/podman: Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 2021-03-24T21:45:26.736 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/podman: Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 2021-03-24T21:45:26.753 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/podman: Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 2021-03-24T21:45:26.786 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/podman: Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 2021-03-24T21:45:27.028 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/podman: Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 2021-03-24T21:45:27.094 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/podman: Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 2021-03-24T21:45:27.095 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/podman: Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 2021-03-24T21:45:27.095 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/podman: Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 2021-03-24T21:45:27.096 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/podman: Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 2021-03-24T21:45:27.096 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/podman: Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 2021-03-24T21:45:27.096 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/podman: Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 2021-03-24T21:45:27.097 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/podman: Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 2021-03-24T21:45:27.097 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/podman: Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 2021-03-24T21:45:27.098 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/podman: Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 2021-03-24T21:45:27.178 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/podman: Copying blob sha256:90565abac5c477f1e35a6f41713cf4006285ddf765d2c0ea4e519c74fe322ebd 2021-03-24T21:45:45.409 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/podman: Writing manifest to image destination 2021-03-24T21:45:45.409 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/podman: Storing signatures 2021-03-24T21:45:59.939 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/podman: 0e9b08a1733c8a345676f97943f84cf88f4c80c7e59c594cd5005bc6262f467d 2021-03-24T21:45:59.962 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: /usr/bin/podman run --rm --ipc=host --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 -e NODE_NAME=smithi125 -e CEPH_USE_RANDOM_NONCE=1 quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 --version 2021-03-24T21:46:00.603 INFO:teuthology.orchestra.run.smithi125.stderr:ceph: ceph version 16.1.0-1166-gd71e96d2 (d71e96d216190f28f757f409532464ab0eca4ea9) pacific (rc) 2021-03-24T21:46:11.146 INFO:teuthology.orchestra.run.smithi125.stderr:Ceph version: ceph version 16.1.0-1166-gd71e96d2 (d71e96d216190f28f757f409532464ab0eca4ea9) pacific (rc) 2021-03-24T21:46:11.146 INFO:teuthology.orchestra.run.smithi125.stderr:Extracting ceph user uid/gid from container image... 2021-03-24T21:46:11.147 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: /usr/bin/podman run --rm --ipc=host --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 -e NODE_NAME=smithi125 -e CEPH_USE_RANDOM_NONCE=1 quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 -c %u %g /var/lib/ceph 2021-03-24T21:46:11.546 INFO:teuthology.orchestra.run.smithi125.stderr:stat: 167 167 2021-03-24T21:46:12.204 INFO:teuthology.orchestra.run.smithi125.stderr:Creating initial keys... 2021-03-24T21:46:12.204 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: /usr/bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 -e NODE_NAME=smithi125 -e CEPH_USE_RANDOM_NONCE=1 quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 --gen-print-key 2021-03-24T21:46:12.540 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-authtool: AQAks1tg7gLpHxAAUZipRVr3NT1xsG3J7mEMLA== 2021-03-24T21:46:13.263 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: /usr/bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 -e NODE_NAME=smithi125 -e CEPH_USE_RANDOM_NONCE=1 quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 --gen-print-key 2021-03-24T21:46:13.639 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-authtool: AQAls1tgIerZJRAAXBnao01sRxkcsBj1EqTRkg== 2021-03-24T21:46:14.320 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: /usr/bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 -e NODE_NAME=smithi125 -e CEPH_USE_RANDOM_NONCE=1 quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 --gen-print-key 2021-03-24T21:46:14.687 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-authtool: AQAms1tgonlwJxAAo7TVAtZvODO7mp7clFB22g== 2021-03-24T21:46:15.381 INFO:teuthology.orchestra.run.smithi125.stderr:Creating initial monmap... 2021-03-24T21:46:15.381 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: /usr/bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 -e NODE_NAME=smithi125 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmpzaxjij07:/tmp/monmap:z quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 --create --clobber --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c --addv a [v1:172.21.15.125:6789] /tmp/monmap 2021-03-24T21:46:15.748 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2021-03-24T21:46:15.748 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 3cfa339a-8cea-11eb-90b2-001a4aab830c 2021-03-24T21:46:15.748 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2021-03-24T21:46:16.503 INFO:teuthology.orchestra.run.smithi125.stderr:monmaptool for a [v1:172.21.15.125:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2021-03-24T21:46:16.503 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/monmaptool: set fsid to 3cfa339a-8cea-11eb-90b2-001a4aab830c 2021-03-24T21:46:16.503 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2021-03-24T21:46:16.504 INFO:teuthology.orchestra.run.smithi125.stderr: 2021-03-24T21:46:16.504 INFO:teuthology.orchestra.run.smithi125.stderr:Creating mon... 2021-03-24T21:46:16.504 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: /usr/bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 -e NODE_NAME=smithi125 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c:/var/log/ceph:z -v /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a:/var/lib/ceph/mon/ceph-a:z -v /tmp/ceph-tmps0xcpvgx:/tmp/keyring:z -v /tmp/ceph-tmpzaxjij07:/tmp/monmap:z quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 --mkfs -i a --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2021-03-24T21:46:16.886 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 0 set uid:gid to 167:167 (ceph:ceph) 2021-03-24T21:46:16.916 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 1 imported monmap: 2021-03-24T21:46:16.917 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: epoch 0 2021-03-24T21:46:16.917 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c 2021-03-24T21:46:16.917 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: last_changed 2021-03-24T21:46:15.748211+0000 2021-03-24T21:46:16.918 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: created 2021-03-24T21:46:15.748211+0000 2021-03-24T21:46:16.918 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2021-03-24T21:46:16.918 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: election_strategy: 1 2021-03-24T21:46:16.918 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: 0: v1:172.21.15.125:6789/0 mon.a 2021-03-24T21:46:16.918 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: 2021-03-24T21:46:16.919 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 0 /usr/bin/ceph-mon: set fsid to 3cfa339a-8cea-11eb-90b2-001a4aab830c 2021-03-24T21:46:16.919 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: RocksDB version: 6.8.1 2021-03-24T21:46:16.919 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: 2021-03-24T21:46:16.919 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2021-03-24T21:46:16.920 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Compile date Mar 24 2021 2021-03-24T21:46:16.920 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: DB SUMMARY 2021-03-24T21:46:16.920 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: 2021-03-24T21:46:16.920 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 0, files: 2021-03-24T21:46:16.921 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: 2021-03-24T21:46:16.921 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 2021-03-24T21:46:16.921 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: 2021-03-24T21:46:16.921 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.error_if_exists: 0 2021-03-24T21:46:16.922 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.create_if_missing: 1 2021-03-24T21:46:16.922 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.paranoid_checks: 1 2021-03-24T21:46:16.922 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.env: 0x55a8481631c0 2021-03-24T21:46:16.922 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.fs: Posix File System 2021-03-24T21:46:16.923 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.info_log: 0x55a84a156b60 2021-03-24T21:46:16.923 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.max_file_opening_threads: 16 2021-03-24T21:46:16.923 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.statistics: (nil) 2021-03-24T21:46:16.923 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.use_fsync: 0 2021-03-24T21:46:16.924 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.max_log_file_size: 0 2021-03-24T21:46:16.924 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2021-03-24T21:46:16.924 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.log_file_time_to_roll: 0 2021-03-24T21:46:16.924 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.keep_log_file_num: 1000 2021-03-24T21:46:16.925 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.recycle_log_file_num: 0 2021-03-24T21:46:16.925 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.allow_fallocate: 1 2021-03-24T21:46:16.925 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.allow_mmap_reads: 0 2021-03-24T21:46:16.925 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.allow_mmap_writes: 0 2021-03-24T21:46:16.925 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.use_direct_reads: 0 2021-03-24T21:46:16.926 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2021-03-24T21:46:16.926 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.create_missing_column_families: 0 2021-03-24T21:46:16.926 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.db_log_dir: 2021-03-24T21:46:16.926 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2021-03-24T21:46:16.927 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.table_cache_numshardbits: 6 2021-03-24T21:46:16.927 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.max_subcompactions: 1 2021-03-24T21:46:16.927 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.max_background_flushes: -1 2021-03-24T21:46:16.927 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.WAL_ttl_seconds: 0 2021-03-24T21:46:16.928 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.WAL_size_limit_MB: 0 2021-03-24T21:46:16.928 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2021-03-24T21:46:16.928 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2021-03-24T21:46:16.928 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.is_fd_close_on_exec: 1 2021-03-24T21:46:16.928 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.advise_random_on_open: 1 2021-03-24T21:46:16.929 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.db_write_buffer_size: 0 2021-03-24T21:46:16.929 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.write_buffer_manager: 0x55a84a1612f0 2021-03-24T21:46:16.929 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2021-03-24T21:46:16.929 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2021-03-24T21:46:16.930 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2021-03-24T21:46:16.930 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.use_adaptive_mutex: 0 2021-03-24T21:46:16.930 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.rate_limiter: (nil) 2021-03-24T21:46:16.930 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2021-03-24T21:46:16.930 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.wal_recovery_mode: 2 2021-03-24T21:46:16.931 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.enable_thread_tracking: 0 2021-03-24T21:46:16.931 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.enable_pipelined_write: 0 2021-03-24T21:46:16.931 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.unordered_write: 0 2021-03-24T21:46:16.931 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2021-03-24T21:46:16.932 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2021-03-24T21:46:16.932 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2021-03-24T21:46:16.932 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2021-03-24T21:46:16.932 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.row_cache: None 2021-03-24T21:46:16.932 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.wal_filter: None 2021-03-24T21:46:16.933 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2021-03-24T21:46:16.933 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.allow_ingest_behind: 0 2021-03-24T21:46:16.933 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.preserve_deletes: 0 2021-03-24T21:46:16.933 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.two_write_queues: 0 2021-03-24T21:46:16.933 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.manual_wal_flush: 0 2021-03-24T21:46:16.934 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.atomic_flush: 0 2021-03-24T21:46:16.934 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2021-03-24T21:46:16.934 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.persist_stats_to_disk: 0 2021-03-24T21:46:16.934 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.write_dbid_to_manifest: 0 2021-03-24T21:46:16.935 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.log_readahead_size: 0 2021-03-24T21:46:16.935 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2021-03-24T21:46:16.935 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.max_background_jobs: 2 2021-03-24T21:46:16.935 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.max_background_compactions: -1 2021-03-24T21:46:16.935 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2021-03-24T21:46:16.936 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2021-03-24T21:46:16.936 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.delayed_write_rate : 16777216 2021-03-24T21:46:16.936 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.max_total_wal_size: 0 2021-03-24T21:46:16.936 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2021-03-24T21:46:16.937 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.stats_dump_period_sec: 600 2021-03-24T21:46:16.937 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.stats_persist_period_sec: 600 2021-03-24T21:46:16.937 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2021-03-24T21:46:16.937 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.max_open_files: -1 2021-03-24T21:46:16.937 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.bytes_per_sync: 0 2021-03-24T21:46:16.938 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.wal_bytes_per_sync: 0 2021-03-24T21:46:16.938 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.strict_bytes_per_sync: 0 2021-03-24T21:46:16.938 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Options.compaction_readahead_size: 0 2021-03-24T21:46:16.938 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Compression algorithms supported: 2021-03-24T21:46:16.939 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2021-03-24T21:46:16.939 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: kZSTD supported: 0 2021-03-24T21:46:16.939 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: kXpressCompression supported: 0 2021-03-24T21:46:16.939 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: kLZ4HCCompression supported: 1 2021-03-24T21:46:16.940 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: kLZ4Compression supported: 1 2021-03-24T21:46:16.940 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: kBZip2Compression supported: 0 2021-03-24T21:46:16.940 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: kZlibCompression supported: 1 2021-03-24T21:46:16.940 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: kSnappyCompression supported: 1 2021-03-24T21:46:16.941 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: Fast CRC32 supported: Supported on x86 2021-03-24T21:46:16.941 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.884+0000 7fd28f93d700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2021-03-24T21:46:16.941 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: 2021-03-24T21:46:16.941 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 2021-03-24T21:46:16.941 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: 2021-03-24T21:46:16.942 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2021-03-24T21:46:16.942 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: 2021-03-24T21:46:16.942 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2021-03-24T21:46:16.942 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.merge_operator: 2021-03-24T21:46:16.943 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.compaction_filter: None 2021-03-24T21:46:16.943 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.compaction_filter_factory: None 2021-03-24T21:46:16.943 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.memtable_factory: SkipListFactory 2021-03-24T21:46:16.943 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.table_factory: BlockBasedTable 2021-03-24T21:46:16.943 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55a84a06c210) 2021-03-24T21:46:16.944 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2021-03-24T21:46:16.944 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2021-03-24T21:46:16.944 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2021-03-24T21:46:16.944 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2021-03-24T21:46:16.944 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: index_type: 0 2021-03-24T21:46:16.945 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2021-03-24T21:46:16.945 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: index_shortening: 1 2021-03-24T21:46:16.945 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2021-03-24T21:46:16.946 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2021-03-24T21:46:16.946 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: checksum: 1 2021-03-24T21:46:16.946 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2021-03-24T21:46:16.946 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: block_cache: 0x55a84a0a4d10 2021-03-24T21:46:16.946 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2021-03-24T21:46:16.947 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: block_cache_options: 2021-03-24T21:46:16.947 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: capacity : 536870912 2021-03-24T21:46:16.947 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2021-03-24T21:46:16.947 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2021-03-24T21:46:16.947 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2021-03-24T21:46:16.948 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2021-03-24T21:46:16.948 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2021-03-24T21:46:16.948 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: block_size: 4096 2021-03-24T21:46:16.948 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2021-03-24T21:46:16.948 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2021-03-24T21:46:16.949 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2021-03-24T21:46:16.949 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2021-03-24T21:46:16.949 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: partition_filters: 0 2021-03-24T21:46:16.949 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2021-03-24T21:46:16.949 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2021-03-24T21:46:16.950 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2021-03-24T21:46:16.950 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: verify_compression: 0 2021-03-24T21:46:16.950 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2021-03-24T21:46:16.950 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: format_version: 2 2021-03-24T21:46:16.951 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2021-03-24T21:46:16.951 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: block_align: 0 2021-03-24T21:46:16.951 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: 2021-03-24T21:46:16.951 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.write_buffer_size: 33554432 2021-03-24T21:46:16.951 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.max_write_buffer_number: 2 2021-03-24T21:46:16.952 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.compression: NoCompression 2021-03-24T21:46:16.952 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.bottommost_compression: Disabled 2021-03-24T21:46:16.952 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.prefix_extractor: nullptr 2021-03-24T21:46:16.952 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2021-03-24T21:46:16.952 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.num_levels: 7 2021-03-24T21:46:16.953 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2021-03-24T21:46:16.953 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2021-03-24T21:46:16.953 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2021-03-24T21:46:16.953 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2021-03-24T21:46:16.953 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2021-03-24T21:46:16.954 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2021-03-24T21:46:16.954 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2021-03-24T21:46:16.954 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2021-03-24T21:46:16.954 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2021-03-24T21:46:16.955 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.compression_opts.window_bits: -14 2021-03-24T21:46:16.955 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.compression_opts.level: 32767 2021-03-24T21:46:16.955 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.compression_opts.strategy: 0 2021-03-24T21:46:16.955 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2021-03-24T21:46:16.956 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2021-03-24T21:46:16.956 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.compression_opts.enabled: false 2021-03-24T21:46:16.956 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2021-03-24T21:46:16.957 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2021-03-24T21:46:16.957 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2021-03-24T21:46:16.957 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.target_file_size_base: 67108864 2021-03-24T21:46:16.957 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.target_file_size_multiplier: 1 2021-03-24T21:46:16.958 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2021-03-24T21:46:16.958 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2021-03-24T21:46:16.958 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2021-03-24T21:46:16.958 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2021-03-24T21:46:16.959 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2021-03-24T21:46:16.959 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2021-03-24T21:46:16.959 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2021-03-24T21:46:16.959 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2021-03-24T21:46:16.959 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2021-03-24T21:46:16.960 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2021-03-24T21:46:16.960 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2021-03-24T21:46:16.960 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2021-03-24T21:46:16.960 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.arena_block_size: 4194304 2021-03-24T21:46:16.960 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2021-03-24T21:46:16.961 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2021-03-24T21:46:16.961 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2021-03-24T21:46:16.961 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.disable_auto_compactions: 0 2021-03-24T21:46:16.961 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2021-03-24T21:46:16.962 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2021-03-24T21:46:16.962 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2021-03-24T21:46:16.962 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2021-03-24T21:46:16.962 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2021-03-24T21:46:16.962 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2021-03-24T21:46:16.963 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2021-03-24T21:46:16.963 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2021-03-24T21:46:16.963 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2021-03-24T21:46:16.963 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2021-03-24T21:46:16.963 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.table_properties_collectors: 2021-03-24T21:46:16.964 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.inplace_update_support: 0 2021-03-24T21:46:16.964 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.inplace_update_num_locks: 10000 2021-03-24T21:46:16.964 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2021-03-24T21:46:16.964 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2021-03-24T21:46:16.965 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.memtable_huge_page_size: 0 2021-03-24T21:46:16.965 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.bloom_locality: 0 2021-03-24T21:46:16.965 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.max_successive_merges: 0 2021-03-24T21:46:16.966 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.optimize_filters_for_hits: 0 2021-03-24T21:46:16.966 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.paranoid_file_checks: 0 2021-03-24T21:46:16.966 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.force_consistency_checks: 0 2021-03-24T21:46:16.966 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.report_bg_io_stats: 0 2021-03-24T21:46:16.966 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.ttl: 2592000 2021-03-24T21:46:16.967 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: Options.periodic_compaction_seconds: 0 2021-03-24T21:46:16.967 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2021-03-24T21:46:16.967 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: 2021-03-24T21:46:16.967 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2021-03-24T21:46:16.968 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: 2021-03-24T21:46:16.968 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd28f93d700 4 rocksdb: DB pointer 0x55a84a16b800 2021-03-24T21:46:16.968 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd2787db700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2021-03-24T21:46:16.968 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.888+0000 7fd2787db700 4 rocksdb: [db_impl/db_impl.cc:851] 2021-03-24T21:46:16.968 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2021-03-24T21:46:16.969 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2021-03-24T21:46:16.969 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2021-03-24T21:46:16.969 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2021-03-24T21:46:16.969 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2021-03-24T21:46:16.969 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2021-03-24T21:46:16.970 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2021-03-24T21:46:16.970 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2021-03-24T21:46:16.970 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: 2021-03-24T21:46:16.970 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2021-03-24T21:46:16.970 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-24T21:46:16.971 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-24T21:46:16.971 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-03-24T21:46:16.971 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-03-24T21:46:16.971 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: 2021-03-24T21:46:16.971 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2021-03-24T21:46:16.972 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-24T21:46:16.972 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-24T21:46:16.972 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2021-03-24T21:46:16.972 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2021-03-24T21:46:16.972 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2021-03-24T21:46:16.973 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2021-03-24T21:46:16.973 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2021-03-24T21:46:16.973 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2021-03-24T21:46:16.973 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-24T21:46:16.973 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-24T21:46:16.974 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-03-24T21:46:16.974 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: 2021-03-24T21:46:16.974 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2021-03-24T21:46:16.974 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: 2021-03-24T21:46:16.975 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2021-03-24T21:46:16.975 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-24T21:46:16.975 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-24T21:46:16.975 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-03-24T21:46:16.975 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-03-24T21:46:16.976 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: 2021-03-24T21:46:16.976 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2021-03-24T21:46:16.976 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-24T21:46:16.976 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-24T21:46:16.977 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2021-03-24T21:46:16.977 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2021-03-24T21:46:16.977 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2021-03-24T21:46:16.977 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2021-03-24T21:46:16.977 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2021-03-24T21:46:16.978 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2021-03-24T21:46:16.978 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-24T21:46:16.978 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-24T21:46:16.978 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-03-24T21:46:16.978 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: 2021-03-24T21:46:16.979 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2021-03-24T21:46:16.979 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: 2021-03-24T21:46:16.979 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.892+0000 7fd28f93d700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2021-03-24T21:46:16.979 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.892+0000 7fd28f93d700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2021-03-24T21:46:16.980 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph-mon: debug 2021-03-24T21:46:16.892+0000 7fd28f93d700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-a for mon.a 2021-03-24T21:46:17.553 INFO:teuthology.orchestra.run.smithi125.stderr:create mon.a on 2021-03-24T21:46:17.553 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c 2021-03-24T21:46:17.560 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: systemctl disable ceph.target 2021-03-24T21:46:17.572 INFO:teuthology.orchestra.run.smithi125.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2021-03-24T21:46:17.768 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: systemctl enable ceph.target 2021-03-24T21:46:17.775 INFO:teuthology.orchestra.run.smithi125.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /etc/systemd/system/ceph.target. 2021-03-24T21:46:17.941 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: systemctl start ceph.target 2021-03-24T21:46:17.950 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: systemctl enable ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c.target 2021-03-24T21:46:17.957 INFO:teuthology.orchestra.run.smithi125.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c.target -> /etc/systemd/system/ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c.target. 2021-03-24T21:46:17.957 INFO:teuthology.orchestra.run.smithi125.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c.target -> /etc/systemd/system/ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c.target. 2021-03-24T21:46:18.131 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: systemctl start ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c.target 2021-03-24T21:46:18.140 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: systemctl daemon-reload 2021-03-24T21:46:18.353 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: systemctl stop ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c@mon.a 2021-03-24T21:46:18.366 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: systemctl reset-failed ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c@mon.a 2021-03-24T21:46:18.371 INFO:teuthology.orchestra.run.smithi125.stderr:systemctl: Failed to reset failed state of unit ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c@mon.a.service: Unit ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c@mon.a.service not loaded. 2021-03-24T21:46:18.371 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: systemctl enable ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c@mon.a 2021-03-24T21:46:18.378 INFO:teuthology.orchestra.run.smithi125.stderr:systemctl: Created symlink /etc/systemd/system/ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c.target.wants/ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c@mon.a.service -> /etc/systemd/system/ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c@.service. 2021-03-24T21:46:18.573 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: systemctl start ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c@mon.a 2021-03-24T21:46:18.905 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:18 smithi125 systemd[1]: Starting Ceph mon.a for 3cfa339a-8cea-11eb-90b2-001a4aab830c... 2021-03-24T21:46:19.180 INFO:teuthology.orchestra.run.smithi125.stderr:firewalld does not appear to be present 2021-03-24T21:46:19.180 INFO:teuthology.orchestra.run.smithi125.stderr:Not possible to enable service . firewalld.service is not available 2021-03-24T21:46:19.180 INFO:teuthology.orchestra.run.smithi125.stderr:Not possible to open ports <[]>. firewalld.service is not available 2021-03-24T21:46:19.181 INFO:teuthology.orchestra.run.smithi125.stderr:Waiting for mon to start... 2021-03-24T21:46:19.181 INFO:teuthology.orchestra.run.smithi125.stderr:Waiting for mon... 2021-03-24T21:46:19.181 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: /usr/bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 -e NODE_NAME=smithi125 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a:/var/lib/ceph/mon/ceph-a:z -v /tmp/ceph-tmp9x0iyaqg:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp51umemfe:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 status 2021-03-24T21:46:19.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:18 smithi125 podman[19271]: 2021-03-24 21:46:18.959363274 +0000 UTC m=+0.187830232 container create 0e322dbc5182a9ba164ded61f2fd01a5c416754cf9e92ae51da566c66d242eb5 (image=quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-mon.a, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20201204, maintainer=Dimitri Savineau , org.label-schema.license=GPLv2, GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06, ceph=True, RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d, io.buildah.version=1.18.0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Base Image, GIT_BRANCH=HEAD) 2021-03-24T21:46:19.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:19 smithi125 podman[19271]: 2021-03-24 21:46:19.076020166 +0000 UTC m=+0.304487131 container init 0e322dbc5182a9ba164ded61f2fd01a5c416754cf9e92ae51da566c66d242eb5 (image=quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-mon.a, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.18.0, maintainer=Dimitri Savineau , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d, org.label-schema.build-date=20201204, org.label-schema.name=CentOS Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2021-03-24T21:46:19.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:19 smithi125 podman[19271]: 2021-03-24 21:46:19.16762355 +0000 UTC m=+0.396090506 container start 0e322dbc5182a9ba164ded61f2fd01a5c416754cf9e92ae51da566c66d242eb5 (image=quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-mon.a, GIT_CLEAN=True, RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d, org.label-schema.build-date=20201204, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Base Image, org.label-schema.license=GPLv2, ceph=True, maintainer=Dimitri Savineau , GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06, io.buildah.version=1.18.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2021-03-24T21:46:19.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:19 smithi125 bash[19271]: 0e322dbc5182a9ba164ded61f2fd01a5c416754cf9e92ae51da566c66d242eb5 2021-03-24T21:46:19.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:19 smithi125 conmon[19340]: cluster 2021-03-24T21:46:19.163600+0000 mon.a (mon. 2021-03-24T21:46:19.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:19 smithi125 conmon[19340]: 0) 0 : [INF] mkfs 3cfa339a-8cea-11eb-90b2-001a4aab830c 2021-03-24T21:46:19.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:19 smithi125 conmon[19340]: cluster 2021-03-24T21:46:19.149305+0000 mon.a (mon.0 2021-03-24T21:46:19.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:19 smithi125 conmon[19340]: ) 1 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2021-03-24T21:46:19.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:19 smithi125 systemd[1]: Started Ceph mon.a for 3cfa339a-8cea-11eb-90b2-001a4aab830c. 2021-03-24T21:46:19.784 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: cluster: 2021-03-24T21:46:19.785 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: id: 3cfa339a-8cea-11eb-90b2-001a4aab830c 2021-03-24T21:46:19.785 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: health: HEALTH_OK 2021-03-24T21:46:19.785 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: 2021-03-24T21:46:19.786 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: services: 2021-03-24T21:46:19.786 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 0.626036s) 2021-03-24T21:46:19.786 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: mgr: no daemons active 2021-03-24T21:46:19.786 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2021-03-24T21:46:19.787 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: 2021-03-24T21:46:19.787 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: data: 2021-03-24T21:46:19.787 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2021-03-24T21:46:19.787 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2021-03-24T21:46:19.787 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2021-03-24T21:46:19.788 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: pgs: 2021-03-24T21:46:19.788 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: 2021-03-24T21:46:20.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:20 smithi125 conmon[19340]: cluster 2021-03-24T21:46:19.160666+0000 mon.a (mon.0) 2 : cluster 2021-03-24T21:46:20.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:20 smithi125 conmon[19340]: [INF] mon.a is new leader, mons a in quorum (ranks 0) 2021-03-24T21:46:20.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:20 smithi125 conmon[19340]: cluster 2021-03-24T21:46:19.162380+0000 mon.a ( 2021-03-24T21:46:20.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:20 smithi125 conmon[19340]: mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {a=v1:172.21.15.125:6789/0} 2021-03-24T21:46:20.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:20 smithi125 conmon[19340]: cluster 2021-03- 2021-03-24T21:46:20.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:20 smithi125 conmon[19340]: 24T21:46:19.167343 2021-03-24T21:46:20.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:20 smithi125 conmon[19340]: +0000 mon.a (mon.0) 4 : cluster 2021-03-24T21:46:20.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:20 smithi125 conmon[19340]: [DBG] fsmap 2021-03-24T21:46:20.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:20 smithi125 conmon[19340]: cluster 2021-03-24 2021-03-24T21:46:20.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:20 smithi125 conmon[19340]: T21:46:19.172405+0000 2021-03-24T21:46:20.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:20 smithi125 conmon[19340]: mon.a (mon.0) 5 : cluster [DBG] 2021-03-24T21:46:20.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:20 smithi125 conmon[19340]: osdmap e1: 0 total, 0 up, 0 in 2021-03-24T21:46:20.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:20 smithi125 conmon[19340]: cluster 2021-03-24T21:46:20.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:20 smithi125 conmon[19340]: 2021-03-24T21:46 2021-03-24T21:46:20.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:20 smithi125 conmon[19340]: :19.173098+0000 mon.a (mon.0) 2021-03-24T21:46:20.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:20 smithi125 conmon[19340]: 6 : cluster [DBG] mgrmap e1: no daemons active 2021-03-24T21:46:20.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:20 smithi125 conmon[19340]: audit 2021 2021-03-24T21:46:20.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:20 smithi125 conmon[19340]: -03-24T21:46 2021-03-24T21:46:20.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:20 smithi125 conmon[19340]: :19.786660+0000 mon.a (mon.0) 2021-03-24T21:46:20.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:20 smithi125 conmon[19340]: 7 : audit [DBG] from='client.? v1:172.21.15.125:0/2685165013' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2021-03-24T21:46:20.761 INFO:teuthology.orchestra.run.smithi125.stderr:mon is available 2021-03-24T21:46:20.761 INFO:teuthology.orchestra.run.smithi125.stderr:Assimilating anything we can from ceph.conf... 2021-03-24T21:46:20.761 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: /usr/bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 -e NODE_NAME=smithi125 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp9x0iyaqg:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp51umemfe:/etc/ceph/ceph.conf:z -v /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a:/var/lib/ceph/mon/ceph-a:z quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 config assimilate-conf -i /var/lib/ceph/mon/ceph-a/config 2021-03-24T21:46:21.369 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: 2021-03-24T21:46:21.369 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: [global] 2021-03-24T21:46:21.370 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: fsid = 3cfa339a-8cea-11eb-90b2-001a4aab830c 2021-03-24T21:46:21.370 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: mon_host = [v1:172.21.15.125:6789] 2021-03-24T21:46:21.370 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2021-03-24T21:46:21.370 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2021-03-24T21:46:21.371 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2021-03-24T21:46:21.371 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2021-03-24T21:46:21.371 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: 2021-03-24T21:46:21.371 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: [mgr] 2021-03-24T21:46:21.371 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2021-03-24T21:46:21.372 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: 2021-03-24T21:46:21.372 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: [osd] 2021-03-24T21:46:21.372 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2021-03-24T21:46:21.372 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2021-03-24T21:46:21.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:21 smithi125 conmon[19340]: audit 2021-03-24T21:46:21.366915+0000 mon.a (mon.0) 8 : audit 2021-03-24T21:46:21.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:21 smithi125 conmon[19340]: [INF] from='client.? v1:172.21.15.125:0/1481967656' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2021-03-24T21:46:21.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:21 smithi125 conmon[19340]: audit 2021-03-24T21:46:21.371634+0000 2021-03-24T21:46:21.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:21 smithi125 conmon[19340]: mon.a (mon.0) 9 : audit [INF] from='client.? v1:172.21.15.125:0/1481967656' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2021-03-24T21:46:22.077 INFO:teuthology.orchestra.run.smithi125.stderr:Generating new minimal ceph.conf... 2021-03-24T21:46:22.077 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: /usr/bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 -e NODE_NAME=smithi125 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp9x0iyaqg:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp51umemfe:/etc/ceph/ceph.conf:z -v /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a:/var/lib/ceph/mon/ceph-a:z quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-a/config 2021-03-24T21:46:22.905 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:22 smithi125 conmon[19340]: audit 2021-03-24 2021-03-24T21:46:22.905 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:22 smithi125 conmon[19340]: T21:46:22.629058+0000 mon.a (mon.0) 10 : audit [DBG] from='client.? v1:172.21.15.125:0/2525651213' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:46:23.302 INFO:teuthology.orchestra.run.smithi125.stderr:Restarting the monitor... 2021-03-24T21:46:23.302 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: systemctl restart ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c@mon.a 2021-03-24T21:46:23.572 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:23 smithi125 systemd[1]: Stopping Ceph mon.a for 3cfa339a-8cea-11eb-90b2-001a4aab830c... 2021-03-24T21:46:23.573 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:23 smithi125 conmon[19340]: debug 2021-03-24T21:46:23.424+0000 7f0d0f836700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2021-03-24T21:46:23.573 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:23 smithi125 conmon[19340]: debug 2021-03-24T21:46:23.573 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:23 smithi125 conmon[19340]: 2021-03-24T21:46:23.424+0000 7f0d0f836700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2021-03-24T21:46:23.832 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:23 smithi125 podman[19732]: 2021-03-24 21:46:23.632197203 +0000 UTC m=+0.298860000 container stop 0e322dbc5182a9ba164ded61f2fd01a5c416754cf9e92ae51da566c66d242eb5 (image=quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-mon.a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Dimitri Savineau , ceph=True, org.label-schema.build-date=20201204, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Base Image, io.buildah.version=1.18.0, GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2021-03-24T21:46:23.832 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:23 smithi125 podman[19732]: 2021-03-24 21:46:23.696636745 +0000 UTC m=+0.363299496 container died 0e322dbc5182a9ba164ded61f2fd01a5c416754cf9e92ae51da566c66d242eb5 (image=quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-mon.a) 2021-03-24T21:46:23.833 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:23 smithi125 podman[19732]: 0e322dbc5182a9ba164ded61f2fd01a5c416754cf9e92ae51da566c66d242eb5 2021-03-24T21:46:24.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 podman[19773]: 2021-03-24 21:46:24.275193203 +0000 UTC m=+0.680902949 container remove 0e322dbc5182a9ba164ded61f2fd01a5c416754cf9e92ae51da566c66d242eb5 (image=quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-mon.a, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.18.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Base Image, RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d, maintainer=Dimitri Savineau , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20201204, GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06, org.label-schema.license=GPLv2) 2021-03-24T21:46:24.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 systemd[1]: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c@mon.a.service: Succeeded. 2021-03-24T21:46:24.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 systemd[1]: Stopped Ceph mon.a for 3cfa339a-8cea-11eb-90b2-001a4aab830c. 2021-03-24T21:46:24.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 systemd[1]: Starting Ceph mon.a for 3cfa339a-8cea-11eb-90b2-001a4aab830c... 2021-03-24T21:46:24.884 INFO:teuthology.orchestra.run.smithi125.stderr:Setting mon public_network to 172.21.0.0/20 2021-03-24T21:46:24.884 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: /usr/bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 -e NODE_NAME=smithi125 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp9x0iyaqg:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp51umemfe:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 config set mon public_network 172.21.0.0/20 2021-03-24T21:46:24.971 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 podman[19896]: 2021-03-24 21:46:24.683499191 +0000 UTC m=+0.193632016 container create a5962ed33b2cca9c92b2a549a7a165919460c89770f3afaa739e6f7a7e3e25e4 (image=quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-mon.a, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Dimitri Savineau , RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d, GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.18.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20201204, org.label-schema.name=CentOS Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2021-03-24T21:46:24.971 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 podman[19896]: 2021-03-24 21:46:24.841822996 +0000 UTC m=+0.351955818 container init a5962ed33b2cca9c92b2a549a7a165919460c89770f3afaa739e6f7a7e3e25e4 (image=quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-mon.a, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.18.0, maintainer=Dimitri Savineau , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20201204, ceph=True, GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06, org.label-schema.name=CentOS Base Image) 2021-03-24T21:46:24.971 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 podman[19896]: 2021-03-24 21:46:24.875084138 +0000 UTC m=+0.385216944 container start a5962ed33b2cca9c92b2a549a7a165919460c89770f3afaa739e6f7a7e3e25e4 (image=quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-mon.a, RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d, org.label-schema.license=GPLv2, org.label-schema.build-date=20201204, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Base Image, maintainer=Dimitri Savineau , org.label-schema.schema-version=1.0, GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.18.0) 2021-03-24T21:46:24.972 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 bash[19896]: a5962ed33b2cca9c92b2a549a7a165919460c89770f3afaa739e6f7a7e3e25e4 2021-03-24T21:46:24.972 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.872+0000 7f212eb06700 0 set uid:gid to 167:167 (ceph:ceph) 2021-03-24T21:46:24.972 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.872+0000 7f212eb06700 0 ceph version 16.1.0-1166-gd71e96d2 (d71e96d216190f28f757f409532464ab0eca4ea9) pacific (rc), process ceph-mon, pid 2 2021-03-24T21:46:24.972 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.872+0000 7f212eb06700 0 pidfile_write: ignore empty --pid-file 2021-03-24T21:46:24.973 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 0 load: jerasure load: lrc load: isa 2021-03-24T21:46:24.973 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 systemd[1]: Started Ceph mon.a for 3cfa339a-8cea-11eb-90b2-001a4aab830c. 2021-03-24T21:46:24.973 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: RocksDB version: 6.8.1 2021-03-24T21:46:24.973 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: 2021-03-24T21:46:24.974 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2021-03-24T21:46:24.974 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Compile date Mar 24 2021 2021-03-24T21:46:24.974 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: DB SUMMARY 2021-03-24T21:46:24.974 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: 2021-03-24T21:46:24.974 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: CURRENT file: CURRENT 2021-03-24T21:46:24.975 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: 2021-03-24T21:46:24.975 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: IDENTITY file: IDENTITY 2021-03-24T21:46:24.975 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: 2021-03-24T21:46:24.977 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2021-03-24T21:46:24.977 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: 2021-03-24T21:46:24.978 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000004.sst 2021-03-24T21:46:24.978 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: 2021-03-24T21:46:24.978 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000006.log size: 85157 ; 2021-03-24T21:46:24.978 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: 2021-03-24T21:46:24.979 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.error_if_exists: 0 2021-03-24T21:46:24.979 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.create_if_missing: 0 2021-03-24T21:46:24.979 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.paranoid_checks: 1 2021-03-24T21:46:24.979 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.env: 0x559a4f36f1c0 2021-03-24T21:46:24.980 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.fs: Posix File System 2021-03-24T21:46:24.980 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.info_log: 0x559a50c30700 2021-03-24T21:46:24.980 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.max_file_opening_threads: 16 2021-03-24T21:46:24.980 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.980 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.statistics: (nil) 2021-03-24T21:46:24.981 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.use_fsync: 0 2021-03-24T21:46:24.981 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.max_log_file_size: 0 2021-03-24T21:46:24.981 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2021-03-24T21:46:24.981 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.log_file_time_to_roll: 0 2021-03-24T21:46:24.982 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.keep_log_file_num: 1000 2021-03-24T21:46:24.982 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.recycle_log_file_num: 0 2021-03-24T21:46:24.982 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.allow_fallocate: 1 2021-03-24T21:46:24.982 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.allow_mmap_reads: 0 2021-03-24T21:46:24.983 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.allow_mmap_writes: 0 2021-03-24T21:46:24.983 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.use_direct_reads: 0 2021-03-24T21:46:24.983 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2021-03-24T21:46:24.983 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.create_missing_column_families: 0 2021-03-24T21:46:24.984 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.db_log_dir: 2021-03-24T21:46:24.984 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2021-03-24T21:46:24.984 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.table_cache_numshardbits: 6 2021-03-24T21:46:24.984 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.max_subcompactions: 1 2021-03-24T21:46:24.984 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.max_background_flushes: -1 2021-03-24T21:46:24.985 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.WAL_ttl_seconds: 0 2021-03-24T21:46:24.985 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.WAL_size_limit_MB: 0 2021-03-24T21:46:24.985 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2021-03-24T21:46:24.985 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2021-03-24T21:46:24.986 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.is_fd_close_on_exec: 1 2021-03-24T21:46:24.986 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.advise_random_on_open: 1 2021-03-24T21:46:24.986 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.db_write_buffer_size: 0 2021-03-24T21:46:24.986 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.write_buffer_manager: 0x559a50cc9ce0 2021-03-24T21:46:24.986 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2021-03-24T21:46:24.987 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2021-03-24T21:46:24.987 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2021-03-24T21:46:24.987 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.use_adaptive_mutex: 0 2021-03-24T21:46:24.987 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.rate_limiter: (nil) 2021-03-24T21:46:24.988 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2021-03-24T21:46:24.988 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.wal_recovery_mode: 2 2021-03-24T21:46:24.988 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.enable_thread_tracking: 0 2021-03-24T21:46:24.988 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.enable_pipelined_write: 0 2021-03-24T21:46:24.988 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.unordered_write: 0 2021-03-24T21:46:24.989 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2021-03-24T21:46:24.989 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2021-03-24T21:46:24.989 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2021-03-24T21:46:24.989 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2021-03-24T21:46:24.990 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.row_cache: None 2021-03-24T21:46:24.990 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.wal_filter: None 2021-03-24T21:46:24.990 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2021-03-24T21:46:24.990 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.allow_ingest_behind: 0 2021-03-24T21:46:24.991 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.preserve_deletes: 0 2021-03-24T21:46:24.991 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.two_write_queues: 0 2021-03-24T21:46:24.991 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.manual_wal_flush: 0 2021-03-24T21:46:24.991 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.atomic_flush: 0 2021-03-24T21:46:24.991 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2021-03-24T21:46:24.992 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.persist_stats_to_disk: 0 2021-03-24T21:46:24.992 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.write_dbid_to_manifest: 0 2021-03-24T21:46:24.992 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.log_readahead_size: 0 2021-03-24T21:46:24.992 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2021-03-24T21:46:24.992 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.max_background_jobs: 2 2021-03-24T21:46:24.993 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.max_background_compactions: -1 2021-03-24T21:46:24.993 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2021-03-24T21:46:24.993 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2021-03-24T21:46:24.993 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.delayed_write_rate : 16777216 2021-03-24T21:46:24.994 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.max_total_wal_size: 0 2021-03-24T21:46:24.994 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2021-03-24T21:46:24.994 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.stats_dump_period_sec: 600 2021-03-24T21:46:24.994 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.stats_persist_period_sec: 600 2021-03-24T21:46:24.995 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2021-03-24T21:46:24.995 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.max_open_files: -1 2021-03-24T21:46:24.995 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.bytes_per_sync: 0 2021-03-24T21:46:24.995 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.wal_bytes_per_sync: 0 2021-03-24T21:46:24.995 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.strict_bytes_per_sync: 0 2021-03-24T21:46:24.996 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.compaction_readahead_size: 0 2021-03-24T21:46:24.996 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Compression algorithms supported: 2021-03-24T21:46:24.996 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2021-03-24T21:46:24.996 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: kZSTD supported: 0 2021-03-24T21:46:24.996 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: kXpressCompression supported: 0 2021-03-24T21:46:24.997 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: kLZ4HCCompression supported: 1 2021-03-24T21:46:24.997 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: kLZ4Compression supported: 1 2021-03-24T21:46:24.997 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: kBZip2Compression supported: 0 2021-03-24T21:46:24.997 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: kZlibCompression supported: 1 2021-03-24T21:46:24.998 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: kSnappyCompression supported: 1 2021-03-24T21:46:24.998 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Fast CRC32 supported: Supported on x86 2021-03-24T21:46:24.998 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000005 2021-03-24T21:46:24.998 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: 2021-03-24T21:46:24.999 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2021-03-24T21:46:24.999 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: 2021-03-24T21:46:24.999 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2021-03-24T21:46:24.999 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.merge_operator: 2021-03-24T21:46:24.999 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:25.000 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.compaction_filter: None 2021-03-24T21:46:25.000 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.compaction_filter_factory: None 2021-03-24T21:46:25.000 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.memtable_factory: SkipListFactory 2021-03-24T21:46:25.000 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.table_factory: BlockBasedTable 2021-03-24T21:46:25.001 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x559a50bd4218) 2021-03-24T21:46:25.001 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: cache_index_and_filter_blocks: 1 2021-03-24T21:46:25.001 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: cache_index_and_filter_blocks_with_high_priority: 0 2021-03-24T21:46:25.001 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: pin_l0_filter_and_index_blocks_in_cache: 0 2021-03-24T21:46:25.001 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: pin_top_level_index_and_filter: 1 2021-03-24T21:46:25.002 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: index_type: 0 2021-03-24T21:46:25.002 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: data_block_index_type: 0 2021-03-24T21:46:25.002 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: index_shortening: 1 2021-03-24T21:46:25.002 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: data_block_hash_table_util_ratio: 0.750000 2021-03-24T21:46:25.003 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: hash_index_allow_collision: 1 2021-03-24T21:46:25.003 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: checksum: 1 2021-03-24T21:46:25.003 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: no_block_cache: 0 2021-03-24T21:46:25.003 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: block_cache: 0x559a50c0cf10 2021-03-24T21:46:25.003 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: block_cache_name: BinnedLRUCache 2021-03-24T21:46:25.004 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: block_cache_options: 2021-03-24T21:46:25.004 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: capacity : 536870912 2021-03-24T21:46:25.004 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: num_shard_bits : 4 2021-03-24T21:46:25.004 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: strict_capacity_limit : 0 2021-03-24T21:46:25.004 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: high_pri_pool_ratio: 0.000 2021-03-24T21:46:25.005 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: block_cache_compressed: (nil) 2021-03-24T21:46:25.005 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: persistent_cache: (nil) 2021-03-24T21:46:25.005 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: block_size: 4096 2021-03-24T21:46:25.005 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: block_size_deviation: 10 2021-03-24T21:46:25.005 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: block_restart_interval: 16 2021-03-24T21:46:25.006 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: index_block_restart_interval: 1 2021-03-24T21:46:25.006 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: metadata_block_size: 4096 2021-03-24T21:46:25.006 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: partition_filters: 0 2021-03-24T21:46:25.006 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: use_delta_encoding: 1 2021-03-24T21:46:25.007 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: filter_policy: rocksdb.BuiltinBloomFilter 2021-03-24T21:46:25.007 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: whole_key_filtering: 1 2021-03-24T21:46:25.007 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: verify_compression: 0 2021-03-24T21:46:25.007 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: read_amp_bytes_per_bit: 0 2021-03-24T21:46:25.008 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: format_version: 2 2021-03-24T21:46:25.008 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: enable_index_compression: 1 2021-03-24T21:46:25.008 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: block_align: 0 2021-03-24T21:46:25.008 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: 2021-03-24T21:46:25.008 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.write_buffer_size: 33554432 2021-03-24T21:46:25.009 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.max_write_buffer_number: 2 2021-03-24T21:46:25.009 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.compression: NoCompression 2021-03-24T21:46:25.009 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.bottommost_compression: Disabled 2021-03-24T21:46:25.009 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.prefix_extractor: nullptr 2021-03-24T21:46:25.009 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2021-03-24T21:46:25.010 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.num_levels: 7 2021-03-24T21:46:25.010 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2021-03-24T21:46:25.010 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2021-03-24T21:46:25.010 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2021-03-24T21:46:25.010 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2021-03-24T21:46:25.011 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2021-03-24T21:46:25.011 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2021-03-24T21:46:25.011 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2021-03-24T21:46:25.011 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2021-03-24T21:46:25.012 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2021-03-24T21:46:25.012 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.compression_opts.window_bits: -14 2021-03-24T21:46:25.012 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.compression_opts.level: 32767 2021-03-24T21:46:25.012 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.compression_opts.strategy: 0 2021-03-24T21:46:25.012 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2021-03-24T21:46:25.013 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2021-03-24T21:46:25.013 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.compression_opts.enabled: false 2021-03-24T21:46:25.013 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2021-03-24T21:46:25.013 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2021-03-24T21:46:25.013 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2021-03-24T21:46:25.014 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.target_file_size_base: 67108864 2021-03-24T21:46:25.014 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.target_file_size_multiplier: 1 2021-03-24T21:46:25.014 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2021-03-24T21:46:25.014 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2021-03-24T21:46:25.014 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2021-03-24T21:46:25.015 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2021-03-24T21:46:25.015 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2021-03-24T21:46:25.015 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2021-03-24T21:46:25.015 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2021-03-24T21:46:25.015 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2021-03-24T21:46:25.016 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2021-03-24T21:46:25.016 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2021-03-24T21:46:25.016 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2021-03-24T21:46:25.016 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2021-03-24T21:46:25.016 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.arena_block_size: 4194304 2021-03-24T21:46:25.017 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2021-03-24T21:46:25.017 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2021-03-24T21:46:25.017 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2021-03-24T21:46:25.017 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.disable_auto_compactions: 0 2021-03-24T21:46:25.017 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2021-03-24T21:46:25.018 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2021-03-24T21:46:25.018 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2021-03-24T21:46:25.018 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2021-03-24T21:46:25.018 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2021-03-24T21:46:25.019 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2021-03-24T21:46:25.019 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2021-03-24T21:46:25.019 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2021-03-24T21:46:25.019 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2021-03-24T21:46:25.019 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2021-03-24T21:46:25.020 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.table_properties_collectors: 2021-03-24T21:46:25.020 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.inplace_update_support: 0 2021-03-24T21:46:25.020 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.inplace_update_num_locks: 10000 2021-03-24T21:46:25.020 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2021-03-24T21:46:25.020 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2021-03-24T21:46:25.021 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.memtable_huge_page_size: 0 2021-03-24T21:46:25.021 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.bloom_locality: 0 2021-03-24T21:46:25.021 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.max_successive_merges: 0 2021-03-24T21:46:25.021 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.optimize_filters_for_hits: 0 2021-03-24T21:46:25.021 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.paranoid_file_checks: 0 2021-03-24T21:46:25.022 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.force_consistency_checks: 0 2021-03-24T21:46:25.022 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.report_bg_io_stats: 0 2021-03-24T21:46:25.022 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.ttl: 2592000 2021-03-24T21:46:25.022 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: Options.periodic_compaction_seconds: 0 2021-03-24T21:46:25.022 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2021-03-24T21:46:25.023 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: 2021-03-24T21:46:25.023 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2021-03-24T21:46:25.023 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: 2021-03-24T21:46:25.023 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616622384888054, "job": 1, "event": "recovery_started", "log_files": [6]} 2021-03-24T21:46:25.024 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.880+0000 7f212eb06700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2021-03-24T21:46:25.024 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.884+0000 7f212eb06700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2021-03-24T21:46:25.024 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.884+0000 7f212eb06700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616622384890588, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 81616, "table_properties": {"data_size": 79893, "index_size": 328, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 581, "raw_key_size": 9953, "raw_average_key_size": 47, "raw_value_size": 74247, "raw_average_value_size": 353, "num_data_blocks": 11, "num_entries": 210, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1616622384, "oldest_key_time": 3, "file_creation_time": 0}} 2021-03-24T21:46:25.024 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.884+0000 7f212eb06700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2021-03-24T21:46:25.024 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: 2021-03-24T21:46:25.025 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.884+0000 7f212eb06700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616622384891670, "job": 1, "event": "recovery_finished"} 2021-03-24T21:46:25.025 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.888+0000 7f212eb06700 4 rocksdb: DB pointer 0x559a50cd3800 2021-03-24T21:46:25.025 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.888+0000 7f2115923700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2021-03-24T21:46:25.025 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.888+0000 7f2115923700 4 rocksdb: [db_impl/db_impl.cc:851] 2021-03-24T21:46:25.025 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: ** DB Stats ** 2021-03-24T21:46:25.026 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: Uptime(secs): 0.0 total, 0.0 interval 2021-03-24T21:46:25.026 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2021-03-24T21:46:25.026 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2021-03-24T21:46:25.026 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2021-03-24T21:46:25.027 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2021-03-24T21:46:25.027 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2021-03-24T21:46:25.027 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2021-03-24T21:46:25.027 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: 2021-03-24T21:46:25.027 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: ** Compaction Stats [default] ** 2021-03-24T21:46:25.028 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-24T21:46:25.028 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-24T21:46:25.028 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: L0 2/0 81.32 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 55.6 0.00 0.00 1 0.001 0 0 2021-03-24T21:46:25.028 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: Sum 2/0 81.32 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 55.6 0.00 0.00 1 0.001 0 0 2021-03-24T21:46:25.028 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 55.6 0.00 0.00 1 0.001 0 0 2021-03-24T21:46:25.029 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: 2021-03-24T21:46:25.029 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: ** Compaction Stats [default] ** 2021-03-24T21:46:25.029 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-24T21:46:25.029 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-24T21:46:25.029 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 55.6 0.00 0.00 1 0.001 0 0 2021-03-24T21:46:25.030 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: Uptime(secs): 0.0 total, 0.0 interval 2021-03-24T21:46:25.030 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: Flush(GB): cumulative 0.000, interval 0.000 2021-03-24T21:46:25.030 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: AddFile(GB): cumulative 0.000, interval 0.000 2021-03-24T21:46:25.030 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: AddFile(Total Files): cumulative 0, interval 0 2021-03-24T21:46:25.031 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: AddFile(L0 Files): cumulative 0, interval 0 2021-03-24T21:46:25.031 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: AddFile(Keys): cumulative 0, interval 0 2021-03-24T21:46:25.031 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: Cumulative compaction: 0.00 GB write, 10.23 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-24T21:46:25.031 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: Interval compaction: 0.00 GB write, 10.23 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-24T21:46:25.031 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-03-24T21:46:25.032 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: 2021-03-24T21:46:25.032 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: ** File Read Latency Histogram By Level [default] ** 2021-03-24T21:46:25.032 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: 2021-03-24T21:46:25.032 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: ** Compaction Stats [default] ** 2021-03-24T21:46:25.032 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-24T21:46:25.033 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-24T21:46:25.033 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: L0 2/0 81.32 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 55.6 0.00 0.00 1 0.001 0 0 2021-03-24T21:46:25.033 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: Sum 2/0 81.32 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 55.6 0.00 0.00 1 0.001 0 0 2021-03-24T21:46:25.033 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-03-24T21:46:25.034 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: 2021-03-24T21:46:25.034 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: ** Compaction Stats [default] ** 2021-03-24T21:46:25.035 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-24T21:46:25.035 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-24T21:46:25.035 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 55.6 0.00 0.00 1 0.001 0 0 2021-03-24T21:46:25.035 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: Uptime(secs): 0.0 total, 0.0 interval 2021-03-24T21:46:25.036 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: Flush(GB): cumulative 0.000, interval 0.000 2021-03-24T21:46:25.036 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: AddFile(GB): cumulative 0.000, interval 0.000 2021-03-24T21:46:25.036 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: AddFile(Total Files): cumulative 0, interval 0 2021-03-24T21:46:25.036 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: AddFile(L0 Files): cumulative 0, interval 0 2021-03-24T21:46:25.036 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: AddFile(Keys): cumulative 0, interval 0 2021-03-24T21:46:25.037 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: Cumulative compaction: 0.00 GB write, 10.12 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-24T21:46:25.037 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-24T21:46:25.037 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-03-24T21:46:25.037 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: 2021-03-24T21:46:25.038 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: ** File Read Latency Histogram By Level [default] ** 2021-03-24T21:46:25.038 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: 2021-03-24T21:46:25.038 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.888+0000 7f212eb06700 0 starting mon.a rank 0 at public addrs v1:172.21.15.125:6789/0 at bind addrs v1:172.21.15.125:6789/0 mon_data /var/lib/ceph/mon/ceph-a fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c 2021-03-24T21:46:25.038 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.892+0000 7f212eb06700 1 mon.a@-1(???) e1 preinit fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c 2021-03-24T21:46:25.038 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.892+0000 7f212eb06700 0 mon.a@-1(???).mds e1 new map 2021-03-24T21:46:25.039 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.892+0000 7f212eb06700 0 mon.a@-1(???).mds e1 print_map 2021-03-24T21:46:25.039 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: e1 2021-03-24T21:46:25.039 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: enable_multiple, ever_enabled_multiple: 1,1 2021-03-24T21:46:25.039 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2021-03-24T21:46:25.040 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: legacy client fscid: -1 2021-03-24T21:46:25.040 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: 2021-03-24T21:46:25.040 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: No filesystems configured 2021-03-24T21:46:25.040 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: 2021-03-24T21:46:25.040 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.892+0000 7f212eb06700 0 mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2021-03-24T21:46:25.041 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.892+0000 7f212eb06700 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2021-03-24T21:46:25.041 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.892+0000 7f212eb06700 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2021-03-24T21:46:25.041 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.892+0000 7f212eb06700 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2021-03-24T21:46:25.041 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: 2021-03-24T21:46:25.041 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: audit 2021-03-24T21:46:25.042 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: 2021-03-24T21:46:22. 2021-03-24T21:46:25.042 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: 629058+0000 mon.a (mon.0) 10 : 2021-03-24T21:46:25.042 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: audit [DBG] from='client.? v1:172.21.15.125:0/2525651213' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:46:25.042 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.892+0000 7f212eb06700 1 mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2021-03-24T21:46:25.043 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.892+0000 7f212eb06700 20 expand_channel_meta expand map: {default=false} 2021-03-24T21:46:25.043 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.892+0000 7f212eb06700 20 expand_channel_meta from 'false' to 'false' 2021-03-24T21:46:25.043 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.892+0000 7f212eb06700 20 expand_channel_meta expanded map: {default=false} 2021-03-24T21:46:25.043 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.892+0000 7f212eb06700 20 expand_channel_meta expand map: {default=info} 2021-03-24T21:46:25.043 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.892+0000 7f212eb06700 20 expand_channel_meta from 'info' to 'info' 2021-03-24T21:46:25.044 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.892+0000 7f212eb06700 20 expand_channel_meta expanded map: {default=info} 2021-03-24T21:46:25.044 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.892+0000 7f212eb06700 20 expand_channel_meta expand map: {default=daemon} 2021-03-24T21:46:25.044 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.892+0000 7f212eb06700 20 expand_channel_meta from 'daemon' to 'daemon' 2021-03-24T21:46:25.044 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.892+0000 7f212eb06700 20 expand_channel_meta expanded map: {default=daemon} 2021-03-24T21:46:25.044 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.892+0000 7f212eb06700 20 expand_channel_meta expand map: {default=debug} 2021-03-24T21:46:25.045 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.892+0000 7f212eb06700 20 expand_channel_meta from 'debug' to 'debug' 2021-03-24T21:46:25.045 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: debug 2021-03-24T21:46:24.892+0000 7f212eb06700 20 expand_channel_meta expanded map: {default=debug} 2021-03-24T21:46:25.045 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: cluster 2021-03-24T21:46:24.921353+0000 mon.a (mon.0) 2021-03-24T21:46:25.045 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: 1 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2021-03-24T21:46:25.045 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: cluster 2021-03-24T21:46:24. 2021-03-24T21:46:25.046 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: 921479+0000 mon.a (mon.0) 2 : cluster [DBG] 2021-03-24T21:46:25.046 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: monmap e1: 1 mons at {a=v1:172.21.15.125:6789/0} 2021-03-24T21:46:25.046 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:46:25.046 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: 03-24T21:46 2021-03-24T21:46:25.047 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: :24.921583+0000 mon.a (mon 2021-03-24T21:46:25.047 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: .0) 3 : cluster [DBG] 2021-03-24T21:46:25.047 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: fsmap 2021-03-24T21:46:25.047 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:46:25.047 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: 03-24T21:46 2021-03-24T21:46:25.048 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: :24.921661+0000 2021-03-24T21:46:25.048 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: mon.a (mon.0) 2021-03-24T21:46:25.048 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2021-03-24T21:46:25.048 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: 2021-03-24T21:46:25.049 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: cluster 2021-03-24T21:46:25.049 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: 2021-03-24T21:46:24. 2021-03-24T21:46:25.049 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: 922425+0000 mon.a (mon.0 2021-03-24T21:46:25.049 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:24 smithi125 conmon[19936]: ) 5 : cluster [DBG] mgrmap e1: no daemons active 2021-03-24T21:46:26.243 INFO:teuthology.orchestra.run.smithi125.stderr:Wrote config to /etc/ceph/ceph.conf 2021-03-24T21:46:26.244 INFO:teuthology.orchestra.run.smithi125.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2021-03-24T21:46:26.244 INFO:teuthology.orchestra.run.smithi125.stderr:Creating mgr... 2021-03-24T21:46:26.245 INFO:teuthology.orchestra.run.smithi125.stderr:Verifying port 9283 ... 2021-03-24T21:46:26.247 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: systemctl daemon-reload 2021-03-24T21:46:26.468 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: systemctl stop ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c@mgr.y 2021-03-24T21:46:26.477 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: systemctl reset-failed ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c@mgr.y 2021-03-24T21:46:26.483 INFO:teuthology.orchestra.run.smithi125.stderr:systemctl: Failed to reset failed state of unit ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c@mgr.y.service: Unit ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c@mgr.y.service not loaded. 2021-03-24T21:46:26.484 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: systemctl enable ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c@mgr.y 2021-03-24T21:46:26.490 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:26 smithi125 conmon[19936]: audit 2021-03-24T21:46:25.474651+0000 2021-03-24T21:46:26.491 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:26 smithi125 conmon[19936]: mon.a (mon.0) 6 : audit [INF] from='client.? v1:172.21.15.125:0/568504404' entity='client.admin' 2021-03-24T21:46:26.491 INFO:teuthology.orchestra.run.smithi125.stderr:systemctl: Created symlink /etc/systemd/system/ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c.target.wants/ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c@mgr.y.service -> /etc/systemd/system/ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c@.service. 2021-03-24T21:46:26.679 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: systemctl start ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c@mgr.y 2021-03-24T21:46:26.905 INFO:journalctl@ceph.mgr.y.smithi125.stdout:Mar 24 21:46:26 smithi125 systemd[1]: Starting Ceph mgr.y for 3cfa339a-8cea-11eb-90b2-001a4aab830c... 2021-03-24T21:46:27.244 INFO:teuthology.orchestra.run.smithi125.stderr:firewalld does not appear to be present 2021-03-24T21:46:27.244 INFO:teuthology.orchestra.run.smithi125.stderr:Not possible to enable service . firewalld.service is not available 2021-03-24T21:46:27.244 INFO:teuthology.orchestra.run.smithi125.stderr:Not possible to open ports <[]>. firewalld.service is not available 2021-03-24T21:46:27.245 INFO:teuthology.orchestra.run.smithi125.stderr:firewalld does not appear to be present 2021-03-24T21:46:27.245 INFO:teuthology.orchestra.run.smithi125.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2021-03-24T21:46:27.245 INFO:teuthology.orchestra.run.smithi125.stderr:Waiting for mgr to start... 2021-03-24T21:46:27.245 INFO:teuthology.orchestra.run.smithi125.stderr:Waiting for mgr... 2021-03-24T21:46:27.246 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: /usr/bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 -e NODE_NAME=smithi125 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp9x0iyaqg:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp51umemfe:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 status -f json-pretty 2021-03-24T21:46:27.405 INFO:journalctl@ceph.mgr.y.smithi125.stdout:Mar 24 21:46:27 smithi125 podman[20217]: 2021-03-24 21:46:27.049841194 +0000 UTC m=+0.176544875 container create eefbe0f3becaf185de6cc49ec83fb1e021fc04270f84b939020a4036f823830b (image=quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-mgr.y, RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Dimitri Savineau , org.label-schema.build-date=20201204, io.buildah.version=1.18.0, org.label-schema.name=CentOS Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06) 2021-03-24T21:46:27.405 INFO:journalctl@ceph.mgr.y.smithi125.stdout:Mar 24 21:46:27 smithi125 podman[20217]: 2021-03-24 21:46:27.199803383 +0000 UTC m=+0.326507064 container init eefbe0f3becaf185de6cc49ec83fb1e021fc04270f84b939020a4036f823830b (image=quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-mgr.y, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06, maintainer=Dimitri Savineau , org.label-schema.build-date=20201204, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Base Image, RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.18.0, CEPH_POINT_RELEASE=) 2021-03-24T21:46:27.405 INFO:journalctl@ceph.mgr.y.smithi125.stdout:Mar 24 21:46:27 smithi125 podman[20217]: 2021-03-24 21:46:27.233007646 +0000 UTC m=+0.359711286 container start eefbe0f3becaf185de6cc49ec83fb1e021fc04270f84b939020a4036f823830b (image=quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-mgr.y, GIT_CLEAN=True, org.label-schema.build-date=20201204, RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d, org.label-schema.vendor=CentOS, maintainer=Dimitri Savineau , GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06, GIT_BRANCH=HEAD, io.buildah.version=1.18.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2021-03-24T21:46:27.406 INFO:journalctl@ceph.mgr.y.smithi125.stdout:Mar 24 21:46:27 smithi125 bash[20217]: eefbe0f3becaf185de6cc49ec83fb1e021fc04270f84b939020a4036f823830b 2021-03-24T21:46:27.406 INFO:journalctl@ceph.mgr.y.smithi125.stdout:Mar 24 21:46:27 smithi125 systemd[1]: Started Ceph mgr.y for 3cfa339a-8cea-11eb-90b2-001a4aab830c. 2021-03-24T21:46:27.804 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: 2021-03-24T21:46:27.804 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: { 2021-03-24T21:46:27.805 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "fsid": "3cfa339a-8cea-11eb-90b2-001a4aab830c", 2021-03-24T21:46:27.805 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "health": { 2021-03-24T21:46:27.805 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2021-03-24T21:46:27.805 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "checks": {}, 2021-03-24T21:46:27.805 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "mutes": [] 2021-03-24T21:46:27.806 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: }, 2021-03-24T21:46:27.806 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "election_epoch": 5, 2021-03-24T21:46:27.806 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "quorum": [ 2021-03-24T21:46:27.806 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: 0 2021-03-24T21:46:27.807 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: ], 2021-03-24T21:46:27.807 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "quorum_names": [ 2021-03-24T21:46:27.807 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "a" 2021-03-24T21:46:27.807 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: ], 2021-03-24T21:46:27.809 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "quorum_age": 2, 2021-03-24T21:46:27.809 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "monmap": { 2021-03-24T21:46:27.809 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "epoch": 1, 2021-03-24T21:46:27.809 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2021-03-24T21:46:27.810 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "num_mons": 1 2021-03-24T21:46:27.810 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: }, 2021-03-24T21:46:27.810 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "osdmap": { 2021-03-24T21:46:27.810 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "epoch": 1, 2021-03-24T21:46:27.811 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "num_osds": 0, 2021-03-24T21:46:27.811 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "num_up_osds": 0, 2021-03-24T21:46:27.811 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "osd_up_since": 0, 2021-03-24T21:46:27.811 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "num_in_osds": 0, 2021-03-24T21:46:27.815 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "osd_in_since": 0, 2021-03-24T21:46:27.815 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2021-03-24T21:46:27.816 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: }, 2021-03-24T21:46:27.816 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "pgmap": { 2021-03-24T21:46:27.816 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "pgs_by_state": [], 2021-03-24T21:46:27.816 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "num_pgs": 0, 2021-03-24T21:46:27.816 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "num_pools": 0, 2021-03-24T21:46:27.817 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "num_objects": 0, 2021-03-24T21:46:27.817 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "data_bytes": 0, 2021-03-24T21:46:27.817 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "bytes_used": 0, 2021-03-24T21:46:27.817 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "bytes_avail": 0, 2021-03-24T21:46:27.817 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "bytes_total": 0 2021-03-24T21:46:27.818 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: }, 2021-03-24T21:46:27.818 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "fsmap": { 2021-03-24T21:46:27.818 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "epoch": 1, 2021-03-24T21:46:27.818 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "by_rank": [], 2021-03-24T21:46:27.819 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "up:standby": 0 2021-03-24T21:46:27.819 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: }, 2021-03-24T21:46:27.819 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "mgrmap": { 2021-03-24T21:46:27.819 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "available": false, 2021-03-24T21:46:27.819 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "num_standbys": 0, 2021-03-24T21:46:27.820 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "modules": [ 2021-03-24T21:46:27.820 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "iostat", 2021-03-24T21:46:27.820 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "restful" 2021-03-24T21:46:27.820 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: ], 2021-03-24T21:46:27.820 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "services": {} 2021-03-24T21:46:27.821 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: }, 2021-03-24T21:46:27.821 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "servicemap": { 2021-03-24T21:46:27.821 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "epoch": 1, 2021-03-24T21:46:27.821 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "modified": "2021-03-24T21:46:19.164471+0000", 2021-03-24T21:46:27.821 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "services": {} 2021-03-24T21:46:27.822 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: }, 2021-03-24T21:46:27.822 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "progress_events": {} 2021-03-24T21:46:27.822 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: } 2021-03-24T21:46:28.155 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:27 smithi125 conmon[19936]: audit 2021-03-24T21:46: 2021-03-24T21:46:28.155 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:27 smithi125 conmon[19936]: 27.805898+0000 mon.a (mon.0) 7 : audit [DBG] from='client.? v1:172.21.15.125:0/905194780' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2021-03-24T21:46:28.564 INFO:teuthology.orchestra.run.smithi125.stderr:mgr not available, waiting (1/15)... 2021-03-24T21:46:30.567 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: /usr/bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 -e NODE_NAME=smithi125 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp9x0iyaqg:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp51umemfe:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 status -f json-pretty 2021-03-24T21:46:31.097 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: 2021-03-24T21:46:31.098 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: { 2021-03-24T21:46:31.099 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "fsid": "3cfa339a-8cea-11eb-90b2-001a4aab830c", 2021-03-24T21:46:31.099 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "health": { 2021-03-24T21:46:31.099 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2021-03-24T21:46:31.100 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "checks": {}, 2021-03-24T21:46:31.100 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "mutes": [] 2021-03-24T21:46:31.100 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: }, 2021-03-24T21:46:31.102 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "election_epoch": 5, 2021-03-24T21:46:31.103 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "quorum": [ 2021-03-24T21:46:31.103 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: 0 2021-03-24T21:46:31.103 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: ], 2021-03-24T21:46:31.103 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "quorum_names": [ 2021-03-24T21:46:31.104 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "a" 2021-03-24T21:46:31.104 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: ], 2021-03-24T21:46:31.104 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "quorum_age": 6, 2021-03-24T21:46:31.104 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "monmap": { 2021-03-24T21:46:31.104 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "epoch": 1, 2021-03-24T21:46:31.105 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2021-03-24T21:46:31.105 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "num_mons": 1 2021-03-24T21:46:31.105 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: }, 2021-03-24T21:46:31.105 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "osdmap": { 2021-03-24T21:46:31.105 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "epoch": 1, 2021-03-24T21:46:31.106 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "num_osds": 0, 2021-03-24T21:46:31.106 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "num_up_osds": 0, 2021-03-24T21:46:31.106 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "osd_up_since": 0, 2021-03-24T21:46:31.106 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "num_in_osds": 0, 2021-03-24T21:46:31.110 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "osd_in_since": 0, 2021-03-24T21:46:31.110 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2021-03-24T21:46:31.110 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: }, 2021-03-24T21:46:31.111 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "pgmap": { 2021-03-24T21:46:31.111 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "pgs_by_state": [], 2021-03-24T21:46:31.111 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "num_pgs": 0, 2021-03-24T21:46:31.111 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "num_pools": 0, 2021-03-24T21:46:31.111 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "num_objects": 0, 2021-03-24T21:46:31.112 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "data_bytes": 0, 2021-03-24T21:46:31.112 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "bytes_used": 0, 2021-03-24T21:46:31.112 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "bytes_avail": 0, 2021-03-24T21:46:31.112 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "bytes_total": 0 2021-03-24T21:46:31.113 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: }, 2021-03-24T21:46:31.113 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "fsmap": { 2021-03-24T21:46:31.113 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "epoch": 1, 2021-03-24T21:46:31.113 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "by_rank": [], 2021-03-24T21:46:31.113 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "up:standby": 0 2021-03-24T21:46:31.114 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: }, 2021-03-24T21:46:31.114 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "mgrmap": { 2021-03-24T21:46:31.114 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "available": false, 2021-03-24T21:46:31.114 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "num_standbys": 0, 2021-03-24T21:46:31.115 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "modules": [ 2021-03-24T21:46:31.115 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "iostat", 2021-03-24T21:46:31.115 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "restful" 2021-03-24T21:46:31.115 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: ], 2021-03-24T21:46:31.115 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "services": {} 2021-03-24T21:46:31.116 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: }, 2021-03-24T21:46:31.116 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "servicemap": { 2021-03-24T21:46:31.116 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "epoch": 1, 2021-03-24T21:46:31.116 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "modified": "2021-03-24T21:46:19.164471+0000", 2021-03-24T21:46:31.116 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "services": {} 2021-03-24T21:46:31.117 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: }, 2021-03-24T21:46:31.117 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "progress_events": {} 2021-03-24T21:46:31.117 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: } 2021-03-24T21:46:31.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:31 smithi125 conmon[19936]: audit 2021-03-24T21:46:31.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:31 smithi125 conmon[19936]: 2021-03-24T21:46:31.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:31 smithi125 conmon[19936]: 2021 2021-03-24T21:46:31.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:31 smithi125 conmon[19936]: - 2021-03-24T21:46:31.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:31 smithi125 conmon[19936]: 03 2021-03-24T21:46:31.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:31 smithi125 conmon[19936]: -24T21:46: 2021-03-24T21:46:31.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:31 smithi125 conmon[19936]: 31.100948+0000 mon.a (mon.0) 8 : audit [DBG] from='client.? v1:172.21.15.125:0/2217326250' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2021-03-24T21:46:31.865 INFO:teuthology.orchestra.run.smithi125.stderr:mgr not available, waiting (2/15)... 2021-03-24T21:46:32.906 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:32 smithi125 conmon[19936]: cluster 2021-03-24T21:46:32.394325+0000 2021-03-24T21:46:32.906 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:32 smithi125 conmon[19936]: mon.a (mon.0) 9 : cluster [INF] Activating manager daemon y 2021-03-24T21:46:32.906 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:32 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:46:32.907 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:32 smithi125 conmon[19936]: -24T21:46:32 2021-03-24T21:46:32.907 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:32 smithi125 conmon[19936]: .400820+0000 mon.a (mon.0 2021-03-24T21:46:32.907 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:32 smithi125 conmon[19936]: ) 10 : cluster [DBG] mgrmap e2: y(active, starting, since 0.00665438s) 2021-03-24T21:46:32.907 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:32 smithi125 conmon[19936]: audit 2021 2021-03-24T21:46:32.908 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:32 smithi125 conmon[19936]: -03-24T21:46:32. 2021-03-24T21:46:32.908 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:32 smithi125 conmon[19936]: 404378+0000 mon.a (mon.0) 2021-03-24T21:46:32.908 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:32 smithi125 conmon[19936]: 11 : audit [DBG] from='mgr.14100 v1:172.21.15.125:0/1223609390' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-03-24T21:46:32.908 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:32 smithi125 conmon[19936]: audit 2021-03-24T21:46:32.909 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:32 smithi125 conmon[19936]: 2021-03-24T 2021-03-24T21:46:32.909 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:32 smithi125 conmon[19936]: 21:46:32.404941+0000 2021-03-24T21:46:32.909 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:32 smithi125 conmon[19936]: mon.a (mon.0) 12 : 2021-03-24T21:46:32.909 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:32 smithi125 conmon[19936]: audit [DBG] from='mgr.14100 v1:172.21.15.125:0/1223609390' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-03-24T21:46:32.909 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:32 smithi125 conmon[19936]: audit 2021-03-24T21:46:32.910 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:32 smithi125 conmon[19936]: 2021-03-24T 2021-03-24T21:46:32.910 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:32 smithi125 conmon[19936]: 21:46:32.405483+0000 2021-03-24T21:46:32.910 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:32 smithi125 conmon[19936]: mon.a (mon.0) 13 : audit 2021-03-24T21:46:32.910 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:32 smithi125 conmon[19936]: [DBG] from='mgr.14100 v1:172.21.15.125:0/1223609390' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-03-24T21:46:32.910 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:32 smithi125 conmon[19936]: audit 2021-03-24T21:46:32.911 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:32 smithi125 conmon[19936]: 2021-03-24T21:46: 2021-03-24T21:46:32.911 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:32 smithi125 conmon[19936]: 32.406149+0000 mon.a (mon.0) 14 2021-03-24T21:46:32.911 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:32 smithi125 conmon[19936]: : audit [DBG] from='mgr.14100 v1:172.21.15.125:0/1223609390' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-03-24T21:46:32.911 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:32 smithi125 conmon[19936]: audit 2021-03 2021-03-24T21:46:32.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:32 smithi125 conmon[19936]: -24T21:46 2021-03-24T21:46:32.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:32 smithi125 conmon[19936]: :32.406382+0000 mon.a ( 2021-03-24T21:46:32.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:32 smithi125 conmon[19936]: mon.0) 15 : audit 2021-03-24T21:46:32.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:32 smithi125 conmon[19936]: [DBG] from='mgr.14100 v1:172.21.15.125:0/1223609390' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-03-24T21:46:32.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:32 smithi125 conmon[19936]: audit 2021-03-24T21:46:32.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:32 smithi125 conmon[19936]: 2021-03-24T21:46:32 2021-03-24T21:46:32.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:32 smithi125 conmon[19936]: .406601+0000 mon.a (mon.0) 16 : 2021-03-24T21:46:32.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:32 smithi125 conmon[19936]: audit [DBG] from='mgr.14100 v1:172.21.15.125:0/1223609390' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-03-24T21:46:32.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:32 smithi125 conmon[19936]: audit 2021-03-24 2021-03-24T21:46:32.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:32 smithi125 conmon[19936]: T21:46:32.406941+0000 mon.a ( 2021-03-24T21:46:32.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:32 smithi125 conmon[19936]: mon.0) 17 : audit [DBG] from='mgr.14100 v1:172.21.15.125:0/1223609390' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2021-03-24T21:46:32.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:32 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:46:32.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:32 smithi125 conmon[19936]: 24T21:46:32.410214+0000 mon.a (mon.0 2021-03-24T21:46:32.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:32 smithi125 conmon[19936]: ) 18 : cluster [INF] Manager daemon y is now available 2021-03-24T21:46:32.915 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:32 smithi125 conmon[19936]: audit 2021 2021-03-24T21:46:32.915 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:32 smithi125 conmon[19936]: -03-24T21:46:32.433871+0000 mon.a 2021-03-24T21:46:32.915 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:32 smithi125 conmon[19936]: (mon.0) 19 : audit [INF] from='mgr.14100 v1:172.21.15.125:0/1223609390' entity='mgr.y' 2021-03-24T21:46:32.915 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:32 smithi125 conmon[19936]: audit 2021 2021-03-24T21:46:32.915 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:32 smithi125 conmon[19936]: -03-24T21:46:32.437413 2021-03-24T21:46:32.916 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:32 smithi125 conmon[19936]: +0000 mon.a (mon.0) 20 : audit [INF] 2021-03-24T21:46:32.916 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:32 smithi125 conmon[19936]: from='mgr.14100 v1:172.21.15.125:0/1223609390' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T21:46:32.916 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:32 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:46:32.916 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:32 smithi125 conmon[19936]: 03-24T21: 2021-03-24T21:46:32.917 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:32 smithi125 conmon[19936]: 46:32.439914+0000 mon.a (mon.0 2021-03-24T21:46:32.917 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:32 smithi125 conmon[19936]: ) 21 : audit [INF] from='mgr.14100 v1:172.21.15.125:0/1223609390' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T21:46:33.868 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: /usr/bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 -e NODE_NAME=smithi125 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp9x0iyaqg:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp51umemfe:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 status -f json-pretty 2021-03-24T21:46:33.905 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:33 smithi125 conmon[19936]: audit 2021-03-24T21:46:32.449558+0000 mon.a (mon.0) 22 : audit 2021-03-24T21:46:33.905 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:33 smithi125 conmon[19936]: [INF] from='mgr.14100 v1:172.21.15.125:0/1223609390' entity='mgr.y' 2021-03-24T21:46:33.905 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:33 smithi125 conmon[19936]: audit 2021-03-24T21:46:32.460326+0000 mon.a (mon.0) 23 : audit 2021-03-24T21:46:33.905 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:33 smithi125 conmon[19936]: [INF] from='mgr.14100 v1:172.21.15.125:0/1223609390' entity='mgr.y' 2021-03-24T21:46:33.906 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:33 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:46:33.906 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:33 smithi125 conmon[19936]: 24T21:46:33.406364+0000 mon.a (mon.0) 2021-03-24T21:46:33.906 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:33 smithi125 conmon[19936]: 24 : cluster [DBG] mgrmap e3: y(active, since 1.01219s) 2021-03-24T21:46:34.629 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: 2021-03-24T21:46:34.855 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: { 2021-03-24T21:46:34.856 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "fsid": "3cfa339a-8cea-11eb-90b2-001a4aab830c", 2021-03-24T21:46:34.856 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "health": { 2021-03-24T21:46:34.863 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2021-03-24T21:46:34.863 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "checks": {}, 2021-03-24T21:46:34.863 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "mutes": [] 2021-03-24T21:46:34.864 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: }, 2021-03-24T21:46:34.864 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "election_epoch": 5, 2021-03-24T21:46:34.864 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "quorum": [ 2021-03-24T21:46:34.864 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: 0 2021-03-24T21:46:34.865 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: ], 2021-03-24T21:46:34.865 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "quorum_names": [ 2021-03-24T21:46:34.865 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "a" 2021-03-24T21:46:34.865 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: ], 2021-03-24T21:46:34.865 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "quorum_age": 9, 2021-03-24T21:46:34.866 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "monmap": { 2021-03-24T21:46:34.866 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "epoch": 1, 2021-03-24T21:46:34.866 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2021-03-24T21:46:34.866 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "num_mons": 1 2021-03-24T21:46:34.866 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: }, 2021-03-24T21:46:34.867 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "osdmap": { 2021-03-24T21:46:34.867 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "epoch": 1, 2021-03-24T21:46:34.867 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "num_osds": 0, 2021-03-24T21:46:34.867 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "num_up_osds": 0, 2021-03-24T21:46:34.868 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "osd_up_since": 0, 2021-03-24T21:46:34.868 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "num_in_osds": 0, 2021-03-24T21:46:34.868 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "osd_in_since": 0, 2021-03-24T21:46:34.868 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2021-03-24T21:46:34.868 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: }, 2021-03-24T21:46:34.869 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "pgmap": { 2021-03-24T21:46:34.869 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "pgs_by_state": [], 2021-03-24T21:46:34.869 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "num_pgs": 0, 2021-03-24T21:46:34.869 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "num_pools": 0, 2021-03-24T21:46:34.869 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "num_objects": 0, 2021-03-24T21:46:34.870 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "data_bytes": 0, 2021-03-24T21:46:34.870 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "bytes_used": 0, 2021-03-24T21:46:34.870 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "bytes_avail": 0, 2021-03-24T21:46:34.870 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "bytes_total": 0 2021-03-24T21:46:34.871 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: }, 2021-03-24T21:46:34.871 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "fsmap": { 2021-03-24T21:46:34.871 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "epoch": 1, 2021-03-24T21:46:34.871 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "by_rank": [], 2021-03-24T21:46:34.871 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "up:standby": 0 2021-03-24T21:46:34.872 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: }, 2021-03-24T21:46:34.872 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "mgrmap": { 2021-03-24T21:46:34.872 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "available": true, 2021-03-24T21:46:34.872 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "num_standbys": 0, 2021-03-24T21:46:34.873 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "modules": [ 2021-03-24T21:46:34.873 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "iostat", 2021-03-24T21:46:34.873 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "restful" 2021-03-24T21:46:34.873 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: ], 2021-03-24T21:46:34.873 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "services": {} 2021-03-24T21:46:34.874 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: }, 2021-03-24T21:46:34.874 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "servicemap": { 2021-03-24T21:46:34.874 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "epoch": 1, 2021-03-24T21:46:34.874 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "modified": "2021-03-24T21:46:19.164471+0000", 2021-03-24T21:46:34.874 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "services": {} 2021-03-24T21:46:34.875 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: }, 2021-03-24T21:46:34.875 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "progress_events": {} 2021-03-24T21:46:34.875 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: } 2021-03-24T21:46:35.341 INFO:teuthology.orchestra.run.smithi125.stderr:mgr is available 2021-03-24T21:46:35.342 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: /usr/bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 -e NODE_NAME=smithi125 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp9x0iyaqg:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp51umemfe:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 config-key set mgr/cephadm/ssh_user root 2021-03-24T21:46:35.607 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:35 smithi125 conmon[19936]: cluster 2021-03-24T21:46:34.463018+0000 mon.a 2021-03-24T21:46:35.608 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:35 smithi125 conmon[19936]: (mon.0) 25 : cluster [DBG] mgrmap e4: y(active, since 2s) 2021-03-24T21:46:35.608 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:35 smithi125 conmon[19936]: audit 2021-03-24T21:46:35.608 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:35 smithi125 conmon[19936]: 2021-03-24T21: 2021-03-24T21:46:35.608 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:35 smithi125 conmon[19936]: 46:34.632320+0000 2021-03-24T21:46:35.608 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:35 smithi125 conmon[19936]: mon.a (mon.0 2021-03-24T21:46:35.609 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:35 smithi125 conmon[19936]: ) 26 : audit 2021-03-24T21:46:35.609 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:35 smithi125 conmon[19936]: [DBG] from='client.? v1:172.21.15.125:0/3224682699' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2021-03-24T21:46:36.100 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: set mgr/cephadm/ssh_user 2021-03-24T21:46:36.782 INFO:teuthology.orchestra.run.smithi125.stderr:Enabling cephadm module... 2021-03-24T21:46:36.783 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: /usr/bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 -e NODE_NAME=smithi125 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp9x0iyaqg:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp51umemfe:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 mgr module enable cephadm 2021-03-24T21:46:37.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:37 smithi125 conmon[19936]: audit 2021-03-24T21 2021-03-24T21:46:37.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:37 smithi125 conmon[19936]: :46:36.100603+0000 mon.a (mon.0) 27 : audit [INF] from='client.? v1:172.21.15.125:0/2967292038' entity='client.admin' 2021-03-24T21:46:38.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:38 smithi125 conmon[19936]: audit 2021-03-24T21:46:37.496949+0000 mon.a (mon.0) 28 : audit [INF] from='client.? v1:172.21.15.125:0/1452410642' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2021-03-24T21:46:38.405 INFO:journalctl@ceph.mgr.y.smithi125.stdout:Mar 24 21:46:38 smithi125 conmon[20260]: ignoring --setuser ceph since I am not root 2021-03-24T21:46:38.405 INFO:journalctl@ceph.mgr.y.smithi125.stdout:Mar 24 21:46:38 smithi125 conmon[20260]: ignoring --setgroup ceph since I am not root 2021-03-24T21:46:38.915 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: /usr/bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 -e NODE_NAME=smithi125 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp9x0iyaqg:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp51umemfe:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 mgr stat 2021-03-24T21:46:39.162 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:39 smithi125 conmon[19936]: audit 2021-03-24T21 2021-03-24T21:46:39.162 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:39 smithi125 conmon[19936]: :46:38.107352+0000 mon.a (mon.0) 29 : audit [INF] from='client.? v1:172.21.15.125:0/1452410642' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2021-03-24T21:46:39.162 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:39 smithi125 conmon[19936]: cluster 2021-03-24T21:46:39.162 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:39 smithi125 conmon[19936]: 2021-03-24T21 2021-03-24T21:46:39.163 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:39 smithi125 conmon[19936]: :46:38. 2021-03-24T21:46:39.163 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:39 smithi125 conmon[19936]: 107484+0000 mon.a 2021-03-24T21:46:39.163 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:39 smithi125 conmon[19936]: (mon.0) 2021-03-24T21:46:39.163 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:39 smithi125 conmon[19936]: 30 : cluster 2021-03-24T21:46:39.164 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:39 smithi125 conmon[19936]: [DBG] mgrmap e5: y(active, since 5s) 2021-03-24T21:46:39.615 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: { 2021-03-24T21:46:39.615 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "epoch": 5, 2021-03-24T21:46:39.615 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "available": true, 2021-03-24T21:46:39.616 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "active_name": "y", 2021-03-24T21:46:39.616 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "num_standby": 0 2021-03-24T21:46:39.616 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: } 2021-03-24T21:46:40.307 INFO:teuthology.orchestra.run.smithi125.stderr:Waiting for the mgr to restart... 2021-03-24T21:46:40.307 INFO:teuthology.orchestra.run.smithi125.stderr:Waiting for mgr epoch 5... 2021-03-24T21:46:40.307 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: /usr/bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 -e NODE_NAME=smithi125 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp9x0iyaqg:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp51umemfe:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 tell mgr mgr_status 2021-03-24T21:46:40.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:40 smithi125 conmon[19936]: audit 2021-03-24T21:46:39.618470+0000 mon.a (mon.0) 31 : 2021-03-24T21:46:40.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:40 smithi125 conmon[19936]: audit [DBG] from='client.? v1:172.21.15.125:0/2717893598' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2021-03-24T21:46:44.268 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: { 2021-03-24T21:46:44.268 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2021-03-24T21:46:44.269 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "initialized": true 2021-03-24T21:46:44.269 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: } 2021-03-24T21:46:44.462 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: cluster 2021-03-24T21:46:43.197674+0000 mon.a (mon.0) 32 : 2021-03-24T21:46:44.463 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: cluster [INF] Active manager daemon y restarted 2021-03-24T21:46:44.463 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: cluster 2021-03-24T21:46:43.198604+0000 2021-03-24T21:46:44.463 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: mon.a (mon.0) 33 : cluster [INF] Activating manager daemon y 2021-03-24T21:46:44.463 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: cluster 2021-03-24T21:46:44.464 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: 2021-03-24T21:46:43. 2021-03-24T21:46:44.464 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: 202324+0000 mon.a (mon.0) 34 : cluster 2021-03-24T21:46:44.464 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: [DBG] osdmap e2: 0 total, 0 up, 0 in 2021-03-24T21:46:44.464 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: cluster 2021-03-24T21:46:44.465 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: 2021-03-24T21:46: 2021-03-24T21:46:44.465 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: 43.254078+0000 mon.a (mon.0) 35 : 2021-03-24T21:46:44.465 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: cluster [DBG] mgrmap e6: y(active, starting, since 0.0556204s) 2021-03-24T21:46:44.465 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: audit 2021 2021-03-24T21:46:44.466 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: -03-24T21:46: 2021-03-24T21:46:44.466 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: 43.258396+0000 mon.a (mon.0) 2021-03-24T21:46:44.466 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: 36 : audit [DBG] from='mgr.14116 v1:172.21.15.125:0/339399267' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2021-03-24T21:46:44.466 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:46:44.467 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: 03-24T21:46:43. 2021-03-24T21:46:44.467 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: 259696+0000 mon.a (mon.0) 37 : audit 2021-03-24T21:46:44.467 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: [DBG] from='mgr.14116 v1:172.21.15.125:0/339399267' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-03-24T21:46:44.467 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: audit 2021-03-24 2021-03-24T21:46:44.467 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: T21:46:43.260025+0000 mon.a 2021-03-24T21:46:44.468 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: (mon.0) 38 : audit [DBG] from='mgr.14116 v1:172.21.15.125:0/339399267' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-03-24T21:46:44.468 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: 2021-03-24T21:46:44.468 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: audit 2021-03-24T21:46:44.468 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: 2021-03-24T21:46:43 2021-03-24T21:46:44.469 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: .260299+0000 mon.a (mon.0) 39 : 2021-03-24T21:46:44.469 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: audit [DBG] from='mgr.14116 v1:172.21.15.125:0/339399267' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-03-24T21:46:44.469 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:46:44.469 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: 24T21:46:43.263750+0000 2021-03-24T21:46:44.470 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: mon.a (mon.0) 40 : cluster [INF] 2021-03-24T21:46:44.470 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: Manager daemon y is now available 2021-03-24T21:46:44.470 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: audit 2021-03-24T 2021-03-24T21:46:44.470 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: 21:46:43.288324+0000 mon.a ( 2021-03-24T21:46:44.471 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: mon.0) 41 : audit [INF] from='mgr.14116 v1:172.21.15.125:0/339399267' entity='mgr.y' 2021-03-24T21:46:44.471 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: audit 2021 2021-03-24T21:46:44.471 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: -03-24T21:46:43.295624+0000 2021-03-24T21:46:44.471 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: mon.a (mon.0) 42 : audit [INF] from='mgr.14116 v1:172.21.15.125:0/339399267' entity='mgr.y' 2021-03-24T21:46:44.471 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: audit 2021-03-24 2021-03-24T21:46:44.472 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: T21:46:43.302201+0000 mon.a (mon.0) 2021-03-24T21:46:44.472 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: 43 : audit [INF] from='mgr.14116 v1:172.21.15.125:0/339399267' entity='mgr.y' 2021-03-24T21:46:44.472 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: audit 2021-03-24T21:46:44.472 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: 2021-03-24T21:46:43.406423+0000 mon.a (mon 2021-03-24T21:46:44.472 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: .0) 44 : audit [INF] from='mgr.14116 v1:172.21.15.125:0/339399267' entity='mgr.y' 2021-03-24T21:46:44.473 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:46:44.473 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: 03-24T21:46:43.407588+0000 2021-03-24T21:46:44.473 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: mon.a (mon.0) 45 : audit [DBG] from='mgr.14116 v1:172.21.15.125:0/339399267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:46:44.473 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: audit 2021-03-24T21 2021-03-24T21:46:44.474 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: :46:43.410176+0000 mon.a (mon.0 2021-03-24T21:46:44.474 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: ) 46 : audit [DBG] from='mgr.14116 v1:172.21.15.125:0/339399267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:46:44.474 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: audit 2021-03-24T21:46:44.474 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: 2021-03-24T21:46:43.421083+0000 2021-03-24T21:46:44.474 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: mon.a (mon.0) 47 : audit [DBG] 2021-03-24T21:46:44.475 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: from='mgr.14116 v1:172.21.15.125:0/339399267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:46:44.475 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: audit 2021 2021-03-24T21:46:44.475 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: -03-24T21:46:43.434645+0000 mon.a (mon. 2021-03-24T21:46:44.475 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: 0) 48 : audit [INF] from='mgr.14116 v1:172.21.15.125:0/339399267' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T21:46:44.475 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: audit 2021-03 2021-03-24T21:46:44.476 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: -24T21:46:43.436590+0000 mon.a (mon.0 2021-03-24T21:46:44.476 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:44 smithi125 conmon[19936]: ) 49 : audit [INF] from='mgr.14116 v1:172.21.15.125:0/339399267' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T21:46:45.040 INFO:teuthology.orchestra.run.smithi125.stderr:mgr epoch 5 is available 2021-03-24T21:46:45.040 INFO:teuthology.orchestra.run.smithi125.stderr:Setting orchestrator backend to cephadm... 2021-03-24T21:46:45.040 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: /usr/bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 -e NODE_NAME=smithi125 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp9x0iyaqg:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp51umemfe:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 orch set backend cephadm 2021-03-24T21:46:45.309 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:45 smithi125 conmon[19936]: audit 2021-03-24T21:46:44.257003+0000 mgr.y (mgr.14116) 1 : 2021-03-24T21:46:45.309 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:45 smithi125 conmon[19936]: audit [DBG] from='client.14120 v1:172.21.15.125:0/150291749' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2021-03-24T21:46:45.310 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:45 smithi125 conmon[19936]: cluster 2021-03-24T21:46:44.258931+0000 mon.a (mon.0) 2021-03-24T21:46:45.310 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:45 smithi125 conmon[19936]: 50 : cluster [DBG] mgrmap e7: y(active, since 1.06047s) 2021-03-24T21:46:45.310 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:45 smithi125 conmon[19936]: audit 2021-03-24T21:46:45.310 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:45 smithi125 conmon[19936]: 2021-03-24T21:46:44.271457+0000 mgr.y ( 2021-03-24T21:46:45.311 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:45 smithi125 conmon[19936]: mgr.14116) 2 : audit [DBG] from='client.14120 v1:172.21.15.125:0/150291749' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2021-03-24T21:46:46.590 INFO:teuthology.orchestra.run.smithi125.stderr:Generating ssh key... 2021-03-24T21:46:46.590 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: /usr/bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 -e NODE_NAME=smithi125 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp9x0iyaqg:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp51umemfe:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 cephadm generate-key 2021-03-24T21:46:46.882 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:46 smithi125 conmon[19936]: audit 2021-03-24T21:46: 2021-03-24T21:46:46.883 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:46 smithi125 conmon[19936]: 45.772201+0000 mgr.y (mgr.14116) 3 : audit [DBG] from='client.14126 v1:172.21.15.125:0/3895380513' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:46:46.883 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:46 smithi125 conmon[19936]: audit 2021-03-24T21:46:45 2021-03-24T21:46:46.883 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:46 smithi125 conmon[19936]: .778694+0000 mon.a (mon.0) 51 : audit [INF] from='mgr.14116 v1:172.21.15.125:0/339399267' entity='mgr.y' 2021-03-24T21:46:46.884 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:46 smithi125 conmon[19936]: audit 2021-03-24T21:46:45.784976 2021-03-24T21:46:46.884 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:46 smithi125 conmon[19936]: +0000 mon.a (mon.0) 52 : audit [DBG] from='mgr.14116 v1:172.21.15.125:0/339399267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:46:46.884 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:46 smithi125 conmon[19936]: 2021-03-24T21:46:46.884 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:46 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:46:46.884 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:46 smithi125 conmon[19936]: 24T21:46:45.911072+0000 mon.a (mon.0) 53 : cluster [DBG] 2021-03-24T21:46:46.885 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:46 smithi125 conmon[19936]: mgrmap e8: y(active, since 2s) 2021-03-24T21:46:47.905 INFO:journalctl@ceph.mgr.y.smithi125.stdout:Mar 24 21:46:47 smithi125 conmon[20260]: Generating public/private rsa key pair. 2021-03-24T21:46:47.905 INFO:journalctl@ceph.mgr.y.smithi125.stdout:Mar 24 21:46:47 smithi125 conmon[20260]: Your identification has been saved in /tmp/tmpofm3eg4z/key. 2021-03-24T21:46:47.905 INFO:journalctl@ceph.mgr.y.smithi125.stdout:Mar 24 21:46:47 smithi125 conmon[20260]: Your public key has been saved in /tmp/tmpofm3eg4z/key.pub. 2021-03-24T21:46:47.906 INFO:journalctl@ceph.mgr.y.smithi125.stdout:Mar 24 21:46:47 smithi125 conmon[20260]: The key fingerprint is: 2021-03-24T21:46:47.906 INFO:journalctl@ceph.mgr.y.smithi125.stdout:Mar 24 21:46:47 smithi125 conmon[20260]: SHA256:emK13Ge6gN8PCWZU7gEBfLH/fRmGXVcYPgugfNnh5KY ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c 2021-03-24T21:46:47.906 INFO:journalctl@ceph.mgr.y.smithi125.stdout:Mar 24 21:46:47 smithi125 conmon[20260]: The key's randomart image is: 2021-03-24T21:46:47.906 INFO:journalctl@ceph.mgr.y.smithi125.stdout:Mar 24 21:46:47 smithi125 conmon[20260]: +---[RSA 3072]----+ 2021-03-24T21:46:47.907 INFO:journalctl@ceph.mgr.y.smithi125.stdout:Mar 24 21:46:47 smithi125 conmon[20260]: | ...+oo o .o.| 2021-03-24T21:46:47.907 INFO:journalctl@ceph.mgr.y.smithi125.stdout:Mar 24 21:46:47 smithi125 conmon[20260]: | ...* B o. .| 2021-03-24T21:46:47.907 INFO:journalctl@ceph.mgr.y.smithi125.stdout:Mar 24 21:46:47 smithi125 conmon[20260]: | .= = * o o| 2021-03-24T21:46:47.907 INFO:journalctl@ceph.mgr.y.smithi125.stdout:Mar 24 21:46:47 smithi125 conmon[20260]: | . + + .ooo| 2021-03-24T21:46:47.908 INFO:journalctl@ceph.mgr.y.smithi125.stdout:Mar 24 21:46:47 smithi125 conmon[20260]: | S E ..+ | 2021-03-24T21:46:47.908 INFO:journalctl@ceph.mgr.y.smithi125.stdout:Mar 24 21:46:47 smithi125 conmon[20260]: | B + o .. o| 2021-03-24T21:46:47.908 INFO:journalctl@ceph.mgr.y.smithi125.stdout:Mar 24 21:46:47 smithi125 conmon[20260]: | = = + + ...| 2021-03-24T21:46:47.908 INFO:journalctl@ceph.mgr.y.smithi125.stdout:Mar 24 21:46:47 smithi125 conmon[20260]: | . + o = . | 2021-03-24T21:46:47.908 INFO:journalctl@ceph.mgr.y.smithi125.stdout:Mar 24 21:46:47 smithi125 conmon[20260]: | . +o. | 2021-03-24T21:46:47.909 INFO:journalctl@ceph.mgr.y.smithi125.stdout:Mar 24 21:46:47 smithi125 conmon[20260]: +----[SHA256]-----+ 2021-03-24T21:46:48.373 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: /usr/bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 -e NODE_NAME=smithi125 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp9x0iyaqg:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp51umemfe:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 cephadm get-pub-key 2021-03-24T21:46:48.904 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:48 smithi125 conmon[19936]: cephadm 2021-03-24T21:46 2021-03-24T21:46:48.905 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:48 smithi125 conmon[19936]: :47.298190+0000 mgr.y (mgr.14116) 5 : cephadm [INF] Generating ssh key... 2021-03-24T21:46:48.905 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:48 smithi125 conmon[19936]: audit 2021-03-24 2021-03-24T21:46:48.905 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:48 smithi125 conmon[19936]: T21:46:47.632894+0000 mon.a (mon.0) 54 : audit [INF] from='mgr.14116 v1:172.21.15.125:0/339399267' entity='mgr.y' 2021-03-24T21:46:48.905 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:48 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:46:48.906 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:48 smithi125 conmon[19936]: 03-24T21:46:47.635486+0000 mon.a ( 2021-03-24T21:46:48.906 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:48 smithi125 conmon[19936]: mon.0) 55 : audit [INF] from='mgr.14116 v1:172.21.15.125:0/339399267' entity='mgr.y' 2021-03-24T21:46:49.051 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4xtPjYt+IIFBmwPlLYx2JQeC8afARhmXSLE+WVwrRzKFitQSiqBJa0LzmIt2VjanZXZPCYPpyrMWYRNYUX7IPn5+SSD65uX3o3wr2DxOcGBJocHmta6Esw7DuZxBZfQ9gmXs2Eja2oR8uH5R2hJxKgG3KRpyNNgWwPgzP1YtPGY7qLIfDOwGOEs7HTTSPTABavhOP04674AmhdvMzyq43KhhPtO7y5x0p96NC9dX70tdERAzi5Zm7Ut+/kwori+u/S/dC0FjnGWrjSWRSF6raop7M8s999A6soqUsXx5XmZpWnP4f0l9srtGN5ji9d2IqFUmvR5XkfHITq/SueMeHs6JyE5wFLIYIl5+d8epcOpUIdoNeq40W7+9J5HisnUjXGr0rHV3yje9CnfR1NUXQV1vQ9fSLt0ypSg3TZwMd7DvUXEns856E8TtOtVJFmviPldcQCvogsPl0lDY20Pqz3PPLyUymMOVq6LxSQFa44K+buH9mGNJzEHaWW6GPHWk= ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c 2021-03-24T21:46:49.814 INFO:teuthology.orchestra.run.smithi125.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2021-03-24T21:46:49.814 INFO:teuthology.orchestra.run.smithi125.stderr:Adding key to root@localhost authorized_keys... 2021-03-24T21:46:49.815 INFO:teuthology.orchestra.run.smithi125.stderr:Adding host smithi125... 2021-03-24T21:46:49.816 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: /usr/bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 -e NODE_NAME=smithi125 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp9x0iyaqg:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp51umemfe:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 orch host add smithi125 2021-03-24T21:46:49.904 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:49 smithi125 conmon[19936]: audit 2021-03-24T21:46:49.055291+0000 mgr.y (mgr.14116) 2021-03-24T21:46:49.905 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:49 smithi125 conmon[19936]: 6 : audit [DBG] from='client.14130 v1:172.21.15.125:0/2589230359' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:46:50.904 INFO:journalctl@ceph.mgr.y.smithi125.stdout:Mar 24 21:46:50 smithi125 conmon[20260]: Warning: Permanently added 'smithi125' (ECDSA) to the list of known hosts. 2021-03-24T21:46:51.318 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: Added host 'smithi125' 2021-03-24T21:46:51.525 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:51 smithi125 conmon[19936]: audit 2021-03-24T21:46:50.522051 2021-03-24T21:46:51.525 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:51 smithi125 conmon[19936]: +0000 mgr.y (mgr.14116) 7 : audit [DBG] from='client.14132 v1:172.21.15.125:0/2646360010' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi125", "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:46:52.256 INFO:teuthology.orchestra.run.smithi125.stderr:Deploying unmanaged mon service... 2021-03-24T21:46:52.257 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: /usr/bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 -e NODE_NAME=smithi125 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp9x0iyaqg:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp51umemfe:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 orch apply mon --unmanaged 2021-03-24T21:46:52.563 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:52 smithi125 conmon[19936]: audit 2021-03-24T21:46 2021-03-24T21:46:52.564 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:52 smithi125 conmon[19936]: :51.321564+0000 mon.a (mon.0) 56 : audit [INF] from='mgr.14116 v1:172.21.15.125:0/339399267' entity='mgr.y' 2021-03-24T21:46:52.564 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:52 smithi125 conmon[19936]: cephadm 2021-03-24T21:46:52.564 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:52 smithi125 conmon[19936]: 2021-03-24T21:46:51.322247+0000 mgr.y (mgr.14116) 8 : cephadm 2021-03-24T21:46:52.564 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:52 smithi125 conmon[19936]: [INF] Added host smithi125 2021-03-24T21:46:52.565 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:52 smithi125 conmon[19936]: audit 2021-03-24T21 2021-03-24T21:46:52.565 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:52 smithi125 conmon[19936]: :46:51.322678+0000 mon.a (mon.0 2021-03-24T21:46:52.565 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:52 smithi125 conmon[19936]: ) 57 : audit [DBG] from='mgr.14116 v1:172.21.15.125:0/339399267' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:46:53.064 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: Scheduled mon update... 2021-03-24T21:46:53.839 INFO:teuthology.orchestra.run.smithi125.stderr:Deploying unmanaged mgr service... 2021-03-24T21:46:53.839 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: /usr/bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 -e NODE_NAME=smithi125 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp9x0iyaqg:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp51umemfe:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 orch apply mgr --unmanaged 2021-03-24T21:46:54.089 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:54 smithi125 conmon[19936]: audit 2021-03-24T21:46:53.061184+0000 mgr.y (mgr.14116) 9 : audit [DBG] from='client.14134 v1:172.21.15.125:0/367818035' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:46:54.090 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:54 smithi125 conmon[19936]: cephadm 2021-03-24T21:46:53 2021-03-24T21:46:54.090 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:54 smithi125 conmon[19936]: .062892+0000 mgr.y (mgr.14116) 10 : cephadm [INF] Saving service mon spec with placement count:5 2021-03-24T21:46:54.090 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:54 smithi125 conmon[19936]: audit 2021-03-24T21:46:53.067731+0000 mon.a (mon.0) 58 : audit [INF] from='mgr.14116 v1:172.21.15.125:0/339399267' entity='mgr.y' 2021-03-24T21:46:54.548 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: Scheduled mgr update... 2021-03-24T21:46:55.246 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: /usr/bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 -e NODE_NAME=smithi125 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp9x0iyaqg:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp51umemfe:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 config set mgr mgr/cephadm/container_init True --force 2021-03-24T21:46:55.904 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:55 smithi125 conmon[19936]: audit 2021-03-24T21:46:54.547176+0000 mgr.y (mgr.14116) 11 : audit 2021-03-24T21:46:55.905 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:55 smithi125 conmon[19936]: [DBG] from='client.14136 v1:172.21.15.125:0/1104469840' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:46:55.905 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:55 smithi125 conmon[19936]: cephadm 2021-03-24T21:46:54.547773+0000 mgr.y (mgr.14116 2021-03-24T21:46:55.905 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:55 smithi125 conmon[19936]: ) 12 : cephadm [INF] Saving service mgr spec with placement count:2 2021-03-24T21:46:55.905 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:55 smithi125 conmon[19936]: audit 2021-03-24T21:46:54.552159+0000 mon.a 2021-03-24T21:46:55.906 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:55 smithi125 conmon[19936]: (mon.0) 59 : audit [INF] from='mgr.14116 v1:172.21.15.125:0/339399267' entity='mgr.y' 2021-03-24T21:46:56.657 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: /usr/bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 -e NODE_NAME=smithi125 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp9x0iyaqg:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp51umemfe:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 config set mgr mgr/dashboard/ssl_server_port 8443 2021-03-24T21:46:57.154 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:56 smithi125 conmon[19936]: audit 2021-03-24T21:46:55.929262+0000 mon.a (mon.0) 60 : audit 2021-03-24T21:46:57.155 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:56 smithi125 conmon[19936]: [INF] from='client.? v1:172.21.15.125:0/4092188006' entity='client.admin' 2021-03-24T21:46:58.106 INFO:teuthology.orchestra.run.smithi125.stderr:Enabling the dashboard module... 2021-03-24T21:46:58.106 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: /usr/bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 -e NODE_NAME=smithi125 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp9x0iyaqg:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp51umemfe:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 mgr module enable dashboard 2021-03-24T21:46:58.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:58 smithi125 conmon[19936]: audit 2021-03-24T21:46:57.375282+0000 mon.a (mon.0) 61 : audit 2021-03-24T21:46:58.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:58 smithi125 conmon[19936]: [INF] from='client.? v1:172.21.15.125:0/193246041' entity='client.admin' 2021-03-24T21:46:59.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:59 smithi125 conmon[19936]: audit 2021-03-24T21:46 2021-03-24T21:46:59.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:46:59 smithi125 conmon[19936]: :58.823209+0000 mon.a (mon.0) 62 : audit [INF] from='client.? v1:172.21.15.125:0/1498246329' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2021-03-24T21:46:59.655 INFO:journalctl@ceph.mgr.y.smithi125.stdout:Mar 24 21:46:59 smithi125 conmon[20260]: ignoring --setuser ceph since I am not root 2021-03-24T21:46:59.656 INFO:journalctl@ceph.mgr.y.smithi125.stdout:Mar 24 21:46:59 smithi125 conmon[20260]: ignoring --setgroup ceph since I am not root 2021-03-24T21:47:00.088 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: /usr/bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 -e NODE_NAME=smithi125 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp9x0iyaqg:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp51umemfe:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 mgr stat 2021-03-24T21:47:00.622 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:00 smithi125 conmon[19936]: audit 2021-03-24T21:46: 2021-03-24T21:47:00.623 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:00 smithi125 conmon[19936]: 59.382073+0000 mon.a (mon.0) 63 : audit [INF] from='client.? v1:172.21.15.125:0/1498246329' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2021-03-24T21:47:00.623 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:00 smithi125 conmon[19936]: cluster 2021-03-24T21:47:00.623 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:00 smithi125 conmon[19936]: 2021-03-24T21: 2021-03-24T21:47:00.623 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:00 smithi125 conmon[19936]: 46:59.382159+0000 mon.a ( 2021-03-24T21:47:00.624 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:00 smithi125 conmon[19936]: mon.0) 64 : cluster 2021-03-24T21:47:00.624 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:00 smithi125 conmon[19936]: [DBG] mgrmap e9: y(active, since 16s) 2021-03-24T21:47:00.827 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: { 2021-03-24T21:47:00.827 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "epoch": 9, 2021-03-24T21:47:00.827 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "available": true, 2021-03-24T21:47:00.828 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "active_name": "y", 2021-03-24T21:47:00.828 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "num_standby": 0 2021-03-24T21:47:00.828 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: } 2021-03-24T21:47:01.619 INFO:teuthology.orchestra.run.smithi125.stderr:Waiting for the mgr to restart... 2021-03-24T21:47:01.619 INFO:teuthology.orchestra.run.smithi125.stderr:Waiting for mgr epoch 9... 2021-03-24T21:47:01.620 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: /usr/bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 -e NODE_NAME=smithi125 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp9x0iyaqg:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp51umemfe:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 tell mgr mgr_status 2021-03-24T21:47:01.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:01 smithi125 conmon[19936]: audit 2021-03-24T21:47:00.830677+0000 mon.a (mon.0) 65 : 2021-03-24T21:47:01.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:01 smithi125 conmon[19936]: audit [DBG] from='client.? v1:172.21.15.125:0/3257974941' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2021-03-24T21:47:04.730 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:04 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:47:04.730 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:04 smithi125 conmon[19936]: 47:04.501998+0000 mon.a (mon.0) 66 : cluster [INF] Active manager daemon y restarted 2021-03-24T21:47:04.730 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:04 smithi125 conmon[19936]: cluster 2021-03-24T21:47:04.730 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:04 smithi125 conmon[19936]: 2021-03-24T21:47:04. 2021-03-24T21:47:04.731 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:04 smithi125 conmon[19936]: 502952+0000 mon.a (mon.0) 2021-03-24T21:47:04.731 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:04 smithi125 conmon[19936]: 67 : cluster [INF] Activating manager daemon y 2021-03-24T21:47:04.731 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:04 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:47:04.731 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:04 smithi125 conmon[19936]: -03-24T21:47:04 2021-03-24T21:47:04.731 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:04 smithi125 conmon[19936]: .506649+0000 mon.a (mon. 2021-03-24T21:47:04.732 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:04 smithi125 conmon[19936]: 0) 68 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2021-03-24T21:47:04.732 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:04 smithi125 conmon[19936]: 2021-03-24T21:47:05.154 INFO:journalctl@ceph.mgr.y.smithi125.stdout:Mar 24 21:47:04 smithi125 conmon[20260]: Warning: Permanently added 'smithi125' (ECDSA) to the list of known hosts. 2021-03-24T21:47:05.575 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: { 2021-03-24T21:47:05.576 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2021-03-24T21:47:05.576 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: "initialized": true 2021-03-24T21:47:05.576 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: } 2021-03-24T21:47:05.865 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:05 smithi125 conmon[19936]: cluster 2021-03-24T21:47: 2021-03-24T21:47:05.865 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:05 smithi125 conmon[19936]: 04.558863+0000 mon.a (mon.0) 69 : cluster [DBG] mgrmap e10: y(active, starting, since 0.05607s) 2021-03-24T21:47:05.865 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:05 smithi125 conmon[19936]: audit 2021-03-24T21:47: 2021-03-24T21:47:05.865 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:05 smithi125 conmon[19936]: 04.568570+0000 mon.a (mon.0) 70 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2021-03-24T21:47:05.866 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:05 smithi125 conmon[19936]: audit 2021-03-24T21:47:04.569778 2021-03-24T21:47:05.866 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:05 smithi125 conmon[19936]: +0000 mon.a (mon.0) 71 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-03-24T21:47:05.866 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:05 smithi125 conmon[19936]: audit 2021-03-24T21 2021-03-24T21:47:05.866 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:05 smithi125 conmon[19936]: :47:04.570229+0000 mon.a (mon.0) 72 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-03-24T21:47:05.867 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:05 smithi125 conmon[19936]: audit 2021-03-24T21:47:04.570658+0000 mon.a (mon.0) 73 : audit [DBG] 2021-03-24T21:47:05.867 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:05 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-03-24T21:47:05.867 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:05 smithi125 conmon[19936]: cluster 2021-03-24T21:47:05.867 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:05 smithi125 conmon[19936]: 2021-03-24T21:47:04.574279+0000 2021-03-24T21:47:05.868 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:05 smithi125 conmon[19936]: mon.a (mon.0) 74 : cluster [INF] Manager daemon y is now available 2021-03-24T21:47:05.868 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:05 smithi125 conmon[19936]: audit 2021-03-24 2021-03-24T21:47:05.868 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:05 smithi125 conmon[19936]: T21:47:04.685136+0000 mon.a (mon.0) 2021-03-24T21:47:05.868 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:05 smithi125 conmon[19936]: 75 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:47:05.869 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:05 smithi125 conmon[19936]: audit 2021-03-24T21:47:05.869 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:05 smithi125 conmon[19936]: 2021-03-24T21:47:04.689467+0000 2021-03-24T21:47:05.869 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:05 smithi125 conmon[19936]: mon.a (mon.0) 76 : 2021-03-24T21:47:05.869 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:05 smithi125 conmon[19936]: audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:47:05.869 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:05 smithi125 conmon[19936]: audit 2021-03-24T21:47:05.870 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:05 smithi125 conmon[19936]: 2021-03-24T21 2021-03-24T21:47:05.870 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:05 smithi125 conmon[19936]: :47:04.724440+0000 2021-03-24T21:47:05.870 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:05 smithi125 conmon[19936]: mon.a (mon.0) 77 : audit 2021-03-24T21:47:05.870 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:05 smithi125 conmon[19936]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T21:47:05.870 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:05 smithi125 conmon[19936]: audit 2021-03-24T21:47:05.871 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:05 smithi125 conmon[19936]: 2021-03-24T 2021-03-24T21:47:05.871 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:05 smithi125 conmon[19936]: 21:47:04.729041+0000 2021-03-24T21:47:05.871 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:05 smithi125 conmon[19936]: mon.a (mon.0) 78 2021-03-24T21:47:05.871 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:05 smithi125 conmon[19936]: : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T21:47:06.663 INFO:teuthology.orchestra.run.smithi125.stderr:mgr epoch 9 is available 2021-03-24T21:47:06.663 INFO:teuthology.orchestra.run.smithi125.stderr:Generating a dashboard self-signed certificate... 2021-03-24T21:47:06.664 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: /usr/bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 -e NODE_NAME=smithi125 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp9x0iyaqg:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp51umemfe:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 dashboard create-self-signed-cert 2021-03-24T21:47:06.851 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:06 smithi125 conmon[19936]: audit 2021-03-24T21:47:05.567775+0000 mgr.y (mgr.14144) 1 : audit 2021-03-24T21:47:06.851 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:06 smithi125 conmon[19936]: [DBG] from='client.14148 v1:172.21.15.125:0/3217612113' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2021-03-24T21:47:06.852 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:06 smithi125 conmon[19936]: cluster 2021-03-24T21:47:05.569479+0000 2021-03-24T21:47:06.852 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:06 smithi125 conmon[19936]: mon.a (mon.0) 79 : cluster [DBG] mgrmap e11: y(active, since 1.06668s) 2021-03-24T21:47:06.852 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:06 smithi125 conmon[19936]: audit 2021 2021-03-24T21:47:06.852 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:06 smithi125 conmon[19936]: -03-24T21:47: 2021-03-24T21:47:06.852 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:06 smithi125 conmon[19936]: 05.579770+0000 mgr.y (mgr. 2021-03-24T21:47:06.853 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:06 smithi125 conmon[19936]: 14144) 2 : audit [DBG] from='client.14148 v1:172.21.15.125:0/3217612113' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2021-03-24T21:47:06.853 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:06 smithi125 conmon[19936]: 2021-03-24T21:47:07.704 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:07 smithi125 conmon[19936]: cluster 2021-03-24T21:47 2021-03-24T21:47:07.704 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:07 smithi125 conmon[19936]: :06.582880+0000 mon.a (mon.0) 80 : cluster [DBG] mgrmap e12: y(active, since 2s) 2021-03-24T21:47:07.775 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: Self-signed certificate created 2021-03-24T21:47:09.154 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:08 smithi125 conmon[19936]: audit 2021-03-24T21:47: 2021-03-24T21:47:09.155 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:08 smithi125 conmon[19936]: 07.623576+0000 mgr.y (mgr.14144) 3 : audit [DBG] from='client.14154 v1:172.21.15.125:0/625323572' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:47:09.155 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:08 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:47:09.155 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:08 smithi125 conmon[19936]: 03-24T21:47:07.776265+0000 mon.a (mon.0) 81 : audit [INF] 2021-03-24T21:47:09.156 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:08 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:47:09.156 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:08 smithi125 conmon[19936]: audit 2021-03-24T21:47:09.156 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:08 smithi125 conmon[19936]: 2021-03-24T21:47:07.778821+0000 2021-03-24T21:47:09.156 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:08 smithi125 conmon[19936]: mon.a (mon.0) 82 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:47:09.156 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:08 smithi125 conmon[19936]: 2021-03-24T21:47:09.613 INFO:teuthology.orchestra.run.smithi125.stderr:Creating initial admin user... 2021-03-24T21:47:09.614 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: /usr/bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 -e NODE_NAME=smithi125 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp9x0iyaqg:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp51umemfe:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpphawey1_:/tmp/dashboard.pw:z quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2021-03-24T21:47:10.617 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$TTm5wb5emYBNrFSqoaQpQ.h9WZv1YdtakVRPBBCkD/zxDd0nPnRf6", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1616622430, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2021-03-24T21:47:10.889 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:10 smithi125 conmon[19936]: audit 2021-03-24T21:47:09.794322 2021-03-24T21:47:10.890 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:10 smithi125 conmon[19936]: +0000 mon.a (mon.0) 83 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:47:10.890 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:10 smithi125 conmon[19936]: audit 2021-03-24 2021-03-24T21:47:10.890 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:10 smithi125 conmon[19936]: T21:47:10.154209+0000 mon.a (mon.0) 84 : audit 2021-03-24T21:47:10.891 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:10 smithi125 conmon[19936]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:47:10.891 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:10 smithi125 conmon[19936]: audit 2021-03-24T21:47:10.891 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:10 smithi125 conmon[19936]: 2021-03-24T21:47:10.384590+0000 mgr.y 2021-03-24T21:47:10.891 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:10 smithi125 conmon[19936]: (mgr.14144) 4 : audit [DBG] from='client.14156 v1:172.21.15.125:0/1137644963' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:47:10.892 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:10 smithi125 conmon[19936]: audit 2021-03 2021-03-24T21:47:10.892 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:10 smithi125 conmon[19936]: -24T21:47:10.621444+0000 2021-03-24T21:47:10.892 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:10 smithi125 conmon[19936]: mon.a (mon.0) 85 : audit [INF] 2021-03-24T21:47:10.892 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:10 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:47:11.684 INFO:teuthology.orchestra.run.smithi125.stderr:Fetching dashboard port number... 2021-03-24T21:47:11.684 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: /usr/bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 -e NODE_NAME=smithi125 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp9x0iyaqg:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp51umemfe:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 config get mgr mgr/dashboard/ssl_server_port 2021-03-24T21:47:12.352 INFO:teuthology.orchestra.run.smithi125.stderr:/usr/bin/ceph: 8443 2021-03-24T21:47:12.429 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:12 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:47:12.429 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:12 smithi125 conmon[19936]: 47:11.161088+0000 mon.a (mon.0) 86 : cluster [DBG] mgrmap e13: y(active, since 6s) 2021-03-24T21:47:12.430 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:12 smithi125 conmon[19936]: audit 2021-03-24T21:47:12.430 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:12 smithi125 conmon[19936]: 2021-03-24T21 2021-03-24T21:47:12.430 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:12 smithi125 conmon[19936]: :47:11.764342+0000 mon.a 2021-03-24T21:47:12.430 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:12 smithi125 conmon[19936]: (mon.0) 87 : audit 2021-03-24T21:47:12.431 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:12 smithi125 conmon[19936]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:47:13.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:13 smithi125 conmon[19936]: audit 2021-03-24T21:47:12. 2021-03-24T21:47:13.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:13 smithi125 conmon[19936]: 357113+0000 mon.a (mon.0) 88 : audit [DBG] from='client.? v1:172.21.15.125:0/1731748236' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2021-03-24T21:47:13.794 INFO:teuthology.orchestra.run.smithi125.stderr:firewalld does not appear to be present 2021-03-24T21:47:13.794 INFO:teuthology.orchestra.run.smithi125.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2021-03-24T21:47:13.798 INFO:teuthology.orchestra.run.smithi125.stderr:Ceph Dashboard is now available at: 2021-03-24T21:47:13.799 INFO:teuthology.orchestra.run.smithi125.stderr: 2021-03-24T21:47:13.799 INFO:teuthology.orchestra.run.smithi125.stderr: URL: https://smithi125.front.sepia.ceph.com:8443/ 2021-03-24T21:47:13.799 INFO:teuthology.orchestra.run.smithi125.stderr: User: admin 2021-03-24T21:47:13.799 INFO:teuthology.orchestra.run.smithi125.stderr: Password: zugsi7071o 2021-03-24T21:47:13.799 INFO:teuthology.orchestra.run.smithi125.stderr: 2021-03-24T21:47:13.800 INFO:teuthology.orchestra.run.smithi125.stderr:You can access the Ceph CLI with: 2021-03-24T21:47:13.801 INFO:teuthology.orchestra.run.smithi125.stderr: 2021-03-24T21:47:13.801 INFO:teuthology.orchestra.run.smithi125.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2021-03-24T21:47:13.801 INFO:teuthology.orchestra.run.smithi125.stderr: 2021-03-24T21:47:13.801 INFO:teuthology.orchestra.run.smithi125.stderr:Please consider enabling telemetry to help improve Ceph: 2021-03-24T21:47:13.802 INFO:teuthology.orchestra.run.smithi125.stderr: 2021-03-24T21:47:13.802 INFO:teuthology.orchestra.run.smithi125.stderr: ceph telemetry on 2021-03-24T21:47:13.802 INFO:teuthology.orchestra.run.smithi125.stderr: 2021-03-24T21:47:13.802 INFO:teuthology.orchestra.run.smithi125.stderr:For more information see: 2021-03-24T21:47:13.803 INFO:teuthology.orchestra.run.smithi125.stderr: 2021-03-24T21:47:13.803 INFO:teuthology.orchestra.run.smithi125.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2021-03-24T21:47:13.803 INFO:teuthology.orchestra.run.smithi125.stderr: 2021-03-24T21:47:13.803 INFO:teuthology.orchestra.run.smithi125.stderr:Bootstrap complete. 2021-03-24T21:47:13.803 INFO:teuthology.orchestra.run.smithi125.stderr:Releasing lock 140164061439264 on /run/cephadm/3cfa339a-8cea-11eb-90b2-001a4aab830c.lock 2021-03-24T21:47:13.804 INFO:teuthology.orchestra.run.smithi125.stderr:Lock 140164061439264 released on /run/cephadm/3cfa339a-8cea-11eb-90b2-001a4aab830c.lock 2021-03-24T21:47:13.826 INFO:tasks.cephadm:Fetching config... 2021-03-24T21:47:13.826 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2021-03-24T21:47:13.827 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2021-03-24T21:47:13.834 INFO:tasks.cephadm:Fetching client.admin keyring... 2021-03-24T21:47:13.834 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2021-03-24T21:47:13.835 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2021-03-24T21:47:13.840 INFO:tasks.cephadm:Fetching mon keyring... 2021-03-24T21:47:13.841 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2021-03-24T21:47:13.841 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd if=/var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/keyring of=/dev/stdout 2021-03-24T21:47:13.852 INFO:tasks.cephadm:Fetching pub ssh key... 2021-03-24T21:47:13.852 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2021-03-24T21:47:13.853 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2021-03-24T21:47:13.901 INFO:tasks.cephadm:Installing pub ssh key for root users... 2021-03-24T21:47:13.902 DEBUG:teuthology.orchestra.run.smithi125:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4xtPjYt+IIFBmwPlLYx2JQeC8afARhmXSLE+WVwrRzKFitQSiqBJa0LzmIt2VjanZXZPCYPpyrMWYRNYUX7IPn5+SSD65uX3o3wr2DxOcGBJocHmta6Esw7DuZxBZfQ9gmXs2Eja2oR8uH5R2hJxKgG3KRpyNNgWwPgzP1YtPGY7qLIfDOwGOEs7HTTSPTABavhOP04674AmhdvMzyq43KhhPtO7y5x0p96NC9dX70tdERAzi5Zm7Ut+/kwori+u/S/dC0FjnGWrjSWRSF6raop7M8s999A6soqUsXx5XmZpWnP4f0l9srtGN5ji9d2IqFUmvR5XkfHITq/SueMeHs6JyE5wFLIYIl5+d8epcOpUIdoNeq40W7+9J5HisnUjXGr0rHV3yje9CnfR1NUXQV1vQ9fSLt0ypSg3TZwMd7DvUXEns856E8TtOtVJFmviPldcQCvogsPl0lDY20Pqz3PPLyUymMOVq6LxSQFa44K+buH9mGNJzEHaWW6GPHWk= ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2021-03-24T21:47:13.924 INFO:teuthology.orchestra.run.smithi125.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4xtPjYt+IIFBmwPlLYx2JQeC8afARhmXSLE+WVwrRzKFitQSiqBJa0LzmIt2VjanZXZPCYPpyrMWYRNYUX7IPn5+SSD65uX3o3wr2DxOcGBJocHmta6Esw7DuZxBZfQ9gmXs2Eja2oR8uH5R2hJxKgG3KRpyNNgWwPgzP1YtPGY7qLIfDOwGOEs7HTTSPTABavhOP04674AmhdvMzyq43KhhPtO7y5x0p96NC9dX70tdERAzi5Zm7Ut+/kwori+u/S/dC0FjnGWrjSWRSF6raop7M8s999A6soqUsXx5XmZpWnP4f0l9srtGN5ji9d2IqFUmvR5XkfHITq/SueMeHs6JyE5wFLIYIl5+d8epcOpUIdoNeq40W7+9J5HisnUjXGr0rHV3yje9CnfR1NUXQV1vQ9fSLt0ypSg3TZwMd7DvUXEns856E8TtOtVJFmviPldcQCvogsPl0lDY20Pqz3PPLyUymMOVq6LxSQFa44K+buH9mGNJzEHaWW6GPHWk= ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c 2021-03-24T21:47:13.931 DEBUG:teuthology.orchestra.run.smithi142:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4xtPjYt+IIFBmwPlLYx2JQeC8afARhmXSLE+WVwrRzKFitQSiqBJa0LzmIt2VjanZXZPCYPpyrMWYRNYUX7IPn5+SSD65uX3o3wr2DxOcGBJocHmta6Esw7DuZxBZfQ9gmXs2Eja2oR8uH5R2hJxKgG3KRpyNNgWwPgzP1YtPGY7qLIfDOwGOEs7HTTSPTABavhOP04674AmhdvMzyq43KhhPtO7y5x0p96NC9dX70tdERAzi5Zm7Ut+/kwori+u/S/dC0FjnGWrjSWRSF6raop7M8s999A6soqUsXx5XmZpWnP4f0l9srtGN5ji9d2IqFUmvR5XkfHITq/SueMeHs6JyE5wFLIYIl5+d8epcOpUIdoNeq40W7+9J5HisnUjXGr0rHV3yje9CnfR1NUXQV1vQ9fSLt0ypSg3TZwMd7DvUXEns856E8TtOtVJFmviPldcQCvogsPl0lDY20Pqz3PPLyUymMOVq6LxSQFa44K+buH9mGNJzEHaWW6GPHWk= ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2021-03-24T21:47:13.959 INFO:teuthology.orchestra.run.smithi142.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4xtPjYt+IIFBmwPlLYx2JQeC8afARhmXSLE+WVwrRzKFitQSiqBJa0LzmIt2VjanZXZPCYPpyrMWYRNYUX7IPn5+SSD65uX3o3wr2DxOcGBJocHmta6Esw7DuZxBZfQ9gmXs2Eja2oR8uH5R2hJxKgG3KRpyNNgWwPgzP1YtPGY7qLIfDOwGOEs7HTTSPTABavhOP04674AmhdvMzyq43KhhPtO7y5x0p96NC9dX70tdERAzi5Zm7Ut+/kwori+u/S/dC0FjnGWrjSWRSF6raop7M8s999A6soqUsXx5XmZpWnP4f0l9srtGN5ji9d2IqFUmvR5XkfHITq/SueMeHs6JyE5wFLIYIl5+d8epcOpUIdoNeq40W7+9J5HisnUjXGr0rHV3yje9CnfR1NUXQV1vQ9fSLt0ypSg3TZwMd7DvUXEns856E8TtOtVJFmviPldcQCvogsPl0lDY20Pqz3PPLyUymMOVq6LxSQFa44K+buH9mGNJzEHaWW6GPHWk= ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c 2021-03-24T21:47:13.968 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph config set mgr mgr/cephadm/allow_ptrace true 2021-03-24T21:47:17.106 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi142 2021-03-24T21:47:17.106 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2021-03-24T21:47:17.107 DEBUG:teuthology.orchestra.run.smithi142:> dd of=/etc/ceph/ceph.conf 2021-03-24T21:47:17.114 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2021-03-24T21:47:17.114 DEBUG:teuthology.orchestra.run.smithi142:> dd of=/etc/ceph/ceph.client.admin.keyring 2021-03-24T21:47:17.121 INFO:tasks.cephadm:Adding host smithi142 to orchestrator... 2021-03-24T21:47:17.122 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph orch host add smithi142 2021-03-24T21:47:17.904 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:17 smithi125 conmon[19936]: audit 2021-03-24T21:47: 2021-03-24T21:47:17.905 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:17 smithi125 conmon[19936]: 16.466903+0000 mon.a (mon.0) 89 : audit [INF] from='client.? v1:172.21.15.125:0/2972153199' entity='client.admin' 2021-03-24T21:47:17.905 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:17 smithi125 conmon[19936]: audit 2021 2021-03-24T21:47:17.905 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:17 smithi125 conmon[19936]: -03-24T21:47:16.695771+0000 mon.a (mon.0 2021-03-24T21:47:17.905 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:17 smithi125 conmon[19936]: ) 90 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:47:17.906 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:17 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:47:17.906 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:17 smithi125 conmon[19936]: 03-24T21:47:17.066766 2021-03-24T21:47:17.906 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:17 smithi125 conmon[19936]: +0000 mon.a (mon.0) 91 2021-03-24T21:47:17.906 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:17 smithi125 conmon[19936]: : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:47:17.907 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:17 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:47:17.907 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:17 smithi125 conmon[19936]: 03-24T21:47:17.071215+0000 2021-03-24T21:47:17.907 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:17 smithi125 conmon[19936]: mon.a (mon.0) 92 : 2021-03-24T21:47:17.907 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:17 smithi125 conmon[19936]: audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:47:17.908 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:17 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:47:17.908 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:17 smithi125 conmon[19936]: 03-24T21:47:17.073644+0000 2021-03-24T21:47:17.908 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:17 smithi125 conmon[19936]: mon.a (mon.0) 93 : 2021-03-24T21:47:17.908 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:17 smithi125 conmon[19936]: audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:47:17.908 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:17 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:47:17.909 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:17 smithi125 conmon[19936]: 03-24T21:47:17.079047+0000 2021-03-24T21:47:17.909 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:17 smithi125 conmon[19936]: mon.a (mon.0) 94 2021-03-24T21:47:17.909 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:17 smithi125 conmon[19936]: : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:47:26.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:25 smithi125 conmon[19936]: cluster 2021-03-24T21:47 2021-03-24T21:47:26.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:25 smithi125 conmon[19936]: :24.571193+0000 mgr.y (mgr.14144) 5 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:47:28.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:27 smithi125 conmon[19936]: cluster 2021-03-24T21:47 2021-03-24T21:47:28.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:27 smithi125 conmon[19936]: :26.571582+0000 mgr.y (mgr.14144) 6 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:47:30.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:29 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:47:30.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:29 smithi125 conmon[19936]: 47:28.571975+0000 mgr.y (mgr.14144) 7 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:47:32.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:31 smithi125 conmon[19936]: cluster 2021-03-24T21:47: 2021-03-24T21:47:32.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:31 smithi125 conmon[19936]: 30.572354+0000 mgr.y (mgr.14144) 8 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:47:34.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:33 smithi125 conmon[19936]: cluster 2021-03-24T21:47:32.572750+0000 mgr.y (mgr.14144) 2021-03-24T21:47:34.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:33 smithi125 conmon[19936]: 9 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:47:36.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:35 smithi125 conmon[19936]: cluster 2021-03-24T21:47:34.573095+0000 mgr.y (mgr.14144) 10 2021-03-24T21:47:36.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:35 smithi125 conmon[19936]: : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:47:38.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:37 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:47:38.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:37 smithi125 conmon[19936]: :47:36.573451+0000 mgr.y (mgr.14144) 11 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:47:40.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:39 smithi125 conmon[19936]: cluster 2021-03-24T21:47:38.573846+0000 mgr.y (mgr.14144) 12 2021-03-24T21:47:40.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:39 smithi125 conmon[19936]: : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:47:42.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:41 smithi125 conmon[19936]: cluster 2021-03-24T21:47:40.574211+0000 mgr.y (mgr.14144 2021-03-24T21:47:42.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:41 smithi125 conmon[19936]: ) 13 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:47:44.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:43 smithi125 conmon[19936]: cluster 2021-03-24T21:47:42.574585+0000 mgr.y (mgr.14144) 2021-03-24T21:47:44.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:43 smithi125 conmon[19936]: 14 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:47:46.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:45 smithi125 conmon[19936]: cluster 2021-03-24T21:47:44.574953+0000 mgr.y (mgr.14144) 15 : 2021-03-24T21:47:46.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:45 smithi125 conmon[19936]: cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:47:48.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:47 smithi125 conmon[19936]: cluster 2021-03-24T21:47:46.575329+0000 mgr.y (mgr.14144) 2021-03-24T21:47:48.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:47 smithi125 conmon[19936]: 16 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:47:50.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:49 smithi125 conmon[19936]: cluster 2021-03-24T21:47:48.575698+0000 mgr.y (mgr.14144) 17 2021-03-24T21:47:50.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:49 smithi125 conmon[19936]: : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:47:52.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:51 smithi125 conmon[19936]: cluster 2021-03-24T21:47:50.576078+0000 mgr.y (mgr.14144) 18 : cluster 2021-03-24T21:47:52.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:51 smithi125 conmon[19936]: [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:47:54.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:53 smithi125 conmon[19936]: cluster 2021-03-24T21:47:52.576468+0000 mgr.y (mgr.14144) 19 2021-03-24T21:47:54.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:53 smithi125 conmon[19936]: : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:47:56.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:55 smithi125 conmon[19936]: cluster 2021-03-24T21:47:54.576823+0000 mgr.y (mgr.14144) 20 : cluster 2021-03-24T21:47:56.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:55 smithi125 conmon[19936]: [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:47:58.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:57 smithi125 conmon[19936]: cluster 2021-03-24T21:47:56.577212+0000 mgr.y (mgr.14144) 2021-03-24T21:47:58.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:57 smithi125 conmon[19936]: 21 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:00.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:59 smithi125 conmon[19936]: cluster 2021-03-24T21:47:58. 2021-03-24T21:48:00.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:47:59 smithi125 conmon[19936]: 577607+0000 mgr.y (mgr.14144) 22 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:02.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:01 smithi125 conmon[19936]: cluster 2021-03-24T21:48:00.577989+0000 mgr.y (mgr.14144) 23 : 2021-03-24T21:48:02.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:01 smithi125 conmon[19936]: cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:03.653 INFO:journalctl@ceph.mgr.y.smithi125.stdout:Mar 24 21:48:03 smithi125 conmon[20260]: Warning: Permanently added 'smithi142,172.21.15.142' (ECDSA) to the list of known hosts. 2021-03-24T21:48:03.952 INFO:teuthology.orchestra.run.smithi142.stdout:Added host 'smithi142' 2021-03-24T21:48:04.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:03 smithi125 conmon[19936]: cluster 2021-03-24T21:48 2021-03-24T21:48:04.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:03 smithi125 conmon[19936]: :02.578380+0000 mgr.y (mgr.14144) 24 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:04.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:03 smithi125 conmon[19936]: audit 2021-03- 2021-03-24T21:48:04.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:03 smithi125 conmon[19936]: 24T21:48:03.144918+0000 mgr.y (mgr.14144) 25 : audit [DBG] from='client.14162 v1:172.21.15.142:0/356246719' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi142", "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:48:04.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:03 smithi125 conmon[19936]: 2021-03-24T21:48:04.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:03 smithi125 conmon[19936]: audit 2021 2021-03-24T21:48:04.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:03 smithi125 conmon[19936]: -03-24T21:48:03.956259+0000 mon.a (mon.0) 95 : audit [INF] 2021-03-24T21:48:04.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:03 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:48:04.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:03 smithi125 conmon[19936]: audit 2021-03-24T21:48:03. 2021-03-24T21:48:04.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:03 smithi125 conmon[19936]: 957736+0000 mon.a (mon.0) 96 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:48:04.779 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph orch host ls --format=json 2021-03-24T21:48:05.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:05 smithi125 conmon[19936]: cephadm 2021-03-24T21:48:03.957109+0000 mgr.y (mgr.14144) 26 2021-03-24T21:48:05.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:05 smithi125 conmon[19936]: : cephadm [INF] Added host smithi142 2021-03-24T21:48:05.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:05 smithi125 conmon[19936]: audit 2021-03-24T21:48:04.275382+0000 mon.a (mon.0 2021-03-24T21:48:05.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:05 smithi125 conmon[19936]: ) 97 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:48:05.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:05 smithi125 conmon[19936]: audit 2021-03-24T21 2021-03-24T21:48:05.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:05 smithi125 conmon[19936]: :48:04.726732+0000 mon.a (mon.0) 98 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T21:48:05.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:05 smithi125 conmon[19936]: audit 2021-03-24T21: 2021-03-24T21:48:05.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:05 smithi125 conmon[19936]: 48:04.730596+0000 mon.a (mon.0) 99 : 2021-03-24T21:48:05.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:05 smithi125 conmon[19936]: audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T21:48:05.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:05 smithi125 conmon[19936]: audit 2021-03-24T21:48:04 2021-03-24T21:48:05.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:05 smithi125 conmon[19936]: .810097+0000 mon.a (mon.0) 100 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:48:05.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:05 smithi125 conmon[19936]: 2021-03-24T21:48:06.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:06 smithi125 conmon[19936]: cluster 2021-03-24T21:48 2021-03-24T21:48:06.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:06 smithi125 conmon[19936]: :04.578817+0000 mgr.y (mgr.14144) 27 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:07.448 INFO:teuthology.orchestra.run.smithi142.stdout: 2021-03-24T21:48:07.448 INFO:teuthology.orchestra.run.smithi142.stdout:[{"addr": "smithi125", "hostname": "smithi125", "labels": [], "status": ""}, {"addr": "smithi142", "hostname": "smithi142", "labels": [], "status": ""}] 2021-03-24T21:48:08.198 INFO:tasks.cephadm:Setting crush tunables to default 2021-03-24T21:48:08.199 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd crush tunables default 2021-03-24T21:48:08.636 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:08 smithi125 conmon[19936]: cluster 2021-03-24T21:48: 2021-03-24T21:48:08.637 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:08 smithi125 conmon[19936]: 06.579244+0000 mgr.y (mgr.14144) 28 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:09.544 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:09 smithi125 conmon[19936]: audit 2021-03-24T21:48:07.452813+0000 mgr.y (mgr.14144) 29 2021-03-24T21:48:09.545 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:09 smithi125 conmon[19936]: : audit [DBG] from='client.14164 v1:172.21.15.142:0/1630139042' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T21:48:10.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:10 smithi125 conmon[19936]: cluster 2021-03-24T21:48:08.579646+0000 mgr.y (mgr.14144) 2021-03-24T21:48:10.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:10 smithi125 conmon[19936]: 30 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:10.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:10 smithi125 conmon[19936]: audit 2021-03-24T21:48:09 2021-03-24T21:48:10.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:10 smithi125 conmon[19936]: .757086+0000 mon.a (mon.0) 101 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:48:10.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:10 smithi125 conmon[19936]: audit 2021 2021-03-24T21:48:10.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:10 smithi125 conmon[19936]: -03-24T21:48: 2021-03-24T21:48:10.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:10 smithi125 conmon[19936]: 10.106909+0000 mon.a (mon.0) 2021-03-24T21:48:10.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:10 smithi125 conmon[19936]: 102 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:48:10.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:10 smithi125 conmon[19936]: audit 2021-03 2021-03-24T21:48:10.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:10 smithi125 conmon[19936]: -24T21:48:10.113584 2021-03-24T21:48:10.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:10 smithi125 conmon[19936]: +0000 mon.a (mon.0) 103 : audit 2021-03-24T21:48:10.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:10 smithi125 conmon[19936]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:48:10.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:10 smithi125 conmon[19936]: audit 2021-03-24 2021-03-24T21:48:10.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:10 smithi125 conmon[19936]: T21:48:10.229738+0000 mon.a 2021-03-24T21:48:10.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:10 smithi125 conmon[19936]: (mon.0) 104 : audit [INF] from='client.? v1:172.21.15.125:0/575061834' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2021-03-24T21:48:10.657 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:10 smithi125 conmon[19936]: 2021-03-24T21:48:11.114 INFO:teuthology.orchestra.run.smithi125.stderr:adjusted tunables profile to default 2021-03-24T21:48:11.877 INFO:tasks.cephadm:Adding mon.a on smithi125 2021-03-24T21:48:11.878 INFO:tasks.cephadm:Adding mon.c on smithi125 2021-03-24T21:48:11.878 INFO:tasks.cephadm:Adding mon.b on smithi142 2021-03-24T21:48:11.878 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph orch apply mon '3;smithi125:[v1:172.21.15.125:6789]=a;smithi125:[v1:172.21.15.125:6790]=c;smithi142:[v1:172.21.15.142:6789]=b' 2021-03-24T21:48:12.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:12 smithi125 conmon[19936]: cluster 2021-03-24T21:48:10.580135+0000 mgr.y (mgr.14144) 31 2021-03-24T21:48:12.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:12 smithi125 conmon[19936]: : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:12.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:12 smithi125 conmon[19936]: audit 2021-03-24T21:48:11.114968+0000 mon.a (mon.0 2021-03-24T21:48:12.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:12 smithi125 conmon[19936]: ) 105 : audit [INF] from='client.? v1:172.21.15.125:0/575061834' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2021-03-24T21:48:12.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:12 smithi125 conmon[19936]: cluster 2021-03-24T21:48:12.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:12 smithi125 conmon[19936]: 2021-03-24T21:48:11.115095+0000 mon.a (mon.0) 106 2021-03-24T21:48:12.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:12 smithi125 conmon[19936]: : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2021-03-24T21:48:12.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:12 smithi125 conmon[19936]: 2021-03-24T21:48:13.908 INFO:teuthology.orchestra.run.smithi142.stdout:Scheduled mon update... 2021-03-24T21:48:14.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:14 smithi125 conmon[19936]: cluster 2021-03-24T21:48:12.580641+0000 mgr.y (mgr.14144) 32 : cluster [DBG] 2021-03-24T21:48:14.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:14 smithi125 conmon[19936]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:14.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:14 smithi125 conmon[19936]: audit 2021-03-24T21:48:13.914099+0000 mon.a (mon.0) 2021-03-24T21:48:14.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:14 smithi125 conmon[19936]: 107 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:48:14.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:14 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:48:14.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:14 smithi125 conmon[19936]: 03-24T21:48:13.915308 2021-03-24T21:48:14.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:14 smithi125 conmon[19936]: +0000 mon.a (mon.0) 108 2021-03-24T21:48:14.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:14 smithi125 conmon[19936]: : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:48:14.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:14 smithi125 conmon[19936]: audit 2021 2021-03-24T21:48:14.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:14 smithi125 conmon[19936]: -03-24T21:48:13 2021-03-24T21:48:14.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:14 smithi125 conmon[19936]: .923649+0000 mon.a (mon.0 2021-03-24T21:48:14.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:14 smithi125 conmon[19936]: ) 109 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:48:14.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:14 smithi125 conmon[19936]: audit 2021 2021-03-24T21:48:14.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:14 smithi125 conmon[19936]: -03-24T21:48:13 2021-03-24T21:48:14.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:14 smithi125 conmon[19936]: .925640+0000 mon.a (mon.0 2021-03-24T21:48:14.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:14 smithi125 conmon[19936]: ) 110 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-03-24T21:48:14.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:14 smithi125 conmon[19936]: audit 2021 2021-03-24T21:48:14.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:14 smithi125 conmon[19936]: -03-24T21:48:13 2021-03-24T21:48:14.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:14 smithi125 conmon[19936]: .927138+0000 mon.a (mon.0 2021-03-24T21:48:14.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:14 smithi125 conmon[19936]: ) 111 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:48:14.621 DEBUG:teuthology.orchestra.run.smithi125:mon.c> sudo journalctl -f -n 0 -u ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c@mon.c.service 2021-03-24T21:48:14.624 DEBUG:teuthology.orchestra.run.smithi142:mon.b> sudo journalctl -f -n 0 -u ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c@mon.b.service 2021-03-24T21:48:14.627 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2021-03-24T21:48:14.627 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph mon dump -f json 2021-03-24T21:48:14.672 INFO:journalctl@ceph.mon.c.smithi125.stdout:-- Logs begin at Wed 2021-03-03 23:51:07 UTC. -- 2021-03-24T21:48:14.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:-- Logs begin at Wed 2021-03-03 23:51:07 UTC. -- 2021-03-24T21:48:15.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:15 smithi125 conmon[19936]: audit 2021-03-24T21:48:13.906924+0000 mgr.y (mgr.14144) 33 : audit [DBG] 2021-03-24T21:48:15.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:15 smithi125 conmon[19936]: from='client.14168 v1:172.21.15.142:0/2479045516' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi125:[v1:172.21.15.125:6789]=a;smithi125:[v1:172.21.15.125:6790]=c;smithi142:[v1:172.21.15.142:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:48:15.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:15 smithi125 conmon[19936]: cephadm 2021-03-24T21:48:13.909742+0000 mgr.y (mgr. 2021-03-24T21:48:15.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:15 smithi125 conmon[19936]: 14144) 34 : cephadm [INF] Saving service mon spec with placement smithi125:[v1:172.21.15.125:6789]=a;smithi125:[v1:172.21.15.125:6790]=c;smithi142:[v1:172.21.15.142:6789]=b;count:3 2021-03-24T21:48:15.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:15 smithi125 conmon[19936]: cephadm 2021-03-24T 2021-03-24T21:48:15.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:15 smithi125 conmon[19936]: 21:48:13.928681+0000 mgr.y (mgr 2021-03-24T21:48:15.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:15 smithi125 conmon[19936]: .14144) 35 : cephadm [INF] Deploying daemon mon.b on smithi142 2021-03-24T21:48:16.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:16 smithi125 conmon[19936]: cluster 2021-03-24T21:48: 2021-03-24T21:48:16.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:16 smithi125 conmon[19936]: 14.581056+0000 mgr.y (mgr.14144) 36 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:18.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:18 smithi125 conmon[19936]: cluster 2021-03-24T21:48 2021-03-24T21:48:18.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:18 smithi125 conmon[19936]: :16.581509+0000 mgr.y (mgr.14144) 37 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:18.404 INFO:teuthology.orchestra.run.smithi142.stdout: 2021-03-24T21:48:18.404 INFO:teuthology.orchestra.run.smithi142.stdout:{"epoch":1,"fsid":"3cfa339a-8cea-11eb-90b2-001a4aab830c","modified":"2021-03-24T21:46:15.748211Z","created":"2021-03-24T21:46:15.748211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6789","nonce":0}]},"addr":"172.21.15.125:6789/0","public_addr":"172.21.15.125:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2021-03-24T21:48:18.407 INFO:teuthology.orchestra.run.smithi142.stderr:dumped monmap epoch 1 2021-03-24T21:48:19.113 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 systemd[1]: Starting Ceph mon.b for 3cfa339a-8cea-11eb-90b2-001a4aab830c... 2021-03-24T21:48:19.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:19 smithi125 conmon[19936]: audit 2021-03-24T21: 2021-03-24T21:48:19.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:19 smithi125 conmon[19936]: 48:18.410063+0000 mon.a (mon.0) 112 : audit [DBG] from='client.? v1:172.21.15.142:0/161333581' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-03-24T21:48:20.114 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 podman[20084]: 2021-03-24 21:48:19.684711932 +0000 UTC m=+0.205311958 container create 9ad3bd69664523c515c6b2760a2601735aa7b1a475ad4d7fc18067147ac5db77 (image=quay.ceph.io/ceph-ci/ceph@sha256:e9db06906b8d0a24f3ef6835ac5aa4b8eea2950a2560bad78f83d4f101eca0db, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-mon.b, io.buildah.version=1.18.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20201204, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, maintainer=Dimitri Savineau , CEPH_POINT_RELEASE=, GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06, RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2021-03-24T21:48:20.115 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 podman[20084]: 2021-03-24 21:48:19.817980155 +0000 UTC m=+0.338580184 container init 9ad3bd69664523c515c6b2760a2601735aa7b1a475ad4d7fc18067147ac5db77 (image=quay.ceph.io/ceph-ci/ceph@sha256:e9db06906b8d0a24f3ef6835ac5aa4b8eea2950a2560bad78f83d4f101eca0db, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-mon.b, maintainer=Dimitri Savineau , org.label-schema.name=CentOS Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20201204, io.buildah.version=1.18.0, GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06) 2021-03-24T21:48:20.115 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.852+0000 7f5ed7f66700 0 set uid:gid to 167:167 (ceph:ceph) 2021-03-24T21:48:20.115 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.852+0000 7f5ed7f66700 0 ceph version 16.1.0-1166-gd71e96d2 (d71e96d216190f28f757f409532464ab0eca4ea9) pacific (rc), process ceph-mon, pid 2 2021-03-24T21:48:20.115 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.852+0000 7f5ed7f66700 0 pidfile_write: ignore empty --pid-file 2021-03-24T21:48:20.116 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 0 load: jerasure load: lrc load: isa 2021-03-24T21:48:20.116 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: RocksDB version: 6.8.1 2021-03-24T21:48:20.116 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: 2021-03-24T21:48:20.116 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2021-03-24T21:48:20.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Compile date Mar 24 2021 2021-03-24T21:48:20.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: DB SUMMARY 2021-03-24T21:48:20.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: 2021-03-24T21:48:20.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: CURRENT file: CURRENT 2021-03-24T21:48:20.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: 2021-03-24T21:48:20.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: IDENTITY file: IDENTITY 2021-03-24T21:48:20.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: 2021-03-24T21:48:20.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: MANIFEST file: MANIFEST-000001 size: 13 Bytes 2021-03-24T21:48:20.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: 2021-03-24T21:48:20.119 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2021-03-24T21:48:20.119 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: 2021-03-24T21:48:20.119 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000003.log size: 476 ; 2021-03-24T21:48:20.119 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: 2021-03-24T21:48:20.119 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.error_if_exists: 0 2021-03-24T21:48:20.120 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.create_if_missing: 0 2021-03-24T21:48:20.122 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.paranoid_checks: 1 2021-03-24T21:48:20.122 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.env: 0x560f73e691c0 2021-03-24T21:48:20.122 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.fs: Posix File System 2021-03-24T21:48:20.123 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.info_log: 0x560f75c4e6a0 2021-03-24T21:48:20.123 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.max_file_opening_threads: 16 2021-03-24T21:48:20.123 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.statistics: (nil) 2021-03-24T21:48:20.123 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.use_fsync: 0 2021-03-24T21:48:20.123 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.max_log_file_size: 0 2021-03-24T21:48:20.124 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2021-03-24T21:48:20.124 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.log_file_time_to_roll: 0 2021-03-24T21:48:20.124 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.keep_log_file_num: 1000 2021-03-24T21:48:20.124 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.recycle_log_file_num: 0 2021-03-24T21:48:20.124 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.allow_fallocate: 1 2021-03-24T21:48:20.125 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.allow_mmap_reads: 0 2021-03-24T21:48:20.125 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.allow_mmap_writes: 0 2021-03-24T21:48:20.125 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.use_direct_reads: 0 2021-03-24T21:48:20.125 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:20.126 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2021-03-24T21:48:20.126 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.create_missing_column_families: 0 2021-03-24T21:48:20.126 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.db_log_dir: 2021-03-24T21:48:20.126 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-b/store.db 2021-03-24T21:48:20.126 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.table_cache_numshardbits: 6 2021-03-24T21:48:20.127 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.max_subcompactions: 1 2021-03-24T21:48:20.127 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.max_background_flushes: -1 2021-03-24T21:48:20.127 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.WAL_ttl_seconds: 0 2021-03-24T21:48:20.127 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.WAL_size_limit_MB: 0 2021-03-24T21:48:20.128 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2021-03-24T21:48:20.128 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2021-03-24T21:48:20.128 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.is_fd_close_on_exec: 1 2021-03-24T21:48:20.128 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.advise_random_on_open: 1 2021-03-24T21:48:20.128 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.db_write_buffer_size: 0 2021-03-24T21:48:20.129 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.write_buffer_manager: 0x560f75ce7d40 2021-03-24T21:48:20.129 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2021-03-24T21:48:20.129 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2021-03-24T21:48:20.129 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2021-03-24T21:48:20.129 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.use_adaptive_mutex: 0 2021-03-24T21:48:20.130 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.rate_limiter: (nil) 2021-03-24T21:48:20.130 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2021-03-24T21:48:20.130 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.wal_recovery_mode: 2 2021-03-24T21:48:20.130 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.enable_thread_tracking: 0 2021-03-24T21:48:20.130 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.enable_pipelined_write: 0 2021-03-24T21:48:20.131 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.unordered_write: 0 2021-03-24T21:48:20.131 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2021-03-24T21:48:20.131 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: 2021-03-24T21:48:20.131 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2021-03-24T21:48:20.132 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2021-03-24T21:48:20.132 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2021-03-24T21:48:20.132 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.row_cache: None 2021-03-24T21:48:20.132 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.wal_filter: None 2021-03-24T21:48:20.133 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2021-03-24T21:48:20.133 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.allow_ingest_behind: 0 2021-03-24T21:48:20.133 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.preserve_deletes: 0 2021-03-24T21:48:20.133 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.two_write_queues: 0 2021-03-24T21:48:20.133 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 bash[20084]: 9ad3bd69664523c515c6b2760a2601735aa7b1a475ad4d7fc18067147ac5db77 2021-03-24T21:48:20.134 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.manual_wal_flush: 0 2021-03-24T21:48:20.134 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.atomic_flush: 0 2021-03-24T21:48:20.134 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2021-03-24T21:48:20.134 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.persist_stats_to_disk: 0 2021-03-24T21:48:20.135 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.write_dbid_to_manifest: 0 2021-03-24T21:48:20.135 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.log_readahead_size: 0 2021-03-24T21:48:20.135 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2021-03-24T21:48:20.135 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.max_background_jobs: 2 2021-03-24T21:48:20.135 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.max_background_compactions: -1 2021-03-24T21:48:20.136 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2021-03-24T21:48:20.136 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2021-03-24T21:48:20.136 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.delayed_write_rate : 16777216 2021-03-24T21:48:20.136 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.max_total_wal_size: 0 2021-03-24T21:48:20.136 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2021-03-24T21:48:20.137 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:20.137 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.stats_dump_period_sec: 600 2021-03-24T21:48:20.137 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.stats_persist_period_sec: 600 2021-03-24T21:48:20.137 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2021-03-24T21:48:20.137 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.max_open_files: -1 2021-03-24T21:48:20.138 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.bytes_per_sync: 0 2021-03-24T21:48:20.138 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.wal_bytes_per_sync: 0 2021-03-24T21:48:20.138 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.strict_bytes_per_sync: 0 2021-03-24T21:48:20.138 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.compaction_readahead_size: 0 2021-03-24T21:48:20.139 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Compression algorithms supported: 2021-03-24T21:48:20.139 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2021-03-24T21:48:20.139 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: kZSTD supported: 0 2021-03-24T21:48:20.139 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: kXpressCompression supported: 0 2021-03-24T21:48:20.139 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: kLZ4HCCompression supported: 1 2021-03-24T21:48:20.140 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: kLZ4Compression supported: 1 2021-03-24T21:48:20.140 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: kBZip2Compression supported: 0 2021-03-24T21:48:20.140 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: kZlibCompression supported: 1 2021-03-24T21:48:20.140 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:20.140 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: kSnappyCompression supported: 1 2021-03-24T21:48:20.141 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Fast CRC32 supported: Supported on x86 2021-03-24T21:48:20.141 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000001 2021-03-24T21:48:20.141 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: 2021-03-24T21:48:20.141 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2021-03-24T21:48:20.141 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: 2021-03-24T21:48:20.142 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2021-03-24T21:48:20.142 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.merge_operator: 2021-03-24T21:48:20.142 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.compaction_filter: None 2021-03-24T21:48:20.142 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.compaction_filter_factory: None 2021-03-24T21:48:20.143 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.memtable_factory: SkipListFactory 2021-03-24T21:48:20.143 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.table_factory: BlockBasedTable 2021-03-24T21:48:20.143 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560f75bf2218) 2021-03-24T21:48:20.143 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: cache_index_and_filter_blocks: 1 2021-03-24T21:48:20.143 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: cache_index_and_filter_blocks_with_high_priority: 0 2021-03-24T21:48:20.144 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: pin_l0_filter_and_index_blocks_in_cache: 0 2021-03-24T21:48:20.144 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: pin_top_level_index_and_filter: 1 2021-03-24T21:48:20.144 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: index_type: 0 2021-03-24T21:48:20.144 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: data_block_index_type: 0 2021-03-24T21:48:20.144 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: index_shortening: 1 2021-03-24T21:48:20.145 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: data_block_hash_table_util_ratio: 0.750000 2021-03-24T21:48:20.145 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: hash_index_allow_collision: 1 2021-03-24T21:48:20.145 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: checksum: 1 2021-03-24T21:48:20.145 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: no_block_cache: 0 2021-03-24T21:48:20.145 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: block_cache: 0x560f75c2af10 2021-03-24T21:48:20.146 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: block_cache_name: BinnedLRUCache 2021-03-24T21:48:20.146 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: block_cache_options: 2021-03-24T21:48:20.146 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: capacity : 536870912 2021-03-24T21:48:20.146 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: num_shard_bits : 4 2021-03-24T21:48:20.146 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: strict_capacity_limit : 0 2021-03-24T21:48:20.147 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: high_pri_pool_ratio: 0.000 2021-03-24T21:48:20.147 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: block_cache_compressed: (nil) 2021-03-24T21:48:20.147 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: persistent_cache: (nil) 2021-03-24T21:48:20.147 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: block_size: 4096 2021-03-24T21:48:20.148 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: block_size_deviation: 10 2021-03-24T21:48:20.148 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: block_restart_interval: 16 2021-03-24T21:48:20.148 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: index_block_restart_interval: 1 2021-03-24T21:48:20.148 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: metadata_block_size: 4096 2021-03-24T21:48:20.148 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: partition_filters: 0 2021-03-24T21:48:20.149 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: use_delta_encoding: 1 2021-03-24T21:48:20.149 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: filter_policy: rocksdb.BuiltinBloomFilter 2021-03-24T21:48:20.149 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: whole_key_filtering: 1 2021-03-24T21:48:20.149 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: verify_compression: 0 2021-03-24T21:48:20.149 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: read_amp_bytes_per_bit: 0 2021-03-24T21:48:20.150 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: format_version: 2 2021-03-24T21:48:20.150 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: enable_index_compression: 1 2021-03-24T21:48:20.150 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: block_align: 0 2021-03-24T21:48:20.150 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: 2021-03-24T21:48:20.151 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.write_buffer_size: 33554432 2021-03-24T21:48:20.151 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.max_write_buffer_number: 2 2021-03-24T21:48:20.151 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.compression: NoCompression 2021-03-24T21:48:20.151 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.bottommost_compression: Disabled 2021-03-24T21:48:20.151 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.prefix_extractor: nullptr 2021-03-24T21:48:20.152 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2021-03-24T21:48:20.152 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: 2021-03-24T21:48:20.152 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.num_levels: 7 2021-03-24T21:48:20.152 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2021-03-24T21:48:20.152 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2021-03-24T21:48:20.153 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2021-03-24T21:48:20.153 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2021-03-24T21:48:20.153 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2021-03-24T21:48:20.153 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2021-03-24T21:48:20.153 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2021-03-24T21:48:20.154 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2021-03-24T21:48:20.154 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2021-03-24T21:48:20.154 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.compression_opts.window_bits: -14 2021-03-24T21:48:20.154 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.compression_opts.level: 32767 2021-03-24T21:48:20.155 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.compression_opts.strategy: 0 2021-03-24T21:48:20.155 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2021-03-24T21:48:20.155 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2021-03-24T21:48:20.155 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.compression_opts.enabled: false 2021-03-24T21:48:20.155 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2021-03-24T21:48:20.156 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2021-03-24T21:48:20.156 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2021-03-24T21:48:20.156 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.target_file_size_base: 67108864 2021-03-24T21:48:20.156 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.target_file_size_multiplier: 1 2021-03-24T21:48:20.156 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2021-03-24T21:48:20.157 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2021-03-24T21:48:20.157 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2021-03-24T21:48:20.157 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2021-03-24T21:48:20.157 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2021-03-24T21:48:20.157 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2021-03-24T21:48:20.158 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2021-03-24T21:48:20.158 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2021-03-24T21:48:20.158 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2021-03-24T21:48:20.158 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2021-03-24T21:48:20.158 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2021-03-24T21:48:20.159 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2021-03-24T21:48:20.159 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.arena_block_size: 4194304 2021-03-24T21:48:20.159 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2021-03-24T21:48:20.159 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2021-03-24T21:48:20.159 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2021-03-24T21:48:20.160 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.disable_auto_compactions: 0 2021-03-24T21:48:20.160 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2021-03-24T21:48:20.160 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2021-03-24T21:48:20.160 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2021-03-24T21:48:20.161 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2021-03-24T21:48:20.161 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2021-03-24T21:48:20.161 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2021-03-24T21:48:20.161 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2021-03-24T21:48:20.161 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2021-03-24T21:48:20.162 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2021-03-24T21:48:20.162 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2021-03-24T21:48:20.162 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.table_properties_collectors: 2021-03-24T21:48:20.162 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.inplace_update_support: 0 2021-03-24T21:48:20.162 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.inplace_update_num_locks: 10000 2021-03-24T21:48:20.163 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2021-03-24T21:48:20.163 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2021-03-24T21:48:20.163 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.memtable_huge_page_size: 0 2021-03-24T21:48:20.163 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.bloom_locality: 0 2021-03-24T21:48:20.163 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.max_successive_merges: 0 2021-03-24T21:48:20.164 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.optimize_filters_for_hits: 0 2021-03-24T21:48:20.164 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.paranoid_file_checks: 0 2021-03-24T21:48:20.164 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.force_consistency_checks: 0 2021-03-24T21:48:20.164 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.report_bg_io_stats: 0 2021-03-24T21:48:20.165 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.ttl: 2592000 2021-03-24T21:48:20.165 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.856+0000 7f5ed7f66700 4 rocksdb: Options.periodic_compaction_seconds: 0 2021-03-24T21:48:20.165 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.860+0000 7f5ed7f66700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2021-03-24T21:48:20.165 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: 2021-03-24T21:48:20.165 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.860+0000 7f5ed7f66700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2021-03-24T21:48:20.166 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: 2021-03-24T21:48:20.166 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.860+0000 7f5ed7f66700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616622499862329, "job": 1, "event": "recovery_started", "log_files": [3]} 2021-03-24T21:48:20.166 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.860+0000 7f5ed7f66700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #3 mode 2 2021-03-24T21:48:20.166 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.860+0000 7f5ed7f66700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2021-03-24T21:48:20.166 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.860+0000 7f5ed7f66700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616622499863597, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 4, "file_size": 1402, "table_properties": {"data_size": 488, "index_size": 40, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 366, "raw_average_value_size": 73, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1616622499, "oldest_key_time": 3, "file_creation_time": 0}} 2021-03-24T21:48:20.167 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.860+0000 7f5ed7f66700 4 rocksdb: [version_set.cc:3826] Creating manifest 5 2021-03-24T21:48:20.167 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: 2021-03-24T21:48:20.167 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.860+0000 7f5ed7f66700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616622499864515, "job": 1, "event": "recovery_finished"} 2021-03-24T21:48:20.167 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.864+0000 7f5ed7f66700 4 rocksdb: DB pointer 0x560f75cf1800 2021-03-24T21:48:20.167 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.864+0000 7f5ebed83700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2021-03-24T21:48:20.168 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.864+0000 7f5ebed83700 4 rocksdb: [db_impl/db_impl.cc:851] 2021-03-24T21:48:20.168 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: ** DB Stats ** 2021-03-24T21:48:20.168 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: Uptime(secs): 0.0 total, 0.0 interval 2021-03-24T21:48:20.168 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2021-03-24T21:48:20.168 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2021-03-24T21:48:20.169 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2021-03-24T21:48:20.169 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2021-03-24T21:48:20.169 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2021-03-24T21:48:20.169 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2021-03-24T21:48:20.170 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: 2021-03-24T21:48:20.170 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: ** Compaction Stats [default] ** 2021-03-24T21:48:20.170 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-24T21:48:20.170 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-24T21:48:20.170 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: L0 1/0 1.37 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.1 0.00 0.00 1 0.001 0 0 2021-03-24T21:48:20.171 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: Sum 1/0 1.37 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.1 0.00 0.00 1 0.001 0 0 2021-03-24T21:48:20.171 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.1 0.00 0.00 1 0.001 0 0 2021-03-24T21:48:20.171 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: 2021-03-24T21:48:20.171 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: ** Compaction Stats [default] ** 2021-03-24T21:48:20.171 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-24T21:48:20.172 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-24T21:48:20.172 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.1 0.00 0.00 1 0.001 0 0 2021-03-24T21:48:20.172 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: Uptime(secs): 0.0 total, 0.0 interval 2021-03-24T21:48:20.172 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: Flush(GB): cumulative 0.000, interval 0.000 2021-03-24T21:48:20.172 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: AddFile(GB): cumulative 0.000, interval 0.000 2021-03-24T21:48:20.173 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: AddFile(Total Files): cumulative 0, interval 0 2021-03-24T21:48:20.173 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: AddFile(L0 Files): cumulative 0, interval 0 2021-03-24T21:48:20.173 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: AddFile(Keys): cumulative 0, interval 0 2021-03-24T21:48:20.173 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: Cumulative compaction: 0.00 GB write, 0.25 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-24T21:48:20.174 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: Interval compaction: 0.00 GB write, 0.25 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-24T21:48:20.174 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-03-24T21:48:20.174 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: 2021-03-24T21:48:20.174 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: ** File Read Latency Histogram By Level [default] ** 2021-03-24T21:48:20.174 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: 2021-03-24T21:48:20.175 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: ** Compaction Stats [default] ** 2021-03-24T21:48:20.175 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-24T21:48:20.175 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-24T21:48:20.175 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: L0 1/0 1.37 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.1 0.00 0.00 1 0.001 0 0 2021-03-24T21:48:20.176 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: Sum 1/0 1.37 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.1 0.00 0.00 1 0.001 0 0 2021-03-24T21:48:20.176 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-03-24T21:48:20.176 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: 2021-03-24T21:48:20.176 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: ** Compaction Stats [default] ** 2021-03-24T21:48:20.176 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-24T21:48:20.177 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-24T21:48:20.177 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.1 0.00 0.00 1 0.001 0 0 2021-03-24T21:48:20.177 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: Uptime(secs): 0.0 total, 0.0 interval 2021-03-24T21:48:20.177 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: Flush(GB): cumulative 0.000, interval 0.000 2021-03-24T21:48:20.177 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: AddFile(GB): cumulative 0.000, interval 0.000 2021-03-24T21:48:20.178 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: AddFile(Total Files): cumulative 0, interval 0 2021-03-24T21:48:20.178 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: AddFile(L0 Files): cumulative 0, interval 0 2021-03-24T21:48:20.178 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: AddFile(Keys): cumulative 0, interval 0 2021-03-24T21:48:20.178 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: Cumulative compaction: 0.00 GB write, 0.25 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-24T21:48:20.178 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-24T21:48:20.179 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-03-24T21:48:20.179 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: 2021-03-24T21:48:20.179 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: ** File Read Latency Histogram By Level [default] ** 2021-03-24T21:48:20.179 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: 2021-03-24T21:48:20.180 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.864+0000 7f5ed7f66700 0 mon.b does not exist in monmap, will attempt to join an existing cluster 2021-03-24T21:48:20.180 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.864+0000 7f5ed7f66700 0 using public_addrv v1:172.21.15.142:6789/0 2021-03-24T21:48:20.180 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 podman[20084]: 2021-03-24 21:48:19.867952867 +0000 UTC m=+0.388552911 container start 9ad3bd69664523c515c6b2760a2601735aa7b1a475ad4d7fc18067147ac5db77 (image=quay.ceph.io/ceph-ci/ceph@sha256:e9db06906b8d0a24f3ef6835ac5aa4b8eea2950a2560bad78f83d4f101eca0db, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-mon.b, ceph=True, org.label-schema.name=CentOS Base Image, GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Dimitri Savineau , org.label-schema.schema-version=1.0, org.label-schema.build-date=20201204, io.buildah.version=1.18.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2021-03-24T21:48:20.180 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.868+0000 7f5ed7f66700 0 starting mon.b rank -1 at public addrs v1:172.21.15.142:6789/0 at bind addrs v1:172.21.15.142:6789/0 mon_data /var/lib/ceph/mon/ceph-b fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c 2021-03-24T21:48:20.180 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.872+0000 7f5ed7f66700 1 mon.b@-1(???) e0 preinit fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c 2021-03-24T21:48:20.181 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 systemd[1]: Started Ceph mon.b for 3cfa339a-8cea-11eb-90b2-001a4aab830c. 2021-03-24T21:48:20.181 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.900+0000 7f5ebcd7f700 4 rocksdb: [db_impl/db_impl_write.cc:1668] [default] New memtable created with log file: #9. Immutable memtables: 0. 2021-03-24T21:48:20.181 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: 2021-03-24T21:48:20.181 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.900+0000 7f5ec6d93700 4 rocksdb: (Original Log Time 2021/03/24-21:48:19.903894) [db_impl/db_impl_compaction_flush.cc:2198] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 2021-03-24T21:48:20.181 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.900+0000 7f5ec6d93700 4 rocksdb: [flush_job.cc:321] [default] [JOB 3] Flushing memtable with next log file: 9 2021-03-24T21:48:20.182 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: 2021-03-24T21:48:20.182 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.900+0000 7f5ec6d93700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616622499903988, "job": 3, "event": "flush_started", "num_memtables": 1, "num_entries": 1077, "num_deletes": 20, "total_data_size": 3525509, "memory_usage": 3545592, "flush_reason": "Manual Compaction"} 2021-03-24T21:48:20.182 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.900+0000 7f5ec6d93700 4 rocksdb: [flush_job.cc:350] [default] [JOB 3] Level-0 flush table #10: started 2021-03-24T21:48:20.182 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.900+0000 7f5ec1588700 0 mon.b@-1(synchronizing).mds e1 new map 2021-03-24T21:48:20.183 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.900+0000 7f5ec1588700 0 mon.b@-1(synchronizing).mds e1 print_map 2021-03-24T21:48:20.183 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: e1 2021-03-24T21:48:20.183 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: enable_multiple, ever_enabled_multiple: 1,1 2021-03-24T21:48:20.183 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2021-03-24T21:48:20.183 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: legacy client fscid: -1 2021-03-24T21:48:20.184 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: 2021-03-24T21:48:20.184 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: No filesystems configured 2021-03-24T21:48:20.184 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: 2021-03-24T21:48:20.184 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.900+0000 7f5ec1588700 1 mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2021-03-24T21:48:20.184 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.900+0000 7f5ec1588700 1 mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2021-03-24T21:48:20.185 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.900+0000 7f5ec1588700 1 mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2021-03-24T21:48:20.185 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.900+0000 7f5ec1588700 1 mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2021-03-24T21:48:20.185 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.900+0000 7f5ec1588700 1 mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2021-03-24T21:48:20.185 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:20.186 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: 2021-03-24T21:48:19.900+0000 7f5ec1588700 1 mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2021-03-24T21:48:20.186 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:20.186 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: 2021-03-24T21:48:19.900+0000 7f5ec1588700 0 mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2021-03-24T21:48:20.186 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: 2021-03-24T21:48:20.186 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:20.187 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: 2021-03-24T21:48:19.900+0000 7f5ec1588700 0 mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2021-03-24T21:48:20.187 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: 2021-03-24T21:48:20.187 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:20.187 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: 2021-03-24T21:48:19.900+0000 7f5ec1588700 0 mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2021-03-24T21:48:20.187 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: 2021-03-24T21:48:20.188 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: audit 2021-03-24T21:48:20.188 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T2021-03-24T21:48:19.900+0000 7f5ec1588700 0 mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2021-03-24T21:48:20.188 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: 21 2021-03-24T21:48:20.188 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: :48:18.410063+0000 mon.a (mon.0) 112 : audit [DBG] from='client.? v1:172.21.15.142:0/161333581' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-03-24T21:48:20.189 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: 2021-03-24T21:48:20.189 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.904+0000 7f5ec1588700 1 mon.b@-1(synchronizing).paxosservice(auth 1..4) refresh upgraded, format 0 -> 3 2021-03-24T21:48:20.189 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.904+0000 7f5ec1588700 20 expand_channel_meta expand map: {default=false} 2021-03-24T21:48:20.189 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.904+0000 7f5ec1588700 20 expand_channel_meta from 'false' to 'false' 2021-03-24T21:48:20.189 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.904+0000 7f5ec1588700 20 expand_channel_meta expanded map: {default=false} 2021-03-24T21:48:20.190 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.904+0000 7f5ec1588700 20 expand_channel_meta expand map: {default=info} 2021-03-24T21:48:20.190 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.904+0000 7f5ec1588700 20 expand_channel_meta from 'info' to 'info' 2021-03-24T21:48:20.190 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.904+0000 7f5ec1588700 20 expand_channel_meta expanded map: {default=info} 2021-03-24T21:48:20.190 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.904+0000 7f5ec1588700 20 expand_channel_meta expand map: {default=daemon} 2021-03-24T21:48:20.190 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.904+0000 7f5ec1588700 20 expand_channel_meta from 'daemon' to 'daemon' 2021-03-24T21:48:20.191 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.904+0000 7f5ec1588700 20 expand_channel_meta expanded map: {default=daemon} 2021-03-24T21:48:20.191 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.904+0000 7f5ec1588700 20 expand_channel_meta expand map: {default=debug} 2021-03-24T21:48:20.191 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:20.191 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: 2021-03-24T21:48:19.904+0000 7f5ec1588700 20 expand_channel_meta from 'debug' to 'debug' 2021-03-24T21:48:20.192 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.904+0000 7f5ec6d93700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616622499909149, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 10, "file_size": 3329928, "table_properties": {"data_size": 3321780, "index_size": 5696, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 1605, "raw_key_size": 18401, "raw_average_key_size": 29, "raw_value_size": 3306193, "raw_average_value_size": 5247, "num_data_blocks": 185, "num_entries": 630, "num_deletions": 20, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1616622499, "oldest_key_time": 1616622499, "file_creation_time": 1616622499}} 2021-03-24T21:48:20.192 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.904+0000 7f5ec1588700 20 expand_channel_meta expanded map: {default=debug} 2021-03-24T21:48:20.192 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.904+0000 7f5ec6d93700 4 rocksdb: [flush_job.cc:401] [default] [JOB 3] Level-0 flush table #10: 3329928 bytes OK 2021-03-24T21:48:20.192 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:19 smithi142 conmon[20125]: debug 2021-03-24T21:48:19.904+0000 7f5ec1588700 10 mon.b@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete,mon_osdmap_full_prune_interval,mon_osdmap_full_prune_min,mon_osdmap_full_prune_txsize 2021-03-24T21:48:20.402 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2021-03-24T21:48:20.402 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph mon dump -f json 2021-03-24T21:48:25.205 INFO:teuthology.orchestra.run.smithi142.stdout: 2021-03-24T21:48:25.205 INFO:teuthology.orchestra.run.smithi142.stdout:{"epoch":2,"fsid":"3cfa339a-8cea-11eb-90b2-001a4aab830c","modified":"2021-03-24T21:48:19.916096Z","created":"2021-03-24T21:46:15.748211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6789","nonce":0}]},"addr":"172.21.15.125:6789/0","public_addr":"172.21.15.125:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2021-03-24T21:48:25.208 INFO:teuthology.orchestra.run.smithi142.stderr:dumped monmap epoch 2 2021-03-24T21:48:25.288 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: audit 2021-03-24T21:48 2021-03-24T21:48:25.288 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: :19.921760+0000 mon.a (mon.0) 116 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:48:25.288 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: cluster 2021-03-24T21:48:25.289 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: 2021-03-24T21:48:19 2021-03-24T21:48:25.289 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: .922075+0000 mon.a (mon. 2021-03-24T21:48:25.289 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: 0) 117 : cluster [INF] mon.a calling monitor election 2021-03-24T21:48:25.290 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: 2021-03-24T21:48:25.290 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:48:25.290 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: -03-24T21:48: 2021-03-24T21:48:25.291 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: 20.582246+0000 mgr.y (mgr. 2021-03-24T21:48:25.291 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: 14144) 39 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:25.291 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: audit 2021 2021-03-24T21:48:25.292 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: -03-24T21:48:20 2021-03-24T21:48:25.292 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: .914782+0000 mon.a (mon.0 2021-03-24T21:48:25.293 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: ) 118 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-03-24T21:48:25.293 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:48:25.293 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: -03-24T21:48:21 2021-03-24T21:48:25.294 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: .914628+0000 mon.b (mon.1 2021-03-24T21:48:25.294 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: ) 1 : cluster [INF] mon.b calling monitor election 2021-03-24T21:48:25.294 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: audit 2021 2021-03-24T21:48:25.294 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: -03-24T21:48:21 2021-03-24T21:48:25.294 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: .914836+0000 mon.a (mon.0) 2021-03-24T21:48:25.295 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: 119 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-03-24T21:48:25.295 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:48:25.295 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: -03-24T21:48:22 2021-03-24T21:48:25.295 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: .582604+0000 mgr.y (mgr.14144 2021-03-24T21:48:25.296 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: ) 40 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:25.296 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: audit 2021-03-24T21:48:25.296 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: 2021-03-24T21:48:22 2021-03-24T21:48:25.296 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: .915094+0000 mon.a (mon.0 2021-03-24T21:48:25.297 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: ) 120 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-03-24T21:48:25.297 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: audit 2021 2021-03-24T21:48:25.297 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: -03-24T21:48: 2021-03-24T21:48:25.297 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: 23.915169+0000 mon.a (mon. 2021-03-24T21:48:25.297 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: 0) 121 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-03-24T21:48:25.298 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: audit 2021-03-24T21:48:25.298 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: 2021-03-24T21:48: 2021-03-24T21:48:25.298 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: 24.915288+0000 mon.a (mon 2021-03-24T21:48:25.298 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: .0) 122 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-03-24T21:48:25.299 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: 2021-03-24T21:48:25.299 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:48:25.299 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: 24T21:48:24.946625+0000 2021-03-24T21:48:25.299 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: mon.a (mon.0) 123 : 2021-03-24T21:48:25.299 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2021-03-24T21:48:25.300 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:48:25.300 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: 03-24T21:48:24.952646+0000 mon.a ( 2021-03-24T21:48:25.300 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: mon.0) 124 : cluster [DBG] monmap e2: 2 mons at {a=v1:172.21.15.125:6789/0,b=v1:172.21.15.142:6789/0} 2021-03-24T21:48:25.301 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: cluster 2021-03-24T21:48:25.301 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: 2021-03-24T21:48 2021-03-24T21:48:25.301 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: :24.952757+0000 mon.a 2021-03-24T21:48:25.301 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: (mon.0) 125 : cluster 2021-03-24T21:48:25.302 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: [DBG] fsmap 2021-03-24T21:48:25.302 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:48:25.302 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: -03-24T21:48:24 2021-03-24T21:48:25.302 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: .952813+0000 mon.a (mon. 2021-03-24T21:48:25.302 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: 0) 126 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2021-03-24T21:48:25.303 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: 2021-03-24T21:48:25.303 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: cluster 2021-03-24T21:48:25.303 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: 2021-03-24T21:48 2021-03-24T21:48:25.303 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: :24.953119+0000 mon.a (mon. 2021-03-24T21:48:25.304 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: 0) 127 : cluster [DBG] mgrmap e13: y(active, since 80s) 2021-03-24T21:48:25.304 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: 2021-03-24T21:48:25.304 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: cluster 2021-03-24T21:48:25.304 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: 2021-03-24T21: 2021-03-24T21:48:25.305 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: 48:24.959029+0000 mon.a ( 2021-03-24T21:48:25.305 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: mon.0) 128 : cluster [INF] 2021-03-24T21:48:25.305 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:25 smithi142 conmon[20125]: overall HEALTH_OK 2021-03-24T21:48:25.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: audit 2021-03-24T21:48:19.921760 2021-03-24T21:48:25.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: +0000 mon.a (mon.0) 116 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:48:25.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:48:25.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: -03-24T21:48:19.922075+0000 2021-03-24T21:48:25.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: mon.a (mon.0) 117 : cluster [INF] mon.a calling monitor election 2021-03-24T21:48:25.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: 2021-03-24T21:48:25.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:48:25.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: -03-24T21:48: 2021-03-24T21:48:25.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: 20.582246+0000 mgr.y (mgr. 2021-03-24T21:48:25.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: 14144) 39 : cluster [DBG] 2021-03-24T21:48:25.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:25.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: audit 2021-03-24T21:48:25.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: 2021-03-24T21:48 2021-03-24T21:48:25.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: :20.914782+0000 2021-03-24T21:48:25.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: mon.a (mon.0) 118 : 2021-03-24T21:48:25.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-03-24T21:48:25.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:48:25.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: -03-24T21:48: 2021-03-24T21:48:25.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: 21.914628+0000 mon.b (mon. 2021-03-24T21:48:25.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: 1) 1 : cluster [INF] mon.b calling monitor election 2021-03-24T21:48:25.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: 2021-03-24T21:48:25.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: audit 2021-03- 2021-03-24T21:48:25.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: 24T21:48:21.914836 2021-03-24T21:48:25.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: +0000 mon.a (mon.0) 2021-03-24T21:48:25.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: 119 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-03-24T21:48:25.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:48:25.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: -03-24T21:48:22 2021-03-24T21:48:25.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: .582604+0000 mgr.y (mgr.14144 2021-03-24T21:48:25.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: ) 40 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:25.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: audit 2021-03-24T21:48:25.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: 2021-03-24T21:48: 2021-03-24T21:48:25.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: 22.915094+0000 mon.a (mon. 2021-03-24T21:48:25.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: 0) 120 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-03-24T21:48:25.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: 2021-03-24T21:48:25.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: audit 2021-03-24T 2021-03-24T21:48:25.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: 21:48:23.915169+0000 2021-03-24T21:48:25.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: mon.a (mon.0) 121 : 2021-03-24T21:48:25.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-03-24T21:48:25.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: audit 2021-03-24T21:48:25.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: 2021-03-24T21:48 2021-03-24T21:48:25.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: :24.915288+0000 mon.a (mon 2021-03-24T21:48:25.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: .0) 122 : audit [DBG] 2021-03-24T21:48:25.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-03-24T21:48:25.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: cluster 2021-03-24T21:48:25.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: 2021-03-24T21:48: 2021-03-24T21:48:25.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: 24.946625+0000 mon.a (mon.0) 123 : cluster 2021-03-24T21:48:25.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2021-03-24T21:48:25.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:48:25.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: 24T21:48:24.952646+0000 2021-03-24T21:48:25.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: mon.a (mon.0) 124 : cluster 2021-03-24T21:48:25.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: [DBG] monmap e2: 2 mons at {a=v1:172.21.15.125:6789/0,b=v1:172.21.15.142:6789/0} 2021-03-24T21:48:25.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: cluster 2021-03-24T21:48:25.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: 2021-03-24T21:48: 2021-03-24T21:48:25.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: 24.952757+0000 mon.a (mon. 2021-03-24T21:48:25.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: 0) 125 : cluster [DBG] fsmap 2021-03-24T21:48:25.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: 2021-03-24T21:48:25.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T21:48:25.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: T21:48:24.952813+0000 mon.a ( 2021-03-24T21:48:25.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: mon.0) 126 : cluster [DBG] 2021-03-24T21:48:25.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: osdmap e4: 0 total, 0 up, 0 in 2021-03-24T21:48:25.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: cluster 2021-03-24T21:48:25.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: 2021-03-24T21:48 2021-03-24T21:48:25.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: :24.953119+0000 mon.a (mon 2021-03-24T21:48:25.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: .0) 127 : cluster [DBG] mgrmap e13: y(active, since 80s) 2021-03-24T21:48:25.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:48:25.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: -03-24T21:48:24 2021-03-24T21:48:25.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: .959029+0000 mon.a (mon.0) 128 : cluster 2021-03-24T21:48:25.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:25 smithi125 conmon[19936]: [INF] overall HEALTH_OK 2021-03-24T21:48:26.192 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:26 smithi125 conmon[19936]: cluster 2021-03-24T21:48 2021-03-24T21:48:26.192 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:26 smithi125 conmon[19936]: :24.582931+0000 mgr.y (mgr.14144) 41 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:26.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:26 smithi142 conmon[20125]: cluster 2021-03-24T21:48:24.582931+0000 2021-03-24T21:48:26.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:26 smithi142 conmon[20125]: mgr.y (mgr.14144) 41 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:26.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:26 smithi142 conmon[20125]: cephadm 2021-03-24T21:48 2021-03-24T21:48:26.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:26 smithi142 conmon[20125]: :24.957555+0000 mgr.y (mgr.14144) 42 : cephadm [INF] Deploying daemon mon.c on smithi125 2021-03-24T21:48:26.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:26 smithi142 conmon[20125]: 2021-03-24T21:48:26.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:26 smithi142 conmon[20125]: audit 2021-03-24T21:48:26.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:26 smithi142 conmon[20125]: 2021-03-24T21:48:25.210867+0000 2021-03-24T21:48:26.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:26 smithi142 conmon[20125]: mon.a (mon.0) 129 : audit [DBG] 2021-03-24T21:48:26.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:26 smithi142 conmon[20125]: from='client.? v1:172.21.15.142:0/967661132' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-03-24T21:48:26.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:26 smithi142 conmon[20125]: audit 2021-03-24T21:48:26.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:26 smithi142 conmon[20125]: 2021-03-24T21:48:25.915532+0000 2021-03-24T21:48:26.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:26 smithi142 conmon[20125]: mon.a (mon.0) 130 : audit [DBG] 2021-03-24T21:48:26.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:26 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-03-24T21:48:26.642 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:26 smithi125 conmon[19936]: cephadm 2021-03- 2021-03-24T21:48:26.642 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:26 smithi125 conmon[19936]: 24T21:48:24.957555+0000 mgr.y (mgr.14144) 42 : 2021-03-24T21:48:26.642 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:26 smithi125 conmon[19936]: cephadm [INF] Deploying daemon mon.c on smithi125 2021-03-24T21:48:26.642 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:26 smithi125 conmon[19936]: audit 2021-03-24T21:48:25.210867+0000 2021-03-24T21:48:26.643 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:26 smithi125 conmon[19936]: mon.a (mon.0) 129 : audit [DBG] from='client.? v1:172.21.15.142:0/967661132' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-03-24T21:48:26.643 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:26 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:48:26.643 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:26 smithi125 conmon[19936]: 03-24T21:48:25.915532+0000 mon.a (mon.0) 130 2021-03-24T21:48:26.643 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:26 smithi125 conmon[19936]: : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-03-24T21:48:26.902 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2021-03-24T21:48:26.903 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph mon dump -f json 2021-03-24T21:48:28.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:28 smithi142 conmon[20125]: cluster 2021-03-24T21:48:26 2021-03-24T21:48:28.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:28 smithi142 conmon[20125]: .583339+0000 mgr.y (mgr.14144) 43 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:28.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[19936]: cluster 2021-03-24T21:48:26. 2021-03-24T21:48:28.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[19936]: 583339+0000 mgr.y (mgr.14144) 43 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:28.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 systemd[1]: Starting Ceph mon.c for 3cfa339a-8cea-11eb-90b2-001a4aab830c... 2021-03-24T21:48:28.903 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 podman[24886]: 2021-03-24 21:48:28.489423363 +0000 UTC m=+0.194312886 container create 428b209d3b90504be67047b67cfeb021ee5afb07e85000441af02551a5c30cef (image=quay.ceph.io/ceph-ci/ceph@sha256:e9db06906b8d0a24f3ef6835ac5aa4b8eea2950a2560bad78f83d4f101eca0db, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-mon.c, io.buildah.version=1.18.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Dimitri Savineau , org.label-schema.name=CentOS Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06, org.label-schema.build-date=20201204, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d) 2021-03-24T21:48:28.904 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 podman[24886]: 2021-03-24 21:48:28.60611243 +0000 UTC m=+0.311001953 container init 428b209d3b90504be67047b67cfeb021ee5afb07e85000441af02551a5c30cef (image=quay.ceph.io/ceph-ci/ceph@sha256:e9db06906b8d0a24f3ef6835ac5aa4b8eea2950a2560bad78f83d4f101eca0db, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-mon.c, io.buildah.version=1.18.0, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20201204, GIT_BRANCH=HEAD, GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06, RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d, maintainer=Dimitri Savineau , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True) 2021-03-24T21:48:28.904 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.632+0000 7f1d79d5f700 0 set uid:gid to 167:167 (ceph:ceph) 2021-03-24T21:48:28.904 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.632+0000 7f1d79d5f700 0 ceph version 16.1.0-1166-gd71e96d2 (d71e96d216190f28f757f409532464ab0eca4ea9) pacific (rc), process ceph-mon, pid 2 2021-03-24T21:48:28.904 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.632+0000 7f1d79d5f700 0 pidfile_write: ignore empty --pid-file 2021-03-24T21:48:28.904 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 podman[24886]: 2021-03-24 21:48:28.647760064 +0000 UTC m=+0.352649570 container start 428b209d3b90504be67047b67cfeb021ee5afb07e85000441af02551a5c30cef (image=quay.ceph.io/ceph-ci/ceph@sha256:e9db06906b8d0a24f3ef6835ac5aa4b8eea2950a2560bad78f83d4f101eca0db, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-mon.c, GIT_BRANCH=HEAD, GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06, maintainer=Dimitri Savineau , ceph=True, org.label-schema.name=CentOS Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20201204, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d, org.label-schema.license=GPLv2, io.buildah.version=1.18.0) 2021-03-24T21:48:28.905 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 bash[24886]: 428b209d3b90504be67047b67cfeb021ee5afb07e85000441af02551a5c30cef 2021-03-24T21:48:28.905 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.640+0000 7f1d79d5f700 0 load: jerasure load: lrc load: isa 2021-03-24T21:48:28.905 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: RocksDB version: 6.8.1 2021-03-24T21:48:28.905 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.906 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2021-03-24T21:48:28.906 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Compile date Mar 24 2021 2021-03-24T21:48:28.906 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: DB SUMMARY 2021-03-24T21:48:28.906 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.906 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: CURRENT file: CURRENT 2021-03-24T21:48:28.907 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.907 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: IDENTITY file: IDENTITY 2021-03-24T21:48:28.907 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.907 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: MANIFEST file: MANIFEST-000001 size: 13 Bytes 2021-03-24T21:48:28.907 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.908 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2021-03-24T21:48:28.908 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.908 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000003.log size: 566 ; 2021-03-24T21:48:28.908 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.909 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.error_if_exists: 0 2021-03-24T21:48:28.909 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.909 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.create_if_missing: 0 2021-03-24T21:48:28.909 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.paranoid_checks: 1 2021-03-24T21:48:28.909 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.910 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.env: 0x560aa40c01c0 2021-03-24T21:48:28.910 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.fs: Posix File System 2021-03-24T21:48:28.910 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.910 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.info_log: 0x560aa54ae720 2021-03-24T21:48:28.911 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.max_file_opening_threads: 16 2021-03-24T21:48:28.911 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.statistics: (nil) 2021-03-24T21:48:28.911 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.use_fsync: 0 2021-03-24T21:48:28.911 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.max_log_file_size: 0 2021-03-24T21:48:28.911 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2021-03-24T21:48:28.912 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.log_file_time_to_roll: 0 2021-03-24T21:48:28.912 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.keep_log_file_num: 1000 2021-03-24T21:48:28.912 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.recycle_log_file_num: 0 2021-03-24T21:48:28.912 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.912 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.allow_fallocate: 1 2021-03-24T21:48:28.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.allow_mmap_reads: 0 2021-03-24T21:48:28.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.allow_mmap_writes: 0 2021-03-24T21:48:28.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.use_direct_reads: 0 2021-03-24T21:48:28.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2021-03-24T21:48:28.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.create_missing_column_families: 0 2021-03-24T21:48:28.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.db_log_dir: 2021-03-24T21:48:28.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2021-03-24T21:48:28.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.table_cache_numshardbits: 6 2021-03-24T21:48:28.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.max_subcompactions: 1 2021-03-24T21:48:28.917 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.917 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.max_background_flushes: -1 2021-03-24T21:48:28.917 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.917 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.WAL_ttl_seconds: 0 2021-03-24T21:48:28.917 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.WAL_size_limit_MB: 0 2021-03-24T21:48:28.918 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2021-03-24T21:48:28.918 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2021-03-24T21:48:28.918 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.918 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.is_fd_close_on_exec: 1 2021-03-24T21:48:28.919 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.advise_random_on_open: 1 2021-03-24T21:48:28.919 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.db_write_buffer_size: 0 2021-03-24T21:48:28.919 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.write_buffer_manager: 0x560aa5547da0 2021-03-24T21:48:28.919 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2021-03-24T21:48:28.919 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2021-03-24T21:48:28.920 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.920 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2021-03-24T21:48:28.920 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.use_adaptive_mutex: 0 2021-03-24T21:48:28.920 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.rate_limiter: (nil) 2021-03-24T21:48:28.920 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2021-03-24T21:48:28.921 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.wal_recovery_mode: 2 2021-03-24T21:48:28.921 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.enable_thread_tracking: 0 2021-03-24T21:48:28.921 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.921 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.enable_pipelined_write: 0 2021-03-24T21:48:28.921 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.unordered_write: 0 2021-03-24T21:48:28.922 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2021-03-24T21:48:28.922 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2021-03-24T21:48:28.922 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2021-03-24T21:48:28.922 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2021-03-24T21:48:28.923 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.row_cache: None 2021-03-24T21:48:28.923 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.wal_filter: None 2021-03-24T21:48:28.923 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2021-03-24T21:48:28.923 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.allow_ingest_behind: 0 2021-03-24T21:48:28.923 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.preserve_deletes: 0 2021-03-24T21:48:28.924 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.two_write_queues: 0 2021-03-24T21:48:28.924 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.manual_wal_flush: 0 2021-03-24T21:48:28.924 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.atomic_flush: 0 2021-03-24T21:48:28.924 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2021-03-24T21:48:28.924 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.persist_stats_to_disk: 0 2021-03-24T21:48:28.925 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.write_dbid_to_manifest: 0 2021-03-24T21:48:28.925 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.log_readahead_size: 0 2021-03-24T21:48:28.925 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2021-03-24T21:48:28.925 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.max_background_jobs: 2 2021-03-24T21:48:28.925 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.max_background_compactions: -1 2021-03-24T21:48:28.926 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2021-03-24T21:48:28.926 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2021-03-24T21:48:28.926 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.delayed_write_rate : 16777216 2021-03-24T21:48:28.926 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.max_total_wal_size: 0 2021-03-24T21:48:28.927 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2021-03-24T21:48:28.927 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.stats_dump_period_sec: 600 2021-03-24T21:48:28.927 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.stats_persist_period_sec: 600 2021-03-24T21:48:28.927 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2021-03-24T21:48:28.927 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.max_open_files: -1 2021-03-24T21:48:28.928 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.928 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.bytes_per_sync: 0 2021-03-24T21:48:28.928 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.wal_bytes_per_sync: 0 2021-03-24T21:48:28.928 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.strict_bytes_per_sync: 0 2021-03-24T21:48:28.929 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.compaction_readahead_size: 0 2021-03-24T21:48:28.929 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Compression algorithms supported: 2021-03-24T21:48:28.929 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2021-03-24T21:48:28.929 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: kZSTD supported: 0 2021-03-24T21:48:28.929 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.930 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.930 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: kXpressCompression supported: 0 2021-03-24T21:48:28.930 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.930 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: kLZ4HCCompression supported: 1 2021-03-24T21:48:28.931 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.931 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: kLZ4Compression supported: 1 2021-03-24T21:48:28.931 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: kBZip2Compression supported: 0 2021-03-24T21:48:28.931 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.931 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.932 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: kZlibCompression supported: 1 2021-03-24T21:48:28.932 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.932 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: kSnappyCompression supported: 1 2021-03-24T21:48:28.932 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.933 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Fast CRC32 supported: Supported on x86 2021-03-24T21:48:28.933 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.933 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000001 2021-03-24T21:48:28.933 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.933 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2021-03-24T21:48:28.934 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.934 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2021-03-24T21:48:28.934 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.merge_operator: 2021-03-24T21:48:28.934 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.934 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.compaction_filter: None 2021-03-24T21:48:28.935 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.compaction_filter_factory: None 2021-03-24T21:48:28.935 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.935 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.935 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.memtable_factory: SkipListFactory 2021-03-24T21:48:28.936 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.table_factory: BlockBasedTable 2021-03-24T21:48:28.936 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560aa5452218) 2021-03-24T21:48:28.936 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: cache_index_and_filter_blocks: 1 2021-03-24T21:48:28.936 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: cache_index_and_filter_blocks_with_high_priority: 0 2021-03-24T21:48:28.936 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: pin_l0_filter_and_index_blocks_in_cache: 0 2021-03-24T21:48:28.937 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: pin_top_level_index_and_filter: 1 2021-03-24T21:48:28.937 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: index_type: 0 2021-03-24T21:48:28.937 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: data_block_index_type: 0 2021-03-24T21:48:28.937 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: index_shortening: 1 2021-03-24T21:48:28.937 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: data_block_hash_table_util_ratio: 0.750000 2021-03-24T21:48:28.938 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: hash_index_allow_collision: 1 2021-03-24T21:48:28.938 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: checksum: 1 2021-03-24T21:48:28.938 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: no_block_cache: 0 2021-03-24T21:48:28.938 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: block_cache: 0x560aa548af10 2021-03-24T21:48:28.939 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: block_cache_name: BinnedLRUCache 2021-03-24T21:48:28.939 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: block_cache_options: 2021-03-24T21:48:28.939 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: capacity : 536870912 2021-03-24T21:48:28.939 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: num_shard_bits : 4 2021-03-24T21:48:28.939 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: strict_capacity_limit : 0 2021-03-24T21:48:28.940 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: high_pri_pool_ratio: 0.000 2021-03-24T21:48:28.940 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: block_cache_compressed: (nil) 2021-03-24T21:48:28.940 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: persistent_cache: (nil) 2021-03-24T21:48:28.940 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: block_size: 4096 2021-03-24T21:48:28.940 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: block_size_deviation: 10 2021-03-24T21:48:28.941 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: block_restart_interval: 16 2021-03-24T21:48:28.941 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: index_block_restart_interval: 1 2021-03-24T21:48:28.941 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: metadata_block_size: 4096 2021-03-24T21:48:28.941 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: partition_filters: 0 2021-03-24T21:48:28.941 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: use_delta_encoding: 1 2021-03-24T21:48:28.942 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: filter_policy: rocksdb.BuiltinBloomFilter 2021-03-24T21:48:28.942 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: whole_key_filtering: 1 2021-03-24T21:48:28.942 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: verify_compression: 0 2021-03-24T21:48:28.942 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: read_amp_bytes_per_bit: 0 2021-03-24T21:48:28.942 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: format_version: 2 2021-03-24T21:48:28.943 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: enable_index_compression: 1 2021-03-24T21:48:28.943 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: block_align: 0 2021-03-24T21:48:28.943 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.943 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.write_buffer_size: 33554432 2021-03-24T21:48:28.943 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.max_write_buffer_number: 2 2021-03-24T21:48:28.944 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.compression: NoCompression 2021-03-24T21:48:28.944 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.bottommost_compression: Disabled 2021-03-24T21:48:28.944 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.prefix_extractor: nullptr 2021-03-24T21:48:28.944 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2021-03-24T21:48:28.945 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.num_levels: 7 2021-03-24T21:48:28.945 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2021-03-24T21:48:28.945 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2021-03-24T21:48:28.945 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2021-03-24T21:48:28.945 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.946 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2021-03-24T21:48:28.946 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.946 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2021-03-24T21:48:28.946 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2021-03-24T21:48:28.946 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2021-03-24T21:48:28.947 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.947 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2021-03-24T21:48:28.947 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.947 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.948 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2021-03-24T21:48:28.948 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.948 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.compression_opts.window_bits: -14 2021-03-24T21:48:28.948 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.compression_opts.level: 32767 2021-03-24T21:48:28.948 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.compression_opts.strategy: 0 2021-03-24T21:48:28.949 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.949 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2021-03-24T21:48:28.949 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2021-03-24T21:48:28.949 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.compression_opts.enabled: false 2021-03-24T21:48:28.949 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.950 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2021-03-24T21:48:28.950 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.950 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2021-03-24T21:48:28.950 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.951 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2021-03-24T21:48:28.951 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.951 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.target_file_size_base: 67108864 2021-03-24T21:48:28.951 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.target_file_size_multiplier: 1 2021-03-24T21:48:28.951 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.952 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.952 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2021-03-24T21:48:28.952 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.952 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2021-03-24T21:48:28.952 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2021-03-24T21:48:28.953 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.953 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2021-03-24T21:48:28.953 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2021-03-24T21:48:28.953 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.954 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2021-03-24T21:48:28.954 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2021-03-24T21:48:28.954 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2021-03-24T21:48:28.954 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.954 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2021-03-24T21:48:28.955 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.955 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2021-03-24T21:48:28.955 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2021-03-24T21:48:28.955 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2021-03-24T21:48:28.955 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.arena_block_size: 4194304 2021-03-24T21:48:28.956 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2021-03-24T21:48:28.956 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.956 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.956 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2021-03-24T21:48:28.957 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2021-03-24T21:48:28.957 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.disable_auto_compactions: 0 2021-03-24T21:48:28.957 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2021-03-24T21:48:28.957 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.957 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2021-03-24T21:48:28.958 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.958 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2021-03-24T21:48:28.958 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.958 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2021-03-24T21:48:28.958 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2021-03-24T21:48:28.959 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.959 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.959 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2021-03-24T21:48:28.959 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2021-03-24T21:48:28.960 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2021-03-24T21:48:28.960 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2021-03-24T21:48:28.960 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2021-03-24T21:48:28.960 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.960 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.table_properties_collectors: 2021-03-24T21:48:28.961 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.inplace_update_support: 0 2021-03-24T21:48:28.961 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.961 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.inplace_update_num_locks: 10000 2021-03-24T21:48:28.961 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.961 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2021-03-24T21:48:28.962 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2021-03-24T21:48:28.962 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.memtable_huge_page_size: 0 2021-03-24T21:48:28.962 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.bloom_locality: 0 2021-03-24T21:48:28.962 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.963 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.max_successive_merges: 0 2021-03-24T21:48:28.963 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.optimize_filters_for_hits: 0 2021-03-24T21:48:28.963 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.963 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.paranoid_file_checks: 0 2021-03-24T21:48:28.963 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.force_consistency_checks: 0 2021-03-24T21:48:28.964 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.report_bg_io_stats: 0 2021-03-24T21:48:28.964 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.ttl: 2592000 2021-03-24T21:48:28.964 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: Options.periodic_compaction_seconds: 0 2021-03-24T21:48:28.964 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.965 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2021-03-24T21:48:28.965 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.965 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2021-03-24T21:48:28.965 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.965 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616622508651865, "job": 1, "event": "recovery_started", "log_files": [3]} 2021-03-24T21:48:28.966 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #3 mode 2 2021-03-24T21:48:28.966 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.966 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.644+0000 7f1d79d5f700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2021-03-24T21:48:28.966 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.648+0000 7f1d79d5f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616622508652757, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 4, "file_size": 1492, "table_properties": {"data_size": 578, "index_size": 40, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 456, "raw_average_value_size": 91, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1616622508, "oldest_key_time": 3, "file_creation_time": 0}} 2021-03-24T21:48:28.967 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.648+0000 7f1d79d5f700 4 rocksdb: [version_set.cc:3826] Creating manifest 5 2021-03-24T21:48:28.967 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.967 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.648+0000 7f1d79d5f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616622508653419, "job": 1, "event": "recovery_finished"} 2021-03-24T21:48:28.967 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.648+0000 7f1d79d5f700 4 rocksdb: DB pointer 0x560aa5551800 2021-03-24T21:48:28.967 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.648+0000 7f1d79d5f700 0 mon.c does not exist in monmap, will attempt to join an existing cluster 2021-03-24T21:48:28.968 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.648+0000 7f1d79d5f700 0 using public_addrv v1:172.21.15.125:6790/0 2021-03-24T21:48:28.968 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.652+0000 7f1d60b7c700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2021-03-24T21:48:28.968 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.652+0000 7f1d60b7c700 4 rocksdb: [db_impl/db_impl.cc:851] 2021-03-24T21:48:28.968 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: ** DB Stats ** 2021-03-24T21:48:28.969 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: Uptime(secs): 0.0 total, 0.0 interval 2021-03-24T21:48:28.969 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2021-03-24T21:48:28.969 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2021-03-24T21:48:28.969 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2021-03-24T21:48:28.969 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2021-03-24T21:48:28.970 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2021-03-24T21:48:28.970 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2021-03-24T21:48:28.970 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.970 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: ** Compaction Stats [default] ** 2021-03-24T21:48:28.970 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-24T21:48:28.971 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-24T21:48:28.971 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: L0 1/0 1.46 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.8 0.00 0.00 1 0.001 0 0 2021-03-24T21:48:28.971 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: Sum 1/0 1.46 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.8 0.00 0.00 1 0.001 0 0 2021-03-24T21:48:28.971 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.8 0.00 0.00 1 0.001 0 0 2021-03-24T21:48:28.971 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.972 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: ** Compaction Stats [default] ** 2021-03-24T21:48:28.972 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-24T21:48:28.972 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-24T21:48:28.972 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.8 0.00 0.00 1 0.001 0 0 2021-03-24T21:48:28.973 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: Uptime(secs): 0.0 total, 0.0 interval 2021-03-24T21:48:28.973 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: Flush(GB): cumulative 0.000, interval 0.000 2021-03-24T21:48:28.973 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: AddFile(GB): cumulative 0.000, interval 0.000 2021-03-24T21:48:28.973 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: AddFile(Total Files): cumulative 0, interval 0 2021-03-24T21:48:28.973 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: AddFile(L0 Files): cumulative 0, interval 0 2021-03-24T21:48:28.974 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: AddFile(Keys): cumulative 0, interval 0 2021-03-24T21:48:28.974 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: Cumulative compaction: 0.00 GB write, 0.22 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-24T21:48:28.974 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: Interval compaction: 0.00 GB write, 0.22 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-24T21:48:28.974 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-03-24T21:48:28.974 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.975 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: ** File Read Latency Histogram By Level [default] ** 2021-03-24T21:48:28.975 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.975 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: ** Compaction Stats [default] ** 2021-03-24T21:48:28.975 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-24T21:48:28.975 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-24T21:48:28.976 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: L0 1/0 1.46 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.8 0.00 0.00 1 0.001 0 0 2021-03-24T21:48:28.976 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: Sum 1/0 1.46 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.8 0.00 0.00 1 0.001 0 0 2021-03-24T21:48:28.976 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-03-24T21:48:28.976 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.977 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: ** Compaction Stats [default] ** 2021-03-24T21:48:28.977 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-24T21:48:28.977 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-24T21:48:28.977 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.8 0.00 0.00 1 0.001 0 0 2021-03-24T21:48:28.977 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: Uptime(secs): 0.0 total, 0.0 interval 2021-03-24T21:48:28.978 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: Flush(GB): cumulative 0.000, interval 0.000 2021-03-24T21:48:28.978 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: AddFile(GB): cumulative 0.000, interval 0.000 2021-03-24T21:48:28.978 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: AddFile(Total Files): cumulative 0, interval 0 2021-03-24T21:48:28.978 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: AddFile(L0 Files): cumulative 0, interval 0 2021-03-24T21:48:28.979 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: AddFile(Keys): cumulative 0, interval 0 2021-03-24T21:48:28.979 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: Cumulative compaction: 0.00 GB write, 0.22 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-24T21:48:28.979 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-24T21:48:28.979 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-03-24T21:48:28.979 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.980 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: ** File Read Latency Histogram By Level [default] ** 2021-03-24T21:48:28.980 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.980 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.652+0000 7f1d79d5f700 0 starting mon.c rank -1 at public addrs v1:172.21.15.125:6790/0 at bind addrs v1:172.21.15.125:6790/0 mon_data /var/lib/ceph/mon/ceph-c fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c 2021-03-24T21:48:28.980 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.652+0000 7f1d79d5f700 1 mon.c@-1(???) e0 preinit fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c 2021-03-24T21:48:28.980 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 systemd[1]: Started Ceph mon.c for 3cfa339a-8cea-11eb-90b2-001a4aab830c. 2021-03-24T21:48:28.981 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.680+0000 7f1d63381700 0 mon.c@-1(synchronizing).mds e1 new map 2021-03-24T21:48:28.981 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.680+0000 7f1d63381700 0 mon.c@-1(synchronizing).mds e1 print_map 2021-03-24T21:48:28.981 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: e1 2021-03-24T21:48:28.981 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: enable_multiple, ever_enabled_multiple: 1,1 2021-03-24T21:48:28.982 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2021-03-24T21:48:28.982 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: legacy client fscid: -1 2021-03-24T21:48:28.982 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.982 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: No filesystems configured 2021-03-24T21:48:28.982 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.983 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.680+0000 7f1d5eb78700 4 rocksdb: [db_impl/db_impl_write.cc:1668] [default] New memtable created with log file: #9. Immutable memtables: 0. 2021-03-24T21:48:28.983 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.983 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.680+0000 7f1d63381700 1 mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2021-03-24T21:48:28.983 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.680+0000 7f1d68b8c700 4 rocksdb: (Original Log Time 2021/03/24-21:48:28.688425) [db_impl/db_impl_compaction_flush.cc:2198] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 2021-03-24T21:48:28.983 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.684+0000 7f1d68b8c700 4 rocksdb: [flush_job.cc:321] [default] [JOB 3] Flushing memtable with next log file: 9 2021-03-24T21:48:28.984 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:28.984 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.684+0000 7f1d68b8c700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616622508688550, "job": 3, "event": "flush_started", "num_memtables": 1, "num_entries": 1123, "num_deletes": 23, "total_data_size": 3667775, "memory_usage": 3688664, "flush_reason": "Manual Compaction"} 2021-03-24T21:48:28.984 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.684+0000 7f1d68b8c700 4 rocksdb: [flush_job.cc:350] [default] [JOB 3] Level-0 flush table #10: started 2021-03-24T21:48:28.984 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.684+0000 7f1d63381700 1 mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2021-03-24T21:48:28.984 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.684+0000 7f1d63381700 1 mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2021-03-24T21:48:28.985 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.684+0000 7f1d63381700 1 mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2021-03-24T21:48:28.985 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.684+0000 7f1d63381700 1 mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2021-03-24T21:48:28.985 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.684+0000 7f1d63381700 1 mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2021-03-24T21:48:28.985 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.684+0000 7f1d63381700 0 mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2021-03-24T21:48:28.986 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.684+0000 7f1d63381700 0 mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2021-03-24T21:48:28.986 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.684+0000 7f1d63381700 0 mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2021-03-24T21:48:28.986 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.684+0000 7f1d63381700 0 mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2021-03-24T21:48:28.986 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: cluster 2021-03-24T21:48:28.986 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 2021-03-24T21:48:26.583339+0000 mgr.y (mgr.14144) 2021-03-24T21:48:28.987 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: 43 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:28.987 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.684+0000 7f1d63381700 1 mon.c@-1(synchronizing).paxosservice(auth 1..5) refresh upgraded, format 0 -> 3 2021-03-24T21:48:28.987 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.684+0000 7f1d63381700 20 expand_channel_meta expand map: {default=false} 2021-03-24T21:48:28.987 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.684+0000 7f1d63381700 20 expand_channel_meta from 'false' to 'false' 2021-03-24T21:48:28.987 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.684+0000 7f1d63381700 20 expand_channel_meta expanded map: {default=false} 2021-03-24T21:48:28.988 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.684+0000 7f1d63381700 20 expand_channel_meta expand map: {default=info} 2021-03-24T21:48:28.988 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.684+0000 7f1d63381700 20 expand_channel_meta from 'info' to 'info' 2021-03-24T21:48:28.988 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.684+0000 7f1d63381700 20 expand_channel_meta expanded map: {default=info} 2021-03-24T21:48:28.988 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.684+0000 7f1d63381700 20 expand_channel_meta expand map: {default=daemon} 2021-03-24T21:48:28.988 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.684+0000 7f1d63381700 20 expand_channel_meta from 'daemon' to 'daemon' 2021-03-24T21:48:28.989 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.684+0000 7f1d63381700 20 expand_channel_meta expanded map: {default=daemon} 2021-03-24T21:48:28.989 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.684+0000 7f1d63381700 20 expand_channel_meta expand map: {default=debug} 2021-03-24T21:48:28.989 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.684+0000 7f1d63381700 20 expand_channel_meta from 'debug' to 'debug' 2021-03-24T21:48:28.989 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.684+0000 7f1d63381700 20 expand_channel_meta expanded map: {default=debug} 2021-03-24T21:48:28.989 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:28 smithi125 conmon[24927]: debug 2021-03-24T21:48:28.684+0000 7f1d63381700 10 mon.c@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete,mon_osdmap_full_prune_interval,mon_osdmap_full_prune_min,mon_osdmap_full_prune_txsize 2021-03-24T21:48:33.729 INFO:teuthology.orchestra.run.smithi142.stdout: 2021-03-24T21:48:33.729 INFO:teuthology.orchestra.run.smithi142.stdout:{"epoch":3,"fsid":"3cfa339a-8cea-11eb-90b2-001a4aab830c","modified":"2021-03-24T21:48:28.695448Z","created":"2021-03-24T21:46:15.748211Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6789","nonce":0}]},"addr":"172.21.15.125:6789/0","public_addr":"172.21.15.125:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6790","nonce":0}]},"addr":"172.21.15.125:6790/0","public_addr":"172.21.15.125:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2021-03-24T21:48:33.734 INFO:teuthology.orchestra.run.smithi142.stderr:dumped monmap epoch 3 2021-03-24T21:48:34.114 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: cluster 2021-03-24T21:48 2021-03-24T21:48:34.115 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: :28.583801+0000 mgr.y (mgr.14144) 44 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:34.115 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: cluster 2021-03-24T21 2021-03-24T21:48:34.115 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: :48:28.702225+0000 mon.b (mon.1) 2 : cluster [INF] mon.b calling monitor election 2021-03-24T21:48:34.115 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: cluster 2021-03-24T21:48:28 2021-03-24T21:48:34.116 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: .702976+0000 mon.a (mon.0) 134 : cluster [INF] 2021-03-24T21:48:34.116 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: mon.a calling monitor election 2021-03-24T21:48:34.116 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:48:34.116 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: 03-24T21:48:28.930705+0000 mon.a (mon.0) 135 : audit [DBG] 2021-03-24T21:48:34.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: from='client.? v1:172.21.15.142:0/2987575319' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-03-24T21:48:34.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:48:34.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: 03-24T21:48:29.693921+0000 mon.a (mon.0) 136 : audit 2021-03-24T21:48:34.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-03-24T21:48:34.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: cluster 2021-03-24T21:48:30.584271+0000 mgr.y 2021-03-24T21:48:34.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: (mgr.14144) 45 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:34.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: audit 2021-03-24T 2021-03-24T21:48:34.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: 21:48:30.694494+0000 mon.a (mon.0) 137 : audit [DBG] 2021-03-24T21:48:34.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-03-24T21:48:34.119 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:48:34.119 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: 24T21:48:30.697301+0000 mon.c (mon.2) 1 : cluster 2021-03-24T21:48:34.119 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: [INF] mon.c calling monitor election 2021-03-24T21:48:34.119 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: audit 2021-03-24T21:48:34.119 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: 2021-03-24T21:48:31.694340+0000 mon.a (mon.0) 138 2021-03-24T21:48:34.120 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-03-24T21:48:34.120 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: cluster 2021-03-24T21:48:32.584764+0000 2021-03-24T21:48:34.120 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: mgr.y (mgr.14144) 46 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:34.120 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: 2021-03-24T21:48:34.120 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: audit 2021-03 2021-03-24T21:48:34.121 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: -24T21:48:32.694499+0000 mon.a (mon.0) 139 : audit [DBG] 2021-03-24T21:48:34.121 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-03-24T21:48:34.121 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:48:34.121 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: 03-24T21:48:33.694687+0000 mon.a (mon.0) 140 : 2021-03-24T21:48:34.121 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-03-24T21:48:34.122 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:48:34.122 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: 21:48:33.708316+0000 mon.a (mon.0) 141 : 2021-03-24T21:48:34.122 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: cluster [INF] mon.a calling monitor election 2021-03-24T21:48:34.122 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:48:34.123 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: 21:48:33.728110+0000 mon.a (mon.0) 142 : cluster [INF] 2021-03-24T21:48:34.123 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2021-03-24T21:48:34.123 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:48:34.123 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: -24T21:48:33.733718+0000 mon.a (mon.0) 143 : cluster [DBG] 2021-03-24T21:48:34.123 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: monmap e3: 3 mons at {a=v1:172.21.15.125:6789/0,b=v1:172.21.15.142:6789/0,c=v1:172.21.15.125:6790/0} 2021-03-24T21:48:34.124 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:48:34.124 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: 24T21:48:33.733814+0000 mon.a (mon.0) 144 : cluster [DBG] 2021-03-24T21:48:34.124 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: fsmap 2021-03-24T21:48:34.124 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:48:34.124 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: -03-24T21:48:33.733861+0000 mon.a (mon.0) 145 : 2021-03-24T21:48:34.125 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2021-03-24T21:48:34.125 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: cluster 2021-03-24T21:48:33.734165+0000 mon.a ( 2021-03-24T21:48:34.125 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: mon.0) 146 : cluster [DBG] mgrmap e13: y(active, since 89s) 2021-03-24T21:48:34.125 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:48:34.125 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: 48:33.743349+0000 mon.a (mon.0) 147 : cluster [INF] 2021-03-24T21:48:34.126 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: overall HEALTH_OK 2021-03-24T21:48:34.126 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:48:34.126 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: 03-24T21:48:33.746987+0000 mon.a (mon.0) 148 : audit 2021-03-24T21:48:34.126 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:48:34.127 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: audit 2021-03-24T21:48:34.127 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: 2021-03-24T21:48:33.752008+0000 mon.a (mon.0) 2021-03-24T21:48:34.127 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:33 smithi142 conmon[20125]: 149 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:48:34.154 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:48:34.154 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: :48:28.583801+0000 mgr.y (mgr.14144) 44 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:34.154 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:48:34.155 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: -24T21:48:28.702225+0000 mon.b (mon.1) 2 : cluster [INF] mon.b calling monitor election 2021-03-24T21:48:34.155 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:48:34.155 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: -24T21:48:28.702976+0000 mon.a (mon.0) 134 2021-03-24T21:48:34.155 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: : cluster [INF] mon.a calling monitor election 2021-03-24T21:48:34.155 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: audit 2021-03-24T21 2021-03-24T21:48:34.156 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: :48:28.930705+0000 mon.a (mon.0) 135 : 2021-03-24T21:48:34.156 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: audit [DBG] from='client.? v1:172.21.15.142:0/2987575319' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-03-24T21:48:34.156 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: audit 2021-03-24T21 2021-03-24T21:48:34.156 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: :48:29.693921+0000 mon.a (mon.0) 136 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-03-24T21:48:34.157 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: 2021-03-24T21:48:34.157 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: cluster 2021-03-24T21:48: 2021-03-24T21:48:34.157 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: 30.584271+0000 mgr.y (mgr.14144) 45 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:34.157 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: audit 2021-03-24T21:48:30.694494+0000 mon.a 2021-03-24T21:48:34.157 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: (mon.0) 137 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-03-24T21:48:34.158 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:48:34.158 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: :48:30.697301+0000 mon.c (mon.2) 1 : cluster [INF] 2021-03-24T21:48:34.158 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: mon.c calling monitor election 2021-03-24T21:48:34.158 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: audit 2021-03-24T21:48:34.158 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: 2021-03-24T 2021-03-24T21:48:34.159 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: 21:48:31.694340+0000 mon.a (mon.0) 2021-03-24T21:48:34.159 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: 138 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-03-24T21:48:34.159 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: cluster 2021-03-24T21:48 2021-03-24T21:48:34.159 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: :32.584764+0000 mgr.y (mgr.14144) 46 : cluster 2021-03-24T21:48:34.160 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:34.160 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: audit 2021-03-24T21:48:34.160 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: 2021-03-24T21:48:32.694499+0000 mon.a (mon.0) 139 : 2021-03-24T21:48:34.160 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-03-24T21:48:34.160 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: audit 2021-03-24T21:48:33. 2021-03-24T21:48:34.161 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: 694687+0000 mon.a (mon.0) 140 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-03-24T21:48:34.161 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: cluster 2021-03-24T21:48:33.708316 2021-03-24T21:48:34.161 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: +0000 mon.a (mon.0) 141 : cluster [INF] mon.a calling monitor election 2021-03-24T21:48:34.161 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: cluster 2021-03-24T21:48 2021-03-24T21:48:34.161 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: :33.728110+0000 mon.a (mon.0) 142 : cluster [INF] 2021-03-24T21:48:34.162 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2021-03-24T21:48:34.162 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:48:34.162 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: 48:33.733718+0000 mon.a (mon.0) 143 : cluster [DBG] monmap e3: 3 mons at {a=v1:172.21.15.125:6789/0,b=v1:172.21.15.142:6789/0,c=v1:172.21.15.125:6790/0} 2021-03-24T21:48:34.162 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:48:34.162 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: 48:33.733814+0000 mon.a (mon.0) 144 2021-03-24T21:48:34.163 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: : cluster [DBG] fsmap 2021-03-24T21:48:34.163 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:48:34.163 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: 48:28.583801+0000 mgr.y (mgr.14144) 44 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:34.164 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: cluster 2021-03-24T21:48:34.164 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: 2021-03-24T21:48 2021-03-24T21:48:34.164 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: :28.702225+0000 2021-03-24T21:48:34.164 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: mon.b (mon.1) 2 : cluster 2021-03-24T21:48:34.164 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: [INF] mon.b calling monitor election 2021-03-24T21:48:34.165 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: cluster 2021-03-24T21:48:34.165 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: 2021-03-24T 2021-03-24T21:48:34.165 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: 21:48:28.702976+0000 2021-03-24T21:48:34.165 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: mon.a (mon.0) 134 : 2021-03-24T21:48:34.165 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: cluster [INF] mon.a calling monitor election 2021-03-24T21:48:34.166 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: audit 2021-03-24T21:48:34.166 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: 2021-03-24T21 2021-03-24T21:48:34.166 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: :48:28.930705+0000 2021-03-24T21:48:34.166 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: mon.a (mon.0) 135 : audit 2021-03-24T21:48:34.166 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: [DBG] from='client.? v1:172.21.15.142:0/2987575319' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-03-24T21:48:34.167 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: audit 2021-03-24T21:48:34.167 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: 2021-03-24T21 2021-03-24T21:48:34.167 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: :48:29.693921+0000 2021-03-24T21:48:34.167 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: mon.a (mon.0) 136 : 2021-03-24T21:48:34.167 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-03-24T21:48:34.168 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: cluster 2021-03-24T21:48:34.168 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: 2021-03-24 2021-03-24T21:48:34.168 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: T21:48:30.584271 2021-03-24T21:48:34.168 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: +0000 mgr.y (mgr.14144) 45 2021-03-24T21:48:34.168 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:34.169 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: audit 2021-03-24T21:48:34.169 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: 2021-03-24T 2021-03-24T21:48:34.169 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: 21:48:30.694494+0000 2021-03-24T21:48:34.169 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: mon.a (mon.0) 137 : audit 2021-03-24T21:48:34.169 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-03-24T21:48:34.170 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: cluster 2021-03-24T21:48:34.170 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: 2021-03-24T21:48: 2021-03-24T21:48:34.170 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: 30.697301+0000 mon.c (mon. 2021-03-24T21:48:34.170 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: 2) 1 : cluster [INF] 2021-03-24T21:48:34.170 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: mon.c calling monitor election 2021-03-24T21:48:34.171 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: audit 2021-03-24T21:48:34.171 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: 2021-03-24 2021-03-24T21:48:34.171 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: T21:48:31. 2021-03-24T21:48:34.171 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: 694340+0000 mon.a (mon 2021-03-24T21:48:34.171 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: .0) 138 : audit 2021-03-24T21:48:34.172 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-03-24T21:48:34.172 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: cluster 2021-03-24T21:48:34.172 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: 2021-03-24T21:48 2021-03-24T21:48:34.172 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: :32.584764+0000 2021-03-24T21:48:34.172 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: mgr.y (mgr.14144 2021-03-24T21:48:34.173 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: ) 46 : cluster [DBG] 2021-03-24T21:48:34.173 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:34.173 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: audit 2021-03-24T21:48:34.173 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: 2021-03-24T 2021-03-24T21:48:34.173 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: 21:48:32. 2021-03-24T21:48:34.174 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: 694499+0000 mon.a (mon 2021-03-24T21:48:34.174 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: .0) 139 : audit 2021-03-24T21:48:34.174 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-03-24T21:48:34.174 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: audit 2021-03-24T21:48:34.175 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: 2021-03-24T21: 2021-03-24T21:48:34.175 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: cluster 2021-03-24T21:48:33. 2021-03-24T21:48:34.175 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: 733861+0000 mon.a (mon.0) 145 : cluster 2021-03-24T21:48:34.175 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: [DBG] osdmap e4: 0 total, 0 up, 0 in 2021-03-24T21:48:34.176 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: cluster 2021-03-24T21:48 2021-03-24T21:48:34.176 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: :33.734165+0000 mon.a (mon.0) 146 : 2021-03-24T21:48:34.176 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: cluster [DBG] mgrmap e13: y(active, since 89s) 2021-03-24T21:48:34.176 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:48:34.176 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: 03-24T21:48:33.743349+0000 2021-03-24T21:48:34.177 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: mon.a (mon.0) 147 : cluster [INF] overall HEALTH_OK 2021-03-24T21:48:34.177 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: 2021-03-24T21:48:34.177 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: audit 2021-03-24T21:48: 2021-03-24T21:48:34.177 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: 33.746987+0000 mon.a (mon.0) 148 : 2021-03-24T21:48:34.177 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:48:34.178 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T21:48:34.178 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: 24T21:48:33.752008+0000 mon.a 2021-03-24T21:48:34.178 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: (mon.0) 149 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:48:34.178 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[24927]: 2021-03-24T21:48:34.178 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: 48:33.694687 2021-03-24T21:48:34.179 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: +0000 mon.a (mon. 2021-03-24T21:48:34.179 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: 0) 140 : audit [DBG] 2021-03-24T21:48:34.179 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-03-24T21:48:34.179 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: cluster 2021-03-24T21:48:34.180 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: 2021-03-24T21 2021-03-24T21:48:34.180 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: :48:33.708316+0000 2021-03-24T21:48:34.180 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: mon.a (mon.0 2021-03-24T21:48:34.180 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: ) 141 : cluster [INF] mon.a calling monitor election 2021-03-24T21:48:34.180 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: 2021-03-24T21:48:34.181 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:48:34.181 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: 03-24T21: 2021-03-24T21:48:34.181 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: 48:33.728110+0000 2021-03-24T21:48:34.181 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: mon.a (mon.0 2021-03-24T21:48:34.182 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: ) 142 : cluster [INF] 2021-03-24T21:48:34.182 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2021-03-24T21:48:34.182 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: cluster 2021-03-24T21:48:34.182 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: 2021-03-24 2021-03-24T21:48:34.182 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: T21:48:33 2021-03-24T21:48:34.183 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: .733718+0000 mon.a ( 2021-03-24T21:48:34.183 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: mon.0) 143 : 2021-03-24T21:48:34.183 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: cluster [DBG] monmap e3: 3 mons at {a=v1:172.21.15.125:6789/0,b=v1:172.21.15.142:6789/0,c=v1:172.21.15.125:6790/0} 2021-03-24T21:48:34.183 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: cluster 2021-03-24T21:48:34.183 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: 2021-03-24T21: 2021-03-24T21:48:34.184 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: 48:33.733814+0000 2021-03-24T21:48:34.184 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: mon.a (mon.0 2021-03-24T21:48:34.184 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: ) 144 : cluster [DBG] 2021-03-24T21:48:34.184 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: fsmap 2021-03-24T21:48:34.184 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: cluster 2021-03-24T21:48:34.185 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: 2021-03-24 2021-03-24T21:48:34.185 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: T21:48:33 2021-03-24T21:48:34.185 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: .733861+0000 mon.a ( 2021-03-24T21:48:34.185 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: mon.0) 145 : 2021-03-24T21:48:34.185 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2021-03-24T21:48:34.186 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: cluster 2021-03-24T21:48:34.186 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: 2021-03-24T21: 2021-03-24T21:48:34.186 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: 48:33.734165+0000 2021-03-24T21:48:34.186 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: mon.a (mon.0 2021-03-24T21:48:34.186 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: ) 146 : cluster [DBG] 2021-03-24T21:48:34.187 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: mgrmap e13: y(active, since 89s) 2021-03-24T21:48:34.187 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: cluster 2021-03-24T21:48:34.187 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: 2021-03-24T21:48:33 2021-03-24T21:48:34.187 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: .743349+0000 mon.a ( 2021-03-24T21:48:34.187 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: mon.0) 147 : 2021-03-24T21:48:34.188 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: cluster [INF] overall HEALTH_OK 2021-03-24T21:48:34.188 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: audit 2021-03-24T21:48:34.188 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: 2021-03-24T21: 2021-03-24T21:48:34.188 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: 48:33.746987 2021-03-24T21:48:34.189 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: +0000 mon.a (mon. 2021-03-24T21:48:34.189 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: 0) 148 : audit 2021-03-24T21:48:34.189 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:48:34.189 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: audit 2021-03-24T21:48:34.189 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: 2021-03-24T21 2021-03-24T21:48:34.189 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: :48:33.752008 2021-03-24T21:48:34.190 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: +0000 mon.a (mon 2021-03-24T21:48:34.190 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: .0) 149 : audit 2021-03-24T21:48:34.190 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:33 smithi125 conmon[19936]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:48:34.568 INFO:tasks.cephadm:Generating final ceph.conf file... 2021-03-24T21:48:34.568 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph config generate-minimal-conf 2021-03-24T21:48:35.153 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:35 smithi125 conmon[24927]: audit 2021-03-24T21:48:34. 2021-03-24T21:48:35.153 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:35 smithi125 conmon[24927]: 048711+0000 mon.a (mon.0) 150 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:48:35.154 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:35 smithi125 conmon[24927]: cephadm 2021-03-24 2021-03-24T21:48:35.154 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:35 smithi125 conmon[24927]: T21:48:34.049869+0000 mgr.y (mgr.14144) 47 : cephadm [INF] 2021-03-24T21:48:35.154 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:35 smithi125 conmon[24927]: Reconfiguring mon.a (unknown last config time)... 2021-03-24T21:48:35.154 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:35 smithi125 conmon[24927]: audit 2021-03-24T21:48: 2021-03-24T21:48:35.155 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:35 smithi125 conmon[24927]: 34.050311+0000 mon.a (mon.0) 151 : 2021-03-24T21:48:35.155 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:35 smithi125 conmon[24927]: audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-03-24T21:48:35.155 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:35 smithi125 conmon[24927]: audit 2021-03-24T21:48:34.051587+0000 mon.a 2021-03-24T21:48:35.155 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:35 smithi125 conmon[24927]: (mon.0) 152 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2021-03-24T21:48:35.156 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:35 smithi125 conmon[24927]: audit 2021-03-24T21:48:34.052703+0000 mon.a 2021-03-24T21:48:35.156 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:35 smithi125 conmon[24927]: (mon.0) 153 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:48:35.156 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:35 smithi125 conmon[24927]: cephadm 2021-03-24T21:48:34.053583+0000 mgr.y 2021-03-24T21:48:35.156 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:35 smithi125 conmon[24927]: (mgr.14144) 48 : cephadm [INF] Reconfiguring daemon mon.a on smithi125 2021-03-24T21:48:35.157 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:35 smithi125 conmon[24927]: audit 2021-03-24T21:48 2021-03-24T21:48:35.157 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:35 smithi125 conmon[24927]: :34.694272+0000 mon.a (mon.0) 154 : audit [DBG] 2021-03-24T21:48:35.157 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:35 smithi125 conmon[24927]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-03-24T21:48:35.157 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:35 smithi125 conmon[19936]: audit 2021-03-24T21: 2021-03-24T21:48:35.158 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:35 smithi125 conmon[19936]: 48:34.048711+0000 mon.a (mon.0) 150 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:48:35.158 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:35 smithi125 conmon[19936]: cephadm 2021-03-24T21: 2021-03-24T21:48:35.158 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:35 smithi125 conmon[19936]: 48:34.049869+0000 mgr.y (mgr.14144) 47 2021-03-24T21:48:35.158 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:35 smithi125 conmon[19936]: : cephadm [INF] Reconfiguring mon.a (unknown last config time)... 2021-03-24T21:48:35.159 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:35 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:48:35.159 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:35 smithi125 conmon[19936]: 03-24T21:48:34.050311 2021-03-24T21:48:35.159 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:35 smithi125 conmon[19936]: +0000 mon.a (mon.0) 151 : 2021-03-24T21:48:35.159 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:35 smithi125 conmon[19936]: audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-03-24T21:48:35.159 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:35 smithi125 conmon[19936]: audit 2021-03-24T21:48 2021-03-24T21:48:35.160 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:35 smithi125 conmon[19936]: :34.051587+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2021-03-24T21:48:35.160 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:35 smithi125 conmon[19936]: 2021-03-24T21:48:35.160 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:35 smithi125 conmon[19936]: audit 2021-03-24 2021-03-24T21:48:35.160 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:35 smithi125 conmon[19936]: T21:48:34.052703+0000 mon.a (mon.0) 153 : 2021-03-24T21:48:35.161 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:35 smithi125 conmon[19936]: audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:48:35.161 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:35 smithi125 conmon[19936]: cephadm 2021-03-24T21:48:35.161 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:35 smithi125 conmon[19936]: 2021-03-24T21:48:34.053583+0000 mgr.y 2021-03-24T21:48:35.161 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:35 smithi125 conmon[19936]: (mgr.14144) 48 : cephadm [INF] Reconfiguring daemon mon.a on smithi125 2021-03-24T21:48:35.161 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:35 smithi125 conmon[19936]: 2021-03-24T21:48:35.162 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:35 smithi125 conmon[19936]: audit 2021-03- 2021-03-24T21:48:35.162 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:35 smithi125 conmon[19936]: 24T21:48:34.694272+0000 mon.a (mon.0 2021-03-24T21:48:35.162 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:35 smithi125 conmon[19936]: ) 154 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-03-24T21:48:35.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:35 smithi142 conmon[20125]: audit 2021-03-24T21:48:34.048711+0000 mon.a (mon.0) 150 : audit 2021-03-24T21:48:35.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:35 smithi142 conmon[20125]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:48:35.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:35 smithi142 conmon[20125]: cephadm 2021-03-24T21:48:34.049869+0000 2021-03-24T21:48:35.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:35 smithi142 conmon[20125]: mgr.y (mgr.14144) 47 : cephadm [INF] 2021-03-24T21:48:35.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:35 smithi142 conmon[20125]: Reconfiguring mon.a (unknown last config time)... 2021-03-24T21:48:35.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:35 smithi142 conmon[20125]: audit 2021 2021-03-24T21:48:35.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:35 smithi142 conmon[20125]: -03-24T21:48:34. 2021-03-24T21:48:35.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:35 smithi142 conmon[20125]: 050311+0000 mon.a (mon.0) 2021-03-24T21:48:35.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:35 smithi142 conmon[20125]: 151 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-03-24T21:48:35.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:35 smithi142 conmon[20125]: audit 2021 2021-03-24T21:48:35.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:35 smithi142 conmon[20125]: -03-24T21:48: 2021-03-24T21:48:35.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:35 smithi142 conmon[20125]: 34.051587+0000 mon.a (mon 2021-03-24T21:48:35.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:35 smithi142 conmon[20125]: .0) 152 : audit [DBG] 2021-03-24T21:48:35.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:35 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2021-03-24T21:48:35.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:35 smithi142 conmon[20125]: audit 2021 2021-03-24T21:48:35.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:35 smithi142 conmon[20125]: -03-24T21:48: 2021-03-24T21:48:35.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:35 smithi142 conmon[20125]: 34.052703+0000 mon.a (mon 2021-03-24T21:48:35.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:35 smithi142 conmon[20125]: .0) 153 : audit [DBG] 2021-03-24T21:48:35.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:35 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:48:35.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:35 smithi142 conmon[20125]: cephadm 2021 2021-03-24T21:48:35.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:35 smithi142 conmon[20125]: -03-24T21:48: 2021-03-24T21:48:35.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:35 smithi142 conmon[20125]: 34.053583+0000 mgr.y (mgr 2021-03-24T21:48:35.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:35 smithi142 conmon[20125]: .14144) 48 : cephadm [INF] 2021-03-24T21:48:35.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:35 smithi142 conmon[20125]: Reconfiguring daemon mon.a on smithi125 2021-03-24T21:48:35.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:35 smithi142 conmon[20125]: audit 2021 2021-03-24T21:48:35.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:35 smithi142 conmon[20125]: -03-24T21:48: 2021-03-24T21:48:35.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:35 smithi142 conmon[20125]: 34.694272+0000 mon.a (mon 2021-03-24T21:48:35.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:35 smithi142 conmon[20125]: .0) 154 : audit [DBG] 2021-03-24T21:48:35.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:35 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-03-24T21:48:36.153 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:36 smithi125 conmon[19936]: cluster 2021-03-24T21:48:34.585154+0000 2021-03-24T21:48:36.154 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:36 smithi125 conmon[19936]: mgr.y (mgr.14144) 49 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:36.154 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:36 smithi125 conmon[19936]: audit 2021-03-24T21:48:35. 2021-03-24T21:48:36.154 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:36 smithi125 conmon[19936]: 615076+0000 mon.a (mon.0) 155 : audit 2021-03-24T21:48:36.155 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:36 smithi125 conmon[19936]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:48:36.155 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:36 smithi125 conmon[19936]: audit 2021-03-24T21:48: 2021-03-24T21:48:36.155 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:36 smithi125 conmon[19936]: 35.616683+0000 mon.a (mon.0) 156 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-03-24T21:48:36.156 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:36 smithi125 conmon[19936]: audit 2021-03-24T21: 2021-03-24T21:48:36.156 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:36 smithi125 conmon[19936]: 48:35.618187+0000 mon.a (mon.0) 157 : audit [DBG] 2021-03-24T21:48:36.156 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:36 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2021-03-24T21:48:36.157 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:36 smithi125 conmon[19936]: audit 2021 2021-03-24T21:48:36.157 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:36 smithi125 conmon[19936]: -03-24T21:48:35.619497+0000 mon.a (mon.0) 158 : audit [DBG] 2021-03-24T21:48:36.158 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:36 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:48:36.158 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:36 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:48:36.158 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:36 smithi125 conmon[24927]: 48:34.585154+0000 mgr.y (mgr.14144) 49 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:36.159 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:36 smithi125 conmon[24927]: audit 2021-03-24T21:48:35.615076 2021-03-24T21:48:36.159 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:36 smithi125 conmon[24927]: +0000 mon.a (mon.0) 155 : audit [INF] 2021-03-24T21:48:36.159 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:36 smithi125 conmon[24927]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:48:36.159 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:36 smithi125 conmon[24927]: audit 2021-03-24T21 2021-03-24T21:48:36.159 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:36 smithi125 conmon[24927]: :48:35.616683+0000 mon.a (mon.0 2021-03-24T21:48:36.160 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:36 smithi125 conmon[24927]: ) 156 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-03-24T21:48:36.160 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:36 smithi125 conmon[24927]: audit 2021- 2021-03-24T21:48:36.161 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:36 smithi125 conmon[24927]: 03-24T21:48:35.618187+0000 2021-03-24T21:48:36.161 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:36 smithi125 conmon[24927]: mon.a (mon.0) 157 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2021-03-24T21:48:36.161 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:36 smithi125 conmon[24927]: audit 2021 2021-03-24T21:48:36.161 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:36 smithi125 conmon[24927]: -03-24T21:48:35.619497+0000 2021-03-24T21:48:36.162 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:36 smithi125 conmon[24927]: mon.a (mon.0) 158 : audit [DBG] 2021-03-24T21:48:36.162 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:36 smithi125 conmon[24927]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:48:36.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:36 smithi142 conmon[20125]: cluster 2021-03-24T21:48:34.585154+0000 mgr.y (mgr.14144) 49 2021-03-24T21:48:36.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:36 smithi142 conmon[20125]: : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:36.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:36 smithi142 conmon[20125]: audit 2021-03-24T21:48:35.615076+0000 2021-03-24T21:48:36.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:36 smithi142 conmon[20125]: mon.a (mon.0) 155 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:48:36.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:36 smithi142 conmon[20125]: audit 2021-03-24T21 2021-03-24T21:48:36.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:36 smithi142 conmon[20125]: :48:35.616683+0000 mon.a (mon.0 2021-03-24T21:48:36.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:36 smithi142 conmon[20125]: ) 156 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-03-24T21:48:36.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:36 smithi142 conmon[20125]: audit 2021-03 2021-03-24T21:48:36.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:36 smithi142 conmon[20125]: -24T21:48:35.618187+0000 mon.a 2021-03-24T21:48:36.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:36 smithi142 conmon[20125]: (mon.0) 157 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2021-03-24T21:48:36.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:36 smithi142 conmon[20125]: 2021-03-24T21:48:36.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:36 smithi142 conmon[20125]: audit 2021-03-24T21:48:36.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:36 smithi142 conmon[20125]: 2021-03-24T21:48:35.619497+0000 mon.a 2021-03-24T21:48:36.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:36 smithi142 conmon[20125]: (mon.0) 158 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:48:36.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:36 smithi142 conmon[20125]: 2021-03-24T21:48:37.055 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:37 smithi125 conmon[19936]: cephadm 2021-03-24T21:48:35.616171 2021-03-24T21:48:37.056 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:37 smithi125 conmon[19936]: +0000 mgr.y (mgr.14144) 50 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2021-03-24T21:48:37.056 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:37 smithi125 conmon[19936]: cephadm 2021-03-24T 2021-03-24T21:48:37.056 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:37 smithi125 conmon[19936]: 21:48:35.620495+0000 mgr.y (mgr.14144) 51 : cephadm [INF] Reconfiguring daemon mon.c on smithi125 2021-03-24T21:48:37.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:37 smithi142 conmon[20125]: cephadm 2021-03-24T21: 2021-03-24T21:48:37.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:37 smithi142 conmon[20125]: 48:35.616171+0000 mgr.y (mgr.14144) 50 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2021-03-24T21:48:37.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:37 smithi142 conmon[20125]: cephadm 2021-03-24T21:48:37.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:37 smithi142 conmon[20125]: 2021-03-24T21:48:35.620495+0000 mgr.y (mgr.14144 2021-03-24T21:48:37.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:37 smithi142 conmon[20125]: ) 51 : cephadm [INF] Reconfiguring daemon mon.c on smithi125 2021-03-24T21:48:37.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:37 smithi125 conmon[24927]: cephadm 2021-03- 2021-03-24T21:48:37.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:37 smithi125 conmon[24927]: 24T21:48:35.616171+0000 mgr.y (mgr.14144) 50 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2021-03-24T21:48:37.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:37 smithi125 conmon[24927]: cephadm 2021-03-24T21:48:37.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:37 smithi125 conmon[24927]: 2021-03-24T 2021-03-24T21:48:37.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:37 smithi125 conmon[24927]: 21:48:35.620495+0000 mgr.y (mgr. 2021-03-24T21:48:37.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:37 smithi125 conmon[24927]: 14144) 51 : cephadm [INF] Reconfiguring daemon mon.c on smithi125 2021-03-24T21:48:37.815 INFO:teuthology.orchestra.run.smithi125.stdout:# minimal ceph.conf for 3cfa339a-8cea-11eb-90b2-001a4aab830c 2021-03-24T21:48:37.816 INFO:teuthology.orchestra.run.smithi125.stdout:[global] 2021-03-24T21:48:37.816 INFO:teuthology.orchestra.run.smithi125.stdout: fsid = 3cfa339a-8cea-11eb-90b2-001a4aab830c 2021-03-24T21:48:37.816 INFO:teuthology.orchestra.run.smithi125.stdout: mon_host = 172.21.15.125:6789/0 172.21.15.142:6789/0 v1:172.21.15.125:6790/0 2021-03-24T21:48:38.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:38 smithi142 conmon[20125]: cluster 2021-03-24T21:48:36.586317+0000 mgr.y (mgr.14144) 2021-03-24T21:48:38.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:38 smithi142 conmon[20125]: 52 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:38.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:38 smithi142 conmon[20125]: audit 2021-03-24T21:48:37.802346+0000 2021-03-24T21:48:38.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:38 smithi142 conmon[20125]: mon.a (mon.0) 159 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:48:38.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:38 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:48:38.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:38 smithi142 conmon[20125]: 03-24T21:48:37.803374+0000 mon.a ( 2021-03-24T21:48:38.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:38 smithi142 conmon[20125]: mon.0) 160 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-03-24T21:48:38.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:38 smithi142 conmon[20125]: audit 2021-03-24T21 2021-03-24T21:48:38.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:38 smithi142 conmon[20125]: :48:37.804244+0000 mon.a (mon.0) 161 : 2021-03-24T21:48:38.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:38 smithi142 conmon[20125]: audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2021-03-24T21:48:38.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:38 smithi142 conmon[20125]: audit 2021-03-24T21:48: 2021-03-24T21:48:38.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:38 smithi142 conmon[20125]: 37.805255+0000 mon.a (mon.0) 162 2021-03-24T21:48:38.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:38 smithi142 conmon[20125]: : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:48:38.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:38 smithi142 conmon[20125]: audit 2021-03-24T21 2021-03-24T21:48:38.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:38 smithi142 conmon[20125]: :48:37.821553+0000 mon.a (mon 2021-03-24T21:48:38.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:38 smithi142 conmon[20125]: .0) 163 : audit 2021-03-24T21:48:38.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:38 smithi142 conmon[20125]: [DBG] from='client.? v1:172.21.15.125:0/1351986201' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:48:38.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:38 smithi125 conmon[19936]: cluster 2021-03-24T21:48:36.586317+0000 2021-03-24T21:48:38.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:38 smithi125 conmon[19936]: mgr.y (mgr.14144) 52 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:38.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:38 smithi125 conmon[19936]: audit 2021-03-24T21:48:37.802346+0000 mon.a 2021-03-24T21:48:38.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:38 smithi125 conmon[19936]: (mon.0) 159 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:48:38.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:38 smithi125 conmon[19936]: audit 2021-03-24T21 2021-03-24T21:48:38.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:38 smithi125 conmon[19936]: :48:37.803374+0000 mon.a (mon.0) 160 2021-03-24T21:48:38.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:38 smithi125 conmon[19936]: : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-03-24T21:48:38.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:38 smithi125 conmon[19936]: audit 2021-03-24T21:48:37.804244+0000 mon.a 2021-03-24T21:48:38.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:38 smithi125 conmon[19936]: (mon.0) 161 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2021-03-24T21:48:38.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:38 smithi125 conmon[19936]: audit 2021-03-24T21:48:37.805255+0000 mon.a 2021-03-24T21:48:38.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:38 smithi125 conmon[19936]: (mon.0) 162 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:48:38.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:38 smithi125 conmon[19936]: audit 2021-03 2021-03-24T21:48:38.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:38 smithi125 conmon[19936]: -24T21:48:37.821553+0000 mon.a 2021-03-24T21:48:38.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:38 smithi125 conmon[19936]: (mon.0) 163 : audit [DBG] from='client.? v1:172.21.15.125:0/1351986201' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:48:38.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:38 smithi125 conmon[19936]: 2021-03-24T21:48:38.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:38 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:48:38.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:38 smithi125 conmon[24927]: 48:36.586317+0000 mgr.y (mgr.14144) 52 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:38.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:38 smithi125 conmon[24927]: audit 2021-03-24T21:48: 2021-03-24T21:48:38.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:38 smithi125 conmon[24927]: 37.802346+0000 mon.a (mon.0) 159 : 2021-03-24T21:48:38.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:38 smithi125 conmon[24927]: audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:48:38.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:38 smithi125 conmon[24927]: audit 2021- 2021-03-24T21:48:38.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:38 smithi125 conmon[24927]: 03-24T21:48:37.803374+0000 mon.a 2021-03-24T21:48:38.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:38 smithi125 conmon[24927]: (mon.0) 160 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-03-24T21:48:38.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:38 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T21:48:38.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:38 smithi125 conmon[24927]: 24T21:48:37.804244+0000 mon.a (mon 2021-03-24T21:48:38.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:38 smithi125 conmon[24927]: .0) 161 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2021-03-24T21:48:38.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:38 smithi125 conmon[24927]: audit 2021-03-24 2021-03-24T21:48:38.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:38 smithi125 conmon[24927]: T21:48:37.805255+0000 mon.a (mon. 2021-03-24T21:48:38.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:38 smithi125 conmon[24927]: 0) 162 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:48:38.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:38 smithi125 conmon[24927]: audit 2021-03-24 2021-03-24T21:48:38.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:38 smithi125 conmon[24927]: T21:48:37.821553+0000 mon.a (mon. 2021-03-24T21:48:38.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:38 smithi125 conmon[24927]: 0) 163 : audit [DBG] from='client.? v1:172.21.15.125:0/1351986201' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:48:38.464 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2021-03-24T21:48:38.465 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2021-03-24T21:48:38.465 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd of=/etc/ceph/ceph.conf 2021-03-24T21:48:38.475 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2021-03-24T21:48:38.476 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2021-03-24T21:48:38.486 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2021-03-24T21:48:38.486 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/etc/ceph/ceph.conf 2021-03-24T21:48:38.503 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2021-03-24T21:48:38.504 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2021-03-24T21:48:38.562 INFO:tasks.cephadm:Adding mgr.y on smithi125 2021-03-24T21:48:38.563 INFO:tasks.cephadm:Adding mgr.x on smithi142 2021-03-24T21:48:38.563 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph orch apply mgr '2;smithi125=y;smithi142=x' 2021-03-24T21:48:39.350 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:39 smithi142 conmon[20125]: cephadm 2021-03-24T21:48:37.803022+0000 mgr.y (mgr.14144) 53 : cephadm 2021-03-24T21:48:39.351 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:39 smithi142 conmon[20125]: [INF] Reconfiguring mon.b (monmap changed)... 2021-03-24T21:48:39.351 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:39 smithi142 conmon[20125]: cephadm 2021-03-24T21:48:37.806402+0000 mgr.y (mgr.14144) 2021-03-24T21:48:39.351 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:39 smithi142 conmon[20125]: 54 : cephadm [INF] Reconfiguring daemon mon.b on smithi142 2021-03-24T21:48:39.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:39 smithi125 conmon[24927]: cephadm 2021-03-24T 2021-03-24T21:48:39.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:39 smithi125 conmon[24927]: 21:48:37.803022+0000 mgr.y (mgr.14144) 53 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2021-03-24T21:48:39.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:39 smithi125 conmon[24927]: cephadm 2021-03-24T21:48:37.806402+0000 mgr.y (mgr.14144) 54 : cephadm [INF] Reconfiguring daemon mon.b on smithi142 2021-03-24T21:48:39.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:39 smithi125 conmon[19936]: cephadm 2021-03-24T 2021-03-24T21:48:39.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:39 smithi125 conmon[19936]: 21:48:37.803022+0000 mgr.y (mgr.14144) 53 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2021-03-24T21:48:39.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:39 smithi125 conmon[19936]: cephadm 2021-03-24T21:48:39.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:39 smithi125 conmon[19936]: 2021-03-24T21:48:37.806402+0000 mgr.y (mgr.14144) 54 2021-03-24T21:48:39.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:39 smithi125 conmon[19936]: : cephadm [INF] Reconfiguring daemon mon.b on smithi142 2021-03-24T21:48:40.428 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:40 smithi142 conmon[20125]: cluster 2021-03-24T21:48: 2021-03-24T21:48:40.428 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:40 smithi142 conmon[20125]: 38.586749+0000 mgr.y (mgr.14144) 55 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:40.429 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:40 smithi142 conmon[20125]: audit 2021-03-24T21 2021-03-24T21:48:40.429 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:40 smithi142 conmon[20125]: :48:39.291034+0000 mon.a (mon.0) 164 : audit [INF] 2021-03-24T21:48:40.429 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:40 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:48:40.429 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:40 smithi142 conmon[20125]: audit 2021-03-24T21:48:40.430 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:40 smithi142 conmon[20125]: 2021-03-24T21:48:39.292447+0000 mon.a 2021-03-24T21:48:40.430 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:40 smithi142 conmon[20125]: (mon.0) 165 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:48:40.430 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:40 smithi142 conmon[20125]: 2021-03-24T21:48:40.431 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:40 smithi142 conmon[20125]: audit 2021-03-24 2021-03-24T21:48:40.431 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:40 smithi142 conmon[20125]: T21:48:39.301952+0000 mon.a (mon.0) 2021-03-24T21:48:40.431 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:40 smithi142 conmon[20125]: 166 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:48:40.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:40 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:48:40.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:40 smithi125 conmon[19936]: 21:48:38.586749+0000 mgr.y (mgr.14144) 55 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:40.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:40 smithi125 conmon[19936]: audit 2021-03-24T21: 2021-03-24T21:48:40.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:40 smithi125 conmon[19936]: 48:39.291034+0000 mon.a (mon.0 2021-03-24T21:48:40.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:40 smithi125 conmon[19936]: ) 164 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:48:40.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:40 smithi125 conmon[19936]: audit 2021-03- 2021-03-24T21:48:40.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:40 smithi125 conmon[19936]: 24T21:48:39.292447+0000 mon.a 2021-03-24T21:48:40.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:40 smithi125 conmon[19936]: (mon.0) 165 : audit [DBG] 2021-03-24T21:48:40.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:40 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:48:40.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:40 smithi125 conmon[19936]: audit 2021-03- 2021-03-24T21:48:40.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:40 smithi125 conmon[19936]: 24T21:48:39.301952+0000 mon.a 2021-03-24T21:48:40.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:40 smithi125 conmon[19936]: (mon.0) 166 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:48:40.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:40 smithi125 conmon[19936]: 2021-03-24T21:48:40.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:40 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:48:40.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:40 smithi125 conmon[24927]: 48:38.586749+0000 mgr.y (mgr.14144) 55 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:40.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:40 smithi125 conmon[24927]: audit 2021-03-24T21:48:40.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:40 smithi125 conmon[24927]: 2021-03-24T21:48:39.291034+0000 2021-03-24T21:48:40.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:40 smithi125 conmon[24927]: mon.a (mon.0) 164 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:48:40.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:40 smithi125 conmon[24927]: 2021-03-24T21:48:40.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:40 smithi125 conmon[24927]: audit 2021-03-24T21:48:40.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:40 smithi125 conmon[24927]: 2021-03-24T21:48:39 2021-03-24T21:48:40.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:40 smithi125 conmon[24927]: .292447+0000 mon.a (mon.0) 165 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:48:40.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:40 smithi125 conmon[24927]: audit 2021-03-24T 2021-03-24T21:48:40.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:40 smithi125 conmon[24927]: 21:48:39.301952+0000 mon.a (mon. 2021-03-24T21:48:40.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:40 smithi125 conmon[24927]: 0) 166 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:48:40.871 INFO:teuthology.orchestra.run.smithi142.stdout:Scheduled mgr update... 2021-03-24T21:48:41.671 DEBUG:teuthology.orchestra.run.smithi142:mgr.x> sudo journalctl -f -n 0 -u ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c@mgr.x.service 2021-03-24T21:48:41.674 INFO:tasks.cephadm:Deploying OSDs... 2021-03-24T21:48:41.674 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2021-03-24T21:48:41.675 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/scratch_devs of=/dev/stdout 2021-03-24T21:48:41.682 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-03-24T21:48:41.683 DEBUG:teuthology.orchestra.run.smithi125:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-03-24T21:48:41.690 INFO:teuthology.orchestra.run.smithi125.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-03-24T21:48:41.691 INFO:teuthology.orchestra.run.smithi125.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-24T21:48:41.691 INFO:teuthology.orchestra.run.smithi125.stdout:Device: 6h/6d Inode: 346 Links: 1 2021-03-24T21:48:41.691 INFO:teuthology.orchestra.run.smithi125.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-24T21:48:41.691 INFO:teuthology.orchestra.run.smithi125.stdout:Access: 2021-03-24 21:41:46.031653849 +0000 2021-03-24T21:48:41.692 INFO:teuthology.orchestra.run.smithi125.stdout:Modify: 2021-03-24 21:41:45.883653850 +0000 2021-03-24T21:48:41.692 INFO:teuthology.orchestra.run.smithi125.stdout:Change: 2021-03-24 21:41:45.883653850 +0000 2021-03-24T21:48:41.692 INFO:teuthology.orchestra.run.smithi125.stdout: Birth: - 2021-03-24T21:48:41.701 INFO:teuthology.orchestra.run.smithi125.stderr:1+0 records in 2021-03-24T21:48:41.701 INFO:teuthology.orchestra.run.smithi125.stderr:1+0 records out 2021-03-24T21:48:41.702 INFO:teuthology.orchestra.run.smithi125.stderr:512 bytes copied, 0.000444941 s, 1.2 MB/s 2021-03-24T21:48:41.706 DEBUG:teuthology.orchestra.run.smithi125:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-03-24T21:48:41.712 INFO:teuthology.orchestra.run.smithi125.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-03-24T21:48:41.712 INFO:teuthology.orchestra.run.smithi125.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-24T21:48:41.713 INFO:teuthology.orchestra.run.smithi125.stdout:Device: 6h/6d Inode: 351 Links: 1 2021-03-24T21:48:41.713 INFO:teuthology.orchestra.run.smithi125.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-24T21:48:41.713 INFO:teuthology.orchestra.run.smithi125.stdout:Access: 2021-03-24 21:41:46.467653844 +0000 2021-03-24T21:48:41.713 INFO:teuthology.orchestra.run.smithi125.stdout:Modify: 2021-03-24 21:41:46.291653846 +0000 2021-03-24T21:48:41.714 INFO:teuthology.orchestra.run.smithi125.stdout:Change: 2021-03-24 21:41:46.291653846 +0000 2021-03-24T21:48:41.714 INFO:teuthology.orchestra.run.smithi125.stdout: Birth: - 2021-03-24T21:48:41.716 INFO:journalctl@ceph.mgr.x.smithi142.stdout:-- Logs begin at Wed 2021-03-03 23:51:07 UTC. -- 2021-03-24T21:48:41.723 INFO:teuthology.orchestra.run.smithi125.stderr:1+0 records in 2021-03-24T21:48:41.723 INFO:teuthology.orchestra.run.smithi125.stderr:1+0 records out 2021-03-24T21:48:41.724 INFO:teuthology.orchestra.run.smithi125.stderr:512 bytes copied, 0.00040323 s, 1.3 MB/s 2021-03-24T21:48:41.728 DEBUG:teuthology.orchestra.run.smithi125:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-03-24T21:48:41.734 INFO:teuthology.orchestra.run.smithi125.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-03-24T21:48:41.734 INFO:teuthology.orchestra.run.smithi125.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-24T21:48:41.735 INFO:teuthology.orchestra.run.smithi125.stdout:Device: 6h/6d Inode: 358 Links: 1 2021-03-24T21:48:41.735 INFO:teuthology.orchestra.run.smithi125.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-24T21:48:41.735 INFO:teuthology.orchestra.run.smithi125.stdout:Access: 2021-03-24 21:41:46.485241998 +0000 2021-03-24T21:48:41.735 INFO:teuthology.orchestra.run.smithi125.stdout:Modify: 2021-03-24 21:41:46.325242000 +0000 2021-03-24T21:48:41.735 INFO:teuthology.orchestra.run.smithi125.stdout:Change: 2021-03-24 21:41:46.325242000 +0000 2021-03-24T21:48:41.736 INFO:teuthology.orchestra.run.smithi125.stdout: Birth: - 2021-03-24T21:48:41.740 INFO:teuthology.orchestra.run.smithi125.stderr:1+0 records in 2021-03-24T21:48:41.740 INFO:teuthology.orchestra.run.smithi125.stderr:1+0 records out 2021-03-24T21:48:41.741 INFO:teuthology.orchestra.run.smithi125.stderr:512 bytes copied, 0.00039345 s, 1.3 MB/s 2021-03-24T21:48:41.744 DEBUG:teuthology.orchestra.run.smithi125:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-03-24T21:48:41.791 INFO:teuthology.orchestra.run.smithi125.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-03-24T21:48:41.792 INFO:teuthology.orchestra.run.smithi125.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-24T21:48:41.792 INFO:teuthology.orchestra.run.smithi125.stdout:Device: 6h/6d Inode: 366 Links: 1 2021-03-24T21:48:41.792 INFO:teuthology.orchestra.run.smithi125.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-24T21:48:41.792 INFO:teuthology.orchestra.run.smithi125.stdout:Access: 2021-03-24 21:41:46.965241993 +0000 2021-03-24T21:48:41.793 INFO:teuthology.orchestra.run.smithi125.stdout:Modify: 2021-03-24 21:41:46.785241995 +0000 2021-03-24T21:48:41.793 INFO:teuthology.orchestra.run.smithi125.stdout:Change: 2021-03-24 21:41:46.785241995 +0000 2021-03-24T21:48:41.793 INFO:teuthology.orchestra.run.smithi125.stdout: Birth: - 2021-03-24T21:48:41.802 INFO:teuthology.orchestra.run.smithi125.stderr:1+0 records in 2021-03-24T21:48:41.802 INFO:teuthology.orchestra.run.smithi125.stderr:1+0 records out 2021-03-24T21:48:41.803 INFO:teuthology.orchestra.run.smithi125.stderr:512 bytes copied, 0.000395863 s, 1.3 MB/s 2021-03-24T21:48:41.807 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2021-03-24T21:48:41.808 DEBUG:teuthology.orchestra.run.smithi142:> dd if=/scratch_devs of=/dev/stdout 2021-03-24T21:48:41.815 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-03-24T21:48:41.815 DEBUG:teuthology.orchestra.run.smithi142:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-03-24T21:48:41.864 INFO:teuthology.orchestra.run.smithi142.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-03-24T21:48:41.865 INFO:teuthology.orchestra.run.smithi142.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-24T21:48:41.865 INFO:teuthology.orchestra.run.smithi142.stdout:Device: 6h/6d Inode: 344 Links: 1 2021-03-24T21:48:41.865 INFO:teuthology.orchestra.run.smithi142.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-24T21:48:41.865 INFO:teuthology.orchestra.run.smithi142.stdout:Access: 2021-03-24 21:41:46.282859894 +0000 2021-03-24T21:48:41.865 INFO:teuthology.orchestra.run.smithi142.stdout:Modify: 2021-03-24 21:41:46.138859895 +0000 2021-03-24T21:48:41.866 INFO:teuthology.orchestra.run.smithi142.stdout:Change: 2021-03-24 21:41:46.138859895 +0000 2021-03-24T21:48:41.866 INFO:teuthology.orchestra.run.smithi142.stdout: Birth: - 2021-03-24T21:48:41.875 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records in 2021-03-24T21:48:41.875 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records out 2021-03-24T21:48:41.876 INFO:teuthology.orchestra.run.smithi142.stderr:512 bytes copied, 0.000358292 s, 1.4 MB/s 2021-03-24T21:48:41.878 DEBUG:teuthology.orchestra.run.smithi142:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-03-24T21:48:41.883 INFO:teuthology.orchestra.run.smithi142.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-03-24T21:48:41.884 INFO:teuthology.orchestra.run.smithi142.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-24T21:48:41.884 INFO:teuthology.orchestra.run.smithi142.stdout:Device: 6h/6d Inode: 352 Links: 1 2021-03-24T21:48:41.884 INFO:teuthology.orchestra.run.smithi142.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-24T21:48:41.884 INFO:teuthology.orchestra.run.smithi142.stdout:Access: 2021-03-24 21:41:46.746859889 +0000 2021-03-24T21:48:41.884 INFO:teuthology.orchestra.run.smithi142.stdout:Modify: 2021-03-24 21:41:46.558859891 +0000 2021-03-24T21:48:41.885 INFO:teuthology.orchestra.run.smithi142.stdout:Change: 2021-03-24 21:41:46.558859891 +0000 2021-03-24T21:48:41.885 INFO:teuthology.orchestra.run.smithi142.stdout: Birth: - 2021-03-24T21:48:41.889 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:41 smithi142 conmon[20125]: cluster 2021-03-24T21:48:40.587155+0000 mgr.y (mgr. 2021-03-24T21:48:41.889 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:41 smithi142 conmon[20125]: 14144) 56 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:41.890 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:41 smithi142 conmon[20125]: audit 2021-03-24T21:48:40.869084+0000 mgr.y (mgr.14144) 57 : audit [DBG] from='client.24109 v1:172.21.15.142:0/891420461' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi125=y;smithi142=x", "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:48:41.890 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:41 smithi142 conmon[20125]: cephadm 2021-03-24T21:48:40.870362+0000 mgr.y (mgr.14144) 58 : cephadm [INF] Saving service mgr spec with placement smithi125=y;smithi142=x;count:2 2021-03-24T21:48:41.890 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:41 smithi142 conmon[20125]: audit 2021-03-24T21:48:40.876658+0000 2021-03-24T21:48:41.890 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:41 smithi142 conmon[20125]: mon.a (mon.0) 167 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:48:41.890 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:41 smithi142 conmon[20125]: audit 2021-03-24T21:48:41.891 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:41 smithi142 conmon[20125]: 2021-03-24T21:48:40.878004+0000 mon.a (mon.0) 168 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:48:41.891 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:41 smithi142 conmon[20125]: audit 2021-03- 2021-03-24T21:48:41.891 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:41 smithi142 conmon[20125]: 24T21:48:40.887414+0000 mon.a (mon.0) 169 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:48:41.891 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:41 smithi142 conmon[20125]: audit 2021-03-24T21:48:40.889234+0000 mon.a (mon.0) 170 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-03-24T21:48:41.892 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:41 smithi142 conmon[20125]: audit 2021-03-24T21:48:40.894677+0000 mon.a (mon.0) 171 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2021-03-24T21:48:41.892 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:41 smithi142 conmon[20125]: audit 2021-03-24T21:48:40.896082+0000 mon.a (mon. 2021-03-24T21:48:41.892 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:41 smithi142 conmon[20125]: 0) 172 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2021-03-24T21:48:41.892 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:41 smithi142 conmon[20125]: audit 2021-03-24T21:48:40.897828+0000 mon.a (mon.0) 173 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:48:41.892 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:41 smithi142 conmon[20125]: cephadm 2021 2021-03-24T21:48:41.893 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:41 smithi142 conmon[20125]: -03-24T21:48:40.899237+0000 mgr.y (mgr.14144) 59 : cephadm [INF] Deploying daemon mgr.x on smithi142 2021-03-24T21:48:41.893 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records in 2021-03-24T21:48:41.893 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records out 2021-03-24T21:48:41.894 INFO:teuthology.orchestra.run.smithi142.stderr:512 bytes copied, 0.00029461 s, 1.7 MB/s 2021-03-24T21:48:41.895 DEBUG:teuthology.orchestra.run.smithi142:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-03-24T21:48:41.900 INFO:teuthology.orchestra.run.smithi142.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-03-24T21:48:41.900 INFO:teuthology.orchestra.run.smithi142.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-24T21:48:41.901 INFO:teuthology.orchestra.run.smithi142.stdout:Device: 6h/6d Inode: 357 Links: 1 2021-03-24T21:48:41.901 INFO:teuthology.orchestra.run.smithi142.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-24T21:48:41.901 INFO:teuthology.orchestra.run.smithi142.stdout:Access: 2021-03-24 21:41:47.170859884 +0000 2021-03-24T21:48:41.901 INFO:teuthology.orchestra.run.smithi142.stdout:Modify: 2021-03-24 21:41:46.990859886 +0000 2021-03-24T21:48:41.901 INFO:teuthology.orchestra.run.smithi142.stdout:Change: 2021-03-24 21:41:46.990859886 +0000 2021-03-24T21:48:41.902 INFO:teuthology.orchestra.run.smithi142.stdout: Birth: - 2021-03-24T21:48:41.906 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records in 2021-03-24T21:48:41.907 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records out 2021-03-24T21:48:41.907 INFO:teuthology.orchestra.run.smithi142.stderr:512 bytes copied, 0.00035075 s, 1.5 MB/s 2021-03-24T21:48:41.911 DEBUG:teuthology.orchestra.run.smithi142:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-03-24T21:48:41.961 INFO:teuthology.orchestra.run.smithi142.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-03-24T21:48:41.961 INFO:teuthology.orchestra.run.smithi142.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-24T21:48:41.961 INFO:teuthology.orchestra.run.smithi142.stdout:Device: 6h/6d Inode: 363 Links: 1 2021-03-24T21:48:41.962 INFO:teuthology.orchestra.run.smithi142.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-24T21:48:41.962 INFO:teuthology.orchestra.run.smithi142.stdout:Access: 2021-03-24 21:41:47.594859880 +0000 2021-03-24T21:48:41.962 INFO:teuthology.orchestra.run.smithi142.stdout:Modify: 2021-03-24 21:41:47.426859881 +0000 2021-03-24T21:48:41.962 INFO:teuthology.orchestra.run.smithi142.stdout:Change: 2021-03-24 21:41:47.426859881 +0000 2021-03-24T21:48:41.963 INFO:teuthology.orchestra.run.smithi142.stdout: Birth: - 2021-03-24T21:48:41.968 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records in 2021-03-24T21:48:41.969 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records out 2021-03-24T21:48:41.969 INFO:teuthology.orchestra.run.smithi142.stderr:512 bytes copied, 0.000306382 s, 1.7 MB/s 2021-03-24T21:48:41.972 INFO:tasks.cephadm:Deploying osd.0 on smithi125 with /dev/vg_nvme/lv_4... 2021-03-24T21:48:41.972 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2021-03-24T21:48:41.985 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[19936]: cluster 2021-03-24T21:48:40. 2021-03-24T21:48:41.985 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[19936]: 587155+0000 mgr.y (mgr.14144) 56 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:41.985 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[19936]: audit 2021-03-24T21:48:40.869084+0000 mgr.y 2021-03-24T21:48:41.986 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[19936]: (mgr.14144) 57 : audit [DBG] from='client.24109 v1:172.21.15.142:0/891420461' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi125=y;smithi142=x", "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:48:41.986 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[19936]: cephadm 2021-03-24 2021-03-24T21:48:41.986 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[19936]: T21:48:40.870362+0000 mgr.y (mgr.14144) 58 : cephadm [INF] Saving service mgr spec with placement smithi125=y;smithi142=x;count:2 2021-03-24T21:48:41.986 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[19936]: audit 2021-03-24T21:48:40.876658+0000 2021-03-24T21:48:41.987 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[19936]: mon.a (mon.0) 167 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:48:41.987 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[19936]: audit 2021-03-24T21:48: 2021-03-24T21:48:41.987 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[19936]: 40.878004+0000 mon.a (mon.0) 168 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:48:41.987 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[19936]: audit 2021-03-24T21:48: 2021-03-24T21:48:41.987 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[19936]: 40.887414+0000 mon.a (mon.0) 169 : audit [INF] 2021-03-24T21:48:41.988 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:48:41.988 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[19936]: audit 2021-03-24T 2021-03-24T21:48:41.988 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[19936]: 21:48:40.889234+0000 mon.a (mon 2021-03-24T21:48:41.988 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[19936]: .0) 170 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-03-24T21:48:41.989 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[19936]: audit 2021-03- 2021-03-24T21:48:41.989 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[19936]: 24T21:48:40.894677+0000 mon.a 2021-03-24T21:48:41.989 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[19936]: (mon.0) 171 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2021-03-24T21:48:41.989 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[19936]: 2021-03-24T21:48:41.990 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:48:41.990 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[19936]: 03-24T21:48:40.896082+0000 mon.a (mon.0) 172 : audit [DBG] 2021-03-24T21:48:41.990 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2021-03-24T21:48:41.990 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:48:41.990 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[19936]: 03-24T21:48:40.897828+0000 mon.a (mon.0 2021-03-24T21:48:41.991 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[19936]: ) 173 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:48:41.991 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[19936]: cephadm 2021-03-24T21 2021-03-24T21:48:41.991 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[19936]: :48:40.899237+0000 mgr.y (mgr.14144) 59 2021-03-24T21:48:41.991 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[19936]: : cephadm [INF] Deploying daemon mgr.x on smithi142 2021-03-24T21:48:41.992 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[24927]: cluster 2021-03-24T21:48:40 2021-03-24T21:48:41.992 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[24927]: .587155+0000 mgr.y (mgr.14144) 56 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:41.992 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[24927]: audit 2021-03-24T21:48:41.992 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[24927]: 2021-03-24T21:48:40.869084+0000 mgr.y (mgr. 2021-03-24T21:48:41.993 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[24927]: 14144) 57 : audit [DBG] from='client.24109 v1:172.21.15.142:0/891420461' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi125=y;smithi142=x", "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:48:41.993 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[24927]: cephadm 2021-03-24T21:48:41.993 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[24927]: 2021-03-24 2021-03-24T21:48:41.993 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[24927]: T21:48:40.870362+0000 mgr.y (mgr. 2021-03-24T21:48:41.994 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[24927]: 14144) 58 : cephadm [INF] Saving service mgr spec with placement smithi125=y;smithi142=x;count:2 2021-03-24T21:48:41.994 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T21:48:41.994 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[24927]: 24T21:48:40.876658+0000 mon.a 2021-03-24T21:48:41.994 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[24927]: (mon.0) 167 : audit 2021-03-24T21:48:41.994 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[24927]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:48:41.995 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[24927]: audit 2021-03-24T21: 2021-03-24T21:48:41.995 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[24927]: 48:40.878004+0000 mon.a (mon.0) 168 : audit 2021-03-24T21:48:41.995 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[24927]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:48:41.995 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[24927]: audit 2021-03-24T21:48 2021-03-24T21:48:41.996 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[24927]: :40.887414+0000 mon.a (mon.0) 169 : audit 2021-03-24T21:48:41.996 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[24927]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:48:41.996 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[24927]: audit 2021-03-24T21:48:41.996 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[24927]: 2021-03-24T21:48:40. 2021-03-24T21:48:41.996 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[24927]: 889234+0000 mon.a (mon.0) 170 : audit [INF] 2021-03-24T21:48:41.997 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[24927]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-03-24T21:48:41.997 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[24927]: audit 2021 2021-03-24T21:48:41.997 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[24927]: -03-24T21:48:40.894677+0000 mon.a (mon.0) 171 2021-03-24T21:48:41.997 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[24927]: : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2021-03-24T21:48:41.998 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[24927]: audit 2021-03-24T21: 2021-03-24T21:48:41.998 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[24927]: 48:40.896082+0000 mon.a (mon.0) 172 : audit [DBG] 2021-03-24T21:48:41.998 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[24927]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2021-03-24T21:48:41.998 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T21:48:41.998 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[24927]: 24T21:48:40.897828+0000 mon.a (mon.0) 173 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:48:41.999 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[24927]: cephadm 2021-03-24T21:48:41.999 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[24927]: 2021-03-24T21:48:40.899237 2021-03-24T21:48:41.999 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:41 smithi125 conmon[24927]: +0000 mgr.y (mgr.14144) 59 : cephadm [INF] Deploying daemon mgr.x on smithi142 2021-03-24T21:48:43.363 INFO:journalctl@ceph.mgr.x.smithi142.stdout:Mar 24 21:48:43 smithi142 systemd[1]: Starting Ceph mgr.x for 3cfa339a-8cea-11eb-90b2-001a4aab830c... 2021-03-24T21:48:43.863 INFO:journalctl@ceph.mgr.x.smithi142.stdout:Mar 24 21:48:43 smithi142 podman[21510]: 2021-03-24 21:48:43.581166125 +0000 UTC m=+0.212842837 container create 4d2fc5efd5618d6f8fe6bf0c2caf2720514de1bcca505ea1e2d147614f487001 (image=quay.ceph.io/ceph-ci/ceph@sha256:e9db06906b8d0a24f3ef6835ac5aa4b8eea2950a2560bad78f83d4f101eca0db, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-mgr.x, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06, io.buildah.version=1.18.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20201204, RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d, CEPH_POINT_RELEASE=, maintainer=Dimitri Savineau , org.label-schema.schema-version=1.0) 2021-03-24T21:48:43.864 INFO:journalctl@ceph.mgr.x.smithi142.stdout:Mar 24 21:48:43 smithi142 podman[21510]: 2021-03-24 21:48:43.722834728 +0000 UTC m=+0.354511420 container init 4d2fc5efd5618d6f8fe6bf0c2caf2720514de1bcca505ea1e2d147614f487001 (image=quay.ceph.io/ceph-ci/ceph@sha256:e9db06906b8d0a24f3ef6835ac5aa4b8eea2950a2560bad78f83d4f101eca0db, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-mgr.x, io.buildah.version=1.18.0, org.label-schema.build-date=20201204, GIT_CLEAN=True, GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06, RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Dimitri Savineau ) 2021-03-24T21:48:43.864 INFO:journalctl@ceph.mgr.x.smithi142.stdout:Mar 24 21:48:43 smithi142 podman[21510]: 2021-03-24 21:48:43.764416741 +0000 UTC m=+0.396093379 container start 4d2fc5efd5618d6f8fe6bf0c2caf2720514de1bcca505ea1e2d147614f487001 (image=quay.ceph.io/ceph-ci/ceph@sha256:e9db06906b8d0a24f3ef6835ac5aa4b8eea2950a2560bad78f83d4f101eca0db, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-mgr.x, maintainer=Dimitri Savineau , ceph=True, io.buildah.version=1.18.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06, org.label-schema.build-date=20201204, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d, GIT_BRANCH=HEAD) 2021-03-24T21:48:43.864 INFO:journalctl@ceph.mgr.x.smithi142.stdout:Mar 24 21:48:43 smithi142 bash[21510]: 4d2fc5efd5618d6f8fe6bf0c2caf2720514de1bcca505ea1e2d147614f487001 2021-03-24T21:48:43.865 INFO:journalctl@ceph.mgr.x.smithi142.stdout:Mar 24 21:48:43 smithi142 systemd[1]: Started Ceph mgr.x for 3cfa339a-8cea-11eb-90b2-001a4aab830c. 2021-03-24T21:48:44.153 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:43 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:48:44.153 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:43 smithi125 conmon[19936]: 48:42.587481+0000 mgr.y (mgr.14144) 60 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:44.153 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:43 smithi125 conmon[19936]: audit 2021-03-24T21:48:44.154 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:43 smithi125 conmon[19936]: 2021-03-24T21:48 2021-03-24T21:48:44.154 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:43 smithi125 conmon[19936]: :43.817999+0000 mon.a (mon.0) 174 : audit 2021-03-24T21:48:44.154 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:43 smithi125 conmon[19936]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:48:44.154 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:43 smithi125 conmon[19936]: audit 2021-03-24T21:48:43.819636 2021-03-24T21:48:44.155 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:43 smithi125 conmon[19936]: +0000 mon.a (mon.0) 175 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:48:44.155 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:43 smithi125 conmon[19936]: 2021-03-24T21:48:44.155 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:43 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:48:44.155 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:43 smithi125 conmon[24927]: 21:48:42.587481+0000 mgr.y (mgr.14144) 60 : cluster [DBG] 2021-03-24T21:48:44.156 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:43 smithi125 conmon[24927]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:44.156 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:43 smithi125 conmon[24927]: audit 2021-03-24T21:48:44.156 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:43 smithi125 conmon[24927]: 2021-03-24T21:48:43.817999+0000 mon.a ( 2021-03-24T21:48:44.156 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:43 smithi125 conmon[24927]: mon.0) 174 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:48:44.157 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:43 smithi125 conmon[24927]: audit 2021 2021-03-24T21:48:44.157 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:43 smithi125 conmon[24927]: -03-24T21:48:43.819636+0000 2021-03-24T21:48:44.157 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:43 smithi125 conmon[24927]: mon.a (mon.0) 175 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:48:44.157 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:43 smithi125 conmon[24927]: 2021-03-24T21:48:44.338 INFO:teuthology.orchestra.run.smithi125.stderr:--> Zapping: /dev/vg_nvme/lv_4 2021-03-24T21:48:44.338 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2021-03-24T21:48:44.339 INFO:teuthology.orchestra.run.smithi125.stderr: stderr: 10+0 records in 2021-03-24T21:48:44.339 INFO:teuthology.orchestra.run.smithi125.stderr:10+0 records out 2021-03-24T21:48:44.339 INFO:teuthology.orchestra.run.smithi125.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0189486 s, 553 MB/s 2021-03-24T21:48:44.339 INFO:teuthology.orchestra.run.smithi125.stderr:--> Zapping successful for: 2021-03-24T21:48:44.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:43 smithi142 conmon[20125]: cluster 2021-03-24T21:48 2021-03-24T21:48:44.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:43 smithi142 conmon[20125]: :42.587481+0000 mgr.y (mgr.14144) 60 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:44.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:43 smithi142 conmon[20125]: audit 2021-03 2021-03-24T21:48:44.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:43 smithi142 conmon[20125]: -24T21:48:43.817999+0000 mon.a 2021-03-24T21:48:44.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:43 smithi142 conmon[20125]: (mon.0) 174 : audit [INF] 2021-03-24T21:48:44.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:43 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:48:44.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:43 smithi142 conmon[20125]: audit 2021 2021-03-24T21:48:44.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:43 smithi142 conmon[20125]: -03-24T21:48:43.819636+0000 2021-03-24T21:48:44.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:43 smithi142 conmon[20125]: mon.a (mon.0) 175 : audit 2021-03-24T21:48:44.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:43 smithi142 conmon[20125]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:48:45.110 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph orch daemon add osd smithi125:vg_nvme/lv_4 2021-03-24T21:48:45.997 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:45 smithi125 conmon[24927]: cluster 2021-03-24T21:48:44.587859+0000 2021-03-24T21:48:45.997 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:45 smithi125 conmon[24927]: mgr.y (mgr.14144) 61 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:45.997 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:45 smithi125 conmon[24927]: audit 2021-03-24T21:48:44.877744+0000 2021-03-24T21:48:45.998 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:45 smithi125 conmon[24927]: mon.a (mon.0) 176 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:48:45.998 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:45 smithi125 conmon[24927]: audit 2021-03-24T 2021-03-24T21:48:45.998 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:45 smithi125 conmon[24927]: 21:48:44.884484+0000 mon.a (mon.0) 177 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:48:45.998 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:45 smithi125 conmon[24927]: cephadm 2021-03-24T21:48:45.999 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:45 smithi125 conmon[24927]: 2021-03-24T21:48:44.886346+0000 mgr.y (mgr.14144) 62 : cephadm [INF] 2021-03-24T21:48:45.999 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:45 smithi125 conmon[24927]: Reconfiguring mgr.y (unknown last config time)... 2021-03-24T21:48:45.999 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:45 smithi125 conmon[24927]: audit 2021-03-24T 2021-03-24T21:48:45.999 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:45 smithi125 conmon[24927]: 21:48:44.889191+0000 mon.a (mon.0) 178 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-03-24T21:48:46.000 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:45 smithi125 conmon[24927]: audit 2021-03-24T21:48:44.890703 2021-03-24T21:48:46.000 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:45 smithi125 conmon[24927]: +0000 mon.a (mon.0) 179 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2021-03-24T21:48:46.000 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:45 smithi125 conmon[24927]: audit 2021-03-24T21:48:44 2021-03-24T21:48:46.000 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:45 smithi125 conmon[24927]: .891911+0000 mon.a (mon.0) 180 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:48:46.000 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:45 smithi125 conmon[24927]: cephadm 2021-03-24T 2021-03-24T21:48:46.001 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:45 smithi125 conmon[24927]: 21:48:44.892908+0000 mgr.y (mgr.14144) 2021-03-24T21:48:46.001 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:45 smithi125 conmon[24927]: 63 : cephadm [INF] Reconfiguring daemon mgr.y on smithi125 2021-03-24T21:48:46.001 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:45 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:48:46.002 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:45 smithi125 conmon[19936]: 48:44.587859+0000 mgr.y (mgr.14144) 61 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:46.002 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:45 smithi125 conmon[19936]: audit 2021-03-24T21: 2021-03-24T21:48:46.002 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:45 smithi125 conmon[19936]: 48:44.877744+0000 mon.a ( 2021-03-24T21:48:46.002 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:45 smithi125 conmon[19936]: mon.0) 176 : audit [INF] 2021-03-24T21:48:46.003 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:45 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:48:46.003 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:45 smithi125 conmon[19936]: audit 2021-03-24 2021-03-24T21:48:46.003 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:45 smithi125 conmon[19936]: T21:48:44.884484+0000 2021-03-24T21:48:46.003 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:45 smithi125 conmon[19936]: mon.a (mon.0) 177 : audit [INF] 2021-03-24T21:48:46.003 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:45 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:48:46.004 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:45 smithi125 conmon[19936]: cephadm 2021-03-24T 2021-03-24T21:48:46.004 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:45 smithi125 conmon[19936]: 21:48:44.886346+0000 mgr.y 2021-03-24T21:48:46.004 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:45 smithi125 conmon[19936]: (mgr.14144) 62 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2021-03-24T21:48:46.004 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:45 smithi125 conmon[19936]: 2021-03-24T21:48:46.004 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:45 smithi125 conmon[19936]: audit 2021-03-24T21:48:46.005 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:45 smithi125 conmon[19936]: 2021-03-24T21:48: 2021-03-24T21:48:46.005 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:45 smithi125 conmon[19936]: 44.889191+0000 mon.a (mon.0) 178 2021-03-24T21:48:46.005 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:45 smithi125 conmon[19936]: : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-03-24T21:48:46.005 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:45 smithi125 conmon[19936]: audit 2021-03 2021-03-24T21:48:46.005 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:45 smithi125 conmon[19936]: -24T21:48:44.890703 2021-03-24T21:48:46.006 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:45 smithi125 conmon[19936]: +0000 mon.a (mon.0) 179 2021-03-24T21:48:46.006 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:45 smithi125 conmon[19936]: : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2021-03-24T21:48:46.006 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:45 smithi125 conmon[19936]: audit 2021 2021-03-24T21:48:46.006 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:45 smithi125 conmon[19936]: -03-24T21:48:44.891911+0000 2021-03-24T21:48:46.007 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:45 smithi125 conmon[19936]: mon.a (mon.0) 180 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:48:46.007 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:45 smithi125 conmon[19936]: 2021-03-24T21:48:46.007 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:45 smithi125 conmon[19936]: cephadm 2021-03-24T21:48:44 2021-03-24T21:48:46.007 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:45 smithi125 conmon[19936]: .892908+0000 mgr.y (mgr.14144) 63 2021-03-24T21:48:46.007 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:45 smithi125 conmon[19936]: : cephadm [INF] Reconfiguring daemon mgr.y on smithi125 2021-03-24T21:48:46.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:45 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:48:46.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:45 smithi142 conmon[20125]: 48:44.587859+0000 mgr.y (mgr.14144) 61 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:46.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:45 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:48:46.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:45 smithi142 conmon[20125]: 03-24T21:48: 2021-03-24T21:48:46.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:45 smithi142 conmon[20125]: 44.877744+0000 mon.a ( 2021-03-24T21:48:46.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:45 smithi142 conmon[20125]: mon.0) 176 : audit 2021-03-24T21:48:46.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:45 smithi142 conmon[20125]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:48:46.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:45 smithi142 conmon[20125]: audit 2021 2021-03-24T21:48:46.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:45 smithi142 conmon[20125]: -03-24T21:48: 2021-03-24T21:48:46.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:45 smithi142 conmon[20125]: 44.884484+0000 mon.a ( 2021-03-24T21:48:46.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:45 smithi142 conmon[20125]: mon.0) 177 : audit 2021-03-24T21:48:46.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:45 smithi142 conmon[20125]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:48:46.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:45 smithi142 conmon[20125]: cephadm 2021-03-24T21:48:46.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:45 smithi142 conmon[20125]: 2021-03-24T21 2021-03-24T21:48:46.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:45 smithi142 conmon[20125]: :48:44.886346+0000 mgr.y 2021-03-24T21:48:46.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:45 smithi142 conmon[20125]: (mgr.14144) 62 : 2021-03-24T21:48:46.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:45 smithi142 conmon[20125]: cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2021-03-24T21:48:46.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:45 smithi142 conmon[20125]: audit 2021 2021-03-24T21:48:46.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:45 smithi142 conmon[20125]: -03-24T21:48 2021-03-24T21:48:46.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:45 smithi142 conmon[20125]: :44.889191+0000 mon.a 2021-03-24T21:48:46.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:45 smithi142 conmon[20125]: (mon.0) 178 2021-03-24T21:48:46.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:45 smithi142 conmon[20125]: : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-03-24T21:48:46.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:45 smithi142 conmon[20125]: audit 2021 2021-03-24T21:48:46.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:45 smithi142 conmon[20125]: -03-24T21:48 2021-03-24T21:48:46.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:45 smithi142 conmon[20125]: :44.890703+0000 mon.a ( 2021-03-24T21:48:46.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:45 smithi142 conmon[20125]: mon.0) 179 : audit 2021-03-24T21:48:46.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:45 smithi142 conmon[20125]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2021-03-24T21:48:46.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:45 smithi142 conmon[20125]: audit 2021-03-24T21:48:46.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:45 smithi142 conmon[20125]: 2021-03-24T21 2021-03-24T21:48:46.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:45 smithi142 conmon[20125]: :48:44.891911+0000 2021-03-24T21:48:46.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:45 smithi142 conmon[20125]: mon.a (mon.0) 2021-03-24T21:48:46.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:45 smithi142 conmon[20125]: 180 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:48:46.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:45 smithi142 conmon[20125]: 2021-03-24T21:48:46.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:45 smithi142 conmon[20125]: cephadm 2021-03- 2021-03-24T21:48:46.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:45 smithi142 conmon[20125]: 24T21:48:44.892908+0000 2021-03-24T21:48:46.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:45 smithi142 conmon[20125]: mgr.y (mgr.14144) 63 : 2021-03-24T21:48:46.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:45 smithi142 conmon[20125]: cephadm [INF] Reconfiguring daemon mgr.y on smithi125 2021-03-24T21:48:47.652 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:47 smithi125 conmon[19936]: audit 2021-03-24T21: 2021-03-24T21:48:47.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:47 smithi125 conmon[19936]: 48:46.381517+0000 mon.a (mon.0) 181 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:48:47.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:47 smithi125 conmon[19936]: audit 2021-03-24T 2021-03-24T21:48:47.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:47 smithi125 conmon[19936]: 21:48:46.382929+0000 mon.a (mon.0 2021-03-24T21:48:47.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:47 smithi125 conmon[19936]: ) 182 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:48:47.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:47 smithi125 conmon[19936]: audit 2021-03-24T21:48:47.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:47 smithi125 conmon[19936]: 2021-03-24T21:48:46.389131 2021-03-24T21:48:47.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:47 smithi125 conmon[19936]: +0000 mon.a (mon.0) 183 : audit [INF] 2021-03-24T21:48:47.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:47 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:48:47.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:47 smithi125 conmon[24927]: audit 2021-03-24T21:48 2021-03-24T21:48:47.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:47 smithi125 conmon[24927]: :46.381517+0000 mon.a (mon.0) 181 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:48:47.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:47 smithi125 conmon[24927]: audit 2021-03-24T21:48:47.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:47 smithi125 conmon[24927]: 2021-03-24T21:48:46.382929+0000 mon.a (mon.0) 182 : audit 2021-03-24T21:48:47.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:47 smithi125 conmon[24927]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:48:47.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:47 smithi125 conmon[24927]: audit 2021-03-24T21:48:47.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:47 smithi125 conmon[24927]: 2021-03-24T21:48:46.389131 2021-03-24T21:48:47.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:47 smithi125 conmon[24927]: +0000 mon.a (mon.0) 183 : audit [INF] 2021-03-24T21:48:47.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:47 smithi125 conmon[24927]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:48:47.863 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:47 smithi142 conmon[20125]: audit 2021-03-24T21 2021-03-24T21:48:47.864 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:47 smithi142 conmon[20125]: :48:46.381517+0000 mon.a (mon.0) 181 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:48:47.864 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:47 smithi142 conmon[20125]: audit 2021-03-24T21:48:47.864 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:47 smithi142 conmon[20125]: 2021-03-24T21 2021-03-24T21:48:47.865 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:47 smithi142 conmon[20125]: :48:46.382929 2021-03-24T21:48:47.865 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:47 smithi142 conmon[20125]: +0000 mon.a (mon.0 2021-03-24T21:48:47.865 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:47 smithi142 conmon[20125]: ) 182 : audit [DBG] 2021-03-24T21:48:47.865 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:47 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:48:47.865 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:47 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:48:47.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:47 smithi142 conmon[20125]: 03-24T21: 2021-03-24T21:48:47.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:47 smithi142 conmon[20125]: 48:46.389131+0000 2021-03-24T21:48:47.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:47 smithi142 conmon[20125]: mon.a (mon.0) 2021-03-24T21:48:47.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:47 smithi142 conmon[20125]: 183 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:48:47.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:47 smithi142 conmon[20125]: 2021-03-24T21:48:48.565 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:48 smithi125 conmon[19936]: cluster 2021-03-24T21:48 2021-03-24T21:48:48.565 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:48 smithi125 conmon[19936]: :46.588257+0000 mgr.y (mgr.14144) 64 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:48.566 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:48 smithi125 conmon[19936]: audit 2021-03-24T21:48:47.823559+0000 2021-03-24T21:48:48.566 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:48 smithi125 conmon[19936]: mon.a (mon.0) 184 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-24T21:48:48.566 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:48 smithi125 conmon[19936]: 2021-03-24T21:48:48.566 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:48 smithi125 conmon[19936]: audit 2021-03-24T21:48:48.567 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:48 smithi125 conmon[19936]: 2021-03-24T21:48:47. 2021-03-24T21:48:48.567 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:48 smithi125 conmon[19936]: 826412+0000 mon.a (mon.0) 185 : audit 2021-03-24T21:48:48.567 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:48 smithi125 conmon[19936]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-24T21:48:48.567 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:48 smithi125 conmon[19936]: audit 2021-03-24T21:48:48.567 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:48 smithi125 conmon[19936]: 2021-03 2021-03-24T21:48:48.568 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:48 smithi125 conmon[19936]: -24T21:48:47. 2021-03-24T21:48:48.568 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:48 smithi125 conmon[19936]: 827633+0000 mon.a (mon.0) 2021-03-24T21:48:48.568 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:48 smithi125 conmon[19936]: 186 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:48:48.568 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:48 smithi125 conmon[24927]: cluster 2021-03-24T21:48:46.588257+0000 mgr.y (mgr.14144) 64 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:48.569 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:48 smithi125 conmon[24927]: audit 2021-03-24T21:48:47.823559+0000 mon.a (mon.0) 184 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-24T21:48:48.569 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:48 smithi125 conmon[24927]: audit 2021-03-24T 2021-03-24T21:48:48.569 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:48 smithi125 conmon[24927]: 21:48:47.826412+0000 mon.a (mon.0) 185 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-24T21:48:48.569 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:48 smithi125 conmon[24927]: audit 2021-03-24T21:48:47.827633+0000 mon.a (mon.0) 186 : 2021-03-24T21:48:48.570 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:48 smithi125 conmon[24927]: audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:48:48.863 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:48 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:48:48.864 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:48 smithi142 conmon[20125]: 48:46.588257+0000 mgr.y (mgr.14144) 64 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:48.864 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:48 smithi142 conmon[20125]: audit 2021-03-24T21:48:48.864 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:48 smithi142 conmon[20125]: 2021-03-24T21: 2021-03-24T21:48:48.864 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:48 smithi142 conmon[20125]: 48:47.823559+0000 2021-03-24T21:48:48.865 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:48 smithi142 conmon[20125]: mon.a (mon.0) 2021-03-24T21:48:48.865 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:48 smithi142 conmon[20125]: 184 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-24T21:48:48.865 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:48 smithi142 conmon[20125]: 2021-03-24T21:48:48.865 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:48 smithi142 conmon[20125]: audit 2021-03-24T21:48:48.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:48 smithi142 conmon[20125]: 2021-03- 2021-03-24T21:48:48.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:48 smithi142 conmon[20125]: 24T21:48 2021-03-24T21:48:48.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:48 smithi142 conmon[20125]: :47.826412+0000 2021-03-24T21:48:48.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:48 smithi142 conmon[20125]: mon.a (mon.0) 185 2021-03-24T21:48:48.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:48 smithi142 conmon[20125]: : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-24T21:48:48.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:48 smithi142 conmon[20125]: audit 2021-03-24T21:48:48.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:48 smithi142 conmon[20125]: 2021-03-24T 2021-03-24T21:48:48.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:48 smithi142 conmon[20125]: 21:48:47.827633 2021-03-24T21:48:48.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:48 smithi142 conmon[20125]: +0000 mon.a (mon. 2021-03-24T21:48:48.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:48 smithi142 conmon[20125]: 0) 186 : audit [DBG] 2021-03-24T21:48:48.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:48 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:48:49.547 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:49 smithi125 conmon[19936]: audit 2021-03-24T21:48:47.822090+0000 mgr.y (mgr.14144) 65 : audit [DBG] from='client.34115 v1:172.21.15.125:0/1619282178' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi125:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:48:49.547 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:49 smithi125 conmon[19936]: cephadm 2021-03-24T21:48:47.826055+0000 mgr.y (mgr.14144) 66 : cephadm [INF] Applying service osd.None on host smithi125... 2021-03-24T21:48:49.548 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:49 smithi125 conmon[19936]: cluster 2021-03-24T21:48:48.919467+0000 mon.a (mon.0) 187 : cluster [DBG] Standby manager daemon x started 2021-03-24T21:48:49.548 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:49 smithi125 conmon[19936]: audit 2021-03-24T21:48:48.920697+0000 mon.b (mon.1) 3 : 2021-03-24T21:48:49.548 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:49 smithi125 conmon[19936]: audit [DBG] from='mgr.? v1:172.21.15.142:0/1104290420' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2021-03-24T21:48:49.548 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:49 smithi125 conmon[19936]: audit 2021-03-24T21:48:48.921392+0000 mon.b (mon.1) 4 : audit [DBG] from='mgr.? v1:172.21.15.142:0/1104290420' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2021-03-24T21:48:49.549 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:49 smithi125 conmon[19936]: audit 2021-03-24T21:48:48.923244+0000 2021-03-24T21:48:49.549 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:49 smithi125 conmon[19936]: mon.b (mon.1) 5 : audit [DBG] from='mgr.? v1:172.21.15.142:0/1104290420' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2021-03-24T21:48:49.549 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:49 smithi125 conmon[19936]: audit 2021-03-24T21:48:48.923938+0000 mon.b (mon.1) 6 : audit [DBG] from='mgr.? v1:172.21.15.142:0/1104290420' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2021-03-24T21:48:49.550 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:49 smithi125 conmon[24927]: audit 2021-03-24T21: 2021-03-24T21:48:49.550 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:49 smithi125 conmon[24927]: 48:47.822090+0000 mgr.y (mgr.14144) 65 : audit [DBG] from='client.34115 v1:172.21.15.125:0/1619282178' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi125:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:48:49.550 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:49 smithi125 conmon[24927]: cephadm 2021-03-24T21:48: 2021-03-24T21:48:49.550 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:49 smithi125 conmon[24927]: 47.826055+0000 mgr.y (mgr.14144) 66 : cephadm 2021-03-24T21:48:49.551 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:49 smithi125 conmon[24927]: [INF] Applying service osd.None on host smithi125... 2021-03-24T21:48:49.551 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:49 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:48:49.551 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:49 smithi125 conmon[24927]: 21:48:48.919467+0000 mon.a (mon.0 2021-03-24T21:48:49.551 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:49 smithi125 conmon[24927]: ) 187 : cluster [DBG] Standby manager daemon x started 2021-03-24T21:48:49.552 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:49 smithi125 conmon[24927]: audit 2021-03 2021-03-24T21:48:49.552 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:49 smithi125 conmon[24927]: -24T21:48:48.920697+0000 mon.b (mon 2021-03-24T21:48:49.552 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:49 smithi125 conmon[24927]: .1) 3 : audit [DBG] from='mgr.? v1:172.21.15.142:0/1104290420' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2021-03-24T21:48:49.552 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:49 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T21:48:49.552 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:49 smithi125 conmon[24927]: 24T21:48:48.921392+0000 mon.b (mon 2021-03-24T21:48:49.553 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:49 smithi125 conmon[24927]: .1) 4 : audit [DBG] from='mgr.? v1:172.21.15.142:0/1104290420' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2021-03-24T21:48:49.553 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:49 smithi125 conmon[24927]: audit 2021 2021-03-24T21:48:49.553 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:49 smithi125 conmon[24927]: -03-24T21:48:48.923244+0000 2021-03-24T21:48:49.553 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:49 smithi125 conmon[24927]: mon.b (mon.1) 5 : audit [DBG] from='mgr.? v1:172.21.15.142:0/1104290420' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2021-03-24T21:48:49.553 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:49 smithi125 conmon[24927]: audit 2021 2021-03-24T21:48:49.554 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:49 smithi125 conmon[24927]: -03-24T21:48:48.923938+0000 mon.b 2021-03-24T21:48:49.554 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:49 smithi125 conmon[24927]: (mon.1) 6 : audit [DBG] from='mgr.? v1:172.21.15.142:0/1104290420' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2021-03-24T21:48:49.863 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:49 smithi142 conmon[20125]: audit 2021-03-24T21:48:47.822090+0000 mgr.y (mgr.14144) 65 : audit 2021-03-24T21:48:49.864 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:49 smithi142 conmon[20125]: [DBG] from='client.34115 v1:172.21.15.125:0/1619282178' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi125:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:48:49.864 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:49 smithi142 conmon[20125]: cephadm 2021-03-24T21:48:47.826055+0000 mgr.y (mgr.14144) 66 : cephadm [INF] Applying service osd.None on host smithi125... 2021-03-24T21:48:49.864 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:49 smithi142 conmon[20125]: 2021-03-24T21:48:49.864 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:49 smithi142 conmon[20125]: cluster 2021-03-24T21:48:48.919467+0000 mon.a (mon.0) 187 : 2021-03-24T21:48:49.865 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:49 smithi142 conmon[20125]: cluster [DBG] Standby manager daemon x started 2021-03-24T21:48:49.865 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:49 smithi142 conmon[20125]: audit 2021-03-24T21 2021-03-24T21:48:49.865 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:49 smithi142 conmon[20125]: :48:48.920697+0000 mon.b (mon.1) 3 : audit [DBG] from='mgr.? v1:172.21.15.142:0/1104290420' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2021-03-24T21:48:49.865 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:49 smithi142 conmon[20125]: audit 2021-03-24T21:48:49.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:49 smithi142 conmon[20125]: 2021-03-24T21:48:48.921392+0000 mon.b ( 2021-03-24T21:48:49.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:49 smithi142 conmon[20125]: mon.1) 4 : audit [DBG] from='mgr.? v1:172.21.15.142:0/1104290420' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2021-03-24T21:48:49.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:49 smithi142 conmon[20125]: audit 2021-03-24T21:48: 2021-03-24T21:48:49.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:49 smithi142 conmon[20125]: 48.923244+0000 mon.b (mon.1) 5 : audit [DBG] 2021-03-24T21:48:49.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:49 smithi142 conmon[20125]: from='mgr.? v1:172.21.15.142:0/1104290420' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2021-03-24T21:48:49.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:49 smithi142 conmon[20125]: audit 2021-03 2021-03-24T21:48:49.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:49 smithi142 conmon[20125]: -24T21:48:48.923938+0000 mon.b (mon.1) 6 : audit [DBG] 2021-03-24T21:48:49.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:49 smithi142 conmon[20125]: from='mgr.? v1:172.21.15.142:0/1104290420' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2021-03-24T21:48:50.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:50 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:48:50.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:50 smithi125 conmon[19936]: 21:48:48.588703+0000 mgr.y (mgr.14144) 67 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:50.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:50 smithi125 conmon[19936]: cluster 2021-03-24T21:48:50.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:50 smithi125 conmon[19936]: 2021-03-24T21:48:49.404576+0000 mon.a (mon.0) 2021-03-24T21:48:50.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:50 smithi125 conmon[19936]: 188 : cluster [DBG] mgrmap e14: y(active, since 104s), standbys: x 2021-03-24T21:48:50.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:50 smithi125 conmon[19936]: audit 2021 2021-03-24T21:48:50.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:50 smithi125 conmon[19936]: -03-24T21:48:49.405905 2021-03-24T21:48:50.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:50 smithi125 conmon[19936]: +0000 mon.a (mon.0) 189 : audit 2021-03-24T21:48:50.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:50 smithi125 conmon[19936]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2021-03-24T21:48:50.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:50 smithi125 conmon[24927]: cluster 2021-03-24T21:48 2021-03-24T21:48:50.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:50 smithi125 conmon[24927]: :48.588703+0000 mgr.y (mgr.14144) 67 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:50.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:50 smithi125 conmon[24927]: cluster 2021-03-24T21:48:49.404576+0000 mon.a (mon 2021-03-24T21:48:50.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:50 smithi125 conmon[24927]: .0) 188 : cluster [DBG] mgrmap e14: y(active, since 104s), standbys: x 2021-03-24T21:48:50.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:50 smithi125 conmon[24927]: audit 2021-03-24T21:48:50.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:50 smithi125 conmon[24927]: 2021-03-24T21:48 2021-03-24T21:48:50.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:50 smithi125 conmon[24927]: :49.405905+0000 mon.a (mon.0) 189 : audit 2021-03-24T21:48:50.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:50 smithi125 conmon[24927]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2021-03-24T21:48:50.863 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:50 smithi142 conmon[20125]: cluster 2021-03-24T21:48:48. 2021-03-24T21:48:50.863 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:50 smithi142 conmon[20125]: 588703+0000 mgr.y (mgr.14144) 67 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:50.864 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:50 smithi142 conmon[20125]: cluster 2021-03-24T21:48:49.404576+0000 mon.a ( 2021-03-24T21:48:50.864 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:50 smithi142 conmon[20125]: mon.0) 188 : cluster [DBG] mgrmap e14: y(active, since 104s), standbys: x 2021-03-24T21:48:50.864 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:50 smithi142 conmon[20125]: audit 2021-03-24T21:48: 2021-03-24T21:48:50.864 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:50 smithi142 conmon[20125]: 49.405905+0000 mon.a (mon.0) 189 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2021-03-24T21:48:51.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:51 smithi125 conmon[19936]: audit 2021-03-24T21:48:50.418870+0000 2021-03-24T21:48:51.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:51 smithi125 conmon[19936]: mon.a (mon.0) 190 : audit [INF] from='client.? v1:172.21.15.125:0/1985683263' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f8f9a69-4b85-4054-8b11-36d35574494c"}]: dispatch 2021-03-24T21:48:51.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:51 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:48:51.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:51 smithi125 conmon[19936]: 03-24T21:48:50.425614+0000 mon.a (mon.0) 191 : audit [INF] from='client.? v1:172.21.15.125:0/1985683263' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1f8f9a69-4b85-4054-8b11-36d35574494c"}]': finished 2021-03-24T21:48:51.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:51 smithi125 conmon[19936]: cluster 2021-03-24T21:48:51.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:51 smithi125 conmon[19936]: 2021-03-24T21:48 2021-03-24T21:48:51.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:51 smithi125 conmon[19936]: :50.425666+0000 mon.a (mon. 2021-03-24T21:48:51.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:51 smithi125 conmon[19936]: 0) 192 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2021-03-24T21:48:51.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:51 smithi125 conmon[19936]: audit 2021 2021-03-24T21:48:51.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:51 smithi125 conmon[19936]: -03-24T21:48: 2021-03-24T21:48:51.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:51 smithi125 conmon[19936]: 50.425761+0000 mon.a (mon.0) 193 2021-03-24T21:48:51.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:51 smithi125 conmon[19936]: : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-03-24T21:48:51.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:51 smithi125 conmon[19936]: audit 2021-03 2021-03-24T21:48:51.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:51 smithi125 conmon[19936]: -24T21:48:51.053665 2021-03-24T21:48:51.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:51 smithi125 conmon[19936]: +0000 mon.a (mon.0) 194 : audit 2021-03-24T21:48:51.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:51 smithi125 conmon[19936]: [DBG] from='client.? v1:172.21.15.125:0/4144171253' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-24T21:48:51.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:51 smithi125 conmon[24927]: audit 2021-03-24T 2021-03-24T21:48:51.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:51 smithi125 conmon[24927]: 21:48:50.418870+0000 mon.a (mon.0) 190 : audit [INF] from='client.? v1:172.21.15.125:0/1985683263' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f8f9a69-4b85-4054-8b11-36d35574494c"}]: dispatch 2021-03-24T21:48:51.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:51 smithi125 conmon[24927]: audit 2021-03-24T21:48:51.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:51 smithi125 conmon[24927]: 2021-03-24T21:48:50 2021-03-24T21:48:51.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:51 smithi125 conmon[24927]: .425614+0000 mon.a (mon.0) 191 : audit 2021-03-24T21:48:51.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:51 smithi125 conmon[24927]: [INF] from='client.? v1:172.21.15.125:0/1985683263' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1f8f9a69-4b85-4054-8b11-36d35574494c"}]': finished 2021-03-24T21:48:51.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:51 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:48:51.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:51 smithi125 conmon[24927]: 24T21:48:50.425666+0000 2021-03-24T21:48:51.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:51 smithi125 conmon[24927]: mon.a (mon.0) 192 : cluster [DBG] 2021-03-24T21:48:51.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:51 smithi125 conmon[24927]: osdmap e5: 1 total, 0 up, 1 in 2021-03-24T21:48:51.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:51 smithi125 conmon[24927]: audit 2021-03-24 2021-03-24T21:48:51.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:51 smithi125 conmon[24927]: T21:48:50.425761+0000 2021-03-24T21:48:51.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:51 smithi125 conmon[24927]: mon.a (mon.0) 193 : audit 2021-03-24T21:48:51.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:51 smithi125 conmon[24927]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-03-24T21:48:51.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:51 smithi125 conmon[24927]: audit 2021-03-24T21:48:51.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:51 smithi125 conmon[24927]: 2021-03-24T21: 2021-03-24T21:48:51.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:51 smithi125 conmon[24927]: 48:51.053665+0000 mon.a (mon.0 2021-03-24T21:48:51.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:51 smithi125 conmon[24927]: ) 194 : audit [DBG] from='client.? v1:172.21.15.125:0/4144171253' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-24T21:48:51.863 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:51 smithi142 conmon[20125]: audit 2021-03-24T21:48: 2021-03-24T21:48:51.863 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:51 smithi142 conmon[20125]: 50.418870+0000 mon.a (mon.0) 190 : audit [INF] from='client.? v1:172.21.15.125:0/1985683263' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f8f9a69-4b85-4054-8b11-36d35574494c"}]: dispatch 2021-03-24T21:48:51.864 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:51 smithi142 conmon[20125]: audit 2021-03-24T21:48:50 2021-03-24T21:48:51.864 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:51 smithi142 conmon[20125]: .425614+0000 mon.a (mon.0) 191 : audit [INF] from='client.? v1:172.21.15.125:0/1985683263' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1f8f9a69-4b85-4054-8b11-36d35574494c"}]': finished 2021-03-24T21:48:51.864 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:51 smithi142 conmon[20125]: cluster 2021-03-24T21 2021-03-24T21:48:51.864 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:51 smithi142 conmon[20125]: :48:50.425666+0000 mon.a (mon.0) 192 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2021-03-24T21:48:51.865 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:51 smithi142 conmon[20125]: audit 2021-03-24T21:48:50.425761+0000 2021-03-24T21:48:51.865 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:51 smithi142 conmon[20125]: mon.a (mon.0) 193 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-03-24T21:48:51.865 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:51 smithi142 conmon[20125]: audit 2021-03-24T21:48:51.053665+0000 mon.a (mon 2021-03-24T21:48:51.865 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:51 smithi142 conmon[20125]: .0) 194 : audit [DBG] from='client.? v1:172.21.15.125:0/4144171253' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-24T21:48:52.863 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:52 smithi142 conmon[20125]: cluster 2021-03-24T21:48: 2021-03-24T21:48:53.046 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:52 smithi142 conmon[20125]: 50.589106+0000 mgr.y (mgr.14144) 68 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:53.049 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:52 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:48:53.049 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:52 smithi125 conmon[19936]: 21:48:50.589106+0000 mgr.y (mgr.14144) 68 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:53.050 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:52 smithi125 conmon[24927]: cluster 2021-03-24T21:48:50 2021-03-24T21:48:53.051 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:52 smithi125 conmon[24927]: .589106+0000 mgr.y (mgr.14144) 68 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:54.863 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:54 smithi142 conmon[20125]: cluster 2021-03-24T21:48: 2021-03-24T21:48:54.863 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:54 smithi142 conmon[20125]: 52.589535+0000 mgr.y (mgr.14144) 69 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:54.902 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:54 smithi125 conmon[19936]: cluster 2021-03-24T21:48 2021-03-24T21:48:54.903 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:54 smithi125 conmon[19936]: :52.589535+0000 mgr.y (mgr.14144) 69 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:54.903 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:54 smithi125 conmon[24927]: cluster 2021-03-24T21:48:52.589535+0000 mgr.y (mgr.14144) 69 : 2021-03-24T21:48:54.903 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:54 smithi125 conmon[24927]: cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:56.681 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:56 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T21:48:56.682 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:56 smithi125 conmon[19936]: T21:48:54.589942+0000 mgr.y (mgr.14144) 70 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:56.682 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:56 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:48:56.683 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:56 smithi125 conmon[24927]: T21:48:54.589942+0000 mgr.y (mgr.14144) 70 : 2021-03-24T21:48:56.683 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:56 smithi125 conmon[24927]: cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:56.863 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:56 smithi142 conmon[20125]: cluster 2021-03-24T21:48:54.589942+0000 mgr.y (mgr.14144) 70 2021-03-24T21:48:56.863 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:56 smithi142 conmon[20125]: : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:57.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:57 smithi125 conmon[24927]: audit 2021-03-24T21:48:57.402062+0000 2021-03-24T21:48:57.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:57 smithi125 conmon[24927]: mon.a (mon.0) 195 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2021-03-24T21:48:57.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:57 smithi125 conmon[24927]: audit 2021-03-24 2021-03-24T21:48:57.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:57 smithi125 conmon[24927]: T21:48:57.403398+0000 mon.a (mon.0) 196 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:48:57.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:57 smithi125 conmon[19936]: audit 2021-03-24T21 2021-03-24T21:48:57.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:57 smithi125 conmon[19936]: :48:57.402062+0000 mon.a (mon.0) 195 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2021-03-24T21:48:57.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:57 smithi125 conmon[19936]: audit 2021-03-24T21:48:57.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:57 smithi125 conmon[19936]: 2021-03-24T21:48:57 2021-03-24T21:48:57.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:57 smithi125 conmon[19936]: .403398+0000 mon.a (mon 2021-03-24T21:48:57.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:57 smithi125 conmon[19936]: .0) 196 : audit 2021-03-24T21:48:57.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:57 smithi125 conmon[19936]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:48:57.863 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:57 smithi142 conmon[20125]: audit 2021-03-24T21:48:57.402062+0000 mon.a (mon.0) 195 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2021-03-24T21:48:57.863 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:57 smithi142 conmon[20125]: 2021-03-24T21:48:57.863 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:57 smithi142 conmon[20125]: audit 2021-03-24T21:48:57.403398+0000 mon.a (mon.0) 2021-03-24T21:48:57.864 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:57 smithi142 conmon[20125]: 196 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:48:58.652 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:58 smithi125 conmon[19936]: cluster 2021-03-24T21:48:56.590377+0000 2021-03-24T21:48:58.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:58 smithi125 conmon[19936]: mgr.y (mgr.14144) 71 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:58.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:58 smithi125 conmon[19936]: cephadm 2021-03-24T21:48: 2021-03-24T21:48:58.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:58 smithi125 conmon[19936]: 57.404541+0000 mgr.y (mgr.14144) 72 : cephadm [INF] Deploying daemon osd.0 on smithi125 2021-03-24T21:48:58.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:58 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:48:58.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:58 smithi125 conmon[24927]: 21:48:56.590377+0000 mgr.y (mgr.14144) 71 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:58.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:58 smithi125 conmon[24927]: cephadm 2021- 2021-03-24T21:48:58.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:58 smithi125 conmon[24927]: 03-24T21:48:57.404541+0000 mgr.y 2021-03-24T21:48:58.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:58 smithi125 conmon[24927]: (mgr.14144) 72 : cephadm [INF] Deploying daemon osd.0 on smithi125 2021-03-24T21:48:58.863 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:58 smithi142 conmon[20125]: cluster 2021-03-24T21:48:56.590377+0000 mgr.y (mgr.14144) 2021-03-24T21:48:58.863 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:58 smithi142 conmon[20125]: 71 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:58.863 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:58 smithi142 conmon[20125]: cephadm 2021-03-24T21:48:57.404541+0000 2021-03-24T21:48:58.864 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:58 smithi142 conmon[20125]: mgr.y (mgr.14144) 72 : cephadm [INF] Deploying daemon osd.0 on smithi125 2021-03-24T21:48:59.790 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:59 smithi125 conmon[19936]: cluster 2021-03-24T21:48:58.590841+0000 mgr.y 2021-03-24T21:48:59.790 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:48:59 smithi125 conmon[19936]: (mgr.14144) 73 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:59.790 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:59 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:48:59.791 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:48:59 smithi125 conmon[24927]: 48:58.590841+0000 mgr.y (mgr.14144) 73 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:48:59.863 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:59 smithi142 conmon[20125]: cluster 2021-03-24T21:48: 2021-03-24T21:48:59.863 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:48:59 smithi142 conmon[20125]: 58.590841+0000 mgr.y (mgr.14144) 73 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:49:01.688 INFO:teuthology.orchestra.run.smithi125.stdout:Created osd(s) 0 on host 'smithi125' 2021-03-24T21:49:02.152 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:01 smithi125 conmon[24927]: cluster 2021-03-24T21:49:00.591276+0000 mgr.y (mgr.14144) 74 2021-03-24T21:49:02.153 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:01 smithi125 conmon[24927]: : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:49:02.153 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:01 smithi125 conmon[24927]: audit 2021-03-24T21:49:01.681150+0000 mon.a (mon.0) 197 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:49:02.153 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:01 smithi125 conmon[24927]: audit 2021-03-24T21:49:01.686941+0000 mon.a (mon.0) 198 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:02.153 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:01 smithi125 conmon[24927]: audit 2021-03-24T21:49:01.694651+0000 mon.a 2021-03-24T21:49:02.154 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:01 smithi125 conmon[24927]: (mon.0) 199 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:02.154 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:01 smithi125 conmon[19936]: cluster 2021-03-24T21:49:00.591276+0000 2021-03-24T21:49:02.154 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:01 smithi125 conmon[19936]: mgr.y (mgr.14144) 74 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:49:02.154 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:01 smithi125 conmon[19936]: audit 2021-03-24T21:49:01.681150+0000 mon.a (mon.0 2021-03-24T21:49:02.155 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:01 smithi125 conmon[19936]: ) 197 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:49:02.155 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:01 smithi125 conmon[19936]: audit 2021-03 2021-03-24T21:49:02.155 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:01 smithi125 conmon[19936]: -24T21:49:01.686941+0000 mon.a (mon.0 2021-03-24T21:49:02.155 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:01 smithi125 conmon[19936]: ) 198 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:02.156 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:01 smithi125 conmon[19936]: audit 2021-03-24T21:49:02.156 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:01 smithi125 conmon[19936]: 2021-03-24T21: 2021-03-24T21:49:02.156 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:01 smithi125 conmon[19936]: 49:01.694651+0000 mon.a (mon.0) 199 : audit 2021-03-24T21:49:02.156 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:01 smithi125 conmon[19936]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:02.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:01 smithi142 conmon[20125]: cluster 2021-03-24T21:49:00.591276+0000 mgr.y (mgr.14144) 74 : 2021-03-24T21:49:02.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:01 smithi142 conmon[20125]: cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:49:02.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:01 smithi142 conmon[20125]: audit 2021-03-24T21:49:01.681150+0000 mon.a (mon.0) 197 : 2021-03-24T21:49:02.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:01 smithi142 conmon[20125]: audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:49:02.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:01 smithi142 conmon[20125]: audit 2021-03-24T21:49:02.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:01 smithi142 conmon[20125]: 2021-03-24T21:49:01.686941+0000 mon.a ( 2021-03-24T21:49:02.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:01 smithi142 conmon[20125]: mon.0) 198 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:02.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:01 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:49:02.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:01 smithi142 conmon[20125]: 03-24T21:49:01.694651+0000 2021-03-24T21:49:02.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:01 smithi142 conmon[20125]: mon.a (mon.0) 199 : audit [INF] 2021-03-24T21:49:02.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:01 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:02.511 DEBUG:teuthology.orchestra.run.smithi125:osd.0> sudo journalctl -f -n 0 -u ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c@osd.0.service 2021-03-24T21:49:02.514 INFO:tasks.cephadm:Deploying osd.1 on smithi125 with /dev/vg_nvme/lv_3... 2021-03-24T21:49:02.514 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2021-03-24T21:49:02.541 INFO:journalctl@ceph.osd.0.smithi125.stdout:-- Logs begin at Wed 2021-03-03 23:51:07 UTC. -- 2021-03-24T21:49:04.302 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:03 smithi125 conmon[24927]: cluster 2021-03-24T21:49:02.591642+0000 mgr.y (mgr.14144) 75 : 2021-03-24T21:49:04.302 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:03 smithi125 conmon[24927]: cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:49:04.303 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:03 smithi125 conmon[19936]: cluster 2021-03-24T21:49 2021-03-24T21:49:04.303 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:03 smithi125 conmon[19936]: :02.591642+0000 mgr.y (mgr.14144) 75 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:49:04.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:03 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:49:04.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:03 smithi142 conmon[20125]: 49:02.591642+0000 mgr.y (mgr.14144) 75 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:49:04.940 INFO:teuthology.orchestra.run.smithi125.stderr:--> Zapping: /dev/vg_nvme/lv_3 2021-03-24T21:49:04.940 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2021-03-24T21:49:04.940 INFO:teuthology.orchestra.run.smithi125.stderr: stderr: 10+0 records in 2021-03-24T21:49:04.941 INFO:teuthology.orchestra.run.smithi125.stderr:10+0 records out 2021-03-24T21:49:04.941 INFO:teuthology.orchestra.run.smithi125.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0213816 s, 490 MB/s 2021-03-24T21:49:04.941 INFO:teuthology.orchestra.run.smithi125.stderr:--> Zapping successful for: 2021-03-24T21:49:05.402 INFO:journalctl@ceph.osd.0.smithi125.stdout:Mar 24 21:49:05 smithi125 conmon[28026]: debug 2021-03-24T21:49:05.280+0000 7f61e401e080 -1 osd.0 0 log_to_monitors {default=true} 2021-03-24T21:49:05.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:05 smithi125 conmon[19936]: audit 2021-03-24T 2021-03-24T21:49:05.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:05 smithi125 conmon[19936]: 21:49:04.302092+0000 mon.a (mon.0) 200 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:05.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:05 smithi125 conmon[19936]: audit 2021-03-24T21:49:05.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:05 smithi125 conmon[19936]: 2021-03- 2021-03-24T21:49:05.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:05 smithi125 conmon[19936]: 24T21:49:04.728362 2021-03-24T21:49:05.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:05 smithi125 conmon[19936]: +0000 mon.a (mon.0) 201 2021-03-24T21:49:05.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:05 smithi125 conmon[19936]: : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T21:49:05.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:05 smithi125 conmon[19936]: audit 2021 2021-03-24T21:49:05.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:05 smithi125 conmon[19936]: -03-24T21:49:04 2021-03-24T21:49:05.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:05 smithi125 conmon[19936]: .732109+0000 mon.a (mon.0 2021-03-24T21:49:05.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:05 smithi125 conmon[19936]: ) 202 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T21:49:05.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:05 smithi125 conmon[19936]: 2021-03-24T21:49:05.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:05 smithi125 conmon[19936]: audit 2021-03-24T21:49: 2021-03-24T21:49:05.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:05 smithi125 conmon[19936]: 05.292950+0000 mon.a (mon.0) 203 : 2021-03-24T21:49:05.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:05 smithi125 conmon[19936]: audit [INF] from='osd.0 v1:172.21.15.125:6801/1197472172' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2021-03-24T21:49:05.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:05 smithi125 conmon[24927]: audit 2021-03-24T21:49:04.302092 2021-03-24T21:49:05.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:05 smithi125 conmon[24927]: +0000 mon.a (mon.0) 200 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:05.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:05 smithi125 conmon[24927]: audit 2021- 2021-03-24T21:49:05.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:05 smithi125 conmon[24927]: 03-24T21:49:04.728362+0000 mon.a (mon.0) 201 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T21:49:05.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:05 smithi125 conmon[24927]: audit 2021-03-24T21 2021-03-24T21:49:05.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:05 smithi125 conmon[24927]: :49:04.732109+0000 mon.a (mon.0) 202 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T21:49:05.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:05 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T21:49:05.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:05 smithi125 conmon[24927]: 24T21:49:05.292950+0000 mon.a (mon.0 2021-03-24T21:49:05.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:05 smithi125 conmon[24927]: ) 203 : audit [INF] from='osd.0 v1:172.21.15.125:6801/1197472172' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2021-03-24T21:49:05.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:05 smithi142 conmon[20125]: audit 2021-03-24T21:49 2021-03-24T21:49:05.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:05 smithi142 conmon[20125]: :04.302092+0000 mon.a (mon.0) 200 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:05.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:05 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:49:05.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:05 smithi142 conmon[20125]: 03-24T21:49:04.728362+0000 mon.a (mon.0) 201 2021-03-24T21:49:05.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:05 smithi142 conmon[20125]: : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T21:49:05.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:05 smithi142 conmon[20125]: audit 2021-03-24T21:49 2021-03-24T21:49:05.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:05 smithi142 conmon[20125]: :04.732109+0000 mon.a (mon.0) 2021-03-24T21:49:05.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:05 smithi142 conmon[20125]: 202 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T21:49:05.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:05 smithi142 conmon[20125]: audit 2021-03-24T21 2021-03-24T21:49:05.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:05 smithi142 conmon[20125]: :49:05.292950+0000 mon.a (mon. 2021-03-24T21:49:05.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:05 smithi142 conmon[20125]: 0) 203 : audit [INF] from='osd.0 v1:172.21.15.125:6801/1197472172' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2021-03-24T21:49:06.387 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph orch daemon add osd smithi125:vg_nvme/lv_3 2021-03-24T21:49:06.577 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:06 smithi125 conmon[24927]: cluster 2021-03-24T21:49:04 2021-03-24T21:49:06.577 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:06 smithi125 conmon[24927]: .591969+0000 mgr.y (mgr.14144) 76 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:49:06.577 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:06 smithi125 conmon[24927]: audit 2021-03-24T21:49:05 2021-03-24T21:49:06.578 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:06 smithi125 conmon[24927]: .968024+0000 mon.a (mon.0) 204 : audit [INF] 2021-03-24T21:49:06.578 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:06 smithi125 conmon[24927]: from='osd.0 v1:172.21.15.125:6801/1197472172' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2021-03-24T21:49:06.578 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:06 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:49:06.578 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:06 smithi125 conmon[24927]: 24T21:49:05.968192+0000 mon.a (mon.0) 205 : 2021-03-24T21:49:06.579 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:06 smithi125 conmon[24927]: cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2021-03-24T21:49:06.579 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:06 smithi125 conmon[24927]: audit 2021-03-24T21: 2021-03-24T21:49:06.579 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:06 smithi125 conmon[24927]: 49:05.968474+0000 mon.a (mon 2021-03-24T21:49:06.579 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:06 smithi125 conmon[24927]: .0) 206 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-03-24T21:49:06.580 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:06 smithi125 conmon[24927]: audit 2021-03-24T 2021-03-24T21:49:06.580 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:06 smithi125 conmon[24927]: 21:49:05.969463+0000 mon.a (mon.0) 207 : audit 2021-03-24T21:49:06.580 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:06 smithi125 conmon[24927]: [INF] from='osd.0 v1:172.21.15.125:6801/1197472172' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]: dispatch 2021-03-24T21:49:06.580 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:06 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:49:06.581 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:06 smithi125 conmon[19936]: :49:04.591969+0000 mgr.y (mgr.14144) 76 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:49:06.581 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:06 smithi125 conmon[19936]: audit 2021 2021-03-24T21:49:06.581 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:06 smithi125 conmon[19936]: -03-24T21:49:05.968024+0000 mon.a 2021-03-24T21:49:06.581 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:06 smithi125 conmon[19936]: (mon.0) 204 : audit [INF] 2021-03-24T21:49:06.582 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:06 smithi125 conmon[19936]: from='osd.0 v1:172.21.15.125:6801/1197472172' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2021-03-24T21:49:06.582 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:06 smithi125 conmon[19936]: cluster 2021-03-24T21:49:06.582 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:06 smithi125 conmon[19936]: 2021-03-24T21:49:05 2021-03-24T21:49:06.582 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:06 smithi125 conmon[19936]: .968192+0000 mon.a (mon.0) 205 2021-03-24T21:49:06.582 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:06 smithi125 conmon[19936]: : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2021-03-24T21:49:06.583 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:06 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:49:06.583 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:06 smithi125 conmon[19936]: 03-24T21:49:05.968474+0000 2021-03-24T21:49:06.583 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:06 smithi125 conmon[19936]: mon.a (mon.0) 206 : audit 2021-03-24T21:49:06.583 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:06 smithi125 conmon[19936]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-03-24T21:49:06.584 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:06 smithi125 conmon[19936]: audit 2021-03-24T21:49:06.584 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:06 smithi125 conmon[19936]: 2021-03-24T21:49:05. 2021-03-24T21:49:06.584 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:06 smithi125 conmon[19936]: 969463+0000 mon.a (mon.0) 207 : audit [INF] 2021-03-24T21:49:06.584 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:06 smithi125 conmon[19936]: from='osd.0 v1:172.21.15.125:6801/1197472172' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]: dispatch 2021-03-24T21:49:06.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:06 smithi142 conmon[20125]: cluster 2021-03-24T21:49:04.591969+0000 mgr.y (mgr.14144) 2021-03-24T21:49:06.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:06 smithi142 conmon[20125]: 76 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:49:06.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:06 smithi142 conmon[20125]: audit 2021-03-24T21:49:05.968024+0000 mon.a (mon.0) 2021-03-24T21:49:06.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:06 smithi142 conmon[20125]: 204 : audit [INF] from='osd.0 v1:172.21.15.125:6801/1197472172' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2021-03-24T21:49:06.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:06 smithi142 conmon[20125]: cluster 2021-03-24T21:49:05.968192+0000 mon.a ( 2021-03-24T21:49:06.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:06 smithi142 conmon[20125]: mon.0) 205 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2021-03-24T21:49:06.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:06 smithi142 conmon[20125]: audit 2021-03 2021-03-24T21:49:06.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:06 smithi142 conmon[20125]: -24T21:49:05.968474+0000 mon.a (mon.0) 206 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-03-24T21:49:06.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:06 smithi142 conmon[20125]: audit 2021-03-24T 2021-03-24T21:49:06.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:06 smithi142 conmon[20125]: 21:49:05.969463+0000 mon.a (mon.0) 207 : audit 2021-03-24T21:49:06.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:06 smithi142 conmon[20125]: [INF] from='osd.0 v1:172.21.15.125:6801/1197472172' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]: dispatch 2021-03-24T21:49:07.152 INFO:journalctl@ceph.osd.0.smithi125.stdout:Mar 24 21:49:06 smithi125 conmon[28026]: debug 2021-03-24T21:49:06.964+0000 7f61da58f700 -1 osd.0 0 waiting for initial osdmap 2021-03-24T21:49:08.304 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:07 smithi125 conmon[19936]: cluster 2021-03-24T21:49:08.305 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:07 smithi125 conmon[19936]: 2021-03-24T21:49:06.592341+0000 mgr.y (mgr.14144) 77 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:49:08.305 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:07 smithi125 conmon[19936]: audit 2021-03-24T21:49:08.305 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:07 smithi125 conmon[19936]: 2021-03-24T21:49:06.965416+0000 mon.a (mon.0) 208 : 2021-03-24T21:49:08.305 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:07 smithi125 conmon[19936]: audit [INF] from='osd.0 v1:172.21.15.125:6801/1197472172' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]': finished 2021-03-24T21:49:08.306 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:07 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T21:49:08.306 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:07 smithi125 conmon[19936]: T21:49:06.965517+0000 mon.a (mon.0) 209 2021-03-24T21:49:08.306 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:07 smithi125 conmon[19936]: : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2021-03-24T21:49:08.306 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:07 smithi125 conmon[19936]: audit 2021-03-24T21:49:06.965630 2021-03-24T21:49:08.307 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:07 smithi125 conmon[19936]: +0000 mon.a (mon.0) 210 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-03-24T21:49:08.307 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:07 smithi125 conmon[19936]: audit 2021-03-24T21:49:06. 2021-03-24T21:49:08.307 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:07 smithi125 conmon[19936]: 969174+0000 mon.a (mon.0) 211 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-03-24T21:49:08.308 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:07 smithi125 conmon[24927]: cluster 2021-03-24T21:49:06.592341+0000 2021-03-24T21:49:08.308 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:07 smithi125 conmon[24927]: mgr.y (mgr.14144) 77 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:49:08.308 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:07 smithi125 conmon[24927]: audit 2021-03-24T21:49:08.309 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:07 smithi125 conmon[24927]: 2021-03-24T21:49:06.965416+0000 2021-03-24T21:49:08.309 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:07 smithi125 conmon[24927]: mon.a (mon.0) 208 : audit [INF] from='osd.0 v1:172.21.15.125:6801/1197472172' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]': finished 2021-03-24T21:49:08.309 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:07 smithi125 conmon[24927]: cluster 2021-03-24T21:49:08.309 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:07 smithi125 conmon[24927]: 2021-03-24T21:49:06.965517+0000 mon.a 2021-03-24T21:49:08.310 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:07 smithi125 conmon[24927]: (mon.0) 209 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2021-03-24T21:49:08.310 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:07 smithi125 conmon[24927]: audit 2021-03-24T 2021-03-24T21:49:08.310 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:07 smithi125 conmon[24927]: 21:49:06.965630+0000 mon.a (mon.0) 210 2021-03-24T21:49:08.310 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:07 smithi125 conmon[24927]: : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-03-24T21:49:08.311 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:07 smithi125 conmon[24927]: audit 2021- 2021-03-24T21:49:08.311 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:07 smithi125 conmon[24927]: 03-24T21:49:06.969174+0000 mon.a ( 2021-03-24T21:49:08.311 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:07 smithi125 conmon[24927]: mon.0) 211 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-03-24T21:49:08.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:07 smithi142 conmon[20125]: cluster 2021-03-24T21:49:06.592341+0000 mgr.y (mgr.14144 2021-03-24T21:49:08.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:07 smithi142 conmon[20125]: ) 77 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:49:08.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:07 smithi142 conmon[20125]: audit 2021-03-24T21:49: 2021-03-24T21:49:08.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:07 smithi142 conmon[20125]: 06.965416+0000 mon.a (mon.0) 208 : audit [INF] from='osd.0 v1:172.21.15.125:6801/1197472172' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]': finished 2021-03-24T21:49:08.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:07 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:49:08.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:07 smithi142 conmon[20125]: -03-24T21:49: 2021-03-24T21:49:08.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:07 smithi142 conmon[20125]: 06.965517+0000 mon.a (mon.0) 209 2021-03-24T21:49:08.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:07 smithi142 conmon[20125]: : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2021-03-24T21:49:08.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:07 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:49:08.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:07 smithi142 conmon[20125]: 03-24T21:49:06 2021-03-24T21:49:08.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:07 smithi142 conmon[20125]: .965630+0000 mon.a (mon.0) 210 : 2021-03-24T21:49:08.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:07 smithi142 conmon[20125]: audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-03-24T21:49:08.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:07 smithi142 conmon[20125]: audit 2021-03- 2021-03-24T21:49:08.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:07 smithi142 conmon[20125]: 24T21:49:06.969174 2021-03-24T21:49:08.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:07 smithi142 conmon[20125]: +0000 mon.a (mon.0) 211 : audit 2021-03-24T21:49:08.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:07 smithi142 conmon[20125]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-03-24T21:49:09.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:08 smithi142 conmon[20125]: cluster 2021-03-24T21:49:06.318446+0000 osd.0 (osd.0) 1 2021-03-24T21:49:09.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:08 smithi142 conmon[20125]: : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:49:09.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:08 smithi142 conmon[20125]: cluster 2021-03-24T21 2021-03-24T21:49:09.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:08 smithi142 conmon[20125]: :49:06.318670+0000 osd.0 (osd.0 2021-03-24T21:49:09.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:08 smithi142 conmon[20125]: ) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:49:09.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:08 smithi142 conmon[20125]: cluster 2021-03-24T21:49:09.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:08 smithi142 conmon[20125]: 2021-03-24T21:49: 2021-03-24T21:49:09.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:08 smithi142 conmon[20125]: 07.966301+0000 mon.a (mon 2021-03-24T21:49:09.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:08 smithi142 conmon[20125]: .0) 212 : cluster [INF] 2021-03-24T21:49:09.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:08 smithi142 conmon[20125]: osd.0 v1:172.21.15.125:6801/1197472172 boot 2021-03-24T21:49:09.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:08 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:49:09.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:08 smithi142 conmon[20125]: -03-24T21:49: 2021-03-24T21:49:09.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:08 smithi142 conmon[20125]: 07.966342+0000 mon.a (mon. 2021-03-24T21:49:09.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:08 smithi142 conmon[20125]: 0) 213 : cluster [DBG] 2021-03-24T21:49:09.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:08 smithi142 conmon[20125]: osdmap e8: 1 total, 1 up, 1 in 2021-03-24T21:49:09.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:08 smithi142 conmon[20125]: audit 2021 2021-03-24T21:49:09.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:08 smithi142 conmon[20125]: -03-24T21:49 2021-03-24T21:49:09.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:08 smithi142 conmon[20125]: :07.968140+0000 mon.a ( 2021-03-24T21:49:09.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:08 smithi142 conmon[20125]: mon.0) 214 : audit [DBG] 2021-03-24T21:49:09.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:08 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-03-24T21:49:09.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:08 smithi125 conmon[19936]: cluster 2021-03-24T21:49:06. 2021-03-24T21:49:09.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:08 smithi125 conmon[19936]: 318446+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:49:09.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:08 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:49:09.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:08 smithi125 conmon[19936]: -03-24T21:49:06.318670+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:49:09.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:08 smithi125 conmon[19936]: cluster 2021-03-24T21:49:09.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:08 smithi125 conmon[19936]: 2021-03-24T21:49:07.966301+0000 mon.a (mon 2021-03-24T21:49:09.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:08 smithi125 conmon[19936]: .0) 212 : cluster [INF] osd.0 v1:172.21.15.125:6801/1197472172 boot 2021-03-24T21:49:09.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:08 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:49:09.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:08 smithi125 conmon[19936]: -24T21:49:07.966342+0000 mon.a (mon. 2021-03-24T21:49:09.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:08 smithi125 conmon[19936]: 0) 213 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2021-03-24T21:49:09.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:08 smithi125 conmon[19936]: audit 2021 2021-03-24T21:49:09.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:08 smithi125 conmon[19936]: -03-24T21:49:07.968140+0000 mon.a (mon. 2021-03-24T21:49:09.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:08 smithi125 conmon[19936]: 0) 214 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-03-24T21:49:09.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:08 smithi125 conmon[24927]: cluster 2021-03-24T21:49:06.318446+0000 osd.0 (osd 2021-03-24T21:49:09.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:08 smithi125 conmon[24927]: .0) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:49:09.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:08 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:49:09.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:08 smithi125 conmon[24927]: -24T21:49:06.318670+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:49:09.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:08 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:49:09.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:08 smithi125 conmon[24927]: 24T21:49:07.966301+0000 mon.a (mon.0 2021-03-24T21:49:09.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:08 smithi125 conmon[24927]: ) 212 : cluster [INF] osd.0 v1:172.21.15.125:6801/1197472172 boot 2021-03-24T21:49:09.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:08 smithi125 conmon[24927]: cluster 2021-03-24T21:49:09.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:08 smithi125 conmon[24927]: 2021-03-24T21:49:07.966342 2021-03-24T21:49:09.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:08 smithi125 conmon[24927]: +0000 mon.a (mon.0) 213 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2021-03-24T21:49:09.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:08 smithi125 conmon[24927]: audit 2021-03-24T21:49:09.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:08 smithi125 conmon[24927]: 2021-03-24T21:49:07.968140+0000 mon.a (mon.0) 214 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-03-24T21:49:09.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:08 smithi125 conmon[24927]: 2021-03-24T21:49:10.120 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[24927]: cluster 2021-03-24T21:49:08.592768+0000 mgr.y 2021-03-24T21:49:10.121 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[24927]: (mgr.14144) 78 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:49:10.121 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[24927]: cluster 2021-03-24T21:49:08.985299+0000 mon.a (mon. 2021-03-24T21:49:10.121 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[24927]: 0) 215 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2021-03-24T21:49:10.121 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[24927]: cephadm 2021-03-24T21:49:09.260870 2021-03-24T21:49:10.121 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[24927]: +0000 mgr.y (mgr.14144) 79 : cephadm [INF] Detected new or changed devices on smithi125 2021-03-24T21:49:10.122 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[24927]: audit 2021-03-24T21:49:10.122 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[24927]: 2021-03-24T21:49:09.268573+0000 mon.a (mon.0) 216 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:10.122 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[24927]: audit 2021-03-24T21:49:09.274208 2021-03-24T21:49:10.122 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[24927]: +0000 mon.a (mon.0) 217 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:10.123 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[24927]: audit 2021-03-24T21:49:09.275811+0000 mon.a (mon.0 2021-03-24T21:49:10.123 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[24927]: ) 218 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:49:10.123 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[24927]: audit 2021- 2021-03-24T21:49:10.123 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[24927]: 03-24T21:49:09.282058+0000 mon.a (mon.0) 219 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:10.123 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[24927]: audit 2021-03-24T21:49:10.124 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[24927]: 2021-03-24T21:49:09.394777+0000 mgr.y (mgr.14144) 80 : audit [DBG] from='client.34142 v1:172.21.15.125:0/325939900' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi125:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:49:10.124 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[24927]: 2021-03-24T21:49:10.124 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[24927]: audit 2021-03-24T21:49:09.395650+0000 mon.a 2021-03-24T21:49:10.124 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[24927]: (mon.0) 220 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-24T21:49:10.125 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[24927]: cephadm 2021-03 2021-03-24T21:49:10.125 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[24927]: -24T21:49:09.397700+0000 mgr.y (mgr.14144) 81 : cephadm [INF] Applying service osd.None on host smithi125... 2021-03-24T21:49:10.125 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[24927]: audit 2021 2021-03-24T21:49:10.125 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[24927]: -03-24T21:49:09.398126+0000 mon.a (mon.0) 221 : audit 2021-03-24T21:49:10.125 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[24927]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-24T21:49:10.126 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[24927]: audit 2021- 2021-03-24T21:49:10.126 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[24927]: 03-24T21:49:09.399370+0000 mon.a (mon.0) 222 : audit 2021-03-24T21:49:10.126 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[24927]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:49:10.126 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[19936]: cluster 2021-03-24T21:49:08.592768+0000 mgr.y (mgr. 2021-03-24T21:49:10.127 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[19936]: 14144) 78 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:49:10.127 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[19936]: cluster 2021-03-24T21:49:08.985299+0000 mon.a (mon.0) 215 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2021-03-24T21:49:10.127 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[19936]: cephadm 2021-03-24T21:49:10.127 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[19936]: 2021-03-24 2021-03-24T21:49:10.128 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[19936]: T21:49:09.260870+0000 2021-03-24T21:49:10.128 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[19936]: mgr.y (mgr.14144) 79 : cephadm 2021-03-24T21:49:10.128 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[19936]: [INF] Detected new or changed devices on smithi125 2021-03-24T21:49:10.128 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[19936]: audit 2021 2021-03-24T21:49:10.128 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[19936]: -03-24T21:49:09 2021-03-24T21:49:10.129 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[19936]: .268573+0000 mon.a (mon.0 2021-03-24T21:49:10.129 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[19936]: ) 216 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:10.129 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[19936]: 2021-03-24T21:49:10.129 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[19936]: audit 2021-03-24T21:49:10.129 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[19936]: 2021-03-24T21: 2021-03-24T21:49:10.130 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[19936]: 49:09.274208+0000 mon.a 2021-03-24T21:49:10.130 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[19936]: (mon.0) 217 : audit 2021-03-24T21:49:10.130 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[19936]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:10.130 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[19936]: audit 2021 2021-03-24T21:49:10.131 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[19936]: -03-24T21:49: 2021-03-24T21:49:10.131 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[19936]: 09.275811+0000 mon.a (mon 2021-03-24T21:49:10.131 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[19936]: .0) 218 : audit [DBG] 2021-03-24T21:49:10.131 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:49:10.131 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[19936]: audit 2021-03-24T21:49:10.132 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[19936]: 2021-03-24T21: 2021-03-24T21:49:10.132 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[19936]: 49:09.282058+0000 mon.a ( 2021-03-24T21:49:10.132 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[19936]: mon.0) 219 : audit [INF] 2021-03-24T21:49:10.132 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:10.132 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[19936]: audit 2021 2021-03-24T21:49:10.133 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[19936]: -03-24T21:49:09 2021-03-24T21:49:10.133 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[19936]: .394777+0000 mgr.y (mgr. 2021-03-24T21:49:10.133 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[19936]: 14144) 80 : audit [DBG] from='client.34142 v1:172.21.15.125:0/325939900' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi125:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:49:10.133 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[19936]: 2021-03-24T21:49:10.133 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[19936]: audit 2021-03-24T21:49:10.134 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[19936]: 2021-03- 2021-03-24T21:49:10.134 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[19936]: 24T21:49:09.395650 2021-03-24T21:49:10.134 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[19936]: +0000 mon.a (mon.0) 2021-03-24T21:49:10.134 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[19936]: 220 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-24T21:49:10.134 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[19936]: cephadm 2021 2021-03-24T21:49:10.135 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[19936]: -03-24T21:49:09 2021-03-24T21:49:10.135 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[19936]: .397700+0000 mgr.y (mgr. 2021-03-24T21:49:10.135 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[19936]: 14144) 81 : cephadm [INF] Applying service osd.None on host smithi125... 2021-03-24T21:49:10.135 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[19936]: 2021-03-24T21:49:10.136 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[19936]: audit 2021-03-24T21:49:10.136 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[19936]: 2021-03-24T21: 2021-03-24T21:49:10.136 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[19936]: 49:09.398126+0000 mon.a ( 2021-03-24T21:49:10.136 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[19936]: mon.0) 221 : audit [INF] 2021-03-24T21:49:10.136 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-24T21:49:10.137 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[19936]: audit 2021 2021-03-24T21:49:10.137 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[19936]: -03-24T21:49:09.399370+0000 2021-03-24T21:49:10.137 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[19936]: mon.a (mon.0) 222 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:49:10.137 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:09 smithi125 conmon[19936]: 2021-03-24T21:49:10.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:09 smithi142 conmon[20125]: cluster 2021-03-24T21:49: 2021-03-24T21:49:10.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:09 smithi142 conmon[20125]: 08.592768+0000 mgr.y (mgr.14144) 78 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-24T21:49:10.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:09 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:49:10.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:09 smithi142 conmon[20125]: 21:49:08.985299+0000 mon.a (mon.0) 215 : cluster 2021-03-24T21:49:10.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:09 smithi142 conmon[20125]: [DBG] osdmap e9: 1 total, 1 up, 1 in 2021-03-24T21:49:10.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:09 smithi142 conmon[20125]: cephadm 2021-03-24T21:49:10.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:09 smithi142 conmon[20125]: 2021-03-24T21:49:09.260870+0000 2021-03-24T21:49:10.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:09 smithi142 conmon[20125]: mgr.y (mgr.14144) 79 : cephadm [INF] 2021-03-24T21:49:10.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:09 smithi142 conmon[20125]: Detected new or changed devices on smithi125 2021-03-24T21:49:10.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:09 smithi142 conmon[20125]: audit 2021-03-24T21:49:10.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:09 smithi142 conmon[20125]: 2021-03-24T21:49:09.268573+0000 2021-03-24T21:49:10.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:09 smithi142 conmon[20125]: mon.a (mon.0) 216 : audit [INF] 2021-03-24T21:49:10.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:09 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:10.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:09 smithi142 conmon[20125]: audit 2021 2021-03-24T21:49:10.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:09 smithi142 conmon[20125]: -03-24T21:49:09.274208+0000 mon.a 2021-03-24T21:49:10.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:09 smithi142 conmon[20125]: (mon.0) 217 : audit [INF] 2021-03-24T21:49:10.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:09 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:10.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:09 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:49:10.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:09 smithi142 conmon[20125]: 03-24T21:49:09.275811+0000 mon.a 2021-03-24T21:49:10.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:09 smithi142 conmon[20125]: (mon.0) 218 : audit [DBG] 2021-03-24T21:49:10.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:09 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:49:10.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:09 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:49:10.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:09 smithi142 conmon[20125]: 03-24T21:49:09.282058+0000 mon.a ( 2021-03-24T21:49:10.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:09 smithi142 conmon[20125]: mon.0) 219 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:10.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:09 smithi142 conmon[20125]: audit 2021-03- 2021-03-24T21:49:10.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:09 smithi142 conmon[20125]: 24T21:49:09.394777+0000 2021-03-24T21:49:10.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:09 smithi142 conmon[20125]: mgr.y (mgr.14144) 80 : audit [DBG] 2021-03-24T21:49:10.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:09 smithi142 conmon[20125]: from='client.34142 v1:172.21.15.125:0/325939900' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi125:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:49:10.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:09 smithi142 conmon[20125]: audit 2021 2021-03-24T21:49:10.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:09 smithi142 conmon[20125]: -03-24T21:49:09.395650+0000 mon.a 2021-03-24T21:49:10.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:09 smithi142 conmon[20125]: (mon.0) 220 : audit [DBG] 2021-03-24T21:49:10.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:09 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-24T21:49:10.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:09 smithi142 conmon[20125]: cephadm 2021- 2021-03-24T21:49:10.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:09 smithi142 conmon[20125]: 03-24T21:49:09.397700+0000 mgr.y (mgr. 2021-03-24T21:49:10.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:09 smithi142 conmon[20125]: 14144) 81 : cephadm [INF] Applying service osd.None on host smithi125... 2021-03-24T21:49:10.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:09 smithi142 conmon[20125]: audit 2021-03- 2021-03-24T21:49:10.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:09 smithi142 conmon[20125]: 24T21:49:09.398126+0000 mon.a (mon. 2021-03-24T21:49:10.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:09 smithi142 conmon[20125]: 0) 221 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-24T21:49:10.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:09 smithi142 conmon[20125]: audit 2021-03-24T21:49:09.399370+0000 mon.a (mon 2021-03-24T21:49:10.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:09 smithi142 conmon[20125]: .0) 222 : audit 2021-03-24T21:49:10.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:09 smithi142 conmon[20125]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:49:12.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:11 smithi142 conmon[20125]: cluster 2021-03-24T21:49:10.593279+0000 mgr.y 2021-03-24T21:49:12.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:11 smithi142 conmon[20125]: (mgr.14144) 82 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-24T21:49:12.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:11 smithi142 conmon[20125]: audit 2021-03-24 2021-03-24T21:49:12.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:11 smithi142 conmon[20125]: T21:49:11.859271+0000 mon.c (mon.2) 2 : audit 2021-03-24T21:49:12.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:11 smithi142 conmon[20125]: [INF] from='client.? v1:172.21.15.125:0/3228142756' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48db0723-5806-4e5c-9919-f3d7f52ad169"}]: dispatch 2021-03-24T21:49:12.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:11 smithi142 conmon[20125]: audit 2021-03-24T21:49: 2021-03-24T21:49:12.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:11 smithi142 conmon[20125]: 11.859995+0000 mon.a (mon.0) 223 2021-03-24T21:49:12.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:11 smithi142 conmon[20125]: : audit [INF] from='client.34148 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48db0723-5806-4e5c-9919-f3d7f52ad169"}]: dispatch 2021-03-24T21:49:12.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:11 smithi142 conmon[20125]: audit 2021-03-24T21: 2021-03-24T21:49:12.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:11 smithi142 conmon[20125]: 49:11.867120+0000 mon.a (mon.0) 2021-03-24T21:49:12.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:11 smithi142 conmon[20125]: 224 : audit [INF] from='client.34148 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "48db0723-5806-4e5c-9919-f3d7f52ad169"}]': finished 2021-03-24T21:49:12.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:11 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:49:12.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:11 smithi142 conmon[20125]: 21:49:11.867253+0000 mon.a (mon 2021-03-24T21:49:12.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:11 smithi142 conmon[20125]: .0) 225 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2021-03-24T21:49:12.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:11 smithi142 conmon[20125]: audit 2021-03 2021-03-24T21:49:12.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:11 smithi142 conmon[20125]: -24T21:49:11.867417+0000 2021-03-24T21:49:12.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:11 smithi142 conmon[20125]: mon.a (mon.0) 226 : audit [DBG] 2021-03-24T21:49:12.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:11 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-03-24T21:49:12.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:11 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:49:12.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:11 smithi125 conmon[19936]: 21:49:10.593279+0000 mgr.y (mgr.14144) 82 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-24T21:49:12.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:11 smithi125 conmon[19936]: audit 2021-03-24T21:49:12.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:11 smithi125 conmon[19936]: 2021-03-24 2021-03-24T21:49:12.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:11 smithi125 conmon[19936]: T21:49:11.859271+0000 mon.c (mon.2 2021-03-24T21:49:12.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:11 smithi125 conmon[19936]: ) 2 : audit [INF] from='client.? v1:172.21.15.125:0/3228142756' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48db0723-5806-4e5c-9919-f3d7f52ad169"}]: dispatch 2021-03-24T21:49:12.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:11 smithi125 conmon[19936]: audit 2021 2021-03-24T21:49:12.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:11 smithi125 conmon[19936]: -03-24T21:49:11. 2021-03-24T21:49:12.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:11 smithi125 conmon[19936]: 859995+0000 mon.a (mon.0) 2021-03-24T21:49:12.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:11 smithi125 conmon[19936]: 223 : audit [INF] from='client.34148 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48db0723-5806-4e5c-9919-f3d7f52ad169"}]: dispatch 2021-03-24T21:49:12.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:11 smithi125 conmon[19936]: audit 2021-03 2021-03-24T21:49:12.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:11 smithi125 conmon[19936]: -24T21:49:11.867120+0000 mon.a (mon.0 2021-03-24T21:49:12.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:11 smithi125 conmon[19936]: ) 224 : audit [INF] from='client.34148 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "48db0723-5806-4e5c-9919-f3d7f52ad169"}]': finished 2021-03-24T21:49:12.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:11 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T21:49:12.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:11 smithi125 conmon[19936]: T21:49:11.867253+0000 mon.a (mon 2021-03-24T21:49:12.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:11 smithi125 conmon[19936]: .0) 225 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2021-03-24T21:49:12.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:11 smithi125 conmon[19936]: audit 2021-03- 2021-03-24T21:49:12.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:11 smithi125 conmon[19936]: 24T21:49:11.867417+0000 2021-03-24T21:49:12.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:11 smithi125 conmon[19936]: mon.a (mon.0) 226 : audit 2021-03-24T21:49:12.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:11 smithi125 conmon[19936]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-03-24T21:49:12.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:11 smithi125 conmon[24927]: cluster 2021-03-24T21:49 2021-03-24T21:49:12.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:11 smithi125 conmon[24927]: :10.593279+0000 mgr.y (mgr.14144) 82 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-24T21:49:12.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:11 smithi125 conmon[24927]: audit 2021-03-24T21:49:11 2021-03-24T21:49:12.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:11 smithi125 conmon[24927]: .859271+0000 mon.c (mon.2) 2 : audit [INF] 2021-03-24T21:49:12.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:11 smithi125 conmon[24927]: from='client.? v1:172.21.15.125:0/3228142756' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48db0723-5806-4e5c-9919-f3d7f52ad169"}]: dispatch 2021-03-24T21:49:12.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:11 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T21:49:12.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:11 smithi125 conmon[24927]: 24T21:49:11.859995+0000 mon.a (mon. 2021-03-24T21:49:12.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:11 smithi125 conmon[24927]: 0) 223 : audit [INF] from='client.34148 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48db0723-5806-4e5c-9919-f3d7f52ad169"}]: dispatch 2021-03-24T21:49:12.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:11 smithi125 conmon[24927]: audit 2021-03-24 2021-03-24T21:49:12.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:11 smithi125 conmon[24927]: T21:49:11.867120+0000 mon.a (mon. 2021-03-24T21:49:12.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:11 smithi125 conmon[24927]: 0) 224 : audit [INF] from='client.34148 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "48db0723-5806-4e5c-9919-f3d7f52ad169"}]': finished 2021-03-24T21:49:12.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:11 smithi125 conmon[24927]: cluster 2021-03-24T21:49:12.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:11 smithi125 conmon[24927]: 2021-03 2021-03-24T21:49:12.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:11 smithi125 conmon[24927]: -24T21:49:11.867253+0000 mon.a ( 2021-03-24T21:49:12.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:11 smithi125 conmon[24927]: mon.0) 225 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2021-03-24T21:49:12.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:11 smithi125 conmon[24927]: audit 2021 2021-03-24T21:49:12.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:11 smithi125 conmon[24927]: -03-24T21:49:11.867417+0000 mon.a ( 2021-03-24T21:49:12.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:11 smithi125 conmon[24927]: mon.0) 226 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-03-24T21:49:13.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:12 smithi142 conmon[20125]: audit 2021-03-24T21:49:12.498804+0000 mon.a (mon.0) 227 2021-03-24T21:49:13.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:12 smithi142 conmon[20125]: : audit [DBG] from='client.? v1:172.21.15.125:0/4013702344' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-24T21:49:13.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:12 smithi125 conmon[19936]: audit 2021-03-24T21:49:12.498804+0000 mon.a (mon.0) 227 : audit [DBG] from='client.? v1:172.21.15.125:0/4013702344' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-24T21:49:13.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:12 smithi125 conmon[24927]: audit 2021-03-24T21:49 2021-03-24T21:49:13.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:12 smithi125 conmon[24927]: :12.498804+0000 mon.a (mon.0) 227 : audit [DBG] from='client.? v1:172.21.15.125:0/4013702344' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-24T21:49:14.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:13 smithi142 conmon[20125]: cluster 2021-03-24T21:49:12.593677+0000 mgr.y (mgr.14144) 83 : 2021-03-24T21:49:14.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:13 smithi142 conmon[20125]: cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-24T21:49:14.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:14 smithi125 conmon[19936]: cluster 2021-03-24T21:49:12 2021-03-24T21:49:14.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:14 smithi125 conmon[19936]: .593677+0000 mgr.y (mgr.14144) 83 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-24T21:49:14.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:14 smithi125 conmon[24927]: cluster 2021-03-24T21:49: 2021-03-24T21:49:14.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:14 smithi125 conmon[24927]: 12.593677+0000 mgr.y (mgr.14144) 83 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-24T21:49:16.250 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:16 smithi125 conmon[19936]: cluster 2021-03-24T21:49:14 2021-03-24T21:49:16.251 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:16 smithi125 conmon[19936]: .594104+0000 mgr.y (mgr.14144) 84 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-24T21:49:16.251 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:16 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:49:16.252 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:16 smithi125 conmon[24927]: 24T21:49:14.594104+0000 mgr.y (mgr.14144) 2021-03-24T21:49:16.252 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:16 smithi125 conmon[24927]: 84 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-24T21:49:16.252 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:16 smithi125 conmon[24927]: 2021-03-24T21:49:16.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:16 smithi142 conmon[20125]: cluster 2021-03-24T21:49:14.594104+0000 mgr.y (mgr.14144) 84 : 2021-03-24T21:49:16.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:16 smithi142 conmon[20125]: cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-24T21:49:18.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:18 smithi142 conmon[20125]: cluster 2021-03-24T21:49:16.594567+0000 mgr.y (mgr.14144) 85 2021-03-24T21:49:18.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:18 smithi142 conmon[20125]: : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-24T21:49:18.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:18 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:49:18.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:18 smithi125 conmon[19936]: 49:16.594567+0000 mgr.y (mgr.14144) 85 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-24T21:49:18.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:18 smithi125 conmon[24927]: cluster 2021-03-24T21:49:16.594567+0000 2021-03-24T21:49:18.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:18 smithi125 conmon[24927]: mgr.y (mgr.14144) 85 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-24T21:49:19.152 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:19 smithi125 conmon[19936]: audit 2021-03-24T21:49:18.856857+0000 mon.a (mon. 2021-03-24T21:49:19.152 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:19 smithi125 conmon[19936]: 0) 228 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2021-03-24T21:49:19.153 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:19 smithi125 conmon[19936]: audit 2021-03-24T21:49:18.858118+0000 mon.a (mon.0) 229 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:49:19.153 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:19 smithi125 conmon[24927]: audit 2021-03-24 2021-03-24T21:49:19.153 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:19 smithi125 conmon[24927]: T21:49:18.856857+0000 mon.a (mon.0) 2021-03-24T21:49:19.153 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:19 smithi125 conmon[24927]: 228 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2021-03-24T21:49:19.154 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:19 smithi125 conmon[24927]: 2021-03-24T21:49:19.154 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:19 smithi125 conmon[24927]: audit 2021-03-24T21:49:19.154 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:19 smithi125 conmon[24927]: 2021-03-24T21 2021-03-24T21:49:19.154 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:19 smithi125 conmon[24927]: :49:18.858118 2021-03-24T21:49:19.155 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:19 smithi125 conmon[24927]: +0000 mon.a (mon. 2021-03-24T21:49:19.155 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:19 smithi125 conmon[24927]: 0) 229 : audit 2021-03-24T21:49:19.155 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:19 smithi125 conmon[24927]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:49:19.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:19 smithi142 conmon[20125]: audit 2021-03-24T21: 2021-03-24T21:49:19.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:19 smithi142 conmon[20125]: 49:18.856857+0000 mon.a (mon.0) 228 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2021-03-24T21:49:19.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:19 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:49:19.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:19 smithi142 conmon[20125]: 03-24T21:49:18.858118+0000 mon.a ( 2021-03-24T21:49:19.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:19 smithi142 conmon[20125]: mon.0) 229 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:49:20.152 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:20 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:49:20.153 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:20 smithi125 conmon[19936]: 21:49:18.595013+0000 mgr.y (mgr.14144) 86 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-24T21:49:20.153 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:20 smithi125 conmon[19936]: cephadm 2021-03-24T21:49:18.859292+0000 mgr.y (mgr.14144) 87 : 2021-03-24T21:49:20.153 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:20 smithi125 conmon[19936]: cephadm [INF] Deploying daemon osd.1 on smithi125 2021-03-24T21:49:20.154 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:20 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:49:20.154 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:20 smithi125 conmon[24927]: T21:49:18.595013+0000 mgr.y (mgr.14144) 86 : cluster 2021-03-24T21:49:20.154 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:20 smithi125 conmon[24927]: [DBG] pgmap v67: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-24T21:49:20.154 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:20 smithi125 conmon[24927]: cephadm 2021-03-24T21:49:20.155 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:20 smithi125 conmon[24927]: 2021-03-24T21:49: 2021-03-24T21:49:20.155 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:20 smithi125 conmon[24927]: 18.859292+0000 mgr.y (mgr. 2021-03-24T21:49:20.155 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:20 smithi125 conmon[24927]: 14144) 87 : cephadm [INF] Deploying daemon osd.1 on smithi125 2021-03-24T21:49:20.155 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:20 smithi125 conmon[24927]: 2021-03-24T21:49:20.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:20 smithi142 conmon[20125]: cluster 2021-03-24T21:49 2021-03-24T21:49:20.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:20 smithi142 conmon[20125]: :18.595013+0000 mgr.y (mgr.14144) 86 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-24T21:49:20.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:20 smithi142 conmon[20125]: cephadm 2021-03-24T21:49:20.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:20 smithi142 conmon[20125]: 2021-03-24T21:49: 2021-03-24T21:49:20.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:20 smithi142 conmon[20125]: 18.859292+0000 mgr.y (mgr.14144) 87 : 2021-03-24T21:49:20.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:20 smithi142 conmon[20125]: cephadm [INF] Deploying daemon osd.1 on smithi125 2021-03-24T21:49:22.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:22 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:49:22.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:22 smithi142 conmon[20125]: 49:20.595446+0000 mgr.y (mgr.14144) 88 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-24T21:49:22.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:22 smithi125 conmon[19936]: cluster 2021-03-24T21:49:20. 2021-03-24T21:49:22.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:22 smithi125 conmon[19936]: 595446+0000 mgr.y (mgr.14144) 88 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-24T21:49:22.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:22 smithi125 conmon[24927]: cluster 2021-03-24T21:49:20 2021-03-24T21:49:22.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:22 smithi125 conmon[24927]: .595446+0000 mgr.y (mgr.14144) 88 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-24T21:49:23.163 INFO:teuthology.orchestra.run.smithi125.stdout:Created osd(s) 1 on host 'smithi125' 2021-03-24T21:49:23.938 DEBUG:teuthology.orchestra.run.smithi125:osd.1> sudo journalctl -f -n 0 -u ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c@osd.1.service 2021-03-24T21:49:23.941 INFO:tasks.cephadm:Deploying osd.2 on smithi125 with /dev/vg_nvme/lv_2... 2021-03-24T21:49:23.942 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2021-03-24T21:49:23.962 INFO:journalctl@ceph.osd.1.smithi125.stdout:-- Logs begin at Wed 2021-03-03 23:51:07 UTC. -- 2021-03-24T21:49:24.326 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:24 smithi125 conmon[24927]: cluster 2021-03-24T21:49:22.595805+0000 mgr.y (mgr.14144) 89 : cluster 2021-03-24T21:49:24.327 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:24 smithi125 conmon[24927]: [DBG] pgmap v69: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-24T21:49:24.327 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:24 smithi125 conmon[24927]: audit 2021- 2021-03-24T21:49:24.327 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:24 smithi125 conmon[24927]: 03-24T21:49:23.155672+0000 mon.a (mon.0) 230 : 2021-03-24T21:49:24.327 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:24 smithi125 conmon[24927]: audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:49:24.328 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:24 smithi125 conmon[24927]: audit 2021-03-24T21:49 2021-03-24T21:49:24.328 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:24 smithi125 conmon[24927]: :23.161916+0000 mon.a (mon.0) 231 : 2021-03-24T21:49:24.328 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:24 smithi125 conmon[24927]: audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:24.328 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:24 smithi125 conmon[24927]: audit 2021-03-24T21 2021-03-24T21:49:24.328 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:24 smithi125 conmon[24927]: :49:23.169500+0000 mon.a (mon.0) 2021-03-24T21:49:24.329 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:24 smithi125 conmon[24927]: 232 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:24.329 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:24 smithi125 conmon[19936]: cluster 2021-03-24T21:49:22.595805+0000 mgr.y (mgr.14144) 2021-03-24T21:49:24.330 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:24 smithi125 conmon[19936]: 89 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-24T21:49:24.330 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:24 smithi125 conmon[19936]: audit 2021-03-24T21:49:23.155672+0000 mon.a (mon.0) 230 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:49:24.330 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:24 smithi125 conmon[19936]: audit 2021-03-24T21:49:23.161916 2021-03-24T21:49:24.331 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:24 smithi125 conmon[19936]: +0000 mon.a (mon.0) 231 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:24.331 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:24 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:49:24.331 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:24 smithi125 conmon[19936]: 03-24T21:49:23.169500+0000 mon.a (mon.0) 2021-03-24T21:49:24.332 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:24 smithi125 conmon[19936]: 232 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:24.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:24 smithi142 conmon[20125]: cluster 2021-03-24T21:49:22.595805+0000 mgr.y (mgr.14144) 89 : cluster [DBG] 2021-03-24T21:49:24.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:24 smithi142 conmon[20125]: pgmap v69: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-24T21:49:24.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:24 smithi142 conmon[20125]: audit 2021-03-24T21:49:23.155672+0000 mon.a (mon.0) 230 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:49:24.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:24 smithi142 conmon[20125]: 2021-03-24T21:49:24.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:24 smithi142 conmon[20125]: audit 2021-03-24T 2021-03-24T21:49:24.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:24 smithi142 conmon[20125]: 21:49:23.161916+0000 mon.a (mon.0) 231 2021-03-24T21:49:24.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:24 smithi142 conmon[20125]: : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:24.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:24 smithi142 conmon[20125]: audit 2021-03-24 2021-03-24T21:49:24.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:24 smithi142 conmon[20125]: T21:49:23.169500+0000 mon.a ( 2021-03-24T21:49:24.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:24 smithi142 conmon[20125]: mon.0) 232 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:26.302 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:26 smithi125 conmon[24927]: cluster 2021-03-24T21:49:24.596180+0000 mgr.y (mgr.14144) 90 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-24T21:49:26.303 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:26 smithi125 conmon[24927]: audit 2021-03-24T21:49:25.801710+0000 mon.a (mon.0) 233 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:26.303 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:26 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:49:26.304 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:26 smithi125 conmon[19936]: 21:49:24.596180+0000 mgr.y (mgr.14144) 90 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-24T21:49:26.304 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:26 smithi125 conmon[19936]: audit 2021-03- 2021-03-24T21:49:26.304 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:26 smithi125 conmon[19936]: 24T21:49:25.801710+0000 mon.a (mon.0 2021-03-24T21:49:26.304 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:26 smithi125 conmon[19936]: ) 233 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:26.304 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:26 smithi125 conmon[19936]: 2021-03-24T21:49:26.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:26 smithi142 conmon[20125]: cluster 2021-03-24T21:49:24.596180+0000 mgr.y (mgr.14144) 90 : cluster [DBG] 2021-03-24T21:49:26.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:26 smithi142 conmon[20125]: pgmap v70: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-24T21:49:26.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:26 smithi142 conmon[20125]: audit 2021-03-24T21:49:25.801710+0000 mon.a (mon.0) 233 : audit [INF] 2021-03-24T21:49:26.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:26 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:26.370 INFO:teuthology.orchestra.run.smithi125.stderr:--> Zapping: /dev/vg_nvme/lv_2 2021-03-24T21:49:26.370 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2021-03-24T21:49:26.370 INFO:teuthology.orchestra.run.smithi125.stderr: stderr: 10+0 records in 2021-03-24T21:49:26.370 INFO:teuthology.orchestra.run.smithi125.stderr:10+0 records out 2021-03-24T21:49:26.371 INFO:teuthology.orchestra.run.smithi125.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0162445 s, 645 MB/s 2021-03-24T21:49:26.371 INFO:teuthology.orchestra.run.smithi125.stderr:--> Zapping successful for: 2021-03-24T21:49:26.902 INFO:journalctl@ceph.osd.1.smithi125.stdout:Mar 24 21:49:26 smithi125 conmon[31297]: debug 2021-03-24T21:49:26.712+0000 7f879f0f9080 -1 osd.1 0 log_to_monitors {default=true} 2021-03-24T21:49:27.319 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:27 smithi125 conmon[19936]: audit 2021-03-24T21:49:26.722380 2021-03-24T21:49:27.319 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:27 smithi125 conmon[19936]: +0000 mon.c (mon.2) 3 : audit [INF] from='osd.1 v1:172.21.15.125:6805/113845288' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2021-03-24T21:49:27.319 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:27 smithi125 conmon[19936]: audit 2021-03-24T21:49:26.723041+0000 mon.a 2021-03-24T21:49:27.319 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:27 smithi125 conmon[19936]: (mon.0) 234 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2021-03-24T21:49:27.320 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:27 smithi125 conmon[24927]: audit 2021-03-24T21:49 2021-03-24T21:49:27.320 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:27 smithi125 conmon[24927]: :26.722380+0000 mon.c (mon.2) 3 : audit [INF] from='osd.1 v1:172.21.15.125:6805/113845288' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2021-03-24T21:49:27.320 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:27 smithi125 conmon[24927]: audit 2021-03-24T21:49 2021-03-24T21:49:27.321 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:27 smithi125 conmon[24927]: :26.723041+0000 mon.a (mon.0) 2021-03-24T21:49:27.321 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:27 smithi125 conmon[24927]: 234 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2021-03-24T21:49:27.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:27 smithi142 conmon[20125]: audit 2021-03-24T21:49:26.722380+0000 mon.c (mon.2) 3 2021-03-24T21:49:27.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:27 smithi142 conmon[20125]: : audit [INF] from='osd.1 v1:172.21.15.125:6805/113845288' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2021-03-24T21:49:27.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:27 smithi142 conmon[20125]: audit 2021-03-24T21:49:26.723041+0000 2021-03-24T21:49:27.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:27 smithi142 conmon[20125]: mon.a (mon.0) 234 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2021-03-24T21:49:27.924 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph orch daemon add osd smithi125:vg_nvme/lv_2 2021-03-24T21:49:28.152 INFO:journalctl@ceph.osd.1.smithi125.stdout:Mar 24 21:49:28 smithi125 conmon[31297]: debug 2021-03-24T21:49:28.048+0000 7f8796e6d700 -1 osd.1 0 waiting for initial osdmap 2021-03-24T21:49:28.152 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:28 smithi125 conmon[24927]: cluster 2021-03-24T21:49:28.153 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:28 smithi125 conmon[24927]: 2021-03-24T21:49:26.596745+0000 2021-03-24T21:49:28.153 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:28 smithi125 conmon[24927]: mgr.y (mgr.14144) 91 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-24T21:49:28.153 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:28 smithi125 conmon[24927]: audit 2021-03-24T21:49:27.044867+0000 mon.a ( 2021-03-24T21:49:28.153 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:28 smithi125 conmon[24927]: mon.0) 235 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2021-03-24T21:49:28.154 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:28 smithi125 conmon[24927]: cluster 2021-03-24T21:49:27.044989+0000 mon.a (mon.0 2021-03-24T21:49:28.154 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:28 smithi125 conmon[24927]: ) 236 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2021-03-24T21:49:28.154 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:28 smithi125 conmon[24927]: audit 2021-03-24T21:49:27.045209+0000 mon.a (mon.0 2021-03-24T21:49:28.154 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:28 smithi125 conmon[24927]: ) 237 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-03-24T21:49:28.154 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:28 smithi125 conmon[24927]: audit 2021-03-24T21:49:27.046024+0000 mon.c (mon 2021-03-24T21:49:28.155 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:28 smithi125 conmon[24927]: .2) 4 : audit [INF] from='osd.1 v1:172.21.15.125:6805/113845288' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]: dispatch 2021-03-24T21:49:28.155 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:28 smithi125 conmon[24927]: audit 2021-03-24T21:49:27.046544+0000 mon.a (mon.0 2021-03-24T21:49:28.155 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:28 smithi125 conmon[24927]: ) 238 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]: dispatch 2021-03-24T21:49:28.155 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:28 smithi125 conmon[19936]: cluster 2021-03-24T21:49:26.596745+0000 2021-03-24T21:49:28.156 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:28 smithi125 conmon[19936]: mgr.y (mgr.14144) 91 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-24T21:49:28.156 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:28 smithi125 conmon[19936]: audit 2021-03-24T21:49:27.044867+0000 mon.a (mon.0) 2021-03-24T21:49:28.156 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:28 smithi125 conmon[19936]: 235 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2021-03-24T21:49:28.156 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:28 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:49:28.157 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:28 smithi125 conmon[19936]: 03-24T21:49:27.044989+0000 mon.a (mon.0) 236 : cluster [DBG] 2021-03-24T21:49:28.157 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:28 smithi125 conmon[19936]: osdmap e11: 2 total, 1 up, 2 in 2021-03-24T21:49:28.157 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:28 smithi125 conmon[19936]: audit 2021-03-24T 2021-03-24T21:49:28.157 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:28 smithi125 conmon[19936]: 21:49:27.045209+0000 mon.a (mon.0) 237 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-03-24T21:49:28.157 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:28 smithi125 conmon[19936]: audit 2021-03-24T21: 2021-03-24T21:49:28.158 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:28 smithi125 conmon[19936]: 49:27.046024+0000 mon.c (mon.2) 4 : audit 2021-03-24T21:49:28.158 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:28 smithi125 conmon[19936]: [INF] from='osd.1 v1:172.21.15.125:6805/113845288' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]: dispatch 2021-03-24T21:49:28.158 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:28 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:49:28.158 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:28 smithi125 conmon[19936]: 03-24T21:49:27.046544+0000 mon.a (mon.0) 238 : audit [INF] 2021-03-24T21:49:28.159 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:28 smithi125 conmon[19936]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]: dispatch 2021-03-24T21:49:28.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:28 smithi142 conmon[20125]: cluster 2021-03-24T21:49:26.596745+0000 mgr.y (mgr.14144) 91 2021-03-24T21:49:28.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:28 smithi142 conmon[20125]: : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-24T21:49:28.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:28 smithi142 conmon[20125]: audit 2021-03-24T21:49:27.044867+0000 2021-03-24T21:49:28.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:28 smithi142 conmon[20125]: mon.a (mon.0) 235 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2021-03-24T21:49:28.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:28 smithi142 conmon[20125]: cluster 2021-03-24T21 2021-03-24T21:49:28.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:28 smithi142 conmon[20125]: :49:27.044989+0000 mon.a (mon. 2021-03-24T21:49:28.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:28 smithi142 conmon[20125]: 0) 236 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2021-03-24T21:49:28.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:28 smithi142 conmon[20125]: audit 2021-03 2021-03-24T21:49:28.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:28 smithi142 conmon[20125]: -24T21:49:27.045209+0000 mon.a 2021-03-24T21:49:28.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:28 smithi142 conmon[20125]: (mon.0) 237 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-03-24T21:49:28.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:28 smithi142 conmon[20125]: 2021-03-24T21:49:28.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:28 smithi142 conmon[20125]: audit 2021-03-24T21:49:28.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:28 smithi142 conmon[20125]: 2021-03-24T21:49:27.046024+0000 2021-03-24T21:49:28.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:28 smithi142 conmon[20125]: mon.c (mon.2) 4 : audit [INF] 2021-03-24T21:49:28.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:28 smithi142 conmon[20125]: from='osd.1 v1:172.21.15.125:6805/113845288' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]: dispatch 2021-03-24T21:49:28.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:28 smithi142 conmon[20125]: audit 2021-03-24T21:49:28.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:28 smithi142 conmon[20125]: 2021-03-24T21:49:27.046544+0000 2021-03-24T21:49:28.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:28 smithi142 conmon[20125]: mon.a (mon.0) 238 : audit [INF] 2021-03-24T21:49:28.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:28 smithi142 conmon[20125]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]: dispatch 2021-03-24T21:49:29.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:29 smithi142 conmon[20125]: audit 2021-03-24T21:49:28.046188+0000 mon.a (mon.0) 2021-03-24T21:49:29.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:29 smithi142 conmon[20125]: 239 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]': finished 2021-03-24T21:49:29.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:29 smithi142 conmon[20125]: cluster 2021-03-24T21:49:28.046283 2021-03-24T21:49:29.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:29 smithi142 conmon[20125]: +0000 mon.a (mon.0) 240 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2021-03-24T21:49:29.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:29 smithi142 conmon[20125]: audit 2021-03- 2021-03-24T21:49:29.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:29 smithi142 conmon[20125]: 24T21:49:28.047631+0000 mon.a 2021-03-24T21:49:29.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:29 smithi142 conmon[20125]: (mon.0) 241 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-03-24T21:49:29.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:29 smithi142 conmon[20125]: 2021-03-24T21:49:29.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:29 smithi142 conmon[20125]: audit 2021-03-24T21:49:29.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:29 smithi142 conmon[20125]: 2021-03-24T21:49:28.053949+0000 mon.a 2021-03-24T21:49:29.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:29 smithi142 conmon[20125]: (mon.0) 242 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-03-24T21:49:29.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:29 smithi142 conmon[20125]: 2021-03-24T21:49:29.373 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:29 smithi125 conmon[19936]: audit 2021-03-24T21:49:28.046188+0000 mon.a (mon.0) 2021-03-24T21:49:29.373 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:29 smithi125 conmon[19936]: 239 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]': finished 2021-03-24T21:49:29.373 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:29 smithi125 conmon[19936]: cluster 2021-03-24T21:49:28.046283+0000 mon.a (mon.0) 240 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2021-03-24T21:49:29.373 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:29 smithi125 conmon[19936]: audit 2021-03-24T21:49:28.047631+0000 mon.a (mon.0) 241 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-03-24T21:49:29.374 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:29 smithi125 conmon[19936]: audit 2021-03-24T21:49:29.374 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:29 smithi125 conmon[19936]: 2021-03-24T21:49:28 2021-03-24T21:49:29.374 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:29 smithi125 conmon[19936]: .053949+0000 mon.a (mon.0) 242 : audit 2021-03-24T21:49:29.374 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:29 smithi125 conmon[19936]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-03-24T21:49:29.375 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:29 smithi125 conmon[24927]: audit 2021-03-24T21:49:28. 2021-03-24T21:49:29.375 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:29 smithi125 conmon[24927]: 046188+0000 mon.a (mon.0) 239 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]': finished 2021-03-24T21:49:29.375 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:29 smithi125 conmon[24927]: cluster 2021-03-24T21:49 2021-03-24T21:49:29.375 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:29 smithi125 conmon[24927]: :28.046283+0000 mon.a (mon.0) 240 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2021-03-24T21:49:29.376 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:29 smithi125 conmon[24927]: audit 2021-03-24T 2021-03-24T21:49:29.376 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:29 smithi125 conmon[24927]: 21:49:28.047631+0000 mon.a (mon.0) 2021-03-24T21:49:29.376 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:29 smithi125 conmon[24927]: 241 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-03-24T21:49:29.376 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:29 smithi125 conmon[24927]: audit 2021-03 2021-03-24T21:49:29.376 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:29 smithi125 conmon[24927]: -24T21:49:28.053949+0000 mon.a (mon. 2021-03-24T21:49:29.377 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:29 smithi125 conmon[24927]: 0) 242 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-03-24T21:49:30.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:30 smithi142 conmon[20125]: cluster 2021-03-24T21:49:27.704115 2021-03-24T21:49:30.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:30 smithi142 conmon[20125]: +0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:49:30.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:30 smithi142 conmon[20125]: cluster 2021-03-24T21:49: 2021-03-24T21:49:30.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:30 smithi142 conmon[20125]: 27.704335+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:49:30.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:30 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:49:30.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:30 smithi142 conmon[20125]: -24T21:49:28.597260+0000 mgr.y ( 2021-03-24T21:49:30.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:30 smithi142 conmon[20125]: mgr.14144) 92 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-24T21:49:30.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:30 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:49:30.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:30 smithi142 conmon[20125]: 03-24T21:49:29.057602+0000 mon.a 2021-03-24T21:49:30.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:30 smithi142 conmon[20125]: (mon.0) 243 : cluster [INF] osd.1 v1:172.21.15.125:6805/113845288 boot 2021-03-24T21:49:30.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:30 smithi142 conmon[20125]: cluster 2021-03-24T21:49:30.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:30 smithi142 conmon[20125]: 2021-03-24T21:49:29.057769+0000 2021-03-24T21:49:30.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:30 smithi142 conmon[20125]: mon.a (mon.0) 244 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2021-03-24T21:49:30.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:30 smithi142 conmon[20125]: 2021-03-24T21:49:30.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:30 smithi142 conmon[20125]: audit 2021-03-24T21 2021-03-24T21:49:30.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:30 smithi142 conmon[20125]: :49:29.058629+0000 mon.a (mon.0 2021-03-24T21:49:30.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:30 smithi142 conmon[20125]: ) 245 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-03-24T21:49:30.371 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:30 smithi125 conmon[19936]: cluster 2021-03-24T21:49:27. 2021-03-24T21:49:30.372 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:30 smithi125 conmon[19936]: 704115+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:49:30.372 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:30 smithi125 conmon[19936]: cluster 2021-03-24T21:49: 2021-03-24T21:49:30.372 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:30 smithi125 conmon[19936]: 27.704335+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:49:30.372 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:30 smithi125 conmon[19936]: cluster 2021-03-24T21:49 2021-03-24T21:49:30.372 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:30 smithi125 conmon[19936]: :28.597260+0000 mgr.y (mgr.14144) 92 : cluster [DBG] 2021-03-24T21:49:30.373 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:30 smithi125 conmon[19936]: pgmap v74: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-24T21:49:30.373 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:30 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:49:30.373 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:30 smithi125 conmon[19936]: 21:49:29.057602+0000 mon.a ( 2021-03-24T21:49:30.373 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:30 smithi125 conmon[19936]: mon.0) 243 : cluster [INF] osd.1 v1:172.21.15.125:6805/113845288 boot 2021-03-24T21:49:30.374 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:30 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:49:30.374 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:30 smithi125 conmon[19936]: -03-24T21:49:29.057769 2021-03-24T21:49:30.374 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:30 smithi125 conmon[19936]: +0000 mon.a (mon.0) 244 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2021-03-24T21:49:30.374 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:30 smithi125 conmon[19936]: audit 2021-03-24 2021-03-24T21:49:30.374 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:30 smithi125 conmon[19936]: T21:49:29.058629+0000 mon.a (mon.0) 245 : 2021-03-24T21:49:30.375 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:30 smithi125 conmon[19936]: audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-03-24T21:49:30.375 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:30 smithi125 conmon[24927]: cluster 2021-03-24T21:49: 2021-03-24T21:49:30.375 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:30 smithi125 conmon[24927]: 27.704115+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:49:30.375 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:30 smithi125 conmon[24927]: cluster 2021-03-24T21:49:27.704335 2021-03-24T21:49:30.376 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:30 smithi125 conmon[24927]: +0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:49:30.376 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:30 smithi125 conmon[24927]: 2021-03-24T21:49:30.376 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:30 smithi125 conmon[24927]: cluster 2021-03-24T21:49: 2021-03-24T21:49:30.376 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:30 smithi125 conmon[24927]: 28.597260+0000 mgr.y (mgr.14144) 92 : 2021-03-24T21:49:30.377 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:30 smithi125 conmon[24927]: cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-24T21:49:30.377 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:30 smithi125 conmon[24927]: cluster 2021-03-24T21:49:30.377 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:30 smithi125 conmon[24927]: 2021- 2021-03-24T21:49:30.377 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:30 smithi125 conmon[24927]: 03-24T21:49:29.057602+0000 mon.a (mon.0 2021-03-24T21:49:30.377 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:30 smithi125 conmon[24927]: ) 243 : cluster [INF] osd.1 v1:172.21.15.125:6805/113845288 boot 2021-03-24T21:49:30.378 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:30 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:49:30.378 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:30 smithi125 conmon[24927]: -03-24T21:49:29.057769+0000 mon.a (mon.0) 2021-03-24T21:49:30.378 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:30 smithi125 conmon[24927]: 244 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2021-03-24T21:49:30.378 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:30 smithi125 conmon[24927]: audit 2021-03-24T21:49:30.379 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:30 smithi125 conmon[24927]: 2021-03- 2021-03-24T21:49:30.379 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:30 smithi125 conmon[24927]: 24T21:49:29.058629+0000 mon.a (mon.0 2021-03-24T21:49:30.379 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:30 smithi125 conmon[24927]: ) 245 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-03-24T21:49:31.336 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:31 smithi125 conmon[19936]: cluster 2021-03-24T21:49:30.074298+0000 mon.a (mon.0) 246 : cluster [DBG] 2021-03-24T21:49:31.337 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:31 smithi125 conmon[19936]: osdmap e14: 2 total, 2 up, 2 in 2021-03-24T21:49:31.337 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:31 smithi125 conmon[19936]: audit 2021-03-24T21:49:30.710983+0000 mon.a (mon.0) 247 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:31.337 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:31 smithi125 conmon[19936]: audit 2021 2021-03-24T21:49:31.338 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:31 smithi125 conmon[19936]: -03-24T21:49:30.715584+0000 mon.a (mon.0) 248 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:31.338 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:31 smithi125 conmon[19936]: audit 2021-03-24T21:49:30. 2021-03-24T21:49:31.338 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:31 smithi125 conmon[19936]: 717819+0000 mon.a (mon.0) 249 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:49:31.338 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:31 smithi125 conmon[19936]: audit 2021-03-24T21:49:30.723809+0000 mon.a (mon 2021-03-24T21:49:31.339 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:31 smithi125 conmon[19936]: .0) 250 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:31.339 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:31 smithi125 conmon[19936]: audit 2021-03 2021-03-24T21:49:31.339 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:31 smithi125 conmon[19936]: -24T21:49:30.838946+0000 mon.a (mon.0) 251 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-24T21:49:31.339 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:31 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:49:31.339 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:31 smithi125 conmon[19936]: 03-24T21:49:30.840771+0000 2021-03-24T21:49:31.340 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:31 smithi125 conmon[19936]: mon.a (mon.0) 252 : audit 2021-03-24T21:49:31.340 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:31 smithi125 conmon[19936]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-24T21:49:31.340 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:31 smithi125 conmon[19936]: audit 2021-03-24T21:49:31.340 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:31 smithi125 conmon[19936]: 2021-03- 2021-03-24T21:49:31.341 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:31 smithi125 conmon[19936]: 24T21:49:30.841815+0000 mon.a (mon 2021-03-24T21:49:31.341 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:31 smithi125 conmon[19936]: .0) 253 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:49:31.341 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:31 smithi125 conmon[24927]: cluster 2021-03-24T21:49: 2021-03-24T21:49:31.341 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:31 smithi125 conmon[24927]: 30.074298+0000 mon.a (mon.0) 246 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2021-03-24T21:49:31.342 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:31 smithi125 conmon[24927]: audit 2021-03-24 2021-03-24T21:49:31.342 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:31 smithi125 conmon[24927]: T21:49:30.710983+0000 mon.a (mon.0) 247 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:31.342 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:31 smithi125 conmon[24927]: audit 2021-03-24T21:49:31.342 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:31 smithi125 conmon[24927]: 2021-03-24T21:49:30.715584+0000 mon.a (mon.0) 248 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:31.342 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:31 smithi125 conmon[24927]: audit 2021-03-24T21:49:30.717819+0000 2021-03-24T21:49:31.343 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:31 smithi125 conmon[24927]: mon.a (mon.0) 249 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:49:31.343 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:31 smithi125 conmon[24927]: audit 2021-03-24T21:49:30.723809+0000 mon.a (mon.0) 250 2021-03-24T21:49:31.343 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:31 smithi125 conmon[24927]: : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:31.343 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:31 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T21:49:31.344 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:31 smithi125 conmon[24927]: 24T21:49:30.838946+0000 mon.a (mon.0) 251 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-24T21:49:31.344 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:31 smithi125 conmon[24927]: audit 2021-03-24T21:49:30.840771+0000 mon.a ( 2021-03-24T21:49:31.344 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:31 smithi125 conmon[24927]: mon.0) 252 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-24T21:49:31.344 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:31 smithi125 conmon[24927]: audit 2021-03-24T21:49:31.344 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:31 smithi125 conmon[24927]: 2021-03-24T21:49:30.841815+0000 mon.a (mon.0) 253 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:49:31.345 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:31 smithi125 conmon[24927]: 2021-03-24T21:49:31.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:31 smithi142 conmon[20125]: cluster 2021-03-24T21:49:30.074298+0000 mon.a (mon.0) 246 : 2021-03-24T21:49:31.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:31 smithi142 conmon[20125]: cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2021-03-24T21:49:31.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:31 smithi142 conmon[20125]: audit 2021-03-24T21:49:30.710983+0000 mon.a (mon. 2021-03-24T21:49:31.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:31 smithi142 conmon[20125]: 0) 247 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:31.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:31 smithi142 conmon[20125]: audit 2021-03-24T21:49:31.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:31 smithi142 conmon[20125]: 2021-03-24T21:49:30.715584+0000 2021-03-24T21:49:31.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:31 smithi142 conmon[20125]: mon.a (mon.0) 248 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:31.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:31 smithi142 conmon[20125]: 2021-03-24T21:49:31.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:31 smithi142 conmon[20125]: audit 2021-03-24T21:49:31.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:31 smithi142 conmon[20125]: 2021-03-24T21:49:30.717819+0000 2021-03-24T21:49:31.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:31 smithi142 conmon[20125]: mon.a (mon.0) 249 : audit [DBG] 2021-03-24T21:49:31.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:31 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:49:31.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:31 smithi142 conmon[20125]: audit 2021-03-24T21:49:31.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:31 smithi142 conmon[20125]: 2021-03-24T21:49:30.723809+0000 2021-03-24T21:49:31.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:31 smithi142 conmon[20125]: mon.a (mon.0) 250 : audit [INF] 2021-03-24T21:49:31.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:31 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:31.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:31 smithi142 conmon[20125]: audit 2021-03-24T21:49:31.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:31 smithi142 conmon[20125]: 2021-03-24T21:49:30.838946+0000 2021-03-24T21:49:31.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:31 smithi142 conmon[20125]: mon.a (mon.0) 251 : audit [DBG] 2021-03-24T21:49:31.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:31 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-24T21:49:31.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:31 smithi142 conmon[20125]: audit 2021-03-24T21:49:31.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:31 smithi142 conmon[20125]: 2021-03-24T21:49:30.840771+0000 2021-03-24T21:49:31.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:31 smithi142 conmon[20125]: mon.a (mon.0) 252 : audit [INF] 2021-03-24T21:49:31.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:31 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-24T21:49:31.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:31 smithi142 conmon[20125]: audit 2021-03-24T21:49:31.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:31 smithi142 conmon[20125]: 2021-03-24T21:49:30.841815+0000 mon.a ( 2021-03-24T21:49:31.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:31 smithi142 conmon[20125]: mon.0) 253 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:49:32.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:32 smithi142 conmon[20125]: cluster 2021-03-24T21:49:30.597724+0000 mgr.y (mgr.14144) 93 2021-03-24T21:49:32.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:32 smithi142 conmon[20125]: : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2021-03-24T21:49:32.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:32 smithi142 conmon[20125]: cephadm 2021-03-24T21:49:30.703344+0000 2021-03-24T21:49:32.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:32 smithi142 conmon[20125]: mgr.y (mgr.14144) 94 : cephadm [INF] Detected new or changed devices on smithi125 2021-03-24T21:49:32.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:32 smithi142 conmon[20125]: audit 2021-03-24T21:49:32.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:32 smithi142 conmon[20125]: 2021-03- 2021-03-24T21:49:32.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:32 smithi142 conmon[20125]: 24T21:49:30.838110+0000 mgr.y 2021-03-24T21:49:32.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:32 smithi142 conmon[20125]: (mgr.14144) 95 : audit [DBG] from='client.14256 v1:172.21.15.125:0/156848273' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi125:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:49:32.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:32 smithi142 conmon[20125]: cephadm 2021-03-24T21:49:32.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:32 smithi142 conmon[20125]: 2021-03 2021-03-24T21:49:32.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:32 smithi142 conmon[20125]: -24T21:49:30.840349+0000 2021-03-24T21:49:32.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:32 smithi142 conmon[20125]: mgr.y (mgr.14144) 96 : cephadm [INF] 2021-03-24T21:49:32.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:32 smithi142 conmon[20125]: Applying service osd.None on host smithi125... 2021-03-24T21:49:32.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:32 smithi125 conmon[19936]: cluster 2021-03-24T21:49 2021-03-24T21:49:32.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:32 smithi125 conmon[19936]: :30.597724+0000 mgr.y (mgr.14144) 93 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2021-03-24T21:49:32.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:32 smithi125 conmon[19936]: cephadm 2021-03-24T 2021-03-24T21:49:32.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:32 smithi125 conmon[19936]: 21:49:30.703344+0000 mgr.y 2021-03-24T21:49:32.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:32 smithi125 conmon[19936]: (mgr.14144) 94 : cephadm 2021-03-24T21:49:32.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:32 smithi125 conmon[19936]: [INF] Detected new or changed devices on smithi125 2021-03-24T21:49:32.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:32 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:49:32.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:32 smithi125 conmon[19936]: 03-24T21:49:30.838110 2021-03-24T21:49:32.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:32 smithi125 conmon[19936]: +0000 mgr.y (mgr.14144) 2021-03-24T21:49:32.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:32 smithi125 conmon[19936]: 95 : audit [DBG] from='client.14256 v1:172.21.15.125:0/156848273' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi125:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:49:32.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:32 smithi125 conmon[19936]: cephadm 2021- 2021-03-24T21:49:32.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:32 smithi125 conmon[19936]: 03-24T21:49:30.840349 2021-03-24T21:49:32.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:32 smithi125 conmon[19936]: +0000 mgr.y (mgr.14144) 96 2021-03-24T21:49:32.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:32 smithi125 conmon[19936]: : cephadm [INF] Applying service osd.None on host smithi125... 2021-03-24T21:49:32.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:32 smithi125 conmon[24927]: cluster 2021-03-24T21:49 2021-03-24T21:49:32.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:32 smithi125 conmon[24927]: :30.597724+0000 mgr.y (mgr.14144) 93 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2021-03-24T21:49:32.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:32 smithi125 conmon[24927]: cephadm 2021-03-24T21:49:32.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:32 smithi125 conmon[24927]: 2021-03-24T21:49:30 2021-03-24T21:49:32.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:32 smithi125 conmon[24927]: .703344+0000 mgr.y (mgr.14144 2021-03-24T21:49:32.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:32 smithi125 conmon[24927]: ) 94 : cephadm [INF] Detected new or changed devices on smithi125 2021-03-24T21:49:32.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:32 smithi125 conmon[24927]: audit 2021- 2021-03-24T21:49:32.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:32 smithi125 conmon[24927]: 03-24T21:49:30.838110+0000 2021-03-24T21:49:32.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:32 smithi125 conmon[24927]: mgr.y (mgr.14144) 95 : 2021-03-24T21:49:32.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:32 smithi125 conmon[24927]: audit [DBG] from='client.14256 v1:172.21.15.125:0/156848273' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi125:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:49:32.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:32 smithi125 conmon[24927]: cephadm 2021-03-24T21:49:32.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:32 smithi125 conmon[24927]: 2021-03-24T21:49:30.840349 2021-03-24T21:49:32.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:32 smithi125 conmon[24927]: +0000 mgr.y (mgr.14144) 96 : cephadm 2021-03-24T21:49:32.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:32 smithi125 conmon[24927]: [INF] Applying service osd.None on host smithi125... 2021-03-24T21:49:34.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:34 smithi142 conmon[20125]: cluster 2021-03-24T21:49:32 2021-03-24T21:49:34.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:34 smithi142 conmon[20125]: .598250+0000 mgr.y (mgr.14144) 97 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2021-03-24T21:49:34.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:34 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:49:34.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:34 smithi142 conmon[20125]: 03-24T21:49:33.329060+0000 mon.a (mon.0) 254 : audit [INF] 2021-03-24T21:49:34.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:34 smithi142 conmon[20125]: from='client.? v1:172.21.15.125:0/3833671345' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d6a361f8-4e4d-4f46-8ab9-01bf71dbb2c8"}]: dispatch 2021-03-24T21:49:34.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:34 smithi142 conmon[20125]: audit 2021-03-24T21:49: 2021-03-24T21:49:34.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:34 smithi142 conmon[20125]: 33.337534+0000 mon.a (mon.0) 255 : audit 2021-03-24T21:49:34.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:34 smithi142 conmon[20125]: [INF] from='client.? v1:172.21.15.125:0/3833671345' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d6a361f8-4e4d-4f46-8ab9-01bf71dbb2c8"}]': finished 2021-03-24T21:49:34.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:34 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:49:34.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:34 smithi142 conmon[20125]: 49:33.337642+0000 mon.a (mon.0) 256 2021-03-24T21:49:34.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:34 smithi142 conmon[20125]: : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2021-03-24T21:49:34.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:34 smithi142 conmon[20125]: audit 2021-03-24 2021-03-24T21:49:34.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:34 smithi142 conmon[20125]: T21:49:33.337789+0000 mon.a (mon. 2021-03-24T21:49:34.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:34 smithi142 conmon[20125]: 0) 257 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-24T21:49:34.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:34 smithi142 conmon[20125]: audit 2021-03 2021-03-24T21:49:34.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:34 smithi142 conmon[20125]: -24T21:49:33.967139+0000 mon.c ( 2021-03-24T21:49:34.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:34 smithi142 conmon[20125]: mon.2) 5 : audit [DBG] from='client.? v1:172.21.15.125:0/460191087' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-24T21:49:34.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:34 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:49:34.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:34 smithi125 conmon[24927]: 21:49:32.598250+0000 mgr.y (mgr.14144) 97 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2021-03-24T21:49:34.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:34 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T21:49:34.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:34 smithi125 conmon[24927]: 24T21:49:33.329060+0000 2021-03-24T21:49:34.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:34 smithi125 conmon[24927]: mon.a (mon.0) 254 : audit 2021-03-24T21:49:34.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:34 smithi125 conmon[24927]: [INF] from='client.? v1:172.21.15.125:0/3833671345' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d6a361f8-4e4d-4f46-8ab9-01bf71dbb2c8"}]: dispatch 2021-03-24T21:49:34.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:34 smithi125 conmon[24927]: audit 2021-03 2021-03-24T21:49:34.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:34 smithi125 conmon[24927]: -24T21:49:33.337534+0000 mon.a (mon.0 2021-03-24T21:49:34.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:34 smithi125 conmon[24927]: ) 255 : audit [INF] from='client.? v1:172.21.15.125:0/3833671345' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d6a361f8-4e4d-4f46-8ab9-01bf71dbb2c8"}]': finished 2021-03-24T21:49:34.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:34 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:49:34.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:34 smithi125 conmon[24927]: 21:49:33.337642+0000 mon.a (mon.0) 256 2021-03-24T21:49:34.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:34 smithi125 conmon[24927]: : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2021-03-24T21:49:34.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:34 smithi125 conmon[24927]: audit 2021-03-24 2021-03-24T21:49:34.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:34 smithi125 conmon[24927]: T21:49:33.337789+0000 mon.a (mon.0) 2021-03-24T21:49:34.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:34 smithi125 conmon[24927]: 257 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-24T21:49:34.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:34 smithi125 conmon[24927]: audit 2021-03 2021-03-24T21:49:34.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:34 smithi125 conmon[24927]: -24T21:49:33.967139+0000 mon.c (mon. 2021-03-24T21:49:34.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:34 smithi125 conmon[24927]: 2) 5 : audit [DBG] from='client.? v1:172.21.15.125:0/460191087' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-24T21:49:34.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:34 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:49:34.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:34 smithi125 conmon[19936]: :49:32.598250+0000 mgr.y (mgr.14144) 97 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2021-03-24T21:49:34.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:34 smithi125 conmon[19936]: audit 2021-03-24T21:49:34.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:34 smithi125 conmon[19936]: 2021-03- 2021-03-24T21:49:34.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:34 smithi125 conmon[19936]: 24T21:49:33.329060 2021-03-24T21:49:34.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:34 smithi125 conmon[19936]: +0000 mon.a (mon.0) 254 2021-03-24T21:49:34.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:34 smithi125 conmon[19936]: : audit [INF] from='client.? v1:172.21.15.125:0/3833671345' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d6a361f8-4e4d-4f46-8ab9-01bf71dbb2c8"}]: dispatch 2021-03-24T21:49:34.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:34 smithi125 conmon[19936]: 2021-03-24T21:49:34.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:34 smithi125 conmon[19936]: audit 2021-03-24T21:49:34.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:34 smithi125 conmon[19936]: 2021-03-24T21:49 2021-03-24T21:49:34.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:34 smithi125 conmon[19936]: :33.337534+0000 mon.a (mon 2021-03-24T21:49:34.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:34 smithi125 conmon[19936]: .0) 255 : audit [INF] from='client.? v1:172.21.15.125:0/3833671345' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d6a361f8-4e4d-4f46-8ab9-01bf71dbb2c8"}]': finished 2021-03-24T21:49:34.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:34 smithi125 conmon[19936]: 2021-03-24T21:49:34.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:34 smithi125 conmon[19936]: cluster 2021-03-24T21:49:34.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:34 smithi125 conmon[19936]: 2021-03-24 2021-03-24T21:49:34.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:34 smithi125 conmon[19936]: T21:49:33.337642+0000 2021-03-24T21:49:34.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:34 smithi125 conmon[19936]: mon.a (mon.0) 256 : cluster 2021-03-24T21:49:34.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:34 smithi125 conmon[19936]: [DBG] osdmap e15: 3 total, 2 up, 3 in 2021-03-24T21:49:34.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:34 smithi125 conmon[19936]: audit 2021 2021-03-24T21:49:34.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:34 smithi125 conmon[19936]: -03-24T21:49:33. 2021-03-24T21:49:34.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:34 smithi125 conmon[19936]: 337789+0000 mon.a (mon.0) 2021-03-24T21:49:34.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:34 smithi125 conmon[19936]: 257 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-24T21:49:34.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:34 smithi125 conmon[19936]: audit 2021-03-24T21:49:34.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:34 smithi125 conmon[19936]: 2021-03 2021-03-24T21:49:34.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:34 smithi125 conmon[19936]: -24T21:49:33.967139 2021-03-24T21:49:34.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:34 smithi125 conmon[19936]: +0000 mon.c (mon.2) 5 : audit 2021-03-24T21:49:34.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:34 smithi125 conmon[19936]: [DBG] from='client.? v1:172.21.15.125:0/460191087' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-24T21:49:36.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:36 smithi142 conmon[20125]: cluster 2021-03-24T21:49:34.598713+0000 mgr.y (mgr.14144) 98 : 2021-03-24T21:49:36.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:36 smithi142 conmon[20125]: cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-24T21:49:36.369 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:36 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:49:36.369 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:36 smithi125 conmon[19936]: 21:49:34.598713+0000 mgr.y (mgr.14144) 98 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-24T21:49:36.370 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:36 smithi125 conmon[24927]: cluster 2021-03-24T21:49 2021-03-24T21:49:36.370 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:36 smithi125 conmon[24927]: :34.598713+0000 mgr.y (mgr.14144) 98 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-24T21:49:38.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:38 smithi142 conmon[20125]: cluster 2021-03-24T21:49:36.599212+0000 mgr.y (mgr.14144) 99 2021-03-24T21:49:38.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:38 smithi142 conmon[20125]: : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-24T21:49:38.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:38 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:49:38.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:38 smithi125 conmon[19936]: 49:36.599212+0000 mgr.y (mgr.14144) 99 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-24T21:49:38.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:38 smithi125 conmon[24927]: cluster 2021-03-24T21:49 2021-03-24T21:49:38.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:38 smithi125 conmon[24927]: :36.599212+0000 mgr.y (mgr.14144) 99 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-24T21:49:40.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:40 smithi142 conmon[20125]: cluster 2021-03-24T21:49:38.599569+0000 mgr.y (mgr.14144) 100 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-24T21:49:40.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:40 smithi125 conmon[19936]: cluster 2021-03-24T21:49:38. 2021-03-24T21:49:40.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:40 smithi125 conmon[19936]: 599569+0000 mgr.y (mgr.14144) 100 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-24T21:49:40.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:40 smithi125 conmon[24927]: cluster 2021-03-24T21:49:38 2021-03-24T21:49:40.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:40 smithi125 conmon[24927]: .599569+0000 mgr.y (mgr.14144) 100 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-24T21:49:41.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:41 smithi142 conmon[20125]: audit 2021-03-24T21:49:40.280808+0000 mon.a (mon.0 2021-03-24T21:49:41.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:41 smithi142 conmon[20125]: ) 258 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2021-03-24T21:49:41.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:41 smithi142 conmon[20125]: audit 2021-03-24 2021-03-24T21:49:41.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:41 smithi142 conmon[20125]: T21:49:40.282216+0000 mon.a (mon. 2021-03-24T21:49:41.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:41 smithi142 conmon[20125]: 0) 259 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:49:41.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:41 smithi142 conmon[20125]: 2021-03-24T21:49:41.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:41 smithi142 conmon[20125]: cephadm 2021 2021-03-24T21:49:41.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:41 smithi142 conmon[20125]: -03-24T21:49:40 2021-03-24T21:49:41.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:41 smithi142 conmon[20125]: .283395+0000 mgr.y (mgr. 2021-03-24T21:49:41.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:41 smithi142 conmon[20125]: 14144) 101 : cephadm [INF] Deploying daemon osd.2 on smithi125 2021-03-24T21:49:41.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:41 smithi142 conmon[20125]: 2021-03-24T21:49:41.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:41 smithi125 conmon[24927]: audit 2021-03-24T21:49:40.280808+0000 mon.a ( 2021-03-24T21:49:41.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:41 smithi125 conmon[24927]: mon.0) 258 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2021-03-24T21:49:41.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:41 smithi125 conmon[24927]: audit 2021- 2021-03-24T21:49:41.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:41 smithi125 conmon[24927]: 03-24T21:49:40.282216+0000 mon.a (mon.0) 259 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:49:41.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:41 smithi125 conmon[24927]: cephadm 2021 2021-03-24T21:49:41.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:41 smithi125 conmon[24927]: -03-24T21:49:40.283395+0000 mgr.y ( 2021-03-24T21:49:41.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:41 smithi125 conmon[24927]: mgr.14144) 101 : cephadm [INF] Deploying daemon osd.2 on smithi125 2021-03-24T21:49:41.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:41 smithi125 conmon[19936]: audit 2021-03-24T21:49 2021-03-24T21:49:41.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:41 smithi125 conmon[19936]: :40.280808+0000 mon.a (mon.0) 258 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2021-03-24T21:49:41.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:41 smithi125 conmon[19936]: audit 2021-03-24T21:49:41.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:41 smithi125 conmon[19936]: 2021-03- 2021-03-24T21:49:41.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:41 smithi125 conmon[19936]: 24T21:49 2021-03-24T21:49:41.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:41 smithi125 conmon[19936]: :40.282216+0000 2021-03-24T21:49:41.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:41 smithi125 conmon[19936]: mon.a (mon.0 2021-03-24T21:49:41.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:41 smithi125 conmon[19936]: ) 259 : audit [DBG] 2021-03-24T21:49:41.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:41 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:49:41.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:41 smithi125 conmon[19936]: cephadm 2021-03-24T21:49:40.283395+0000 2021-03-24T21:49:41.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:41 smithi125 conmon[19936]: mgr.y (mgr.14144) 101 : cephadm 2021-03-24T21:49:41.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:41 smithi125 conmon[19936]: [INF] Deploying daemon osd.2 on smithi125 2021-03-24T21:49:42.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:42 smithi142 conmon[20125]: cluster 2021-03-24T21:49:40.599943+0000 mgr.y (mgr. 2021-03-24T21:49:42.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:42 smithi142 conmon[20125]: 14144) 102 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-24T21:49:42.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:42 smithi125 conmon[19936]: cluster 2021-03-24T21:49: 2021-03-24T21:49:42.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:42 smithi125 conmon[19936]: 40.599943+0000 mgr.y (mgr.14144) 102 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-24T21:49:42.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:42 smithi125 conmon[24927]: cluster 2021-03-24T21:49:40. 2021-03-24T21:49:42.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:42 smithi125 conmon[24927]: 599943+0000 mgr.y (mgr.14144) 102 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-24T21:49:44.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:44 smithi142 conmon[20125]: cluster 2021-03-24T21:49:42.600401+0000 mgr.y (mgr.14144) 103 : 2021-03-24T21:49:44.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:44 smithi142 conmon[20125]: cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-24T21:49:44.401 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:44 smithi125 conmon[24927]: cluster 2021-03-24T21:49: 2021-03-24T21:49:44.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:44 smithi125 conmon[24927]: 42.600401+0000 mgr.y (mgr.14144) 103 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-24T21:49:44.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:44 smithi125 conmon[19936]: cluster 2021-03-24T21:49:42.600401+0000 mgr.y 2021-03-24T21:49:44.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:44 smithi125 conmon[19936]: (mgr.14144) 103 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-24T21:49:44.670 INFO:teuthology.orchestra.run.smithi125.stdout:Created osd(s) 2 on host 'smithi125' 2021-03-24T21:49:45.152 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:45 smithi125 conmon[24927]: audit 2021-03-24T21: 2021-03-24T21:49:45.152 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:45 smithi125 conmon[24927]: 49:44.661653+0000 mon.a (mon.0) 260 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:49:45.152 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:45 smithi125 conmon[24927]: audit 2021-03-24T21:49:44 2021-03-24T21:49:45.152 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:45 smithi125 conmon[24927]: .669598+0000 mon.a (mon.0) 261 : audit 2021-03-24T21:49:45.153 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:45 smithi125 conmon[24927]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:45.153 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:45 smithi125 conmon[24927]: audit 2021-03-24T21 2021-03-24T21:49:45.153 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:45 smithi125 conmon[24927]: :49:44.677158+0000 mon.a (mon.0) 2021-03-24T21:49:45.153 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:45 smithi125 conmon[24927]: 262 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:45.154 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:45 smithi125 conmon[19936]: audit 2021-03-24T21 2021-03-24T21:49:45.154 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:45 smithi125 conmon[19936]: :49:44.661653+0000 mon.a (mon.0) 260 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:49:45.154 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:45 smithi125 conmon[19936]: audit 2021-03 2021-03-24T21:49:45.154 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:45 smithi125 conmon[19936]: -24T21: 2021-03-24T21:49:45.155 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:45 smithi125 conmon[19936]: 49:44.669598+0000 2021-03-24T21:49:45.155 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:45 smithi125 conmon[19936]: mon.a (mon.0) 2021-03-24T21:49:45.155 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:45 smithi125 conmon[19936]: 261 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:45.155 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:45 smithi125 conmon[19936]: 2021-03-24T21:49:45.155 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:45 smithi125 conmon[19936]: audit 2021-03-24T21:49:45.156 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:45 smithi125 conmon[19936]: 2021-03-24 2021-03-24T21:49:45.156 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:45 smithi125 conmon[19936]: T21:49:44. 2021-03-24T21:49:45.156 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:45 smithi125 conmon[19936]: 677158+0000 mon.a (mon.0 2021-03-24T21:49:45.156 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:45 smithi125 conmon[19936]: ) 262 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:45.156 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:45 smithi125 conmon[19936]: 2021-03-24T21:49:45.467 DEBUG:teuthology.orchestra.run.smithi125:osd.2> sudo journalctl -f -n 0 -u ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c@osd.2.service 2021-03-24T21:49:45.470 INFO:tasks.cephadm:Deploying osd.3 on smithi125 with /dev/vg_nvme/lv_1... 2021-03-24T21:49:45.470 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2021-03-24T21:49:45.493 INFO:journalctl@ceph.osd.2.smithi125.stdout:-- Logs begin at Wed 2021-03-03 23:51:07 UTC. -- 2021-03-24T21:49:45.648 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:45 smithi142 conmon[20125]: audit 2021-03-24T21:49:44.661653+0000 mon.a (mon.0) 260 : 2021-03-24T21:49:45.648 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:45 smithi142 conmon[20125]: audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:49:45.648 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:45 smithi142 conmon[20125]: audit 2021-03-24T21:49:44.669598+0000 mon.a (mon 2021-03-24T21:49:45.648 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:45 smithi142 conmon[20125]: .0) 261 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:45.649 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:45 smithi142 conmon[20125]: audit 2021-03-24T21:49:45.649 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:45 smithi142 conmon[20125]: 2021-03-24T21:49:44.677158+0000 2021-03-24T21:49:45.649 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:45 smithi142 conmon[20125]: mon.a (mon.0) 262 : audit [INF] 2021-03-24T21:49:45.649 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:45 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:46.372 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:46 smithi125 conmon[19936]: cluster 2021-03-24T21:49: 2021-03-24T21:49:46.372 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:46 smithi125 conmon[19936]: 44.600739+0000 mgr.y (mgr.14144) 104 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-24T21:49:46.373 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:46 smithi125 conmon[24927]: cluster 2021-03-24T21:49:44. 2021-03-24T21:49:46.373 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:46 smithi125 conmon[24927]: 600739+0000 mgr.y (mgr.14144) 104 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-24T21:49:46.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:46 smithi142 conmon[20125]: cluster 2021-03-24T21:49:44.600739+0000 mgr.y (mgr.14144) 104 2021-03-24T21:49:46.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:46 smithi142 conmon[20125]: : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-24T21:49:47.971 INFO:teuthology.orchestra.run.smithi125.stderr:--> Zapping: /dev/vg_nvme/lv_1 2021-03-24T21:49:47.972 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2021-03-24T21:49:47.972 INFO:teuthology.orchestra.run.smithi125.stderr: stderr: 10+0 records in 2021-03-24T21:49:47.972 INFO:teuthology.orchestra.run.smithi125.stderr:10+0 records out 2021-03-24T21:49:47.972 INFO:teuthology.orchestra.run.smithi125.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0159047 s, 659 MB/s 2021-03-24T21:49:47.973 INFO:teuthology.orchestra.run.smithi125.stderr:--> Zapping successful for: 2021-03-24T21:49:48.146 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:48 smithi125 conmon[19936]: cluster 2021-03-24T21:49:46.601235+0000 mgr.y (mgr.14144) 105 2021-03-24T21:49:48.146 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:48 smithi125 conmon[19936]: : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-24T21:49:48.146 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:48 smithi125 conmon[19936]: audit 2021-03-24T21:49:47.513609+0000 mon.a (mon.0) 263 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:48.147 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:48 smithi125 conmon[19936]: audit 2021-03-24T21:49 2021-03-24T21:49:48.147 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:48 smithi125 conmon[19936]: :47.985957+0000 mon.a (mon.0) 264 2021-03-24T21:49:48.147 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:48 smithi125 conmon[19936]: : audit [INF] from='osd.2 v1:172.21.15.125:6809/246939163' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2021-03-24T21:49:48.147 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:48 smithi125 conmon[24927]: cluster 2021-03-24T21:49:46.601235+0000 mgr.y (mgr.14144) 105 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-24T21:49:48.148 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:48 smithi125 conmon[24927]: audit 2021-03-24T21:49:47.513609+0000 mon.a (mon.0) 263 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:48.148 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:48 smithi125 conmon[24927]: audit 2021-03-24T21:49:47.985957+0000 mon.a (mon.0) 264 : audit [INF] from='osd.2 v1:172.21.15.125:6809/246939163' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2021-03-24T21:49:48.148 INFO:journalctl@ceph.osd.2.smithi125.stdout:Mar 24 21:49:47 smithi125 conmon[34620]: debug 2021-03-24T21:49:47.976+0000 7f472d56d080 -1 osd.2 0 log_to_monitors {default=true} 2021-03-24T21:49:48.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:48 smithi142 conmon[20125]: cluster 2021-03-24T21:49: 2021-03-24T21:49:48.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:48 smithi142 conmon[20125]: 46.601235+0000 mgr.y (mgr.14144) 105 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-24T21:49:48.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:48 smithi142 conmon[20125]: audit 2021-03-24T21:49 2021-03-24T21:49:48.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:48 smithi142 conmon[20125]: :47.513609+0000 mon.a (mon.0) 263 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:48.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:48 smithi142 conmon[20125]: audit 2021-03-24T21:49:47.985957 2021-03-24T21:49:48.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:48 smithi142 conmon[20125]: +0000 mon.a (mon.0) 264 : audit [INF] from='osd.2 v1:172.21.15.125:6809/246939163' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2021-03-24T21:49:48.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:48 smithi142 conmon[20125]: 2021-03-24T21:49:48.884 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph orch daemon add osd smithi125:vg_nvme/lv_1 2021-03-24T21:49:49.772 INFO:journalctl@ceph.osd.2.smithi125.stdout:Mar 24 21:49:49 smithi125 conmon[34620]: debug 2021-03-24T21:49:49.773 INFO:journalctl@ceph.osd.2.smithi125.stdout:Mar 24 21:49:49 smithi125 conmon[34620]: 2021-03-24T21:49:49.524+0000 7f4723ade700 -1 osd.2 0 waiting for initial osdmap 2021-03-24T21:49:49.773 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:49 smithi125 conmon[24927]: audit 2021-03-24T21:49:48.519140 2021-03-24T21:49:49.774 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:49 smithi125 conmon[24927]: +0000 mon.a (mon.0) 265 : audit [INF] from='osd.2 v1:172.21.15.125:6809/246939163' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2021-03-24T21:49:49.774 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:49 smithi125 conmon[24927]: cluster 2021-03-24T21:49:49.774 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:49 smithi125 conmon[24927]: 2021-03-24T21:49:48.519395+0000 mon.a (mon.0) 266 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2021-03-24T21:49:49.774 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:49 smithi125 conmon[24927]: audit 2021-03-24T21: 2021-03-24T21:49:49.775 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:49 smithi125 conmon[24927]: 49:48.519911+0000 mon.a (mon.0) 267 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-24T21:49:49.775 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:49 smithi125 conmon[24927]: audit 2021-03-24T21:49: 2021-03-24T21:49:49.775 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:49 smithi125 conmon[24927]: 48.521039+0000 mon.a (mon.0) 268 : audit [INF] from='osd.2 v1:172.21.15.125:6809/246939163' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]: dispatch 2021-03-24T21:49:49.775 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:49 smithi125 conmon[24927]: cluster 2021-03-24T21:49:48.601703 2021-03-24T21:49:49.776 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:49 smithi125 conmon[24927]: +0000 mgr.y (mgr.14144) 106 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-24T21:49:49.777 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:49 smithi125 conmon[19936]: audit 2021-03-24T21:49:48.519140 2021-03-24T21:49:49.777 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:49 smithi125 conmon[19936]: +0000 mon.a (mon.0) 265 : audit [INF] from='osd.2 v1:172.21.15.125:6809/246939163' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2021-03-24T21:49:49.778 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:49 smithi125 conmon[19936]: cluster 2021-03-24T21:49:48.519395+0000 2021-03-24T21:49:49.778 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:49 smithi125 conmon[19936]: mon.a (mon.0) 266 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2021-03-24T21:49:49.779 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:49 smithi125 conmon[19936]: audit 2021-03-24T21:49:48. 2021-03-24T21:49:49.779 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:49 smithi125 conmon[19936]: 519911+0000 mon.a (mon.0) 267 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-24T21:49:49.779 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:49 smithi125 conmon[19936]: audit 2021-03-24T21:49:49.779 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:49 smithi125 conmon[19936]: 2021-03-24T21:49:48.521039+0000 mon.a (mon.0) 268 : audit [INF] from='osd.2 v1:172.21.15.125:6809/246939163' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]: dispatch 2021-03-24T21:49:49.779 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:49 smithi125 conmon[19936]: cluster 2021-03-24T21:49:48.601703+0000 mgr.y (mgr.14144) 106 : cluster 2021-03-24T21:49:49.780 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:49 smithi125 conmon[19936]: [DBG] pgmap v88: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-24T21:49:49.862 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:49 smithi142 conmon[20125]: audit 2021-03-24T21:49:48.519140 2021-03-24T21:49:49.863 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:49 smithi142 conmon[20125]: +0000 mon.a (mon.0) 265 : audit [INF] from='osd.2 v1:172.21.15.125:6809/246939163' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2021-03-24T21:49:49.863 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:49 smithi142 conmon[20125]: cluster 2021-03-24T21:49:48.519395+0000 2021-03-24T21:49:49.863 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:49 smithi142 conmon[20125]: mon.a (mon.0) 266 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2021-03-24T21:49:49.863 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:49 smithi142 conmon[20125]: audit 2021-03-24T21:49:49.863 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:49 smithi142 conmon[20125]: 2021-03-24T 2021-03-24T21:49:49.864 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:49 smithi142 conmon[20125]: 21:49:48.519911+0000 mon.a (mon.0) 267 : 2021-03-24T21:49:49.864 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:49 smithi142 conmon[20125]: audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-24T21:49:49.864 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:49 smithi142 conmon[20125]: audit 2021-03-24T21:49:49.864 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:49 smithi142 conmon[20125]: 2021-03-24T21:49: 2021-03-24T21:49:49.865 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:49 smithi142 conmon[20125]: 48.521039+0000 mon.a (mon.0) 268 2021-03-24T21:49:49.865 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:49 smithi142 conmon[20125]: : audit [INF] from='osd.2 v1:172.21.15.125:6809/246939163' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]: dispatch 2021-03-24T21:49:49.865 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:49 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:49:49.865 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:49 smithi142 conmon[20125]: -24T21:49:48.601703+0000 mgr.y (mgr. 2021-03-24T21:49:49.865 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:49 smithi142 conmon[20125]: 14144) 106 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-24T21:49:51.151 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:51 smithi125 conmon[19936]: audit 2021-03-24T21: 2021-03-24T21:49:51.152 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:51 smithi125 conmon[19936]: 49:49.523307+0000 mon.a (mon.0) 269 : audit [INF] from='osd.2 v1:172.21.15.125:6809/246939163' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]': finished 2021-03-24T21:49:51.152 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:51 smithi125 conmon[19936]: cluster 2021-03-24T21:49:49.523568+0000 mon.a ( 2021-03-24T21:49:51.152 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:51 smithi125 conmon[19936]: mon.0) 270 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2021-03-24T21:49:51.152 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:51 smithi125 conmon[19936]: audit 2021-03-24T21:49:51.153 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:51 smithi125 conmon[19936]: 2021-03-24T21:49:49.524934+0000 mon.a (mon 2021-03-24T21:49:51.153 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:51 smithi125 conmon[19936]: .0) 271 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-24T21:49:51.153 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:51 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:49:51.153 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:51 smithi125 conmon[19936]: 03-24T21:49:49.531719+0000 mon.a (mon.0 2021-03-24T21:49:51.153 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:51 smithi125 conmon[19936]: ) 272 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-24T21:49:51.154 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:51 smithi125 conmon[24927]: audit 2021- 2021-03-24T21:49:51.154 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:51 smithi125 conmon[24927]: 03-24T21:49:49.523307+0000 mon.a (mon 2021-03-24T21:49:51.154 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:51 smithi125 conmon[24927]: .0) 269 : audit [INF] 2021-03-24T21:49:51.155 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:51 smithi125 conmon[24927]: from='osd.2 v1:172.21.15.125:6809/246939163' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]': finished 2021-03-24T21:49:51.155 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:51 smithi125 conmon[24927]: cluster 2021-03-24T21:49:51.155 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:51 smithi125 conmon[24927]: 2021-03-24T21: 2021-03-24T21:49:51.155 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:51 smithi125 conmon[24927]: 49:49.523568+0000 2021-03-24T21:49:51.155 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:51 smithi125 conmon[24927]: mon.a (mon.0) 2021-03-24T21:49:51.156 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:51 smithi125 conmon[24927]: 270 : cluster [DBG] 2021-03-24T21:49:51.156 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:51 smithi125 conmon[24927]: osdmap e17: 3 total, 2 up, 3 in 2021-03-24T21:49:51.156 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:51 smithi125 conmon[24927]: audit 2021-03-24T21:49:51.156 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:51 smithi125 conmon[24927]: 2021- 2021-03-24T21:49:51.157 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:51 smithi125 conmon[24927]: 03-24T21 2021-03-24T21:49:51.157 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:51 smithi125 conmon[24927]: :49:49 2021-03-24T21:49:51.157 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:51 smithi125 conmon[24927]: .524934+0000 mon.a 2021-03-24T21:49:51.157 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:51 smithi125 conmon[24927]: (mon.0 2021-03-24T21:49:51.158 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:51 smithi125 conmon[24927]: ) 271 : audit 2021-03-24T21:49:51.158 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:51 smithi125 conmon[24927]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-24T21:49:51.158 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:51 smithi125 conmon[24927]: audit 2021-03-24 2021-03-24T21:49:51.158 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:51 smithi125 conmon[24927]: T21:49:49.531719+0000 mon.a (mon.0) 272 : 2021-03-24T21:49:51.159 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:51 smithi125 conmon[24927]: audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-24T21:49:51.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:51 smithi142 conmon[20125]: audit 2021-03-24T21:49:49 2021-03-24T21:49:51.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:51 smithi142 conmon[20125]: .523307+0000 mon.a (mon.0) 269 : audit [INF] from='osd.2 v1:172.21.15.125:6809/246939163' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]': finished 2021-03-24T21:49:51.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:51 smithi142 conmon[20125]: cluster 2021-03-24T21:49:49.523568+0000 mon.a ( 2021-03-24T21:49:51.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:51 smithi142 conmon[20125]: mon.0) 270 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2021-03-24T21:49:51.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:51 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:49:51.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:51 smithi142 conmon[20125]: 03-24T21:49:49.524934+0000 mon.a (mon.0) 271 : audit 2021-03-24T21:49:51.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:51 smithi142 conmon[20125]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-24T21:49:51.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:51 smithi142 conmon[20125]: audit 2021-03-24T21:49:51.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:51 smithi142 conmon[20125]: 2021-03-24T21:49:49.531719+0000 mon.a (mon.0) 272 : 2021-03-24T21:49:51.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:51 smithi142 conmon[20125]: audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-24T21:49:52.152 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[19936]: cluster 2021-03-24T21:49:48.988809+0000 2021-03-24T21:49:52.153 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[19936]: osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:49:52.153 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[19936]: cluster 2021-03-24T21:49:48.989041+0000 2021-03-24T21:49:52.153 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[19936]: osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:49:52.153 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[19936]: cluster 2021-03-24T21:49: 2021-03-24T21:49:52.153 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[19936]: 50.537173+0000 mon.a (mon.0) 273 : cluster 2021-03-24T21:49:52.154 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[19936]: [INF] osd.2 v1:172.21.15.125:6809/246939163 boot 2021-03-24T21:49:52.154 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:49:52.154 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[19936]: :49:50.537252+0000 mon.a (mon.0) 274 : 2021-03-24T21:49:52.154 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[19936]: cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2021-03-24T21:49:52.155 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:49:52.155 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[19936]: 03-24T21:49:50.537851+0000 mon.a (mon 2021-03-24T21:49:52.155 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[19936]: .0) 275 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-24T21:49:52.155 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[19936]: audit 2021 2021-03-24T21:49:52.156 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[19936]: -03-24T21:49:50.538847+0000 mon.a (mon 2021-03-24T21:49:52.156 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[19936]: .0) 276 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]: dispatch 2021-03-24T21:49:52.156 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:49:52.156 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[19936]: 03-24T21:49:50.602209+0000 mgr.y (mgr 2021-03-24T21:49:52.156 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[19936]: .14144) 107 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-24T21:49:52.157 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:49:52.157 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[19936]: 03-24T21:49:51.343063+0000 mgr.y (mgr 2021-03-24T21:49:52.157 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[19936]: .14144) 108 : audit [DBG] from='client.34196 v1:172.21.15.125:0/300549484' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi125:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:49:52.157 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[19936]: audit 2021 2021-03-24T21:49:52.158 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[19936]: -03-24T21:49:51.535423+0000 mon.a ( 2021-03-24T21:49:52.158 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[19936]: mon.0) 277 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]': finished 2021-03-24T21:49:52.158 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:49:52.158 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[19936]: 03-24T21:49:51.535517+0000 mon.a (mon.0) 278 2021-03-24T21:49:52.158 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[19936]: : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2021-03-24T21:49:52.159 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[19936]: audit 2021-03-24T21 2021-03-24T21:49:52.159 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[19936]: :49:51.537792+0000 mon.a (mon.0) 279 : audit 2021-03-24T21:49:52.159 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[19936]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2021-03-24T21:49:52.159 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: cluster 2021-03-24T21:49 2021-03-24T21:49:52.160 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: :48.988809+0000 osd.2 (osd.2) 1 : cluster [DBG] 2021-03-24T21:49:52.160 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: purged_snaps scrub starts 2021-03-24T21:49:52.160 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:49:52.160 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: 03-24T21:49:48. 2021-03-24T21:49:52.161 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: 989041+0000 osd.2 (osd. 2021-03-24T21:49:52.161 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: 2) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:49:52.161 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: 2021-03-24T21:49:52.161 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: cluster 2021-03-24T21:49:52.162 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: 2021-03-24T21:49:50 2021-03-24T21:49:52.162 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: .537173+0000 mon.a 2021-03-24T21:49:52.162 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: (mon.0) 2021-03-24T21:49:52.162 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: 273 : cluster 2021-03-24T21:49:52.163 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: [INF] osd.2 v1:172.21.15.125:6809/246939163 boot 2021-03-24T21:49:52.163 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:49:52.163 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: -03-24T21:49:50.537252+0000 mon.a (mon.0) 274 2021-03-24T21:49:52.163 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2021-03-24T21:49:52.163 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: audit 2021-03-24T21:49:52.164 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: 2021-03-24T21:49:50 2021-03-24T21:49:52.164 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: .537851+0000 mon.a (mon 2021-03-24T21:49:52.164 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: .0) 275 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-24T21:49:52.164 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: 2021-03-24T21:49:52.165 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: audit 2021-03-24 2021-03-24T21:49:52.165 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: T21:49:50.538847 2021-03-24T21:49:52.165 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: +0000 mon.a (mon.0 2021-03-24T21:49:52.165 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: ) 276 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]: dispatch 2021-03-24T21:49:52.165 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: 2021-03-24T21:49:52.166 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: cluster 2021-03-24T21:49:52.166 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: 2021-03- 2021-03-24T21:49:52.166 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: 24T21:49:50 2021-03-24T21:49:52.166 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: .602209+0000 mgr.y (mgr. 2021-03-24T21:49:52.167 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: 14144) 107 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-24T21:49:52.167 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: audit 2021-03-24T21:49:52.167 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: 2021-03-24T 2021-03-24T21:49:52.167 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: 21:49:51.343063 2021-03-24T21:49:52.167 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: +0000 mgr.y (mgr.14144) 108 2021-03-24T21:49:52.168 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: : audit [DBG] from='client.34196 v1:172.21.15.125:0/300549484' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi125:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:49:52.168 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: audit 2021-03-24T21:49:52.168 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: 2021-03-24T21:49:51 2021-03-24T21:49:52.168 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: .535423+0000 mon.a (mon.0) 2021-03-24T21:49:52.168 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: 277 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]': finished 2021-03-24T21:49:52.169 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: cluster 2021-03-24T21:49:52.169 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: 2021-03 2021-03-24T21:49:52.169 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: -24T21:49:51 2021-03-24T21:49:52.169 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: .535517+0000 mon.a ( 2021-03-24T21:49:52.170 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: mon.0) 278 2021-03-24T21:49:52.170 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: : cluster [DBG] 2021-03-24T21:49:52.170 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: osdmap e19: 3 total, 3 up, 3 in 2021-03-24T21:49:52.170 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: audit 2021 2021-03-24T21:49:52.170 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: -03-24T21:49 2021-03-24T21:49:52.171 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: :51.537792+0000 2021-03-24T21:49:52.171 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: mon.a (mon. 2021-03-24T21:49:52.171 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: 0) 279 : audit 2021-03-24T21:49:52.171 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:52 smithi125 conmon[24927]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2021-03-24T21:49:52.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:52 smithi142 conmon[20125]: cluster 2021-03-24T21:49:48.988809+0000 osd.2 (osd.2) 1 : cluster 2021-03-24T21:49:52.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:52 smithi142 conmon[20125]: [DBG] purged_snaps scrub starts 2021-03-24T21:49:52.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:52 smithi142 conmon[20125]: cluster 2021-03-24T21:49:48.989041+0000 osd.2 (osd.2) 2 : 2021-03-24T21:49:52.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:52 smithi142 conmon[20125]: cluster [DBG] purged_snaps scrub ok 2021-03-24T21:49:52.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:52 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:49:52.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:52 smithi142 conmon[20125]: 03-24T21:49:50.537173+0000 mon.a (mon. 2021-03-24T21:49:52.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:52 smithi142 conmon[20125]: 0) 273 : cluster [INF] osd.2 v1:172.21.15.125:6809/246939163 boot 2021-03-24T21:49:52.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:52 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:49:52.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:52 smithi142 conmon[20125]: 24T21:49:50.537252+0000 mon.a 2021-03-24T21:49:52.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:52 smithi142 conmon[20125]: (mon.0) 274 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2021-03-24T21:49:52.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:52 smithi142 conmon[20125]: 2021-03-24T21:49:52.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:52 smithi142 conmon[20125]: audit 2021 2021-03-24T21:49:52.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:52 smithi142 conmon[20125]: -03-24T21:49:50.537851+0000 mon.a ( 2021-03-24T21:49:52.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:52 smithi142 conmon[20125]: mon.0) 275 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-24T21:49:52.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:52 smithi142 conmon[20125]: audit 2021-03 2021-03-24T21:49:52.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:52 smithi142 conmon[20125]: -24T21:49:50.538847+0000 2021-03-24T21:49:52.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:52 smithi142 conmon[20125]: mon.a (mon.0) 276 : audit [INF] 2021-03-24T21:49:52.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:52 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]: dispatch 2021-03-24T21:49:52.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:52 smithi142 conmon[20125]: cluster 2021-03-24T21:49:52.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:52 smithi142 conmon[20125]: 2021-03-24T21:49:50.602209+0000 mgr.y 2021-03-24T21:49:52.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:52 smithi142 conmon[20125]: (mgr.14144) 107 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-24T21:49:52.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:52 smithi142 conmon[20125]: 2021-03-24T21:49:52.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:52 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:49:52.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:52 smithi142 conmon[20125]: 03-24T21:49:51.343063+0000 mgr.y (mgr 2021-03-24T21:49:52.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:52 smithi142 conmon[20125]: .14144) 108 : audit [DBG] from='client.34196 v1:172.21.15.125:0/300549484' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi125:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:49:52.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:52 smithi142 conmon[20125]: audit 2021-03- 2021-03-24T21:49:52.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:52 smithi142 conmon[20125]: 24T21:49:51.535423+0000 mon.a 2021-03-24T21:49:52.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:52 smithi142 conmon[20125]: (mon.0) 277 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]': finished 2021-03-24T21:49:52.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:52 smithi142 conmon[20125]: 2021-03-24T21:49:52.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:52 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:49:52.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:52 smithi142 conmon[20125]: 03-24T21:49:51.535517+0000 mon.a (mon 2021-03-24T21:49:52.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:52 smithi142 conmon[20125]: .0) 278 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2021-03-24T21:49:52.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:52 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:49:52.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:52 smithi142 conmon[20125]: 03-24T21:49:51.537792+0000 2021-03-24T21:49:52.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:52 smithi142 conmon[20125]: mon.a (mon.0) 279 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2021-03-24T21:49:53.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[24927]: audit 2021-03-24T21:49:52.543864+0000 2021-03-24T21:49:53.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[24927]: mon.a (mon.0) 280 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2021-03-24T21:49:53.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[24927]: cluster 2021-03-24T21:49:52. 2021-03-24T21:49:53.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[24927]: 544118+0000 mon.a (mon.0) 281 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2021-03-24T21:49:53.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[24927]: audit 2021-03-24T21: 2021-03-24T21:49:53.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[24927]: 49:52.549477+0000 mon.a (mon.0) 282 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-24T21:49:53.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[24927]: cephadm 2021-03-24T21:49:52.552235+0000 2021-03-24T21:49:53.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[24927]: mgr.y (mgr.14144) 109 : cephadm [INF] Applying service osd.None on host smithi125... 2021-03-24T21:49:53.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[24927]: audit 2021-03-24T21:49:53.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[24927]: 2021-03-24T21:49:52.552761+0000 mon.a ( 2021-03-24T21:49:53.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[24927]: mon.0) 283 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-24T21:49:53.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[24927]: audit 2021-03-24T21:49 2021-03-24T21:49:53.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[24927]: :52.553754+0000 mon.a (mon.0) 284 : 2021-03-24T21:49:53.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[24927]: audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:49:53.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[24927]: cluster 2021-03-24T21:49:53.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[24927]: 2021-03-24T21:49:52.602790+0000 mgr.y (mgr 2021-03-24T21:49:53.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[24927]: .14144) 110 : cluster [DBG] pgmap v94: 1 pgs: 1 unknown; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-03-24T21:49:53.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[24927]: cephadm 2021-03- 2021-03-24T21:49:53.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[24927]: 24T21:49:52.931181+0000 mgr.y (mgr.14144 2021-03-24T21:49:53.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[24927]: ) 111 : cephadm [INF] Detected new or changed devices on smithi125 2021-03-24T21:49:53.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[24927]: audit 2021-03-24T21:49:52 2021-03-24T21:49:53.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[24927]: .940985+0000 mon.a (mon.0) 285 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:53.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[24927]: audit 2021-03-24T21:49:53.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[24927]: 2021-03-24T 2021-03-24T21:49:53.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[24927]: 21:49:53.264600+0000 mon.a (mon 2021-03-24T21:49:53.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[24927]: .0) 286 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:53.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[24927]: audit 2021-03-24T21:49:53.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[24927]: 2021-03-24T21:49:53. 2021-03-24T21:49:53.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[24927]: 267163+0000 mon.a (mon.0) 287 : audit [DBG] 2021-03-24T21:49:53.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[24927]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:49:53.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[24927]: audit 2021-03-24T21:49:53.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[24927]: 2021-03-24T21:49:53.275387+0000 mon.a (mon.0 2021-03-24T21:49:53.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[24927]: ) 288 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:53.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: audit 2021-03-24T21:49: 2021-03-24T21:49:53.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: 52.543864+0000 mon.a (mon.0) 280 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2021-03-24T21:49:53.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:49:53.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: :49:52.544118+0000 mon.a (mon.0) 281 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2021-03-24T21:49:53.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: audit 2021-03-24T21:49:53.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: 2021-03-24T21:49: 2021-03-24T21:49:53.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: 52.549477+0000 mon.a ( 2021-03-24T21:49:53.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: mon.0) 282 : audit 2021-03-24T21:49:53.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-24T21:49:53.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: cephadm 2021-03-24T21:49:53.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: 2021-03-24T21: 2021-03-24T21:49:53.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: 49:52.552235+0000 mgr.y 2021-03-24T21:49:53.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: (mgr.14144) 109 : cephadm [INF] 2021-03-24T21:49:53.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: Applying service osd.None on host smithi125... 2021-03-24T21:49:53.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: audit 2021-03-24T21:49:53.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: 2021-03-24T21:49:52.552761 2021-03-24T21:49:53.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: +0000 mon.a (mon.0) 2021-03-24T21:49:53.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: 283 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-24T21:49:53.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: 2021-03-24T21:49:53.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: audit 2021-03-24T21:49:53.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: 2021-03-24T21: 2021-03-24T21:49:53.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: 49:52.553754+0000 2021-03-24T21:49:53.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: mon.a (mon.0) 284 : audit 2021-03-24T21:49:53.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:49:53.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: cluster 2021-03-24T21:49:53.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: 2021-03-24T21:49:52.602790+0000 2021-03-24T21:49:53.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: mgr.y (mgr.14144) 110 : cluster [DBG] 2021-03-24T21:49:53.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: pgmap v94: 1 pgs: 1 unknown; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-03-24T21:49:53.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: cephadm 2021-03-24T21 2021-03-24T21:49:53.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: :49:52.931181+0000 mgr.y (mgr 2021-03-24T21:49:53.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: .14144) 111 : cephadm [INF] Detected new or changed devices on smithi125 2021-03-24T21:49:53.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:49:53.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: 03-24T21: 2021-03-24T21:49:53.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: 49:52.940985 2021-03-24T21:49:53.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: +0000 mon.a (mon.0) 285 : audit [INF] 2021-03-24T21:49:53.669 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:53.669 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: audit 2021-03-24T21:49:53.669 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: 2021-03-24T 2021-03-24T21:49:53.669 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: 21:49:53 2021-03-24T21:49:53.670 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: .264600+0000 mon.a 2021-03-24T21:49:53.670 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: (mon.0) 2021-03-24T21:49:53.670 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: 286 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:53.670 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: 2021-03-24T21:49:53.670 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: audit 2021-03-24T21:49:53.671 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: 2021-03- 2021-03-24T21:49:53.671 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: 24T21:49:53 2021-03-24T21:49:53.671 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: .267163+0000 mon.a ( 2021-03-24T21:49:53.671 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: mon.0) 287 2021-03-24T21:49:53.671 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: : audit [DBG] 2021-03-24T21:49:53.672 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:49:53.672 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: audit 2021-03-24T21:49:53.672 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: 2021-03-24T21 2021-03-24T21:49:53.672 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: :49:53.275387+0000 2021-03-24T21:49:53.673 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: mon.a (mon.0 2021-03-24T21:49:53.673 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: ) 288 : audit [INF] 2021-03-24T21:49:53.673 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:53 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:53.862 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:53 smithi142 conmon[20125]: audit 2021-03-24T21:49:52.543864 2021-03-24T21:49:53.863 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:53 smithi142 conmon[20125]: +0000 mon.a (mon.0) 280 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2021-03-24T21:49:53.863 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:53 smithi142 conmon[20125]: cluster 2021-03-24T21:49:52.544118+0000 mon.a (mon.0) 2021-03-24T21:49:53.863 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:53 smithi142 conmon[20125]: 281 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2021-03-24T21:49:53.864 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:53 smithi142 conmon[20125]: audit 2021-03-24T21 2021-03-24T21:49:53.864 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:53 smithi142 conmon[20125]: :49:52.549477+0000 mon.a (mon.0) 282 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-24T21:49:53.864 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:53 smithi142 conmon[20125]: cephadm 2021-03-24T21:49:52.552235+0000 mgr.y (mgr 2021-03-24T21:49:53.864 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:53 smithi142 conmon[20125]: .14144) 109 : cephadm [INF] Applying service osd.None on host smithi125... 2021-03-24T21:49:53.865 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:53 smithi142 conmon[20125]: audit 2021-03-24T21:49:52.552761 2021-03-24T21:49:53.865 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:53 smithi142 conmon[20125]: +0000 mon.a (mon.0) 283 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-24T21:49:53.865 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:53 smithi142 conmon[20125]: audit 2021-03-24T21: 2021-03-24T21:49:53.865 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:53 smithi142 conmon[20125]: 49:52.553754+0000 mon.a (mon.0) 284 : audit 2021-03-24T21:49:53.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:53 smithi142 conmon[20125]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:49:53.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:53 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:49:53.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:53 smithi142 conmon[20125]: 03-24T21:49:52.602790+0000 mgr.y (mgr.14144) 110 : cluster 2021-03-24T21:49:53.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:53 smithi142 conmon[20125]: [DBG] pgmap v94: 1 pgs: 1 unknown; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-03-24T21:49:53.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:53 smithi142 conmon[20125]: cephadm 2021- 2021-03-24T21:49:53.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:53 smithi142 conmon[20125]: 03-24T21:49:52.931181+0000 mgr.y (mgr.14144) 111 : cephadm 2021-03-24T21:49:53.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:53 smithi142 conmon[20125]: [INF] Detected new or changed devices on smithi125 2021-03-24T21:49:53.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:53 smithi142 conmon[20125]: audit 2021 2021-03-24T21:49:53.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:53 smithi142 conmon[20125]: -03-24T21:49:52.940985+0000 mon.a (mon.0) 285 : 2021-03-24T21:49:53.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:53 smithi142 conmon[20125]: audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:53.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:53 smithi142 conmon[20125]: audit 2021 2021-03-24T21:49:53.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:53 smithi142 conmon[20125]: -03-24T21:49:53.264600+0000 mon.a (mon.0) 286 : 2021-03-24T21:49:53.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:53 smithi142 conmon[20125]: audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:53.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:53 smithi142 conmon[20125]: audit 2021 2021-03-24T21:49:53.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:53 smithi142 conmon[20125]: -03-24T21:49:53.267163+0000 mon.a (mon.0) 287 : 2021-03-24T21:49:53.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:53 smithi142 conmon[20125]: audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:49:53.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:53 smithi142 conmon[20125]: audit 2021 2021-03-24T21:49:53.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:53 smithi142 conmon[20125]: -03-24T21:49:53.275387+0000 mon.a (mon.0) 288 : audit 2021-03-24T21:49:53.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:53 smithi142 conmon[20125]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:49:55.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:55 smithi142 conmon[20125]: cluster 2021-03-24T21:49:54.283276+0000 mon.a (mon.0) 289 : cluster [DBG] 2021-03-24T21:49:55.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:55 smithi142 conmon[20125]: osdmap e21: 3 total, 3 up, 3 in 2021-03-24T21:49:55.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:55 smithi142 conmon[20125]: audit 2021-03-24T21:49:55.054116+0000 mon.a (mon.0) 290 : audit [INF] from='client.? v1:172.21.15.125:0/2064061112' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "716a41ce-7a28-400f-bf75-4c1ad99c7f94"}]: dispatch 2021-03-24T21:49:55.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:55 smithi142 conmon[20125]: 2021-03-24T21:49:55.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:55 smithi142 conmon[20125]: audit 2021-03-24T21 2021-03-24T21:49:55.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:55 smithi142 conmon[20125]: :49:55.061413+0000 mon.a (mon.0) 291 : 2021-03-24T21:49:55.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:55 smithi142 conmon[20125]: audit [INF] from='client.? v1:172.21.15.125:0/2064061112' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "716a41ce-7a28-400f-bf75-4c1ad99c7f94"}]': finished 2021-03-24T21:49:55.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:55 smithi142 conmon[20125]: cluster 2021-03-24T21:49: 2021-03-24T21:49:55.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:55 smithi142 conmon[20125]: 55.061477+0000 mon.a (mon.0) 292 2021-03-24T21:49:55.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:55 smithi142 conmon[20125]: : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2021-03-24T21:49:55.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:55 smithi142 conmon[20125]: audit 2021-03-24T21 2021-03-24T21:49:55.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:55 smithi142 conmon[20125]: :49:55.061694+0000 mon.a (mon. 2021-03-24T21:49:55.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:55 smithi142 conmon[20125]: 0) 293 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-03-24T21:49:55.652 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:55 smithi125 conmon[19936]: cluster 2021-03-24T21:49:54. 2021-03-24T21:49:55.652 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:55 smithi125 conmon[19936]: 283276+0000 mon.a (mon.0) 289 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2021-03-24T21:49:55.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:55 smithi125 conmon[19936]: audit 2021-03-24T 2021-03-24T21:49:55.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:55 smithi125 conmon[19936]: 21:49:55.054116+0000 mon.a (mon.0) 290 : audit [INF] from='client.? v1:172.21.15.125:0/2064061112' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "716a41ce-7a28-400f-bf75-4c1ad99c7f94"}]: dispatch 2021-03-24T21:49:55.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:55 smithi125 conmon[19936]: audit 2021-03-24T21:49:55.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:55 smithi125 conmon[19936]: 2021-03-24T21:49:55.061413+0000 2021-03-24T21:49:55.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:55 smithi125 conmon[19936]: mon.a (mon.0) 291 : audit [INF] from='client.? v1:172.21.15.125:0/2064061112' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "716a41ce-7a28-400f-bf75-4c1ad99c7f94"}]': finished 2021-03-24T21:49:55.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:55 smithi125 conmon[19936]: 2021-03-24T21:49:55.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:55 smithi125 conmon[19936]: cluster 2021-03-24T21:49: 2021-03-24T21:49:55.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:55 smithi125 conmon[19936]: 55.061477+0000 mon.a (mon.0) 292 : 2021-03-24T21:49:55.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:55 smithi125 conmon[19936]: cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2021-03-24T21:49:55.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:55 smithi125 conmon[19936]: audit 2021-03-24 2021-03-24T21:49:55.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:55 smithi125 conmon[19936]: T21:49:55.061694+0000 mon.a (mon. 2021-03-24T21:49:55.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:55 smithi125 conmon[19936]: 0) 293 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-03-24T21:49:55.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:55 smithi125 conmon[24927]: cluster 2021-03-24T21:49:54 2021-03-24T21:49:55.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:55 smithi125 conmon[24927]: .283276+0000 mon.a (mon.0) 289 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2021-03-24T21:49:55.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:55 smithi125 conmon[24927]: audit 2021-03-24T21:49:55.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:55 smithi125 conmon[24927]: 2021-03-24T21:49:55.054116+0000 mon.a 2021-03-24T21:49:55.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:55 smithi125 conmon[24927]: (mon.0) 290 : audit [INF] from='client.? v1:172.21.15.125:0/2064061112' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "716a41ce-7a28-400f-bf75-4c1ad99c7f94"}]: dispatch 2021-03-24T21:49:55.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:55 smithi125 conmon[24927]: audit 2021-03-24T21:49:55.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:55 smithi125 conmon[24927]: 2021-03-24T21:49 2021-03-24T21:49:55.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:55 smithi125 conmon[24927]: :55.061413+0000 mon.a (mon.0) 291 : 2021-03-24T21:49:55.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:55 smithi125 conmon[24927]: audit [INF] from='client.? v1:172.21.15.125:0/2064061112' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "716a41ce-7a28-400f-bf75-4c1ad99c7f94"}]': finished 2021-03-24T21:49:55.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:55 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:49:55.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:55 smithi125 conmon[24927]: 03-24T21:49:55.061477+0000 mon.a ( 2021-03-24T21:49:55.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:55 smithi125 conmon[24927]: mon.0) 292 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2021-03-24T21:49:55.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:55 smithi125 conmon[24927]: audit 2021-03-24T21:49:55.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:55 smithi125 conmon[24927]: 2021-03-24T21:49:55.061694+0000 mon.a ( 2021-03-24T21:49:55.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:55 smithi125 conmon[24927]: mon.0) 293 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-03-24T21:49:56.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:56 smithi142 conmon[20125]: cluster 2021-03-24T21:49:54.603222+0000 mgr.y (mgr.14144) 112 : 2021-03-24T21:49:56.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:56 smithi142 conmon[20125]: cluster [DBG] pgmap v96: 1 pgs: 1 unknown; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-03-24T21:49:56.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:56 smithi142 conmon[20125]: audit 2021-03-24T21:49:55.703352+0000 mon.c (mon.2) 6 : audit 2021-03-24T21:49:56.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:56 smithi142 conmon[20125]: [DBG] from='client.? v1:172.21.15.125:0/29058032' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-24T21:49:56.651 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:56 smithi125 conmon[19936]: cluster 2021-03-24T21:49:54.603222+0000 2021-03-24T21:49:56.652 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:56 smithi125 conmon[19936]: mgr.y (mgr.14144) 112 : cluster [DBG] pgmap v96: 1 pgs: 1 unknown; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-03-24T21:49:56.652 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:56 smithi125 conmon[19936]: audit 2021-03-24T21:49:55.703352+0000 2021-03-24T21:49:56.652 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:56 smithi125 conmon[19936]: mon.c (mon.2) 6 : audit [DBG] from='client.? v1:172.21.15.125:0/29058032' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-24T21:49:56.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:56 smithi125 conmon[24927]: cluster 2021-03-24T21:49:54.603222+0000 mgr.y (mgr.14144) 112 : 2021-03-24T21:49:56.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:56 smithi125 conmon[24927]: cluster [DBG] pgmap v96: 1 pgs: 1 unknown; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-03-24T21:49:56.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:56 smithi125 conmon[24927]: audit 2021-03-24T21:49: 2021-03-24T21:49:56.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:56 smithi125 conmon[24927]: 55.703352+0000 mon.c (mon.2) 6 : audit [DBG] from='client.? v1:172.21.15.125:0/29058032' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-24T21:49:56.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:56 smithi125 conmon[24927]: 2021-03-24T21:49:58.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:58 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:49:58.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:49:58 smithi125 conmon[19936]: 21:49:56.603718+0000 mgr.y (mgr.14144) 113 : cluster [DBG] pgmap v98: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-03-24T21:49:58.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:58 smithi125 conmon[24927]: cluster 2021-03-24T21:49:58.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:58 smithi125 conmon[24927]: 2021-03 2021-03-24T21:49:58.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:49:58 smithi125 conmon[24927]: -24T21:49:56.603718+0000 mgr.y (mgr.14144) 113 : cluster [DBG] pgmap v98: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-03-24T21:49:58.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:58 smithi142 conmon[20125]: cluster 2021-03-24T21:49:56.603718+0000 mgr.y (mgr.14144) 113 2021-03-24T21:49:58.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:49:58 smithi142 conmon[20125]: : cluster [DBG] pgmap v98: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-03-24T21:50:00.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:00 smithi142 conmon[20125]: cluster 2021-03-24T21:49:58.604227+0000 mgr.y (mgr.14144) 2021-03-24T21:50:00.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:00 smithi142 conmon[20125]: 114 : cluster [DBG] pgmap v99: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-03-24T21:50:00.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:00 smithi142 conmon[20125]: cluster 2021-03-24T21:50:00.000141+0000 2021-03-24T21:50:00.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:00 smithi142 conmon[20125]: mon.a (mon.0) 294 : cluster [INF] overall HEALTH_OK 2021-03-24T21:50:00.651 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:00 smithi125 conmon[24927]: cluster 2021-03-24T21:49:58. 2021-03-24T21:50:00.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:00 smithi125 conmon[24927]: 604227+0000 mgr.y (mgr.14144) 114 : cluster [DBG] pgmap v99: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-03-24T21:50:00.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:00 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:50:00.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:00 smithi125 conmon[24927]: -24T21:50:00.000141+0000 mon.a (mon.0) 294 : cluster [INF] 2021-03-24T21:50:00.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:00 smithi125 conmon[24927]: overall HEALTH_OK 2021-03-24T21:50:00.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:00 smithi125 conmon[19936]: cluster 2021-03-24T21:49:58 2021-03-24T21:50:00.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:00 smithi125 conmon[19936]: .604227+0000 mgr.y (mgr.14144) 114 : cluster [DBG] pgmap v99: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-03-24T21:50:00.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:00 smithi125 conmon[19936]: cluster 2021-03-24T21:50:00.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:00 smithi125 conmon[19936]: 2021-03-24T21:50:00.000141+0000 mon.a (mon 2021-03-24T21:50:00.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:00 smithi125 conmon[19936]: .0) 294 : cluster [INF] overall HEALTH_OK 2021-03-24T21:50:02.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:02 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:50:02.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:02 smithi125 conmon[24927]: :50:00.604789+0000 mgr.y (mgr.14144) 115 : cluster [DBG] pgmap v100: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-03-24T21:50:02.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:02 smithi125 conmon[24927]: audit 2021 2021-03-24T21:50:02.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:02 smithi125 conmon[24927]: -03-24T21:50:02.188989+0000 2021-03-24T21:50:02.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:02 smithi125 conmon[24927]: mon.a (mon.0) 295 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2021-03-24T21:50:02.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:02 smithi125 conmon[24927]: 2021-03-24T21:50:02.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:02 smithi125 conmon[24927]: audit 2021-03-24T21:50:02 2021-03-24T21:50:02.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:02 smithi125 conmon[24927]: .190695+0000 mon.a (mon.0) 296 : audit 2021-03-24T21:50:02.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:02 smithi125 conmon[24927]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:50:02.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:02 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:50:02.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:02 smithi125 conmon[19936]: 24T21:50:00.604789+0000 mgr.y (mgr.14144) 115 : cluster [DBG] pgmap v100: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-03-24T21:50:02.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:02 smithi125 conmon[19936]: 2021-03-24T21:50:02.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:02 smithi125 conmon[19936]: audit 2021-03-24T21 2021-03-24T21:50:02.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:02 smithi125 conmon[19936]: :50:02.188989+0000 mon.a (mon. 2021-03-24T21:50:02.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:02 smithi125 conmon[19936]: 0) 295 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2021-03-24T21:50:02.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:02 smithi125 conmon[19936]: audit 2021 2021-03-24T21:50:02.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:02 smithi125 conmon[19936]: -03-24T21:50:02.190695 2021-03-24T21:50:02.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:02 smithi125 conmon[19936]: +0000 mon.a (mon.0) 296 : audit 2021-03-24T21:50:02.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:02 smithi125 conmon[19936]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:50:02.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:02 smithi142 conmon[20125]: cluster 2021-03-24T21:50:00.604789+0000 mgr.y (mgr.14144) 115 2021-03-24T21:50:02.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:02 smithi142 conmon[20125]: : cluster [DBG] pgmap v100: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-03-24T21:50:02.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:02 smithi142 conmon[20125]: audit 2021-03-24T21:50:02.188989+0000 mon.a (mon.0) 295 : audit 2021-03-24T21:50:02.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:02 smithi142 conmon[20125]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2021-03-24T21:50:02.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:02 smithi142 conmon[20125]: audit 2021-03-24T21:50:02.190695+0000 mon.a (mon.0) 296 2021-03-24T21:50:02.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:02 smithi142 conmon[20125]: : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:50:03.600 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:03 smithi125 conmon[19936]: cephadm 2021-03-24T21:50:02.192125+0000 mgr.y (mgr.14144 2021-03-24T21:50:03.600 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:03 smithi125 conmon[19936]: ) 116 : cephadm [INF] Deploying daemon osd.3 on smithi125 2021-03-24T21:50:03.601 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:03 smithi125 conmon[24927]: cephadm 2021-03-24T21:50 2021-03-24T21:50:03.601 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:03 smithi125 conmon[24927]: :02.192125+0000 mgr.y (mgr.14144) 116 : cephadm [INF] Deploying daemon osd.3 on smithi125 2021-03-24T21:50:03.611 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:03 smithi142 conmon[20125]: cephadm 2021-03-24T21:50:02.192125+0000 mgr.y (mgr.14144) 2021-03-24T21:50:03.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:03 smithi142 conmon[20125]: 116 : cephadm [INF] Deploying daemon osd.3 on smithi125 2021-03-24T21:50:04.608 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:04 smithi125 conmon[19936]: cluster 2021-03-24T21:50:02. 2021-03-24T21:50:04.609 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:04 smithi125 conmon[19936]: 605209+0000 mgr.y (mgr.14144) 117 : cluster [DBG] pgmap v101: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-03-24T21:50:04.609 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:04 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:50:04.609 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:04 smithi125 conmon[24927]: 21:50:02.605209+0000 mgr.y (mgr.14144) 117 : cluster 2021-03-24T21:50:04.610 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:04 smithi125 conmon[24927]: [DBG] pgmap v101: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-03-24T21:50:04.611 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:04 smithi142 conmon[20125]: cluster 2021-03-24T21 2021-03-24T21:50:04.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:04 smithi142 conmon[20125]: :50:02.605209+0000 mgr.y (mgr.14144) 117 : cluster [DBG] pgmap v101: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-03-24T21:50:05.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:05 smithi142 conmon[20125]: audit 2021-03-24T21: 2021-03-24T21:50:05.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:05 smithi142 conmon[20125]: 50:04.729984+0000 mon.a (mon.0) 297 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T21:50:05.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:05 smithi142 conmon[20125]: audit 2021-03-24T21:50:05.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:05 smithi142 conmon[20125]: 2021-03-24T21:50:04.733762+0000 mon.a (mon.0 2021-03-24T21:50:05.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:05 smithi142 conmon[20125]: ) 298 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T21:50:05.651 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:05 smithi125 conmon[19936]: audit 2021-03-24T21:50:04.729984 2021-03-24T21:50:05.652 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:05 smithi125 conmon[19936]: +0000 mon.a (mon.0) 297 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T21:50:05.652 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:05 smithi125 conmon[19936]: audit 2021-03-24T21:50:04.733762 2021-03-24T21:50:05.652 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:05 smithi125 conmon[19936]: +0000 mon.a (mon.0) 298 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T21:50:05.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:05 smithi125 conmon[24927]: audit 2021-03-24T21:50 2021-03-24T21:50:05.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:05 smithi125 conmon[24927]: :04.729984+0000 mon.a (mon.0) 297 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T21:50:05.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:05 smithi125 conmon[24927]: audit 2021-03-24T21:50:05.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:05 smithi125 conmon[24927]: 2021-03-24T21:50:04.733762+0000 mon.a (mon. 2021-03-24T21:50:05.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:05 smithi125 conmon[24927]: 0) 298 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T21:50:06.435 INFO:teuthology.orchestra.run.smithi125.stdout:Created osd(s) 3 on host 'smithi125' 2021-03-24T21:50:06.508 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:06 smithi125 conmon[19936]: cluster 2021-03-24T21:50:04.605660+0000 2021-03-24T21:50:06.509 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:06 smithi125 conmon[19936]: mgr.y (mgr.14144) 118 : cluster [DBG] pgmap v102: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-03-24T21:50:06.509 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:06 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:50:06.509 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:06 smithi125 conmon[24927]: 50:04.605660+0000 mgr.y (mgr.14144) 118 : cluster [DBG] pgmap v102: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-03-24T21:50:06.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:06 smithi142 conmon[20125]: cluster 2021-03-24T21:50:04.605660+0000 mgr.y (mgr.14144) 2021-03-24T21:50:06.659 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:06 smithi142 conmon[20125]: 118 : cluster [DBG] pgmap v102: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-03-24T21:50:07.259 DEBUG:teuthology.orchestra.run.smithi125:osd.3> sudo journalctl -f -n 0 -u ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c@osd.3.service 2021-03-24T21:50:07.262 INFO:tasks.cephadm:Deploying osd.4 on smithi142 with /dev/vg_nvme/lv_4... 2021-03-24T21:50:07.262 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2021-03-24T21:50:07.294 INFO:journalctl@ceph.osd.3.smithi125.stdout:-- Logs begin at Wed 2021-03-03 23:51:07 UTC. -- 2021-03-24T21:50:07.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:07 smithi142 conmon[20125]: audit 2021-03-24T21:50:06.429337+0000 mon.a (mon.0) 299 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:50:07.642 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:07 smithi142 conmon[20125]: 2021-03-24T21:50:07.643 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:07 smithi142 conmon[20125]: audit 2021-03-24T21:50:06.434570+0000 mon.a (mon.0) 300 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:07.643 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:07 smithi142 conmon[20125]: audit 2021 2021-03-24T21:50:07.643 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:07 smithi142 conmon[20125]: -03-24T21:50:06.441786 2021-03-24T21:50:07.643 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:07 smithi142 conmon[20125]: +0000 mon.a (mon.0) 301 : audit 2021-03-24T21:50:07.643 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:07 smithi142 conmon[20125]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:07.644 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:07 smithi142 conmon[20125]: cluster 2021-03-24T21 2021-03-24T21:50:07.644 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:07 smithi142 conmon[20125]: :50:06.692576+0000 osd.0 (osd. 2021-03-24T21:50:07.644 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:07 smithi142 conmon[20125]: 0) 3 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:50:07.644 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:07 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:50:07.645 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:07 smithi142 conmon[20125]: -24T21:50:06.692747+0000 2021-03-24T21:50:07.645 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:07 smithi142 conmon[20125]: osd.0 (osd.0) 4 : cluster [DBG] 2021-03-24T21:50:07.645 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:07 smithi142 conmon[20125]: purged_snaps scrub ok 2021-03-24T21:50:07.651 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:07 smithi125 conmon[24927]: audit 2021-03-24T21:50:06.429337+0000 mon.a (mon 2021-03-24T21:50:07.651 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:07 smithi125 conmon[24927]: .0) 299 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:50:07.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:07 smithi125 conmon[24927]: audit 2021-03-24T21:50:06 2021-03-24T21:50:07.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:07 smithi125 conmon[24927]: .434570+0000 mon.a (mon.0) 300 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:07.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:07 smithi125 conmon[24927]: audit 2021- 2021-03-24T21:50:07.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:07 smithi125 conmon[24927]: 03-24T21:50:06.441786+0000 mon.a (mon.0) 301 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:07.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:07 smithi125 conmon[24927]: cluster 2021-03-24T21:50:06.692576 2021-03-24T21:50:07.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:07 smithi125 conmon[24927]: +0000 osd.0 (osd.0) 3 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:50:07.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:07 smithi125 conmon[24927]: cluster 2021-03-24T21:50: 2021-03-24T21:50:07.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:07 smithi125 conmon[24927]: 06.692747+0000 osd.0 (osd.0) 4 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:50:07.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:07 smithi125 conmon[19936]: audit 2021-03-24T21:50:06.429337 2021-03-24T21:50:07.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:07 smithi125 conmon[19936]: +0000 mon.a (mon.0) 299 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:50:07.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:07 smithi125 conmon[19936]: audit 2021-03-24T21:50:06.434570+0000 mon.a ( 2021-03-24T21:50:07.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:07 smithi125 conmon[19936]: mon.0) 300 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:07.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:07 smithi125 conmon[19936]: audit 2021-03- 2021-03-24T21:50:07.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:07 smithi125 conmon[19936]: 24T21:50:06.441786+0000 mon.a (mon.0) 301 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:07.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:07 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:50:07.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:07 smithi125 conmon[19936]: 03-24T21:50:06.692576+0000 osd.0 (osd.0) 2021-03-24T21:50:07.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:07 smithi125 conmon[19936]: 3 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:50:07.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:07 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:50:07.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:07 smithi125 conmon[19936]: 24T21:50:06.692747+0000 osd.0 (osd.0) 4 : 2021-03-24T21:50:07.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:07 smithi125 conmon[19936]: cluster [DBG] purged_snaps scrub ok 2021-03-24T21:50:08.847 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:08 smithi142 conmon[20125]: cluster 2021-03-24T21:50:06.606151+0000 mgr.y (mgr.14144) 119 : cluster 2021-03-24T21:50:08.847 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:08 smithi142 conmon[20125]: [DBG] pgmap v103: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-03-24T21:50:08.847 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:08 smithi142 conmon[20125]: cluster 2021-03-24T21:50:07.449925+0000 mon.a (mon. 2021-03-24T21:50:08.847 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:08 smithi142 conmon[20125]: 0) 302 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2021-03-24T21:50:08.848 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:08 smithi142 conmon[20125]: audit 2021-03-24T21:50:08.848 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:08 smithi142 conmon[20125]: 2021-03-24T21:50:07 2021-03-24T21:50:08.848 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:08 smithi142 conmon[20125]: .450239+0000 mon.a (mon.0) 303 : 2021-03-24T21:50:08.848 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:08 smithi142 conmon[20125]: audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-03-24T21:50:08.901 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:08 smithi125 conmon[19936]: cluster 2021-03-24T21:50:06.606151+0000 mgr.y (mgr.14144) 119 2021-03-24T21:50:08.901 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:08 smithi125 conmon[19936]: : cluster [DBG] pgmap v103: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-03-24T21:50:08.902 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:08 smithi125 conmon[19936]: cluster 2021-03-24T21:50:07.449925+0000 mon.a (mon.0) 302 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2021-03-24T21:50:08.902 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:08 smithi125 conmon[19936]: audit 2021-03-24T21:50:07.450239+0000 mon.a 2021-03-24T21:50:08.902 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:08 smithi125 conmon[19936]: (mon.0) 303 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-03-24T21:50:08.903 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:08 smithi125 conmon[24927]: cluster 2021-03-24T21:50:06.606151 2021-03-24T21:50:08.903 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:08 smithi125 conmon[24927]: +0000 mgr.y (mgr.14144) 119 : cluster [DBG] pgmap v103: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-03-24T21:50:08.903 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:08 smithi125 conmon[24927]: cluster 2021-03-24T21:50 2021-03-24T21:50:08.903 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:08 smithi125 conmon[24927]: :07.449925+0000 mon.a (mon.0) 302 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2021-03-24T21:50:08.903 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:08 smithi125 conmon[24927]: audit 2021-03-24T21:50:07.450239 2021-03-24T21:50:08.904 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:08 smithi125 conmon[24927]: +0000 mon.a (mon.0) 303 2021-03-24T21:50:08.904 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:08 smithi125 conmon[24927]: : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-03-24T21:50:09.512 INFO:teuthology.orchestra.run.smithi142.stderr:--> Zapping: /dev/vg_nvme/lv_4 2021-03-24T21:50:09.512 INFO:teuthology.orchestra.run.smithi142.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2021-03-24T21:50:09.512 INFO:teuthology.orchestra.run.smithi142.stderr: stderr: 10+0 records in 2021-03-24T21:50:09.512 INFO:teuthology.orchestra.run.smithi142.stderr:10+0 records out 2021-03-24T21:50:09.513 INFO:teuthology.orchestra.run.smithi142.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.020233 s, 518 MB/s 2021-03-24T21:50:09.513 INFO:teuthology.orchestra.run.smithi142.stderr:--> Zapping successful for: 2021-03-24T21:50:09.862 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:09 smithi142 conmon[20125]: cluster 2021-03-24T21:50:08.606554+0000 2021-03-24T21:50:09.862 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:09 smithi142 conmon[20125]: mgr.y (mgr.14144) 120 : cluster [DBG] pgmap v105: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-03-24T21:50:09.862 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:09 smithi142 conmon[20125]: audit 2021-03-24T21:50: 2021-03-24T21:50:09.863 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:09 smithi142 conmon[20125]: 08.709098+0000 mon.a (mon.0) 304 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:09.901 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:09 smithi125 conmon[24927]: cluster 2021-03-24T21:50:08. 2021-03-24T21:50:09.901 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:09 smithi125 conmon[24927]: 606554+0000 mgr.y (mgr.14144) 120 : cluster [DBG] pgmap v105: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-03-24T21:50:09.902 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:09 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T21:50:09.902 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:09 smithi125 conmon[24927]: 24T21:50:08.709098+0000 mon.a (mon 2021-03-24T21:50:09.902 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:09 smithi125 conmon[24927]: .0) 304 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:09.903 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:09 smithi125 conmon[19936]: cluster 2021-03-24T21:50:08 2021-03-24T21:50:09.903 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:09 smithi125 conmon[19936]: .606554+0000 mgr.y (mgr.14144) 120 : cluster [DBG] pgmap v105: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-03-24T21:50:09.903 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:09 smithi125 conmon[19936]: audit 2021-03-24T21:50:08. 2021-03-24T21:50:09.903 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:09 smithi125 conmon[19936]: 709098+0000 mon.a (mon.0) 304 : audit [INF] 2021-03-24T21:50:09.903 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:09 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:10.299 INFO:journalctl@ceph.osd.3.smithi125.stdout:Mar 24 21:50:10 smithi125 conmon[37997]: debug 2021-03-24T21:50:10.044+0000 7f1741f78080 -1 osd.3 0 log_to_monitors {default=true} 2021-03-24T21:50:10.360 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph orch daemon add osd smithi142:vg_nvme/lv_4 2021-03-24T21:50:11.042 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:10 smithi142 conmon[20125]: audit 2021-03-24T21:50:10.056654+0000 2021-03-24T21:50:11.042 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:10 smithi142 conmon[20125]: mon.a (mon.0) 305 : audit [INF] from='osd.3 v1:172.21.15.125:6813/2209636173' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2021-03-24T21:50:11.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:10 smithi125 conmon[19936]: audit 2021-03-24T21:50:10 2021-03-24T21:50:11.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:10 smithi125 conmon[19936]: .056654+0000 mon.a (mon.0) 305 : audit [INF] from='osd.3 v1:172.21.15.125:6813/2209636173' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2021-03-24T21:50:11.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:10 smithi125 conmon[24927]: audit 2021-03-24T21 2021-03-24T21:50:11.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:10 smithi125 conmon[24927]: :50:10.056654+0000 mon.a (mon.0) 305 : audit [INF] from='osd.3 v1:172.21.15.125:6813/2209636173' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2021-03-24T21:50:12.151 INFO:journalctl@ceph.osd.3.smithi125.stdout:Mar 24 21:50:12 smithi125 conmon[37997]: debug 2021-03-24T21:50:12.012+0000 7f1739cec700 -1 osd.3 0 waiting for initial osdmap 2021-03-24T21:50:12.152 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:12 smithi125 conmon[24927]: cluster 2021-03-24T21:50:10.607049+0000 mgr.y (mgr.14144 2021-03-24T21:50:12.152 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:12 smithi125 conmon[24927]: ) 121 : cluster [DBG] pgmap v106: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-03-24T21:50:12.152 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:12 smithi125 conmon[24927]: audit 2021-03-24T21:50: 2021-03-24T21:50:12.152 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:12 smithi125 conmon[24927]: 11.003643+0000 mon.a (mon.0) 306 : audit [INF] from='osd.3 v1:172.21.15.125:6813/2209636173' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2021-03-24T21:50:12.153 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:12 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:50:12.153 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:12 smithi125 conmon[24927]: 21:50:11.003789+0000 mon.a ( 2021-03-24T21:50:12.153 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:12 smithi125 conmon[24927]: mon.0) 307 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2021-03-24T21:50:12.153 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:12 smithi125 conmon[24927]: audit 2021- 2021-03-24T21:50:12.154 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:12 smithi125 conmon[24927]: 03-24T21:50:11.004043+0000 mon.a (mon 2021-03-24T21:50:12.154 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:12 smithi125 conmon[24927]: .0) 308 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-03-24T21:50:12.154 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:12 smithi125 conmon[24927]: 2021-03-24T21:50:12.154 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:12 smithi125 conmon[24927]: audit 2021-03-24T21:50:12.154 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:12 smithi125 conmon[24927]: 2021-03-24T21:50 2021-03-24T21:50:12.155 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:12 smithi125 conmon[24927]: :11.004603+0000 mon.a (mon.0) 309 : 2021-03-24T21:50:12.155 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:12 smithi125 conmon[24927]: audit [INF] from='osd.3 v1:172.21.15.125:6813/2209636173' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]: dispatch 2021-03-24T21:50:12.155 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:12 smithi125 conmon[19936]: cluster 2021-03-24T21:50:10 2021-03-24T21:50:12.155 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:12 smithi125 conmon[19936]: .607049+0000 mgr.y (mgr.14144) 121 : cluster [DBG] pgmap v106: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-03-24T21:50:12.156 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:12 smithi125 conmon[19936]: audit 2021-03-24T21:50:12.156 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:12 smithi125 conmon[19936]: 2021-03-24T21:50:11 2021-03-24T21:50:12.156 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:12 smithi125 conmon[19936]: .003643+0000 mon.a 2021-03-24T21:50:12.156 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:12 smithi125 conmon[19936]: (mon.0) 306 : 2021-03-24T21:50:12.157 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:12 smithi125 conmon[19936]: audit [INF] from='osd.3 v1:172.21.15.125:6813/2209636173' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2021-03-24T21:50:12.157 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:12 smithi125 conmon[19936]: 2021-03-24T21:50:12.157 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:12 smithi125 conmon[19936]: cluster 2021-03-24T21:50:12.157 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:12 smithi125 conmon[19936]: 2021-03-24T21: 2021-03-24T21:50:12.158 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:12 smithi125 conmon[19936]: 50:11.003789+0000 2021-03-24T21:50:12.158 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:12 smithi125 conmon[19936]: mon.a (mon.0) 307 : 2021-03-24T21:50:12.158 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:12 smithi125 conmon[19936]: cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2021-03-24T21:50:12.158 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:12 smithi125 conmon[19936]: audit 2021 2021-03-24T21:50:12.159 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:12 smithi125 conmon[19936]: -03-24T21:50:11.004043+0000 2021-03-24T21:50:12.159 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:12 smithi125 conmon[19936]: mon.a (mon.0) 308 : audit [DBG] 2021-03-24T21:50:12.159 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:12 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-03-24T21:50:12.159 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:12 smithi125 conmon[19936]: audit 2021-03-24T21:50: 2021-03-24T21:50:12.160 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:12 smithi125 conmon[19936]: 11.004603+0000 mon.a (mon.0) 309 : 2021-03-24T21:50:12.160 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:12 smithi125 conmon[19936]: audit [INF] from='osd.3 v1:172.21.15.125:6813/2209636173' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]: dispatch 2021-03-24T21:50:12.361 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:12 smithi142 conmon[20125]: cluster 2021-03-24T21:50:10.607049+0000 mgr.y 2021-03-24T21:50:12.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:12 smithi142 conmon[20125]: (mgr.14144) 121 : cluster [DBG] pgmap v106: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-03-24T21:50:12.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:12 smithi142 conmon[20125]: audit 2021-03-24T21:50:11.003643+0000 mon.a (mon. 2021-03-24T21:50:12.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:12 smithi142 conmon[20125]: 0) 306 : audit [INF] from='osd.3 v1:172.21.15.125:6813/2209636173' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2021-03-24T21:50:12.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:12 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:50:12.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:12 smithi142 conmon[20125]: T21:50:11.003789+0000 mon.a (mon.0) 307 : cluster 2021-03-24T21:50:12.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:12 smithi142 conmon[20125]: [DBG] osdmap e24: 4 total, 3 up, 4 in 2021-03-24T21:50:12.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:12 smithi142 conmon[20125]: audit 2021-03-24T21:50:11.004043 2021-03-24T21:50:12.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:12 smithi142 conmon[20125]: +0000 mon.a (mon.0) 308 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-03-24T21:50:12.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:12 smithi142 conmon[20125]: audit 2021-03 2021-03-24T21:50:12.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:12 smithi142 conmon[20125]: -24T21:50:11.004603+0000 mon.a (mon.0) 309 : audit [INF] from='osd.3 v1:172.21.15.125:6813/2209636173' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]: dispatch 2021-03-24T21:50:13.208 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:13 smithi142 conmon[20125]: audit 2021-03-24T21:50:12.009123+0000 mon.a (mon.0) 310 : 2021-03-24T21:50:13.208 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:13 smithi142 conmon[20125]: audit [INF] from='osd.3 v1:172.21.15.125:6813/2209636173' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]': finished 2021-03-24T21:50:13.208 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:13 smithi142 conmon[20125]: cluster 2021-03-24T21:50:12. 2021-03-24T21:50:13.208 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:13 smithi142 conmon[20125]: 009385+0000 mon.a (mon.0) 311 : cluster [DBG] 2021-03-24T21:50:13.209 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:13 smithi142 conmon[20125]: osdmap e25: 4 total, 3 up, 4 in 2021-03-24T21:50:13.209 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:13 smithi142 conmon[20125]: audit 2021 2021-03-24T21:50:13.209 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:13 smithi142 conmon[20125]: -03-24T21:50:12 2021-03-24T21:50:13.209 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:13 smithi142 conmon[20125]: .010327+0000 mon.a (mon.0 2021-03-24T21:50:13.210 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:13 smithi142 conmon[20125]: ) 312 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-03-24T21:50:13.210 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:13 smithi142 conmon[20125]: 2021-03-24T21:50:13.210 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:13 smithi142 conmon[20125]: audit 2021 2021-03-24T21:50:13.210 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:13 smithi142 conmon[20125]: -03-24T21:50: 2021-03-24T21:50:13.210 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:13 smithi142 conmon[20125]: 12.018671+0000 mon.a (mon 2021-03-24T21:50:13.211 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:13 smithi142 conmon[20125]: .0) 313 : audit [DBG] 2021-03-24T21:50:13.211 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:13 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-03-24T21:50:13.211 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:13 smithi142 conmon[20125]: audit 2021 2021-03-24T21:50:13.211 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:13 smithi142 conmon[20125]: -03-24T21:50 2021-03-24T21:50:13.212 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:13 smithi142 conmon[20125]: :12.498858+0000 mon.a ( 2021-03-24T21:50:13.212 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:13 smithi142 conmon[20125]: mon.0) 314 : audit [DBG] 2021-03-24T21:50:13.212 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:13 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-24T21:50:13.212 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:13 smithi142 conmon[20125]: audit 2021-03-24T21:50:13.212 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:13 smithi142 conmon[20125]: 2021-03-24T21:50 2021-03-24T21:50:13.213 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:13 smithi142 conmon[20125]: :12.502406+0000 mon.a ( 2021-03-24T21:50:13.213 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:13 smithi142 conmon[20125]: mon.0) 315 : audit [INF] 2021-03-24T21:50:13.213 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:13 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-24T21:50:13.213 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:13 smithi142 conmon[20125]: audit 2021-03-24T21:50:13.213 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:13 smithi142 conmon[20125]: 2021-03-24T21:50 2021-03-24T21:50:13.214 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:13 smithi142 conmon[20125]: :12.503696+0000 mon.a (mon 2021-03-24T21:50:13.214 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:13 smithi142 conmon[20125]: .0) 316 : audit [DBG] 2021-03-24T21:50:13.214 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:13 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:50:13.214 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:13 smithi142 conmon[20125]: audit 2021 2021-03-24T21:50:13.214 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:13 smithi142 conmon[20125]: -03-24T21:50: 2021-03-24T21:50:13.215 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:13 smithi142 conmon[20125]: 12.789105+0000 mon.a (mon. 2021-03-24T21:50:13.215 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:13 smithi142 conmon[20125]: 0) 317 : audit [INF] 2021-03-24T21:50:13.215 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:13 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:13.215 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:13 smithi142 conmon[20125]: audit 2021-03-24T21:50:13.215 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:13 smithi142 conmon[20125]: 2021-03-24T21:50: 2021-03-24T21:50:13.216 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:13 smithi142 conmon[20125]: 12.796666+0000 mon.a (mon 2021-03-24T21:50:13.216 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:13 smithi142 conmon[20125]: .0) 318 : audit [INF] 2021-03-24T21:50:13.216 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:13 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:13.216 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:13 smithi142 conmon[20125]: audit 2021-03-24T21:50:13.216 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:13 smithi142 conmon[20125]: 2021-03-24T21: 2021-03-24T21:50:13.217 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:13 smithi142 conmon[20125]: 50:12.798347+0000 mon.a 2021-03-24T21:50:13.217 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:13 smithi142 conmon[20125]: (mon.0) 319 : audit 2021-03-24T21:50:13.217 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:13 smithi142 conmon[20125]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:50:13.217 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:13 smithi142 conmon[20125]: audit 2021 2021-03-24T21:50:13.218 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:13 smithi142 conmon[20125]: -03-24T21:50:12 2021-03-24T21:50:13.218 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:13 smithi142 conmon[20125]: .805618+0000 mon.a (mon.0 2021-03-24T21:50:13.218 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:13 smithi142 conmon[20125]: ) 320 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:13.218 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:13 smithi142 conmon[20125]: 2021-03-24T21:50:13.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[19936]: audit 2021-03-24T21:50:12.009123+0000 mon.a 2021-03-24T21:50:13.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[19936]: (mon.0) 310 : audit [INF] from='osd.3 v1:172.21.15.125:6813/2209636173' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]': finished 2021-03-24T21:50:13.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[19936]: cluster 2021-03-24T21:50:12.009385+0000 mon.a (mon.0) 311 : cluster [DBG] 2021-03-24T21:50:13.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[19936]: osdmap e25: 4 total, 3 up, 4 in 2021-03-24T21:50:13.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[19936]: audit 2021-03-24T21:50:13.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[19936]: 2021-03-24T21:50 2021-03-24T21:50:13.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[19936]: :12.010327+0000 mon.a (mon 2021-03-24T21:50:13.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[19936]: .0) 312 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-03-24T21:50:13.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[19936]: audit 2021 2021-03-24T21:50:13.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[19936]: -03-24T21:50:12 2021-03-24T21:50:13.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[19936]: .018671+0000 mon.a (mon. 2021-03-24T21:50:13.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[19936]: 0) 313 : audit [DBG] 2021-03-24T21:50:13.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-03-24T21:50:13.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[19936]: audit 2021-03-24T21:50:13.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[19936]: 2021-03-24T21 2021-03-24T21:50:13.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[19936]: :50:12.498858+0000 2021-03-24T21:50:13.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[19936]: mon.a (mon.0) 314 : 2021-03-24T21:50:13.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[19936]: audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-24T21:50:13.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[19936]: audit 2021-03-24T21:50:13.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[19936]: 2021-03 2021-03-24T21:50:13.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[19936]: -24T21:50:12 2021-03-24T21:50:13.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[19936]: .502406+0000 mon.a (mon. 2021-03-24T21:50:13.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[19936]: 0) 315 : audit [INF] 2021-03-24T21:50:13.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-24T21:50:13.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[19936]: audit 2021-03-24T21:50:13.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[19936]: 2021-03-24T21: 2021-03-24T21:50:13.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[19936]: 50:12.503696+0000 mon.a 2021-03-24T21:50:13.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[19936]: (mon.0) 316 : audit 2021-03-24T21:50:13.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[19936]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:50:13.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[19936]: audit 2021-03-24T21:50:13.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[19936]: 2021-03-24T21:50: 2021-03-24T21:50:13.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[19936]: 12.789105+0000 mon.a ( 2021-03-24T21:50:13.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[19936]: mon.0) 317 : audit [INF] 2021-03-24T21:50:13.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:13.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[19936]: audit 2021-03-24T21: 2021-03-24T21:50:13.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[19936]: 50:12.796666+0000 mon.a (mon.0) 2021-03-24T21:50:13.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[19936]: 318 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:13.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[19936]: audit 2021 2021-03-24T21:50:13.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[19936]: -03-24T21:50:12.798347+0000 2021-03-24T21:50:13.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[19936]: mon.a (mon.0) 319 : audit [DBG] 2021-03-24T21:50:13.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:50:13.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[19936]: audit 2021-03-24T21:50: 2021-03-24T21:50:13.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[19936]: 12.805618+0000 mon.a (mon.0) 320 : audit 2021-03-24T21:50:13.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[19936]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:13.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[24927]: audit 2021-03-24T21:50:13.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[24927]: 2021-03-24T21:50:13.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[24927]: 2021 2021-03-24T21:50:13.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[24927]: - 2021-03-24T21:50:13.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[24927]: 03 2021-03-24T21:50:13.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[24927]: - 2021-03-24T21:50:13.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[24927]: 24 2021-03-24T21:50:13.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[24927]: T 2021-03-24T21:50:13.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[24927]: 21:50:12. 2021-03-24T21:50:13.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[24927]: 009123+0000 mon.a (mon.0) 310 : audit [INF] from='osd.3 v1:172.21.15.125:6813/2209636173' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]': finished 2021-03-24T21:50:13.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[24927]: cluster 2021-03-24T21:50:13.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[24927]: 2021-03-24T21:50 2021-03-24T21:50:13.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[24927]: :12.009385+0000 mon.a (mon.0) 311 : cluster [DBG] osdmap e25: 4 total, 3 up, 4 in 2021-03-24T21:50:13.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[24927]: audit 2021-03-24T21:50: 2021-03-24T21:50:13.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[24927]: 12.010327+0000 mon.a (mon.0) 312 : audit 2021-03-24T21:50:13.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[24927]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-03-24T21:50:13.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[24927]: audit 2021-03-24T21:50:13.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[24927]: 2021-03-24T21:50:12.018671+0000 mon.a (mon.0) 313 : audit 2021-03-24T21:50:13.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[24927]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-03-24T21:50:13.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[24927]: audit 2021-03-24T21:50:12. 2021-03-24T21:50:13.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[24927]: 498858+0000 mon.a (mon.0) 314 : audit 2021-03-24T21:50:13.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[24927]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-24T21:50:13.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[24927]: audit 2021-03-24T21:50:13.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[24927]: 2021-03-24T21 2021-03-24T21:50:13.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[24927]: :50:12.502406+0000 mon.a (mon.0 2021-03-24T21:50:13.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[24927]: ) 315 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-24T21:50:13.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[24927]: audit 2021-03-24T21:50:13.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[24927]: 2021-03-24T21:50:12. 2021-03-24T21:50:13.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[24927]: 503696+0000 mon.a (mon.0) 316 : audit [DBG] 2021-03-24T21:50:13.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[24927]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:50:13.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[24927]: audit 2021 2021-03-24T21:50:13.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[24927]: -03-24T21:50:12.789105+0000 mon.a (mon.0) 317 2021-03-24T21:50:13.420 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[24927]: : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:13.420 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[24927]: audit 2021-03-24T21:50:12 2021-03-24T21:50:13.420 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[24927]: .796666+0000 mon.a (mon.0) 318 : audit [INF] 2021-03-24T21:50:13.420 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[24927]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:13.420 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[24927]: audit 2021 2021-03-24T21:50:13.421 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[24927]: -03-24T21:50:12.798347+0000 mon.a (mon 2021-03-24T21:50:13.421 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[24927]: .0) 319 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:50:13.421 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[24927]: audit 2021-03-24 2021-03-24T21:50:13.421 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[24927]: T21:50:12.805618+0000 mon.a 2021-03-24T21:50:13.421 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[24927]: (mon.0) 320 : audit [INF] 2021-03-24T21:50:13.422 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:13 smithi125 conmon[24927]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:14.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:14 smithi142 conmon[20125]: cluster 2021-03-24T21:50:11 2021-03-24T21:50:14.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:14 smithi142 conmon[20125]: .025530+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:50:14.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:14 smithi142 conmon[20125]: cluster 2021-03-24T21 2021-03-24T21:50:14.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:14 smithi142 conmon[20125]: :50:11.025692+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:50:14.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:14 smithi142 conmon[20125]: audit 2021-03-24T21:50:14.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:14 smithi142 conmon[20125]: 2021-03-24T21:50:12.497335+0000 2021-03-24T21:50:14.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:14 smithi142 conmon[20125]: mgr.y (mgr.14144) 122 : audit [DBG] from='client.24223 v1:172.21.15.142:0/1779746764' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:50:14.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:14 smithi142 conmon[20125]: cephadm 2021- 2021-03-24T21:50:14.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:14 smithi142 conmon[20125]: 03-24T21:50:12.501877+0000 mgr.y (mgr 2021-03-24T21:50:14.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:14 smithi142 conmon[20125]: .14144) 123 : cephadm [INF] Applying service osd.None on host smithi142... 2021-03-24T21:50:14.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:14 smithi142 conmon[20125]: cluster 2021-03-24T21:50:14.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:14 smithi142 conmon[20125]: 2021-03-24T21:50:12.607487+0000 2021-03-24T21:50:14.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:14 smithi142 conmon[20125]: mgr.y (mgr.14144) 124 : cluster [DBG] pgmap v109: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-03-24T21:50:14.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:14 smithi142 conmon[20125]: 2021-03-24T21:50:14.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:14 smithi142 conmon[20125]: cephadm 2021-03-24T21:50:12. 2021-03-24T21:50:14.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:14 smithi142 conmon[20125]: 778837+0000 mgr.y (mgr.14144) 125 : cephadm 2021-03-24T21:50:14.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:14 smithi142 conmon[20125]: [INF] Detected new or changed devices on smithi125 2021-03-24T21:50:14.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:14 smithi142 conmon[20125]: audit 2021-03-24T21:50 2021-03-24T21:50:14.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:14 smithi142 conmon[20125]: :13.015365+0000 mon.a (mon.0) 321 : 2021-03-24T21:50:14.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:14 smithi142 conmon[20125]: audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-03-24T21:50:14.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:14 smithi142 conmon[20125]: cluster 2021-03-24T21 2021-03-24T21:50:14.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:14 smithi142 conmon[20125]: :50:13.022195+0000 mon.a (mon.0 2021-03-24T21:50:14.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:14 smithi142 conmon[20125]: ) 322 : cluster [INF] osd.3 v1:172.21.15.125:6813/2209636173 boot 2021-03-24T21:50:14.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:14 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:50:14.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:14 smithi142 conmon[20125]: 03-24T21:50:13.022246+0000 mon.a 2021-03-24T21:50:14.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:14 smithi142 conmon[20125]: (mon.0) 323 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2021-03-24T21:50:14.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:14 smithi142 conmon[20125]: audit 2021-03-24T21:50:14.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:14 smithi142 conmon[20125]: 2021-03-24T21:50:13.023215+0000 2021-03-24T21:50:14.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:14 smithi142 conmon[20125]: mon.a (mon.0) 324 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-03-24T21:50:14.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:14 smithi142 conmon[20125]: 2021-03-24T21:50:14.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[24927]: cluster 2021-03-24T21:50:11.025530+0000 2021-03-24T21:50:14.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[24927]: osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:50:14.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[24927]: cluster 2021-03-24T21:50:11.025692+0000 osd.3 (osd. 2021-03-24T21:50:14.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[24927]: 3) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:50:14.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[24927]: audit 2021-03-24T21:50:12 2021-03-24T21:50:14.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[24927]: .497335+0000 mgr.y (mgr.14144) 122 : audit [DBG] from='client.24223 v1:172.21.15.142:0/1779746764' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:50:14.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[24927]: cephadm 2021-03-24T21:50:12.501877+0000 mgr.y (mgr.14144 2021-03-24T21:50:14.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[24927]: ) 123 : cephadm [INF] Applying service osd.None on host smithi142... 2021-03-24T21:50:14.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[24927]: cluster 2021-03-24T21:50 2021-03-24T21:50:14.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[24927]: :12.607487+0000 mgr.y (mgr.14144 2021-03-24T21:50:14.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[24927]: ) 124 : cluster [DBG] pgmap v109: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-03-24T21:50:14.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[24927]: cephadm 2021- 2021-03-24T21:50:14.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[24927]: 03-24T21:50:12.778837+0000 2021-03-24T21:50:14.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[24927]: mgr.y (mgr.14144) 125 : cephadm 2021-03-24T21:50:14.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[24927]: [INF] Detected new or changed devices on smithi125 2021-03-24T21:50:14.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[24927]: audit 2021- 2021-03-24T21:50:14.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[24927]: 03-24T21:50:13.015365 2021-03-24T21:50:14.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[24927]: +0000 mon.a (mon.0) 321 : 2021-03-24T21:50:14.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[24927]: audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-03-24T21:50:14.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[24927]: cluster 2021-03-24T21:50:14.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[24927]: 2021-03-24 2021-03-24T21:50:14.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[24927]: T21:50:13.022195+0000 mon.a (mon.0) 2021-03-24T21:50:14.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[24927]: 322 : cluster [INF] osd.3 v1:172.21.15.125:6813/2209636173 boot 2021-03-24T21:50:14.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:50:14.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[24927]: -03-24T21:50:13. 2021-03-24T21:50:14.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[24927]: 022246+0000 mon.a (mon.0) 323 2021-03-24T21:50:14.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[24927]: : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2021-03-24T21:50:14.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[24927]: audit 2021-03-24T21:50:14.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[24927]: 2021-03 2021-03-24T21:50:14.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[24927]: -24T21:50:13.023215 2021-03-24T21:50:14.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[24927]: +0000 mon.a (mon.0) 324 : audit [DBG] 2021-03-24T21:50:14.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[24927]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-03-24T21:50:14.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[19936]: cluster 2021-03-24T21:50:11. 2021-03-24T21:50:14.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[19936]: 025530+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:50:14.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[19936]: cluster 2021-03-24T21:50:11.025692+0000 osd.3 (osd.3 2021-03-24T21:50:14.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[19936]: ) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:50:14.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[19936]: audit 2021-03- 2021-03-24T21:50:14.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[19936]: 24T21:50:12.497335+0000 mgr.y (mgr.14144) 122 : audit [DBG] from='client.24223 v1:172.21.15.142:0/1779746764' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:50:14.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[19936]: cephadm 2021-03-24 2021-03-24T21:50:14.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[19936]: T21:50:12.501877+0000 mgr.y (mgr.14144) 123 2021-03-24T21:50:14.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[19936]: : cephadm [INF] Applying service osd.None on host smithi142... 2021-03-24T21:50:14.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[19936]: cluster 2021-03-24T21:50:14.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[19936]: 2021-03-24T21:50:12 2021-03-24T21:50:14.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[19936]: .607487+0000 mgr.y (mgr.14144) 124 2021-03-24T21:50:14.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[19936]: : cluster [DBG] pgmap v109: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-03-24T21:50:14.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[19936]: cephadm 2021- 2021-03-24T21:50:14.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[19936]: 03-24T21:50:12 2021-03-24T21:50:14.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[19936]: .778837+0000 mgr.y (mgr.14144) 125 2021-03-24T21:50:14.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[19936]: : cephadm [INF] Detected new or changed devices on smithi125 2021-03-24T21:50:14.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:50:14.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[19936]: 03-24T21:50:13. 2021-03-24T21:50:14.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[19936]: 015365+0000 mon.a (mon.0) 321 : 2021-03-24T21:50:14.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[19936]: audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-03-24T21:50:14.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:50:14.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[19936]: -24T21:50:13. 2021-03-24T21:50:14.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[19936]: 022195+0000 mon.a (mon.0 2021-03-24T21:50:14.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[19936]: ) 322 : cluster [INF] osd.3 v1:172.21.15.125:6813/2209636173 boot 2021-03-24T21:50:14.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[19936]: 2021-03-24T21:50:14.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:50:14.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[19936]: -24T21:50:13.022246+0000 2021-03-24T21:50:14.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[19936]: mon.a (mon.0) 323 : cluster [DBG] 2021-03-24T21:50:14.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[19936]: osdmap e26: 4 total, 4 up, 4 in 2021-03-24T21:50:14.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[19936]: audit 2021-03-24T21:50:14.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[19936]: 2021-03- 2021-03-24T21:50:14.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[19936]: 24T21:50:13.023215 2021-03-24T21:50:14.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[19936]: +0000 mon.a (mon.0) 324 : audit 2021-03-24T21:50:14.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:14 smithi125 conmon[19936]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-03-24T21:50:15.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:15 smithi142 conmon[20125]: cluster 2021-03-24T21:50:14.031191+0000 mon.a (mon.0) 325 : cluster 2021-03-24T21:50:15.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:15 smithi142 conmon[20125]: [DBG] osdmap e27: 4 total, 4 up, 4 in 2021-03-24T21:50:15.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:15 smithi142 conmon[20125]: audit 2021-03-24T21:50:14.922575+0000 mon.b (mon 2021-03-24T21:50:15.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:15 smithi142 conmon[20125]: .1) 7 : audit [INF] from='client.? v1:172.21.15.142:0/2624801269' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f9bdd8d2-b292-49da-ad4f-99682122f543"}]: dispatch 2021-03-24T21:50:15.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:15 smithi142 conmon[20125]: audit 2021-03-24T 2021-03-24T21:50:15.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:15 smithi142 conmon[20125]: 21:50:14.926390+0000 mon.a ( 2021-03-24T21:50:15.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:15 smithi142 conmon[20125]: mon.0) 326 : audit [INF] from='client.24229 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f9bdd8d2-b292-49da-ad4f-99682122f543"}]: dispatch 2021-03-24T21:50:15.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:15 smithi142 conmon[20125]: audit 2021 2021-03-24T21:50:15.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:15 smithi142 conmon[20125]: -03-24T21:50: 2021-03-24T21:50:15.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:15 smithi142 conmon[20125]: 14.935438+0000 mon.a (mon.0) 327 2021-03-24T21:50:15.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:15 smithi142 conmon[20125]: : audit [INF] from='client.24229 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f9bdd8d2-b292-49da-ad4f-99682122f543"}]': finished 2021-03-24T21:50:15.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:15 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:50:15.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:15 smithi142 conmon[20125]: 24T21:50:14.935665+0000 2021-03-24T21:50:15.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:15 smithi142 conmon[20125]: mon.a (mon.0) 328 : cluster [DBG] 2021-03-24T21:50:15.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:15 smithi142 conmon[20125]: osdmap e28: 5 total, 4 up, 5 in 2021-03-24T21:50:15.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:15 smithi142 conmon[20125]: audit 2021-03-24T21:50:15.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:15 smithi142 conmon[20125]: 2021-03-24T21:50: 2021-03-24T21:50:15.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:15 smithi142 conmon[20125]: 14.935916+0000 mon.a (mon.0) 329 2021-03-24T21:50:15.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:15 smithi142 conmon[20125]: : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-03-24T21:50:15.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:15 smithi125 conmon[19936]: cluster 2021-03-24T21:50:14.031191 2021-03-24T21:50:15.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:15 smithi125 conmon[19936]: +0000 mon.a (mon.0) 325 : cluster [DBG] osdmap e27: 4 total, 4 up, 4 in 2021-03-24T21:50:15.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:15 smithi125 conmon[19936]: audit 2021-03-24T21:50:14.922575+0000 mon.b (mon.1 2021-03-24T21:50:15.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:15 smithi125 conmon[19936]: ) 7 : audit [INF] from='client.? v1:172.21.15.142:0/2624801269' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f9bdd8d2-b292-49da-ad4f-99682122f543"}]: dispatch 2021-03-24T21:50:15.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:15 smithi125 conmon[19936]: audit 2021-03-24T21:50:14.926390 2021-03-24T21:50:15.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:15 smithi125 conmon[19936]: +0000 mon.a (mon.0) 326 : audit 2021-03-24T21:50:15.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:15 smithi125 conmon[19936]: [INF] from='client.24229 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f9bdd8d2-b292-49da-ad4f-99682122f543"}]: dispatch 2021-03-24T21:50:15.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:15 smithi125 conmon[19936]: audit 2021-03-24T21:50:15.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:15 smithi125 conmon[19936]: 2021-03-24T21:50:14.935438+0000 mon.a (mon.0 2021-03-24T21:50:15.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:15 smithi125 conmon[19936]: ) 327 : audit [INF] from='client.24229 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f9bdd8d2-b292-49da-ad4f-99682122f543"}]': finished 2021-03-24T21:50:15.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:15 smithi125 conmon[19936]: cluster 2021-03-24T21:50:15.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:15 smithi125 conmon[19936]: 2021-03-24T21:50:14.935665+0000 mon.a (mon.0 2021-03-24T21:50:15.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:15 smithi125 conmon[19936]: ) 328 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2021-03-24T21:50:15.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:15 smithi125 conmon[19936]: audit 2021-03-24T21:50:14.935916+0000 2021-03-24T21:50:15.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:15 smithi125 conmon[19936]: mon.a (mon.0) 329 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-03-24T21:50:15.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:15 smithi125 conmon[24927]: cluster 2021-03-24T21:50:14.031191 2021-03-24T21:50:15.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:15 smithi125 conmon[24927]: +0000 mon.a (mon.0) 325 : cluster [DBG] osdmap e27: 4 total, 4 up, 4 in 2021-03-24T21:50:15.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:15 smithi125 conmon[24927]: audit 2021-03-24T21: 2021-03-24T21:50:15.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:15 smithi125 conmon[24927]: 50:14.922575+0000 mon.b (mon.1) 7 : audit [INF] from='client.? v1:172.21.15.142:0/2624801269' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f9bdd8d2-b292-49da-ad4f-99682122f543"}]: dispatch 2021-03-24T21:50:15.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:15 smithi125 conmon[24927]: audit 2021 2021-03-24T21:50:15.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:15 smithi125 conmon[24927]: -03-24T21:50:14.926390+0000 mon.a ( 2021-03-24T21:50:15.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:15 smithi125 conmon[24927]: mon.0) 326 : audit [INF] from='client.24229 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f9bdd8d2-b292-49da-ad4f-99682122f543"}]: dispatch 2021-03-24T21:50:15.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:15 smithi125 conmon[24927]: audit 2021-03-24T21:50:15.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:15 smithi125 conmon[24927]: 2021-03-24T21:50:14. 2021-03-24T21:50:15.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:15 smithi125 conmon[24927]: 935438+0000 mon.a (mon.0) 327 : audit 2021-03-24T21:50:15.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:15 smithi125 conmon[24927]: [INF] from='client.24229 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f9bdd8d2-b292-49da-ad4f-99682122f543"}]': finished 2021-03-24T21:50:15.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:15 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:50:15.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:15 smithi125 conmon[24927]: 21:50:14.935665+0000 mon.a (mon 2021-03-24T21:50:15.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:15 smithi125 conmon[24927]: .0) 328 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2021-03-24T21:50:15.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:15 smithi125 conmon[24927]: audit 2021 2021-03-24T21:50:15.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:15 smithi125 conmon[24927]: -03-24T21:50:14.935916+0000 2021-03-24T21:50:15.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:15 smithi125 conmon[24927]: mon.a (mon.0) 329 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-03-24T21:50:15.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:15 smithi125 conmon[24927]: 2021-03-24T21:50:16.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:16 smithi142 conmon[20125]: cluster 2021-03-24T21:50:14.607970+0000 2021-03-24T21:50:16.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:16 smithi142 conmon[20125]: mgr.y (mgr.14144) 126 : cluster [DBG] pgmap v112: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-24T21:50:16.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:16 smithi142 conmon[20125]: audit 2021-03-24T21:50:15.577560+0000 mon.b (mon.1) 8 2021-03-24T21:50:16.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:16 smithi142 conmon[20125]: : audit [DBG] from='client.? v1:172.21.15.142:0/1717422512' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-24T21:50:16.401 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:16 smithi125 conmon[24927]: cluster 2021-03-24T21:50 2021-03-24T21:50:16.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:16 smithi125 conmon[24927]: :14.607970+0000 mgr.y (mgr.14144) 126 : cluster [DBG] pgmap v112: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-24T21:50:16.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:16 smithi125 conmon[24927]: audit 2021-03-24T21:50:15. 2021-03-24T21:50:16.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:16 smithi125 conmon[24927]: 577560+0000 mon.b (mon.1) 8 : audit [DBG] 2021-03-24T21:50:16.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:16 smithi125 conmon[24927]: from='client.? v1:172.21.15.142:0/1717422512' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-24T21:50:16.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:16 smithi125 conmon[19936]: cluster 2021-03-24T21:50: 2021-03-24T21:50:16.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:16 smithi125 conmon[19936]: 14.607970+0000 mgr.y (mgr.14144) 126 : cluster [DBG] pgmap v112: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-24T21:50:16.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:16 smithi125 conmon[19936]: audit 2021-03-24 2021-03-24T21:50:16.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:16 smithi125 conmon[19936]: T21:50:15.577560+0000 2021-03-24T21:50:16.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:16 smithi125 conmon[19936]: mon.b (mon.1) 8 : audit [DBG] from='client.? v1:172.21.15.142:0/1717422512' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-24T21:50:18.361 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:18 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:50:18.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:18 smithi142 conmon[20125]: 50:16.608540+0000 mgr.y (mgr.14144) 127 : cluster [DBG] pgmap v114: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-24T21:50:18.401 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:18 smithi125 conmon[24927]: cluster 2021-03-24T21:50:16. 2021-03-24T21:50:18.401 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:18 smithi125 conmon[24927]: 608540+0000 mgr.y (mgr.14144) 127 : cluster [DBG] pgmap v114: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-24T21:50:18.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:18 smithi125 conmon[19936]: cluster 2021-03-24T21:50:16.608540 2021-03-24T21:50:18.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:18 smithi125 conmon[19936]: +0000 mgr.y (mgr.14144) 127 : cluster [DBG] pgmap v114: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-24T21:50:20.320 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:20 smithi142 conmon[20125]: cluster 2021-03-24T21:50:18.608998+0000 mgr.y 2021-03-24T21:50:20.320 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:20 smithi142 conmon[20125]: (mgr.14144) 128 : cluster [DBG] pgmap v115: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-24T21:50:20.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:20 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:50:20.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:20 smithi125 conmon[19936]: 21:50:18.608998+0000 mgr.y (mgr.14144) 128 : cluster [DBG] pgmap v115: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-24T21:50:20.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:20 smithi125 conmon[24927]: cluster 2021-03-24T21:50 2021-03-24T21:50:20.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:20 smithi125 conmon[24927]: :18.608998+0000 mgr.y (mgr.14144) 128 : cluster [DBG] pgmap v115: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-24T21:50:22.320 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:22 smithi142 conmon[20125]: cluster 2021-03-24T21:50:20.609633+0000 mgr.y (mgr.14144) 129 : cluster 2021-03-24T21:50:22.320 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:22 smithi142 conmon[20125]: [DBG] pgmap v116: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-24T21:50:22.321 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:22 smithi142 conmon[20125]: audit 2021-03-24T21:50:21.565291+0000 mon.a (mon.0) 330 2021-03-24T21:50:22.321 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:22 smithi142 conmon[20125]: : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2021-03-24T21:50:22.321 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:22 smithi142 conmon[20125]: audit 2021-03-24T21:50:22.321 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:22 smithi142 conmon[20125]: 2021-03-24T21:50:21.566988+0000 mon.a (mon 2021-03-24T21:50:22.322 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:22 smithi142 conmon[20125]: .0) 331 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:50:22.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:22 smithi125 conmon[19936]: cluster 2021-03-24T21:50: 2021-03-24T21:50:22.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:22 smithi125 conmon[19936]: 20.609633+0000 mgr.y (mgr.14144) 129 : cluster [DBG] pgmap v116: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-24T21:50:22.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:22 smithi125 conmon[19936]: audit 2021-03-24T 2021-03-24T21:50:22.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:22 smithi125 conmon[19936]: 21:50:21.565291+0000 mon.a (mon.0) 330 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2021-03-24T21:50:22.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:22 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:50:22.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:22 smithi125 conmon[19936]: 03-24T21:50:21.566988+0000 2021-03-24T21:50:22.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:22 smithi125 conmon[19936]: mon.a (mon.0) 331 : audit [DBG] 2021-03-24T21:50:22.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:22 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:50:22.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:22 smithi125 conmon[24927]: cluster 2021-03-24T21:50 2021-03-24T21:50:22.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:22 smithi125 conmon[24927]: :20.609633+0000 mgr.y (mgr.14144) 129 : cluster [DBG] pgmap v116: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-24T21:50:22.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:22 smithi125 conmon[24927]: audit 2021-03-24T21:50:22.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:22 smithi125 conmon[24927]: 2021-03-24T21:50:21.565291+0000 mon.a (mon.0) 330 : audit 2021-03-24T21:50:22.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:22 smithi125 conmon[24927]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2021-03-24T21:50:22.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:22 smithi125 conmon[24927]: audit 2021-03-24T21:50:22.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:22 smithi125 conmon[24927]: 2021-03-24T21:50:21.566988+0000 mon.a (mon 2021-03-24T21:50:22.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:22 smithi125 conmon[24927]: .0) 331 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:50:22.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:22 smithi125 conmon[24927]: 2021-03-24T21:50:23.262 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:23 smithi142 conmon[20125]: cephadm 2021-03-24T21:50:21.568437+0000 mgr.y (mgr.14144) 130 : 2021-03-24T21:50:23.262 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:23 smithi142 conmon[20125]: cephadm [INF] Deploying daemon osd.4 on smithi142 2021-03-24T21:50:23.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:23 smithi125 conmon[19936]: cephadm 2021-03-24T21:50: 2021-03-24T21:50:23.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:23 smithi125 conmon[19936]: 21.568437+0000 mgr.y (mgr.14144) 130 : cephadm [INF] Deploying daemon osd.4 on smithi142 2021-03-24T21:50:23.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:23 smithi125 conmon[24927]: cephadm 2021-03-24 2021-03-24T21:50:23.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:23 smithi125 conmon[24927]: T21:50:21.568437+0000 mgr.y (mgr.14144) 130 : cephadm [INF] Deploying daemon osd.4 on smithi142 2021-03-24T21:50:24.111 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:24 smithi142 conmon[20125]: cluster 2021-03-24T21:50:22.610289+0000 mgr.y (mgr.14144) 131 : cluster [DBG] 2021-03-24T21:50:24.112 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:24 smithi142 conmon[20125]: pgmap v117: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-24T21:50:24.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:24 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:50:24.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:24 smithi125 conmon[19936]: 21:50:22.610289+0000 mgr.y (mgr.14144) 131 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-24T21:50:24.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:24 smithi125 conmon[24927]: cluster 2021-03-24T21:50:22.610289+0000 mgr.y (mgr.14144) 131 : 2021-03-24T21:50:24.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:24 smithi125 conmon[24927]: cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-24T21:50:25.624 INFO:teuthology.orchestra.run.smithi142.stdout:Created osd(s) 4 on host 'smithi142' 2021-03-24T21:50:26.111 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:26 smithi142 conmon[20125]: cluster 2021-03-24T21:50:24.610835+0000 mgr.y (mgr.14144) 132 : cluster 2021-03-24T21:50:26.112 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:26 smithi142 conmon[20125]: [DBG] pgmap v118: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-24T21:50:26.112 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:26 smithi142 conmon[20125]: audit 2021-03-24T21:50:25.612254+0000 mon.a (mon.0) 332 : audit 2021-03-24T21:50:26.112 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:26 smithi142 conmon[20125]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:50:26.113 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:26 smithi142 conmon[20125]: audit 2021-03-24T21:50:26.113 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:26 smithi142 conmon[20125]: 2021-03-24T21:50:25.620181+0000 mon.a (mon 2021-03-24T21:50:26.113 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:26 smithi142 conmon[20125]: .0) 333 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:26.113 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:26 smithi142 conmon[20125]: audit 2021-03-24T21:50:26.113 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:26 smithi142 conmon[20125]: 2021-03- 2021-03-24T21:50:26.114 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:26 smithi142 conmon[20125]: 24T21:50:25.631521+0000 2021-03-24T21:50:26.114 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:26 smithi142 conmon[20125]: mon.a (mon.0) 334 : audit [INF] 2021-03-24T21:50:26.114 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:26 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:26.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:26 smithi125 conmon[19936]: cluster 2021-03-24T21:50:24 2021-03-24T21:50:26.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:26 smithi125 conmon[19936]: .610835+0000 mgr.y (mgr.14144) 132 : cluster [DBG] pgmap v118: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-24T21:50:26.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:26 smithi125 conmon[19936]: audit 2021-03-24T21:50:25.612254+0000 2021-03-24T21:50:26.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:26 smithi125 conmon[19936]: mon.a (mon.0) 332 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:50:26.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:26 smithi125 conmon[19936]: audit 2021-03- 2021-03-24T21:50:26.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:26 smithi125 conmon[19936]: 24T21:50:25.620181+0000 mon.a (mon.0) 333 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:26.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:26 smithi125 conmon[19936]: 2021-03-24T21:50:26.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:26 smithi125 conmon[19936]: audit 2021 2021-03-24T21:50:26.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:26 smithi125 conmon[19936]: -03-24T21:50:25.631521+0000 mon.a ( 2021-03-24T21:50:26.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:26 smithi125 conmon[19936]: mon.0) 334 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:26.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:26 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:50:26.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:26 smithi125 conmon[24927]: T21:50:24.610835+0000 mgr.y (mgr.14144) 132 : cluster [DBG] 2021-03-24T21:50:26.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:26 smithi125 conmon[24927]: pgmap v118: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-24T21:50:26.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:26 smithi125 conmon[24927]: audit 2021-03-24T21:50:26.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:26 smithi125 conmon[24927]: 2021-03-24T21:50:25.612254+0000 2021-03-24T21:50:26.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:26 smithi125 conmon[24927]: mon.a (mon.0) 332 : audit [DBG] 2021-03-24T21:50:26.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:26 smithi125 conmon[24927]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:50:26.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:26 smithi125 conmon[24927]: audit 2021-03-24T21:50:26.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:26 smithi125 conmon[24927]: 2021-03-24T21: 2021-03-24T21:50:26.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:26 smithi125 conmon[24927]: 50:25.620181+0000 mon.a (mon.0) 333 : 2021-03-24T21:50:26.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:26 smithi125 conmon[24927]: audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:26.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:26 smithi125 conmon[24927]: audit 2021- 2021-03-24T21:50:26.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:26 smithi125 conmon[24927]: 03-24T21:50:25.631521+0000 mon.a 2021-03-24T21:50:26.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:26 smithi125 conmon[24927]: (mon.0) 334 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:26.420 DEBUG:teuthology.orchestra.run.smithi142:osd.4> sudo journalctl -f -n 0 -u ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c@osd.4.service 2021-03-24T21:50:26.422 INFO:tasks.cephadm:Deploying osd.5 on smithi142 with /dev/vg_nvme/lv_3... 2021-03-24T21:50:26.423 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2021-03-24T21:50:26.463 INFO:journalctl@ceph.osd.4.smithi142.stdout:-- Logs begin at Wed 2021-03-03 23:51:07 UTC. -- 2021-03-24T21:50:28.335 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:28 smithi142 conmon[20125]: cluster 2021-03-24T21:50:26.611376+0000 mgr.y (mgr.14144) 133 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-24T21:50:28.335 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:28 smithi142 conmon[20125]: 2021-03-24T21:50:28.336 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:28 smithi142 conmon[20125]: audit 2021-03-24T21:50:28.075937+0000 mon.a (mon.0) 335 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:28.401 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:28 smithi125 conmon[24927]: cluster 2021-03-24T21:50:26. 2021-03-24T21:50:28.401 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:28 smithi125 conmon[24927]: 611376+0000 mgr.y (mgr.14144) 133 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-24T21:50:28.401 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:28 smithi125 conmon[24927]: audit 2021 2021-03-24T21:50:28.401 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:28 smithi125 conmon[24927]: -03-24T21:50:28.075937+0000 mon.a (mon.0) 335 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:28.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:28 smithi125 conmon[24927]: 2021-03-24T21:50:28.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:28 smithi125 conmon[19936]: cluster 2021-03-24T21:50:26 2021-03-24T21:50:28.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:28 smithi125 conmon[19936]: .611376+0000 mgr.y (mgr.14144) 133 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-24T21:50:28.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:28 smithi125 conmon[19936]: audit 2021-03-24T21:50:28.075937+0000 2021-03-24T21:50:28.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:28 smithi125 conmon[19936]: mon.a (mon.0) 335 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:28.995 INFO:teuthology.orchestra.run.smithi142.stderr:--> Zapping: /dev/vg_nvme/lv_3 2021-03-24T21:50:28.996 INFO:teuthology.orchestra.run.smithi142.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2021-03-24T21:50:28.996 INFO:teuthology.orchestra.run.smithi142.stderr: stderr: 10+0 records in 2021-03-24T21:50:28.996 INFO:teuthology.orchestra.run.smithi142.stderr:10+0 records out 2021-03-24T21:50:28.997 INFO:teuthology.orchestra.run.smithi142.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0145925 s, 719 MB/s 2021-03-24T21:50:28.997 INFO:teuthology.orchestra.run.smithi142.stderr:--> Zapping successful for: 2021-03-24T21:50:29.211 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:29 smithi142 conmon[20125]: cluster 2021-03-24T21:50:28.441181+0000 osd.1 (osd.1) 3 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:50:29.211 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:29 smithi142 conmon[20125]: cluster 2021-03-24T21:50:28.441376+0000 osd.1 (osd.1) 4 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:50:29.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:29 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:50:29.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:29 smithi125 conmon[19936]: 50:28.441181+0000 osd.1 (osd.1) 3 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:50:29.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:29 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:50:29.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:29 smithi125 conmon[19936]: 03-24T21:50:28.441376+0000 osd.1 (osd.1) 4 : cluster [DBG] 2021-03-24T21:50:29.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:29 smithi125 conmon[19936]: purged_snaps scrub ok 2021-03-24T21:50:29.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:29 smithi125 conmon[24927]: cluster 2021-03-24T21:50: 2021-03-24T21:50:29.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:29 smithi125 conmon[24927]: 28.441181+0000 osd.1 (osd.1) 3 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:50:29.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:29 smithi125 conmon[24927]: cluster 2021-03-24T21:50:28.441376 2021-03-24T21:50:29.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:29 smithi125 conmon[24927]: +0000 osd.1 (osd.1) 4 : cluster [DBG] 2021-03-24T21:50:29.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:29 smithi125 conmon[24927]: purged_snaps scrub ok 2021-03-24T21:50:29.503 INFO:journalctl@ceph.osd.4.smithi142.stdout:Mar 24 21:50:29 smithi142 conmon[24076]: debug 2021-03-24T21:50:29.212+0000 7fc5889cd080 -1 osd.4 0 log_to_monitors {default=true} 2021-03-24T21:50:30.181 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph orch daemon add osd smithi142:vg_nvme/lv_3 2021-03-24T21:50:30.236 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:30 smithi142 conmon[20125]: cluster 2021-03-24T21:50:28 2021-03-24T21:50:30.237 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:30 smithi142 conmon[20125]: .611823+0000 mgr.y (mgr.14144) 134 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-24T21:50:30.237 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:30 smithi142 conmon[20125]: cluster 2021-03-24T21:50:29.094872 2021-03-24T21:50:30.237 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:30 smithi142 conmon[20125]: +0000 mon.a (mon.0) 336 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2021-03-24T21:50:30.237 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:30 smithi142 conmon[20125]: audit 2021-03-24T21: 2021-03-24T21:50:30.237 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:30 smithi142 conmon[20125]: 50:29.095137+0000 mon.a (mon.0) 2021-03-24T21:50:30.238 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:30 smithi142 conmon[20125]: 337 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-03-24T21:50:30.238 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:30 smithi142 conmon[20125]: audit 2021-03-24T 2021-03-24T21:50:30.238 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:30 smithi142 conmon[20125]: 21:50:29.222066+0000 mon.b (mon. 2021-03-24T21:50:30.238 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:30 smithi142 conmon[20125]: 1) 9 : audit [INF] from='osd.4 v1:172.21.15.142:6800/1816503330' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-03-24T21:50:30.239 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:30 smithi142 conmon[20125]: audit 2021-03-24 2021-03-24T21:50:30.239 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:30 smithi142 conmon[20125]: T21:50:29.225539+0000 mon.a ( 2021-03-24T21:50:30.239 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:30 smithi142 conmon[20125]: mon.0) 338 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-03-24T21:50:30.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:30 smithi125 conmon[19936]: cluster 2021-03-24T21:50:28.611823+0000 mgr.y (mgr.14144) 134 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-24T21:50:30.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:30 smithi125 conmon[19936]: cluster 2021-03-24T21:50:29.094872+0000 mon.a (mon.0) 336 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2021-03-24T21:50:30.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:30 smithi125 conmon[19936]: audit 2021-03-24T21:50:29.095137+0000 mon.a (mon.0) 337 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-03-24T21:50:30.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:30 smithi125 conmon[19936]: audit 2021-03-24T21:50:29.222066+0000 mon.b (mon. 2021-03-24T21:50:30.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:30 smithi125 conmon[19936]: 1) 9 : audit [INF] from='osd.4 v1:172.21.15.142:6800/1816503330' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-03-24T21:50:30.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:30 smithi125 conmon[19936]: audit 2021-03-24T21:50:29.225539+0000 mon.a (mon.0) 338 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-03-24T21:50:30.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:30 smithi125 conmon[24927]: cluster 2021-03-24T21:50 2021-03-24T21:50:30.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:30 smithi125 conmon[24927]: :28.611823+0000 mgr.y (mgr.14144) 134 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-24T21:50:30.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:30 smithi125 conmon[24927]: cluster 2021-03-24T21:50:29.094872 2021-03-24T21:50:30.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:30 smithi125 conmon[24927]: +0000 mon.a (mon.0) 336 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2021-03-24T21:50:30.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:30 smithi125 conmon[24927]: 2021-03-24T21:50:30.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:30 smithi125 conmon[24927]: audit 2021-03-24T21:50: 2021-03-24T21:50:30.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:30 smithi125 conmon[24927]: 29.095137+0000 mon.a (mon.0) 337 : 2021-03-24T21:50:30.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:30 smithi125 conmon[24927]: audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-03-24T21:50:30.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:30 smithi125 conmon[24927]: audit 2021-03 2021-03-24T21:50:30.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:30 smithi125 conmon[24927]: -24T21:50:29.222066+0000 mon.b ( 2021-03-24T21:50:30.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:30 smithi125 conmon[24927]: mon.1) 9 : audit [INF] from='osd.4 v1:172.21.15.142:6800/1816503330' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-03-24T21:50:30.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:30 smithi125 conmon[24927]: audit 2021-03 2021-03-24T21:50:30.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:30 smithi125 conmon[24927]: -24T21:50:29.225539+0000 mon.a ( 2021-03-24T21:50:30.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:30 smithi125 conmon[24927]: mon.0) 338 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-03-24T21:50:31.361 INFO:journalctl@ceph.osd.4.smithi142.stdout:Mar 24 21:50:31 smithi142 conmon[24076]: debug 2021-03-24T21:50:31.112+0000 7fc57ef3e700 -1 osd.4 0 waiting for initial osdmap 2021-03-24T21:50:31.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:31 smithi142 conmon[20125]: audit 2021-03-24T21:50:30.103654+0000 mon.b (mon.1) 10 2021-03-24T21:50:31.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:31 smithi142 conmon[20125]: : audit [INF] from='osd.4 v1:172.21.15.142:6800/1816503330' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2021-03-24T21:50:31.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:31 smithi142 conmon[20125]: audit 2021-03-24T21:50:30.104628+0000 mon.a (mon.0) 339 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2021-03-24T21:50:31.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:31 smithi142 conmon[20125]: cluster 2021-03-24T21 2021-03-24T21:50:31.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:31 smithi142 conmon[20125]: :50:30.104801+0000 mon.a (mon.0) 340 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2021-03-24T21:50:31.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:31 smithi142 conmon[20125]: audit 2021-03-24T21:50:30. 2021-03-24T21:50:31.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:31 smithi142 conmon[20125]: 105026+0000 mon.a (mon.0) 341 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-03-24T21:50:31.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:31 smithi142 conmon[20125]: audit 2021-03-24T 2021-03-24T21:50:31.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:31 smithi142 conmon[20125]: 21:50:30.107265+0000 mon.a (mon.0) 342 : audit 2021-03-24T21:50:31.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:31 smithi142 conmon[20125]: [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2021-03-24T21:50:31.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:31 smithi125 conmon[19936]: audit 2021-03-24T21:50: 2021-03-24T21:50:31.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:31 smithi125 conmon[19936]: 30.103654+0000 mon.b (mon.1) 10 : audit [INF] from='osd.4 v1:172.21.15.142:6800/1816503330' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2021-03-24T21:50:31.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:31 smithi125 conmon[19936]: audit 2021-03-24T 2021-03-24T21:50:31.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:31 smithi125 conmon[19936]: 21:50:30.104628+0000 mon.a (mon.0) 339 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2021-03-24T21:50:31.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:31 smithi125 conmon[19936]: cluster 2021-03-24T21:50 2021-03-24T21:50:31.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:31 smithi125 conmon[19936]: :30.104801+0000 mon.a (mon.0) 340 : cluster 2021-03-24T21:50:31.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:31 smithi125 conmon[19936]: [DBG] osdmap e30: 5 total, 4 up, 5 in 2021-03-24T21:50:31.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:31 smithi125 conmon[19936]: audit 2021-03-24T21:50:31.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:31 smithi125 conmon[19936]: 2021-03-24T21:50:30.105026+0000 2021-03-24T21:50:31.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:31 smithi125 conmon[19936]: mon.a (mon.0) 341 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-03-24T21:50:31.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:31 smithi125 conmon[19936]: audit 2021-03- 2021-03-24T21:50:31.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:31 smithi125 conmon[19936]: 24T21:50:30.107265+0000 mon.a 2021-03-24T21:50:31.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:31 smithi125 conmon[19936]: (mon.0) 342 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2021-03-24T21:50:31.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:31 smithi125 conmon[24927]: audit 2021-03-24T21:50:31.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:31 smithi125 conmon[24927]: 2021-03-24T21: 2021-03-24T21:50:31.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:31 smithi125 conmon[24927]: 50:30.103654+0000 mon.b (mon.1) 10 : audit [INF] from='osd.4 v1:172.21.15.142:6800/1816503330' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2021-03-24T21:50:31.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:31 smithi125 conmon[24927]: audit 2021-03-24T21:50:31.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:31 smithi125 conmon[24927]: 2021-03-24T21 2021-03-24T21:50:31.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:31 smithi125 conmon[24927]: :50:30.104628+0000 mon.a (mon.0) 339 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2021-03-24T21:50:31.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:31 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:50:31.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:31 smithi125 conmon[24927]: :50:30.104801+0000 mon.a (mon.0) 340 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2021-03-24T21:50:31.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:31 smithi125 conmon[24927]: audit 2021-03-24T21:50:30.105026+0000 2021-03-24T21:50:31.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:31 smithi125 conmon[24927]: mon.a (mon.0) 341 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-03-24T21:50:31.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:31 smithi125 conmon[24927]: audit 2021-03-24T21:50:30.107265 2021-03-24T21:50:31.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:31 smithi125 conmon[24927]: +0000 mon.a (mon.0) 342 : audit [INF] 2021-03-24T21:50:31.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:31 smithi125 conmon[24927]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2021-03-24T21:50:32.361 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:32 smithi142 conmon[20125]: cluster 2021-03-24T21:50:30.612335+0000 mgr.y ( 2021-03-24T21:50:32.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:32 smithi142 conmon[20125]: mgr.14144) 135 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-24T21:50:32.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:32 smithi142 conmon[20125]: audit 2021-03-24T21:50:31.108880+0000 mon.a (mon.0) 343 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2021-03-24T21:50:32.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:32 smithi142 conmon[20125]: cluster 2021-03-24T21:50:31.109050+0000 mon.a (mon.0) 344 : cluster [DBG] osdmap e31: 5 total, 4 up, 5 in 2021-03-24T21:50:32.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:32 smithi142 conmon[20125]: 2021-03-24T21:50:32.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:32 smithi142 conmon[20125]: audit 2021-03-24 2021-03-24T21:50:32.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:32 smithi142 conmon[20125]: T21:50:31.111356+0000 mon.a (mon.0) 345 : audit [DBG] 2021-03-24T21:50:32.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:32 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-03-24T21:50:32.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:32 smithi142 conmon[20125]: audit 2021-03- 2021-03-24T21:50:32.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:32 smithi142 conmon[20125]: 24T21:50:31.117333+0000 mon.a (mon.0) 346 : audit [DBG] 2021-03-24T21:50:32.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:32 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-03-24T21:50:32.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:32 smithi125 conmon[19936]: cluster 2021-03-24T21:50:30.612335+0000 mgr.y (mgr 2021-03-24T21:50:32.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:32 smithi125 conmon[19936]: .14144) 135 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-24T21:50:32.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:32 smithi125 conmon[19936]: audit 2021-03-24T21:50:31.108880+0000 mon.a (mon.0) 343 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2021-03-24T21:50:32.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:32 smithi125 conmon[19936]: cluster 2021-03-24T21:50:31.109050 2021-03-24T21:50:32.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:32 smithi125 conmon[19936]: +0000 mon.a (mon.0) 344 : cluster [DBG] osdmap e31: 5 total, 4 up, 5 in 2021-03-24T21:50:32.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:32 smithi125 conmon[19936]: audit 2021-03-24 2021-03-24T21:50:32.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:32 smithi125 conmon[19936]: T21:50:31.111356+0000 mon.a (mon.0) 345 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-03-24T21:50:32.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:32 smithi125 conmon[19936]: 2021-03-24T21:50:32.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:32 smithi125 conmon[19936]: audit 2021 2021-03-24T21:50:32.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:32 smithi125 conmon[19936]: -03-24T21:50:31.117333+0000 mon.a (mon.0) 346 : 2021-03-24T21:50:32.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:32 smithi125 conmon[19936]: audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-03-24T21:50:32.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:32 smithi125 conmon[24927]: cluster 2021-03-24T21:50:32.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:32 smithi125 conmon[24927]: 2021-03-24T21:50:32.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:32 smithi125 conmon[24927]: 2021-03-24T21:50:30.612335+0000 mgr.y (mgr.14144) 135 : 2021-03-24T21:50:32.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:32 smithi125 conmon[24927]: cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-24T21:50:32.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:32 smithi125 conmon[24927]: audit 2021-03-24T21:50:31.108880+0000 mon.a (mon.0) 343 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2021-03-24T21:50:32.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:32 smithi125 conmon[24927]: 2021-03-24T21:50:32.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:32 smithi125 conmon[24927]: cluster 2021-03-24T21:50:31.109050+0000 mon.a (mon.0) 344 : cluster [DBG] osdmap e31: 5 total, 4 up, 5 in 2021-03-24T21:50:32.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:32 smithi125 conmon[24927]: audit 2021-03-24T21:50:31.111356+0000 mon.a (mon.0) 345 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-03-24T21:50:32.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:32 smithi125 conmon[24927]: audit 2021-03-24T21:50:32.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:32 smithi125 conmon[24927]: 2021-03-24T21:50:31.117333+0000 mon.a (mon.0) 346 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-03-24T21:50:33.401 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:33 smithi125 conmon[24927]: cluster 2021-03-24T21:50:30.195373+0000 osd.4 ( 2021-03-24T21:50:33.401 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:33 smithi125 conmon[24927]: osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:50:33.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:33 smithi125 conmon[24927]: cluster 2021-03-24T21:50:30.195584+0000 osd.4 (osd.4) 2 : cluster [DBG] 2021-03-24T21:50:33.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:33 smithi125 conmon[24927]: purged_snaps scrub ok 2021-03-24T21:50:33.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:33 smithi125 conmon[24927]: audit 2021-03-24T21:50:32.115114+0000 mon.a (mon.0 2021-03-24T21:50:33.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:33 smithi125 conmon[24927]: ) 347 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-03-24T21:50:33.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:33 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:50:33.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:33 smithi125 conmon[24927]: 21:50:32.122801+0000 mon.a (mon.0) 348 : cluster [INF] osd.4 v1:172.21.15.142:6800/1816503330 boot 2021-03-24T21:50:33.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:33 smithi125 conmon[24927]: cluster 2021-03-24T21:50:32.122902+0000 mon.a (mon 2021-03-24T21:50:33.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:33 smithi125 conmon[24927]: .0) 349 : cluster [DBG] osdmap e32: 5 total, 5 up, 5 in 2021-03-24T21:50:33.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:33 smithi125 conmon[24927]: audit 2021-03-24T21:50:33.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:33 smithi125 conmon[24927]: 2021-03-24T21:50:32.124058+0000 mon.a (mon.0) 350 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-03-24T21:50:33.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:33 smithi125 conmon[24927]: audit 2021-03-24T21:50: 2021-03-24T21:50:33.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:33 smithi125 conmon[24927]: 33.070330+0000 mon.a (mon.0) 351 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:33.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:33 smithi125 conmon[24927]: cluster 2021-03-24T21:50:33.123940+0000 mon.a (mon.0) 2021-03-24T21:50:33.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:33 smithi125 conmon[24927]: 352 : cluster [DBG] osdmap e33: 5 total, 5 up, 5 in 2021-03-24T21:50:33.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:33 smithi125 conmon[19936]: cluster 2021-03-24T21:50:30.195373+0000 2021-03-24T21:50:33.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:33 smithi125 conmon[19936]: osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:50:33.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:33 smithi125 conmon[19936]: cluster 2021-03-24T21:50:30.195584+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:50:33.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:33 smithi125 conmon[19936]: audit 2021-03-24T21:50:32.115114 2021-03-24T21:50:33.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:33 smithi125 conmon[19936]: +0000 mon.a (mon.0) 347 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-03-24T21:50:33.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:33 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:50:33.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:33 smithi125 conmon[19936]: 03-24T21:50:32.122801+0000 mon.a (mon.0 2021-03-24T21:50:33.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:33 smithi125 conmon[19936]: ) 348 : cluster [INF] osd.4 v1:172.21.15.142:6800/1816503330 boot 2021-03-24T21:50:33.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:33 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:50:33.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:33 smithi125 conmon[19936]: -03-24T21:50:32.122902+0000 mon.a ( 2021-03-24T21:50:33.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:33 smithi125 conmon[19936]: mon.0) 349 : cluster [DBG] osdmap e32: 5 total, 5 up, 5 in 2021-03-24T21:50:33.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:33 smithi125 conmon[19936]: audit 2021-03-24T21:50:33.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:33 smithi125 conmon[19936]: 2021-03-24T 2021-03-24T21:50:33.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:33 smithi125 conmon[19936]: 21:50:32.124058+0000 mon.a (mon.0) 2021-03-24T21:50:33.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:33 smithi125 conmon[19936]: 350 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-03-24T21:50:33.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:33 smithi125 conmon[19936]: audit 2021-03 2021-03-24T21:50:33.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:33 smithi125 conmon[19936]: -24T21:50:33.070330+0000 mon.a (mon 2021-03-24T21:50:33.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:33 smithi125 conmon[19936]: .0) 351 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:33.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:33 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:50:33.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:33 smithi125 conmon[19936]: 03-24T21:50:33.123940+0000 mon.a 2021-03-24T21:50:33.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:33 smithi125 conmon[19936]: (mon.0) 352 : cluster [DBG] osdmap e33: 5 total, 5 up, 5 in 2021-03-24T21:50:33.611 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:33 smithi142 conmon[20125]: cluster 2021-03-24T21:50:30.195373+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:50:33.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:33 smithi142 conmon[20125]: cluster 2021-03-24T21:50:30.195584+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:50:33.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:33 smithi142 conmon[20125]: audit 2021-03-24T21:50:32.115114+0000 mon.a (mon.0) 2021-03-24T21:50:33.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:33 smithi142 conmon[20125]: 347 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-03-24T21:50:33.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:33 smithi142 conmon[20125]: cluster 2021-03-24T21:50:32.122801+0000 mon.a (mon.0 2021-03-24T21:50:33.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:33 smithi142 conmon[20125]: ) 348 : cluster [INF] osd.4 v1:172.21.15.142:6800/1816503330 boot 2021-03-24T21:50:33.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:33 smithi142 conmon[20125]: cluster 2021-03-24T21:50:32.122902+0000 mon.a 2021-03-24T21:50:33.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:33 smithi142 conmon[20125]: (mon.0) 349 : cluster [DBG] osdmap e32: 5 total, 5 up, 5 in 2021-03-24T21:50:33.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:33 smithi142 conmon[20125]: audit 2021-03-24T21:50:32. 2021-03-24T21:50:33.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:33 smithi142 conmon[20125]: 124058+0000 mon.a (mon.0) 350 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-03-24T21:50:33.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:33 smithi142 conmon[20125]: audit 2021-03-24T21 2021-03-24T21:50:33.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:33 smithi142 conmon[20125]: :50:33.070330+0000 mon.a (mon.0) 351 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:33.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:33 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:50:33.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:33 smithi142 conmon[20125]: -24T21:50:33.123940+0000 mon.a (mon.0) 352 : cluster [DBG] osdmap e33: 5 total, 5 up, 5 in 2021-03-24T21:50:34.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:34 smithi142 conmon[20125]: cluster 2021-03-24T21:50:32.612811+0000 mgr.y (mgr.14144) 136 : cluster 2021-03-24T21:50:34.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:34 smithi142 conmon[20125]: [DBG] pgmap v126: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-24T21:50:34.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:34 smithi142 conmon[20125]: cephadm 2021-03-24T21:50:33.061689+0000 mgr.y (mgr 2021-03-24T21:50:34.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:34 smithi142 conmon[20125]: .14144) 137 : cephadm [INF] Detected new or changed devices on smithi142 2021-03-24T21:50:34.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:34 smithi142 conmon[20125]: audit 2021 2021-03-24T21:50:34.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:34 smithi142 conmon[20125]: -03-24T21:50:33. 2021-03-24T21:50:34.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:34 smithi142 conmon[20125]: 185550+0000 mgr.y (mgr.14144) 2021-03-24T21:50:34.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:34 smithi142 conmon[20125]: 138 : audit [DBG] from='client.24250 v1:172.21.15.142:0/197151611' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:50:34.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:34 smithi142 conmon[20125]: audit 2021-03-24T21:50:34.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:34 smithi142 conmon[20125]: 2021-03-24T21:50 2021-03-24T21:50:34.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:34 smithi142 conmon[20125]: :33.186910+0000 mon.a (mon 2021-03-24T21:50:34.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:34 smithi142 conmon[20125]: .0) 353 : audit [DBG] 2021-03-24T21:50:34.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:34 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-24T21:50:34.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:34 smithi142 conmon[20125]: cephadm 2021-03-24T21:50:34.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:34 smithi142 conmon[20125]: 2021-03-24T21:50 2021-03-24T21:50:34.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:34 smithi142 conmon[20125]: :33.189543+0000 mgr.y ( 2021-03-24T21:50:34.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:34 smithi142 conmon[20125]: mgr.14144) 139 : cephadm [INF] 2021-03-24T21:50:34.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:34 smithi142 conmon[20125]: Applying service osd.None on host smithi142... 2021-03-24T21:50:34.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:34 smithi142 conmon[20125]: audit 2021-03-24T21:50:34.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:34 smithi142 conmon[20125]: 2021-03-24T21:50 2021-03-24T21:50:34.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:34 smithi142 conmon[20125]: :33.190115+0000 mon.a ( 2021-03-24T21:50:34.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:34 smithi142 conmon[20125]: mon.0) 354 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-24T21:50:34.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:34 smithi142 conmon[20125]: audit 2021 2021-03-24T21:50:34.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:34 smithi142 conmon[20125]: -03-24T21:50: 2021-03-24T21:50:34.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:34 smithi142 conmon[20125]: 33.191604+0000 mon.a (mon. 2021-03-24T21:50:34.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:34 smithi142 conmon[20125]: 0) 355 : audit [DBG] 2021-03-24T21:50:34.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:34 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:50:34.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:34 smithi142 conmon[20125]: audit 2021-03-24T21:50:34.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:34 smithi142 conmon[20125]: 2021-03-24T21: 2021-03-24T21:50:34.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:34 smithi142 conmon[20125]: 50:33.385376+0000 mon.a ( 2021-03-24T21:50:34.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:34 smithi142 conmon[20125]: mon.0) 356 : audit 2021-03-24T21:50:34.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:34 smithi142 conmon[20125]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:34.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:34 smithi142 conmon[20125]: audit 2021-03-24T21:50:34.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:34 smithi142 conmon[20125]: 2021-03-24T21 2021-03-24T21:50:34.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:34 smithi142 conmon[20125]: :50:33.388318+0000 2021-03-24T21:50:34.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:34 smithi142 conmon[20125]: mon.a (mon.0) 357 : 2021-03-24T21:50:34.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:34 smithi142 conmon[20125]: audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:50:34.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:34 smithi142 conmon[20125]: audit 2021 2021-03-24T21:50:34.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:34 smithi142 conmon[20125]: -03-24T21:50: 2021-03-24T21:50:34.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:34 smithi142 conmon[20125]: 33.395447+0000 mon.a (mon 2021-03-24T21:50:34.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:34 smithi142 conmon[20125]: .0) 358 : audit [INF] 2021-03-24T21:50:34.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:34 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:34.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:34 smithi142 conmon[20125]: cluster 2021-03-24T21:50:34.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:34 smithi142 conmon[20125]: 2021-03-24T21:50 2021-03-24T21:50:34.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:34 smithi142 conmon[20125]: :34.125369+0000 mon.a (mon 2021-03-24T21:50:34.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:34 smithi142 conmon[20125]: .0) 359 : cluster [DBG] 2021-03-24T21:50:34.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:34 smithi142 conmon[20125]: osdmap e34: 5 total, 5 up, 5 in 2021-03-24T21:50:34.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[19936]: cluster 2021-03-24T21:50:32.612811+0000 mgr.y (mgr. 2021-03-24T21:50:34.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[19936]: 14144) 136 : cluster [DBG] pgmap v126: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-24T21:50:34.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[19936]: cephadm 2021-03-24T21:50:33.061689+0000 mgr.y (mgr.14144 2021-03-24T21:50:34.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[19936]: ) 137 : cephadm [INF] Detected new or changed devices on smithi142 2021-03-24T21:50:34.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[19936]: audit 2021-03 2021-03-24T21:50:34.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[19936]: -24T21:50:33. 2021-03-24T21:50:34.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[19936]: 185550+0000 mgr.y (mgr.14144 2021-03-24T21:50:34.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[19936]: ) 138 : audit [DBG] from='client.24250 v1:172.21.15.142:0/197151611' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:50:34.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[19936]: 2021-03-24T21:50:34.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[19936]: audit 2021-03- 2021-03-24T21:50:34.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[19936]: 24T21:50:33. 2021-03-24T21:50:34.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[19936]: 186910+0000 mon.a (mon.0 2021-03-24T21:50:34.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[19936]: ) 353 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-24T21:50:34.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[19936]: cephadm 2021- 2021-03-24T21:50:34.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[19936]: 03-24T21:50:33.189543 2021-03-24T21:50:34.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[19936]: +0000 mgr.y (mgr.14144) 139 : cephadm 2021-03-24T21:50:34.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[19936]: [INF] Applying service osd.None on host smithi142... 2021-03-24T21:50:34.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[19936]: audit 2021-03-24T21 2021-03-24T21:50:34.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[19936]: :50:33.190115+0000 mon.a 2021-03-24T21:50:34.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[19936]: (mon.0) 354 : audit 2021-03-24T21:50:34.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[19936]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-24T21:50:34.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[19936]: audit 2021-03- 2021-03-24T21:50:34.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[19936]: 24T21:50:33. 2021-03-24T21:50:34.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[19936]: 191604+0000 mon.a (mon.0) 355 : audit 2021-03-24T21:50:34.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[19936]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:50:34.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[19936]: audit 2021-03-24T21:50:34.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[19936]: 2021-03-24T21:50:33.385376+0000 mon.a 2021-03-24T21:50:34.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[19936]: (mon.0) 356 : audit 2021-03-24T21:50:34.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[19936]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:34.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[19936]: audit 2021-03-24T21:50:34.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[19936]: 2021-03-24T21:50: 2021-03-24T21:50:34.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[19936]: 33.388318+0000 mon.a (mon. 2021-03-24T21:50:34.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[19936]: 0) 357 : audit [DBG] 2021-03-24T21:50:34.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:50:34.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[19936]: audit 2021-03-24T21:50:33. 2021-03-24T21:50:34.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[19936]: 395447+0000 mon.a (mon.0) 2021-03-24T21:50:34.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[19936]: 358 : audit [INF] 2021-03-24T21:50:34.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:34.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[19936]: cluster 2021-03-24T21:50:34.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[19936]: 2021-03-24T21:50 2021-03-24T21:50:34.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[19936]: :34.125369+0000 mon.a ( 2021-03-24T21:50:34.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[19936]: mon.0) 359 : cluster 2021-03-24T21:50:34.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[19936]: [DBG] osdmap e34: 5 total, 5 up, 5 in 2021-03-24T21:50:34.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[24927]: cluster 2021-03-24T21:50 2021-03-24T21:50:34.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[24927]: :32.612811+0000 mgr.y (mgr.14144) 136 : cluster [DBG] pgmap v126: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-24T21:50:34.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[24927]: cephadm 2021-03- 2021-03-24T21:50:34.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[24927]: 24T21:50:33.061689+0000 2021-03-24T21:50:34.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[24927]: mgr.y (mgr.14144) 137 : cephadm [INF] 2021-03-24T21:50:34.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[24927]: Detected new or changed devices on smithi142 2021-03-24T21:50:34.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[24927]: audit 2021-03-24T21:50: 2021-03-24T21:50:34.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[24927]: 33.185550+0000 mgr.y (mgr.14144) 138 2021-03-24T21:50:34.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[24927]: : audit [DBG] from='client.24250 v1:172.21.15.142:0/197151611' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:50:34.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[24927]: audit 2021-03 2021-03-24T21:50:34.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[24927]: -24T21:50:33.186910+0000 2021-03-24T21:50:34.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[24927]: mon.a (mon.0) 353 : audit 2021-03-24T21:50:34.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[24927]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-24T21:50:34.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[24927]: cephadm 2021-03 2021-03-24T21:50:34.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[24927]: -24T21:50: 2021-03-24T21:50:34.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[24927]: 33.189543+0000 mgr.y (mgr 2021-03-24T21:50:34.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[24927]: .14144) 139 : cephadm [INF] 2021-03-24T21:50:34.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[24927]: Applying service osd.None on host smithi142... 2021-03-24T21:50:34.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[24927]: audit 2021-03-24 2021-03-24T21:50:34.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[24927]: T21:50:33.190115 2021-03-24T21:50:34.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[24927]: +0000 mon.a (mon.0) 354 : 2021-03-24T21:50:34.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[24927]: audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-24T21:50:34.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[24927]: audit 2021-03-24T21:50:34.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[24927]: 2021-03-24T21: 2021-03-24T21:50:34.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[24927]: 50:33.191604+0000 mon.a ( 2021-03-24T21:50:34.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[24927]: mon.0) 355 : audit [DBG] 2021-03-24T21:50:34.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[24927]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:50:34.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T21:50:34.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[24927]: 24T21:50:33.385376 2021-03-24T21:50:34.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[24927]: +0000 mon.a (mon.0) 356 : audit 2021-03-24T21:50:34.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[24927]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:34.420 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[24927]: audit 2021 2021-03-24T21:50:34.420 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[24927]: -03-24T21:50 2021-03-24T21:50:34.420 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[24927]: :33.388318+0000 mon.a (mon.0) 357 : audit [DBG] 2021-03-24T21:50:34.420 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[24927]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:50:34.421 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[24927]: audit 2021-03-24 2021-03-24T21:50:34.421 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[24927]: T21:50:33.395447 2021-03-24T21:50:34.421 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[24927]: +0000 mon.a (mon.0) 358 : audit 2021-03-24T21:50:34.421 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[24927]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:34.421 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:50:34.422 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[24927]: 03-24T21:50: 2021-03-24T21:50:34.422 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[24927]: 34.125369+0000 mon.a (mon.0 2021-03-24T21:50:34.422 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:34 smithi125 conmon[24927]: ) 359 : cluster [DBG] osdmap e34: 5 total, 5 up, 5 in 2021-03-24T21:50:36.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:36 smithi125 conmon[19936]: cluster 2021-03-24T21:50:34 2021-03-24T21:50:36.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:36 smithi125 conmon[19936]: .613306+0000 mgr.y (mgr.14144) 140 : cluster [DBG] pgmap v129: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-24T21:50:36.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:36 smithi125 conmon[19936]: audit 2021-03-24T21:50:36.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:36 smithi125 conmon[19936]: 2021-03-24T21:50:35.879868+0000 mon.b (mon. 2021-03-24T21:50:36.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:36 smithi125 conmon[19936]: 1) 11 : audit [INF] from='client.? v1:172.21.15.142:0/967325557' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec0d43aa-6905-43f0-9e32-e19d4cf5129d"}]: dispatch 2021-03-24T21:50:36.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:36 smithi125 conmon[19936]: audit 2021-03- 2021-03-24T21:50:36.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:36 smithi125 conmon[19936]: 24T21:50:35.883617+0000 mon.a (mon.0) 2021-03-24T21:50:36.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:36 smithi125 conmon[19936]: 360 : audit [INF] from='client.24256 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec0d43aa-6905-43f0-9e32-e19d4cf5129d"}]: dispatch 2021-03-24T21:50:36.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:36 smithi125 conmon[19936]: audit 2021-03-24T21:50:35. 2021-03-24T21:50:36.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:36 smithi125 conmon[19936]: 894132+0000 mon.a (mon.0) 361 : audit [INF] 2021-03-24T21:50:36.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:36 smithi125 conmon[19936]: from='client.24256 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ec0d43aa-6905-43f0-9e32-e19d4cf5129d"}]': finished 2021-03-24T21:50:36.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:36 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:50:36.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:36 smithi125 conmon[19936]: 03-24T21:50:35.894320+0000 mon.a (mon.0) 362 : cluster 2021-03-24T21:50:36.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:36 smithi125 conmon[19936]: [DBG] osdmap e35: 6 total, 5 up, 6 in 2021-03-24T21:50:36.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:36 smithi125 conmon[19936]: audit 2021-03 2021-03-24T21:50:36.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:36 smithi125 conmon[19936]: -24T21:50:35.894662+0000 mon.a (mon.0) 363 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-03-24T21:50:36.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:36 smithi125 conmon[19936]: 2021-03-24T21:50:36.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:36 smithi125 conmon[24927]: cluster 2021-03-24T21:50 2021-03-24T21:50:36.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:36 smithi125 conmon[24927]: :34.613306+0000 mgr.y (mgr.14144) 140 : cluster [DBG] pgmap v129: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-24T21:50:36.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:36 smithi125 conmon[24927]: audit 2021-03-24T21:50:36.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:36 smithi125 conmon[24927]: 2021-03-24T21:50: 2021-03-24T21:50:36.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:36 smithi125 conmon[24927]: 35.879868+0000 mon.b (mon.1) 11 2021-03-24T21:50:36.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:36 smithi125 conmon[24927]: : audit [INF] from='client.? v1:172.21.15.142:0/967325557' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec0d43aa-6905-43f0-9e32-e19d4cf5129d"}]: dispatch 2021-03-24T21:50:36.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:36 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T21:50:36.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:36 smithi125 conmon[24927]: 24T21:50:35.883617+0000 mon.a 2021-03-24T21:50:36.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:36 smithi125 conmon[24927]: (mon.0) 360 : audit [INF] from='client.24256 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec0d43aa-6905-43f0-9e32-e19d4cf5129d"}]: dispatch 2021-03-24T21:50:36.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:36 smithi125 conmon[24927]: 2021-03-24T21:50:36.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:36 smithi125 conmon[24927]: audit 2021-03-24T21:50:36.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:36 smithi125 conmon[24927]: 2021-03-24T21:50:35.894132+0000 mon.a 2021-03-24T21:50:36.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:36 smithi125 conmon[24927]: (mon.0) 361 : audit [INF] from='client.24256 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ec0d43aa-6905-43f0-9e32-e19d4cf5129d"}]': finished 2021-03-24T21:50:36.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:36 smithi125 conmon[24927]: cluster 2021-03-24T21:50:35.894320+0000 2021-03-24T21:50:36.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:36 smithi125 conmon[24927]: mon.a (mon.0) 362 : cluster [DBG] osdmap e35: 6 total, 5 up, 6 in 2021-03-24T21:50:36.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:36 smithi125 conmon[24927]: audit 2021-03-24T21:50:35 2021-03-24T21:50:36.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:36 smithi125 conmon[24927]: .894662+0000 mon.a (mon.0) 363 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-03-24T21:50:36.611 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:36 smithi142 conmon[20125]: cluster 2021-03-24T21:50: 2021-03-24T21:50:36.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:36 smithi142 conmon[20125]: 34.613306+0000 mgr.y (mgr.14144) 140 : cluster [DBG] pgmap v129: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-24T21:50:36.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:36 smithi142 conmon[20125]: audit 2021-03-24 2021-03-24T21:50:36.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:36 smithi142 conmon[20125]: T21:50:35.879868+0000 mon.b (mon.1) 11 2021-03-24T21:50:36.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:36 smithi142 conmon[20125]: : audit [INF] from='client.? v1:172.21.15.142:0/967325557' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec0d43aa-6905-43f0-9e32-e19d4cf5129d"}]: dispatch 2021-03-24T21:50:36.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:36 smithi142 conmon[20125]: 2021-03-24T21:50:36.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:36 smithi142 conmon[20125]: audit 2021-03-24T21:50:36.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:36 smithi142 conmon[20125]: 2021-03-24T21:50: 2021-03-24T21:50:36.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:36 smithi142 conmon[20125]: 35.883617+0000 mon.a (mon 2021-03-24T21:50:36.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:36 smithi142 conmon[20125]: .0) 360 : audit [INF] 2021-03-24T21:50:36.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:36 smithi142 conmon[20125]: from='client.24256 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec0d43aa-6905-43f0-9e32-e19d4cf5129d"}]: dispatch 2021-03-24T21:50:36.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:36 smithi142 conmon[20125]: audit 2021-03-24T21 2021-03-24T21:50:36.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:36 smithi142 conmon[20125]: :50:35.894132+0000 mon.a 2021-03-24T21:50:36.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:36 smithi142 conmon[20125]: (mon.0) 361 : audit 2021-03-24T21:50:36.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:36 smithi142 conmon[20125]: [INF] from='client.24256 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ec0d43aa-6905-43f0-9e32-e19d4cf5129d"}]': finished 2021-03-24T21:50:36.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:36 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:50:36.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:36 smithi142 conmon[20125]: 24T21:50:35.894320 2021-03-24T21:50:36.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:36 smithi142 conmon[20125]: +0000 mon.a (mon.0) 362 2021-03-24T21:50:36.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:36 smithi142 conmon[20125]: : cluster [DBG] osdmap e35: 6 total, 5 up, 6 in 2021-03-24T21:50:36.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:36 smithi142 conmon[20125]: audit 2021 2021-03-24T21:50:36.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:36 smithi142 conmon[20125]: -03-24T21:50:35 2021-03-24T21:50:36.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:36 smithi142 conmon[20125]: .894662+0000 mon.a (mon.0 2021-03-24T21:50:36.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:36 smithi142 conmon[20125]: ) 363 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-03-24T21:50:37.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:37 smithi125 conmon[19936]: audit 2021-03-24T21:50: 2021-03-24T21:50:37.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:37 smithi125 conmon[19936]: 36.514554+0000 mon.b (mon.1) 12 : audit [DBG] from='client.? v1:172.21.15.142:0/2874186890' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-24T21:50:37.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:37 smithi125 conmon[24927]: audit 2021-03-24T21:50:36 2021-03-24T21:50:37.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:37 smithi125 conmon[24927]: .514554+0000 mon.b (mon.1) 12 : audit [DBG] from='client.? v1:172.21.15.142:0/2874186890' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-24T21:50:37.611 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:37 smithi142 conmon[20125]: audit 2021-03-24T21:50:36.514554+0000 mon.b (mon.1) 12 : 2021-03-24T21:50:37.611 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:37 smithi142 conmon[20125]: audit [DBG] from='client.? v1:172.21.15.142:0/2874186890' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-24T21:50:38.401 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:38 smithi125 conmon[24927]: cluster 2021-03-24T21:50 2021-03-24T21:50:38.401 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:38 smithi125 conmon[24927]: :36.613794+0000 mgr.y (mgr.14144) 141 : cluster [DBG] pgmap v131: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-24T21:50:38.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:38 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:50:38.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:38 smithi125 conmon[19936]: 21:50:36.613794+0000 mgr.y (mgr.14144) 141 : cluster [DBG] pgmap v131: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-24T21:50:38.611 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:38 smithi142 conmon[20125]: cluster 2021-03-24T21:50:36 2021-03-24T21:50:38.611 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:38 smithi142 conmon[20125]: .613794+0000 mgr.y (mgr.14144) 141 : cluster [DBG] pgmap v131: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-24T21:50:40.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:40 smithi125 conmon[19936]: cluster 2021-03-24T21:50: 2021-03-24T21:50:40.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:40 smithi125 conmon[19936]: 38.614330+0000 mgr.y (mgr.14144) 142 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-24T21:50:40.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:40 smithi125 conmon[24927]: cluster 2021-03-24T21:50:38. 2021-03-24T21:50:40.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:40 smithi125 conmon[24927]: 614330+0000 mgr.y (mgr.14144) 142 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-24T21:50:40.416 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:40 smithi142 conmon[20125]: cluster 2021-03-24T21:50:38.614330+0000 mgr.y (mgr.14144) 2021-03-24T21:50:40.417 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:40 smithi142 conmon[20125]: 142 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-24T21:50:42.400 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:42 smithi125 conmon[19936]: cluster 2021-03-24T21:50:40.614875+0000 mgr.y (mgr.14144) 143 2021-03-24T21:50:42.514 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:42 smithi125 conmon[19936]: : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-24T21:50:42.515 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:42 smithi125 conmon[24927]: cluster 2021-03-24T21:50:40.614875+0000 2021-03-24T21:50:42.515 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:42 smithi125 conmon[24927]: mgr.y (mgr.14144) 143 : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-24T21:50:42.611 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:42 smithi142 conmon[20125]: cluster 2021-03-24T21:50:40.614875+0000 mgr.y (mgr.14144) 143 : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-24T21:50:43.458 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:43 smithi142 conmon[20125]: audit 2021-03-24T21:50:42 2021-03-24T21:50:43.458 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:43 smithi142 conmon[20125]: .892446+0000 mon.a (mon.0) 364 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2021-03-24T21:50:43.458 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:43 smithi142 conmon[20125]: audit 2021-03-24T21:50:43.459 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:43 smithi142 conmon[20125]: 2021-03-24T21:50: 2021-03-24T21:50:43.459 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:43 smithi142 conmon[20125]: 42.894259+0000 mon.a (mon 2021-03-24T21:50:43.459 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:43 smithi142 conmon[20125]: .0) 365 : audit 2021-03-24T21:50:43.459 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:43 smithi142 conmon[20125]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:50:43.650 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:43 smithi125 conmon[19936]: audit 2021-03-24T21:50:42 2021-03-24T21:50:43.651 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:43 smithi125 conmon[19936]: .892446+0000 mon.a (mon.0) 364 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2021-03-24T21:50:43.651 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:43 smithi125 conmon[19936]: audit 2021-03-24T21:50:42.894259+0000 mon.a (mon.0 2021-03-24T21:50:43.651 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:43 smithi125 conmon[19936]: ) 365 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:50:43.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:43 smithi125 conmon[24927]: audit 2021-03-24T21:50:42 2021-03-24T21:50:43.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:43 smithi125 conmon[24927]: .892446+0000 mon.a (mon.0) 364 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2021-03-24T21:50:43.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:43 smithi125 conmon[24927]: audit 2021-03-24T21:50:43.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:43 smithi125 conmon[24927]: 2021-03-24T21:50:42.894259 2021-03-24T21:50:43.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:43 smithi125 conmon[24927]: +0000 mon.a (mon.0) 365 : audit 2021-03-24T21:50:43.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:43 smithi125 conmon[24927]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:50:44.426 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:44 smithi142 conmon[20125]: cluster 2021-03-24T21:50:42.615382 2021-03-24T21:50:44.427 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:44 smithi142 conmon[20125]: +0000 mgr.y (mgr.14144) 144 : cluster [DBG] pgmap v134: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-24T21:50:44.427 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:44 smithi142 conmon[20125]: cephadm 2021-03-24T21:50:42.895772+0000 mgr.y (mgr. 2021-03-24T21:50:44.427 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:44 smithi142 conmon[20125]: 14144) 145 : cephadm [INF] Deploying daemon osd.5 on smithi142 2021-03-24T21:50:44.651 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:44 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:50:44.651 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:44 smithi125 conmon[19936]: :50:42.615382+0000 mgr.y (mgr.14144) 144 : cluster [DBG] pgmap v134: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-24T21:50:44.651 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:44 smithi125 conmon[19936]: cephadm 2021-03-24T21:50:44.651 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:44 smithi125 conmon[19936]: 2021-03-24T21:50:42.895772 2021-03-24T21:50:44.652 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:44 smithi125 conmon[19936]: +0000 mgr.y (mgr.14144) 145 : 2021-03-24T21:50:44.652 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:44 smithi125 conmon[19936]: cephadm [INF] Deploying daemon osd.5 on smithi142 2021-03-24T21:50:44.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:44 smithi125 conmon[24927]: cluster 2021-03-24T21:50 2021-03-24T21:50:44.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:44 smithi125 conmon[24927]: :42.615382+0000 mgr.y (mgr.14144) 144 : cluster [DBG] pgmap v134: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-24T21:50:44.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:44 smithi125 conmon[24927]: cephadm 2021-03 2021-03-24T21:50:44.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:44 smithi125 conmon[24927]: -24T21:50:42.895772+0000 mgr.y 2021-03-24T21:50:44.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:44 smithi125 conmon[24927]: (mgr.14144) 145 : cephadm [INF] Deploying daemon osd.5 on smithi142 2021-03-24T21:50:44.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:44 smithi125 conmon[24927]: 2021-03-24T21:50:46.361 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:46 smithi142 conmon[20125]: cluster 2021-03-24T21:50:44.615881+0000 mgr.y (mgr.14144) 146 : 2021-03-24T21:50:46.361 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:46 smithi142 conmon[20125]: cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-24T21:50:46.650 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:46 smithi125 conmon[19936]: cluster 2021-03-24T21:50:44.615881 2021-03-24T21:50:46.651 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:46 smithi125 conmon[19936]: +0000 mgr.y (mgr.14144) 146 : cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-24T21:50:46.651 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:46 smithi125 conmon[24927]: cluster 2021-03-24T21:50:44. 2021-03-24T21:50:46.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:46 smithi125 conmon[24927]: 615881+0000 mgr.y (mgr.14144) 146 : cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-24T21:50:47.249 INFO:teuthology.orchestra.run.smithi142.stdout:Created osd(s) 5 on host 'smithi142' 2021-03-24T21:50:48.014 DEBUG:teuthology.orchestra.run.smithi142:osd.5> sudo journalctl -f -n 0 -u ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c@osd.5.service 2021-03-24T21:50:48.018 INFO:tasks.cephadm:Deploying osd.6 on smithi142 with /dev/vg_nvme/lv_2... 2021-03-24T21:50:48.019 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2021-03-24T21:50:48.056 INFO:journalctl@ceph.osd.5.smithi142.stdout:-- Logs begin at Wed 2021-03-03 23:51:07 UTC. -- 2021-03-24T21:50:48.361 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:48 smithi142 conmon[20125]: cluster 2021-03-24T21:50:46.616400+0000 2021-03-24T21:50:48.361 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:48 smithi142 conmon[20125]: mgr.y (mgr.14144) 147 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-24T21:50:48.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:48 smithi142 conmon[20125]: audit 2021-03-24T21:50: 2021-03-24T21:50:48.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:48 smithi142 conmon[20125]: 47.239640+0000 mon.a (mon.0) 366 : audit [DBG] 2021-03-24T21:50:48.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:48 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:50:48.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:48 smithi142 conmon[20125]: audit 2021 2021-03-24T21:50:48.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:48 smithi142 conmon[20125]: -03-24T21:50:47. 2021-03-24T21:50:48.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:48 smithi142 conmon[20125]: 248479+0000 mon.a (mon.0) 2021-03-24T21:50:48.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:48 smithi142 conmon[20125]: 367 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:48.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:48 smithi142 conmon[20125]: audit 2021 2021-03-24T21:50:48.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:48 smithi142 conmon[20125]: -03-24T21:50:47 2021-03-24T21:50:48.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:48 smithi142 conmon[20125]: .257363+0000 mon.a (mon. 2021-03-24T21:50:48.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:48 smithi142 conmon[20125]: 0) 368 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:48.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:48 smithi142 conmon[20125]: 2021-03-24T21:50:48.651 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:48 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:50:48.651 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:48 smithi125 conmon[19936]: 21:50:46.616400+0000 mgr.y (mgr.14144) 147 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-24T21:50:48.651 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:48 smithi125 conmon[19936]: audit 2021-03-24T21:50:48.651 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:48 smithi125 conmon[19936]: 2021-03-24T21:50:47.239640+0000 mon.a (mon.0) 366 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:50:48.652 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:48 smithi125 conmon[19936]: audit 2021-03-24T21:50:47.248479+0000 2021-03-24T21:50:48.652 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:48 smithi125 conmon[19936]: mon.a (mon.0) 367 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:48.652 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:48 smithi125 conmon[19936]: audit 2021-03-24T21:50:47.257363+0000 mon.a (mon.0 2021-03-24T21:50:48.652 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:48 smithi125 conmon[19936]: ) 368 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:48.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:48 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:50:48.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:48 smithi125 conmon[24927]: 21:50:46.616400+0000 mgr.y (mgr.14144) 147 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-24T21:50:48.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:48 smithi125 conmon[24927]: audit 2021-03-24T21:50:48.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:48 smithi125 conmon[24927]: 2021-03-24T21:50:47 2021-03-24T21:50:48.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:48 smithi125 conmon[24927]: .239640+0000 mon.a (mon.0) 366 : audit 2021-03-24T21:50:48.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:48 smithi125 conmon[24927]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:50:48.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:48 smithi125 conmon[24927]: audit 2021-03-24T21:50 2021-03-24T21:50:48.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:48 smithi125 conmon[24927]: :47.248479+0000 mon.a (mon.0) 367 : 2021-03-24T21:50:48.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:48 smithi125 conmon[24927]: audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:48.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:48 smithi125 conmon[24927]: audit 2021-03-24T21:50:48.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:48 smithi125 conmon[24927]: 2021-03-24 2021-03-24T21:50:48.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:48 smithi125 conmon[24927]: T21:50:47.257363+0000 mon.a (mon.0) 2021-03-24T21:50:48.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:48 smithi125 conmon[24927]: 368 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:50.442 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:50 smithi142 conmon[20125]: cluster 2021-03-24T21:50:48.616943+0000 mgr.y (mgr.14144) 148 : cluster [DBG] pgmap v137: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-24T21:50:50.442 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:50 smithi142 conmon[20125]: cluster 2021-03-24T21:50:49.050205+0000 osd.2 (osd.2) 3 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:50:50.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:50 smithi142 conmon[20125]: cluster 2021-03-24T21:50:49.050396+0000 osd.2 (osd.2) 4 : cluster 2021-03-24T21:50:50.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:50 smithi142 conmon[20125]: [DBG] purged_snaps scrub ok 2021-03-24T21:50:50.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:50 smithi142 conmon[20125]: cluster 2021-03-24T21:50:49.193588+0000 mon.a ( 2021-03-24T21:50:50.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:50 smithi142 conmon[20125]: mon.0) 369 : cluster [DBG] osdmap e36: 6 total, 5 up, 6 in 2021-03-24T21:50:50.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:50 smithi142 conmon[20125]: audit 2021-03-24T21 2021-03-24T21:50:50.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:50 smithi142 conmon[20125]: :50:49.193793+0000 mon.a (mon.0) 370 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-03-24T21:50:50.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:50 smithi142 conmon[20125]: audit 2021-03-24T21:50:49.305509+0000 2021-03-24T21:50:50.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:50 smithi142 conmon[20125]: mon.a (mon.0) 371 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:50.651 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:50 smithi125 conmon[24927]: cluster 2021-03-24T21:50:50.651 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:50 smithi125 conmon[24927]: 2021-03 2021-03-24T21:50:50.651 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:50 smithi125 conmon[24927]: -24T21:50:48.616943+0000 mgr.y (mgr.14144) 148 : cluster [DBG] 2021-03-24T21:50:50.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:50 smithi125 conmon[24927]: pgmap v137: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-24T21:50:50.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:50 smithi125 conmon[24927]: cluster 2021-03-24T21:50:50.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:50 smithi125 conmon[24927]: 2021-03-24 2021-03-24T21:50:50.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:50 smithi125 conmon[24927]: T21:50:49.050205+0000 osd.2 (osd.2) 2021-03-24T21:50:50.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:50 smithi125 conmon[24927]: 3 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:50:50.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:50 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:50:50.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:50 smithi125 conmon[24927]: -03-24T21:50:49.050396+0000 osd.2 ( 2021-03-24T21:50:50.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:50 smithi125 conmon[24927]: osd.2) 4 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:50:50.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:50 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:50:50.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:50 smithi125 conmon[24927]: -03-24T21:50:49.193588+0000 mon.a 2021-03-24T21:50:50.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:50 smithi125 conmon[24927]: (mon.0) 369 : cluster [DBG] osdmap e36: 6 total, 5 up, 6 in 2021-03-24T21:50:50.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:50 smithi125 conmon[24927]: audit 2021-03-24T21:50:50.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:50 smithi125 conmon[24927]: 2021-03-24T21:50:49.193793+0000 2021-03-24T21:50:50.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:50 smithi125 conmon[24927]: mon.a (mon.0) 370 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-03-24T21:50:50.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:50 smithi125 conmon[24927]: 2021-03-24T21:50:50.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:50 smithi125 conmon[24927]: audit 2021-03-24T21:50:50.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:50 smithi125 conmon[24927]: 2021-03-24T21:50:49.305509+0000 mon.a ( 2021-03-24T21:50:50.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:50 smithi125 conmon[24927]: mon.0) 371 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:50.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:50 smithi125 conmon[19936]: cluster 2021-03-24T21:50 2021-03-24T21:50:50.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:50 smithi125 conmon[19936]: :48.616943+0000 mgr.y (mgr.14144) 148 : cluster [DBG] pgmap v137: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-24T21:50:50.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:50 smithi125 conmon[19936]: cluster 2021-03-24T21:50 2021-03-24T21:50:50.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:50 smithi125 conmon[19936]: :49.050205+0000 osd.2 (osd.2) 3 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:50:50.657 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:50 smithi125 conmon[19936]: cluster 2021-03-24T21:50:50.657 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:50 smithi125 conmon[19936]: 2021-03-24T21:50 2021-03-24T21:50:50.657 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:50 smithi125 conmon[19936]: :49.050396+0000 osd.2 (osd.2 2021-03-24T21:50:50.657 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:50 smithi125 conmon[19936]: ) 4 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:50:50.657 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:50 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:50:50.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:50 smithi125 conmon[19936]: -03-24T21:50: 2021-03-24T21:50:50.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:50 smithi125 conmon[19936]: 49.193588+0000 mon.a (mon.0) 369 2021-03-24T21:50:50.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:50 smithi125 conmon[19936]: : cluster [DBG] osdmap e36: 6 total, 5 up, 6 in 2021-03-24T21:50:50.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:50 smithi125 conmon[19936]: audit 2021-03 2021-03-24T21:50:50.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:50 smithi125 conmon[19936]: -24T21:50:49.193793 2021-03-24T21:50:50.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:50 smithi125 conmon[19936]: +0000 mon.a (mon.0) 370 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-03-24T21:50:50.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:50 smithi125 conmon[19936]: 2021-03-24T21:50:50.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:50 smithi125 conmon[19936]: audit 2021-03-24T21:50:50.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:50 smithi125 conmon[19936]: 2021-03-24T21:50:49.305509+0000 mon.a (mon.0) 371 : 2021-03-24T21:50:50.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:50 smithi125 conmon[19936]: audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:51.111 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 21:50:50 smithi142 conmon[27337]: debug 2021-03-24T21:50:50.804+0000 7f54c2aaa080 -1 osd.5 0 log_to_monitors {default=true} 2021-03-24T21:50:51.335 INFO:teuthology.orchestra.run.smithi142.stderr:--> Zapping: /dev/vg_nvme/lv_2 2021-03-24T21:50:51.335 INFO:teuthology.orchestra.run.smithi142.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2021-03-24T21:50:51.336 INFO:teuthology.orchestra.run.smithi142.stderr: stderr: 10+0 records in 2021-03-24T21:50:51.336 INFO:teuthology.orchestra.run.smithi142.stderr:10+0 records out 2021-03-24T21:50:51.336 INFO:teuthology.orchestra.run.smithi142.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0165474 s, 634 MB/s 2021-03-24T21:50:51.336 INFO:teuthology.orchestra.run.smithi142.stderr:--> Zapping successful for: 2021-03-24T21:50:51.594 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:51 smithi142 conmon[20125]: audit 2021-03-24T21:50:50.813723+0000 mon.b (mon.1) 13 : audit 2021-03-24T21:50:51.594 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:51 smithi142 conmon[20125]: [INF] from='osd.5 v1:172.21.15.142:6804/396400804' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-03-24T21:50:51.595 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:51 smithi142 conmon[20125]: audit 2021-03-24T21:50:50.817192+0000 mon.a (mon.0) 372 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-03-24T21:50:51.650 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:51 smithi125 conmon[19936]: audit 2021-03-24T21:50:50 2021-03-24T21:50:51.651 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:51 smithi125 conmon[19936]: .813723+0000 mon.b (mon.1) 13 : audit [INF] from='osd.5 v1:172.21.15.142:6804/396400804' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-03-24T21:50:51.651 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:51 smithi125 conmon[19936]: audit 2021-03-24T 2021-03-24T21:50:51.651 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:51 smithi125 conmon[19936]: 21:50:50.817192+0000 mon.a (mon.0) 372 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-03-24T21:50:51.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:51 smithi125 conmon[24927]: audit 2021-03-24T21:50:50 2021-03-24T21:50:51.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:51 smithi125 conmon[24927]: .813723+0000 mon.b (mon.1) 13 : audit [INF] from='osd.5 v1:172.21.15.142:6804/396400804' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-03-24T21:50:51.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:51 smithi125 conmon[24927]: audit 2021 2021-03-24T21:50:51.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:51 smithi125 conmon[24927]: -03-24T21:50:50.817192+0000 mon.a (mon.0) 372 2021-03-24T21:50:51.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:51 smithi125 conmon[24927]: : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-03-24T21:50:52.319 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph orch daemon add osd smithi142:vg_nvme/lv_2 2021-03-24T21:50:52.360 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 21:50:52 smithi142 conmon[27337]: debug 2021-03-24T21:50:52.212+0000 7f54b901b700 -1 osd.5 0 waiting for initial osdmap 2021-03-24T21:50:52.361 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:52 smithi142 conmon[20125]: cluster 2021-03-24T21:50:50. 2021-03-24T21:50:52.361 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:52 smithi142 conmon[20125]: 617462+0000 mgr.y (mgr.14144) 149 : cluster [DBG] pgmap v139: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-24T21:50:52.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:52 smithi142 conmon[20125]: audit 2021-03-24T21:50:51.203925+0000 mon.b (mon.1) 14 : audit 2021-03-24T21:50:52.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:52 smithi142 conmon[20125]: [INF] from='osd.5 v1:172.21.15.142:6804/396400804' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2021-03-24T21:50:52.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:52 smithi142 conmon[20125]: audit 2021-03-24T21:50:51.205746+0000 mon.a 2021-03-24T21:50:52.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:52 smithi142 conmon[20125]: (mon.0) 373 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2021-03-24T21:50:52.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:52 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:50:52.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:52 smithi142 conmon[20125]: 03-24T21:50:51.205867+0000 mon.a (mon.0) 374 : cluster [DBG] osdmap e37: 6 total, 5 up, 6 in 2021-03-24T21:50:52.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:52 smithi142 conmon[20125]: audit 2021-03-24T21:50:51.206040 2021-03-24T21:50:52.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:52 smithi142 conmon[20125]: +0000 mon.a (mon.0) 375 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-03-24T21:50:52.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:52 smithi142 conmon[20125]: audit 2021-03-24T21:50:51.207142+0000 mon.a (mon. 2021-03-24T21:50:52.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:52 smithi142 conmon[20125]: 0) 376 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2021-03-24T21:50:52.651 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:52 smithi125 conmon[24927]: cluster 2021-03-24T21:50:50.617462+0000 mgr.y (mgr 2021-03-24T21:50:52.711 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:52 smithi125 conmon[24927]: .14144) 149 : cluster [DBG] pgmap v139: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-24T21:50:52.711 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:52 smithi125 conmon[24927]: audit 2021-03-24T21:50:51.203925+0000 mon.b (mon 2021-03-24T21:50:52.711 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:52 smithi125 conmon[24927]: .1) 14 : audit [INF] from='osd.5 v1:172.21.15.142:6804/396400804' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2021-03-24T21:50:52.711 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:52 smithi125 conmon[24927]: audit 2021 2021-03-24T21:50:52.712 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:52 smithi125 conmon[24927]: -03-24T21:50:51.205746+0000 mon.a (mon.0) 373 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2021-03-24T21:50:52.712 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:52 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:50:52.712 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:52 smithi125 conmon[24927]: -03-24T21:50:51.205867+0000 mon.a (mon.0) 374 : cluster 2021-03-24T21:50:52.712 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:52 smithi125 conmon[24927]: [DBG] osdmap e37: 6 total, 5 up, 6 in 2021-03-24T21:50:52.713 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:52 smithi125 conmon[24927]: audit 2021-03-24T21:50:51.206040+0000 mon.a ( 2021-03-24T21:50:52.713 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:52 smithi125 conmon[24927]: mon.0) 375 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-03-24T21:50:52.713 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:52 smithi125 conmon[24927]: audit 2021-03-24T21:50:51.207142 2021-03-24T21:50:52.713 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:52 smithi125 conmon[24927]: +0000 mon.a (mon.0) 376 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2021-03-24T21:50:52.714 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:52 smithi125 conmon[19936]: cluster 2021-03-24T21:50:50 2021-03-24T21:50:52.714 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:52 smithi125 conmon[19936]: .617462+0000 mgr.y (mgr.14144) 149 : cluster [DBG] pgmap v139: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-24T21:50:52.714 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:52 smithi125 conmon[19936]: audit 2021-03-24 2021-03-24T21:50:52.714 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:52 smithi125 conmon[19936]: T21:50:51.203925+0000 mon.b (mon.1) 14 : audit 2021-03-24T21:50:52.715 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:52 smithi125 conmon[19936]: [INF] from='osd.5 v1:172.21.15.142:6804/396400804' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2021-03-24T21:50:52.715 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:52 smithi125 conmon[19936]: audit 2021-03-24T21: 2021-03-24T21:50:52.715 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:52 smithi125 conmon[19936]: 50:51.205746+0000 mon.a (mon 2021-03-24T21:50:52.716 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:52 smithi125 conmon[19936]: .0) 373 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2021-03-24T21:50:52.716 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:52 smithi125 conmon[19936]: 2021-03-24T21:50:52.716 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:52 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:50:52.716 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:52 smithi125 conmon[19936]: -03-24T21:50:51.205867+0000 2021-03-24T21:50:52.716 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:52 smithi125 conmon[19936]: mon.a (mon.0) 374 : cluster 2021-03-24T21:50:52.717 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:52 smithi125 conmon[19936]: [DBG] osdmap e37: 6 total, 5 up, 6 in 2021-03-24T21:50:52.717 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:52 smithi125 conmon[19936]: audit 2021-03-24T21: 2021-03-24T21:50:52.717 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:52 smithi125 conmon[19936]: 50:51.206040+0000 mon.a (mon 2021-03-24T21:50:52.717 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:52 smithi125 conmon[19936]: .0) 375 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-03-24T21:50:52.717 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:52 smithi125 conmon[19936]: 2021-03-24T21:50:52.718 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:52 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:50:52.718 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:52 smithi125 conmon[19936]: 03-24T21:50:51.207142+0000 2021-03-24T21:50:52.718 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:52 smithi125 conmon[19936]: mon.a (mon.0) 376 : audit [INF] 2021-03-24T21:50:52.718 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:52 smithi125 conmon[19936]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2021-03-24T21:50:53.361 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:53 smithi142 conmon[20125]: audit 2021-03-24T21:50:52.210318+0000 2021-03-24T21:50:53.361 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:53 smithi142 conmon[20125]: mon.a (mon.0) 377 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2021-03-24T21:50:53.361 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:53 smithi142 conmon[20125]: cluster 2021-03-24T21:50:52.210584+0000 mon.a ( 2021-03-24T21:50:53.361 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:53 smithi142 conmon[20125]: mon.0) 378 : cluster [DBG] osdmap e38: 6 total, 5 up, 6 in 2021-03-24T21:50:53.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:53 smithi142 conmon[20125]: audit 2021-03-24T21:50:53.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:53 smithi142 conmon[20125]: 2021-03-24T21:50:52.211370+0000 mon.a (mon 2021-03-24T21:50:53.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:53 smithi142 conmon[20125]: .0) 379 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-03-24T21:50:53.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:53 smithi142 conmon[20125]: audit 2021-03-24T21:50:53.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:53 smithi142 conmon[20125]: 2021-03-24T21:50:52. 2021-03-24T21:50:53.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:53 smithi142 conmon[20125]: 218768+0000 mon.a (mon.0) 380 2021-03-24T21:50:53.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:53 smithi142 conmon[20125]: : audit [DBG] 2021-03-24T21:50:53.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:53 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-03-24T21:50:53.651 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:53 smithi125 conmon[19936]: audit 2021-03-24T21:50:52.210318+0000 mon.a (mon.0) 377 2021-03-24T21:50:53.651 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:53 smithi125 conmon[19936]: : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2021-03-24T21:50:53.651 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:53 smithi125 conmon[19936]: cluster 2021-03-24T21:50:52.210584+0000 mon.a (mon.0) 378 : cluster [DBG] osdmap e38: 6 total, 5 up, 6 in 2021-03-24T21:50:53.652 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:53 smithi125 conmon[19936]: audit 2021-03-24T21:50:52.211370+0000 2021-03-24T21:50:53.652 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:53 smithi125 conmon[19936]: mon.a (mon.0) 379 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-03-24T21:50:53.652 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:53 smithi125 conmon[19936]: audit 2021-03-24T21 2021-03-24T21:50:53.652 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:53 smithi125 conmon[19936]: :50:52.218768+0000 mon.a (mon.0) 380 : audit 2021-03-24T21:50:53.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:53 smithi125 conmon[19936]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-03-24T21:50:53.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:53 smithi125 conmon[24927]: audit 2021-03-24T21:50:52.210318+0000 mon.a (mon.0) 2021-03-24T21:50:53.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:53 smithi125 conmon[24927]: 377 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2021-03-24T21:50:53.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:53 smithi125 conmon[24927]: cluster 2021-03-24T21:50:52.210584+0000 mon.a (mon.0) 378 : cluster [DBG] 2021-03-24T21:50:53.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:53 smithi125 conmon[24927]: osdmap e38: 6 total, 5 up, 6 in 2021-03-24T21:50:53.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:53 smithi125 conmon[24927]: audit 2021-03-24T21:50:52.211370+0000 mon.a (mon. 2021-03-24T21:50:53.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:53 smithi125 conmon[24927]: 0) 379 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-03-24T21:50:53.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:53 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T21:50:53.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:53 smithi125 conmon[24927]: 24T21:50:52.218768+0000 mon.a (mon.0) 380 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-03-24T21:50:54.508 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:54 smithi142 conmon[20125]: cluster 2021-03-24T21:50:51. 2021-03-24T21:50:54.508 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:54 smithi142 conmon[20125]: 840403+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:50:54.508 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:54 smithi142 conmon[20125]: cluster 2021-03-24T21:50:51.840623 2021-03-24T21:50:54.509 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:54 smithi142 conmon[20125]: +0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:50:54.509 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:54 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:50:54.509 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:54 smithi142 conmon[20125]: 21:50:52.617999+0000 mgr.y (mgr.14144) 150 2021-03-24T21:50:54.509 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:54 smithi142 conmon[20125]: : cluster [DBG] pgmap v142: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-24T21:50:54.509 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:54 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:50:54.510 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:54 smithi142 conmon[20125]: T21:50:53.212377+0000 mon.a (mon. 2021-03-24T21:50:54.510 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:54 smithi142 conmon[20125]: 0) 381 : cluster [INF] osd.5 v1:172.21.15.142:6804/396400804 boot 2021-03-24T21:50:54.510 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:54 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:50:54.510 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:54 smithi142 conmon[20125]: 21:50:53.212547+0000 mon.a (mon.0 2021-03-24T21:50:54.511 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:54 smithi142 conmon[20125]: ) 382 : cluster [DBG] osdmap e39: 6 total, 6 up, 6 in 2021-03-24T21:50:54.511 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:54 smithi142 conmon[20125]: audit 2021 2021-03-24T21:50:54.511 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:54 smithi142 conmon[20125]: -03-24T21:50:53.212944+0000 mon.a 2021-03-24T21:50:54.511 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:54 smithi142 conmon[20125]: (mon.0) 383 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-03-24T21:50:54.511 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:54 smithi142 conmon[20125]: cluster 2021-03-24T21:50:54.512 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:54 smithi142 conmon[20125]: 2021-03-24T21:50 2021-03-24T21:50:54.512 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:54 smithi142 conmon[20125]: :54.222594+0000 mon.a (mon.0) 384 : 2021-03-24T21:50:54.512 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:54 smithi142 conmon[20125]: cluster [DBG] osdmap e40: 6 total, 6 up, 6 in 2021-03-24T21:50:54.651 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:54 smithi125 conmon[24927]: cluster 2021-03-24T21:50:51.840403+0000 2021-03-24T21:50:54.651 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:54 smithi125 conmon[24927]: osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:50:54.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:54 smithi125 conmon[24927]: cluster 2021-03-24T21:50:51.840623+0000 2021-03-24T21:50:54.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:54 smithi125 conmon[24927]: osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:50:54.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:54 smithi125 conmon[24927]: cluster 2021-03-24T21:50:52. 2021-03-24T21:50:54.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:54 smithi125 conmon[24927]: 617999+0000 mgr.y (mgr.14144) 150 : cluster [DBG] pgmap v142: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-24T21:50:54.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:54 smithi125 conmon[24927]: cluster 2021-03-24T21:50:54.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:54 smithi125 conmon[24927]: 2021-03-24T21:50:53.212377+0000 mon.a (mon.0) 2021-03-24T21:50:54.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:54 smithi125 conmon[24927]: 381 : cluster [INF] osd.5 v1:172.21.15.142:6804/396400804 boot 2021-03-24T21:50:54.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:54 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:50:54.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:54 smithi125 conmon[24927]: 50:53.212547+0000 mon.a (mon.0) 382 : cluster [DBG] osdmap e39: 6 total, 6 up, 6 in 2021-03-24T21:50:54.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:54 smithi125 conmon[24927]: audit 2021-03-24T21: 2021-03-24T21:50:54.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:54 smithi125 conmon[24927]: 50:53.212944+0000 mon.a (mon.0) 383 : 2021-03-24T21:50:54.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:54 smithi125 conmon[24927]: audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-03-24T21:50:54.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:54 smithi125 conmon[24927]: cluster 2021-03-24T21:50:54.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:54 smithi125 conmon[24927]: 2021-03-24T21:50:54.222594+0000 mon.a (mon 2021-03-24T21:50:54.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:54 smithi125 conmon[24927]: .0) 384 : cluster [DBG] osdmap e40: 6 total, 6 up, 6 in 2021-03-24T21:50:54.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:54 smithi125 conmon[19936]: cluster 2021-03-24T21:50:51. 2021-03-24T21:50:54.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:54 smithi125 conmon[19936]: 840403+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:50:54.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:54 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:50:54.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:54 smithi125 conmon[19936]: 24T21:50:51.840623+0000 osd.5 (osd.5) 2 2021-03-24T21:50:54.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:54 smithi125 conmon[19936]: : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:50:54.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:54 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:50:54.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:54 smithi125 conmon[19936]: 03-24T21:50:52.617999+0000 2021-03-24T21:50:54.657 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:54 smithi125 conmon[19936]: mgr.y (mgr.14144) 150 2021-03-24T21:50:54.657 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:54 smithi125 conmon[19936]: : cluster [DBG] pgmap v142: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-24T21:50:54.657 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:54 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:50:54.657 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:54 smithi125 conmon[19936]: 03-24T21:50:53.212377+0000 mon.a 2021-03-24T21:50:54.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:54 smithi125 conmon[19936]: (mon.0) 381 : 2021-03-24T21:50:54.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:54 smithi125 conmon[19936]: cluster [INF] osd.5 v1:172.21.15.142:6804/396400804 boot 2021-03-24T21:50:54.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:54 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:50:54.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:54 smithi125 conmon[19936]: 03-24T21:50:53. 2021-03-24T21:50:54.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:54 smithi125 conmon[19936]: 212547+0000 mon.a (mon.0) 2021-03-24T21:50:54.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:54 smithi125 conmon[19936]: 382 : cluster [DBG] 2021-03-24T21:50:54.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:54 smithi125 conmon[19936]: osdmap e39: 6 total, 6 up, 6 in 2021-03-24T21:50:54.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:54 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:50:54.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:54 smithi125 conmon[19936]: 03-24T21:50:53. 2021-03-24T21:50:54.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:54 smithi125 conmon[19936]: 212944+0000 mon.a (mon 2021-03-24T21:50:54.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:54 smithi125 conmon[19936]: .0) 383 : audit 2021-03-24T21:50:54.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:54 smithi125 conmon[19936]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-03-24T21:50:54.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:54 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:50:54.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:54 smithi125 conmon[19936]: 03-24T21:50:54.222594 2021-03-24T21:50:54.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:54 smithi125 conmon[19936]: +0000 mon.a (mon.0 2021-03-24T21:50:54.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:54 smithi125 conmon[19936]: ) 384 : cluster [DBG] 2021-03-24T21:50:54.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:54 smithi125 conmon[19936]: osdmap e40: 6 total, 6 up, 6 in 2021-03-24T21:50:56.611 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:56 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:50:56.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:56 smithi142 conmon[20125]: 24T21:50:54.618494+0000 mgr.y (mgr.14144) 151 : cluster [DBG] pgmap v145: 1 pgs: 1 remapped+peering; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:50:56.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:56 smithi142 conmon[20125]: cluster 2021-03-24T21:50:56.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:56 smithi142 conmon[20125]: 2021-03- 2021-03-24T21:50:56.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:56 smithi142 conmon[20125]: 24T21:50:55.242434+0000 mon.a (mon 2021-03-24T21:50:56.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:56 smithi142 conmon[20125]: .0) 385 : cluster [DBG] osdmap e41: 6 total, 6 up, 6 in 2021-03-24T21:50:56.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:56 smithi142 conmon[20125]: cephadm 2021 2021-03-24T21:50:56.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:56 smithi142 conmon[20125]: -03-24T21:50:55.293524+0000 2021-03-24T21:50:56.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:56 smithi142 conmon[20125]: mgr.y (mgr.14144) 152 : cephadm [INF] 2021-03-24T21:50:56.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:56 smithi142 conmon[20125]: Detected new or changed devices on smithi142 2021-03-24T21:50:56.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:56 smithi142 conmon[20125]: audit 2021-03-24T21:50: 2021-03-24T21:50:56.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:56 smithi142 conmon[20125]: 55.304289+0000 mon.a (mon.0) 386 : audit 2021-03-24T21:50:56.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:56 smithi142 conmon[20125]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:56.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:56 smithi142 conmon[20125]: audit 2021-03-24T21:50:56.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:56 smithi142 conmon[20125]: 2021-03-24T21:50:55 2021-03-24T21:50:56.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:56 smithi142 conmon[20125]: .314632+0000 mon.a (mon.0) 387 : 2021-03-24T21:50:56.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:56 smithi142 conmon[20125]: audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:56.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:56 smithi142 conmon[20125]: audit 2021-03-24T21:50:56.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:56 smithi142 conmon[20125]: 2021-03-24T21:50 2021-03-24T21:50:56.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:56 smithi142 conmon[20125]: :55.316769+0000 mon.a (mon.0) 388 2021-03-24T21:50:56.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:56 smithi142 conmon[20125]: : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:50:56.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:56 smithi142 conmon[20125]: audit 2021-03-24T 2021-03-24T21:50:56.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:56 smithi142 conmon[20125]: 21:50:55.415341+0000 mgr.y (mgr.14144) 153 : audit 2021-03-24T21:50:56.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:56 smithi142 conmon[20125]: [DBG] from='client.24277 v1:172.21.15.142:0/2277522098' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:50:56.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:56 smithi142 conmon[20125]: audit 2021-03-24T 2021-03-24T21:50:56.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:56 smithi142 conmon[20125]: 21:50:55.416747+0000 mon.a (mon.0) 389 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-24T21:50:56.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:56 smithi142 conmon[20125]: cephadm 2021-03-24T21:50:55.419145+0000 2021-03-24T21:50:56.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:56 smithi142 conmon[20125]: mgr.y (mgr.14144) 154 : cephadm [INF] Applying service osd.None on host smithi142... 2021-03-24T21:50:56.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:56 smithi142 conmon[20125]: audit 2021-03-24T21:50:55. 2021-03-24T21:50:56.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:56 smithi142 conmon[20125]: 419625+0000 mon.a (mon.0) 390 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-24T21:50:56.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:56 smithi142 conmon[20125]: audit 2021-03-24T21:50:55 2021-03-24T21:50:56.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:56 smithi142 conmon[20125]: .421166+0000 mon.a (mon.0) 391 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:50:56.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:56 smithi142 conmon[20125]: audit 2021-03-24T 2021-03-24T21:50:56.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:56 smithi142 conmon[20125]: 21:50:55.644417+0000 mon.a (mon.0) 392 : 2021-03-24T21:50:56.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:56 smithi142 conmon[20125]: audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:56.652 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[19936]: cluster 2021-03-24T21:50 2021-03-24T21:50:56.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[19936]: :54.618494+0000 mgr.y (mgr.14144) 151 : cluster [DBG] pgmap v145: 1 pgs: 1 remapped+peering; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:50:56.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[19936]: cluster 2021-03-24T21:50:56.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[19936]: 2021-03-24T 2021-03-24T21:50:56.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[19936]: 21:50:55.242434+0000 mon.a (mon 2021-03-24T21:50:56.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[19936]: .0) 385 : cluster [DBG] osdmap e41: 6 total, 6 up, 6 in 2021-03-24T21:50:56.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[19936]: cephadm 2021-03 2021-03-24T21:50:56.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[19936]: -24T21:50:55.293524+0000 mgr.y ( 2021-03-24T21:50:56.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[19936]: mgr.14144) 152 : cephadm [INF] Detected new or changed devices on smithi142 2021-03-24T21:50:56.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[19936]: audit 2021-03-24T21:50:55.304289 2021-03-24T21:50:56.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[19936]: +0000 mon.a (mon.0) 386 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:56.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[19936]: audit 2021-03-24T21: 2021-03-24T21:50:56.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[19936]: 50:55.314632+0000 mon.a ( 2021-03-24T21:50:56.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[19936]: mon.0) 387 : audit [INF] 2021-03-24T21:50:56.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:56.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[19936]: audit 2021 2021-03-24T21:50:56.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[19936]: -03-24T21:50: 2021-03-24T21:50:56.657 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[19936]: 55.316769+0000 mon.a (mon. 2021-03-24T21:50:56.657 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[19936]: 0) 388 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:50:56.657 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[19936]: 2021-03-24T21:50:56.657 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[19936]: audit 2021-03-24T21:50:56.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[19936]: 2021- 2021-03-24T21:50:56.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[19936]: 03-24T21:50:55 2021-03-24T21:50:56.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[19936]: .415341+0000 mgr.y (mgr.14144 2021-03-24T21:50:56.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[19936]: ) 153 : audit [DBG] from='client.24277 v1:172.21.15.142:0/2277522098' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:50:56.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[19936]: audit 2021-03-24T21:50:56.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[19936]: 2021-03-24T21:50: 2021-03-24T21:50:56.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[19936]: 55.416747+0000 mon.a (mon. 2021-03-24T21:50:56.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[19936]: 0) 389 : audit [DBG] 2021-03-24T21:50:56.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-24T21:50:56.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[19936]: cephadm 2021-03-24T21:50:56.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[19936]: 2021-03-24T 2021-03-24T21:50:56.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[19936]: 21:50:55.419145+0000 mgr.y (mgr 2021-03-24T21:50:56.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[19936]: .14144) 154 : cephadm [INF] Applying service osd.None on host smithi142... 2021-03-24T21:50:56.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[19936]: audit 2021-03-24T21 2021-03-24T21:50:56.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[19936]: :50:55.419625+0000 mon.a (mon. 2021-03-24T21:50:56.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[19936]: 0) 390 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-24T21:50:56.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[19936]: audit 2021 2021-03-24T21:50:56.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[19936]: -03-24T21:50:55.421166 2021-03-24T21:50:56.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[19936]: +0000 mon.a (mon.0) 2021-03-24T21:50:56.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[19936]: 391 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:50:56.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:50:56.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[19936]: 03-24T21:50:55.644417 2021-03-24T21:50:56.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[19936]: +0000 mon.a (mon.0) 392 2021-03-24T21:50:56.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[19936]: : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:56.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:50:56.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[24927]: T21:50:54.618494+0000 mgr.y (mgr.14144) 151 : cluster [DBG] pgmap v145: 1 pgs: 1 remapped+peering; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:50:56.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[24927]: 2021-03-24T21:50:56.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[24927]: cluster 2021-03-24T21:50:56.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[24927]: 2021-03-24T 2021-03-24T21:50:56.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[24927]: 21:50:55.242434+0000 2021-03-24T21:50:56.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[24927]: mon.a (mon.0) 385 : cluster 2021-03-24T21:50:56.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[24927]: [DBG] osdmap e41: 6 total, 6 up, 6 in 2021-03-24T21:50:56.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[24927]: cephadm 2021-03 2021-03-24T21:50:56.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[24927]: -24T21:50:55. 2021-03-24T21:50:56.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[24927]: 293524+0000 mgr.y (mgr.14144) 152 : 2021-03-24T21:50:56.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[24927]: cephadm [INF] Detected new or changed devices on smithi142 2021-03-24T21:50:56.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[24927]: audit 2021-03 2021-03-24T21:50:56.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[24927]: -24T21:50:55. 2021-03-24T21:50:56.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[24927]: 304289+0000 mon.a (mon.0 2021-03-24T21:50:56.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[24927]: ) 386 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:56.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[24927]: audit 2021-03-24T21:50:56.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[24927]: 2021-03-24T21: 2021-03-24T21:50:56.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[24927]: 50:55.314632+0000 mon.a ( 2021-03-24T21:50:56.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[24927]: mon.0) 387 : audit [INF] 2021-03-24T21:50:56.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[24927]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:56.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[24927]: audit 2021-03-24T21:50:56.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[24927]: 2021-03-24T21:50:55.316769 2021-03-24T21:50:56.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[24927]: +0000 mon.a (mon.0) 388 : audit 2021-03-24T21:50:56.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[24927]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:50:56.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[24927]: audit 2021 2021-03-24T21:50:56.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[24927]: -03-24T21:50:55.415341+0000 2021-03-24T21:50:56.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[24927]: mgr.y (mgr.14144) 153 : audit [DBG] from='client.24277 v1:172.21.15.142:0/2277522098' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:50:56.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[24927]: 2021-03-24T21:50:56.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[24927]: audit 2021-03-24T21:50:56.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[24927]: 2021-03-24T21:50:55.416747+0000 2021-03-24T21:50:56.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[24927]: mon.a (mon.0) 389 : audit 2021-03-24T21:50:56.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[24927]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-24T21:50:56.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[24927]: cephadm 2021 2021-03-24T21:50:56.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[24927]: -03-24T21:50:55.419145+0000 2021-03-24T21:50:56.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[24927]: mgr.y (mgr.14144) 154 : cephadm 2021-03-24T21:50:56.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[24927]: [INF] Applying service osd.None on host smithi142... 2021-03-24T21:50:56.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[24927]: audit 2021-03-24T21:50:56.672 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[24927]: 2021-03-24T21:50: 2021-03-24T21:50:56.672 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[24927]: 55.419625+0000 mon.a (mon.0) 390 : audit 2021-03-24T21:50:56.672 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[24927]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-24T21:50:56.673 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[24927]: audit 2021-03 2021-03-24T21:50:56.673 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[24927]: -24T21:50:55. 2021-03-24T21:50:56.673 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[24927]: 421166+0000 mon.a (mon.0) 391 : 2021-03-24T21:50:56.673 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[24927]: audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:50:56.674 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[24927]: audit 2021-03-24T21:50:56.674 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[24927]: 2021-03- 2021-03-24T21:50:56.674 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[24927]: 24T21:50:55.644417 2021-03-24T21:50:56.674 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[24927]: +0000 mon.a (mon.0) 392 : audit [INF] 2021-03-24T21:50:56.674 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:56 smithi125 conmon[24927]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:50:58.611 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:58 smithi142 conmon[20125]: cluster 2021-03-24T21:50:56.619070+0000 mgr.y (mgr.14144) 155 : cluster [DBG] 2021-03-24T21:50:58.611 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:58 smithi142 conmon[20125]: pgmap v147: 1 pgs: 1 remapped+peering; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:50:58.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:58 smithi142 conmon[20125]: audit 2021-03-24T21:50:58.104861+0000 mon.b (mon.1) 15 : audit 2021-03-24T21:50:58.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:58 smithi142 conmon[20125]: [INF] from='client.? v1:172.21.15.142:0/4021230770' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a26a487e-ed26-4d92-98ea-896739302921"}]: dispatch 2021-03-24T21:50:58.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:58 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:50:58.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:58 smithi142 conmon[20125]: 03-24T21:50:58.108462+0000 mon.a (mon.0) 2021-03-24T21:50:58.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:58 smithi142 conmon[20125]: 393 : audit [INF] from='client.24283 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a26a487e-ed26-4d92-98ea-896739302921"}]: dispatch 2021-03-24T21:50:58.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:58 smithi142 conmon[20125]: audit 2021-03- 2021-03-24T21:50:58.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:58 smithi142 conmon[20125]: 24T21:50:58.118345+0000 mon.a 2021-03-24T21:50:58.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:58 smithi142 conmon[20125]: (mon.0) 394 : audit [INF] from='client.24283 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a26a487e-ed26-4d92-98ea-896739302921"}]': finished 2021-03-24T21:50:58.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:58 smithi142 conmon[20125]: 2021-03-24T21:50:58.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:58 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:50:58.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:58 smithi142 conmon[20125]: -03-24T21:50:58.118511+0000 mon.a ( 2021-03-24T21:50:58.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:58 smithi142 conmon[20125]: mon.0) 395 : cluster [DBG] osdmap e42: 7 total, 6 up, 7 in 2021-03-24T21:50:58.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:58 smithi142 conmon[20125]: audit 2021 2021-03-24T21:50:58.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:58 smithi142 conmon[20125]: -03-24T21:50:58.118800 2021-03-24T21:50:58.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:58 smithi142 conmon[20125]: +0000 mon.a (mon.0) 396 : audit 2021-03-24T21:50:58.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:58 smithi142 conmon[20125]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-03-24T21:50:58.651 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:58 smithi125 conmon[24927]: cluster 2021-03-24T21:50: 2021-03-24T21:50:58.651 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:58 smithi125 conmon[24927]: 56.619070+0000 mgr.y (mgr.14144) 155 : cluster [DBG] pgmap v147: 1 pgs: 1 remapped+peering; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:50:58.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:58 smithi125 conmon[24927]: audit 2021-03-24T21:50:58.104861 2021-03-24T21:50:58.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:58 smithi125 conmon[24927]: +0000 mon.b (mon.1) 15 : audit [INF] from='client.? v1:172.21.15.142:0/4021230770' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a26a487e-ed26-4d92-98ea-896739302921"}]: dispatch 2021-03-24T21:50:58.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:58 smithi125 conmon[24927]: 2021-03-24T21:50:58.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:58 smithi125 conmon[24927]: audit 2021-03-24T21: 2021-03-24T21:50:58.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:58 smithi125 conmon[24927]: 50:58.108462+0000 mon.a (mon.0) 2021-03-24T21:50:58.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:58 smithi125 conmon[24927]: 393 : audit [INF] from='client.24283 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a26a487e-ed26-4d92-98ea-896739302921"}]: dispatch 2021-03-24T21:50:58.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:58 smithi125 conmon[24927]: audit 2021-03 2021-03-24T21:50:58.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:58 smithi125 conmon[24927]: -24T21:50:58.118345+0000 mon.a (mon 2021-03-24T21:50:58.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:58 smithi125 conmon[24927]: .0) 394 : audit [INF] from='client.24283 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a26a487e-ed26-4d92-98ea-896739302921"}]': finished 2021-03-24T21:50:58.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:58 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:50:58.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:58 smithi125 conmon[24927]: -03-24T21:50:58.118511+0000 mon.a 2021-03-24T21:50:58.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:58 smithi125 conmon[24927]: (mon.0) 395 : cluster [DBG] osdmap e42: 7 total, 6 up, 7 in 2021-03-24T21:50:58.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:58 smithi125 conmon[24927]: audit 2021-03-24T21:50:58.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:58 smithi125 conmon[24927]: 2021-03-24T21:50:58.118800+0000 mon.a 2021-03-24T21:50:58.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:58 smithi125 conmon[24927]: (mon.0) 396 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-03-24T21:50:58.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:58 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:50:58.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:58 smithi125 conmon[19936]: 21:50:56.619070+0000 mgr.y (mgr.14144) 155 : cluster [DBG] pgmap v147: 1 pgs: 1 remapped+peering; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:50:58.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:58 smithi125 conmon[19936]: audit 2021-03-24T21 2021-03-24T21:50:58.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:58 smithi125 conmon[19936]: :50:58.104861+0000 mon.b (mon. 2021-03-24T21:50:58.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:58 smithi125 conmon[19936]: 1) 15 : audit [INF] from='client.? v1:172.21.15.142:0/4021230770' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a26a487e-ed26-4d92-98ea-896739302921"}]: dispatch 2021-03-24T21:50:58.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:58 smithi125 conmon[19936]: audit 2021 2021-03-24T21:50:58.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:58 smithi125 conmon[19936]: -03-24T21:50: 2021-03-24T21:50:58.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:58 smithi125 conmon[19936]: 58.108462+0000 mon.a (mon. 2021-03-24T21:50:58.657 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:58 smithi125 conmon[19936]: 0) 393 : audit [INF] from='client.24283 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a26a487e-ed26-4d92-98ea-896739302921"}]: dispatch 2021-03-24T21:50:58.657 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:58 smithi125 conmon[19936]: 2021-03-24T21:50:58.657 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:58 smithi125 conmon[19936]: audit 2021-03-24T21: 2021-03-24T21:50:58.657 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:58 smithi125 conmon[19936]: 50:58.118345+0000 mon.a (mon.0) 2021-03-24T21:50:58.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:58 smithi125 conmon[19936]: 394 : audit [INF] from='client.24283 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a26a487e-ed26-4d92-98ea-896739302921"}]': finished 2021-03-24T21:50:58.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:58 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:50:58.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:58 smithi125 conmon[19936]: 24T21:50:58.118511 2021-03-24T21:50:58.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:58 smithi125 conmon[19936]: +0000 mon.a (mon.0 2021-03-24T21:50:58.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:58 smithi125 conmon[19936]: ) 395 : cluster [DBG] osdmap e42: 7 total, 6 up, 7 in 2021-03-24T21:50:58.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:58 smithi125 conmon[19936]: 2021-03-24T21:50:58.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:58 smithi125 conmon[19936]: audit 2021-03-24 2021-03-24T21:50:58.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:58 smithi125 conmon[19936]: T21:50:58.118800+0000 mon.a 2021-03-24T21:50:58.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:58 smithi125 conmon[19936]: (mon.0) 396 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-03-24T21:50:59.611 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:59 smithi142 conmon[20125]: cluster 2021-03-24T21:50:58.728273+0000 osd.5 (osd.5) 3 : cluster [DBG] 2021-03-24T21:50:59.611 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:59 smithi142 conmon[20125]: 1.0 scrub ok 2021-03-24T21:50:59.611 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:59 smithi142 conmon[20125]: audit 2021-03-24T21:50:58.775021+0000 mon.b (mon.1) 16 : audit 2021-03-24T21:50:59.611 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:50:59 smithi142 conmon[20125]: [DBG] from='client.? v1:172.21.15.142:0/603799674' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-24T21:50:59.650 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:59 smithi125 conmon[19936]: cluster 2021-03-24T21:50:58.728273+0000 2021-03-24T21:50:59.651 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:59 smithi125 conmon[19936]: osd.5 (osd.5) 3 : cluster [DBG] 1.0 scrub ok 2021-03-24T21:50:59.651 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:59 smithi125 conmon[19936]: audit 2021-03-24T21 2021-03-24T21:50:59.651 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:59 smithi125 conmon[19936]: :50:58.775021+0000 mon.b (mon.1) 16 : audit [DBG] 2021-03-24T21:50:59.651 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:50:59 smithi125 conmon[19936]: from='client.? v1:172.21.15.142:0/603799674' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-24T21:50:59.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:59 smithi125 conmon[24927]: cluster 2021-03-24T21:50:58.728273+0000 2021-03-24T21:50:59.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:59 smithi125 conmon[24927]: osd.5 (osd.5) 3 : cluster [DBG] 1.0 scrub ok 2021-03-24T21:50:59.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:59 smithi125 conmon[24927]: audit 2021-03-24T21 2021-03-24T21:50:59.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:50:59 smithi125 conmon[24927]: :50:58.775021+0000 mon.b (mon.1) 16 : audit [DBG] from='client.? v1:172.21.15.142:0/603799674' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-24T21:51:00.610 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:00 smithi142 conmon[20125]: cluster 2021-03-24T21:50:58.619627+0000 mgr.y (mgr.14144) 156 : 2021-03-24T21:51:00.611 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:00 smithi142 conmon[20125]: cluster [DBG] pgmap v149: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:51:00.650 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:00 smithi125 conmon[19936]: cluster 2021-03-24T21:50:58 2021-03-24T21:51:00.651 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:00 smithi125 conmon[19936]: .619627+0000 mgr.y (mgr.14144) 156 : cluster [DBG] pgmap v149: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:51:00.651 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:00 smithi125 conmon[24927]: cluster 2021-03-24T21:50:58.619627+0000 2021-03-24T21:51:00.651 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:00 smithi125 conmon[24927]: mgr.y (mgr.14144) 156 : cluster [DBG] pgmap v149: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:51:02.596 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:02 smithi142 conmon[20125]: cluster 2021-03-24T21:51:00.620290+0000 mgr.y (mgr.14144) 157 2021-03-24T21:51:02.597 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:02 smithi142 conmon[20125]: : cluster [DBG] pgmap v150: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:51:02.650 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:02 smithi125 conmon[19936]: cluster 2021-03-24T21:51:00.620290+0000 2021-03-24T21:51:02.651 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:02 smithi125 conmon[19936]: mgr.y (mgr.14144) 157 : cluster [DBG] pgmap v150: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:51:02.651 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:02 smithi125 conmon[24927]: cluster 2021-03-24T21:51 2021-03-24T21:51:02.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:02 smithi125 conmon[24927]: :00.620290+0000 mgr.y (mgr.14144) 157 : cluster [DBG] pgmap v150: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:51:04.550 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:04 smithi142 conmon[20125]: cluster 2021-03-24T21:51:02.620847+0000 mgr.y (mgr.14144) 158 : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:51:04.650 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:04 smithi125 conmon[19936]: cluster 2021-03-24T21:51 2021-03-24T21:51:04.651 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:04 smithi125 conmon[19936]: :02.620847+0000 mgr.y (mgr.14144) 158 : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:51:04.651 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:04 smithi125 conmon[24927]: cluster 2021-03-24T21:51:02.620847+0000 mgr.y (mgr.14144) 158 : cluster 2021-03-24T21:51:04.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:04 smithi125 conmon[24927]: [DBG] pgmap v151: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:51:05.610 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:05 smithi142 conmon[20125]: audit 2021-03-24T21:51:04.731998+0000 mon.a (mon.0) 397 : 2021-03-24T21:51:05.611 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:05 smithi142 conmon[20125]: audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T21:51:05.611 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:05 smithi142 conmon[20125]: audit 2021-03-24T21:51:04.735859+0000 mon.a (mon. 2021-03-24T21:51:05.611 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:05 smithi142 conmon[20125]: 0) 398 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T21:51:05.650 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:05 smithi125 conmon[19936]: audit 2021-03-24T21:51: 2021-03-24T21:51:05.651 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:05 smithi125 conmon[19936]: 04.731998+0000 mon.a (mon.0) 397 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T21:51:05.651 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:05 smithi125 conmon[19936]: audit 2021-03-24T21:51:05.651 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:05 smithi125 conmon[19936]: 2021-03-24T21:51 2021-03-24T21:51:05.651 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:05 smithi125 conmon[19936]: :04.735859+0000 mon.a (mon.0) 2021-03-24T21:51:05.652 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:05 smithi125 conmon[19936]: 398 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T21:51:05.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:05 smithi125 conmon[24927]: audit 2021-03-24T21 2021-03-24T21:51:05.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:05 smithi125 conmon[24927]: :51:04.731998+0000 mon.a (mon.0) 397 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T21:51:05.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:05 smithi125 conmon[24927]: audit 2021-03-24T21:51:05.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:05 smithi125 conmon[24927]: 2021-03-24T21:51:04. 2021-03-24T21:51:05.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:05 smithi125 conmon[24927]: 735859+0000 mon.a (mon.0) 398 : audit 2021-03-24T21:51:05.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:05 smithi125 conmon[24927]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T21:51:06.611 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:06 smithi142 conmon[20125]: cluster 2021-03-24T21:51:04.621326 2021-03-24T21:51:06.611 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:06 smithi142 conmon[20125]: +0000 mgr.y (mgr.14144) 159 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:51:06.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:06 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:51:06.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:06 smithi142 conmon[20125]: 03-24T21:51:05.442873+0000 2021-03-24T21:51:06.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:06 smithi142 conmon[20125]: mon.a (mon.0) 2021-03-24T21:51:06.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:06 smithi142 conmon[20125]: 399 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2021-03-24T21:51:06.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:06 smithi142 conmon[20125]: audit 2021-03-24T21:51:06.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:06 smithi142 conmon[20125]: 2021-03-24T21:51:05 2021-03-24T21:51:06.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:06 smithi142 conmon[20125]: .444492+0000 mon.a (mon.0) 2021-03-24T21:51:06.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:06 smithi142 conmon[20125]: 400 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:51:06.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:06 smithi142 conmon[20125]: cephadm 2021-03-24T21:51:06.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:06 smithi142 conmon[20125]: 2021-03-24T 2021-03-24T21:51:06.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:06 smithi142 conmon[20125]: 21:51:05.445928+0000 2021-03-24T21:51:06.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:06 smithi142 conmon[20125]: mgr.y (mgr.14144) 2021-03-24T21:51:06.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:06 smithi142 conmon[20125]: 160 : cephadm [INF] Deploying daemon osd.6 on smithi142 2021-03-24T21:51:06.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:06 smithi142 conmon[20125]: 2021-03-24T21:51:06.650 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:06 smithi125 conmon[19936]: cluster 2021-03-24T21:51: 2021-03-24T21:51:06.651 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:06 smithi125 conmon[19936]: 04.621326+0000 mgr.y (mgr.14144) 159 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:51:06.651 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:06 smithi125 conmon[19936]: audit 2021-03 2021-03-24T21:51:06.651 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:06 smithi125 conmon[19936]: -24T21:51:05.442873+0000 mon.a (mon 2021-03-24T21:51:06.651 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:06 smithi125 conmon[19936]: .0) 399 : audit [INF] 2021-03-24T21:51:06.652 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:06 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2021-03-24T21:51:06.652 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:06 smithi125 conmon[19936]: audit 2021-03 2021-03-24T21:51:06.652 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:06 smithi125 conmon[19936]: -24T21:51:05. 2021-03-24T21:51:06.652 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:06 smithi125 conmon[19936]: 444492+0000 mon.a (mon.0) 400 : audit 2021-03-24T21:51:06.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:06 smithi125 conmon[19936]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:51:06.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:06 smithi125 conmon[19936]: cephadm 2021-03 2021-03-24T21:51:06.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:06 smithi125 conmon[19936]: -24T21:51:05. 2021-03-24T21:51:06.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:06 smithi125 conmon[19936]: 445928+0000 mgr.y (mgr.14144 2021-03-24T21:51:06.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:06 smithi125 conmon[19936]: ) 160 : cephadm [INF] Deploying daemon osd.6 on smithi142 2021-03-24T21:51:06.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:06 smithi125 conmon[24927]: cluster 2021-03-24T21:51:04 2021-03-24T21:51:06.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:06 smithi125 conmon[24927]: .621326+0000 mgr.y (mgr.14144) 159 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:51:06.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:06 smithi125 conmon[24927]: audit 2021-03-24T21:51: 2021-03-24T21:51:06.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:06 smithi125 conmon[24927]: 05.442873+0000 mon.a (mon.0) 399 : 2021-03-24T21:51:06.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:06 smithi125 conmon[24927]: audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2021-03-24T21:51:06.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:06 smithi125 conmon[24927]: audit 2021-03-24 2021-03-24T21:51:06.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:06 smithi125 conmon[24927]: T21:51:05.444492+0000 2021-03-24T21:51:06.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:06 smithi125 conmon[24927]: mon.a (mon.0) 400 : 2021-03-24T21:51:06.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:06 smithi125 conmon[24927]: audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:51:06.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:06 smithi125 conmon[24927]: cephadm 2021 2021-03-24T21:51:06.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:06 smithi125 conmon[24927]: -03-24T21:51:05. 2021-03-24T21:51:06.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:06 smithi125 conmon[24927]: 445928+0000 mgr.y (mgr.14144) 160 : cephadm 2021-03-24T21:51:06.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:06 smithi125 conmon[24927]: [INF] Deploying daemon osd.6 on smithi142 2021-03-24T21:51:08.576 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:08 smithi142 conmon[20125]: cluster 2021-03-24T21:51:06.621878+0000 mgr.y (mgr.14144) 2021-03-24T21:51:08.576 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:08 smithi142 conmon[20125]: 161 : cluster [DBG] pgmap v153: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:51:08.576 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:08 smithi142 conmon[20125]: cluster 2021-03-24T21:51:06.797328+0000 osd.0 (osd.0) 5 : cluster [DBG] 2021-03-24T21:51:08.577 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:08 smithi142 conmon[20125]: purged_snaps scrub starts 2021-03-24T21:51:08.577 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:08 smithi142 conmon[20125]: cluster 2021-03-24T21:51:06. 2021-03-24T21:51:08.577 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:08 smithi142 conmon[20125]: 797528+0000 osd.0 (osd.0) 6 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:51:08.577 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:08 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:51:08.578 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:08 smithi142 conmon[20125]: 21:51:07.287623+0000 mon.a (mon.0) 401 2021-03-24T21:51:08.578 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:08 smithi142 conmon[20125]: : cluster [DBG] osdmap e43: 7 total, 6 up, 7 in 2021-03-24T21:51:08.578 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:08 smithi142 conmon[20125]: audit 2021-03-24T21:51:07.287951+0000 2021-03-24T21:51:08.578 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:08 smithi142 conmon[20125]: mon.a (mon.0) 402 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-03-24T21:51:08.650 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:08 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:51:08.651 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:08 smithi125 conmon[19936]: 51:06.621878+0000 mgr.y (mgr.14144) 161 : cluster [DBG] pgmap v153: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:51:08.651 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:08 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:51:08.651 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:08 smithi125 conmon[19936]: :51:06.797328+0000 osd.0 (osd.0) 5 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:51:08.651 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:08 smithi125 conmon[19936]: cluster 2021-03-24T21:51:08.652 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:08 smithi125 conmon[19936]: 2021-03-24T21:51:06.797528+0000 osd.0 (osd.0 2021-03-24T21:51:08.652 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:08 smithi125 conmon[19936]: ) 6 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:51:08.652 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:08 smithi125 conmon[19936]: cluster 2021-03-24T21:51:07 2021-03-24T21:51:08.652 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:08 smithi125 conmon[19936]: .287623+0000 mon.a (mon.0) 401 : cluster [DBG] osdmap e43: 7 total, 6 up, 7 in 2021-03-24T21:51:08.652 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:08 smithi125 conmon[19936]: audit 2021-03- 2021-03-24T21:51:08.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:08 smithi125 conmon[19936]: 24T21:51:07.287951+0000 mon.a (mon. 2021-03-24T21:51:08.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:08 smithi125 conmon[19936]: 0) 402 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-03-24T21:51:08.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:08 smithi125 conmon[24927]: cluster 2021-03-24T21:51:06.621878+0000 2021-03-24T21:51:08.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:08 smithi125 conmon[24927]: mgr.y (mgr.14144) 161 : cluster [DBG] pgmap v153: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:51:08.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:08 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:51:08.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:08 smithi125 conmon[24927]: :51:06.797328+0000 osd.0 (osd.0) 5 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:51:08.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:08 smithi125 conmon[24927]: cluster 2021-03-24T21:51 2021-03-24T21:51:08.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:08 smithi125 conmon[24927]: :06.797528+0000 osd.0 (osd.0) 6 : cluster [DBG] 2021-03-24T21:51:08.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:08 smithi125 conmon[24927]: purged_snaps scrub ok 2021-03-24T21:51:08.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:08 smithi125 conmon[24927]: cluster 2021-03-24T21:51:08.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:08 smithi125 conmon[24927]: 2021-03-24T21:51:07.287623+0000 mon.a (mon.0) 401 : 2021-03-24T21:51:08.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:08 smithi125 conmon[24927]: cluster [DBG] osdmap e43: 7 total, 6 up, 7 in 2021-03-24T21:51:08.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:08 smithi125 conmon[24927]: audit 2021- 2021-03-24T21:51:08.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:08 smithi125 conmon[24927]: 03-24T21:51:07.287951+0000 mon.a (mon.0) 402 : audit [DBG] 2021-03-24T21:51:08.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:08 smithi125 conmon[24927]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-03-24T21:51:10.021 INFO:teuthology.orchestra.run.smithi142.stdout:Created osd(s) 6 on host 'smithi142' 2021-03-24T21:51:10.481 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:10 smithi142 conmon[20125]: cluster 2021-03-24T21:51:08.622402+0000 mgr.y (mgr.14144) 162 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:51:10.482 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:10 smithi142 conmon[20125]: audit 2021-03-24T21:51:10.013716+0000 mon.a (mon.0) 403 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:51:10.482 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:10 smithi142 conmon[20125]: audit 2021-03-24T21:51:10.482 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:10 smithi142 conmon[20125]: 2021-03-24T21:51:10.021506+0000 mon.a (mon.0) 404 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:10.483 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:10 smithi142 conmon[20125]: audit 2021-03-24T 2021-03-24T21:51:10.483 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:10 smithi142 conmon[20125]: 21:51:10.029383+0000 mon.a (mon.0 2021-03-24T21:51:10.483 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:10 smithi142 conmon[20125]: ) 405 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:10.650 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:10 smithi125 conmon[19936]: cluster 2021-03-24T21:51:08. 2021-03-24T21:51:10.651 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:10 smithi125 conmon[19936]: 622402+0000 mgr.y (mgr.14144) 162 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:51:10.651 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:10 smithi125 conmon[19936]: audit 2021-03-24T21:51:10.651 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:10 smithi125 conmon[19936]: 2021-03-24T21:51:10.013716+0000 mon.a (mon.0) 403 : audit [DBG] 2021-03-24T21:51:10.652 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:10 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:51:10.652 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:10 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:51:10.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:10 smithi125 conmon[19936]: 03-24T21:51:10.021506+0000 mon.a (mon.0) 404 : audit [INF] 2021-03-24T21:51:10.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:10 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:10.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:10 smithi125 conmon[19936]: audit 2021 2021-03-24T21:51:10.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:10 smithi125 conmon[19936]: -03-24T21:51:10.029383+0000 mon.a (mon.0) 405 : audit 2021-03-24T21:51:10.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:10 smithi125 conmon[19936]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:10.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:10 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:51:10.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:10 smithi125 conmon[24927]: 51:08.622402+0000 mgr.y (mgr.14144) 162 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:51:10.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:10 smithi125 conmon[24927]: audit 2021- 2021-03-24T21:51:10.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:10 smithi125 conmon[24927]: 03-24T21:51:10 2021-03-24T21:51:10.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:10 smithi125 conmon[24927]: .013716+0000 mon.a (mon.0) 403 : 2021-03-24T21:51:10.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:10 smithi125 conmon[24927]: audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:51:10.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:10 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T21:51:10.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:10 smithi125 conmon[24927]: 24T21:51:10.021506+0000 mon.a (mon 2021-03-24T21:51:10.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:10 smithi125 conmon[24927]: .0) 404 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:10.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:10 smithi125 conmon[24927]: audit 2021 2021-03-24T21:51:10.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:10 smithi125 conmon[24927]: -03-24T21:51:10.029383+0000 2021-03-24T21:51:10.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:10 smithi125 conmon[24927]: mon.a (mon.0) 405 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:10.904 DEBUG:teuthology.orchestra.run.smithi142:osd.6> sudo journalctl -f -n 0 -u ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c@osd.6.service 2021-03-24T21:51:10.906 INFO:tasks.cephadm:Deploying osd.7 on smithi142 with /dev/vg_nvme/lv_1... 2021-03-24T21:51:10.907 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2021-03-24T21:51:10.935 INFO:journalctl@ceph.osd.6.smithi142.stdout:-- Logs begin at Wed 2021-03-03 23:51:07 UTC. -- 2021-03-24T21:51:12.586 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:12 smithi142 conmon[20125]: cluster 2021-03-24T21:51:10.622911+0000 mgr.y (mgr.14144) 163 : cluster 2021-03-24T21:51:12.587 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:12 smithi142 conmon[20125]: [DBG] pgmap v156: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:51:12.587 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:12 smithi142 conmon[20125]: cluster 2021-03-24T21:51 2021-03-24T21:51:12.587 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:12 smithi142 conmon[20125]: :11.204178+0000 osd.3 (osd.3) 3 : cluster [DBG] 2021-03-24T21:51:12.588 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:12 smithi142 conmon[20125]: purged_snaps scrub starts 2021-03-24T21:51:12.588 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:12 smithi142 conmon[20125]: cluster 2021-03-24T21:51:11 2021-03-24T21:51:12.588 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:12 smithi142 conmon[20125]: .204376+0000 osd.3 (osd.3) 4 : cluster 2021-03-24T21:51:12.589 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:12 smithi142 conmon[20125]: [DBG] purged_snaps scrub ok 2021-03-24T21:51:12.589 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:12 smithi142 conmon[20125]: cluster 2021-03-24T21:51:12.589 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:12 smithi142 conmon[20125]: 2021-03-24T21:51:11.310650+0000 mon.a (mon.0) 406 : cluster [DBG] osdmap e44: 7 total, 6 up, 7 in 2021-03-24T21:51:12.589 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:12 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:51:12.590 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:12 smithi142 conmon[20125]: 03-24T21:51:11.310946+0000 mon.a (mon 2021-03-24T21:51:12.590 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:12 smithi142 conmon[20125]: .0) 407 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-03-24T21:51:12.651 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:12 smithi125 conmon[19936]: cluster 2021-03-24T21:51: 2021-03-24T21:51:12.982 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:12 smithi125 conmon[19936]: 10.622911+0000 mgr.y (mgr.14144) 163 : cluster [DBG] pgmap v156: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:51:12.983 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:12 smithi125 conmon[19936]: cluster 2021-03-24T21:51:12.983 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:12 smithi125 conmon[19936]: 2021-03-24T21:51:11.204178+0000 osd.3 (osd.3) 2021-03-24T21:51:12.983 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:12 smithi125 conmon[19936]: 3 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:51:12.983 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:12 smithi125 conmon[19936]: cluster 2021-03-24T21:51: 2021-03-24T21:51:12.984 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:12 smithi125 conmon[19936]: 11.204376+0000 osd.3 (osd.3) 4 : cluster [DBG] 2021-03-24T21:51:12.984 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:12 smithi125 conmon[19936]: purged_snaps scrub ok 2021-03-24T21:51:12.984 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:12 smithi125 conmon[19936]: cluster 2021-03-24T21:51:12.984 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:12 smithi125 conmon[19936]: 2021-03-24T21:51:11.310650+0000 mon.a (mon.0 2021-03-24T21:51:12.984 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:12 smithi125 conmon[19936]: ) 406 : cluster [DBG] osdmap e44: 7 total, 6 up, 7 in 2021-03-24T21:51:12.985 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:12 smithi125 conmon[19936]: audit 2021-03-24T21:51 2021-03-24T21:51:12.985 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:12 smithi125 conmon[19936]: :11.310946+0000 2021-03-24T21:51:12.985 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:12 smithi125 conmon[19936]: mon.a (mon.0) 407 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-03-24T21:51:12.985 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:12 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:51:12.986 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:12 smithi125 conmon[24927]: 21:51:10.622911+0000 mgr.y (mgr.14144) 163 : cluster [DBG] pgmap v156: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:51:12.986 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:12 smithi125 conmon[24927]: cluster 2021-03-24T21:51:11.204178+0000 osd.3 (osd. 2021-03-24T21:51:12.986 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:12 smithi125 conmon[24927]: 3) 3 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:51:12.986 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:12 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:51:12.987 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:12 smithi125 conmon[24927]: T21:51:11.204376+0000 osd.3 (osd. 2021-03-24T21:51:12.987 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:12 smithi125 conmon[24927]: 3) 4 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:51:12.987 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:12 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:51:12.987 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:12 smithi125 conmon[24927]: -24T21:51:11.310650+0000 mon.a (mon 2021-03-24T21:51:12.987 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:12 smithi125 conmon[24927]: .0) 406 : cluster [DBG] osdmap e44: 7 total, 6 up, 7 in 2021-03-24T21:51:12.988 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:12 smithi125 conmon[24927]: audit 2021-03-24T21:51:12.988 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:12 smithi125 conmon[24927]: 2021-03-24T21:51:11.310946+0000 2021-03-24T21:51:12.988 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:12 smithi125 conmon[24927]: mon.a (mon.0) 407 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-03-24T21:51:13.551 INFO:journalctl@ceph.osd.6.smithi142.stdout:Mar 24 21:51:13 smithi142 conmon[30596]: debug 2021-03-24T21:51:13.292+0000 7ff0223c2080 -1 osd.6 0 log_to_monitors {default=true} 2021-03-24T21:51:13.627 INFO:teuthology.orchestra.run.smithi142.stderr:--> Zapping: /dev/vg_nvme/lv_1 2021-03-24T21:51:13.672 INFO:teuthology.orchestra.run.smithi142.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2021-03-24T21:51:13.673 INFO:teuthology.orchestra.run.smithi142.stderr: stderr: 10+0 records in 2021-03-24T21:51:13.673 INFO:teuthology.orchestra.run.smithi142.stderr:10+0 records out 2021-03-24T21:51:13.673 INFO:teuthology.orchestra.run.smithi142.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0175918 s, 596 MB/s 2021-03-24T21:51:13.673 INFO:teuthology.orchestra.run.smithi142.stderr:--> Zapping successful for: 2021-03-24T21:51:13.969 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:13 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:51:14.150 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:13 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:51:14.151 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:13 smithi125 conmon[19936]: :51:12.623513+0000 mgr.y (mgr.14144) 164 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:51:14.151 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:13 smithi125 conmon[19936]: audit 2021-03-24T21:51 2021-03-24T21:51:14.151 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:13 smithi125 conmon[19936]: :12.893940+0000 mon.a (mon.0) 408 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:14.152 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:13 smithi125 conmon[19936]: audit 2021-03-24T21:51:13 2021-03-24T21:51:14.152 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:13 smithi125 conmon[19936]: .300480+0000 mon.b (mon.1) 17 : audit [INF] 2021-03-24T21:51:14.152 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:13 smithi125 conmon[19936]: from='osd.6 v1:172.21.15.142:6808/4048483913' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2021-03-24T21:51:14.152 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:13 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:51:14.152 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:13 smithi125 conmon[19936]: 03-24T21:51:13.303845+0000 mon.a (mon. 2021-03-24T21:51:14.153 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:13 smithi125 conmon[19936]: 0) 409 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2021-03-24T21:51:14.153 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:13 smithi125 conmon[24927]: cluster 2021-03-24T21:51 2021-03-24T21:51:14.153 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:13 smithi125 conmon[24927]: :12.623513+0000 mgr.y (mgr.14144) 164 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:51:14.153 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:13 smithi125 conmon[24927]: audit 2021-03-24T21:51:12 2021-03-24T21:51:14.154 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:13 smithi125 conmon[24927]: .893940+0000 mon.a (mon.0) 408 : audit 2021-03-24T21:51:14.154 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:13 smithi125 conmon[24927]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:14.154 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:13 smithi125 conmon[24927]: audit 2021-03-24T21 2021-03-24T21:51:14.154 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:13 smithi125 conmon[24927]: :51:13.300480+0000 mon.b (mon.1 2021-03-24T21:51:14.155 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:13 smithi125 conmon[24927]: ) 17 : audit [INF] from='osd.6 v1:172.21.15.142:6808/4048483913' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2021-03-24T21:51:14.155 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:13 smithi125 conmon[24927]: audit 2021-03 2021-03-24T21:51:14.155 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:13 smithi125 conmon[24927]: -24T21:51:13.303845+0000 mon.a 2021-03-24T21:51:14.155 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:13 smithi125 conmon[24927]: (mon.0) 409 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2021-03-24T21:51:14.361 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:13 smithi142 conmon[20125]: 51:12.623513+0000 mgr.y (mgr.14144) 164 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:51:14.361 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:13 smithi142 conmon[20125]: audit 2021-03-24T21:51:14.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:13 smithi142 conmon[20125]: 2021-03-24T21:51:12.893940 2021-03-24T21:51:14.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:13 smithi142 conmon[20125]: +0000 mon.a (mon.0) 408 : audit [INF] 2021-03-24T21:51:14.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:13 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:14.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:13 smithi142 conmon[20125]: audit 2021-03-24T21:51:13. 2021-03-24T21:51:14.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:13 smithi142 conmon[20125]: 300480+0000 mon.b (mon.1) 17 : audit [INF] 2021-03-24T21:51:14.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:13 smithi142 conmon[20125]: from='osd.6 v1:172.21.15.142:6808/4048483913' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2021-03-24T21:51:14.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:13 smithi142 conmon[20125]: audit 2021-03-24T21:51:13.303845 2021-03-24T21:51:14.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:13 smithi142 conmon[20125]: +0000 mon.a (mon.0) 409 : audit [INF] 2021-03-24T21:51:14.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:13 smithi142 conmon[20125]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2021-03-24T21:51:14.997 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph orch daemon add osd smithi142:vg_nvme/lv_1 2021-03-24T21:51:15.211 INFO:journalctl@ceph.osd.6.smithi142.stdout:Mar 24 21:51:14 smithi142 conmon[30596]: debug 2021-03-24T21:51:14.920+0000 7ff018933700 -1 osd.6 0 waiting for initial osdmap 2021-03-24T21:51:15.211 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:14 smithi142 conmon[20125]: audit 2021-03-24T21:51:13.904519+0000 mon.b (mon.1) 18 : audit 2021-03-24T21:51:15.212 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:14 smithi142 conmon[20125]: [INF] from='osd.6 v1:172.21.15.142:6808/4048483913' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2021-03-24T21:51:15.212 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:14 smithi142 conmon[20125]: audit 2021-03-24T21:51:13.905725+0000 mon.a (mon.0) 410 : 2021-03-24T21:51:15.212 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:14 smithi142 conmon[20125]: audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2021-03-24T21:51:15.213 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:14 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:51:15.213 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:14 smithi142 conmon[20125]: -24T21:51:13.905883+0000 mon.a (mon.0 2021-03-24T21:51:15.214 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:14 smithi142 conmon[20125]: ) 411 : cluster [DBG] osdmap e45: 7 total, 6 up, 7 in 2021-03-24T21:51:15.214 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:14 smithi142 conmon[20125]: audit 2021-03-24 2021-03-24T21:51:15.215 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:14 smithi142 conmon[20125]: T21:51:13.906174+0000 mon.a ( 2021-03-24T21:51:15.215 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:14 smithi142 conmon[20125]: mon.0) 412 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-03-24T21:51:15.216 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:14 smithi142 conmon[20125]: audit 2021 2021-03-24T21:51:15.216 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:14 smithi142 conmon[20125]: -03-24T21:51:13.907951 2021-03-24T21:51:15.217 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:14 smithi142 conmon[20125]: +0000 mon.a (mon.0) 413 : audit 2021-03-24T21:51:15.217 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:14 smithi142 conmon[20125]: [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2021-03-24T21:51:15.401 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:14 smithi125 conmon[24927]: audit 2021-03-24T21:51:13.904519+0000 mon.b (mon.1) 18 : audit [INF] from='osd.6 v1:172.21.15.142:6808/4048483913' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2021-03-24T21:51:15.401 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:14 smithi125 conmon[24927]: audit 2021-03-24T21 2021-03-24T21:51:15.401 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:14 smithi125 conmon[24927]: :51:13.905725+0000 mon.a (mon.0) 410 : 2021-03-24T21:51:15.401 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:14 smithi125 conmon[24927]: audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2021-03-24T21:51:15.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:14 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:51:15.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:14 smithi125 conmon[24927]: :51:13.905883+0000 mon.a (mon.0) 411 : cluster 2021-03-24T21:51:15.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:14 smithi125 conmon[24927]: [DBG] osdmap e45: 7 total, 6 up, 7 in 2021-03-24T21:51:15.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:14 smithi125 conmon[24927]: audit 2021-03-24T21:51:13 2021-03-24T21:51:15.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:14 smithi125 conmon[24927]: .906174+0000 mon.a (mon.0) 412 : audit 2021-03-24T21:51:15.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:14 smithi125 conmon[24927]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-03-24T21:51:15.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:14 smithi125 conmon[24927]: audit 2021-03-24T21:51:13. 2021-03-24T21:51:15.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:14 smithi125 conmon[24927]: 907951+0000 mon.a (mon.0) 413 : audit [INF] 2021-03-24T21:51:15.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:14 smithi125 conmon[24927]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2021-03-24T21:51:15.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:14 smithi125 conmon[19936]: audit 2021-03-24T21 2021-03-24T21:51:15.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:14 smithi125 conmon[19936]: :51:13.904519+0000 mon.b (mon.1) 18 : audit [INF] from='osd.6 v1:172.21.15.142:6808/4048483913' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2021-03-24T21:51:15.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:14 smithi125 conmon[19936]: audit 2021-03-24T21:51:15.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:14 smithi125 conmon[19936]: 2021 2021-03-24T21:51:15.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:14 smithi125 conmon[19936]: -03-24T21:51 2021-03-24T21:51:15.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:14 smithi125 conmon[19936]: :13.905725+0000 mon.a (mon.0) 410 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2021-03-24T21:51:15.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:14 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:51:15.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:14 smithi125 conmon[19936]: 03-24T21:51:13.905883+0000 mon.a 2021-03-24T21:51:15.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:14 smithi125 conmon[19936]: (mon.0) 411 : cluster [DBG] osdmap e45: 7 total, 6 up, 7 in 2021-03-24T21:51:15.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:14 smithi125 conmon[19936]: audit 2021 2021-03-24T21:51:15.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:14 smithi125 conmon[19936]: -03-24T21:51: 2021-03-24T21:51:15.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:14 smithi125 conmon[19936]: 13.906174+0000 mon.a (mon 2021-03-24T21:51:15.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:14 smithi125 conmon[19936]: .0) 412 : audit [DBG] 2021-03-24T21:51:15.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:14 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-03-24T21:51:15.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:14 smithi125 conmon[19936]: audit 2021-03- 2021-03-24T21:51:15.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:14 smithi125 conmon[19936]: 24T21:51:13.907951 2021-03-24T21:51:15.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:14 smithi125 conmon[19936]: +0000 mon.a (mon.0) 2021-03-24T21:51:15.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:14 smithi125 conmon[19936]: 413 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2021-03-24T21:51:16.361 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:15 smithi142 conmon[20125]: cluster 2021-03-24T21:51:14. 2021-03-24T21:51:16.361 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:15 smithi142 conmon[20125]: 624161+0000 mgr.y (mgr.14144) 165 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:51:16.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:15 smithi142 conmon[20125]: audit 2021-03- 2021-03-24T21:51:16.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:15 smithi142 conmon[20125]: 24T21:51:14.918134+0000 mon.a (mon.0) 414 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2021-03-24T21:51:16.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:15 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:51:16.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:15 smithi142 conmon[20125]: T21:51:14.918274+0000 mon.a (mon.0) 415 2021-03-24T21:51:16.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:15 smithi142 conmon[20125]: : cluster [DBG] osdmap e46: 7 total, 6 up, 7 in 2021-03-24T21:51:16.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:15 smithi142 conmon[20125]: audit 2021-03-24T21:51:14.918685+0000 2021-03-24T21:51:16.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:15 smithi142 conmon[20125]: mon.a (mon.0) 416 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-03-24T21:51:16.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:15 smithi142 conmon[20125]: 2021-03-24T21:51:16.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:15 smithi142 conmon[20125]: audit 2021-03- 2021-03-24T21:51:16.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:15 smithi142 conmon[20125]: 24T21:51:14.925044+0000 mon.a (mon.0) 417 : audit 2021-03-24T21:51:16.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:15 smithi142 conmon[20125]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-03-24T21:51:16.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:15 smithi142 conmon[20125]: cluster 2021-03-24T21:51:16.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:15 smithi142 conmon[20125]: 2021-03-24T21:51:15.009786+0000 mon.a (mon.0) 418 : 2021-03-24T21:51:16.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:15 smithi142 conmon[20125]: cluster [INF] osd.6 v1:172.21.15.142:6808/4048483913 boot 2021-03-24T21:51:16.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:15 smithi142 conmon[20125]: cluster 2021-03-24T21:51:15.009912+0000 2021-03-24T21:51:16.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:15 smithi142 conmon[20125]: mon.a (mon.0) 419 : cluster [DBG] osdmap e47: 7 total, 7 up, 7 in 2021-03-24T21:51:16.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:15 smithi142 conmon[20125]: audit 2021-03- 2021-03-24T21:51:16.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:15 smithi142 conmon[20125]: 24T21:51:15.010908+0000 mon.a (mon.0) 2021-03-24T21:51:16.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:15 smithi142 conmon[20125]: 420 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-03-24T21:51:16.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:15 smithi125 conmon[19936]: cluster 2021-03-24T21:51: 2021-03-24T21:51:16.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:15 smithi125 conmon[19936]: 14.624161+0000 mgr.y (mgr.14144) 165 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:51:16.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:15 smithi125 conmon[19936]: audit 2021-03-24T21: 2021-03-24T21:51:16.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:15 smithi125 conmon[19936]: 51:14.918134+0000 mon.a (mon.0) 414 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2021-03-24T21:51:16.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:15 smithi125 conmon[19936]: cluster 2021-03-24T21:51: 2021-03-24T21:51:16.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:15 smithi125 conmon[19936]: 14.918274+0000 mon.a (mon.0) 415 : cluster [DBG] osdmap e46: 7 total, 6 up, 7 in 2021-03-24T21:51:16.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:15 smithi125 conmon[19936]: 2021-03-24T21:51:16.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:15 smithi125 conmon[19936]: audit 2021-03-24T21: 2021-03-24T21:51:16.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:15 smithi125 conmon[19936]: 51:14.918685+0000 mon.a (mon.0) 416 : audit [DBG] 2021-03-24T21:51:16.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:15 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-03-24T21:51:16.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:15 smithi125 conmon[19936]: audit 2021-03-24T21:51:16.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:15 smithi125 conmon[19936]: 2021-03-24T21:51:14.925044+0000 2021-03-24T21:51:16.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:15 smithi125 conmon[19936]: mon.a (mon.0) 417 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-03-24T21:51:16.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:15 smithi125 conmon[19936]: 2021-03-24T21:51:16.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:15 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:51:16.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:15 smithi125 conmon[19936]: :51:15.009786+0000 mon.a (mon.0) 418 2021-03-24T21:51:16.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:15 smithi125 conmon[19936]: : cluster [INF] osd.6 v1:172.21.15.142:6808/4048483913 boot 2021-03-24T21:51:16.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:15 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:51:16.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:15 smithi125 conmon[19936]: 21:51:15.009912+0000 mon.a (mon. 2021-03-24T21:51:16.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:15 smithi125 conmon[19936]: 0) 419 : cluster [DBG] osdmap e47: 7 total, 7 up, 7 in 2021-03-24T21:51:16.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:15 smithi125 conmon[19936]: audit 2021-03- 2021-03-24T21:51:16.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:15 smithi125 conmon[19936]: 24T21:51:15.010908+0000 mon.a ( 2021-03-24T21:51:16.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:15 smithi125 conmon[19936]: mon.0) 420 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-03-24T21:51:16.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:15 smithi125 conmon[24927]: cluster 2021-03-24T21:51: 2021-03-24T21:51:16.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:15 smithi125 conmon[24927]: 14.624161+0000 mgr.y (mgr.14144) 165 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:51:16.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:15 smithi125 conmon[24927]: audit 2021-03-24 2021-03-24T21:51:16.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:15 smithi125 conmon[24927]: T21:51:14.918134+0000 mon.a (mon 2021-03-24T21:51:16.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:15 smithi125 conmon[24927]: .0) 414 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2021-03-24T21:51:16.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:15 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:51:16.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:15 smithi125 conmon[24927]: T21:51:14.918274+0000 mon.a (mon.0) 415 2021-03-24T21:51:16.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:15 smithi125 conmon[24927]: : cluster [DBG] osdmap e46: 7 total, 6 up, 7 in 2021-03-24T21:51:16.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:15 smithi125 conmon[24927]: audit 2021-03-24T21:51:16.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:15 smithi125 conmon[24927]: 2021-03-24T21:51:14.918685+0000 mon.a (mon. 2021-03-24T21:51:16.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:15 smithi125 conmon[24927]: 0) 416 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-03-24T21:51:16.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:15 smithi125 conmon[24927]: audit 2021-03-24T21:51:14.925044 2021-03-24T21:51:16.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:15 smithi125 conmon[24927]: +0000 mon.a (mon.0) 417 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-03-24T21:51:16.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:15 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:51:16.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:15 smithi125 conmon[24927]: 21:51:15.009786+0000 mon.a (mon.0) 418 : 2021-03-24T21:51:16.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:15 smithi125 conmon[24927]: cluster [INF] osd.6 v1:172.21.15.142:6808/4048483913 boot 2021-03-24T21:51:16.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:15 smithi125 conmon[24927]: cluster 2021-03-24T21:51:16.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:15 smithi125 conmon[24927]: 2021-03-24T21:51:15.009912+0000 mon.a (mon.0) 419 : 2021-03-24T21:51:16.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:15 smithi125 conmon[24927]: cluster [DBG] osdmap e47: 7 total, 7 up, 7 in 2021-03-24T21:51:16.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:15 smithi125 conmon[24927]: audit 2021-03-24T21:51:16.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:15 smithi125 conmon[24927]: 2021-03-24T21:51:15.010908+0000 mon.a (mon.0) 2021-03-24T21:51:16.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:15 smithi125 conmon[24927]: 420 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-03-24T21:51:17.164 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:17 smithi142 conmon[20125]: cluster 2021-03-24T21:51:14.327730+0000 osd.6 (osd.6) 1 2021-03-24T21:51:17.165 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:17 smithi142 conmon[20125]: : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:51:17.165 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:17 smithi142 conmon[20125]: cluster 2021-03-24T21:51:14.328019+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:51:17.165 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:17 smithi142 conmon[20125]: cluster 2021-03-24T21:51:16.007482+0000 mon.a (mon. 2021-03-24T21:51:17.165 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:17 smithi142 conmon[20125]: 0) 421 : cluster [DBG] osdmap e48: 7 total, 7 up, 7 in 2021-03-24T21:51:17.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:17 smithi125 conmon[19936]: cluster 2021-03-24T21:51:14. 2021-03-24T21:51:17.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:17 smithi125 conmon[19936]: 327730+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:51:17.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:17 smithi125 conmon[19936]: cluster 2021-03-24T21:51:14.328019+0000 2021-03-24T21:51:17.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:17 smithi125 conmon[19936]: osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:51:17.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:17 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:51:17.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:17 smithi125 conmon[19936]: :51:16.007482+0000 mon.a (mon.0) 421 : cluster [DBG] 2021-03-24T21:51:17.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:17 smithi125 conmon[19936]: osdmap e48: 7 total, 7 up, 7 in 2021-03-24T21:51:17.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:17 smithi125 conmon[24927]: cluster 2021-03-24T21:51:14.327730+0000 2021-03-24T21:51:17.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:17 smithi125 conmon[24927]: osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:51:17.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:17 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:51:17.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:17 smithi125 conmon[24927]: :51:14.328019+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:51:17.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:17 smithi125 conmon[24927]: cluster 2021-03-24T21:51:17.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:17 smithi125 conmon[24927]: 2021-03-24T21:51:16.007482+0000 mon.a 2021-03-24T21:51:17.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:17 smithi125 conmon[24927]: (mon.0) 421 : cluster [DBG] osdmap e48: 7 total, 7 up, 7 in 2021-03-24T21:51:18.165 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:18 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:51:18.376 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:18 smithi142 conmon[20125]: -24T21:51:16.624753+0000 mgr.y (mgr.14144) 166 : 2021-03-24T21:51:18.376 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:18 smithi142 conmon[20125]: cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-03-24T21:51:18.376 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:18 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:51:18.377 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:18 smithi142 conmon[20125]: -03-24T21: 2021-03-24T21:51:18.377 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:18 smithi142 conmon[20125]: 51:17.015631 2021-03-24T21:51:18.377 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:18 smithi142 conmon[20125]: +0000 mon.a (mon 2021-03-24T21:51:18.377 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:18 smithi142 conmon[20125]: .0) 422 : 2021-03-24T21:51:18.378 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:18 smithi142 conmon[20125]: cluster [DBG] osdmap e49: 7 total, 7 up, 7 in 2021-03-24T21:51:18.378 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:18 smithi142 conmon[20125]: 2021-03-24T21:51:18.378 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:18 smithi142 conmon[20125]: audit 2021-03-24T21:51:18.378 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:18 smithi142 conmon[20125]: 2021-03-24 2021-03-24T21:51:18.379 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:18 smithi142 conmon[20125]: T21:51: 2021-03-24T21:51:18.379 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:18 smithi142 conmon[20125]: 17.524138+0000 2021-03-24T21:51:18.379 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:18 smithi142 conmon[20125]: mon.a (mon. 2021-03-24T21:51:18.379 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:18 smithi142 conmon[20125]: 0) 423 : audit 2021-03-24T21:51:18.379 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:18 smithi142 conmon[20125]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-24T21:51:18.380 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:18 smithi142 conmon[20125]: audit 2021-03-24T21:51:18.380 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:18 smithi142 conmon[20125]: 2021-03-24T21 2021-03-24T21:51:18.380 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:18 smithi142 conmon[20125]: :51:17. 2021-03-24T21:51:18.380 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:18 smithi142 conmon[20125]: 527323+0000 mon.a ( 2021-03-24T21:51:18.380 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:18 smithi142 conmon[20125]: mon.0) 424 : 2021-03-24T21:51:18.381 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:18 smithi142 conmon[20125]: audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-24T21:51:18.381 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:18 smithi142 conmon[20125]: 2021-03-24T21:51:18.381 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:18 smithi142 conmon[20125]: audit 2021-03-24T21:51:18.381 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:18 smithi142 conmon[20125]: 2021-03- 2021-03-24T21:51:18.382 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:18 smithi142 conmon[20125]: 24T21:51 2021-03-24T21:51:18.382 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:18 smithi142 conmon[20125]: :17.528875+0000 2021-03-24T21:51:18.382 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:18 smithi142 conmon[20125]: mon.a (mon. 2021-03-24T21:51:18.382 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:18 smithi142 conmon[20125]: 0) 425 : audit 2021-03-24T21:51:18.382 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:18 smithi142 conmon[20125]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:51:18.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:18 smithi125 conmon[19936]: cluster 2021-03-24T21:51: 2021-03-24T21:51:18.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:18 smithi125 conmon[19936]: 16.624753+0000 mgr.y (mgr.14144) 166 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-03-24T21:51:18.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:18 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T21:51:18.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:18 smithi125 conmon[19936]: T21:51:17.015631+0000 mon.a (mon.0) 422 : cluster [DBG] osdmap e49: 7 total, 7 up, 7 in 2021-03-24T21:51:18.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:18 smithi125 conmon[19936]: audit 2021 2021-03-24T21:51:18.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:18 smithi125 conmon[19936]: -03-24T21:51:17.524138+0000 mon.a 2021-03-24T21:51:18.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:18 smithi125 conmon[19936]: (mon.0) 423 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-24T21:51:18.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:18 smithi125 conmon[19936]: audit 2021 2021-03-24T21:51:18.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:18 smithi125 conmon[19936]: -03-24T21:51:17.527323+0000 2021-03-24T21:51:18.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:18 smithi125 conmon[19936]: mon.a (mon.0) 424 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-24T21:51:18.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:18 smithi125 conmon[19936]: audit 2021-03-24T21:51:18.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:18 smithi125 conmon[19936]: 2021-03-24T21:51:17. 2021-03-24T21:51:18.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:18 smithi125 conmon[19936]: 528875+0000 mon.a (mon.0) 425 : audit [DBG] 2021-03-24T21:51:18.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:18 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:51:18.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:18 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:51:18.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:18 smithi125 conmon[24927]: T21:51:16.624753+0000 mgr.y (mgr.14144) 166 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-03-24T21:51:18.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:18 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:51:18.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:18 smithi125 conmon[24927]: -03-24T21:51:17 2021-03-24T21:51:18.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:18 smithi125 conmon[24927]: .015631+0000 mon.a (mon. 2021-03-24T21:51:18.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:18 smithi125 conmon[24927]: 0) 422 : cluster [DBG] 2021-03-24T21:51:18.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:18 smithi125 conmon[24927]: osdmap e49: 7 total, 7 up, 7 in 2021-03-24T21:51:18.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:18 smithi125 conmon[24927]: audit 2021-03-24T21 2021-03-24T21:51:18.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:18 smithi125 conmon[24927]: :51:17.524138+0000 mon.a (mon.0) 423 : 2021-03-24T21:51:18.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:18 smithi125 conmon[24927]: audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-24T21:51:18.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:18 smithi125 conmon[24927]: audit 2021-03-24T21:51:18.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:18 smithi125 conmon[24927]: 2021-03-24T21:51:17.527323+0000 mon.a (mon.0) 424 : audit [INF] 2021-03-24T21:51:18.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:18 smithi125 conmon[24927]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-24T21:51:18.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:18 smithi125 conmon[24927]: audit 2021-03-24T 2021-03-24T21:51:18.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:18 smithi125 conmon[24927]: 21:51:17.528875+0000 mon.a (mon.0) 425 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:51:19.329 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:19 smithi142 conmon[20125]: audit 2021-03-24T21:51:17.522621+0000 mgr.y (mgr.14144 2021-03-24T21:51:19.329 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:19 smithi142 conmon[20125]: ) 167 : audit [DBG] from='client.24304 v1:172.21.15.142:0/2018747614' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:51:19.329 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:19 smithi142 conmon[20125]: cephadm 2021-03-24 2021-03-24T21:51:19.330 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:19 smithi142 conmon[20125]: T21:51:17.526842+0000 mgr.y ( 2021-03-24T21:51:19.330 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:19 smithi142 conmon[20125]: mgr.14144) 168 : cephadm [INF] 2021-03-24T21:51:19.330 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:19 smithi142 conmon[20125]: Applying service osd.None on host smithi142... 2021-03-24T21:51:19.330 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:19 smithi142 conmon[20125]: cephadm 2021-03-24T21:51:19.331 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:19 smithi142 conmon[20125]: 2021-03-24T 2021-03-24T21:51:19.331 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:19 smithi142 conmon[20125]: 21:51:18.148019 2021-03-24T21:51:19.331 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:19 smithi142 conmon[20125]: +0000 mgr.y (mgr.14144) 2021-03-24T21:51:19.331 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:19 smithi142 conmon[20125]: 169 : cephadm [INF] Detected new or changed devices on smithi142 2021-03-24T21:51:19.331 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:19 smithi142 conmon[20125]: audit 2021-03-24T21:51:19.332 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:19 smithi142 conmon[20125]: 2021-03-24T21 2021-03-24T21:51:19.332 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:19 smithi142 conmon[20125]: :51:18.157198+0000 2021-03-24T21:51:19.332 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:19 smithi142 conmon[20125]: mon.a (mon.0) 426 : 2021-03-24T21:51:19.332 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:19 smithi142 conmon[20125]: audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:19.333 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:19 smithi142 conmon[20125]: audit 2021-03-24T21:51:19.333 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:19 smithi142 conmon[20125]: 2021-03-24 2021-03-24T21:51:19.333 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:19 smithi142 conmon[20125]: T21:51:18.164779 2021-03-24T21:51:19.333 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:19 smithi142 conmon[20125]: +0000 mon.a (mon.0) 427 2021-03-24T21:51:19.333 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:19 smithi142 conmon[20125]: : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:19.334 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:19 smithi142 conmon[20125]: audit 2021-03-24T21:51:19.334 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:19 smithi142 conmon[20125]: 2021-03-24T21 2021-03-24T21:51:19.334 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:19 smithi142 conmon[20125]: :51:18.167268 2021-03-24T21:51:19.334 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:19 smithi142 conmon[20125]: +0000 mon.a (mon. 2021-03-24T21:51:19.335 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:19 smithi142 conmon[20125]: 0) 428 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:51:19.335 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:19 smithi142 conmon[20125]: 2021-03-24T21:51:19.335 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:19 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:51:19.336 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:19 smithi142 conmon[20125]: 03-24T21: 2021-03-24T21:51:19.336 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:19 smithi142 conmon[20125]: 51:18.174725+0000 mon.a 2021-03-24T21:51:19.336 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:19 smithi142 conmon[20125]: (mon.0) 429 : audit 2021-03-24T21:51:19.336 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:19 smithi142 conmon[20125]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:19.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:19 smithi125 conmon[24927]: audit 2021-03-24T21:51 2021-03-24T21:51:19.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:19 smithi125 conmon[24927]: :17.522621+0000 mgr.y (mgr.14144) 167 : audit [DBG] from='client.24304 v1:172.21.15.142:0/2018747614' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:51:19.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:19 smithi125 conmon[24927]: cephadm 2021-03-24T21:51:17.526842 2021-03-24T21:51:19.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:19 smithi125 conmon[24927]: +0000 mgr.y (mgr.14144) 168 : cephadm [INF] Applying service osd.None on host smithi142... 2021-03-24T21:51:19.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:19 smithi125 conmon[24927]: 2021-03-24T21:51:19.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:19 smithi125 conmon[24927]: cephadm 2021- 2021-03-24T21:51:19.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:19 smithi125 conmon[24927]: 03-24T21:51:18.148019+0000 mgr.y 2021-03-24T21:51:19.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:19 smithi125 conmon[24927]: (mgr.14144) 169 : cephadm [INF] Detected new or changed devices on smithi142 2021-03-24T21:51:19.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:19 smithi125 conmon[24927]: audit 2021 2021-03-24T21:51:19.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:19 smithi125 conmon[24927]: -03-24T21:51:18.157198+0000 mon.a 2021-03-24T21:51:19.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:19 smithi125 conmon[24927]: (mon.0) 426 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:19.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:19 smithi125 conmon[24927]: audit 2021 2021-03-24T21:51:19.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:19 smithi125 conmon[24927]: -03-24T21:51:18.164779+0000 mon.a 2021-03-24T21:51:19.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:19 smithi125 conmon[24927]: (mon.0) 427 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:19.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:19 smithi125 conmon[24927]: 2021-03-24T21:51:19.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:19 smithi125 conmon[24927]: audit 2021-03-24T21 2021-03-24T21:51:19.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:19 smithi125 conmon[24927]: :51:18.167268+0000 mon.a (mon. 2021-03-24T21:51:19.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:19 smithi125 conmon[24927]: 0) 428 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:51:19.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:19 smithi125 conmon[24927]: audit 2021 2021-03-24T21:51:19.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:19 smithi125 conmon[24927]: -03-24T21:51:18.174725+0000 2021-03-24T21:51:19.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:19 smithi125 conmon[24927]: mon.a (mon.0) 429 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:19.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:19 smithi125 conmon[24927]: 2021-03-24T21:51:19.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:19 smithi125 conmon[19936]: audit 2021-03-24T 2021-03-24T21:51:19.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:19 smithi125 conmon[19936]: 21:51:17.522621+0000 mgr.y (mgr.14144) 167 : audit [DBG] from='client.24304 v1:172.21.15.142:0/2018747614' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:51:19.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:19 smithi125 conmon[19936]: cephadm 2021- 2021-03-24T21:51:19.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:19 smithi125 conmon[19936]: 03-24T21:51:17.526842+0000 mgr.y ( 2021-03-24T21:51:19.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:19 smithi125 conmon[19936]: mgr.14144) 168 : cephadm [INF] Applying service osd.None on host smithi142... 2021-03-24T21:51:19.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:19 smithi125 conmon[19936]: cephadm 2021-03-24T21:51:19.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:19 smithi125 conmon[19936]: 2021-03-24T21:51 2021-03-24T21:51:19.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:19 smithi125 conmon[19936]: :18.148019+0000 mgr.y ( 2021-03-24T21:51:19.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:19 smithi125 conmon[19936]: mgr.14144) 169 : cephadm [INF] 2021-03-24T21:51:19.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:19 smithi125 conmon[19936]: Detected new or changed devices on smithi142 2021-03-24T21:51:19.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:19 smithi125 conmon[19936]: audit 2021-03-24T21:51:19.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:19 smithi125 conmon[19936]: 2021-03- 2021-03-24T21:51:19.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:19 smithi125 conmon[19936]: 24T21:51:18.157198+0000 2021-03-24T21:51:19.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:19 smithi125 conmon[19936]: mon.a (mon.0) 426 2021-03-24T21:51:19.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:19 smithi125 conmon[19936]: : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:19.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:19 smithi125 conmon[19936]: audit 2021 2021-03-24T21:51:19.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:19 smithi125 conmon[19936]: -03-24T21:51: 2021-03-24T21:51:19.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:19 smithi125 conmon[19936]: 18.164779+0000 mon.a (mon. 2021-03-24T21:51:19.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:19 smithi125 conmon[19936]: 0) 427 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:19.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:19 smithi125 conmon[19936]: 2021-03-24T21:51:19.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:19 smithi125 conmon[19936]: audit 2021-03-24T21:51:19.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:19 smithi125 conmon[19936]: 2021-03-24T21:51 2021-03-24T21:51:19.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:19 smithi125 conmon[19936]: :18.167268+0000 mon.a ( 2021-03-24T21:51:19.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:19 smithi125 conmon[19936]: mon.0) 428 : audit [DBG] 2021-03-24T21:51:19.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:19 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:51:19.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:19 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:51:19.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:19 smithi125 conmon[19936]: 03-24T21:51: 2021-03-24T21:51:19.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:19 smithi125 conmon[19936]: 18.174725+0000 mon.a (mon 2021-03-24T21:51:19.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:19 smithi125 conmon[19936]: .0) 429 : audit [INF] 2021-03-24T21:51:19.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:19 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:20.610 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:20 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:51:20.611 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:20 smithi142 conmon[20125]: 51:18.625348+0000 mgr.y (mgr.14144) 170 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-03-24T21:51:20.651 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:20 smithi125 conmon[19936]: cluster 2021-03-24T21:51: 2021-03-24T21:51:20.651 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:20 smithi125 conmon[19936]: 18.625348+0000 mgr.y (mgr.14144) 170 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-03-24T21:51:20.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:20 smithi125 conmon[24927]: cluster 2021-03-24T21:51 2021-03-24T21:51:20.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:20 smithi125 conmon[24927]: :18.625348+0000 mgr.y (mgr.14144) 170 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-03-24T21:51:21.611 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:21 smithi142 conmon[20125]: audit 2021-03-24 2021-03-24T21:51:21.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:21 smithi142 conmon[20125]: T21:51:20.210501+0000 mon.b (mon.1) 19 : audit [INF] from='client.? v1:172.21.15.142:0/226247977' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "682ba5ce-0172-48e3-82f8-57a3f98c911b"}]: dispatch 2021-03-24T21:51:21.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:21 smithi142 conmon[20125]: audit 2021-03-24T21: 2021-03-24T21:51:21.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:21 smithi142 conmon[20125]: 51:20.213191+0000 mon.a (mon.0) 2021-03-24T21:51:21.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:21 smithi142 conmon[20125]: 430 : audit [INF] from='client.24310 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "682ba5ce-0172-48e3-82f8-57a3f98c911b"}]: dispatch 2021-03-24T21:51:21.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:21 smithi142 conmon[20125]: audit 2021-03-24T21 2021-03-24T21:51:21.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:21 smithi142 conmon[20125]: :51:20.221440+0000 mon.a (mon. 2021-03-24T21:51:21.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:21 smithi142 conmon[20125]: 0) 431 : audit [INF] from='client.24310 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "682ba5ce-0172-48e3-82f8-57a3f98c911b"}]': finished 2021-03-24T21:51:21.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:21 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:51:21.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:21 smithi142 conmon[20125]: 24T21:51:20.221616+0000 mon.a 2021-03-24T21:51:21.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:21 smithi142 conmon[20125]: (mon.0) 432 : cluster [DBG] osdmap e50: 8 total, 7 up, 8 in 2021-03-24T21:51:21.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:21 smithi142 conmon[20125]: 2021-03-24T21:51:21.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:21 smithi142 conmon[20125]: audit 2021-03-24T21:51:21.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:21 smithi142 conmon[20125]: 2021-03-24T21:51:20.221872+0000 mon.a 2021-03-24T21:51:21.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:21 smithi142 conmon[20125]: (mon.0) 433 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-03-24T21:51:21.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:21 smithi142 conmon[20125]: 2021-03-24T21:51:21.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:21 smithi142 conmon[20125]: audit 2021-03-24T21:51:21.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:21 smithi142 conmon[20125]: 2021-03-24T21:51:20.855687+0000 mon.b 2021-03-24T21:51:21.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:21 smithi142 conmon[20125]: (mon.1) 20 : audit [DBG] from='client.? v1:172.21.15.142:0/3460101721' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-24T21:51:21.652 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:21 smithi125 conmon[19936]: audit 2021-03-24T21:51:20.210501+0000 mon.b (mon.1) 19 : audit [INF] from='client.? v1:172.21.15.142:0/226247977' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "682ba5ce-0172-48e3-82f8-57a3f98c911b"}]: dispatch 2021-03-24T21:51:21.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:21 smithi125 conmon[19936]: 2021-03-24T21:51:21.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:21 smithi125 conmon[19936]: audit 2021-03-24T21:51:20.213191+0000 2021-03-24T21:51:21.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:21 smithi125 conmon[19936]: mon.a (mon.0) 430 : audit [INF] from='client.24310 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "682ba5ce-0172-48e3-82f8-57a3f98c911b"}]: dispatch 2021-03-24T21:51:21.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:21 smithi125 conmon[19936]: 2021-03-24T21:51:21.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:21 smithi125 conmon[19936]: audit 2021-03-24T21 2021-03-24T21:51:21.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:21 smithi125 conmon[19936]: :51:20.221440+0000 mon.a (mon.0) 431 : audit [INF] from='client.24310 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "682ba5ce-0172-48e3-82f8-57a3f98c911b"}]': finished 2021-03-24T21:51:21.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:21 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:51:21.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:21 smithi125 conmon[19936]: -24T21:51:20.221616+0000 mon.a 2021-03-24T21:51:21.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:21 smithi125 conmon[19936]: (mon.0) 432 : cluster 2021-03-24T21:51:21.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:21 smithi125 conmon[19936]: [DBG] osdmap e50: 8 total, 7 up, 8 in 2021-03-24T21:51:21.657 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:21 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:51:21.657 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:21 smithi125 conmon[19936]: 03-24T21:51:20.221872+0000 2021-03-24T21:51:21.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:21 smithi125 conmon[19936]: mon.a (mon.0) 433 : audit 2021-03-24T21:51:21.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:21 smithi125 conmon[19936]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-03-24T21:51:21.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:21 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:51:21.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:21 smithi125 conmon[19936]: 03-24T21:51:20.855687+0000 2021-03-24T21:51:21.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:21 smithi125 conmon[19936]: mon.b (mon.1) 20 : audit 2021-03-24T21:51:21.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:21 smithi125 conmon[19936]: [DBG] from='client.? v1:172.21.15.142:0/3460101721' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-24T21:51:21.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:21 smithi125 conmon[24927]: audit 2021-03-24T21: 2021-03-24T21:51:21.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:21 smithi125 conmon[24927]: 51:20.210501+0000 mon.b (mon.1) 19 : audit [INF] from='client.? v1:172.21.15.142:0/226247977' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "682ba5ce-0172-48e3-82f8-57a3f98c911b"}]: dispatch 2021-03-24T21:51:21.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:21 smithi125 conmon[24927]: audit 2021-03 2021-03-24T21:51:21.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:21 smithi125 conmon[24927]: -24T21:51:20.213191 2021-03-24T21:51:21.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:21 smithi125 conmon[24927]: +0000 mon.a (mon.0) 430 : audit 2021-03-24T21:51:21.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:21 smithi125 conmon[24927]: [INF] from='client.24310 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "682ba5ce-0172-48e3-82f8-57a3f98c911b"}]: dispatch 2021-03-24T21:51:21.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:21 smithi125 conmon[24927]: audit 2021-03-24T21:51:21.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:21 smithi125 conmon[24927]: 2021-03- 2021-03-24T21:51:21.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:21 smithi125 conmon[24927]: 24T21:51:20.221440+0000 mon.a (mon 2021-03-24T21:51:21.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:21 smithi125 conmon[24927]: .0) 431 : audit [INF] from='client.24310 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "682ba5ce-0172-48e3-82f8-57a3f98c911b"}]': finished 2021-03-24T21:51:21.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:21 smithi125 conmon[24927]: cluster 2021-03-24T21:51: 2021-03-24T21:51:21.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:21 smithi125 conmon[24927]: 20.221616+0000 mon.a (mon.0) 2021-03-24T21:51:21.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:21 smithi125 conmon[24927]: 432 : cluster [DBG] osdmap e50: 8 total, 7 up, 8 in 2021-03-24T21:51:21.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:21 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T21:51:21.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:21 smithi125 conmon[24927]: 24T21:51:20.221872+0000 2021-03-24T21:51:21.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:21 smithi125 conmon[24927]: mon.a (mon.0) 433 : 2021-03-24T21:51:21.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:21 smithi125 conmon[24927]: audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-03-24T21:51:21.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:21 smithi125 conmon[24927]: audit 2021-03-24T21:51:21.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:21 smithi125 conmon[24927]: 2021-03-24T21 2021-03-24T21:51:21.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:21 smithi125 conmon[24927]: :51:20.855687+0000 2021-03-24T21:51:21.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:21 smithi125 conmon[24927]: mon.b (mon.1) 20 : audit 2021-03-24T21:51:21.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:21 smithi125 conmon[24927]: [DBG] from='client.? v1:172.21.15.142:0/3460101721' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-24T21:51:22.611 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:22 smithi142 conmon[20125]: cluster 2021-03-24T21:51:20.625878+0000 mgr.y (mgr.14144) 171 : cluster 2021-03-24T21:51:22.611 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:22 smithi142 conmon[20125]: [DBG] pgmap v168: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-03-24T21:51:22.652 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:22 smithi125 conmon[19936]: cluster 2021-03-24T21:51 2021-03-24T21:51:22.652 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:22 smithi125 conmon[19936]: :20.625878+0000 mgr.y (mgr.14144) 171 : cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-03-24T21:51:22.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:22 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:51:22.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:22 smithi125 conmon[24927]: 51:20.625878+0000 mgr.y (mgr.14144) 171 : cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-03-24T21:51:24.609 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:24 smithi142 conmon[20125]: cluster 2021-03-24T21:51:22.626463+0000 mgr.y (mgr 2021-03-24T21:51:24.610 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:24 smithi142 conmon[20125]: .14144) 172 : cluster [DBG] pgmap v169: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-03-24T21:51:24.652 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:24 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:51:24.652 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:24 smithi125 conmon[19936]: :51:22.626463+0000 mgr.y (mgr.14144) 172 : cluster [DBG] pgmap v169: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-03-24T21:51:24.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:24 smithi125 conmon[24927]: cluster 2021-03-24T21:51 2021-03-24T21:51:24.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:24 smithi125 conmon[24927]: :22.626463+0000 mgr.y (mgr.14144) 172 : cluster [DBG] pgmap v169: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-03-24T21:51:26.596 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:26 smithi142 conmon[20125]: cluster 2021-03-24T21:51:24.627018+0000 mgr.y ( 2021-03-24T21:51:26.597 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:26 smithi142 conmon[20125]: mgr.14144) 173 : cluster [DBG] pgmap v170: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2021-03-24T21:51:26.652 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:26 smithi125 conmon[19936]: cluster 2021-03-24T21:51:24.627018 2021-03-24T21:51:26.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:26 smithi125 conmon[19936]: +0000 mgr.y (mgr.14144) 173 : cluster [DBG] pgmap v170: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2021-03-24T21:51:26.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:26 smithi125 conmon[24927]: cluster 2021-03-24T21:51:24. 2021-03-24T21:51:26.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:26 smithi125 conmon[24927]: 627018+0000 mgr.y (mgr.14144) 173 : cluster [DBG] pgmap v170: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2021-03-24T21:51:28.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:28 smithi142 conmon[20125]: cluster 2021-03-24T21:51:26.627583+0000 mgr.y (mgr 2021-03-24T21:51:28.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:28 smithi142 conmon[20125]: .14144) 174 : cluster [DBG] pgmap v171: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2021-03-24T21:51:28.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:28 smithi142 conmon[20125]: audit 2021-03-24T21:51:27.387336+0000 2021-03-24T21:51:28.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:28 smithi142 conmon[20125]: mon.a (mon.0) 434 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2021-03-24T21:51:28.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:28 smithi142 conmon[20125]: audit 2021-03-24 2021-03-24T21:51:28.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:28 smithi142 conmon[20125]: T21:51:27.388970+0000 mon.a (mon.0) 435 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:51:28.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:28 smithi142 conmon[20125]: cephadm 2021-03-24T 2021-03-24T21:51:28.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:28 smithi142 conmon[20125]: 21:51:27.390401+0000 mgr.y (mgr.14144) 175 2021-03-24T21:51:28.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:28 smithi142 conmon[20125]: : cephadm [INF] Deploying daemon osd.7 on smithi142 2021-03-24T21:51:28.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:28 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:51:28.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:28 smithi125 conmon[19936]: 21:51:26.627583+0000 mgr.y (mgr.14144) 174 : cluster [DBG] pgmap v171: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2021-03-24T21:51:28.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:28 smithi125 conmon[19936]: audit 2021-03-24T21:51:27.387336+0000 mon.a (mon.0 2021-03-24T21:51:28.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:28 smithi125 conmon[19936]: ) 434 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2021-03-24T21:51:28.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:28 smithi125 conmon[19936]: audit 2021 2021-03-24T21:51:28.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:28 smithi125 conmon[19936]: -03-24T21:51:27 2021-03-24T21:51:28.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:28 smithi125 conmon[19936]: .388970+0000 mon.a (mon.0 2021-03-24T21:51:28.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:28 smithi125 conmon[19936]: ) 435 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:51:28.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:28 smithi125 conmon[19936]: cephadm 2021 2021-03-24T21:51:28.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:28 smithi125 conmon[19936]: -03-24T21:51:27. 2021-03-24T21:51:28.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:28 smithi125 conmon[19936]: 390401+0000 mgr.y (mgr.14144 2021-03-24T21:51:28.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:28 smithi125 conmon[19936]: ) 175 : cephadm [INF] Deploying daemon osd.7 on smithi142 2021-03-24T21:51:28.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:28 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:51:28.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:28 smithi125 conmon[24927]: T21:51:26.627583+0000 mgr.y (mgr.14144) 174 : cluster [DBG] pgmap v171: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2021-03-24T21:51:28.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:28 smithi125 conmon[24927]: 2021-03-24T21:51:28.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:28 smithi125 conmon[24927]: audit 2021-03-24T21:51:28.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:28 smithi125 conmon[24927]: 2021-03-24T21:51 2021-03-24T21:51:28.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:28 smithi125 conmon[24927]: :27.387336+0000 mon.a (mon.0) 2021-03-24T21:51:28.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:28 smithi125 conmon[24927]: 434 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2021-03-24T21:51:28.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:28 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T21:51:28.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:28 smithi125 conmon[24927]: 24T21:51:27.388970+0000 mon.a 2021-03-24T21:51:28.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:28 smithi125 conmon[24927]: (mon.0) 435 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:51:28.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:28 smithi125 conmon[24927]: 2021-03-24T21:51:28.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:28 smithi125 conmon[24927]: cephadm 2021-03-24T21:51:28.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:28 smithi125 conmon[24927]: 2021-03-24T21:51: 2021-03-24T21:51:28.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:28 smithi125 conmon[24927]: 27.390401+0000 mgr.y (mgr. 2021-03-24T21:51:28.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:28 smithi125 conmon[24927]: 14144) 175 : cephadm [INF] Deploying daemon osd.7 on smithi142 2021-03-24T21:51:30.529 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:30 smithi142 conmon[20125]: cluster 2021-03-24T21:51:28.628150+0000 mgr.y (mgr.14144) 176 : cluster 2021-03-24T21:51:30.530 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:30 smithi142 conmon[20125]: [DBG] pgmap v172: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2021-03-24T21:51:30.530 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:30 smithi142 conmon[20125]: cluster 2021-03-24T21:51:28.965481+0000 osd.1 (osd.1) 5 : 2021-03-24T21:51:30.530 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:30 smithi142 conmon[20125]: cluster [DBG] purged_snaps scrub starts 2021-03-24T21:51:30.531 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:30 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:51:30.531 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:30 smithi142 conmon[20125]: 03-24T21:51:28.965676+0000 osd.1 (osd. 2021-03-24T21:51:30.531 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:30 smithi142 conmon[20125]: 1) 6 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:51:30.531 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:30 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:51:30.531 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:30 smithi142 conmon[20125]: 24T21:51:29.251541+0000 mon.a ( 2021-03-24T21:51:30.532 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:30 smithi142 conmon[20125]: mon.0) 436 : cluster [DBG] osdmap e51: 8 total, 7 up, 8 in 2021-03-24T21:51:30.532 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:30 smithi142 conmon[20125]: audit 2021-03 2021-03-24T21:51:30.532 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:30 smithi142 conmon[20125]: -24T21:51:29.251914+0000 mon.a ( 2021-03-24T21:51:30.532 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:30 smithi142 conmon[20125]: mon.0) 437 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-03-24T21:51:30.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:30 smithi125 conmon[24927]: cluster 2021-03-24T21:51:28.628150 2021-03-24T21:51:30.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:30 smithi125 conmon[24927]: +0000 mgr.y (mgr.14144) 176 : cluster [DBG] pgmap v172: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2021-03-24T21:51:30.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:30 smithi125 conmon[24927]: cluster 2021-03-24T21:51:28 2021-03-24T21:51:30.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:30 smithi125 conmon[24927]: .965481+0000 osd.1 (osd.1) 5 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:51:30.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:30 smithi125 conmon[24927]: cluster 2021-03-24T21:51:30.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:30 smithi125 conmon[24927]: 2021-03-24T21:51:28.965676+0000 osd.1 (osd.1) 6 : 2021-03-24T21:51:30.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:30 smithi125 conmon[24927]: cluster [DBG] purged_snaps scrub ok 2021-03-24T21:51:30.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:30 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:51:30.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:30 smithi125 conmon[24927]: 24T21:51:29.251541+0000 mon.a (mon.0) 436 : cluster [DBG] osdmap e51: 8 total, 7 up, 8 in 2021-03-24T21:51:30.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:30 smithi125 conmon[24927]: audit 2021-03 2021-03-24T21:51:30.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:30 smithi125 conmon[24927]: -24T21:51:29.251914+0000 mon.a (mon.0) 2021-03-24T21:51:30.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:30 smithi125 conmon[24927]: 437 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-03-24T21:51:30.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:30 smithi125 conmon[19936]: cluster 2021-03-24T21:51:28. 2021-03-24T21:51:30.657 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:30 smithi125 conmon[19936]: 628150+0000 mgr.y (mgr.14144) 176 : cluster [DBG] pgmap v172: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2021-03-24T21:51:30.657 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:30 smithi125 conmon[19936]: cluster 2021-03-24T21:51:28. 2021-03-24T21:51:30.657 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:30 smithi125 conmon[19936]: 965481+0000 osd.1 (osd.1) 5 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:51:30.657 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:30 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:51:30.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:30 smithi125 conmon[19936]: -24T21:51:28.965676+0000 2021-03-24T21:51:30.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:30 smithi125 conmon[19936]: osd.1 (osd.1) 6 : cluster [DBG] 2021-03-24T21:51:30.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:30 smithi125 conmon[19936]: purged_snaps scrub ok 2021-03-24T21:51:30.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:30 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:51:30.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:30 smithi125 conmon[19936]: -24T21:51:29.251541+0000 mon.a (mon.0) 436 2021-03-24T21:51:30.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:30 smithi125 conmon[19936]: : cluster [DBG] osdmap e51: 8 total, 7 up, 8 in 2021-03-24T21:51:30.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:30 smithi125 conmon[19936]: audit 2021 2021-03-24T21:51:30.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:30 smithi125 conmon[19936]: -03-24T21:51: 2021-03-24T21:51:30.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:30 smithi125 conmon[19936]: 29.251914+0000 mon.a (mon.0) 2021-03-24T21:51:30.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:30 smithi125 conmon[19936]: 437 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-03-24T21:51:31.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:31 smithi142 conmon[20125]: cluster 2021-03-24T21 2021-03-24T21:51:31.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:31 smithi142 conmon[20125]: :51:30.467238+0000 osd.4 (osd.4) 3 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:51:31.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:31 smithi142 conmon[20125]: cluster 2021-03-24T21:51:30.467440+0000 osd.4 (osd.4) 4 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:51:31.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:31 smithi125 conmon[24927]: cluster 2021-03-24T21:51:30.467238+0000 osd.4 2021-03-24T21:51:31.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:31 smithi125 conmon[24927]: (osd.4) 3 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:51:31.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:31 smithi125 conmon[24927]: cluster 2021-03-24T21:51:30.467440+0000 osd.4 (osd.4) 4 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:51:31.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:31 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:51:31.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:31 smithi125 conmon[19936]: 51:30.467238+0000 osd.4 (osd.4) 3 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:51:31.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:31 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:51:31.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:31 smithi125 conmon[19936]: -24T21:51:30.467440+0000 osd.4 (osd.4) 4 : cluster [DBG] 2021-03-24T21:51:31.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:31 smithi125 conmon[19936]: purged_snaps scrub ok 2021-03-24T21:51:31.733 INFO:teuthology.orchestra.run.smithi142.stdout:Created osd(s) 7 on host 'smithi142' 2021-03-24T21:51:32.492 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:32 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:51:32.492 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:32 smithi142 conmon[20125]: 51:30.628812+0000 mgr.y (mgr.14144) 177 : cluster [DBG] pgmap v174: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2021-03-24T21:51:32.492 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:32 smithi142 conmon[20125]: cluster 2021-03-24T21 2021-03-24T21:51:32.493 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:32 smithi142 conmon[20125]: :51:31.266516+0000 mon.a 2021-03-24T21:51:32.493 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:32 smithi142 conmon[20125]: (mon.0) 438 : cluster 2021-03-24T21:51:32.493 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:32 smithi142 conmon[20125]: [DBG] osdmap e52: 8 total, 7 up, 8 in 2021-03-24T21:51:32.493 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:32 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:51:32.493 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:32 smithi142 conmon[20125]: 03-24T21:51:31 2021-03-24T21:51:32.494 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:32 smithi142 conmon[20125]: .266918+0000 mon.a (mon.0 2021-03-24T21:51:32.494 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:32 smithi142 conmon[20125]: ) 439 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-03-24T21:51:32.494 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:32 smithi142 conmon[20125]: audit 2021-03-24T 2021-03-24T21:51:32.494 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:32 smithi142 conmon[20125]: 21:51:31.719981+0000 mon.a ( 2021-03-24T21:51:32.494 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:32 smithi142 conmon[20125]: mon.0) 440 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:51:32.495 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:32 smithi142 conmon[20125]: 2021-03-24T21:51:32.495 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:32 smithi142 conmon[20125]: audit 2021-03-24T21:51:32.495 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:32 smithi142 conmon[20125]: 2021-03-24T21:51:31.728172+0000 2021-03-24T21:51:32.495 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:32 smithi142 conmon[20125]: mon.a (mon.0) 441 : audit [INF] 2021-03-24T21:51:32.496 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:32 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:32.496 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:32 smithi142 conmon[20125]: audit 2021-03-24T21:51:32.496 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:32 smithi142 conmon[20125]: 2021-03-24T21:51:31.738052+0000 2021-03-24T21:51:32.496 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:32 smithi142 conmon[20125]: mon.a (mon.0) 442 : audit [INF] 2021-03-24T21:51:32.496 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:32 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:32.522 DEBUG:teuthology.orchestra.run.smithi142:osd.7> sudo journalctl -f -n 0 -u ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c@osd.7.service 2021-03-24T21:51:32.524 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi125 2021-03-24T21:51:32.525 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph orch apply rgw foo.a --placement '1;smithi125=foo.a' 2021-03-24T21:51:32.560 INFO:journalctl@ceph.osd.7.smithi142.stdout:-- Logs begin at Wed 2021-03-03 23:51:07 UTC. -- 2021-03-24T21:51:32.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:32 smithi125 conmon[24927]: cluster 2021-03-24T21:51:30. 2021-03-24T21:51:32.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:32 smithi125 conmon[24927]: 628812+0000 mgr.y (mgr.14144) 177 : cluster [DBG] pgmap v174: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2021-03-24T21:51:32.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:32 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:51:32.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:32 smithi125 conmon[24927]: 03-24T21:51:31.266516+0000 mon.a (mon.0) 438 : 2021-03-24T21:51:32.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:32 smithi125 conmon[24927]: cluster [DBG] osdmap e52: 8 total, 7 up, 8 in 2021-03-24T21:51:32.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:32 smithi125 conmon[24927]: audit 2021-03-24T21:51 2021-03-24T21:51:32.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:32 smithi125 conmon[24927]: :31.266918+0000 mon.a (mon.0) 439 2021-03-24T21:51:32.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:32 smithi125 conmon[24927]: : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-03-24T21:51:32.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:32 smithi125 conmon[24927]: audit 2021-03-24T21: 2021-03-24T21:51:32.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:32 smithi125 conmon[24927]: 51:31.719981+0000 mon.a (mon.0) 440 : audit 2021-03-24T21:51:32.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:32 smithi125 conmon[24927]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:51:32.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:32 smithi125 conmon[24927]: audit 2021-03-24T21:51:32.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:32 smithi125 conmon[24927]: 2021-03-24T21:51:31.728172+0000 mon.a (mon 2021-03-24T21:51:32.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:32 smithi125 conmon[24927]: .0) 441 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:32.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:32 smithi125 conmon[24927]: audit 2021 2021-03-24T21:51:32.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:32 smithi125 conmon[24927]: -03-24T21:51:31.738052+0000 2021-03-24T21:51:32.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:32 smithi125 conmon[24927]: mon.a (mon.0) 442 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:32.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:32 smithi125 conmon[24927]: 2021-03-24T21:51:32.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:32 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:51:32.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:32 smithi125 conmon[19936]: 03-24T21:51:30.628812+0000 mgr.y (mgr. 2021-03-24T21:51:32.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:32 smithi125 conmon[19936]: 14144) 177 : cluster [DBG] 2021-03-24T21:51:32.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:32 smithi125 conmon[19936]: pgmap v174: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2021-03-24T21:51:32.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:32 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:51:32.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:32 smithi125 conmon[19936]: 03-24T21:51:31.266516+0000 mon.a ( 2021-03-24T21:51:32.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:32 smithi125 conmon[19936]: mon.0) 438 : cluster [DBG] 2021-03-24T21:51:32.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:32 smithi125 conmon[19936]: osdmap e52: 8 total, 7 up, 8 in 2021-03-24T21:51:32.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:32 smithi125 conmon[19936]: audit 2021-03-24T21:51:32.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:32 smithi125 conmon[19936]: 2021-03- 2021-03-24T21:51:32.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:32 smithi125 conmon[19936]: 24T21:51:31.266918+0000 2021-03-24T21:51:32.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:32 smithi125 conmon[19936]: mon.a (mon.0) 439 : audit [DBG] 2021-03-24T21:51:32.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:32 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-03-24T21:51:32.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:32 smithi125 conmon[19936]: audit 2021-03- 2021-03-24T21:51:32.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:32 smithi125 conmon[19936]: 24T21:51:31.719981+0000 mon.a (mon.0 2021-03-24T21:51:32.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:32 smithi125 conmon[19936]: ) 440 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:51:32.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:32 smithi125 conmon[19936]: audit 2021-03-24T21 2021-03-24T21:51:32.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:32 smithi125 conmon[19936]: :51:31.728172+0000 mon.a (mon.0) 441 2021-03-24T21:51:32.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:32 smithi125 conmon[19936]: : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:32.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:32 smithi125 conmon[19936]: audit 2021-03-24T21:51:31. 2021-03-24T21:51:32.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:32 smithi125 conmon[19936]: 738052+0000 mon.a (mon.0) 442 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:34.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:34 smithi142 conmon[20125]: cluster 2021-03-24T21 2021-03-24T21:51:34.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:34 smithi142 conmon[20125]: :51:32.629410+0000 mgr.y (mgr.14144) 178 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2021-03-24T21:51:34.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:34 smithi125 conmon[19936]: cluster 2021-03-24T21:51:32.629410 2021-03-24T21:51:34.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:34 smithi125 conmon[19936]: +0000 mgr.y (mgr.14144) 178 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2021-03-24T21:51:34.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:34 smithi125 conmon[24927]: cluster 2021-03-24T21:51 2021-03-24T21:51:34.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:34 smithi125 conmon[24927]: :32.629410+0000 mgr.y (mgr.14144) 178 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2021-03-24T21:51:35.136 INFO:teuthology.orchestra.run.smithi142.stdout:Scheduled rgw.foo.a update... 2021-03-24T21:51:35.394 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 24 21:51:35 smithi142 conmon[33921]: debug 2021-03-24T21:51:35.377+0000 7fc7ddfb5080 -1 osd.7 0 log_to_monitors {default=true} 2021-03-24T21:51:35.991 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:35 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:51:35.992 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:35 smithi142 conmon[20125]: 51:34.629992+0000 mgr.y (mgr.14144) 179 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2021-03-24T21:51:35.992 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:35 smithi142 conmon[20125]: audit 2021-03-24T21:51:35.992 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:35 smithi142 conmon[20125]: 2021-03-24T21:51:34.764545+0000 mon.a (mon.0) 443 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:35.992 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:35 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:51:35.993 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:35 smithi142 conmon[20125]: 03-24T21:51:35.132797+0000 mgr.y (mgr 2021-03-24T21:51:35.993 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:35 smithi142 conmon[20125]: .14144) 180 : audit [DBG] from='client.24328 v1:172.21.15.142:0/1947903221' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi125=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:51:35.993 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:35 smithi142 conmon[20125]: cephadm 2021-03 2021-03-24T21:51:35.993 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:35 smithi142 conmon[20125]: -24T21:51:35.134298+0000 mgr.y ( 2021-03-24T21:51:35.994 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:35 smithi142 conmon[20125]: mgr.14144) 181 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi125=foo.a;count:1 2021-03-24T21:51:35.994 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:35 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:51:35.994 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:35 smithi142 conmon[20125]: 03-24T21:51:35.141097+0000 mon.a 2021-03-24T21:51:35.994 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:35 smithi142 conmon[20125]: (mon.0) 444 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:35.995 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:35 smithi142 conmon[20125]: audit 2021 2021-03-24T21:51:35.995 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:35 smithi142 conmon[20125]: -03-24T21:51:35.383340+0000 2021-03-24T21:51:35.995 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:35 smithi142 conmon[20125]: mon.b (mon.1) 21 : audit [INF] from='osd.7 v1:172.21.15.142:6812/3678129570' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2021-03-24T21:51:35.996 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:35 smithi142 conmon[20125]: 2021-03-24T21:51:35.997 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:35 smithi142 conmon[20125]: audit 2021-03-24T21:51:35. 2021-03-24T21:51:35.997 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:35 smithi142 conmon[20125]: 385409+0000 mon.a (mon.0) 445 : audit 2021-03-24T21:51:35.997 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:35 smithi142 conmon[20125]: [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2021-03-24T21:51:36.063 DEBUG:teuthology.orchestra.run.smithi125:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c@rgw.foo.a.service 2021-03-24T21:51:36.065 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi142 2021-03-24T21:51:36.066 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd pool create iscsi 3 3 replicated 2021-03-24T21:51:36.077 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:35 smithi125 conmon[24927]: cluster 2021-03-24T21:51: 2021-03-24T21:51:36.139 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:35 smithi125 conmon[24927]: 34.629992+0000 mgr.y (mgr.14144) 179 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2021-03-24T21:51:36.139 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:35 smithi125 conmon[24927]: audit 2021-03-24T21:51:34.764545+0000 2021-03-24T21:51:36.140 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:35 smithi125 conmon[24927]: mon.a (mon.0) 443 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:36.140 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:35 smithi125 conmon[24927]: audit 2021 2021-03-24T21:51:36.140 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:35 smithi125 conmon[24927]: -03-24T21:51:35.132797+0000 mgr.y ( 2021-03-24T21:51:36.140 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:35 smithi125 conmon[24927]: mgr.14144) 180 : audit [DBG] from='client.24328 v1:172.21.15.142:0/1947903221' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi125=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:51:36.141 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:35 smithi125 conmon[24927]: cephadm 2021 2021-03-24T21:51:36.141 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:35 smithi125 conmon[24927]: -03-24T21:51:35.134298+0000 2021-03-24T21:51:36.141 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:35 smithi125 conmon[24927]: mgr.y (mgr.14144) 181 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi125=foo.a;count:1 2021-03-24T21:51:36.141 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:35 smithi125 conmon[24927]: audit 2021-03-24T 2021-03-24T21:51:36.141 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:35 smithi125 conmon[24927]: 21:51:35.141097+0000 mon.a (mon.0) 444 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:36.142 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:35 smithi125 conmon[24927]: audit 2021-03-24T21:51:36.142 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:35 smithi125 conmon[24927]: 2021-03-24 2021-03-24T21:51:36.142 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:35 smithi125 conmon[24927]: T21:51:35.383340+0000 mon.b (mon.1 2021-03-24T21:51:36.142 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:35 smithi125 conmon[24927]: ) 21 : audit [INF] from='osd.7 v1:172.21.15.142:6812/3678129570' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2021-03-24T21:51:36.142 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:35 smithi125 conmon[24927]: audit 2021 2021-03-24T21:51:36.143 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:35 smithi125 conmon[24927]: -03-24T21:51:35.385409+0000 mon.a ( 2021-03-24T21:51:36.143 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:35 smithi125 conmon[24927]: mon.0) 445 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2021-03-24T21:51:36.143 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:35 smithi125 conmon[19936]: cluster 2021-03-24T21:51: 2021-03-24T21:51:36.144 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:35 smithi125 conmon[19936]: 34.629992+0000 mgr.y (mgr.14144) 179 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2021-03-24T21:51:36.144 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:35 smithi125 conmon[19936]: audit 2021-03-24T21:51:36.144 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:35 smithi125 conmon[19936]: 2021-03-24T21:51:34. 2021-03-24T21:51:36.144 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:35 smithi125 conmon[19936]: 764545+0000 mon.a (mon.0) 2021-03-24T21:51:36.145 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:35 smithi125 conmon[19936]: 443 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:36.145 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:35 smithi125 conmon[19936]: audit 2021 2021-03-24T21:51:36.145 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:35 smithi125 conmon[19936]: -03-24T21:51:35. 2021-03-24T21:51:36.145 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:35 smithi125 conmon[19936]: 132797+0000 mgr.y (mgr.14144) 2021-03-24T21:51:36.146 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:35 smithi125 conmon[19936]: 180 : audit [DBG] from='client.24328 v1:172.21.15.142:0/1947903221' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi125=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:51:36.146 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:35 smithi125 conmon[19936]: cephadm 2021 2021-03-24T21:51:36.146 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:35 smithi125 conmon[19936]: -03-24T21:51:35 2021-03-24T21:51:36.146 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:35 smithi125 conmon[19936]: .134298+0000 mgr.y (mgr. 2021-03-24T21:51:36.146 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:35 smithi125 conmon[19936]: 14144) 181 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi125=foo.a;count:1 2021-03-24T21:51:36.147 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:35 smithi125 conmon[19936]: 2021-03-24T21:51:36.147 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:35 smithi125 conmon[19936]: audit 2021-03-24T21:51:36.147 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:35 smithi125 conmon[19936]: 2021-03 2021-03-24T21:51:36.147 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:35 smithi125 conmon[19936]: -24T21:51:35. 2021-03-24T21:51:36.147 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:35 smithi125 conmon[19936]: 141097+0000 mon.a (mon.0 2021-03-24T21:51:36.148 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:35 smithi125 conmon[19936]: ) 444 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:36.148 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:35 smithi125 conmon[19936]: 2021-03-24T21:51:36.148 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:35 smithi125 conmon[19936]: audit 2021 2021-03-24T21:51:36.148 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:35 smithi125 conmon[19936]: -03-24T21:51: 2021-03-24T21:51:36.148 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:35 smithi125 conmon[19936]: 35.383340+0000 mon.b (mon 2021-03-24T21:51:36.149 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:35 smithi125 conmon[19936]: .1) 21 : audit [INF] 2021-03-24T21:51:36.149 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:35 smithi125 conmon[19936]: from='osd.7 v1:172.21.15.142:6812/3678129570' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2021-03-24T21:51:36.149 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:35 smithi125 conmon[19936]: audit 2021-03-24T21:51:36.149 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:35 smithi125 conmon[19936]: 2021-03-24T21:51 2021-03-24T21:51:36.150 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:35 smithi125 conmon[19936]: :35.385409+0000 mon.a ( 2021-03-24T21:51:36.150 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:35 smithi125 conmon[19936]: mon.0) 445 : audit 2021-03-24T21:51:36.150 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:35 smithi125 conmon[19936]: [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2021-03-24T21:51:36.150 INFO:journalctl@ceph.rgw.foo.a.smithi125.stdout:-- Logs begin at Wed 2021-03-03 23:51:07 UTC. -- 2021-03-24T21:51:37.358 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 24 21:51:37 smithi142 conmon[33921]: debug 2021-03-24T21:51:37.285+0000 7fc7d4526700 -1 osd.7 0 waiting for initial osdmap 2021-03-24T21:51:37.359 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:37 smithi142 conmon[20125]: audit 2021-03-24T21:51:36.277411+0000 mon.b (mon. 2021-03-24T21:51:37.359 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:37 smithi142 conmon[20125]: 1) 22 : audit [INF] from='osd.7 v1:172.21.15.142:6812/3678129570' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2021-03-24T21:51:37.359 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:37 smithi142 conmon[20125]: audit 2021-03-24T21:51:36.277757+0000 mon.a (mon.0) 446 2021-03-24T21:51:37.359 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:37 smithi142 conmon[20125]: : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2021-03-24T21:51:37.360 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:37 smithi142 conmon[20125]: cluster 2021-03-24T21:51:36.277955+0000 2021-03-24T21:51:37.360 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:37 smithi142 conmon[20125]: mon.a (mon.0) 447 : cluster [DBG] osdmap e53: 8 total, 7 up, 8 in 2021-03-24T21:51:37.360 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:37 smithi142 conmon[20125]: audit 2021-03-24T 2021-03-24T21:51:37.360 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:37 smithi142 conmon[20125]: 21:51:36.278348+0000 mon.a (mon.0 2021-03-24T21:51:37.361 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:37 smithi142 conmon[20125]: ) 448 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-03-24T21:51:37.361 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:37 smithi142 conmon[20125]: audit 2021 2021-03-24T21:51:37.361 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:37 smithi142 conmon[20125]: -03-24T21:51:36.280098+0000 2021-03-24T21:51:37.361 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:37 smithi142 conmon[20125]: mon.a (mon.0) 449 : audit [INF] 2021-03-24T21:51:37.361 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:37 smithi142 conmon[20125]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2021-03-24T21:51:37.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:37 smithi125 conmon[19936]: audit 2021-03-24T21:51:36.277411+0000 mon.b (mon.1) 22 : audit [INF] 2021-03-24T21:51:37.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:37 smithi125 conmon[19936]: from='osd.7 v1:172.21.15.142:6812/3678129570' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2021-03-24T21:51:37.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:37 smithi125 conmon[19936]: audit 2021-03-24T21:51:36.277757+0000 mon.a (mon.0) 446 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2021-03-24T21:51:37.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:37 smithi125 conmon[19936]: cluster 2021-03-24T21:51:36.277955+0000 mon.a (mon.0) 447 : cluster [DBG] osdmap e53: 8 total, 7 up, 8 in 2021-03-24T21:51:37.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:37 smithi125 conmon[19936]: audit 2021-03-24T21:51:36. 2021-03-24T21:51:37.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:37 smithi125 conmon[19936]: 278348+0000 mon.a (mon.0) 448 : audit 2021-03-24T21:51:37.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:37 smithi125 conmon[19936]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-03-24T21:51:37.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:37 smithi125 conmon[19936]: audit 2021-03-24 2021-03-24T21:51:37.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:37 smithi125 conmon[19936]: T21:51:36.280098+0000 mon.a ( 2021-03-24T21:51:37.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:37 smithi125 conmon[19936]: mon.0) 449 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2021-03-24T21:51:37.657 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:37 smithi125 conmon[19936]: 2021-03-24T21:51:37.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:37 smithi125 conmon[24927]: audit 2021-03-24T21:51:36.277411+0000 2021-03-24T21:51:37.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:37 smithi125 conmon[24927]: mon.b (mon.1) 22 : audit [INF] from='osd.7 v1:172.21.15.142:6812/3678129570' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2021-03-24T21:51:37.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:37 smithi125 conmon[24927]: audit 2021-03-24T21:51:36 2021-03-24T21:51:37.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:37 smithi125 conmon[24927]: .277757+0000 mon.a (mon.0) 446 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2021-03-24T21:51:37.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:37 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:51:37.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:37 smithi125 conmon[24927]: 21:51:36.277955+0000 mon.a (mon.0) 447 2021-03-24T21:51:37.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:37 smithi125 conmon[24927]: : cluster [DBG] osdmap e53: 8 total, 7 up, 8 in 2021-03-24T21:51:37.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:37 smithi125 conmon[24927]: audit 2021-03-24T 2021-03-24T21:51:37.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:37 smithi125 conmon[24927]: 21:51:36.278348+0000 mon.a (mon.0) 448 : 2021-03-24T21:51:37.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:37 smithi125 conmon[24927]: audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-03-24T21:51:37.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:37 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T21:51:37.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:37 smithi125 conmon[24927]: 24T21:51:36.280098+0000 mon.a (mon.0 2021-03-24T21:51:37.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:37 smithi125 conmon[24927]: ) 449 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2021-03-24T21:51:38.350 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:38 smithi142 conmon[20125]: cluster 2021-03-24T21:51:36.630524+0000 mgr.y (mgr.14144) 182 : cluster 2021-03-24T21:51:38.351 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:38 smithi142 conmon[20125]: [DBG] pgmap v179: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2021-03-24T21:51:38.351 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:38 smithi142 conmon[20125]: audit 2021-03-24T21:51:37.281456+0000 mon.a (mon.0) 450 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2021-03-24T21:51:38.351 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:38 smithi142 conmon[20125]: cluster 2021-03-24T21:51:37.281645+0000 mon.a (mon.0) 451 : cluster [DBG] osdmap e54: 8 total, 7 up, 8 in 2021-03-24T21:51:38.352 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:38 smithi142 conmon[20125]: audit 2021-03-24T21:51:37.282435+0000 mon.a (mon.0) 452 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-03-24T21:51:38.352 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:38 smithi142 conmon[20125]: audit 2021-03-24T 2021-03-24T21:51:38.353 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:38 smithi142 conmon[20125]: 21:51:37.289761+0000 mon.a (mon.0) 453 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-03-24T21:51:38.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:38 smithi125 conmon[19936]: cluster 2021-03-24T21:51:36.630524+0000 mgr.y (mgr.14144 2021-03-24T21:51:38.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:38 smithi125 conmon[19936]: ) 182 : cluster [DBG] pgmap v179: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2021-03-24T21:51:38.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:38 smithi125 conmon[19936]: audit 2021-03-24T21:51:37.281456+0000 mon.a (mon.0) 450 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2021-03-24T21:51:38.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:38 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:51:38.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:38 smithi125 conmon[19936]: 03-24T21:51:37.281645+0000 mon.a (mon.0) 451 : cluster [DBG] osdmap e54: 8 total, 7 up, 8 in 2021-03-24T21:51:38.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:38 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:51:38.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:38 smithi125 conmon[19936]: 03-24T21:51:37.282435+0000 mon.a (mon.0) 452 : audit [DBG] 2021-03-24T21:51:38.657 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:38 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-03-24T21:51:38.657 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:38 smithi125 conmon[19936]: audit 2021-03-24T21:51:38.657 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:38 smithi125 conmon[19936]: 2021-03-24T21:51:37.289761+0000 mon.a (mon.0) 453 : audit [DBG] 2021-03-24T21:51:38.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:38 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-03-24T21:51:38.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:38 smithi125 conmon[24927]: cluster 2021-03-24T21:51:36.630524+0000 mgr.y (mgr 2021-03-24T21:51:38.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:38 smithi125 conmon[24927]: .14144) 182 : cluster [DBG] pgmap v179: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2021-03-24T21:51:38.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:38 smithi125 conmon[24927]: audit 2021-03-24T21:51:37 2021-03-24T21:51:38.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:38 smithi125 conmon[24927]: .281456+0000 mon.a (mon.0) 450 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2021-03-24T21:51:38.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:38 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:51:38.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:38 smithi125 conmon[24927]: 24T21:51:37.281645+0000 mon.a (mon.0) 2021-03-24T21:51:38.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:38 smithi125 conmon[24927]: 451 : cluster [DBG] osdmap e54: 8 total, 7 up, 8 in 2021-03-24T21:51:38.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:38 smithi125 conmon[24927]: audit 2021-03-24T21:51:38.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:38 smithi125 conmon[24927]: 2021-03-24T21:51:37.282435+0000 mon.a (mon 2021-03-24T21:51:38.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:38 smithi125 conmon[24927]: .0) 452 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-03-24T21:51:38.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:38 smithi125 conmon[24927]: audit 2021-03-24T21:51:38.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:38 smithi125 conmon[24927]: 2021-03-24 2021-03-24T21:51:38.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:38 smithi125 conmon[24927]: T21:51:37.289761+0000 mon.a (mon.0) 2021-03-24T21:51:38.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:38 smithi125 conmon[24927]: 453 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-03-24T21:51:39.294 INFO:teuthology.orchestra.run.smithi142.stderr:pool 'iscsi' created 2021-03-24T21:51:39.566 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:39 smithi142 conmon[20125]: cluster 2021-03-24T21:51:36.348332+0000 osd.7 (osd.7 2021-03-24T21:51:39.567 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:39 smithi142 conmon[20125]: ) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:51:39.567 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:39 smithi142 conmon[20125]: cluster 2021-03-24T21:51:36.348512+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:51:39.567 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:39 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:51:39.567 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:39 smithi142 conmon[20125]: 03-24T21:51:38.292521+0000 mon.a (mon.0) 454 : cluster [INF] osd.7 v1:172.21.15.142:6812/3678129570 boot 2021-03-24T21:51:39.568 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:39 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:51:39.568 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:39 smithi142 conmon[20125]: 21:51:38.292593+0000 mon.a (mon.0) 455 : cluster [DBG] osdmap e55: 8 total, 8 up, 8 in 2021-03-24T21:51:39.568 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:39 smithi142 conmon[20125]: audit 2021-03-24T 2021-03-24T21:51:39.568 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:39 smithi142 conmon[20125]: 21:51:38.293609+0000 mon.a (mon.0) 456 : 2021-03-24T21:51:39.569 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:39 smithi142 conmon[20125]: audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-03-24T21:51:39.569 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:39 smithi142 conmon[20125]: audit 2021-03-24T21:51:38.750254+0000 2021-03-24T21:51:39.569 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:39 smithi142 conmon[20125]: mon.b (mon.1) 23 : audit [INF] from='client.? v1:172.21.15.142:0/609110312' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2021-03-24T21:51:39.569 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:39 smithi142 conmon[20125]: audit 2021-03-24T21:51:38.752369+0000 mon.a (mon 2021-03-24T21:51:39.569 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:39 smithi142 conmon[20125]: .0) 457 : audit [INF] from='client.24337 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2021-03-24T21:51:39.570 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:39 smithi142 conmon[20125]: audit 2021-03-24T21:51:39.570 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:39 smithi142 conmon[20125]: 2021-03-24T21:51:39.293336+0000 mon.a (mon.0) 458 : audit 2021-03-24T21:51:39.570 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:39 smithi142 conmon[20125]: [INF] from='client.24337 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2021-03-24T21:51:39.570 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:39 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:51:39.571 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:39 smithi142 conmon[20125]: 03-24T21:51:39.293469+0000 mon.a (mon.0) 459 : cluster [DBG] osdmap e56: 8 total, 8 up, 8 in 2021-03-24T21:51:39.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:51:39.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[24927]: :51:36.348332+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:51:39.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:51:39.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[24927]: 03-24T21:51:36.348512+0000 2021-03-24T21:51:39.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[24927]: osd.7 (osd.7 2021-03-24T21:51:39.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[24927]: ) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:51:39.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:51:39.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[24927]: -03-24T21:51:38.292521+0000 mon.a 2021-03-24T21:51:39.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[24927]: (mon.0) 454 : cluster [INF] 2021-03-24T21:51:39.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[24927]: osd.7 v1:172.21.15.142:6812/3678129570 boot 2021-03-24T21:51:39.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:51:39.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[24927]: T21:51:38.292593+0000 2021-03-24T21:51:39.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[24927]: mon.a (mon.0) 455 : cluster 2021-03-24T21:51:39.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[24927]: [DBG] osdmap e55: 8 total, 8 up, 8 in 2021-03-24T21:51:39.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[24927]: audit 2021-03-24T21:51: 2021-03-24T21:51:39.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[24927]: 38.293609+0000 mon.a (mon.0) 456 : 2021-03-24T21:51:39.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[24927]: audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-03-24T21:51:39.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[24927]: audit 2021- 2021-03-24T21:51:39.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[24927]: 03-24T21:51:38.750254 2021-03-24T21:51:39.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[24927]: +0000 mon.b (mon.1) 23 : audit [INF] 2021-03-24T21:51:39.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[24927]: from='client.? v1:172.21.15.142:0/609110312' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2021-03-24T21:51:39.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[24927]: audit 2021-03-24T21:51:38 2021-03-24T21:51:39.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[24927]: .752369+0000 mon.a (mon.0) 457 2021-03-24T21:51:39.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[24927]: : audit [INF] from='client.24337 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2021-03-24T21:51:39.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[24927]: audit 2021- 2021-03-24T21:51:39.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[24927]: 03-24T21:51:39.293336+0000 2021-03-24T21:51:39.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[24927]: mon.a (mon.0 2021-03-24T21:51:39.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[24927]: ) 458 : audit [INF] from='client.24337 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2021-03-24T21:51:39.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[24927]: 2021-03-24T21:51:39.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:51:39.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[24927]: T21:51:39.293469+0000 mon.a (mon. 2021-03-24T21:51:39.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[24927]: 0) 459 : cluster 2021-03-24T21:51:39.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[24927]: [DBG] osdmap e56: 8 total, 8 up, 8 in 2021-03-24T21:51:39.670 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:51:39.670 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[19936]: :51:36.348332+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:51:39.670 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:51:39.671 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[19936]: 24T21:51:36 2021-03-24T21:51:39.671 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[19936]: .348512+0000 osd.7 ( 2021-03-24T21:51:39.672 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[19936]: osd.7) 2 : cluster 2021-03-24T21:51:39.672 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[19936]: [DBG] purged_snaps scrub ok 2021-03-24T21:51:39.672 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:51:39.673 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[19936]: -03-24T21:51 2021-03-24T21:51:39.673 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[19936]: :38.292521+0000 2021-03-24T21:51:39.673 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[19936]: mon.a (mon.0) 2021-03-24T21:51:39.674 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[19936]: 454 : cluster [INF] osd.7 v1:172.21.15.142:6812/3678129570 boot 2021-03-24T21:51:39.674 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[19936]: 2021-03-24T21:51:39.675 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[19936]: cluster 2021-03-24T21:51:39.675 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[19936]: 2021-03 2021-03-24T21:51:39.675 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[19936]: -24T21:51: 2021-03-24T21:51:39.676 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[19936]: 38.292593+0000 mon.a 2021-03-24T21:51:39.676 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[19936]: (mon.0) 455 2021-03-24T21:51:39.676 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[19936]: : cluster [DBG] osdmap e55: 8 total, 8 up, 8 in 2021-03-24T21:51:39.677 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[19936]: audit 2021 2021-03-24T21:51:39.677 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[19936]: -03-24T21:51 2021-03-24T21:51:39.677 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[19936]: :38.293609+0000 2021-03-24T21:51:39.678 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[19936]: mon.a (mon.0) 2021-03-24T21:51:39.678 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[19936]: 456 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-03-24T21:51:39.679 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[19936]: 2021-03-24T21:51:39.679 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[19936]: audit 2021 2021-03-24T21:51:39.680 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[19936]: -03-24T21: 2021-03-24T21:51:39.680 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[19936]: 51:38.750254+0000 2021-03-24T21:51:39.680 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[19936]: mon.b (mon.1 2021-03-24T21:51:39.681 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[19936]: ) 23 : audit [INF] 2021-03-24T21:51:39.681 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[19936]: from='client.? v1:172.21.15.142:0/609110312' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2021-03-24T21:51:39.682 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[19936]: audit 2021-03-24T21:51:39.682 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[19936]: 2021-03-24 2021-03-24T21:51:39.682 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[19936]: T21:51:38. 2021-03-24T21:51:39.683 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[19936]: 752369+0000 mon.a (mon 2021-03-24T21:51:39.683 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[19936]: .0) 457 : audit 2021-03-24T21:51:39.683 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[19936]: [INF] from='client.24337 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2021-03-24T21:51:39.683 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[19936]: audit 2021 2021-03-24T21:51:39.684 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[19936]: -03-24T21: 2021-03-24T21:51:39.684 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[19936]: 51:39.293336 2021-03-24T21:51:39.684 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[19936]: +0000 mon.a (mon 2021-03-24T21:51:39.685 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[19936]: .0) 458 : 2021-03-24T21:51:39.685 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[19936]: audit [INF] from='client.24337 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2021-03-24T21:51:39.686 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[19936]: 2021-03-24T21:51:39.686 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[19936]: cluster 2021-03-24T21:51:39.686 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[19936]: 2021-03-24T21: 2021-03-24T21:51:39.687 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[19936]: 51:39.293469+0000 2021-03-24T21:51:39.687 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[19936]: mon.a (mon.0 2021-03-24T21:51:39.688 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[19936]: ) 459 : cluster 2021-03-24T21:51:39.688 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:39 smithi125 conmon[19936]: [DBG] osdmap e56: 8 total, 8 up, 8 in 2021-03-24T21:51:40.502 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd pool application enable iscsi rbd 2021-03-24T21:51:40.579 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:40 smithi142 conmon[20125]: cluster 2021-03-24T21:51:38.631153+0000 mgr.y (mgr 2021-03-24T21:51:40.579 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:40 smithi142 conmon[20125]: .14144) 183 : cluster [DBG] pgmap v182: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2021-03-24T21:51:40.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:40 smithi125 conmon[24927]: cluster 2021-03-24T21:51: 2021-03-24T21:51:40.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:40 smithi125 conmon[24927]: 38.631153+0000 mgr.y (mgr.14144) 183 : cluster [DBG] pgmap v182: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2021-03-24T21:51:40.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:40 smithi125 conmon[19936]: cluster 2021-03-24T21:51:40.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:40 smithi125 conmon[19936]: 2021-03-24T21:51:38.631153+0000 mgr.y 2021-03-24T21:51:40.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:40 smithi125 conmon[19936]: (mgr.14144) 183 : cluster [DBG] pgmap v182: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2021-03-24T21:51:41.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:41 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:51:41.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:41 smithi142 conmon[20125]: 51:40.318904+0000 mon.a (mon.0) 460 : cluster [DBG] osdmap e57: 8 total, 8 up, 8 in 2021-03-24T21:51:41.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:41 smithi142 conmon[20125]: audit 2021 2021-03-24T21:51:41.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:41 smithi142 conmon[20125]: -03-24T21:51:40.717119+0000 mon.a 2021-03-24T21:51:41.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:41 smithi142 conmon[20125]: (mon.0) 461 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:41.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:41 smithi142 conmon[20125]: 2021-03-24T21:51:41.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:41 smithi142 conmon[20125]: audit 2021-03-24T21:51:41.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:41 smithi142 conmon[20125]: 2021-03-24T21:51:41.027056 2021-03-24T21:51:41.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:41 smithi142 conmon[20125]: +0000 mon.a (mon.0) 462 : audit [INF] 2021-03-24T21:51:41.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:41 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:41.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:41 smithi142 conmon[20125]: audit 2021-03- 2021-03-24T21:51:41.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:41 smithi142 conmon[20125]: 24T21:51:41.036144+0000 mon.a ( 2021-03-24T21:51:41.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:41 smithi142 conmon[20125]: mon.0) 463 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:41.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:41 smithi142 conmon[20125]: audit 2021-03 2021-03-24T21:51:41.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:41 smithi142 conmon[20125]: -24T21:51:41.037030+0000 2021-03-24T21:51:41.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:41 smithi142 conmon[20125]: mon.a (mon.0) 464 : audit [INF] 2021-03-24T21:51:41.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:41 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2021-03-24T21:51:41.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:41 smithi142 conmon[20125]: audit 2021-03-24T21:51:41.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:41 smithi142 conmon[20125]: 2021-03-24T21:51:41.041658+0000 mon.a (mon 2021-03-24T21:51:41.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:41 smithi142 conmon[20125]: .0) 465 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2021-03-24T21:51:41.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:41 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:51:41.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:41 smithi142 conmon[20125]: 03-24T21:51:41.048680+0000 2021-03-24T21:51:41.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:41 smithi142 conmon[20125]: mon.a (mon.0) 466 : audit [INF] 2021-03-24T21:51:41.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:41 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:41.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:41 smithi142 conmon[20125]: audit 2021-03-24T21:51:41.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:41 smithi142 conmon[20125]: 2021-03-24T21:51:41.049890+0000 2021-03-24T21:51:41.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:41 smithi142 conmon[20125]: mon.a (mon.0) 467 : audit [DBG] 2021-03-24T21:51:41.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:41 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:51:41.621 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:41 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:51:41.621 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:41 smithi125 conmon[24927]: 51:40.318904+0000 mon.a (mon.0) 460 : cluster [DBG] osdmap e57: 8 total, 8 up, 8 in 2021-03-24T21:51:41.621 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:41 smithi125 conmon[24927]: audit 2021-03-24T21:51:40.717119+0000 mon.a (mon.0) 461 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:41.621 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:41 smithi125 conmon[24927]: audit 2021-03-24T21:51:41.027056+0000 mon.a (mon.0) 462 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:41.622 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:41 smithi125 conmon[24927]: audit 2021-03-24T21:51:41.036144+0000 mon.a (mon.0) 463 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:41.622 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:41 smithi125 conmon[24927]: audit 2021-03-24T21:51:41.037030+0000 mon.a (mon.0) 464 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2021-03-24T21:51:41.622 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:41 smithi125 conmon[24927]: audit 2021-03-24T21:51:41.041658+0000 mon.a (mon.0) 465 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2021-03-24T21:51:41.622 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:41 smithi125 conmon[24927]: audit 2021-03-24T21:51:41.622 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:41 smithi125 conmon[24927]: 2021-03-24T21:51:41.048680+0000 mon.a (mon.0) 466 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:41.623 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:41 smithi125 conmon[24927]: audit 2021-03-24T21:51:41.049890+0000 mon.a (mon. 2021-03-24T21:51:41.623 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:41 smithi125 conmon[24927]: 0) 467 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:51:41.623 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:41 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:51:41.624 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:41 smithi125 conmon[19936]: 21:51:40.318904+0000 mon.a (mon.0) 460 : cluster [DBG] osdmap e57: 8 total, 8 up, 8 in 2021-03-24T21:51:41.624 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:41 smithi125 conmon[19936]: audit 2021-03-24T21:51:41.624 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:41 smithi125 conmon[19936]: 2021-03-24T21:51 2021-03-24T21:51:41.624 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:41 smithi125 conmon[19936]: :40.717119+0000 mon.a (mon. 2021-03-24T21:51:41.624 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:41 smithi125 conmon[19936]: 0) 461 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:41.625 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:41 smithi125 conmon[19936]: 2021-03-24T21:51:41.625 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:41 smithi125 conmon[19936]: audit 2021-03- 2021-03-24T21:51:41.625 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:41 smithi125 conmon[19936]: 24T21:51:41.027056+0000 mon.a 2021-03-24T21:51:41.625 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:41 smithi125 conmon[19936]: (mon.0) 462 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:41.625 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:41 smithi125 conmon[19936]: audit 2021-03-24T21:51:41.626 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:41 smithi125 conmon[19936]: 2021- 2021-03-24T21:51:41.626 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:41 smithi125 conmon[19936]: 03-24T21:51:41. 2021-03-24T21:51:41.626 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:41 smithi125 conmon[19936]: 036144+0000 mon.a (mon.0) 2021-03-24T21:51:41.626 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:41 smithi125 conmon[19936]: 463 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:41.627 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:41 smithi125 conmon[19936]: audit 2021-03-24 2021-03-24T21:51:41.627 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:41 smithi125 conmon[19936]: T21:51:41.037030+0000 mon.a (mon.0) 2021-03-24T21:51:41.627 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:41 smithi125 conmon[19936]: 464 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2021-03-24T21:51:41.627 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:41 smithi125 conmon[19936]: 2021-03-24T21:51:41.627 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:41 smithi125 conmon[19936]: audit 2021-03-24T21:51:41.628 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:41 smithi125 conmon[19936]: 2021-03-24 2021-03-24T21:51:41.628 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:41 smithi125 conmon[19936]: T21:51:41.041658+0000 mon.a (mon 2021-03-24T21:51:41.628 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:41 smithi125 conmon[19936]: .0) 465 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2021-03-24T21:51:41.628 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:41 smithi125 conmon[19936]: 2021-03-24T21:51:41.628 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:41 smithi125 conmon[19936]: audit 2021 2021-03-24T21:51:41.629 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:41 smithi125 conmon[19936]: -03-24T21:51: 2021-03-24T21:51:41.629 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:41 smithi125 conmon[19936]: 41.048680+0000 mon.a (mon 2021-03-24T21:51:41.629 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:41 smithi125 conmon[19936]: .0) 466 : audit [INF] 2021-03-24T21:51:41.629 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:41 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:41.629 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:41 smithi125 conmon[19936]: audit 2021-03-24T21:51:41.630 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:41 smithi125 conmon[19936]: 2021-03-24T21 2021-03-24T21:51:41.630 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:41 smithi125 conmon[19936]: :51:41.049890+0000 2021-03-24T21:51:41.630 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:41 smithi125 conmon[19936]: mon.a (mon.0) 467 : audit 2021-03-24T21:51:41.630 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:41 smithi125 conmon[19936]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:51:42.326 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:42 smithi125 conmon[19936]: cluster 2021-03-24T21:51:40 2021-03-24T21:51:42.608 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:42 smithi125 conmon[19936]: .631777+0000 mgr.y (mgr.14144) 184 : cluster [DBG] pgmap v185: 4 pgs: 3 unknown, 1 remapped+peering; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-03-24T21:51:42.609 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:42 smithi125 conmon[19936]: cephadm 2021-03-24T21:51:40.707373+0000 mgr.y (mgr. 2021-03-24T21:51:42.609 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:42 smithi125 conmon[19936]: 14144) 185 : cephadm [INF] Detected new or changed devices on smithi142 2021-03-24T21:51:42.609 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:42 smithi125 conmon[19936]: cephadm 2021-03- 2021-03-24T21:51:42.610 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:42 smithi125 conmon[19936]: 24T21:51:41.029988+0000 mgr.y (mgr.14144 2021-03-24T21:51:42.610 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:42 smithi125 conmon[19936]: ) 186 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi125=foo.a;count:1 2021-03-24T21:51:42.610 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:42 smithi125 conmon[19936]: cephadm 2021-03 2021-03-24T21:51:42.610 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:42 smithi125 conmon[19936]: -24T21:51:41.051226+0000 mgr.y 2021-03-24T21:51:42.610 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:42 smithi125 conmon[19936]: (mgr. 2021-03-24T21:51:42.611 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:42 smithi125 conmon[19936]: 14144) 187 : cephadm [INF] Deploying daemon rgw.foo.a on smithi125 2021-03-24T21:51:42.611 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:42 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:51:42.611 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:42 smithi125 conmon[19936]: -03-24T21:51:41.323903+0000 2021-03-24T21:51:42.611 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:42 smithi125 conmon[19936]: mon.a (mon.0) 468 : cluster 2021-03-24T21:51:42.612 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:42 smithi125 conmon[19936]: [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2021-03-24T21:51:42.612 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:42 smithi125 conmon[19936]: cluster 2021-03-24T21:51:41.335803 2021-03-24T21:51:42.612 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:42 smithi125 conmon[19936]: +0000 mon.a (mon.0) 469 : cluster [DBG] osdmap e58: 8 total, 8 up, 8 in 2021-03-24T21:51:42.613 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:42 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:51:42.613 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:42 smithi125 conmon[24927]: 51:40.631777+0000 mgr.y (mgr.14144) 184 : cluster [DBG] pgmap v185: 4 pgs: 3 unknown, 1 remapped+peering; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-03-24T21:51:42.613 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:42 smithi125 conmon[24927]: cephadm 2021-03-24T21 2021-03-24T21:51:42.613 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:42 smithi125 conmon[24927]: :51:40.707373+0000 mgr.y (mgr.14144) 2021-03-24T21:51:42.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:42 smithi125 conmon[24927]: 185 : cephadm [INF] Detected new or changed devices on smithi142 2021-03-24T21:51:42.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:42 smithi125 conmon[24927]: cephadm 2021-03-24T21:51:42.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:42 smithi125 conmon[24927]: 2021-03-24 2021-03-24T21:51:42.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:42 smithi125 conmon[24927]: T21:51:41.029988+0000 mgr.y (mgr 2021-03-24T21:51:42.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:42 smithi125 conmon[24927]: .14144) 186 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi125=foo.a;count:1 2021-03-24T21:51:42.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:42 smithi125 conmon[24927]: cephadm 2021-03-24T21:51:42.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:42 smithi125 conmon[24927]: 2021-03-24T21:51:41.051226+0000 2021-03-24T21:51:42.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:42 smithi125 conmon[24927]: mgr.y (mgr.14144) 187 : cephadm [INF] Deploying daemon rgw.foo.a on smithi125 2021-03-24T21:51:42.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:42 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:51:42.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:42 smithi125 conmon[24927]: :51:41.323903+0000 mon.a (mon.0) 468 : cluster 2021-03-24T21:51:42.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:42 smithi125 conmon[24927]: [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2021-03-24T21:51:42.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:42 smithi125 conmon[24927]: cluster 2021-03-24T21:51:42.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:42 smithi125 conmon[24927]: 2021-03-24T21:51:41.335803+0000 mon.a (mon.0 2021-03-24T21:51:42.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:42 smithi125 conmon[24927]: ) 469 : cluster [DBG] osdmap e58: 8 total, 8 up, 8 in 2021-03-24T21:51:42.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:42 smithi142 conmon[20125]: cluster 2021-03-24T21:51:40.631777+0000 mgr.y (mgr.14144) 184 : cluster [DBG] pgmap v185: 4 pgs: 3 unknown, 1 remapped+peering; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-03-24T21:51:42.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:42 smithi142 conmon[20125]: cephadm 2021-03-24T21:51:40.707373+0000 mgr.y (mgr.14144) 185 : cephadm [INF] 2021-03-24T21:51:42.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:42 smithi142 conmon[20125]: Detected new or changed devices on smithi142 2021-03-24T21:51:42.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:42 smithi142 conmon[20125]: cephadm 2021-03-24T 2021-03-24T21:51:42.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:42 smithi142 conmon[20125]: 21:51:41.029988+0000 mgr.y (mgr. 2021-03-24T21:51:42.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:42 smithi142 conmon[20125]: 14144) 186 : cephadm [INF] 2021-03-24T21:51:42.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:42 smithi142 conmon[20125]: Saving service rgw.foo.a spec with placement smithi125=foo.a;count:1 2021-03-24T21:51:42.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:42 smithi142 conmon[20125]: cephadm 2021-03-24T21:51:42.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:42 smithi142 conmon[20125]: 2021-03-24T 2021-03-24T21:51:42.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:42 smithi142 conmon[20125]: 21:51:41. 2021-03-24T21:51:42.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:42 smithi142 conmon[20125]: 051226+0000 mgr.y (mgr 2021-03-24T21:51:42.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:42 smithi142 conmon[20125]: .14144) 187 : cephadm 2021-03-24T21:51:42.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:42 smithi142 conmon[20125]: [INF] Deploying daemon rgw.foo.a on smithi125 2021-03-24T21:51:42.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:42 smithi142 conmon[20125]: cluster 2021-03-24T21:51:42.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:42 smithi142 conmon[20125]: 2021-03-24T21 2021-03-24T21:51:42.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:42 smithi142 conmon[20125]: :51:41.323903+0000 2021-03-24T21:51:42.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:42 smithi142 conmon[20125]: mon.a (mon.0 2021-03-24T21:51:42.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:42 smithi142 conmon[20125]: ) 468 : cluster [WRN] 2021-03-24T21:51:42.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:42 smithi142 conmon[20125]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2021-03-24T21:51:42.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:42 smithi142 conmon[20125]: cluster 2021-03-24T21:51:42.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:42 smithi142 conmon[20125]: 2021-03-24T21:51 2021-03-24T21:51:42.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:42 smithi142 conmon[20125]: :41.335803+0000 2021-03-24T21:51:42.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:42 smithi142 conmon[20125]: mon.a (mon.0) 2021-03-24T21:51:42.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:42 smithi142 conmon[20125]: 469 : cluster [DBG] 2021-03-24T21:51:42.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:42 smithi142 conmon[20125]: osdmap e58: 8 total, 8 up, 8 in 2021-03-24T21:51:43.154 INFO:journalctl@ceph.rgw.foo.a.smithi125.stdout:Mar 24 21:51:43 smithi125 systemd[1]: Starting Ceph rgw.foo.a for 3cfa339a-8cea-11eb-90b2-001a4aab830c... 2021-03-24T21:51:43.344 INFO:teuthology.orchestra.run.smithi142.stderr:enabled application 'rbd' on pool 'iscsi' 2021-03-24T21:51:43.585 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:43 smithi125 conmon[19936]: audit 2021-03-24 2021-03-24T21:51:43.585 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:43 smithi125 conmon[19936]: T21:51:42.733372+0000 mon.b (mon.1) 24 : audit [INF] from='client.? v1:172.21.15.142:0/2848231465' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]: dispatch 2021-03-24T21:51:43.585 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:43 smithi125 conmon[19936]: audit 2021-03-24T21:51:42.735370+0000 mon.a (mon.0) 470 : audit [INF] from='client.24343 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]: dispatch 2021-03-24T21:51:43.586 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:43 smithi125 conmon[24927]: audit 2021-03-24T21:51:42.733372+0000 mon.b (mon.1) 24 : audit [INF] from='client.? v1:172.21.15.142:0/2848231465' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]: dispatch 2021-03-24T21:51:43.586 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:43 smithi125 conmon[24927]: audit 2021-03-24T21:51:42.735370+0000 mon.a (mon.0) 470 : audit [INF] from='client.24343 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]: dispatch 2021-03-24T21:51:43.587 INFO:journalctl@ceph.rgw.foo.a.smithi125.stdout:Mar 24 21:51:43 smithi125 podman[39554]: 2021-03-24 21:51:43.504138734 +0000 UTC m=+0.197747891 container create 61e4a6b08cd7fc827795d0d06d44bf1e79a47788556b17d8f9f9b2739f8e2461 (image=quay.ceph.io/ceph-ci/ceph@sha256:e9db06906b8d0a24f3ef6835ac5aa4b8eea2950a2560bad78f83d4f101eca0db, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-rgw.foo.a, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20201204, GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06, GIT_CLEAN=True, io.buildah.version=1.18.0, CEPH_POINT_RELEASE=, maintainer=Dimitri Savineau , org.label-schema.vendor=CentOS) 2021-03-24T21:51:43.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:43 smithi142 conmon[20125]: audit 2021-03-24T21:51: 2021-03-24T21:51:43.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:43 smithi142 conmon[20125]: 42.733372+0000 mon.b (mon.1) 24 : audit [INF] from='client.? v1:172.21.15.142:0/2848231465' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]: dispatch 2021-03-24T21:51:43.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:43 smithi142 conmon[20125]: audit 2021 2021-03-24T21:51:43.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:43 smithi142 conmon[20125]: -03-24T21:51:42.735370+0000 mon.a 2021-03-24T21:51:43.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:43 smithi142 conmon[20125]: (mon.0) 470 : audit 2021-03-24T21:51:43.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:43 smithi142 conmon[20125]: [INF] from='client.24343 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]: dispatch 2021-03-24T21:51:43.905 INFO:journalctl@ceph.rgw.foo.a.smithi125.stdout:Mar 24 21:51:43 smithi125 podman[39554]: 2021-03-24 21:51:43.662463717 +0000 UTC m=+0.356072907 container init 61e4a6b08cd7fc827795d0d06d44bf1e79a47788556b17d8f9f9b2739f8e2461 (image=quay.ceph.io/ceph-ci/ceph@sha256:e9db06906b8d0a24f3ef6835ac5aa4b8eea2950a2560bad78f83d4f101eca0db, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-rgw.foo.a, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d, CEPH_POINT_RELEASE=, org.label-schema.build-date=20201204, io.buildah.version=1.18.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Base Image, maintainer=Dimitri Savineau , GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2021-03-24T21:51:43.906 INFO:journalctl@ceph.rgw.foo.a.smithi125.stdout:Mar 24 21:51:43 smithi125 podman[39554]: 2021-03-24 21:51:43.704032111 +0000 UTC m=+0.397641219 container start 61e4a6b08cd7fc827795d0d06d44bf1e79a47788556b17d8f9f9b2739f8e2461 (image=quay.ceph.io/ceph-ci/ceph@sha256:e9db06906b8d0a24f3ef6835ac5aa4b8eea2950a2560bad78f83d4f101eca0db, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-rgw.foo.a, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20201204, GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.18.0, maintainer=Dimitri Savineau , org.label-schema.vendor=CentOS, RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d, org.label-schema.name=CentOS Base Image, ceph=True, GIT_BRANCH=HEAD) 2021-03-24T21:51:43.906 INFO:journalctl@ceph.rgw.foo.a.smithi125.stdout:Mar 24 21:51:43 smithi125 bash[39554]: 61e4a6b08cd7fc827795d0d06d44bf1e79a47788556b17d8f9f9b2739f8e2461 2021-03-24T21:51:43.906 INFO:journalctl@ceph.rgw.foo.a.smithi125.stdout:Mar 24 21:51:43 smithi125 systemd[1]: Started Ceph rgw.foo.a for 3cfa339a-8cea-11eb-90b2-001a4aab830c. 2021-03-24T21:51:44.130 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph orch apply iscsi iscsi user password --placement '1;smithi142=iscsi.a' 2021-03-24T21:51:44.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:44 smithi142 conmon[20125]: cluster 2021-03-24T21:51:42.632376+0000 mgr.y (mgr.14144) 188 : cluster [DBG] 2021-03-24T21:51:44.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:44 smithi142 conmon[20125]: pgmap v187: 4 pgs: 1 active+clean, 1 remapped+peering, 2 unknown; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-03-24T21:51:44.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:44 smithi142 conmon[20125]: audit 2021-03-24T21:51:43.340103+0000 mon.a (mon.0) 471 : audit [INF] from='client.24343 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]': finished 2021-03-24T21:51:44.364 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:44 smithi142 conmon[20125]: cluster 2021-03-24T21:51:43.340206+0000 mon.a (mon.0) 472 : cluster [DBG] osdmap e59: 8 total, 8 up, 8 in 2021-03-24T21:51:44.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:44 smithi142 conmon[20125]: audit 2021-03 2021-03-24T21:51:44.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:44 smithi142 conmon[20125]: -24T21:51:44.165306+0000 mon.a (mon.0) 473 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:44.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:44 smithi142 conmon[20125]: audit 2021-03-24T21:51:44.166209+0000 mon.a (mon.0) 474 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:51:44.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:44 smithi125 conmon[19936]: cluster 2021-03-24T21:51:42.632376+0000 mgr.y (mgr.14144) 188 : cluster [DBG] pgmap v187: 4 pgs: 1 active+clean, 1 remapped+peering, 2 unknown; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-03-24T21:51:44.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:44 smithi125 conmon[19936]: audit 2021-03-24T21:51:43.340103+0000 mon.a (mon.0) 471 : audit [INF] from='client.24343 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]': finished 2021-03-24T21:51:44.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:44 smithi125 conmon[19936]: cluster 2021-03-24T21:51:43.340206+0000 mon.a (mon.0) 472 : cluster [DBG] osdmap e59: 8 total, 8 up, 8 in 2021-03-24T21:51:44.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:44 smithi125 conmon[19936]: audit 2021-03-24T21:51:44.165306+0000 mon.a (mon.0) 473 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:44.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:44 smithi125 conmon[19936]: audit 2021-03-24T21:51:44.166209+0000 mon.a (mon.0) 474 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:51:44.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:44 smithi125 conmon[24927]: cluster 2021-03-24T21:51:42.632376+0000 mgr.y (mgr.14144) 188 : cluster [DBG] 2021-03-24T21:51:44.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:44 smithi125 conmon[24927]: pgmap v187: 4 pgs: 1 active+clean, 1 remapped+peering, 2 unknown; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-03-24T21:51:44.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:44 smithi125 conmon[24927]: audit 2021-03-24T21:51:43.340103+0000 mon.a (mon.0) 471 : audit [INF] from='client.24343 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]': finished 2021-03-24T21:51:44.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:44 smithi125 conmon[24927]: cluster 2021-03-24T21:51:43.340206+0000 mon.a (mon.0) 472 : cluster [DBG] osdmap e59: 8 total, 8 up, 8 in 2021-03-24T21:51:44.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:44 smithi125 conmon[24927]: audit 2021-03-24T21:51:44.165306+0000 mon.a (mon.0) 473 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:44.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:44 smithi125 conmon[24927]: audit 2021-03-24T21:51:44.166209+0000 mon.a (mon.0) 474 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:51:45.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:45 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:51:45.740 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:45 smithi142 conmon[20125]: 21:51:44.355392+0000 mon.a (mon.0) 475 : cluster [DBG] osdmap e60: 8 total, 8 up, 8 in 2021-03-24T21:51:45.741 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:45 smithi142 conmon[20125]: audit 2021-03-24T21:51:45.741 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:45 smithi142 conmon[20125]: 2021-03-24T21:51:44.357303+0000 mon.c (mon.2) 7 : audit 2021-03-24T21:51:45.741 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:45 smithi142 conmon[20125]: [INF] from='client.? v1:172.21.15.125:0/3436348424' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2021-03-24T21:51:45.741 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:45 smithi142 conmon[20125]: audit 2021-03-24T21:51:45.742 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:45 smithi142 conmon[20125]: 2021-03-24T21:51:44.358241+0000 mon.a (mon 2021-03-24T21:51:45.742 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:45 smithi142 conmon[20125]: .0) 476 : audit [INF] from='client.34352 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2021-03-24T21:51:45.743 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:45 smithi125 conmon[24927]: cluster 2021-03-24T21:51:44.355392+0000 mon.a (mon.0) 475 : cluster [DBG] osdmap e60: 8 total, 8 up, 8 in 2021-03-24T21:51:45.743 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:45 smithi125 conmon[24927]: audit 2021-03-24T21:51:44.357303+0000 mon.c (mon.2) 7 : audit [INF] from='client.? v1:172.21.15.125:0/3436348424' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2021-03-24T21:51:45.744 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:45 smithi125 conmon[24927]: audit 2021-03-24T21:51:44.358241+0000 mon.a (mon.0) 476 : audit [INF] from='client.34352 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2021-03-24T21:51:45.744 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:45 smithi125 conmon[19936]: cluster 2021-03-24T21:51:44.355392+0000 mon.a (mon.0) 2021-03-24T21:51:45.744 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:45 smithi125 conmon[19936]: 475 : cluster [DBG] osdmap e60: 8 total, 8 up, 8 in 2021-03-24T21:51:45.744 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:45 smithi125 conmon[19936]: audit 2021-03-24T21:51:44.357303+0000 mon.c (mon.2) 7 : audit [INF] from='client.? v1:172.21.15.125:0/3436348424' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2021-03-24T21:51:45.745 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:45 smithi125 conmon[19936]: audit 2021-03-24T21:51:44.358241+0000 mon.a (mon.0) 476 : audit [INF] from='client.34352 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2021-03-24T21:51:46.309 INFO:teuthology.orchestra.run.smithi142.stdout:Scheduled iscsi.iscsi update... 2021-03-24T21:51:46.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:51:46.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: 51:44.632843+0000 mgr.y (mgr.14144) 189 : cluster [DBG] pgmap v190: 36 pgs: 32 unknown, 4 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-03-24T21:51:46.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:51:46.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: 51:45.347211+0000 mon.a (mon 2021-03-24T21:51:46.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: .0) 477 : cluster [INF] 2021-03-24T21:51:46.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2021-03-24T21:51:46.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:51:46.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: 03-24T21:51:45.347310+0000 2021-03-24T21:51:46.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: mon.a (mon.0) 478 : 2021-03-24T21:51:46.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: cluster [INF] Cluster is now healthy 2021-03-24T21:51:46.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:51:46.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: 03-24T21:51:45.362654 2021-03-24T21:51:46.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: +0000 mon.a (mon.0) 479 2021-03-24T21:51:46.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: : audit [INF] from='client.34352 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2021-03-24T21:51:46.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:51:46.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: 03-24T21:51:45. 2021-03-24T21:51:46.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: 362823+0000 mon.a (mon.0) 2021-03-24T21:51:46.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: 480 : cluster [DBG] osdmap e61: 8 total, 8 up, 8 in 2021-03-24T21:51:46.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:51:46.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: 03-24T21:51:46.100672 2021-03-24T21:51:46.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: +0000 mon.a (mon.0) 2021-03-24T21:51:46.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: 481 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:46.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:51:46.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: 03-24T21:51:46.106165 2021-03-24T21:51:46.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: +0000 mon.a (mon.0) 2021-03-24T21:51:46.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: 482 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:46.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:51:46.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: 03-24T21:51:46. 2021-03-24T21:51:46.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: 109105+0000 mon.a (mon.0) 2021-03-24T21:51:46.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: 483 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:51:46.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: audit 2021 2021-03-24T21:51:46.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: -03-24T21:51:46. 2021-03-24T21:51:46.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: 117764+0000 mon.a (mon.0 2021-03-24T21:51:46.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: ) 484 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:46.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: 2021-03-24T21:51:46.625 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:51:46.625 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: 03-24T21:51:46.312782+0000 2021-03-24T21:51:46.625 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: mon.a (mon.0) 485 : 2021-03-24T21:51:46.625 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:46.626 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:51:46.626 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: 03-24T21:51:46.314204 2021-03-24T21:51:46.627 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: +0000 mon.a (mon.0) 486 2021-03-24T21:51:46.627 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:51:46.627 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:51:46.627 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: 03-24T21:51:46. 2021-03-24T21:51:46.628 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: 322378+0000 mon.a (mon.0 2021-03-24T21:51:46.628 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: ) 487 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:46.628 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: 2021-03-24T21:51:46.628 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: audit 2021-03-24T21:51:46.629 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: 2021-03-24T21:51:46. 2021-03-24T21:51:46.629 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: 334871+0000 mon.a (mon.0) 2021-03-24T21:51:46.629 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: 488 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2021-03-24T21:51:46.629 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: audit 2021 2021-03-24T21:51:46.630 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: -03-24T21:51:46.340268+0000 2021-03-24T21:51:46.630 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: mon.a (mon.0) 489 : 2021-03-24T21:51:46.631 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2021-03-24T21:51:46.631 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: audit 2021-03-24T21:51:46.631 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: 2021-03-24T21:51:46. 2021-03-24T21:51:46.632 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: 354908+0000 mon.a (mon.0 2021-03-24T21:51:46.632 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: ) 490 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:51:46.632 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:46 smithi142 conmon[20125]: 2021-03-24T21:51:46.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:46 smithi125 conmon[19936]: cluster 2021-03-24T21:51:44.632843+0000 mgr.y (mgr.14144) 2021-03-24T21:51:46.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:46 smithi125 conmon[19936]: 189 : cluster [DBG] pgmap v190: 36 pgs: 32 unknown, 4 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-03-24T21:51:46.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:46 smithi125 conmon[19936]: cluster 2021-03-24T21:51:45.347211+0000 mon.a (mon.0) 477 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2021-03-24T21:51:46.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:46 smithi125 conmon[19936]: cluster 2021-03-24T21:51:45.347310+0000 mon.a (mon.0) 478 : cluster [INF] Cluster is now healthy 2021-03-24T21:51:46.657 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:46 smithi125 conmon[19936]: audit 2021-03-24T21:51:45.362654+0000 mon.a (mon.0) 479 : audit [INF] from='client.34352 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2021-03-24T21:51:46.657 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:46 smithi125 conmon[19936]: cluster 2021-03-24T21:51:45.362823+0000 mon.a (mon.0) 480 : cluster [DBG] osdmap e61: 8 total, 8 up, 8 in 2021-03-24T21:51:46.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:46 smithi125 conmon[19936]: audit 2021-03-24T21:51:46.100672+0000 2021-03-24T21:51:46.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:46 smithi125 conmon[19936]: mon.a (mon.0) 481 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:46.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:46 smithi125 conmon[19936]: audit 2021-03-24T21:51:46.106165+0000 mon.a (mon.0) 482 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:46.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:46 smithi125 conmon[19936]: audit 2021-03-24T21:51: 2021-03-24T21:51:46.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:46 smithi125 conmon[19936]: 46.109105+0000 mon.a (mon.0) 483 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:51:46.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:46 smithi125 conmon[19936]: audit 2021-03-24T21:51:46.117764+0000 mon.a (mon.0) 484 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:46.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:46 smithi125 conmon[19936]: audit 2021-03-24T21:51:46.312782+0000 mon.a (mon.0) 485 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:46.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:46 smithi125 conmon[19936]: audit 2021-03-24T21: 2021-03-24T21:51:46.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:46 smithi125 conmon[19936]: 51:46.314204+0000 mon.a (mon.0) 486 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:51:46.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:46 smithi125 conmon[19936]: audit 2021-03-24T21:51:46 2021-03-24T21:51:46.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:46 smithi125 conmon[19936]: .322378+0000 mon.a (mon.0) 487 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:46.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:46 smithi125 conmon[19936]: audit 2021-03-24T21:51:46.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:46 smithi125 conmon[19936]: 2021-03-24T21:51:46.334871+0000 mon.a (mon.0) 488 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2021-03-24T21:51:46.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:46 smithi125 conmon[19936]: audit 2021-03-24T21:51:46.340268+0000 mon.a (mon.0) 489 : audit 2021-03-24T21:51:46.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:46 smithi125 conmon[19936]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2021-03-24T21:51:46.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:46 smithi125 conmon[19936]: audit 2021-03-24T21:51: 2021-03-24T21:51:46.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:46 smithi125 conmon[19936]: 46.354908+0000 mon.a (mon.0) 490 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:51:46.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:46 smithi125 conmon[24927]: cluster 2021-03-24T21:51:44.632843+0000 mgr.y (mgr.14144 2021-03-24T21:51:46.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:46 smithi125 conmon[24927]: ) 189 : cluster [DBG] pgmap v190: 36 pgs: 32 unknown, 4 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-03-24T21:51:46.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:46 smithi125 conmon[24927]: cluster 2021-03-24T21:51:45.347211+0000 mon.a (mon.0) 477 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2021-03-24T21:51:46.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:46 smithi125 conmon[24927]: cluster 2021-03-24T21:51:45.347310+0000 mon.a (mon.0) 478 : 2021-03-24T21:51:46.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:46 smithi125 conmon[24927]: cluster [INF] Cluster is now healthy 2021-03-24T21:51:46.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:46 smithi125 conmon[24927]: audit 2021-03-24T21:51:46.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:46 smithi125 conmon[24927]: 2021-03-24T21:51:45.362654+0000 mon.a (mon.0) 479 : audit [INF] from='client.34352 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2021-03-24T21:51:46.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:46 smithi125 conmon[24927]: cluster 2021-03-24T21:51:45.362823+0000 2021-03-24T21:51:46.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:46 smithi125 conmon[24927]: mon.a (mon.0) 480 : cluster [DBG] osdmap e61: 8 total, 8 up, 8 in 2021-03-24T21:51:46.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:46 smithi125 conmon[24927]: audit 2021-03-24T21:51:46. 2021-03-24T21:51:46.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:46 smithi125 conmon[24927]: 100672+0000 mon.a (mon.0) 481 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:46.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:46 smithi125 conmon[24927]: audit 2021-03-24T21:51:46.106165+0000 mon.a 2021-03-24T21:51:46.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:46 smithi125 conmon[24927]: (mon.0) 482 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:46.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:46 smithi125 conmon[24927]: audit 2021-03-24T21:51:46.109105+0000 mon.a (mon.0) 2021-03-24T21:51:46.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:46 smithi125 conmon[24927]: 483 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:51:46.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:46 smithi125 conmon[24927]: audit 2021-03-24 2021-03-24T21:51:46.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:46 smithi125 conmon[24927]: T21:51:46.117764+0000 mon.a (mon.0) 484 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:46.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:46 smithi125 conmon[24927]: audit 2021-03-24T21:51:46.312782+0000 mon.a ( 2021-03-24T21:51:46.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:46 smithi125 conmon[24927]: mon.0) 485 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:46.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:46 smithi125 conmon[24927]: audit 2021-03-24T21:51:46.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:46 smithi125 conmon[24927]: 2021-03-24T21:51:46.314204+0000 mon.a (mon.0) 486 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:51:46.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:46 smithi125 conmon[24927]: audit 2021-03-24T21:51:46.322378+0000 2021-03-24T21:51:46.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:46 smithi125 conmon[24927]: mon.a (mon.0) 487 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:46.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:46 smithi125 conmon[24927]: audit 2021-03-24T21:51:46.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:46 smithi125 conmon[24927]: 2021-03-24T21:51:46.334871+0000 mon.a (mon.0) 488 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2021-03-24T21:51:46.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:46 smithi125 conmon[24927]: audit 2021-03-24T21:51:46.340268+0000 mon.a (mon.0) 489 2021-03-24T21:51:46.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:46 smithi125 conmon[24927]: : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2021-03-24T21:51:46.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:46 smithi125 conmon[24927]: audit 2021-03-24T21:51: 2021-03-24T21:51:46.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:46 smithi125 conmon[24927]: 46.354908+0000 mon.a (mon.0) 490 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-24T21:51:47.205 DEBUG:teuthology.orchestra.run.smithi142:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c@iscsi.iscsi.a.service 2021-03-24T21:51:47.209 INFO:tasks.cephadm:Adding prometheus.a on smithi142 2021-03-24T21:51:47.210 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph orch apply prometheus '1;smithi142=a' 2021-03-24T21:51:47.243 INFO:journalctl@ceph.iscsi.iscsi.a.smithi142.stdout:-- Logs begin at Wed 2021-03-03 23:51:07 UTC. -- 2021-03-24T21:51:47.462 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:47 smithi142 conmon[20125]: audit 2021-03-24T 2021-03-24T21:51:47.766 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:47 smithi142 conmon[20125]: 21:51:46.304682+0000 mgr.y (mgr.14144) 190 : audit [DBG] 2021-03-24T21:51:47.766 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:47 smithi142 conmon[20125]: from='client.24358 v1:172.21.15.142:0/2623068419' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "iscsi", "api_user": "user", "api_password": "password", "placement": "1;smithi142=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:51:47.766 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:47 smithi142 conmon[20125]: cephadm 2021- 2021-03-24T21:51:47.767 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:47 smithi142 conmon[20125]: 03-24T21:51:46.305849+0000 2021-03-24T21:51:47.767 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:47 smithi142 conmon[20125]: mgr.y (mgr.14144) 191 : 2021-03-24T21:51:47.767 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:47 smithi142 conmon[20125]: cephadm [INF] Saving service iscsi.iscsi spec with placement smithi142=iscsi.a;count:1 2021-03-24T21:51:47.767 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:47 smithi142 conmon[20125]: cephadm 2021-03-24T21:51:47.767 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:47 smithi142 conmon[20125]: 2021-03-24T21: 2021-03-24T21:51:47.768 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:47 smithi142 conmon[20125]: 51:46.324364+0000 2021-03-24T21:51:47.768 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:47 smithi142 conmon[20125]: mgr.y (mgr.14144) 2021-03-24T21:51:47.768 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:47 smithi142 conmon[20125]: 192 : cephadm [INF] 2021-03-24T21:51:47.768 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:47 smithi142 conmon[20125]: Checking pool "iscsi" exists for service iscsi.iscsi 2021-03-24T21:51:47.768 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:47 smithi142 conmon[20125]: cephadm 2021-03-24T21:51:47.769 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:47 smithi142 conmon[20125]: 2021-03-24T21: 2021-03-24T21:51:47.769 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:47 smithi142 conmon[20125]: 51:46.356233+0000 2021-03-24T21:51:47.769 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:47 smithi142 conmon[20125]: mgr.y (mgr.14144) 2021-03-24T21:51:47.769 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:47 smithi142 conmon[20125]: 193 : cephadm [INF] 2021-03-24T21:51:47.770 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:47 smithi142 conmon[20125]: Deploying daemon iscsi.iscsi.a on smithi142 2021-03-24T21:51:47.770 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:47 smithi142 conmon[20125]: cluster 2021-03-24T21:51:47.770 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:47 smithi142 conmon[20125]: 2021-03-24T21 2021-03-24T21:51:47.770 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:47 smithi142 conmon[20125]: :51:46.376047+0000 2021-03-24T21:51:47.770 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:47 smithi142 conmon[20125]: mon.a (mon.0 2021-03-24T21:51:47.771 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:47 smithi142 conmon[20125]: ) 491 : cluster [DBG] 2021-03-24T21:51:47.771 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:47 smithi142 conmon[20125]: osdmap e62: 8 total, 8 up, 8 in 2021-03-24T21:51:47.771 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:47 smithi142 conmon[20125]: audit 2021-03-24T21:51:47.771 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:47 smithi142 conmon[20125]: 2021-03-24T21 2021-03-24T21:51:47.771 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:47 smithi142 conmon[20125]: :51:46.379324+0000 2021-03-24T21:51:47.772 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:47 smithi142 conmon[20125]: mon.c (mon.2) 2021-03-24T21:51:47.772 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:47 smithi142 conmon[20125]: 8 : audit [INF] from='client.? v1:172.21.15.125:0/3436348424' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2021-03-24T21:51:47.772 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:47 smithi142 conmon[20125]: 2021-03-24T21:51:47.772 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:47 smithi142 conmon[20125]: audit 2021-03-24T21:51:47.772 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:47 smithi142 conmon[20125]: 2021-03-24T 2021-03-24T21:51:47.773 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:47 smithi142 conmon[20125]: 21:51:46.380159 2021-03-24T21:51:47.773 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:47 smithi142 conmon[20125]: +0000 mon.a (mon. 2021-03-24T21:51:47.773 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:47 smithi142 conmon[20125]: 0) 492 : audit 2021-03-24T21:51:47.773 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:47 smithi142 conmon[20125]: [INF] from='client.34352 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2021-03-24T21:51:47.774 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:47 smithi125 conmon[19936]: audit 2021-03-24T21 2021-03-24T21:51:47.774 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:47 smithi125 conmon[19936]: :51:46.304682+0000 mgr.y (mgr.14144) 190 : audit [DBG] from='client.24358 v1:172.21.15.142:0/2623068419' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "iscsi", "api_user": "user", "api_password": "password", "placement": "1;smithi142=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:51:47.775 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:47 smithi125 conmon[19936]: cephadm 2021-03-24T21:51:46.305849+0000 mgr.y (mgr.14144) 191 : cephadm [INF] Saving service iscsi.iscsi spec with placement smithi142=iscsi.a;count:1 2021-03-24T21:51:47.775 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:47 smithi125 conmon[19936]: cephadm 2021-03-24T21:51:46.324364+0000 mgr.y (mgr.14144) 192 : cephadm [INF] Checking pool "iscsi" exists for service iscsi.iscsi 2021-03-24T21:51:47.775 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:47 smithi125 conmon[19936]: cephadm 2021-03-24T21:51:46.356233+0000 mgr.y (mgr.14144) 193 : cephadm [INF] Deploying daemon iscsi.iscsi.a on smithi142 2021-03-24T21:51:47.775 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:47 smithi125 conmon[19936]: cluster 2021-03-24T21:51:46.376047+0000 mon.a (mon.0) 491 : cluster [DBG] osdmap e62: 8 total, 8 up, 8 in 2021-03-24T21:51:47.776 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:47 smithi125 conmon[19936]: audit 2021-03-24T21:51:46.379324+0000 mon.c (mon.2) 8 : audit [INF] from='client.? v1:172.21.15.125:0/3436348424' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2021-03-24T21:51:47.776 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:47 smithi125 conmon[19936]: audit 2021-03-24T 2021-03-24T21:51:47.776 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:47 smithi125 conmon[19936]: 21:51:46.380159+0000 mon.a (mon.0) 492 : audit [INF] from='client.34352 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2021-03-24T21:51:47.776 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:47 smithi125 conmon[24927]: audit 2021-03-24T21:51:47.777 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:47 smithi125 conmon[24927]: 2021-03-24T21:51:47.777 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:47 smithi125 conmon[24927]: 2021-03-24T21: 2021-03-24T21:51:47.777 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:47 smithi125 conmon[24927]: 51:46.304682+0000 mgr.y (mgr.14144) 190 : audit [DBG] from='client.24358 v1:172.21.15.142:0/2623068419' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "iscsi", "api_user": "user", "api_password": "password", "placement": "1;smithi142=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:51:47.777 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:47 smithi125 conmon[24927]: cephadm 2021-03-24T21:51:47.778 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:47 smithi125 conmon[24927]: 2021-03-24T21:51:47.778 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:47 smithi125 conmon[24927]: 2021 2021-03-24T21:51:47.778 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:47 smithi125 conmon[24927]: - 2021-03-24T21:51:47.778 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:47 smithi125 conmon[24927]: 03 2021-03-24T21:51:47.778 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:47 smithi125 conmon[24927]: - 2021-03-24T21:51:47.779 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:47 smithi125 conmon[24927]: 24 2021-03-24T21:51:47.779 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:47 smithi125 conmon[24927]: T 2021-03-24T21:51:47.779 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:47 smithi125 conmon[24927]: 21 2021-03-24T21:51:47.779 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:47 smithi125 conmon[24927]: :51:46.305849+0000 mgr.y (mgr 2021-03-24T21:51:47.780 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:47 smithi125 conmon[24927]: .14144) 191 : cephadm [INF] Saving service iscsi.iscsi spec with placement smithi142=iscsi.a;count:1 2021-03-24T21:51:47.780 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:47 smithi125 conmon[24927]: cephadm 2021-03- 2021-03-24T21:51:47.780 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:47 smithi125 conmon[24927]: 24T21:51:46.324364+0000 mgr.y 2021-03-24T21:51:47.780 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:47 smithi125 conmon[24927]: (mgr.14144) 192 : cephadm [INF] 2021-03-24T21:51:47.780 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:47 smithi125 conmon[24927]: Checking pool "iscsi" exists for service iscsi.iscsi 2021-03-24T21:51:47.781 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:47 smithi125 conmon[24927]: cephadm 2021-03-24T 2021-03-24T21:51:47.781 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:47 smithi125 conmon[24927]: 21:51:46.356233+0000 mgr.y (mgr. 2021-03-24T21:51:47.781 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:47 smithi125 conmon[24927]: 14144) 193 : cephadm [INF] Deploying daemon iscsi.iscsi.a on smithi142 2021-03-24T21:51:47.781 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:47 smithi125 conmon[24927]: cluster 2021-03-24T21:51:47.781 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:47 smithi125 conmon[24927]: 2021-03-24T21:51: 2021-03-24T21:51:47.782 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:47 smithi125 conmon[24927]: 46.376047+0000 mon.a (mon.0) 2021-03-24T21:51:47.782 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:47 smithi125 conmon[24927]: 491 : cluster [DBG] osdmap e62: 8 total, 8 up, 8 in 2021-03-24T21:51:47.782 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:47 smithi125 conmon[24927]: audit 2021 2021-03-24T21:51:47.782 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:47 smithi125 conmon[24927]: -03-24T21:51:46.379324 2021-03-24T21:51:47.782 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:47 smithi125 conmon[24927]: +0000 mon.c (mon.2) 8 : audit 2021-03-24T21:51:47.783 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:47 smithi125 conmon[24927]: [INF] from='client.? v1:172.21.15.125:0/3436348424' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2021-03-24T21:51:47.783 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:47 smithi125 conmon[24927]: audit 2021-03 2021-03-24T21:51:47.783 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:47 smithi125 conmon[24927]: -24T21:51:46.380159+0000 2021-03-24T21:51:47.783 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:47 smithi125 conmon[24927]: mon.a (mon.0) 492 : audit 2021-03-24T21:51:47.784 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:47 smithi125 conmon[24927]: [INF] from='client.34352 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2021-03-24T21:51:48.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:48 smithi125 conmon[24927]: cluster 2021-03-24T21:51:48.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:48 smithi125 conmon[24927]: 2021-03-24T21:51:48.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:48 smithi125 conmon[24927]: 2021 2021-03-24T21:51:48.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:48 smithi125 conmon[24927]: - 2021-03-24T21:51:48.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:48 smithi125 conmon[24927]: 03-24T21:51:46.633499+0000 mgr.y ( 2021-03-24T21:51:48.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:48 smithi125 conmon[24927]: mgr.14144) 194 : cluster [DBG] pgmap v193: 68 pgs: 4 creating+peering, 58 unknown, 6 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-03-24T21:51:48.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:48 smithi125 conmon[24927]: audit 2021-03-24T21:51:47.398770+0000 mon.a ( 2021-03-24T21:51:48.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:48 smithi125 conmon[24927]: mon.0) 493 : audit [INF] from='client.34352 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2021-03-24T21:51:48.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:48 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:51:48.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:48 smithi125 conmon[24927]: -03-24T21:51:47.398963+0000 mon.a 2021-03-24T21:51:48.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:48 smithi125 conmon[24927]: (mon.0) 494 : cluster [DBG] osdmap e63: 8 total, 8 up, 8 in 2021-03-24T21:51:48.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:48 smithi125 conmon[24927]: cluster 2021-03-24T21:51:48.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:48 smithi125 conmon[24927]: 2021-03-24T21:51:47.399873+0000 2021-03-24T21:51:48.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:48 smithi125 conmon[24927]: mon.a (mon.0) 495 : cluster [DBG] mgrmap e15: y(active, since 4m), standbys: x 2021-03-24T21:51:48.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:48 smithi125 conmon[19936]: cluster 2021-03-24T21:51:48.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:48 smithi125 conmon[19936]: 2021-03-24T21:51:48.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:48 smithi125 conmon[19936]: 2021 2021-03-24T21:51:48.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:48 smithi125 conmon[19936]: - 2021-03-24T21:51:48.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:48 smithi125 conmon[19936]: 03 2021-03-24T21:51:48.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:48 smithi125 conmon[19936]: - 2021-03-24T21:51:48.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:48 smithi125 conmon[19936]: 24 2021-03-24T21:51:48.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:48 smithi125 conmon[19936]: T21:51:46.633499 2021-03-24T21:51:48.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:48 smithi125 conmon[19936]: +0000 mgr.y (mgr.14144) 194 : cluster [DBG] pgmap v193: 68 pgs: 4 creating+peering, 58 unknown, 6 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-03-24T21:51:48.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:48 smithi125 conmon[19936]: audit 2021-03-24T21:51:48.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:48 smithi125 conmon[19936]: 2021-03-24T21:51:47.398770+0000 mon.a (mon 2021-03-24T21:51:48.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:48 smithi125 conmon[19936]: .0) 493 : audit [INF] from='client.34352 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2021-03-24T21:51:48.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:48 smithi125 conmon[19936]: cluster 2021-03-24T21:51:47.398963 2021-03-24T21:51:48.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:48 smithi125 conmon[19936]: +0000 mon.a (mon.0) 494 : cluster [DBG] osdmap e63: 8 total, 8 up, 8 in 2021-03-24T21:51:48.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:48 smithi125 conmon[19936]: cluster 2021-03-24T21:51:47.399873+0000 mon.a (mon.0 2021-03-24T21:51:48.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:48 smithi125 conmon[19936]: ) 495 : cluster [DBG] mgrmap e15: y(active, since 4m), standbys: x 2021-03-24T21:51:48.663 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:48 smithi142 conmon[20125]: cluster 2021-03-24T21:51:46.633499+0000 mgr.y (mgr. 2021-03-24T21:51:48.663 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:48 smithi142 conmon[20125]: 14144) 194 : cluster [DBG] pgmap v193: 68 pgs: 4 creating+peering, 58 unknown, 6 active+clean; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-03-24T21:51:48.663 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:48 smithi142 conmon[20125]: audit 2021-03-24T21:51:47.398770+0000 mon.a (mon.0) 493 : audit [INF] from='client.34352 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2021-03-24T21:51:48.664 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:48 smithi142 conmon[20125]: cluster 2021-03-24T21:51:47.398963+0000 2021-03-24T21:51:48.664 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:48 smithi142 conmon[20125]: mon.a (mon.0) 494 : 2021-03-24T21:51:48.664 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:48 smithi142 conmon[20125]: cluster [DBG] osdmap e63: 8 total, 8 up, 8 in 2021-03-24T21:51:48.664 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:48 smithi142 conmon[20125]: cluster 2021-03-24T21:51:47.399873 2021-03-24T21:51:48.665 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:48 smithi142 conmon[20125]: +0000 mon.a (mon.0) 495 : cluster [DBG] mgrmap e15: y(active, since 4m), standbys: x 2021-03-24T21:51:49.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:49 smithi142 conmon[20125]: cluster 2021-03-24T21:51:48.395041+0000 2021-03-24T21:51:49.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:49 smithi142 conmon[20125]: mon.a (mon.0) 496 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2021-03-24T21:51:49.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:49 smithi142 conmon[20125]: audit 2021-03-24T21:51:48.398628+0000 mon.c (mon.2) 9 : audit [INF] 2021-03-24T21:51:49.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:49 smithi142 conmon[20125]: from='client.? v1:172.21.15.125:0/3436348424' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2021-03-24T21:51:49.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:49 smithi142 conmon[20125]: audit 2021-03-24 2021-03-24T21:51:49.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:49 smithi142 conmon[20125]: T21:51:48.400449+0000 mon.a (mon.0) 497 : audit [INF] 2021-03-24T21:51:49.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:49 smithi142 conmon[20125]: from='client.34352 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2021-03-24T21:51:49.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:49 smithi142 conmon[20125]: audit 2021-03-24T21:51:49.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:49 smithi142 conmon[20125]: 2021-03-24T21:51:49.399894+0000 mon.a (mon. 2021-03-24T21:51:49.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:49 smithi142 conmon[20125]: 0) 498 : audit [INF] from='client.34352 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2021-03-24T21:51:49.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:49 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:51:49.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:49 smithi142 conmon[20125]: T21:51:49.400178+0000 mon.a (mon. 2021-03-24T21:51:49.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:49 smithi142 conmon[20125]: 0) 499 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2021-03-24T21:51:49.906 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:49 smithi125 conmon[24927]: cluster 2021-03-24T21:51:49.906 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:49 smithi125 conmon[24927]: 2021-03-24T21:51:49.906 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:49 smithi125 conmon[24927]: 2021 2021-03-24T21:51:49.907 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:49 smithi125 conmon[24927]: - 2021-03-24T21:51:49.907 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:49 smithi125 conmon[24927]: 03 2021-03-24T21:51:49.907 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:49 smithi125 conmon[24927]: -24T21:51:48.395041+0000 mon.a ( 2021-03-24T21:51:49.907 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:49 smithi125 conmon[24927]: mon.0) 496 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2021-03-24T21:51:49.908 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:49 smithi125 conmon[24927]: audit 2021-03-24T21:51:48.398628+0000 mon.c ( 2021-03-24T21:51:49.908 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:49 smithi125 conmon[24927]: mon.2) 9 : audit [INF] from='client.? v1:172.21.15.125:0/3436348424' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2021-03-24T21:51:49.908 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:49 smithi125 conmon[24927]: audit 2021-03-24T21:51: 2021-03-24T21:51:49.908 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:49 smithi125 conmon[24927]: 48.400449+0000 mon.a (mon.0) 497 : audit [INF] from='client.34352 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2021-03-24T21:51:49.908 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:49 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T21:51:49.909 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:49 smithi125 conmon[24927]: 24T21:51:49.399894+0000 mon.a (mon.0) 498 : audit [INF] 2021-03-24T21:51:49.909 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:49 smithi125 conmon[24927]: from='client.34352 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2021-03-24T21:51:49.909 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:49 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:51:49.909 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:49 smithi125 conmon[24927]: 21:51:49.400178+0000 mon.a (mon.0) 499 : cluster 2021-03-24T21:51:49.910 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:49 smithi125 conmon[24927]: [DBG] osdmap e65: 8 total, 8 up, 8 in 2021-03-24T21:51:49.910 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:49 smithi125 conmon[19936]: cluster 2021-03-24T21:51:49.910 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:49 smithi125 conmon[19936]: 2021-03-24T21:51:49.911 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:49 smithi125 conmon[19936]: 2021 2021-03-24T21:51:49.911 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:49 smithi125 conmon[19936]: - 2021-03-24T21:51:49.911 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:49 smithi125 conmon[19936]: 03 2021-03-24T21:51:49.911 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:49 smithi125 conmon[19936]: - 2021-03-24T21:51:49.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:49 smithi125 conmon[19936]: 24 2021-03-24T21:51:49.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:49 smithi125 conmon[19936]: T 2021-03-24T21:51:49.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:49 smithi125 conmon[19936]: 21 2021-03-24T21:51:49.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:49 smithi125 conmon[19936]: : 2021-03-24T21:51:49.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:49 smithi125 conmon[19936]: 51 2021-03-24T21:51:49.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:49 smithi125 conmon[19936]: : 2021-03-24T21:51:49.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:49 smithi125 conmon[19936]: 48 2021-03-24T21:51:49.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:49 smithi125 conmon[19936]: . 2021-03-24T21:51:49.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:49 smithi125 conmon[19936]: 395041 2021-03-24T21:51:49.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:49 smithi125 conmon[19936]: +0000 2021-03-24T21:51:49.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:49 smithi125 conmon[19936]: 2021-03-24T21:51:49.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:49 smithi125 conmon[19936]: mon.a 2021-03-24T21:51:49.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:49 smithi125 conmon[19936]: ( 2021-03-24T21:51:49.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:49 smithi125 conmon[19936]: mon 2021-03-24T21:51:49.915 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:49 smithi125 conmon[19936]: . 2021-03-24T21:51:49.915 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:49 smithi125 conmon[19936]: 0 2021-03-24T21:51:49.915 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:49 smithi125 conmon[19936]: ) 2021-03-24T21:51:49.915 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:49 smithi125 conmon[19936]: 496 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2021-03-24T21:51:49.915 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:49 smithi125 conmon[19936]: audit 2021-03-24T21:51:48.398628 2021-03-24T21:51:49.916 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:49 smithi125 conmon[19936]: +0000 mon.c (mon.2) 9 : audit [INF] from='client.? v1:172.21.15.125:0/3436348424' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2021-03-24T21:51:49.916 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:49 smithi125 conmon[19936]: audit 2021-03 2021-03-24T21:51:49.916 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:49 smithi125 conmon[19936]: -24T21:51:48.400449+0000 mon.a (mon.0 2021-03-24T21:51:49.916 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:49 smithi125 conmon[19936]: ) 497 : audit [INF] from='client.34352 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2021-03-24T21:51:49.916 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:49 smithi125 conmon[19936]: audit 2021-03-24T21:51:49.399894+0000 mon.a ( 2021-03-24T21:51:49.917 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:49 smithi125 conmon[19936]: mon.0) 498 : audit [INF] from='client.34352 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2021-03-24T21:51:49.917 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:49 smithi125 conmon[19936]: cluster 2021-03-24T21:51: 2021-03-24T21:51:49.917 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:49 smithi125 conmon[19936]: 49.400178+0000 mon.a (mon.0) 499 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2021-03-24T21:51:49.917 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:49 smithi125 conmon[19936]: 2021-03-24T21:51:50.580 INFO:journalctl@ceph.iscsi.iscsi.a.smithi142.stdout:Mar 24 21:51:50 smithi142 bash[36457]: log file path now is '/var/log/tcmu-runner.log' 2021-03-24T21:51:50.750 INFO:teuthology.orchestra.run.smithi142.stdout:Scheduled prometheus update... 2021-03-24T21:51:50.864 INFO:journalctl@ceph.iscsi.iscsi.a.smithi142.stdout:Mar 24 21:51:50 smithi142 podman[36566]: 2021-03-24 21:51:50.582832683 +0000 UTC m=+0.296951270 container create 047cd4bcf8340cecc067dbcfa00d33423901896f7ee3243696a688c706fb979d (image=quay.ceph.io/ceph-ci/ceph@sha256:e9db06906b8d0a24f3ef6835ac5aa4b8eea2950a2560bad78f83d4f101eca0db, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-iscsi.iscsi.a, CEPH_POINT_RELEASE=, maintainer=Dimitri Savineau , ceph=True, RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.18.0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20201204, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06) 2021-03-24T21:51:50.865 INFO:journalctl@ceph.iscsi.iscsi.a.smithi142.stdout:Mar 24 21:51:50 smithi142 podman[36566]: 2021-03-24 21:51:50.724537937 +0000 UTC m=+0.438656521 container init 047cd4bcf8340cecc067dbcfa00d33423901896f7ee3243696a688c706fb979d (image=quay.ceph.io/ceph-ci/ceph@sha256:e9db06906b8d0a24f3ef6835ac5aa4b8eea2950a2560bad78f83d4f101eca0db, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-iscsi.iscsi.a, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.18.0, CEPH_POINT_RELEASE=, maintainer=Dimitri Savineau , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20201204, org.label-schema.name=CentOS Base Image, RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d, ceph=True, GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2021-03-24T21:51:50.865 INFO:journalctl@ceph.iscsi.iscsi.a.smithi142.stdout:Mar 24 21:51:50 smithi142 podman[36566]: 2021-03-24 21:51:50.766007013 +0000 UTC m=+0.480125525 container start 047cd4bcf8340cecc067dbcfa00d33423901896f7ee3243696a688c706fb979d (image=quay.ceph.io/ceph-ci/ceph@sha256:e9db06906b8d0a24f3ef6835ac5aa4b8eea2950a2560bad78f83d4f101eca0db, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-iscsi.iscsi.a, org.label-schema.build-date=20201204, org.label-schema.name=CentOS Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06, GIT_BRANCH=HEAD, maintainer=Dimitri Savineau , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.18.0) 2021-03-24T21:51:50.865 INFO:journalctl@ceph.iscsi.iscsi.a.smithi142.stdout:Mar 24 21:51:50 smithi142 bash[36566]: 047cd4bcf8340cecc067dbcfa00d33423901896f7ee3243696a688c706fb979d 2021-03-24T21:51:50.865 INFO:journalctl@ceph.iscsi.iscsi.a.smithi142.stdout:Mar 24 21:51:50 smithi142 systemd[1]: Started Ceph iscsi.iscsi.a for 3cfa339a-8cea-11eb-90b2-001a4aab830c. 2021-03-24T21:51:50.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:50 smithi142 conmon[20125]: cluster 2021-03-24T21:51:50.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:50 smithi142 conmon[20125]: 2021-03-24T21:51:50.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:50 smithi142 conmon[20125]: 2021 2021-03-24T21:51:50.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:50 smithi142 conmon[20125]: - 2021-03-24T21:51:50.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:50 smithi142 conmon[20125]: 03 2021-03-24T21:51:50.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:50 smithi142 conmon[20125]: - 2021-03-24T21:51:50.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:50 smithi142 conmon[20125]: 24 2021-03-24T21:51:50.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:50 smithi142 conmon[20125]: T 2021-03-24T21:51:50.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:50 smithi142 conmon[20125]: 21 2021-03-24T21:51:50.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:50 smithi142 conmon[20125]: : 2021-03-24T21:51:50.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:50 smithi142 conmon[20125]: 51 2021-03-24T21:51:50.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:50 smithi142 conmon[20125]: : 2021-03-24T21:51:50.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:50 smithi142 conmon[20125]: 48 2021-03-24T21:51:50.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:50 smithi142 conmon[20125]: . 2021-03-24T21:51:50.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:50 smithi142 conmon[20125]: 634192 2021-03-24T21:51:50.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:50 smithi142 conmon[20125]: +0000 2021-03-24T21:51:50.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:50 smithi142 conmon[20125]: 2021-03-24T21:51:50.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:50 smithi142 conmon[20125]: mgr.y 2021-03-24T21:51:50.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:50 smithi142 conmon[20125]: ( 2021-03-24T21:51:50.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:50 smithi142 conmon[20125]: mgr 2021-03-24T21:51:50.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:50 smithi142 conmon[20125]: . 2021-03-24T21:51:50.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:50 smithi142 conmon[20125]: 14144 2021-03-24T21:51:50.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:50 smithi142 conmon[20125]: ) 2021-03-24T21:51:50.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:50 smithi142 conmon[20125]: 195 2021-03-24T21:51:50.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:50 smithi142 conmon[20125]: : 2021-03-24T21:51:50.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:50 smithi142 conmon[20125]: cluster 2021-03-24T21:51:50.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:50 smithi142 conmon[20125]: 2021-03-24T21:51:50.872 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:50 smithi142 conmon[20125]: [DBG] 2021-03-24T21:51:50.872 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:50 smithi142 conmon[20125]: 2021-03-24T21:51:50.872 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:50 smithi142 conmon[20125]: pgmap v196: 100 pgs: 4 creating+peering, 70 unknown, 26 active+clean; 358 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-03-24T21:51:50.872 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:50 smithi142 conmon[20125]: 2021-03-24T21:51:50.873 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:50 smithi142 conmon[20125]: cluster 2021-03-24T21:51:50.873 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:50 smithi142 conmon[20125]: 2021-03-24T21:51:50.873 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:50 smithi142 conmon[20125]: 2021 2021-03-24T21:51:50.873 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:50 smithi142 conmon[20125]: - 2021-03-24T21:51:50.873 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:50 smithi142 conmon[20125]: 03 2021-03-24T21:51:50.874 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:50 smithi142 conmon[20125]: - 2021-03-24T21:51:50.874 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:50 smithi142 conmon[20125]: 24 2021-03-24T21:51:50.874 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:50 smithi142 conmon[20125]: T 2021-03-24T21:51:50.874 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:50 smithi142 conmon[20125]: 21 2021-03-24T21:51:50.874 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:50 smithi142 conmon[20125]: : 2021-03-24T21:51:50.875 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:50 smithi142 conmon[20125]: 51 2021-03-24T21:51:50.875 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:50 smithi142 conmon[20125]: :49.809569+0000 osd.2 (osd.2) 5 : 2021-03-24T21:51:50.875 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:50 smithi142 conmon[20125]: cluster [DBG] purged_snaps scrub starts 2021-03-24T21:51:50.875 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:50 smithi142 conmon[20125]: cluster 2021-03-24T21:51: 2021-03-24T21:51:50.875 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:50 smithi142 conmon[20125]: 49.809714+0000 osd.2 (osd.2) 6 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:51:51.156 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:50 smithi125 conmon[19936]: cluster 2021-03-24T21:51:48.634192+0000 mgr.y (mgr 2021-03-24T21:51:51.156 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:50 smithi125 conmon[19936]: .14144) 195 : cluster [DBG] pgmap v196: 100 pgs: 4 creating+peering, 70 unknown, 26 active+clean; 358 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-03-24T21:51:51.156 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:50 smithi125 conmon[19936]: cluster 2021-03-24T21:51:49.809569+0000 osd.2 (osd.2) 5 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:51:51.157 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:50 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:51:51.157 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:50 smithi125 conmon[19936]: :51:49.809714+0000 osd.2 (osd.2) 6 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:51:51.157 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:50 smithi125 conmon[24927]: cluster 2021-03-24T21:51:51.157 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:50 smithi125 conmon[24927]: 2021-03-24T21:51:51.158 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:50 smithi125 conmon[24927]: 2021 2021-03-24T21:51:51.158 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:50 smithi125 conmon[24927]: - 2021-03-24T21:51:51.158 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:50 smithi125 conmon[24927]: 03 2021-03-24T21:51:51.158 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:50 smithi125 conmon[24927]: - 2021-03-24T21:51:51.159 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:50 smithi125 conmon[24927]: 24T21:51:48.634192 2021-03-24T21:51:51.159 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:50 smithi125 conmon[24927]: +0000 mgr.y (mgr.14144) 195 : cluster [DBG] pgmap v196: 100 pgs: 4 creating+peering, 70 unknown, 26 active+clean; 358 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-03-24T21:51:51.159 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:50 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:51:51.159 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:50 smithi125 conmon[24927]: -03-24T21:51:49.809569 2021-03-24T21:51:51.159 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:50 smithi125 conmon[24927]: +0000 osd.2 (osd.2) 2021-03-24T21:51:51.160 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:50 smithi125 conmon[24927]: 5 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:51:51.160 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:50 smithi125 conmon[24927]: 2021-03-24T21:51:51.160 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:50 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:51:51.160 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:50 smithi125 conmon[24927]: -03-24T21:51:49.809714 2021-03-24T21:51:51.161 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:50 smithi125 conmon[24927]: +0000 osd.2 (osd.2) 6 : cluster 2021-03-24T21:51:51.161 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:50 smithi125 conmon[24927]: [DBG] purged_snaps scrub ok 2021-03-24T21:51:51.648 DEBUG:teuthology.orchestra.run.smithi142:prometheus.a> sudo journalctl -f -n 0 -u ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c@prometheus.a.service 2021-03-24T21:51:51.652 INFO:tasks.cephadm:Adding node-exporter.a on smithi125 2021-03-24T21:51:51.652 INFO:tasks.cephadm:Adding node-exporter.b on smithi142 2021-03-24T21:51:51.653 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph orch apply node-exporter '2;smithi125=a;smithi142=b' 2021-03-24T21:51:51.667 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:-- Logs begin at Wed 2021-03-03 23:51:07 UTC. -- 2021-03-24T21:51:51.864 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:51 smithi142 conmon[20125]: audit 2021-03-24T21:51:50.359874+0000 mgr.y 2021-03-24T21:51:51.864 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:51 smithi142 conmon[20125]: (mgr.14144) 196 : audit [DBG] from='client.24370 v1:172.21.15.142:0/1692354001' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi142=a", "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:51:51.865 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:51 smithi142 conmon[20125]: cephadm 2021-03-24T21:51:50.361265+0000 mgr.y (mgr.14144) 197 : cephadm [INF] Saving service prometheus spec with placement smithi142=a;count:1 2021-03-24T21:51:51.865 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:51 smithi142 conmon[20125]: cluster 2021-03-24T21:51:50.634863+0000 mgr.y (mgr.14144) 198 : cluster [DBG] pgmap v198: 100 pgs: 12 creating+peering, 20 unknown, 68 active+clean; 3.3 KiB data, 44 MiB used, 715 GiB / 715 GiB avail; 7.1 KiB/s rd, 7.1 KiB/s wr, 25 op/s 2021-03-24T21:51:51.865 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:51 smithi142 conmon[20125]: cluster 2021-03-24T21:51:50.743085+0000 mon.a (mon.0) 500 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2021-03-24T21:51:51.865 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:51 smithi142 conmon[20125]: audit 2021-03-24T21:51:50.751010+0000 mon.a (mon.0) 501 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:51.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:51 smithi142 conmon[20125]: audit 2021-03-24T21:51:50.753534+0000 mon.b (mon.1) 25 : audit [INF] from='client.? v1:172.21.15.125:0/3051439499' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2021-03-24T21:51:51.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:51 smithi142 conmon[20125]: audit 2021-03-24T21:51:50.754909+0000 mon.a (mon.0) 502 : audit [INF] from='client.24376 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2021-03-24T21:51:51.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:51 smithi142 conmon[20125]: audit 2021-03-24T21:51:50.803673+0000 mon.a (mon.0) 503 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:51.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:51 smithi142 conmon[20125]: audit 2021-03-24T21:51:50.805211+0000 mon.a (mon.0) 504 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:51:51.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:51 smithi142 conmon[20125]: audit 2021-03-24T21:51:51.396645+0000 mon.b (mon.1) 26 : audit [DBG] from='client.? v1:172.21.15.142:0/976398904' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2021-03-24T21:51:51.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:51 smithi142 conmon[20125]: audit 2021-03-24T21:51:51.745033+0000 mon.a (mon.0) 505 : audit [INF] from='client.24376 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2021-03-24T21:51:51.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:51 smithi142 conmon[20125]: cluster 2021-03-24T21:51:51.745157+0000 mon.a (mon.0) 506 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2021-03-24T21:51:52.156 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[19936]: audit 2021-03-24T21:51:50.359874+0000 mgr.y (mgr.14144) 196 : audit [DBG] from='client.24370 v1:172.21.15.142:0/1692354001' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi142=a", "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:51:52.156 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[19936]: cephadm 2021-03-24T21:51:50.361265+0000 mgr.y (mgr.14144) 197 : cephadm [INF] Saving service prometheus spec with placement smithi142=a;count:1 2021-03-24T21:51:52.157 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[19936]: cluster 2021-03-24T21:51:50.634863+0000 mgr.y (mgr.14144) 198 : cluster [DBG] pgmap v198: 100 pgs: 12 creating+peering, 20 unknown, 68 active+clean; 3.3 KiB data, 44 MiB used, 715 GiB / 715 GiB avail; 7.1 KiB/s rd, 7.1 KiB/s wr, 25 op/s 2021-03-24T21:51:52.157 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[19936]: cluster 2021-03-24T21:51:52.157 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[19936]: 2021-03-24T21:51:50.743085+0000 mon.a (mon.0) 500 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2021-03-24T21:51:52.157 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[19936]: audit 2021-03 2021-03-24T21:51:52.158 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[19936]: -24T21:51:50.751010+0000 mon.a (mon.0) 501 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:52.158 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[19936]: audit 2021-03- 2021-03-24T21:51:52.158 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[19936]: 24T21:51:50.753534+0000 mon.b (mon.1) 25 2021-03-24T21:51:52.158 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[19936]: : audit [INF] from='client.? v1:172.21.15.125:0/3051439499' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2021-03-24T21:51:52.158 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[19936]: audit 2021-03-24T21:51:52.159 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[19936]: 2021-03-24T21:51:50.754909+0000 mon.a (mon.0 2021-03-24T21:51:52.159 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[19936]: ) 502 : audit [INF] from='client.24376 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2021-03-24T21:51:52.159 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[19936]: audit 2021-03-24T21:51:50.803673 2021-03-24T21:51:52.159 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[19936]: +0000 mon.a (mon.0) 503 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:52.160 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[19936]: audit 2021-03-24T21 2021-03-24T21:51:52.160 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[19936]: :51:50.805211+0000 mon.a (mon.0) 504 : 2021-03-24T21:51:52.160 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[19936]: audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:51:52.160 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[19936]: audit 2021-03-24T21 2021-03-24T21:51:52.160 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[19936]: :51:51.396645+0000 mon.b (mon.1) 26 : audit [DBG] from='client.? v1:172.21.15.142:0/976398904' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2021-03-24T21:51:52.161 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[19936]: audit 2021-03-24T21:51:51.745033+0000 mon.a (mon.0) 2021-03-24T21:51:52.161 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[19936]: 505 : audit [INF] from='client.24376 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2021-03-24T21:51:52.161 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T21:51:52.161 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[19936]: T21:51:51.745157+0000 mon.a (mon.0) 506 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2021-03-24T21:51:52.162 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: audit 2021-03-24T21:51:52.162 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 2021-03-24T21:51:52.163 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 2021 2021-03-24T21:51:52.163 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: - 2021-03-24T21:51:52.163 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 03 2021-03-24T21:51:52.164 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: - 2021-03-24T21:51:52.164 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 24 2021-03-24T21:51:52.164 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: T 2021-03-24T21:51:52.164 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 21 2021-03-24T21:51:52.165 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: : 2021-03-24T21:51:52.165 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 51 2021-03-24T21:51:52.165 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: : 2021-03-24T21:51:52.165 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 50 2021-03-24T21:51:52.165 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: . 2021-03-24T21:51:52.166 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 359874 2021-03-24T21:51:52.166 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: +0000 2021-03-24T21:51:52.166 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 2021-03-24T21:51:52.166 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: mgr.y 2021-03-24T21:51:52.166 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: ( 2021-03-24T21:51:52.167 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: mgr 2021-03-24T21:51:52.167 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: . 2021-03-24T21:51:52.167 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 14144 2021-03-24T21:51:52.167 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: ) 2021-03-24T21:51:52.168 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 196 2021-03-24T21:51:52.168 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: : 2021-03-24T21:51:52.168 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: audit 2021-03-24T21:51:52.168 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 2021-03-24T21:51:52.168 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: [DBG] 2021-03-24T21:51:52.169 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 2021-03-24T21:51:52.169 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: from='client.24370 v1:172.21.15.142:0/1692354001' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi142=a", "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:51:52.169 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 2021-03-24T21:51:52.169 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: cephadm 2021-03-24T21:51:52.169 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 2021-03-24T21:51:52.170 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 2021 2021-03-24T21:51:52.170 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: - 2021-03-24T21:51:52.170 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 03 2021-03-24T21:51:52.170 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: - 2021-03-24T21:51:52.171 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 24 2021-03-24T21:51:52.171 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: T 2021-03-24T21:51:52.171 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 21 2021-03-24T21:51:52.171 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: : 2021-03-24T21:51:52.172 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 51 2021-03-24T21:51:52.172 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: : 2021-03-24T21:51:52.172 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 50 2021-03-24T21:51:52.172 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: . 2021-03-24T21:51:52.172 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 361265 2021-03-24T21:51:52.173 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: +0000 2021-03-24T21:51:52.173 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 2021-03-24T21:51:52.173 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: mgr.y 2021-03-24T21:51:52.173 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: ( 2021-03-24T21:51:52.173 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: mgr 2021-03-24T21:51:52.174 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: . 2021-03-24T21:51:52.174 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 14144 2021-03-24T21:51:52.174 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: ) 2021-03-24T21:51:52.174 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 197 2021-03-24T21:51:52.174 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: : 2021-03-24T21:51:52.175 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: cephadm 2021-03-24T21:51:52.175 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 2021-03-24T21:51:52.175 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: [INF] 2021-03-24T21:51:52.175 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 2021-03-24T21:51:52.176 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: Saving service prometheus spec with placement smithi142=a;count:1 2021-03-24T21:51:52.176 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 2021-03-24T21:51:52.176 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: cluster 2021-03-24T21:51:52.176 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 2021-03-24T21:51:52.176 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 2021 2021-03-24T21:51:52.177 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: - 2021-03-24T21:51:52.177 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 03 2021-03-24T21:51:52.177 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: - 2021-03-24T21:51:52.177 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 24 2021-03-24T21:51:52.177 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: T 2021-03-24T21:51:52.178 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 21 2021-03-24T21:51:52.178 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: : 2021-03-24T21:51:52.178 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 51 2021-03-24T21:51:52.178 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: : 2021-03-24T21:51:52.179 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 50 2021-03-24T21:51:52.179 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: . 2021-03-24T21:51:52.179 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 634863 2021-03-24T21:51:52.179 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: +0000 2021-03-24T21:51:52.179 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 2021-03-24T21:51:52.180 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: mgr.y 2021-03-24T21:51:52.180 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: ( 2021-03-24T21:51:52.180 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: mgr 2021-03-24T21:51:52.180 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: . 2021-03-24T21:51:52.180 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 14144 2021-03-24T21:51:52.181 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: ) 2021-03-24T21:51:52.181 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 198 2021-03-24T21:51:52.181 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: : 2021-03-24T21:51:52.181 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: cluster 2021-03-24T21:51:52.181 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 2021-03-24T21:51:52.182 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: [DBG] 2021-03-24T21:51:52.182 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 2021-03-24T21:51:52.182 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: pgmap v198: 100 pgs: 12 creating+peering, 20 unknown, 68 active+clean; 3.3 KiB data, 44 MiB used, 715 GiB / 715 GiB avail; 7.1 KiB/s rd, 7.1 KiB/s wr, 25 op/s 2021-03-24T21:51:52.182 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 2021-03-24T21:51:52.182 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: cluster 2021-03-24T21:51:52.183 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 2021-03-24T21:51:52.183 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 2021 2021-03-24T21:51:52.183 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: - 2021-03-24T21:51:52.183 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 03 2021-03-24T21:51:52.184 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: - 2021-03-24T21:51:52.184 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 24 2021-03-24T21:51:52.184 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: T 2021-03-24T21:51:52.184 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 21 2021-03-24T21:51:52.184 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: : 2021-03-24T21:51:52.185 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 51 2021-03-24T21:51:52.185 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: : 2021-03-24T21:51:52.185 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 50 2021-03-24T21:51:52.185 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: . 2021-03-24T21:51:52.185 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 743085 2021-03-24T21:51:52.186 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: +0000 2021-03-24T21:51:52.186 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 2021-03-24T21:51:52.186 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: mon.a 2021-03-24T21:51:52.186 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: ( 2021-03-24T21:51:52.186 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: mon 2021-03-24T21:51:52.187 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: . 2021-03-24T21:51:52.187 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 0 2021-03-24T21:51:52.187 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: ) 2021-03-24T21:51:52.187 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 500 2021-03-24T21:51:52.187 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: : 2021-03-24T21:51:52.188 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: cluster 2021-03-24T21:51:52.188 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 2021-03-24T21:51:52.188 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: [DBG] 2021-03-24T21:51:52.188 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 2021-03-24T21:51:52.189 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: osdmap e66: 8 total, 8 up, 8 in 2021-03-24T21:51:52.189 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 2021-03-24T21:51:52.189 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: audit 2021-03-24T21:51:52.189 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 2021-03-24T21:51:52.189 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 2021 2021-03-24T21:51:52.190 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: - 2021-03-24T21:51:52.190 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 03 2021-03-24T21:51:52.190 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: - 2021-03-24T21:51:52.190 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 24 2021-03-24T21:51:52.190 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: T 2021-03-24T21:51:52.191 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 21 2021-03-24T21:51:52.191 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: : 2021-03-24T21:51:52.191 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 51 2021-03-24T21:51:52.191 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: : 2021-03-24T21:51:52.192 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 50 2021-03-24T21:51:52.192 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: . 2021-03-24T21:51:52.192 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 751010 2021-03-24T21:51:52.192 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: +0000 2021-03-24T21:51:52.192 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 2021-03-24T21:51:52.193 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: mon.a 2021-03-24T21:51:52.193 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: ( 2021-03-24T21:51:52.193 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: mon 2021-03-24T21:51:52.193 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: . 2021-03-24T21:51:52.193 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 0 2021-03-24T21:51:52.194 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: ) 2021-03-24T21:51:52.194 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 501 2021-03-24T21:51:52.194 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: : 2021-03-24T21:51:52.194 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: audit 2021-03-24T21:51:52.195 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 2021-03-24T21:51:52.195 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: [INF] 2021-03-24T21:51:52.195 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:52.195 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: audit 2021-03-24T21:51:50.753534+0000 mon.b ( 2021-03-24T21:51:52.195 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: mon.1) 25 : audit [INF] from='client.? v1:172.21.15.125:0/3051439499' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2021-03-24T21:51:52.196 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: audit 2021-03-24T21:51:52.196 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 2021- 2021-03-24T21:51:52.196 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 03-24T21:51:50.754909+0000 2021-03-24T21:51:52.196 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: mon.a (mon.0) 502 : audit [INF] from='client.24376 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2021-03-24T21:51:52.197 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 2021-03-24T21:51:52.197 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: audit 2021-03-24T21:51:52.197 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 2021-03-24T21:51:50.803673+0000 mon.a (mon.0) 503 2021-03-24T21:51:52.197 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:52.198 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: audit 2021-03-24T21:51:52.198 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 2021-03- 2021-03-24T21:51:52.198 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 24T21:51:50.805211+0000 mon.a (mon.0 2021-03-24T21:51:52.198 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: ) 504 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:51:52.198 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: audit 2021-03-24T21:51:52.199 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 2021-03-24 2021-03-24T21:51:52.199 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: T21:51:51.396645+0000 mon.b (mon.1) 2021-03-24T21:51:52.199 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 26 : audit [DBG] from='client.? v1:172.21.15.142:0/976398904' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2021-03-24T21:51:52.199 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: audit 2021-03-24T21:51:52.200 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 2021-03-24T21: 2021-03-24T21:51:52.200 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: 51:51.745033+0000 mon.a (mon.0) 505 2021-03-24T21:51:52.200 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: : audit [INF] from='client.24376 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2021-03-24T21:51:52.200 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:51:52.201 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: -03-24T21:51:51.745157+0000 mon.a ( 2021-03-24T21:51:52.201 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:51 smithi125 conmon[24927]: mon.0) 506 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2021-03-24T21:51:52.864 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:52 smithi142 conmon[20125]: audit 2021-03-24T21:51:51.759303+0000 mon.b (mon.1) 27 : audit 2021-03-24T21:51:52.865 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:52 smithi142 conmon[20125]: [INF] from='client.? v1:172.21.15.125:0/3051439499' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2021-03-24T21:51:52.865 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:52 smithi142 conmon[20125]: audit 2021-03-24T21:51:51.762808+0000 mon.a (mon.0) 507 : audit [INF] from='client.24376 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2021-03-24T21:51:53.156 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:52 smithi125 conmon[24927]: audit 2021-03-24T21:51:53.156 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:52 smithi125 conmon[24927]: 2021-03-24T21:51:53.156 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:52 smithi125 conmon[24927]: 2021-03-24T21:51:51. 2021-03-24T21:51:53.157 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:52 smithi125 conmon[24927]: 759303+0000 mon.b (mon.1) 27 : audit [INF] from='client.? v1:172.21.15.125:0/3051439499' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2021-03-24T21:51:53.157 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:52 smithi125 conmon[24927]: audit 2021-03-24T21:51:53.157 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:52 smithi125 conmon[24927]: 2021-03-24T21:51:51.762808+0000 mon.a (mon.0) 507 2021-03-24T21:51:53.157 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:52 smithi125 conmon[24927]: : audit [INF] from='client.24376 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2021-03-24T21:51:53.158 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:52 smithi125 conmon[19936]: audit 2021-03-24T 2021-03-24T21:51:53.158 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:52 smithi125 conmon[19936]: 21:51:51.759303+0000 mon.b (mon.1) 27 : audit [INF] from='client.? v1:172.21.15.125:0/3051439499' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2021-03-24T21:51:53.158 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:52 smithi125 conmon[19936]: audit 2021-03-24T21:51:53.159 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:52 smithi125 conmon[19936]: 2021-03-24T21:51:51 2021-03-24T21:51:53.159 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:52 smithi125 conmon[19936]: .762808+0000 mon.a 2021-03-24T21:51:53.159 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:52 smithi125 conmon[19936]: (mon.0) 2021-03-24T21:51:53.160 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:52 smithi125 conmon[19936]: 507 : audit 2021-03-24T21:51:53.160 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:52 smithi125 conmon[19936]: [INF] from='client.24376 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2021-03-24T21:51:53.864 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:53 smithi142 conmon[20125]: cluster 2021-03-24T21:51:52.635550+0000 mgr.y (mgr 2021-03-24T21:51:53.865 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:53 smithi142 conmon[20125]: .14144) 199 : cluster [DBG] pgmap v201: 132 pgs: 13 creating+peering, 39 unknown, 80 active+clean; 5.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail; 6.8 KiB/s rd, 6.8 KiB/s wr, 25 op/s 2021-03-24T21:51:53.865 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:53 smithi142 conmon[20125]: audit 2021-03-24T21:51:52.760632+0000 mon.a ( 2021-03-24T21:51:53.865 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:53 smithi142 conmon[20125]: mon.0) 508 : audit [INF] from='client.24376 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2021-03-24T21:51:53.865 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:53 smithi142 conmon[20125]: audit 2021-03-24T21:51:53.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:53 smithi142 conmon[20125]: 2021-03-24T21:51:52 2021-03-24T21:51:53.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:53 smithi142 conmon[20125]: .760692+0000 mon.b (mon.1) 28 2021-03-24T21:51:53.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:53 smithi142 conmon[20125]: : audit [INF] from='client.? v1:172.21.15.125:0/3051439499' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2021-03-24T21:51:53.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:53 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:51:53.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:53 smithi142 conmon[20125]: -03-24T21:51:52.760769+0000 2021-03-24T21:51:53.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:53 smithi142 conmon[20125]: mon.a (mon.0) 509 : cluster 2021-03-24T21:51:53.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:53 smithi142 conmon[20125]: [DBG] osdmap e68: 8 total, 8 up, 8 in 2021-03-24T21:51:53.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:53 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:51:53.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:53 smithi142 conmon[20125]: 03-24T21:51:52.763035+0000 2021-03-24T21:51:53.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:53 smithi142 conmon[20125]: mon.a (mon.0) 510 : audit 2021-03-24T21:51:53.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:53 smithi142 conmon[20125]: [INF] from='client.24376 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2021-03-24T21:51:54.102 INFO:teuthology.orchestra.run.smithi142.stdout:Scheduled node-exporter update... 2021-03-24T21:51:54.156 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:53 smithi125 conmon[19936]: cluster 2021-03-24T21:51:52.635550+0000 mgr.y (mgr.14144) 199 2021-03-24T21:51:54.156 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:53 smithi125 conmon[19936]: : cluster [DBG] pgmap v201: 132 pgs: 13 creating+peering, 39 unknown, 80 active+clean; 5.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail; 6.8 KiB/s rd, 6.8 KiB/s wr, 25 op/s 2021-03-24T21:51:54.156 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:53 smithi125 conmon[19936]: audit 2021-03-24T21:51:52.760632+0000 mon.a (mon.0) 508 : audit [INF] from='client.24376 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2021-03-24T21:51:54.157 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:53 smithi125 conmon[19936]: audit 2021-03-24T21:51:52.760692+0000 mon.b (mon.1) 28 : audit [INF] from='client.? v1:172.21.15.125:0/3051439499' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2021-03-24T21:51:54.157 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:53 smithi125 conmon[19936]: 2021-03-24T21:51:54.157 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:53 smithi125 conmon[19936]: cluster 2021-03-24T21:51:52.760769+0000 mon.a (mon.0) 509 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2021-03-24T21:51:54.157 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:53 smithi125 conmon[19936]: audit 2021-03-24T21:51:52.763035+0000 mon.a (mon.0) 510 : audit [INF] from='client.24376 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2021-03-24T21:51:54.158 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:53 smithi125 conmon[24927]: cluster 2021-03-24T21:51:54.158 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:53 smithi125 conmon[24927]: 2021-03-24T21:51:52.635550+0000 mgr.y 2021-03-24T21:51:54.158 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:53 smithi125 conmon[24927]: (mgr.14144) 199 : cluster [DBG] pgmap v201: 132 pgs: 13 creating+peering, 39 unknown, 80 active+clean; 5.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail; 6.8 KiB/s rd, 6.8 KiB/s wr, 25 op/s 2021-03-24T21:51:54.158 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:53 smithi125 conmon[24927]: audit 2021-03-24 2021-03-24T21:51:54.159 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:53 smithi125 conmon[24927]: T21:51:52.760632+0000 mon.a (mon.0) 508 : audit [INF] from='client.24376 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2021-03-24T21:51:54.159 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:53 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T21:51:54.159 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:53 smithi125 conmon[24927]: 24T21:51:52.760692+0000 mon.b (mon.1) 28 : audit [INF] from='client.? v1:172.21.15.125:0/3051439499' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2021-03-24T21:51:54.159 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:53 smithi125 conmon[24927]: 2021-03-24T21:51:54.160 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:53 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:51:54.160 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:53 smithi125 conmon[24927]: 24T21:51:52.760769+0000 mon.a (mon.0) 509 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2021-03-24T21:51:54.160 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:53 smithi125 conmon[24927]: audit 2021-03-24T21:51:52 2021-03-24T21:51:54.160 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:53 smithi125 conmon[24927]: .763035+0000 mon.a (mon.0) 510 : audit [INF] from='client.24376 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2021-03-24T21:51:54.865 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:54 smithi142 conmon[20125]: cluster 2021-03-24T21:51:52.833945+0000 osd.5 (osd.5) 4 : cluster 2021-03-24T21:51:54.865 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:54 smithi142 conmon[20125]: [DBG] purged_snaps scrub starts 2021-03-24T21:51:54.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:54 smithi142 conmon[20125]: cluster 2021-03-24T21:51:52.834135+0000 osd.5 (osd.5) 5 : 2021-03-24T21:51:54.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:54 smithi142 conmon[20125]: cluster [DBG] purged_snaps scrub ok 2021-03-24T21:51:54.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:54 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:51:54.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:54 smithi142 conmon[20125]: 03-24T21:51:53.763055+0000 mon.a (mon. 2021-03-24T21:51:54.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:54 smithi142 conmon[20125]: 0) 511 : audit [INF] from='client.24376 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2021-03-24T21:51:54.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:54 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:51:54.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:54 smithi142 conmon[20125]: 24T21:51:53.763269+0000 mon.a ( 2021-03-24T21:51:54.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:54 smithi142 conmon[20125]: mon.0) 512 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2021-03-24T21:51:54.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:54 smithi142 conmon[20125]: audit 2021-03 2021-03-24T21:51:54.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:54 smithi142 conmon[20125]: -24T21:51:54.097705+0000 mgr.y (mgr 2021-03-24T21:51:54.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:54 smithi142 conmon[20125]: .14144) 200 : audit [DBG] from='client.24400 v1:172.21.15.142:0/1174518353' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi125=a;smithi142=b", "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:51:54.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:54 smithi142 conmon[20125]: cephadm 2021-03-24T21:51:54.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:54 smithi142 conmon[20125]: 2021-03- 2021-03-24T21:51:54.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:54 smithi142 conmon[20125]: 24T21:51:54.098945+0000 mgr.y 2021-03-24T21:51:54.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:54 smithi142 conmon[20125]: (mgr.14144) 201 : cephadm [INF] Saving service node-exporter spec with placement smithi125=a;smithi142=b;count:2 2021-03-24T21:51:54.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:54 smithi142 conmon[20125]: 2021-03-24T21:51:54.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:54 smithi142 conmon[20125]: audit 2021-03-24 2021-03-24T21:51:54.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:54 smithi142 conmon[20125]: T21:51:54.105163+0000 mon.a (mon.0) 2021-03-24T21:51:54.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:54 smithi142 conmon[20125]: 513 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:54.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:54 smithi142 conmon[20125]: audit 2021-03-24T 2021-03-24T21:51:54.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:54 smithi142 conmon[20125]: 21:51:54.349118+0000 mon.a (mon. 2021-03-24T21:51:54.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:54 smithi142 conmon[20125]: 0) 514 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:54.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:54 smithi142 conmon[20125]: audit 2021-03- 2021-03-24T21:51:54.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:54 smithi142 conmon[20125]: 24T21:51:54.355682+0000 mon.a 2021-03-24T21:51:54.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:54 smithi142 conmon[20125]: (mon.0) 515 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:54.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:54 smithi142 conmon[20125]: 2021-03-24T21:51:54.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:54 smithi142 conmon[20125]: cephadm 2021 2021-03-24T21:51:54.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:54 smithi142 conmon[20125]: -03-24T21:51:54.364226+0000 mgr.y 2021-03-24T21:51:54.872 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:54 smithi142 conmon[20125]: (mgr.14144) 202 : cephadm [INF] Deploying daemon prometheus.a on smithi142 2021-03-24T21:51:54.872 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:54 smithi142 conmon[20125]: 2021-03-24T21:51:55.156 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[19936]: cluster 2021-03-24T21:51 2021-03-24T21:51:55.157 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[19936]: :52.833945+0000 osd.5 (osd.5) 4 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:51:55.157 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[19936]: cluster 2021-03-24T21:51:55.157 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[19936]: 2021-03-24T21:51: 2021-03-24T21:51:55.158 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[19936]: 52.834135+0000 osd.5 (osd.5) 5 2021-03-24T21:51:55.158 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[19936]: : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:51:55.158 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[19936]: audit 2021-03- 2021-03-24T21:51:55.158 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[19936]: 24T21:51:53.763055+0000 mon.a 2021-03-24T21:51:55.158 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[19936]: (mon.0) 511 : audit [INF] from='client.24376 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2021-03-24T21:51:55.159 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[19936]: 2021-03-24T21:51:55.159 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[19936]: cluster 2021-03-24T21:51 2021-03-24T21:51:55.159 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[19936]: :53.763269+0000 mon.a (mon.0) 512 : cluster [DBG] 2021-03-24T21:51:55.159 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[19936]: osdmap e69: 8 total, 8 up, 8 in 2021-03-24T21:51:55.160 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[19936]: audit 2021-03-24T21:51:55.160 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[19936]: 2021-03-24T21:51:54. 2021-03-24T21:51:55.160 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[19936]: 097705+0000 mgr.y (mgr.14144) 2021-03-24T21:51:55.160 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[19936]: 200 : audit [DBG] from='client.24400 v1:172.21.15.142:0/1174518353' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi125=a;smithi142=b", "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:51:55.160 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[19936]: cephadm 2021-03-24T 2021-03-24T21:51:55.161 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[19936]: 21:51:54.098945+0000 mgr.y (mgr.14144) 201 : cephadm [INF] 2021-03-24T21:51:55.161 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[19936]: Saving service node-exporter spec with placement smithi125=a;smithi142=b;count:2 2021-03-24T21:51:55.161 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:51:55.161 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[19936]: 03-24T21:51:54.105163+0000 mon.a (mon.0) 513 : audit [INF] 2021-03-24T21:51:55.161 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:55.162 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[19936]: audit 2021-03-24T21:51:55.162 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[19936]: 2021-03-24T21:51:54.349118+0000 mon.a (mon.0) 514 : audit 2021-03-24T21:51:55.162 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[19936]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:55.162 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[19936]: audit 2021-03-24T21:51: 2021-03-24T21:51:55.162 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[19936]: 54.355682+0000 mon.a (mon.0) 515 : audit [INF] 2021-03-24T21:51:55.163 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:55.163 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[19936]: cephadm 2021-03-24T21:51:55.163 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[19936]: 2021-03-24T21:51 2021-03-24T21:51:55.163 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[19936]: :54.364226+0000 mgr.y (mgr.14144) 202 : 2021-03-24T21:51:55.164 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[19936]: cephadm [INF] Deploying daemon prometheus.a on smithi142 2021-03-24T21:51:55.164 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:51:55.164 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[24927]: T21:51:52.833945+0000 osd.5 (osd.5) 4 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:51:55.164 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:51:55.165 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[24927]: T21:51:52.834135+0000 osd.5 2021-03-24T21:51:55.165 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[24927]: (osd.5) 5 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:51:55.165 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[24927]: audit 2021- 2021-03-24T21:51:55.165 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[24927]: 03-24T21:51:53.763055+0000 mon.a 2021-03-24T21:51:55.165 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[24927]: (mon.0) 511 : audit [INF] from='client.24376 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2021-03-24T21:51:55.166 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:51:55.166 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[24927]: :51:53.763269+0000 mon.a (mon.0) 512 2021-03-24T21:51:55.166 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[24927]: : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2021-03-24T21:51:55.166 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T21:51:55.166 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[24927]: 24T21:51:54.097705+0000 mgr.y (mgr 2021-03-24T21:51:55.167 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[24927]: .14144) 200 : audit [DBG] from='client.24400 v1:172.21.15.142:0/1174518353' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi125=a;smithi142=b", "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:51:55.167 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[24927]: cephadm 2021-03-24T21:51: 2021-03-24T21:51:55.167 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[24927]: 54.098945+0000 mgr.y (mgr.14144) 201 : cephadm [INF] 2021-03-24T21:51:55.167 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[24927]: Saving service node-exporter spec with placement smithi125=a;smithi142=b;count:2 2021-03-24T21:51:55.168 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[24927]: audit 2021 2021-03-24T21:51:55.168 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[24927]: -03-24T21:51:54.105163+0000 mon.a (mon.0) 513 2021-03-24T21:51:55.168 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[24927]: : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:55.168 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T21:51:55.168 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[24927]: 24T21:51:54.349118+0000 mon.a (mon. 2021-03-24T21:51:55.169 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[24927]: 0) 514 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:55.169 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[24927]: audit 2021 2021-03-24T21:51:55.169 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[24927]: -03-24T21:51:54.355682+0000 mon.a 2021-03-24T21:51:55.169 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[24927]: (mon.0) 515 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:51:55.169 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[24927]: cephadm 2021-03-24T 2021-03-24T21:51:55.170 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[24927]: 21:51:54.364226+0000 mgr.y (mgr.14144) 202 2021-03-24T21:51:55.170 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:54 smithi125 conmon[24927]: : cephadm [INF] Deploying daemon prometheus.a on smithi142 2021-03-24T21:51:55.363 DEBUG:teuthology.orchestra.run.smithi125:node-exporter.a> sudo journalctl -f -n 0 -u ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c@node-exporter.a.service 2021-03-24T21:51:55.366 DEBUG:teuthology.orchestra.run.smithi142:node-exporter.b> sudo journalctl -f -n 0 -u ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c@node-exporter.b.service 2021-03-24T21:51:55.368 INFO:tasks.cephadm:Adding alertmanager.a on smithi125 2021-03-24T21:51:55.369 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph orch apply alertmanager '1;smithi125=a' 2021-03-24T21:51:55.386 INFO:journalctl@ceph.node-exporter.b.smithi142.stdout:-- Logs begin at Wed 2021-03-03 23:51:07 UTC. -- 2021-03-24T21:51:55.406 INFO:journalctl@ceph.node-exporter.a.smithi125.stdout:-- Logs begin at Wed 2021-03-03 23:51:07 UTC. -- 2021-03-24T21:51:56.107 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:56 smithi142 conmon[20125]: cluster 2021-03-24T21:51:54 2021-03-24T21:51:56.108 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:56 smithi142 conmon[20125]: .636639+0000 mgr.y (mgr.14144) 203 : cluster [DBG] pgmap v204: 132 pgs: 2 creating+peering, 3 unknown, 127 active+clean; 21 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 30 KiB/s wr, 158 op/s 2021-03-24T21:51:56.108 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:56 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:51:56.108 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:56 smithi142 conmon[20125]: -24T21:51:55.018543+0000 mon.a (mon.0) 516 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2021-03-24T21:51:56.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:56 smithi125 conmon[19936]: cluster 2021-03-24T21:51 2021-03-24T21:51:56.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:56 smithi125 conmon[19936]: :54.636639+0000 mgr.y (mgr.14144) 203 : cluster [DBG] pgmap v204: 132 pgs: 2 creating+peering, 3 unknown, 127 active+clean; 21 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 30 KiB/s wr, 158 op/s 2021-03-24T21:51:56.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:56 smithi125 conmon[19936]: cluster 2021-03-24T21:51:55.018543+0000 mon.a 2021-03-24T21:51:56.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:56 smithi125 conmon[19936]: (mon.0) 516 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2021-03-24T21:51:56.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:56 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:51:56.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:56 smithi125 conmon[24927]: 21:51:54.636639+0000 mgr.y (mgr.14144) 203 : cluster [DBG] pgmap v204: 132 pgs: 2 creating+peering, 3 unknown, 127 active+clean; 21 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 30 KiB/s wr, 158 op/s 2021-03-24T21:51:56.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:56 smithi125 conmon[24927]: cluster 2021-03-24T21:51:56.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:56 smithi125 conmon[24927]: 2021-03-24T21:51:55. 2021-03-24T21:51:56.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:56 smithi125 conmon[24927]: 018543+0000 mon.a (mon.0) 516 : cluster 2021-03-24T21:51:56.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:56 smithi125 conmon[24927]: [DBG] osdmap e70: 8 total, 8 up, 8 in 2021-03-24T21:51:58.114 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:58 smithi142 conmon[20125]: cluster 2021-03-24T21:51 2021-03-24T21:51:58.115 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:58 smithi142 conmon[20125]: :56.637492+0000 mgr.y (mgr.14144) 204 : cluster [DBG] pgmap v206: 132 pgs: 132 active+clean; 24 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 26 KiB/s wr, 198 op/s 2021-03-24T21:51:58.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:58 smithi125 conmon[19936]: cluster 2021-03-24T21:51:56.637492+0000 2021-03-24T21:51:58.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:58 smithi125 conmon[19936]: mgr.y (mgr.14144) 204 : cluster [DBG] pgmap v206: 132 pgs: 132 active+clean; 24 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 26 KiB/s wr, 198 op/s 2021-03-24T21:51:58.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:58 smithi125 conmon[24927]: cluster 2021-03-24T21:51: 2021-03-24T21:51:58.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:58 smithi125 conmon[24927]: 56.637492+0000 mgr.y (mgr.14144) 204 : cluster [DBG] pgmap v206: 132 pgs: 132 active+clean; 24 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 26 KiB/s wr, 198 op/s 2021-03-24T21:51:58.722 INFO:teuthology.orchestra.run.smithi142.stdout:Scheduled alertmanager update... 2021-03-24T21:52:00.115 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:59 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:52:00.115 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:59 smithi142 conmon[20125]: 51:58.638399+0000 mgr.y (mgr.14144) 205 : cluster [DBG] pgmap v207: 132 pgs: 132 active+clean; 25 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 22 KiB/s wr, 220 op/s 2021-03-24T21:52:00.115 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:59 smithi142 conmon[20125]: audit 2021-03-24T 2021-03-24T21:52:00.116 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:59 smithi142 conmon[20125]: 21:51:58.716299+0000 mgr.y (mgr.14144) 206 : audit [DBG] from='client.24406 v1:172.21.15.142:0/3546318051' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi125=a", "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:52:00.116 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:59 smithi142 conmon[20125]: cephadm 2021-03-24T21:51: 2021-03-24T21:52:00.116 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:59 smithi142 conmon[20125]: 58.717681+0000 mgr.y (mgr.14144) 207 2021-03-24T21:52:00.116 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:59 smithi142 conmon[20125]: : cephadm [INF] Saving service alertmanager spec with placement smithi125=a;count:1 2021-03-24T21:52:00.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:59 smithi142 conmon[20125]: audit 2021-03- 2021-03-24T21:52:00.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:59 smithi142 conmon[20125]: 24T21:51:58.724497+0000 mon.a ( 2021-03-24T21:52:00.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:51:59 smithi142 conmon[20125]: mon.0) 517 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:52:00.156 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:59 smithi125 conmon[19936]: cluster 2021-03-24T21:51:58. 2021-03-24T21:52:00.157 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:59 smithi125 conmon[19936]: 638399+0000 mgr.y (mgr.14144) 205 : cluster [DBG] pgmap v207: 132 pgs: 132 active+clean; 25 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 22 KiB/s wr, 220 op/s 2021-03-24T21:52:00.157 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:59 smithi125 conmon[19936]: audit 2021-03-24T21:51:58.716299+0000 mgr.y (mgr 2021-03-24T21:52:00.157 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:59 smithi125 conmon[19936]: .14144) 206 : audit [DBG] from='client.24406 v1:172.21.15.142:0/3546318051' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi125=a", "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:52:00.157 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:59 smithi125 conmon[19936]: cephadm 2021-03-24 2021-03-24T21:52:00.158 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:59 smithi125 conmon[19936]: T21:51:58.717681+0000 mgr.y (mgr. 2021-03-24T21:52:00.158 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:59 smithi125 conmon[19936]: 14144) 207 : cephadm [INF] Saving service alertmanager spec with placement smithi125=a;count:1 2021-03-24T21:52:00.158 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:59 smithi125 conmon[19936]: audit 2021-03-24T21:52:00.158 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:59 smithi125 conmon[19936]: 2021-03- 2021-03-24T21:52:00.159 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:59 smithi125 conmon[19936]: 24T21:51:58.724497+0000 mon.a 2021-03-24T21:52:00.159 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:59 smithi125 conmon[19936]: (mon.0) 517 : audit 2021-03-24T21:52:00.159 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:51:59 smithi125 conmon[19936]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:52:00.159 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:59 smithi125 conmon[24927]: cluster 2021-03-24T21:52:00.160 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:59 smithi125 conmon[24927]: 2021-03-24T21 2021-03-24T21:52:00.160 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:59 smithi125 conmon[24927]: :51:58.638399+0000 mgr.y (mgr.14144) 205 : cluster [DBG] pgmap v207: 132 pgs: 132 active+clean; 25 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 22 KiB/s wr, 220 op/s 2021-03-24T21:52:00.160 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:59 smithi125 conmon[24927]: audit 2021 2021-03-24T21:52:00.161 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:59 smithi125 conmon[24927]: -03-24T21:51:58.716299+0000 2021-03-24T21:52:00.161 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:59 smithi125 conmon[24927]: mgr.y (mgr.14144) 206 : audit [DBG] from='client.24406 v1:172.21.15.142:0/3546318051' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi125=a", "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:52:00.161 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:59 smithi125 conmon[24927]: cephadm 2021-03-24T 2021-03-24T21:52:00.161 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:59 smithi125 conmon[24927]: 21:51:58.717681+0000 mgr.y (mgr.14144) 2021-03-24T21:52:00.162 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:59 smithi125 conmon[24927]: 207 : cephadm [INF] Saving service alertmanager spec with placement smithi125=a;count:1 2021-03-24T21:52:00.162 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:59 smithi125 conmon[24927]: audit 2021-03-24T21: 2021-03-24T21:52:00.162 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:59 smithi125 conmon[24927]: 51:58.724497+0000 mon.a (mon.0) 517 2021-03-24T21:52:00.162 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:51:59 smithi125 conmon[24927]: : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:52:01.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:01 smithi142 conmon[20125]: cluster 2021-03-24T21:52:00.022174+0000 mon.a (mon. 2021-03-24T21:52:01.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:01 smithi142 conmon[20125]: 0) 518 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2021-03-24T21:52:01.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:01 smithi125 conmon[24927]: cluster 2021-03-24T21:52 2021-03-24T21:52:01.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:01 smithi125 conmon[24927]: :00.022174+0000 mon.a (mon.0) 518 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2021-03-24T21:52:01.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:01 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:52:01.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:01 smithi125 conmon[19936]: 21:52:00.022174+0000 mon.a (mon.0) 518 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2021-03-24T21:52:01.435 DEBUG:teuthology.orchestra.run.smithi125:alertmanager.a> sudo journalctl -f -n 0 -u ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c@alertmanager.a.service 2021-03-24T21:52:01.452 INFO:tasks.cephadm:Adding grafana.a on smithi142 2021-03-24T21:52:01.452 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph orch apply grafana '1;smithi142=a' 2021-03-24T21:52:01.470 INFO:journalctl@ceph.alertmanager.a.smithi125.stdout:-- Logs begin at Wed 2021-03-03 23:51:07 UTC. -- 2021-03-24T21:52:02.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:02 smithi142 conmon[20125]: cluster 2021-03-24T21 2021-03-24T21:52:02.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:02 smithi142 conmon[20125]: :52:00.639140+0000 mgr.y (mgr.14144) 208 : cluster [DBG] pgmap v209: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 14 KiB/s wr, 197 op/s 2021-03-24T21:52:02.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:02 smithi125 conmon[19936]: cluster 2021-03-24T21:52:00 2021-03-24T21:52:02.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:02 smithi125 conmon[19936]: .639140+0000 mgr.y (mgr.14144) 208 : cluster [DBG] pgmap v209: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 14 KiB/s wr, 197 op/s 2021-03-24T21:52:02.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:02 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:52:02.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:02 smithi125 conmon[24927]: 52:00.639140+0000 mgr.y (mgr.14144) 208 : cluster [DBG] pgmap v209: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 14 KiB/s wr, 197 op/s 2021-03-24T21:52:04.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:04 smithi142 conmon[20125]: cluster 2021-03-24T21:52:02.639929+0000 mgr.y (mgr.14144) 209 : cluster [DBG] pgmap v210: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 3.5 KiB/s wr, 117 op/s 2021-03-24T21:52:04.365 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:04 smithi142 conmon[20125]: 2021-03-24T21:52:04.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:04 smithi125 conmon[19936]: cluster 2021-03-24T21:52:02 2021-03-24T21:52:04.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:04 smithi125 conmon[19936]: .639929+0000 mgr.y (mgr.14144) 209 : cluster [DBG] pgmap v210: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 3.5 KiB/s wr, 117 op/s 2021-03-24T21:52:04.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:04 smithi125 conmon[24927]: cluster 2021-03-24T21:52:02. 2021-03-24T21:52:04.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:04 smithi125 conmon[24927]: 639929+0000 mgr.y (mgr.14144) 209 : cluster [DBG] pgmap v210: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 3.5 KiB/s wr, 117 op/s 2021-03-24T21:52:05.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:05 smithi142 conmon[20125]: audit 2021-03-24T21:52:04.592909+0000 mon.a (mon.0 2021-03-24T21:52:05.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:05 smithi142 conmon[20125]: ) 519 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2021-03-24T21:52:05.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:05 smithi142 conmon[20125]: audit 2021-03-24T21:52:04.593582+0000 mon.a (mon. 2021-03-24T21:52:05.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:05 smithi142 conmon[20125]: 0) 520 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2021-03-24T21:52:05.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:05 smithi142 conmon[20125]: audit 2021 2021-03-24T21:52:05.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:05 smithi142 conmon[20125]: -03-24T21:52:04. 2021-03-24T21:52:05.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:05 smithi142 conmon[20125]: 594019+0000 mon.a (mon.0 2021-03-24T21:52:05.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:05 smithi142 conmon[20125]: ) 521 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2021-03-24T21:52:05.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:05 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:52:05.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:05 smithi142 conmon[20125]: 03-24T21:52:04. 2021-03-24T21:52:05.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:05 smithi142 conmon[20125]: 734094+0000 mon.a (mon.0 2021-03-24T21:52:05.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:05 smithi142 conmon[20125]: ) 522 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T21:52:05.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:05 smithi142 conmon[20125]: 2021-03-24T21:52:05.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:05 smithi142 conmon[20125]: audit 2021-03-24 2021-03-24T21:52:05.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:05 smithi142 conmon[20125]: T21:52:04.738650+0000 2021-03-24T21:52:05.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:05 smithi142 conmon[20125]: mon.a (mon.0) 523 2021-03-24T21:52:05.373 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:05 smithi142 conmon[20125]: : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T21:52:05.373 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:05 smithi142 conmon[20125]: audit 2021-03-24T21:52:05.373 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:05 smithi142 conmon[20125]: 2021-03-24T21:52 2021-03-24T21:52:05.374 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:05 smithi142 conmon[20125]: :05.024880+0000 mon.a ( 2021-03-24T21:52:05.374 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:05 smithi142 conmon[20125]: mon.0) 524 : audit [INF] 2021-03-24T21:52:05.375 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:05 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2021-03-24T21:52:05.375 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:05 smithi142 conmon[20125]: audit 2021-03 2021-03-24T21:52:05.376 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:05 smithi142 conmon[20125]: -24T21:52:05 2021-03-24T21:52:05.376 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:05 smithi142 conmon[20125]: .025074+0000 mon.a (mon. 2021-03-24T21:52:05.376 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:05 smithi142 conmon[20125]: 0) 525 : audit [INF] 2021-03-24T21:52:05.377 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:05 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]': finished 2021-03-24T21:52:05.377 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:05 smithi142 conmon[20125]: audit 2021-03-24T21 2021-03-24T21:52:05.378 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:05 smithi142 conmon[20125]: :52:05.025210+0000 mon.a (mon.0) 526 : 2021-03-24T21:52:05.378 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:05 smithi142 conmon[20125]: audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2021-03-24T21:52:05.378 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:05 smithi142 conmon[20125]: cluster 2021-03-24T21:52:05.025295 2021-03-24T21:52:05.379 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:05 smithi142 conmon[20125]: +0000 mon.a (mon.0) 527 : cluster 2021-03-24T21:52:05.379 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:05 smithi142 conmon[20125]: [DBG] osdmap e72: 8 total, 8 up, 8 in 2021-03-24T21:52:05.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:05 smithi125 conmon[19936]: audit 2021-03-24T21:52:04.592909+0000 mon.a (mon.0) 519 : 2021-03-24T21:52:05.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:05 smithi125 conmon[19936]: audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2021-03-24T21:52:05.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:05 smithi125 conmon[19936]: audit 2021-03-24T21:52:04.593582+0000 mon.a (mon.0) 520 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2021-03-24T21:52:05.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:05 smithi125 conmon[19936]: audit 2021-03-24T21:52:04.594019+0000 2021-03-24T21:52:05.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:05 smithi125 conmon[19936]: mon.a (mon.0) 521 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2021-03-24T21:52:05.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:05 smithi125 conmon[19936]: audit 2021-03-24T21 2021-03-24T21:52:05.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:05 smithi125 conmon[19936]: :52:04.734094+0000 mon.a (mon.0) 522 : audit [INF] 2021-03-24T21:52:05.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:05 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T21:52:05.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:05 smithi125 conmon[19936]: audit 2021-03-24T21:52:04 2021-03-24T21:52:05.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:05 smithi125 conmon[19936]: .738650+0000 mon.a (mon.0) 523 : 2021-03-24T21:52:05.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:05 smithi125 conmon[19936]: audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T21:52:05.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:05 smithi125 conmon[19936]: audit 2021-03-24T21: 2021-03-24T21:52:05.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:05 smithi125 conmon[19936]: 52:05.024880+0000 mon.a (mon.0) 2021-03-24T21:52:05.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:05 smithi125 conmon[19936]: 524 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2021-03-24T21:52:05.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:05 smithi125 conmon[19936]: audit 2021-03-24T21:52:05.025074+0000 mon.a (mon. 2021-03-24T21:52:05.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:05 smithi125 conmon[19936]: 0) 525 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]': finished 2021-03-24T21:52:05.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:05 smithi125 conmon[19936]: audit 2021-03-24T21:52:05.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:05 smithi125 conmon[19936]: 2021-03-24T21:52:05.025210+0000 mon.a (mon.0) 526 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2021-03-24T21:52:05.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:05 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:52:05.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:05 smithi125 conmon[19936]: -03-24T21:52:05.025295+0000 mon.a ( 2021-03-24T21:52:05.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:05 smithi125 conmon[19936]: mon.0) 527 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2021-03-24T21:52:05.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:05 smithi125 conmon[24927]: audit 2021-03-24T21:52:04. 2021-03-24T21:52:05.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:05 smithi125 conmon[24927]: 592909+0000 mon.a (mon.0) 519 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2021-03-24T21:52:05.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:05 smithi125 conmon[24927]: audit 2021-03-24T21:52:05.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:05 smithi125 conmon[24927]: 2021-03-24T21:52:04.593582+0000 mon.a (mon.0) 520 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2021-03-24T21:52:05.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:05 smithi125 conmon[24927]: audit 2021-03-24 2021-03-24T21:52:05.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:05 smithi125 conmon[24927]: T21:52:04.594019+0000 mon.a (mon.0) 2021-03-24T21:52:05.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:05 smithi125 conmon[24927]: 521 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2021-03-24T21:52:05.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:05 smithi125 conmon[24927]: audit 2021-03 2021-03-24T21:52:05.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:05 smithi125 conmon[24927]: -24T21:52:04.734094+0000 mon.a ( 2021-03-24T21:52:05.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:05 smithi125 conmon[24927]: mon.0) 522 : audit [INF] 2021-03-24T21:52:05.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:05 smithi125 conmon[24927]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T21:52:05.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:05 smithi125 conmon[24927]: audit 2021-03-24T21:52 2021-03-24T21:52:05.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:05 smithi125 conmon[24927]: :04.738650+0000 mon.a (mon.0) 523 : audit [INF] 2021-03-24T21:52:05.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:05 smithi125 conmon[24927]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T21:52:05.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:05 smithi125 conmon[24927]: audit 2021-03-24T21:52:05. 2021-03-24T21:52:05.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:05 smithi125 conmon[24927]: 024880+0000 mon.a (mon.0) 524 : audit [INF] 2021-03-24T21:52:05.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:05 smithi125 conmon[24927]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2021-03-24T21:52:05.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:05 smithi125 conmon[24927]: audit 2021 2021-03-24T21:52:05.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:05 smithi125 conmon[24927]: -03-24T21:52:05.025074+0000 mon.a (mon.0) 525 : audit [INF] 2021-03-24T21:52:05.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:05 smithi125 conmon[24927]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]': finished 2021-03-24T21:52:05.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:05 smithi125 conmon[24927]: audit 2021-03 2021-03-24T21:52:05.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:05 smithi125 conmon[24927]: -24T21:52:05.025210+0000 mon.a (mon. 2021-03-24T21:52:05.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:05 smithi125 conmon[24927]: 0) 526 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2021-03-24T21:52:05.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:05 smithi125 conmon[24927]: cluster 2021-03-24T21:52:05 2021-03-24T21:52:05.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:05 smithi125 conmon[24927]: .025295+0000 mon.a (mon.0) 527 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2021-03-24T21:52:05.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:05 smithi125 conmon[24927]: 2021-03-24T21:52:06.115 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:06 smithi142 conmon[20125]: cluster 2021-03-24T21:52:06.115 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:06 smithi142 conmon[20125]: 2021-03-24T21:52:06.116 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:06 smithi142 conmon[20125]: 2021-03-24T21: 2021-03-24T21:52:06.116 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:06 smithi142 conmon[20125]: 52:04.641179+0000 mgr.y (mgr.14144) 210 : cluster [DBG] pgmap v211: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 2.9 KiB/s wr, 98 op/s 2021-03-24T21:52:06.116 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:06 smithi142 conmon[20125]: cluster 2021-03-24T21:52: 2021-03-24T21:52:06.116 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:06 smithi142 conmon[20125]: 04.932125+0000 osd.7 (osd.7) 3 : cluster [DBG] 1.0 scrub ok 2021-03-24T21:52:06.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:06 smithi125 conmon[19936]: cluster 2021-03-24T21:52:06.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:06 smithi125 conmon[19936]: 2021-03-24T21:52:06.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:06 smithi125 conmon[19936]: 2021 2021-03-24T21:52:06.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:06 smithi125 conmon[19936]: -03-24T21:52:04.641179+0000 mgr.y (mgr 2021-03-24T21:52:06.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:06 smithi125 conmon[19936]: .14144) 210 : cluster [DBG] pgmap v211: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 2.9 KiB/s wr, 98 op/s 2021-03-24T21:52:06.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:06 smithi125 conmon[19936]: cluster 2021-03-24T21:52:06.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:06 smithi125 conmon[19936]: 2021-03-24T21:52:04.932125+0000 osd.7 2021-03-24T21:52:06.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:06 smithi125 conmon[19936]: (osd.7) 3 : cluster [DBG] 1.0 scrub ok 2021-03-24T21:52:06.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:06 smithi125 conmon[24927]: cluster 2021-03-24T21:52:04.641179+0000 mgr.y 2021-03-24T21:52:06.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:06 smithi125 conmon[24927]: (mgr.14144) 210 : cluster [DBG] pgmap v211: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 2.9 KiB/s wr, 98 op/s 2021-03-24T21:52:06.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:06 smithi125 conmon[24927]: cluster 2021-03-24T21:52:04.932125+0000 osd.7 (osd.7) 3 : cluster [DBG] 1.0 scrub ok 2021-03-24T21:52:07.041 INFO:teuthology.orchestra.run.smithi142.stdout:Scheduled grafana update... 2021-03-24T21:52:07.145 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:07 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:52:07.239 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:07 smithi142 conmon[20125]: T21:52:06.047124+0000 mon.a (mon.0) 528 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2021-03-24T21:52:07.239 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:07 smithi142 conmon[20125]: audit 2021-03-24 2021-03-24T21:52:07.239 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:07 smithi142 conmon[20125]: T21:52:07.042847+0000 mon.a 2021-03-24T21:52:07.240 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:07 smithi142 conmon[20125]: (mon.0) 529 : audit 2021-03-24T21:52:07.240 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:07 smithi142 conmon[20125]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:52:07.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:07 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:52:07.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:07 smithi125 conmon[19936]: 21:52:06.047124+0000 mon.a (mon.0) 528 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2021-03-24T21:52:07.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:07 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:52:07.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:07 smithi125 conmon[19936]: 03-24T21:52:07.042847+0000 mon.a ( 2021-03-24T21:52:07.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:07 smithi125 conmon[19936]: mon.0) 529 : audit [INF] 2021-03-24T21:52:07.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:07 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:52:07.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:07 smithi125 conmon[24927]: cluster 2021-03-24T21:52:07.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:07 smithi125 conmon[24927]: 2021-03- 2021-03-24T21:52:07.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:07 smithi125 conmon[24927]: 24T21:52:06.047124+0000 mon.a (mon.0) 528 : cluster 2021-03-24T21:52:07.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:07 smithi125 conmon[24927]: [DBG] osdmap e73: 8 total, 8 up, 8 in 2021-03-24T21:52:07.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:07 smithi125 conmon[24927]: audit 2021 2021-03-24T21:52:07.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:07 smithi125 conmon[24927]: -03-24T21:52:07. 2021-03-24T21:52:07.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:07 smithi125 conmon[24927]: 042847+0000 mon.a (mon.0) 529 2021-03-24T21:52:07.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:07 smithi125 conmon[24927]: : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:52:08.261 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:08 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:52:08.261 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:08 smithi142 conmon[20125]: 52:06.641852+0000 mgr.y (mgr.14144) 211 : cluster [DBG] pgmap v214: 132 pgs: 1 peering, 131 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 772 B/s rd, 0 op/s; 16 B/s, 1 objects/s recovering 2021-03-24T21:52:08.261 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:08 smithi142 conmon[20125]: audit 2021-03- 2021-03-24T21:52:08.261 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:08 smithi142 conmon[20125]: 24T21:52:07.035529+0000 mgr.y (mgr.14144) 212 2021-03-24T21:52:08.262 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:08 smithi142 conmon[20125]: : audit [DBG] from='client.24412 v1:172.21.15.142:0/1195521755' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi142=a", "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:52:08.262 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:08 smithi142 conmon[20125]: cephadm 2021 2021-03-24T21:52:08.262 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:08 smithi142 conmon[20125]: -03-24T21:52: 2021-03-24T21:52:08.262 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:08 smithi142 conmon[20125]: 07.036788+0000 mgr.y ( 2021-03-24T21:52:08.263 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:08 smithi142 conmon[20125]: mgr.14144) 213 : 2021-03-24T21:52:08.263 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:08 smithi142 conmon[20125]: cephadm [INF] Saving service grafana spec with placement smithi142=a;count:1 2021-03-24T21:52:08.390 DEBUG:teuthology.orchestra.run.smithi142:grafana.a> sudo journalctl -f -n 0 -u ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c@grafana.a.service 2021-03-24T21:52:08.393 INFO:tasks.cephadm:Setting up client nodes... 2021-03-24T21:52:08.393 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2021-03-24T21:52:08.403 INFO:journalctl@ceph.grafana.a.smithi142.stdout:-- Logs begin at Wed 2021-03-03 23:51:07 UTC. -- 2021-03-24T21:52:08.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:08 smithi125 conmon[19936]: cluster 2021-03-24T21:52:06.641852+0000 2021-03-24T21:52:08.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:08 smithi125 conmon[19936]: mgr.y (mgr.14144) 211 : cluster [DBG] pgmap v214: 132 pgs: 1 peering, 131 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 772 B/s rd, 0 op/s; 16 B/s, 1 objects/s recovering 2021-03-24T21:52:08.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:08 smithi125 conmon[19936]: audit 2021-03-24T21:52:07.035529+0000 mgr.y (mgr.14144) 212 : audit [DBG] from='client.24412 v1:172.21.15.142:0/1195521755' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi142=a", "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:52:08.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:08 smithi125 conmon[19936]: cephadm 2021-03-24T21:52:07.036788+0000 mgr.y (mgr.14144) 213 : cephadm [INF] Saving service grafana spec with placement smithi142=a;count:1 2021-03-24T21:52:08.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:08 smithi125 conmon[24927]: cluster 2021-03-24T21:52:06 2021-03-24T21:52:08.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:08 smithi125 conmon[24927]: .641852+0000 mgr.y (mgr.14144) 211 : cluster [DBG] pgmap v214: 132 pgs: 1 peering, 131 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 772 B/s rd, 0 op/s; 16 B/s, 1 objects/s recovering 2021-03-24T21:52:08.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:08 smithi125 conmon[24927]: audit 2021-03-24T21:52:08.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:08 smithi125 conmon[24927]: 2021 2021-03-24T21:52:08.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:08 smithi125 conmon[24927]: -03-24T21:52 2021-03-24T21:52:08.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:08 smithi125 conmon[24927]: :07.035529+0000 mgr.y (mgr.14144) 212 : audit [DBG] from='client.24412 v1:172.21.15.142:0/1195521755' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi142=a", "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:52:08.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:08 smithi125 conmon[24927]: cephadm 2021- 2021-03-24T21:52:08.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:08 smithi125 conmon[24927]: 03-24T21:52:07.036788+0000 mgr.y (mgr.14144 2021-03-24T21:52:08.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:08 smithi125 conmon[24927]: ) 213 : cephadm [INF] Saving service grafana spec with placement smithi142=a;count:1 2021-03-24T21:52:08.865 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:52:08 smithi142 systemd[1]: Starting Ceph prometheus.a for 3cfa339a-8cea-11eb-90b2-001a4aab830c... 2021-03-24T21:52:09.335 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:09 smithi142 conmon[20125]: cluster 2021-03-24T21:52 2021-03-24T21:52:09.335 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:09 smithi142 conmon[20125]: :07.610397+0000 osd.0 (osd.0) 7 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:52:09.336 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:09 smithi142 conmon[20125]: cluster 2021-03-24T21:52:07 2021-03-24T21:52:09.336 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:09 smithi142 conmon[20125]: .610584+0000 osd.0 (osd.0) 8 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:52:09.336 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:09 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:52:09.336 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:09 smithi142 conmon[20125]: -24T21:52:08.069392+0000 mon.a ( 2021-03-24T21:52:09.337 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:09 smithi142 conmon[20125]: mon.0) 530 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2021-03-24T21:52:09.337 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:52:09 smithi142 podman[38367]: 2021-03-24 21:52:09.145362422 +0000 UTC m=+0.197331165 container create e182b9c92b75659c3f661b7f31111701b62b50ddc70fa69a90f215597f57669d (image=docker.io/prom/prometheus:v2.18.1, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-prometheus.a, maintainer=The Prometheus Authors ) 2021-03-24T21:52:09.337 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:52:09 smithi142 podman[38367]: 2021-03-24 21:52:09.28704477 +0000 UTC m=+0.339013513 container init e182b9c92b75659c3f661b7f31111701b62b50ddc70fa69a90f215597f57669d (image=docker.io/prom/prometheus:v2.18.1, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-prometheus.a, maintainer=The Prometheus Authors ) 2021-03-24T21:52:09.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:09 smithi125 conmon[24927]: cluster 2021-03-24T21:52:07.610397+0000 osd.0 (osd. 2021-03-24T21:52:09.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:09 smithi125 conmon[24927]: 0) 7 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:52:09.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:09 smithi125 conmon[24927]: cluster 2021-03-24T21:52:07.610584+0000 osd.0 (osd 2021-03-24T21:52:09.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:09 smithi125 conmon[24927]: .0) 8 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:52:09.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:09 smithi125 conmon[24927]: cluster 2021-03-24T21:52:09.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:09 smithi125 conmon[24927]: 2021-03-24T21:52:08.069392+0000 mon.a 2021-03-24T21:52:09.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:09 smithi125 conmon[24927]: (mon.0) 530 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2021-03-24T21:52:09.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:09 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:52:09.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:09 smithi125 conmon[19936]: :52:07.610397+0000 osd.0 (osd.0) 7 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:52:09.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:09 smithi125 conmon[19936]: cluster 2021-03-24T21:52:09.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:09 smithi125 conmon[19936]: 2021-03-24T21:52:07.610584 2021-03-24T21:52:09.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:09 smithi125 conmon[19936]: +0000 osd.0 (osd.0) 8 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:52:09.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:09 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:52:09.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:09 smithi125 conmon[19936]: 03-24T21:52:08.069392+0000 mon.a ( 2021-03-24T21:52:09.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:09 smithi125 conmon[19936]: mon.0) 530 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2021-03-24T21:52:09.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:09 smithi125 conmon[19936]: 2021-03-24T21:52:09.616 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:52:09 smithi142 podman[38367]: 2021-03-24 21:52:09.337056602 +0000 UTC m=+0.389025394 container start e182b9c92b75659c3f661b7f31111701b62b50ddc70fa69a90f215597f57669d (image=docker.io/prom/prometheus:v2.18.1, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-prometheus.a, maintainer=The Prometheus Authors ) 2021-03-24T21:52:09.616 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:52:09 smithi142 bash[38367]: e182b9c92b75659c3f661b7f31111701b62b50ddc70fa69a90f215597f57669d 2021-03-24T21:52:09.616 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:52:09 smithi142 conmon[38407]: level=info ts=2021-03-24T21:52:09.341Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2021-03-24T21:52:09.616 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:52:09 smithi142 conmon[38407]: level=info ts=2021-03-24T21:52:09.341Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2021-03-24T21:52:09.617 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:52:09 smithi142 conmon[38407]: level=info ts=2021-03-24T21:52:09.341Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2021-03-24T21:52:09.617 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:52:09 smithi142 conmon[38407]: level=info ts=2021-03-24T21:52:09.341Z caller=main.go:339 host_details="(Linux 5.4.0-70-generic #78-Ubuntu SMP Fri Mar 19 13:29:52 UTC 2021 x86_64 smithi142 (none))" 2021-03-24T21:52:09.617 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:52:09 smithi142 conmon[38407]: level=info ts=2021-03-24T21:52:09.341Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2021-03-24T21:52:09.617 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:52:09 smithi142 conmon[38407]: level=info ts=2021-03-24T21:52:09.341Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2021-03-24T21:52:09.618 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:52:09 smithi142 conmon[38407]: level=info ts=2021-03-24T21:52:09.342Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2021-03-24T21:52:09.618 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:52:09 smithi142 conmon[38407]: level=info ts=2021-03-24T21:52:09.342Z caller=main.go:678 msg="Starting TSDB ..." 2021-03-24T21:52:09.618 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:52:09 smithi142 systemd[1]: Started Ceph prometheus.a for 3cfa339a-8cea-11eb-90b2-001a4aab830c. 2021-03-24T21:52:09.618 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:52:09 smithi142 conmon[38407]: level=info ts=2021-03-24T21:52:09.348Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2021-03-24T21:52:09.618 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:52:09 smithi142 conmon[38407]: level=info ts=2021-03-24T21:52:09.348Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2021-03-24T21:52:09.619 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:52:09 smithi142 conmon[38407]: level=info ts=2021-03-24T21:52:09.348Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=275.877µs 2021-03-24T21:52:09.619 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:52:09 smithi142 conmon[38407]: level=info ts=2021-03-24T21:52:09.350Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2021-03-24T21:52:09.619 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:52:09 smithi142 conmon[38407]: level=info ts=2021-03-24T21:52:09.350Z caller=main.go:695 msg="TSDB started" 2021-03-24T21:52:09.619 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:52:09 smithi142 conmon[38407]: level=info ts=2021-03-24T21:52:09.350Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2021-03-24T21:52:09.620 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:52:09 smithi142 conmon[38407]: level=info ts=2021-03-24T21:52:09.351Z caller=main.go:827 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml 2021-03-24T21:52:09.620 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:52:09 smithi142 conmon[38407]: level=info ts=2021-03-24T21:52:09.351Z caller=main.go:646 msg="Server is ready to receive web requests." 2021-03-24T21:52:10.522 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:10 smithi125 conmon[24927]: cluster 2021-03-24T21:52 2021-03-24T21:52:10.523 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:10 smithi125 conmon[24927]: :08.642527+0000 mgr.y (mgr.14144) 214 : cluster [DBG] pgmap v216: 132 pgs: 1 peering, 131 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 18 B/s, 1 objects/s recovering 2021-03-24T21:52:10.523 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:10 smithi125 conmon[24927]: audit 2021-03-24T21: 2021-03-24T21:52:10.523 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:10 smithi125 conmon[24927]: 52:09.378050+0000 mon.a (mon.0) 531 : audit [INF] 2021-03-24T21:52:10.524 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:10 smithi125 conmon[24927]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:52:10.524 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:10 smithi125 conmon[24927]: cephadm 2021-03-24 2021-03-24T21:52:10.524 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:10 smithi125 conmon[24927]: T21:52:09.380745+0000 mgr.y (mgr.14144) 215 : cephadm 2021-03-24T21:52:10.524 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:10 smithi125 conmon[24927]: [INF] Deploying daemon node-exporter.a on smithi125 2021-03-24T21:52:10.525 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:10 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:52:10.525 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:10 smithi125 conmon[19936]: 24T21:52:08.642527+0000 mgr.y (mgr.14144) 214 : cluster [DBG] pgmap v216: 132 pgs: 1 peering, 131 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 18 B/s, 1 objects/s recovering 2021-03-24T21:52:10.525 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:10 smithi125 conmon[19936]: audit 2021-03-24T21:52:10.525 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:10 smithi125 conmon[19936]: 2021-03-24T21:52:09. 2021-03-24T21:52:10.525 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:10 smithi125 conmon[19936]: 378050+0000 mon.a (mon.0) 531 : audit 2021-03-24T21:52:10.526 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:10 smithi125 conmon[19936]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:52:10.526 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:10 smithi125 conmon[19936]: cephadm 2021-03-24T21: 2021-03-24T21:52:10.526 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:10 smithi125 conmon[19936]: 52:09.380745+0000 mgr.y (mgr.14144 2021-03-24T21:52:10.526 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:10 smithi125 conmon[19936]: ) 215 : cephadm [INF] Deploying daemon node-exporter.a on smithi125 2021-03-24T21:52:10.865 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:10 smithi142 conmon[20125]: cluster 2021-03-24T21:52:08.642527 2021-03-24T21:52:10.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:10 smithi142 conmon[20125]: +0000 mgr.y (mgr.14144) 214 : cluster [DBG] pgmap v216: 132 pgs: 1 peering, 131 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 18 B/s, 1 objects/s recovering 2021-03-24T21:52:10.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:10 smithi142 conmon[20125]: audit 2021-03-24T21:52:09 2021-03-24T21:52:10.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:10 smithi142 conmon[20125]: .378050+0000 mon.a (mon.0) 531 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:52:10.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:10 smithi142 conmon[20125]: cephadm 2021-03-24T 2021-03-24T21:52:10.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:10 smithi142 conmon[20125]: 21:52:09.380745+0000 mgr.y (mgr.14144 2021-03-24T21:52:10.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:10 smithi142 conmon[20125]: ) 215 : cephadm [INF] Deploying daemon node-exporter.a on smithi125 2021-03-24T21:52:10.907 INFO:journalctl@ceph.node-exporter.a.smithi125.stdout:Mar 24 21:52:10 smithi125 bash[41106]: Trying to pull docker.io/prom/node-exporter:v0.18.1... 2021-03-24T21:52:10.963 INFO:teuthology.orchestra.run.smithi125.stdout:[client.0] 2021-03-24T21:52:11.014 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQCKtFtgp90WORAAwGT0G0CIScz9ns6fiurStw== 2021-03-24T21:52:11.382 INFO:journalctl@ceph.node-exporter.a.smithi125.stdout:Mar 24 21:52:11 smithi125 bash[41106]: Getting image source signatures 2021-03-24T21:52:11.496 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:11 smithi125 conmon[19936]: audit 2021-03-24T21:52:10. 2021-03-24T21:52:11.497 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:11 smithi125 conmon[19936]: 956551+0000 mon.c (mon.2) 10 : audit [INF] from='client.? v1:172.21.15.125:0/3274162431' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-03-24T21:52:11.497 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:11 smithi125 conmon[19936]: audit 2021-03-24T21:52:10 2021-03-24T21:52:11.497 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:11 smithi125 conmon[19936]: .957453+0000 mon.a (mon.0) 532 : audit [INF] from='client.34415 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-03-24T21:52:11.497 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:11 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:52:11.498 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:11 smithi125 conmon[19936]: 03-24T21:52:10.964499+0000 mon.a 2021-03-24T21:52:11.498 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:11 smithi125 conmon[19936]: (mon.0) 533 : audit [INF] from='client.34415 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2021-03-24T21:52:11.498 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:11 smithi125 conmon[24927]: audit 2021-03-24T21:52 2021-03-24T21:52:11.499 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:11 smithi125 conmon[24927]: :10.956551+0000 mon.c (mon.2) 10 : audit [INF] from='client.? v1:172.21.15.125:0/3274162431' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-03-24T21:52:11.499 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:11 smithi125 conmon[24927]: audit 2021 2021-03-24T21:52:11.499 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:11 smithi125 conmon[24927]: -03-24T21:52:10.957453+0000 mon.a ( 2021-03-24T21:52:11.499 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:11 smithi125 conmon[24927]: mon.0) 532 : audit [INF] from='client.34415 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-03-24T21:52:11.499 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:11 smithi125 conmon[24927]: audit 2021-03-24T21:52:11.500 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:11 smithi125 conmon[24927]: 2021-03-24T21:52:10.964499+0000 2021-03-24T21:52:11.500 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:11 smithi125 conmon[24927]: mon.a (mon.0) 533 : audit [INF] from='client.34415 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2021-03-24T21:52:11.865 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:11 smithi142 conmon[20125]: audit 2021-03-24T21:52:10.956551+0000 mon.c (mon.2) 10 : 2021-03-24T21:52:11.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:11 smithi142 conmon[20125]: audit [INF] from='client.? v1:172.21.15.125:0/3274162431' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-03-24T21:52:11.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:11 smithi142 conmon[20125]: audit 2021-03-24T21:52:10.957453+0000 mon.a (mon.0) 532 : 2021-03-24T21:52:11.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:11 smithi142 conmon[20125]: audit [INF] from='client.34415 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-03-24T21:52:11.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:11 smithi142 conmon[20125]: audit 2021 2021-03-24T21:52:11.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:11 smithi142 conmon[20125]: -03-24T21:52:10.964499+0000 mon.a (mon 2021-03-24T21:52:11.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:11 smithi142 conmon[20125]: .0) 533 : audit [INF] from='client.34415 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2021-03-24T21:52:11.911 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2021-03-24T21:52:11.911 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2021-03-24T21:52:11.912 DEBUG:teuthology.orchestra.run.smithi125:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2021-03-24T21:52:11.966 INFO:journalctl@ceph.node-exporter.a.smithi125.stdout:Mar 24 21:52:11 smithi125 bash[41106]: Copying blob sha256:3589a6efd9ce96b2f2218a80bacb84f60d5e584d024023617b0f56b87fe798c3 2021-03-24T21:52:11.966 INFO:journalctl@ceph.node-exporter.a.smithi125.stdout:Mar 24 21:52:11 smithi125 bash[41106]: Copying blob sha256:190160031744b0810999dd6f293478be0b41524654c70782cf09437095430f2f 2021-03-24T21:52:11.966 INFO:journalctl@ceph.node-exporter.a.smithi125.stdout:Mar 24 21:52:11 smithi125 bash[41106]: Copying blob sha256:49a2d53aa1af115e87f7e23a7fe4b7a3e12304d37982fb6528b035d0f68a3b56 2021-03-24T21:52:11.979 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2021-03-24T21:52:12.225 INFO:journalctl@ceph.node-exporter.a.smithi125.stdout:Mar 24 21:52:12 smithi125 bash[41106]: Copying config sha256:e5a616e4b9cf68dfcad7782b78e118be4310022e874d52da85c55923fb615f87 2021-03-24T21:52:12.226 INFO:journalctl@ceph.node-exporter.a.smithi125.stdout:Mar 24 21:52:12 smithi125 bash[41106]: Writing manifest to image destination 2021-03-24T21:52:12.657 INFO:journalctl@ceph.node-exporter.a.smithi125.stdout:Mar 24 21:52:12 smithi125 bash[41106]: Storing signatures 2021-03-24T21:52:12.657 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:12 smithi125 conmon[19936]: cluster 2021-03-24T21:52:12.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:12 smithi125 conmon[19936]: 2021-03-24T21:52:10.643663+0000 mgr.y (mgr.14144) 216 : cluster 2021-03-24T21:52:12.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:12 smithi125 conmon[19936]: [DBG] pgmap v217: 132 pgs: 1 peering, 131 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 226 B/s, 2 objects/s recovering 2021-03-24T21:52:12.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:12 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:52:12.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:12 smithi125 conmon[19936]: -03-24T21:52:11.677556+0000 osd.3 2021-03-24T21:52:12.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:12 smithi125 conmon[19936]: (osd.3) 5 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:52:12.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:12 smithi125 conmon[19936]: cluster 2021-03-24T21:52:12.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:12 smithi125 conmon[19936]: 2021-03-24T21:52:11.677770 2021-03-24T21:52:12.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:12 smithi125 conmon[19936]: +0000 osd.3 (osd.3) 6 : cluster [DBG] 2021-03-24T21:52:12.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:12 smithi125 conmon[19936]: purged_snaps scrub ok 2021-03-24T21:52:12.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:12 smithi125 conmon[24927]: cluster 2021-03-24T21:52: 2021-03-24T21:52:12.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:12 smithi125 conmon[24927]: 10.643663+0000 mgr.y (mgr.14144) 216 : cluster [DBG] pgmap v217: 132 pgs: 1 peering, 131 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 226 B/s, 2 objects/s recovering 2021-03-24T21:52:12.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:12 smithi125 conmon[24927]: cluster 2021-03-24T21:52:11.677556+0000 2021-03-24T21:52:12.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:12 smithi125 conmon[24927]: osd.3 (osd.3) 5 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:52:12.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:12 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:52:12.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:12 smithi125 conmon[24927]: 03-24T21:52:11.677770+0000 osd.3 (osd. 2021-03-24T21:52:12.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:12 smithi125 conmon[24927]: 3) 6 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:52:12.687 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:12 smithi142 conmon[20125]: cluster 2021-03-24T21:52: 2021-03-24T21:52:12.688 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:12 smithi142 conmon[20125]: 10.643663+0000 mgr.y (mgr.14144) 216 : cluster [DBG] pgmap v217: 132 pgs: 1 peering, 131 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 226 B/s, 2 objects/s recovering 2021-03-24T21:52:12.688 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:12 smithi142 conmon[20125]: cluster 2021-03-24T21:52:11 2021-03-24T21:52:12.688 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:12 smithi142 conmon[20125]: .677556+0000 osd.3 (osd.3) 5 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:52:12.688 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:12 smithi142 conmon[20125]: cluster 2021-03-24T21:52:11.677770+0000 2021-03-24T21:52:12.689 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:12 smithi142 conmon[20125]: osd.3 (osd.3) 6 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:52:13.657 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:13 smithi125 conmon[19936]: cluster 2021-03-24T21:52:12.396861+0000 mon.a (mon. 2021-03-24T21:52:13.657 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:13 smithi125 conmon[19936]: 0) 534 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2021-03-24T21:52:13.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:13 smithi125 conmon[24927]: cluster 2021-03-24T21:52 2021-03-24T21:52:13.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:13 smithi125 conmon[24927]: :12.396861+0000 mon.a (mon.0) 534 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2021-03-24T21:52:13.658 INFO:journalctl@ceph.node-exporter.a.smithi125.stdout:Mar 24 21:52:13 smithi125 podman[41106]: 2021-03-24 21:52:13.500335928 +0000 UTC m=+2.754507302 image pull 2021-03-24T21:52:13.659 INFO:journalctl@ceph.node-exporter.a.smithi125.stdout:Mar 24 21:52:13 smithi125 podman[41106]: 2021-03-24 21:52:13.638459333 +0000 UTC m=+2.892630624 container create 16a3b0e1d3edbe295fe2f70e12405bbbbe1271e40904abf6fe7ddb2a4399cf12 (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-node-exporter.a, maintainer=The Prometheus Authors ) 2021-03-24T21:52:13.683 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:13 smithi142 conmon[20125]: cluster 2021-03-24T21:52:12.396861+0000 mon.a (mon.0) 534 : 2021-03-24T21:52:13.683 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:13 smithi142 conmon[20125]: cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2021-03-24T21:52:14.158 INFO:journalctl@ceph.node-exporter.a.smithi125.stdout:Mar 24 21:52:13 smithi125 podman[41106]: 2021-03-24 21:52:13.780114105 +0000 UTC m=+3.034285463 container init 16a3b0e1d3edbe295fe2f70e12405bbbbe1271e40904abf6fe7ddb2a4399cf12 (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-node-exporter.a, maintainer=The Prometheus Authors ) 2021-03-24T21:52:14.158 INFO:journalctl@ceph.node-exporter.a.smithi125.stdout:Mar 24 21:52:13 smithi125 conmon[41241]: time="2021-03-24T21:52:13Z" level=info msg="Starting node_exporter (version=0.18.1, branch=HEAD, revision=3db77732e925c08f675d7404a8c46466b2ece83e)" source="node_exporter.go:156" 2021-03-24T21:52:14.159 INFO:journalctl@ceph.node-exporter.a.smithi125.stdout:Mar 24 21:52:13 smithi125 conmon[41241]: time="2021-03-24T21:52:13Z" level=info msg="Build context (go=go1.12.5, user=root@b50852a1acba, date=20190604-16:41:18)" source="node_exporter.go:157" 2021-03-24T21:52:14.159 INFO:journalctl@ceph.node-exporter.a.smithi125.stdout:Mar 24 21:52:13 smithi125 conmon[41241]: time="2021-03-24T21:52:13Z" level=info msg="Enabled collectors:" source="node_exporter.go:97" 2021-03-24T21:52:14.159 INFO:journalctl@ceph.node-exporter.a.smithi125.stdout:Mar 24 21:52:13 smithi125 conmon[41241]: time="2021-03-24T21:52:13Z" level=info msg=" - arp" source="node_exporter.go:104" 2021-03-24T21:52:14.159 INFO:journalctl@ceph.node-exporter.a.smithi125.stdout:Mar 24 21:52:13 smithi125 conmon[41241]: time="2021-03-24T21:52:13Z" level=info msg=" - bcache" source="node_exporter.go:104" 2021-03-24T21:52:14.159 INFO:journalctl@ceph.node-exporter.a.smithi125.stdout:Mar 24 21:52:13 smithi125 conmon[41241]: time="2021-03-24T21:52:13Z" level=info msg=" - bonding" source="node_exporter.go:104" 2021-03-24T21:52:14.160 INFO:journalctl@ceph.node-exporter.a.smithi125.stdout:Mar 24 21:52:13 smithi125 conmon[41241]: time="2021-03-24T21:52:13Z" level=info msg=" - conntrack" source="node_exporter.go:104" 2021-03-24T21:52:14.160 INFO:journalctl@ceph.node-exporter.a.smithi125.stdout:Mar 24 21:52:13 smithi125 conmon[41241]: time="2021-03-24T21:52:13Z" level=info msg=" - cpu" source="node_exporter.go:104" 2021-03-24T21:52:14.160 INFO:journalctl@ceph.node-exporter.a.smithi125.stdout:Mar 24 21:52:13 smithi125 conmon[41241]: time="2021-03-24T21:52:13Z" level=info msg=" - cpufreq" source="node_exporter.go:104" 2021-03-24T21:52:14.160 INFO:journalctl@ceph.node-exporter.a.smithi125.stdout:Mar 24 21:52:13 smithi125 conmon[41241]: time="2021-03-24T21:52:13Z" level=info msg=" - diskstats" source="node_exporter.go:104" 2021-03-24T21:52:14.161 INFO:journalctl@ceph.node-exporter.a.smithi125.stdout:Mar 24 21:52:13 smithi125 conmon[41241]: time="2021-03-24T21:52:13Z" level=info msg=" - edac" source="node_exporter.go:104" 2021-03-24T21:52:14.161 INFO:journalctl@ceph.node-exporter.a.smithi125.stdout:Mar 24 21:52:13 smithi125 conmon[41241]: time="2021-03-24T21:52:13Z" level=info msg=" - entropy" source="node_exporter.go:104" 2021-03-24T21:52:14.161 INFO:journalctl@ceph.node-exporter.a.smithi125.stdout:Mar 24 21:52:13 smithi125 conmon[41241]: time="2021-03-24T21:52:13Z" level=info msg=" - filefd" source="node_exporter.go:104" 2021-03-24T21:52:14.161 INFO:journalctl@ceph.node-exporter.a.smithi125.stdout:Mar 24 21:52:13 smithi125 conmon[41241]: time="2021-03-24T21:52:13Z" level=info msg=" - filesystem" source="node_exporter.go:104" 2021-03-24T21:52:14.161 INFO:journalctl@ceph.node-exporter.a.smithi125.stdout:Mar 24 21:52:13 smithi125 conmon[41241]: time="2021-03-24T21:52:13Z" level=info msg=" - hwmon" source="node_exporter.go:104" 2021-03-24T21:52:14.162 INFO:journalctl@ceph.node-exporter.a.smithi125.stdout:Mar 24 21:52:13 smithi125 conmon[41241]: time="2021-03-24T21:52:13Z" level=info msg=" - infiniband" source="node_exporter.go:104" 2021-03-24T21:52:14.162 INFO:journalctl@ceph.node-exporter.a.smithi125.stdout:Mar 24 21:52:13 smithi125 conmon[41241]: time="2021-03-24T21:52:13Z" level=info msg=" - ipvs" source="node_exporter.go:104" 2021-03-24T21:52:14.162 INFO:journalctl@ceph.node-exporter.a.smithi125.stdout:Mar 24 21:52:13 smithi125 conmon[41241]: time="2021-03-24T21:52:13Z" level=info msg=" - loadavg" source="node_exporter.go:104" 2021-03-24T21:52:14.163 INFO:journalctl@ceph.node-exporter.a.smithi125.stdout:Mar 24 21:52:13 smithi125 conmon[41241]: time="2021-03-24T21:52:13Z" level=info msg=" - mdadm" source="node_exporter.go:104" 2021-03-24T21:52:14.163 INFO:journalctl@ceph.node-exporter.a.smithi125.stdout:Mar 24 21:52:13 smithi125 conmon[41241]: time="2021-03-24T21:52:13Z" level=info msg=" - meminfo" source="node_exporter.go:104" 2021-03-24T21:52:14.163 INFO:journalctl@ceph.node-exporter.a.smithi125.stdout:Mar 24 21:52:13 smithi125 conmon[41241]: time="2021-03-24T21:52:13Z" level=info msg=" - netclass" source="node_exporter.go:104" 2021-03-24T21:52:14.163 INFO:journalctl@ceph.node-exporter.a.smithi125.stdout:Mar 24 21:52:13 smithi125 conmon[41241]: time="2021-03-24T21:52:13Z" level=info msg=" - netdev" source="node_exporter.go:104" 2021-03-24T21:52:14.163 INFO:journalctl@ceph.node-exporter.a.smithi125.stdout:Mar 24 21:52:13 smithi125 conmon[41241]: time="2021-03-24T21:52:13Z" level=info msg=" - netstat" source="node_exporter.go:104" 2021-03-24T21:52:14.164 INFO:journalctl@ceph.node-exporter.a.smithi125.stdout:Mar 24 21:52:13 smithi125 conmon[41241]: time="2021-03-24T21:52:13Z" level=info msg=" - nfs" source="node_exporter.go:104" 2021-03-24T21:52:14.164 INFO:journalctl@ceph.node-exporter.a.smithi125.stdout:Mar 24 21:52:13 smithi125 conmon[41241]: time="2021-03-24T21:52:13Z" level=info msg=" - nfsd" source="node_exporter.go:104" 2021-03-24T21:52:14.164 INFO:journalctl@ceph.node-exporter.a.smithi125.stdout:Mar 24 21:52:13 smithi125 conmon[41241]: time="2021-03-24T21:52:13Z" level=info msg=" - pressure" source="node_exporter.go:104" 2021-03-24T21:52:14.164 INFO:journalctl@ceph.node-exporter.a.smithi125.stdout:Mar 24 21:52:13 smithi125 conmon[41241]: time="2021-03-24T21:52:13Z" level=info msg=" - sockstat" source="node_exporter.go:104" 2021-03-24T21:52:14.165 INFO:journalctl@ceph.node-exporter.a.smithi125.stdout:Mar 24 21:52:13 smithi125 conmon[41241]: time="2021-03-24T21:52:13Z" level=info msg=" - stat" source="node_exporter.go:104" 2021-03-24T21:52:14.165 INFO:journalctl@ceph.node-exporter.a.smithi125.stdout:Mar 24 21:52:13 smithi125 conmon[41241]: time="2021-03-24T21:52:13Z" level=info msg=" - textfile" source="node_exporter.go:104" 2021-03-24T21:52:14.165 INFO:journalctl@ceph.node-exporter.a.smithi125.stdout:Mar 24 21:52:13 smithi125 conmon[41241]: time="2021-03-24T21:52:13Z" level=info msg=" - time" source="node_exporter.go:104" 2021-03-24T21:52:14.165 INFO:journalctl@ceph.node-exporter.a.smithi125.stdout:Mar 24 21:52:13 smithi125 conmon[41241]: time="2021-03-24T21:52:13Z" level=info msg=" - uname" source="node_exporter.go:104" 2021-03-24T21:52:14.165 INFO:journalctl@ceph.node-exporter.a.smithi125.stdout:Mar 24 21:52:13 smithi125 conmon[41241]: time="2021-03-24T21:52:13Z" level=info msg=" - vmstat" source="node_exporter.go:104" 2021-03-24T21:52:14.166 INFO:journalctl@ceph.node-exporter.a.smithi125.stdout:Mar 24 21:52:13 smithi125 conmon[41241]: time="2021-03-24T21:52:13Z" level=info msg=" - xfs" source="node_exporter.go:104" 2021-03-24T21:52:14.166 INFO:journalctl@ceph.node-exporter.a.smithi125.stdout:Mar 24 21:52:13 smithi125 conmon[41241]: time="2021-03-24T21:52:13Z" level=info msg=" - zfs" source="node_exporter.go:104" 2021-03-24T21:52:14.166 INFO:journalctl@ceph.node-exporter.a.smithi125.stdout:Mar 24 21:52:13 smithi125 conmon[41241]: time="2021-03-24T21:52:13Z" level=info msg="Listening on :9100" source="node_exporter.go:170" 2021-03-24T21:52:14.166 INFO:journalctl@ceph.node-exporter.a.smithi125.stdout:Mar 24 21:52:13 smithi125 podman[41106]: 2021-03-24 21:52:13.838456301 +0000 UTC m=+3.092627609 container start 16a3b0e1d3edbe295fe2f70e12405bbbbe1271e40904abf6fe7ddb2a4399cf12 (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-node-exporter.a, maintainer=The Prometheus Authors ) 2021-03-24T21:52:14.166 INFO:journalctl@ceph.node-exporter.a.smithi125.stdout:Mar 24 21:52:13 smithi125 bash[41106]: 16a3b0e1d3edbe295fe2f70e12405bbbbe1271e40904abf6fe7ddb2a4399cf12 2021-03-24T21:52:14.167 INFO:journalctl@ceph.node-exporter.a.smithi125.stdout:Mar 24 21:52:13 smithi125 systemd[1]: Started Ceph node-exporter.a for 3cfa339a-8cea-11eb-90b2-001a4aab830c. 2021-03-24T21:52:14.565 INFO:teuthology.orchestra.run.smithi142.stdout:[client.1] 2021-03-24T21:52:14.566 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQCOtFtgmhVrIRAAZ3vtdB6NJxaGcFTc6H8rrQ== 2021-03-24T21:52:14.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:14 smithi142 conmon[20125]: cluster 2021-03-24T21:52:12. 2021-03-24T21:52:14.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:14 smithi142 conmon[20125]: 644234+0000 mgr.y (mgr.14144) 217 : cluster [DBG] pgmap v219: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 775 B/s rd, 0 op/s; 365 B/s, 4 objects/s recovering 2021-03-24T21:52:14.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:14 smithi142 conmon[20125]: audit 2021-03-24T21:52:14.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:14 smithi142 conmon[20125]: 2021-03-24T21:52:13.887782+0000 mon.a (mon.0) 535 2021-03-24T21:52:14.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:14 smithi142 conmon[20125]: : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:52:14.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:14 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:52:14.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:14 smithi125 conmon[24927]: -24T21:52:12.644234+0000 mgr.y (mgr.14144) 217 : cluster [DBG] pgmap v219: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 775 B/s rd, 0 op/s; 365 B/s, 4 objects/s recovering 2021-03-24T21:52:14.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:14 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T21:52:14.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:14 smithi125 conmon[24927]: 24T21:52:13.887782+0000 mon.a (mon. 2021-03-24T21:52:14.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:14 smithi125 conmon[24927]: 0) 535 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:52:14.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:14 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:52:14.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:14 smithi125 conmon[19936]: 52:12.644234+0000 mgr.y (mgr.14144) 217 : cluster [DBG] pgmap v219: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 775 B/s rd, 0 op/s; 365 B/s, 4 objects/s recovering 2021-03-24T21:52:14.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:14 smithi125 conmon[19936]: audit 2021-03- 2021-03-24T21:52:14.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:14 smithi125 conmon[19936]: 24T21:52:13.887782 2021-03-24T21:52:14.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:14 smithi125 conmon[19936]: +0000 mon.a (mon.0) 535 2021-03-24T21:52:14.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:14 smithi125 conmon[19936]: : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:52:15.614 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2021-03-24T21:52:15.615 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2021-03-24T21:52:15.615 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2021-03-24T21:52:15.637 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd dump --format=json 2021-03-24T21:52:15.657 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:15 smithi125 conmon[19936]: cephadm 2021-03-24T21:52:13.889038+0000 mgr.y (mgr.14144) 218 2021-03-24T21:52:15.657 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:15 smithi125 conmon[19936]: : cephadm [INF] Deploying daemon node-exporter.b on smithi142 2021-03-24T21:52:15.657 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:15 smithi125 conmon[19936]: audit 2021-03-24T21:52:14.558847+0000 mon.b (mon.1) 29 : audit [INF] 2021-03-24T21:52:15.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:15 smithi125 conmon[19936]: from='client.? v1:172.21.15.142:0/3470659043' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-03-24T21:52:15.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:15 smithi125 conmon[19936]: audit 2021-03-24T21:52: 2021-03-24T21:52:15.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:15 smithi125 conmon[19936]: 14.560291+0000 mon.a (mon.0) 536 : audit [INF] from='client.24424 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-03-24T21:52:15.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:15 smithi125 conmon[19936]: 2021-03-24T21:52:15.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:15 smithi125 conmon[19936]: audit 2021 2021-03-24T21:52:15.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:15 smithi125 conmon[19936]: -03-24T21:52:14.567471+0000 mon.a ( 2021-03-24T21:52:15.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:15 smithi125 conmon[19936]: mon.0) 537 : audit [INF] from='client.24424 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2021-03-24T21:52:15.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:15 smithi125 conmon[19936]: cluster 2021-03-24T21:52:15.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:15 smithi125 conmon[19936]: 2021-03-24T21:52:15 2021-03-24T21:52:15.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:15 smithi125 conmon[19936]: .031497+0000 mon.a (mon.0 2021-03-24T21:52:15.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:15 smithi125 conmon[19936]: ) 538 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2021-03-24T21:52:15.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:15 smithi125 conmon[24927]: cephadm 2021-03-24T21:52 2021-03-24T21:52:15.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:15 smithi125 conmon[24927]: :13.889038+0000 mgr.y (mgr.14144) 218 : cephadm [INF] Deploying daemon node-exporter.b on smithi142 2021-03-24T21:52:15.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:15 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T21:52:15.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:15 smithi125 conmon[24927]: 24T21:52:14.558847+0000 mon.b (mon.1 2021-03-24T21:52:15.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:15 smithi125 conmon[24927]: ) 29 : audit [INF] from='client.? v1:172.21.15.142:0/3470659043' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-03-24T21:52:15.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:15 smithi125 conmon[24927]: audit 2021-03-24T21:52:14. 2021-03-24T21:52:15.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:15 smithi125 conmon[24927]: 560291+0000 mon.a (mon.0) 536 : audit 2021-03-24T21:52:15.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:15 smithi125 conmon[24927]: [INF] from='client.24424 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-03-24T21:52:15.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:15 smithi125 conmon[24927]: audit 2021-03-24T21:52:14 2021-03-24T21:52:15.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:15 smithi125 conmon[24927]: .567471+0000 mon.a (mon.0) 537 : audit [INF] from='client.24424 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2021-03-24T21:52:15.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:15 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:52:15.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:15 smithi125 conmon[24927]: 21:52:15.031497+0000 mon.a (mon.0 2021-03-24T21:52:15.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:15 smithi125 conmon[24927]: ) 538 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2021-03-24T21:52:15.720 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:15 smithi142 conmon[20125]: cephadm 2021-03-24T21:52: 2021-03-24T21:52:15.721 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:15 smithi142 conmon[20125]: 13.889038+0000 mgr.y (mgr.14144) 218 : cephadm [INF] Deploying daemon node-exporter.b on smithi142 2021-03-24T21:52:15.721 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:15 smithi142 conmon[20125]: audit 2021-03-24 2021-03-24T21:52:15.721 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:15 smithi142 conmon[20125]: T21:52:14.558847+0000 mon.b (mon.1) 29 : audit [INF] from='client.? v1:172.21.15.142:0/3470659043' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-03-24T21:52:15.721 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:15 smithi142 conmon[20125]: audit 2021-03-24T21:52:15.722 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:15 smithi142 conmon[20125]: 2021-03-24T21:52:14.560291+0000 2021-03-24T21:52:15.722 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:15 smithi142 conmon[20125]: mon.a (mon.0) 536 : audit [INF] from='client.24424 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-03-24T21:52:15.722 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:15 smithi142 conmon[20125]: 2021-03-24T21:52:15.722 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:15 smithi142 conmon[20125]: audit 2021-03-24T21:52:14.567471 2021-03-24T21:52:15.723 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:15 smithi142 conmon[20125]: +0000 mon.a (mon.0) 537 : audit [INF] 2021-03-24T21:52:15.723 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:15 smithi142 conmon[20125]: from='client.24424 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2021-03-24T21:52:15.723 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:15 smithi142 conmon[20125]: cluster 2021-03-24T21:52:15 2021-03-24T21:52:15.723 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:15 smithi142 conmon[20125]: .031497+0000 mon.a (mon.0) 538 : cluster 2021-03-24T21:52:15.723 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:15 smithi142 conmon[20125]: [DBG] osdmap e76: 8 total, 8 up, 8 in 2021-03-24T21:52:15.945 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:52:16.034 INFO:journalctl@ceph.node-exporter.b.smithi142.stdout:Mar 24 21:52:15 smithi142 bash[38849]: Trying to pull docker.io/prom/node-exporter:v0.18.1... 2021-03-24T21:52:16.365 INFO:journalctl@ceph.node-exporter.b.smithi142.stdout:Mar 24 21:52:16 smithi142 bash[38849]: Getting image source signatures 2021-03-24T21:52:16.366 INFO:journalctl@ceph.node-exporter.b.smithi142.stdout:Mar 24 21:52:16 smithi142 bash[38849]: Copying blob sha256:49a2d53aa1af115e87f7e23a7fe4b7a3e12304d37982fb6528b035d0f68a3b56 2021-03-24T21:52:16.366 INFO:journalctl@ceph.node-exporter.b.smithi142.stdout:Mar 24 21:52:16 smithi142 bash[38849]: Copying blob sha256:190160031744b0810999dd6f293478be0b41524654c70782cf09437095430f2f 2021-03-24T21:52:16.366 INFO:journalctl@ceph.node-exporter.b.smithi142.stdout:Mar 24 21:52:16 smithi142 bash[38849]: Copying blob sha256:3589a6efd9ce96b2f2218a80bacb84f60d5e584d024023617b0f56b87fe798c3 2021-03-24T21:52:16.546 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:16 smithi125 conmon[19936]: cluster 2021-03-24T21:52:14. 2021-03-24T21:52:16.764 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:16 smithi125 conmon[19936]: 645424+0000 mgr.y (mgr.14144) 219 : cluster [DBG] pgmap v220: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 287 B/s, 2 objects/s recovering 2021-03-24T21:52:16.764 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:16 smithi125 conmon[19936]: cluster 2021-03-24T21:52:15.330756 2021-03-24T21:52:16.764 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:16 smithi125 conmon[19936]: +0000 osd.6 (osd.6) 3 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:52:16.765 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:16 smithi125 conmon[19936]: cluster 2021-03-24T21:52:16.765 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:16 smithi125 conmon[19936]: 2021-03-24T21:52:15.330960+0000 osd.6 (osd 2021-03-24T21:52:16.765 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:16 smithi125 conmon[19936]: .6) 4 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:52:16.765 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:16 smithi125 conmon[19936]: cluster 2021-03-24T21:52:16.043010+0000 2021-03-24T21:52:16.765 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:16 smithi125 conmon[19936]: mon.a (mon.0) 539 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2021-03-24T21:52:16.766 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:16 smithi125 conmon[24927]: cluster 2021-03-24T21:52: 2021-03-24T21:52:16.766 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:16 smithi125 conmon[24927]: 14.645424+0000 mgr.y (mgr.14144) 219 : cluster [DBG] pgmap v220: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 287 B/s, 2 objects/s recovering 2021-03-24T21:52:16.766 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:16 smithi125 conmon[24927]: cluster 2021-03-24T21:52:16.767 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:16 smithi125 conmon[24927]: 2021- 2021-03-24T21:52:16.767 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:16 smithi125 conmon[24927]: 03-24T21:52:15.330756+0000 2021-03-24T21:52:16.767 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:16 smithi125 conmon[24927]: osd.6 (osd.6) 3 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:52:16.767 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:16 smithi125 conmon[24927]: 2021-03-24T21:52:16.767 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:16 smithi125 conmon[24927]: cluster 2021-03-24T21:52:16.768 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:16 smithi125 conmon[24927]: 2021-03-24T21:52:15.330960+0000 2021-03-24T21:52:16.768 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:16 smithi125 conmon[24927]: osd.6 (osd.6) 4 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:52:16.768 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:16 smithi125 conmon[24927]: cluster 2021-03-24T21:52:16.768 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:16 smithi125 conmon[24927]: 2021-03-24T21:52:16.043010 2021-03-24T21:52:16.769 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:16 smithi125 conmon[24927]: +0000 mon.a (mon.0) 539 : cluster 2021-03-24T21:52:16.769 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:16 smithi125 conmon[24927]: [DBG] osdmap e77: 8 total, 8 up, 8 in 2021-03-24T21:52:16.866 INFO:journalctl@ceph.node-exporter.b.smithi142.stdout:Mar 24 21:52:16 smithi142 bash[38849]: Copying config sha256:e5a616e4b9cf68dfcad7782b78e118be4310022e874d52da85c55923fb615f87 2021-03-24T21:52:16.866 INFO:journalctl@ceph.node-exporter.b.smithi142.stdout:Mar 24 21:52:16 smithi142 bash[38849]: Writing manifest to image destination 2021-03-24T21:52:16.866 INFO:journalctl@ceph.node-exporter.b.smithi142.stdout:Mar 24 21:52:16 smithi142 bash[38849]: Storing signatures 2021-03-24T21:52:16.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:16 smithi142 conmon[20125]: cluster 2021-03-24T21 2021-03-24T21:52:16.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:16 smithi142 conmon[20125]: :52:14.645424+0000 mgr.y (mgr.14144) 219 : cluster [DBG] pgmap v220: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 287 B/s, 2 objects/s recovering 2021-03-24T21:52:16.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:16 smithi142 conmon[20125]: cluster 2021-03-24T21:52:16.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:16 smithi142 conmon[20125]: 2021-03-24T21:52:15.330756+0000 osd.6 (osd.6) 3 2021-03-24T21:52:16.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:16 smithi142 conmon[20125]: : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:52:16.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:16 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:52:16.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:16 smithi142 conmon[20125]: T21:52:15.330960+0000 osd.6 (osd. 2021-03-24T21:52:16.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:16 smithi142 conmon[20125]: 6) 4 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:52:16.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:16 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:52:16.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:16 smithi142 conmon[20125]: 03-24T21:52:16.043010+0000 2021-03-24T21:52:16.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:16 smithi142 conmon[20125]: mon.a (mon.0) 539 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2021-03-24T21:52:18.115 INFO:journalctl@ceph.node-exporter.b.smithi142.stdout:Mar 24 21:52:17 smithi142 podman[38849]: 2021-03-24 21:52:17.851547053 +0000 UTC m=+2.191436978 image pull 2021-03-24T21:52:18.116 INFO:journalctl@ceph.node-exporter.b.smithi142.stdout:Mar 24 21:52:18 smithi142 podman[38849]: 2021-03-24 21:52:18.010316552 +0000 UTC m=+2.350206434 container create ecfd82dd7f74f17cc23edd415f443d8e397e204f458704b1a98a1132eecd8801 (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-node-exporter.b, maintainer=The Prometheus Authors ) 2021-03-24T21:52:18.228 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-24T21:52:18.229 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":77,"fsid":"3cfa339a-8cea-11eb-90b2-001a4aab830c","created":"2021-03-24T21:46:19.163312+0000","modified":"2021-03-24T21:52:16.036908+0000","last_up_change":"2021-03-24T21:51:38.283730+0000","last_in_change":"2021-03-24T21:51:20.214398+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-24T21:49:51.001357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"iscsi","create_time":"2021-03-24T21:51:39.190981+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":".rgw.root","create_time":"2021-03-24T21:51:44.159145+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2021-03-24T21:51:46.090924+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2021-03-24T21:51:47.842521+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2021-03-24T21:51:50.730906+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"1f8f9a69-4b85-4054-8b11-36d35574494c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6801","nonce":1197472172}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6802","nonce":1197472172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6804","nonce":1197472172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6803","nonce":1197472172}]},"public_addr":"172.21.15.125:6801/1197472172","cluster_addr":"172.21.15.125:6802/1197472172","heartbeat_back_addr":"172.21.15.125:6804/1197472172","heartbeat_front_addr":"172.21.15.125:6803/1197472172","state":["exists","up"]},{"osd":1,"uuid":"48db0723-5806-4e5c-9919-f3d7f52ad169","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6805","nonce":113845288}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6806","nonce":113845288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6808","nonce":113845288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6807","nonce":113845288}]},"public_addr":"172.21.15.125:6805/113845288","cluster_addr":"172.21.15.125:6806/113845288","heartbeat_back_addr":"172.21.15.125:6808/113845288","heartbeat_front_addr":"172.21.15.125:6807/113845288","state":["exists","up"]},{"osd":2,"uuid":"d6a361f8-4e4d-4f46-8ab9-01bf71dbb2c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6809","nonce":246939163}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6810","nonce":246939163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6812","nonce":246939163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6811","nonce":246939163}]},"public_addr":"172.21.15.125:6809/246939163","cluster_addr":"172.21.15.125:6810/246939163","heartbeat_back_addr":"172.21.15.125:6812/246939163","heartbeat_front_addr":"172.21.15.125:6811/246939163","state":["exists","up"]},{"osd":3,"uuid":"716a41ce-7a28-400f-bf75-4c1ad99c7f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6813","nonce":2209636173}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6814","nonce":2209636173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6816","nonce":2209636173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6815","nonce":2209636173}]},"public_addr":"172.21.15.125:6813/2209636173","cluster_addr":"172.21.15.125:6814/2209636173","heartbeat_back_addr":"172.21.15.125:6816/2209636173","heartbeat_front_addr":"172.21.15.125:6815/2209636173","state":["exists","up"]},{"osd":4,"uuid":"f9bdd8d2-b292-49da-ad4f-99682122f543","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.142:6800","nonce":1816503330}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.142:6801","nonce":1816503330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.142:6803","nonce":1816503330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.142:6802","nonce":1816503330}]},"public_addr":"172.21.15.142:6800/1816503330","cluster_addr":"172.21.15.142:6801/1816503330","heartbeat_back_addr":"172.21.15.142:6803/1816503330","heartbeat_front_addr":"172.21.15.142:6802/1816503330","state":["exists","up"]},{"osd":5,"uuid":"ec0d43aa-6905-43f0-9e32-e19d4cf5129d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.142:6804","nonce":396400804}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.142:6805","nonce":396400804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.142:6807","nonce":396400804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.142:6806","nonce":396400804}]},"public_addr":"172.21.15.142:6804/396400804","cluster_addr":"172.21.15.142:6805/396400804","heartbeat_back_addr":"172.21.15.142:6807/396400804","heartbeat_front_addr":"172.21.15.142:6806/396400804","state":["exists","up"]},{"osd":6,"uuid":"a26a487e-ed26-4d92-98ea-896739302921","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.142:6808","nonce":4048483913}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.142:6809","nonce":4048483913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.142:6811","nonce":4048483913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.142:6810","nonce":4048483913}]},"public_addr":"172.21.15.142:6808/4048483913","cluster_addr":"172.21.15.142:6809/4048483913","heartbeat_back_addr":"172.21.15.142:6811/4048483913","heartbeat_front_addr":"172.21.15.142:6810/4048483913","state":["exists","up"]},{"osd":7,"uuid":"682ba5ce-0172-48e3-82f8-57a3f98c911b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.142:6812","nonce":3678129570}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.142:6813","nonce":3678129570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.142:6815","nonce":3678129570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.142:6814","nonce":3678129570}]},"public_addr":"172.21.15.142:6812/3678129570","cluster_addr":"172.21.15.142:6813/3678129570","heartbeat_back_addr":"172.21.15.142:6815/3678129570","heartbeat_front_addr":"172.21.15.142:6814/3678129570","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T21:52:07.610589+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T21:51:28.965683+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T21:51:49.809720+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T21:52:11.677778+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T21:51:30.467445+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T21:51:52.834140+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T21:52:15.330965+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T21:51:36.348518+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.1c","mappings":[{"from":1,"to":5}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:0/3077905055":"2021-03-25T21:47:04.502055+0000","172.21.15.125:0/2210544773":"2021-03-25T21:47:04.502055+0000","172.21.15.125:6800/3176807740":"2021-03-25T21:47:04.502055+0000","172.21.15.125:6800/3531208850":"2021-03-25T21:46:43.197724+0000","172.21.15.125:0/3337261959":"2021-03-25T21:46:43.197724+0000","172.21.15.125:0/1919150488":"2021-03-25T21:46:43.197724+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-03-24T21:52:18.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:52:18.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[20125]: -24T21:52:16.646186+0000 mgr.y (mgr.14144) 220 : cluster [DBG] pgmap v223: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 174 B/s, 1 objects/s recovering 2021-03-24T21:52:18.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:52:18.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[20125]: 03-24T21: 2021-03-24T21:52:18.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[20125]: 52:18.228195+0000 2021-03-24T21:52:18.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[20125]: mon.c (mon.2) 2021-03-24T21:52:18.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[20125]: 11 : audit [DBG] from='client.? v1:172.21.15.125:0/2985987131' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-03-24T21:52:18.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[20125]: 2021-03-24T21:52:18.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[20125]: audit 2021 2021-03-24T21:52:18.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[20125]: -03-24T21: 2021-03-24T21:52:18.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[20125]: 52:18.244756+0000 2021-03-24T21:52:18.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[20125]: mon.a (mon.0) 2021-03-24T21:52:18.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[20125]: 540 : audit [INF] 2021-03-24T21:52:18.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:52:18.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[20125]: audit 2021 2021-03-24T21:52:18.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[20125]: -03-24T21: 2021-03-24T21:52:18.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[20125]: 52:18.245945+0000 2021-03-24T21:52:18.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[20125]: mon.a (mon.0) 2021-03-24T21:52:18.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[20125]: 541 : audit [DBG] 2021-03-24T21:52:18.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:52:18.621 INFO:journalctl@ceph.node-exporter.b.smithi142.stdout:Mar 24 21:52:18 smithi142 podman[38849]: 2021-03-24 21:52:18.168641509 +0000 UTC m=+2.508531384 container init ecfd82dd7f74f17cc23edd415f443d8e397e204f458704b1a98a1132eecd8801 (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-node-exporter.b, maintainer=The Prometheus Authors ) 2021-03-24T21:52:18.621 INFO:journalctl@ceph.node-exporter.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[38933]: time="2021-03-24T21:52:18Z" level=info msg="Starting node_exporter (version=0.18.1, branch=HEAD, revision=3db77732e925c08f675d7404a8c46466b2ece83e)" source="node_exporter.go:156" 2021-03-24T21:52:18.622 INFO:journalctl@ceph.node-exporter.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[38933]: time="2021-03-24T21:52:18Z" level=info msg="Build context (go=go1.12.5, user=root@b50852a1acba, date=20190604-16:41:18)" source="node_exporter.go:157" 2021-03-24T21:52:18.622 INFO:journalctl@ceph.node-exporter.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[38933]: time="2021-03-24T21:52:18Z" level=info msg="Enabled collectors:" source="node_exporter.go:97" 2021-03-24T21:52:18.622 INFO:journalctl@ceph.node-exporter.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[38933]: time="2021-03-24T21:52:18Z" level=info msg=" - arp" source="node_exporter.go:104" 2021-03-24T21:52:18.622 INFO:journalctl@ceph.node-exporter.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[38933]: time="2021-03-24T21:52:18Z" level=info msg=" - bcache" source="node_exporter.go:104" 2021-03-24T21:52:18.622 INFO:journalctl@ceph.node-exporter.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[38933]: time="2021-03-24T21:52:18Z" level=info msg=" - bonding" source="node_exporter.go:104" 2021-03-24T21:52:18.623 INFO:journalctl@ceph.node-exporter.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[38933]: time="2021-03-24T21:52:18Z" level=info msg=" - conntrack" source="node_exporter.go:104" 2021-03-24T21:52:18.623 INFO:journalctl@ceph.node-exporter.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[38933]: time="2021-03-24T21:52:18Z" level=info msg=" - cpu" source="node_exporter.go:104" 2021-03-24T21:52:18.623 INFO:journalctl@ceph.node-exporter.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[38933]: time="2021-03-24T21:52:18Z" level=info msg=" - cpufreq" source="node_exporter.go:104" 2021-03-24T21:52:18.623 INFO:journalctl@ceph.node-exporter.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[38933]: time="2021-03-24T21:52:18Z" level=info msg=" - diskstats" source="node_exporter.go:104" 2021-03-24T21:52:18.624 INFO:journalctl@ceph.node-exporter.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[38933]: time="2021-03-24T21:52:18Z" level=info msg=" - edac" source="node_exporter.go:104" 2021-03-24T21:52:18.624 INFO:journalctl@ceph.node-exporter.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[38933]: time="2021-03-24T21:52:18Z" level=info msg=" - entropy" source="node_exporter.go:104" 2021-03-24T21:52:18.624 INFO:journalctl@ceph.node-exporter.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[38933]: time="2021-03-24T21:52:18Z" level=info msg=" - filefd" source="node_exporter.go:104" 2021-03-24T21:52:18.624 INFO:journalctl@ceph.node-exporter.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[38933]: time="2021-03-24T21:52:18Z" level=info msg=" - filesystem" source="node_exporter.go:104" 2021-03-24T21:52:18.624 INFO:journalctl@ceph.node-exporter.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[38933]: time="2021-03-24T21:52:18Z" level=info msg=" - hwmon" source="node_exporter.go:104" 2021-03-24T21:52:18.625 INFO:journalctl@ceph.node-exporter.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[38933]: time="2021-03-24T21:52:18Z" level=info msg=" - infiniband" source="node_exporter.go:104" 2021-03-24T21:52:18.625 INFO:journalctl@ceph.node-exporter.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[38933]: time="2021-03-24T21:52:18Z" level=info msg=" - ipvs" source="node_exporter.go:104" 2021-03-24T21:52:18.625 INFO:journalctl@ceph.node-exporter.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[38933]: time="2021-03-24T21:52:18Z" level=info msg=" - loadavg" source="node_exporter.go:104" 2021-03-24T21:52:18.625 INFO:journalctl@ceph.node-exporter.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[38933]: time="2021-03-24T21:52:18Z" level=info msg=" - mdadm" source="node_exporter.go:104" 2021-03-24T21:52:18.625 INFO:journalctl@ceph.node-exporter.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[38933]: time="2021-03-24T21:52:18Z" level=info msg=" - meminfo" source="node_exporter.go:104" 2021-03-24T21:52:18.626 INFO:journalctl@ceph.node-exporter.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[38933]: time="2021-03-24T21:52:18Z" level=info msg=" - netclass" source="node_exporter.go:104" 2021-03-24T21:52:18.626 INFO:journalctl@ceph.node-exporter.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[38933]: time="2021-03-24T21:52:18Z" level=info msg=" - netdev" source="node_exporter.go:104" 2021-03-24T21:52:18.626 INFO:journalctl@ceph.node-exporter.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[38933]: time="2021-03-24T21:52:18Z" level=info msg=" - netstat" source="node_exporter.go:104" 2021-03-24T21:52:18.626 INFO:journalctl@ceph.node-exporter.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[38933]: time="2021-03-24T21:52:18Z" level=info msg=" - nfs" source="node_exporter.go:104" 2021-03-24T21:52:18.627 INFO:journalctl@ceph.node-exporter.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[38933]: time="2021-03-24T21:52:18Z" level=info msg=" - nfsd" source="node_exporter.go:104" 2021-03-24T21:52:18.627 INFO:journalctl@ceph.node-exporter.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[38933]: time="2021-03-24T21:52:18Z" level=info msg=" - pressure" source="node_exporter.go:104" 2021-03-24T21:52:18.627 INFO:journalctl@ceph.node-exporter.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[38933]: time="2021-03-24T21:52:18Z" level=info msg=" - sockstat" source="node_exporter.go:104" 2021-03-24T21:52:18.627 INFO:journalctl@ceph.node-exporter.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[38933]: time="2021-03-24T21:52:18Z" level=info msg=" - stat" source="node_exporter.go:104" 2021-03-24T21:52:18.627 INFO:journalctl@ceph.node-exporter.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[38933]: time="2021-03-24T21:52:18Z" level=info msg=" - textfile" source="node_exporter.go:104" 2021-03-24T21:52:18.628 INFO:journalctl@ceph.node-exporter.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[38933]: time="2021-03-24T21:52:18Z" level=info msg=" - time" source="node_exporter.go:104" 2021-03-24T21:52:18.628 INFO:journalctl@ceph.node-exporter.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[38933]: time="2021-03-24T21:52:18Z" level=info msg=" - uname" source="node_exporter.go:104" 2021-03-24T21:52:18.628 INFO:journalctl@ceph.node-exporter.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[38933]: time="2021-03-24T21:52:18Z" level=info msg=" - vmstat" source="node_exporter.go:104" 2021-03-24T21:52:18.628 INFO:journalctl@ceph.node-exporter.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[38933]: time="2021-03-24T21:52:18Z" level=info msg=" - xfs" source="node_exporter.go:104" 2021-03-24T21:52:18.628 INFO:journalctl@ceph.node-exporter.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[38933]: time="2021-03-24T21:52:18Z" level=info msg=" - zfs" source="node_exporter.go:104" 2021-03-24T21:52:18.629 INFO:journalctl@ceph.node-exporter.b.smithi142.stdout:Mar 24 21:52:18 smithi142 conmon[38933]: time="2021-03-24T21:52:18Z" level=info msg="Listening on :9100" source="node_exporter.go:170" 2021-03-24T21:52:18.629 INFO:journalctl@ceph.node-exporter.b.smithi142.stdout:Mar 24 21:52:18 smithi142 podman[38849]: 2021-03-24 21:52:18.201956648 +0000 UTC m=+2.541846530 container start ecfd82dd7f74f17cc23edd415f443d8e397e204f458704b1a98a1132eecd8801 (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-node-exporter.b, maintainer=The Prometheus Authors ) 2021-03-24T21:52:18.629 INFO:journalctl@ceph.node-exporter.b.smithi142.stdout:Mar 24 21:52:18 smithi142 bash[38849]: ecfd82dd7f74f17cc23edd415f443d8e397e204f458704b1a98a1132eecd8801 2021-03-24T21:52:18.629 INFO:journalctl@ceph.node-exporter.b.smithi142.stdout:Mar 24 21:52:18 smithi142 systemd[1]: Started Ceph node-exporter.b for 3cfa339a-8cea-11eb-90b2-001a4aab830c. 2021-03-24T21:52:18.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:18 smithi125 conmon[19936]: cluster 2021-03-24T21:52:16.646186+0000 2021-03-24T21:52:18.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:18 smithi125 conmon[19936]: mgr.y (mgr.14144) 220 : cluster [DBG] pgmap v223: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 174 B/s, 1 objects/s recovering 2021-03-24T21:52:18.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:18 smithi125 conmon[19936]: audit 2021-03-24T21:52:18.228195+0000 2021-03-24T21:52:18.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:18 smithi125 conmon[19936]: mon.c (mon.2) 11 : audit [DBG] from='client.? v1:172.21.15.125:0/2985987131' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-03-24T21:52:18.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:18 smithi125 conmon[19936]: audit 2021-03-24T21:52:18.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:18 smithi125 conmon[19936]: 2021-03-24T21:52:18.244756+0000 2021-03-24T21:52:18.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:18 smithi125 conmon[19936]: mon.a (mon.0) 540 : audit [INF] 2021-03-24T21:52:18.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:18 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:52:18.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:18 smithi125 conmon[19936]: audit 2021 2021-03-24T21:52:18.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:18 smithi125 conmon[19936]: -03-24T21:52:18.245945+0000 mon.a 2021-03-24T21:52:18.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:18 smithi125 conmon[19936]: (mon.0) 541 : audit 2021-03-24T21:52:18.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:18 smithi125 conmon[19936]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:52:18.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:18 smithi125 conmon[24927]: cluster 2021-03-24T21:52:18.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:18 smithi125 conmon[24927]: 2021-03-24T21:52:18.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:18 smithi125 conmon[24927]: 2021 2021-03-24T21:52:18.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:18 smithi125 conmon[24927]: - 2021-03-24T21:52:18.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:18 smithi125 conmon[24927]: 03-24T21:52 2021-03-24T21:52:18.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:18 smithi125 conmon[24927]: :16.646186+0000 mgr.y (mgr.14144) 220 : cluster [DBG] pgmap v223: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 174 B/s, 1 objects/s recovering 2021-03-24T21:52:18.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:18 smithi125 conmon[24927]: audit 2021-03-24T 2021-03-24T21:52:18.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:18 smithi125 conmon[24927]: 21:52:18.228195+0000 mon.c (mon.2) 11 : audit [DBG] 2021-03-24T21:52:18.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:18 smithi125 conmon[24927]: from='client.? v1:172.21.15.125:0/2985987131' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-03-24T21:52:18.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:18 smithi125 conmon[24927]: audit 2021-03-24T21:52: 2021-03-24T21:52:18.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:18 smithi125 conmon[24927]: 18.244756+0000 mon.a (mon.0) 2021-03-24T21:52:18.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:18 smithi125 conmon[24927]: 540 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:52:18.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:18 smithi125 conmon[24927]: audit 2021-03-24T 2021-03-24T21:52:18.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:18 smithi125 conmon[24927]: 21:52:18.245945+0000 mon.a ( 2021-03-24T21:52:18.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:18 smithi125 conmon[24927]: mon.0) 541 : audit [DBG] 2021-03-24T21:52:18.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:18 smithi125 conmon[24927]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:52:19.144 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-03-24T21:49:51.001357+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'iscsi', 'create_time': '2021-03-24T21:51:39.190981+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2021-03-24T21:51:44.159145+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2021-03-24T21:51:46.090924+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2021-03-24T21:51:47.842521+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2021-03-24T21:51:50.730906+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 8}, 'application_metadata': {'rgw': {}}}] 2021-03-24T21:52:19.145 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd pool get device_health_metrics pg_num 2021-03-24T21:52:19.614 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:52:20.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:20 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:52:20.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:20 smithi125 conmon[24927]: 21:52:18.647080+0000 mgr.y (mgr.14144) 221 : cluster [DBG] pgmap v224: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2021-03-24T21:52:20.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:20 smithi125 conmon[24927]: cluster 2021-03-24T21:52 2021-03-24T21:52:20.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:20 smithi125 conmon[24927]: :20.025909+0000 mon.a (mon 2021-03-24T21:52:20.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:20 smithi125 conmon[24927]: .0) 542 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2021-03-24T21:52:20.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:20 smithi125 conmon[19936]: cluster 2021-03-24T21:52 2021-03-24T21:52:20.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:20 smithi125 conmon[19936]: :18.647080+0000 mgr.y (mgr.14144) 221 : cluster [DBG] pgmap v224: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2021-03-24T21:52:20.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:20 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T21:52:20.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:20 smithi125 conmon[19936]: T21:52:20.025909+0000 2021-03-24T21:52:20.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:20 smithi125 conmon[19936]: mon.a (mon.0) 542 2021-03-24T21:52:20.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:20 smithi125 conmon[19936]: : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2021-03-24T21:52:20.819 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:20 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:52:20.820 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:20 smithi142 conmon[20125]: 52:18.647080+0000 mgr.y (mgr.14144) 221 : cluster [DBG] pgmap v224: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2021-03-24T21:52:20.820 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:20 smithi142 conmon[20125]: cluster 2021-03-24T21:52:20.820 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:20 smithi142 conmon[20125]: 2021-03-24T21:52:20.025909+0000 mon.a (mon.0 2021-03-24T21:52:20.820 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:20 smithi142 conmon[20125]: ) 542 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2021-03-24T21:52:21.720 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2021-03-24T21:52:22.818 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:22 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:52:22.818 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:22 smithi125 conmon[19936]: 21:52:20.647834+0000 mgr.y (mgr.14144) 222 : cluster [DBG] pgmap v226: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-03-24T21:52:22.819 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:22 smithi125 conmon[19936]: audit 2021-03-24T21:52:21.706067+0000 mon.a (mon. 2021-03-24T21:52:22.819 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:22 smithi125 conmon[19936]: 0) 543 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:52:22.819 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:22 smithi125 conmon[19936]: audit 2021-03-24T21:52:21 2021-03-24T21:52:22.819 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:22 smithi125 conmon[19936]: .722076+0000 mon.c (mon.2) 12 : audit 2021-03-24T21:52:22.819 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:22 smithi125 conmon[19936]: [DBG] from='client.? v1:172.21.15.125:0/3104083234' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2021-03-24T21:52:22.820 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:22 smithi125 conmon[24927]: cluster 2021-03-24T21:52 2021-03-24T21:52:22.820 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:22 smithi125 conmon[24927]: :20.647834+0000 mgr.y (mgr.14144) 222 : cluster [DBG] pgmap v226: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-03-24T21:52:22.820 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:22 smithi125 conmon[24927]: audit 2021-03-24T21:52:22.821 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:22 smithi125 conmon[24927]: 2021-03-24T21:52:21.706067+0000 mon.a (mon. 2021-03-24T21:52:22.821 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:22 smithi125 conmon[24927]: 0) 543 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:52:22.821 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:22 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T21:52:22.821 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:22 smithi125 conmon[24927]: 24T21:52:21.722076+0000 mon.c 2021-03-24T21:52:22.821 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:22 smithi125 conmon[24927]: (mon.2) 12 : audit [DBG] from='client.? v1:172.21.15.125:0/3104083234' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2021-03-24T21:52:22.822 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:22 smithi125 conmon[24927]: 2021-03-24T21:52:22.846 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd pool get iscsi pg_num 2021-03-24T21:52:22.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:22 smithi142 conmon[20125]: cluster 2021-03-24T21:52:20.647834+0000 mgr.y (mgr.14144) 222 : cluster 2021-03-24T21:52:22.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:22 smithi142 conmon[20125]: [DBG] pgmap v226: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-03-24T21:52:22.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:22 smithi142 conmon[20125]: audit 2021-03-24T21:52:21.706067+0000 mon.a (mon 2021-03-24T21:52:22.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:22 smithi142 conmon[20125]: .0) 543 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:52:22.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:22 smithi142 conmon[20125]: audit 2021 2021-03-24T21:52:22.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:22 smithi142 conmon[20125]: -03-24T21:52:21 2021-03-24T21:52:22.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:22 smithi142 conmon[20125]: .722076+0000 mon.c (mon.2 2021-03-24T21:52:22.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:22 smithi142 conmon[20125]: ) 12 : audit [DBG] from='client.? v1:172.21.15.125:0/3104083234' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2021-03-24T21:52:23.149 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:52:24.116 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:23 smithi142 conmon[20125]: cluster 2021-03-24T21:52:22.648632+0000 mgr.y (mgr.14144) 223 : cluster 2021-03-24T21:52:24.116 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:23 smithi142 conmon[20125]: [DBG] pgmap v227: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 671 B/s rd, 0 op/s 2021-03-24T21:52:24.116 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:23 smithi142 conmon[20125]: audit 2021-03-24T21:52:22.829347+0000 2021-03-24T21:52:24.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:23 smithi142 conmon[20125]: mon.a (mon.0) 544 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:52:24.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:23 smithi142 conmon[20125]: audit 2021-03-24T21:52:24.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:23 smithi142 conmon[20125]: 2021-03-24T21:52: 2021-03-24T21:52:24.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:23 smithi142 conmon[20125]: 23.142184+0000 mon.a (mon.0) 545 2021-03-24T21:52:24.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:23 smithi142 conmon[20125]: : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:52:24.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:23 smithi142 conmon[20125]: cephadm 2021-03-24 2021-03-24T21:52:24.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:23 smithi142 conmon[20125]: T21:52:23.154246+0000 mgr.y (mgr 2021-03-24T21:52:24.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:23 smithi142 conmon[20125]: .14144) 224 : cephadm [INF] Deploying daemon alertmanager.a on smithi125 2021-03-24T21:52:24.158 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:23 smithi125 conmon[19936]: cluster 2021-03-24T21:52: 2021-03-24T21:52:24.222 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:23 smithi125 conmon[19936]: 22.648632+0000 mgr.y (mgr.14144) 223 : cluster [DBG] pgmap v227: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 671 B/s rd, 0 op/s 2021-03-24T21:52:24.222 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:23 smithi125 conmon[19936]: audit 2021-03 2021-03-24T21:52:24.222 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:23 smithi125 conmon[19936]: -24T21:52:22.829347+0000 mon.a (mon 2021-03-24T21:52:24.223 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:23 smithi125 conmon[19936]: .0) 544 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:52:24.223 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:23 smithi125 conmon[19936]: audit 2021 2021-03-24T21:52:24.223 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:23 smithi125 conmon[19936]: -03-24T21:52:23.142184+0000 2021-03-24T21:52:24.223 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:23 smithi125 conmon[19936]: mon.a (mon.0) 545 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:52:24.224 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:23 smithi125 conmon[19936]: 2021-03-24T21:52:24.224 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:23 smithi125 conmon[19936]: cephadm 2021-03-24T21:52: 2021-03-24T21:52:24.224 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:23 smithi125 conmon[19936]: 23.154246+0000 mgr.y (mgr.14144) 224 : cephadm [INF] 2021-03-24T21:52:24.224 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:23 smithi125 conmon[19936]: Deploying daemon alertmanager.a on smithi125 2021-03-24T21:52:24.225 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:23 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:52:24.225 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:23 smithi125 conmon[24927]: 24T21:52:22.648632+0000 mgr.y (mgr.14144) 223 : cluster [DBG] pgmap v227: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 671 B/s rd, 0 op/s 2021-03-24T21:52:24.225 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:23 smithi125 conmon[24927]: 2021-03-24T21:52:24.225 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:23 smithi125 conmon[24927]: audit 2021-03-24T21:52:24.225 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:23 smithi125 conmon[24927]: 2021-03-24T21:52:22. 2021-03-24T21:52:24.226 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:23 smithi125 conmon[24927]: 829347+0000 mon.a (mon.0) 544 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:52:24.226 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:23 smithi125 conmon[24927]: 2021-03-24T21:52:24.226 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:23 smithi125 conmon[24927]: audit 2021-03-24T21:52:24.226 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:23 smithi125 conmon[24927]: 2021-03-24T 2021-03-24T21:52:24.227 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:23 smithi125 conmon[24927]: 21:52:23.142184+0000 mon.a (mon.0) 545 2021-03-24T21:52:24.227 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:23 smithi125 conmon[24927]: : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:52:24.227 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:23 smithi125 conmon[24927]: cephadm 2021-03-24T21:52:23. 2021-03-24T21:52:24.227 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:23 smithi125 conmon[24927]: 154246+0000 mgr.y (mgr.14144) 224 : cephadm [INF] Deploying daemon alertmanager.a on smithi125 2021-03-24T21:52:25.519 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 3 2021-03-24T21:52:26.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:26 smithi142 conmon[20125]: cluster 2021-03-24T21:52: 2021-03-24T21:52:26.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:26 smithi142 conmon[20125]: 24.649489+0000 mgr.y (mgr.14144) 225 : cluster [DBG] pgmap v228: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-24T21:52:26.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:26 smithi142 conmon[20125]: audit 2021-03-24T21:52: 2021-03-24T21:52:26.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:26 smithi142 conmon[20125]: 25.520502+0000 mon.a (mon.0) 546 : audit [DBG] from='client.? v1:172.21.15.125:0/1939572147' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "iscsi", "var": "pg_num"}]: dispatch 2021-03-24T21:52:26.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:26 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:52:26.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:26 smithi125 conmon[24927]: 52:24.649489+0000 mgr.y (mgr.14144) 225 : cluster [DBG] pgmap v228: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-24T21:52:26.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:26 smithi125 conmon[24927]: audit 2021-03-24T21: 2021-03-24T21:52:26.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:26 smithi125 conmon[24927]: 52:25.520502+0000 mon.a (mon.0) 546 2021-03-24T21:52:26.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:26 smithi125 conmon[24927]: : audit [DBG] from='client.? v1:172.21.15.125:0/1939572147' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "iscsi", "var": "pg_num"}]: dispatch 2021-03-24T21:52:26.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:26 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:52:26.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:26 smithi125 conmon[19936]: 21:52:24.649489+0000 mgr.y (mgr.14144) 225 : cluster [DBG] pgmap v228: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-24T21:52:26.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:26 smithi125 conmon[19936]: audit 2021-03-24T21:52:26.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:26 smithi125 conmon[19936]: 2021-03 2021-03-24T21:52:26.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:26 smithi125 conmon[19936]: -24T21:52:25.520502+0000 mon.a 2021-03-24T21:52:26.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:26 smithi125 conmon[19936]: (mon.0) 546 : audit [DBG] from='client.? v1:172.21.15.125:0/1939572147' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "iscsi", "var": "pg_num"}]: dispatch 2021-03-24T21:52:27.990 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd pool get .rgw.root pg_num 2021-03-24T21:52:28.158 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:28 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:52:28.158 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:28 smithi125 conmon[19936]: 21:52:26.650320+0000 mgr.y (mgr.14144) 226 : cluster [DBG] pgmap v229: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-24T21:52:28.158 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:28 smithi125 conmon[24927]: cluster 2021-03-24T21:52:26.650320+0000 2021-03-24T21:52:28.159 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:28 smithi125 conmon[24927]: mgr.y (mgr.14144) 226 : cluster [DBG] pgmap v229: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-24T21:52:28.313 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:52:28.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:28 smithi142 conmon[20125]: cluster 2021-03-24T21:52 2021-03-24T21:52:28.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:28 smithi142 conmon[20125]: :26.650320+0000 mgr.y (mgr.14144) 226 : cluster [DBG] pgmap v229: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-24T21:52:30.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:30 smithi142 conmon[20125]: cluster 2021-03-24T21:52:28.651218+0000 mgr.y (mgr.14144 2021-03-24T21:52:30.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:30 smithi142 conmon[20125]: ) 227 : cluster [DBG] pgmap v230: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-24T21:52:30.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:30 smithi142 conmon[20125]: cluster 2021-03-24T21:52:28.967072+0000 osd.1 (osd 2021-03-24T21:52:30.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:30 smithi142 conmon[20125]: .1) 7 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:52:30.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:30 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:52:30.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:30 smithi142 conmon[20125]: 21:52:28.967286+0000 osd.1 (osd. 2021-03-24T21:52:30.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:30 smithi142 conmon[20125]: 1) 8 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:52:30.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:30 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:52:30.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:30 smithi142 conmon[20125]: -03-24T21:52:29 2021-03-24T21:52:30.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:30 smithi142 conmon[20125]: .042535+0000 mon.a (mon.0) 547 : 2021-03-24T21:52:30.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:30 smithi142 conmon[20125]: cluster [DBG] osdmap e79: 8 total, 8 up, 8 in 2021-03-24T21:52:30.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:30 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:52:30.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:30 smithi142 conmon[20125]: 24T21:52:30.032553+0000 2021-03-24T21:52:30.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:30 smithi142 conmon[20125]: mon.a (mon.0) 548 : cluster [DBG] 2021-03-24T21:52:30.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:30 smithi142 conmon[20125]: osdmap e80: 8 total, 8 up, 8 in 2021-03-24T21:52:30.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:30 smithi125 conmon[19936]: cluster 2021-03-24T21:52:28.651218+0000 mgr.y (mgr.14144) 2021-03-24T21:52:30.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:30 smithi125 conmon[19936]: 227 : cluster [DBG] pgmap v230: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-24T21:52:30.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:30 smithi125 conmon[19936]: cluster 2021-03-24T21:52:28.967072+0000 osd.1 (osd.1) 7 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:52:30.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:30 smithi125 conmon[19936]: cluster 2021-03-24T21:52:28.967286+0000 osd.1 (osd.1) 8 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:52:30.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:30 smithi125 conmon[19936]: cluster 2021-03-24T21:52:29.042535+0000 mon.a (mon.0) 547 : cluster [DBG] osdmap e79: 8 total, 8 up, 8 in 2021-03-24T21:52:30.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:30 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:52:30.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:30 smithi125 conmon[19936]: -03-24T21:52:30.032553+0000 mon.a (mon.0) 548 : cluster [DBG] osdmap e80: 8 total, 8 up, 8 in 2021-03-24T21:52:30.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:30 smithi125 conmon[24927]: cluster 2021-03-24T21:52:28.651218+0000 mgr.y (mgr.14144) 2021-03-24T21:52:30.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:30 smithi125 conmon[24927]: 227 : cluster [DBG] pgmap v230: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-24T21:52:30.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:30 smithi125 conmon[24927]: cluster 2021-03-24T21:52:28.967072+0000 osd.1 (osd.1) 7 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:52:30.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:30 smithi125 conmon[24927]: cluster 2021-03-24T21:52:28.967286+0000 osd.1 (osd.1) 8 : cluster [DBG] 2021-03-24T21:52:30.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:30 smithi125 conmon[24927]: purged_snaps scrub ok 2021-03-24T21:52:30.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:30 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:52:30.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:30 smithi125 conmon[24927]: 24T21:52:29.042535+0000 mon.a (mon.0) 547 : cluster [DBG] osdmap e79: 8 total, 8 up, 8 in 2021-03-24T21:52:30.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:30 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:52:30.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:30 smithi125 conmon[24927]: 21:52:30.032553+0000 mon.a (mon.0) 548 : cluster 2021-03-24T21:52:30.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:30 smithi125 conmon[24927]: [DBG] osdmap e80: 8 total, 8 up, 8 in 2021-03-24T21:52:32.299 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:32 smithi125 conmon[19936]: cluster 2021-03-24T21:52:30.579664+0000 osd.4 (osd.4) 5 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:52:32.299 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:32 smithi125 conmon[19936]: cluster 2021-03-24T21:52:30.579850+0000 osd.4 (osd.4) 6 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:52:32.300 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:32 smithi125 conmon[19936]: cluster 2021-03-24T21:52:32.300 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:32 smithi125 conmon[19936]: 2021-03-24T21:52:30.651978+0000 mgr.y (mgr. 2021-03-24T21:52:32.300 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:32 smithi125 conmon[19936]: 14144) 228 : cluster [DBG] pgmap v233: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-24T21:52:32.300 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:32 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:52:32.301 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:32 smithi125 conmon[19936]: 03-24T21:52:31.055724+0000 mon.a 2021-03-24T21:52:32.301 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:32 smithi125 conmon[19936]: (mon.0) 549 : cluster [DBG] 2021-03-24T21:52:32.301 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:32 smithi125 conmon[19936]: osdmap e81: 8 total, 8 up, 8 in 2021-03-24T21:52:32.301 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:32 smithi125 conmon[24927]: cluster 2021-03-24T21:52:30.579664+0000 2021-03-24T21:52:32.302 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:32 smithi125 conmon[24927]: osd.4 (osd.4) 5 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:52:32.302 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:32 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:52:32.302 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:32 smithi125 conmon[24927]: :52:30.579850+0000 osd.4 (osd.4) 6 : cluster 2021-03-24T21:52:32.302 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:32 smithi125 conmon[24927]: [DBG] purged_snaps scrub ok 2021-03-24T21:52:32.303 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:32 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:52:32.303 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:32 smithi125 conmon[24927]: -03-24T21:52:30 2021-03-24T21:52:32.303 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:32 smithi125 conmon[24927]: .651978+0000 mgr.y (mgr.14144 2021-03-24T21:52:32.303 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:32 smithi125 conmon[24927]: ) 228 : cluster [DBG] pgmap v233: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-24T21:52:32.303 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:32 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:52:32.304 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:32 smithi125 conmon[24927]: 03-24T21:52:31.055724+0000 mon.a 2021-03-24T21:52:32.304 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:32 smithi125 conmon[24927]: (mon.0) 549 : cluster [DBG] osdmap e81: 8 total, 8 up, 8 in 2021-03-24T21:52:32.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:32 smithi142 conmon[20125]: cluster 2021-03-24T21:52:30.579664+0000 osd.4 (osd.4) 5 : cluster 2021-03-24T21:52:32.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:32 smithi142 conmon[20125]: [DBG] purged_snaps scrub starts 2021-03-24T21:52:32.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:32 smithi142 conmon[20125]: cluster 2021-03-24T21:52:30.579850+0000 osd.4 (osd. 2021-03-24T21:52:32.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:32 smithi142 conmon[20125]: 4) 6 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:52:32.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:32 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:52:32.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:32 smithi142 conmon[20125]: -03-24T21:52:30.651978+0000 2021-03-24T21:52:32.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:32 smithi142 conmon[20125]: mgr.y (mgr.14144) 228 : cluster 2021-03-24T21:52:32.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:32 smithi142 conmon[20125]: [DBG] pgmap v233: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-24T21:52:32.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:32 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:52:32.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:32 smithi142 conmon[20125]: T21:52:31.055724+0000 mon.a (mon.0) 549 : cluster [DBG] osdmap e81: 8 total, 8 up, 8 in 2021-03-24T21:52:32.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:32 smithi142 conmon[20125]: 2021-03-24T21:52:32.391 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 32 2021-03-24T21:52:33.324 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:33 smithi125 conmon[19936]: audit 2021-03-24T21:52:32.392953 2021-03-24T21:52:33.324 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:33 smithi125 conmon[19936]: +0000 mon.c (mon.2) 13 : audit [DBG] from='client.? v1:172.21.15.125:0/1779567826' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".rgw.root", "var": "pg_num"}]: dispatch 2021-03-24T21:52:33.325 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:33 smithi125 conmon[24927]: audit 2021-03-24T21:52:32.392953+0000 2021-03-24T21:52:33.325 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:33 smithi125 conmon[24927]: mon.c (mon.2) 13 : audit [DBG] from='client.? v1:172.21.15.125:0/1779567826' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".rgw.root", "var": "pg_num"}]: dispatch 2021-03-24T21:52:33.326 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd pool get default.rgw.log pg_num 2021-03-24T21:52:33.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:33 smithi142 conmon[20125]: audit 2021-03-24T21:52:32.392953+0000 mon.c (mon.2) 13 2021-03-24T21:52:33.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:33 smithi142 conmon[20125]: : audit [DBG] from='client.? v1:172.21.15.125:0/1779567826' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".rgw.root", "var": "pg_num"}]: dispatch 2021-03-24T21:52:33.624 INFO:journalctl@ceph.alertmanager.a.smithi125.stdout:Mar 24 21:52:33 smithi125 podman[43346]: 2021-03-24 21:52:33.559189927 +0000 UTC m=+0.165742192 volume create 126442d08ccc7f97f30d2ca9a4aa84ec4a9f9183c23cf478ccbec0c527b8bdea 2021-03-24T21:52:33.856 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:52:33.876 INFO:journalctl@ceph.alertmanager.a.smithi125.stdout:Mar 24 21:52:33 smithi125 podman[43346]: 2021-03-24 21:52:33.62599159 +0000 UTC m=+0.232543924 container create 667e2697a3313686f9a894caa448dbda821e104361686019b600081da215c9bd (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-alertmanager.a, maintainer=The Prometheus Authors ) 2021-03-24T21:52:33.877 INFO:journalctl@ceph.alertmanager.a.smithi125.stdout:Mar 24 21:52:33 smithi125 podman[43346]: 2021-03-24 21:52:33.809219587 +0000 UTC m=+0.415771927 container init 667e2697a3313686f9a894caa448dbda821e104361686019b600081da215c9bd (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-alertmanager.a, maintainer=The Prometheus Authors ) 2021-03-24T21:52:33.877 INFO:journalctl@ceph.alertmanager.a.smithi125.stdout:Mar 24 21:52:33 smithi125 podman[43346]: 2021-03-24 21:52:33.834172046 +0000 UTC m=+0.440724330 container start 667e2697a3313686f9a894caa448dbda821e104361686019b600081da215c9bd (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-alertmanager.a, maintainer=The Prometheus Authors ) 2021-03-24T21:52:33.877 INFO:journalctl@ceph.alertmanager.a.smithi125.stdout:Mar 24 21:52:33 smithi125 bash[43346]: 667e2697a3313686f9a894caa448dbda821e104361686019b600081da215c9bd 2021-03-24T21:52:33.877 INFO:journalctl@ceph.alertmanager.a.smithi125.stdout:Mar 24 21:52:33 smithi125 systemd[1]: Started Ceph alertmanager.a for 3cfa339a-8cea-11eb-90b2-001a4aab830c. 2021-03-24T21:52:33.878 INFO:journalctl@ceph.alertmanager.a.smithi125.stdout:Mar 24 21:52:33 smithi125 conmon[43414]: level=info ts=2021-03-24T21:52:33.847Z caller=main.go:231 msg="Starting Alertmanager" version="(version=0.20.0, branch=HEAD, revision=f74be0400a6243d10bb53812d6fa408ad71ff32d)" 2021-03-24T21:52:33.878 INFO:journalctl@ceph.alertmanager.a.smithi125.stdout:Mar 24 21:52:33 smithi125 conmon[43414]: level=info ts=2021-03-24T21:52:33.847Z caller=main.go:232 build_context="(go=go1.13.5, user=root@00c3106655f8, date=20191211-14:13:14)" 2021-03-24T21:52:33.878 INFO:journalctl@ceph.alertmanager.a.smithi125.stdout:Mar 24 21:52:33 smithi125 conmon[43414]: level=info ts=2021-03-24T21:52:33.850Z caller=cluster.go:161 component=cluster msg="setting advertise address explicitly" addr=172.21.15.125 port=9094 2021-03-24T21:52:33.878 INFO:journalctl@ceph.alertmanager.a.smithi125.stdout:Mar 24 21:52:33 smithi125 conmon[43414]: level=info ts=2021-03-24T21:52:33.851Z caller=cluster.go:623 component=cluster msg="Waiting for gossip to settle..." interval=2s 2021-03-24T21:52:33.879 INFO:journalctl@ceph.alertmanager.a.smithi125.stdout:Mar 24 21:52:33 smithi125 conmon[43414]: level=info ts=2021-03-24T21:52:33.878Z caller=coordinator.go:119 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2021-03-24T21:52:34.158 INFO:journalctl@ceph.alertmanager.a.smithi125.stdout:Mar 24 21:52:33 smithi125 conmon[43414]: level=info ts=2021-03-24T21:52:33.878Z caller=coordinator.go:131 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2021-03-24T21:52:34.158 INFO:journalctl@ceph.alertmanager.a.smithi125.stdout:Mar 24 21:52:33 smithi125 conmon[43414]: level=info ts=2021-03-24T21:52:33.882Z caller=main.go:497 msg=Listening address=:9093 2021-03-24T21:52:34.159 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:34 smithi125 conmon[19936]: cluster 2021-03-24T21:52 2021-03-24T21:52:34.159 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:34 smithi125 conmon[19936]: :32.652684+0000 mgr.y (mgr.14144) 229 : cluster [DBG] pgmap v235: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-03-24T21:52:34.159 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:34 smithi125 conmon[19936]: audit 2021-03-24T21:52:33.883783+0000 2021-03-24T21:52:34.159 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:34 smithi125 conmon[19936]: mon.a (mon.0) 550 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:52:34.160 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:34 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:52:34.160 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:34 smithi125 conmon[24927]: 52:32.652684+0000 mgr.y (mgr.14144) 229 : cluster [DBG] pgmap v235: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-03-24T21:52:34.160 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:34 smithi125 conmon[24927]: audit 2021-03-24T21:52:33.883783+0000 mon.a (mon.0) 550 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:52:34.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:34 smithi142 conmon[20125]: cluster 2021-03-24T21:52:32.652684+0000 mgr.y ( 2021-03-24T21:52:34.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:34 smithi142 conmon[20125]: mgr.14144) 229 : cluster [DBG] pgmap v235: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-03-24T21:52:34.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:34 smithi142 conmon[20125]: audit 2021-03-24T21:52:33.883783+0000 mon.a (mon.0) 2021-03-24T21:52:34.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:34 smithi142 conmon[20125]: 550 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:52:35.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:35 smithi142 conmon[20125]: audit 2021-03-24T21:52 2021-03-24T21:52:35.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:35 smithi142 conmon[20125]: :34.102615+0000 mon.a (mon.0) 551 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:52:35.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:35 smithi142 conmon[20125]: audit 2021-03-24 2021-03-24T21:52:35.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:35 smithi142 conmon[20125]: T21:52:34.110038+0000 mon.a (mon.0) 552 : audit 2021-03-24T21:52:35.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:35 smithi142 conmon[20125]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:52:35.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:35 smithi142 conmon[20125]: audit 2021-03-24T21:52 2021-03-24T21:52:35.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:35 smithi142 conmon[20125]: :34.112097+0000 mon.a (mon.0) 2021-03-24T21:52:35.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:35 smithi142 conmon[20125]: 553 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2021-03-24T21:52:35.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:35 smithi142 conmon[20125]: audit 2021-03-24 2021-03-24T21:52:35.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:35 smithi142 conmon[20125]: T21:52:34.112862+0000 2021-03-24T21:52:35.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:35 smithi142 conmon[20125]: mgr.y (mgr.14144) 230 : 2021-03-24T21:52:35.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:35 smithi142 conmon[20125]: audit [DBG] from='mon.0 v1:172.21.15.125:0/3961911888' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2021-03-24T21:52:35.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:35 smithi142 conmon[20125]: audit 2021-03-24T21: 2021-03-24T21:52:35.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:35 smithi142 conmon[20125]: 52:34.119997+0000 mon.a (mon.0) 554 : audit 2021-03-24T21:52:35.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:35 smithi142 conmon[20125]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:52:35.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:35 smithi142 conmon[20125]: cephadm 2021-03-24T21:52:35.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:35 smithi142 conmon[20125]: 2021-03-24T21:52:34.125247+0000 2021-03-24T21:52:35.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:35 smithi142 conmon[20125]: mgr.y (mgr.14144) 231 : cephadm [INF] 2021-03-24T21:52:35.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:35 smithi142 conmon[20125]: Deploying daemon grafana.a on smithi142 2021-03-24T21:52:35.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:35 smithi142 conmon[20125]: cluster 2021-03-24T21:52:35.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:35 smithi142 conmon[20125]: 2021-03-24T21:52:35.038570+0000 2021-03-24T21:52:35.373 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:35 smithi142 conmon[20125]: mon.a (mon.0) 555 : cluster [DBG] osdmap e82: 8 total, 8 up, 8 in 2021-03-24T21:52:35.373 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:35 smithi142 conmon[20125]: 2021-03-24T21:52:35.394 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:35 smithi125 conmon[24927]: audit 2021-03-24T21: 2021-03-24T21:52:35.395 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:35 smithi125 conmon[24927]: 52:34.102615+0000 mon.a (mon.0) 551 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:52:35.395 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:35 smithi125 conmon[24927]: audit 2021-03-24T21:52 2021-03-24T21:52:35.395 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:35 smithi125 conmon[24927]: :34.110038+0000 mon.a (mon.0) 552 : audit 2021-03-24T21:52:35.396 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:35 smithi125 conmon[24927]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:52:35.396 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:35 smithi125 conmon[24927]: audit 2021-03-24T 2021-03-24T21:52:35.396 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:35 smithi125 conmon[24927]: 21:52:34.112097+0000 mon.a (mon. 2021-03-24T21:52:35.396 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:35 smithi125 conmon[24927]: 0) 553 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2021-03-24T21:52:35.396 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:35 smithi125 conmon[24927]: audit 2021 2021-03-24T21:52:35.397 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:35 smithi125 conmon[24927]: -03-24T21:52:34.112862+0000 2021-03-24T21:52:35.397 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:35 smithi125 conmon[24927]: mgr.y (mgr.14144) 230 : audit [DBG] 2021-03-24T21:52:35.397 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:35 smithi125 conmon[24927]: from='mon.0 v1:172.21.15.125:0/3961911888' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2021-03-24T21:52:35.397 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:35 smithi125 conmon[24927]: audit 2021- 2021-03-24T21:52:35.397 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:35 smithi125 conmon[24927]: 03-24T21:52:34.119997+0000 2021-03-24T21:52:35.398 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:35 smithi125 conmon[24927]: mon.a (mon.0) 554 : audit [INF] 2021-03-24T21:52:35.398 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:35 smithi125 conmon[24927]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:52:35.398 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:35 smithi125 conmon[24927]: cephadm 2021- 2021-03-24T21:52:35.398 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:35 smithi125 conmon[24927]: 03-24T21:52:34.125247+0000 2021-03-24T21:52:35.398 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:35 smithi125 conmon[24927]: mgr.y (mgr.14144) 231 : cephadm [INF] Deploying daemon grafana.a on smithi142 2021-03-24T21:52:35.399 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:35 smithi125 conmon[24927]: 2021-03-24T21:52:35.399 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:35 smithi125 conmon[24927]: cluster 2021-03-24T21:52:35.399 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:35 smithi125 conmon[24927]: 2021-03-24T21:52:35.038570+0000 2021-03-24T21:52:35.399 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:35 smithi125 conmon[24927]: mon.a (mon.0) 555 : cluster [DBG] osdmap e82: 8 total, 8 up, 8 in 2021-03-24T21:52:35.399 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:35 smithi125 conmon[24927]: 2021-03-24T21:52:35.400 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:35 smithi125 conmon[19936]: audit 2021-03-24T21 2021-03-24T21:52:35.400 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:35 smithi125 conmon[19936]: :52:34.102615+0000 mon.a (mon.0) 551 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:52:35.400 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:35 smithi125 conmon[19936]: audit 2021-03-24T21:52:35.400 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:35 smithi125 conmon[19936]: 2021-03-24T21:52:34.110038+0000 mon.a (mon.0) 552 2021-03-24T21:52:35.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:35 smithi125 conmon[19936]: : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:52:35.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:35 smithi125 conmon[19936]: audit 2021-03-24T21:52 2021-03-24T21:52:35.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:35 smithi125 conmon[19936]: :34.112097+0000 mon.a (mon.0) 553 : audit [INF] 2021-03-24T21:52:35.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:35 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2021-03-24T21:52:35.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:35 smithi125 conmon[19936]: audit 2021-03-24T21:52:35.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:35 smithi125 conmon[19936]: 2021-03-24T21:52:34.112862+0000 mgr.y (mgr.14144) 230 : 2021-03-24T21:52:35.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:35 smithi125 conmon[19936]: audit [DBG] from='mon.0 v1:172.21.15.125:0/3961911888' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2021-03-24T21:52:35.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:35 smithi125 conmon[19936]: audit 2021-03-24T21:52:35.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:35 smithi125 conmon[19936]: 2021-03- 2021-03-24T21:52:35.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:35 smithi125 conmon[19936]: 24T21:52:34.119997+0000 mon.a (mon.0) 2021-03-24T21:52:35.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:35 smithi125 conmon[19936]: 554 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:52:35.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:35 smithi125 conmon[19936]: cephadm 2021-03-24T21:52:34 2021-03-24T21:52:35.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:35 smithi125 conmon[19936]: .125247+0000 mgr.y (mgr.14144) 231 : cephadm [INF] Deploying daemon grafana.a on smithi142 2021-03-24T21:52:35.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:35 smithi125 conmon[19936]: 2021-03-24T21:52:35.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:35 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:52:35.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:35 smithi125 conmon[19936]: 24T21:52:35.038570+0000 mon.a (mon.0) 555 : 2021-03-24T21:52:35.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:35 smithi125 conmon[19936]: cluster [DBG] osdmap e82: 8 total, 8 up, 8 in 2021-03-24T21:52:35.856 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 32 2021-03-24T21:52:36.121 INFO:journalctl@ceph.alertmanager.a.smithi125.stdout:Mar 24 21:52:35 smithi125 conmon[43414]: level=info ts=2021-03-24T21:52:35.851Z caller=cluster.go:648 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000136552s 2021-03-24T21:52:36.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:36 smithi125 conmon[19936]: cluster 2021-03-24T21:52:34. 2021-03-24T21:52:36.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:36 smithi125 conmon[19936]: 654028+0000 mgr.y (mgr.14144) 232 : cluster [DBG] pgmap v236: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2021-03-24T21:52:36.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:36 smithi125 conmon[19936]: audit 2021-03-24T21:52:35. 2021-03-24T21:52:36.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:36 smithi125 conmon[19936]: 856222+0000 mon.b (mon.1) 30 : audit [DBG] from='client.? v1:172.21.15.125:0/3656960179' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.log", "var": "pg_num"}]: dispatch 2021-03-24T21:52:36.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:36 smithi125 conmon[24927]: cluster 2021-03-24T21:52:36.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:36 smithi125 conmon[24927]: 2021-03-24T21:52:34.654028+0000 mgr.y (mgr.14144) 232 : cluster [DBG] pgmap v236: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2021-03-24T21:52:36.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:36 smithi125 conmon[24927]: audit 2021-03-24T21:52:36.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:36 smithi125 conmon[24927]: 2021-03-24T21:52:35 2021-03-24T21:52:36.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:36 smithi125 conmon[24927]: .856222+0000 mon.b (mon.1) 30 : audit 2021-03-24T21:52:36.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:36 smithi125 conmon[24927]: [DBG] from='client.? v1:172.21.15.125:0/3656960179' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.log", "var": "pg_num"}]: dispatch 2021-03-24T21:52:36.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:36 smithi142 conmon[20125]: cluster 2021-03-24T21:52:34. 2021-03-24T21:52:36.906 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:36 smithi142 conmon[20125]: 654028+0000 mgr.y (mgr.14144) 232 : cluster [DBG] pgmap v236: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2021-03-24T21:52:36.906 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:36 smithi142 conmon[20125]: audit 2021-03-24T21:52:35.856222+0000 mon.b (mon.1 2021-03-24T21:52:36.907 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:36 smithi142 conmon[20125]: ) 30 : audit [DBG] from='client.? v1:172.21.15.125:0/3656960179' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.log", "var": "pg_num"}]: dispatch 2021-03-24T21:52:36.908 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd pool get default.rgw.control pg_num 2021-03-24T21:52:37.228 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:52:38.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:38 smithi125 conmon[24927]: cluster 2021-03-24T21:52:36 2021-03-24T21:52:38.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:38 smithi125 conmon[24927]: .618733+0000 osd.7 (osd.7) 4 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:52:38.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:38 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:52:38.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:38 smithi125 conmon[24927]: 21:52:36.618953+0000 osd.7 (osd.7) 5 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:52:38.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:38 smithi125 conmon[24927]: cluster 2021-03-24T21:52:36 2021-03-24T21:52:38.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:38 smithi125 conmon[24927]: .654896+0000 mgr.y (mgr.14144) 233 : cluster [DBG] pgmap v238: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 772 B/s rd, 0 op/s 2021-03-24T21:52:38.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:38 smithi125 conmon[24927]: 2021-03-24T21:52:38.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:38 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:52:38.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:38 smithi125 conmon[24927]: 52:37.135575+0000 mon.a (mon.0) 556 : cluster [DBG] osdmap e83: 8 total, 8 up, 8 in 2021-03-24T21:52:38.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:38 smithi125 conmon[19936]: cluster 2021-03-24T21:52 2021-03-24T21:52:38.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:38 smithi125 conmon[19936]: :36.618733+0000 osd.7 (osd.7) 4 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:52:38.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:38 smithi125 conmon[19936]: cluster 2021-03-24T21:52:36. 2021-03-24T21:52:38.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:38 smithi125 conmon[19936]: 618953+0000 osd.7 (osd.7) 5 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:52:38.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:38 smithi125 conmon[19936]: cluster 2021-03-24T21:52:36 2021-03-24T21:52:38.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:38 smithi125 conmon[19936]: .654896+0000 mgr.y (mgr.14144) 233 : cluster [DBG] pgmap v238: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 772 B/s rd, 0 op/s 2021-03-24T21:52:38.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:38 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:52:38.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:38 smithi125 conmon[19936]: -03-24T21:52:37.135575+0000 mon.a 2021-03-24T21:52:38.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:38 smithi125 conmon[19936]: (mon.0) 556 : cluster [DBG] osdmap e83: 8 total, 8 up, 8 in 2021-03-24T21:52:38.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:38 smithi142 conmon[20125]: cluster 2021-03-24T21:52:36 2021-03-24T21:52:38.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:38 smithi142 conmon[20125]: .618733+0000 osd.7 (osd.7) 4 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:52:38.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:38 smithi142 conmon[20125]: cluster 2021-03-24T21:52:36.618953+0000 osd.7 (osd 2021-03-24T21:52:38.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:38 smithi142 conmon[20125]: .7) 5 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:52:38.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:38 smithi142 conmon[20125]: cluster 2021-03-24T21:52:36. 2021-03-24T21:52:38.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:38 smithi142 conmon[20125]: 654896+0000 mgr.y (mgr.14144) 233 : cluster [DBG] 2021-03-24T21:52:38.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:38 smithi142 conmon[20125]: pgmap v238: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 772 B/s rd, 0 op/s 2021-03-24T21:52:38.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:38 smithi142 conmon[20125]: cluster 2021-03-24T21:52: 2021-03-24T21:52:38.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:38 smithi142 conmon[20125]: 37.135575+0000 mon.a (mon.0) 556 : 2021-03-24T21:52:38.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:38 smithi142 conmon[20125]: cluster [DBG] osdmap e83: 8 total, 8 up, 8 in 2021-03-24T21:52:39.007 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 32 2021-03-24T21:52:39.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:39 smithi125 conmon[24927]: audit 2021-03-24T21:52:39.007772+0000 mon.a 2021-03-24T21:52:39.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:39 smithi125 conmon[24927]: (mon.0) 557 : audit [DBG] from='client.? v1:172.21.15.125:0/3753910332' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.control", "var": "pg_num"}]: dispatch 2021-03-24T21:52:39.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:39 smithi125 conmon[19936]: audit 2021-03-24T21:52:39 2021-03-24T21:52:39.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:39 smithi125 conmon[19936]: .007772+0000 mon.a (mon.0) 557 : audit [DBG] from='client.? v1:172.21.15.125:0/3753910332' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.control", "var": "pg_num"}]: dispatch 2021-03-24T21:52:39.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:39 smithi142 conmon[20125]: audit 2021-03-24T21:52:39.007772 2021-03-24T21:52:39.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:39 smithi142 conmon[20125]: +0000 mon.a (mon.0) 557 : audit [DBG] from='client.? v1:172.21.15.125:0/3753910332' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.control", "var": "pg_num"}]: dispatch 2021-03-24T21:52:39.814 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd pool get default.rgw.meta pg_num 2021-03-24T21:52:40.124 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:52:40.145 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:40 smithi125 conmon[19936]: cluster 2021-03-24T21:52:38. 2021-03-24T21:52:40.145 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:40 smithi125 conmon[19936]: 655978+0000 mgr.y (mgr.14144) 234 : cluster [DBG] pgmap v240: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 673 B/s rd, 0 op/s 2021-03-24T21:52:40.145 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:40 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:52:40.145 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:40 smithi125 conmon[19936]: 03-24T21:52:40.035955+0000 mon.a (mon.0) 558 : cluster [DBG] osdmap e84: 8 total, 8 up, 8 in 2021-03-24T21:52:40.146 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:40 smithi125 conmon[24927]: cluster 2021-03-24T21:52:40.146 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:40 smithi125 conmon[24927]: 2021-03-24T21:52:38.655978+0000 mgr.y (mgr.14144) 234 : cluster 2021-03-24T21:52:40.146 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:40 smithi125 conmon[24927]: [DBG] pgmap v240: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 673 B/s rd, 0 op/s 2021-03-24T21:52:40.146 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:40 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:52:40.147 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:40 smithi125 conmon[24927]: -24T21:52:40.035955+0000 2021-03-24T21:52:40.147 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:40 smithi125 conmon[24927]: mon.a (mon.0) 558 : cluster [DBG] osdmap e84: 8 total, 8 up, 8 in 2021-03-24T21:52:40.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:40 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:52:40.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:40 smithi142 conmon[20125]: 52:38.655978+0000 mgr.y (mgr.14144) 234 : cluster [DBG] pgmap v240: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 673 B/s rd, 0 op/s 2021-03-24T21:52:40.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:40 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:52:40.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:40 smithi142 conmon[20125]: 24T21:52:40.035955+0000 mon.a (mon.0) 558 2021-03-24T21:52:40.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:40 smithi142 conmon[20125]: : cluster [DBG] osdmap e84: 8 total, 8 up, 8 in 2021-03-24T21:52:42.087 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 32 2021-03-24T21:52:42.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:42 smithi125 conmon[19936]: cluster 2021-03-24T21:52:40 2021-03-24T21:52:42.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:42 smithi125 conmon[19936]: .656782+0000 mgr.y (mgr.14144) 235 : cluster [DBG] pgmap v242: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-03-24T21:52:42.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:42 smithi125 conmon[19936]: audit 2021-03-24T21:52:42.088129+0000 mon.a (mon.0) 559 : audit [DBG] 2021-03-24T21:52:42.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:42 smithi125 conmon[19936]: from='client.? v1:172.21.15.125:0/2862052045' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.meta", "var": "pg_num"}]: dispatch 2021-03-24T21:52:42.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:42 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:52:42.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:42 smithi125 conmon[24927]: 52:40.656782+0000 mgr.y (mgr.14144) 235 : cluster [DBG] pgmap v242: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-03-24T21:52:42.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:42 smithi125 conmon[24927]: audit 2021-03-24T21:52:42.088129+0000 2021-03-24T21:52:42.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:42 smithi125 conmon[24927]: mon.a (mon.0) 559 : audit [DBG] from='client.? v1:172.21.15.125:0/2862052045' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.meta", "var": "pg_num"}]: dispatch 2021-03-24T21:52:42.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:42 smithi142 conmon[20125]: cluster 2021-03-24T21:52:40.656782 2021-03-24T21:52:42.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:42 smithi142 conmon[20125]: +0000 mgr.y (mgr.14144) 235 : cluster [DBG] pgmap v242: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-03-24T21:52:42.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:42 smithi142 conmon[20125]: audit 2021-03-24T21:52:42.088129+0000 mon.a (mon.0) 559 : audit [DBG] from='client.? v1:172.21.15.125:0/2862052045' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.meta", "var": "pg_num"}]: dispatch 2021-03-24T21:52:42.896 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2021-03-24T21:52:42.897 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2021-03-24T21:52:42.897 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph mgr dump --format=json 2021-03-24T21:52:43.198 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:52:44.152 INFO:journalctl@ceph.alertmanager.a.smithi125.stdout:Mar 24 21:52:43 smithi125 conmon[43414]: level=info ts=2021-03-24T21:52:43.852Z caller=cluster.go:640 component=cluster msg="gossip settled; proceeding" elapsed=10.000865281s 2021-03-24T21:52:44.153 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:44 smithi125 conmon[19936]: cluster 2021-03-24T21:52:42.657549 2021-03-24T21:52:44.153 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:44 smithi125 conmon[19936]: +0000 mgr.y (mgr.14144) 236 : cluster [DBG] pgmap v243: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 671 B/s rd, 0 op/s 2021-03-24T21:52:44.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:44 smithi125 conmon[24927]: cluster 2021-03-24T21:52: 2021-03-24T21:52:44.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:44 smithi125 conmon[24927]: 42.657549+0000 mgr.y (mgr.14144) 236 : cluster [DBG] pgmap v243: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 671 B/s rd, 0 op/s 2021-03-24T21:52:44.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:44 smithi142 conmon[20125]: cluster 2021-03-24T21:52:42.657549+0000 2021-03-24T21:52:44.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:44 smithi142 conmon[20125]: mgr.y (mgr.14144) 236 : cluster [DBG] pgmap v243: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 671 B/s rd, 0 op/s 2021-03-24T21:52:45.189 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-24T21:52:45.978 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":15,"active_gid":14144,"active_name":"y","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6800","nonce":1485106819}]},"active_addr":"172.21.15.125:6800/1485106819","active_change":"2021-03-24T21:47:04.502770+0000","active_mgr_features":4540138297136906239,"available":true,"standbys":[{"gid":24115,"name":"x","mgr_features":4540138297136906239,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"haproxy","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"haproxy","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-mgr.y:8443/"},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":3,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.125:0","nonce":4130482302}]},{"addrvec":[{"type":"v2","addr":"172.21.15.125:0","nonce":961196040}]},{"addrvec":[{"type":"v2","addr":"172.21.15.125:0","nonce":1824394534}]}]}} 2021-03-24T21:52:45.985 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2021-03-24T21:52:45.985 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2021-03-24T21:52:45.985 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd dump --format=json 2021-03-24T21:52:46.300 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:52:46.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:46 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:52:46.461 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:46 smithi125 conmon[19936]: -24T21:52:44.658660+0000 mgr.y (mgr.14144) 237 : cluster 2021-03-24T21:52:46.461 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:46 smithi125 conmon[19936]: [DBG] pgmap v244: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-24T21:52:46.461 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:46 smithi125 conmon[19936]: audit 2021-03-24T21:52:46.461 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:46 smithi125 conmon[19936]: 2021-03- 2021-03-24T21:52:46.462 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:46 smithi125 conmon[19936]: 24T21: 2021-03-24T21:52:46.462 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:46 smithi125 conmon[19936]: 52:45. 2021-03-24T21:52:46.462 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:46 smithi125 conmon[19936]: 178562+0000 mon.a 2021-03-24T21:52:46.462 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:46 smithi125 conmon[19936]: (mon.0) 2021-03-24T21:52:46.463 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:46 smithi125 conmon[19936]: 560 : audit [DBG] 2021-03-24T21:52:46.463 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:46 smithi125 conmon[19936]: from='client.? v1:172.21.15.125:0/3853952150' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2021-03-24T21:52:46.463 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:46 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:52:46.463 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:46 smithi125 conmon[19936]: :52:45.797860+0000 osd.5 (osd.5) 6 : cluster [DBG] 3.1c scrub ok 2021-03-24T21:52:46.464 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:46 smithi125 conmon[24927]: cluster 2021-03-24T21:52:44.658660+0000 mgr.y (mgr.14144) 237 : 2021-03-24T21:52:46.464 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:46 smithi125 conmon[24927]: cluster [DBG] pgmap v244: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-24T21:52:46.464 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:46 smithi125 conmon[24927]: audit 2021-03-24T21:52:45.178562+0000 mon.a (mon.0) 560 : audit [DBG] from='client.? v1:172.21.15.125:0/3853952150' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2021-03-24T21:52:46.465 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:46 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:52:46.465 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:46 smithi125 conmon[24927]: 24T21:52:45.797860+0000 osd.5 (osd.5) 6 : cluster [DBG] 3.1c scrub ok 2021-03-24T21:52:46.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:46 smithi142 conmon[20125]: cluster 2021-03-24T21:52:44 2021-03-24T21:52:46.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:46 smithi142 conmon[20125]: .658660+0000 mgr.y (mgr.14144) 237 : cluster [DBG] pgmap v244: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-24T21:52:46.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:46 smithi142 conmon[20125]: audit 2021-03-24T21:52:45.178562+0000 mon.a 2021-03-24T21:52:46.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:46 smithi142 conmon[20125]: (mon.0) 560 : audit [DBG] from='client.? v1:172.21.15.125:0/3853952150' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2021-03-24T21:52:46.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:46 smithi142 conmon[20125]: cluster 2021-03-24T21:52:46.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:46 smithi142 conmon[20125]: 2021-03-24T21:52:45.797860+0000 osd.5 (osd.5) 6 : cluster 2021-03-24T21:52:46.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:46 smithi142 conmon[20125]: [DBG] 3.1c scrub ok 2021-03-24T21:52:47.480 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:47 smithi125 conmon[19936]: cluster 2021-03-24T21:52:46.620306 2021-03-24T21:52:47.481 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:47 smithi125 conmon[19936]: +0000 osd.3 (osd.3) 7 : cluster [DBG] 3.b scrub ok 2021-03-24T21:52:47.481 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:47 smithi125 conmon[24927]: cluster 2021-03-24T21:52:46.620306+0000 osd.3 (osd.3) 7 : 2021-03-24T21:52:47.482 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:47 smithi125 conmon[24927]: cluster [DBG] 3.b scrub ok 2021-03-24T21:52:47.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:47 smithi142 conmon[20125]: cluster 2021-03-24T21:52:46. 2021-03-24T21:52:47.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:47 smithi142 conmon[20125]: 620306+0000 osd.3 (osd.3) 7 : cluster [DBG] 3.b scrub ok 2021-03-24T21:52:48.178 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-24T21:52:48.179 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":84,"fsid":"3cfa339a-8cea-11eb-90b2-001a4aab830c","created":"2021-03-24T21:46:19.163312+0000","modified":"2021-03-24T21:52:40.029101+0000","last_up_change":"2021-03-24T21:51:38.283730+0000","last_in_change":"2021-03-24T21:51:20.214398+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-24T21:49:51.001357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"iscsi","create_time":"2021-03-24T21:51:39.190981+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":".rgw.root","create_time":"2021-03-24T21:51:44.159145+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2021-03-24T21:51:46.090924+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2021-03-24T21:51:47.842521+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2021-03-24T21:51:50.730906+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"1f8f9a69-4b85-4054-8b11-36d35574494c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6801","nonce":1197472172}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6802","nonce":1197472172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6804","nonce":1197472172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6803","nonce":1197472172}]},"public_addr":"172.21.15.125:6801/1197472172","cluster_addr":"172.21.15.125:6802/1197472172","heartbeat_back_addr":"172.21.15.125:6804/1197472172","heartbeat_front_addr":"172.21.15.125:6803/1197472172","state":["exists","up"]},{"osd":1,"uuid":"48db0723-5806-4e5c-9919-f3d7f52ad169","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6805","nonce":113845288}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6806","nonce":113845288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6808","nonce":113845288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6807","nonce":113845288}]},"public_addr":"172.21.15.125:6805/113845288","cluster_addr":"172.21.15.125:6806/113845288","heartbeat_back_addr":"172.21.15.125:6808/113845288","heartbeat_front_addr":"172.21.15.125:6807/113845288","state":["exists","up"]},{"osd":2,"uuid":"d6a361f8-4e4d-4f46-8ab9-01bf71dbb2c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6809","nonce":246939163}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6810","nonce":246939163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6812","nonce":246939163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6811","nonce":246939163}]},"public_addr":"172.21.15.125:6809/246939163","cluster_addr":"172.21.15.125:6810/246939163","heartbeat_back_addr":"172.21.15.125:6812/246939163","heartbeat_front_addr":"172.21.15.125:6811/246939163","state":["exists","up"]},{"osd":3,"uuid":"716a41ce-7a28-400f-bf75-4c1ad99c7f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6813","nonce":2209636173}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6814","nonce":2209636173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6816","nonce":2209636173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6815","nonce":2209636173}]},"public_addr":"172.21.15.125:6813/2209636173","cluster_addr":"172.21.15.125:6814/2209636173","heartbeat_back_addr":"172.21.15.125:6816/2209636173","heartbeat_front_addr":"172.21.15.125:6815/2209636173","state":["exists","up"]},{"osd":4,"uuid":"f9bdd8d2-b292-49da-ad4f-99682122f543","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.142:6800","nonce":1816503330}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.142:6801","nonce":1816503330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.142:6803","nonce":1816503330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.142:6802","nonce":1816503330}]},"public_addr":"172.21.15.142:6800/1816503330","cluster_addr":"172.21.15.142:6801/1816503330","heartbeat_back_addr":"172.21.15.142:6803/1816503330","heartbeat_front_addr":"172.21.15.142:6802/1816503330","state":["exists","up"]},{"osd":5,"uuid":"ec0d43aa-6905-43f0-9e32-e19d4cf5129d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.142:6804","nonce":396400804}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.142:6805","nonce":396400804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.142:6807","nonce":396400804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.142:6806","nonce":396400804}]},"public_addr":"172.21.15.142:6804/396400804","cluster_addr":"172.21.15.142:6805/396400804","heartbeat_back_addr":"172.21.15.142:6807/396400804","heartbeat_front_addr":"172.21.15.142:6806/396400804","state":["exists","up"]},{"osd":6,"uuid":"a26a487e-ed26-4d92-98ea-896739302921","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.142:6808","nonce":4048483913}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.142:6809","nonce":4048483913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.142:6811","nonce":4048483913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.142:6810","nonce":4048483913}]},"public_addr":"172.21.15.142:6808/4048483913","cluster_addr":"172.21.15.142:6809/4048483913","heartbeat_back_addr":"172.21.15.142:6811/4048483913","heartbeat_front_addr":"172.21.15.142:6810/4048483913","state":["exists","up"]},{"osd":7,"uuid":"682ba5ce-0172-48e3-82f8-57a3f98c911b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.142:6812","nonce":3678129570}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.142:6813","nonce":3678129570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.142:6815","nonce":3678129570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.142:6814","nonce":3678129570}]},"public_addr":"172.21.15.142:6812/3678129570","cluster_addr":"172.21.15.142:6813/3678129570","heartbeat_back_addr":"172.21.15.142:6815/3678129570","heartbeat_front_addr":"172.21.15.142:6814/3678129570","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T21:52:07.610589+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T21:52:28.967293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T21:51:49.809720+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T21:52:11.677778+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T21:52:30.579857+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T21:51:52.834140+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T21:52:15.330965+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T21:52:36.618960+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.1c","mappings":[{"from":1,"to":5}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:0/3077905055":"2021-03-25T21:47:04.502055+0000","172.21.15.125:0/2210544773":"2021-03-25T21:47:04.502055+0000","172.21.15.125:6800/3176807740":"2021-03-25T21:47:04.502055+0000","172.21.15.125:6800/3531208850":"2021-03-25T21:46:43.197724+0000","172.21.15.125:0/3337261959":"2021-03-25T21:46:43.197724+0000","172.21.15.125:0/1919150488":"2021-03-25T21:46:43.197724+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-03-24T21:52:48.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:48 smithi142 conmon[20125]: cluster 2021-03-24T21:52:46. 2021-03-24T21:52:48.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:48 smithi142 conmon[20125]: 659540+0000 mgr.y (mgr.14144) 238 : cluster [DBG] pgmap v245: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-03-24T21:52:48.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:48 smithi142 conmon[20125]: cluster 2021-03-24T21:52:46.847223+0000 osd.7 (osd.7 2021-03-24T21:52:48.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:48 smithi142 conmon[20125]: ) 6 : cluster [DBG] 3.e scrub ok 2021-03-24T21:52:48.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:48 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:52:48.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:48 smithi142 conmon[20125]: 21:52:47.126820+0000 osd.6 (osd.6) 5 : cluster [DBG] 4.a scrub ok 2021-03-24T21:52:48.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:48 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:52:48.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:48 smithi142 conmon[20125]: T21:52:47.514419+0000 osd.2 (osd.2) 7 2021-03-24T21:52:48.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:48 smithi142 conmon[20125]: : cluster [DBG] 2.1 scrub ok 2021-03-24T21:52:48.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:48 smithi142 conmon[20125]: cluster 2021-03-24T21:52:47.709428+0000 2021-03-24T21:52:48.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:48 smithi142 conmon[20125]: osd.5 (osd.5) 7 : cluster [DBG] 3.c scrub ok 2021-03-24T21:52:48.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:48 smithi142 conmon[20125]: 2021-03-24T21:52:48.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[19936]: cluster 2021-03-24T21:52 2021-03-24T21:52:48.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[19936]: :46.659540+0000 mgr.y (mgr.14144) 238 : cluster [DBG] pgmap v245: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-03-24T21:52:48.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[19936]: cluster 2021-03-24T21:52:46 2021-03-24T21:52:48.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[19936]: .847223+0000 osd.7 (osd.7) 6 : cluster [DBG] 3.e scrub ok 2021-03-24T21:52:48.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:52:48.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[19936]: 21:52:47.126820+0000 osd.6 (osd.6) 5 : cluster [DBG] 4.a scrub ok 2021-03-24T21:52:48.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[19936]: cluster 2021-03-24T21:52:47 2021-03-24T21:52:48.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[19936]: .514419+0000 osd.2 (osd.2) 7 : cluster [DBG] 2021-03-24T21:52:48.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[19936]: 2.1 scrub ok 2021-03-24T21:52:48.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:52:48.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[19936]: -03-24T21:52:47.709428+0000 osd.5 (osd.5) 7 : cluster 2021-03-24T21:52:48.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[19936]: [DBG] 3.c scrub ok 2021-03-24T21:52:48.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:52:48.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[24927]: 03-24T21:52:46.659540+0000 mgr.y ( 2021-03-24T21:52:48.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[24927]: mgr.14144) 238 2021-03-24T21:52:48.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[24927]: : cluster [DBG] 2021-03-24T21:52:48.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[24927]: pgmap v245: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-03-24T21:52:48.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[24927]: cluster 2021-03-24T21:52:48.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[24927]: 2021-03 2021-03-24T21:52:48.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[24927]: -24T 2021-03-24T21:52:48.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[24927]: 21:52: 2021-03-24T21:52:48.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[24927]: 46.847223+0000 2021-03-24T21:52:48.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[24927]: osd.7 (osd. 2021-03-24T21:52:48.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[24927]: 7) 6 : 2021-03-24T21:52:48.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[24927]: cluster [DBG] 2021-03-24T21:52:48.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[24927]: 3.e scrub ok 2021-03-24T21:52:48.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[24927]: cluster 2021-03-24T21:52:48.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[24927]: 2021-03-24 2021-03-24T21:52:48.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[24927]: T21:52 2021-03-24T21:52:48.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[24927]: :47 2021-03-24T21:52:48.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[24927]: .126820+0000 2021-03-24T21:52:48.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[24927]: osd.6 (osd.6 2021-03-24T21:52:48.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[24927]: ) 5 : cluster 2021-03-24T21:52:48.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[24927]: [DBG] 4.a scrub ok 2021-03-24T21:52:48.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[24927]: 2021-03-24T21:52:48.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[24927]: cluster 2021-03-24T21:52:48.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[24927]: 2021-03 2021-03-24T21:52:48.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[24927]: -24T21 2021-03-24T21:52:48.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[24927]: :52: 2021-03-24T21:52:48.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[24927]: 47.514419 2021-03-24T21:52:48.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[24927]: +0000 osd.2 2021-03-24T21:52:48.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[24927]: (osd. 2021-03-24T21:52:48.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[24927]: 2) 7 : 2021-03-24T21:52:48.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[24927]: cluster [DBG] 2.1 scrub ok 2021-03-24T21:52:48.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[24927]: 2021-03-24T21:52:48.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[24927]: cluster 2021-03-24T21:52:48.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[24927]: 2021-03- 2021-03-24T21:52:48.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[24927]: 24T21: 2021-03-24T21:52:48.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[24927]: 52:47.709428 2021-03-24T21:52:48.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[24927]: +0000 osd.5 2021-03-24T21:52:48.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[24927]: (osd. 2021-03-24T21:52:48.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[24927]: 5) 7 : 2021-03-24T21:52:48.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[24927]: cluster [DBG] 3.c scrub ok 2021-03-24T21:52:48.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:48 smithi125 conmon[24927]: 2021-03-24T21:52:49.029 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2021-03-24T21:52:49.030 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd dump --format=json 2021-03-24T21:52:49.340 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:52:49.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:49 smithi125 conmon[24927]: audit 2021-03-24T21:52:48.169674+0000 mon.c (mon.2 2021-03-24T21:52:49.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:49 smithi125 conmon[24927]: ) 14 : audit [DBG] from='client.? v1:172.21.15.125:0/3124454138' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-03-24T21:52:49.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:49 smithi125 conmon[19936]: audit 2021-03-24T21 2021-03-24T21:52:49.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:49 smithi125 conmon[19936]: :52:48.169674+0000 mon.c (mon.2) 14 : audit [DBG] from='client.? v1:172.21.15.125:0/3124454138' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-03-24T21:52:49.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:49 smithi142 conmon[20125]: audit 2021-03-24T21:52:48.169674+0000 mon.c (mon.2) 14 : audit 2021-03-24T21:52:49.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:49 smithi142 conmon[20125]: [DBG] from='client.? v1:172.21.15.125:0/3124454138' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-03-24T21:52:50.492 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:50 smithi125 conmon[19936]: cluster 2021-03-24T21:52: 2021-03-24T21:52:50.493 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:50 smithi125 conmon[19936]: 48.334127+0000 osd.4 (osd.4) 7 : cluster [DBG] 3.9 scrub ok 2021-03-24T21:52:50.493 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:50 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:52:50.493 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:50 smithi125 conmon[19936]: 24T21:52:48.487778+0000 osd.0 (osd.0) 9 : cluster 2021-03-24T21:52:50.493 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:50 smithi125 conmon[19936]: [DBG] 3.1 scrub ok 2021-03-24T21:52:50.494 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:50 smithi125 conmon[19936]: cluster 2021-03-24T21:52:50.494 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:50 smithi125 conmon[19936]: 2021-03-24T21:52:48 2021-03-24T21:52:50.494 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:50 smithi125 conmon[19936]: .660702+0000 mgr.y (mgr.14144) 239 : 2021-03-24T21:52:50.494 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:50 smithi125 conmon[19936]: cluster [DBG] pgmap v246: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-24T21:52:50.494 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:50 smithi125 conmon[19936]: cluster 2021-03-24T21:52:50.495 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:50 smithi125 conmon[19936]: 2021-03-24T21:52:49 2021-03-24T21:52:50.495 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:50 smithi125 conmon[19936]: .553945+0000 osd.2 (osd.2) 8 : 2021-03-24T21:52:50.495 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:50 smithi125 conmon[19936]: cluster [DBG] 4.1c scrub ok 2021-03-24T21:52:50.496 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:50 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:52:50.496 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:50 smithi125 conmon[24927]: 21:52:48.334127+0000 osd.4 (osd.4) 7 : cluster [DBG] 3.9 scrub ok 2021-03-24T21:52:50.496 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:50 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:52:50.496 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:50 smithi125 conmon[24927]: :52:48.487778+0000 osd.0 (osd. 2021-03-24T21:52:50.497 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:50 smithi125 conmon[24927]: 0) 9 : cluster [DBG] 3.1 scrub ok 2021-03-24T21:52:50.497 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:50 smithi125 conmon[24927]: cluster 2021-03-24T21:52:50.497 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:50 smithi125 conmon[24927]: 2021-03-24T21:52 2021-03-24T21:52:50.497 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:50 smithi125 conmon[24927]: :48.660702+0000 mgr.y (mgr.14144) 2021-03-24T21:52:50.498 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:50 smithi125 conmon[24927]: 239 : cluster [DBG] pgmap v246: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-24T21:52:50.498 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:50 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:52:50.498 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:50 smithi125 conmon[24927]: -24T21:52:49. 2021-03-24T21:52:50.498 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:50 smithi125 conmon[24927]: 553945+0000 osd.2 (osd.2) 8 : cluster 2021-03-24T21:52:50.499 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:50 smithi125 conmon[24927]: [DBG] 4.1c scrub ok 2021-03-24T21:52:50.545 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:50 smithi142 conmon[20125]: cluster 2021-03-24T21:52: 2021-03-24T21:52:50.546 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:50 smithi142 conmon[20125]: 48.334127+0000 osd.4 (osd.4) 7 : cluster [DBG] 3.9 scrub ok 2021-03-24T21:52:50.546 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:50 smithi142 conmon[20125]: cluster 2021-03-24T21:52:50.546 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:50 smithi142 conmon[20125]: 2021-03-24T21:52:48.487778+0000 osd.0 (osd. 2021-03-24T21:52:50.547 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:50 smithi142 conmon[20125]: 0) 9 : cluster [DBG] 3.1 scrub ok 2021-03-24T21:52:50.547 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:50 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:52:50.547 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:50 smithi142 conmon[20125]: -03-24T21:52:48.660702+0000 2021-03-24T21:52:50.547 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:50 smithi142 conmon[20125]: mgr.y (mgr.14144) 239 : cluster [DBG] pgmap v246: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-24T21:52:50.547 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:50 smithi142 conmon[20125]: 2021-03-24T21:52:50.548 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:50 smithi142 conmon[20125]: cluster 2021-03-24T21:52:49 2021-03-24T21:52:50.548 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:50 smithi142 conmon[20125]: .553945+0000 osd.2 (osd.2) 8 : cluster 2021-03-24T21:52:50.548 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:50 smithi142 conmon[20125]: [DBG] 4.1c scrub ok 2021-03-24T21:52:51.230 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-24T21:52:51.231 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":85,"fsid":"3cfa339a-8cea-11eb-90b2-001a4aab830c","created":"2021-03-24T21:46:19.163312+0000","modified":"2021-03-24T21:52:51.181580+0000","last_up_change":"2021-03-24T21:51:38.283730+0000","last_in_change":"2021-03-24T21:51:20.214398+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-24T21:49:51.001357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"iscsi","create_time":"2021-03-24T21:51:39.190981+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":".rgw.root","create_time":"2021-03-24T21:51:44.159145+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2021-03-24T21:51:46.090924+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2021-03-24T21:51:47.842521+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2021-03-24T21:51:50.730906+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"1f8f9a69-4b85-4054-8b11-36d35574494c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6801","nonce":1197472172}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6802","nonce":1197472172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6804","nonce":1197472172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6803","nonce":1197472172}]},"public_addr":"172.21.15.125:6801/1197472172","cluster_addr":"172.21.15.125:6802/1197472172","heartbeat_back_addr":"172.21.15.125:6804/1197472172","heartbeat_front_addr":"172.21.15.125:6803/1197472172","state":["exists","up"]},{"osd":1,"uuid":"48db0723-5806-4e5c-9919-f3d7f52ad169","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6805","nonce":113845288}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6806","nonce":113845288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6808","nonce":113845288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6807","nonce":113845288}]},"public_addr":"172.21.15.125:6805/113845288","cluster_addr":"172.21.15.125:6806/113845288","heartbeat_back_addr":"172.21.15.125:6808/113845288","heartbeat_front_addr":"172.21.15.125:6807/113845288","state":["exists","up"]},{"osd":2,"uuid":"d6a361f8-4e4d-4f46-8ab9-01bf71dbb2c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6809","nonce":246939163}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6810","nonce":246939163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6812","nonce":246939163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6811","nonce":246939163}]},"public_addr":"172.21.15.125:6809/246939163","cluster_addr":"172.21.15.125:6810/246939163","heartbeat_back_addr":"172.21.15.125:6812/246939163","heartbeat_front_addr":"172.21.15.125:6811/246939163","state":["exists","up"]},{"osd":3,"uuid":"716a41ce-7a28-400f-bf75-4c1ad99c7f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6813","nonce":2209636173}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6814","nonce":2209636173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6816","nonce":2209636173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.125:6815","nonce":2209636173}]},"public_addr":"172.21.15.125:6813/2209636173","cluster_addr":"172.21.15.125:6814/2209636173","heartbeat_back_addr":"172.21.15.125:6816/2209636173","heartbeat_front_addr":"172.21.15.125:6815/2209636173","state":["exists","up"]},{"osd":4,"uuid":"f9bdd8d2-b292-49da-ad4f-99682122f543","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.142:6800","nonce":1816503330}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.142:6801","nonce":1816503330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.142:6803","nonce":1816503330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.142:6802","nonce":1816503330}]},"public_addr":"172.21.15.142:6800/1816503330","cluster_addr":"172.21.15.142:6801/1816503330","heartbeat_back_addr":"172.21.15.142:6803/1816503330","heartbeat_front_addr":"172.21.15.142:6802/1816503330","state":["exists","up"]},{"osd":5,"uuid":"ec0d43aa-6905-43f0-9e32-e19d4cf5129d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.142:6804","nonce":396400804}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.142:6805","nonce":396400804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.142:6807","nonce":396400804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.142:6806","nonce":396400804}]},"public_addr":"172.21.15.142:6804/396400804","cluster_addr":"172.21.15.142:6805/396400804","heartbeat_back_addr":"172.21.15.142:6807/396400804","heartbeat_front_addr":"172.21.15.142:6806/396400804","state":["exists","up"]},{"osd":6,"uuid":"a26a487e-ed26-4d92-98ea-896739302921","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.142:6808","nonce":4048483913}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.142:6809","nonce":4048483913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.142:6811","nonce":4048483913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.142:6810","nonce":4048483913}]},"public_addr":"172.21.15.142:6808/4048483913","cluster_addr":"172.21.15.142:6809/4048483913","heartbeat_back_addr":"172.21.15.142:6811/4048483913","heartbeat_front_addr":"172.21.15.142:6810/4048483913","state":["exists","up"]},{"osd":7,"uuid":"682ba5ce-0172-48e3-82f8-57a3f98c911b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.142:6812","nonce":3678129570}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.142:6813","nonce":3678129570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.142:6815","nonce":3678129570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.142:6814","nonce":3678129570}]},"public_addr":"172.21.15.142:6812/3678129570","cluster_addr":"172.21.15.142:6813/3678129570","heartbeat_back_addr":"172.21.15.142:6815/3678129570","heartbeat_front_addr":"172.21.15.142:6814/3678129570","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T21:52:07.610589+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T21:52:28.967293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T21:52:50.723077+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T21:52:11.677778+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T21:52:30.579857+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T21:51:52.834140+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T21:52:15.330965+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T21:52:36.618960+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.1c","mappings":[{"from":1,"to":5}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.125:0/3077905055":"2021-03-25T21:47:04.502055+0000","172.21.15.125:0/2210544773":"2021-03-25T21:47:04.502055+0000","172.21.15.125:6800/3176807740":"2021-03-25T21:47:04.502055+0000","172.21.15.125:6800/3531208850":"2021-03-25T21:46:43.197724+0000","172.21.15.125:0/3337261959":"2021-03-25T21:46:43.197724+0000","172.21.15.125:0/1919150488":"2021-03-25T21:46:43.197724+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-03-24T21:52:51.542 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:51 smithi142 conmon[20125]: cluster 2021-03-24T21:52:51.543 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:51 smithi142 conmon[20125]: 2021-03-24T21: 2021-03-24T21:52:51.543 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:51 smithi142 conmon[20125]: 52:50.722882+0000 osd.2 (osd.2) 9 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:52:51.543 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:51 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:52:51.543 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:51 smithi142 conmon[20125]: T21:52:50.723072+0000 osd.2 (osd.2 2021-03-24T21:52:51.544 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:51 smithi142 conmon[20125]: ) 10 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:52:51.544 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:51 smithi142 conmon[20125]: cluster 2021-03-24T21:52:51.544 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:51 smithi142 conmon[20125]: 2021-03-24T21:52 2021-03-24T21:52:51.544 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:51 smithi142 conmon[20125]: :50.788597+0000 osd.5 (osd 2021-03-24T21:52:51.545 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:51 smithi142 conmon[20125]: .5) 8 : cluster [DBG] 3.1d scrub ok 2021-03-24T21:52:51.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:51 smithi125 conmon[24927]: cluster 2021-03-24T21:52:50.722882+0000 osd.2 (osd.2) 9 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:52:51.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:51 smithi125 conmon[24927]: cluster 2021-03-24T21:52:50.723072+0000 osd.2 (osd.2) 10 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:52:51.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:51 smithi125 conmon[24927]: cluster 2021-03-24T21:52:50.788597+0000 osd.5 (osd.5) 8 : cluster [DBG] 3.1d scrub ok 2021-03-24T21:52:51.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:51 smithi125 conmon[19936]: cluster 2021-03-24T21:52:50.722882+0000 osd.2 (osd.2) 9 : cluster [DBG] 2021-03-24T21:52:51.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:51 smithi125 conmon[19936]: purged_snaps scrub starts 2021-03-24T21:52:51.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:51 smithi125 conmon[19936]: cluster 2021-03-24T21:52:50.723072+0000 osd.2 (osd.2) 10 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:52:51.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:51 smithi125 conmon[19936]: cluster 2021-03-24T21:52:50.788597+0000 osd.5 (osd.5) 8 : cluster [DBG] 3.1d scrub ok 2021-03-24T21:52:52.013 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph tell osd.0 flush_pg_stats 2021-03-24T21:52:52.116 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:52 smithi142 systemd[1]: Starting Ceph grafana.a for 3cfa339a-8cea-11eb-90b2-001a4aab830c... 2021-03-24T21:52:52.337 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:52:52.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:52 smithi125 conmon[19936]: cluster 2021-03-24T21:52:50.302066+0000 osd.4 (osd.4) 8 : cluster [DBG] 4.d scrub ok 2021-03-24T21:52:52.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:52 smithi125 conmon[19936]: cluster 2021-03-24T21:52:50.661800+0000 mgr.y (mgr.14144) 240 : cluster [DBG] pgmap v247: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 963 B/s rd, 0 op/s 2021-03-24T21:52:52.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:52 smithi125 conmon[19936]: cluster 2021-03-24T21:52:50.807551+0000 osd.7 (osd.7) 7 : cluster [DBG] 2.0 scrub ok 2021-03-24T21:52:52.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:52 smithi125 conmon[19936]: cluster 2021-03-24T21:52:51.192166+0000 mon.a (mon.0) 561 : cluster [DBG] osdmap e85: 8 total, 8 up, 8 in 2021-03-24T21:52:52.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:52 smithi125 conmon[19936]: audit 2021-03-24T21:52:51.230007+0000 mon.c (mon.2) 15 : audit [DBG] from='client.? v1:172.21.15.125:0/884826942' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-03-24T21:52:52.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:52 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:52:52.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:52 smithi125 conmon[24927]: T21:52:50.302066+0000 osd.4 (osd.4) 8 2021-03-24T21:52:52.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:52 smithi125 conmon[24927]: : cluster [DBG] 4.d scrub ok 2021-03-24T21:52:52.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:52 smithi125 conmon[24927]: cluster 2021-03-24T21:52:50.661800+0000 2021-03-24T21:52:52.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:52 smithi125 conmon[24927]: mgr.y (mgr.14144) 240 : cluster [DBG] pgmap v247: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 963 B/s rd, 0 op/s 2021-03-24T21:52:52.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:52 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:52:52.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:52 smithi125 conmon[24927]: -24T21:52:50.807551+0000 osd.7 (osd.7) 7 : cluster [DBG] 2021-03-24T21:52:52.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:52 smithi125 conmon[24927]: 2.0 scrub ok 2021-03-24T21:52:52.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:52 smithi125 conmon[24927]: cluster 2021-03-24T21:52:51. 2021-03-24T21:52:52.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:52 smithi125 conmon[24927]: 192166+0000 mon.a (mon.0) 561 : cluster 2021-03-24T21:52:52.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:52 smithi125 conmon[24927]: [DBG] osdmap e85: 8 total, 8 up, 8 in 2021-03-24T21:52:52.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:52 smithi125 conmon[24927]: audit 2021-03-24T21: 2021-03-24T21:52:52.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:52 smithi125 conmon[24927]: 52:51.230007+0000 mon.c (mon.2) 2021-03-24T21:52:52.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:52 smithi125 conmon[24927]: 15 : audit [DBG] from='client.? v1:172.21.15.125:0/884826942' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-03-24T21:52:52.503 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:52 smithi142 podman[40110]: 2021-03-24 21:52:52.411984725 +0000 UTC m=+0.203128170 container create 9213e39ce16a4dd9826b888f5b6f9b3e8f3db1527ed810847f8719c157bfba97 (image=docker.io/ceph/ceph-grafana:6.7.4, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-grafana.a, summary=Grafana Container configured for Ceph mgr/dashboard integration, org.label-schema.vendor=CentOS, io.buildah.version=1.18.0, org.label-schema.build-date=20201204, maintainer=Paul Cuzner , org.label-schema.name=CentOS Base Image, description=Ceph Grafana Container, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2021-03-24T21:52:52.504 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:52 smithi142 conmon[20125]: cluster 2021-03-24T21:52 2021-03-24T21:52:52.504 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:52 smithi142 conmon[20125]: :50.302066+0000 osd.4 (osd.4) 8 : cluster [DBG] 4.d scrub ok 2021-03-24T21:52:52.504 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:52 smithi142 conmon[20125]: cluster 2021-03-24T21:52:52.504 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:52 smithi142 conmon[20125]: 2021-03-24T21:52:50.661800+0000 mgr.y 2021-03-24T21:52:52.505 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:52 smithi142 conmon[20125]: (mgr.14144) 240 : cluster [DBG] 2021-03-24T21:52:52.505 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:52 smithi142 conmon[20125]: pgmap v247: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 963 B/s rd, 0 op/s 2021-03-24T21:52:52.505 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:52 smithi142 conmon[20125]: cluster 2021-03-24T21:52:52.505 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:52 smithi142 conmon[20125]: 2021-03-24T21:52:50.807551+0000 2021-03-24T21:52:52.506 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:52 smithi142 conmon[20125]: osd.7 (osd.7) 7 : cluster [DBG] 2021-03-24T21:52:52.506 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:52 smithi142 conmon[20125]: 2.0 scrub ok 2021-03-24T21:52:52.506 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:52 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:52:52.506 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:52 smithi142 conmon[20125]: 03-24T21:52 2021-03-24T21:52:52.506 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:52 smithi142 conmon[20125]: :51.192166+0000 mon.a (mon.0 2021-03-24T21:52:52.507 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:52 smithi142 conmon[20125]: ) 561 : cluster [DBG] osdmap e85: 8 total, 8 up, 8 in 2021-03-24T21:52:52.507 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:52 smithi142 conmon[20125]: 2021-03-24T21:52:52.507 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:52 smithi142 conmon[20125]: audit 2021 2021-03-24T21:52:52.507 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:52 smithi142 conmon[20125]: -03-24T21:52 2021-03-24T21:52:52.507 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:52 smithi142 conmon[20125]: :51.230007+0000 mon.c ( 2021-03-24T21:52:52.508 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:52 smithi142 conmon[20125]: mon.2) 15 : audit [DBG] 2021-03-24T21:52:52.508 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:52 smithi142 conmon[20125]: from='client.? v1:172.21.15.125:0/884826942' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-03-24T21:52:52.842 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:52 smithi142 podman[40110]: 2021-03-24 21:52:52.611989683 +0000 UTC m=+0.403133149 container init 9213e39ce16a4dd9826b888f5b6f9b3e8f3db1527ed810847f8719c157bfba97 (image=docker.io/ceph/ceph-grafana:6.7.4, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-grafana.a, org.label-schema.build-date=20201204, description=Ceph Grafana Container, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.18.0, maintainer=Paul Cuzner , org.label-schema.vendor=CentOS, summary=Grafana Container configured for Ceph mgr/dashboard integration) 2021-03-24T21:52:52.842 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:52 smithi142 conmon[40152]: t=2021-03-24T21:52:52+0000 lvl=info msg="Starting Grafana" logger=server version=6.7.4 commit=8e44bbc5f5 branch=HEAD compiled=2020-05-26T17:35:38+0000 2021-03-24T21:52:52.842 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:52 smithi142 conmon[40152]: t=2021-03-24T21:52:52+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2021-03-24T21:52:52.843 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:52 smithi142 bash[40110]: 9213e39ce16a4dd9826b888f5b6f9b3e8f3db1527ed810847f8719c157bfba97 2021-03-24T21:52:52.843 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:52 smithi142 conmon[40152]: t=2021-03-24T21:52:52+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2021-03-24T21:52:52.843 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:52 smithi142 conmon[40152]: t=2021-03-24T21:52:52+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2021-03-24T21:52:52.843 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:52 smithi142 conmon[40152]: t=2021-03-24T21:52:52+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2021-03-24T21:52:52.844 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:52 smithi142 conmon[40152]: t=2021-03-24T21:52:52+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2021-03-24T21:52:52.844 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:52 smithi142 conmon[40152]: t=2021-03-24T21:52:52+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2021-03-24T21:52:52.844 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:52 smithi142 conmon[40152]: t=2021-03-24T21:52:52+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2021-03-24T21:52:52.844 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:52 smithi142 conmon[40152]: t=2021-03-24T21:52:52+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2021-03-24T21:52:52.844 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:52 smithi142 conmon[40152]: t=2021-03-24T21:52:52+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2021-03-24T21:52:52.845 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:52 smithi142 conmon[40152]: t=2021-03-24T21:52:52+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2021-03-24T21:52:52.845 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:52 smithi142 conmon[40152]: t=2021-03-24T21:52:52+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2021-03-24T21:52:52.845 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:52 smithi142 conmon[40152]: t=2021-03-24T21:52:52+0000 lvl=info msg="App mode production" logger=settings 2021-03-24T21:52:52.845 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:52 smithi142 podman[40110]: 2021-03-24 21:52:52.645250912 +0000 UTC m=+0.436394364 container start 9213e39ce16a4dd9826b888f5b6f9b3e8f3db1527ed810847f8719c157bfba97 (image=docker.io/ceph/ceph-grafana:6.7.4, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-grafana.a, org.label-schema.vendor=CentOS, maintainer=Paul Cuzner , description=Ceph Grafana Container, org.label-schema.build-date=20201204, org.label-schema.schema-version=1.0, summary=Grafana Container configured for Ceph mgr/dashboard integration, org.label-schema.name=CentOS Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.18.0) 2021-03-24T21:52:52.845 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:52 smithi142 conmon[40152]: t=2021-03-24T21:52:52+0000 lvl=info msg="Initializing SqlStore" logger=server 2021-03-24T21:52:52.846 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:52 smithi142 conmon[40152]: t=2021-03-24T21:52:52+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2021-03-24T21:52:52.846 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:52 smithi142 conmon[40152]: t=2021-03-24T21:52:52+0000 lvl=info msg="Starting DB migration" logger=migrator 2021-03-24T21:52:52.846 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:52 smithi142 conmon[40152]: t=2021-03-24T21:52:52+0000 lvl=info msg="Executing migration" logger=migrator id="create migration_log table" 2021-03-24T21:52:52.846 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:52 smithi142 systemd[1]: Started Ceph grafana.a for 3cfa339a-8cea-11eb-90b2-001a4aab830c. 2021-03-24T21:52:52.847 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:52 smithi142 conmon[40152]: t=2021-03-24T21:52:52+0000 lvl=info msg="Executing migration" logger=migrator id="create user table" 2021-03-24T21:52:52.847 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:52 smithi142 conmon[40152]: t=2021-03-24T21:52:52+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.login" 2021-03-24T21:52:53.116 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:52 smithi142 conmon[40152]: t=2021-03-24T21:52:52+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.email" 2021-03-24T21:52:53.117 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:52 smithi142 conmon[40152]: t=2021-03-24T21:52:52+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_login - v1" 2021-03-24T21:52:53.117 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:53 smithi142 conmon[40152]: t=2021-03-24T21:52:53+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_email - v1" 2021-03-24T21:52:53.117 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:53 smithi142 conmon[40152]: t=2021-03-24T21:52:53+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table user to user_v1 - v1" 2021-03-24T21:52:53.437 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:53 smithi142 conmon[40152]: t=2021-03-24T21:52:53+0000 lvl=info msg="Executing migration" logger=migrator id="create user table v2" 2021-03-24T21:52:53.438 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:53 smithi142 conmon[40152]: t=2021-03-24T21:52:53+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_login - v2" 2021-03-24T21:52:53.438 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:53 smithi142 conmon[40152]: t=2021-03-24T21:52:53+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_email - v2" 2021-03-24T21:52:53.439 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:53 smithi142 conmon[20125]: cluster 2021-03-24T21:52:51.450324+0000 osd.0 (osd.0) 10 2021-03-24T21:52:53.439 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:53 smithi142 conmon[20125]: : cluster [DBG] 3.12 scrub ok 2021-03-24T21:52:53.439 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:53 smithi142 conmon[20125]: cluster 2021-03-24T21:52:52.579779+0000 osd.3 (osd.3) 8 : cluster [DBG] 4.10 scrub ok 2021-03-24T21:52:53.439 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:53 smithi142 conmon[20125]: audit 2021-03-24T21:52:52 2021-03-24T21:52:53.440 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:53 smithi142 conmon[20125]: .685724+0000 mon.a (mon.0) 562 : audit 2021-03-24T21:52:53.440 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:53 smithi142 conmon[20125]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:52:53.440 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:53 smithi142 conmon[20125]: audit 2021-03-24T21:52 2021-03-24T21:52:53.440 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:53 smithi142 conmon[20125]: :52.687296+0000 mon.a (mon.0) 563 2021-03-24T21:52:53.440 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:53 smithi142 conmon[20125]: : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:52:53.441 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:53 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:52:53.441 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:53 smithi142 conmon[20125]: 52:52.702299+0000 osd.1 (osd.1 2021-03-24T21:52:53.441 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:53 smithi142 conmon[20125]: ) 9 : cluster [DBG] 5.12 scrub ok 2021-03-24T21:52:53.483 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:53 smithi125 conmon[19936]: cluster 2021-03-24T21:52: 2021-03-24T21:52:53.483 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:53 smithi125 conmon[19936]: 51.450324+0000 osd.0 (osd.0) 10 : cluster [DBG] 3.12 scrub ok 2021-03-24T21:52:53.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:53 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:52:53.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:53 smithi125 conmon[19936]: :52:52.579779+0000 osd.3 (osd.3) 8 2021-03-24T21:52:53.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:53 smithi125 conmon[19936]: : cluster [DBG] 4.10 scrub ok 2021-03-24T21:52:53.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:53 smithi125 conmon[19936]: audit 2021-03-24T21:52:52.685724+0000 2021-03-24T21:52:53.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:53 smithi125 conmon[19936]: mon.a (mon.0) 562 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:52:53.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:53 smithi125 conmon[19936]: audit 2021-03-24T21:52:52.687296+0000 mon.a (mon 2021-03-24T21:52:53.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:53 smithi125 conmon[19936]: .0) 563 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:52:53.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:53 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:52:53.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:53 smithi125 conmon[19936]: 52:52.702299+0000 osd.1 (osd.1) 2021-03-24T21:52:53.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:53 smithi125 conmon[19936]: 9 : cluster [DBG] 5.12 scrub ok 2021-03-24T21:52:53.486 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:53 smithi125 conmon[24927]: cluster 2021-03-24T21:52:51. 2021-03-24T21:52:53.486 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:53 smithi125 conmon[24927]: 450324+0000 osd.0 (osd.0) 10 : cluster [DBG] 3.12 scrub ok 2021-03-24T21:52:53.486 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:53 smithi125 conmon[24927]: cluster 2021-03-24T21:52:52 2021-03-24T21:52:53.487 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:53 smithi125 conmon[24927]: .579779+0000 osd.3 (osd.3) 8 : cluster [DBG] 4.10 scrub ok 2021-03-24T21:52:53.487 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:53 smithi125 conmon[24927]: audit 2021- 2021-03-24T21:52:53.487 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:53 smithi125 conmon[24927]: 03-24T21:52:52.685724+0000 2021-03-24T21:52:53.487 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:53 smithi125 conmon[24927]: mon.a (mon.0) 562 : audit [INF] 2021-03-24T21:52:53.488 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:53 smithi125 conmon[24927]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:52:53.488 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:53 smithi125 conmon[24927]: audit 2021-03-24T21:52: 2021-03-24T21:52:53.488 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:53 smithi125 conmon[24927]: 52.687296+0000 mon.a (mon.0) 563 : 2021-03-24T21:52:53.488 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:53 smithi125 conmon[24927]: audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:52:53.489 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:53 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:52:53.489 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:53 smithi125 conmon[24927]: 52:52.702299+0000 osd.1 (osd.1) 2021-03-24T21:52:53.490 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:53 smithi125 conmon[24927]: 9 : cluster [DBG] 5.12 scrub ok 2021-03-24T21:52:53.741 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:53 smithi142 conmon[40152]: t=2021-03-24T21:52:53+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2021-03-24T21:52:53.741 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:53 smithi142 conmon[40152]: t=2021-03-24T21:52:53+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table user_v1" 2021-03-24T21:52:53.741 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:53 smithi142 conmon[40152]: t=2021-03-24T21:52:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add column help_flags1 to user table" 2021-03-24T21:52:54.074 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:53 smithi142 conmon[40152]: t=2021-03-24T21:52:53+0000 lvl=info msg="Executing migration" logger=migrator id="Update user table charset" 2021-03-24T21:52:54.075 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:53 smithi142 conmon[40152]: t=2021-03-24T21:52:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add last_seen_at column to user" 2021-03-24T21:52:54.075 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:53 smithi142 conmon[40152]: t=2021-03-24T21:52:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add missing user data" 2021-03-24T21:52:54.075 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:53 smithi142 conmon[40152]: t=2021-03-24T21:52:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_disabled column to user" 2021-03-24T21:52:54.075 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:54 smithi142 conmon[40152]: t=2021-03-24T21:52:54+0000 lvl=info msg="Executing migration" logger=migrator id="create temp user table v1-7" 2021-03-24T21:52:54.306 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:54 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:52:54.307 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:54 smithi125 conmon[19936]: 21:52:52.662559+0000 mgr.y (mgr.14144) 241 : cluster [DBG] pgmap v249: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-24T21:52:54.307 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:54 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:52:54.307 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:54 smithi125 conmon[19936]: -03-24T21:52:52.829372+0000 2021-03-24T21:52:54.308 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:54 smithi125 conmon[19936]: osd.7 (osd.7) 8 : cluster [DBG] 5.10 scrub ok 2021-03-24T21:52:54.308 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:54 smithi125 conmon[19936]: 2021-03-24T21:52:54.308 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:54 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:52:54.308 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:54 smithi125 conmon[19936]: -03-24T21:52:52.990123+0000 2021-03-24T21:52:54.309 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:54 smithi125 conmon[19936]: osd.5 (osd.5) 9 : 2021-03-24T21:52:54.309 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:54 smithi125 conmon[19936]: cluster [DBG] purged_snaps scrub starts 2021-03-24T21:52:54.309 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:54 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:52:54.309 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:54 smithi125 conmon[19936]: -03-24T21:52: 2021-03-24T21:52:54.309 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:54 smithi125 conmon[19936]: 52.990297+0000 osd.5 (osd. 2021-03-24T21:52:54.310 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:54 smithi125 conmon[19936]: 5) 10 : cluster [DBG] 2021-03-24T21:52:54.310 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:54 smithi125 conmon[19936]: purged_snaps scrub ok 2021-03-24T21:52:54.310 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:54 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:52:54.310 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:54 smithi125 conmon[19936]: -03-24T21:52: 2021-03-24T21:52:54.310 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:54 smithi125 conmon[19936]: 53.689362+0000 mon.a (mon 2021-03-24T21:52:54.311 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:54 smithi125 conmon[19936]: .0) 564 : cluster [DBG] 2021-03-24T21:52:54.311 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:54 smithi125 conmon[19936]: osdmap e86: 8 total, 8 up, 8 in 2021-03-24T21:52:54.311 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:54 smithi125 conmon[24927]: cluster 2021-03-24T21:52: 2021-03-24T21:52:54.311 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:54 smithi125 conmon[24927]: 52.662559+0000 mgr.y (mgr.14144) 241 : cluster [DBG] pgmap v249: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-24T21:52:54.312 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:54 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:52:54.312 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:54 smithi125 conmon[24927]: -24T21:52:52.829372+0000 2021-03-24T21:52:54.312 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:54 smithi125 conmon[24927]: osd.7 (osd.7) 8 : cluster [DBG] 2021-03-24T21:52:54.312 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:54 smithi125 conmon[24927]: 5.10 scrub ok 2021-03-24T21:52:54.312 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:54 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:52:54.313 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:54 smithi125 conmon[24927]: -03-24T21:52:52 2021-03-24T21:52:54.313 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:54 smithi125 conmon[24927]: .990123+0000 osd.5 (osd.5) 9 2021-03-24T21:52:54.313 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:54 smithi125 conmon[24927]: : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:52:54.313 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:54 smithi125 conmon[24927]: cluster 2021-03-24T21:52:54.314 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:54 smithi125 conmon[24927]: 2021-03-24T21:52:52 2021-03-24T21:52:54.314 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:54 smithi125 conmon[24927]: .990297+0000 osd.5 (osd.5) 10 2021-03-24T21:52:54.314 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:54 smithi125 conmon[24927]: : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:52:54.314 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:54 smithi125 conmon[24927]: cluster 2021-03-24T21:52:54.315 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:54 smithi125 conmon[24927]: 2021-03-24T21:52:53. 2021-03-24T21:52:54.315 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:54 smithi125 conmon[24927]: 689362+0000 mon.a (mon.0) 564 : cluster 2021-03-24T21:52:54.315 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:54 smithi125 conmon[24927]: [DBG] osdmap e86: 8 total, 8 up, 8 in 2021-03-24T21:52:54.366 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:54 smithi142 conmon[40152]: t=2021-03-24T21:52:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v1-7" 2021-03-24T21:52:54.367 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:54 smithi142 conmon[40152]: t=2021-03-24T21:52:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v1-7" 2021-03-24T21:52:54.367 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:54 smithi142 conmon[40152]: t=2021-03-24T21:52:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v1-7" 2021-03-24T21:52:54.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:54 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:52:54.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:54 smithi142 conmon[20125]: 52:52.662559+0000 mgr.y (mgr.14144) 241 : cluster [DBG] pgmap v249: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-24T21:52:54.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:54 smithi142 conmon[20125]: cluster 2021-03-24T21:52 2021-03-24T21:52:54.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:54 smithi142 conmon[20125]: :52.829372+0000 osd.7 (osd.7) 8 : cluster [DBG] 5.10 scrub ok 2021-03-24T21:52:54.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:54 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:52:54.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:54 smithi142 conmon[20125]: T21:52:52.990123+0000 osd.5 (osd 2021-03-24T21:52:54.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:54 smithi142 conmon[20125]: .5) 9 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:52:54.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:54 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:52:54.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:54 smithi142 conmon[20125]: -03-24T21:52:52.990297 2021-03-24T21:52:54.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:54 smithi142 conmon[20125]: +0000 osd.5 (osd.5) 10 : cluster 2021-03-24T21:52:54.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:54 smithi142 conmon[20125]: [DBG] purged_snaps scrub ok 2021-03-24T21:52:54.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:54 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:52:54.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:54 smithi142 conmon[20125]: 52:53.689362+0000 mon.a (mon. 2021-03-24T21:52:54.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:54 smithi142 conmon[20125]: 0) 564 : cluster [DBG] osdmap e86: 8 total, 8 up, 8 in 2021-03-24T21:52:54.728 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:54 smithi142 conmon[40152]: t=2021-03-24T21:52:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v1-7" 2021-03-24T21:52:54.728 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:54 smithi142 conmon[40152]: t=2021-03-24T21:52:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update temp_user table charset" 2021-03-24T21:52:54.729 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:54 smithi142 conmon[40152]: t=2021-03-24T21:52:54+0000 lvl=info msg="Executing migration" logger=migrator id="create star table" 2021-03-24T21:52:54.729 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:54 smithi142 conmon[40152]: t=2021-03-24T21:52:54+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index star.user_id_dashboard_id" 2021-03-24T21:52:55.011 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:54 smithi142 conmon[40152]: t=2021-03-24T21:52:54+0000 lvl=info msg="Executing migration" logger=migrator id="create org table v1" 2021-03-24T21:52:55.012 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:54 smithi142 conmon[40152]: t=2021-03-24T21:52:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_name - v1" 2021-03-24T21:52:55.012 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:54 smithi142 conmon[40152]: t=2021-03-24T21:52:54+0000 lvl=info msg="Executing migration" logger=migrator id="create org_user table v1" 2021-03-24T21:52:55.177 INFO:teuthology.orchestra.run.smithi125.stdout:34359738415 2021-03-24T21:52:55.177 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph tell osd.1 flush_pg_stats 2021-03-24T21:52:55.303 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:55 smithi142 conmon[40152]: t=2021-03-24T21:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_org_id - v1" 2021-03-24T21:52:55.303 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:55 smithi142 conmon[40152]: t=2021-03-24T21:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_user_org_id_user_id - v1" 2021-03-24T21:52:55.303 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:55 smithi142 conmon[40152]: t=2021-03-24T21:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Update org table charset" 2021-03-24T21:52:55.303 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:55 smithi142 conmon[40152]: t=2021-03-24T21:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Update org_user table charset" 2021-03-24T21:52:55.304 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:55 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:52:55.304 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:55 smithi142 conmon[20125]: 21:52:54.212435+0000 osd.4 (osd.4) 9 : cluster [DBG] 3.1a deep-scrub ok 2021-03-24T21:52:55.304 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:55 smithi142 conmon[20125]: cluster 2021-03-24T21:52:55.304 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:55 smithi142 conmon[20125]: 2021-03-24T21:52:54. 2021-03-24T21:52:55.305 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:55 smithi142 conmon[20125]: 597487+0000 osd.3 (osd.3 2021-03-24T21:52:55.305 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:55 smithi142 conmon[20125]: ) 9 : cluster [DBG] 4.11 deep-scrub ok 2021-03-24T21:52:55.305 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:55 smithi142 conmon[20125]: 2021-03-24T21:52:55.305 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:55 smithi142 conmon[20125]: cluster 2021-03-24T21:52:55.306 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:55 smithi142 conmon[20125]: 2021-03-24 2021-03-24T21:52:55.306 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:55 smithi142 conmon[20125]: T21:52:54.714039+0000 2021-03-24T21:52:55.306 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:55 smithi142 conmon[20125]: osd.1 (osd.1) 10 2021-03-24T21:52:55.306 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:55 smithi142 conmon[20125]: : cluster [DBG] 4.f scrub ok 2021-03-24T21:52:55.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:55 smithi125 conmon[24927]: cluster 2021-03-24T21:52:54 2021-03-24T21:52:55.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:55 smithi125 conmon[24927]: .212435+0000 osd.4 (osd.4) 9 : cluster [DBG] 3.1a deep-scrub ok 2021-03-24T21:52:55.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:55 smithi125 conmon[24927]: cluster 2021-03-24T21:52:55.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:55 smithi125 conmon[24927]: 2021-03-24T21:52:54.597487+0000 osd.3 (osd.3) 9 : cluster 2021-03-24T21:52:55.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:55 smithi125 conmon[24927]: [DBG] 4.11 deep-scrub ok 2021-03-24T21:52:55.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:55 smithi125 conmon[24927]: cluster 2021-03-24T21:52:55.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:55 smithi125 conmon[24927]: 2021-03-24T21:52 2021-03-24T21:52:55.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:55 smithi125 conmon[24927]: :54.714039+0000 osd.1 (osd.1) 10 2021-03-24T21:52:55.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:55 smithi125 conmon[24927]: : cluster [DBG] 4.f scrub ok 2021-03-24T21:52:55.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:55 smithi125 conmon[19936]: cluster 2021-03-24T21:52 2021-03-24T21:52:55.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:55 smithi125 conmon[19936]: :54.212435+0000 osd.4 (osd.4) 9 : cluster [DBG] 3.1a deep-scrub ok 2021-03-24T21:52:55.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:55 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:52:55.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:55 smithi125 conmon[19936]: 24T21:52:54.597487+0000 osd.3 (osd.3) 9 : cluster [DBG] 4.11 deep-scrub ok 2021-03-24T21:52:55.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:55 smithi125 conmon[19936]: 2021-03-24T21:52:55.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:55 smithi125 conmon[19936]: cluster 2021-03-24T21:52:54. 2021-03-24T21:52:55.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:55 smithi125 conmon[19936]: 714039+0000 osd.1 (osd.1) 10 : cluster 2021-03-24T21:52:55.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:55 smithi125 conmon[19936]: [DBG] 4.f scrub ok 2021-03-24T21:52:55.492 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:52:55.568 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:55 smithi142 conmon[40152]: t=2021-03-24T21:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate all Read Only Viewers to Viewers" 2021-03-24T21:52:55.568 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:55 smithi142 conmon[40152]: t=2021-03-24T21:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard table" 2021-03-24T21:52:55.866 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:55 smithi142 conmon[40152]: t=2021-03-24T21:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard.account_id" 2021-03-24T21:52:55.867 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:55 smithi142 conmon[40152]: t=2021-03-24T21:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_account_id_slug" 2021-03-24T21:52:56.177 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:55 smithi142 conmon[40152]: t=2021-03-24T21:52:55+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_tag table" 2021-03-24T21:52:56.178 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:56 smithi142 conmon[40152]: t=2021-03-24T21:52:56+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_tag.dasboard_id_term" 2021-03-24T21:52:56.178 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:56 smithi142 conmon[40152]: t=2021-03-24T21:52:56+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2021-03-24T21:52:56.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:56 smithi125 conmon[19936]: cluster 2021-03-24T21:52:54.663673 2021-03-24T21:52:56.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:56 smithi125 conmon[19936]: +0000 mgr.y (mgr.14144) 242 : cluster [DBG] pgmap v251: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-24T21:52:56.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:56 smithi125 conmon[19936]: cluster 2021-03-24T21:52:55.233205+0000 osd.4 (osd.4) 2021-03-24T21:52:56.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:56 smithi125 conmon[19936]: 10 : cluster [DBG] 4.e scrub ok 2021-03-24T21:52:56.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:56 smithi125 conmon[24927]: cluster 2021-03-24T21:52:54.663673+0000 mgr.y (mgr 2021-03-24T21:52:56.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:56 smithi125 conmon[24927]: .14144) 242 : cluster [DBG] pgmap v251: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-24T21:52:56.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:56 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:52:56.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:56 smithi125 conmon[24927]: 03-24T21:52:55.233205+0000 osd.4 (osd. 2021-03-24T21:52:56.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:56 smithi125 conmon[24927]: 4) 10 : cluster [DBG] 4.e scrub ok 2021-03-24T21:52:56.453 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:56 smithi142 conmon[40152]: t=2021-03-24T21:52:56+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard to dashboard_v1 - v1" 2021-03-24T21:52:56.453 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:56 smithi142 conmon[40152]: t=2021-03-24T21:52:56+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard v2" 2021-03-24T21:52:56.454 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:56 smithi142 conmon[20125]: cluster 2021-03-24T21:52:54.663673+0000 2021-03-24T21:52:56.454 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:56 smithi142 conmon[20125]: mgr.y (mgr.14144) 242 : cluster [DBG] pgmap v251: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-24T21:52:56.454 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:56 smithi142 conmon[20125]: cluster 2021-03-24T21:52:55.233205+0000 osd.4 (osd.4) 10 : cluster [DBG] 4.e scrub ok 2021-03-24T21:52:56.722 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:56 smithi142 conmon[40152]: t=2021-03-24T21:52:56+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_org_id - v2" 2021-03-24T21:52:56.722 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:56 smithi142 conmon[40152]: t=2021-03-24T21:52:56+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_org_id_slug - v2" 2021-03-24T21:52:57.007 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:56 smithi142 conmon[40152]: t=2021-03-24T21:52:56+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard v1 to v2" 2021-03-24T21:52:57.007 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:56 smithi142 conmon[40152]: t=2021-03-24T21:52:56+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_v1" 2021-03-24T21:52:57.007 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:56 smithi142 conmon[40152]: t=2021-03-24T21:52:56+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard.data to mediumtext v1" 2021-03-24T21:52:57.268 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:57 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:52:57.269 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:57 smithi142 conmon[20125]: 52:55.828801+0000 osd.7 (osd.7) 9 : cluster [DBG] 6.3 scrub ok 2021-03-24T21:52:57.269 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:57 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:52:57.269 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:57 smithi142 conmon[20125]: -24T21:52:55.846753+0000 osd.5 (osd.5) 11 : cluster [DBG] 5.f scrub ok 2021-03-24T21:52:57.270 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:57 smithi142 conmon[20125]: cluster 2021-03-24T21 2021-03-24T21:52:57.270 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:57 smithi142 conmon[20125]: :52:56.678096+0000 osd.3 (osd.3) 10 2021-03-24T21:52:57.270 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:57 smithi142 conmon[20125]: : cluster [DBG] 5.14 deep-scrub ok 2021-03-24T21:52:57.270 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:57 smithi142 conmon[40152]: t=2021-03-24T21:52:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add column updated_by in dashboard - v2" 2021-03-24T21:52:57.271 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:57 smithi142 conmon[40152]: t=2021-03-24T21:52:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add column created_by in dashboard - v2" 2021-03-24T21:52:57.271 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:57 smithi142 conmon[40152]: t=2021-03-24T21:52:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add column gnetId in dashboard" 2021-03-24T21:52:57.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:57 smithi125 conmon[19936]: cluster 2021-03-24T21:52:55. 2021-03-24T21:52:57.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:57 smithi125 conmon[19936]: 828801+0000 osd.7 (osd.7) 9 : cluster [DBG] 6.3 scrub ok 2021-03-24T21:52:57.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:57 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:52:57.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:57 smithi125 conmon[19936]: 52:55.846753+0000 osd.5 (osd.5) 11 : cluster [DBG] 5.f scrub ok 2021-03-24T21:52:57.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:57 smithi125 conmon[19936]: cluster 2021-03-24T21:52:56. 2021-03-24T21:52:57.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:57 smithi125 conmon[19936]: 678096+0000 osd.3 (osd.3) 10 : cluster [DBG] 5.14 deep-scrub ok 2021-03-24T21:52:57.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:57 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:52:57.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:57 smithi125 conmon[24927]: :52:55.828801+0000 osd.7 (osd.7) 9 : cluster [DBG] 2021-03-24T21:52:57.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:57 smithi125 conmon[24927]: 6.3 scrub ok 2021-03-24T21:52:57.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:57 smithi125 conmon[24927]: cluster 2021-03-24T21:52:57.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:57 smithi125 conmon[24927]: 2021-03-24T21:52:57.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:57 smithi125 conmon[24927]: 2021 2021-03-24T21:52:57.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:57 smithi125 conmon[24927]: -03-24T21:52 2021-03-24T21:52:57.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:57 smithi125 conmon[24927]: :55.846753+0000 osd.5 (osd.5) 11 : cluster [DBG] 2021-03-24T21:52:57.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:57 smithi125 conmon[24927]: 5.f scrub ok 2021-03-24T21:52:57.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:57 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:52:57.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:57 smithi125 conmon[24927]: -03-24T21:52:56.678096 2021-03-24T21:52:57.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:57 smithi125 conmon[24927]: +0000 osd.3 (osd.3) 10 : cluster 2021-03-24T21:52:57.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:57 smithi125 conmon[24927]: [DBG] 5.14 deep-scrub ok 2021-03-24T21:52:57.559 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:57 smithi142 conmon[40152]: t=2021-03-24T21:52:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for gnetId in dashboard" 2021-03-24T21:52:57.829 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:57 smithi142 conmon[40152]: t=2021-03-24T21:52:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_id in dashboard" 2021-03-24T21:52:57.830 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:57 smithi142 conmon[40152]: t=2021-03-24T21:52:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for plugin_id in dashboard" 2021-03-24T21:52:57.830 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:57 smithi142 conmon[40152]: t=2021-03-24T21:52:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_id in dashboard_tag" 2021-03-24T21:52:58.116 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:57 smithi142 conmon[40152]: t=2021-03-24T21:52:57+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard table charset" 2021-03-24T21:52:58.117 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:57 smithi142 conmon[40152]: t=2021-03-24T21:52:57+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_tag table charset" 2021-03-24T21:52:58.117 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:57 smithi142 conmon[40152]: t=2021-03-24T21:52:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add column folder_id in dashboard" 2021-03-24T21:52:58.117 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:58 smithi142 conmon[40152]: t=2021-03-24T21:52:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add column isFolder in dashboard" 2021-03-24T21:52:58.385 INFO:teuthology.orchestra.run.smithi125.stdout:55834574892 2021-03-24T21:52:58.386 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph tell osd.2 flush_pg_stats 2021-03-24T21:52:58.473 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:58 smithi142 conmon[40152]: t=2021-03-24T21:52:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add column has_acl in dashboard" 2021-03-24T21:52:58.473 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:58 smithi142 conmon[40152]: t=2021-03-24T21:52:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in dashboard" 2021-03-24T21:52:58.474 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:58 smithi142 conmon[40152]: t=2021-03-24T21:52:58+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in dashboard" 2021-03-24T21:52:58.474 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:58 smithi142 conmon[40152]: t=2021-03-24T21:52:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index dashboard_org_id_uid" 2021-03-24T21:52:58.474 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:58 smithi142 conmon[40152]: t=2021-03-24T21:52:58+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_slug" 2021-03-24T21:52:58.475 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:58 smithi142 conmon[20125]: cluster 2021-03-24T21:52:56.664360+0000 mgr.y (mgr.14144) 243 : cluster 2021-03-24T21:52:58.475 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:58 smithi142 conmon[20125]: [DBG] pgmap v252: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-24T21:52:58.475 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:58 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:52:58.475 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:58 smithi142 conmon[20125]: T21:52:56.855298+0000 osd.5 (osd.5) 12 : cluster 2021-03-24T21:52:58.476 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:58 smithi142 conmon[20125]: [DBG] 5.1b scrub ok 2021-03-24T21:52:58.476 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:58 smithi142 conmon[20125]: audit 2021-03-24T21:52:58.476 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:58 smithi142 conmon[20125]: 2021-03-24T21:52:57. 2021-03-24T21:52:58.476 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:58 smithi142 conmon[20125]: 572416+0000 mon.a (mon.0) 565 : audit 2021-03-24T21:52:58.476 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:58 smithi142 conmon[20125]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:52:58.477 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:58 smithi142 conmon[20125]: cluster 2021-03-24T21 2021-03-24T21:52:58.477 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:58 smithi142 conmon[20125]: :52:57.660796+0000 osd.3 (osd. 2021-03-24T21:52:58.477 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:58 smithi142 conmon[20125]: 3) 11 : cluster [DBG] 5.17 scrub ok 2021-03-24T21:52:58.540 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:58 smithi125 conmon[19936]: cluster 2021-03-24T21:52 2021-03-24T21:52:58.541 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:58 smithi125 conmon[19936]: :56.664360+0000 mgr.y (mgr.14144) 243 : cluster [DBG] pgmap v252: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-24T21:52:58.541 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:58 smithi125 conmon[19936]: cluster 2021-03-24T21:52:56. 2021-03-24T21:52:58.541 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:58 smithi125 conmon[19936]: 855298+0000 osd.5 (osd.5) 12 : cluster [DBG] 5.1b scrub ok 2021-03-24T21:52:58.541 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:58 smithi125 conmon[19936]: audit 2021-03- 2021-03-24T21:52:58.542 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:58 smithi125 conmon[19936]: 24T21:52:57.572416+0000 mon.a (mon.0) 565 : 2021-03-24T21:52:58.542 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:58 smithi125 conmon[19936]: audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:52:58.542 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:58 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:52:58.542 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:58 smithi125 conmon[19936]: 03-24T21:52:57.660796+0000 osd.3 (osd.3) 11 : cluster 2021-03-24T21:52:58.543 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:58 smithi125 conmon[19936]: [DBG] 5.17 scrub ok 2021-03-24T21:52:58.543 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:58 smithi125 conmon[24927]: cluster 2021-03-24T21:52 2021-03-24T21:52:58.543 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:58 smithi125 conmon[24927]: :56.664360+0000 mgr.y (mgr.14144) 243 : cluster [DBG] pgmap v252: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-24T21:52:58.543 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:58 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:52:58.544 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:58 smithi125 conmon[24927]: -03-24T21:52:56.855298+0000 osd.5 2021-03-24T21:52:58.544 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:58 smithi125 conmon[24927]: (osd.5) 12 : cluster [DBG] 5.1b scrub ok 2021-03-24T21:52:58.544 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:58 smithi125 conmon[24927]: audit 2021- 2021-03-24T21:52:58.544 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:58 smithi125 conmon[24927]: 03-24T21:52:57.572416+0000 mon.a (mon. 2021-03-24T21:52:58.545 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:58 smithi125 conmon[24927]: 0) 565 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:52:58.545 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:58 smithi125 conmon[24927]: cluster 2021-03-24T21:52:58.545 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:58 smithi125 conmon[24927]: 2021-03-24T21:52 2021-03-24T21:52:58.545 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:58 smithi125 conmon[24927]: :57.660796+0000 osd.3 (osd.3) 11 2021-03-24T21:52:58.545 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:58 smithi125 conmon[24927]: : cluster [DBG] 5.17 scrub ok 2021-03-24T21:52:58.735 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:58 smithi142 conmon[40152]: t=2021-03-24T21:52:58+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard title length" 2021-03-24T21:52:58.736 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:58 smithi142 conmon[40152]: t=2021-03-24T21:52:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index for dashboard_org_id_title_folder_id" 2021-03-24T21:52:58.736 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:58 smithi142 conmon[40152]: t=2021-03-24T21:52:58+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning" 2021-03-24T21:52:58.736 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:58 smithi142 conmon[40152]: t=2021-03-24T21:52:58+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2021-03-24T21:52:58.796 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:52:59.040 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:58 smithi142 conmon[40152]: t=2021-03-24T21:52:58+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning v2" 2021-03-24T21:52:59.041 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:58 smithi142 conmon[40152]: t=2021-03-24T21:52:58+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2021-03-24T21:52:59.041 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:58 smithi142 conmon[40152]: t=2021-03-24T21:52:58+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2021-03-24T21:52:59.041 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:58 smithi142 conmon[40152]: t=2021-03-24T21:52:58+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard_provisioning v1 to v2" 2021-03-24T21:52:59.367 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:59 smithi142 conmon[40152]: t=2021-03-24T21:52:59+0000 lvl=info msg="Executing migration" logger=migrator id="drop dashboard_provisioning_tmp_qwerty" 2021-03-24T21:52:59.367 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:59 smithi142 conmon[40152]: t=2021-03-24T21:52:59+0000 lvl=info msg="Executing migration" logger=migrator id="Add check_sum column" 2021-03-24T21:52:59.368 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:59 smithi142 conmon[40152]: t=2021-03-24T21:52:59+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table" 2021-03-24T21:52:59.368 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:59 smithi142 conmon[40152]: t=2021-03-24T21:52:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index data_source.account_id" 2021-03-24T21:52:59.688 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:59 smithi142 conmon[40152]: t=2021-03-24T21:52:59+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index data_source.account_id_name" 2021-03-24T21:52:59.689 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:59 smithi142 conmon[40152]: t=2021-03-24T21:52:59+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_data_source_account_id - v1" 2021-03-24T21:52:59.689 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:59 smithi142 conmon[40152]: t=2021-03-24T21:52:59+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_data_source_account_id_name - v1" 2021-03-24T21:52:59.689 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:59 smithi142 conmon[40152]: t=2021-03-24T21:52:59+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table data_source to data_source_v1 - v1" 2021-03-24T21:52:59.910 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[19936]: cluster 2021-03-24T21:52:57.861238 2021-03-24T21:52:59.910 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[19936]: +0000 osd.5 (osd.5) 13 : cluster [DBG] 4.8 scrub ok 2021-03-24T21:52:59.910 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[19936]: cluster 2021-03-24T21:52:58.665342+0000 mgr.y 2021-03-24T21:52:59.911 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[19936]: (mgr.14144) 244 : cluster [DBG] pgmap v253: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-03-24T21:52:59.911 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:52:59.911 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[19936]: 24T21:52:58.676054+0000 osd.3 (osd.3) 12 : cluster [DBG] 6.1b scrub ok 2021-03-24T21:52:59.911 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[19936]: audit 2021-03-24 2021-03-24T21:52:59.911 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[19936]: T21:52:58.765437+0000 mon.a (mon.0) 566 : 2021-03-24T21:52:59.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[19936]: audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:52:59.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[19936]: audit 2021-03-24T21:52:58.773375+0000 mon.a 2021-03-24T21:52:59.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[19936]: (mon.0) 567 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:52:59.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[19936]: cephadm 2021-03-24T 2021-03-24T21:52:59.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[19936]: 21:52:58.777467+0000 mgr.y (mgr.14144) 2021-03-24T21:52:59.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[19936]: 245 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2021-03-24T21:52:59.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[19936]: cephadm 2021-03-24T21:52:59.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[19936]: 2021-03-24T21:52:58.782544 2021-03-24T21:52:59.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[19936]: +0000 mgr.y (mgr.14144) 246 : 2021-03-24T21:52:59.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[19936]: cephadm [INF] Reconfiguring daemon alertmanager.a on smithi125 2021-03-24T21:52:59.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:52:59.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[19936]: -03-24T21:52:59.074406 2021-03-24T21:52:59.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[19936]: +0000 osd.6 (osd.6) 6 2021-03-24T21:52:59.915 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[19936]: : cluster [DBG] 5.1f scrub ok 2021-03-24T21:52:59.915 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:52:59.915 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[19936]: 03-24T21:52:59.180230+0000 2021-03-24T21:52:59.915 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[19936]: osd.4 (osd.4) 11 : 2021-03-24T21:52:59.915 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[19936]: cluster [DBG] 6.1a scrub ok 2021-03-24T21:52:59.916 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[19936]: cluster 2021-03-24T21:52:59.916 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[19936]: 2021-03-24T21:52:59 2021-03-24T21:52:59.916 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[19936]: .579518+0000 osd.1 (osd.1) 11 2021-03-24T21:52:59.916 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[19936]: : cluster [DBG] 5.1d scrub ok 2021-03-24T21:52:59.917 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:52:59.917 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[24927]: :52:57.861238+0000 osd.5 (osd.5) 13 : cluster [DBG] 4.8 scrub ok 2021-03-24T21:52:59.917 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[24927]: 2021-03-24T21:52:59.917 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:52:59.918 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[24927]: 21:52:58.665342+0000 mgr.y 2021-03-24T21:52:59.918 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[24927]: (mgr.14144) 244 : cluster 2021-03-24T21:52:59.918 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[24927]: [DBG] pgmap v253: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-03-24T21:52:59.918 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[24927]: cluster 2021-03-24T21:52:59.918 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[24927]: 2021-03-24T21:52 2021-03-24T21:52:59.919 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[24927]: :58.676054+0000 osd.3 (osd 2021-03-24T21:52:59.919 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[24927]: .3) 12 : cluster [DBG] 2021-03-24T21:52:59.919 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[24927]: 6.1b scrub ok 2021-03-24T21:52:59.919 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[24927]: audit 2021 2021-03-24T21:52:59.919 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[24927]: -03-24T21:52: 2021-03-24T21:52:59.920 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[24927]: 58.765437+0000 mon.a (mon 2021-03-24T21:52:59.920 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[24927]: .0) 566 : audit [INF] 2021-03-24T21:52:59.920 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[24927]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:52:59.920 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[24927]: audit 2021-03-24T21:52:59.921 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[24927]: 2021-03-24T21: 2021-03-24T21:52:59.921 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[24927]: 52:58.773375+0000 mon.a (mon.0 2021-03-24T21:52:59.921 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[24927]: ) 567 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:52:59.921 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[24927]: cephadm 2021 2021-03-24T21:52:59.921 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[24927]: -03-24T21:52:58.777467+0000 2021-03-24T21:52:59.922 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[24927]: mgr.y (mgr.14144) 245 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2021-03-24T21:52:59.922 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[24927]: 2021-03-24T21:52:59.922 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[24927]: cephadm 2021-03-24T21:52:59.922 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[24927]: 2021-03-24T21:52:58.782544+0000 2021-03-24T21:52:59.922 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[24927]: mgr.y (mgr.14144) 246 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi125 2021-03-24T21:52:59.923 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[24927]: 2021-03-24T21:52:59.923 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[24927]: cluster 2021-03-24T21:52:59.923 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[24927]: 2021-03-24T21:52:59.923 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[24927]: 2021- 2021-03-24T21:52:59.924 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[24927]: 03-24T21:52:59.074406+0000 2021-03-24T21:52:59.924 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[24927]: osd.6 (osd.6) 6 : 2021-03-24T21:52:59.924 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[24927]: cluster [DBG] 5.1f scrub ok 2021-03-24T21:52:59.924 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:52:59.924 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[24927]: -24T21:52:59.180230+0000 osd.4 (osd 2021-03-24T21:52:59.925 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[24927]: .4) 11 : cluster [DBG] 6.1a scrub ok 2021-03-24T21:52:59.925 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:52:59.925 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[24927]: -24T21:52:59.579518+0000 osd.1 (osd.1 2021-03-24T21:52:59.925 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:52:59 smithi125 conmon[24927]: ) 11 : cluster [DBG] 5.1d scrub ok 2021-03-24T21:53:00.023 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:59 smithi142 conmon[40152]: t=2021-03-24T21:52:59+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table v2" 2021-03-24T21:53:00.023 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:59 smithi142 conmon[40152]: t=2021-03-24T21:52:59+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_data_source_org_id - v2" 2021-03-24T21:53:00.023 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:59 smithi142 conmon[40152]: t=2021-03-24T21:52:59+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2021-03-24T21:53:00.024 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:52:59 smithi142 conmon[40152]: t=2021-03-24T21:52:59+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2021-03-24T21:53:00.024 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:59 smithi142 conmon[20125]: cluster 2021-03-24T21:52 2021-03-24T21:53:00.024 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:59 smithi142 conmon[20125]: :57.861238+0000 osd.5 (osd.5) 13 : cluster [DBG] 4.8 scrub ok 2021-03-24T21:53:00.025 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:59 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:53:00.025 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:59 smithi142 conmon[20125]: 24T21:52:58.665342+0000 mgr.y (mgr.14144) 244 : cluster [DBG] pgmap v253: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-03-24T21:53:00.025 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:59 smithi142 conmon[20125]: cluster 2021-03-24T21 2021-03-24T21:53:00.025 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:59 smithi142 conmon[20125]: :52:58.676054+0000 osd.3 (osd.3) 12 : cluster 2021-03-24T21:53:00.026 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:59 smithi142 conmon[20125]: [DBG] 6.1b scrub ok 2021-03-24T21:53:00.026 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:59 smithi142 conmon[20125]: audit 2021-03-24T21:53:00.026 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:59 smithi142 conmon[20125]: 2021-03-24T21:52:58.765437+0000 mon.a (mon.0) 566 : 2021-03-24T21:53:00.026 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:59 smithi142 conmon[20125]: audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:53:00.026 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:59 smithi142 conmon[20125]: audit 2021-03-24T21:52:58.773375+0000 mon.a 2021-03-24T21:53:00.027 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:59 smithi142 conmon[20125]: (mon.0) 567 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:53:00.027 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:59 smithi142 conmon[20125]: cephadm 2021-03- 2021-03-24T21:53:00.027 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:59 smithi142 conmon[20125]: 24T21:52:58.777467+0000 mgr.y (mgr.14144) 2021-03-24T21:53:00.027 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:59 smithi142 conmon[20125]: 245 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2021-03-24T21:53:00.028 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:59 smithi142 conmon[20125]: cephadm 2021-03-24T21:52:58.782544+0000 2021-03-24T21:53:00.028 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:59 smithi142 conmon[20125]: mgr.y (mgr.14144) 246 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi125 2021-03-24T21:53:00.028 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:59 smithi142 conmon[20125]: cluster 2021-03-24T21:52:59. 2021-03-24T21:53:00.028 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:59 smithi142 conmon[20125]: 074406+0000 osd.6 (osd.6) 6 : cluster [DBG] 2021-03-24T21:53:00.028 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:59 smithi142 conmon[20125]: 5.1f scrub ok 2021-03-24T21:53:00.029 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:59 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:53:00.029 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:59 smithi142 conmon[20125]: 03-24T21:52:59.180230+0000 osd.4 (osd.4) 11 : cluster 2021-03-24T21:53:00.029 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:59 smithi142 conmon[20125]: [DBG] 6.1a scrub ok 2021-03-24T21:53:00.029 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:59 smithi142 conmon[20125]: cluster 2021-03-24T21:52:59.579518+0000 osd.1 2021-03-24T21:53:00.029 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:59 smithi142 conmon[20125]: (osd.1) 11 : cluster [DBG] 2021-03-24T21:53:00.030 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:52:59 smithi142 conmon[20125]: 5.1d scrub ok 2021-03-24T21:53:00.301 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:00 smithi142 conmon[40152]: t=2021-03-24T21:53:00+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2021-03-24T21:53:00.301 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:00 smithi142 conmon[40152]: t=2021-03-24T21:53:00+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2021-03-24T21:53:00.302 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:00 smithi142 conmon[40152]: t=2021-03-24T21:53:00+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2021-03-24T21:53:00.601 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:00 smithi142 conmon[40152]: t=2021-03-24T21:53:00+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2021-03-24T21:53:00.602 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:00 smithi142 conmon[40152]: t=2021-03-24T21:53:00+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2021-03-24T21:53:00.602 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:00 smithi142 conmon[40152]: t=2021-03-24T21:53:00+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2021-03-24T21:53:00.602 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:00 smithi142 conmon[40152]: t=2021-03-24T21:53:00+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2021-03-24T21:53:00.867 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:00 smithi142 conmon[40152]: t=2021-03-24T21:53:00+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2021-03-24T21:53:00.867 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:00 smithi142 conmon[40152]: t=2021-03-24T21:53:00+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2021-03-24T21:53:00.867 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:00 smithi142 conmon[40152]: t=2021-03-24T21:53:00+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2021-03-24T21:53:00.868 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:00 smithi142 conmon[40152]: t=2021-03-24T21:53:00+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2021-03-24T21:53:00.868 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:00 smithi142 conmon[40152]: t=2021-03-24T21:53:00+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2021-03-24T21:53:00.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:00 smithi142 conmon[20125]: cluster 2021-03-24T21:52:59 2021-03-24T21:53:00.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:00 smithi142 conmon[20125]: .520849+0000 osd.2 (osd.2) 11 : cluster [DBG] 5.b scrub ok 2021-03-24T21:53:00.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:00 smithi142 conmon[20125]: cluster 2021-03-24T21:53:00.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:00 smithi142 conmon[20125]: 2021-03-24T21:52:59.706496+0000 osd.3 (osd.3) 13 : 2021-03-24T21:53:00.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:00 smithi142 conmon[20125]: cluster [DBG] 6.6 deep-scrub ok 2021-03-24T21:53:00.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:00 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:53:00.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:00 smithi142 conmon[20125]: 21:53:00.300656+0000 osd.0 (osd.0 2021-03-24T21:53:00.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:00 smithi142 conmon[20125]: ) 11 : cluster [DBG] 4.16 scrub ok 2021-03-24T21:53:00.909 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:00 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:53:00.909 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:00 smithi125 conmon[24927]: 52:59.520849+0000 osd.2 (osd.2) 11 : cluster [DBG] 5.b scrub ok 2021-03-24T21:53:00.909 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:00 smithi125 conmon[24927]: cluster 2021-03-24T21:53:00.909 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:00 smithi125 conmon[24927]: 2021-03-24T21:52:59.706496+0000 osd.3 (osd.3) 13 : cluster [DBG] 2021-03-24T21:53:00.910 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:00 smithi125 conmon[24927]: 6.6 deep-scrub ok 2021-03-24T21:53:00.910 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:00 smithi125 conmon[24927]: cluster 2021-03-24T21:53:00.910 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:00 smithi125 conmon[24927]: 2021-03-24T21:53:00.300656+0000 osd.0 (osd.0) 11 2021-03-24T21:53:00.910 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:00 smithi125 conmon[24927]: : cluster [DBG] 4.16 scrub ok 2021-03-24T21:53:00.911 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:00 smithi125 conmon[19936]: cluster 2021-03-24T21:52 2021-03-24T21:53:00.911 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:00 smithi125 conmon[19936]: :59.520849+0000 osd.2 (osd.2) 11 : cluster [DBG] 5.b scrub ok 2021-03-24T21:53:00.911 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:00 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:53:00.911 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:00 smithi125 conmon[19936]: -03-24T21:52:59.706496+0000 osd.3 2021-03-24T21:53:00.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:00 smithi125 conmon[19936]: (osd.3) 13 : cluster 2021-03-24T21:53:00.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:00 smithi125 conmon[19936]: [DBG] 6.6 deep-scrub ok 2021-03-24T21:53:00.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:00 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:53:00.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:00 smithi125 conmon[19936]: 03-24T21:53:00.300656+0000 osd.0 (osd. 2021-03-24T21:53:00.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:00 smithi125 conmon[19936]: 0) 11 : cluster [DBG] 4.16 scrub ok 2021-03-24T21:53:00.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:00 smithi125 conmon[19936]: 2021-03-24T21:53:01.253 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:00 smithi142 conmon[40152]: t=2021-03-24T21:53:00+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2021-03-24T21:53:01.253 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:00 smithi142 conmon[40152]: t=2021-03-24T21:53:00+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2021-03-24T21:53:01.253 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:01 smithi142 conmon[40152]: t=2021-03-24T21:53:01+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2021-03-24T21:53:01.254 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:01 smithi142 conmon[40152]: t=2021-03-24T21:53:01+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2021-03-24T21:53:01.254 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:01 smithi142 conmon[40152]: t=2021-03-24T21:53:01+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2021-03-24T21:53:01.254 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:01 smithi142 conmon[40152]: t=2021-03-24T21:53:01+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2021-03-24T21:53:01.514 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:01 smithi142 conmon[40152]: t=2021-03-24T21:53:01+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2021-03-24T21:53:01.515 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:01 smithi142 conmon[40152]: t=2021-03-24T21:53:01+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2021-03-24T21:53:01.515 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:01 smithi142 conmon[40152]: t=2021-03-24T21:53:01+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2021-03-24T21:53:01.799 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:01 smithi142 conmon[40152]: t=2021-03-24T21:53:01+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2021-03-24T21:53:01.800 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:01 smithi142 conmon[40152]: t=2021-03-24T21:53:01+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2021-03-24T21:53:01.800 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:01 smithi142 conmon[40152]: t=2021-03-24T21:53:01+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2021-03-24T21:53:01.800 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:01 smithi142 conmon[40152]: t=2021-03-24T21:53:01+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2021-03-24T21:53:02.052 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:01 smithi142 conmon[40152]: t=2021-03-24T21:53:01+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2021-03-24T21:53:02.052 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:01 smithi142 conmon[40152]: t=2021-03-24T21:53:01+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2021-03-24T21:53:02.052 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:01 smithi142 conmon[40152]: t=2021-03-24T21:53:01+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2021-03-24T21:53:02.053 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:02 smithi142 conmon[40152]: t=2021-03-24T21:53:02+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2021-03-24T21:53:02.159 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:02 smithi125 conmon[19936]: cluster 2021-03-24T21:53: 2021-03-24T21:53:02.159 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:02 smithi125 conmon[19936]: 00.666431+0000 mgr.y (mgr.14144) 247 : cluster [DBG] pgmap v254: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-03-24T21:53:02.159 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:02 smithi125 conmon[19936]: cluster 2021-03-24T21:53:00.682704 2021-03-24T21:53:02.159 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:02 smithi125 conmon[19936]: +0000 osd.3 (osd.3) 14 : cluster [DBG] 2021-03-24T21:53:02.160 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:02 smithi125 conmon[19936]: 3.7 scrub ok 2021-03-24T21:53:02.160 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:02 smithi125 conmon[19936]: cluster 2021-03-24T21:53:00 2021-03-24T21:53:02.160 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:02 smithi125 conmon[19936]: .845329+0000 osd.7 (osd.7) 10 : cluster 2021-03-24T21:53:02.160 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:02 smithi125 conmon[19936]: [DBG] 5.1a scrub ok 2021-03-24T21:53:02.161 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:02 smithi125 conmon[24927]: cluster 2021-03-24T21:53:02.161 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:02 smithi125 conmon[24927]: 2021-03 2021-03-24T21:53:02.161 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:02 smithi125 conmon[24927]: -24T21:53:00.666431+0000 mgr.y (mgr.14144) 247 : cluster 2021-03-24T21:53:02.161 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:02 smithi125 conmon[24927]: [DBG] pgmap v254: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-03-24T21:53:02.162 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:02 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:53:02.162 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:02 smithi125 conmon[24927]: 03-24T21:53:00.682704+0000 osd.3 2021-03-24T21:53:02.162 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:02 smithi125 conmon[24927]: (osd.3) 14 : cluster [DBG] 3.7 scrub ok 2021-03-24T21:53:02.162 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:02 smithi125 conmon[24927]: 2021-03-24T21:53:02.162 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:02 smithi125 conmon[24927]: cluster 2021-03-24T21:53: 2021-03-24T21:53:02.163 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:02 smithi125 conmon[24927]: 00.845329+0000 osd.7 (osd.7) 10 : cluster 2021-03-24T21:53:02.163 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:02 smithi125 conmon[24927]: [DBG] 5.1a scrub ok 2021-03-24T21:53:02.311 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:02 smithi142 conmon[40152]: t=2021-03-24T21:53:02+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2021-03-24T21:53:02.311 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:02 smithi142 conmon[40152]: t=2021-03-24T21:53:02+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2021-03-24T21:53:02.311 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:02 smithi142 conmon[20125]: cluster 2021-03-24T21:53:00.666431+0000 2021-03-24T21:53:02.312 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:02 smithi142 conmon[20125]: mgr.y (mgr.14144) 247 : cluster [DBG] pgmap v254: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-03-24T21:53:02.312 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:02 smithi142 conmon[20125]: cluster 2021-03-24T21:53:00.682704 2021-03-24T21:53:02.312 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:02 smithi142 conmon[20125]: +0000 osd.3 (osd.3) 14 : cluster [DBG] 3.7 scrub ok 2021-03-24T21:53:02.312 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:02 smithi142 conmon[20125]: cluster 2021-03-24T21:53 2021-03-24T21:53:02.312 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:02 smithi142 conmon[20125]: :00.845329+0000 osd.7 (osd.7) 10 : cluster [DBG] 5.1a scrub ok 2021-03-24T21:53:02.492 INFO:teuthology.orchestra.run.smithi125.stdout:77309411369 2021-03-24T21:53:02.493 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph tell osd.3 flush_pg_stats 2021-03-24T21:53:02.561 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:02 smithi142 conmon[40152]: t=2021-03-24T21:53:02+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2021-03-24T21:53:02.561 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:02 smithi142 conmon[40152]: t=2021-03-24T21:53:02+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2021-03-24T21:53:02.863 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:02 smithi142 conmon[40152]: t=2021-03-24T21:53:02+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2021-03-24T21:53:02.864 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:02 smithi142 conmon[40152]: t=2021-03-24T21:53:02+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2021-03-24T21:53:02.864 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:02 smithi142 conmon[40152]: t=2021-03-24T21:53:02+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2021-03-24T21:53:02.864 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:02 smithi142 conmon[40152]: t=2021-03-24T21:53:02+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2021-03-24T21:53:02.864 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:02 smithi142 conmon[40152]: t=2021-03-24T21:53:02+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2021-03-24T21:53:03.117 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:02 smithi142 conmon[40152]: t=2021-03-24T21:53:02+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2021-03-24T21:53:03.117 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:02 smithi142 conmon[40152]: t=2021-03-24T21:53:02+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2021-03-24T21:53:03.117 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:03 smithi142 conmon[40152]: t=2021-03-24T21:53:03+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2021-03-24T21:53:03.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:03 smithi142 conmon[20125]: cluster 2021-03-24T21:53:01.832725+0000 osd.5 (osd.5) 14 : cluster 2021-03-24T21:53:03.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:03 smithi142 conmon[20125]: [DBG] 6.7 scrub ok 2021-03-24T21:53:03.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:03 smithi142 conmon[20125]: cluster 2021-03-24T21:53:02.100234+0000 2021-03-24T21:53:03.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:03 smithi142 conmon[20125]: osd.6 (osd.6) 7 : cluster [DBG] 3.10 scrub ok 2021-03-24T21:53:03.119 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:03 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:53:03.119 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:03 smithi142 conmon[20125]: -03-24T21:53:02.553030+0000 osd.1 (osd.1) 12 : cluster [DBG] 5.19 scrub ok 2021-03-24T21:53:03.119 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:03 smithi142 conmon[20125]: 2021-03-24T21:53:03.280 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:53:03.349 INFO:journalctl@ceph.alertmanager.a.smithi125.stdout:Mar 24 21:53:03 smithi125 systemd[1]: Stopping Ceph alertmanager.a for 3cfa339a-8cea-11eb-90b2-001a4aab830c... 2021-03-24T21:53:03.349 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:03 smithi125 conmon[19936]: cluster 2021-03-24T21:53:01 2021-03-24T21:53:03.350 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:03 smithi125 conmon[19936]: .832725+0000 osd.5 (osd.5) 14 : cluster [DBG] 6.7 scrub ok 2021-03-24T21:53:03.350 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:03 smithi125 conmon[19936]: cluster 2021-03-24T21:53:03.350 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:03 smithi125 conmon[19936]: 2021-03-24T21:53:02.100234+0000 2021-03-24T21:53:03.350 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:03 smithi125 conmon[19936]: osd.6 (osd.6) 7 : 2021-03-24T21:53:03.350 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:03 smithi125 conmon[19936]: cluster [DBG] 3.10 scrub ok 2021-03-24T21:53:03.351 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:03 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:53:03.351 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:03 smithi125 conmon[19936]: -24T21:53:02.553030+0000 osd.1 (osd.1 2021-03-24T21:53:03.351 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:03 smithi125 conmon[19936]: ) 12 : cluster [DBG] 2021-03-24T21:53:03.351 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:03 smithi125 conmon[19936]: 5.19 scrub ok 2021-03-24T21:53:03.352 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:03 smithi125 conmon[24927]: cluster 2021-03-24T21:53:01.832725+0000 osd.5 (osd 2021-03-24T21:53:03.352 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:03 smithi125 conmon[24927]: .5) 14 : cluster [DBG] 6.7 scrub ok 2021-03-24T21:53:03.352 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:03 smithi125 conmon[24927]: cluster 2021-03-24T21:53:02.100234+0000 osd.6 (osd.6) 7 : cluster [DBG] 3.10 scrub ok 2021-03-24T21:53:03.352 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:03 smithi125 conmon[24927]: 2021-03-24T21:53:03.353 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:03 smithi125 conmon[24927]: cluster 2021-03-24T21:53:02.553030+0000 osd.1 (osd 2021-03-24T21:53:03.353 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:03 smithi125 conmon[24927]: .1) 12 : cluster [DBG] 5.19 scrub ok 2021-03-24T21:53:03.500 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:03 smithi142 conmon[40152]: t=2021-03-24T21:53:03+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2021-03-24T21:53:03.500 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:03 smithi142 conmon[40152]: t=2021-03-24T21:53:03+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2021-03-24T21:53:03.500 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:03 smithi142 conmon[40152]: t=2021-03-24T21:53:03+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2021-03-24T21:53:03.500 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:03 smithi142 conmon[40152]: t=2021-03-24T21:53:03+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2021-03-24T21:53:03.658 INFO:journalctl@ceph.alertmanager.a.smithi125.stdout:Mar 24 21:53:03 smithi125 conmon[43414]: level=info ts=2021-03-24T21:53:03.349Z caller=main.go:536 msg="Received SIGTERM, exiting gracefully..." 2021-03-24T21:53:03.659 INFO:journalctl@ceph.alertmanager.a.smithi125.stdout:Mar 24 21:53:03 smithi125 podman[46714]: 2021-03-24 21:53:03.449925173 +0000 UTC m=+0.196428341 container stop 667e2697a3313686f9a894caa448dbda821e104361686019b600081da215c9bd (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-alertmanager.a, maintainer=The Prometheus Authors ) 2021-03-24T21:53:03.659 INFO:journalctl@ceph.alertmanager.a.smithi125.stdout:Mar 24 21:53:03 smithi125 podman[46714]: 2021-03-24 21:53:03.528687372 +0000 UTC m=+0.275190504 container died 667e2697a3313686f9a894caa448dbda821e104361686019b600081da215c9bd (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-alertmanager.a) 2021-03-24T21:53:03.659 INFO:journalctl@ceph.alertmanager.a.smithi125.stdout:Mar 24 21:53:03 smithi125 podman[46714]: 667e2697a3313686f9a894caa448dbda821e104361686019b600081da215c9bd 2021-03-24T21:53:03.754 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:03 smithi142 conmon[40152]: t=2021-03-24T21:53:03+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2021-03-24T21:53:03.755 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:03 smithi142 conmon[40152]: t=2021-03-24T21:53:03+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2021-03-24T21:53:03.755 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:03 smithi142 conmon[40152]: t=2021-03-24T21:53:03+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2021-03-24T21:53:04.068 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:03 smithi142 conmon[40152]: t=2021-03-24T21:53:03+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2021-03-24T21:53:04.068 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:03 smithi142 conmon[40152]: t=2021-03-24T21:53:03+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2021-03-24T21:53:04.068 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:03 smithi142 conmon[40152]: t=2021-03-24T21:53:03+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2021-03-24T21:53:04.069 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:03 smithi142 conmon[40152]: t=2021-03-24T21:53:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2021-03-24T21:53:04.367 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:04 smithi142 conmon[40152]: t=2021-03-24T21:53:04+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2021-03-24T21:53:04.367 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:04 smithi142 conmon[40152]: t=2021-03-24T21:53:04+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2021-03-24T21:53:04.367 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:04 smithi142 conmon[40152]: t=2021-03-24T21:53:04+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2021-03-24T21:53:04.368 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:04 smithi142 conmon[40152]: t=2021-03-24T21:53:04+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2021-03-24T21:53:04.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:04 smithi142 conmon[20125]: cluster 2021-03-24T21:53:02.667302+0000 mgr.y (mgr.14144) 248 : cluster 2021-03-24T21:53:04.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:04 smithi142 conmon[20125]: [DBG] pgmap v255: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-24T21:53:04.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:04 smithi142 conmon[20125]: cluster 2021-03-24T21:53:02.722956+0000 osd.3 (osd.3) 15 2021-03-24T21:53:04.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:04 smithi142 conmon[20125]: : cluster [DBG] 4.14 scrub ok 2021-03-24T21:53:04.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:04 smithi142 conmon[20125]: cluster 2021-03-24T21:53:02.842529+0000 osd.7 ( 2021-03-24T21:53:04.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:04 smithi142 conmon[20125]: osd.7) 11 : cluster [DBG] 6.8 scrub ok 2021-03-24T21:53:04.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:04 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:53:04.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:04 smithi142 conmon[20125]: 21:53:03.595573+0000 osd.1 (osd. 2021-03-24T21:53:04.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:04 smithi142 conmon[20125]: 1) 13 : cluster [DBG] 4.2 scrub ok 2021-03-24T21:53:04.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:04 smithi125 conmon[19936]: cluster 2021-03-24T21:53 2021-03-24T21:53:04.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:04 smithi125 conmon[19936]: :02.667302+0000 mgr.y (mgr.14144) 248 : cluster [DBG] pgmap v255: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-24T21:53:04.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:04 smithi125 conmon[19936]: cluster 2021-03-24T21:53: 2021-03-24T21:53:04.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:04 smithi125 conmon[19936]: 02.722956+0000 osd.3 (osd.3) 15 : cluster [DBG] 4.14 scrub ok 2021-03-24T21:53:04.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:04 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T21:53:04.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:04 smithi125 conmon[19936]: T21:53:02.842529+0000 osd.7 (osd.7) 11 : cluster [DBG] 6.8 scrub ok 2021-03-24T21:53:04.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:04 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:53:04.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:04 smithi125 conmon[19936]: 24T21:53:03.595573+0000 osd.1 (osd. 2021-03-24T21:53:04.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:04 smithi125 conmon[19936]: 1) 13 : cluster [DBG] 4.2 scrub ok 2021-03-24T21:53:04.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:04 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:53:04.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:04 smithi125 conmon[24927]: T21:53:02.667302+0000 mgr.y (mgr.14144) 248 : cluster [DBG] pgmap v255: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-24T21:53:04.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:04 smithi125 conmon[24927]: cluster 2021-03-24T21:53:04.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:04 smithi125 conmon[24927]: 2021-03-24T21:53 2021-03-24T21:53:04.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:04 smithi125 conmon[24927]: :02.722956+0000 2021-03-24T21:53:04.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:04 smithi125 conmon[24927]: osd.3 (osd.3) 15 : cluster [DBG] 2021-03-24T21:53:04.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:04 smithi125 conmon[24927]: 4.14 scrub ok 2021-03-24T21:53:04.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:04 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:53:04.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:04 smithi125 conmon[24927]: T21:53:02.842529+0000 2021-03-24T21:53:04.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:04 smithi125 conmon[24927]: osd.7 (osd.7) 11 : cluster [DBG] 2021-03-24T21:53:04.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:04 smithi125 conmon[24927]: 6.8 scrub ok 2021-03-24T21:53:04.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:04 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:53:04.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:04 smithi125 conmon[24927]: 21:53:03.595573+0000 osd.1 ( 2021-03-24T21:53:04.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:04 smithi125 conmon[24927]: osd.1) 13 : cluster [DBG] 4.2 scrub ok 2021-03-24T21:53:04.706 INFO:journalctl@ceph.alertmanager.a.smithi125.stdout:Mar 24 21:53:04 smithi125 podman[46766]: 2021-03-24 21:53:04.436643297 +0000 UTC m=+1.029627747 container remove 667e2697a3313686f9a894caa448dbda821e104361686019b600081da215c9bd (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-alertmanager.a, maintainer=The Prometheus Authors ) 2021-03-24T21:53:04.706 INFO:journalctl@ceph.alertmanager.a.smithi125.stdout:Mar 24 21:53:04 smithi125 podman[46766]: 2021-03-24 21:53:04.453623437 +0000 UTC m=+1.046607936 volume remove 126442d08ccc7f97f30d2ca9a4aa84ec4a9f9183c23cf478ccbec0c527b8bdea 2021-03-24T21:53:04.707 INFO:journalctl@ceph.alertmanager.a.smithi125.stdout:Mar 24 21:53:04 smithi125 systemd[1]: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c@alertmanager.a.service: Succeeded. 2021-03-24T21:53:04.707 INFO:journalctl@ceph.alertmanager.a.smithi125.stdout:Mar 24 21:53:04 smithi125 systemd[1]: Stopped Ceph alertmanager.a for 3cfa339a-8cea-11eb-90b2-001a4aab830c. 2021-03-24T21:53:04.707 INFO:journalctl@ceph.alertmanager.a.smithi125.stdout:Mar 24 21:53:04 smithi125 systemd[1]: Starting Ceph alertmanager.a for 3cfa339a-8cea-11eb-90b2-001a4aab830c... 2021-03-24T21:53:04.723 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:04 smithi142 conmon[40152]: t=2021-03-24T21:53:04+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2021-03-24T21:53:04.723 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:04 smithi142 conmon[40152]: t=2021-03-24T21:53:04+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2021-03-24T21:53:04.723 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:04 smithi142 conmon[40152]: t=2021-03-24T21:53:04+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2021-03-24T21:53:04.724 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:04 smithi142 conmon[40152]: t=2021-03-24T21:53:04+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2021-03-24T21:53:05.025 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:04 smithi142 conmon[40152]: t=2021-03-24T21:53:04+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2021-03-24T21:53:05.025 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:04 smithi142 conmon[40152]: t=2021-03-24T21:53:04+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2021-03-24T21:53:05.026 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:04 smithi142 conmon[40152]: t=2021-03-24T21:53:04+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2021-03-24T21:53:05.026 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:04 smithi142 conmon[40152]: t=2021-03-24T21:53:04+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2021-03-24T21:53:05.335 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:05 smithi142 conmon[40152]: t=2021-03-24T21:53:05+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2021-03-24T21:53:05.335 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:05 smithi142 conmon[40152]: t=2021-03-24T21:53:05+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2021-03-24T21:53:05.335 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:05 smithi142 conmon[40152]: t=2021-03-24T21:53:05+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2021-03-24T21:53:05.336 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:05 smithi142 conmon[40152]: t=2021-03-24T21:53:05+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2021-03-24T21:53:05.336 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:05 smithi142 conmon[20125]: cluster 2021-03-24T21:53: 2021-03-24T21:53:05.336 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:05 smithi142 conmon[20125]: 03.836451+0000 osd.7 (osd.7) 12 : cluster [DBG] 5.9 scrub ok 2021-03-24T21:53:05.336 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:05 smithi142 conmon[20125]: cluster 2021-03-24T21:53: 2021-03-24T21:53:05.337 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:05 smithi142 conmon[20125]: 04.262973+0000 osd.0 (osd.0) 12 : cluster [DBG] 3.1b scrub ok 2021-03-24T21:53:05.337 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:05 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:53:05.337 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:05 smithi142 conmon[20125]: 21:53:04.624881+0000 osd.2 (osd.2) 12 : cluster [DBG] 5.d scrub ok 2021-03-24T21:53:05.337 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:05 smithi142 conmon[20125]: cluster 2021-03-24T21:53:04. 2021-03-24T21:53:05.337 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:05 smithi142 conmon[20125]: 637873+0000 osd.1 (osd.1) 14 : cluster [DBG] 2021-03-24T21:53:05.338 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:05 smithi142 conmon[20125]: 4.12 scrub ok 2021-03-24T21:53:05.338 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:05 smithi142 conmon[20125]: audit 2021-03-24T21:53:04. 2021-03-24T21:53:05.338 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:05 smithi142 conmon[20125]: 741814+0000 mon.a (mon.0) 568 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T21:53:05.338 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:05 smithi142 conmon[20125]: audit 2021-03-24T 2021-03-24T21:53:05.339 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:05 smithi142 conmon[20125]: 21:53:04.743392+0000 mon.a (mon.0) 569 : 2021-03-24T21:53:05.339 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:05 smithi142 conmon[20125]: audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T21:53:05.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:05 smithi125 conmon[19936]: cluster 2021-03-24T21:53:03. 2021-03-24T21:53:05.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:05 smithi125 conmon[19936]: 836451+0000 osd.7 (osd.7) 12 : cluster [DBG] 5.9 scrub ok 2021-03-24T21:53:05.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:05 smithi125 conmon[19936]: cluster 2021-03-24T21:53:04.262973+0000 osd.0 ( 2021-03-24T21:53:05.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:05 smithi125 conmon[19936]: osd.0) 12 : cluster [DBG] 3.1b scrub ok 2021-03-24T21:53:05.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:05 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:53:05.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:05 smithi125 conmon[19936]: 53:04.624881+0000 osd.2 (osd.2) 2021-03-24T21:53:05.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:05 smithi125 conmon[19936]: 12 : cluster [DBG] 5.d scrub ok 2021-03-24T21:53:05.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:05 smithi125 conmon[19936]: cluster 2021-03-24T21:53:04 2021-03-24T21:53:05.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:05 smithi125 conmon[19936]: .637873+0000 osd.1 (osd.1) 14 : cluster [DBG] 2021-03-24T21:53:05.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:05 smithi125 conmon[19936]: 4.12 scrub ok 2021-03-24T21:53:05.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:05 smithi125 conmon[19936]: audit 2021-03-24T21:53:04 2021-03-24T21:53:05.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:05 smithi125 conmon[19936]: .741814+0000 mon.a (mon.0) 568 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T21:53:05.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:05 smithi125 conmon[19936]: 2021-03-24T21:53:05.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:05 smithi125 conmon[19936]: audit 2021-03- 2021-03-24T21:53:05.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:05 smithi125 conmon[19936]: 24T21:53:04.743392+0000 mon.a (mon.0) 569 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T21:53:05.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:05 smithi125 conmon[24927]: cluster 2021-03-24T21:53:03.836451+0000 osd.7 (osd 2021-03-24T21:53:05.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:05 smithi125 conmon[24927]: .7) 12 : cluster [DBG] 5.9 scrub ok 2021-03-24T21:53:05.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:05 smithi125 conmon[24927]: cluster 2021-03-24T21:53:05.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:05 smithi125 conmon[24927]: 2021-03-24T21:53:04.262973+0000 osd.0 (osd.0) 12 : cluster 2021-03-24T21:53:05.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:05 smithi125 conmon[24927]: [DBG] 3.1b scrub ok 2021-03-24T21:53:05.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:05 smithi125 conmon[24927]: cluster 2021-03-24T21:53:05.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:05 smithi125 conmon[24927]: 2021-03-24T21:53:04.624881+0000 osd.2 (osd.2) 12 : cluster 2021-03-24T21:53:05.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:05 smithi125 conmon[24927]: [DBG] 5.d scrub ok 2021-03-24T21:53:05.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:05 smithi125 conmon[24927]: cluster 2021-03-24T21:53:05.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:05 smithi125 conmon[24927]: 2021-03-24T21:53:04.637873+0000 osd.1 (osd.1) 14 : cluster 2021-03-24T21:53:05.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:05 smithi125 conmon[24927]: [DBG] 4.12 scrub ok 2021-03-24T21:53:05.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:05 smithi125 conmon[24927]: audit 2021-03-24T21:53:04.741814+0000 2021-03-24T21:53:05.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:05 smithi125 conmon[24927]: mon.a (mon.0) 568 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T21:53:05.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:05 smithi125 conmon[24927]: audit 2021-03-24T21:53:04 2021-03-24T21:53:05.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:05 smithi125 conmon[24927]: .743392+0000 mon.a (mon.0) 569 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T21:53:05.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:05 smithi125 conmon[24927]: 2021-03-24T21:53:05.616 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:05 smithi142 conmon[40152]: t=2021-03-24T21:53:05+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2021-03-24T21:53:05.617 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:05 smithi142 conmon[40152]: t=2021-03-24T21:53:05+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2021-03-24T21:53:05.617 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:05 smithi142 conmon[40152]: t=2021-03-24T21:53:05+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2021-03-24T21:53:05.617 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:05 smithi142 conmon[40152]: t=2021-03-24T21:53:05+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2021-03-24T21:53:05.750 INFO:journalctl@ceph.alertmanager.a.smithi125.stdout:Mar 24 21:53:05 smithi125 podman[46963]: 2021-03-24 21:53:05.628102922 +0000 UTC m=+0.198963013 volume create e775b1ffe392f975a680ce008f33665028c758a916c9fd2b663908393ff89b30 2021-03-24T21:53:05.750 INFO:journalctl@ceph.alertmanager.a.smithi125.stdout:Mar 24 21:53:05 smithi125 podman[46963]: 2021-03-24 21:53:05.661459953 +0000 UTC m=+0.232320084 container create dd9caa0c673d996c3a79dd3cc756ee53636584c5abc8cadadb9af5ad7759411a (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-alertmanager.a, maintainer=The Prometheus Authors ) 2021-03-24T21:53:05.988 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:05 smithi142 conmon[40152]: t=2021-03-24T21:53:05+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2021-03-24T21:53:05.989 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:05 smithi142 conmon[40152]: t=2021-03-24T21:53:05+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2021-03-24T21:53:05.989 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:05 smithi142 conmon[40152]: t=2021-03-24T21:53:05+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2021-03-24T21:53:05.989 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:05 smithi142 conmon[40152]: t=2021-03-24T21:53:05+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2021-03-24T21:53:06.014 INFO:journalctl@ceph.alertmanager.a.smithi125.stdout:Mar 24 21:53:05 smithi125 podman[46963]: 2021-03-24 21:53:05.861412462 +0000 UTC m=+0.432272590 container init dd9caa0c673d996c3a79dd3cc756ee53636584c5abc8cadadb9af5ad7759411a (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-alertmanager.a, maintainer=The Prometheus Authors ) 2021-03-24T21:53:06.014 INFO:journalctl@ceph.alertmanager.a.smithi125.stdout:Mar 24 21:53:05 smithi125 conmon[47044]: level=info ts=2021-03-24T21:53:05.900Z caller=main.go:231 msg="Starting Alertmanager" version="(version=0.20.0, branch=HEAD, revision=f74be0400a6243d10bb53812d6fa408ad71ff32d)" 2021-03-24T21:53:06.015 INFO:journalctl@ceph.alertmanager.a.smithi125.stdout:Mar 24 21:53:05 smithi125 conmon[47044]: level=info ts=2021-03-24T21:53:05.900Z caller=main.go:232 build_context="(go=go1.13.5, user=root@00c3106655f8, date=20191211-14:13:14)" 2021-03-24T21:53:06.015 INFO:journalctl@ceph.alertmanager.a.smithi125.stdout:Mar 24 21:53:05 smithi125 conmon[47044]: level=info ts=2021-03-24T21:53:05.902Z caller=cluster.go:161 component=cluster msg="setting advertise address explicitly" addr=172.21.15.125 port=9094 2021-03-24T21:53:06.015 INFO:journalctl@ceph.alertmanager.a.smithi125.stdout:Mar 24 21:53:05 smithi125 conmon[47044]: level=info ts=2021-03-24T21:53:05.903Z caller=cluster.go:623 component=cluster msg="Waiting for gossip to settle..." interval=2s 2021-03-24T21:53:06.015 INFO:journalctl@ceph.alertmanager.a.smithi125.stdout:Mar 24 21:53:05 smithi125 podman[46963]: 2021-03-24 21:53:05.927998993 +0000 UTC m=+0.498859161 container start dd9caa0c673d996c3a79dd3cc756ee53636584c5abc8cadadb9af5ad7759411a (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-alertmanager.a, maintainer=The Prometheus Authors ) 2021-03-24T21:53:06.015 INFO:journalctl@ceph.alertmanager.a.smithi125.stdout:Mar 24 21:53:05 smithi125 bash[46963]: dd9caa0c673d996c3a79dd3cc756ee53636584c5abc8cadadb9af5ad7759411a 2021-03-24T21:53:06.016 INFO:journalctl@ceph.alertmanager.a.smithi125.stdout:Mar 24 21:53:05 smithi125 conmon[47044]: level=info ts=2021-03-24T21:53:05.928Z caller=coordinator.go:119 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2021-03-24T21:53:06.016 INFO:journalctl@ceph.alertmanager.a.smithi125.stdout:Mar 24 21:53:05 smithi125 conmon[47044]: level=info ts=2021-03-24T21:53:05.928Z caller=coordinator.go:131 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2021-03-24T21:53:06.016 INFO:journalctl@ceph.alertmanager.a.smithi125.stdout:Mar 24 21:53:05 smithi125 conmon[47044]: level=info ts=2021-03-24T21:53:05.933Z caller=main.go:497 msg=Listening address=:9093 2021-03-24T21:53:06.016 INFO:journalctl@ceph.alertmanager.a.smithi125.stdout:Mar 24 21:53:05 smithi125 systemd[1]: Started Ceph alertmanager.a for 3cfa339a-8cea-11eb-90b2-001a4aab830c. 2021-03-24T21:53:06.271 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:06 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:53:06.271 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:06 smithi125 conmon[24927]: :53:04.668411+0000 mgr.y (mgr.14144) 249 : cluster [DBG] pgmap v256: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2021-03-24T21:53:06.272 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:06 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:53:06.272 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:06 smithi125 conmon[24927]: 53:04.714335+0000 osd.3 (osd.3) 16 2021-03-24T21:53:06.272 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:06 smithi125 conmon[24927]: : cluster [DBG] 5.0 scrub ok 2021-03-24T21:53:06.272 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:06 smithi125 conmon[24927]: cluster 2021-03-24T21:53:06.273 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:06 smithi125 conmon[24927]: 2021-03-24T21:53: 2021-03-24T21:53:06.273 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:06 smithi125 conmon[24927]: 04.837469+0000 osd.5 (osd. 2021-03-24T21:53:06.273 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:06 smithi125 conmon[24927]: 5) 15 : cluster [DBG] 2021-03-24T21:53:06.273 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:06 smithi125 conmon[24927]: 3.5 scrub ok 2021-03-24T21:53:06.273 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:06 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:53:06.273 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:06 smithi125 conmon[24927]: -03-24T21:53: 2021-03-24T21:53:06.274 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:06 smithi125 conmon[24927]: 05.313223+0000 osd.0 (osd. 2021-03-24T21:53:06.274 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:06 smithi125 conmon[24927]: 0) 13 : cluster [DBG] 4.3 scrub ok 2021-03-24T21:53:06.274 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:06 smithi125 conmon[24927]: 2021-03-24T21:53:06.274 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:06 smithi125 conmon[24927]: audit 2021 2021-03-24T21:53:06.275 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:06 smithi125 conmon[24927]: -03-24T21:53 2021-03-24T21:53:06.275 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:06 smithi125 conmon[24927]: :05.968242+0000 mon.a ( 2021-03-24T21:53:06.275 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:06 smithi125 conmon[24927]: mon.0) 570 : audit [INF] 2021-03-24T21:53:06.275 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:06 smithi125 conmon[24927]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:53:06.275 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:06 smithi125 conmon[19936]: cluster 2021-03-24T21:53:04.668411+0000 2021-03-24T21:53:06.276 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:06 smithi125 conmon[19936]: mgr.y (mgr.14144) 249 : cluster [DBG] pgmap v256: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2021-03-24T21:53:06.276 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:06 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T21:53:06.276 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:06 smithi125 conmon[19936]: T21:53:04.714335+0000 osd.3 (osd.3) 16 : cluster [DBG] 5.0 scrub ok 2021-03-24T21:53:06.276 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:06 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:53:06.276 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:06 smithi125 conmon[19936]: -24T21:53:04.837469+0000 osd.5 ( 2021-03-24T21:53:06.277 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:06 smithi125 conmon[19936]: osd.5) 15 : cluster [DBG] 3.5 scrub ok 2021-03-24T21:53:06.277 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:06 smithi125 conmon[19936]: 2021-03-24T21:53:06.277 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:06 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:53:06.277 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:06 smithi125 conmon[19936]: 53:05.313223+0000 osd.0 (osd.0) 13 2021-03-24T21:53:06.277 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:06 smithi125 conmon[19936]: : cluster [DBG] 4.3 scrub ok 2021-03-24T21:53:06.278 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:06 smithi125 conmon[19936]: audit 2021-03 2021-03-24T21:53:06.278 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:06 smithi125 conmon[19936]: -24T21:53:05.968242+0000 mon.a (mon.0) 2021-03-24T21:53:06.278 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:06 smithi125 conmon[19936]: 570 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:53:06.324 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:05 smithi142 conmon[40152]: t=2021-03-24T21:53:05+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2021-03-24T21:53:06.324 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:06 smithi142 conmon[40152]: t=2021-03-24T21:53:06+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2021-03-24T21:53:06.324 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:06 smithi142 conmon[40152]: t=2021-03-24T21:53:06+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2021-03-24T21:53:06.325 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:06 smithi142 conmon[40152]: t=2021-03-24T21:53:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2021-03-24T21:53:06.325 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:06 smithi142 conmon[20125]: cluster 2021-03-24T21:53: 2021-03-24T21:53:06.325 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:06 smithi142 conmon[20125]: 04.668411+0000 mgr.y (mgr.14144) 249 : cluster [DBG] pgmap v256: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2021-03-24T21:53:06.325 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:06 smithi142 conmon[20125]: cluster 2021-03-24T21:53:06.326 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:06 smithi142 conmon[20125]: 2021-03-24T21:53:04.714335+0000 osd.3 (osd.3 2021-03-24T21:53:06.326 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:06 smithi142 conmon[20125]: ) 16 : cluster [DBG] 5.0 scrub ok 2021-03-24T21:53:06.326 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:06 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:53:06.326 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:06 smithi142 conmon[20125]: T21:53:04.837469+0000 osd.5 ( 2021-03-24T21:53:06.326 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:06 smithi142 conmon[20125]: osd.5) 15 : cluster [DBG] 3.5 scrub ok 2021-03-24T21:53:06.327 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:06 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:53:06.327 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:06 smithi142 conmon[20125]: 03-24T21:53:05.313223+0000 2021-03-24T21:53:06.327 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:06 smithi142 conmon[20125]: osd.0 (osd.0) 13 : cluster [DBG] 2021-03-24T21:53:06.327 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:06 smithi142 conmon[20125]: 4.3 scrub ok 2021-03-24T21:53:06.327 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:06 smithi142 conmon[20125]: audit 2021-03-24T21:53:06.328 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:06 smithi142 conmon[20125]: 2021-03-24T21:53:05.968242+0000 2021-03-24T21:53:06.328 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:06 smithi142 conmon[20125]: mon.a (mon.0) 570 : audit [INF] 2021-03-24T21:53:06.328 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:06 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:53:06.601 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:06 smithi142 conmon[40152]: t=2021-03-24T21:53:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2021-03-24T21:53:06.601 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:06 smithi142 conmon[40152]: t=2021-03-24T21:53:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2021-03-24T21:53:06.867 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:06 smithi142 conmon[40152]: t=2021-03-24T21:53:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2021-03-24T21:53:06.867 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:06 smithi142 conmon[40152]: t=2021-03-24T21:53:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2021-03-24T21:53:06.867 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:06 smithi142 conmon[40152]: t=2021-03-24T21:53:06+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2021-03-24T21:53:07.242 INFO:teuthology.orchestra.run.smithi125.stdout:111669149734 2021-03-24T21:53:07.243 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph tell osd.4 flush_pg_stats 2021-03-24T21:53:07.280 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:07 smithi142 conmon[40152]: t=2021-03-24T21:53:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2021-03-24T21:53:07.280 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:07 smithi142 conmon[40152]: t=2021-03-24T21:53:07+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2021-03-24T21:53:07.280 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:07 smithi142 conmon[20125]: cluster 2021-03-24T21:53:05.847401+0000 2021-03-24T21:53:07.281 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:07 smithi142 conmon[20125]: osd.7 (osd.7) 13 : cluster [DBG] 6.1c scrub ok 2021-03-24T21:53:07.281 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:07 smithi142 conmon[20125]: cephadm 2021-03-24T21:53:05.969574+0000 mgr.y (mgr.14144) 250 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2021-03-24T21:53:07.281 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:07 smithi142 conmon[20125]: 2021-03-24T21:53:07.281 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:07 smithi142 conmon[20125]: cephadm 2021-03-24T21:53:05.972230+0000 mgr.y (mgr.14144 2021-03-24T21:53:07.281 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:07 smithi142 conmon[20125]: ) 251 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi142 2021-03-24T21:53:07.282 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:07 smithi142 conmon[20125]: cluster 2021-03-24T21:53:06.309057+0000 osd.0 (osd 2021-03-24T21:53:07.282 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:07 smithi142 conmon[20125]: .0) 14 : cluster [DBG] 6.16 scrub ok 2021-03-24T21:53:07.282 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:07 smithi142 conmon[20125]: cluster 2021-03-24T21 2021-03-24T21:53:07.282 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:07 smithi142 conmon[20125]: :53:06.800399+0000 osd.5 (osd.5) 16 : cluster [DBG] 2021-03-24T21:53:07.282 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:07 smithi142 conmon[20125]: 2.2 deep-scrub ok 2021-03-24T21:53:07.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:07 smithi125 conmon[19936]: cluster 2021-03-24T21:53:05. 2021-03-24T21:53:07.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:07 smithi125 conmon[19936]: 847401+0000 osd.7 (osd.7) 13 : cluster [DBG] 6.1c scrub ok 2021-03-24T21:53:07.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:07 smithi125 conmon[19936]: cephadm 2021-03-24T 2021-03-24T21:53:07.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:07 smithi125 conmon[19936]: 21:53:05.969574+0000 mgr.y (mgr.14144) 250 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2021-03-24T21:53:07.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:07 smithi125 conmon[19936]: cephadm 2021-03-24T21:53:05.972230+0000 mgr.y (mgr 2021-03-24T21:53:07.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:07 smithi125 conmon[19936]: .14144) 251 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi142 2021-03-24T21:53:07.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:07 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:53:07.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:07 smithi125 conmon[19936]: 03-24T21:53:06.309057+0000 osd.0 (osd.0) 14 : cluster [DBG] 2021-03-24T21:53:07.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:07 smithi125 conmon[19936]: 6.16 scrub ok 2021-03-24T21:53:07.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:07 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:53:07.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:07 smithi125 conmon[19936]: 53:06.800399+0000 osd.5 (osd.5) 16 : cluster [DBG] 2.2 deep-scrub ok 2021-03-24T21:53:07.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:07 smithi125 conmon[24927]: cluster 2021-03-24T21:53:07.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:07 smithi125 conmon[24927]: 2021-03-24 2021-03-24T21:53:07.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:07 smithi125 conmon[24927]: T21:53:05.847401+0000 osd.7 (osd.7) 13 : cluster [DBG] 2021-03-24T21:53:07.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:07 smithi125 conmon[24927]: 6.1c scrub ok 2021-03-24T21:53:07.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:07 smithi125 conmon[24927]: cephadm 2021 2021-03-24T21:53:07.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:07 smithi125 conmon[24927]: -03-24T21:53:05 2021-03-24T21:53:07.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:07 smithi125 conmon[24927]: .969574+0000 mgr.y (mgr. 2021-03-24T21:53:07.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:07 smithi125 conmon[24927]: 14144) 250 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2021-03-24T21:53:07.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:07 smithi125 conmon[24927]: 2021-03-24T21:53:07.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:07 smithi125 conmon[24927]: cephadm 2021-03-24T21:53:07.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:07 smithi125 conmon[24927]: 2021-03-24T21: 2021-03-24T21:53:07.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:07 smithi125 conmon[24927]: 53:05.972230+0000 mgr.y ( 2021-03-24T21:53:07.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:07 smithi125 conmon[24927]: mgr.14144) 251 : cephadm 2021-03-24T21:53:07.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:07 smithi125 conmon[24927]: [INF] Reconfiguring daemon prometheus.a on smithi142 2021-03-24T21:53:07.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:07 smithi125 conmon[24927]: cluster 2021-03-24T21:53:07.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:07 smithi125 conmon[24927]: 2021-03- 2021-03-24T21:53:07.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:07 smithi125 conmon[24927]: 24T21:53:06.309057 2021-03-24T21:53:07.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:07 smithi125 conmon[24927]: +0000 osd.0 (osd.0) 14 2021-03-24T21:53:07.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:07 smithi125 conmon[24927]: : cluster [DBG] 6.16 scrub ok 2021-03-24T21:53:07.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:07 smithi125 conmon[24927]: cluster 2021-03-24T21:53:07.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:07 smithi125 conmon[24927]: 2021-03-24T21:53:06.800399+0000 2021-03-24T21:53:07.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:07 smithi125 conmon[24927]: osd.5 (osd.5) 16 : cluster [DBG] 2.2 deep-scrub ok 2021-03-24T21:53:07.561 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:53:07.616 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:07 smithi142 conmon[40152]: t=2021-03-24T21:53:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2021-03-24T21:53:07.617 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:07 smithi142 conmon[40152]: t=2021-03-24T21:53:07+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2021-03-24T21:53:08.052 INFO:journalctl@ceph.alertmanager.a.smithi125.stdout:Mar 24 21:53:07 smithi125 conmon[47044]: level=info ts=2021-03-24T21:53:07.903Z caller=cluster.go:648 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.00012893s 2021-03-24T21:53:08.088 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:07 smithi142 conmon[40152]: t=2021-03-24T21:53:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2021-03-24T21:53:08.088 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:07 smithi142 conmon[40152]: t=2021-03-24T21:53:07+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2021-03-24T21:53:08.366 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:08 smithi142 conmon[40152]: t=2021-03-24T21:53:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2021-03-24T21:53:08.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:08 smithi142 conmon[20125]: cluster 2021-03-24T21:53 2021-03-24T21:53:08.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:08 smithi142 conmon[20125]: :06.669374+0000 mgr.y (mgr.14144) 252 : cluster [DBG] pgmap v257: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-03-24T21:53:08.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:08 smithi125 conmon[19936]: cluster 2021-03-24T21:53 2021-03-24T21:53:08.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:08 smithi125 conmon[19936]: :06.669374+0000 mgr.y (mgr.14144) 252 : cluster [DBG] pgmap v257: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-03-24T21:53:08.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:08 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:53:08.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:08 smithi125 conmon[24927]: T21:53:06.669374+0000 mgr.y (mgr.14144) 252 : cluster [DBG] pgmap v257: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-03-24T21:53:08.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:08 smithi125 conmon[24927]: 2021-03-24T21:53:08.884 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:08 smithi142 conmon[40152]: t=2021-03-24T21:53:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2021-03-24T21:53:08.884 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:08 smithi142 conmon[40152]: t=2021-03-24T21:53:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2021-03-24T21:53:09.135 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:09 smithi125 conmon[19936]: cluster 2021-03-24T21:53 2021-03-24T21:53:09.136 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:09 smithi125 conmon[19936]: :08.691527+0000 osd.1 (osd.1) 15 : cluster [DBG] 3.4 scrub ok 2021-03-24T21:53:09.136 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:09 smithi125 conmon[19936]: cluster 2021-03-24T21:53:08. 2021-03-24T21:53:09.136 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:09 smithi125 conmon[19936]: 815146+0000 osd.5 (osd.5) 17 : cluster [DBG] 2021-03-24T21:53:09.136 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:09 smithi125 conmon[19936]: 5.15 deep-scrub ok 2021-03-24T21:53:09.137 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:09 smithi125 conmon[24927]: cluster 2021-03-24T21:53:08.691527+0000 osd.1 (osd.1) 15 : cluster [DBG] 3.4 scrub ok 2021-03-24T21:53:09.137 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:09 smithi125 conmon[24927]: cluster 2021-03-24T21:53:08.815146+0000 osd.5 (osd.5) 17 : cluster [DBG] 5.15 deep-scrub ok 2021-03-24T21:53:09.234 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:09 smithi142 conmon[20125]: cluster 2021-03-24T21:53:08.691527+0000 osd.1 (osd.1 2021-03-24T21:53:09.234 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:09 smithi142 conmon[20125]: ) 15 : cluster [DBG] 3.4 scrub ok 2021-03-24T21:53:09.235 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:09 smithi142 conmon[20125]: cluster 2021-03-24T21:53:08.815146+0000 2021-03-24T21:53:09.235 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:09 smithi142 conmon[20125]: osd.5 (osd.5) 17 : cluster [DBG] 5.15 deep-scrub ok 2021-03-24T21:53:09.235 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:08 smithi142 conmon[40152]: t=2021-03-24T21:53:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2021-03-24T21:53:09.235 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:08 smithi142 conmon[40152]: t=2021-03-24T21:53:08+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2021-03-24T21:53:09.236 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:09 smithi142 conmon[40152]: t=2021-03-24T21:53:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2021-03-24T21:53:09.567 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:09 smithi142 conmon[40152]: t=2021-03-24T21:53:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2021-03-24T21:53:09.567 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:09 smithi142 conmon[40152]: t=2021-03-24T21:53:09+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2021-03-24T21:53:09.827 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:09 smithi142 conmon[40152]: t=2021-03-24T21:53:09+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2021-03-24T21:53:09.827 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:09 smithi142 conmon[40152]: t=2021-03-24T21:53:09+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2021-03-24T21:53:10.109 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:09 smithi142 conmon[40152]: t=2021-03-24T21:53:09+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2021-03-24T21:53:10.109 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:09 smithi142 conmon[40152]: t=2021-03-24T21:53:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2021-03-24T21:53:10.276 INFO:teuthology.orchestra.run.smithi125.stdout:137438953505 2021-03-24T21:53:10.277 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph tell osd.5 flush_pg_stats 2021-03-24T21:53:10.367 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:10 smithi142 conmon[40152]: t=2021-03-24T21:53:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2021-03-24T21:53:10.367 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:10 smithi142 conmon[40152]: t=2021-03-24T21:53:10+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2021-03-24T21:53:10.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:10 smithi142 conmon[20125]: cluster 2021-03-24T21:53:08 2021-03-24T21:53:10.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:10 smithi142 conmon[20125]: .426186+0000 osd.0 (osd.0) 15 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:53:10.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:10 smithi142 conmon[20125]: cluster 2021-03-24T21:53:08.426331+0000 osd.0 (osd.0 2021-03-24T21:53:10.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:10 smithi142 conmon[20125]: ) 16 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:53:10.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:10 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:53:10.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:10 smithi142 conmon[20125]: -24T21:53:08.670211+0000 2021-03-24T21:53:10.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:10 smithi142 conmon[20125]: mgr.y (mgr.14144) 2021-03-24T21:53:10.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:10 smithi142 conmon[20125]: 253 : cluster [DBG] pgmap v258: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-03-24T21:53:10.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:10 smithi142 conmon[20125]: 2021-03-24T21:53:10.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:10 smithi142 conmon[20125]: cluster 2021-03-24T21:53:10.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:10 smithi142 conmon[20125]: 2021-03-24T21:53:08. 2021-03-24T21:53:10.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:10 smithi142 conmon[20125]: 841300+0000 osd.7 (osd.7) 14 2021-03-24T21:53:10.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:10 smithi142 conmon[20125]: : cluster [DBG] 3.15 scrub ok 2021-03-24T21:53:10.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:10 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:53:10.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:10 smithi142 conmon[20125]: 24T21:53:09. 2021-03-24T21:53:10.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:10 smithi142 conmon[20125]: 103010+0000 mon.a (mon.0) 571 2021-03-24T21:53:10.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:10 smithi142 conmon[20125]: : cluster [DBG] osdmap e87: 8 total, 8 up, 8 in 2021-03-24T21:53:10.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:10 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:53:10.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:10 smithi142 conmon[20125]: T21:53:09.740856+0000 2021-03-24T21:53:10.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:10 smithi142 conmon[20125]: osd.1 (osd.1) 16 : cluster 2021-03-24T21:53:10.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:10 smithi142 conmon[20125]: [DBG] 3.19 scrub ok 2021-03-24T21:53:10.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:10 smithi125 conmon[24927]: cluster 2021-03-24T21:53:08.426186 2021-03-24T21:53:10.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:10 smithi125 conmon[24927]: +0000 osd.0 (osd.0) 15 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:53:10.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:10 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:53:10.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:10 smithi125 conmon[24927]: 03-24T21:53:08.426331+0000 osd.0 (osd.0) 2021-03-24T21:53:10.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:10 smithi125 conmon[24927]: 16 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:53:10.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:10 smithi125 conmon[24927]: cluster 2021-03-24T21:53:10.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:10 smithi125 conmon[24927]: 2021-03-24T21:53:08.670211+0000 mgr.y (mgr.14144) 253 : cluster 2021-03-24T21:53:10.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:10 smithi125 conmon[24927]: [DBG] pgmap v258: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-03-24T21:53:10.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:10 smithi125 conmon[24927]: cluster 2021-03-24T21:53:10.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:10 smithi125 conmon[24927]: 2021-03-24T21:53:10.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:10 smithi125 conmon[24927]: 2021-03-24 2021-03-24T21:53:10.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:10 smithi125 conmon[24927]: T21:53:08.841300+0000 osd.7 (osd.7) 2021-03-24T21:53:10.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:10 smithi125 conmon[24927]: 14 : cluster [DBG] 3.15 scrub ok 2021-03-24T21:53:10.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:10 smithi125 conmon[24927]: cluster 2021-03-24T21:53:10.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:10 smithi125 conmon[24927]: 2021-03-24T21:53:09.103010+0000 mon.a (mon 2021-03-24T21:53:10.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:10 smithi125 conmon[24927]: .0) 571 : cluster [DBG] osdmap e87: 8 total, 8 up, 8 in 2021-03-24T21:53:10.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:10 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:53:10.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:10 smithi125 conmon[24927]: T21:53:09.740856+0000 osd.1 (osd.1 2021-03-24T21:53:10.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:10 smithi125 conmon[24927]: ) 16 : cluster [DBG] 3.19 scrub ok 2021-03-24T21:53:10.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:10 smithi125 conmon[19936]: cluster 2021-03-24T21:53:08.426186+0000 osd.0 (osd. 2021-03-24T21:53:10.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:10 smithi125 conmon[19936]: 0) 15 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:53:10.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:10 smithi125 conmon[19936]: cluster 2021-03-24T21:53:08.426331+0000 osd.0 (osd.0) 16 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:53:10.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:10 smithi125 conmon[19936]: cluster 2021-03-24T21:53:08.670211+0000 mgr.y (mgr.14144) 253 : cluster [DBG] pgmap v258: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-03-24T21:53:10.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:10 smithi125 conmon[19936]: cluster 2021-03-24T21:53:08.841300+0000 osd.7 ( 2021-03-24T21:53:10.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:10 smithi125 conmon[19936]: osd.7) 14 : cluster [DBG] 3.15 scrub ok 2021-03-24T21:53:10.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:10 smithi125 conmon[19936]: cluster 2021-03-24T21:53:09.103010+0000 mon.a 2021-03-24T21:53:10.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:10 smithi125 conmon[19936]: (mon.0) 571 : cluster [DBG] osdmap e87: 8 total, 8 up, 8 in 2021-03-24T21:53:10.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:10 smithi125 conmon[19936]: cluster 2021-03-24T21:53:09.740856+0000 osd.1 (osd.1) 16 : cluster [DBG] 2021-03-24T21:53:10.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:10 smithi125 conmon[19936]: 3.19 scrub ok 2021-03-24T21:53:10.585 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:53:11.081 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:10 smithi142 conmon[40152]: t=2021-03-24T21:53:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2021-03-24T21:53:11.333 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:11 smithi142 conmon[40152]: t=2021-03-24T21:53:11+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2021-03-24T21:53:11.333 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:11 smithi142 conmon[40152]: t=2021-03-24T21:53:11+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2021-03-24T21:53:11.334 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:11 smithi142 conmon[20125]: cluster 2021-03-24T21:53:09.854297+0000 osd.5 (osd.5) 18 : 2021-03-24T21:53:11.334 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:11 smithi142 conmon[20125]: cluster [DBG] 6.a scrub ok 2021-03-24T21:53:11.334 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:11 smithi142 conmon[20125]: cluster 2021-03-24T21:53:09.874675+0000 osd.7 2021-03-24T21:53:11.335 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:11 smithi142 conmon[20125]: (osd.7) 15 : cluster [DBG] 5.4 scrub ok 2021-03-24T21:53:11.335 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:11 smithi142 conmon[20125]: cluster 2021-03-24T21:53:11.335 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:11 smithi142 conmon[20125]: 2021-03-24T21:53: 2021-03-24T21:53:11.335 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:11 smithi142 conmon[20125]: 10.104795+0000 osd.4 (osd.4) 12 2021-03-24T21:53:11.336 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:11 smithi142 conmon[20125]: : cluster [DBG] 4.1a scrub ok 2021-03-24T21:53:11.336 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:53:11 smithi142 systemd[1]: Stopping Ceph prometheus.a for 3cfa339a-8cea-11eb-90b2-001a4aab830c... 2021-03-24T21:53:11.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:11 smithi125 conmon[19936]: cluster 2021-03-24T21:53:09.854297+0000 osd.5 (osd.5) 18 : cluster 2021-03-24T21:53:11.542 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:11 smithi125 conmon[19936]: [DBG] 6.a scrub ok 2021-03-24T21:53:11.542 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:11 smithi125 conmon[19936]: cluster 2021-03-24T21:53:09.874675+0000 osd.7 (osd.7) 15 : cluster [DBG] 2021-03-24T21:53:11.543 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:11 smithi125 conmon[19936]: 5.4 scrub ok 2021-03-24T21:53:11.543 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:11 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T21:53:11.543 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:11 smithi125 conmon[19936]: T21:53:10.104795+0000 osd.4 (osd.4) 2021-03-24T21:53:11.543 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:11 smithi125 conmon[19936]: 12 : cluster [DBG] 2021-03-24T21:53:11.544 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:11 smithi125 conmon[19936]: 4.1a scrub ok 2021-03-24T21:53:11.544 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:11 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:53:11.544 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:11 smithi125 conmon[24927]: 21:53:09.854297+0000 osd.5 (osd.5) 18 2021-03-24T21:53:11.545 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:11 smithi125 conmon[24927]: : cluster [DBG] 6.a scrub ok 2021-03-24T21:53:11.545 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:11 smithi125 conmon[24927]: 2021-03-24T21:53:11.545 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:11 smithi125 conmon[24927]: cluster 2021-03-24T21:53:11.545 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:11 smithi125 conmon[24927]: 2021-03- 2021-03-24T21:53:11.546 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:11 smithi125 conmon[24927]: 24T21:53 2021-03-24T21:53:11.546 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:11 smithi125 conmon[24927]: :09.874675+0000 2021-03-24T21:53:11.546 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:11 smithi125 conmon[24927]: osd.7 (osd. 2021-03-24T21:53:11.546 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:11 smithi125 conmon[24927]: 7) 15 : cluster 2021-03-24T21:53:11.546 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:11 smithi125 conmon[24927]: [DBG] 5.4 scrub ok 2021-03-24T21:53:11.547 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:11 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:53:11.547 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:11 smithi125 conmon[24927]: T21:53:10.104795+0000 osd.4 (osd.4) 12 : cluster [DBG] 4.1a scrub ok 2021-03-24T21:53:11.617 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:11 smithi142 conmon[40152]: t=2021-03-24T21:53:11+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2021-03-24T21:53:11.618 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:11 smithi142 conmon[40152]: t=2021-03-24T21:53:11+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2021-03-24T21:53:11.618 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:11 smithi142 conmon[40152]: t=2021-03-24T21:53:11+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2021-03-24T21:53:11.618 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:53:11 smithi142 conmon[38407]: level=warn ts=2021-03-24T21:53:11.423Z caller=main.go:524 msg="Received SIGTERM, exiting gracefully..." 2021-03-24T21:53:11.619 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:53:11 smithi142 conmon[38407]: level=info ts=2021-03-24T21:53:11.424Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2021-03-24T21:53:11.619 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:53:11 smithi142 conmon[38407]: level=info ts=2021-03-24T21:53:11.424Z caller=main.go:561 msg="Stopping notify discovery manager..." 2021-03-24T21:53:11.619 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:53:11 smithi142 conmon[38407]: level=info ts=2021-03-24T21:53:11.424Z caller=main.go:583 msg="Stopping scrape manager..." 2021-03-24T21:53:11.619 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:53:11 smithi142 conmon[38407]: level=info ts=2021-03-24T21:53:11.424Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2021-03-24T21:53:11.620 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:53:11 smithi142 conmon[38407]: level=info ts=2021-03-24T21:53:11.424Z caller=main.go:543 msg="Scrape discovery manager stopped" 2021-03-24T21:53:11.620 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:53:11 smithi142 conmon[38407]: level=info ts=2021-03-24T21:53:11.424Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2021-03-24T21:53:11.620 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:53:11 smithi142 conmon[38407]: level=info ts=2021-03-24T21:53:11.424Z caller=main.go:557 msg="Notify discovery manager stopped" 2021-03-24T21:53:11.620 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:53:11 smithi142 conmon[38407]: level=info ts=2021-03-24T21:53:11.424Z caller=main.go:577 msg="Scrape manager stopped" 2021-03-24T21:53:11.620 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:53:11 smithi142 conmon[38407]: level=info ts=2021-03-24T21:53:11.424Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2021-03-24T21:53:11.621 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:53:11 smithi142 conmon[38407]: level=info ts=2021-03-24T21:53:11.424Z caller=main.go:749 msg="Notifier manager stopped" 2021-03-24T21:53:11.621 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:53:11 smithi142 conmon[38407]: level=info ts=2021-03-24T21:53:11.424Z caller=main.go:761 msg="See you next time!" 2021-03-24T21:53:12.017 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:11 smithi142 conmon[40152]: t=2021-03-24T21:53:11+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2021-03-24T21:53:12.341 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:12 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:53:12.341 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:12 smithi125 conmon[24927]: :53:10.429766+0000 osd.0 (osd.0) 17 : cluster [DBG] 3.17 scrub ok 2021-03-24T21:53:12.342 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:12 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:53:12.342 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:12 smithi125 conmon[24927]: -24T21:53:10.671176+0000 mgr.y ( 2021-03-24T21:53:12.342 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:12 smithi125 conmon[24927]: mgr.14144) 254 : cluster [DBG] pgmap v260: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-24T21:53:12.342 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:12 smithi125 conmon[24927]: cluster 2021-03-24T21:53:12.343 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:12 smithi125 conmon[24927]: 2021-03-24T 2021-03-24T21:53:12.343 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:12 smithi125 conmon[24927]: 21:53:11.623345+0000 osd.2 (osd 2021-03-24T21:53:12.343 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:12 smithi125 conmon[24927]: .2) 13 : cluster [DBG] 5.a scrub ok 2021-03-24T21:53:12.343 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:12 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:53:12.344 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:12 smithi125 conmon[24927]: -24T21:53:11.648459+0000 2021-03-24T21:53:12.344 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:12 smithi125 conmon[24927]: osd.3 (osd.3) 17 : cluster [DBG] 2021-03-24T21:53:12.344 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:12 smithi125 conmon[24927]: 3.18 scrub ok 2021-03-24T21:53:12.344 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:12 smithi125 conmon[24927]: cluster 2021-03-24T21:53:12.344 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:12 smithi125 conmon[24927]: 2021-03-24T21:53:11.723689+0000 osd.1 (osd. 2021-03-24T21:53:12.345 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:12 smithi125 conmon[24927]: 1) 17 : cluster [DBG] 6.4 scrub ok 2021-03-24T21:53:12.345 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:12 smithi125 conmon[19936]: cluster 2021-03-24T21:53:10 2021-03-24T21:53:12.345 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:12 smithi125 conmon[19936]: .429766+0000 osd.0 (osd.0) 17 : cluster [DBG] 3.17 scrub ok 2021-03-24T21:53:12.346 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:12 smithi125 conmon[19936]: cluster 2021-03-24T21:53:10.671176+0000 mgr.y ( 2021-03-24T21:53:12.346 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:12 smithi125 conmon[19936]: mgr.14144) 254 : cluster [DBG] pgmap v260: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-24T21:53:12.346 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:12 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:53:12.346 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:12 smithi125 conmon[19936]: 03-24T21:53:11.623345+0000 osd.2 ( 2021-03-24T21:53:12.346 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:12 smithi125 conmon[19936]: osd.2) 13 : cluster [DBG] 5.a scrub ok 2021-03-24T21:53:12.347 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:12 smithi125 conmon[19936]: cluster 2021-03-24T21:53:12.347 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:12 smithi125 conmon[19936]: 2021-03-24T21:53:11.648459+0000 2021-03-24T21:53:12.347 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:12 smithi125 conmon[19936]: osd.3 (osd.3) 17 : cluster [DBG] 3.18 scrub ok 2021-03-24T21:53:12.347 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:12 smithi125 conmon[19936]: 2021-03-24T21:53:12.348 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:12 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:53:12.348 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:12 smithi125 conmon[19936]: -03-24T21:53:11.723689+0000 osd.1 2021-03-24T21:53:12.348 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:12 smithi125 conmon[19936]: (osd.1) 17 : cluster [DBG] 6.4 scrub ok 2021-03-24T21:53:12.367 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:12 smithi142 conmon[40152]: t=2021-03-24T21:53:12+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2021-03-24T21:53:12.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:12 smithi142 conmon[20125]: cluster 2021-03-24T21:53: 2021-03-24T21:53:12.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:12 smithi142 conmon[20125]: 10.429766+0000 osd.0 (osd.0) 17 : cluster [DBG] 3.17 scrub ok 2021-03-24T21:53:12.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:12 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:53:12.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:12 smithi142 conmon[20125]: 21:53:10.671176+0000 mgr.y (mgr.14144) 254 : cluster [DBG] pgmap v260: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-24T21:53:12.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:12 smithi142 conmon[20125]: cluster 2021-03-24T21:53:11.623345+0000 osd.2 2021-03-24T21:53:12.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:12 smithi142 conmon[20125]: (osd.2) 13 : cluster [DBG] 5.a scrub ok 2021-03-24T21:53:12.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:12 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:53:12.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:12 smithi142 conmon[20125]: 53:11.648459+0000 osd.3 (osd.3) 17 : cluster 2021-03-24T21:53:12.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:12 smithi142 conmon[20125]: [DBG] 3.18 scrub ok 2021-03-24T21:53:12.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:12 smithi142 conmon[20125]: cluster 2021-03-24T21:53:12.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:12 smithi142 conmon[20125]: 2021-03-24T21:53:11.723689+0000 osd.1 (osd.1) 17 2021-03-24T21:53:12.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:12 smithi142 conmon[20125]: : cluster [DBG] 6.4 scrub ok 2021-03-24T21:53:12.814 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:53:12 smithi142 podman[41365]: 2021-03-24 21:53:12.641766734 +0000 UTC m=+1.197169426 container remove e182b9c92b75659c3f661b7f31111701b62b50ddc70fa69a90f215597f57669d (image=docker.io/prom/prometheus:v2.18.1, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-prometheus.a, maintainer=The Prometheus Authors ) 2021-03-24T21:53:12.814 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:53:12 smithi142 podman[41326]: e182b9c92b75659c3f661b7f31111701b62b50ddc70fa69a90f215597f57669d 2021-03-24T21:53:12.814 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:53:12 smithi142 systemd[1]: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c@prometheus.a.service: Succeeded. 2021-03-24T21:53:12.815 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:53:12 smithi142 systemd[1]: Stopped Ceph prometheus.a for 3cfa339a-8cea-11eb-90b2-001a4aab830c. 2021-03-24T21:53:12.815 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:53:12 smithi142 systemd[1]: Starting Ceph prometheus.a for 3cfa339a-8cea-11eb-90b2-001a4aab830c... 2021-03-24T21:53:12.815 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:12 smithi142 conmon[40152]: t=2021-03-24T21:53:12+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2021-03-24T21:53:12.815 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:12 smithi142 conmon[40152]: t=2021-03-24T21:53:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2021-03-24T21:53:12.816 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:12 smithi142 conmon[40152]: t=2021-03-24T21:53:12+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2021-03-24T21:53:13.117 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:12 smithi142 conmon[40152]: t=2021-03-24T21:53:12+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2021-03-24T21:53:13.117 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:12 smithi142 conmon[40152]: t=2021-03-24T21:53:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2021-03-24T21:53:13.117 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:12 smithi142 conmon[40152]: t=2021-03-24T21:53:12+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2021-03-24T21:53:13.118 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:13 smithi142 conmon[40152]: t=2021-03-24T21:53:13+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2021-03-24T21:53:13.168 INFO:teuthology.orchestra.run.smithi125.stdout:167503724574 2021-03-24T21:53:13.268 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph tell osd.6 flush_pg_stats 2021-03-24T21:53:13.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:13 smithi125 conmon[19936]: cluster 2021-03-24T21:53:11.925115+0000 osd.5 (osd.5) 19 : cluster [DBG] 2021-03-24T21:53:13.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:13 smithi125 conmon[19936]: 6.d scrub ok 2021-03-24T21:53:13.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:13 smithi125 conmon[19936]: cluster 2021-03-24T21:53:12.030239+0000 osd.4 (osd.4) 13 : cluster [DBG] 6.17 scrub ok 2021-03-24T21:53:13.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:13 smithi125 conmon[19936]: cluster 2021-03-24T21:53:12.604741+0000 osd.3 (osd.3) 18 : cluster [DBG] 4.19 scrub ok 2021-03-24T21:53:13.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:13 smithi125 conmon[19936]: cluster 2021-03-24T21:53:12.648789+0000 osd.3 (osd.3) 19 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:53:13.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:13 smithi125 conmon[19936]: cluster 2021-03-24T21:53: 2021-03-24T21:53:13.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:13 smithi125 conmon[19936]: 12.648910+0000 osd.3 (osd.3) 20 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:53:13.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:13 smithi125 conmon[24927]: cluster 2021-03-24T21:53:13.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:13 smithi125 conmon[24927]: 2021-03 2021-03-24T21:53:13.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:13 smithi125 conmon[24927]: -24T21:53:11.925115+0000 osd.5 (osd.5) 19 : cluster [DBG] 6.d scrub ok 2021-03-24T21:53:13.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:13 smithi125 conmon[24927]: cluster 2021-03-24T21:53:13.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:13 smithi125 conmon[24927]: 2021-03-24T21:53:12.030239+0000 2021-03-24T21:53:13.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:13 smithi125 conmon[24927]: osd.4 (osd.4) 13 : cluster [DBG] 6.17 scrub ok 2021-03-24T21:53:13.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:13 smithi125 conmon[24927]: 2021-03-24T21:53:13.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:13 smithi125 conmon[24927]: cluster 2021-03-24T21:53:12.604741+0000 2021-03-24T21:53:13.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:13 smithi125 conmon[24927]: osd.3 (osd.3) 18 : cluster [DBG] 4.19 scrub ok 2021-03-24T21:53:13.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:13 smithi125 conmon[24927]: cluster 2021-03-24T21:53:13.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:13 smithi125 conmon[24927]: 2021-03-24T21:53: 2021-03-24T21:53:13.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:13 smithi125 conmon[24927]: 12.648789+0000 osd.3 (osd.3) 19 : cluster 2021-03-24T21:53:13.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:13 smithi125 conmon[24927]: [DBG] purged_snaps scrub starts 2021-03-24T21:53:13.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:13 smithi125 conmon[24927]: cluster 2021-03-24T21:53:13.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:13 smithi125 conmon[24927]: 2021-03-24T21:53:12.648910 2021-03-24T21:53:13.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:13 smithi125 conmon[24927]: +0000 osd.3 (osd.3) 20 : cluster [DBG] 2021-03-24T21:53:13.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:13 smithi125 conmon[24927]: purged_snaps scrub ok 2021-03-24T21:53:13.431 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:53:13 smithi142 podman[41487]: 2021-03-24 21:53:13.183348225 +0000 UTC m=+0.331552481 container create 3c5e4fffcf831fd58198da50dff407fb0ac50e251484bcb250111d026d764ff9 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-prometheus.a, maintainer=The Prometheus Authors ) 2021-03-24T21:53:13.432 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:13 smithi142 conmon[40152]: t=2021-03-24T21:53:13+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2021-03-24T21:53:13.432 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:13 smithi142 conmon[40152]: t=2021-03-24T21:53:13+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2021-03-24T21:53:13.433 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:13 smithi142 conmon[20125]: cluster 2021-03-24T21:53:11.925115+0000 2021-03-24T21:53:13.433 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:13 smithi142 conmon[20125]: osd.5 (osd.5) 19 : cluster [DBG] 6.d scrub ok 2021-03-24T21:53:13.434 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:13 smithi142 conmon[20125]: cluster 2021-03-24T21:53:12.030239+0000 osd.4 (osd.4) 13 : cluster [DBG] 6.17 scrub ok 2021-03-24T21:53:13.434 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:13 smithi142 conmon[20125]: cluster 2021-03-24T21:53:12.604741+0000 osd.3 (osd 2021-03-24T21:53:13.435 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:13 smithi142 conmon[20125]: .3) 18 : cluster [DBG] 4.19 scrub ok 2021-03-24T21:53:13.435 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:13 smithi142 conmon[20125]: cluster 2021-03-24T21:53:12.648789+0000 osd.3 (osd.3) 19 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:53:13.435 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:13 smithi142 conmon[20125]: cluster 2021-03-24T21:53:12.648910+0000 osd.3 (osd.3) 20 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:53:13.581 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:53:13.745 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:13 smithi142 conmon[40152]: t=2021-03-24T21:53:13+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2021-03-24T21:53:13.792 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:13 smithi142 conmon[40152]: t=2021-03-24T21:53:13+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2021-03-24T21:53:13.793 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:13 smithi142 conmon[40152]: t=2021-03-24T21:53:13+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2021-03-24T21:53:13.794 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:53:13 smithi142 podman[41487]: 2021-03-24 21:53:13.441566689 +0000 UTC m=+0.589770992 container init 3c5e4fffcf831fd58198da50dff407fb0ac50e251484bcb250111d026d764ff9 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-prometheus.a, maintainer=The Prometheus Authors ) 2021-03-24T21:53:13.794 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:53:13 smithi142 conmon[41527]: level=info ts=2021-03-24T21:53:13.496Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2021-03-24T21:53:13.795 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:53:13 smithi142 conmon[41527]: level=info ts=2021-03-24T21:53:13.496Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2021-03-24T21:53:13.795 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:53:13 smithi142 conmon[41527]: level=info ts=2021-03-24T21:53:13.496Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2021-03-24T21:53:13.796 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:53:13 smithi142 conmon[41527]: level=info ts=2021-03-24T21:53:13.496Z caller=main.go:339 host_details="(Linux 5.4.0-70-generic #78-Ubuntu SMP Fri Mar 19 13:29:52 UTC 2021 x86_64 smithi142 (none))" 2021-03-24T21:53:13.796 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:53:13 smithi142 conmon[41527]: level=info ts=2021-03-24T21:53:13.496Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2021-03-24T21:53:13.797 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:53:13 smithi142 conmon[41527]: level=info ts=2021-03-24T21:53:13.496Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2021-03-24T21:53:13.797 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:53:13 smithi142 conmon[41527]: level=info ts=2021-03-24T21:53:13.499Z caller=main.go:678 msg="Starting TSDB ..." 2021-03-24T21:53:13.797 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:53:13 smithi142 conmon[41527]: level=info ts=2021-03-24T21:53:13.499Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2021-03-24T21:53:13.798 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:53:13 smithi142 conmon[41527]: level=info ts=2021-03-24T21:53:13.502Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2021-03-24T21:53:13.798 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:53:13 smithi142 conmon[41527]: level=info ts=2021-03-24T21:53:13.503Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2021-03-24T21:53:13.799 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:53:13 smithi142 conmon[41527]: level=info ts=2021-03-24T21:53:13.503Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2021-03-24T21:53:13.799 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:53:13 smithi142 conmon[41527]: level=info ts=2021-03-24T21:53:13.503Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=461.26µs 2021-03-24T21:53:13.800 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:53:13 smithi142 conmon[41527]: level=info ts=2021-03-24T21:53:13.504Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2021-03-24T21:53:13.800 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:53:13 smithi142 conmon[41527]: level=info ts=2021-03-24T21:53:13.504Z caller=main.go:695 msg="TSDB started" 2021-03-24T21:53:13.800 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:53:13 smithi142 conmon[41527]: level=info ts=2021-03-24T21:53:13.505Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2021-03-24T21:53:13.801 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:53:13 smithi142 conmon[41527]: level=info ts=2021-03-24T21:53:13.506Z caller=main.go:827 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml 2021-03-24T21:53:13.802 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:53:13 smithi142 conmon[41527]: level=info ts=2021-03-24T21:53:13.506Z caller=main.go:646 msg="Server is ready to receive web requests." 2021-03-24T21:53:13.802 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:53:13 smithi142 podman[41487]: 2021-03-24 21:53:13.508516757 +0000 UTC m=+0.656720993 container start 3c5e4fffcf831fd58198da50dff407fb0ac50e251484bcb250111d026d764ff9 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-prometheus.a, maintainer=The Prometheus Authors ) 2021-03-24T21:53:13.803 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:53:13 smithi142 bash[41487]: 3c5e4fffcf831fd58198da50dff407fb0ac50e251484bcb250111d026d764ff9 2021-03-24T21:53:13.803 INFO:journalctl@ceph.prometheus.a.smithi142.stdout:Mar 24 21:53:13 smithi142 systemd[1]: Started Ceph prometheus.a for 3cfa339a-8cea-11eb-90b2-001a4aab830c. 2021-03-24T21:53:14.008 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:13 smithi142 conmon[40152]: t=2021-03-24T21:53:13+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2021-03-24T21:53:14.008 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:13 smithi142 conmon[40152]: t=2021-03-24T21:53:13+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2021-03-24T21:53:14.008 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:13 smithi142 conmon[40152]: t=2021-03-24T21:53:13+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2021-03-24T21:53:14.009 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[40152]: t=2021-03-24T21:53:14+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2021-03-24T21:53:14.333 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[40152]: t=2021-03-24T21:53:14+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2021-03-24T21:53:14.334 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[40152]: t=2021-03-24T21:53:14+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2021-03-24T21:53:14.335 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[40152]: t=2021-03-24T21:53:14+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2021-03-24T21:53:14.336 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: cluster 2021-03-24T21 2021-03-24T21:53:14.336 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: :53:12.428485+0000 osd.0 (osd.0) 18 : cluster [DBG] 3.1f scrub ok 2021-03-24T21:53:14.337 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:53:14.337 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: 24T21:53:12.672158 2021-03-24T21:53:14.337 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: +0000 mgr.y (mgr.14144) 2021-03-24T21:53:14.338 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: 255 : cluster [DBG] pgmap v261: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-24T21:53:14.338 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:53:14.339 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: -03-24T21:53:12 2021-03-24T21:53:14.339 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: .871261+0000 osd.7 (osd. 2021-03-24T21:53:14.340 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: 7) 16 : cluster [DBG] 2021-03-24T21:53:14.340 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: 3.11 scrub ok 2021-03-24T21:53:14.341 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:53:14.341 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: -03-24T21:53 2021-03-24T21:53:14.342 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: :13.076747+0000 osd.4 ( 2021-03-24T21:53:14.342 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: osd.4) 14 : cluster 2021-03-24T21:53:14.342 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: [DBG] 4.1 scrub ok 2021-03-24T21:53:14.343 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: cluster 2021-03-24T21:53:14.343 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: 2021-03-24T21: 2021-03-24T21:53:14.343 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: 53:13.125871+0000 mon.a ( 2021-03-24T21:53:14.343 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: mon.0) 572 : cluster 2021-03-24T21:53:14.344 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: [DBG] osdmap e88: 8 total, 8 up, 8 in 2021-03-24T21:53:14.344 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:53:14.344 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: 24T21:53:13. 2021-03-24T21:53:14.344 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: 143316+0000 osd.6 (osd.6 2021-03-24T21:53:14.345 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: ) 8 : cluster [DBG] 5.11 scrub ok 2021-03-24T21:53:14.345 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: audit 2021-03-24T21:53:14.345 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: 2021-03-24T21: 2021-03-24T21:53:14.345 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: 53:13.547909+0000 mon.a ( 2021-03-24T21:53:14.346 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: mon.0) 573 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:53:14.346 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: audit 2021-03-24T21:53:14.346 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: 2021-03-24T21: 2021-03-24T21:53:14.346 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: 53:13.551417+0000 mon.a 2021-03-24T21:53:14.347 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: (mon.0) 574 : audit [DBG] 2021-03-24T21:53:14.347 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2021-03-24T21:53:14.347 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: audit 2021-03- 2021-03-24T21:53:14.347 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: 24T21:53:13.553243 2021-03-24T21:53:14.347 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: +0000 mon.a (mon.0) 2021-03-24T21:53:14.348 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: 575 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi125:9093"}]: dispatch 2021-03-24T21:53:14.348 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: audit 2021-03-24T21:53:14.348 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: 2021-03-24T21:53: 2021-03-24T21:53:14.348 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: 13.560003+0000 mon.a (mon 2021-03-24T21:53:14.348 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: .0) 576 : audit [INF] 2021-03-24T21:53:14.349 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:53:14.349 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: audit 2021-03-24T21:53:14.349 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: 2021-03-24T21: 2021-03-24T21:53:14.349 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: 53:13.565775+0000 mon.a 2021-03-24T21:53:14.349 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: (mon.0) 577 : audit 2021-03-24T21:53:14.350 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2021-03-24T21:53:14.350 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: audit 2021 2021-03-24T21:53:14.350 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: -03-24T21:53:13 2021-03-24T21:53:14.350 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: .574264+0000 mon.a (mon. 2021-03-24T21:53:14.351 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: 0) 578 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2021-03-24T21:53:14.351 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: 2021-03-24T21:53:14.351 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:53:14.351 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: 03-24T21:53:13 2021-03-24T21:53:14.351 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: .579916+0000 mon.a (mon.0 2021-03-24T21:53:14.352 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: ) 579 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:53:14.352 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: 2021-03-24T21:53:14.352 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: audit 2021 2021-03-24T21:53:14.352 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: -03-24T21:53 2021-03-24T21:53:14.352 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: :13.599067+0000 mon.a 2021-03-24T21:53:14.353 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: (mon.0) 580 : audit 2021-03-24T21:53:14.353 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi142"}]: dispatch 2021-03-24T21:53:14.353 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: audit 2021-03-24T21:53:14.353 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: 2021-03-24T21: 2021-03-24T21:53:14.353 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: 53:13.611531+0000 mon.a 2021-03-24T21:53:14.354 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: (mon.0) 581 : audit 2021-03-24T21:53:14.354 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:53:14.354 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: audit 2021-03-24T21:53:14.354 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: 2021-03-24T21:53 2021-03-24T21:53:14.354 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: :13.615114+0000 mon.a ( 2021-03-24T21:53:14.355 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: mon.0) 582 : audit [DBG] 2021-03-24T21:53:14.355 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2021-03-24T21:53:14.355 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: audit 2021 2021-03-24T21:53:14.355 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: -03-24T21:53:13 2021-03-24T21:53:14.356 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: .616663+0000 mon.a (mon. 2021-03-24T21:53:14.356 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: 0) 583 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi142:9095"}]: dispatch 2021-03-24T21:53:14.356 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: 2021-03-24T21:53:14.356 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: audit 2021 2021-03-24T21:53:14.356 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: -03-24T21:53: 2021-03-24T21:53:14.357 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: 13.622939+0000 mon.a (mon 2021-03-24T21:53:14.357 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: .0) 584 : audit [INF] 2021-03-24T21:53:14.357 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:53:14.357 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: audit 2021-03-24T21:53:14.357 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: 2021-03-24T21: 2021-03-24T21:53:14.358 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: 53:13.629273+0000 mon.a ( 2021-03-24T21:53:14.358 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: mon.0) 585 : audit 2021-03-24T21:53:14.358 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2021-03-24T21:53:14.358 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:53:14.359 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: 03-24T21:53:13 2021-03-24T21:53:14.359 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: .636846+0000 mon.a (mon.0 2021-03-24T21:53:14.359 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: ) 586 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi142:3000"}]: dispatch 2021-03-24T21:53:14.359 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: 2021-03-24T21:53:14.359 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: audit 2021-03- 2021-03-24T21:53:14.360 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: 24T21:53:13. 2021-03-24T21:53:14.360 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: 643096+0000 mon.a (mon.0 2021-03-24T21:53:14.360 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: ) 587 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:53:14.360 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:53:14.360 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: -03-24T21: 2021-03-24T21:53:14.361 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: 53:13.645923+0000 2021-03-24T21:53:14.361 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: osd.3 (osd.3) 21 : 2021-03-24T21:53:14.361 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: cluster [DBG] 4.17 scrub ok 2021-03-24T21:53:14.361 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: audit 2021-03-24T21:53:14.361 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: 2021-03-24T21: 2021-03-24T21:53:14.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: 53:13.646809+0000 2021-03-24T21:53:14.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: mon.a (mon.0) 2021-03-24T21:53:14.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: 588 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:53:14.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[20125]: 2021-03-24T21:53:14.363 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:53:14.363 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: 53:12.428485+0000 osd.0 (osd.0) 18 : cluster [DBG] 3.1f scrub ok 2021-03-24T21:53:14.363 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: cluster 2021-03-24T21:53:14.364 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: 2021-03-24T21:53:12.672158+0000 mgr.y (mgr. 2021-03-24T21:53:14.364 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: 14144) 255 : cluster [DBG] pgmap v261: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-24T21:53:14.364 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:53:14.364 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: T21:53:12.871261+0000 osd.7 (osd. 2021-03-24T21:53:14.365 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: 7) 16 : cluster [DBG] 3.11 scrub ok 2021-03-24T21:53:14.365 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:53:14.365 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: -03-24T21:53:13.076747 2021-03-24T21:53:14.365 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: +0000 osd.4 (osd.4) 14 : cluster [DBG] 4.1 scrub ok 2021-03-24T21:53:14.365 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:53:14.366 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: 03-24T21:53:13. 2021-03-24T21:53:14.366 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: 125871+0000 mon.a (mon.0) 572 : cluster 2021-03-24T21:53:14.366 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: [DBG] osdmap e88: 8 total, 8 up, 8 in 2021-03-24T21:53:14.366 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: cluster 2021-03-24T21:53:14.367 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: 2021-03-24T21:53:13.143316+0000 2021-03-24T21:53:14.367 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: osd.6 (osd.6) 8 : cluster [DBG] 2021-03-24T21:53:14.367 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: 5.11 scrub ok 2021-03-24T21:53:14.367 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: audit 2021 2021-03-24T21:53:14.368 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: -03-24T21:53:13.547909+0000 mon.a (mon.0 2021-03-24T21:53:14.368 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: ) 573 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:53:14.368 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: audit 2021-03-24T21: 2021-03-24T21:53:14.368 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: 53:13.551417+0000 mon.a (mon.0) 574 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2021-03-24T21:53:14.368 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: audit 2021-03-24T21:53:13.553243+0000 2021-03-24T21:53:14.369 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: mon.a (mon.0) 575 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi125:9093"}]: dispatch 2021-03-24T21:53:14.369 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: audit 2021- 2021-03-24T21:53:14.369 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: 03-24T21:53:13.560003+0000 mon.a (mon 2021-03-24T21:53:14.370 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: .0) 576 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:53:14.370 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: audit 2021-03-24T21:53:14.370 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: 2021-03-24T 2021-03-24T21:53:14.370 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: 21:53:13.565775+0000 mon.a (mon.0 2021-03-24T21:53:14.371 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: ) 577 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2021-03-24T21:53:14.371 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: audit 2021- 2021-03-24T21:53:14.371 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: 03-24T21:53:13.574264+0000 mon.a ( 2021-03-24T21:53:14.372 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: mon.0) 578 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2021-03-24T21:53:14.372 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: audit 2021- 2021-03-24T21:53:14.373 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: 03-24T21:53:13.579916+0000 mon.a ( 2021-03-24T21:53:14.373 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: mon.0) 579 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:53:14.373 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: audit 2021-03 2021-03-24T21:53:14.374 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: -24T21:53:13.599067+0000 mon.a (mon 2021-03-24T21:53:14.374 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: .0) 580 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi142"}]: dispatch 2021-03-24T21:53:14.375 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: audit 2021 2021-03-24T21:53:14.375 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: -03-24T21:53:13.611531+0000 mon.a 2021-03-24T21:53:14.376 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: cluster 2021-03-24T21:53:12.428485+0000 osd.0 (osd 2021-03-24T21:53:14.376 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: .0) 18 : cluster [DBG] 3.1f scrub ok 2021-03-24T21:53:14.377 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: cluster 2021-03-24T21:53:12 2021-03-24T21:53:14.377 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: .672158+0000 mgr.y (mgr.14144) 255 : cluster [DBG] pgmap v261: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-24T21:53:14.378 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: cluster 2021-03-24T21:53:14.378 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: 2021-03-24T21:53: 2021-03-24T21:53:14.379 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: 12.871261+0000 osd.7 (osd. 2021-03-24T21:53:14.379 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: 7) 16 : cluster [DBG] 3.11 scrub ok 2021-03-24T21:53:14.379 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: 2021-03-24T21:53:14.380 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:53:14.380 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: -03-24T21:53: 2021-03-24T21:53:14.381 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: 13.076747+0000 osd.4 (osd. 2021-03-24T21:53:14.381 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: 4) 14 : cluster [DBG] 4.1 scrub ok 2021-03-24T21:53:14.381 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: 2021-03-24T21:53:14.382 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T21:53:14.382 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: T21:53:13.125871+0000 2021-03-24T21:53:14.383 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: mon.a (mon.0) 572 : 2021-03-24T21:53:14.383 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: cluster [DBG] osdmap e88: 8 total, 8 up, 8 in 2021-03-24T21:53:14.384 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: cluster 2021-03-24T21:53:14.384 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: 2021-03-24T21: 2021-03-24T21:53:14.384 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: 53:13.143316+0000 osd.6 ( 2021-03-24T21:53:14.385 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: osd.6) 8 : cluster [DBG] 2021-03-24T21:53:14.385 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: 5.11 scrub ok 2021-03-24T21:53:14.386 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: audit 2021-03-24T21:53:14.386 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: 2021-03-24T21:53: 2021-03-24T21:53:14.386 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: 13.547909+0000 mon.a (mon. 2021-03-24T21:53:14.387 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: 0) 573 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:53:14.387 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: audit 2021 2021-03-24T21:53:14.388 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: -03-24T21:53:13 2021-03-24T21:53:14.388 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: .551417+0000 mon.a (mon.0) 2021-03-24T21:53:14.388 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: 574 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2021-03-24T21:53:14.389 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: audit 2021 2021-03-24T21:53:14.389 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: -03-24T21:53:13. 2021-03-24T21:53:14.390 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: 553243+0000 mon.a (mon.0 2021-03-24T21:53:14.390 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: ) 575 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi125:9093"}]: dispatch 2021-03-24T21:53:14.391 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: audit 2021-03-24T21:53:14.391 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: 2021-03 2021-03-24T21:53:14.391 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: -24T21:53:13. 2021-03-24T21:53:14.392 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: 560003+0000 mon.a (mon.0) 576 2021-03-24T21:53:14.392 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:53:14.392 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: audit 2021-03-24T21:53:14.393 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: 2021-03-24T21:53: 2021-03-24T21:53:14.393 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: 13.565775+0000 mon.a (mon.0) 2021-03-24T21:53:14.393 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: 577 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2021-03-24T21:53:14.393 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: audit 2021-03 2021-03-24T21:53:14.393 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: -24T21:53:13.574264+0000 mon.a (mon.0 2021-03-24T21:53:14.394 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: ) 578 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2021-03-24T21:53:14.394 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: audit 2021-03-24T21:53:14.394 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: 2021-03-24T21:53:13.579916+0000 2021-03-24T21:53:14.394 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: mon.a (mon.0) 579 : audit 2021-03-24T21:53:14.394 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:53:14.395 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:53:14.395 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: (mon.0) 581 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:53:14.396 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: audit 2021-03-24T21:53:14.396 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: 2021-03-24T21:53:13.615114+0000 2021-03-24T21:53:14.397 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: mon.a (mon.0) 582 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2021-03-24T21:53:14.397 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: audit 2021-03-24T21:53:14.397 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: 2021- 2021-03-24T21:53:14.397 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: 03-24T21:53:13.616663+0000 mon.a (mon.0) 2021-03-24T21:53:14.397 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: 583 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi142:9095"}]: dispatch 2021-03-24T21:53:14.398 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: audit 2021 2021-03-24T21:53:14.398 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: -03-24T21:53:13.622939+0000 mon.a ( 2021-03-24T21:53:14.398 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: mon.0) 584 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:53:14.398 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: audit 2021-03-24T21:53:14.399 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: 2021-03-24T21:53:13.629273+0000 mon.a (mon. 2021-03-24T21:53:14.399 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: 0) 585 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2021-03-24T21:53:14.399 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: audit 2021-03-24T21:53:14.399 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: 2021-03-24T21:53:13.636846 2021-03-24T21:53:14.399 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: +0000 mon.a (mon.0) 586 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi142:3000"}]: dispatch 2021-03-24T21:53:14.400 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: 2021-03-24T21:53:14.400 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: audit 2021-03-24T21:53:13 2021-03-24T21:53:14.400 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: .643096+0000 mon.a (mon.0) 587 : audit 2021-03-24T21:53:14.400 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:53:14.400 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: cluster 2021-03-24T21:53:14.401 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: 2021-03-24T21:53:13.645923 2021-03-24T21:53:14.401 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: +0000 osd.3 (osd.3) 21 : cluster [DBG] 2021-03-24T21:53:14.401 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: 4.17 scrub ok 2021-03-24T21:53:14.401 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: audit 2021-03-24T 2021-03-24T21:53:14.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: 21:53:13.646809+0000 mon.a (mon. 2021-03-24T21:53:14.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[24927]: 0) 588 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:53:14.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: 03-24T21:53:13.599067+0000 2021-03-24T21:53:14.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: mon.a (mon.0) 580 : audit [INF] 2021-03-24T21:53:14.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi142"}]: dispatch 2021-03-24T21:53:14.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: audit 2021-03-24T21:53:14.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: 2021-03-24T21:53:13.611531 2021-03-24T21:53:14.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: +0000 mon.a (mon.0) 581 2021-03-24T21:53:14.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:53:14.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:53:14.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: 03-24T21:53:13.615114+0000 mon.a (mon 2021-03-24T21:53:14.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: .0) 582 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2021-03-24T21:53:14.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: 2021-03-24T21:53:14.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:53:14.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: 03-24T21:53:13.616663+0000 2021-03-24T21:53:14.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: mon.a (mon.0) 583 : audit 2021-03-24T21:53:14.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi142:9095"}]: dispatch 2021-03-24T21:53:14.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: audit 2021-03-24T21: 2021-03-24T21:53:14.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: 53:13.622939+0000 mon.a (mon. 2021-03-24T21:53:14.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: 0) 584 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:53:14.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: audit 2021-03-24T21 2021-03-24T21:53:14.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: :53:13.629273+0000 mon.a (mon.0) 585 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2021-03-24T21:53:14.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:53:14.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: 03-24T21:53 2021-03-24T21:53:14.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: :13.636846+0000 mon.a (mon. 2021-03-24T21:53:14.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: 0) 586 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi142:3000"}]: dispatch 2021-03-24T21:53:14.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: 2021-03-24T21:53:14.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: audit 2021-03-24T21:53:14.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: 2021 2021-03-24T21:53:14.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: - 2021-03-24T21:53:14.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: 03-24T21:53:13 2021-03-24T21:53:14.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: .643096+0000 mon.a (mon.0) 587 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:53:14.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:53:14.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: 24T21:53:13.645923+0000 osd.3 2021-03-24T21:53:14.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: (osd.3) 21 : cluster [DBG] 2021-03-24T21:53:14.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: 4.17 scrub ok 2021-03-24T21:53:14.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: audit 2021-03-24T21:53:14.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: 2021-03-24 2021-03-24T21:53:14.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: T21:53:13.646809+0000 mon.a (mon.0 2021-03-24T21:53:14.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:14 smithi125 conmon[19936]: ) 588 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:53:14.617 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[40152]: t=2021-03-24T21:53:14+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2021-03-24T21:53:14.623 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[40152]: t=2021-03-24T21:53:14+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2021-03-24T21:53:14.623 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[40152]: t=2021-03-24T21:53:14+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2021-03-24T21:53:15.002 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[40152]: t=2021-03-24T21:53:14+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2021-03-24T21:53:15.005 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[40152]: t=2021-03-24T21:53:14+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2021-03-24T21:53:15.006 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:14 smithi142 conmon[40152]: t=2021-03-24T21:53:14+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2021-03-24T21:53:15.298 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:15 smithi125 conmon[19936]: cluster 2021-03-24T21:53:13.461937+0000 osd.0 (osd.0) 19 : cluster [DBG] 6.10 scrub ok 2021-03-24T21:53:15.299 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:15 smithi125 conmon[19936]: audit 2021-03-24T21:53:13.551925+0000 mgr.y (mgr.14144) 256 : audit [DBG] from='mon.0 v1:172.21.15.125:0/3961911888' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2021-03-24T21:53:15.299 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:15 smithi125 conmon[19936]: audit 2021-03-24T21:53:13.553826+0000 mgr.y (mgr.14144) 257 : audit [DBG] from='mon.0 v1:172.21.15.125:0/3961911888' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi125:9093"}]: dispatch 2021-03-24T21:53:15.300 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:15 smithi125 conmon[19936]: audit 2021-03-24T21:53:13.566379+0000 mgr.y (mgr.14144) 258 : audit [DBG] from='mon.0 v1:172.21.15.125:0/3961911888' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2021-03-24T21:53:15.300 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:15 smithi125 conmon[19936]: cephadm 2021-03-24T21:53:13.573932+0000 mgr.y (mgr.14144) 259 : cephadm [INF] Adding iSCSI gateway http://:@172.21.15.142:5000 to Dashboard 2021-03-24T21:53:15.301 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:15 smithi125 conmon[19936]: audit 2021-03-24T21:53:13.574738+0000 mgr.y (mgr.14144) 260 : audit [DBG] from='mon.0 v1:172.21.15.125:0/3961911888' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2021-03-24T21:53:15.301 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:15 smithi125 conmon[19936]: audit 2021-03-24T21:53:13.600108+0000 mgr.y (mgr.14144) 261 : audit [DBG] from='mon.0 v1:172.21.15.125:0/3961911888' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi142"}]: dispatch 2021-03-24T21:53:15.302 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:15 smithi125 conmon[19936]: audit 2021-03-24T21:53:15.302 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:15 smithi125 conmon[19936]: 2021-03-24T21:53:13.615556+0000 mgr.y (mgr.14144) 262 : audit [DBG] from='mon.0 v1:172.21.15.125:0/3961911888' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2021-03-24T21:53:15.302 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:15 smithi125 conmon[19936]: audit 2021-03-24T21:53:13.617089+0000 mgr.y (mgr.14144) 263 : audit [DBG] from='mon.0 v1:172.21.15.125:0/3961911888' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi142:9095"}]: dispatch 2021-03-24T21:53:15.303 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:15 smithi125 conmon[19936]: audit 2021-03-24T21:53:13.629961+0000 mgr.y (mgr.14144) 264 : audit [DBG] from='mon.0 v1:172.21.15.125:0/3961911888' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2021-03-24T21:53:15.303 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:15 smithi125 conmon[19936]: audit 2021-03-24T21:53:13.637266+0000 mgr.y (mgr.14144) 265 : audit [DBG] from='mon.0 v1:172.21.15.125:0/3961911888' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi142:3000"}]: dispatch 2021-03-24T21:53:15.304 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:15 smithi125 conmon[19936]: cluster 2021-03-24T21:53:14.576678+0000 osd.2 (osd.2) 14 : cluster [DBG] 6.f scrub ok 2021-03-24T21:53:15.304 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:15 smithi125 conmon[19936]: cluster 2021-03-24T21:53:14.699830+0000 osd.1 (osd.1) 18 : cluster [DBG] 6.1d scrub ok 2021-03-24T21:53:15.305 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:15 smithi125 conmon[24927]: cluster 2021-03-24T21:53:13.461937 2021-03-24T21:53:15.306 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:15 smithi125 conmon[24927]: +0000 osd.0 (osd.0) 19 : cluster [DBG] 6.10 scrub ok 2021-03-24T21:53:15.306 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:15 smithi125 conmon[24927]: audit 2021-03-24T21:53:15.307 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:15 smithi125 conmon[24927]: 2021-03-24T21:53:13.551925+0000 mgr.y (mgr.14144) 256 : audit [DBG] 2021-03-24T21:53:15.307 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:15 smithi125 conmon[24927]: from='mon.0 v1:172.21.15.125:0/3961911888' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2021-03-24T21:53:15.308 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:15 smithi125 conmon[24927]: audit 2021-03-24T21:53 2021-03-24T21:53:15.308 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:15 smithi125 conmon[24927]: :13.553826+0000 mgr.y (mgr.14144) 257 : audit 2021-03-24T21:53:15.308 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:15 smithi125 conmon[24927]: [DBG] from='mon.0 v1:172.21.15.125:0/3961911888' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi125:9093"}]: dispatch 2021-03-24T21:53:15.309 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:15 smithi125 conmon[24927]: audit 2021-03-24T21:53 2021-03-24T21:53:15.309 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:15 smithi125 conmon[24927]: :13.566379+0000 mgr.y (mgr.14144) 258 : 2021-03-24T21:53:15.310 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:15 smithi125 conmon[24927]: audit [DBG] from='mon.0 v1:172.21.15.125:0/3961911888' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2021-03-24T21:53:15.310 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:15 smithi125 conmon[24927]: cephadm 2021-03-24T 2021-03-24T21:53:15.311 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:15 smithi125 conmon[24927]: 21:53:13.573932+0000 mgr.y (mgr.14144) 2021-03-24T21:53:15.311 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:15 smithi125 conmon[24927]: 259 : cephadm [INF] Adding iSCSI gateway http://:@172.21.15.142:5000 to Dashboard 2021-03-24T21:53:15.311 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:15 smithi125 conmon[24927]: audit 2021-03-24T21:53:15.312 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:15 smithi125 conmon[24927]: 2021-03-24T21:53:13.574738+0000 mgr.y 2021-03-24T21:53:15.312 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:15 smithi125 conmon[24927]: (mgr.14144) 260 : audit [DBG] from='mon.0 v1:172.21.15.125:0/3961911888' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2021-03-24T21:53:15.313 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:15 smithi125 conmon[24927]: audit 2021 2021-03-24T21:53:15.313 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:15 smithi125 conmon[24927]: -03-24T21:53:13.600108+0000 mgr.y (mgr 2021-03-24T21:53:15.314 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:15 smithi125 conmon[24927]: .14144) 261 : audit [DBG] from='mon.0 v1:172.21.15.125:0/3961911888' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi142"}]: dispatch 2021-03-24T21:53:15.314 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:15 smithi125 conmon[24927]: audit 2021-03-24T21:53:15.315 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:15 smithi125 conmon[24927]: 2021-03 2021-03-24T21:53:15.315 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:15 smithi125 conmon[24927]: -24T21:53:13.615556+0000 mgr.y (mgr. 2021-03-24T21:53:15.316 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:15 smithi125 conmon[24927]: 14144) 262 : audit [DBG] from='mon.0 v1:172.21.15.125:0/3961911888' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2021-03-24T21:53:15.316 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:15 smithi125 conmon[24927]: audit 2021-03-24T21:53:15.317 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:15 smithi125 conmon[24927]: 2021-03-24T21:53:13.617089+0000 mgr.y 2021-03-24T21:53:15.317 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:15 smithi125 conmon[24927]: (mgr.14144) 263 : audit [DBG] from='mon.0 v1:172.21.15.125:0/3961911888' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi142:9095"}]: dispatch 2021-03-24T21:53:15.318 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:15 smithi125 conmon[24927]: audit 2021-03-24T21:53:15.318 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:15 smithi125 conmon[24927]: 2021-03-24T21:53:13.629961+0000 mgr.y 2021-03-24T21:53:15.318 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:15 smithi125 conmon[24927]: (mgr.14144) 264 : audit [DBG] from='mon.0 v1:172.21.15.125:0/3961911888' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2021-03-24T21:53:15.319 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:15 smithi125 conmon[24927]: audit 2021-03-24T21:53:15.319 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:15 smithi125 conmon[24927]: 2021-03-24T21:53:13.637266+0000 mgr.y 2021-03-24T21:53:15.320 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:15 smithi125 conmon[24927]: (mgr.14144) 265 : audit [DBG] from='mon.0 v1:172.21.15.125:0/3961911888' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi142:3000"}]: dispatch 2021-03-24T21:53:15.320 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:15 smithi125 conmon[24927]: cluster 2021-03-24T21:53:14.576678+0000 osd.2 (osd.2) 14 : cluster [DBG] 6.f scrub ok 2021-03-24T21:53:15.321 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:15 smithi125 conmon[24927]: cluster 2021-03-24T21:53:14.699830+0000 osd.1 (osd.1) 18 : cluster [DBG] 6.1d scrub ok 2021-03-24T21:53:15.336 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:15 smithi142 conmon[40152]: t=2021-03-24T21:53:15+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2021-03-24T21:53:15.337 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:15 smithi142 conmon[40152]: t=2021-03-24T21:53:15+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2021-03-24T21:53:15.337 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:15 smithi142 conmon[40152]: t=2021-03-24T21:53:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2021-03-24T21:53:15.338 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:15 smithi142 conmon[40152]: t=2021-03-24T21:53:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2021-03-24T21:53:15.339 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:15 smithi142 conmon[20125]: cluster 2021-03-24T21:53:13.461937 2021-03-24T21:53:15.339 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:15 smithi142 conmon[20125]: +0000 osd.0 (osd.0) 19 : cluster [DBG] 6.10 scrub ok 2021-03-24T21:53:15.340 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:15 smithi142 conmon[20125]: audit 2021-03-24T21:53:13.551925+0000 2021-03-24T21:53:15.340 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:15 smithi142 conmon[20125]: mgr.y (mgr.14144) 256 : audit [DBG] from='mon.0 v1:172.21.15.125:0/3961911888' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2021-03-24T21:53:15.341 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:15 smithi142 conmon[20125]: audit 2021-03-24T21:53:13 2021-03-24T21:53:15.341 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:15 smithi142 conmon[20125]: .553826+0000 mgr.y (mgr.14144) 257 : audit 2021-03-24T21:53:15.341 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:15 smithi142 conmon[20125]: [DBG] from='mon.0 v1:172.21.15.125:0/3961911888' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi125:9093"}]: dispatch 2021-03-24T21:53:15.342 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:15 smithi142 conmon[20125]: audit 2021-03-24T21 2021-03-24T21:53:15.342 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:15 smithi142 conmon[20125]: :53:13.566379+0000 mgr.y (mgr.14144) 2021-03-24T21:53:15.343 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:15 smithi142 conmon[20125]: 258 : audit [DBG] from='mon.0 v1:172.21.15.125:0/3961911888' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2021-03-24T21:53:15.343 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:15 smithi142 conmon[20125]: cephadm 2021-03- 2021-03-24T21:53:15.345 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:15 smithi142 conmon[20125]: 24T21:53:13.573932+0000 mgr.y (mgr 2021-03-24T21:53:15.345 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:15 smithi142 conmon[20125]: .14144) 259 : cephadm [INF] Adding iSCSI gateway http://:@172.21.15.142:5000 to Dashboard 2021-03-24T21:53:15.346 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:15 smithi142 conmon[20125]: audit 2021 2021-03-24T21:53:15.346 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:15 smithi142 conmon[20125]: -03-24T21:53:13.574738+0000 mgr.y 2021-03-24T21:53:15.347 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:15 smithi142 conmon[20125]: (mgr.14144) 260 : audit [DBG] from='mon.0 v1:172.21.15.125:0/3961911888' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2021-03-24T21:53:15.347 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:15 smithi142 conmon[20125]: audit 2021 2021-03-24T21:53:15.347 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:15 smithi142 conmon[20125]: -03-24T21:53:13.600108+0000 mgr.y 2021-03-24T21:53:15.348 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:15 smithi142 conmon[20125]: (mgr.14144) 261 : audit [DBG] from='mon.0 v1:172.21.15.125:0/3961911888' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi142"}]: dispatch 2021-03-24T21:53:15.348 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:15 smithi142 conmon[20125]: audit 2021-03-24T21:53:15.349 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:15 smithi142 conmon[20125]: 2021-03-24 2021-03-24T21:53:15.349 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:15 smithi142 conmon[20125]: T21:53:13.615556+0000 mgr.y (mgr. 2021-03-24T21:53:15.350 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:15 smithi142 conmon[20125]: 14144) 262 : audit [DBG] from='mon.0 v1:172.21.15.125:0/3961911888' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2021-03-24T21:53:15.350 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:15 smithi142 conmon[20125]: audit 2021-03-24T21:53:15.350 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:15 smithi142 conmon[20125]: 2021-03-24T21:53:13.617089+0000 2021-03-24T21:53:15.351 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:15 smithi142 conmon[20125]: mgr.y (mgr.14144) 263 : audit [DBG] from='mon.0 v1:172.21.15.125:0/3961911888' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi142:9095"}]: dispatch 2021-03-24T21:53:15.351 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:15 smithi142 conmon[20125]: audit 2021-03-24T21:53:15.352 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:15 smithi142 conmon[20125]: 2021-03-24T21:53:13.629961+0000 2021-03-24T21:53:15.352 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:15 smithi142 conmon[20125]: mgr.y (mgr.14144) 264 : audit [DBG] 2021-03-24T21:53:15.353 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:15 smithi142 conmon[20125]: from='mon.0 v1:172.21.15.125:0/3961911888' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2021-03-24T21:53:15.353 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:15 smithi142 conmon[20125]: audit 2021 2021-03-24T21:53:15.354 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:15 smithi142 conmon[20125]: -03-24T21:53:13.637266+0000 2021-03-24T21:53:15.354 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:15 smithi142 conmon[20125]: mgr.y (mgr.14144) 265 : audit [DBG] from='mon.0 v1:172.21.15.125:0/3961911888' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi142:3000"}]: dispatch 2021-03-24T21:53:15.355 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:15 smithi142 conmon[20125]: 2021-03-24T21:53:15.356 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:15 smithi142 conmon[20125]: cluster 2021-03-24T21:53: 2021-03-24T21:53:15.356 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:15 smithi142 conmon[20125]: 14.576678+0000 osd.2 (osd.2) 14 : 2021-03-24T21:53:15.356 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:15 smithi142 conmon[20125]: cluster [DBG] 6.f scrub ok 2021-03-24T21:53:15.357 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:15 smithi142 conmon[20125]: cluster 2021-03-24T21:53:15.357 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:15 smithi142 conmon[20125]: 2021-03-24T21:53:14. 2021-03-24T21:53:15.357 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:15 smithi142 conmon[20125]: 699830+0000 osd.1 (osd.1) 18 : cluster 2021-03-24T21:53:15.357 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:15 smithi142 conmon[20125]: [DBG] 6.1d scrub ok 2021-03-24T21:53:15.617 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:15 smithi142 conmon[40152]: t=2021-03-24T21:53:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2021-03-24T21:53:15.617 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:15 smithi142 conmon[40152]: t=2021-03-24T21:53:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2021-03-24T21:53:15.617 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:15 smithi142 conmon[40152]: t=2021-03-24T21:53:15+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2021-03-24T21:53:15.983 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:15 smithi142 conmon[40152]: t=2021-03-24T21:53:15+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2021-03-24T21:53:15.984 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:15 smithi142 conmon[40152]: t=2021-03-24T21:53:15+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2021-03-24T21:53:15.984 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:15 smithi142 conmon[40152]: t=2021-03-24T21:53:15+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2021-03-24T21:53:16.149 INFO:journalctl@ceph.alertmanager.a.smithi125.stdout:Mar 24 21:53:15 smithi125 conmon[47044]: level=info ts=2021-03-24T21:53:15.904Z caller=cluster.go:640 component=cluster msg="gossip settled; proceeding" elapsed=10.000871409s 2021-03-24T21:53:16.158 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:16 smithi125 conmon[19936]: cluster 2021-03-24T21:53:14.673370+0000 2021-03-24T21:53:16.159 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:16 smithi125 conmon[19936]: mgr.y (mgr.14144) 266 : cluster [DBG] pgmap v263: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-24T21:53:16.159 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:16 smithi125 conmon[19936]: cluster 2021-03-24T21:53:15.622937+0000 osd.2 (osd.2) 15 : cluster [DBG] 2021-03-24T21:53:16.159 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:16 smithi125 conmon[19936]: 6.14 scrub ok 2021-03-24T21:53:16.160 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:16 smithi125 conmon[19936]: cluster 2021-03-24T21:53 2021-03-24T21:53:16.160 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:16 smithi125 conmon[19936]: :15.719427+0000 osd.1 (osd.1) 19 : cluster [DBG] 4.4 scrub ok 2021-03-24T21:53:16.160 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:16 smithi125 conmon[24927]: cluster 2021-03-24T21:53 2021-03-24T21:53:16.160 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:16 smithi125 conmon[24927]: :14.673370+0000 mgr.y (mgr.14144) 266 : cluster [DBG] pgmap v263: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-24T21:53:16.161 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:16 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:53:16.161 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:16 smithi125 conmon[24927]: -24T21:53: 2021-03-24T21:53:16.161 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:16 smithi125 conmon[24927]: 15.622937+0000 osd.2 (osd. 2021-03-24T21:53:16.161 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:16 smithi125 conmon[24927]: 2) 15 : cluster [DBG] 6.14 scrub ok 2021-03-24T21:53:16.162 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:16 smithi125 conmon[24927]: 2021-03-24T21:53:16.162 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:16 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:53:16.162 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:16 smithi125 conmon[24927]: 03-24T21:53 2021-03-24T21:53:16.162 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:16 smithi125 conmon[24927]: :15.719427+0000 osd.1 (osd 2021-03-24T21:53:16.163 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:16 smithi125 conmon[24927]: .1) 19 : cluster [DBG] 2021-03-24T21:53:16.163 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:16 smithi125 conmon[24927]: 4.4 scrub ok 2021-03-24T21:53:16.291 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:16 smithi142 conmon[20125]: cluster 2021-03-24T21:53:14.673370+0000 mgr.y ( 2021-03-24T21:53:16.292 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:16 smithi142 conmon[20125]: mgr.14144) 266 : cluster [DBG] pgmap v263: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-24T21:53:16.292 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:16 smithi142 conmon[20125]: cluster 2021-03-24T21:53:15.622937+0000 osd.2 (osd.2) 15 : cluster [DBG] 6.14 scrub ok 2021-03-24T21:53:16.292 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:16 smithi142 conmon[20125]: cluster 2021-03-24T21:53:15. 2021-03-24T21:53:16.293 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:16 smithi142 conmon[20125]: 719427+0000 osd.1 (osd.1) 19 : cluster [DBG] 4.4 scrub ok 2021-03-24T21:53:16.293 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:15 smithi142 conmon[40152]: t=2021-03-24T21:53:15+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2021-03-24T21:53:16.293 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:16 smithi142 conmon[40152]: t=2021-03-24T21:53:16+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2021-03-24T21:53:16.293 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:16 smithi142 conmon[40152]: t=2021-03-24T21:53:16+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2021-03-24T21:53:16.446 INFO:teuthology.orchestra.run.smithi125.stdout:201863462938 2021-03-24T21:53:16.446 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph tell osd.7 flush_pg_stats 2021-03-24T21:53:16.575 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:16 smithi142 conmon[40152]: t=2021-03-24T21:53:16+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2021-03-24T21:53:16.769 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:53:16.868 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:16 smithi142 conmon[40152]: t=2021-03-24T21:53:16+0000 lvl=info msg="Created default admin" logger=sqlstore user=admin 2021-03-24T21:53:16.868 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:16 smithi142 conmon[40152]: t=2021-03-24T21:53:16+0000 lvl=info msg="Initializing HTTPServer" logger=server 2021-03-24T21:53:16.868 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:16 smithi142 conmon[40152]: t=2021-03-24T21:53:16+0000 lvl=info msg="Initializing BackendPluginManager" logger=server 2021-03-24T21:53:16.869 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:16 smithi142 conmon[40152]: t=2021-03-24T21:53:16+0000 lvl=info msg="Initializing PluginManager" logger=server 2021-03-24T21:53:16.869 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:16 smithi142 conmon[40152]: t=2021-03-24T21:53:16+0000 lvl=info msg="Starting plugin search" logger=plugins 2021-03-24T21:53:16.869 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:16 smithi142 conmon[40152]: t=2021-03-24T21:53:16+0000 lvl=info msg="Registering plugin" logger=plugins name="Pie Chart" 2021-03-24T21:53:16.869 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:16 smithi142 conmon[40152]: t=2021-03-24T21:53:16+0000 lvl=info msg="Registering plugin" logger=plugins name="Status Panel" 2021-03-24T21:53:16.869 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:16 smithi142 conmon[40152]: t=2021-03-24T21:53:16+0000 lvl=info msg="Initializing HooksService" logger=server 2021-03-24T21:53:16.870 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:16 smithi142 conmon[40152]: t=2021-03-24T21:53:16+0000 lvl=info msg="Initializing OSSLicensingService" logger=server 2021-03-24T21:53:16.870 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:16 smithi142 conmon[40152]: t=2021-03-24T21:53:16+0000 lvl=info msg="Initializing InternalMetricsService" logger=server 2021-03-24T21:53:16.870 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:16 smithi142 conmon[40152]: t=2021-03-24T21:53:16+0000 lvl=info msg="Initializing RemoteCache" logger=server 2021-03-24T21:53:16.870 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:16 smithi142 conmon[40152]: t=2021-03-24T21:53:16+0000 lvl=info msg="Initializing RenderingService" logger=server 2021-03-24T21:53:16.871 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:16 smithi142 conmon[40152]: t=2021-03-24T21:53:16+0000 lvl=info msg="Initializing AlertEngine" logger=server 2021-03-24T21:53:16.871 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:16 smithi142 conmon[40152]: t=2021-03-24T21:53:16+0000 lvl=info msg="Initializing QuotaService" logger=server 2021-03-24T21:53:16.871 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:16 smithi142 conmon[40152]: t=2021-03-24T21:53:16+0000 lvl=info msg="Initializing ServerLockService" logger=server 2021-03-24T21:53:16.871 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:16 smithi142 conmon[40152]: t=2021-03-24T21:53:16+0000 lvl=info msg="Initializing UserAuthTokenService" logger=server 2021-03-24T21:53:16.872 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:16 smithi142 conmon[40152]: t=2021-03-24T21:53:16+0000 lvl=info msg="Initializing DatasourceCacheService" logger=server 2021-03-24T21:53:16.872 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:16 smithi142 conmon[40152]: t=2021-03-24T21:53:16+0000 lvl=info msg="Initializing LoginService" logger=server 2021-03-24T21:53:16.872 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:16 smithi142 conmon[40152]: t=2021-03-24T21:53:16+0000 lvl=info msg="Initializing SearchService" logger=server 2021-03-24T21:53:16.872 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:16 smithi142 conmon[40152]: t=2021-03-24T21:53:16+0000 lvl=info msg="Initializing TracingService" logger=server 2021-03-24T21:53:16.872 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:16 smithi142 conmon[40152]: t=2021-03-24T21:53:16+0000 lvl=info msg="Initializing UsageStatsService" logger=server 2021-03-24T21:53:16.873 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:16 smithi142 conmon[40152]: t=2021-03-24T21:53:16+0000 lvl=info msg="Initializing CleanUpService" logger=server 2021-03-24T21:53:16.873 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:16 smithi142 conmon[40152]: t=2021-03-24T21:53:16+0000 lvl=info msg="Initializing NotificationService" logger=server 2021-03-24T21:53:16.873 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:16 smithi142 conmon[40152]: t=2021-03-24T21:53:16+0000 lvl=info msg="Initializing provisioningServiceImpl" logger=server 2021-03-24T21:53:16.873 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:16 smithi142 conmon[40152]: t=2021-03-24T21:53:16+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2021-03-24T21:53:16.873 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:16 smithi142 conmon[40152]: t=2021-03-24T21:53:16+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 2021-03-24T21:53:16.874 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:16 smithi142 conmon[40152]: t=2021-03-24T21:53:16+0000 lvl=info msg="Backend rendering via phantomJS" logger=rendering renderer=phantomJS 2021-03-24T21:53:16.874 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:16 smithi142 conmon[40152]: t=2021-03-24T21:53:16+0000 lvl=warn msg="phantomJS is deprecated and will be removed in a future release. You should consider migrating from phantomJS to grafana-image-renderer plugin. Read more at https://grafana.com/docs/grafana/latest/administration/image_rendering/" logger=rendering renderer=phantomJS 2021-03-24T21:53:16.874 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:16 smithi142 conmon[40152]: t=2021-03-24T21:53:16+0000 lvl=info msg="Initializing Stream Manager" 2021-03-24T21:53:16.874 INFO:journalctl@ceph.grafana.a.smithi142.stdout:Mar 24 21:53:16 smithi142 conmon[40152]: t=2021-03-24T21:53:16+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2021-03-24T21:53:17.240 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:17 smithi125 conmon[24927]: cluster 2021-03-24T21:53:15.485063+0000 osd.0 (osd.0) 20 : cluster [DBG] 4.6 scrub ok 2021-03-24T21:53:17.240 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:17 smithi125 conmon[24927]: cluster 2021-03-24T21:53:15.868297+0000 osd.7 (osd.7) 17 : cluster [DBG] 3.13 scrub ok 2021-03-24T21:53:17.241 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:17 smithi125 conmon[24927]: cluster 2021-03-24T21:53:15.901131+0000 osd.5 (osd.5) 20 : cluster [DBG] 4.15 scrub ok 2021-03-24T21:53:17.241 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:17 smithi125 conmon[24927]: cluster 2021-03-24T21:53:15.925599+0000 osd.6 (osd.6) 9 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:53:17.241 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:17 smithi125 conmon[24927]: cluster 2021-03-24T21:53:17.241 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:17 smithi125 conmon[24927]: 2021-03-24T21:53:15.925760+0000 osd.6 (osd.6) 10 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:53:17.242 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:17 smithi125 conmon[24927]: cluster 2021-03-24T21:53:16.143770+0000 mon.a (mon.0) 589 2021-03-24T21:53:17.242 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:17 smithi125 conmon[24927]: : cluster [DBG] osdmap e89: 8 total, 8 up, 8 in 2021-03-24T21:53:17.243 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:17 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:53:17.243 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:17 smithi125 conmon[19936]: 53:15.485063+0000 osd.0 (osd.0) 20 : cluster [DBG] 4.6 scrub ok 2021-03-24T21:53:17.243 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:17 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T21:53:17.243 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:17 smithi125 conmon[19936]: T21:53:15.868297 2021-03-24T21:53:17.244 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:17 smithi125 conmon[19936]: +0000 osd.7 (osd.7) 17 : 2021-03-24T21:53:17.244 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:17 smithi125 conmon[19936]: cluster [DBG] 3.13 scrub ok 2021-03-24T21:53:17.244 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:17 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:53:17.244 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:17 smithi125 conmon[19936]: 24T21:53:15.901131 2021-03-24T21:53:17.244 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:17 smithi125 conmon[19936]: +0000 osd.5 (osd.5) 20 : cluster 2021-03-24T21:53:17.245 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:17 smithi125 conmon[19936]: [DBG] 4.15 scrub ok 2021-03-24T21:53:17.245 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:17 smithi125 conmon[19936]: cluster 2021-03-24T21:53:17.245 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:17 smithi125 conmon[19936]: 2021-03-24 2021-03-24T21:53:17.245 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:17 smithi125 conmon[19936]: T21:53:15.925599 2021-03-24T21:53:17.246 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:17 smithi125 conmon[19936]: +0000 osd.6 (osd.6 2021-03-24T21:53:17.246 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:17 smithi125 conmon[19936]: ) 9 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:53:17.246 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:17 smithi125 conmon[19936]: 2021-03-24T21:53:17.247 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:17 smithi125 conmon[19936]: cluster 2021-03-24T21:53:17.247 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:17 smithi125 conmon[19936]: 2021-03-24T21:53: 2021-03-24T21:53:17.247 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:17 smithi125 conmon[19936]: 15.925760+0000 osd.6 (osd. 2021-03-24T21:53:17.247 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:17 smithi125 conmon[19936]: 6) 10 : cluster 2021-03-24T21:53:17.247 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:17 smithi125 conmon[19936]: [DBG] purged_snaps scrub ok 2021-03-24T21:53:17.248 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:17 smithi125 conmon[19936]: cluster 2021-03-24T21:53:17.248 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:17 smithi125 conmon[19936]: 2021-03-24T21:53 2021-03-24T21:53:17.248 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:17 smithi125 conmon[19936]: :16.143770+0000 2021-03-24T21:53:17.248 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:17 smithi125 conmon[19936]: mon.a (mon. 2021-03-24T21:53:17.249 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:17 smithi125 conmon[19936]: 0) 589 : cluster 2021-03-24T21:53:17.249 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:17 smithi125 conmon[19936]: [DBG] osdmap e89: 8 total, 8 up, 8 in 2021-03-24T21:53:17.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:17 smithi142 conmon[20125]: cluster 2021-03-24T21:53:15.485063+0000 osd.0 (osd.0) 20 : cluster 2021-03-24T21:53:17.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:17 smithi142 conmon[20125]: [DBG] 4.6 scrub ok 2021-03-24T21:53:17.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:17 smithi142 conmon[20125]: cluster 2021-03-24T21:53:15.868297+0000 osd.7 (osd.7) 17 : cluster [DBG] 2021-03-24T21:53:17.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:17 smithi142 conmon[20125]: 3.13 scrub ok 2021-03-24T21:53:17.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:17 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:53:17.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:17 smithi142 conmon[20125]: T21:53:15.901131+0000 osd.5 (osd.5) 2021-03-24T21:53:17.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:17 smithi142 conmon[20125]: 20 : cluster [DBG] 4.15 scrub ok 2021-03-24T21:53:17.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:17 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:53:17.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:17 smithi142 conmon[20125]: 21:53:15.925599+0000 osd.6 (osd 2021-03-24T21:53:17.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:17 smithi142 conmon[20125]: .6) 9 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:53:17.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:17 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:53:17.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:17 smithi142 conmon[20125]: 24T21:53:15.925760+0000 osd.6 2021-03-24T21:53:17.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:17 smithi142 conmon[20125]: (osd.6) 10 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:53:17.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:17 smithi142 conmon[20125]: 2021-03-24T21:53:17.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:17 smithi142 conmon[20125]: cluster 2021-03-24T21:53:17.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:17 smithi142 conmon[20125]: 2021-03-24T21:53:16.143770+0000 mon.a ( 2021-03-24T21:53:17.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:17 smithi142 conmon[20125]: mon.0) 589 : cluster [DBG] osdmap e89: 8 total, 8 up, 8 in 2021-03-24T21:53:18.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:18 smithi142 conmon[20125]: cluster 2021-03-24T21:53: 2021-03-24T21:53:18.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:18 smithi142 conmon[20125]: 16.673945+0000 mgr.y (mgr.14144) 267 : cluster [DBG] pgmap v265: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-03-24T21:53:18.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:18 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:53:18.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:18 smithi142 conmon[20125]: -24T21:53:17.246362+0000 osd.6 (osd.6) 11 : cluster [DBG] 4.1f scrub ok 2021-03-24T21:53:18.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:18 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:53:18.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:18 smithi142 conmon[20125]: 03-24T21:53:17.716446+0000 osd.3 (osd 2021-03-24T21:53:18.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:18 smithi142 conmon[20125]: .3) 22 : cluster [DBG] 4.0 scrub ok 2021-03-24T21:53:18.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:18 smithi125 conmon[24927]: cluster 2021-03-24T21:53:18.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:18 smithi125 conmon[24927]: 2021-03-24T21 2021-03-24T21:53:18.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:18 smithi125 conmon[24927]: :53:16.673945+0000 mgr.y (mgr.14144) 267 : cluster [DBG] 2021-03-24T21:53:18.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:18 smithi125 conmon[24927]: pgmap v265: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-03-24T21:53:18.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:18 smithi125 conmon[24927]: cluster 2021-03-24T21:53 2021-03-24T21:53:18.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:18 smithi125 conmon[24927]: :17.246362+0000 osd.6 (osd.6 2021-03-24T21:53:18.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:18 smithi125 conmon[24927]: ) 11 : cluster [DBG] 4.1f scrub ok 2021-03-24T21:53:18.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:18 smithi125 conmon[24927]: 2021-03-24T21:53:18.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:18 smithi125 conmon[24927]: cluster 2021-03-24T21:53:18.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:18 smithi125 conmon[24927]: 2021-03-24T21: 2021-03-24T21:53:18.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:18 smithi125 conmon[24927]: 53:17.716446+0000 osd.3 (osd. 2021-03-24T21:53:18.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:18 smithi125 conmon[24927]: 3) 22 : cluster [DBG] 4.0 scrub ok 2021-03-24T21:53:18.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:18 smithi125 conmon[19936]: cluster 2021-03-24T21:53 2021-03-24T21:53:18.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:18 smithi125 conmon[19936]: :16.673945+0000 mgr.y (mgr.14144) 267 : cluster [DBG] pgmap v265: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-03-24T21:53:18.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:18 smithi125 conmon[19936]: cluster 2021-03-24T21:53:17.246362+0000 2021-03-24T21:53:18.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:18 smithi125 conmon[19936]: osd.6 (osd.6) 11 : cluster 2021-03-24T21:53:18.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:18 smithi125 conmon[19936]: [DBG] 4.1f scrub ok 2021-03-24T21:53:18.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:18 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:53:18.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:18 smithi125 conmon[19936]: -24T21:53:17.716446+0000 2021-03-24T21:53:18.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:18 smithi125 conmon[19936]: osd.3 (osd.3) 22 : cluster 2021-03-24T21:53:18.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:18 smithi125 conmon[19936]: [DBG] 4.0 scrub ok 2021-03-24T21:53:19.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:19 smithi125 conmon[19936]: audit 2021-03-24T21:53:18.328925 2021-03-24T21:53:19.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:19 smithi125 conmon[19936]: +0000 mon.a (mon.0) 590 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:53:19.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:19 smithi125 conmon[19936]: cluster 2021-03-24T21:53:18 2021-03-24T21:53:19.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:19 smithi125 conmon[19936]: .557112+0000 osd.2 (osd.2) 16 : cluster [DBG] 5.8 scrub ok 2021-03-24T21:53:19.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:19 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:53:19.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:19 smithi125 conmon[19936]: 03-24T21:53:18.706735+0000 osd.3 (osd.3) 23 : cluster [DBG] 3.1e scrub ok 2021-03-24T21:53:19.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:19 smithi125 conmon[19936]: audit 2021-03-24T21 2021-03-24T21:53:19.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:19 smithi125 conmon[19936]: :53:19.028655+0000 mon.a (mon.0) 2021-03-24T21:53:19.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:19 smithi125 conmon[19936]: 591 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:53:19.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:19 smithi125 conmon[19936]: audit 2021-03-24T21:53:19.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:19 smithi125 conmon[19936]: 2021-03-24T21 2021-03-24T21:53:19.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:19 smithi125 conmon[19936]: :53:19.034116+0000 mon.a (mon.0) 592 : audit 2021-03-24T21:53:19.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:19 smithi125 conmon[19936]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:53:19.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:19 smithi125 conmon[19936]: audit 2021 2021-03-24T21:53:19.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:19 smithi125 conmon[19936]: -03-24T21:53:19.038554 2021-03-24T21:53:19.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:19 smithi125 conmon[19936]: +0000 mon.a (mon.0) 593 : audit 2021-03-24T21:53:19.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:19 smithi125 conmon[19936]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:53:19.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:19 smithi125 conmon[24927]: audit 2021-03-24T21: 2021-03-24T21:53:19.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:19 smithi125 conmon[24927]: 53:18.328925+0000 mon.a (mon.0) 590 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:53:19.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:19 smithi125 conmon[24927]: cluster 2021-03-24T21:53:19.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:19 smithi125 conmon[24927]: 2021-03-24T21: 2021-03-24T21:53:19.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:19 smithi125 conmon[24927]: 53:18.557112+0000 osd.2 (osd.2) 16 : cluster 2021-03-24T21:53:19.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:19 smithi125 conmon[24927]: [DBG] 5.8 scrub ok 2021-03-24T21:53:19.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:19 smithi125 conmon[24927]: cluster 2021-03-24T21:53:18 2021-03-24T21:53:19.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:19 smithi125 conmon[24927]: .706735+0000 osd.3 (osd.3) 23 : cluster 2021-03-24T21:53:19.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:19 smithi125 conmon[24927]: [DBG] 3.1e scrub ok 2021-03-24T21:53:19.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:19 smithi125 conmon[24927]: audit 2021-03-24T21:53 2021-03-24T21:53:19.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:19 smithi125 conmon[24927]: :19.028655+0000 mon.a (mon. 2021-03-24T21:53:19.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:19 smithi125 conmon[24927]: 0) 591 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:53:19.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:19 smithi125 conmon[24927]: 2021-03-24T21:53:19.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:19 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T21:53:19.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:19 smithi125 conmon[24927]: 24T21:53:19.034116 2021-03-24T21:53:19.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:19 smithi125 conmon[24927]: +0000 mon.a (mon.0) 2021-03-24T21:53:19.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:19 smithi125 conmon[24927]: 592 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:53:19.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:19 smithi125 conmon[24927]: audit 2021-03-24T21:53:19.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:19 smithi125 conmon[24927]: 2021-03-24T21:53: 2021-03-24T21:53:19.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:19 smithi125 conmon[24927]: 19.038554+0000 mon.a (mon.0) 593 2021-03-24T21:53:19.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:19 smithi125 conmon[24927]: : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:53:19.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:19 smithi142 conmon[20125]: audit 2021-03-24T21:53:18.328925+0000 2021-03-24T21:53:19.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:19 smithi142 conmon[20125]: mon.a (mon.0) 590 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:53:19.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:19 smithi142 conmon[20125]: cluster 2021-03-24T21:53:18.557112+0000 osd.2 (osd.2) 2021-03-24T21:53:19.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:19 smithi142 conmon[20125]: 16 : cluster [DBG] 5.8 scrub ok 2021-03-24T21:53:19.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:19 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:53:19.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:19 smithi142 conmon[20125]: 03-24T21:53:18.706735+0000 osd.3 (osd. 2021-03-24T21:53:19.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:19 smithi142 conmon[20125]: 3) 23 : cluster [DBG] 3.1e scrub ok 2021-03-24T21:53:19.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:19 smithi142 conmon[20125]: audit 2021-03 2021-03-24T21:53:19.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:19 smithi142 conmon[20125]: -24T21:53:19.028655+0000 2021-03-24T21:53:19.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:19 smithi142 conmon[20125]: mon.a (mon.0) 591 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:53:19.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:19 smithi142 conmon[20125]: 2021-03-24T21:53:19.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:19 smithi142 conmon[20125]: audit 2021-03-24T21:53:19.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:19 smithi142 conmon[20125]: 2021-03-24T21:53:19.034116+0000 mon.a 2021-03-24T21:53:19.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:19 smithi142 conmon[20125]: (mon.0) 592 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:53:19.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:19 smithi142 conmon[20125]: audit 2021 2021-03-24T21:53:19.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:19 smithi142 conmon[20125]: -03-24T21:53:19.038554 2021-03-24T21:53:19.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:19 smithi142 conmon[20125]: +0000 mon.a (mon.0) 593 : audit [DBG] 2021-03-24T21:53:19.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:19 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:53:19.806 INFO:teuthology.orchestra.run.smithi125.stdout:236223201302 2021-03-24T21:53:19.807 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd last-stat-seq osd.0 2021-03-24T21:53:20.110 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:53:20.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:20 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:53:20.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:20 smithi125 conmon[24927]: :53:18.674714+0000 mgr.y (mgr.14144) 268 : cluster [DBG] pgmap v266: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-03-24T21:53:20.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:20 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:53:20.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:20 smithi125 conmon[24927]: :53:18.753022+0000 osd.1 (osd.1) 20 : cluster 2021-03-24T21:53:20.618 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:20 smithi125 conmon[24927]: [DBG] 3.0 scrub ok 2021-03-24T21:53:20.618 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:20 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:53:20.618 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:20 smithi125 conmon[24927]: -03-24T21:53:19.717231+0000 osd.1 (osd.1) 21 : 2021-03-24T21:53:20.618 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:20 smithi125 conmon[24927]: cluster [DBG] 4.7 deep-scrub ok 2021-03-24T21:53:20.619 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:20 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:53:20.619 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:20 smithi125 conmon[24927]: 21:53:19.812808+0000 osd.5 (osd.5) 2021-03-24T21:53:20.619 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:20 smithi125 conmon[24927]: 21 : cluster [DBG] 5.16 scrub ok 2021-03-24T21:53:20.619 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:20 smithi125 conmon[24927]: cluster 2021-03-24T21:53: 2021-03-24T21:53:20.619 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:20 smithi125 conmon[24927]: 20.163044+0000 osd.4 (osd.4) 15 : 2021-03-24T21:53:20.620 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:20 smithi125 conmon[24927]: cluster [DBG] 4.c scrub ok 2021-03-24T21:53:20.620 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:20 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:53:20.620 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:20 smithi125 conmon[24927]: 24T21:53:20.278996+0000 osd.6 (osd.6) 12 : 2021-03-24T21:53:20.620 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:20 smithi125 conmon[24927]: cluster [DBG] 4.5 scrub ok 2021-03-24T21:53:20.621 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:20 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:53:20.621 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:20 smithi125 conmon[19936]: 53:18.674714+0000 mgr.y (mgr.14144) 268 : cluster [DBG] pgmap v266: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-03-24T21:53:20.621 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:20 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:53:20.621 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:20 smithi125 conmon[19936]: -03-24T21:53:18.753022 2021-03-24T21:53:20.621 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:20 smithi125 conmon[19936]: +0000 osd.1 (osd.1) 20 : cluster [DBG] 2021-03-24T21:53:20.622 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:20 smithi125 conmon[19936]: 3.0 scrub ok 2021-03-24T21:53:20.622 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:20 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T21:53:20.622 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:20 smithi125 conmon[19936]: T21:53:19.717231+0000 2021-03-24T21:53:20.623 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:20 smithi125 conmon[19936]: osd.1 (osd.1) 21 : 2021-03-24T21:53:20.623 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:20 smithi125 conmon[19936]: cluster [DBG] 4.7 deep-scrub ok 2021-03-24T21:53:20.623 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:20 smithi125 conmon[19936]: cluster 2021-03-24T21:53:20.623 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:20 smithi125 conmon[19936]: 2021-03-24 2021-03-24T21:53:20.623 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:20 smithi125 conmon[19936]: T21:53:19.812808+0000 osd.5 (osd 2021-03-24T21:53:20.624 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:20 smithi125 conmon[19936]: .5) 21 : cluster [DBG] 2021-03-24T21:53:20.624 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:20 smithi125 conmon[19936]: 5.16 scrub ok 2021-03-24T21:53:20.624 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:20 smithi125 conmon[19936]: cluster 2021-03-24T21:53 2021-03-24T21:53:20.624 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:20 smithi125 conmon[19936]: :20.163044+0000 osd.4 (osd. 2021-03-24T21:53:20.624 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:20 smithi125 conmon[19936]: 4) 15 : cluster [DBG] 4.c scrub ok 2021-03-24T21:53:20.625 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:20 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:53:20.625 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:20 smithi125 conmon[19936]: 03-24T21:53:20.278996+0000 2021-03-24T21:53:20.626 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:20 smithi125 conmon[19936]: osd.6 (osd.6) 12 : cluster 2021-03-24T21:53:20.626 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:20 smithi125 conmon[19936]: [DBG] 4.5 scrub ok 2021-03-24T21:53:20.626 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:20 smithi142 conmon[20125]: cluster 2021-03-24T21:53:18.674714+0000 2021-03-24T21:53:20.627 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:20 smithi142 conmon[20125]: mgr.y (mgr.14144) 268 : cluster [DBG] pgmap v266: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-03-24T21:53:20.627 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:20 smithi142 conmon[20125]: cluster 2021-03-24T21:53:18.753022+0000 osd.1 2021-03-24T21:53:20.627 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:20 smithi142 conmon[20125]: (osd.1) 20 : cluster [DBG] 3.0 scrub ok 2021-03-24T21:53:20.627 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:20 smithi142 conmon[20125]: cluster 2021-03-24T21:53:20.628 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:20 smithi142 conmon[20125]: 2021-03-24T21:53 2021-03-24T21:53:20.628 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:20 smithi142 conmon[20125]: :19.717231+0000 osd.1 (osd.1) 21 : cluster 2021-03-24T21:53:20.628 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:20 smithi142 conmon[20125]: [DBG] 4.7 deep-scrub ok 2021-03-24T21:53:20.628 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:20 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:53:20.629 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:20 smithi142 conmon[20125]: T21:53:19.812808+0000 osd.5 (osd.5) 21 2021-03-24T21:53:20.629 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:20 smithi142 conmon[20125]: : cluster [DBG] 5.16 scrub ok 2021-03-24T21:53:20.629 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:20 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:53:20.629 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:20 smithi142 conmon[20125]: 24T21:53:20.163044+0000 osd.4 (osd.4) 2021-03-24T21:53:20.630 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:20 smithi142 conmon[20125]: 15 : cluster [DBG] 4.c scrub ok 2021-03-24T21:53:20.630 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:20 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:53:20.630 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:20 smithi142 conmon[20125]: 24T21:53:20.278996+0000 osd.6 (osd.6 2021-03-24T21:53:20.630 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:20 smithi142 conmon[20125]: ) 12 : cluster [DBG] 4.5 scrub ok 2021-03-24T21:53:21.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:21 smithi142 conmon[20125]: cluster 2021-03-24T21:53:19.926627+0000 osd.7 (osd. 2021-03-24T21:53:21.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:21 smithi142 conmon[20125]: 7) 18 : cluster [DBG] 3.f deep-scrub ok 2021-03-24T21:53:21.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:21 smithi142 conmon[20125]: cluster 2021-03-24T21:53:20.736421+0000 osd.3 (osd.3) 24 : cluster [DBG] 3.2 scrub ok 2021-03-24T21:53:21.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:21 smithi142 conmon[20125]: cluster 2021-03-24T21:53: 2021-03-24T21:53:21.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:21 smithi142 conmon[20125]: 21.207689+0000 osd.4 (osd.4) 16 : cluster [DBG] 4.13 scrub ok 2021-03-24T21:53:21.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:21 smithi125 conmon[24927]: cluster 2021-03-24T21:53:19. 2021-03-24T21:53:21.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:21 smithi125 conmon[24927]: 926627+0000 osd.7 (osd.7) 18 : cluster [DBG] 3.f deep-scrub ok 2021-03-24T21:53:21.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:21 smithi125 conmon[24927]: cluster 2021-03-24T21:53:20.736421 2021-03-24T21:53:21.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:21 smithi125 conmon[24927]: +0000 osd.3 (osd.3) 24 : cluster [DBG] 3.2 scrub ok 2021-03-24T21:53:21.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:21 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:53:21.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:21 smithi125 conmon[24927]: T21:53:21.207689+0000 osd.4 (osd.4) 16 : cluster [DBG] 4.13 scrub ok 2021-03-24T21:53:21.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:21 smithi125 conmon[19936]: cluster 2021-03-24T21:53:19 2021-03-24T21:53:21.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:21 smithi125 conmon[19936]: .926627+0000 osd.7 (osd.7) 18 : cluster [DBG] 3.f deep-scrub ok 2021-03-24T21:53:21.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:21 smithi125 conmon[19936]: cluster 2021-03-24T21:53 2021-03-24T21:53:21.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:21 smithi125 conmon[19936]: :20.736421+0000 osd.3 (osd.3) 24 : cluster [DBG] 3.2 scrub ok 2021-03-24T21:53:21.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:21 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:53:21.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:21 smithi125 conmon[19936]: 03-24T21:53:21. 2021-03-24T21:53:21.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:21 smithi125 conmon[19936]: 207689+0000 osd.4 (osd. 2021-03-24T21:53:21.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:21 smithi125 conmon[19936]: 4) 16 : cluster [DBG] 2021-03-24T21:53:21.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:21 smithi125 conmon[19936]: 4.13 scrub ok 2021-03-24T21:53:22.105 INFO:teuthology.orchestra.run.smithi125.stdout:34359738420 2021-03-24T21:53:22.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:22 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:53:22.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:22 smithi125 conmon[24927]: 53:20.675729+0000 mgr.y (mgr.14144) 269 : cluster [DBG] pgmap v267: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-24T21:53:22.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:22 smithi125 conmon[24927]: cluster 2021-03-24T21:53:20.835984+0000 2021-03-24T21:53:22.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:22 smithi125 conmon[24927]: osd.5 (osd.5) 22 : cluster [DBG] 5.7 scrub ok 2021-03-24T21:53:22.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:22 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:53:22.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:22 smithi125 conmon[24927]: -03-24T21:53:21.533867+0000 osd.2 (osd 2021-03-24T21:53:22.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:22 smithi125 conmon[24927]: .2) 17 : cluster [DBG] 5.6 scrub ok 2021-03-24T21:53:22.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:22 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:53:22.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:22 smithi125 conmon[24927]: -03-24T21:53:21.584992+0000 osd.0 2021-03-24T21:53:22.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:22 smithi125 conmon[24927]: (osd.0) 21 : cluster [DBG] 4.1e deep-scrub ok 2021-03-24T21:53:22.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:22 smithi125 conmon[24927]: audit 2021 2021-03-24T21:53:22.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:22 smithi125 conmon[24927]: -03-24T21:53:22.106054+0000 mon.a (mon 2021-03-24T21:53:22.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:22 smithi125 conmon[24927]: .0) 594 : audit [DBG] from='client.? v1:172.21.15.125:0/3263262115' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2021-03-24T21:53:22.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:22 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:53:22.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:22 smithi125 conmon[24927]: 24T21:53:22.203848+0000 osd.6 (osd 2021-03-24T21:53:22.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:22 smithi125 conmon[24927]: .6) 13 : cluster [DBG] 3.a deep-scrub ok 2021-03-24T21:53:22.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:22 smithi125 conmon[19936]: cluster 2021-03-24T21:53: 2021-03-24T21:53:22.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:22 smithi125 conmon[19936]: 20.675729+0000 mgr.y (mgr.14144) 269 : cluster [DBG] pgmap v267: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-24T21:53:22.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:22 smithi125 conmon[19936]: cluster 2021-03-24T21:53:22.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:22 smithi125 conmon[19936]: 2021-03-24T21:53:20.835984 2021-03-24T21:53:22.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:22 smithi125 conmon[19936]: +0000 osd.5 (osd.5) 2021-03-24T21:53:22.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:22 smithi125 conmon[19936]: 22 : cluster [DBG] 5.7 scrub ok 2021-03-24T21:53:22.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:22 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:53:22.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:22 smithi125 conmon[19936]: 03-24T21:53:21.533867+0000 osd.2 ( 2021-03-24T21:53:22.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:22 smithi125 conmon[19936]: osd.2) 17 : cluster [DBG] 5.6 scrub ok 2021-03-24T21:53:22.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:22 smithi125 conmon[19936]: cluster 2021-03-24T21:53:22.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:22 smithi125 conmon[19936]: 2021-03-24T21:53 2021-03-24T21:53:22.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:22 smithi125 conmon[19936]: :21.584992+0000 osd.0 (osd 2021-03-24T21:53:22.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:22 smithi125 conmon[19936]: .0) 21 : cluster [DBG] 2021-03-24T21:53:22.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:22 smithi125 conmon[19936]: 4.1e deep-scrub ok 2021-03-24T21:53:22.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:22 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:53:22.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:22 smithi125 conmon[19936]: 03-24T21:53: 2021-03-24T21:53:22.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:22 smithi125 conmon[19936]: 22.106054+0000 mon.a (mon. 2021-03-24T21:53:22.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:22 smithi125 conmon[19936]: 0) 594 : audit [DBG] 2021-03-24T21:53:22.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:22 smithi125 conmon[19936]: from='client.? v1:172.21.15.125:0/3263262115' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2021-03-24T21:53:22.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:22 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:53:22.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:22 smithi125 conmon[19936]: 03-24T21:53:22 2021-03-24T21:53:22.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:22 smithi125 conmon[19936]: .203848+0000 osd.6 (osd. 2021-03-24T21:53:22.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:22 smithi125 conmon[19936]: 6) 13 : cluster [DBG] 3.a deep-scrub ok 2021-03-24T21:53:22.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:22 smithi125 conmon[19936]: 2021-03-24T21:53:22.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:22 smithi142 conmon[20125]: cluster 2021-03-24T21:53:20.675729+0000 2021-03-24T21:53:22.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:22 smithi142 conmon[20125]: mgr.y (mgr.14144) 269 : cluster [DBG] pgmap v267: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-24T21:53:22.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:22 smithi142 conmon[20125]: cluster 2021-03-24T21:53:20.835984+0000 osd.5 (osd.5) 22 : cluster [DBG] 2021-03-24T21:53:22.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:22 smithi142 conmon[20125]: 5.7 scrub ok 2021-03-24T21:53:22.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:22 smithi142 conmon[20125]: cluster 2021-03-24T21:53:21.533867+0000 osd.2 (osd 2021-03-24T21:53:22.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:22 smithi142 conmon[20125]: .2) 17 : cluster [DBG] 5.6 scrub ok 2021-03-24T21:53:22.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:22 smithi142 conmon[20125]: cluster 2021-03-24T21:53 2021-03-24T21:53:22.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:22 smithi142 conmon[20125]: :21.584992+0000 osd.0 (osd.0) 21 2021-03-24T21:53:22.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:22 smithi142 conmon[20125]: : cluster [DBG] 4.1e deep-scrub ok 2021-03-24T21:53:22.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:22 smithi142 conmon[20125]: audit 2021-03-24T21:53:22.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:22 smithi142 conmon[20125]: 2021-03-24 2021-03-24T21:53:22.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:22 smithi142 conmon[20125]: T21:53:22.106054+0000 mon.a (mon.0) 594 : audit 2021-03-24T21:53:22.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:22 smithi142 conmon[20125]: [DBG] from='client.? v1:172.21.15.125:0/3263262115' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2021-03-24T21:53:22.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:22 smithi142 conmon[20125]: cluster 2021-03-24T21:53:22.203848+0000 2021-03-24T21:53:22.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:22 smithi142 conmon[20125]: osd.6 (osd.6) 13 : cluster [DBG] 3.a deep-scrub ok 2021-03-24T21:53:23.047 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738415 got 34359738420 for osd.0 2021-03-24T21:53:23.051 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd last-stat-seq osd.1 2021-03-24T21:53:23.369 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:53:23.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:23 smithi142 conmon[20125]: cluster 2021-03-24T21:53:21.832905+0000 osd.5 (osd.5) 23 : cluster 2021-03-24T21:53:23.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:23 smithi142 conmon[20125]: [DBG] 3.16 scrub ok 2021-03-24T21:53:23.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:23 smithi142 conmon[20125]: cluster 2021-03-24T21:53:22.724822+0000 osd.3 (osd.3) 25 : cluster 2021-03-24T21:53:23.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:23 smithi142 conmon[20125]: [DBG] 6.b deep-scrub ok 2021-03-24T21:53:23.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:23 smithi142 conmon[20125]: audit 2021-03-24T21:53:23.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:23 smithi142 conmon[20125]: 2021-03-24T21:53:22.933324+0000 mon.a ( 2021-03-24T21:53:23.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:23 smithi142 conmon[20125]: mon.0) 595 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:53:23.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:23 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:53:23.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:23 smithi142 conmon[20125]: 03-24T21:53:22.940975+0000 2021-03-24T21:53:23.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:23 smithi142 conmon[20125]: mon.a (mon.0) 596 : audit [INF] 2021-03-24T21:53:23.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:23 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:53:23.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:23 smithi142 conmon[20125]: cluster 2021-03-24T21:53:23.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:23 smithi142 conmon[20125]: 2021-03-24T21:53:23.246311+0000 osd.6 2021-03-24T21:53:23.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:23 smithi142 conmon[20125]: (osd.6) 14 : cluster [DBG] 5.2 scrub ok 2021-03-24T21:53:23.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:23 smithi142 conmon[20125]: 2021-03-24T21:53:23.640 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:23 smithi125 conmon[24927]: cluster 2021-03-24T21:53: 2021-03-24T21:53:23.640 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:23 smithi125 conmon[24927]: 21.832905+0000 osd.5 (osd.5) 23 : cluster [DBG] 3.16 scrub ok 2021-03-24T21:53:23.640 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:23 smithi125 conmon[24927]: cluster 2021-03-24T21:53:23.641 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:23 smithi125 conmon[24927]: 2021-03-24T21:53:22. 2021-03-24T21:53:23.641 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:23 smithi125 conmon[24927]: 724822+0000 osd.3 (osd.3) 25 : cluster [DBG] 2021-03-24T21:53:23.641 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:23 smithi125 conmon[24927]: 6.b deep-scrub ok 2021-03-24T21:53:23.641 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:23 smithi125 conmon[24927]: audit 2021-03-24T21:53:23.642 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:23 smithi125 conmon[24927]: 2021-03-24T21:53:22.933324+0000 mon.a (mon 2021-03-24T21:53:23.642 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:23 smithi125 conmon[24927]: .0) 595 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:53:23.642 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:23 smithi125 conmon[24927]: audit 2021-03-24T21:53:23.642 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:23 smithi125 conmon[24927]: 2021-03-24T21:53:22.940975+0000 mon.a (mon.0) 596 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:53:23.642 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:23 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:53:23.643 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:23 smithi125 conmon[24927]: -03-24T21:53:23.246311+0000 2021-03-24T21:53:23.643 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:23 smithi125 conmon[24927]: osd.6 (osd.6) 14 : cluster [DBG] 5.2 scrub ok 2021-03-24T21:53:23.643 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:23 smithi125 conmon[19936]: cluster 2021-03-24T21:53:21.832905+0000 osd.5 2021-03-24T21:53:23.643 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:23 smithi125 conmon[19936]: (osd.5) 23 : cluster [DBG] 3.16 scrub ok 2021-03-24T21:53:23.644 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:23 smithi125 conmon[19936]: cluster 2021-03-24T21:53:22.724822+0000 osd.3 (osd.3) 25 : 2021-03-24T21:53:23.644 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:23 smithi125 conmon[19936]: cluster [DBG] 6.b deep-scrub ok 2021-03-24T21:53:23.644 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:23 smithi125 conmon[19936]: audit 2021-03-24T 2021-03-24T21:53:23.644 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:23 smithi125 conmon[19936]: 21:53:22.933324+0000 mon.a (mon.0) 595 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:53:23.645 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:23 smithi125 conmon[19936]: audit 2021-03-24 2021-03-24T21:53:23.645 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:23 smithi125 conmon[19936]: T21:53:22.940975+0000 mon.a (mon.0) 596 : 2021-03-24T21:53:23.645 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:23 smithi125 conmon[19936]: audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:53:23.645 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:23 smithi125 conmon[19936]: cluster 2021-03-24T21:53: 2021-03-24T21:53:23.645 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:23 smithi125 conmon[19936]: 23.246311+0000 osd.6 (osd.6) 14 : cluster [DBG] 5.2 scrub ok 2021-03-24T21:53:24.359 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:24 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:53:24.360 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:24 smithi125 conmon[19936]: 21:53:22.676523+0000 mgr.y (mgr.14144) 270 : cluster [DBG] pgmap v268: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-03-24T21:53:24.360 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:24 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:53:24.360 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:24 smithi125 conmon[19936]: 03-24T21:53:23.578671+0000 osd.0 (osd.0 2021-03-24T21:53:24.360 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:24 smithi125 conmon[19936]: ) 22 : cluster [DBG] 5.5 scrub ok 2021-03-24T21:53:24.361 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:24 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:53:24.361 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:24 smithi125 conmon[19936]: -24T21:53:23.707908+0000 osd.3 2021-03-24T21:53:24.361 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:24 smithi125 conmon[19936]: (osd.3) 26 : cluster [DBG] 6.11 deep-scrub ok 2021-03-24T21:53:24.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:24 smithi142 conmon[20125]: cluster 2021-03-24T21:53:22 2021-03-24T21:53:24.839 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:24 smithi142 conmon[20125]: .676523+0000 mgr.y (mgr.14144) 270 : cluster [DBG] pgmap v268: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-03-24T21:53:24.840 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:24 smithi142 conmon[20125]: cluster 2021-03-24T21:53:24.840 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:24 smithi142 conmon[20125]: 2021-03-24T21:53:23.578671+0000 2021-03-24T21:53:24.840 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:24 smithi142 conmon[20125]: osd.0 (osd.0) 22 : 2021-03-24T21:53:24.841 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:24 smithi142 conmon[20125]: cluster [DBG] 5.5 scrub ok 2021-03-24T21:53:24.841 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:24 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:53:24.841 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:24 smithi142 conmon[20125]: -03-24T21:53:23. 2021-03-24T21:53:24.841 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:24 smithi142 conmon[20125]: 707908+0000 osd.3 (osd.3) 2021-03-24T21:53:24.841 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:24 smithi142 conmon[20125]: 26 : cluster [DBG] 6.11 deep-scrub ok 2021-03-24T21:53:24.842 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:24 smithi125 conmon[24927]: cluster 2021-03-24T21:53:22.676523+0000 2021-03-24T21:53:24.842 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:24 smithi125 conmon[24927]: mgr.y (mgr.14144) 270 : cluster [DBG] pgmap v268: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-03-24T21:53:24.843 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:24 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:53:24.843 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:24 smithi125 conmon[24927]: -24T21:53:23.578671+0000 osd.0 ( 2021-03-24T21:53:24.843 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:24 smithi125 conmon[24927]: osd.0) 22 : cluster [DBG] 2021-03-24T21:53:24.843 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:24 smithi125 conmon[24927]: 5.5 scrub ok 2021-03-24T21:53:24.844 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:24 smithi125 conmon[24927]: cluster 2021-03-24T21:53:24.844 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:24 smithi125 conmon[24927]: 2021-03-24T21:53:23. 2021-03-24T21:53:24.844 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:24 smithi125 conmon[24927]: 707908+0000 osd.3 (osd.3 2021-03-24T21:53:24.844 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:24 smithi125 conmon[24927]: ) 26 : cluster [DBG] 6.11 deep-scrub ok 2021-03-24T21:53:24.844 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:24 smithi125 conmon[24927]: 2021-03-24T21:53:25.272 INFO:teuthology.orchestra.run.smithi125.stdout:55834574897 2021-03-24T21:53:25.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:25 smithi142 conmon[20125]: cluster 2021-03-24T21:53:23.895958+0000 osd.5 (osd.5) 24 : cluster 2021-03-24T21:53:25.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:25 smithi142 conmon[20125]: [DBG] 6.19 scrub ok 2021-03-24T21:53:25.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:25 smithi142 conmon[20125]: cluster 2021-03-24T21:53:23.926876+0000 osd.7 (osd.7) 19 : cluster 2021-03-24T21:53:25.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:25 smithi142 conmon[20125]: [DBG] 6.5 scrub ok 2021-03-24T21:53:25.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:25 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:53:25.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:25 smithi142 conmon[20125]: -03-24T21:53:24.621742 2021-03-24T21:53:25.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:25 smithi142 conmon[20125]: +0000 osd.0 (osd.0) 23 2021-03-24T21:53:25.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:25 smithi142 conmon[20125]: : cluster [DBG] 3.6 scrub ok 2021-03-24T21:53:25.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:25 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:53:25.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:25 smithi142 conmon[20125]: -03-24T21:53: 2021-03-24T21:53:25.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:25 smithi142 conmon[20125]: 25.220420+0000 osd.4 (osd.4) 17 2021-03-24T21:53:25.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:25 smithi142 conmon[20125]: : cluster [DBG] 5.1 deep-scrub ok 2021-03-24T21:53:25.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:25 smithi142 conmon[20125]: audit 2021-03 2021-03-24T21:53:25.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:25 smithi142 conmon[20125]: -24T21:53:25. 2021-03-24T21:53:25.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:25 smithi142 conmon[20125]: 272436+0000 mon.a (mon.0) 597 : audit 2021-03-24T21:53:25.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:25 smithi142 conmon[20125]: [DBG] from='client.? v1:172.21.15.125:0/4225651875' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2021-03-24T21:53:25.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:25 smithi125 conmon[19936]: cluster 2021-03-24T21:53:23.895958+0000 osd.5 (osd 2021-03-24T21:53:25.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:25 smithi125 conmon[19936]: .5) 24 : cluster [DBG] 6.19 scrub ok 2021-03-24T21:53:25.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:25 smithi125 conmon[19936]: cluster 2021-03-24T21:53:23.926876+0000 osd.7 (osd.7) 19 : cluster [DBG] 6.5 scrub ok 2021-03-24T21:53:25.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:25 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:53:25.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:25 smithi125 conmon[19936]: -03-24T21:53:24.621742+0000 osd.0 (osd.0) 23 : cluster [DBG] 3.6 scrub ok 2021-03-24T21:53:25.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:25 smithi125 conmon[19936]: cluster 2021-03-24T21:53: 2021-03-24T21:53:25.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:25 smithi125 conmon[19936]: 25.220420+0000 osd.4 (osd.4) 17 : cluster [DBG] 5.1 deep-scrub ok 2021-03-24T21:53:25.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:25 smithi125 conmon[19936]: audit 2021-03 2021-03-24T21:53:25.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:25 smithi125 conmon[19936]: -24T21:53:25.272436+0000 mon.a (mon.0) 597 : audit [DBG] from='client.? v1:172.21.15.125:0/4225651875' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2021-03-24T21:53:25.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:25 smithi125 conmon[24927]: cluster 2021-03-24T21:53:25.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:25 smithi125 conmon[24927]: 2021-03-24T21:53:23.895958+0000 osd.5 (osd.5) 24 : cluster [DBG] 6.19 scrub ok 2021-03-24T21:53:25.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:25 smithi125 conmon[24927]: cluster 2021-03-24T21:53:23.926876+0000 2021-03-24T21:53:25.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:25 smithi125 conmon[24927]: osd.7 (osd.7) 19 : cluster [DBG] 6.5 scrub ok 2021-03-24T21:53:25.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:25 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:53:25.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:25 smithi125 conmon[24927]: -24T21:53:24.621742+0000 osd.0 (osd.0 2021-03-24T21:53:25.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:25 smithi125 conmon[24927]: ) 23 : cluster [DBG] 3.6 scrub ok 2021-03-24T21:53:25.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:25 smithi125 conmon[24927]: cluster 2021-03-24T21:53:25.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:25 smithi125 conmon[24927]: 2021-03-24T21:53:25.220420+0000 osd.4 (osd.4) 17 : cluster [DBG] 5.1 deep-scrub ok 2021-03-24T21:53:25.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:25 smithi125 conmon[24927]: audit 2021-03-24T 2021-03-24T21:53:25.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:25 smithi125 conmon[24927]: 21:53:25.272436+0000 mon.a (mon.0) 597 : audit [DBG] from='client.? v1:172.21.15.125:0/4225651875' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2021-03-24T21:53:26.162 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574892 got 55834574897 for osd.1 2021-03-24T21:53:26.162 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd last-stat-seq osd.2 2021-03-24T21:53:26.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:26 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:53:26.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:26 smithi125 conmon[24927]: 24T21:53:24.677629+0000 mgr.y (mgr.14144) 271 : cluster [DBG] 2021-03-24T21:53:26.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:26 smithi125 conmon[24927]: pgmap v269: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-24T21:53:26.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:26 smithi125 conmon[24927]: cluster 2021-03-24T21:53:26.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:26 smithi125 conmon[24927]: 2021-03 2021-03-24T21:53:26.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:26 smithi125 conmon[24927]: -24T21:53: 2021-03-24T21:53:26.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:26 smithi125 conmon[24927]: 24.913225+0000 osd.7 ( 2021-03-24T21:53:26.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:26 smithi125 conmon[24927]: osd.7) 20 : cluster 2021-03-24T21:53:26.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:26 smithi125 conmon[24927]: [DBG] 6.15 scrub ok 2021-03-24T21:53:26.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:26 smithi125 conmon[24927]: cluster 2021-03-24T21:53:26.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:26 smithi125 conmon[24927]: 2021-03-24T21: 2021-03-24T21:53:26.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:26 smithi125 conmon[24927]: 53:25.593349+0000 2021-03-24T21:53:26.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:26 smithi125 conmon[24927]: osd.0 (osd.0) 24 : 2021-03-24T21:53:26.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:26 smithi125 conmon[24927]: cluster [DBG] 6.0 scrub ok 2021-03-24T21:53:26.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:26 smithi125 conmon[19936]: cluster 2021-03-24T21:53:24.677629 2021-03-24T21:53:26.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:26 smithi125 conmon[19936]: +0000 mgr.y (mgr.14144) 271 : cluster [DBG] pgmap v269: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-24T21:53:26.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:26 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:53:26.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:26 smithi125 conmon[19936]: :53:24.913225+0000 osd.7 (osd.7) 20 : cluster [DBG] 6.15 scrub ok 2021-03-24T21:53:26.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:26 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:53:26.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:26 smithi125 conmon[19936]: 03-24T21:53:25.593349+0000 osd.0 ( 2021-03-24T21:53:26.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:26 smithi125 conmon[19936]: osd.0) 24 : cluster [DBG] 6.0 scrub ok 2021-03-24T21:53:26.481 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:53:26.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:26 smithi142 conmon[20125]: cluster 2021-03-24T21:53:24.677629+0000 mgr.y (mgr.14144) 271 : cluster 2021-03-24T21:53:26.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:26 smithi142 conmon[20125]: [DBG] pgmap v269: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-24T21:53:26.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:26 smithi142 conmon[20125]: cluster 2021-03-24T21:53:24.913225+0000 osd.7 (osd.7) 20 : 2021-03-24T21:53:26.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:26 smithi142 conmon[20125]: cluster [DBG] 6.15 scrub ok 2021-03-24T21:53:26.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:26 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:53:26.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:26 smithi142 conmon[20125]: -03-24T21:53:25.593349+0000 osd.0 (osd. 2021-03-24T21:53:26.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:26 smithi142 conmon[20125]: 0) 24 : cluster [DBG] 6.0 scrub ok 2021-03-24T21:53:27.379 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:27 smithi125 conmon[24927]: cluster 2021-03-24T21:53:26. 2021-03-24T21:53:27.379 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:27 smithi125 conmon[24927]: 584666+0000 osd.0 (osd.0) 25 : cluster [DBG] 5.3 deep-scrub ok 2021-03-24T21:53:27.379 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:27 smithi125 conmon[24927]: cluster 2021-03-24T21:53:27.238593 2021-03-24T21:53:27.380 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:27 smithi125 conmon[24927]: +0000 osd.4 (osd.4) 18 : cluster [DBG] 5.1c scrub ok 2021-03-24T21:53:27.380 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:27 smithi125 conmon[24927]: 2021-03-24T21:53:27.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:27 smithi125 conmon[19936]: cluster 2021-03-24T21:53:27.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:27 smithi125 conmon[19936]: 2021-03-24T21:53:26.584666+0000 osd.0 (osd.0) 25 : cluster [DBG] 5.3 deep-scrub ok 2021-03-24T21:53:27.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:27 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:53:27.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:27 smithi125 conmon[19936]: -03-24T21:53:27. 2021-03-24T21:53:27.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:27 smithi125 conmon[19936]: 238593+0000 osd.4 (osd.4) 18 : 2021-03-24T21:53:27.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:27 smithi125 conmon[19936]: cluster [DBG] 5.1c scrub ok 2021-03-24T21:53:27.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:27 smithi142 conmon[20125]: cluster 2021-03-24T21:53:26.584666+0000 osd.0 (osd.0) 25 : 2021-03-24T21:53:27.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:27 smithi142 conmon[20125]: cluster [DBG] 5.3 deep-scrub ok 2021-03-24T21:53:27.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:27 smithi142 conmon[20125]: cluster 2021-03-24T21:53:27.238593+0000 osd.4 (osd.4) 2021-03-24T21:53:27.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:27 smithi142 conmon[20125]: 18 : cluster [DBG] 5.1c scrub ok 2021-03-24T21:53:28.436 INFO:teuthology.orchestra.run.smithi125.stdout:77309411375 2021-03-24T21:53:28.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:28 smithi125 conmon[19936]: cluster 2021-03-24T21:53:28.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:28 smithi125 conmon[19936]: 2021-03-24T21:53:26.678518+0000 mgr.y (mgr.14144) 272 : cluster [DBG] pgmap v270: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 971 B/s rd, 0 op/s 2021-03-24T21:53:28.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:28 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:53:28.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:28 smithi125 conmon[19936]: 53:26.908788+0000 osd.7 (osd.7) 21 2021-03-24T21:53:28.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:28 smithi125 conmon[19936]: : cluster [DBG] 3.d scrub ok 2021-03-24T21:53:28.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:28 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T21:53:28.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:28 smithi125 conmon[19936]: T21:53:27.604218+0000 osd.0 (osd 2021-03-24T21:53:28.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:28 smithi125 conmon[19936]: .0) 26 : cluster [DBG] 6.18 scrub ok 2021-03-24T21:53:28.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:28 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:53:28.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:28 smithi125 conmon[19936]: -03-24T21:53:27.700973+0000 osd.3 2021-03-24T21:53:28.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:28 smithi125 conmon[19936]: (osd.3) 27 : cluster [DBG] 6.c scrub ok 2021-03-24T21:53:28.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:28 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:53:28.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:28 smithi125 conmon[19936]: -03-24T21:53:28.219889+0000 2021-03-24T21:53:28.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:28 smithi125 conmon[19936]: osd.4 (osd.4) 19 : cluster [DBG] 2021-03-24T21:53:28.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:28 smithi125 conmon[19936]: 3.3 deep-scrub ok 2021-03-24T21:53:28.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:28 smithi125 conmon[24927]: cluster 2021-03-24T21:53:28.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:28 smithi125 conmon[24927]: 2021-03-24T 2021-03-24T21:53:28.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:28 smithi125 conmon[24927]: 21:53:26.678518+0000 mgr.y (mgr.14144) 272 : cluster [DBG] pgmap v270: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 971 B/s rd, 0 op/s 2021-03-24T21:53:28.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:28 smithi125 conmon[24927]: cluster 2021-03-24T21:53:28.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:28 smithi125 conmon[24927]: 2021-03-24T21:53 2021-03-24T21:53:28.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:28 smithi125 conmon[24927]: :26.908788+0000 osd.7 (osd 2021-03-24T21:53:28.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:28 smithi125 conmon[24927]: .7) 21 : cluster [DBG] 2021-03-24T21:53:28.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:28 smithi125 conmon[24927]: 3.d scrub ok 2021-03-24T21:53:28.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:28 smithi125 conmon[24927]: cluster 2021-03-24T21:53:28.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:28 smithi125 conmon[24927]: 2021-03-24T21:53:27. 2021-03-24T21:53:28.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:28 smithi125 conmon[24927]: 604218+0000 osd.0 (osd.0) 26 : cluster [DBG] 2021-03-24T21:53:28.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:28 smithi125 conmon[24927]: 6.18 scrub ok 2021-03-24T21:53:28.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:28 smithi125 conmon[24927]: cluster 2021-03-24T21:53:28.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:28 smithi125 conmon[24927]: 2021-03-24T21:53:27.700973+0000 osd.3 2021-03-24T21:53:28.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:28 smithi125 conmon[24927]: (osd.3) 27 : cluster [DBG] 6.c scrub ok 2021-03-24T21:53:28.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:28 smithi125 conmon[24927]: 2021-03-24T21:53:28.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:28 smithi125 conmon[24927]: cluster 2021-03-24T21:53: 2021-03-24T21:53:28.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:28 smithi125 conmon[24927]: 28.219889+0000 osd.4 (osd.4) 2021-03-24T21:53:28.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:28 smithi125 conmon[24927]: 19 : cluster [DBG] 3.3 deep-scrub ok 2021-03-24T21:53:28.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:28 smithi142 conmon[20125]: cluster 2021-03-24T21:53:26.678518+0000 mgr.y (mgr.14144) 272 : cluster 2021-03-24T21:53:28.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:28 smithi142 conmon[20125]: [DBG] pgmap v270: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 971 B/s rd, 0 op/s 2021-03-24T21:53:28.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:28 smithi142 conmon[20125]: cluster 2021-03-24T21:53:26.908788+0000 osd.7 (osd.7) 21 : cluster [DBG] 3.d scrub ok 2021-03-24T21:53:28.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:28 smithi142 conmon[20125]: cluster 2021-03-24T21:53:27.604218+0000 osd.0 (osd.0) 26 : cluster [DBG] 2021-03-24T21:53:28.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:28 smithi142 conmon[20125]: 6.18 scrub ok 2021-03-24T21:53:28.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:28 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:53:28.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:28 smithi142 conmon[20125]: T21:53:27.700973+0000 osd.3 (osd.3) 27 : cluster [DBG] 6.c scrub ok 2021-03-24T21:53:28.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:28 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:53:28.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:28 smithi142 conmon[20125]: 53:28.219889+0000 osd.4 (osd.4) 19 : cluster [DBG] 2021-03-24T21:53:28.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:28 smithi142 conmon[20125]: 3.3 deep-scrub ok 2021-03-24T21:53:29.337 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411369 got 77309411375 for osd.2 2021-03-24T21:53:29.338 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd last-stat-seq osd.3 2021-03-24T21:53:29.649 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:53:29.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:29 smithi125 conmon[19936]: cluster 2021-03-24T21:53:29.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:29 smithi125 conmon[19936]: 2021-03-24T21:53:29.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:29 smithi125 conmon[19936]: 2021 2021-03-24T21:53:29.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:29 smithi125 conmon[19936]: -03-24T21 2021-03-24T21:53:29.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:29 smithi125 conmon[19936]: :53:27.877619+0000 osd.7 (osd.7) 22 : cluster [DBG] 6.12 scrub ok 2021-03-24T21:53:29.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:29 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:53:29.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:29 smithi125 conmon[19936]: 03-24T21:53:28.435982+0000 mon.c (mon.2) 16 : audit 2021-03-24T21:53:29.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:29 smithi125 conmon[19936]: [DBG] from='client.? v1:172.21.15.125:0/1362303252' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2021-03-24T21:53:29.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:29 smithi125 conmon[19936]: cluster 2021-03-24T21:53:29.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:29 smithi125 conmon[19936]: 2021-03-24T21:53:28.691257 2021-03-24T21:53:29.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:29 smithi125 conmon[19936]: +0000 osd.1 (osd.1) 22 : cluster [DBG] 2021-03-24T21:53:29.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:29 smithi125 conmon[19936]: 6.1 scrub ok 2021-03-24T21:53:29.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:29 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:53:29.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:29 smithi125 conmon[24927]: 21:53:27.877619+0000 osd.7 (osd.7) 22 : cluster [DBG] 6.12 scrub ok 2021-03-24T21:53:29.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:29 smithi125 conmon[24927]: audit 2021-03-24T21:53:29.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:29 smithi125 conmon[24927]: 2021-03-24T21:53:28. 2021-03-24T21:53:29.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:29 smithi125 conmon[24927]: 435982+0000 mon.c (mon.2 2021-03-24T21:53:29.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:29 smithi125 conmon[24927]: ) 16 : audit [DBG] from='client.? v1:172.21.15.125:0/1362303252' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2021-03-24T21:53:29.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:29 smithi125 conmon[24927]: cluster 2021-03-24T21:53:29.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:29 smithi125 conmon[24927]: 2021-03-24T 2021-03-24T21:53:29.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:29 smithi125 conmon[24927]: 21:53:28. 2021-03-24T21:53:29.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:29 smithi125 conmon[24927]: 691257+0000 osd.1 (osd 2021-03-24T21:53:29.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:29 smithi125 conmon[24927]: .1) 22 : cluster 2021-03-24T21:53:29.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:29 smithi125 conmon[24927]: [DBG] 6.1 scrub ok 2021-03-24T21:53:29.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:29 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:53:29.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:29 smithi142 conmon[20125]: 21:53:27.877619+0000 osd.7 (osd.7) 22 : cluster [DBG] 6.12 scrub ok 2021-03-24T21:53:29.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:29 smithi142 conmon[20125]: audit 2021-03- 2021-03-24T21:53:29.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:29 smithi142 conmon[20125]: 24T21:53:28.435982+0000 mon.c (mon.2) 16 : audit [DBG] from='client.? v1:172.21.15.125:0/1362303252' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2021-03-24T21:53:29.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:29 smithi142 conmon[20125]: cluster 2021-03-24T21:53:29.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:29 smithi142 conmon[20125]: 2021-03-24T21:53:29.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:29 smithi142 conmon[20125]: 2021 2021-03-24T21:53:29.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:29 smithi142 conmon[20125]: - 2021-03-24T21:53:29.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:29 smithi142 conmon[20125]: 03-24T21:53:28.691257 2021-03-24T21:53:29.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:29 smithi142 conmon[20125]: +0000 osd.1 (osd.1) 22 : cluster [DBG] 6.1 scrub ok 2021-03-24T21:53:30.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:30 smithi125 conmon[19936]: cluster 2021-03-24T21:53:28.679428+0000 mgr.y (mgr.14144) 273 : cluster [DBG] pgmap v271: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-03-24T21:53:30.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:30 smithi125 conmon[19936]: cluster 2021-03-24T21:53:29.268264+0000 osd.4 (osd.4) 20 : cluster [DBG] 5.e scrub ok 2021-03-24T21:53:30.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:30 smithi125 conmon[19936]: cluster 2021-03-24T21:53:29.735662+0000 osd.1 (osd.1) 23 : cluster [DBG] 5.c scrub ok 2021-03-24T21:53:30.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:30 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:53:30.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:30 smithi125 conmon[24927]: 24T21:53:28.679428+0000 mgr.y (mgr.14144) 273 2021-03-24T21:53:30.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:30 smithi125 conmon[24927]: : cluster [DBG] pgmap v271: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-03-24T21:53:30.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:30 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:53:30.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:30 smithi125 conmon[24927]: -03-24T21: 2021-03-24T21:53:30.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:30 smithi125 conmon[24927]: 53:29.268264+0000 2021-03-24T21:53:30.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:30 smithi125 conmon[24927]: osd.4 (osd.4 2021-03-24T21:53:30.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:30 smithi125 conmon[24927]: ) 20 : cluster [DBG] 2021-03-24T21:53:30.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:30 smithi125 conmon[24927]: 5.e scrub ok 2021-03-24T21:53:30.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:30 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:53:30.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:30 smithi125 conmon[24927]: -03-24T21: 2021-03-24T21:53:30.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:30 smithi125 conmon[24927]: 53:29.735662+0000 2021-03-24T21:53:30.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:30 smithi125 conmon[24927]: osd.1 (osd.1 2021-03-24T21:53:30.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:30 smithi125 conmon[24927]: ) 23 : cluster [DBG] 2021-03-24T21:53:30.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:30 smithi125 conmon[24927]: 5.c scrub ok 2021-03-24T21:53:30.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:30 smithi142 conmon[20125]: cluster 2021-03-24T21:53:28.679428+0000 mgr.y (mgr.14144) 273 : cluster 2021-03-24T21:53:30.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:30 smithi142 conmon[20125]: [DBG] pgmap v271: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-03-24T21:53:30.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:30 smithi142 conmon[20125]: cluster 2021-03-24T21:53:29.268264+0000 osd.4 (osd.4) 20 : cluster [DBG] 5.e scrub ok 2021-03-24T21:53:30.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:30 smithi142 conmon[20125]: cluster 2021-03-24T21:53:29.735662+0000 osd.1 (osd.1) 23 : cluster [DBG] 5.c scrub ok 2021-03-24T21:53:30.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:30 smithi142 conmon[20125]: 2021-03-24T21:53:31.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:31 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:53:31.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:31 smithi125 conmon[24927]: -24T21:53:29.773517+0000 osd.1 (osd. 2021-03-24T21:53:31.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:31 smithi125 conmon[24927]: 1) 24 : cluster [DBG] 2021-03-24T21:53:31.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:31 smithi125 conmon[24927]: purged_snaps scrub starts 2021-03-24T21:53:31.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:31 smithi125 conmon[24927]: cluster 2021-03-24T21:53:31.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:31 smithi125 conmon[24927]: 2021-03-24T21 2021-03-24T21:53:31.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:31 smithi125 conmon[24927]: :53:29.773693 2021-03-24T21:53:31.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:31 smithi125 conmon[24927]: +0000 osd.1 (osd. 2021-03-24T21:53:31.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:31 smithi125 conmon[24927]: 1) 25 : 2021-03-24T21:53:31.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:31 smithi125 conmon[24927]: cluster [DBG] purged_snaps scrub ok 2021-03-24T21:53:31.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:31 smithi125 conmon[24927]: cluster 2021-03-24T21:53:31.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:31 smithi125 conmon[24927]: 2021-03-24T21: 2021-03-24T21:53:31.420 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:31 smithi125 conmon[24927]: 53:30.399954 2021-03-24T21:53:31.420 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:31 smithi125 conmon[24927]: +0000 mon.a (mon. 2021-03-24T21:53:31.420 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:31 smithi125 conmon[24927]: 0) 598 : cluster 2021-03-24T21:53:31.420 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:31 smithi125 conmon[24927]: [DBG] osdmap e90: 8 total, 8 up, 8 in 2021-03-24T21:53:31.421 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:31 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:53:31.421 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:31 smithi125 conmon[24927]: 53:30.660613+0000 osd.0 (osd.0) 27 : cluster [DBG] 5.1e scrub ok 2021-03-24T21:53:31.421 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:31 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:53:31.422 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:31 smithi125 conmon[24927]: -24T21:53:30.836503+0000 2021-03-24T21:53:31.422 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:31 smithi125 conmon[24927]: osd.4 (osd.4) 21 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:53:31.423 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:31 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:53:31.423 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:31 smithi125 conmon[24927]: 24T21:53:30.836695+0000 osd.4 (osd.4) 22 : cluster [DBG] 2021-03-24T21:53:31.424 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:31 smithi125 conmon[24927]: purged_snaps scrub ok 2021-03-24T21:53:31.424 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:31 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:53:31.424 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:31 smithi125 conmon[24927]: -03-24T21:53:30.906494+0000 osd.7 (osd.7) 23 2021-03-24T21:53:31.425 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:31 smithi125 conmon[24927]: : cluster [DBG] 1.0 scrub ok 2021-03-24T21:53:31.539 INFO:teuthology.orchestra.run.smithi125.stdout:111669149739 2021-03-24T21:53:31.758 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:31 smithi125 conmon[19936]: cluster 2021-03-24T21:53:31.759 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:31 smithi125 conmon[19936]: 2021-03-24T21:53:29.773517+0000 osd.1 (osd.1) 24 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:53:31.759 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:31 smithi125 conmon[19936]: 2021-03-24T21:53:31.759 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:31 smithi125 conmon[19936]: cluster 2021-03-24T21:53:29. 2021-03-24T21:53:31.760 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:31 smithi125 conmon[19936]: 773693+0000 osd.1 (osd.1) 25 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:53:31.760 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:31 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:53:31.760 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:31 smithi125 conmon[19936]: 21:53:30.399954+0000 2021-03-24T21:53:31.760 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:31 smithi125 conmon[19936]: mon.a (mon.0) 598 2021-03-24T21:53:31.760 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:31 smithi125 conmon[19936]: : cluster 2021-03-24T21:53:31.761 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:31 smithi125 conmon[19936]: [DBG] osdmap e90: 8 total, 8 up, 8 in 2021-03-24T21:53:31.761 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:31 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:53:31.761 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:31 smithi125 conmon[19936]: -24T21:53:30.660613+0000 osd.0 (osd.0) 27 : cluster 2021-03-24T21:53:31.761 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:31 smithi125 conmon[19936]: [DBG] 5.1e scrub ok 2021-03-24T21:53:31.761 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:31 smithi125 conmon[19936]: cluster 2021-03-24T21:53:31.762 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:31 smithi125 conmon[19936]: 2021-03-24T21:53:30.836503+0000 osd.4 (osd.4 2021-03-24T21:53:31.762 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:31 smithi125 conmon[19936]: ) 21 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:53:31.762 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:31 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T21:53:31.762 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:31 smithi125 conmon[19936]: T21:53:30.836695+0000 osd.4 (osd.4) 22 : 2021-03-24T21:53:31.763 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:31 smithi125 conmon[19936]: cluster [DBG] purged_snaps scrub ok 2021-03-24T21:53:31.763 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:31 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:53:31.763 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:31 smithi125 conmon[19936]: -03-24T21:53:30.906494+0000 osd.7 (osd.7) 23 : cluster [DBG] 1.0 scrub ok 2021-03-24T21:53:31.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:31 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:53:31.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:31 smithi142 conmon[20125]: 53:29.773517+0000 osd.1 (osd.1) 24 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:53:31.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:31 smithi142 conmon[20125]: cluster 2021-03-24T21:53:29.773693 2021-03-24T21:53:31.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:31 smithi142 conmon[20125]: +0000 osd.1 (osd.1) 25 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:53:31.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:31 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:53:31.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:31 smithi142 conmon[20125]: 21:53:30.399954+0000 mon.a (mon.0) 598 : cluster [DBG] osdmap e90: 8 total, 8 up, 8 in 2021-03-24T21:53:31.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:31 smithi142 conmon[20125]: cluster 2021-03-24T21:53:30. 2021-03-24T21:53:31.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:31 smithi142 conmon[20125]: 660613+0000 osd.0 (osd.0) 27 : cluster [DBG] 5.1e scrub ok 2021-03-24T21:53:31.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:31 smithi142 conmon[20125]: cluster 2021-03-24T21:53 2021-03-24T21:53:31.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:31 smithi142 conmon[20125]: :30.836503+0000 osd.4 (osd.4) 21 : cluster [DBG] 2021-03-24T21:53:31.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:31 smithi142 conmon[20125]: purged_snaps scrub starts 2021-03-24T21:53:31.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:31 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:53:31.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:31 smithi142 conmon[20125]: 03-24T21:53:30.836695+0000 osd.4 (osd.4) 22 : cluster 2021-03-24T21:53:31.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:31 smithi142 conmon[20125]: [DBG] purged_snaps scrub ok 2021-03-24T21:53:31.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:31 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:53:31.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:31 smithi142 conmon[20125]: -03-24T21:53:30.906494+0000 osd.7 (osd.7) 23 : cluster [DBG] 2021-03-24T21:53:31.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:31 smithi142 conmon[20125]: 1.0 scrub ok 2021-03-24T21:53:32.494 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149734 got 111669149739 for osd.3 2021-03-24T21:53:32.495 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd last-stat-seq osd.4 2021-03-24T21:53:32.797 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:53:32.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:32 smithi142 conmon[20125]: cluster 2021-03-24T21:53:30. 2021-03-24T21:53:32.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:32 smithi142 conmon[20125]: 680160+0000 mgr.y (mgr.14144) 274 : cluster [DBG] pgmap v273: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-24T21:53:32.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:32 smithi142 conmon[20125]: cluster 2021-03-24T21:53:31.424426+0000 mon.a (mon. 2021-03-24T21:53:32.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:32 smithi142 conmon[20125]: 0) 599 : cluster [DBG] osdmap e91: 8 total, 8 up, 8 in 2021-03-24T21:53:32.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:32 smithi142 conmon[20125]: audit 2021-03-24T21:53:31.538306+0000 2021-03-24T21:53:32.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:32 smithi142 conmon[20125]: mon.c (mon.2) 17 : audit [DBG] from='client.? v1:172.21.15.125:0/802316941' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2021-03-24T21:53:32.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:32 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:53:32.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:32 smithi142 conmon[20125]: -03-24T21:53:31.653325+0000 osd.0 (osd.0) 28 : cluster [DBG] 2021-03-24T21:53:32.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:32 smithi142 conmon[20125]: 4.b scrub ok 2021-03-24T21:53:32.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:32 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:53:32.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:32 smithi142 conmon[20125]: -24T21:53:31.665423+0000 osd.3 (osd.3) 28 : cluster [DBG] 2021-03-24T21:53:32.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:32 smithi142 conmon[20125]: 5.13 scrub ok 2021-03-24T21:53:32.909 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:32 smithi125 conmon[24927]: cluster 2021-03-24T21:53:30.680160+0000 mgr.y (mgr.14144) 274 2021-03-24T21:53:32.910 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:32 smithi125 conmon[24927]: : cluster [DBG] pgmap v273: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-24T21:53:32.910 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:32 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:53:32.910 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:32 smithi125 conmon[24927]: T21:53:31.424426+0000 mon.a (mon.0) 599 : cluster [DBG] 2021-03-24T21:53:32.911 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:32 smithi125 conmon[24927]: osdmap e91: 8 total, 8 up, 8 in 2021-03-24T21:53:32.911 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:32 smithi125 conmon[24927]: audit 2021-03-24T21:53: 2021-03-24T21:53:32.911 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:32 smithi125 conmon[24927]: 31.538306+0000 mon.c (mon.2) 17 : audit [DBG] from='client.? v1:172.21.15.125:0/802316941' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2021-03-24T21:53:32.911 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:32 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:53:32.911 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:32 smithi125 conmon[24927]: 03-24T21:53:31. 2021-03-24T21:53:32.912 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:32 smithi125 conmon[24927]: 653325+0000 osd.0 (osd.0) 28 : cluster 2021-03-24T21:53:32.912 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:32 smithi125 conmon[24927]: [DBG] 4.b scrub ok 2021-03-24T21:53:32.912 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:32 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:53:32.912 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:32 smithi125 conmon[24927]: 24T21:53:31.665423+0000 2021-03-24T21:53:32.912 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:32 smithi125 conmon[24927]: osd.3 (osd.3) 28 : cluster [DBG] 2021-03-24T21:53:32.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:32 smithi125 conmon[24927]: 5.13 scrub ok 2021-03-24T21:53:32.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:32 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:53:32.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:32 smithi125 conmon[19936]: 21:53:30.680160+0000 mgr.y (mgr.14144) 274 : cluster [DBG] pgmap v273: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-24T21:53:32.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:32 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T21:53:32.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:32 smithi125 conmon[19936]: T21:53:31.424426+0000 2021-03-24T21:53:32.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:32 smithi125 conmon[19936]: mon.a (mon.0) 599 : cluster [DBG] 2021-03-24T21:53:32.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:32 smithi125 conmon[19936]: osdmap e91: 8 total, 8 up, 8 in 2021-03-24T21:53:32.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:32 smithi125 conmon[19936]: audit 2021-03- 2021-03-24T21:53:32.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:32 smithi125 conmon[19936]: 24T21:53:31.538306 2021-03-24T21:53:32.915 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:32 smithi125 conmon[19936]: +0000 mon.c (mon.2) 17 : audit 2021-03-24T21:53:32.915 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:32 smithi125 conmon[19936]: [DBG] from='client.? v1:172.21.15.125:0/802316941' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2021-03-24T21:53:32.915 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:32 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:53:32.915 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:32 smithi125 conmon[19936]: 24T21:53:31.653325+0000 2021-03-24T21:53:32.915 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:32 smithi125 conmon[19936]: osd.0 (osd.0) 28 : cluster [DBG] 2021-03-24T21:53:32.916 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:32 smithi125 conmon[19936]: 4.b scrub ok 2021-03-24T21:53:32.916 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:32 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:53:32.916 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:32 smithi125 conmon[19936]: 21:53:31.665423+0000 osd.3 ( 2021-03-24T21:53:32.916 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:32 smithi125 conmon[19936]: osd.3) 28 : cluster [DBG] 5.13 scrub ok 2021-03-24T21:53:34.609 INFO:teuthology.orchestra.run.smithi125.stdout:137438953509 2021-03-24T21:53:34.625 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:34 smithi125 conmon[19936]: cluster 2021-03-24T21:53:32.680804+0000 mgr.y (mgr.14144) 275 : 2021-03-24T21:53:34.625 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:34 smithi125 conmon[19936]: cluster [DBG] pgmap v275: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-03-24T21:53:34.625 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:34 smithi125 conmon[19936]: cluster 2021-03-24T21:53:33.289989+0000 osd.4 (osd.4) 23 : cluster [DBG] 4.1b deep-scrub ok 2021-03-24T21:53:34.626 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:34 smithi125 conmon[24927]: cluster 2021-03-24T21:53:32. 2021-03-24T21:53:34.626 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:34 smithi125 conmon[24927]: 680804+0000 mgr.y (mgr.14144) 275 : cluster [DBG] pgmap v275: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-03-24T21:53:34.626 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:34 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:53:34.627 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:34 smithi125 conmon[24927]: 24T21:53:33.289989+0000 osd.4 (osd.4) 23 : cluster [DBG] 4.1b deep-scrub ok 2021-03-24T21:53:34.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:34 smithi142 conmon[20125]: cluster 2021-03-24T21:53:32. 2021-03-24T21:53:34.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:34 smithi142 conmon[20125]: 680804+0000 mgr.y (mgr.14144) 275 : cluster [DBG] pgmap v275: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-03-24T21:53:34.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:34 smithi142 conmon[20125]: cluster 2021-03-24T21 2021-03-24T21:53:34.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:34 smithi142 conmon[20125]: :53:33.289989+0000 osd.4 (osd.4) 23 : cluster [DBG] 4.1b deep-scrub ok 2021-03-24T21:53:35.487 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953505 got 137438953509 for osd.4 2021-03-24T21:53:35.488 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd last-stat-seq osd.5 2021-03-24T21:53:35.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:35 smithi125 conmon[24927]: audit 2021-03-24T21:53:34. 2021-03-24T21:53:35.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:35 smithi125 conmon[24927]: 609964+0000 mon.a (mon.0) 600 : audit [DBG] from='client.? v1:172.21.15.125:0/745612390' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2021-03-24T21:53:35.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:35 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:53:35.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:35 smithi125 conmon[19936]: 03-24T21:53:34.609964+0000 mon.a (mon.0) 600 : audit [DBG] from='client.? v1:172.21.15.125:0/745612390' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2021-03-24T21:53:35.805 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:53:35.892 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:35 smithi142 conmon[20125]: audit 2021-03-24T21:53:34.609964+0000 mon.a (mon.0) 600 : audit 2021-03-24T21:53:35.892 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:35 smithi142 conmon[20125]: [DBG] from='client.? v1:172.21.15.125:0/745612390' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2021-03-24T21:53:36.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:36 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:53:36.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:36 smithi125 conmon[19936]: 53:34.682075+0000 mgr.y (mgr.14144) 276 : cluster [DBG] pgmap v276: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-03-24T21:53:36.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:36 smithi125 conmon[19936]: cluster 2021-03-24T21:53:35.609162+0000 osd.3 (osd.3) 29 : cluster [DBG] 3.8 scrub ok 2021-03-24T21:53:36.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:36 smithi125 conmon[19936]: cluster 2021-03-24T21:53:35.636292+0000 osd.0 (osd.0 2021-03-24T21:53:36.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:36 smithi125 conmon[19936]: ) 29 : cluster [DBG] 6.9 deep-scrub ok 2021-03-24T21:53:36.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:36 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:53:36.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:36 smithi125 conmon[24927]: 21:53:34.682075+0000 mgr.y (mgr.14144) 276 : cluster [DBG] pgmap v276: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-03-24T21:53:36.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:36 smithi125 conmon[24927]: cluster 2021-03-24T21:53:36.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:36 smithi125 conmon[24927]: 2021-03-24 2021-03-24T21:53:36.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:36 smithi125 conmon[24927]: T21:53:35.609162+0000 osd.3 (osd.3) 29 2021-03-24T21:53:36.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:36 smithi125 conmon[24927]: : cluster [DBG] 3.8 scrub ok 2021-03-24T21:53:36.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:36 smithi125 conmon[24927]: cluster 2021-03-24T21:53:36.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:36 smithi125 conmon[24927]: 2021-03-24T 2021-03-24T21:53:36.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:36 smithi125 conmon[24927]: 21:53:35.636292+0000 osd.0 2021-03-24T21:53:36.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:36 smithi125 conmon[24927]: (osd.0) 29 : cluster [DBG] 6.9 deep-scrub ok 2021-03-24T21:53:36.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:36 smithi125 conmon[24927]: 2021-03-24T21:53:36.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:36 smithi142 conmon[20125]: cluster 2021-03-24T21:53:34. 2021-03-24T21:53:36.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:36 smithi142 conmon[20125]: 682075+0000 mgr.y (mgr.14144) 276 : cluster [DBG] pgmap v276: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-03-24T21:53:36.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:36 smithi142 conmon[20125]: cluster 2021-03-24T21 2021-03-24T21:53:36.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:36 smithi142 conmon[20125]: :53:35.609162+0000 osd.3 (osd.3) 29 : cluster [DBG] 2021-03-24T21:53:36.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:36 smithi142 conmon[20125]: 3.8 scrub ok 2021-03-24T21:53:36.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:36 smithi142 conmon[20125]: cluster 2021-03-24T21:53:36.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:36 smithi142 conmon[20125]: 2021-03-24T21:53:35.636292+0000 osd.0 2021-03-24T21:53:36.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:36 smithi142 conmon[20125]: (osd.0) 29 : cluster [DBG] 6.9 deep-scrub ok 2021-03-24T21:53:37.451 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:37 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:53:37.452 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:37 smithi125 conmon[19936]: 21:53:36.316859+0000 osd.4 (osd.4) 24 : cluster [DBG] 3.14 scrub ok 2021-03-24T21:53:37.452 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:37 smithi125 conmon[19936]: cluster 2021-03-24T21:53:37.452 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:37 smithi125 conmon[19936]: 2021-03-24T21:53:37.452 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:37 smithi125 conmon[19936]: 2021 2021-03-24T21:53:37.453 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:37 smithi125 conmon[19936]: - 2021-03-24T21:53:37.453 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:37 smithi125 conmon[19936]: 03 2021-03-24T21:53:37.453 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:37 smithi125 conmon[19936]: - 2021-03-24T21:53:37.453 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:37 smithi125 conmon[19936]: 24 2021-03-24T21:53:37.454 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:37 smithi125 conmon[19936]: T 2021-03-24T21:53:37.454 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:37 smithi125 conmon[19936]: 21:53:36.912855+0000 osd.7 2021-03-24T21:53:37.454 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:37 smithi125 conmon[19936]: (osd.7) 24 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:53:37.454 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:37 smithi125 conmon[19936]: cluster 2021-03-24T21:53:37.454 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:37 smithi125 conmon[19936]: 2021- 2021-03-24T21:53:37.455 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:37 smithi125 conmon[19936]: 03-24T21:53:36.913051 2021-03-24T21:53:37.455 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:37 smithi125 conmon[19936]: +0000 osd.7 (osd.7) 25 : cluster [DBG] 2021-03-24T21:53:37.455 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:37 smithi125 conmon[19936]: purged_snaps scrub ok 2021-03-24T21:53:37.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:37 smithi142 conmon[20125]: cluster 2021-03-24T21:53:36. 2021-03-24T21:53:37.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:37 smithi142 conmon[20125]: 316859+0000 osd.4 (osd.4) 24 : cluster [DBG] 3.14 scrub ok 2021-03-24T21:53:37.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:37 smithi142 conmon[20125]: cluster 2021-03-24T21:53:36.912855+0000 osd.7 (osd.7) 2021-03-24T21:53:37.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:37 smithi142 conmon[20125]: 24 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:53:37.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:37 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:53:37.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:37 smithi142 conmon[20125]: T21:53:36.913051+0000 osd.7 (osd.7) 25 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:53:37.909 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:37 smithi125 conmon[24927]: cluster 2021-03-24T21:53:37.910 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:37 smithi125 conmon[24927]: 2021-03-24T21:53:37.910 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:37 smithi125 conmon[24927]: 2021 2021-03-24T21:53:37.910 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:37 smithi125 conmon[24927]: - 2021-03-24T21:53:37.910 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:37 smithi125 conmon[24927]: 03 2021-03-24T21:53:37.911 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:37 smithi125 conmon[24927]: - 2021-03-24T21:53:37.911 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:37 smithi125 conmon[24927]: 24 2021-03-24T21:53:37.911 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:37 smithi125 conmon[24927]: T 2021-03-24T21:53:37.911 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:37 smithi125 conmon[24927]: 21 2021-03-24T21:53:37.912 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:37 smithi125 conmon[24927]: : 2021-03-24T21:53:37.912 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:37 smithi125 conmon[24927]: 53:36.316859+0000 2021-03-24T21:53:37.912 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:37 smithi125 conmon[24927]: osd.4 (osd.4) 24 : 2021-03-24T21:53:37.912 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:37 smithi125 conmon[24927]: cluster [DBG] 2021-03-24T21:53:37.912 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:37 smithi125 conmon[24927]: 3.14 scrub ok 2021-03-24T21:53:37.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:37 smithi125 conmon[24927]: cluster 2021-03-24T21:53:37.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:37 smithi125 conmon[24927]: 2021-03 2021-03-24T21:53:37.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:37 smithi125 conmon[24927]: -24T21:53:36.912855+0000 2021-03-24T21:53:37.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:37 smithi125 conmon[24927]: osd.7 (osd.7) 24 2021-03-24T21:53:37.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:37 smithi125 conmon[24927]: : cluster [DBG] 2021-03-24T21:53:37.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:37 smithi125 conmon[24927]: purged_snaps scrub starts 2021-03-24T21:53:37.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:37 smithi125 conmon[24927]: cluster 2021-03-24T21:53:37.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:37 smithi125 conmon[24927]: 2021- 2021-03-24T21:53:37.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:37 smithi125 conmon[24927]: 03-24T21:53:36.913051 2021-03-24T21:53:37.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:37 smithi125 conmon[24927]: +0000 osd.7 (osd. 2021-03-24T21:53:37.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:37 smithi125 conmon[24927]: 7) 25 : cluster 2021-03-24T21:53:37.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:37 smithi125 conmon[24927]: [DBG] purged_snaps scrub ok 2021-03-24T21:53:37.925 INFO:teuthology.orchestra.run.smithi125.stdout:167503724579 2021-03-24T21:53:38.805 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:38 smithi125 conmon[19936]: cluster 2021-03-24T21:53:36.682885+0000 mgr.y (mgr.14144) 277 : cluster [DBG] 2021-03-24T21:53:38.805 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:38 smithi125 conmon[19936]: pgmap v277: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-03-24T21:53:38.806 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:38 smithi125 conmon[19936]: cluster 2021-03-24T21:53:37.334329+0000 osd.4 (osd.4) 25 : cluster [DBG] 4.9 scrub ok 2021-03-24T21:53:38.806 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:38 smithi125 conmon[19936]: cluster 2021-03-24T21:53: 2021-03-24T21:53:38.806 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:38 smithi125 conmon[19936]: 37.455697+0000 mon.a (mon.0) 601 : cluster [DBG] osdmap e92: 8 total, 8 up, 8 in 2021-03-24T21:53:38.806 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:38 smithi125 conmon[19936]: audit 2021-03-24T21: 2021-03-24T21:53:38.807 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:38 smithi125 conmon[19936]: 53:37.925620+0000 mon.a (mon.0) 602 : audit [DBG] from='client.? v1:172.21.15.125:0/1185712925' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2021-03-24T21:53:38.807 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:38 smithi125 conmon[24927]: cluster 2021-03-24T21:53 2021-03-24T21:53:38.807 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:38 smithi125 conmon[24927]: :36.682885+0000 mgr.y (mgr.14144) 277 : cluster [DBG] pgmap v277: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-03-24T21:53:38.808 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:38 smithi125 conmon[24927]: cluster 2021-03-24T21:53:38.808 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:38 smithi125 conmon[24927]: 2021-03-24T21:53:37.334329+0000 osd.4 (osd 2021-03-24T21:53:38.808 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:38 smithi125 conmon[24927]: .4) 25 : cluster [DBG] 4.9 scrub ok 2021-03-24T21:53:38.808 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:38 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:53:38.809 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:38 smithi125 conmon[24927]: -24T21:53:37.455697+0000 mon.a 2021-03-24T21:53:38.809 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:38 smithi125 conmon[24927]: (mon.0) 601 : cluster [DBG] osdmap e92: 8 total, 8 up, 8 in 2021-03-24T21:53:38.809 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:38 smithi125 conmon[24927]: 2021-03-24T21:53:38.809 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:38 smithi125 conmon[24927]: audit 2021-03-24T21:53:38.809 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:38 smithi125 conmon[24927]: 2021-03-24T21:53:37.925620+0000 mon.a 2021-03-24T21:53:38.810 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:38 smithi125 conmon[24927]: (mon.0) 602 : audit [DBG] from='client.? v1:172.21.15.125:0/1185712925' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2021-03-24T21:53:38.810 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:38 smithi125 conmon[24927]: 2021-03-24T21:53:38.835 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724574 got 167503724579 for osd.5 2021-03-24T21:53:38.836 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd last-stat-seq osd.6 2021-03-24T21:53:38.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:38 smithi142 conmon[20125]: cluster 2021-03-24T21:53:36.682885+0000 mgr.y (mgr.14144) 277 : cluster [DBG] 2021-03-24T21:53:38.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:38 smithi142 conmon[20125]: pgmap v277: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-03-24T21:53:38.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:38 smithi142 conmon[20125]: cluster 2021-03-24T21:53:37.334329+0000 osd.4 (osd.4) 25 : 2021-03-24T21:53:38.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:38 smithi142 conmon[20125]: cluster [DBG] 4.9 scrub ok 2021-03-24T21:53:38.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:38 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:53:38.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:38 smithi142 conmon[20125]: -24T21:53:37.455697+0000 mon.a (mon.0 2021-03-24T21:53:38.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:38 smithi142 conmon[20125]: ) 601 : cluster [DBG] osdmap e92: 8 total, 8 up, 8 in 2021-03-24T21:53:38.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:38 smithi142 conmon[20125]: audit 2021-03-24 2021-03-24T21:53:38.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:38 smithi142 conmon[20125]: T21:53:37.925620+0000 mon.a ( 2021-03-24T21:53:38.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:38 smithi142 conmon[20125]: mon.0) 602 : audit [DBG] from='client.? v1:172.21.15.125:0/1185712925' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2021-03-24T21:53:39.162 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:53:40.730 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:40 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:53:40.730 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:40 smithi125 conmon[19936]: 21:53:38.683834+0000 mgr.y (mgr.14144) 278 : cluster [DBG] pgmap v279: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2021-03-24T21:53:40.730 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:40 smithi125 conmon[19936]: cluster 2021-03-24T21:53:40.730 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:40 smithi125 conmon[19936]: 2021-03- 2021-03-24T21:53:40.731 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:40 smithi125 conmon[19936]: 24T21:53:39.277416+0000 2021-03-24T21:53:40.731 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:40 smithi125 conmon[19936]: osd.4 (osd.4) 26 : 2021-03-24T21:53:40.731 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:40 smithi125 conmon[19936]: cluster [DBG] 6.1e scrub ok 2021-03-24T21:53:40.732 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:40 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:53:40.732 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:40 smithi125 conmon[24927]: 24T21:53:38.683834+0000 mgr.y (mgr.14144) 278 : 2021-03-24T21:53:40.732 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:40 smithi125 conmon[24927]: cluster [DBG] pgmap v279: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2021-03-24T21:53:40.732 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:40 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:53:40.732 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:40 smithi125 conmon[24927]: 03-24T21:53:39.277416 2021-03-24T21:53:40.733 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:40 smithi125 conmon[24927]: +0000 osd.4 (osd.4) 2021-03-24T21:53:40.733 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:40 smithi125 conmon[24927]: 26 : cluster [DBG] 6.1e scrub ok 2021-03-24T21:53:40.733 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:40 smithi125 conmon[24927]: 2021-03-24T21:53:40.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:40 smithi142 conmon[20125]: cluster 2021-03-24T21:53:38.683834+0000 mgr.y (mgr.14144) 278 : cluster 2021-03-24T21:53:40.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:40 smithi142 conmon[20125]: [DBG] pgmap v279: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2021-03-24T21:53:40.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:40 smithi142 conmon[20125]: cluster 2021-03-24T21:53:39.277416+0000 osd.4 (osd.4) 26 : 2021-03-24T21:53:40.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:40 smithi142 conmon[20125]: cluster [DBG] 6.1e scrub ok 2021-03-24T21:53:41.122 INFO:teuthology.orchestra.run.smithi125.stdout:201863462943 2021-03-24T21:53:41.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:41 smithi125 conmon[19936]: cluster 2021-03-24T21:53: 2021-03-24T21:53:41.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:41 smithi125 conmon[19936]: 40.289975+0000 osd.4 (osd.4) 27 : cluster [DBG] 6.e scrub ok 2021-03-24T21:53:41.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:41 smithi125 conmon[19936]: audit 2021-03-24T21:53:41.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:41 smithi125 conmon[19936]: 2021-03-24T21:53:41.122775+0000 mon.c (mon.2) 18 : audit 2021-03-24T21:53:41.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:41 smithi125 conmon[19936]: [DBG] from='client.? v1:172.21.15.125:0/1713460877' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2021-03-24T21:53:41.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:41 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:53:41.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:41 smithi125 conmon[24927]: 53:40.289975+0000 osd.4 (osd.4) 27 : cluster [DBG] 6.e scrub ok 2021-03-24T21:53:41.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:41 smithi125 conmon[24927]: audit 2021-03-24T21 2021-03-24T21:53:41.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:41 smithi125 conmon[24927]: :53:41.122775+0000 mon.c 2021-03-24T21:53:41.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:41 smithi125 conmon[24927]: (mon.2) 18 : audit [DBG] from='client.? v1:172.21.15.125:0/1713460877' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2021-03-24T21:53:41.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:41 smithi125 conmon[24927]: 2021-03-24T21:53:41.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:41 smithi142 conmon[20125]: cluster 2021-03-24T21:53:40.289975+0000 osd.4 (osd.4) 27 : cluster [DBG] 6.e scrub ok 2021-03-24T21:53:41.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:41 smithi142 conmon[20125]: audit 2021-03-24T21:53:41.122775+0000 mon.c (mon.2) 18 : audit [DBG] from='client.? v1:172.21.15.125:0/1713460877' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2021-03-24T21:53:41.993 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462938 got 201863462943 for osd.6 2021-03-24T21:53:41.994 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd last-stat-seq osd.7 2021-03-24T21:53:42.301 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:53:42.468 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:42 smithi125 conmon[24927]: cluster 2021-03-24T21:53:40. 2021-03-24T21:53:42.468 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:42 smithi125 conmon[19936]: cluster 2021-03-24T21:53: 2021-03-24T21:53:42.468 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:42 smithi125 conmon[19936]: 40.685012+0000 mgr.y (mgr.14144) 279 : cluster [DBG] pgmap v280: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 994 B/s rd, 0 op/s 2021-03-24T21:53:42.469 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:42 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T21:53:42.469 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:42 smithi125 conmon[19936]: T21:53:41.295859+0000 2021-03-24T21:53:42.469 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:42 smithi125 conmon[19936]: osd.4 (osd.4) 28 : cluster 2021-03-24T21:53:42.469 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:42 smithi125 conmon[19936]: [DBG] 4.18 scrub ok 2021-03-24T21:53:42.748 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:42 smithi125 conmon[24927]: 685012+0000 mgr.y (mgr.14144) 279 : cluster [DBG] pgmap v280: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 994 B/s rd, 0 op/s 2021-03-24T21:53:42.749 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:42 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:53:42.749 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:42 smithi125 conmon[24927]: 53:41.295859+0000 osd.4 (osd.4) 28 : cluster [DBG] 4.18 scrub ok 2021-03-24T21:53:42.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:42 smithi142 conmon[20125]: cluster 2021-03-24T21:53:40.685012+0000 mgr.y (mgr.14144) 279 : cluster 2021-03-24T21:53:42.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:42 smithi142 conmon[20125]: [DBG] pgmap v280: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 994 B/s rd, 0 op/s 2021-03-24T21:53:42.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:42 smithi142 conmon[20125]: cluster 2021-03-24T21:53:41.295859+0000 osd.4 (osd.4 2021-03-24T21:53:42.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:42 smithi142 conmon[20125]: ) 28 : cluster [DBG] 4.18 scrub ok 2021-03-24T21:53:43.770 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:43 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:53:43.771 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:43 smithi125 conmon[24927]: 53:42.611529+0000 osd.3 (osd.3) 30 : cluster [DBG] 4.1d scrub ok 2021-03-24T21:53:43.771 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:43 smithi125 conmon[19936]: cluster 2021-03-24T21:53 2021-03-24T21:53:43.772 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:43 smithi125 conmon[19936]: :42.611529+0000 osd.3 (osd.3) 30 : cluster [DBG] 4.1d scrub ok 2021-03-24T21:53:43.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:43 smithi142 conmon[20125]: cluster 2021-03-24T21:53: 2021-03-24T21:53:43.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:43 smithi142 conmon[20125]: 42.611529+0000 osd.3 (osd.3) 30 : cluster [DBG] 4.1d scrub ok 2021-03-24T21:53:44.200 INFO:teuthology.orchestra.run.smithi125.stdout:236223201307 2021-03-24T21:53:44.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:44 smithi125 conmon[19936]: cluster 2021-03-24T21:53:42 2021-03-24T21:53:44.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:44 smithi125 conmon[19936]: .685750+0000 mgr.y (mgr.14144) 280 : cluster [DBG] pgmap v281: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2021-03-24T21:53:44.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:44 smithi125 conmon[19936]: audit 2021-03 2021-03-24T21:53:44.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:44 smithi125 conmon[19936]: -24T21:53:44.200742+0000 mon.a (mon.0) 603 : audit [DBG] from='client.? v1:172.21.15.125:0/3357942926' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2021-03-24T21:53:44.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:44 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:53:44.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:44 smithi125 conmon[24927]: 53:42.685750+0000 mgr.y (mgr.14144) 280 : cluster [DBG] pgmap v281: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2021-03-24T21:53:44.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:44 smithi125 conmon[24927]: audit 2021-03-24T21 2021-03-24T21:53:44.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:44 smithi125 conmon[24927]: :53:44.200742+0000 mon.a (mon. 2021-03-24T21:53:44.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:44 smithi125 conmon[24927]: 0) 603 : audit [DBG] from='client.? v1:172.21.15.125:0/3357942926' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2021-03-24T21:53:44.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:44 smithi142 conmon[20125]: cluster 2021-03-24T21:53:42.685750+0000 mgr.y (mgr.14144) 280 : cluster 2021-03-24T21:53:44.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:44 smithi142 conmon[20125]: [DBG] pgmap v281: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2021-03-24T21:53:44.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:44 smithi142 conmon[20125]: audit 2021-03-24T21:53:44.200742+0000 mon.a (mon.0) 603 : audit 2021-03-24T21:53:44.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:44 smithi142 conmon[20125]: [DBG] from='client.? v1:172.21.15.125:0/3357942926' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2021-03-24T21:53:45.015 INFO:tasks.cephadm.ceph_manager.ceph:need seq 236223201302 got 236223201307 for osd.7 2021-03-24T21:53:45.016 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2021-03-24T21:53:45.016 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph pg dump --format=json 2021-03-24T21:53:45.314 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:53:46.159 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:46 smithi125 conmon[24927]: cluster 2021-03-24T21:53 2021-03-24T21:53:46.160 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:46 smithi125 conmon[24927]: :44.657749+0000 osd.3 (osd.3) 31 : cluster [DBG] 6.1f scrub ok 2021-03-24T21:53:46.160 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:46 smithi125 conmon[24927]: cluster 2021-03-24T21:53:46.160 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:46 smithi125 conmon[24927]: 2021-03-24T21:53:44.686839+0000 mgr.y (mgr. 2021-03-24T21:53:46.160 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:46 smithi125 conmon[24927]: 14144) 281 : cluster [DBG] pgmap v282: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-24T21:53:46.161 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:46 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:53:46.161 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:46 smithi125 conmon[19936]: 03-24T21:53:44.657749+0000 osd.3 (osd.3) 31 : cluster [DBG] 6.1f scrub ok 2021-03-24T21:53:46.161 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:46 smithi125 conmon[19936]: cluster 2021-03-24T21:53:46.161 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:46 smithi125 conmon[19936]: 2021-03-24T21:53:44.686839 2021-03-24T21:53:46.162 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:46 smithi125 conmon[19936]: +0000 mgr.y (mgr.14144) 281 : cluster [DBG] pgmap v282: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-24T21:53:46.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:46 smithi142 conmon[20125]: cluster 2021-03-24T21:53:44.657749+0000 osd.3 (osd.3) 31 : cluster 2021-03-24T21:53:46.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:46 smithi142 conmon[20125]: [DBG] 6.1f scrub ok 2021-03-24T21:53:46.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:46 smithi142 conmon[20125]: cluster 2021-03-24T21:53:44.686839+0000 mgr.y (mgr.14144) 281 : 2021-03-24T21:53:46.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:46 smithi142 conmon[20125]: cluster [DBG] pgmap v282: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-24T21:53:47.156 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-24T21:53:47.158 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2021-03-24T21:53:47.331 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:47 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:53:47.331 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:47 smithi125 conmon[19936]: :53:45.349766+0000 osd.4 (osd.4) 29 : cluster [DBG] 6.2 scrub ok 2021-03-24T21:53:47.332 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:47 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:53:47.332 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:47 smithi125 conmon[24927]: 53:45.349766+0000 osd.4 (osd.4) 29 : cluster [DBG] 6.2 scrub ok 2021-03-24T21:53:47.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:47 smithi142 conmon[20125]: cluster 2021-03-24T21:53:45.349766+0000 osd.4 (osd.4) 29 : cluster 2021-03-24T21:53:47.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:47 smithi142 conmon[20125]: [DBG] 6.2 scrub ok 2021-03-24T21:53:48.020 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":283,"stamp":"2021-03-24T21:53:46.687162+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":25236,"num_objects":316,"num_object_clones":0,"num_object_copies":948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":316,"num_whiteouts":0,"num_read":1191,"num_read_kb":691,"num_write":550,"num_write_kb":165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":2412,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":570,"ondisk_log_size":570,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":53328,"kb_used_data":6864,"kb_used_omap":0,"kb_used_meta":46400,"kb_avail":749907888,"statfs":{"total":767960285184,"available":767905677312,"internally_reserved":0,"allocated":7028736,"data_stored":2529507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":47513600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004839"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":"38","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.421899+0000","last_change":"2021-03-24T21:52:59.180417+0000","last_active":"2021-03-24T21:53:31.421899+0000","last_peered":"2021-03-24T21:53:31.421899+0000","last_clean":"2021-03-24T21:53:31.421899+0000","last_became_active":"2021-03-24T21:51:51.759485+0000","last_became_peered":"2021-03-24T21:51:51.759485+0000","last_unstale":"2021-03-24T21:53:31.421899+0000","last_undegraded":"2021-03-24T21:53:31.421899+0000","last_fullsized":"2021-03-24T21:53:31.421899+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:59.180240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:52:59.180240+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"41","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.425115+0000","last_change":"2021-03-24T21:53:02.553145+0000","last_active":"2021-03-24T21:53:31.425115+0000","last_peered":"2021-03-24T21:53:31.425115+0000","last_clean":"2021-03-24T21:53:31.425115+0000","last_became_active":"2021-03-24T21:51:49.415254+0000","last_became_peered":"2021-03-24T21:51:49.415254+0000","last_unstale":"2021-03-24T21:53:31.425115+0000","last_undegraded":"2021-03-24T21:53:31.425115+0000","last_fullsized":"2021-03-24T21:53:31.425115+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:02.553036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:02.553036+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":"43","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.651528+0000","last_change":"2021-03-24T21:53:12.428679+0000","last_active":"2021-03-24T21:53:31.651528+0000","last_peered":"2021-03-24T21:53:31.651528+0000","last_clean":"2021-03-24T21:53:31.651528+0000","last_became_active":"2021-03-24T21:51:45.581000+0000","last_became_peered":"2021-03-24T21:51:45.581000+0000","last_unstale":"2021-03-24T21:53:31.651528+0000","last_undegraded":"2021-03-24T21:53:31.651528+0000","last_fullsized":"2021-03-24T21:53:31.651528+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:12.428497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:12.428497+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"69'15","reported_seq":"70","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:41.296073+0000","last_change":"2021-03-24T21:53:41.296073+0000","last_active":"2021-03-24T21:53:41.296073+0000","last_peered":"2021-03-24T21:53:41.296073+0000","last_clean":"2021-03-24T21:53:41.296073+0000","last_became_active":"2021-03-24T21:51:47.410353+0000","last_became_peered":"2021-03-24T21:51:47.410353+0000","last_unstale":"2021-03-24T21:53:41.296073+0000","last_undegraded":"2021-03-24T21:53:41.296073+0000","last_fullsized":"2021-03-24T21:53:41.296073+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:53:41.295875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:41.295875+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"38","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.664833+0000","last_change":"2021-03-24T21:52:58.676144+0000","last_active":"2021-03-24T21:53:31.664833+0000","last_peered":"2021-03-24T21:53:31.664833+0000","last_clean":"2021-03-24T21:53:31.664833+0000","last_became_active":"2021-03-24T21:51:51.915108+0000","last_became_peered":"2021-03-24T21:51:51.915108+0000","last_unstale":"2021-03-24T21:53:31.664833+0000","last_undegraded":"2021-03-24T21:53:31.664833+0000","last_fullsized":"2021-03-24T21:53:31.664833+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:58.676060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:52:58.676060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"35","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.421623+0000","last_change":"2021-03-24T21:51:49.410454+0000","last_active":"2021-03-24T21:53:31.421623+0000","last_peered":"2021-03-24T21:53:31.421623+0000","last_clean":"2021-03-24T21:53:31.421623+0000","last_became_active":"2021-03-24T21:51:49.409015+0000","last_became_peered":"2021-03-24T21:51:49.409015+0000","last_unstale":"2021-03-24T21:53:31.421623+0000","last_undegraded":"2021-03-24T21:53:31.421623+0000","last_fullsized":"2021-03-24T21:53:31.421623+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:51:48.384232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":"44","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.665525+0000","last_change":"2021-03-24T21:53:18.706895+0000","last_active":"2021-03-24T21:53:31.665525+0000","last_peered":"2021-03-24T21:53:31.665525+0000","last_clean":"2021-03-24T21:53:31.665525+0000","last_became_active":"2021-03-24T21:51:45.581465+0000","last_became_peered":"2021-03-24T21:51:45.581465+0000","last_unstale":"2021-03-24T21:53:31.665525+0000","last_undegraded":"2021-03-24T21:53:31.665525+0000","last_fullsized":"2021-03-24T21:53:31.665525+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:18.706745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:18.706745+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"69'20","reported_seq":"77","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.664691+0000","last_change":"2021-03-24T21:53:12.604981+0000","last_active":"2021-03-24T21:53:31.664691+0000","last_peered":"2021-03-24T21:53:31.664691+0000","last_clean":"2021-03-24T21:53:31.664691+0000","last_became_active":"2021-03-24T21:51:47.405776+0000","last_became_peered":"2021-03-24T21:51:47.405776+0000","last_unstale":"2021-03-24T21:53:31.664691+0000","last_undegraded":"2021-03-24T21:53:31.664691+0000","last_fullsized":"2021-03-24T21:53:31.664691+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'20","last_scrub_stamp":"2021-03-24T21:53:12.604757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:12.604757+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":22,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"37","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.651120+0000","last_change":"2021-03-24T21:53:27.604422+0000","last_active":"2021-03-24T21:53:31.651120+0000","last_peered":"2021-03-24T21:53:31.651120+0000","last_clean":"2021-03-24T21:53:31.651120+0000","last_became_active":"2021-03-24T21:51:51.758026+0000","last_became_peered":"2021-03-24T21:51:51.758026+0000","last_unstale":"2021-03-24T21:53:31.651120+0000","last_undegraded":"2021-03-24T21:53:31.651120+0000","last_fullsized":"2021-03-24T21:53:31.651120+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:27.604230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:27.604230+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"41","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.873310+0000","last_change":"2021-03-24T21:52:56.855476+0000","last_active":"2021-03-24T21:53:31.873310+0000","last_peered":"2021-03-24T21:53:31.873310+0000","last_clean":"2021-03-24T21:53:31.873310+0000","last_became_active":"2021-03-24T21:51:49.413342+0000","last_became_peered":"2021-03-24T21:51:49.413342+0000","last_unstale":"2021-03-24T21:53:31.873310+0000","last_undegraded":"2021-03-24T21:53:31.873310+0000","last_fullsized":"2021-03-24T21:53:31.873310+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:56.855305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:52:56.855305+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":"45","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.873823+0000","last_change":"2021-03-24T21:52:50.788792+0000","last_active":"2021-03-24T21:53:31.873823+0000","last_peered":"2021-03-24T21:53:31.873823+0000","last_clean":"2021-03-24T21:53:31.873823+0000","last_became_active":"2021-03-24T21:51:45.382348+0000","last_became_peered":"2021-03-24T21:51:45.382348+0000","last_unstale":"2021-03-24T21:53:31.873823+0000","last_undegraded":"2021-03-24T21:53:31.873823+0000","last_fullsized":"2021-03-24T21:53:31.873823+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:50.788606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:52:50.788606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"69'13","reported_seq":"66","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.422100+0000","last_change":"2021-03-24T21:53:10.104959+0000","last_active":"2021-03-24T21:53:31.422100+0000","last_peered":"2021-03-24T21:53:31.422100+0000","last_clean":"2021-03-24T21:53:31.422100+0000","last_became_active":"2021-03-24T21:51:47.407488+0000","last_became_peered":"2021-03-24T21:51:47.407488+0000","last_unstale":"2021-03-24T21:53:31.422100+0000","last_undegraded":"2021-03-24T21:53:31.422100+0000","last_fullsized":"2021-03-24T21:53:31.422100+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2021-03-24T21:53:10.104806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:10.104806+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"39","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.872795+0000","last_change":"2021-03-24T21:53:23.896132+0000","last_active":"2021-03-24T21:53:31.872795+0000","last_peered":"2021-03-24T21:53:31.872795+0000","last_clean":"2021-03-24T21:53:31.872795+0000","last_became_active":"2021-03-24T21:51:51.765338+0000","last_became_peered":"2021-03-24T21:51:51.765338+0000","last_unstale":"2021-03-24T21:53:31.872795+0000","last_undegraded":"2021-03-24T21:53:31.872795+0000","last_fullsized":"2021-03-24T21:53:31.872795+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:23.895966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:23.895966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"39","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.897282+0000","last_change":"2021-03-24T21:53:00.845488+0000","last_active":"2021-03-24T21:53:31.897282+0000","last_peered":"2021-03-24T21:53:31.897282+0000","last_clean":"2021-03-24T21:53:31.897282+0000","last_became_active":"2021-03-24T21:51:49.411327+0000","last_became_peered":"2021-03-24T21:51:49.411327+0000","last_unstale":"2021-03-24T21:53:31.897282+0000","last_undegraded":"2021-03-24T21:53:31.897282+0000","last_fullsized":"2021-03-24T21:53:31.897282+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:00.845338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:00.845338+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":"45","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.873275+0000","last_change":"2021-03-24T21:52:45.798051+0000","last_active":"2021-03-24T21:53:31.873275+0000","last_peered":"2021-03-24T21:53:31.873275+0000","last_clean":"2021-03-24T21:53:31.873275+0000","last_became_active":"2021-03-24T21:51:45.381362+0000","last_became_peered":"2021-03-24T21:51:45.381362+0000","last_unstale":"2021-03-24T21:53:31.873275+0000","last_undegraded":"2021-03-24T21:53:31.873275+0000","last_fullsized":"2021-03-24T21:53:31.873275+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:45.797873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:52:45.797873+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"91'9","reported_seq":"57","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:33.290258+0000","last_change":"2021-03-24T21:53:33.290178+0000","last_active":"2021-03-24T21:53:33.290258+0000","last_peered":"2021-03-24T21:53:33.290258+0000","last_clean":"2021-03-24T21:53:33.290258+0000","last_became_active":"2021-03-24T21:51:47.416379+0000","last_became_peered":"2021-03-24T21:51:47.416379+0000","last_unstale":"2021-03-24T21:53:33.290258+0000","last_undegraded":"2021-03-24T21:53:33.290258+0000","last_fullsized":"2021-03-24T21:53:33.290258+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2021-03-24T21:53:33.290000+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2021-03-24T21:53:33.290000+0000","last_clean_scrub_stamp":"2021-03-24T21:53:33.290000+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":8,"num_write":7,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"38","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:39.277601+0000","last_change":"2021-03-24T21:53:39.277601+0000","last_active":"2021-03-24T21:53:39.277601+0000","last_peered":"2021-03-24T21:53:39.277601+0000","last_clean":"2021-03-24T21:53:39.277601+0000","last_became_active":"2021-03-24T21:51:51.910950+0000","last_became_peered":"2021-03-24T21:51:51.910950+0000","last_unstale":"2021-03-24T21:53:39.277601+0000","last_undegraded":"2021-03-24T21:53:39.277601+0000","last_fullsized":"2021-03-24T21:53:39.277601+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:39.277425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:39.277425+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"41","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.425640+0000","last_change":"2021-03-24T21:52:59.579706+0000","last_active":"2021-03-24T21:53:31.425640+0000","last_peered":"2021-03-24T21:53:31.425640+0000","last_clean":"2021-03-24T21:53:31.425640+0000","last_became_active":"2021-03-24T21:51:49.411914+0000","last_became_peered":"2021-03-24T21:51:49.411914+0000","last_unstale":"2021-03-24T21:53:31.425640+0000","last_undegraded":"2021-03-24T21:53:31.425640+0000","last_fullsized":"2021-03-24T21:53:31.425640+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:59.579528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:52:59.579528+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":"43","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.651619+0000","last_change":"2021-03-24T21:53:04.263170+0000","last_active":"2021-03-24T21:53:31.651619+0000","last_peered":"2021-03-24T21:53:31.651619+0000","last_clean":"2021-03-24T21:53:31.651619+0000","last_became_active":"2021-03-24T21:51:45.381533+0000","last_became_peered":"2021-03-24T21:51:45.381533+0000","last_unstale":"2021-03-24T21:53:31.651619+0000","last_undegraded":"2021-03-24T21:53:31.651619+0000","last_fullsized":"2021-03-24T21:53:31.651619+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:04.262989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:04.262989+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"69'15","reported_seq":"93","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.423314+0000","last_change":"2021-03-24T21:52:49.554069+0000","last_active":"2021-03-24T21:53:31.423314+0000","last_peered":"2021-03-24T21:53:31.423314+0000","last_clean":"2021-03-24T21:53:31.423314+0000","last_became_active":"2021-03-24T21:52:06.056116+0000","last_became_peered":"2021-03-24T21:52:06.056116+0000","last_unstale":"2021-03-24T21:53:31.423314+0000","last_undegraded":"2021-03-24T21:53:31.423314+0000","last_fullsized":"2021-03-24T21:53:31.423314+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:52:49.553951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:52:49.553951+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"33","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.665181+0000","last_change":"2021-03-24T21:51:51.915774+0000","last_active":"2021-03-24T21:53:31.665181+0000","last_peered":"2021-03-24T21:53:31.665181+0000","last_clean":"2021-03-24T21:53:31.665181+0000","last_became_active":"2021-03-24T21:51:51.914955+0000","last_became_peered":"2021-03-24T21:51:51.914955+0000","last_unstale":"2021-03-24T21:53:31.665181+0000","last_undegraded":"2021-03-24T21:53:31.665181+0000","last_fullsized":"2021-03-24T21:53:31.665181+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:51:50.731137+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"40","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.422406+0000","last_change":"2021-03-24T21:53:27.238846+0000","last_active":"2021-03-24T21:53:31.422406+0000","last_peered":"2021-03-24T21:53:31.422406+0000","last_clean":"2021-03-24T21:53:31.422406+0000","last_became_active":"2021-03-24T21:51:49.410920+0000","last_became_peered":"2021-03-24T21:51:49.410920+0000","last_unstale":"2021-03-24T21:53:31.422406+0000","last_undegraded":"2021-03-24T21:53:31.422406+0000","last_fullsized":"2021-03-24T21:53:31.422406+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:27.238614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:27.238614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":"44","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.422173+0000","last_change":"2021-03-24T21:52:54.212593+0000","last_active":"2021-03-24T21:53:31.422173+0000","last_peered":"2021-03-24T21:53:31.422173+0000","last_clean":"2021-03-24T21:53:31.422173+0000","last_became_active":"2021-03-24T21:51:45.579326+0000","last_became_peered":"2021-03-24T21:51:45.579326+0000","last_unstale":"2021-03-24T21:53:31.422173+0000","last_undegraded":"2021-03-24T21:53:31.422173+0000","last_fullsized":"2021-03-24T21:53:31.422173+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:54.212459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:52:54.212459+0000","last_clean_scrub_stamp":"2021-03-24T21:52:54.212459+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"69'16","reported_seq":"73","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:42.611713+0000","last_change":"2021-03-24T21:53:42.611713+0000","last_active":"2021-03-24T21:53:42.611713+0000","last_peered":"2021-03-24T21:53:42.611713+0000","last_clean":"2021-03-24T21:53:42.611713+0000","last_became_active":"2021-03-24T21:51:47.411418+0000","last_became_peered":"2021-03-24T21:51:47.411418+0000","last_unstale":"2021-03-24T21:53:42.611713+0000","last_undegraded":"2021-03-24T21:53:42.611713+0000","last_fullsized":"2021-03-24T21:53:42.611713+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T21:53:42.611540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:42.611540+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":37,"num_read_kb":20,"num_write":18,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"37","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.897890+0000","last_change":"2021-03-24T21:53:05.847583+0000","last_active":"2021-03-24T21:53:31.897890+0000","last_peered":"2021-03-24T21:53:31.897890+0000","last_clean":"2021-03-24T21:53:31.897890+0000","last_became_active":"2021-03-24T21:51:51.761492+0000","last_became_peered":"2021-03-24T21:51:51.761492+0000","last_unstale":"2021-03-24T21:53:31.897890+0000","last_undegraded":"2021-03-24T21:53:31.897890+0000","last_fullsized":"2021-03-24T21:53:31.897890+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:05.847412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:05.847412+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"65'2","reported_seq":"65","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:38.805577+0000","last_change":"2021-03-24T21:52:59.074600+0000","last_active":"2021-03-24T21:53:38.805577+0000","last_peered":"2021-03-24T21:53:38.805577+0000","last_clean":"2021-03-24T21:53:38.805577+0000","last_became_active":"2021-03-24T21:51:49.419519+0000","last_became_peered":"2021-03-24T21:51:49.419519+0000","last_unstale":"2021-03-24T21:53:38.805577+0000","last_undegraded":"2021-03-24T21:53:38.805577+0000","last_fullsized":"2021-03-24T21:53:38.805577+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2021-03-24T21:52:59.074417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:52:59.074417+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":"45","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.425196+0000","last_change":"2021-03-24T21:53:09.741058+0000","last_active":"2021-03-24T21:53:31.425196+0000","last_peered":"2021-03-24T21:53:31.425196+0000","last_clean":"2021-03-24T21:53:31.425196+0000","last_became_active":"2021-03-24T21:51:45.374975+0000","last_became_peered":"2021-03-24T21:51:45.374975+0000","last_unstale":"2021-03-24T21:53:31.425196+0000","last_undegraded":"2021-03-24T21:53:31.425196+0000","last_fullsized":"2021-03-24T21:53:31.425196+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:09.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:09.740867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"89'27","reported_seq":"79","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.652282+0000","last_change":"2021-03-24T21:53:21.585108+0000","last_active":"2021-03-24T21:53:31.652282+0000","last_peered":"2021-03-24T21:53:31.652282+0000","last_clean":"2021-03-24T21:53:31.652282+0000","last_became_active":"2021-03-24T21:51:47.688995+0000","last_became_peered":"2021-03-24T21:51:47.688995+0000","last_unstale":"2021-03-24T21:53:31.652282+0000","last_undegraded":"2021-03-24T21:53:31.652282+0000","last_fullsized":"2021-03-24T21:53:31.652282+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'27","last_scrub_stamp":"2021-03-24T21:53:21.584999+0000","last_deep_scrub":"89'27","last_deep_scrub_stamp":"2021-03-24T21:53:21.584999+0000","last_clean_scrub_stamp":"2021-03-24T21:53:21.584999+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"39","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.425714+0000","last_change":"2021-03-24T21:53:14.700106+0000","last_active":"2021-03-24T21:53:31.425714+0000","last_peered":"2021-03-24T21:53:31.425714+0000","last_clean":"2021-03-24T21:53:31.425714+0000","last_became_active":"2021-03-24T21:51:51.758605+0000","last_became_peered":"2021-03-24T21:51:51.758605+0000","last_unstale":"2021-03-24T21:53:31.425714+0000","last_undegraded":"2021-03-24T21:53:31.425714+0000","last_fullsized":"2021-03-24T21:53:31.425714+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:14.699852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:14.699852+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"39","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.652614+0000","last_change":"2021-03-24T21:53:30.660884+0000","last_active":"2021-03-24T21:53:31.652614+0000","last_peered":"2021-03-24T21:53:31.652614+0000","last_clean":"2021-03-24T21:53:31.652614+0000","last_became_active":"2021-03-24T21:51:49.409745+0000","last_became_peered":"2021-03-24T21:51:49.409745+0000","last_unstale":"2021-03-24T21:53:31.652614+0000","last_undegraded":"2021-03-24T21:53:31.652614+0000","last_fullsized":"2021-03-24T21:53:31.652614+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:30.660636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:30.660636+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":"44","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.663842+0000","last_change":"2021-03-24T21:53:11.648668+0000","last_active":"2021-03-24T21:53:31.663842+0000","last_peered":"2021-03-24T21:53:31.663842+0000","last_clean":"2021-03-24T21:53:31.663842+0000","last_became_active":"2021-03-24T21:51:45.373294+0000","last_became_peered":"2021-03-24T21:51:45.373294+0000","last_unstale":"2021-03-24T21:53:31.663842+0000","last_undegraded":"2021-03-24T21:53:31.663842+0000","last_fullsized":"2021-03-24T21:53:31.663842+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:11.648470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:11.648470+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"69'17","reported_seq":"106","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.638810+0000","last_change":"2021-03-24T21:53:17.246451+0000","last_active":"2021-03-24T21:53:31.638810+0000","last_peered":"2021-03-24T21:53:31.638810+0000","last_clean":"2021-03-24T21:53:31.638810+0000","last_became_active":"2021-03-24T21:52:06.054746+0000","last_became_peered":"2021-03-24T21:52:06.054746+0000","last_unstale":"2021-03-24T21:53:31.638810+0000","last_undegraded":"2021-03-24T21:53:31.638810+0000","last_fullsized":"2021-03-24T21:53:31.638810+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'17","last_scrub_stamp":"2021-03-24T21:53:17.246367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:17.246367+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":38,"num_read_kb":19,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"37","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.897223+0000","last_change":"2021-03-24T21:53:27.877846+0000","last_active":"2021-03-24T21:53:31.897223+0000","last_peered":"2021-03-24T21:53:31.897223+0000","last_clean":"2021-03-24T21:53:31.897223+0000","last_became_active":"2021-03-24T21:51:51.762150+0000","last_became_peered":"2021-03-24T21:51:51.762150+0000","last_unstale":"2021-03-24T21:53:31.897223+0000","last_undegraded":"2021-03-24T21:53:31.897223+0000","last_fullsized":"2021-03-24T21:53:31.897223+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:27.877627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:27.877627+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"40","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.637439+0000","last_change":"2021-03-24T21:53:13.143455+0000","last_active":"2021-03-24T21:53:31.637439+0000","last_peered":"2021-03-24T21:53:31.637439+0000","last_clean":"2021-03-24T21:53:31.637439+0000","last_became_active":"2021-03-24T21:51:49.414221+0000","last_became_peered":"2021-03-24T21:51:49.414221+0000","last_unstale":"2021-03-24T21:53:31.637439+0000","last_undegraded":"2021-03-24T21:53:31.637439+0000","last_fullsized":"2021-03-24T21:53:31.637439+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:13.143327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:13.143327+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":"43","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.651552+0000","last_change":"2021-03-24T21:53:10.429942+0000","last_active":"2021-03-24T21:53:31.651552+0000","last_peered":"2021-03-24T21:53:31.651552+0000","last_clean":"2021-03-24T21:53:31.651552+0000","last_became_active":"2021-03-24T21:51:45.379789+0000","last_became_peered":"2021-03-24T21:51:45.379789+0000","last_unstale":"2021-03-24T21:53:31.651552+0000","last_undegraded":"2021-03-24T21:53:31.651552+0000","last_fullsized":"2021-03-24T21:53:31.651552+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:10.429783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:10.429783+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"69'12","reported_seq":"95","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.663955+0000","last_change":"2021-03-24T21:52:52.579983+0000","last_active":"2021-03-24T21:53:31.663955+0000","last_peered":"2021-03-24T21:53:31.663955+0000","last_clean":"2021-03-24T21:53:31.663955+0000","last_became_active":"2021-03-24T21:52:06.053964+0000","last_became_peered":"2021-03-24T21:52:06.053964+0000","last_unstale":"2021-03-24T21:53:31.663955+0000","last_undegraded":"2021-03-24T21:53:31.663955+0000","last_fullsized":"2021-03-24T21:53:31.663955+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2021-03-24T21:52:52.579796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:52:52.579796+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":12,"num_write":12,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"33","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.664451+0000","last_change":"2021-03-24T21:51:51.912859+0000","last_active":"2021-03-24T21:53:31.664451+0000","last_peered":"2021-03-24T21:53:31.664451+0000","last_clean":"2021-03-24T21:53:31.664451+0000","last_became_active":"2021-03-24T21:51:51.911828+0000","last_became_peered":"2021-03-24T21:51:51.911828+0000","last_unstale":"2021-03-24T21:53:31.664451+0000","last_undegraded":"2021-03-24T21:53:31.664451+0000","last_fullsized":"2021-03-24T21:53:31.664451+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:51:50.731137+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"39","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.897228+0000","last_change":"2021-03-24T21:52:52.829549+0000","last_active":"2021-03-24T21:53:31.897228+0000","last_peered":"2021-03-24T21:53:31.897228+0000","last_clean":"2021-03-24T21:53:31.897228+0000","last_became_active":"2021-03-24T21:51:49.422591+0000","last_became_peered":"2021-03-24T21:51:49.422591+0000","last_unstale":"2021-03-24T21:53:31.897228+0000","last_undegraded":"2021-03-24T21:53:31.897228+0000","last_fullsized":"2021-03-24T21:53:31.897228+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:52.829381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:52:52.829381+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":"45","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.873993+0000","last_change":"2021-03-24T21:53:21.833111+0000","last_active":"2021-03-24T21:53:31.873993+0000","last_peered":"2021-03-24T21:53:31.873993+0000","last_clean":"2021-03-24T21:53:31.873993+0000","last_became_active":"2021-03-24T21:51:45.385892+0000","last_became_peered":"2021-03-24T21:51:45.385892+0000","last_unstale":"2021-03-24T21:53:31.873993+0000","last_undegraded":"2021-03-24T21:53:31.873993+0000","last_fullsized":"2021-03-24T21:53:31.873993+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:21.832921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:21.832921+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"86'22","reported_seq":"75","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.664519+0000","last_change":"2021-03-24T21:52:54.597687+0000","last_active":"2021-03-24T21:53:31.664519+0000","last_peered":"2021-03-24T21:53:31.664519+0000","last_clean":"2021-03-24T21:53:31.664519+0000","last_became_active":"2021-03-24T21:51:47.687561+0000","last_became_peered":"2021-03-24T21:51:47.687561+0000","last_unstale":"2021-03-24T21:53:31.664519+0000","last_undegraded":"2021-03-24T21:53:31.664519+0000","last_fullsized":"2021-03-24T21:53:31.664519+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'22","last_scrub_stamp":"2021-03-24T21:52:54.597505+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2021-03-24T21:52:54.597505+0000","last_clean_scrub_stamp":"2021-03-24T21:52:54.597505+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":16,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"37","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.650465+0000","last_change":"2021-03-24T21:53:13.462014+0000","last_active":"2021-03-24T21:53:31.650465+0000","last_peered":"2021-03-24T21:53:31.650465+0000","last_clean":"2021-03-24T21:53:31.650465+0000","last_became_active":"2021-03-24T21:51:51.757253+0000","last_became_peered":"2021-03-24T21:51:51.757253+0000","last_unstale":"2021-03-24T21:53:31.650465+0000","last_undegraded":"2021-03-24T21:53:31.650465+0000","last_fullsized":"2021-03-24T21:53:31.650465+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:13.461940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:13.461940+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"40","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.665594+0000","last_change":"2021-03-24T21:53:31.665546+0000","last_active":"2021-03-24T21:53:31.665594+0000","last_peered":"2021-03-24T21:53:31.665594+0000","last_clean":"2021-03-24T21:53:31.665594+0000","last_became_active":"2021-03-24T21:51:49.418515+0000","last_became_peered":"2021-03-24T21:51:49.418515+0000","last_unstale":"2021-03-24T21:53:31.665594+0000","last_undegraded":"2021-03-24T21:53:31.665594+0000","last_fullsized":"2021-03-24T21:53:31.665594+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:31.665434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:31.665434+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":"43","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.898116+0000","last_change":"2021-03-24T21:53:08.841487+0000","last_active":"2021-03-24T21:53:31.898116+0000","last_peered":"2021-03-24T21:53:31.898116+0000","last_clean":"2021-03-24T21:53:31.898116+0000","last_became_active":"2021-03-24T21:51:45.379383+0000","last_became_peered":"2021-03-24T21:51:45.379383+0000","last_unstale":"2021-03-24T21:53:31.898116+0000","last_undegraded":"2021-03-24T21:53:31.898116+0000","last_fullsized":"2021-03-24T21:53:31.898116+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:08.841310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:08.841310+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"69'15","reported_seq":"71","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.425868+0000","last_change":"2021-03-24T21:53:04.637959+0000","last_active":"2021-03-24T21:53:31.425868+0000","last_peered":"2021-03-24T21:53:31.425868+0000","last_clean":"2021-03-24T21:53:31.425868+0000","last_became_active":"2021-03-24T21:51:47.403130+0000","last_became_peered":"2021-03-24T21:51:47.403130+0000","last_unstale":"2021-03-24T21:53:31.425868+0000","last_undegraded":"2021-03-24T21:53:31.425868+0000","last_fullsized":"2021-03-24T21:53:31.425868+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:53:04.637877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:04.637877+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"38","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.664068+0000","last_change":"2021-03-24T21:53:23.708000+0000","last_active":"2021-03-24T21:53:31.664068+0000","last_peered":"2021-03-24T21:53:31.664068+0000","last_clean":"2021-03-24T21:53:31.664068+0000","last_became_active":"2021-03-24T21:51:51.764871+0000","last_became_peered":"2021-03-24T21:51:51.764871+0000","last_unstale":"2021-03-24T21:53:31.664068+0000","last_undegraded":"2021-03-24T21:53:31.664068+0000","last_fullsized":"2021-03-24T21:53:31.664068+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:23.707914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:23.707914+0000","last_clean_scrub_stamp":"2021-03-24T21:53:23.707914+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"41","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.425768+0000","last_change":"2021-03-24T21:52:52.702414+0000","last_active":"2021-03-24T21:53:31.425768+0000","last_peered":"2021-03-24T21:53:31.425768+0000","last_clean":"2021-03-24T21:53:31.425768+0000","last_became_active":"2021-03-24T21:51:49.415559+0000","last_became_peered":"2021-03-24T21:51:49.415559+0000","last_unstale":"2021-03-24T21:53:31.425768+0000","last_undegraded":"2021-03-24T21:53:31.425768+0000","last_fullsized":"2021-03-24T21:53:31.425768+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:52.702305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:52:52.702305+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":"44","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:36.317204+0000","last_change":"2021-03-24T21:53:36.317121+0000","last_active":"2021-03-24T21:53:36.317204+0000","last_peered":"2021-03-24T21:53:36.317204+0000","last_clean":"2021-03-24T21:53:36.317204+0000","last_became_active":"2021-03-24T21:51:45.379862+0000","last_became_peered":"2021-03-24T21:51:45.379862+0000","last_unstale":"2021-03-24T21:53:36.317204+0000","last_undegraded":"2021-03-24T21:53:36.317204+0000","last_fullsized":"2021-03-24T21:53:36.317204+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:36.316893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:36.316893+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"69'15","reported_seq":"69","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.422789+0000","last_change":"2021-03-24T21:53:21.207831+0000","last_active":"2021-03-24T21:53:31.422789+0000","last_peered":"2021-03-24T21:53:31.422789+0000","last_clean":"2021-03-24T21:53:31.422789+0000","last_became_active":"2021-03-24T21:51:47.416319+0000","last_became_peered":"2021-03-24T21:51:47.416319+0000","last_unstale":"2021-03-24T21:53:31.422789+0000","last_undegraded":"2021-03-24T21:53:31.422789+0000","last_fullsized":"2021-03-24T21:53:31.422789+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:53:21.207696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:21.207696+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"37","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.651788+0000","last_change":"2021-03-24T21:53:06.309234+0000","last_active":"2021-03-24T21:53:31.651788+0000","last_peered":"2021-03-24T21:53:31.651788+0000","last_clean":"2021-03-24T21:53:31.651788+0000","last_became_active":"2021-03-24T21:51:51.756584+0000","last_became_peered":"2021-03-24T21:51:51.756584+0000","last_unstale":"2021-03-24T21:53:31.651788+0000","last_undegraded":"2021-03-24T21:53:31.651788+0000","last_fullsized":"2021-03-24T21:53:31.651788+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:06.309066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:06.309066+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"86'3","reported_seq":"66","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.647356+0000","last_change":"2021-03-24T21:53:08.815321+0000","last_active":"2021-03-24T21:53:37.647356+0000","last_peered":"2021-03-24T21:53:37.647356+0000","last_clean":"2021-03-24T21:53:37.647356+0000","last_became_active":"2021-03-24T21:51:49.419663+0000","last_became_peered":"2021-03-24T21:51:49.419663+0000","last_unstale":"2021-03-24T21:53:37.647356+0000","last_undegraded":"2021-03-24T21:53:37.647356+0000","last_fullsized":"2021-03-24T21:53:37.647356+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'3","last_scrub_stamp":"2021-03-24T21:53:08.815163+0000","last_deep_scrub":"86'3","last_deep_scrub_stamp":"2021-03-24T21:53:08.815163+0000","last_clean_scrub_stamp":"2021-03-24T21:53:08.815163+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":"43","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.897500+0000","last_change":"2021-03-24T21:53:15.868395+0000","last_active":"2021-03-24T21:53:31.897500+0000","last_peered":"2021-03-24T21:53:31.897500+0000","last_clean":"2021-03-24T21:53:31.897500+0000","last_became_active":"2021-03-24T21:51:45.579271+0000","last_became_peered":"2021-03-24T21:51:45.579271+0000","last_unstale":"2021-03-24T21:53:31.897500+0000","last_undegraded":"2021-03-24T21:53:31.897500+0000","last_fullsized":"2021-03-24T21:53:31.897500+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:15.868303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:15.868303+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"69'14","reported_seq":"65","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.665534+0000","last_change":"2021-03-24T21:53:02.723125+0000","last_active":"2021-03-24T21:53:31.665534+0000","last_peered":"2021-03-24T21:53:31.665534+0000","last_clean":"2021-03-24T21:53:31.665534+0000","last_became_active":"2021-03-24T21:51:47.686103+0000","last_became_peered":"2021-03-24T21:51:47.686103+0000","last_unstale":"2021-03-24T21:53:31.665534+0000","last_undegraded":"2021-03-24T21:53:31.665534+0000","last_fullsized":"2021-03-24T21:53:31.665534+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'14","last_scrub_stamp":"2021-03-24T21:53:02.722970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:02.722970+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":27,"num_read_kb":14,"num_write":14,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"38","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.423236+0000","last_change":"2021-03-24T21:53:12.030422+0000","last_active":"2021-03-24T21:53:31.423236+0000","last_peered":"2021-03-24T21:53:31.423236+0000","last_clean":"2021-03-24T21:53:31.423236+0000","last_became_active":"2021-03-24T21:51:51.762876+0000","last_became_peered":"2021-03-24T21:51:51.762876+0000","last_unstale":"2021-03-24T21:53:31.423236+0000","last_undegraded":"2021-03-24T21:53:31.423236+0000","last_fullsized":"2021-03-24T21:53:31.423236+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:12.030251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:12.030251+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"86'3","reported_seq":"66","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:38.805692+0000","last_change":"2021-03-24T21:52:56.678231+0000","last_active":"2021-03-24T21:53:38.805692+0000","last_peered":"2021-03-24T21:53:38.805692+0000","last_clean":"2021-03-24T21:53:38.805692+0000","last_became_active":"2021-03-24T21:51:49.419797+0000","last_became_peered":"2021-03-24T21:51:49.419797+0000","last_unstale":"2021-03-24T21:53:38.805692+0000","last_undegraded":"2021-03-24T21:53:38.805692+0000","last_fullsized":"2021-03-24T21:53:38.805692+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'3","last_scrub_stamp":"2021-03-24T21:52:56.678111+0000","last_deep_scrub":"86'3","last_deep_scrub_stamp":"2021-03-24T21:52:56.678111+0000","last_clean_scrub_stamp":"2021-03-24T21:52:56.678111+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":"43","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.650894+0000","last_change":"2021-03-24T21:52:51.450624+0000","last_active":"2021-03-24T21:53:31.650894+0000","last_peered":"2021-03-24T21:53:31.650894+0000","last_clean":"2021-03-24T21:53:31.650894+0000","last_became_active":"2021-03-24T21:51:45.381516+0000","last_became_peered":"2021-03-24T21:51:45.381516+0000","last_unstale":"2021-03-24T21:53:31.650894+0000","last_undegraded":"2021-03-24T21:53:31.650894+0000","last_fullsized":"2021-03-24T21:53:31.650894+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:51.450347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:52:51.450347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"69'13","reported_seq":"67","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.872410+0000","last_change":"2021-03-24T21:53:15.901347+0000","last_active":"2021-03-24T21:53:31.872410+0000","last_peered":"2021-03-24T21:53:31.872410+0000","last_clean":"2021-03-24T21:53:31.872410+0000","last_became_active":"2021-03-24T21:51:47.689288+0000","last_became_peered":"2021-03-24T21:51:47.689288+0000","last_unstale":"2021-03-24T21:53:31.872410+0000","last_undegraded":"2021-03-24T21:53:31.872410+0000","last_fullsized":"2021-03-24T21:53:31.872410+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2021-03-24T21:53:15.901150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:15.901150+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"39","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.423555+0000","last_change":"2021-03-24T21:53:15.623047+0000","last_active":"2021-03-24T21:53:31.423555+0000","last_peered":"2021-03-24T21:53:31.423555+0000","last_clean":"2021-03-24T21:53:31.423555+0000","last_became_active":"2021-03-24T21:51:51.754085+0000","last_became_peered":"2021-03-24T21:51:51.754085+0000","last_unstale":"2021-03-24T21:53:31.423555+0000","last_undegraded":"2021-03-24T21:53:31.423555+0000","last_fullsized":"2021-03-24T21:53:31.423555+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:15.622946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:15.622946+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"40","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.665904+0000","last_change":"2021-03-24T21:52:57.661001+0000","last_active":"2021-03-24T21:53:31.665904+0000","last_peered":"2021-03-24T21:53:31.665904+0000","last_clean":"2021-03-24T21:53:31.665904+0000","last_became_active":"2021-03-24T21:51:49.411882+0000","last_became_peered":"2021-03-24T21:51:49.411882+0000","last_unstale":"2021-03-24T21:53:31.665904+0000","last_undegraded":"2021-03-24T21:53:31.665904+0000","last_fullsized":"2021-03-24T21:53:31.665904+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:57.660827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:52:57.660827+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":"43","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.897188+0000","last_change":"2021-03-24T21:53:12.871443+0000","last_active":"2021-03-24T21:53:31.897188+0000","last_peered":"2021-03-24T21:53:31.897188+0000","last_clean":"2021-03-24T21:53:31.897188+0000","last_became_active":"2021-03-24T21:51:45.378937+0000","last_became_peered":"2021-03-24T21:51:45.378937+0000","last_unstale":"2021-03-24T21:53:31.897188+0000","last_undegraded":"2021-03-24T21:53:31.897188+0000","last_fullsized":"2021-03-24T21:53:31.897188+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:12.871269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:12.871269+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"69'16","reported_seq":"71","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.652435+0000","last_change":"2021-03-24T21:53:00.300847+0000","last_active":"2021-03-24T21:53:31.652435+0000","last_peered":"2021-03-24T21:53:31.652435+0000","last_clean":"2021-03-24T21:53:31.652435+0000","last_became_active":"2021-03-24T21:51:47.687595+0000","last_became_peered":"2021-03-24T21:51:47.687595+0000","last_unstale":"2021-03-24T21:53:31.652435+0000","last_undegraded":"2021-03-24T21:53:31.652435+0000","last_fullsized":"2021-03-24T21:53:31.652435+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T21:53:00.300673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:00.300673+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":38,"num_read_kb":18,"num_write":17,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"37","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.898110+0000","last_change":"2021-03-24T21:53:24.913418+0000","last_active":"2021-03-24T21:53:31.898110+0000","last_peered":"2021-03-24T21:53:31.898110+0000","last_clean":"2021-03-24T21:53:31.898110+0000","last_became_active":"2021-03-24T21:51:51.914615+0000","last_became_peered":"2021-03-24T21:51:51.914615+0000","last_unstale":"2021-03-24T21:53:31.898110+0000","last_undegraded":"2021-03-24T21:53:31.898110+0000","last_fullsized":"2021-03-24T21:53:31.898110+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:24.913238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:24.913238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"41","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.874139+0000","last_change":"2021-03-24T21:53:19.812985+0000","last_active":"2021-03-24T21:53:31.874139+0000","last_peered":"2021-03-24T21:53:31.874139+0000","last_clean":"2021-03-24T21:53:31.874139+0000","last_became_active":"2021-03-24T21:51:49.423159+0000","last_became_peered":"2021-03-24T21:51:49.423159+0000","last_unstale":"2021-03-24T21:53:31.874139+0000","last_undegraded":"2021-03-24T21:53:31.874139+0000","last_fullsized":"2021-03-24T21:53:31.874139+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:19.812822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:19.812822+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":"44","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.637365+0000","last_change":"2021-03-24T21:53:02.100432+0000","last_active":"2021-03-24T21:53:31.637365+0000","last_peered":"2021-03-24T21:53:31.637365+0000","last_clean":"2021-03-24T21:53:31.637365+0000","last_became_active":"2021-03-24T21:51:45.377984+0000","last_became_peered":"2021-03-24T21:51:45.377984+0000","last_unstale":"2021-03-24T21:53:31.637365+0000","last_undegraded":"2021-03-24T21:53:31.637365+0000","last_fullsized":"2021-03-24T21:53:31.637365+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:02.100243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:02.100243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"69'7","reported_seq":"53","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.665783+0000","last_change":"2021-03-24T21:53:13.646018+0000","last_active":"2021-03-24T21:53:31.665783+0000","last_peered":"2021-03-24T21:53:31.665783+0000","last_clean":"2021-03-24T21:53:31.665783+0000","last_became_active":"2021-03-24T21:51:47.687445+0000","last_became_peered":"2021-03-24T21:51:47.687445+0000","last_unstale":"2021-03-24T21:53:31.665783+0000","last_undegraded":"2021-03-24T21:53:31.665783+0000","last_fullsized":"2021-03-24T21:53:31.665783+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'7","last_scrub_stamp":"2021-03-24T21:53:13.645928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:13.645928+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":7,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"39","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.873441+0000","last_change":"2021-03-24T21:53:09.854407+0000","last_active":"2021-03-24T21:53:31.873441+0000","last_peered":"2021-03-24T21:53:31.873441+0000","last_clean":"2021-03-24T21:53:31.873441+0000","last_became_active":"2021-03-24T21:51:51.913733+0000","last_became_peered":"2021-03-24T21:51:51.913733+0000","last_unstale":"2021-03-24T21:53:31.873441+0000","last_undegraded":"2021-03-24T21:53:31.873441+0000","last_fullsized":"2021-03-24T21:53:31.873441+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:09.854304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:09.854304+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"89'4","reported_seq":"48","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.898706+0000","last_change":"2021-03-24T21:53:19.926732+0000","last_active":"2021-03-24T21:53:31.898706+0000","last_peered":"2021-03-24T21:53:31.898706+0000","last_clean":"2021-03-24T21:53:31.898706+0000","last_became_active":"2021-03-24T21:51:45.385110+0000","last_became_peered":"2021-03-24T21:51:45.385110+0000","last_unstale":"2021-03-24T21:53:31.898706+0000","last_undegraded":"2021-03-24T21:53:31.898706+0000","last_fullsized":"2021-03-24T21:53:31.898706+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'4","last_scrub_stamp":"2021-03-24T21:53:19.926637+0000","last_deep_scrub":"89'4","last_deep_scrub_stamp":"2021-03-24T21:53:19.926637+0000","last_clean_scrub_stamp":"2021-03-24T21:53:19.926637+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"69'18","reported_seq":"74","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.872294+0000","last_change":"2021-03-24T21:52:57.861464+0000","last_active":"2021-03-24T21:53:31.872294+0000","last_peered":"2021-03-24T21:53:31.872294+0000","last_clean":"2021-03-24T21:53:31.872294+0000","last_became_active":"2021-03-24T21:51:47.685994+0000","last_became_peered":"2021-03-24T21:51:47.685994+0000","last_unstale":"2021-03-24T21:53:31.872294+0000","last_undegraded":"2021-03-24T21:53:31.872294+0000","last_fullsized":"2021-03-24T21:53:31.872294+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'18","last_scrub_stamp":"2021-03-24T21:52:57.861253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:52:57.861253+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":20,"num_write":19,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"65'2","reported_seq":"64","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:38.805536+0000","last_change":"2021-03-24T21:53:03.836648+0000","last_active":"2021-03-24T21:53:38.805536+0000","last_peered":"2021-03-24T21:53:38.805536+0000","last_clean":"2021-03-24T21:53:38.805536+0000","last_became_active":"2021-03-24T21:51:49.415167+0000","last_became_peered":"2021-03-24T21:51:49.415167+0000","last_unstale":"2021-03-24T21:53:38.805536+0000","last_undegraded":"2021-03-24T21:53:38.805536+0000","last_fullsized":"2021-03-24T21:53:38.805536+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2021-03-24T21:53:03.836463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:03.836463+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"38","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.664849+0000","last_change":"2021-03-24T21:53:22.725035+0000","last_active":"2021-03-24T21:53:31.664849+0000","last_peered":"2021-03-24T21:53:31.664849+0000","last_clean":"2021-03-24T21:53:31.664849+0000","last_became_active":"2021-03-24T21:51:51.762035+0000","last_became_peered":"2021-03-24T21:51:51.762035+0000","last_unstale":"2021-03-24T21:53:31.664849+0000","last_undegraded":"2021-03-24T21:53:31.664849+0000","last_fullsized":"2021-03-24T21:53:31.664849+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:22.724840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:22.724840+0000","last_clean_scrub_stamp":"2021-03-24T21:53:22.724840+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":"43","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.898481+0000","last_change":"2021-03-24T21:52:46.847404+0000","last_active":"2021-03-24T21:53:31.898481+0000","last_peered":"2021-03-24T21:53:31.898481+0000","last_clean":"2021-03-24T21:53:31.898481+0000","last_became_active":"2021-03-24T21:51:45.380984+0000","last_became_peered":"2021-03-24T21:51:45.380984+0000","last_unstale":"2021-03-24T21:53:31.898481+0000","last_undegraded":"2021-03-24T21:53:31.898481+0000","last_fullsized":"2021-03-24T21:53:31.898481+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:46.847232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:52:46.847232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"69'15","reported_seq":"71","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:37.334642+0000","last_change":"2021-03-24T21:53:37.334527+0000","last_active":"2021-03-24T21:53:37.334642+0000","last_peered":"2021-03-24T21:53:37.334642+0000","last_clean":"2021-03-24T21:53:37.334642+0000","last_became_active":"2021-03-24T21:51:47.410650+0000","last_became_peered":"2021-03-24T21:51:47.410650+0000","last_unstale":"2021-03-24T21:53:37.334642+0000","last_undegraded":"2021-03-24T21:53:37.334642+0000","last_fullsized":"2021-03-24T21:53:37.334642+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:53:37.334345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:37.334345+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":19,"num_write":17,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"41","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.422753+0000","last_change":"2021-03-24T21:53:18.557275+0000","last_active":"2021-03-24T21:53:31.422753+0000","last_peered":"2021-03-24T21:53:31.422753+0000","last_clean":"2021-03-24T21:53:31.422753+0000","last_became_active":"2021-03-24T21:51:49.415403+0000","last_became_peered":"2021-03-24T21:51:49.415403+0000","last_unstale":"2021-03-24T21:53:31.422753+0000","last_undegraded":"2021-03-24T21:53:31.422753+0000","last_fullsized":"2021-03-24T21:53:31.422753+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:18.557122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:18.557122+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"37","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.896739+0000","last_change":"2021-03-24T21:53:02.842718+0000","last_active":"2021-03-24T21:53:31.896739+0000","last_peered":"2021-03-24T21:53:31.896739+0000","last_clean":"2021-03-24T21:53:31.896739+0000","last_became_active":"2021-03-24T21:51:51.762513+0000","last_became_peered":"2021-03-24T21:51:51.762513+0000","last_unstale":"2021-03-24T21:53:31.896739+0000","last_undegraded":"2021-03-24T21:53:31.896739+0000","last_fullsized":"2021-03-24T21:53:31.896739+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:02.842537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:02.842537+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"41","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.423107+0000","last_change":"2021-03-24T21:52:59.521026+0000","last_active":"2021-03-24T21:53:31.423107+0000","last_peered":"2021-03-24T21:53:31.423107+0000","last_clean":"2021-03-24T21:53:31.423107+0000","last_became_active":"2021-03-24T21:51:49.417699+0000","last_became_peered":"2021-03-24T21:51:49.417699+0000","last_unstale":"2021-03-24T21:53:31.423107+0000","last_undegraded":"2021-03-24T21:53:31.423107+0000","last_fullsized":"2021-03-24T21:53:31.423107+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:59.520858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:52:59.520858+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":"43","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.898420+0000","last_change":"2021-03-24T21:53:26.908991+0000","last_active":"2021-03-24T21:53:31.898420+0000","last_peered":"2021-03-24T21:53:31.898420+0000","last_clean":"2021-03-24T21:53:31.898420+0000","last_became_active":"2021-03-24T21:51:45.384545+0000","last_became_peered":"2021-03-24T21:51:45.384545+0000","last_unstale":"2021-03-24T21:53:31.898420+0000","last_undegraded":"2021-03-24T21:53:31.898420+0000","last_fullsized":"2021-03-24T21:53:31.898420+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:26.908821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:26.908821+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"69'23","reported_seq":"81","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.638227+0000","last_change":"2021-03-24T21:52:47.126913+0000","last_active":"2021-03-24T21:53:31.638227+0000","last_peered":"2021-03-24T21:53:31.638227+0000","last_clean":"2021-03-24T21:53:31.638227+0000","last_became_active":"2021-03-24T21:51:47.685830+0000","last_became_peered":"2021-03-24T21:51:47.685830+0000","last_unstale":"2021-03-24T21:53:31.638227+0000","last_undegraded":"2021-03-24T21:53:31.638227+0000","last_fullsized":"2021-03-24T21:53:31.638227+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'23","last_scrub_stamp":"2021-03-24T21:52:47.126826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:52:47.126826+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":44,"num_read_kb":25,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"37","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:35.636450+0000","last_change":"2021-03-24T21:53:35.636404+0000","last_active":"2021-03-24T21:53:35.636450+0000","last_peered":"2021-03-24T21:53:35.636450+0000","last_clean":"2021-03-24T21:53:35.636450+0000","last_became_active":"2021-03-24T21:51:51.763077+0000","last_became_peered":"2021-03-24T21:51:51.763077+0000","last_unstale":"2021-03-24T21:53:35.636450+0000","last_undegraded":"2021-03-24T21:53:35.636450+0000","last_fullsized":"2021-03-24T21:53:35.636450+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:35.636299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:35.636299+0000","last_clean_scrub_stamp":"2021-03-24T21:53:35.636299+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"41","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.422936+0000","last_change":"2021-03-24T21:53:11.623536+0000","last_active":"2021-03-24T21:53:31.422936+0000","last_peered":"2021-03-24T21:53:31.422936+0000","last_clean":"2021-03-24T21:53:31.422936+0000","last_became_active":"2021-03-24T21:51:49.411921+0000","last_became_peered":"2021-03-24T21:51:49.411921+0000","last_unstale":"2021-03-24T21:53:31.422936+0000","last_undegraded":"2021-03-24T21:53:31.422936+0000","last_fullsized":"2021-03-24T21:53:31.422936+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:11.623353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:11.623353+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"45","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.873529+0000","last_change":"2021-03-24T21:52:47.709622+0000","last_active":"2021-03-24T21:53:31.873529+0000","last_peered":"2021-03-24T21:53:31.873529+0000","last_clean":"2021-03-24T21:53:31.873529+0000","last_became_active":"2021-03-24T21:51:45.383017+0000","last_became_peered":"2021-03-24T21:51:45.383017+0000","last_unstale":"2021-03-24T21:53:31.873529+0000","last_undegraded":"2021-03-24T21:53:31.873529+0000","last_fullsized":"2021-03-24T21:53:31.873529+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:47.709437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:52:47.709437+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"69'14","reported_seq":"67","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.653542+0000","last_change":"2021-03-24T21:53:31.653478+0000","last_active":"2021-03-24T21:53:31.653542+0000","last_peered":"2021-03-24T21:53:31.653542+0000","last_clean":"2021-03-24T21:53:31.653542+0000","last_became_active":"2021-03-24T21:51:47.408588+0000","last_became_peered":"2021-03-24T21:51:47.408588+0000","last_unstale":"2021-03-24T21:53:31.653542+0000","last_undegraded":"2021-03-24T21:53:31.653542+0000","last_fullsized":"2021-03-24T21:53:31.653542+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'14","last_scrub_stamp":"2021-03-24T21:53:31.653334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:31.653334+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":16,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"38","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.664854+0000","last_change":"2021-03-24T21:52:59.706737+0000","last_active":"2021-03-24T21:53:31.664854+0000","last_peered":"2021-03-24T21:53:31.664854+0000","last_clean":"2021-03-24T21:53:31.664854+0000","last_became_active":"2021-03-24T21:51:51.759514+0000","last_became_peered":"2021-03-24T21:51:51.759514+0000","last_unstale":"2021-03-24T21:53:31.664854+0000","last_undegraded":"2021-03-24T21:53:31.664854+0000","last_fullsized":"2021-03-24T21:53:31.664854+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:59.706511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:52:59.706511+0000","last_clean_scrub_stamp":"2021-03-24T21:52:59.706511+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"49","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.873143+0000","last_change":"2021-03-24T21:53:06.800587+0000","last_active":"2021-03-24T21:53:31.873143+0000","last_peered":"2021-03-24T21:53:31.873143+0000","last_clean":"2021-03-24T21:53:31.873143+0000","last_became_active":"2021-03-24T21:51:40.350011+0000","last_became_peered":"2021-03-24T21:51:40.350011+0000","last_unstale":"2021-03-24T21:53:31.873143+0000","last_undegraded":"2021-03-24T21:53:31.873143+0000","last_fullsized":"2021-03-24T21:53:31.873143+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:06.800417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:06.800417+0000","last_clean_scrub_stamp":"2021-03-24T21:53:06.800417+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"89'2","reported_seq":"47","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.422148+0000","last_change":"2021-03-24T21:53:28.220091+0000","last_active":"2021-03-24T21:53:31.422148+0000","last_peered":"2021-03-24T21:53:31.422148+0000","last_clean":"2021-03-24T21:53:31.422148+0000","last_became_active":"2021-03-24T21:51:45.375765+0000","last_became_peered":"2021-03-24T21:51:45.375765+0000","last_unstale":"2021-03-24T21:53:31.422148+0000","last_undegraded":"2021-03-24T21:53:31.422148+0000","last_fullsized":"2021-03-24T21:53:31.422148+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2021-03-24T21:53:28.219900+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2021-03-24T21:53:28.219900+0000","last_clean_scrub_stamp":"2021-03-24T21:53:28.219900+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"69'25","reported_seq":"90","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.425468+0000","last_change":"2021-03-24T21:53:15.719556+0000","last_active":"2021-03-24T21:53:31.425468+0000","last_peered":"2021-03-24T21:53:31.425468+0000","last_clean":"2021-03-24T21:53:31.425468+0000","last_became_active":"2021-03-24T21:51:47.411647+0000","last_became_peered":"2021-03-24T21:51:47.411647+0000","last_unstale":"2021-03-24T21:53:31.425468+0000","last_undegraded":"2021-03-24T21:53:31.425468+0000","last_fullsized":"2021-03-24T21:53:31.425468+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'25","last_scrub_stamp":"2021-03-24T21:53:15.719434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:15.719434+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":54,"num_read_kb":31,"num_write":28,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"39","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.651499+0000","last_change":"2021-03-24T21:53:23.578854+0000","last_active":"2021-03-24T21:53:31.651499+0000","last_peered":"2021-03-24T21:53:31.651499+0000","last_clean":"2021-03-24T21:53:31.651499+0000","last_became_active":"2021-03-24T21:51:49.421253+0000","last_became_peered":"2021-03-24T21:51:49.421253+0000","last_unstale":"2021-03-24T21:53:31.651499+0000","last_undegraded":"2021-03-24T21:53:31.651499+0000","last_fullsized":"2021-03-24T21:53:31.651499+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:23.578679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:23.578679+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"37","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.898617+0000","last_change":"2021-03-24T21:53:23.927068+0000","last_active":"2021-03-24T21:53:31.898617+0000","last_peered":"2021-03-24T21:53:31.898617+0000","last_clean":"2021-03-24T21:53:31.898617+0000","last_became_active":"2021-03-24T21:51:51.912075+0000","last_became_peered":"2021-03-24T21:51:51.912075+0000","last_unstale":"2021-03-24T21:53:31.898617+0000","last_undegraded":"2021-03-24T21:53:31.898617+0000","last_fullsized":"2021-03-24T21:53:31.898617+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:23.926884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:23.926884+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"61'1","reported_seq":"48","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.425535+0000","last_change":"2021-03-24T21:53:18.753108+0000","last_active":"2021-03-24T21:53:31.425535+0000","last_peered":"2021-03-24T21:53:31.425535+0000","last_clean":"2021-03-24T21:53:31.425535+0000","last_became_active":"2021-03-24T21:51:45.580337+0000","last_became_peered":"2021-03-24T21:51:45.580337+0000","last_unstale":"2021-03-24T21:53:31.425535+0000","last_undegraded":"2021-03-24T21:53:31.425535+0000","last_fullsized":"2021-03-24T21:53:31.425535+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2021-03-24T21:53:18.753026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:18.753026+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"49","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.422852+0000","last_change":"2021-03-24T21:52:47.514577+0000","last_active":"2021-03-24T21:53:31.422852+0000","last_peered":"2021-03-24T21:53:31.422852+0000","last_clean":"2021-03-24T21:53:31.422852+0000","last_became_active":"2021-03-24T21:51:40.351000+0000","last_became_peered":"2021-03-24T21:51:40.351000+0000","last_unstale":"2021-03-24T21:53:31.422852+0000","last_undegraded":"2021-03-24T21:53:31.422852+0000","last_fullsized":"2021-03-24T21:53:31.422852+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:47.514427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:39.286101+0000","last_clean_scrub_stamp":"2021-03-24T21:52:47.514427+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"89'22","reported_seq":"81","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.425810+0000","last_change":"2021-03-24T21:53:19.717364+0000","last_active":"2021-03-24T21:53:31.425810+0000","last_peered":"2021-03-24T21:53:31.425810+0000","last_clean":"2021-03-24T21:53:31.425810+0000","last_became_active":"2021-03-24T21:51:47.411696+0000","last_became_peered":"2021-03-24T21:51:47.411696+0000","last_unstale":"2021-03-24T21:53:31.425810+0000","last_undegraded":"2021-03-24T21:53:31.425810+0000","last_fullsized":"2021-03-24T21:53:31.425810+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'22","last_scrub_stamp":"2021-03-24T21:53:19.717244+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2021-03-24T21:53:19.717244+0000","last_clean_scrub_stamp":"2021-03-24T21:53:19.717244+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":36,"num_read_kb":21,"num_write":18,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"41","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.423845+0000","last_change":"2021-03-24T21:53:21.534119+0000","last_active":"2021-03-24T21:53:31.423845+0000","last_peered":"2021-03-24T21:53:31.423845+0000","last_clean":"2021-03-24T21:53:31.423845+0000","last_became_active":"2021-03-24T21:51:49.418848+0000","last_became_peered":"2021-03-24T21:51:49.418848+0000","last_unstale":"2021-03-24T21:53:31.423845+0000","last_undegraded":"2021-03-24T21:53:31.423845+0000","last_fullsized":"2021-03-24T21:53:31.423845+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:21.533882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:21.533882+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"39","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.874172+0000","last_change":"2021-03-24T21:53:01.832920+0000","last_active":"2021-03-24T21:53:31.874172+0000","last_peered":"2021-03-24T21:53:31.874172+0000","last_clean":"2021-03-24T21:53:31.874172+0000","last_became_active":"2021-03-24T21:51:51.765236+0000","last_became_peered":"2021-03-24T21:51:51.765236+0000","last_unstale":"2021-03-24T21:53:31.874172+0000","last_undegraded":"2021-03-24T21:53:31.874172+0000","last_fullsized":"2021-03-24T21:53:31.874172+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:01.832740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:01.832740+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"53","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.896967+0000","last_change":"2021-03-24T21:53:30.906666+0000","last_active":"2021-03-24T21:53:31.896967+0000","last_peered":"2021-03-24T21:53:31.896967+0000","last_clean":"2021-03-24T21:53:31.896967+0000","last_became_active":"2021-03-24T21:51:40.349967+0000","last_became_peered":"2021-03-24T21:51:40.349967+0000","last_unstale":"2021-03-24T21:53:31.896967+0000","last_undegraded":"2021-03-24T21:53:31.896967+0000","last_fullsized":"2021-03-24T21:53:31.896967+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:30.906502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:49:51.529051+0000","last_clean_scrub_stamp":"2021-03-24T21:53:30.906502+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"44","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.664955+0000","last_change":"2021-03-24T21:53:20.736644+0000","last_active":"2021-03-24T21:53:31.664955+0000","last_peered":"2021-03-24T21:53:31.664955+0000","last_clean":"2021-03-24T21:53:31.664955+0000","last_became_active":"2021-03-24T21:51:45.383779+0000","last_became_peered":"2021-03-24T21:51:45.383779+0000","last_unstale":"2021-03-24T21:53:31.664955+0000","last_undegraded":"2021-03-24T21:53:31.664955+0000","last_fullsized":"2021-03-24T21:53:31.664955+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:20.736432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:20.736432+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"69'15","reported_seq":"66","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.638700+0000","last_change":"2021-03-24T21:53:20.279112+0000","last_active":"2021-03-24T21:53:31.638700+0000","last_peered":"2021-03-24T21:53:31.638700+0000","last_clean":"2021-03-24T21:53:31.638700+0000","last_became_active":"2021-03-24T21:51:47.688873+0000","last_became_peered":"2021-03-24T21:51:47.688873+0000","last_unstale":"2021-03-24T21:53:31.638700+0000","last_undegraded":"2021-03-24T21:53:31.638700+0000","last_fullsized":"2021-03-24T21:53:31.638700+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:53:20.279004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:20.279004+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":27,"num_read_kb":15,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"39","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.898039+0000","last_change":"2021-03-24T21:53:09.874798+0000","last_active":"2021-03-24T21:53:31.898039+0000","last_peered":"2021-03-24T21:53:31.898039+0000","last_clean":"2021-03-24T21:53:31.898039+0000","last_became_active":"2021-03-24T21:51:49.413251+0000","last_became_peered":"2021-03-24T21:51:49.413251+0000","last_unstale":"2021-03-24T21:53:31.898039+0000","last_undegraded":"2021-03-24T21:53:31.898039+0000","last_fullsized":"2021-03-24T21:53:31.898039+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:09.874681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:09.874681+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"39","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.425249+0000","last_change":"2021-03-24T21:53:11.723900+0000","last_active":"2021-03-24T21:53:31.425249+0000","last_peered":"2021-03-24T21:53:31.425249+0000","last_clean":"2021-03-24T21:53:31.425249+0000","last_became_active":"2021-03-24T21:51:51.762039+0000","last_became_peered":"2021-03-24T21:51:51.762039+0000","last_unstale":"2021-03-24T21:53:31.425249+0000","last_undegraded":"2021-03-24T21:53:31.425249+0000","last_fullsized":"2021-03-24T21:53:31.425249+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:11.723709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:11.723709+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"43","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.651013+0000","last_change":"2021-03-24T21:52:48.487965+0000","last_active":"2021-03-24T21:53:31.651013+0000","last_peered":"2021-03-24T21:53:31.651013+0000","last_clean":"2021-03-24T21:53:31.651013+0000","last_became_active":"2021-03-24T21:51:45.376687+0000","last_became_peered":"2021-03-24T21:51:45.376687+0000","last_unstale":"2021-03-24T21:53:31.651013+0000","last_undegraded":"2021-03-24T21:53:31.651013+0000","last_fullsized":"2021-03-24T21:53:31.651013+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:48.487787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:52:48.487787+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"66'5","reported_seq":"168","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:42.319250+0000","last_change":"2021-03-24T21:52:50.807770+0000","last_active":"2021-03-24T21:53:42.319250+0000","last_peered":"2021-03-24T21:53:42.319250+0000","last_clean":"2021-03-24T21:53:42.319250+0000","last_became_active":"2021-03-24T21:51:40.350525+0000","last_became_peered":"2021-03-24T21:51:40.350525+0000","last_unstale":"2021-03-24T21:53:42.319250+0000","last_undegraded":"2021-03-24T21:53:42.319250+0000","last_fullsized":"2021-03-24T21:53:42.319250+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2021-03-24T21:52:50.807568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:39.286101+0000","last_clean_scrub_stamp":"2021-03-24T21:52:50.807568+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":118,"num_read_kb":113,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"69'16","reported_seq":"67","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.651988+0000","last_change":"2021-03-24T21:53:15.485180+0000","last_active":"2021-03-24T21:53:31.651988+0000","last_peered":"2021-03-24T21:53:31.651988+0000","last_clean":"2021-03-24T21:53:31.651988+0000","last_became_active":"2021-03-24T21:51:47.411442+0000","last_became_peered":"2021-03-24T21:51:47.411442+0000","last_unstale":"2021-03-24T21:53:31.651988+0000","last_undegraded":"2021-03-24T21:53:31.651988+0000","last_fullsized":"2021-03-24T21:53:31.651988+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T21:53:15.485072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:15.485072+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":16,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"41","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.874725+0000","last_change":"2021-03-24T21:53:20.836086+0000","last_active":"2021-03-24T21:53:31.874725+0000","last_peered":"2021-03-24T21:53:31.874725+0000","last_clean":"2021-03-24T21:53:31.874725+0000","last_became_active":"2021-03-24T21:51:49.419167+0000","last_became_peered":"2021-03-24T21:51:49.419167+0000","last_unstale":"2021-03-24T21:53:31.874725+0000","last_undegraded":"2021-03-24T21:53:31.874725+0000","last_fullsized":"2021-03-24T21:53:31.874725+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:20.835990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:20.835990+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"39","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.425097+0000","last_change":"2021-03-24T21:53:28.691448+0000","last_active":"2021-03-24T21:53:31.425097+0000","last_peered":"2021-03-24T21:53:31.425097+0000","last_clean":"2021-03-24T21:53:31.425097+0000","last_became_active":"2021-03-24T21:51:51.913581+0000","last_became_peered":"2021-03-24T21:51:51.913581+0000","last_unstale":"2021-03-24T21:53:31.425097+0000","last_undegraded":"2021-03-24T21:53:31.425097+0000","last_fullsized":"2021-03-24T21:53:31.425097+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:28.691270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:28.691270+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"45","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.425366+0000","last_change":"2021-03-24T21:53:08.691716+0000","last_active":"2021-03-24T21:53:31.425366+0000","last_peered":"2021-03-24T21:53:31.425366+0000","last_clean":"2021-03-24T21:53:31.425366+0000","last_became_active":"2021-03-24T21:51:45.580531+0000","last_became_peered":"2021-03-24T21:51:45.580531+0000","last_unstale":"2021-03-24T21:53:31.425366+0000","last_undegraded":"2021-03-24T21:53:31.425366+0000","last_fullsized":"2021-03-24T21:53:31.425366+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:08.691536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:08.691536+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"69'21","reported_seq":"81","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.651725+0000","last_change":"2021-03-24T21:53:05.313409+0000","last_active":"2021-03-24T21:53:31.651725+0000","last_peered":"2021-03-24T21:53:31.651725+0000","last_clean":"2021-03-24T21:53:31.651725+0000","last_became_active":"2021-03-24T21:51:47.686486+0000","last_became_peered":"2021-03-24T21:51:47.686486+0000","last_unstale":"2021-03-24T21:53:31.651725+0000","last_undegraded":"2021-03-24T21:53:31.651725+0000","last_fullsized":"2021-03-24T21:53:31.651725+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'21","last_scrub_stamp":"2021-03-24T21:53:05.313235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:05.313235+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":45,"num_read_kb":27,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"40","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.638362+0000","last_change":"2021-03-24T21:53:23.246488+0000","last_active":"2021-03-24T21:53:31.638362+0000","last_peered":"2021-03-24T21:53:31.638362+0000","last_clean":"2021-03-24T21:53:31.638362+0000","last_became_active":"2021-03-24T21:51:49.416847+0000","last_became_peered":"2021-03-24T21:51:49.416847+0000","last_unstale":"2021-03-24T21:53:31.638362+0000","last_undegraded":"2021-03-24T21:53:31.638362+0000","last_fullsized":"2021-03-24T21:53:31.638362+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:23.246320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:23.246320+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"37","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.650517+0000","last_change":"2021-03-24T21:53:25.593543+0000","last_active":"2021-03-24T21:53:31.650517+0000","last_peered":"2021-03-24T21:53:31.650517+0000","last_clean":"2021-03-24T21:53:31.650517+0000","last_became_active":"2021-03-24T21:51:51.759627+0000","last_became_peered":"2021-03-24T21:51:51.759627+0000","last_unstale":"2021-03-24T21:53:31.650517+0000","last_undegraded":"2021-03-24T21:53:31.650517+0000","last_fullsized":"2021-03-24T21:53:31.650517+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:25.593358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:25.593358+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"45","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.874041+0000","last_change":"2021-03-24T21:53:04.837752+0000","last_active":"2021-03-24T21:53:31.874041+0000","last_peered":"2021-03-24T21:53:31.874041+0000","last_clean":"2021-03-24T21:53:31.874041+0000","last_became_active":"2021-03-24T21:51:45.583885+0000","last_became_peered":"2021-03-24T21:51:45.583885+0000","last_unstale":"2021-03-24T21:53:31.874041+0000","last_undegraded":"2021-03-24T21:53:31.874041+0000","last_fullsized":"2021-03-24T21:53:31.874041+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:04.837488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:04.837488+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"69'16","reported_seq":"70","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.425907+0000","last_change":"2021-03-24T21:53:03.595733+0000","last_active":"2021-03-24T21:53:31.425907+0000","last_peered":"2021-03-24T21:53:31.425907+0000","last_clean":"2021-03-24T21:53:31.425907+0000","last_became_active":"2021-03-24T21:51:47.403006+0000","last_became_peered":"2021-03-24T21:51:47.403006+0000","last_unstale":"2021-03-24T21:53:31.425907+0000","last_undegraded":"2021-03-24T21:53:31.425907+0000","last_fullsized":"2021-03-24T21:53:31.425907+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T21:53:03.595585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:03.595585+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"89'3","reported_seq":"65","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:38.806462+0000","last_change":"2021-03-24T21:53:26.584866+0000","last_active":"2021-03-24T21:53:38.806462+0000","last_peered":"2021-03-24T21:53:38.806462+0000","last_clean":"2021-03-24T21:53:38.806462+0000","last_became_active":"2021-03-24T21:51:49.420173+0000","last_became_peered":"2021-03-24T21:51:49.420173+0000","last_unstale":"2021-03-24T21:53:38.806462+0000","last_undegraded":"2021-03-24T21:53:38.806462+0000","last_fullsized":"2021-03-24T21:53:38.806462+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'3","last_scrub_stamp":"2021-03-24T21:53:26.584678+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-24T21:53:26.584678+0000","last_clean_scrub_stamp":"2021-03-24T21:53:26.584678+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"37","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.897726+0000","last_change":"2021-03-24T21:52:55.828993+0000","last_active":"2021-03-24T21:53:31.897726+0000","last_peered":"2021-03-24T21:53:31.897726+0000","last_clean":"2021-03-24T21:53:31.897726+0000","last_became_active":"2021-03-24T21:51:51.914601+0000","last_became_peered":"2021-03-24T21:51:51.914601+0000","last_unstale":"2021-03-24T21:53:31.897726+0000","last_undegraded":"2021-03-24T21:53:31.897726+0000","last_fullsized":"2021-03-24T21:53:31.897726+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:55.828810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:52:55.828810+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"43","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.652135+0000","last_change":"2021-03-24T21:53:24.621910+0000","last_active":"2021-03-24T21:53:31.652135+0000","last_peered":"2021-03-24T21:53:31.652135+0000","last_clean":"2021-03-24T21:53:31.652135+0000","last_became_active":"2021-03-24T21:51:45.375747+0000","last_became_peered":"2021-03-24T21:51:45.375747+0000","last_unstale":"2021-03-24T21:53:31.652135+0000","last_undegraded":"2021-03-24T21:53:31.652135+0000","last_fullsized":"2021-03-24T21:53:31.652135+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:24.621749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:24.621749+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"69'20","reported_seq":"75","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.422003+0000","last_change":"2021-03-24T21:53:13.077033+0000","last_active":"2021-03-24T21:53:31.422003+0000","last_peered":"2021-03-24T21:53:31.422003+0000","last_clean":"2021-03-24T21:53:31.422003+0000","last_became_active":"2021-03-24T21:51:47.404713+0000","last_became_peered":"2021-03-24T21:51:47.404713+0000","last_unstale":"2021-03-24T21:53:31.422003+0000","last_undegraded":"2021-03-24T21:53:31.422003+0000","last_fullsized":"2021-03-24T21:53:31.422003+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'20","last_scrub_stamp":"2021-03-24T21:53:13.076772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:13.076772+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":39,"num_read_kb":20,"num_write":20,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"65'2","reported_seq":"65","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:38.805729+0000","last_change":"2021-03-24T21:53:04.714525+0000","last_active":"2021-03-24T21:53:38.805729+0000","last_peered":"2021-03-24T21:53:38.805729+0000","last_clean":"2021-03-24T21:53:38.805729+0000","last_became_active":"2021-03-24T21:51:49.421628+0000","last_became_peered":"2021-03-24T21:51:49.421628+0000","last_unstale":"2021-03-24T21:53:38.805729+0000","last_undegraded":"2021-03-24T21:53:38.805729+0000","last_fullsized":"2021-03-24T21:53:38.805729+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2021-03-24T21:53:04.714348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:04.714348+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"33","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.421953+0000","last_change":"2021-03-24T21:51:51.756423+0000","last_active":"2021-03-24T21:53:31.421953+0000","last_peered":"2021-03-24T21:53:31.421953+0000","last_clean":"2021-03-24T21:53:31.421953+0000","last_became_active":"2021-03-24T21:51:51.756114+0000","last_became_peered":"2021-03-24T21:51:51.756114+0000","last_unstale":"2021-03-24T21:53:31.421953+0000","last_undegraded":"2021-03-24T21:53:31.421953+0000","last_fullsized":"2021-03-24T21:53:31.421953+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:51:50.731137+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"44","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.665448+0000","last_change":"2021-03-24T21:53:00.682889+0000","last_active":"2021-03-24T21:53:31.665448+0000","last_peered":"2021-03-24T21:53:31.665448+0000","last_clean":"2021-03-24T21:53:31.665448+0000","last_became_active":"2021-03-24T21:51:45.378704+0000","last_became_peered":"2021-03-24T21:51:45.378704+0000","last_unstale":"2021-03-24T21:53:31.665448+0000","last_undegraded":"2021-03-24T21:53:31.665448+0000","last_fullsized":"2021-03-24T21:53:31.665448+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:00.682713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:00.682713+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"69'22","reported_seq":"82","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.664261+0000","last_change":"2021-03-24T21:53:17.716726+0000","last_active":"2021-03-24T21:53:31.664261+0000","last_peered":"2021-03-24T21:53:31.664261+0000","last_clean":"2021-03-24T21:53:31.664261+0000","last_became_active":"2021-03-24T21:51:47.691459+0000","last_became_peered":"2021-03-24T21:51:47.691459+0000","last_unstale":"2021-03-24T21:53:31.664261+0000","last_undegraded":"2021-03-24T21:53:31.664261+0000","last_fullsized":"2021-03-24T21:53:31.664261+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'22","last_scrub_stamp":"2021-03-24T21:53:17.716470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:17.716470+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":46,"num_read_kb":26,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"40","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.421753+0000","last_change":"2021-03-24T21:53:25.220598+0000","last_active":"2021-03-24T21:53:31.421753+0000","last_peered":"2021-03-24T21:53:31.421753+0000","last_clean":"2021-03-24T21:53:31.421753+0000","last_became_active":"2021-03-24T21:51:49.408761+0000","last_became_peered":"2021-03-24T21:51:49.408761+0000","last_unstale":"2021-03-24T21:53:31.421753+0000","last_undegraded":"2021-03-24T21:53:31.421753+0000","last_fullsized":"2021-03-24T21:53:31.421753+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:25.220436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:25.220436+0000","last_clean_scrub_stamp":"2021-03-24T21:53:25.220436+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"39","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.873607+0000","last_change":"2021-03-24T21:53:11.925302+0000","last_active":"2021-03-24T21:53:31.873607+0000","last_peered":"2021-03-24T21:53:31.873607+0000","last_clean":"2021-03-24T21:53:31.873607+0000","last_became_active":"2021-03-24T21:51:51.760369+0000","last_became_peered":"2021-03-24T21:51:51.760369+0000","last_unstale":"2021-03-24T21:53:31.873607+0000","last_undegraded":"2021-03-24T21:53:31.873607+0000","last_fullsized":"2021-03-24T21:53:31.873607+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:11.925149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:11.925149+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"65'2","reported_seq":"64","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.645845+0000","last_change":"2021-03-24T21:53:29.268452+0000","last_active":"2021-03-24T21:53:37.645845+0000","last_peered":"2021-03-24T21:53:37.645845+0000","last_clean":"2021-03-24T21:53:37.645845+0000","last_became_active":"2021-03-24T21:51:49.414380+0000","last_became_peered":"2021-03-24T21:51:49.414380+0000","last_unstale":"2021-03-24T21:53:37.645845+0000","last_undegraded":"2021-03-24T21:53:37.645845+0000","last_fullsized":"2021-03-24T21:53:37.645845+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2021-03-24T21:53:29.268273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:29.268273+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":"44","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:35.609404+0000","last_change":"2021-03-24T21:53:35.609332+0000","last_active":"2021-03-24T21:53:35.609404+0000","last_peered":"2021-03-24T21:53:35.609404+0000","last_clean":"2021-03-24T21:53:35.609404+0000","last_became_active":"2021-03-24T21:51:45.378863+0000","last_became_peered":"2021-03-24T21:51:45.378863+0000","last_unstale":"2021-03-24T21:53:35.609404+0000","last_undegraded":"2021-03-24T21:53:35.609404+0000","last_fullsized":"2021-03-24T21:53:35.609404+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:35.609173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:35.609173+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"69'17","reported_seq":"72","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.425898+0000","last_change":"2021-03-24T21:52:54.714245+0000","last_active":"2021-03-24T21:53:31.425898+0000","last_peered":"2021-03-24T21:53:31.425898+0000","last_clean":"2021-03-24T21:53:31.425898+0000","last_became_active":"2021-03-24T21:51:47.411126+0000","last_became_peered":"2021-03-24T21:51:47.411126+0000","last_unstale":"2021-03-24T21:53:31.425898+0000","last_undegraded":"2021-03-24T21:53:31.425898+0000","last_fullsized":"2021-03-24T21:53:31.425898+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'17","last_scrub_stamp":"2021-03-24T21:52:54.714050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:52:54.714050+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":19,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"38","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.664862+0000","last_change":"2021-03-24T21:53:27.701155+0000","last_active":"2021-03-24T21:53:31.664862+0000","last_peered":"2021-03-24T21:53:31.664862+0000","last_clean":"2021-03-24T21:53:31.664862+0000","last_became_active":"2021-03-24T21:51:51.912009+0000","last_became_peered":"2021-03-24T21:51:51.912009+0000","last_unstale":"2021-03-24T21:53:31.664862+0000","last_undegraded":"2021-03-24T21:53:31.664862+0000","last_fullsized":"2021-03-24T21:53:31.664862+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:27.701002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:27.701002+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"43","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.874620+0000","last_change":"2021-03-24T21:52:55.846936+0000","last_active":"2021-03-24T21:53:31.874620+0000","last_peered":"2021-03-24T21:53:31.874620+0000","last_clean":"2021-03-24T21:53:31.874620+0000","last_became_active":"2021-03-24T21:51:49.422787+0000","last_became_peered":"2021-03-24T21:51:49.422787+0000","last_unstale":"2021-03-24T21:53:31.874620+0000","last_undegraded":"2021-03-24T21:53:31.874620+0000","last_fullsized":"2021-03-24T21:53:31.874620+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:55.846764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:52:55.846764+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"44","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.422013+0000","last_change":"2021-03-24T21:52:48.334326+0000","last_active":"2021-03-24T21:53:31.422013+0000","last_peered":"2021-03-24T21:53:31.422013+0000","last_clean":"2021-03-24T21:53:31.422013+0000","last_became_active":"2021-03-24T21:51:45.579513+0000","last_became_peered":"2021-03-24T21:51:45.579513+0000","last_unstale":"2021-03-24T21:53:31.422013+0000","last_undegraded":"2021-03-24T21:53:31.422013+0000","last_fullsized":"2021-03-24T21:53:31.422013+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:48.334138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:52:48.334138+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"69'15","reported_seq":"69","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.422946+0000","last_change":"2021-03-24T21:52:55.233331+0000","last_active":"2021-03-24T21:53:31.422946+0000","last_peered":"2021-03-24T21:53:31.422946+0000","last_clean":"2021-03-24T21:53:31.422946+0000","last_became_active":"2021-03-24T21:51:47.410414+0000","last_became_peered":"2021-03-24T21:51:47.410414+0000","last_unstale":"2021-03-24T21:53:31.422946+0000","last_undegraded":"2021-03-24T21:53:31.422946+0000","last_fullsized":"2021-03-24T21:53:31.422946+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:52:55.233213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:52:55.233213+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"39","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.423972+0000","last_change":"2021-03-24T21:53:14.576861+0000","last_active":"2021-03-24T21:53:31.423972+0000","last_peered":"2021-03-24T21:53:31.423972+0000","last_clean":"2021-03-24T21:53:31.423972+0000","last_became_active":"2021-03-24T21:51:51.765067+0000","last_became_peered":"2021-03-24T21:51:51.765067+0000","last_unstale":"2021-03-24T21:53:31.423972+0000","last_undegraded":"2021-03-24T21:53:31.423972+0000","last_fullsized":"2021-03-24T21:53:31.423972+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:14.576687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:14.576687+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"41","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.425582+0000","last_change":"2021-03-24T21:53:29.735847+0000","last_active":"2021-03-24T21:53:31.425582+0000","last_peered":"2021-03-24T21:53:31.425582+0000","last_clean":"2021-03-24T21:53:31.425582+0000","last_became_active":"2021-03-24T21:51:49.407001+0000","last_became_peered":"2021-03-24T21:51:49.407001+0000","last_unstale":"2021-03-24T21:53:31.425582+0000","last_undegraded":"2021-03-24T21:53:31.425582+0000","last_fullsized":"2021-03-24T21:53:31.425582+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:29.735672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:29.735672+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"44","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.637890+0000","last_change":"2021-03-24T21:53:22.204032+0000","last_active":"2021-03-24T21:53:31.637890+0000","last_peered":"2021-03-24T21:53:31.637890+0000","last_clean":"2021-03-24T21:53:31.637890+0000","last_became_active":"2021-03-24T21:51:45.374203+0000","last_became_peered":"2021-03-24T21:51:45.374203+0000","last_unstale":"2021-03-24T21:53:31.637890+0000","last_undegraded":"2021-03-24T21:53:31.637890+0000","last_fullsized":"2021-03-24T21:53:31.637890+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:22.203858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:22.203858+0000","last_clean_scrub_stamp":"2021-03-24T21:53:22.203858+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"69'21","reported_seq":"78","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.422809+0000","last_change":"2021-03-24T21:52:50.302291+0000","last_active":"2021-03-24T21:53:31.422809+0000","last_peered":"2021-03-24T21:53:31.422809+0000","last_clean":"2021-03-24T21:53:31.422809+0000","last_became_active":"2021-03-24T21:51:47.416296+0000","last_became_peered":"2021-03-24T21:51:47.416296+0000","last_unstale":"2021-03-24T21:53:31.422809+0000","last_undegraded":"2021-03-24T21:53:31.422809+0000","last_fullsized":"2021-03-24T21:53:31.422809+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'21","last_scrub_stamp":"2021-03-24T21:52:50.302085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:52:50.302085+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":23,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"38","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:40.290157+0000","last_change":"2021-03-24T21:53:40.290157+0000","last_active":"2021-03-24T21:53:40.290157+0000","last_peered":"2021-03-24T21:53:40.290157+0000","last_clean":"2021-03-24T21:53:40.290157+0000","last_became_active":"2021-03-24T21:51:51.761142+0000","last_became_peered":"2021-03-24T21:51:51.761142+0000","last_unstale":"2021-03-24T21:53:40.290157+0000","last_undegraded":"2021-03-24T21:53:40.290157+0000","last_fullsized":"2021-03-24T21:53:40.290157+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:40.289984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:40.289984+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"65'2","reported_seq":"66","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:38.805700+0000","last_change":"2021-03-24T21:53:04.625006+0000","last_active":"2021-03-24T21:53:38.805700+0000","last_peered":"2021-03-24T21:53:38.805700+0000","last_clean":"2021-03-24T21:53:38.805700+0000","last_became_active":"2021-03-24T21:51:49.416182+0000","last_became_peered":"2021-03-24T21:51:49.416182+0000","last_unstale":"2021-03-24T21:53:38.805700+0000","last_undegraded":"2021-03-24T21:53:38.805700+0000","last_fullsized":"2021-03-24T21:53:38.805700+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2021-03-24T21:53:04.624892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:04.624892+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"44","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.665024+0000","last_change":"2021-03-24T21:52:46.620502+0000","last_active":"2021-03-24T21:53:31.665024+0000","last_peered":"2021-03-24T21:53:31.665024+0000","last_clean":"2021-03-24T21:53:31.665024+0000","last_became_active":"2021-03-24T21:51:45.375830+0000","last_became_peered":"2021-03-24T21:51:45.375830+0000","last_unstale":"2021-03-24T21:53:31.665024+0000","last_undegraded":"2021-03-24T21:53:31.665024+0000","last_fullsized":"2021-03-24T21:53:31.665024+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:46.620317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:52:46.620317+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"69'13","reported_seq":"63","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.422441+0000","last_change":"2021-03-24T21:53:20.163291+0000","last_active":"2021-03-24T21:53:31.422441+0000","last_peered":"2021-03-24T21:53:31.422441+0000","last_clean":"2021-03-24T21:53:31.422441+0000","last_became_active":"2021-03-24T21:51:47.407739+0000","last_became_peered":"2021-03-24T21:51:47.407739+0000","last_unstale":"2021-03-24T21:53:31.422441+0000","last_undegraded":"2021-03-24T21:53:31.422441+0000","last_fullsized":"2021-03-24T21:53:31.422441+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2021-03-24T21:53:20.163065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:20.163065+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":13,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19,"ondisk_log_size":19,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1069,"num_read_kb":574,"num_write":542,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":2412,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966080,"data_stored":70572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":539,"ondisk_log_size":539,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":118,"num_read_kb":113,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201308,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6400,"kb_used_data":824,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738752,"statfs":{"total":95995035648,"available":95988482048,"internally_reserved":0,"allocated":843776,"data_stored":314208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:52:41 2021","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":0.97999999999999998},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":0.878}]},{"osd":1,"last update":"Wed Mar 24 21:52:41 2021","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":2.343,"5min":2.343,"15min":2.343},"last":1.01}]},{"osd":2,"last update":"Wed Mar 24 21:52:41 2021","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.7509999999999999,"5min":1.7509999999999999,"15min":1.7509999999999999},"last":0.84799999999999998},{"interface":"front","average":{"1min":1.012,"5min":1.012,"15min":1.012},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.6850000000000001,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":1.1459999999999999}]},{"osd":3,"last update":"Wed Mar 24 21:52:41 2021","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.9490000000000001,"5min":1.9490000000000001,"15min":1.9490000000000001},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.845,"5min":1.845,"15min":1.845},"last":1.0469999999999999}]},{"osd":4,"last update":"Wed Mar 24 21:52:41 2021","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":2.2000000000000002,"5min":2.2000000000000002,"15min":2.2000000000000002},"last":1.113},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":2.1299999999999999,"5min":2.1299999999999999,"15min":2.1299999999999999},"last":0.92200000000000004}]},{"osd":5,"last update":"Wed Mar 24 21:52:41 2021","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.581,"5min":1.581,"15min":1.581},"last":1.22},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":2.4630000000000001,"5min":2.4630000000000001,"15min":2.4630000000000001},"last":1.25}]},{"osd":6,"last update":"Wed Mar 24 21:52:41 2021","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":2.5179999999999998,"5min":2.5179999999999998,"15min":2.5179999999999998},"last":1.4259999999999999},{"interface":"front","average":{"1min":1.171,"5min":1.171,"15min":1.171},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":2.3069999999999999,"5min":2.3069999999999999,"15min":2.3069999999999999},"last":1.0780000000000001}]}]},{"osd":6,"up_from":47,"seq":201863462945,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6512,"kb_used_data":872,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738640,"statfs":{"total":95995035648,"available":95988367360,"internally_reserved":0,"allocated":892928,"data_stored":317430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:53:21 2021","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.91000000000000003,"15min":0.91200000000000003},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.91200000000000003,"15min":0.91700000000000004},"min":{"1min":0.56799999999999995,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.94599999999999995}]},{"osd":1,"last update":"Wed Mar 24 21:53:21 2021","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.92100000000000004,"15min":0.93200000000000005},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.2470000000000001,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.91100000000000003,"15min":0.91600000000000004},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.2869999999999999,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.63800000000000001}]},{"osd":2,"last update":"Wed Mar 24 21:53:21 2021","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.94899999999999995,"15min":0.95099999999999996},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.96699999999999997,"15min":0.97499999999999998},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.2170000000000001,"5min":1.401,"15min":1.401},"last":0.88700000000000001}]},{"osd":3,"last update":"Wed Mar 24 21:53:21 2021","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.95299999999999996,"15min":0.95699999999999996},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.3089999999999999,"5min":1.4179999999999999,"15min":1.4179999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.022,"15min":1.0149999999999999},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.5009999999999999,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":0.97799999999999998}]},{"osd":4,"last update":"Wed Mar 24 21:53:21 2021","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.92000000000000004,"15min":0.92100000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.26,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.97699999999999998,"5min":0.95099999999999996,"15min":0.94699999999999995},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.3149999999999999,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.96499999999999997}]},{"osd":5,"last update":"Wed Mar 24 21:53:21 2021","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":1.0369999999999999,"15min":1.0449999999999999},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.337,"5min":1.4830000000000001,"15min":1.4830000000000001},"last":1.004},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.04,"15min":1.038},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.4570000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":1.0620000000000001}]},{"osd":7,"last update":"Wed Mar 24 21:53:41 2021","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":1.1359999999999999,"15min":1.1279999999999999},"min":{"1min":0.83799999999999997,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.3560000000000001,"5min":1.5249999999999999,"15min":1.5249999999999999},"last":0.83799999999999997},{"interface":"front","average":{"1min":1.234,"5min":1.1299999999999999,"15min":1.113},"min":{"1min":0.79900000000000004,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.478,"5min":1.478,"15min":1.478},"last":1.04}]}]},{"osd":1,"up_from":13,"seq":55834574902,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7104,"kb_used_data":952,"kb_used_omap":0,"kb_used_meta":6144,"kb_avail":93738048,"statfs":{"total":95995035648,"available":95987761152,"internally_reserved":0,"allocated":974848,"data_stored":320242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6291456},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:53:33 2021","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.629,"15min":0.58699999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.68500000000000005,"15min":0.64100000000000001},"min":{"1min":0.41399999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.3009999999999999,"5min":1.641,"15min":1.641},"last":0.68600000000000005}]},{"osd":2,"last update":"Wed Mar 24 21:52:52 2021","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.64600000000000002,"15min":0.627},"min":{"1min":0.443,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":1.1100000000000001},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.76700000000000002,"15min":0.746},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.173,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":0.84899999999999998}]},{"osd":3,"last update":"Wed Mar 24 21:53:18 2021","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":1.296,"15min":1.4730000000000001},"min":{"1min":0.63100000000000001,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.2849999999999999,"5min":17.032,"15min":17.032},"last":1.2370000000000001},{"interface":"front","average":{"1min":0.90900000000000003,"5min":1.2290000000000001,"15min":1.389},"min":{"1min":0.621,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.135,"5min":16.908999999999999,"15min":16.908999999999999},"last":1.333}]},{"osd":4,"last update":"Wed Mar 24 21:53:41 2021","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.96399999999999997,"15min":0.96399999999999997},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.4510000000000001,"5min":1.617,"15min":1.617},"last":1.4319999999999999},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.093,"15min":1.095},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.706,"5min":1.7150000000000001,"15min":1.7150000000000001},"last":1}]},{"osd":5,"last update":"Wed Mar 24 21:52:55 2021","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":0.999,"15min":0.999},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.3220000000000001,"5min":1.5489999999999999,"15min":1.5489999999999999},"last":0.96099999999999997},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.0640000000000001,"15min":1.0620000000000001},"min":{"1min":0.80000000000000004,"5min":0.748,"15min":0.748},"max":{"1min":1.4199999999999999,"5min":1.6619999999999999,"15min":1.6619999999999999},"last":1.085}]},{"osd":6,"last update":"Wed Mar 24 21:53:18 2021","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.141,"15min":1.1399999999999999},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":1.4239999999999999,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":0.93400000000000005},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.1479999999999999,"15min":1.1619999999999999},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.3819999999999999,"5min":1.806,"15min":1.806},"last":1.278}]},{"osd":7,"last update":"Wed Mar 24 21:53:41 2021","interfaces":[{"interface":"back","average":{"1min":1.2030000000000001,"5min":1.23,"15min":1.234},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.7789999999999999,"5min":1.7789999999999999,"15min":1.7789999999999999},"last":1.1639999999999999},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.175,"15min":1.181},"min":{"1min":0.96499999999999997,"5min":0.90300000000000002,"15min":0.90300000000000002},"max":{"1min":1.3480000000000001,"5min":1.532,"15min":1.532},"last":1.2130000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738425,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6792,"kb_used_data":832,"kb_used_omap":0,"kb_used_meta":5952,"kb_avail":93738360,"statfs":{"total":95995035648,"available":95988080640,"internally_reserved":0,"allocated":851968,"data_stored":315145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6094848},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Mar 24 21:53:37 2021","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.63700000000000001,"15min":0.58399999999999996},"min":{"1min":0.44400000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.98299999999999998,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.67500000000000004,"15min":0.60999999999999999},"min":{"1min":0.46999999999999997,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.78100000000000003}]},{"osd":2,"last update":"Wed Mar 24 21:52:55 2021","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.67500000000000004,"15min":0.66100000000000003},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.879,"5min":1.468,"15min":1.468},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.73099999999999998,"15min":0.69099999999999995},"min":{"1min":0.41799999999999998,"5min":0.376,"15min":0.376},"max":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.55200000000000005}]},{"osd":3,"last update":"Wed Mar 24 21:53:20 2021","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.80500000000000005,"15min":0.80000000000000004},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.115,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.89700000000000002},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.93400000000000005,"15min":0.93200000000000005},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.2430000000000001,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.73099999999999998}]},{"osd":4,"last update":"Wed Mar 24 21:53:37 2021","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.0900000000000001,"15min":1.0960000000000001},"min":{"1min":0.81799999999999995,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.4570000000000001,"5min":1.7,"15min":1.7},"last":1.0229999999999999},{"interface":"front","average":{"1min":1.056,"5min":1.028,"15min":1.014},"min":{"1min":0.78700000000000003,"5min":0.625,"15min":0.625},"max":{"1min":1.375,"5min":1.407,"15min":1.407},"last":1.228}]},{"osd":5,"last update":"Wed Mar 24 21:52:55 2021","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":1.0509999999999999,"15min":1.0620000000000001},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.3919999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":1.143},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.179,"15min":1.177},"min":{"1min":0.86299999999999999,"5min":0.78500000000000003,"15min":0.78500000000000003},"max":{"1min":1.597,"5min":1.746,"15min":1.746},"last":0.95899999999999996}]},{"osd":6,"last update":"Wed Mar 24 21:53:20 2021","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.0669999999999999,"15min":1.0669999999999999},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.3779999999999999,"5min":1.427,"15min":1.427},"last":0.98999999999999999},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.224,"15min":1.22},"min":{"1min":0.877,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.6659999999999999,"5min":1.6659999999999999,"15min":1.6659999999999999},"last":1.0640000000000001}]},{"osd":7,"last update":"Wed Mar 24 21:53:42 2021","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.1739999999999999,"15min":1.1699999999999999},"min":{"1min":0.82099999999999995,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":1.4970000000000001,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":1.28},{"interface":"front","average":{"1min":1.157,"5min":1.131,"15min":1.127},"min":{"1min":0.77100000000000002,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.4319999999999999,"5min":1.4490000000000001,"15min":1.4490000000000001},"last":1.0940000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411379,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6564,"kb_used_data":796,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738588,"statfs":{"total":95995035648,"available":95988314112,"internally_reserved":0,"allocated":815104,"data_stored":314429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:52:55 2021","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.64400000000000002,"15min":0.61799999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.94799999999999995},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.71599999999999997,"15min":0.68100000000000005},"min":{"1min":0.55300000000000005,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.405,"5min":1.405,"15min":1.405},"last":0.79200000000000004}]},{"osd":1,"last update":"Wed Mar 24 21:52:55 2021","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.72199999999999998,"15min":0.69199999999999995},"min":{"1min":0.57499999999999996,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":1.024},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.76600000000000001,"15min":0.72799999999999998},"min":{"1min":0.621,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.444,"5min":1.444,"15min":1.444},"last":1.0529999999999999}]},{"osd":3,"last update":"Wed Mar 24 21:53:17 2021","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.84399999999999997,"15min":0.82399999999999995},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.9890000000000001,"5min":1.9890000000000001,"15min":1.9890000000000001},"last":1.101},{"interface":"front","average":{"1min":0.95699999999999996,"5min":0.92000000000000004,"15min":0.89400000000000002},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":2.0950000000000002,"5min":2.0950000000000002,"15min":2.0950000000000002},"last":1.1359999999999999}]},{"osd":4,"last update":"Wed Mar 24 21:53:40 2021","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.04,"15min":1.036},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":3.3380000000000001,"5min":3.3380000000000001,"15min":3.3380000000000001},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.0680000000000001,"15min":1.046},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.2050000000000001,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":1.2230000000000001}]},{"osd":5,"last update":"Wed Mar 24 21:52:55 2021","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.0329999999999999,"15min":1.0329999999999999},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.47,"5min":1.47,"15min":1.47},"last":1.2},{"interface":"front","average":{"1min":1.224,"5min":1.165,"15min":1.155},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":3.1509999999999998,"5min":3.1509999999999998,"15min":3.1509999999999998},"last":1.2430000000000001}]},{"osd":6,"last update":"Wed Mar 24 21:53:17 2021","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.256,"15min":1.262},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":2.2810000000000001,"5min":2.2810000000000001,"15min":2.2810000000000001},"last":1.552},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.1619999999999999,"15min":1.165},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":1.8069999999999999,"5min":2.016,"15min":2.016},"last":0.98599999999999999}]},{"osd":7,"last update":"Wed Mar 24 21:53:43 2021","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":1.2190000000000001,"15min":1.228},"min":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.6719999999999999,"5min":1.6719999999999999,"15min":1.6719999999999999},"last":1.1100000000000001},{"interface":"front","average":{"1min":1.2290000000000001,"5min":1.246,"15min":1.2490000000000001},"min":{"1min":0.80800000000000005,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":3.5419999999999998,"5min":3.5419999999999998,"15min":3.5419999999999998},"last":1.161}]}]},{"osd":3,"up_from":26,"seq":111669149742,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6828,"kb_used_data":932,"kb_used_omap":0,"kb_used_meta":5888,"kb_avail":93738324,"statfs":{"total":95995035648,"available":95988043776,"internally_reserved":0,"allocated":954368,"data_stored":318598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6029312},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:53:21 2021","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.63300000000000001,"15min":0.63500000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.92900000000000005,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.73299999999999998,"15min":0.72099999999999997},"min":{"1min":0.41499999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.93000000000000005}]},{"osd":1,"last update":"Wed Mar 24 21:53:21 2021","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.74199999999999999,"15min":0.73799999999999999},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.018,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.75600000000000001,"15min":0.755},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.97399999999999998,"5min":1.103,"15min":1.103},"last":0.89800000000000002}]},{"osd":2,"last update":"Wed Mar 24 21:53:21 2021","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.79700000000000004,"15min":0.78700000000000003},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.77500000000000002,"15min":0.76500000000000001},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0449999999999999,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.95599999999999996}]},{"osd":4,"last update":"Wed Mar 24 21:53:40 2021","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.95399999999999996,"15min":0.95399999999999996},"min":{"1min":0.66400000000000003,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.1950000000000001,"5min":1.702,"15min":1.702},"last":1.0629999999999999},{"interface":"front","average":{"1min":1.016,"5min":0.96599999999999997,"15min":0.94399999999999995},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.671,"5min":1.671,"15min":1.671},"last":1.0960000000000001}]},{"osd":5,"last update":"Wed Mar 24 21:52:55 2021","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.0669999999999999,"15min":1.0620000000000001},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":2.423,"5min":2.423,"15min":2.423},"last":1.1279999999999999},{"interface":"front","average":{"1min":0.98999999999999999,"5min":1.0740000000000001,"15min":1.0880000000000001},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.2889999999999999,"5min":1.496,"15min":1.496},"last":1.1930000000000001}]},{"osd":6,"last update":"Wed Mar 24 21:53:21 2021","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.1180000000000001,"15min":1.1259999999999999},"min":{"1min":0.85499999999999998,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.49,"5min":1.49,"15min":1.49},"last":1.3220000000000001},{"interface":"front","average":{"1min":1.155,"5min":1.1539999999999999,"15min":1.1539999999999999},"min":{"1min":0.77600000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":1.7070000000000001,"5min":1.7070000000000001,"15min":1.7070000000000001},"last":1.204}]},{"osd":7,"last update":"Wed Mar 24 21:52:43 2021","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.077,"15min":1.077},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.425,"5min":1.425,"15min":1.425},"last":1.2270000000000001},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"last":1.363}]}]},{"osd":4,"up_from":32,"seq":137438953512,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6724,"kb_used_data":892,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738428,"statfs":{"total":95995035648,"available":95988150272,"internally_reserved":0,"allocated":913408,"data_stored":317246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:53:36 2021","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.84699999999999998,"15min":0.84099999999999997},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.2490000000000001,"5min":1.268,"15min":1.268},"last":1.153},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.89200000000000002,"15min":0.86199999999999999},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.667,"5min":1.667,"15min":1.667},"last":1.165}]},{"osd":1,"last update":"Wed Mar 24 21:53:36 2021","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.92000000000000004,"15min":0.91800000000000004},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.1439999999999999,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.996},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.86899999999999999,"15min":0.84699999999999998},"min":{"1min":0.59199999999999997,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.337,"5min":1.337,"15min":1.337},"last":0.77900000000000003}]},{"osd":2,"last update":"Wed Mar 24 21:53:36 2021","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.90700000000000003,"15min":0.89900000000000002},"min":{"1min":0.57699999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.6279999999999999,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.878,"5min":0.92800000000000005,"15min":0.91200000000000003},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.169,"5min":1.335,"15min":1.335},"last":1.1299999999999999}]},{"osd":3,"last update":"Wed Mar 24 21:53:36 2021","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.92800000000000005,"15min":0.93400000000000005},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.1359999999999999,"5min":1.601,"15min":1.601},"last":0.872},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.97099999999999997,"15min":0.95299999999999996},"min":{"1min":0.754,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.474,"5min":1.474,"15min":1.474},"last":0.93700000000000006}]},{"osd":5,"last update":"Wed Mar 24 21:52:57 2021","interfaces":[{"interface":"back","average":{"1min":1.0149999999999999,"5min":0.96299999999999997,"15min":0.95499999999999996},"min":{"1min":0.59599999999999997,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":1.0660000000000001},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.026,"15min":1},"min":{"1min":0.78300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":2.0760000000000001,"5min":2.0760000000000001,"15min":2.0760000000000001},"last":1.23}]},{"osd":6,"last update":"Wed Mar 24 21:53:18 2021","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":1.0620000000000001,"15min":1.0640000000000001},"min":{"1min":0.79900000000000004,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.2749999999999999,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":1.1859999999999999},{"interface":"front","average":{"1min":1.018,"5min":1.056,"15min":1.0620000000000001},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.357,"5min":1.506,"15min":1.506},"last":1.0469999999999999}]},{"osd":7,"last update":"Wed Mar 24 21:53:41 2021","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":1.1679999999999999,"15min":1.1870000000000001},"min":{"1min":0.624,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.587,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":1.109},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.1910000000000001,"15min":1.2010000000000001},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.702,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":1.2789999999999999}]}]},{"osd":5,"up_from":39,"seq":167503724581,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6404,"kb_used_data":764,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738748,"statfs":{"total":95995035648,"available":95988477952,"internally_reserved":0,"allocated":782336,"data_stored":312209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:52:56 2021","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.89000000000000001,"15min":0.88900000000000001},"min":{"1min":0.623,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":2.29,"5min":2.29,"15min":2.29},"last":0.84599999999999997},{"interface":"front","average":{"1min":1.0489999999999999,"5min":0.91700000000000004,"15min":0.89500000000000002},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":3.1779999999999999,"5min":3.1779999999999999,"15min":3.1779999999999999},"last":1.111}]},{"osd":1,"last update":"Wed Mar 24 21:52:56 2021","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":0.93200000000000005,"15min":0.92500000000000004},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":2.8690000000000002,"5min":2.8690000000000002,"15min":2.8690000000000002},"last":0.97599999999999998},{"interface":"front","average":{"1min":0.94199999999999995,"5min":0.93400000000000005,"15min":0.93300000000000005},"min":{"1min":0.66600000000000004,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":0.80000000000000004}]},{"osd":2,"last update":"Wed Mar 24 21:52:56 2021","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.90000000000000002,"15min":0.90200000000000002},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":1.161},{"interface":"front","average":{"1min":1.1839999999999999,"5min":0.94699999999999995,"15min":0.90700000000000003},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":5.8479999999999999,"5min":5.8479999999999999,"15min":5.8479999999999999},"last":0.89600000000000002}]},{"osd":3,"last update":"Wed Mar 24 21:52:56 2021","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":0.94399999999999995,"15min":0.93600000000000005},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":3.1259999999999999,"5min":3.1259999999999999,"15min":3.1259999999999999},"last":1.056},{"interface":"front","average":{"1min":1.0409999999999999,"5min":0.95999999999999996,"15min":0.94599999999999995},"min":{"1min":0.67400000000000004,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":2.6850000000000001,"5min":2.6850000000000001,"15min":2.6850000000000001},"last":1.081}]},{"osd":4,"last update":"Wed Mar 24 21:52:56 2021","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"min":{"1min":0.55800000000000005,"5min":0.439,"15min":0.439},"max":{"1min":1.1519999999999999,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":1.1890000000000001},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.90700000000000003,"15min":0.89500000000000002},"min":{"1min":0.71499999999999997,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.171,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":1.0169999999999999}]},{"osd":6,"last update":"Wed Mar 24 21:53:19 2021","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":0.89200000000000002,"15min":0.87},"min":{"1min":0.71999999999999997,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":1.2829999999999999},{"interface":"front","average":{"1min":1.153,"5min":0.99299999999999999,"15min":0.96699999999999997},"min":{"1min":0.72299999999999998,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":2.3079999999999998,"5min":2.3079999999999998,"15min":2.3079999999999998},"last":1.2130000000000001}]},{"osd":7,"last update":"Wed Mar 24 21:53:42 2021","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.0549999999999999,"15min":1.0469999999999999},"min":{"1min":0.82899999999999996,"5min":0.624,"15min":0.624},"max":{"1min":1.484,"5min":1.484,"15min":1.484},"last":1.268},{"interface":"front","average":{"1min":1.196,"5min":1.0189999999999999,"15min":0.98899999999999999},"min":{"1min":0.88800000000000001,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":2.1539999999999999,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":1.333}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":7420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":339968,"data_stored":12094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":6670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":11712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":274432,"data_stored":9910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":5638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":9628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":7500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-03-24T21:53:48.030 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph pg dump --format=json 2021-03-24T21:53:48.349 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:53:48.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:48 smithi142 conmon[20125]: cluster 2021-03-24T21:53:46.687787+0000 mgr.y (mgr.14144) 282 : cluster [DBG] 2021-03-24T21:53:48.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:48 smithi142 conmon[20125]: pgmap v283: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-24T21:53:48.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:48 smithi142 conmon[20125]: audit 2021-03-24T21:53:47.139738 2021-03-24T21:53:48.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:48 smithi142 conmon[20125]: +0000 mgr.y (mgr.14144) 283 : audit [DBG] from='client.14667 v1:172.21.15.125:0/3998792218' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T21:53:48.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:48 smithi125 conmon[19936]: cluster 2021-03-24T21:53:46.687787+0000 mgr.y (mgr.14144) 282 : cluster 2021-03-24T21:53:48.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:48 smithi125 conmon[19936]: [DBG] pgmap v283: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-24T21:53:48.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:48 smithi125 conmon[19936]: audit 2021-03-24T21:53:47.139738+0000 mgr.y (mgr.14144) 283 : audit [DBG] from='client.14667 v1:172.21.15.125:0/3998792218' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T21:53:48.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:48 smithi125 conmon[24927]: cluster 2021-03-24T21:53 2021-03-24T21:53:48.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:48 smithi125 conmon[24927]: :46.687787+0000 mgr.y (mgr.14144) 282 : cluster [DBG] pgmap v283: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-24T21:53:48.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:48 smithi125 conmon[24927]: audit 2021-03-24T21:53:47.139738+0000 2021-03-24T21:53:48.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:48 smithi125 conmon[24927]: mgr.y (mgr.14144) 283 : audit [DBG] from='client.14667 v1:172.21.15.125:0/3998792218' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T21:53:48.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:48 smithi125 conmon[24927]: 2021-03-24T21:53:49.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:49 smithi142 conmon[20125]: cluster 2021-03-24T21:53:47. 2021-03-24T21:53:49.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:49 smithi142 conmon[20125]: 540783+0000 osd.3 (osd.3) 32 : cluster [DBG] 6.13 deep-scrub ok 2021-03-24T21:53:49.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:49 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:53:49.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:49 smithi125 conmon[19936]: 21:53:47.540783+0000 osd.3 (osd.3) 32 : cluster [DBG] 6.13 deep-scrub ok 2021-03-24T21:53:49.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:49 smithi125 conmon[24927]: cluster 2021-03-24T21:53 2021-03-24T21:53:49.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:49 smithi125 conmon[24927]: :47.540783+0000 osd.3 (osd.3) 32 : cluster [DBG] 6.13 deep-scrub ok 2021-03-24T21:53:50.159 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:50 smithi125 conmon[19936]: cluster 2021-03-24T21:53:50.159 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:50 smithi125 conmon[19936]: 2021-03- 2021-03-24T21:53:50.159 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:50 smithi125 conmon[19936]: 24T21:53:48.329471+0000 osd.4 (osd.4) 30 : cluster [DBG] 5.18 scrub ok 2021-03-24T21:53:50.160 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:50 smithi125 conmon[19936]: cluster 2021-03-24T21:53:50.160 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:50 smithi125 conmon[19936]: 2021- 2021-03-24T21:53:50.160 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:50 smithi125 conmon[19936]: 03-24T21:53:48.688751+0000 mgr.y (mgr.14144 2021-03-24T21:53:50.160 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:50 smithi125 conmon[19936]: ) 284 : cluster [DBG] pgmap v284: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 911 B/s rd, 0 op/s 2021-03-24T21:53:50.161 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:50 smithi125 conmon[24927]: cluster 2021-03-24T21:53:50.161 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:50 smithi125 conmon[24927]: 2021-03-24T21:53:50.161 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:50 smithi125 conmon[24927]: 2021 2021-03-24T21:53:50.162 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:50 smithi125 conmon[24927]: - 2021-03-24T21:53:50.162 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:50 smithi125 conmon[24927]: 03 2021-03-24T21:53:50.162 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:50 smithi125 conmon[24927]: - 2021-03-24T21:53:50.162 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:50 smithi125 conmon[24927]: 24T21:53:48.329471+0000 osd.4 (osd. 2021-03-24T21:53:50.162 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:50 smithi125 conmon[24927]: 4) 30 : cluster [DBG] 5.18 scrub ok 2021-03-24T21:53:50.163 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:50 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:53:50.163 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:50 smithi125 conmon[24927]: 21:53:48.688751+0000 mgr.y (mgr.14144) 284 : cluster [DBG] pgmap v284: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 911 B/s rd, 0 op/s 2021-03-24T21:53:50.358 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-24T21:53:50.360 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2021-03-24T21:53:50.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:50 smithi142 conmon[20125]: cluster 2021-03-24T21:53:48.329471+0000 osd.4 (osd. 2021-03-24T21:53:50.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:50 smithi142 conmon[20125]: 4) 30 : cluster [DBG] 5.18 scrub ok 2021-03-24T21:53:50.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:50 smithi142 conmon[20125]: cluster 2021-03-24T21:53:48.688751+0000 mgr.y (mgr.14144) 284 : cluster [DBG] pgmap v284: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 911 B/s rd, 0 op/s 2021-03-24T21:53:51.179 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":284,"stamp":"2021-03-24T21:53:48.688099+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":25236,"num_objects":316,"num_object_clones":0,"num_object_copies":948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":316,"num_whiteouts":0,"num_read":1191,"num_read_kb":691,"num_write":550,"num_write_kb":165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":2412,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":570,"ondisk_log_size":570,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":53328,"kb_used_data":6864,"kb_used_omap":0,"kb_used_meta":46400,"kb_avail":749907888,"statfs":{"total":767960285184,"available":767905677312,"internally_reserved":0,"allocated":7028736,"data_stored":2529507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":47513600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.239306"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":"38","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.421899+0000","last_change":"2021-03-24T21:52:59.180417+0000","last_active":"2021-03-24T21:53:31.421899+0000","last_peered":"2021-03-24T21:53:31.421899+0000","last_clean":"2021-03-24T21:53:31.421899+0000","last_became_active":"2021-03-24T21:51:51.759485+0000","last_became_peered":"2021-03-24T21:51:51.759485+0000","last_unstale":"2021-03-24T21:53:31.421899+0000","last_undegraded":"2021-03-24T21:53:31.421899+0000","last_fullsized":"2021-03-24T21:53:31.421899+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:59.180240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:52:59.180240+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"41","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.425115+0000","last_change":"2021-03-24T21:53:02.553145+0000","last_active":"2021-03-24T21:53:31.425115+0000","last_peered":"2021-03-24T21:53:31.425115+0000","last_clean":"2021-03-24T21:53:31.425115+0000","last_became_active":"2021-03-24T21:51:49.415254+0000","last_became_peered":"2021-03-24T21:51:49.415254+0000","last_unstale":"2021-03-24T21:53:31.425115+0000","last_undegraded":"2021-03-24T21:53:31.425115+0000","last_fullsized":"2021-03-24T21:53:31.425115+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:02.553036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:02.553036+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":"43","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.651528+0000","last_change":"2021-03-24T21:53:12.428679+0000","last_active":"2021-03-24T21:53:31.651528+0000","last_peered":"2021-03-24T21:53:31.651528+0000","last_clean":"2021-03-24T21:53:31.651528+0000","last_became_active":"2021-03-24T21:51:45.581000+0000","last_became_peered":"2021-03-24T21:51:45.581000+0000","last_unstale":"2021-03-24T21:53:31.651528+0000","last_undegraded":"2021-03-24T21:53:31.651528+0000","last_fullsized":"2021-03-24T21:53:31.651528+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:12.428497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:12.428497+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"69'15","reported_seq":"70","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:41.296073+0000","last_change":"2021-03-24T21:53:41.296073+0000","last_active":"2021-03-24T21:53:41.296073+0000","last_peered":"2021-03-24T21:53:41.296073+0000","last_clean":"2021-03-24T21:53:41.296073+0000","last_became_active":"2021-03-24T21:51:47.410353+0000","last_became_peered":"2021-03-24T21:51:47.410353+0000","last_unstale":"2021-03-24T21:53:41.296073+0000","last_undegraded":"2021-03-24T21:53:41.296073+0000","last_fullsized":"2021-03-24T21:53:41.296073+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:53:41.295875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:41.295875+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"38","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.664833+0000","last_change":"2021-03-24T21:52:58.676144+0000","last_active":"2021-03-24T21:53:31.664833+0000","last_peered":"2021-03-24T21:53:31.664833+0000","last_clean":"2021-03-24T21:53:31.664833+0000","last_became_active":"2021-03-24T21:51:51.915108+0000","last_became_peered":"2021-03-24T21:51:51.915108+0000","last_unstale":"2021-03-24T21:53:31.664833+0000","last_undegraded":"2021-03-24T21:53:31.664833+0000","last_fullsized":"2021-03-24T21:53:31.664833+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:58.676060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:52:58.676060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"35","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.421623+0000","last_change":"2021-03-24T21:51:49.410454+0000","last_active":"2021-03-24T21:53:31.421623+0000","last_peered":"2021-03-24T21:53:31.421623+0000","last_clean":"2021-03-24T21:53:31.421623+0000","last_became_active":"2021-03-24T21:51:49.409015+0000","last_became_peered":"2021-03-24T21:51:49.409015+0000","last_unstale":"2021-03-24T21:53:31.421623+0000","last_undegraded":"2021-03-24T21:53:31.421623+0000","last_fullsized":"2021-03-24T21:53:31.421623+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:51:48.384232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":"44","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.665525+0000","last_change":"2021-03-24T21:53:18.706895+0000","last_active":"2021-03-24T21:53:31.665525+0000","last_peered":"2021-03-24T21:53:31.665525+0000","last_clean":"2021-03-24T21:53:31.665525+0000","last_became_active":"2021-03-24T21:51:45.581465+0000","last_became_peered":"2021-03-24T21:51:45.581465+0000","last_unstale":"2021-03-24T21:53:31.665525+0000","last_undegraded":"2021-03-24T21:53:31.665525+0000","last_fullsized":"2021-03-24T21:53:31.665525+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:18.706745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:18.706745+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"69'20","reported_seq":"77","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.664691+0000","last_change":"2021-03-24T21:53:12.604981+0000","last_active":"2021-03-24T21:53:31.664691+0000","last_peered":"2021-03-24T21:53:31.664691+0000","last_clean":"2021-03-24T21:53:31.664691+0000","last_became_active":"2021-03-24T21:51:47.405776+0000","last_became_peered":"2021-03-24T21:51:47.405776+0000","last_unstale":"2021-03-24T21:53:31.664691+0000","last_undegraded":"2021-03-24T21:53:31.664691+0000","last_fullsized":"2021-03-24T21:53:31.664691+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'20","last_scrub_stamp":"2021-03-24T21:53:12.604757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:12.604757+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":22,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"37","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.651120+0000","last_change":"2021-03-24T21:53:27.604422+0000","last_active":"2021-03-24T21:53:31.651120+0000","last_peered":"2021-03-24T21:53:31.651120+0000","last_clean":"2021-03-24T21:53:31.651120+0000","last_became_active":"2021-03-24T21:51:51.758026+0000","last_became_peered":"2021-03-24T21:51:51.758026+0000","last_unstale":"2021-03-24T21:53:31.651120+0000","last_undegraded":"2021-03-24T21:53:31.651120+0000","last_fullsized":"2021-03-24T21:53:31.651120+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:27.604230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:27.604230+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"41","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.873310+0000","last_change":"2021-03-24T21:52:56.855476+0000","last_active":"2021-03-24T21:53:31.873310+0000","last_peered":"2021-03-24T21:53:31.873310+0000","last_clean":"2021-03-24T21:53:31.873310+0000","last_became_active":"2021-03-24T21:51:49.413342+0000","last_became_peered":"2021-03-24T21:51:49.413342+0000","last_unstale":"2021-03-24T21:53:31.873310+0000","last_undegraded":"2021-03-24T21:53:31.873310+0000","last_fullsized":"2021-03-24T21:53:31.873310+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:56.855305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:52:56.855305+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":"45","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.873823+0000","last_change":"2021-03-24T21:52:50.788792+0000","last_active":"2021-03-24T21:53:31.873823+0000","last_peered":"2021-03-24T21:53:31.873823+0000","last_clean":"2021-03-24T21:53:31.873823+0000","last_became_active":"2021-03-24T21:51:45.382348+0000","last_became_peered":"2021-03-24T21:51:45.382348+0000","last_unstale":"2021-03-24T21:53:31.873823+0000","last_undegraded":"2021-03-24T21:53:31.873823+0000","last_fullsized":"2021-03-24T21:53:31.873823+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:50.788606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:52:50.788606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"69'13","reported_seq":"66","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.422100+0000","last_change":"2021-03-24T21:53:10.104959+0000","last_active":"2021-03-24T21:53:31.422100+0000","last_peered":"2021-03-24T21:53:31.422100+0000","last_clean":"2021-03-24T21:53:31.422100+0000","last_became_active":"2021-03-24T21:51:47.407488+0000","last_became_peered":"2021-03-24T21:51:47.407488+0000","last_unstale":"2021-03-24T21:53:31.422100+0000","last_undegraded":"2021-03-24T21:53:31.422100+0000","last_fullsized":"2021-03-24T21:53:31.422100+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2021-03-24T21:53:10.104806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:10.104806+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"39","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.872795+0000","last_change":"2021-03-24T21:53:23.896132+0000","last_active":"2021-03-24T21:53:31.872795+0000","last_peered":"2021-03-24T21:53:31.872795+0000","last_clean":"2021-03-24T21:53:31.872795+0000","last_became_active":"2021-03-24T21:51:51.765338+0000","last_became_peered":"2021-03-24T21:51:51.765338+0000","last_unstale":"2021-03-24T21:53:31.872795+0000","last_undegraded":"2021-03-24T21:53:31.872795+0000","last_fullsized":"2021-03-24T21:53:31.872795+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:23.895966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:23.895966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"39","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.897282+0000","last_change":"2021-03-24T21:53:00.845488+0000","last_active":"2021-03-24T21:53:31.897282+0000","last_peered":"2021-03-24T21:53:31.897282+0000","last_clean":"2021-03-24T21:53:31.897282+0000","last_became_active":"2021-03-24T21:51:49.411327+0000","last_became_peered":"2021-03-24T21:51:49.411327+0000","last_unstale":"2021-03-24T21:53:31.897282+0000","last_undegraded":"2021-03-24T21:53:31.897282+0000","last_fullsized":"2021-03-24T21:53:31.897282+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:00.845338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:00.845338+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":"45","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.873275+0000","last_change":"2021-03-24T21:52:45.798051+0000","last_active":"2021-03-24T21:53:31.873275+0000","last_peered":"2021-03-24T21:53:31.873275+0000","last_clean":"2021-03-24T21:53:31.873275+0000","last_became_active":"2021-03-24T21:51:45.381362+0000","last_became_peered":"2021-03-24T21:51:45.381362+0000","last_unstale":"2021-03-24T21:53:31.873275+0000","last_undegraded":"2021-03-24T21:53:31.873275+0000","last_fullsized":"2021-03-24T21:53:31.873275+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:45.797873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:52:45.797873+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"91'9","reported_seq":"57","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:33.290258+0000","last_change":"2021-03-24T21:53:33.290178+0000","last_active":"2021-03-24T21:53:33.290258+0000","last_peered":"2021-03-24T21:53:33.290258+0000","last_clean":"2021-03-24T21:53:33.290258+0000","last_became_active":"2021-03-24T21:51:47.416379+0000","last_became_peered":"2021-03-24T21:51:47.416379+0000","last_unstale":"2021-03-24T21:53:33.290258+0000","last_undegraded":"2021-03-24T21:53:33.290258+0000","last_fullsized":"2021-03-24T21:53:33.290258+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2021-03-24T21:53:33.290000+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2021-03-24T21:53:33.290000+0000","last_clean_scrub_stamp":"2021-03-24T21:53:33.290000+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":8,"num_write":7,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"38","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:39.277601+0000","last_change":"2021-03-24T21:53:39.277601+0000","last_active":"2021-03-24T21:53:39.277601+0000","last_peered":"2021-03-24T21:53:39.277601+0000","last_clean":"2021-03-24T21:53:39.277601+0000","last_became_active":"2021-03-24T21:51:51.910950+0000","last_became_peered":"2021-03-24T21:51:51.910950+0000","last_unstale":"2021-03-24T21:53:39.277601+0000","last_undegraded":"2021-03-24T21:53:39.277601+0000","last_fullsized":"2021-03-24T21:53:39.277601+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:39.277425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:39.277425+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"41","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.425640+0000","last_change":"2021-03-24T21:52:59.579706+0000","last_active":"2021-03-24T21:53:31.425640+0000","last_peered":"2021-03-24T21:53:31.425640+0000","last_clean":"2021-03-24T21:53:31.425640+0000","last_became_active":"2021-03-24T21:51:49.411914+0000","last_became_peered":"2021-03-24T21:51:49.411914+0000","last_unstale":"2021-03-24T21:53:31.425640+0000","last_undegraded":"2021-03-24T21:53:31.425640+0000","last_fullsized":"2021-03-24T21:53:31.425640+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:59.579528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:52:59.579528+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":"43","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.651619+0000","last_change":"2021-03-24T21:53:04.263170+0000","last_active":"2021-03-24T21:53:31.651619+0000","last_peered":"2021-03-24T21:53:31.651619+0000","last_clean":"2021-03-24T21:53:31.651619+0000","last_became_active":"2021-03-24T21:51:45.381533+0000","last_became_peered":"2021-03-24T21:51:45.381533+0000","last_unstale":"2021-03-24T21:53:31.651619+0000","last_undegraded":"2021-03-24T21:53:31.651619+0000","last_fullsized":"2021-03-24T21:53:31.651619+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:04.262989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:04.262989+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"69'15","reported_seq":"93","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.423314+0000","last_change":"2021-03-24T21:52:49.554069+0000","last_active":"2021-03-24T21:53:31.423314+0000","last_peered":"2021-03-24T21:53:31.423314+0000","last_clean":"2021-03-24T21:53:31.423314+0000","last_became_active":"2021-03-24T21:52:06.056116+0000","last_became_peered":"2021-03-24T21:52:06.056116+0000","last_unstale":"2021-03-24T21:53:31.423314+0000","last_undegraded":"2021-03-24T21:53:31.423314+0000","last_fullsized":"2021-03-24T21:53:31.423314+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:52:49.553951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:52:49.553951+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"33","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.665181+0000","last_change":"2021-03-24T21:51:51.915774+0000","last_active":"2021-03-24T21:53:31.665181+0000","last_peered":"2021-03-24T21:53:31.665181+0000","last_clean":"2021-03-24T21:53:31.665181+0000","last_became_active":"2021-03-24T21:51:51.914955+0000","last_became_peered":"2021-03-24T21:51:51.914955+0000","last_unstale":"2021-03-24T21:53:31.665181+0000","last_undegraded":"2021-03-24T21:53:31.665181+0000","last_fullsized":"2021-03-24T21:53:31.665181+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:51:50.731137+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"40","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.422406+0000","last_change":"2021-03-24T21:53:27.238846+0000","last_active":"2021-03-24T21:53:31.422406+0000","last_peered":"2021-03-24T21:53:31.422406+0000","last_clean":"2021-03-24T21:53:31.422406+0000","last_became_active":"2021-03-24T21:51:49.410920+0000","last_became_peered":"2021-03-24T21:51:49.410920+0000","last_unstale":"2021-03-24T21:53:31.422406+0000","last_undegraded":"2021-03-24T21:53:31.422406+0000","last_fullsized":"2021-03-24T21:53:31.422406+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:27.238614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:27.238614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":"44","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.422173+0000","last_change":"2021-03-24T21:52:54.212593+0000","last_active":"2021-03-24T21:53:31.422173+0000","last_peered":"2021-03-24T21:53:31.422173+0000","last_clean":"2021-03-24T21:53:31.422173+0000","last_became_active":"2021-03-24T21:51:45.579326+0000","last_became_peered":"2021-03-24T21:51:45.579326+0000","last_unstale":"2021-03-24T21:53:31.422173+0000","last_undegraded":"2021-03-24T21:53:31.422173+0000","last_fullsized":"2021-03-24T21:53:31.422173+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:54.212459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:52:54.212459+0000","last_clean_scrub_stamp":"2021-03-24T21:52:54.212459+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"69'16","reported_seq":"73","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:42.611713+0000","last_change":"2021-03-24T21:53:42.611713+0000","last_active":"2021-03-24T21:53:42.611713+0000","last_peered":"2021-03-24T21:53:42.611713+0000","last_clean":"2021-03-24T21:53:42.611713+0000","last_became_active":"2021-03-24T21:51:47.411418+0000","last_became_peered":"2021-03-24T21:51:47.411418+0000","last_unstale":"2021-03-24T21:53:42.611713+0000","last_undegraded":"2021-03-24T21:53:42.611713+0000","last_fullsized":"2021-03-24T21:53:42.611713+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T21:53:42.611540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:42.611540+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":37,"num_read_kb":20,"num_write":18,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"37","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.897890+0000","last_change":"2021-03-24T21:53:05.847583+0000","last_active":"2021-03-24T21:53:31.897890+0000","last_peered":"2021-03-24T21:53:31.897890+0000","last_clean":"2021-03-24T21:53:31.897890+0000","last_became_active":"2021-03-24T21:51:51.761492+0000","last_became_peered":"2021-03-24T21:51:51.761492+0000","last_unstale":"2021-03-24T21:53:31.897890+0000","last_undegraded":"2021-03-24T21:53:31.897890+0000","last_fullsized":"2021-03-24T21:53:31.897890+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:05.847412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:05.847412+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"65'2","reported_seq":"65","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:38.805577+0000","last_change":"2021-03-24T21:52:59.074600+0000","last_active":"2021-03-24T21:53:38.805577+0000","last_peered":"2021-03-24T21:53:38.805577+0000","last_clean":"2021-03-24T21:53:38.805577+0000","last_became_active":"2021-03-24T21:51:49.419519+0000","last_became_peered":"2021-03-24T21:51:49.419519+0000","last_unstale":"2021-03-24T21:53:38.805577+0000","last_undegraded":"2021-03-24T21:53:38.805577+0000","last_fullsized":"2021-03-24T21:53:38.805577+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2021-03-24T21:52:59.074417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:52:59.074417+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":"45","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.425196+0000","last_change":"2021-03-24T21:53:09.741058+0000","last_active":"2021-03-24T21:53:31.425196+0000","last_peered":"2021-03-24T21:53:31.425196+0000","last_clean":"2021-03-24T21:53:31.425196+0000","last_became_active":"2021-03-24T21:51:45.374975+0000","last_became_peered":"2021-03-24T21:51:45.374975+0000","last_unstale":"2021-03-24T21:53:31.425196+0000","last_undegraded":"2021-03-24T21:53:31.425196+0000","last_fullsized":"2021-03-24T21:53:31.425196+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:09.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:09.740867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"89'27","reported_seq":"79","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.652282+0000","last_change":"2021-03-24T21:53:21.585108+0000","last_active":"2021-03-24T21:53:31.652282+0000","last_peered":"2021-03-24T21:53:31.652282+0000","last_clean":"2021-03-24T21:53:31.652282+0000","last_became_active":"2021-03-24T21:51:47.688995+0000","last_became_peered":"2021-03-24T21:51:47.688995+0000","last_unstale":"2021-03-24T21:53:31.652282+0000","last_undegraded":"2021-03-24T21:53:31.652282+0000","last_fullsized":"2021-03-24T21:53:31.652282+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'27","last_scrub_stamp":"2021-03-24T21:53:21.584999+0000","last_deep_scrub":"89'27","last_deep_scrub_stamp":"2021-03-24T21:53:21.584999+0000","last_clean_scrub_stamp":"2021-03-24T21:53:21.584999+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"39","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.425714+0000","last_change":"2021-03-24T21:53:14.700106+0000","last_active":"2021-03-24T21:53:31.425714+0000","last_peered":"2021-03-24T21:53:31.425714+0000","last_clean":"2021-03-24T21:53:31.425714+0000","last_became_active":"2021-03-24T21:51:51.758605+0000","last_became_peered":"2021-03-24T21:51:51.758605+0000","last_unstale":"2021-03-24T21:53:31.425714+0000","last_undegraded":"2021-03-24T21:53:31.425714+0000","last_fullsized":"2021-03-24T21:53:31.425714+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:14.699852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:14.699852+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"39","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.652614+0000","last_change":"2021-03-24T21:53:30.660884+0000","last_active":"2021-03-24T21:53:31.652614+0000","last_peered":"2021-03-24T21:53:31.652614+0000","last_clean":"2021-03-24T21:53:31.652614+0000","last_became_active":"2021-03-24T21:51:49.409745+0000","last_became_peered":"2021-03-24T21:51:49.409745+0000","last_unstale":"2021-03-24T21:53:31.652614+0000","last_undegraded":"2021-03-24T21:53:31.652614+0000","last_fullsized":"2021-03-24T21:53:31.652614+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:30.660636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:30.660636+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":"44","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.663842+0000","last_change":"2021-03-24T21:53:11.648668+0000","last_active":"2021-03-24T21:53:31.663842+0000","last_peered":"2021-03-24T21:53:31.663842+0000","last_clean":"2021-03-24T21:53:31.663842+0000","last_became_active":"2021-03-24T21:51:45.373294+0000","last_became_peered":"2021-03-24T21:51:45.373294+0000","last_unstale":"2021-03-24T21:53:31.663842+0000","last_undegraded":"2021-03-24T21:53:31.663842+0000","last_fullsized":"2021-03-24T21:53:31.663842+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:11.648470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:11.648470+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"69'17","reported_seq":"106","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.638810+0000","last_change":"2021-03-24T21:53:17.246451+0000","last_active":"2021-03-24T21:53:31.638810+0000","last_peered":"2021-03-24T21:53:31.638810+0000","last_clean":"2021-03-24T21:53:31.638810+0000","last_became_active":"2021-03-24T21:52:06.054746+0000","last_became_peered":"2021-03-24T21:52:06.054746+0000","last_unstale":"2021-03-24T21:53:31.638810+0000","last_undegraded":"2021-03-24T21:53:31.638810+0000","last_fullsized":"2021-03-24T21:53:31.638810+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'17","last_scrub_stamp":"2021-03-24T21:53:17.246367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:17.246367+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":38,"num_read_kb":19,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"37","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.897223+0000","last_change":"2021-03-24T21:53:27.877846+0000","last_active":"2021-03-24T21:53:31.897223+0000","last_peered":"2021-03-24T21:53:31.897223+0000","last_clean":"2021-03-24T21:53:31.897223+0000","last_became_active":"2021-03-24T21:51:51.762150+0000","last_became_peered":"2021-03-24T21:51:51.762150+0000","last_unstale":"2021-03-24T21:53:31.897223+0000","last_undegraded":"2021-03-24T21:53:31.897223+0000","last_fullsized":"2021-03-24T21:53:31.897223+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:27.877627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:27.877627+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"40","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.637439+0000","last_change":"2021-03-24T21:53:13.143455+0000","last_active":"2021-03-24T21:53:31.637439+0000","last_peered":"2021-03-24T21:53:31.637439+0000","last_clean":"2021-03-24T21:53:31.637439+0000","last_became_active":"2021-03-24T21:51:49.414221+0000","last_became_peered":"2021-03-24T21:51:49.414221+0000","last_unstale":"2021-03-24T21:53:31.637439+0000","last_undegraded":"2021-03-24T21:53:31.637439+0000","last_fullsized":"2021-03-24T21:53:31.637439+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:13.143327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:13.143327+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":"43","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.651552+0000","last_change":"2021-03-24T21:53:10.429942+0000","last_active":"2021-03-24T21:53:31.651552+0000","last_peered":"2021-03-24T21:53:31.651552+0000","last_clean":"2021-03-24T21:53:31.651552+0000","last_became_active":"2021-03-24T21:51:45.379789+0000","last_became_peered":"2021-03-24T21:51:45.379789+0000","last_unstale":"2021-03-24T21:53:31.651552+0000","last_undegraded":"2021-03-24T21:53:31.651552+0000","last_fullsized":"2021-03-24T21:53:31.651552+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:10.429783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:10.429783+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"69'12","reported_seq":"95","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.663955+0000","last_change":"2021-03-24T21:52:52.579983+0000","last_active":"2021-03-24T21:53:31.663955+0000","last_peered":"2021-03-24T21:53:31.663955+0000","last_clean":"2021-03-24T21:53:31.663955+0000","last_became_active":"2021-03-24T21:52:06.053964+0000","last_became_peered":"2021-03-24T21:52:06.053964+0000","last_unstale":"2021-03-24T21:53:31.663955+0000","last_undegraded":"2021-03-24T21:53:31.663955+0000","last_fullsized":"2021-03-24T21:53:31.663955+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2021-03-24T21:52:52.579796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:52:52.579796+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":12,"num_write":12,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"33","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.664451+0000","last_change":"2021-03-24T21:51:51.912859+0000","last_active":"2021-03-24T21:53:31.664451+0000","last_peered":"2021-03-24T21:53:31.664451+0000","last_clean":"2021-03-24T21:53:31.664451+0000","last_became_active":"2021-03-24T21:51:51.911828+0000","last_became_peered":"2021-03-24T21:51:51.911828+0000","last_unstale":"2021-03-24T21:53:31.664451+0000","last_undegraded":"2021-03-24T21:53:31.664451+0000","last_fullsized":"2021-03-24T21:53:31.664451+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:51:50.731137+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"39","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.897228+0000","last_change":"2021-03-24T21:52:52.829549+0000","last_active":"2021-03-24T21:53:31.897228+0000","last_peered":"2021-03-24T21:53:31.897228+0000","last_clean":"2021-03-24T21:53:31.897228+0000","last_became_active":"2021-03-24T21:51:49.422591+0000","last_became_peered":"2021-03-24T21:51:49.422591+0000","last_unstale":"2021-03-24T21:53:31.897228+0000","last_undegraded":"2021-03-24T21:53:31.897228+0000","last_fullsized":"2021-03-24T21:53:31.897228+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:52.829381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:52:52.829381+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":"45","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.873993+0000","last_change":"2021-03-24T21:53:21.833111+0000","last_active":"2021-03-24T21:53:31.873993+0000","last_peered":"2021-03-24T21:53:31.873993+0000","last_clean":"2021-03-24T21:53:31.873993+0000","last_became_active":"2021-03-24T21:51:45.385892+0000","last_became_peered":"2021-03-24T21:51:45.385892+0000","last_unstale":"2021-03-24T21:53:31.873993+0000","last_undegraded":"2021-03-24T21:53:31.873993+0000","last_fullsized":"2021-03-24T21:53:31.873993+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:21.832921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:21.832921+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"86'22","reported_seq":"75","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.664519+0000","last_change":"2021-03-24T21:52:54.597687+0000","last_active":"2021-03-24T21:53:31.664519+0000","last_peered":"2021-03-24T21:53:31.664519+0000","last_clean":"2021-03-24T21:53:31.664519+0000","last_became_active":"2021-03-24T21:51:47.687561+0000","last_became_peered":"2021-03-24T21:51:47.687561+0000","last_unstale":"2021-03-24T21:53:31.664519+0000","last_undegraded":"2021-03-24T21:53:31.664519+0000","last_fullsized":"2021-03-24T21:53:31.664519+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'22","last_scrub_stamp":"2021-03-24T21:52:54.597505+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2021-03-24T21:52:54.597505+0000","last_clean_scrub_stamp":"2021-03-24T21:52:54.597505+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":16,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"37","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.650465+0000","last_change":"2021-03-24T21:53:13.462014+0000","last_active":"2021-03-24T21:53:31.650465+0000","last_peered":"2021-03-24T21:53:31.650465+0000","last_clean":"2021-03-24T21:53:31.650465+0000","last_became_active":"2021-03-24T21:51:51.757253+0000","last_became_peered":"2021-03-24T21:51:51.757253+0000","last_unstale":"2021-03-24T21:53:31.650465+0000","last_undegraded":"2021-03-24T21:53:31.650465+0000","last_fullsized":"2021-03-24T21:53:31.650465+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:13.461940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:13.461940+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"40","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.665594+0000","last_change":"2021-03-24T21:53:31.665546+0000","last_active":"2021-03-24T21:53:31.665594+0000","last_peered":"2021-03-24T21:53:31.665594+0000","last_clean":"2021-03-24T21:53:31.665594+0000","last_became_active":"2021-03-24T21:51:49.418515+0000","last_became_peered":"2021-03-24T21:51:49.418515+0000","last_unstale":"2021-03-24T21:53:31.665594+0000","last_undegraded":"2021-03-24T21:53:31.665594+0000","last_fullsized":"2021-03-24T21:53:31.665594+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:31.665434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:31.665434+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":"43","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.898116+0000","last_change":"2021-03-24T21:53:08.841487+0000","last_active":"2021-03-24T21:53:31.898116+0000","last_peered":"2021-03-24T21:53:31.898116+0000","last_clean":"2021-03-24T21:53:31.898116+0000","last_became_active":"2021-03-24T21:51:45.379383+0000","last_became_peered":"2021-03-24T21:51:45.379383+0000","last_unstale":"2021-03-24T21:53:31.898116+0000","last_undegraded":"2021-03-24T21:53:31.898116+0000","last_fullsized":"2021-03-24T21:53:31.898116+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:08.841310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:08.841310+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"69'15","reported_seq":"71","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.425868+0000","last_change":"2021-03-24T21:53:04.637959+0000","last_active":"2021-03-24T21:53:31.425868+0000","last_peered":"2021-03-24T21:53:31.425868+0000","last_clean":"2021-03-24T21:53:31.425868+0000","last_became_active":"2021-03-24T21:51:47.403130+0000","last_became_peered":"2021-03-24T21:51:47.403130+0000","last_unstale":"2021-03-24T21:53:31.425868+0000","last_undegraded":"2021-03-24T21:53:31.425868+0000","last_fullsized":"2021-03-24T21:53:31.425868+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:53:04.637877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:04.637877+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"38","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.664068+0000","last_change":"2021-03-24T21:53:23.708000+0000","last_active":"2021-03-24T21:53:31.664068+0000","last_peered":"2021-03-24T21:53:31.664068+0000","last_clean":"2021-03-24T21:53:31.664068+0000","last_became_active":"2021-03-24T21:51:51.764871+0000","last_became_peered":"2021-03-24T21:51:51.764871+0000","last_unstale":"2021-03-24T21:53:31.664068+0000","last_undegraded":"2021-03-24T21:53:31.664068+0000","last_fullsized":"2021-03-24T21:53:31.664068+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:23.707914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:23.707914+0000","last_clean_scrub_stamp":"2021-03-24T21:53:23.707914+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"41","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.425768+0000","last_change":"2021-03-24T21:52:52.702414+0000","last_active":"2021-03-24T21:53:31.425768+0000","last_peered":"2021-03-24T21:53:31.425768+0000","last_clean":"2021-03-24T21:53:31.425768+0000","last_became_active":"2021-03-24T21:51:49.415559+0000","last_became_peered":"2021-03-24T21:51:49.415559+0000","last_unstale":"2021-03-24T21:53:31.425768+0000","last_undegraded":"2021-03-24T21:53:31.425768+0000","last_fullsized":"2021-03-24T21:53:31.425768+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:52.702305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:52:52.702305+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":"44","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:36.317204+0000","last_change":"2021-03-24T21:53:36.317121+0000","last_active":"2021-03-24T21:53:36.317204+0000","last_peered":"2021-03-24T21:53:36.317204+0000","last_clean":"2021-03-24T21:53:36.317204+0000","last_became_active":"2021-03-24T21:51:45.379862+0000","last_became_peered":"2021-03-24T21:51:45.379862+0000","last_unstale":"2021-03-24T21:53:36.317204+0000","last_undegraded":"2021-03-24T21:53:36.317204+0000","last_fullsized":"2021-03-24T21:53:36.317204+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:36.316893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:36.316893+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"69'15","reported_seq":"69","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.422789+0000","last_change":"2021-03-24T21:53:21.207831+0000","last_active":"2021-03-24T21:53:31.422789+0000","last_peered":"2021-03-24T21:53:31.422789+0000","last_clean":"2021-03-24T21:53:31.422789+0000","last_became_active":"2021-03-24T21:51:47.416319+0000","last_became_peered":"2021-03-24T21:51:47.416319+0000","last_unstale":"2021-03-24T21:53:31.422789+0000","last_undegraded":"2021-03-24T21:53:31.422789+0000","last_fullsized":"2021-03-24T21:53:31.422789+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:53:21.207696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:21.207696+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"37","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.651788+0000","last_change":"2021-03-24T21:53:06.309234+0000","last_active":"2021-03-24T21:53:31.651788+0000","last_peered":"2021-03-24T21:53:31.651788+0000","last_clean":"2021-03-24T21:53:31.651788+0000","last_became_active":"2021-03-24T21:51:51.756584+0000","last_became_peered":"2021-03-24T21:51:51.756584+0000","last_unstale":"2021-03-24T21:53:31.651788+0000","last_undegraded":"2021-03-24T21:53:31.651788+0000","last_fullsized":"2021-03-24T21:53:31.651788+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:06.309066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:06.309066+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"86'3","reported_seq":"67","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:38.805416+0000","last_change":"2021-03-24T21:53:08.815321+0000","last_active":"2021-03-24T21:53:38.805416+0000","last_peered":"2021-03-24T21:53:38.805416+0000","last_clean":"2021-03-24T21:53:38.805416+0000","last_became_active":"2021-03-24T21:51:49.419663+0000","last_became_peered":"2021-03-24T21:51:49.419663+0000","last_unstale":"2021-03-24T21:53:38.805416+0000","last_undegraded":"2021-03-24T21:53:38.805416+0000","last_fullsized":"2021-03-24T21:53:38.805416+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'3","last_scrub_stamp":"2021-03-24T21:53:08.815163+0000","last_deep_scrub":"86'3","last_deep_scrub_stamp":"2021-03-24T21:53:08.815163+0000","last_clean_scrub_stamp":"2021-03-24T21:53:08.815163+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":"43","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.897500+0000","last_change":"2021-03-24T21:53:15.868395+0000","last_active":"2021-03-24T21:53:31.897500+0000","last_peered":"2021-03-24T21:53:31.897500+0000","last_clean":"2021-03-24T21:53:31.897500+0000","last_became_active":"2021-03-24T21:51:45.579271+0000","last_became_peered":"2021-03-24T21:51:45.579271+0000","last_unstale":"2021-03-24T21:53:31.897500+0000","last_undegraded":"2021-03-24T21:53:31.897500+0000","last_fullsized":"2021-03-24T21:53:31.897500+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:15.868303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:15.868303+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"69'14","reported_seq":"65","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.665534+0000","last_change":"2021-03-24T21:53:02.723125+0000","last_active":"2021-03-24T21:53:31.665534+0000","last_peered":"2021-03-24T21:53:31.665534+0000","last_clean":"2021-03-24T21:53:31.665534+0000","last_became_active":"2021-03-24T21:51:47.686103+0000","last_became_peered":"2021-03-24T21:51:47.686103+0000","last_unstale":"2021-03-24T21:53:31.665534+0000","last_undegraded":"2021-03-24T21:53:31.665534+0000","last_fullsized":"2021-03-24T21:53:31.665534+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'14","last_scrub_stamp":"2021-03-24T21:53:02.722970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:02.722970+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":27,"num_read_kb":14,"num_write":14,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"38","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.423236+0000","last_change":"2021-03-24T21:53:12.030422+0000","last_active":"2021-03-24T21:53:31.423236+0000","last_peered":"2021-03-24T21:53:31.423236+0000","last_clean":"2021-03-24T21:53:31.423236+0000","last_became_active":"2021-03-24T21:51:51.762876+0000","last_became_peered":"2021-03-24T21:51:51.762876+0000","last_unstale":"2021-03-24T21:53:31.423236+0000","last_undegraded":"2021-03-24T21:53:31.423236+0000","last_fullsized":"2021-03-24T21:53:31.423236+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:12.030251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:12.030251+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"86'3","reported_seq":"66","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:38.805692+0000","last_change":"2021-03-24T21:52:56.678231+0000","last_active":"2021-03-24T21:53:38.805692+0000","last_peered":"2021-03-24T21:53:38.805692+0000","last_clean":"2021-03-24T21:53:38.805692+0000","last_became_active":"2021-03-24T21:51:49.419797+0000","last_became_peered":"2021-03-24T21:51:49.419797+0000","last_unstale":"2021-03-24T21:53:38.805692+0000","last_undegraded":"2021-03-24T21:53:38.805692+0000","last_fullsized":"2021-03-24T21:53:38.805692+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'3","last_scrub_stamp":"2021-03-24T21:52:56.678111+0000","last_deep_scrub":"86'3","last_deep_scrub_stamp":"2021-03-24T21:52:56.678111+0000","last_clean_scrub_stamp":"2021-03-24T21:52:56.678111+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":"43","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.650894+0000","last_change":"2021-03-24T21:52:51.450624+0000","last_active":"2021-03-24T21:53:31.650894+0000","last_peered":"2021-03-24T21:53:31.650894+0000","last_clean":"2021-03-24T21:53:31.650894+0000","last_became_active":"2021-03-24T21:51:45.381516+0000","last_became_peered":"2021-03-24T21:51:45.381516+0000","last_unstale":"2021-03-24T21:53:31.650894+0000","last_undegraded":"2021-03-24T21:53:31.650894+0000","last_fullsized":"2021-03-24T21:53:31.650894+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:51.450347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:52:51.450347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"69'13","reported_seq":"67","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.872410+0000","last_change":"2021-03-24T21:53:15.901347+0000","last_active":"2021-03-24T21:53:31.872410+0000","last_peered":"2021-03-24T21:53:31.872410+0000","last_clean":"2021-03-24T21:53:31.872410+0000","last_became_active":"2021-03-24T21:51:47.689288+0000","last_became_peered":"2021-03-24T21:51:47.689288+0000","last_unstale":"2021-03-24T21:53:31.872410+0000","last_undegraded":"2021-03-24T21:53:31.872410+0000","last_fullsized":"2021-03-24T21:53:31.872410+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2021-03-24T21:53:15.901150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:15.901150+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"39","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.423555+0000","last_change":"2021-03-24T21:53:15.623047+0000","last_active":"2021-03-24T21:53:31.423555+0000","last_peered":"2021-03-24T21:53:31.423555+0000","last_clean":"2021-03-24T21:53:31.423555+0000","last_became_active":"2021-03-24T21:51:51.754085+0000","last_became_peered":"2021-03-24T21:51:51.754085+0000","last_unstale":"2021-03-24T21:53:31.423555+0000","last_undegraded":"2021-03-24T21:53:31.423555+0000","last_fullsized":"2021-03-24T21:53:31.423555+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:15.622946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:15.622946+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"40","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.665904+0000","last_change":"2021-03-24T21:52:57.661001+0000","last_active":"2021-03-24T21:53:31.665904+0000","last_peered":"2021-03-24T21:53:31.665904+0000","last_clean":"2021-03-24T21:53:31.665904+0000","last_became_active":"2021-03-24T21:51:49.411882+0000","last_became_peered":"2021-03-24T21:51:49.411882+0000","last_unstale":"2021-03-24T21:53:31.665904+0000","last_undegraded":"2021-03-24T21:53:31.665904+0000","last_fullsized":"2021-03-24T21:53:31.665904+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:57.660827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:52:57.660827+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":"43","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.897188+0000","last_change":"2021-03-24T21:53:12.871443+0000","last_active":"2021-03-24T21:53:31.897188+0000","last_peered":"2021-03-24T21:53:31.897188+0000","last_clean":"2021-03-24T21:53:31.897188+0000","last_became_active":"2021-03-24T21:51:45.378937+0000","last_became_peered":"2021-03-24T21:51:45.378937+0000","last_unstale":"2021-03-24T21:53:31.897188+0000","last_undegraded":"2021-03-24T21:53:31.897188+0000","last_fullsized":"2021-03-24T21:53:31.897188+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:12.871269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:12.871269+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"69'16","reported_seq":"71","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.652435+0000","last_change":"2021-03-24T21:53:00.300847+0000","last_active":"2021-03-24T21:53:31.652435+0000","last_peered":"2021-03-24T21:53:31.652435+0000","last_clean":"2021-03-24T21:53:31.652435+0000","last_became_active":"2021-03-24T21:51:47.687595+0000","last_became_peered":"2021-03-24T21:51:47.687595+0000","last_unstale":"2021-03-24T21:53:31.652435+0000","last_undegraded":"2021-03-24T21:53:31.652435+0000","last_fullsized":"2021-03-24T21:53:31.652435+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T21:53:00.300673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:00.300673+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":38,"num_read_kb":18,"num_write":17,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"37","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.898110+0000","last_change":"2021-03-24T21:53:24.913418+0000","last_active":"2021-03-24T21:53:31.898110+0000","last_peered":"2021-03-24T21:53:31.898110+0000","last_clean":"2021-03-24T21:53:31.898110+0000","last_became_active":"2021-03-24T21:51:51.914615+0000","last_became_peered":"2021-03-24T21:51:51.914615+0000","last_unstale":"2021-03-24T21:53:31.898110+0000","last_undegraded":"2021-03-24T21:53:31.898110+0000","last_fullsized":"2021-03-24T21:53:31.898110+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:24.913238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:24.913238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"41","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.874139+0000","last_change":"2021-03-24T21:53:19.812985+0000","last_active":"2021-03-24T21:53:31.874139+0000","last_peered":"2021-03-24T21:53:31.874139+0000","last_clean":"2021-03-24T21:53:31.874139+0000","last_became_active":"2021-03-24T21:51:49.423159+0000","last_became_peered":"2021-03-24T21:51:49.423159+0000","last_unstale":"2021-03-24T21:53:31.874139+0000","last_undegraded":"2021-03-24T21:53:31.874139+0000","last_fullsized":"2021-03-24T21:53:31.874139+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:19.812822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:19.812822+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":"44","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.637365+0000","last_change":"2021-03-24T21:53:02.100432+0000","last_active":"2021-03-24T21:53:31.637365+0000","last_peered":"2021-03-24T21:53:31.637365+0000","last_clean":"2021-03-24T21:53:31.637365+0000","last_became_active":"2021-03-24T21:51:45.377984+0000","last_became_peered":"2021-03-24T21:51:45.377984+0000","last_unstale":"2021-03-24T21:53:31.637365+0000","last_undegraded":"2021-03-24T21:53:31.637365+0000","last_fullsized":"2021-03-24T21:53:31.637365+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:02.100243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:02.100243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"69'7","reported_seq":"53","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.665783+0000","last_change":"2021-03-24T21:53:13.646018+0000","last_active":"2021-03-24T21:53:31.665783+0000","last_peered":"2021-03-24T21:53:31.665783+0000","last_clean":"2021-03-24T21:53:31.665783+0000","last_became_active":"2021-03-24T21:51:47.687445+0000","last_became_peered":"2021-03-24T21:51:47.687445+0000","last_unstale":"2021-03-24T21:53:31.665783+0000","last_undegraded":"2021-03-24T21:53:31.665783+0000","last_fullsized":"2021-03-24T21:53:31.665783+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'7","last_scrub_stamp":"2021-03-24T21:53:13.645928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:13.645928+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":7,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"39","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.873441+0000","last_change":"2021-03-24T21:53:09.854407+0000","last_active":"2021-03-24T21:53:31.873441+0000","last_peered":"2021-03-24T21:53:31.873441+0000","last_clean":"2021-03-24T21:53:31.873441+0000","last_became_active":"2021-03-24T21:51:51.913733+0000","last_became_peered":"2021-03-24T21:51:51.913733+0000","last_unstale":"2021-03-24T21:53:31.873441+0000","last_undegraded":"2021-03-24T21:53:31.873441+0000","last_fullsized":"2021-03-24T21:53:31.873441+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:09.854304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:09.854304+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"89'4","reported_seq":"48","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.898706+0000","last_change":"2021-03-24T21:53:19.926732+0000","last_active":"2021-03-24T21:53:31.898706+0000","last_peered":"2021-03-24T21:53:31.898706+0000","last_clean":"2021-03-24T21:53:31.898706+0000","last_became_active":"2021-03-24T21:51:45.385110+0000","last_became_peered":"2021-03-24T21:51:45.385110+0000","last_unstale":"2021-03-24T21:53:31.898706+0000","last_undegraded":"2021-03-24T21:53:31.898706+0000","last_fullsized":"2021-03-24T21:53:31.898706+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'4","last_scrub_stamp":"2021-03-24T21:53:19.926637+0000","last_deep_scrub":"89'4","last_deep_scrub_stamp":"2021-03-24T21:53:19.926637+0000","last_clean_scrub_stamp":"2021-03-24T21:53:19.926637+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"69'18","reported_seq":"74","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.872294+0000","last_change":"2021-03-24T21:52:57.861464+0000","last_active":"2021-03-24T21:53:31.872294+0000","last_peered":"2021-03-24T21:53:31.872294+0000","last_clean":"2021-03-24T21:53:31.872294+0000","last_became_active":"2021-03-24T21:51:47.685994+0000","last_became_peered":"2021-03-24T21:51:47.685994+0000","last_unstale":"2021-03-24T21:53:31.872294+0000","last_undegraded":"2021-03-24T21:53:31.872294+0000","last_fullsized":"2021-03-24T21:53:31.872294+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'18","last_scrub_stamp":"2021-03-24T21:52:57.861253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:52:57.861253+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":20,"num_write":19,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"65'2","reported_seq":"64","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:38.805536+0000","last_change":"2021-03-24T21:53:03.836648+0000","last_active":"2021-03-24T21:53:38.805536+0000","last_peered":"2021-03-24T21:53:38.805536+0000","last_clean":"2021-03-24T21:53:38.805536+0000","last_became_active":"2021-03-24T21:51:49.415167+0000","last_became_peered":"2021-03-24T21:51:49.415167+0000","last_unstale":"2021-03-24T21:53:38.805536+0000","last_undegraded":"2021-03-24T21:53:38.805536+0000","last_fullsized":"2021-03-24T21:53:38.805536+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2021-03-24T21:53:03.836463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:03.836463+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"38","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.664849+0000","last_change":"2021-03-24T21:53:22.725035+0000","last_active":"2021-03-24T21:53:31.664849+0000","last_peered":"2021-03-24T21:53:31.664849+0000","last_clean":"2021-03-24T21:53:31.664849+0000","last_became_active":"2021-03-24T21:51:51.762035+0000","last_became_peered":"2021-03-24T21:51:51.762035+0000","last_unstale":"2021-03-24T21:53:31.664849+0000","last_undegraded":"2021-03-24T21:53:31.664849+0000","last_fullsized":"2021-03-24T21:53:31.664849+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:22.724840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:22.724840+0000","last_clean_scrub_stamp":"2021-03-24T21:53:22.724840+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":"43","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.898481+0000","last_change":"2021-03-24T21:52:46.847404+0000","last_active":"2021-03-24T21:53:31.898481+0000","last_peered":"2021-03-24T21:53:31.898481+0000","last_clean":"2021-03-24T21:53:31.898481+0000","last_became_active":"2021-03-24T21:51:45.380984+0000","last_became_peered":"2021-03-24T21:51:45.380984+0000","last_unstale":"2021-03-24T21:53:31.898481+0000","last_undegraded":"2021-03-24T21:53:31.898481+0000","last_fullsized":"2021-03-24T21:53:31.898481+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:46.847232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:52:46.847232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"69'15","reported_seq":"71","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:37.334642+0000","last_change":"2021-03-24T21:53:37.334527+0000","last_active":"2021-03-24T21:53:37.334642+0000","last_peered":"2021-03-24T21:53:37.334642+0000","last_clean":"2021-03-24T21:53:37.334642+0000","last_became_active":"2021-03-24T21:51:47.410650+0000","last_became_peered":"2021-03-24T21:51:47.410650+0000","last_unstale":"2021-03-24T21:53:37.334642+0000","last_undegraded":"2021-03-24T21:53:37.334642+0000","last_fullsized":"2021-03-24T21:53:37.334642+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:53:37.334345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:37.334345+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":19,"num_write":17,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"41","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.422753+0000","last_change":"2021-03-24T21:53:18.557275+0000","last_active":"2021-03-24T21:53:31.422753+0000","last_peered":"2021-03-24T21:53:31.422753+0000","last_clean":"2021-03-24T21:53:31.422753+0000","last_became_active":"2021-03-24T21:51:49.415403+0000","last_became_peered":"2021-03-24T21:51:49.415403+0000","last_unstale":"2021-03-24T21:53:31.422753+0000","last_undegraded":"2021-03-24T21:53:31.422753+0000","last_fullsized":"2021-03-24T21:53:31.422753+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:18.557122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:18.557122+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"37","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.896739+0000","last_change":"2021-03-24T21:53:02.842718+0000","last_active":"2021-03-24T21:53:31.896739+0000","last_peered":"2021-03-24T21:53:31.896739+0000","last_clean":"2021-03-24T21:53:31.896739+0000","last_became_active":"2021-03-24T21:51:51.762513+0000","last_became_peered":"2021-03-24T21:51:51.762513+0000","last_unstale":"2021-03-24T21:53:31.896739+0000","last_undegraded":"2021-03-24T21:53:31.896739+0000","last_fullsized":"2021-03-24T21:53:31.896739+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:02.842537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:02.842537+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"41","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.423107+0000","last_change":"2021-03-24T21:52:59.521026+0000","last_active":"2021-03-24T21:53:31.423107+0000","last_peered":"2021-03-24T21:53:31.423107+0000","last_clean":"2021-03-24T21:53:31.423107+0000","last_became_active":"2021-03-24T21:51:49.417699+0000","last_became_peered":"2021-03-24T21:51:49.417699+0000","last_unstale":"2021-03-24T21:53:31.423107+0000","last_undegraded":"2021-03-24T21:53:31.423107+0000","last_fullsized":"2021-03-24T21:53:31.423107+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:59.520858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:52:59.520858+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":"43","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.898420+0000","last_change":"2021-03-24T21:53:26.908991+0000","last_active":"2021-03-24T21:53:31.898420+0000","last_peered":"2021-03-24T21:53:31.898420+0000","last_clean":"2021-03-24T21:53:31.898420+0000","last_became_active":"2021-03-24T21:51:45.384545+0000","last_became_peered":"2021-03-24T21:51:45.384545+0000","last_unstale":"2021-03-24T21:53:31.898420+0000","last_undegraded":"2021-03-24T21:53:31.898420+0000","last_fullsized":"2021-03-24T21:53:31.898420+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:26.908821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:26.908821+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"69'23","reported_seq":"81","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.638227+0000","last_change":"2021-03-24T21:52:47.126913+0000","last_active":"2021-03-24T21:53:31.638227+0000","last_peered":"2021-03-24T21:53:31.638227+0000","last_clean":"2021-03-24T21:53:31.638227+0000","last_became_active":"2021-03-24T21:51:47.685830+0000","last_became_peered":"2021-03-24T21:51:47.685830+0000","last_unstale":"2021-03-24T21:53:31.638227+0000","last_undegraded":"2021-03-24T21:53:31.638227+0000","last_fullsized":"2021-03-24T21:53:31.638227+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'23","last_scrub_stamp":"2021-03-24T21:52:47.126826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:52:47.126826+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":44,"num_read_kb":25,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"37","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:35.636450+0000","last_change":"2021-03-24T21:53:35.636404+0000","last_active":"2021-03-24T21:53:35.636450+0000","last_peered":"2021-03-24T21:53:35.636450+0000","last_clean":"2021-03-24T21:53:35.636450+0000","last_became_active":"2021-03-24T21:51:51.763077+0000","last_became_peered":"2021-03-24T21:51:51.763077+0000","last_unstale":"2021-03-24T21:53:35.636450+0000","last_undegraded":"2021-03-24T21:53:35.636450+0000","last_fullsized":"2021-03-24T21:53:35.636450+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:35.636299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:35.636299+0000","last_clean_scrub_stamp":"2021-03-24T21:53:35.636299+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"41","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.422936+0000","last_change":"2021-03-24T21:53:11.623536+0000","last_active":"2021-03-24T21:53:31.422936+0000","last_peered":"2021-03-24T21:53:31.422936+0000","last_clean":"2021-03-24T21:53:31.422936+0000","last_became_active":"2021-03-24T21:51:49.411921+0000","last_became_peered":"2021-03-24T21:51:49.411921+0000","last_unstale":"2021-03-24T21:53:31.422936+0000","last_undegraded":"2021-03-24T21:53:31.422936+0000","last_fullsized":"2021-03-24T21:53:31.422936+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:11.623353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:11.623353+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"45","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.873529+0000","last_change":"2021-03-24T21:52:47.709622+0000","last_active":"2021-03-24T21:53:31.873529+0000","last_peered":"2021-03-24T21:53:31.873529+0000","last_clean":"2021-03-24T21:53:31.873529+0000","last_became_active":"2021-03-24T21:51:45.383017+0000","last_became_peered":"2021-03-24T21:51:45.383017+0000","last_unstale":"2021-03-24T21:53:31.873529+0000","last_undegraded":"2021-03-24T21:53:31.873529+0000","last_fullsized":"2021-03-24T21:53:31.873529+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:47.709437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:52:47.709437+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"69'14","reported_seq":"67","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.653542+0000","last_change":"2021-03-24T21:53:31.653478+0000","last_active":"2021-03-24T21:53:31.653542+0000","last_peered":"2021-03-24T21:53:31.653542+0000","last_clean":"2021-03-24T21:53:31.653542+0000","last_became_active":"2021-03-24T21:51:47.408588+0000","last_became_peered":"2021-03-24T21:51:47.408588+0000","last_unstale":"2021-03-24T21:53:31.653542+0000","last_undegraded":"2021-03-24T21:53:31.653542+0000","last_fullsized":"2021-03-24T21:53:31.653542+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'14","last_scrub_stamp":"2021-03-24T21:53:31.653334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:31.653334+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":16,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"38","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.664854+0000","last_change":"2021-03-24T21:52:59.706737+0000","last_active":"2021-03-24T21:53:31.664854+0000","last_peered":"2021-03-24T21:53:31.664854+0000","last_clean":"2021-03-24T21:53:31.664854+0000","last_became_active":"2021-03-24T21:51:51.759514+0000","last_became_peered":"2021-03-24T21:51:51.759514+0000","last_unstale":"2021-03-24T21:53:31.664854+0000","last_undegraded":"2021-03-24T21:53:31.664854+0000","last_fullsized":"2021-03-24T21:53:31.664854+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:59.706511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:52:59.706511+0000","last_clean_scrub_stamp":"2021-03-24T21:52:59.706511+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"49","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.873143+0000","last_change":"2021-03-24T21:53:06.800587+0000","last_active":"2021-03-24T21:53:31.873143+0000","last_peered":"2021-03-24T21:53:31.873143+0000","last_clean":"2021-03-24T21:53:31.873143+0000","last_became_active":"2021-03-24T21:51:40.350011+0000","last_became_peered":"2021-03-24T21:51:40.350011+0000","last_unstale":"2021-03-24T21:53:31.873143+0000","last_undegraded":"2021-03-24T21:53:31.873143+0000","last_fullsized":"2021-03-24T21:53:31.873143+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:06.800417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:06.800417+0000","last_clean_scrub_stamp":"2021-03-24T21:53:06.800417+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"89'2","reported_seq":"47","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.422148+0000","last_change":"2021-03-24T21:53:28.220091+0000","last_active":"2021-03-24T21:53:31.422148+0000","last_peered":"2021-03-24T21:53:31.422148+0000","last_clean":"2021-03-24T21:53:31.422148+0000","last_became_active":"2021-03-24T21:51:45.375765+0000","last_became_peered":"2021-03-24T21:51:45.375765+0000","last_unstale":"2021-03-24T21:53:31.422148+0000","last_undegraded":"2021-03-24T21:53:31.422148+0000","last_fullsized":"2021-03-24T21:53:31.422148+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2021-03-24T21:53:28.219900+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2021-03-24T21:53:28.219900+0000","last_clean_scrub_stamp":"2021-03-24T21:53:28.219900+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"69'25","reported_seq":"90","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.425468+0000","last_change":"2021-03-24T21:53:15.719556+0000","last_active":"2021-03-24T21:53:31.425468+0000","last_peered":"2021-03-24T21:53:31.425468+0000","last_clean":"2021-03-24T21:53:31.425468+0000","last_became_active":"2021-03-24T21:51:47.411647+0000","last_became_peered":"2021-03-24T21:51:47.411647+0000","last_unstale":"2021-03-24T21:53:31.425468+0000","last_undegraded":"2021-03-24T21:53:31.425468+0000","last_fullsized":"2021-03-24T21:53:31.425468+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'25","last_scrub_stamp":"2021-03-24T21:53:15.719434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:15.719434+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":54,"num_read_kb":31,"num_write":28,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"39","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.651499+0000","last_change":"2021-03-24T21:53:23.578854+0000","last_active":"2021-03-24T21:53:31.651499+0000","last_peered":"2021-03-24T21:53:31.651499+0000","last_clean":"2021-03-24T21:53:31.651499+0000","last_became_active":"2021-03-24T21:51:49.421253+0000","last_became_peered":"2021-03-24T21:51:49.421253+0000","last_unstale":"2021-03-24T21:53:31.651499+0000","last_undegraded":"2021-03-24T21:53:31.651499+0000","last_fullsized":"2021-03-24T21:53:31.651499+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:23.578679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:23.578679+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"37","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.898617+0000","last_change":"2021-03-24T21:53:23.927068+0000","last_active":"2021-03-24T21:53:31.898617+0000","last_peered":"2021-03-24T21:53:31.898617+0000","last_clean":"2021-03-24T21:53:31.898617+0000","last_became_active":"2021-03-24T21:51:51.912075+0000","last_became_peered":"2021-03-24T21:51:51.912075+0000","last_unstale":"2021-03-24T21:53:31.898617+0000","last_undegraded":"2021-03-24T21:53:31.898617+0000","last_fullsized":"2021-03-24T21:53:31.898617+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:23.926884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:23.926884+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"61'1","reported_seq":"48","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.425535+0000","last_change":"2021-03-24T21:53:18.753108+0000","last_active":"2021-03-24T21:53:31.425535+0000","last_peered":"2021-03-24T21:53:31.425535+0000","last_clean":"2021-03-24T21:53:31.425535+0000","last_became_active":"2021-03-24T21:51:45.580337+0000","last_became_peered":"2021-03-24T21:51:45.580337+0000","last_unstale":"2021-03-24T21:53:31.425535+0000","last_undegraded":"2021-03-24T21:53:31.425535+0000","last_fullsized":"2021-03-24T21:53:31.425535+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2021-03-24T21:53:18.753026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:18.753026+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"49","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.422852+0000","last_change":"2021-03-24T21:52:47.514577+0000","last_active":"2021-03-24T21:53:31.422852+0000","last_peered":"2021-03-24T21:53:31.422852+0000","last_clean":"2021-03-24T21:53:31.422852+0000","last_became_active":"2021-03-24T21:51:40.351000+0000","last_became_peered":"2021-03-24T21:51:40.351000+0000","last_unstale":"2021-03-24T21:53:31.422852+0000","last_undegraded":"2021-03-24T21:53:31.422852+0000","last_fullsized":"2021-03-24T21:53:31.422852+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:47.514427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:39.286101+0000","last_clean_scrub_stamp":"2021-03-24T21:52:47.514427+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"89'22","reported_seq":"81","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.425810+0000","last_change":"2021-03-24T21:53:19.717364+0000","last_active":"2021-03-24T21:53:31.425810+0000","last_peered":"2021-03-24T21:53:31.425810+0000","last_clean":"2021-03-24T21:53:31.425810+0000","last_became_active":"2021-03-24T21:51:47.411696+0000","last_became_peered":"2021-03-24T21:51:47.411696+0000","last_unstale":"2021-03-24T21:53:31.425810+0000","last_undegraded":"2021-03-24T21:53:31.425810+0000","last_fullsized":"2021-03-24T21:53:31.425810+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'22","last_scrub_stamp":"2021-03-24T21:53:19.717244+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2021-03-24T21:53:19.717244+0000","last_clean_scrub_stamp":"2021-03-24T21:53:19.717244+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":36,"num_read_kb":21,"num_write":18,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"41","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.423845+0000","last_change":"2021-03-24T21:53:21.534119+0000","last_active":"2021-03-24T21:53:31.423845+0000","last_peered":"2021-03-24T21:53:31.423845+0000","last_clean":"2021-03-24T21:53:31.423845+0000","last_became_active":"2021-03-24T21:51:49.418848+0000","last_became_peered":"2021-03-24T21:51:49.418848+0000","last_unstale":"2021-03-24T21:53:31.423845+0000","last_undegraded":"2021-03-24T21:53:31.423845+0000","last_fullsized":"2021-03-24T21:53:31.423845+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:21.533882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:21.533882+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"39","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.874172+0000","last_change":"2021-03-24T21:53:01.832920+0000","last_active":"2021-03-24T21:53:31.874172+0000","last_peered":"2021-03-24T21:53:31.874172+0000","last_clean":"2021-03-24T21:53:31.874172+0000","last_became_active":"2021-03-24T21:51:51.765236+0000","last_became_peered":"2021-03-24T21:51:51.765236+0000","last_unstale":"2021-03-24T21:53:31.874172+0000","last_undegraded":"2021-03-24T21:53:31.874172+0000","last_fullsized":"2021-03-24T21:53:31.874172+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:01.832740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:01.832740+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"53","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.896967+0000","last_change":"2021-03-24T21:53:30.906666+0000","last_active":"2021-03-24T21:53:31.896967+0000","last_peered":"2021-03-24T21:53:31.896967+0000","last_clean":"2021-03-24T21:53:31.896967+0000","last_became_active":"2021-03-24T21:51:40.349967+0000","last_became_peered":"2021-03-24T21:51:40.349967+0000","last_unstale":"2021-03-24T21:53:31.896967+0000","last_undegraded":"2021-03-24T21:53:31.896967+0000","last_fullsized":"2021-03-24T21:53:31.896967+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:30.906502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:49:51.529051+0000","last_clean_scrub_stamp":"2021-03-24T21:53:30.906502+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"44","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.664955+0000","last_change":"2021-03-24T21:53:20.736644+0000","last_active":"2021-03-24T21:53:31.664955+0000","last_peered":"2021-03-24T21:53:31.664955+0000","last_clean":"2021-03-24T21:53:31.664955+0000","last_became_active":"2021-03-24T21:51:45.383779+0000","last_became_peered":"2021-03-24T21:51:45.383779+0000","last_unstale":"2021-03-24T21:53:31.664955+0000","last_undegraded":"2021-03-24T21:53:31.664955+0000","last_fullsized":"2021-03-24T21:53:31.664955+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:20.736432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:20.736432+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"69'15","reported_seq":"66","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.638700+0000","last_change":"2021-03-24T21:53:20.279112+0000","last_active":"2021-03-24T21:53:31.638700+0000","last_peered":"2021-03-24T21:53:31.638700+0000","last_clean":"2021-03-24T21:53:31.638700+0000","last_became_active":"2021-03-24T21:51:47.688873+0000","last_became_peered":"2021-03-24T21:51:47.688873+0000","last_unstale":"2021-03-24T21:53:31.638700+0000","last_undegraded":"2021-03-24T21:53:31.638700+0000","last_fullsized":"2021-03-24T21:53:31.638700+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:53:20.279004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:20.279004+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":27,"num_read_kb":15,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"39","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.898039+0000","last_change":"2021-03-24T21:53:09.874798+0000","last_active":"2021-03-24T21:53:31.898039+0000","last_peered":"2021-03-24T21:53:31.898039+0000","last_clean":"2021-03-24T21:53:31.898039+0000","last_became_active":"2021-03-24T21:51:49.413251+0000","last_became_peered":"2021-03-24T21:51:49.413251+0000","last_unstale":"2021-03-24T21:53:31.898039+0000","last_undegraded":"2021-03-24T21:53:31.898039+0000","last_fullsized":"2021-03-24T21:53:31.898039+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:09.874681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:09.874681+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"39","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.425249+0000","last_change":"2021-03-24T21:53:11.723900+0000","last_active":"2021-03-24T21:53:31.425249+0000","last_peered":"2021-03-24T21:53:31.425249+0000","last_clean":"2021-03-24T21:53:31.425249+0000","last_became_active":"2021-03-24T21:51:51.762039+0000","last_became_peered":"2021-03-24T21:51:51.762039+0000","last_unstale":"2021-03-24T21:53:31.425249+0000","last_undegraded":"2021-03-24T21:53:31.425249+0000","last_fullsized":"2021-03-24T21:53:31.425249+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:11.723709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:11.723709+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"43","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.651013+0000","last_change":"2021-03-24T21:52:48.487965+0000","last_active":"2021-03-24T21:53:31.651013+0000","last_peered":"2021-03-24T21:53:31.651013+0000","last_clean":"2021-03-24T21:53:31.651013+0000","last_became_active":"2021-03-24T21:51:45.376687+0000","last_became_peered":"2021-03-24T21:51:45.376687+0000","last_unstale":"2021-03-24T21:53:31.651013+0000","last_undegraded":"2021-03-24T21:53:31.651013+0000","last_fullsized":"2021-03-24T21:53:31.651013+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:48.487787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:52:48.487787+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"66'5","reported_seq":"168","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:42.319250+0000","last_change":"2021-03-24T21:52:50.807770+0000","last_active":"2021-03-24T21:53:42.319250+0000","last_peered":"2021-03-24T21:53:42.319250+0000","last_clean":"2021-03-24T21:53:42.319250+0000","last_became_active":"2021-03-24T21:51:40.350525+0000","last_became_peered":"2021-03-24T21:51:40.350525+0000","last_unstale":"2021-03-24T21:53:42.319250+0000","last_undegraded":"2021-03-24T21:53:42.319250+0000","last_fullsized":"2021-03-24T21:53:42.319250+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2021-03-24T21:52:50.807568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:39.286101+0000","last_clean_scrub_stamp":"2021-03-24T21:52:50.807568+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":118,"num_read_kb":113,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"69'16","reported_seq":"67","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.651988+0000","last_change":"2021-03-24T21:53:15.485180+0000","last_active":"2021-03-24T21:53:31.651988+0000","last_peered":"2021-03-24T21:53:31.651988+0000","last_clean":"2021-03-24T21:53:31.651988+0000","last_became_active":"2021-03-24T21:51:47.411442+0000","last_became_peered":"2021-03-24T21:51:47.411442+0000","last_unstale":"2021-03-24T21:53:31.651988+0000","last_undegraded":"2021-03-24T21:53:31.651988+0000","last_fullsized":"2021-03-24T21:53:31.651988+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T21:53:15.485072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:15.485072+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":16,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"41","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.874725+0000","last_change":"2021-03-24T21:53:20.836086+0000","last_active":"2021-03-24T21:53:31.874725+0000","last_peered":"2021-03-24T21:53:31.874725+0000","last_clean":"2021-03-24T21:53:31.874725+0000","last_became_active":"2021-03-24T21:51:49.419167+0000","last_became_peered":"2021-03-24T21:51:49.419167+0000","last_unstale":"2021-03-24T21:53:31.874725+0000","last_undegraded":"2021-03-24T21:53:31.874725+0000","last_fullsized":"2021-03-24T21:53:31.874725+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:20.835990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:20.835990+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"39","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.425097+0000","last_change":"2021-03-24T21:53:28.691448+0000","last_active":"2021-03-24T21:53:31.425097+0000","last_peered":"2021-03-24T21:53:31.425097+0000","last_clean":"2021-03-24T21:53:31.425097+0000","last_became_active":"2021-03-24T21:51:51.913581+0000","last_became_peered":"2021-03-24T21:51:51.913581+0000","last_unstale":"2021-03-24T21:53:31.425097+0000","last_undegraded":"2021-03-24T21:53:31.425097+0000","last_fullsized":"2021-03-24T21:53:31.425097+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:28.691270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:28.691270+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"45","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.425366+0000","last_change":"2021-03-24T21:53:08.691716+0000","last_active":"2021-03-24T21:53:31.425366+0000","last_peered":"2021-03-24T21:53:31.425366+0000","last_clean":"2021-03-24T21:53:31.425366+0000","last_became_active":"2021-03-24T21:51:45.580531+0000","last_became_peered":"2021-03-24T21:51:45.580531+0000","last_unstale":"2021-03-24T21:53:31.425366+0000","last_undegraded":"2021-03-24T21:53:31.425366+0000","last_fullsized":"2021-03-24T21:53:31.425366+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:08.691536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:08.691536+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"69'21","reported_seq":"81","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.651725+0000","last_change":"2021-03-24T21:53:05.313409+0000","last_active":"2021-03-24T21:53:31.651725+0000","last_peered":"2021-03-24T21:53:31.651725+0000","last_clean":"2021-03-24T21:53:31.651725+0000","last_became_active":"2021-03-24T21:51:47.686486+0000","last_became_peered":"2021-03-24T21:51:47.686486+0000","last_unstale":"2021-03-24T21:53:31.651725+0000","last_undegraded":"2021-03-24T21:53:31.651725+0000","last_fullsized":"2021-03-24T21:53:31.651725+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'21","last_scrub_stamp":"2021-03-24T21:53:05.313235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:05.313235+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":45,"num_read_kb":27,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"40","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.638362+0000","last_change":"2021-03-24T21:53:23.246488+0000","last_active":"2021-03-24T21:53:31.638362+0000","last_peered":"2021-03-24T21:53:31.638362+0000","last_clean":"2021-03-24T21:53:31.638362+0000","last_became_active":"2021-03-24T21:51:49.416847+0000","last_became_peered":"2021-03-24T21:51:49.416847+0000","last_unstale":"2021-03-24T21:53:31.638362+0000","last_undegraded":"2021-03-24T21:53:31.638362+0000","last_fullsized":"2021-03-24T21:53:31.638362+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:23.246320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:23.246320+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"37","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.650517+0000","last_change":"2021-03-24T21:53:25.593543+0000","last_active":"2021-03-24T21:53:31.650517+0000","last_peered":"2021-03-24T21:53:31.650517+0000","last_clean":"2021-03-24T21:53:31.650517+0000","last_became_active":"2021-03-24T21:51:51.759627+0000","last_became_peered":"2021-03-24T21:51:51.759627+0000","last_unstale":"2021-03-24T21:53:31.650517+0000","last_undegraded":"2021-03-24T21:53:31.650517+0000","last_fullsized":"2021-03-24T21:53:31.650517+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:25.593358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:25.593358+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"45","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.874041+0000","last_change":"2021-03-24T21:53:04.837752+0000","last_active":"2021-03-24T21:53:31.874041+0000","last_peered":"2021-03-24T21:53:31.874041+0000","last_clean":"2021-03-24T21:53:31.874041+0000","last_became_active":"2021-03-24T21:51:45.583885+0000","last_became_peered":"2021-03-24T21:51:45.583885+0000","last_unstale":"2021-03-24T21:53:31.874041+0000","last_undegraded":"2021-03-24T21:53:31.874041+0000","last_fullsized":"2021-03-24T21:53:31.874041+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:04.837488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:04.837488+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"69'16","reported_seq":"70","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.425907+0000","last_change":"2021-03-24T21:53:03.595733+0000","last_active":"2021-03-24T21:53:31.425907+0000","last_peered":"2021-03-24T21:53:31.425907+0000","last_clean":"2021-03-24T21:53:31.425907+0000","last_became_active":"2021-03-24T21:51:47.403006+0000","last_became_peered":"2021-03-24T21:51:47.403006+0000","last_unstale":"2021-03-24T21:53:31.425907+0000","last_undegraded":"2021-03-24T21:53:31.425907+0000","last_fullsized":"2021-03-24T21:53:31.425907+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T21:53:03.595585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:03.595585+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"89'3","reported_seq":"65","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:38.806462+0000","last_change":"2021-03-24T21:53:26.584866+0000","last_active":"2021-03-24T21:53:38.806462+0000","last_peered":"2021-03-24T21:53:38.806462+0000","last_clean":"2021-03-24T21:53:38.806462+0000","last_became_active":"2021-03-24T21:51:49.420173+0000","last_became_peered":"2021-03-24T21:51:49.420173+0000","last_unstale":"2021-03-24T21:53:38.806462+0000","last_undegraded":"2021-03-24T21:53:38.806462+0000","last_fullsized":"2021-03-24T21:53:38.806462+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'3","last_scrub_stamp":"2021-03-24T21:53:26.584678+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-24T21:53:26.584678+0000","last_clean_scrub_stamp":"2021-03-24T21:53:26.584678+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"37","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.897726+0000","last_change":"2021-03-24T21:52:55.828993+0000","last_active":"2021-03-24T21:53:31.897726+0000","last_peered":"2021-03-24T21:53:31.897726+0000","last_clean":"2021-03-24T21:53:31.897726+0000","last_became_active":"2021-03-24T21:51:51.914601+0000","last_became_peered":"2021-03-24T21:51:51.914601+0000","last_unstale":"2021-03-24T21:53:31.897726+0000","last_undegraded":"2021-03-24T21:53:31.897726+0000","last_fullsized":"2021-03-24T21:53:31.897726+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:55.828810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:52:55.828810+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"43","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.652135+0000","last_change":"2021-03-24T21:53:24.621910+0000","last_active":"2021-03-24T21:53:31.652135+0000","last_peered":"2021-03-24T21:53:31.652135+0000","last_clean":"2021-03-24T21:53:31.652135+0000","last_became_active":"2021-03-24T21:51:45.375747+0000","last_became_peered":"2021-03-24T21:51:45.375747+0000","last_unstale":"2021-03-24T21:53:31.652135+0000","last_undegraded":"2021-03-24T21:53:31.652135+0000","last_fullsized":"2021-03-24T21:53:31.652135+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:24.621749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:24.621749+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"69'20","reported_seq":"75","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.422003+0000","last_change":"2021-03-24T21:53:13.077033+0000","last_active":"2021-03-24T21:53:31.422003+0000","last_peered":"2021-03-24T21:53:31.422003+0000","last_clean":"2021-03-24T21:53:31.422003+0000","last_became_active":"2021-03-24T21:51:47.404713+0000","last_became_peered":"2021-03-24T21:51:47.404713+0000","last_unstale":"2021-03-24T21:53:31.422003+0000","last_undegraded":"2021-03-24T21:53:31.422003+0000","last_fullsized":"2021-03-24T21:53:31.422003+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'20","last_scrub_stamp":"2021-03-24T21:53:13.076772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:13.076772+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":39,"num_read_kb":20,"num_write":20,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"65'2","reported_seq":"65","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:38.805729+0000","last_change":"2021-03-24T21:53:04.714525+0000","last_active":"2021-03-24T21:53:38.805729+0000","last_peered":"2021-03-24T21:53:38.805729+0000","last_clean":"2021-03-24T21:53:38.805729+0000","last_became_active":"2021-03-24T21:51:49.421628+0000","last_became_peered":"2021-03-24T21:51:49.421628+0000","last_unstale":"2021-03-24T21:53:38.805729+0000","last_undegraded":"2021-03-24T21:53:38.805729+0000","last_fullsized":"2021-03-24T21:53:38.805729+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2021-03-24T21:53:04.714348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:04.714348+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"38","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:45.349966+0000","last_change":"2021-03-24T21:53:45.349966+0000","last_active":"2021-03-24T21:53:45.349966+0000","last_peered":"2021-03-24T21:53:45.349966+0000","last_clean":"2021-03-24T21:53:45.349966+0000","last_became_active":"2021-03-24T21:51:51.756114+0000","last_became_peered":"2021-03-24T21:51:51.756114+0000","last_unstale":"2021-03-24T21:53:45.349966+0000","last_undegraded":"2021-03-24T21:53:45.349966+0000","last_fullsized":"2021-03-24T21:53:45.349966+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:45.349781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:45.349781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"44","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.665448+0000","last_change":"2021-03-24T21:53:00.682889+0000","last_active":"2021-03-24T21:53:31.665448+0000","last_peered":"2021-03-24T21:53:31.665448+0000","last_clean":"2021-03-24T21:53:31.665448+0000","last_became_active":"2021-03-24T21:51:45.378704+0000","last_became_peered":"2021-03-24T21:51:45.378704+0000","last_unstale":"2021-03-24T21:53:31.665448+0000","last_undegraded":"2021-03-24T21:53:31.665448+0000","last_fullsized":"2021-03-24T21:53:31.665448+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:00.682713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:00.682713+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"69'22","reported_seq":"82","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.664261+0000","last_change":"2021-03-24T21:53:17.716726+0000","last_active":"2021-03-24T21:53:31.664261+0000","last_peered":"2021-03-24T21:53:31.664261+0000","last_clean":"2021-03-24T21:53:31.664261+0000","last_became_active":"2021-03-24T21:51:47.691459+0000","last_became_peered":"2021-03-24T21:51:47.691459+0000","last_unstale":"2021-03-24T21:53:31.664261+0000","last_undegraded":"2021-03-24T21:53:31.664261+0000","last_fullsized":"2021-03-24T21:53:31.664261+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'22","last_scrub_stamp":"2021-03-24T21:53:17.716470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:17.716470+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":46,"num_read_kb":26,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"40","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.421753+0000","last_change":"2021-03-24T21:53:25.220598+0000","last_active":"2021-03-24T21:53:31.421753+0000","last_peered":"2021-03-24T21:53:31.421753+0000","last_clean":"2021-03-24T21:53:31.421753+0000","last_became_active":"2021-03-24T21:51:49.408761+0000","last_became_peered":"2021-03-24T21:51:49.408761+0000","last_unstale":"2021-03-24T21:53:31.421753+0000","last_undegraded":"2021-03-24T21:53:31.421753+0000","last_fullsized":"2021-03-24T21:53:31.421753+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:25.220436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:25.220436+0000","last_clean_scrub_stamp":"2021-03-24T21:53:25.220436+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"39","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.873607+0000","last_change":"2021-03-24T21:53:11.925302+0000","last_active":"2021-03-24T21:53:31.873607+0000","last_peered":"2021-03-24T21:53:31.873607+0000","last_clean":"2021-03-24T21:53:31.873607+0000","last_became_active":"2021-03-24T21:51:51.760369+0000","last_became_peered":"2021-03-24T21:51:51.760369+0000","last_unstale":"2021-03-24T21:53:31.873607+0000","last_undegraded":"2021-03-24T21:53:31.873607+0000","last_fullsized":"2021-03-24T21:53:31.873607+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:11.925149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:11.925149+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"65'2","reported_seq":"65","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:38.805416+0000","last_change":"2021-03-24T21:53:29.268452+0000","last_active":"2021-03-24T21:53:38.805416+0000","last_peered":"2021-03-24T21:53:38.805416+0000","last_clean":"2021-03-24T21:53:38.805416+0000","last_became_active":"2021-03-24T21:51:49.414380+0000","last_became_peered":"2021-03-24T21:51:49.414380+0000","last_unstale":"2021-03-24T21:53:38.805416+0000","last_undegraded":"2021-03-24T21:53:38.805416+0000","last_fullsized":"2021-03-24T21:53:38.805416+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2021-03-24T21:53:29.268273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:29.268273+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":"44","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:35.609404+0000","last_change":"2021-03-24T21:53:35.609332+0000","last_active":"2021-03-24T21:53:35.609404+0000","last_peered":"2021-03-24T21:53:35.609404+0000","last_clean":"2021-03-24T21:53:35.609404+0000","last_became_active":"2021-03-24T21:51:45.378863+0000","last_became_peered":"2021-03-24T21:51:45.378863+0000","last_unstale":"2021-03-24T21:53:35.609404+0000","last_undegraded":"2021-03-24T21:53:35.609404+0000","last_fullsized":"2021-03-24T21:53:35.609404+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:35.609173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:35.609173+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"69'17","reported_seq":"72","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.425898+0000","last_change":"2021-03-24T21:52:54.714245+0000","last_active":"2021-03-24T21:53:31.425898+0000","last_peered":"2021-03-24T21:53:31.425898+0000","last_clean":"2021-03-24T21:53:31.425898+0000","last_became_active":"2021-03-24T21:51:47.411126+0000","last_became_peered":"2021-03-24T21:51:47.411126+0000","last_unstale":"2021-03-24T21:53:31.425898+0000","last_undegraded":"2021-03-24T21:53:31.425898+0000","last_fullsized":"2021-03-24T21:53:31.425898+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'17","last_scrub_stamp":"2021-03-24T21:52:54.714050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:52:54.714050+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":19,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"38","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.664862+0000","last_change":"2021-03-24T21:53:27.701155+0000","last_active":"2021-03-24T21:53:31.664862+0000","last_peered":"2021-03-24T21:53:31.664862+0000","last_clean":"2021-03-24T21:53:31.664862+0000","last_became_active":"2021-03-24T21:51:51.912009+0000","last_became_peered":"2021-03-24T21:51:51.912009+0000","last_unstale":"2021-03-24T21:53:31.664862+0000","last_undegraded":"2021-03-24T21:53:31.664862+0000","last_fullsized":"2021-03-24T21:53:31.664862+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:27.701002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:27.701002+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"43","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.874620+0000","last_change":"2021-03-24T21:52:55.846936+0000","last_active":"2021-03-24T21:53:31.874620+0000","last_peered":"2021-03-24T21:53:31.874620+0000","last_clean":"2021-03-24T21:53:31.874620+0000","last_became_active":"2021-03-24T21:51:49.422787+0000","last_became_peered":"2021-03-24T21:51:49.422787+0000","last_unstale":"2021-03-24T21:53:31.874620+0000","last_undegraded":"2021-03-24T21:53:31.874620+0000","last_fullsized":"2021-03-24T21:53:31.874620+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:55.846764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:52:55.846764+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"44","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.422013+0000","last_change":"2021-03-24T21:52:48.334326+0000","last_active":"2021-03-24T21:53:31.422013+0000","last_peered":"2021-03-24T21:53:31.422013+0000","last_clean":"2021-03-24T21:53:31.422013+0000","last_became_active":"2021-03-24T21:51:45.579513+0000","last_became_peered":"2021-03-24T21:51:45.579513+0000","last_unstale":"2021-03-24T21:53:31.422013+0000","last_undegraded":"2021-03-24T21:53:31.422013+0000","last_fullsized":"2021-03-24T21:53:31.422013+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:48.334138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:52:48.334138+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"69'15","reported_seq":"69","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.422946+0000","last_change":"2021-03-24T21:52:55.233331+0000","last_active":"2021-03-24T21:53:31.422946+0000","last_peered":"2021-03-24T21:53:31.422946+0000","last_clean":"2021-03-24T21:53:31.422946+0000","last_became_active":"2021-03-24T21:51:47.410414+0000","last_became_peered":"2021-03-24T21:51:47.410414+0000","last_unstale":"2021-03-24T21:53:31.422946+0000","last_undegraded":"2021-03-24T21:53:31.422946+0000","last_fullsized":"2021-03-24T21:53:31.422946+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:52:55.233213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:52:55.233213+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"39","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.423972+0000","last_change":"2021-03-24T21:53:14.576861+0000","last_active":"2021-03-24T21:53:31.423972+0000","last_peered":"2021-03-24T21:53:31.423972+0000","last_clean":"2021-03-24T21:53:31.423972+0000","last_became_active":"2021-03-24T21:51:51.765067+0000","last_became_peered":"2021-03-24T21:51:51.765067+0000","last_unstale":"2021-03-24T21:53:31.423972+0000","last_undegraded":"2021-03-24T21:53:31.423972+0000","last_fullsized":"2021-03-24T21:53:31.423972+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:14.576687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:14.576687+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"41","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.425582+0000","last_change":"2021-03-24T21:53:29.735847+0000","last_active":"2021-03-24T21:53:31.425582+0000","last_peered":"2021-03-24T21:53:31.425582+0000","last_clean":"2021-03-24T21:53:31.425582+0000","last_became_active":"2021-03-24T21:51:49.407001+0000","last_became_peered":"2021-03-24T21:51:49.407001+0000","last_unstale":"2021-03-24T21:53:31.425582+0000","last_undegraded":"2021-03-24T21:53:31.425582+0000","last_fullsized":"2021-03-24T21:53:31.425582+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:29.735672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:29.735672+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"44","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.637890+0000","last_change":"2021-03-24T21:53:22.204032+0000","last_active":"2021-03-24T21:53:31.637890+0000","last_peered":"2021-03-24T21:53:31.637890+0000","last_clean":"2021-03-24T21:53:31.637890+0000","last_became_active":"2021-03-24T21:51:45.374203+0000","last_became_peered":"2021-03-24T21:51:45.374203+0000","last_unstale":"2021-03-24T21:53:31.637890+0000","last_undegraded":"2021-03-24T21:53:31.637890+0000","last_fullsized":"2021-03-24T21:53:31.637890+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:22.203858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:22.203858+0000","last_clean_scrub_stamp":"2021-03-24T21:53:22.203858+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"69'21","reported_seq":"78","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.422809+0000","last_change":"2021-03-24T21:52:50.302291+0000","last_active":"2021-03-24T21:53:31.422809+0000","last_peered":"2021-03-24T21:53:31.422809+0000","last_clean":"2021-03-24T21:53:31.422809+0000","last_became_active":"2021-03-24T21:51:47.416296+0000","last_became_peered":"2021-03-24T21:51:47.416296+0000","last_unstale":"2021-03-24T21:53:31.422809+0000","last_undegraded":"2021-03-24T21:53:31.422809+0000","last_fullsized":"2021-03-24T21:53:31.422809+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'21","last_scrub_stamp":"2021-03-24T21:52:50.302085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:52:50.302085+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":23,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"38","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:40.290157+0000","last_change":"2021-03-24T21:53:40.290157+0000","last_active":"2021-03-24T21:53:40.290157+0000","last_peered":"2021-03-24T21:53:40.290157+0000","last_clean":"2021-03-24T21:53:40.290157+0000","last_became_active":"2021-03-24T21:51:51.761142+0000","last_became_peered":"2021-03-24T21:51:51.761142+0000","last_unstale":"2021-03-24T21:53:40.290157+0000","last_undegraded":"2021-03-24T21:53:40.290157+0000","last_fullsized":"2021-03-24T21:53:40.290157+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:40.289984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:40.289984+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"65'2","reported_seq":"66","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:38.805700+0000","last_change":"2021-03-24T21:53:04.625006+0000","last_active":"2021-03-24T21:53:38.805700+0000","last_peered":"2021-03-24T21:53:38.805700+0000","last_clean":"2021-03-24T21:53:38.805700+0000","last_became_active":"2021-03-24T21:51:49.416182+0000","last_became_peered":"2021-03-24T21:51:49.416182+0000","last_unstale":"2021-03-24T21:53:38.805700+0000","last_undegraded":"2021-03-24T21:53:38.805700+0000","last_fullsized":"2021-03-24T21:53:38.805700+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2021-03-24T21:53:04.624892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:04.624892+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"44","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.665024+0000","last_change":"2021-03-24T21:52:46.620502+0000","last_active":"2021-03-24T21:53:31.665024+0000","last_peered":"2021-03-24T21:53:31.665024+0000","last_clean":"2021-03-24T21:53:31.665024+0000","last_became_active":"2021-03-24T21:51:45.375830+0000","last_became_peered":"2021-03-24T21:51:45.375830+0000","last_unstale":"2021-03-24T21:53:31.665024+0000","last_undegraded":"2021-03-24T21:53:31.665024+0000","last_fullsized":"2021-03-24T21:53:31.665024+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:46.620317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:52:46.620317+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"69'13","reported_seq":"63","reported_epoch":"91","state":"active+clean","last_fresh":"2021-03-24T21:53:31.422441+0000","last_change":"2021-03-24T21:53:20.163291+0000","last_active":"2021-03-24T21:53:31.422441+0000","last_peered":"2021-03-24T21:53:31.422441+0000","last_clean":"2021-03-24T21:53:31.422441+0000","last_became_active":"2021-03-24T21:51:47.407739+0000","last_became_peered":"2021-03-24T21:51:47.407739+0000","last_unstale":"2021-03-24T21:53:31.422441+0000","last_undegraded":"2021-03-24T21:53:31.422441+0000","last_fullsized":"2021-03-24T21:53:31.422441+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2021-03-24T21:53:20.163065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:20.163065+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":13,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19,"ondisk_log_size":19,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1069,"num_read_kb":574,"num_write":542,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":2412,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966080,"data_stored":70572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":539,"ondisk_log_size":539,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":118,"num_read_kb":113,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201308,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6400,"kb_used_data":824,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738752,"statfs":{"total":95995035648,"available":95988482048,"internally_reserved":0,"allocated":843776,"data_stored":314208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:52:41 2021","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":0.97999999999999998},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":0.878}]},{"osd":1,"last update":"Wed Mar 24 21:52:41 2021","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":2.343,"5min":2.343,"15min":2.343},"last":1.01}]},{"osd":2,"last update":"Wed Mar 24 21:52:41 2021","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.7509999999999999,"5min":1.7509999999999999,"15min":1.7509999999999999},"last":0.84799999999999998},{"interface":"front","average":{"1min":1.012,"5min":1.012,"15min":1.012},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.6850000000000001,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":1.1459999999999999}]},{"osd":3,"last update":"Wed Mar 24 21:52:41 2021","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.9490000000000001,"5min":1.9490000000000001,"15min":1.9490000000000001},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.845,"5min":1.845,"15min":1.845},"last":1.0469999999999999}]},{"osd":4,"last update":"Wed Mar 24 21:52:41 2021","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":2.2000000000000002,"5min":2.2000000000000002,"15min":2.2000000000000002},"last":1.113},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":2.1299999999999999,"5min":2.1299999999999999,"15min":2.1299999999999999},"last":0.92200000000000004}]},{"osd":5,"last update":"Wed Mar 24 21:52:41 2021","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.581,"5min":1.581,"15min":1.581},"last":1.22},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":2.4630000000000001,"5min":2.4630000000000001,"15min":2.4630000000000001},"last":1.25}]},{"osd":6,"last update":"Wed Mar 24 21:52:41 2021","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":2.5179999999999998,"5min":2.5179999999999998,"15min":2.5179999999999998},"last":1.4259999999999999},{"interface":"front","average":{"1min":1.171,"5min":1.171,"15min":1.171},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":2.3069999999999999,"5min":2.3069999999999999,"15min":2.3069999999999999},"last":1.0780000000000001}]}]},{"osd":6,"up_from":47,"seq":201863462945,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6512,"kb_used_data":872,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738640,"statfs":{"total":95995035648,"available":95988367360,"internally_reserved":0,"allocated":892928,"data_stored":317430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:53:21 2021","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.91000000000000003,"15min":0.91200000000000003},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.91200000000000003,"15min":0.91700000000000004},"min":{"1min":0.56799999999999995,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.94599999999999995}]},{"osd":1,"last update":"Wed Mar 24 21:53:21 2021","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.92100000000000004,"15min":0.93200000000000005},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.2470000000000001,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.91100000000000003,"15min":0.91600000000000004},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.2869999999999999,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.63800000000000001}]},{"osd":2,"last update":"Wed Mar 24 21:53:21 2021","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.94899999999999995,"15min":0.95099999999999996},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.96699999999999997,"15min":0.97499999999999998},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.2170000000000001,"5min":1.401,"15min":1.401},"last":0.88700000000000001}]},{"osd":3,"last update":"Wed Mar 24 21:53:21 2021","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.95299999999999996,"15min":0.95699999999999996},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.3089999999999999,"5min":1.4179999999999999,"15min":1.4179999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.022,"15min":1.0149999999999999},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.5009999999999999,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":0.97799999999999998}]},{"osd":4,"last update":"Wed Mar 24 21:53:21 2021","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.92000000000000004,"15min":0.92100000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.26,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.97699999999999998,"5min":0.95099999999999996,"15min":0.94699999999999995},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.3149999999999999,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.96499999999999997}]},{"osd":5,"last update":"Wed Mar 24 21:53:21 2021","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":1.0369999999999999,"15min":1.0449999999999999},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.337,"5min":1.4830000000000001,"15min":1.4830000000000001},"last":1.004},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.04,"15min":1.038},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.4570000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":1.0620000000000001}]},{"osd":7,"last update":"Wed Mar 24 21:53:41 2021","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":1.1359999999999999,"15min":1.1279999999999999},"min":{"1min":0.83799999999999997,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.3560000000000001,"5min":1.5249999999999999,"15min":1.5249999999999999},"last":0.83799999999999997},{"interface":"front","average":{"1min":1.234,"5min":1.1299999999999999,"15min":1.113},"min":{"1min":0.79900000000000004,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.478,"5min":1.478,"15min":1.478},"last":1.04}]}]},{"osd":1,"up_from":13,"seq":55834574902,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7104,"kb_used_data":952,"kb_used_omap":0,"kb_used_meta":6144,"kb_avail":93738048,"statfs":{"total":95995035648,"available":95987761152,"internally_reserved":0,"allocated":974848,"data_stored":320242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6291456},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:53:33 2021","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.629,"15min":0.58699999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.68500000000000005,"15min":0.64100000000000001},"min":{"1min":0.41399999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.3009999999999999,"5min":1.641,"15min":1.641},"last":0.68600000000000005}]},{"osd":2,"last update":"Wed Mar 24 21:52:52 2021","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.64600000000000002,"15min":0.627},"min":{"1min":0.443,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":1.1100000000000001},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.76700000000000002,"15min":0.746},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.173,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":0.84899999999999998}]},{"osd":3,"last update":"Wed Mar 24 21:53:18 2021","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":1.296,"15min":1.4730000000000001},"min":{"1min":0.63100000000000001,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.2849999999999999,"5min":17.032,"15min":17.032},"last":1.2370000000000001},{"interface":"front","average":{"1min":0.90900000000000003,"5min":1.2290000000000001,"15min":1.389},"min":{"1min":0.621,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.135,"5min":16.908999999999999,"15min":16.908999999999999},"last":1.333}]},{"osd":4,"last update":"Wed Mar 24 21:53:41 2021","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.96399999999999997,"15min":0.96399999999999997},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.4510000000000001,"5min":1.617,"15min":1.617},"last":1.4319999999999999},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.093,"15min":1.095},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.706,"5min":1.7150000000000001,"15min":1.7150000000000001},"last":1}]},{"osd":5,"last update":"Wed Mar 24 21:52:55 2021","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":0.999,"15min":0.999},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.3220000000000001,"5min":1.5489999999999999,"15min":1.5489999999999999},"last":0.96099999999999997},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.0640000000000001,"15min":1.0620000000000001},"min":{"1min":0.80000000000000004,"5min":0.748,"15min":0.748},"max":{"1min":1.4199999999999999,"5min":1.6619999999999999,"15min":1.6619999999999999},"last":1.085}]},{"osd":6,"last update":"Wed Mar 24 21:53:18 2021","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.141,"15min":1.1399999999999999},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":1.4239999999999999,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":0.93400000000000005},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.1479999999999999,"15min":1.1619999999999999},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.3819999999999999,"5min":1.806,"15min":1.806},"last":1.278}]},{"osd":7,"last update":"Wed Mar 24 21:53:41 2021","interfaces":[{"interface":"back","average":{"1min":1.2030000000000001,"5min":1.23,"15min":1.234},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.7789999999999999,"5min":1.7789999999999999,"15min":1.7789999999999999},"last":1.1639999999999999},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.175,"15min":1.181},"min":{"1min":0.96499999999999997,"5min":0.90300000000000002,"15min":0.90300000000000002},"max":{"1min":1.3480000000000001,"5min":1.532,"15min":1.532},"last":1.2130000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738425,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6792,"kb_used_data":832,"kb_used_omap":0,"kb_used_meta":5952,"kb_avail":93738360,"statfs":{"total":95995035648,"available":95988080640,"internally_reserved":0,"allocated":851968,"data_stored":315145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6094848},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Mar 24 21:53:37 2021","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.63700000000000001,"15min":0.58399999999999996},"min":{"1min":0.44400000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.98299999999999998,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.67500000000000004,"15min":0.60999999999999999},"min":{"1min":0.46999999999999997,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.78100000000000003}]},{"osd":2,"last update":"Wed Mar 24 21:52:55 2021","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.67500000000000004,"15min":0.66100000000000003},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.879,"5min":1.468,"15min":1.468},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.73099999999999998,"15min":0.69099999999999995},"min":{"1min":0.41799999999999998,"5min":0.376,"15min":0.376},"max":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.55200000000000005}]},{"osd":3,"last update":"Wed Mar 24 21:53:20 2021","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.80500000000000005,"15min":0.80000000000000004},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.115,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.89700000000000002},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.93400000000000005,"15min":0.93200000000000005},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.2430000000000001,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.73099999999999998}]},{"osd":4,"last update":"Wed Mar 24 21:53:37 2021","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.0900000000000001,"15min":1.0960000000000001},"min":{"1min":0.81799999999999995,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.4570000000000001,"5min":1.7,"15min":1.7},"last":1.0229999999999999},{"interface":"front","average":{"1min":1.056,"5min":1.028,"15min":1.014},"min":{"1min":0.78700000000000003,"5min":0.625,"15min":0.625},"max":{"1min":1.375,"5min":1.407,"15min":1.407},"last":1.228}]},{"osd":5,"last update":"Wed Mar 24 21:52:55 2021","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":1.0509999999999999,"15min":1.0620000000000001},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.3919999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":1.143},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.179,"15min":1.177},"min":{"1min":0.86299999999999999,"5min":0.78500000000000003,"15min":0.78500000000000003},"max":{"1min":1.597,"5min":1.746,"15min":1.746},"last":0.95899999999999996}]},{"osd":6,"last update":"Wed Mar 24 21:53:20 2021","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.0669999999999999,"15min":1.0669999999999999},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.3779999999999999,"5min":1.427,"15min":1.427},"last":0.98999999999999999},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.224,"15min":1.22},"min":{"1min":0.877,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.6659999999999999,"5min":1.6659999999999999,"15min":1.6659999999999999},"last":1.0640000000000001}]},{"osd":7,"last update":"Wed Mar 24 21:53:42 2021","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.1739999999999999,"15min":1.1699999999999999},"min":{"1min":0.82099999999999995,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":1.4970000000000001,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":1.28},{"interface":"front","average":{"1min":1.157,"5min":1.131,"15min":1.127},"min":{"1min":0.77100000000000002,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.4319999999999999,"5min":1.4490000000000001,"15min":1.4490000000000001},"last":1.0940000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411379,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6564,"kb_used_data":796,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738588,"statfs":{"total":95995035648,"available":95988314112,"internally_reserved":0,"allocated":815104,"data_stored":314429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:52:55 2021","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.64400000000000002,"15min":0.61799999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.94799999999999995},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.71599999999999997,"15min":0.68100000000000005},"min":{"1min":0.55300000000000005,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.405,"5min":1.405,"15min":1.405},"last":0.79200000000000004}]},{"osd":1,"last update":"Wed Mar 24 21:52:55 2021","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.72199999999999998,"15min":0.69199999999999995},"min":{"1min":0.57499999999999996,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":1.024},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.76600000000000001,"15min":0.72799999999999998},"min":{"1min":0.621,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.444,"5min":1.444,"15min":1.444},"last":1.0529999999999999}]},{"osd":3,"last update":"Wed Mar 24 21:53:17 2021","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.84399999999999997,"15min":0.82399999999999995},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.9890000000000001,"5min":1.9890000000000001,"15min":1.9890000000000001},"last":1.101},{"interface":"front","average":{"1min":0.95699999999999996,"5min":0.92000000000000004,"15min":0.89400000000000002},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":2.0950000000000002,"5min":2.0950000000000002,"15min":2.0950000000000002},"last":1.1359999999999999}]},{"osd":4,"last update":"Wed Mar 24 21:53:40 2021","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.04,"15min":1.036},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":3.3380000000000001,"5min":3.3380000000000001,"15min":3.3380000000000001},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.0680000000000001,"15min":1.046},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.2050000000000001,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":1.2230000000000001}]},{"osd":5,"last update":"Wed Mar 24 21:52:55 2021","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.0329999999999999,"15min":1.0329999999999999},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.47,"5min":1.47,"15min":1.47},"last":1.2},{"interface":"front","average":{"1min":1.224,"5min":1.165,"15min":1.155},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":3.1509999999999998,"5min":3.1509999999999998,"15min":3.1509999999999998},"last":1.2430000000000001}]},{"osd":6,"last update":"Wed Mar 24 21:53:17 2021","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.256,"15min":1.262},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":2.2810000000000001,"5min":2.2810000000000001,"15min":2.2810000000000001},"last":1.552},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.1619999999999999,"15min":1.165},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":1.8069999999999999,"5min":2.016,"15min":2.016},"last":0.98599999999999999}]},{"osd":7,"last update":"Wed Mar 24 21:53:43 2021","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":1.2190000000000001,"15min":1.228},"min":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.6719999999999999,"5min":1.6719999999999999,"15min":1.6719999999999999},"last":1.1100000000000001},{"interface":"front","average":{"1min":1.2290000000000001,"5min":1.246,"15min":1.2490000000000001},"min":{"1min":0.80800000000000005,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":3.5419999999999998,"5min":3.5419999999999998,"15min":3.5419999999999998},"last":1.161}]}]},{"osd":3,"up_from":26,"seq":111669149742,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6828,"kb_used_data":932,"kb_used_omap":0,"kb_used_meta":5888,"kb_avail":93738324,"statfs":{"total":95995035648,"available":95988043776,"internally_reserved":0,"allocated":954368,"data_stored":318598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6029312},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:53:21 2021","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.63300000000000001,"15min":0.63500000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.92900000000000005,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.73299999999999998,"15min":0.72099999999999997},"min":{"1min":0.41499999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.93000000000000005}]},{"osd":1,"last update":"Wed Mar 24 21:53:21 2021","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.74199999999999999,"15min":0.73799999999999999},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.018,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.75600000000000001,"15min":0.755},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.97399999999999998,"5min":1.103,"15min":1.103},"last":0.89800000000000002}]},{"osd":2,"last update":"Wed Mar 24 21:53:21 2021","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.79700000000000004,"15min":0.78700000000000003},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.77500000000000002,"15min":0.76500000000000001},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0449999999999999,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.95599999999999996}]},{"osd":4,"last update":"Wed Mar 24 21:53:40 2021","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.95399999999999996,"15min":0.95399999999999996},"min":{"1min":0.66400000000000003,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.1950000000000001,"5min":1.702,"15min":1.702},"last":1.0629999999999999},{"interface":"front","average":{"1min":1.016,"5min":0.96599999999999997,"15min":0.94399999999999995},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.671,"5min":1.671,"15min":1.671},"last":1.0960000000000001}]},{"osd":5,"last update":"Wed Mar 24 21:52:55 2021","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.0669999999999999,"15min":1.0620000000000001},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":2.423,"5min":2.423,"15min":2.423},"last":1.1279999999999999},{"interface":"front","average":{"1min":0.98999999999999999,"5min":1.0740000000000001,"15min":1.0880000000000001},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.2889999999999999,"5min":1.496,"15min":1.496},"last":1.1930000000000001}]},{"osd":6,"last update":"Wed Mar 24 21:53:21 2021","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.1180000000000001,"15min":1.1259999999999999},"min":{"1min":0.85499999999999998,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.49,"5min":1.49,"15min":1.49},"last":1.3220000000000001},{"interface":"front","average":{"1min":1.155,"5min":1.1539999999999999,"15min":1.1539999999999999},"min":{"1min":0.77600000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":1.7070000000000001,"5min":1.7070000000000001,"15min":1.7070000000000001},"last":1.204}]},{"osd":7,"last update":"Wed Mar 24 21:52:43 2021","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.077,"15min":1.077},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.425,"5min":1.425,"15min":1.425},"last":1.2270000000000001},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"last":1.363}]}]},{"osd":4,"up_from":32,"seq":137438953513,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6724,"kb_used_data":892,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738428,"statfs":{"total":95995035648,"available":95988150272,"internally_reserved":0,"allocated":913408,"data_stored":317246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:53:36 2021","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.84699999999999998,"15min":0.84099999999999997},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.2490000000000001,"5min":1.268,"15min":1.268},"last":1.1459999999999999},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.89200000000000002,"15min":0.86199999999999999},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.667,"5min":1.667,"15min":1.667},"last":0.84599999999999997}]},{"osd":1,"last update":"Wed Mar 24 21:53:36 2021","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.92000000000000004,"15min":0.91800000000000004},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.1439999999999999,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.86899999999999999,"15min":0.84699999999999998},"min":{"1min":0.59199999999999997,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.337,"5min":1.337,"15min":1.337},"last":1.071}]},{"osd":2,"last update":"Wed Mar 24 21:53:36 2021","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.90700000000000003,"15min":0.89900000000000002},"min":{"1min":0.57699999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.6279999999999999,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":0.747},{"interface":"front","average":{"1min":0.878,"5min":0.92800000000000005,"15min":0.91200000000000003},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.169,"5min":1.335,"15min":1.335},"last":0.99099999999999999}]},{"osd":3,"last update":"Wed Mar 24 21:53:36 2021","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.92800000000000005,"15min":0.93400000000000005},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.1359999999999999,"5min":1.601,"15min":1.601},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.97099999999999997,"15min":0.95299999999999996},"min":{"1min":0.754,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.474,"5min":1.474,"15min":1.474},"last":0.86599999999999999}]},{"osd":5,"last update":"Wed Mar 24 21:52:57 2021","interfaces":[{"interface":"back","average":{"1min":1.0149999999999999,"5min":0.96299999999999997,"15min":0.95499999999999996},"min":{"1min":0.59599999999999997,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":0.95599999999999996},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.026,"15min":1},"min":{"1min":0.78300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":2.0760000000000001,"5min":2.0760000000000001,"15min":2.0760000000000001},"last":1.03}]},{"osd":6,"last update":"Wed Mar 24 21:53:18 2021","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":1.0620000000000001,"15min":1.0640000000000001},"min":{"1min":0.79900000000000004,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.2749999999999999,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":0.93400000000000005},{"interface":"front","average":{"1min":1.018,"5min":1.056,"15min":1.0620000000000001},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.357,"5min":1.506,"15min":1.506},"last":1.272}]},{"osd":7,"last update":"Wed Mar 24 21:53:41 2021","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":1.1679999999999999,"15min":1.1870000000000001},"min":{"1min":0.624,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.587,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":1.222},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.1910000000000001,"15min":1.2010000000000001},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.702,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":1.1779999999999999}]}]},{"osd":5,"up_from":39,"seq":167503724582,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6404,"kb_used_data":764,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738748,"statfs":{"total":95995035648,"available":95988477952,"internally_reserved":0,"allocated":782336,"data_stored":312209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:52:56 2021","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.89000000000000001,"15min":0.88900000000000001},"min":{"1min":0.623,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":2.29,"5min":2.29,"15min":2.29},"last":0.96699999999999997},{"interface":"front","average":{"1min":1.0489999999999999,"5min":0.91700000000000004,"15min":0.89500000000000002},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":3.1779999999999999,"5min":3.1779999999999999,"15min":3.1779999999999999},"last":0.90400000000000003}]},{"osd":1,"last update":"Wed Mar 24 21:52:56 2021","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":0.93200000000000005,"15min":0.92500000000000004},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":2.8690000000000002,"5min":2.8690000000000002,"15min":2.8690000000000002},"last":1.0309999999999999},{"interface":"front","average":{"1min":0.94199999999999995,"5min":0.93400000000000005,"15min":0.93300000000000005},"min":{"1min":0.66600000000000004,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":0.88}]},{"osd":2,"last update":"Wed Mar 24 21:52:56 2021","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.90000000000000002,"15min":0.90200000000000002},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.70399999999999996},{"interface":"front","average":{"1min":1.1839999999999999,"5min":0.94699999999999995,"15min":0.90700000000000003},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":5.8479999999999999,"5min":5.8479999999999999,"15min":5.8479999999999999},"last":0.76700000000000002}]},{"osd":3,"last update":"Wed Mar 24 21:52:56 2021","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":0.94399999999999995,"15min":0.93600000000000005},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":3.1259999999999999,"5min":3.1259999999999999,"15min":3.1259999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":1.0409999999999999,"5min":0.95999999999999996,"15min":0.94599999999999995},"min":{"1min":0.67400000000000004,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":2.6850000000000001,"5min":2.6850000000000001,"15min":2.6850000000000001},"last":0.82199999999999995}]},{"osd":4,"last update":"Wed Mar 24 21:52:56 2021","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"min":{"1min":0.55800000000000005,"5min":0.439,"15min":0.439},"max":{"1min":1.1519999999999999,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":1.0549999999999999},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.90700000000000003,"15min":0.89500000000000002},"min":{"1min":0.71499999999999997,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.171,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.996}]},{"osd":6,"last update":"Wed Mar 24 21:53:19 2021","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":0.89200000000000002,"15min":0.87},"min":{"1min":0.71999999999999997,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":1.1419999999999999},{"interface":"front","average":{"1min":1.153,"5min":0.99299999999999999,"15min":0.96699999999999997},"min":{"1min":0.72299999999999998,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":2.3079999999999998,"5min":2.3079999999999998,"15min":2.3079999999999998},"last":1.077}]},{"osd":7,"last update":"Wed Mar 24 21:53:42 2021","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.0549999999999999,"15min":1.0469999999999999},"min":{"1min":0.82899999999999996,"5min":0.624,"15min":0.624},"max":{"1min":1.484,"5min":1.484,"15min":1.484},"last":1.165},{"interface":"front","average":{"1min":1.196,"5min":1.0189999999999999,"15min":0.98899999999999999},"min":{"1min":0.88800000000000001,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":2.1539999999999999,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":1.2150000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":7420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":339968,"data_stored":12094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":6670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":11712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":274432,"data_stored":9910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":5638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":9628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":7500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-03-24T21:53:51.189 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2021-03-24T21:53:51.189 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2021-03-24T21:53:51.189 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2021-03-24T21:53:51.190 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph health --format=json 2021-03-24T21:53:51.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:51 smithi142 conmon[20125]: audit 2021-03-24T21:53:51.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:51 smithi142 conmon[20125]: 2021-03-24 2021-03-24T21:53:51.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:51 smithi142 conmon[20125]: T21:53:50.332936+0000 mgr.y (mgr.14144) 285 : audit [DBG] from='client.34589 v1:172.21.15.125:0/1149532475' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T21:53:51.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:51 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:53:51.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:51 smithi142 conmon[20125]: -24T21:53:50.963652+0000 2021-03-24T21:53:51.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:51 smithi142 conmon[20125]: osd.2 (osd.2) 18 : cluster 2021-03-24T21:53:51.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:51 smithi142 conmon[20125]: [DBG] purged_snaps scrub starts 2021-03-24T21:53:51.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:51 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:53:51.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:51 smithi142 conmon[20125]: -03-24T21:53:50 2021-03-24T21:53:51.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:51 smithi142 conmon[20125]: .963820+0000 osd.2 (osd.2 2021-03-24T21:53:51.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:51 smithi142 conmon[20125]: ) 19 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:53:51.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:51 smithi125 conmon[19936]: audit 2021-03-24T21:53:50. 2021-03-24T21:53:51.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:51 smithi125 conmon[19936]: 332936+0000 mgr.y (mgr.14144) 285 : audit [DBG] from='client.34589 v1:172.21.15.125:0/1149532475' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T21:53:51.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:51 smithi125 conmon[19936]: cluster 2021-03-24T21:53:51.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:51 smithi125 conmon[19936]: 2021-03-24 2021-03-24T21:53:51.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:51 smithi125 conmon[19936]: T21:53:50.963652 2021-03-24T21:53:51.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:51 smithi125 conmon[19936]: +0000 osd.2 (osd.2) 2021-03-24T21:53:51.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:51 smithi125 conmon[19936]: 18 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:53:51.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:51 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:53:51.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:51 smithi125 conmon[19936]: -03-24T21:53:50 2021-03-24T21:53:51.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:51 smithi125 conmon[19936]: .963820+0000 osd.2 (osd 2021-03-24T21:53:51.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:51 smithi125 conmon[19936]: .2) 19 : cluster 2021-03-24T21:53:51.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:51 smithi125 conmon[19936]: [DBG] purged_snaps scrub ok 2021-03-24T21:53:51.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:51 smithi125 conmon[24927]: audit 2021-03-24T21:53:51.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:51 smithi125 conmon[24927]: 2021-03 2021-03-24T21:53:51.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:51 smithi125 conmon[24927]: -24T21:53:50.332936+0000 mgr.y (mgr.14144) 285 : audit [DBG] from='client.34589 v1:172.21.15.125:0/1149532475' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T21:53:51.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:51 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:53:51.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:51 smithi125 conmon[24927]: 03-24T21:53:50.963652+0000 osd.2 (osd.2) 18 2021-03-24T21:53:51.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:51 smithi125 conmon[24927]: : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:53:51.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:51 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:53:51.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:51 smithi125 conmon[24927]: 03-24T21:53:50.963820+0000 osd.2 ( 2021-03-24T21:53:51.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:51 smithi125 conmon[24927]: osd.2) 19 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:53:51.509 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:53:52.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:52 smithi142 conmon[20125]: cluster 2021-03-24T21:53:50. 2021-03-24T21:53:52.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:52 smithi142 conmon[20125]: 689790+0000 mgr.y (mgr.14144) 286 : cluster [DBG] pgmap v285: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-24T21:53:52.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:52 smithi142 conmon[20125]: cluster 2021-03-24T21:53:51.100224+0000 mon.a ( 2021-03-24T21:53:52.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:52 smithi142 conmon[20125]: mon.0) 604 : cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2021-03-24T21:53:52.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:52 smithi125 conmon[24927]: cluster 2021-03-24T21:53:50 2021-03-24T21:53:52.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:52 smithi125 conmon[24927]: .689790+0000 mgr.y (mgr.14144) 286 : cluster [DBG] pgmap v285: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-24T21:53:52.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:52 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:53:52.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:52 smithi125 conmon[24927]: -24T21:53:51.100224+0000 mon.a (mon.0) 604 : cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2021-03-24T21:53:52.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:52 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:53:52.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:52 smithi125 conmon[19936]: 03-24T21:53:50.689790+0000 mgr.y (mgr.14144) 286 : cluster [DBG] pgmap v285: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-24T21:53:52.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:52 smithi125 conmon[19936]: cluster 2021-03-24T21:53: 2021-03-24T21:53:52.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:52 smithi125 conmon[19936]: 51.100224+0000 mon.a (mon.0) 604 : 2021-03-24T21:53:52.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:52 smithi125 conmon[19936]: cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2021-03-24T21:53:53.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:53 smithi142 conmon[20125]: cluster 2021-03-24T21:53: 2021-03-24T21:53:53.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:53 smithi142 conmon[20125]: 51.651153+0000 osd.7 (osd.7) 26 : cluster [DBG] 3.e scrub ok 2021-03-24T21:53:53.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:53 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:53:53.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:53 smithi125 conmon[19936]: 21:53:51.651153+0000 osd.7 (osd.7) 26 : cluster [DBG] 3.e scrub ok 2021-03-24T21:53:53.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:53 smithi125 conmon[19936]: 2021-03-24T21:53:53.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:53 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:53:53.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:53 smithi125 conmon[24927]: 53:51.651153+0000 osd.7 (osd.7) 26 : cluster [DBG] 3.e scrub ok 2021-03-24T21:53:53.506 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-24T21:53:54.113 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:54 smithi125 conmon[24927]: cluster 2021-03-24T21:53 2021-03-24T21:53:54.114 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:54 smithi125 conmon[24927]: :52.690532+0000 mgr.y (mgr.14144) 287 : cluster [DBG] pgmap v287: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-24T21:53:54.114 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:54 smithi125 conmon[24927]: audit 2021-03-24T21: 2021-03-24T21:53:54.114 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:54 smithi125 conmon[24927]: 53:53.506612+0000 mon.c (mon.2) 19 2021-03-24T21:53:54.115 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:54 smithi125 conmon[24927]: : audit [DBG] from='client.? v1:172.21.15.125:0/2569462266' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2021-03-24T21:53:54.115 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:54 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:53:54.115 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:54 smithi125 conmon[24927]: -24T21:53:53.735725+0000 osd.5 ( 2021-03-24T21:53:54.349 INFO:teuthology.orchestra.run.smithi125.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2021-03-24T21:53:54.355 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2021-03-24T21:53:54.355 INFO:tasks.cephadm:Setup complete, yielding 2021-03-24T21:53:54.356 INFO:teuthology.run_tasks:Running task thrashosds... 2021-03-24T21:53:54.367 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2021-03-24T21:53:54.367 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-sage2-testing-2021-03-24-1043-pacific'}, 'ceph': {'conf': {'global': {'ms bind msgr2': False, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'mon_bind_msgr2': False, 'sha1': 'd71e96d216190f28f757f409532464ab0eca4ea9'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'cephadm': {'cephadm_mode': 'root'}, 'install': {'ceph': {'sha1': 'd71e96d216190f28f757f409532464ab0eca4ea9'}}, 'workunit': {'branch': 'wip-sage2-testing-2021-03-24-1043-pacific', 'sha1': 'd71e96d216190f28f757f409532464ab0eca4ea9'}} 2021-03-24T21:53:54.367 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2021-03-24T21:53:54.368 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2021-03-24T21:53:54.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:54 smithi142 conmon[20125]: cluster 2021-03-24T21:53:52. 2021-03-24T21:53:54.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:54 smithi142 conmon[20125]: 690532+0000 mgr.y (mgr.14144) 287 : cluster [DBG] pgmap v287: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-24T21:53:54.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:54 smithi142 conmon[20125]: audit 2021-03-24T21:53:53.506612+0000 mon.c ( 2021-03-24T21:53:54.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:54 smithi142 conmon[20125]: mon.2) 19 : audit [DBG] from='client.? v1:172.21.15.125:0/2569462266' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2021-03-24T21:53:54.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:54 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:53:54.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:54 smithi142 conmon[20125]: 03-24T21:53:53.735725+0000 osd.5 (osd.5) 25 : cluster [DBG] 2021-03-24T21:53:54.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:54 smithi142 conmon[20125]: purged_snaps scrub starts 2021-03-24T21:53:54.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:54 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:53:54.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:54 smithi142 conmon[20125]: -03-24T21:53:53.735923+0000 osd.5 (osd.5) 26 : cluster 2021-03-24T21:53:54.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:54 smithi142 conmon[20125]: [DBG] purged_snaps scrub ok 2021-03-24T21:53:54.382 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:54 smithi125 conmon[24927]: osd.5) 25 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:53:54.382 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:54 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:53:54.382 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:54 smithi125 conmon[24927]: -03-24T21:53:53.735923+0000 2021-03-24T21:53:54.382 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:54 smithi125 conmon[24927]: osd.5 (osd.5) 26 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:53:54.383 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:54 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:53:54.383 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:54 smithi125 conmon[19936]: 53:52.690532+0000 mgr.y (mgr.14144) 287 : cluster [DBG] pgmap v287: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-24T21:53:54.383 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:54 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:53:54.384 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:54 smithi125 conmon[19936]: 03-24T21:53:53.506612+0000 mon.c ( 2021-03-24T21:53:54.384 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:54 smithi125 conmon[19936]: mon.2) 19 : audit [DBG] from='client.? v1:172.21.15.125:0/2569462266' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2021-03-24T21:53:54.384 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:54 smithi125 conmon[19936]: cluster 2021-03-24T21:53:54.384 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:54 smithi125 conmon[19936]: 2021-03- 2021-03-24T21:53:54.384 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:54 smithi125 conmon[19936]: 24T21:53:53.735725+0000 osd.5 2021-03-24T21:53:54.385 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:54 smithi125 conmon[19936]: (osd.5) 25 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:53:54.385 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:54 smithi125 conmon[19936]: 2021-03-24T21:53:54.385 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:54 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:53:54.385 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:54 smithi125 conmon[19936]: 21:53:53.735923+0000 osd.5 2021-03-24T21:53:54.385 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:54 smithi125 conmon[19936]: (osd.5) 26 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:53:54.386 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:54 smithi125 conmon[19936]: 2021-03-24T21:53:54.689 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:53:55.296 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:55 smithi125 conmon[19936]: cluster 2021-03-24T21:53: 2021-03-24T21:53:55.296 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:55 smithi125 conmon[19936]: 54.124096+0000 mon.a (mon.0) 605 : cluster [DBG] osdmap e94: 8 total, 8 up, 8 in 2021-03-24T21:53:55.297 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:55 smithi125 conmon[24927]: cluster 2021-03-24T21:53 2021-03-24T21:53:55.297 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:55 smithi125 conmon[24927]: :54.124096+0000 mon.a (mon.0) 605 : cluster [DBG] osdmap e94: 8 total, 8 up, 8 in 2021-03-24T21:53:55.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:55 smithi142 conmon[20125]: cluster 2021-03-24T21:53:54.124096 2021-03-24T21:53:55.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:55 smithi142 conmon[20125]: +0000 mon.a (mon.0) 605 : cluster [DBG] osdmap e94: 8 total, 8 up, 8 in 2021-03-24T21:53:56.227 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:56 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:53:56.228 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:56 smithi125 conmon[24927]: 21:53:54.691246+0000 mgr.y (mgr.14144) 288 : cluster [DBG] pgmap v289: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-24T21:53:56.228 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:56 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:53:56.228 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:56 smithi125 conmon[24927]: :53:55.497371+0000 osd.2 (osd. 2021-03-24T21:53:56.229 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:56 smithi125 conmon[24927]: 2) 20 : cluster [DBG] 2.1 scrub ok 2021-03-24T21:53:56.229 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:56 smithi125 conmon[24927]: 2021-03-24T21:53:56.229 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:56 smithi125 conmon[19936]: cluster 2021-03-24T21:53: 2021-03-24T21:53:56.230 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:56 smithi125 conmon[19936]: 54.691246+0000 mgr.y (mgr.14144) 288 : cluster [DBG] pgmap v289: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-24T21:53:56.230 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:56 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:53:56.230 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:56 smithi125 conmon[19936]: 24T21:53:55.497371+0000 osd.2 (osd.2) 20 : cluster [DBG] 2.1 scrub ok 2021-03-24T21:53:56.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:56 smithi142 conmon[20125]: cluster 2021-03-24T21:53:54.691246+0000 mgr.y (mgr.14144) 288 : cluster 2021-03-24T21:53:56.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:56 smithi142 conmon[20125]: [DBG] pgmap v289: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-24T21:53:56.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:56 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:53:56.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:56 smithi142 conmon[20125]: 03-24T21:53:55.497371+0000 osd.2 (osd.2) 2021-03-24T21:53:56.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:56 smithi142 conmon[20125]: 20 : cluster [DBG] 2.1 scrub ok 2021-03-24T21:53:57.442 INFO:tasks.thrashosds:Beginning thrashosds... 2021-03-24T21:53:57.442 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2021-03-24T21:53:57.443 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph pg dump --format=json 2021-03-24T21:53:57.793 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:53:57.909 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:57 smithi125 conmon[19936]: audit 2021-03-24T21:53:56.641838+0000 mon.a ( 2021-03-24T21:53:57.909 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:57 smithi125 conmon[19936]: mon.0) 606 : audit [INF] from='client.? v1:172.21.15.125:0/2244175082' entity='client.admin' 2021-03-24T21:53:57.910 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:57 smithi125 conmon[19936]: audit 2021-03-24T21:53:56.649979+0000 mon.a (mon.0) 607 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:53:57.910 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:57 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:53:57.910 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:57 smithi125 conmon[19936]: -24T21:53:56.692130+0000 mgr.y (mgr.14144) 289 : cluster [DBG] pgmap v290: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-24T21:53:57.910 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:57 smithi125 conmon[19936]: audit 2021-03-24T21:53:57.911 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:57 smithi125 conmon[19936]: 2021-03-24T21:53:56.950384+0000 mon.a (mon.0) 608 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:53:57.911 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:57 smithi125 conmon[19936]: 2021-03-24T21:53:57.911 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:57 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T21:53:57.911 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:57 smithi125 conmon[24927]: 24T21:53:56.641838+0000 mon.a (mon.0) 606 : audit 2021-03-24T21:53:57.912 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:57 smithi125 conmon[24927]: [INF] from='client.? v1:172.21.15.125:0/2244175082' entity='client.admin' 2021-03-24T21:53:57.912 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:57 smithi125 conmon[24927]: audit 2021-03-24T 2021-03-24T21:53:57.912 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:57 smithi125 conmon[24927]: 21:53:56.649979+0000 mon.a (mon.0) 607 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:53:57.912 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:57 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:53:57.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:57 smithi125 conmon[24927]: :53:56.692130+0000 mgr.y (mgr.14144 2021-03-24T21:53:57.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:57 smithi125 conmon[24927]: ) 289 : cluster [DBG] pgmap v290: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-24T21:53:57.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:57 smithi125 conmon[24927]: audit 2021- 2021-03-24T21:53:57.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:57 smithi125 conmon[24927]: 03-24T21:53:56.950384+0000 mon.a 2021-03-24T21:53:57.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:57 smithi125 conmon[24927]: (mon.0) 608 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:53:57.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:57 smithi125 conmon[24927]: 2021-03-24T21:53:58.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:57 smithi142 conmon[20125]: audit 2021-03-24T21:53: 2021-03-24T21:53:58.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:57 smithi142 conmon[20125]: 56.641838+0000 mon.a (mon.0) 606 : audit [INF] from='client.? v1:172.21.15.125:0/2244175082' entity='client.admin' 2021-03-24T21:53:58.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:57 smithi142 conmon[20125]: audit 2021-03-24T21: 2021-03-24T21:53:58.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:57 smithi142 conmon[20125]: 53:56.649979+0000 mon.a (mon.0) 607 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:53:58.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:57 smithi142 conmon[20125]: cluster 2021-03-24T21:53:58.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:57 smithi142 conmon[20125]: 2021-03-24T21:53:56.692130+0000 2021-03-24T21:53:58.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:57 smithi142 conmon[20125]: mgr.y (mgr.14144) 289 : cluster [DBG] 2021-03-24T21:53:58.119 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:57 smithi142 conmon[20125]: pgmap v290: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-24T21:53:58.119 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:57 smithi142 conmon[20125]: audit 2021-03-24T21: 2021-03-24T21:53:58.119 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:57 smithi142 conmon[20125]: 53:56.950384+0000 mon.a (mon.0) 2021-03-24T21:53:58.119 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:57 smithi142 conmon[20125]: 608 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:53:58.909 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:58 smithi125 conmon[19936]: cluster 2021-03-24T21:53:57 2021-03-24T21:53:58.909 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:58 smithi125 conmon[19936]: .501341+0000 osd.3 (osd.3) 33 : cluster [DBG] 3.b scrub ok 2021-03-24T21:53:58.910 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:58 smithi125 conmon[24927]: cluster 2021-03-24T21:53:57 2021-03-24T21:53:58.910 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:58 smithi125 conmon[24927]: .501341+0000 osd.3 (osd.3) 33 : cluster [DBG] 3.b scrub ok 2021-03-24T21:53:59.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:58 smithi142 conmon[20125]: cluster 2021-03-24T21:53:57.501341+0000 osd.3 (osd.3) 33 : cluster 2021-03-24T21:53:59.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:58 smithi142 conmon[20125]: [DBG] 3.b scrub ok 2021-03-24T21:53:59.718 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-24T21:53:59.721 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2021-03-24T21:53:59.909 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:59 smithi125 conmon[24927]: cluster 2021-03-24T21:53:58.587141+0000 osd.2 (osd.2) 2021-03-24T21:53:59.909 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:59 smithi125 conmon[24927]: 21 : cluster [DBG] 4.1c scrub ok 2021-03-24T21:53:59.909 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:53:59 smithi125 conmon[24927]: cluster 2021-03-24T21:53:58.693080+0000 mgr.y (mgr.14144) 290 : cluster [DBG] pgmap v291: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-03-24T21:53:59.910 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:59 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:53:59.910 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:59 smithi125 conmon[19936]: 21:53:58.587141+0000 osd.2 (osd.2) 21 : cluster [DBG] 4.1c scrub ok 2021-03-24T21:53:59.910 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:59 smithi125 conmon[19936]: cluster 2021-03-24T21:53:59.911 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:59 smithi125 conmon[19936]: 2021-03-24 2021-03-24T21:53:59.911 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:59 smithi125 conmon[19936]: T21:53:58. 2021-03-24T21:53:59.911 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:59 smithi125 conmon[19936]: 693080+0000 mgr.y ( 2021-03-24T21:53:59.911 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:59 smithi125 conmon[19936]: mgr.14144) 290 2021-03-24T21:53:59.911 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:59 smithi125 conmon[19936]: : cluster [DBG] 2021-03-24T21:53:59.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:53:59 smithi125 conmon[19936]: pgmap v291: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-03-24T21:54:00.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:59 smithi142 conmon[20125]: cluster 2021-03-24T21:53 2021-03-24T21:54:00.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:59 smithi142 conmon[20125]: :58.587141+0000 osd.2 (osd.2) 21 : cluster [DBG] 4.1c scrub ok 2021-03-24T21:54:00.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:59 smithi142 conmon[20125]: cluster 2021-03-24T21:54:00.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:59 smithi142 conmon[20125]: 2021-03-24T21:53:58.693080+0000 mgr.y (mgr.14144) 2021-03-24T21:54:00.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:53:59 smithi142 conmon[20125]: 290 : cluster [DBG] pgmap v291: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-03-24T21:54:00.510 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":291,"stamp":"2021-03-24T21:53:58.692451+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":25236,"num_objects":316,"num_object_clones":0,"num_object_copies":948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":316,"num_whiteouts":0,"num_read":1201,"num_read_kb":701,"num_write":550,"num_write_kb":165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":2412,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":570,"ondisk_log_size":570,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":53676,"kb_used_data":7020,"kb_used_omap":0,"kb_used_meta":46592,"kb_avail":749907540,"statfs":{"total":767960285184,"available":767905320960,"internally_reserved":0,"allocated":7188480,"data_stored":2610276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":47710208},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003454"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":"40","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.099187+0000","last_change":"2021-03-24T21:52:59.180417+0000","last_active":"2021-03-24T21:53:51.099187+0000","last_peered":"2021-03-24T21:53:51.099187+0000","last_clean":"2021-03-24T21:53:51.099187+0000","last_became_active":"2021-03-24T21:51:51.759485+0000","last_became_peered":"2021-03-24T21:51:51.759485+0000","last_unstale":"2021-03-24T21:53:51.099187+0000","last_undegraded":"2021-03-24T21:53:51.099187+0000","last_fullsized":"2021-03-24T21:53:51.099187+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:59.180240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:52:59.180240+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"43","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.100701+0000","last_change":"2021-03-24T21:53:02.553145+0000","last_active":"2021-03-24T21:53:51.100701+0000","last_peered":"2021-03-24T21:53:51.100701+0000","last_clean":"2021-03-24T21:53:51.100701+0000","last_became_active":"2021-03-24T21:51:49.415254+0000","last_became_peered":"2021-03-24T21:51:49.415254+0000","last_unstale":"2021-03-24T21:53:51.100701+0000","last_undegraded":"2021-03-24T21:53:51.100701+0000","last_fullsized":"2021-03-24T21:53:51.100701+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:02.553036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:02.553036+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":"44","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.453764+0000","last_change":"2021-03-24T21:53:12.428679+0000","last_active":"2021-03-24T21:53:37.453764+0000","last_peered":"2021-03-24T21:53:37.453764+0000","last_clean":"2021-03-24T21:53:37.453764+0000","last_became_active":"2021-03-24T21:51:45.581000+0000","last_became_peered":"2021-03-24T21:51:45.581000+0000","last_unstale":"2021-03-24T21:53:37.453764+0000","last_undegraded":"2021-03-24T21:53:37.453764+0000","last_fullsized":"2021-03-24T21:53:37.453764+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:12.428497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:12.428497+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"69'15","reported_seq":"72","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.098411+0000","last_change":"2021-03-24T21:53:41.296073+0000","last_active":"2021-03-24T21:53:51.098411+0000","last_peered":"2021-03-24T21:53:51.098411+0000","last_clean":"2021-03-24T21:53:51.098411+0000","last_became_active":"2021-03-24T21:51:47.410353+0000","last_became_peered":"2021-03-24T21:51:47.410353+0000","last_unstale":"2021-03-24T21:53:51.098411+0000","last_undegraded":"2021-03-24T21:53:51.098411+0000","last_fullsized":"2021-03-24T21:53:51.098411+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:53:41.295875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:41.295875+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"39","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.642530+0000","last_change":"2021-03-24T21:52:58.676144+0000","last_active":"2021-03-24T21:53:37.642530+0000","last_peered":"2021-03-24T21:53:37.642530+0000","last_clean":"2021-03-24T21:53:37.642530+0000","last_became_active":"2021-03-24T21:51:51.915108+0000","last_became_peered":"2021-03-24T21:51:51.915108+0000","last_unstale":"2021-03-24T21:53:37.642530+0000","last_undegraded":"2021-03-24T21:53:37.642530+0000","last_fullsized":"2021-03-24T21:53:37.642530+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:58.676060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:52:58.676060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"42","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.098406+0000","last_change":"2021-03-24T21:53:48.329647+0000","last_active":"2021-03-24T21:53:51.098406+0000","last_peered":"2021-03-24T21:53:51.098406+0000","last_clean":"2021-03-24T21:53:51.098406+0000","last_became_active":"2021-03-24T21:51:49.409015+0000","last_became_peered":"2021-03-24T21:51:49.409015+0000","last_unstale":"2021-03-24T21:53:51.098406+0000","last_undegraded":"2021-03-24T21:53:51.098406+0000","last_fullsized":"2021-03-24T21:53:51.098406+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:48.329480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:48.329480+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":"45","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.643926+0000","last_change":"2021-03-24T21:53:18.706895+0000","last_active":"2021-03-24T21:53:37.643926+0000","last_peered":"2021-03-24T21:53:37.643926+0000","last_clean":"2021-03-24T21:53:37.643926+0000","last_became_active":"2021-03-24T21:51:45.581465+0000","last_became_peered":"2021-03-24T21:51:45.581465+0000","last_unstale":"2021-03-24T21:53:37.643926+0000","last_undegraded":"2021-03-24T21:53:37.643926+0000","last_fullsized":"2021-03-24T21:53:37.643926+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:18.706745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:18.706745+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"69'20","reported_seq":"78","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.641956+0000","last_change":"2021-03-24T21:53:12.604981+0000","last_active":"2021-03-24T21:53:37.641956+0000","last_peered":"2021-03-24T21:53:37.641956+0000","last_clean":"2021-03-24T21:53:37.641956+0000","last_became_active":"2021-03-24T21:51:47.405776+0000","last_became_peered":"2021-03-24T21:51:47.405776+0000","last_unstale":"2021-03-24T21:53:37.641956+0000","last_undegraded":"2021-03-24T21:53:37.641956+0000","last_fullsized":"2021-03-24T21:53:37.641956+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'20","last_scrub_stamp":"2021-03-24T21:53:12.604757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:12.604757+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":22,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"38","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.452406+0000","last_change":"2021-03-24T21:53:27.604422+0000","last_active":"2021-03-24T21:53:37.452406+0000","last_peered":"2021-03-24T21:53:37.452406+0000","last_clean":"2021-03-24T21:53:37.452406+0000","last_became_active":"2021-03-24T21:51:51.758026+0000","last_became_peered":"2021-03-24T21:51:51.758026+0000","last_unstale":"2021-03-24T21:53:37.452406+0000","last_undegraded":"2021-03-24T21:53:37.452406+0000","last_fullsized":"2021-03-24T21:53:37.452406+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:27.604230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:27.604230+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"43","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.549752+0000","last_change":"2021-03-24T21:52:56.855476+0000","last_active":"2021-03-24T21:53:51.549752+0000","last_peered":"2021-03-24T21:53:51.549752+0000","last_clean":"2021-03-24T21:53:51.549752+0000","last_became_active":"2021-03-24T21:51:49.413342+0000","last_became_peered":"2021-03-24T21:51:49.413342+0000","last_unstale":"2021-03-24T21:53:51.549752+0000","last_undegraded":"2021-03-24T21:53:51.549752+0000","last_fullsized":"2021-03-24T21:53:51.549752+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:56.855305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:52:56.855305+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":"47","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.550271+0000","last_change":"2021-03-24T21:52:50.788792+0000","last_active":"2021-03-24T21:53:51.550271+0000","last_peered":"2021-03-24T21:53:51.550271+0000","last_clean":"2021-03-24T21:53:51.550271+0000","last_became_active":"2021-03-24T21:51:45.382348+0000","last_became_peered":"2021-03-24T21:51:45.382348+0000","last_unstale":"2021-03-24T21:53:51.550271+0000","last_undegraded":"2021-03-24T21:53:51.550271+0000","last_fullsized":"2021-03-24T21:53:51.550271+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:50.788606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:52:50.788606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"69'13","reported_seq":"68","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.099593+0000","last_change":"2021-03-24T21:53:10.104959+0000","last_active":"2021-03-24T21:53:51.099593+0000","last_peered":"2021-03-24T21:53:51.099593+0000","last_clean":"2021-03-24T21:53:51.099593+0000","last_became_active":"2021-03-24T21:51:47.407488+0000","last_became_peered":"2021-03-24T21:51:47.407488+0000","last_unstale":"2021-03-24T21:53:51.099593+0000","last_undegraded":"2021-03-24T21:53:51.099593+0000","last_fullsized":"2021-03-24T21:53:51.099593+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2021-03-24T21:53:10.104806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:10.104806+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"41","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.549132+0000","last_change":"2021-03-24T21:53:23.896132+0000","last_active":"2021-03-24T21:53:51.549132+0000","last_peered":"2021-03-24T21:53:51.549132+0000","last_clean":"2021-03-24T21:53:51.549132+0000","last_became_active":"2021-03-24T21:51:51.765338+0000","last_became_peered":"2021-03-24T21:51:51.765338+0000","last_unstale":"2021-03-24T21:53:51.549132+0000","last_undegraded":"2021-03-24T21:53:51.549132+0000","last_fullsized":"2021-03-24T21:53:51.549132+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:23.895966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:23.895966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"40","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.451458+0000","last_change":"2021-03-24T21:53:00.845488+0000","last_active":"2021-03-24T21:53:37.451458+0000","last_peered":"2021-03-24T21:53:37.451458+0000","last_clean":"2021-03-24T21:53:37.451458+0000","last_became_active":"2021-03-24T21:51:49.411327+0000","last_became_peered":"2021-03-24T21:51:49.411327+0000","last_unstale":"2021-03-24T21:53:37.451458+0000","last_undegraded":"2021-03-24T21:53:37.451458+0000","last_fullsized":"2021-03-24T21:53:37.451458+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:00.845338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:00.845338+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":"47","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.549645+0000","last_change":"2021-03-24T21:52:45.798051+0000","last_active":"2021-03-24T21:53:51.549645+0000","last_peered":"2021-03-24T21:53:51.549645+0000","last_clean":"2021-03-24T21:53:51.549645+0000","last_became_active":"2021-03-24T21:51:45.381362+0000","last_became_peered":"2021-03-24T21:51:45.381362+0000","last_unstale":"2021-03-24T21:53:51.549645+0000","last_undegraded":"2021-03-24T21:53:51.549645+0000","last_fullsized":"2021-03-24T21:53:51.549645+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:45.797873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:52:45.797873+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"91'9","reported_seq":"59","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.099621+0000","last_change":"2021-03-24T21:53:33.290178+0000","last_active":"2021-03-24T21:53:51.099621+0000","last_peered":"2021-03-24T21:53:51.099621+0000","last_clean":"2021-03-24T21:53:51.099621+0000","last_became_active":"2021-03-24T21:51:47.416379+0000","last_became_peered":"2021-03-24T21:51:47.416379+0000","last_unstale":"2021-03-24T21:53:51.099621+0000","last_undegraded":"2021-03-24T21:53:51.099621+0000","last_fullsized":"2021-03-24T21:53:51.099621+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2021-03-24T21:53:33.290000+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2021-03-24T21:53:33.290000+0000","last_clean_scrub_stamp":"2021-03-24T21:53:33.290000+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":8,"num_write":7,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"40","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.100374+0000","last_change":"2021-03-24T21:53:39.277601+0000","last_active":"2021-03-24T21:53:51.100374+0000","last_peered":"2021-03-24T21:53:51.100374+0000","last_clean":"2021-03-24T21:53:51.100374+0000","last_became_active":"2021-03-24T21:51:51.910950+0000","last_became_peered":"2021-03-24T21:51:51.910950+0000","last_unstale":"2021-03-24T21:53:51.100374+0000","last_undegraded":"2021-03-24T21:53:51.100374+0000","last_fullsized":"2021-03-24T21:53:51.100374+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:39.277425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:39.277425+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"43","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.102599+0000","last_change":"2021-03-24T21:52:59.579706+0000","last_active":"2021-03-24T21:53:51.102599+0000","last_peered":"2021-03-24T21:53:51.102599+0000","last_clean":"2021-03-24T21:53:51.102599+0000","last_became_active":"2021-03-24T21:51:49.411914+0000","last_became_peered":"2021-03-24T21:51:49.411914+0000","last_unstale":"2021-03-24T21:53:51.102599+0000","last_undegraded":"2021-03-24T21:53:51.102599+0000","last_fullsized":"2021-03-24T21:53:51.102599+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:59.579528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:52:59.579528+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":"44","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.453682+0000","last_change":"2021-03-24T21:53:04.263170+0000","last_active":"2021-03-24T21:53:37.453682+0000","last_peered":"2021-03-24T21:53:37.453682+0000","last_clean":"2021-03-24T21:53:37.453682+0000","last_became_active":"2021-03-24T21:51:45.381533+0000","last_became_peered":"2021-03-24T21:51:45.381533+0000","last_unstale":"2021-03-24T21:53:37.453682+0000","last_undegraded":"2021-03-24T21:53:37.453682+0000","last_fullsized":"2021-03-24T21:53:37.453682+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:04.262989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:04.262989+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"69'15","reported_seq":"95","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.242772+0000","last_change":"2021-03-24T21:52:49.554069+0000","last_active":"2021-03-24T21:53:51.242772+0000","last_peered":"2021-03-24T21:53:51.242772+0000","last_clean":"2021-03-24T21:53:51.242772+0000","last_became_active":"2021-03-24T21:52:06.056116+0000","last_became_peered":"2021-03-24T21:52:06.056116+0000","last_unstale":"2021-03-24T21:53:51.242772+0000","last_undegraded":"2021-03-24T21:53:51.242772+0000","last_fullsized":"2021-03-24T21:53:51.242772+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:52:49.553951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:52:49.553951+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"39","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:44.658024+0000","last_change":"2021-03-24T21:53:44.657926+0000","last_active":"2021-03-24T21:53:44.658024+0000","last_peered":"2021-03-24T21:53:44.658024+0000","last_clean":"2021-03-24T21:53:44.658024+0000","last_became_active":"2021-03-24T21:51:51.914955+0000","last_became_peered":"2021-03-24T21:51:51.914955+0000","last_unstale":"2021-03-24T21:53:44.658024+0000","last_undegraded":"2021-03-24T21:53:44.658024+0000","last_fullsized":"2021-03-24T21:53:44.658024+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:44.657761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:44.657761+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"42","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.099336+0000","last_change":"2021-03-24T21:53:27.238846+0000","last_active":"2021-03-24T21:53:51.099336+0000","last_peered":"2021-03-24T21:53:51.099336+0000","last_clean":"2021-03-24T21:53:51.099336+0000","last_became_active":"2021-03-24T21:51:49.410920+0000","last_became_peered":"2021-03-24T21:51:49.410920+0000","last_unstale":"2021-03-24T21:53:51.099336+0000","last_undegraded":"2021-03-24T21:53:51.099336+0000","last_fullsized":"2021-03-24T21:53:51.099336+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:27.238614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:27.238614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":"46","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.100343+0000","last_change":"2021-03-24T21:52:54.212593+0000","last_active":"2021-03-24T21:53:51.100343+0000","last_peered":"2021-03-24T21:53:51.100343+0000","last_clean":"2021-03-24T21:53:51.100343+0000","last_became_active":"2021-03-24T21:51:45.579326+0000","last_became_peered":"2021-03-24T21:51:45.579326+0000","last_unstale":"2021-03-24T21:53:51.100343+0000","last_undegraded":"2021-03-24T21:53:51.100343+0000","last_fullsized":"2021-03-24T21:53:51.100343+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:54.212459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:52:54.212459+0000","last_clean_scrub_stamp":"2021-03-24T21:52:54.212459+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"69'16","reported_seq":"74","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:42.611784+0000","last_change":"2021-03-24T21:53:42.611713+0000","last_active":"2021-03-24T21:53:42.611784+0000","last_peered":"2021-03-24T21:53:42.611784+0000","last_clean":"2021-03-24T21:53:42.611784+0000","last_became_active":"2021-03-24T21:51:47.411418+0000","last_became_peered":"2021-03-24T21:51:47.411418+0000","last_unstale":"2021-03-24T21:53:42.611784+0000","last_undegraded":"2021-03-24T21:53:42.611784+0000","last_fullsized":"2021-03-24T21:53:42.611784+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T21:53:42.611540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:42.611540+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":37,"num_read_kb":20,"num_write":18,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"38","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.452051+0000","last_change":"2021-03-24T21:53:05.847583+0000","last_active":"2021-03-24T21:53:37.452051+0000","last_peered":"2021-03-24T21:53:37.452051+0000","last_clean":"2021-03-24T21:53:37.452051+0000","last_became_active":"2021-03-24T21:51:51.761492+0000","last_became_peered":"2021-03-24T21:51:51.761492+0000","last_unstale":"2021-03-24T21:53:37.452051+0000","last_undegraded":"2021-03-24T21:53:37.452051+0000","last_fullsized":"2021-03-24T21:53:37.452051+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:05.847412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:05.847412+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"65'2","reported_seq":"69","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:53.817325+0000","last_change":"2021-03-24T21:52:59.074600+0000","last_active":"2021-03-24T21:53:53.817325+0000","last_peered":"2021-03-24T21:53:53.817325+0000","last_clean":"2021-03-24T21:53:53.817325+0000","last_became_active":"2021-03-24T21:51:49.419519+0000","last_became_peered":"2021-03-24T21:51:49.419519+0000","last_unstale":"2021-03-24T21:53:53.817325+0000","last_undegraded":"2021-03-24T21:53:53.817325+0000","last_fullsized":"2021-03-24T21:53:53.817325+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2021-03-24T21:52:59.074417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:52:59.074417+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":"47","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.100870+0000","last_change":"2021-03-24T21:53:09.741058+0000","last_active":"2021-03-24T21:53:51.100870+0000","last_peered":"2021-03-24T21:53:51.100870+0000","last_clean":"2021-03-24T21:53:51.100870+0000","last_became_active":"2021-03-24T21:51:45.374975+0000","last_became_peered":"2021-03-24T21:51:45.374975+0000","last_unstale":"2021-03-24T21:53:51.100870+0000","last_undegraded":"2021-03-24T21:53:51.100870+0000","last_fullsized":"2021-03-24T21:53:51.100870+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:09.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:09.740867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"89'27","reported_seq":"80","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.454460+0000","last_change":"2021-03-24T21:53:21.585108+0000","last_active":"2021-03-24T21:53:37.454460+0000","last_peered":"2021-03-24T21:53:37.454460+0000","last_clean":"2021-03-24T21:53:37.454460+0000","last_became_active":"2021-03-24T21:51:47.688995+0000","last_became_peered":"2021-03-24T21:51:47.688995+0000","last_unstale":"2021-03-24T21:53:37.454460+0000","last_undegraded":"2021-03-24T21:53:37.454460+0000","last_fullsized":"2021-03-24T21:53:37.454460+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'27","last_scrub_stamp":"2021-03-24T21:53:21.584999+0000","last_deep_scrub":"89'27","last_deep_scrub_stamp":"2021-03-24T21:53:21.584999+0000","last_clean_scrub_stamp":"2021-03-24T21:53:21.584999+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"41","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.102757+0000","last_change":"2021-03-24T21:53:14.700106+0000","last_active":"2021-03-24T21:53:51.102757+0000","last_peered":"2021-03-24T21:53:51.102757+0000","last_clean":"2021-03-24T21:53:51.102757+0000","last_became_active":"2021-03-24T21:51:51.758605+0000","last_became_peered":"2021-03-24T21:51:51.758605+0000","last_unstale":"2021-03-24T21:53:51.102757+0000","last_undegraded":"2021-03-24T21:53:51.102757+0000","last_fullsized":"2021-03-24T21:53:51.102757+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:14.699852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:14.699852+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"40","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.454692+0000","last_change":"2021-03-24T21:53:30.660884+0000","last_active":"2021-03-24T21:53:37.454692+0000","last_peered":"2021-03-24T21:53:37.454692+0000","last_clean":"2021-03-24T21:53:37.454692+0000","last_became_active":"2021-03-24T21:51:49.409745+0000","last_became_peered":"2021-03-24T21:51:49.409745+0000","last_unstale":"2021-03-24T21:53:37.454692+0000","last_undegraded":"2021-03-24T21:53:37.454692+0000","last_fullsized":"2021-03-24T21:53:37.454692+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:30.660636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:30.660636+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":"45","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.641476+0000","last_change":"2021-03-24T21:53:11.648668+0000","last_active":"2021-03-24T21:53:37.641476+0000","last_peered":"2021-03-24T21:53:37.641476+0000","last_clean":"2021-03-24T21:53:37.641476+0000","last_became_active":"2021-03-24T21:51:45.373294+0000","last_became_peered":"2021-03-24T21:51:45.373294+0000","last_unstale":"2021-03-24T21:53:37.641476+0000","last_undegraded":"2021-03-24T21:53:37.641476+0000","last_fullsized":"2021-03-24T21:53:37.641476+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:11.648470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:11.648470+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"69'17","reported_seq":"108","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.745417+0000","last_change":"2021-03-24T21:53:17.246451+0000","last_active":"2021-03-24T21:53:51.745417+0000","last_peered":"2021-03-24T21:53:51.745417+0000","last_clean":"2021-03-24T21:53:51.745417+0000","last_became_active":"2021-03-24T21:52:06.054746+0000","last_became_peered":"2021-03-24T21:52:06.054746+0000","last_unstale":"2021-03-24T21:53:51.745417+0000","last_undegraded":"2021-03-24T21:53:51.745417+0000","last_fullsized":"2021-03-24T21:53:51.745417+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'17","last_scrub_stamp":"2021-03-24T21:53:17.246367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:17.246367+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":38,"num_read_kb":19,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"38","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.451376+0000","last_change":"2021-03-24T21:53:27.877846+0000","last_active":"2021-03-24T21:53:37.451376+0000","last_peered":"2021-03-24T21:53:37.451376+0000","last_clean":"2021-03-24T21:53:37.451376+0000","last_became_active":"2021-03-24T21:51:51.762150+0000","last_became_peered":"2021-03-24T21:51:51.762150+0000","last_unstale":"2021-03-24T21:53:37.451376+0000","last_undegraded":"2021-03-24T21:53:37.451376+0000","last_fullsized":"2021-03-24T21:53:37.451376+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:27.877627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:27.877627+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"42","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.743897+0000","last_change":"2021-03-24T21:53:13.143455+0000","last_active":"2021-03-24T21:53:51.743897+0000","last_peered":"2021-03-24T21:53:51.743897+0000","last_clean":"2021-03-24T21:53:51.743897+0000","last_became_active":"2021-03-24T21:51:49.414221+0000","last_became_peered":"2021-03-24T21:51:49.414221+0000","last_unstale":"2021-03-24T21:53:51.743897+0000","last_undegraded":"2021-03-24T21:53:51.743897+0000","last_fullsized":"2021-03-24T21:53:51.743897+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:13.143327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:13.143327+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":"44","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.453911+0000","last_change":"2021-03-24T21:53:10.429942+0000","last_active":"2021-03-24T21:53:37.453911+0000","last_peered":"2021-03-24T21:53:37.453911+0000","last_clean":"2021-03-24T21:53:37.453911+0000","last_became_active":"2021-03-24T21:51:45.379789+0000","last_became_peered":"2021-03-24T21:51:45.379789+0000","last_unstale":"2021-03-24T21:53:37.453911+0000","last_undegraded":"2021-03-24T21:53:37.453911+0000","last_fullsized":"2021-03-24T21:53:37.453911+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:10.429783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:10.429783+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"69'12","reported_seq":"96","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.641460+0000","last_change":"2021-03-24T21:52:52.579983+0000","last_active":"2021-03-24T21:53:37.641460+0000","last_peered":"2021-03-24T21:53:37.641460+0000","last_clean":"2021-03-24T21:53:37.641460+0000","last_became_active":"2021-03-24T21:52:06.053964+0000","last_became_peered":"2021-03-24T21:52:06.053964+0000","last_unstale":"2021-03-24T21:53:37.641460+0000","last_undegraded":"2021-03-24T21:53:37.641460+0000","last_fullsized":"2021-03-24T21:53:37.641460+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2021-03-24T21:52:52.579796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:52:52.579796+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":12,"num_write":12,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"39","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:47.541082+0000","last_change":"2021-03-24T21:53:47.540975+0000","last_active":"2021-03-24T21:53:47.541082+0000","last_peered":"2021-03-24T21:53:47.541082+0000","last_clean":"2021-03-24T21:53:47.541082+0000","last_became_active":"2021-03-24T21:51:51.911828+0000","last_became_peered":"2021-03-24T21:51:51.911828+0000","last_unstale":"2021-03-24T21:53:47.541082+0000","last_undegraded":"2021-03-24T21:53:47.541082+0000","last_fullsized":"2021-03-24T21:53:47.541082+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:47.540793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:47.540793+0000","last_clean_scrub_stamp":"2021-03-24T21:53:47.540793+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"40","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.452028+0000","last_change":"2021-03-24T21:52:52.829549+0000","last_active":"2021-03-24T21:53:37.452028+0000","last_peered":"2021-03-24T21:53:37.452028+0000","last_clean":"2021-03-24T21:53:37.452028+0000","last_became_active":"2021-03-24T21:51:49.422591+0000","last_became_peered":"2021-03-24T21:51:49.422591+0000","last_unstale":"2021-03-24T21:53:37.452028+0000","last_undegraded":"2021-03-24T21:53:37.452028+0000","last_fullsized":"2021-03-24T21:53:37.452028+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:52.829381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:52:52.829381+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":"47","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.550276+0000","last_change":"2021-03-24T21:53:21.833111+0000","last_active":"2021-03-24T21:53:51.550276+0000","last_peered":"2021-03-24T21:53:51.550276+0000","last_clean":"2021-03-24T21:53:51.550276+0000","last_became_active":"2021-03-24T21:51:45.385892+0000","last_became_peered":"2021-03-24T21:51:45.385892+0000","last_unstale":"2021-03-24T21:53:51.550276+0000","last_undegraded":"2021-03-24T21:53:51.550276+0000","last_fullsized":"2021-03-24T21:53:51.550276+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:21.832921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:21.832921+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"86'22","reported_seq":"76","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.641785+0000","last_change":"2021-03-24T21:52:54.597687+0000","last_active":"2021-03-24T21:53:37.641785+0000","last_peered":"2021-03-24T21:53:37.641785+0000","last_clean":"2021-03-24T21:53:37.641785+0000","last_became_active":"2021-03-24T21:51:47.687561+0000","last_became_peered":"2021-03-24T21:51:47.687561+0000","last_unstale":"2021-03-24T21:53:37.641785+0000","last_undegraded":"2021-03-24T21:53:37.641785+0000","last_fullsized":"2021-03-24T21:53:37.641785+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'22","last_scrub_stamp":"2021-03-24T21:52:54.597505+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2021-03-24T21:52:54.597505+0000","last_clean_scrub_stamp":"2021-03-24T21:52:54.597505+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":16,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"38","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.451193+0000","last_change":"2021-03-24T21:53:13.462014+0000","last_active":"2021-03-24T21:53:37.451193+0000","last_peered":"2021-03-24T21:53:37.451193+0000","last_clean":"2021-03-24T21:53:37.451193+0000","last_became_active":"2021-03-24T21:51:51.757253+0000","last_became_peered":"2021-03-24T21:51:51.757253+0000","last_unstale":"2021-03-24T21:53:37.451193+0000","last_undegraded":"2021-03-24T21:53:37.451193+0000","last_fullsized":"2021-03-24T21:53:37.451193+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:13.461940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:13.461940+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"41","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.642662+0000","last_change":"2021-03-24T21:53:31.665546+0000","last_active":"2021-03-24T21:53:37.642662+0000","last_peered":"2021-03-24T21:53:37.642662+0000","last_clean":"2021-03-24T21:53:37.642662+0000","last_became_active":"2021-03-24T21:51:49.418515+0000","last_became_peered":"2021-03-24T21:51:49.418515+0000","last_unstale":"2021-03-24T21:53:37.642662+0000","last_undegraded":"2021-03-24T21:53:37.642662+0000","last_fullsized":"2021-03-24T21:53:37.642662+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:31.665434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:31.665434+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":"44","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.451863+0000","last_change":"2021-03-24T21:53:08.841487+0000","last_active":"2021-03-24T21:53:37.451863+0000","last_peered":"2021-03-24T21:53:37.451863+0000","last_clean":"2021-03-24T21:53:37.451863+0000","last_became_active":"2021-03-24T21:51:45.379383+0000","last_became_peered":"2021-03-24T21:51:45.379383+0000","last_unstale":"2021-03-24T21:53:37.451863+0000","last_undegraded":"2021-03-24T21:53:37.451863+0000","last_fullsized":"2021-03-24T21:53:37.451863+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:08.841310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:08.841310+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"69'15","reported_seq":"73","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.101931+0000","last_change":"2021-03-24T21:53:04.637959+0000","last_active":"2021-03-24T21:53:51.101931+0000","last_peered":"2021-03-24T21:53:51.101931+0000","last_clean":"2021-03-24T21:53:51.101931+0000","last_became_active":"2021-03-24T21:51:47.403130+0000","last_became_peered":"2021-03-24T21:51:47.403130+0000","last_unstale":"2021-03-24T21:53:51.101931+0000","last_undegraded":"2021-03-24T21:53:51.101931+0000","last_fullsized":"2021-03-24T21:53:51.101931+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:53:04.637877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:04.637877+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"39","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.641507+0000","last_change":"2021-03-24T21:53:23.708000+0000","last_active":"2021-03-24T21:53:37.641507+0000","last_peered":"2021-03-24T21:53:37.641507+0000","last_clean":"2021-03-24T21:53:37.641507+0000","last_became_active":"2021-03-24T21:51:51.764871+0000","last_became_peered":"2021-03-24T21:51:51.764871+0000","last_unstale":"2021-03-24T21:53:37.641507+0000","last_undegraded":"2021-03-24T21:53:37.641507+0000","last_fullsized":"2021-03-24T21:53:37.641507+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:23.707914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:23.707914+0000","last_clean_scrub_stamp":"2021-03-24T21:53:23.707914+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"43","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.101229+0000","last_change":"2021-03-24T21:52:52.702414+0000","last_active":"2021-03-24T21:53:51.101229+0000","last_peered":"2021-03-24T21:53:51.101229+0000","last_clean":"2021-03-24T21:53:51.101229+0000","last_became_active":"2021-03-24T21:51:49.415559+0000","last_became_peered":"2021-03-24T21:51:49.415559+0000","last_unstale":"2021-03-24T21:53:51.101229+0000","last_undegraded":"2021-03-24T21:53:51.101229+0000","last_fullsized":"2021-03-24T21:53:51.101229+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:52.702305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:52:52.702305+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":"46","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.099609+0000","last_change":"2021-03-24T21:53:36.317121+0000","last_active":"2021-03-24T21:53:51.099609+0000","last_peered":"2021-03-24T21:53:51.099609+0000","last_clean":"2021-03-24T21:53:51.099609+0000","last_became_active":"2021-03-24T21:51:45.379862+0000","last_became_peered":"2021-03-24T21:51:45.379862+0000","last_unstale":"2021-03-24T21:53:51.099609+0000","last_undegraded":"2021-03-24T21:53:51.099609+0000","last_fullsized":"2021-03-24T21:53:51.099609+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:36.316893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:36.316893+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"69'15","reported_seq":"71","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.099732+0000","last_change":"2021-03-24T21:53:21.207831+0000","last_active":"2021-03-24T21:53:51.099732+0000","last_peered":"2021-03-24T21:53:51.099732+0000","last_clean":"2021-03-24T21:53:51.099732+0000","last_became_active":"2021-03-24T21:51:47.416319+0000","last_became_peered":"2021-03-24T21:51:47.416319+0000","last_unstale":"2021-03-24T21:53:51.099732+0000","last_undegraded":"2021-03-24T21:53:51.099732+0000","last_fullsized":"2021-03-24T21:53:51.099732+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:53:21.207696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:21.207696+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"38","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.452389+0000","last_change":"2021-03-24T21:53:06.309234+0000","last_active":"2021-03-24T21:53:37.452389+0000","last_peered":"2021-03-24T21:53:37.452389+0000","last_clean":"2021-03-24T21:53:37.452389+0000","last_became_active":"2021-03-24T21:51:51.756584+0000","last_became_peered":"2021-03-24T21:51:51.756584+0000","last_unstale":"2021-03-24T21:53:37.452389+0000","last_undegraded":"2021-03-24T21:53:37.452389+0000","last_fullsized":"2021-03-24T21:53:37.452389+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:06.309066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:06.309066+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"86'3","reported_seq":"71","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:53.817486+0000","last_change":"2021-03-24T21:53:08.815321+0000","last_active":"2021-03-24T21:53:53.817486+0000","last_peered":"2021-03-24T21:53:53.817486+0000","last_clean":"2021-03-24T21:53:53.817486+0000","last_became_active":"2021-03-24T21:51:49.419663+0000","last_became_peered":"2021-03-24T21:51:49.419663+0000","last_unstale":"2021-03-24T21:53:53.817486+0000","last_undegraded":"2021-03-24T21:53:53.817486+0000","last_fullsized":"2021-03-24T21:53:53.817486+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'3","last_scrub_stamp":"2021-03-24T21:53:08.815163+0000","last_deep_scrub":"86'3","last_deep_scrub_stamp":"2021-03-24T21:53:08.815163+0000","last_clean_scrub_stamp":"2021-03-24T21:53:08.815163+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":"44","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.452046+0000","last_change":"2021-03-24T21:53:15.868395+0000","last_active":"2021-03-24T21:53:37.452046+0000","last_peered":"2021-03-24T21:53:37.452046+0000","last_clean":"2021-03-24T21:53:37.452046+0000","last_became_active":"2021-03-24T21:51:45.579271+0000","last_became_peered":"2021-03-24T21:51:45.579271+0000","last_unstale":"2021-03-24T21:53:37.452046+0000","last_undegraded":"2021-03-24T21:53:37.452046+0000","last_fullsized":"2021-03-24T21:53:37.452046+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:15.868303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:15.868303+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"69'14","reported_seq":"66","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.643317+0000","last_change":"2021-03-24T21:53:02.723125+0000","last_active":"2021-03-24T21:53:37.643317+0000","last_peered":"2021-03-24T21:53:37.643317+0000","last_clean":"2021-03-24T21:53:37.643317+0000","last_became_active":"2021-03-24T21:51:47.686103+0000","last_became_peered":"2021-03-24T21:51:47.686103+0000","last_unstale":"2021-03-24T21:53:37.643317+0000","last_undegraded":"2021-03-24T21:53:37.643317+0000","last_fullsized":"2021-03-24T21:53:37.643317+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'14","last_scrub_stamp":"2021-03-24T21:53:02.722970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:02.722970+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":27,"num_read_kb":14,"num_write":14,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"40","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.101014+0000","last_change":"2021-03-24T21:53:12.030422+0000","last_active":"2021-03-24T21:53:51.101014+0000","last_peered":"2021-03-24T21:53:51.101014+0000","last_clean":"2021-03-24T21:53:51.101014+0000","last_became_active":"2021-03-24T21:51:51.762876+0000","last_became_peered":"2021-03-24T21:51:51.762876+0000","last_unstale":"2021-03-24T21:53:51.101014+0000","last_undegraded":"2021-03-24T21:53:51.101014+0000","last_fullsized":"2021-03-24T21:53:51.101014+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:12.030251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:12.030251+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"86'3","reported_seq":"69","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.746520+0000","last_change":"2021-03-24T21:52:56.678231+0000","last_active":"2021-03-24T21:53:51.746520+0000","last_peered":"2021-03-24T21:53:51.746520+0000","last_clean":"2021-03-24T21:53:51.746520+0000","last_became_active":"2021-03-24T21:51:49.419797+0000","last_became_peered":"2021-03-24T21:51:49.419797+0000","last_unstale":"2021-03-24T21:53:51.746520+0000","last_undegraded":"2021-03-24T21:53:51.746520+0000","last_fullsized":"2021-03-24T21:53:51.746520+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'3","last_scrub_stamp":"2021-03-24T21:52:56.678111+0000","last_deep_scrub":"86'3","last_deep_scrub_stamp":"2021-03-24T21:52:56.678111+0000","last_clean_scrub_stamp":"2021-03-24T21:52:56.678111+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":"44","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.452487+0000","last_change":"2021-03-24T21:52:51.450624+0000","last_active":"2021-03-24T21:53:37.452487+0000","last_peered":"2021-03-24T21:53:37.452487+0000","last_clean":"2021-03-24T21:53:37.452487+0000","last_became_active":"2021-03-24T21:51:45.381516+0000","last_became_peered":"2021-03-24T21:51:45.381516+0000","last_unstale":"2021-03-24T21:53:37.452487+0000","last_undegraded":"2021-03-24T21:53:37.452487+0000","last_fullsized":"2021-03-24T21:53:37.452487+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:51.450347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:52:51.450347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"69'13","reported_seq":"69","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.550774+0000","last_change":"2021-03-24T21:53:15.901347+0000","last_active":"2021-03-24T21:53:51.550774+0000","last_peered":"2021-03-24T21:53:51.550774+0000","last_clean":"2021-03-24T21:53:51.550774+0000","last_became_active":"2021-03-24T21:51:47.689288+0000","last_became_peered":"2021-03-24T21:51:47.689288+0000","last_unstale":"2021-03-24T21:53:51.550774+0000","last_undegraded":"2021-03-24T21:53:51.550774+0000","last_fullsized":"2021-03-24T21:53:51.550774+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2021-03-24T21:53:15.901150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:15.901150+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"41","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.243227+0000","last_change":"2021-03-24T21:53:15.623047+0000","last_active":"2021-03-24T21:53:51.243227+0000","last_peered":"2021-03-24T21:53:51.243227+0000","last_clean":"2021-03-24T21:53:51.243227+0000","last_became_active":"2021-03-24T21:51:51.754085+0000","last_became_peered":"2021-03-24T21:51:51.754085+0000","last_unstale":"2021-03-24T21:53:51.243227+0000","last_undegraded":"2021-03-24T21:53:51.243227+0000","last_fullsized":"2021-03-24T21:53:51.243227+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:15.622946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:15.622946+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"41","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.643361+0000","last_change":"2021-03-24T21:52:57.661001+0000","last_active":"2021-03-24T21:53:37.643361+0000","last_peered":"2021-03-24T21:53:37.643361+0000","last_clean":"2021-03-24T21:53:37.643361+0000","last_became_active":"2021-03-24T21:51:49.411882+0000","last_became_peered":"2021-03-24T21:51:49.411882+0000","last_unstale":"2021-03-24T21:53:37.643361+0000","last_undegraded":"2021-03-24T21:53:37.643361+0000","last_fullsized":"2021-03-24T21:53:37.643361+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:57.660827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:52:57.660827+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":"44","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.451503+0000","last_change":"2021-03-24T21:53:12.871443+0000","last_active":"2021-03-24T21:53:37.451503+0000","last_peered":"2021-03-24T21:53:37.451503+0000","last_clean":"2021-03-24T21:53:37.451503+0000","last_became_active":"2021-03-24T21:51:45.378937+0000","last_became_peered":"2021-03-24T21:51:45.378937+0000","last_unstale":"2021-03-24T21:53:37.451503+0000","last_undegraded":"2021-03-24T21:53:37.451503+0000","last_fullsized":"2021-03-24T21:53:37.451503+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:12.871269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:12.871269+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"69'16","reported_seq":"72","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.454547+0000","last_change":"2021-03-24T21:53:00.300847+0000","last_active":"2021-03-24T21:53:37.454547+0000","last_peered":"2021-03-24T21:53:37.454547+0000","last_clean":"2021-03-24T21:53:37.454547+0000","last_became_active":"2021-03-24T21:51:47.687595+0000","last_became_peered":"2021-03-24T21:51:47.687595+0000","last_unstale":"2021-03-24T21:53:37.454547+0000","last_undegraded":"2021-03-24T21:53:37.454547+0000","last_fullsized":"2021-03-24T21:53:37.454547+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T21:53:00.300673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:00.300673+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":38,"num_read_kb":18,"num_write":17,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"38","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.451872+0000","last_change":"2021-03-24T21:53:24.913418+0000","last_active":"2021-03-24T21:53:37.451872+0000","last_peered":"2021-03-24T21:53:37.451872+0000","last_clean":"2021-03-24T21:53:37.451872+0000","last_became_active":"2021-03-24T21:51:51.914615+0000","last_became_peered":"2021-03-24T21:51:51.914615+0000","last_unstale":"2021-03-24T21:53:37.451872+0000","last_undegraded":"2021-03-24T21:53:37.451872+0000","last_fullsized":"2021-03-24T21:53:37.451872+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:24.913238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:24.913238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"43","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.550335+0000","last_change":"2021-03-24T21:53:19.812985+0000","last_active":"2021-03-24T21:53:51.550335+0000","last_peered":"2021-03-24T21:53:51.550335+0000","last_clean":"2021-03-24T21:53:51.550335+0000","last_became_active":"2021-03-24T21:51:49.423159+0000","last_became_peered":"2021-03-24T21:51:49.423159+0000","last_unstale":"2021-03-24T21:53:51.550335+0000","last_undegraded":"2021-03-24T21:53:51.550335+0000","last_fullsized":"2021-03-24T21:53:51.550335+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:19.812822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:19.812822+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":"46","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.743643+0000","last_change":"2021-03-24T21:53:02.100432+0000","last_active":"2021-03-24T21:53:51.743643+0000","last_peered":"2021-03-24T21:53:51.743643+0000","last_clean":"2021-03-24T21:53:51.743643+0000","last_became_active":"2021-03-24T21:51:45.377984+0000","last_became_peered":"2021-03-24T21:51:45.377984+0000","last_unstale":"2021-03-24T21:53:51.743643+0000","last_undegraded":"2021-03-24T21:53:51.743643+0000","last_fullsized":"2021-03-24T21:53:51.743643+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:02.100243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:02.100243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"69'7","reported_seq":"54","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.643229+0000","last_change":"2021-03-24T21:53:13.646018+0000","last_active":"2021-03-24T21:53:37.643229+0000","last_peered":"2021-03-24T21:53:37.643229+0000","last_clean":"2021-03-24T21:53:37.643229+0000","last_became_active":"2021-03-24T21:51:47.687445+0000","last_became_peered":"2021-03-24T21:51:47.687445+0000","last_unstale":"2021-03-24T21:53:37.643229+0000","last_undegraded":"2021-03-24T21:53:37.643229+0000","last_fullsized":"2021-03-24T21:53:37.643229+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'7","last_scrub_stamp":"2021-03-24T21:53:13.645928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:13.645928+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":7,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"41","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.549848+0000","last_change":"2021-03-24T21:53:09.854407+0000","last_active":"2021-03-24T21:53:51.549848+0000","last_peered":"2021-03-24T21:53:51.549848+0000","last_clean":"2021-03-24T21:53:51.549848+0000","last_became_active":"2021-03-24T21:51:51.913733+0000","last_became_peered":"2021-03-24T21:51:51.913733+0000","last_unstale":"2021-03-24T21:53:51.549848+0000","last_undegraded":"2021-03-24T21:53:51.549848+0000","last_fullsized":"2021-03-24T21:53:51.549848+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:09.854304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:09.854304+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"89'4","reported_seq":"49","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.452464+0000","last_change":"2021-03-24T21:53:19.926732+0000","last_active":"2021-03-24T21:53:37.452464+0000","last_peered":"2021-03-24T21:53:37.452464+0000","last_clean":"2021-03-24T21:53:37.452464+0000","last_became_active":"2021-03-24T21:51:45.385110+0000","last_became_peered":"2021-03-24T21:51:45.385110+0000","last_unstale":"2021-03-24T21:53:37.452464+0000","last_undegraded":"2021-03-24T21:53:37.452464+0000","last_fullsized":"2021-03-24T21:53:37.452464+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'4","last_scrub_stamp":"2021-03-24T21:53:19.926637+0000","last_deep_scrub":"89'4","last_deep_scrub_stamp":"2021-03-24T21:53:19.926637+0000","last_clean_scrub_stamp":"2021-03-24T21:53:19.926637+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"69'18","reported_seq":"76","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.549133+0000","last_change":"2021-03-24T21:52:57.861464+0000","last_active":"2021-03-24T21:53:51.549133+0000","last_peered":"2021-03-24T21:53:51.549133+0000","last_clean":"2021-03-24T21:53:51.549133+0000","last_became_active":"2021-03-24T21:51:47.685994+0000","last_became_peered":"2021-03-24T21:51:47.685994+0000","last_unstale":"2021-03-24T21:53:51.549133+0000","last_undegraded":"2021-03-24T21:53:51.549133+0000","last_fullsized":"2021-03-24T21:53:51.549133+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'18","last_scrub_stamp":"2021-03-24T21:52:57.861253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:52:57.861253+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":20,"num_write":19,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"65'2","reported_seq":"67","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.651817+0000","last_change":"2021-03-24T21:53:03.836648+0000","last_active":"2021-03-24T21:53:51.651817+0000","last_peered":"2021-03-24T21:53:51.651817+0000","last_clean":"2021-03-24T21:53:51.651817+0000","last_became_active":"2021-03-24T21:51:49.415167+0000","last_became_peered":"2021-03-24T21:51:49.415167+0000","last_unstale":"2021-03-24T21:53:51.651817+0000","last_undegraded":"2021-03-24T21:53:51.651817+0000","last_fullsized":"2021-03-24T21:53:51.651817+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2021-03-24T21:53:03.836463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:03.836463+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"39","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.642260+0000","last_change":"2021-03-24T21:53:22.725035+0000","last_active":"2021-03-24T21:53:37.642260+0000","last_peered":"2021-03-24T21:53:37.642260+0000","last_clean":"2021-03-24T21:53:37.642260+0000","last_became_active":"2021-03-24T21:51:51.762035+0000","last_became_peered":"2021-03-24T21:51:51.762035+0000","last_unstale":"2021-03-24T21:53:37.642260+0000","last_undegraded":"2021-03-24T21:53:37.642260+0000","last_fullsized":"2021-03-24T21:53:37.642260+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:22.724840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:22.724840+0000","last_clean_scrub_stamp":"2021-03-24T21:53:22.724840+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":"49","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:51.651550+0000","last_change":"2021-03-24T21:53:51.651338+0000","last_active":"2021-03-24T21:53:51.651550+0000","last_peered":"2021-03-24T21:53:51.651550+0000","last_clean":"2021-03-24T21:53:51.651550+0000","last_became_active":"2021-03-24T21:51:45.380984+0000","last_became_peered":"2021-03-24T21:51:45.380984+0000","last_unstale":"2021-03-24T21:53:51.651550+0000","last_undegraded":"2021-03-24T21:53:51.651550+0000","last_fullsized":"2021-03-24T21:53:51.651550+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:51.651164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:51.651164+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"69'15","reported_seq":"73","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.098982+0000","last_change":"2021-03-24T21:53:37.334527+0000","last_active":"2021-03-24T21:53:51.098982+0000","last_peered":"2021-03-24T21:53:51.098982+0000","last_clean":"2021-03-24T21:53:51.098982+0000","last_became_active":"2021-03-24T21:51:47.410650+0000","last_became_peered":"2021-03-24T21:51:47.410650+0000","last_unstale":"2021-03-24T21:53:51.098982+0000","last_undegraded":"2021-03-24T21:53:51.098982+0000","last_fullsized":"2021-03-24T21:53:51.098982+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:53:37.334345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:37.334345+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":19,"num_write":17,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"43","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.242372+0000","last_change":"2021-03-24T21:53:18.557275+0000","last_active":"2021-03-24T21:53:51.242372+0000","last_peered":"2021-03-24T21:53:51.242372+0000","last_clean":"2021-03-24T21:53:51.242372+0000","last_became_active":"2021-03-24T21:51:49.415403+0000","last_became_peered":"2021-03-24T21:51:49.415403+0000","last_unstale":"2021-03-24T21:53:51.242372+0000","last_undegraded":"2021-03-24T21:53:51.242372+0000","last_fullsized":"2021-03-24T21:53:51.242372+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:18.557122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:18.557122+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"38","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.451044+0000","last_change":"2021-03-24T21:53:02.842718+0000","last_active":"2021-03-24T21:53:37.451044+0000","last_peered":"2021-03-24T21:53:37.451044+0000","last_clean":"2021-03-24T21:53:37.451044+0000","last_became_active":"2021-03-24T21:51:51.762513+0000","last_became_peered":"2021-03-24T21:51:51.762513+0000","last_unstale":"2021-03-24T21:53:37.451044+0000","last_undegraded":"2021-03-24T21:53:37.451044+0000","last_fullsized":"2021-03-24T21:53:37.451044+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:02.842537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:02.842537+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"43","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.242696+0000","last_change":"2021-03-24T21:52:59.521026+0000","last_active":"2021-03-24T21:53:51.242696+0000","last_peered":"2021-03-24T21:53:51.242696+0000","last_clean":"2021-03-24T21:53:51.242696+0000","last_became_active":"2021-03-24T21:51:49.417699+0000","last_became_peered":"2021-03-24T21:51:49.417699+0000","last_unstale":"2021-03-24T21:53:51.242696+0000","last_undegraded":"2021-03-24T21:53:51.242696+0000","last_fullsized":"2021-03-24T21:53:51.242696+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:59.520858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:52:59.520858+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":"44","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.452472+0000","last_change":"2021-03-24T21:53:26.908991+0000","last_active":"2021-03-24T21:53:37.452472+0000","last_peered":"2021-03-24T21:53:37.452472+0000","last_clean":"2021-03-24T21:53:37.452472+0000","last_became_active":"2021-03-24T21:51:45.384545+0000","last_became_peered":"2021-03-24T21:51:45.384545+0000","last_unstale":"2021-03-24T21:53:37.452472+0000","last_undegraded":"2021-03-24T21:53:37.452472+0000","last_fullsized":"2021-03-24T21:53:37.452472+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:26.908821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:26.908821+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"69'23","reported_seq":"83","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.744687+0000","last_change":"2021-03-24T21:52:47.126913+0000","last_active":"2021-03-24T21:53:51.744687+0000","last_peered":"2021-03-24T21:53:51.744687+0000","last_clean":"2021-03-24T21:53:51.744687+0000","last_became_active":"2021-03-24T21:51:47.685830+0000","last_became_peered":"2021-03-24T21:51:47.685830+0000","last_unstale":"2021-03-24T21:53:51.744687+0000","last_undegraded":"2021-03-24T21:53:51.744687+0000","last_fullsized":"2021-03-24T21:53:51.744687+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'23","last_scrub_stamp":"2021-03-24T21:52:47.126826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:52:47.126826+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":44,"num_read_kb":25,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"38","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.451627+0000","last_change":"2021-03-24T21:53:35.636404+0000","last_active":"2021-03-24T21:53:37.451627+0000","last_peered":"2021-03-24T21:53:37.451627+0000","last_clean":"2021-03-24T21:53:37.451627+0000","last_became_active":"2021-03-24T21:51:51.763077+0000","last_became_peered":"2021-03-24T21:51:51.763077+0000","last_unstale":"2021-03-24T21:53:37.451627+0000","last_undegraded":"2021-03-24T21:53:37.451627+0000","last_fullsized":"2021-03-24T21:53:37.451627+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:35.636299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:35.636299+0000","last_clean_scrub_stamp":"2021-03-24T21:53:35.636299+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"43","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.242799+0000","last_change":"2021-03-24T21:53:11.623536+0000","last_active":"2021-03-24T21:53:51.242799+0000","last_peered":"2021-03-24T21:53:51.242799+0000","last_clean":"2021-03-24T21:53:51.242799+0000","last_became_active":"2021-03-24T21:51:49.411921+0000","last_became_peered":"2021-03-24T21:51:49.411921+0000","last_unstale":"2021-03-24T21:53:51.242799+0000","last_undegraded":"2021-03-24T21:53:51.242799+0000","last_fullsized":"2021-03-24T21:53:51.242799+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:11.623353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:11.623353+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"47","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.549824+0000","last_change":"2021-03-24T21:52:47.709622+0000","last_active":"2021-03-24T21:53:51.549824+0000","last_peered":"2021-03-24T21:53:51.549824+0000","last_clean":"2021-03-24T21:53:51.549824+0000","last_became_active":"2021-03-24T21:51:45.383017+0000","last_became_peered":"2021-03-24T21:51:45.383017+0000","last_unstale":"2021-03-24T21:53:51.549824+0000","last_undegraded":"2021-03-24T21:53:51.549824+0000","last_fullsized":"2021-03-24T21:53:51.549824+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:47.709437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:52:47.709437+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"69'14","reported_seq":"68","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.453303+0000","last_change":"2021-03-24T21:53:31.653478+0000","last_active":"2021-03-24T21:53:37.453303+0000","last_peered":"2021-03-24T21:53:37.453303+0000","last_clean":"2021-03-24T21:53:37.453303+0000","last_became_active":"2021-03-24T21:51:47.408588+0000","last_became_peered":"2021-03-24T21:51:47.408588+0000","last_unstale":"2021-03-24T21:53:37.453303+0000","last_undegraded":"2021-03-24T21:53:37.453303+0000","last_fullsized":"2021-03-24T21:53:37.453303+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'14","last_scrub_stamp":"2021-03-24T21:53:31.653334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:31.653334+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":16,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"39","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.643619+0000","last_change":"2021-03-24T21:52:59.706737+0000","last_active":"2021-03-24T21:53:37.643619+0000","last_peered":"2021-03-24T21:53:37.643619+0000","last_clean":"2021-03-24T21:53:37.643619+0000","last_became_active":"2021-03-24T21:51:51.759514+0000","last_became_peered":"2021-03-24T21:51:51.759514+0000","last_unstale":"2021-03-24T21:53:37.643619+0000","last_undegraded":"2021-03-24T21:53:37.643619+0000","last_fullsized":"2021-03-24T21:53:37.643619+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:59.706511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:52:59.706511+0000","last_clean_scrub_stamp":"2021-03-24T21:52:59.706511+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"51","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.549448+0000","last_change":"2021-03-24T21:53:06.800587+0000","last_active":"2021-03-24T21:53:51.549448+0000","last_peered":"2021-03-24T21:53:51.549448+0000","last_clean":"2021-03-24T21:53:51.549448+0000","last_became_active":"2021-03-24T21:51:40.350011+0000","last_became_peered":"2021-03-24T21:51:40.350011+0000","last_unstale":"2021-03-24T21:53:51.549448+0000","last_undegraded":"2021-03-24T21:53:51.549448+0000","last_fullsized":"2021-03-24T21:53:51.549448+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:06.800417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:06.800417+0000","last_clean_scrub_stamp":"2021-03-24T21:53:06.800417+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"89'2","reported_seq":"49","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.099230+0000","last_change":"2021-03-24T21:53:28.220091+0000","last_active":"2021-03-24T21:53:51.099230+0000","last_peered":"2021-03-24T21:53:51.099230+0000","last_clean":"2021-03-24T21:53:51.099230+0000","last_became_active":"2021-03-24T21:51:45.375765+0000","last_became_peered":"2021-03-24T21:51:45.375765+0000","last_unstale":"2021-03-24T21:53:51.099230+0000","last_undegraded":"2021-03-24T21:53:51.099230+0000","last_fullsized":"2021-03-24T21:53:51.099230+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2021-03-24T21:53:28.219900+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2021-03-24T21:53:28.219900+0000","last_clean_scrub_stamp":"2021-03-24T21:53:28.219900+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"69'25","reported_seq":"92","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.102258+0000","last_change":"2021-03-24T21:53:15.719556+0000","last_active":"2021-03-24T21:53:51.102258+0000","last_peered":"2021-03-24T21:53:51.102258+0000","last_clean":"2021-03-24T21:53:51.102258+0000","last_became_active":"2021-03-24T21:51:47.411647+0000","last_became_peered":"2021-03-24T21:51:47.411647+0000","last_unstale":"2021-03-24T21:53:51.102258+0000","last_undegraded":"2021-03-24T21:53:51.102258+0000","last_fullsized":"2021-03-24T21:53:51.102258+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'25","last_scrub_stamp":"2021-03-24T21:53:15.719434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:15.719434+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":54,"num_read_kb":31,"num_write":28,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"40","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.452544+0000","last_change":"2021-03-24T21:53:23.578854+0000","last_active":"2021-03-24T21:53:37.452544+0000","last_peered":"2021-03-24T21:53:37.452544+0000","last_clean":"2021-03-24T21:53:37.452544+0000","last_became_active":"2021-03-24T21:51:49.421253+0000","last_became_peered":"2021-03-24T21:51:49.421253+0000","last_unstale":"2021-03-24T21:53:37.452544+0000","last_undegraded":"2021-03-24T21:53:37.452544+0000","last_fullsized":"2021-03-24T21:53:37.452544+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:23.578679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:23.578679+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"38","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.452769+0000","last_change":"2021-03-24T21:53:23.927068+0000","last_active":"2021-03-24T21:53:37.452769+0000","last_peered":"2021-03-24T21:53:37.452769+0000","last_clean":"2021-03-24T21:53:37.452769+0000","last_became_active":"2021-03-24T21:51:51.912075+0000","last_became_peered":"2021-03-24T21:51:51.912075+0000","last_unstale":"2021-03-24T21:53:37.452769+0000","last_undegraded":"2021-03-24T21:53:37.452769+0000","last_fullsized":"2021-03-24T21:53:37.452769+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:23.926884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:23.926884+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"61'1","reported_seq":"50","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.100473+0000","last_change":"2021-03-24T21:53:18.753108+0000","last_active":"2021-03-24T21:53:51.100473+0000","last_peered":"2021-03-24T21:53:51.100473+0000","last_clean":"2021-03-24T21:53:51.100473+0000","last_became_active":"2021-03-24T21:51:45.580337+0000","last_became_peered":"2021-03-24T21:51:45.580337+0000","last_unstale":"2021-03-24T21:53:51.100473+0000","last_undegraded":"2021-03-24T21:53:51.100473+0000","last_fullsized":"2021-03-24T21:53:51.100473+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2021-03-24T21:53:18.753026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:18.753026+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"56","reported_epoch":"94","state":"active+clean","last_fresh":"2021-03-24T21:53:55.497565+0000","last_change":"2021-03-24T21:53:55.497565+0000","last_active":"2021-03-24T21:53:55.497565+0000","last_peered":"2021-03-24T21:53:55.497565+0000","last_clean":"2021-03-24T21:53:55.497565+0000","last_became_active":"2021-03-24T21:51:40.351000+0000","last_became_peered":"2021-03-24T21:51:40.351000+0000","last_unstale":"2021-03-24T21:53:55.497565+0000","last_undegraded":"2021-03-24T21:53:55.497565+0000","last_fullsized":"2021-03-24T21:53:55.497565+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:55.497382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:39.286101+0000","last_clean_scrub_stamp":"2021-03-24T21:53:55.497382+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"89'22","reported_seq":"83","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.103132+0000","last_change":"2021-03-24T21:53:19.717364+0000","last_active":"2021-03-24T21:53:51.103132+0000","last_peered":"2021-03-24T21:53:51.103132+0000","last_clean":"2021-03-24T21:53:51.103132+0000","last_became_active":"2021-03-24T21:51:47.411696+0000","last_became_peered":"2021-03-24T21:51:47.411696+0000","last_unstale":"2021-03-24T21:53:51.103132+0000","last_undegraded":"2021-03-24T21:53:51.103132+0000","last_fullsized":"2021-03-24T21:53:51.103132+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'22","last_scrub_stamp":"2021-03-24T21:53:19.717244+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2021-03-24T21:53:19.717244+0000","last_clean_scrub_stamp":"2021-03-24T21:53:19.717244+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":36,"num_read_kb":21,"num_write":18,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"43","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.243292+0000","last_change":"2021-03-24T21:53:21.534119+0000","last_active":"2021-03-24T21:53:51.243292+0000","last_peered":"2021-03-24T21:53:51.243292+0000","last_clean":"2021-03-24T21:53:51.243292+0000","last_became_active":"2021-03-24T21:51:49.418848+0000","last_became_peered":"2021-03-24T21:51:49.418848+0000","last_unstale":"2021-03-24T21:53:51.243292+0000","last_undegraded":"2021-03-24T21:53:51.243292+0000","last_fullsized":"2021-03-24T21:53:51.243292+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:21.533882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:21.533882+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"41","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.550400+0000","last_change":"2021-03-24T21:53:01.832920+0000","last_active":"2021-03-24T21:53:51.550400+0000","last_peered":"2021-03-24T21:53:51.550400+0000","last_clean":"2021-03-24T21:53:51.550400+0000","last_became_active":"2021-03-24T21:51:51.765236+0000","last_became_peered":"2021-03-24T21:51:51.765236+0000","last_unstale":"2021-03-24T21:53:51.550400+0000","last_undegraded":"2021-03-24T21:53:51.550400+0000","last_fullsized":"2021-03-24T21:53:51.550400+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:01.832740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:01.832740+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"54","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.451159+0000","last_change":"2021-03-24T21:53:30.906666+0000","last_active":"2021-03-24T21:53:37.451159+0000","last_peered":"2021-03-24T21:53:37.451159+0000","last_clean":"2021-03-24T21:53:37.451159+0000","last_became_active":"2021-03-24T21:51:40.349967+0000","last_became_peered":"2021-03-24T21:51:40.349967+0000","last_unstale":"2021-03-24T21:53:37.451159+0000","last_undegraded":"2021-03-24T21:53:37.451159+0000","last_fullsized":"2021-03-24T21:53:37.451159+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:30.906502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:49:51.529051+0000","last_clean_scrub_stamp":"2021-03-24T21:53:30.906502+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"45","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.643186+0000","last_change":"2021-03-24T21:53:20.736644+0000","last_active":"2021-03-24T21:53:37.643186+0000","last_peered":"2021-03-24T21:53:37.643186+0000","last_clean":"2021-03-24T21:53:37.643186+0000","last_became_active":"2021-03-24T21:51:45.383779+0000","last_became_peered":"2021-03-24T21:51:45.383779+0000","last_unstale":"2021-03-24T21:53:37.643186+0000","last_undegraded":"2021-03-24T21:53:37.643186+0000","last_fullsized":"2021-03-24T21:53:37.643186+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:20.736432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:20.736432+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"69'15","reported_seq":"68","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.745180+0000","last_change":"2021-03-24T21:53:20.279112+0000","last_active":"2021-03-24T21:53:51.745180+0000","last_peered":"2021-03-24T21:53:51.745180+0000","last_clean":"2021-03-24T21:53:51.745180+0000","last_became_active":"2021-03-24T21:51:47.688873+0000","last_became_peered":"2021-03-24T21:51:47.688873+0000","last_unstale":"2021-03-24T21:53:51.745180+0000","last_undegraded":"2021-03-24T21:53:51.745180+0000","last_fullsized":"2021-03-24T21:53:51.745180+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:53:20.279004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:20.279004+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":27,"num_read_kb":15,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"40","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.452352+0000","last_change":"2021-03-24T21:53:09.874798+0000","last_active":"2021-03-24T21:53:37.452352+0000","last_peered":"2021-03-24T21:53:37.452352+0000","last_clean":"2021-03-24T21:53:37.452352+0000","last_became_active":"2021-03-24T21:51:49.413251+0000","last_became_peered":"2021-03-24T21:51:49.413251+0000","last_unstale":"2021-03-24T21:53:37.452352+0000","last_undegraded":"2021-03-24T21:53:37.452352+0000","last_fullsized":"2021-03-24T21:53:37.452352+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:09.874681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:09.874681+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"41","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.102098+0000","last_change":"2021-03-24T21:53:11.723900+0000","last_active":"2021-03-24T21:53:51.102098+0000","last_peered":"2021-03-24T21:53:51.102098+0000","last_clean":"2021-03-24T21:53:51.102098+0000","last_became_active":"2021-03-24T21:51:51.762039+0000","last_became_peered":"2021-03-24T21:51:51.762039+0000","last_unstale":"2021-03-24T21:53:51.102098+0000","last_undegraded":"2021-03-24T21:53:51.102098+0000","last_fullsized":"2021-03-24T21:53:51.102098+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:11.723709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:11.723709+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"44","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.454059+0000","last_change":"2021-03-24T21:52:48.487965+0000","last_active":"2021-03-24T21:53:37.454059+0000","last_peered":"2021-03-24T21:53:37.454059+0000","last_clean":"2021-03-24T21:53:37.454059+0000","last_became_active":"2021-03-24T21:51:45.376687+0000","last_became_peered":"2021-03-24T21:51:45.376687+0000","last_unstale":"2021-03-24T21:53:37.454059+0000","last_undegraded":"2021-03-24T21:53:37.454059+0000","last_fullsized":"2021-03-24T21:53:37.454059+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:48.487787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:52:48.487787+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"66'5","reported_seq":"179","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:52.339336+0000","last_change":"2021-03-24T21:52:50.807770+0000","last_active":"2021-03-24T21:53:52.339336+0000","last_peered":"2021-03-24T21:53:52.339336+0000","last_clean":"2021-03-24T21:53:52.339336+0000","last_became_active":"2021-03-24T21:51:40.350525+0000","last_became_peered":"2021-03-24T21:51:40.350525+0000","last_unstale":"2021-03-24T21:53:52.339336+0000","last_undegraded":"2021-03-24T21:53:52.339336+0000","last_fullsized":"2021-03-24T21:53:52.339336+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2021-03-24T21:52:50.807568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:39.286101+0000","last_clean_scrub_stamp":"2021-03-24T21:52:50.807568+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":128,"num_read_kb":123,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"69'16","reported_seq":"68","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.454741+0000","last_change":"2021-03-24T21:53:15.485180+0000","last_active":"2021-03-24T21:53:37.454741+0000","last_peered":"2021-03-24T21:53:37.454741+0000","last_clean":"2021-03-24T21:53:37.454741+0000","last_became_active":"2021-03-24T21:51:47.411442+0000","last_became_peered":"2021-03-24T21:51:47.411442+0000","last_unstale":"2021-03-24T21:53:37.454741+0000","last_undegraded":"2021-03-24T21:53:37.454741+0000","last_fullsized":"2021-03-24T21:53:37.454741+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T21:53:15.485072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:15.485072+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":16,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"43","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.550993+0000","last_change":"2021-03-24T21:53:20.836086+0000","last_active":"2021-03-24T21:53:51.550993+0000","last_peered":"2021-03-24T21:53:51.550993+0000","last_clean":"2021-03-24T21:53:51.550993+0000","last_became_active":"2021-03-24T21:51:49.419167+0000","last_became_peered":"2021-03-24T21:51:49.419167+0000","last_unstale":"2021-03-24T21:53:51.550993+0000","last_undegraded":"2021-03-24T21:53:51.550993+0000","last_fullsized":"2021-03-24T21:53:51.550993+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:20.835990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:20.835990+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"41","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.101704+0000","last_change":"2021-03-24T21:53:28.691448+0000","last_active":"2021-03-24T21:53:51.101704+0000","last_peered":"2021-03-24T21:53:51.101704+0000","last_clean":"2021-03-24T21:53:51.101704+0000","last_became_active":"2021-03-24T21:51:51.913581+0000","last_became_peered":"2021-03-24T21:51:51.913581+0000","last_unstale":"2021-03-24T21:53:51.101704+0000","last_undegraded":"2021-03-24T21:53:51.101704+0000","last_fullsized":"2021-03-24T21:53:51.101704+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:28.691270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:28.691270+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"47","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.102244+0000","last_change":"2021-03-24T21:53:08.691716+0000","last_active":"2021-03-24T21:53:51.102244+0000","last_peered":"2021-03-24T21:53:51.102244+0000","last_clean":"2021-03-24T21:53:51.102244+0000","last_became_active":"2021-03-24T21:51:45.580531+0000","last_became_peered":"2021-03-24T21:51:45.580531+0000","last_unstale":"2021-03-24T21:53:51.102244+0000","last_undegraded":"2021-03-24T21:53:51.102244+0000","last_fullsized":"2021-03-24T21:53:51.102244+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:08.691536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:08.691536+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"69'21","reported_seq":"82","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.453915+0000","last_change":"2021-03-24T21:53:05.313409+0000","last_active":"2021-03-24T21:53:37.453915+0000","last_peered":"2021-03-24T21:53:37.453915+0000","last_clean":"2021-03-24T21:53:37.453915+0000","last_became_active":"2021-03-24T21:51:47.686486+0000","last_became_peered":"2021-03-24T21:51:47.686486+0000","last_unstale":"2021-03-24T21:53:37.453915+0000","last_undegraded":"2021-03-24T21:53:37.453915+0000","last_fullsized":"2021-03-24T21:53:37.453915+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'21","last_scrub_stamp":"2021-03-24T21:53:05.313235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:05.313235+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":45,"num_read_kb":27,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"42","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.744790+0000","last_change":"2021-03-24T21:53:23.246488+0000","last_active":"2021-03-24T21:53:51.744790+0000","last_peered":"2021-03-24T21:53:51.744790+0000","last_clean":"2021-03-24T21:53:51.744790+0000","last_became_active":"2021-03-24T21:51:49.416847+0000","last_became_peered":"2021-03-24T21:51:49.416847+0000","last_unstale":"2021-03-24T21:53:51.744790+0000","last_undegraded":"2021-03-24T21:53:51.744790+0000","last_fullsized":"2021-03-24T21:53:51.744790+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:23.246320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:23.246320+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"38","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.451252+0000","last_change":"2021-03-24T21:53:25.593543+0000","last_active":"2021-03-24T21:53:37.451252+0000","last_peered":"2021-03-24T21:53:37.451252+0000","last_clean":"2021-03-24T21:53:37.451252+0000","last_became_active":"2021-03-24T21:51:51.759627+0000","last_became_peered":"2021-03-24T21:51:51.759627+0000","last_unstale":"2021-03-24T21:53:37.451252+0000","last_undegraded":"2021-03-24T21:53:37.451252+0000","last_fullsized":"2021-03-24T21:53:37.451252+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:25.593358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:25.593358+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"47","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.550554+0000","last_change":"2021-03-24T21:53:04.837752+0000","last_active":"2021-03-24T21:53:51.550554+0000","last_peered":"2021-03-24T21:53:51.550554+0000","last_clean":"2021-03-24T21:53:51.550554+0000","last_became_active":"2021-03-24T21:51:45.583885+0000","last_became_peered":"2021-03-24T21:51:45.583885+0000","last_unstale":"2021-03-24T21:53:51.550554+0000","last_undegraded":"2021-03-24T21:53:51.550554+0000","last_fullsized":"2021-03-24T21:53:51.550554+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:04.837488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:04.837488+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"69'16","reported_seq":"72","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.101470+0000","last_change":"2021-03-24T21:53:03.595733+0000","last_active":"2021-03-24T21:53:51.101470+0000","last_peered":"2021-03-24T21:53:51.101470+0000","last_clean":"2021-03-24T21:53:51.101470+0000","last_became_active":"2021-03-24T21:51:47.403006+0000","last_became_peered":"2021-03-24T21:51:47.403006+0000","last_unstale":"2021-03-24T21:53:51.101470+0000","last_undegraded":"2021-03-24T21:53:51.101470+0000","last_fullsized":"2021-03-24T21:53:51.101470+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T21:53:03.595585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:03.595585+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"89'3","reported_seq":"68","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.098401+0000","last_change":"2021-03-24T21:53:26.584866+0000","last_active":"2021-03-24T21:53:51.098401+0000","last_peered":"2021-03-24T21:53:51.098401+0000","last_clean":"2021-03-24T21:53:51.098401+0000","last_became_active":"2021-03-24T21:51:49.420173+0000","last_became_peered":"2021-03-24T21:51:49.420173+0000","last_unstale":"2021-03-24T21:53:51.098401+0000","last_undegraded":"2021-03-24T21:53:51.098401+0000","last_fullsized":"2021-03-24T21:53:51.098401+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'3","last_scrub_stamp":"2021-03-24T21:53:26.584678+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-24T21:53:26.584678+0000","last_clean_scrub_stamp":"2021-03-24T21:53:26.584678+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"38","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.452493+0000","last_change":"2021-03-24T21:52:55.828993+0000","last_active":"2021-03-24T21:53:37.452493+0000","last_peered":"2021-03-24T21:53:37.452493+0000","last_clean":"2021-03-24T21:53:37.452493+0000","last_became_active":"2021-03-24T21:51:51.914601+0000","last_became_peered":"2021-03-24T21:51:51.914601+0000","last_unstale":"2021-03-24T21:53:37.452493+0000","last_undegraded":"2021-03-24T21:53:37.452493+0000","last_fullsized":"2021-03-24T21:53:37.452493+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:55.828810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:52:55.828810+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"44","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.454274+0000","last_change":"2021-03-24T21:53:24.621910+0000","last_active":"2021-03-24T21:53:37.454274+0000","last_peered":"2021-03-24T21:53:37.454274+0000","last_clean":"2021-03-24T21:53:37.454274+0000","last_became_active":"2021-03-24T21:51:45.375747+0000","last_became_peered":"2021-03-24T21:51:45.375747+0000","last_unstale":"2021-03-24T21:53:37.454274+0000","last_undegraded":"2021-03-24T21:53:37.454274+0000","last_fullsized":"2021-03-24T21:53:37.454274+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:24.621749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:24.621749+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"69'20","reported_seq":"77","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.098841+0000","last_change":"2021-03-24T21:53:13.077033+0000","last_active":"2021-03-24T21:53:51.098841+0000","last_peered":"2021-03-24T21:53:51.098841+0000","last_clean":"2021-03-24T21:53:51.098841+0000","last_became_active":"2021-03-24T21:51:47.404713+0000","last_became_peered":"2021-03-24T21:51:47.404713+0000","last_unstale":"2021-03-24T21:53:51.098841+0000","last_undegraded":"2021-03-24T21:53:51.098841+0000","last_fullsized":"2021-03-24T21:53:51.098841+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'20","last_scrub_stamp":"2021-03-24T21:53:13.076772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:13.076772+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":39,"num_read_kb":20,"num_write":20,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"65'2","reported_seq":"68","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.744468+0000","last_change":"2021-03-24T21:53:04.714525+0000","last_active":"2021-03-24T21:53:51.744468+0000","last_peered":"2021-03-24T21:53:51.744468+0000","last_clean":"2021-03-24T21:53:51.744468+0000","last_became_active":"2021-03-24T21:51:49.421628+0000","last_became_peered":"2021-03-24T21:51:49.421628+0000","last_unstale":"2021-03-24T21:53:51.744468+0000","last_undegraded":"2021-03-24T21:53:51.744468+0000","last_fullsized":"2021-03-24T21:53:51.744468+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2021-03-24T21:53:04.714348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:04.714348+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"40","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.099337+0000","last_change":"2021-03-24T21:53:45.349966+0000","last_active":"2021-03-24T21:53:51.099337+0000","last_peered":"2021-03-24T21:53:51.099337+0000","last_clean":"2021-03-24T21:53:51.099337+0000","last_became_active":"2021-03-24T21:51:51.756114+0000","last_became_peered":"2021-03-24T21:51:51.756114+0000","last_unstale":"2021-03-24T21:53:51.099337+0000","last_undegraded":"2021-03-24T21:53:51.099337+0000","last_fullsized":"2021-03-24T21:53:51.099337+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:45.349781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:45.349781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"45","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.643007+0000","last_change":"2021-03-24T21:53:00.682889+0000","last_active":"2021-03-24T21:53:37.643007+0000","last_peered":"2021-03-24T21:53:37.643007+0000","last_clean":"2021-03-24T21:53:37.643007+0000","last_became_active":"2021-03-24T21:51:45.378704+0000","last_became_peered":"2021-03-24T21:51:45.378704+0000","last_unstale":"2021-03-24T21:53:37.643007+0000","last_undegraded":"2021-03-24T21:53:37.643007+0000","last_fullsized":"2021-03-24T21:53:37.643007+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:00.682713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:00.682713+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"69'22","reported_seq":"83","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.641789+0000","last_change":"2021-03-24T21:53:17.716726+0000","last_active":"2021-03-24T21:53:37.641789+0000","last_peered":"2021-03-24T21:53:37.641789+0000","last_clean":"2021-03-24T21:53:37.641789+0000","last_became_active":"2021-03-24T21:51:47.691459+0000","last_became_peered":"2021-03-24T21:51:47.691459+0000","last_unstale":"2021-03-24T21:53:37.641789+0000","last_undegraded":"2021-03-24T21:53:37.641789+0000","last_fullsized":"2021-03-24T21:53:37.641789+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'22","last_scrub_stamp":"2021-03-24T21:53:17.716470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:17.716470+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":46,"num_read_kb":26,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"42","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.098326+0000","last_change":"2021-03-24T21:53:25.220598+0000","last_active":"2021-03-24T21:53:51.098326+0000","last_peered":"2021-03-24T21:53:51.098326+0000","last_clean":"2021-03-24T21:53:51.098326+0000","last_became_active":"2021-03-24T21:51:49.408761+0000","last_became_peered":"2021-03-24T21:51:49.408761+0000","last_unstale":"2021-03-24T21:53:51.098326+0000","last_undegraded":"2021-03-24T21:53:51.098326+0000","last_fullsized":"2021-03-24T21:53:51.098326+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:25.220436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:25.220436+0000","last_clean_scrub_stamp":"2021-03-24T21:53:25.220436+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"41","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.550050+0000","last_change":"2021-03-24T21:53:11.925302+0000","last_active":"2021-03-24T21:53:51.550050+0000","last_peered":"2021-03-24T21:53:51.550050+0000","last_clean":"2021-03-24T21:53:51.550050+0000","last_became_active":"2021-03-24T21:51:51.760369+0000","last_became_peered":"2021-03-24T21:51:51.760369+0000","last_unstale":"2021-03-24T21:53:51.550050+0000","last_undegraded":"2021-03-24T21:53:51.550050+0000","last_fullsized":"2021-03-24T21:53:51.550050+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:11.925149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:11.925149+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"65'2","reported_seq":"69","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:53.817549+0000","last_change":"2021-03-24T21:53:29.268452+0000","last_active":"2021-03-24T21:53:53.817549+0000","last_peered":"2021-03-24T21:53:53.817549+0000","last_clean":"2021-03-24T21:53:53.817549+0000","last_became_active":"2021-03-24T21:51:49.414380+0000","last_became_peered":"2021-03-24T21:51:49.414380+0000","last_unstale":"2021-03-24T21:53:53.817549+0000","last_undegraded":"2021-03-24T21:53:53.817549+0000","last_fullsized":"2021-03-24T21:53:53.817549+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2021-03-24T21:53:29.268273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:29.268273+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":"45","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.641726+0000","last_change":"2021-03-24T21:53:35.609332+0000","last_active":"2021-03-24T21:53:37.641726+0000","last_peered":"2021-03-24T21:53:37.641726+0000","last_clean":"2021-03-24T21:53:37.641726+0000","last_became_active":"2021-03-24T21:51:45.378863+0000","last_became_peered":"2021-03-24T21:51:45.378863+0000","last_unstale":"2021-03-24T21:53:37.641726+0000","last_undegraded":"2021-03-24T21:53:37.641726+0000","last_fullsized":"2021-03-24T21:53:37.641726+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:35.609173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:35.609173+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"69'17","reported_seq":"74","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.103242+0000","last_change":"2021-03-24T21:52:54.714245+0000","last_active":"2021-03-24T21:53:51.103242+0000","last_peered":"2021-03-24T21:53:51.103242+0000","last_clean":"2021-03-24T21:53:51.103242+0000","last_became_active":"2021-03-24T21:51:47.411126+0000","last_became_peered":"2021-03-24T21:51:47.411126+0000","last_unstale":"2021-03-24T21:53:51.103242+0000","last_undegraded":"2021-03-24T21:53:51.103242+0000","last_fullsized":"2021-03-24T21:53:51.103242+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'17","last_scrub_stamp":"2021-03-24T21:52:54.714050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:52:54.714050+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":19,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"39","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.642233+0000","last_change":"2021-03-24T21:53:27.701155+0000","last_active":"2021-03-24T21:53:37.642233+0000","last_peered":"2021-03-24T21:53:37.642233+0000","last_clean":"2021-03-24T21:53:37.642233+0000","last_became_active":"2021-03-24T21:51:51.912009+0000","last_became_peered":"2021-03-24T21:51:51.912009+0000","last_unstale":"2021-03-24T21:53:37.642233+0000","last_undegraded":"2021-03-24T21:53:37.642233+0000","last_fullsized":"2021-03-24T21:53:37.642233+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:27.701002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:27.701002+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"45","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.550888+0000","last_change":"2021-03-24T21:52:55.846936+0000","last_active":"2021-03-24T21:53:51.550888+0000","last_peered":"2021-03-24T21:53:51.550888+0000","last_clean":"2021-03-24T21:53:51.550888+0000","last_became_active":"2021-03-24T21:51:49.422787+0000","last_became_peered":"2021-03-24T21:51:49.422787+0000","last_unstale":"2021-03-24T21:53:51.550888+0000","last_undegraded":"2021-03-24T21:53:51.550888+0000","last_fullsized":"2021-03-24T21:53:51.550888+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:55.846764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:52:55.846764+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"46","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.098521+0000","last_change":"2021-03-24T21:52:48.334326+0000","last_active":"2021-03-24T21:53:51.098521+0000","last_peered":"2021-03-24T21:53:51.098521+0000","last_clean":"2021-03-24T21:53:51.098521+0000","last_became_active":"2021-03-24T21:51:45.579513+0000","last_became_peered":"2021-03-24T21:51:45.579513+0000","last_unstale":"2021-03-24T21:53:51.098521+0000","last_undegraded":"2021-03-24T21:53:51.098521+0000","last_fullsized":"2021-03-24T21:53:51.098521+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:48.334138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:52:48.334138+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"69'15","reported_seq":"71","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.100209+0000","last_change":"2021-03-24T21:52:55.233331+0000","last_active":"2021-03-24T21:53:51.100209+0000","last_peered":"2021-03-24T21:53:51.100209+0000","last_clean":"2021-03-24T21:53:51.100209+0000","last_became_active":"2021-03-24T21:51:47.410414+0000","last_became_peered":"2021-03-24T21:51:47.410414+0000","last_unstale":"2021-03-24T21:53:51.100209+0000","last_undegraded":"2021-03-24T21:53:51.100209+0000","last_fullsized":"2021-03-24T21:53:51.100209+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:52:55.233213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:52:55.233213+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"41","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.243312+0000","last_change":"2021-03-24T21:53:14.576861+0000","last_active":"2021-03-24T21:53:51.243312+0000","last_peered":"2021-03-24T21:53:51.243312+0000","last_clean":"2021-03-24T21:53:51.243312+0000","last_became_active":"2021-03-24T21:51:51.765067+0000","last_became_peered":"2021-03-24T21:51:51.765067+0000","last_unstale":"2021-03-24T21:53:51.243312+0000","last_undegraded":"2021-03-24T21:53:51.243312+0000","last_fullsized":"2021-03-24T21:53:51.243312+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:14.576687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:14.576687+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"43","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.102421+0000","last_change":"2021-03-24T21:53:29.735847+0000","last_active":"2021-03-24T21:53:51.102421+0000","last_peered":"2021-03-24T21:53:51.102421+0000","last_clean":"2021-03-24T21:53:51.102421+0000","last_became_active":"2021-03-24T21:51:49.407001+0000","last_became_peered":"2021-03-24T21:51:49.407001+0000","last_unstale":"2021-03-24T21:53:51.102421+0000","last_undegraded":"2021-03-24T21:53:51.102421+0000","last_fullsized":"2021-03-24T21:53:51.102421+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:29.735672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:29.735672+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"46","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.744368+0000","last_change":"2021-03-24T21:53:22.204032+0000","last_active":"2021-03-24T21:53:51.744368+0000","last_peered":"2021-03-24T21:53:51.744368+0000","last_clean":"2021-03-24T21:53:51.744368+0000","last_became_active":"2021-03-24T21:51:45.374203+0000","last_became_peered":"2021-03-24T21:51:45.374203+0000","last_unstale":"2021-03-24T21:53:51.744368+0000","last_undegraded":"2021-03-24T21:53:51.744368+0000","last_fullsized":"2021-03-24T21:53:51.744368+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:22.203858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:22.203858+0000","last_clean_scrub_stamp":"2021-03-24T21:53:22.203858+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"69'21","reported_seq":"80","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.100054+0000","last_change":"2021-03-24T21:52:50.302291+0000","last_active":"2021-03-24T21:53:51.100054+0000","last_peered":"2021-03-24T21:53:51.100054+0000","last_clean":"2021-03-24T21:53:51.100054+0000","last_became_active":"2021-03-24T21:51:47.416296+0000","last_became_peered":"2021-03-24T21:51:47.416296+0000","last_unstale":"2021-03-24T21:53:51.100054+0000","last_undegraded":"2021-03-24T21:53:51.100054+0000","last_fullsized":"2021-03-24T21:53:51.100054+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'21","last_scrub_stamp":"2021-03-24T21:52:50.302085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:52:50.302085+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":23,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"40","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.100495+0000","last_change":"2021-03-24T21:53:40.290157+0000","last_active":"2021-03-24T21:53:51.100495+0000","last_peered":"2021-03-24T21:53:51.100495+0000","last_clean":"2021-03-24T21:53:51.100495+0000","last_became_active":"2021-03-24T21:51:51.761142+0000","last_became_peered":"2021-03-24T21:51:51.761142+0000","last_unstale":"2021-03-24T21:53:51.100495+0000","last_undegraded":"2021-03-24T21:53:51.100495+0000","last_fullsized":"2021-03-24T21:53:51.100495+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:40.289984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:40.289984+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"65'2","reported_seq":"70","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:53.817452+0000","last_change":"2021-03-24T21:53:04.625006+0000","last_active":"2021-03-24T21:53:53.817452+0000","last_peered":"2021-03-24T21:53:53.817452+0000","last_clean":"2021-03-24T21:53:53.817452+0000","last_became_active":"2021-03-24T21:51:49.416182+0000","last_became_peered":"2021-03-24T21:51:49.416182+0000","last_unstale":"2021-03-24T21:53:53.817452+0000","last_undegraded":"2021-03-24T21:53:53.817452+0000","last_fullsized":"2021-03-24T21:53:53.817452+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2021-03-24T21:53:04.624892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:04.624892+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"45","reported_epoch":"92","state":"active+clean","last_fresh":"2021-03-24T21:53:37.642691+0000","last_change":"2021-03-24T21:52:46.620502+0000","last_active":"2021-03-24T21:53:37.642691+0000","last_peered":"2021-03-24T21:53:37.642691+0000","last_clean":"2021-03-24T21:53:37.642691+0000","last_became_active":"2021-03-24T21:51:45.375830+0000","last_became_peered":"2021-03-24T21:51:45.375830+0000","last_unstale":"2021-03-24T21:53:37.642691+0000","last_undegraded":"2021-03-24T21:53:37.642691+0000","last_fullsized":"2021-03-24T21:53:37.642691+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:46.620317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:52:46.620317+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"69'13","reported_seq":"65","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.099436+0000","last_change":"2021-03-24T21:53:20.163291+0000","last_active":"2021-03-24T21:53:51.099436+0000","last_peered":"2021-03-24T21:53:51.099436+0000","last_clean":"2021-03-24T21:53:51.099436+0000","last_became_active":"2021-03-24T21:51:47.407739+0000","last_became_peered":"2021-03-24T21:51:47.407739+0000","last_unstale":"2021-03-24T21:53:51.099436+0000","last_undegraded":"2021-03-24T21:53:51.099436+0000","last_fullsized":"2021-03-24T21:53:51.099436+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2021-03-24T21:53:20.163065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:20.163065+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":13,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19,"ondisk_log_size":19,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1069,"num_read_kb":574,"num_write":542,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":2412,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966080,"data_stored":70572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":539,"ondisk_log_size":539,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":128,"num_read_kb":123,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201310,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6476,"kb_used_data":836,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738676,"statfs":{"total":95995035648,"available":95988404224,"internally_reserved":0,"allocated":856064,"data_stored":320421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:53:44 2021","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.90900000000000003,"15min":0.90200000000000002},"min":{"1min":0.68100000000000005,"5min":0.439,"15min":0.439},"max":{"1min":2.2240000000000002,"5min":2.2240000000000002,"15min":2.2240000000000002},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.97199999999999998,"15min":0.97299999999999998},"min":{"1min":0.71299999999999997,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":2.1859999999999999,"5min":2.1859999999999999,"15min":2.1859999999999999},"last":0.74299999999999999}]},{"osd":1,"last update":"Wed Mar 24 21:53:44 2021","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.85199999999999998,"15min":0.83799999999999997},"min":{"1min":0.62,"5min":0.5,"15min":0.5},"max":{"1min":1.7090000000000001,"5min":1.7090000000000001,"15min":1.7090000000000001},"last":1.099},{"interface":"front","average":{"1min":1.0289999999999999,"5min":0.97999999999999998,"15min":0.97199999999999998},"min":{"1min":0.68200000000000005,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.5600000000000001,"5min":2.343,"15min":2.343},"last":1.083}]},{"osd":2,"last update":"Wed Mar 24 21:53:44 2021","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.022,"15min":1.0029999999999999},"min":{"1min":0.69599999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":4.944,"5min":4.944,"15min":4.944},"last":1.0469999999999999},{"interface":"front","average":{"1min":1.01,"5min":1.0109999999999999,"15min":1.0109999999999999},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.357,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":0.71699999999999997}]},{"osd":3,"last update":"Wed Mar 24 21:53:44 2021","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.042,"15min":1.044},"min":{"1min":0.70299999999999996,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.538,"5min":1.9490000000000001,"15min":1.9490000000000001},"last":0.88200000000000001},{"interface":"front","average":{"1min":1.1100000000000001,"5min":0.998,"15min":0.97899999999999998},"min":{"1min":0.77100000000000002,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.4330000000000001,"5min":1.845,"15min":1.845},"last":1.008}]},{"osd":4,"last update":"Wed Mar 24 21:53:44 2021","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":0.94099999999999995,"15min":0.92800000000000005},"min":{"1min":0.71799999999999997,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.3460000000000001,"5min":2.2000000000000002,"15min":2.2000000000000002},"last":0.96799999999999997},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.0780000000000001,"15min":1.075},"min":{"1min":0.91900000000000004,"5min":0.504,"15min":0.504},"max":{"1min":1.381,"5min":2.1299999999999999,"15min":2.1299999999999999},"last":1.121}]},{"osd":5,"last update":"Wed Mar 24 21:53:44 2021","interfaces":[{"interface":"back","average":{"1min":1.1299999999999999,"5min":0.997,"15min":0.97499999999999998},"min":{"1min":0.77000000000000002,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.516,"5min":1.581,"15min":1.581},"last":1.212},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.1519999999999999,"15min":1.141},"min":{"1min":0.92500000000000004,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.395,"5min":2.4630000000000001,"15min":2.4630000000000001},"last":1.1359999999999999}]},{"osd":6,"last update":"Wed Mar 24 21:53:44 2021","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.1379999999999999,"15min":1.117},"min":{"1min":0.99199999999999999,"5min":0.62,"15min":0.62},"max":{"1min":1.4590000000000001,"5min":2.5179999999999998,"15min":2.5179999999999998},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.238,"5min":1.1839999999999999,"15min":1.175},"min":{"1min":1.0529999999999999,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.4730000000000001,"5min":2.3069999999999999,"15min":2.3069999999999999},"last":1.1659999999999999}]}]},{"osd":6,"up_from":47,"seq":201863462947,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6536,"kb_used_data":896,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738616,"statfs":{"total":95995035648,"available":95988342784,"internally_reserved":0,"allocated":917504,"data_stored":329856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:53:21 2021","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.91000000000000003,"15min":0.91200000000000003},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":1.1180000000000001},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.91200000000000003,"15min":0.91700000000000004},"min":{"1min":0.56799999999999995,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.75600000000000001}]},{"osd":1,"last update":"Wed Mar 24 21:53:21 2021","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.92100000000000004,"15min":0.93200000000000005},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.2470000000000001,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.91100000000000003,"15min":0.91600000000000004},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.2869999999999999,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":1.073}]},{"osd":2,"last update":"Wed Mar 24 21:53:21 2021","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.94899999999999995,"15min":0.95099999999999996},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.96699999999999997,"15min":0.97499999999999998},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.2170000000000001,"5min":1.401,"15min":1.401},"last":1.0329999999999999}]},{"osd":3,"last update":"Wed Mar 24 21:53:21 2021","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.95299999999999996,"15min":0.95699999999999996},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.3089999999999999,"5min":1.4179999999999999,"15min":1.4179999999999999},"last":0.84699999999999998},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.022,"15min":1.0149999999999999},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.5009999999999999,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":0.999}]},{"osd":4,"last update":"Wed Mar 24 21:53:21 2021","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.92000000000000004,"15min":0.92100000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.26,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.97699999999999998,"5min":0.95099999999999996,"15min":0.94699999999999995},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.3149999999999999,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.95599999999999996}]},{"osd":5,"last update":"Wed Mar 24 21:53:21 2021","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":1.0369999999999999,"15min":1.0449999999999999},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.337,"5min":1.4830000000000001,"15min":1.4830000000000001},"last":0.83299999999999996},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.04,"15min":1.038},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.4570000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":1.0580000000000001}]},{"osd":7,"last update":"Wed Mar 24 21:53:41 2021","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":1.1359999999999999,"15min":1.1279999999999999},"min":{"1min":0.83799999999999997,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.3560000000000001,"5min":1.5249999999999999,"15min":1.5249999999999999},"last":1.0940000000000001},{"interface":"front","average":{"1min":1.234,"5min":1.1299999999999999,"15min":1.113},"min":{"1min":0.79900000000000004,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.478,"5min":1.478,"15min":1.478},"last":1.1619999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574904,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7128,"kb_used_data":976,"kb_used_omap":0,"kb_used_meta":6144,"kb_avail":93738024,"statfs":{"total":95995035648,"available":95987736576,"internally_reserved":0,"allocated":999424,"data_stored":332668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6291456},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:53:33 2021","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.629,"15min":0.58699999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.68500000000000005,"15min":0.64100000000000001},"min":{"1min":0.41399999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.3009999999999999,"5min":1.641,"15min":1.641},"last":0.89800000000000002}]},{"osd":2,"last update":"Wed Mar 24 21:53:54 2021","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.65800000000000003,"15min":0.63100000000000001},"min":{"1min":0.40300000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.78300000000000003,"15min":0.751},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.046,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":0.75800000000000001}]},{"osd":3,"last update":"Wed Mar 24 21:53:18 2021","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":1.296,"15min":1.4730000000000001},"min":{"1min":0.63100000000000001,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.2849999999999999,"5min":17.032,"15min":17.032},"last":0.996},{"interface":"front","average":{"1min":0.90900000000000003,"5min":1.2290000000000001,"15min":1.389},"min":{"1min":0.621,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.135,"5min":16.908999999999999,"15min":16.908999999999999},"last":0.84999999999999998}]},{"osd":4,"last update":"Wed Mar 24 21:53:41 2021","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.96399999999999997,"15min":0.96399999999999997},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.4510000000000001,"5min":1.617,"15min":1.617},"last":1.274},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.093,"15min":1.095},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.706,"5min":1.7150000000000001,"15min":1.7150000000000001},"last":0.94999999999999996}]},{"osd":5,"last update":"Wed Mar 24 21:52:55 2021","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":0.999,"15min":0.999},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.3220000000000001,"5min":1.5489999999999999,"15min":1.5489999999999999},"last":1.448},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.0640000000000001,"15min":1.0620000000000001},"min":{"1min":0.80000000000000004,"5min":0.748,"15min":0.748},"max":{"1min":1.4199999999999999,"5min":1.6619999999999999,"15min":1.6619999999999999},"last":1.24}]},{"osd":6,"last update":"Wed Mar 24 21:53:18 2021","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.141,"15min":1.1399999999999999},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":1.4239999999999999,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":1.212},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.1479999999999999,"15min":1.1619999999999999},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.3819999999999999,"5min":1.806,"15min":1.806},"last":1.5149999999999999}]},{"osd":7,"last update":"Wed Mar 24 21:53:41 2021","interfaces":[{"interface":"back","average":{"1min":1.2030000000000001,"5min":1.23,"15min":1.234},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.7789999999999999,"5min":1.7789999999999999,"15min":1.7789999999999999},"last":1.367},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.175,"15min":1.181},"min":{"1min":0.96499999999999997,"5min":0.90300000000000002,"15min":0.90300000000000002},"max":{"1min":1.3480000000000001,"5min":1.532,"15min":1.532},"last":1.399}]}]},{"osd":0,"up_from":8,"seq":34359738427,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6804,"kb_used_data":844,"kb_used_omap":0,"kb_used_meta":5952,"kb_avail":93738348,"statfs":{"total":95995035648,"available":95988068352,"internally_reserved":0,"allocated":864256,"data_stored":321358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6094848},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Mar 24 21:53:37 2021","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.63700000000000001,"15min":0.58399999999999996},"min":{"1min":0.44400000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.98299999999999998,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":1.2170000000000001},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.67500000000000004,"15min":0.60999999999999999},"min":{"1min":0.46999999999999997,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.749}]},{"osd":2,"last update":"Wed Mar 24 21:52:55 2021","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.67500000000000004,"15min":0.66100000000000003},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.879,"5min":1.468,"15min":1.468},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.73099999999999998,"15min":0.69099999999999995},"min":{"1min":0.41799999999999998,"5min":0.376,"15min":0.376},"max":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":1.381}]},{"osd":3,"last update":"Wed Mar 24 21:53:20 2021","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.80500000000000005,"15min":0.80000000000000004},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.115,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":1.843},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.93400000000000005,"15min":0.93200000000000005},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.2430000000000001,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":1.597}]},{"osd":4,"last update":"Wed Mar 24 21:53:37 2021","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.0900000000000001,"15min":1.0960000000000001},"min":{"1min":0.81799999999999995,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.4570000000000001,"5min":1.7,"15min":1.7},"last":1.528},{"interface":"front","average":{"1min":1.056,"5min":1.028,"15min":1.014},"min":{"1min":0.78700000000000003,"5min":0.625,"15min":0.625},"max":{"1min":1.375,"5min":1.407,"15min":1.407},"last":1.1419999999999999}]},{"osd":5,"last update":"Wed Mar 24 21:52:55 2021","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":1.0509999999999999,"15min":1.0620000000000001},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.3919999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.98499999999999999},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.179,"15min":1.177},"min":{"1min":0.86299999999999999,"5min":0.78500000000000003,"15min":0.78500000000000003},"max":{"1min":1.597,"5min":1.746,"15min":1.746},"last":1.895}]},{"osd":6,"last update":"Wed Mar 24 21:53:20 2021","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.0669999999999999,"15min":1.0669999999999999},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.3779999999999999,"5min":1.427,"15min":1.427},"last":1.0620000000000001},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.224,"15min":1.22},"min":{"1min":0.877,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.6659999999999999,"5min":1.6659999999999999,"15min":1.6659999999999999},"last":1.8180000000000001}]},{"osd":7,"last update":"Wed Mar 24 21:53:42 2021","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.1739999999999999,"15min":1.1699999999999999},"min":{"1min":0.82099999999999995,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":1.4970000000000001,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":1.409},{"interface":"front","average":{"1min":1.157,"5min":1.131,"15min":1.127},"min":{"1min":0.77100000000000002,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.4319999999999999,"5min":1.4490000000000001,"15min":1.4490000000000001},"last":1.27}]}]},{"osd":2,"up_from":18,"seq":77309411381,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6588,"kb_used_data":820,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738564,"statfs":{"total":95995035648,"available":95988289536,"internally_reserved":0,"allocated":839680,"data_stored":326855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:53:55 2021","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.66500000000000004,"15min":0.625},"min":{"1min":0.38300000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.3640000000000001,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.72399999999999998,"15min":0.68400000000000005},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.0740000000000001,"5min":1.405,"15min":1.405},"last":0.69699999999999995}]},{"osd":1,"last update":"Wed Mar 24 21:53:55 2021","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.74299999999999999,"15min":0.69899999999999995},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.093,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.78700000000000003,"15min":0.73499999999999999},"min":{"1min":0.38300000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.6240000000000001,"5min":1.6240000000000001,"15min":1.6240000000000001},"last":0.84699999999999998}]},{"osd":3,"last update":"Wed Mar 24 21:53:17 2021","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.84399999999999997,"15min":0.82399999999999995},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.9890000000000001,"5min":1.9890000000000001,"15min":1.9890000000000001},"last":0.93200000000000005},{"interface":"front","average":{"1min":0.95699999999999996,"5min":0.92000000000000004,"15min":0.89400000000000002},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":2.0950000000000002,"5min":2.0950000000000002,"15min":2.0950000000000002},"last":0.96699999999999997}]},{"osd":4,"last update":"Wed Mar 24 21:53:40 2021","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.04,"15min":1.036},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":3.3380000000000001,"5min":3.3380000000000001,"15min":3.3380000000000001},"last":0.998},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.0680000000000001,"15min":1.046},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.2050000000000001,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":1.222}]},{"osd":5,"last update":"Wed Mar 24 21:53:55 2021","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":1.034,"15min":1.0329999999999999},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.536,"5min":1.536,"15min":1.536},"last":0.89500000000000002},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.157,"15min":1.153},"min":{"1min":0.70199999999999996,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.732,"5min":3.1509999999999998,"15min":3.1509999999999998},"last":1.1459999999999999}]},{"osd":6,"last update":"Wed Mar 24 21:53:17 2021","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.256,"15min":1.262},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":2.2810000000000001,"5min":2.2810000000000001,"15min":2.2810000000000001},"last":1.109},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.1619999999999999,"15min":1.165},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":1.8069999999999999,"5min":2.016,"15min":2.016},"last":1.2909999999999999}]},{"osd":7,"last update":"Wed Mar 24 21:53:43 2021","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":1.2190000000000001,"15min":1.228},"min":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.6719999999999999,"5min":1.6719999999999999,"15min":1.6719999999999999},"last":1.3540000000000001},{"interface":"front","average":{"1min":1.2290000000000001,"5min":1.246,"15min":1.2490000000000001},"min":{"1min":0.80800000000000005,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":3.5419999999999998,"5min":3.5419999999999998,"15min":3.5419999999999998},"last":1.2030000000000001}]}]},{"osd":3,"up_from":26,"seq":111669149744,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6904,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":5952,"kb_avail":93738248,"statfs":{"total":95995035648,"available":95987965952,"internally_reserved":0,"allocated":966656,"data_stored":324811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6094848},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:53:21 2021","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.63300000000000001,"15min":0.63500000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.92900000000000005,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.73299999999999998,"15min":0.72099999999999997},"min":{"1min":0.41499999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.72799999999999998}]},{"osd":1,"last update":"Wed Mar 24 21:53:21 2021","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.74199999999999999,"15min":0.73799999999999999},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.018,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.75600000000000001,"15min":0.755},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.97399999999999998,"5min":1.103,"15min":1.103},"last":0.98499999999999999}]},{"osd":2,"last update":"Wed Mar 24 21:53:21 2021","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.79700000000000004,"15min":0.78700000000000003},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":1.115},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.77500000000000002,"15min":0.76500000000000001},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0449999999999999,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.70199999999999996}]},{"osd":4,"last update":"Wed Mar 24 21:53:40 2021","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.95399999999999996,"15min":0.95399999999999996},"min":{"1min":0.66400000000000003,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.1950000000000001,"5min":1.702,"15min":1.702},"last":0.89900000000000002},{"interface":"front","average":{"1min":1.016,"5min":0.96599999999999997,"15min":0.94399999999999995},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.671,"5min":1.671,"15min":1.671},"last":1.048}]},{"osd":5,"last update":"Wed Mar 24 21:52:55 2021","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.0669999999999999,"15min":1.0620000000000001},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":2.423,"5min":2.423,"15min":2.423},"last":1.2170000000000001},{"interface":"front","average":{"1min":0.98999999999999999,"5min":1.0740000000000001,"15min":1.0880000000000001},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.2889999999999999,"5min":1.496,"15min":1.496},"last":1.083}]},{"osd":6,"last update":"Wed Mar 24 21:53:21 2021","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.1180000000000001,"15min":1.1259999999999999},"min":{"1min":0.85499999999999998,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.49,"5min":1.49,"15min":1.49},"last":1.143},{"interface":"front","average":{"1min":1.155,"5min":1.1539999999999999,"15min":1.1539999999999999},"min":{"1min":0.77600000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":1.7070000000000001,"5min":1.7070000000000001,"15min":1.7070000000000001},"last":1.27}]},{"osd":7,"last update":"Wed Mar 24 21:53:47 2021","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":1.0900000000000001,"15min":1.081},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.5800000000000001,"5min":1.5800000000000001,"15min":1.5800000000000001},"last":1.171},{"interface":"front","average":{"1min":1.2010000000000001,"5min":1.1679999999999999,"15min":1.1619999999999999},"min":{"1min":0.90400000000000003,"5min":0.751,"15min":0.751},"max":{"1min":1.556,"5min":1.7410000000000001,"15min":1.7410000000000001},"last":1.238}]}]},{"osd":4,"up_from":32,"seq":137438953515,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6748,"kb_used_data":916,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738404,"statfs":{"total":95995035648,"available":95988125696,"internally_reserved":0,"allocated":937984,"data_stored":329672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:53:36 2021","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.84699999999999998,"15min":0.84099999999999997},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.2490000000000001,"5min":1.268,"15min":1.268},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.89200000000000002,"15min":0.86199999999999999},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.667,"5min":1.667,"15min":1.667},"last":1.079}]},{"osd":1,"last update":"Wed Mar 24 21:53:36 2021","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.92000000000000004,"15min":0.91800000000000004},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.1439999999999999,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.86899999999999999,"15min":0.84699999999999998},"min":{"1min":0.59199999999999997,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.337,"5min":1.337,"15min":1.337},"last":1.2190000000000001}]},{"osd":2,"last update":"Wed Mar 24 21:53:36 2021","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.90700000000000003,"15min":0.89900000000000002},"min":{"1min":0.57699999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.6279999999999999,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.878,"5min":0.92800000000000005,"15min":0.91200000000000003},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.169,"5min":1.335,"15min":1.335},"last":0.78700000000000003}]},{"osd":3,"last update":"Wed Mar 24 21:53:36 2021","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.92800000000000005,"15min":0.93400000000000005},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.1359999999999999,"5min":1.601,"15min":1.601},"last":0.92400000000000004},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.97099999999999997,"15min":0.95299999999999996},"min":{"1min":0.754,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.474,"5min":1.474,"15min":1.474},"last":1.115}]},{"osd":5,"last update":"Wed Mar 24 21:52:57 2021","interfaces":[{"interface":"back","average":{"1min":1.0149999999999999,"5min":0.96299999999999997,"15min":0.95499999999999996},"min":{"1min":0.59599999999999997,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":0.81299999999999994},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.026,"15min":1},"min":{"1min":0.78300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":2.0760000000000001,"5min":2.0760000000000001,"15min":2.0760000000000001},"last":0.97599999999999998}]},{"osd":6,"last update":"Wed Mar 24 21:53:18 2021","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":1.0620000000000001,"15min":1.0640000000000001},"min":{"1min":0.79900000000000004,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.2749999999999999,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":0.83699999999999997},{"interface":"front","average":{"1min":1.018,"5min":1.056,"15min":1.0620000000000001},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.357,"5min":1.506,"15min":1.506},"last":1.004}]},{"osd":7,"last update":"Wed Mar 24 21:53:41 2021","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":1.1679999999999999,"15min":1.1870000000000001},"min":{"1min":0.624,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.587,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.1910000000000001,"15min":1.2010000000000001},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.702,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":1.1859999999999999}]}]},{"osd":5,"up_from":39,"seq":167503724584,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6492,"kb_used_data":788,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738660,"statfs":{"total":95995035648,"available":95988387840,"internally_reserved":0,"allocated":806912,"data_stored":324635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:52:56 2021","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.89000000000000001,"15min":0.88900000000000001},"min":{"1min":0.623,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":2.29,"5min":2.29,"15min":2.29},"last":0.87},{"interface":"front","average":{"1min":1.0489999999999999,"5min":0.91700000000000004,"15min":0.89500000000000002},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":3.1779999999999999,"5min":3.1779999999999999,"15min":3.1779999999999999},"last":0.75800000000000001}]},{"osd":1,"last update":"Wed Mar 24 21:52:56 2021","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":0.93200000000000005,"15min":0.92500000000000004},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":2.8690000000000002,"5min":2.8690000000000002,"15min":2.8690000000000002},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.94199999999999995,"5min":0.93400000000000005,"15min":0.93300000000000005},"min":{"1min":0.66600000000000004,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":0.78800000000000003}]},{"osd":2,"last update":"Wed Mar 24 21:52:56 2021","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.90000000000000002,"15min":0.90200000000000002},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.95999999999999996},{"interface":"front","average":{"1min":1.1839999999999999,"5min":0.94699999999999995,"15min":0.90700000000000003},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":5.8479999999999999,"5min":5.8479999999999999,"15min":5.8479999999999999},"last":0.94199999999999995}]},{"osd":3,"last update":"Wed Mar 24 21:52:56 2021","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":0.94399999999999995,"15min":0.93600000000000005},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":3.1259999999999999,"5min":3.1259999999999999,"15min":3.1259999999999999},"last":0.98499999999999999},{"interface":"front","average":{"1min":1.0409999999999999,"5min":0.95999999999999996,"15min":0.94599999999999995},"min":{"1min":0.67400000000000004,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":2.6850000000000001,"5min":2.6850000000000001,"15min":2.6850000000000001},"last":1.05}]},{"osd":4,"last update":"Wed Mar 24 21:52:56 2021","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"min":{"1min":0.55800000000000005,"5min":0.439,"15min":0.439},"max":{"1min":1.1519999999999999,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":1.085},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.90700000000000003,"15min":0.89500000000000002},"min":{"1min":0.71499999999999997,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.171,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":1.1140000000000001}]},{"osd":6,"last update":"Wed Mar 24 21:53:19 2021","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":0.89200000000000002,"15min":0.87},"min":{"1min":0.71999999999999997,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":1.3100000000000001},{"interface":"front","average":{"1min":1.153,"5min":0.99299999999999999,"15min":0.96699999999999997},"min":{"1min":0.72299999999999998,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":2.3079999999999998,"5min":2.3079999999999998,"15min":2.3079999999999998},"last":1.1639999999999999}]},{"osd":7,"last update":"Wed Mar 24 21:53:42 2021","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.0549999999999999,"15min":1.0469999999999999},"min":{"1min":0.82899999999999996,"5min":0.624,"15min":0.624},"max":{"1min":1.484,"5min":1.484,"15min":1.484},"last":1.393},{"interface":"front","average":{"1min":1.196,"5min":1.0189999999999999,"15min":0.98899999999999999},"min":{"1min":0.88800000000000001,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":2.1539999999999999,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":1.2210000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":7420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":339968,"data_stored":12094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":6670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":11712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":274432,"data_stored":9910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":5638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":9628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":7500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-03-24T21:54:00.518 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph pg dump --format=json 2021-03-24T21:54:00.842 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:54:00.909 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:00 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:54:00.909 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:00 smithi125 conmon[19936]: 21:53:58.783404+0000 osd.1 (osd.1) 26 : cluster [DBG] 5.12 scrub ok 2021-03-24T21:54:00.909 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:00 smithi125 conmon[19936]: audit 2021-03-24T21 2021-03-24T21:54:00.910 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:00 smithi125 conmon[19936]: :53:59.697324+0000 mgr.y ( 2021-03-24T21:54:00.910 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:00 smithi125 conmon[19936]: mgr.14144) 291 : audit [DBG] 2021-03-24T21:54:00.910 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:00 smithi125 conmon[19936]: from='client.34607 v1:172.21.15.125:0/2159662114' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T21:54:00.911 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:00 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:54:00.911 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:00 smithi125 conmon[24927]: 21:53:58.783404+0000 osd.1 (osd.1) 26 : cluster 2021-03-24T21:54:00.911 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:00 smithi125 conmon[24927]: [DBG] 5.12 scrub ok 2021-03-24T21:54:00.911 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:00 smithi125 conmon[24927]: audit 2021 2021-03-24T21:54:00.911 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:00 smithi125 conmon[24927]: -03-24T21:53: 2021-03-24T21:54:00.912 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:00 smithi125 conmon[24927]: 59.697324+0000 mgr.y ( 2021-03-24T21:54:00.912 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:00 smithi125 conmon[24927]: mgr.14144) 291 : audit 2021-03-24T21:54:00.912 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:00 smithi125 conmon[24927]: [DBG] from='client.34607 v1:172.21.15.125:0/2159662114' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T21:54:01.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:00 smithi142 conmon[20125]: cluster 2021-03-24T21:53: 2021-03-24T21:54:01.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:00 smithi142 conmon[20125]: 58.783404+0000 osd.1 (osd.1) 26 : cluster [DBG] 5.12 scrub ok 2021-03-24T21:54:01.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:00 smithi142 conmon[20125]: audit 2021 2021-03-24T21:54:01.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:00 smithi142 conmon[20125]: -03-24T21:53:59.697324+0000 mgr.y (mgr 2021-03-24T21:54:01.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:00 smithi142 conmon[20125]: .14144) 291 : audit [DBG] 2021-03-24T21:54:01.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:00 smithi142 conmon[20125]: from='client.34607 v1:172.21.15.125:0/2159662114' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T21:54:01.909 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:01 smithi125 conmon[19936]: cluster 2021-03-24T21:54 2021-03-24T21:54:01.909 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:01 smithi125 conmon[19936]: :00.693826+0000 mgr.y (mgr.14144) 292 : cluster [DBG] pgmap v292: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-03-24T21:54:01.910 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:01 smithi125 conmon[24927]: cluster 2021-03-24T21:54: 2021-03-24T21:54:01.910 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:01 smithi125 conmon[24927]: 00.693826+0000 mgr.y (mgr.14144) 292 : cluster [DBG] pgmap v292: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-03-24T21:54:02.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:01 smithi142 conmon[20125]: cluster 2021-03-24T21:54: 2021-03-24T21:54:02.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:01 smithi142 conmon[20125]: 00.693826+0000 mgr.y (mgr.14144) 292 : cluster [DBG] pgmap v292: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-03-24T21:54:02.763 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-24T21:54:02.765 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2021-03-24T21:54:02.953 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:02 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:54:02.953 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:02 smithi125 conmon[24927]: 54:00.911003+0000 osd.5 (osd.5) 27 : cluster [DBG] 3.c scrub ok 2021-03-24T21:54:02.954 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:02 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:54:02.954 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:02 smithi125 conmon[19936]: -24T21:54:00.911003+0000 osd.5 (osd.5) 27 2021-03-24T21:54:02.954 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:02 smithi125 conmon[19936]: : cluster [DBG] 3.c scrub ok 2021-03-24T21:54:02.954 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:02 smithi125 conmon[19936]: 2021-03-24T21:54:03.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:02 smithi142 conmon[20125]: cluster 2021-03-24T21:54:00.911003+0000 osd.5 (osd.5) 27 : 2021-03-24T21:54:03.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:02 smithi142 conmon[20125]: cluster [DBG] 3.c scrub ok 2021-03-24T21:54:03.624 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":293,"stamp":"2021-03-24T21:54:02.694037+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":25236,"num_objects":316,"num_object_clones":0,"num_object_copies":948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":316,"num_whiteouts":0,"num_read":1206,"num_read_kb":706,"num_write":550,"num_write_kb":165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":2412,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":570,"ondisk_log_size":570,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":53776,"kb_used_data":7056,"kb_used_omap":0,"kb_used_meta":46656,"kb_avail":749907440,"statfs":{"total":767960285184,"available":767905218560,"internally_reserved":0,"allocated":7225344,"data_stored":2628915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":47775744},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003943"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":"40","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.099187+0000","last_change":"2021-03-24T21:52:59.180417+0000","last_active":"2021-03-24T21:53:51.099187+0000","last_peered":"2021-03-24T21:53:51.099187+0000","last_clean":"2021-03-24T21:53:51.099187+0000","last_became_active":"2021-03-24T21:51:51.759485+0000","last_became_peered":"2021-03-24T21:51:51.759485+0000","last_unstale":"2021-03-24T21:53:51.099187+0000","last_undegraded":"2021-03-24T21:53:51.099187+0000","last_fullsized":"2021-03-24T21:53:51.099187+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:59.180240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:52:59.180240+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"43","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.100701+0000","last_change":"2021-03-24T21:53:02.553145+0000","last_active":"2021-03-24T21:53:51.100701+0000","last_peered":"2021-03-24T21:53:51.100701+0000","last_clean":"2021-03-24T21:53:51.100701+0000","last_became_active":"2021-03-24T21:51:49.415254+0000","last_became_peered":"2021-03-24T21:51:49.415254+0000","last_unstale":"2021-03-24T21:53:51.100701+0000","last_undegraded":"2021-03-24T21:53:51.100701+0000","last_fullsized":"2021-03-24T21:53:51.100701+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:02.553036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:02.553036+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":"45","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.098236+0000","last_change":"2021-03-24T21:53:12.428679+0000","last_active":"2021-03-24T21:53:51.098236+0000","last_peered":"2021-03-24T21:53:51.098236+0000","last_clean":"2021-03-24T21:53:51.098236+0000","last_became_active":"2021-03-24T21:51:45.581000+0000","last_became_peered":"2021-03-24T21:51:45.581000+0000","last_unstale":"2021-03-24T21:53:51.098236+0000","last_undegraded":"2021-03-24T21:53:51.098236+0000","last_fullsized":"2021-03-24T21:53:51.098236+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:12.428497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:12.428497+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"69'15","reported_seq":"72","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.098411+0000","last_change":"2021-03-24T21:53:41.296073+0000","last_active":"2021-03-24T21:53:51.098411+0000","last_peered":"2021-03-24T21:53:51.098411+0000","last_clean":"2021-03-24T21:53:51.098411+0000","last_became_active":"2021-03-24T21:51:47.410353+0000","last_became_peered":"2021-03-24T21:51:47.410353+0000","last_unstale":"2021-03-24T21:53:51.098411+0000","last_undegraded":"2021-03-24T21:53:51.098411+0000","last_fullsized":"2021-03-24T21:53:51.098411+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:53:41.295875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:41.295875+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"40","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.745145+0000","last_change":"2021-03-24T21:52:58.676144+0000","last_active":"2021-03-24T21:53:51.745145+0000","last_peered":"2021-03-24T21:53:51.745145+0000","last_clean":"2021-03-24T21:53:51.745145+0000","last_became_active":"2021-03-24T21:51:51.915108+0000","last_became_peered":"2021-03-24T21:51:51.915108+0000","last_unstale":"2021-03-24T21:53:51.745145+0000","last_undegraded":"2021-03-24T21:53:51.745145+0000","last_fullsized":"2021-03-24T21:53:51.745145+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:58.676060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:52:58.676060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"42","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.098406+0000","last_change":"2021-03-24T21:53:48.329647+0000","last_active":"2021-03-24T21:53:51.098406+0000","last_peered":"2021-03-24T21:53:51.098406+0000","last_clean":"2021-03-24T21:53:51.098406+0000","last_became_active":"2021-03-24T21:51:49.409015+0000","last_became_peered":"2021-03-24T21:51:49.409015+0000","last_unstale":"2021-03-24T21:53:51.098406+0000","last_undegraded":"2021-03-24T21:53:51.098406+0000","last_fullsized":"2021-03-24T21:53:51.098406+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:48.329480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:48.329480+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":"46","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.746761+0000","last_change":"2021-03-24T21:53:18.706895+0000","last_active":"2021-03-24T21:53:51.746761+0000","last_peered":"2021-03-24T21:53:51.746761+0000","last_clean":"2021-03-24T21:53:51.746761+0000","last_became_active":"2021-03-24T21:51:45.581465+0000","last_became_peered":"2021-03-24T21:51:45.581465+0000","last_unstale":"2021-03-24T21:53:51.746761+0000","last_undegraded":"2021-03-24T21:53:51.746761+0000","last_fullsized":"2021-03-24T21:53:51.746761+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:18.706745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:18.706745+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"69'20","reported_seq":"79","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.745368+0000","last_change":"2021-03-24T21:53:12.604981+0000","last_active":"2021-03-24T21:53:51.745368+0000","last_peered":"2021-03-24T21:53:51.745368+0000","last_clean":"2021-03-24T21:53:51.745368+0000","last_became_active":"2021-03-24T21:51:47.405776+0000","last_became_peered":"2021-03-24T21:51:47.405776+0000","last_unstale":"2021-03-24T21:53:51.745368+0000","last_undegraded":"2021-03-24T21:53:51.745368+0000","last_fullsized":"2021-03-24T21:53:51.745368+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'20","last_scrub_stamp":"2021-03-24T21:53:12.604757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:12.604757+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":22,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"39","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.096999+0000","last_change":"2021-03-24T21:53:27.604422+0000","last_active":"2021-03-24T21:53:51.096999+0000","last_peered":"2021-03-24T21:53:51.096999+0000","last_clean":"2021-03-24T21:53:51.096999+0000","last_became_active":"2021-03-24T21:51:51.758026+0000","last_became_peered":"2021-03-24T21:51:51.758026+0000","last_unstale":"2021-03-24T21:53:51.096999+0000","last_undegraded":"2021-03-24T21:53:51.096999+0000","last_fullsized":"2021-03-24T21:53:51.096999+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:27.604230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:27.604230+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"43","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.549752+0000","last_change":"2021-03-24T21:52:56.855476+0000","last_active":"2021-03-24T21:53:51.549752+0000","last_peered":"2021-03-24T21:53:51.549752+0000","last_clean":"2021-03-24T21:53:51.549752+0000","last_became_active":"2021-03-24T21:51:49.413342+0000","last_became_peered":"2021-03-24T21:51:49.413342+0000","last_unstale":"2021-03-24T21:53:51.549752+0000","last_undegraded":"2021-03-24T21:53:51.549752+0000","last_fullsized":"2021-03-24T21:53:51.549752+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:56.855305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:52:56.855305+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":"47","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.550271+0000","last_change":"2021-03-24T21:52:50.788792+0000","last_active":"2021-03-24T21:53:51.550271+0000","last_peered":"2021-03-24T21:53:51.550271+0000","last_clean":"2021-03-24T21:53:51.550271+0000","last_became_active":"2021-03-24T21:51:45.382348+0000","last_became_peered":"2021-03-24T21:51:45.382348+0000","last_unstale":"2021-03-24T21:53:51.550271+0000","last_undegraded":"2021-03-24T21:53:51.550271+0000","last_fullsized":"2021-03-24T21:53:51.550271+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:50.788606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:52:50.788606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"69'13","reported_seq":"68","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.099593+0000","last_change":"2021-03-24T21:53:10.104959+0000","last_active":"2021-03-24T21:53:51.099593+0000","last_peered":"2021-03-24T21:53:51.099593+0000","last_clean":"2021-03-24T21:53:51.099593+0000","last_became_active":"2021-03-24T21:51:47.407488+0000","last_became_peered":"2021-03-24T21:51:47.407488+0000","last_unstale":"2021-03-24T21:53:51.099593+0000","last_undegraded":"2021-03-24T21:53:51.099593+0000","last_fullsized":"2021-03-24T21:53:51.099593+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2021-03-24T21:53:10.104806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:10.104806+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"41","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.549132+0000","last_change":"2021-03-24T21:53:23.896132+0000","last_active":"2021-03-24T21:53:51.549132+0000","last_peered":"2021-03-24T21:53:51.549132+0000","last_clean":"2021-03-24T21:53:51.549132+0000","last_became_active":"2021-03-24T21:51:51.765338+0000","last_became_peered":"2021-03-24T21:51:51.765338+0000","last_unstale":"2021-03-24T21:53:51.549132+0000","last_undegraded":"2021-03-24T21:53:51.549132+0000","last_fullsized":"2021-03-24T21:53:51.549132+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:23.895966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:23.895966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"41","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.651766+0000","last_change":"2021-03-24T21:53:00.845488+0000","last_active":"2021-03-24T21:53:51.651766+0000","last_peered":"2021-03-24T21:53:51.651766+0000","last_clean":"2021-03-24T21:53:51.651766+0000","last_became_active":"2021-03-24T21:51:49.411327+0000","last_became_peered":"2021-03-24T21:51:49.411327+0000","last_unstale":"2021-03-24T21:53:51.651766+0000","last_undegraded":"2021-03-24T21:53:51.651766+0000","last_fullsized":"2021-03-24T21:53:51.651766+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:00.845338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:00.845338+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":"47","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.549645+0000","last_change":"2021-03-24T21:52:45.798051+0000","last_active":"2021-03-24T21:53:51.549645+0000","last_peered":"2021-03-24T21:53:51.549645+0000","last_clean":"2021-03-24T21:53:51.549645+0000","last_became_active":"2021-03-24T21:51:45.381362+0000","last_became_peered":"2021-03-24T21:51:45.381362+0000","last_unstale":"2021-03-24T21:53:51.549645+0000","last_undegraded":"2021-03-24T21:53:51.549645+0000","last_fullsized":"2021-03-24T21:53:51.549645+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:45.797873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:52:45.797873+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"91'9","reported_seq":"59","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.099621+0000","last_change":"2021-03-24T21:53:33.290178+0000","last_active":"2021-03-24T21:53:51.099621+0000","last_peered":"2021-03-24T21:53:51.099621+0000","last_clean":"2021-03-24T21:53:51.099621+0000","last_became_active":"2021-03-24T21:51:47.416379+0000","last_became_peered":"2021-03-24T21:51:47.416379+0000","last_unstale":"2021-03-24T21:53:51.099621+0000","last_undegraded":"2021-03-24T21:53:51.099621+0000","last_fullsized":"2021-03-24T21:53:51.099621+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2021-03-24T21:53:33.290000+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2021-03-24T21:53:33.290000+0000","last_clean_scrub_stamp":"2021-03-24T21:53:33.290000+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":8,"num_write":7,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"40","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.100374+0000","last_change":"2021-03-24T21:53:39.277601+0000","last_active":"2021-03-24T21:53:51.100374+0000","last_peered":"2021-03-24T21:53:51.100374+0000","last_clean":"2021-03-24T21:53:51.100374+0000","last_became_active":"2021-03-24T21:51:51.910950+0000","last_became_peered":"2021-03-24T21:51:51.910950+0000","last_unstale":"2021-03-24T21:53:51.100374+0000","last_undegraded":"2021-03-24T21:53:51.100374+0000","last_fullsized":"2021-03-24T21:53:51.100374+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:39.277425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:39.277425+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"43","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.102599+0000","last_change":"2021-03-24T21:52:59.579706+0000","last_active":"2021-03-24T21:53:51.102599+0000","last_peered":"2021-03-24T21:53:51.102599+0000","last_clean":"2021-03-24T21:53:51.102599+0000","last_became_active":"2021-03-24T21:51:49.411914+0000","last_became_peered":"2021-03-24T21:51:49.411914+0000","last_unstale":"2021-03-24T21:53:51.102599+0000","last_undegraded":"2021-03-24T21:53:51.102599+0000","last_fullsized":"2021-03-24T21:53:51.102599+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:59.579528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:52:59.579528+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":"45","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.098099+0000","last_change":"2021-03-24T21:53:04.263170+0000","last_active":"2021-03-24T21:53:51.098099+0000","last_peered":"2021-03-24T21:53:51.098099+0000","last_clean":"2021-03-24T21:53:51.098099+0000","last_became_active":"2021-03-24T21:51:45.381533+0000","last_became_peered":"2021-03-24T21:51:45.381533+0000","last_unstale":"2021-03-24T21:53:51.098099+0000","last_undegraded":"2021-03-24T21:53:51.098099+0000","last_fullsized":"2021-03-24T21:53:51.098099+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:04.262989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:04.262989+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"69'15","reported_seq":"100","reported_epoch":"94","state":"active+clean","last_fresh":"2021-03-24T21:53:58.587333+0000","last_change":"2021-03-24T21:53:58.587333+0000","last_active":"2021-03-24T21:53:58.587333+0000","last_peered":"2021-03-24T21:53:58.587333+0000","last_clean":"2021-03-24T21:53:58.587333+0000","last_became_active":"2021-03-24T21:52:06.056116+0000","last_became_peered":"2021-03-24T21:52:06.056116+0000","last_unstale":"2021-03-24T21:53:58.587333+0000","last_undegraded":"2021-03-24T21:53:58.587333+0000","last_fullsized":"2021-03-24T21:53:58.587333+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:53:58.587152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:58.587152+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"40","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.746441+0000","last_change":"2021-03-24T21:53:44.657926+0000","last_active":"2021-03-24T21:53:51.746441+0000","last_peered":"2021-03-24T21:53:51.746441+0000","last_clean":"2021-03-24T21:53:51.746441+0000","last_became_active":"2021-03-24T21:51:51.914955+0000","last_became_peered":"2021-03-24T21:51:51.914955+0000","last_unstale":"2021-03-24T21:53:51.746441+0000","last_undegraded":"2021-03-24T21:53:51.746441+0000","last_fullsized":"2021-03-24T21:53:51.746441+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:44.657761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:44.657761+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"42","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.099336+0000","last_change":"2021-03-24T21:53:27.238846+0000","last_active":"2021-03-24T21:53:51.099336+0000","last_peered":"2021-03-24T21:53:51.099336+0000","last_clean":"2021-03-24T21:53:51.099336+0000","last_became_active":"2021-03-24T21:51:49.410920+0000","last_became_peered":"2021-03-24T21:51:49.410920+0000","last_unstale":"2021-03-24T21:53:51.099336+0000","last_undegraded":"2021-03-24T21:53:51.099336+0000","last_fullsized":"2021-03-24T21:53:51.099336+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:27.238614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:27.238614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":"46","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.100343+0000","last_change":"2021-03-24T21:52:54.212593+0000","last_active":"2021-03-24T21:53:51.100343+0000","last_peered":"2021-03-24T21:53:51.100343+0000","last_clean":"2021-03-24T21:53:51.100343+0000","last_became_active":"2021-03-24T21:51:45.579326+0000","last_became_peered":"2021-03-24T21:51:45.579326+0000","last_unstale":"2021-03-24T21:53:51.100343+0000","last_undegraded":"2021-03-24T21:53:51.100343+0000","last_fullsized":"2021-03-24T21:53:51.100343+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:54.212459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:52:54.212459+0000","last_clean_scrub_stamp":"2021-03-24T21:52:54.212459+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"69'16","reported_seq":"75","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.746310+0000","last_change":"2021-03-24T21:53:42.611713+0000","last_active":"2021-03-24T21:53:51.746310+0000","last_peered":"2021-03-24T21:53:51.746310+0000","last_clean":"2021-03-24T21:53:51.746310+0000","last_became_active":"2021-03-24T21:51:47.411418+0000","last_became_peered":"2021-03-24T21:51:47.411418+0000","last_unstale":"2021-03-24T21:53:51.746310+0000","last_undegraded":"2021-03-24T21:53:51.746310+0000","last_fullsized":"2021-03-24T21:53:51.746310+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T21:53:42.611540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:42.611540+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":37,"num_read_kb":20,"num_write":18,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"39","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.652524+0000","last_change":"2021-03-24T21:53:05.847583+0000","last_active":"2021-03-24T21:53:51.652524+0000","last_peered":"2021-03-24T21:53:51.652524+0000","last_clean":"2021-03-24T21:53:51.652524+0000","last_became_active":"2021-03-24T21:51:51.761492+0000","last_became_peered":"2021-03-24T21:51:51.761492+0000","last_unstale":"2021-03-24T21:53:51.652524+0000","last_undegraded":"2021-03-24T21:53:51.652524+0000","last_fullsized":"2021-03-24T21:53:51.652524+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:05.847412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:05.847412+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"65'2","reported_seq":"70","reported_epoch":"94","state":"active+clean","last_fresh":"2021-03-24T21:53:55.070341+0000","last_change":"2021-03-24T21:52:59.074600+0000","last_active":"2021-03-24T21:53:55.070341+0000","last_peered":"2021-03-24T21:53:55.070341+0000","last_clean":"2021-03-24T21:53:55.070341+0000","last_became_active":"2021-03-24T21:51:49.419519+0000","last_became_peered":"2021-03-24T21:51:49.419519+0000","last_unstale":"2021-03-24T21:53:55.070341+0000","last_undegraded":"2021-03-24T21:53:55.070341+0000","last_fullsized":"2021-03-24T21:53:55.070341+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2021-03-24T21:52:59.074417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:52:59.074417+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":"47","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.100870+0000","last_change":"2021-03-24T21:53:09.741058+0000","last_active":"2021-03-24T21:53:51.100870+0000","last_peered":"2021-03-24T21:53:51.100870+0000","last_clean":"2021-03-24T21:53:51.100870+0000","last_became_active":"2021-03-24T21:51:45.374975+0000","last_became_peered":"2021-03-24T21:51:45.374975+0000","last_unstale":"2021-03-24T21:53:51.100870+0000","last_undegraded":"2021-03-24T21:53:51.100870+0000","last_fullsized":"2021-03-24T21:53:51.100870+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:09.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:09.740867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"89'27","reported_seq":"81","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.098031+0000","last_change":"2021-03-24T21:53:21.585108+0000","last_active":"2021-03-24T21:53:51.098031+0000","last_peered":"2021-03-24T21:53:51.098031+0000","last_clean":"2021-03-24T21:53:51.098031+0000","last_became_active":"2021-03-24T21:51:47.688995+0000","last_became_peered":"2021-03-24T21:51:47.688995+0000","last_unstale":"2021-03-24T21:53:51.098031+0000","last_undegraded":"2021-03-24T21:53:51.098031+0000","last_fullsized":"2021-03-24T21:53:51.098031+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'27","last_scrub_stamp":"2021-03-24T21:53:21.584999+0000","last_deep_scrub":"89'27","last_deep_scrub_stamp":"2021-03-24T21:53:21.584999+0000","last_clean_scrub_stamp":"2021-03-24T21:53:21.584999+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"41","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.102757+0000","last_change":"2021-03-24T21:53:14.700106+0000","last_active":"2021-03-24T21:53:51.102757+0000","last_peered":"2021-03-24T21:53:51.102757+0000","last_clean":"2021-03-24T21:53:51.102757+0000","last_became_active":"2021-03-24T21:51:51.758605+0000","last_became_peered":"2021-03-24T21:51:51.758605+0000","last_unstale":"2021-03-24T21:53:51.102757+0000","last_undegraded":"2021-03-24T21:53:51.102757+0000","last_fullsized":"2021-03-24T21:53:51.102757+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:14.699852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:14.699852+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"41","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.098328+0000","last_change":"2021-03-24T21:53:30.660884+0000","last_active":"2021-03-24T21:53:51.098328+0000","last_peered":"2021-03-24T21:53:51.098328+0000","last_clean":"2021-03-24T21:53:51.098328+0000","last_became_active":"2021-03-24T21:51:49.409745+0000","last_became_peered":"2021-03-24T21:51:49.409745+0000","last_unstale":"2021-03-24T21:53:51.098328+0000","last_undegraded":"2021-03-24T21:53:51.098328+0000","last_fullsized":"2021-03-24T21:53:51.098328+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:30.660636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:30.660636+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":"46","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.743860+0000","last_change":"2021-03-24T21:53:11.648668+0000","last_active":"2021-03-24T21:53:51.743860+0000","last_peered":"2021-03-24T21:53:51.743860+0000","last_clean":"2021-03-24T21:53:51.743860+0000","last_became_active":"2021-03-24T21:51:45.373294+0000","last_became_peered":"2021-03-24T21:51:45.373294+0000","last_unstale":"2021-03-24T21:53:51.743860+0000","last_undegraded":"2021-03-24T21:53:51.743860+0000","last_fullsized":"2021-03-24T21:53:51.743860+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:11.648470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:11.648470+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"69'17","reported_seq":"108","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.745417+0000","last_change":"2021-03-24T21:53:17.246451+0000","last_active":"2021-03-24T21:53:51.745417+0000","last_peered":"2021-03-24T21:53:51.745417+0000","last_clean":"2021-03-24T21:53:51.745417+0000","last_became_active":"2021-03-24T21:52:06.054746+0000","last_became_peered":"2021-03-24T21:52:06.054746+0000","last_unstale":"2021-03-24T21:53:51.745417+0000","last_undegraded":"2021-03-24T21:53:51.745417+0000","last_fullsized":"2021-03-24T21:53:51.745417+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'17","last_scrub_stamp":"2021-03-24T21:53:17.246367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:17.246367+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":38,"num_read_kb":19,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"39","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.651538+0000","last_change":"2021-03-24T21:53:27.877846+0000","last_active":"2021-03-24T21:53:51.651538+0000","last_peered":"2021-03-24T21:53:51.651538+0000","last_clean":"2021-03-24T21:53:51.651538+0000","last_became_active":"2021-03-24T21:51:51.762150+0000","last_became_peered":"2021-03-24T21:51:51.762150+0000","last_unstale":"2021-03-24T21:53:51.651538+0000","last_undegraded":"2021-03-24T21:53:51.651538+0000","last_fullsized":"2021-03-24T21:53:51.651538+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:27.877627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:27.877627+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"42","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.743897+0000","last_change":"2021-03-24T21:53:13.143455+0000","last_active":"2021-03-24T21:53:51.743897+0000","last_peered":"2021-03-24T21:53:51.743897+0000","last_clean":"2021-03-24T21:53:51.743897+0000","last_became_active":"2021-03-24T21:51:49.414221+0000","last_became_peered":"2021-03-24T21:51:49.414221+0000","last_unstale":"2021-03-24T21:53:51.743897+0000","last_undegraded":"2021-03-24T21:53:51.743897+0000","last_fullsized":"2021-03-24T21:53:51.743897+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:13.143327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:13.143327+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":"45","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.098396+0000","last_change":"2021-03-24T21:53:10.429942+0000","last_active":"2021-03-24T21:53:51.098396+0000","last_peered":"2021-03-24T21:53:51.098396+0000","last_clean":"2021-03-24T21:53:51.098396+0000","last_became_active":"2021-03-24T21:51:45.379789+0000","last_became_peered":"2021-03-24T21:51:45.379789+0000","last_unstale":"2021-03-24T21:53:51.098396+0000","last_undegraded":"2021-03-24T21:53:51.098396+0000","last_fullsized":"2021-03-24T21:53:51.098396+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:10.429783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:10.429783+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"69'12","reported_seq":"97","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.744085+0000","last_change":"2021-03-24T21:52:52.579983+0000","last_active":"2021-03-24T21:53:51.744085+0000","last_peered":"2021-03-24T21:53:51.744085+0000","last_clean":"2021-03-24T21:53:51.744085+0000","last_became_active":"2021-03-24T21:52:06.053964+0000","last_became_peered":"2021-03-24T21:52:06.053964+0000","last_unstale":"2021-03-24T21:53:51.744085+0000","last_undegraded":"2021-03-24T21:53:51.744085+0000","last_fullsized":"2021-03-24T21:53:51.744085+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2021-03-24T21:52:52.579796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:52:52.579796+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":12,"num_write":12,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"40","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.744862+0000","last_change":"2021-03-24T21:53:47.540975+0000","last_active":"2021-03-24T21:53:51.744862+0000","last_peered":"2021-03-24T21:53:51.744862+0000","last_clean":"2021-03-24T21:53:51.744862+0000","last_became_active":"2021-03-24T21:51:51.911828+0000","last_became_peered":"2021-03-24T21:51:51.911828+0000","last_unstale":"2021-03-24T21:53:51.744862+0000","last_undegraded":"2021-03-24T21:53:51.744862+0000","last_fullsized":"2021-03-24T21:53:51.744862+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:47.540793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:47.540793+0000","last_clean_scrub_stamp":"2021-03-24T21:53:47.540793+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"41","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.651297+0000","last_change":"2021-03-24T21:52:52.829549+0000","last_active":"2021-03-24T21:53:51.651297+0000","last_peered":"2021-03-24T21:53:51.651297+0000","last_clean":"2021-03-24T21:53:51.651297+0000","last_became_active":"2021-03-24T21:51:49.422591+0000","last_became_peered":"2021-03-24T21:51:49.422591+0000","last_unstale":"2021-03-24T21:53:51.651297+0000","last_undegraded":"2021-03-24T21:53:51.651297+0000","last_fullsized":"2021-03-24T21:53:51.651297+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:52.829381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:52:52.829381+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":"47","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.550276+0000","last_change":"2021-03-24T21:53:21.833111+0000","last_active":"2021-03-24T21:53:51.550276+0000","last_peered":"2021-03-24T21:53:51.550276+0000","last_clean":"2021-03-24T21:53:51.550276+0000","last_became_active":"2021-03-24T21:51:45.385892+0000","last_became_peered":"2021-03-24T21:51:45.385892+0000","last_unstale":"2021-03-24T21:53:51.550276+0000","last_undegraded":"2021-03-24T21:53:51.550276+0000","last_fullsized":"2021-03-24T21:53:51.550276+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:21.832921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:21.832921+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"86'22","reported_seq":"77","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.745121+0000","last_change":"2021-03-24T21:52:54.597687+0000","last_active":"2021-03-24T21:53:51.745121+0000","last_peered":"2021-03-24T21:53:51.745121+0000","last_clean":"2021-03-24T21:53:51.745121+0000","last_became_active":"2021-03-24T21:51:47.687561+0000","last_became_peered":"2021-03-24T21:51:47.687561+0000","last_unstale":"2021-03-24T21:53:51.745121+0000","last_undegraded":"2021-03-24T21:53:51.745121+0000","last_fullsized":"2021-03-24T21:53:51.745121+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'22","last_scrub_stamp":"2021-03-24T21:52:54.597505+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2021-03-24T21:52:54.597505+0000","last_clean_scrub_stamp":"2021-03-24T21:52:54.597505+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":16,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"39","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.096305+0000","last_change":"2021-03-24T21:53:13.462014+0000","last_active":"2021-03-24T21:53:51.096305+0000","last_peered":"2021-03-24T21:53:51.096305+0000","last_clean":"2021-03-24T21:53:51.096305+0000","last_became_active":"2021-03-24T21:51:51.757253+0000","last_became_peered":"2021-03-24T21:51:51.757253+0000","last_unstale":"2021-03-24T21:53:51.096305+0000","last_undegraded":"2021-03-24T21:53:51.096305+0000","last_fullsized":"2021-03-24T21:53:51.096305+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:13.461940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:13.461940+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"42","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.745261+0000","last_change":"2021-03-24T21:53:31.665546+0000","last_active":"2021-03-24T21:53:51.745261+0000","last_peered":"2021-03-24T21:53:51.745261+0000","last_clean":"2021-03-24T21:53:51.745261+0000","last_became_active":"2021-03-24T21:51:49.418515+0000","last_became_peered":"2021-03-24T21:51:49.418515+0000","last_unstale":"2021-03-24T21:53:51.745261+0000","last_undegraded":"2021-03-24T21:53:51.745261+0000","last_fullsized":"2021-03-24T21:53:51.745261+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:31.665434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:31.665434+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":"45","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.652747+0000","last_change":"2021-03-24T21:53:08.841487+0000","last_active":"2021-03-24T21:53:51.652747+0000","last_peered":"2021-03-24T21:53:51.652747+0000","last_clean":"2021-03-24T21:53:51.652747+0000","last_became_active":"2021-03-24T21:51:45.379383+0000","last_became_peered":"2021-03-24T21:51:45.379383+0000","last_unstale":"2021-03-24T21:53:51.652747+0000","last_undegraded":"2021-03-24T21:53:51.652747+0000","last_fullsized":"2021-03-24T21:53:51.652747+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:08.841310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:08.841310+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"69'15","reported_seq":"73","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.101931+0000","last_change":"2021-03-24T21:53:04.637959+0000","last_active":"2021-03-24T21:53:51.101931+0000","last_peered":"2021-03-24T21:53:51.101931+0000","last_clean":"2021-03-24T21:53:51.101931+0000","last_became_active":"2021-03-24T21:51:47.403130+0000","last_became_peered":"2021-03-24T21:51:47.403130+0000","last_unstale":"2021-03-24T21:53:51.101931+0000","last_undegraded":"2021-03-24T21:53:51.101931+0000","last_fullsized":"2021-03-24T21:53:51.101931+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:53:04.637877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:04.637877+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"40","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.744258+0000","last_change":"2021-03-24T21:53:23.708000+0000","last_active":"2021-03-24T21:53:51.744258+0000","last_peered":"2021-03-24T21:53:51.744258+0000","last_clean":"2021-03-24T21:53:51.744258+0000","last_became_active":"2021-03-24T21:51:51.764871+0000","last_became_peered":"2021-03-24T21:51:51.764871+0000","last_unstale":"2021-03-24T21:53:51.744258+0000","last_undegraded":"2021-03-24T21:53:51.744258+0000","last_fullsized":"2021-03-24T21:53:51.744258+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:23.707914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:23.707914+0000","last_clean_scrub_stamp":"2021-03-24T21:53:23.707914+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"48","reported_epoch":"94","state":"active+clean","last_fresh":"2021-03-24T21:53:58.783603+0000","last_change":"2021-03-24T21:53:58.783603+0000","last_active":"2021-03-24T21:53:58.783603+0000","last_peered":"2021-03-24T21:53:58.783603+0000","last_clean":"2021-03-24T21:53:58.783603+0000","last_became_active":"2021-03-24T21:51:49.415559+0000","last_became_peered":"2021-03-24T21:51:49.415559+0000","last_unstale":"2021-03-24T21:53:58.783603+0000","last_undegraded":"2021-03-24T21:53:58.783603+0000","last_fullsized":"2021-03-24T21:53:58.783603+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:58.783414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:58.783414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":"46","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.099609+0000","last_change":"2021-03-24T21:53:36.317121+0000","last_active":"2021-03-24T21:53:51.099609+0000","last_peered":"2021-03-24T21:53:51.099609+0000","last_clean":"2021-03-24T21:53:51.099609+0000","last_became_active":"2021-03-24T21:51:45.379862+0000","last_became_peered":"2021-03-24T21:51:45.379862+0000","last_unstale":"2021-03-24T21:53:51.099609+0000","last_undegraded":"2021-03-24T21:53:51.099609+0000","last_fullsized":"2021-03-24T21:53:51.099609+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:36.316893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:36.316893+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"69'15","reported_seq":"71","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.099732+0000","last_change":"2021-03-24T21:53:21.207831+0000","last_active":"2021-03-24T21:53:51.099732+0000","last_peered":"2021-03-24T21:53:51.099732+0000","last_clean":"2021-03-24T21:53:51.099732+0000","last_became_active":"2021-03-24T21:51:47.416319+0000","last_became_peered":"2021-03-24T21:51:47.416319+0000","last_unstale":"2021-03-24T21:53:51.099732+0000","last_undegraded":"2021-03-24T21:53:51.099732+0000","last_fullsized":"2021-03-24T21:53:51.099732+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:53:21.207696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:21.207696+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"39","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.097681+0000","last_change":"2021-03-24T21:53:06.309234+0000","last_active":"2021-03-24T21:53:51.097681+0000","last_peered":"2021-03-24T21:53:51.097681+0000","last_clean":"2021-03-24T21:53:51.097681+0000","last_became_active":"2021-03-24T21:51:51.756584+0000","last_became_peered":"2021-03-24T21:51:51.756584+0000","last_unstale":"2021-03-24T21:53:51.097681+0000","last_undegraded":"2021-03-24T21:53:51.097681+0000","last_fullsized":"2021-03-24T21:53:51.097681+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:06.309066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:06.309066+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"86'3","reported_seq":"71","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:53.817486+0000","last_change":"2021-03-24T21:53:08.815321+0000","last_active":"2021-03-24T21:53:53.817486+0000","last_peered":"2021-03-24T21:53:53.817486+0000","last_clean":"2021-03-24T21:53:53.817486+0000","last_became_active":"2021-03-24T21:51:49.419663+0000","last_became_peered":"2021-03-24T21:51:49.419663+0000","last_unstale":"2021-03-24T21:53:53.817486+0000","last_undegraded":"2021-03-24T21:53:53.817486+0000","last_fullsized":"2021-03-24T21:53:53.817486+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'3","last_scrub_stamp":"2021-03-24T21:53:08.815163+0000","last_deep_scrub":"86'3","last_deep_scrub_stamp":"2021-03-24T21:53:08.815163+0000","last_clean_scrub_stamp":"2021-03-24T21:53:08.815163+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":"45","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.651716+0000","last_change":"2021-03-24T21:53:15.868395+0000","last_active":"2021-03-24T21:53:51.651716+0000","last_peered":"2021-03-24T21:53:51.651716+0000","last_clean":"2021-03-24T21:53:51.651716+0000","last_became_active":"2021-03-24T21:51:45.579271+0000","last_became_peered":"2021-03-24T21:51:45.579271+0000","last_unstale":"2021-03-24T21:53:51.651716+0000","last_undegraded":"2021-03-24T21:53:51.651716+0000","last_fullsized":"2021-03-24T21:53:51.651716+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:15.868303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:15.868303+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"69'14","reported_seq":"67","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.746302+0000","last_change":"2021-03-24T21:53:02.723125+0000","last_active":"2021-03-24T21:53:51.746302+0000","last_peered":"2021-03-24T21:53:51.746302+0000","last_clean":"2021-03-24T21:53:51.746302+0000","last_became_active":"2021-03-24T21:51:47.686103+0000","last_became_peered":"2021-03-24T21:51:47.686103+0000","last_unstale":"2021-03-24T21:53:51.746302+0000","last_undegraded":"2021-03-24T21:53:51.746302+0000","last_fullsized":"2021-03-24T21:53:51.746302+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'14","last_scrub_stamp":"2021-03-24T21:53:02.722970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:02.722970+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":27,"num_read_kb":14,"num_write":14,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"40","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.101014+0000","last_change":"2021-03-24T21:53:12.030422+0000","last_active":"2021-03-24T21:53:51.101014+0000","last_peered":"2021-03-24T21:53:51.101014+0000","last_clean":"2021-03-24T21:53:51.101014+0000","last_became_active":"2021-03-24T21:51:51.762876+0000","last_became_peered":"2021-03-24T21:51:51.762876+0000","last_unstale":"2021-03-24T21:53:51.101014+0000","last_undegraded":"2021-03-24T21:53:51.101014+0000","last_fullsized":"2021-03-24T21:53:51.101014+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:12.030251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:12.030251+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"86'3","reported_seq":"71","reported_epoch":"94","state":"active+clean","last_fresh":"2021-03-24T21:53:54.122873+0000","last_change":"2021-03-24T21:52:56.678231+0000","last_active":"2021-03-24T21:53:54.122873+0000","last_peered":"2021-03-24T21:53:54.122873+0000","last_clean":"2021-03-24T21:53:54.122873+0000","last_became_active":"2021-03-24T21:51:49.419797+0000","last_became_peered":"2021-03-24T21:51:49.419797+0000","last_unstale":"2021-03-24T21:53:54.122873+0000","last_undegraded":"2021-03-24T21:53:54.122873+0000","last_fullsized":"2021-03-24T21:53:54.122873+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'3","last_scrub_stamp":"2021-03-24T21:52:56.678111+0000","last_deep_scrub":"86'3","last_deep_scrub_stamp":"2021-03-24T21:52:56.678111+0000","last_clean_scrub_stamp":"2021-03-24T21:52:56.678111+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":"45","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.097419+0000","last_change":"2021-03-24T21:52:51.450624+0000","last_active":"2021-03-24T21:53:51.097419+0000","last_peered":"2021-03-24T21:53:51.097419+0000","last_clean":"2021-03-24T21:53:51.097419+0000","last_became_active":"2021-03-24T21:51:45.381516+0000","last_became_peered":"2021-03-24T21:51:45.381516+0000","last_unstale":"2021-03-24T21:53:51.097419+0000","last_undegraded":"2021-03-24T21:53:51.097419+0000","last_fullsized":"2021-03-24T21:53:51.097419+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:51.450347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:52:51.450347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"69'13","reported_seq":"69","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.550774+0000","last_change":"2021-03-24T21:53:15.901347+0000","last_active":"2021-03-24T21:53:51.550774+0000","last_peered":"2021-03-24T21:53:51.550774+0000","last_clean":"2021-03-24T21:53:51.550774+0000","last_became_active":"2021-03-24T21:51:47.689288+0000","last_became_peered":"2021-03-24T21:51:47.689288+0000","last_unstale":"2021-03-24T21:53:51.550774+0000","last_undegraded":"2021-03-24T21:53:51.550774+0000","last_fullsized":"2021-03-24T21:53:51.550774+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2021-03-24T21:53:15.901150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:15.901150+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"41","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.243227+0000","last_change":"2021-03-24T21:53:15.623047+0000","last_active":"2021-03-24T21:53:51.243227+0000","last_peered":"2021-03-24T21:53:51.243227+0000","last_clean":"2021-03-24T21:53:51.243227+0000","last_became_active":"2021-03-24T21:51:51.754085+0000","last_became_peered":"2021-03-24T21:51:51.754085+0000","last_unstale":"2021-03-24T21:53:51.243227+0000","last_undegraded":"2021-03-24T21:53:51.243227+0000","last_fullsized":"2021-03-24T21:53:51.243227+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:15.622946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:15.622946+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"42","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.747144+0000","last_change":"2021-03-24T21:52:57.661001+0000","last_active":"2021-03-24T21:53:51.747144+0000","last_peered":"2021-03-24T21:53:51.747144+0000","last_clean":"2021-03-24T21:53:51.747144+0000","last_became_active":"2021-03-24T21:51:49.411882+0000","last_became_peered":"2021-03-24T21:51:49.411882+0000","last_unstale":"2021-03-24T21:53:51.747144+0000","last_undegraded":"2021-03-24T21:53:51.747144+0000","last_fullsized":"2021-03-24T21:53:51.747144+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:57.660827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:52:57.660827+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":"45","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.651358+0000","last_change":"2021-03-24T21:53:12.871443+0000","last_active":"2021-03-24T21:53:51.651358+0000","last_peered":"2021-03-24T21:53:51.651358+0000","last_clean":"2021-03-24T21:53:51.651358+0000","last_became_active":"2021-03-24T21:51:45.378937+0000","last_became_peered":"2021-03-24T21:51:45.378937+0000","last_unstale":"2021-03-24T21:53:51.651358+0000","last_undegraded":"2021-03-24T21:53:51.651358+0000","last_fullsized":"2021-03-24T21:53:51.651358+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:12.871269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:12.871269+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"69'16","reported_seq":"73","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.098156+0000","last_change":"2021-03-24T21:53:00.300847+0000","last_active":"2021-03-24T21:53:51.098156+0000","last_peered":"2021-03-24T21:53:51.098156+0000","last_clean":"2021-03-24T21:53:51.098156+0000","last_became_active":"2021-03-24T21:51:47.687595+0000","last_became_peered":"2021-03-24T21:51:47.687595+0000","last_unstale":"2021-03-24T21:53:51.098156+0000","last_undegraded":"2021-03-24T21:53:51.098156+0000","last_fullsized":"2021-03-24T21:53:51.098156+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T21:53:00.300673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:00.300673+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":38,"num_read_kb":18,"num_write":17,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"39","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.652523+0000","last_change":"2021-03-24T21:53:24.913418+0000","last_active":"2021-03-24T21:53:51.652523+0000","last_peered":"2021-03-24T21:53:51.652523+0000","last_clean":"2021-03-24T21:53:51.652523+0000","last_became_active":"2021-03-24T21:51:51.914615+0000","last_became_peered":"2021-03-24T21:51:51.914615+0000","last_unstale":"2021-03-24T21:53:51.652523+0000","last_undegraded":"2021-03-24T21:53:51.652523+0000","last_fullsized":"2021-03-24T21:53:51.652523+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:24.913238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:24.913238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"43","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.550335+0000","last_change":"2021-03-24T21:53:19.812985+0000","last_active":"2021-03-24T21:53:51.550335+0000","last_peered":"2021-03-24T21:53:51.550335+0000","last_clean":"2021-03-24T21:53:51.550335+0000","last_became_active":"2021-03-24T21:51:49.423159+0000","last_became_peered":"2021-03-24T21:51:49.423159+0000","last_unstale":"2021-03-24T21:53:51.550335+0000","last_undegraded":"2021-03-24T21:53:51.550335+0000","last_fullsized":"2021-03-24T21:53:51.550335+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:19.812822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:19.812822+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":"46","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.743643+0000","last_change":"2021-03-24T21:53:02.100432+0000","last_active":"2021-03-24T21:53:51.743643+0000","last_peered":"2021-03-24T21:53:51.743643+0000","last_clean":"2021-03-24T21:53:51.743643+0000","last_became_active":"2021-03-24T21:51:45.377984+0000","last_became_peered":"2021-03-24T21:51:45.377984+0000","last_unstale":"2021-03-24T21:53:51.743643+0000","last_undegraded":"2021-03-24T21:53:51.743643+0000","last_fullsized":"2021-03-24T21:53:51.743643+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:02.100243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:02.100243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"69'7","reported_seq":"55","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.746975+0000","last_change":"2021-03-24T21:53:13.646018+0000","last_active":"2021-03-24T21:53:51.746975+0000","last_peered":"2021-03-24T21:53:51.746975+0000","last_clean":"2021-03-24T21:53:51.746975+0000","last_became_active":"2021-03-24T21:51:47.687445+0000","last_became_peered":"2021-03-24T21:51:47.687445+0000","last_unstale":"2021-03-24T21:53:51.746975+0000","last_undegraded":"2021-03-24T21:53:51.746975+0000","last_fullsized":"2021-03-24T21:53:51.746975+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'7","last_scrub_stamp":"2021-03-24T21:53:13.645928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:13.645928+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":7,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"41","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.549848+0000","last_change":"2021-03-24T21:53:09.854407+0000","last_active":"2021-03-24T21:53:51.549848+0000","last_peered":"2021-03-24T21:53:51.549848+0000","last_clean":"2021-03-24T21:53:51.549848+0000","last_became_active":"2021-03-24T21:51:51.913733+0000","last_became_peered":"2021-03-24T21:51:51.913733+0000","last_unstale":"2021-03-24T21:53:51.549848+0000","last_undegraded":"2021-03-24T21:53:51.549848+0000","last_fullsized":"2021-03-24T21:53:51.549848+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:09.854304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:09.854304+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"89'4","reported_seq":"50","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.653904+0000","last_change":"2021-03-24T21:53:19.926732+0000","last_active":"2021-03-24T21:53:51.653904+0000","last_peered":"2021-03-24T21:53:51.653904+0000","last_clean":"2021-03-24T21:53:51.653904+0000","last_became_active":"2021-03-24T21:51:45.385110+0000","last_became_peered":"2021-03-24T21:51:45.385110+0000","last_unstale":"2021-03-24T21:53:51.653904+0000","last_undegraded":"2021-03-24T21:53:51.653904+0000","last_fullsized":"2021-03-24T21:53:51.653904+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'4","last_scrub_stamp":"2021-03-24T21:53:19.926637+0000","last_deep_scrub":"89'4","last_deep_scrub_stamp":"2021-03-24T21:53:19.926637+0000","last_clean_scrub_stamp":"2021-03-24T21:53:19.926637+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"69'18","reported_seq":"76","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.549133+0000","last_change":"2021-03-24T21:52:57.861464+0000","last_active":"2021-03-24T21:53:51.549133+0000","last_peered":"2021-03-24T21:53:51.549133+0000","last_clean":"2021-03-24T21:53:51.549133+0000","last_became_active":"2021-03-24T21:51:47.685994+0000","last_became_peered":"2021-03-24T21:51:47.685994+0000","last_unstale":"2021-03-24T21:53:51.549133+0000","last_undegraded":"2021-03-24T21:53:51.549133+0000","last_fullsized":"2021-03-24T21:53:51.549133+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'18","last_scrub_stamp":"2021-03-24T21:52:57.861253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:52:57.861253+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":20,"num_write":19,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"65'2","reported_seq":"69","reported_epoch":"94","state":"active+clean","last_fresh":"2021-03-24T21:53:55.069931+0000","last_change":"2021-03-24T21:53:03.836648+0000","last_active":"2021-03-24T21:53:55.069931+0000","last_peered":"2021-03-24T21:53:55.069931+0000","last_clean":"2021-03-24T21:53:55.069931+0000","last_became_active":"2021-03-24T21:51:49.415167+0000","last_became_peered":"2021-03-24T21:51:49.415167+0000","last_unstale":"2021-03-24T21:53:55.069931+0000","last_undegraded":"2021-03-24T21:53:55.069931+0000","last_fullsized":"2021-03-24T21:53:55.069931+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2021-03-24T21:53:03.836463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:03.836463+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"40","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.745032+0000","last_change":"2021-03-24T21:53:22.725035+0000","last_active":"2021-03-24T21:53:51.745032+0000","last_peered":"2021-03-24T21:53:51.745032+0000","last_clean":"2021-03-24T21:53:51.745032+0000","last_became_active":"2021-03-24T21:51:51.762035+0000","last_became_peered":"2021-03-24T21:51:51.762035+0000","last_unstale":"2021-03-24T21:53:51.745032+0000","last_undegraded":"2021-03-24T21:53:51.745032+0000","last_fullsized":"2021-03-24T21:53:51.745032+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:22.724840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:22.724840+0000","last_clean_scrub_stamp":"2021-03-24T21:53:22.724840+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":"50","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.653745+0000","last_change":"2021-03-24T21:53:51.651338+0000","last_active":"2021-03-24T21:53:51.653745+0000","last_peered":"2021-03-24T21:53:51.653745+0000","last_clean":"2021-03-24T21:53:51.653745+0000","last_became_active":"2021-03-24T21:51:45.380984+0000","last_became_peered":"2021-03-24T21:51:45.380984+0000","last_unstale":"2021-03-24T21:53:51.653745+0000","last_undegraded":"2021-03-24T21:53:51.653745+0000","last_fullsized":"2021-03-24T21:53:51.653745+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:51.651164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:51.651164+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"69'15","reported_seq":"73","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.098982+0000","last_change":"2021-03-24T21:53:37.334527+0000","last_active":"2021-03-24T21:53:51.098982+0000","last_peered":"2021-03-24T21:53:51.098982+0000","last_clean":"2021-03-24T21:53:51.098982+0000","last_became_active":"2021-03-24T21:51:47.410650+0000","last_became_peered":"2021-03-24T21:51:47.410650+0000","last_unstale":"2021-03-24T21:53:51.098982+0000","last_undegraded":"2021-03-24T21:53:51.098982+0000","last_fullsized":"2021-03-24T21:53:51.098982+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:53:37.334345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:37.334345+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":19,"num_write":17,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"43","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.242372+0000","last_change":"2021-03-24T21:53:18.557275+0000","last_active":"2021-03-24T21:53:51.242372+0000","last_peered":"2021-03-24T21:53:51.242372+0000","last_clean":"2021-03-24T21:53:51.242372+0000","last_became_active":"2021-03-24T21:51:49.415403+0000","last_became_peered":"2021-03-24T21:51:49.415403+0000","last_unstale":"2021-03-24T21:53:51.242372+0000","last_undegraded":"2021-03-24T21:53:51.242372+0000","last_fullsized":"2021-03-24T21:53:51.242372+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:18.557122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:18.557122+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"39","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.650958+0000","last_change":"2021-03-24T21:53:02.842718+0000","last_active":"2021-03-24T21:53:51.650958+0000","last_peered":"2021-03-24T21:53:51.650958+0000","last_clean":"2021-03-24T21:53:51.650958+0000","last_became_active":"2021-03-24T21:51:51.762513+0000","last_became_peered":"2021-03-24T21:51:51.762513+0000","last_unstale":"2021-03-24T21:53:51.650958+0000","last_undegraded":"2021-03-24T21:53:51.650958+0000","last_fullsized":"2021-03-24T21:53:51.650958+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:02.842537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:02.842537+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"43","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.242696+0000","last_change":"2021-03-24T21:52:59.521026+0000","last_active":"2021-03-24T21:53:51.242696+0000","last_peered":"2021-03-24T21:53:51.242696+0000","last_clean":"2021-03-24T21:53:51.242696+0000","last_became_active":"2021-03-24T21:51:49.417699+0000","last_became_peered":"2021-03-24T21:51:49.417699+0000","last_unstale":"2021-03-24T21:53:51.242696+0000","last_undegraded":"2021-03-24T21:53:51.242696+0000","last_fullsized":"2021-03-24T21:53:51.242696+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:59.520858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:52:59.520858+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":"45","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.653352+0000","last_change":"2021-03-24T21:53:26.908991+0000","last_active":"2021-03-24T21:53:51.653352+0000","last_peered":"2021-03-24T21:53:51.653352+0000","last_clean":"2021-03-24T21:53:51.653352+0000","last_became_active":"2021-03-24T21:51:45.384545+0000","last_became_peered":"2021-03-24T21:51:45.384545+0000","last_unstale":"2021-03-24T21:53:51.653352+0000","last_undegraded":"2021-03-24T21:53:51.653352+0000","last_fullsized":"2021-03-24T21:53:51.653352+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:26.908821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:26.908821+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"69'23","reported_seq":"83","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.744687+0000","last_change":"2021-03-24T21:52:47.126913+0000","last_active":"2021-03-24T21:53:51.744687+0000","last_peered":"2021-03-24T21:53:51.744687+0000","last_clean":"2021-03-24T21:53:51.744687+0000","last_became_active":"2021-03-24T21:51:47.685830+0000","last_became_peered":"2021-03-24T21:51:47.685830+0000","last_unstale":"2021-03-24T21:53:51.744687+0000","last_undegraded":"2021-03-24T21:53:51.744687+0000","last_fullsized":"2021-03-24T21:53:51.744687+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'23","last_scrub_stamp":"2021-03-24T21:52:47.126826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:52:47.126826+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":44,"num_read_kb":25,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"39","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.096442+0000","last_change":"2021-03-24T21:53:35.636404+0000","last_active":"2021-03-24T21:53:51.096442+0000","last_peered":"2021-03-24T21:53:51.096442+0000","last_clean":"2021-03-24T21:53:51.096442+0000","last_became_active":"2021-03-24T21:51:51.763077+0000","last_became_peered":"2021-03-24T21:51:51.763077+0000","last_unstale":"2021-03-24T21:53:51.096442+0000","last_undegraded":"2021-03-24T21:53:51.096442+0000","last_fullsized":"2021-03-24T21:53:51.096442+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:35.636299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:35.636299+0000","last_clean_scrub_stamp":"2021-03-24T21:53:35.636299+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"43","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.242799+0000","last_change":"2021-03-24T21:53:11.623536+0000","last_active":"2021-03-24T21:53:51.242799+0000","last_peered":"2021-03-24T21:53:51.242799+0000","last_clean":"2021-03-24T21:53:51.242799+0000","last_became_active":"2021-03-24T21:51:49.411921+0000","last_became_peered":"2021-03-24T21:51:49.411921+0000","last_unstale":"2021-03-24T21:53:51.242799+0000","last_undegraded":"2021-03-24T21:53:51.242799+0000","last_fullsized":"2021-03-24T21:53:51.242799+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:11.623353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:11.623353+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"47","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.549824+0000","last_change":"2021-03-24T21:52:47.709622+0000","last_active":"2021-03-24T21:53:51.549824+0000","last_peered":"2021-03-24T21:53:51.549824+0000","last_clean":"2021-03-24T21:53:51.549824+0000","last_became_active":"2021-03-24T21:51:45.383017+0000","last_became_peered":"2021-03-24T21:51:45.383017+0000","last_unstale":"2021-03-24T21:53:51.549824+0000","last_undegraded":"2021-03-24T21:53:51.549824+0000","last_fullsized":"2021-03-24T21:53:51.549824+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:47.709437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:52:47.709437+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"69'14","reported_seq":"69","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.098185+0000","last_change":"2021-03-24T21:53:31.653478+0000","last_active":"2021-03-24T21:53:51.098185+0000","last_peered":"2021-03-24T21:53:51.098185+0000","last_clean":"2021-03-24T21:53:51.098185+0000","last_became_active":"2021-03-24T21:51:47.408588+0000","last_became_peered":"2021-03-24T21:51:47.408588+0000","last_unstale":"2021-03-24T21:53:51.098185+0000","last_undegraded":"2021-03-24T21:53:51.098185+0000","last_fullsized":"2021-03-24T21:53:51.098185+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'14","last_scrub_stamp":"2021-03-24T21:53:31.653334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:31.653334+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":16,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"40","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.746242+0000","last_change":"2021-03-24T21:52:59.706737+0000","last_active":"2021-03-24T21:53:51.746242+0000","last_peered":"2021-03-24T21:53:51.746242+0000","last_clean":"2021-03-24T21:53:51.746242+0000","last_became_active":"2021-03-24T21:51:51.759514+0000","last_became_peered":"2021-03-24T21:51:51.759514+0000","last_unstale":"2021-03-24T21:53:51.746242+0000","last_undegraded":"2021-03-24T21:53:51.746242+0000","last_fullsized":"2021-03-24T21:53:51.746242+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:59.706511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:52:59.706511+0000","last_clean_scrub_stamp":"2021-03-24T21:52:59.706511+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"51","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.549448+0000","last_change":"2021-03-24T21:53:06.800587+0000","last_active":"2021-03-24T21:53:51.549448+0000","last_peered":"2021-03-24T21:53:51.549448+0000","last_clean":"2021-03-24T21:53:51.549448+0000","last_became_active":"2021-03-24T21:51:40.350011+0000","last_became_peered":"2021-03-24T21:51:40.350011+0000","last_unstale":"2021-03-24T21:53:51.549448+0000","last_undegraded":"2021-03-24T21:53:51.549448+0000","last_fullsized":"2021-03-24T21:53:51.549448+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:06.800417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:06.800417+0000","last_clean_scrub_stamp":"2021-03-24T21:53:06.800417+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"89'2","reported_seq":"49","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.099230+0000","last_change":"2021-03-24T21:53:28.220091+0000","last_active":"2021-03-24T21:53:51.099230+0000","last_peered":"2021-03-24T21:53:51.099230+0000","last_clean":"2021-03-24T21:53:51.099230+0000","last_became_active":"2021-03-24T21:51:45.375765+0000","last_became_peered":"2021-03-24T21:51:45.375765+0000","last_unstale":"2021-03-24T21:53:51.099230+0000","last_undegraded":"2021-03-24T21:53:51.099230+0000","last_fullsized":"2021-03-24T21:53:51.099230+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2021-03-24T21:53:28.219900+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2021-03-24T21:53:28.219900+0000","last_clean_scrub_stamp":"2021-03-24T21:53:28.219900+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"69'25","reported_seq":"92","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.102258+0000","last_change":"2021-03-24T21:53:15.719556+0000","last_active":"2021-03-24T21:53:51.102258+0000","last_peered":"2021-03-24T21:53:51.102258+0000","last_clean":"2021-03-24T21:53:51.102258+0000","last_became_active":"2021-03-24T21:51:47.411647+0000","last_became_peered":"2021-03-24T21:51:47.411647+0000","last_unstale":"2021-03-24T21:53:51.102258+0000","last_undegraded":"2021-03-24T21:53:51.102258+0000","last_fullsized":"2021-03-24T21:53:51.102258+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'25","last_scrub_stamp":"2021-03-24T21:53:15.719434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:15.719434+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":54,"num_read_kb":31,"num_write":28,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"41","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.097635+0000","last_change":"2021-03-24T21:53:23.578854+0000","last_active":"2021-03-24T21:53:51.097635+0000","last_peered":"2021-03-24T21:53:51.097635+0000","last_clean":"2021-03-24T21:53:51.097635+0000","last_became_active":"2021-03-24T21:51:49.421253+0000","last_became_peered":"2021-03-24T21:51:49.421253+0000","last_unstale":"2021-03-24T21:53:51.097635+0000","last_undegraded":"2021-03-24T21:53:51.097635+0000","last_fullsized":"2021-03-24T21:53:51.097635+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:23.578679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:23.578679+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"39","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.653495+0000","last_change":"2021-03-24T21:53:23.927068+0000","last_active":"2021-03-24T21:53:51.653495+0000","last_peered":"2021-03-24T21:53:51.653495+0000","last_clean":"2021-03-24T21:53:51.653495+0000","last_became_active":"2021-03-24T21:51:51.912075+0000","last_became_peered":"2021-03-24T21:51:51.912075+0000","last_unstale":"2021-03-24T21:53:51.653495+0000","last_undegraded":"2021-03-24T21:53:51.653495+0000","last_fullsized":"2021-03-24T21:53:51.653495+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:23.926884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:23.926884+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"61'1","reported_seq":"50","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.100473+0000","last_change":"2021-03-24T21:53:18.753108+0000","last_active":"2021-03-24T21:53:51.100473+0000","last_peered":"2021-03-24T21:53:51.100473+0000","last_clean":"2021-03-24T21:53:51.100473+0000","last_became_active":"2021-03-24T21:51:45.580337+0000","last_became_peered":"2021-03-24T21:51:45.580337+0000","last_unstale":"2021-03-24T21:53:51.100473+0000","last_undegraded":"2021-03-24T21:53:51.100473+0000","last_fullsized":"2021-03-24T21:53:51.100473+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2021-03-24T21:53:18.753026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:18.753026+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"56","reported_epoch":"94","state":"active+clean","last_fresh":"2021-03-24T21:53:55.497565+0000","last_change":"2021-03-24T21:53:55.497565+0000","last_active":"2021-03-24T21:53:55.497565+0000","last_peered":"2021-03-24T21:53:55.497565+0000","last_clean":"2021-03-24T21:53:55.497565+0000","last_became_active":"2021-03-24T21:51:40.351000+0000","last_became_peered":"2021-03-24T21:51:40.351000+0000","last_unstale":"2021-03-24T21:53:55.497565+0000","last_undegraded":"2021-03-24T21:53:55.497565+0000","last_fullsized":"2021-03-24T21:53:55.497565+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:55.497382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:39.286101+0000","last_clean_scrub_stamp":"2021-03-24T21:53:55.497382+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"89'22","reported_seq":"83","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.103132+0000","last_change":"2021-03-24T21:53:19.717364+0000","last_active":"2021-03-24T21:53:51.103132+0000","last_peered":"2021-03-24T21:53:51.103132+0000","last_clean":"2021-03-24T21:53:51.103132+0000","last_became_active":"2021-03-24T21:51:47.411696+0000","last_became_peered":"2021-03-24T21:51:47.411696+0000","last_unstale":"2021-03-24T21:53:51.103132+0000","last_undegraded":"2021-03-24T21:53:51.103132+0000","last_fullsized":"2021-03-24T21:53:51.103132+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'22","last_scrub_stamp":"2021-03-24T21:53:19.717244+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2021-03-24T21:53:19.717244+0000","last_clean_scrub_stamp":"2021-03-24T21:53:19.717244+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":36,"num_read_kb":21,"num_write":18,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"43","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.243292+0000","last_change":"2021-03-24T21:53:21.534119+0000","last_active":"2021-03-24T21:53:51.243292+0000","last_peered":"2021-03-24T21:53:51.243292+0000","last_clean":"2021-03-24T21:53:51.243292+0000","last_became_active":"2021-03-24T21:51:49.418848+0000","last_became_peered":"2021-03-24T21:51:49.418848+0000","last_unstale":"2021-03-24T21:53:51.243292+0000","last_undegraded":"2021-03-24T21:53:51.243292+0000","last_fullsized":"2021-03-24T21:53:51.243292+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:21.533882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:21.533882+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"41","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.550400+0000","last_change":"2021-03-24T21:53:01.832920+0000","last_active":"2021-03-24T21:53:51.550400+0000","last_peered":"2021-03-24T21:53:51.550400+0000","last_clean":"2021-03-24T21:53:51.550400+0000","last_became_active":"2021-03-24T21:51:51.765236+0000","last_became_peered":"2021-03-24T21:51:51.765236+0000","last_unstale":"2021-03-24T21:53:51.550400+0000","last_undegraded":"2021-03-24T21:53:51.550400+0000","last_fullsized":"2021-03-24T21:53:51.550400+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:01.832740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:01.832740+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"55","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.651060+0000","last_change":"2021-03-24T21:53:30.906666+0000","last_active":"2021-03-24T21:53:51.651060+0000","last_peered":"2021-03-24T21:53:51.651060+0000","last_clean":"2021-03-24T21:53:51.651060+0000","last_became_active":"2021-03-24T21:51:40.349967+0000","last_became_peered":"2021-03-24T21:51:40.349967+0000","last_unstale":"2021-03-24T21:53:51.651060+0000","last_undegraded":"2021-03-24T21:53:51.651060+0000","last_fullsized":"2021-03-24T21:53:51.651060+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:30.906502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:49:51.529051+0000","last_clean_scrub_stamp":"2021-03-24T21:53:30.906502+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"46","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.745126+0000","last_change":"2021-03-24T21:53:20.736644+0000","last_active":"2021-03-24T21:53:51.745126+0000","last_peered":"2021-03-24T21:53:51.745126+0000","last_clean":"2021-03-24T21:53:51.745126+0000","last_became_active":"2021-03-24T21:51:45.383779+0000","last_became_peered":"2021-03-24T21:51:45.383779+0000","last_unstale":"2021-03-24T21:53:51.745126+0000","last_undegraded":"2021-03-24T21:53:51.745126+0000","last_fullsized":"2021-03-24T21:53:51.745126+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:20.736432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:20.736432+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"69'15","reported_seq":"68","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.745180+0000","last_change":"2021-03-24T21:53:20.279112+0000","last_active":"2021-03-24T21:53:51.745180+0000","last_peered":"2021-03-24T21:53:51.745180+0000","last_clean":"2021-03-24T21:53:51.745180+0000","last_became_active":"2021-03-24T21:51:47.688873+0000","last_became_peered":"2021-03-24T21:51:47.688873+0000","last_unstale":"2021-03-24T21:53:51.745180+0000","last_undegraded":"2021-03-24T21:53:51.745180+0000","last_fullsized":"2021-03-24T21:53:51.745180+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:53:20.279004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:20.279004+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":27,"num_read_kb":15,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"41","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.652905+0000","last_change":"2021-03-24T21:53:09.874798+0000","last_active":"2021-03-24T21:53:51.652905+0000","last_peered":"2021-03-24T21:53:51.652905+0000","last_clean":"2021-03-24T21:53:51.652905+0000","last_became_active":"2021-03-24T21:51:49.413251+0000","last_became_peered":"2021-03-24T21:51:49.413251+0000","last_unstale":"2021-03-24T21:53:51.652905+0000","last_undegraded":"2021-03-24T21:53:51.652905+0000","last_fullsized":"2021-03-24T21:53:51.652905+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:09.874681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:09.874681+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"41","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.102098+0000","last_change":"2021-03-24T21:53:11.723900+0000","last_active":"2021-03-24T21:53:51.102098+0000","last_peered":"2021-03-24T21:53:51.102098+0000","last_clean":"2021-03-24T21:53:51.102098+0000","last_became_active":"2021-03-24T21:51:51.762039+0000","last_became_peered":"2021-03-24T21:51:51.762039+0000","last_unstale":"2021-03-24T21:53:51.102098+0000","last_undegraded":"2021-03-24T21:53:51.102098+0000","last_fullsized":"2021-03-24T21:53:51.102098+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:11.723709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:11.723709+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"45","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.097498+0000","last_change":"2021-03-24T21:52:48.487965+0000","last_active":"2021-03-24T21:53:51.097498+0000","last_peered":"2021-03-24T21:53:51.097498+0000","last_clean":"2021-03-24T21:53:51.097498+0000","last_became_active":"2021-03-24T21:51:45.376687+0000","last_became_peered":"2021-03-24T21:51:45.376687+0000","last_unstale":"2021-03-24T21:53:51.097498+0000","last_undegraded":"2021-03-24T21:53:51.097498+0000","last_fullsized":"2021-03-24T21:53:51.097498+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:48.487787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:52:48.487787+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"66'5","reported_seq":"185","reported_epoch":"94","state":"active+clean","last_fresh":"2021-03-24T21:53:57.349383+0000","last_change":"2021-03-24T21:52:50.807770+0000","last_active":"2021-03-24T21:53:57.349383+0000","last_peered":"2021-03-24T21:53:57.349383+0000","last_clean":"2021-03-24T21:53:57.349383+0000","last_became_active":"2021-03-24T21:51:40.350525+0000","last_became_peered":"2021-03-24T21:51:40.350525+0000","last_unstale":"2021-03-24T21:53:57.349383+0000","last_undegraded":"2021-03-24T21:53:57.349383+0000","last_fullsized":"2021-03-24T21:53:57.349383+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2021-03-24T21:52:50.807568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:39.286101+0000","last_clean_scrub_stamp":"2021-03-24T21:52:50.807568+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":133,"num_read_kb":128,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"69'16","reported_seq":"69","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.097806+0000","last_change":"2021-03-24T21:53:15.485180+0000","last_active":"2021-03-24T21:53:51.097806+0000","last_peered":"2021-03-24T21:53:51.097806+0000","last_clean":"2021-03-24T21:53:51.097806+0000","last_became_active":"2021-03-24T21:51:47.411442+0000","last_became_peered":"2021-03-24T21:51:47.411442+0000","last_unstale":"2021-03-24T21:53:51.097806+0000","last_undegraded":"2021-03-24T21:53:51.097806+0000","last_fullsized":"2021-03-24T21:53:51.097806+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T21:53:15.485072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:15.485072+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":16,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"43","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.550993+0000","last_change":"2021-03-24T21:53:20.836086+0000","last_active":"2021-03-24T21:53:51.550993+0000","last_peered":"2021-03-24T21:53:51.550993+0000","last_clean":"2021-03-24T21:53:51.550993+0000","last_became_active":"2021-03-24T21:51:49.419167+0000","last_became_peered":"2021-03-24T21:51:49.419167+0000","last_unstale":"2021-03-24T21:53:51.550993+0000","last_undegraded":"2021-03-24T21:53:51.550993+0000","last_fullsized":"2021-03-24T21:53:51.550993+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:20.835990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:20.835990+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"41","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.101704+0000","last_change":"2021-03-24T21:53:28.691448+0000","last_active":"2021-03-24T21:53:51.101704+0000","last_peered":"2021-03-24T21:53:51.101704+0000","last_clean":"2021-03-24T21:53:51.101704+0000","last_became_active":"2021-03-24T21:51:51.913581+0000","last_became_peered":"2021-03-24T21:51:51.913581+0000","last_unstale":"2021-03-24T21:53:51.101704+0000","last_undegraded":"2021-03-24T21:53:51.101704+0000","last_fullsized":"2021-03-24T21:53:51.101704+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:28.691270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:28.691270+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"47","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.102244+0000","last_change":"2021-03-24T21:53:08.691716+0000","last_active":"2021-03-24T21:53:51.102244+0000","last_peered":"2021-03-24T21:53:51.102244+0000","last_clean":"2021-03-24T21:53:51.102244+0000","last_became_active":"2021-03-24T21:51:45.580531+0000","last_became_peered":"2021-03-24T21:51:45.580531+0000","last_unstale":"2021-03-24T21:53:51.102244+0000","last_undegraded":"2021-03-24T21:53:51.102244+0000","last_fullsized":"2021-03-24T21:53:51.102244+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:08.691536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:08.691536+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"69'21","reported_seq":"83","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.098227+0000","last_change":"2021-03-24T21:53:05.313409+0000","last_active":"2021-03-24T21:53:51.098227+0000","last_peered":"2021-03-24T21:53:51.098227+0000","last_clean":"2021-03-24T21:53:51.098227+0000","last_became_active":"2021-03-24T21:51:47.686486+0000","last_became_peered":"2021-03-24T21:51:47.686486+0000","last_unstale":"2021-03-24T21:53:51.098227+0000","last_undegraded":"2021-03-24T21:53:51.098227+0000","last_fullsized":"2021-03-24T21:53:51.098227+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'21","last_scrub_stamp":"2021-03-24T21:53:05.313235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:05.313235+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":45,"num_read_kb":27,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"42","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.744790+0000","last_change":"2021-03-24T21:53:23.246488+0000","last_active":"2021-03-24T21:53:51.744790+0000","last_peered":"2021-03-24T21:53:51.744790+0000","last_clean":"2021-03-24T21:53:51.744790+0000","last_became_active":"2021-03-24T21:51:49.416847+0000","last_became_peered":"2021-03-24T21:51:49.416847+0000","last_unstale":"2021-03-24T21:53:51.744790+0000","last_undegraded":"2021-03-24T21:53:51.744790+0000","last_fullsized":"2021-03-24T21:53:51.744790+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:23.246320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:23.246320+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"39","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.096441+0000","last_change":"2021-03-24T21:53:25.593543+0000","last_active":"2021-03-24T21:53:51.096441+0000","last_peered":"2021-03-24T21:53:51.096441+0000","last_clean":"2021-03-24T21:53:51.096441+0000","last_became_active":"2021-03-24T21:51:51.759627+0000","last_became_peered":"2021-03-24T21:51:51.759627+0000","last_unstale":"2021-03-24T21:53:51.096441+0000","last_undegraded":"2021-03-24T21:53:51.096441+0000","last_fullsized":"2021-03-24T21:53:51.096441+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:25.593358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:25.593358+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"47","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.550554+0000","last_change":"2021-03-24T21:53:04.837752+0000","last_active":"2021-03-24T21:53:51.550554+0000","last_peered":"2021-03-24T21:53:51.550554+0000","last_clean":"2021-03-24T21:53:51.550554+0000","last_became_active":"2021-03-24T21:51:45.583885+0000","last_became_peered":"2021-03-24T21:51:45.583885+0000","last_unstale":"2021-03-24T21:53:51.550554+0000","last_undegraded":"2021-03-24T21:53:51.550554+0000","last_fullsized":"2021-03-24T21:53:51.550554+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:04.837488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:04.837488+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"69'16","reported_seq":"72","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.101470+0000","last_change":"2021-03-24T21:53:03.595733+0000","last_active":"2021-03-24T21:53:51.101470+0000","last_peered":"2021-03-24T21:53:51.101470+0000","last_clean":"2021-03-24T21:53:51.101470+0000","last_became_active":"2021-03-24T21:51:47.403006+0000","last_became_peered":"2021-03-24T21:51:47.403006+0000","last_unstale":"2021-03-24T21:53:51.101470+0000","last_undegraded":"2021-03-24T21:53:51.101470+0000","last_fullsized":"2021-03-24T21:53:51.101470+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T21:53:03.595585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:03.595585+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"89'3","reported_seq":"70","reported_epoch":"94","state":"active+clean","last_fresh":"2021-03-24T21:53:55.067734+0000","last_change":"2021-03-24T21:53:26.584866+0000","last_active":"2021-03-24T21:53:55.067734+0000","last_peered":"2021-03-24T21:53:55.067734+0000","last_clean":"2021-03-24T21:53:55.067734+0000","last_became_active":"2021-03-24T21:51:49.420173+0000","last_became_peered":"2021-03-24T21:51:49.420173+0000","last_unstale":"2021-03-24T21:53:55.067734+0000","last_undegraded":"2021-03-24T21:53:55.067734+0000","last_fullsized":"2021-03-24T21:53:55.067734+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'3","last_scrub_stamp":"2021-03-24T21:53:26.584678+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-24T21:53:26.584678+0000","last_clean_scrub_stamp":"2021-03-24T21:53:26.584678+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"39","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.652234+0000","last_change":"2021-03-24T21:52:55.828993+0000","last_active":"2021-03-24T21:53:51.652234+0000","last_peered":"2021-03-24T21:53:51.652234+0000","last_clean":"2021-03-24T21:53:51.652234+0000","last_became_active":"2021-03-24T21:51:51.914601+0000","last_became_peered":"2021-03-24T21:51:51.914601+0000","last_unstale":"2021-03-24T21:53:51.652234+0000","last_undegraded":"2021-03-24T21:53:51.652234+0000","last_fullsized":"2021-03-24T21:53:51.652234+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:55.828810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:52:55.828810+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"45","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.097917+0000","last_change":"2021-03-24T21:53:24.621910+0000","last_active":"2021-03-24T21:53:51.097917+0000","last_peered":"2021-03-24T21:53:51.097917+0000","last_clean":"2021-03-24T21:53:51.097917+0000","last_became_active":"2021-03-24T21:51:45.375747+0000","last_became_peered":"2021-03-24T21:51:45.375747+0000","last_unstale":"2021-03-24T21:53:51.097917+0000","last_undegraded":"2021-03-24T21:53:51.097917+0000","last_fullsized":"2021-03-24T21:53:51.097917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:24.621749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:24.621749+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"69'20","reported_seq":"77","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.098841+0000","last_change":"2021-03-24T21:53:13.077033+0000","last_active":"2021-03-24T21:53:51.098841+0000","last_peered":"2021-03-24T21:53:51.098841+0000","last_clean":"2021-03-24T21:53:51.098841+0000","last_became_active":"2021-03-24T21:51:47.404713+0000","last_became_peered":"2021-03-24T21:51:47.404713+0000","last_unstale":"2021-03-24T21:53:51.098841+0000","last_undegraded":"2021-03-24T21:53:51.098841+0000","last_fullsized":"2021-03-24T21:53:51.098841+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'20","last_scrub_stamp":"2021-03-24T21:53:13.076772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:13.076772+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":39,"num_read_kb":20,"num_write":20,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"65'2","reported_seq":"70","reported_epoch":"94","state":"active+clean","last_fresh":"2021-03-24T21:53:54.121129+0000","last_change":"2021-03-24T21:53:04.714525+0000","last_active":"2021-03-24T21:53:54.121129+0000","last_peered":"2021-03-24T21:53:54.121129+0000","last_clean":"2021-03-24T21:53:54.121129+0000","last_became_active":"2021-03-24T21:51:49.421628+0000","last_became_peered":"2021-03-24T21:51:49.421628+0000","last_unstale":"2021-03-24T21:53:54.121129+0000","last_undegraded":"2021-03-24T21:53:54.121129+0000","last_fullsized":"2021-03-24T21:53:54.121129+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2021-03-24T21:53:04.714348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:04.714348+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"40","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.099337+0000","last_change":"2021-03-24T21:53:45.349966+0000","last_active":"2021-03-24T21:53:51.099337+0000","last_peered":"2021-03-24T21:53:51.099337+0000","last_clean":"2021-03-24T21:53:51.099337+0000","last_became_active":"2021-03-24T21:51:51.756114+0000","last_became_peered":"2021-03-24T21:51:51.756114+0000","last_unstale":"2021-03-24T21:53:51.099337+0000","last_undegraded":"2021-03-24T21:53:51.099337+0000","last_fullsized":"2021-03-24T21:53:51.099337+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:45.349781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:45.349781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"46","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.746746+0000","last_change":"2021-03-24T21:53:00.682889+0000","last_active":"2021-03-24T21:53:51.746746+0000","last_peered":"2021-03-24T21:53:51.746746+0000","last_clean":"2021-03-24T21:53:51.746746+0000","last_became_active":"2021-03-24T21:51:45.378704+0000","last_became_peered":"2021-03-24T21:51:45.378704+0000","last_unstale":"2021-03-24T21:53:51.746746+0000","last_undegraded":"2021-03-24T21:53:51.746746+0000","last_fullsized":"2021-03-24T21:53:51.746746+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:00.682713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:00.682713+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"69'22","reported_seq":"84","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.744340+0000","last_change":"2021-03-24T21:53:17.716726+0000","last_active":"2021-03-24T21:53:51.744340+0000","last_peered":"2021-03-24T21:53:51.744340+0000","last_clean":"2021-03-24T21:53:51.744340+0000","last_became_active":"2021-03-24T21:51:47.691459+0000","last_became_peered":"2021-03-24T21:51:47.691459+0000","last_unstale":"2021-03-24T21:53:51.744340+0000","last_undegraded":"2021-03-24T21:53:51.744340+0000","last_fullsized":"2021-03-24T21:53:51.744340+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'22","last_scrub_stamp":"2021-03-24T21:53:17.716470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:17.716470+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":46,"num_read_kb":26,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"42","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.098326+0000","last_change":"2021-03-24T21:53:25.220598+0000","last_active":"2021-03-24T21:53:51.098326+0000","last_peered":"2021-03-24T21:53:51.098326+0000","last_clean":"2021-03-24T21:53:51.098326+0000","last_became_active":"2021-03-24T21:51:49.408761+0000","last_became_peered":"2021-03-24T21:51:49.408761+0000","last_unstale":"2021-03-24T21:53:51.098326+0000","last_undegraded":"2021-03-24T21:53:51.098326+0000","last_fullsized":"2021-03-24T21:53:51.098326+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:25.220436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:25.220436+0000","last_clean_scrub_stamp":"2021-03-24T21:53:25.220436+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"41","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.550050+0000","last_change":"2021-03-24T21:53:11.925302+0000","last_active":"2021-03-24T21:53:51.550050+0000","last_peered":"2021-03-24T21:53:51.550050+0000","last_clean":"2021-03-24T21:53:51.550050+0000","last_became_active":"2021-03-24T21:51:51.760369+0000","last_became_peered":"2021-03-24T21:51:51.760369+0000","last_unstale":"2021-03-24T21:53:51.550050+0000","last_undegraded":"2021-03-24T21:53:51.550050+0000","last_fullsized":"2021-03-24T21:53:51.550050+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:11.925149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:11.925149+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"65'2","reported_seq":"69","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:53.817549+0000","last_change":"2021-03-24T21:53:29.268452+0000","last_active":"2021-03-24T21:53:53.817549+0000","last_peered":"2021-03-24T21:53:53.817549+0000","last_clean":"2021-03-24T21:53:53.817549+0000","last_became_active":"2021-03-24T21:51:49.414380+0000","last_became_peered":"2021-03-24T21:51:49.414380+0000","last_unstale":"2021-03-24T21:53:53.817549+0000","last_undegraded":"2021-03-24T21:53:53.817549+0000","last_fullsized":"2021-03-24T21:53:53.817549+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2021-03-24T21:53:29.268273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:29.268273+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":"46","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.744271+0000","last_change":"2021-03-24T21:53:35.609332+0000","last_active":"2021-03-24T21:53:51.744271+0000","last_peered":"2021-03-24T21:53:51.744271+0000","last_clean":"2021-03-24T21:53:51.744271+0000","last_became_active":"2021-03-24T21:51:45.378863+0000","last_became_peered":"2021-03-24T21:51:45.378863+0000","last_unstale":"2021-03-24T21:53:51.744271+0000","last_undegraded":"2021-03-24T21:53:51.744271+0000","last_fullsized":"2021-03-24T21:53:51.744271+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:35.609173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:35.609173+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"69'17","reported_seq":"74","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.103242+0000","last_change":"2021-03-24T21:52:54.714245+0000","last_active":"2021-03-24T21:53:51.103242+0000","last_peered":"2021-03-24T21:53:51.103242+0000","last_clean":"2021-03-24T21:53:51.103242+0000","last_became_active":"2021-03-24T21:51:47.411126+0000","last_became_peered":"2021-03-24T21:51:47.411126+0000","last_unstale":"2021-03-24T21:53:51.103242+0000","last_undegraded":"2021-03-24T21:53:51.103242+0000","last_fullsized":"2021-03-24T21:53:51.103242+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'17","last_scrub_stamp":"2021-03-24T21:52:54.714050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:52:54.714050+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":19,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"40","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.745750+0000","last_change":"2021-03-24T21:53:27.701155+0000","last_active":"2021-03-24T21:53:51.745750+0000","last_peered":"2021-03-24T21:53:51.745750+0000","last_clean":"2021-03-24T21:53:51.745750+0000","last_became_active":"2021-03-24T21:51:51.912009+0000","last_became_peered":"2021-03-24T21:51:51.912009+0000","last_unstale":"2021-03-24T21:53:51.745750+0000","last_undegraded":"2021-03-24T21:53:51.745750+0000","last_fullsized":"2021-03-24T21:53:51.745750+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:27.701002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:27.701002+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"45","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.550888+0000","last_change":"2021-03-24T21:52:55.846936+0000","last_active":"2021-03-24T21:53:51.550888+0000","last_peered":"2021-03-24T21:53:51.550888+0000","last_clean":"2021-03-24T21:53:51.550888+0000","last_became_active":"2021-03-24T21:51:49.422787+0000","last_became_peered":"2021-03-24T21:51:49.422787+0000","last_unstale":"2021-03-24T21:53:51.550888+0000","last_undegraded":"2021-03-24T21:53:51.550888+0000","last_fullsized":"2021-03-24T21:53:51.550888+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:55.846764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:52:55.846764+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"46","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.098521+0000","last_change":"2021-03-24T21:52:48.334326+0000","last_active":"2021-03-24T21:53:51.098521+0000","last_peered":"2021-03-24T21:53:51.098521+0000","last_clean":"2021-03-24T21:53:51.098521+0000","last_became_active":"2021-03-24T21:51:45.579513+0000","last_became_peered":"2021-03-24T21:51:45.579513+0000","last_unstale":"2021-03-24T21:53:51.098521+0000","last_undegraded":"2021-03-24T21:53:51.098521+0000","last_fullsized":"2021-03-24T21:53:51.098521+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:48.334138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:52:48.334138+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"69'15","reported_seq":"71","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.100209+0000","last_change":"2021-03-24T21:52:55.233331+0000","last_active":"2021-03-24T21:53:51.100209+0000","last_peered":"2021-03-24T21:53:51.100209+0000","last_clean":"2021-03-24T21:53:51.100209+0000","last_became_active":"2021-03-24T21:51:47.410414+0000","last_became_peered":"2021-03-24T21:51:47.410414+0000","last_unstale":"2021-03-24T21:53:51.100209+0000","last_undegraded":"2021-03-24T21:53:51.100209+0000","last_fullsized":"2021-03-24T21:53:51.100209+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:52:55.233213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:52:55.233213+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"41","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.243312+0000","last_change":"2021-03-24T21:53:14.576861+0000","last_active":"2021-03-24T21:53:51.243312+0000","last_peered":"2021-03-24T21:53:51.243312+0000","last_clean":"2021-03-24T21:53:51.243312+0000","last_became_active":"2021-03-24T21:51:51.765067+0000","last_became_peered":"2021-03-24T21:51:51.765067+0000","last_unstale":"2021-03-24T21:53:51.243312+0000","last_undegraded":"2021-03-24T21:53:51.243312+0000","last_fullsized":"2021-03-24T21:53:51.243312+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:14.576687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:14.576687+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"43","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.102421+0000","last_change":"2021-03-24T21:53:29.735847+0000","last_active":"2021-03-24T21:53:51.102421+0000","last_peered":"2021-03-24T21:53:51.102421+0000","last_clean":"2021-03-24T21:53:51.102421+0000","last_became_active":"2021-03-24T21:51:49.407001+0000","last_became_peered":"2021-03-24T21:51:49.407001+0000","last_unstale":"2021-03-24T21:53:51.102421+0000","last_undegraded":"2021-03-24T21:53:51.102421+0000","last_fullsized":"2021-03-24T21:53:51.102421+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:29.735672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:29.735672+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"46","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.744368+0000","last_change":"2021-03-24T21:53:22.204032+0000","last_active":"2021-03-24T21:53:51.744368+0000","last_peered":"2021-03-24T21:53:51.744368+0000","last_clean":"2021-03-24T21:53:51.744368+0000","last_became_active":"2021-03-24T21:51:45.374203+0000","last_became_peered":"2021-03-24T21:51:45.374203+0000","last_unstale":"2021-03-24T21:53:51.744368+0000","last_undegraded":"2021-03-24T21:53:51.744368+0000","last_fullsized":"2021-03-24T21:53:51.744368+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:22.203858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:22.203858+0000","last_clean_scrub_stamp":"2021-03-24T21:53:22.203858+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"69'21","reported_seq":"80","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.100054+0000","last_change":"2021-03-24T21:52:50.302291+0000","last_active":"2021-03-24T21:53:51.100054+0000","last_peered":"2021-03-24T21:53:51.100054+0000","last_clean":"2021-03-24T21:53:51.100054+0000","last_became_active":"2021-03-24T21:51:47.416296+0000","last_became_peered":"2021-03-24T21:51:47.416296+0000","last_unstale":"2021-03-24T21:53:51.100054+0000","last_undegraded":"2021-03-24T21:53:51.100054+0000","last_fullsized":"2021-03-24T21:53:51.100054+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'21","last_scrub_stamp":"2021-03-24T21:52:50.302085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:52:50.302085+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":23,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"40","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.100495+0000","last_change":"2021-03-24T21:53:40.290157+0000","last_active":"2021-03-24T21:53:51.100495+0000","last_peered":"2021-03-24T21:53:51.100495+0000","last_clean":"2021-03-24T21:53:51.100495+0000","last_became_active":"2021-03-24T21:51:51.761142+0000","last_became_peered":"2021-03-24T21:51:51.761142+0000","last_unstale":"2021-03-24T21:53:51.100495+0000","last_undegraded":"2021-03-24T21:53:51.100495+0000","last_fullsized":"2021-03-24T21:53:51.100495+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:40.289984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:40.289984+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"65'2","reported_seq":"71","reported_epoch":"94","state":"active+clean","last_fresh":"2021-03-24T21:53:55.068643+0000","last_change":"2021-03-24T21:53:04.625006+0000","last_active":"2021-03-24T21:53:55.068643+0000","last_peered":"2021-03-24T21:53:55.068643+0000","last_clean":"2021-03-24T21:53:55.068643+0000","last_became_active":"2021-03-24T21:51:49.416182+0000","last_became_peered":"2021-03-24T21:51:49.416182+0000","last_unstale":"2021-03-24T21:53:55.068643+0000","last_undegraded":"2021-03-24T21:53:55.068643+0000","last_fullsized":"2021-03-24T21:53:55.068643+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2021-03-24T21:53:04.624892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:04.624892+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"51","reported_epoch":"94","state":"active+clean","last_fresh":"2021-03-24T21:53:57.501519+0000","last_change":"2021-03-24T21:53:57.501519+0000","last_active":"2021-03-24T21:53:57.501519+0000","last_peered":"2021-03-24T21:53:57.501519+0000","last_clean":"2021-03-24T21:53:57.501519+0000","last_became_active":"2021-03-24T21:51:45.375830+0000","last_became_peered":"2021-03-24T21:51:45.375830+0000","last_unstale":"2021-03-24T21:53:57.501519+0000","last_undegraded":"2021-03-24T21:53:57.501519+0000","last_fullsized":"2021-03-24T21:53:57.501519+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:57.501351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:57.501351+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"69'13","reported_seq":"65","reported_epoch":"93","state":"active+clean","last_fresh":"2021-03-24T21:53:51.099436+0000","last_change":"2021-03-24T21:53:20.163291+0000","last_active":"2021-03-24T21:53:51.099436+0000","last_peered":"2021-03-24T21:53:51.099436+0000","last_clean":"2021-03-24T21:53:51.099436+0000","last_became_active":"2021-03-24T21:51:47.407739+0000","last_became_peered":"2021-03-24T21:51:47.407739+0000","last_unstale":"2021-03-24T21:53:51.099436+0000","last_undegraded":"2021-03-24T21:53:51.099436+0000","last_fullsized":"2021-03-24T21:53:51.099436+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2021-03-24T21:53:20.163065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:20.163065+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":13,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19,"ondisk_log_size":19,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1069,"num_read_kb":574,"num_write":542,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":2412,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966080,"data_stored":70572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":539,"ondisk_log_size":539,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":133,"num_read_kb":128,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201311,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6488,"kb_used_data":848,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738664,"statfs":{"total":95995035648,"available":95988391936,"internally_reserved":0,"allocated":868352,"data_stored":326634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:53:44 2021","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.90900000000000003,"15min":0.90200000000000002},"min":{"1min":0.68100000000000005,"5min":0.439,"15min":0.439},"max":{"1min":2.2240000000000002,"5min":2.2240000000000002,"15min":2.2240000000000002},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.97199999999999998,"15min":0.97299999999999998},"min":{"1min":0.71299999999999997,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":2.1859999999999999,"5min":2.1859999999999999,"15min":2.1859999999999999},"last":0.94899999999999995}]},{"osd":1,"last update":"Wed Mar 24 21:53:44 2021","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.85199999999999998,"15min":0.83799999999999997},"min":{"1min":0.62,"5min":0.5,"15min":0.5},"max":{"1min":1.7090000000000001,"5min":1.7090000000000001,"15min":1.7090000000000001},"last":1.0700000000000001},{"interface":"front","average":{"1min":1.0289999999999999,"5min":0.97999999999999998,"15min":0.97199999999999998},"min":{"1min":0.68200000000000005,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.5600000000000001,"5min":2.343,"15min":2.343},"last":0.84699999999999998}]},{"osd":2,"last update":"Wed Mar 24 21:53:44 2021","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.022,"15min":1.0029999999999999},"min":{"1min":0.69599999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":4.944,"5min":4.944,"15min":4.944},"last":0.98799999999999999},{"interface":"front","average":{"1min":1.01,"5min":1.0109999999999999,"15min":1.0109999999999999},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.357,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":1.222}]},{"osd":3,"last update":"Wed Mar 24 21:53:44 2021","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.042,"15min":1.044},"min":{"1min":0.70299999999999996,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.538,"5min":1.9490000000000001,"15min":1.9490000000000001},"last":1.0980000000000001},{"interface":"front","average":{"1min":1.1100000000000001,"5min":0.998,"15min":0.97899999999999998},"min":{"1min":0.77100000000000002,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.4330000000000001,"5min":1.845,"15min":1.845},"last":0.89700000000000002}]},{"osd":4,"last update":"Wed Mar 24 21:53:44 2021","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":0.94099999999999995,"15min":0.92800000000000005},"min":{"1min":0.71799999999999997,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.3460000000000001,"5min":2.2000000000000002,"15min":2.2000000000000002},"last":1.046},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.0780000000000001,"15min":1.075},"min":{"1min":0.91900000000000004,"5min":0.504,"15min":0.504},"max":{"1min":1.381,"5min":2.1299999999999999,"15min":2.1299999999999999},"last":1.2589999999999999}]},{"osd":5,"last update":"Wed Mar 24 21:53:44 2021","interfaces":[{"interface":"back","average":{"1min":1.1299999999999999,"5min":0.997,"15min":0.97499999999999998},"min":{"1min":0.77000000000000002,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.516,"5min":1.581,"15min":1.581},"last":1.0249999999999999},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.1519999999999999,"15min":1.141},"min":{"1min":0.92500000000000004,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.395,"5min":2.4630000000000001,"15min":2.4630000000000001},"last":1.282}]},{"osd":6,"last update":"Wed Mar 24 21:53:44 2021","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.1379999999999999,"15min":1.117},"min":{"1min":0.99199999999999999,"5min":0.62,"15min":0.62},"max":{"1min":1.4590000000000001,"5min":2.5179999999999998,"15min":2.5179999999999998},"last":1.1539999999999999},{"interface":"front","average":{"1min":1.238,"5min":1.1839999999999999,"15min":1.175},"min":{"1min":1.0529999999999999,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.4730000000000001,"5min":2.3069999999999999,"15min":2.3069999999999999},"last":1.1990000000000001}]}]},{"osd":6,"up_from":47,"seq":201863462948,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6536,"kb_used_data":896,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738616,"statfs":{"total":95995035648,"available":95988342784,"internally_reserved":0,"allocated":917504,"data_stored":329856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:53:21 2021","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.91000000000000003,"15min":0.91200000000000003},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":1.2030000000000001},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.91200000000000003,"15min":0.91700000000000004},"min":{"1min":0.56799999999999995,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.95999999999999996}]},{"osd":1,"last update":"Wed Mar 24 21:53:21 2021","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.92100000000000004,"15min":0.93200000000000005},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.2470000000000001,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.91100000000000003,"15min":0.91600000000000004},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.2869999999999999,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":1.0209999999999999}]},{"osd":2,"last update":"Wed Mar 24 21:53:21 2021","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.94899999999999995,"15min":0.95099999999999996},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.96699999999999997,"15min":0.97499999999999998},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.2170000000000001,"5min":1.401,"15min":1.401},"last":0.81999999999999995}]},{"osd":3,"last update":"Wed Mar 24 21:53:21 2021","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.95299999999999996,"15min":0.95699999999999996},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.3089999999999999,"5min":1.4179999999999999,"15min":1.4179999999999999},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.022,"15min":1.0149999999999999},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.5009999999999999,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":1.0980000000000001}]},{"osd":4,"last update":"Wed Mar 24 21:53:21 2021","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.92000000000000004,"15min":0.92100000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.26,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":1.149},{"interface":"front","average":{"1min":0.97699999999999998,"5min":0.95099999999999996,"15min":0.94699999999999995},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.3149999999999999,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":1.0489999999999999}]},{"osd":5,"last update":"Wed Mar 24 21:53:21 2021","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":1.0369999999999999,"15min":1.0449999999999999},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.337,"5min":1.4830000000000001,"15min":1.4830000000000001},"last":1.302},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.04,"15min":1.038},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.4570000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":1.1850000000000001}]},{"osd":7,"last update":"Wed Mar 24 21:53:41 2021","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":1.1359999999999999,"15min":1.1279999999999999},"min":{"1min":0.83799999999999997,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.3560000000000001,"5min":1.5249999999999999,"15min":1.5249999999999999},"last":1.363},{"interface":"front","average":{"1min":1.234,"5min":1.1299999999999999,"15min":1.113},"min":{"1min":0.79900000000000004,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.478,"5min":1.478,"15min":1.478},"last":1.55}]}]},{"osd":1,"up_from":13,"seq":55834574905,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7192,"kb_used_data":976,"kb_used_omap":0,"kb_used_meta":6208,"kb_avail":93737960,"statfs":{"total":95995035648,"available":95987671040,"internally_reserved":0,"allocated":999424,"data_stored":332668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6356992},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:53:33 2021","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.629,"15min":0.58699999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.68500000000000005,"15min":0.64100000000000001},"min":{"1min":0.41399999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.3009999999999999,"5min":1.641,"15min":1.641},"last":0.67900000000000005}]},{"osd":2,"last update":"Wed Mar 24 21:53:54 2021","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.65800000000000003,"15min":0.63100000000000001},"min":{"1min":0.40300000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.78300000000000003,"15min":0.751},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.046,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":0.81599999999999995}]},{"osd":3,"last update":"Wed Mar 24 21:53:18 2021","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":1.296,"15min":1.4730000000000001},"min":{"1min":0.63100000000000001,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.2849999999999999,"5min":17.032,"15min":17.032},"last":0.93600000000000005},{"interface":"front","average":{"1min":0.90900000000000003,"5min":1.2290000000000001,"15min":1.389},"min":{"1min":0.621,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.135,"5min":16.908999999999999,"15min":16.908999999999999},"last":1.373}]},{"osd":4,"last update":"Wed Mar 24 21:53:41 2021","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.96399999999999997,"15min":0.96399999999999997},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.4510000000000001,"5min":1.617,"15min":1.617},"last":1.2609999999999999},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.093,"15min":1.095},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.706,"5min":1.7150000000000001,"15min":1.7150000000000001},"last":1.1000000000000001}]},{"osd":5,"last update":"Wed Mar 24 21:53:59 2021","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.0069999999999999,"15min":1.002},"min":{"1min":0.67500000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.5289999999999999,"5min":1.5489999999999999,"15min":1.5489999999999999},"last":1.1259999999999999},{"interface":"front","average":{"1min":1.173,"5min":1.0860000000000001,"15min":1.0700000000000001},"min":{"1min":0.80800000000000005,"5min":0.748,"15min":0.748},"max":{"1min":1.667,"5min":1.667,"15min":1.667},"last":0.98899999999999999}]},{"osd":6,"last update":"Wed Mar 24 21:53:18 2021","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.141,"15min":1.1399999999999999},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":1.4239999999999999,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":1.1930000000000001},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.1479999999999999,"15min":1.1619999999999999},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.3819999999999999,"5min":1.806,"15min":1.806},"last":1.319}]},{"osd":7,"last update":"Wed Mar 24 21:53:41 2021","interfaces":[{"interface":"back","average":{"1min":1.2030000000000001,"5min":1.23,"15min":1.234},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.7789999999999999,"5min":1.7789999999999999,"15min":1.7789999999999999},"last":1.343},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.175,"15min":1.181},"min":{"1min":0.96499999999999997,"5min":0.90300000000000002,"15min":0.90300000000000002},"max":{"1min":1.3480000000000001,"5min":1.532,"15min":1.532},"last":1.3029999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738428,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6816,"kb_used_data":856,"kb_used_omap":0,"kb_used_meta":5952,"kb_avail":93738336,"statfs":{"total":95995035648,"available":95988056064,"internally_reserved":0,"allocated":876544,"data_stored":327571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6094848},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Mar 24 21:53:37 2021","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.63700000000000001,"15min":0.58399999999999996},"min":{"1min":0.44400000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.98299999999999998,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.67500000000000004,"15min":0.60999999999999999},"min":{"1min":0.46999999999999997,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.78900000000000003}]},{"osd":2,"last update":"Wed Mar 24 21:53:56 2021","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.68300000000000005,"15min":0.66400000000000003},"min":{"1min":0.47299999999999998,"5min":0.314,"15min":0.314},"max":{"1min":0.92600000000000005,"5min":1.468,"15min":1.468},"last":0.755},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.754,"15min":0.69799999999999995},"min":{"1min":0.41199999999999998,"5min":0.376,"15min":0.376},"max":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.92900000000000005}]},{"osd":3,"last update":"Wed Mar 24 21:53:20 2021","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.80500000000000005,"15min":0.80000000000000004},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.115,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.94999999999999996},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.93400000000000005,"15min":0.93200000000000005},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.2430000000000001,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":1}]},{"osd":4,"last update":"Wed Mar 24 21:53:37 2021","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.0900000000000001,"15min":1.0960000000000001},"min":{"1min":0.81799999999999995,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.4570000000000001,"5min":1.7,"15min":1.7},"last":1.1879999999999999},{"interface":"front","average":{"1min":1.056,"5min":1.028,"15min":1.014},"min":{"1min":0.78700000000000003,"5min":0.625,"15min":0.625},"max":{"1min":1.375,"5min":1.407,"15min":1.407},"last":1.2350000000000001}]},{"osd":5,"last update":"Wed Mar 24 21:53:56 2021","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":1.0489999999999999,"15min":1.0609999999999999},"min":{"1min":0.74299999999999999,"5min":0.62,"15min":0.62},"max":{"1min":1.3080000000000001,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":1.2709999999999999},{"interface":"front","average":{"1min":1.194,"5min":1.1819999999999999,"15min":1.1779999999999999},"min":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":1.895,"5min":1.895,"15min":1.895},"last":1.4419999999999999}]},{"osd":6,"last update":"Wed Mar 24 21:53:20 2021","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.0669999999999999,"15min":1.0669999999999999},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.3779999999999999,"5min":1.427,"15min":1.427},"last":1.4199999999999999},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.224,"15min":1.22},"min":{"1min":0.877,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.6659999999999999,"5min":1.6659999999999999,"15min":1.6659999999999999},"last":1.5149999999999999}]},{"osd":7,"last update":"Wed Mar 24 21:53:42 2021","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.1739999999999999,"15min":1.1699999999999999},"min":{"1min":0.82099999999999995,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":1.4970000000000001,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":1.375},{"interface":"front","average":{"1min":1.157,"5min":1.131,"15min":1.127},"min":{"1min":0.77100000000000002,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.4319999999999999,"5min":1.4490000000000001,"15min":1.4490000000000001},"last":1.357}]}]},{"osd":2,"up_from":18,"seq":77309411382,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6588,"kb_used_data":820,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738564,"statfs":{"total":95995035648,"available":95988289536,"internally_reserved":0,"allocated":839680,"data_stored":326855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:53:55 2021","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.66500000000000004,"15min":0.625},"min":{"1min":0.38300000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.3640000000000001,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.72399999999999998,"15min":0.68400000000000005},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.0740000000000001,"5min":1.405,"15min":1.405},"last":0.69699999999999995}]},{"osd":1,"last update":"Wed Mar 24 21:53:55 2021","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.74299999999999999,"15min":0.69899999999999995},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.093,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.78700000000000003,"15min":0.73499999999999999},"min":{"1min":0.38300000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.6240000000000001,"5min":1.6240000000000001,"15min":1.6240000000000001},"last":0.84699999999999998}]},{"osd":3,"last update":"Wed Mar 24 21:53:17 2021","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.84399999999999997,"15min":0.82399999999999995},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.9890000000000001,"5min":1.9890000000000001,"15min":1.9890000000000001},"last":0.93200000000000005},{"interface":"front","average":{"1min":0.95699999999999996,"5min":0.92000000000000004,"15min":0.89400000000000002},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":2.0950000000000002,"5min":2.0950000000000002,"15min":2.0950000000000002},"last":0.96699999999999997}]},{"osd":4,"last update":"Wed Mar 24 21:53:40 2021","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.04,"15min":1.036},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":3.3380000000000001,"5min":3.3380000000000001,"15min":3.3380000000000001},"last":0.998},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.0680000000000001,"15min":1.046},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.2050000000000001,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":1.222}]},{"osd":5,"last update":"Wed Mar 24 21:53:55 2021","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":1.034,"15min":1.0329999999999999},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.536,"5min":1.536,"15min":1.536},"last":0.89500000000000002},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.157,"15min":1.153},"min":{"1min":0.70199999999999996,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.732,"5min":3.1509999999999998,"15min":3.1509999999999998},"last":1.1459999999999999}]},{"osd":6,"last update":"Wed Mar 24 21:53:17 2021","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.256,"15min":1.262},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":2.2810000000000001,"5min":2.2810000000000001,"15min":2.2810000000000001},"last":1.109},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.1619999999999999,"15min":1.165},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":1.8069999999999999,"5min":2.016,"15min":2.016},"last":1.2909999999999999}]},{"osd":7,"last update":"Wed Mar 24 21:53:43 2021","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":1.2190000000000001,"15min":1.228},"min":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.6719999999999999,"5min":1.6719999999999999,"15min":1.6719999999999999},"last":1.3540000000000001},{"interface":"front","average":{"1min":1.2290000000000001,"5min":1.246,"15min":1.2490000000000001},"min":{"1min":0.80800000000000005,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":3.5419999999999998,"5min":3.5419999999999998,"15min":3.5419999999999998},"last":1.2030000000000001}]}]},{"osd":3,"up_from":26,"seq":111669149745,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6916,"kb_used_data":956,"kb_used_omap":0,"kb_used_meta":5952,"kb_avail":93738236,"statfs":{"total":95995035648,"available":95987953664,"internally_reserved":0,"allocated":978944,"data_stored":331024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6094848},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:53:21 2021","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.63300000000000001,"15min":0.63500000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.92900000000000005,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.73299999999999998,"15min":0.72099999999999997},"min":{"1min":0.41499999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.57299999999999995}]},{"osd":1,"last update":"Wed Mar 24 21:53:21 2021","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.74199999999999999,"15min":0.73799999999999999},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.018,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.75600000000000001,"15min":0.755},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.97399999999999998,"5min":1.103,"15min":1.103},"last":0.82399999999999995}]},{"osd":2,"last update":"Wed Mar 24 21:53:21 2021","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.79700000000000004,"15min":0.78700000000000003},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.77500000000000002,"15min":0.76500000000000001},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0449999999999999,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.92500000000000004}]},{"osd":4,"last update":"Wed Mar 24 21:53:40 2021","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.95399999999999996,"15min":0.95399999999999996},"min":{"1min":0.66400000000000003,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.1950000000000001,"5min":1.702,"15min":1.702},"last":1.05},{"interface":"front","average":{"1min":1.016,"5min":0.96599999999999997,"15min":0.94399999999999995},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.671,"5min":1.671,"15min":1.671},"last":0.69299999999999995}]},{"osd":5,"last update":"Wed Mar 24 21:53:57 2021","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.0669999999999999,"15min":1.0620000000000001},"min":{"1min":0.627,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.7490000000000001,"5min":2.423,"15min":2.423},"last":0.75600000000000001},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.0580000000000001,"15min":1.083},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.256,"5min":1.496,"15min":1.496},"last":1.133}]},{"osd":6,"last update":"Wed Mar 24 21:53:21 2021","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.1180000000000001,"15min":1.1259999999999999},"min":{"1min":0.85499999999999998,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.49,"5min":1.49,"15min":1.49},"last":0.77700000000000002},{"interface":"front","average":{"1min":1.155,"5min":1.1539999999999999,"15min":1.1539999999999999},"min":{"1min":0.77600000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":1.7070000000000001,"5min":1.7070000000000001,"15min":1.7070000000000001},"last":0.96699999999999997}]},{"osd":7,"last update":"Wed Mar 24 21:53:47 2021","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":1.0900000000000001,"15min":1.081},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.5800000000000001,"5min":1.5800000000000001,"15min":1.5800000000000001},"last":1.268},{"interface":"front","average":{"1min":1.2010000000000001,"5min":1.1679999999999999,"15min":1.1619999999999999},"min":{"1min":0.90400000000000003,"5min":0.751,"15min":0.751},"max":{"1min":1.556,"5min":1.7410000000000001,"15min":1.7410000000000001},"last":0.94899999999999995}]}]},{"osd":4,"up_from":32,"seq":137438953515,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6748,"kb_used_data":916,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738404,"statfs":{"total":95995035648,"available":95988125696,"internally_reserved":0,"allocated":937984,"data_stored":329672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:53:36 2021","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.84699999999999998,"15min":0.84099999999999997},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.2490000000000001,"5min":1.268,"15min":1.268},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.89200000000000002,"15min":0.86199999999999999},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.667,"5min":1.667,"15min":1.667},"last":1.079}]},{"osd":1,"last update":"Wed Mar 24 21:53:36 2021","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.92000000000000004,"15min":0.91800000000000004},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.1439999999999999,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.86899999999999999,"15min":0.84699999999999998},"min":{"1min":0.59199999999999997,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.337,"5min":1.337,"15min":1.337},"last":1.2190000000000001}]},{"osd":2,"last update":"Wed Mar 24 21:53:36 2021","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.90700000000000003,"15min":0.89900000000000002},"min":{"1min":0.57699999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.6279999999999999,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.878,"5min":0.92800000000000005,"15min":0.91200000000000003},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.169,"5min":1.335,"15min":1.335},"last":0.78700000000000003}]},{"osd":3,"last update":"Wed Mar 24 21:53:36 2021","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.92800000000000005,"15min":0.93400000000000005},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.1359999999999999,"5min":1.601,"15min":1.601},"last":0.92400000000000004},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.97099999999999997,"15min":0.95299999999999996},"min":{"1min":0.754,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.474,"5min":1.474,"15min":1.474},"last":1.115}]},{"osd":5,"last update":"Wed Mar 24 21:52:57 2021","interfaces":[{"interface":"back","average":{"1min":1.0149999999999999,"5min":0.96299999999999997,"15min":0.95499999999999996},"min":{"1min":0.59599999999999997,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":0.81299999999999994},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.026,"15min":1},"min":{"1min":0.78300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":2.0760000000000001,"5min":2.0760000000000001,"15min":2.0760000000000001},"last":0.97599999999999998}]},{"osd":6,"last update":"Wed Mar 24 21:53:18 2021","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":1.0620000000000001,"15min":1.0640000000000001},"min":{"1min":0.79900000000000004,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.2749999999999999,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":0.83699999999999997},{"interface":"front","average":{"1min":1.018,"5min":1.056,"15min":1.0620000000000001},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.357,"5min":1.506,"15min":1.506},"last":1.004}]},{"osd":7,"last update":"Wed Mar 24 21:53:41 2021","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":1.1679999999999999,"15min":1.1870000000000001},"min":{"1min":0.624,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.587,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.1910000000000001,"15min":1.2010000000000001},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.702,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":1.1859999999999999}]}]},{"osd":5,"up_from":39,"seq":167503724584,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6492,"kb_used_data":788,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738660,"statfs":{"total":95995035648,"available":95988387840,"internally_reserved":0,"allocated":806912,"data_stored":324635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:52:56 2021","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.89000000000000001,"15min":0.88900000000000001},"min":{"1min":0.623,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":2.29,"5min":2.29,"15min":2.29},"last":0.87},{"interface":"front","average":{"1min":1.0489999999999999,"5min":0.91700000000000004,"15min":0.89500000000000002},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":3.1779999999999999,"5min":3.1779999999999999,"15min":3.1779999999999999},"last":0.75800000000000001}]},{"osd":1,"last update":"Wed Mar 24 21:52:56 2021","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":0.93200000000000005,"15min":0.92500000000000004},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":2.8690000000000002,"5min":2.8690000000000002,"15min":2.8690000000000002},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.94199999999999995,"5min":0.93400000000000005,"15min":0.93300000000000005},"min":{"1min":0.66600000000000004,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":0.78800000000000003}]},{"osd":2,"last update":"Wed Mar 24 21:52:56 2021","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.90000000000000002,"15min":0.90200000000000002},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.95999999999999996},{"interface":"front","average":{"1min":1.1839999999999999,"5min":0.94699999999999995,"15min":0.90700000000000003},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":5.8479999999999999,"5min":5.8479999999999999,"15min":5.8479999999999999},"last":0.94199999999999995}]},{"osd":3,"last update":"Wed Mar 24 21:52:56 2021","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":0.94399999999999995,"15min":0.93600000000000005},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":3.1259999999999999,"5min":3.1259999999999999,"15min":3.1259999999999999},"last":0.98499999999999999},{"interface":"front","average":{"1min":1.0409999999999999,"5min":0.95999999999999996,"15min":0.94599999999999995},"min":{"1min":0.67400000000000004,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":2.6850000000000001,"5min":2.6850000000000001,"15min":2.6850000000000001},"last":1.05}]},{"osd":4,"last update":"Wed Mar 24 21:52:56 2021","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"min":{"1min":0.55800000000000005,"5min":0.439,"15min":0.439},"max":{"1min":1.1519999999999999,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":1.085},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.90700000000000003,"15min":0.89500000000000002},"min":{"1min":0.71499999999999997,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.171,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":1.1140000000000001}]},{"osd":6,"last update":"Wed Mar 24 21:53:19 2021","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":0.89200000000000002,"15min":0.87},"min":{"1min":0.71999999999999997,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":1.3100000000000001},{"interface":"front","average":{"1min":1.153,"5min":0.99299999999999999,"15min":0.96699999999999997},"min":{"1min":0.72299999999999998,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":2.3079999999999998,"5min":2.3079999999999998,"15min":2.3079999999999998},"last":1.1639999999999999}]},{"osd":7,"last update":"Wed Mar 24 21:53:42 2021","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.0549999999999999,"15min":1.0469999999999999},"min":{"1min":0.82899999999999996,"5min":0.624,"15min":0.624},"max":{"1min":1.484,"5min":1.484,"15min":1.484},"last":1.393},{"interface":"front","average":{"1min":1.196,"5min":1.0189999999999999,"15min":0.98899999999999999},"min":{"1min":0.88800000000000001,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":2.1539999999999999,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":1.2210000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":7420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":339968,"data_stored":12094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":6670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":11712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":274432,"data_stored":9910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":5638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":9628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":7500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-03-24T21:54:03.631 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2021-03-24T21:54:03.632 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd dump 2021-03-24T21:54:03.909 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:03 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:54:03.909 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:03 smithi125 conmon[24927]: 54:02.296805+0000 osd.4 (osd.4) 31 : cluster [DBG] 4.e scrub ok 2021-03-24T21:54:03.909 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:03 smithi125 conmon[24927]: cluster 2021-03-24T21:54 2021-03-24T21:54:03.910 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:03 smithi125 conmon[24927]: :02.694475+0000 mgr.y (mgr.14144) 293 : cluster 2021-03-24T21:54:03.910 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:03 smithi125 conmon[24927]: [DBG] pgmap v293: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-24T21:54:03.910 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:03 smithi125 conmon[24927]: audit 2021-03-24T21: 2021-03-24T21:54:03.910 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:03 smithi125 conmon[24927]: 54:02.740686+0000 mgr.y (mgr.14144) 294 2021-03-24T21:54:03.911 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:03 smithi125 conmon[24927]: : audit [DBG] from='client.34613 v1:172.21.15.125:0/2142727820' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T21:54:03.911 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:03 smithi125 conmon[19936]: cluster 2021-03-24T21:54:02.296805+0000 osd.4 (osd.4) 31 : cluster 2021-03-24T21:54:03.911 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:03 smithi125 conmon[19936]: [DBG] 4.e scrub ok 2021-03-24T21:54:03.911 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:03 smithi125 conmon[19936]: cluster 2021-03-24T21:54:02.694475+0000 mgr.y (mgr.14144) 293 : cluster [DBG] pgmap v293: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-24T21:54:03.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:03 smithi125 conmon[19936]: audit 2021-03-24T21:54:02 2021-03-24T21:54:03.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:03 smithi125 conmon[19936]: .740686+0000 mgr.y (mgr.14144) 294 : audit [DBG] from='client.34613 v1:172.21.15.125:0/2142727820' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T21:54:03.953 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:54:04.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:03 smithi142 conmon[20125]: cluster 2021-03-24T21:54:02.296805+0000 osd.4 (osd.4) 31 2021-03-24T21:54:04.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:03 smithi142 conmon[20125]: : cluster [DBG] 4.e scrub ok 2021-03-24T21:54:04.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:03 smithi142 conmon[20125]: cluster 2021-03-24T21:54:02.694475+0000 2021-03-24T21:54:04.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:03 smithi142 conmon[20125]: mgr.y (mgr.14144) 293 : cluster [DBG] pgmap v293: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-24T21:54:04.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:03 smithi142 conmon[20125]: audit 2021-03 2021-03-24T21:54:04.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:03 smithi142 conmon[20125]: -24T21:54:02.740686 2021-03-24T21:54:04.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:03 smithi142 conmon[20125]: +0000 mgr.y (mgr.14144) 294 : audit [DBG] 2021-03-24T21:54:04.119 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:03 smithi142 conmon[20125]: from='client.34613 v1:172.21.15.125:0/2142727820' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T21:54:04.909 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:04 smithi125 conmon[24927]: cluster 2021-03-24T21:54 2021-03-24T21:54:04.909 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:04 smithi125 conmon[24927]: :03.731421+0000 osd.1 (osd.1) 27 : cluster [DBG] 4.f scrub ok 2021-03-24T21:54:04.910 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:04 smithi125 conmon[19936]: cluster 2021-03-24T21:54: 2021-03-24T21:54:04.910 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:04 smithi125 conmon[19936]: 03.731421+0000 osd.1 (osd.1) 27 : cluster [DBG] 4.f scrub ok 2021-03-24T21:54:05.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:04 smithi142 conmon[20125]: cluster 2021-03-24T21:54 2021-03-24T21:54:05.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:04 smithi142 conmon[20125]: :03.731421+0000 osd.1 (osd.1) 27 : cluster [DBG] 4.f scrub ok 2021-03-24T21:54:05.909 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:05 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:54:05.909 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:05 smithi125 conmon[24927]: 54:04.695436+0000 mgr.y (mgr.14144) 295 : cluster [DBG] pgmap v294: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2021-03-24T21:54:05.910 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:05 smithi125 conmon[24927]: cluster 2021-03-24T21:54:04.706120+0000 2021-03-24T21:54:05.910 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:05 smithi125 conmon[24927]: osd.7 (osd.7) 27 : cluster [DBG] 5.9 scrub ok 2021-03-24T21:54:05.910 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:05 smithi125 conmon[24927]: audit 2021 2021-03-24T21:54:05.910 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:05 smithi125 conmon[24927]: -03-24T21:54:04.745752+0000 mon.a 2021-03-24T21:54:05.911 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:05 smithi125 conmon[24927]: (mon.0) 609 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T21:54:05.911 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:05 smithi125 conmon[24927]: audit 2021 2021-03-24T21:54:05.911 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:05 smithi125 conmon[24927]: -03-24T21:54:04.748394+0000 mon.a 2021-03-24T21:54:05.911 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:05 smithi125 conmon[24927]: (mon.0) 610 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T21:54:05.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:05 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:54:05.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:05 smithi125 conmon[19936]: 54:04.695436+0000 mgr.y (mgr.14144) 295 : cluster [DBG] pgmap v294: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2021-03-24T21:54:05.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:05 smithi125 conmon[19936]: cluster 2021-03-24T21:54:05.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:05 smithi125 conmon[19936]: 2021-03-24T21 2021-03-24T21:54:05.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:05 smithi125 conmon[19936]: :54:04.706120 2021-03-24T21:54:05.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:05 smithi125 conmon[19936]: +0000 osd.7 (osd. 2021-03-24T21:54:05.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:05 smithi125 conmon[19936]: 7) 27 : cluster 2021-03-24T21:54:05.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:05 smithi125 conmon[19936]: [DBG] 5.9 scrub ok 2021-03-24T21:54:05.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:05 smithi125 conmon[19936]: audit 2021 2021-03-24T21:54:05.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:05 smithi125 conmon[19936]: -03-24T21:54: 2021-03-24T21:54:05.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:05 smithi125 conmon[19936]: 04.745752+0000 mon.a (mon 2021-03-24T21:54:05.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:05 smithi125 conmon[19936]: .0) 609 : audit [INF] 2021-03-24T21:54:05.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:05 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T21:54:05.915 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:05 smithi125 conmon[19936]: audit 2021-03-24T21:54:05.915 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:05 smithi125 conmon[19936]: 2021-03-24T21: 2021-03-24T21:54:05.915 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:05 smithi125 conmon[19936]: 54:04.748394+0000 2021-03-24T21:54:05.915 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:05 smithi125 conmon[19936]: mon.a (mon.0) 610 2021-03-24T21:54:05.916 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:05 smithi125 conmon[19936]: : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T21:54:05.916 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:05 smithi125 conmon[19936]: 2021-03-24T21:54:05.924 INFO:teuthology.orchestra.run.smithi125.stdout:epoch 94 2021-03-24T21:54:05.925 INFO:teuthology.orchestra.run.smithi125.stdout:fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c 2021-03-24T21:54:05.925 INFO:teuthology.orchestra.run.smithi125.stdout:created 2021-03-24T21:46:19.163312+0000 2021-03-24T21:54:05.925 INFO:teuthology.orchestra.run.smithi125.stdout:modified 2021-03-24T21:53:54.107674+0000 2021-03-24T21:54:05.925 INFO:teuthology.orchestra.run.smithi125.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2021-03-24T21:54:05.926 INFO:teuthology.orchestra.run.smithi125.stdout:crush_version 18 2021-03-24T21:54:05.926 INFO:teuthology.orchestra.run.smithi125.stdout:full_ratio 0.95 2021-03-24T21:54:05.927 INFO:teuthology.orchestra.run.smithi125.stdout:backfillfull_ratio 0.9 2021-03-24T21:54:05.927 INFO:teuthology.orchestra.run.smithi125.stdout:nearfull_ratio 0.85 2021-03-24T21:54:05.927 INFO:teuthology.orchestra.run.smithi125.stdout:require_min_compat_client luminous 2021-03-24T21:54:05.927 INFO:teuthology.orchestra.run.smithi125.stdout:min_compat_client luminous 2021-03-24T21:54:05.928 INFO:teuthology.orchestra.run.smithi125.stdout:require_osd_release pacific 2021-03-24T21:54:05.928 INFO:teuthology.orchestra.run.smithi125.stdout:stretch_mode_enabled false 2021-03-24T21:54:05.928 INFO:teuthology.orchestra.run.smithi125.stdout:pool 1 'device_health_metrics' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_min 1 application mgr_devicehealth 2021-03-24T21:54:05.928 INFO:teuthology.orchestra.run.smithi125.stdout:pool 2 'iscsi' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 3 pgp_num 3 autoscale_mode off last_change 59 flags hashpspool stripe_width 0 application rbd 2021-03-24T21:54:05.929 INFO:teuthology.orchestra.run.smithi125.stdout:pool 3 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 62 flags hashpspool stripe_width 0 application rgw 2021-03-24T21:54:05.929 INFO:teuthology.orchestra.run.smithi125.stdout:pool 4 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 64 flags hashpspool stripe_width 0 application rgw 2021-03-24T21:54:05.929 INFO:teuthology.orchestra.run.smithi125.stdout:pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 66 flags hashpspool stripe_width 0 application rgw 2021-03-24T21:54:05.929 INFO:teuthology.orchestra.run.smithi125.stdout:pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 69 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 8 application rgw 2021-03-24T21:54:05.930 INFO:teuthology.orchestra.run.smithi125.stdout:max_osd 8 2021-03-24T21:54:05.930 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0 up in weight 1 up_from 8 up_thru 66 down_at 0 last_clean_interval [0,0) v1:172.21.15.125:6801/1197472172 v1:172.21.15.125:6802/1197472172 exists,up 1f8f9a69-4b85-4054-8b11-36d35574494c 2021-03-24T21:54:05.930 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1 up in weight 1 up_from 13 up_thru 66 down_at 0 last_clean_interval [0,0) v1:172.21.15.125:6805/113845288 v1:172.21.15.125:6806/113845288 exists,up 48db0723-5806-4e5c-9919-f3d7f52ad169 2021-03-24T21:54:05.930 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2 up in weight 1 up_from 18 up_thru 72 down_at 0 last_clean_interval [0,0) v1:172.21.15.125:6809/246939163 v1:172.21.15.125:6810/246939163 exists,up d6a361f8-4e4d-4f46-8ab9-01bf71dbb2c8 2021-03-24T21:54:05.931 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3 up in weight 1 up_from 26 up_thru 72 down_at 0 last_clean_interval [0,0) v1:172.21.15.125:6813/2209636173 v1:172.21.15.125:6814/2209636173 exists,up 716a41ce-7a28-400f-bf75-4c1ad99c7f94 2021-03-24T21:54:05.931 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4 up in weight 1 up_from 32 up_thru 66 down_at 0 last_clean_interval [0,0) v1:172.21.15.142:6800/1816503330 v1:172.21.15.142:6801/1816503330 exists,up f9bdd8d2-b292-49da-ad4f-99682122f543 2021-03-24T21:54:05.931 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5 up in weight 1 up_from 39 up_thru 66 down_at 0 last_clean_interval [0,0) v1:172.21.15.142:6804/396400804 v1:172.21.15.142:6805/396400804 exists,up ec0d43aa-6905-43f0-9e32-e19d4cf5129d 2021-03-24T21:54:05.931 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6 up in weight 1 up_from 47 up_thru 72 down_at 0 last_clean_interval [0,0) v1:172.21.15.142:6808/4048483913 v1:172.21.15.142:6809/4048483913 exists,up a26a487e-ed26-4d92-98ea-896739302921 2021-03-24T21:54:05.932 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7 up in weight 1 up_from 55 up_thru 66 down_at 0 last_clean_interval [0,0) v1:172.21.15.142:6812/3678129570 v1:172.21.15.142:6813/3678129570 exists,up 682ba5ce-0172-48e3-82f8-57a3f98c911b 2021-03-24T21:54:05.932 INFO:teuthology.orchestra.run.smithi125.stdout:pg_upmap_items 4.10 [1,2] 2021-03-24T21:54:05.932 INFO:teuthology.orchestra.run.smithi125.stdout:pg_upmap_items 4.1c [1,5] 2021-03-24T21:54:05.932 INFO:teuthology.orchestra.run.smithi125.stdout:pg_upmap_items 4.1f [1,2] 2021-03-24T21:54:05.933 INFO:teuthology.orchestra.run.smithi125.stdout:blocklist 172.21.15.125:0/3077905055 expires 2021-03-25T21:47:04.502055+0000 2021-03-24T21:54:05.933 INFO:teuthology.orchestra.run.smithi125.stdout:blocklist 172.21.15.125:0/2210544773 expires 2021-03-25T21:47:04.502055+0000 2021-03-24T21:54:05.933 INFO:teuthology.orchestra.run.smithi125.stdout:blocklist 172.21.15.125:6800/3176807740 expires 2021-03-25T21:47:04.502055+0000 2021-03-24T21:54:05.933 INFO:teuthology.orchestra.run.smithi125.stdout:blocklist 172.21.15.125:6800/3531208850 expires 2021-03-25T21:46:43.197724+0000 2021-03-24T21:54:05.933 INFO:teuthology.orchestra.run.smithi125.stdout:blocklist 172.21.15.125:0/3337261959 expires 2021-03-25T21:46:43.197724+0000 2021-03-24T21:54:05.934 INFO:teuthology.orchestra.run.smithi125.stdout:blocklist 172.21.15.125:0/1919150488 expires 2021-03-25T21:46:43.197724+0000 2021-03-24T21:54:06.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:05 smithi142 conmon[20125]: cluster 2021-03-24T21:54:04.695436+0000 mgr.y (mgr.14144) 295 : cluster 2021-03-24T21:54:06.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:05 smithi142 conmon[20125]: [DBG] pgmap v294: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2021-03-24T21:54:06.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:05 smithi142 conmon[20125]: cluster 2021-03-24T21:54:04.706120+0000 osd.7 (osd.7) 27 : cluster [DBG] 2021-03-24T21:54:06.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:05 smithi142 conmon[20125]: 5.9 scrub ok 2021-03-24T21:54:06.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:05 smithi142 conmon[20125]: audit 2021-03-24 2021-03-24T21:54:06.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:05 smithi142 conmon[20125]: T21:54:04.745752+0000 mon.a (mon.0) 2021-03-24T21:54:06.119 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:05 smithi142 conmon[20125]: 609 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T21:54:06.119 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:05 smithi142 conmon[20125]: audit 2021-03-24T 2021-03-24T21:54:06.119 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:05 smithi142 conmon[20125]: 21:54:04.748394+0000 mon.a (mon 2021-03-24T21:54:06.119 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:05 smithi142 conmon[20125]: .0) 610 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T21:54:06.747 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 66 down_at 0 last_clean_interval [0,0) v1:172.21.15.125:6801/1197472172 v1:172.21.15.125:6802/1197472172 exists,up 1f8f9a69-4b85-4054-8b11-36d35574494c', 'osd.1 up in weight 1 up_from 13 up_thru 66 down_at 0 last_clean_interval [0,0) v1:172.21.15.125:6805/113845288 v1:172.21.15.125:6806/113845288 exists,up 48db0723-5806-4e5c-9919-f3d7f52ad169', 'osd.2 up in weight 1 up_from 18 up_thru 72 down_at 0 last_clean_interval [0,0) v1:172.21.15.125:6809/246939163 v1:172.21.15.125:6810/246939163 exists,up d6a361f8-4e4d-4f46-8ab9-01bf71dbb2c8', 'osd.3 up in weight 1 up_from 26 up_thru 72 down_at 0 last_clean_interval [0,0) v1:172.21.15.125:6813/2209636173 v1:172.21.15.125:6814/2209636173 exists,up 716a41ce-7a28-400f-bf75-4c1ad99c7f94', 'osd.4 up in weight 1 up_from 32 up_thru 66 down_at 0 last_clean_interval [0,0) v1:172.21.15.142:6800/1816503330 v1:172.21.15.142:6801/1816503330 exists,up f9bdd8d2-b292-49da-ad4f-99682122f543', 'osd.5 up in weight 1 up_from 39 up_thru 66 down_at 0 last_clean_interval [0,0) v1:172.21.15.142:6804/396400804 v1:172.21.15.142:6805/396400804 exists,up ec0d43aa-6905-43f0-9e32-e19d4cf5129d', 'osd.6 up in weight 1 up_from 47 up_thru 72 down_at 0 last_clean_interval [0,0) v1:172.21.15.142:6808/4048483913 v1:172.21.15.142:6809/4048483913 exists,up a26a487e-ed26-4d92-98ea-896739302921', 'osd.7 up in weight 1 up_from 55 up_thru 66 down_at 0 last_clean_interval [0,0) v1:172.21.15.142:6812/3678129570 v1:172.21.15.142:6813/3678129570 exists,up 682ba5ce-0172-48e3-82f8-57a3f98c911b'] 2021-03-24T21:54:06.748 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon mon.a config show 2021-03-24T21:54:07.057 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:54:07.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:06 smithi142 conmon[20125]: cluster 2021-03-24T21:54:05.473338+0000 osd.0 (osd.0) 30 : cluster 2021-03-24T21:54:07.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:06 smithi142 conmon[20125]: [DBG] 3.12 deep-scrub ok 2021-03-24T21:54:07.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:06 smithi142 conmon[20125]: audit 2021-03-24T21:54:05.924643+0000 mon.c (mon.2 2021-03-24T21:54:07.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:06 smithi142 conmon[20125]: ) 20 : audit [DBG] from='client.? v1:172.21.15.125:0/2651570391' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2021-03-24T21:54:07.159 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:06 smithi125 conmon[24927]: cluster 2021-03-24T21:54:05.473338+0000 osd.0 (osd.0) 2021-03-24T21:54:07.159 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:06 smithi125 conmon[24927]: 30 : cluster [DBG] 3.12 deep-scrub ok 2021-03-24T21:54:07.159 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:06 smithi125 conmon[24927]: audit 2021-03-24T21:54:05.924643+0000 mon.c (mon.2) 20 : audit [DBG] from='client.? v1:172.21.15.125:0/2651570391' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2021-03-24T21:54:07.160 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:06 smithi125 conmon[19936]: cluster 2021-03-24T21:54:05.473338+0000 osd.0 (osd.0) 2021-03-24T21:54:07.160 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:06 smithi125 conmon[19936]: 30 : cluster [DBG] 3.12 deep-scrub ok 2021-03-24T21:54:07.160 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:06 smithi125 conmon[19936]: audit 2021-03-24T21:54:05.924643 2021-03-24T21:54:07.160 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:06 smithi125 conmon[19936]: +0000 mon.c (mon.2) 20 : audit [DBG] from='client.? v1:172.21.15.125:0/2651570391' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2021-03-24T21:54:08.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:07 smithi142 conmon[20125]: cluster 2021-03-24T21:54:06.651091+0000 osd.1 (osd.1) 28 : cluster 2021-03-24T21:54:08.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:07 smithi142 conmon[20125]: [DBG] 4.12 scrub ok 2021-03-24T21:54:08.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:07 smithi142 conmon[20125]: cluster 2021-03-24T21:54:06.696338+0000 mgr.y (mgr.14144) 296 : 2021-03-24T21:54:08.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:07 smithi142 conmon[20125]: cluster [DBG] pgmap v295: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-03-24T21:54:08.159 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:07 smithi125 conmon[19936]: cluster 2021-03-24T21:54:06.651091+0000 2021-03-24T21:54:08.159 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:07 smithi125 conmon[19936]: osd.1 (osd.1) 28 : cluster [DBG] 4.12 scrub ok 2021-03-24T21:54:08.160 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:07 smithi125 conmon[19936]: cluster 2021-03-24T21:54:06.696338+0000 mgr.y 2021-03-24T21:54:08.160 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:07 smithi125 conmon[19936]: (mgr.14144) 296 : cluster [DBG] pgmap v295: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-03-24T21:54:08.160 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:07 smithi125 conmon[24927]: cluster 2021-03-24T21:54:06.651091+0000 osd.1 ( 2021-03-24T21:54:08.161 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:07 smithi125 conmon[24927]: osd.1) 28 : cluster [DBG] 4.12 scrub ok 2021-03-24T21:54:08.161 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:07 smithi125 conmon[24927]: cluster 2021-03-24T21:54:06.696338+0000 mgr.y (mgr.14144) 296 : cluster 2021-03-24T21:54:08.161 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:07 smithi125 conmon[24927]: [DBG] pgmap v295: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-03-24T21:54:08.836 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-24T21:54:08.836 INFO:teuthology.orchestra.run.smithi125.stdout: "name": "mon.a", 2021-03-24T21:54:08.837 INFO:teuthology.orchestra.run.smithi125.stdout: "cluster": "ceph", 2021-03-24T21:54:08.837 INFO:teuthology.orchestra.run.smithi125.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2021-03-24T21:54:08.837 INFO:teuthology.orchestra.run.smithi125.stdout: "admin_socket_mode": "", 2021-03-24T21:54:08.837 INFO:teuthology.orchestra.run.smithi125.stdout: "auth_client_required": "cephx, none", 2021-03-24T21:54:08.838 INFO:teuthology.orchestra.run.smithi125.stdout: "auth_cluster_required": "cephx", 2021-03-24T21:54:08.838 INFO:teuthology.orchestra.run.smithi125.stdout: "auth_debug": "true", 2021-03-24T21:54:08.838 INFO:teuthology.orchestra.run.smithi125.stdout: "auth_mon_ticket_ttl": "43200.000000", 2021-03-24T21:54:08.839 INFO:teuthology.orchestra.run.smithi125.stdout: "auth_service_required": "cephx", 2021-03-24T21:54:08.839 INFO:teuthology.orchestra.run.smithi125.stdout: "auth_service_ticket_ttl": "3600.000000", 2021-03-24T21:54:08.839 INFO:teuthology.orchestra.run.smithi125.stdout: "auth_supported": "", 2021-03-24T21:54:08.840 INFO:teuthology.orchestra.run.smithi125.stdout: "bdev_aio": "true", 2021-03-24T21:54:08.840 INFO:teuthology.orchestra.run.smithi125.stdout: "bdev_aio_max_queue_depth": "1024", 2021-03-24T21:54:08.840 INFO:teuthology.orchestra.run.smithi125.stdout: "bdev_aio_poll_ms": "250", 2021-03-24T21:54:08.841 INFO:teuthology.orchestra.run.smithi125.stdout: "bdev_aio_reap_max": "16", 2021-03-24T21:54:08.841 INFO:teuthology.orchestra.run.smithi125.stdout: "bdev_async_discard": "false", 2021-03-24T21:54:08.841 INFO:teuthology.orchestra.run.smithi125.stdout: "bdev_block_size": "4096", 2021-03-24T21:54:08.841 INFO:teuthology.orchestra.run.smithi125.stdout: "bdev_debug_aio": "false", 2021-03-24T21:54:08.842 INFO:teuthology.orchestra.run.smithi125.stdout: "bdev_debug_aio_log_age": "5.000000", 2021-03-24T21:54:08.842 INFO:teuthology.orchestra.run.smithi125.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2021-03-24T21:54:08.842 INFO:teuthology.orchestra.run.smithi125.stdout: "bdev_debug_inflight_ios": "false", 2021-03-24T21:54:08.843 INFO:teuthology.orchestra.run.smithi125.stdout: "bdev_enable_discard": "false", 2021-03-24T21:54:08.843 INFO:teuthology.orchestra.run.smithi125.stdout: "bdev_flock_retry": "3", 2021-03-24T21:54:08.843 INFO:teuthology.orchestra.run.smithi125.stdout: "bdev_flock_retry_interval": "0.100000", 2021-03-24T21:54:08.843 INFO:teuthology.orchestra.run.smithi125.stdout: "bdev_inject_crash": "0", 2021-03-24T21:54:08.844 INFO:teuthology.orchestra.run.smithi125.stdout: "bdev_inject_crash_flush_delay": "2", 2021-03-24T21:54:08.844 INFO:teuthology.orchestra.run.smithi125.stdout: "bdev_ioring": "false", 2021-03-24T21:54:08.844 INFO:teuthology.orchestra.run.smithi125.stdout: "bdev_ioring_hipri": "false", 2021-03-24T21:54:08.844 INFO:teuthology.orchestra.run.smithi125.stdout: "bdev_ioring_sqthread_poll": "false", 2021-03-24T21:54:08.845 INFO:teuthology.orchestra.run.smithi125.stdout: "bdev_nvme_unbind_from_kernel": "false", 2021-03-24T21:54:08.845 INFO:teuthology.orchestra.run.smithi125.stdout: "bdev_type": "", 2021-03-24T21:54:08.845 INFO:teuthology.orchestra.run.smithi125.stdout: "bluefs_alloc_size": "1048576", 2021-03-24T21:54:08.845 INFO:teuthology.orchestra.run.smithi125.stdout: "bluefs_allocator": "hybrid", 2021-03-24T21:54:08.846 INFO:teuthology.orchestra.run.smithi125.stdout: "bluefs_buffered_io": "false", 2021-03-24T21:54:08.846 INFO:teuthology.orchestra.run.smithi125.stdout: "bluefs_check_for_zeros": "false", 2021-03-24T21:54:08.846 INFO:teuthology.orchestra.run.smithi125.stdout: "bluefs_compact_log_sync": "false", 2021-03-24T21:54:08.846 INFO:teuthology.orchestra.run.smithi125.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2021-03-24T21:54:08.847 INFO:teuthology.orchestra.run.smithi125.stdout: "bluefs_log_compact_min_size": "16777216", 2021-03-24T21:54:08.847 INFO:teuthology.orchestra.run.smithi125.stdout: "bluefs_log_replay_check_allocations": "true", 2021-03-24T21:54:08.847 INFO:teuthology.orchestra.run.smithi125.stdout: "bluefs_max_log_runway": "4194304", 2021-03-24T21:54:08.847 INFO:teuthology.orchestra.run.smithi125.stdout: "bluefs_max_prefetch": "1048576", 2021-03-24T21:54:08.848 INFO:teuthology.orchestra.run.smithi125.stdout: "bluefs_min_flush_size": "524288", 2021-03-24T21:54:08.848 INFO:teuthology.orchestra.run.smithi125.stdout: "bluefs_min_log_runway": "1048576", 2021-03-24T21:54:08.849 INFO:teuthology.orchestra.run.smithi125.stdout: "bluefs_replay_recovery": "false", 2021-03-24T21:54:08.849 INFO:teuthology.orchestra.run.smithi125.stdout: "bluefs_replay_recovery_disable_compact": "false", 2021-03-24T21:54:08.849 INFO:teuthology.orchestra.run.smithi125.stdout: "bluefs_shared_alloc_size": "65536", 2021-03-24T21:54:08.850 INFO:teuthology.orchestra.run.smithi125.stdout: "bluefs_sync_write": "false", 2021-03-24T21:54:08.850 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2021-03-24T21:54:08.850 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2021-03-24T21:54:08.850 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2021-03-24T21:54:08.851 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_allocator": "hybrid", 2021-03-24T21:54:08.851 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2021-03-24T21:54:08.851 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2021-03-24T21:54:08.852 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2021-03-24T21:54:08.852 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_bitmapallocator_span_size": "1024", 2021-03-24T21:54:08.852 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_blobid_prealloc": "10240", 2021-03-24T21:54:08.852 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_block_create": "true", 2021-03-24T21:54:08.853 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_block_db_create": "false", 2021-03-24T21:54:08.853 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_block_db_path": "", 2021-03-24T21:54:08.853 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_block_db_size": "0", 2021-03-24T21:54:08.853 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_block_path": "", 2021-03-24T21:54:08.854 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_block_preallocate_file": "false", 2021-03-24T21:54:08.854 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_block_size": "107374182400", 2021-03-24T21:54:08.854 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_block_wal_create": "false", 2021-03-24T21:54:08.855 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_block_wal_path": "", 2021-03-24T21:54:08.855 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_block_wal_size": "100663296", 2021-03-24T21:54:08.855 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_bluefs": "true", 2021-03-24T21:54:08.856 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2021-03-24T21:54:08.856 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_bluefs_env_mirror": "false", 2021-03-24T21:54:08.856 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_cache_autotune": "true", 2021-03-24T21:54:08.856 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_cache_autotune_interval": "5.000000", 2021-03-24T21:54:08.857 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2021-03-24T21:54:08.857 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_cache_kv_ratio": "0.450000", 2021-03-24T21:54:08.857 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_cache_meta_ratio": "0.450000", 2021-03-24T21:54:08.858 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_cache_size": "0", 2021-03-24T21:54:08.858 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_cache_size_hdd": "1073741824", 2021-03-24T21:54:08.858 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_cache_size_ssd": "3221225472", 2021-03-24T21:54:08.858 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_cache_trim_interval": "0.050000", 2021-03-24T21:54:08.859 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_cache_trim_max_skip_pinned": "64", 2021-03-24T21:54:08.859 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_cache_type": "2q", 2021-03-24T21:54:08.859 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_clone_cow": "true", 2021-03-24T21:54:08.859 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_compression_algorithm": "snappy", 2021-03-24T21:54:08.860 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_compression_max_blob_size": "0", 2021-03-24T21:54:08.860 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2021-03-24T21:54:08.860 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2021-03-24T21:54:08.860 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_compression_min_blob_size": "0", 2021-03-24T21:54:08.860 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2021-03-24T21:54:08.861 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_compression_min_blob_size_ssd": "8192", 2021-03-24T21:54:08.861 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_compression_mode": "none", 2021-03-24T21:54:08.861 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_compression_required_ratio": "0.875000", 2021-03-24T21:54:08.861 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_csum_type": "crc32c", 2021-03-24T21:54:08.862 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_debug_enforce_settings": "default", 2021-03-24T21:54:08.862 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_debug_freelist": "false", 2021-03-24T21:54:08.862 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_debug_fsck_abort": "false", 2021-03-24T21:54:08.862 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_debug_inject_bug21040": "false", 2021-03-24T21:54:08.863 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2021-03-24T21:54:08.863 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_debug_inject_read_err": "false", 2021-03-24T21:54:08.863 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_debug_no_reuse_blocks": "false", 2021-03-24T21:54:08.863 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_debug_omit_block_device_write": "false", 2021-03-24T21:54:08.864 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_debug_omit_kv_commit": "false", 2021-03-24T21:54:08.864 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2021-03-24T21:54:08.864 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_debug_prefill": "0.000000", 2021-03-24T21:54:08.864 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_debug_prefragment_max": "1048576", 2021-03-24T21:54:08.865 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_debug_random_read_err": "0.000000", 2021-03-24T21:54:08.865 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2021-03-24T21:54:08.865 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_debug_small_allocations": "0", 2021-03-24T21:54:08.865 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2021-03-24T21:54:08.866 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_default_buffered_read": "true", 2021-03-24T21:54:08.866 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_default_buffered_write": "false", 2021-03-24T21:54:08.866 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_deferred_batch_ops": "0", 2021-03-24T21:54:08.866 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2021-03-24T21:54:08.867 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2021-03-24T21:54:08.867 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2021-03-24T21:54:08.867 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_extent_map_shard_max_size": "1200", 2021-03-24T21:54:08.867 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_extent_map_shard_min_size": "150", 2021-03-24T21:54:08.868 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_extent_map_shard_target_size": "500", 2021-03-24T21:54:08.868 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2021-03-24T21:54:08.868 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_freelist_blocks_per_key": "128", 2021-03-24T21:54:08.868 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2021-03-24T21:54:08.869 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2021-03-24T21:54:08.869 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_fsck_on_mkfs": "true", 2021-03-24T21:54:08.869 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2021-03-24T21:54:08.869 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_fsck_on_mount": "false", 2021-03-24T21:54:08.870 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_fsck_on_mount_deep": "false", 2021-03-24T21:54:08.870 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_fsck_on_umount": "false", 2021-03-24T21:54:08.870 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_fsck_on_umount_deep": "false", 2021-03-24T21:54:08.870 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2021-03-24T21:54:08.871 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_fsck_quick_fix_threads": "2", 2021-03-24T21:54:08.871 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2021-03-24T21:54:08.871 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_gc_enable_blob_threshold": "0", 2021-03-24T21:54:08.871 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_gc_enable_total_threshold": "0", 2021-03-24T21:54:08.871 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2021-03-24T21:54:08.872 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_ignore_data_csum": "false", 2021-03-24T21:54:08.872 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2021-03-24T21:54:08.872 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_kvbackend": "rocksdb", 2021-03-24T21:54:08.872 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_log_collection_list_age": "60.000000", 2021-03-24T21:54:08.873 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2021-03-24T21:54:08.873 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_log_op_age": "5.000000", 2021-03-24T21:54:08.873 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_max_alloc_size": "0", 2021-03-24T21:54:08.873 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_max_blob_size": "0", 2021-03-24T21:54:08.874 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_max_blob_size_hdd": "65536", 2021-03-24T21:54:08.874 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_max_blob_size_ssd": "65536", 2021-03-24T21:54:08.874 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_max_defer_interval": "3.000000", 2021-03-24T21:54:08.874 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_max_deferred_txc": "32", 2021-03-24T21:54:08.875 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_min_alloc_size": "0", 2021-03-24T21:54:08.875 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_min_alloc_size_hdd": "4096", 2021-03-24T21:54:08.875 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_min_alloc_size_ssd": "4096", 2021-03-24T21:54:08.875 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_nid_prealloc": "1024", 2021-03-24T21:54:08.876 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_prefer_deferred_size": "0", 2021-03-24T21:54:08.876 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2021-03-24T21:54:08.876 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2021-03-24T21:54:08.876 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_retry_disk_reads": "3", 2021-03-24T21:54:08.876 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_rocksdb_cf": "true", 2021-03-24T21:54:08.877 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P", 2021-03-24T21:54:08.877 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_rocksdb_options": "compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824", 2021-03-24T21:54:08.877 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_rocksdb_options_annex": "", 2021-03-24T21:54:08.877 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_spdk_coremask": "0x1", 2021-03-24T21:54:08.878 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_spdk_io_sleep": "5", 2021-03-24T21:54:08.878 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_spdk_max_io_completion": "0", 2021-03-24T21:54:08.878 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_spdk_mem": "512", 2021-03-24T21:54:08.878 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_sync_submit_transaction": "false", 2021-03-24T21:54:08.879 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_throttle_bytes": "67108864", 2021-03-24T21:54:08.879 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_throttle_cost_per_io": "0", 2021-03-24T21:54:08.879 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2021-03-24T21:54:08.879 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2021-03-24T21:54:08.880 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2021-03-24T21:54:08.880 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_throttle_trace_rate": "0.000000", 2021-03-24T21:54:08.880 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_tracing": "false", 2021-03-24T21:54:08.880 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2021-03-24T21:54:08.881 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_volume_selection_reserved": "0", 2021-03-24T21:54:08.881 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2021-03-24T21:54:08.881 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2021-03-24T21:54:08.881 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_warn_on_legacy_statfs": "true", 2021-03-24T21:54:08.881 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2021-03-24T21:54:08.882 INFO:teuthology.orchestra.run.smithi125.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2021-03-24T21:54:08.882 INFO:teuthology.orchestra.run.smithi125.stdout: "cephadm_path": "/usr/sbin/cephadm", 2021-03-24T21:54:08.882 INFO:teuthology.orchestra.run.smithi125.stdout: "cephfs_mirror_action_update_interval": "2", 2021-03-24T21:54:08.882 INFO:teuthology.orchestra.run.smithi125.stdout: "cephfs_mirror_directory_scan_interval": "10", 2021-03-24T21:54:08.883 INFO:teuthology.orchestra.run.smithi125.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2021-03-24T21:54:08.883 INFO:teuthology.orchestra.run.smithi125.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2021-03-24T21:54:08.883 INFO:teuthology.orchestra.run.smithi125.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2021-03-24T21:54:08.883 INFO:teuthology.orchestra.run.smithi125.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2021-03-24T21:54:08.884 INFO:teuthology.orchestra.run.smithi125.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2021-03-24T21:54:08.884 INFO:teuthology.orchestra.run.smithi125.stdout: "cephx_cluster_require_signatures": "false", 2021-03-24T21:54:08.884 INFO:teuthology.orchestra.run.smithi125.stdout: "cephx_cluster_require_version": "1", 2021-03-24T21:54:08.884 INFO:teuthology.orchestra.run.smithi125.stdout: "cephx_require_signatures": "false", 2021-03-24T21:54:08.885 INFO:teuthology.orchestra.run.smithi125.stdout: "cephx_require_version": "1", 2021-03-24T21:54:08.885 INFO:teuthology.orchestra.run.smithi125.stdout: "cephx_service_require_signatures": "false", 2021-03-24T21:54:08.885 INFO:teuthology.orchestra.run.smithi125.stdout: "cephx_service_require_version": "1", 2021-03-24T21:54:08.885 INFO:teuthology.orchestra.run.smithi125.stdout: "cephx_sign_messages": "true", 2021-03-24T21:54:08.886 INFO:teuthology.orchestra.run.smithi125.stdout: "chdir": "", 2021-03-24T21:54:08.886 INFO:teuthology.orchestra.run.smithi125.stdout: "client_acl_type": "", 2021-03-24T21:54:08.886 INFO:teuthology.orchestra.run.smithi125.stdout: "client_asio_thread_count": "2", 2021-03-24T21:54:08.886 INFO:teuthology.orchestra.run.smithi125.stdout: "client_cache_mid": "0.750000", 2021-03-24T21:54:08.887 INFO:teuthology.orchestra.run.smithi125.stdout: "client_cache_size": "16384", 2021-03-24T21:54:08.887 INFO:teuthology.orchestra.run.smithi125.stdout: "client_caps_release_delay": "5", 2021-03-24T21:54:08.887 INFO:teuthology.orchestra.run.smithi125.stdout: "client_check_pool_perm": "true", 2021-03-24T21:54:08.887 INFO:teuthology.orchestra.run.smithi125.stdout: "client_debug_force_sync_read": "false", 2021-03-24T21:54:08.888 INFO:teuthology.orchestra.run.smithi125.stdout: "client_debug_getattr_caps": "false", 2021-03-24T21:54:08.888 INFO:teuthology.orchestra.run.smithi125.stdout: "client_debug_inject_tick_delay": "0", 2021-03-24T21:54:08.888 INFO:teuthology.orchestra.run.smithi125.stdout: "client_die_on_failed_dentry_invalidate": "true", 2021-03-24T21:54:08.888 INFO:teuthology.orchestra.run.smithi125.stdout: "client_die_on_failed_remount": "false", 2021-03-24T21:54:08.889 INFO:teuthology.orchestra.run.smithi125.stdout: "client_dirsize_rbytes": "true", 2021-03-24T21:54:08.889 INFO:teuthology.orchestra.run.smithi125.stdout: "client_force_lazyio": "false", 2021-03-24T21:54:08.889 INFO:teuthology.orchestra.run.smithi125.stdout: "client_fs": "", 2021-03-24T21:54:08.889 INFO:teuthology.orchestra.run.smithi125.stdout: "client_inject_fixed_oldest_tid": "false", 2021-03-24T21:54:08.890 INFO:teuthology.orchestra.run.smithi125.stdout: "client_inject_release_failure": "false", 2021-03-24T21:54:08.890 INFO:teuthology.orchestra.run.smithi125.stdout: "client_max_inline_size": "4096", 2021-03-24T21:54:08.890 INFO:teuthology.orchestra.run.smithi125.stdout: "client_mds_namespace": "", 2021-03-24T21:54:08.890 INFO:teuthology.orchestra.run.smithi125.stdout: "client_metadata": "", 2021-03-24T21:54:08.891 INFO:teuthology.orchestra.run.smithi125.stdout: "client_mount_gid": "-1", 2021-03-24T21:54:08.891 INFO:teuthology.orchestra.run.smithi125.stdout: "client_mount_timeout": "300.000000", 2021-03-24T21:54:08.891 INFO:teuthology.orchestra.run.smithi125.stdout: "client_mount_uid": "-1", 2021-03-24T21:54:08.892 INFO:teuthology.orchestra.run.smithi125.stdout: "client_mountpoint": "/", 2021-03-24T21:54:08.892 INFO:teuthology.orchestra.run.smithi125.stdout: "client_notify_timeout": "10", 2021-03-24T21:54:08.892 INFO:teuthology.orchestra.run.smithi125.stdout: "client_oc": "true", 2021-03-24T21:54:08.892 INFO:teuthology.orchestra.run.smithi125.stdout: "client_oc_max_dirty": "104857600", 2021-03-24T21:54:08.893 INFO:teuthology.orchestra.run.smithi125.stdout: "client_oc_max_dirty_age": "5.000000", 2021-03-24T21:54:08.893 INFO:teuthology.orchestra.run.smithi125.stdout: "client_oc_max_objects": "1000", 2021-03-24T21:54:08.893 INFO:teuthology.orchestra.run.smithi125.stdout: "client_oc_size": "209715200", 2021-03-24T21:54:08.893 INFO:teuthology.orchestra.run.smithi125.stdout: "client_oc_target_dirty": "8388608", 2021-03-24T21:54:08.894 INFO:teuthology.orchestra.run.smithi125.stdout: "client_permissions": "true", 2021-03-24T21:54:08.894 INFO:teuthology.orchestra.run.smithi125.stdout: "client_quota_df": "true", 2021-03-24T21:54:08.894 INFO:teuthology.orchestra.run.smithi125.stdout: "client_readahead_max_bytes": "0", 2021-03-24T21:54:08.894 INFO:teuthology.orchestra.run.smithi125.stdout: "client_readahead_max_periods": "4", 2021-03-24T21:54:08.895 INFO:teuthology.orchestra.run.smithi125.stdout: "client_readahead_min": "131072", 2021-03-24T21:54:08.895 INFO:teuthology.orchestra.run.smithi125.stdout: "client_reconnect_stale": "false", 2021-03-24T21:54:08.895 INFO:teuthology.orchestra.run.smithi125.stdout: "client_shutdown_timeout": "30", 2021-03-24T21:54:08.895 INFO:teuthology.orchestra.run.smithi125.stdout: "client_snapdir": ".snap", 2021-03-24T21:54:08.896 INFO:teuthology.orchestra.run.smithi125.stdout: "client_tick_interval": "1", 2021-03-24T21:54:08.896 INFO:teuthology.orchestra.run.smithi125.stdout: "client_trace": "", 2021-03-24T21:54:08.896 INFO:teuthology.orchestra.run.smithi125.stdout: "client_try_dentry_invalidate": "false", 2021-03-24T21:54:08.896 INFO:teuthology.orchestra.run.smithi125.stdout: "client_use_faked_inos": "false", 2021-03-24T21:54:08.897 INFO:teuthology.orchestra.run.smithi125.stdout: "client_use_random_mds": "false", 2021-03-24T21:54:08.897 INFO:teuthology.orchestra.run.smithi125.stdout: "clog_to_graylog": "false", 2021-03-24T21:54:08.897 INFO:teuthology.orchestra.run.smithi125.stdout: "clog_to_graylog_host": "127.0.0.1", 2021-03-24T21:54:08.897 INFO:teuthology.orchestra.run.smithi125.stdout: "clog_to_graylog_port": "12201", 2021-03-24T21:54:08.898 INFO:teuthology.orchestra.run.smithi125.stdout: "clog_to_monitors": "default=true", 2021-03-24T21:54:08.898 INFO:teuthology.orchestra.run.smithi125.stdout: "clog_to_syslog": "false", 2021-03-24T21:54:08.898 INFO:teuthology.orchestra.run.smithi125.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2021-03-24T21:54:08.898 INFO:teuthology.orchestra.run.smithi125.stdout: "clog_to_syslog_level": "info", 2021-03-24T21:54:08.899 INFO:teuthology.orchestra.run.smithi125.stdout: "cluster_addr": "-", 2021-03-24T21:54:08.899 INFO:teuthology.orchestra.run.smithi125.stdout: "cluster_network": "", 2021-03-24T21:54:08.899 INFO:teuthology.orchestra.run.smithi125.stdout: "cluster_network_interface": "", 2021-03-24T21:54:08.899 INFO:teuthology.orchestra.run.smithi125.stdout: "compressor_zlib_isal": "false", 2021-03-24T21:54:08.900 INFO:teuthology.orchestra.run.smithi125.stdout: "compressor_zlib_level": "5", 2021-03-24T21:54:08.900 INFO:teuthology.orchestra.run.smithi125.stdout: "compressor_zlib_winsize": "-15", 2021-03-24T21:54:08.900 INFO:teuthology.orchestra.run.smithi125.stdout: "compressor_zstd_level": "1", 2021-03-24T21:54:08.901 INFO:teuthology.orchestra.run.smithi125.stdout: "container_image": "quay.ceph.io/ceph-ci/ceph@sha256:e9db06906b8d0a24f3ef6835ac5aa4b8eea2950a2560bad78f83d4f101eca0db", 2021-03-24T21:54:08.901 INFO:teuthology.orchestra.run.smithi125.stdout: "crash_dir": "/var/lib/ceph/crash", 2021-03-24T21:54:08.901 INFO:teuthology.orchestra.run.smithi125.stdout: "crimson_osd_obc_lru_size": "10", 2021-03-24T21:54:08.901 INFO:teuthology.orchestra.run.smithi125.stdout: "crimson_osd_scheduler_concurrency": "0", 2021-03-24T21:54:08.902 INFO:teuthology.orchestra.run.smithi125.stdout: "crush_location": "", 2021-03-24T21:54:08.902 INFO:teuthology.orchestra.run.smithi125.stdout: "crush_location_hook": "", 2021-03-24T21:54:08.902 INFO:teuthology.orchestra.run.smithi125.stdout: "crush_location_hook_timeout": "10", 2021-03-24T21:54:08.902 INFO:teuthology.orchestra.run.smithi125.stdout: "daemonize": "false", 2021-03-24T21:54:08.903 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_allow_any_pool_priority": "false", 2021-03-24T21:54:08.903 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_asok": "1/5", 2021-03-24T21:54:08.903 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_asok_assert_abort": "false", 2021-03-24T21:54:08.903 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_asserts_on_shutdown": "true", 2021-03-24T21:54:08.904 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_auth": "1/5", 2021-03-24T21:54:08.904 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_bdev": "1/3", 2021-03-24T21:54:08.904 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_bluefs": "1/5", 2021-03-24T21:54:08.905 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_bluestore": "1/5", 2021-03-24T21:54:08.905 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_buffer": "0/1", 2021-03-24T21:54:08.905 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_cephfs_mirror": "0/5", 2021-03-24T21:54:08.905 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_civetweb": "1/10", 2021-03-24T21:54:08.906 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_client": "0/5", 2021-03-24T21:54:08.906 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_compressor": "1/5", 2021-03-24T21:54:08.906 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_context": "0/1", 2021-03-24T21:54:08.906 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_crush": "1/1", 2021-03-24T21:54:08.907 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_crypto": "1/5", 2021-03-24T21:54:08.907 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_deliberately_leak_memory": "false", 2021-03-24T21:54:08.907 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_disable_randomized_ping": "false", 2021-03-24T21:54:08.907 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_dpdk": "1/5", 2021-03-24T21:54:08.908 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_eventtrace": "1/5", 2021-03-24T21:54:08.908 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_filer": "0/1", 2021-03-24T21:54:08.908 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_filestore": "1/3", 2021-03-24T21:54:08.908 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_finisher": "1/1", 2021-03-24T21:54:08.908 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_fuse": "1/5", 2021-03-24T21:54:08.909 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_heartbeat_testing_span": "0", 2021-03-24T21:54:08.909 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_heartbeatmap": "1/5", 2021-03-24T21:54:08.910 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_immutable_obj_cache": "0/5", 2021-03-24T21:54:08.910 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_javaclient": "1/5", 2021-03-24T21:54:08.910 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_journal": "1/3", 2021-03-24T21:54:08.910 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_journaler": "0/5", 2021-03-24T21:54:08.911 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_kstore": "1/5", 2021-03-24T21:54:08.911 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_leveldb": "4/5", 2021-03-24T21:54:08.911 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_lockdep": "0/1", 2021-03-24T21:54:08.911 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_mds": "1/5", 2021-03-24T21:54:08.912 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_mds_balancer": "1/5", 2021-03-24T21:54:08.912 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_mds_locker": "1/5", 2021-03-24T21:54:08.912 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_mds_log": "1/5", 2021-03-24T21:54:08.912 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_mds_log_expire": "1/5", 2021-03-24T21:54:08.913 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_mds_migrator": "1/5", 2021-03-24T21:54:08.913 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_memdb": "4/5", 2021-03-24T21:54:08.913 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_mgr": "1/5", 2021-03-24T21:54:08.913 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_mgrc": "1/5", 2021-03-24T21:54:08.913 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_mon": "20/20", 2021-03-24T21:54:08.914 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_monc": "0/10", 2021-03-24T21:54:08.914 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_ms": "1/1", 2021-03-24T21:54:08.914 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_none": "0/5", 2021-03-24T21:54:08.914 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_objclass": "0/5", 2021-03-24T21:54:08.915 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_objectcacher": "0/5", 2021-03-24T21:54:08.915 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_objecter": "0/1", 2021-03-24T21:54:08.915 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_optracker": "0/5", 2021-03-24T21:54:08.915 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_osd": "1/5", 2021-03-24T21:54:08.916 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_paxos": "20/20", 2021-03-24T21:54:08.916 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_perfcounter": "1/5", 2021-03-24T21:54:08.916 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_prioritycache": "1/5", 2021-03-24T21:54:08.916 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_rados": "0/5", 2021-03-24T21:54:08.917 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_rbd": "0/5", 2021-03-24T21:54:08.917 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_rbd_mirror": "0/5", 2021-03-24T21:54:08.917 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_rbd_pwl": "0/5", 2021-03-24T21:54:08.917 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_rbd_replay": "0/5", 2021-03-24T21:54:08.918 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_refs": "0/0", 2021-03-24T21:54:08.918 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_reserver": "1/1", 2021-03-24T21:54:08.918 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_rgw": "1/5", 2021-03-24T21:54:08.919 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_rgw_sync": "1/5", 2021-03-24T21:54:08.919 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_rocksdb": "4/5", 2021-03-24T21:54:08.919 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_striper": "0/1", 2021-03-24T21:54:08.919 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_test": "0/5", 2021-03-24T21:54:08.919 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_throttle": "1/1", 2021-03-24T21:54:08.920 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_timer": "0/1", 2021-03-24T21:54:08.920 INFO:teuthology.orchestra.run.smithi125.stdout: "debug_tp": "0/5", 2021-03-24T21:54:08.920 INFO:teuthology.orchestra.run.smithi125.stdout: "device_failure_prediction_mode": "none", 2021-03-24T21:54:08.920 INFO:teuthology.orchestra.run.smithi125.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2021-03-24T21:54:08.921 INFO:teuthology.orchestra.run.smithi125.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2021-03-24T21:54:08.921 INFO:teuthology.orchestra.run.smithi125.stdout: "err_to_graylog": "false", 2021-03-24T21:54:08.921 INFO:teuthology.orchestra.run.smithi125.stdout: "err_to_stderr": "true", 2021-03-24T21:54:08.921 INFO:teuthology.orchestra.run.smithi125.stdout: "err_to_syslog": "false", 2021-03-24T21:54:08.922 INFO:teuthology.orchestra.run.smithi125.stdout: "event_tracing": "false", 2021-03-24T21:54:08.922 INFO:teuthology.orchestra.run.smithi125.stdout: "fake_statfs_for_testing": "0", 2021-03-24T21:54:08.922 INFO:teuthology.orchestra.run.smithi125.stdout: "fatal_signal_handlers": "true", 2021-03-24T21:54:08.922 INFO:teuthology.orchestra.run.smithi125.stdout: "filer_max_purge_ops": "10", 2021-03-24T21:54:08.923 INFO:teuthology.orchestra.run.smithi125.stdout: "filer_max_truncate_ops": "128", 2021-03-24T21:54:08.923 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_apply_finisher_threads": "1", 2021-03-24T21:54:08.923 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_blackhole": "false", 2021-03-24T21:54:08.923 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_btrfs_clone_range": "true", 2021-03-24T21:54:08.924 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_btrfs_snap": "true", 2021-03-24T21:54:08.924 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_caller_concurrency": "10", 2021-03-24T21:54:08.924 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_collect_device_partition_information": "true", 2021-03-24T21:54:08.924 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_commit_timeout": "600.000000", 2021-03-24T21:54:08.924 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_debug_inject_read_err": "false", 2021-03-24T21:54:08.925 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_debug_omap_check": "false", 2021-03-24T21:54:08.925 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_debug_random_read_err": "0.000000", 2021-03-24T21:54:08.925 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_debug_verify_split": "false", 2021-03-24T21:54:08.925 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_dump_file": "", 2021-03-24T21:54:08.926 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2021-03-24T21:54:08.926 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_expected_throughput_ops": "200.000000", 2021-03-24T21:54:08.926 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_fadvise": "true", 2021-03-24T21:54:08.927 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_fail_eio": "true", 2021-03-24T21:54:08.927 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_fd_cache_shards": "16", 2021-03-24T21:54:08.927 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_fd_cache_size": "128", 2021-03-24T21:54:08.927 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_fiemap": "false", 2021-03-24T21:54:08.927 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_fiemap_threshold": "4096", 2021-03-24T21:54:08.928 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_fsync_flushes_journal_data": "false", 2021-03-24T21:54:08.928 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_index_retry_probability": "0.000000", 2021-03-24T21:54:08.928 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_inject_stall": "0", 2021-03-24T21:54:08.928 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_journal_parallel": "false", 2021-03-24T21:54:08.929 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_journal_trailing": "false", 2021-03-24T21:54:08.929 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_journal_writeahead": "false", 2021-03-24T21:54:08.929 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_kill_at": "0", 2021-03-24T21:54:08.929 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_max_alloc_hint_size": "1048576", 2021-03-24T21:54:08.930 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_max_inline_xattr_size": "0", 2021-03-24T21:54:08.930 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2021-03-24T21:54:08.930 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_max_inline_xattr_size_other": "512", 2021-03-24T21:54:08.930 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2021-03-24T21:54:08.930 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_max_inline_xattrs": "0", 2021-03-24T21:54:08.931 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2021-03-24T21:54:08.931 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_max_inline_xattrs_other": "2", 2021-03-24T21:54:08.931 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_max_inline_xattrs_xfs": "10", 2021-03-24T21:54:08.931 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_max_sync_interval": "5.000000", 2021-03-24T21:54:08.932 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_max_xattr_value_size": "0", 2021-03-24T21:54:08.932 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2021-03-24T21:54:08.932 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_max_xattr_value_size_other": "1024", 2021-03-24T21:54:08.932 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2021-03-24T21:54:08.933 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_merge_threshold": "-10", 2021-03-24T21:54:08.933 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_min_sync_interval": "0.010000", 2021-03-24T21:54:08.933 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_odsync_write": "false", 2021-03-24T21:54:08.933 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_omap_backend": "rocksdb", 2021-03-24T21:54:08.934 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_omap_backend_path": "", 2021-03-24T21:54:08.934 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_omap_header_cache_size": "1024", 2021-03-24T21:54:08.934 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_ondisk_finisher_threads": "1", 2021-03-24T21:54:08.934 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_op_thread_suicide_timeout": "180", 2021-03-24T21:54:08.934 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_op_thread_timeout": "60", 2021-03-24T21:54:08.935 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_op_threads": "2", 2021-03-24T21:54:08.935 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_punch_hole": "false", 2021-03-24T21:54:08.935 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2021-03-24T21:54:08.936 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2021-03-24T21:54:08.936 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2021-03-24T21:54:08.936 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_queue_high_threshhold": "0.900000", 2021-03-24T21:54:08.936 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_queue_low_threshhold": "0.300000", 2021-03-24T21:54:08.937 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_queue_max_bytes": "104857600", 2021-03-24T21:54:08.937 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2021-03-24T21:54:08.937 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2021-03-24T21:54:08.938 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2021-03-24T21:54:08.938 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_queue_max_ops": "50", 2021-03-24T21:54:08.938 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2021-03-24T21:54:08.938 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_seek_data_hole": "false", 2021-03-24T21:54:08.939 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_sloppy_crc": "false", 2021-03-24T21:54:08.939 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_sloppy_crc_block_size": "65536", 2021-03-24T21:54:08.939 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_splice": "false", 2021-03-24T21:54:08.939 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_split_multiple": "2", 2021-03-24T21:54:08.940 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_split_rand_factor": "20", 2021-03-24T21:54:08.940 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_update_to": "1000", 2021-03-24T21:54:08.940 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2021-03-24T21:54:08.940 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2021-03-24T21:54:08.941 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2021-03-24T21:54:08.941 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2021-03-24T21:54:08.941 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2021-03-24T21:54:08.941 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2021-03-24T21:54:08.942 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_wbthrottle_enable": "true", 2021-03-24T21:54:08.942 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2021-03-24T21:54:08.942 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2021-03-24T21:54:08.942 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2021-03-24T21:54:08.943 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2021-03-24T21:54:08.943 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2021-03-24T21:54:08.943 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2021-03-24T21:54:08.943 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_xfs_extsize": "false", 2021-03-24T21:54:08.944 INFO:teuthology.orchestra.run.smithi125.stdout: "filestore_zfs_snap": "false", 2021-03-24T21:54:08.944 INFO:teuthology.orchestra.run.smithi125.stdout: "fio_dir": "/tmp/fio", 2021-03-24T21:54:08.944 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3cfa339a-8cea-11eb-90b2-001a4aab830c", 2021-03-24T21:54:08.945 INFO:teuthology.orchestra.run.smithi125.stdout: "fuse_allow_other": "true", 2021-03-24T21:54:08.945 INFO:teuthology.orchestra.run.smithi125.stdout: "fuse_atomic_o_trunc": "true", 2021-03-24T21:54:08.945 INFO:teuthology.orchestra.run.smithi125.stdout: "fuse_big_writes": "true", 2021-03-24T21:54:08.946 INFO:teuthology.orchestra.run.smithi125.stdout: "fuse_debug": "false", 2021-03-24T21:54:08.946 INFO:teuthology.orchestra.run.smithi125.stdout: "fuse_default_permissions": "false", 2021-03-24T21:54:08.946 INFO:teuthology.orchestra.run.smithi125.stdout: "fuse_disable_pagecache": "false", 2021-03-24T21:54:08.946 INFO:teuthology.orchestra.run.smithi125.stdout: "fuse_max_write": "0", 2021-03-24T21:54:08.947 INFO:teuthology.orchestra.run.smithi125.stdout: "fuse_multithreaded": "true", 2021-03-24T21:54:08.947 INFO:teuthology.orchestra.run.smithi125.stdout: "fuse_require_active_mds": "true", 2021-03-24T21:54:08.947 INFO:teuthology.orchestra.run.smithi125.stdout: "fuse_set_user_groups": "true", 2021-03-24T21:54:08.947 INFO:teuthology.orchestra.run.smithi125.stdout: "fuse_splice_move": "true", 2021-03-24T21:54:08.948 INFO:teuthology.orchestra.run.smithi125.stdout: "fuse_splice_read": "true", 2021-03-24T21:54:08.948 INFO:teuthology.orchestra.run.smithi125.stdout: "fuse_splice_write": "true", 2021-03-24T21:54:08.948 INFO:teuthology.orchestra.run.smithi125.stdout: "fuse_syncfs_on_mksnap": "true", 2021-03-24T21:54:08.948 INFO:teuthology.orchestra.run.smithi125.stdout: "fuse_use_invalidate_cb": "true", 2021-03-24T21:54:08.949 INFO:teuthology.orchestra.run.smithi125.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2021-03-24T21:54:08.949 INFO:teuthology.orchestra.run.smithi125.stdout: "gss_target_name": "ceph", 2021-03-24T21:54:08.949 INFO:teuthology.orchestra.run.smithi125.stdout: "heartbeat_file": "", 2021-03-24T21:54:08.949 INFO:teuthology.orchestra.run.smithi125.stdout: "heartbeat_inject_failure": "0", 2021-03-24T21:54:08.950 INFO:teuthology.orchestra.run.smithi125.stdout: "heartbeat_interval": "5", 2021-03-24T21:54:08.950 INFO:teuthology.orchestra.run.smithi125.stdout: "host": "", 2021-03-24T21:54:08.950 INFO:teuthology.orchestra.run.smithi125.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2021-03-24T21:54:08.950 INFO:teuthology.orchestra.run.smithi125.stdout: "immutable_object_cache_max_inflight_ops": "128", 2021-03-24T21:54:08.951 INFO:teuthology.orchestra.run.smithi125.stdout: "immutable_object_cache_max_size": "1073741824", 2021-03-24T21:54:08.951 INFO:teuthology.orchestra.run.smithi125.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2021-03-24T21:54:08.951 INFO:teuthology.orchestra.run.smithi125.stdout: "immutable_object_cache_qos_bps_burst": "0", 2021-03-24T21:54:08.951 INFO:teuthology.orchestra.run.smithi125.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2021-03-24T21:54:08.952 INFO:teuthology.orchestra.run.smithi125.stdout: "immutable_object_cache_qos_bps_limit": "0", 2021-03-24T21:54:08.952 INFO:teuthology.orchestra.run.smithi125.stdout: "immutable_object_cache_qos_iops_burst": "0", 2021-03-24T21:54:08.952 INFO:teuthology.orchestra.run.smithi125.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2021-03-24T21:54:08.953 INFO:teuthology.orchestra.run.smithi125.stdout: "immutable_object_cache_qos_iops_limit": "0", 2021-03-24T21:54:08.953 INFO:teuthology.orchestra.run.smithi125.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2021-03-24T21:54:08.953 INFO:teuthology.orchestra.run.smithi125.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2021-03-24T21:54:08.954 INFO:teuthology.orchestra.run.smithi125.stdout: "immutable_object_cache_watermark": "0.900000", 2021-03-24T21:54:08.954 INFO:teuthology.orchestra.run.smithi125.stdout: "inject_early_sigterm": "false", 2021-03-24T21:54:08.954 INFO:teuthology.orchestra.run.smithi125.stdout: "journal_aio": "true", 2021-03-24T21:54:08.954 INFO:teuthology.orchestra.run.smithi125.stdout: "journal_align_min_size": "65536", 2021-03-24T21:54:08.955 INFO:teuthology.orchestra.run.smithi125.stdout: "journal_block_align": "true", 2021-03-24T21:54:08.955 INFO:teuthology.orchestra.run.smithi125.stdout: "journal_block_size": "4096", 2021-03-24T21:54:08.955 INFO:teuthology.orchestra.run.smithi125.stdout: "journal_dio": "true", 2021-03-24T21:54:08.955 INFO:teuthology.orchestra.run.smithi125.stdout: "journal_discard": "false", 2021-03-24T21:54:08.956 INFO:teuthology.orchestra.run.smithi125.stdout: "journal_force_aio": "false", 2021-03-24T21:54:08.956 INFO:teuthology.orchestra.run.smithi125.stdout: "journal_ignore_corruption": "false", 2021-03-24T21:54:08.956 INFO:teuthology.orchestra.run.smithi125.stdout: "journal_max_write_bytes": "10485760", 2021-03-24T21:54:08.956 INFO:teuthology.orchestra.run.smithi125.stdout: "journal_max_write_entries": "100", 2021-03-24T21:54:08.957 INFO:teuthology.orchestra.run.smithi125.stdout: "journal_replay_from": "0", 2021-03-24T21:54:08.957 INFO:teuthology.orchestra.run.smithi125.stdout: "journal_throttle_high_multiple": "0.000000", 2021-03-24T21:54:08.957 INFO:teuthology.orchestra.run.smithi125.stdout: "journal_throttle_high_threshhold": "0.900000", 2021-03-24T21:54:08.957 INFO:teuthology.orchestra.run.smithi125.stdout: "journal_throttle_low_threshhold": "0.600000", 2021-03-24T21:54:08.957 INFO:teuthology.orchestra.run.smithi125.stdout: "journal_throttle_max_multiple": "0.000000", 2021-03-24T21:54:08.958 INFO:teuthology.orchestra.run.smithi125.stdout: "journal_write_header_frequency": "0", 2021-03-24T21:54:08.958 INFO:teuthology.orchestra.run.smithi125.stdout: "journal_zero_on_create": "false", 2021-03-24T21:54:08.958 INFO:teuthology.orchestra.run.smithi125.stdout: "journaler_prefetch_periods": "10", 2021-03-24T21:54:08.958 INFO:teuthology.orchestra.run.smithi125.stdout: "journaler_prezero_periods": "5", 2021-03-24T21:54:08.959 INFO:teuthology.orchestra.run.smithi125.stdout: "journaler_write_head_interval": "15", 2021-03-24T21:54:08.959 INFO:teuthology.orchestra.run.smithi125.stdout: "key": "", 2021-03-24T21:54:08.959 INFO:teuthology.orchestra.run.smithi125.stdout: "keyfile": "", 2021-03-24T21:54:08.960 INFO:teuthology.orchestra.run.smithi125.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2021-03-24T21:54:08.960 INFO:teuthology.orchestra.run.smithi125.stdout: "kstore_backend": "rocksdb", 2021-03-24T21:54:08.960 INFO:teuthology.orchestra.run.smithi125.stdout: "kstore_default_stripe_size": "65536", 2021-03-24T21:54:08.961 INFO:teuthology.orchestra.run.smithi125.stdout: "kstore_fsck_on_mount": "false", 2021-03-24T21:54:08.961 INFO:teuthology.orchestra.run.smithi125.stdout: "kstore_fsck_on_mount_deep": "true", 2021-03-24T21:54:08.961 INFO:teuthology.orchestra.run.smithi125.stdout: "kstore_max_bytes": "67108864", 2021-03-24T21:54:08.962 INFO:teuthology.orchestra.run.smithi125.stdout: "kstore_max_ops": "512", 2021-03-24T21:54:08.962 INFO:teuthology.orchestra.run.smithi125.stdout: "kstore_nid_prealloc": "1024", 2021-03-24T21:54:08.962 INFO:teuthology.orchestra.run.smithi125.stdout: "kstore_onode_map_size": "1024", 2021-03-24T21:54:08.962 INFO:teuthology.orchestra.run.smithi125.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2021-03-24T21:54:08.963 INFO:teuthology.orchestra.run.smithi125.stdout: "kstore_sync_submit_transaction": "false", 2021-03-24T21:54:08.963 INFO:teuthology.orchestra.run.smithi125.stdout: "kstore_sync_transaction": "false", 2021-03-24T21:54:08.963 INFO:teuthology.orchestra.run.smithi125.stdout: "leveldb_block_size": "65536", 2021-03-24T21:54:08.963 INFO:teuthology.orchestra.run.smithi125.stdout: "leveldb_bloom_size": "0", 2021-03-24T21:54:08.964 INFO:teuthology.orchestra.run.smithi125.stdout: "leveldb_cache_size": "536870912", 2021-03-24T21:54:08.964 INFO:teuthology.orchestra.run.smithi125.stdout: "leveldb_compact_on_mount": "false", 2021-03-24T21:54:08.964 INFO:teuthology.orchestra.run.smithi125.stdout: "leveldb_compression": "false", 2021-03-24T21:54:08.964 INFO:teuthology.orchestra.run.smithi125.stdout: "leveldb_log": "", 2021-03-24T21:54:08.965 INFO:teuthology.orchestra.run.smithi125.stdout: "leveldb_log_to_ceph_log": "true", 2021-03-24T21:54:08.965 INFO:teuthology.orchestra.run.smithi125.stdout: "leveldb_max_open_files": "0", 2021-03-24T21:54:08.965 INFO:teuthology.orchestra.run.smithi125.stdout: "leveldb_paranoid": "false", 2021-03-24T21:54:08.965 INFO:teuthology.orchestra.run.smithi125.stdout: "leveldb_write_buffer_size": "33554432", 2021-03-24T21:54:08.966 INFO:teuthology.orchestra.run.smithi125.stdout: "librados_thread_count": "2", 2021-03-24T21:54:08.966 INFO:teuthology.orchestra.run.smithi125.stdout: "lockdep": "false", 2021-03-24T21:54:08.966 INFO:teuthology.orchestra.run.smithi125.stdout: "lockdep_force_backtrace": "false", 2021-03-24T21:54:08.966 INFO:teuthology.orchestra.run.smithi125.stdout: "log_coarse_timestamps": "true", 2021-03-24T21:54:08.967 INFO:teuthology.orchestra.run.smithi125.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2021-03-24T21:54:08.967 INFO:teuthology.orchestra.run.smithi125.stdout: "log_flush_on_exit": "false", 2021-03-24T21:54:08.967 INFO:teuthology.orchestra.run.smithi125.stdout: "log_graylog_host": "127.0.0.1", 2021-03-24T21:54:08.968 INFO:teuthology.orchestra.run.smithi125.stdout: "log_graylog_port": "12201", 2021-03-24T21:54:08.968 INFO:teuthology.orchestra.run.smithi125.stdout: "log_max_new": "1000", 2021-03-24T21:54:08.968 INFO:teuthology.orchestra.run.smithi125.stdout: "log_max_recent": "10000", 2021-03-24T21:54:08.968 INFO:teuthology.orchestra.run.smithi125.stdout: "log_stderr_prefix": "debug ", 2021-03-24T21:54:08.969 INFO:teuthology.orchestra.run.smithi125.stdout: "log_stop_at_utilization": "0.970000", 2021-03-24T21:54:08.969 INFO:teuthology.orchestra.run.smithi125.stdout: "log_to_file": "true", 2021-03-24T21:54:08.969 INFO:teuthology.orchestra.run.smithi125.stdout: "log_to_graylog": "false", 2021-03-24T21:54:08.969 INFO:teuthology.orchestra.run.smithi125.stdout: "log_to_stderr": "false", 2021-03-24T21:54:08.970 INFO:teuthology.orchestra.run.smithi125.stdout: "log_to_syslog": "false", 2021-03-24T21:54:08.970 INFO:teuthology.orchestra.run.smithi125.stdout: "max_rotating_auth_attempts": "10", 2021-03-24T21:54:08.970 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_action_on_write_error": "1", 2021-03-24T21:54:08.971 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_alternate_name_max": "8192", 2021-03-24T21:54:08.971 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_asio_thread_count": "2", 2021-03-24T21:54:08.971 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_bal_export_pin": "true", 2021-03-24T21:54:08.972 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_bal_fragment_dirs": "true", 2021-03-24T21:54:08.972 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2021-03-24T21:54:08.972 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_bal_fragment_interval": "5", 2021-03-24T21:54:08.972 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_bal_fragment_size_max": "100000", 2021-03-24T21:54:08.972 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_bal_idle_threshold": "0.000000", 2021-03-24T21:54:08.973 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_bal_interval": "10", 2021-03-24T21:54:08.973 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_bal_max": "-1", 2021-03-24T21:54:08.973 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_bal_max_until": "-1", 2021-03-24T21:54:08.973 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_bal_merge_size": "50", 2021-03-24T21:54:08.974 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_bal_midchunk": "0.300000", 2021-03-24T21:54:08.974 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_bal_min_rebalance": "0.100000", 2021-03-24T21:54:08.974 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_bal_min_start": "0.200000", 2021-03-24T21:54:08.975 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_bal_minchunk": "0.001000", 2021-03-24T21:54:08.975 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_bal_mode": "0", 2021-03-24T21:54:08.975 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_bal_need_max": "1.200000", 2021-03-24T21:54:08.975 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_bal_need_min": "0.800000", 2021-03-24T21:54:08.975 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_bal_replicate_threshold": "8000.000000", 2021-03-24T21:54:08.976 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_bal_sample_interval": "3.000000", 2021-03-24T21:54:08.976 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_bal_split_bits": "3", 2021-03-24T21:54:08.976 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_bal_split_rd": "25000.000000", 2021-03-24T21:54:08.976 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_bal_split_size": "10000", 2021-03-24T21:54:08.977 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_bal_split_wr": "10000.000000", 2021-03-24T21:54:08.977 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_bal_target_decay": "10.000000", 2021-03-24T21:54:08.977 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2021-03-24T21:54:08.977 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_beacon_grace": "15.000000", 2021-03-24T21:54:08.978 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_beacon_interval": "4.000000", 2021-03-24T21:54:08.978 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_cache_memory_limit": "4294967296", 2021-03-24T21:54:08.978 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_cache_mid": "0.700000", 2021-03-24T21:54:08.978 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_cache_release_free_interval": "10", 2021-03-24T21:54:08.979 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_cache_reservation": "0.050000", 2021-03-24T21:54:08.979 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_cache_trim_decay_rate": "1.000000", 2021-03-24T21:54:08.979 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_cache_trim_interval": "1", 2021-03-24T21:54:08.980 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_cache_trim_threshold": "262144", 2021-03-24T21:54:08.980 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2021-03-24T21:54:08.980 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2021-03-24T21:54:08.980 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_client_delegate_inos_pct": "50", 2021-03-24T21:54:08.981 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_client_prealloc_inos": "1000", 2021-03-24T21:54:08.981 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2021-03-24T21:54:08.981 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_damage_table_max_entries": "10000", 2021-03-24T21:54:08.981 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2021-03-24T21:54:08.982 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_debug_auth_pins": "false", 2021-03-24T21:54:08.982 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_debug_frag": "false", 2021-03-24T21:54:08.982 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_debug_scatterstat": "false", 2021-03-24T21:54:08.982 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_debug_subtrees": "false", 2021-03-24T21:54:08.983 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_decay_halflife": "5.000000", 2021-03-24T21:54:08.983 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_default_dir_hash": "2", 2021-03-24T21:54:08.983 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_defer_session_stale": "true", 2021-03-24T21:54:08.983 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_deny_all_reconnect": "false", 2021-03-24T21:54:08.984 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_dir_keys_per_op": "16384", 2021-03-24T21:54:08.984 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_dir_max_commit_size": "10", 2021-03-24T21:54:08.984 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_dirstat_min_interval": "1.000000", 2021-03-24T21:54:08.984 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_dump_cache_after_rejoin": "false", 2021-03-24T21:54:08.985 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_dump_cache_on_map": "false", 2021-03-24T21:54:08.985 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_dump_cache_threshold_file": "0", 2021-03-24T21:54:08.985 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2021-03-24T21:54:08.985 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_early_reply": "true", 2021-03-24T21:54:08.986 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_enable_op_tracker": "true", 2021-03-24T21:54:08.986 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_enforce_unique_name": "true", 2021-03-24T21:54:08.986 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_export_ephemeral_distributed": "true", 2021-03-24T21:54:08.986 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2021-03-24T21:54:08.987 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_export_ephemeral_random": "true", 2021-03-24T21:54:08.987 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_export_ephemeral_random_max": "0.010000", 2021-03-24T21:54:08.987 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_forward_all_requests_to_auth": "false", 2021-03-24T21:54:08.987 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_freeze_tree_timeout": "30.000000", 2021-03-24T21:54:08.988 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2021-03-24T21:54:08.988 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_health_cache_threshold": "1.500000", 2021-03-24T21:54:08.988 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_health_summarize_threshold": "10", 2021-03-24T21:54:08.989 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_heartbeat_grace": "15.000000", 2021-03-24T21:54:08.989 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_inject_migrator_session_race": "false", 2021-03-24T21:54:08.989 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2021-03-24T21:54:08.989 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_join_fs": "", 2021-03-24T21:54:08.990 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_journal_format": "1", 2021-03-24T21:54:08.990 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_kill_create_at": "0", 2021-03-24T21:54:08.990 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_kill_export_at": "0", 2021-03-24T21:54:08.990 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_kill_import_at": "0", 2021-03-24T21:54:08.991 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_kill_journal_at": "0", 2021-03-24T21:54:08.991 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_kill_journal_expire_at": "0", 2021-03-24T21:54:08.991 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_kill_journal_replay_at": "0", 2021-03-24T21:54:08.991 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_kill_link_at": "0", 2021-03-24T21:54:08.992 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_kill_mdstable_at": "0", 2021-03-24T21:54:08.992 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_kill_openc_at": "0", 2021-03-24T21:54:08.992 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_kill_rename_at": "0", 2021-03-24T21:54:08.992 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_log_events_per_segment": "1024", 2021-03-24T21:54:08.993 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_log_max_events": "-1", 2021-03-24T21:54:08.993 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_log_max_segments": "128", 2021-03-24T21:54:08.993 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_log_pause": "false", 2021-03-24T21:54:08.993 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_log_segment_size": "0", 2021-03-24T21:54:08.993 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_log_skip_corrupt_events": "false", 2021-03-24T21:54:08.994 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_log_warn_factor": "2.000000", 2021-03-24T21:54:08.994 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_max_caps_per_client": "1048576", 2021-03-24T21:54:08.994 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_max_completed_flushes": "100000", 2021-03-24T21:54:08.994 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_max_completed_requests": "100000", 2021-03-24T21:54:08.995 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_max_export_size": "20971520", 2021-03-24T21:54:08.995 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_max_file_recover": "32", 2021-03-24T21:54:08.995 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_max_purge_files": "64", 2021-03-24T21:54:08.995 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_max_purge_ops": "8192", 2021-03-24T21:54:08.996 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2021-03-24T21:54:08.996 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_max_retries_on_remount_failure": "5", 2021-03-24T21:54:08.996 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_max_scrub_ops_in_progress": "5", 2021-03-24T21:54:08.997 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_max_snaps_per_dir": "100", 2021-03-24T21:54:08.997 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_max_xattr_pairs_size": "65536", 2021-03-24T21:54:08.997 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_metrics_update_interval": "2", 2021-03-24T21:54:08.997 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_min_caps_per_client": "100", 2021-03-24T21:54:08.998 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_min_caps_working_set": "10000", 2021-03-24T21:54:08.998 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_mon_shutdown_timeout": "5.000000", 2021-03-24T21:54:08.998 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_numa_node": "-1", 2021-03-24T21:54:08.998 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_oft_prefetch_dirfrags": "true", 2021-03-24T21:54:08.999 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_op_complaint_time": "30.000000", 2021-03-24T21:54:08.999 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_op_history_duration": "600", 2021-03-24T21:54:08.999 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_op_history_size": "20", 2021-03-24T21:54:08.999 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_op_log_threshold": "5", 2021-03-24T21:54:09.000 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_ping_grace": "15", 2021-03-24T21:54:09.000 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_ping_interval": "5", 2021-03-24T21:54:09.000 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2021-03-24T21:54:09.000 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_recall_global_max_decay_threshold": "131072", 2021-03-24T21:54:09.001 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_recall_max_caps": "30000", 2021-03-24T21:54:09.001 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_recall_max_decay_rate": "1.500000", 2021-03-24T21:54:09.001 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_recall_max_decay_threshold": "131072", 2021-03-24T21:54:09.001 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_recall_warning_decay_rate": "60.000000", 2021-03-24T21:54:09.002 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_recall_warning_threshold": "262144", 2021-03-24T21:54:09.002 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_reconnect_timeout": "45.000000", 2021-03-24T21:54:09.002 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_replay_interval": "1.000000", 2021-03-24T21:54:09.002 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_replay_unsafe_with_closed_session": "false", 2021-03-24T21:54:09.003 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_request_load_average_decay_rate": "60.000000", 2021-03-24T21:54:09.003 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_root_ino_gid": "0", 2021-03-24T21:54:09.003 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_root_ino_uid": "0", 2021-03-24T21:54:09.003 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_scatter_nudge_interval": "5.000000", 2021-03-24T21:54:09.003 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_session_blocklist_on_evict": "true", 2021-03-24T21:54:09.004 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_session_blocklist_on_timeout": "true", 2021-03-24T21:54:09.004 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2021-03-24T21:54:09.004 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_session_cache_liveness_magnitude": "10", 2021-03-24T21:54:09.004 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_session_cap_acquisition_decay_rate": "10.000000", 2021-03-24T21:54:09.005 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_session_cap_acquisition_throttle": "500000", 2021-03-24T21:54:09.005 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2021-03-24T21:54:09.005 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_sessionmap_keys_per_op": "1024", 2021-03-24T21:54:09.006 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_shutdown_check": "0", 2021-03-24T21:54:09.006 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_skip_ino": "0", 2021-03-24T21:54:09.006 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_snap_max_uid": "4294967294", 2021-03-24T21:54:09.007 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_snap_min_uid": "0", 2021-03-24T21:54:09.007 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_snap_rstat": "false", 2021-03-24T21:54:09.007 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_task_status_update_interval": "2.000000", 2021-03-24T21:54:09.007 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_thrash_exports": "0", 2021-03-24T21:54:09.008 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_thrash_fragments": "0", 2021-03-24T21:54:09.008 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_tick_interval": "5.000000", 2021-03-24T21:54:09.008 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_verify_backtrace": "1", 2021-03-24T21:54:09.008 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_verify_scatter": "false", 2021-03-24T21:54:09.009 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_wipe_ino_prealloc": "false", 2021-03-24T21:54:09.009 INFO:teuthology.orchestra.run.smithi125.stdout: "mds_wipe_sessions": "false", 2021-03-24T21:54:09.009 INFO:teuthology.orchestra.run.smithi125.stdout: "mempool_debug": "false", 2021-03-24T21:54:09.009 INFO:teuthology.orchestra.run.smithi125.stdout: "memstore_debug_omit_block_device_write": "false", 2021-03-24T21:54:09.010 INFO:teuthology.orchestra.run.smithi125.stdout: "memstore_device_bytes": "1073741824", 2021-03-24T21:54:09.010 INFO:teuthology.orchestra.run.smithi125.stdout: "memstore_page_set": "false", 2021-03-24T21:54:09.010 INFO:teuthology.orchestra.run.smithi125.stdout: "memstore_page_size": "65536", 2021-03-24T21:54:09.010 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr_client_bytes": "134217728", 2021-03-24T21:54:09.011 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr_client_messages": "512", 2021-03-24T21:54:09.011 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2021-03-24T21:54:09.011 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr_connect_retry_interval": "1.000000", 2021-03-24T21:54:09.011 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2021-03-24T21:54:09.012 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2021-03-24T21:54:09.012 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr_disabled_modules": "", 2021-03-24T21:54:09.012 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr_initial_modules": "restful iostat", 2021-03-24T21:54:09.012 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr_mds_bytes": "134217728", 2021-03-24T21:54:09.013 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr_mds_messages": "128", 2021-03-24T21:54:09.013 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2021-03-24T21:54:09.013 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr_mon_bytes": "134217728", 2021-03-24T21:54:09.013 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr_mon_messages": "128", 2021-03-24T21:54:09.014 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr_osd_bytes": "536870912", 2021-03-24T21:54:09.014 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr_osd_messages": "8192", 2021-03-24T21:54:09.014 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr_service_beacon_grace": "60.000000", 2021-03-24T21:54:09.014 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr_stats_period": "5", 2021-03-24T21:54:09.015 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr_stats_threshold": "5", 2021-03-24T21:54:09.015 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr_tick_period": "2", 2021-03-24T21:54:09.015 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_accept_timeout_factor": "2.000000", 2021-03-24T21:54:09.016 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_allow_pool_delete": "true", 2021-03-24T21:54:09.016 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_allow_pool_size_one": "false", 2021-03-24T21:54:09.016 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_auth_validate_all_caps": "true", 2021-03-24T21:54:09.016 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2021-03-24T21:54:09.017 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2021-03-24T21:54:09.017 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_client_bytes": "104857600", 2021-03-24T21:54:09.017 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_client_directed_command_retry": "2", 2021-03-24T21:54:09.017 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_client_hunt_interval": "3.000000", 2021-03-24T21:54:09.018 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2021-03-24T21:54:09.018 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2021-03-24T21:54:09.018 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2021-03-24T21:54:09.019 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_client_hunt_parallel": "3", 2021-03-24T21:54:09.019 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_client_log_interval": "1.000000", 2021-03-24T21:54:09.019 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_client_max_log_entries_per_message": "1000", 2021-03-24T21:54:09.019 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_client_ping_interval": "10.000000", 2021-03-24T21:54:09.020 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_client_ping_timeout": "30.000000", 2021-03-24T21:54:09.020 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_clock_drift_allowed": "1.000000", 2021-03-24T21:54:09.020 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2021-03-24T21:54:09.021 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2021-03-24T21:54:09.021 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_cluster_log_file_level": "debug", 2021-03-24T21:54:09.021 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_cluster_log_to_file": "false", 2021-03-24T21:54:09.022 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_cluster_log_to_graylog": "false", 2021-03-24T21:54:09.022 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2021-03-24T21:54:09.022 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_cluster_log_to_graylog_port": "12201", 2021-03-24T21:54:09.022 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_cluster_log_to_stderr": "true", 2021-03-24T21:54:09.023 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_cluster_log_to_syslog": "default=false", 2021-03-24T21:54:09.023 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2021-03-24T21:54:09.023 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_cluster_log_to_syslog_level": "info", 2021-03-24T21:54:09.023 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_compact_on_bootstrap": "false", 2021-03-24T21:54:09.024 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_compact_on_start": "false", 2021-03-24T21:54:09.024 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_compact_on_trim": "true", 2021-03-24T21:54:09.024 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_con_tracker_persist_interval": "10", 2021-03-24T21:54:09.024 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_con_tracker_score_halflife": "43200", 2021-03-24T21:54:09.025 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_config_key_max_entry_size": "65536", 2021-03-24T21:54:09.025 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_cpu_threads": "4", 2021-03-24T21:54:09.025 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_crush_min_required_version": "hammer", 2021-03-24T21:54:09.025 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_daemon_bytes": "419430400", 2021-03-24T21:54:09.026 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2021-03-24T21:54:09.026 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_data_avail_crit": "5", 2021-03-24T21:54:09.026 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_data_avail_warn": "5", 2021-03-24T21:54:09.026 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_data_size_warn": "16106127360", 2021-03-24T21:54:09.027 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_debug_block_osdmap_trim": "false", 2021-03-24T21:54:09.027 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_debug_deprecated_as_obsolete": "false", 2021-03-24T21:54:09.027 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_debug_dump_json": "false", 2021-03-24T21:54:09.027 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2021-03-24T21:54:09.028 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_debug_dump_transactions": "false", 2021-03-24T21:54:09.028 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_debug_extra_checks": "false", 2021-03-24T21:54:09.028 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_debug_no_initial_persistent_features": "false", 2021-03-24T21:54:09.029 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2021-03-24T21:54:09.029 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_debug_no_require_octopus": "false", 2021-03-24T21:54:09.029 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_debug_no_require_pacific": "false", 2021-03-24T21:54:09.029 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2021-03-24T21:54:09.030 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_delta_reset_interval": "10.000000", 2021-03-24T21:54:09.030 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_dns_srv_name": "ceph-mon", 2021-03-24T21:54:09.030 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_election_default_strategy": "1", 2021-03-24T21:54:09.031 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_election_timeout": "5.000000", 2021-03-24T21:54:09.031 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2021-03-24T21:54:09.031 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_elector_ping_divisor": "2", 2021-03-24T21:54:09.031 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_elector_ping_timeout": "2.000000", 2021-03-24T21:54:09.032 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_enable_op_tracker": "true", 2021-03-24T21:54:09.032 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_fake_pool_delete": "false", 2021-03-24T21:54:09.032 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_force_quorum_join": "false", 2021-03-24T21:54:09.033 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_globalid_prealloc": "10000", 2021-03-24T21:54:09.033 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_health_detail_to_clog": "true", 2021-03-24T21:54:09.033 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_health_log_update_period": "5", 2021-03-24T21:54:09.033 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_health_max_detail": "50", 2021-03-24T21:54:09.034 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_health_to_clog": "true", 2021-03-24T21:54:09.034 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_health_to_clog_interval": "600", 2021-03-24T21:54:09.034 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2021-03-24T21:54:09.035 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_host": "172.21.15.125:6789/0", 2021-03-24T21:54:09.035 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_host_override": "", 2021-03-24T21:54:09.035 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_initial_members": "", 2021-03-24T21:54:09.035 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2021-03-24T21:54:09.035 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2021-03-24T21:54:09.036 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_inject_transaction_delay_max": "10.000000", 2021-03-24T21:54:09.036 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2021-03-24T21:54:09.036 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_keyvaluedb": "rocksdb", 2021-03-24T21:54:09.036 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_lease": "5.000000", 2021-03-24T21:54:09.037 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2021-03-24T21:54:09.037 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_lease_renew_interval_factor": "0.600000", 2021-03-24T21:54:09.037 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_log_max_summary": "50", 2021-03-24T21:54:09.037 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_max_log_entries_per_event": "4096", 2021-03-24T21:54:09.038 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_max_log_epochs": "500", 2021-03-24T21:54:09.038 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_max_mdsmap_epochs": "500", 2021-03-24T21:54:09.038 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_max_mgrmap_epochs": "500", 2021-03-24T21:54:09.039 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_max_osd": "10000", 2021-03-24T21:54:09.039 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_max_pg_per_osd": "10000", 2021-03-24T21:54:09.039 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_max_pool_pg_num": "65536", 2021-03-24T21:54:09.039 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_max_snap_prune_per_epoch": "100", 2021-03-24T21:54:09.040 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_mds_blocklist_interval": "86400.000000", 2021-03-24T21:54:09.040 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_mds_force_trim_to": "0", 2021-03-24T21:54:09.040 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_mds_skip_sanity": "false", 2021-03-24T21:54:09.041 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_memory_autotune": "true", 2021-03-24T21:54:09.041 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_memory_target": "2147483648", 2021-03-24T21:54:09.041 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_mgr_beacon_grace": "30", 2021-03-24T21:54:09.041 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2021-03-24T21:54:09.042 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_mgr_digest_period": "5", 2021-03-24T21:54:09.042 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_mgr_inactive_grace": "60", 2021-03-24T21:54:09.042 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_mgr_mkfs_grace": "240", 2021-03-24T21:54:09.042 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2021-03-24T21:54:09.043 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_min_osdmap_epochs": "50", 2021-03-24T21:54:09.043 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_op_complaint_time": "30", 2021-03-24T21:54:09.043 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_op_history_duration": "600", 2021-03-24T21:54:09.043 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_op_history_size": "20", 2021-03-24T21:54:09.044 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_op_history_slow_op_size": "20", 2021-03-24T21:54:09.044 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_op_history_slow_op_threshold": "10", 2021-03-24T21:54:09.044 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_op_log_threshold": "5", 2021-03-24T21:54:09.044 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_osd_adjust_down_out_interval": "true", 2021-03-24T21:54:09.044 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2021-03-24T21:54:09.045 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2021-03-24T21:54:09.045 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_osd_auto_mark_in": "false", 2021-03-24T21:54:09.045 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_osd_auto_mark_new_in": "true", 2021-03-24T21:54:09.045 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2021-03-24T21:54:09.046 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2021-03-24T21:54:09.046 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_osd_cache_size": "500", 2021-03-24T21:54:09.047 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_osd_cache_size_min": "134217728", 2021-03-24T21:54:09.047 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_osd_crush_smoke_test": "true", 2021-03-24T21:54:09.047 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_osd_destroyed_out_interval": "600", 2021-03-24T21:54:09.048 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_osd_down_out_interval": "600", 2021-03-24T21:54:09.048 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_osd_down_out_subtree_limit": "rack", 2021-03-24T21:54:09.048 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2021-03-24T21:54:09.048 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_osd_force_trim_to": "0", 2021-03-24T21:54:09.049 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_osd_full_ratio": "0.950000", 2021-03-24T21:54:09.049 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2021-03-24T21:54:09.049 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_osd_laggy_halflife": "3600", 2021-03-24T21:54:09.049 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_osd_laggy_max_interval": "300", 2021-03-24T21:54:09.050 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_osd_laggy_weight": "0.300000", 2021-03-24T21:54:09.050 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2021-03-24T21:54:09.050 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_osd_max_creating_pgs": "1024", 2021-03-24T21:54:09.050 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_osd_max_initial_pgs": "1024", 2021-03-24T21:54:09.051 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_osd_min_down_reporters": "2", 2021-03-24T21:54:09.051 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_osd_min_in_ratio": "0.750000", 2021-03-24T21:54:09.051 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_osd_min_up_ratio": "0.300000", 2021-03-24T21:54:09.051 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_osd_nearfull_ratio": "0.850000", 2021-03-24T21:54:09.052 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_osd_prime_pg_temp": "true", 2021-03-24T21:54:09.052 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2021-03-24T21:54:09.052 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2021-03-24T21:54:09.053 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_osd_report_timeout": "900", 2021-03-24T21:54:09.053 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_osd_reporter_subtree_level": "osd", 2021-03-24T21:54:09.053 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2021-03-24T21:54:09.054 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_osd_warn_num_repaired": "10", 2021-03-24T21:54:09.054 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_osd_warn_op_age": "32.000000", 2021-03-24T21:54:09.054 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_osdmap_full_prune_enabled": "true", 2021-03-24T21:54:09.054 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_osdmap_full_prune_interval": "2", 2021-03-24T21:54:09.055 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_osdmap_full_prune_min": "15", 2021-03-24T21:54:09.055 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_osdmap_full_prune_txsize": "2", 2021-03-24T21:54:09.055 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2021-03-24T21:54:09.055 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_pg_stuck_threshold": "60", 2021-03-24T21:54:09.056 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2021-03-24T21:54:09.056 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_pg_warn_min_objects": "10000", 2021-03-24T21:54:09.056 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_pg_warn_min_per_osd": "0", 2021-03-24T21:54:09.057 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_pg_warn_min_pool_objects": "1000", 2021-03-24T21:54:09.057 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_pool_quota_crit_threshold": "0", 2021-03-24T21:54:09.057 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_pool_quota_warn_threshold": "0", 2021-03-24T21:54:09.057 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_probe_timeout": "2.000000", 2021-03-24T21:54:09.058 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_reweight_max_change": "0.050000", 2021-03-24T21:54:09.058 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_reweight_max_osds": "4", 2021-03-24T21:54:09.058 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_reweight_min_bytes_per_osd": "10", 2021-03-24T21:54:09.058 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_reweight_min_pgs_per_osd": "4", 2021-03-24T21:54:09.059 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2021-03-24T21:54:09.059 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2021-03-24T21:54:09.059 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2021-03-24T21:54:09.059 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_scrub_interval": "86400", 2021-03-24T21:54:09.059 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_scrub_max_keys": "100", 2021-03-24T21:54:09.060 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_scrub_timeout": "300", 2021-03-24T21:54:09.060 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_session_timeout": "300", 2021-03-24T21:54:09.060 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_smart_report_timeout": "5", 2021-03-24T21:54:09.060 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_stat_smooth_intervals": "6", 2021-03-24T21:54:09.061 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2021-03-24T21:54:09.061 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_stretch_pool_min_size": "2", 2021-03-24T21:54:09.061 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_stretch_pool_size": "4", 2021-03-24T21:54:09.061 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2021-03-24T21:54:09.062 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_subscribe_interval": "86400.000000", 2021-03-24T21:54:09.062 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_sync_debug": "false", 2021-03-24T21:54:09.062 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_sync_max_payload_keys": "2000", 2021-03-24T21:54:09.062 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_sync_max_payload_size": "1048576", 2021-03-24T21:54:09.063 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_sync_provider_kill_at": "0", 2021-03-24T21:54:09.063 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_sync_requester_kill_at": "0", 2021-03-24T21:54:09.063 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_sync_timeout": "60.000000", 2021-03-24T21:54:09.063 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_target_pg_per_osd": "100", 2021-03-24T21:54:09.063 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_tick_interval": "5", 2021-03-24T21:54:09.064 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_timecheck_interval": "300.000000", 2021-03-24T21:54:09.064 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_timecheck_skew_interval": "30.000000", 2021-03-24T21:54:09.064 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_warn_older_version_delay": "604800", 2021-03-24T21:54:09.064 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2021-03-24T21:54:09.065 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2021-03-24T21:54:09.065 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2021-03-24T21:54:09.065 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2021-03-24T21:54:09.065 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_warn_on_misplaced": "false", 2021-03-24T21:54:09.066 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2021-03-24T21:54:09.066 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_warn_on_older_version": "true", 2021-03-24T21:54:09.066 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2021-03-24T21:54:09.066 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_warn_on_pool_no_app": "true", 2021-03-24T21:54:09.067 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_warn_on_pool_no_redundancy": "true", 2021-03-24T21:54:09.067 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2021-03-24T21:54:09.067 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2021-03-24T21:54:09.068 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2021-03-24T21:54:09.068 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_warn_on_too_few_osds": "false", 2021-03-24T21:54:09.068 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2021-03-24T21:54:09.068 INFO:teuthology.orchestra.run.smithi125.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2021-03-24T21:54:09.069 INFO:teuthology.orchestra.run.smithi125.stdout: "monmap": "", 2021-03-24T21:54:09.069 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_async_max_op_threads": "5", 2021-03-24T21:54:09.069 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_async_op_threads": "3", 2021-03-24T21:54:09.069 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_async_rdma_buffer_size": "131072", 2021-03-24T21:54:09.069 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_async_rdma_cm": "false", 2021-03-24T21:54:09.070 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_async_rdma_device_name": "", 2021-03-24T21:54:09.070 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_async_rdma_dscp": "96", 2021-03-24T21:54:09.070 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_async_rdma_enable_hugepage": "false", 2021-03-24T21:54:09.070 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_async_rdma_gid_idx": "0", 2021-03-24T21:54:09.071 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_async_rdma_local_gid": "", 2021-03-24T21:54:09.071 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_async_rdma_polling_us": "1000", 2021-03-24T21:54:09.071 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_async_rdma_port_num": "1", 2021-03-24T21:54:09.071 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_async_rdma_receive_buffers": "32768", 2021-03-24T21:54:09.072 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_async_rdma_receive_queue_len": "4096", 2021-03-24T21:54:09.072 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_async_rdma_roce_ver": "1", 2021-03-24T21:54:09.072 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_async_rdma_send_buffers": "1024", 2021-03-24T21:54:09.072 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_async_rdma_sl": "3", 2021-03-24T21:54:09.073 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_async_rdma_support_srq": "true", 2021-03-24T21:54:09.073 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_async_rdma_type": "ib", 2021-03-24T21:54:09.073 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_bind_before_connect": "false", 2021-03-24T21:54:09.073 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_bind_ipv4": "true", 2021-03-24T21:54:09.074 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_bind_ipv6": "false", 2021-03-24T21:54:09.074 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_bind_msgr1": "true", 2021-03-24T21:54:09.074 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_bind_msgr2": "false", 2021-03-24T21:54:09.074 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_bind_port_max": "7300", 2021-03-24T21:54:09.075 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_bind_port_min": "6800", 2021-03-24T21:54:09.075 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_bind_prefer_ipv4": "false", 2021-03-24T21:54:09.075 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_bind_retry_count": "3", 2021-03-24T21:54:09.075 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_bind_retry_delay": "5", 2021-03-24T21:54:09.076 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_blackhole_client": "false", 2021-03-24T21:54:09.076 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_blackhole_mds": "false", 2021-03-24T21:54:09.076 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_blackhole_mgr": "false", 2021-03-24T21:54:09.077 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_blackhole_mon": "false", 2021-03-24T21:54:09.077 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_blackhole_osd": "false", 2021-03-24T21:54:09.077 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_client_mode": "crc secure", 2021-03-24T21:54:09.077 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_cluster_mode": "crc secure", 2021-03-24T21:54:09.077 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_cluster_type": "", 2021-03-24T21:54:09.078 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_connection_idle_timeout": "900", 2021-03-24T21:54:09.078 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_connection_ready_timeout": "10", 2021-03-24T21:54:09.078 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_crc_data": "true", 2021-03-24T21:54:09.078 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_crc_header": "true", 2021-03-24T21:54:09.079 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_die_on_bad_msg": "false", 2021-03-24T21:54:09.079 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_die_on_bug": "true", 2021-03-24T21:54:09.079 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_die_on_old_message": "true", 2021-03-24T21:54:09.079 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_die_on_skipped_message": "false", 2021-03-24T21:54:09.080 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_die_on_unhandled_msg": "false", 2021-03-24T21:54:09.080 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_dispatch_throttle_bytes": "104857600", 2021-03-24T21:54:09.080 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_dpdk_coremask": "0xF", 2021-03-24T21:54:09.080 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_dpdk_debug_allow_loopback": "false", 2021-03-24T21:54:09.081 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2021-03-24T21:54:09.081 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_dpdk_host_ipv4_addr": "", 2021-03-24T21:54:09.081 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_dpdk_hugepages": "", 2021-03-24T21:54:09.081 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_dpdk_hw_flow_control": "true", 2021-03-24T21:54:09.081 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2021-03-24T21:54:09.082 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_dpdk_lro": "true", 2021-03-24T21:54:09.082 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_dpdk_memory_channel": "4", 2021-03-24T21:54:09.082 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2021-03-24T21:54:09.082 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_dpdk_pmd": "", 2021-03-24T21:54:09.083 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_dpdk_port_id": "0", 2021-03-24T21:54:09.083 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2021-03-24T21:54:09.083 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_dump_corrupt_message_level": "1", 2021-03-24T21:54:09.083 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_dump_on_send": "false", 2021-03-24T21:54:09.084 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_initial_backoff": "0.200000", 2021-03-24T21:54:09.084 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_inject_delay_max": "1.000000", 2021-03-24T21:54:09.084 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_inject_delay_probability": "0.000000", 2021-03-24T21:54:09.084 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_inject_delay_type": "", 2021-03-24T21:54:09.084 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_inject_internal_delays": "0.000000", 2021-03-24T21:54:09.085 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_inject_socket_failures": "0", 2021-03-24T21:54:09.085 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_learn_addr_from_peer": "true", 2021-03-24T21:54:09.085 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_max_accept_failures": "4", 2021-03-24T21:54:09.086 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_max_backoff": "15.000000", 2021-03-24T21:54:09.086 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_mon_client_mode": "secure crc", 2021-03-24T21:54:09.086 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_mon_cluster_mode": "secure crc", 2021-03-24T21:54:09.086 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_mon_service_mode": "secure crc", 2021-03-24T21:54:09.087 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2021-03-24T21:54:09.087 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_pq_min_cost": "65536", 2021-03-24T21:54:09.087 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_public_type": "", 2021-03-24T21:54:09.087 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_service_mode": "crc secure", 2021-03-24T21:54:09.088 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_tcp_listen_backlog": "512", 2021-03-24T21:54:09.088 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_tcp_nodelay": "true", 2021-03-24T21:54:09.088 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_tcp_prefetch_max_size": "4096", 2021-03-24T21:54:09.088 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_tcp_rcvbuf": "0", 2021-03-24T21:54:09.088 INFO:teuthology.orchestra.run.smithi125.stdout: "ms_type": "async+posix", 2021-03-24T21:54:09.089 INFO:teuthology.orchestra.run.smithi125.stdout: "no_config_file": "false", 2021-03-24T21:54:09.089 INFO:teuthology.orchestra.run.smithi125.stdout: "objecter_completion_locks_per_session": "32", 2021-03-24T21:54:09.089 INFO:teuthology.orchestra.run.smithi125.stdout: "objecter_debug_inject_relock_delay": "false", 2021-03-24T21:54:09.089 INFO:teuthology.orchestra.run.smithi125.stdout: "objecter_inflight_op_bytes": "104857600", 2021-03-24T21:54:09.090 INFO:teuthology.orchestra.run.smithi125.stdout: "objecter_inflight_ops": "1024", 2021-03-24T21:54:09.090 INFO:teuthology.orchestra.run.smithi125.stdout: "objecter_inject_no_watch_ping": "false", 2021-03-24T21:54:09.090 INFO:teuthology.orchestra.run.smithi125.stdout: "objecter_retry_writes_after_first_reply": "false", 2021-03-24T21:54:09.090 INFO:teuthology.orchestra.run.smithi125.stdout: "objecter_tick_interval": "5.000000", 2021-03-24T21:54:09.091 INFO:teuthology.orchestra.run.smithi125.stdout: "objecter_timeout": "10.000000", 2021-03-24T21:54:09.091 INFO:teuthology.orchestra.run.smithi125.stdout: "objectstore_blackhole": "false", 2021-03-24T21:54:09.091 INFO:teuthology.orchestra.run.smithi125.stdout: "openssl_engine_opts": "", 2021-03-24T21:54:09.091 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_agent_delay_time": "5.000000", 2021-03-24T21:54:09.091 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_agent_hist_halflife": "1000", 2021-03-24T21:54:09.092 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_agent_max_low_ops": "2", 2021-03-24T21:54:09.092 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_agent_max_ops": "4", 2021-03-24T21:54:09.092 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_agent_min_evict_effort": "0.100000", 2021-03-24T21:54:09.092 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_agent_quantize_effort": "0.100000", 2021-03-24T21:54:09.093 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_agent_slop": "0.020000", 2021-03-24T21:54:09.093 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_allow_recovery_below_min_size": "true", 2021-03-24T21:54:09.093 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_asio_thread_count": "2", 2021-03-24T21:54:09.093 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_async_recovery_min_cost": "100", 2021-03-24T21:54:09.094 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_auto_mark_unfound_lost": "false", 2021-03-24T21:54:09.094 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_backfill_retry_interval": "30.000000", 2021-03-24T21:54:09.094 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_backfill_scan_max": "512", 2021-03-24T21:54:09.094 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_backfill_scan_min": "64", 2021-03-24T21:54:09.095 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_backoff_on_degraded": "false", 2021-03-24T21:54:09.095 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_backoff_on_peering": "false", 2021-03-24T21:54:09.095 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_backoff_on_unfound": "true", 2021-03-24T21:54:09.095 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_beacon_report_interval": "300", 2021-03-24T21:54:09.095 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_bench_duration": "30", 2021-03-24T21:54:09.096 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_bench_large_size_max_throughput": "104857600", 2021-03-24T21:54:09.096 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_bench_max_block_size": "67108864", 2021-03-24T21:54:09.096 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_bench_small_size_max_iops": "100", 2021-03-24T21:54:09.096 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_blkin_trace_all": "false", 2021-03-24T21:54:09.097 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2021-03-24T21:54:09.097 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2021-03-24T21:54:09.097 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_check_for_log_corruption": "false", 2021-03-24T21:54:09.097 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_check_max_object_name_len_on_startup": "true", 2021-03-24T21:54:09.097 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2021-03-24T21:54:09.098 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2021-03-24T21:54:09.098 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2021-03-24T21:54:09.098 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_class_update_on_start": "true", 2021-03-24T21:54:09.098 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_client_message_cap": "0", 2021-03-24T21:54:09.099 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_client_message_size_cap": "524288000", 2021-03-24T21:54:09.099 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_client_op_priority": "63", 2021-03-24T21:54:09.099 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_client_watch_timeout": "30", 2021-03-24T21:54:09.099 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_command_max_records": "256", 2021-03-24T21:54:09.100 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_command_thread_suicide_timeout": "900", 2021-03-24T21:54:09.100 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_command_thread_timeout": "600", 2021-03-24T21:54:09.100 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_compact_on_start": "false", 2021-03-24T21:54:09.100 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_copyfrom_max_chunk": "8388608", 2021-03-24T21:54:09.100 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_crush_chooseleaf_type": "1", 2021-03-24T21:54:09.101 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_crush_initial_weight": "-1.000000", 2021-03-24T21:54:09.101 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_crush_update_on_start": "true", 2021-03-24T21:54:09.101 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_crush_update_weight_set": "true", 2021-03-24T21:54:09.101 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2021-03-24T21:54:09.102 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2021-03-24T21:54:09.102 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2021-03-24T21:54:09.102 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_debug_drop_ping_duration": "0", 2021-03-24T21:54:09.102 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_debug_drop_ping_probability": "0.000000", 2021-03-24T21:54:09.103 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_debug_feed_pullee": "-1", 2021-03-24T21:54:09.103 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_debug_inject_copyfrom_error": "false", 2021-03-24T21:54:09.103 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2021-03-24T21:54:09.103 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2021-03-24T21:54:09.104 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_debug_misdirected_ops": "false", 2021-03-24T21:54:09.104 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_debug_no_acting_change": "false", 2021-03-24T21:54:09.104 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_debug_no_purge_strays": "false", 2021-03-24T21:54:09.104 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_debug_op_order": "false", 2021-03-24T21:54:09.104 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_debug_pg_log_writeout": "false", 2021-03-24T21:54:09.105 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_debug_pretend_recovery_active": "false", 2021-03-24T21:54:09.105 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_debug_random_push_read_error": "0.000000", 2021-03-24T21:54:09.105 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2021-03-24T21:54:09.105 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_debug_shutdown": "false", 2021-03-24T21:54:09.106 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2021-03-24T21:54:09.106 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2021-03-24T21:54:09.106 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_debug_verify_cached_snaps": "false", 2021-03-24T21:54:09.106 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_debug_verify_missing_on_start": "false", 2021-03-24T21:54:09.107 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_debug_verify_snaps": "false", 2021-03-24T21:54:09.107 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_debug_verify_stray_on_activate": "false", 2021-03-24T21:54:09.107 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_deep_scrub_interval": "604800.000000", 2021-03-24T21:54:09.107 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_deep_scrub_keys": "1024", 2021-03-24T21:54:09.108 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2021-03-24T21:54:09.108 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2021-03-24T21:54:09.108 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2021-03-24T21:54:09.108 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_deep_scrub_stride": "524288", 2021-03-24T21:54:09.109 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2021-03-24T21:54:09.109 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_default_data_pool_replay_window": "45", 2021-03-24T21:54:09.109 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_default_notify_timeout": "30", 2021-03-24T21:54:09.109 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_delete_sleep": "0.000000", 2021-03-24T21:54:09.110 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_delete_sleep_hdd": "5.000000", 2021-03-24T21:54:09.110 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_delete_sleep_hybrid": "1.000000", 2021-03-24T21:54:09.110 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_delete_sleep_ssd": "1.000000", 2021-03-24T21:54:09.110 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_discard_disconnected_ops": "true", 2021-03-24T21:54:09.110 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_enable_op_tracker": "true", 2021-03-24T21:54:09.111 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_erasure_code_plugins": "jerasure lrc isa", 2021-03-24T21:54:09.111 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_failsafe_full_ratio": "0.970000", 2021-03-24T21:54:09.111 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_fast_fail_on_connection_refused": "true", 2021-03-24T21:54:09.112 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_fast_info": "true", 2021-03-24T21:54:09.112 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_fast_shutdown": "true", 2021-03-24T21:54:09.112 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_fast_shutdown_notify_mon": "false", 2021-03-24T21:54:09.112 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_find_best_info_ignore_history_les": "false", 2021-03-24T21:54:09.113 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_force_auth_primary_missing_objects": "100", 2021-03-24T21:54:09.113 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2021-03-24T21:54:09.113 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_function_tracing": "false", 2021-03-24T21:54:09.113 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_heartbeat_grace": "20", 2021-03-24T21:54:09.114 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_heartbeat_interval": "6", 2021-03-24T21:54:09.114 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2021-03-24T21:54:09.114 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_heartbeat_min_peers": "10", 2021-03-24T21:54:09.114 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_heartbeat_min_size": "2000", 2021-03-24T21:54:09.114 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_heartbeat_stale": "600", 2021-03-24T21:54:09.115 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2021-03-24T21:54:09.115 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_hit_set_max_size": "100000", 2021-03-24T21:54:09.115 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_hit_set_min_size": "1000", 2021-03-24T21:54:09.115 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_hit_set_namespace": ".ceph-internal", 2021-03-24T21:54:09.116 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_ignore_stale_divergent_priors": "false", 2021-03-24T21:54:09.116 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2021-03-24T21:54:09.116 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_inject_failure_on_pg_removal": "false", 2021-03-24T21:54:09.116 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2021-03-24T21:54:09.117 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_journal_flush_on_shutdown": "true", 2021-03-24T21:54:09.117 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_journal_size": "5120", 2021-03-24T21:54:09.117 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_kill_backfill_at": "0", 2021-03-24T21:54:09.117 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_loop_before_reset_tphandle": "64", 2021-03-24T21:54:09.118 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_map_cache_size": "50", 2021-03-24T21:54:09.118 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_map_dedup": "true", 2021-03-24T21:54:09.118 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_map_message_max": "40", 2021-03-24T21:54:09.118 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_map_message_max_bytes": "10485760", 2021-03-24T21:54:09.119 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_map_share_max_epochs": "40", 2021-03-24T21:54:09.119 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_max_attr_name_len": "100", 2021-03-24T21:54:09.119 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_max_attr_size": "0", 2021-03-24T21:54:09.119 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_max_backfills": "1", 2021-03-24T21:54:09.120 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_max_markdown_count": "5", 2021-03-24T21:54:09.120 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_max_markdown_period": "600", 2021-03-24T21:54:09.120 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_max_object_name_len": "2048", 2021-03-24T21:54:09.121 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_max_object_namespace_len": "256", 2021-03-24T21:54:09.121 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_max_object_size": "134217728", 2021-03-24T21:54:09.121 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2021-03-24T21:54:09.121 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_max_omap_entries_per_request": "1024", 2021-03-24T21:54:09.122 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_max_pg_blocked_by": "16", 2021-03-24T21:54:09.122 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_max_pg_log_entries": "10000", 2021-03-24T21:54:09.122 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2021-03-24T21:54:09.122 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_max_pgls": "1024", 2021-03-24T21:54:09.123 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_max_push_cost": "8388608", 2021-03-24T21:54:09.123 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_max_push_objects": "10", 2021-03-24T21:54:09.123 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_max_scrubs": "1", 2021-03-24T21:54:09.123 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2021-03-24T21:54:09.123 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_max_trimming_pgs": "2", 2021-03-24T21:54:09.124 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_max_write_op_reply_len": "32", 2021-03-24T21:54:09.124 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_max_write_size": "90", 2021-03-24T21:54:09.124 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_mclock_cost_per_byte_usec": "0.000000", 2021-03-24T21:54:09.124 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_mclock_cost_per_byte_usec_hdd": "5.200000", 2021-03-24T21:54:09.125 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_mclock_cost_per_byte_usec_ssd": "0.011000", 2021-03-24T21:54:09.125 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_mclock_cost_per_io_usec": "0.000000", 2021-03-24T21:54:09.125 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_mclock_cost_per_io_usec_hdd": "25000.000000", 2021-03-24T21:54:09.125 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_mclock_cost_per_io_usec_ssd": "50.000000", 2021-03-24T21:54:09.126 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_mclock_max_capacity_iops": "0.000000", 2021-03-24T21:54:09.126 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2021-03-24T21:54:09.126 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2021-03-24T21:54:09.126 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_mclock_profile": "high_client_ops", 2021-03-24T21:54:09.127 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2021-03-24T21:54:09.127 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_mclock_scheduler_background_best_effort_lim": "999999", 2021-03-24T21:54:09.127 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_mclock_scheduler_background_best_effort_res": "1", 2021-03-24T21:54:09.127 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2021-03-24T21:54:09.128 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_mclock_scheduler_background_recovery_lim": "999999", 2021-03-24T21:54:09.128 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_mclock_scheduler_background_recovery_res": "1", 2021-03-24T21:54:09.128 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2021-03-24T21:54:09.129 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_mclock_scheduler_client_lim": "999999", 2021-03-24T21:54:09.129 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_mclock_scheduler_client_res": "1", 2021-03-24T21:54:09.129 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_mclock_scheduler_client_wgt": "1", 2021-03-24T21:54:09.129 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_memory_base": "805306368", 2021-03-24T21:54:09.130 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_memory_cache_min": "134217728", 2021-03-24T21:54:09.130 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_memory_cache_resize_interval": "1.000000", 2021-03-24T21:54:09.130 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_memory_expected_fragmentation": "0.150000", 2021-03-24T21:54:09.130 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_memory_target": "4294967296", 2021-03-24T21:54:09.131 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2021-03-24T21:54:09.131 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_min_pg_log_entries": "250", 2021-03-24T21:54:09.131 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_min_recovery_priority": "0", 2021-03-24T21:54:09.131 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_mon_heartbeat_interval": "30", 2021-03-24T21:54:09.132 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2021-03-24T21:54:09.132 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_mon_report_interval": "5", 2021-03-24T21:54:09.132 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_mon_report_max_in_flight": "2", 2021-03-24T21:54:09.132 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_mon_shutdown_timeout": "5.000000", 2021-03-24T21:54:09.132 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_num_cache_shards": "32", 2021-03-24T21:54:09.133 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_num_op_tracker_shard": "32", 2021-03-24T21:54:09.133 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_numa_auto_affinity": "true", 2021-03-24T21:54:09.133 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_numa_node": "-1", 2021-03-24T21:54:09.133 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_numa_prefer_iface": "true", 2021-03-24T21:54:09.134 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_object_clean_region_max_num_intervals": "10", 2021-03-24T21:54:09.134 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_objecter_finishers": "1", 2021-03-24T21:54:09.134 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_objectstore": "bluestore", 2021-03-24T21:54:09.134 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_objectstore_fuse": "false", 2021-03-24T21:54:09.135 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_objectstore_tracing": "false", 2021-03-24T21:54:09.135 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_op_complaint_time": "30.000000", 2021-03-24T21:54:09.135 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_op_history_duration": "600", 2021-03-24T21:54:09.135 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_op_history_size": "20", 2021-03-24T21:54:09.136 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_op_history_slow_op_size": "20", 2021-03-24T21:54:09.136 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2021-03-24T21:54:09.136 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_op_log_threshold": "5", 2021-03-24T21:54:09.136 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_op_num_shards": "0", 2021-03-24T21:54:09.137 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_op_num_shards_hdd": "5", 2021-03-24T21:54:09.137 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_op_num_shards_ssd": "8", 2021-03-24T21:54:09.137 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_op_num_threads_per_shard": "0", 2021-03-24T21:54:09.138 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2021-03-24T21:54:09.138 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2021-03-24T21:54:09.138 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2021-03-24T21:54:09.138 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_op_pq_min_cost": "65536", 2021-03-24T21:54:09.139 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_op_queue": "wpq", 2021-03-24T21:54:09.139 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_op_queue_cut_off": "high", 2021-03-24T21:54:09.139 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_op_thread_suicide_timeout": "150", 2021-03-24T21:54:09.139 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_op_thread_timeout": "15", 2021-03-24T21:54:09.140 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_open_classes_on_start": "true", 2021-03-24T21:54:09.140 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_os_flags": "0", 2021-03-24T21:54:09.140 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_peering_op_priority": "255", 2021-03-24T21:54:09.140 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_pg_delete_cost": "1048576", 2021-03-24T21:54:09.141 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_pg_delete_priority": "5", 2021-03-24T21:54:09.141 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2021-03-24T21:54:09.141 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2021-03-24T21:54:09.141 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_pg_log_dups_tracked": "3000", 2021-03-24T21:54:09.141 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_pg_log_trim_max": "10000", 2021-03-24T21:54:09.142 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_pg_log_trim_min": "100", 2021-03-24T21:54:09.142 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2021-03-24T21:54:09.142 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_pg_object_context_cache_count": "64", 2021-03-24T21:54:09.142 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_pg_stat_report_interval_max": "500", 2021-03-24T21:54:09.143 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2021-03-24T21:54:09.143 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_pool_default_cache_min_evict_age": "0", 2021-03-24T21:54:09.143 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_pool_default_cache_min_flush_age": "0", 2021-03-24T21:54:09.143 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2021-03-24T21:54:09.144 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2021-03-24T21:54:09.144 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2021-03-24T21:54:09.144 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_pool_default_crush_rule": "-1", 2021-03-24T21:54:09.144 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_pool_default_ec_fast_read": "false", 2021-03-24T21:54:09.145 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd", 2021-03-24T21:54:09.145 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_pool_default_flag_hashpspool": "true", 2021-03-24T21:54:09.145 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_pool_default_flag_nodelete": "false", 2021-03-24T21:54:09.145 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_pool_default_flag_nopgchange": "false", 2021-03-24T21:54:09.146 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_pool_default_flag_nosizechange": "false", 2021-03-24T21:54:09.146 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_pool_default_flags": "0", 2021-03-24T21:54:09.146 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2021-03-24T21:54:09.146 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_pool_default_min_size": "0", 2021-03-24T21:54:09.147 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2021-03-24T21:54:09.147 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_pool_default_pg_num": "32", 2021-03-24T21:54:09.147 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_pool_default_pgp_num": "0", 2021-03-24T21:54:09.148 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2021-03-24T21:54:09.148 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_pool_default_size": "3", 2021-03-24T21:54:09.148 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_pool_default_type": "replicated", 2021-03-24T21:54:09.148 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2021-03-24T21:54:09.149 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_pool_use_gmt_hitset": "true", 2021-03-24T21:54:09.149 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_push_per_object_cost": "1000", 2021-03-24T21:54:09.149 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_read_ec_check_for_errors": "false", 2021-03-24T21:54:09.149 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_recover_clone_overlap": "true", 2021-03-24T21:54:09.149 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_recover_clone_overlap_limit": "10", 2021-03-24T21:54:09.150 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_recovery_cost": "20971520", 2021-03-24T21:54:09.150 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_recovery_delay_start": "0.000000", 2021-03-24T21:54:09.150 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_recovery_max_active": "0", 2021-03-24T21:54:09.150 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_recovery_max_active_hdd": "3", 2021-03-24T21:54:09.151 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_recovery_max_active_ssd": "10", 2021-03-24T21:54:09.151 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_recovery_max_chunk": "8388608", 2021-03-24T21:54:09.151 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2021-03-24T21:54:09.151 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_recovery_max_single_start": "1", 2021-03-24T21:54:09.152 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_recovery_op_priority": "3", 2021-03-24T21:54:09.152 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_recovery_op_warn_multiple": "16", 2021-03-24T21:54:09.152 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_recovery_priority": "5", 2021-03-24T21:54:09.152 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_recovery_retry_interval": "30.000000", 2021-03-24T21:54:09.153 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_recovery_sleep": "0.000000", 2021-03-24T21:54:09.153 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_recovery_sleep_hdd": "0.100000", 2021-03-24T21:54:09.153 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2021-03-24T21:54:09.153 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_recovery_sleep_ssd": "0.000000", 2021-03-24T21:54:09.154 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_repair_during_recovery": "false", 2021-03-24T21:54:09.154 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_requested_scrub_priority": "120", 2021-03-24T21:54:09.154 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_rollback_to_cluster_snap": "", 2021-03-24T21:54:09.154 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_scrub_auto_repair": "false", 2021-03-24T21:54:09.155 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_scrub_auto_repair_num_errors": "5", 2021-03-24T21:54:09.155 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_scrub_backoff_ratio": "0.660000", 2021-03-24T21:54:09.155 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_scrub_begin_hour": "0", 2021-03-24T21:54:09.155 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_scrub_begin_week_day": "0", 2021-03-24T21:54:09.156 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_scrub_chunk_max": "25", 2021-03-24T21:54:09.156 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_scrub_chunk_min": "5", 2021-03-24T21:54:09.156 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_scrub_cost": "52428800", 2021-03-24T21:54:09.156 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_scrub_during_recovery": "false", 2021-03-24T21:54:09.157 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_scrub_end_hour": "0", 2021-03-24T21:54:09.157 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_scrub_end_week_day": "0", 2021-03-24T21:54:09.157 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_scrub_extended_sleep": "0.000000", 2021-03-24T21:54:09.157 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2021-03-24T21:54:09.158 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_scrub_invalid_stats": "true", 2021-03-24T21:54:09.158 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_scrub_load_threshold": "0.500000", 2021-03-24T21:54:09.158 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_scrub_max_interval": "604800.000000", 2021-03-24T21:54:09.158 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_scrub_max_preemptions": "5", 2021-03-24T21:54:09.159 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_scrub_min_interval": "86400.000000", 2021-03-24T21:54:09.159 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_scrub_priority": "5", 2021-03-24T21:54:09.159 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_scrub_sleep": "0.000000", 2021-03-24T21:54:09.159 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_shutdown_pgref_assert": "false", 2021-03-24T21:54:09.160 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_skip_data_digest": "false", 2021-03-24T21:54:09.160 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_smart_report_timeout": "5", 2021-03-24T21:54:09.160 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_snap_trim_cost": "1048576", 2021-03-24T21:54:09.160 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_snap_trim_priority": "5", 2021-03-24T21:54:09.160 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_snap_trim_sleep": "0.000000", 2021-03-24T21:54:09.161 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2021-03-24T21:54:09.161 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2021-03-24T21:54:09.161 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2021-03-24T21:54:09.161 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2021-03-24T21:54:09.162 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_target_transaction_size": "30", 2021-03-24T21:54:09.162 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_tier_default_cache_hit_set_count": "4", 2021-03-24T21:54:09.162 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2021-03-24T21:54:09.162 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2021-03-24T21:54:09.163 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2021-03-24T21:54:09.163 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2021-03-24T21:54:09.163 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2021-03-24T21:54:09.164 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2021-03-24T21:54:09.164 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_tier_default_cache_mode": "writeback", 2021-03-24T21:54:09.164 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2021-03-24T21:54:09.164 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_tier_promote_max_objects_sec": "25", 2021-03-24T21:54:09.165 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_tracing": "false", 2021-03-24T21:54:09.165 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_use_stale_snap": "false", 2021-03-24T21:54:09.165 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2021-03-24T21:54:09.166 INFO:teuthology.orchestra.run.smithi125.stdout: "osdc_blkin_trace_all": "false", 2021-03-24T21:54:09.166 INFO:teuthology.orchestra.run.smithi125.stdout: "paxos_kill_at": "0", 2021-03-24T21:54:09.166 INFO:teuthology.orchestra.run.smithi125.stdout: "paxos_max_join_drift": "10", 2021-03-24T21:54:09.166 INFO:teuthology.orchestra.run.smithi125.stdout: "paxos_min": "500", 2021-03-24T21:54:09.167 INFO:teuthology.orchestra.run.smithi125.stdout: "paxos_min_wait": "0.050000", 2021-03-24T21:54:09.167 INFO:teuthology.orchestra.run.smithi125.stdout: "paxos_propose_interval": "1.000000", 2021-03-24T21:54:09.167 INFO:teuthology.orchestra.run.smithi125.stdout: "paxos_service_trim_max": "500", 2021-03-24T21:54:09.167 INFO:teuthology.orchestra.run.smithi125.stdout: "paxos_service_trim_min": "10", 2021-03-24T21:54:09.168 INFO:teuthology.orchestra.run.smithi125.stdout: "paxos_stash_full_interval": "25", 2021-03-24T21:54:09.168 INFO:teuthology.orchestra.run.smithi125.stdout: "paxos_trim_max": "500", 2021-03-24T21:54:09.168 INFO:teuthology.orchestra.run.smithi125.stdout: "paxos_trim_min": "250", 2021-03-24T21:54:09.168 INFO:teuthology.orchestra.run.smithi125.stdout: "perf": "true", 2021-03-24T21:54:09.168 INFO:teuthology.orchestra.run.smithi125.stdout: "pid_file": "", 2021-03-24T21:54:09.169 INFO:teuthology.orchestra.run.smithi125.stdout: "plugin_crypto_accelerator": "crypto_isal", 2021-03-24T21:54:09.169 INFO:teuthology.orchestra.run.smithi125.stdout: "plugin_dir": "/usr/lib64/ceph", 2021-03-24T21:54:09.169 INFO:teuthology.orchestra.run.smithi125.stdout: "public_addr": "-", 2021-03-24T21:54:09.169 INFO:teuthology.orchestra.run.smithi125.stdout: "public_addrv": "", 2021-03-24T21:54:09.170 INFO:teuthology.orchestra.run.smithi125.stdout: "public_bind_addr": "-", 2021-03-24T21:54:09.170 INFO:teuthology.orchestra.run.smithi125.stdout: "public_network": "", 2021-03-24T21:54:09.170 INFO:teuthology.orchestra.run.smithi125.stdout: "public_network_interface": "", 2021-03-24T21:54:09.170 INFO:teuthology.orchestra.run.smithi125.stdout: "qat_compressor_enabled": "false", 2021-03-24T21:54:09.171 INFO:teuthology.orchestra.run.smithi125.stdout: "rados_mon_op_timeout": "0", 2021-03-24T21:54:09.171 INFO:teuthology.orchestra.run.smithi125.stdout: "rados_osd_op_timeout": "0", 2021-03-24T21:54:09.171 INFO:teuthology.orchestra.run.smithi125.stdout: "rados_tracing": "false", 2021-03-24T21:54:09.171 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_atime_update_interval": "60", 2021-03-24T21:54:09.172 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2021-03-24T21:54:09.172 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_balance_parent_reads": "false", 2021-03-24T21:54:09.172 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_balance_snap_reads": "false", 2021-03-24T21:54:09.173 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_blkin_trace_all": "false", 2021-03-24T21:54:09.173 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_blocklist_expire_seconds": "0", 2021-03-24T21:54:09.173 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_blocklist_on_break_lock": "true", 2021-03-24T21:54:09.174 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_cache": "true", 2021-03-24T21:54:09.174 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_cache_block_writes_upfront": "false", 2021-03-24T21:54:09.174 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_cache_max_dirty": "25165824", 2021-03-24T21:54:09.174 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_cache_max_dirty_age": "1.000000", 2021-03-24T21:54:09.175 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_cache_max_dirty_object": "0", 2021-03-24T21:54:09.175 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_cache_policy": "writearound", 2021-03-24T21:54:09.175 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_cache_size": "33554432", 2021-03-24T21:54:09.175 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_cache_target_dirty": "16777216", 2021-03-24T21:54:09.175 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_cache_writethrough_until_flush": "true", 2021-03-24T21:54:09.176 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_clone_copy_on_read": "false", 2021-03-24T21:54:09.176 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_compression_hint": "none", 2021-03-24T21:54:09.176 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_concurrent_management_ops": "10", 2021-03-24T21:54:09.176 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_config_pool_override_update_timestamp": "0", 2021-03-24T21:54:09.177 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_default_clone_format": "auto", 2021-03-24T21:54:09.177 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_default_data_pool": "", 2021-03-24T21:54:09.177 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_default_features": "61", 2021-03-24T21:54:09.177 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_default_format": "2", 2021-03-24T21:54:09.178 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_default_map_options": "", 2021-03-24T21:54:09.178 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_default_order": "22", 2021-03-24T21:54:09.178 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_default_pool": "rbd", 2021-03-24T21:54:09.178 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2021-03-24T21:54:09.179 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_default_stripe_count": "0", 2021-03-24T21:54:09.179 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_default_stripe_unit": "0", 2021-03-24T21:54:09.179 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_disable_zero_copy_writes": "true", 2021-03-24T21:54:09.179 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_discard_granularity_bytes": "65536", 2021-03-24T21:54:09.180 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_discard_on_zeroed_write_same": "true", 2021-03-24T21:54:09.180 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_enable_alloc_hint": "true", 2021-03-24T21:54:09.180 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2021-03-24T21:54:09.180 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_io_scheduler": "simple", 2021-03-24T21:54:09.181 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2021-03-24T21:54:09.181 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_journal_commit_age": "5.000000", 2021-03-24T21:54:09.181 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2021-03-24T21:54:09.181 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_journal_max_payload_bytes": "16384", 2021-03-24T21:54:09.182 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_journal_object_flush_age": "0.000000", 2021-03-24T21:54:09.182 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_journal_object_flush_bytes": "1048576", 2021-03-24T21:54:09.182 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_journal_object_flush_interval": "0", 2021-03-24T21:54:09.183 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2021-03-24T21:54:09.183 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2021-03-24T21:54:09.183 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_journal_order": "24", 2021-03-24T21:54:09.183 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_journal_pool": "", 2021-03-24T21:54:09.184 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_journal_splay_width": "4", 2021-03-24T21:54:09.184 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_localize_parent_reads": "false", 2021-03-24T21:54:09.184 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_localize_snap_reads": "false", 2021-03-24T21:54:09.184 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2021-03-24T21:54:09.185 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2021-03-24T21:54:09.185 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2021-03-24T21:54:09.185 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2021-03-24T21:54:09.185 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2021-03-24T21:54:09.186 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2021-03-24T21:54:09.186 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_mirror_image_policy_type": "simple", 2021-03-24T21:54:09.186 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2021-03-24T21:54:09.186 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_mirror_image_state_check_interval": "30", 2021-03-24T21:54:09.187 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2021-03-24T21:54:09.187 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2021-03-24T21:54:09.187 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2021-03-24T21:54:09.187 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2021-03-24T21:54:09.188 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2021-03-24T21:54:09.188 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_mirror_memory_autotune": "true", 2021-03-24T21:54:09.188 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_mirror_memory_base": "805306368", 2021-03-24T21:54:09.188 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2021-03-24T21:54:09.189 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_mirror_memory_cache_min": "134217728", 2021-03-24T21:54:09.189 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2021-03-24T21:54:09.189 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2021-03-24T21:54:09.189 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_mirror_memory_target": "4294967296", 2021-03-24T21:54:09.190 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_mirror_perf_stats_prio": "5", 2021-03-24T21:54:09.190 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2021-03-24T21:54:09.190 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2021-03-24T21:54:09.190 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_mirroring_delete_delay": "0", 2021-03-24T21:54:09.191 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_mirroring_max_mirroring_snapshots": "3", 2021-03-24T21:54:09.191 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_mirroring_replay_delay": "0", 2021-03-24T21:54:09.191 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2021-03-24T21:54:09.191 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2021-03-24T21:54:09.192 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_move_to_trash_on_remove": "false", 2021-03-24T21:54:09.192 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2021-03-24T21:54:09.192 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_mtime_update_interval": "60", 2021-03-24T21:54:09.192 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_non_blocking_aio": "true", 2021-03-24T21:54:09.193 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_op_thread_timeout": "60", 2021-03-24T21:54:09.193 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_op_threads": "1", 2021-03-24T21:54:09.193 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_parent_cache_enabled": "false", 2021-03-24T21:54:09.193 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_persistent_cache_log_periodic_stats": "false", 2021-03-24T21:54:09.194 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_persistent_cache_mode": "disabled", 2021-03-24T21:54:09.194 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_persistent_cache_path": "/tmp", 2021-03-24T21:54:09.194 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_persistent_cache_size": "1073741824", 2021-03-24T21:54:09.194 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_plugins": "", 2021-03-24T21:54:09.195 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_qos_bps_burst": "0", 2021-03-24T21:54:09.195 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_qos_bps_burst_seconds": "1", 2021-03-24T21:54:09.195 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_qos_bps_limit": "0", 2021-03-24T21:54:09.195 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_qos_iops_burst": "0", 2021-03-24T21:54:09.196 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_qos_iops_burst_seconds": "1", 2021-03-24T21:54:09.196 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_qos_iops_limit": "0", 2021-03-24T21:54:09.196 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_qos_read_bps_burst": "0", 2021-03-24T21:54:09.196 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2021-03-24T21:54:09.196 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_qos_read_bps_limit": "0", 2021-03-24T21:54:09.197 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_qos_read_iops_burst": "0", 2021-03-24T21:54:09.197 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2021-03-24T21:54:09.197 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_qos_read_iops_limit": "0", 2021-03-24T21:54:09.197 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_qos_schedule_tick_min": "50", 2021-03-24T21:54:09.198 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_qos_write_bps_burst": "0", 2021-03-24T21:54:09.198 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2021-03-24T21:54:09.198 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_qos_write_bps_limit": "0", 2021-03-24T21:54:09.198 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_qos_write_iops_burst": "0", 2021-03-24T21:54:09.199 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2021-03-24T21:54:09.199 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_qos_write_iops_limit": "0", 2021-03-24T21:54:09.199 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_quiesce_notification_attempts": "10", 2021-03-24T21:54:09.199 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_read_from_replica_policy": "default", 2021-03-24T21:54:09.200 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2021-03-24T21:54:09.200 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_readahead_max_bytes": "524288", 2021-03-24T21:54:09.200 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_readahead_trigger_requests": "10", 2021-03-24T21:54:09.200 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_request_timed_out_seconds": "30", 2021-03-24T21:54:09.201 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_skip_partial_discard": "true", 2021-03-24T21:54:09.201 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2021-03-24T21:54:09.201 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_tracing": "false", 2021-03-24T21:54:09.201 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_validate_names": "true", 2021-03-24T21:54:09.202 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd_validate_pool": "true", 2021-03-24T21:54:09.202 INFO:teuthology.orchestra.run.smithi125.stdout: "restapi_base_url": "", 2021-03-24T21:54:09.202 INFO:teuthology.orchestra.run.smithi125.stdout: "restapi_log_level": "", 2021-03-24T21:54:09.202 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_acl_grants_max_num": "100", 2021-03-24T21:54:09.203 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_admin_entry": "admin", 2021-03-24T21:54:09.203 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_barbican_url": "", 2021-03-24T21:54:09.203 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2021-03-24T21:54:09.203 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_bucket_default_quota_max_size": "-1", 2021-03-24T21:54:09.204 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_bucket_index_max_aio": "128", 2021-03-24T21:54:09.204 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_bucket_quota_cache_size": "10000", 2021-03-24T21:54:09.204 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_bucket_quota_soft_threshold": "0.950000", 2021-03-24T21:54:09.204 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_bucket_quota_ttl": "600", 2021-03-24T21:54:09.204 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_cache_enabled": "true", 2021-03-24T21:54:09.205 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_cache_expiry_interval": "900", 2021-03-24T21:54:09.205 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_cache_lru_size": "10000", 2021-03-24T21:54:09.205 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_content_length_compat": "false", 2021-03-24T21:54:09.205 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_copy_obj_progress": "true", 2021-03-24T21:54:09.206 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2021-03-24T21:54:09.206 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_cors_rules_max_num": "100", 2021-03-24T21:54:09.206 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_cross_domain_policy": "", 2021-03-24T21:54:09.206 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_crypt_default_encryption_key": "", 2021-03-24T21:54:09.207 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_crypt_kmip_addr": "", 2021-03-24T21:54:09.207 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_crypt_kmip_ca_path": "", 2021-03-24T21:54:09.207 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_crypt_kmip_client_cert": "", 2021-03-24T21:54:09.207 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_crypt_kmip_client_key": "", 2021-03-24T21:54:09.208 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_crypt_kmip_kms_key_template": "", 2021-03-24T21:54:09.208 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_crypt_kmip_password": "", 2021-03-24T21:54:09.208 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2021-03-24T21:54:09.209 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_crypt_kmip_username": "", 2021-03-24T21:54:09.209 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_crypt_require_ssl": "true", 2021-03-24T21:54:09.209 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2021-03-24T21:54:09.209 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2021-03-24T21:54:09.210 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_crypt_suppress_logs": "true", 2021-03-24T21:54:09.210 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_crypt_vault_addr": "", 2021-03-24T21:54:09.210 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_crypt_vault_auth": "token", 2021-03-24T21:54:09.210 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_crypt_vault_namespace": "", 2021-03-24T21:54:09.211 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_crypt_vault_prefix": "", 2021-03-24T21:54:09.211 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_crypt_vault_secret_engine": "transit", 2021-03-24T21:54:09.211 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_crypt_vault_token_file": "", 2021-03-24T21:54:09.211 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_curl_buffersize": "524288", 2021-03-24T21:54:09.212 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_curl_low_speed_limit": "1024", 2021-03-24T21:54:09.212 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_curl_low_speed_time": "300", 2021-03-24T21:54:09.212 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_curl_wait_timeout_ms": "1000", 2021-03-24T21:54:09.212 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2021-03-24T21:54:09.213 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_data_log_backing": "auto", 2021-03-24T21:54:09.213 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_data_log_changes_size": "1000", 2021-03-24T21:54:09.213 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_data_log_num_shards": "128", 2021-03-24T21:54:09.213 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_data_log_obj_prefix": "data_log", 2021-03-24T21:54:09.214 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_data_log_window": "30", 2021-03-24T21:54:09.214 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_data_notify_interval_msec": "200", 2021-03-24T21:54:09.214 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_default_realm_info_oid": "default.realm", 2021-03-24T21:54:09.214 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_default_region_info_oid": "default.region", 2021-03-24T21:54:09.215 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_default_zone_info_oid": "default.zone", 2021-03-24T21:54:09.215 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2021-03-24T21:54:09.215 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_defer_to_bucket_acls": "", 2021-03-24T21:54:09.215 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_delete_multi_obj_max_num": "1000", 2021-03-24T21:54:09.216 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_dmclock_admin_lim": "0.000000", 2021-03-24T21:54:09.216 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_dmclock_admin_res": "100.000000", 2021-03-24T21:54:09.216 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2021-03-24T21:54:09.216 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_dmclock_auth_lim": "0.000000", 2021-03-24T21:54:09.217 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_dmclock_auth_res": "200.000000", 2021-03-24T21:54:09.217 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2021-03-24T21:54:09.217 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_dmclock_data_lim": "0.000000", 2021-03-24T21:54:09.218 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_dmclock_data_res": "500.000000", 2021-03-24T21:54:09.218 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_dmclock_data_wgt": "500.000000", 2021-03-24T21:54:09.218 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2021-03-24T21:54:09.218 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_dmclock_metadata_res": "500.000000", 2021-03-24T21:54:09.219 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2021-03-24T21:54:09.219 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_dns_name": "", 2021-03-24T21:54:09.219 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_dns_s3website_name": "", 2021-03-24T21:54:09.219 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_dynamic_resharding": "true", 2021-03-24T21:54:09.219 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2021-03-24T21:54:09.220 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_enable_gc_threads": "true", 2021-03-24T21:54:09.220 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_enable_lc_threads": "true", 2021-03-24T21:54:09.220 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_enable_ops_log": "false", 2021-03-24T21:54:09.220 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_enable_quota_threads": "true", 2021-03-24T21:54:09.221 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_enable_static_website": "false", 2021-03-24T21:54:09.221 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_enable_usage_log": "false", 2021-03-24T21:54:09.221 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_enforce_swift_acls": "true", 2021-03-24T21:54:09.221 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_exit_timeout_secs": "120", 2021-03-24T21:54:09.222 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_expose_bucket": "false", 2021-03-24T21:54:09.222 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_extended_http_attrs": "", 2021-03-24T21:54:09.222 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_fcgi_socket_backlog": "1024", 2021-03-24T21:54:09.222 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2021-03-24T21:54:09.222 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_frontends": "beast port=7480", 2021-03-24T21:54:09.223 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_gc_max_concurrent_io": "10", 2021-03-24T21:54:09.223 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_gc_max_deferred": "50", 2021-03-24T21:54:09.223 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2021-03-24T21:54:09.223 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_gc_max_objs": "32", 2021-03-24T21:54:09.224 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_gc_max_queue_size": "134213632", 2021-03-24T21:54:09.224 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_gc_max_trim_chunk": "16", 2021-03-24T21:54:09.224 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_gc_obj_min_wait": "7200", 2021-03-24T21:54:09.224 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_gc_processor_max_time": "3600", 2021-03-24T21:54:09.225 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_gc_processor_period": "3600", 2021-03-24T21:54:09.225 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_get_obj_max_req_size": "4194304", 2021-03-24T21:54:09.225 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_get_obj_window_size": "16777216", 2021-03-24T21:54:09.225 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_healthcheck_disabling_path": "", 2021-03-24T21:54:09.226 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_host": "", 2021-03-24T21:54:09.226 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_ignore_get_invalid_range": "false", 2021-03-24T21:54:09.226 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_init_timeout": "300", 2021-03-24T21:54:09.226 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2021-03-24T21:54:09.227 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_keystone_accepted_admin_roles": "", 2021-03-24T21:54:09.227 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2021-03-24T21:54:09.227 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_keystone_admin_domain": "", 2021-03-24T21:54:09.227 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_keystone_admin_password": "", 2021-03-24T21:54:09.228 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_keystone_admin_password_path": "", 2021-03-24T21:54:09.228 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_keystone_admin_project": "", 2021-03-24T21:54:09.228 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_keystone_admin_tenant": "", 2021-03-24T21:54:09.228 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_keystone_admin_token": "", 2021-03-24T21:54:09.228 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_keystone_admin_token_path": "", 2021-03-24T21:54:09.229 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_keystone_admin_user": "", 2021-03-24T21:54:09.229 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_keystone_api_version": "2", 2021-03-24T21:54:09.229 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_keystone_barbican_domain": "", 2021-03-24T21:54:09.229 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_keystone_barbican_password": "", 2021-03-24T21:54:09.230 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_keystone_barbican_project": "", 2021-03-24T21:54:09.230 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_keystone_barbican_tenant": "", 2021-03-24T21:54:09.230 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_keystone_barbican_user": "", 2021-03-24T21:54:09.230 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_keystone_implicit_tenants": "false", 2021-03-24T21:54:09.231 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_keystone_token_cache_size": "10000", 2021-03-24T21:54:09.231 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_keystone_url": "", 2021-03-24T21:54:09.231 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_keystone_verify_ssl": "true", 2021-03-24T21:54:09.231 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_lc_debug_interval": "-1", 2021-03-24T21:54:09.232 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_lc_lock_max_time": "90", 2021-03-24T21:54:09.232 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_lc_max_objs": "32", 2021-03-24T21:54:09.232 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_lc_max_rules": "1000", 2021-03-24T21:54:09.232 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_lc_max_worker": "3", 2021-03-24T21:54:09.233 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_lc_max_wp_worker": "3", 2021-03-24T21:54:09.233 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_lc_thread_delay": "0", 2021-03-24T21:54:09.233 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2021-03-24T21:54:09.233 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_ldap_dnattr": "uid", 2021-03-24T21:54:09.233 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2021-03-24T21:54:09.234 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_ldap_searchfilter": "", 2021-03-24T21:54:09.234 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2021-03-24T21:54:09.234 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_ldap_uri": "ldaps://", 2021-03-24T21:54:09.234 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2021-03-24T21:54:09.235 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_list_bucket_min_readahead": "1000", 2021-03-24T21:54:09.235 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_list_buckets_max_chunk": "1000", 2021-03-24T21:54:09.235 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_log_http_headers": "", 2021-03-24T21:54:09.235 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_log_nonexistent_bucket": "false", 2021-03-24T21:54:09.236 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2021-03-24T21:54:09.236 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_log_object_name_utc": "false", 2021-03-24T21:54:09.236 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_luarocks_location": "/tmp/luarocks", 2021-03-24T21:54:09.236 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_max_attr_name_len": "0", 2021-03-24T21:54:09.237 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_max_attr_size": "0", 2021-03-24T21:54:09.237 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_max_attrs_num_in_req": "0", 2021-03-24T21:54:09.237 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_max_chunk_size": "4194304", 2021-03-24T21:54:09.237 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_max_concurrent_requests": "1024", 2021-03-24T21:54:09.238 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_max_dynamic_shards": "1999", 2021-03-24T21:54:09.238 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_max_listing_results": "1000", 2021-03-24T21:54:09.238 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_max_notify_retries": "3", 2021-03-24T21:54:09.238 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_max_objs_per_shard": "100000", 2021-03-24T21:54:09.239 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_max_put_param_size": "1048576", 2021-03-24T21:54:09.239 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_max_put_size": "5368709120", 2021-03-24T21:54:09.239 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_max_slo_entries": "1000", 2021-03-24T21:54:09.239 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_md_log_max_shards": "64", 2021-03-24T21:54:09.239 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_md_notify_interval_msec": "200", 2021-03-24T21:54:09.240 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_mime_types_file": "/etc/mime.types", 2021-03-24T21:54:09.240 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_mp_lock_max_time": "600", 2021-03-24T21:54:09.240 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_multipart_min_part_size": "5242880", 2021-03-24T21:54:09.240 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_multipart_part_upload_limit": "10000", 2021-03-24T21:54:09.241 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_nfs_fhcache_partitions": "3", 2021-03-24T21:54:09.241 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_nfs_fhcache_size": "2017", 2021-03-24T21:54:09.241 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2021-03-24T21:54:09.241 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_nfs_lru_lanes": "5", 2021-03-24T21:54:09.242 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_nfs_max_gc": "300", 2021-03-24T21:54:09.242 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_nfs_namespace_expire_secs": "300", 2021-03-24T21:54:09.242 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_nfs_run_gc_threads": "false", 2021-03-24T21:54:09.242 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_nfs_run_lc_threads": "false", 2021-03-24T21:54:09.242 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_nfs_run_quota_threads": "false", 2021-03-24T21:54:09.243 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_nfs_run_sync_thread": "false", 2021-03-24T21:54:09.243 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_nfs_s3_fast_attrs": "false", 2021-03-24T21:54:09.243 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_nfs_write_completion_interval_s": "10", 2021-03-24T21:54:09.244 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_num_async_rados_threads": "32", 2021-03-24T21:54:09.244 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_num_control_oids": "8", 2021-03-24T21:54:09.244 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_num_rados_handles": "1", 2021-03-24T21:54:09.244 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_numa_node": "-1", 2021-03-24T21:54:09.244 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_obj_stripe_size": "4194304", 2021-03-24T21:54:09.245 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_obj_tombstone_cache_size": "1000", 2021-03-24T21:54:09.245 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_objexp_chunk_size": "100", 2021-03-24T21:54:09.245 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_objexp_gc_interval": "600", 2021-03-24T21:54:09.245 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_objexp_hints_num_shards": "127", 2021-03-24T21:54:09.246 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_olh_pending_timeout_sec": "3600", 2021-03-24T21:54:09.246 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_op_thread_suicide_timeout": "0", 2021-03-24T21:54:09.246 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_op_thread_timeout": "600", 2021-03-24T21:54:09.246 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_op_tracing": "false", 2021-03-24T21:54:09.247 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_opa_token": "", 2021-03-24T21:54:09.247 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_opa_url": "", 2021-03-24T21:54:09.247 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_opa_verify_ssl": "true", 2021-03-24T21:54:09.247 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_ops_log_data_backlog": "5242880", 2021-03-24T21:54:09.247 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_ops_log_rados": "true", 2021-03-24T21:54:09.248 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_ops_log_socket_path": "", 2021-03-24T21:54:09.248 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_override_bucket_index_max_shards": "0", 2021-03-24T21:54:09.248 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2021-03-24T21:54:09.248 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_period_push_interval": "2.000000", 2021-03-24T21:54:09.249 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_period_push_interval_max": "30.000000", 2021-03-24T21:54:09.249 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_period_root_pool": ".rgw.root", 2021-03-24T21:54:09.249 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_port": "", 2021-03-24T21:54:09.249 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_print_continue": "true", 2021-03-24T21:54:09.249 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_print_prohibited_content_length": "false", 2021-03-24T21:54:09.250 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_put_obj_max_window_size": "67108864", 2021-03-24T21:54:09.250 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_put_obj_min_window_size": "16777216", 2021-03-24T21:54:09.250 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2021-03-24T21:54:09.250 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_rados_pool_pg_num_min": "8", 2021-03-24T21:54:09.251 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_rados_pool_recovery_priority": "5", 2021-03-24T21:54:09.251 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_rados_tracing": "false", 2021-03-24T21:54:09.251 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_realm": "", 2021-03-24T21:54:09.252 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_realm_root_pool": ".rgw.root", 2021-03-24T21:54:09.252 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_region": "", 2021-03-24T21:54:09.252 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_region_root_pool": ".rgw.root", 2021-03-24T21:54:09.252 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_relaxed_region_enforcement": "false", 2021-03-24T21:54:09.253 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_relaxed_s3_bucket_names": "false", 2021-03-24T21:54:09.253 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2021-03-24T21:54:09.253 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_request_uri": "", 2021-03-24T21:54:09.253 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_reshard_batch_size": "64", 2021-03-24T21:54:09.254 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_reshard_bucket_lock_duration": "360", 2021-03-24T21:54:09.254 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_reshard_max_aio": "128", 2021-03-24T21:54:09.254 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_reshard_num_logs": "16", 2021-03-24T21:54:09.254 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_reshard_thread_interval": "600", 2021-03-24T21:54:09.255 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_resolve_cname": "false", 2021-03-24T21:54:09.255 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_rest_getusage_op_compat": "false", 2021-03-24T21:54:09.255 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_run_sync_thread": "true", 2021-03-24T21:54:09.255 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_s3_auth_order": "sts, external, local", 2021-03-24T21:54:09.256 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_s3_auth_use_keystone": "false", 2021-03-24T21:54:09.256 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_s3_auth_use_ldap": "false", 2021-03-24T21:54:09.256 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_s3_auth_use_rados": "true", 2021-03-24T21:54:09.256 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_s3_auth_use_sts": "false", 2021-03-24T21:54:09.257 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_s3_success_create_obj_status": "0", 2021-03-24T21:54:09.257 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_safe_max_objects_per_shard": "102400", 2021-03-24T21:54:09.257 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_scheduler_type": "throttler", 2021-03-24T21:54:09.257 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_script_uri": "", 2021-03-24T21:54:09.257 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_service_provider_name": "", 2021-03-24T21:54:09.258 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_shard_warning_threshold": "90.000000", 2021-03-24T21:54:09.258 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_socket_path": "", 2021-03-24T21:54:09.258 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_sts_client_id": "", 2021-03-24T21:54:09.258 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_sts_client_secret": "", 2021-03-24T21:54:09.259 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_sts_entry": "sts", 2021-03-24T21:54:09.259 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_sts_key": "sts", 2021-03-24T21:54:09.259 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_sts_max_session_duration": "43200", 2021-03-24T21:54:09.259 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_sts_min_session_duration": "900", 2021-03-24T21:54:09.260 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_sts_token_introspection_url": "", 2021-03-24T21:54:09.260 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_swift_account_in_url": "false", 2021-03-24T21:54:09.260 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_swift_auth_entry": "auth", 2021-03-24T21:54:09.261 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_swift_auth_url": "", 2021-03-24T21:54:09.261 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_swift_custom_header": "", 2021-03-24T21:54:09.261 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_swift_enforce_content_length": "false", 2021-03-24T21:54:09.261 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_swift_need_stats": "true", 2021-03-24T21:54:09.262 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_swift_tenant_name": "", 2021-03-24T21:54:09.262 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_swift_token_expiration": "86400", 2021-03-24T21:54:09.262 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_swift_url": "", 2021-03-24T21:54:09.262 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_swift_url_prefix": "swift", 2021-03-24T21:54:09.263 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_swift_versioning_enabled": "false", 2021-03-24T21:54:09.263 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2021-03-24T21:54:09.263 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_sync_lease_period": "120", 2021-03-24T21:54:09.263 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2021-03-24T21:54:09.263 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_sync_log_trim_interval": "1200", 2021-03-24T21:54:09.264 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_sync_log_trim_max_buckets": "16", 2021-03-24T21:54:09.264 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2021-03-24T21:54:09.264 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2021-03-24T21:54:09.264 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_sync_obj_etag_verify": "false", 2021-03-24T21:54:09.265 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_sync_trace_history_size": "4096", 2021-03-24T21:54:09.265 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_sync_trace_per_node_log_size": "32", 2021-03-24T21:54:09.265 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2021-03-24T21:54:09.265 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_thread_pool_size": "512", 2021-03-24T21:54:09.266 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_torrent_comment": "", 2021-03-24T21:54:09.266 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_torrent_createby": "", 2021-03-24T21:54:09.266 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_torrent_encoding": "", 2021-03-24T21:54:09.266 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_torrent_flag": "false", 2021-03-24T21:54:09.266 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_torrent_origin": "", 2021-03-24T21:54:09.267 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_torrent_sha_unit": "524288", 2021-03-24T21:54:09.267 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_torrent_tracker": "", 2021-03-24T21:54:09.267 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_trust_forwarded_https": "false", 2021-03-24T21:54:09.267 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_usage_log_flush_threshold": "1024", 2021-03-24T21:54:09.268 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_usage_log_tick_interval": "30", 2021-03-24T21:54:09.268 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_usage_max_shards": "32", 2021-03-24T21:54:09.268 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_usage_max_user_shards": "1", 2021-03-24T21:54:09.268 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_use_opa_authz": "false", 2021-03-24T21:54:09.269 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_user_default_quota_max_objects": "-1", 2021-03-24T21:54:09.269 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_user_default_quota_max_size": "-1", 2021-03-24T21:54:09.269 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_user_max_buckets": "1000", 2021-03-24T21:54:09.270 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2021-03-24T21:54:09.270 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_user_quota_sync_idle_users": "false", 2021-03-24T21:54:09.270 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_user_quota_sync_interval": "86400", 2021-03-24T21:54:09.270 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_user_quota_sync_wait_time": "86400", 2021-03-24T21:54:09.271 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_user_unique_email": "true", 2021-03-24T21:54:09.271 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_verify_ssl": "true", 2021-03-24T21:54:09.271 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_website_routing_rules_max_num": "50", 2021-03-24T21:54:09.271 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_zone": "", 2021-03-24T21:54:09.271 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_zone_root_pool": ".rgw.root", 2021-03-24T21:54:09.272 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_zonegroup": "", 2021-03-24T21:54:09.272 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2021-03-24T21:54:09.272 INFO:teuthology.orchestra.run.smithi125.stdout: "rocksdb_block_size": "4096", 2021-03-24T21:54:09.272 INFO:teuthology.orchestra.run.smithi125.stdout: "rocksdb_bloom_bits_per_key": "20", 2021-03-24T21:54:09.273 INFO:teuthology.orchestra.run.smithi125.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2021-03-24T21:54:09.273 INFO:teuthology.orchestra.run.smithi125.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2021-03-24T21:54:09.273 INFO:teuthology.orchestra.run.smithi125.stdout: "rocksdb_cache_row_ratio": "0.000000", 2021-03-24T21:54:09.273 INFO:teuthology.orchestra.run.smithi125.stdout: "rocksdb_cache_shard_bits": "4", 2021-03-24T21:54:09.274 INFO:teuthology.orchestra.run.smithi125.stdout: "rocksdb_cache_size": "536870912", 2021-03-24T21:54:09.274 INFO:teuthology.orchestra.run.smithi125.stdout: "rocksdb_cache_type": "binned_lru", 2021-03-24T21:54:09.274 INFO:teuthology.orchestra.run.smithi125.stdout: "rocksdb_collect_compaction_stats": "false", 2021-03-24T21:54:09.274 INFO:teuthology.orchestra.run.smithi125.stdout: "rocksdb_collect_extended_stats": "false", 2021-03-24T21:54:09.275 INFO:teuthology.orchestra.run.smithi125.stdout: "rocksdb_collect_memory_stats": "false", 2021-03-24T21:54:09.275 INFO:teuthology.orchestra.run.smithi125.stdout: "rocksdb_delete_range_threshold": "1048576", 2021-03-24T21:54:09.275 INFO:teuthology.orchestra.run.smithi125.stdout: "rocksdb_index_type": "binary_search", 2021-03-24T21:54:09.275 INFO:teuthology.orchestra.run.smithi125.stdout: "rocksdb_log_to_ceph_log": "true", 2021-03-24T21:54:09.275 INFO:teuthology.orchestra.run.smithi125.stdout: "rocksdb_metadata_block_size": "4096", 2021-03-24T21:54:09.276 INFO:teuthology.orchestra.run.smithi125.stdout: "rocksdb_partition_filters": "false", 2021-03-24T21:54:09.276 INFO:teuthology.orchestra.run.smithi125.stdout: "rocksdb_perf": "false", 2021-03-24T21:54:09.276 INFO:teuthology.orchestra.run.smithi125.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2021-03-24T21:54:09.276 INFO:teuthology.orchestra.run.smithi125.stdout: "rotating_keys_bootstrap_timeout": "30", 2021-03-24T21:54:09.277 INFO:teuthology.orchestra.run.smithi125.stdout: "rotating_keys_renewal_timeout": "10", 2021-03-24T21:54:09.277 INFO:teuthology.orchestra.run.smithi125.stdout: "run_dir": "/var/run/ceph", 2021-03-24T21:54:09.277 INFO:teuthology.orchestra.run.smithi125.stdout: "setgroup": "ceph", 2021-03-24T21:54:09.277 INFO:teuthology.orchestra.run.smithi125.stdout: "setuser": "ceph", 2021-03-24T21:54:09.278 INFO:teuthology.orchestra.run.smithi125.stdout: "setuser_match_path": "", 2021-03-24T21:54:09.278 INFO:teuthology.orchestra.run.smithi125.stdout: "target_max_misplaced_ratio": "0.050000", 2021-03-24T21:54:09.278 INFO:teuthology.orchestra.run.smithi125.stdout: "thp": "false", 2021-03-24T21:54:09.278 INFO:teuthology.orchestra.run.smithi125.stdout: "threadpool_default_timeout": "60", 2021-03-24T21:54:09.279 INFO:teuthology.orchestra.run.smithi125.stdout: "threadpool_empty_queue_max_wait": "2", 2021-03-24T21:54:09.279 INFO:teuthology.orchestra.run.smithi125.stdout: "throttler_perf_counter": "true" 2021-03-24T21:54:09.279 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-24T21:54:09.280 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:08 smithi142 conmon[20125]: cluster 2021-03-24T21:54:07.669859+0000 osd.7 2021-03-24T21:54:09.280 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:08 smithi142 conmon[20125]: (osd.7) 28 : cluster [DBG] 2.0 scrub ok 2021-03-24T21:54:09.281 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:08 smithi125 conmon[19936]: cluster 2021-03-24T21:54:07.669859+0000 osd.7 (osd.7) 28 : cluster 2021-03-24T21:54:09.281 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:08 smithi125 conmon[19936]: [DBG] 2.0 scrub ok 2021-03-24T21:54:09.281 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:08 smithi125 conmon[24927]: cluster 2021-03-24T21:54:07.669859+0000 osd.7 (osd.7) 28 : 2021-03-24T21:54:09.282 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:08 smithi125 conmon[24927]: cluster [DBG] 2.0 scrub ok 2021-03-24T21:54:09.726 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2021-03-24T21:54:10.041 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:54:10.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:09 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:54:10.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:09 smithi142 conmon[20125]: 21:54:08.697501+0000 mgr.y (mgr.14144) 297 : cluster [DBG] pgmap v296: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-03-24T21:54:10.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:09 smithi142 conmon[20125]: cluster 2021-03-24T21:54:10.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:09 smithi142 conmon[20125]: 2021 2021-03-24T21:54:10.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:09 smithi142 conmon[20125]: -03-24T21: 2021-03-24T21:54:10.119 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:09 smithi142 conmon[20125]: 54:09.072949 2021-03-24T21:54:10.119 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:09 smithi142 conmon[20125]: +0000 osd.0 ( 2021-03-24T21:54:10.119 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:09 smithi142 conmon[20125]: osd.0) 31 2021-03-24T21:54:10.119 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:09 smithi142 conmon[20125]: : cluster [DBG] 2021-03-24T21:54:10.120 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:09 smithi142 conmon[20125]: purged_snaps scrub starts 2021-03-24T21:54:10.120 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:09 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:54:10.120 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:09 smithi142 conmon[20125]: -03-24T21:54 2021-03-24T21:54:10.120 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:09 smithi142 conmon[20125]: :09.073131+0000 2021-03-24T21:54:10.120 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:09 smithi142 conmon[20125]: osd.0 (osd. 2021-03-24T21:54:10.121 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:09 smithi142 conmon[20125]: 0) 32 : cluster 2021-03-24T21:54:10.121 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:09 smithi142 conmon[20125]: [DBG] purged_snaps scrub ok 2021-03-24T21:54:10.121 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:09 smithi142 conmon[20125]: 2021-03-24T21:54:10.159 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:09 smithi125 conmon[24927]: cluster 2021-03-24T21:54:08.697501+0000 mgr.y (mgr.14144) 297 : cluster 2021-03-24T21:54:10.159 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:09 smithi125 conmon[24927]: [DBG] pgmap v296: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-03-24T21:54:10.159 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:09 smithi125 conmon[24927]: cluster 2021-03-24T21:54:09.072949+0000 osd.0 (osd.0) 31 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:54:10.160 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:09 smithi125 conmon[24927]: cluster 2021-03-24T21:54:10.160 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:09 smithi125 conmon[24927]: 2021-03-24T21:54:09.073131+0000 osd.0 (osd.0) 32 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:54:10.160 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:09 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:54:10.161 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:09 smithi125 conmon[19936]: 54:08.697501+0000 mgr.y (mgr.14144) 297 : cluster [DBG] pgmap v296: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2021-03-24T21:54:10.161 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:09 smithi125 conmon[19936]: cluster 2021-03-24T21:54:10.161 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:09 smithi125 conmon[19936]: 2021- 2021-03-24T21:54:10.161 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:09 smithi125 conmon[19936]: 03-24T21:54 2021-03-24T21:54:10.161 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:09 smithi125 conmon[19936]: :09.072949+0000 2021-03-24T21:54:10.162 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:09 smithi125 conmon[19936]: osd.0 (osd.0) 31 2021-03-24T21:54:10.162 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:09 smithi125 conmon[19936]: : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:54:10.162 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:09 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:54:10.162 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:09 smithi125 conmon[19936]: -03-24T21:54:09.073131+0000 2021-03-24T21:54:10.163 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:09 smithi125 conmon[19936]: osd.0 (osd.0) 32 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:54:10.909 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:10 smithi125 conmon[19936]: cluster 2021-03-24T21:54 2021-03-24T21:54:10.910 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:10 smithi125 conmon[19936]: :09.127700+0000 osd.5 (osd.5) 28 : cluster [DBG] 3.1c scrub ok 2021-03-24T21:54:10.910 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:10 smithi125 conmon[19936]: cluster 2021-03-24T21:54:09.727921+0000 2021-03-24T21:54:10.910 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:10 smithi125 conmon[19936]: mon.a (mon.0) 611 : cluster [DBG] osdmap e95: 8 total, 8 up, 8 in 2021-03-24T21:54:10.911 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:10 smithi125 conmon[24927]: cluster 2021-03-24T21:54:09.127700+0000 2021-03-24T21:54:10.911 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:10 smithi125 conmon[24927]: osd.5 (osd.5) 28 : cluster [DBG] 3.1c scrub ok 2021-03-24T21:54:10.911 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:10 smithi125 conmon[24927]: cluster 2021-03-24T21:54:10.911 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:10 smithi125 conmon[24927]: 2021-03-24T21:54:09.727921+0000 mon.a 2021-03-24T21:54:10.912 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:10 smithi125 conmon[24927]: (mon.0) 611 : cluster [DBG] 2021-03-24T21:54:10.912 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:10 smithi125 conmon[24927]: osdmap e95: 8 total, 8 up, 8 in 2021-03-24T21:54:11.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:10 smithi142 conmon[20125]: cluster 2021-03-24T21:54:09.127700+0000 osd.5 (osd.5) 28 : cluster 2021-03-24T21:54:11.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:10 smithi142 conmon[20125]: [DBG] 3.1c scrub ok 2021-03-24T21:54:11.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:10 smithi142 conmon[20125]: cluster 2021-03-24T21:54:09.727921+0000 mon.a (mon.0) 611 : cluster [DBG] 2021-03-24T21:54:11.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:10 smithi142 conmon[20125]: osdmap e95: 8 total, 8 up, 8 in 2021-03-24T21:54:11.892 INFO:teuthology.orchestra.run.smithi125.stdout:mon.a: {} 2021-03-24T21:54:11.907 INFO:teuthology.orchestra.run.smithi125.stdout:mon.b: {} 2021-03-24T21:54:11.923 INFO:teuthology.orchestra.run.smithi125.stdout:mon.c: {} 2021-03-24T21:54:11.925 INFO:teuthology.orchestra.run.smithi125.stderr:mon.a: mon_osd_down_out_interval = '0' (not observed, change may require restart) 2021-03-24T21:54:11.925 INFO:teuthology.orchestra.run.smithi125.stderr:mon.b: mon_osd_down_out_interval = '0' (not observed, change may require restart) 2021-03-24T21:54:11.925 INFO:teuthology.orchestra.run.smithi125.stderr:mon.c: mon_osd_down_out_interval = '0' (not observed, change may require restart) 2021-03-24T21:54:12.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:12 smithi142 conmon[20125]: cluster 2021-03-24T21:54:10 2021-03-24T21:54:12.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:12 smithi142 conmon[20125]: .421241+0000 osd.0 (osd.0) 33 : cluster [DBG] 4.3 scrub ok 2021-03-24T21:54:12.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:12 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:54:12.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:12 smithi142 conmon[20125]: 24T21:54:10.519407+0000 osd.3 (osd. 2021-03-24T21:54:12.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:12 smithi142 conmon[20125]: 3) 34 : cluster [DBG] 6.6 scrub ok 2021-03-24T21:54:12.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:12 smithi142 conmon[20125]: 2021-03-24T21:54:12.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:12 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:54:12.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:12 smithi142 conmon[20125]: -03-24T21:54:10 2021-03-24T21:54:12.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:12 smithi142 conmon[20125]: .698290+0000 mgr.y (mgr.14144 2021-03-24T21:54:12.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:12 smithi142 conmon[20125]: ) 298 : cluster [DBG] pgmap v298: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-24T21:54:12.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:12 smithi142 conmon[20125]: audit 2021 2021-03-24T21:54:12.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:12 smithi142 conmon[20125]: -03-24T21:54:11 2021-03-24T21:54:12.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:12 smithi142 conmon[20125]: .876153+0000 mon.a (mon. 2021-03-24T21:54:12.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:12 smithi142 conmon[20125]: 0) 612 : audit [DBG] from='client.? v1:172.21.15.125:0/3695198138' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-03-24T21:54:12.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:12 smithi142 conmon[20125]: 2021-03-24T21:54:12.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:12 smithi125 conmon[24927]: cluster 2021-03-24T21:54 2021-03-24T21:54:12.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:12 smithi125 conmon[24927]: :10.421241+0000 osd.0 (osd.0) 33 : cluster [DBG] 4.3 scrub ok 2021-03-24T21:54:12.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:12 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:54:12.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:12 smithi125 conmon[24927]: 21:54:10.519407+0000 osd.3 (osd.3) 2021-03-24T21:54:12.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:12 smithi125 conmon[24927]: 34 : cluster [DBG] 6.6 scrub ok 2021-03-24T21:54:12.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:12 smithi125 conmon[24927]: cluster 2021-03-24T21:54:12.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:12 smithi125 conmon[24927]: 2021-03-24T21:54:10.698290+0000 2021-03-24T21:54:12.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:12 smithi125 conmon[24927]: mgr.y (mgr.14144) 298 : cluster [DBG] pgmap v298: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-24T21:54:12.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:12 smithi125 conmon[24927]: 2021-03-24T21:54:12.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:12 smithi125 conmon[24927]: audit 2021-03-24T21:54:12.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:12 smithi125 conmon[24927]: 2021-03-24T21:54:11.876153+0000 mon.a (mon. 2021-03-24T21:54:12.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:12 smithi125 conmon[24927]: 0) 612 : audit [DBG] from='client.? v1:172.21.15.125:0/3695198138' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-03-24T21:54:12.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:12 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:54:12.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:12 smithi125 conmon[19936]: 21:54:10.421241+0000 osd.0 (osd.0) 33 : cluster [DBG] 4.3 scrub ok 2021-03-24T21:54:12.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:12 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:54:12.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:12 smithi125 conmon[19936]: 21:54:10.519407+0000 osd.3 (osd.3) 34 2021-03-24T21:54:12.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:12 smithi125 conmon[19936]: : cluster [DBG] 6.6 scrub ok 2021-03-24T21:54:12.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:12 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:54:12.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:12 smithi125 conmon[19936]: 03-24T21:54:10.698290+0000 2021-03-24T21:54:12.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:12 smithi125 conmon[19936]: mgr.y (mgr.14144) 298 : cluster 2021-03-24T21:54:12.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:12 smithi125 conmon[19936]: [DBG] pgmap v298: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-24T21:54:12.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:12 smithi125 conmon[19936]: audit 2021-03-24T21:54:12.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:12 smithi125 conmon[19936]: 2021-03-24T21:54 2021-03-24T21:54:12.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:12 smithi125 conmon[19936]: :11.876153+0000 mon.a (mon.0) 612 : audit 2021-03-24T21:54:12.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:12 smithi125 conmon[19936]: [DBG] from='client.? v1:172.21.15.125:0/3695198138' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-03-24T21:54:12.740 INFO:teuthology.run_tasks:Running task rados... 2021-03-24T21:54:12.757 INFO:tasks.rados:Beginning rados... 2021-03-24T21:54:12.758 DEBUG:teuthology.run_tasks:Unwinding manager rados 2021-03-24T21:54:12.817 INFO:tasks.rados:joining rados 2021-03-24T21:54:12.817 INFO:tasks.thrashosds.thrasher:starting do_thrash 2021-03-24T21:54:12.817 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-03-24T21:54:12.818 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-24T21:54:12.818 INFO:tasks.thrashosds.thrasher:Removing osd 6, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2021-03-24T21:54:12.819 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd out 6 2021-03-24T21:54:12.819 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2021-03-24T21:54:12.820 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:12.820 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2021-03-24T21:54:12.821 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-24T21:54:12.821 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2021-03-24T21:54:12.822 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.0 dump_ops_in_flight 2021-03-24T21:54:12.822 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2021-03-24T21:54:12.822 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd set noscrub 2021-03-24T21:54:12.823 INFO:tasks.rados:clients are ['client.0', 'client.1'] 2021-03-24T21:54:12.823 INFO:tasks.rados:starting run 0 out of 1 2021-03-24T21:54:12.823 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2021-03-24T21:54:12.824 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd pool create unique_pool_0 16 2021-03-24T21:54:13.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:13 smithi142 conmon[20125]: cluster 2021-03-24T21:54:11 2021-03-24T21:54:13.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:13 smithi142 conmon[20125]: .285981+0000 osd.4 (osd.4) 32 : cluster [DBG] 3.1a scrub ok 2021-03-24T21:54:13.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:13 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:54:13.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:13 smithi142 conmon[20125]: T21:54:12.170924+0000 osd.5 (osd.5 2021-03-24T21:54:13.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:13 smithi142 conmon[20125]: ) 29 : cluster [DBG] 5.f scrub ok 2021-03-24T21:54:13.129 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:54:13.149 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:54:13.193 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:54:13.216 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:54:13.241 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:54:13.357 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:13 smithi125 conmon[24927]: cluster 2021-03-24T21:54:11.285981+0000 osd.4 (osd.4) 32 : cluster [DBG] 3.1a scrub ok 2021-03-24T21:54:13.357 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:13 smithi125 conmon[24927]: cluster 2021-03-24T21:54:12.170924+0000 osd.5 (osd.5) 29 : cluster [DBG] 5.f scrub ok 2021-03-24T21:54:13.358 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:13 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:54:13.358 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:13 smithi125 conmon[19936]: -03-24T21:54:11.285981+0000 osd.4 (osd.4) 32 : cluster [DBG] 3.1a scrub ok 2021-03-24T21:54:13.358 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:13 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:54:13.358 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:13 smithi125 conmon[19936]: -24T21:54:12.170924+0000 osd.5 2021-03-24T21:54:13.359 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:13 smithi125 conmon[19936]: (osd.5) 29 : cluster [DBG] 5.f scrub ok 2021-03-24T21:54:13.592 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:13.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:13.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:54:13.699 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:14.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:14 smithi142 conmon[20125]: cluster 2021-03-24T21:54:12.699114+0000 mgr.y (mgr.14144) 299 : cluster 2021-03-24T21:54:14.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:14 smithi142 conmon[20125]: [DBG] pgmap v299: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-24T21:54:14.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:14 smithi125 conmon[19936]: cluster 2021-03-24T21:54:12.699114+0000 mgr.y (mgr.14144) 299 : cluster [DBG] 2021-03-24T21:54:14.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:14 smithi125 conmon[19936]: pgmap v299: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-24T21:54:14.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:14 smithi125 conmon[24927]: cluster 2021-03-24T21:54:12. 2021-03-24T21:54:14.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:14 smithi125 conmon[24927]: 699114+0000 mgr.y (mgr.14144) 299 : cluster [DBG] pgmap v299: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-24T21:54:15.295 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:15 smithi125 conmon[19936]: cluster 2021-03-24T21:54:13.291449+0000 2021-03-24T21:54:15.296 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:15 smithi125 conmon[19936]: osd.4 (osd.4) 33 : cluster [DBG] 6.17 scrub ok 2021-03-24T21:54:15.296 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:15 smithi125 conmon[19936]: cluster 2021-03-24T21:54:13.485035+0000 osd.3 (osd.3) 35 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:54:15.296 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:15 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:54:15.297 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:15 smithi125 conmon[19936]: 54:13.485196+0000 osd.3 (osd 2021-03-24T21:54:15.297 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:15 smithi125 conmon[19936]: .3) 36 : cluster [DBG] 2021-03-24T21:54:15.297 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:15 smithi125 conmon[19936]: purged_snaps scrub ok 2021-03-24T21:54:15.297 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:15 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:54:15.297 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:15 smithi125 conmon[19936]: -03-24T21:54:14 2021-03-24T21:54:15.298 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:15 smithi125 conmon[19936]: .096756+0000 mon.a (mon. 2021-03-24T21:54:15.298 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:15 smithi125 conmon[19936]: 0) 613 : cluster [DBG] osdmap e96: 8 total, 8 up, 8 in 2021-03-24T21:54:15.298 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:15 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:54:15.298 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:15 smithi125 conmon[19936]: 24T21:54:14.710691+0000 osd.1 2021-03-24T21:54:15.299 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:15 smithi125 conmon[19936]: (osd.1) 29 : cluster [DBG] 5.19 deep-scrub ok 2021-03-24T21:54:15.299 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:15 smithi125 conmon[19936]: 2021-03-24T21:54:15.299 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:15 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:54:15.299 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:15 smithi125 conmon[24927]: T21:54:13.291449+0000 osd.4 (osd.4) 33 : cluster [DBG] 2021-03-24T21:54:15.300 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:15 smithi125 conmon[24927]: 6.17 scrub ok 2021-03-24T21:54:15.300 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:15 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:54:15.300 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:15 smithi125 conmon[24927]: -24T21:54:13 2021-03-24T21:54:15.300 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:15 smithi125 conmon[24927]: .485035+0000 osd.3 (osd. 2021-03-24T21:54:15.301 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:15 smithi125 conmon[24927]: 3) 35 : cluster [DBG] 2021-03-24T21:54:15.301 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:15 smithi125 conmon[24927]: purged_snaps scrub starts 2021-03-24T21:54:15.301 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:15 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:54:15.301 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:15 smithi125 conmon[24927]: 24T21:54:13. 2021-03-24T21:54:15.301 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:15 smithi125 conmon[24927]: 485196+0000 osd.3 (osd.3 2021-03-24T21:54:15.302 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:15 smithi125 conmon[24927]: ) 36 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:54:15.302 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:15 smithi125 conmon[24927]: 2021-03-24T21:54:15.302 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:15 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:54:15.302 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:15 smithi125 conmon[24927]: 21:54:14.096756+0000 2021-03-24T21:54:15.303 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:15 smithi125 conmon[24927]: mon.a (mon.0) 613 2021-03-24T21:54:15.303 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:15 smithi125 conmon[24927]: : cluster [DBG] osdmap e96: 8 total, 8 up, 8 in 2021-03-24T21:54:15.303 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:15 smithi125 conmon[24927]: cluster 2021-03-24T21:54:15.304 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:15 smithi125 conmon[24927]: 2021-03-24T21: 2021-03-24T21:54:15.304 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:15 smithi125 conmon[24927]: 54:14.710691+0000 2021-03-24T21:54:15.304 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:15 smithi125 conmon[24927]: osd.1 (osd.1) 2021-03-24T21:54:15.304 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:15 smithi125 conmon[24927]: 29 : cluster [DBG] 5.19 deep-scrub ok 2021-03-24T21:54:15.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:15 smithi142 conmon[20125]: cluster 2021-03-24T21:54: 2021-03-24T21:54:15.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:15 smithi142 conmon[20125]: 13.291449+0000 osd.4 (osd.4) 33 : cluster [DBG] 6.17 scrub ok 2021-03-24T21:54:15.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:15 smithi142 conmon[20125]: cluster 2021-03-24T21:54:13.485035 2021-03-24T21:54:15.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:15 smithi142 conmon[20125]: +0000 osd.3 (osd.3) 35 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:54:15.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:15 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:54:15.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:15 smithi142 conmon[20125]: 21:54:13.485196+0000 osd.3 (osd.3 2021-03-24T21:54:15.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:15 smithi142 conmon[20125]: ) 36 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:54:15.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:15 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:54:15.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:15 smithi142 conmon[20125]: -03-24T21:54:14.096756+0000 2021-03-24T21:54:15.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:15 smithi142 conmon[20125]: mon.a (mon.0) 613 : cluster [DBG] osdmap e96: 8 total, 8 up, 8 in 2021-03-24T21:54:15.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:15 smithi142 conmon[20125]: 2021-03-24T21:54:15.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:15 smithi142 conmon[20125]: cluster 2021-03-24T21:54: 2021-03-24T21:54:15.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:15 smithi142 conmon[20125]: 14.710691+0000 osd.1 (osd.1) 29 : 2021-03-24T21:54:15.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:15 smithi142 conmon[20125]: cluster [DBG] 5.19 deep-scrub ok 2021-03-24T21:54:16.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:16 smithi142 conmon[20125]: cluster 2021-03-24T21:54:14 2021-03-24T21:54:16.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:16 smithi142 conmon[20125]: .574728+0000 osd.3 (osd.3) 37 : cluster [DBG] 4.10 scrub ok 2021-03-24T21:54:16.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:16 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:54:16.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:16 smithi142 conmon[20125]: 54:14.699921+0000 mgr.y (mgr.14144) 300 : cluster [DBG] pgmap v301: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-24T21:54:16.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:16 smithi125 conmon[19936]: cluster 2021-03-24T21:54:14. 2021-03-24T21:54:16.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:16 smithi125 conmon[19936]: 574728+0000 osd.3 (osd.3) 37 : cluster [DBG] 4.10 scrub ok 2021-03-24T21:54:16.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:16 smithi125 conmon[19936]: cluster 2021-03-24T21:54:14 2021-03-24T21:54:16.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:16 smithi125 conmon[19936]: .699921+0000 mgr.y (mgr.14144) 300 : cluster [DBG] pgmap v301: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-24T21:54:16.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:16 smithi125 conmon[24927]: cluster 2021-03-24T21:54 2021-03-24T21:54:16.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:16 smithi125 conmon[24927]: :14.574728+0000 osd.3 (osd.3) 37 : cluster [DBG] 4.10 scrub ok 2021-03-24T21:54:16.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:16 smithi125 conmon[24927]: cluster 2021-03-24T21:54:14.699921+0000 2021-03-24T21:54:16.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:16 smithi125 conmon[24927]: mgr.y (mgr.14144) 300 : cluster [DBG] pgmap v301: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-24T21:54:17.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:17 smithi142 conmon[20125]: cluster 2021-03-24T21:54:17.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:17 smithi142 conmon[20125]: 2021-03-24T21: 2021-03-24T21:54:17.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:17 smithi142 conmon[20125]: 54:15.969730+0000 osd.6 (osd.6) 15 : cluster [DBG] 5.1f scrub ok 2021-03-24T21:54:17.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:17 smithi142 conmon[20125]: cluster 2021-03-24T21:54:17.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:17 smithi142 conmon[20125]: 2021-03-24 2021-03-24T21:54:17.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:17 smithi142 conmon[20125]: T21:54:16.721494+0000 osd.1 2021-03-24T21:54:17.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:17 smithi142 conmon[20125]: (osd.1) 30 : cluster [DBG] 6.4 scrub ok 2021-03-24T21:54:17.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:17 smithi142 conmon[20125]: 2021-03-24T21:54:17.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:17 smithi125 conmon[19936]: cluster 2021-03-24T21:54 2021-03-24T21:54:17.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:17 smithi125 conmon[19936]: :15.969730+0000 osd.6 (osd.6) 15 : cluster [DBG] 5.1f scrub ok 2021-03-24T21:54:17.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:17 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:54:17.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:17 smithi125 conmon[19936]: 24T21:54:16.721494+0000 osd.1 (osd.1) 30 : cluster [DBG] 2021-03-24T21:54:17.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:17 smithi125 conmon[19936]: 6.4 scrub ok 2021-03-24T21:54:17.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:17 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:54:17.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:17 smithi125 conmon[24927]: 24T21:54:15.969730+0000 osd.6 (osd.6) 15 : cluster [DBG] 5.1f scrub ok 2021-03-24T21:54:17.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:17 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:54:17.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:17 smithi125 conmon[24927]: -03-24T21:54:16 2021-03-24T21:54:17.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:17 smithi125 conmon[24927]: .721494+0000 osd.1 (osd.1) 2021-03-24T21:54:17.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:17 smithi125 conmon[24927]: 30 : cluster [DBG] 6.4 scrub ok 2021-03-24T21:54:18.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:18 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:54:18.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:18 smithi125 conmon[19936]: 21:54:16.700778+0000 mgr.y (mgr.14144) 301 : cluster [DBG] pgmap v302: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-24T21:54:18.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:18 smithi125 conmon[19936]: cluster 2021-03-24T21:54:18.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:18 smithi125 conmon[19936]: 2021-03-24T21:54 2021-03-24T21:54:18.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:18 smithi125 conmon[19936]: :16.916128+0000 osd.6 (osd 2021-03-24T21:54:18.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:18 smithi125 conmon[19936]: .6) 16 : cluster [DBG] 2021-03-24T21:54:18.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:18 smithi125 conmon[19936]: purged_snaps scrub starts 2021-03-24T21:54:18.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:18 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:54:18.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:18 smithi125 conmon[19936]: -03-24T21:54: 2021-03-24T21:54:18.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:18 smithi125 conmon[19936]: 16.916297+0000 osd.6 (osd. 2021-03-24T21:54:18.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:18 smithi125 conmon[19936]: 6) 17 : cluster 2021-03-24T21:54:18.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:18 smithi125 conmon[19936]: [DBG] purged_snaps scrub ok 2021-03-24T21:54:18.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:18 smithi125 conmon[19936]: cluster 2021-03-24T21:54:18.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:18 smithi125 conmon[19936]: 2021-03-24T21:54 2021-03-24T21:54:18.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:18 smithi125 conmon[19936]: :17.122101+0000 mon.a ( 2021-03-24T21:54:18.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:18 smithi125 conmon[19936]: mon.0) 614 : cluster [DBG] osdmap e97: 8 total, 8 up, 8 in 2021-03-24T21:54:18.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:18 smithi125 conmon[19936]: cluster 2021-03-24T21:54:18.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:18 smithi125 conmon[19936]: 2021-03-24T21: 2021-03-24T21:54:18.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:18 smithi125 conmon[19936]: 54:17.493851+0000 osd.3 (osd.3 2021-03-24T21:54:18.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:18 smithi125 conmon[19936]: ) 38 : cluster [DBG] 5.0 scrub ok 2021-03-24T21:54:18.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:18 smithi125 conmon[24927]: cluster 2021-03-24T21:54:16. 2021-03-24T21:54:18.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:18 smithi125 conmon[24927]: 700778+0000 mgr.y (mgr.14144) 301 : cluster [DBG] pgmap v302: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-24T21:54:18.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:18 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:54:18.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:18 smithi125 conmon[24927]: 54:16.916128+0000 osd.6 (osd.6) 16 : 2021-03-24T21:54:18.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:18 smithi125 conmon[24927]: cluster [DBG] purged_snaps scrub starts 2021-03-24T21:54:18.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:18 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:54:18.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:18 smithi125 conmon[24927]: -24T21:54:16. 2021-03-24T21:54:18.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:18 smithi125 conmon[24927]: 916297+0000 osd.6 (osd.6) 17 : cluster 2021-03-24T21:54:18.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:18 smithi125 conmon[24927]: [DBG] purged_snaps scrub ok 2021-03-24T21:54:18.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:18 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:54:18.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:18 smithi125 conmon[24927]: -24T21:54:17.122101+0000 mon.a (mon 2021-03-24T21:54:18.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:18 smithi125 conmon[24927]: .0) 614 : cluster [DBG] osdmap e97: 8 total, 8 up, 8 in 2021-03-24T21:54:18.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:18 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:54:18.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:18 smithi125 conmon[24927]: 21:54:17.493851+0000 osd.3 (osd. 2021-03-24T21:54:18.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:18 smithi125 conmon[24927]: 3) 38 : cluster [DBG] 5.0 scrub ok 2021-03-24T21:54:18.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:18 smithi142 conmon[20125]: cluster 2021-03-24T21:54:16 2021-03-24T21:54:18.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:18 smithi142 conmon[20125]: .700778+0000 mgr.y (mgr.14144) 301 : cluster [DBG] pgmap v302: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-24T21:54:18.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:18 smithi142 conmon[20125]: cluster 2021-03-24T21 2021-03-24T21:54:18.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:18 smithi142 conmon[20125]: :54:16.916128+0000 osd.6 (osd.6) 16 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:54:18.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:18 smithi142 conmon[20125]: cluster 2021-03-24T21:54:18.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:18 smithi142 conmon[20125]: 2021-03- 2021-03-24T21:54:18.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:18 smithi142 conmon[20125]: 24T21:54:16.916297+0000 osd.6 (osd 2021-03-24T21:54:18.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:18 smithi142 conmon[20125]: .6) 17 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:54:18.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:18 smithi142 conmon[20125]: cluster 2021-03-24T21:54:18.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:18 smithi142 conmon[20125]: 2021-03-24T21:54:17.122101+0000 2021-03-24T21:54:18.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:18 smithi142 conmon[20125]: mon.a (mon.0) 614 : cluster [DBG] osdmap e97: 8 total, 8 up, 8 in 2021-03-24T21:54:18.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:18 smithi142 conmon[20125]: 2021-03-24T21:54:18.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:18 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:54:18.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:18 smithi142 conmon[20125]: -03-24T21:54:17.493851+0000 2021-03-24T21:54:18.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:18 smithi142 conmon[20125]: osd.3 (osd.3) 38 : cluster [DBG] 5.0 scrub ok 2021-03-24T21:54:18.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:18 smithi142 conmon[20125]: 2021-03-24T21:54:19.552 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:19.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:19.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:54:19.660 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:19.717 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:19.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:19.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:54:19.822 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:19.873 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:19.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:19.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:54:19.947 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-24T21:54:19.948 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-24T21:54:19.948 INFO:teuthology.orchestra.run.smithi125.stdout: "num_ops": 0 2021-03-24T21:54:19.948 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-24T21:54:19.978 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:20.027 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:20.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:20.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:54:20.131 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:20.144 INFO:teuthology.orchestra.run.smithi125.stderr:noscrub is set 2021-03-24T21:54:20.177 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:20.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:20.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:54:20.256 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:20 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:54:20.257 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:20 smithi125 conmon[24927]: 24T21:54:18.594188+0000 osd.0 (osd.0) 34 : cluster 2021-03-24T21:54:20.257 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:20 smithi125 conmon[24927]: [DBG] 3.1 scrub ok 2021-03-24T21:54:20.257 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:20 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:54:20.258 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:20 smithi125 conmon[24927]: 24T21:54:18.701705+0000 mgr.y 2021-03-24T21:54:20.258 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:20 smithi125 conmon[24927]: (mgr.14144) 302 : cluster [DBG] 2021-03-24T21:54:20.258 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:20 smithi125 conmon[24927]: pgmap v304: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-03-24T21:54:20.259 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:20 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:54:20.259 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:20 smithi125 conmon[24927]: 54:18.973097+0000 osd.6 (osd.6) 18 : cluster [DBG] 4.a scrub ok 2021-03-24T21:54:20.259 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:20 smithi125 conmon[24927]: cluster 2021-03-24T21:54: 2021-03-24T21:54:20.259 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:20 smithi125 conmon[24927]: 19.157250+0000 osd.5 (osd.5) 30 : cluster 2021-03-24T21:54:20.259 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:20 smithi125 conmon[24927]: [DBG] 4.8 deep-scrub ok 2021-03-24T21:54:20.260 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:20 smithi125 conmon[24927]: audit 2021-03-24T21:54:19. 2021-03-24T21:54:20.260 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:20 smithi125 conmon[24927]: 796650+0000 mon.a (mon.0) 615 : audit [INF] from='client.? v1:172.21.15.125:0/3867333052' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T21:54:20.260 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:20 smithi125 conmon[24927]: 2021-03-24T21:54:20.260 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:20 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:54:20.261 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:20 smithi125 conmon[19936]: 03-24T21:54:18.594188+0000 osd.0 (osd.0) 34 : 2021-03-24T21:54:20.261 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:20 smithi125 conmon[19936]: cluster [DBG] 3.1 scrub ok 2021-03-24T21:54:20.261 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:20 smithi125 conmon[19936]: cluster 2021-03-24T21:54:20.261 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:20 smithi125 conmon[19936]: 2021- 2021-03-24T21:54:20.262 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:20 smithi125 conmon[19936]: 03-24T21:54:18.701705+0000 2021-03-24T21:54:20.262 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:20 smithi125 conmon[19936]: mgr.y (mgr.14144) 302 : cluster [DBG] 2021-03-24T21:54:20.262 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:20 smithi125 conmon[19936]: pgmap v304: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-03-24T21:54:20.262 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:20 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:54:20.263 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:20 smithi125 conmon[19936]: -03-24T21:54 2021-03-24T21:54:20.263 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:20 smithi125 conmon[19936]: :18.973097+0000 osd.6 2021-03-24T21:54:20.263 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:20 smithi125 conmon[19936]: (osd.6) 18 : 2021-03-24T21:54:20.263 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:20 smithi125 conmon[19936]: cluster [DBG] 4.a scrub ok 2021-03-24T21:54:20.263 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:20 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:54:20.264 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:20 smithi125 conmon[19936]: -03-24T21:54 2021-03-24T21:54:20.264 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:20 smithi125 conmon[19936]: :19.157250+0000 osd.5 2021-03-24T21:54:20.264 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:20 smithi125 conmon[19936]: (osd.5) 30 : 2021-03-24T21:54:20.264 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:20 smithi125 conmon[19936]: cluster [DBG] 4.8 deep-scrub ok 2021-03-24T21:54:20.265 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:20 smithi125 conmon[19936]: audit 2021 2021-03-24T21:54:20.265 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:20 smithi125 conmon[19936]: -03-24T 2021-03-24T21:54:20.265 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:20 smithi125 conmon[19936]: 21:54 2021-03-24T21:54:20.265 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:20 smithi125 conmon[19936]: :19.796650+0000 mon.a 2021-03-24T21:54:20.265 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:20 smithi125 conmon[19936]: (mon.0) 615 : 2021-03-24T21:54:20.266 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:20 smithi125 conmon[19936]: audit [INF] from='client.? v1:172.21.15.125:0/3867333052' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T21:54:20.282 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:54:20.338 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:54:20.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:20.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:54:20.404 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T21:54:20.425 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T21:54:20.447 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:20.450 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T21:54:20.461 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:20 smithi142 conmon[20125]: cluster 2021-03-24T21:54:18. 2021-03-24T21:54:20.461 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:20 smithi142 conmon[20125]: 594188+0000 osd.0 (osd.0) 34 : cluster [DBG] 3.1 scrub ok 2021-03-24T21:54:20.461 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:20 smithi142 conmon[20125]: cluster 2021-03-24T21:54 2021-03-24T21:54:20.462 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:20 smithi142 conmon[20125]: :18.701705+0000 mgr.y (mgr.14144) 302 : cluster [DBG] 2021-03-24T21:54:20.462 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:20 smithi142 conmon[20125]: pgmap v304: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-03-24T21:54:20.462 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:20 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:54:20.462 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:20 smithi142 conmon[20125]: 03-24T21:54:18.973097 2021-03-24T21:54:20.463 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:20 smithi142 conmon[20125]: +0000 osd.6 (osd.6) 18 : 2021-03-24T21:54:20.463 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:20 smithi142 conmon[20125]: cluster [DBG] 4.a scrub ok 2021-03-24T21:54:20.463 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:20 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:54:20.463 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:20 smithi142 conmon[20125]: 03-24T21:54:19.157250 2021-03-24T21:54:20.464 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:20 smithi142 conmon[20125]: +0000 osd.5 (osd.5) 30 2021-03-24T21:54:20.464 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:20 smithi142 conmon[20125]: : cluster [DBG] 4.8 deep-scrub ok 2021-03-24T21:54:20.464 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:20 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:54:20.464 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:20 smithi142 conmon[20125]: 03-24T21:54:19.796650 2021-03-24T21:54:20.464 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:20 smithi142 conmon[20125]: +0000 mon.a (mon.0) 615 2021-03-24T21:54:20.465 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:20 smithi142 conmon[20125]: : audit [INF] from='client.? v1:172.21.15.125:0/3867333052' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T21:54:20.476 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T21:54:20.500 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T21:54:20.505 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:20.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:20.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:54:20.521 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T21:54:20.544 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T21:54:20.567 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T21:54:20.568 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-24T21:54:20.568 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-24T21:54:20.569 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-24T21:54:20.569 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-24T21:54:20.569 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-24T21:54:20.569 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-24T21:54:20.570 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-24T21:54:20.570 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-24T21:54:20.610 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:20.666 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:20.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:20.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:54:20.771 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:54:20.829 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:54:20.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:20.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:54:20.936 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:20.997 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:21.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:21.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:54:21.102 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:21.151 INFO:teuthology.orchestra.run.smithi125.stderr:pool 'unique_pool_0' created 2021-03-24T21:54:21.155 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:21.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:21.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:54:21.259 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:21.310 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:21.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:21.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:54:21.386 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[19936]: cluster 2021-03-24T21:54 2021-03-24T21:54:21.386 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[19936]: :19.588790+0000 osd.0 (osd.0) 35 : cluster [DBG] 4.16 scrub ok 2021-03-24T21:54:21.386 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:54:21.387 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[19936]: 21:54:20.101454+0000 osd.5 (osd.5) 31 : cluster [DBG] 3.5 scrub ok 2021-03-24T21:54:21.387 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:54:21.387 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[19936]: 24T21:54:20.121112+0000 mon.a (mon 2021-03-24T21:54:21.387 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[19936]: .0) 616 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:54:21.388 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:54:21.388 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[19936]: 03-24T21:54:20 2021-03-24T21:54:21.388 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[19936]: .140678+0000 mon.a ( 2021-03-24T21:54:21.388 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[19936]: mon.0) 617 : 2021-03-24T21:54:21.388 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[19936]: audit [INF] from='client.? v1:172.21.15.125:0/3867333052' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T21:54:21.389 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:54:21.389 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[19936]: -03-24T21: 2021-03-24T21:54:21.389 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[19936]: 54:20.140786+0000 2021-03-24T21:54:21.389 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[19936]: mon.a (mon.0) 2021-03-24T21:54:21.390 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[19936]: 618 : cluster [DBG] 2021-03-24T21:54:21.390 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[19936]: osdmap e98: 8 total, 8 up, 8 in 2021-03-24T21:54:21.390 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[19936]: audit 2021-03-24T21:54:21.390 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[19936]: 2021-03-24T21 2021-03-24T21:54:21.390 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[19936]: :54:20.379290+0000 2021-03-24T21:54:21.391 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[19936]: mon.c (mon.2 2021-03-24T21:54:21.391 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[19936]: ) 21 : audit [DBG] 2021-03-24T21:54:21.391 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[19936]: from='client.? v1:172.21.15.125:0/3783613405' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:54:21.391 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[19936]: audit 2021-03-24T21:54:21.392 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[19936]: 2021-03-24T21 2021-03-24T21:54:21.392 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[19936]: :54:20.640921 2021-03-24T21:54:21.392 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[19936]: +0000 mon.c (mon. 2021-03-24T21:54:21.392 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[19936]: 2) 22 : audit 2021-03-24T21:54:21.392 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[19936]: [INF] from='client.? v1:172.21.15.125:0/735388148' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2021-03-24T21:54:21.393 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[19936]: audit 2021-03-24T21:54:21.393 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[19936]: 2021-03-24T21 2021-03-24T21:54:21.393 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[19936]: :54:20.641359 2021-03-24T21:54:21.394 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[19936]: +0000 mon.a (mon.0 2021-03-24T21:54:21.394 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[19936]: ) 619 : audit 2021-03-24T21:54:21.394 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[19936]: [INF] from='client.34649 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2021-03-24T21:54:21.394 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:54:21.395 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[24927]: 24T21:54:19.588790+0000 osd.0 (osd.0) 35 : cluster [DBG] 2021-03-24T21:54:21.395 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[24927]: 4.16 scrub ok 2021-03-24T21:54:21.395 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:54:21.395 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[24927]: 03-24T21:54: 2021-03-24T21:54:21.396 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[24927]: 20.101454+0000 osd.5 (osd 2021-03-24T21:54:21.396 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[24927]: .5) 31 : cluster [DBG] 2021-03-24T21:54:21.396 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[24927]: 3.5 scrub ok 2021-03-24T21:54:21.396 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:54:21.396 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[24927]: 03-24T21:54:20 2021-03-24T21:54:21.397 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[24927]: .121112+0000 mon.a (mon.0 2021-03-24T21:54:21.397 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[24927]: ) 616 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:54:21.397 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[24927]: 2021-03-24T21:54:21.397 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T21:54:21.397 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[24927]: 24T21:54:20.140678 2021-03-24T21:54:21.398 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[24927]: +0000 mon.a (mon.0) 617 2021-03-24T21:54:21.398 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[24927]: : audit [INF] from='client.? v1:172.21.15.125:0/3867333052' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T21:54:21.398 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:54:21.398 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[24927]: -03-24T21:54 2021-03-24T21:54:21.399 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[24927]: :20.140786+0000 mon.a (mon.0) 2021-03-24T21:54:21.399 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[24927]: 618 : cluster [DBG] osdmap e98: 8 total, 8 up, 8 in 2021-03-24T21:54:21.399 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[24927]: audit 2021-03-24T 2021-03-24T21:54:21.399 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[24927]: 21:54:20.379290+0000 mon.c (mon. 2021-03-24T21:54:21.399 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[24927]: 2) 21 : audit [DBG] from='client.? v1:172.21.15.125:0/3783613405' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:54:21.400 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[24927]: 2021-03-24T21:54:21.400 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[24927]: audit 2021-03-24 2021-03-24T21:54:21.400 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[24927]: T21:54:20.640921+0000 mon.c 2021-03-24T21:54:21.400 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[24927]: (mon.2) 22 : audit [INF] 2021-03-24T21:54:21.400 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[24927]: from='client.? v1:172.21.15.125:0/735388148' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2021-03-24T21:54:21.401 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[24927]: audit 2021-03-24T21:54:21.401 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[24927]: 2021-03-24T21:54:20.641359+0000 2021-03-24T21:54:21.401 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[24927]: mon.a (mon.0) 619 : audit [INF] 2021-03-24T21:54:21.401 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:21 smithi125 conmon[24927]: from='client.34649 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2021-03-24T21:54:21.416 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:54:21.428 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:21 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:54:21.429 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:21 smithi142 conmon[20125]: 54:19.588790+0000 osd.0 (osd.0) 35 : cluster [DBG] 4.16 scrub ok 2021-03-24T21:54:21.429 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:21 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:54:21.429 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:21 smithi142 conmon[20125]: 03-24T21:54:20.101454+0000 osd.5 (osd.5) 2021-03-24T21:54:21.430 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:21 smithi142 conmon[20125]: 31 : cluster [DBG] 3.5 scrub ok 2021-03-24T21:54:21.430 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:21 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:54:21.430 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:21 smithi142 conmon[20125]: T21:54:20.121112+0000 mon.a ( 2021-03-24T21:54:21.430 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:21 smithi142 conmon[20125]: mon.0) 616 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:54:21.431 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:21 smithi142 conmon[20125]: audit 2021-03-24T21:54:21.431 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:21 smithi142 conmon[20125]: 2021-03-24T21:54:20. 2021-03-24T21:54:21.431 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:21 smithi142 conmon[20125]: 140678+0000 mon.a (mon.0) 617 : audit 2021-03-24T21:54:21.431 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:21 smithi142 conmon[20125]: [INF] from='client.? v1:172.21.15.125:0/3867333052' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T21:54:21.432 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:21 smithi142 conmon[20125]: cluster 2021-03-24T21 2021-03-24T21:54:21.432 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:21 smithi142 conmon[20125]: :54:20.140786+0000 mon.a (mon. 2021-03-24T21:54:21.432 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:21 smithi142 conmon[20125]: 0) 618 : cluster [DBG] osdmap e98: 8 total, 8 up, 8 in 2021-03-24T21:54:21.432 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:21 smithi142 conmon[20125]: audit 2021-03- 2021-03-24T21:54:21.433 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:21 smithi142 conmon[20125]: 24T21:54:20.379290+0000 mon.c ( 2021-03-24T21:54:21.433 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:21 smithi142 conmon[20125]: mon.2) 21 : audit [DBG] from='client.? v1:172.21.15.125:0/3783613405' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:54:21.433 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:21 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:54:21.433 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:21 smithi142 conmon[20125]: 03-24T21:54:20.640921+0000 2021-03-24T21:54:21.434 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:21 smithi142 conmon[20125]: mon.c (mon.2) 22 : audit [INF] 2021-03-24T21:54:21.434 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:21 smithi142 conmon[20125]: from='client.? v1:172.21.15.125:0/735388148' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2021-03-24T21:54:21.434 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:21 smithi142 conmon[20125]: audit 2021-03-24T21:54:21.434 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:21 smithi142 conmon[20125]: 2021-03-24T21:54:20.641359+0000 2021-03-24T21:54:21.434 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:21 smithi142 conmon[20125]: mon.a (mon.0) 619 : audit [INF] 2021-03-24T21:54:21.435 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:21 smithi142 conmon[20125]: from='client.34649 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2021-03-24T21:54:21.467 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:54:21.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:21.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:54:21.573 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:21.635 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:21.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:21.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:54:21.742 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:21.806 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:21.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:21.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:54:21.913 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:21.969 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:21.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:21.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:54:22.075 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:22.128 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:22.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:22.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:54:22.233 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:22.240 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:22 smithi125 conmon[24927]: cluster 2021-03-24T21:54:22.240 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:22 smithi125 conmon[24927]: 2021-03-24T21:54:22.240 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:22 smithi125 conmon[24927]: 2021 2021-03-24T21:54:22.240 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:22 smithi125 conmon[24927]: -03-24T21:54:20.702258+0000 2021-03-24T21:54:22.241 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:22 smithi125 conmon[24927]: mgr.y (mgr.14144) 303 : cluster [DBG] pgmap v306: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-03-24T21:54:22.241 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:22 smithi125 conmon[24927]: audit 2021-03-24T21:54:21.144764+0000 mon.a (mon.0 2021-03-24T21:54:22.241 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:22 smithi125 conmon[24927]: ) 620 : audit [INF] from='client.34649 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2021-03-24T21:54:22.241 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:22 smithi125 conmon[24927]: cluster 2021-03-24T21:54:21.144822+0000 2021-03-24T21:54:22.242 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:22 smithi125 conmon[24927]: mon.a (mon.0) 621 : cluster [DBG] osdmap e99: 8 total, 8 up, 8 in 2021-03-24T21:54:22.242 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:22 smithi125 conmon[19936]: cluster 2021-03-24T21:54:20.702258+0000 mgr.y (mgr.14144) 303 2021-03-24T21:54:22.242 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:22 smithi125 conmon[19936]: : cluster [DBG] pgmap v306: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-03-24T21:54:22.243 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:22 smithi125 conmon[19936]: audit 2021-03-24T21:54:21.144764+0000 mon.a (mon.0) 620 : audit 2021-03-24T21:54:22.243 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:22 smithi125 conmon[19936]: [INF] from='client.34649 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2021-03-24T21:54:22.243 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:22 smithi125 conmon[19936]: cluster 2021-03-24T21:54:22.243 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:22 smithi125 conmon[19936]: 2021-03-24T21:54:22.244 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:22 smithi125 conmon[19936]: 2021 2021-03-24T21:54:22.244 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:22 smithi125 conmon[19936]: - 2021-03-24T21:54:22.244 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:22 smithi125 conmon[19936]: 03 2021-03-24T21:54:22.244 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:22 smithi125 conmon[19936]: - 2021-03-24T21:54:22.244 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:22 smithi125 conmon[19936]: 24 2021-03-24T21:54:22.245 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:22 smithi125 conmon[19936]: T 2021-03-24T21:54:22.245 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:22 smithi125 conmon[19936]: 21 2021-03-24T21:54:22.245 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:22 smithi125 conmon[19936]: : 2021-03-24T21:54:22.245 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:22 smithi125 conmon[19936]: 54 2021-03-24T21:54:22.245 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:22 smithi125 conmon[19936]: : 2021-03-24T21:54:22.246 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:22 smithi125 conmon[19936]: 21 2021-03-24T21:54:22.246 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:22 smithi125 conmon[19936]: . 2021-03-24T21:54:22.246 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:22 smithi125 conmon[19936]: 144822 2021-03-24T21:54:22.246 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:22 smithi125 conmon[19936]: +0000 2021-03-24T21:54:22.247 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:22 smithi125 conmon[19936]: 2021-03-24T21:54:22.247 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:22 smithi125 conmon[19936]: mon.a 2021-03-24T21:54:22.247 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:22 smithi125 conmon[19936]: ( 2021-03-24T21:54:22.247 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:22 smithi125 conmon[19936]: mon 2021-03-24T21:54:22.248 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:22 smithi125 conmon[19936]: . 2021-03-24T21:54:22.248 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:22 smithi125 conmon[19936]: 0 2021-03-24T21:54:22.248 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:22 smithi125 conmon[19936]: ) 2021-03-24T21:54:22.248 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:22 smithi125 conmon[19936]: 621 : cluster [DBG] osdmap e99: 8 total, 8 up, 8 in 2021-03-24T21:54:22.290 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:22.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:22.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:54:22.397 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:22.456 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:22.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:22.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:54:22.562 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:22.576 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:22 smithi142 conmon[20125]: cluster 2021-03-24T21:54:20.702258+0000 mgr.y (mgr. 2021-03-24T21:54:22.576 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:22 smithi142 conmon[20125]: 14144) 303 : cluster [DBG] pgmap v306: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-03-24T21:54:22.576 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:22 smithi142 conmon[20125]: audit 2021-03-24T21:54:21.144764+0000 mon.a (mon.0) 620 : audit [INF] from='client.34649 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2021-03-24T21:54:22.576 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:22 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:54:22.577 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:22 smithi142 conmon[20125]: 24T21:54:21.144822+0000 mon.a (mon.0) 621 : cluster [DBG] osdmap e99: 8 total, 8 up, 8 in 2021-03-24T21:54:22.624 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:22.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:22.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:54:22.729 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:22.788 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:22.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:22.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:54:22.892 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:22.935 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:22.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:22.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:54:23.043 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:23.100 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:23.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:23.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:54:23.207 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:23.250 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:23.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:23.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:54:23.315 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:23 smithi125 conmon[19936]: debug 2021-03-24T21:54:23.044+0000 7f2118128700 -1 mon.a@0(leader).osd e100 definitely_dead 0 2021-03-24T21:54:23.315 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:23 smithi125 conmon[19936]: cluster 2021-03-24T21:54:23.356 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:23.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:23 smithi142 conmon[20125]: cluster 2021-03-24T21:54:23.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:23 smithi142 conmon[20125]: 2021-03-24T21:54:23.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:23 smithi142 conmon[20125]: 2021 2021-03-24T21:54:23.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:23 smithi142 conmon[20125]: - 2021-03-24T21:54:23.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:23 smithi142 conmon[20125]: 03 2021-03-24T21:54:23.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:23 smithi142 conmon[20125]: - 2021-03-24T21:54:23.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:23 smithi142 conmon[20125]: 24 2021-03-24T21:54:23.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:23 smithi142 conmon[20125]: T 2021-03-24T21:54:23.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:23 smithi142 conmon[20125]: 21 2021-03-24T21:54:23.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:23 smithi142 conmon[20125]: : 2021-03-24T21:54:23.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:23 smithi142 conmon[20125]: 54 2021-03-24T21:54:23.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:23 smithi142 conmon[20125]: : 2021-03-24T21:54:23.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:23 smithi142 conmon[20125]: 22 2021-03-24T21:54:23.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:23 smithi142 conmon[20125]: . 2021-03-24T21:54:23.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:23 smithi142 conmon[20125]: 151806 2021-03-24T21:54:23.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:23 smithi142 conmon[20125]: +0000 2021-03-24T21:54:23.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:23 smithi142 conmon[20125]: 2021-03-24T21:54:23.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:23 smithi142 conmon[20125]: mon.a 2021-03-24T21:54:23.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:23 smithi142 conmon[20125]: ( 2021-03-24T21:54:23.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:23 smithi142 conmon[20125]: mon 2021-03-24T21:54:23.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:23 smithi142 conmon[20125]: .0) 622 : cluster [DBG] osdmap e100: 8 total, 8 up, 8 in 2021-03-24T21:54:23.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:23 smithi142 conmon[20125]: audit 2021-03-24T21:54:23.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:23 smithi142 conmon[20125]: 2021-03-24T21:54:23.373 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:23 smithi142 conmon[20125]: 2021-03-24T21:54: 2021-03-24T21:54:23.373 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:23 smithi142 conmon[20125]: 23.047849+0000 mon.a (mon.0) 623 : audit [INF] from='client.? v1:172.21.15.125:0/697022427' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2021-03-24T21:54:23.373 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:23 smithi142 conmon[20125]: cluster 2021-03-24T21:54: 2021-03-24T21:54:23.374 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:23 smithi142 conmon[20125]: 23.048014+0000 mon.a (mon.0) 624 : cluster [INF] Client client.admin marked osd.6 out, while it was still marked up 2021-03-24T21:54:23.416 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:23.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:23.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:54:23.502 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.0 dump_blocked_ops 2021-03-24T21:54:23.522 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:23.601 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:23.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:23 smithi125 conmon[24927]: cluster 2021-03-24T21:54:22.151806+0000 mon.a (mon. 2021-03-24T21:54:23.606 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:23 smithi125 conmon[24927]: 0) 622 : cluster [DBG] osdmap e100: 8 total, 8 up, 8 in 2021-03-24T21:54:23.606 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:23 smithi125 conmon[24927]: audit 2021-03-24T21:54:23.606 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:23 smithi125 conmon[24927]: 2021-03-24T21:54:23.047849+0000 mon.a (mon.0) 623 2021-03-24T21:54:23.606 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:23 smithi125 conmon[24927]: : audit [INF] from='client.? v1:172.21.15.125:0/697022427' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2021-03-24T21:54:23.606 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:23 smithi125 conmon[24927]: cluster 2021-03-24T21:54 2021-03-24T21:54:23.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:23 smithi125 conmon[24927]: :23.048014+0000 mon.a (mon.0) 624 : cluster 2021-03-24T21:54:23.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:23 smithi125 conmon[24927]: [INF] Client client.admin marked osd.6 out, while it was still marked up 2021-03-24T21:54:23.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:23.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:54:23.609 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:23 smithi125 conmon[19936]: 2021-03-24T21:54:23.609 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:23 smithi125 conmon[19936]: 2021-03-24T21:54:22.151806+0000 mon.a 2021-03-24T21:54:23.609 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:23 smithi125 conmon[19936]: (mon.0) 622 : cluster [DBG] osdmap e100: 8 total, 8 up, 8 in 2021-03-24T21:54:23.610 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:23 smithi125 conmon[19936]: audit 2021-03-24T21:54: 2021-03-24T21:54:23.610 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:23 smithi125 conmon[19936]: 23.047849+0000 mon.a (mon.0) 623 : audit [INF] from='client.? v1:172.21.15.125:0/697022427' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2021-03-24T21:54:23.610 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:23 smithi125 conmon[19936]: cluster 2021-03-24T21:54:23.610 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:23 smithi125 conmon[19936]: 2021-03-24T21:54:23.610 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:23 smithi125 conmon[19936]: 2021 2021-03-24T21:54:23.611 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:23 smithi125 conmon[19936]: - 2021-03-24T21:54:23.611 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:23 smithi125 conmon[19936]: 03 2021-03-24T21:54:23.611 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:23 smithi125 conmon[19936]: - 2021-03-24T21:54:23.611 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:23 smithi125 conmon[19936]: 24 2021-03-24T21:54:23.611 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:23 smithi125 conmon[19936]: T 2021-03-24T21:54:23.612 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:23 smithi125 conmon[19936]: 21 2021-03-24T21:54:23.612 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:23 smithi125 conmon[19936]: : 2021-03-24T21:54:23.612 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:23 smithi125 conmon[19936]: 54 2021-03-24T21:54:23.612 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:23 smithi125 conmon[19936]: : 2021-03-24T21:54:23.613 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:23 smithi125 conmon[19936]: 23 2021-03-24T21:54:23.613 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:23 smithi125 conmon[19936]: . 2021-03-24T21:54:23.613 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:23 smithi125 conmon[19936]: 048014 2021-03-24T21:54:23.613 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:23 smithi125 conmon[19936]: +0000 2021-03-24T21:54:23.613 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:23 smithi125 conmon[19936]: 2021-03-24T21:54:23.614 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:23 smithi125 conmon[19936]: mon.a 2021-03-24T21:54:23.614 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:23 smithi125 conmon[19936]: ( 2021-03-24T21:54:23.614 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:23 smithi125 conmon[19936]: mon 2021-03-24T21:54:23.614 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:23 smithi125 conmon[19936]: . 2021-03-24T21:54:23.614 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:23 smithi125 conmon[19936]: 0 2021-03-24T21:54:23.615 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:23 smithi125 conmon[19936]: ) 2021-03-24T21:54:23.615 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:23 smithi125 conmon[19936]: 624 2021-03-24T21:54:23.615 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:23 smithi125 conmon[19936]: : 2021-03-24T21:54:23.615 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:23 smithi125 conmon[19936]: cluster 2021-03-24T21:54:23.616 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:23 smithi125 conmon[19936]: 2021-03-24T21:54:23.616 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:23 smithi125 conmon[19936]: [INF] 2021-03-24T21:54:23.616 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:23 smithi125 conmon[19936]: 2021-03-24T21:54:23.616 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:23 smithi125 conmon[19936]: Client client.admin marked osd.6 out, while it was still marked up 2021-03-24T21:54:23.616 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:23 smithi125 conmon[19936]: 2021-03-24T21:54:23.695 INFO:teuthology.orchestra.run.smithi125.stderr:marked out osd.6. 2021-03-24T21:54:23.709 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:23.757 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2021-03-24T21:54:23.772 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:23.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:23.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:54:23.877 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:23.932 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:24.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:24.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:54:24.171 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:24.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:24 smithi142 conmon[20125]: cluster 2021-03-24T21:54:22.702927+0000 mgr.y (mgr 2021-03-24T21:54:24.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:24 smithi142 conmon[20125]: .14144) 304 : cluster [DBG] pgmap v309: 148 pgs: 2 creating+peering, 14 unknown, 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 916 B/s rd, 0 op/s 2021-03-24T21:54:24.194 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:24 smithi142 conmon[20125]: audit 2021-03-24T21:54:24.195 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:24 smithi142 conmon[20125]: 2021-03-24T21:54:23.172020+0000 2021-03-24T21:54:24.195 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:24 smithi142 conmon[20125]: mon.a (mon.0) 625 : audit [INF] from='client.? v1:172.21.15.125:0/697022427' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2021-03-24T21:54:24.195 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:24 smithi142 conmon[20125]: cluster 2021-03-24T21:54:23.172604+0000 mon.a (mon.0 2021-03-24T21:54:24.195 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:24 smithi142 conmon[20125]: ) 626 : cluster [DBG] osdmap e101: 8 total, 8 up, 7 in 2021-03-24T21:54:24.230 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:24.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:24.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:54:24.335 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:24.384 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:24.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:24.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:54:24.490 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:24.503 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[24927]: cluster 2021-03-24T21:54:22.702927+0000 2021-03-24T21:54:24.503 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[24927]: mgr.y (mgr.14144) 304 : cluster [DBG] pgmap v309: 148 pgs: 2 creating+peering, 14 unknown, 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 916 B/s rd, 0 op/s 2021-03-24T21:54:24.503 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[24927]: audit 2021-03-24T21:54: 2021-03-24T21:54:24.504 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[24927]: 23.172020+0000 mon.a (mon.0) 625 : audit 2021-03-24T21:54:24.504 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[24927]: [INF] from='client.? v1:172.21.15.125:0/697022427' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2021-03-24T21:54:24.504 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:54:24.504 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[24927]: -24T21:54:23.172604+0000 2021-03-24T21:54:24.505 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[24927]: mon.a (mon.0) 626 : cluster [DBG] osdmap e101: 8 total, 8 up, 7 in 2021-03-24T21:54:24.505 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[24927]: 2021-03-24T21:54:24.506 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: cluster 2021-03-24T21:54:24.506 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: 2021-03-24T21:54:24.507 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: 2021 2021-03-24T21:54:24.507 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: - 2021-03-24T21:54:24.507 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: 03 2021-03-24T21:54:24.507 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: - 2021-03-24T21:54:24.507 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: 24 2021-03-24T21:54:24.508 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: T 2021-03-24T21:54:24.508 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: 21 2021-03-24T21:54:24.508 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: : 2021-03-24T21:54:24.508 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: 54 2021-03-24T21:54:24.509 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: : 2021-03-24T21:54:24.509 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: 22 2021-03-24T21:54:24.509 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: . 2021-03-24T21:54:24.509 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: 702927 2021-03-24T21:54:24.509 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: +0000 2021-03-24T21:54:24.510 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: 2021-03-24T21:54:24.510 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: mgr.y 2021-03-24T21:54:24.510 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: ( 2021-03-24T21:54:24.510 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: mgr 2021-03-24T21:54:24.511 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: . 2021-03-24T21:54:24.511 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: 14144 2021-03-24T21:54:24.511 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: ) 2021-03-24T21:54:24.511 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: 304 2021-03-24T21:54:24.511 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: : 2021-03-24T21:54:24.512 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: cluster 2021-03-24T21:54:24.512 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: 2021-03-24T21:54:24.512 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: [DBG] 2021-03-24T21:54:24.512 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: 2021-03-24T21:54:24.512 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: pgmap v309: 148 pgs: 2 creating+peering, 14 unknown, 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 916 B/s rd, 0 op/s 2021-03-24T21:54:24.513 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: 2021-03-24T21:54:24.513 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: audit 2021-03-24T21:54:24.513 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: 2021-03-24T21:54:24.513 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: 2021 2021-03-24T21:54:24.514 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: - 2021-03-24T21:54:24.514 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: 03 2021-03-24T21:54:24.514 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: - 2021-03-24T21:54:24.514 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: 24 2021-03-24T21:54:24.514 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: T 2021-03-24T21:54:24.515 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: 21 2021-03-24T21:54:24.515 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: : 2021-03-24T21:54:24.515 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: 54 2021-03-24T21:54:24.515 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: : 2021-03-24T21:54:24.516 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: 23 2021-03-24T21:54:24.516 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: . 2021-03-24T21:54:24.516 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: 172020 2021-03-24T21:54:24.516 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: +0000 2021-03-24T21:54:24.516 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: 2021-03-24T21:54:24.517 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: mon.a 2021-03-24T21:54:24.517 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: ( 2021-03-24T21:54:24.517 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: mon 2021-03-24T21:54:24.517 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: . 2021-03-24T21:54:24.517 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: 0 2021-03-24T21:54:24.518 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: ) 2021-03-24T21:54:24.518 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: 625 2021-03-24T21:54:24.518 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: : 2021-03-24T21:54:24.518 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: audit 2021-03-24T21:54:24.519 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: 2021-03-24T21:54:24.519 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: [INF] 2021-03-24T21:54:24.519 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: 2021-03-24T21:54:24.519 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: from='client.? v1:172.21.15.125:0/697022427' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2021-03-24T21:54:24.519 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: 2021-03-24T21:54:24.520 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: cluster 2021-03-24T21:54:24.520 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: 2021-03-24T21:54:24.520 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: 2021 2021-03-24T21:54:24.520 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: - 2021-03-24T21:54:24.520 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: 03 2021-03-24T21:54:24.521 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: - 2021-03-24T21:54:24.521 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: 24 2021-03-24T21:54:24.521 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: T 2021-03-24T21:54:24.521 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: 21 2021-03-24T21:54:24.521 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: : 2021-03-24T21:54:24.522 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: 54 2021-03-24T21:54:24.522 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: : 2021-03-24T21:54:24.522 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: 23 2021-03-24T21:54:24.522 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: . 2021-03-24T21:54:24.523 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: 172604 2021-03-24T21:54:24.523 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: +0000 2021-03-24T21:54:24.523 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: 2021-03-24T21:54:24.523 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: mon.a 2021-03-24T21:54:24.523 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: ( 2021-03-24T21:54:24.524 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: mon 2021-03-24T21:54:24.524 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: . 2021-03-24T21:54:24.524 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: 0 2021-03-24T21:54:24.524 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: ) 2021-03-24T21:54:24.524 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: 626 2021-03-24T21:54:24.525 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: : 2021-03-24T21:54:24.525 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: cluster 2021-03-24T21:54:24.525 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: 2021-03-24T21:54:24.525 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: [DBG] 2021-03-24T21:54:24.526 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: 2021-03-24T21:54:24.526 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: osdmap e101: 8 total, 8 up, 7 in 2021-03-24T21:54:24.526 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:24 smithi125 conmon[19936]: 2021-03-24T21:54:24.544 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:24.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:24.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:54:24.588 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:54:24.645 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:54:24.650 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:24.694 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:24.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:24.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:54:24.799 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:24.845 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:24.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:24.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:54:24.898 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-24T21:54:24.950 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:24.995 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:24.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:24.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:54:25.100 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:25.154 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:25.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:25.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:54:25.258 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:25.304 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:25.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:25.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:54:25.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:25 smithi142 conmon[20125]: cluster 2021-03-24T21 2021-03-24T21:54:25.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:25 smithi142 conmon[20125]: :54:24.172501+0000 mon.a (mon.0) 627 : cluster [DBG] osdmap e102: 8 total, 8 up, 7 in 2021-03-24T21:54:25.409 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:25.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:25 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:54:25.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:25 smithi125 conmon[19936]: 24T21:54:24.172501+0000 mon.a (mon.0) 627 : cluster [DBG] osdmap e102: 8 total, 8 up, 7 in 2021-03-24T21:54:25.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:25 smithi125 conmon[24927]: cluster 2021-03-24T21:54:24 2021-03-24T21:54:25.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:25 smithi125 conmon[24927]: .172501+0000 mon.a (mon.0) 627 : cluster [DBG] osdmap e102: 8 total, 8 up, 7 in 2021-03-24T21:54:25.462 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:25.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:25.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:54:25.502 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-24T21:54:25.568 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:25.625 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:25.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:25.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:54:25.730 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:25.786 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:25.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:25.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:54:25.891 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:25.947 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:25.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:25.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:54:26.053 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:26.102 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:26.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:26.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:54:26.161 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:54:26.207 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:26.258 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:26.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:26.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:54:26.363 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:26.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:26 smithi125 conmon[19936]: cluster 2021-03-24T21:54:24.703601+0000 mgr.y (mgr.14144) 305 : cluster [DBG] 2021-03-24T21:54:26.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:26 smithi125 conmon[19936]: pgmap v312: 148 pgs: 20 peering, 2 creating+peering, 2 unknown, 124 active+clean; 25 KiB data, 47 MiB used, 626 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-03-24T21:54:26.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:26 smithi125 conmon[19936]: cluster 2021-03-24T21:54:25.186314+0000 mon.a (mon.0) 628 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2021-03-24T21:54:26.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:26 smithi125 conmon[24927]: cluster 2021-03-24T21:54:24 2021-03-24T21:54:26.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:26 smithi125 conmon[24927]: .703601+0000 mgr.y (mgr.14144) 305 : cluster [DBG] pgmap v312: 148 pgs: 20 peering, 2 creating+peering, 2 unknown, 124 active+clean; 25 KiB data, 47 MiB used, 626 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-03-24T21:54:26.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:26 smithi125 conmon[24927]: cluster 2021-03-24T21:54: 2021-03-24T21:54:26.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:26 smithi125 conmon[24927]: 25.186314+0000 mon.a (mon.0) 628 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2021-03-24T21:54:26.421 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:26.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:26.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:54:26.526 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:26.529 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:54:26.577 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:26.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:26.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:54:26.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:26 smithi142 conmon[20125]: cluster 2021-03-24T21:54:24.703601+0000 mgr.y (mgr.14144) 305 : 2021-03-24T21:54:26.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:26 smithi142 conmon[20125]: cluster [DBG] pgmap v312: 148 pgs: 20 peering, 2 creating+peering, 2 unknown, 124 active+clean; 25 KiB data, 47 MiB used, 626 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-03-24T21:54:26.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:26 smithi142 conmon[20125]: cluster 2021-03-24T21:54:25.186314+0000 mon.a 2021-03-24T21:54:26.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:26 smithi142 conmon[20125]: (mon.0) 628 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2021-03-24T21:54:26.682 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:54:26.740 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:54:26.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:26.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:54:26.845 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:26.900 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:26.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:26.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:54:27.007 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:27.063 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:27.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:27.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:54:27.169 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:27.226 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:27.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:27.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:54:27.332 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:27.386 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:27.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:27.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:54:27.490 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:27.547 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:27.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:27.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:54:27.652 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:27.707 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:27.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:27.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:54:27.813 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:27.870 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:27.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:27.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:54:27.948 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-24T21:54:27.949 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T21:54:27.976 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:28.034 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:28.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:28.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:54:28.139 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:28.192 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:28.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:28.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:54:28.298 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:28.359 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:28.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:28.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:54:28.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:28 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:54:28.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:28 smithi125 conmon[24927]: 54:26.704561+0000 mgr.y (mgr.14144) 306 : cluster [DBG] pgmap v313: 148 pgs: 20 peering, 128 active+clean; 25 KiB data, 51 MiB used, 626 GiB / 626 GiB avail; 552 B/s rd, 0 op/s; 288 B/s, 2 objects/s recovering 2021-03-24T21:54:28.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:28 smithi125 conmon[19936]: cluster 2021-03-24T21:54:26. 2021-03-24T21:54:28.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:28 smithi125 conmon[19936]: 704561+0000 mgr.y (mgr.14144) 306 : cluster [DBG] pgmap v313: 148 pgs: 20 peering, 128 active+clean; 25 KiB data, 51 MiB used, 626 GiB / 626 GiB avail; 552 B/s rd, 0 op/s; 288 B/s, 2 objects/s recovering 2021-03-24T21:54:28.464 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:28.527 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:28.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:28.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:54:28.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:28 smithi142 conmon[20125]: cluster 2021-03-24T21:54:26.704561+0000 mgr.y (mgr.14144) 306 : cluster 2021-03-24T21:54:28.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:28 smithi142 conmon[20125]: [DBG] pgmap v313: 148 pgs: 20 peering, 128 active+clean; 25 KiB data, 51 MiB used, 626 GiB / 626 GiB avail; 552 B/s rd, 0 op/s; 288 B/s, 2 objects/s recovering 2021-03-24T21:54:28.633 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:28.681 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:28.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:28.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:54:28.786 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:28.842 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:28.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:28.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:54:28.949 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:29.005 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:29.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:29.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:54:29.111 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:29.168 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:29.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:29.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:54:29.274 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:29.333 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:29.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:29.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:54:29.438 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:29.492 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:29.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:29.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:54:29.557 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-03-24T21:54:29.557 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-24T21:54:29.557 INFO:tasks.thrashosds.thrasher:Removing osd 1, in_osds are: [0, 1, 2, 3, 4, 5, 7] 2021-03-24T21:54:29.558 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd out 1 2021-03-24T21:54:29.600 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:29.655 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:29.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:29.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:54:29.761 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:29.787 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.0 dump_historic_ops 2021-03-24T21:54:29.813 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:29.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:29.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:54:29.918 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:29.971 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:29.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:29.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:54:30.077 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:30.131 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:30.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:30.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:54:30.237 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:30.292 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:30.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:30.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:54:30.399 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:30.449 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:30.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:30.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:54:30.554 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:30 smithi125 conmon[19936]: cluster 2021-03-24T21:54:28.705485 2021-03-24T21:54:30.554 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:30 smithi125 conmon[19936]: +0000 mgr.y (mgr.14144) 307 : cluster [DBG] pgmap v314: 148 pgs: 13 peering, 135 active+clean; 25 KiB data, 52 MiB used, 626 GiB / 626 GiB avail; 312 B/s rd, 0 op/s; 1.2 KiB/s, 14 objects/s recovering 2021-03-24T21:54:30.554 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:30 smithi125 conmon[24927]: cluster 2021-03-24T21:54:28. 2021-03-24T21:54:30.555 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:30 smithi125 conmon[24927]: 705485+0000 mgr.y (mgr.14144) 307 : cluster [DBG] pgmap v314: 148 pgs: 13 peering, 135 active+clean; 25 KiB data, 52 MiB used, 626 GiB / 626 GiB avail; 312 B/s rd, 0 op/s; 1.2 KiB/s, 14 objects/s recovering 2021-03-24T21:54:30.555 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:30.616 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:30.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:30 smithi142 conmon[20125]: cluster 2021-03-24T21:54:28.705485+0000 mgr.y (mgr.14144) 307 : cluster 2021-03-24T21:54:30.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:30 smithi142 conmon[20125]: [DBG] pgmap v314: 148 pgs: 13 peering, 135 active+clean; 25 KiB data, 52 MiB used, 626 GiB / 626 GiB avail; 312 B/s rd, 0 op/s; 1.2 KiB/s, 14 objects/s recovering 2021-03-24T21:54:30.617 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:54:30.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:30.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:54:30.695 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:54:30.722 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:30.773 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:30.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:30.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:54:30.878 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:30.929 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:30.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:30.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:54:31.036 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:31.042 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T21:54:31.063 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T21:54:31.085 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T21:54:31.095 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:31.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:31.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:54:31.108 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T21:54:31.136 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T21:54:31.160 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T21:54:31.195 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T21:54:31.201 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:31.236 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T21:54:31.237 INFO:teuthology.orchestra.run.smithi125.stderr:enabled application 'rados' on pool 'unique_pool_0' 2021-03-24T21:54:31.244 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-24T21:54:31.245 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-24T21:54:31.245 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-24T21:54:31.245 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-24T21:54:31.245 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-24T21:54:31.246 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-24T21:54:31.246 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-24T21:54:31.246 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-24T21:54:31.272 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:31.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:31.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:54:31.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:31 smithi142 conmon[20125]: cluster 2021-03-24T21:54:29 2021-03-24T21:54:31.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:31 smithi142 conmon[20125]: .993324+0000 osd.1 (osd.1) 31 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:54:31.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:31 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:54:31.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:31 smithi142 conmon[20125]: 21:54:29.993524+0000 osd.1 (osd.1) 32 : cluster [DBG] 2021-03-24T21:54:31.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:31 smithi142 conmon[20125]: purged_snaps scrub ok 2021-03-24T21:54:31.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:31 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:54:31.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:31 smithi142 conmon[20125]: 21:54:30.224658+0000 2021-03-24T21:54:31.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:31 smithi142 conmon[20125]: mon.a (mon.0) 629 : cluster 2021-03-24T21:54:31.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:31 smithi142 conmon[20125]: [DBG] osdmap e103: 8 total, 8 up, 7 in 2021-03-24T21:54:31.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:31 smithi142 conmon[20125]: audit 2021-03 2021-03-24T21:54:31.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:31 smithi142 conmon[20125]: -24T21:54:30.602330 2021-03-24T21:54:31.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:31 smithi142 conmon[20125]: +0000 mon.a (mon.0) 2021-03-24T21:54:31.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:31 smithi142 conmon[20125]: 630 : audit [INF] from='client.? v1:172.21.15.125:0/2691093462' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2021-03-24T21:54:31.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:31 smithi142 conmon[20125]: audit 2021-03-24T21:54:31.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:31 smithi142 conmon[20125]: 2021-03-24T21:54 2021-03-24T21:54:31.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:31 smithi142 conmon[20125]: :31.018216+0000 mon.c (mon 2021-03-24T21:54:31.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:31 smithi142 conmon[20125]: .2) 23 : audit [DBG] from='client.? v1:172.21.15.125:0/1094296628' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:54:31.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:31 smithi142 conmon[20125]: 2021-03-24T21:54:31.376 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:54:31.430 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:54:31.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:31.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:54:31.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:31 smithi125 conmon[19936]: cluster 2021-03-24T21:54:31.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:31 smithi125 conmon[19936]: 2021-03-24T21:54:29.993324+0000 osd.1 (osd.1) 31 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:54:31.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:31 smithi125 conmon[19936]: cluster 2021-03-24T21:54:29.993524+0000 osd.1 (osd.1) 32 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:54:31.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:31 smithi125 conmon[19936]: cluster 2021-03-24T21:54:30.224658+0000 mon.a (mon.0) 629 : cluster [DBG] osdmap e103: 8 total, 8 up, 7 in 2021-03-24T21:54:31.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:31 smithi125 conmon[19936]: audit 2021-03-24T21:54:30.602330+0000 mon.a (mon.0 2021-03-24T21:54:31.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:31 smithi125 conmon[19936]: ) 630 : audit [INF] from='client.? v1:172.21.15.125:0/2691093462' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2021-03-24T21:54:31.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:31 smithi125 conmon[19936]: audit 2021-03-24T21:54 2021-03-24T21:54:31.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:31 smithi125 conmon[19936]: :31.018216+0000 mon.c (mon.2) 23 : audit [DBG] from='client.? v1:172.21.15.125:0/1094296628' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:54:31.486 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:31 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:54:31.486 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:31 smithi125 conmon[24927]: 21:54:29.993324+0000 osd.1 (osd.1) 31 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:54:31.486 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:31 smithi125 conmon[24927]: cluster 2021-03-24T21:54:31.486 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:31 smithi125 conmon[24927]: 2021-03-24T21:54:29.993524+0000 osd.1 (osd. 2021-03-24T21:54:31.487 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:31 smithi125 conmon[24927]: 1) 32 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:54:31.487 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:31 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:54:31.487 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:31 smithi125 conmon[24927]: 03-24T21:54:30.224658+0000 2021-03-24T21:54:31.487 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:31 smithi125 conmon[24927]: mon.a (mon.0) 629 : cluster [DBG] 2021-03-24T21:54:31.488 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:31 smithi125 conmon[24927]: osdmap e103: 8 total, 8 up, 7 in 2021-03-24T21:54:31.488 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:31 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T21:54:31.488 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:31 smithi125 conmon[24927]: 24T21:54:30.602330+0000 2021-03-24T21:54:31.488 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:31 smithi125 conmon[24927]: mon.a (mon.0) 630 : audit [INF] 2021-03-24T21:54:31.488 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:31 smithi125 conmon[24927]: from='client.? v1:172.21.15.125:0/2691093462' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2021-03-24T21:54:31.489 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:31 smithi125 conmon[24927]: audit 2021-03 2021-03-24T21:54:31.489 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:31 smithi125 conmon[24927]: -24T21:54:31.018216+0000 2021-03-24T21:54:31.489 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:31 smithi125 conmon[24927]: mon.c (mon.2) 23 : audit 2021-03-24T21:54:31.489 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:31 smithi125 conmon[24927]: [DBG] from='client.? v1:172.21.15.125:0/1094296628' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:54:31.537 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:31.594 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:31.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:31.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:54:31.700 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:31.757 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:31.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:31.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:54:31.863 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:31.921 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:31.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:31.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:54:32.028 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:32.083 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:32.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:32.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:54:32.190 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:32.251 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:32.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:32.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:54:32.355 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:32.414 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:32.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:32.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:54:32.462 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:32 smithi125 conmon[19936]: cluster 2021-03-24T21:54:30.706069+0000 mgr.y (mgr.14144) 308 2021-03-24T21:54:32.463 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:32 smithi125 conmon[19936]: : cluster [DBG] pgmap v316: 148 pgs: 148 active+clean; 25 KiB data, 52 MiB used, 626 GiB / 626 GiB avail; 950 B/s rd, 0 op/s; 1.3 KiB/s, 15 objects/s recovering 2021-03-24T21:54:32.463 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:32 smithi125 conmon[19936]: cluster 2021-03-24T21:54:31.216038+0000 mon.a (mon.0) 631 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2021-03-24T21:54:32.463 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:32 smithi125 conmon[19936]: audit 2021-03-24T21:54:31.229845+0000 mon.a (mon.0) 632 : audit [INF] from='client.? v1:172.21.15.125:0/2691093462' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2021-03-24T21:54:32.463 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:32 smithi125 conmon[19936]: cluster 2021-03-24T21:54: 2021-03-24T21:54:32.464 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:32 smithi125 conmon[19936]: 31.229927+0000 mon.a (mon.0) 633 : cluster [DBG] osdmap e104: 8 total, 8 up, 7 in 2021-03-24T21:54:32.464 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:32 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:54:32.464 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:32 smithi125 conmon[24927]: 54:30.706069+0000 mgr.y (mgr.14144) 308 : cluster [DBG] pgmap v316: 148 pgs: 148 active+clean; 25 KiB data, 52 MiB used, 626 GiB / 626 GiB avail; 950 B/s rd, 0 op/s; 1.3 KiB/s, 15 objects/s recovering 2021-03-24T21:54:32.464 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:32 smithi125 conmon[24927]: cluster 2021-03-24T21:54:32.465 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:32 smithi125 conmon[24927]: 2021-03-24T21:54:31.216038+0000 2021-03-24T21:54:32.465 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:32 smithi125 conmon[24927]: mon.a (mon.0) 631 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2021-03-24T21:54:32.465 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:32 smithi125 conmon[24927]: 2021-03-24T21:54:32.465 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:32 smithi125 conmon[24927]: audit 2021-03-24T21:54:32.466 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:32 smithi125 conmon[24927]: 2021-03 2021-03-24T21:54:32.466 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:32 smithi125 conmon[24927]: -24T21:54:31.229845+0000 mon.a ( 2021-03-24T21:54:32.466 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:32 smithi125 conmon[24927]: mon.0) 632 : audit [INF] from='client.? v1:172.21.15.125:0/2691093462' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2021-03-24T21:54:32.466 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:32 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:54:32.467 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:32 smithi125 conmon[24927]: 03-24T21:54:31.229927+0000 2021-03-24T21:54:32.467 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:32 smithi125 conmon[24927]: mon.a (mon.0) 633 : cluster 2021-03-24T21:54:32.467 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:32 smithi125 conmon[24927]: [DBG] osdmap e104: 8 total, 8 up, 7 in 2021-03-24T21:54:32.523 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:32.578 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:32.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:32.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:54:32.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:32 smithi142 conmon[20125]: cluster 2021-03-24T21:54:30.706069+0000 mgr.y (mgr.14144) 308 : cluster [DBG] pgmap v316: 148 pgs: 148 active+clean; 25 KiB data, 52 MiB used, 626 GiB / 626 GiB avail; 950 B/s rd, 0 op/s; 1.3 KiB/s, 15 objects/s recovering 2021-03-24T21:54:32.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:32 smithi142 conmon[20125]: cluster 2021-03-24T21:54:31.216038+0000 mon.a (mon.0) 631 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2021-03-24T21:54:32.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:32 smithi142 conmon[20125]: audit 2021-03-24 2021-03-24T21:54:32.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:32 smithi142 conmon[20125]: T21:54:31.229845+0000 mon.a (mon.0) 632 : audit [INF] from='client.? v1:172.21.15.125:0/2691093462' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2021-03-24T21:54:32.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:32 smithi142 conmon[20125]: cluster 2021-03-24T21:54 2021-03-24T21:54:32.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:32 smithi142 conmon[20125]: :31.229927+0000 mon.a (mon.0) 633 2021-03-24T21:54:32.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:32 smithi142 conmon[20125]: : cluster [DBG] osdmap e104: 8 total, 8 up, 7 in 2021-03-24T21:54:32.682 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:32.733 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:32.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:32.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:54:32.837 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:32.893 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:32.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:32.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:54:32.999 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:54:33.052 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:54:33.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:33.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:54:33.160 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:33.222 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:33.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:33.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:54:33.257 INFO:teuthology.orchestra.run.smithi125.stderr:nodeep-scrub is set 2021-03-24T21:54:33.328 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:54:33.341 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:33 smithi142 conmon[20125]: cluster 2021-03-24T21:54:31.711440+0000 2021-03-24T21:54:33.341 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:33 smithi142 conmon[20125]: osd.4 (osd.4) 34 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:54:33.342 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:33 smithi142 conmon[20125]: cluster 2021-03-24T21:54:31.711592+0000 osd.4 (osd.4) 35 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:54:33.342 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:33 smithi142 conmon[20125]: cluster 2021-03-24T21:54:32.237766+0000 mon.a ( 2021-03-24T21:54:33.342 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:33 smithi142 conmon[20125]: mon.0) 634 : cluster [DBG] osdmap e105: 8 total, 8 up, 7 in 2021-03-24T21:54:33.342 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:33 smithi142 conmon[20125]: audit 2021-03-24T21:54:32. 2021-03-24T21:54:33.343 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:33 smithi142 conmon[20125]: 735907+0000 mon.a (mon.0) 635 : audit [INF] from='client.? v1:172.21.15.125:0/3939161147' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:54:33.388 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:54:33.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:33.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:54:33.495 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:33.519 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:33 smithi125 conmon[19936]: cluster 2021-03-24T21:54:31. 2021-03-24T21:54:33.520 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:33 smithi125 conmon[19936]: 711440+0000 osd.4 (osd.4) 34 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:54:33.520 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:33 smithi125 conmon[19936]: cluster 2021-03-24T21:54 2021-03-24T21:54:33.521 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:33 smithi125 conmon[19936]: :31.711592+0000 osd.4 (osd.4) 35 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:54:33.521 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:33 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T21:54:33.521 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:33 smithi125 conmon[19936]: T21:54:32.237766+0000 mon.a (mon. 2021-03-24T21:54:33.522 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:33 smithi125 conmon[19936]: 0) 634 : cluster [DBG] osdmap e105: 8 total, 8 up, 7 in 2021-03-24T21:54:33.522 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:33 smithi125 conmon[19936]: audit 2021 2021-03-24T21:54:33.523 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:33 smithi125 conmon[19936]: -03-24T21:54:32.735907+0000 2021-03-24T21:54:33.523 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:33 smithi125 conmon[19936]: mon.a (mon.0) 635 : audit [INF] 2021-03-24T21:54:33.524 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:33 smithi125 conmon[19936]: from='client.? v1:172.21.15.125:0/3939161147' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:54:33.524 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:33 smithi125 conmon[24927]: cluster 2021-03-24T21:54:31.711440+0000 osd.4 (osd 2021-03-24T21:54:33.524 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:33 smithi125 conmon[24927]: .4) 34 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:54:33.525 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:33 smithi125 conmon[24927]: cluster 2021-03-24T21:54:31.711592+0000 osd.4 (osd.4) 35 : cluster [DBG] 2021-03-24T21:54:33.525 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:33 smithi125 conmon[24927]: purged_snaps scrub ok 2021-03-24T21:54:33.526 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:33 smithi125 conmon[24927]: cluster 2021-03-24T21:54:32.237766+0000 mon.a ( 2021-03-24T21:54:33.526 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:33 smithi125 conmon[24927]: mon.0) 634 : cluster [DBG] osdmap e105: 8 total, 8 up, 7 in 2021-03-24T21:54:33.526 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:33 smithi125 conmon[24927]: audit 2021 2021-03-24T21:54:33.526 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:33 smithi125 conmon[24927]: -03-24T21:54:32.735907+0000 mon.a (mon.0) 635 : audit [INF] 2021-03-24T21:54:33.527 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:33 smithi125 conmon[24927]: from='client.? v1:172.21.15.125:0/3939161147' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:54:33.559 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:33.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:33.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:54:33.665 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:33.721 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:33.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:33.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:54:33.827 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:33.889 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:33.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:33.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:54:33.996 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:34.062 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:34.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:34.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:54:34.168 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:34.225 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:34.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:34.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:54:34.331 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:34.385 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:34.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:34.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:54:34.459 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:34 smithi125 conmon[19936]: cluster 2021-03-24T21:54:32. 2021-03-24T21:54:34.459 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:34 smithi125 conmon[19936]: 706647+0000 mgr.y (mgr.14144) 309 : cluster [DBG] pgmap v319: 148 pgs: 148 active+clean; 25 KiB data, 52 MiB used, 626 GiB / 626 GiB avail; 682 B/s rd, 0 op/s; 1.3 KiB/s, 16 objects/s recovering 2021-03-24T21:54:34.459 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:34 smithi125 conmon[19936]: cluster 2021-03-24T21:54:33.235696+0000 mon.a (mon 2021-03-24T21:54:34.460 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:34 smithi125 conmon[19936]: .0) 636 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:54:34.460 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:34 smithi125 conmon[19936]: audit 2021-03- 2021-03-24T21:54:34.460 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:34 smithi125 conmon[19936]: 24T21:54:33.249854+0000 mon.a (mon.0) 637 : audit [INF] from='client.? v1:172.21.15.125:0/3939161147' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T21:54:34.460 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:34 smithi125 conmon[19936]: cluster 2021-03-24T21:54:33.249960 2021-03-24T21:54:34.460 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:34 smithi125 conmon[19936]: +0000 mon.a (mon.0) 638 : cluster [DBG] 2021-03-24T21:54:34.461 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:34 smithi125 conmon[19936]: osdmap e106: 8 total, 8 up, 7 in 2021-03-24T21:54:34.461 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:34 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:54:34.461 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:34 smithi125 conmon[24927]: 21:54:32.706647+0000 mgr.y (mgr.14144) 309 : cluster [DBG] pgmap v319: 148 pgs: 148 active+clean; 25 KiB data, 52 MiB used, 626 GiB / 626 GiB avail; 682 B/s rd, 0 op/s; 1.3 KiB/s, 16 objects/s recovering 2021-03-24T21:54:34.461 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:34 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:54:34.462 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:34 smithi125 conmon[24927]: -24T21:54:33.235696+0000 mon.a (mon 2021-03-24T21:54:34.462 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:34 smithi125 conmon[24927]: .0) 636 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:54:34.462 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:34 smithi125 conmon[24927]: audit 2021 2021-03-24T21:54:34.463 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:34 smithi125 conmon[24927]: -03-24T21: 2021-03-24T21:54:34.463 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:34 smithi125 conmon[24927]: 54:33.249854+0000 mon.a (mon.0) 2021-03-24T21:54:34.463 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:34 smithi125 conmon[24927]: 637 : audit [INF] from='client.? v1:172.21.15.125:0/3939161147' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T21:54:34.463 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:34 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:54:34.463 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:34 smithi125 conmon[24927]: 24T21:54:33.249960 2021-03-24T21:54:34.464 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:34 smithi125 conmon[24927]: +0000 mon.a (mon.0) 638 : cluster [DBG] 2021-03-24T21:54:34.464 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:34 smithi125 conmon[24927]: osdmap e106: 8 total, 8 up, 7 in 2021-03-24T21:54:34.493 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:34.553 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:34.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:34.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:54:34.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:34 smithi142 conmon[20125]: cluster 2021-03-24T21:54:32. 2021-03-24T21:54:34.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:34 smithi142 conmon[20125]: 706647+0000 mgr.y (mgr.14144) 309 : cluster [DBG] pgmap v319: 148 pgs: 148 active+clean; 25 KiB data, 52 MiB used, 626 GiB / 626 GiB avail; 682 B/s rd, 0 op/s; 1.3 KiB/s, 16 objects/s recovering 2021-03-24T21:54:34.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:34 smithi142 conmon[20125]: cluster 2021-03-24T21:54:33.235696+0000 mon.a 2021-03-24T21:54:34.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:34 smithi142 conmon[20125]: (mon.0) 636 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:54:34.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:34 smithi142 conmon[20125]: audit 2021-03-24T21:54: 2021-03-24T21:54:34.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:34 smithi142 conmon[20125]: 33.249854+0000 mon.a (mon.0) 637 : audit [INF] from='client.? v1:172.21.15.125:0/3939161147' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T21:54:34.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:34 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:54:34.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:34 smithi142 conmon[20125]: T21:54:33.249960+0000 mon.a (mon.0) 638 : cluster [DBG] osdmap e106: 8 total, 8 up, 7 in 2021-03-24T21:54:34.660 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:34.717 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:34.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:34.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:54:34.824 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:34.879 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:34.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:34.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:54:34.984 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:35.037 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:35.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:35.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:54:35.076 DEBUG:teuthology.orchestra.run.smithi125:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0 2021-03-24T21:54:35.094 INFO:tasks.rados.rados.0.smithi125.stdout:adding op weight read -> 100 2021-03-24T21:54:35.095 INFO:tasks.rados.rados.0.smithi125.stdout:adding op weight write -> 50 2021-03-24T21:54:35.095 INFO:tasks.rados.rados.0.smithi125.stdout:adding op weight delete -> 50 2021-03-24T21:54:35.095 INFO:tasks.rados.rados.0.smithi125.stdout:adding op weight snap_create -> 50 2021-03-24T21:54:35.095 INFO:tasks.rados.rados.0.smithi125.stdout:adding op weight snap_remove -> 50 2021-03-24T21:54:35.096 INFO:tasks.rados.rados.0.smithi125.stdout:adding op weight rollback -> 50 2021-03-24T21:54:35.096 INFO:tasks.rados.rados.0.smithi125.stdout:adding op weight copy_from -> 50 2021-03-24T21:54:35.096 INFO:tasks.rados.rados.0.smithi125.stdout:adding op weight write_excl -> 50 2021-03-24T21:54:35.096 INFO:tasks.rados.rados.0.smithi125.stdout:ceph version 16.1.0-1166-gd71e96d2 (d71e96d216190f28f757f409532464ab0eca4ea9) pacific (rc) 2021-03-24T21:54:35.097 INFO:tasks.rados.rados.0.smithi125.stdout:Configuration: 2021-03-24T21:54:35.097 INFO:tasks.rados.rados.0.smithi125.stdout: Number of operations: 4000 2021-03-24T21:54:35.097 INFO:tasks.rados.rados.0.smithi125.stdout: Number of objects: 50 2021-03-24T21:54:35.097 INFO:tasks.rados.rados.0.smithi125.stdout: Max in flight operations: 16 2021-03-24T21:54:35.097 INFO:tasks.rados.rados.0.smithi125.stdout: Object size (in bytes): 4000000 2021-03-24T21:54:35.098 INFO:tasks.rados.rados.0.smithi125.stdout: Write stride min: 400000 2021-03-24T21:54:35.098 INFO:tasks.rados.rados.0.smithi125.stdout: Write stride max: 800000 2021-03-24T21:54:35.143 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:35.199 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:35.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:35.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:54:35.305 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:35.363 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:35.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:35.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:54:35.471 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:35.524 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:35.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:35.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:54:35.631 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:35.688 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:35.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:35.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:54:35.793 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:35.843 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:35.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:35.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:54:35.948 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:36.029 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:36.033 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:35 smithi142 conmon[20125]: cluster 2021-03-24T21:54:34 2021-03-24T21:54:36.033 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:35 smithi142 conmon[20125]: .707859+0000 mgr.y (mgr.14144) 310 : cluster [DBG] pgmap v321: 148 pgs: 148 active+clean; 25 KiB data, 52 MiB used, 626 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-03-24T21:54:36.034 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:35 smithi142 conmon[20125]: audit 2021-03-24T21:54: 2021-03-24T21:54:36.034 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:35 smithi142 conmon[20125]: 34.749722+0000 mon.a (mon.0) 639 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:54:36.034 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:35 smithi142 conmon[20125]: audit 2021-03-24T21:54:35.244773+0000 mon.a (mon.0) 640 : audit [INF] from='client.? v1:172.21.15.125:0/2843332919' entity='client.0' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]: dispatch 2021-03-24T21:54:36.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:36.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:54:36.040 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-24T21:54:36.040 INFO:teuthology.orchestra.run.smithi125.stdout: "size": 20, 2021-03-24T21:54:36.041 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 600, 2021-03-24T21:54:36.041 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [ 2021-03-24T21:54:36.041 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.042 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.34352.0:2 3.12 3:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected e61)", 2021-03-24T21:54:36.042 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:51:45.369344+0000", 2021-03-24T21:54:36.042 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 170.668104132, 2021-03-24T21:54:36.042 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012509175000000001, 2021-03-24T21:54:36.043 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:54:36.043 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "reached pg", 2021-03-24T21:54:36.043 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:54:36.043 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.34352", 2021-03-24T21:54:36.044 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/3436348424", 2021-03-24T21:54:36.044 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 2 2021-03-24T21:54:36.044 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.044 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:54:36.045 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.045 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:54:36.045 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:45.369344+0000", 2021-03-24T21:54:36.045 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:54:36.045 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.046 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.046 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:54:36.046 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:45.369344+0000", 2021-03-24T21:54:36.046 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999971 2021-03-24T21:54:36.047 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.047 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.047 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:54:36.047 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:45.369341+0000", 2021-03-24T21:54:36.047 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0037999999999999e-05 2021-03-24T21:54:36.048 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.048 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.048 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:54:36.048 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:45.369351+0000", 2021-03-24T21:54:36.049 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7516e-05 2021-03-24T21:54:36.049 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.049 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.049 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:54:36.050 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:45.369368+0000", 2021-03-24T21:54:36.050 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.5650000000000007e-06 2021-03-24T21:54:36.050 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.051 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.051 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:54:36.051 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:45.369378+0000", 2021-03-24T21:54:36.051 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.7278000000000001e-05 2021-03-24T21:54:36.052 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.052 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.052 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:54:36.052 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:45.369455+0000", 2021-03-24T21:54:36.052 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.062e-05 2021-03-24T21:54:36.053 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.053 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.053 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T21:54:36.053 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:45.369466+0000", 2021-03-24T21:54:36.054 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012278427999999999 2021-03-24T21:54:36.054 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.054 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.054 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:54:36.055 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:45.381744+0000", 2021-03-24T21:54:36.055 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000108446 2021-03-24T21:54:36.055 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.055 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.055 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:54:36.056 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:45.381853+0000", 2021-03-24T21:54:36.056 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012509175000000001 2021-03-24T21:54:36.056 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:36.056 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:54:36.056 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:36.057 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.057 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.057 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.34352.0:89 4.b 4:d467b91b:::bucket.sync-target-hints.:head [call,read 0~0] snapc 0=[] ondisk+read+known_if_redirected e63)", 2021-03-24T21:54:36.057 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:51:47.404436+0000", 2021-03-24T21:54:36.058 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 168.63301163099999, 2021-03-24T21:54:36.058 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0046623719999999997, 2021-03-24T21:54:36.058 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:54:36.058 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "reached pg", 2021-03-24T21:54:36.059 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:54:36.059 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.34352", 2021-03-24T21:54:36.059 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/3436348424", 2021-03-24T21:54:36.059 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 89 2021-03-24T21:54:36.059 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.060 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:54:36.060 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.060 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:54:36.060 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:47.404436+0000", 2021-03-24T21:54:36.061 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:54:36.061 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.061 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.061 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:54:36.061 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:47.404436+0000", 2021-03-24T21:54:36.062 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999976 2021-03-24T21:54:36.062 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.062 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.062 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:54:36.063 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:47.404434+0000", 2021-03-24T21:54:36.063 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.113e-05 2021-03-24T21:54:36.063 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.063 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.064 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:54:36.064 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:47.404445+0000", 2021-03-24T21:54:36.064 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3396e-05 2021-03-24T21:54:36.064 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.064 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.065 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:54:36.065 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:47.404458+0000", 2021-03-24T21:54:36.065 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.5750000000000001e-06 2021-03-24T21:54:36.065 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.065 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.066 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:54:36.066 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:47.404467+0000", 2021-03-24T21:54:36.066 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.1062000000000005e-05 2021-03-24T21:54:36.066 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.067 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.067 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:54:36.067 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:47.404538+0000", 2021-03-24T21:54:36.067 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.7660000000000004e-06 2021-03-24T21:54:36.067 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.068 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.068 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T21:54:36.068 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:47.404546+0000", 2021-03-24T21:54:36.068 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0043799640000000001 2021-03-24T21:54:36.068 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.069 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.069 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:54:36.069 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:47.408926+0000", 2021-03-24T21:54:36.069 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00017204000000000001 2021-03-24T21:54:36.070 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.070 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.070 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:54:36.070 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:47.409098+0000", 2021-03-24T21:54:36.070 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0046623719999999997 2021-03-24T21:54:36.071 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:36.071 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:54:36.071 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:36.071 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.072 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.072 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.34352.0:101 5.3 5:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected e65)", 2021-03-24T21:54:36.072 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:51:49.437022+0000", 2021-03-24T21:54:36.072 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 166.60042584999999, 2021-03-24T21:54:36.072 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0040905560000000004, 2021-03-24T21:54:36.073 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:54:36.073 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:54:36.073 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:54:36.073 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.34352", 2021-03-24T21:54:36.073 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/3436348424", 2021-03-24T21:54:36.074 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 101 2021-03-24T21:54:36.074 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.074 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:54:36.074 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.075 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:54:36.075 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:49.437022+0000", 2021-03-24T21:54:36.075 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:54:36.075 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.075 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.076 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:54:36.076 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:49.437022+0000", 2021-03-24T21:54:36.076 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999976 2021-03-24T21:54:36.076 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.077 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.077 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:54:36.077 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:49.437019+0000", 2021-03-24T21:54:36.077 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5597999999999999e-05 2021-03-24T21:54:36.077 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.078 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.078 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:54:36.078 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:49.437035+0000", 2021-03-24T21:54:36.078 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5478999999999999e-05 2021-03-24T21:54:36.079 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.079 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.079 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:54:36.079 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:49.437050+0000", 2021-03-24T21:54:36.079 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9340000000000003e-06 2021-03-24T21:54:36.080 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.080 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.080 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:54:36.080 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:49.437056+0000", 2021-03-24T21:54:36.081 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5031000000000002e-05 2021-03-24T21:54:36.081 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.081 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.081 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:54:36.081 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:49.437111+0000", 2021-03-24T21:54:36.082 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.0860999999999996e-05 2021-03-24T21:54:36.082 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.082 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.082 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:54:36.082 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:49.437202+0000", 2021-03-24T21:54:36.083 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00013260600000000001 2021-03-24T21:54:36.083 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.083 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.083 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 5,6", 2021-03-24T21:54:36.084 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:49.437335+0000", 2021-03-24T21:54:36.084 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00027442500000000002 2021-03-24T21:54:36.084 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.084 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.084 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:54:36.085 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:49.437609+0000", 2021-03-24T21:54:36.085 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00046454500000000002 2021-03-24T21:54:36.085 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.085 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.086 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:54:36.086 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:49.438074+0000", 2021-03-24T21:54:36.086 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0029229949999999998 2021-03-24T21:54:36.086 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.086 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.087 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:54:36.087 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:49.440997+0000", 2021-03-24T21:54:36.087 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.4333999999999999e-05 2021-03-24T21:54:36.087 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.087 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.088 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:54:36.088 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:49.441051+0000", 2021-03-24T21:54:36.088 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.1198000000000006e-05 2021-03-24T21:54:36.088 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.089 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.089 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:54:36.089 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:49.441112+0000", 2021-03-24T21:54:36.089 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0040905560000000004 2021-03-24T21:54:36.089 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:36.090 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:54:36.090 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:36.090 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.090 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.090 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.34352.0:391 4.3 4:c35aa124:::data_log.14:head [call fifo.create_meta in=15b] snapc 0=[] ondisk+write+known_if_redirected e65)", 2021-03-24T21:54:36.091 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:51:49.666923+0000", 2021-03-24T21:54:36.091 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 166.370524988, 2021-03-24T21:54:36.091 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0028829010000000002, 2021-03-24T21:54:36.091 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:54:36.092 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:54:36.092 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:54:36.092 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.34352", 2021-03-24T21:54:36.092 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/3436348424", 2021-03-24T21:54:36.092 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 391 2021-03-24T21:54:36.093 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.093 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:54:36.093 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.093 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:54:36.093 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:49.666923+0000", 2021-03-24T21:54:36.094 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:54:36.094 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.094 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.094 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:54:36.095 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:49.666923+0000", 2021-03-24T21:54:36.095 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999962 2021-03-24T21:54:36.095 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.095 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.095 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:54:36.096 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:49.666919+0000", 2021-03-24T21:54:36.096 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5461999999999999e-05 2021-03-24T21:54:36.096 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.096 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.096 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:54:36.097 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:49.666935+0000", 2021-03-24T21:54:36.097 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.4309e-05 2021-03-24T21:54:36.097 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.098 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.098 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:54:36.098 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:49.666969+0000", 2021-03-24T21:54:36.098 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3944999999999999e-05 2021-03-24T21:54:36.098 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.099 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.099 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:54:36.099 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:49.666983+0000", 2021-03-24T21:54:36.099 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.5465999999999997e-05 2021-03-24T21:54:36.099 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.100 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.100 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:54:36.100 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:49.667078+0000", 2021-03-24T21:54:36.100 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00020629099999999999 2021-03-24T21:54:36.101 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.101 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.101 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:54:36.101 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:49.667285+0000", 2021-03-24T21:54:36.101 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00035986699999999999 2021-03-24T21:54:36.102 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.102 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.102 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 5,7", 2021-03-24T21:54:36.102 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:49.667644+0000", 2021-03-24T21:54:36.102 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00075813699999999998 2021-03-24T21:54:36.103 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.103 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.103 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:54:36.103 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:49.668403+0000", 2021-03-24T21:54:36.104 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00071055999999999995 2021-03-24T21:54:36.104 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.104 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.104 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:54:36.104 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:49.669113+0000", 2021-03-24T21:54:36.105 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00060654100000000005 2021-03-24T21:54:36.105 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.105 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.105 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:54:36.106 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:49.669720+0000", 2021-03-24T21:54:36.106 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6458999999999997e-05 2021-03-24T21:54:36.106 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.106 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.106 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:54:36.107 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:49.669766+0000", 2021-03-24T21:54:36.107 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9507e-05 2021-03-24T21:54:36.107 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.107 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.108 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:54:36.108 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:49.669806+0000", 2021-03-24T21:54:36.108 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0028829010000000002 2021-03-24T21:54:36.108 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:36.109 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:54:36.109 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:36.109 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.109 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.109 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.34352.0:490 4.b 4:d415e779:::data_log.63:head [call fifo.get_meta in=12b] snapc 0=[] ondisk+read+known_if_redirected e65)", 2021-03-24T21:54:36.110 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:51:49.906303+0000", 2021-03-24T21:54:36.110 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 166.13114495600001, 2021-03-24T21:54:36.110 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0035619250000000001, 2021-03-24T21:54:36.110 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:54:36.111 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T21:54:36.111 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:54:36.111 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.34352", 2021-03-24T21:54:36.111 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/3436348424", 2021-03-24T21:54:36.111 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 490 2021-03-24T21:54:36.112 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.112 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:54:36.112 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.112 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:54:36.112 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:49.906303+0000", 2021-03-24T21:54:36.113 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:54:36.113 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.113 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.113 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:54:36.114 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:49.906303+0000", 2021-03-24T21:54:36.114 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999976 2021-03-24T21:54:36.114 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.114 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.115 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:54:36.115 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:49.906300+0000", 2021-03-24T21:54:36.115 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5770999999999999e-05 2021-03-24T21:54:36.115 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.115 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.116 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:54:36.116 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:49.906316+0000", 2021-03-24T21:54:36.116 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.3130999999999999e-05 2021-03-24T21:54:36.116 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.116 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.117 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:54:36.117 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:49.906339+0000", 2021-03-24T21:54:36.117 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0399e-05 2021-03-24T21:54:36.117 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.118 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.118 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:54:36.118 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:49.906349+0000", 2021-03-24T21:54:36.118 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.7670000000000006e-05 2021-03-24T21:54:36.118 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.119 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.119 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:54:36.119 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:49.906437+0000", 2021-03-24T21:54:36.119 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00016473200000000001 2021-03-24T21:54:36.119 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.120 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.120 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:54:36.120 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:49.906602+0000", 2021-03-24T21:54:36.120 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.003262792 2021-03-24T21:54:36.121 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.121 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.121 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:54:36.121 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:49.909865+0000", 2021-03-24T21:54:36.121 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0035619250000000001 2021-03-24T21:54:36.122 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:36.122 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:54:36.122 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:36.122 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.123 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.123 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.34352.0:535 4.b 4:d1061236:::data_log.86:head [call fifo.create_meta in=15b] snapc 0=[] ondisk+write+known_if_redirected e65)", 2021-03-24T21:54:36.123 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:51:50.033218+0000", 2021-03-24T21:54:36.123 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 166.00422952400001, 2021-03-24T21:54:36.124 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018885586999999999, 2021-03-24T21:54:36.124 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:54:36.124 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:54:36.124 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:54:36.124 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.34352", 2021-03-24T21:54:36.125 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/3436348424", 2021-03-24T21:54:36.125 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 535 2021-03-24T21:54:36.125 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.125 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:54:36.125 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.126 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:54:36.126 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.033218+0000", 2021-03-24T21:54:36.126 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:54:36.126 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.127 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.127 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:54:36.127 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.033218+0000", 2021-03-24T21:54:36.127 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999962 2021-03-24T21:54:36.127 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.128 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.128 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:54:36.128 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.033214+0000", 2021-03-24T21:54:36.128 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.8168e-05 2021-03-24T21:54:36.128 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.129 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.129 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:54:36.130 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.033232+0000", 2021-03-24T21:54:36.130 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.2484999999999998e-05 2021-03-24T21:54:36.130 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.130 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.130 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:54:36.131 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.033265+0000", 2021-03-24T21:54:36.131 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1425e-05 2021-03-24T21:54:36.131 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.131 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.132 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:54:36.132 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.033276+0000", 2021-03-24T21:54:36.132 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010019800000000001 2021-03-24T21:54:36.132 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.132 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.133 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:54:36.133 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.033376+0000", 2021-03-24T21:54:36.133 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00025655600000000001 2021-03-24T21:54:36.133 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.133 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.134 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:54:36.134 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.033633+0000", 2021-03-24T21:54:36.134 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00041837800000000001 2021-03-24T21:54:36.134 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.135 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.135 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,4", 2021-03-24T21:54:36.135 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.034051+0000", 2021-03-24T21:54:36.135 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0074551269999999998 2021-03-24T21:54:36.135 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.136 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.136 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:54:36.136 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.041507+0000", 2021-03-24T21:54:36.136 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.010253246000000001 2021-03-24T21:54:36.136 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.137 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.137 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:54:36.137 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.051760+0000", 2021-03-24T21:54:36.137 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00022099399999999999 2021-03-24T21:54:36.138 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.138 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.138 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:54:36.138 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.051981+0000", 2021-03-24T21:54:36.138 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.4010000000000005e-05 2021-03-24T21:54:36.139 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.139 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.139 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:54:36.139 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.052055+0000", 2021-03-24T21:54:36.139 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.8992000000000001e-05 2021-03-24T21:54:36.140 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.140 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.140 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:54:36.140 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.052104+0000", 2021-03-24T21:54:36.141 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018885586999999999 2021-03-24T21:54:36.141 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:36.141 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:54:36.141 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:36.141 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.142 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.142 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.34352.0:536 4.b 4:d1061236:::data_log.86:head [call fifo.get_meta in=12b] snapc 0=[] ondisk+read+known_if_redirected e65)", 2021-03-24T21:54:36.142 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:51:50.052252+0000", 2021-03-24T21:54:36.142 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 165.98519535, 2021-03-24T21:54:36.142 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002980262, 2021-03-24T21:54:36.143 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:54:36.143 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T21:54:36.143 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:54:36.143 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.34352", 2021-03-24T21:54:36.144 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/3436348424", 2021-03-24T21:54:36.144 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 536 2021-03-24T21:54:36.144 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.144 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:54:36.144 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.145 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:54:36.145 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.052252+0000", 2021-03-24T21:54:36.145 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:54:36.145 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.145 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.146 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:54:36.146 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.052252+0000", 2021-03-24T21:54:36.146 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T21:54:36.146 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.147 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.147 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:54:36.147 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.052250+0000", 2021-03-24T21:54:36.147 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.1480000000000006e-06 2021-03-24T21:54:36.147 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.148 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.148 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:54:36.148 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.052259+0000", 2021-03-24T21:54:36.148 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.6758000000000001e-05 2021-03-24T21:54:36.148 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.149 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.149 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:54:36.149 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.052275+0000", 2021-03-24T21:54:36.149 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.6799999999999993e-06 2021-03-24T21:54:36.150 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.150 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.150 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:54:36.150 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.052283+0000", 2021-03-24T21:54:36.150 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.5197000000000005e-05 2021-03-24T21:54:36.151 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.151 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.151 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:54:36.152 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.052358+0000", 2021-03-24T21:54:36.152 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000113565 2021-03-24T21:54:36.152 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.152 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.153 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:54:36.153 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.052472+0000", 2021-03-24T21:54:36.153 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0027607840000000001 2021-03-24T21:54:36.153 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.153 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.154 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:54:36.154 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.055233+0000", 2021-03-24T21:54:36.154 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002980262 2021-03-24T21:54:36.154 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:36.155 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:54:36.155 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:36.155 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.155 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.156 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.34352.0:541 4.16 4:6b9e929a:::data_log.89:head [call fifo.create_meta in=15b] snapc 0=[] ondisk+write+known_if_redirected e65)", 2021-03-24T21:54:36.156 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:51:50.073601+0000", 2021-03-24T21:54:36.156 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 165.96384698899999, 2021-03-24T21:54:36.156 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018947499, 2021-03-24T21:54:36.156 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:54:36.157 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:54:36.157 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:54:36.157 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.34352", 2021-03-24T21:54:36.157 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/3436348424", 2021-03-24T21:54:36.158 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 541 2021-03-24T21:54:36.158 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.158 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:54:36.158 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.158 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:54:36.159 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.073601+0000", 2021-03-24T21:54:36.159 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:54:36.159 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.159 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.160 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:54:36.160 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.073601+0000", 2021-03-24T21:54:36.160 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999952 2021-03-24T21:54:36.160 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.161 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.161 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:54:36.161 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.073596+0000", 2021-03-24T21:54:36.161 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9911999999999998e-05 2021-03-24T21:54:36.162 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.162 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.162 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:54:36.163 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.073616+0000", 2021-03-24T21:54:36.163 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.9751999999999999e-05 2021-03-24T21:54:36.163 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.163 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.163 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:54:36.164 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.073646+0000", 2021-03-24T21:54:36.164 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2867999999999999e-05 2021-03-24T21:54:36.164 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.164 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.164 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:54:36.165 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.073659+0000", 2021-03-24T21:54:36.165 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.5937000000000002e-05 2021-03-24T21:54:36.165 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.165 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.166 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:54:36.166 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.073735+0000", 2021-03-24T21:54:36.166 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00017380799999999999 2021-03-24T21:54:36.166 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.167 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.167 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:54:36.167 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.073908+0000", 2021-03-24T21:54:36.167 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000260556 2021-03-24T21:54:36.167 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.168 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.168 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 3,7", 2021-03-24T21:54:36.168 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.074169+0000", 2021-03-24T21:54:36.168 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0024465329999999999 2021-03-24T21:54:36.169 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.169 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.169 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:54:36.169 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.076615+0000", 2021-03-24T21:54:36.169 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.015545275000000001 2021-03-24T21:54:36.170 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.170 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.170 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:54:36.170 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.092161+0000", 2021-03-24T21:54:36.171 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00026825299999999998 2021-03-24T21:54:36.171 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.171 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.171 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:54:36.171 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.092429+0000", 2021-03-24T21:54:36.172 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.1335999999999997e-05 2021-03-24T21:54:36.172 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.172 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.172 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:54:36.173 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.092490+0000", 2021-03-24T21:54:36.173 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.7896999999999998e-05 2021-03-24T21:54:36.173 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.173 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.173 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:54:36.174 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.092548+0000", 2021-03-24T21:54:36.174 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018947499 2021-03-24T21:54:36.174 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:36.174 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:54:36.175 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:36.175 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.175 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.175 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.34352.0:562 4.6 4:620d8b71:::data_log.99:head [call fifo.get_meta in=12b] snapc 0=[] ondisk+read+known_if_redirected e65)", 2021-03-24T21:54:36.175 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:51:50.137609+0000", 2021-03-24T21:54:36.176 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 165.89983876299999, 2021-03-24T21:54:36.176 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0031071390000000001, 2021-03-24T21:54:36.176 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:54:36.176 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T21:54:36.176 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:54:36.177 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.34352", 2021-03-24T21:54:36.177 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/3436348424", 2021-03-24T21:54:36.177 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 562 2021-03-24T21:54:36.177 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.178 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:54:36.178 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.178 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:54:36.178 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.137609+0000", 2021-03-24T21:54:36.179 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:54:36.179 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.179 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.179 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:54:36.179 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.137609+0000", 2021-03-24T21:54:36.180 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999976 2021-03-24T21:54:36.180 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.180 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.180 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:54:36.181 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.137606+0000", 2021-03-24T21:54:36.181 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2146e-05 2021-03-24T21:54:36.181 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.182 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.182 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:54:36.182 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.137619+0000", 2021-03-24T21:54:36.183 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.4513000000000001e-05 2021-03-24T21:54:36.184 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.184 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.184 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:54:36.184 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.137643+0000", 2021-03-24T21:54:36.184 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.6271999999999999e-05 2021-03-24T21:54:36.185 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.185 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.186 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:54:36.186 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.137659+0000", 2021-03-24T21:54:36.186 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.6815000000000001e-05 2021-03-24T21:54:36.186 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.187 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.187 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:54:36.188 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.137746+0000", 2021-03-24T21:54:36.188 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000116087 2021-03-24T21:54:36.188 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.188 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.188 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:54:36.189 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.137862+0000", 2021-03-24T21:54:36.189 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0028537789999999999 2021-03-24T21:54:36.189 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.189 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.190 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:54:36.190 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.140716+0000", 2021-03-24T21:54:36.190 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0031071390000000001 2021-03-24T21:54:36.190 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:36.190 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:54:36.191 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:36.191 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.191 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.191 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.34352.0:577 4.6 4:63b3b181:::data_log.107:head [call fifo.create_meta in=15b] snapc 0=[] ondisk+write+known_if_redirected e65)", 2021-03-24T21:54:36.191 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:51:50.177185+0000", 2021-03-24T21:54:36.192 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 165.86026303599999, 2021-03-24T21:54:36.192 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.003029047, 2021-03-24T21:54:36.192 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:54:36.192 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:54:36.193 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:54:36.193 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.34352", 2021-03-24T21:54:36.193 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/3436348424", 2021-03-24T21:54:36.193 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 577 2021-03-24T21:54:36.193 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.194 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:54:36.194 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.194 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:54:36.194 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.177185+0000", 2021-03-24T21:54:36.195 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:54:36.195 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.195 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.195 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:54:36.196 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.177185+0000", 2021-03-24T21:54:36.196 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999967 2021-03-24T21:54:36.196 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.196 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.196 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:54:36.197 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.177181+0000", 2021-03-24T21:54:36.197 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.4841000000000001e-05 2021-03-24T21:54:36.197 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.197 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.197 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:54:36.198 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.177196+0000", 2021-03-24T21:54:36.198 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.2357000000000002e-05 2021-03-24T21:54:36.198 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.199 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.199 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:54:36.199 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.177228+0000", 2021-03-24T21:54:36.199 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2995e-05 2021-03-24T21:54:36.199 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.200 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.200 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:54:36.200 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.177241+0000", 2021-03-24T21:54:36.200 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000118601 2021-03-24T21:54:36.201 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.201 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.201 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:54:36.201 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.177360+0000", 2021-03-24T21:54:36.201 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000353519 2021-03-24T21:54:36.202 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.202 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.202 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:54:36.202 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.177714+0000", 2021-03-24T21:54:36.203 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00065359299999999995 2021-03-24T21:54:36.203 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.203 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.203 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,2", 2021-03-24T21:54:36.203 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.178367+0000", 2021-03-24T21:54:36.204 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00083148499999999997 2021-03-24T21:54:36.204 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.204 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.204 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:54:36.205 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.179199+0000", 2021-03-24T21:54:36.205 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00069680600000000001 2021-03-24T21:54:36.205 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.205 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.205 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:54:36.206 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.179895+0000", 2021-03-24T21:54:36.206 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00015430700000000001 2021-03-24T21:54:36.206 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.206 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.207 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:54:36.207 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.180050+0000", 2021-03-24T21:54:36.207 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.5409999999999996e-05 2021-03-24T21:54:36.207 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.208 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.208 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:54:36.208 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.180145+0000", 2021-03-24T21:54:36.209 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.8565000000000005e-05 2021-03-24T21:54:36.209 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.209 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.209 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:54:36.210 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.180214+0000", 2021-03-24T21:54:36.210 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.003029047 2021-03-24T21:54:36.210 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:36.210 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:54:36.211 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:36.211 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.211 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.211 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.24376.0:2 4.19 e68/62 4:9a6dd8c0:gc::gc.0:head v 68'6, mlcod=65'4)", 2021-03-24T21:54:36.211 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:51:53.767556+0000", 2021-03-24T21:54:36.212 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 162.269891683, 2021-03-24T21:54:36.212 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0049905790000000002, 2021-03-24T21:54:36.212 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:54:36.212 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:54:36.213 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:54:36.213 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.213 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:54:36.213 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.767556+0000", 2021-03-24T21:54:36.213 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:54:36.214 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.214 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.214 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:54:36.214 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.767556+0000", 2021-03-24T21:54:36.215 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999967 2021-03-24T21:54:36.215 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.215 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.215 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:54:36.215 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.767553+0000", 2021-03-24T21:54:36.216 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.1046000000000001e-05 2021-03-24T21:54:36.216 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.216 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.216 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:54:36.217 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.767574+0000", 2021-03-24T21:54:36.217 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.1986999999999997e-05 2021-03-24T21:54:36.217 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.217 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.218 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:54:36.218 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.767606+0000", 2021-03-24T21:54:36.218 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3263e-05 2021-03-24T21:54:36.218 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.218 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.219 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:54:36.219 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.767619+0000", 2021-03-24T21:54:36.219 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.8910000000000001e-05 2021-03-24T21:54:36.219 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.220 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.220 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:54:36.220 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.767708+0000", 2021-03-24T21:54:36.220 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.2972999999999997e-05 2021-03-24T21:54:36.220 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.221 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.221 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:54:36.221 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.767751+0000", 2021-03-24T21:54:36.221 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0047271819999999999 2021-03-24T21:54:36.222 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.222 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.222 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:54:36.222 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.772478+0000", 2021-03-24T21:54:36.222 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.8670000000000005e-05 2021-03-24T21:54:36.223 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.223 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.223 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:54:36.223 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.772547+0000", 2021-03-24T21:54:36.224 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0049905790000000002 2021-03-24T21:54:36.224 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:36.224 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:54:36.224 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:36.224 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.225 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.225 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24376.0:4 4.b 4:d242335b:gc::gc.2:head [create,call version.check_conds in=18b,call rgw_gc.rgw_gc_queue_init in=23b,call version.set in=10b] snapc 0=[] ondisk+write+known_if_redirected e68)", 2021-03-24T21:54:36.225 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:51:53.776900+0000", 2021-03-24T21:54:36.225 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 162.260547995, 2021-03-24T21:54:36.226 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.004889727, 2021-03-24T21:54:36.226 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:54:36.226 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:54:36.226 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:54:36.226 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24376", 2021-03-24T21:54:36.227 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/3051439499", 2021-03-24T21:54:36.227 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4 2021-03-24T21:54:36.227 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.227 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:54:36.228 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.228 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:54:36.228 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.776900+0000", 2021-03-24T21:54:36.228 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:54:36.228 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.229 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.229 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:54:36.229 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.776900+0000", 2021-03-24T21:54:36.229 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T21:54:36.229 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.230 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.230 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:54:36.230 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.776898+0000", 2021-03-24T21:54:36.230 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.9719999999999998e-06 2021-03-24T21:54:36.231 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.231 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.231 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:54:36.231 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.776907+0000", 2021-03-24T21:54:36.231 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2831e-05 2021-03-24T21:54:36.232 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.232 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.232 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:54:36.232 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.776919+0000", 2021-03-24T21:54:36.232 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.1859999999999997e-06 2021-03-24T21:54:36.233 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.233 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.233 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:54:36.233 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.776926+0000", 2021-03-24T21:54:36.234 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010092199999999999 2021-03-24T21:54:36.234 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.234 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.234 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:54:36.234 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.777027+0000", 2021-03-24T21:54:36.235 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000171881 2021-03-24T21:54:36.235 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.235 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.235 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:54:36.235 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.777198+0000", 2021-03-24T21:54:36.236 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000290864 2021-03-24T21:54:36.236 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.236 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.236 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,4", 2021-03-24T21:54:36.237 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.777489+0000", 2021-03-24T21:54:36.237 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00050524299999999995 2021-03-24T21:54:36.237 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.237 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.237 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:54:36.238 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.777995+0000", 2021-03-24T21:54:36.238 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00034218399999999999 2021-03-24T21:54:36.238 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.238 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.238 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:54:36.239 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.778337+0000", 2021-03-24T21:54:36.239 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.003336866 2021-03-24T21:54:36.239 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.239 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.240 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:54:36.240 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.781674+0000", 2021-03-24T21:54:36.240 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.8089000000000005e-05 2021-03-24T21:54:36.240 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.240 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.241 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:54:36.241 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.781742+0000", 2021-03-24T21:54:36.241 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.7744999999999998e-05 2021-03-24T21:54:36.241 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.241 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.242 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:54:36.242 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.781789+0000", 2021-03-24T21:54:36.242 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.004889727 2021-03-24T21:54:36.242 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:36.243 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:54:36.243 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:36.243 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.243 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.243 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(osd.1.0:89 4.7 e89/62 4:e078a471:::data_log.102:head v 89'16, mlcod=69'14)", 2021-03-24T21:54:36.244 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:53:19.709853+0000", 2021-03-24T21:54:36.244 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 76.327594970999996, 2021-03-24T21:54:36.244 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0032427659999999998, 2021-03-24T21:54:36.244 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:54:36.244 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:54:36.245 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:54:36.245 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.245 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:54:36.245 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:19.709853+0000", 2021-03-24T21:54:36.246 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:54:36.246 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.246 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.246 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:54:36.246 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:19.709853+0000", 2021-03-24T21:54:36.247 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999971 2021-03-24T21:54:36.247 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.247 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.247 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:54:36.247 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:19.709850+0000", 2021-03-24T21:54:36.248 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0669000000000001e-05 2021-03-24T21:54:36.248 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.248 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.248 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:54:36.249 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:19.709871+0000", 2021-03-24T21:54:36.249 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.5168e-05 2021-03-24T21:54:36.249 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.249 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.249 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:54:36.250 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:19.709896+0000", 2021-03-24T21:54:36.250 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2024e-05 2021-03-24T21:54:36.250 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.250 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.250 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:54:36.251 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:19.709908+0000", 2021-03-24T21:54:36.251 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.6446999999999995e-05 2021-03-24T21:54:36.251 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.251 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.252 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:54:36.252 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:19.709984+0000", 2021-03-24T21:54:36.252 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.3562000000000002e-05 2021-03-24T21:54:36.252 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.252 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.253 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:54:36.253 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:19.710018+0000", 2021-03-24T21:54:36.253 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.003051677 2021-03-24T21:54:36.253 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.253 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.254 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:54:36.254 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:19.713070+0000", 2021-03-24T21:54:36.254 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.5874999999999999e-05 2021-03-24T21:54:36.254 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.255 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.255 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:54:36.255 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:19.713096+0000", 2021-03-24T21:54:36.255 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0032427659999999998 2021-03-24T21:54:36.255 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:36.256 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:54:36.256 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:36.256 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.256 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.257 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop_reply(osd.0.0:85 4.1e e89/62 ondisk, result = 0)", 2021-03-24T21:54:36.257 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:53:21.577655+0000", 2021-03-24T21:54:36.257 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 74.459792614999998, 2021-03-24T21:54:36.257 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0038674719999999998, 2021-03-24T21:54:36.257 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:54:36.258 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T21:54:36.258 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:54:36.258 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.258 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:54:36.259 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.577655+0000", 2021-03-24T21:54:36.259 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:54:36.259 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.259 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.259 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:54:36.260 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.577655+0000", 2021-03-24T21:54:36.260 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999971 2021-03-24T21:54:36.260 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.260 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.260 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:54:36.261 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.577652+0000", 2021-03-24T21:54:36.261 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3643e-05 2021-03-24T21:54:36.261 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.261 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.262 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:54:36.262 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.577666+0000", 2021-03-24T21:54:36.262 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.8561999999999999e-05 2021-03-24T21:54:36.262 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.262 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.263 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:54:36.263 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.577695+0000", 2021-03-24T21:54:36.263 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.6764000000000001e-05 2021-03-24T21:54:36.263 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.264 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.264 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:54:36.264 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.577711+0000", 2021-03-24T21:54:36.264 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0036806389999999999 2021-03-24T21:54:36.264 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.265 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.265 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:54:36.265 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.581392+0000", 2021-03-24T21:54:36.265 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2309e-05 2021-03-24T21:54:36.265 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.266 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.266 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:54:36.266 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.581404+0000", 2021-03-24T21:54:36.266 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000118288 2021-03-24T21:54:36.267 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.267 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.267 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:54:36.267 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.581523+0000", 2021-03-24T21:54:36.267 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0038674719999999998 2021-03-24T21:54:36.268 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:36.268 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:54:36.268 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:36.268 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.268 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.269 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop_reply(osd.0.0:85 4.1e e89/62 ondisk, result = 0)", 2021-03-24T21:54:36.269 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:53:21.577691+0000", 2021-03-24T21:54:36.269 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 74.459756419000001, 2021-03-24T21:54:36.269 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0042575219999999997, 2021-03-24T21:54:36.270 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:54:36.270 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T21:54:36.270 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:54:36.270 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.270 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:54:36.271 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.577691+0000", 2021-03-24T21:54:36.271 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:54:36.271 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.271 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.271 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:54:36.272 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.577691+0000", 2021-03-24T21:54:36.272 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999976 2021-03-24T21:54:36.272 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.272 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.273 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:54:36.273 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.577689+0000", 2021-03-24T21:54:36.273 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0180000000000001e-05 2021-03-24T21:54:36.273 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.273 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.274 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:54:36.274 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.577699+0000", 2021-03-24T21:54:36.274 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9069e-05 2021-03-24T21:54:36.274 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.274 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.275 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:54:36.275 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.577718+0000", 2021-03-24T21:54:36.275 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2486e-05 2021-03-24T21:54:36.275 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.276 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.276 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:54:36.276 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.577731+0000", 2021-03-24T21:54:36.276 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0039041520000000001 2021-03-24T21:54:36.276 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.277 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.277 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:54:36.277 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.581635+0000", 2021-03-24T21:54:36.277 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.9635999999999999e-05 2021-03-24T21:54:36.277 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.278 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.278 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:54:36.278 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.581665+0000", 2021-03-24T21:54:36.278 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00028426900000000003 2021-03-24T21:54:36.279 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.279 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.279 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:54:36.279 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.581949+0000", 2021-03-24T21:54:36.279 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0042575219999999997 2021-03-24T21:54:36.280 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:36.280 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:54:36.280 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:36.280 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.280 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.281 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop_reply(osd.0.0:86 4.1e e89/62 ondisk, result = 0)", 2021-03-24T21:54:36.281 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:53:21.577894+0000", 2021-03-24T21:54:36.281 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 74.459553240999995, 2021-03-24T21:54:36.281 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0041483229999999998, 2021-03-24T21:54:36.281 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:54:36.282 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T21:54:36.282 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:54:36.282 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.282 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:54:36.283 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.577894+0000", 2021-03-24T21:54:36.283 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:54:36.283 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.283 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.283 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:54:36.284 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.577894+0000", 2021-03-24T21:54:36.284 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999971 2021-03-24T21:54:36.284 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.284 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.285 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:54:36.285 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.577892+0000", 2021-03-24T21:54:36.285 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3539e-05 2021-03-24T21:54:36.285 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.285 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.286 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:54:36.286 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.577905+0000", 2021-03-24T21:54:36.286 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.8794e-05 2021-03-24T21:54:36.286 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.286 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.287 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:54:36.287 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.577924+0000", 2021-03-24T21:54:36.287 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.9560000000000003e-06 2021-03-24T21:54:36.287 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.288 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.288 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:54:36.288 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.577933+0000", 2021-03-24T21:54:36.288 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0040476050000000001 2021-03-24T21:54:36.288 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.289 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.289 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:54:36.289 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.581981+0000", 2021-03-24T21:54:36.289 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.423e-05 2021-03-24T21:54:36.289 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.290 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.290 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:54:36.290 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.582005+0000", 2021-03-24T21:54:36.290 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.7985999999999998e-05 2021-03-24T21:54:36.291 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.291 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.291 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:54:36.291 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.582043+0000", 2021-03-24T21:54:36.291 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0041483229999999998 2021-03-24T21:54:36.292 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:36.292 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:54:36.292 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:36.292 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.292 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.293 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop_reply(osd.0.0:86 4.1e e89/62 ondisk, result = 0)", 2021-03-24T21:54:36.293 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:53:21.578385+0000", 2021-03-24T21:54:36.293 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 74.459062811999999, 2021-03-24T21:54:36.293 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0040039029999999996, 2021-03-24T21:54:36.293 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:54:36.294 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T21:54:36.294 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:54:36.294 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.294 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:54:36.295 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.578385+0000", 2021-03-24T21:54:36.295 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:54:36.295 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.295 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.295 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:54:36.296 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.578385+0000", 2021-03-24T21:54:36.296 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999976 2021-03-24T21:54:36.296 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.296 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.296 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:54:36.297 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.578383+0000", 2021-03-24T21:54:36.297 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.4447000000000001e-05 2021-03-24T21:54:36.297 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.297 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.298 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:54:36.298 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.578397+0000", 2021-03-24T21:54:36.298 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.0698000000000002e-05 2021-03-24T21:54:36.298 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.298 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.299 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:54:36.299 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.578438+0000", 2021-03-24T21:54:36.299 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.4589999999999999e-05 2021-03-24T21:54:36.299 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.300 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.300 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:54:36.300 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.578452+0000", 2021-03-24T21:54:36.300 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0036506120000000001 2021-03-24T21:54:36.300 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.301 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.301 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:54:36.301 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.582103+0000", 2021-03-24T21:54:36.301 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.2202000000000001e-05 2021-03-24T21:54:36.301 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.302 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.302 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:54:36.302 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.582135+0000", 2021-03-24T21:54:36.302 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00025361699999999998 2021-03-24T21:54:36.303 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.303 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.303 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:54:36.303 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.582389+0000", 2021-03-24T21:54:36.303 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0040039029999999996 2021-03-24T21:54:36.304 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:36.304 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:54:36.304 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:36.304 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.305 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.305 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop_reply(osd.0.0:87 4.1e e89/62 ondisk, result = 0)", 2021-03-24T21:54:36.305 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:53:21.578944+0000", 2021-03-24T21:54:36.305 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 74.458503676999996, 2021-03-24T21:54:36.305 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0035300710000000001, 2021-03-24T21:54:36.306 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:54:36.306 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T21:54:36.306 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:54:36.306 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.307 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:54:36.307 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.578944+0000", 2021-03-24T21:54:36.307 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:54:36.307 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.307 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.308 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:54:36.308 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.578944+0000", 2021-03-24T21:54:36.308 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999971 2021-03-24T21:54:36.308 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.308 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.309 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:54:36.309 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.578941+0000", 2021-03-24T21:54:36.309 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.6161999999999999e-05 2021-03-24T21:54:36.309 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.310 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.310 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:54:36.310 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.578957+0000", 2021-03-24T21:54:36.310 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.3706e-05 2021-03-24T21:54:36.310 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.311 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.311 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:54:36.311 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.578981+0000", 2021-03-24T21:54:36.311 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7799999999999999e-05 2021-03-24T21:54:36.311 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.312 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.312 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:54:36.312 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.578999+0000", 2021-03-24T21:54:36.312 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0034349070000000001 2021-03-24T21:54:36.313 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.313 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.313 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:54:36.313 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.582434+0000", 2021-03-24T21:54:36.313 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.6824e-05 2021-03-24T21:54:36.314 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.314 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.314 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:54:36.314 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.582450+0000", 2021-03-24T21:54:36.315 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.3734999999999999e-05 2021-03-24T21:54:36.315 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.315 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.315 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:54:36.315 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.582474+0000", 2021-03-24T21:54:36.316 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0035300710000000001 2021-03-24T21:54:36.316 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:36.316 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:54:36.316 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:36.317 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.317 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.317 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop_reply(osd.0.0:87 4.1e e89/62 ondisk, result = 0)", 2021-03-24T21:54:36.317 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:53:21.579132+0000", 2021-03-24T21:54:36.317 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 74.458315689000003, 2021-03-24T21:54:36.318 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0035170560000000002, 2021-03-24T21:54:36.318 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:54:36.318 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T21:54:36.318 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:54:36.319 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.319 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:54:36.319 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.579132+0000", 2021-03-24T21:54:36.319 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:54:36.319 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.320 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.320 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:54:36.320 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.579132+0000", 2021-03-24T21:54:36.320 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999967 2021-03-24T21:54:36.321 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.321 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.321 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:54:36.321 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.579129+0000", 2021-03-24T21:54:36.321 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5339999999999999e-05 2021-03-24T21:54:36.322 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.322 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.322 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:54:36.322 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.579144+0000", 2021-03-24T21:54:36.323 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.2164999999999999e-05 2021-03-24T21:54:36.323 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.323 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.323 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:54:36.323 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.579166+0000", 2021-03-24T21:54:36.324 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9616999999999999e-05 2021-03-24T21:54:36.324 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.324 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.324 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:54:36.324 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.579186+0000", 2021-03-24T21:54:36.325 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0033324439999999999 2021-03-24T21:54:36.325 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.325 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.325 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:54:36.326 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.582518+0000", 2021-03-24T21:54:36.326 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1046000000000001e-05 2021-03-24T21:54:36.326 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.326 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.326 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:54:36.327 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.582529+0000", 2021-03-24T21:54:36.327 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000119954 2021-03-24T21:54:36.327 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.327 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.328 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:54:36.328 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.582649+0000", 2021-03-24T21:54:36.328 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0035170560000000002 2021-03-24T21:54:36.328 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:36.328 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:54:36.329 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:36.329 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.329 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.329 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop_reply(osd.0.0:88 4.1e e89/62 ondisk, result = 0)", 2021-03-24T21:54:36.329 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:53:21.579140+0000", 2021-03-24T21:54:36.330 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 74.458307836000003, 2021-03-24T21:54:36.330 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0036380850000000001, 2021-03-24T21:54:36.330 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:54:36.330 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T21:54:36.331 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:54:36.331 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.331 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:54:36.331 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.579140+0000", 2021-03-24T21:54:36.331 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:54:36.332 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.332 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.332 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:54:36.332 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.579140+0000", 2021-03-24T21:54:36.332 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999976 2021-03-24T21:54:36.333 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.333 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.333 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:54:36.333 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.579137+0000", 2021-03-24T21:54:36.334 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.9960000000000005e-06 2021-03-24T21:54:36.334 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.334 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.334 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:54:36.334 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.579147+0000", 2021-03-24T21:54:36.335 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.4273e-05 2021-03-24T21:54:36.335 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.335 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.335 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:54:36.336 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.579162+0000", 2021-03-24T21:54:36.336 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.3238999999999998e-05 2021-03-24T21:54:36.336 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.336 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.336 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:54:36.337 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.579195+0000", 2021-03-24T21:54:36.337 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0035486979999999999 2021-03-24T21:54:36.337 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.337 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.338 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:54:36.338 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.582744+0000", 2021-03-24T21:54:36.338 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.31e-05 2021-03-24T21:54:36.338 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.338 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.339 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:54:36.339 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.582757+0000", 2021-03-24T21:54:36.339 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.1202000000000001e-05 2021-03-24T21:54:36.339 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:36.340 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:36.340 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:54:36.340 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:53:21.582778+0000", 2021-03-24T21:54:36.340 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0036380850000000001 2021-03-24T21:54:36.340 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:36.341 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:54:36.341 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:36.341 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:36.341 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:54:36.341 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-24T21:54:36.342 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:36.343 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-24T21:54:36.344 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:35 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:54:36.344 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:35 smithi125 conmon[24927]: 21:54:34.707859+0000 mgr.y (mgr.14144) 310 : cluster [DBG] pgmap v321: 148 pgs: 148 active+clean; 25 KiB data, 52 MiB used, 626 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-03-24T21:54:36.345 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:35 smithi125 conmon[24927]: audit 2021 2021-03-24T21:54:36.345 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:35 smithi125 conmon[24927]: -03-24T21:54:34. 2021-03-24T21:54:36.345 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:35 smithi125 conmon[24927]: 749722+0000 mon.a (mon.0) 639 2021-03-24T21:54:36.345 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:35 smithi125 conmon[24927]: : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:54:36.346 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:35 smithi125 conmon[24927]: audit 2021-03 2021-03-24T21:54:36.346 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:35 smithi125 conmon[24927]: -24T21:54:35.244773 2021-03-24T21:54:36.346 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:35 smithi125 conmon[24927]: +0000 mon.a (mon.0) 640 : 2021-03-24T21:54:36.346 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:35 smithi125 conmon[24927]: audit [INF] from='client.? v1:172.21.15.125:0/2843332919' entity='client.0' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]: dispatch 2021-03-24T21:54:36.347 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:35 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T21:54:36.347 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:35 smithi125 conmon[19936]: T21:54:34.707859+0000 mgr.y (mgr.14144) 310 : cluster [DBG] pgmap v321: 148 pgs: 148 active+clean; 25 KiB data, 52 MiB used, 626 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-03-24T21:54:36.347 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:35 smithi125 conmon[19936]: audit 2021-03-24T21:54:36.347 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:35 smithi125 conmon[19936]: 2021-03-24T 2021-03-24T21:54:36.348 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:35 smithi125 conmon[19936]: 21:54:34.749722+0000 mon.a (mon.0) 639 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:54:36.348 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:35 smithi125 conmon[19936]: 2021-03-24T21:54:36.348 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:35 smithi125 conmon[19936]: audit 2021-03-24 2021-03-24T21:54:36.348 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:35 smithi125 conmon[19936]: T21:54:35.244773+0000 mon.a (mon. 2021-03-24T21:54:36.349 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:35 smithi125 conmon[19936]: 0) 640 : audit [INF] from='client.? v1:172.21.15.125:0/2843332919' entity='client.0' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]: dispatch 2021-03-24T21:54:36.362 INFO:tasks.rados.rados.0.smithi125.stdout:1: write initial oid 1 2021-03-24T21:54:36.363 INFO:tasks.rados.rados.0.smithi125.stdout:1: seq_num 0 ranges {416807=475249,1542129=474678} 2021-03-24T21:54:36.363 INFO:tasks.rados.rados.0.smithi125.stdout:1: writing smithi12556470-1 from 416807 to 892056 tid 1 2021-03-24T21:54:36.363 INFO:tasks.rados.rados.0.smithi125.stdout:1: writing smithi12556470-1 from 1542129 to 2016807 tid 2 2021-03-24T21:54:36.363 INFO:tasks.rados.rados.0.smithi125.stdout:2: write initial oid 2 2021-03-24T21:54:36.363 INFO:tasks.rados.rados.0.smithi125.stdout:2: seq_num 1 ranges {416807=475249,1542129=474678} 2021-03-24T21:54:36.364 INFO:tasks.rados.rados.0.smithi125.stdout:2: writing smithi12556470-2 from 416807 to 892056 tid 1 2021-03-24T21:54:36.364 INFO:tasks.rados.rados.0.smithi125.stdout:2: writing smithi12556470-2 from 1542129 to 2016807 tid 2 2021-03-24T21:54:36.364 INFO:tasks.rados.rados.0.smithi125.stdout:3: write initial oid 3 2021-03-24T21:54:36.364 INFO:tasks.rados.rados.0.smithi125.stdout:1: finishing write tid 1 to smithi12556470-1 2021-03-24T21:54:36.365 INFO:tasks.rados.rados.0.smithi125.stdout:3: seq_num 2 ranges {433614=550498,1600611=433003} 2021-03-24T21:54:36.365 INFO:tasks.rados.rados.0.smithi125.stdout:3: writing smithi12556470-3 from 433614 to 984112 tid 1 2021-03-24T21:54:36.365 INFO:tasks.rados.rados.0.smithi125.stdout:3: writing smithi12556470-3 from 1600611 to 2033614 tid 2 2021-03-24T21:54:36.365 INFO:tasks.rados.rados.0.smithi125.stdout:4: write initial oid 4 2021-03-24T21:54:36.365 INFO:tasks.rados.rados.0.smithi125.stdout:1: finishing write tid 2 to smithi12556470-1 2021-03-24T21:54:36.366 INFO:tasks.rados.rados.0.smithi125.stdout:4: seq_num 3 ranges {450421=625747,1659093=391328} 2021-03-24T21:54:36.366 INFO:tasks.rados.rados.0.smithi125.stdout:4: writing smithi12556470-4 from 450421 to 1076168 tid 1 2021-03-24T21:54:36.366 INFO:tasks.rados.rados.0.smithi125.stdout:4: writing smithi12556470-4 from 1659093 to 2050421 tid 2 2021-03-24T21:54:36.366 INFO:tasks.rados.rados.0.smithi125.stdout:5: write initial oid 5 2021-03-24T21:54:36.367 INFO:tasks.rados.rados.0.smithi125.stdout:1: finishing write tid 3 to smithi12556470-1 2021-03-24T21:54:36.367 INFO:tasks.rados.rados.0.smithi125.stdout:5: seq_num 4 ranges {467228=700996,1717575=349653} 2021-03-24T21:54:36.367 INFO:tasks.rados.rados.0.smithi125.stdout:5: writing smithi12556470-5 from 467228 to 1168224 tid 1 2021-03-24T21:54:36.367 INFO:tasks.rados.rados.0.smithi125.stdout:5: writing smithi12556470-5 from 1717575 to 2067228 tid 2 2021-03-24T21:54:36.367 INFO:tasks.rados.rados.0.smithi125.stdout:6: write initial oid 6 2021-03-24T21:54:36.368 INFO:tasks.rados.rados.0.smithi125.stdout:6: seq_num 5 ranges {484035=776245,2059704=24331} 2021-03-24T21:54:36.368 INFO:tasks.rados.rados.0.smithi125.stdout:6: writing smithi12556470-6 from 484035 to 1260280 tid 1 2021-03-24T21:54:36.368 INFO:tasks.rados.rados.0.smithi125.stdout:6: writing smithi12556470-6 from 2059704 to 2084035 tid 2 2021-03-24T21:54:36.368 INFO:tasks.rados.rados.0.smithi125.stdout:7: write initial oid 7 2021-03-24T21:54:36.369 INFO:tasks.rados.rados.0.smithi125.stdout:7: seq_num 6 ranges {500842=451494,1718186=382656} 2021-03-24T21:54:36.369 INFO:tasks.rados.rados.0.smithi125.stdout:7: writing smithi12556470-7 from 500842 to 952336 tid 1 2021-03-24T21:54:36.369 INFO:tasks.rados.rados.0.smithi125.stdout:7: writing smithi12556470-7 from 1718186 to 2100842 tid 2 2021-03-24T21:54:36.369 INFO:tasks.rados.rados.0.smithi125.stdout:8: write initial oid 8 2021-03-24T21:54:36.369 INFO:tasks.rados.rados.0.smithi125.stdout:8: seq_num 7 ranges {517649=526743,1776668=340981} 2021-03-24T21:54:36.370 INFO:tasks.rados.rados.0.smithi125.stdout:8: writing smithi12556470-8 from 517649 to 1044392 tid 1 2021-03-24T21:54:36.370 INFO:tasks.rados.rados.0.smithi125.stdout:8: writing smithi12556470-8 from 1776668 to 2117649 tid 2 2021-03-24T21:54:36.370 INFO:tasks.rados.rados.0.smithi125.stdout:9: write initial oid 9 2021-03-24T21:54:36.370 INFO:tasks.rados.rados.0.smithi125.stdout:9: seq_num 8 ranges {534456=718345,1951503=182953} 2021-03-24T21:54:36.371 INFO:tasks.rados.rados.0.smithi125.stdout:9: writing smithi12556470-9 from 534456 to 1252801 tid 1 2021-03-24T21:54:36.371 INFO:tasks.rados.rados.0.smithi125.stdout:9: writing smithi12556470-9 from 1951503 to 2134456 tid 2 2021-03-24T21:54:36.371 INFO:tasks.rados.rados.0.smithi125.stdout:10: write initial oid 10 2021-03-24T21:54:36.371 INFO:tasks.rados.rados.0.smithi125.stdout:10: seq_num 9 ranges {551263=793594,1893632=257631} 2021-03-24T21:54:36.371 INFO:tasks.rados.rados.0.smithi125.stdout:10: writing smithi12556470-10 from 551263 to 1344857 tid 1 2021-03-24T21:54:36.372 INFO:tasks.rados.rados.0.smithi125.stdout:10: writing smithi12556470-10 from 1893632 to 2151263 tid 2 2021-03-24T21:54:36.372 INFO:tasks.rados.rados.0.smithi125.stdout:11: write initial oid 11 2021-03-24T21:54:36.372 INFO:tasks.rados.rados.0.smithi125.stdout:11: seq_num 10 ranges {568070=468843,1552114=615956} 2021-03-24T21:54:36.372 INFO:tasks.rados.rados.0.smithi125.stdout:11: writing smithi12556470-11 from 568070 to 1036913 tid 1 2021-03-24T21:54:36.372 INFO:tasks.rados.rados.0.smithi125.stdout:11: writing smithi12556470-11 from 1552114 to 2168070 tid 2 2021-03-24T21:54:36.373 INFO:tasks.rados.rados.0.smithi125.stdout:12: write initial oid 12 2021-03-24T21:54:36.373 INFO:tasks.rados.rados.0.smithi125.stdout:12: seq_num 11 ranges {584877=544092,1610596=562003} 2021-03-24T21:54:36.373 INFO:tasks.rados.rados.0.smithi125.stdout:12: writing smithi12556470-12 from 584877 to 1128969 tid 1 2021-03-24T21:54:36.373 INFO:tasks.rados.rados.0.smithi125.stdout:12: writing smithi12556470-12 from 1610596 to 2172599 tid 2 2021-03-24T21:54:36.374 INFO:tasks.rados.rados.0.smithi125.stdout:13: write initial oid 13 2021-03-24T21:54:36.374 INFO:tasks.rados.rados.0.smithi125.stdout:13: seq_num 12 ranges {601684=619341,1669078=532606} 2021-03-24T21:54:36.374 INFO:tasks.rados.rados.0.smithi125.stdout:13: writing smithi12556470-13 from 601684 to 1221025 tid 1 2021-03-24T21:54:36.374 INFO:tasks.rados.rados.0.smithi125.stdout:13: writing smithi12556470-13 from 1669078 to 2201684 tid 2 2021-03-24T21:54:36.374 INFO:tasks.rados.rados.0.smithi125.stdout:14: write initial oid 14 2021-03-24T21:54:36.375 INFO:tasks.rados.rados.0.smithi125.stdout:14: seq_num 13 ranges {618491=694590,2011207=207284} 2021-03-24T21:54:36.375 INFO:tasks.rados.rados.0.smithi125.stdout:14: writing smithi12556470-14 from 618491 to 1313081 tid 1 2021-03-24T21:54:36.375 INFO:tasks.rados.rados.0.smithi125.stdout:14: writing smithi12556470-14 from 2011207 to 2218491 tid 2 2021-03-24T21:54:36.375 INFO:tasks.rados.rados.0.smithi125.stdout:15: write initial oid 15 2021-03-24T21:54:36.376 INFO:tasks.rados.rados.0.smithi125.stdout:15: seq_num 14 ranges {635298=769839,2069689=165609} 2021-03-24T21:54:36.376 INFO:tasks.rados.rados.0.smithi125.stdout:15: writing smithi12556470-15 from 635298 to 1405137 tid 1 2021-03-24T21:54:36.376 INFO:tasks.rados.rados.0.smithi125.stdout:15: writing smithi12556470-15 from 2069689 to 2235298 tid 2 2021-03-24T21:54:36.376 INFO:tasks.rados.rados.0.smithi125.stdout:16: write initial oid 16 2021-03-24T21:54:36.376 INFO:tasks.rados.rados.0.smithi125.stdout:16: seq_num 15 ranges {652105=445088,1728171=452988} 2021-03-24T21:54:36.377 INFO:tasks.rados.rados.0.smithi125.stdout:16: writing smithi12556470-16 from 652105 to 1097193 tid 1 2021-03-24T21:54:36.377 INFO:tasks.rados.rados.0.smithi125.stdout:16: writing smithi12556470-16 from 1728171 to 2181159 tid 2 2021-03-24T21:54:36.377 INFO:tasks.rados.rados.0.smithi125.stdout: waiting on 16 2021-03-24T21:54:36.377 INFO:tasks.rados.rados.0.smithi125.stdout:1: finishing write tid 4 to smithi12556470-1 2021-03-24T21:54:36.377 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2021-03-24T21:54:36.378 INFO:tasks.rados.rados.0.smithi125.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2021-03-24T21:54:36.378 INFO:tasks.rados.rados.0.smithi125.stdout:2: finishing write tid 1 to smithi12556470-2 2021-03-24T21:54:36.378 INFO:tasks.rados.rados.0.smithi125.stdout:1: done (15 left) 2021-03-24T21:54:36.378 INFO:tasks.rados.rados.0.smithi125.stdout:17: write initial oid 17 2021-03-24T21:54:36.379 INFO:tasks.rados.rados.0.smithi125.stdout:17: seq_num 16 ranges {668912=636690,1903006=365906} 2021-03-24T21:54:36.379 INFO:tasks.rados.rados.0.smithi125.stdout:17: writing smithi12556470-17 from 668912 to 1305602 tid 1 2021-03-24T21:54:36.379 INFO:tasks.rados.rados.0.smithi125.stdout:17: writing smithi12556470-17 from 1903006 to 2268912 tid 2 2021-03-24T21:54:36.379 INFO:tasks.rados.rados.0.smithi125.stdout: waiting on 16 2021-03-24T21:54:36.379 INFO:tasks.rados.rados.0.smithi125.stdout:2: finishing write tid 2 to smithi12556470-2 2021-03-24T21:54:36.380 INFO:tasks.rados.rados.0.smithi125.stdout:2: finishing write tid 3 to smithi12556470-2 2021-03-24T21:54:36.380 INFO:tasks.rados.rados.0.smithi125.stdout:2: finishing write tid 4 to smithi12556470-2 2021-03-24T21:54:36.380 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 2 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2021-03-24T21:54:36.380 INFO:tasks.rados.rados.0.smithi125.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2021-03-24T21:54:36.380 INFO:tasks.rados.rados.0.smithi125.stdout:3: finishing write tid 1 to smithi12556470-3 2021-03-24T21:54:36.381 INFO:tasks.rados.rados.0.smithi125.stdout:3: finishing write tid 2 to smithi12556470-3 2021-03-24T21:54:36.381 INFO:tasks.rados.rados.0.smithi125.stdout:2: done (15 left) 2021-03-24T21:54:36.381 INFO:tasks.rados.rados.0.smithi125.stdout:18: write initial oid 18 2021-03-24T21:54:36.381 INFO:tasks.rados.rados.0.smithi125.stdout:18: seq_num 17 ranges {685719=711939,1845135=440584} 2021-03-24T21:54:36.381 INFO:tasks.rados.rados.0.smithi125.stdout:18: writing smithi12556470-18 from 685719 to 1397658 tid 1 2021-03-24T21:54:36.382 INFO:tasks.rados.rados.0.smithi125.stdout:18: writing smithi12556470-18 from 1845135 to 2285719 tid 2 2021-03-24T21:54:36.382 INFO:tasks.rados.rados.0.smithi125.stdout: waiting on 16 2021-03-24T21:54:36.382 INFO:tasks.rados.rados.0.smithi125.stdout:3: finishing write tid 3 to smithi12556470-3 2021-03-24T21:54:36.382 INFO:tasks.rados.rados.0.smithi125.stdout:3: finishing write tid 4 to smithi12556470-3 2021-03-24T21:54:36.383 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 3 v 3 (ObjNum 2 snap 0 seq_num 2) dirty exists 2021-03-24T21:54:36.383 INFO:tasks.rados.rados.0.smithi125.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2021-03-24T21:54:36.383 INFO:tasks.rados.rados.0.smithi125.stdout:4: finishing write tid 1 to smithi12556470-4 2021-03-24T21:54:36.383 INFO:tasks.rados.rados.0.smithi125.stdout:4: finishing write tid 2 to smithi12556470-4 2021-03-24T21:54:36.384 INFO:tasks.rados.rados.0.smithi125.stdout:4: finishing write tid 3 to smithi12556470-4 2021-03-24T21:54:36.384 INFO:tasks.rados.rados.0.smithi125.stdout:4: finishing write tid 4 to smithi12556470-4 2021-03-24T21:54:36.384 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 4 v 6 (ObjNum 3 snap 0 seq_num 3) dirty exists 2021-03-24T21:54:36.384 INFO:tasks.rados.rados.0.smithi125.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2021-03-24T21:54:36.384 INFO:tasks.rados.rados.0.smithi125.stdout:3: done (15 left) 2021-03-24T21:54:36.385 INFO:tasks.rados.rados.0.smithi125.stdout:4: done (14 left) 2021-03-24T21:54:36.385 INFO:tasks.rados.rados.0.smithi125.stdout:19: write initial oid 19 2021-03-24T21:54:36.385 INFO:tasks.rados.rados.0.smithi125.stdout:19: seq_num 18 ranges {702526=787188,1903617=398909} 2021-03-24T21:54:36.385 INFO:tasks.rados.rados.0.smithi125.stdout:19: writing smithi12556470-19 from 702526 to 1489714 tid 1 2021-03-24T21:54:36.385 INFO:tasks.rados.rados.0.smithi125.stdout:19: writing smithi12556470-19 from 1903617 to 2302526 tid 2 2021-03-24T21:54:36.386 INFO:tasks.rados.rados.0.smithi125.stdout:20: write initial oid 20 2021-03-24T21:54:36.386 INFO:tasks.rados.rados.0.smithi125.stdout:20: seq_num 19 ranges {719333=462437,1962099=357234} 2021-03-24T21:54:36.386 INFO:tasks.rados.rados.0.smithi125.stdout:20: writing smithi12556470-20 from 719333 to 1181770 tid 1 2021-03-24T21:54:36.386 INFO:tasks.rados.rados.0.smithi125.stdout:20: writing smithi12556470-20 from 1962099 to 2319333 tid 2 2021-03-24T21:54:36.386 INFO:tasks.rados.rados.0.smithi125.stdout: waiting on 16 2021-03-24T21:54:36.387 INFO:tasks.rados.rados.0.smithi125.stdout:5: finishing write tid 1 to smithi12556470-5 2021-03-24T21:54:36.387 INFO:tasks.rados.rados.0.smithi125.stdout:5: finishing write tid 2 to smithi12556470-5 2021-03-24T21:54:36.387 INFO:tasks.rados.rados.0.smithi125.stdout:5: finishing write tid 3 to smithi12556470-5 2021-03-24T21:54:36.387 INFO:tasks.rados.rados.0.smithi125.stdout:5: finishing write tid 4 to smithi12556470-5 2021-03-24T21:54:36.388 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 5 v 3 (ObjNum 4 snap 0 seq_num 4) dirty exists 2021-03-24T21:54:36.388 INFO:tasks.rados.rados.0.smithi125.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2021-03-24T21:54:36.388 INFO:tasks.rados.rados.0.smithi125.stdout:6: finishing write tid 1 to smithi12556470-6 2021-03-24T21:54:36.388 INFO:tasks.rados.rados.0.smithi125.stdout:6: finishing write tid 2 to smithi12556470-6 2021-03-24T21:54:36.388 INFO:tasks.rados.rados.0.smithi125.stdout:6: finishing write tid 3 to smithi12556470-6 2021-03-24T21:54:36.389 INFO:tasks.rados.rados.0.smithi125.stdout:6: finishing write tid 4 to smithi12556470-6 2021-03-24T21:54:36.389 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 6 v 3 (ObjNum 5 snap 0 seq_num 5) dirty exists 2021-03-24T21:54:36.389 INFO:tasks.rados.rados.0.smithi125.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2021-03-24T21:54:36.389 INFO:tasks.rados.rados.0.smithi125.stdout:7: finishing write tid 1 to smithi12556470-7 2021-03-24T21:54:36.390 INFO:tasks.rados.rados.0.smithi125.stdout:7: finishing write tid 2 to smithi12556470-7 2021-03-24T21:54:36.390 INFO:tasks.rados.rados.0.smithi125.stdout:7: finishing write tid 3 to smithi12556470-7 2021-03-24T21:54:36.390 INFO:tasks.rados.rados.0.smithi125.stdout:7: finishing write tid 4 to smithi12556470-7 2021-03-24T21:54:36.390 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 7 v 6 (ObjNum 6 snap 0 seq_num 6) dirty exists 2021-03-24T21:54:36.390 INFO:tasks.rados.rados.0.smithi125.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2021-03-24T21:54:36.391 INFO:tasks.rados.rados.0.smithi125.stdout:8: finishing write tid 1 to smithi12556470-8 2021-03-24T21:54:36.391 INFO:tasks.rados.rados.0.smithi125.stdout:8: finishing write tid 2 to smithi12556470-8 2021-03-24T21:54:36.391 INFO:tasks.rados.rados.0.smithi125.stdout:8: finishing write tid 3 to smithi12556470-8 2021-03-24T21:54:36.391 INFO:tasks.rados.rados.0.smithi125.stdout:8: finishing write tid 4 to smithi12556470-8 2021-03-24T21:54:36.391 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 8 v 3 (ObjNum 7 snap 0 seq_num 7) dirty exists 2021-03-24T21:54:36.392 INFO:tasks.rados.rados.0.smithi125.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2021-03-24T21:54:36.392 INFO:tasks.rados.rados.0.smithi125.stdout:9: finishing write tid 1 to smithi12556470-9 2021-03-24T21:54:36.392 INFO:tasks.rados.rados.0.smithi125.stdout:9: finishing write tid 2 to smithi12556470-9 2021-03-24T21:54:36.392 INFO:tasks.rados.rados.0.smithi125.stdout:9: finishing write tid 3 to smithi12556470-9 2021-03-24T21:54:36.393 INFO:tasks.rados.rados.0.smithi125.stdout:9: finishing write tid 4 to smithi12556470-9 2021-03-24T21:54:36.393 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 9 v 3 (ObjNum 8 snap 0 seq_num 8) dirty exists 2021-03-24T21:54:36.393 INFO:tasks.rados.rados.0.smithi125.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2021-03-24T21:54:36.393 INFO:tasks.rados.rados.0.smithi125.stdout:10: finishing write tid 1 to smithi12556470-10 2021-03-24T21:54:36.393 INFO:tasks.rados.rados.0.smithi125.stdout:10: finishing write tid 2 to smithi12556470-10 2021-03-24T21:54:36.394 INFO:tasks.rados.rados.0.smithi125.stdout:10: finishing write tid 3 to smithi12556470-10 2021-03-24T21:54:36.394 INFO:tasks.rados.rados.0.smithi125.stdout:10: finishing write tid 4 to smithi12556470-10 2021-03-24T21:54:36.394 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 10 v 3 (ObjNum 9 snap 0 seq_num 9) dirty exists 2021-03-24T21:54:36.394 INFO:tasks.rados.rados.0.smithi125.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2021-03-24T21:54:36.395 INFO:tasks.rados.rados.0.smithi125.stdout:11: finishing write tid 1 to smithi12556470-11 2021-03-24T21:54:36.395 INFO:tasks.rados.rados.0.smithi125.stdout:11: finishing write tid 2 to smithi12556470-11 2021-03-24T21:54:36.395 INFO:tasks.rados.rados.0.smithi125.stdout:11: finishing write tid 3 to smithi12556470-11 2021-03-24T21:54:36.395 INFO:tasks.rados.rados.0.smithi125.stdout:11: finishing write tid 4 to smithi12556470-11 2021-03-24T21:54:36.395 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 11 v 9 (ObjNum 10 snap 0 seq_num 10) dirty exists 2021-03-24T21:54:36.396 INFO:tasks.rados.rados.0.smithi125.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2021-03-24T21:54:36.396 INFO:tasks.rados.rados.0.smithi125.stdout:12: finishing write tid 1 to smithi12556470-12 2021-03-24T21:54:36.396 INFO:tasks.rados.rados.0.smithi125.stdout:12: finishing write tid 2 to smithi12556470-12 2021-03-24T21:54:36.396 INFO:tasks.rados.rados.0.smithi125.stdout:12: finishing write tid 3 to smithi12556470-12 2021-03-24T21:54:36.396 INFO:tasks.rados.rados.0.smithi125.stdout:5: done (15 left) 2021-03-24T21:54:36.397 INFO:tasks.rados.rados.0.smithi125.stdout:6: done (14 left) 2021-03-24T21:54:36.397 INFO:tasks.rados.rados.0.smithi125.stdout:7: done (13 left) 2021-03-24T21:54:36.397 INFO:tasks.rados.rados.0.smithi125.stdout:8: done (12 left) 2021-03-24T21:54:36.397 INFO:tasks.rados.rados.0.smithi125.stdout:9: done (11 left) 2021-03-24T21:54:36.398 INFO:tasks.rados.rados.0.smithi125.stdout:10: done (10 left) 2021-03-24T21:54:36.398 INFO:tasks.rados.rados.0.smithi125.stdout:11: done (9 left) 2021-03-24T21:54:36.398 INFO:tasks.rados.rados.0.smithi125.stdout:21: write initial oid 21 2021-03-24T21:54:36.398 INFO:tasks.rados.rados.0.smithi125.stdout:21: seq_num 20 ranges {736140=537686,2020581=315559} 2021-03-24T21:54:36.398 INFO:tasks.rados.rados.0.smithi125.stdout:21: writing smithi12556470-21 from 736140 to 1273826 tid 1 2021-03-24T21:54:36.399 INFO:tasks.rados.rados.0.smithi125.stdout:21: writing smithi12556470-21 from 2020581 to 2336140 tid 2 2021-03-24T21:54:36.399 INFO:tasks.rados.rados.0.smithi125.stdout:22: write initial oid 22 2021-03-24T21:54:36.399 INFO:tasks.rados.rados.0.smithi125.stdout:22: seq_num 21 ranges {752947=612935,1962710=390237} 2021-03-24T21:54:36.401 INFO:tasks.rados.rados.0.smithi125.stdout:22: writing smithi12556470-22 from 752947 to 1365882 tid 1 2021-03-24T21:54:36.402 INFO:tasks.rados.rados.0.smithi125.stdout:22: writing smithi12556470-22 from 1962710 to 2352947 tid 2 2021-03-24T21:54:36.402 INFO:tasks.rados.rados.0.smithi125.stdout:23: write initial oid 23 2021-03-24T21:54:36.402 INFO:tasks.rados.rados.0.smithi125.stdout:23: seq_num 22 ranges {769754=688184,2021192=348562} 2021-03-24T21:54:36.402 INFO:tasks.rados.rados.0.smithi125.stdout:23: writing smithi12556470-23 from 769754 to 1457938 tid 1 2021-03-24T21:54:36.403 INFO:tasks.rados.rados.0.smithi125.stdout:23: writing smithi12556470-23 from 2021192 to 2369754 tid 2 2021-03-24T21:54:36.403 INFO:tasks.rados.rados.0.smithi125.stdout:24: write initial oid 24 2021-03-24T21:54:36.403 INFO:tasks.rados.rados.0.smithi125.stdout:24: seq_num 23 ranges {786561=479786,1796027=467664} 2021-03-24T21:54:36.403 INFO:tasks.rados.rados.0.smithi125.stdout:24: writing smithi12556470-24 from 786561 to 1266347 tid 1 2021-03-24T21:54:36.404 INFO:tasks.rados.rados.0.smithi125.stdout:24: writing smithi12556470-24 from 1796027 to 2263691 tid 2 2021-03-24T21:54:36.404 INFO:tasks.rados.rados.0.smithi125.stdout:25: write initial oid 25 2021-03-24T21:54:36.404 INFO:tasks.rados.rados.0.smithi125.stdout:25: seq_num 24 ranges {403368=555035,1454509=727675} 2021-03-24T21:54:36.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:36.405 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:36.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:54:36.406 INFO:tasks.rados.rados.0.smithi125.stdout:25: writing smithi12556470-25 from 403368 to 958403 tid 1 2021-03-24T21:54:36.412 INFO:tasks.rados.rados.0.smithi125.stdout:25: writing smithi12556470-25 from 1454509 to 2182184 tid 2 2021-03-24T21:54:36.413 INFO:tasks.rados.rados.0.smithi125.stdout:26: write initial oid 26 2021-03-24T21:54:36.413 INFO:tasks.rados.rados.0.smithi125.stdout:26: seq_num 25 ranges {420175=630284,1796638=471333} 2021-03-24T21:54:36.419 INFO:tasks.rados.rados.0.smithi125.stdout:26: writing smithi12556470-26 from 420175 to 1050459 tid 1 2021-03-24T21:54:36.426 INFO:tasks.rados.rados.0.smithi125.stdout:26: writing smithi12556470-26 from 1796638 to 2267971 tid 2 2021-03-24T21:54:36.427 INFO:tasks.rados.rados.0.smithi125.stdout:27: write initial oid 27 2021-03-24T21:54:36.427 INFO:tasks.rados.rados.0.smithi125.stdout:27: seq_num 26 ranges {436982=705533,1855120=581862} 2021-03-24T21:54:36.433 INFO:tasks.rados.rados.0.smithi125.stdout:27: writing smithi12556470-27 from 436982 to 1142515 tid 1 2021-03-24T21:54:36.442 INFO:tasks.rados.rados.0.smithi125.stdout:27: writing smithi12556470-27 from 1855120 to 2436982 tid 2 2021-03-24T21:54:36.442 INFO:tasks.rados.rados.0.smithi125.stdout: waiting on 16 2021-03-24T21:54:36.443 INFO:tasks.rados.rados.0.smithi125.stdout:12: finishing write tid 4 to smithi12556470-12 2021-03-24T21:54:36.443 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 12 v 6 (ObjNum 11 snap 0 seq_num 11) dirty exists 2021-03-24T21:54:36.443 INFO:tasks.rados.rados.0.smithi125.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2021-03-24T21:54:36.443 INFO:tasks.rados.rados.0.smithi125.stdout:13: finishing write tid 1 to smithi12556470-13 2021-03-24T21:54:36.443 INFO:tasks.rados.rados.0.smithi125.stdout:13: finishing write tid 2 to smithi12556470-13 2021-03-24T21:54:36.444 INFO:tasks.rados.rados.0.smithi125.stdout:13: finishing write tid 3 to smithi12556470-13 2021-03-24T21:54:36.444 INFO:tasks.rados.rados.0.smithi125.stdout:13: finishing write tid 4 to smithi12556470-13 2021-03-24T21:54:36.444 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 13 v 9 (ObjNum 12 snap 0 seq_num 12) dirty exists 2021-03-24T21:54:36.444 INFO:tasks.rados.rados.0.smithi125.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2021-03-24T21:54:36.445 INFO:tasks.rados.rados.0.smithi125.stdout:14: finishing write tid 1 to smithi12556470-14 2021-03-24T21:54:36.445 INFO:tasks.rados.rados.0.smithi125.stdout:14: finishing write tid 2 to smithi12556470-14 2021-03-24T21:54:36.445 INFO:tasks.rados.rados.0.smithi125.stdout:14: finishing write tid 3 to smithi12556470-14 2021-03-24T21:54:36.445 INFO:tasks.rados.rados.0.smithi125.stdout:14: finishing write tid 4 to smithi12556470-14 2021-03-24T21:54:36.445 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 14 v 6 (ObjNum 13 snap 0 seq_num 13) dirty exists 2021-03-24T21:54:36.446 INFO:tasks.rados.rados.0.smithi125.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2021-03-24T21:54:36.446 INFO:tasks.rados.rados.0.smithi125.stdout:15: finishing write tid 1 to smithi12556470-15 2021-03-24T21:54:36.446 INFO:tasks.rados.rados.0.smithi125.stdout:15: finishing write tid 2 to smithi12556470-15 2021-03-24T21:54:36.446 INFO:tasks.rados.rados.0.smithi125.stdout:15: finishing write tid 3 to smithi12556470-15 2021-03-24T21:54:36.447 INFO:tasks.rados.rados.0.smithi125.stdout:15: finishing write tid 4 to smithi12556470-15 2021-03-24T21:54:36.447 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 15 v 9 (ObjNum 14 snap 0 seq_num 14) dirty exists 2021-03-24T21:54:36.447 INFO:tasks.rados.rados.0.smithi125.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2021-03-24T21:54:36.447 INFO:tasks.rados.rados.0.smithi125.stdout:16: finishing write tid 1 to smithi12556470-16 2021-03-24T21:54:36.447 INFO:tasks.rados.rados.0.smithi125.stdout:12: done (15 left) 2021-03-24T21:54:36.448 INFO:tasks.rados.rados.0.smithi125.stdout:13: done (14 left) 2021-03-24T21:54:36.448 INFO:tasks.rados.rados.0.smithi125.stdout:14: done (13 left) 2021-03-24T21:54:36.448 INFO:tasks.rados.rados.0.smithi125.stdout:15: done (12 left) 2021-03-24T21:54:36.448 INFO:tasks.rados.rados.0.smithi125.stdout:28: write initial oid 28 2021-03-24T21:54:36.448 INFO:tasks.rados.rados.0.smithi125.stdout:28: seq_num 27 ranges {453789=780782,1913602=475002} 2021-03-24T21:54:36.449 INFO:tasks.rados.rados.0.smithi125.stdout:28: writing smithi12556470-28 from 453789 to 1234571 tid 1 2021-03-24T21:54:36.455 INFO:tasks.rados.rados.0.smithi125.stdout:28: writing smithi12556470-28 from 1913602 to 2388604 tid 2 2021-03-24T21:54:36.455 INFO:tasks.rados.rados.0.smithi125.stdout:29: write initial oid 29 2021-03-24T21:54:36.455 INFO:tasks.rados.rados.0.smithi125.stdout:29: seq_num 28 ranges {470596=456031,1572084=618660} 2021-03-24T21:54:36.460 INFO:tasks.rados.rados.0.smithi125.stdout:29: writing smithi12556470-29 from 470596 to 926627 tid 1 2021-03-24T21:54:36.468 INFO:tasks.rados.rados.0.smithi125.stdout:29: writing smithi12556470-29 from 1572084 to 2190744 tid 2 2021-03-24T21:54:36.468 INFO:tasks.rados.rados.0.smithi125.stdout:30: write initial oid 30 2021-03-24T21:54:36.469 INFO:tasks.rados.rados.0.smithi125.stdout:16: finishing write tid 2 to smithi12556470-16 2021-03-24T21:54:36.469 INFO:tasks.rados.rados.0.smithi125.stdout:30: seq_num 29 ranges {487403=531280,1514213=478671} 2021-03-24T21:54:36.474 INFO:tasks.rados.rados.0.smithi125.stdout:30: writing smithi12556470-30 from 487403 to 1018683 tid 1 2021-03-24T21:54:36.480 INFO:tasks.rados.rados.0.smithi125.stdout:30: writing smithi12556470-30 from 1514213 to 1992884 tid 2 2021-03-24T21:54:36.480 INFO:tasks.rados.rados.0.smithi125.stdout:31: write initial oid 31 2021-03-24T21:54:36.481 INFO:tasks.rados.rados.0.smithi125.stdout:31: seq_num 30 ranges {504210=606529,1572695=622329} 2021-03-24T21:54:36.486 INFO:tasks.rados.rados.0.smithi125.stdout:31: writing smithi12556470-31 from 504210 to 1110739 tid 1 2021-03-24T21:54:36.493 INFO:tasks.rados.rados.0.smithi125.stdout:31: writing smithi12556470-31 from 1572695 to 2195024 tid 2 2021-03-24T21:54:36.494 INFO:tasks.rados.rados.0.smithi125.stdout: waiting on 16 2021-03-24T21:54:36.494 INFO:tasks.rados.rados.0.smithi125.stdout:16: finishing write tid 3 to smithi12556470-16 2021-03-24T21:54:36.494 INFO:tasks.rados.rados.0.smithi125.stdout:16: finishing write tid 4 to smithi12556470-16 2021-03-24T21:54:36.494 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 16 v 6 (ObjNum 15 snap 0 seq_num 15) dirty exists 2021-03-24T21:54:36.495 INFO:tasks.rados.rados.0.smithi125.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2021-03-24T21:54:36.495 INFO:tasks.rados.rados.0.smithi125.stdout:17: finishing write tid 1 to smithi12556470-17 2021-03-24T21:54:36.495 INFO:tasks.rados.rados.0.smithi125.stdout:16: done (15 left) 2021-03-24T21:54:36.495 INFO:tasks.rados.rados.0.smithi125.stdout:32: write initial oid 32 2021-03-24T21:54:36.495 INFO:tasks.rados.rados.0.smithi125.stdout:32: seq_num 31 ranges {521017=798131,1747530=482340} 2021-03-24T21:54:36.501 INFO:tasks.rados.rados.0.smithi125.stdout:32: writing smithi12556470-32 from 521017 to 1319148 tid 1 2021-03-24T21:54:36.506 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:36.508 INFO:tasks.rados.rados.0.smithi125.stdout:32: writing smithi12556470-32 from 1747530 to 2229870 tid 2 2021-03-24T21:54:36.508 INFO:tasks.rados.rados.0.smithi125.stdout: waiting on 16 2021-03-24T21:54:36.509 INFO:tasks.rados.rados.0.smithi125.stdout:17: finishing write tid 2 to smithi12556470-17 2021-03-24T21:54:36.509 INFO:tasks.rados.rados.0.smithi125.stdout:17: finishing write tid 3 to smithi12556470-17 2021-03-24T21:54:36.509 INFO:tasks.rados.rados.0.smithi125.stdout:17: finishing write tid 4 to smithi12556470-17 2021-03-24T21:54:36.509 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 17 v 6 (ObjNum 16 snap 0 seq_num 16) dirty exists 2021-03-24T21:54:36.509 INFO:tasks.rados.rados.0.smithi125.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2021-03-24T21:54:36.510 INFO:tasks.rados.rados.0.smithi125.stdout:18: finishing write tid 1 to smithi12556470-18 2021-03-24T21:54:36.510 INFO:tasks.rados.rados.0.smithi125.stdout:18: finishing write tid 2 to smithi12556470-18 2021-03-24T21:54:36.510 INFO:tasks.rados.rados.0.smithi125.stdout:18: finishing write tid 3 to smithi12556470-18 2021-03-24T21:54:36.510 INFO:tasks.rados.rados.0.smithi125.stdout:18: finishing write tid 4 to smithi12556470-18 2021-03-24T21:54:36.511 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 18 v 3 (ObjNum 17 snap 0 seq_num 17) dirty exists 2021-03-24T21:54:36.511 INFO:tasks.rados.rados.0.smithi125.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2021-03-24T21:54:36.511 INFO:tasks.rados.rados.0.smithi125.stdout:19: finishing write tid 1 to smithi12556470-19 2021-03-24T21:54:36.511 INFO:tasks.rados.rados.0.smithi125.stdout:19: finishing write tid 2 to smithi12556470-19 2021-03-24T21:54:36.511 INFO:tasks.rados.rados.0.smithi125.stdout:19: finishing write tid 3 to smithi12556470-19 2021-03-24T21:54:36.512 INFO:tasks.rados.rados.0.smithi125.stdout:19: finishing write tid 4 to smithi12556470-19 2021-03-24T21:54:36.512 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 19 v 12 (ObjNum 18 snap 0 seq_num 18) dirty exists 2021-03-24T21:54:36.512 INFO:tasks.rados.rados.0.smithi125.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2021-03-24T21:54:36.512 INFO:tasks.rados.rados.0.smithi125.stdout:20: finishing write tid 1 to smithi12556470-20 2021-03-24T21:54:36.513 INFO:tasks.rados.rados.0.smithi125.stdout:20: finishing write tid 2 to smithi12556470-20 2021-03-24T21:54:36.513 INFO:tasks.rados.rados.0.smithi125.stdout:20: finishing write tid 3 to smithi12556470-20 2021-03-24T21:54:36.513 INFO:tasks.rados.rados.0.smithi125.stdout:20: finishing write tid 4 to smithi12556470-20 2021-03-24T21:54:36.513 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 20 v 3 (ObjNum 19 snap 0 seq_num 19) dirty exists 2021-03-24T21:54:36.513 INFO:tasks.rados.rados.0.smithi125.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2021-03-24T21:54:36.514 INFO:tasks.rados.rados.0.smithi125.stdout:21: finishing write tid 1 to smithi12556470-21 2021-03-24T21:54:36.514 INFO:tasks.rados.rados.0.smithi125.stdout:21: finishing write tid 2 to smithi12556470-21 2021-03-24T21:54:36.514 INFO:tasks.rados.rados.0.smithi125.stdout:21: finishing write tid 3 to smithi12556470-21 2021-03-24T21:54:36.515 INFO:tasks.rados.rados.0.smithi125.stdout:21: finishing write tid 4 to smithi12556470-21 2021-03-24T21:54:36.515 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 21 v 3 (ObjNum 20 snap 0 seq_num 20) dirty exists 2021-03-24T21:54:36.515 INFO:tasks.rados.rados.0.smithi125.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2021-03-24T21:54:36.515 INFO:tasks.rados.rados.0.smithi125.stdout:22: finishing write tid 1 to smithi12556470-22 2021-03-24T21:54:36.515 INFO:tasks.rados.rados.0.smithi125.stdout:22: finishing write tid 2 to smithi12556470-22 2021-03-24T21:54:36.516 INFO:tasks.rados.rados.0.smithi125.stdout:22: finishing write tid 3 to smithi12556470-22 2021-03-24T21:54:36.516 INFO:tasks.rados.rados.0.smithi125.stdout:22: finishing write tid 4 to smithi12556470-22 2021-03-24T21:54:36.516 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 22 v 6 (ObjNum 21 snap 0 seq_num 21) dirty exists 2021-03-24T21:54:36.516 INFO:tasks.rados.rados.0.smithi125.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2021-03-24T21:54:36.517 INFO:tasks.rados.rados.0.smithi125.stdout:23: finishing write tid 1 to smithi12556470-23 2021-03-24T21:54:36.517 INFO:tasks.rados.rados.0.smithi125.stdout:23: finishing write tid 2 to smithi12556470-23 2021-03-24T21:54:36.517 INFO:tasks.rados.rados.0.smithi125.stdout:23: finishing write tid 3 to smithi12556470-23 2021-03-24T21:54:36.517 INFO:tasks.rados.rados.0.smithi125.stdout:23: finishing write tid 4 to smithi12556470-23 2021-03-24T21:54:36.518 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 23 v 9 (ObjNum 22 snap 0 seq_num 22) dirty exists 2021-03-24T21:54:36.518 INFO:tasks.rados.rados.0.smithi125.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2021-03-24T21:54:36.518 INFO:tasks.rados.rados.0.smithi125.stdout:24: finishing write tid 1 to smithi12556470-24 2021-03-24T21:54:36.518 INFO:tasks.rados.rados.0.smithi125.stdout:17: done (15 left) 2021-03-24T21:54:36.518 INFO:tasks.rados.rados.0.smithi125.stdout:18: done (14 left) 2021-03-24T21:54:36.519 INFO:tasks.rados.rados.0.smithi125.stdout:19: done (13 left) 2021-03-24T21:54:36.519 INFO:tasks.rados.rados.0.smithi125.stdout:20: done (12 left) 2021-03-24T21:54:36.519 INFO:tasks.rados.rados.0.smithi125.stdout:21: done (11 left) 2021-03-24T21:54:36.519 INFO:tasks.rados.rados.0.smithi125.stdout:22: done (10 left) 2021-03-24T21:54:36.520 INFO:tasks.rados.rados.0.smithi125.stdout:23: done (9 left) 2021-03-24T21:54:36.520 INFO:tasks.rados.rados.0.smithi125.stdout:33: write initial oid 33 2021-03-24T21:54:36.520 INFO:tasks.rados.rados.0.smithi125.stdout:33: seq_num 32 ranges {537824=473380,1806012=625998} 2021-03-24T21:54:36.521 INFO:tasks.rados.rados.0.smithi125.stdout:33: writing smithi12556470-33 from 537824 to 1011204 tid 1 2021-03-24T21:54:36.521 INFO:tasks.rados.rados.0.smithi125.stdout:33: writing smithi12556470-33 from 1806012 to 2432010 tid 2 2021-03-24T21:54:36.522 INFO:tasks.rados.rados.0.smithi125.stdout:34: write initial oid 34 2021-03-24T21:54:36.522 INFO:tasks.rados.rados.0.smithi125.stdout:34: seq_num 33 ranges {554631=548629,1748141=486009} 2021-03-24T21:54:36.527 INFO:tasks.rados.rados.0.smithi125.stdout:34: writing smithi12556470-34 from 554631 to 1103260 tid 1 2021-03-24T21:54:36.534 INFO:tasks.rados.rados.0.smithi125.stdout:34: writing smithi12556470-34 from 1748141 to 2234150 tid 2 2021-03-24T21:54:36.534 INFO:tasks.rados.rados.0.smithi125.stdout:35: write initial oid 35 2021-03-24T21:54:36.534 INFO:tasks.rados.rados.0.smithi125.stdout:35: seq_num 34 ranges {571438=623878,1806623=629667} 2021-03-24T21:54:36.541 INFO:tasks.rados.rados.0.smithi125.stdout:35: writing smithi12556470-35 from 571438 to 1195316 tid 1 2021-03-24T21:54:36.548 INFO:tasks.rados.rados.0.smithi125.stdout:35: writing smithi12556470-35 from 1806623 to 2436290 tid 2 2021-03-24T21:54:36.549 INFO:tasks.rados.rados.0.smithi125.stdout:36: write initial oid 36 2021-03-24T21:54:36.549 INFO:tasks.rados.rados.0.smithi125.stdout:36: seq_num 35 ranges {588245=699127,1865105=489678} 2021-03-24T21:54:36.556 INFO:tasks.rados.rados.0.smithi125.stdout:36: writing smithi12556470-36 from 588245 to 1287372 tid 1 2021-03-24T21:54:36.563 INFO:tasks.rados.rados.0.smithi125.stdout:36: writing smithi12556470-36 from 1865105 to 2354783 tid 2 2021-03-24T21:54:36.563 INFO:tasks.rados.rados.0.smithi125.stdout:37: write initial oid 37 2021-03-24T21:54:36.563 INFO:tasks.rados.rados.0.smithi125.stdout:37: seq_num 36 ranges {605052=774376,1923587=633336} 2021-03-24T21:54:36.571 INFO:tasks.rados.rados.0.smithi125.stdout:37: writing smithi12556470-37 from 605052 to 1379428 tid 1 2021-03-24T21:54:36.578 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:36.579 INFO:tasks.rados.rados.0.smithi125.stdout:37: writing smithi12556470-37 from 1923587 to 2556923 tid 2 2021-03-24T21:54:36.579 INFO:tasks.rados.rados.0.smithi125.stdout:38: write initial oid 38 2021-03-24T21:54:36.579 INFO:tasks.rados.rados.0.smithi125.stdout:24: finishing write tid 2 to smithi12556470-24 2021-03-24T21:54:36.580 INFO:tasks.rados.rados.0.smithi125.stdout:38: seq_num 37 ranges {621859=449625,1865716=756143} 2021-03-24T21:54:36.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:36.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:54:36.585 INFO:tasks.rados.rados.0.smithi125.stdout:38: writing smithi12556470-38 from 621859 to 1071484 tid 1 2021-03-24T21:54:36.595 INFO:tasks.rados.rados.0.smithi125.stdout:38: writing smithi12556470-38 from 1865716 to 2621859 tid 2 2021-03-24T21:54:36.595 INFO:tasks.rados.rados.0.smithi125.stdout:39: write initial oid 39 2021-03-24T21:54:36.595 INFO:tasks.rados.rados.0.smithi125.stdout:39: seq_num 38 ranges {638666=524874,1924198=637005} 2021-03-24T21:54:36.601 INFO:tasks.rados.rados.0.smithi125.stdout:39: writing smithi12556470-39 from 638666 to 1163540 tid 1 2021-03-24T21:54:36.610 INFO:tasks.rados.rados.0.smithi125.stdout:39: writing smithi12556470-39 from 1924198 to 2561203 tid 2 2021-03-24T21:54:36.610 INFO:tasks.rados.rados.0.smithi125.stdout: waiting on 16 2021-03-24T21:54:36.611 INFO:tasks.rados.rados.0.smithi125.stdout:24: finishing write tid 3 to smithi12556470-24 2021-03-24T21:54:36.611 INFO:tasks.rados.rados.0.smithi125.stdout:24: finishing write tid 4 to smithi12556470-24 2021-03-24T21:54:36.611 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 24 v 9 (ObjNum 23 snap 0 seq_num 23) dirty exists 2021-03-24T21:54:36.611 INFO:tasks.rados.rados.0.smithi125.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2021-03-24T21:54:36.611 INFO:tasks.rados.rados.0.smithi125.stdout:25: finishing write tid 1 to smithi12556470-25 2021-03-24T21:54:36.612 INFO:tasks.rados.rados.0.smithi125.stdout:25: finishing write tid 2 to smithi12556470-25 2021-03-24T21:54:36.612 INFO:tasks.rados.rados.0.smithi125.stdout:25: finishing write tid 3 to smithi12556470-25 2021-03-24T21:54:36.612 INFO:tasks.rados.rados.0.smithi125.stdout:24: done (15 left) 2021-03-24T21:54:36.612 INFO:tasks.rados.rados.0.smithi125.stdout:40: write initial oid 40 2021-03-24T21:54:36.613 INFO:tasks.rados.rados.0.smithi125.stdout:40: seq_num 39 ranges {655473=716476,2099033=556440} 2021-03-24T21:54:36.616 INFO:tasks.rados.rados.0.smithi125.stdout:40: writing smithi12556470-40 from 655473 to 1371949 tid 1 2021-03-24T21:54:36.624 INFO:tasks.rados.rados.0.smithi125.stdout:40: writing smithi12556470-40 from 2099033 to 2655473 tid 2 2021-03-24T21:54:36.624 INFO:tasks.rados.rados.0.smithi125.stdout: waiting on 16 2021-03-24T21:54:36.625 INFO:tasks.rados.rados.0.smithi125.stdout:25: finishing write tid 4 to smithi12556470-25 2021-03-24T21:54:36.625 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 25 v 6 (ObjNum 24 snap 0 seq_num 24) dirty exists 2021-03-24T21:54:36.625 INFO:tasks.rados.rados.0.smithi125.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2021-03-24T21:54:36.625 INFO:tasks.rados.rados.0.smithi125.stdout:26: finishing write tid 1 to smithi12556470-26 2021-03-24T21:54:36.625 INFO:tasks.rados.rados.0.smithi125.stdout:26: finishing write tid 2 to smithi12556470-26 2021-03-24T21:54:36.626 INFO:tasks.rados.rados.0.smithi125.stdout:26: finishing write tid 3 to smithi12556470-26 2021-03-24T21:54:36.626 INFO:tasks.rados.rados.0.smithi125.stdout:26: finishing write tid 4 to smithi12556470-26 2021-03-24T21:54:36.626 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 26 v 3 (ObjNum 25 snap 0 seq_num 25) dirty exists 2021-03-24T21:54:36.626 INFO:tasks.rados.rados.0.smithi125.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2021-03-24T21:54:36.627 INFO:tasks.rados.rados.0.smithi125.stdout:27: finishing write tid 1 to smithi12556470-27 2021-03-24T21:54:36.627 INFO:tasks.rados.rados.0.smithi125.stdout:25: done (15 left) 2021-03-24T21:54:36.627 INFO:tasks.rados.rados.0.smithi125.stdout:26: done (14 left) 2021-03-24T21:54:36.627 INFO:tasks.rados.rados.0.smithi125.stdout:41: write initial oid 41 2021-03-24T21:54:36.628 INFO:tasks.rados.rados.0.smithi125.stdout:41: seq_num 40 ranges {672280=791725,2157515=514765} 2021-03-24T21:54:36.634 INFO:tasks.rados.rados.0.smithi125.stdout:41: writing smithi12556470-41 from 672280 to 1464005 tid 1 2021-03-24T21:54:36.643 INFO:tasks.rados.rados.0.smithi125.stdout:41: writing smithi12556470-41 from 2157515 to 2672280 tid 2 2021-03-24T21:54:36.643 INFO:tasks.rados.rados.0.smithi125.stdout:42: write initial oid 42 2021-03-24T21:54:36.644 INFO:tasks.rados.rados.0.smithi125.stdout:42: seq_num 41 ranges {689087=466974,1699644=784332} 2021-03-24T21:54:36.650 INFO:tasks.rados.rados.0.smithi125.stdout:42: writing smithi12556470-42 from 689087 to 1156061 tid 1 2021-03-24T21:54:36.658 INFO:tasks.rados.rados.0.smithi125.stdout:42: writing smithi12556470-42 from 1699644 to 2483976 tid 2 2021-03-24T21:54:36.658 INFO:tasks.rados.rados.0.smithi125.stdout: waiting on 16 2021-03-24T21:54:36.658 INFO:tasks.rados.rados.0.smithi125.stdout:27: finishing write tid 2 to smithi12556470-27 2021-03-24T21:54:36.659 INFO:tasks.rados.rados.0.smithi125.stdout:27: finishing write tid 3 to smithi12556470-27 2021-03-24T21:54:36.659 INFO:tasks.rados.rados.0.smithi125.stdout:27: finishing write tid 4 to smithi12556470-27 2021-03-24T21:54:36.659 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 27 v 12 (ObjNum 26 snap 0 seq_num 26) dirty exists 2021-03-24T21:54:36.659 INFO:tasks.rados.rados.0.smithi125.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2021-03-24T21:54:36.659 INFO:tasks.rados.rados.0.smithi125.stdout:28: finishing write tid 1 to smithi12556470-28 2021-03-24T21:54:36.660 INFO:tasks.rados.rados.0.smithi125.stdout:28: finishing write tid 2 to smithi12556470-28 2021-03-24T21:54:36.660 INFO:tasks.rados.rados.0.smithi125.stdout:27: done (15 left) 2021-03-24T21:54:36.660 INFO:tasks.rados.rados.0.smithi125.stdout:43: write initial oid 43 2021-03-24T21:54:36.661 INFO:tasks.rados.rados.0.smithi125.stdout:43: seq_num 42 ranges {705894=542223,1758126=644343} 2021-03-24T21:54:36.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:36 smithi125 conmon[19936]: debug 2021-03-24T21:54:36.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:36 smithi125 conmon[19936]: 2021-03-24T21:54:36.539+0000 7f2118128700 -1 mon.a@0(leader).osd e107 definitely_dead 0 2021-03-24T21:54:36.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:36 smithi125 conmon[19936]: 2021-03-24T21:54:36.669 INFO:tasks.rados.rados.0.smithi125.stdout:43: writing smithi12556470-43 from 705894 to 1248117 tid 1 2021-03-24T21:54:36.677 INFO:tasks.rados.rados.0.smithi125.stdout:43: writing smithi12556470-43 from 1758126 to 2402469 tid 2 2021-03-24T21:54:36.677 INFO:tasks.rados.rados.0.smithi125.stdout: waiting on 16 2021-03-24T21:54:36.678 INFO:tasks.rados.rados.0.smithi125.stdout:28: finishing write tid 3 to smithi12556470-28 2021-03-24T21:54:36.678 INFO:tasks.rados.rados.0.smithi125.stdout:28: finishing write tid 4 to smithi12556470-28 2021-03-24T21:54:36.678 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 28 v 9 (ObjNum 27 snap 0 seq_num 27) dirty exists 2021-03-24T21:54:36.679 INFO:tasks.rados.rados.0.smithi125.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2021-03-24T21:54:36.679 INFO:tasks.rados.rados.0.smithi125.stdout:29: finishing write tid 1 to smithi12556470-29 2021-03-24T21:54:36.679 INFO:tasks.rados.rados.0.smithi125.stdout:28: done (15 left) 2021-03-24T21:54:36.680 INFO:tasks.rados.rados.0.smithi125.stdout:44: write initial oid 44 2021-03-24T21:54:36.680 INFO:tasks.rados.rados.0.smithi125.stdout:44: seq_num 43 ranges {722701=617472,1816608=788001} 2021-03-24T21:54:36.683 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:54:36.684 INFO:tasks.rados.rados.0.smithi125.stdout:44: writing smithi12556470-44 from 722701 to 1340173 tid 1 2021-03-24T21:54:36.691 INFO:tasks.rados.rados.0.smithi125.stdout:44: writing smithi12556470-44 from 1816608 to 2604609 tid 2 2021-03-24T21:54:36.691 INFO:tasks.rados.rados.0.smithi125.stdout: waiting on 16 2021-03-24T21:54:36.692 INFO:tasks.rados.rados.0.smithi125.stdout:29: finishing write tid 2 to smithi12556470-29 2021-03-24T21:54:36.692 INFO:tasks.rados.rados.0.smithi125.stdout:29: finishing write tid 3 to smithi12556470-29 2021-03-24T21:54:36.692 INFO:tasks.rados.rados.0.smithi125.stdout:29: finishing write tid 4 to smithi12556470-29 2021-03-24T21:54:36.692 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 29 v 6 (ObjNum 28 snap 0 seq_num 28) dirty exists 2021-03-24T21:54:36.692 INFO:tasks.rados.rados.0.smithi125.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2021-03-24T21:54:36.693 INFO:tasks.rados.rados.0.smithi125.stdout:30: finishing write tid 1 to smithi12556470-30 2021-03-24T21:54:36.693 INFO:tasks.rados.rados.0.smithi125.stdout:29: done (15 left) 2021-03-24T21:54:36.693 INFO:tasks.rados.rados.0.smithi125.stdout:45: write initial oid 45 2021-03-24T21:54:36.693 INFO:tasks.rados.rados.0.smithi125.stdout:45: seq_num 44 ranges {739508=692721,1875090=648012} 2021-03-24T21:54:36.699 INFO:tasks.rados.rados.0.smithi125.stdout:45: writing smithi12556470-45 from 739508 to 1432229 tid 1 2021-03-24T21:54:36.706 INFO:tasks.rados.rados.0.smithi125.stdout:45: writing smithi12556470-45 from 1875090 to 2523102 tid 2 2021-03-24T21:54:36.707 INFO:tasks.rados.rados.0.smithi125.stdout: waiting on 16 2021-03-24T21:54:36.707 INFO:tasks.rados.rados.0.smithi125.stdout:30: finishing write tid 2 to smithi12556470-30 2021-03-24T21:54:36.707 INFO:tasks.rados.rados.0.smithi125.stdout:30: finishing write tid 3 to smithi12556470-30 2021-03-24T21:54:36.707 INFO:tasks.rados.rados.0.smithi125.stdout:30: finishing write tid 4 to smithi12556470-30 2021-03-24T21:54:36.708 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 30 v 6 (ObjNum 29 snap 0 seq_num 29) dirty exists 2021-03-24T21:54:36.708 INFO:tasks.rados.rados.0.smithi125.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2021-03-24T21:54:36.708 INFO:tasks.rados.rados.0.smithi125.stdout:31: finishing write tid 1 to smithi12556470-31 2021-03-24T21:54:36.708 INFO:tasks.rados.rados.0.smithi125.stdout:31: finishing write tid 2 to smithi12556470-31 2021-03-24T21:54:36.709 INFO:tasks.rados.rados.0.smithi125.stdout:31: finishing write tid 3 to smithi12556470-31 2021-03-24T21:54:36.709 INFO:tasks.rados.rados.0.smithi125.stdout:31: finishing write tid 4 to smithi12556470-31 2021-03-24T21:54:36.709 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 31 v 9 (ObjNum 30 snap 0 seq_num 30) dirty exists 2021-03-24T21:54:36.710 INFO:tasks.rados.rados.0.smithi125.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2021-03-24T21:54:36.710 INFO:tasks.rados.rados.0.smithi125.stdout:32: finishing write tid 1 to smithi12556470-32 2021-03-24T21:54:36.710 INFO:tasks.rados.rados.0.smithi125.stdout:30: done (15 left) 2021-03-24T21:54:36.710 INFO:tasks.rados.rados.0.smithi125.stdout:31: done (14 left) 2021-03-24T21:54:36.711 INFO:tasks.rados.rados.0.smithi125.stdout:46: write initial oid 46 2021-03-24T21:54:36.711 INFO:tasks.rados.rados.0.smithi125.stdout:46: seq_num 45 ranges {756315=767970,1933572=791670} 2021-03-24T21:54:36.714 INFO:tasks.rados.rados.0.smithi125.stdout:46: writing smithi12556470-46 from 756315 to 1524285 tid 1 2021-03-24T21:54:36.721 INFO:tasks.rados.rados.0.smithi125.stdout:46: writing smithi12556470-46 from 1933572 to 2725242 tid 2 2021-03-24T21:54:36.721 INFO:tasks.rados.rados.0.smithi125.stdout:47: write initial oid 47 2021-03-24T21:54:36.722 INFO:tasks.rados.rados.0.smithi125.stdout:47: seq_num 46 ranges {773122=559572,1992054=651681} 2021-03-24T21:54:36.728 INFO:tasks.rados.rados.0.smithi125.stdout:47: writing smithi12556470-47 from 773122 to 1332694 tid 1 2021-03-24T21:54:36.736 INFO:tasks.rados.rados.0.smithi125.stdout:47: writing smithi12556470-47 from 1992054 to 2643735 tid 2 2021-03-24T21:54:36.736 INFO:tasks.rados.rados.0.smithi125.stdout: waiting on 16 2021-03-24T21:54:36.737 INFO:tasks.rados.rados.0.smithi125.stdout:32: finishing write tid 2 to smithi12556470-32 2021-03-24T21:54:36.737 INFO:tasks.rados.rados.0.smithi125.stdout:32: finishing write tid 3 to smithi12556470-32 2021-03-24T21:54:36.737 INFO:tasks.rados.rados.0.smithi125.stdout:32: finishing write tid 4 to smithi12556470-32 2021-03-24T21:54:36.737 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 32 v 12 (ObjNum 31 snap 0 seq_num 31) dirty exists 2021-03-24T21:54:36.738 INFO:tasks.rados.rados.0.smithi125.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2021-03-24T21:54:36.738 INFO:tasks.rados.rados.0.smithi125.stdout:33: finishing write tid 1 to smithi12556470-33 2021-03-24T21:54:36.738 INFO:tasks.rados.rados.0.smithi125.stdout:32: done (15 left) 2021-03-24T21:54:36.738 INFO:tasks.rados.rados.0.smithi125.stdout:48: write initial oid 48 2021-03-24T21:54:36.739 INFO:tasks.rados.rados.0.smithi125.stdout:48: seq_num 47 ranges {789929=634821,2050536=739393} 2021-03-24T21:54:36.745 INFO:tasks.rados.rados.0.smithi125.stdout:48: writing smithi12556470-48 from 789929 to 1424750 tid 1 2021-03-24T21:54:36.749 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:54:36.753 INFO:tasks.rados.rados.0.smithi125.stdout:48: writing smithi12556470-48 from 2050536 to 2789929 tid 2 2021-03-24T21:54:36.754 INFO:tasks.rados.rados.0.smithi125.stdout: waiting on 16 2021-03-24T21:54:36.754 INFO:tasks.rados.rados.0.smithi125.stdout:33: finishing write tid 2 to smithi12556470-33 2021-03-24T21:54:36.755 INFO:tasks.rados.rados.0.smithi125.stdout:33: finishing write tid 3 to smithi12556470-33 2021-03-24T21:54:36.755 INFO:tasks.rados.rados.0.smithi125.stdout:33: finishing write tid 4 to smithi12556470-33 2021-03-24T21:54:36.755 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 33 v 12 (ObjNum 32 snap 0 seq_num 32) dirty exists 2021-03-24T21:54:36.756 INFO:tasks.rados.rados.0.smithi125.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2021-03-24T21:54:36.756 INFO:tasks.rados.rados.0.smithi125.stdout:33: done (15 left) 2021-03-24T21:54:36.757 INFO:tasks.rados.rados.0.smithi125.stdout:49: write initial oid 49 2021-03-24T21:54:36.757 INFO:tasks.rados.rados.0.smithi125.stdout:49: seq_num 48 ranges {406736=710070,1709018=655350} 2021-03-24T21:54:36.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:36.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:54:36.760 INFO:tasks.rados.rados.0.smithi125.stdout:49: writing smithi12556470-49 from 406736 to 1116806 tid 1 2021-03-24T21:54:36.766 INFO:tasks.rados.rados.0.smithi125.stdout:49: writing smithi12556470-49 from 1709018 to 2364368 tid 2 2021-03-24T21:54:36.767 INFO:tasks.rados.rados.0.smithi125.stdout: waiting on 16 2021-03-24T21:54:36.767 INFO:tasks.rados.rados.0.smithi125.stdout:34: finishing write tid 1 to smithi12556470-34 2021-03-24T21:54:36.767 INFO:tasks.rados.rados.0.smithi125.stdout:34: finishing write tid 2 to smithi12556470-34 2021-03-24T21:54:36.767 INFO:tasks.rados.rados.0.smithi125.stdout:34: finishing write tid 3 to smithi12556470-34 2021-03-24T21:54:36.768 INFO:tasks.rados.rados.0.smithi125.stdout:34: finishing write tid 4 to smithi12556470-34 2021-03-24T21:54:36.768 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 34 v 15 (ObjNum 33 snap 0 seq_num 33) dirty exists 2021-03-24T21:54:36.768 INFO:tasks.rados.rados.0.smithi125.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2021-03-24T21:54:36.768 INFO:tasks.rados.rados.0.smithi125.stdout:35: finishing write tid 1 to smithi12556470-35 2021-03-24T21:54:36.769 INFO:tasks.rados.rados.0.smithi125.stdout:35: finishing write tid 2 to smithi12556470-35 2021-03-24T21:54:36.769 INFO:tasks.rados.rados.0.smithi125.stdout:35: finishing write tid 3 to smithi12556470-35 2021-03-24T21:54:36.769 INFO:tasks.rados.rados.0.smithi125.stdout:35: finishing write tid 4 to smithi12556470-35 2021-03-24T21:54:36.769 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 35 v 3 (ObjNum 34 snap 0 seq_num 34) dirty exists 2021-03-24T21:54:36.769 INFO:tasks.rados.rados.0.smithi125.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2021-03-24T21:54:36.770 INFO:tasks.rados.rados.0.smithi125.stdout:34: done (15 left) 2021-03-24T21:54:36.770 INFO:tasks.rados.rados.0.smithi125.stdout:35: done (14 left) 2021-03-24T21:54:36.770 INFO:tasks.rados.rados.0.smithi125.stdout:50: write initial oid 50 2021-03-24T21:54:36.770 INFO:tasks.rados.rados.0.smithi125.stdout:50: seq_num 49 ranges {423543=785319,1767500=799008} 2021-03-24T21:54:36.772 INFO:tasks.rados.rados.0.smithi125.stdout:50: writing smithi12556470-50 from 423543 to 1208862 tid 1 2021-03-24T21:54:36.781 INFO:tasks.rados.rados.0.smithi125.stdout:50: writing smithi12556470-50 from 1767500 to 2566508 tid 2 2021-03-24T21:54:36.781 INFO:tasks.rados.rados.0.smithi125.stdout:51: copy_from oid 2 from oid 11 current snap is 0 2021-03-24T21:54:36.782 INFO:tasks.rados.rados.0.smithi125.stdout: waiting on 16 2021-03-24T21:54:36.782 INFO:tasks.rados.rados.0.smithi125.stdout:36: finishing write tid 1 to smithi12556470-36 2021-03-24T21:54:36.782 INFO:tasks.rados.rados.0.smithi125.stdout:36: finishing write tid 2 to smithi12556470-36 2021-03-24T21:54:36.908 INFO:tasks.rados.rados.0.smithi125.stdout:36: finishing write tid 3 to smithi12556470-36 2021-03-24T21:54:36.909 INFO:tasks.rados.rados.0.smithi125.stdout:36: finishing write tid 4 to smithi12556470-36 2021-03-24T21:54:36.909 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 36 v 12 (ObjNum 35 snap 0 seq_num 35) dirty exists 2021-03-24T21:54:36.909 INFO:tasks.rados.rados.0.smithi125.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2021-03-24T21:54:36.910 INFO:tasks.rados.rados.0.smithi125.stdout:37: finishing write tid 1 to smithi12556470-37 2021-03-24T21:54:36.910 INFO:tasks.rados.rados.0.smithi125.stdout:37: finishing write tid 2 to smithi12556470-37 2021-03-24T21:54:36.910 INFO:tasks.rados.rados.0.smithi125.stdout:37: finishing write tid 3 to smithi12556470-37 2021-03-24T21:54:36.910 INFO:tasks.rados.rados.0.smithi125.stdout:37: finishing write tid 4 to smithi12556470-37 2021-03-24T21:54:36.911 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 37 v 3 (ObjNum 36 snap 0 seq_num 36) dirty exists 2021-03-24T21:54:36.911 INFO:tasks.rados.rados.0.smithi125.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2021-03-24T21:54:36.911 INFO:tasks.rados.rados.0.smithi125.stdout:36: done (15 left) 2021-03-24T21:54:36.911 INFO:tasks.rados.rados.0.smithi125.stdout:37: done (14 left) 2021-03-24T21:54:36.912 INFO:tasks.rados.rados.0.smithi125.stdout:52: write oid 18 current snap is 0 2021-03-24T21:54:36.912 INFO:tasks.rados.rados.0.smithi125.stdout:52: seq_num 50 ranges {440350=460568,1309629=542666,2323973=516377} 2021-03-24T21:54:36.912 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:36.915 INFO:tasks.rados.rados.0.smithi125.stdout:52: writing smithi12556470-18 from 440350 to 900918 tid 1 2021-03-24T21:54:36.916 INFO:tasks.rados.rados.0.smithi125.stdout:52: writing smithi12556470-18 from 1309629 to 1852295 tid 2 2021-03-24T21:54:36.916 INFO:tasks.rados.rados.0.smithi125.stdout:52: writing smithi12556470-18 from 2323973 to 2840350 tid 3 2021-03-24T21:54:36.916 INFO:tasks.rados.rados.0.smithi125.stdout:53: rollback oid 30 current snap is 0 2021-03-24T21:54:36.916 INFO:tasks.rados.rados.0.smithi125.stdout:53: done (15 left) 2021-03-24T21:54:36.917 INFO:tasks.rados.rados.0.smithi125.stdout:54: write (excl) oid 22 current snap is 0 2021-03-24T21:54:36.917 INFO:tasks.rados.rados.0.smithi125.stdout:38: finishing write tid 1 to smithi12556470-38 2021-03-24T21:54:36.917 INFO:tasks.rados.rados.0.smithi125.stdout:54: seq_num 51 ranges {457157=535817,1768111=402677} 2021-03-24T21:54:36.918 INFO:tasks.rados.rados.0.smithi125.stdout:54: writing smithi12556470-22 from 457157 to 992974 tid 1 2021-03-24T21:54:36.918 INFO:tasks.rados.rados.0.smithi125.stdout:54: writing smithi12556470-22 from 1768111 to 2170788 tid 2 2021-03-24T21:54:36.918 INFO:tasks.rados.rados.0.smithi125.stdout: waiting on 16 2021-03-24T21:54:36.918 INFO:tasks.rados.rados.0.smithi125.stdout:38: finishing write tid 2 to smithi12556470-38 2021-03-24T21:54:36.919 INFO:tasks.rados.rados.0.smithi125.stdout:38: finishing write tid 3 to smithi12556470-38 2021-03-24T21:54:36.919 INFO:tasks.rados.rados.0.smithi125.stdout:38: finishing write tid 4 to smithi12556470-38 2021-03-24T21:54:36.919 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 38 v 3 (ObjNum 37 snap 0 seq_num 37) dirty exists 2021-03-24T21:54:36.919 INFO:tasks.rados.rados.0.smithi125.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2021-03-24T21:54:36.919 INFO:tasks.rados.rados.0.smithi125.stdout:39: finishing write tid 1 to smithi12556470-39 2021-03-24T21:54:36.920 INFO:tasks.rados.rados.0.smithi125.stdout:38: done (15 left) 2021-03-24T21:54:36.920 INFO:tasks.rados.rados.0.smithi125.stdout:55: snap_create 2021-03-24T21:54:36.920 INFO:tasks.rados.rados.0.smithi125.stdout:39: finishing write tid 2 to smithi12556470-39 2021-03-24T21:54:36.920 INFO:tasks.rados.rados.0.smithi125.stdout:39: finishing write tid 3 to smithi12556470-39 2021-03-24T21:54:36.921 INFO:tasks.rados.rados.0.smithi125.stdout:39: finishing write tid 4 to smithi12556470-39 2021-03-24T21:54:36.921 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 39 v 15 (ObjNum 38 snap 0 seq_num 38) dirty exists 2021-03-24T21:54:36.921 INFO:tasks.rados.rados.0.smithi125.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2021-03-24T21:54:36.921 INFO:tasks.rados.rados.0.smithi125.stdout:40: finishing write tid 1 to smithi12556470-40 2021-03-24T21:54:36.921 INFO:tasks.rados.rados.0.smithi125.stdout:40: finishing write tid 2 to smithi12556470-40 2021-03-24T21:54:36.922 INFO:tasks.rados.rados.0.smithi125.stdout:40: finishing write tid 3 to smithi12556470-40 2021-03-24T21:54:36.922 INFO:tasks.rados.rados.0.smithi125.stdout:40: finishing write tid 4 to smithi12556470-40 2021-03-24T21:54:36.922 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 40 v 15 (ObjNum 39 snap 0 seq_num 39) dirty exists 2021-03-24T21:54:36.922 INFO:tasks.rados.rados.0.smithi125.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2021-03-24T21:54:36.923 INFO:tasks.rados.rados.0.smithi125.stdout:41: finishing write tid 1 to smithi12556470-41 2021-03-24T21:54:36.923 INFO:tasks.rados.rados.0.smithi125.stdout:41: finishing write tid 2 to smithi12556470-41 2021-03-24T21:54:36.923 INFO:tasks.rados.rados.0.smithi125.stdout:41: finishing write tid 3 to smithi12556470-41 2021-03-24T21:54:36.924 INFO:tasks.rados.rados.0.smithi125.stdout:41: finishing write tid 4 to smithi12556470-41 2021-03-24T21:54:36.924 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 41 v 12 (ObjNum 40 snap 0 seq_num 40) dirty exists 2021-03-24T21:54:36.924 INFO:tasks.rados.rados.0.smithi125.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2021-03-24T21:54:36.924 INFO:tasks.rados.rados.0.smithi125.stdout:42: finishing write tid 1 to smithi12556470-42 2021-03-24T21:54:36.924 INFO:tasks.rados.rados.0.smithi125.stdout:42: finishing write tid 2 to smithi12556470-42 2021-03-24T21:54:36.925 INFO:tasks.rados.rados.0.smithi125.stdout:42: finishing write tid 3 to smithi12556470-42 2021-03-24T21:54:36.925 INFO:tasks.rados.rados.0.smithi125.stdout:42: finishing write tid 4 to smithi12556470-42 2021-03-24T21:54:36.925 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 42 v 6 (ObjNum 41 snap 0 seq_num 41) dirty exists 2021-03-24T21:54:36.926 INFO:tasks.rados.rados.0.smithi125.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2021-03-24T21:54:36.926 INFO:tasks.rados.rados.0.smithi125.stdout:43: finishing write tid 1 to smithi12556470-43 2021-03-24T21:54:36.926 INFO:tasks.rados.rados.0.smithi125.stdout:43: finishing write tid 2 to smithi12556470-43 2021-03-24T21:54:36.926 INFO:tasks.rados.rados.0.smithi125.stdout:43: finishing write tid 3 to smithi12556470-43 2021-03-24T21:54:36.927 INFO:tasks.rados.rados.0.smithi125.stdout:43: finishing write tid 4 to smithi12556470-43 2021-03-24T21:54:36.927 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 43 v 18 (ObjNum 42 snap 0 seq_num 42) dirty exists 2021-03-24T21:54:36.927 INFO:tasks.rados.rados.0.smithi125.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2021-03-24T21:54:36.927 INFO:tasks.rados.rados.0.smithi125.stdout:44: finishing write tid 1 to smithi12556470-44 2021-03-24T21:54:36.927 INFO:tasks.rados.rados.0.smithi125.stdout:44: finishing write tid 2 to smithi12556470-44 2021-03-24T21:54:36.928 INFO:tasks.rados.rados.0.smithi125.stdout:44: finishing write tid 3 to smithi12556470-44 2021-03-24T21:54:36.928 INFO:tasks.rados.rados.0.smithi125.stdout:44: finishing write tid 4 to smithi12556470-44 2021-03-24T21:54:36.928 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 44 v 9 (ObjNum 43 snap 0 seq_num 43) dirty exists 2021-03-24T21:54:36.928 INFO:tasks.rados.rados.0.smithi125.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2021-03-24T21:54:36.928 INFO:tasks.rados.rados.0.smithi125.stdout:45: finishing write tid 1 to smithi12556470-45 2021-03-24T21:54:36.929 INFO:tasks.rados.rados.0.smithi125.stdout:45: finishing write tid 2 to smithi12556470-45 2021-03-24T21:54:36.929 INFO:tasks.rados.rados.0.smithi125.stdout:45: finishing write tid 3 to smithi12556470-45 2021-03-24T21:54:36.929 INFO:tasks.rados.rados.0.smithi125.stdout:45: finishing write tid 4 to smithi12556470-45 2021-03-24T21:54:36.929 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 45 v 12 (ObjNum 44 snap 0 seq_num 44) dirty exists 2021-03-24T21:54:36.929 INFO:tasks.rados.rados.0.smithi125.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2021-03-24T21:54:36.930 INFO:tasks.rados.rados.0.smithi125.stdout:46: finishing write tid 1 to smithi12556470-46 2021-03-24T21:54:36.930 INFO:tasks.rados.rados.0.smithi125.stdout:46: finishing write tid 2 to smithi12556470-46 2021-03-24T21:54:36.930 INFO:tasks.rados.rados.0.smithi125.stdout:46: finishing write tid 3 to smithi12556470-46 2021-03-24T21:54:36.930 INFO:tasks.rados.rados.0.smithi125.stdout:46: finishing write tid 4 to smithi12556470-46 2021-03-24T21:54:36.931 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 46 v 6 (ObjNum 45 snap 0 seq_num 45) dirty exists 2021-03-24T21:54:36.931 INFO:tasks.rados.rados.0.smithi125.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2021-03-24T21:54:36.931 INFO:tasks.rados.rados.0.smithi125.stdout:47: finishing write tid 1 to smithi12556470-47 2021-03-24T21:54:36.931 INFO:tasks.rados.rados.0.smithi125.stdout:47: finishing write tid 2 to smithi12556470-47 2021-03-24T21:54:36.931 INFO:tasks.rados.rados.0.smithi125.stdout:47: finishing write tid 3 to smithi12556470-47 2021-03-24T21:54:36.932 INFO:tasks.rados.rados.0.smithi125.stdout:47: finishing write tid 4 to smithi12556470-47 2021-03-24T21:54:36.932 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 47 v 3 (ObjNum 46 snap 0 seq_num 46) dirty exists 2021-03-24T21:54:36.932 INFO:tasks.rados.rados.0.smithi125.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2021-03-24T21:54:36.932 INFO:tasks.rados.rados.0.smithi125.stdout:48: finishing write tid 1 to smithi12556470-48 2021-03-24T21:54:36.933 INFO:tasks.rados.rados.0.smithi125.stdout:48: finishing write tid 2 to smithi12556470-48 2021-03-24T21:54:36.933 INFO:tasks.rados.rados.0.smithi125.stdout:48: finishing write tid 3 to smithi12556470-48 2021-03-24T21:54:36.933 INFO:tasks.rados.rados.0.smithi125.stdout:48: finishing write tid 4 to smithi12556470-48 2021-03-24T21:54:36.933 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 48 v 15 (ObjNum 47 snap 0 seq_num 47) dirty exists 2021-03-24T21:54:36.934 INFO:tasks.rados.rados.0.smithi125.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2021-03-24T21:54:36.934 INFO:tasks.rados.rados.0.smithi125.stdout:49: finishing write tid 1 to smithi12556470-49 2021-03-24T21:54:36.934 INFO:tasks.rados.rados.0.smithi125.stdout:49: finishing write tid 2 to smithi12556470-49 2021-03-24T21:54:36.935 INFO:tasks.rados.rados.0.smithi125.stdout:49: finishing write tid 3 to smithi12556470-49 2021-03-24T21:54:36.935 INFO:tasks.rados.rados.0.smithi125.stdout:49: finishing write tid 4 to smithi12556470-49 2021-03-24T21:54:36.935 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 49 v 18 (ObjNum 48 snap 0 seq_num 48) dirty exists 2021-03-24T21:54:36.935 INFO:tasks.rados.rados.0.smithi125.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2021-03-24T21:54:36.935 INFO:tasks.rados.rados.0.smithi125.stdout:50: finishing write tid 1 to smithi12556470-50 2021-03-24T21:54:36.936 INFO:tasks.rados.rados.0.smithi125.stdout:50: finishing write tid 2 to smithi12556470-50 2021-03-24T21:54:36.936 INFO:tasks.rados.rados.0.smithi125.stdout:50: finishing write tid 3 to smithi12556470-50 2021-03-24T21:54:36.936 INFO:tasks.rados.rados.0.smithi125.stdout:50: finishing write tid 4 to smithi12556470-50 2021-03-24T21:54:36.936 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 50 v 12 (ObjNum 49 snap 0 seq_num 49) dirty exists 2021-03-24T21:54:36.937 INFO:tasks.rados.rados.0.smithi125.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2021-03-24T21:54:36.937 INFO:tasks.rados.rados.0.smithi125.stdout:52: finishing write tid 1 to smithi12556470-18 2021-03-24T21:54:36.937 INFO:tasks.rados.rados.0.smithi125.stdout:51: finishing copy_from to smithi12556470-2 2021-03-24T21:54:36.937 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 2 v 9 (ObjNum 10 snap 0 seq_num 10) dirty exists 2021-03-24T21:54:36.937 INFO:tasks.rados.rados.0.smithi125.stdout:51: finishing copy_from racing read to smithi12556470-2 2021-03-24T21:54:36.938 INFO:tasks.rados.rados.0.smithi125.stdout:52: finishing write tid 2 to smithi12556470-18 2021-03-24T21:54:36.938 INFO:tasks.rados.rados.0.smithi125.stdout:52: finishing write tid 3 to smithi12556470-18 2021-03-24T21:54:36.938 INFO:tasks.rados.rados.0.smithi125.stdout:52: finishing write tid 4 to smithi12556470-18 2021-03-24T21:54:36.938 INFO:tasks.rados.rados.0.smithi125.stdout:52: finishing write tid 5 to smithi12556470-18 2021-03-24T21:54:36.938 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 18 v 10 (ObjNum 50 snap 0 seq_num 50) dirty exists 2021-03-24T21:54:36.939 INFO:tasks.rados.rados.0.smithi125.stdout:52: left oid 18 (ObjNum 50 snap 0 seq_num 50) 2021-03-24T21:54:36.939 INFO:tasks.rados.rados.0.smithi125.stdout:54: finishing write tid 1 to smithi12556470-22 2021-03-24T21:54:36.939 INFO:tasks.rados.rados.0.smithi125.stdout:54: finishing write tid 2 to smithi12556470-22 2021-03-24T21:54:36.939 INFO:tasks.rados.rados.0.smithi125.stdout:54: finishing write tid 3 to smithi12556470-22 2021-03-24T21:54:36.939 INFO:tasks.rados.rados.0.smithi125.stdout:54: finishing write tid 4 to smithi12556470-22 2021-03-24T21:54:36.940 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 22 v 15 (ObjNum 51 snap 0 seq_num 51) dirty exists 2021-03-24T21:54:36.940 INFO:tasks.rados.rados.0.smithi125.stdout:54: left oid 22 (ObjNum 51 snap 0 seq_num 51) 2021-03-24T21:54:36.994 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:36.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:37.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:54:37.086 INFO:tasks.rados.rados.0.smithi125.stdout:39: done (15 left) 2021-03-24T21:54:37.087 INFO:tasks.rados.rados.0.smithi125.stdout:40: done (14 left) 2021-03-24T21:54:37.087 INFO:tasks.rados.rados.0.smithi125.stdout:41: done (13 left) 2021-03-24T21:54:37.087 INFO:tasks.rados.rados.0.smithi125.stdout:42: done (12 left) 2021-03-24T21:54:37.087 INFO:tasks.rados.rados.0.smithi125.stdout:43: done (11 left) 2021-03-24T21:54:37.088 INFO:tasks.rados.rados.0.smithi125.stdout:44: done (10 left) 2021-03-24T21:54:37.088 INFO:tasks.rados.rados.0.smithi125.stdout:45: done (9 left) 2021-03-24T21:54:37.088 INFO:tasks.rados.rados.0.smithi125.stdout:46: done (8 left) 2021-03-24T21:54:37.088 INFO:tasks.rados.rados.0.smithi125.stdout:47: done (7 left) 2021-03-24T21:54:37.089 INFO:tasks.rados.rados.0.smithi125.stdout:48: done (6 left) 2021-03-24T21:54:37.089 INFO:tasks.rados.rados.0.smithi125.stdout:49: done (5 left) 2021-03-24T21:54:37.089 INFO:tasks.rados.rados.0.smithi125.stdout:50: done (4 left) 2021-03-24T21:54:37.089 INFO:tasks.rados.rados.0.smithi125.stdout:51: done (3 left) 2021-03-24T21:54:37.089 INFO:tasks.rados.rados.0.smithi125.stdout:52: done (2 left) 2021-03-24T21:54:37.090 INFO:tasks.rados.rados.0.smithi125.stdout:54: done (1 left) 2021-03-24T21:54:37.090 INFO:tasks.rados.rados.0.smithi125.stdout:55: done (0 left) 2021-03-24T21:54:37.090 INFO:tasks.rados.rados.0.smithi125.stdout:56: delete oid 34 current snap is 1 2021-03-24T21:54:37.101 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:37.104 INFO:teuthology.orchestra.run.smithi125.stderr:marked out osd.1. 2021-03-24T21:54:37.121 INFO:tasks.rados.rados.0.smithi125.stdout:56: done (0 left) 2021-03-24T21:54:37.121 INFO:tasks.rados.rados.0.smithi125.stdout:57: snap_create 2021-03-24T21:54:37.148 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:37.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:37.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:54:37.228 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:37 smithi125 conmon[19936]: audit 2021-03-24T21 2021-03-24T21:54:37.229 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:37 smithi125 conmon[19936]: :54:36.074885+0000 mon.a (mon.0) 641 : audit [INF] from='client.? v1:172.21.15.125:0/2843332919' entity='client.0' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]': finished 2021-03-24T21:54:37.229 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:37 smithi125 conmon[19936]: cluster 2021-03-24T21:54 2021-03-24T21:54:37.229 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:37 smithi125 conmon[19936]: :36.075000+0000 mon.a (mon.0) 642 : cluster [DBG] osdmap e107: 8 total, 8 up, 7 in 2021-03-24T21:54:37.230 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:37 smithi125 conmon[19936]: audit 2021-03-24T21 2021-03-24T21:54:37.230 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:37 smithi125 conmon[19936]: :54:36.542058+0000 mon.c (mon.2) 2021-03-24T21:54:37.230 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:37 smithi125 conmon[19936]: 24 : audit [INF] from='client.? v1:172.21.15.125:0/603975034' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2021-03-24T21:54:37.230 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:37 smithi125 conmon[19936]: audit 2021-03-24T21 2021-03-24T21:54:37.231 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:37 smithi125 conmon[19936]: :54:36.542543+0000 mon.a (mon. 2021-03-24T21:54:37.231 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:37 smithi125 conmon[19936]: 0) 643 : audit [INF] from='client.34685 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2021-03-24T21:54:37.231 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:37 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:54:37.231 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:37 smithi125 conmon[19936]: -03-24T21:54:36. 2021-03-24T21:54:37.232 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:37 smithi125 conmon[19936]: 542684+0000 mon.a (mon.0) 644 : 2021-03-24T21:54:37.232 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:37 smithi125 conmon[19936]: cluster [INF] Client client.admin marked osd.1 out, while it was still marked up 2021-03-24T21:54:37.232 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:37 smithi125 conmon[24927]: audit 2021-03-24T21:54:36.074885+0000 mon.a (mon.0) 641 : audit [INF] from='client.? v1:172.21.15.125:0/2843332919' entity='client.0' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]': finished 2021-03-24T21:54:37.233 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:37 smithi125 conmon[24927]: 2021-03-24T21:54:37.233 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:37 smithi125 conmon[24927]: cluster 2021-03-24T21:54:36.075000+0000 mon.a (mon.0) 642 : cluster [DBG] osdmap e107: 8 total, 8 up, 7 in 2021-03-24T21:54:37.233 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:37 smithi125 conmon[24927]: audit 2021-03-24T21:54:36.542058+0000 2021-03-24T21:54:37.234 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:37 smithi125 conmon[24927]: mon.c (mon.2) 24 : audit [INF] from='client.? v1:172.21.15.125:0/603975034' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2021-03-24T21:54:37.234 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:37 smithi125 conmon[24927]: audit 2021-03-24T21:54:36.542543+0000 mon.a (mon.0) 643 : audit [INF] from='client.34685 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2021-03-24T21:54:37.234 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:37 smithi125 conmon[24927]: cluster 2021-03-24T21:54:36.542684+0000 mon.a (mon.0) 644 : cluster [INF] Client client.admin marked osd.1 out, while it was still marked up 2021-03-24T21:54:37.253 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:37.302 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:37.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:37.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:54:37.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:37 smithi142 conmon[20125]: audit 2021-03-24T21:54:37.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:37 smithi142 conmon[20125]: 2021-03-24T21:54:37.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:37 smithi142 conmon[20125]: 2021-03-24T21:54 2021-03-24T21:54:37.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:37 smithi142 conmon[20125]: :36.074885+0000 mon.a (mon.0) 641 : audit [INF] from='client.? v1:172.21.15.125:0/2843332919' entity='client.0' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]': finished 2021-03-24T21:54:37.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:37 smithi142 conmon[20125]: cluster 2021-03-24T21:54: 2021-03-24T21:54:37.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:37 smithi142 conmon[20125]: 36.075000+0000 mon.a (mon.0) 642 : cluster [DBG] 2021-03-24T21:54:37.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:37 smithi142 conmon[20125]: osdmap e107: 8 total, 8 up, 7 in 2021-03-24T21:54:37.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:37 smithi142 conmon[20125]: audit 2021 2021-03-24T21:54:37.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:37 smithi142 conmon[20125]: -03-24T21:54:36.542058+0000 mon.c 2021-03-24T21:54:37.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:37 smithi142 conmon[20125]: (mon.2) 24 : audit [INF] from='client.? v1:172.21.15.125:0/603975034' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2021-03-24T21:54:37.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:37 smithi142 conmon[20125]: 2021-03-24T21:54:37.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:37 smithi142 conmon[20125]: audit 2021-03-24 2021-03-24T21:54:37.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:37 smithi142 conmon[20125]: T21:54:36.542543+0000 mon.a ( 2021-03-24T21:54:37.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:37 smithi142 conmon[20125]: mon.0) 643 : audit [INF] from='client.34685 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2021-03-24T21:54:37.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:37 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:54:37.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:37 smithi142 conmon[20125]: 03-24T21:54:36.542684 2021-03-24T21:54:37.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:37 smithi142 conmon[20125]: +0000 mon.a (mon.0) 644 : cluster 2021-03-24T21:54:37.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:37 smithi142 conmon[20125]: [INF] Client client.admin marked osd.1 out, while it was still marked up 2021-03-24T21:54:37.408 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:37.410 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.1 dump_ops_in_flight 2021-03-24T21:54:37.465 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:37.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:37.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:54:37.571 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:37.725 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd unset noscrub 2021-03-24T21:54:37.779 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:37.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:37.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:54:37.886 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:37.947 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:37.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:37.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:54:38.054 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:38.100 INFO:tasks.rados.rados.0.smithi125.stdout:57: done (0 left) 2021-03-24T21:54:38.100 INFO:tasks.rados.rados.0.smithi125.stdout:58: write oid 33 current snap is 2 2021-03-24T21:54:38.100 INFO:tasks.rados.rados.0.smithi125.stdout:58: seq_num 53 ranges {490771=686315,1885075=406346} 2021-03-24T21:54:38.109 INFO:tasks.rados.rados.0.smithi125.stdout:58: writing smithi12556470-33 from 490771 to 1177086 tid 1 2021-03-24T21:54:38.118 INFO:tasks.rados.rados.0.smithi125.stdout:58: writing smithi12556470-33 from 1885075 to 2291421 tid 2 2021-03-24T21:54:38.119 INFO:tasks.rados.rados.0.smithi125.stdout:59: delete oid 3 current snap is 2 2021-03-24T21:54:38.142 INFO:tasks.rados.rados.0.smithi125.stdout:59: done (1 left) 2021-03-24T21:54:38.142 INFO:tasks.rados.rados.0.smithi125.stdout:60: write oid 12 current snap is 2 2021-03-24T21:54:38.143 INFO:tasks.rados.rados.0.smithi125.stdout:60: seq_num 55 ranges {524385=553166,1602039=410015,2577441=346944} 2021-03-24T21:54:38.149 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:38.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:38.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:54:38.157 INFO:tasks.rados.rados.0.smithi125.stdout:60: writing smithi12556470-12 from 524385 to 1077551 tid 1 2021-03-24T21:54:38.162 INFO:tasks.rados.rados.0.smithi125.stdout:60: writing smithi12556470-12 from 1602039 to 2012054 tid 2 2021-03-24T21:54:38.169 INFO:tasks.rados.rados.0.smithi125.stdout:60: writing smithi12556470-12 from 2577441 to 2924385 tid 3 2021-03-24T21:54:38.169 INFO:tasks.rados.rados.0.smithi125.stdout:61: rollback oid 47 current snap is 2 2021-03-24T21:54:38.169 INFO:tasks.rados.rados.0.smithi125.stdout:58: finishing write tid 1 to smithi12556470-33 2021-03-24T21:54:38.170 INFO:tasks.rados.rados.0.smithi125.stdout:58: finishing write tid 2 to smithi12556470-33 2021-03-24T21:54:38.170 INFO:tasks.rados.rados.0.smithi125.stdout:58: finishing write tid 3 to smithi12556470-33 2021-03-24T21:54:38.170 INFO:tasks.rados.rados.0.smithi125.stdout:58: finishing write tid 4 to smithi12556470-33 2021-03-24T21:54:38.170 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 33 v 19 (ObjNum 53 snap 2 seq_num 53) dirty exists 2021-03-24T21:54:38.171 INFO:tasks.rados.rados.0.smithi125.stdout:58: left oid 33 (ObjNum 53 snap 2 seq_num 53) 2021-03-24T21:54:38.171 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 47 to 1 2021-03-24T21:54:38.171 INFO:tasks.rados.rados.0.smithi125.stdout:58: done (2 left) 2021-03-24T21:54:38.171 INFO:tasks.rados.rados.0.smithi125.stdout:62: snap_remove snap 0 2021-03-24T21:54:38.208 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:54:38.208 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:54:38.222 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:54:38.258 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:38.307 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:38.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:38.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:54:38.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:38 smithi142 conmon[20125]: cluster 2021-03-24T21:54:36.708672+0000 mgr.y (mgr. 2021-03-24T21:54:38.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:38 smithi142 conmon[20125]: 14144) 311 : cluster [DBG] pgmap v323: 148 pgs: 148 active+clean; 2.0 MiB data, 63 MiB used, 626 GiB / 626 GiB avail; 1.1 KiB/s rd, 176 KiB/s wr, 2 op/s 2021-03-24T21:54:38.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:38 smithi142 conmon[20125]: audit 2021-03-24T21:54:37.085650+0000 mon.a (mon.0) 645 : audit [INF] from='client.34685 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2021-03-24T21:54:38.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:38 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:54:38.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:38 smithi142 conmon[20125]: T21:54:37.085795+0000 mon.a (mon.0) 646 : cluster [DBG] osdmap e108: 8 total, 8 up, 6 in 2021-03-24T21:54:38.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:38 smithi125 conmon[24927]: cluster 2021-03-24T21:54:36.708672+0000 2021-03-24T21:54:38.558 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:38 smithi125 conmon[24927]: mgr.y (mgr.14144) 311 : cluster [DBG] pgmap v323: 148 pgs: 148 active+clean; 2.0 MiB data, 63 MiB used, 626 GiB / 626 GiB avail; 1.1 KiB/s rd, 176 KiB/s wr, 2 op/s 2021-03-24T21:54:38.558 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:38 smithi125 conmon[24927]: audit 2021-03-24T21:54:37.085650+0000 mon.a (mon 2021-03-24T21:54:38.558 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:38 smithi125 conmon[24927]: .0) 645 : audit [INF] from='client.34685 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2021-03-24T21:54:38.559 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:38 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:54:38.559 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:38 smithi125 conmon[24927]: T21:54:37.085795+0000 mon.a (mon.0) 646 : cluster [DBG] 2021-03-24T21:54:38.559 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:38 smithi125 conmon[24927]: osdmap e108: 8 total, 8 up, 6 in 2021-03-24T21:54:38.560 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:38 smithi125 conmon[19936]: cluster 2021-03-24T21:54:38.560 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:38 smithi125 conmon[19936]: 2021-03-24T21:54:38.560 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:38 smithi125 conmon[19936]: 2021 2021-03-24T21:54:38.560 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:38 smithi125 conmon[19936]: - 2021-03-24T21:54:38.560 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:38 smithi125 conmon[19936]: 03 2021-03-24T21:54:38.561 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:38 smithi125 conmon[19936]: - 2021-03-24T21:54:38.561 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:38 smithi125 conmon[19936]: 24 2021-03-24T21:54:38.561 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:38 smithi125 conmon[19936]: T 2021-03-24T21:54:38.561 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:38 smithi125 conmon[19936]: 21 2021-03-24T21:54:38.562 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:38 smithi125 conmon[19936]: : 2021-03-24T21:54:38.562 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:38 smithi125 conmon[19936]: 54 2021-03-24T21:54:38.562 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:38 smithi125 conmon[19936]: : 2021-03-24T21:54:38.562 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:38 smithi125 conmon[19936]: 36 2021-03-24T21:54:38.563 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:38 smithi125 conmon[19936]: . 2021-03-24T21:54:38.563 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:38 smithi125 conmon[19936]: 708672+0000 mgr.y (mgr.14144) 2021-03-24T21:54:38.563 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:38 smithi125 conmon[19936]: 311 : cluster [DBG] pgmap v323: 148 pgs: 148 active+clean; 2.0 MiB data, 63 MiB used, 626 GiB / 626 GiB avail; 1.1 KiB/s rd, 176 KiB/s wr, 2 op/s 2021-03-24T21:54:38.563 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:38 smithi125 conmon[19936]: audit 2021-03 2021-03-24T21:54:38.563 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:38 smithi125 conmon[19936]: -24T21:54:37.085650+0000 mon.a (mon.0) 645 : audit 2021-03-24T21:54:38.564 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:38 smithi125 conmon[19936]: [INF] from='client.34685 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2021-03-24T21:54:38.564 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:38 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:54:38.564 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:38 smithi125 conmon[19936]: 24T21:54:37.085795+0000 mon.a (mon 2021-03-24T21:54:38.564 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:38 smithi125 conmon[19936]: .0) 646 : cluster [DBG] osdmap e108: 8 total, 8 up, 6 in 2021-03-24T21:54:38.565 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:38.615 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:38.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:38.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:54:38.722 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:38.769 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:38.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:38.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:54:38.874 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:38.925 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:38.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:38.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:54:39.032 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:39.081 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:39.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:39.200 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:39 smithi142 conmon[20125]: cluster 2021-03-24T21:54:39.201 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:39 smithi142 conmon[20125]: 2021-03-24T21: 2021-03-24T21:54:39.201 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:39 smithi142 conmon[20125]: 54:37.757808+0000 osd.7 (osd.7) 29 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:54:39.201 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:39 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:54:39.201 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:39 smithi142 conmon[20125]: T21:54:37.757967+0000 osd.7 ( 2021-03-24T21:54:39.202 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:39 smithi142 conmon[20125]: osd.7) 30 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:54:39.202 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:39 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:54:39.202 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:39 smithi142 conmon[20125]: 24T21:54:38.094774+0000 mon.a ( 2021-03-24T21:54:39.202 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:39 smithi142 conmon[20125]: mon.0) 647 : cluster [DBG] osdmap e109: 8 total, 8 up, 6 in 2021-03-24T21:54:39.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:54:39.204 INFO:tasks.rados.rados.0.smithi125.stdout:62: done (2 left) 2021-03-24T21:54:39.205 INFO:tasks.rados.rados.0.smithi125.stdout:63: read oid 28 snap -1 2021-03-24T21:54:39.205 INFO:tasks.rados.rados.0.smithi125.stdout:63: expect (ObjNum 27 snap 0 seq_num 27) 2021-03-24T21:54:39.205 INFO:tasks.rados.rados.0.smithi125.stdout:64: read oid 33 snap -1 2021-03-24T21:54:39.205 INFO:tasks.rados.rados.0.smithi125.stdout:64: expect (ObjNum 53 snap 2 seq_num 53) 2021-03-24T21:54:39.206 INFO:tasks.rados.rados.0.smithi125.stdout:65: delete oid 2 current snap is 2 2021-03-24T21:54:39.206 INFO:tasks.rados.rados.0.smithi125.stdout:61: finishing rollback tid 0 to smithi12556470-47 2021-03-24T21:54:39.206 INFO:tasks.rados.rados.0.smithi125.stdout:61: finishing rollback tid 1 to smithi12556470-47 2021-03-24T21:54:39.206 INFO:tasks.rados.rados.0.smithi125.stdout:61: finishing rollback tid 2 to smithi12556470-47 2021-03-24T21:54:39.206 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 47 v 7 (ObjNum 46 snap 0 seq_num 46) dirty exists 2021-03-24T21:54:39.207 INFO:tasks.rados.rados.0.smithi125.stdout:60: finishing write tid 1 to smithi12556470-12 2021-03-24T21:54:39.207 INFO:tasks.rados.rados.0.smithi125.stdout:60: finishing write tid 2 to smithi12556470-12 2021-03-24T21:54:39.207 INFO:tasks.rados.rados.0.smithi125.stdout:60: finishing write tid 3 to smithi12556470-12 2021-03-24T21:54:39.207 INFO:tasks.rados.rados.0.smithi125.stdout:60: finishing write tid 4 to smithi12556470-12 2021-03-24T21:54:39.207 INFO:tasks.rados.rados.0.smithi125.stdout:60: finishing write tid 5 to smithi12556470-12 2021-03-24T21:54:39.208 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 12 v 22 (ObjNum 55 snap 2 seq_num 55) dirty exists 2021-03-24T21:54:39.208 INFO:tasks.rados.rados.0.smithi125.stdout:60: left oid 12 (ObjNum 55 snap 2 seq_num 55) 2021-03-24T21:54:39.208 INFO:tasks.rados.rados.0.smithi125.stdout:64: expect (ObjNum 53 snap 2 seq_num 53) 2021-03-24T21:54:39.279 INFO:tasks.rados.rados.0.smithi125.stdout:63: expect (ObjNum 27 snap 0 seq_num 27) 2021-03-24T21:54:39.305 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:39.313 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T21:54:39.313 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[19936]: T21:54:37.757808+0000 osd.7 (osd.7) 29 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:54:39.314 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[19936]: cluster 2021-03-24T21:54:39.314 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[19936]: 2021-03-24T21:54:39.314 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[19936]: 2021 2021-03-24T21:54:39.314 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[19936]: - 2021-03-24T21:54:39.314 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[19936]: 03 2021-03-24T21:54:39.315 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[19936]: - 2021-03-24T21:54:39.315 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[19936]: 24 2021-03-24T21:54:39.315 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[19936]: T 2021-03-24T21:54:39.315 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[19936]: 21 2021-03-24T21:54:39.315 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[19936]: : 2021-03-24T21:54:39.316 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[19936]: 54 2021-03-24T21:54:39.316 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[19936]: : 2021-03-24T21:54:39.316 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[19936]: 37 2021-03-24T21:54:39.316 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[19936]: . 2021-03-24T21:54:39.317 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[19936]: 757967 2021-03-24T21:54:39.317 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[19936]: +0000 2021-03-24T21:54:39.317 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[19936]: 2021-03-24T21:54:39.317 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[19936]: osd.7 2021-03-24T21:54:39.317 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[19936]: ( 2021-03-24T21:54:39.318 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[19936]: osd.7) 30 : 2021-03-24T21:54:39.318 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[19936]: cluster [DBG] purged_snaps scrub ok 2021-03-24T21:54:39.318 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[19936]: cluster 2021-03-24T21:54: 2021-03-24T21:54:39.318 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[19936]: 38.094774+0000 mon.a (mon.0) 647 : cluster 2021-03-24T21:54:39.319 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[19936]: [DBG] osdmap e109: 8 total, 8 up, 6 in 2021-03-24T21:54:39.319 INFO:journalctl@ceph.osd.3.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[37997]: debug 2021-03-24T21:54:39.123+0000 7f17173db700 -1 osd.3 pg_epoch: 110 pg[7.3( v 110'24 (0'0,110'24] local-lis/les=99/100 n=7 ec=99/99 lis/c=99/99 les/c/f=100/100/0 sis=99) [3,2,4] r=0 lpr=99 luod=109'22 crt=110'24 lcod 109'21 mlcod 109'21 active+clean+snaptrim trimq=[2~1]] removing snap head 2021-03-24T21:54:39.319 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: cluster 2021-03-24T21:54:39.320 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: 2021-03-24T21:54:39.320 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: 2021 2021-03-24T21:54:39.320 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: - 2021-03-24T21:54:39.320 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: 03 2021-03-24T21:54:39.321 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: - 2021-03-24T21:54:39.321 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: 24 2021-03-24T21:54:39.321 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: T 2021-03-24T21:54:39.321 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: 21 2021-03-24T21:54:39.322 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: : 2021-03-24T21:54:39.322 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: 54 2021-03-24T21:54:39.322 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: : 2021-03-24T21:54:39.322 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: 37 2021-03-24T21:54:39.322 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: . 2021-03-24T21:54:39.323 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: 757808 2021-03-24T21:54:39.323 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: +0000 2021-03-24T21:54:39.323 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: 2021-03-24T21:54:39.323 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: osd.7 2021-03-24T21:54:39.323 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: ( 2021-03-24T21:54:39.324 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: osd 2021-03-24T21:54:39.324 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: . 2021-03-24T21:54:39.324 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: 7 2021-03-24T21:54:39.324 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: ) 2021-03-24T21:54:39.325 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: 29 2021-03-24T21:54:39.325 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: : 2021-03-24T21:54:39.325 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: cluster 2021-03-24T21:54:39.325 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: 2021-03-24T21:54:39.325 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: [DBG] 2021-03-24T21:54:39.326 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: 2021-03-24T21:54:39.326 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: purged_snaps scrub starts 2021-03-24T21:54:39.326 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: 2021-03-24T21:54:39.326 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: cluster 2021-03-24T21:54:39.327 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: 2021-03-24T21:54:39.327 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: 2021 2021-03-24T21:54:39.327 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: - 2021-03-24T21:54:39.327 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: 03 2021-03-24T21:54:39.327 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: - 2021-03-24T21:54:39.328 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: 24 2021-03-24T21:54:39.328 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: T 2021-03-24T21:54:39.328 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: 21 2021-03-24T21:54:39.328 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: : 2021-03-24T21:54:39.328 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: 54 2021-03-24T21:54:39.329 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: : 2021-03-24T21:54:39.329 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: 37 2021-03-24T21:54:39.329 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: . 2021-03-24T21:54:39.329 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: 757967 2021-03-24T21:54:39.329 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: +0000 2021-03-24T21:54:39.330 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: 2021-03-24T21:54:39.330 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: osd.7 2021-03-24T21:54:39.330 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: ( 2021-03-24T21:54:39.330 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: osd 2021-03-24T21:54:39.331 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: . 2021-03-24T21:54:39.331 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: 7 2021-03-24T21:54:39.331 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: ) 2021-03-24T21:54:39.331 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: 30 2021-03-24T21:54:39.331 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: : 2021-03-24T21:54:39.332 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: cluster 2021-03-24T21:54:39.332 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: 2021-03-24T21:54:39.332 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: [DBG] 2021-03-24T21:54:39.332 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: 2021-03-24T21:54:39.333 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: purged_snaps scrub ok 2021-03-24T21:54:39.333 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: 2021-03-24T21:54:39.333 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: cluster 2021-03-24T21:54:39.333 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: 2021-03-24T21:54:39.333 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: 2021 2021-03-24T21:54:39.334 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: - 2021-03-24T21:54:39.334 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: 03 2021-03-24T21:54:39.334 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: - 2021-03-24T21:54:39.334 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: 24 2021-03-24T21:54:39.334 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: T 2021-03-24T21:54:39.335 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: 21 2021-03-24T21:54:39.335 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: : 2021-03-24T21:54:39.335 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: 54 2021-03-24T21:54:39.335 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: : 2021-03-24T21:54:39.336 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: 38 2021-03-24T21:54:39.336 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: . 2021-03-24T21:54:39.336 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: 094774 2021-03-24T21:54:39.336 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: +0000 2021-03-24T21:54:39.336 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: 2021-03-24T21:54:39.337 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: mon.a 2021-03-24T21:54:39.337 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: ( 2021-03-24T21:54:39.337 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: mon 2021-03-24T21:54:39.337 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: . 2021-03-24T21:54:39.337 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: 0 2021-03-24T21:54:39.338 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: ) 2021-03-24T21:54:39.338 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:39 smithi125 conmon[24927]: 647 : cluster [DBG] osdmap e109: 8 total, 8 up, 6 in 2021-03-24T21:54:39.351 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:39.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:39.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:54:39.364 INFO:tasks.rados.rados.0.smithi125.stdout:60: done (4 left) 2021-03-24T21:54:39.364 INFO:tasks.rados.rados.0.smithi125.stdout:61: done (3 left) 2021-03-24T21:54:39.365 INFO:tasks.rados.rados.0.smithi125.stdout:63: done (2 left) 2021-03-24T21:54:39.365 INFO:tasks.rados.rados.0.smithi125.stdout:64: done (1 left) 2021-03-24T21:54:39.366 INFO:tasks.rados.rados.0.smithi125.stdout:65: done (0 left) 2021-03-24T21:54:39.366 INFO:tasks.rados.rados.0.smithi125.stdout:66: copy_from oid 29 from oid 27 current snap is 2 2021-03-24T21:54:39.366 INFO:tasks.rados.rados.0.smithi125.stdout:67: snap_create 2021-03-24T21:54:39.376 INFO:tasks.rados.rados.0.smithi125.stdout:66: finishing copy_from to smithi12556470-29 2021-03-24T21:54:39.376 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 29 v 12 (ObjNum 26 snap 0 seq_num 26) dirty exists 2021-03-24T21:54:39.377 INFO:tasks.rados.rados.0.smithi125.stdout:66: finishing copy_from racing read to smithi12556470-29 2021-03-24T21:54:39.456 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:39.501 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:39.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:39.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:54:39.605 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:39.657 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:39.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:39.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:54:39.762 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:39.811 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:39.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:39.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:54:39.917 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:39.972 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:40.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:40.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:54:40.130 INFO:tasks.rados.rados.0.smithi125.stdout:66: done (1 left) 2021-03-24T21:54:40.130 INFO:tasks.rados.rados.0.smithi125.stdout:67: done (0 left) 2021-03-24T21:54:40.130 INFO:tasks.rados.rados.0.smithi125.stdout:68: snap_remove snap 1 2021-03-24T21:54:40.161 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:40.215 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:40.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:40.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:54:40.320 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:40.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:40 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:54:40.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:40 smithi142 conmon[20125]: 24T21:54:38.709691+0000 mgr.y (mgr.14144) 312 : cluster [DBG] 2021-03-24T21:54:40.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:40 smithi142 conmon[20125]: pgmap v326: 148 pgs: 1 active+degraded, 1 activating, 3 activating+degraded, 7 active+recovery_wait+degraded, 2 active+recovering, 134 active+clean; 32 MiB data, 91 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 2.8 MiB/s wr, 15 op/s; 76/990 objects degraded (7.677%); 14/990 objects misplaced (1.414%); 1.4 MiB/s, 2 objects/s recovering 2021-03-24T21:54:40.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:40 smithi142 conmon[20125]: cluster 2021-03-24T21:54:40.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:40 smithi142 conmon[20125]: 2021-03- 2021-03-24T21:54:40.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:40 smithi142 conmon[20125]: 24T21: 2021-03-24T21:54:40.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:40 smithi142 conmon[20125]: 54:39.102251 2021-03-24T21:54:40.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:40 smithi142 conmon[20125]: +0000 mon.a ( 2021-03-24T21:54:40.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:40 smithi142 conmon[20125]: mon.0) 2021-03-24T21:54:40.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:40 smithi142 conmon[20125]: 648 : cluster 2021-03-24T21:54:40.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:40 smithi142 conmon[20125]: [WRN] Health check failed: Degraded data redundancy: 76/990 objects degraded (7.677%), 11 pgs degraded (PG_DEGRADED) 2021-03-24T21:54:40.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:40 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:54:40.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:40 smithi142 conmon[20125]: 21:54:39.114503+0000 mon.a (mon.0) 649 : cluster [DBG] 2021-03-24T21:54:40.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:40 smithi142 conmon[20125]: osdmap e110: 8 total, 8 up, 6 in 2021-03-24T21:54:40.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:40 smithi142 conmon[20125]: audit 2021 2021-03-24T21:54:40.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:40 smithi142 conmon[20125]: -03-24T21:54: 2021-03-24T21:54:40.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:40 smithi142 conmon[20125]: 39.756761+0000 mon.a (mon 2021-03-24T21:54:40.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:40 smithi142 conmon[20125]: .0) 650 : audit [INF] 2021-03-24T21:54:40.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:40 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:54:40.399 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:40.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:40.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:54:40.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:40 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:54:40.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:40 smithi125 conmon[19936]: :54:38.709691+0000 mgr.y (mgr.14144) 312 : cluster [DBG] pgmap v326: 148 pgs: 1 active+degraded, 1 activating, 3 activating+degraded, 7 active+recovery_wait+degraded, 2 active+recovering, 134 active+clean; 32 MiB data, 91 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 2.8 MiB/s wr, 15 op/s; 76/990 objects degraded (7.677%); 14/990 objects misplaced (1.414%); 1.4 MiB/s, 2 objects/s recovering 2021-03-24T21:54:40.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:40 smithi125 conmon[19936]: cluster 2021-03-24T21:54:39. 2021-03-24T21:54:40.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:40 smithi125 conmon[19936]: 102251+0000 mon.a (mon.0) 648 : cluster [WRN] 2021-03-24T21:54:40.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:40 smithi125 conmon[19936]: Health check failed: Degraded data redundancy: 76/990 objects degraded (7.677%), 11 pgs degraded (PG_DEGRADED) 2021-03-24T21:54:40.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:40 smithi125 conmon[19936]: cluster 2021-03-24T21:54:40.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:40 smithi125 conmon[19936]: 2021-03-24T21:54:39.114503+0000 mon.a (mon. 2021-03-24T21:54:40.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:40 smithi125 conmon[19936]: 0) 649 : cluster [DBG] osdmap e110: 8 total, 8 up, 6 in 2021-03-24T21:54:40.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:40 smithi125 conmon[19936]: audit 2021 2021-03-24T21:54:40.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:40 smithi125 conmon[19936]: -03-24T21:54:39.756761+0000 mon.a (mon 2021-03-24T21:54:40.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:40 smithi125 conmon[19936]: .0) 650 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:54:40.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:40 smithi125 conmon[24927]: cluster 2021-03-24T21:54: 2021-03-24T21:54:40.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:40 smithi125 conmon[24927]: 38.709691+0000 mgr.y (mgr.14144) 312 : cluster [DBG] pgmap v326: 148 pgs: 1 active+degraded, 1 activating, 3 activating+degraded, 7 active+recovery_wait+degraded, 2 active+recovering, 134 active+clean; 32 MiB data, 91 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 2.8 MiB/s wr, 15 op/s; 76/990 objects degraded (7.677%); 14/990 objects misplaced (1.414%); 1.4 MiB/s, 2 objects/s recovering 2021-03-24T21:54:40.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:40 smithi125 conmon[24927]: cluster 2021-03-24T21:54:39.102251+0000 2021-03-24T21:54:40.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:40 smithi125 conmon[24927]: mon.a (mon.0) 648 : cluster [WRN] Health check failed: Degraded data redundancy: 76/990 objects degraded (7.677%), 11 pgs degraded (PG_DEGRADED) 2021-03-24T21:54:40.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:40 smithi125 conmon[24927]: cluster 2021-03-24T21:54:40.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:40 smithi125 conmon[24927]: 2021-03 2021-03-24T21:54:40.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:40 smithi125 conmon[24927]: -24T21:54:39.114503 2021-03-24T21:54:40.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:40 smithi125 conmon[24927]: +0000 mon.a (mon.0) 2021-03-24T21:54:40.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:40 smithi125 conmon[24927]: 649 : cluster [DBG] osdmap e110: 8 total, 8 up, 6 in 2021-03-24T21:54:40.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:40 smithi125 conmon[24927]: audit 2021 2021-03-24T21:54:40.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:40 smithi125 conmon[24927]: -03-24T21:54:39. 2021-03-24T21:54:40.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:40 smithi125 conmon[24927]: 756761+0000 mon.a (mon.0) 650 : audit 2021-03-24T21:54:40.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:40 smithi125 conmon[24927]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:54:40.505 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:40.556 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:40.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:40.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:54:40.661 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:54:40.710 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:54:40.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:40.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:54:40.814 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:40.872 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:40.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:40.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:54:40.977 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:41.033 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:41.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:41.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:54:41.137 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:41.141 INFO:tasks.rados.rados.0.smithi125.stdout:68: done (0 left) 2021-03-24T21:54:41.142 INFO:tasks.rados.rados.0.smithi125.stdout:69: read oid 23 snap -1 2021-03-24T21:54:41.142 INFO:tasks.rados.rados.0.smithi125.stdout:69: expect (ObjNum 22 snap 0 seq_num 22) 2021-03-24T21:54:41.142 INFO:tasks.rados.rados.0.smithi125.stdout:70: read oid 24 snap -1 2021-03-24T21:54:41.142 INFO:tasks.rados.rados.0.smithi125.stdout:70: expect (ObjNum 23 snap 0 seq_num 23) 2021-03-24T21:54:41.143 INFO:tasks.rados.rados.0.smithi125.stdout:71: read oid 41 snap -1 2021-03-24T21:54:41.143 INFO:tasks.rados.rados.0.smithi125.stdout:71: expect (ObjNum 40 snap 0 seq_num 40) 2021-03-24T21:54:41.143 INFO:tasks.rados.rados.0.smithi125.stdout:72: snap_remove snap 2 2021-03-24T21:54:41.183 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:41.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:41.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:54:41.289 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:41.359 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:41.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:41.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:54:41.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:41 smithi125 conmon[19936]: cluster 2021-03-24T21:54:41.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:41 smithi125 conmon[19936]: 2021-03- 2021-03-24T21:54:41.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:41 smithi125 conmon[19936]: 24T21:54:40.129600+0000 mon.a (mon.0) 651 : cluster [DBG] osdmap e111: 8 total, 8 up, 6 in 2021-03-24T21:54:41.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:41 smithi125 conmon[24927]: cluster 2021-03-24T21:54:41.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:41 smithi125 conmon[24927]: 2021-03-24T21:54: 2021-03-24T21:54:41.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:41 smithi125 conmon[24927]: 40.129600+0000 mon.a (mon.0) 651 : cluster [DBG] osdmap e111: 8 total, 8 up, 6 in 2021-03-24T21:54:41.411 INFO:journalctl@ceph.osd.2.smithi125.stdout:Mar 24 21:54:41 smithi125 conmon[34620]: debug 2021-03-24T21:54:41.412 INFO:journalctl@ceph.osd.2.smithi125.stdout:Mar 24 21:54:41 smithi125 conmon[34620]: 2021-03-24T21:54:41.143+0000 7f47059d6700 -1 osd.2 pg_epoch: 112 pg[7.8( v 110'10 (0'0,110'10] local-lis/les=99/100 n=3 ec=99/99 lis/c=99/99 les/c/f=100/100/0 sis=99) [2,7,4] r=0 lpr=99 crt=110'10 lcod 109'8 mlcod 109'8 active+clean+snaptrim trimq=[3~1] ps=[2~1]] removing snap head 2021-03-24T21:54:41.412 INFO:journalctl@ceph.osd.2.smithi125.stdout:Mar 24 21:54:41 smithi125 conmon[34620]: 2021-03-24T21:54:41.412 INFO:journalctl@ceph.osd.3.smithi125.stdout:Mar 24 21:54:41 smithi125 conmon[37997]: debug 2021-03-24T21:54:41.412 INFO:journalctl@ceph.osd.3.smithi125.stdout:Mar 24 21:54:41 smithi125 conmon[37997]: 2021-03-24T21:54:41.139+0000 7f1716bda700 -1 osd.3 pg_epoch: 112 pg[7.c( v 110'15 (0'0,110'15] local-lis/les=101/102 n=5 ec=99/99 lis/c=101/101 les/c/f=102/102/0 sis=101) [3,7,2] r=0 lpr=101 crt=110'15 lcod 107'13 mlcod 107'13 active+clean+snaptrim trimq=[3~1] ps=[2~1]] removing snap head 2021-03-24T21:54:41.413 INFO:journalctl@ceph.osd.3.smithi125.stdout:Mar 24 21:54:41 smithi125 conmon[37997]: 2021-03-24T21:54:41.463 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:41.526 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:41.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:41.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:54:41.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:41 smithi142 conmon[20125]: cluster 2021-03-24T21:54:41.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:41 smithi142 conmon[20125]: 2021-03-24T21:54:41.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:41 smithi142 conmon[20125]: 2021 2021-03-24T21:54:41.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:41 smithi142 conmon[20125]: - 2021-03-24T21:54:41.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:41 smithi142 conmon[20125]: 03 2021-03-24T21:54:41.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:41 smithi142 conmon[20125]: - 2021-03-24T21:54:41.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:41 smithi142 conmon[20125]: 24 2021-03-24T21:54:41.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:41 smithi142 conmon[20125]: T 2021-03-24T21:54:41.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:41 smithi142 conmon[20125]: 21 2021-03-24T21:54:41.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:41 smithi142 conmon[20125]: : 2021-03-24T21:54:41.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:41 smithi142 conmon[20125]: 54 2021-03-24T21:54:41.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:41 smithi142 conmon[20125]: : 2021-03-24T21:54:41.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:41 smithi142 conmon[20125]: 40 2021-03-24T21:54:41.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:41 smithi142 conmon[20125]: . 2021-03-24T21:54:41.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:41 smithi142 conmon[20125]: 129600 2021-03-24T21:54:41.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:41 smithi142 conmon[20125]: +0000 2021-03-24T21:54:41.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:41 smithi142 conmon[20125]: 2021-03-24T21:54:41.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:41 smithi142 conmon[20125]: mon.a 2021-03-24T21:54:41.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:41 smithi142 conmon[20125]: ( 2021-03-24T21:54:41.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:41 smithi142 conmon[20125]: mon 2021-03-24T21:54:41.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:41 smithi142 conmon[20125]: .0) 651 : cluster [DBG] osdmap e111: 8 total, 8 up, 6 in 2021-03-24T21:54:41.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:41 smithi142 conmon[20125]: 2021-03-24T21:54:41.633 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:41.687 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:41.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:41.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:54:41.793 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:41.847 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:41.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:41.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:54:41.951 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:42.006 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:42.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:42.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:54:42.110 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:42.148 INFO:tasks.rados.rados.0.smithi125.stdout:72: done (3 left) 2021-03-24T21:54:42.148 INFO:tasks.rados.rados.0.smithi125.stdout:73: write oid 47 current snap is 3 2021-03-24T21:54:42.148 INFO:tasks.rados.rados.0.smithi125.stdout:73: seq_num 57 ranges {557999=703664,1719003=413684,2632287=325712} 2021-03-24T21:54:42.152 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-24T21:54:42.153 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-24T21:54:42.153 INFO:teuthology.orchestra.run.smithi125.stdout: "num_ops": 0 2021-03-24T21:54:42.153 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-24T21:54:42.156 INFO:tasks.rados.rados.0.smithi125.stdout:73: writing smithi12556470-47 from 557999 to 1261663 tid 1 2021-03-24T21:54:42.162 INFO:tasks.rados.rados.0.smithi125.stdout:73: writing smithi12556470-47 from 1719003 to 2132687 tid 2 2021-03-24T21:54:42.167 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:42.170 INFO:tasks.rados.rados.0.smithi125.stdout:73: writing smithi12556470-47 from 2632287 to 2957999 tid 3 2021-03-24T21:54:42.170 INFO:tasks.rados.rados.0.smithi125.stdout:74: copy_from oid 48 from oid 25 current snap is 3 2021-03-24T21:54:42.170 INFO:tasks.rados.rados.0.smithi125.stdout:70: expect (ObjNum 23 snap 0 seq_num 23) 2021-03-24T21:54:42.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:42.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:54:42.266 INFO:tasks.rados.rados.0.smithi125.stdout:69: expect (ObjNum 22 snap 0 seq_num 22) 2021-03-24T21:54:42.273 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:42.287 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:42 smithi142 conmon[20125]: cluster 2021-03-24T21:54:40.710562+0000 mgr.y (mgr.14144) 313 : cluster 2021-03-24T21:54:42.287 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:42 smithi142 conmon[20125]: [DBG] pgmap v329: 148 pgs: 1 active+recovery_wait, 5 active+recovering, 12 active+recovery_wait+degraded, 3 activating+degraded, 1 activating, 125 active+clean, 1 active+degraded; 114 MiB data, 200 MiB used, 536 GiB / 536 GiB avail; 11 KiB/s rd, 14 MiB/s wr, 75 op/s; 103/1104 objects degraded (9.330%); 16/1104 objects misplaced (1.449%); 5.4 MiB/s, 16 objects/s recovering 2021-03-24T21:54:42.287 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:42 smithi142 conmon[20125]: cluster 2021-03-24T21:54:41.138921+0000 mon.a (mon.0) 2021-03-24T21:54:42.287 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:42 smithi142 conmon[20125]: 652 : cluster [DBG] osdmap e112: 8 total, 8 up, 6 in 2021-03-24T21:54:42.332 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:42.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:42.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:54:42.346 INFO:tasks.rados.rados.0.smithi125.stdout:71: expect (ObjNum 40 snap 0 seq_num 40) 2021-03-24T21:54:42.398 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:42 smithi125 conmon[24927]: cluster 2021-03-24T21:54:40.710562+0000 mgr.y (mgr.14144) 313 : cluster [DBG] pgmap v329: 148 pgs: 1 active+recovery_wait, 5 active+recovering, 12 active+recovery_wait+degraded, 3 activating+degraded, 1 activating, 125 active+clean, 1 active+degraded; 114 MiB data, 200 MiB used, 536 GiB / 536 GiB avail; 11 KiB/s rd, 14 MiB/s wr, 75 op/s; 103/1104 objects degraded (9.330%); 16/1104 objects misplaced (1.449%); 5.4 MiB/s, 16 objects/s recovering 2021-03-24T21:54:42.399 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:42 smithi125 conmon[24927]: cluster 2021-03-24T21:54:41.138921+0000 mon.a (mon.0) 652 : cluster [DBG] osdmap e112: 8 total, 8 up, 6 in 2021-03-24T21:54:42.399 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:42 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:54:42.399 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:42 smithi125 conmon[19936]: :54:40.710562+0000 mgr.y (mgr.14144) 313 : cluster [DBG] pgmap v329: 148 pgs: 1 active+recovery_wait, 5 active+recovering, 12 active+recovery_wait+degraded, 3 activating+degraded, 1 activating, 125 active+clean, 1 active+degraded; 114 MiB data, 200 MiB used, 536 GiB / 536 GiB avail; 11 KiB/s rd, 14 MiB/s wr, 75 op/s; 103/1104 objects degraded (9.330%); 16/1104 objects misplaced (1.449%); 5.4 MiB/s, 16 objects/s recovering 2021-03-24T21:54:42.400 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:42 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:54:42.400 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:42 smithi125 conmon[19936]: -24T21:54:41 2021-03-24T21:54:42.400 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:42 smithi125 conmon[19936]: .138921+0000 mon.a ( 2021-03-24T21:54:42.400 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:42 smithi125 conmon[19936]: mon.0) 652 : cluster 2021-03-24T21:54:42.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:42 smithi125 conmon[19936]: [DBG] osdmap e112: 8 total, 8 up, 6 in 2021-03-24T21:54:42.434 INFO:tasks.rados.rados.0.smithi125.stdout:73: finishing write tid 1 to smithi12556470-47 2021-03-24T21:54:42.434 INFO:tasks.rados.rados.0.smithi125.stdout:73: finishing write tid 2 to smithi12556470-47 2021-03-24T21:54:42.434 INFO:tasks.rados.rados.0.smithi125.stdout:73: finishing write tid 3 to smithi12556470-47 2021-03-24T21:54:42.435 INFO:tasks.rados.rados.0.smithi125.stdout:73: finishing write tid 4 to smithi12556470-47 2021-03-24T21:54:42.435 INFO:tasks.rados.rados.0.smithi125.stdout:73: finishing write tid 5 to smithi12556470-47 2021-03-24T21:54:42.435 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 47 v 15 (ObjNum 57 snap 3 seq_num 57) dirty exists 2021-03-24T21:54:42.435 INFO:tasks.rados.rados.0.smithi125.stdout:73: left oid 47 (ObjNum 57 snap 3 seq_num 57) 2021-03-24T21:54:42.436 INFO:tasks.rados.rados.0.smithi125.stdout:74: finishing copy_from to smithi12556470-48 2021-03-24T21:54:42.436 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 48 v 6 (ObjNum 24 snap 0 seq_num 24) dirty exists 2021-03-24T21:54:42.436 INFO:tasks.rados.rados.0.smithi125.stdout:74: finishing copy_from racing read to smithi12556470-48 2021-03-24T21:54:42.436 INFO:tasks.rados.rados.0.smithi125.stdout:69: done (4 left) 2021-03-24T21:54:42.437 INFO:tasks.rados.rados.0.smithi125.stdout:70: done (3 left) 2021-03-24T21:54:42.437 INFO:tasks.rados.rados.0.smithi125.stdout:71: done (2 left) 2021-03-24T21:54:42.437 INFO:tasks.rados.rados.0.smithi125.stdout:73: done (1 left) 2021-03-24T21:54:42.437 INFO:tasks.rados.rados.0.smithi125.stdout:74: done (0 left) 2021-03-24T21:54:42.437 INFO:tasks.rados.rados.0.smithi125.stdout:75: copy_from oid 50 from oid 22 current snap is 3 2021-03-24T21:54:42.438 INFO:tasks.rados.rados.0.smithi125.stdout:76: snap_create 2021-03-24T21:54:42.438 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:54:42.453 INFO:tasks.rados.rados.0.smithi125.stdout:75: finishing copy_from to smithi12556470-50 2021-03-24T21:54:42.454 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 50 v 15 (ObjNum 51 snap 0 seq_num 51) dirty exists 2021-03-24T21:54:42.454 INFO:tasks.rados.rados.0.smithi125.stdout:75: finishing copy_from racing read to smithi12556470-50 2021-03-24T21:54:42.498 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:54:42.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:42.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:54:42.604 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:42.658 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:42.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:42.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:54:42.764 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:42.824 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:42.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:42.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:54:42.930 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:42.985 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:42.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:42.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:54:43.092 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:43.161 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:43.162 INFO:tasks.rados.rados.0.smithi125.stdout:75: done (1 left) 2021-03-24T21:54:43.163 INFO:tasks.rados.rados.0.smithi125.stdout:76: done (0 left) 2021-03-24T21:54:43.163 INFO:tasks.rados.rados.0.smithi125.stdout:77: copy_from oid 22 from oid 49 current snap is 4 2021-03-24T21:54:43.163 INFO:tasks.rados.rados.0.smithi125.stdout:78: read oid 39 snap 3 2021-03-24T21:54:43.163 INFO:tasks.rados.rados.0.smithi125.stdout:78: expect (ObjNum 38 snap 0 seq_num 38) 2021-03-24T21:54:43.164 INFO:tasks.rados.rados.0.smithi125.stdout:79: write oid 30 current snap is 4 2021-03-24T21:54:43.164 INFO:tasks.rados.rados.0.smithi125.stdout:79: seq_num 58 ranges {574806=778913,2061132=557342} 2021-03-24T21:54:43.165 INFO:teuthology.orchestra.run.smithi125.stderr:noscrub is unset 2021-03-24T21:54:43.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:43.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:54:43.171 INFO:tasks.rados.rados.0.smithi125.stdout:79: writing smithi12556470-30 from 574806 to 1353719 tid 1 2021-03-24T21:54:43.180 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 7] out_osds: [6, 1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-03-24T21:54:43.180 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-24T21:54:43.180 INFO:tasks.thrashosds.thrasher:inject_pause on 1 2021-03-24T21:54:43.181 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2021-03-24T21:54:43.181 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2021-03-24T21:54:43.181 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.1 config set filestore_inject_stall 3 2021-03-24T21:54:43.184 INFO:tasks.rados.rados.0.smithi125.stdout:79: writing smithi12556470-30 from 2061132 to 2618474 tid 2 2021-03-24T21:54:43.184 INFO:tasks.rados.rados.0.smithi125.stdout:80: write (excl) oid 29 current snap is 4 2021-03-24T21:54:43.184 INFO:tasks.rados.rados.0.smithi125.stdout:80: seq_num 59 ranges {591613=454162,1719614=417353,2570780=420833} 2021-03-24T21:54:43.190 INFO:tasks.rados.rados.0.smithi125.stdout:80: writing smithi12556470-29 from 591613 to 1045775 tid 1 2021-03-24T21:54:43.196 INFO:tasks.rados.rados.0.smithi125.stdout:80: writing smithi12556470-29 from 1719614 to 2136967 tid 2 2021-03-24T21:54:43.201 INFO:tasks.rados.rados.0.smithi125.stdout:80: writing smithi12556470-29 from 2570780 to 2991613 tid 3 2021-03-24T21:54:43.201 INFO:tasks.rados.rados.0.smithi125.stdout:81: read oid 36 snap 3 2021-03-24T21:54:43.202 INFO:tasks.rados.rados.0.smithi125.stdout:81: expect (ObjNum 35 snap 0 seq_num 35) 2021-03-24T21:54:43.202 INFO:tasks.rados.rados.0.smithi125.stdout:82: copy_from oid 42 from oid 28 current snap is 4 2021-03-24T21:54:43.202 INFO:tasks.rados.rados.0.smithi125.stdout:78: expect (ObjNum 38 snap 0 seq_num 38) 2021-03-24T21:54:43.236 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.1 dump_blocked_ops 2021-03-24T21:54:43.268 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:43.319 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:43.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:43.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:54:43.333 INFO:tasks.rados.rados.0.smithi125.stdout:79: finishing write tid 1 to smithi12556470-30 2021-03-24T21:54:43.333 INFO:tasks.rados.rados.0.smithi125.stdout:79: finishing write tid 2 to smithi12556470-30 2021-03-24T21:54:43.333 INFO:tasks.rados.rados.0.smithi125.stdout:79: finishing write tid 3 to smithi12556470-30 2021-03-24T21:54:43.333 INFO:tasks.rados.rados.0.smithi125.stdout:79: finishing write tid 4 to smithi12556470-30 2021-03-24T21:54:43.334 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 30 v 14 (ObjNum 58 snap 4 seq_num 58) dirty exists 2021-03-24T21:54:43.334 INFO:tasks.rados.rados.0.smithi125.stdout:79: left oid 30 (ObjNum 58 snap 4 seq_num 58) 2021-03-24T21:54:43.334 INFO:tasks.rados.rados.0.smithi125.stdout:77: finishing copy_from to smithi12556470-22 2021-03-24T21:54:43.334 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 22 v 18 (ObjNum 48 snap 0 seq_num 48) dirty exists 2021-03-24T21:54:43.334 INFO:tasks.rados.rados.0.smithi125.stdout:77: finishing copy_from racing read to smithi12556470-22 2021-03-24T21:54:43.335 INFO:tasks.rados.rados.0.smithi125.stdout:80: finishing write tid 1 to smithi12556470-29 2021-03-24T21:54:43.335 INFO:tasks.rados.rados.0.smithi125.stdout:80: finishing write tid 2 to smithi12556470-29 2021-03-24T21:54:43.335 INFO:tasks.rados.rados.0.smithi125.stdout:77: done (5 left) 2021-03-24T21:54:43.335 INFO:tasks.rados.rados.0.smithi125.stdout:78: done (4 left) 2021-03-24T21:54:43.336 INFO:tasks.rados.rados.0.smithi125.stdout:79: done (3 left) 2021-03-24T21:54:43.336 INFO:tasks.rados.rados.0.smithi125.stdout:83: rollback oid 41 current snap is 4 2021-03-24T21:54:43.336 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 41 to 3 2021-03-24T21:54:43.336 INFO:tasks.rados.rados.0.smithi125.stdout:80: finishing write tid 3 to smithi12556470-29 2021-03-24T21:54:43.336 INFO:tasks.rados.rados.0.smithi125.stdout:80: finishing write tid 4 to smithi12556470-29 2021-03-24T21:54:43.337 INFO:tasks.rados.rados.0.smithi125.stdout:80: finishing write tid 5 to smithi12556470-29 2021-03-24T21:54:43.337 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 29 v 16 (ObjNum 59 snap 4 seq_num 59) dirty exists 2021-03-24T21:54:43.337 INFO:tasks.rados.rados.0.smithi125.stdout:80: left oid 29 (ObjNum 59 snap 4 seq_num 59) 2021-03-24T21:54:43.337 INFO:tasks.rados.rados.0.smithi125.stdout:81: expect (ObjNum 35 snap 0 seq_num 35) 2021-03-24T21:54:43.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:43 smithi142 conmon[20125]: cluster 2021-03-24T21:54 2021-03-24T21:54:43.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:43 smithi142 conmon[20125]: :42.146036+0000 mon.a (mon.0) 653 : cluster [DBG] osdmap e113: 8 total, 8 up, 6 in 2021-03-24T21:54:43.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:43 smithi142 conmon[20125]: audit 2021-03-24T21:54:43.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:43 smithi142 conmon[20125]: 2021-03-24T21:54:42.287276+0000 mon.c (mon.2) 25 : audit 2021-03-24T21:54:43.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:43 smithi142 conmon[20125]: [INF] from='client.? v1:172.21.15.125:0/3674138739' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T21:54:43.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:43 smithi142 conmon[20125]: audit 2021-03-24T21 2021-03-24T21:54:43.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:43 smithi142 conmon[20125]: :54:42.287760+0000 mon.a (mon.0) 2021-03-24T21:54:43.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:43 smithi142 conmon[20125]: 654 : audit [INF] from='client.34691 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T21:54:43.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:43 smithi125 conmon[24927]: cluster 2021-03-24T21:54:42.146036+0000 mon.a (mon.0) 653 : cluster [DBG] osdmap e113: 8 total, 8 up, 6 in 2021-03-24T21:54:43.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:43 smithi125 conmon[24927]: audit 2021-03-24T21:54:42.287276+0000 mon.c (mon.2) 25 : audit [INF] from='client.? v1:172.21.15.125:0/3674138739' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T21:54:43.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:43 smithi125 conmon[24927]: audit 2021-03-24T21:54:42.287760+0000 mon.a (mon.0) 654 : audit [INF] from='client.34691 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T21:54:43.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:43 smithi125 conmon[19936]: cluster 2021-03-24T21:54:42.146036 2021-03-24T21:54:43.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:43 smithi125 conmon[19936]: +0000 mon.a (mon.0) 653 : cluster [DBG] osdmap e113: 8 total, 8 up, 6 in 2021-03-24T21:54:43.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:43 smithi125 conmon[19936]: audit 2021-03-24T21:54: 2021-03-24T21:54:43.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:43 smithi125 conmon[19936]: 42.287276+0000 mon.c (mon.2) 25 : audit [INF] from='client.? v1:172.21.15.125:0/3674138739' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T21:54:43.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:43 smithi125 conmon[19936]: audit 2021-03-24T21:54:43.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:43 smithi125 conmon[19936]: 2021-03 2021-03-24T21:54:43.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:43 smithi125 conmon[19936]: -24T21:54:42.287760+0000 mon.a 2021-03-24T21:54:43.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:43 smithi125 conmon[19936]: (mon.0) 654 : audit [INF] from='client.34691 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T21:54:43.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:43 smithi125 conmon[19936]: 2021-03-24T21:54:43.424 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:54:43.445 INFO:tasks.rados.rados.0.smithi125.stdout:82: finishing copy_from to smithi12556470-42 2021-03-24T21:54:43.445 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 42 v 9 (ObjNum 27 snap 0 seq_num 27) dirty exists 2021-03-24T21:54:43.446 INFO:tasks.rados.rados.0.smithi125.stdout:82: finishing copy_from racing read to smithi12556470-42 2021-03-24T21:54:43.446 INFO:tasks.rados.rados.0.smithi125.stdout:83: finishing rollback tid 0 to smithi12556470-41 2021-03-24T21:54:43.446 INFO:tasks.rados.rados.0.smithi125.stdout:83: finishing rollback tid 1 to smithi12556470-41 2021-03-24T21:54:43.446 INFO:tasks.rados.rados.0.smithi125.stdout:83: finishing rollback tid 2 to smithi12556470-41 2021-03-24T21:54:43.446 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 41 v 16 (ObjNum 40 snap 0 seq_num 40) dirty exists 2021-03-24T21:54:43.447 INFO:tasks.rados.rados.0.smithi125.stdout:80: done (3 left) 2021-03-24T21:54:43.447 INFO:tasks.rados.rados.0.smithi125.stdout:81: done (2 left) 2021-03-24T21:54:43.447 INFO:tasks.rados.rados.0.smithi125.stdout:82: done (1 left) 2021-03-24T21:54:43.447 INFO:tasks.rados.rados.0.smithi125.stdout:83: done (0 left) 2021-03-24T21:54:43.448 INFO:tasks.rados.rados.0.smithi125.stdout:84: read oid 10 snap -1 2021-03-24T21:54:43.448 INFO:tasks.rados.rados.0.smithi125.stdout:84: expect (ObjNum 9 snap 0 seq_num 9) 2021-03-24T21:54:43.448 INFO:tasks.rados.rados.0.smithi125.stdout:85: read oid 24 snap -1 2021-03-24T21:54:43.448 INFO:tasks.rados.rados.0.smithi125.stdout:85: expect (ObjNum 23 snap 0 seq_num 23) 2021-03-24T21:54:43.448 INFO:tasks.rados.rados.0.smithi125.stdout:86: read oid 49 snap 3 2021-03-24T21:54:43.449 INFO:tasks.rados.rados.0.smithi125.stdout:86: expect (ObjNum 48 snap 0 seq_num 48) 2021-03-24T21:54:43.449 INFO:tasks.rados.rados.0.smithi125.stdout:87: read oid 37 snap -1 2021-03-24T21:54:43.449 INFO:tasks.rados.rados.0.smithi125.stdout:87: expect (ObjNum 36 snap 0 seq_num 36) 2021-03-24T21:54:43.449 INFO:tasks.rados.rados.0.smithi125.stdout:88: copy_from oid 9 from oid 5 current snap is 4 2021-03-24T21:54:43.449 INFO:tasks.rados.rados.0.smithi125.stdout:89: read oid 2 snap -1 2021-03-24T21:54:43.450 INFO:tasks.rados.rados.0.smithi125.stdout:89: expect deleted 2021-03-24T21:54:43.450 INFO:tasks.rados.rados.0.smithi125.stdout:90: read oid 34 snap -1 2021-03-24T21:54:43.450 INFO:tasks.rados.rados.0.smithi125.stdout:90: expect deleted 2021-03-24T21:54:43.450 INFO:tasks.rados.rados.0.smithi125.stdout:91: write oid 35 current snap is 4 2021-03-24T21:54:43.451 INFO:tasks.rados.rados.0.smithi125.stdout:91: seq_num 60 ranges {608420=529411,1778096=561011,2881850=126570} 2021-03-24T21:54:43.459 INFO:tasks.rados.rados.0.smithi125.stdout:91: writing smithi12556470-35 from 608420 to 1137831 tid 1 2021-03-24T21:54:43.468 INFO:tasks.rados.rados.0.smithi125.stdout:91: writing smithi12556470-35 from 1778096 to 2339107 tid 2 2021-03-24T21:54:43.472 INFO:tasks.rados.rados.0.smithi125.stdout:91: writing smithi12556470-35 from 2881850 to 3008420 tid 3 2021-03-24T21:54:43.473 INFO:tasks.rados.rados.0.smithi125.stdout:92: read oid 17 snap -1 2021-03-24T21:54:43.473 INFO:tasks.rados.rados.0.smithi125.stdout:92: expect (ObjNum 16 snap 0 seq_num 16) 2021-03-24T21:54:43.473 INFO:tasks.rados.rados.0.smithi125.stdout:93: write (excl) oid 3 current snap is 4 2021-03-24T21:54:43.473 INFO:tasks.rados.rados.0.smithi125.stdout:93: seq_num 61 ranges {625227=721013,1952931=704669} 2021-03-24T21:54:43.480 INFO:tasks.rados.rados.0.smithi125.stdout:93: writing smithi12556470-3 from 625227 to 1346240 tid 1 2021-03-24T21:54:43.484 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:54:43.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:43.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:54:43.489 INFO:tasks.rados.rados.0.smithi125.stdout:93: writing smithi12556470-3 from 1952931 to 2657600 tid 2 2021-03-24T21:54:43.489 INFO:tasks.rados.rados.0.smithi125.stdout:94: write oid 43 current snap is 4 2021-03-24T21:54:43.490 INFO:tasks.rados.rados.0.smithi125.stdout:94: seq_num 62 ranges {642034=796262,1895060=564680} 2021-03-24T21:54:43.497 INFO:tasks.rados.rados.0.smithi125.stdout:94: writing smithi12556470-43 from 642034 to 1438296 tid 1 2021-03-24T21:54:43.503 INFO:tasks.rados.rados.0.smithi125.stdout:94: writing smithi12556470-43 from 1895060 to 2459740 tid 2 2021-03-24T21:54:43.503 INFO:tasks.rados.rados.0.smithi125.stdout:95: read oid 31 snap -1 2021-03-24T21:54:43.503 INFO:tasks.rados.rados.0.smithi125.stdout:95: expect (ObjNum 30 snap 0 seq_num 30) 2021-03-24T21:54:43.504 INFO:tasks.rados.rados.0.smithi125.stdout:96: copy_from oid 28 from oid 41 current snap is 4 2021-03-24T21:54:43.504 INFO:tasks.rados.rados.0.smithi125.stdout:84: expect (ObjNum 9 snap 0 seq_num 9) 2021-03-24T21:54:43.584 INFO:tasks.rados.rados.0.smithi125.stdout:85: expect (ObjNum 23 snap 0 seq_num 23) 2021-03-24T21:54:43.590 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:54:43.644 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:54:43.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:43.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:54:43.667 INFO:tasks.rados.rados.0.smithi125.stdout:88: finishing copy_from to smithi12556470-9 2021-03-24T21:54:43.667 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 9 v 3 (ObjNum 4 snap 0 seq_num 4) dirty exists 2021-03-24T21:54:43.668 INFO:tasks.rados.rados.0.smithi125.stdout:88: finishing copy_from racing read to smithi12556470-9 2021-03-24T21:54:43.668 INFO:tasks.rados.rados.0.smithi125.stdout:87: expect (ObjNum 36 snap 0 seq_num 36) 2021-03-24T21:54:43.749 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:43.753 INFO:tasks.rados.rados.0.smithi125.stdout:91: finishing write tid 1 to smithi12556470-35 2021-03-24T21:54:43.753 INFO:tasks.rados.rados.0.smithi125.stdout:86: expect (ObjNum 48 snap 0 seq_num 48) 2021-03-24T21:54:43.802 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:43.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:43.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:54:43.846 INFO:tasks.rados.rados.0.smithi125.stdout:91: finishing write tid 2 to smithi12556470-35 2021-03-24T21:54:43.846 INFO:tasks.rados.rados.0.smithi125.stdout:91: finishing write tid 3 to smithi12556470-35 2021-03-24T21:54:43.846 INFO:tasks.rados.rados.0.smithi125.stdout:91: finishing write tid 4 to smithi12556470-35 2021-03-24T21:54:43.847 INFO:tasks.rados.rados.0.smithi125.stdout:91: finishing write tid 5 to smithi12556470-35 2021-03-24T21:54:43.847 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 35 v 8 (ObjNum 60 snap 4 seq_num 60) dirty exists 2021-03-24T21:54:43.847 INFO:tasks.rados.rados.0.smithi125.stdout:91: left oid 35 (ObjNum 60 snap 4 seq_num 60) 2021-03-24T21:54:43.847 INFO:tasks.rados.rados.0.smithi125.stdout:92: expect (ObjNum 16 snap 0 seq_num 16) 2021-03-24T21:54:43.906 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:54:43.922 INFO:tasks.rados.rados.0.smithi125.stdout:93: finishing write tid 1 to smithi12556470-3 2021-03-24T21:54:43.923 INFO:tasks.rados.rados.0.smithi125.stdout:93: finishing write tid 2 to smithi12556470-3 2021-03-24T21:54:43.923 INFO:tasks.rados.rados.0.smithi125.stdout:93: finishing write tid 3 to smithi12556470-3 2021-03-24T21:54:43.923 INFO:tasks.rados.rados.0.smithi125.stdout:93: finishing write tid 4 to smithi12556470-3 2021-03-24T21:54:43.923 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 3 v 17 (ObjNum 61 snap 4 seq_num 61) dirty exists 2021-03-24T21:54:43.924 INFO:tasks.rados.rados.0.smithi125.stdout:93: left oid 3 (ObjNum 61 snap 4 seq_num 61) 2021-03-24T21:54:43.924 INFO:tasks.rados.rados.0.smithi125.stdout:94: finishing write tid 1 to smithi12556470-43 2021-03-24T21:54:43.924 INFO:tasks.rados.rados.0.smithi125.stdout:96: finishing copy_from to smithi12556470-28 2021-03-24T21:54:43.924 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 28 v 16 (ObjNum 40 snap 0 seq_num 40) dirty exists 2021-03-24T21:54:43.925 INFO:tasks.rados.rados.0.smithi125.stdout:96: finishing copy_from racing read to smithi12556470-28 2021-03-24T21:54:43.925 INFO:tasks.rados.rados.0.smithi125.stdout:84: done (12 left) 2021-03-24T21:54:43.925 INFO:tasks.rados.rados.0.smithi125.stdout:85: done (11 left) 2021-03-24T21:54:43.925 INFO:tasks.rados.rados.0.smithi125.stdout:86: done (10 left) 2021-03-24T21:54:43.925 INFO:tasks.rados.rados.0.smithi125.stdout:87: done (9 left) 2021-03-24T21:54:43.926 INFO:tasks.rados.rados.0.smithi125.stdout:88: done (8 left) 2021-03-24T21:54:43.926 INFO:tasks.rados.rados.0.smithi125.stdout:89: done (7 left) 2021-03-24T21:54:43.926 INFO:tasks.rados.rados.0.smithi125.stdout:90: done (6 left) 2021-03-24T21:54:43.926 INFO:tasks.rados.rados.0.smithi125.stdout:91: done (5 left) 2021-03-24T21:54:43.926 INFO:tasks.rados.rados.0.smithi125.stdout:92: done (4 left) 2021-03-24T21:54:43.927 INFO:tasks.rados.rados.0.smithi125.stdout:93: done (3 left) 2021-03-24T21:54:43.927 INFO:tasks.rados.rados.0.smithi125.stdout:96: done (2 left) 2021-03-24T21:54:43.927 INFO:tasks.rados.rados.0.smithi125.stdout:97: snap_remove snap 3 2021-03-24T21:54:43.962 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:54:43.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:43.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:54:44.068 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:44.085 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T21:54:44.085 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T21:54:44.085 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T21:54:44.085 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T21:54:44.086 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T21:54:44.096 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T21:54:44.123 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T21:54:44.124 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:44.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:44.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:54:44.143 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:54:44.149 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T21:54:44.152 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-24T21:54:44.152 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-24T21:54:44.152 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-24T21:54:44.153 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-24T21:54:44.153 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-24T21:54:44.153 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-24T21:54:44.153 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-24T21:54:44.154 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-24T21:54:44.160 INFO:tasks.rados.rados.0.smithi125.stdout:97: done (2 left) 2021-03-24T21:54:44.160 INFO:tasks.rados.rados.0.smithi125.stdout:98: write oid 13 current snap is 4 2021-03-24T21:54:44.161 INFO:tasks.rados.rados.0.smithi125.stdout:98: seq_num 63 ranges {658841=471511,1553542=708338,2964119=94722} 2021-03-24T21:54:44.169 INFO:tasks.rados.rados.0.smithi125.stdout:98: writing smithi12556470-13 from 658841 to 1130352 tid 1 2021-03-24T21:54:44.176 INFO:tasks.rados.rados.0.smithi125.stdout:98: writing smithi12556470-13 from 1553542 to 2261880 tid 2 2021-03-24T21:54:44.181 INFO:tasks.rados.rados.0.smithi125.stdout:98: writing smithi12556470-13 from 2964119 to 3058841 tid 3 2021-03-24T21:54:44.181 INFO:tasks.rados.rados.0.smithi125.stdout:99: read oid 28 snap -1 2021-03-24T21:54:44.181 INFO:tasks.rados.rados.0.smithi125.stdout:99: expect (ObjNum 40 snap 0 seq_num 40) 2021-03-24T21:54:44.182 INFO:tasks.rados.rados.0.smithi125.stdout:100: rollback oid 10 current snap is 4 2021-03-24T21:54:44.182 INFO:tasks.rados.rados.0.smithi125.stdout:100: done (4 left) 2021-03-24T21:54:44.182 INFO:tasks.rados.rados.0.smithi125.stdout:101: read oid 38 snap -1 2021-03-24T21:54:44.182 INFO:tasks.rados.rados.0.smithi125.stdout:101: expect (ObjNum 37 snap 0 seq_num 37) 2021-03-24T21:54:44.183 INFO:tasks.rados.rados.0.smithi125.stdout:102: snap_create 2021-03-24T21:54:44.183 INFO:tasks.rados.rados.0.smithi125.stdout:94: finishing write tid 2 to smithi12556470-43 2021-03-24T21:54:44.183 INFO:tasks.rados.rados.0.smithi125.stdout:94: finishing write tid 3 to smithi12556470-43 2021-03-24T21:54:44.184 INFO:tasks.rados.rados.0.smithi125.stdout:94: finishing write tid 4 to smithi12556470-43 2021-03-24T21:54:44.184 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 43 v 24 (ObjNum 62 snap 4 seq_num 62) dirty exists 2021-03-24T21:54:44.184 INFO:tasks.rados.rados.0.smithi125.stdout:94: left oid 43 (ObjNum 62 snap 4 seq_num 62) 2021-03-24T21:54:44.184 INFO:tasks.rados.rados.0.smithi125.stdout:95: expect (ObjNum 30 snap 0 seq_num 30) 2021-03-24T21:54:44.187 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:54:44.231 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:44.285 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:44.286 INFO:tasks.rados.rados.0.smithi125.stdout:98: finishing write tid 1 to smithi12556470-13 2021-03-24T21:54:44.286 INFO:tasks.rados.rados.0.smithi125.stdout:98: finishing write tid 2 to smithi12556470-13 2021-03-24T21:54:44.286 INFO:tasks.rados.rados.0.smithi125.stdout:98: finishing write tid 3 to smithi12556470-13 2021-03-24T21:54:44.287 INFO:tasks.rados.rados.0.smithi125.stdout:98: finishing write tid 4 to smithi12556470-13 2021-03-24T21:54:44.287 INFO:tasks.rados.rados.0.smithi125.stdout:98: finishing write tid 5 to smithi12556470-13 2021-03-24T21:54:44.287 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 13 v 32 (ObjNum 63 snap 4 seq_num 63) dirty exists 2021-03-24T21:54:44.287 INFO:tasks.rados.rados.0.smithi125.stdout:98: left oid 13 (ObjNum 63 snap 4 seq_num 63) 2021-03-24T21:54:44.288 INFO:tasks.rados.rados.0.smithi125.stdout:101: expect (ObjNum 37 snap 0 seq_num 37) 2021-03-24T21:54:44.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:44.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:54:44.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:44 smithi142 conmon[20125]: cluster 2021-03-24T21:54: 2021-03-24T21:54:44.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:44 smithi142 conmon[20125]: 42.711456+0000 mgr.y (mgr.14144) 314 : cluster [DBG] pgmap v332: 148 pgs: 1 active+recovery_wait, 5 active+recovering, 12 active+recovery_wait+degraded, 3 activating+degraded, 1 activating, 125 active+clean, 1 active+degraded; 117 MiB data, 213 MiB used, 536 GiB / 536 GiB avail; 8.5 KiB/s rd, 11 MiB/s wr, 59 op/s; 103/1107 objects degraded (9.304%); 16/1107 objects misplaced (1.445%); 3.5 MiB/s, 17 objects/s recovering 2021-03-24T21:54:44.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:44 smithi142 conmon[20125]: cluster 2021-03-24T21:54:44.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:44 smithi142 conmon[20125]: 2021-03-24T21:54:43.143184+0000 mon.a 2021-03-24T21:54:44.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:44 smithi142 conmon[20125]: (mon.0) 655 : cluster 2021-03-24T21:54:44.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:44 smithi142 conmon[20125]: [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:54:44.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:44 smithi142 conmon[20125]: audit 2021 2021-03-24T21:54:44.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:44 smithi142 conmon[20125]: -03-24T21:54:43. 2021-03-24T21:54:44.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:44 smithi142 conmon[20125]: 159444+0000 mon.a (mon.0 2021-03-24T21:54:44.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:44 smithi142 conmon[20125]: ) 656 : audit [INF] from='client.34691 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T21:54:44.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:44 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:54:44.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:44 smithi142 conmon[20125]: -03-24T21:54:43 2021-03-24T21:54:44.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:44 smithi142 conmon[20125]: .159576+0000 mon.a (mon. 2021-03-24T21:54:44.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:44 smithi142 conmon[20125]: 0) 657 : cluster [DBG] osdmap e114: 8 total, 8 up, 6 in 2021-03-24T21:54:44.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:44 smithi142 conmon[20125]: 2021-03-24T21:54:44.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:44 smithi142 conmon[20125]: cluster 2021-03-24T21:54:44.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:44 smithi142 conmon[20125]: 2021- 2021-03-24T21:54:44.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:44 smithi142 conmon[20125]: 03-24T21:54:43 2021-03-24T21:54:44.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:44 smithi142 conmon[20125]: .286324+0000 osd.2 (osd. 2021-03-24T21:54:44.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:44 smithi142 conmon[20125]: 2) 22 : cluster [DBG] 3.4 scrub ok 2021-03-24T21:54:44.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:44 smithi142 conmon[20125]: 2021-03-24T21:54:44.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:44 smithi142 conmon[20125]: cluster 2021-03-24T21:54:44.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:44 smithi142 conmon[20125]: 2021-03-24T21:54 2021-03-24T21:54:44.373 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:44 smithi142 conmon[20125]: :43.482301+0000 osd.3 ( 2021-03-24T21:54:44.373 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:44 smithi142 conmon[20125]: osd.3) 39 : cluster [DBG] 2021-03-24T21:54:44.373 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:44 smithi142 conmon[20125]: 6.1b scrub ok 2021-03-24T21:54:44.373 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:44 smithi142 conmon[20125]: audit 2021 2021-03-24T21:54:44.374 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:44 smithi142 conmon[20125]: -03-24T21:54:43 2021-03-24T21:54:44.374 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:44 smithi142 conmon[20125]: .961263+0000 mon.a (mon. 2021-03-24T21:54:44.374 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:44 smithi142 conmon[20125]: 0) 658 : audit [DBG] 2021-03-24T21:54:44.374 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:44 smithi142 conmon[20125]: from='client.? v1:172.21.15.125:0/3258619749' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:54:44.375 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:44 smithi125 conmon[24927]: cluster 2021-03-24T21:54:42.711456+0000 2021-03-24T21:54:44.375 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:44 smithi125 conmon[24927]: mgr.y (mgr.14144) 314 : cluster [DBG] pgmap v332: 148 pgs: 1 active+recovery_wait, 5 active+recovering, 12 active+recovery_wait+degraded, 3 activating+degraded, 1 activating, 125 active+clean, 1 active+degraded; 117 MiB data, 213 MiB used, 536 GiB / 536 GiB avail; 8.5 KiB/s rd, 11 MiB/s wr, 59 op/s; 103/1107 objects degraded (9.304%); 16/1107 objects misplaced (1.445%); 3.5 MiB/s, 17 objects/s recovering 2021-03-24T21:54:44.375 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:44 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:54:44.376 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:44 smithi125 conmon[24927]: 24T21:54:43.143184+0000 mon.a (mon.0) 655 : cluster [WRN] 2021-03-24T21:54:44.376 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:44 smithi125 conmon[24927]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:54:44.376 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:44 smithi125 conmon[24927]: audit 2021-03-24T 2021-03-24T21:54:44.376 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:44 smithi125 conmon[24927]: 21:54:43.159444+0000 mon.a (mon.0 2021-03-24T21:54:44.377 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:44 smithi125 conmon[24927]: ) 656 : audit [INF] from='client.34691 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T21:54:44.377 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:44 smithi125 conmon[24927]: cluster 2021-03-24T21:54:43. 2021-03-24T21:54:44.377 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:44 smithi125 conmon[24927]: 159576+0000 mon.a (mon.0) 657 : cluster [DBG] osdmap e114: 8 total, 8 up, 6 in 2021-03-24T21:54:44.377 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:44 smithi125 conmon[24927]: cluster 2021-03-24T21:54 2021-03-24T21:54:44.377 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:44 smithi125 conmon[24927]: :43.286324+0000 osd.2 (osd.2) 22 : 2021-03-24T21:54:44.378 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:44 smithi125 conmon[24927]: cluster [DBG] 3.4 scrub ok 2021-03-24T21:54:44.378 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:44 smithi125 conmon[24927]: cluster 2021-03-24T21:54: 2021-03-24T21:54:44.378 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:44 smithi125 conmon[24927]: 43.482301+0000 osd.3 (osd.3) 39 2021-03-24T21:54:44.378 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:44 smithi125 conmon[24927]: : cluster [DBG] 6.1b scrub ok 2021-03-24T21:54:44.379 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:44 smithi125 conmon[24927]: audit 2021-03-24T21:54: 2021-03-24T21:54:44.379 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:44 smithi125 conmon[24927]: 43.961263+0000 mon.a (mon.0) 658 : 2021-03-24T21:54:44.379 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:44 smithi125 conmon[24927]: audit [DBG] from='client.? v1:172.21.15.125:0/3258619749' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:54:44.379 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:44 smithi125 conmon[19936]: cluster 2021-03-24T21:54:42.711456+0000 mgr.y (mgr. 2021-03-24T21:54:44.380 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:44 smithi125 conmon[19936]: 14144) 314 : cluster [DBG] pgmap v332: 148 pgs: 1 active+recovery_wait, 5 active+recovering, 12 active+recovery_wait+degraded, 3 activating+degraded, 1 activating, 125 active+clean, 1 active+degraded; 117 MiB data, 213 MiB used, 536 GiB / 536 GiB avail; 8.5 KiB/s rd, 11 MiB/s wr, 59 op/s; 103/1107 objects degraded (9.304%); 16/1107 objects misplaced (1.445%); 3.5 MiB/s, 17 objects/s recovering 2021-03-24T21:54:44.380 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:44 smithi125 conmon[19936]: cluster 2021-03-24T21:54:43.143184+0000 mon.a (mon.0) 655 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:54:44.380 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:44 smithi125 conmon[19936]: audit 2021-03-24 2021-03-24T21:54:44.380 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:44 smithi125 conmon[19936]: T21:54:43.159444+0000 mon.a (mon.0) 656 : audit [INF] from='client.34691 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T21:54:44.381 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:44 smithi125 conmon[19936]: cluster 2021-03-24T21:54:43.159576+0000 mon.a (mon.0) 657 : cluster [DBG] 2021-03-24T21:54:44.381 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:44 smithi125 conmon[19936]: osdmap e114: 8 total, 8 up, 6 in 2021-03-24T21:54:44.381 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:44 smithi125 conmon[19936]: cluster 2021-03-24T21:54:43.286324+0000 osd.2 (osd.2) 22 : cluster 2021-03-24T21:54:44.381 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:44 smithi125 conmon[19936]: [DBG] 3.4 scrub ok 2021-03-24T21:54:44.381 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:44 smithi125 conmon[19936]: cluster 2021-03-24T21:54:43.482301+0000 osd.3 (osd. 2021-03-24T21:54:44.382 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:44 smithi125 conmon[19936]: 3) 39 : cluster [DBG] 6.1b scrub ok 2021-03-24T21:54:44.382 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:44 smithi125 conmon[19936]: audit 2021-03-24T21:54:43.961263+0000 mon.a 2021-03-24T21:54:44.382 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:44 smithi125 conmon[19936]: (mon.0) 658 : audit [DBG] from='client.? v1:172.21.15.125:0/3258619749' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:54:44.382 INFO:tasks.rados.rados.0.smithi125.stdout:99: expect (ObjNum 40 snap 0 seq_num 40) 2021-03-24T21:54:44.390 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:54:44.444 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:54:44.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:44.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:54:44.549 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:44.604 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:44.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:44.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:54:44.711 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:44.766 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:44.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:44.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:54:44.871 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:44.929 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:44.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:44.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:54:45.036 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:45.080 INFO:tasks.rados.rados.0.smithi125.stdout:94: done (5 left) 2021-03-24T21:54:45.081 INFO:tasks.rados.rados.0.smithi125.stdout:95: done (4 left) 2021-03-24T21:54:45.081 INFO:tasks.rados.rados.0.smithi125.stdout:98: done (3 left) 2021-03-24T21:54:45.081 INFO:tasks.rados.rados.0.smithi125.stdout:99: done (2 left) 2021-03-24T21:54:45.081 INFO:tasks.rados.rados.0.smithi125.stdout:101: done (1 left) 2021-03-24T21:54:45.082 INFO:tasks.rados.rados.0.smithi125.stdout:102: done (0 left) 2021-03-24T21:54:45.082 INFO:tasks.rados.rados.0.smithi125.stdout:103: copy_from oid 31 from oid 29 current snap is 5 2021-03-24T21:54:45.082 INFO:tasks.rados.rados.0.smithi125.stdout:104: read oid 50 snap -1 2021-03-24T21:54:45.082 INFO:tasks.rados.rados.0.smithi125.stdout:104: expect (ObjNum 51 snap 0 seq_num 51) 2021-03-24T21:54:45.083 INFO:tasks.rados.rados.0.smithi125.stdout:105: read oid 35 snap -1 2021-03-24T21:54:45.083 INFO:tasks.rados.rados.0.smithi125.stdout:105: expect (ObjNum 60 snap 4 seq_num 60) 2021-03-24T21:54:45.083 INFO:tasks.rados.rados.0.smithi125.stdout:106: snap_remove snap 4 2021-03-24T21:54:45.096 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:45.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:45.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:54:45.202 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:45.255 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:45.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:45.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:54:45.286 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:45 smithi125 conmon[19936]: cluster 2021-03-24T21:54:43.406871+0000 osd.0 (osd.0) 36 : cluster 2021-03-24T21:54:45.286 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:45 smithi125 conmon[19936]: [DBG] 3.1b scrub ok 2021-03-24T21:54:45.286 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:45 smithi125 conmon[19936]: cluster 2021-03-24T21:54:43.877596+0000 osd.5 (osd.5) 32 : cluster 2021-03-24T21:54:45.287 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:45 smithi125 conmon[19936]: [DBG] 3.1d scrub ok 2021-03-24T21:54:45.287 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:45 smithi125 conmon[19936]: cluster 2021-03-24T21:54:44 2021-03-24T21:54:45.287 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:45 smithi125 conmon[19936]: .159471+0000 mon.a (mon.0) 659 : cluster [DBG] osdmap e115: 8 total, 8 up, 6 in 2021-03-24T21:54:45.287 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:45 smithi125 conmon[19936]: cluster 2021-03-24T21:54:45.068543+0000 mon.a (mon. 2021-03-24T21:54:45.288 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:45 smithi125 conmon[19936]: 0) 660 : cluster [WRN] Health check update: Degraded data redundancy: 103/1107 objects degraded (9.304%), 16 pgs degraded (PG_DEGRADED) 2021-03-24T21:54:45.288 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:45 smithi125 conmon[19936]: cluster 2021-03-24T21:54:45.288 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:45 smithi125 conmon[19936]: 2021-03-24T21:54:45.077767+0000 mon.a (mon.0) 661 : cluster [DBG] osdmap e116: 8 total, 8 up, 6 in 2021-03-24T21:54:45.289 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:45 smithi125 conmon[24927]: cluster 2021-03-24T21:54 2021-03-24T21:54:45.289 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:45 smithi125 conmon[24927]: :43.406871+0000 osd.0 (osd.0) 36 : cluster [DBG] 3.1b scrub ok 2021-03-24T21:54:45.289 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:45 smithi125 conmon[24927]: cluster 2021-03-24T21:54: 2021-03-24T21:54:45.289 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:45 smithi125 conmon[24927]: 43.877596+0000 osd.5 (osd.5) 32 : cluster 2021-03-24T21:54:45.290 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:45 smithi125 conmon[24927]: [DBG] 3.1d scrub ok 2021-03-24T21:54:45.290 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:45 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:54:45.290 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:45 smithi125 conmon[24927]: :54:44.159471+0000 mon.a (mon.0 2021-03-24T21:54:45.290 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:45 smithi125 conmon[24927]: ) 659 : cluster [DBG] osdmap e115: 8 total, 8 up, 6 in 2021-03-24T21:54:45.291 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:45 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:54:45.291 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:45 smithi125 conmon[24927]: -24T21:54:45.068543+0000 mon.a ( 2021-03-24T21:54:45.291 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:45 smithi125 conmon[24927]: mon.0) 660 : cluster [WRN] Health check update: Degraded data redundancy: 103/1107 objects degraded (9.304%), 16 pgs degraded (PG_DEGRADED) 2021-03-24T21:54:45.291 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:45 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:54:45.291 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:45 smithi125 conmon[24927]: 03-24T21:54:45.077767+0000 mon.a 2021-03-24T21:54:45.292 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:45 smithi125 conmon[24927]: (mon.0) 661 : cluster [DBG] osdmap e116: 8 total, 8 up, 6 in 2021-03-24T21:54:45.362 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:45.374 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:45 smithi142 conmon[20125]: cluster 2021-03-24T21:54:43.406871+0000 osd.0 (osd.0) 36 : 2021-03-24T21:54:45.413 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:45.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:45.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:54:45.518 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:54:45.574 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:54:45.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:45.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:54:45.681 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:45.746 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:45.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:45.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:54:45.852 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:45.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:45 smithi142 conmon[20125]: cluster [DBG] 3.1b scrub ok 2021-03-24T21:54:45.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:45 smithi142 conmon[20125]: cluster 2021-03-24T21:54:43.877596+0000 osd.5 2021-03-24T21:54:45.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:45 smithi142 conmon[20125]: (osd.5) 32 : cluster [DBG] 3.1d scrub ok 2021-03-24T21:54:45.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:45 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:54:45.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:45 smithi142 conmon[20125]: -03-24T21:54:44.159471 2021-03-24T21:54:45.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:45 smithi142 conmon[20125]: +0000 mon.a (mon.0) 659 : cluster 2021-03-24T21:54:45.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:45 smithi142 conmon[20125]: [DBG] osdmap e115: 8 total, 8 up, 6 in 2021-03-24T21:54:45.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:45 smithi142 conmon[20125]: cluster 2021-03-24T21:54:45.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:45 smithi142 conmon[20125]: 2021-03-24T21:54 2021-03-24T21:54:45.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:45 smithi142 conmon[20125]: :45.068543+0000 mon.a (mon.0) 660 : cluster [WRN] Health check update: Degraded data redundancy: 103/1107 objects degraded (9.304%), 16 pgs degraded (PG_DEGRADED) 2021-03-24T21:54:45.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:45 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:54:45.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:45 smithi142 conmon[20125]: 03-24T21:54:45.077767 2021-03-24T21:54:45.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:45 smithi142 conmon[20125]: +0000 mon.a (mon.0) 661 2021-03-24T21:54:45.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:45 smithi142 conmon[20125]: : cluster [DBG] osdmap e116: 8 total, 8 up, 6 in 2021-03-24T21:54:45.910 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:45.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:45.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:54:46.016 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:46.075 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:46.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:46.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:54:46.083 INFO:tasks.rados.rados.0.smithi125.stdout:106: done (3 left) 2021-03-24T21:54:46.084 INFO:tasks.rados.rados.0.smithi125.stdout:107: read oid 34 snap -1 2021-03-24T21:54:46.084 INFO:tasks.rados.rados.0.smithi125.stdout:107: expect deleted 2021-03-24T21:54:46.084 INFO:tasks.rados.rados.0.smithi125.stdout:108: copy_from oid 12 from oid 26 current snap is 5 2021-03-24T21:54:46.084 INFO:tasks.rados.rados.0.smithi125.stdout:105: expect (ObjNum 60 snap 4 seq_num 60) 2021-03-24T21:54:46.169 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-24T21:54:46.184 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:46.236 INFO:tasks.rados.rados.0.smithi125.stdout:104: expect (ObjNum 51 snap 0 seq_num 51) 2021-03-24T21:54:46.266 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:46.270 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:46 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:54:46.270 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:46 smithi125 conmon[19936]: 21:54:44.392797+0000 osd.0 (osd.0) 37 : cluster [DBG] 6.16 scrub ok 2021-03-24T21:54:46.271 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:46 smithi125 conmon[19936]: cluster 2021-03-24T21:54:46.271 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:46 smithi125 conmon[19936]: 2021-03-24T 2021-03-24T21:54:46.271 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:46 smithi125 conmon[19936]: 21:54:44.712362+0000 2021-03-24T21:54:46.271 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:46 smithi125 conmon[19936]: mgr.y (mgr.14144) 315 2021-03-24T21:54:46.271 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:46 smithi125 conmon[19936]: : cluster [DBG] pgmap v335: 148 pgs: 148 active+clean; 119 MiB data, 255 MiB used, 536 GiB / 536 GiB avail; 21 MiB/s rd, 4.7 MiB/s wr, 33 op/s; 3.0 MiB/s, 34 objects/s recovering 2021-03-24T21:54:46.272 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:46 smithi125 conmon[19936]: cluster 2021-03-24T21:54:46.272 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:46 smithi125 conmon[19936]: 2021-03-24 2021-03-24T21:54:46.272 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:46 smithi125 conmon[19936]: T21:54:44.878911 2021-03-24T21:54:46.272 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:46 smithi125 conmon[19936]: +0000 osd.5 (osd.5) 2021-03-24T21:54:46.273 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:46 smithi125 conmon[19936]: 33 : cluster [DBG] 5.1b scrub ok 2021-03-24T21:54:46.273 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:46 smithi125 conmon[19936]: cluster 2021-03-24T21:54:46.273 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:46 smithi125 conmon[19936]: 2021-03-24 2021-03-24T21:54:46.273 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:46 smithi125 conmon[19936]: T21:54:44.933251 2021-03-24T21:54:46.273 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:46 smithi125 conmon[19936]: +0000 osd.7 (osd.7) 2021-03-24T21:54:46.274 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:46 smithi125 conmon[19936]: 31 : cluster [DBG] 6.3 scrub ok 2021-03-24T21:54:46.274 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:46 smithi125 conmon[19936]: cluster 2021-03-24T21:54:46.274 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:46 smithi125 conmon[19936]: 2021-03-24T21: 2021-03-24T21:54:46.274 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:46 smithi125 conmon[19936]: 54:45.155732+0000 mon.a ( 2021-03-24T21:54:46.275 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:46 smithi125 conmon[19936]: mon.0) 662 : cluster 2021-03-24T21:54:46.275 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:46 smithi125 conmon[19936]: [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 103/1107 objects degraded (9.304%), 16 pgs degraded) 2021-03-24T21:54:46.275 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:46 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:54:46.275 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:46 smithi125 conmon[19936]: -03-24T21 2021-03-24T21:54:46.275 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:46 smithi125 conmon[19936]: :54:46.078038 2021-03-24T21:54:46.276 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:46 smithi125 conmon[19936]: +0000 mon.a (mon.0 2021-03-24T21:54:46.276 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:46 smithi125 conmon[19936]: ) 663 : cluster [DBG] osdmap e117: 8 total, 8 up, 6 in 2021-03-24T21:54:46.276 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:46 smithi125 conmon[24927]: cluster 2021-03-24T21:54:46.277 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:46 smithi125 conmon[24927]: 2021-03-24T21:54:44.392797+0000 osd.0 (osd.0 2021-03-24T21:54:46.277 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:46 smithi125 conmon[24927]: ) 37 : cluster [DBG] 6.16 scrub ok 2021-03-24T21:54:46.277 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:46 smithi125 conmon[24927]: cluster 2021-03-24T21:54:44.712362+0000 mgr.y (mgr.14144) 315 : cluster [DBG] pgmap v335: 148 pgs: 148 active+clean; 119 MiB data, 255 MiB used, 536 GiB / 536 GiB avail; 21 MiB/s rd, 4.7 MiB/s wr, 33 op/s; 3.0 MiB/s, 34 objects/s recovering 2021-03-24T21:54:46.277 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:46 smithi125 conmon[24927]: cluster 2021-03-24T21:54:44.878911+0000 osd.5 (osd.5) 33 : cluster [DBG] 5.1b scrub ok 2021-03-24T21:54:46.278 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:46 smithi125 conmon[24927]: cluster 2021-03-24T21:54:46.278 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:46 smithi125 conmon[24927]: 2021-03-24T21:54:44.933251+0000 osd.7 (osd.7) 31 : cluster [DBG] 6.3 scrub ok 2021-03-24T21:54:46.278 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:46 smithi125 conmon[24927]: cluster 2021-03-24T21:54:46.278 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:46 smithi125 conmon[24927]: 2021-03-24T21:54:45.155732+0000 mon.a (mon.0) 662 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 103/1107 objects degraded (9.304%), 16 pgs degraded) 2021-03-24T21:54:46.279 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:46 smithi125 conmon[24927]: cluster 2021-03-24T21:54:46.279 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:46 smithi125 conmon[24927]: 2021-03-24T21:54:46.078038+0000 mon.a (mon.0) 663 : cluster [DBG] osdmap e117: 8 total, 8 up, 6 in 2021-03-24T21:54:46.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:46.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:54:46.381 INFO:tasks.rados.rados.0.smithi125.stdout:103: finishing copy_from to smithi12556470-31 2021-03-24T21:54:46.382 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 31 v 16 (ObjNum 59 snap 4 seq_num 59) dirty exists 2021-03-24T21:54:46.382 INFO:tasks.rados.rados.0.smithi125.stdout:103: finishing copy_from racing read to smithi12556470-31 2021-03-24T21:54:46.382 INFO:tasks.rados.rados.0.smithi125.stdout:103: done (4 left) 2021-03-24T21:54:46.383 INFO:tasks.rados.rados.0.smithi125.stdout:104: done (3 left) 2021-03-24T21:54:46.383 INFO:tasks.rados.rados.0.smithi125.stdout:105: done (2 left) 2021-03-24T21:54:46.383 INFO:tasks.rados.rados.0.smithi125.stdout:109: read oid 17 snap -1 2021-03-24T21:54:46.383 INFO:tasks.rados.rados.0.smithi125.stdout:109: expect (ObjNum 16 snap 0 seq_num 16) 2021-03-24T21:54:46.383 INFO:tasks.rados.rados.0.smithi125.stdout:110: copy_from oid 41 from oid 35 current snap is 5 2021-03-24T21:54:46.384 INFO:tasks.rados.rados.0.smithi125.stdout:108: finishing copy_from to smithi12556470-12 2021-03-24T21:54:46.384 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 12 v 3 (ObjNum 25 snap 0 seq_num 25) dirty exists 2021-03-24T21:54:46.384 INFO:tasks.rados.rados.0.smithi125.stdout:108: finishing copy_from racing read to smithi12556470-12 2021-03-24T21:54:46.384 INFO:tasks.rados.rados.0.smithi125.stdout:107: done (3 left) 2021-03-24T21:54:46.384 INFO:tasks.rados.rados.0.smithi125.stdout:108: done (2 left) 2021-03-24T21:54:46.385 INFO:tasks.rados.rados.0.smithi125.stdout:111: write oid 7 current snap is 5 2021-03-24T21:54:46.385 INFO:tasks.rados.rados.0.smithi125.stdout:111: seq_num 64 ranges {675648=546760,2012024=568349} 2021-03-24T21:54:46.385 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:46.388 INFO:tasks.rados.rados.0.smithi125.stdout:111: writing smithi12556470-7 from 675648 to 1222408 tid 1 2021-03-24T21:54:46.396 INFO:tasks.rados.rados.0.smithi125.stdout:111: writing smithi12556470-7 from 2012024 to 2580373 tid 2 2021-03-24T21:54:46.396 INFO:tasks.rados.rados.0.smithi125.stdout:112: copy_from oid 8 from oid 43 current snap is 5 2021-03-24T21:54:46.397 INFO:tasks.rados.rados.0.smithi125.stdout:109: expect (ObjNum 16 snap 0 seq_num 16) 2021-03-24T21:54:46.444 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:46.448 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:46 smithi142 conmon[20125]: cluster 2021-03-24T21 2021-03-24T21:54:46.448 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:46 smithi142 conmon[20125]: :54:44.392797+0000 osd.0 (osd.0) 37 : cluster [DBG] 6.16 scrub ok 2021-03-24T21:54:46.449 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:46 smithi142 conmon[20125]: cluster 2021-03-24T21:54:46.449 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:46 smithi142 conmon[20125]: 2021-03-24T21:54:44. 2021-03-24T21:54:46.449 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:46 smithi142 conmon[20125]: 712362+0000 mgr.y (mgr.14144) 315 : cluster [DBG] pgmap v335: 148 pgs: 148 active+clean; 119 MiB data, 255 MiB used, 536 GiB / 536 GiB avail; 21 MiB/s rd, 4.7 MiB/s wr, 33 op/s; 3.0 MiB/s, 34 objects/s recovering 2021-03-24T21:54:46.449 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:46 smithi142 conmon[20125]: 2021-03-24T21:54:46.450 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:46 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:54:46.450 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:46 smithi142 conmon[20125]: 03-24T21:54:44.878911+0000 osd.5 (osd.5) 33 : cluster [DBG] 2021-03-24T21:54:46.450 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:46 smithi142 conmon[20125]: 5.1b scrub ok 2021-03-24T21:54:46.450 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:46 smithi142 conmon[20125]: cluster 2021-03-24T21:54:46.450 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:46 smithi142 conmon[20125]: 2021-03-24T21:54:44.933251+0000 osd.7 2021-03-24T21:54:46.451 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:46 smithi142 conmon[20125]: (osd.7) 31 : cluster [DBG] 6.3 scrub ok 2021-03-24T21:54:46.451 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:46 smithi142 conmon[20125]: 2021-03-24T21:54:46.451 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:46 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:54:46.451 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:46 smithi142 conmon[20125]: 21:54:45.155732+0000 mon.a (mon 2021-03-24T21:54:46.452 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:46 smithi142 conmon[20125]: .0) 662 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 103/1107 objects degraded (9.304%), 16 pgs degraded) 2021-03-24T21:54:46.452 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:46 smithi142 conmon[20125]: cluster 2021-03-24T21:54:46.452 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:46 smithi142 conmon[20125]: 2021-03-24 2021-03-24T21:54:46.452 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:46 smithi142 conmon[20125]: T21:54:46.078038+0000 mon.a ( 2021-03-24T21:54:46.452 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:46 smithi142 conmon[20125]: mon.0) 663 : cluster [DBG] osdmap e117: 8 total, 8 up, 6 in 2021-03-24T21:54:46.453 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:46 smithi142 conmon[20125]: 2021-03-24T21:54:46.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:46.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:54:46.481 INFO:tasks.rados.rados.0.smithi125.stdout:111: finishing write tid 1 to smithi12556470-7 2021-03-24T21:54:46.482 INFO:tasks.rados.rados.0.smithi125.stdout:111: finishing write tid 2 to smithi12556470-7 2021-03-24T21:54:46.482 INFO:tasks.rados.rados.0.smithi125.stdout:111: finishing write tid 3 to smithi12556470-7 2021-03-24T21:54:46.482 INFO:tasks.rados.rados.0.smithi125.stdout:111: finishing write tid 4 to smithi12556470-7 2021-03-24T21:54:46.482 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 7 v 20 (ObjNum 64 snap 5 seq_num 64) dirty exists 2021-03-24T21:54:46.483 INFO:tasks.rados.rados.0.smithi125.stdout:111: left oid 7 (ObjNum 64 snap 5 seq_num 64) 2021-03-24T21:54:46.483 INFO:tasks.rados.rados.0.smithi125.stdout:110: finishing copy_from to smithi12556470-41 2021-03-24T21:54:46.483 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 41 v 8 (ObjNum 60 snap 4 seq_num 60) dirty exists 2021-03-24T21:54:46.483 INFO:tasks.rados.rados.0.smithi125.stdout:110: finishing copy_from racing read to smithi12556470-41 2021-03-24T21:54:46.484 INFO:tasks.rados.rados.0.smithi125.stdout:112: finishing copy_from to smithi12556470-8 2021-03-24T21:54:46.484 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 8 v 24 (ObjNum 62 snap 4 seq_num 62) dirty exists 2021-03-24T21:54:46.484 INFO:tasks.rados.rados.0.smithi125.stdout:112: finishing copy_from racing read to smithi12556470-8 2021-03-24T21:54:46.484 INFO:tasks.rados.rados.0.smithi125.stdout:109: done (3 left) 2021-03-24T21:54:46.484 INFO:tasks.rados.rados.0.smithi125.stdout:110: done (2 left) 2021-03-24T21:54:46.485 INFO:tasks.rados.rados.0.smithi125.stdout:111: done (1 left) 2021-03-24T21:54:46.485 INFO:tasks.rados.rados.0.smithi125.stdout:112: done (0 left) 2021-03-24T21:54:46.485 INFO:tasks.rados.rados.0.smithi125.stdout:113: write (excl) oid 45 current snap is 5 2021-03-24T21:54:46.485 INFO:tasks.rados.rados.0.smithi125.stdout:113: seq_num 65 ranges {692455=622009,2070506=712007} 2021-03-24T21:54:46.496 INFO:tasks.rados.rados.0.smithi125.stdout:113: writing smithi12556470-45 from 692455 to 1314464 tid 1 2021-03-24T21:54:46.505 INFO:tasks.rados.rados.0.smithi125.stdout:113: writing smithi12556470-45 from 2070506 to 2782513 tid 2 2021-03-24T21:54:46.505 INFO:tasks.rados.rados.0.smithi125.stdout:114: snap_create 2021-03-24T21:54:46.505 INFO:tasks.rados.rados.0.smithi125.stdout:113: finishing write tid 1 to smithi12556470-45 2021-03-24T21:54:46.509 INFO:tasks.rados.rados.0.smithi125.stdout:113: finishing write tid 2 to smithi12556470-45 2021-03-24T21:54:46.509 INFO:tasks.rados.rados.0.smithi125.stdout:113: finishing write tid 3 to smithi12556470-45 2021-03-24T21:54:46.510 INFO:tasks.rados.rados.0.smithi125.stdout:113: finishing write tid 4 to smithi12556470-45 2021-03-24T21:54:46.510 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 45 v 22 (ObjNum 65 snap 5 seq_num 65) dirty exists 2021-03-24T21:54:46.510 INFO:tasks.rados.rados.0.smithi125.stdout:113: left oid 45 (ObjNum 65 snap 5 seq_num 65) 2021-03-24T21:54:46.555 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:46.619 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:46.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:46.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:54:46.724 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:46.779 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:46.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:46.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:54:46.885 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:46.948 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:46.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:46.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:54:47.054 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:47.104 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:47.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:47.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:54:47.142 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-24T21:54:47.183 INFO:tasks.rados.rados.0.smithi125.stdout:113: done (1 left) 2021-03-24T21:54:47.184 INFO:tasks.rados.rados.0.smithi125.stdout:114: done (0 left) 2021-03-24T21:54:47.184 INFO:tasks.rados.rados.0.smithi125.stdout:115: snap_create 2021-03-24T21:54:47.210 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:47.260 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:47.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:47.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:54:47.367 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:54:47.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:47 smithi125 conmon[19936]: cluster 2021-03-24T21:54:45.900712 2021-03-24T21:54:47.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:47 smithi125 conmon[19936]: +0000 osd.5 (osd.5) 34 : cluster [DBG] 6.7 scrub ok 2021-03-24T21:54:47.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:47 smithi125 conmon[24927]: cluster 2021-03-24T21:54:47.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:47 smithi125 conmon[24927]: 2021-03-24T21:54:47.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:47 smithi125 conmon[24927]: 2021 2021-03-24T21:54:47.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:47 smithi125 conmon[24927]: - 2021-03-24T21:54:47.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:47 smithi125 conmon[24927]: 03 2021-03-24T21:54:47.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:47 smithi125 conmon[24927]: - 2021-03-24T21:54:47.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:47 smithi125 conmon[24927]: 24 2021-03-24T21:54:47.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:47 smithi125 conmon[24927]: T 2021-03-24T21:54:47.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:47 smithi125 conmon[24927]: 21 2021-03-24T21:54:47.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:47 smithi125 conmon[24927]: : 2021-03-24T21:54:47.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:47 smithi125 conmon[24927]: 54 2021-03-24T21:54:47.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:47 smithi125 conmon[24927]: : 2021-03-24T21:54:47.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:47 smithi125 conmon[24927]: 45 2021-03-24T21:54:47.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:47 smithi125 conmon[24927]: . 2021-03-24T21:54:47.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:47 smithi125 conmon[24927]: 900712 2021-03-24T21:54:47.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:47 smithi125 conmon[24927]: +0000 2021-03-24T21:54:47.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:47 smithi125 conmon[24927]: 2021-03-24T21:54:47.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:47 smithi125 conmon[24927]: osd.5 2021-03-24T21:54:47.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:47 smithi125 conmon[24927]: ( 2021-03-24T21:54:47.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:47 smithi125 conmon[24927]: osd 2021-03-24T21:54:47.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:47 smithi125 conmon[24927]: . 2021-03-24T21:54:47.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:47 smithi125 conmon[24927]: 5 2021-03-24T21:54:47.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:47 smithi125 conmon[24927]: ) 2021-03-24T21:54:47.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:47 smithi125 conmon[24927]: 34 2021-03-24T21:54:47.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:47 smithi125 conmon[24927]: : 2021-03-24T21:54:47.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:47 smithi125 conmon[24927]: cluster 2021-03-24T21:54:47.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:47 smithi125 conmon[24927]: 2021-03-24T21:54:47.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:47 smithi125 conmon[24927]: [DBG] 2021-03-24T21:54:47.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:47 smithi125 conmon[24927]: 6.7 scrub ok 2021-03-24T21:54:47.423 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:54:47.427 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:54:47.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:47.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:54:47.501 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-24T21:54:47.501 INFO:teuthology.orchestra.run.smithi125.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2021-03-24T21:54:47.502 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-24T21:54:47.513 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:54:47.529 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:47.577 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:47.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:47.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:54:47.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:47 smithi142 conmon[20125]: cluster 2021-03-24T21:54:45.900712+0000 osd.5 (osd.5) 34 : cluster [DBG] 6.7 scrub ok 2021-03-24T21:54:47.681 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:47.740 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:47.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:47.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:54:47.847 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:47.900 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:47.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:47.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:54:47.988 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-24T21:54:47.989 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T21:54:48.005 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:48.059 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:48.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:48.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:54:48.165 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:48.188 INFO:tasks.rados.rados.0.smithi125.stdout:115: done (0 left) 2021-03-24T21:54:48.188 INFO:tasks.rados.rados.0.smithi125.stdout:116: delete oid 39 current snap is 7 2021-03-24T21:54:48.194 INFO:tasks.rados.rados.0.smithi125.stdout:116: done (0 left) 2021-03-24T21:54:48.195 INFO:tasks.rados.rados.0.smithi125.stdout:117: snap_remove snap 6 2021-03-24T21:54:48.221 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:48.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:48.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:54:48.326 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:48.379 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:48.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:48.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:54:48.486 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:48.499 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:48 smithi142 conmon[20125]: cluster 2021-03-24T21:54:46. 2021-03-24T21:54:48.500 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:48 smithi142 conmon[20125]: 713215+0000 mgr.y (mgr.14144) 316 : cluster [DBG] pgmap v338: 148 pgs: 148 active+clean; 121 MiB data, 268 MiB used, 536 GiB / 536 GiB avail; 24 MiB/s rd, 5.0 MiB/s wr, 35 op/s; 3.0 MiB/s, 29 objects/s recovering 2021-03-24T21:54:48.500 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:48 smithi142 conmon[20125]: cluster 2021-03-24T21:54:46.859025+0000 osd.7 ( 2021-03-24T21:54:48.500 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:48 smithi142 conmon[20125]: osd.7) 32 : cluster [DBG] 6.8 scrub ok 2021-03-24T21:54:48.500 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:48 smithi142 conmon[20125]: cluster 2021-03-24T21:54:47. 2021-03-24T21:54:48.501 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:48 smithi142 conmon[20125]: 183165+0000 mon.a (mon.0) 664 : cluster [DBG] 2021-03-24T21:54:48.501 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:48 smithi142 conmon[20125]: osdmap e118: 8 total, 8 up, 6 in 2021-03-24T21:54:48.544 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:48.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:48.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:54:48.648 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:48.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:48 smithi125 conmon[19936]: cluster 2021-03-24T21:54:46.713215+0000 2021-03-24T21:54:48.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:48 smithi125 conmon[19936]: mgr.y (mgr.14144) 316 : cluster [DBG] pgmap v338: 148 pgs: 148 active+clean; 121 MiB data, 268 MiB used, 536 GiB / 536 GiB avail; 24 MiB/s rd, 5.0 MiB/s wr, 35 op/s; 3.0 MiB/s, 29 objects/s recovering 2021-03-24T21:54:48.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:48 smithi125 conmon[19936]: cluster 2021-03-24T21:54:46.859025+0000 osd.7 (osd.7) 32 : cluster [DBG] 6.8 scrub ok 2021-03-24T21:54:48.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:48 smithi125 conmon[19936]: cluster 2021-03-24T21:54:47.183165+0000 mon.a (mon.0) 664 : cluster [DBG] osdmap e118: 8 total, 8 up, 6 in 2021-03-24T21:54:48.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:48 smithi125 conmon[24927]: cluster 2021-03-24T21:54:48.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:48 smithi125 conmon[24927]: 2021-03-24T21:54:48.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:48 smithi125 conmon[24927]: 2021 2021-03-24T21:54:48.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:48 smithi125 conmon[24927]: - 2021-03-24T21:54:48.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:48 smithi125 conmon[24927]: 03 2021-03-24T21:54:48.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:48 smithi125 conmon[24927]: - 2021-03-24T21:54:48.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:48 smithi125 conmon[24927]: 24 2021-03-24T21:54:48.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:48 smithi125 conmon[24927]: T 2021-03-24T21:54:48.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:48 smithi125 conmon[24927]: 21 2021-03-24T21:54:48.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:48 smithi125 conmon[24927]: : 2021-03-24T21:54:48.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:48 smithi125 conmon[24927]: 54:46.713215+0000 2021-03-24T21:54:48.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:48 smithi125 conmon[24927]: mgr.y (mgr.14144) 316 : cluster [DBG] pgmap v338: 148 pgs: 148 active+clean; 121 MiB data, 268 MiB used, 536 GiB / 536 GiB avail; 24 MiB/s rd, 5.0 MiB/s wr, 35 op/s; 3.0 MiB/s, 29 objects/s recovering 2021-03-24T21:54:48.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:48 smithi125 conmon[24927]: cluster 2021-03-24T21:54:48.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:48 smithi125 conmon[24927]: 2021-03- 2021-03-24T21:54:48.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:48 smithi125 conmon[24927]: 24T21:54:46.859025+0000 osd.7 (osd. 2021-03-24T21:54:48.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:48 smithi125 conmon[24927]: 7) 32 : cluster [DBG] 6.8 scrub ok 2021-03-24T21:54:48.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:48 smithi125 conmon[24927]: cluster 2021-03-24T21:54:48.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:48 smithi125 conmon[24927]: 2021- 2021-03-24T21:54:48.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:48 smithi125 conmon[24927]: 03-24T21:54:47.183165+0000 mon.a (mon.0) 664 2021-03-24T21:54:48.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:48 smithi125 conmon[24927]: : cluster [DBG] osdmap e118: 8 total, 8 up, 6 in 2021-03-24T21:54:48.710 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:48.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:48.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:54:48.815 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:54:48.873 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:54:48.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:48.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:54:48.980 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:49.042 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:49.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:49.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:54:49.149 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:49.205 INFO:tasks.rados.rados.0.smithi125.stdout:117: done (0 left) 2021-03-24T21:54:49.205 INFO:tasks.rados.rados.0.smithi125.stdout:118: read oid 14 snap 5 2021-03-24T21:54:49.205 INFO:tasks.rados.rados.0.smithi125.stdout:118: expect (ObjNum 13 snap 0 seq_num 13) 2021-03-24T21:54:49.205 INFO:tasks.rados.rados.0.smithi125.stdout:119: write oid 47 current snap is 7 2021-03-24T21:54:49.206 INFO:tasks.rados.rados.0.smithi125.stdout:119: seq_num 67 ranges {726069=772507,2071117=715676} 2021-03-24T21:54:49.209 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:49.214 INFO:tasks.rados.rados.0.smithi125.stdout:119: writing smithi12556470-47 from 726069 to 1498576 tid 1 2021-03-24T21:54:49.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:49.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:54:49.222 INFO:tasks.rados.rados.0.smithi125.stdout:119: writing smithi12556470-47 from 2071117 to 2786793 tid 2 2021-03-24T21:54:49.222 INFO:tasks.rados.rados.0.smithi125.stdout:120: read oid 13 snap -1 2021-03-24T21:54:49.222 INFO:tasks.rados.rados.0.smithi125.stdout:120: expect (ObjNum 63 snap 4 seq_num 63) 2021-03-24T21:54:49.223 INFO:tasks.rados.rados.0.smithi125.stdout:121: write (excl) oid 30 current snap is 7 2021-03-24T21:54:49.223 INFO:tasks.rados.rados.0.smithi125.stdout:121: seq_num 68 ranges {742876=447756,1729599=575687,3101234=41642} 2021-03-24T21:54:49.229 INFO:tasks.rados.rados.0.smithi125.stdout:121: writing smithi12556470-30 from 742876 to 1190632 tid 1 2021-03-24T21:54:49.235 INFO:tasks.rados.rados.0.smithi125.stdout:121: writing smithi12556470-30 from 1729599 to 2305286 tid 2 2021-03-24T21:54:49.240 INFO:tasks.rados.rados.0.smithi125.stdout:121: writing smithi12556470-30 from 3101234 to 3142876 tid 3 2021-03-24T21:54:49.240 INFO:tasks.rados.rados.0.smithi125.stdout:122: write (excl) oid 39 current snap is 7 2021-03-24T21:54:49.241 INFO:tasks.rados.rados.0.smithi125.stdout:122: seq_num 69 ranges {759683=639358,1904434=719345,3128657=31026} 2021-03-24T21:54:49.249 INFO:tasks.rados.rados.0.smithi125.stdout:122: writing smithi12556470-39 from 759683 to 1399041 tid 1 2021-03-24T21:54:49.258 INFO:tasks.rados.rados.0.smithi125.stdout:122: writing smithi12556470-39 from 1904434 to 2623779 tid 2 2021-03-24T21:54:49.261 INFO:tasks.rados.rados.0.smithi125.stdout:122: writing smithi12556470-39 from 3128657 to 3159683 tid 3 2021-03-24T21:54:49.261 INFO:tasks.rados.rados.0.smithi125.stdout:118: expect (ObjNum 13 snap 0 seq_num 13) 2021-03-24T21:54:49.317 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:49.343 INFO:tasks.rados.rados.0.smithi125.stdout:118: done (4 left) 2021-03-24T21:54:49.343 INFO:tasks.rados.rados.0.smithi125.stdout:123: write oid 33 current snap is 7 2021-03-24T21:54:49.343 INFO:tasks.rados.rados.0.smithi125.stdout:121: finishing write tid 1 to smithi12556470-30 2021-03-24T21:54:49.344 INFO:tasks.rados.rados.0.smithi125.stdout:123: seq_num 70 ranges {776490=714607,2246563=579356} 2021-03-24T21:54:49.351 INFO:tasks.rados.rados.0.smithi125.stdout:123: writing smithi12556470-33 from 776490 to 1491097 tid 1 2021-03-24T21:54:49.359 INFO:tasks.rados.rados.0.smithi125.stdout:123: writing smithi12556470-33 from 2246563 to 2825919 tid 2 2021-03-24T21:54:49.359 INFO:tasks.rados.rados.0.smithi125.stdout:124: rollback oid 3 current snap is 7 2021-03-24T21:54:49.359 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 3 to 5 2021-03-24T21:54:49.360 INFO:tasks.rados.rados.0.smithi125.stdout:119: finishing write tid 1 to smithi12556470-47 2021-03-24T21:54:49.360 INFO:tasks.rados.rados.0.smithi125.stdout:119: finishing write tid 2 to smithi12556470-47 2021-03-24T21:54:49.360 INFO:tasks.rados.rados.0.smithi125.stdout:119: finishing write tid 3 to smithi12556470-47 2021-03-24T21:54:49.360 INFO:tasks.rados.rados.0.smithi125.stdout:119: finishing write tid 4 to smithi12556470-47 2021-03-24T21:54:49.361 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 47 v 19 (ObjNum 67 snap 7 seq_num 67) dirty exists 2021-03-24T21:54:49.361 INFO:tasks.rados.rados.0.smithi125.stdout:119: left oid 47 (ObjNum 67 snap 7 seq_num 67) 2021-03-24T21:54:49.361 INFO:tasks.rados.rados.0.smithi125.stdout:122: finishing write tid 1 to smithi12556470-39 2021-03-24T21:54:49.361 INFO:tasks.rados.rados.0.smithi125.stdout:120: expect (ObjNum 63 snap 4 seq_num 63) 2021-03-24T21:54:49.379 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:49.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:49.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:54:49.486 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:49.493 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:49 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:54:49.494 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:49 smithi125 conmon[24927]: 54:48.186159+0000 mon.a (mon.0) 665 : cluster [DBG] osdmap e119: 8 total, 8 up, 6 in 2021-03-24T21:54:49.494 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:49 smithi125 conmon[24927]: cluster 2021-03-24T21:54:48. 2021-03-24T21:54:49.494 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:49 smithi125 conmon[24927]: 541406+0000 osd.3 (osd.3) 40 : cluster 2021-03-24T21:54:49.494 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:49 smithi125 conmon[24927]: [DBG] 3.19 scrub ok 2021-03-24T21:54:49.495 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:49 smithi125 conmon[19936]: cluster 2021-03-24T21:54:48.186159 2021-03-24T21:54:49.495 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:49 smithi125 conmon[19936]: +0000 mon.a (mon.0) 665 : cluster [DBG] osdmap e119: 8 total, 8 up, 6 in 2021-03-24T21:54:49.495 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:49 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:54:49.495 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:49 smithi125 conmon[19936]: 21:54:48.541406+0000 osd.3 (osd.3) 40 : cluster [DBG] 3.19 scrub ok 2021-03-24T21:54:49.502 INFO:tasks.rados.rados.0.smithi125.stdout:121: finishing write tid 2 to smithi12556470-30 2021-03-24T21:54:49.503 INFO:tasks.rados.rados.0.smithi125.stdout:121: finishing write tid 3 to smithi12556470-30 2021-03-24T21:54:49.503 INFO:tasks.rados.rados.0.smithi125.stdout:122: finishing write tid 2 to smithi12556470-39 2021-03-24T21:54:49.503 INFO:tasks.rados.rados.0.smithi125.stdout:121: finishing write tid 4 to smithi12556470-30 2021-03-24T21:54:49.503 INFO:tasks.rados.rados.0.smithi125.stdout:122: finishing write tid 3 to smithi12556470-39 2021-03-24T21:54:49.504 INFO:tasks.rados.rados.0.smithi125.stdout:121: finishing write tid 5 to smithi12556470-30 2021-03-24T21:54:49.504 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 30 v 21 (ObjNum 68 snap 7 seq_num 68) dirty exists 2021-03-24T21:54:49.504 INFO:tasks.rados.rados.0.smithi125.stdout:121: left oid 30 (ObjNum 68 snap 7 seq_num 68) 2021-03-24T21:54:49.504 INFO:tasks.rados.rados.0.smithi125.stdout:122: finishing write tid 4 to smithi12556470-39 2021-03-24T21:54:49.505 INFO:tasks.rados.rados.0.smithi125.stdout:122: finishing write tid 5 to smithi12556470-39 2021-03-24T21:54:49.505 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 39 v 31 (ObjNum 69 snap 7 seq_num 69) dirty exists 2021-03-24T21:54:49.505 INFO:tasks.rados.rados.0.smithi125.stdout:122: left oid 39 (ObjNum 69 snap 7 seq_num 69) 2021-03-24T21:54:49.505 INFO:tasks.rados.rados.0.smithi125.stdout:119: done (5 left) 2021-03-24T21:54:49.505 INFO:tasks.rados.rados.0.smithi125.stdout:120: done (4 left) 2021-03-24T21:54:49.506 INFO:tasks.rados.rados.0.smithi125.stdout:121: done (3 left) 2021-03-24T21:54:49.506 INFO:tasks.rados.rados.0.smithi125.stdout:122: done (2 left) 2021-03-24T21:54:49.506 INFO:tasks.rados.rados.0.smithi125.stdout:125: delete oid 28 current snap is 7 2021-03-24T21:54:49.506 INFO:tasks.rados.rados.0.smithi125.stdout:123: finishing write tid 1 to smithi12556470-33 2021-03-24T21:54:49.507 INFO:tasks.rados.rados.0.smithi125.stdout:123: finishing write tid 2 to smithi12556470-33 2021-03-24T21:54:49.507 INFO:tasks.rados.rados.0.smithi125.stdout:123: finishing write tid 3 to smithi12556470-33 2021-03-24T21:54:49.507 INFO:tasks.rados.rados.0.smithi125.stdout:123: finishing write tid 4 to smithi12556470-33 2021-03-24T21:54:49.507 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 33 v 35 (ObjNum 70 snap 7 seq_num 70) dirty exists 2021-03-24T21:54:49.507 INFO:tasks.rados.rados.0.smithi125.stdout:123: left oid 33 (ObjNum 70 snap 7 seq_num 70) 2021-03-24T21:54:49.508 INFO:tasks.rados.rados.0.smithi125.stdout:124: finishing rollback tid 0 to smithi12556470-3 2021-03-24T21:54:49.508 INFO:tasks.rados.rados.0.smithi125.stdout:124: finishing rollback tid 1 to smithi12556470-3 2021-03-24T21:54:49.508 INFO:tasks.rados.rados.0.smithi125.stdout:124: finishing rollback tid 2 to smithi12556470-3 2021-03-24T21:54:49.508 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 3 v 23 (ObjNum 61 snap 4 seq_num 61) dirty exists 2021-03-24T21:54:49.509 INFO:tasks.rados.rados.0.smithi125.stdout:123: done (2 left) 2021-03-24T21:54:49.509 INFO:tasks.rados.rados.0.smithi125.stdout:124: done (1 left) 2021-03-24T21:54:49.509 INFO:tasks.rados.rados.0.smithi125.stdout:125: done (0 left) 2021-03-24T21:54:49.509 INFO:tasks.rados.rados.0.smithi125.stdout:126: write oid 17 current snap is 7 2021-03-24T21:54:49.510 INFO:tasks.rados.rados.0.smithi125.stdout:126: seq_num 72 ranges {410104=465105,1563527=583025,2810926=399178} 2021-03-24T21:54:49.510 INFO:tasks.rados.rados.0.smithi125.stdout:126: writing smithi12556470-17 from 410104 to 875209 tid 1 2021-03-24T21:54:49.520 INFO:tasks.rados.rados.0.smithi125.stdout:126: writing smithi12556470-17 from 1563527 to 2146552 tid 2 2021-03-24T21:54:49.527 INFO:tasks.rados.rados.0.smithi125.stdout:126: writing smithi12556470-17 from 2810926 to 3210104 tid 3 2021-03-24T21:54:49.527 INFO:tasks.rados.rados.0.smithi125.stdout:127: rollback oid 44 current snap is 7 2021-03-24T21:54:49.527 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 44 to 5 2021-03-24T21:54:49.528 INFO:tasks.rados.rados.0.smithi125.stdout:126: finishing write tid 1 to smithi12556470-17 2021-03-24T21:54:49.528 INFO:tasks.rados.rados.0.smithi125.stdout:126: finishing write tid 2 to smithi12556470-17 2021-03-24T21:54:49.528 INFO:tasks.rados.rados.0.smithi125.stdout:128: rollback oid 32 current snap is 7 2021-03-24T21:54:49.528 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 32 to 5 2021-03-24T21:54:49.528 INFO:tasks.rados.rados.0.smithi125.stdout:129: write (excl) oid 31 current snap is 7 2021-03-24T21:54:49.529 INFO:tasks.rados.rados.0.smithi125.stdout:129: seq_num 73 ranges {426911=540354,1622009=726683,3121996=104915} 2021-03-24T21:54:49.533 INFO:tasks.rados.rados.0.smithi125.stdout:129: writing smithi12556470-31 from 426911 to 967265 tid 1 2021-03-24T21:54:49.541 INFO:tasks.rados.rados.0.smithi125.stdout:129: writing smithi12556470-31 from 1622009 to 2348692 tid 2 2021-03-24T21:54:49.544 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:49.548 INFO:tasks.rados.rados.0.smithi125.stdout:129: writing smithi12556470-31 from 3121996 to 3226911 tid 3 2021-03-24T21:54:49.548 INFO:tasks.rados.rados.0.smithi125.stdout:130: copy_from oid 1 from oid 23 current snap is 7 2021-03-24T21:54:49.549 INFO:tasks.rados.rados.0.smithi125.stdout:131: read oid 33 snap -1 2021-03-24T21:54:49.549 INFO:tasks.rados.rados.0.smithi125.stdout:131: expect (ObjNum 70 snap 7 seq_num 70) 2021-03-24T21:54:49.549 INFO:tasks.rados.rados.0.smithi125.stdout:132: write (excl) oid 15 current snap is 7 2021-03-24T21:54:49.550 INFO:tasks.rados.rados.0.smithi125.stdout:132: seq_num 74 ranges {443718=615603,1564138=470341,2633066=610652} 2021-03-24T21:54:49.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:49.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:54:49.555 INFO:tasks.rados.rados.0.smithi125.stdout:132: writing smithi12556470-15 from 443718 to 1059321 tid 1 2021-03-24T21:54:49.561 INFO:tasks.rados.rados.0.smithi125.stdout:132: writing smithi12556470-15 from 1564138 to 2034479 tid 2 2021-03-24T21:54:49.568 INFO:tasks.rados.rados.0.smithi125.stdout:132: writing smithi12556470-15 from 2633066 to 3243718 tid 3 2021-03-24T21:54:49.569 INFO:tasks.rados.rados.0.smithi125.stdout:133: snap_create 2021-03-24T21:54:49.569 INFO:tasks.rados.rados.0.smithi125.stdout:127: finishing rollback tid 0 to smithi12556470-44 2021-03-24T21:54:49.569 INFO:tasks.rados.rados.0.smithi125.stdout:126: finishing write tid 3 to smithi12556470-17 2021-03-24T21:54:49.569 INFO:tasks.rados.rados.0.smithi125.stdout:127: finishing rollback tid 1 to smithi12556470-44 2021-03-24T21:54:49.569 INFO:tasks.rados.rados.0.smithi125.stdout:128: finishing rollback tid 0 to smithi12556470-32 2021-03-24T21:54:49.570 INFO:tasks.rados.rados.0.smithi125.stdout:127: finishing rollback tid 2 to smithi12556470-44 2021-03-24T21:54:49.570 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 44 v 26 (ObjNum 43 snap 0 seq_num 43) dirty exists 2021-03-24T21:54:49.570 INFO:tasks.rados.rados.0.smithi125.stdout:126: finishing write tid 4 to smithi12556470-17 2021-03-24T21:54:49.570 INFO:tasks.rados.rados.0.smithi125.stdout:128: finishing rollback tid 1 to smithi12556470-32 2021-03-24T21:54:49.571 INFO:tasks.rados.rados.0.smithi125.stdout:126: finishing write tid 5 to smithi12556470-17 2021-03-24T21:54:49.571 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 17 v 24 (ObjNum 72 snap 7 seq_num 72) dirty exists 2021-03-24T21:54:49.571 INFO:tasks.rados.rados.0.smithi125.stdout:126: left oid 17 (ObjNum 72 snap 7 seq_num 72) 2021-03-24T21:54:49.571 INFO:tasks.rados.rados.0.smithi125.stdout:128: finishing rollback tid 2 to smithi12556470-32 2021-03-24T21:54:49.572 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 32 v 36 (ObjNum 31 snap 0 seq_num 31) dirty exists 2021-03-24T21:54:49.572 INFO:tasks.rados.rados.0.smithi125.stdout:129: finishing write tid 1 to smithi12556470-31 2021-03-24T21:54:49.572 INFO:tasks.rados.rados.0.smithi125.stdout:129: finishing write tid 2 to smithi12556470-31 2021-03-24T21:54:49.572 INFO:tasks.rados.rados.0.smithi125.stdout:129: finishing write tid 3 to smithi12556470-31 2021-03-24T21:54:49.573 INFO:tasks.rados.rados.0.smithi125.stdout:129: finishing write tid 4 to smithi12556470-31 2021-03-24T21:54:49.573 INFO:tasks.rados.rados.0.smithi125.stdout:129: finishing write tid 5 to smithi12556470-31 2021-03-24T21:54:49.573 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 31 v 41 (ObjNum 73 snap 7 seq_num 73) dirty exists 2021-03-24T21:54:49.573 INFO:tasks.rados.rados.0.smithi125.stdout:129: left oid 31 (ObjNum 73 snap 7 seq_num 73) 2021-03-24T21:54:49.573 INFO:tasks.rados.rados.0.smithi125.stdout:131: expect (ObjNum 70 snap 7 seq_num 70) 2021-03-24T21:54:49.603 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.1 dump_historic_ops 2021-03-24T21:54:49.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:49 smithi142 conmon[20125]: cluster 2021-03-24T21:54:48 2021-03-24T21:54:49.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:49 smithi142 conmon[20125]: .186159+0000 mon.a (mon.0) 665 : cluster [DBG] osdmap e119: 8 total, 8 up, 6 in 2021-03-24T21:54:49.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:49 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:54:49.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:49 smithi142 conmon[20125]: -24T21:54:48.541406+0000 osd.3 (osd.3) 40 : cluster [DBG] 3.19 scrub ok 2021-03-24T21:54:49.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:49 smithi142 conmon[20125]: 2021-03-24T21:54:49.653 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:49.717 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:49.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:49.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:54:49.781 INFO:tasks.rados.rados.0.smithi125.stdout:132: finishing write tid 1 to smithi12556470-15 2021-03-24T21:54:49.782 INFO:tasks.rados.rados.0.smithi125.stdout:130: finishing copy_from to smithi12556470-1 2021-03-24T21:54:49.782 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 1 v 9 (ObjNum 22 snap 0 seq_num 22) dirty exists 2021-03-24T21:54:49.783 INFO:tasks.rados.rados.0.smithi125.stdout:130: finishing copy_from racing read to smithi12556470-1 2021-03-24T21:54:49.783 INFO:tasks.rados.rados.0.smithi125.stdout:132: finishing write tid 2 to smithi12556470-15 2021-03-24T21:54:49.784 INFO:tasks.rados.rados.0.smithi125.stdout:132: finishing write tid 3 to smithi12556470-15 2021-03-24T21:54:49.784 INFO:tasks.rados.rados.0.smithi125.stdout:132: finishing write tid 4 to smithi12556470-15 2021-03-24T21:54:49.784 INFO:tasks.rados.rados.0.smithi125.stdout:132: finishing write tid 5 to smithi12556470-15 2021-03-24T21:54:49.785 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 15 v 25 (ObjNum 74 snap 7 seq_num 74) dirty exists 2021-03-24T21:54:49.785 INFO:tasks.rados.rados.0.smithi125.stdout:132: left oid 15 (ObjNum 74 snap 7 seq_num 74) 2021-03-24T21:54:49.824 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:49.879 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:49.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:49.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:54:49.986 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:50.038 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:50.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:50.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:54:50.083 INFO:tasks.rados.rados.0.smithi125.stdout:126: done (7 left) 2021-03-24T21:54:50.083 INFO:tasks.rados.rados.0.smithi125.stdout:127: done (6 left) 2021-03-24T21:54:50.084 INFO:tasks.rados.rados.0.smithi125.stdout:128: done (5 left) 2021-03-24T21:54:50.084 INFO:tasks.rados.rados.0.smithi125.stdout:129: done (4 left) 2021-03-24T21:54:50.084 INFO:tasks.rados.rados.0.smithi125.stdout:130: done (3 left) 2021-03-24T21:54:50.084 INFO:tasks.rados.rados.0.smithi125.stdout:131: done (2 left) 2021-03-24T21:54:50.085 INFO:tasks.rados.rados.0.smithi125.stdout:132: done (1 left) 2021-03-24T21:54:50.085 INFO:tasks.rados.rados.0.smithi125.stdout:133: done (0 left) 2021-03-24T21:54:50.085 INFO:tasks.rados.rados.0.smithi125.stdout:134: snap_create 2021-03-24T21:54:50.145 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:50.201 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:50.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:50.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:54:50.306 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:50.319 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:50 smithi125 conmon[24927]: cluster 2021-03-24T21:54:48.394149+0000 osd.0 2021-03-24T21:54:50.319 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:50 smithi125 conmon[24927]: (osd.0) 38 : cluster [DBG] 3.17 scrub ok 2021-03-24T21:54:50.319 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:50 smithi125 conmon[24927]: cluster 2021-03-24T21:54:48.713770+0000 mgr.y (mgr.14144) 317 : cluster [DBG] pgmap v341: 148 pgs: 148 active+clean; 123 MiB data, 285 MiB used, 536 GiB / 536 GiB avail; 11 MiB/s rd, 2.7 MiB/s wr, 14 op/s 2021-03-24T21:54:50.319 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:50 smithi125 conmon[24927]: cluster 2021-03-24T21:54:49.204401+0000 mon.a (mon.0) 666 : cluster [DBG] osdmap e120: 8 total, 8 up, 6 in 2021-03-24T21:54:50.320 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:50 smithi125 conmon[24927]: audit 2021-03-24T21:54:49.764953+0000 mon.a (mon.0) 667 2021-03-24T21:54:50.320 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:50 smithi125 conmon[24927]: : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:54:50.320 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:50 smithi125 conmon[24927]: cluster 2021-03-24T21:54: 2021-03-24T21:54:50.320 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:50 smithi125 conmon[24927]: 50.079679+0000 mon.a (mon.0) 668 : cluster [DBG] osdmap e121: 8 total, 8 up, 6 in 2021-03-24T21:54:50.321 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:50 smithi125 conmon[19936]: cluster 2021-03-24T21:54:48.394149 2021-03-24T21:54:50.321 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:50 smithi125 conmon[19936]: +0000 osd.0 (osd.0) 38 : cluster [DBG] 3.17 scrub ok 2021-03-24T21:54:50.321 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:50 smithi125 conmon[19936]: cluster 2021-03-24T21:54:48.713770+0000 mgr.y (mgr.14144) 317 : cluster 2021-03-24T21:54:50.321 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:50 smithi125 conmon[19936]: [DBG] pgmap v341: 148 pgs: 148 active+clean; 123 MiB data, 285 MiB used, 536 GiB / 536 GiB avail; 11 MiB/s rd, 2.7 MiB/s wr, 14 op/s 2021-03-24T21:54:50.321 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:50 smithi125 conmon[19936]: cluster 2021-03-24T21:54:50.322 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:50 smithi125 conmon[19936]: 2021-03-24T21:54:49.204401+0000 mon.a (mon.0) 666 2021-03-24T21:54:50.322 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:50 smithi125 conmon[19936]: : cluster [DBG] osdmap e120: 8 total, 8 up, 6 in 2021-03-24T21:54:50.322 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:50 smithi125 conmon[19936]: audit 2021-03-24T21:54:50.323 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:50 smithi125 conmon[19936]: 2021-03-24T21:54:49.764953+0000 2021-03-24T21:54:50.323 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:50 smithi125 conmon[19936]: mon.a (mon.0) 667 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:54:50.323 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:50 smithi125 conmon[19936]: cluster 2021-03-24T21:54:50 2021-03-24T21:54:50.323 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:50 smithi125 conmon[19936]: .079679+0000 mon.a (mon.0) 668 : cluster [DBG] osdmap e121: 8 total, 8 up, 6 in 2021-03-24T21:54:50.368 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:50.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:50.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:54:50.474 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:50.488 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:50 smithi142 conmon[20125]: cluster 2021-03-24T21:54:48.394149+0000 osd.0 (osd.0) 38 : 2021-03-24T21:54:50.488 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:50 smithi142 conmon[20125]: cluster [DBG] 3.17 scrub ok 2021-03-24T21:54:50.488 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:50 smithi142 conmon[20125]: cluster 2021-03-24T21:54:48.713770+0000 mgr.y (mgr.14144) 2021-03-24T21:54:50.489 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:50 smithi142 conmon[20125]: 317 : cluster [DBG] pgmap v341: 148 pgs: 148 active+clean; 123 MiB data, 285 MiB used, 536 GiB / 536 GiB avail; 11 MiB/s rd, 2.7 MiB/s wr, 14 op/s 2021-03-24T21:54:50.489 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:50 smithi142 conmon[20125]: cluster 2021-03-24T21:54: 2021-03-24T21:54:50.489 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:50 smithi142 conmon[20125]: 49.204401+0000 mon.a (mon.0) 666 2021-03-24T21:54:50.489 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:50 smithi142 conmon[20125]: : cluster [DBG] osdmap e120: 8 total, 8 up, 6 in 2021-03-24T21:54:50.490 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:50 smithi142 conmon[20125]: audit 2021 2021-03-24T21:54:50.490 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:50 smithi142 conmon[20125]: -03-24T21:54:49.764953 2021-03-24T21:54:50.490 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:50 smithi142 conmon[20125]: +0000 mon.a (mon.0) 667 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:54:50.490 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:50 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:54:50.491 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:50 smithi142 conmon[20125]: T21:54:50.079679+0000 mon.a ( 2021-03-24T21:54:50.491 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:50 smithi142 conmon[20125]: mon.0) 668 : cluster [DBG] osdmap e121: 8 total, 8 up, 6 in 2021-03-24T21:54:50.531 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:50.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:50.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:54:50.638 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:54:50.699 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:54:50.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:50.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:54:50.806 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:50.864 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:50.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:50.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:54:50.970 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:51.017 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:51.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:51.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:54:51.084 INFO:tasks.rados.rados.0.smithi125.stdout:134: done (0 left) 2021-03-24T21:54:51.084 INFO:tasks.rados.rados.0.smithi125.stdout:135: snap_remove snap 5 2021-03-24T21:54:51.124 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:51.179 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:51.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:51.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:54:51.286 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:54:51.347 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:54:51.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:51.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:54:51.377 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:54:51.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:51 smithi125 conmon[19936]: cluster 2021-03-24T21:54:49.323780+0000 osd.4 (osd.4) 2021-03-24T21:54:51.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:51 smithi125 conmon[19936]: 36 : cluster [DBG] 5.1d scrub ok 2021-03-24T21:54:51.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:51 smithi125 conmon[19936]: cluster 2021-03-24T21:54:49.409291+0000 osd.0 (osd.0) 39 : cluster [DBG] 6.18 scrub ok 2021-03-24T21:54:51.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:51 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:54:51.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:51 smithi125 conmon[19936]: 54:49.807576+0000 osd.7 (osd.7) 33 : cluster [DBG] 5.4 scrub ok 2021-03-24T21:54:51.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:51 smithi125 conmon[19936]: cluster 2021-03-24T21:54:51.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:51 smithi125 conmon[19936]: 2021-03-24T21:54:49.991957+0000 osd.5 (osd.5) 35 2021-03-24T21:54:51.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:51 smithi125 conmon[19936]: : cluster [DBG] 6.1d scrub ok 2021-03-24T21:54:51.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:51 smithi125 conmon[19936]: cluster 2021-03-24T21:54 2021-03-24T21:54:51.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:51 smithi125 conmon[19936]: :51.083785+0000 mon.a (mon.0) 669 : cluster [DBG] 2021-03-24T21:54:51.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:51 smithi125 conmon[19936]: osdmap e122: 8 total, 8 up, 6 in 2021-03-24T21:54:51.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:51 smithi125 conmon[24927]: cluster 2021-03-24T21:54:49.323780+0000 osd.4 (osd. 2021-03-24T21:54:51.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:51 smithi125 conmon[24927]: 4) 36 : cluster [DBG] 5.1d scrub ok 2021-03-24T21:54:51.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:51 smithi125 conmon[24927]: cluster 2021-03-24T21:54:49.409291+0000 osd.0 (osd.0) 39 : cluster [DBG] 6.18 scrub ok 2021-03-24T21:54:51.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:51 smithi125 conmon[24927]: cluster 2021-03-24T21:54:49.807576+0000 osd.7 (osd.7) 2021-03-24T21:54:51.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:51 smithi125 conmon[24927]: 33 : cluster [DBG] 5.4 scrub ok 2021-03-24T21:54:51.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:51 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:54:51.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:51 smithi125 conmon[24927]: -03-24T21:54:49.991957+0000 osd.5 (osd.5) 35 : cluster [DBG] 6.1d scrub ok 2021-03-24T21:54:51.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:51 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:54:51.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:51 smithi125 conmon[24927]: 24T21:54:51.083785+0000 mon.a (mon.0) 669 : cluster [DBG] osdmap e122: 8 total, 8 up, 6 in 2021-03-24T21:54:51.451 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:51.464 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:51 smithi142 conmon[20125]: cluster 2021-03-24T21:54:49.323780+0000 osd.4 (osd.4) 36 : cluster [DBG] 5.1d scrub ok 2021-03-24T21:54:51.465 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:51 smithi142 conmon[20125]: cluster 2021-03-24T21:54:49.409291+0000 osd.0 2021-03-24T21:54:51.465 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:51 smithi142 conmon[20125]: (osd.0) 39 : cluster [DBG] 6.18 scrub ok 2021-03-24T21:54:51.465 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:51 smithi142 conmon[20125]: cluster 2021-03-24T21:54:49.807576+0000 osd.7 (osd.7) 33 : cluster [DBG] 5.4 scrub ok 2021-03-24T21:54:51.465 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:51 smithi142 conmon[20125]: cluster 2021-03-24T21:54:49.991957+0000 osd.5 (osd.5) 35 : cluster [DBG] 6.1d scrub ok 2021-03-24T21:54:51.466 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:51 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:54:51.466 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:51 smithi142 conmon[20125]: -24T21:54:51.083785+0000 mon.a (mon.0) 669 : cluster [DBG] osdmap e122: 8 total, 8 up, 6 in 2021-03-24T21:54:51.509 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:51.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:51.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:54:51.616 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:51.673 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:51.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:51.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:54:51.778 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:51.831 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:51.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:51.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:54:51.936 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:54:51.994 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:54:51.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:51.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:54:52.081 INFO:tasks.rados.rados.0.smithi125.stdout:135: done (0 left) 2021-03-24T21:54:52.082 INFO:tasks.rados.rados.0.smithi125.stdout:136: write oid 21 current snap is 9 2021-03-24T21:54:52.082 INFO:tasks.rados.rados.0.smithi125.stdout:136: seq_num 75 ranges {460525=690852,1622620=730352,3060489=200036} 2021-03-24T21:54:52.088 INFO:tasks.rados.rados.0.smithi125.stdout:136: writing smithi12556470-21 from 460525 to 1151377 tid 1 2021-03-24T21:54:52.096 INFO:tasks.rados.rados.0.smithi125.stdout:136: writing smithi12556470-21 from 1622620 to 2352972 tid 2 2021-03-24T21:54:52.100 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:52.102 INFO:tasks.rados.rados.0.smithi125.stdout:136: writing smithi12556470-21 from 3060489 to 3260525 tid 3 2021-03-24T21:54:52.102 INFO:tasks.rados.rados.0.smithi125.stdout:137: read oid 47 snap -1 2021-03-24T21:54:52.102 INFO:tasks.rados.rados.0.smithi125.stdout:137: expect (ObjNum 67 snap 7 seq_num 67) 2021-03-24T21:54:52.102 INFO:tasks.rados.rados.0.smithi125.stdout:138: delete oid 19 current snap is 9 2021-03-24T21:54:52.130 INFO:tasks.rados.rados.0.smithi125.stdout:138: done (2 left) 2021-03-24T21:54:52.131 INFO:tasks.rados.rados.0.smithi125.stdout:139: write oid 15 current snap is 9 2021-03-24T21:54:52.131 INFO:tasks.rados.rados.0.smithi125.stdout:139: seq_num 77 ranges {494139=557703,1455937=734021,2948041=346098} 2021-03-24T21:54:52.137 INFO:tasks.rados.rados.0.smithi125.stdout:139: writing smithi12556470-15 from 494139 to 1051842 tid 1 2021-03-24T21:54:52.144 INFO:tasks.rados.rados.0.smithi125.stdout:139: writing smithi12556470-15 from 1455937 to 2189958 tid 2 2021-03-24T21:54:52.150 INFO:tasks.rados.rados.0.smithi125.stdout:139: writing smithi12556470-15 from 2948041 to 3294139 tid 3 2021-03-24T21:54:52.151 INFO:tasks.rados.rados.0.smithi125.stdout:140: write (excl) oid 11 current snap is 9 2021-03-24T21:54:52.151 INFO:tasks.rados.rados.0.smithi125.stdout:140: seq_num 78 ranges {510946=632952,1798066=477679,2742758=457217} 2021-03-24T21:54:52.156 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:52.157 INFO:tasks.rados.rados.0.smithi125.stdout:140: writing smithi12556470-11 from 510946 to 1143898 tid 1 2021-03-24T21:54:52.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:52.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:54:52.163 INFO:tasks.rados.rados.0.smithi125.stdout:140: writing smithi12556470-11 from 1798066 to 2275745 tid 2 2021-03-24T21:54:52.169 INFO:tasks.rados.rados.0.smithi125.stdout:140: writing smithi12556470-11 from 2742758 to 3199975 tid 3 2021-03-24T21:54:52.169 INFO:tasks.rados.rados.0.smithi125.stdout:141: write (excl) oid 1 current snap is 9 2021-03-24T21:54:52.170 INFO:tasks.rados.rados.0.smithi125.stdout:136: finishing write tid 1 to smithi12556470-21 2021-03-24T21:54:52.170 INFO:tasks.rados.rados.0.smithi125.stdout:141: seq_num 79 ranges {527753=708201,1856548=737690,3286534=41219} 2021-03-24T21:54:52.176 INFO:tasks.rados.rados.0.smithi125.stdout:141: writing smithi12556470-1 from 527753 to 1235954 tid 1 2021-03-24T21:54:52.184 INFO:tasks.rados.rados.0.smithi125.stdout:141: writing smithi12556470-1 from 1856548 to 2594238 tid 2 2021-03-24T21:54:52.189 INFO:tasks.rados.rados.0.smithi125.stdout:141: writing smithi12556470-1 from 3286534 to 3327753 tid 3 2021-03-24T21:54:52.189 INFO:tasks.rados.rados.0.smithi125.stdout:142: delete oid 14 current snap is 9 2021-03-24T21:54:52.189 INFO:tasks.rados.rados.0.smithi125.stdout:136: finishing write tid 2 to smithi12556470-21 2021-03-24T21:54:52.189 INFO:tasks.rados.rados.0.smithi125.stdout:136: finishing write tid 3 to smithi12556470-21 2021-03-24T21:54:52.190 INFO:tasks.rados.rados.0.smithi125.stdout:136: finishing write tid 4 to smithi12556470-21 2021-03-24T21:54:52.190 INFO:tasks.rados.rados.0.smithi125.stdout:136: finishing write tid 5 to smithi12556470-21 2021-03-24T21:54:52.190 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 21 v 31 (ObjNum 75 snap 9 seq_num 75) dirty exists 2021-03-24T21:54:52.190 INFO:tasks.rados.rados.0.smithi125.stdout:136: left oid 21 (ObjNum 75 snap 9 seq_num 75) 2021-03-24T21:54:52.191 INFO:tasks.rados.rados.0.smithi125.stdout:137: expect (ObjNum 67 snap 7 seq_num 67) 2021-03-24T21:54:52.200 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T21:54:52.200 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T21:54:52.201 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T21:54:52.201 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T21:54:52.201 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T21:54:52.201 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T21:54:52.201 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T21:54:52.202 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T21:54:52.202 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-24T21:54:52.202 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-24T21:54:52.203 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-24T21:54:52.203 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-24T21:54:52.203 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-24T21:54:52.203 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-24T21:54:52.203 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-24T21:54:52.204 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-24T21:54:52.263 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:52.318 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:52.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:52.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:54:52.344 INFO:tasks.rados.rados.0.smithi125.stdout:139: finishing write tid 1 to smithi12556470-15 2021-03-24T21:54:52.344 INFO:tasks.rados.rados.0.smithi125.stdout:139: finishing write tid 2 to smithi12556470-15 2021-03-24T21:54:52.344 INFO:tasks.rados.rados.0.smithi125.stdout:139: finishing write tid 3 to smithi12556470-15 2021-03-24T21:54:52.344 INFO:tasks.rados.rados.0.smithi125.stdout:139: finishing write tid 4 to smithi12556470-15 2021-03-24T21:54:52.345 INFO:tasks.rados.rados.0.smithi125.stdout:139: finishing write tid 5 to smithi12556470-15 2021-03-24T21:54:52.345 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 15 v 32 (ObjNum 77 snap 9 seq_num 77) dirty exists 2021-03-24T21:54:52.345 INFO:tasks.rados.rados.0.smithi125.stdout:139: left oid 15 (ObjNum 77 snap 9 seq_num 77) 2021-03-24T21:54:52.345 INFO:tasks.rados.rados.0.smithi125.stdout:140: finishing write tid 1 to smithi12556470-11 2021-03-24T21:54:52.346 INFO:tasks.rados.rados.0.smithi125.stdout:140: finishing write tid 2 to smithi12556470-11 2021-03-24T21:54:52.346 INFO:tasks.rados.rados.0.smithi125.stdout:136: done (5 left) 2021-03-24T21:54:52.346 INFO:tasks.rados.rados.0.smithi125.stdout:137: done (4 left) 2021-03-24T21:54:52.346 INFO:tasks.rados.rados.0.smithi125.stdout:139: done (3 left) 2021-03-24T21:54:52.346 INFO:tasks.rados.rados.0.smithi125.stdout:142: done (2 left) 2021-03-24T21:54:52.347 INFO:tasks.rados.rados.0.smithi125.stdout:143: rollback oid 15 current snap is 9 2021-03-24T21:54:52.347 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 15 to 7 2021-03-24T21:54:52.347 INFO:tasks.rados.rados.0.smithi125.stdout:140: finishing write tid 3 to smithi12556470-11 2021-03-24T21:54:52.347 INFO:tasks.rados.rados.0.smithi125.stdout:140: finishing write tid 4 to smithi12556470-11 2021-03-24T21:54:52.347 INFO:tasks.rados.rados.0.smithi125.stdout:140: finishing write tid 5 to smithi12556470-11 2021-03-24T21:54:52.348 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 11 v 30 (ObjNum 78 snap 9 seq_num 78) dirty exists 2021-03-24T21:54:52.348 INFO:tasks.rados.rados.0.smithi125.stdout:140: left oid 11 (ObjNum 78 snap 9 seq_num 78) 2021-03-24T21:54:52.348 INFO:tasks.rados.rados.0.smithi125.stdout:141: finishing write tid 1 to smithi12556470-1 2021-03-24T21:54:52.348 INFO:tasks.rados.rados.0.smithi125.stdout:141: finishing write tid 2 to smithi12556470-1 2021-03-24T21:54:52.349 INFO:tasks.rados.rados.0.smithi125.stdout:141: finishing write tid 3 to smithi12556470-1 2021-03-24T21:54:52.349 INFO:tasks.rados.rados.0.smithi125.stdout:141: finishing write tid 4 to smithi12556470-1 2021-03-24T21:54:52.349 INFO:tasks.rados.rados.0.smithi125.stdout:141: finishing write tid 5 to smithi12556470-1 2021-03-24T21:54:52.349 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 1 v 35 (ObjNum 79 snap 9 seq_num 79) dirty exists 2021-03-24T21:54:52.349 INFO:tasks.rados.rados.0.smithi125.stdout:141: left oid 1 (ObjNum 79 snap 9 seq_num 79) 2021-03-24T21:54:52.350 INFO:tasks.rados.rados.0.smithi125.stdout:140: done (2 left) 2021-03-24T21:54:52.350 INFO:tasks.rados.rados.0.smithi125.stdout:141: done (1 left) 2021-03-24T21:54:52.350 INFO:tasks.rados.rados.0.smithi125.stdout:144: snap_remove snap 8 2021-03-24T21:54:52.378 INFO:journalctl@ceph.osd.0.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[28026]: debug 2021-03-24T21:54:52.379 INFO:journalctl@ceph.osd.0.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[28026]: 2021-03-24T21:54:52.115+0000 7f61bbc86700 -1 osd.0 pg_epoch: 123 pg[7.4( v 120'19 (0'0,120'19] local-lis/les=108/109 n=5 ec=99/99 lis/c=108/108 les/c/f=109/110/0 sis=108) [0,4,5] r=0 lpr=108 crt=120'19 lcod 115'17 mlcod 115'17 active+clean+snaptrim trimq=[c~1] ps=[8~1,a~1,d~1]] removing snap head 2021-03-24T21:54:52.379 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[19936]: cluster 2021-03-24T21:54 2021-03-24T21:54:52.379 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[19936]: :50.292015+0000 osd.4 (osd.4) 37 : cluster [DBG] 6.1a scrub ok 2021-03-24T21:54:52.380 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[19936]: cluster 2021-03-24T21:54 2021-03-24T21:54:52.380 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[19936]: :50.714403+0000 mgr.y (mgr. 2021-03-24T21:54:52.380 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[19936]: 14144) 318 : cluster [DBG] 2021-03-24T21:54:52.380 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[19936]: pgmap v344: 148 pgs: 148 active+clean; 120 MiB data, 289 MiB used, 536 GiB / 536 GiB avail; 17 MiB/s rd, 5.1 MiB/s wr, 25 op/s 2021-03-24T21:54:52.381 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:54:52.381 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[19936]: -03-24T21:54:51 2021-03-24T21:54:52.381 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[19936]: .015004+0000 2021-03-24T21:54:52.381 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[19936]: osd.5 (osd.5) 36 : cluster [DBG] 2021-03-24T21:54:52.382 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[19936]: 4.2 scrub ok 2021-03-24T21:54:52.382 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:54:52.382 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[19936]: 21:54:51.202625+0000 osd.2 2021-03-24T21:54:52.382 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[19936]: (osd.2) 23 : cluster [DBG] 5.b scrub ok 2021-03-24T21:54:52.382 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[19936]: cluster 2021-03-24T21:54:52.383 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[19936]: 2021-03- 2021-03-24T21:54:52.383 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[19936]: 24T21:54:51.585046+0000 osd.3 ( 2021-03-24T21:54:52.383 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[19936]: osd.3) 41 : cluster [DBG] 4.11 scrub ok 2021-03-24T21:54:52.383 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:54:52.383 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[19936]: -03-24T21:54:51 2021-03-24T21:54:52.384 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[19936]: .616502+0000 osd.2 (osd.2) 24 : 2021-03-24T21:54:52.384 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[19936]: cluster [DBG] purged_snaps scrub starts 2021-03-24T21:54:52.384 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:54:52.384 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[19936]: 24T21:54:51.616675 2021-03-24T21:54:52.385 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[19936]: +0000 osd.2 (osd.2) 25 2021-03-24T21:54:52.385 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[19936]: : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:54:52.385 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[19936]: audit 2021 2021-03-24T21:54:52.385 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[19936]: -03-24T21:54: 2021-03-24T21:54:52.385 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[19936]: 51.852211+0000 mon.c (mon. 2021-03-24T21:54:52.386 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[19936]: 2) 26 : audit [DBG] from='client.? v1:172.21.15.125:0/3265369720' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:54:52.386 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:54:52.386 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[19936]: -03-24T21:54:52. 2021-03-24T21:54:52.386 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[19936]: 081684+0000 mon.a (mon.0) 2021-03-24T21:54:52.387 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[19936]: 670 : cluster [DBG] osdmap e123: 8 total, 8 up, 6 in 2021-03-24T21:54:52.387 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[24927]: cluster 2021-03-24T21:54:52.387 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[24927]: 2021-03-24T21:54:52.387 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[24927]: 2021 2021-03-24T21:54:52.388 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[24927]: - 2021-03-24T21:54:52.388 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[24927]: 03 2021-03-24T21:54:52.388 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[24927]: - 2021-03-24T21:54:52.388 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[24927]: 24 2021-03-24T21:54:52.388 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[24927]: T 2021-03-24T21:54:52.389 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[24927]: 21 2021-03-24T21:54:52.389 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[24927]: : 2021-03-24T21:54:52.389 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[24927]: 54 2021-03-24T21:54:52.389 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[24927]: : 2021-03-24T21:54:52.390 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[24927]: 50.292015+0000 osd.4 (osd.4) 2021-03-24T21:54:52.390 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[24927]: 37 : cluster [DBG] 6.1a scrub ok 2021-03-24T21:54:52.390 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[24927]: cluster 2021-03-24T21:54:52.390 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[24927]: 2021-03-24T21:54:50.714403+0000 mgr.y (mgr.14144) 318 2021-03-24T21:54:52.391 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[24927]: : cluster [DBG] pgmap v344: 148 pgs: 148 active+clean; 120 MiB data, 289 MiB used, 536 GiB / 536 GiB avail; 17 MiB/s rd, 5.1 MiB/s wr, 25 op/s 2021-03-24T21:54:52.391 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:54:52.391 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[24927]: -03-24T21:54:51 2021-03-24T21:54:52.391 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[24927]: .015004+0000 osd.5 (osd.5 2021-03-24T21:54:52.391 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[24927]: ) 36 : cluster [DBG] 4.2 scrub ok 2021-03-24T21:54:52.392 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[24927]: cluster 2021-03-24T21:54:52.392 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[24927]: 2021-03-24 2021-03-24T21:54:52.392 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[24927]: T21:54:51.202625+0000 osd.2 (osd.2) 2021-03-24T21:54:52.392 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[24927]: 23 : cluster [DBG] 5.b scrub ok 2021-03-24T21:54:52.392 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:54:52.393 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[24927]: 54:51.585046+0000 osd.3 (osd 2021-03-24T21:54:52.393 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[24927]: .3) 41 : cluster [DBG] 4.11 scrub ok 2021-03-24T21:54:52.393 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[24927]: 2021-03-24T21:54:52.393 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[24927]: cluster 2021-03-24T21:54: 2021-03-24T21:54:52.394 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[24927]: 51.616502+0000 osd.2 (osd.2) 24 : cluster 2021-03-24T21:54:52.394 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[24927]: [DBG] purged_snaps scrub starts 2021-03-24T21:54:52.394 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:54:52.394 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[24927]: 24T21:54:51.616675+0000 2021-03-24T21:54:52.395 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[24927]: osd.2 (osd.2) 25 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:54:52.395 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[24927]: 2021-03-24T21:54:52.395 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[24927]: audit 2021-03-24T21:54:52.395 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[24927]: 2021-03-24T21:54:51.852211+0000 2021-03-24T21:54:52.395 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[24927]: mon.c (mon.2) 26 : audit 2021-03-24T21:54:52.396 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[24927]: [DBG] from='client.? v1:172.21.15.125:0/3265369720' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:54:52.396 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[24927]: cluster 2021-03-24T21:54 2021-03-24T21:54:52.396 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[24927]: :52.081684+0000 mon.a (mon.0 2021-03-24T21:54:52.397 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:52 smithi125 conmon[24927]: ) 670 : cluster [DBG] osdmap e123: 8 total, 8 up, 6 in 2021-03-24T21:54:52.423 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:52.485 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:52.487 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:52 smithi142 conmon[20125]: cluster 2021-03-24T21:54:50.292015+0000 osd.4 (osd.4) 37 : cluster 2021-03-24T21:54:52.488 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:52 smithi142 conmon[20125]: [DBG] 6.1a scrub ok 2021-03-24T21:54:52.488 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:52 smithi142 conmon[20125]: cluster 2021-03-24T21:54:50.714403+0000 mgr.y (mgr.14144) 318 : cluster 2021-03-24T21:54:52.488 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:52 smithi142 conmon[20125]: [DBG] pgmap v344: 148 pgs: 148 active+clean; 120 MiB data, 289 MiB used, 536 GiB / 536 GiB avail; 17 MiB/s rd, 5.1 MiB/s wr, 25 op/s 2021-03-24T21:54:52.488 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:52 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:54:52.489 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:52 smithi142 conmon[20125]: 03-24T21:54:51.015004+0000 osd.5 (osd 2021-03-24T21:54:52.489 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:52 smithi142 conmon[20125]: .5) 36 : cluster [DBG] 4.2 scrub ok 2021-03-24T21:54:52.489 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:52 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:54:52.489 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:52 smithi142 conmon[20125]: -03-24T21:54:51.202625 2021-03-24T21:54:52.490 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:52 smithi142 conmon[20125]: +0000 osd.2 (osd.2) 23 : cluster 2021-03-24T21:54:52.490 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:52 smithi142 conmon[20125]: [DBG] 5.b scrub ok 2021-03-24T21:54:52.490 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:52 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:54:52.490 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:52 smithi142 conmon[20125]: -03-24T21:54: 2021-03-24T21:54:52.490 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:52 smithi142 conmon[20125]: 51.585046+0000 osd.3 ( 2021-03-24T21:54:52.491 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:52 smithi142 conmon[20125]: osd.3) 41 : cluster [DBG] 4.11 scrub ok 2021-03-24T21:54:52.491 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:52 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:54:52.491 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:52 smithi142 conmon[20125]: 24T21:54:51.616502+0000 osd.2 (osd.2) 24 : cluster 2021-03-24T21:54:52.491 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:52 smithi142 conmon[20125]: [DBG] purged_snaps scrub starts 2021-03-24T21:54:52.492 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:52 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:54:52.492 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:52 smithi142 conmon[20125]: T21:54:51.616675+0000 2021-03-24T21:54:52.492 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:52 smithi142 conmon[20125]: osd.2 (osd.2) 25 : cluster [DBG] 2021-03-24T21:54:52.492 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:52 smithi142 conmon[20125]: purged_snaps scrub ok 2021-03-24T21:54:52.492 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:52 smithi142 conmon[20125]: audit 2021-03-24 2021-03-24T21:54:52.493 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:52 smithi142 conmon[20125]: T21:54:51.852211+0000 2021-03-24T21:54:52.493 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:52 smithi142 conmon[20125]: mon.c (mon.2) 26 : audit 2021-03-24T21:54:52.493 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:52 smithi142 conmon[20125]: [DBG] from='client.? v1:172.21.15.125:0/3265369720' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:54:52.493 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:52 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:54:52.493 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:52 smithi142 conmon[20125]: 24T21:54:52.081684+0000 2021-03-24T21:54:52.494 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:52 smithi142 conmon[20125]: mon.a (mon.0) 670 : cluster 2021-03-24T21:54:52.494 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:52 smithi142 conmon[20125]: [DBG] osdmap e123: 8 total, 8 up, 6 in 2021-03-24T21:54:52.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:52.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:54:52.596 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:52.658 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:52.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:52.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:54:52.764 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:52.818 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:52.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:52.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:54:52.925 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:52.982 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:52.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:52.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:54:53.089 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:53.148 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:53.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:53.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:54:53.228 INFO:tasks.rados.rados.0.smithi125.stdout:144: done (1 left) 2021-03-24T21:54:53.229 INFO:tasks.rados.rados.0.smithi125.stdout:145: read oid 11 snap -1 2021-03-24T21:54:53.229 INFO:tasks.rados.rados.0.smithi125.stdout:145: expect (ObjNum 78 snap 9 seq_num 78) 2021-03-24T21:54:53.229 INFO:tasks.rados.rados.0.smithi125.stdout:146: write (excl) oid 22 current snap is 9 2021-03-24T21:54:53.229 INFO:tasks.rados.rados.0.smithi125.stdout:143: finishing rollback tid 0 to smithi12556470-15 2021-03-24T21:54:53.230 INFO:tasks.rados.rados.0.smithi125.stdout:146: seq_num 81 ranges {561367=458699,1573512=741359,2941380=419987} 2021-03-24T21:54:53.233 INFO:teuthology.orchestra.run.smithi125.stderr:nodeep-scrub is unset 2021-03-24T21:54:53.235 INFO:tasks.rados.rados.0.smithi125.stdout:146: writing smithi12556470-22 from 561367 to 1020066 tid 1 2021-03-24T21:54:53.243 INFO:tasks.rados.rados.0.smithi125.stdout:146: writing smithi12556470-22 from 1573512 to 2314871 tid 2 2021-03-24T21:54:53.249 INFO:tasks.rados.rados.0.smithi125.stdout:146: writing smithi12556470-22 from 2941380 to 3361367 tid 3 2021-03-24T21:54:53.250 INFO:tasks.rados.rados.0.smithi125.stdout:147: write oid 21 current snap is 9 2021-03-24T21:54:53.250 INFO:tasks.rados.rados.0.smithi125.stdout:147: seq_num 82 ranges {578174=533948,1515641=485017,2736097=502305} 2021-03-24T21:54:53.253 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:53.256 INFO:tasks.rados.rados.0.smithi125.stdout:147: writing smithi12556470-21 from 578174 to 1112122 tid 1 2021-03-24T21:54:53.261 INFO:tasks.rados.rados.0.smithi125.stdout:147: writing smithi12556470-21 from 1515641 to 2000658 tid 2 2021-03-24T21:54:53.269 INFO:tasks.rados.rados.0.smithi125.stdout:147: writing smithi12556470-21 from 2736097 to 3238402 tid 3 2021-03-24T21:54:53.269 INFO:tasks.rados.rados.0.smithi125.stdout:148: snap_create 2021-03-24T21:54:53.269 INFO:tasks.rados.rados.0.smithi125.stdout:143: finishing rollback tid 1 to smithi12556470-15 2021-03-24T21:54:53.269 INFO:tasks.rados.rados.0.smithi125.stdout:143: finishing rollback tid 2 to smithi12556470-15 2021-03-24T21:54:53.270 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 15 v 35 (ObjNum 74 snap 7 seq_num 74) dirty exists 2021-03-24T21:54:53.270 INFO:tasks.rados.rados.0.smithi125.stdout:146: finishing write tid 1 to smithi12556470-22 2021-03-24T21:54:53.270 INFO:tasks.rados.rados.0.smithi125.stdout:145: expect (ObjNum 78 snap 9 seq_num 78) 2021-03-24T21:54:53.299 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:53.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:53.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:54:53.406 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:53.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:53 smithi125 conmon[24927]: audit 2021-03-24T21:54:53.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:53 smithi125 conmon[24927]: 2021-03-24T21:54:52.455211+0000 mon.a (mon.0) 671 : audit 2021-03-24T21:54:53.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:53 smithi125 conmon[24927]: [INF] from='client.? v1:172.21.15.125:0/1645589003' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:54:53.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:53 smithi125 conmon[19936]: audit 2021-03-24T21:54: 2021-03-24T21:54:53.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:53 smithi125 conmon[19936]: 52.455211+0000 mon.a (mon.0) 671 : audit [INF] from='client.? v1:172.21.15.125:0/1645589003' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:54:53.437 INFO:tasks.rados.rados.0.smithi125.stdout:146: finishing write tid 2 to smithi12556470-22 2021-03-24T21:54:53.438 INFO:tasks.rados.rados.0.smithi125.stdout:146: finishing write tid 3 to smithi12556470-22 2021-03-24T21:54:53.438 INFO:tasks.rados.rados.0.smithi125.stdout:146: finishing write tid 4 to smithi12556470-22 2021-03-24T21:54:53.438 INFO:tasks.rados.rados.0.smithi125.stdout:146: finishing write tid 5 to smithi12556470-22 2021-03-24T21:54:53.439 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 22 v 40 (ObjNum 81 snap 9 seq_num 81) dirty exists 2021-03-24T21:54:53.439 INFO:tasks.rados.rados.0.smithi125.stdout:146: left oid 22 (ObjNum 81 snap 9 seq_num 81) 2021-03-24T21:54:53.439 INFO:tasks.rados.rados.0.smithi125.stdout:147: finishing write tid 1 to smithi12556470-21 2021-03-24T21:54:53.439 INFO:tasks.rados.rados.0.smithi125.stdout:147: finishing write tid 2 to smithi12556470-21 2021-03-24T21:54:53.439 INFO:tasks.rados.rados.0.smithi125.stdout:147: finishing write tid 3 to smithi12556470-21 2021-03-24T21:54:53.440 INFO:tasks.rados.rados.0.smithi125.stdout:147: finishing write tid 4 to smithi12556470-21 2021-03-24T21:54:53.440 INFO:tasks.rados.rados.0.smithi125.stdout:147: finishing write tid 5 to smithi12556470-21 2021-03-24T21:54:53.440 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 21 v 44 (ObjNum 82 snap 9 seq_num 82) dirty exists 2021-03-24T21:54:53.440 INFO:tasks.rados.rados.0.smithi125.stdout:147: left oid 21 (ObjNum 82 snap 9 seq_num 82) 2021-03-24T21:54:53.451 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:53.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:53.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:54:53.556 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:54:53.567 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:53 smithi142 conmon[20125]: audit 2021-03-24T21:54: 2021-03-24T21:54:53.567 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:53 smithi142 conmon[20125]: 52.455211+0000 mon.a (mon.0) 671 : audit [INF] from='client.? v1:172.21.15.125:0/1645589003' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:54:53.614 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:54:53.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:53.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:54:53.720 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:53.782 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:53.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:53.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:54:53.887 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:53.943 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:53.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:53.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:54:54.047 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:54.103 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:54.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:54.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:54:54.209 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:54.237 INFO:tasks.rados.rados.0.smithi125.stdout:143: done (4 left) 2021-03-24T21:54:54.237 INFO:tasks.rados.rados.0.smithi125.stdout:145: done (3 left) 2021-03-24T21:54:54.237 INFO:tasks.rados.rados.0.smithi125.stdout:146: done (2 left) 2021-03-24T21:54:54.238 INFO:tasks.rados.rados.0.smithi125.stdout:147: done (1 left) 2021-03-24T21:54:54.238 INFO:tasks.rados.rados.0.smithi125.stdout:148: done (0 left) 2021-03-24T21:54:54.238 INFO:tasks.rados.rados.0.smithi125.stdout:149: read oid 29 snap 9 2021-03-24T21:54:54.238 INFO:tasks.rados.rados.0.smithi125.stdout:149: expect (ObjNum 59 snap 4 seq_num 59) 2021-03-24T21:54:54.239 INFO:tasks.rados.rados.0.smithi125.stdout:150: write (excl) oid 21 current snap is 10 2021-03-24T21:54:54.239 INFO:tasks.rados.rados.0.smithi125.stdout:150: seq_num 83 ranges {594981=609197,1974123=745028,3279873=115108} 2021-03-24T21:54:54.244 INFO:tasks.rados.rados.0.smithi125.stdout:150: writing smithi12556470-21 from 594981 to 1204178 tid 1 2021-03-24T21:54:54.254 INFO:tasks.rados.rados.0.smithi125.stdout:150: writing smithi12556470-21 from 1974123 to 2719151 tid 2 2021-03-24T21:54:54.258 INFO:tasks.rados.rados.0.smithi125.stdout:150: writing smithi12556470-21 from 3279873 to 3394981 tid 3 2021-03-24T21:54:54.258 INFO:tasks.rados.rados.0.smithi125.stdout:151: delete oid 23 current snap is 10 2021-03-24T21:54:54.259 INFO:tasks.rados.rados.0.smithi125.stdout:149: expect (ObjNum 59 snap 4 seq_num 59) 2021-03-24T21:54:54.273 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:54:54.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:54.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:54:54.383 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:54.399 INFO:tasks.rados.rados.0.smithi125.stdout:150: finishing write tid 1 to smithi12556470-21 2021-03-24T21:54:54.399 INFO:tasks.rados.rados.0.smithi125.stdout:150: finishing write tid 2 to smithi12556470-21 2021-03-24T21:54:54.400 INFO:tasks.rados.rados.0.smithi125.stdout:150: finishing write tid 3 to smithi12556470-21 2021-03-24T21:54:54.400 INFO:tasks.rados.rados.0.smithi125.stdout:150: finishing write tid 4 to smithi12556470-21 2021-03-24T21:54:54.400 INFO:tasks.rados.rados.0.smithi125.stdout:150: finishing write tid 5 to smithi12556470-21 2021-03-24T21:54:54.400 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 21 v 49 (ObjNum 83 snap 10 seq_num 83) dirty exists 2021-03-24T21:54:54.401 INFO:tasks.rados.rados.0.smithi125.stdout:150: left oid 21 (ObjNum 83 snap 10 seq_num 83) 2021-03-24T21:54:54.401 INFO:tasks.rados.rados.0.smithi125.stdout:149: done (2 left) 2021-03-24T21:54:54.401 INFO:tasks.rados.rados.0.smithi125.stdout:150: done (1 left) 2021-03-24T21:54:54.401 INFO:tasks.rados.rados.0.smithi125.stdout:151: done (0 left) 2021-03-24T21:54:54.401 INFO:tasks.rados.rados.0.smithi125.stdout:152: read oid 28 snap 7 2021-03-24T21:54:54.402 INFO:tasks.rados.rados.0.smithi125.stdout:152: expect deleted 2021-03-24T21:54:54.402 INFO:tasks.rados.rados.0.smithi125.stdout:153: snap_remove snap 9 2021-03-24T21:54:54.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:54 smithi125 conmon[24927]: cluster 2021-03-24T21:54:52.385560+0000 2021-03-24T21:54:54.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:54 smithi125 conmon[24927]: osd.0 (osd.0) 40 : cluster [DBG] 3.10 scrub ok 2021-03-24T21:54:54.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:54 smithi125 conmon[24927]: cluster 2021-03-24T21:54:54.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:54 smithi125 conmon[24927]: 2021-03-24T21:54:52.715071+0000 mgr.y (mgr.14144) 319 : cluster [DBG] pgmap v347: 148 pgs: 148 active+clean; 120 MiB data, 296 MiB used, 536 GiB / 536 GiB avail; 8.8 MiB/s rd, 3.3 MiB/s wr, 15 op/s 2021-03-24T21:54:54.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:54 smithi125 conmon[24927]: 2021-03-24T21:54:54.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:54 smithi125 conmon[24927]: cluster 2021-03-24T21:54 2021-03-24T21:54:54.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:54 smithi125 conmon[24927]: :53.212988+0000 mon.a (mon.0) 672 : cluster 2021-03-24T21:54:54.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:54 smithi125 conmon[24927]: [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T21:54:54.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:54 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:54:54.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:54 smithi125 conmon[24927]: -24T21:54:53.213041+0000 mon.a (mon. 2021-03-24T21:54:54.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:54 smithi125 conmon[24927]: 0) 673 : cluster [INF] Cluster is now healthy 2021-03-24T21:54:54.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:54 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T21:54:54.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:54 smithi125 conmon[24927]: 24T21:54:53.227123+0000 mon.a (mon.0) 2021-03-24T21:54:54.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:54 smithi125 conmon[24927]: 674 : audit [INF] from='client.? v1:172.21.15.125:0/1645589003' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T21:54:54.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:54 smithi125 conmon[24927]: cluster 2021-03-24T21:54:53.227251+0000 mon.a (mon.0) 675 : cluster [DBG] osdmap e124: 8 total, 8 up, 6 in 2021-03-24T21:54:54.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:54 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:54:54.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:54 smithi125 conmon[19936]: 24T21:54:52.385560+0000 osd.0 (osd.0) 40 : cluster 2021-03-24T21:54:54.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:54 smithi125 conmon[19936]: [DBG] 3.10 scrub ok 2021-03-24T21:54:54.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:54 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:54:54.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:54 smithi125 conmon[19936]: :54:52.715071+0000 mgr.y (mgr.14144) 319 : cluster [DBG] pgmap v347: 148 pgs: 148 active+clean; 120 MiB data, 296 MiB used, 536 GiB / 536 GiB avail; 8.8 MiB/s rd, 3.3 MiB/s wr, 15 op/s 2021-03-24T21:54:54.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:54 smithi125 conmon[19936]: cluster 2021-03-24T21:54:54.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:54 smithi125 conmon[19936]: 2021-03-24T21:54:53.212988+0000 mon.a (mon.0) 672 : cluster [INF] 2021-03-24T21:54:54.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:54 smithi125 conmon[19936]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T21:54:54.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:54 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:54:54.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:54 smithi125 conmon[19936]: 21:54:53.213041+0000 mon.a (mon.0) 673 : cluster [INF] Cluster is now healthy 2021-03-24T21:54:54.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:54 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:54:54.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:54 smithi125 conmon[19936]: 03-24T21:54:53.227123+0000 2021-03-24T21:54:54.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:54 smithi125 conmon[19936]: mon.a (mon.0) 674 : audit [INF] 2021-03-24T21:54:54.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:54 smithi125 conmon[19936]: from='client.? v1:172.21.15.125:0/1645589003' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T21:54:54.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:54 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:54:54.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:54 smithi125 conmon[19936]: -24T21:54:53.227251+0000 2021-03-24T21:54:54.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:54 smithi125 conmon[19936]: mon.a (mon.0) 675 : cluster [DBG] osdmap e124: 8 total, 8 up, 6 in 2021-03-24T21:54:54.441 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:54.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:54.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:54:54.547 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:54:54.559 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:54 smithi142 conmon[20125]: cluster 2021-03-24T21:54:52.385560+0000 osd.0 (osd.0) 40 : cluster 2021-03-24T21:54:54.560 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:54 smithi142 conmon[20125]: [DBG] 3.10 scrub ok 2021-03-24T21:54:54.560 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:54 smithi142 conmon[20125]: cluster 2021-03-24T21:54:52.715071+0000 mgr.y (mgr.14144) 319 : cluster 2021-03-24T21:54:54.560 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:54 smithi142 conmon[20125]: [DBG] pgmap v347: 148 pgs: 148 active+clean; 120 MiB data, 296 MiB used, 536 GiB / 536 GiB avail; 8.8 MiB/s rd, 3.3 MiB/s wr, 15 op/s 2021-03-24T21:54:54.560 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:54 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:54:54.561 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:54 smithi142 conmon[20125]: -03-24T21:54:53.212988+0000 mon.a (mon.0 2021-03-24T21:54:54.561 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:54 smithi142 conmon[20125]: ) 672 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T21:54:54.561 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:54 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:54:54.561 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:54 smithi142 conmon[20125]: 24T21:54:53.213041+0000 mon.a 2021-03-24T21:54:54.561 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:54 smithi142 conmon[20125]: (mon.0) 673 : cluster [INF] Cluster is now healthy 2021-03-24T21:54:54.562 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:54 smithi142 conmon[20125]: 2021-03-24T21:54:54.562 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:54 smithi142 conmon[20125]: audit 2021 2021-03-24T21:54:54.562 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:54 smithi142 conmon[20125]: -03-24T21:54:53.227123+0000 mon.a ( 2021-03-24T21:54:54.562 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:54 smithi142 conmon[20125]: mon.0) 674 : audit [INF] from='client.? v1:172.21.15.125:0/1645589003' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T21:54:54.563 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:54 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:54:54.563 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:54 smithi142 conmon[20125]: -24T21:54:53.227251+0000 2021-03-24T21:54:54.563 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:54 smithi142 conmon[20125]: mon.a (mon.0) 675 : cluster [DBG] 2021-03-24T21:54:54.563 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:54 smithi142 conmon[20125]: osdmap e124: 8 total, 8 up, 6 in 2021-03-24T21:54:54.576 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 7] out_osds: [6, 1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-03-24T21:54:54.576 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-24T21:54:54.576 INFO:tasks.thrashosds.thrasher:inject_pause on 7 2021-03-24T21:54:54.576 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2021-03-24T21:54:54.577 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2021-03-24T21:54:54.577 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.7 config set filestore_inject_stall 3 2021-03-24T21:54:54.601 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:54:54.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:54.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:54:54.706 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:54.765 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:54.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:54.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:54:54.869 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:54.874 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T21:54:54.912 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-24T21:54:54.912 INFO:teuthology.orchestra.run.smithi125.stdout: "size": 20, 2021-03-24T21:54:54.913 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 600, 2021-03-24T21:54:54.913 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [ 2021-03-24T21:54:54.913 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.913 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.34352.0:535 4.b e65/62 4:d1061236:::data_log.86:head v 65'3, mlcod=65'1)", 2021-03-24T21:54:54.913 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:51:50.034175+0000", 2021-03-24T21:54:54.914 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 184.873019715, 2021-03-24T21:54:54.914 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.017611492999999999, 2021-03-24T21:54:54.914 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:54:54.914 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:54:54.915 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:54:54.915 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.915 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:54:54.915 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.034175+0000", 2021-03-24T21:54:54.915 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:54:54.916 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.916 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.916 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:54:54.916 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.034175+0000", 2021-03-24T21:54:54.917 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T21:54:54.917 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.917 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.917 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:54:54.917 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.034173+0000", 2021-03-24T21:54:54.918 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3985000000000001e-05 2021-03-24T21:54:54.918 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.918 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.918 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:54:54.919 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.034187+0000", 2021-03-24T21:54:54.919 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7768e-05 2021-03-24T21:54:54.919 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.919 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.919 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:54:54.920 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.034205+0000", 2021-03-24T21:54:54.920 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4919999999999998e-06 2021-03-24T21:54:54.920 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.920 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.920 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:54:54.921 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.034211+0000", 2021-03-24T21:54:54.921 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5782000000000002e-05 2021-03-24T21:54:54.921 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.921 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.922 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:54:54.922 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.034267+0000", 2021-03-24T21:54:54.922 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.4258000000000003e-05 2021-03-24T21:54:54.922 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.923 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.923 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:54:54.923 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.034301+0000", 2021-03-24T21:54:54.923 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.017442103 2021-03-24T21:54:54.923 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.924 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.924 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:54:54.924 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.051743+0000", 2021-03-24T21:54:54.924 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3061e-05 2021-03-24T21:54:54.924 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.925 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.925 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:54:54.925 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.051786+0000", 2021-03-24T21:54:54.925 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.017611492999999999 2021-03-24T21:54:54.926 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:54.926 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:54:54.926 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:54.926 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.927 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.927 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24376.0:221 4.2 4:479ba34f:::obj_delete_at_hint.0000000021:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e69)", 2021-03-24T21:54:54.927 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:51:53.942625+0000", 2021-03-24T21:54:54.927 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 180.96457007399999, 2021-03-24T21:54:54.927 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0044149480000000001, 2021-03-24T21:54:54.928 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:54:54.928 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:54:54.928 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:54:54.928 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24376", 2021-03-24T21:54:54.928 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/3051439499", 2021-03-24T21:54:54.929 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 221 2021-03-24T21:54:54.929 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.929 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:54:54.929 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.930 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:54:54.930 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.942625+0000", 2021-03-24T21:54:54.930 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:54:54.930 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.930 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.931 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:54:54.931 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.942625+0000", 2021-03-24T21:54:54.931 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T21:54:54.931 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.931 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.932 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:54:54.932 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.942623+0000", 2021-03-24T21:54:54.932 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.4860000000000003e-06 2021-03-24T21:54:54.932 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.933 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.933 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:54:54.933 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.942632+0000", 2021-03-24T21:54:54.934 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7484000000000001e-05 2021-03-24T21:54:54.935 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.935 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.935 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:54:54.935 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.942650+0000", 2021-03-24T21:54:54.936 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.131e-06 2021-03-24T21:54:54.936 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.936 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.936 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:54:54.937 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.942657+0000", 2021-03-24T21:54:54.937 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.9171999999999993e-05 2021-03-24T21:54:54.937 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.937 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.937 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:54:54.938 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.942736+0000", 2021-03-24T21:54:54.938 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000181931 2021-03-24T21:54:54.938 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.938 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.939 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:54:54.939 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.942918+0000", 2021-03-24T21:54:54.939 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00019801000000000001 2021-03-24T21:54:54.939 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.940 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.940 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 4,5", 2021-03-24T21:54:54.940 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.943116+0000", 2021-03-24T21:54:54.940 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00048897899999999995 2021-03-24T21:54:54.940 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.941 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.941 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:54:54.941 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.943605+0000", 2021-03-24T21:54:54.941 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000288492 2021-03-24T21:54:54.942 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.942 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.942 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:54:54.942 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.943893+0000", 2021-03-24T21:54:54.943 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0030618160000000002 2021-03-24T21:54:54.943 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.943 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.943 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:54:54.944 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.946955+0000", 2021-03-24T21:54:54.944 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.7068999999999999e-05 2021-03-24T21:54:54.944 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.944 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.944 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:54:54.945 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.947002+0000", 2021-03-24T21:54:54.945 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.7311999999999998e-05 2021-03-24T21:54:54.945 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.945 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.946 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:54:54.946 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.947040+0000", 2021-03-24T21:54:54.946 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0044149480000000001 2021-03-24T21:54:54.946 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:54.946 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:54:54.947 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:54.947 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.947 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.947 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:33 7.b 7:d821c111:::smithi12556470-9:head [write 534456~718345,stat] snapc 0=[] ondisk+write+known_if_redirected e107)", 2021-03-24T21:54:54.948 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.193321+0000", 2021-03-24T21:54:54.948 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 18.713873455000002, 2021-03-24T21:54:54.948 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0060392889999999998, 2021-03-24T21:54:54.948 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:54:54.948 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:54:54.949 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:54:54.949 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:54:54.949 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:54:54.949 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 33 2021-03-24T21:54:54.950 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.950 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:54:54.950 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.950 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:54:54.951 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.193321+0000", 2021-03-24T21:54:54.951 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:54:54.951 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.951 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.952 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:54:54.952 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.193321+0000", 2021-03-24T21:54:54.952 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999976 2021-03-24T21:54:54.952 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.953 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.953 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:54:54.953 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.193319+0000", 2021-03-24T21:54:54.953 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00038596399999999998 2021-03-24T21:54:54.954 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.954 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.954 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:54:54.954 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.193705+0000", 2021-03-24T21:54:54.954 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.8202e-05 2021-03-24T21:54:54.955 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.955 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.955 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:54:54.955 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.193723+0000", 2021-03-24T21:54:54.955 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.6779999999999995e-06 2021-03-24T21:54:54.956 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.956 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.956 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:54:54.956 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.193733+0000", 2021-03-24T21:54:54.956 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.7935000000000005e-05 2021-03-24T21:54:54.957 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.957 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.957 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:54:54.957 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.193810+0000", 2021-03-24T21:54:54.957 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00016634899999999999 2021-03-24T21:54:54.958 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.958 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.958 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:54:54.958 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.193977+0000", 2021-03-24T21:54:54.958 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00021176300000000001 2021-03-24T21:54:54.959 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.959 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.959 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,5", 2021-03-24T21:54:54.959 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.194189+0000", 2021-03-24T21:54:54.959 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0020706510000000002 2021-03-24T21:54:54.960 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.960 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.960 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:54:54.960 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.196259+0000", 2021-03-24T21:54:54.961 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.4345000000000002e-05 2021-03-24T21:54:54.961 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.961 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.961 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:54:54.961 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.196284+0000", 2021-03-24T21:54:54.962 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0028126039999999998 2021-03-24T21:54:54.962 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.962 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.962 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:54:54.962 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.199096+0000", 2021-03-24T21:54:54.963 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.7195999999999997e-05 2021-03-24T21:54:54.963 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.963 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.963 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:54:54.963 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.199173+0000", 2021-03-24T21:54:54.964 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00018708400000000001 2021-03-24T21:54:54.964 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.964 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.964 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:54:54.964 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.199360+0000", 2021-03-24T21:54:54.965 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0060392889999999998 2021-03-24T21:54:54.965 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:54.965 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:54:54.965 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:54.966 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.966 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.966 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:34 7.b 7:d821c111:::smithi12556470-9:head [write 1951503~182953] snapc 0=[] ondisk+write+known_if_redirected e107)", 2021-03-24T21:54:54.966 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.197792+0000", 2021-03-24T21:54:54.967 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 18.709402382, 2021-03-24T21:54:54.967 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0040527949999999997, 2021-03-24T21:54:54.967 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:54:54.967 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:54:54.968 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:54:54.968 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:54:54.968 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:54:54.968 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 34 2021-03-24T21:54:54.968 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.969 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:54:54.969 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.969 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:54:54.969 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.197792+0000", 2021-03-24T21:54:54.969 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:54:54.970 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.970 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.970 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:54:54.970 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.197792+0000", 2021-03-24T21:54:54.971 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999976 2021-03-24T21:54:54.971 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.971 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.971 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:54:54.972 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.197790+0000", 2021-03-24T21:54:54.972 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.4402999999999997e-05 2021-03-24T21:54:54.972 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.972 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.973 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:54:54.973 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.197884+0000", 2021-03-24T21:54:54.973 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9775999999999998e-05 2021-03-24T21:54:54.973 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.973 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.974 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:54:54.974 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.197904+0000", 2021-03-24T21:54:54.974 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.4100000000000008e-06 2021-03-24T21:54:54.974 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.974 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.975 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:54:54.975 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.197912+0000", 2021-03-24T21:54:54.975 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.2970000000000001e-05 2021-03-24T21:54:54.975 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.975 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.976 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:54:54.976 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.197985+0000", 2021-03-24T21:54:54.976 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000144194 2021-03-24T21:54:54.976 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.976 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.977 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T21:54:54.977 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.198130+0000", 2021-03-24T21:54:54.977 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0013126240000000001 2021-03-24T21:54:54.977 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.977 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.978 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:54:54.978 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.199442+0000", 2021-03-24T21:54:54.978 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.276e-05 2021-03-24T21:54:54.978 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.978 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.979 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:54:54.979 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.199535+0000", 2021-03-24T21:54:54.979 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00021443300000000001 2021-03-24T21:54:54.979 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.979 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.980 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,5", 2021-03-24T21:54:54.980 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.199749+0000", 2021-03-24T21:54:54.980 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001662048 2021-03-24T21:54:54.980 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.980 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.981 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:54:54.981 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.201411+0000", 2021-03-24T21:54:54.981 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.5590000000000006e-06 2021-03-24T21:54:54.981 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.982 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.982 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:54:54.982 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.201420+0000", 2021-03-24T21:54:54.982 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000318401 2021-03-24T21:54:54.983 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.983 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.983 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:54:54.984 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.201738+0000", 2021-03-24T21:54:54.984 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.8393999999999997e-05 2021-03-24T21:54:54.984 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.984 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.984 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:54:54.985 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.201797+0000", 2021-03-24T21:54:54.985 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.8189999999999998e-05 2021-03-24T21:54:54.986 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.986 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.986 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:54:54.986 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.201845+0000", 2021-03-24T21:54:54.987 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0040527949999999997 2021-03-24T21:54:54.987 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:54.987 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:54:54.987 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:54.987 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.988 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.988 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:35 7.b 7:d821c111:::smithi12556470-9:head [setxattr _header (55) in=7b,truncate 2134456] snapc 0=[] ondisk+write+known_if_redirected e107)", 2021-03-24T21:54:54.988 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.197928+0000", 2021-03-24T21:54:54.988 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 18.709267003000001, 2021-03-24T21:54:54.989 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0041834120000000001, 2021-03-24T21:54:54.989 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:54:54.989 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:54:54.989 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:54:54.989 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:54:54.990 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:54:54.990 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 35 2021-03-24T21:54:54.990 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.990 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:54:54.991 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.991 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:54:54.991 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.197928+0000", 2021-03-24T21:54:54.991 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:54:54.991 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.992 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.992 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:54:54.992 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.197928+0000", 2021-03-24T21:54:54.992 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T21:54:54.993 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.993 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.993 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:54:54.993 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.197926+0000", 2021-03-24T21:54:54.994 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.653e-06 2021-03-24T21:54:54.994 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.994 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.994 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:54:54.994 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.197935+0000", 2021-03-24T21:54:54.995 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1636e-05 2021-03-24T21:54:54.995 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.995 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.995 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:54:54.996 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.197946+0000", 2021-03-24T21:54:54.996 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.9549999999999998e-06 2021-03-24T21:54:54.996 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.996 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.996 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:54:54.997 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.197955+0000", 2021-03-24T21:54:54.997 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00023729900000000001 2021-03-24T21:54:54.997 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.997 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.998 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:54:54.998 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.198193+0000", 2021-03-24T21:54:54.998 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000111729 2021-03-24T21:54:54.998 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:54.999 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:54.999 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T21:54:54.999 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.198304+0000", 2021-03-24T21:54:54.999 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0021900470000000001 2021-03-24T21:54:54.999 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.000 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.000 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:54:55.000 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.200494+0000", 2021-03-24T21:54:55.000 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00020013600000000001 2021-03-24T21:54:55.001 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.001 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.001 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:54:55.001 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.200694+0000", 2021-03-24T21:54:55.002 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00025853600000000002 2021-03-24T21:54:55.002 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.002 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.002 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,5", 2021-03-24T21:54:55.003 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.200953+0000", 2021-03-24T21:54:55.003 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00048642299999999999 2021-03-24T21:54:55.003 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.003 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.003 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:54:55.004 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.201439+0000", 2021-03-24T21:54:55.004 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00047360399999999997 2021-03-24T21:54:55.005 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.005 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.005 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:54:55.005 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.201913+0000", 2021-03-24T21:54:55.005 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.1973999999999998e-05 2021-03-24T21:54:55.006 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.006 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.006 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:54:55.006 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.201995+0000", 2021-03-24T21:54:55.007 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.7000999999999999e-05 2021-03-24T21:54:55.007 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.007 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.007 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:54:55.007 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.202052+0000", 2021-03-24T21:54:55.008 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9033999999999998e-05 2021-03-24T21:54:55.008 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.008 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.008 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:54:55.008 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.202111+0000", 2021-03-24T21:54:55.009 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0041834120000000001 2021-03-24T21:54:55.009 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:55.009 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:54:55.009 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:55.010 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.010 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.010 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:36 7.b 7:d821c111:::smithi12556470-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e107)", 2021-03-24T21:54:55.011 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.197999+0000", 2021-03-24T21:54:55.011 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 18.709195948000001, 2021-03-24T21:54:55.011 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0044019999999999997, 2021-03-24T21:54:55.011 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:54:55.012 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T21:54:55.012 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:54:55.012 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:54:55.012 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:54:55.013 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 36 2021-03-24T21:54:55.013 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.013 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:54:55.013 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.014 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:54:55.014 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.197999+0000", 2021-03-24T21:54:55.014 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:54:55.014 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.015 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.015 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:54:55.015 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.197999+0000", 2021-03-24T21:54:55.015 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T21:54:55.015 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.016 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.016 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:54:55.016 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.197997+0000", 2021-03-24T21:54:55.016 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.576e-06 2021-03-24T21:54:55.016 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.017 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.017 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:54:55.017 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.198005+0000", 2021-03-24T21:54:55.017 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2581e-05 2021-03-24T21:54:55.017 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.018 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.018 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:54:55.018 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.198017+0000", 2021-03-24T21:54:55.018 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.6400000000000003e-06 2021-03-24T21:54:55.018 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.019 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.019 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:54:55.019 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.198026+0000", 2021-03-24T21:54:55.019 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00031787599999999998 2021-03-24T21:54:55.020 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.020 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.020 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:54:55.020 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.198344+0000", 2021-03-24T21:54:55.021 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.3399999999999993e-05 2021-03-24T21:54:55.021 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.021 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.021 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T21:54:55.021 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.198437+0000", 2021-03-24T21:54:55.022 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0028437660000000002 2021-03-24T21:54:55.022 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.022 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.022 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:54:55.023 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.201281+0000", 2021-03-24T21:54:55.023 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2627999999999996e-05 2021-03-24T21:54:55.023 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.023 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.023 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T21:54:55.024 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.201364+0000", 2021-03-24T21:54:55.024 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000821461 2021-03-24T21:54:55.024 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.024 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.024 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:54:55.025 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.202185+0000", 2021-03-24T21:54:55.025 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010005699999999999 2021-03-24T21:54:55.025 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.025 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.026 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:54:55.026 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.202285+0000", 2021-03-24T21:54:55.026 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000115427 2021-03-24T21:54:55.026 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.026 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.027 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:54:55.027 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.202401+0000", 2021-03-24T21:54:55.027 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0044019999999999997 2021-03-24T21:54:55.027 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:55.028 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:54:55.028 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:55.028 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.028 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.028 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:53 7.b 7:da3cc97e:::smithi12556470-14:head [write 618491~694590,stat] snapc 0=[] ondisk+write+known_if_redirected e107)", 2021-03-24T21:54:55.029 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.255785+0000", 2021-03-24T21:54:55.029 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 18.651409607000002, 2021-03-24T21:54:55.029 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0052466329999999997, 2021-03-24T21:54:55.029 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:54:55.030 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:54:55.030 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:54:55.030 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:54:55.030 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:54:55.030 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 53 2021-03-24T21:54:55.031 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.031 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:54:55.031 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.031 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:54:55.032 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.255785+0000", 2021-03-24T21:54:55.032 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:54:55.032 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.032 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.032 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:54:55.033 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.255785+0000", 2021-03-24T21:54:55.033 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999967 2021-03-24T21:54:55.033 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.033 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.034 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:54:55.034 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.255781+0000", 2021-03-24T21:54:55.034 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00027362600000000001 2021-03-24T21:54:55.034 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.034 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.035 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:54:55.035 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.256055+0000", 2021-03-24T21:54:55.035 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.9417e-05 2021-03-24T21:54:55.035 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.036 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.036 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:54:55.036 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.256084+0000", 2021-03-24T21:54:55.036 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1114000000000001e-05 2021-03-24T21:54:55.037 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.037 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.037 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:54:55.037 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.256096+0000", 2021-03-24T21:54:55.037 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00014862799999999999 2021-03-24T21:54:55.038 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.038 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.038 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:54:55.038 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.256244+0000", 2021-03-24T21:54:55.039 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000355247 2021-03-24T21:54:55.039 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.039 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.039 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:54:55.039 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.256599+0000", 2021-03-24T21:54:55.040 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00049522699999999999 2021-03-24T21:54:55.040 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.040 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.040 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,5", 2021-03-24T21:54:55.041 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.257095+0000", 2021-03-24T21:54:55.041 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0015160180000000001 2021-03-24T21:54:55.041 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.041 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.041 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:54:55.042 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.258611+0000", 2021-03-24T21:54:55.042 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0010272020000000001 2021-03-24T21:54:55.042 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.042 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.042 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:54:55.043 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.259638+0000", 2021-03-24T21:54:55.043 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001131791 2021-03-24T21:54:55.043 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.043 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.044 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:54:55.044 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.260770+0000", 2021-03-24T21:54:55.044 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012263000000000001 2021-03-24T21:54:55.044 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.044 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.045 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:54:55.045 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.260892+0000", 2021-03-24T21:54:55.045 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000139309 2021-03-24T21:54:55.045 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.046 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.046 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:54:55.046 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.261032+0000", 2021-03-24T21:54:55.046 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0052466329999999997 2021-03-24T21:54:55.047 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:55.047 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:54:55.047 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:55.047 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.048 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.048 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:62 7.7 e107/101 7:e7492b26:::smithi12556470-16:head v 107'5, mlcod=107'3)", 2021-03-24T21:54:55.048 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.287402+0000", 2021-03-24T21:54:55.048 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 18.619792189000002, 2021-03-24T21:54:55.048 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0054094420000000004, 2021-03-24T21:54:55.049 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:54:55.049 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:54:55.049 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:54:55.049 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.050 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:54:55.050 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.287402+0000", 2021-03-24T21:54:55.050 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:54:55.050 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.051 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.051 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:54:55.051 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.287402+0000", 2021-03-24T21:54:55.051 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999976 2021-03-24T21:54:55.052 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.052 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.052 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:54:55.052 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.287400+0000", 2021-03-24T21:54:55.052 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000213122 2021-03-24T21:54:55.053 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.053 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.053 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:54:55.053 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.287613+0000", 2021-03-24T21:54:55.054 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.8102e-05 2021-03-24T21:54:55.054 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.054 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.054 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:54:55.055 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.287641+0000", 2021-03-24T21:54:55.055 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.8050000000000001e-06 2021-03-24T21:54:55.055 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.055 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.056 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:54:55.056 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.287651+0000", 2021-03-24T21:54:55.056 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9985000000000003e-05 2021-03-24T21:54:55.056 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.056 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.057 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:54:55.057 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.287711+0000", 2021-03-24T21:54:55.057 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.1588999999999999e-05 2021-03-24T21:54:55.057 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.058 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.058 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:54:55.058 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.287743+0000", 2021-03-24T21:54:55.058 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0050277150000000003 2021-03-24T21:54:55.059 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.059 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.059 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:54:55.059 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.292770+0000", 2021-03-24T21:54:55.059 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.1461000000000002e-05 2021-03-24T21:54:55.060 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.060 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.060 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:54:55.060 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.292812+0000", 2021-03-24T21:54:55.061 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0054094420000000004 2021-03-24T21:54:55.061 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:55.061 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:54:55.061 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:55.061 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.062 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.062 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:63 7.7 e107/101 7:e7492b26:::smithi12556470-16:head v 107'6, mlcod=107'3)", 2021-03-24T21:54:55.062 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.288459+0000", 2021-03-24T21:54:55.062 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 18.618735418, 2021-03-24T21:54:55.063 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0043744769999999999, 2021-03-24T21:54:55.063 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:54:55.063 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:54:55.063 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:54:55.063 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.064 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:54:55.064 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.288459+0000", 2021-03-24T21:54:55.064 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:54:55.064 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.065 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.065 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:54:55.065 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.288459+0000", 2021-03-24T21:54:55.065 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T21:54:55.066 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.066 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.066 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:54:55.067 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.288458+0000", 2021-03-24T21:54:55.067 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.2450000000000004e-06 2021-03-24T21:54:55.067 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.067 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.068 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:54:55.068 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.288467+0000", 2021-03-24T21:54:55.068 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5017e-05 2021-03-24T21:54:55.068 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.069 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.069 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:54:55.069 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.288482+0000", 2021-03-24T21:54:55.069 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.5370000000000003e-06 2021-03-24T21:54:55.070 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.070 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.070 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:54:55.070 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.288488+0000", 2021-03-24T21:54:55.070 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6026999999999998e-05 2021-03-24T21:54:55.071 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.071 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.071 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:54:55.071 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.288534+0000", 2021-03-24T21:54:55.072 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9232000000000002e-05 2021-03-24T21:54:55.072 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.072 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.072 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:54:55.073 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.288554+0000", 2021-03-24T21:54:55.073 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0042627170000000001 2021-03-24T21:54:55.073 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.073 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.074 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:54:55.074 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.292816+0000", 2021-03-24T21:54:55.074 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7295e-05 2021-03-24T21:54:55.074 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.075 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.075 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:54:55.075 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.292834+0000", 2021-03-24T21:54:55.075 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0043744769999999999 2021-03-24T21:54:55.075 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:55.076 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:54:55.076 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:55.076 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.076 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.077 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:82 7.d 7:b2218182:::smithi12556470-21:head [write 2020581~315559] snapc 0=[] ondisk+write+known_if_redirected e107)", 2021-03-24T21:54:55.077 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.359676+0000", 2021-03-24T21:54:55.077 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 18.547518146000002, 2021-03-24T21:54:55.077 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0053478029999999999, 2021-03-24T21:54:55.077 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:54:55.078 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:54:55.078 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:54:55.078 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:54:55.078 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:54:55.079 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 82 2021-03-24T21:54:55.079 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.079 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:54:55.079 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.080 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:54:55.080 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359676+0000", 2021-03-24T21:54:55.080 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:54:55.080 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.081 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.081 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:54:55.081 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359676+0000", 2021-03-24T21:54:55.082 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T21:54:55.082 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.082 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.083 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:54:55.083 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359674+0000", 2021-03-24T21:54:55.084 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000130075 2021-03-24T21:54:55.084 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.085 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.085 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:54:55.085 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359804+0000", 2021-03-24T21:54:55.086 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.2365999999999999e-05 2021-03-24T21:54:55.086 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.087 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.087 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:54:55.087 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359827+0000", 2021-03-24T21:54:55.088 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.6319999999999997e-06 2021-03-24T21:54:55.088 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.088 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.089 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:54:55.089 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359835+0000", 2021-03-24T21:54:55.089 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.2173000000000002e-05 2021-03-24T21:54:55.089 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.090 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.090 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:54:55.090 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359908+0000", 2021-03-24T21:54:55.091 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00011309199999999999 2021-03-24T21:54:55.091 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.091 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.091 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:54:55.092 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.360021+0000", 2021-03-24T21:54:55.092 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00020120199999999999 2021-03-24T21:54:55.092 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.092 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.093 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,3", 2021-03-24T21:54:55.093 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.360222+0000", 2021-03-24T21:54:55.093 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0013165919999999999 2021-03-24T21:54:55.093 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.094 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.094 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:54:55.094 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.361538+0000", 2021-03-24T21:54:55.094 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00059871600000000001 2021-03-24T21:54:55.095 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.095 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.095 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:54:55.095 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.362137+0000", 2021-03-24T21:54:55.095 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0027486580000000002 2021-03-24T21:54:55.096 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.096 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.096 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:54:55.096 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.364886+0000", 2021-03-24T21:54:55.097 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.852e-05 2021-03-24T21:54:55.097 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.097 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.097 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:54:55.097 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.364964+0000", 2021-03-24T21:54:55.098 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9901000000000001e-05 2021-03-24T21:54:55.098 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.098 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.098 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:54:55.099 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.365024+0000", 2021-03-24T21:54:55.099 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0053478029999999999 2021-03-24T21:54:55.099 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:55.099 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:54:55.100 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:55.100 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.100 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.100 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:83 7.d 7:b2218182:::smithi12556470-21:head [setxattr _header (56) in=7b,truncate 2336140] snapc 0=[] ondisk+write+known_if_redirected e107)", 2021-03-24T21:54:55.100 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.359856+0000", 2021-03-24T21:54:55.101 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 18.54733843, 2021-03-24T21:54:55.106 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0053737860000000002, 2021-03-24T21:54:55.106 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:54:55.107 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:54:55.107 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:54:55.107 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:54:55.107 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:54:55.108 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 83 2021-03-24T21:54:55.108 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.108 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:54:55.108 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.109 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:54:55.109 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359856+0000", 2021-03-24T21:54:55.109 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:54:55.109 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.110 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.110 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:54:55.110 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359856+0000", 2021-03-24T21:54:55.110 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T21:54:55.111 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.113 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.114 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:54:55.114 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359854+0000", 2021-03-24T21:54:55.114 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.8180000000000004e-06 2021-03-24T21:54:55.114 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.115 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.115 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:54:55.115 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359862+0000", 2021-03-24T21:54:55.115 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.0979999999999996e-06 2021-03-24T21:54:55.116 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.116 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.116 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:54:55.116 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359869+0000", 2021-03-24T21:54:55.117 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2680000000000002e-06 2021-03-24T21:54:55.117 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.117 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.117 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:54:55.117 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359875+0000", 2021-03-24T21:54:55.118 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00094160000000000001 2021-03-24T21:54:55.118 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.118 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.118 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:54:55.119 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.360816+0000", 2021-03-24T21:54:55.119 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.3867000000000002e-05 2021-03-24T21:54:55.119 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.119 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.120 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:54:55.120 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.360900+0000", 2021-03-24T21:54:55.120 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00015279599999999999 2021-03-24T21:54:55.120 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.120 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.121 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,3", 2021-03-24T21:54:55.121 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.361053+0000", 2021-03-24T21:54:55.121 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00049272900000000002 2021-03-24T21:54:55.121 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.122 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.122 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:54:55.122 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.361546+0000", 2021-03-24T21:54:55.122 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00077420799999999997 2021-03-24T21:54:55.123 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.123 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.123 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:54:55.123 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.362320+0000", 2021-03-24T21:54:55.124 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0027734880000000002 2021-03-24T21:54:55.124 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.124 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.124 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:54:55.124 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.365093+0000", 2021-03-24T21:54:55.125 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.1638999999999997e-05 2021-03-24T21:54:55.125 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.125 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.125 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:54:55.125 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.365155+0000", 2021-03-24T21:54:55.126 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.4994999999999996e-05 2021-03-24T21:54:55.126 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.126 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.126 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:54:55.127 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.365230+0000", 2021-03-24T21:54:55.127 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0053737860000000002 2021-03-24T21:54:55.127 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:55.127 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:54:55.127 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:55.128 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.128 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.128 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:84 7.d 7:b2218182:::smithi12556470-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e107)", 2021-03-24T21:54:55.128 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.359915+0000", 2021-03-24T21:54:55.128 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 18.547279873000001, 2021-03-24T21:54:55.129 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0056067570000000004, 2021-03-24T21:54:55.129 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:54:55.129 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T21:54:55.129 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:54:55.130 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:54:55.130 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:54:55.130 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 84 2021-03-24T21:54:55.130 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.131 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:54:55.131 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.131 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:54:55.131 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359915+0000", 2021-03-24T21:54:55.131 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:54:55.132 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.132 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.132 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:54:55.132 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359915+0000", 2021-03-24T21:54:55.132 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T21:54:55.133 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.133 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.133 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:54:55.133 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359914+0000", 2021-03-24T21:54:55.134 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.8439999999999997e-06 2021-03-24T21:54:55.134 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.134 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.134 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:54:55.134 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359918+0000", 2021-03-24T21:54:55.135 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0258000000000001e-05 2021-03-24T21:54:55.135 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.135 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.135 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:54:55.136 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359929+0000", 2021-03-24T21:54:55.136 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.8179999999999998e-06 2021-03-24T21:54:55.136 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.136 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.136 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:54:55.137 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359935+0000", 2021-03-24T21:54:55.137 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0013444279999999999 2021-03-24T21:54:55.137 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.137 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.138 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:54:55.138 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.361280+0000", 2021-03-24T21:54:55.138 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.7156999999999993e-05 2021-03-24T21:54:55.138 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.139 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.139 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T21:54:55.139 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.361357+0000", 2021-03-24T21:54:55.139 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0039437489999999999 2021-03-24T21:54:55.139 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.140 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.140 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:54:55.140 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.365301+0000", 2021-03-24T21:54:55.140 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000100756 2021-03-24T21:54:55.140 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.141 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.141 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:54:55.141 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.365402+0000", 2021-03-24T21:54:55.141 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00011997099999999999 2021-03-24T21:54:55.142 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.142 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.142 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:54:55.142 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.365522+0000", 2021-03-24T21:54:55.143 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0056067570000000004 2021-03-24T21:54:55.143 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:55.143 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:54:55.143 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:55.144 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.144 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.144 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:121 7.b 7:d841eed8:::smithi12556470-31:head [write 504210~606529,stat] snapc 0=[] ondisk+write+known_if_redirected e107)", 2021-03-24T21:54:55.144 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.487723+0000", 2021-03-24T21:54:55.144 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 18.419472034999998, 2021-03-24T21:54:55.145 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0043501010000000003, 2021-03-24T21:54:55.145 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:54:55.145 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:54:55.145 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:54:55.145 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:54:55.146 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:54:55.146 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 121 2021-03-24T21:54:55.146 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.146 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:54:55.147 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.147 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:54:55.147 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.487723+0000", 2021-03-24T21:54:55.147 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:54:55.148 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.148 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.148 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:54:55.148 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.487723+0000", 2021-03-24T21:54:55.148 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999962 2021-03-24T21:54:55.149 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.149 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.149 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:54:55.149 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.487719+0000", 2021-03-24T21:54:55.150 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00036455999999999999 2021-03-24T21:54:55.150 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.150 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.150 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:54:55.151 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.488084+0000", 2021-03-24T21:54:55.151 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.7018000000000001e-05 2021-03-24T21:54:55.151 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.151 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.152 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:54:55.152 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.488111+0000", 2021-03-24T21:54:55.152 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1592e-05 2021-03-24T21:54:55.152 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.152 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.153 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:54:55.153 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.488122+0000", 2021-03-24T21:54:55.153 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.7318999999999996e-05 2021-03-24T21:54:55.154 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.154 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.154 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:54:55.154 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.488200+0000", 2021-03-24T21:54:55.154 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00019132599999999999 2021-03-24T21:54:55.155 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.155 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.155 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:54:55.155 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.488391+0000", 2021-03-24T21:54:55.156 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00022982699999999999 2021-03-24T21:54:55.156 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.156 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.156 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,5", 2021-03-24T21:54:55.156 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.488621+0000", 2021-03-24T21:54:55.157 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001590948 2021-03-24T21:54:55.157 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.157 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.157 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:54:55.157 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.490212+0000", 2021-03-24T21:54:55.158 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00096796600000000001 2021-03-24T21:54:55.158 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.158 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.158 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:54:55.159 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.491180+0000", 2021-03-24T21:54:55.159 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00076374199999999998 2021-03-24T21:54:55.159 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.159 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.159 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:54:55.160 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.491943+0000", 2021-03-24T21:54:55.160 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.4624999999999998e-05 2021-03-24T21:54:55.160 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.160 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.160 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:54:55.161 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.492018+0000", 2021-03-24T21:54:55.161 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.4759000000000002e-05 2021-03-24T21:54:55.161 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.161 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.162 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:54:55.162 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.492073+0000", 2021-03-24T21:54:55.162 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0043501010000000003 2021-03-24T21:54:55.162 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:55.162 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:54:55.163 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:55.163 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.163 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.163 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:124 7.b 7:d841eed8:::smithi12556470-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e107)", 2021-03-24T21:54:55.164 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.494851+0000", 2021-03-24T21:54:55.164 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 18.412343344, 2021-03-24T21:54:55.164 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0039240020000000002, 2021-03-24T21:54:55.164 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:54:55.164 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T21:54:55.165 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:54:55.165 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:54:55.165 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:54:55.165 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 124 2021-03-24T21:54:55.165 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.166 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:54:55.166 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.166 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:54:55.166 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.494851+0000", 2021-03-24T21:54:55.167 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:54:55.167 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.167 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.167 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:54:55.167 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.494851+0000", 2021-03-24T21:54:55.168 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T21:54:55.168 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.169 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.169 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:54:55.169 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.494850+0000", 2021-03-24T21:54:55.170 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.8899999999999998e-06 2021-03-24T21:54:55.170 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.170 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.170 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:54:55.171 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.494855+0000", 2021-03-24T21:54:55.171 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.5820000000000004e-06 2021-03-24T21:54:55.171 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.171 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.171 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:54:55.172 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.494863+0000", 2021-03-24T21:54:55.172 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.9119999999999997e-06 2021-03-24T21:54:55.172 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.172 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.173 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:54:55.173 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.494868+0000", 2021-03-24T21:54:55.173 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0023491929999999999 2021-03-24T21:54:55.173 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.174 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.174 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:54:55.174 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.497217+0000", 2021-03-24T21:54:55.174 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.4723999999999998e-05 2021-03-24T21:54:55.175 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.175 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.175 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T21:54:55.175 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.497292+0000", 2021-03-24T21:54:55.175 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001295278 2021-03-24T21:54:55.176 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.176 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.176 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:54:55.176 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.498587+0000", 2021-03-24T21:54:55.177 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.5016000000000001e-05 2021-03-24T21:54:55.177 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.177 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.177 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:54:55.177 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.498672+0000", 2021-03-24T21:54:55.178 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010340099999999999 2021-03-24T21:54:55.178 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.178 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.178 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:54:55.179 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.498775+0000", 2021-03-24T21:54:55.179 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0039240020000000002 2021-03-24T21:54:55.179 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:55.179 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:54:55.180 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:55.180 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.180 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.180 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:125 7.b 7:de4c3db5:::smithi12556470-32:head [write 521017~798131,stat] snapc 0=[] ondisk+write+known_if_redirected e107)", 2021-03-24T21:54:55.181 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.502054+0000", 2021-03-24T21:54:55.181 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 18.405140675999998, 2021-03-24T21:54:55.181 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0041060309999999996, 2021-03-24T21:54:55.181 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:54:55.182 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:54:55.182 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:54:55.182 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:54:55.182 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:54:55.183 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 125 2021-03-24T21:54:55.183 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.183 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:54:55.183 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.184 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:54:55.184 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.502054+0000", 2021-03-24T21:54:55.184 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:54:55.184 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.185 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.185 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:54:55.185 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.502054+0000", 2021-03-24T21:54:55.185 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999976 2021-03-24T21:54:55.186 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.186 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.186 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:54:55.186 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.502052+0000", 2021-03-24T21:54:55.186 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00030999899999999998 2021-03-24T21:54:55.187 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.187 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.187 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:54:55.187 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.502362+0000", 2021-03-24T21:54:55.187 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.1806000000000001e-05 2021-03-24T21:54:55.188 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.188 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.188 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:54:55.188 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.502383+0000", 2021-03-24T21:54:55.189 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.6349999999999995e-06 2021-03-24T21:54:55.189 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.189 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.189 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:54:55.189 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.502392+0000", 2021-03-24T21:54:55.190 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.0788000000000002e-05 2021-03-24T21:54:55.190 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.190 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.190 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:54:55.191 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.502443+0000", 2021-03-24T21:54:55.191 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00013060400000000001 2021-03-24T21:54:55.191 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.191 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.191 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:54:55.192 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.502573+0000", 2021-03-24T21:54:55.192 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00017257399999999999 2021-03-24T21:54:55.192 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.192 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.192 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,5", 2021-03-24T21:54:55.193 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.502746+0000", 2021-03-24T21:54:55.193 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00132138 2021-03-24T21:54:55.193 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.193 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.194 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:54:55.194 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.504067+0000", 2021-03-24T21:54:55.194 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0011008719999999999 2021-03-24T21:54:55.194 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.195 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.195 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:54:55.195 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.505168+0000", 2021-03-24T21:54:55.195 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00081040500000000004 2021-03-24T21:54:55.196 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.196 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.196 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:54:55.196 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.505979+0000", 2021-03-24T21:54:55.196 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.2396000000000001e-05 2021-03-24T21:54:55.197 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.197 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.197 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:54:55.197 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.506071+0000", 2021-03-24T21:54:55.197 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.8867000000000001e-05 2021-03-24T21:54:55.198 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.198 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.198 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:54:55.198 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.506160+0000", 2021-03-24T21:54:55.199 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0041060309999999996 2021-03-24T21:54:55.199 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:55.199 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:54:55.199 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:55.199 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.200 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.200 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:137 7.0 e107/99 7:0ece3580:::smithi12556470-35:head v 107'1, mlcod=0'0)", 2021-03-24T21:54:55.200 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.545040+0000", 2021-03-24T21:54:55.200 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 18.362155002000002, 2021-03-24T21:54:55.201 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0054328359999999999, 2021-03-24T21:54:55.201 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:54:55.201 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:54:55.201 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:54:55.201 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.202 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:54:55.202 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.545040+0000", 2021-03-24T21:54:55.202 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:54:55.202 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.203 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.203 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:54:55.203 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.545040+0000", 2021-03-24T21:54:55.203 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999957 2021-03-24T21:54:55.203 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.204 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.204 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:54:55.204 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.545035+0000", 2021-03-24T21:54:55.204 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.004385702 2021-03-24T21:54:55.204 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.205 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.205 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:54:55.205 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.549421+0000", 2021-03-24T21:54:55.205 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.6047999999999999e-05 2021-03-24T21:54:55.206 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.206 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.206 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:54:55.206 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.549447+0000", 2021-03-24T21:54:55.207 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0923e-05 2021-03-24T21:54:55.207 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.207 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.207 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:54:55.207 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.549458+0000", 2021-03-24T21:54:55.208 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.0572000000000002e-05 2021-03-24T21:54:55.208 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.208 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.208 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:54:55.208 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.549519+0000", 2021-03-24T21:54:55.209 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.2843999999999999e-05 2021-03-24T21:54:55.209 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.209 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.209 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:54:55.210 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.549552+0000", 2021-03-24T21:54:55.210 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00088233300000000001 2021-03-24T21:54:55.210 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.210 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.210 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:54:55.211 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.550434+0000", 2021-03-24T21:54:55.211 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.8571000000000001e-05 2021-03-24T21:54:55.211 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.212 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.212 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:54:55.212 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.550472+0000", 2021-03-24T21:54:55.212 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0054328359999999999 2021-03-24T21:54:55.212 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:55.213 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:54:55.213 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:55.213 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.213 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.214 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:160 7.b 7:da3d170a:::smithi12556470-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e107)", 2021-03-24T21:54:55.214 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.625320+0000", 2021-03-24T21:54:55.214 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 18.281874146, 2021-03-24T21:54:55.214 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0037863419999999998, 2021-03-24T21:54:55.215 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:54:55.215 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T21:54:55.215 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:54:55.215 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:54:55.215 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:54:55.216 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 160 2021-03-24T21:54:55.216 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.216 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:54:55.216 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.217 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:54:55.217 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.625320+0000", 2021-03-24T21:54:55.217 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:54:55.217 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.217 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.218 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:54:55.218 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.625320+0000", 2021-03-24T21:54:55.218 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T21:54:55.218 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.219 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.219 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:54:55.219 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.625319+0000", 2021-03-24T21:54:55.219 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5809999999999996e-06 2021-03-24T21:54:55.220 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.220 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.220 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:54:55.220 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.625325+0000", 2021-03-24T21:54:55.220 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.4336e-05 2021-03-24T21:54:55.221 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.221 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.221 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:54:55.221 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.625339+0000", 2021-03-24T21:54:55.222 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.3259999999999996e-06 2021-03-24T21:54:55.222 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.222 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.222 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:54:55.222 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.625348+0000", 2021-03-24T21:54:55.223 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0021449469999999999 2021-03-24T21:54:55.223 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.223 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.223 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:54:55.224 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.627492+0000", 2021-03-24T21:54:55.224 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010118100000000001 2021-03-24T21:54:55.224 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.224 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.225 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T21:54:55.225 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.627594+0000", 2021-03-24T21:54:55.225 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001193237 2021-03-24T21:54:55.225 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.225 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.226 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:54:55.226 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.628787+0000", 2021-03-24T21:54:55.226 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000121074 2021-03-24T21:54:55.226 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.227 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.227 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:54:55.227 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.628908+0000", 2021-03-24T21:54:55.227 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000198884 2021-03-24T21:54:55.227 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.228 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.228 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:54:55.228 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.629107+0000", 2021-03-24T21:54:55.228 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0037863419999999998 2021-03-24T21:54:55.228 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:55.229 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:54:55.229 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:55.229 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.229 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.230 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:178 7.d 7:ba8515f1:::smithi12556470-45:head [write 1875090~648012] snapc 0=[] ondisk+write+known_if_redirected e107)", 2021-03-24T21:54:55.230 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.706281+0000", 2021-03-24T21:54:55.230 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 18.200913717999999, 2021-03-24T21:54:55.230 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0046999620000000002, 2021-03-24T21:54:55.231 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:54:55.231 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:54:55.231 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:54:55.231 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:54:55.231 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:54:55.232 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 178 2021-03-24T21:54:55.232 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.232 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:54:55.232 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.233 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:54:55.233 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706281+0000", 2021-03-24T21:54:55.233 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:54:55.234 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.234 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.234 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:54:55.234 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706281+0000", 2021-03-24T21:54:55.234 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999971 2021-03-24T21:54:55.235 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.235 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.235 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:54:55.235 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706278+0000", 2021-03-24T21:54:55.236 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00022174299999999999 2021-03-24T21:54:55.236 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.236 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.236 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:54:55.237 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706500+0000", 2021-03-24T21:54:55.237 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.3224e-05 2021-03-24T21:54:55.237 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.237 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.237 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:54:55.238 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706523+0000", 2021-03-24T21:54:55.238 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7773000000000001e-05 2021-03-24T21:54:55.238 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.238 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.239 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:54:55.239 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706541+0000", 2021-03-24T21:54:55.239 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000101025 2021-03-24T21:54:55.239 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.239 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.240 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:54:55.240 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706642+0000", 2021-03-24T21:54:55.240 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00019631299999999999 2021-03-24T21:54:55.240 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.240 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.241 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:54:55.241 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706838+0000", 2021-03-24T21:54:55.241 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000349081 2021-03-24T21:54:55.241 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.242 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.242 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,3", 2021-03-24T21:54:55.242 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.707187+0000", 2021-03-24T21:54:55.242 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002206515 2021-03-24T21:54:55.242 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.243 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.243 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:54:55.243 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.709394+0000", 2021-03-24T21:54:55.243 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00031391600000000002 2021-03-24T21:54:55.244 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.244 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.244 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:54:55.244 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.709708+0000", 2021-03-24T21:54:55.244 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0011238960000000001 2021-03-24T21:54:55.245 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.245 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.245 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:54:55.245 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.710832+0000", 2021-03-24T21:54:55.246 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.9524000000000006e-05 2021-03-24T21:54:55.246 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.246 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.246 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:54:55.246 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.710911+0000", 2021-03-24T21:54:55.247 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.9733999999999993e-05 2021-03-24T21:54:55.247 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.247 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.247 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:54:55.248 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.710981+0000", 2021-03-24T21:54:55.248 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0046999620000000002 2021-03-24T21:54:55.248 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:55.248 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:54:55.248 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:55.249 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.249 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.249 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:179 7.d 7:ba8515f1:::smithi12556470-45:head [setxattr _header (56) in=7b,truncate 2739508] snapc 0=[] ondisk+write+known_if_redirected e107)", 2021-03-24T21:54:55.249 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.706566+0000", 2021-03-24T21:54:55.249 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 18.200628952999999, 2021-03-24T21:54:55.250 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.004679417, 2021-03-24T21:54:55.250 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:54:55.250 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:54:55.250 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:54:55.251 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:54:55.251 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:54:55.251 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 179 2021-03-24T21:54:55.251 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.252 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:54:55.252 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.252 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:54:55.252 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706566+0000", 2021-03-24T21:54:55.253 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:54:55.253 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.253 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.253 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:54:55.254 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706566+0000", 2021-03-24T21:54:55.254 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T21:54:55.254 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.254 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.255 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:54:55.255 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706564+0000", 2021-03-24T21:54:55.255 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.3519999999999996e-06 2021-03-24T21:54:55.255 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.255 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.256 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:54:55.256 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706573+0000", 2021-03-24T21:54:55.256 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.4070000000000001e-05 2021-03-24T21:54:55.256 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.257 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.257 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:54:55.257 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706587+0000", 2021-03-24T21:54:55.257 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.7489999999999993e-06 2021-03-24T21:54:55.257 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.258 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.258 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:54:55.258 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706597+0000", 2021-03-24T21:54:55.258 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001623022 2021-03-24T21:54:55.259 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.259 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.259 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:54:55.259 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.708220+0000", 2021-03-24T21:54:55.259 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00013183199999999999 2021-03-24T21:54:55.260 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.260 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.260 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:54:55.260 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.708352+0000", 2021-03-24T21:54:55.261 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00025474500000000001 2021-03-24T21:54:55.261 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.261 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.261 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,3", 2021-03-24T21:54:55.261 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.708606+0000", 2021-03-24T21:54:55.262 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00080621600000000001 2021-03-24T21:54:55.262 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.262 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.262 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:54:55.262 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.709413+0000", 2021-03-24T21:54:55.263 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00047969100000000003 2021-03-24T21:54:55.263 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.263 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.263 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:54:55.264 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.709892+0000", 2021-03-24T21:54:55.264 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001199615 2021-03-24T21:54:55.264 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.264 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.264 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:54:55.265 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.711092+0000", 2021-03-24T21:54:55.265 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0001032 2021-03-24T21:54:55.265 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.266 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.266 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:54:55.266 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.711195+0000", 2021-03-24T21:54:55.266 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.9874999999999999e-05 2021-03-24T21:54:55.267 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.267 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.267 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:54:55.267 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.711245+0000", 2021-03-24T21:54:55.267 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.004679417 2021-03-24T21:54:55.268 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:55.268 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:54:55.268 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:55.268 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.268 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.269 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:180 7.d 7:ba8515f1:::smithi12556470-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e107)", 2021-03-24T21:54:55.269 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.706637+0000", 2021-03-24T21:54:55.269 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 18.200557655000001, 2021-03-24T21:54:55.269 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0049820649999999999, 2021-03-24T21:54:55.270 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:54:55.270 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T21:54:55.270 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:54:55.270 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:54:55.270 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:54:55.271 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 180 2021-03-24T21:54:55.271 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.271 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:54:55.271 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.271 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:54:55.272 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706637+0000", 2021-03-24T21:54:55.272 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:54:55.272 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.272 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.272 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:54:55.273 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706637+0000", 2021-03-24T21:54:55.273 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T21:54:55.273 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.273 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.274 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:54:55.274 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706636+0000", 2021-03-24T21:54:55.274 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5099999999999998e-06 2021-03-24T21:54:55.274 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.275 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.275 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:54:55.275 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706641+0000", 2021-03-24T21:54:55.275 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.6740000000000003e-06 2021-03-24T21:54:55.275 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.276 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.276 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:54:55.276 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706650+0000", 2021-03-24T21:54:55.276 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.7699999999999998e-06 2021-03-24T21:54:55.276 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.277 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.277 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:54:55.277 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706656+0000", 2021-03-24T21:54:55.277 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0023962749999999998 2021-03-24T21:54:55.278 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.278 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.278 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:54:55.278 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.709052+0000", 2021-03-24T21:54:55.278 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000193926 2021-03-24T21:54:55.279 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.279 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.279 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T21:54:55.279 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.709246+0000", 2021-03-24T21:54:55.280 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002080279 2021-03-24T21:54:55.280 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.280 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.280 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:54:55.281 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.711326+0000", 2021-03-24T21:54:55.281 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00013017399999999999 2021-03-24T21:54:55.281 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.281 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.281 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:54:55.282 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.711456+0000", 2021-03-24T21:54:55.282 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000162767 2021-03-24T21:54:55.282 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:54:55.282 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:54:55.283 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:54:55.283 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.711619+0000", 2021-03-24T21:54:55.283 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0049820649999999999 2021-03-24T21:54:55.283 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:55.284 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:54:55.284 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:55.284 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:54:55.284 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:54:55.284 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-24T21:54:55.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:55.286 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:55.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:54:55.288 INFO:tasks.rados.rados.0.smithi125.stdout:153: done (1 left) 2021-03-24T21:54:55.289 INFO:tasks.rados.rados.0.smithi125.stdout:154: write oid 44 current snap is 10 2021-03-24T21:54:55.289 INFO:tasks.rados.rados.0.smithi125.stdout:154: seq_num 85 ranges {628595=476048,1807440=632344,2934719=452474} 2021-03-24T21:54:55.289 INFO:tasks.rados.rados.0.smithi125.stdout:154: writing smithi12556470-44 from 628595 to 1104643 tid 1 2021-03-24T21:54:55.289 INFO:tasks.rados.rados.0.smithi125.stdout:154: writing smithi12556470-44 from 1807440 to 2439784 tid 2 2021-03-24T21:54:55.290 INFO:tasks.rados.rados.0.smithi125.stdout:154: writing smithi12556470-44 from 2934719 to 3387193 tid 3 2021-03-24T21:54:55.290 INFO:tasks.rados.rados.0.smithi125.stdout:155: read oid 34 snap -1 2021-03-24T21:54:55.290 INFO:tasks.rados.rados.0.smithi125.stdout:155: expect deleted 2021-03-24T21:54:55.290 INFO:tasks.rados.rados.0.smithi125.stdout:156: snap_create 2021-03-24T21:54:55.291 INFO:tasks.rados.rados.0.smithi125.stdout:154: finishing write tid 1 to smithi12556470-44 2021-03-24T21:54:55.291 INFO:tasks.rados.rados.0.smithi125.stdout:154: finishing write tid 2 to smithi12556470-44 2021-03-24T21:54:55.291 INFO:tasks.rados.rados.0.smithi125.stdout:154: finishing write tid 3 to smithi12556470-44 2021-03-24T21:54:55.291 INFO:tasks.rados.rados.0.smithi125.stdout:154: finishing write tid 4 to smithi12556470-44 2021-03-24T21:54:55.291 INFO:tasks.rados.rados.0.smithi125.stdout:154: finishing write tid 5 to smithi12556470-44 2021-03-24T21:54:55.292 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 44 v 56 (ObjNum 85 snap 10 seq_num 85) dirty exists 2021-03-24T21:54:55.292 INFO:tasks.rados.rados.0.smithi125.stdout:154: left oid 44 (ObjNum 85 snap 10 seq_num 85) 2021-03-24T21:54:55.388 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:55.400 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:55 smithi142 conmon[20125]: cluster 2021-03-24T21:54 2021-03-24T21:54:55.401 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:55 smithi142 conmon[20125]: :53.267320+0000 osd.4 (osd.4) 38 : cluster [DBG] 3.9 scrub ok 2021-03-24T21:54:55.401 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:55 smithi142 conmon[20125]: cluster 2021-03-24T21:54:55.401 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:55 smithi142 conmon[20125]: 2021-03-24T21:54:53.873066 2021-03-24T21:54:55.401 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:55 smithi142 conmon[20125]: +0000 osd.5 (osd.5) 37 2021-03-24T21:54:55.401 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:55 smithi142 conmon[20125]: : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:54:55.402 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:55 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:54:55.402 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:55 smithi142 conmon[20125]: -03-24T21:54:53. 2021-03-24T21:54:55.402 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:55 smithi142 conmon[20125]: 873221+0000 osd.5 (osd.5) 2021-03-24T21:54:55.402 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:55 smithi142 conmon[20125]: 38 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:54:55.403 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:55 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:54:55.403 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:55 smithi142 conmon[20125]: -03-24T21:54:54 2021-03-24T21:54:55.403 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:55 smithi142 conmon[20125]: .139650+0000 osd.2 (osd.2 2021-03-24T21:54:55.403 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:55 smithi142 conmon[20125]: ) 26 : cluster [DBG] 5.a scrub ok 2021-03-24T21:54:55.404 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:55 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:54:55.404 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:55 smithi142 conmon[20125]: -03-24T21:54:54 2021-03-24T21:54:55.404 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:55 smithi142 conmon[20125]: .236571+0000 mon.a (mon.0) 2021-03-24T21:54:55.404 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:55 smithi142 conmon[20125]: 676 : cluster [DBG] osdmap e125: 8 total, 8 up, 6 in 2021-03-24T21:54:55.404 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:55 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:54:55.405 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:55 smithi142 conmon[20125]: -03-24T21:54:54. 2021-03-24T21:54:55.405 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:55 smithi142 conmon[20125]: 541173+0000 osd.3 (osd.3) 2021-03-24T21:54:55.405 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:55 smithi142 conmon[20125]: 42 : cluster [DBG] 4.19 scrub ok 2021-03-24T21:54:55.405 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:55 smithi142 conmon[20125]: cluster 2021-03-24T21:54:55.079577+0000 mon.a (mon.0) 677 : cluster 2021-03-24T21:54:55.405 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:55 smithi142 conmon[20125]: [DBG] osdmap e126: 8 total, 8 up, 6 in 2021-03-24T21:54:55.439 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:55.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:55.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:54:55.448 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:55 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:54:55.448 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:55 smithi125 conmon[19936]: :54:53.267320+0000 osd.4 (osd.4) 38 : cluster [DBG] 3.9 scrub ok 2021-03-24T21:54:55.545 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:55.602 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:55.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:55.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:54:55.709 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:55.724 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:55 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:54:55.724 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:55 smithi125 conmon[24927]: 54:53.267320+0000 osd.4 (osd.4) 38 : cluster [DBG] 3.9 scrub ok 2021-03-24T21:54:55.725 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:55 smithi125 conmon[24927]: cluster 2021-03-24T21:54:53.873066+0000 osd.5 ( 2021-03-24T21:54:55.725 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:55 smithi125 conmon[24927]: osd.5) 37 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:54:55.725 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:55 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:54:55.726 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:55 smithi125 conmon[24927]: 21:54:53.873221+0000 osd.5 2021-03-24T21:54:55.726 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:55 smithi125 conmon[24927]: (osd.5) 38 : cluster [DBG] 2021-03-24T21:54:55.726 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:55 smithi125 conmon[24927]: purged_snaps scrub ok 2021-03-24T21:54:55.726 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:55 smithi125 conmon[24927]: cluster 2021-03-24T21:54:55.727 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:55 smithi125 conmon[24927]: 2021-03-24T21:54:54.139650+0000 osd.2 2021-03-24T21:54:55.727 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:55 smithi125 conmon[24927]: (osd.2) 26 : cluster [DBG] 5.a scrub ok 2021-03-24T21:54:55.727 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:55 smithi125 conmon[24927]: 2021-03-24T21:54:55.727 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:55 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:54:55.728 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:55 smithi125 conmon[24927]: -24T21:54:54.236571+0000 mon.a (mon. 2021-03-24T21:54:55.728 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:55 smithi125 conmon[24927]: 0) 676 : cluster [DBG] osdmap e125: 8 total, 8 up, 6 in 2021-03-24T21:54:55.728 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:55 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:54:55.728 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:55 smithi125 conmon[24927]: :54:54.541173+0000 osd.3 (osd. 2021-03-24T21:54:55.729 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:55 smithi125 conmon[24927]: 3) 42 : cluster [DBG] 4.19 scrub ok 2021-03-24T21:54:55.729 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:55 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:54:55.729 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:55 smithi125 conmon[24927]: :54:55.079577+0000 mon.a (mon. 2021-03-24T21:54:55.730 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:55 smithi125 conmon[24927]: 0) 677 : cluster [DBG] osdmap e126: 8 total, 8 up, 6 in 2021-03-24T21:54:55.730 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:55 smithi125 conmon[19936]: cluster 2021-03-24T21:54:53.873066+0000 osd.5 (osd.5) 37 : cluster [DBG] 2021-03-24T21:54:55.731 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:55 smithi125 conmon[19936]: purged_snaps scrub starts 2021-03-24T21:54:55.731 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:55 smithi125 conmon[19936]: cluster 2021-03-24T21:54:53.873221+0000 osd.5 (osd.5) 38 : 2021-03-24T21:54:55.731 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:55 smithi125 conmon[19936]: cluster [DBG] purged_snaps scrub ok 2021-03-24T21:54:55.732 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:55 smithi125 conmon[19936]: cluster 2021-03-24T21:54:54.139650 2021-03-24T21:54:55.732 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:55 smithi125 conmon[19936]: +0000 osd.2 (osd.2) 26 : cluster [DBG] 5.a scrub ok 2021-03-24T21:54:55.732 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:55 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:54:55.732 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:55 smithi125 conmon[19936]: 03-24T21:54:54.236571+0000 mon.a (mon.0) 676 : cluster [DBG] osdmap e125: 8 total, 8 up, 6 in 2021-03-24T21:54:55.732 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:55 smithi125 conmon[19936]: cluster 2021-03-24T21:54: 2021-03-24T21:54:55.733 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:55 smithi125 conmon[19936]: 54.541173+0000 osd.3 (osd.3) 42 : 2021-03-24T21:54:55.733 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:55 smithi125 conmon[19936]: cluster [DBG] 4.19 scrub ok 2021-03-24T21:54:55.733 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:55 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:54:55.733 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:55 smithi125 conmon[19936]: 54:55.079577+0000 mon.a ( 2021-03-24T21:54:55.734 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:55 smithi125 conmon[19936]: mon.0) 677 : cluster [DBG] osdmap e126: 8 total, 8 up, 6 in 2021-03-24T21:54:55.766 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:55.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:55.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:54:55.873 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:55.890 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.2 dump_ops_in_flight 2021-03-24T21:54:55.932 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:55.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:55.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:54:56.039 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:56.095 INFO:tasks.rados.rados.0.smithi125.stdout:152: done (3 left) 2021-03-24T21:54:56.096 INFO:tasks.rados.rados.0.smithi125.stdout:154: done (2 left) 2021-03-24T21:54:56.096 INFO:tasks.rados.rados.0.smithi125.stdout:155: done (1 left) 2021-03-24T21:54:56.096 INFO:tasks.rados.rados.0.smithi125.stdout:156: done (0 left) 2021-03-24T21:54:56.096 INFO:tasks.rados.rados.0.smithi125.stdout:157: rollback oid 38 current snap is 11 2021-03-24T21:54:56.097 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 38 to 10 2021-03-24T21:54:56.097 INFO:tasks.rados.rados.0.smithi125.stdout:158: write oid 33 current snap is 11 2021-03-24T21:54:56.097 INFO:tasks.rados.rados.0.smithi125.stdout:158: seq_num 86 ranges {645402=551297,1749569=492355,2845789=599613} 2021-03-24T21:54:56.098 INFO:tasks.rados.rados.0.smithi125.stdout:158: writing smithi12556470-33 from 645402 to 1196699 tid 1 2021-03-24T21:54:56.102 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:56.105 INFO:tasks.rados.rados.0.smithi125.stdout:158: writing smithi12556470-33 from 1749569 to 2241924 tid 2 2021-03-24T21:54:56.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:56.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:54:56.112 INFO:tasks.rados.rados.0.smithi125.stdout:158: writing smithi12556470-33 from 2845789 to 3445402 tid 3 2021-03-24T21:54:56.112 INFO:tasks.rados.rados.0.smithi125.stdout:159: read oid 34 snap 10 2021-03-24T21:54:56.113 INFO:tasks.rados.rados.0.smithi125.stdout:159: expect deleted 2021-03-24T21:54:56.113 INFO:tasks.rados.rados.0.smithi125.stdout:157: finishing rollback tid 0 to smithi12556470-38 2021-03-24T21:54:56.113 INFO:tasks.rados.rados.0.smithi125.stdout:157: finishing rollback tid 1 to smithi12556470-38 2021-03-24T21:54:56.114 INFO:tasks.rados.rados.0.smithi125.stdout:157: finishing rollback tid 2 to smithi12556470-38 2021-03-24T21:54:56.114 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 38 v 7 (ObjNum 37 snap 0 seq_num 37) dirty exists 2021-03-24T21:54:56.114 INFO:tasks.rados.rados.0.smithi125.stdout:158: finishing write tid 1 to smithi12556470-33 2021-03-24T21:54:56.114 INFO:tasks.rados.rados.0.smithi125.stdout:158: finishing write tid 2 to smithi12556470-33 2021-03-24T21:54:56.115 INFO:tasks.rados.rados.0.smithi125.stdout:157: done (2 left) 2021-03-24T21:54:56.115 INFO:tasks.rados.rados.0.smithi125.stdout:160: write (excl) oid 16 current snap is 11 2021-03-24T21:54:56.115 INFO:tasks.rados.rados.0.smithi125.stdout:160: seq_num 87 ranges {662209=626546,1808051=636013,2873212=588997} 2021-03-24T21:54:56.120 INFO:tasks.rados.rados.0.smithi125.stdout:160: writing smithi12556470-16 from 662209 to 1288755 tid 1 2021-03-24T21:54:56.127 INFO:tasks.rados.rados.0.smithi125.stdout:160: writing smithi12556470-16 from 1808051 to 2444064 tid 2 2021-03-24T21:54:56.133 INFO:tasks.rados.rados.0.smithi125.stdout:160: writing smithi12556470-16 from 2873212 to 3462209 tid 3 2021-03-24T21:54:56.134 INFO:tasks.rados.rados.0.smithi125.stdout:161: read oid 31 snap -1 2021-03-24T21:54:56.134 INFO:tasks.rados.rados.0.smithi125.stdout:161: expect (ObjNum 73 snap 7 seq_num 73) 2021-03-24T21:54:56.134 INFO:tasks.rados.rados.0.smithi125.stdout:162: delete oid 49 current snap is 11 2021-03-24T21:54:56.134 INFO:tasks.rados.rados.0.smithi125.stdout:158: finishing write tid 3 to smithi12556470-33 2021-03-24T21:54:56.135 INFO:tasks.rados.rados.0.smithi125.stdout:158: finishing write tid 4 to smithi12556470-33 2021-03-24T21:54:56.135 INFO:tasks.rados.rados.0.smithi125.stdout:158: finishing write tid 5 to smithi12556470-33 2021-03-24T21:54:56.135 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 33 v 44 (ObjNum 86 snap 11 seq_num 86) dirty exists 2021-03-24T21:54:56.135 INFO:tasks.rados.rados.0.smithi125.stdout:158: left oid 33 (ObjNum 86 snap 11 seq_num 86) 2021-03-24T21:54:56.135 INFO:tasks.rados.rados.0.smithi125.stdout:160: finishing write tid 1 to smithi12556470-16 2021-03-24T21:54:56.136 INFO:tasks.rados.rados.0.smithi125.stdout:160: finishing write tid 2 to smithi12556470-16 2021-03-24T21:54:56.137 INFO:tasks.rados.rados.0.smithi125.stdout:160: finishing write tid 3 to smithi12556470-16 2021-03-24T21:54:56.138 INFO:tasks.rados.rados.0.smithi125.stdout:160: finishing write tid 4 to smithi12556470-16 2021-03-24T21:54:56.138 INFO:tasks.rados.rados.0.smithi125.stdout:160: finishing write tid 5 to smithi12556470-16 2021-03-24T21:54:56.138 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 16 v 49 (ObjNum 87 snap 11 seq_num 87) dirty exists 2021-03-24T21:54:56.139 INFO:tasks.rados.rados.0.smithi125.stdout:160: left oid 16 (ObjNum 87 snap 11 seq_num 87) 2021-03-24T21:54:56.140 INFO:tasks.rados.rados.0.smithi125.stdout:158: done (4 left) 2021-03-24T21:54:56.140 INFO:tasks.rados.rados.0.smithi125.stdout:159: done (3 left) 2021-03-24T21:54:56.140 INFO:tasks.rados.rados.0.smithi125.stdout:160: done (2 left) 2021-03-24T21:54:56.140 INFO:tasks.rados.rados.0.smithi125.stdout:162: done (1 left) 2021-03-24T21:54:56.140 INFO:tasks.rados.rados.0.smithi125.stdout:163: write (excl) oid 24 current snap is 11 2021-03-24T21:54:56.141 INFO:tasks.rados.rados.0.smithi125.stdout:163: seq_num 89 ranges {695823=777044,1925015=639682,3328058=167765} 2021-03-24T21:54:56.147 INFO:tasks.rados.rados.0.smithi125.stdout:163: writing smithi12556470-24 from 695823 to 1472867 tid 1 2021-03-24T21:54:56.155 INFO:tasks.rados.rados.0.smithi125.stdout:163: writing smithi12556470-24 from 1925015 to 2564697 tid 2 2021-03-24T21:54:56.161 INFO:tasks.rados.rados.0.smithi125.stdout:163: writing smithi12556470-24 from 3328058 to 3495823 tid 3 2021-03-24T21:54:56.161 INFO:tasks.rados.rados.0.smithi125.stdout:164: snap_create 2021-03-24T21:54:56.162 INFO:tasks.rados.rados.0.smithi125.stdout:161: expect (ObjNum 73 snap 7 seq_num 73) 2021-03-24T21:54:56.211 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:56.217 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:54:56.263 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:56.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:56.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:54:56.300 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-24T21:54:56.358 INFO:tasks.rados.rados.0.smithi125.stdout:163: finishing write tid 1 to smithi12556470-24 2021-03-24T21:54:56.423 INFO:tasks.rados.rados.0.smithi125.stdout:163: finishing write tid 2 to smithi12556470-24 2021-03-24T21:54:56.423 INFO:tasks.rados.rados.0.smithi125.stdout:163: finishing write tid 3 to smithi12556470-24 2021-03-24T21:54:56.423 INFO:tasks.rados.rados.0.smithi125.stdout:163: finishing write tid 4 to smithi12556470-24 2021-03-24T21:54:56.424 INFO:tasks.rados.rados.0.smithi125.stdout:163: finishing write tid 5 to smithi12556470-24 2021-03-24T21:54:56.424 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 24 v 54 (ObjNum 89 snap 11 seq_num 89) dirty exists 2021-03-24T21:54:56.424 INFO:tasks.rados.rados.0.smithi125.stdout:163: left oid 24 (ObjNum 89 snap 11 seq_num 89) 2021-03-24T21:54:56.425 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:56.472 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:56 smithi125 conmon[19936]: cluster 2021-03-24T21:54:54.371093+0000 2021-03-24T21:54:56.472 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:56 smithi125 conmon[19936]: osd.0 (osd.0) 41 : cluster [DBG] 6.10 scrub ok 2021-03-24T21:54:56.472 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:56 smithi125 conmon[19936]: cluster 2021-03-24T21:54:54.715932+0000 mgr.y (mgr 2021-03-24T21:54:56.472 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:56 smithi125 conmon[19936]: .14144) 320 : cluster [DBG] pgmap v350: 148 pgs: 148 active+clean; 129 MiB data, 325 MiB used, 536 GiB / 536 GiB avail; 9.8 MiB/s rd, 4.8 MiB/s wr, 32 op/s 2021-03-24T21:54:56.473 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:56 smithi125 conmon[19936]: cluster 2021-03-24T21:54:55.067955+0000 2021-03-24T21:54:56.473 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:56 smithi125 conmon[19936]: osd.5 (osd.5) 39 : cluster [DBG] 2021-03-24T21:54:56.473 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:56 smithi125 conmon[19936]: 6.a scrub ok 2021-03-24T21:54:56.474 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:56 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:54:56.474 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:56 smithi125 conmon[19936]: 03-24T21:54:55.493703+0000 osd.3 2021-03-24T21:54:56.474 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:56 smithi125 conmon[19936]: (osd.3) 43 : cluster [DBG] 5.17 deep-scrub ok 2021-03-24T21:54:56.474 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:56 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:54:56.475 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:56 smithi125 conmon[19936]: 03-24T21:54:56.088064+0000 mon.a ( 2021-03-24T21:54:56.475 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:56 smithi125 conmon[19936]: mon.0) 678 : cluster [DBG] osdmap e127: 8 total, 8 up, 6 in 2021-03-24T21:54:56.475 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:56 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:54:56.475 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:56 smithi125 conmon[24927]: :54:54.371093+0000 osd.0 (osd.0) 41 : cluster [DBG] 6.10 scrub ok 2021-03-24T21:54:56.476 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:56 smithi125 conmon[24927]: 2021-03-24T21:54:56.476 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:56 smithi125 conmon[24927]: cluster 2021-03-24T21:54:56.476 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:56 smithi125 conmon[24927]: 2021-03-24T 2021-03-24T21:54:56.477 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:56 smithi125 conmon[24927]: 21:54:54. 2021-03-24T21:54:56.477 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:56 smithi125 conmon[24927]: 715932+0000 mgr.y ( 2021-03-24T21:54:56.477 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:56 smithi125 conmon[24927]: mgr.14144) 320 : cluster 2021-03-24T21:54:56.477 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:56 smithi125 conmon[24927]: [DBG] pgmap v350: 148 pgs: 148 active+clean; 129 MiB data, 325 MiB used, 536 GiB / 536 GiB avail; 9.8 MiB/s rd, 4.8 MiB/s wr, 32 op/s 2021-03-24T21:54:56.478 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:56 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:54:56.478 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:56 smithi125 conmon[24927]: 03-24T21:54:55 2021-03-24T21:54:56.478 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:56 smithi125 conmon[24927]: .067955+0000 osd.5 2021-03-24T21:54:56.478 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:56 smithi125 conmon[24927]: (osd.5) 39 2021-03-24T21:54:56.478 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:56 smithi125 conmon[24927]: : cluster [DBG] 6.a scrub ok 2021-03-24T21:54:56.479 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:56 smithi125 conmon[24927]: 2021-03-24T21:54:56.479 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:56 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:54:56.479 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:56 smithi125 conmon[24927]: 24T21:54: 2021-03-24T21:54:56.479 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:56 smithi125 conmon[24927]: 55.493703+0000 2021-03-24T21:54:56.480 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:56 smithi125 conmon[24927]: osd.3 (osd.3 2021-03-24T21:54:56.480 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:56 smithi125 conmon[24927]: ) 43 : cluster [DBG] 5.17 deep-scrub ok 2021-03-24T21:54:56.480 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:56 smithi125 conmon[24927]: 2021-03-24T21:54:56.480 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:56 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:54:56.480 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:56 smithi125 conmon[24927]: 24T21:54 2021-03-24T21:54:56.481 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:56 smithi125 conmon[24927]: :56.088064+0000 2021-03-24T21:54:56.481 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:56 smithi125 conmon[24927]: mon.a (mon.0 2021-03-24T21:54:56.481 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:56 smithi125 conmon[24927]: ) 678 : cluster [DBG] osdmap e127: 8 total, 8 up, 6 in 2021-03-24T21:54:56.481 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:56 smithi125 conmon[24927]: 2021-03-24T21:54:56.482 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:56.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:56.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:54:56.507 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd set noscrub 2021-03-24T21:54:56.508 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:56 smithi142 conmon[20125]: cluster 2021-03-24T21:54:54.371093+0000 osd.0 (osd.0) 41 : cluster 2021-03-24T21:54:56.508 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:56 smithi142 conmon[20125]: [DBG] 6.10 scrub ok 2021-03-24T21:54:56.509 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:56 smithi142 conmon[20125]: cluster 2021-03-24T21:54:54.715932+0000 mgr.y (mgr.14144) 2021-03-24T21:54:56.509 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:56 smithi142 conmon[20125]: 320 : cluster [DBG] pgmap v350: 148 pgs: 148 active+clean; 129 MiB data, 325 MiB used, 536 GiB / 536 GiB avail; 9.8 MiB/s rd, 4.8 MiB/s wr, 32 op/s 2021-03-24T21:54:56.509 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:56 smithi142 conmon[20125]: cluster 2021-03-24T21:54:55 2021-03-24T21:54:56.509 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:56 smithi142 conmon[20125]: .067955+0000 osd.5 (osd.5) 39 2021-03-24T21:54:56.510 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:56 smithi142 conmon[20125]: : cluster [DBG] 6.a scrub ok 2021-03-24T21:54:56.510 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:56 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:54:56.510 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:56 smithi142 conmon[20125]: 24T21:54:55.493703+0000 osd.3 ( 2021-03-24T21:54:56.510 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:56 smithi142 conmon[20125]: osd.3) 43 : cluster [DBG] 5.17 deep-scrub ok 2021-03-24T21:54:56.511 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:56 smithi142 conmon[20125]: cluster 2021-03-24T21 2021-03-24T21:54:56.511 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:56 smithi142 conmon[20125]: :54:56.088064+0000 mon.a (mon.0) 678 : cluster [DBG] osdmap e127: 8 total, 8 up, 6 in 2021-03-24T21:54:56.584 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:56.648 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:56.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:56.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:54:56.674 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:54:56.754 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:56.804 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:56.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:56.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:54:56.909 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:56.961 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:56.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:56.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:54:57.049 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2021-03-24T21:54:57.049 INFO:teuthology.orchestra.run.smithi142.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2021-03-24T21:54:57.050 INFO:teuthology.orchestra.run.smithi142.stdout:} 2021-03-24T21:54:57.066 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:57.094 INFO:tasks.rados.rados.0.smithi125.stdout:161: done (2 left) 2021-03-24T21:54:57.096 INFO:tasks.rados.rados.0.smithi125.stdout:163: done (1 left) 2021-03-24T21:54:57.096 INFO:tasks.rados.rados.0.smithi125.stdout:164: done (0 left) 2021-03-24T21:54:57.096 INFO:tasks.rados.rados.0.smithi125.stdout:165: snap_create 2021-03-24T21:54:57.122 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:57.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:57.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:54:57.226 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:57.275 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:57.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:57.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:54:57.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:57 smithi142 conmon[20125]: cluster 2021-03-24T21:54:56.160318+0000 osd.2 (osd.2) 27 : cluster 2021-03-24T21:54:57.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:57 smithi142 conmon[20125]: [DBG] 5.d scrub ok 2021-03-24T21:54:57.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:57 smithi142 conmon[20125]: cluster 2021-03-24T21:54:56.226747+0000 osd.4 (osd.4) 39 : cluster [DBG] 4.d scrub ok 2021-03-24T21:54:57.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:57 smithi142 conmon[20125]: cluster 2021-03-24T21:54:57.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:57 smithi142 conmon[20125]: 2021-03-24T21:54:56.508638+0000 osd.3 (osd.3) 44 : cluster [DBG] 3.7 scrub ok 2021-03-24T21:54:57.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:57 smithi142 conmon[20125]: audit 2021-03-24T21:54:56.956453 2021-03-24T21:54:57.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:57 smithi142 conmon[20125]: +0000 mon.a (mon.0) 679 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:54:57.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:57 smithi142 conmon[20125]: cluster 2021-03-24T21:54:57 2021-03-24T21:54:57.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:57 smithi142 conmon[20125]: .092908+0000 mon.a (mon.0) 680 : cluster [DBG] osdmap e128: 8 total, 8 up, 6 in 2021-03-24T21:54:57.380 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:57.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:57 smithi125 conmon[19936]: cluster 2021-03-24T21:54 2021-03-24T21:54:57.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:57 smithi125 conmon[19936]: :56.160318+0000 osd.2 (osd.2) 27 : cluster [DBG] 5.d scrub ok 2021-03-24T21:54:57.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:57 smithi125 conmon[19936]: cluster 2021-03-24T21:54 2021-03-24T21:54:57.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:57 smithi125 conmon[19936]: :56.226747+0000 osd.4 (osd.4) 39 : cluster [DBG] 4.d scrub ok 2021-03-24T21:54:57.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:57 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:54:57.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:57 smithi125 conmon[19936]: 03-24T21:54:56.508638+0000 osd.3 ( 2021-03-24T21:54:57.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:57 smithi125 conmon[19936]: osd.3) 44 : cluster [DBG] 3.7 scrub ok 2021-03-24T21:54:57.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:57 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:54:57.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:57 smithi125 conmon[19936]: 03-24T21:54:56.956453+0000 mon.a 2021-03-24T21:54:57.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:57 smithi125 conmon[19936]: (mon.0) 679 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:54:57.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:57 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:54:57.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:57 smithi125 conmon[19936]: 03-24T21:54:57.092908+0000 mon.a ( 2021-03-24T21:54:57.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:57 smithi125 conmon[19936]: mon.0) 680 : cluster [DBG] osdmap e128: 8 total, 8 up, 6 in 2021-03-24T21:54:57.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:57 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:54:57.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:57 smithi125 conmon[24927]: 21:54:56.160318+0000 osd.2 (osd.2) 27 : cluster [DBG] 5.d scrub ok 2021-03-24T21:54:57.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:57 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:54:57.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:57 smithi125 conmon[24927]: -24T21:54:56.226747+0000 2021-03-24T21:54:57.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:57 smithi125 conmon[24927]: osd.4 (osd.4) 39 : cluster 2021-03-24T21:54:57.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:57 smithi125 conmon[24927]: [DBG] 4.d scrub ok 2021-03-24T21:54:57.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:57 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:54:57.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:57 smithi125 conmon[24927]: 21:54:56.508638+0000 2021-03-24T21:54:57.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:57 smithi125 conmon[24927]: osd.3 (osd.3) 44 : cluster [DBG] 2021-03-24T21:54:57.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:57 smithi125 conmon[24927]: 3.7 scrub ok 2021-03-24T21:54:57.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:57 smithi125 conmon[24927]: audit 2021-03 2021-03-24T21:54:57.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:57 smithi125 conmon[24927]: -24T21:54:56.956453 2021-03-24T21:54:57.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:57 smithi125 conmon[24927]: +0000 mon.a (mon.0) 679 2021-03-24T21:54:57.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:57 smithi125 conmon[24927]: : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:54:57.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:57 smithi125 conmon[24927]: cluster 2021-03-24T21:54:57.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:57 smithi125 conmon[24927]: 2021-03-24T21 2021-03-24T21:54:57.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:57 smithi125 conmon[24927]: :54:57.092908+0000 mon.a (mon.0) 680 : 2021-03-24T21:54:57.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:57 smithi125 conmon[24927]: cluster [DBG] osdmap e128: 8 total, 8 up, 6 in 2021-03-24T21:54:57.440 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:57.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:57.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:54:57.547 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:57.609 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:54:57.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:57.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:54:57.715 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:57.755 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:54:57.776 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:57.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:57.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:54:57.882 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:57.943 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:57.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:57.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:54:58.048 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:58.096 INFO:tasks.rados.rados.0.smithi125.stdout:165: done (0 left) 2021-03-24T21:54:58.097 INFO:tasks.rados.rados.0.smithi125.stdout:166: write (excl) oid 5 current snap is 13 2021-03-24T21:54:58.097 INFO:tasks.rados.rados.0.smithi125.stdout:166: seq_num 90 ranges {712630=452293,1583497=499693,2555481=425187} 2021-03-24T21:54:58.103 INFO:tasks.rados.rados.0.smithi125.stdout:166: writing smithi12556470-5 from 712630 to 1164923 tid 1 2021-03-24T21:54:58.104 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:58.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:58.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:54:58.109 INFO:tasks.rados.rados.0.smithi125.stdout:166: writing smithi12556470-5 from 1583497 to 2083190 tid 2 2021-03-24T21:54:58.116 INFO:tasks.rados.rados.0.smithi125.stdout:166: writing smithi12556470-5 from 2555481 to 2980668 tid 3 2021-03-24T21:54:58.116 INFO:tasks.rados.rados.0.smithi125.stdout:167: read oid 22 snap -1 2021-03-24T21:54:58.116 INFO:tasks.rados.rados.0.smithi125.stdout:167: expect (ObjNum 81 snap 9 seq_num 81) 2021-03-24T21:54:58.116 INFO:tasks.rados.rados.0.smithi125.stdout:168: snap_remove snap 7 2021-03-24T21:54:58.209 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:58.264 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:58.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:58.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:54:58.353 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:58 smithi125 conmon[19936]: cluster 2021-03-24T21:54:56. 2021-03-24T21:54:58.374 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:58.435 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:54:58.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:58.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:54:58.542 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:58.555 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:58 smithi142 conmon[20125]: cluster 2021-03-24T21:54: 2021-03-24T21:54:58.556 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:58 smithi142 conmon[20125]: 56.716460+0000 mgr.y (mgr.14144) 321 : cluster [DBG] pgmap v353: 148 pgs: 148 active+clean; 129 MiB data, 322 MiB used, 536 GiB / 536 GiB avail; 9.8 MiB/s rd, 4.8 MiB/s wr, 32 op/s 2021-03-24T21:54:58.556 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:58 smithi142 conmon[20125]: audit 2021-03-24T21:54: 2021-03-24T21:54:58.556 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:58 smithi142 conmon[20125]: 57.271770+0000 mon.a (mon.0) 681 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:54:58.556 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:58 smithi142 conmon[20125]: cluster 2021-03-24T21:54:57.325708+0000 2021-03-24T21:54:58.556 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:58 smithi142 conmon[20125]: osd.0 (osd.0) 42 : cluster [DBG] 4.6 scrub ok 2021-03-24T21:54:58.557 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:58 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:54:58.557 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:58 smithi142 conmon[20125]: 24T21:54:58.093883+0000 mon.a (mon.0) 2021-03-24T21:54:58.557 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:58 smithi142 conmon[20125]: 682 : cluster [DBG] osdmap e129: 8 total, 8 up, 6 in 2021-03-24T21:54:58.603 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:58.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:58.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:54:58.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:58 smithi125 conmon[19936]: 716460+0000 mgr.y (mgr.14144) 321 : cluster [DBG] pgmap v353: 148 pgs: 148 active+clean; 129 MiB data, 322 MiB used, 536 GiB / 536 GiB avail; 9.8 MiB/s rd, 4.8 MiB/s wr, 32 op/s 2021-03-24T21:54:58.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:58 smithi125 conmon[19936]: audit 2021-03-24T21:54:57.271770+0000 2021-03-24T21:54:58.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:58 smithi125 conmon[19936]: mon.a (mon.0) 681 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:54:58.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:58 smithi125 conmon[19936]: cluster 2021-03-24T21:54:58.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:58 smithi125 conmon[19936]: 2021-03-24T21:54:57.325708+0000 osd.0 (osd.0) 42 : cluster [DBG] 4.6 scrub ok 2021-03-24T21:54:58.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:58 smithi125 conmon[19936]: 2021-03-24T21:54:58.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:58 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:54:58.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:58 smithi125 conmon[19936]: :54:58.093883+0000 mon.a (mon.0) 682 : cluster [DBG] osdmap e129: 8 total, 8 up, 6 in 2021-03-24T21:54:58.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:58 smithi125 conmon[19936]: 2021-03-24T21:54:58.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:58 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:54:58.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:58 smithi125 conmon[24927]: 21:54:56.716460+0000 mgr.y (mgr.14144) 321 : cluster [DBG] pgmap v353: 148 pgs: 148 active+clean; 129 MiB data, 322 MiB used, 536 GiB / 536 GiB avail; 9.8 MiB/s rd, 4.8 MiB/s wr, 32 op/s 2021-03-24T21:54:58.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:58 smithi125 conmon[24927]: audit 2021- 2021-03-24T21:54:58.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:58 smithi125 conmon[24927]: 03-24T21:54:57.271770+0000 2021-03-24T21:54:58.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:58 smithi125 conmon[24927]: mon.a (mon.0) 681 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:54:58.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:58 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:54:58.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:58 smithi125 conmon[24927]: -03-24T21:54:57.325708+0000 osd.0 2021-03-24T21:54:58.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:58 smithi125 conmon[24927]: (osd.0) 42 : cluster [DBG] 4.6 scrub ok 2021-03-24T21:54:58.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:58 smithi125 conmon[24927]: cluster 2021-03-24T21:54:58.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:58 smithi125 conmon[24927]: 2021-03 2021-03-24T21:54:58.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:58 smithi125 conmon[24927]: -24T21:54:58. 2021-03-24T21:54:58.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:58 smithi125 conmon[24927]: 093883+0000 mon.a (mon.0 2021-03-24T21:54:58.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:58 smithi125 conmon[24927]: ) 682 : cluster [DBG] osdmap e129: 8 total, 8 up, 6 in 2021-03-24T21:54:58.711 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:58.769 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:58.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:58.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:54:58.874 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:58.924 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:58.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:58.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:54:58.984 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-24T21:54:59.039 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-24T21:54:59.040 INFO:teuthology.orchestra.run.smithi125.stdout: "num_ops": 0 2021-03-24T21:54:59.040 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-24T21:54:59.040 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:59.093 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:59.098 INFO:tasks.rados.rados.0.smithi125.stdout:168: done (2 left) 2021-03-24T21:54:59.098 INFO:tasks.rados.rados.0.smithi125.stdout:169: copy_from oid 47 from oid 43 current snap is 13 2021-03-24T21:54:59.099 INFO:tasks.rados.rados.0.smithi125.stdout:170: snap_create 2021-03-24T21:54:59.100 INFO:tasks.rados.rados.0.smithi125.stdout:166: finishing write tid 1 to smithi12556470-5 2021-03-24T21:54:59.100 INFO:tasks.rados.rados.0.smithi125.stdout:166: finishing write tid 2 to smithi12556470-5 2021-03-24T21:54:59.100 INFO:tasks.rados.rados.0.smithi125.stdout:166: finishing write tid 3 to smithi12556470-5 2021-03-24T21:54:59.100 INFO:tasks.rados.rados.0.smithi125.stdout:166: finishing write tid 4 to smithi12556470-5 2021-03-24T21:54:59.101 INFO:tasks.rados.rados.0.smithi125.stdout:166: finishing write tid 5 to smithi12556470-5 2021-03-24T21:54:59.101 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 5 v 35 (ObjNum 90 snap 13 seq_num 90) dirty exists 2021-03-24T21:54:59.101 INFO:tasks.rados.rados.0.smithi125.stdout:166: left oid 5 (ObjNum 90 snap 13 seq_num 90) 2021-03-24T21:54:59.101 INFO:tasks.rados.rados.0.smithi125.stdout:167: expect (ObjNum 81 snap 9 seq_num 81) 2021-03-24T21:54:59.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:59.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:54:59.204 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:59.249 INFO:journalctl@ceph.osd.0.smithi125.stdout:Mar 24 21:54:59 smithi125 conmon[28026]: debug 2021-03-24T21:54:59.103+0000 7f61b9481700 -1 osd.0 pg_epoch: 130 pg[7.9( v 130'37 (0'0,130'37] local-lis/les=108/109 n=6 ec=99/99 lis/c=108/108 les/c/f=109/109/0 sis=108) [0,4,7] r=0 lpr=108 luod=129'35 crt=130'37 lcod 129'34 mlcod 129'34 active+clean+snaptrim trimq=[f~1] ps=[c~2,10~1,13~1]] removing snap head 2021-03-24T21:54:59.258 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:54:59.260 INFO:tasks.rados.rados.0.smithi125.stdout:169: finishing copy_from to smithi12556470-47 2021-03-24T21:54:59.261 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 47 v 24 (ObjNum 62 snap 4 seq_num 62) dirty exists 2021-03-24T21:54:59.261 INFO:tasks.rados.rados.0.smithi125.stdout:169: finishing copy_from racing read to smithi12556470-47 2021-03-24T21:54:59.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:59.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:54:59.364 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:59.367 INFO:journalctl@ceph.osd.4.smithi142.stdout:Mar 24 21:54:59 smithi142 conmon[24076]: debug 2021-03-24T21:54:59.093+0000 7fc561e38700 -1 osd.4 pg_epoch: 130 pg[7.1( v 127'43 (0'0,127'43] local-lis/les=108/109 n=10 ec=99/99 lis/c=108/108 les/c/f=109/111/0 sis=108) [4,3,0] r=0 lpr=108 crt=127'43 lcod 124'41 mlcod 124'41 active+clean+snaptrim trimq=[f~1] ps=[c~2,10~1,13~1]] removing snap head 2021-03-24T21:54:59.367 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 21:54:59 smithi142 conmon[27337]: debug 2021-03-24T21:54:59.105+0000 7f5499710700 -1 osd.5 pg_epoch: 130 pg[7.b( v 124'49 (0'0,124'49] local-lis/les=108/109 n=7 ec=99/99 lis/c=108/108 les/c/f=109/109/0 sis=108) [5,2,4] r=0 lpr=108 crt=124'49 lcod 123'47 mlcod 123'47 active+clean+snaptrim trimq=[f~1] ps=[c~2,10~1,13~1]] removing snap head 2021-03-24T21:54:59.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:59 smithi142 conmon[20125]: cluster 2021-03-24T21:54:57.741460+0000 osd.7 (osd.7) 34 2021-03-24T21:54:59.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:59 smithi142 conmon[20125]: : cluster [DBG] 5.1a scrub ok 2021-03-24T21:54:59.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:59 smithi142 conmon[20125]: cluster 2021-03-24T21:54:59.096818+0000 2021-03-24T21:54:59.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:54:59 smithi142 conmon[20125]: mon.a (mon.0) 683 : cluster [DBG] osdmap e130: 8 total, 8 up, 6 in 2021-03-24T21:54:59.414 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:54:59.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:59.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:54:59.518 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:59.531 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:59 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:54:59.532 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:59 smithi125 conmon[19936]: 54:57.741460+0000 osd.7 (osd.7) 34 : cluster [DBG] 5.1a scrub ok 2021-03-24T21:54:59.532 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:59 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:54:59.532 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:59 smithi125 conmon[19936]: -03-24T21:54:59.096818+0000 mon.a ( 2021-03-24T21:54:59.532 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:54:59 smithi125 conmon[19936]: mon.0) 683 : cluster [DBG] osdmap e130: 8 total, 8 up, 6 in 2021-03-24T21:54:59.533 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:59 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:54:59.533 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:59 smithi125 conmon[24927]: 21:54:57.741460+0000 osd.7 (osd.7) 34 : cluster [DBG] 5.1a scrub ok 2021-03-24T21:54:59.533 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:59 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:54:59.533 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:59 smithi125 conmon[24927]: 54:59.096818+0000 mon.a (mon.0) 683 2021-03-24T21:54:59.534 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:54:59 smithi125 conmon[24927]: : cluster [DBG] osdmap e130: 8 total, 8 up, 6 in 2021-03-24T21:54:59.573 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:54:59.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:59.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:54:59.679 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:59.767 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:54:59.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:59.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:54:59.874 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:54:59.927 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:54:59.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:54:59.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:55:00.033 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:00.095 INFO:tasks.rados.rados.0.smithi125.stdout:166: done (3 left) 2021-03-24T21:55:00.096 INFO:tasks.rados.rados.0.smithi125.stdout:167: done (2 left) 2021-03-24T21:55:00.096 INFO:tasks.rados.rados.0.smithi125.stdout:169: done (1 left) 2021-03-24T21:55:00.096 INFO:tasks.rados.rados.0.smithi125.stdout:170: done (0 left) 2021-03-24T21:55:00.097 INFO:tasks.rados.rados.0.smithi125.stdout:171: write (excl) oid 42 current snap is 14 2021-03-24T21:55:00.097 INFO:tasks.rados.rados.0.smithi125.stdout:171: seq_num 91 ranges {729437=527542,1925626=643351,3266551=262886} 2021-03-24T21:55:00.098 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:00.098 INFO:tasks.rados.rados.0.smithi125.stdout:171: writing smithi12556470-42 from 729437 to 1256979 tid 1 2021-03-24T21:55:00.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:00.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:55:00.106 INFO:tasks.rados.rados.0.smithi125.stdout:171: writing smithi12556470-42 from 1925626 to 2568977 tid 2 2021-03-24T21:55:00.112 INFO:tasks.rados.rados.0.smithi125.stdout:171: writing smithi12556470-42 from 3266551 to 3529437 tid 3 2021-03-24T21:55:00.112 INFO:tasks.rados.rados.0.smithi125.stdout:172: copy_from oid 6 from oid 31 current snap is 14 2021-03-24T21:55:00.113 INFO:tasks.rados.rados.0.smithi125.stdout:171: finishing write tid 1 to smithi12556470-42 2021-03-24T21:55:00.113 INFO:tasks.rados.rados.0.smithi125.stdout:171: finishing write tid 2 to smithi12556470-42 2021-03-24T21:55:00.113 INFO:tasks.rados.rados.0.smithi125.stdout:173: snap_remove snap 11 2021-03-24T21:55:00.202 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:00.253 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:00.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:00.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:55:00.262 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.2 dump_blocked_ops 2021-03-24T21:55:00.357 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:00.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:00 smithi142 conmon[20125]: cluster 2021-03-24T21:54: 2021-03-24T21:55:00.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:00 smithi142 conmon[20125]: 58.717420+0000 mgr.y (mgr.14144) 322 : cluster [DBG] pgmap v356: 148 pgs: 148 active+clean; 129 MiB data, 322 MiB used, 536 GiB / 536 GiB avail; 7.4 MiB/s rd, 0 B/s wr, 5 op/s 2021-03-24T21:55:00.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:00 smithi142 conmon[20125]: cluster 2021-03-24T21:55:00.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:00 smithi142 conmon[20125]: 2021-03-24T21:55:00.087862+0000 2021-03-24T21:55:00.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:00 smithi142 conmon[20125]: mon.a (mon.0) 684 : cluster 2021-03-24T21:55:00.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:00 smithi142 conmon[20125]: [DBG] osdmap e131: 8 total, 8 up, 6 in 2021-03-24T21:55:00.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:00 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:55:00.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:00 smithi142 conmon[20125]: -03-24T21:55:00. 2021-03-24T21:55:00.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:00 smithi142 conmon[20125]: 135321+0000 osd.4 (osd.4 2021-03-24T21:55:00.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:00 smithi142 conmon[20125]: ) 40 : cluster [DBG] 4.1 scrub ok 2021-03-24T21:55:00.415 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:00.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:00.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:55:00.423 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T21:55:00.447 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T21:55:00.468 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T21:55:00.488 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T21:55:00.511 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T21:55:00.522 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:00.535 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T21:55:00.561 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T21:55:00.570 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:00 smithi125 conmon[19936]: cluster 2021-03-24T21:54:58.717420+0000 mgr.y (mgr.14144 2021-03-24T21:55:00.570 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:00 smithi125 conmon[19936]: ) 322 : cluster [DBG] pgmap v356: 148 pgs: 148 active+clean; 129 MiB data, 322 MiB used, 536 GiB / 536 GiB avail; 7.4 MiB/s rd, 0 B/s wr, 5 op/s 2021-03-24T21:55:00.571 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:00 smithi125 conmon[19936]: cluster 2021-03-24T21:55:00.087862+0000 mon.a (mon.0) 684 : cluster [DBG] osdmap e131: 8 total, 8 up, 6 in 2021-03-24T21:55:00.571 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:00 smithi125 conmon[19936]: cluster 2021-03-24T21:55:00.135321+0000 osd.4 (osd.4) 40 : cluster 2021-03-24T21:55:00.572 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:00 smithi125 conmon[19936]: [DBG] 4.1 scrub ok 2021-03-24T21:55:00.572 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:00 smithi125 conmon[24927]: cluster 2021-03-24T21:54:58.717420+0000 mgr.y (mgr.14144 2021-03-24T21:55:00.573 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:00 smithi125 conmon[24927]: ) 322 : cluster [DBG] pgmap v356: 148 pgs: 148 active+clean; 129 MiB data, 322 MiB used, 536 GiB / 536 GiB avail; 7.4 MiB/s rd, 0 B/s wr, 5 op/s 2021-03-24T21:55:00.573 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:00 smithi125 conmon[24927]: cluster 2021-03-24T21:55:00.087862+0000 mon.a (mon 2021-03-24T21:55:00.573 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:00 smithi125 conmon[24927]: .0) 684 : cluster [DBG] osdmap e131: 8 total, 8 up, 6 in 2021-03-24T21:55:00.574 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:00 smithi125 conmon[24927]: cluster 2021-03-24T21:55 2021-03-24T21:55:00.574 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:00 smithi125 conmon[24927]: :00.135321+0000 osd.4 (osd.4) 40 : cluster [DBG] 4.1 scrub ok 2021-03-24T21:55:00.579 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:00.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:00.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:55:00.586 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T21:55:00.588 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-24T21:55:00.589 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-24T21:55:00.589 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-24T21:55:00.589 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-24T21:55:00.590 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-24T21:55:00.590 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-24T21:55:00.591 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-24T21:55:00.591 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-24T21:55:00.612 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:55:00.684 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:00.819 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:00.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:00.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:55:00.924 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:00.983 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:00.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:00.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:55:01.088 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:01.105 INFO:tasks.rados.rados.0.smithi125.stdout:173: done (2 left) 2021-03-24T21:55:01.105 INFO:tasks.rados.rados.0.smithi125.stdout:174: copy_from oid 15 from oid 7 current snap is 14 2021-03-24T21:55:01.105 INFO:tasks.rados.rados.0.smithi125.stdout:171: finishing write tid 3 to smithi12556470-42 2021-03-24T21:55:01.106 INFO:tasks.rados.rados.0.smithi125.stdout:171: finishing write tid 4 to smithi12556470-42 2021-03-24T21:55:01.106 INFO:tasks.rados.rados.0.smithi125.stdout:171: finishing write tid 5 to smithi12556470-42 2021-03-24T21:55:01.106 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 42 v 30 (ObjNum 91 snap 14 seq_num 91) dirty exists 2021-03-24T21:55:01.106 INFO:tasks.rados.rados.0.smithi125.stdout:171: left oid 42 (ObjNum 91 snap 14 seq_num 91) 2021-03-24T21:55:01.107 INFO:tasks.rados.rados.0.smithi125.stdout:172: finishing copy_from to smithi12556470-6 2021-03-24T21:55:01.107 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 6 v 41 (ObjNum 73 snap 7 seq_num 73) dirty exists 2021-03-24T21:55:01.107 INFO:tasks.rados.rados.0.smithi125.stdout:172: finishing copy_from racing read to smithi12556470-6 2021-03-24T21:55:01.107 INFO:tasks.rados.rados.0.smithi125.stdout:171: done (2 left) 2021-03-24T21:55:01.107 INFO:tasks.rados.rados.0.smithi125.stdout:172: done (1 left) 2021-03-24T21:55:01.108 INFO:tasks.rados.rados.0.smithi125.stdout:175: delete oid 39 current snap is 14 2021-03-24T21:55:01.116 INFO:tasks.rados.rados.0.smithi125.stdout:175: done (1 left) 2021-03-24T21:55:01.116 INFO:tasks.rados.rados.0.smithi125.stdout:176: rollback oid 19 current snap is 14 2021-03-24T21:55:01.116 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 19 to 13 2021-03-24T21:55:01.117 INFO:tasks.rados.rados.0.smithi125.stdout:177: delete oid 11 current snap is 14 2021-03-24T21:55:01.117 INFO:tasks.rados.rados.0.smithi125.stdout:176: finishing rollback tid 1 to smithi12556470-19 2021-03-24T21:55:01.117 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 19 v 0 (ObjNum 1112276168 snap 21916 seq_num 4) dirty dne 2021-03-24T21:55:01.119 INFO:tasks.rados.rados.0.smithi125.stdout:176: done (2 left) 2021-03-24T21:55:01.119 INFO:tasks.rados.rados.0.smithi125.stdout:177: done (1 left) 2021-03-24T21:55:01.119 INFO:tasks.rados.rados.0.smithi125.stdout:178: snap_create 2021-03-24T21:55:01.127 INFO:tasks.rados.rados.0.smithi125.stdout:174: finishing copy_from to smithi12556470-15 2021-03-24T21:55:01.127 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 15 v 20 (ObjNum 64 snap 5 seq_num 64) dirty exists 2021-03-24T21:55:01.128 INFO:tasks.rados.rados.0.smithi125.stdout:174: finishing copy_from racing read to smithi12556470-15 2021-03-24T21:55:01.141 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:01.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:01.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:55:01.246 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:01.319 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:01.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:01.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:55:01.363 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:01 smithi125 conmon[19936]: audit 2021-03-24T21:55:00.398052 2021-03-24T21:55:01.363 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:01 smithi125 conmon[19936]: +0000 mon.a (mon.0) 685 : audit [DBG] from='client.? v1:172.21.15.125:0/3943396179' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:55:01.364 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:01 smithi125 conmon[19936]: cluster 2021-03-24T21:55:01.100472+0000 mon.a (mon.0) 686 : cluster 2021-03-24T21:55:01.364 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:01 smithi125 conmon[19936]: [DBG] osdmap e132: 8 total, 8 up, 6 in 2021-03-24T21:55:01.364 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:01 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:55:01.364 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:01 smithi125 conmon[19936]: 21:55:01.169452+0000 osd.4 (osd.4) 41 : cluster [DBG] 2021-03-24T21:55:01.365 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:01 smithi125 conmon[19936]: 3.a deep-scrub ok 2021-03-24T21:55:01.365 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:01 smithi125 conmon[24927]: audit 2021-03-24T21 2021-03-24T21:55:01.365 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:01 smithi125 conmon[24927]: :55:00.398052+0000 mon.a (mon.0) 685 : audit [DBG] from='client.? v1:172.21.15.125:0/3943396179' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:55:01.366 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:01 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:55:01.366 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:01 smithi125 conmon[24927]: 03-24T21:55:01.100472 2021-03-24T21:55:01.366 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:01 smithi125 conmon[24927]: +0000 mon.a (mon.0 2021-03-24T21:55:01.366 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:01 smithi125 conmon[24927]: ) 686 : cluster [DBG] 2021-03-24T21:55:01.367 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:01 smithi125 conmon[24927]: osdmap e132: 8 total, 8 up, 6 in 2021-03-24T21:55:01.367 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:01 smithi125 conmon[24927]: cluster 2021-03-24T21:55:01.367 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:01 smithi125 conmon[24927]: 2021-03-24T 2021-03-24T21:55:01.367 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:01 smithi125 conmon[24927]: 21:55:01.169452+0000 osd.4 (osd 2021-03-24T21:55:01.368 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:01 smithi125 conmon[24927]: .4) 41 : cluster [DBG] 3.a deep-scrub ok 2021-03-24T21:55:01.424 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:01.483 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:01.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:01.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:55:01.590 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:01.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:01 smithi142 conmon[20125]: audit 2021-03-24T21:55: 2021-03-24T21:55:01.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:01 smithi142 conmon[20125]: 00.398052+0000 mon.a (mon.0) 685 : audit [DBG] from='client.? v1:172.21.15.125:0/3943396179' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:55:01.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:01 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:55:01.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:01 smithi142 conmon[20125]: 55:01.100472+0000 mon.a (mon.0) 686 : cluster [DBG] osdmap e132: 8 total, 8 up, 6 in 2021-03-24T21:55:01.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:01 smithi142 conmon[20125]: 2021-03-24T21:55:01.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:01 smithi142 conmon[20125]: cluster 2021-03-24T21:55:01.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:01 smithi142 conmon[20125]: 2021-03-24T21:55:01. 2021-03-24T21:55:01.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:01 smithi142 conmon[20125]: 169452+0000 osd.4 (osd.4) 41 2021-03-24T21:55:01.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:01 smithi142 conmon[20125]: : cluster [DBG] 3.a deep-scrub ok 2021-03-24T21:55:01.648 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:01.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:01.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:55:01.755 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:01.820 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:01.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:01.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:01.926 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:01.977 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:01.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:01.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:55:02.083 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:02.103 INFO:tasks.rados.rados.0.smithi125.stdout:174: done (1 left) 2021-03-24T21:55:02.104 INFO:tasks.rados.rados.0.smithi125.stdout:178: done (0 left) 2021-03-24T21:55:02.104 INFO:tasks.rados.rados.0.smithi125.stdout:179: write (excl) oid 12 current snap is 15 2021-03-24T21:55:02.104 INFO:tasks.rados.rados.0.smithi125.stdout:179: seq_num 94 ranges {779858=469642,1817425=507031,2781526=586628} 2021-03-24T21:55:02.111 INFO:tasks.rados.rados.0.smithi125.stdout:179: writing smithi12556470-12 from 779858 to 1249500 tid 1 2021-03-24T21:55:02.119 INFO:tasks.rados.rados.0.smithi125.stdout:179: writing smithi12556470-12 from 1817425 to 2324456 tid 2 2021-03-24T21:55:02.126 INFO:tasks.rados.rados.0.smithi125.stdout:179: writing smithi12556470-12 from 2781526 to 3368154 tid 3 2021-03-24T21:55:02.127 INFO:tasks.rados.rados.0.smithi125.stdout:180: snap_create 2021-03-24T21:55:02.127 INFO:tasks.rados.rados.0.smithi125.stdout:179: finishing write tid 1 to smithi12556470-12 2021-03-24T21:55:02.127 INFO:tasks.rados.rados.0.smithi125.stdout:179: finishing write tid 2 to smithi12556470-12 2021-03-24T21:55:02.133 INFO:tasks.rados.rados.0.smithi125.stdout:179: finishing write tid 3 to smithi12556470-12 2021-03-24T21:55:02.133 INFO:tasks.rados.rados.0.smithi125.stdout:179: finishing write tid 4 to smithi12556470-12 2021-03-24T21:55:02.133 INFO:tasks.rados.rados.0.smithi125.stdout:179: finishing write tid 5 to smithi12556470-12 2021-03-24T21:55:02.134 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 12 v 45 (ObjNum 94 snap 15 seq_num 94) dirty exists 2021-03-24T21:55:02.134 INFO:tasks.rados.rados.0.smithi125.stdout:179: left oid 12 (ObjNum 94 snap 15 seq_num 94) 2021-03-24T21:55:02.149 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:02.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:02.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:55:02.257 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:02.318 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:02.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:02.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:55:02.424 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:02.458 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:02 smithi125 conmon[19936]: cluster 2021-03-24T21:55: 2021-03-24T21:55:02.458 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:02 smithi125 conmon[19936]: 00.718148+0000 mgr.y (mgr.14144) 323 : cluster [DBG] pgmap v359: 148 pgs: 148 active+clean; 142 MiB data, 343 MiB used, 536 GiB / 536 GiB avail; 14 MiB/s rd, 3.0 MiB/s wr, 20 op/s 2021-03-24T21:55:02.458 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:02 smithi125 conmon[19936]: cluster 2021-03-24T21:55:01.577752+0000 osd.3 (osd.3) 45 : cluster [DBG] 2021-03-24T21:55:02.459 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:02 smithi125 conmon[19936]: 5.14 scrub ok 2021-03-24T21:55:02.459 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:02 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:55:02.460 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:02 smithi125 conmon[19936]: -24T21:55:02.101701+0000 mon.a (mon.0) 687 : 2021-03-24T21:55:02.460 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:02 smithi125 conmon[19936]: cluster [DBG] osdmap e133: 8 total, 8 up, 6 in 2021-03-24T21:55:02.461 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:02 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:55:02.461 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:02 smithi125 conmon[19936]: 21:55:02.129298+0000 osd.4 (osd.4) 2021-03-24T21:55:02.462 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:02 smithi125 conmon[19936]: 42 : cluster [DBG] 5.1c scrub ok 2021-03-24T21:55:02.462 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:02 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:55:02.463 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:02 smithi125 conmon[24927]: 21:55:00.718148+0000 mgr.y (mgr.14144) 323 : cluster [DBG] pgmap v359: 148 pgs: 148 active+clean; 142 MiB data, 343 MiB used, 536 GiB / 536 GiB avail; 14 MiB/s rd, 3.0 MiB/s wr, 20 op/s 2021-03-24T21:55:02.463 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:02 smithi125 conmon[24927]: cluster 2021-03-24T21:55:02.464 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:02 smithi125 conmon[24927]: 2021-03-24T21:55: 2021-03-24T21:55:02.464 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:02 smithi125 conmon[24927]: 01.577752+0000 osd.3 (osd. 2021-03-24T21:55:02.464 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:02 smithi125 conmon[24927]: 3) 45 : cluster [DBG] 5.14 scrub ok 2021-03-24T21:55:02.465 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:02 smithi125 conmon[24927]: 2021-03-24T21:55:02.465 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:02 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:55:02.466 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:02 smithi125 conmon[24927]: :55:02.101701+0000 2021-03-24T21:55:02.466 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:02 smithi125 conmon[24927]: mon.a (mon.0) 687 : cluster 2021-03-24T21:55:02.467 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:02 smithi125 conmon[24927]: [DBG] osdmap e133: 8 total, 8 up, 6 in 2021-03-24T21:55:02.467 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:02 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:55:02.468 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:02 smithi125 conmon[24927]: -24T21:55:02. 2021-03-24T21:55:02.468 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:02 smithi125 conmon[24927]: 129298+0000 osd.4 (osd.4) 2021-03-24T21:55:02.468 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:02 smithi125 conmon[24927]: 42 : cluster [DBG] 5.1c scrub ok 2021-03-24T21:55:02.488 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:02.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:02.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:55:02.594 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:02.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:02 smithi142 conmon[20125]: cluster 2021-03-24T21:55:00.718148+0000 mgr.y (mgr.14144) 323 : cluster [DBG] 2021-03-24T21:55:02.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:02 smithi142 conmon[20125]: pgmap v359: 148 pgs: 148 active+clean; 142 MiB data, 343 MiB used, 536 GiB / 536 GiB avail; 14 MiB/s rd, 3.0 MiB/s wr, 20 op/s 2021-03-24T21:55:02.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:02 smithi142 conmon[20125]: cluster 2021-03-24T21:55:01.577752+0000 osd.3 (osd.3) 45 2021-03-24T21:55:02.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:02 smithi142 conmon[20125]: : cluster [DBG] 5.14 scrub ok 2021-03-24T21:55:02.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:02 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:55:02.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:02 smithi142 conmon[20125]: -03-24T21:55:02. 2021-03-24T21:55:02.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:02 smithi142 conmon[20125]: 101701+0000 mon.a (mon. 2021-03-24T21:55:02.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:02 smithi142 conmon[20125]: 0) 687 : cluster [DBG] osdmap e133: 8 total, 8 up, 6 in 2021-03-24T21:55:02.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:02 smithi142 conmon[20125]: 2021-03-24T21:55:02.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:02 smithi142 conmon[20125]: cluster 2021-03-24T21:55:02.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:02 smithi142 conmon[20125]: 2021-03-24T21:55 2021-03-24T21:55:02.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:02 smithi142 conmon[20125]: :02.129298+0000 osd.4 ( 2021-03-24T21:55:02.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:02 smithi142 conmon[20125]: osd.4) 42 : cluster [DBG] 2021-03-24T21:55:02.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:02 smithi142 conmon[20125]: 5.1c scrub ok 2021-03-24T21:55:02.649 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:02.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:02.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:55:02.756 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:02.812 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:02.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:02.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:55:02.919 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:02.970 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:02.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:02.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:03.006 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 7] out_osds: [6, 1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-03-24T21:55:03.006 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-24T21:55:03.007 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 0.664524 2021-03-24T21:55:03.007 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd primary-affinity 5 0.6645241636505669 2021-03-24T21:55:03.076 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:03.105 INFO:tasks.rados.rados.0.smithi125.stdout:179: done (1 left) 2021-03-24T21:55:03.105 INFO:tasks.rados.rados.0.smithi125.stdout:180: done (0 left) 2021-03-24T21:55:03.106 INFO:tasks.rados.rados.0.smithi125.stdout:181: delete oid 7 current snap is 16 2021-03-24T21:55:03.111 INFO:tasks.rados.rados.0.smithi125.stdout:181: done (0 left) 2021-03-24T21:55:03.111 INFO:tasks.rados.rados.0.smithi125.stdout:182: snap_create 2021-03-24T21:55:03.114 INFO:teuthology.orchestra.run.smithi125.stderr:noscrub is set 2021-03-24T21:55:03.131 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:03.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:03.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:55:03.235 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:03.292 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:03.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:03.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:55:03.320 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:55:03.396 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:03.457 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:03.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:03.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:55:03.564 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:03.578 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:03 smithi125 conmon[19936]: audit 2021-03-24 2021-03-24T21:55:03.578 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:03 smithi125 conmon[19936]: T21:55:02.932845+0000 mon.c (mon.2) 27 : audit [INF] from='client.? v1:172.21.15.125:0/923985132' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T21:55:03.578 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:03 smithi125 conmon[19936]: audit 2021-03-24T21:55:03.579 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:03 smithi125 conmon[19936]: 2021-03-24T21 2021-03-24T21:55:03.579 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:03 smithi125 conmon[19936]: :55:02.933665+0000 2021-03-24T21:55:03.579 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:03 smithi125 conmon[19936]: mon.a (mon.0) 688 2021-03-24T21:55:03.579 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:03 smithi125 conmon[19936]: : audit [INF] from='client.34721 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T21:55:03.580 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:03 smithi125 conmon[19936]: cluster 2021-03-24T21:55:03 2021-03-24T21:55:03.580 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:03 smithi125 conmon[19936]: .095795+0000 mon.a (mon.0) 689 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:55:03.580 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:03 smithi125 conmon[19936]: audit 2021-03 2021-03-24T21:55:03.580 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:03 smithi125 conmon[19936]: -24T21:55:03.104435+0000 mon.a (mon.0) 690 : audit [INF] 2021-03-24T21:55:03.581 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:03 smithi125 conmon[19936]: from='client.34721 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T21:55:03.581 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:03 smithi125 conmon[19936]: cluster 2021-03-24T21:55 2021-03-24T21:55:03.581 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:03 smithi125 conmon[19936]: :03.104654+0000 mon.a (mon.0) 691 : 2021-03-24T21:55:03.581 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:03 smithi125 conmon[19936]: cluster [DBG] osdmap e134: 8 total, 8 up, 6 in 2021-03-24T21:55:03.582 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:03 smithi125 conmon[24927]: audit 2021-03-24T21:55: 2021-03-24T21:55:03.582 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:03 smithi125 conmon[24927]: 02.932845+0000 mon.c (mon.2) 27 : audit [INF] from='client.? v1:172.21.15.125:0/923985132' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T21:55:03.582 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:03 smithi125 conmon[24927]: audit 2021-03-24T21:55:02.933665+0000 2021-03-24T21:55:03.583 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:03 smithi125 conmon[24927]: mon.a (mon.0) 688 : audit 2021-03-24T21:55:03.583 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:03 smithi125 conmon[24927]: [INF] from='client.34721 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T21:55:03.583 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:03 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:55:03.584 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:03 smithi125 conmon[24927]: 21:55:03.095795+0000 mon.a (mon 2021-03-24T21:55:03.584 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:03 smithi125 conmon[24927]: .0) 689 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:55:03.584 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:03 smithi125 conmon[24927]: audit 2021 2021-03-24T21:55:03.584 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:03 smithi125 conmon[24927]: -03-24T21:55:03.104435+0000 2021-03-24T21:55:03.584 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:03 smithi125 conmon[24927]: mon.a (mon.0) 690 : audit [INF] 2021-03-24T21:55:03.585 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:03 smithi125 conmon[24927]: from='client.34721 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T21:55:03.585 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:03 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:55:03.585 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:03 smithi125 conmon[24927]: 03-24T21:55:03. 2021-03-24T21:55:03.585 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:03 smithi125 conmon[24927]: 104654+0000 mon.a (mon.0) 2021-03-24T21:55:03.586 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:03 smithi125 conmon[24927]: 691 : cluster [DBG] osdmap e134: 8 total, 8 up, 6 in 2021-03-24T21:55:03.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:03 smithi142 conmon[20125]: audit 2021-03-24T21:55:02.932845+0000 mon.c (mon.2) 27 : audit [INF] 2021-03-24T21:55:03.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:03 smithi142 conmon[20125]: from='client.? v1:172.21.15.125:0/923985132' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T21:55:03.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:03 smithi142 conmon[20125]: audit 2021-03-24T21:55:02.933665+0000 mon.a (mon.0) 688 : audit [INF] 2021-03-24T21:55:03.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:03 smithi142 conmon[20125]: from='client.34721 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T21:55:03.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:03 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:55:03.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:03 smithi142 conmon[20125]: 21:55:03.095795+0000 mon.a (mon.0) 689 2021-03-24T21:55:03.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:03 smithi142 conmon[20125]: : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:55:03.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:03 smithi142 conmon[20125]: audit 2021-03-24T21 2021-03-24T21:55:03.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:03 smithi142 conmon[20125]: :55:03.104435+0000 mon.a (mon. 2021-03-24T21:55:03.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:03 smithi142 conmon[20125]: 0) 690 : audit [INF] from='client.34721 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T21:55:03.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:03 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:55:03.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:03 smithi142 conmon[20125]: T21:55:03.104654+0000 mon.a ( 2021-03-24T21:55:03.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:03 smithi142 conmon[20125]: mon.0) 691 : cluster [DBG] osdmap e134: 8 total, 8 up, 6 in 2021-03-24T21:55:03.621 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:03.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:03.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:55:03.727 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:03.785 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:03.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:03.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:55:03.891 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:03.947 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:03.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:03.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:55:04.053 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:04.106 INFO:tasks.rados.rados.0.smithi125.stdout:182: done (0 left) 2021-03-24T21:55:04.107 INFO:tasks.rados.rados.0.smithi125.stdout:183: read oid 32 snap 15 2021-03-24T21:55:04.107 INFO:tasks.rados.rados.0.smithi125.stdout:183: expect (ObjNum 31 snap 0 seq_num 31) 2021-03-24T21:55:04.107 INFO:tasks.rados.rados.0.smithi125.stdout:184: snap_create 2021-03-24T21:55:04.112 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:04.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:04.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:55:04.121 INFO:tasks.rados.rados.0.smithi125.stdout:183: expect (ObjNum 31 snap 0 seq_num 31) 2021-03-24T21:55:04.157 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-24T21:55:04.217 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:04.266 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:04.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:04.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:04.357 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:04 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:55:04.357 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:04 smithi125 conmon[24927]: T21:55:02.719042+0000 mgr.y (mgr.14144) 324 : cluster [DBG] pgmap v362: 148 pgs: 148 active+clean; 144 MiB data, 350 MiB used, 536 GiB / 536 GiB avail; 6.7 MiB/s rd, 3.3 MiB/s wr, 17 op/s 2021-03-24T21:55:04.357 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:04 smithi125 conmon[24927]: cluster 2021-03-24T21:55 2021-03-24T21:55:04.358 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:04 smithi125 conmon[24927]: :03.195840+0000 osd.5 (osd.5) 40 : cluster [DBG] 2.2 scrub ok 2021-03-24T21:55:04.358 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:04 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:55:04.358 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:04 smithi125 conmon[24927]: -24T21:55:04.106180+0000 mon.a (mon.0) 2021-03-24T21:55:04.358 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:04 smithi125 conmon[24927]: 692 : cluster [DBG] osdmap e135: 8 total, 8 up, 6 in 2021-03-24T21:55:04.359 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:04 smithi125 conmon[19936]: cluster 2021-03-24T21:55:02. 2021-03-24T21:55:04.359 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:04 smithi125 conmon[19936]: 719042+0000 mgr.y (mgr.14144) 324 : cluster [DBG] pgmap v362: 148 pgs: 148 active+clean; 144 MiB data, 350 MiB used, 536 GiB / 536 GiB avail; 6.7 MiB/s rd, 3.3 MiB/s wr, 17 op/s 2021-03-24T21:55:04.360 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:04 smithi125 conmon[19936]: cluster 2021-03-24T21:55:03.195840 2021-03-24T21:55:04.360 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:04 smithi125 conmon[19936]: +0000 osd.5 (osd.5) 40 : cluster [DBG] 2.2 scrub ok 2021-03-24T21:55:04.360 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:04 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:55:04.361 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:04 smithi125 conmon[19936]: 24T21:55:04.106180 2021-03-24T21:55:04.361 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:04 smithi125 conmon[19936]: +0000 mon.a (mon.0 2021-03-24T21:55:04.361 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:04 smithi125 conmon[19936]: ) 692 : cluster [DBG] 2021-03-24T21:55:04.362 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:04 smithi125 conmon[19936]: osdmap e135: 8 total, 8 up, 6 in 2021-03-24T21:55:04.372 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:04.427 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:04.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:04.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:04.533 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:04.589 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:04.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:04.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:55:04.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:04 smithi142 conmon[20125]: cluster 2021-03-24T21:55 2021-03-24T21:55:04.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:04 smithi142 conmon[20125]: :02.719042+0000 mgr.y (mgr.14144) 324 : cluster [DBG] pgmap v362: 148 pgs: 148 active+clean; 144 MiB data, 350 MiB used, 536 GiB / 536 GiB avail; 6.7 MiB/s rd, 3.3 MiB/s wr, 17 op/s 2021-03-24T21:55:04.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:04 smithi142 conmon[20125]: cluster 2021-03-24T21 2021-03-24T21:55:04.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:04 smithi142 conmon[20125]: :55:03.195840+0000 osd.5 (osd.5) 40 : cluster [DBG] 2021-03-24T21:55:04.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:04 smithi142 conmon[20125]: 2.2 scrub ok 2021-03-24T21:55:04.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:04 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:55:04.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:04 smithi142 conmon[20125]: 03-24T21:55:04.106180+0000 mon.a (mon. 2021-03-24T21:55:04.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:04 smithi142 conmon[20125]: 0) 692 : cluster [DBG] osdmap e135: 8 total, 8 up, 6 in 2021-03-24T21:55:04.695 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:04.755 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:04.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:04.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:55:04.826 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:55:04.862 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:04.916 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:04.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:04.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:05.022 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:05.083 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:05.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:05.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:55:05.090 INFO:tasks.rados.rados.0.smithi125.stdout:183: done (1 left) 2021-03-24T21:55:05.090 INFO:tasks.rados.rados.0.smithi125.stdout:184: done (0 left) 2021-03-24T21:55:05.090 INFO:tasks.rados.rados.0.smithi125.stdout:185: copy_from oid 19 from oid 22 current snap is 18 2021-03-24T21:55:05.091 INFO:tasks.rados.rados.0.smithi125.stdout:186: read oid 21 snap 13 2021-03-24T21:55:05.091 INFO:tasks.rados.rados.0.smithi125.stdout:186: expect (ObjNum 83 snap 10 seq_num 83) 2021-03-24T21:55:05.091 INFO:tasks.rados.rados.0.smithi125.stdout:187: write oid 49 current snap is 18 2021-03-24T21:55:05.091 INFO:tasks.rados.rados.0.smithi125.stdout:187: seq_num 96 ranges {413472=620140,1818036=510700,3120019=493453} 2021-03-24T21:55:05.095 INFO:tasks.rados.rados.0.smithi125.stdout:187: writing smithi12556470-49 from 413472 to 1033612 tid 1 2021-03-24T21:55:05.104 INFO:tasks.rados.rados.0.smithi125.stdout:187: writing smithi12556470-49 from 1818036 to 2328736 tid 2 2021-03-24T21:55:05.112 INFO:tasks.rados.rados.0.smithi125.stdout:187: writing smithi12556470-49 from 3120019 to 3613472 tid 3 2021-03-24T21:55:05.113 INFO:tasks.rados.rados.0.smithi125.stdout:188: read oid 24 snap -1 2021-03-24T21:55:05.113 INFO:tasks.rados.rados.0.smithi125.stdout:188: expect (ObjNum 89 snap 11 seq_num 89) 2021-03-24T21:55:05.113 INFO:tasks.rados.rados.0.smithi125.stdout:189: delete oid 43 current snap is 18 2021-03-24T21:55:05.113 INFO:tasks.rados.rados.0.smithi125.stdout:187: finishing write tid 1 to smithi12556470-49 2021-03-24T21:55:05.113 INFO:tasks.rados.rados.0.smithi125.stdout:186: expect (ObjNum 83 snap 10 seq_num 83) 2021-03-24T21:55:05.190 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:05.192 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-24T21:55:05.192 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T21:55:05.248 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:05.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:05.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:55:05.354 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:05.375 INFO:tasks.rados.rados.0.smithi125.stdout:187: finishing write tid 2 to smithi12556470-49 2021-03-24T21:55:05.376 INFO:tasks.rados.rados.0.smithi125.stdout:186: done (4 left) 2021-03-24T21:55:05.376 INFO:tasks.rados.rados.0.smithi125.stdout:189: done (3 left) 2021-03-24T21:55:05.376 INFO:tasks.rados.rados.0.smithi125.stdout:190: copy_from oid 33 from oid 13 current snap is 18 2021-03-24T21:55:05.376 INFO:tasks.rados.rados.0.smithi125.stdout:188: expect (ObjNum 89 snap 11 seq_num 89) 2021-03-24T21:55:05.408 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:05.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:05.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:55:05.455 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:05 smithi125 conmon[19936]: audit 2021-03-24T21:55:04.751560+0000 mon.a (mon. 2021-03-24T21:55:05.455 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:05 smithi125 conmon[19936]: 0) 693 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T21:55:05.456 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:05 smithi125 conmon[19936]: audit 2021-03-24T21:55:04.753999+0000 mon.a (mon 2021-03-24T21:55:05.456 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:05 smithi125 conmon[19936]: .0) 694 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T21:55:05.456 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:05 smithi125 conmon[19936]: audit 2021-03-24T21:55:05.456 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:05 smithi125 conmon[19936]: 2021-03-24T21:55:05.046295+0000 mon.a (mon.0) 695 : 2021-03-24T21:55:05.457 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:05 smithi125 conmon[19936]: audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 2]}]: dispatch 2021-03-24T21:55:05.457 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:05 smithi125 conmon[19936]: audit 2021-03-24T21:55:05 2021-03-24T21:55:05.457 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:05 smithi125 conmon[19936]: .087086+0000 mon.a (mon.0) 696 : audit 2021-03-24T21:55:05.457 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:05 smithi125 conmon[19936]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 2]}]': finished 2021-03-24T21:55:05.458 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:05 smithi125 conmon[19936]: cluster 2021-03-24T21:55:05. 2021-03-24T21:55:05.458 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:05 smithi125 conmon[19936]: 087646+0000 mon.a (mon.0) 697 : cluster [DBG] 2021-03-24T21:55:05.458 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:05 smithi125 conmon[19936]: osdmap e136: 8 total, 8 up, 6 in 2021-03-24T21:55:05.459 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:05 smithi125 conmon[24927]: audit 2021-03-24T21:55 2021-03-24T21:55:05.459 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:05 smithi125 conmon[24927]: :04.751560+0000 mon.a (mon.0) 693 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T21:55:05.459 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:05 smithi125 conmon[24927]: audit 2021 2021-03-24T21:55:05.459 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:05 smithi125 conmon[24927]: -03-24T21:55 2021-03-24T21:55:05.460 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:05 smithi125 conmon[24927]: :04.753999+0000 mon.a (mon. 2021-03-24T21:55:05.460 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:05 smithi125 conmon[24927]: 0) 694 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T21:55:05.460 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:05 smithi125 conmon[24927]: audit 2021- 2021-03-24T21:55:05.460 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:05 smithi125 conmon[24927]: 03-24T21:55:05.046295+0000 2021-03-24T21:55:05.461 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:05 smithi125 conmon[24927]: mon.a (mon.0) 695 : audit [INF] 2021-03-24T21:55:05.461 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:05 smithi125 conmon[24927]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 2]}]: dispatch 2021-03-24T21:55:05.461 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:05 smithi125 conmon[24927]: audit 2021-03-24T 2021-03-24T21:55:05.461 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:05 smithi125 conmon[24927]: 21:55:05.087086+0000 mon.a 2021-03-24T21:55:05.462 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:05 smithi125 conmon[24927]: (mon.0) 696 : audit 2021-03-24T21:55:05.462 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:05 smithi125 conmon[24927]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 2]}]': finished 2021-03-24T21:55:05.462 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:05 smithi125 conmon[24927]: cluster 2021-03-24T21:55:05.087646+0000 mon.a 2021-03-24T21:55:05.462 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:05 smithi125 conmon[24927]: (mon.0) 697 : cluster [DBG] 2021-03-24T21:55:05.462 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:05 smithi125 conmon[24927]: osdmap e136: 8 total, 8 up, 6 in 2021-03-24T21:55:05.512 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:05.534 INFO:tasks.rados.rados.0.smithi125.stdout:185: finishing copy_from to smithi12556470-19 2021-03-24T21:55:05.535 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 19 v 40 (ObjNum 81 snap 9 seq_num 81) dirty exists 2021-03-24T21:55:05.535 INFO:tasks.rados.rados.0.smithi125.stdout:185: finishing copy_from racing read to smithi12556470-19 2021-03-24T21:55:05.535 INFO:tasks.rados.rados.0.smithi125.stdout:187: finishing write tid 3 to smithi12556470-49 2021-03-24T21:55:05.536 INFO:tasks.rados.rados.0.smithi125.stdout:187: finishing write tid 4 to smithi12556470-49 2021-03-24T21:55:05.536 INFO:tasks.rados.rados.0.smithi125.stdout:187: finishing write tid 5 to smithi12556470-49 2021-03-24T21:55:05.536 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 49 v 58 (ObjNum 96 snap 18 seq_num 96) dirty exists 2021-03-24T21:55:05.536 INFO:tasks.rados.rados.0.smithi125.stdout:187: left oid 49 (ObjNum 96 snap 18 seq_num 96) 2021-03-24T21:55:05.537 INFO:tasks.rados.rados.0.smithi125.stdout:185: done (3 left) 2021-03-24T21:55:05.537 INFO:tasks.rados.rados.0.smithi125.stdout:187: done (2 left) 2021-03-24T21:55:05.537 INFO:tasks.rados.rados.0.smithi125.stdout:188: done (1 left) 2021-03-24T21:55:05.537 INFO:tasks.rados.rados.0.smithi125.stdout:191: snap_create 2021-03-24T21:55:05.538 INFO:tasks.rados.rados.0.smithi125.stdout:190: finishing copy_from to smithi12556470-33 2021-03-24T21:55:05.538 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 33 v 32 (ObjNum 63 snap 4 seq_num 63) dirty exists 2021-03-24T21:55:05.538 INFO:tasks.rados.rados.0.smithi125.stdout:190: finishing copy_from racing read to smithi12556470-33 2021-03-24T21:55:05.570 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:05.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:05.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:05.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:05 smithi142 conmon[20125]: audit 2021-03-24T21:55: 2021-03-24T21:55:05.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:05 smithi142 conmon[20125]: 04.751560+0000 mon.a (mon.0) 693 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T21:55:05.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:05 smithi142 conmon[20125]: audit 2021-03 2021-03-24T21:55:05.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:05 smithi142 conmon[20125]: -24T21:55:04.753999+0000 mon.a (mon 2021-03-24T21:55:05.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:05 smithi142 conmon[20125]: .0) 694 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T21:55:05.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:05 smithi142 conmon[20125]: 2021-03-24T21:55:05.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:05 smithi142 conmon[20125]: audit 2021-03-24T21: 2021-03-24T21:55:05.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:05 smithi142 conmon[20125]: 55:05.046295+0000 mon.a (mon.0) 2021-03-24T21:55:05.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:05 smithi142 conmon[20125]: 695 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 2]}]: dispatch 2021-03-24T21:55:05.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:05 smithi142 conmon[20125]: audit 2021-03-24T21 2021-03-24T21:55:05.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:05 smithi142 conmon[20125]: :55:05.087086+0000 mon.a (mon.0 2021-03-24T21:55:05.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:05 smithi142 conmon[20125]: ) 696 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 2]}]': finished 2021-03-24T21:55:05.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:05 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:55:05.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:05 smithi142 conmon[20125]: 55:05.087646+0000 mon.a (mon.0) 697 2021-03-24T21:55:05.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:05 smithi142 conmon[20125]: : cluster [DBG] osdmap e136: 8 total, 8 up, 6 in 2021-03-24T21:55:05.676 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:05.734 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:05.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:05.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:55:05.840 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:05.896 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:05.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:05.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:55:06.001 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:06.056 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:06.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:06.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:55:06.106 INFO:tasks.rados.rados.0.smithi125.stdout:190: done (1 left) 2021-03-24T21:55:06.107 INFO:tasks.rados.rados.0.smithi125.stdout:191: done (0 left) 2021-03-24T21:55:06.107 INFO:tasks.rados.rados.0.smithi125.stdout:192: copy_from oid 41 from oid 40 current snap is 19 2021-03-24T21:55:06.107 INFO:tasks.rados.rados.0.smithi125.stdout:193: copy_from oid 6 from oid 47 current snap is 19 2021-03-24T21:55:06.108 INFO:tasks.rados.rados.0.smithi125.stdout:194: read oid 3 snap -1 2021-03-24T21:55:06.108 INFO:tasks.rados.rados.0.smithi125.stdout:194: expect (ObjNum 61 snap 4 seq_num 61) 2021-03-24T21:55:06.108 INFO:tasks.rados.rados.0.smithi125.stdout:195: rollback oid 48 current snap is 19 2021-03-24T21:55:06.108 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 48 to 18 2021-03-24T21:55:06.109 INFO:tasks.rados.rados.0.smithi125.stdout:196: write (excl) oid 45 current snap is 19 2021-03-24T21:55:06.109 INFO:tasks.rados.rados.0.smithi125.stdout:196: seq_num 98 ranges {447086=770638,1935000=798016,3458512=188574} 2021-03-24T21:55:06.114 INFO:tasks.rados.rados.0.smithi125.stdout:196: writing smithi12556470-45 from 447086 to 1217724 tid 1 2021-03-24T21:55:06.125 INFO:tasks.rados.rados.0.smithi125.stdout:196: writing smithi12556470-45 from 1935000 to 2733016 tid 2 2021-03-24T21:55:06.130 INFO:tasks.rados.rados.0.smithi125.stdout:196: writing smithi12556470-45 from 3458512 to 3647086 tid 3 2021-03-24T21:55:06.130 INFO:tasks.rados.rados.0.smithi125.stdout:197: read oid 50 snap -1 2021-03-24T21:55:06.131 INFO:tasks.rados.rados.0.smithi125.stdout:197: expect (ObjNum 51 snap 0 seq_num 51) 2021-03-24T21:55:06.131 INFO:tasks.rados.rados.0.smithi125.stdout:198: write (excl) oid 44 current snap is 19 2021-03-24T21:55:06.131 INFO:tasks.rados.rados.0.smithi125.stdout:198: seq_num 99 ranges {463893=562240,1593482=658027,2685935=559341} 2021-03-24T21:55:06.136 INFO:tasks.rados.rados.0.smithi125.stdout:198: writing smithi12556470-44 from 463893 to 1026133 tid 1 2021-03-24T21:55:06.143 INFO:tasks.rados.rados.0.smithi125.stdout:198: writing smithi12556470-44 from 1593482 to 2251509 tid 2 2021-03-24T21:55:06.149 INFO:tasks.rados.rados.0.smithi125.stdout:198: writing smithi12556470-44 from 2685935 to 3245276 tid 3 2021-03-24T21:55:06.149 INFO:tasks.rados.rados.0.smithi125.stdout:199: copy_from oid 9 from oid 30 current snap is 19 2021-03-24T21:55:06.150 INFO:tasks.rados.rados.0.smithi125.stdout:195: finishing rollback tid 0 to smithi12556470-48 2021-03-24T21:55:06.150 INFO:tasks.rados.rados.0.smithi125.stdout:195: finishing rollback tid 1 to smithi12556470-48 2021-03-24T21:55:06.150 INFO:tasks.rados.rados.0.smithi125.stdout:194: expect (ObjNum 61 snap 4 seq_num 61) 2021-03-24T21:55:06.164 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:06.217 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:06.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:06.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:55:06.256 INFO:tasks.rados.rados.0.smithi125.stdout:196: finishing write tid 1 to smithi12556470-45 2021-03-24T21:55:06.256 INFO:tasks.rados.rados.0.smithi125.stdout:195: finishing rollback tid 2 to smithi12556470-48 2021-03-24T21:55:06.256 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 48 v 62 (ObjNum 24 snap 0 seq_num 24) dirty exists 2021-03-24T21:55:06.257 INFO:tasks.rados.rados.0.smithi125.stdout:197: expect (ObjNum 51 snap 0 seq_num 51) 2021-03-24T21:55:06.325 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:06.329 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:06 smithi125 conmon[19936]: cluster 2021-03-24T21:55: 2021-03-24T21:55:06.329 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:06 smithi125 conmon[19936]: 04.719888+0000 mgr.y (mgr.14144) 325 : cluster [DBG] pgmap v365: 148 pgs: 148 active+clean; 138 MiB data, 353 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 2.2 MiB/s wr, 6 op/s 2021-03-24T21:55:06.330 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:06 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:55:06.330 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:06 smithi125 conmon[19936]: -03-24T21:55:06.104974+0000 mon.a (mon.0) 698 : cluster [DBG] osdmap e137: 8 total, 8 up, 6 in 2021-03-24T21:55:06.371 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:06.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:06.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:55:06.394 INFO:tasks.rados.rados.0.smithi125.stdout:196: finishing write tid 2 to smithi12556470-45 2021-03-24T21:55:06.394 INFO:tasks.rados.rados.0.smithi125.stdout:196: finishing write tid 3 to smithi12556470-45 2021-03-24T21:55:06.394 INFO:tasks.rados.rados.0.smithi125.stdout:196: finishing write tid 4 to smithi12556470-45 2021-03-24T21:55:06.394 INFO:tasks.rados.rados.0.smithi125.stdout:196: finishing write tid 5 to smithi12556470-45 2021-03-24T21:55:06.395 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 45 v 67 (ObjNum 98 snap 19 seq_num 98) dirty exists 2021-03-24T21:55:06.395 INFO:tasks.rados.rados.0.smithi125.stdout:196: left oid 45 (ObjNum 98 snap 19 seq_num 98) 2021-03-24T21:55:06.395 INFO:tasks.rados.rados.0.smithi125.stdout:192: finishing copy_from to smithi12556470-41 2021-03-24T21:55:06.395 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 41 v 15 (ObjNum 39 snap 0 seq_num 39) dirty exists 2021-03-24T21:55:06.396 INFO:tasks.rados.rados.0.smithi125.stdout:194: done (7 left) 2021-03-24T21:55:06.396 INFO:tasks.rados.rados.0.smithi125.stdout:195: done (6 left) 2021-03-24T21:55:06.396 INFO:tasks.rados.rados.0.smithi125.stdout:196: done (5 left) 2021-03-24T21:55:06.396 INFO:tasks.rados.rados.0.smithi125.stdout:197: done (4 left) 2021-03-24T21:55:06.396 INFO:tasks.rados.rados.0.smithi125.stdout:200: write (excl) oid 48 current snap is 19 2021-03-24T21:55:06.397 INFO:tasks.rados.rados.0.smithi125.stdout:192: finishing copy_from racing read to smithi12556470-41 2021-03-24T21:55:06.397 INFO:tasks.rados.rados.0.smithi125.stdout:198: finishing write tid 1 to smithi12556470-44 2021-03-24T21:55:06.397 INFO:tasks.rados.rados.0.smithi125.stdout:193: finishing copy_from to smithi12556470-6 2021-03-24T21:55:06.397 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 6 v 24 (ObjNum 62 snap 4 seq_num 62) dirty exists 2021-03-24T21:55:06.397 INFO:tasks.rados.rados.0.smithi125.stdout:193: finishing copy_from racing read to smithi12556470-6 2021-03-24T21:55:06.398 INFO:tasks.rados.rados.0.smithi125.stdout:198: finishing write tid 2 to smithi12556470-44 2021-03-24T21:55:06.398 INFO:tasks.rados.rados.0.smithi125.stdout:198: finishing write tid 3 to smithi12556470-44 2021-03-24T21:55:06.398 INFO:tasks.rados.rados.0.smithi125.stdout:198: finishing write tid 4 to smithi12556470-44 2021-03-24T21:55:06.399 INFO:tasks.rados.rados.0.smithi125.stdout:198: finishing write tid 5 to smithi12556470-44 2021-03-24T21:55:06.399 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 44 v 72 (ObjNum 99 snap 19 seq_num 99) dirty exists 2021-03-24T21:55:06.399 INFO:tasks.rados.rados.0.smithi125.stdout:198: left oid 44 (ObjNum 99 snap 19 seq_num 99) 2021-03-24T21:55:06.399 INFO:tasks.rados.rados.0.smithi125.stdout:199: finishing copy_from to smithi12556470-9 2021-03-24T21:55:06.399 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 9 v 21 (ObjNum 68 snap 7 seq_num 68) dirty exists 2021-03-24T21:55:06.400 INFO:tasks.rados.rados.0.smithi125.stdout:199: finishing copy_from racing read to smithi12556470-9 2021-03-24T21:55:06.400 INFO:tasks.rados.rados.0.smithi125.stdout:200: seq_num 100 ranges {480700=637489,1651964=401685,2713358=770613} 2021-03-24T21:55:06.405 INFO:tasks.rados.rados.0.smithi125.stdout:200: writing smithi12556470-48 from 480700 to 1118189 tid 1 2021-03-24T21:55:06.410 INFO:tasks.rados.rados.0.smithi125.stdout:200: writing smithi12556470-48 from 1651964 to 2053649 tid 2 2021-03-24T21:55:06.418 INFO:tasks.rados.rados.0.smithi125.stdout:200: writing smithi12556470-48 from 2713358 to 3483971 tid 3 2021-03-24T21:55:06.419 INFO:tasks.rados.rados.0.smithi125.stdout:192: done (4 left) 2021-03-24T21:55:06.419 INFO:tasks.rados.rados.0.smithi125.stdout:193: done (3 left) 2021-03-24T21:55:06.419 INFO:tasks.rados.rados.0.smithi125.stdout:198: done (2 left) 2021-03-24T21:55:06.419 INFO:tasks.rados.rados.0.smithi125.stdout:199: done (1 left) 2021-03-24T21:55:06.420 INFO:tasks.rados.rados.0.smithi125.stdout:201: snap_create 2021-03-24T21:55:06.420 INFO:tasks.rados.rados.0.smithi125.stdout:200: finishing write tid 1 to smithi12556470-48 2021-03-24T21:55:06.420 INFO:tasks.rados.rados.0.smithi125.stdout:200: finishing write tid 2 to smithi12556470-48 2021-03-24T21:55:06.425 INFO:tasks.rados.rados.0.smithi125.stdout:200: finishing write tid 3 to smithi12556470-48 2021-03-24T21:55:06.425 INFO:tasks.rados.rados.0.smithi125.stdout:200: finishing write tid 4 to smithi12556470-48 2021-03-24T21:55:06.426 INFO:tasks.rados.rados.0.smithi125.stdout:200: finishing write tid 5 to smithi12556470-48 2021-03-24T21:55:06.426 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 48 v 66 (ObjNum 100 snap 19 seq_num 100) dirty exists 2021-03-24T21:55:06.426 INFO:tasks.rados.rados.0.smithi125.stdout:200: left oid 48 (ObjNum 100 snap 19 seq_num 100) 2021-03-24T21:55:06.476 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:06.509 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-24T21:55:06.527 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:06.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:06.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:55:06.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:06 smithi142 conmon[20125]: cluster 2021-03-24T21:55:04.719888+0000 mgr.y (mgr.14144) 325 : 2021-03-24T21:55:06.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:06 smithi142 conmon[20125]: cluster [DBG] pgmap v365: 148 pgs: 148 active+clean; 138 MiB data, 353 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 2.2 MiB/s wr, 6 op/s 2021-03-24T21:55:06.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:06 smithi142 conmon[20125]: cluster 2021-03-24T21:55:06.104974+0000 mon.a (mon.0 2021-03-24T21:55:06.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:06 smithi142 conmon[20125]: ) 698 : cluster [DBG] osdmap e137: 8 total, 8 up, 6 in 2021-03-24T21:55:06.635 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:06.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:06 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:55:06.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:06 smithi125 conmon[24927]: 55:04.719888+0000 mgr.y (mgr.14144) 325 : cluster [DBG] pgmap v365: 148 pgs: 148 active+clean; 138 MiB data, 353 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 2.2 MiB/s wr, 6 op/s 2021-03-24T21:55:06.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:06 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:55:06.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:06 smithi125 conmon[24927]: 03-24T21:55:06.104974 2021-03-24T21:55:06.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:06 smithi125 conmon[24927]: +0000 mon.a (mon.0) 698 : cluster [DBG] 2021-03-24T21:55:06.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:06 smithi125 conmon[24927]: osdmap e137: 8 total, 8 up, 6 in 2021-03-24T21:55:06.689 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:06.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:06.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:55:06.795 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:06.854 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:06.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:06.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:55:06.962 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:07.025 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:07.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:07.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:07.131 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:07.188 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:07.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:07.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:55:07.209 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.2 dump_historic_ops 2021-03-24T21:55:07.294 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:07.345 INFO:tasks.rados.rados.0.smithi125.stdout:200: done (1 left) 2021-03-24T21:55:07.345 INFO:tasks.rados.rados.0.smithi125.stdout:201: done (0 left) 2021-03-24T21:55:07.346 INFO:tasks.rados.rados.0.smithi125.stdout:202: write oid 27 current snap is 20 2021-03-24T21:55:07.346 INFO:tasks.rados.rados.0.smithi125.stdout:202: seq_num 101 ranges {497507=712738,1710446=661696,3140781=556726} 2021-03-24T21:55:07.351 INFO:tasks.rados.rados.0.smithi125.stdout:202: writing smithi12556470-27 from 497507 to 1210245 tid 1 2021-03-24T21:55:07.358 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:07.359 INFO:tasks.rados.rados.0.smithi125.stdout:202: writing smithi12556470-27 from 1710446 to 2372142 tid 2 2021-03-24T21:55:07.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:07.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:55:07.366 INFO:tasks.rados.rados.0.smithi125.stdout:202: writing smithi12556470-27 from 3140781 to 3697507 tid 3 2021-03-24T21:55:07.366 INFO:tasks.rados.rados.0.smithi125.stdout:203: write (excl) oid 20 current snap is 20 2021-03-24T21:55:07.366 INFO:tasks.rados.rados.0.smithi125.stdout:202: finishing write tid 1 to smithi12556470-27 2021-03-24T21:55:07.367 INFO:tasks.rados.rados.0.smithi125.stdout:203: seq_num 102 ranges {514314=787987,1768928=405354,2768204=509510} 2021-03-24T21:55:07.373 INFO:tasks.rados.rados.0.smithi125.stdout:203: writing smithi12556470-20 from 514314 to 1302301 tid 1 2021-03-24T21:55:07.379 INFO:tasks.rados.rados.0.smithi125.stdout:203: writing smithi12556470-20 from 1768928 to 2174282 tid 2 2021-03-24T21:55:07.385 INFO:tasks.rados.rados.0.smithi125.stdout:203: writing smithi12556470-20 from 2768204 to 3277714 tid 3 2021-03-24T21:55:07.385 INFO:tasks.rados.rados.0.smithi125.stdout:204: write (excl) oid 5 current snap is 20 2021-03-24T21:55:07.386 INFO:tasks.rados.rados.0.smithi125.stdout:202: finishing write tid 2 to smithi12556470-27 2021-03-24T21:55:07.386 INFO:tasks.rados.rados.0.smithi125.stdout:204: seq_num 103 ranges {531121=463236,1711057=665365,3079274=651847} 2021-03-24T21:55:07.391 INFO:tasks.rados.rados.0.smithi125.stdout:204: writing smithi12556470-5 from 531121 to 994357 tid 1 2021-03-24T21:55:07.399 INFO:tasks.rados.rados.0.smithi125.stdout:204: writing smithi12556470-5 from 1711057 to 2376422 tid 2 2021-03-24T21:55:07.407 INFO:tasks.rados.rados.0.smithi125.stdout:204: writing smithi12556470-5 from 3079274 to 3731121 tid 3 2021-03-24T21:55:07.408 INFO:tasks.rados.rados.0.smithi125.stdout:205: delete oid 49 current snap is 20 2021-03-24T21:55:07.408 INFO:tasks.rados.rados.0.smithi125.stdout:202: finishing write tid 3 to smithi12556470-27 2021-03-24T21:55:07.408 INFO:tasks.rados.rados.0.smithi125.stdout:202: finishing write tid 4 to smithi12556470-27 2021-03-24T21:55:07.408 INFO:tasks.rados.rados.0.smithi125.stdout:202: finishing write tid 5 to smithi12556470-27 2021-03-24T21:55:07.409 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 27 v 49 (ObjNum 101 snap 20 seq_num 101) dirty exists 2021-03-24T21:55:07.409 INFO:tasks.rados.rados.0.smithi125.stdout:202: left oid 27 (ObjNum 101 snap 20 seq_num 101) 2021-03-24T21:55:07.409 INFO:tasks.rados.rados.0.smithi125.stdout:203: finishing write tid 1 to smithi12556470-20 2021-03-24T21:55:07.409 INFO:tasks.rados.rados.0.smithi125.stdout:203: finishing write tid 2 to smithi12556470-20 2021-03-24T21:55:07.410 INFO:tasks.rados.rados.0.smithi125.stdout:203: finishing write tid 3 to smithi12556470-20 2021-03-24T21:55:07.410 INFO:tasks.rados.rados.0.smithi125.stdout:203: finishing write tid 4 to smithi12556470-20 2021-03-24T21:55:07.410 INFO:tasks.rados.rados.0.smithi125.stdout:203: finishing write tid 5 to smithi12556470-20 2021-03-24T21:55:07.410 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 20 v 26 (ObjNum 102 snap 20 seq_num 102) dirty exists 2021-03-24T21:55:07.411 INFO:tasks.rados.rados.0.smithi125.stdout:203: left oid 20 (ObjNum 102 snap 20 seq_num 102) 2021-03-24T21:55:07.411 INFO:tasks.rados.rados.0.smithi125.stdout:204: finishing write tid 1 to smithi12556470-5 2021-03-24T21:55:07.411 INFO:tasks.rados.rados.0.smithi125.stdout:204: finishing write tid 2 to smithi12556470-5 2021-03-24T21:55:07.412 INFO:tasks.rados.rados.0.smithi125.stdout:202: done (3 left) 2021-03-24T21:55:07.412 INFO:tasks.rados.rados.0.smithi125.stdout:203: done (2 left) 2021-03-24T21:55:07.412 INFO:tasks.rados.rados.0.smithi125.stdout:205: done (1 left) 2021-03-24T21:55:07.412 INFO:tasks.rados.rados.0.smithi125.stdout:206: snap_remove snap 14 2021-03-24T21:55:07.464 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:07.515 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:07.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:07.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:55:07.622 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:07.676 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:07.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:07.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:55:07.781 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:07.835 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:07.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:07.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:55:07.941 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:07.991 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:07.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:07.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:55:08.095 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:08.133 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:55:08.156 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:55:08.160 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:08.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:08.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:55:08.265 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:08.311 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:08.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:08.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:55:08.345 INFO:tasks.rados.rados.0.smithi125.stdout:206: done (1 left) 2021-03-24T21:55:08.346 INFO:tasks.rados.rados.0.smithi125.stdout:207: snap_remove snap 18 2021-03-24T21:55:08.416 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:08.428 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:08 smithi125 conmon[19936]: cluster 2021-03-24T21:55:08.428 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:08 smithi125 conmon[19936]: 2021-03-24T21:55:06.720580+0000 2021-03-24T21:55:08.428 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:08 smithi125 conmon[19936]: mgr.y (mgr.14144) 326 : cluster [DBG] pgmap v368: 148 pgs: 148 active+clean; 138 MiB data, 358 MiB used, 536 GiB / 536 GiB avail; 3.2 MiB/s rd, 1.9 MiB/s wr, 7 op/s 2021-03-24T21:55:08.428 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:08 smithi125 conmon[19936]: cluster 2021-03-24T21:55:08.429 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:08 smithi125 conmon[19936]: 2021-03-24T21:55:07.340387+0000 mon.a (mon.0) 699 : cluster [DBG] osdmap e138: 8 total, 8 up, 6 in 2021-03-24T21:55:08.429 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:08 smithi125 conmon[24927]: cluster 2021-03-24T21:55:08.429 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:08 smithi125 conmon[24927]: 2021-03-24T21:55:08.430 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:08 smithi125 conmon[24927]: 2021 2021-03-24T21:55:08.430 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:08 smithi125 conmon[24927]: - 2021-03-24T21:55:08.430 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:08 smithi125 conmon[24927]: 03 2021-03-24T21:55:08.430 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:08 smithi125 conmon[24927]: - 2021-03-24T21:55:08.431 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:08 smithi125 conmon[24927]: 24 2021-03-24T21:55:08.431 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:08 smithi125 conmon[24927]: T 2021-03-24T21:55:08.431 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:08 smithi125 conmon[24927]: 21 2021-03-24T21:55:08.431 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:08 smithi125 conmon[24927]: : 2021-03-24T21:55:08.431 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:08 smithi125 conmon[24927]: 55 2021-03-24T21:55:08.432 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:08 smithi125 conmon[24927]: : 2021-03-24T21:55:08.432 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:08 smithi125 conmon[24927]: 06 2021-03-24T21:55:08.432 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:08 smithi125 conmon[24927]: . 2021-03-24T21:55:08.432 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:08 smithi125 conmon[24927]: 720580 2021-03-24T21:55:08.433 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:08 smithi125 conmon[24927]: +0000 2021-03-24T21:55:08.433 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:08 smithi125 conmon[24927]: 2021-03-24T21:55:08.433 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:08 smithi125 conmon[24927]: mgr.y 2021-03-24T21:55:08.433 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:08 smithi125 conmon[24927]: ( 2021-03-24T21:55:08.433 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:08 smithi125 conmon[24927]: mgr 2021-03-24T21:55:08.434 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:08 smithi125 conmon[24927]: . 2021-03-24T21:55:08.434 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:08 smithi125 conmon[24927]: 14144 2021-03-24T21:55:08.434 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:08 smithi125 conmon[24927]: ) 2021-03-24T21:55:08.434 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:08 smithi125 conmon[24927]: 326 2021-03-24T21:55:08.435 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:08 smithi125 conmon[24927]: : cluster [DBG] pgmap v368: 148 pgs: 148 active+clean; 138 MiB data, 358 MiB used, 536 GiB / 536 GiB avail; 3.2 MiB/s rd, 1.9 MiB/s wr, 7 op/s 2021-03-24T21:55:08.435 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:08 smithi125 conmon[24927]: cluster 2021-03-24T21:55:08.435 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:08 smithi125 conmon[24927]: 2021-03-24T21:55:07. 2021-03-24T21:55:08.435 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:08 smithi125 conmon[24927]: 340387+0000 mon.a (mon.0) 2021-03-24T21:55:08.435 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:08 smithi125 conmon[24927]: 699 : cluster [DBG] osdmap e138: 8 total, 8 up, 6 in 2021-03-24T21:55:08.461 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T21:55:08.467 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:08.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:08.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:55:08.488 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T21:55:08.507 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T21:55:08.529 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T21:55:08.555 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T21:55:08.573 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:08.584 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T21:55:08.609 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T21:55:08.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:08 smithi142 conmon[20125]: cluster 2021-03-24T21:55:06. 2021-03-24T21:55:08.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:08 smithi142 conmon[20125]: 720580+0000 mgr.y (mgr.14144) 326 : cluster [DBG] pgmap v368: 148 pgs: 148 active+clean; 138 MiB data, 358 MiB used, 536 GiB / 536 GiB avail; 3.2 MiB/s rd, 1.9 MiB/s wr, 7 op/s 2021-03-24T21:55:08.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:08 smithi142 conmon[20125]: cluster 2021-03-24T21:55:07.340387+0000 mon.a (mon.0) 699 : cluster [DBG] osdmap e138: 8 total, 8 up, 6 in 2021-03-24T21:55:08.629 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:08.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:08.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:55:08.636 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T21:55:08.637 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-24T21:55:08.638 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-24T21:55:08.638 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-24T21:55:08.639 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-24T21:55:08.639 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-24T21:55:08.639 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-24T21:55:08.639 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-24T21:55:08.639 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-24T21:55:08.736 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:08.794 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:08.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:08.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:55:08.900 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:08.955 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:08.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:08.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:55:09.061 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:09.122 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:09.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:09.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:55:09.226 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:09.280 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:09.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:09.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:55:09.352 INFO:tasks.rados.rados.0.smithi125.stdout:207: done (1 left) 2021-03-24T21:55:09.352 INFO:tasks.rados.rados.0.smithi125.stdout:208: rollback oid 19 current snap is 20 2021-03-24T21:55:09.353 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 19 to 15 2021-03-24T21:55:09.353 INFO:tasks.rados.rados.0.smithi125.stdout:209: read oid 49 snap -1 2021-03-24T21:55:09.353 INFO:tasks.rados.rados.0.smithi125.stdout:209: expect deleted 2021-03-24T21:55:09.353 INFO:tasks.rados.rados.0.smithi125.stdout:210: write (excl) oid 50 current snap is 20 2021-03-24T21:55:09.353 INFO:tasks.rados.rados.0.smithi125.stdout:210: seq_num 105 ranges {564735=613734,1828021=669034,3134120=630615} 2021-03-24T21:55:09.358 INFO:tasks.rados.rados.0.smithi125.stdout:210: writing smithi12556470-50 from 564735 to 1178469 tid 1 2021-03-24T21:55:09.366 INFO:tasks.rados.rados.0.smithi125.stdout:210: writing smithi12556470-50 from 1828021 to 2497055 tid 2 2021-03-24T21:55:09.370 INFO:teuthology.orchestra.run.smithi125.stderr:set osd.5 primary-affinity to 0.664524 (8435502) 2021-03-24T21:55:09.375 INFO:tasks.rados.rados.0.smithi125.stdout:210: writing smithi12556470-50 from 3134120 to 3764735 tid 3 2021-03-24T21:55:09.375 INFO:tasks.rados.rados.0.smithi125.stdout:211: copy_from oid 43 from oid 29 current snap is 20 2021-03-24T21:55:09.376 INFO:tasks.rados.rados.0.smithi125.stdout:204: finishing write tid 3 to smithi12556470-5 2021-03-24T21:55:09.376 INFO:tasks.rados.rados.0.smithi125.stdout:204: finishing write tid 4 to smithi12556470-5 2021-03-24T21:55:09.376 INFO:tasks.rados.rados.0.smithi125.stdout:204: finishing write tid 5 to smithi12556470-5 2021-03-24T21:55:09.377 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 5 v 48 (ObjNum 103 snap 20 seq_num 103) dirty exists 2021-03-24T21:55:09.377 INFO:tasks.rados.rados.0.smithi125.stdout:204: left oid 5 (ObjNum 103 snap 20 seq_num 103) 2021-03-24T21:55:09.377 INFO:tasks.rados.rados.0.smithi125.stdout:208: finishing rollback tid 0 to smithi12556470-19 2021-03-24T21:55:09.377 INFO:tasks.rados.rados.0.smithi125.stdout:210: finishing write tid 1 to smithi12556470-50 2021-03-24T21:55:09.377 INFO:tasks.rados.rados.0.smithi125.stdout:208: finishing rollback tid 1 to smithi12556470-19 2021-03-24T21:55:09.378 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 19 v 77 (ObjNum 1115288944 snap 21916 seq_num 1116784912) dirty dne 2021-03-24T21:55:09.378 INFO:tasks.rados.rados.0.smithi125.stdout:210: finishing write tid 2 to smithi12556470-50 2021-03-24T21:55:09.378 INFO:tasks.rados.rados.0.smithi125.stdout:204: done (4 left) 2021-03-24T21:55:09.378 INFO:tasks.rados.rados.0.smithi125.stdout:208: done (3 left) 2021-03-24T21:55:09.379 INFO:tasks.rados.rados.0.smithi125.stdout:209: done (2 left) 2021-03-24T21:55:09.379 INFO:tasks.rados.rados.0.smithi125.stdout:212: copy_from oid 37 from oid 27 current snap is 20 2021-03-24T21:55:09.379 INFO:tasks.rados.rados.0.smithi125.stdout:213: snap_create 2021-03-24T21:55:09.380 INFO:tasks.rados.rados.0.smithi125.stdout:210: finishing write tid 3 to smithi12556470-50 2021-03-24T21:55:09.380 INFO:tasks.rados.rados.0.smithi125.stdout:210: finishing write tid 4 to smithi12556470-50 2021-03-24T21:55:09.381 INFO:tasks.rados.rados.0.smithi125.stdout:210: finishing write tid 5 to smithi12556470-50 2021-03-24T21:55:09.381 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 50 v 35 (ObjNum 105 snap 20 seq_num 105) dirty exists 2021-03-24T21:55:09.381 INFO:tasks.rados.rados.0.smithi125.stdout:210: left oid 50 (ObjNum 105 snap 20 seq_num 105) 2021-03-24T21:55:09.386 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:09.389 INFO:tasks.rados.rados.0.smithi125.stdout:211: finishing copy_from to smithi12556470-43 2021-03-24T21:55:09.389 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 43 v 16 (ObjNum 59 snap 4 seq_num 59) dirty exists 2021-03-24T21:55:09.389 INFO:tasks.rados.rados.0.smithi125.stdout:211: finishing copy_from racing read to smithi12556470-43 2021-03-24T21:55:09.397 INFO:tasks.rados.rados.0.smithi125.stdout:212: finishing copy_from to smithi12556470-37 2021-03-24T21:55:09.398 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 37 v 49 (ObjNum 101 snap 20 seq_num 101) dirty exists 2021-03-24T21:55:09.398 INFO:tasks.rados.rados.0.smithi125.stdout:212: finishing copy_from racing read to smithi12556470-37 2021-03-24T21:55:09.436 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:09.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:09.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:55:09.542 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:09.556 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:09 smithi142 conmon[20125]: cluster 2021-03-24T21:55:09.556 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:09 smithi142 conmon[20125]: 2021-03-24T21:55:09.556 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:09 smithi142 conmon[20125]: 2021 2021-03-24T21:55:09.556 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:09 smithi142 conmon[20125]: - 2021-03-24T21:55:09.556 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:09 smithi142 conmon[20125]: 03 2021-03-24T21:55:09.557 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:09 smithi142 conmon[20125]: - 2021-03-24T21:55:09.557 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:09 smithi142 conmon[20125]: 24 2021-03-24T21:55:09.557 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:09 smithi142 conmon[20125]: T 2021-03-24T21:55:09.557 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:09 smithi142 conmon[20125]: 21 2021-03-24T21:55:09.558 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:09 smithi142 conmon[20125]: : 2021-03-24T21:55:09.558 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:09 smithi142 conmon[20125]: 55 2021-03-24T21:55:09.558 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:09 smithi142 conmon[20125]: : 2021-03-24T21:55:09.558 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:09 smithi142 conmon[20125]: 08 2021-03-24T21:55:09.558 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:09 smithi142 conmon[20125]: . 2021-03-24T21:55:09.559 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:09 smithi142 conmon[20125]: 343502 2021-03-24T21:55:09.559 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:09 smithi142 conmon[20125]: +0000 2021-03-24T21:55:09.559 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:09 smithi142 conmon[20125]: 2021-03-24T21:55:09.559 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:09 smithi142 conmon[20125]: mon.a 2021-03-24T21:55:09.559 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:09 smithi142 conmon[20125]: ( 2021-03-24T21:55:09.560 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:09 smithi142 conmon[20125]: mon 2021-03-24T21:55:09.560 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:09 smithi142 conmon[20125]: . 2021-03-24T21:55:09.560 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:09 smithi142 conmon[20125]: 0 2021-03-24T21:55:09.560 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:09 smithi142 conmon[20125]: ) 2021-03-24T21:55:09.561 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:09 smithi142 conmon[20125]: 700 2021-03-24T21:55:09.561 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:09 smithi142 conmon[20125]: : 2021-03-24T21:55:09.561 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:09 smithi142 conmon[20125]: cluster 2021-03-24T21:55:09.561 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:09 smithi142 conmon[20125]: 2021-03-24T21:55:09.562 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:09 smithi142 conmon[20125]: [DBG] 2021-03-24T21:55:09.562 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:09 smithi142 conmon[20125]: 2021-03-24T21:55:09.562 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:09 smithi142 conmon[20125]: osdmap e139: 8 total, 8 up, 6 in 2021-03-24T21:55:09.562 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:09 smithi142 conmon[20125]: 2021-03-24T21:55:09.563 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:09 smithi142 conmon[20125]: audit 2021-03-24T21:55:09.563 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:09 smithi142 conmon[20125]: 2021-03-24T21:55:09.563 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:09 smithi142 conmon[20125]: 2021 2021-03-24T21:55:09.563 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:09 smithi142 conmon[20125]: - 2021-03-24T21:55:09.564 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:09 smithi142 conmon[20125]: 03 2021-03-24T21:55:09.564 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:09 smithi142 conmon[20125]: - 2021-03-24T21:55:09.564 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:09 smithi142 conmon[20125]: 24 2021-03-24T21:55:09.564 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:09 smithi142 conmon[20125]: T 2021-03-24T21:55:09.564 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:09 smithi142 conmon[20125]: 21 2021-03-24T21:55:09.565 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:09 smithi142 conmon[20125]: : 2021-03-24T21:55:09.565 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:09 smithi142 conmon[20125]: 55 2021-03-24T21:55:09.565 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:09 smithi142 conmon[20125]: : 2021-03-24T21:55:09.565 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:09 smithi142 conmon[20125]: 08 2021-03-24T21:55:09.565 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:09 smithi142 conmon[20125]: . 2021-03-24T21:55:09.566 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:09 smithi142 conmon[20125]: 379413+0000 mon.a (mon.0) 2021-03-24T21:55:09.566 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:09 smithi142 conmon[20125]: 701 : audit [INF] from='client.? v1:172.21.15.125:0/3626648473' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.6645241636505669}]: dispatch 2021-03-24T21:55:09.566 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:09 smithi142 conmon[20125]: audit 2021-03-24T21:55:09.566 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:09 smithi142 conmon[20125]: 2021-03-24T21:55:08. 2021-03-24T21:55:09.566 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:09 smithi142 conmon[20125]: 438541+0000 mon.c (mon.2) 28 : audit 2021-03-24T21:55:09.567 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:09 smithi142 conmon[20125]: [DBG] from='client.? v1:172.21.15.125:0/4069657978' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:55:09.600 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:09.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:09.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:55:09.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:09 smithi125 conmon[24927]: cluster 2021-03-24T21:55:08.343502+0000 mon.a (mon.0) 2021-03-24T21:55:09.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:09 smithi125 conmon[24927]: 700 : cluster [DBG] osdmap e139: 8 total, 8 up, 6 in 2021-03-24T21:55:09.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:09 smithi125 conmon[24927]: audit 2021-03-24T21:55:08.379413+0000 mon.a (mon.0) 701 : audit [INF] from='client.? v1:172.21.15.125:0/3626648473' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.6645241636505669}]: dispatch 2021-03-24T21:55:09.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:09 smithi125 conmon[24927]: audit 2021-03-24T21:55:08.438541+0000 mon.c (mon.2 2021-03-24T21:55:09.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:09 smithi125 conmon[24927]: ) 28 : audit [DBG] from='client.? v1:172.21.15.125:0/4069657978' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:55:09.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:09 smithi125 conmon[19936]: cluster 2021-03-24T21:55:08.343502+0000 mon.a (mon 2021-03-24T21:55:09.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:09 smithi125 conmon[19936]: .0) 700 : cluster [DBG] osdmap e139: 8 total, 8 up, 6 in 2021-03-24T21:55:09.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:09 smithi125 conmon[19936]: audit 2021-03-24T21:55:08.379413+0000 mon.a (mon.0) 701 : audit [INF] from='client.? v1:172.21.15.125:0/3626648473' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.6645241636505669}]: dispatch 2021-03-24T21:55:09.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:09 smithi125 conmon[19936]: audit 2021-03-24T21:55:09.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:09 smithi125 conmon[19936]: 2021-03-24T21:55:08.438541+0000 mon.c (mon.2) 28 : audit [DBG] from='client.? v1:172.21.15.125:0/4069657978' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:55:09.706 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:09.762 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:09.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:09.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:55:09.868 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:09.925 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:09.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:09.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:55:10.031 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:10.087 INFO:tasks.rados.rados.0.smithi125.stdout:210: done (3 left) 2021-03-24T21:55:10.088 INFO:tasks.rados.rados.0.smithi125.stdout:211: done (2 left) 2021-03-24T21:55:10.088 INFO:tasks.rados.rados.0.smithi125.stdout:212: done (1 left) 2021-03-24T21:55:10.088 INFO:tasks.rados.rados.0.smithi125.stdout:213: done (0 left) 2021-03-24T21:55:10.088 INFO:tasks.rados.rados.0.smithi125.stdout:214: write oid 24 current snap is 21 2021-03-24T21:55:10.089 INFO:tasks.rados.rados.0.smithi125.stdout:214: seq_num 106 ranges {581542=688983,1886503=412692,2761543=670951} 2021-03-24T21:55:10.092 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:10.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:10.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:55:10.107 INFO:tasks.rados.rados.0.smithi125.stdout:214: writing smithi12556470-24 from 581542 to 1270525 tid 1 2021-03-24T21:55:10.114 INFO:tasks.rados.rados.0.smithi125.stdout:214: writing smithi12556470-24 from 1886503 to 2299195 tid 2 2021-03-24T21:55:10.121 INFO:tasks.rados.rados.0.smithi125.stdout:214: writing smithi12556470-24 from 2761543 to 3432494 tid 3 2021-03-24T21:55:10.121 INFO:tasks.rados.rados.0.smithi125.stdout:215: read oid 34 snap -1 2021-03-24T21:55:10.121 INFO:tasks.rados.rados.0.smithi125.stdout:215: expect deleted 2021-03-24T21:55:10.121 INFO:tasks.rados.rados.0.smithi125.stdout:216: rollback oid 36 current snap is 21 2021-03-24T21:55:10.122 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 36 to 10 2021-03-24T21:55:10.122 INFO:tasks.rados.rados.0.smithi125.stdout:214: finishing write tid 1 to smithi12556470-24 2021-03-24T21:55:10.122 INFO:tasks.rados.rados.0.smithi125.stdout:214: finishing write tid 2 to smithi12556470-24 2021-03-24T21:55:10.122 INFO:tasks.rados.rados.0.smithi125.stdout:217: snap_create 2021-03-24T21:55:10.151 INFO:tasks.rados.rados.0.smithi125.stdout:216: finishing rollback tid 0 to smithi12556470-36 2021-03-24T21:55:10.152 INFO:tasks.rados.rados.0.smithi125.stdout:216: finishing rollback tid 1 to smithi12556470-36 2021-03-24T21:55:10.152 INFO:tasks.rados.rados.0.smithi125.stdout:216: finishing rollback tid 2 to smithi12556470-36 2021-03-24T21:55:10.152 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 36 v 49 (ObjNum 35 snap 0 seq_num 35) dirty exists 2021-03-24T21:55:10.153 INFO:tasks.rados.rados.0.smithi125.stdout:214: finishing write tid 3 to smithi12556470-24 2021-03-24T21:55:10.153 INFO:tasks.rados.rados.0.smithi125.stdout:214: finishing write tid 4 to smithi12556470-24 2021-03-24T21:55:10.154 INFO:tasks.rados.rados.0.smithi125.stdout:214: finishing write tid 5 to smithi12556470-24 2021-03-24T21:55:10.154 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 24 v 71 (ObjNum 106 snap 21 seq_num 106) dirty exists 2021-03-24T21:55:10.154 INFO:tasks.rados.rados.0.smithi125.stdout:214: left oid 24 (ObjNum 106 snap 21 seq_num 106) 2021-03-24T21:55:10.202 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:10.259 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:10.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:10.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:55:10.363 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:10 smithi125 conmon[19936]: cluster 2021-03-24T21:55: 2021-03-24T21:55:10.365 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:10.420 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:10.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:10.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:55:10.527 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:10.590 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:10.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:10.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:10.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:10 smithi142 conmon[20125]: cluster 2021-03-24T21:55:08.721146+0000 mgr.y (mgr.14144) 327 : cluster 2021-03-24T21:55:10.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:10 smithi142 conmon[20125]: [DBG] pgmap v371: 148 pgs: 148 active+clean; 150 MiB data, 381 MiB used, 536 GiB / 536 GiB avail; 5.9 MiB/s rd, 2.4 MiB/s wr, 9 op/s 2021-03-24T21:55:10.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:10 smithi142 conmon[20125]: audit 2021-03-24T21:55:09.347616+0000 mon.a (mon.0) 702 : audit 2021-03-24T21:55:10.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:10 smithi142 conmon[20125]: [INF] from='client.? v1:172.21.15.125:0/3626648473' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.6645241636505669}]': finished 2021-03-24T21:55:10.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:10 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:55:10.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:10 smithi142 conmon[20125]: 03-24T21:55:09.347829+0000 mon.a (mon. 2021-03-24T21:55:10.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:10 smithi142 conmon[20125]: 0) 703 : cluster [DBG] osdmap e140: 8 total, 8 up, 6 in 2021-03-24T21:55:10.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:10 smithi142 conmon[20125]: cluster 2021-03-24T21:55:10.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:10 smithi142 conmon[20125]: 2021-03-24 2021-03-24T21:55:10.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:10 smithi142 conmon[20125]: T21:55:10.070430+0000 osd.0 (osd.0) 43 : 2021-03-24T21:55:10.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:10 smithi142 conmon[20125]: cluster [DBG] purged_snaps scrub starts 2021-03-24T21:55:10.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:10 smithi142 conmon[20125]: cluster 2021-03-24T21:55:10.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:10 smithi142 conmon[20125]: 2021-03- 2021-03-24T21:55:10.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:10 smithi142 conmon[20125]: 24T21:55:10.070595 2021-03-24T21:55:10.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:10 smithi142 conmon[20125]: +0000 osd.0 (osd.0) 44 : cluster 2021-03-24T21:55:10.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:10 smithi142 conmon[20125]: [DBG] purged_snaps scrub ok 2021-03-24T21:55:10.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:10 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:55:10.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:10 smithi142 conmon[20125]: -24T21:55:10. 2021-03-24T21:55:10.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:10 smithi142 conmon[20125]: 086481+0000 mon.a (mon.0) 2021-03-24T21:55:10.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:10 smithi142 conmon[20125]: 704 : cluster [DBG] osdmap e141: 8 total, 8 up, 6 in 2021-03-24T21:55:10.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:10 smithi125 conmon[19936]: 08.721146+0000 mgr.y (mgr.14144) 327 : cluster [DBG] pgmap v371: 148 pgs: 148 active+clean; 150 MiB data, 381 MiB used, 536 GiB / 536 GiB avail; 5.9 MiB/s rd, 2.4 MiB/s wr, 9 op/s 2021-03-24T21:55:10.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:10 smithi125 conmon[19936]: audit 2021-03-24T21:55:10.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:10 smithi125 conmon[19936]: 2021-03-24T21:55:09.347616+0000 2021-03-24T21:55:10.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:10 smithi125 conmon[19936]: mon.a (mon.0) 702 : audit [INF] from='client.? v1:172.21.15.125:0/3626648473' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.6645241636505669}]': finished 2021-03-24T21:55:10.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:10 smithi125 conmon[19936]: cluster 2021-03-24T21:55:09.347829+0000 mon.a (mon.0) 703 : cluster [DBG] osdmap e140: 8 total, 8 up, 6 in 2021-03-24T21:55:10.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:10 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:55:10.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:10 smithi125 conmon[19936]: 55:10.070430+0000 osd.0 (osd.0) 43 2021-03-24T21:55:10.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:10 smithi125 conmon[19936]: : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:55:10.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:10 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:55:10.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:10 smithi125 conmon[19936]: 03-24T21:55:10.070595+0000 osd.0 2021-03-24T21:55:10.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:10 smithi125 conmon[19936]: (osd.0) 44 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:55:10.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:10 smithi125 conmon[19936]: cluster 2021-03-24T21:55:10.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:10 smithi125 conmon[19936]: 2021-03-24T21:55:10.086481+0000 2021-03-24T21:55:10.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:10 smithi125 conmon[19936]: mon.a (mon.0) 704 : cluster [DBG] osdmap e141: 8 total, 8 up, 6 in 2021-03-24T21:55:10.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:10 smithi125 conmon[19936]: 2021-03-24T21:55:10.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:10 smithi125 conmon[24927]: cluster 2021-03-24T21:55 2021-03-24T21:55:10.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:10 smithi125 conmon[24927]: :08.721146+0000 mgr.y (mgr.14144) 327 : cluster [DBG] pgmap v371: 148 pgs: 148 active+clean; 150 MiB data, 381 MiB used, 536 GiB / 536 GiB avail; 5.9 MiB/s rd, 2.4 MiB/s wr, 9 op/s 2021-03-24T21:55:10.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:10 smithi125 conmon[24927]: audit 2021-03-24T21:55: 2021-03-24T21:55:10.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:10 smithi125 conmon[24927]: 09.347616+0000 mon.a (mon.0) 702 : 2021-03-24T21:55:10.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:10 smithi125 conmon[24927]: audit [INF] from='client.? v1:172.21.15.125:0/3626648473' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.6645241636505669}]': finished 2021-03-24T21:55:10.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:10 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:55:10.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:10 smithi125 conmon[24927]: T21:55:09.347829+0000 mon.a (mon. 2021-03-24T21:55:10.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:10 smithi125 conmon[24927]: 0) 703 : cluster [DBG] osdmap e140: 8 total, 8 up, 6 in 2021-03-24T21:55:10.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:10 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:55:10.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:10 smithi125 conmon[24927]: -24T21:55:10.070430+0000 osd.0 ( 2021-03-24T21:55:10.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:10 smithi125 conmon[24927]: osd.0) 43 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:55:10.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:10 smithi125 conmon[24927]: cluster 2021-03-24T21:55:10.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:10 smithi125 conmon[24927]: 2021-03-24T21:55:10.070595+0000 2021-03-24T21:55:10.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:10 smithi125 conmon[24927]: osd.0 (osd.0) 44 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:55:10.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:10 smithi125 conmon[24927]: cluster 2021-03-24T21:55:10.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:10 smithi125 conmon[24927]: 2021-03-24T21:55:10.086481+0000 2021-03-24T21:55:10.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:10 smithi125 conmon[24927]: mon.a (mon.0) 704 : cluster [DBG] osdmap e141: 8 total, 8 up, 6 in 2021-03-24T21:55:10.697 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:10.752 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:10.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:10.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:10.858 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:10.917 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:10.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:10.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:55:11.022 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:11.083 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:11.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:11.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:55:11.103 INFO:tasks.rados.rados.0.smithi125.stdout:214: done (3 left) 2021-03-24T21:55:11.103 INFO:tasks.rados.rados.0.smithi125.stdout:215: done (2 left) 2021-03-24T21:55:11.103 INFO:tasks.rados.rados.0.smithi125.stdout:216: done (1 left) 2021-03-24T21:55:11.103 INFO:tasks.rados.rados.0.smithi125.stdout:217: done (0 left) 2021-03-24T21:55:11.104 INFO:tasks.rados.rados.0.smithi125.stdout:218: snap_remove snap 20 2021-03-24T21:55:11.188 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:11.244 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:11.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:11.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:55:11.350 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:11.407 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:11.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:11.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:55:11.512 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:11.570 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:11.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:11.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:11.676 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:11.731 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:11.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:11.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:11.838 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:11.896 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:11.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:11.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:55:12.001 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:12.056 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:12.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:12.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:12.112 INFO:tasks.rados.rados.0.smithi125.stdout:218: done (0 left) 2021-03-24T21:55:12.113 INFO:tasks.rados.rados.0.smithi125.stdout:219: write oid 20 current snap is 22 2021-03-24T21:55:12.113 INFO:tasks.rados.rados.0.smithi125.stdout:219: seq_num 107 ranges {598349=480585,1544985=672703,2905319=482223} 2021-03-24T21:55:12.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:12 smithi142 conmon[20125]: cluster 2021-03-24T21:55:10.721782+0000 mgr.y (mgr.14144) 328 : cluster 2021-03-24T21:55:12.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:12 smithi142 conmon[20125]: [DBG] pgmap v374: 148 pgs: 148 active+clean; 169 MiB data, 429 MiB used, 536 GiB / 536 GiB avail; 13 MiB/s rd, 6.6 MiB/s wr, 25 op/s 2021-03-24T21:55:12.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:12 smithi142 conmon[20125]: cluster 2021-03-24T21:55:11.101670+0000 mon.a (mon. 2021-03-24T21:55:12.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:12 smithi142 conmon[20125]: 0) 705 : cluster [DBG] osdmap e142: 8 total, 8 up, 6 in 2021-03-24T21:55:12.120 INFO:tasks.rados.rados.0.smithi125.stdout:219: writing smithi12556470-20 from 598349 to 1078934 tid 1 2021-03-24T21:55:12.128 INFO:tasks.rados.rados.0.smithi125.stdout:219: writing smithi12556470-20 from 1544985 to 2217688 tid 2 2021-03-24T21:55:12.137 INFO:tasks.rados.rados.0.smithi125.stdout:219: writing smithi12556470-20 from 2905319 to 3387542 tid 3 2021-03-24T21:55:12.138 INFO:tasks.rados.rados.0.smithi125.stdout:220: write oid 32 current snap is 22 2021-03-24T21:55:12.138 INFO:tasks.rados.rados.0.smithi125.stdout:220: seq_num 108 ranges {615156=555834,1603467=416361,2816389=693495} 2021-03-24T21:55:12.145 INFO:tasks.rados.rados.0.smithi125.stdout:220: writing smithi12556470-32 from 615156 to 1170990 tid 1 2021-03-24T21:55:12.150 INFO:tasks.rados.rados.0.smithi125.stdout:220: writing smithi12556470-32 from 1603467 to 2019828 tid 2 2021-03-24T21:55:12.159 INFO:tasks.rados.rados.0.smithi125.stdout:220: writing smithi12556470-32 from 2816389 to 3509884 tid 3 2021-03-24T21:55:12.159 INFO:tasks.rados.rados.0.smithi125.stdout:221: delete oid 17 current snap is 22 2021-03-24T21:55:12.160 INFO:tasks.rados.rados.0.smithi125.stdout:219: finishing write tid 1 to smithi12556470-20 2021-03-24T21:55:12.160 INFO:tasks.rados.rados.0.smithi125.stdout:219: finishing write tid 2 to smithi12556470-20 2021-03-24T21:55:12.160 INFO:tasks.rados.rados.0.smithi125.stdout:219: finishing write tid 3 to smithi12556470-20 2021-03-24T21:55:12.160 INFO:tasks.rados.rados.0.smithi125.stdout:219: finishing write tid 4 to smithi12556470-20 2021-03-24T21:55:12.161 INFO:tasks.rados.rados.0.smithi125.stdout:219: finishing write tid 5 to smithi12556470-20 2021-03-24T21:55:12.161 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 20 v 40 (ObjNum 107 snap 22 seq_num 107) dirty exists 2021-03-24T21:55:12.161 INFO:tasks.rados.rados.0.smithi125.stdout:219: left oid 20 (ObjNum 107 snap 22 seq_num 107) 2021-03-24T21:55:12.161 INFO:tasks.rados.rados.0.smithi125.stdout:220: finishing write tid 1 to smithi12556470-32 2021-03-24T21:55:12.161 INFO:tasks.rados.rados.0.smithi125.stdout:220: finishing write tid 2 to smithi12556470-32 2021-03-24T21:55:12.162 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:12.163 INFO:tasks.rados.rados.0.smithi125.stdout:219: done (2 left) 2021-03-24T21:55:12.163 INFO:tasks.rados.rados.0.smithi125.stdout:221: done (1 left) 2021-03-24T21:55:12.164 INFO:tasks.rados.rados.0.smithi125.stdout:222: write (excl) oid 29 current snap is 22 2021-03-24T21:55:12.164 INFO:tasks.rados.rados.0.smithi125.stdout:222: seq_num 110 ranges {648770=706332,2120431=420030,3271235=432392} 2021-03-24T21:55:12.170 INFO:tasks.rados.rados.0.smithi125.stdout:222: writing smithi12556470-29 from 648770 to 1355102 tid 1 2021-03-24T21:55:12.178 INFO:tasks.rados.rados.0.smithi125.stdout:222: writing smithi12556470-29 from 2120431 to 2540461 tid 2 2021-03-24T21:55:12.186 INFO:tasks.rados.rados.0.smithi125.stdout:222: writing smithi12556470-29 from 3271235 to 3703627 tid 3 2021-03-24T21:55:12.186 INFO:tasks.rados.rados.0.smithi125.stdout:223: copy_from oid 34 from oid 7 current snap is 22 2021-03-24T21:55:12.186 INFO:tasks.rados.rados.0.smithi125.stdout:220: finishing write tid 3 to smithi12556470-32 2021-03-24T21:55:12.187 INFO:tasks.rados.rados.0.smithi125.stdout:220: finishing write tid 4 to smithi12556470-32 2021-03-24T21:55:12.187 INFO:tasks.rados.rados.0.smithi125.stdout:220: finishing write tid 5 to smithi12556470-32 2021-03-24T21:55:12.187 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 32 v 67 (ObjNum 108 snap 22 seq_num 108) dirty exists 2021-03-24T21:55:12.187 INFO:tasks.rados.rados.0.smithi125.stdout:220: left oid 32 (ObjNum 108 snap 22 seq_num 108) 2021-03-24T21:55:12.188 INFO:tasks.rados.rados.0.smithi125.stdout:222: finishing write tid 1 to smithi12556470-29 2021-03-24T21:55:12.188 INFO:tasks.rados.rados.0.smithi125.stdout:222: finishing write tid 2 to smithi12556470-29 2021-03-24T21:55:12.188 INFO:tasks.rados.rados.0.smithi125.stdout:220: done (2 left) 2021-03-24T21:55:12.188 INFO:tasks.rados.rados.0.smithi125.stdout:224: delete oid 48 current snap is 22 2021-03-24T21:55:12.189 INFO:tasks.rados.rados.0.smithi125.stdout:223: finishing copy_from to smithi12556470-34 2021-03-24T21:55:12.189 INFO:tasks.rados.rados.0.smithi125.stdout:223: got expected ENOENT (src dne) 2021-03-24T21:55:12.189 INFO:tasks.rados.rados.0.smithi125.stdout:223: finishing copy_from racing read to smithi12556470-34 2021-03-24T21:55:12.190 INFO:tasks.rados.rados.0.smithi125.stdout:223: done (2 left) 2021-03-24T21:55:12.190 INFO:tasks.rados.rados.0.smithi125.stdout:224: done (1 left) 2021-03-24T21:55:12.190 INFO:tasks.rados.rados.0.smithi125.stdout:225: snap_remove snap 12 2021-03-24T21:55:12.223 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:12.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:12.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:55:12.264 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:12 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:55:12.265 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:12 smithi125 conmon[19936]: 21:55:10.721782+0000 mgr.y (mgr.14144) 328 : cluster [DBG] pgmap v374: 148 pgs: 148 active+clean; 169 MiB data, 429 MiB used, 536 GiB / 536 GiB avail; 13 MiB/s rd, 6.6 MiB/s wr, 25 op/s 2021-03-24T21:55:12.265 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:12 smithi125 conmon[19936]: cluster 2021-03-24T21:55:11.101670+0000 mon.a (mon.0) 705 : 2021-03-24T21:55:12.265 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:12 smithi125 conmon[19936]: cluster [DBG] osdmap e142: 8 total, 8 up, 6 in 2021-03-24T21:55:12.265 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:12 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:55:12.266 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:12 smithi125 conmon[24927]: 55:10.721782+0000 mgr.y (mgr.14144) 328 : cluster [DBG] pgmap v374: 148 pgs: 148 active+clean; 169 MiB data, 429 MiB used, 536 GiB / 536 GiB avail; 13 MiB/s rd, 6.6 MiB/s wr, 25 op/s 2021-03-24T21:55:12.266 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:12 smithi125 conmon[24927]: cluster 2021-03-24T21:55: 2021-03-24T21:55:12.266 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:12 smithi125 conmon[24927]: 11.101670+0000 mon.a (mon.0) 705 : cluster 2021-03-24T21:55:12.267 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:12 smithi125 conmon[24927]: [DBG] osdmap e142: 8 total, 8 up, 6 in 2021-03-24T21:55:12.327 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:12.382 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:12.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:12.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:55:12.489 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:12.548 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:12.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:12.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:55:12.653 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:12.712 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:12.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:12.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:55:12.792 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-24T21:55:12.792 INFO:teuthology.orchestra.run.smithi125.stdout: "size": 20, 2021-03-24T21:55:12.793 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 600, 2021-03-24T21:55:12.793 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [ 2021-03-24T21:55:12.793 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.793 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:204 7.a e107/99 7:5f18d753:::smithi12556470-18:head v 107'8, mlcod=107'6)", 2021-03-24T21:55:12.794 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.794062+0000", 2021-03-24T21:55:12.794 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 35.993732393000002, 2021-03-24T21:55:12.794 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0070242760000000003, 2021-03-24T21:55:12.794 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:55:12.795 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:55:12.795 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:55:12.795 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.795 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:55:12.796 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.794062+0000", 2021-03-24T21:55:12.796 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:55:12.796 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.796 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.797 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:55:12.797 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.794062+0000", 2021-03-24T21:55:12.797 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T21:55:12.797 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.798 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.798 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:55:12.798 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.794060+0000", 2021-03-24T21:55:12.798 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000274751 2021-03-24T21:55:12.798 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.799 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.799 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:55:12.799 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.794335+0000", 2021-03-24T21:55:12.799 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.645e-05 2021-03-24T21:55:12.800 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.800 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.800 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:55:12.800 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.794352+0000", 2021-03-24T21:55:12.801 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6470000000000001e-06 2021-03-24T21:55:12.801 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.801 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.801 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:55:12.802 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.794358+0000", 2021-03-24T21:55:12.802 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.6215e-05 2021-03-24T21:55:12.802 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.802 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.802 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:12.803 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.794445+0000", 2021-03-24T21:55:12.803 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.9925000000000001e-05 2021-03-24T21:55:12.803 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.803 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.804 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:55:12.804 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.794494+0000", 2021-03-24T21:55:12.804 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0065218330000000003 2021-03-24T21:55:12.804 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.805 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.805 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:55:12.805 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.801016+0000", 2021-03-24T21:55:12.805 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.9933999999999998e-05 2021-03-24T21:55:12.806 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.806 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.806 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:55:12.806 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.801086+0000", 2021-03-24T21:55:12.806 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0070242760000000003 2021-03-24T21:55:12.807 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:12.807 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:55:12.807 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:12.807 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.808 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.808 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:219 7.8 7:16b6db15:::smithi12556470-3:head [delete] snapc 3=[3,2] ondisk+write+peerstat_old+known_if_redirected e109)", 2021-03-24T21:55:12.808 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:38.120417+0000", 2021-03-24T21:55:12.808 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 34.667377049000002, 2021-03-24T21:55:12.808 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0083189059999999992, 2021-03-24T21:55:12.809 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:55:12.809 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:55:12.809 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:55:12.809 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:55:12.810 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:55:12.810 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 219 2021-03-24T21:55:12.810 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.810 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:55:12.811 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.811 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:55:12.811 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.120417+0000", 2021-03-24T21:55:12.811 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:55:12.812 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.812 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.812 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:55:12.812 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.120417+0000", 2021-03-24T21:55:12.813 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999976 2021-03-24T21:55:12.813 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.813 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.813 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:55:12.814 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.120415+0000", 2021-03-24T21:55:12.814 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.8439999999999993e-06 2021-03-24T21:55:12.814 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.814 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.815 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:55:12.815 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.120425+0000", 2021-03-24T21:55:12.815 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.2237000000000001e-05 2021-03-24T21:55:12.815 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.815 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.816 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:55:12.816 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.120447+0000", 2021-03-24T21:55:12.816 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0397000000000001e-05 2021-03-24T21:55:12.816 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.816 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.817 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:55:12.817 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.120457+0000", 2021-03-24T21:55:12.817 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00043770000000000001 2021-03-24T21:55:12.818 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.818 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.818 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:12.818 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.120895+0000", 2021-03-24T21:55:12.818 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000103293 2021-03-24T21:55:12.819 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.819 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.819 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:55:12.819 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.120998+0000", 2021-03-24T21:55:12.820 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00023456400000000001 2021-03-24T21:55:12.820 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.820 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.820 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 4,7", 2021-03-24T21:55:12.820 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.121233+0000", 2021-03-24T21:55:12.821 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002657132 2021-03-24T21:55:12.821 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.821 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.821 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:55:12.822 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.123890+0000", 2021-03-24T21:55:12.822 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0030868089999999998 2021-03-24T21:55:12.822 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.822 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.822 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:55:12.823 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.126977+0000", 2021-03-24T21:55:12.823 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001630371 2021-03-24T21:55:12.823 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.823 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.824 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:55:12.824 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.128607+0000", 2021-03-24T21:55:12.824 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.9499999999999997e-05 2021-03-24T21:55:12.825 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.825 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.825 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:55:12.825 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.128657+0000", 2021-03-24T21:55:12.825 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.9561999999999995e-05 2021-03-24T21:55:12.826 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.826 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.826 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:55:12.826 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.128736+0000", 2021-03-24T21:55:12.827 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0083189059999999992 2021-03-24T21:55:12.827 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:12.827 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:55:12.827 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:12.827 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.828 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.828 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "MOSDPGPush(7.0 109/108 [PushOp(7:0ece3580:::smithi12556470-35:head, version: 107'3, data_included: [571438~623878,1806623~424698], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7:0ece3580:::smithi12556470-35:head@107'3, size: 2571438, copy_subset: [0~2571438], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2231321, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2021-03-24T21:55:12.828 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:38.142023+0000", 2021-03-24T21:55:12.828 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 34.645771189999998, 2021-03-24T21:55:12.829 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.013359005, 2021-03-24T21:55:12.829 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:55:12.829 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T21:55:12.829 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:55:12.830 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.830 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:55:12.830 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.142023+0000", 2021-03-24T21:55:12.830 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:55:12.830 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.831 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.831 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:55:12.831 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.142023+0000", 2021-03-24T21:55:12.832 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T21:55:12.832 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.832 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.832 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:55:12.832 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.142022+0000", 2021-03-24T21:55:12.833 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012155871 2021-03-24T21:55:12.833 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.833 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.833 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:55:12.834 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.154178+0000", 2021-03-24T21:55:12.834 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.9859e-05 2021-03-24T21:55:12.834 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.835 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.835 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:55:12.835 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.154208+0000", 2021-03-24T21:55:12.835 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.4072e-05 2021-03-24T21:55:12.836 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.836 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.836 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:55:12.836 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.154222+0000", 2021-03-24T21:55:12.836 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00023070499999999999 2021-03-24T21:55:12.837 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.837 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.837 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:12.837 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.154452+0000", 2021-03-24T21:55:12.838 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.3880000000000006e-06 2021-03-24T21:55:12.838 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.838 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.838 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:55:12.838 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.154462+0000", 2021-03-24T21:55:12.839 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00092047799999999996 2021-03-24T21:55:12.839 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.839 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.839 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:55:12.840 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.155382+0000", 2021-03-24T21:55:12.840 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.013359005 2021-03-24T21:55:12.840 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:12.840 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:55:12.840 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:12.841 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.841 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.841 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "MOSDPGPushReply(4.12 109/108 [PushReplyOp(4:4bf29001:::data_log.60:head)])", 2021-03-24T21:55:12.841 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:38.158493+0000", 2021-03-24T21:55:12.841 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 34.629301388999998, 2021-03-24T21:55:12.842 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0063302080000000004, 2021-03-24T21:55:12.842 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:55:12.842 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "reached pg", 2021-03-24T21:55:12.842 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:55:12.843 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.843 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:55:12.843 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.158493+0000", 2021-03-24T21:55:12.843 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:55:12.844 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.844 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.844 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:55:12.844 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.158493+0000", 2021-03-24T21:55:12.844 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T21:55:12.845 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.845 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.845 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:55:12.845 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.158492+0000", 2021-03-24T21:55:12.846 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.3470000000000001e-06 2021-03-24T21:55:12.846 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.846 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.846 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:55:12.846 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.158499+0000", 2021-03-24T21:55:12.847 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2478e-05 2021-03-24T21:55:12.847 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.847 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.847 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:55:12.848 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.158511+0000", 2021-03-24T21:55:12.848 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9422000000000001e-05 2021-03-24T21:55:12.848 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.848 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.848 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:55:12.849 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.158531+0000", 2021-03-24T21:55:12.849 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0060486180000000004 2021-03-24T21:55:12.849 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.849 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.850 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:12.850 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.164580+0000", 2021-03-24T21:55:12.850 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000243663 2021-03-24T21:55:12.850 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.850 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.851 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:55:12.851 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.164823+0000", 2021-03-24T21:55:12.851 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0063302080000000004 2021-03-24T21:55:12.851 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:12.852 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:55:12.852 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:12.852 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.852 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.852 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(osd.7.0:20 7.5 e112/99 7:ad88ec1b:::smithi12556470-47:3 v 112'11, mlcod=109'7)", 2021-03-24T21:55:12.853 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:41.149282+0000", 2021-03-24T21:55:12.853 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 31.638512107, 2021-03-24T21:55:12.853 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.010071468, 2021-03-24T21:55:12.853 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:55:12.854 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:55:12.854 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:55:12.854 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.854 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:55:12.854 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:41.149282+0000", 2021-03-24T21:55:12.855 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:55:12.855 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.855 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.855 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:55:12.856 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:41.149282+0000", 2021-03-24T21:55:12.856 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999976 2021-03-24T21:55:12.856 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.856 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.856 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:55:12.857 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:41.149280+0000", 2021-03-24T21:55:12.857 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.4156e-05 2021-03-24T21:55:12.857 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.857 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.857 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:55:12.858 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:41.149294+0000", 2021-03-24T21:55:12.858 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.365e-05 2021-03-24T21:55:12.858 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.858 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.859 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:55:12.859 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:41.149308+0000", 2021-03-24T21:55:12.859 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.737e-06 2021-03-24T21:55:12.859 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.860 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.860 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:55:12.860 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:41.149314+0000", 2021-03-24T21:55:12.860 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.4080999999999999e-05 2021-03-24T21:55:12.860 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.861 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.861 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:12.861 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:41.149368+0000", 2021-03-24T21:55:12.861 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.7549000000000003e-05 2021-03-24T21:55:12.861 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.862 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.862 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:55:12.862 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:41.149406+0000", 2021-03-24T21:55:12.862 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0099096319999999998 2021-03-24T21:55:12.863 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.863 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.863 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:55:12.863 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:41.159316+0000", 2021-03-24T21:55:12.863 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.8166e-05 2021-03-24T21:55:12.864 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.864 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.864 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:55:12.864 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:41.159354+0000", 2021-03-24T21:55:12.864 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.010071468 2021-03-24T21:55:12.865 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:12.865 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:55:12.865 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:12.865 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.866 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.866 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:277 7.8 7:19ff009c:::smithi12556470-42:head [copy-from ver 9] snapc 8=[8] ondisk+write+known_if_redirected e114)", 2021-03-24T21:55:12.866 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:43.202214+0000", 2021-03-24T21:55:12.866 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 29.585580487000001, 2021-03-24T21:55:12.866 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.015212942, 2021-03-24T21:55:12.867 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:55:12.867 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:55:12.867 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:55:12.867 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:55:12.867 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:55:12.868 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 277 2021-03-24T21:55:12.868 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.868 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:55:12.868 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.869 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:55:12.869 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.202214+0000", 2021-03-24T21:55:12.869 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:55:12.869 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.869 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.870 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:55:12.870 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.202214+0000", 2021-03-24T21:55:12.870 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999971 2021-03-24T21:55:12.870 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.871 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.871 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:55:12.871 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.202211+0000", 2021-03-24T21:55:12.871 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1984000000000001e-05 2021-03-24T21:55:12.872 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.872 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.872 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:55:12.872 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.202223+0000", 2021-03-24T21:55:12.872 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7306e-05 2021-03-24T21:55:12.873 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.873 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.873 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:55:12.873 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.202240+0000", 2021-03-24T21:55:12.874 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.6490000000000003e-06 2021-03-24T21:55:12.874 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.874 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.874 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:55:12.874 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.202250+0000", 2021-03-24T21:55:12.875 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.6559999999999999e-05 2021-03-24T21:55:12.875 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.875 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.875 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:12.876 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.202327+0000", 2021-03-24T21:55:12.876 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.1945000000000001e-05 2021-03-24T21:55:12.876 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.876 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.876 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:55:12.877 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.202419+0000", 2021-03-24T21:55:12.877 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0048950820000000003 2021-03-24T21:55:12.877 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.877 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.878 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 4,7", 2021-03-24T21:55:12.878 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.207314+0000", 2021-03-24T21:55:12.878 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0050236980000000001 2021-03-24T21:55:12.878 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.878 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.879 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:55:12.879 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.212337+0000", 2021-03-24T21:55:12.879 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0048611870000000003 2021-03-24T21:55:12.879 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.880 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.880 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:55:12.880 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.217199+0000", 2021-03-24T21:55:12.880 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1696000000000001e-05 2021-03-24T21:55:12.880 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.881 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.881 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:55:12.881 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.217250+0000", 2021-03-24T21:55:12.881 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.4931000000000004e-05 2021-03-24T21:55:12.882 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.882 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.882 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:55:12.882 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.217335+0000", 2021-03-24T21:55:12.882 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.1592e-05 2021-03-24T21:55:12.883 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.883 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.883 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:55:12.883 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.217427+0000", 2021-03-24T21:55:12.884 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.015212942 2021-03-24T21:55:12.884 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:12.884 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:55:12.884 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:12.884 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.885 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.885 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:278 7.8 7:19ff009c:::smithi12556470-42:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e114)", 2021-03-24T21:55:12.885 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:43.202267+0000", 2021-03-24T21:55:12.885 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 29.585526932, 2021-03-24T21:55:12.885 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.015574229, 2021-03-24T21:55:12.886 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:55:12.886 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T21:55:12.886 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:55:12.886 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:55:12.887 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:55:12.887 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 278 2021-03-24T21:55:12.887 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.887 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:55:12.888 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.888 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:55:12.888 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.202267+0000", 2021-03-24T21:55:12.888 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:55:12.888 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.889 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.889 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:55:12.889 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.202267+0000", 2021-03-24T21:55:12.889 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T21:55:12.890 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.890 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.890 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:55:12.890 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.202267+0000", 2021-03-24T21:55:12.890 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6749999999999999e-06 2021-03-24T21:55:12.891 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.891 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.891 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:55:12.891 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.202272+0000", 2021-03-24T21:55:12.892 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.8089999999999998e-06 2021-03-24T21:55:12.892 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.892 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.892 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:55:12.892 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.202278+0000", 2021-03-24T21:55:12.893 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.354e-06 2021-03-24T21:55:12.893 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.893 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.893 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:55:12.894 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.202282+0000", 2021-03-24T21:55:12.894 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00032993900000000001 2021-03-24T21:55:12.894 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.894 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.894 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:12.895 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.202611+0000", 2021-03-24T21:55:12.895 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.3274000000000004e-05 2021-03-24T21:55:12.895 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.895 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.896 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-24T21:55:12.896 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.202685+0000", 2021-03-24T21:55:12.896 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0085844809999999997 2021-03-24T21:55:12.896 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.896 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.897 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:12.897 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.211269+0000", 2021-03-24T21:55:12.897 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.8164000000000004e-05 2021-03-24T21:55:12.897 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.897 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.898 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T21:55:12.898 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.211367+0000", 2021-03-24T21:55:12.898 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0061402770000000004 2021-03-24T21:55:12.898 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.899 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.899 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:12.899 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.217508+0000", 2021-03-24T21:55:12.899 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00016125499999999999 2021-03-24T21:55:12.899 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.900 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.900 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:55:12.900 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.217669+0000", 2021-03-24T21:55:12.900 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00017273599999999999 2021-03-24T21:55:12.901 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.901 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.901 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:55:12.901 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.217842+0000", 2021-03-24T21:55:12.901 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.015574229 2021-03-24T21:55:12.902 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:12.902 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:55:12.902 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:12.902 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.903 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.903 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:294 7.b e114/108 7:d821c111:::smithi12556470-9:head v 114'20, mlcod=0'0)", 2021-03-24T21:55:12.903 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:43.452985+0000", 2021-03-24T21:55:12.903 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 29.334808891000002, 2021-03-24T21:55:12.903 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0058441530000000004, 2021-03-24T21:55:12.904 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:55:12.904 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:55:12.904 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:55:12.904 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.904 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:55:12.905 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.452985+0000", 2021-03-24T21:55:12.905 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:55:12.905 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.905 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.905 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:55:12.906 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.452985+0000", 2021-03-24T21:55:12.906 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999952 2021-03-24T21:55:12.906 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.906 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.907 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:55:12.907 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.452981+0000", 2021-03-24T21:55:12.907 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0040413840000000003 2021-03-24T21:55:12.907 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.907 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.908 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:55:12.908 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.457022+0000", 2021-03-24T21:55:12.908 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.6670999999999999e-05 2021-03-24T21:55:12.908 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.908 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.909 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:55:12.909 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.457049+0000", 2021-03-24T21:55:12.909 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.8829999999999995e-06 2021-03-24T21:55:12.909 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.909 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.910 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:55:12.910 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.457058+0000", 2021-03-24T21:55:12.910 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.1901000000000002e-05 2021-03-24T21:55:12.910 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.911 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.911 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:12.911 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.457130+0000", 2021-03-24T21:55:12.911 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.2412e-05 2021-03-24T21:55:12.911 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.912 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.912 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:55:12.912 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.457172+0000", 2021-03-24T21:55:12.912 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001614735 2021-03-24T21:55:12.912 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.913 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.913 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:55:12.913 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.458787+0000", 2021-03-24T21:55:12.913 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.2954000000000002e-05 2021-03-24T21:55:12.914 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.914 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.914 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:55:12.914 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.458830+0000", 2021-03-24T21:55:12.914 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0058441530000000004 2021-03-24T21:55:12.915 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:12.915 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:55:12.915 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:12.915 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.916 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.916 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(osd.7.0:28 7.5 e123/99 7:ad88ec1b:::smithi12556470-47:d v 123'21, mlcod=120'18)", 2021-03-24T21:55:12.916 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:52.113893+0000", 2021-03-24T21:55:12.916 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 20.673900922000001, 2021-03-24T21:55:12.916 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012019541999999999, 2021-03-24T21:55:12.917 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:55:12.917 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:55:12.917 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:55:12.917 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.917 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:55:12.918 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.113893+0000", 2021-03-24T21:55:12.918 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:55:12.918 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.918 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.919 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:55:12.919 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.113893+0000", 2021-03-24T21:55:12.919 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T21:55:12.919 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.919 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.920 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:55:12.920 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.113892+0000", 2021-03-24T21:55:12.920 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3386e-05 2021-03-24T21:55:12.920 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.921 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.921 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:55:12.921 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.113906+0000", 2021-03-24T21:55:12.921 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0048278569999999996 2021-03-24T21:55:12.922 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.922 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.922 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:55:12.922 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.118734+0000", 2021-03-24T21:55:12.922 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.0820000000000005e-06 2021-03-24T21:55:12.923 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.923 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.923 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:55:12.923 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.118743+0000", 2021-03-24T21:55:12.923 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.7781000000000001e-05 2021-03-24T21:55:12.924 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.924 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.924 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:12.924 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.118800+0000", 2021-03-24T21:55:12.924 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.8793e-05 2021-03-24T21:55:12.925 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.925 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.925 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:55:12.925 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.118829+0000", 2021-03-24T21:55:12.926 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0070437140000000004 2021-03-24T21:55:12.926 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.926 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.926 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:55:12.927 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.125873+0000", 2021-03-24T21:55:12.927 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.0057e-05 2021-03-24T21:55:12.927 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.927 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.927 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:55:12.928 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.125913+0000", 2021-03-24T21:55:12.928 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012019541999999999 2021-03-24T21:55:12.928 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:12.928 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:55:12.928 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:12.929 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.929 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.929 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(osd.5.0:68 7.b e123/108 7:d841eed8:::smithi12556470-31:d v 123'43, mlcod=120'40)", 2021-03-24T21:55:12.929 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:52.121456+0000", 2021-03-24T21:55:12.930 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 20.666337944999999, 2021-03-24T21:55:12.930 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0071538929999999997, 2021-03-24T21:55:12.930 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:55:12.930 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:55:12.931 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:55:12.931 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.931 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:55:12.931 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.121456+0000", 2021-03-24T21:55:12.931 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:55:12.932 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.932 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.932 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:55:12.932 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.121456+0000", 2021-03-24T21:55:12.933 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999995 2021-03-24T21:55:12.933 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.933 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.933 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:55:12.933 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.121456+0000", 2021-03-24T21:55:12.934 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.3319999999999998e-06 2021-03-24T21:55:12.934 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.934 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.934 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:55:12.935 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.121465+0000", 2021-03-24T21:55:12.935 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7739000000000001e-05 2021-03-24T21:55:12.935 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.935 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.935 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:55:12.936 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.121483+0000", 2021-03-24T21:55:12.936 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1691999999999999e-05 2021-03-24T21:55:12.936 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.936 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.936 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:55:12.937 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.121495+0000", 2021-03-24T21:55:12.937 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.7725000000000005e-05 2021-03-24T21:55:12.937 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.937 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.938 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:12.938 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.121572+0000", 2021-03-24T21:55:12.938 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5736999999999999e-05 2021-03-24T21:55:12.938 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.939 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.939 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:55:12.939 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.121588+0000", 2021-03-24T21:55:12.939 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0069713329999999997 2021-03-24T21:55:12.939 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.940 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.940 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:55:12.940 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.128559+0000", 2021-03-24T21:55:12.940 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.0998999999999997e-05 2021-03-24T21:55:12.940 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.941 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.941 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:55:12.941 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.128610+0000", 2021-03-24T21:55:12.941 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0071538929999999997 2021-03-24T21:55:12.942 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:12.942 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:55:12.942 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:12.942 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.942 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.943 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(osd.5.0:69 7.b e123/108 7:de4c3db5:::smithi12556470-32:d v 123'45, mlcod=120'40)", 2021-03-24T21:55:12.943 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:52.122255+0000", 2021-03-24T21:55:12.943 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 20.665539819999999, 2021-03-24T21:55:12.943 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0063817140000000001, 2021-03-24T21:55:12.944 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:55:12.944 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:55:12.944 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:55:12.944 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.944 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:55:12.945 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.122255+0000", 2021-03-24T21:55:12.945 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:55:12.945 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.945 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.945 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:55:12.946 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.122255+0000", 2021-03-24T21:55:12.946 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T21:55:12.946 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.946 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.947 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:55:12.947 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.122253+0000", 2021-03-24T21:55:12.947 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.7179999999999996e-06 2021-03-24T21:55:12.947 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.947 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.948 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:55:12.948 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.122261+0000", 2021-03-24T21:55:12.948 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.0420000000000008e-06 2021-03-24T21:55:12.948 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.948 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.949 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:55:12.949 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.122270+0000", 2021-03-24T21:55:12.949 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.408e-06 2021-03-24T21:55:12.949 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.949 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.950 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:55:12.950 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.122274+0000", 2021-03-24T21:55:12.950 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0012787899999999999 2021-03-24T21:55:12.950 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.951 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.951 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:12.951 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.123553+0000", 2021-03-24T21:55:12.951 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0024999999999999e-05 2021-03-24T21:55:12.951 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.952 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.952 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:55:12.952 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.123573+0000", 2021-03-24T21:55:12.952 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0050443739999999999 2021-03-24T21:55:12.953 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.953 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.953 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:55:12.953 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.128618+0000", 2021-03-24T21:55:12.953 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.8692e-05 2021-03-24T21:55:12.954 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.954 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.954 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:55:12.954 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.128636+0000", 2021-03-24T21:55:12.955 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0063817140000000001 2021-03-24T21:55:12.955 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:12.955 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:55:12.955 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:12.955 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.956 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.956 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:492 7.e 7:777a79d9:::smithi12556470-38:head [rollback 15] snapc 15=[15,f] ondisk+write+known_if_redirected e127)", 2021-03-24T21:55:12.956 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:56.096674+0000", 2021-03-24T21:55:12.956 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 16.691120072, 2021-03-24T21:55:12.956 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0078288200000000002, 2021-03-24T21:55:12.957 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:55:12.957 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:55:12.957 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:55:12.957 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:55:12.958 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:55:12.958 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 492 2021-03-24T21:55:12.958 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.958 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:55:12.958 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.959 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:55:12.959 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:56.096674+0000", 2021-03-24T21:55:12.959 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:55:12.959 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.960 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.960 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:55:12.960 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:56.096674+0000", 2021-03-24T21:55:12.960 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T21:55:12.960 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.961 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.961 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:55:12.961 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:56.096673+0000", 2021-03-24T21:55:12.961 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2190000000000003e-06 2021-03-24T21:55:12.961 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.962 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.962 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:55:12.962 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:56.096679+0000", 2021-03-24T21:55:12.962 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0421000000000001e-05 2021-03-24T21:55:12.963 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.963 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.963 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:55:12.963 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:56.096689+0000", 2021-03-24T21:55:12.964 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5589999999999996e-06 2021-03-24T21:55:12.964 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.964 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.964 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:55:12.964 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:56.096695+0000", 2021-03-24T21:55:12.965 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000162514 2021-03-24T21:55:12.965 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.965 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.965 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:12.965 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:56.096857+0000", 2021-03-24T21:55:12.966 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.4479999999999994e-06 2021-03-24T21:55:12.966 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.966 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.966 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting_for_map not empty", 2021-03-24T21:55:12.967 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:56.096866+0000", 2021-03-24T21:55:12.967 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.003206665 2021-03-24T21:55:12.967 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.967 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.967 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:12.968 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:56.100072+0000", 2021-03-24T21:55:12.968 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.0593999999999994e-05 2021-03-24T21:55:12.968 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.968 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.968 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:55:12.969 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:56.100153+0000", 2021-03-24T21:55:12.969 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000232205 2021-03-24T21:55:12.969 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.969 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.970 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,5", 2021-03-24T21:55:12.970 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:56.100385+0000", 2021-03-24T21:55:12.970 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0020889770000000001 2021-03-24T21:55:12.970 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.970 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.971 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:55:12.971 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:56.102474+0000", 2021-03-24T21:55:12.971 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7376000000000001e-05 2021-03-24T21:55:12.971 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.972 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.972 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:55:12.972 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:56.102491+0000", 2021-03-24T21:55:12.972 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0018621180000000001 2021-03-24T21:55:12.972 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.973 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.973 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:55:12.973 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:56.104353+0000", 2021-03-24T21:55:12.973 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.5879999999999999e-05 2021-03-24T21:55:12.973 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.974 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.974 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:55:12.974 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:56.104429+0000", 2021-03-24T21:55:12.974 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.3757999999999994e-05 2021-03-24T21:55:12.975 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.975 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.975 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:55:12.975 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:56.104503+0000", 2021-03-24T21:55:12.975 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0078288200000000002 2021-03-24T21:55:12.976 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:12.976 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:55:12.976 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:12.976 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.977 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.977 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:493 7.e 7:777a79d9:::smithi12556470-38:head [append 0~0] snapc 15=[15,f] ondisk+write+known_if_redirected e127)", 2021-03-24T21:55:12.977 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:56.096699+0000", 2021-03-24T21:55:12.977 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 16.691094893999999, 2021-03-24T21:55:12.977 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0080784450000000001, 2021-03-24T21:55:12.978 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:55:12.978 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:55:12.978 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:55:12.978 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:55:12.979 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:55:12.979 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 493 2021-03-24T21:55:12.979 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.979 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:55:12.979 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.980 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:55:12.980 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:56.096699+0000", 2021-03-24T21:55:12.980 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:55:12.980 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.981 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.981 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:55:12.981 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:56.096699+0000", 2021-03-24T21:55:12.981 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999995 2021-03-24T21:55:12.981 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.982 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.982 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:55:12.982 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:56.096699+0000", 2021-03-24T21:55:12.982 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.1459999999999998e-06 2021-03-24T21:55:12.983 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.983 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.983 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:55:12.983 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:56.096703+0000", 2021-03-24T21:55:12.983 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.8879999999999999e-06 2021-03-24T21:55:12.984 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.984 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.984 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:55:12.984 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:56.096707+0000", 2021-03-24T21:55:12.985 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.7159999999999999e-06 2021-03-24T21:55:12.985 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.985 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.985 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:55:12.985 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:56.096710+0000", 2021-03-24T21:55:12.986 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000192683 2021-03-24T21:55:12.986 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.986 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.986 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:12.986 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:56.096902+0000", 2021-03-24T21:55:12.987 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.678e-06 2021-03-24T21:55:12.987 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.987 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.987 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting_for_map not empty", 2021-03-24T21:55:12.988 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:56.096910+0000", 2021-03-24T21:55:12.988 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0037767619999999999 2021-03-24T21:55:12.988 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.988 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.988 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:12.989 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:56.100687+0000", 2021-03-24T21:55:12.989 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.7711000000000002e-05 2021-03-24T21:55:12.989 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.989 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.989 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:55:12.990 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:56.100765+0000", 2021-03-24T21:55:12.990 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000174322 2021-03-24T21:55:12.990 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.990 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.991 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,5", 2021-03-24T21:55:12.991 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:56.100939+0000", 2021-03-24T21:55:12.991 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001746566 2021-03-24T21:55:12.991 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.991 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.992 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:55:12.992 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:56.102685+0000", 2021-03-24T21:55:12.992 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0012237089999999999 2021-03-24T21:55:12.992 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.992 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.993 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:55:12.993 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:56.103909+0000", 2021-03-24T21:55:12.993 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00076403500000000004 2021-03-24T21:55:12.993 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.993 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.994 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:55:12.994 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:56.104673+0000", 2021-03-24T21:55:12.994 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3535999999999998e-05 2021-03-24T21:55:12.994 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.995 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.995 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:55:12.995 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:56.104717+0000", 2021-03-24T21:55:12.995 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.1179000000000005e-05 2021-03-24T21:55:12.996 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.996 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.996 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:55:12.996 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:56.104778+0000", 2021-03-24T21:55:12.996 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0080784450000000001 2021-03-24T21:55:12.997 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:12.997 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:55:12.997 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:12.997 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:12.997 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:12.998 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:527 7.5 e130/99 7:ad88ec1b:::smithi12556470-47:head v 130'23, mlcod=120'19)", 2021-03-24T21:55:12.998 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:59.109335+0000", 2021-03-24T21:55:12.998 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 13.678459835, 2021-03-24T21:55:12.998 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.006531706, 2021-03-24T21:55:12.999 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:55:12.999 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:55:12.999 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:55:12.999 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.000 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:55:13.000 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:59.109335+0000", 2021-03-24T21:55:13.000 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:55:13.000 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.000 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.001 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:55:13.001 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:59.109335+0000", 2021-03-24T21:55:13.001 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999995 2021-03-24T21:55:13.001 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.002 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.002 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:55:13.002 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:59.109334+0000", 2021-03-24T21:55:13.002 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0044002909999999998 2021-03-24T21:55:13.003 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.003 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.003 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:55:13.003 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:59.113735+0000", 2021-03-24T21:55:13.003 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5481999999999998e-05 2021-03-24T21:55:13.004 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.004 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.004 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:55:13.004 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:59.113750+0000", 2021-03-24T21:55:13.005 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2369999999999997e-06 2021-03-24T21:55:13.005 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.005 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.005 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:55:13.005 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:59.113756+0000", 2021-03-24T21:55:13.006 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.9341000000000001e-05 2021-03-24T21:55:13.006 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.006 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.006 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:13.006 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:59.113806+0000", 2021-03-24T21:55:13.007 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.8984e-05 2021-03-24T21:55:13.007 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.007 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.007 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:55:13.008 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:59.113825+0000", 2021-03-24T21:55:13.008 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001992101 2021-03-24T21:55:13.008 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.008 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.008 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:55:13.009 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:59.115817+0000", 2021-03-24T21:55:13.009 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.9526e-05 2021-03-24T21:55:13.009 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.009 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.009 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:55:13.010 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:59.115866+0000", 2021-03-24T21:55:13.010 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.006531706 2021-03-24T21:55:13.010 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:13.010 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:55:13.011 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:13.011 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.011 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.011 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:530 7.8 7:19ff009c:::smithi12556470-42:head [write 729437~527542,stat] snapc 19=[19,17,16,15] ondisk+write+known_if_redirected e131)", 2021-03-24T21:55:13.011 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:55:00.098667+0000", 2021-03-24T21:55:13.012 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 12.689126879, 2021-03-24T21:55:13.012 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0061618080000000004, 2021-03-24T21:55:13.012 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:55:13.012 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:55:13.013 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:55:13.013 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:55:13.013 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:55:13.013 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 530 2021-03-24T21:55:13.013 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.014 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:55:13.014 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.014 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:55:13.014 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.098667+0000", 2021-03-24T21:55:13.014 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:55:13.015 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.015 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.015 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:55:13.015 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.098667+0000", 2021-03-24T21:55:13.016 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999938 2021-03-24T21:55:13.016 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.016 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.016 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:55:13.016 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.098662+0000", 2021-03-24T21:55:13.017 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00030417200000000001 2021-03-24T21:55:13.017 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.017 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.017 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:55:13.017 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.098966+0000", 2021-03-24T21:55:13.018 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.7645e-05 2021-03-24T21:55:13.018 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.018 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.018 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:55:13.019 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.098993+0000", 2021-03-24T21:55:13.019 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2942000000000001e-05 2021-03-24T21:55:13.019 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.019 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.019 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:55:13.020 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.099006+0000", 2021-03-24T21:55:13.020 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00011213 2021-03-24T21:55:13.020 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.020 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.021 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:13.021 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.099118+0000", 2021-03-24T21:55:13.021 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000159347 2021-03-24T21:55:13.021 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.021 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.022 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:55:13.022 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.099278+0000", 2021-03-24T21:55:13.022 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00035342000000000001 2021-03-24T21:55:13.022 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.023 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.023 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 4,7", 2021-03-24T21:55:13.023 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.099631+0000", 2021-03-24T21:55:13.023 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002150693 2021-03-24T21:55:13.023 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.024 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.024 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:55:13.024 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.101782+0000", 2021-03-24T21:55:13.024 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0020140480000000001 2021-03-24T21:55:13.025 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.025 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.025 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:55:13.025 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.103796+0000", 2021-03-24T21:55:13.025 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00089429500000000003 2021-03-24T21:55:13.026 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.026 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.026 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:55:13.026 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.104690+0000", 2021-03-24T21:55:13.027 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.4165999999999994e-05 2021-03-24T21:55:13.027 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.027 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.027 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:55:13.027 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.104764+0000", 2021-03-24T21:55:13.028 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4920999999999997e-05 2021-03-24T21:55:13.028 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.028 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.028 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:55:13.029 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.104829+0000", 2021-03-24T21:55:13.029 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0061618080000000004 2021-03-24T21:55:13.029 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:13.029 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:55:13.029 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:13.030 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.030 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.030 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:535 7.3 e131/99 7:cb55abc9:::smithi12556470-6:head v 131'35, mlcod=115'32)", 2021-03-24T21:55:13.030 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:55:00.126442+0000", 2021-03-24T21:55:13.031 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 12.661352668999999, 2021-03-24T21:55:13.031 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0074330719999999998, 2021-03-24T21:55:13.031 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:55:13.031 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:55:13.032 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:55:13.032 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.032 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:55:13.032 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.126442+0000", 2021-03-24T21:55:13.032 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:55:13.033 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.033 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.033 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:55:13.033 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.126442+0000", 2021-03-24T21:55:13.034 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T21:55:13.034 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.034 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.034 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:55:13.034 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.126441+0000", 2021-03-24T21:55:13.035 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0024534040000000002 2021-03-24T21:55:13.035 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.035 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.035 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:55:13.036 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.128894+0000", 2021-03-24T21:55:13.036 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.9875000000000001e-05 2021-03-24T21:55:13.036 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.036 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.036 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:55:13.037 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.128924+0000", 2021-03-24T21:55:13.037 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1225e-05 2021-03-24T21:55:13.037 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.037 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.038 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:55:13.038 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.128935+0000", 2021-03-24T21:55:13.038 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00015186 2021-03-24T21:55:13.038 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.038 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.039 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:13.039 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.129087+0000", 2021-03-24T21:55:13.039 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.3269000000000003e-05 2021-03-24T21:55:13.039 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.040 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.040 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:55:13.040 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.129120+0000", 2021-03-24T21:55:13.040 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0047029699999999999 2021-03-24T21:55:13.040 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.041 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.041 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:55:13.041 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.133823+0000", 2021-03-24T21:55:13.041 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1629000000000003e-05 2021-03-24T21:55:13.041 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.042 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.042 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:55:13.042 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.133875+0000", 2021-03-24T21:55:13.042 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0074330719999999998 2021-03-24T21:55:13.043 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:13.043 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:55:13.043 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:13.043 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.043 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.044 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:632 7.b e140/108 7:dcd8f3ef:::smithi12556470-43:head v 140'62, mlcod=139'59)", 2021-03-24T21:55:13.044 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:55:09.381158+0000", 2021-03-24T21:55:13.044 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 3.406635927, 2021-03-24T21:55:13.044 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00710402, 2021-03-24T21:55:13.045 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:55:13.045 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:55:13.045 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:55:13.045 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.045 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:55:13.046 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:09.381158+0000", 2021-03-24T21:55:13.046 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:55:13.046 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.046 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.047 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:55:13.047 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:09.381158+0000", 2021-03-24T21:55:13.047 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999971 2021-03-24T21:55:13.047 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.047 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.048 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:55:13.048 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:09.381156+0000", 2021-03-24T21:55:13.048 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0046473529999999999 2021-03-24T21:55:13.048 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.049 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.049 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:55:13.049 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:09.385803+0000", 2021-03-24T21:55:13.049 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.3868999999999997e-05 2021-03-24T21:55:13.049 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.050 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.050 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:55:13.050 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:09.385837+0000", 2021-03-24T21:55:13.050 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1899000000000001e-05 2021-03-24T21:55:13.051 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.051 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.051 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:55:13.051 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:09.385849+0000", 2021-03-24T21:55:13.051 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.6557999999999994e-05 2021-03-24T21:55:13.052 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.052 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.052 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:13.052 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:09.385935+0000", 2021-03-24T21:55:13.053 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.6188e-05 2021-03-24T21:55:13.053 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.053 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.053 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:55:13.053 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:09.385972+0000", 2021-03-24T21:55:13.054 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0022319129999999999 2021-03-24T21:55:13.054 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.054 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.054 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:55:13.055 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:09.388203+0000", 2021-03-24T21:55:13.055 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.8990999999999998e-05 2021-03-24T21:55:13.055 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.055 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.056 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:55:13.056 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:09.388262+0000", 2021-03-24T21:55:13.056 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00710402 2021-03-24T21:55:13.056 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:13.056 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:55:13.057 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:13.057 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.057 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.057 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:645 7.c e141/101 7:305739dc:::smithi12556470-36:head v 141'47, mlcod=134'43)", 2021-03-24T21:55:13.057 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:55:10.123986+0000", 2021-03-24T21:55:13.058 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 2.6638080030000002, 2021-03-24T21:55:13.058 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.025623269000000001, 2021-03-24T21:55:13.058 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:55:13.058 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:55:13.059 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:55:13.059 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.059 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:55:13.059 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.123986+0000", 2021-03-24T21:55:13.059 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:55:13.060 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.060 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.060 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:55:13.060 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.123986+0000", 2021-03-24T21:55:13.061 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T21:55:13.061 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.061 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.061 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:55:13.061 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.123985+0000", 2021-03-24T21:55:13.062 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2130000000000001e-05 2021-03-24T21:55:13.062 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.062 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.062 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:55:13.063 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.123997+0000", 2021-03-24T21:55:13.063 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.4334e-05 2021-03-24T21:55:13.063 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.063 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.063 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:55:13.064 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124031+0000", 2021-03-24T21:55:13.064 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.3420000000000002e-06 2021-03-24T21:55:13.064 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.064 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.065 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:55:13.065 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124039+0000", 2021-03-24T21:55:13.065 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.0111999999999997e-05 2021-03-24T21:55:13.065 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.065 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.066 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:13.066 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124089+0000", 2021-03-24T21:55:13.066 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.3839e-05 2021-03-24T21:55:13.066 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.067 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.067 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:55:13.067 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124113+0000", 2021-03-24T21:55:13.067 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.025454029999999999 2021-03-24T21:55:13.067 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.068 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.068 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:55:13.068 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.149567+0000", 2021-03-24T21:55:13.068 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.2976000000000003e-05 2021-03-24T21:55:13.068 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.069 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.069 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:55:13.069 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.149610+0000", 2021-03-24T21:55:13.069 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.025623269000000001 2021-03-24T21:55:13.070 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:13.070 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:55:13.070 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:13.070 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.070 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.071 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:646 7.c e141/101 7:305739dc:::smithi12556470-36:head v 141'48, mlcod=134'43)", 2021-03-24T21:55:13.071 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:55:10.124053+0000", 2021-03-24T21:55:13.071 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 2.6637414869999998, 2021-03-24T21:55:13.071 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.027636099000000001, 2021-03-24T21:55:13.072 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:55:13.072 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:55:13.072 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:55:13.072 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.072 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:55:13.073 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124053+0000", 2021-03-24T21:55:13.073 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:55:13.073 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.073 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.074 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:55:13.074 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124053+0000", 2021-03-24T21:55:13.074 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T21:55:13.074 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.074 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.075 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:55:13.075 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124052+0000", 2021-03-24T21:55:13.075 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.2480000000000002e-06 2021-03-24T21:55:13.075 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.076 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.076 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:55:13.076 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124061+0000", 2021-03-24T21:55:13.076 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6640000000000003e-06 2021-03-24T21:55:13.076 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.077 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.077 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:55:13.077 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124067+0000", 2021-03-24T21:55:13.077 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9349999999999996e-06 2021-03-24T21:55:13.077 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.078 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.078 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:55:13.078 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124071+0000", 2021-03-24T21:55:13.078 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00051871899999999995 2021-03-24T21:55:13.079 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.079 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.079 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:13.079 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124590+0000", 2021-03-24T21:55:13.079 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2456999999999999e-05 2021-03-24T21:55:13.080 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.080 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.080 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:55:13.080 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124602+0000", 2021-03-24T21:55:13.081 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.027032664000000001 2021-03-24T21:55:13.081 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.081 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.081 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:55:13.081 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.151635+0000", 2021-03-24T21:55:13.082 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.4151999999999999e-05 2021-03-24T21:55:13.082 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.082 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.082 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:55:13.083 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.151689+0000", 2021-03-24T21:55:13.083 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.027636099000000001 2021-03-24T21:55:13.083 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:13.083 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:55:13.083 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:13.084 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.084 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.084 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:647 7.c e141/101 7:305739dc:::smithi12556470-36:head v 141'49, mlcod=134'43)", 2021-03-24T21:55:13.084 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:55:10.124445+0000", 2021-03-24T21:55:13.085 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 2.6633495219999999, 2021-03-24T21:55:13.085 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.027276857000000002, 2021-03-24T21:55:13.085 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:55:13.085 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:55:13.085 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:55:13.086 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.086 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:55:13.086 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124445+0000", 2021-03-24T21:55:13.086 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:55:13.087 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.087 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.087 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:55:13.087 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124445+0000", 2021-03-24T21:55:13.087 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T21:55:13.088 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.088 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.088 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:55:13.088 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124444+0000", 2021-03-24T21:55:13.089 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.8339999999999999e-06 2021-03-24T21:55:13.089 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.089 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.089 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:55:13.089 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124451+0000", 2021-03-24T21:55:13.090 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.8640000000000002e-06 2021-03-24T21:55:13.090 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.090 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.090 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:55:13.091 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124460+0000", 2021-03-24T21:55:13.091 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5130000000000004e-06 2021-03-24T21:55:13.091 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.091 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.092 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:55:13.092 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124466+0000", 2021-03-24T21:55:13.092 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00030484700000000002 2021-03-24T21:55:13.092 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.092 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.093 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:13.093 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124771+0000", 2021-03-24T21:55:13.093 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.8258e-05 2021-03-24T21:55:13.093 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.093 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.094 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:55:13.094 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124789+0000", 2021-03-24T21:55:13.094 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.026903249000000001 2021-03-24T21:55:13.094 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.095 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.095 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:55:13.095 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.151692+0000", 2021-03-24T21:55:13.095 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.9533e-05 2021-03-24T21:55:13.096 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:13.096 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:13.096 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:55:13.096 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.151722+0000", 2021-03-24T21:55:13.096 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.027276857000000002 2021-03-24T21:55:13.097 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:13.097 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:55:13.097 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:13.097 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:13.098 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:55:13.098 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-24T21:55:13.098 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:13.122 INFO:tasks.rados.rados.0.smithi125.stdout:225: done (1 left) 2021-03-24T21:55:13.123 INFO:tasks.rados.rados.0.smithi125.stdout:226: rollback oid 26 current snap is 22 2021-03-24T21:55:13.123 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 26 to 21 2021-03-24T21:55:13.123 INFO:tasks.rados.rados.0.smithi125.stdout:222: finishing write tid 3 to smithi12556470-29 2021-03-24T21:55:13.123 INFO:tasks.rados.rados.0.smithi125.stdout:222: finishing write tid 4 to smithi12556470-29 2021-03-24T21:55:13.123 INFO:tasks.rados.rados.0.smithi125.stdout:222: finishing write tid 5 to smithi12556470-29 2021-03-24T21:55:13.124 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 29 v 22 (ObjNum 110 snap 22 seq_num 110) dirty exists 2021-03-24T21:55:13.124 INFO:tasks.rados.rados.0.smithi125.stdout:222: left oid 29 (ObjNum 110 snap 22 seq_num 110) 2021-03-24T21:55:13.124 INFO:tasks.rados.rados.0.smithi125.stdout:222: done (1 left) 2021-03-24T21:55:13.124 INFO:tasks.rados.rados.0.smithi125.stdout:227: snap_remove snap 19 2021-03-24T21:55:13.136 INFO:teuthology.orchestra.run.smithi125.stderr:nodeep-scrub is set 2021-03-24T21:55:13.157 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:13.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:13.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:55:13.262 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:13.276 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:13 smithi125 conmon[19936]: cluster 2021-03-24T21:55:12.111991+0000 mon.a (mon.0) 706 : 2021-03-24T21:55:13.276 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:13 smithi125 conmon[19936]: cluster [DBG] osdmap e143: 8 total, 8 up, 6 in 2021-03-24T21:55:13.276 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:13 smithi125 conmon[19936]: audit 2021-03-24T21:55:12.676432+0000 mon.a (mon.0) 707 : audit [INF] from='client.? v1:172.21.15.125:0/1573460350' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:55:13.277 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:13 smithi125 conmon[24927]: cluster 2021-03-24T21:55:13.277 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:13 smithi125 conmon[24927]: 2021-03-24T21:55:13.277 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:13 smithi125 conmon[24927]: 2021 2021-03-24T21:55:13.277 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:13 smithi125 conmon[24927]: -03-24T21:55:12. 2021-03-24T21:55:13.278 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:13 smithi125 conmon[24927]: 111991+0000 mon.a (mon.0) 706 : cluster [DBG] osdmap e143: 8 total, 8 up, 6 in 2021-03-24T21:55:13.278 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:13 smithi125 conmon[24927]: audit 2021-03 2021-03-24T21:55:13.278 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:13 smithi125 conmon[24927]: -24T21:55:12.676432+0000 mon.a 2021-03-24T21:55:13.278 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:13 smithi125 conmon[24927]: (mon.0) 707 : audit [INF] 2021-03-24T21:55:13.279 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:13 smithi125 conmon[24927]: from='client.? v1:172.21.15.125:0/1573460350' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:55:13.317 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:13.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:13.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:55:13.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:13 smithi142 conmon[20125]: cluster 2021-03-24T21:55:13.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:13 smithi142 conmon[20125]: 2021-03-24T21:55:13.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:13 smithi142 conmon[20125]: 2021 2021-03-24T21:55:13.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:13 smithi142 conmon[20125]: -03-24T21:55 2021-03-24T21:55:13.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:13 smithi142 conmon[20125]: :12.111991+0000 mon.a (mon.0) 706 : cluster [DBG] osdmap e143: 8 total, 8 up, 6 in 2021-03-24T21:55:13.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:13 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:55:13.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:13 smithi142 conmon[20125]: 03-24T21:55:12.676432+0000 mon.a (mon.0) 707 : audit [INF] 2021-03-24T21:55:13.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:13 smithi142 conmon[20125]: from='client.? v1:172.21.15.125:0/1573460350' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:55:13.424 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:13.486 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:13.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:13.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:55:13.593 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:13.653 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:13.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:13.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:55:13.759 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:13.774 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.3 dump_ops_in_flight 2021-03-24T21:55:13.818 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:13.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:13.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:55:13.924 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:13.986 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:13.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:13.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:55:14.093 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:14.132 INFO:tasks.rados.rados.0.smithi125.stdout:227: done (1 left) 2021-03-24T21:55:14.133 INFO:tasks.rados.rados.0.smithi125.stdout:228: read oid 31 snap -1 2021-03-24T21:55:14.133 INFO:tasks.rados.rados.0.smithi125.stdout:228: expect (ObjNum 73 snap 7 seq_num 73) 2021-03-24T21:55:14.133 INFO:tasks.rados.rados.0.smithi125.stdout:229: snap_create 2021-03-24T21:55:14.133 INFO:tasks.rados.rados.0.smithi125.stdout:226: finishing rollback tid 0 to smithi12556470-26 2021-03-24T21:55:14.133 INFO:tasks.rados.rados.0.smithi125.stdout:226: finishing rollback tid 1 to smithi12556470-26 2021-03-24T21:55:14.134 INFO:tasks.rados.rados.0.smithi125.stdout:226: finishing rollback tid 2 to smithi12556470-26 2021-03-24T21:55:14.134 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 26 v 26 (ObjNum 25 snap 0 seq_num 25) dirty exists 2021-03-24T21:55:14.151 INFO:tasks.rados.rados.0.smithi125.stdout:228: expect (ObjNum 73 snap 7 seq_num 73) 2021-03-24T21:55:14.158 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:14.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:14.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:55:14.213 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-24T21:55:14.262 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:14.317 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:14.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:14.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:55:14.367 INFO:journalctl@ceph.osd.4.smithi142.stdout:Mar 24 21:55:14 smithi142 conmon[24076]: debug 2021-03-24T21:55:14.367 INFO:journalctl@ceph.osd.4.smithi142.stdout:Mar 24 21:55:14 smithi142 conmon[24076]: 2021-03-24T21:55:14.133+0000 7fc561e38700 -1 osd.4 pg_epoch: 145 pg[7.1( v 145'85 (0'0,145'85] local-lis/les=108/109 n=12 ec=99/99 lis/c=108/108 les/c/f=109/111/0 sis=108) [4,3,0] r=0 lpr=108 luod=144'83 crt=145'85 lcod 144'81 mlcod 144'81 active+clean+snaptrim trimq=[20~1] ps=[17~1,1b~1,1f~1,23~1]] removing snap head 2021-03-24T21:55:14.368 INFO:journalctl@ceph.osd.4.smithi142.stdout:Mar 24 21:55:14 smithi142 conmon[24076]: 2021-03-24T21:55:14.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:14 smithi142 conmon[20125]: cluster 2021-03-24T21:55:12.722404+0000 mgr.y (mgr.14144) 329 : cluster 2021-03-24T21:55:14.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:14 smithi142 conmon[20125]: [DBG] pgmap v377: 148 pgs: 148 active+clean; 169 MiB data, 440 MiB used, 536 GiB / 536 GiB avail; 10 MiB/s rd, 4.2 MiB/s wr, 17 op/s 2021-03-24T21:55:14.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:14 smithi142 conmon[20125]: cluster 2021-03-24T21:55:13.106901+0000 mon.a (mon.0) 708 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:55:14.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:14 smithi142 conmon[20125]: audit 2021-03-24T21:55:13.118760+0000 2021-03-24T21:55:14.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:14 smithi142 conmon[20125]: mon.a (mon.0) 709 : audit [INF] from='client.? v1:172.21.15.125:0/1573460350' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T21:55:14.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:14 smithi142 conmon[20125]: cluster 2021-03-24T21:55:14.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:14 smithi142 conmon[20125]: 2021-03-24T21:55:13.118835+0000 mon.a (mon.0) 710 : cluster [DBG] 2021-03-24T21:55:14.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:14 smithi142 conmon[20125]: osdmap e144: 8 total, 8 up, 6 in 2021-03-24T21:55:14.389 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:55:14.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:14 smithi125 conmon[19936]: cluster 2021-03-24T21:55: 2021-03-24T21:55:14.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:14 smithi125 conmon[19936]: 12.722404+0000 mgr.y (mgr.14144) 329 : cluster [DBG] pgmap v377: 148 pgs: 148 active+clean; 169 MiB data, 440 MiB used, 536 GiB / 536 GiB avail; 10 MiB/s rd, 4.2 MiB/s wr, 17 op/s 2021-03-24T21:55:14.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:14 smithi125 conmon[19936]: cluster 2021-03-24T21:55:14.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:14 smithi125 conmon[19936]: 2021-03-24T21:55:13.106901+0000 2021-03-24T21:55:14.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:14 smithi125 conmon[19936]: mon.a (mon.0) 708 : cluster 2021-03-24T21:55:14.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:14 smithi125 conmon[19936]: [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:55:14.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:14 smithi125 conmon[19936]: audit 2021-03-24T21:55:13.118760+0000 mon.a (mon.0) 709 : audit [INF] from='client.? v1:172.21.15.125:0/1573460350' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T21:55:14.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:14 smithi125 conmon[19936]: cluster 2021-03-24T21:55:13.118835+0000 mon.a (mon.0) 710 : cluster [DBG] osdmap e144: 8 total, 8 up, 6 in 2021-03-24T21:55:14.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:14 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:55:14.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:14 smithi125 conmon[24927]: 21:55:12.722404+0000 mgr.y (mgr.14144) 329 : cluster [DBG] pgmap v377: 148 pgs: 148 active+clean; 169 MiB data, 440 MiB used, 536 GiB / 536 GiB avail; 10 MiB/s rd, 4.2 MiB/s wr, 17 op/s 2021-03-24T21:55:14.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:14 smithi125 conmon[24927]: cluster 2021-03-24T21:55:13.106901+0000 mon.a (mon.0) 2021-03-24T21:55:14.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:14 smithi125 conmon[24927]: 708 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:55:14.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:14 smithi125 conmon[24927]: audit 2021-03-24T21:55:14.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:14 smithi125 conmon[24927]: 2021-03-24T21:55:13.118760+0000 mon.a (mon.0) 709 : audit [INF] 2021-03-24T21:55:14.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:14 smithi125 conmon[24927]: from='client.? v1:172.21.15.125:0/1573460350' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T21:55:14.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:14 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:55:14.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:14 smithi125 conmon[24927]: -03-24T21:55:13.118835+0000 mon.a (mon.0) 2021-03-24T21:55:14.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:14 smithi125 conmon[24927]: 710 : cluster [DBG] osdmap e144: 8 total, 8 up, 6 in 2021-03-24T21:55:14.425 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:14.483 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:14.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:14.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:55:14.569 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:55:14.589 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:14.640 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:14.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:14.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:55:14.752 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:14.809 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:14.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:14.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:55:14.914 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:14.972 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:14.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:14.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:55:15.079 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:15.141 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:15.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:15.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:55:15.158 INFO:tasks.rados.rados.0.smithi125.stdout:226: done (2 left) 2021-03-24T21:55:15.158 INFO:tasks.rados.rados.0.smithi125.stdout:228: done (1 left) 2021-03-24T21:55:15.158 INFO:tasks.rados.rados.0.smithi125.stdout:229: done (0 left) 2021-03-24T21:55:15.159 INFO:tasks.rados.rados.0.smithi125.stdout:230: write oid 18 current snap is 23 2021-03-24T21:55:15.159 INFO:tasks.rados.rados.0.smithi125.stdout:230: seq_num 112 ranges {682384=456830,1721042=423699,2809728=454936,3731357=151027} 2021-03-24T21:55:15.167 INFO:tasks.rados.rados.0.smithi125.stdout:230: writing smithi12556470-18 from 682384 to 1139214 tid 1 2021-03-24T21:55:15.173 INFO:tasks.rados.rados.0.smithi125.stdout:230: writing smithi12556470-18 from 1721042 to 2144741 tid 2 2021-03-24T21:55:15.179 INFO:tasks.rados.rados.0.smithi125.stdout:230: writing smithi12556470-18 from 2809728 to 3264664 tid 3 2021-03-24T21:55:15.183 INFO:tasks.rados.rados.0.smithi125.stdout:230: writing smithi12556470-18 from 3731357 to 3882384 tid 4 2021-03-24T21:55:15.183 INFO:tasks.rados.rados.0.smithi125.stdout:231: read oid 27 snap -1 2021-03-24T21:55:15.183 INFO:tasks.rados.rados.0.smithi125.stdout:231: expect (ObjNum 101 snap 20 seq_num 101) 2021-03-24T21:55:15.184 INFO:tasks.rados.rados.0.smithi125.stdout:232: snap_remove snap 21 2021-03-24T21:55:15.247 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:15.258 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:15 smithi125 conmon[24927]: cluster 2021-03-24T21:55:13.518301 2021-03-24T21:55:15.259 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:15 smithi125 conmon[24927]: +0000 osd.3 (osd.3) 46 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:55:15.259 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:15 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:55:15.259 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:15 smithi125 conmon[24927]: :55:13.518627+0000 osd.3 (osd.3) 47 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:55:15.259 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:15 smithi125 conmon[24927]: cluster 2021-03-24T21:55:14.131346+0000 mon.a (mon.0) 711 : cluster [DBG] osdmap e145: 8 total, 8 up, 6 in 2021-03-24T21:55:15.260 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:15 smithi125 conmon[19936]: cluster 2021-03-24T21:55:13.518301+0000 osd.3 2021-03-24T21:55:15.260 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:15 smithi125 conmon[19936]: (osd.3) 46 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:55:15.260 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:15 smithi125 conmon[19936]: cluster 2021-03-24T21:55:13.518627+0000 osd.3 (osd.3) 47 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:55:15.261 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:15 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:55:15.261 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:15 smithi125 conmon[19936]: 03-24T21:55:14.131346+0000 mon.a (mon.0) 711 : cluster [DBG] osdmap e145: 8 total, 8 up, 6 in 2021-03-24T21:55:15.300 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:15.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:15.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:55:15.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:15 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:55:15.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:15 smithi142 conmon[20125]: T21:55:13.518301+0000 osd.3 (osd.3) 46 : cluster 2021-03-24T21:55:15.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:15 smithi142 conmon[20125]: [DBG] purged_snaps scrub starts 2021-03-24T21:55:15.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:15 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:55:15.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:15 smithi142 conmon[20125]: -03-24T21:55:13.518627+0000 osd.3 (osd.3) 47 : cluster 2021-03-24T21:55:15.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:15 smithi142 conmon[20125]: [DBG] purged_snaps scrub ok 2021-03-24T21:55:15.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:15 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:55:15.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:15 smithi142 conmon[20125]: 55:14.131346+0000 mon.a (mon 2021-03-24T21:55:15.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:15 smithi142 conmon[20125]: .0) 711 : cluster [DBG] osdmap e145: 8 total, 8 up, 6 in 2021-03-24T21:55:15.405 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:15.464 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:15.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:15.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:15.568 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:15.627 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:15.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:15.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:55:15.733 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:15.792 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:15.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:15.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:55:15.897 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:15.953 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:15.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:15.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:55:16.058 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:16.113 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:16.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:16.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:55:16.169 INFO:tasks.rados.rados.0.smithi125.stdout:230: finishing write tid 1 to smithi12556470-18 2021-03-24T21:55:16.169 INFO:tasks.rados.rados.0.smithi125.stdout:232: done (2 left) 2021-03-24T21:55:16.169 INFO:tasks.rados.rados.0.smithi125.stdout:233: write (excl) oid 29 current snap is 23 2021-03-24T21:55:16.170 INFO:tasks.rados.rados.0.smithi125.stdout:230: finishing write tid 2 to smithi12556470-18 2021-03-24T21:55:16.170 INFO:tasks.rados.rados.0.smithi125.stdout:230: finishing write tid 3 to smithi12556470-18 2021-03-24T21:55:16.170 INFO:tasks.rados.rados.0.smithi125.stdout:230: finishing write tid 4 to smithi12556470-18 2021-03-24T21:55:16.170 INFO:tasks.rados.rados.0.smithi125.stdout:230: finishing write tid 5 to smithi12556470-18 2021-03-24T21:55:16.170 INFO:tasks.rados.rados.0.smithi125.stdout:233: seq_num 113 ranges {699191=532079,1779524=567357,2837151=666208} 2021-03-24T21:55:16.176 INFO:tasks.rados.rados.0.smithi125.stdout:233: writing smithi12556470-29 from 699191 to 1231270 tid 1 2021-03-24T21:55:16.183 INFO:tasks.rados.rados.0.smithi125.stdout:233: writing smithi12556470-29 from 1779524 to 2346881 tid 2 2021-03-24T21:55:16.190 INFO:tasks.rados.rados.0.smithi125.stdout:233: writing smithi12556470-29 from 2837151 to 3503359 tid 3 2021-03-24T21:55:16.190 INFO:tasks.rados.rados.0.smithi125.stdout:234: write (excl) oid 41 current snap is 23 2021-03-24T21:55:16.191 INFO:tasks.rados.rados.0.smithi125.stdout:230: finishing write tid 6 to smithi12556470-18 2021-03-24T21:55:16.191 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 18 v 29 (ObjNum 112 snap 23 seq_num 112) dirty exists 2021-03-24T21:55:16.191 INFO:tasks.rados.rados.0.smithi125.stdout:230: left oid 18 (ObjNum 112 snap 23 seq_num 112) 2021-03-24T21:55:16.191 INFO:tasks.rados.rados.0.smithi125.stdout:234: seq_num 114 ranges {715998=607328,1838006=427368,2864574=477480,3863835=52163} 2021-03-24T21:55:16.197 INFO:tasks.rados.rados.0.smithi125.stdout:234: writing smithi12556470-41 from 715998 to 1323326 tid 1 2021-03-24T21:55:16.203 INFO:tasks.rados.rados.0.smithi125.stdout:234: writing smithi12556470-41 from 1838006 to 2265374 tid 2 2021-03-24T21:55:16.209 INFO:tasks.rados.rados.0.smithi125.stdout:234: writing smithi12556470-41 from 2864574 to 3342054 tid 3 2021-03-24T21:55:16.213 INFO:tasks.rados.rados.0.smithi125.stdout:234: writing smithi12556470-41 from 3863835 to 3915998 tid 4 2021-03-24T21:55:16.213 INFO:tasks.rados.rados.0.smithi125.stdout:230: done (3 left) 2021-03-24T21:55:16.213 INFO:tasks.rados.rados.0.smithi125.stdout:235: read oid 13 snap -1 2021-03-24T21:55:16.213 INFO:tasks.rados.rados.0.smithi125.stdout:235: expect (ObjNum 63 snap 4 seq_num 63) 2021-03-24T21:55:16.214 INFO:tasks.rados.rados.0.smithi125.stdout:236: read oid 5 snap -1 2021-03-24T21:55:16.214 INFO:tasks.rados.rados.0.smithi125.stdout:236: expect (ObjNum 103 snap 20 seq_num 103) 2021-03-24T21:55:16.214 INFO:tasks.rados.rados.0.smithi125.stdout:237: snap_remove snap 13 2021-03-24T21:55:16.220 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:16.274 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:16.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:16.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:55:16.343 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd unset noscrub 2021-03-24T21:55:16.357 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:16 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:55:16.357 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:16 smithi125 conmon[19936]: 55:14.723180+0000 mgr.y (mgr.14144) 330 : cluster [DBG] pgmap v380: 148 pgs: 148 active+clean; 185 MiB data, 479 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 3.6 MiB/s wr, 12 op/s 2021-03-24T21:55:16.357 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:16 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:55:16.358 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:16 smithi125 conmon[19936]: :55:15.157543+0000 mon.a (mon.0) 712 : cluster [DBG] osdmap e146: 8 total, 8 up, 6 in 2021-03-24T21:55:16.358 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:16 smithi125 conmon[19936]: 2021-03-24T21:55:16.358 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:16 smithi125 conmon[24927]: cluster 2021-03-24T21:55:14 2021-03-24T21:55:16.358 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:16 smithi125 conmon[24927]: .723180+0000 mgr.y (mgr.14144) 330 : cluster [DBG] pgmap v380: 148 pgs: 148 active+clean; 185 MiB data, 479 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 3.6 MiB/s wr, 12 op/s 2021-03-24T21:55:16.359 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:16 smithi125 conmon[24927]: cluster 2021-03-24T21:55:16.359 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:16 smithi125 conmon[24927]: 2021-03-24T21:55:15.157543+0000 mon.a (mon. 2021-03-24T21:55:16.359 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:16 smithi125 conmon[24927]: 0) 712 : cluster [DBG] osdmap e146: 8 total, 8 up, 6 in 2021-03-24T21:55:16.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:16 smithi142 conmon[20125]: cluster 2021-03-24T21:55 2021-03-24T21:55:16.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:16 smithi142 conmon[20125]: :14.723180+0000 mgr.y (mgr.14144) 330 : cluster [DBG] pgmap v380: 148 pgs: 148 active+clean; 185 MiB data, 479 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 3.6 MiB/s wr, 12 op/s 2021-03-24T21:55:16.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:16 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:55:16.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:16 smithi142 conmon[20125]: 03-24T21:55:15.157543+0000 mon.a (mon.0 2021-03-24T21:55:16.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:16 smithi142 conmon[20125]: ) 712 : cluster [DBG] osdmap e146: 8 total, 8 up, 6 in 2021-03-24T21:55:16.381 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:16.430 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:16.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:16.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:55:16.534 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:16.587 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:16.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:16.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:55:16.693 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:16.754 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:16.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:16.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:55:16.761 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:55:16.861 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:16.912 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:16.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:16.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:55:17.018 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:17.075 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:17.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:17.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:55:17.136 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-24T21:55:17.136 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-24T21:55:17.137 INFO:teuthology.orchestra.run.smithi125.stdout: "num_ops": 0 2021-03-24T21:55:17.137 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-24T21:55:17.159 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 7] out_osds: [6, 1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-03-24T21:55:17.160 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-24T21:55:17.161 INFO:tasks.thrashosds.thrasher:fixing pg num pool iscsi 2021-03-24T21:55:17.161 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph pg dump --format=json 2021-03-24T21:55:17.178 INFO:tasks.rados.rados.0.smithi125.stdout:237: done (5 left) 2021-03-24T21:55:17.179 INFO:tasks.rados.rados.0.smithi125.stdout:238: delete oid 28 current snap is 23 2021-03-24T21:55:17.179 INFO:tasks.rados.rados.0.smithi125.stdout:231: expect (ObjNum 101 snap 20 seq_num 101) 2021-03-24T21:55:17.182 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:17.245 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:17.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:17.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:55:17.353 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:17.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:17 smithi142 conmon[20125]: cluster 2021-03-24T21:55 2021-03-24T21:55:17.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:17 smithi142 conmon[20125]: :16.168449+0000 mon.a (mon.0) 713 : cluster [DBG] osdmap e147: 8 total, 8 up, 6 in 2021-03-24T21:55:17.368 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T21:55:17.392 INFO:tasks.rados.rados.0.smithi125.stdout:233: finishing write tid 1 to smithi12556470-29 2021-03-24T21:55:17.392 INFO:tasks.rados.rados.0.smithi125.stdout:233: finishing write tid 2 to smithi12556470-29 2021-03-24T21:55:17.392 INFO:tasks.rados.rados.0.smithi125.stdout:233: finishing write tid 3 to smithi12556470-29 2021-03-24T21:55:17.393 INFO:tasks.rados.rados.0.smithi125.stdout:233: finishing write tid 4 to smithi12556470-29 2021-03-24T21:55:17.393 INFO:tasks.rados.rados.0.smithi125.stdout:233: finishing write tid 5 to smithi12556470-29 2021-03-24T21:55:17.393 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 29 v 41 (ObjNum 113 snap 23 seq_num 113) dirty exists 2021-03-24T21:55:17.393 INFO:tasks.rados.rados.0.smithi125.stdout:233: left oid 29 (ObjNum 113 snap 23 seq_num 113) 2021-03-24T21:55:17.394 INFO:tasks.rados.rados.0.smithi125.stdout:234: finishing write tid 1 to smithi12556470-41 2021-03-24T21:55:17.394 INFO:tasks.rados.rados.0.smithi125.stdout:234: finishing write tid 2 to smithi12556470-41 2021-03-24T21:55:17.394 INFO:tasks.rados.rados.0.smithi125.stdout:234: finishing write tid 3 to smithi12556470-41 2021-03-24T21:55:17.394 INFO:tasks.rados.rados.0.smithi125.stdout:234: finishing write tid 4 to smithi12556470-41 2021-03-24T21:55:17.395 INFO:tasks.rados.rados.0.smithi125.stdout:234: finishing write tid 5 to smithi12556470-41 2021-03-24T21:55:17.395 INFO:tasks.rados.rados.0.smithi125.stdout:234: finishing write tid 6 to smithi12556470-41 2021-03-24T21:55:17.395 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 41 v 76 (ObjNum 114 snap 23 seq_num 114) dirty exists 2021-03-24T21:55:17.395 INFO:tasks.rados.rados.0.smithi125.stdout:234: left oid 41 (ObjNum 114 snap 23 seq_num 114) 2021-03-24T21:55:17.395 INFO:tasks.rados.rados.0.smithi125.stdout:235: expect (ObjNum 63 snap 4 seq_num 63) 2021-03-24T21:55:17.402 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T21:55:17.410 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:17.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:17.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:55:17.428 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T21:55:17.438 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:17 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:55:17.438 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:17 smithi125 conmon[19936]: 03-24T21:55:16.168449+0000 mon.a (mon.0) 713 : cluster [DBG] osdmap e147: 8 total, 8 up, 6 in 2021-03-24T21:55:17.438 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:17 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:55:17.439 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:17 smithi125 conmon[24927]: 55:16.168449+0000 mon.a (mon.0) 713 : cluster [DBG] osdmap e147: 8 total, 8 up, 6 in 2021-03-24T21:55:17.457 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T21:55:17.498 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T21:55:17.516 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:17.526 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T21:55:17.547 INFO:tasks.rados.rados.0.smithi125.stdout:236: expect (ObjNum 103 snap 20 seq_num 103) 2021-03-24T21:55:17.550 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T21:55:17.574 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T21:55:17.576 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-24T21:55:17.576 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-24T21:55:17.576 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-24T21:55:17.576 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-24T21:55:17.577 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-24T21:55:17.577 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-24T21:55:17.577 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-24T21:55:17.577 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-24T21:55:17.577 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:17.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:17.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:55:17.641 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:55:17.682 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:17.732 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:17.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:17.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:55:17.775 INFO:tasks.rados.rados.0.smithi125.stdout:231: done (5 left) 2021-03-24T21:55:17.776 INFO:tasks.rados.rados.0.smithi125.stdout:233: done (4 left) 2021-03-24T21:55:17.776 INFO:tasks.rados.rados.0.smithi125.stdout:234: done (3 left) 2021-03-24T21:55:17.776 INFO:tasks.rados.rados.0.smithi125.stdout:235: done (2 left) 2021-03-24T21:55:17.777 INFO:tasks.rados.rados.0.smithi125.stdout:236: done (1 left) 2021-03-24T21:55:17.777 INFO:tasks.rados.rados.0.smithi125.stdout:238: done (0 left) 2021-03-24T21:55:17.778 INFO:tasks.rados.rados.0.smithi125.stdout:239: snap_remove snap 16 2021-03-24T21:55:17.839 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:17.896 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:17.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:17.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:55:18.002 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:18.058 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:18.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:18.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:55:18.165 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:18.202 INFO:tasks.rados.rados.0.smithi125.stdout:239: done (0 left) 2021-03-24T21:55:18.203 INFO:tasks.rados.rados.0.smithi125.stdout:240: write oid 27 current snap is 23 2021-03-24T21:55:18.203 INFO:tasks.rados.rados.0.smithi125.stdout:240: seq_num 116 ranges {749612=474179,1954970=431037,2919420=616377} 2021-03-24T21:55:18.204 INFO:tasks.rados.rados.0.smithi125.stdout:240: writing smithi12556470-27 from 749612 to 1223791 tid 1 2021-03-24T21:55:18.211 INFO:tasks.rados.rados.0.smithi125.stdout:240: writing smithi12556470-27 from 1954970 to 2386007 tid 2 2021-03-24T21:55:18.219 INFO:tasks.rados.rados.0.smithi125.stdout:240: writing smithi12556470-27 from 2919420 to 3535797 tid 3 2021-03-24T21:55:18.219 INFO:tasks.rados.rados.0.smithi125.stdout:241: snap_remove snap 17 2021-03-24T21:55:18.230 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:18.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:18.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:55:18.334 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:18.394 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:18.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:18.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:55:18.500 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:18.514 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:18 smithi125 conmon[19936]: cluster 2021-03-24T21:55: 2021-03-24T21:55:18.514 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:18 smithi125 conmon[19936]: 16.723908+0000 mgr.y (mgr.14144) 331 : cluster [DBG] pgmap v383: 148 pgs: 148 active+clean; 185 MiB data, 483 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 3.6 MiB/s wr, 12 op/s 2021-03-24T21:55:18.514 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:18 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:55:18.515 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:18 smithi125 conmon[19936]: -24T21:55:17.076658+0000 osd.6 (osd.6) 19 : cluster [DBG] 2021-03-24T21:55:18.515 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:18 smithi125 conmon[19936]: purged_snaps scrub starts 2021-03-24T21:55:18.515 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:18 smithi125 conmon[19936]: cluster 2021-03-24T21:55: 2021-03-24T21:55:18.515 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:18 smithi125 conmon[19936]: 17.076823+0000 osd.6 (osd.6) 20 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:55:18.515 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:18 smithi125 conmon[19936]: 2021-03-24T21:55:18.516 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:18 smithi125 conmon[19936]: cluster 2021-03-24T21:55:17.174681+0000 2021-03-24T21:55:18.516 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:18 smithi125 conmon[19936]: mon.a (mon.0) 714 : cluster [DBG] osdmap e148: 8 total, 8 up, 6 in 2021-03-24T21:55:18.516 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:18 smithi125 conmon[19936]: audit 2021-03-24T21:55:17. 2021-03-24T21:55:18.516 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:18 smithi125 conmon[19936]: 344211+0000 mon.c (mon.2) 29 : audit [DBG] 2021-03-24T21:55:18.516 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:18 smithi125 conmon[19936]: from='client.? v1:172.21.15.125:0/2979759918' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:55:18.517 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:18 smithi125 conmon[24927]: cluster 2021-03-24T21:55 2021-03-24T21:55:18.517 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:18 smithi125 conmon[24927]: :16.723908+0000 mgr.y (mgr.14144) 331 : cluster [DBG] pgmap v383: 148 pgs: 148 active+clean; 185 MiB data, 483 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 3.6 MiB/s wr, 12 op/s 2021-03-24T21:55:18.517 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:18 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:55:18.518 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:18 smithi125 conmon[24927]: 21:55:17.076658+0000 osd.6 (osd. 2021-03-24T21:55:18.518 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:18 smithi125 conmon[24927]: 6) 19 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:55:18.518 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:18 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:55:18.518 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:18 smithi125 conmon[24927]: T21:55:17.076823+0000 osd.6 (osd 2021-03-24T21:55:18.518 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:18 smithi125 conmon[24927]: .6) 20 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:55:18.519 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:18 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:55:18.519 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:18 smithi125 conmon[24927]: 24T21:55:17.174681+0000 mon.a (mon 2021-03-24T21:55:18.519 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:18 smithi125 conmon[24927]: .0) 714 : cluster [DBG] osdmap e148: 8 total, 8 up, 6 in 2021-03-24T21:55:18.519 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:18 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T21:55:18.519 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:18 smithi125 conmon[24927]: 24T21:55:17.344211+0000 mon.c ( 2021-03-24T21:55:18.520 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:18 smithi125 conmon[24927]: mon.2) 29 : audit [DBG] from='client.? v1:172.21.15.125:0/2979759918' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:55:18.558 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:18.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:18.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:18.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:18 smithi142 conmon[20125]: cluster 2021-03-24T21:55 2021-03-24T21:55:18.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:18 smithi142 conmon[20125]: :16.723908+0000 mgr.y (mgr.14144) 331 : cluster [DBG] pgmap v383: 148 pgs: 148 active+clean; 185 MiB data, 483 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 3.6 MiB/s wr, 12 op/s 2021-03-24T21:55:18.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:18 smithi142 conmon[20125]: cluster 2021-03-24T21:55:18.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:18 smithi142 conmon[20125]: 2021-03-24T21:55:17.076658 2021-03-24T21:55:18.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:18 smithi142 conmon[20125]: +0000 osd.6 (osd.6) 19 2021-03-24T21:55:18.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:18 smithi142 conmon[20125]: : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:55:18.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:18 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:55:18.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:18 smithi142 conmon[20125]: -03-24T21:55:17. 2021-03-24T21:55:18.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:18 smithi142 conmon[20125]: 076823+0000 osd.6 (osd.6 2021-03-24T21:55:18.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:18 smithi142 conmon[20125]: ) 20 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:55:18.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:18 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:55:18.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:18 smithi142 conmon[20125]: -03-24T21:55:17 2021-03-24T21:55:18.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:18 smithi142 conmon[20125]: .174681+0000 mon.a (mon. 2021-03-24T21:55:18.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:18 smithi142 conmon[20125]: 0) 714 : cluster [DBG] osdmap e148: 8 total, 8 up, 6 in 2021-03-24T21:55:18.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:18 smithi142 conmon[20125]: 2021-03-24T21:55:18.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:18 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:55:18.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:18 smithi142 conmon[20125]: 03-24T21:55: 2021-03-24T21:55:18.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:18 smithi142 conmon[20125]: 17.344211+0000 mon.c (mon 2021-03-24T21:55:18.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:18 smithi142 conmon[20125]: .2) 29 : audit [DBG] 2021-03-24T21:55:18.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:18 smithi142 conmon[20125]: from='client.? v1:172.21.15.125:0/2979759918' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:55:18.663 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:18.714 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:18.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:18.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:18.821 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:18.880 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:18.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:18.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:55:18.986 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:19.045 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:19.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:19.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:19.150 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:19.196 INFO:tasks.rados.rados.0.smithi125.stdout:241: done (1 left) 2021-03-24T21:55:19.197 INFO:tasks.rados.rados.0.smithi125.stdout:242: delete oid 38 current snap is 23 2021-03-24T21:55:19.197 INFO:tasks.rados.rados.0.smithi125.stdout:240: finishing write tid 1 to smithi12556470-27 2021-03-24T21:55:19.197 INFO:tasks.rados.rados.0.smithi125.stdout:240: finishing write tid 2 to smithi12556470-27 2021-03-24T21:55:19.197 INFO:tasks.rados.rados.0.smithi125.stdout:240: finishing write tid 3 to smithi12556470-27 2021-03-24T21:55:19.198 INFO:tasks.rados.rados.0.smithi125.stdout:240: finishing write tid 4 to smithi12556470-27 2021-03-24T21:55:19.198 INFO:tasks.rados.rados.0.smithi125.stdout:240: finishing write tid 5 to smithi12556470-27 2021-03-24T21:55:19.198 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 27 v 82 (ObjNum 116 snap 23 seq_num 116) dirty exists 2021-03-24T21:55:19.198 INFO:tasks.rados.rados.0.smithi125.stdout:240: left oid 27 (ObjNum 116 snap 23 seq_num 116) 2021-03-24T21:55:19.205 INFO:tasks.rados.rados.0.smithi125.stdout:240: done (1 left) 2021-03-24T21:55:19.206 INFO:tasks.rados.rados.0.smithi125.stdout:242: done (0 left) 2021-03-24T21:55:19.206 INFO:tasks.rados.rados.0.smithi125.stdout:243: delete oid 20 current snap is 23 2021-03-24T21:55:19.209 INFO:tasks.rados.rados.0.smithi125.stdout:243: done (0 left) 2021-03-24T21:55:19.210 INFO:tasks.rados.rados.0.smithi125.stdout:244: copy_from oid 48 from oid 47 current snap is 23 2021-03-24T21:55:19.210 INFO:tasks.rados.rados.0.smithi125.stdout:245: copy_from oid 34 from oid 25 current snap is 23 2021-03-24T21:55:19.210 INFO:tasks.rados.rados.0.smithi125.stdout:246: copy_from oid 19 from oid 10 current snap is 23 2021-03-24T21:55:19.211 INFO:tasks.rados.rados.0.smithi125.stdout:247: read oid 9 snap -1 2021-03-24T21:55:19.211 INFO:tasks.rados.rados.0.smithi125.stdout:247: expect (ObjNum 68 snap 7 seq_num 68) 2021-03-24T21:55:19.211 INFO:tasks.rados.rados.0.smithi125.stdout:248: read oid 46 snap -1 2021-03-24T21:55:19.211 INFO:tasks.rados.rados.0.smithi125.stdout:248: expect (ObjNum 45 snap 0 seq_num 45) 2021-03-24T21:55:19.211 INFO:tasks.rados.rados.0.smithi125.stdout:249: snap_remove snap 15 2021-03-24T21:55:19.218 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:19.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:19.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:55:19.325 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:19.380 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:19.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:19.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:55:19.430 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:19 smithi125 conmon[19936]: cluster 2021-03-24T21:55:18.191343+0000 mon.a ( 2021-03-24T21:55:19.430 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:19 smithi125 conmon[19936]: mon.0) 715 : cluster [DBG] osdmap e149: 8 total, 8 up, 6 in 2021-03-24T21:55:19.431 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:19 smithi125 conmon[24927]: cluster 2021-03-24T21:55:18.191343+0000 mon.a (mon.0) 715 : cluster [DBG] osdmap e149: 8 total, 8 up, 6 in 2021-03-24T21:55:19.487 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:19.543 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:19.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:19.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:55:19.561 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.3 dump_blocked_ops 2021-03-24T21:55:19.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:19 smithi142 conmon[20125]: cluster 2021-03-24T21:55:19.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:19 smithi142 conmon[20125]: 2021-03-24T21: 2021-03-24T21:55:19.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:19 smithi142 conmon[20125]: 55:18.191343+0000 mon.a (mon.0) 715 : cluster [DBG] osdmap e149: 8 total, 8 up, 6 in 2021-03-24T21:55:19.650 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:19.702 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:19.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:19.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:55:19.808 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:19.865 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:19.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:19.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:55:19.971 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:20.029 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:20.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:20.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:55:20.103 INFO:tasks.rados.rados.0.smithi125.stdout:249: done (5 left) 2021-03-24T21:55:20.104 INFO:tasks.rados.rados.0.smithi125.stdout:250: rollback oid 26 current snap is 23 2021-03-24T21:55:20.104 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 26 to 10 2021-03-24T21:55:20.104 INFO:tasks.rados.rados.0.smithi125.stdout:248: expect (ObjNum 45 snap 0 seq_num 45) 2021-03-24T21:55:20.135 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:20.185 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:20.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:20.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:55:20.208 INFO:tasks.rados.rados.0.smithi125.stdout:245: finishing copy_from to smithi12556470-34 2021-03-24T21:55:20.209 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 34 v 6 (ObjNum 24 snap 0 seq_num 24) dirty exists 2021-03-24T21:55:20.209 INFO:tasks.rados.rados.0.smithi125.stdout:245: finishing copy_from racing read to smithi12556470-34 2021-03-24T21:55:20.209 INFO:tasks.rados.rados.0.smithi125.stdout:247: expect (ObjNum 68 snap 7 seq_num 68) 2021-03-24T21:55:20.291 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:20.354 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:20.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:20.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:55:20.366 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:20 smithi125 conmon[19936]: cluster 2021-03-24T21:55:18.724609+0000 mgr.y (mgr.14144) 332 : cluster [DBG] pgmap v386: 148 pgs: 148 active+clean; 176 MiB data, 480 MiB used, 536 GiB / 536 GiB avail; 3.0 MiB/s rd, 1 op/s 2021-03-24T21:55:20.367 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:20 smithi125 conmon[19936]: 2021-03-24T21:55:20.367 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:20 smithi125 conmon[19936]: cluster 2021-03-24T21:55:19.194379+0000 mon.a (mon.0) 716 : cluster [DBG] osdmap e150: 8 total, 8 up, 6 in 2021-03-24T21:55:20.367 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:20 smithi125 conmon[19936]: cluster 2021-03-24T21:55:20.098162+0000 mon.a (mon.0) 717 : cluster [DBG] osdmap e151: 8 total, 8 up, 6 in 2021-03-24T21:55:20.368 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:20 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:55:20.368 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:20 smithi125 conmon[24927]: 24T21:55:18.724609+0000 mgr.y (mgr.14144) 332 : 2021-03-24T21:55:20.368 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:20 smithi125 conmon[24927]: cluster [DBG] pgmap v386: 148 pgs: 148 active+clean; 176 MiB data, 480 MiB used, 536 GiB / 536 GiB avail; 3.0 MiB/s rd, 1 op/s 2021-03-24T21:55:20.368 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:20 smithi125 conmon[24927]: cluster 2021-03-24T21:55:20.369 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:20 smithi125 conmon[24927]: 2021-03-24T21: 2021-03-24T21:55:20.369 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:20 smithi125 conmon[24927]: 55:19.194379+0000 2021-03-24T21:55:20.369 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:20 smithi125 conmon[24927]: mon.a (mon.0) 2021-03-24T21:55:20.369 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:20 smithi125 conmon[24927]: 716 : cluster [DBG] 2021-03-24T21:55:20.370 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:20 smithi125 conmon[24927]: osdmap e150: 8 total, 8 up, 6 in 2021-03-24T21:55:20.370 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:20 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:55:20.370 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:20 smithi125 conmon[24927]: -03-24 2021-03-24T21:55:20.370 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:20 smithi125 conmon[24927]: T21:55: 2021-03-24T21:55:20.370 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:20 smithi125 conmon[24927]: 20.098162+0000 2021-03-24T21:55:20.371 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:20 smithi125 conmon[24927]: mon.a (mon.0 2021-03-24T21:55:20.371 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:20 smithi125 conmon[24927]: ) 717 : cluster 2021-03-24T21:55:20.371 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:20 smithi125 conmon[24927]: [DBG] osdmap e151: 8 total, 8 up, 6 in 2021-03-24T21:55:20.402 INFO:tasks.rados.rados.0.smithi125.stdout:244: finishing copy_from to smithi12556470-48 2021-03-24T21:55:20.402 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 48 v 24 (ObjNum 62 snap 4 seq_num 62) dirty exists 2021-03-24T21:55:20.402 INFO:tasks.rados.rados.0.smithi125.stdout:244: finishing copy_from racing read to smithi12556470-48 2021-03-24T21:55:20.403 INFO:tasks.rados.rados.0.smithi125.stdout:246: finishing copy_from to smithi12556470-19 2021-03-24T21:55:20.403 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 19 v 3 (ObjNum 9 snap 0 seq_num 9) dirty exists 2021-03-24T21:55:20.403 INFO:tasks.rados.rados.0.smithi125.stdout:246: finishing copy_from racing read to smithi12556470-19 2021-03-24T21:55:20.403 INFO:tasks.rados.rados.0.smithi125.stdout:250: finishing rollback tid 0 to smithi12556470-26 2021-03-24T21:55:20.403 INFO:tasks.rados.rados.0.smithi125.stdout:250: finishing rollback tid 1 to smithi12556470-26 2021-03-24T21:55:20.404 INFO:tasks.rados.rados.0.smithi125.stdout:244: done (5 left) 2021-03-24T21:55:20.404 INFO:tasks.rados.rados.0.smithi125.stdout:245: done (4 left) 2021-03-24T21:55:20.404 INFO:tasks.rados.rados.0.smithi125.stdout:246: done (3 left) 2021-03-24T21:55:20.404 INFO:tasks.rados.rados.0.smithi125.stdout:247: done (2 left) 2021-03-24T21:55:20.405 INFO:tasks.rados.rados.0.smithi125.stdout:248: done (1 left) 2021-03-24T21:55:20.405 INFO:tasks.rados.rados.0.smithi125.stdout:251: read oid 47 snap 10 2021-03-24T21:55:20.405 INFO:tasks.rados.rados.0.smithi125.stdout:251: expect (ObjNum 67 snap 7 seq_num 67) 2021-03-24T21:55:20.405 INFO:tasks.rados.rados.0.smithi125.stdout:252: copy_from oid 12 from oid 40 current snap is 23 2021-03-24T21:55:20.405 INFO:tasks.rados.rados.0.smithi125.stdout:250: finishing rollback tid 2 to smithi12556470-26 2021-03-24T21:55:20.406 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 26 v 57 (ObjNum 25 snap 0 seq_num 25) dirty exists 2021-03-24T21:55:20.406 INFO:tasks.rados.rados.0.smithi125.stdout:250: done (2 left) 2021-03-24T21:55:20.406 INFO:tasks.rados.rados.0.smithi125.stdout:253: delete oid 30 current snap is 23 2021-03-24T21:55:20.407 INFO:tasks.rados.rados.0.smithi125.stdout:253: done (2 left) 2021-03-24T21:55:20.407 INFO:tasks.rados.rados.0.smithi125.stdout:254: write oid 36 current snap is 23 2021-03-24T21:55:20.407 INFO:tasks.rados.rados.0.smithi125.stdout:254: seq_num 120 ranges {416840=775175,1672545=344295} 2021-03-24T21:55:20.416 INFO:tasks.rados.rados.0.smithi125.stdout:254: writing smithi12556470-36 from 416840 to 1192015 tid 1 2021-03-24T21:55:20.421 INFO:tasks.rados.rados.0.smithi125.stdout:254: writing smithi12556470-36 from 1672545 to 2016840 tid 2 2021-03-24T21:55:20.421 INFO:tasks.rados.rados.0.smithi125.stdout:255: write oid 38 current snap is 23 2021-03-24T21:55:20.421 INFO:tasks.rados.rados.0.smithi125.stdout:251: expect (ObjNum 67 snap 7 seq_num 67) 2021-03-24T21:55:20.461 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:20.516 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:20.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:20.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:55:20.568 INFO:tasks.rados.rados.0.smithi125.stdout:255: seq_num 121 ranges {433647=450424,1331027=582033} 2021-03-24T21:55:20.573 INFO:tasks.rados.rados.0.smithi125.stdout:255: writing smithi12556470-38 from 433647 to 884071 tid 1 2021-03-24T21:55:20.580 INFO:tasks.rados.rados.0.smithi125.stdout:255: writing smithi12556470-38 from 1331027 to 1913060 tid 2 2021-03-24T21:55:20.580 INFO:tasks.rados.rados.0.smithi125.stdout:251: done (3 left) 2021-03-24T21:55:20.581 INFO:tasks.rados.rados.0.smithi125.stdout:256: read oid 1 snap -1 2021-03-24T21:55:20.581 INFO:tasks.rados.rados.0.smithi125.stdout:256: expect (ObjNum 79 snap 9 seq_num 79) 2021-03-24T21:55:20.581 INFO:tasks.rados.rados.0.smithi125.stdout:257: delete oid 34 current snap is 23 2021-03-24T21:55:20.581 INFO:tasks.rados.rados.0.smithi125.stdout:252: finishing copy_from to smithi12556470-12 2021-03-24T21:55:20.582 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 12 v 15 (ObjNum 39 snap 0 seq_num 39) dirty exists 2021-03-24T21:55:20.582 INFO:tasks.rados.rados.0.smithi125.stdout:252: finishing copy_from racing read to smithi12556470-12 2021-03-24T21:55:20.582 INFO:tasks.rados.rados.0.smithi125.stdout:254: finishing write tid 1 to smithi12556470-36 2021-03-24T21:55:20.582 INFO:tasks.rados.rados.0.smithi125.stdout:254: finishing write tid 2 to smithi12556470-36 2021-03-24T21:55:20.582 INFO:tasks.rados.rados.0.smithi125.stdout:254: finishing write tid 3 to smithi12556470-36 2021-03-24T21:55:20.583 INFO:tasks.rados.rados.0.smithi125.stdout:254: finishing write tid 4 to smithi12556470-36 2021-03-24T21:55:20.583 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 36 v 77 (ObjNum 120 snap 23 seq_num 120) dirty exists 2021-03-24T21:55:20.583 INFO:tasks.rados.rados.0.smithi125.stdout:254: left oid 36 (ObjNum 120 snap 23 seq_num 120) 2021-03-24T21:55:20.584 INFO:tasks.rados.rados.0.smithi125.stdout:255: finishing write tid 1 to smithi12556470-38 2021-03-24T21:55:20.584 INFO:tasks.rados.rados.0.smithi125.stdout:252: done (4 left) 2021-03-24T21:55:20.584 INFO:tasks.rados.rados.0.smithi125.stdout:254: done (3 left) 2021-03-24T21:55:20.585 INFO:tasks.rados.rados.0.smithi125.stdout:257: done (2 left) 2021-03-24T21:55:20.585 INFO:tasks.rados.rados.0.smithi125.stdout:258: write (excl) oid 13 current snap is 23 2021-03-24T21:55:20.585 INFO:tasks.rados.rados.0.smithi125.stdout:258: seq_num 123 ranges {467261=717275,1847991=219270} 2021-03-24T21:55:20.590 INFO:tasks.rados.rados.0.smithi125.stdout:258: writing smithi12556470-13 from 467261 to 1184536 tid 1 2021-03-24T21:55:20.596 INFO:tasks.rados.rados.0.smithi125.stdout:258: writing smithi12556470-13 from 1847991 to 2067261 tid 2 2021-03-24T21:55:20.596 INFO:tasks.rados.rados.0.smithi125.stdout:259: rollback oid 23 current snap is 23 2021-03-24T21:55:20.596 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 23 to 22 2021-03-24T21:55:20.597 INFO:tasks.rados.rados.0.smithi125.stdout:255: finishing write tid 2 to smithi12556470-38 2021-03-24T21:55:20.597 INFO:tasks.rados.rados.0.smithi125.stdout:255: finishing write tid 3 to smithi12556470-38 2021-03-24T21:55:20.597 INFO:tasks.rados.rados.0.smithi125.stdout:255: finishing write tid 4 to smithi12556470-38 2021-03-24T21:55:20.597 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 38 v 16 (ObjNum 121 snap 23 seq_num 121) dirty exists 2021-03-24T21:55:20.598 INFO:tasks.rados.rados.0.smithi125.stdout:255: left oid 38 (ObjNum 121 snap 23 seq_num 121) 2021-03-24T21:55:20.598 INFO:tasks.rados.rados.0.smithi125.stdout:258: finishing write tid 1 to smithi12556470-13 2021-03-24T21:55:20.598 INFO:tasks.rados.rados.0.smithi125.stdout:255: done (3 left) 2021-03-24T21:55:20.598 INFO:tasks.rados.rados.0.smithi125.stdout:260: read oid 29 snap -1 2021-03-24T21:55:20.598 INFO:tasks.rados.rados.0.smithi125.stdout:260: expect (ObjNum 113 snap 23 seq_num 113) 2021-03-24T21:55:20.599 INFO:tasks.rados.rados.0.smithi125.stdout:261: write oid 44 current snap is 23 2021-03-24T21:55:20.599 INFO:tasks.rados.rados.0.smithi125.stdout:261: seq_num 124 ranges {484068=792524,1906473=177595} 2021-03-24T21:55:20.604 INFO:tasks.rados.rados.0.smithi125.stdout:261: writing smithi12556470-44 from 484068 to 1276592 tid 1 2021-03-24T21:55:20.609 INFO:tasks.rados.rados.0.smithi125.stdout:261: writing smithi12556470-44 from 1906473 to 2084068 tid 2 2021-03-24T21:55:20.610 INFO:tasks.rados.rados.0.smithi125.stdout:262: read oid 19 snap -1 2021-03-24T21:55:20.610 INFO:tasks.rados.rados.0.smithi125.stdout:262: expect (ObjNum 9 snap 0 seq_num 9) 2021-03-24T21:55:20.610 INFO:tasks.rados.rados.0.smithi125.stdout:263: read oid 26 snap -1 2021-03-24T21:55:20.610 INFO:tasks.rados.rados.0.smithi125.stdout:263: expect (ObjNum 25 snap 0 seq_num 25) 2021-03-24T21:55:20.611 INFO:tasks.rados.rados.0.smithi125.stdout:264: write (excl) oid 47 current snap is 23 2021-03-24T21:55:20.611 INFO:tasks.rados.rados.0.smithi125.stdout:259: finishing rollback tid 1 to smithi12556470-23 2021-03-24T21:55:20.611 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 23 v 0 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-03-24T21:55:20.611 INFO:tasks.rados.rados.0.smithi125.stdout:264: seq_num 125 ranges {500875=467773,1564955=535920} 2021-03-24T21:55:20.615 INFO:tasks.rados.rados.0.smithi125.stdout:264: writing smithi12556470-47 from 500875 to 968648 tid 1 2021-03-24T21:55:20.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:20 smithi142 conmon[20125]: cluster 2021-03-24T21:55:18 2021-03-24T21:55:20.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:20 smithi142 conmon[20125]: .724609+0000 mgr.y (mgr.14144) 332 : cluster [DBG] pgmap v386: 148 pgs: 148 active+clean; 176 MiB data, 480 MiB used, 536 GiB / 536 GiB avail; 3.0 MiB/s rd, 1 op/s 2021-03-24T21:55:20.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:20 smithi142 conmon[20125]: cluster 2021-03-24T21:55:19.194379 2021-03-24T21:55:20.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:20 smithi142 conmon[20125]: +0000 mon.a (mon.0) 716 : cluster [DBG] osdmap e150: 8 total, 8 up, 6 in 2021-03-24T21:55:20.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:20 smithi142 conmon[20125]: cluster 2021-03-24T21:55:20.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:20 smithi142 conmon[20125]: 2021-03-24T21:55:20.098162+0000 mon.a (mon.0) 717 : 2021-03-24T21:55:20.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:20 smithi142 conmon[20125]: cluster [DBG] osdmap e151: 8 total, 8 up, 6 in 2021-03-24T21:55:20.621 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:20.623 INFO:tasks.rados.rados.0.smithi125.stdout:264: writing smithi12556470-47 from 1564955 to 2100875 tid 2 2021-03-24T21:55:20.623 INFO:tasks.rados.rados.0.smithi125.stdout:259: done (7 left) 2021-03-24T21:55:20.623 INFO:tasks.rados.rados.0.smithi125.stdout:265: write oid 6 current snap is 23 2021-03-24T21:55:20.623 INFO:tasks.rados.rados.0.smithi125.stdout:258: finishing write tid 2 to smithi12556470-13 2021-03-24T21:55:20.624 INFO:tasks.rados.rados.0.smithi125.stdout:265: seq_num 126 ranges {517682=543022,1623437=494245} 2021-03-24T21:55:20.628 INFO:tasks.rados.rados.0.smithi125.stdout:265: writing smithi12556470-6 from 517682 to 1060704 tid 1 2021-03-24T21:55:20.634 INFO:tasks.rados.rados.0.smithi125.stdout:265: writing smithi12556470-6 from 1623437 to 2117682 tid 2 2021-03-24T21:55:20.634 INFO:tasks.rados.rados.0.smithi125.stdout:266: write (excl) oid 20 current snap is 23 2021-03-24T21:55:20.635 INFO:tasks.rados.rados.0.smithi125.stdout:258: finishing write tid 3 to smithi12556470-13 2021-03-24T21:55:20.635 INFO:tasks.rados.rados.0.smithi125.stdout:266: seq_num 127 ranges {534489=618271,1565566=568923} 2021-03-24T21:55:20.641 INFO:tasks.rados.rados.0.smithi125.stdout:266: writing smithi12556470-20 from 534489 to 1152760 tid 1 2021-03-24T21:55:20.647 INFO:tasks.rados.rados.0.smithi125.stdout:266: writing smithi12556470-20 from 1565566 to 2134489 tid 2 2021-03-24T21:55:20.647 INFO:tasks.rados.rados.0.smithi125.stdout:267: write (excl) oid 45 current snap is 23 2021-03-24T21:55:20.647 INFO:tasks.rados.rados.0.smithi125.stdout:258: finishing write tid 4 to smithi12556470-13 2021-03-24T21:55:20.647 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 13 v 98 (ObjNum 123 snap 23 seq_num 123) dirty exists 2021-03-24T21:55:20.648 INFO:tasks.rados.rados.0.smithi125.stdout:258: left oid 13 (ObjNum 123 snap 23 seq_num 123) 2021-03-24T21:55:20.648 INFO:tasks.rados.rados.0.smithi125.stdout:267: seq_num 128 ranges {551296=693520,2024048=127248} 2021-03-24T21:55:20.654 INFO:tasks.rados.rados.0.smithi125.stdout:267: writing smithi12556470-45 from 551296 to 1244816 tid 1 2021-03-24T21:55:20.659 INFO:tasks.rados.rados.0.smithi125.stdout:267: writing smithi12556470-45 from 2024048 to 2151296 tid 2 2021-03-24T21:55:20.659 INFO:tasks.rados.rados.0.smithi125.stdout:258: done (9 left) 2021-03-24T21:55:20.660 INFO:tasks.rados.rados.0.smithi125.stdout:268: write (excl) oid 7 current snap is 23 2021-03-24T21:55:20.660 INFO:tasks.rados.rados.0.smithi125.stdout:268: seq_num 129 ranges {568103=768769,2082530=85573} 2021-03-24T21:55:20.667 INFO:tasks.rados.rados.0.smithi125.stdout:268: writing smithi12556470-7 from 568103 to 1336872 tid 1 2021-03-24T21:55:20.672 INFO:tasks.rados.rados.0.smithi125.stdout:268: writing smithi12556470-7 from 2082530 to 2168103 tid 2 2021-03-24T21:55:20.672 INFO:tasks.rados.rados.0.smithi125.stdout:269: read oid 9 snap -1 2021-03-24T21:55:20.672 INFO:tasks.rados.rados.0.smithi125.stdout:269: expect (ObjNum 68 snap 7 seq_num 68) 2021-03-24T21:55:20.673 INFO:tasks.rados.rados.0.smithi125.stdout:270: snap_create 2021-03-24T21:55:20.673 INFO:tasks.rados.rados.0.smithi125.stdout:260: expect (ObjNum 113 snap 23 seq_num 113) 2021-03-24T21:55:20.678 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:20.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:20.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:55:20.785 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:20.844 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:20.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:20.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:55:20.950 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:20.962 INFO:tasks.rados.rados.0.smithi125.stdout:256: expect (ObjNum 79 snap 9 seq_num 79) 2021-03-24T21:55:21.004 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:21.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:21.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:55:21.110 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:21.114 INFO:tasks.rados.rados.0.smithi125.stdout:261: finishing write tid 1 to smithi12556470-44 2021-03-24T21:55:21.115 INFO:tasks.rados.rados.0.smithi125.stdout:261: finishing write tid 2 to smithi12556470-44 2021-03-24T21:55:21.115 INFO:tasks.rados.rados.0.smithi125.stdout:261: finishing write tid 3 to smithi12556470-44 2021-03-24T21:55:21.116 INFO:tasks.rados.rados.0.smithi125.stdout:261: finishing write tid 4 to smithi12556470-44 2021-03-24T21:55:21.116 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 44 v 104 (ObjNum 124 snap 23 seq_num 124) dirty exists 2021-03-24T21:55:21.117 INFO:tasks.rados.rados.0.smithi125.stdout:261: left oid 44 (ObjNum 124 snap 23 seq_num 124) 2021-03-24T21:55:21.117 INFO:tasks.rados.rados.0.smithi125.stdout:263: expect (ObjNum 25 snap 0 seq_num 25) 2021-03-24T21:55:21.167 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:21.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:21.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:55:21.195 INFO:tasks.rados.rados.0.smithi125.stdout:264: finishing write tid 1 to smithi12556470-47 2021-03-24T21:55:21.195 INFO:tasks.rados.rados.0.smithi125.stdout:264: finishing write tid 2 to smithi12556470-47 2021-03-24T21:55:21.195 INFO:tasks.rados.rados.0.smithi125.stdout:264: finishing write tid 3 to smithi12556470-47 2021-03-24T21:55:21.196 INFO:tasks.rados.rados.0.smithi125.stdout:262: expect (ObjNum 9 snap 0 seq_num 9) 2021-03-24T21:55:21.267 INFO:tasks.rados.rados.0.smithi125.stdout:264: finishing write tid 4 to smithi12556470-47 2021-03-24T21:55:21.267 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 47 v 31 (ObjNum 125 snap 23 seq_num 125) dirty exists 2021-03-24T21:55:21.268 INFO:tasks.rados.rados.0.smithi125.stdout:264: left oid 47 (ObjNum 125 snap 23 seq_num 125) 2021-03-24T21:55:21.268 INFO:tasks.rados.rados.0.smithi125.stdout:265: finishing write tid 1 to smithi12556470-6 2021-03-24T21:55:21.269 INFO:tasks.rados.rados.0.smithi125.stdout:265: finishing write tid 2 to smithi12556470-6 2021-03-24T21:55:21.269 INFO:tasks.rados.rados.0.smithi125.stdout:265: finishing write tid 3 to smithi12556470-6 2021-03-24T21:55:21.270 INFO:tasks.rados.rados.0.smithi125.stdout:265: finishing write tid 4 to smithi12556470-6 2021-03-24T21:55:21.270 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 6 v 102 (ObjNum 126 snap 23 seq_num 126) dirty exists 2021-03-24T21:55:21.270 INFO:tasks.rados.rados.0.smithi125.stdout:265: left oid 6 (ObjNum 126 snap 23 seq_num 126) 2021-03-24T21:55:21.271 INFO:tasks.rados.rados.0.smithi125.stdout:266: finishing write tid 1 to smithi12556470-20 2021-03-24T21:55:21.271 INFO:tasks.rados.rados.0.smithi125.stdout:266: finishing write tid 2 to smithi12556470-20 2021-03-24T21:55:21.272 INFO:tasks.rados.rados.0.smithi125.stdout:266: finishing write tid 3 to smithi12556470-20 2021-03-24T21:55:21.272 INFO:tasks.rados.rados.0.smithi125.stdout:256: done (11 left) 2021-03-24T21:55:21.273 INFO:tasks.rados.rados.0.smithi125.stdout:260: done (10 left) 2021-03-24T21:55:21.273 INFO:tasks.rados.rados.0.smithi125.stdout:261: done (9 left) 2021-03-24T21:55:21.273 INFO:tasks.rados.rados.0.smithi125.stdout:262: done (8 left) 2021-03-24T21:55:21.274 INFO:tasks.rados.rados.0.smithi125.stdout:263: done (7 left) 2021-03-24T21:55:21.274 INFO:tasks.rados.rados.0.smithi125.stdout:264: done (6 left) 2021-03-24T21:55:21.274 INFO:tasks.rados.rados.0.smithi125.stdout:265: done (5 left) 2021-03-24T21:55:21.275 INFO:tasks.rados.rados.0.smithi125.stdout:270: done (4 left) 2021-03-24T21:55:21.275 INFO:tasks.rados.rados.0.smithi125.stdout:271: snap_remove snap 22 2021-03-24T21:55:21.276 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:21.315 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:55:21.335 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:21.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:21.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:55:21.440 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:21.485 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:21.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:21.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:55:21.589 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:21.645 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:21.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:21.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:55:21.750 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:21.809 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:21.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:21.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:55:21.914 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:21.975 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:21.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:21.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:22.080 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:22.128 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:22.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:22.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:55:22.221 INFO:tasks.rados.rados.0.smithi125.stdout:271: done (4 left) 2021-03-24T21:55:22.222 INFO:tasks.rados.rados.0.smithi125.stdout:272: copy_from oid 14 from oid 2 current snap is 24 2021-03-24T21:55:22.222 INFO:tasks.rados.rados.0.smithi125.stdout:266: finishing write tid 4 to smithi12556470-20 2021-03-24T21:55:22.222 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 20 v 72 (ObjNum 127 snap 23 seq_num 127) dirty exists 2021-03-24T21:55:22.222 INFO:tasks.rados.rados.0.smithi125.stdout:266: left oid 20 (ObjNum 127 snap 23 seq_num 127) 2021-03-24T21:55:22.222 INFO:tasks.rados.rados.0.smithi125.stdout:267: finishing write tid 1 to smithi12556470-45 2021-03-24T21:55:22.223 INFO:tasks.rados.rados.0.smithi125.stdout:267: finishing write tid 2 to smithi12556470-45 2021-03-24T21:55:22.223 INFO:tasks.rados.rados.0.smithi125.stdout:267: finishing write tid 3 to smithi12556470-45 2021-03-24T21:55:22.223 INFO:tasks.rados.rados.0.smithi125.stdout:267: finishing write tid 4 to smithi12556470-45 2021-03-24T21:55:22.223 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 45 v 108 (ObjNum 128 snap 23 seq_num 128) dirty exists 2021-03-24T21:55:22.223 INFO:tasks.rados.rados.0.smithi125.stdout:267: left oid 45 (ObjNum 128 snap 23 seq_num 128) 2021-03-24T21:55:22.224 INFO:tasks.rados.rados.0.smithi125.stdout:268: finishing write tid 1 to smithi12556470-7 2021-03-24T21:55:22.224 INFO:tasks.rados.rados.0.smithi125.stdout:268: finishing write tid 2 to smithi12556470-7 2021-03-24T21:55:22.224 INFO:tasks.rados.rados.0.smithi125.stdout:268: finishing write tid 3 to smithi12556470-7 2021-03-24T21:55:22.224 INFO:tasks.rados.rados.0.smithi125.stdout:268: finishing write tid 4 to smithi12556470-7 2021-03-24T21:55:22.224 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 7 v 80 (ObjNum 129 snap 23 seq_num 129) dirty exists 2021-03-24T21:55:22.225 INFO:tasks.rados.rados.0.smithi125.stdout:268: left oid 7 (ObjNum 129 snap 23 seq_num 129) 2021-03-24T21:55:22.225 INFO:tasks.rados.rados.0.smithi125.stdout:269: expect (ObjNum 68 snap 7 seq_num 68) 2021-03-24T21:55:22.233 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:22.234 INFO:teuthology.orchestra.run.smithi125.stderr:noscrub is unset 2021-03-24T21:55:22.283 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:22.288 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-24T21:55:22.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:22.291 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2021-03-24T21:55:22.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:55:22.393 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:22.417 INFO:tasks.rados.rados.0.smithi125.stdout:266: done (4 left) 2021-03-24T21:55:22.418 INFO:tasks.rados.rados.0.smithi125.stdout:267: done (3 left) 2021-03-24T21:55:22.418 INFO:tasks.rados.rados.0.smithi125.stdout:268: done (2 left) 2021-03-24T21:55:22.418 INFO:tasks.rados.rados.0.smithi125.stdout:269: done (1 left) 2021-03-24T21:55:22.418 INFO:tasks.rados.rados.0.smithi125.stdout:273: delete oid 18 current snap is 24 2021-03-24T21:55:22.419 INFO:tasks.rados.rados.0.smithi125.stdout:272: finishing copy_from to smithi12556470-14 2021-03-24T21:55:22.419 INFO:tasks.rados.rados.0.smithi125.stdout:272: got expected ENOENT (src dne) 2021-03-24T21:55:22.419 INFO:tasks.rados.rados.0.smithi125.stdout:272: finishing copy_from racing read to smithi12556470-14 2021-03-24T21:55:22.420 INFO:tasks.rados.rados.0.smithi125.stdout:272: done (1 left) 2021-03-24T21:55:22.421 INFO:tasks.rados.rados.0.smithi125.stdout:273: done (0 left) 2021-03-24T21:55:22.421 INFO:tasks.rados.rados.0.smithi125.stdout:274: delete oid 19 current snap is 24 2021-03-24T21:55:22.424 INFO:tasks.rados.rados.0.smithi125.stdout:274: done (0 left) 2021-03-24T21:55:22.425 INFO:tasks.rados.rados.0.smithi125.stdout:275: rollback oid 42 current snap is 24 2021-03-24T21:55:22.425 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 42 to 23 2021-03-24T21:55:22.425 INFO:tasks.rados.rados.0.smithi125.stdout:276: write (excl) oid 40 current snap is 24 2021-03-24T21:55:22.425 INFO:tasks.rados.rados.0.smithi125.stdout:276: seq_num 132 ranges {618524=710869,1857976=360548} 2021-03-24T21:55:22.434 INFO:tasks.rados.rados.0.smithi125.stdout:276: writing smithi12556470-40 from 618524 to 1329393 tid 1 2021-03-24T21:55:22.436 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-24T21:55:22.440 INFO:tasks.rados.rados.0.smithi125.stdout:276: writing smithi12556470-40 from 1857976 to 2218524 tid 2 2021-03-24T21:55:22.441 INFO:tasks.rados.rados.0.smithi125.stdout:277: read oid 26 snap 23 2021-03-24T21:55:22.441 INFO:tasks.rados.rados.0.smithi125.stdout:277: expect (ObjNum 25 snap 0 seq_num 25) 2021-03-24T21:55:22.441 INFO:tasks.rados.rados.0.smithi125.stdout:278: write oid 23 current snap is 24 2021-03-24T21:55:22.441 INFO:tasks.rados.rados.0.smithi125.stdout:275: finishing rollback tid 0 to smithi12556470-42 2021-03-24T21:55:22.442 INFO:tasks.rados.rados.0.smithi125.stdout:275: finishing rollback tid 1 to smithi12556470-42 2021-03-24T21:55:22.442 INFO:tasks.rados.rados.0.smithi125.stdout:275: finishing rollback tid 2 to smithi12556470-42 2021-03-24T21:55:22.442 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 42 v 40 (ObjNum 91 snap 14 seq_num 91) dirty exists 2021-03-24T21:55:22.442 INFO:tasks.rados.rados.0.smithi125.stdout:278: seq_num 133 ranges {635331=786118,1916458=318873} 2021-03-24T21:55:22.448 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:22.450 INFO:tasks.rados.rados.0.smithi125.stdout:278: writing smithi12556470-23 from 635331 to 1421449 tid 1 2021-03-24T21:55:22.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:22.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:55:22.456 INFO:tasks.rados.rados.0.smithi125.stdout:278: writing smithi12556470-23 from 1916458 to 2235331 tid 2 2021-03-24T21:55:22.456 INFO:tasks.rados.rados.0.smithi125.stdout:275: done (3 left) 2021-03-24T21:55:22.456 INFO:tasks.rados.rados.0.smithi125.stdout:279: delete oid 12 current snap is 24 2021-03-24T21:55:22.457 INFO:tasks.rados.rados.0.smithi125.stdout:276: finishing write tid 1 to smithi12556470-40 2021-03-24T21:55:22.457 INFO:tasks.rados.rados.0.smithi125.stdout:277: expect (ObjNum 25 snap 0 seq_num 25) 2021-03-24T21:55:22.541 INFO:tasks.rados.rados.0.smithi125.stdout:276: finishing write tid 2 to smithi12556470-40 2021-03-24T21:55:22.541 INFO:tasks.rados.rados.0.smithi125.stdout:276: finishing write tid 3 to smithi12556470-40 2021-03-24T21:55:22.542 INFO:tasks.rados.rados.0.smithi125.stdout:276: finishing write tid 4 to smithi12556470-40 2021-03-24T21:55:22.542 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 40 v 105 (ObjNum 132 snap 24 seq_num 132) dirty exists 2021-03-24T21:55:22.542 INFO:tasks.rados.rados.0.smithi125.stdout:276: left oid 40 (ObjNum 132 snap 24 seq_num 132) 2021-03-24T21:55:22.542 INFO:tasks.rados.rados.0.smithi125.stdout:278: finishing write tid 1 to smithi12556470-23 2021-03-24T21:55:22.543 INFO:tasks.rados.rados.0.smithi125.stdout:276: done (3 left) 2021-03-24T21:55:22.543 INFO:tasks.rados.rados.0.smithi125.stdout:277: done (2 left) 2021-03-24T21:55:22.543 INFO:tasks.rados.rados.0.smithi125.stdout:279: done (1 left) 2021-03-24T21:55:22.543 INFO:tasks.rados.rados.0.smithi125.stdout:280: copy_from oid 45 from oid 37 current snap is 24 2021-03-24T21:55:22.543 INFO:tasks.rados.rados.0.smithi125.stdout:278: finishing write tid 2 to smithi12556470-23 2021-03-24T21:55:22.544 INFO:tasks.rados.rados.0.smithi125.stdout:278: finishing write tid 3 to smithi12556470-23 2021-03-24T21:55:22.544 INFO:tasks.rados.rados.0.smithi125.stdout:278: finishing write tid 4 to smithi12556470-23 2021-03-24T21:55:22.544 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 23 v 105 (ObjNum 133 snap 24 seq_num 133) dirty exists 2021-03-24T21:55:22.544 INFO:tasks.rados.rados.0.smithi125.stdout:278: left oid 23 (ObjNum 133 snap 24 seq_num 133) 2021-03-24T21:55:22.544 INFO:tasks.rados.rados.0.smithi125.stdout:278: done (1 left) 2021-03-24T21:55:22.545 INFO:tasks.rados.rados.0.smithi125.stdout:281: snap_create 2021-03-24T21:55:22.554 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:22.561 INFO:tasks.rados.rados.0.smithi125.stdout:280: finishing copy_from to smithi12556470-45 2021-03-24T21:55:22.561 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 45 v 49 (ObjNum 101 snap 20 seq_num 101) dirty exists 2021-03-24T21:55:22.562 INFO:tasks.rados.rados.0.smithi125.stdout:280: finishing copy_from racing read to smithi12556470-45 2021-03-24T21:55:22.567 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:22 smithi125 conmon[24927]: cluster 2021-03-24T21:55: 2021-03-24T21:55:22.567 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:22 smithi125 conmon[24927]: 20.725163+0000 mgr.y (mgr.14144) 333 : cluster [DBG] pgmap v389: 148 pgs: 148 active+clean; 182 MiB data, 476 MiB used, 536 GiB / 536 GiB avail; 13 MiB/s rd, 1.5 MiB/s wr, 16 op/s 2021-03-24T21:55:22.567 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:22 smithi125 conmon[24927]: cluster 2021-03-24T21:55:21.205736+0000 mon.a (mon.0) 718 2021-03-24T21:55:22.567 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:22 smithi125 conmon[24927]: : cluster [DBG] osdmap e152: 8 total, 8 up, 6 in 2021-03-24T21:55:22.568 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:22 smithi125 conmon[24927]: audit 2021-03-24T21:55:21.669649+0000 mon.a (mon. 2021-03-24T21:55:22.568 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:22 smithi125 conmon[24927]: 0) 719 : audit [INF] from='client.? v1:172.21.15.125:0/3547647229' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T21:55:22.568 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:22 smithi125 conmon[19936]: cluster 2021-03-24T21:55:20.725163+0000 mgr.y (mgr. 2021-03-24T21:55:22.568 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:22 smithi125 conmon[19936]: 14144) 333 : cluster [DBG] pgmap v389: 148 pgs: 148 active+clean; 182 MiB data, 476 MiB used, 536 GiB / 536 GiB avail; 13 MiB/s rd, 1.5 MiB/s wr, 16 op/s 2021-03-24T21:55:22.569 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:22 smithi125 conmon[19936]: cluster 2021-03-24T21:55:21.205736+0000 mon.a (mon. 2021-03-24T21:55:22.569 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:22 smithi125 conmon[19936]: 0) 718 : cluster [DBG] osdmap e152: 8 total, 8 up, 6 in 2021-03-24T21:55:22.569 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:22 smithi125 conmon[19936]: audit 2021-03- 2021-03-24T21:55:22.569 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:22 smithi125 conmon[19936]: 24T21:55:21.669649+0000 mon.a (mon.0) 719 : audit [INF] from='client.? v1:172.21.15.125:0/3547647229' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T21:55:22.601 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:22.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:22.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:55:22.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:22 smithi142 conmon[20125]: cluster 2021-03-24T21:55:20.725163+0000 2021-03-24T21:55:22.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:22 smithi142 conmon[20125]: mgr.y (mgr.14144) 333 : cluster [DBG] pgmap v389: 148 pgs: 148 active+clean; 182 MiB data, 476 MiB used, 536 GiB / 536 GiB avail; 13 MiB/s rd, 1.5 MiB/s wr, 16 op/s 2021-03-24T21:55:22.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:22 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:55:22.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:22 smithi142 conmon[20125]: T21:55:21.205736+0000 mon.a (mon.0) 718 : 2021-03-24T21:55:22.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:22 smithi142 conmon[20125]: cluster [DBG] osdmap e152: 8 total, 8 up, 6 in 2021-03-24T21:55:22.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:22 smithi142 conmon[20125]: audit 2021-03- 2021-03-24T21:55:22.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:22 smithi142 conmon[20125]: 24T21:55:21.669649+0000 2021-03-24T21:55:22.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:22 smithi142 conmon[20125]: mon.a (mon.0) 719 : audit [INF] from='client.? v1:172.21.15.125:0/3547647229' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T21:55:22.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:22 smithi142 conmon[20125]: 2021-03-24T21:55:22.706 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:22.760 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:22.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:22.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:55:22.866 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:22.921 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:22.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:22.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:55:23.027 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:23.079 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:23.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:23.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:55:23.185 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:23.224 INFO:tasks.rados.rados.0.smithi125.stdout:280: done (1 left) 2021-03-24T21:55:23.224 INFO:tasks.rados.rados.0.smithi125.stdout:281: done (0 left) 2021-03-24T21:55:23.225 INFO:tasks.rados.rados.0.smithi125.stdout:282: delete oid 50 current snap is 25 2021-03-24T21:55:23.229 INFO:tasks.rados.rados.0.smithi125.stdout:282: done (0 left) 2021-03-24T21:55:23.230 INFO:tasks.rados.rados.0.smithi125.stdout:283: write (excl) oid 28 current snap is 25 2021-03-24T21:55:23.230 INFO:tasks.rados.rados.0.smithi125.stdout:283: seq_num 136 ranges {685752=611865,1975551=310201} 2021-03-24T21:55:23.238 INFO:tasks.rados.rados.0.smithi125.stdout:283: writing smithi12556470-28 from 685752 to 1297617 tid 1 2021-03-24T21:55:23.243 INFO:tasks.rados.rados.0.smithi125.stdout:283: writing smithi12556470-28 from 1975551 to 2285752 tid 2 2021-03-24T21:55:23.244 INFO:tasks.rados.rados.0.smithi125.stdout:284: delete oid 26 current snap is 25 2021-03-24T21:55:23.244 INFO:tasks.rados.rados.0.smithi125.stdout:283: finishing write tid 1 to smithi12556470-28 2021-03-24T21:55:23.244 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:23.248 INFO:tasks.rados.rados.0.smithi125.stdout:284: done (1 left) 2021-03-24T21:55:23.248 INFO:tasks.rados.rados.0.smithi125.stdout:285: snap_remove snap 10 2021-03-24T21:55:23.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:23.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:55:23.333 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:23 smithi125 conmon[19936]: cluster 2021-03-24T21:55:22. 2021-03-24T21:55:23.333 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:23 smithi125 conmon[19936]: 201880+0000 mon.a (mon.0) 720 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:55:23.334 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:23 smithi125 conmon[19936]: audit 2021-03-24T21: 2021-03-24T21:55:23.334 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:23 smithi125 conmon[19936]: 55:22.218058+0000 mon.a (mon.0) 721 : audit [INF] from='client.? v1:172.21.15.125:0/3547647229' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T21:55:23.334 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:23 smithi125 conmon[19936]: cluster 2021-03-24T21:55 2021-03-24T21:55:23.334 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:23 smithi125 conmon[19936]: :22.218144+0000 mon.a (mon.0) 2021-03-24T21:55:23.335 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:23 smithi125 conmon[19936]: 722 : cluster [DBG] osdmap e153: 8 total, 8 up, 6 in 2021-03-24T21:55:23.335 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:23 smithi125 conmon[19936]: audit 2021-03 2021-03-24T21:55:23.335 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:23 smithi125 conmon[19936]: -24T21:55:22.264088+0000 2021-03-24T21:55:23.335 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:23 smithi125 conmon[19936]: mgr.y (mgr.14144) 334 : audit [DBG] from='client.34754 v1:172.21.15.125:0/3212216769' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T21:55:23.336 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:23 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T21:55:23.336 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:23 smithi125 conmon[19936]: T21:55:22.555620+0000 osd.3 ( 2021-03-24T21:55:23.336 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:23 smithi125 conmon[19936]: osd.3) 48 : cluster [DBG] 6.11 scrub ok 2021-03-24T21:55:23.336 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:23 smithi125 conmon[24927]: cluster 2021-03-24T21:55:22.201880+0000 2021-03-24T21:55:23.337 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:23 smithi125 conmon[24927]: mon.a (mon.0) 720 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:55:23.337 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:23 smithi125 conmon[24927]: audit 2021-03-24T21:55:22.218058+0000 mon.a (mon.0) 721 : audit [INF] 2021-03-24T21:55:23.337 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:23 smithi125 conmon[24927]: from='client.? v1:172.21.15.125:0/3547647229' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T21:55:23.337 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:23 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:55:23.338 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:23 smithi125 conmon[24927]: :55:22.218144+0000 mon.a (mon.0) 722 : cluster [DBG] osdmap e153: 8 total, 8 up, 6 in 2021-03-24T21:55:23.338 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:23 smithi125 conmon[24927]: audit 2021-03-24T21:55: 2021-03-24T21:55:23.338 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:23 smithi125 conmon[24927]: 22.264088+0000 mgr.y (mgr.14144) 334 : audit [DBG] from='client.34754 v1:172.21.15.125:0/3212216769' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T21:55:23.338 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:23 smithi125 conmon[24927]: 2021-03-24T21:55:23.338 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:23 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:55:23.339 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:23 smithi125 conmon[24927]: 03-24T21:55:22.555620+0000 osd.3 (osd.3) 48 : cluster [DBG] 6.11 scrub ok 2021-03-24T21:55:23.339 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:23 smithi125 conmon[24927]: 2021-03-24T21:55:23.351 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:23.405 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:23.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:23.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:55:23.511 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:23.525 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:23 smithi142 conmon[20125]: cluster 2021-03-24T21:55 2021-03-24T21:55:23.525 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:23 smithi142 conmon[20125]: :22.201880+0000 mon.a (mon.0) 720 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:55:23.525 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:23 smithi142 conmon[20125]: audit 2021-03- 2021-03-24T21:55:23.525 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:23 smithi142 conmon[20125]: 24T21:55:22.218058+0000 mon.a (mon.0) 721 : audit 2021-03-24T21:55:23.526 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:23 smithi142 conmon[20125]: [INF] from='client.? v1:172.21.15.125:0/3547647229' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T21:55:23.526 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:23 smithi142 conmon[20125]: cluster 2021-03-24T21:55:23.526 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:23 smithi142 conmon[20125]: 2021-03-24T21:55:22.218144+0000 mon.a 2021-03-24T21:55:23.526 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:23 smithi142 conmon[20125]: (mon.0) 722 : cluster [DBG] osdmap e153: 8 total, 8 up, 6 in 2021-03-24T21:55:23.527 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:23 smithi142 conmon[20125]: 2021-03-24T21:55:23.527 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:23 smithi142 conmon[20125]: audit 2021 2021-03-24T21:55:23.527 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:23 smithi142 conmon[20125]: -03-24T21:55:22.264088+0000 mgr.y (mgr 2021-03-24T21:55:23.527 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:23 smithi142 conmon[20125]: .14144) 334 : audit [DBG] from='client.34754 v1:172.21.15.125:0/3212216769' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T21:55:23.528 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:23 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:55:23.528 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:23 smithi142 conmon[20125]: -24T21:55:22.555620+0000 2021-03-24T21:55:23.528 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:23 smithi142 conmon[20125]: osd.3 (osd.3) 48 : cluster [DBG] 2021-03-24T21:55:23.528 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:23 smithi142 conmon[20125]: 6.11 scrub ok 2021-03-24T21:55:23.566 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:23.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:23.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:55:23.673 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:23.729 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:23.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:23.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:55:23.836 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:23.887 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:23.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:23.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:23.995 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:24.062 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:24.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:24.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:55:24.167 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:24.222 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:24.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:24.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:55:24.236 INFO:tasks.rados.rados.0.smithi125.stdout:285: done (1 left) 2021-03-24T21:55:24.236 INFO:tasks.rados.rados.0.smithi125.stdout:286: read oid 20 snap 23 2021-03-24T21:55:24.236 INFO:tasks.rados.rados.0.smithi125.stdout:286: expect (ObjNum 127 snap 23 seq_num 127) 2021-03-24T21:55:24.237 INFO:tasks.rados.rados.0.smithi125.stdout:287: snap_create 2021-03-24T21:55:24.237 INFO:tasks.rados.rados.0.smithi125.stdout:283: finishing write tid 2 to smithi12556470-28 2021-03-24T21:55:24.237 INFO:tasks.rados.rados.0.smithi125.stdout:283: finishing write tid 3 to smithi12556470-28 2021-03-24T21:55:24.237 INFO:tasks.rados.rados.0.smithi125.stdout:283: finishing write tid 4 to smithi12556470-28 2021-03-24T21:55:24.238 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 28 v 79 (ObjNum 136 snap 25 seq_num 136) dirty exists 2021-03-24T21:55:24.238 INFO:tasks.rados.rados.0.smithi125.stdout:283: left oid 28 (ObjNum 136 snap 25 seq_num 136) 2021-03-24T21:55:24.245 INFO:tasks.rados.rados.0.smithi125.stdout:286: expect (ObjNum 127 snap 23 seq_num 127) 2021-03-24T21:55:24.327 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":391,"stamp":"2021-03-24T21:55:22.214439+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":193365213,"num_objects":398,"num_object_clones":38,"num_object_copies":1194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":398,"num_whiteouts":6,"num_read":1773,"num_read_kb":384644,"num_write":1059,"num_write_kb":193523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":324,"num_bytes_recovered":35484837,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1442,"ondisk_log_size":1442,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":494460,"kb_used_data":419808,"kb_used_omap":0,"kb_used_meta":73280,"kb_avail":561976452,"statfs":{"total":575970213888,"available":575463886848,"internally_reserved":0,"allocated":429883392,"data_stored":486200639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":75038720},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8503654,"num_objects":4,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":47,"num_read_kb":54483,"num_write":27,"num_write_kb":7206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.026143"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":"107","reported_epoch":"147","state":"active+clean","last_fresh":"2021-03-24T21:55:16.165292+0000","last_change":"2021-03-24T21:54:50.292202+0000","last_active":"2021-03-24T21:55:16.165292+0000","last_peered":"2021-03-24T21:55:16.165292+0000","last_clean":"2021-03-24T21:55:16.165292+0000","last_became_active":"2021-03-24T21:54:38.111248+0000","last_became_peered":"2021-03-24T21:54:38.111248+0000","last_unstale":"2021-03-24T21:55:16.165292+0000","last_undegraded":"2021-03-24T21:55:16.165292+0000","last_fullsized":"2021-03-24T21:55:16.165292+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:50.292025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:54:50.292025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"24","reported_epoch":"149","state":"active+clean","last_fresh":"2021-03-24T21:55:18.198295+0000","last_change":"2021-03-24T21:55:10.100067+0000","last_active":"2021-03-24T21:55:18.198295+0000","last_peered":"2021-03-24T21:55:18.198295+0000","last_clean":"2021-03-24T21:55:18.198295+0000","last_became_active":"2021-03-24T21:55:10.099666+0000","last_became_peered":"2021-03-24T21:55:10.099666+0000","last_unstale":"2021-03-24T21:55:18.198295+0000","last_undegraded":"2021-03-24T21:55:18.198295+0000","last_fullsized":"2021-03-24T21:55:18.198295+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:14.710709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:14.710709+0000","last_clean_scrub_stamp":"2021-03-24T21:54:14.710709+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":"100","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.178151+0000","last_change":"2021-03-24T21:53:12.428679+0000","last_active":"2021-03-24T21:55:17.178151+0000","last_peered":"2021-03-24T21:55:17.178151+0000","last_clean":"2021-03-24T21:55:17.178151+0000","last_became_active":"2021-03-24T21:51:45.581000+0000","last_became_peered":"2021-03-24T21:51:45.581000+0000","last_unstale":"2021-03-24T21:55:17.178151+0000","last_undegraded":"2021-03-24T21:55:17.178151+0000","last_fullsized":"2021-03-24T21:55:17.178151+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:12.428497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:12.428497+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"69'15","reported_seq":"187","reported_epoch":"147","state":"active+clean","last_fresh":"2021-03-24T21:55:16.164792+0000","last_change":"2021-03-24T21:54:39.222600+0000","last_active":"2021-03-24T21:55:16.164792+0000","last_peered":"2021-03-24T21:55:16.164792+0000","last_clean":"2021-03-24T21:55:16.164792+0000","last_became_active":"2021-03-24T21:54:38.127426+0000","last_became_peered":"2021-03-24T21:54:38.127426+0000","last_unstale":"2021-03-24T21:55:16.164792+0000","last_undegraded":"2021-03-24T21:55:16.164792+0000","last_fullsized":"2021-03-24T21:55:16.164792+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:53:41.295875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:41.295875+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2076,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"117","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.181589+0000","last_change":"2021-03-24T21:54:43.482408+0000","last_active":"2021-03-24T21:55:17.181589+0000","last_peered":"2021-03-24T21:55:17.181589+0000","last_clean":"2021-03-24T21:55:17.181589+0000","last_became_active":"2021-03-24T21:54:38.130293+0000","last_became_peered":"2021-03-24T21:54:38.130293+0000","last_unstale":"2021-03-24T21:55:17.181589+0000","last_undegraded":"2021-03-24T21:55:17.181589+0000","last_fullsized":"2021-03-24T21:55:17.181589+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:43.482307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:54:43.482307+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"113","reported_epoch":"147","state":"active+clean","last_fresh":"2021-03-24T21:55:16.165150+0000","last_change":"2021-03-24T21:54:38.120961+0000","last_active":"2021-03-24T21:55:16.165150+0000","last_peered":"2021-03-24T21:55:16.165150+0000","last_clean":"2021-03-24T21:55:16.165150+0000","last_became_active":"2021-03-24T21:54:38.120575+0000","last_became_peered":"2021-03-24T21:54:38.120575+0000","last_unstale":"2021-03-24T21:55:16.165150+0000","last_undegraded":"2021-03-24T21:55:16.165150+0000","last_fullsized":"2021-03-24T21:55:16.165150+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:48.329480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:48.329480+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":"118","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.183367+0000","last_change":"2021-03-24T21:54:38.133632+0000","last_active":"2021-03-24T21:55:17.183367+0000","last_peered":"2021-03-24T21:55:17.183367+0000","last_clean":"2021-03-24T21:55:17.183367+0000","last_became_active":"2021-03-24T21:54:38.130163+0000","last_became_peered":"2021-03-24T21:54:38.130163+0000","last_unstale":"2021-03-24T21:55:17.183367+0000","last_undegraded":"2021-03-24T21:55:17.183367+0000","last_fullsized":"2021-03-24T21:55:17.183367+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:18.706745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:18.706745+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"69'20","reported_seq":"138","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.181559+0000","last_change":"2021-03-24T21:54:54.541380+0000","last_active":"2021-03-24T21:55:17.181559+0000","last_peered":"2021-03-24T21:55:17.181559+0000","last_clean":"2021-03-24T21:55:17.181559+0000","last_became_active":"2021-03-24T21:51:47.405776+0000","last_became_peered":"2021-03-24T21:51:47.405776+0000","last_unstale":"2021-03-24T21:55:17.181559+0000","last_undegraded":"2021-03-24T21:55:17.181559+0000","last_fullsized":"2021-03-24T21:55:17.181559+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'20","last_scrub_stamp":"2021-03-24T21:54:54.541183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:54:54.541183+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":22,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"108","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.173778+0000","last_change":"2021-03-24T21:54:49.409465+0000","last_active":"2021-03-24T21:55:17.173778+0000","last_peered":"2021-03-24T21:55:17.173778+0000","last_clean":"2021-03-24T21:55:17.173778+0000","last_became_active":"2021-03-24T21:54:38.127696+0000","last_became_peered":"2021-03-24T21:54:38.127696+0000","last_unstale":"2021-03-24T21:55:17.173778+0000","last_undegraded":"2021-03-24T21:55:17.173778+0000","last_fullsized":"2021-03-24T21:55:17.173778+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:49.409299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:54:49.409299+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"103","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.186178+0000","last_change":"2021-03-24T21:54:44.879188+0000","last_active":"2021-03-24T21:55:17.186178+0000","last_peered":"2021-03-24T21:55:17.186178+0000","last_clean":"2021-03-24T21:55:17.186178+0000","last_became_active":"2021-03-24T21:51:49.413342+0000","last_became_peered":"2021-03-24T21:51:49.413342+0000","last_unstale":"2021-03-24T21:55:17.186178+0000","last_undegraded":"2021-03-24T21:55:17.186178+0000","last_fullsized":"2021-03-24T21:55:17.186178+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:44.878935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:54:44.878935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":"136","reported_epoch":"151","state":"active+clean","last_fresh":"2021-03-24T21:55:20.100878+0000","last_change":"2021-03-24T21:55:10.097750+0000","last_active":"2021-03-24T21:55:20.100878+0000","last_peered":"2021-03-24T21:55:20.100878+0000","last_clean":"2021-03-24T21:55:20.100878+0000","last_became_active":"2021-03-24T21:55:10.097431+0000","last_became_peered":"2021-03-24T21:55:10.097431+0000","last_unstale":"2021-03-24T21:55:20.100878+0000","last_undegraded":"2021-03-24T21:55:20.100878+0000","last_fullsized":"2021-03-24T21:55:20.100878+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:43.877604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:54:43.877604+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"69'13","reported_seq":"121","reported_epoch":"147","state":"active+clean","last_fresh":"2021-03-24T21:55:16.165757+0000","last_change":"2021-03-24T21:53:10.104959+0000","last_active":"2021-03-24T21:55:16.165757+0000","last_peered":"2021-03-24T21:55:16.165757+0000","last_clean":"2021-03-24T21:55:16.165757+0000","last_became_active":"2021-03-24T21:51:47.407488+0000","last_became_peered":"2021-03-24T21:51:47.407488+0000","last_unstale":"2021-03-24T21:55:16.165757+0000","last_undegraded":"2021-03-24T21:55:16.165757+0000","last_fullsized":"2021-03-24T21:55:16.165757+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2021-03-24T21:53:10.104806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:10.104806+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"105","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.183653+0000","last_change":"2021-03-24T21:54:38.122726+0000","last_active":"2021-03-24T21:55:17.183653+0000","last_peered":"2021-03-24T21:55:17.183653+0000","last_clean":"2021-03-24T21:55:17.183653+0000","last_became_active":"2021-03-24T21:54:38.122349+0000","last_became_peered":"2021-03-24T21:54:38.122349+0000","last_unstale":"2021-03-24T21:55:17.183653+0000","last_undegraded":"2021-03-24T21:55:17.183653+0000","last_fullsized":"2021-03-24T21:55:17.183653+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:23.895966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:23.895966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"111","reported_epoch":"149","state":"active+clean","last_fresh":"2021-03-24T21:55:18.197922+0000","last_change":"2021-03-24T21:54:57.741556+0000","last_active":"2021-03-24T21:55:18.197922+0000","last_peered":"2021-03-24T21:55:18.197922+0000","last_clean":"2021-03-24T21:55:18.197922+0000","last_became_active":"2021-03-24T21:54:38.137706+0000","last_became_peered":"2021-03-24T21:54:38.137706+0000","last_unstale":"2021-03-24T21:55:18.197922+0000","last_undegraded":"2021-03-24T21:55:18.197922+0000","last_fullsized":"2021-03-24T21:55:18.197922+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:57.741464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:54:57.741464+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":"86","reported_epoch":"147","state":"active+clean","last_fresh":"2021-03-24T21:55:16.165865+0000","last_change":"2021-03-24T21:55:10.099630+0000","last_active":"2021-03-24T21:55:16.165865+0000","last_peered":"2021-03-24T21:55:16.165865+0000","last_clean":"2021-03-24T21:55:16.165865+0000","last_became_active":"2021-03-24T21:55:10.098790+0000","last_became_peered":"2021-03-24T21:55:10.098790+0000","last_unstale":"2021-03-24T21:55:16.165865+0000","last_undegraded":"2021-03-24T21:55:16.165865+0000","last_fullsized":"2021-03-24T21:55:16.165865+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:09.127715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:54:09.127715+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"91'9","reported_seq":"129","reported_epoch":"147","state":"active+clean","last_fresh":"2021-03-24T21:55:16.167048+0000","last_change":"2021-03-24T21:54:39.157590+0000","last_active":"2021-03-24T21:55:16.167048+0000","last_peered":"2021-03-24T21:55:16.167048+0000","last_clean":"2021-03-24T21:55:16.167048+0000","last_became_active":"2021-03-24T21:54:38.127055+0000","last_became_peered":"2021-03-24T21:54:38.127055+0000","last_unstale":"2021-03-24T21:55:16.167048+0000","last_undegraded":"2021-03-24T21:55:16.167048+0000","last_fullsized":"2021-03-24T21:55:16.167048+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2021-03-24T21:53:33.290000+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2021-03-24T21:53:33.290000+0000","last_clean_scrub_stamp":"2021-03-24T21:53:33.290000+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":8,"num_write":7,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"102","reported_epoch":"147","state":"active+clean","last_fresh":"2021-03-24T21:55:16.166963+0000","last_change":"2021-03-24T21:54:24.199810+0000","last_active":"2021-03-24T21:55:16.166963+0000","last_peered":"2021-03-24T21:55:16.166963+0000","last_clean":"2021-03-24T21:55:16.166963+0000","last_became_active":"2021-03-24T21:54:24.199444+0000","last_became_peered":"2021-03-24T21:54:24.199444+0000","last_unstale":"2021-03-24T21:55:16.166963+0000","last_undegraded":"2021-03-24T21:55:16.166963+0000","last_fullsized":"2021-03-24T21:55:16.166963+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:39.277425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:39.277425+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"56","reported_epoch":"147","state":"active+clean","last_fresh":"2021-03-24T21:55:16.167117+0000","last_change":"2021-03-24T21:54:49.323954+0000","last_active":"2021-03-24T21:55:16.167117+0000","last_peered":"2021-03-24T21:55:16.167117+0000","last_clean":"2021-03-24T21:55:16.167117+0000","last_became_active":"2021-03-24T21:54:38.138620+0000","last_became_peered":"2021-03-24T21:54:38.138620+0000","last_unstale":"2021-03-24T21:55:16.167117+0000","last_undegraded":"2021-03-24T21:55:16.167117+0000","last_fullsized":"2021-03-24T21:55:16.167117+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:49.323789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:54:49.323789+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":"105","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.176687+0000","last_change":"2021-03-24T21:54:43.406949+0000","last_active":"2021-03-24T21:55:17.176687+0000","last_peered":"2021-03-24T21:55:17.176687+0000","last_clean":"2021-03-24T21:55:17.176687+0000","last_became_active":"2021-03-24T21:51:45.381533+0000","last_became_peered":"2021-03-24T21:51:45.381533+0000","last_unstale":"2021-03-24T21:55:17.176687+0000","last_undegraded":"2021-03-24T21:55:17.176687+0000","last_fullsized":"2021-03-24T21:55:17.176687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:43.406875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:54:43.406875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"69'15","reported_seq":"182","reported_epoch":"151","state":"active+clean","last_fresh":"2021-03-24T21:55:20.102090+0000","last_change":"2021-03-24T21:54:39.063715+0000","last_active":"2021-03-24T21:55:20.102090+0000","last_peered":"2021-03-24T21:55:20.102090+0000","last_clean":"2021-03-24T21:55:20.102090+0000","last_became_active":"2021-03-24T21:54:38.140589+0000","last_became_peered":"2021-03-24T21:54:38.140589+0000","last_unstale":"2021-03-24T21:55:20.102090+0000","last_undegraded":"2021-03-24T21:55:20.102090+0000","last_fullsized":"2021-03-24T21:55:20.102090+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:53:58.587152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:58.587152+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"112","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.182602+0000","last_change":"2021-03-24T21:54:38.144481+0000","last_active":"2021-03-24T21:55:17.182602+0000","last_peered":"2021-03-24T21:55:17.182602+0000","last_clean":"2021-03-24T21:55:17.182602+0000","last_became_active":"2021-03-24T21:54:38.143563+0000","last_became_peered":"2021-03-24T21:54:38.143563+0000","last_unstale":"2021-03-24T21:55:17.182602+0000","last_undegraded":"2021-03-24T21:55:17.182602+0000","last_fullsized":"2021-03-24T21:55:17.182602+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:44.657761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:44.657761+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"100","reported_epoch":"147","state":"active+clean","last_fresh":"2021-03-24T21:55:16.166685+0000","last_change":"2021-03-24T21:55:02.129906+0000","last_active":"2021-03-24T21:55:16.166685+0000","last_peered":"2021-03-24T21:55:16.166685+0000","last_clean":"2021-03-24T21:55:16.166685+0000","last_became_active":"2021-03-24T21:51:49.410920+0000","last_became_peered":"2021-03-24T21:51:49.410920+0000","last_unstale":"2021-03-24T21:55:16.166685+0000","last_undegraded":"2021-03-24T21:55:16.166685+0000","last_fullsized":"2021-03-24T21:55:16.166685+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:02.129719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:55:02.129719+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":"113","reported_epoch":"147","state":"active+clean","last_fresh":"2021-03-24T21:55:16.165881+0000","last_change":"2021-03-24T21:54:38.144167+0000","last_active":"2021-03-24T21:55:16.165881+0000","last_peered":"2021-03-24T21:55:16.165881+0000","last_clean":"2021-03-24T21:55:16.165881+0000","last_became_active":"2021-03-24T21:54:38.143876+0000","last_became_peered":"2021-03-24T21:54:38.143876+0000","last_unstale":"2021-03-24T21:55:16.165881+0000","last_undegraded":"2021-03-24T21:55:16.165881+0000","last_fullsized":"2021-03-24T21:55:16.165881+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:11.286003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:52:54.212459+0000","last_clean_scrub_stamp":"2021-03-24T21:54:11.286003+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"69'16","reported_seq":"158","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.182468+0000","last_change":"2021-03-24T21:54:38.191975+0000","last_active":"2021-03-24T21:55:17.182468+0000","last_peered":"2021-03-24T21:55:17.182468+0000","last_clean":"2021-03-24T21:55:17.182468+0000","last_became_active":"2021-03-24T21:54:38.128246+0000","last_became_peered":"2021-03-24T21:54:38.128246+0000","last_unstale":"2021-03-24T21:55:17.182468+0000","last_undegraded":"2021-03-24T21:55:17.182468+0000","last_fullsized":"2021-03-24T21:55:17.182468+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T21:53:42.611540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:42.611540+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":37,"num_read_kb":20,"num_write":18,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":844,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"95","reported_epoch":"149","state":"active+clean","last_fresh":"2021-03-24T21:55:18.199946+0000","last_change":"2021-03-24T21:53:05.847583+0000","last_active":"2021-03-24T21:55:18.199946+0000","last_peered":"2021-03-24T21:55:18.199946+0000","last_clean":"2021-03-24T21:55:18.199946+0000","last_became_active":"2021-03-24T21:51:51.761492+0000","last_became_peered":"2021-03-24T21:51:51.761492+0000","last_unstale":"2021-03-24T21:55:18.199946+0000","last_undegraded":"2021-03-24T21:55:18.199946+0000","last_fullsized":"2021-03-24T21:55:18.199946+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:05.847412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:05.847412+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"102'3","reported_seq":"81","reported_epoch":"147","state":"active+clean","last_fresh":"2021-03-24T21:55:16.167044+0000","last_change":"2021-03-24T21:54:24.283503+0000","last_active":"2021-03-24T21:55:16.167044+0000","last_peered":"2021-03-24T21:55:16.167044+0000","last_clean":"2021-03-24T21:55:16.167044+0000","last_became_active":"2021-03-24T21:54:24.188093+0000","last_became_peered":"2021-03-24T21:54:24.188093+0000","last_unstale":"2021-03-24T21:55:16.167044+0000","last_undegraded":"2021-03-24T21:55:16.167044+0000","last_fullsized":"2021-03-24T21:55:16.167044+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2021-03-24T21:54:15.969746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:54:15.969746+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":"56","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.181078+0000","last_change":"2021-03-24T21:54:48.541576+0000","last_active":"2021-03-24T21:55:17.181078+0000","last_peered":"2021-03-24T21:55:17.181078+0000","last_clean":"2021-03-24T21:55:17.181078+0000","last_became_active":"2021-03-24T21:54:38.130034+0000","last_became_peered":"2021-03-24T21:54:38.130034+0000","last_unstale":"2021-03-24T21:55:17.181078+0000","last_undegraded":"2021-03-24T21:55:17.181078+0000","last_fullsized":"2021-03-24T21:55:17.181078+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:48.541415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:54:48.541415+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"89'27","reported_seq":"136","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.177542+0000","last_change":"2021-03-24T21:53:21.585108+0000","last_active":"2021-03-24T21:55:17.177542+0000","last_peered":"2021-03-24T21:55:17.177542+0000","last_clean":"2021-03-24T21:55:17.177542+0000","last_became_active":"2021-03-24T21:51:47.688995+0000","last_became_peered":"2021-03-24T21:51:47.688995+0000","last_unstale":"2021-03-24T21:55:17.177542+0000","last_undegraded":"2021-03-24T21:55:17.177542+0000","last_fullsized":"2021-03-24T21:55:17.177542+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'27","last_scrub_stamp":"2021-03-24T21:53:21.584999+0000","last_deep_scrub":"89'27","last_deep_scrub_stamp":"2021-03-24T21:53:21.584999+0000","last_clean_scrub_stamp":"2021-03-24T21:53:21.584999+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"57","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.186626+0000","last_change":"2021-03-24T21:54:49.992150+0000","last_active":"2021-03-24T21:55:17.186626+0000","last_peered":"2021-03-24T21:55:17.186626+0000","last_clean":"2021-03-24T21:55:17.186626+0000","last_became_active":"2021-03-24T21:54:38.113849+0000","last_became_peered":"2021-03-24T21:54:38.113849+0000","last_unstale":"2021-03-24T21:55:17.186626+0000","last_undegraded":"2021-03-24T21:55:17.186626+0000","last_fullsized":"2021-03-24T21:55:17.186626+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:49.991966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:54:49.991966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"96","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.178316+0000","last_change":"2021-03-24T21:53:30.660884+0000","last_active":"2021-03-24T21:55:17.178316+0000","last_peered":"2021-03-24T21:55:17.178316+0000","last_clean":"2021-03-24T21:55:17.178316+0000","last_became_active":"2021-03-24T21:51:49.409745+0000","last_became_peered":"2021-03-24T21:51:49.409745+0000","last_unstale":"2021-03-24T21:55:17.178316+0000","last_undegraded":"2021-03-24T21:55:17.178316+0000","last_fullsized":"2021-03-24T21:55:17.178316+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:30.660636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:30.660636+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":"109","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.180961+0000","last_change":"2021-03-24T21:54:38.146256+0000","last_active":"2021-03-24T21:55:17.180961+0000","last_peered":"2021-03-24T21:55:17.180961+0000","last_clean":"2021-03-24T21:55:17.180961+0000","last_became_active":"2021-03-24T21:54:38.145870+0000","last_became_peered":"2021-03-24T21:54:38.145870+0000","last_unstale":"2021-03-24T21:55:17.180961+0000","last_undegraded":"2021-03-24T21:55:17.180961+0000","last_fullsized":"2021-03-24T21:55:17.180961+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:11.648470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:11.648470+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"69'17","reported_seq":"67","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.177837+0000","last_change":"2021-03-24T21:55:10.101681+0000","last_active":"2021-03-24T21:55:17.177837+0000","last_peered":"2021-03-24T21:55:17.177837+0000","last_clean":"2021-03-24T21:55:17.177837+0000","last_became_active":"2021-03-24T21:55:10.101198+0000","last_became_peered":"2021-03-24T21:55:10.101198+0000","last_unstale":"2021-03-24T21:55:17.177837+0000","last_undegraded":"2021-03-24T21:55:17.177837+0000","last_fullsized":"2021-03-24T21:55:17.177837+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'17","last_scrub_stamp":"2021-03-24T21:53:17.246367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:17.246367+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":37,"num_read_kb":18,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"95","reported_epoch":"149","state":"active+clean","last_fresh":"2021-03-24T21:55:18.198277+0000","last_change":"2021-03-24T21:53:27.877846+0000","last_active":"2021-03-24T21:55:18.198277+0000","last_peered":"2021-03-24T21:55:18.198277+0000","last_clean":"2021-03-24T21:55:18.198277+0000","last_became_active":"2021-03-24T21:51:51.762150+0000","last_became_peered":"2021-03-24T21:51:51.762150+0000","last_unstale":"2021-03-24T21:55:18.198277+0000","last_undegraded":"2021-03-24T21:55:18.198277+0000","last_fullsized":"2021-03-24T21:55:18.198277+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:27.877627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:27.877627+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"57","reported_epoch":"147","state":"active+clean","last_fresh":"2021-03-24T21:55:16.165003+0000","last_change":"2021-03-24T21:54:24.188634+0000","last_active":"2021-03-24T21:55:16.165003+0000","last_peered":"2021-03-24T21:55:16.165003+0000","last_clean":"2021-03-24T21:55:16.165003+0000","last_became_active":"2021-03-24T21:54:24.187568+0000","last_became_peered":"2021-03-24T21:54:24.187568+0000","last_unstale":"2021-03-24T21:55:16.165003+0000","last_undegraded":"2021-03-24T21:55:16.165003+0000","last_fullsized":"2021-03-24T21:55:16.165003+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:13.143327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:13.143327+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":"105","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.179332+0000","last_change":"2021-03-24T21:54:48.394336+0000","last_active":"2021-03-24T21:55:17.179332+0000","last_peered":"2021-03-24T21:55:17.179332+0000","last_clean":"2021-03-24T21:55:17.179332+0000","last_became_active":"2021-03-24T21:51:45.379789+0000","last_became_peered":"2021-03-24T21:51:45.379789+0000","last_unstale":"2021-03-24T21:55:17.179332+0000","last_undegraded":"2021-03-24T21:55:17.179332+0000","last_fullsized":"2021-03-24T21:55:17.179332+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:48.394164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:54:48.394164+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"69'12","reported_seq":"218","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.180962+0000","last_change":"2021-03-24T21:54:39.142185+0000","last_active":"2021-03-24T21:55:17.180962+0000","last_peered":"2021-03-24T21:55:17.180962+0000","last_clean":"2021-03-24T21:55:17.180962+0000","last_became_active":"2021-03-24T21:54:38.134132+0000","last_became_peered":"2021-03-24T21:54:38.134132+0000","last_unstale":"2021-03-24T21:55:17.180962+0000","last_undegraded":"2021-03-24T21:55:17.180962+0000","last_fullsized":"2021-03-24T21:55:17.180962+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2021-03-24T21:54:14.574739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:54:14.574739+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":12,"num_write":12,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":3756,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"103","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.181984+0000","last_change":"2021-03-24T21:54:24.204479+0000","last_active":"2021-03-24T21:55:17.181984+0000","last_peered":"2021-03-24T21:55:17.181984+0000","last_clean":"2021-03-24T21:55:17.181984+0000","last_became_active":"2021-03-24T21:54:24.202686+0000","last_became_peered":"2021-03-24T21:54:24.202686+0000","last_unstale":"2021-03-24T21:55:17.181984+0000","last_undegraded":"2021-03-24T21:55:17.181984+0000","last_fullsized":"2021-03-24T21:55:17.181984+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:47.540793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:47.540793+0000","last_clean_scrub_stamp":"2021-03-24T21:53:47.540793+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"106","reported_epoch":"149","state":"active+clean","last_fresh":"2021-03-24T21:55:18.198300+0000","last_change":"2021-03-24T21:54:24.194725+0000","last_active":"2021-03-24T21:55:18.198300+0000","last_peered":"2021-03-24T21:55:18.198300+0000","last_clean":"2021-03-24T21:55:18.198300+0000","last_became_active":"2021-03-24T21:54:24.194423+0000","last_became_peered":"2021-03-24T21:54:24.194423+0000","last_unstale":"2021-03-24T21:55:18.198300+0000","last_undegraded":"2021-03-24T21:55:18.198300+0000","last_fullsized":"2021-03-24T21:55:18.198300+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:52:52.829381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:52:52.829381+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":"111","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.186777+0000","last_change":"2021-03-24T21:54:38.121712+0000","last_active":"2021-03-24T21:55:17.186777+0000","last_peered":"2021-03-24T21:55:17.186777+0000","last_clean":"2021-03-24T21:55:17.186777+0000","last_became_active":"2021-03-24T21:54:38.121323+0000","last_became_peered":"2021-03-24T21:54:38.121323+0000","last_unstale":"2021-03-24T21:55:17.186777+0000","last_undegraded":"2021-03-24T21:55:17.186777+0000","last_fullsized":"2021-03-24T21:55:17.186777+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:21.832921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:21.832921+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"86'22","reported_seq":"163","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.181376+0000","last_change":"2021-03-24T21:54:51.585204+0000","last_active":"2021-03-24T21:55:17.181376+0000","last_peered":"2021-03-24T21:55:17.181376+0000","last_clean":"2021-03-24T21:55:17.181376+0000","last_became_active":"2021-03-24T21:54:24.188790+0000","last_became_peered":"2021-03-24T21:54:24.188790+0000","last_unstale":"2021-03-24T21:55:17.181376+0000","last_undegraded":"2021-03-24T21:55:17.181376+0000","last_fullsized":"2021-03-24T21:55:17.181376+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'22","last_scrub_stamp":"2021-03-24T21:54:51.585058+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2021-03-24T21:52:54.597505+0000","last_clean_scrub_stamp":"2021-03-24T21:54:51.585058+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":16,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":580,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"108","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.174081+0000","last_change":"2021-03-24T21:54:54.371278+0000","last_active":"2021-03-24T21:55:17.174081+0000","last_peered":"2021-03-24T21:55:17.174081+0000","last_clean":"2021-03-24T21:55:17.174081+0000","last_became_active":"2021-03-24T21:54:38.126662+0000","last_became_peered":"2021-03-24T21:54:38.126662+0000","last_unstale":"2021-03-24T21:55:17.174081+0000","last_undegraded":"2021-03-24T21:55:17.174081+0000","last_fullsized":"2021-03-24T21:55:17.174081+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:54.371106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:54:54.371106+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"105","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.181474+0000","last_change":"2021-03-24T21:54:38.129428+0000","last_active":"2021-03-24T21:55:17.181474+0000","last_peered":"2021-03-24T21:55:17.181474+0000","last_clean":"2021-03-24T21:55:17.181474+0000","last_became_active":"2021-03-24T21:54:38.128067+0000","last_became_peered":"2021-03-24T21:54:38.128067+0000","last_unstale":"2021-03-24T21:55:17.181474+0000","last_undegraded":"2021-03-24T21:55:17.181474+0000","last_fullsized":"2021-03-24T21:55:17.181474+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:31.665434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:31.665434+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":"101","reported_epoch":"149","state":"active+clean","last_fresh":"2021-03-24T21:55:18.200069+0000","last_change":"2021-03-24T21:53:08.841487+0000","last_active":"2021-03-24T21:55:18.200069+0000","last_peered":"2021-03-24T21:55:18.200069+0000","last_clean":"2021-03-24T21:55:18.200069+0000","last_became_active":"2021-03-24T21:51:45.379383+0000","last_became_peered":"2021-03-24T21:51:45.379383+0000","last_unstale":"2021-03-24T21:55:18.200069+0000","last_undegraded":"2021-03-24T21:55:18.200069+0000","last_fullsized":"2021-03-24T21:55:18.200069+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:08.841310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:08.841310+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"69'15","reported_seq":"117","reported_epoch":"151","state":"active+clean","last_fresh":"2021-03-24T21:55:20.099945+0000","last_change":"2021-03-24T21:54:38.182009+0000","last_active":"2021-03-24T21:55:20.099945+0000","last_peered":"2021-03-24T21:55:20.099945+0000","last_clean":"2021-03-24T21:55:20.099945+0000","last_became_active":"2021-03-24T21:54:38.122707+0000","last_became_peered":"2021-03-24T21:54:38.122707+0000","last_unstale":"2021-03-24T21:55:20.099945+0000","last_undegraded":"2021-03-24T21:55:20.099945+0000","last_fullsized":"2021-03-24T21:55:20.099945+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:54:06.651102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:54:06.651102+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":34,"num_read_kb":16,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"94","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.180934+0000","last_change":"2021-03-24T21:53:23.708000+0000","last_active":"2021-03-24T21:55:17.180934+0000","last_peered":"2021-03-24T21:55:17.180934+0000","last_clean":"2021-03-24T21:55:17.180934+0000","last_became_active":"2021-03-24T21:51:51.764871+0000","last_became_peered":"2021-03-24T21:51:51.764871+0000","last_unstale":"2021-03-24T21:55:17.180934+0000","last_undegraded":"2021-03-24T21:55:17.180934+0000","last_fullsized":"2021-03-24T21:55:17.180934+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:23.707914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:23.707914+0000","last_clean_scrub_stamp":"2021-03-24T21:53:23.707914+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"52","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.185906+0000","last_change":"2021-03-24T21:54:38.122046+0000","last_active":"2021-03-24T21:55:17.185906+0000","last_peered":"2021-03-24T21:55:17.185906+0000","last_clean":"2021-03-24T21:55:17.185906+0000","last_became_active":"2021-03-24T21:54:38.120989+0000","last_became_peered":"2021-03-24T21:54:38.120989+0000","last_unstale":"2021-03-24T21:55:17.185906+0000","last_undegraded":"2021-03-24T21:55:17.185906+0000","last_fullsized":"2021-03-24T21:55:17.185906+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:58.783414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:58.783414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":"108","reported_epoch":"147","state":"active+clean","last_fresh":"2021-03-24T21:55:16.166140+0000","last_change":"2021-03-24T21:54:24.181804+0000","last_active":"2021-03-24T21:55:16.166140+0000","last_peered":"2021-03-24T21:55:16.166140+0000","last_clean":"2021-03-24T21:55:16.166140+0000","last_became_active":"2021-03-24T21:54:24.181343+0000","last_became_peered":"2021-03-24T21:54:24.181343+0000","last_unstale":"2021-03-24T21:55:16.166140+0000","last_undegraded":"2021-03-24T21:55:16.166140+0000","last_fullsized":"2021-03-24T21:55:16.166140+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:36.316893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:36.316893+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"69'15","reported_seq":"182","reported_epoch":"147","state":"active+clean","last_fresh":"2021-03-24T21:55:16.167360+0000","last_change":"2021-03-24T21:54:39.243768+0000","last_active":"2021-03-24T21:55:16.167360+0000","last_peered":"2021-03-24T21:55:16.167360+0000","last_clean":"2021-03-24T21:55:16.167360+0000","last_became_active":"2021-03-24T21:54:38.118330+0000","last_became_peered":"2021-03-24T21:54:38.118330+0000","last_unstale":"2021-03-24T21:55:16.167360+0000","last_undegraded":"2021-03-24T21:55:16.167360+0000","last_fullsized":"2021-03-24T21:55:16.167360+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:53:21.207696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:21.207696+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"99","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.177127+0000","last_change":"2021-03-24T21:54:44.392954+0000","last_active":"2021-03-24T21:55:17.177127+0000","last_peered":"2021-03-24T21:55:17.177127+0000","last_clean":"2021-03-24T21:55:17.177127+0000","last_became_active":"2021-03-24T21:51:51.756584+0000","last_became_peered":"2021-03-24T21:51:51.756584+0000","last_unstale":"2021-03-24T21:55:17.177127+0000","last_undegraded":"2021-03-24T21:55:17.177127+0000","last_fullsized":"2021-03-24T21:55:17.177127+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:44.392808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:54:44.392808+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"109'4","reported_seq":"155","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.186857+0000","last_change":"2021-03-24T21:54:39.208463+0000","last_active":"2021-03-24T21:55:17.186857+0000","last_peered":"2021-03-24T21:55:17.186857+0000","last_clean":"2021-03-24T21:55:17.186857+0000","last_became_active":"2021-03-24T21:54:38.130026+0000","last_became_peered":"2021-03-24T21:54:38.130026+0000","last_unstale":"2021-03-24T21:55:17.186857+0000","last_undegraded":"2021-03-24T21:55:17.186857+0000","last_fullsized":"2021-03-24T21:55:17.186857+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'3","last_scrub_stamp":"2021-03-24T21:53:08.815163+0000","last_deep_scrub":"86'3","last_deep_scrub_stamp":"2021-03-24T21:53:08.815163+0000","last_clean_scrub_stamp":"2021-03-24T21:53:08.815163+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":"101","reported_epoch":"149","state":"active+clean","last_fresh":"2021-03-24T21:55:18.198500+0000","last_change":"2021-03-24T21:53:15.868395+0000","last_active":"2021-03-24T21:55:18.198500+0000","last_peered":"2021-03-24T21:55:18.198500+0000","last_clean":"2021-03-24T21:55:18.198500+0000","last_became_active":"2021-03-24T21:51:45.579271+0000","last_became_peered":"2021-03-24T21:51:45.579271+0000","last_unstale":"2021-03-24T21:55:18.198500+0000","last_undegraded":"2021-03-24T21:55:18.198500+0000","last_fullsized":"2021-03-24T21:55:18.198500+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:15.868303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:15.868303+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"69'14","reported_seq":"153","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.182023+0000","last_change":"2021-03-24T21:54:39.825486+0000","last_active":"2021-03-24T21:55:17.182023+0000","last_peered":"2021-03-24T21:55:17.182023+0000","last_clean":"2021-03-24T21:55:17.182023+0000","last_became_active":"2021-03-24T21:54:38.143783+0000","last_became_peered":"2021-03-24T21:54:38.143783+0000","last_unstale":"2021-03-24T21:55:17.182023+0000","last_undegraded":"2021-03-24T21:55:17.182023+0000","last_fullsized":"2021-03-24T21:55:17.182023+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'14","last_scrub_stamp":"2021-03-24T21:53:02.722970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:02.722970+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":27,"num_read_kb":14,"num_write":14,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"98","reported_epoch":"147","state":"active+clean","last_fresh":"2021-03-24T21:55:16.167316+0000","last_change":"2021-03-24T21:54:13.291633+0000","last_active":"2021-03-24T21:55:16.167316+0000","last_peered":"2021-03-24T21:55:16.167316+0000","last_clean":"2021-03-24T21:55:16.167316+0000","last_became_active":"2021-03-24T21:51:51.762876+0000","last_became_peered":"2021-03-24T21:51:51.762876+0000","last_unstale":"2021-03-24T21:55:16.167316+0000","last_undegraded":"2021-03-24T21:55:16.167316+0000","last_fullsized":"2021-03-24T21:55:16.167316+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:13.291461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:54:13.291461+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"86'3","reported_seq":"146","reported_epoch":"149","state":"active+clean","last_fresh":"2021-03-24T21:55:18.196096+0000","last_change":"2021-03-24T21:55:01.577952+0000","last_active":"2021-03-24T21:55:18.196096+0000","last_peered":"2021-03-24T21:55:18.196096+0000","last_clean":"2021-03-24T21:55:18.196096+0000","last_became_active":"2021-03-24T21:51:49.419797+0000","last_became_peered":"2021-03-24T21:51:49.419797+0000","last_unstale":"2021-03-24T21:55:18.196096+0000","last_undegraded":"2021-03-24T21:55:18.196096+0000","last_fullsized":"2021-03-24T21:55:18.196096+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'3","last_scrub_stamp":"2021-03-24T21:55:01.577769+0000","last_deep_scrub":"86'3","last_deep_scrub_stamp":"2021-03-24T21:52:56.678111+0000","last_clean_scrub_stamp":"2021-03-24T21:55:01.577769+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":"105","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.174804+0000","last_change":"2021-03-24T21:54:05.473528+0000","last_active":"2021-03-24T21:55:17.174804+0000","last_peered":"2021-03-24T21:55:17.174804+0000","last_clean":"2021-03-24T21:55:17.174804+0000","last_became_active":"2021-03-24T21:51:45.381516+0000","last_became_peered":"2021-03-24T21:51:45.381516+0000","last_unstale":"2021-03-24T21:55:17.174804+0000","last_undegraded":"2021-03-24T21:55:17.174804+0000","last_fullsized":"2021-03-24T21:55:17.174804+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:05.473348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:05.473348+0000","last_clean_scrub_stamp":"2021-03-24T21:54:05.473348+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"69'13","reported_seq":"124","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.187062+0000","last_change":"2021-03-24T21:53:15.901347+0000","last_active":"2021-03-24T21:55:17.187062+0000","last_peered":"2021-03-24T21:55:17.187062+0000","last_clean":"2021-03-24T21:55:17.187062+0000","last_became_active":"2021-03-24T21:51:47.689288+0000","last_became_peered":"2021-03-24T21:51:47.689288+0000","last_unstale":"2021-03-24T21:55:17.187062+0000","last_undegraded":"2021-03-24T21:55:17.187062+0000","last_fullsized":"2021-03-24T21:55:17.187062+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2021-03-24T21:53:15.901150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:15.901150+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"98","reported_epoch":"151","state":"active+clean","last_fresh":"2021-03-24T21:55:20.102381+0000","last_change":"2021-03-24T21:53:15.623047+0000","last_active":"2021-03-24T21:55:20.102381+0000","last_peered":"2021-03-24T21:55:20.102381+0000","last_clean":"2021-03-24T21:55:20.102381+0000","last_became_active":"2021-03-24T21:51:51.754085+0000","last_became_peered":"2021-03-24T21:51:51.754085+0000","last_unstale":"2021-03-24T21:55:20.102381+0000","last_undegraded":"2021-03-24T21:55:20.102381+0000","last_fullsized":"2021-03-24T21:55:20.102381+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:15.622946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:15.622946+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"110","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.183026+0000","last_change":"2021-03-24T21:54:55.493891+0000","last_active":"2021-03-24T21:55:17.183026+0000","last_peered":"2021-03-24T21:55:17.183026+0000","last_clean":"2021-03-24T21:55:17.183026+0000","last_became_active":"2021-03-24T21:54:38.129390+0000","last_became_peered":"2021-03-24T21:54:38.129390+0000","last_unstale":"2021-03-24T21:55:17.183026+0000","last_undegraded":"2021-03-24T21:55:17.183026+0000","last_fullsized":"2021-03-24T21:55:17.183026+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:55.493713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:55.493713+0000","last_clean_scrub_stamp":"2021-03-24T21:54:55.493713+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":"110","reported_epoch":"149","state":"active+clean","last_fresh":"2021-03-24T21:55:18.197934+0000","last_change":"2021-03-24T21:54:24.196658+0000","last_active":"2021-03-24T21:55:18.197934+0000","last_peered":"2021-03-24T21:55:18.197934+0000","last_clean":"2021-03-24T21:55:18.197934+0000","last_became_active":"2021-03-24T21:54:24.196290+0000","last_became_peered":"2021-03-24T21:54:24.196290+0000","last_unstale":"2021-03-24T21:55:18.197934+0000","last_undegraded":"2021-03-24T21:55:18.197934+0000","last_fullsized":"2021-03-24T21:55:18.197934+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:12.871269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:12.871269+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"69'16","reported_seq":"133","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.177858+0000","last_change":"2021-03-24T21:54:19.588964+0000","last_active":"2021-03-24T21:55:17.177858+0000","last_peered":"2021-03-24T21:55:17.177858+0000","last_clean":"2021-03-24T21:55:17.177858+0000","last_became_active":"2021-03-24T21:51:47.687595+0000","last_became_peered":"2021-03-24T21:51:47.687595+0000","last_unstale":"2021-03-24T21:55:17.177858+0000","last_undegraded":"2021-03-24T21:55:17.177858+0000","last_fullsized":"2021-03-24T21:55:17.177858+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T21:54:19.588797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:54:19.588797+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":38,"num_read_kb":18,"num_write":17,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"104","reported_epoch":"149","state":"active+clean","last_fresh":"2021-03-24T21:55:18.199928+0000","last_change":"2021-03-24T21:54:24.188745+0000","last_active":"2021-03-24T21:55:18.199928+0000","last_peered":"2021-03-24T21:55:18.199928+0000","last_clean":"2021-03-24T21:55:18.199928+0000","last_became_active":"2021-03-24T21:54:24.188386+0000","last_became_peered":"2021-03-24T21:54:24.188386+0000","last_unstale":"2021-03-24T21:55:18.199928+0000","last_undegraded":"2021-03-24T21:55:18.199928+0000","last_fullsized":"2021-03-24T21:55:18.199928+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:24.913238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:24.913238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"107","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.186729+0000","last_change":"2021-03-24T21:54:38.128829+0000","last_active":"2021-03-24T21:55:17.186729+0000","last_peered":"2021-03-24T21:55:17.186729+0000","last_clean":"2021-03-24T21:55:17.186729+0000","last_became_active":"2021-03-24T21:54:38.128409+0000","last_became_peered":"2021-03-24T21:54:38.128409+0000","last_unstale":"2021-03-24T21:55:17.186729+0000","last_undegraded":"2021-03-24T21:55:17.186729+0000","last_fullsized":"2021-03-24T21:55:17.186729+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:19.812822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:19.812822+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":"64","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.173571+0000","last_change":"2021-03-24T21:54:52.385672+0000","last_active":"2021-03-24T21:55:17.173571+0000","last_peered":"2021-03-24T21:55:17.173571+0000","last_clean":"2021-03-24T21:55:17.173571+0000","last_became_active":"2021-03-24T21:54:24.201388+0000","last_became_peered":"2021-03-24T21:54:24.201388+0000","last_unstale":"2021-03-24T21:55:17.173571+0000","last_undegraded":"2021-03-24T21:55:17.173571+0000","last_fullsized":"2021-03-24T21:55:17.173571+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:52.385567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:54:52.385567+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"69'7","reported_seq":"128","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.182879+0000","last_change":"2021-03-24T21:54:39.199780+0000","last_active":"2021-03-24T21:55:17.182879+0000","last_peered":"2021-03-24T21:55:17.182879+0000","last_clean":"2021-03-24T21:55:17.182879+0000","last_became_active":"2021-03-24T21:54:38.144098+0000","last_became_peered":"2021-03-24T21:54:38.144098+0000","last_unstale":"2021-03-24T21:55:17.182879+0000","last_undegraded":"2021-03-24T21:55:17.182879+0000","last_fullsized":"2021-03-24T21:55:17.182879+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'7","last_scrub_stamp":"2021-03-24T21:53:13.645928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:13.645928+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":7,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":158,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"149'89","reported_seq":"274","reported_epoch":"149","state":"active+clean","last_fresh":"2021-03-24T21:55:18.207753+0000","last_change":"2021-03-24T21:55:18.207998+0000","last_active":"2021-03-24T21:55:18.207753+0000","last_peered":"2021-03-24T21:55:18.207753+0000","last_clean":"2021-03-24T21:55:18.207753+0000","last_became_active":"2021-03-24T21:54:38.117586+0000","last_became_peered":"2021-03-24T21:54:38.117586+0000","last_unstale":"2021-03-24T21:55:18.207753+0000","last_undegraded":"2021-03-24T21:55:18.207753+0000","last_fullsized":"2021-03-24T21:55:18.207753+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":89,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21913486,"num_objects":8,"num_object_clones":3,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":62,"num_read_kb":52501,"num_write":46,"num_write_kb":21852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":6771983,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"17","length":"1"},{"start":"19","length":"1"},{"start":"1b","length":"1"},{"start":"1d","length":"1"},{"start":"1f","length":"2"},{"start":"23","length":"2"}]},{"pgid":"6.a","version":"0'0","reported_seq":"69","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.175320+0000","last_change":"2021-03-24T21:55:10.097929+0000","last_active":"2021-03-24T21:55:17.175320+0000","last_peered":"2021-03-24T21:55:17.175320+0000","last_clean":"2021-03-24T21:55:17.175320+0000","last_became_active":"2021-03-24T21:55:10.097570+0000","last_became_peered":"2021-03-24T21:55:10.097570+0000","last_unstale":"2021-03-24T21:55:17.175320+0000","last_undegraded":"2021-03-24T21:55:17.175320+0000","last_fullsized":"2021-03-24T21:55:17.175320+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:55.067965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:54:55.067965+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"89'4","reported_seq":"106","reported_epoch":"149","state":"active+clean","last_fresh":"2021-03-24T21:55:18.200835+0000","last_change":"2021-03-24T21:53:19.926732+0000","last_active":"2021-03-24T21:55:18.200835+0000","last_peered":"2021-03-24T21:55:18.200835+0000","last_clean":"2021-03-24T21:55:18.200835+0000","last_became_active":"2021-03-24T21:51:45.385110+0000","last_became_peered":"2021-03-24T21:51:45.385110+0000","last_unstale":"2021-03-24T21:55:18.200835+0000","last_undegraded":"2021-03-24T21:55:18.200835+0000","last_fullsized":"2021-03-24T21:55:18.200835+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'4","last_scrub_stamp":"2021-03-24T21:53:19.926637+0000","last_deep_scrub":"89'4","last_deep_scrub_stamp":"2021-03-24T21:53:19.926637+0000","last_clean_scrub_stamp":"2021-03-24T21:53:19.926637+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"97'28","reported_seq":"177","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.182980+0000","last_change":"2021-03-24T21:54:24.220625+0000","last_active":"2021-03-24T21:55:17.182980+0000","last_peered":"2021-03-24T21:55:17.182980+0000","last_clean":"2021-03-24T21:55:17.182980+0000","last_became_active":"2021-03-24T21:54:24.190817+0000","last_became_peered":"2021-03-24T21:54:24.190817+0000","last_unstale":"2021-03-24T21:55:17.182980+0000","last_undegraded":"2021-03-24T21:55:17.182980+0000","last_fullsized":"2021-03-24T21:55:17.182980+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'28","last_scrub_stamp":"2021-03-24T21:54:19.157265+0000","last_deep_scrub":"97'28","last_deep_scrub_stamp":"2021-03-24T21:54:19.157265+0000","last_clean_scrub_stamp":"2021-03-24T21:54:19.157265+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":20,"num_write":19,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"109'4","reported_seq":"172","reported_epoch":"149","state":"active+clean","last_fresh":"2021-03-24T21:55:18.881584+0000","last_change":"2021-03-24T21:54:38.223920+0000","last_active":"2021-03-24T21:55:18.881584+0000","last_peered":"2021-03-24T21:55:18.881584+0000","last_clean":"2021-03-24T21:55:18.881584+0000","last_became_active":"2021-03-24T21:54:38.125497+0000","last_became_peered":"2021-03-24T21:54:38.125497+0000","last_unstale":"2021-03-24T21:55:18.881584+0000","last_undegraded":"2021-03-24T21:55:18.881584+0000","last_fullsized":"2021-03-24T21:55:18.881584+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2021-03-24T21:54:04.706153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:54:04.706153+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"149'35","reported_seq":"177","reported_epoch":"149","state":"active+clean","last_fresh":"2021-03-24T21:55:18.206971+0000","last_change":"2021-03-24T21:55:18.207186+0000","last_active":"2021-03-24T21:55:18.206971+0000","last_peered":"2021-03-24T21:55:18.206971+0000","last_clean":"2021-03-24T21:55:18.206971+0000","last_became_active":"2021-03-24T21:54:22.167632+0000","last_became_peered":"2021-03-24T21:54:22.167632+0000","last_unstale":"2021-03-24T21:55:18.206971+0000","last_undegraded":"2021-03-24T21:55:18.206971+0000","last_fullsized":"2021-03-24T21:55:18.206971+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7936411,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":28,"num_write_kb":7401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"17","length":"1"},{"start":"19","length":"1"},{"start":"1b","length":"1"},{"start":"1d","length":"1"},{"start":"1f","length":"2"},{"start":"23","length":"2"}]},{"pgid":"6.b","version":"0'0","reported_seq":"103","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.181679+0000","last_change":"2021-03-24T21:54:38.133788+0000","last_active":"2021-03-24T21:55:17.181679+0000","last_peered":"2021-03-24T21:55:17.181679+0000","last_clean":"2021-03-24T21:55:17.181679+0000","last_became_active":"2021-03-24T21:54:38.129824+0000","last_became_peered":"2021-03-24T21:54:38.129824+0000","last_unstale":"2021-03-24T21:55:17.181679+0000","last_undegraded":"2021-03-24T21:55:17.181679+0000","last_fullsized":"2021-03-24T21:55:17.181679+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:22.724840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:22.724840+0000","last_clean_scrub_stamp":"2021-03-24T21:53:22.724840+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":"115","reported_epoch":"149","state":"active+clean","last_fresh":"2021-03-24T21:55:18.200660+0000","last_change":"2021-03-24T21:54:38.109846+0000","last_active":"2021-03-24T21:55:18.200660+0000","last_peered":"2021-03-24T21:55:18.200660+0000","last_clean":"2021-03-24T21:55:18.200660+0000","last_became_active":"2021-03-24T21:54:38.109439+0000","last_became_peered":"2021-03-24T21:54:38.109439+0000","last_unstale":"2021-03-24T21:55:18.200660+0000","last_undegraded":"2021-03-24T21:55:18.200660+0000","last_fullsized":"2021-03-24T21:55:18.200660+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:51.651164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:51.651164+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"69'15","reported_seq":"153","reported_epoch":"147","state":"active+clean","last_fresh":"2021-03-24T21:55:16.168154+0000","last_change":"2021-03-24T21:54:41.209644+0000","last_active":"2021-03-24T21:55:16.168154+0000","last_peered":"2021-03-24T21:55:16.168154+0000","last_clean":"2021-03-24T21:55:16.168154+0000","last_became_active":"2021-03-24T21:54:38.139863+0000","last_became_peered":"2021-03-24T21:54:38.139863+0000","last_unstale":"2021-03-24T21:55:16.168154+0000","last_undegraded":"2021-03-24T21:55:16.168154+0000","last_fullsized":"2021-03-24T21:55:16.168154+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:53:37.334345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:37.334345+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":19,"num_write":17,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":692,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"109","reported_epoch":"151","state":"active+clean","last_fresh":"2021-03-24T21:55:20.102567+0000","last_change":"2021-03-24T21:54:38.124840+0000","last_active":"2021-03-24T21:55:20.102567+0000","last_peered":"2021-03-24T21:55:20.102567+0000","last_clean":"2021-03-24T21:55:20.102567+0000","last_became_active":"2021-03-24T21:54:38.124317+0000","last_became_peered":"2021-03-24T21:54:38.124317+0000","last_unstale":"2021-03-24T21:55:20.102567+0000","last_undegraded":"2021-03-24T21:55:20.102567+0000","last_fullsized":"2021-03-24T21:55:20.102567+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:18.557122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:18.557122+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"149'88","reported_seq":"259","reported_epoch":"149","state":"active+clean","last_fresh":"2021-03-24T21:55:18.210589+0000","last_change":"2021-03-24T21:55:18.210806+0000","last_active":"2021-03-24T21:55:18.210589+0000","last_peered":"2021-03-24T21:55:18.210589+0000","last_clean":"2021-03-24T21:55:18.210589+0000","last_became_active":"2021-03-24T21:54:38.137395+0000","last_became_peered":"2021-03-24T21:54:38.137395+0000","last_unstale":"2021-03-24T21:55:18.210589+0000","last_undegraded":"2021-03-24T21:55:18.210589+0000","last_fullsized":"2021-03-24T21:55:18.210589+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":88,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17398320,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":57,"num_read_kb":43166,"num_write":43,"num_write_kb":15861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4396481,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"17","length":"1"},{"start":"19","length":"1"},{"start":"1b","length":"1"},{"start":"1d","length":"1"},{"start":"1f","length":"2"},{"start":"23","length":"2"}]},{"pgid":"6.8","version":"0'0","reported_seq":"100","reported_epoch":"149","state":"active+clean","last_fresh":"2021-03-24T21:55:18.197466+0000","last_change":"2021-03-24T21:54:46.859206+0000","last_active":"2021-03-24T21:55:18.197466+0000","last_peered":"2021-03-24T21:55:18.197466+0000","last_clean":"2021-03-24T21:55:18.197466+0000","last_became_active":"2021-03-24T21:51:51.762513+0000","last_became_peered":"2021-03-24T21:51:51.762513+0000","last_unstale":"2021-03-24T21:55:18.197466+0000","last_undegraded":"2021-03-24T21:55:18.197466+0000","last_fullsized":"2021-03-24T21:55:18.197466+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:46.859034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:54:46.859034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"105","reported_epoch":"151","state":"active+clean","last_fresh":"2021-03-24T21:55:20.102802+0000","last_change":"2021-03-24T21:54:51.202761+0000","last_active":"2021-03-24T21:55:20.102802+0000","last_peered":"2021-03-24T21:55:20.102802+0000","last_clean":"2021-03-24T21:55:20.102802+0000","last_became_active":"2021-03-24T21:51:49.417699+0000","last_became_peered":"2021-03-24T21:51:49.417699+0000","last_unstale":"2021-03-24T21:55:20.102802+0000","last_undegraded":"2021-03-24T21:55:20.102802+0000","last_fullsized":"2021-03-24T21:55:20.102802+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:51.202632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:54:51.202632+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":"110","reported_epoch":"149","state":"active+clean","last_fresh":"2021-03-24T21:55:18.200311+0000","last_change":"2021-03-24T21:54:24.189663+0000","last_active":"2021-03-24T21:55:18.200311+0000","last_peered":"2021-03-24T21:55:18.200311+0000","last_clean":"2021-03-24T21:55:18.200311+0000","last_became_active":"2021-03-24T21:54:24.189299+0000","last_became_peered":"2021-03-24T21:54:24.189299+0000","last_unstale":"2021-03-24T21:55:18.200311+0000","last_undegraded":"2021-03-24T21:55:18.200311+0000","last_fullsized":"2021-03-24T21:55:18.200311+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:26.908821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:26.908821+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"69'23","reported_seq":"133","reported_epoch":"149","state":"active+clean","last_fresh":"2021-03-24T21:55:18.197719+0000","last_change":"2021-03-24T21:54:39.420930+0000","last_active":"2021-03-24T21:55:18.197719+0000","last_peered":"2021-03-24T21:55:18.197719+0000","last_clean":"2021-03-24T21:55:18.197719+0000","last_became_active":"2021-03-24T21:54:38.138701+0000","last_became_peered":"2021-03-24T21:54:38.138701+0000","last_unstale":"2021-03-24T21:55:18.197719+0000","last_undegraded":"2021-03-24T21:55:18.197719+0000","last_fullsized":"2021-03-24T21:55:18.197719+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'23","last_scrub_stamp":"2021-03-24T21:54:18.973113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:54:18.973113+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":43,"num_read_kb":24,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":732,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"148'36","reported_seq":"190","reported_epoch":"151","state":"active+clean","last_fresh":"2021-03-24T21:55:20.104965+0000","last_change":"2021-03-24T21:55:20.104965+0000","last_active":"2021-03-24T21:55:20.104965+0000","last_peered":"2021-03-24T21:55:20.104965+0000","last_clean":"2021-03-24T21:55:20.104965+0000","last_became_active":"2021-03-24T21:54:22.166423+0000","last_became_peered":"2021-03-24T21:54:22.166423+0000","last_unstale":"2021-03-24T21:55:20.104965+0000","last_undegraded":"2021-03-24T21:55:20.104965+0000","last_fullsized":"2021-03-24T21:55:20.104965+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7610369,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":13221,"num_write":21,"num_write_kb":7377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"19","length":"1"},{"start":"1c","length":"3"},{"start":"20","length":"1"},{"start":"24","length":"1"}]},{"pgid":"6.9","version":"0'0","reported_seq":"94","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.174837+0000","last_change":"2021-03-24T21:53:35.636404+0000","last_active":"2021-03-24T21:55:17.174837+0000","last_peered":"2021-03-24T21:55:17.174837+0000","last_clean":"2021-03-24T21:55:17.174837+0000","last_became_active":"2021-03-24T21:51:51.763077+0000","last_became_peered":"2021-03-24T21:51:51.763077+0000","last_unstale":"2021-03-24T21:55:17.174837+0000","last_undegraded":"2021-03-24T21:55:17.174837+0000","last_fullsized":"2021-03-24T21:55:17.174837+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:35.636299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:35.636299+0000","last_clean_scrub_stamp":"2021-03-24T21:53:35.636299+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"105","reported_epoch":"151","state":"active+clean","last_fresh":"2021-03-24T21:55:20.100040+0000","last_change":"2021-03-24T21:54:54.139755+0000","last_active":"2021-03-24T21:55:20.100040+0000","last_peered":"2021-03-24T21:55:20.100040+0000","last_clean":"2021-03-24T21:55:20.100040+0000","last_became_active":"2021-03-24T21:51:49.411921+0000","last_became_peered":"2021-03-24T21:51:49.411921+0000","last_unstale":"2021-03-24T21:55:20.100040+0000","last_undegraded":"2021-03-24T21:55:20.100040+0000","last_fullsized":"2021-03-24T21:55:20.100040+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:54.139655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:54:54.139655+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"95","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.181569+0000","last_change":"2021-03-24T21:55:10.102704+0000","last_active":"2021-03-24T21:55:17.181569+0000","last_peered":"2021-03-24T21:55:17.181569+0000","last_clean":"2021-03-24T21:55:17.181569+0000","last_became_active":"2021-03-24T21:55:10.102360+0000","last_became_peered":"2021-03-24T21:55:10.102360+0000","last_unstale":"2021-03-24T21:55:17.181569+0000","last_undegraded":"2021-03-24T21:55:17.181569+0000","last_fullsized":"2021-03-24T21:55:17.181569+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:00.911020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:54:00.911020+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"69'14","reported_seq":"153","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.176449+0000","last_change":"2021-03-24T21:54:39.837964+0000","last_active":"2021-03-24T21:55:17.176449+0000","last_peered":"2021-03-24T21:55:17.176449+0000","last_clean":"2021-03-24T21:55:17.176449+0000","last_became_active":"2021-03-24T21:54:38.138034+0000","last_became_peered":"2021-03-24T21:54:38.138034+0000","last_unstale":"2021-03-24T21:55:17.176449+0000","last_undegraded":"2021-03-24T21:55:17.176449+0000","last_fullsized":"2021-03-24T21:55:17.176449+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'14","last_scrub_stamp":"2021-03-24T21:53:31.653334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:31.653334+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":16,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":894,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"149'91","reported_seq":"272","reported_epoch":"149","state":"active+clean","last_fresh":"2021-03-24T21:55:18.197417+0000","last_change":"2021-03-24T21:55:18.197756+0000","last_active":"2021-03-24T21:55:18.197417+0000","last_peered":"2021-03-24T21:55:18.197417+0000","last_clean":"2021-03-24T21:55:18.197417+0000","last_became_active":"2021-03-24T21:54:38.124237+0000","last_became_peered":"2021-03-24T21:54:38.124237+0000","last_unstale":"2021-03-24T21:55:18.197417+0000","last_undegraded":"2021-03-24T21:55:18.197417+0000","last_fullsized":"2021-03-24T21:55:18.197417+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":91,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24515109,"num_objects":11,"num_object_clones":6,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":77,"num_read_kb":61285,"num_write":56,"num_write_kb":18418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":5158008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"17","length":"1"},{"start":"19","length":"1"},{"start":"1b","length":"1"},{"start":"1d","length":"1"},{"start":"1f","length":"2"},{"start":"23","length":"2"}]},{"pgid":"6.6","version":"0'0","reported_seq":"99","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.183741+0000","last_change":"2021-03-24T21:54:10.519515+0000","last_active":"2021-03-24T21:55:17.183741+0000","last_peered":"2021-03-24T21:55:17.183741+0000","last_clean":"2021-03-24T21:55:17.183741+0000","last_became_active":"2021-03-24T21:51:51.759514+0000","last_became_peered":"2021-03-24T21:51:51.759514+0000","last_unstale":"2021-03-24T21:55:17.183741+0000","last_undegraded":"2021-03-24T21:55:17.183741+0000","last_fullsized":"2021-03-24T21:55:17.183741+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:10.519411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:52:59.706511+0000","last_clean_scrub_stamp":"2021-03-24T21:54:10.519411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"95","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.174940+0000","last_change":"2021-03-24T21:55:10.100603+0000","last_active":"2021-03-24T21:55:17.174940+0000","last_peered":"2021-03-24T21:55:17.174940+0000","last_clean":"2021-03-24T21:55:17.174940+0000","last_became_active":"2021-03-24T21:55:10.100245+0000","last_became_peered":"2021-03-24T21:55:10.100245+0000","last_unstale":"2021-03-24T21:55:17.174940+0000","last_undegraded":"2021-03-24T21:55:17.174940+0000","last_fullsized":"2021-03-24T21:55:17.174940+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:03.195853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:06.800417+0000","last_clean_scrub_stamp":"2021-03-24T21:55:03.195853+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"89'2","reported_seq":"128","reported_epoch":"147","state":"active+clean","last_fresh":"2021-03-24T21:55:16.166485+0000","last_change":"2021-03-24T21:54:41.206984+0000","last_active":"2021-03-24T21:55:16.166485+0000","last_peered":"2021-03-24T21:55:16.166485+0000","last_clean":"2021-03-24T21:55:16.166485+0000","last_became_active":"2021-03-24T21:54:38.144013+0000","last_became_peered":"2021-03-24T21:54:38.144013+0000","last_unstale":"2021-03-24T21:55:16.166485+0000","last_undegraded":"2021-03-24T21:55:16.166485+0000","last_fullsized":"2021-03-24T21:55:16.166485+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2021-03-24T21:53:28.219900+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2021-03-24T21:53:28.219900+0000","last_clean_scrub_stamp":"2021-03-24T21:53:28.219900+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"69'25","reported_seq":"142","reported_epoch":"151","state":"active+clean","last_fresh":"2021-03-24T21:55:20.103162+0000","last_change":"2021-03-24T21:54:39.051748+0000","last_active":"2021-03-24T21:55:20.103162+0000","last_peered":"2021-03-24T21:55:20.103162+0000","last_clean":"2021-03-24T21:55:20.103162+0000","last_became_active":"2021-03-24T21:54:38.141478+0000","last_became_peered":"2021-03-24T21:54:38.141478+0000","last_unstale":"2021-03-24T21:55:20.103162+0000","last_undegraded":"2021-03-24T21:55:20.103162+0000","last_fullsized":"2021-03-24T21:55:20.103162+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'25","last_scrub_stamp":"2021-03-24T21:53:15.719434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:15.719434+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":53,"num_read_kb":30,"num_write":28,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"105","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.177621+0000","last_change":"2021-03-24T21:54:38.137179+0000","last_active":"2021-03-24T21:55:17.177621+0000","last_peered":"2021-03-24T21:55:17.177621+0000","last_clean":"2021-03-24T21:55:17.177621+0000","last_became_active":"2021-03-24T21:54:38.136859+0000","last_became_peered":"2021-03-24T21:54:38.136859+0000","last_unstale":"2021-03-24T21:55:17.177621+0000","last_undegraded":"2021-03-24T21:55:17.177621+0000","last_fullsized":"2021-03-24T21:55:17.177621+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:23.578679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:23.578679+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"149'59","reported_seq":"231","reported_epoch":"149","state":"active+clean","last_fresh":"2021-03-24T21:55:18.200643+0000","last_change":"2021-03-24T21:55:18.200829+0000","last_active":"2021-03-24T21:55:18.200643+0000","last_peered":"2021-03-24T21:55:18.200643+0000","last_clean":"2021-03-24T21:55:18.200643+0000","last_became_active":"2021-03-24T21:54:38.137466+0000","last_became_peered":"2021-03-24T21:54:38.137466+0000","last_unstale":"2021-03-24T21:55:18.200643+0000","last_undegraded":"2021-03-24T21:55:18.200643+0000","last_fullsized":"2021-03-24T21:55:18.200643+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":59,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12709910,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":51,"num_read_kb":54588,"num_write":34,"num_write_kb":15363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4942492,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"17","length":"1"},{"start":"19","length":"1"},{"start":"1b","length":"1"},{"start":"1d","length":"1"},{"start":"1f","length":"2"},{"start":"23","length":"2"}]},{"pgid":"6.5","version":"0'0","reported_seq":"104","reported_epoch":"149","state":"active+clean","last_fresh":"2021-03-24T21:55:18.200554+0000","last_change":"2021-03-24T21:54:24.187342+0000","last_active":"2021-03-24T21:55:18.200554+0000","last_peered":"2021-03-24T21:55:18.200554+0000","last_clean":"2021-03-24T21:55:18.200554+0000","last_became_active":"2021-03-24T21:54:24.186957+0000","last_became_peered":"2021-03-24T21:54:24.186957+0000","last_unstale":"2021-03-24T21:55:18.200554+0000","last_undegraded":"2021-03-24T21:55:18.200554+0000","last_fullsized":"2021-03-24T21:55:18.200554+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:23.926884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:23.926884+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"61'1","reported_seq":"66","reported_epoch":"151","state":"active+clean","last_fresh":"2021-03-24T21:55:20.102734+0000","last_change":"2021-03-24T21:54:38.240632+0000","last_active":"2021-03-24T21:55:20.102734+0000","last_peered":"2021-03-24T21:55:20.102734+0000","last_clean":"2021-03-24T21:55:20.102734+0000","last_became_active":"2021-03-24T21:54:38.123139+0000","last_became_peered":"2021-03-24T21:54:38.123139+0000","last_unstale":"2021-03-24T21:55:20.102734+0000","last_undegraded":"2021-03-24T21:55:20.102734+0000","last_fullsized":"2021-03-24T21:55:20.102734+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2021-03-24T21:53:18.753026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:18.753026+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"113","reported_epoch":"151","state":"active+clean","last_fresh":"2021-03-24T21:55:20.099290+0000","last_change":"2021-03-24T21:53:55.497565+0000","last_active":"2021-03-24T21:55:20.099290+0000","last_peered":"2021-03-24T21:55:20.099290+0000","last_clean":"2021-03-24T21:55:20.099290+0000","last_became_active":"2021-03-24T21:51:40.351000+0000","last_became_peered":"2021-03-24T21:51:40.351000+0000","last_unstale":"2021-03-24T21:55:20.099290+0000","last_undegraded":"2021-03-24T21:55:20.099290+0000","last_fullsized":"2021-03-24T21:55:20.099290+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:55.497382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:39.286101+0000","last_clean_scrub_stamp":"2021-03-24T21:53:55.497382+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"89'22","reported_seq":"135","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.187423+0000","last_change":"2021-03-24T21:54:39.636900+0000","last_active":"2021-03-24T21:55:17.187423+0000","last_peered":"2021-03-24T21:55:17.187423+0000","last_clean":"2021-03-24T21:55:17.187423+0000","last_became_active":"2021-03-24T21:54:38.129459+0000","last_became_peered":"2021-03-24T21:54:38.129459+0000","last_unstale":"2021-03-24T21:55:17.187423+0000","last_undegraded":"2021-03-24T21:55:17.187423+0000","last_fullsized":"2021-03-24T21:55:17.187423+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'22","last_scrub_stamp":"2021-03-24T21:53:19.717244+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2021-03-24T21:53:19.717244+0000","last_clean_scrub_stamp":"2021-03-24T21:53:19.717244+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":36,"num_read_kb":21,"num_write":18,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":644,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"100","reported_epoch":"151","state":"active+clean","last_fresh":"2021-03-24T21:55:20.100944+0000","last_change":"2021-03-24T21:53:21.534119+0000","last_active":"2021-03-24T21:55:20.100944+0000","last_peered":"2021-03-24T21:55:20.100944+0000","last_clean":"2021-03-24T21:55:20.100944+0000","last_became_active":"2021-03-24T21:51:49.418848+0000","last_became_peered":"2021-03-24T21:51:49.418848+0000","last_unstale":"2021-03-24T21:55:20.100944+0000","last_undegraded":"2021-03-24T21:55:20.100944+0000","last_fullsized":"2021-03-24T21:55:20.100944+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:21.533882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:21.533882+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"149'49","reported_seq":"191","reported_epoch":"149","state":"active+clean","last_fresh":"2021-03-24T21:55:18.209941+0000","last_change":"2021-03-24T21:55:18.210152+0000","last_active":"2021-03-24T21:55:18.209941+0000","last_peered":"2021-03-24T21:55:18.209941+0000","last_clean":"2021-03-24T21:55:18.209941+0000","last_became_active":"2021-03-24T21:54:22.162608+0000","last_became_peered":"2021-03-24T21:54:22.162608+0000","last_unstale":"2021-03-24T21:55:18.209941+0000","last_undegraded":"2021-03-24T21:55:18.209941+0000","last_fullsized":"2021-03-24T21:55:18.209941+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":49,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9211372,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":14488,"num_write":26,"num_write_kb":9021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"17","length":"1"},{"start":"19","length":"1"},{"start":"1b","length":"1"},{"start":"1d","length":"1"},{"start":"1f","length":"2"},{"start":"23","length":"2"}]},{"pgid":"6.7","version":"0'0","reported_seq":"101","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.187190+0000","last_change":"2021-03-24T21:54:45.900904+0000","last_active":"2021-03-24T21:55:17.187190+0000","last_peered":"2021-03-24T21:55:17.187190+0000","last_clean":"2021-03-24T21:55:17.187190+0000","last_became_active":"2021-03-24T21:51:51.765236+0000","last_became_peered":"2021-03-24T21:51:51.765236+0000","last_unstale":"2021-03-24T21:55:17.187190+0000","last_undegraded":"2021-03-24T21:55:17.187190+0000","last_fullsized":"2021-03-24T21:55:17.187190+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:45.900726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:54:45.900726+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"129","reported_epoch":"149","state":"active+clean","last_fresh":"2021-03-24T21:55:18.197755+0000","last_change":"2021-03-24T21:54:38.110780+0000","last_active":"2021-03-24T21:55:18.197755+0000","last_peered":"2021-03-24T21:55:18.197755+0000","last_clean":"2021-03-24T21:55:18.197755+0000","last_became_active":"2021-03-24T21:54:38.110327+0000","last_became_peered":"2021-03-24T21:54:38.110327+0000","last_unstale":"2021-03-24T21:55:18.197755+0000","last_undegraded":"2021-03-24T21:55:18.197755+0000","last_fullsized":"2021-03-24T21:55:18.197755+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:30.906502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:49:51.529051+0000","last_clean_scrub_stamp":"2021-03-24T21:53:30.906502+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"109","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.183592+0000","last_change":"2021-03-24T21:54:24.199724+0000","last_active":"2021-03-24T21:55:17.183592+0000","last_peered":"2021-03-24T21:55:17.183592+0000","last_clean":"2021-03-24T21:55:17.183592+0000","last_became_active":"2021-03-24T21:54:24.199349+0000","last_became_peered":"2021-03-24T21:54:24.199349+0000","last_unstale":"2021-03-24T21:55:17.183592+0000","last_undegraded":"2021-03-24T21:55:17.183592+0000","last_fullsized":"2021-03-24T21:55:17.183592+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:20.736432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:20.736432+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"69'15","reported_seq":"114","reported_epoch":"147","state":"active+clean","last_fresh":"2021-03-24T21:55:16.166912+0000","last_change":"2021-03-24T21:54:25.040006+0000","last_active":"2021-03-24T21:55:16.166912+0000","last_peered":"2021-03-24T21:55:16.166912+0000","last_clean":"2021-03-24T21:55:16.166912+0000","last_became_active":"2021-03-24T21:54:24.194711+0000","last_became_peered":"2021-03-24T21:54:24.194711+0000","last_unstale":"2021-03-24T21:55:16.166912+0000","last_undegraded":"2021-03-24T21:55:16.166912+0000","last_fullsized":"2021-03-24T21:55:16.166912+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:53:20.279004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:20.279004+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":26,"num_read_kb":14,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":466,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"102","reported_epoch":"149","state":"active+clean","last_fresh":"2021-03-24T21:55:18.199940+0000","last_change":"2021-03-24T21:54:49.807761+0000","last_active":"2021-03-24T21:55:18.199940+0000","last_peered":"2021-03-24T21:55:18.199940+0000","last_clean":"2021-03-24T21:55:18.199940+0000","last_became_active":"2021-03-24T21:51:49.413251+0000","last_became_peered":"2021-03-24T21:51:49.413251+0000","last_unstale":"2021-03-24T21:55:18.199940+0000","last_undegraded":"2021-03-24T21:55:18.199940+0000","last_fullsized":"2021-03-24T21:55:18.199940+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:49.807585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:54:49.807585+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"144'27","reported_seq":"177","reported_epoch":"150","state":"active+clean","last_fresh":"2021-03-24T21:55:19.206759+0000","last_change":"2021-03-24T21:55:19.206759+0000","last_active":"2021-03-24T21:55:19.206759+0000","last_peered":"2021-03-24T21:55:19.206759+0000","last_clean":"2021-03-24T21:55:19.206759+0000","last_became_active":"2021-03-24T21:54:22.162598+0000","last_became_peered":"2021-03-24T21:54:22.162598+0000","last_unstale":"2021-03-24T21:55:19.206759+0000","last_undegraded":"2021-03-24T21:55:19.206759+0000","last_fullsized":"2021-03-24T21:55:19.206759+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6168103,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":6394,"num_write":16,"num_write_kb":7020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"17","length":"1"},{"start":"19","length":"1"},{"start":"1b","length":"1"},{"start":"1d","length":"4"},{"start":"23","length":"2"}]},{"pgid":"6.4","version":"0'0","reported_seq":"52","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.186692+0000","last_change":"2021-03-24T21:54:38.122715+0000","last_active":"2021-03-24T21:55:17.186692+0000","last_peered":"2021-03-24T21:55:17.186692+0000","last_clean":"2021-03-24T21:55:17.186692+0000","last_became_active":"2021-03-24T21:54:38.122320+0000","last_became_peered":"2021-03-24T21:54:38.122320+0000","last_unstale":"2021-03-24T21:55:17.186692+0000","last_undegraded":"2021-03-24T21:55:17.186692+0000","last_fullsized":"2021-03-24T21:55:17.186692+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:16.721504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:54:16.721504+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"105","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.173553+0000","last_change":"2021-03-24T21:54:18.594379+0000","last_active":"2021-03-24T21:55:17.173553+0000","last_peered":"2021-03-24T21:55:17.173553+0000","last_clean":"2021-03-24T21:55:17.173553+0000","last_became_active":"2021-03-24T21:51:45.376687+0000","last_became_peered":"2021-03-24T21:51:45.376687+0000","last_unstale":"2021-03-24T21:55:17.173553+0000","last_undegraded":"2021-03-24T21:55:17.173553+0000","last_fullsized":"2021-03-24T21:55:17.173553+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:18.594199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:54:18.594199+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"66'5","reported_seq":"339","reported_epoch":"150","state":"active+clean","last_fresh":"2021-03-24T21:55:19.200970+0000","last_change":"2021-03-24T21:54:38.495832+0000","last_active":"2021-03-24T21:55:19.200970+0000","last_peered":"2021-03-24T21:55:19.200970+0000","last_clean":"2021-03-24T21:55:19.200970+0000","last_became_active":"2021-03-24T21:54:38.125434+0000","last_became_peered":"2021-03-24T21:54:38.125434+0000","last_unstale":"2021-03-24T21:55:19.200970+0000","last_undegraded":"2021-03-24T21:55:19.200970+0000","last_fullsized":"2021-03-24T21:55:19.200970+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2021-03-24T21:54:07.669877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:39.286101+0000","last_clean_scrub_stamp":"2021-03-24T21:54:07.669877+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":212,"num_read_kb":207,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"69'16","reported_seq":"160","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.177295+0000","last_change":"2021-03-24T21:54:57.325888+0000","last_active":"2021-03-24T21:55:17.177295+0000","last_peered":"2021-03-24T21:55:17.177295+0000","last_clean":"2021-03-24T21:55:17.177295+0000","last_became_active":"2021-03-24T21:54:38.121714+0000","last_became_peered":"2021-03-24T21:54:38.121714+0000","last_unstale":"2021-03-24T21:55:17.177295+0000","last_undegraded":"2021-03-24T21:55:17.177295+0000","last_fullsized":"2021-03-24T21:55:17.177295+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T21:54:57.325723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:54:57.325723+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":16,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":630,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"107","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.187628+0000","last_change":"2021-03-24T21:54:38.129422+0000","last_active":"2021-03-24T21:55:17.187628+0000","last_peered":"2021-03-24T21:55:17.187628+0000","last_clean":"2021-03-24T21:55:17.187628+0000","last_became_active":"2021-03-24T21:54:38.128708+0000","last_became_peered":"2021-03-24T21:54:38.128708+0000","last_unstale":"2021-03-24T21:55:17.187628+0000","last_undegraded":"2021-03-24T21:55:17.187628+0000","last_fullsized":"2021-03-24T21:55:17.187628+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:20.835990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:20.835990+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"115'10","reported_seq":"167","reported_epoch":"149","state":"active+clean","last_fresh":"2021-03-24T21:55:18.195591+0000","last_change":"2021-03-24T21:55:18.195957+0000","last_active":"2021-03-24T21:55:18.195591+0000","last_peered":"2021-03-24T21:55:18.195591+0000","last_clean":"2021-03-24T21:55:18.195591+0000","last_became_active":"2021-03-24T21:54:38.122313+0000","last_became_peered":"2021-03-24T21:54:38.122313+0000","last_unstale":"2021-03-24T21:55:18.195591+0000","last_undegraded":"2021-03-24T21:55:18.195591+0000","last_fullsized":"2021-03-24T21:55:18.195591+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3008420,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":13133,"num_write":9,"num_write_kb":2415,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1253545,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"17","length":"1"},{"start":"19","length":"1"},{"start":"1b","length":"1"},{"start":"1d","length":"1"},{"start":"1f","length":"2"},{"start":"23","length":"2"}]},{"pgid":"6.1","version":"0'0","reported_seq":"103","reported_epoch":"151","state":"active+clean","last_fresh":"2021-03-24T21:55:20.099521+0000","last_change":"2021-03-24T21:54:38.124075+0000","last_active":"2021-03-24T21:55:20.099521+0000","last_peered":"2021-03-24T21:55:20.099521+0000","last_clean":"2021-03-24T21:55:20.099521+0000","last_became_active":"2021-03-24T21:54:38.122577+0000","last_became_peered":"2021-03-24T21:54:38.122577+0000","last_unstale":"2021-03-24T21:55:20.099521+0000","last_undegraded":"2021-03-24T21:55:20.099521+0000","last_fullsized":"2021-03-24T21:55:20.099521+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:28.691270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:28.691270+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"59","reported_epoch":"151","state":"active+clean","last_fresh":"2021-03-24T21:55:20.100678+0000","last_change":"2021-03-24T21:54:43.286516+0000","last_active":"2021-03-24T21:55:20.100678+0000","last_peered":"2021-03-24T21:55:20.100678+0000","last_clean":"2021-03-24T21:55:20.100678+0000","last_became_active":"2021-03-24T21:54:38.122875+0000","last_became_peered":"2021-03-24T21:54:38.122875+0000","last_unstale":"2021-03-24T21:55:20.100678+0000","last_undegraded":"2021-03-24T21:55:20.100678+0000","last_fullsized":"2021-03-24T21:55:20.100678+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:43.286336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:54:43.286336+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"69'21","reported_seq":"143","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.179468+0000","last_change":"2021-03-24T21:54:10.421436+0000","last_active":"2021-03-24T21:55:17.179468+0000","last_peered":"2021-03-24T21:55:17.179468+0000","last_clean":"2021-03-24T21:55:17.179468+0000","last_became_active":"2021-03-24T21:51:47.686486+0000","last_became_peered":"2021-03-24T21:51:47.686486+0000","last_unstale":"2021-03-24T21:55:17.179468+0000","last_undegraded":"2021-03-24T21:55:17.179468+0000","last_fullsized":"2021-03-24T21:55:17.179468+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'21","last_scrub_stamp":"2021-03-24T21:54:10.421258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:54:10.421258+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":45,"num_read_kb":27,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"59","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.174678+0000","last_change":"2021-03-24T21:54:24.199253+0000","last_active":"2021-03-24T21:55:17.174678+0000","last_peered":"2021-03-24T21:55:17.174678+0000","last_clean":"2021-03-24T21:55:17.174678+0000","last_became_active":"2021-03-24T21:54:24.198895+0000","last_became_peered":"2021-03-24T21:54:24.198895+0000","last_unstale":"2021-03-24T21:55:17.174678+0000","last_undegraded":"2021-03-24T21:55:17.174678+0000","last_fullsized":"2021-03-24T21:55:17.174678+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:23.246320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:23.246320+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"148'95","reported_seq":"265","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.192105+0000","last_change":"2021-03-24T21:55:17.192678+0000","last_active":"2021-03-24T21:55:17.192105+0000","last_peered":"2021-03-24T21:55:17.192105+0000","last_clean":"2021-03-24T21:55:17.192105+0000","last_became_active":"2021-03-24T21:54:38.139355+0000","last_became_peered":"2021-03-24T21:54:38.139355+0000","last_unstale":"2021-03-24T21:55:17.192105+0000","last_undegraded":"2021-03-24T21:55:17.192105+0000","last_fullsized":"2021-03-24T21:55:17.192105+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":95,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13899224,"num_objects":8,"num_object_clones":3,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":3,"num_read":42,"num_read_kb":27023,"num_write":56,"num_write_kb":21109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":6977532,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"17","length":"1"},{"start":"19","length":"1"},{"start":"1b","length":"1"},{"start":"1f","length":"2"},{"start":"23","length":"2"}]},{"pgid":"6.0","version":"0'0","reported_seq":"94","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.173283+0000","last_change":"2021-03-24T21:53:25.593543+0000","last_active":"2021-03-24T21:55:17.173283+0000","last_peered":"2021-03-24T21:55:17.173283+0000","last_clean":"2021-03-24T21:55:17.173283+0000","last_became_active":"2021-03-24T21:51:51.759627+0000","last_became_peered":"2021-03-24T21:51:51.759627+0000","last_unstale":"2021-03-24T21:55:17.173283+0000","last_undegraded":"2021-03-24T21:55:17.173283+0000","last_fullsized":"2021-03-24T21:55:17.173283+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:25.593358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:25.593358+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"107","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.186969+0000","last_change":"2021-03-24T21:54:20.101633+0000","last_active":"2021-03-24T21:55:17.186969+0000","last_peered":"2021-03-24T21:55:17.186969+0000","last_clean":"2021-03-24T21:55:17.186969+0000","last_became_active":"2021-03-24T21:51:45.583885+0000","last_became_peered":"2021-03-24T21:51:45.583885+0000","last_unstale":"2021-03-24T21:55:17.186969+0000","last_undegraded":"2021-03-24T21:55:17.186969+0000","last_fullsized":"2021-03-24T21:55:17.186969+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:20.101463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:54:20.101463+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"69'16","reported_seq":"121","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.185768+0000","last_change":"2021-03-24T21:54:51.015209+0000","last_active":"2021-03-24T21:55:17.185768+0000","last_peered":"2021-03-24T21:55:17.185768+0000","last_clean":"2021-03-24T21:55:17.185768+0000","last_became_active":"2021-03-24T21:54:38.128745+0000","last_became_peered":"2021-03-24T21:54:38.128745+0000","last_unstale":"2021-03-24T21:55:17.185768+0000","last_undegraded":"2021-03-24T21:55:17.185768+0000","last_fullsized":"2021-03-24T21:55:17.185768+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T21:54:51.015019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:54:51.015019+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":32,"num_read_kb":15,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":940,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"102'4","reported_seq":"154","reported_epoch":"149","state":"active+clean","last_fresh":"2021-03-24T21:55:18.194670+0000","last_change":"2021-03-24T21:54:24.305088+0000","last_active":"2021-03-24T21:55:18.194670+0000","last_peered":"2021-03-24T21:55:18.194670+0000","last_clean":"2021-03-24T21:55:18.194670+0000","last_became_active":"2021-03-24T21:54:24.203310+0000","last_became_peered":"2021-03-24T21:54:24.203310+0000","last_unstale":"2021-03-24T21:55:18.194670+0000","last_undegraded":"2021-03-24T21:55:18.194670+0000","last_fullsized":"2021-03-24T21:55:18.194670+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'3","last_scrub_stamp":"2021-03-24T21:53:26.584678+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-24T21:53:26.584678+0000","last_clean_scrub_stamp":"2021-03-24T21:53:26.584678+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"117'4","reported_seq":"157","reported_epoch":"149","state":"active+clean","last_fresh":"2021-03-24T21:55:18.195086+0000","last_change":"2021-03-24T21:55:18.197617+0000","last_active":"2021-03-24T21:55:18.195086+0000","last_peered":"2021-03-24T21:55:18.195086+0000","last_clean":"2021-03-24T21:55:18.195086+0000","last_became_active":"2021-03-24T21:54:38.143356+0000","last_became_peered":"2021-03-24T21:54:38.143356+0000","last_unstale":"2021-03-24T21:55:18.195086+0000","last_undegraded":"2021-03-24T21:55:18.195086+0000","last_fullsized":"2021-03-24T21:55:18.195086+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3042034,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":5,"num_write_kb":3819,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":867724,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"17","length":"1"},{"start":"19","length":"1"},{"start":"1b","length":"1"},{"start":"1d","length":"1"},{"start":"1f","length":"2"},{"start":"23","length":"2"}]},{"pgid":"6.3","version":"0'0","reported_seq":"118","reported_epoch":"149","state":"active+clean","last_fresh":"2021-03-24T21:55:18.198937+0000","last_change":"2021-03-24T21:54:44.933344+0000","last_active":"2021-03-24T21:55:18.198937+0000","last_peered":"2021-03-24T21:55:18.198937+0000","last_clean":"2021-03-24T21:55:18.198937+0000","last_became_active":"2021-03-24T21:54:38.110246+0000","last_became_peered":"2021-03-24T21:54:38.110246+0000","last_unstale":"2021-03-24T21:55:18.198937+0000","last_undegraded":"2021-03-24T21:55:18.198937+0000","last_fullsized":"2021-03-24T21:55:18.198937+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:44.933255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:54:44.933255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"109","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.177271+0000","last_change":"2021-03-24T21:54:38.134945+0000","last_active":"2021-03-24T21:55:17.177271+0000","last_peered":"2021-03-24T21:55:17.177271+0000","last_clean":"2021-03-24T21:55:17.177271+0000","last_became_active":"2021-03-24T21:54:38.134604+0000","last_became_peered":"2021-03-24T21:54:38.134604+0000","last_unstale":"2021-03-24T21:55:17.177271+0000","last_undegraded":"2021-03-24T21:55:17.177271+0000","last_fullsized":"2021-03-24T21:55:17.177271+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:24.621749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:24.621749+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"69'20","reported_seq":"172","reported_epoch":"147","state":"active+clean","last_fresh":"2021-03-24T21:55:16.167881+0000","last_change":"2021-03-24T21:55:00.135489+0000","last_active":"2021-03-24T21:55:16.167881+0000","last_peered":"2021-03-24T21:55:16.167881+0000","last_clean":"2021-03-24T21:55:16.167881+0000","last_became_active":"2021-03-24T21:54:24.189182+0000","last_became_peered":"2021-03-24T21:54:24.189182+0000","last_unstale":"2021-03-24T21:55:16.167881+0000","last_undegraded":"2021-03-24T21:55:16.167881+0000","last_fullsized":"2021-03-24T21:55:16.167881+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'20","last_scrub_stamp":"2021-03-24T21:55:00.135328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:00.135328+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":39,"num_read_kb":20,"num_write":20,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":940,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"109'3","reported_seq":"158","reported_epoch":"149","state":"active+clean","last_fresh":"2021-03-24T21:55:18.192997+0000","last_change":"2021-03-24T21:54:39.228237+0000","last_active":"2021-03-24T21:55:18.192997+0000","last_peered":"2021-03-24T21:55:18.192997+0000","last_clean":"2021-03-24T21:55:18.192997+0000","last_became_active":"2021-03-24T21:54:38.135726+0000","last_became_peered":"2021-03-24T21:54:38.135726+0000","last_unstale":"2021-03-24T21:55:18.192997+0000","last_undegraded":"2021-03-24T21:55:18.192997+0000","last_fullsized":"2021-03-24T21:55:18.192997+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2021-03-24T21:54:17.493861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:54:17.493861+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"149'82","reported_seq":"232","reported_epoch":"149","state":"active+clean","last_fresh":"2021-03-24T21:55:18.222768+0000","last_change":"2021-03-24T21:55:18.203460+0000","last_active":"2021-03-24T21:55:18.222768+0000","last_peered":"2021-03-24T21:55:18.222768+0000","last_clean":"2021-03-24T21:55:18.222768+0000","last_became_active":"2021-03-24T21:54:22.167761+0000","last_became_peered":"2021-03-24T21:54:22.167761+0000","last_unstale":"2021-03-24T21:55:18.222768+0000","last_undegraded":"2021-03-24T21:55:18.222768+0000","last_fullsized":"2021-03-24T21:55:18.222768+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":82,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22813991,"num_objects":9,"num_object_clones":5,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":46,"num_read_kb":43159,"num_write":49,"num_write_kb":21176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"17","length":"1"},{"start":"19","length":"1"},{"start":"1b","length":"1"},{"start":"1d","length":"1"},{"start":"1f","length":"2"},{"start":"23","length":"2"}]},{"pgid":"6.2","version":"0'0","reported_seq":"93","reported_epoch":"147","state":"active+clean","last_fresh":"2021-03-24T21:55:16.165513+0000","last_change":"2021-03-24T21:53:45.349966+0000","last_active":"2021-03-24T21:55:16.165513+0000","last_peered":"2021-03-24T21:55:16.165513+0000","last_clean":"2021-03-24T21:55:16.165513+0000","last_became_active":"2021-03-24T21:51:51.756114+0000","last_became_peered":"2021-03-24T21:51:51.756114+0000","last_unstale":"2021-03-24T21:55:16.165513+0000","last_undegraded":"2021-03-24T21:55:16.165513+0000","last_fullsized":"2021-03-24T21:55:16.165513+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:45.349781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:45.349781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"105","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.182716+0000","last_change":"2021-03-24T21:54:56.508843+0000","last_active":"2021-03-24T21:55:17.182716+0000","last_peered":"2021-03-24T21:55:17.182716+0000","last_clean":"2021-03-24T21:55:17.182716+0000","last_became_active":"2021-03-24T21:51:45.378704+0000","last_became_peered":"2021-03-24T21:51:45.378704+0000","last_unstale":"2021-03-24T21:55:17.182716+0000","last_undegraded":"2021-03-24T21:55:17.182716+0000","last_fullsized":"2021-03-24T21:55:17.182716+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:56.508652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:54:56.508652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"69'22","reported_seq":"138","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.181262+0000","last_change":"2021-03-24T21:53:17.716726+0000","last_active":"2021-03-24T21:55:17.181262+0000","last_peered":"2021-03-24T21:55:17.181262+0000","last_clean":"2021-03-24T21:55:17.181262+0000","last_became_active":"2021-03-24T21:51:47.691459+0000","last_became_peered":"2021-03-24T21:51:47.691459+0000","last_unstale":"2021-03-24T21:55:17.181262+0000","last_undegraded":"2021-03-24T21:55:17.181262+0000","last_fullsized":"2021-03-24T21:55:17.181262+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'22","last_scrub_stamp":"2021-03-24T21:53:17.716470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:17.716470+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":46,"num_read_kb":26,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"95","reported_epoch":"147","state":"active+clean","last_fresh":"2021-03-24T21:55:16.167518+0000","last_change":"2021-03-24T21:53:25.220598+0000","last_active":"2021-03-24T21:55:16.167518+0000","last_peered":"2021-03-24T21:55:16.167518+0000","last_clean":"2021-03-24T21:55:16.167518+0000","last_became_active":"2021-03-24T21:51:49.408761+0000","last_became_peered":"2021-03-24T21:51:49.408761+0000","last_unstale":"2021-03-24T21:55:16.167518+0000","last_undegraded":"2021-03-24T21:55:16.167518+0000","last_fullsized":"2021-03-24T21:55:16.167518+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:25.220436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:25.220436+0000","last_clean_scrub_stamp":"2021-03-24T21:53:25.220436+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"149'67","reported_seq":"211","reported_epoch":"149","state":"active+clean","last_fresh":"2021-03-24T21:55:18.204216+0000","last_change":"2021-03-24T21:55:18.204850+0000","last_active":"2021-03-24T21:55:18.204216+0000","last_peered":"2021-03-24T21:55:18.204216+0000","last_clean":"2021-03-24T21:55:18.204216+0000","last_became_active":"2021-03-24T21:54:24.190419+0000","last_became_peered":"2021-03-24T21:54:24.190419+0000","last_unstale":"2021-03-24T21:55:18.204216+0000","last_undegraded":"2021-03-24T21:55:18.204216+0000","last_fullsized":"2021-03-24T21:55:18.204216+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":67,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11983259,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":23,"num_read_kb":3494,"num_write":38,"num_write_kb":13291,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"17","length":"1"},{"start":"19","length":"1"},{"start":"1b","length":"1"},{"start":"1d","length":"1"},{"start":"1f","length":"2"},{"start":"23","length":"2"}]},{"pgid":"6.d","version":"0'0","reported_seq":"105","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.186784+0000","last_change":"2021-03-24T21:54:38.129995+0000","last_active":"2021-03-24T21:55:17.186784+0000","last_peered":"2021-03-24T21:55:17.186784+0000","last_clean":"2021-03-24T21:55:17.186784+0000","last_became_active":"2021-03-24T21:54:38.129469+0000","last_became_peered":"2021-03-24T21:54:38.129469+0000","last_unstale":"2021-03-24T21:55:17.186784+0000","last_undegraded":"2021-03-24T21:55:17.186784+0000","last_fullsized":"2021-03-24T21:55:17.186784+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:11.925149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:11.925149+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"65'2","reported_seq":"138","reported_epoch":"147","state":"active+clean","last_fresh":"2021-03-24T21:55:16.167258+0000","last_change":"2021-03-24T21:53:29.268452+0000","last_active":"2021-03-24T21:55:16.167258+0000","last_peered":"2021-03-24T21:55:16.167258+0000","last_clean":"2021-03-24T21:55:16.167258+0000","last_became_active":"2021-03-24T21:51:49.414380+0000","last_became_peered":"2021-03-24T21:51:49.414380+0000","last_unstale":"2021-03-24T21:55:16.167258+0000","last_undegraded":"2021-03-24T21:55:16.167258+0000","last_fullsized":"2021-03-24T21:55:16.167258+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2021-03-24T21:53:29.268273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:29.268273+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":"109","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.181153+0000","last_change":"2021-03-24T21:54:38.135257+0000","last_active":"2021-03-24T21:55:17.181153+0000","last_peered":"2021-03-24T21:55:17.181153+0000","last_clean":"2021-03-24T21:55:17.181153+0000","last_became_active":"2021-03-24T21:54:38.134768+0000","last_became_peered":"2021-03-24T21:54:38.134768+0000","last_unstale":"2021-03-24T21:55:17.181153+0000","last_undegraded":"2021-03-24T21:55:17.181153+0000","last_fullsized":"2021-03-24T21:55:17.181153+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:35.609173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:35.609173+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"69'17","reported_seq":"113","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.182783+0000","last_change":"2021-03-24T21:54:38.505905+0000","last_active":"2021-03-24T21:55:17.182783+0000","last_peered":"2021-03-24T21:55:17.182783+0000","last_clean":"2021-03-24T21:55:17.182783+0000","last_became_active":"2021-03-24T21:54:38.130536+0000","last_became_peered":"2021-03-24T21:54:38.130536+0000","last_unstale":"2021-03-24T21:55:17.182783+0000","last_undegraded":"2021-03-24T21:55:17.182783+0000","last_fullsized":"2021-03-24T21:55:17.182783+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'17","last_scrub_stamp":"2021-03-24T21:54:03.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:54:03.731436+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":18,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":422,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"149'92","reported_seq":"256","reported_epoch":"149","state":"active+clean","last_fresh":"2021-03-24T21:55:18.206090+0000","last_change":"2021-03-24T21:55:18.206297+0000","last_active":"2021-03-24T21:55:18.206090+0000","last_peered":"2021-03-24T21:55:18.206090+0000","last_clean":"2021-03-24T21:55:18.206090+0000","last_became_active":"2021-03-24T21:54:38.123502+0000","last_became_peered":"2021-03-24T21:54:38.123502+0000","last_unstale":"2021-03-24T21:55:18.206090+0000","last_undegraded":"2021-03-24T21:55:18.206090+0000","last_fullsized":"2021-03-24T21:55:18.206090+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":92,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17415589,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":24712,"num_write":62,"num_write_kb":20536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":5093012,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"17","length":"1"},{"start":"19","length":"1"},{"start":"1b","length":"1"},{"start":"1d","length":"1"},{"start":"1f","length":"2"},{"start":"23","length":"2"}]},{"pgid":"6.c","version":"0'0","reported_seq":"103","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.181461+0000","last_change":"2021-03-24T21:54:24.196940+0000","last_active":"2021-03-24T21:55:17.181461+0000","last_peered":"2021-03-24T21:55:17.181461+0000","last_clean":"2021-03-24T21:55:17.181461+0000","last_became_active":"2021-03-24T21:54:24.196587+0000","last_became_peered":"2021-03-24T21:54:24.196587+0000","last_unstale":"2021-03-24T21:55:17.181461+0000","last_undegraded":"2021-03-24T21:55:17.181461+0000","last_fullsized":"2021-03-24T21:55:17.181461+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:27.701002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:27.701002+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"77","reported_epoch":"147","state":"active+clean","last_fresh":"2021-03-24T21:55:16.167253+0000","last_change":"2021-03-24T21:55:10.102484+0000","last_active":"2021-03-24T21:55:16.167253+0000","last_peered":"2021-03-24T21:55:16.167253+0000","last_clean":"2021-03-24T21:55:16.167253+0000","last_became_active":"2021-03-24T21:55:10.102113+0000","last_became_peered":"2021-03-24T21:55:10.102113+0000","last_unstale":"2021-03-24T21:55:16.167253+0000","last_undegraded":"2021-03-24T21:55:16.167253+0000","last_fullsized":"2021-03-24T21:55:16.167253+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:12.170934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:54:12.170934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"104","reported_epoch":"147","state":"active+clean","last_fresh":"2021-03-24T21:55:16.167711+0000","last_change":"2021-03-24T21:54:53.267409+0000","last_active":"2021-03-24T21:55:16.167711+0000","last_peered":"2021-03-24T21:55:16.167711+0000","last_clean":"2021-03-24T21:55:16.167711+0000","last_became_active":"2021-03-24T21:51:45.579513+0000","last_became_peered":"2021-03-24T21:51:45.579513+0000","last_unstale":"2021-03-24T21:55:16.167711+0000","last_undegraded":"2021-03-24T21:55:16.167711+0000","last_fullsized":"2021-03-24T21:55:16.167711+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:53.267324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:54:53.267324+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"69'15","reported_seq":"187","reported_epoch":"147","state":"active+clean","last_fresh":"2021-03-24T21:55:16.167214+0000","last_change":"2021-03-24T21:54:40.017534+0000","last_active":"2021-03-24T21:55:16.167214+0000","last_peered":"2021-03-24T21:55:16.167214+0000","last_clean":"2021-03-24T21:55:16.167214+0000","last_became_active":"2021-03-24T21:54:38.127461+0000","last_became_peered":"2021-03-24T21:54:38.127461+0000","last_unstale":"2021-03-24T21:55:16.167214+0000","last_undegraded":"2021-03-24T21:55:16.167214+0000","last_fullsized":"2021-03-24T21:55:16.167214+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:54:02.296830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:54:02.296830+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1476,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"151'16","reported_seq":"169","reported_epoch":"151","state":"active+clean","last_fresh":"2021-03-24T21:55:20.586910+0000","last_change":"2021-03-24T21:55:20.105835+0000","last_active":"2021-03-24T21:55:20.586910+0000","last_peered":"2021-03-24T21:55:20.586910+0000","last_clean":"2021-03-24T21:55:20.586910+0000","last_became_active":"2021-03-24T21:54:22.168397+0000","last_became_peered":"2021-03-24T21:54:22.168397+0000","last_unstale":"2021-03-24T21:55:20.586910+0000","last_undegraded":"2021-03-24T21:55:20.586910+0000","last_fullsized":"2021-03-24T21:55:20.586910+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4655506,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":13,"num_read_kb":11428,"num_write":11,"num_write_kb":2188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"19","length":"1"},{"start":"1c","length":"3"},{"start":"20","length":"1"},{"start":"24","length":"1"}]},{"pgid":"6.f","version":"0'0","reported_seq":"98","reported_epoch":"151","state":"active+clean","last_fresh":"2021-03-24T21:55:20.101891+0000","last_change":"2021-03-24T21:53:14.576861+0000","last_active":"2021-03-24T21:55:20.101891+0000","last_peered":"2021-03-24T21:55:20.101891+0000","last_clean":"2021-03-24T21:55:20.101891+0000","last_became_active":"2021-03-24T21:51:51.765067+0000","last_became_peered":"2021-03-24T21:51:51.765067+0000","last_unstale":"2021-03-24T21:55:20.101891+0000","last_undegraded":"2021-03-24T21:55:20.101891+0000","last_fullsized":"2021-03-24T21:55:20.101891+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:14.576687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:14.576687+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"51","reported_epoch":"147","state":"active+clean","last_fresh":"2021-03-24T21:55:16.166213+0000","last_change":"2021-03-24T21:54:38.139233+0000","last_active":"2021-03-24T21:55:16.166213+0000","last_peered":"2021-03-24T21:55:16.166213+0000","last_clean":"2021-03-24T21:55:16.166213+0000","last_became_active":"2021-03-24T21:54:38.138864+0000","last_became_peered":"2021-03-24T21:54:38.138864+0000","last_unstale":"2021-03-24T21:55:16.166213+0000","last_undegraded":"2021-03-24T21:55:16.166213+0000","last_fullsized":"2021-03-24T21:55:16.166213+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:29.735672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:29.735672+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"71","reported_epoch":"147","state":"active+clean","last_fresh":"2021-03-24T21:55:16.165624+0000","last_change":"2021-03-24T21:55:01.169611+0000","last_active":"2021-03-24T21:55:16.165624+0000","last_peered":"2021-03-24T21:55:16.165624+0000","last_clean":"2021-03-24T21:55:16.165624+0000","last_became_active":"2021-03-24T21:54:38.127136+0000","last_became_peered":"2021-03-24T21:54:38.127136+0000","last_unstale":"2021-03-24T21:55:16.165624+0000","last_undegraded":"2021-03-24T21:55:16.165624+0000","last_fullsized":"2021-03-24T21:55:16.165624+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:01.169461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:55:01.169461+0000","last_clean_scrub_stamp":"2021-03-24T21:55:01.169461+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"69'21","reported_seq":"173","reported_epoch":"147","state":"active+clean","last_fresh":"2021-03-24T21:55:16.166418+0000","last_change":"2021-03-24T21:54:56.226932+0000","last_active":"2021-03-24T21:55:16.166418+0000","last_peered":"2021-03-24T21:55:16.166418+0000","last_clean":"2021-03-24T21:55:16.166418+0000","last_became_active":"2021-03-24T21:54:38.144020+0000","last_became_peered":"2021-03-24T21:54:38.144020+0000","last_unstale":"2021-03-24T21:55:16.166418+0000","last_undegraded":"2021-03-24T21:55:16.166418+0000","last_fullsized":"2021-03-24T21:55:16.166418+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'21","last_scrub_stamp":"2021-03-24T21:54:56.226757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:54:56.226757+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":23,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"149'16","reported_seq":"160","reported_epoch":"149","state":"active+clean","last_fresh":"2021-03-24T21:55:18.205453+0000","last_change":"2021-03-24T21:55:18.205715+0000","last_active":"2021-03-24T21:55:18.205453+0000","last_peered":"2021-03-24T21:55:18.205453+0000","last_clean":"2021-03-24T21:55:18.205453+0000","last_became_active":"2021-03-24T21:54:22.160108+0000","last_became_peered":"2021-03-24T21:54:22.160108+0000","last_unstale":"2021-03-24T21:55:18.205453+0000","last_undegraded":"2021-03-24T21:55:18.205453+0000","last_fullsized":"2021-03-24T21:55:18.205453+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9058874,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":15,"num_read_kb":15270,"num_write":9,"num_write_kb":6511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"17","length":"1"},{"start":"19","length":"1"},{"start":"1b","length":"1"},{"start":"1d","length":"1"},{"start":"1f","length":"2"},{"start":"23","length":"2"}]},{"pgid":"6.e","version":"0'0","reported_seq":"102","reported_epoch":"147","state":"active+clean","last_fresh":"2021-03-24T21:55:16.167376+0000","last_change":"2021-03-24T21:54:38.117075+0000","last_active":"2021-03-24T21:55:16.167376+0000","last_peered":"2021-03-24T21:55:16.167376+0000","last_clean":"2021-03-24T21:55:16.167376+0000","last_became_active":"2021-03-24T21:54:38.116786+0000","last_became_peered":"2021-03-24T21:54:38.116786+0000","last_unstale":"2021-03-24T21:55:16.167376+0000","last_undegraded":"2021-03-24T21:55:16.167376+0000","last_fullsized":"2021-03-24T21:55:16.167376+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:40.289984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:40.289984+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"65'2","reported_seq":"149","reported_epoch":"151","state":"active+clean","last_fresh":"2021-03-24T21:55:20.101284+0000","last_change":"2021-03-24T21:54:56.160424+0000","last_active":"2021-03-24T21:55:20.101284+0000","last_peered":"2021-03-24T21:55:20.101284+0000","last_clean":"2021-03-24T21:55:20.101284+0000","last_became_active":"2021-03-24T21:51:49.416182+0000","last_became_peered":"2021-03-24T21:51:49.416182+0000","last_unstale":"2021-03-24T21:55:20.101284+0000","last_undegraded":"2021-03-24T21:55:20.101284+0000","last_fullsized":"2021-03-24T21:55:20.101284+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2021-03-24T21:54:56.160326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:54:56.160326+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"105","reported_epoch":"148","state":"active+clean","last_fresh":"2021-03-24T21:55:17.183053+0000","last_change":"2021-03-24T21:53:57.501519+0000","last_active":"2021-03-24T21:55:17.183053+0000","last_peered":"2021-03-24T21:55:17.183053+0000","last_clean":"2021-03-24T21:55:17.183053+0000","last_became_active":"2021-03-24T21:51:45.375830+0000","last_became_peered":"2021-03-24T21:51:45.375830+0000","last_unstale":"2021-03-24T21:55:17.183053+0000","last_undegraded":"2021-03-24T21:55:17.183053+0000","last_fullsized":"2021-03-24T21:55:17.183053+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:57.501351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:57.501351+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"69'13","reported_seq":"172","reported_epoch":"147","state":"active+clean","last_fresh":"2021-03-24T21:55:16.166851+0000","last_change":"2021-03-24T21:54:39.194177+0000","last_active":"2021-03-24T21:55:16.166851+0000","last_peered":"2021-03-24T21:55:16.166851+0000","last_clean":"2021-03-24T21:55:16.166851+0000","last_became_active":"2021-03-24T21:54:38.127115+0000","last_became_peered":"2021-03-24T21:54:38.127115+0000","last_unstale":"2021-03-24T21:55:16.166851+0000","last_undegraded":"2021-03-24T21:55:16.166851+0000","last_fullsized":"2021-03-24T21:55:16.166851+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2021-03-24T21:53:20.163065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:20.163065+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":13,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":193339977,"num_objects":82,"num_object_clones":38,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":6,"num_read":497,"num_read_kb":383868,"num_write":509,"num_write_kb":193358,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":35460777,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430796800,"data_stored":490839361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":856,"ondisk_log_size":856,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":212,"num_read_kb":207,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1589,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":5558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1062,"num_read_kb":567,"num_write":542,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":283,"num_bytes_recovered":22082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2027520,"data_stored":72734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":549,"ondisk_log_size":549,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25,"ondisk_log_size":25,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201327,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":90008,"kb_used_data":77840,"kb_used_omap":0,"kb_used_meta":12160,"kb_avail":93655144,"statfs":{"total":95995035648,"available":95902867456,"internally_reserved":0,"allocated":79708160,"data_stored":91751322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":12451840},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:54:47 2021","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.90300000000000002,"15min":0.90000000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.4550000000000001,"5min":2.2240000000000002,"15min":2.2240000000000002},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.94599999999999995,"15min":0.96399999999999997},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.163,"5min":2.1859999999999999,"15min":2.1859999999999999},"last":0.71999999999999997}]},{"osd":1,"last update":"Wed Mar 24 21:54:47 2021","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":0.88800000000000001,"15min":0.84999999999999998},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.4079999999999999,"5min":1.7090000000000001,"15min":1.7090000000000001},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.238,"5min":2.343,"15min":2.343},"last":0.96199999999999997}]},{"osd":2,"last update":"Wed Mar 24 21:54:47 2021","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":1.0029999999999999,"15min":0.997},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.3069999999999999,"5min":4.944,"15min":4.944},"last":1.0700000000000001},{"interface":"front","average":{"1min":1.0089999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.522,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":0.76400000000000001}]},{"osd":3,"last update":"Wed Mar 24 21:54:47 2021","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":1.0269999999999999,"15min":1.0389999999999999},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.397,"5min":1.9490000000000001,"15min":1.9490000000000001},"last":0.89400000000000002},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.022,"15min":0.98699999999999999},"min":{"1min":0.71699999999999997,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.6040000000000001,"5min":1.845,"15min":1.845},"last":1.1200000000000001}]},{"osd":4,"last update":"Wed Mar 24 21:54:47 2021","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":0.95699999999999996,"15min":0.93300000000000005},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.343,"5min":2.2000000000000002,"15min":2.2000000000000002},"last":0.875},{"interface":"front","average":{"1min":1.073,"5min":1.0780000000000001,"15min":1.075},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.4359999999999999,"5min":2.1299999999999999,"15min":2.1299999999999999},"last":1.0009999999999999}]},{"osd":5,"last update":"Wed Mar 24 21:54:47 2021","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.028,"15min":0.98499999999999999},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.5669999999999999,"5min":1.581,"15min":1.581},"last":1.048},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.149,"15min":1.1399999999999999},"min":{"1min":0.70199999999999996,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.425,"5min":2.4630000000000001,"15min":2.4630000000000001},"last":1.0940000000000001}]},{"osd":6,"last update":"Wed Mar 24 21:54:47 2021","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.1559999999999999,"15min":1.123},"min":{"1min":0.73999999999999999,"5min":0.62,"15min":0.62},"max":{"1min":1.5269999999999999,"5min":2.5179999999999998,"15min":2.5179999999999998},"last":1.032},{"interface":"front","average":{"1min":1.181,"5min":1.1859999999999999,"15min":1.1759999999999999},"min":{"1min":0.63200000000000001,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.5,"5min":2.3069999999999999,"15min":2.3069999999999999},"last":1.3700000000000001}]}]},{"osd":6,"up_from":0,"seq":201863462964,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":0,"seq":55834574921,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738444,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":94192,"kb_used_data":80900,"kb_used_omap":0,"kb_used_meta":12736,"kb_avail":93650960,"statfs":{"total":95995035648,"available":95898583040,"internally_reserved":0,"allocated":82841600,"data_stored":95026302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":13041664},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Mar 24 21:54:41 2021","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.70199999999999996,"15min":0.60599999999999998},"min":{"1min":0.45400000000000001,"5min":0.314,"15min":0.314},"max":{"1min":4.8040000000000003,"5min":4.8040000000000003,"15min":4.8040000000000003},"last":0.66000000000000003},{"interface":"front","average":{"1min":1.175,"5min":0.79500000000000004,"15min":0.65000000000000002},"min":{"1min":0.46700000000000003,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":8.8989999999999991,"5min":8.8989999999999991,"15min":8.8989999999999991},"last":0.73299999999999998}]},{"osd":2,"last update":"Wed Mar 24 21:54:58 2021","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":0.75700000000000001,"15min":0.68899999999999995},"min":{"1min":0.40899999999999997,"5min":0.314,"15min":0.314},"max":{"1min":5.8810000000000002,"5min":5.8810000000000002,"15min":5.8810000000000002},"last":0.86899999999999999},{"interface":"front","average":{"1min":1.008,"5min":0.82199999999999995,"15min":0.72099999999999997},"min":{"1min":0.47499999999999998,"5min":0.376,"15min":0.376},"max":{"1min":4.766,"5min":4.766,"15min":4.766},"last":0.91000000000000003}]},{"osd":3,"last update":"Wed Mar 24 21:54:21 2021","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":0.88,"15min":0.82499999999999996},"min":{"1min":0.434,"5min":0.376,"15min":0.376},"max":{"1min":5.9790000000000001,"5min":5.9790000000000001,"15min":5.9790000000000001},"last":1.016},{"interface":"front","average":{"1min":1.1479999999999999,"5min":0.97699999999999998,"15min":0.94599999999999995},"min":{"1min":0.504,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":6.4580000000000002,"5min":6.4580000000000002,"15min":6.4580000000000002},"last":1.0529999999999999}]},{"osd":4,"last update":"Wed Mar 24 21:54:41 2021","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.121,"15min":1.107},"min":{"1min":0.73399999999999999,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":5.0330000000000004,"5min":5.0330000000000004,"15min":5.0330000000000004},"last":1.1890000000000001},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.1080000000000001,"15min":1.04},"min":{"1min":0.85999999999999999,"5min":0.625,"15min":0.625},"max":{"1min":6.1829999999999998,"5min":6.1829999999999998,"15min":6.1829999999999998},"last":1.2330000000000001}]},{"osd":5,"last update":"Wed Mar 24 21:54:58 2021","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":1.0960000000000001,"15min":1.077},"min":{"1min":0.69399999999999995,"5min":0.62,"15min":0.62},"max":{"1min":5.9390000000000001,"5min":5.9390000000000001,"15min":5.9390000000000001},"last":1.3500000000000001},{"interface":"front","average":{"1min":1.359,"5min":1.218,"15min":1.1899999999999999},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":5.0590000000000002,"5min":5.0590000000000002,"15min":5.0590000000000002},"last":1.421}]},{"osd":6,"last update":"Wed Mar 24 21:54:21 2021","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":1.1299999999999999,"15min":1.0880000000000001},"min":{"1min":0.79200000000000004,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":5.9930000000000003,"5min":5.9930000000000003,"15min":5.9930000000000003},"last":1.452},{"interface":"front","average":{"1min":1.429,"5min":1.266,"15min":1.234},"min":{"1min":0.88700000000000001,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":4.9960000000000004,"5min":4.9960000000000004,"15min":4.9960000000000004},"last":1.29}]},{"osd":7,"last update":"Wed Mar 24 21:54:46 2021","interfaces":[{"interface":"back","average":{"1min":1.4810000000000001,"5min":1.2370000000000001,"15min":1.1910000000000001},"min":{"1min":0.95199999999999996,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":6.1319999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"last":1.1060000000000001},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.1879999999999999,"15min":1.1459999999999999},"min":{"1min":1.0129999999999999,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":6.0529999999999999,"5min":6.0529999999999999,"15min":6.0529999999999999},"last":1.526}]}]},{"osd":2,"up_from":18,"seq":77309411398,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":81528,"kb_used_data":69360,"kb_used_omap":0,"kb_used_meta":12160,"kb_avail":93663624,"statfs":{"total":95995035648,"available":95911550976,"internally_reserved":0,"allocated":71024640,"data_stored":80789765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":12451840},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:54:57 2021","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.63},"min":{"1min":0.433,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.228,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.74299999999999999,"15min":0.68999999999999995},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.6140000000000001,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.76800000000000002}]},{"osd":1,"last update":"Wed Mar 24 21:54:57 2021","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.77300000000000002,"15min":0.70899999999999996},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5780000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.79300000000000004,"15min":0.73699999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.0840000000000001,"5min":1.6240000000000001,"15min":1.6240000000000001},"last":0.61599999999999999}]},{"osd":3,"last update":"Wed Mar 24 21:55:20 2021","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.85599999999999998,"15min":0.82799999999999996},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.1659999999999999,"5min":1.9890000000000001,"15min":1.9890000000000001},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.91900000000000004,"15min":0.89400000000000002},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.2210000000000001,"5min":2.0950000000000002,"15min":2.0950000000000002},"last":0.85099999999999998}]},{"osd":4,"last update":"Wed Mar 24 21:54:41 2021","interfaces":[{"interface":"back","average":{"1min":1.1279999999999999,"5min":1.0589999999999999,"15min":1.042},"min":{"1min":0.625,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.8169999999999999,"5min":3.3380000000000001,"15min":3.3380000000000001},"last":2.3149999999999999},{"interface":"front","average":{"1min":1.046,"5min":1.0700000000000001,"15min":1.046},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.446,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":2.1859999999999999}]},{"osd":5,"last update":"Wed Mar 24 21:54:57 2021","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.0509999999999999,"15min":1.0389999999999999},"min":{"1min":0.71199999999999997,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":2.0899999999999999},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.1599999999999999,"15min":1.1539999999999999},"min":{"1min":0.67200000000000004,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.7490000000000001,"5min":3.1509999999999998,"15min":3.1509999999999998},"last":2.5859999999999999}]},{"osd":6,"last update":"Wed Mar 24 21:55:20 2021","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.1990000000000001,"15min":1.2430000000000001},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.403,"5min":2.2810000000000001,"15min":2.2810000000000001},"last":1.2809999999999999},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.1459999999999999,"15min":1.1599999999999999},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.375,"5min":2.016,"15min":2.016},"last":1.0820000000000001}]},{"osd":7,"last update":"Wed Mar 24 21:54:47 2021","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.2150000000000001,"15min":1.2270000000000001},"min":{"1min":0.80900000000000005,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.4750000000000001,"5min":1.6719999999999999,"15min":1.6719999999999999},"last":4.4880000000000004},{"interface":"front","average":{"1min":1.2290000000000001,"5min":1.242,"15min":1.248},"min":{"1min":0.748,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.6759999999999999,"5min":3.5419999999999998,"15min":3.5419999999999998},"last":1.796}]}]},{"osd":3,"up_from":26,"seq":111669149761,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":100316,"kb_used_data":87572,"kb_used_omap":0,"kb_used_meta":12736,"kb_avail":93644836,"statfs":{"total":95995035648,"available":95892312064,"internally_reserved":0,"allocated":89673728,"data_stored":101235619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":13041664},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:54:24 2021","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.46200000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.91100000000000003,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.74299999999999999,"15min":0.72499999999999998},"min":{"1min":0.53600000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.86899999999999999}]},{"osd":1,"last update":"Wed Mar 24 21:54:24 2021","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.754,"15min":0.74199999999999999},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.1419999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":1.081},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.76300000000000001,"15min":0.75700000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.4510000000000001,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":1.8460000000000001}]},{"osd":2,"last update":"Wed Mar 24 21:54:24 2021","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.80500000000000005,"15min":0.78900000000000003},"min":{"1min":0.47499999999999998,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.8500000000000001,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":1.9750000000000001},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.78800000000000003,"15min":0.76900000000000002},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.399,"5min":1.399,"15min":1.399},"last":1.6120000000000001}]},{"osd":4,"last update":"Wed Mar 24 21:54:42 2021","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":0.97899999999999998,"15min":0.96199999999999997},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":2.3090000000000002,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":1.339},{"interface":"front","average":{"1min":1.744,"5min":1.1279999999999999,"15min":0.998},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":14.291,"5min":14.291,"15min":14.291},"last":1.923}]},{"osd":5,"last update":"Wed Mar 24 21:54:58 2021","interfaces":[{"interface":"back","average":{"1min":1.6000000000000001,"5min":1.175,"15min":1.0980000000000001},"min":{"1min":0.67800000000000005,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":12.723000000000001,"5min":12.723000000000001,"15min":12.723000000000001},"last":2.1030000000000002},{"interface":"front","average":{"1min":1.194,"5min":1.0780000000000001,"15min":1.0900000000000001},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":2.375,"5min":2.375,"15min":2.375},"last":1.22}]},{"osd":6,"last update":"Wed Mar 24 21:54:24 2021","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.111,"15min":1.123},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.782,"5min":1.782,"15min":1.782},"last":1.8759999999999999},{"interface":"front","average":{"1min":1.1439999999999999,"5min":1.1519999999999999,"15min":1.153},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.631,"5min":1.7070000000000001,"15min":1.7070000000000001},"last":2.165}]},{"osd":7,"last update":"Wed Mar 24 21:54:48 2021","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.129,"15min":1.0940000000000001},"min":{"1min":0.69899999999999995,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":2.3479999999999999,"5min":2.3479999999999999,"15min":2.3479999999999999},"last":1.29},{"interface":"front","average":{"1min":1.8999999999999999,"5min":1.3160000000000001,"15min":1.212},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":14.727,"5min":14.727,"15min":14.727},"last":1.6870000000000001}]}]},{"osd":4,"up_from":32,"seq":137438953531,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":81904,"kb_used_data":69316,"kb_used_omap":0,"kb_used_meta":12032,"kb_avail":93663248,"statfs":{"total":95995035648,"available":95911165952,"internally_reserved":0,"allocated":70979584,"data_stored":77785862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":12320768},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:54:38 2021","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.85699999999999998,"15min":0.84499999999999997},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.2030000000000001,"5min":1.268,"15min":1.268},"last":0.84799999999999998},{"interface":"front","average":{"1min":1.2509999999999999,"5min":0.97299999999999998,"15min":0.88900000000000001},"min":{"1min":0.68400000000000005,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":6.3929999999999998,"5min":6.3929999999999998,"15min":6.3929999999999998},"last":1.014}]},{"osd":1,"last update":"Wed Mar 24 21:54:38 2021","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.91600000000000004,"15min":0.91700000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.3120000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.89900000000000002,"15min":0.85699999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.873,"5min":1.873,"15min":1.873},"last":0.97799999999999998}]},{"osd":2,"last update":"Wed Mar 24 21:54:38 2021","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":0.94399999999999995,"15min":0.91100000000000003},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":4.4379999999999997,"5min":4.4379999999999997,"15min":4.4379999999999997},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.93899999999999995,"15min":0.91500000000000004},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.28,"5min":1.335,"15min":1.335},"last":0.79600000000000004}]},{"osd":3,"last update":"Wed Mar 24 21:54:38 2021","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.93799999999999994,"15min":0.93700000000000006},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.3240000000000001,"5min":1.601,"15min":1.601},"last":1.069},{"interface":"front","average":{"1min":0.96099999999999997,"5min":0.97499999999999998,"15min":0.95399999999999996},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.272,"5min":1.474,"15min":1.474},"last":1.115}]},{"osd":5,"last update":"Wed Mar 24 21:55:01 2021","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.93500000000000005,"15min":0.94499999999999995},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.2150000000000001,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":0.72899999999999998},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.0569999999999999,"15min":1.0109999999999999},"min":{"1min":0.66800000000000004,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.3200000000000001,"5min":2.0760000000000001,"15min":2.0760000000000001},"last":0.89000000000000001}]},{"osd":6,"last update":"Wed Mar 24 21:54:21 2021","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":1.0469999999999999,"15min":1.0589999999999999},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.4219999999999999,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":1.1639999999999999},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.056,"15min":1.0620000000000001},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":1.0469999999999999}]},{"osd":7,"last update":"Wed Mar 24 21:54:46 2021","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":1.155,"15min":1.1830000000000001},"min":{"1min":0.64200000000000002,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.552,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":1.196},{"interface":"front","average":{"1min":1.1479999999999999,"5min":1.179,"15min":1.1970000000000001},"min":{"1min":0.73399999999999999,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.4199999999999999,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":1.145}]}]},{"osd":5,"up_from":39,"seq":167503724600,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":46512,"kb_used_data":34820,"kb_used_omap":0,"kb_used_meta":11456,"kb_avail":93698640,"statfs":{"total":95995035648,"available":95947407360,"internally_reserved":0,"allocated":35655680,"data_stored":39611769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":11730944},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:55:03 2021","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.872,"15min":0.88300000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.3280000000000001,"5min":2.29,"15min":2.29},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.92000000000000004,"15min":0.89600000000000002},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.272,"5min":3.1779999999999999,"15min":3.1779999999999999},"last":0.754}]},{"osd":1,"last update":"Wed Mar 24 21:55:03 2021","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.92400000000000004,"15min":0.92200000000000004},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.3049999999999999,"5min":2.8690000000000002,"15min":2.8690000000000002},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.92600000000000005,"15min":0.93000000000000005},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.395,"5min":2.0030000000000001,"15min":2.0030000000000001},"last":0.69699999999999995}]},{"osd":2,"last update":"Wed Mar 24 21:55:03 2021","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":0.96899999999999997,"15min":0.92500000000000004},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":6.5199999999999996,"5min":6.5199999999999996,"15min":6.5199999999999996},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.94599999999999995,"15min":0.90700000000000003},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.2030000000000001,"5min":5.8479999999999999,"15min":5.8479999999999999},"last":0.94499999999999995}]},{"osd":3,"last update":"Wed Mar 24 21:55:03 2021","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.95999999999999996,"15min":0.94199999999999995},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.3500000000000001,"5min":3.1259999999999999,"15min":3.1259999999999999},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.95699999999999996,"5min":0.98199999999999998,"15min":0.95399999999999996},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.3089999999999999,"5min":2.6850000000000001,"15min":2.6850000000000001},"last":1.0349999999999999}]},{"osd":4,"last update":"Wed Mar 24 21:55:03 2021","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":0.91400000000000003,"15min":0.85499999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.4339999999999999,"5min":2.2320000000000002,"15min":2.2320000000000002},"last":0.97699999999999998},{"interface":"front","average":{"1min":1.0549999999999999,"5min":0.98999999999999999,"15min":0.92200000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.54,"5min":1.6970000000000001,"15min":1.6970000000000001},"last":1.1850000000000001}]},{"osd":6,"last update":"Wed Mar 24 21:54:20 2021","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":0.93999999999999995,"15min":0.88600000000000001},"min":{"1min":0.875,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.3100000000000001,"5min":2.085,"15min":2.085},"last":1.075},{"interface":"front","average":{"1min":1.127,"5min":1.028,"15min":0.97799999999999998},"min":{"1min":0.746,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.6060000000000001,"5min":2.3079999999999998,"15min":2.3079999999999998},"last":1.1000000000000001}]},{"osd":7,"last update":"Wed Mar 24 21:54:45 2021","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.083,"15min":1.0569999999999999},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.6140000000000001,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":1.24},{"interface":"front","average":{"1min":1.169,"5min":1.0580000000000001,"15min":1.002},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.4890000000000001,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":1.1619999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":14134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":7188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":15432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":11740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":11460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":299008,"data_stored":10618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":81756160,"data_stored":94639128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7032832,"data_stored":6977532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":70111232,"data_stored":80389254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":88555520,"data_stored":100841464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":69971968,"data_stored":77402344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":34656256,"data_stored":39227750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":78712832,"data_stored":91361889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-03-24T21:55:24.337 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd pool set iscsi pgp_num 3 2021-03-24T21:55:24.338 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:24.382 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:24.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:24.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:24.450 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:55:24.477 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:24 smithi125 conmon[19936]: cluster 2021-03-24T21:55: 2021-03-24T21:55:24.477 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:24 smithi125 conmon[19936]: 22.725831+0000 mgr.y (mgr.14144) 335 : cluster [DBG] pgmap v392: 148 pgs: 148 active+clean; 184 MiB data, 483 MiB used, 536 GiB / 536 GiB avail; 10 MiB/s rd, 1.8 MiB/s wr, 16 op/s 2021-03-24T21:55:24.477 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:24 smithi125 conmon[19936]: cluster 2021-03-24T21:55:24.478 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:24 smithi125 conmon[19936]: 2021-03-24T21:55:23.222353 2021-03-24T21:55:24.478 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:24 smithi125 conmon[19936]: +0000 mon.a (mon.0) 723 : cluster [DBG] osdmap e154: 8 total, 8 up, 6 in 2021-03-24T21:55:24.478 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:24 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:55:24.478 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:24 smithi125 conmon[19936]: 24T21:55:23.459240+0000 2021-03-24T21:55:24.479 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:24 smithi125 conmon[19936]: osd.5 (osd.5) 41 2021-03-24T21:55:24.479 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:24 smithi125 conmon[19936]: : cluster [DBG] 4.7 scrub ok 2021-03-24T21:55:24.479 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:24 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:55:24.480 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:24 smithi125 conmon[24927]: 24T21:55:22.725831+0000 mgr.y (mgr.14144) 335 : 2021-03-24T21:55:24.480 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:24 smithi125 conmon[24927]: cluster [DBG] pgmap v392: 148 pgs: 148 active+clean; 184 MiB data, 483 MiB used, 536 GiB / 536 GiB avail; 10 MiB/s rd, 1.8 MiB/s wr, 16 op/s 2021-03-24T21:55:24.480 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:24 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:55:24.480 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:24 smithi125 conmon[24927]: 03-24T 2021-03-24T21:55:24.481 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:24 smithi125 conmon[24927]: 21:55: 2021-03-24T21:55:24.481 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:24 smithi125 conmon[24927]: 23.222353+0000 2021-03-24T21:55:24.481 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:24 smithi125 conmon[24927]: mon.a ( 2021-03-24T21:55:24.481 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:24 smithi125 conmon[24927]: mon.0) 2021-03-24T21:55:24.482 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:24 smithi125 conmon[24927]: 723 : cluster 2021-03-24T21:55:24.482 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:24 smithi125 conmon[24927]: [DBG] osdmap e154: 8 total, 8 up, 6 in 2021-03-24T21:55:24.482 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:24 smithi125 conmon[24927]: 2021-03-24T21:55:24.482 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:24 smithi125 conmon[24927]: cluster 2021-03-24T21:55:24.483 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:24 smithi125 conmon[24927]: 2021-03- 2021-03-24T21:55:24.483 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:24 smithi125 conmon[24927]: 24T21:55: 2021-03-24T21:55:24.483 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:24 smithi125 conmon[24927]: 23.459240+0000 2021-03-24T21:55:24.483 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:24 smithi125 conmon[24927]: osd.5 (osd.5 2021-03-24T21:55:24.483 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:24 smithi125 conmon[24927]: ) 41 : cluster [DBG] 2021-03-24T21:55:24.484 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:24 smithi125 conmon[24927]: 4.7 scrub ok 2021-03-24T21:55:24.484 INFO:journalctl@ceph.osd.0.smithi125.stdout:Mar 24 21:55:24 smithi125 conmon[28026]: debug 2021-03-24T21:55:24.246+0000 7f61bd489700 -1 osd.0 pg_epoch: 155 pg[7.9( v 155'109 (0'0,155'109] local-lis/les=108/109 n=5 ec=99/99 lis/c=108/108 les/c/f=109/109/0 sis=108) [0,4,7] r=0 lpr=108 crt=155'109 lcod 155'107 mlcod 155'107 active+clean+snaptrim trimq=[15~1] ps=[19~1,1c~3,20~1,24~1,28~1]] removing snap head 2021-03-24T21:55:24.484 INFO:journalctl@ceph.osd.0.smithi125.stdout:Mar 24 21:55:24 smithi125 conmon[28026]: debug 2021-03-24T21:55:24.485 INFO:journalctl@ceph.osd.0.smithi125.stdout:Mar 24 21:55:24 smithi125 conmon[28026]: 2021-03-24T21:55:24.250+0000 7f61ba483700 -1 osd.0 pg_epoch: 155 pg[7.7( v 155'101 (0'0,155'101] local-lis/les=108/109 n=8 ec=99/99 lis/c=108/108 les/c/f=109/109/0 sis=108) [0,3,7] r=0 lpr=108 luod=151'99 crt=155'101 lcod 151'97 mlcod 151'97 active+clean+snaptrim trimq=[15~1] ps=[19~1,1c~3,20~1,24~1,28~1]] removing snap head 2021-03-24T21:55:24.485 INFO:journalctl@ceph.osd.0.smithi125.stdout:Mar 24 21:55:24 smithi125 conmon[28026]: 2021-03-24T21:55:24.485 INFO:journalctl@ceph.osd.3.smithi125.stdout:Mar 24 21:55:24 smithi125 conmon[37997]: debug 2021-03-24T21:55:24.486 INFO:journalctl@ceph.osd.3.smithi125.stdout:Mar 24 21:55:24 smithi125 conmon[37997]: 2021-03-24T21:55:24.238+0000 7f1717bdc700 -1 osd.3 pg_epoch: 155 pg[7.a( v 153'47 (0'0,153'47] local-lis/les=99/100 n=5 ec=99/99 lis/c=99/99 les/c/f=100/100/0 sis=99) [3,2,5] r=0 lpr=99 crt=153'47 lcod 153'45 mlcod 153'45 active+clean+snaptrim trimq=[15~1] ps=[19~1,1c~3,20~1,24~1,28~1]] removing snap head 2021-03-24T21:55:24.486 INFO:journalctl@ceph.osd.3.smithi125.stdout:Mar 24 21:55:24 smithi125 conmon[37997]: 2021-03-24T21:55:24.488 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:24.539 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:24.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:24.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:55:24.617 INFO:journalctl@ceph.osd.4.smithi142.stdout:Mar 24 21:55:24 smithi142 conmon[24076]: debug 2021-03-24T21:55:24.236+0000 7fc561e38700 -1 osd.4 pg_epoch: 155 pg[7.1( v 155'107 (0'0,155'107] local-lis/les=108/109 n=9 ec=99/99 lis/c=108/108 les/c/f=109/111/0 sis=108) [4,3,0] r=0 lpr=108 luod=153'105 crt=155'107 lcod 151'103 mlcod 151'103 active+clean+snaptrim trimq=[15~1] ps=[19~1,1c~3,20~1,24~1,28~1]] removing snap head 2021-03-24T21:55:24.617 INFO:journalctl@ceph.osd.4.smithi142.stdout:Mar 24 21:55:24 smithi142 conmon[24076]: debug 2021-03-24T21:55:24.248+0000 7fc55de30700 -1 osd.4 pg_epoch: 155 pg[7.1( v 155'109 (0'0,155'109] local-lis/les=108/109 n=7 ec=99/99 lis/c=108/108 les/c/f=109/111/0 sis=108) [4,3,0] r=0 lpr=108 crt=155'109 lcod 155'107 mlcod 155'107 active+clean+snaptrim trimq=[15~1] ps=[19~1,1c~3,20~1,24~1,28~1]] removing snap head 2021-03-24T21:55:24.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:24 smithi142 conmon[20125]: cluster 2021-03-24T21:55:22 2021-03-24T21:55:24.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:24 smithi142 conmon[20125]: .725831+0000 mgr.y (mgr.14144) 335 : cluster [DBG] pgmap v392: 148 pgs: 148 active+clean; 184 MiB data, 483 MiB used, 536 GiB / 536 GiB avail; 10 MiB/s rd, 1.8 MiB/s wr, 16 op/s 2021-03-24T21:55:24.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:24 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:55:24.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:24 smithi142 conmon[20125]: 24T21:55:23.222353+0000 mon.a (mon.0) 723 : cluster [DBG] osdmap e154: 8 total, 8 up, 6 in 2021-03-24T21:55:24.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:24 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:55:24.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:24 smithi142 conmon[20125]: -03-24T21:55:23.459240+0000 osd.5 2021-03-24T21:55:24.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:24 smithi142 conmon[20125]: (osd.5) 41 : cluster [DBG] 4.7 scrub ok 2021-03-24T21:55:24.647 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:24.707 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:24.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:24.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:55:24.765 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-24T21:55:24.766 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T21:55:24.788 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:55:24.814 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:24.863 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:24.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:24.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:55:24.969 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:25.024 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:25.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:25.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:55:25.104 INFO:tasks.rados.rados.0.smithi125.stdout:283: done (2 left) 2021-03-24T21:55:25.105 INFO:tasks.rados.rados.0.smithi125.stdout:286: done (1 left) 2021-03-24T21:55:25.105 INFO:tasks.rados.rados.0.smithi125.stdout:287: done (0 left) 2021-03-24T21:55:25.106 INFO:tasks.rados.rados.0.smithi125.stdout:288: read oid 39 snap 24 2021-03-24T21:55:25.106 INFO:tasks.rados.rados.0.smithi125.stdout:288: expect deleted 2021-03-24T21:55:25.107 INFO:tasks.rados.rados.0.smithi125.stdout:289: read oid 38 snap -1 2021-03-24T21:55:25.107 INFO:tasks.rados.rados.0.smithi125.stdout:289: expect (ObjNum 121 snap 23 seq_num 121) 2021-03-24T21:55:25.107 INFO:tasks.rados.rados.0.smithi125.stdout:290: copy_from oid 3 from oid 30 current snap is 26 2021-03-24T21:55:25.108 INFO:tasks.rados.rados.0.smithi125.stdout:291: write (excl) oid 35 current snap is 26 2021-03-24T21:55:25.108 INFO:tasks.rados.rados.0.smithi125.stdout:291: seq_num 138 ranges {719366=478716,1808868=510498} 2021-03-24T21:55:25.112 INFO:tasks.rados.rados.0.smithi125.stdout:291: writing smithi12556470-35 from 719366 to 1198082 tid 1 2021-03-24T21:55:25.120 INFO:tasks.rados.rados.0.smithi125.stdout:291: writing smithi12556470-35 from 1808868 to 2319366 tid 2 2021-03-24T21:55:25.120 INFO:tasks.rados.rados.0.smithi125.stdout:292: read oid 4 snap -1 2021-03-24T21:55:25.120 INFO:tasks.rados.rados.0.smithi125.stdout:292: expect (ObjNum 3 snap 0 seq_num 3) 2021-03-24T21:55:25.120 INFO:tasks.rados.rados.0.smithi125.stdout:293: rollback oid 42 current snap is 26 2021-03-24T21:55:25.120 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 42 to 23 2021-03-24T21:55:25.121 INFO:tasks.rados.rados.0.smithi125.stdout:290: finishing copy_from to smithi12556470-3 2021-03-24T21:55:25.121 INFO:tasks.rados.rados.0.smithi125.stdout:290: got expected ENOENT (src dne) 2021-03-24T21:55:25.121 INFO:tasks.rados.rados.0.smithi125.stdout:290: finishing copy_from racing read to smithi12556470-3 2021-03-24T21:55:25.121 INFO:tasks.rados.rados.0.smithi125.stdout:289: expect (ObjNum 121 snap 23 seq_num 121) 2021-03-24T21:55:25.130 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:25.179 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:25.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:25.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:55:25.190 INFO:tasks.rados.rados.0.smithi125.stdout:288: done (5 left) 2021-03-24T21:55:25.190 INFO:tasks.rados.rados.0.smithi125.stdout:289: done (4 left) 2021-03-24T21:55:25.190 INFO:tasks.rados.rados.0.smithi125.stdout:290: done (3 left) 2021-03-24T21:55:25.190 INFO:tasks.rados.rados.0.smithi125.stdout:294: copy_from oid 27 from oid 11 current snap is 26 2021-03-24T21:55:25.191 INFO:tasks.rados.rados.0.smithi125.stdout:291: finishing write tid 1 to smithi12556470-35 2021-03-24T21:55:25.191 INFO:tasks.rados.rados.0.smithi125.stdout:291: finishing write tid 2 to smithi12556470-35 2021-03-24T21:55:25.191 INFO:tasks.rados.rados.0.smithi125.stdout:293: finishing rollback tid 0 to smithi12556470-42 2021-03-24T21:55:25.191 INFO:tasks.rados.rados.0.smithi125.stdout:291: finishing write tid 3 to smithi12556470-35 2021-03-24T21:55:25.192 INFO:tasks.rados.rados.0.smithi125.stdout:293: finishing rollback tid 1 to smithi12556470-42 2021-03-24T21:55:25.192 INFO:tasks.rados.rados.0.smithi125.stdout:293: finishing rollback tid 2 to smithi12556470-42 2021-03-24T21:55:25.192 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 42 v 46 (ObjNum 91 snap 14 seq_num 91) dirty exists 2021-03-24T21:55:25.192 INFO:tasks.rados.rados.0.smithi125.stdout:291: finishing write tid 4 to smithi12556470-35 2021-03-24T21:55:25.192 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 35 v 14 (ObjNum 138 snap 26 seq_num 138) dirty exists 2021-03-24T21:55:25.193 INFO:tasks.rados.rados.0.smithi125.stdout:291: left oid 35 (ObjNum 138 snap 26 seq_num 138) 2021-03-24T21:55:25.193 INFO:tasks.rados.rados.0.smithi125.stdout:292: expect (ObjNum 3 snap 0 seq_num 3) 2021-03-24T21:55:25.258 INFO:tasks.rados.rados.0.smithi125.stdout:291: done (3 left) 2021-03-24T21:55:25.259 INFO:tasks.rados.rados.0.smithi125.stdout:292: done (2 left) 2021-03-24T21:55:25.259 INFO:tasks.rados.rados.0.smithi125.stdout:293: done (1 left) 2021-03-24T21:55:25.259 INFO:tasks.rados.rados.0.smithi125.stdout:295: delete oid 35 current snap is 26 2021-03-24T21:55:25.259 INFO:tasks.rados.rados.0.smithi125.stdout:294: finishing copy_from to smithi12556470-27 2021-03-24T21:55:25.260 INFO:tasks.rados.rados.0.smithi125.stdout:294: got expected ENOENT (src dne) 2021-03-24T21:55:25.260 INFO:tasks.rados.rados.0.smithi125.stdout:294: finishing copy_from racing read to smithi12556470-27 2021-03-24T21:55:25.261 INFO:tasks.rados.rados.0.smithi125.stdout:294: done (1 left) 2021-03-24T21:55:25.261 INFO:tasks.rados.rados.0.smithi125.stdout:295: done (0 left) 2021-03-24T21:55:25.262 INFO:tasks.rados.rados.0.smithi125.stdout:296: snap_create 2021-03-24T21:55:25.285 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:25.307 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:25 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:55:25.308 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:25 smithi125 conmon[24927]: 21:55:23.872733+0000 osd.7 (osd.7) 35 : cluster 2021-03-24T21:55:25.308 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:25 smithi125 conmon[24927]: [DBG] 5.10 scrub ok 2021-03-24T21:55:25.308 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:25 smithi125 conmon[24927]: cluster 2021-03-24T21:55:25.309 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:25 smithi125 conmon[24927]: 2021-03-24T21:55 2021-03-24T21:55:25.309 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:25 smithi125 conmon[24927]: :24.078773+0000 osd.2 ( 2021-03-24T21:55:25.309 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:25 smithi125 conmon[24927]: osd.2) 28 : cluster [DBG] 2021-03-24T21:55:25.309 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:25 smithi125 conmon[24927]: 6.f scrub ok 2021-03-24T21:55:25.309 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:25 smithi125 conmon[24927]: cluster 2021-03-24T21:55:25.310 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:25 smithi125 conmon[24927]: 2021-03 2021-03-24T21:55:25.310 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:25 smithi125 conmon[24927]: -24T21:55:24.235601+0000 2021-03-24T21:55:25.310 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:25 smithi125 conmon[24927]: mon.a (mon.0) 724 : cluster [DBG] 2021-03-24T21:55:25.310 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:25 smithi125 conmon[24927]: osdmap e155: 8 total, 8 up, 6 in 2021-03-24T21:55:25.311 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:25 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:55:25.311 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:25 smithi125 conmon[24927]: -03-24T21:55:24. 2021-03-24T21:55:25.311 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:25 smithi125 conmon[24927]: 548814+0000 osd.3 (osd.3) 49 : 2021-03-24T21:55:25.312 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:25 smithi125 conmon[24927]: cluster [DBG] 6.b scrub ok 2021-03-24T21:55:25.312 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:25 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:55:25.312 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:25 smithi125 conmon[24927]: -03-24T21:55:25 2021-03-24T21:55:25.312 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:25 smithi125 conmon[24927]: .099765+0000 mon.a (mon.0) 725 2021-03-24T21:55:25.312 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:25 smithi125 conmon[24927]: : cluster [DBG] osdmap e156: 8 total, 8 up, 6 in 2021-03-24T21:55:25.313 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:25 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T21:55:25.313 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:25 smithi125 conmon[19936]: T21:55:23.872733+0000 osd.7 (osd.7) 35 : cluster [DBG] 5.10 scrub ok 2021-03-24T21:55:25.313 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:25 smithi125 conmon[19936]: cluster 2021-03-24T21:55:25.314 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:25 smithi125 conmon[19936]: 2021- 2021-03-24T21:55:25.314 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:25 smithi125 conmon[19936]: 03-24T21: 2021-03-24T21:55:25.314 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:25 smithi125 conmon[19936]: 55:24. 2021-03-24T21:55:25.314 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:25 smithi125 conmon[19936]: 078773+0000 2021-03-24T21:55:25.314 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:25 smithi125 conmon[19936]: osd.2 (osd. 2021-03-24T21:55:25.315 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:25 smithi125 conmon[19936]: 2) 28 : cluster 2021-03-24T21:55:25.315 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:25 smithi125 conmon[19936]: [DBG] 6.f scrub ok 2021-03-24T21:55:25.315 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:25 smithi125 conmon[19936]: cluster 2021-03-24T21:55:25.315 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:25 smithi125 conmon[19936]: 2021-03- 2021-03-24T21:55:25.315 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:25 smithi125 conmon[19936]: 24T21:55:24 2021-03-24T21:55:25.316 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:25 smithi125 conmon[19936]: .235601+0000 mon.a ( 2021-03-24T21:55:25.316 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:25 smithi125 conmon[19936]: mon.0) 724 : cluster 2021-03-24T21:55:25.316 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:25 smithi125 conmon[19936]: [DBG] osdmap e155: 8 total, 8 up, 6 in 2021-03-24T21:55:25.316 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:25 smithi125 conmon[19936]: cluster 2021-03-24T21:55:25.317 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:25 smithi125 conmon[19936]: 2021-03- 2021-03-24T21:55:25.317 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:25 smithi125 conmon[19936]: 24T21:55:24. 2021-03-24T21:55:25.317 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:25 smithi125 conmon[19936]: 548814+0000 osd.3 (osd 2021-03-24T21:55:25.317 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:25 smithi125 conmon[19936]: .3) 49 : cluster 2021-03-24T21:55:25.317 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:25 smithi125 conmon[19936]: [DBG] 6.b scrub ok 2021-03-24T21:55:25.318 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:25 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:55:25.318 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:25 smithi125 conmon[19936]: :55:25.099765+0000 mon.a (mon.0) 725 : cluster [DBG] osdmap e156: 8 total, 8 up, 6 in 2021-03-24T21:55:25.340 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:25.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:25.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:55:25.445 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:25.506 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:25.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:25.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:55:25.611 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:25.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:25 smithi142 conmon[20125]: cluster 2021-03-24T21:55:23.872733+0000 osd.7 (osd.7) 35 : cluster 2021-03-24T21:55:25.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:25 smithi142 conmon[20125]: [DBG] 5.10 scrub ok 2021-03-24T21:55:25.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:25 smithi142 conmon[20125]: cluster 2021-03-24T21:55:24.078773+0000 2021-03-24T21:55:25.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:25 smithi142 conmon[20125]: osd.2 (osd.2) 28 : cluster [DBG] 6.f scrub ok 2021-03-24T21:55:25.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:25 smithi142 conmon[20125]: 2021-03-24T21:55:25.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:25 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:55:25.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:25 smithi142 conmon[20125]: 24T21:55:24.235601+0000 mon.a (mon.0 2021-03-24T21:55:25.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:25 smithi142 conmon[20125]: ) 724 : cluster [DBG] osdmap e155: 8 total, 8 up, 6 in 2021-03-24T21:55:25.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:25 smithi142 conmon[20125]: cluster 2021-03-24T21:55 2021-03-24T21:55:25.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:25 smithi142 conmon[20125]: :24.548814+0000 osd.3 (osd.3) 49 : cluster [DBG] 6.b scrub ok 2021-03-24T21:55:25.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:25 smithi142 conmon[20125]: cluster 2021-03-24T21:55:25.099765+0000 mon.a 2021-03-24T21:55:25.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:25 smithi142 conmon[20125]: (mon.0) 725 : cluster [DBG] osdmap e156: 8 total, 8 up, 6 in 2021-03-24T21:55:25.668 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:25.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:25.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:55:25.773 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:25.831 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:25.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:25.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:55:25.938 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:25.993 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:25.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:25.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:55:26.098 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:26.155 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:26.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:26.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:55:26.257 INFO:tasks.rados.rados.0.smithi125.stdout:296: done (0 left) 2021-03-24T21:55:26.257 INFO:tasks.rados.rados.0.smithi125.stdout:297: snap_remove snap 26 2021-03-24T21:55:26.260 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:26.312 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:26.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:26.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:55:26.417 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:26.432 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:26 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:55:26.432 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:26 smithi125 conmon[24927]: 55:24.726627+0000 mgr.y (mgr.14144) 336 : cluster [DBG] pgmap v395: 148 pgs: 148 active+clean; 184 MiB data, 491 MiB used, 536 GiB / 536 GiB avail; 27 MiB/s rd, 6.2 MiB/s wr, 43 op/s 2021-03-24T21:55:26.433 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:26 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:55:26.433 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:26 smithi125 conmon[24927]: T21:55:25.106434+0000 osd.4 2021-03-24T21:55:26.433 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:26 smithi125 conmon[24927]: (osd.4) 43 : cluster [DBG] 5.1 scrub ok 2021-03-24T21:55:26.433 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:26 smithi125 conmon[24927]: 2021-03-24T21:55:26.434 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:26 smithi125 conmon[24927]: cluster 2021-03-24T21:55:26.434 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:26 smithi125 conmon[24927]: 2021-03-24T21:55: 2021-03-24T21:55:26.434 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:26 smithi125 conmon[24927]: 25.582377+0000 osd.3 (osd.3) 50 2021-03-24T21:55:26.434 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:26 smithi125 conmon[24927]: : cluster [DBG] 4.14 scrub ok 2021-03-24T21:55:26.434 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:26 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:55:26.435 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:26 smithi125 conmon[24927]: -24T21:55:26.017416 2021-03-24T21:55:26.435 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:26 smithi125 conmon[24927]: +0000 osd.2 (osd.2) 29 : cluster 2021-03-24T21:55:26.435 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:26 smithi125 conmon[24927]: [DBG] 5.8 scrub ok 2021-03-24T21:55:26.436 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:26 smithi125 conmon[19936]: cluster 2021-03-24T21:55:24.726627+0000 2021-03-24T21:55:26.436 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:26 smithi125 conmon[19936]: mgr.y (mgr.14144) 336 : cluster [DBG] pgmap v395: 148 pgs: 148 active+clean; 184 MiB data, 491 MiB used, 536 GiB / 536 GiB avail; 27 MiB/s rd, 6.2 MiB/s wr, 43 op/s 2021-03-24T21:55:26.436 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:26 smithi125 conmon[19936]: cluster 2021-03-24T21:55:25.106434+0000 2021-03-24T21:55:26.436 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:26 smithi125 conmon[19936]: osd.4 (osd.4) 43 : cluster [DBG] 5.1 scrub ok 2021-03-24T21:55:26.436 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:26 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:55:26.437 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:26 smithi125 conmon[19936]: -24T21:55:25.582377+0000 osd.3 ( 2021-03-24T21:55:26.437 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:26 smithi125 conmon[19936]: osd.3) 50 : cluster [DBG] 4.14 scrub ok 2021-03-24T21:55:26.437 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:26 smithi125 conmon[19936]: cluster 2021-03-24T21:55:26.437 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:26 smithi125 conmon[19936]: 2021-03-24T21:55:26.017416+0000 2021-03-24T21:55:26.438 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:26 smithi125 conmon[19936]: osd.2 (osd.2) 29 : cluster [DBG] 5.8 scrub ok 2021-03-24T21:55:26.438 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:26 smithi125 conmon[19936]: 2021-03-24T21:55:26.479 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:26.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:26.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:55:26.488 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-24T21:55:26.584 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:26.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:26 smithi142 conmon[20125]: cluster 2021-03-24T21:55:24.726627+0000 mgr.y (mgr.14144) 336 : cluster 2021-03-24T21:55:26.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:26 smithi142 conmon[20125]: [DBG] pgmap v395: 148 pgs: 148 active+clean; 184 MiB data, 491 MiB used, 536 GiB / 536 GiB avail; 27 MiB/s rd, 6.2 MiB/s wr, 43 op/s 2021-03-24T21:55:26.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:26 smithi142 conmon[20125]: cluster 2021-03-24T21:55:25.106434+0000 osd.4 (osd.4) 43 2021-03-24T21:55:26.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:26 smithi142 conmon[20125]: : cluster [DBG] 5.1 scrub ok 2021-03-24T21:55:26.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:26 smithi142 conmon[20125]: cluster 2021-03-24T21:55:26.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:26 smithi142 conmon[20125]: 2021-03-24T21:55:25.582377+0000 osd.3 ( 2021-03-24T21:55:26.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:26 smithi142 conmon[20125]: osd.3) 50 : cluster [DBG] 4.14 scrub ok 2021-03-24T21:55:26.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:26 smithi142 conmon[20125]: 2021-03-24T21:55:26.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:26 smithi142 conmon[20125]: cluster 2021-03-24T21:55:26.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:26 smithi142 conmon[20125]: 2021-03-24T21:55:26.017416+0000 2021-03-24T21:55:26.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:26 smithi142 conmon[20125]: osd.2 (osd.2) 29 : cluster [DBG] 5.8 scrub ok 2021-03-24T21:55:26.635 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:26.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:26.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:26.740 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:26.753 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.3 dump_historic_ops 2021-03-24T21:55:26.791 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:26.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:26.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:26.897 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:26.944 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:26.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:26.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:55:27.049 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:27.102 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:27.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:27.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:27.208 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:27.266 INFO:tasks.rados.rados.0.smithi125.stdout:297: done (0 left) 2021-03-24T21:55:27.266 INFO:tasks.rados.rados.0.smithi125.stdout:298: delete oid 20 current snap is 27 2021-03-24T21:55:27.269 INFO:tasks.rados.rados.0.smithi125.stdout:298: done (0 left) 2021-03-24T21:55:27.269 INFO:tasks.rados.rados.0.smithi125.stdout:299: snap_remove snap 24 2021-03-24T21:55:27.270 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:27.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:27.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:55:27.374 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:27.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:27 smithi125 conmon[19936]: cluster 2021-03-24T21:55:26.087575+0000 osd.4 (osd.4 2021-03-24T21:55:27.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:27 smithi125 conmon[19936]: ) 44 : cluster [DBG] 5.11 scrub ok 2021-03-24T21:55:27.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:27 smithi125 conmon[19936]: cluster 2021-03-24T21:55:26.256383+0000 mon.a (mon.0) 726 : cluster [DBG] osdmap e157: 8 total, 8 up, 6 in 2021-03-24T21:55:27.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:27 smithi125 conmon[19936]: cluster 2021-03-24T21:55:26.502887+0000 osd.5 (osd 2021-03-24T21:55:27.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:27 smithi125 conmon[19936]: .5) 42 : cluster [DBG] 3.16 scrub ok 2021-03-24T21:55:27.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:27 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:55:27.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:27 smithi125 conmon[24927]: T21:55:26.087575+0000 osd.4 (osd.4) 44 : cluster [DBG] 5.11 scrub ok 2021-03-24T21:55:27.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:27 smithi125 conmon[24927]: 2021-03-24T21:55:27.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:27 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:55:27.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:27 smithi125 conmon[24927]: 03-24T21:55 2021-03-24T21:55:27.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:27 smithi125 conmon[24927]: :26.256383+0000 2021-03-24T21:55:27.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:27 smithi125 conmon[24927]: mon.a (mon.0) 726 2021-03-24T21:55:27.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:27 smithi125 conmon[24927]: : cluster [DBG] osdmap e157: 8 total, 8 up, 6 in 2021-03-24T21:55:27.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:27 smithi125 conmon[24927]: 2021-03-24T21:55:27.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:27 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:55:27.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:27 smithi125 conmon[24927]: 03-24T21:55 2021-03-24T21:55:27.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:27 smithi125 conmon[24927]: :26.502887+0000 2021-03-24T21:55:27.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:27 smithi125 conmon[24927]: osd.5 (osd.5) 2021-03-24T21:55:27.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:27 smithi125 conmon[24927]: 42 : cluster [DBG] 3.16 scrub ok 2021-03-24T21:55:27.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:27 smithi125 conmon[24927]: 2021-03-24T21:55:27.432 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:27.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:27.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:55:27.539 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:27.552 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:27 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:55:27.552 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:27 smithi142 conmon[20125]: 55:26.087575+0000 osd.4 (osd.4) 44 : cluster [DBG] 5.11 scrub ok 2021-03-24T21:55:27.552 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:27 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:55:27.553 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:27 smithi142 conmon[20125]: 21:55:26.256383+0000 mon.a (mon.0) 726 : cluster [DBG] osdmap e157: 8 total, 8 up, 6 in 2021-03-24T21:55:27.553 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:27 smithi142 conmon[20125]: 2021-03-24T21:55:27.553 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:27 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:55:27.553 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:27 smithi142 conmon[20125]: T21:55:26.502887+0000 osd.5 (osd.5) 42 : cluster [DBG] 3.16 scrub ok 2021-03-24T21:55:27.592 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:27.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:27.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:55:27.699 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:27.752 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:27.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:27.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:27.857 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:27.912 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:27.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:27.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:28.015 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:55:28.019 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:28.052 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:55:28.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:28.208 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:28.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:28.209 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T21:55:28.235 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T21:55:28.253 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T21:55:28.273 INFO:tasks.rados.rados.0.smithi125.stdout:299: done (0 left) 2021-03-24T21:55:28.274 INFO:tasks.rados.rados.0.smithi125.stdout:300: rollback oid 31 current snap is 27 2021-03-24T21:55:28.274 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 31 to 23 2021-03-24T21:55:28.274 INFO:tasks.rados.rados.0.smithi125.stdout:301: rollback oid 19 current snap is 27 2021-03-24T21:55:28.275 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 19 to 25 2021-03-24T21:55:28.275 INFO:tasks.rados.rados.0.smithi125.stdout:302: write (excl) oid 26 current snap is 27 2021-03-24T21:55:28.275 INFO:tasks.rados.rados.0.smithi125.stdout:302: seq_num 141 ranges {769787=704463,2267961=101826} 2021-03-24T21:55:28.283 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T21:55:28.285 INFO:tasks.rados.rados.0.smithi125.stdout:302: writing smithi12556470-26 from 769787 to 1474250 tid 1 2021-03-24T21:55:28.291 INFO:tasks.rados.rados.0.smithi125.stdout:302: writing smithi12556470-26 from 2267961 to 2369787 tid 2 2021-03-24T21:55:28.291 INFO:tasks.rados.rados.0.smithi125.stdout:303: read oid 4 snap -1 2021-03-24T21:55:28.291 INFO:tasks.rados.rados.0.smithi125.stdout:303: expect (ObjNum 3 snap 0 seq_num 3) 2021-03-24T21:55:28.292 INFO:tasks.rados.rados.0.smithi125.stdout:304: copy_from oid 22 from oid 30 current snap is 27 2021-03-24T21:55:28.292 INFO:tasks.rados.rados.0.smithi125.stdout:301: finishing rollback tid 1 to smithi12556470-19 2021-03-24T21:55:28.292 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 19 v 105 (ObjNum -986354224 snap 32741 seq_num 3308613072) dirty dne 2021-03-24T21:55:28.292 INFO:tasks.rados.rados.0.smithi125.stdout:300: finishing rollback tid 0 to smithi12556470-31 2021-03-24T21:55:28.292 INFO:tasks.rados.rados.0.smithi125.stdout:300: finishing rollback tid 1 to smithi12556470-31 2021-03-24T21:55:28.293 INFO:tasks.rados.rados.0.smithi125.stdout:301: done (4 left) 2021-03-24T21:55:28.293 INFO:tasks.rados.rados.0.smithi125.stdout:305: rollback oid 11 current snap is 27 2021-03-24T21:55:28.293 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 11 to 25 2021-03-24T21:55:28.293 INFO:tasks.rados.rados.0.smithi125.stdout:300: finishing rollback tid 2 to smithi12556470-31 2021-03-24T21:55:28.294 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 31 v 117 (ObjNum 73 snap 7 seq_num 73) dirty exists 2021-03-24T21:55:28.294 INFO:tasks.rados.rados.0.smithi125.stdout:302: finishing write tid 1 to smithi12556470-26 2021-03-24T21:55:28.294 INFO:tasks.rados.rados.0.smithi125.stdout:300: done (4 left) 2021-03-24T21:55:28.294 INFO:tasks.rados.rados.0.smithi125.stdout:306: delete oid 37 current snap is 27 2021-03-24T21:55:28.295 INFO:tasks.rados.rados.0.smithi125.stdout:304: finishing copy_from to smithi12556470-22 2021-03-24T21:55:28.295 INFO:tasks.rados.rados.0.smithi125.stdout:304: got expected ENOENT (src dne) 2021-03-24T21:55:28.296 INFO:tasks.rados.rados.0.smithi125.stdout:304: finishing copy_from racing read to smithi12556470-22 2021-03-24T21:55:28.296 INFO:tasks.rados.rados.0.smithi125.stdout:302: finishing write tid 2 to smithi12556470-26 2021-03-24T21:55:28.296 INFO:tasks.rados.rados.0.smithi125.stdout:304: done (4 left) 2021-03-24T21:55:28.296 INFO:tasks.rados.rados.0.smithi125.stdout:306: done (3 left) 2021-03-24T21:55:28.297 INFO:tasks.rados.rados.0.smithi125.stdout:307: write oid 1 current snap is 27 2021-03-24T21:55:28.297 INFO:tasks.rados.rados.0.smithi125.stdout:307: seq_num 143 ranges {403401=454961,1584925=506039} 2021-03-24T21:55:28.297 INFO:tasks.rados.rados.0.smithi125.stdout:307: writing smithi12556470-1 from 403401 to 858362 tid 1 2021-03-24T21:55:28.305 INFO:tasks.rados.rados.0.smithi125.stdout:307: writing smithi12556470-1 from 1584925 to 2090964 tid 2 2021-03-24T21:55:28.305 INFO:tasks.rados.rados.0.smithi125.stdout:308: snap_create 2021-03-24T21:55:28.305 INFO:tasks.rados.rados.0.smithi125.stdout:302: finishing write tid 3 to smithi12556470-26 2021-03-24T21:55:28.306 INFO:tasks.rados.rados.0.smithi125.stdout:302: finishing write tid 4 to smithi12556470-26 2021-03-24T21:55:28.306 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 26 v 76 (ObjNum 141 snap 27 seq_num 141) dirty exists 2021-03-24T21:55:28.306 INFO:tasks.rados.rados.0.smithi125.stdout:302: left oid 26 (ObjNum 141 snap 27 seq_num 141) 2021-03-24T21:55:28.306 INFO:tasks.rados.rados.0.smithi125.stdout:303: expect (ObjNum 3 snap 0 seq_num 3) 2021-03-24T21:55:28.309 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T21:55:28.309 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:28.336 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T21:55:28.356 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:28.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:28.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:55:28.374 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T21:55:28.380 INFO:tasks.rados.rados.0.smithi125.stdout:305: finishing rollback tid 1 to smithi12556470-11 2021-03-24T21:55:28.380 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 11 v 0 (ObjNum 1116400000 snap 21916 seq_num 3308612736) dirty dne 2021-03-24T21:55:28.380 INFO:tasks.rados.rados.0.smithi125.stdout:307: finishing write tid 1 to smithi12556470-1 2021-03-24T21:55:28.381 INFO:tasks.rados.rados.0.smithi125.stdout:307: finishing write tid 2 to smithi12556470-1 2021-03-24T21:55:28.381 INFO:tasks.rados.rados.0.smithi125.stdout:307: finishing write tid 3 to smithi12556470-1 2021-03-24T21:55:28.381 INFO:tasks.rados.rados.0.smithi125.stdout:307: finishing write tid 4 to smithi12556470-1 2021-03-24T21:55:28.381 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 1 v 116 (ObjNum 143 snap 27 seq_num 143) dirty exists 2021-03-24T21:55:28.382 INFO:tasks.rados.rados.0.smithi125.stdout:307: left oid 1 (ObjNum 143 snap 27 seq_num 143) 2021-03-24T21:55:28.401 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T21:55:28.403 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-24T21:55:28.403 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-24T21:55:28.403 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-24T21:55:28.403 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-24T21:55:28.404 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-24T21:55:28.404 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-24T21:55:28.404 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-24T21:55:28.404 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-24T21:55:28.466 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:28.521 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:28.524 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:28 smithi125 conmon[19936]: cluster 2021-03-24T21:55:26.727183 2021-03-24T21:55:28.525 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:28 smithi125 conmon[19936]: +0000 mgr.y (mgr.14144) 337 : cluster [DBG] pgmap v398: 148 pgs: 148 active+clean; 181 MiB data, 464 MiB used, 536 GiB / 536 GiB avail; 29 MiB/s rd, 6.0 MiB/s wr, 45 op/s 2021-03-24T21:55:28.525 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:28 smithi125 conmon[19936]: cluster 2021-03-24T21:55:27.262809+0000 mon.a (mon.0) 2021-03-24T21:55:28.525 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:28 smithi125 conmon[19936]: 727 : cluster [DBG] osdmap e158: 8 total, 8 up, 6 in 2021-03-24T21:55:28.525 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:28 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:55:28.525 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:28 smithi125 conmon[19936]: -24T21:55:27.547484+0000 osd.5 (osd.5) 2021-03-24T21:55:28.526 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:28 smithi125 conmon[19936]: 43 : cluster [DBG] 5.16 scrub ok 2021-03-24T21:55:28.526 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:28 smithi125 conmon[19936]: audit 2021-03- 2021-03-24T21:55:28.526 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:28 smithi125 conmon[19936]: 24T21:55:28.177911+0000 mon.a (mon.0) 2021-03-24T21:55:28.526 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:28 smithi125 conmon[19936]: 728 : audit [DBG] from='client.? v1:172.21.15.125:0/530248716' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:55:28.527 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:28 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:55:28.527 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:28 smithi125 conmon[24927]: :55:26.727183+0000 mgr.y (mgr.14144) 337 : cluster [DBG] pgmap v398: 148 pgs: 148 active+clean; 181 MiB data, 464 MiB used, 536 GiB / 536 GiB avail; 29 MiB/s rd, 6.0 MiB/s wr, 45 op/s 2021-03-24T21:55:28.527 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:28 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:55:28.528 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:28 smithi125 conmon[24927]: -03-24T21:55:27.262809 2021-03-24T21:55:28.528 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:28 smithi125 conmon[24927]: +0000 mon.a (mon.0) 727 : cluster 2021-03-24T21:55:28.528 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:28 smithi125 conmon[24927]: [DBG] osdmap e158: 8 total, 8 up, 6 in 2021-03-24T21:55:28.528 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:28 smithi125 conmon[24927]: cluster 2021-03-24T21:55 2021-03-24T21:55:28.528 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:28 smithi125 conmon[24927]: :27.547484+0000 osd.5 (osd.5) 43 : cluster 2021-03-24T21:55:28.529 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:28 smithi125 conmon[24927]: [DBG] 5.16 scrub ok 2021-03-24T21:55:28.529 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:28 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T21:55:28.529 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:28 smithi125 conmon[24927]: 24T21:55:28.177911+0000 mon.a (mon 2021-03-24T21:55:28.530 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:28 smithi125 conmon[24927]: .0) 728 : audit [DBG] from='client.? v1:172.21.15.125:0/530248716' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:55:28.530 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:28 smithi125 conmon[24927]: 2021-03-24T21:55:28.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:28.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:55:28.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:28 smithi142 conmon[20125]: cluster 2021-03-24T21:55: 2021-03-24T21:55:28.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:28 smithi142 conmon[20125]: 26.727183+0000 mgr.y (mgr.14144) 337 : cluster [DBG] pgmap v398: 148 pgs: 148 active+clean; 181 MiB data, 464 MiB used, 536 GiB / 536 GiB avail; 29 MiB/s rd, 6.0 MiB/s wr, 45 op/s 2021-03-24T21:55:28.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:28 smithi142 conmon[20125]: cluster 2021-03-24T21:55:28.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:28 smithi142 conmon[20125]: 2021-03-24T21:55:27.262809+0000 mon.a (mon.0 2021-03-24T21:55:28.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:28 smithi142 conmon[20125]: ) 727 : cluster [DBG] osdmap e158: 8 total, 8 up, 6 in 2021-03-24T21:55:28.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:28 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:55:28.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:28 smithi142 conmon[20125]: 03-24T21:55:27.547484+0000 2021-03-24T21:55:28.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:28 smithi142 conmon[20125]: osd.5 (osd.5) 43 : cluster [DBG] 5.16 scrub ok 2021-03-24T21:55:28.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:28 smithi142 conmon[20125]: 2021-03-24T21:55:28.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:28 smithi142 conmon[20125]: audit 2021-03-24T21:55:28.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:28 smithi142 conmon[20125]: 2021-03-24 2021-03-24T21:55:28.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:28 smithi142 conmon[20125]: T21:55:28.177911+0000 mon.a ( 2021-03-24T21:55:28.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:28 smithi142 conmon[20125]: mon.0) 728 : audit [DBG] from='client.? v1:172.21.15.125:0/530248716' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:55:28.633 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:28.693 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:28.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:28.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:55:28.800 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:28.861 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:28.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:28.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:55:28.967 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:29.022 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:29.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:29.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:55:29.129 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:29.184 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:29.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:29.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:55:29.283 INFO:tasks.rados.rados.0.smithi125.stdout:302: done (4 left) 2021-03-24T21:55:29.283 INFO:tasks.rados.rados.0.smithi125.stdout:303: done (3 left) 2021-03-24T21:55:29.283 INFO:tasks.rados.rados.0.smithi125.stdout:305: done (2 left) 2021-03-24T21:55:29.283 INFO:tasks.rados.rados.0.smithi125.stdout:307: done (1 left) 2021-03-24T21:55:29.284 INFO:tasks.rados.rados.0.smithi125.stdout:308: done (0 left) 2021-03-24T21:55:29.284 INFO:tasks.rados.rados.0.smithi125.stdout:309: rollback oid 8 current snap is 28 2021-03-24T21:55:29.284 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 8 to 23 2021-03-24T21:55:29.284 INFO:tasks.rados.rados.0.smithi125.stdout:310: snap_create 2021-03-24T21:55:29.291 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:29.295 INFO:tasks.rados.rados.0.smithi125.stdout:309: finishing rollback tid 0 to smithi12556470-8 2021-03-24T21:55:29.295 INFO:tasks.rados.rados.0.smithi125.stdout:309: finishing rollback tid 1 to smithi12556470-8 2021-03-24T21:55:29.296 INFO:tasks.rados.rados.0.smithi125.stdout:309: finishing rollback tid 2 to smithi12556470-8 2021-03-24T21:55:29.296 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 8 v 27 (ObjNum 62 snap 4 seq_num 62) dirty exists 2021-03-24T21:55:29.340 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:29.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:29.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:55:29.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:29 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:55:29.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:29 smithi125 conmon[19936]: :55:28.165393+0000 osd.0 (osd.0) 45 : cluster [DBG] 3.6 scrub ok 2021-03-24T21:55:29.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:29 smithi125 conmon[19936]: cluster 2021-03-24T21:55 2021-03-24T21:55:29.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:29 smithi125 conmon[19936]: :28.270475+0000 mon.a (mon.0) 729 : 2021-03-24T21:55:29.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:29 smithi125 conmon[19936]: cluster [DBG] osdmap e159: 8 total, 8 up, 6 in 2021-03-24T21:55:29.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:29 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:55:29.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:29 smithi125 conmon[19936]: :55:28.556835+0000 osd.5 (osd.5) 44 2021-03-24T21:55:29.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:29 smithi125 conmon[19936]: : cluster [DBG] 6.d scrub ok 2021-03-24T21:55:29.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:29 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:55:29.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:29 smithi125 conmon[19936]: 24T21:55:28.629433+0000 osd.3 (osd 2021-03-24T21:55:29.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:29 smithi125 conmon[19936]: .3) 51 : cluster [DBG] 4.0 scrub ok 2021-03-24T21:55:29.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:29 smithi125 conmon[24927]: cluster 2021-03-24T21:55:28. 2021-03-24T21:55:29.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:29 smithi125 conmon[24927]: 165393+0000 osd.0 (osd.0) 45 : cluster [DBG] 3.6 scrub ok 2021-03-24T21:55:29.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:29 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:55:29.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:29 smithi125 conmon[24927]: 03-24T21:55:28.270475+0000 mon.a (mon.0) 729 : cluster [DBG] 2021-03-24T21:55:29.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:29 smithi125 conmon[24927]: osdmap e159: 8 total, 8 up, 6 in 2021-03-24T21:55:29.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:29 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:55:29.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:29 smithi125 conmon[24927]: 24T21:55:28.556835+0000 osd.5 (osd.5) 44 : cluster 2021-03-24T21:55:29.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:29 smithi125 conmon[24927]: [DBG] 6.d scrub ok 2021-03-24T21:55:29.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:29 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:55:29.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:29 smithi125 conmon[24927]: -03-24T21:55:28.629433+0000 osd.3 (osd.3) 51 2021-03-24T21:55:29.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:29 smithi125 conmon[24927]: : cluster [DBG] 4.0 scrub ok 2021-03-24T21:55:29.446 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:29.507 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:29.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:29.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:29.613 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:29.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:29 smithi142 conmon[20125]: cluster 2021-03-24T21:55:28.165393+0000 osd.0 (osd.0) 45 : cluster 2021-03-24T21:55:29.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:29 smithi142 conmon[20125]: [DBG] 3.6 scrub ok 2021-03-24T21:55:29.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:29 smithi142 conmon[20125]: cluster 2021-03-24T21:55:28.270475+0000 mon.a (mon.0) 2021-03-24T21:55:29.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:29 smithi142 conmon[20125]: 729 : cluster [DBG] osdmap e159: 8 total, 8 up, 6 in 2021-03-24T21:55:29.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:29 smithi142 conmon[20125]: cluster 2021-03-24T21:55:29.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:29 smithi142 conmon[20125]: 2021-03-24T21:55:28.556835+0000 osd.5 2021-03-24T21:55:29.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:29 smithi142 conmon[20125]: (osd.5) 44 : cluster [DBG] 6.d scrub ok 2021-03-24T21:55:29.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:29 smithi142 conmon[20125]: 2021-03-24T21:55:29.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:29 smithi142 conmon[20125]: cluster 2021-03-24T21:55:29.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:29 smithi142 conmon[20125]: 2021-03-24T21:55:28.629433+0000 osd.3 ( 2021-03-24T21:55:29.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:29 smithi142 conmon[20125]: osd.3) 51 : cluster [DBG] 4.0 scrub ok 2021-03-24T21:55:29.674 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:29.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:29.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:55:29.780 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:29.837 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:29.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:29.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:55:29.943 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:29.998 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:30.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:30.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:55:30.101 INFO:tasks.rados.rados.0.smithi125.stdout:309: done (1 left) 2021-03-24T21:55:30.148 INFO:tasks.rados.rados.0.smithi125.stdout:310: done (0 left) 2021-03-24T21:55:30.148 INFO:tasks.rados.rados.0.smithi125.stdout:311: snap_remove snap 28 2021-03-24T21:55:30.149 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:30.202 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:30.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:30.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:30.308 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:30.367 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:30.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:30.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:55:30.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:30 smithi125 conmon[24927]: cluster 2021-03-24T21:55: 2021-03-24T21:55:30.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:30 smithi125 conmon[24927]: 28.727849+0000 mgr.y (mgr.14144) 338 : cluster [DBG] pgmap v401: 148 pgs: 148 active+clean; 167 MiB data, 434 MiB used, 536 GiB / 536 GiB avail; 3.7 MiB/s rd, 242 KiB/s wr, 7 op/s 2021-03-24T21:55:30.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:30 smithi125 conmon[24927]: cluster 2021-03-24T21:55:30.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:30 smithi125 conmon[24927]: 2021-03-24T21:55:29.056126 2021-03-24T21:55:30.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:30 smithi125 conmon[24927]: +0000 osd.2 (osd.2) 30 : cluster [DBG] 2021-03-24T21:55:30.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:30 smithi125 conmon[24927]: 4.4 scrub ok 2021-03-24T21:55:30.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:30 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:55:30.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:30 smithi125 conmon[24927]: 21:55:29.282299+0000 mon.a (mon. 2021-03-24T21:55:30.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:30 smithi125 conmon[24927]: 0) 730 : cluster [DBG] osdmap e160: 8 total, 8 up, 6 in 2021-03-24T21:55:30.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:30 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:55:30.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:30 smithi125 conmon[24927]: 03-24T21:55:29.541304+0000 osd.5 2021-03-24T21:55:30.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:30 smithi125 conmon[24927]: (osd.5) 45 : cluster [DBG] 5.15 scrub ok 2021-03-24T21:55:30.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:30 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:55:30.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:30 smithi125 conmon[24927]: -03-24T21:55:30.096537+0000 mon.a ( 2021-03-24T21:55:30.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:30 smithi125 conmon[24927]: mon.0) 731 : cluster [DBG] osdmap e161: 8 total, 8 up, 6 in 2021-03-24T21:55:30.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:30 smithi125 conmon[24927]: audit 2021-03-24T21:55:30.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:30 smithi125 conmon[24927]: 2021-03-24T21:55:30.153379+0000 2021-03-24T21:55:30.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:30 smithi125 conmon[24927]: mon.a (mon.0) 732 : audit [INF] from='client.? v1:172.21.15.125:0/1679475150' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"}]: dispatch 2021-03-24T21:55:30.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:30 smithi125 conmon[24927]: 2021-03-24T21:55:30.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:30 smithi125 conmon[24927]: cluster 2021-03-24T21:55:30.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:30 smithi125 conmon[24927]: 2021-03-24T21:55:30.166100+0000 2021-03-24T21:55:30.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:30 smithi125 conmon[24927]: osd.4 (osd.4) 45 : cluster [DBG] 4.5 scrub ok 2021-03-24T21:55:30.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:30 smithi125 conmon[24927]: 2021-03-24T21:55:30.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:30 smithi125 conmon[19936]: cluster 2021-03-24T21:55 2021-03-24T21:55:30.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:30 smithi125 conmon[19936]: :28.727849+0000 mgr.y (mgr.14144) 338 : cluster [DBG] pgmap v401: 148 pgs: 148 active+clean; 167 MiB data, 434 MiB used, 536 GiB / 536 GiB avail; 3.7 MiB/s rd, 242 KiB/s wr, 7 op/s 2021-03-24T21:55:30.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:30 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:55:30.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:30 smithi125 conmon[19936]: -24T21:55:29.056126+0000 osd.2 (osd.2) 30 : cluster 2021-03-24T21:55:30.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:30 smithi125 conmon[19936]: [DBG] 4.4 scrub ok 2021-03-24T21:55:30.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:30 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:55:30.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:30 smithi125 conmon[19936]: -03-24T21:55:29.282299+0000 2021-03-24T21:55:30.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:30 smithi125 conmon[19936]: mon.a (mon.0) 730 : cluster [DBG] 2021-03-24T21:55:30.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:30 smithi125 conmon[19936]: osdmap e160: 8 total, 8 up, 6 in 2021-03-24T21:55:30.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:30 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:55:30.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:30 smithi125 conmon[19936]: -24T21:55:29 2021-03-24T21:55:30.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:30 smithi125 conmon[19936]: .541304+0000 osd.5 (osd. 2021-03-24T21:55:30.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:30 smithi125 conmon[19936]: 5) 45 : cluster [DBG] 5.15 scrub ok 2021-03-24T21:55:30.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:30 smithi125 conmon[19936]: 2021-03-24T21:55:30.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:30 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:55:30.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:30 smithi125 conmon[19936]: -24T21:55:30.096537+0000 mon.a 2021-03-24T21:55:30.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:30 smithi125 conmon[19936]: (mon.0) 731 : cluster [DBG] osdmap e161: 8 total, 8 up, 6 in 2021-03-24T21:55:30.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:30 smithi125 conmon[19936]: audit 2021 2021-03-24T21:55:30.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:30 smithi125 conmon[19936]: -03-24T21:55:30.153379+0000 2021-03-24T21:55:30.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:30 smithi125 conmon[19936]: mon.a (mon.0) 732 : audit [INF] from='client.? v1:172.21.15.125:0/1679475150' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"}]: dispatch 2021-03-24T21:55:30.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:30 smithi125 conmon[19936]: 2021-03-24T21:55:30.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:30 smithi125 conmon[19936]: cluster 2021-03-24T21:55:30.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:30 smithi125 conmon[19936]: 2021-03-24 2021-03-24T21:55:30.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:30 smithi125 conmon[19936]: T21:55:30.166100+0000 osd.4 2021-03-24T21:55:30.421 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:30 smithi125 conmon[19936]: (osd.4) 45 : cluster [DBG] 4.5 scrub ok 2021-03-24T21:55:30.421 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:30 smithi125 conmon[19936]: 2021-03-24T21:55:30.473 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:30.535 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:30.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:30.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:30.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:30 smithi142 conmon[20125]: cluster 2021-03-24T21:55:28.727849+0000 mgr.y (mgr.14144) 338 : cluster [DBG] 2021-03-24T21:55:30.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:30 smithi142 conmon[20125]: pgmap v401: 148 pgs: 148 active+clean; 167 MiB data, 434 MiB used, 536 GiB / 536 GiB avail; 3.7 MiB/s rd, 242 KiB/s wr, 7 op/s 2021-03-24T21:55:30.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:30 smithi142 conmon[20125]: cluster 2021-03-24T21:55:29.056126+0000 osd.2 (osd.2) 30 : cluster [DBG] 2021-03-24T21:55:30.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:30 smithi142 conmon[20125]: 4.4 scrub ok 2021-03-24T21:55:30.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:30 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:55:30.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:30 smithi142 conmon[20125]: -24T21:55:29.282299+0000 mon.a (mon. 2021-03-24T21:55:30.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:30 smithi142 conmon[20125]: 0) 730 : cluster [DBG] osdmap e160: 8 total, 8 up, 6 in 2021-03-24T21:55:30.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:30 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:55:30.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:30 smithi142 conmon[20125]: 03-24T21:55:29.541304+0000 2021-03-24T21:55:30.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:30 smithi142 conmon[20125]: osd.5 (osd.5) 45 : cluster [DBG] 2021-03-24T21:55:30.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:30 smithi142 conmon[20125]: 5.15 scrub ok 2021-03-24T21:55:30.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:30 smithi142 conmon[20125]: cluster 2021-03-24T21:55:30.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:30 smithi142 conmon[20125]: 2021-03-24T21:55:30.096537+0000 2021-03-24T21:55:30.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:30 smithi142 conmon[20125]: mon.a (mon.0) 731 : cluster [DBG] 2021-03-24T21:55:30.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:30 smithi142 conmon[20125]: osdmap e161: 8 total, 8 up, 6 in 2021-03-24T21:55:30.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:30 smithi142 conmon[20125]: audit 2021-03-24T21:55:30.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:30 smithi142 conmon[20125]: 2021-03-24T21:55:30.153379+0000 2021-03-24T21:55:30.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:30 smithi142 conmon[20125]: mon.a (mon.0) 732 : audit [INF] 2021-03-24T21:55:30.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:30 smithi142 conmon[20125]: from='client.? v1:172.21.15.125:0/1679475150' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"}]: dispatch 2021-03-24T21:55:30.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:30 smithi142 conmon[20125]: cluster 2021-03-24T21:55:30.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:30 smithi142 conmon[20125]: 2021-03-24T21:55:30.166100+0000 2021-03-24T21:55:30.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:30 smithi142 conmon[20125]: osd.4 (osd.4) 45 : cluster [DBG] 2021-03-24T21:55:30.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:30 smithi142 conmon[20125]: 4.5 scrub ok 2021-03-24T21:55:30.639 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:30.700 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:30.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:30.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:55:30.806 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:30.863 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:30.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:30.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:55:30.970 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:31.019 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:31.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:31.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:55:31.107 INFO:tasks.rados.rados.0.smithi125.stdout:311: done (0 left) 2021-03-24T21:55:31.107 INFO:tasks.rados.rados.0.smithi125.stdout:312: copy_from oid 29 from oid 45 current snap is 29 2021-03-24T21:55:31.108 INFO:tasks.rados.rados.0.smithi125.stdout:313: rollback oid 34 current snap is 29 2021-03-24T21:55:31.108 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 34 to 23 2021-03-24T21:55:31.108 INFO:tasks.rados.rados.0.smithi125.stdout:314: read oid 13 snap -1 2021-03-24T21:55:31.108 INFO:tasks.rados.rados.0.smithi125.stdout:314: expect (ObjNum 123 snap 23 seq_num 123) 2021-03-24T21:55:31.109 INFO:tasks.rados.rados.0.smithi125.stdout:315: read oid 26 snap -1 2021-03-24T21:55:31.109 INFO:tasks.rados.rados.0.smithi125.stdout:315: expect (ObjNum 141 snap 27 seq_num 141) 2021-03-24T21:55:31.109 INFO:tasks.rados.rados.0.smithi125.stdout:316: write oid 17 current snap is 29 2021-03-24T21:55:31.109 INFO:tasks.rados.rados.0.smithi125.stdout:316: seq_num 144 ranges {420208=530210,1527054=766050} 2021-03-24T21:55:31.110 INFO:tasks.rados.rados.0.smithi125.stdout:316: writing smithi12556470-17 from 420208 to 950418 tid 1 2021-03-24T21:55:31.118 INFO:teuthology.orchestra.run.smithi125.stderr:set pool 2 pgp_num to 3 2021-03-24T21:55:31.119 INFO:tasks.rados.rados.0.smithi125.stdout:316: writing smithi12556470-17 from 1527054 to 2293104 tid 2 2021-03-24T21:55:31.120 INFO:tasks.rados.rados.0.smithi125.stdout:317: snap_remove snap 25 2021-03-24T21:55:31.125 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:31.169 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:31.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:31.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:31.274 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:31.321 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:31.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:31.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:55:31.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:31 smithi125 conmon[24927]: cluster 2021-03-24T21:55:29.899912+0000 2021-03-24T21:55:31.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:31 smithi125 conmon[24927]: osd.7 (osd.7) 36 : cluster [DBG] 6.1c scrub ok 2021-03-24T21:55:31.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:31 smithi125 conmon[24927]: cluster 2021-03-24T21:55:30.102188+0000 osd.2 (osd 2021-03-24T21:55:31.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:31 smithi125 conmon[24927]: .2) 31 : cluster [DBG] 3.0 scrub ok 2021-03-24T21:55:31.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:31 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:55:31.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:31 smithi125 conmon[24927]: -03-24T21:55:30.165351+0000 osd.1 (osd 2021-03-24T21:55:31.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:31 smithi125 conmon[24927]: .1) 33 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:55:31.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:31 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:55:31.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:31 smithi125 conmon[24927]: 55:30.165437+0000 osd.1 (osd.1) 34 : cluster [DBG] 2021-03-24T21:55:31.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:31 smithi125 conmon[24927]: purged_snaps scrub ok 2021-03-24T21:55:31.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:31 smithi125 conmon[24927]: cluster 2021-03-24T21:55:31 2021-03-24T21:55:31.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:31 smithi125 conmon[24927]: .084857+0000 osd.4 (osd.4) 46 : cluster [DBG] 2021-03-24T21:55:31.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:31 smithi125 conmon[24927]: 4.1a scrub ok 2021-03-24T21:55:31.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:31 smithi125 conmon[24927]: audit 2021-03-24T21:55:31 2021-03-24T21:55:31.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:31 smithi125 conmon[24927]: .102492+0000 mon.a (mon.0) 733 : audit [INF] 2021-03-24T21:55:31.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:31 smithi125 conmon[24927]: from='client.? v1:172.21.15.125:0/1679475150' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"}]': finished 2021-03-24T21:55:31.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:31 smithi125 conmon[24927]: cluster 2021-03-24T21:55: 2021-03-24T21:55:31.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:31 smithi125 conmon[24927]: 31.102678+0000 mon.a (mon.0) 734 : cluster 2021-03-24T21:55:31.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:31 smithi125 conmon[24927]: [DBG] osdmap e162: 8 total, 8 up, 6 in 2021-03-24T21:55:31.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:31 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:55:31.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:31 smithi125 conmon[19936]: 21:55:29.899912+0000 osd.7 (osd.7) 36 : cluster [DBG] 2021-03-24T21:55:31.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:31 smithi125 conmon[19936]: 6.1c scrub ok 2021-03-24T21:55:31.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:31 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:55:31.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:31 smithi125 conmon[19936]: :55:30.102188+0000 osd.2 (osd.2) 31 : cluster [DBG] 3.0 scrub ok 2021-03-24T21:55:31.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:31 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:55:31.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:31 smithi125 conmon[19936]: 24T21:55:30.165351+0000 osd.1 (osd.1) 33 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:55:31.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:31 smithi125 conmon[19936]: 2021-03-24T21:55:31.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:31 smithi125 conmon[19936]: cluster 2021-03-24T21:55:31.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:31 smithi125 conmon[19936]: 2021-03-24T21:55:30. 2021-03-24T21:55:31.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:31 smithi125 conmon[19936]: 165437+0000 osd.1 (osd.1) 34 : cluster 2021-03-24T21:55:31.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:31 smithi125 conmon[19936]: [DBG] purged_snaps scrub ok 2021-03-24T21:55:31.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:31 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:55:31.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:31 smithi125 conmon[19936]: -03-24T21:55:31 2021-03-24T21:55:31.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:31 smithi125 conmon[19936]: .084857+0000 osd.4 (osd.4) 46 : cluster 2021-03-24T21:55:31.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:31 smithi125 conmon[19936]: [DBG] 4.1a scrub ok 2021-03-24T21:55:31.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:31 smithi125 conmon[19936]: audit 2021-03-24T21:55:31.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:31 smithi125 conmon[19936]: 2021-03-24T 2021-03-24T21:55:31.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:31 smithi125 conmon[19936]: 21:55:31.102492+0000 mon.a (mon 2021-03-24T21:55:31.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:31 smithi125 conmon[19936]: .0) 733 : audit [INF] from='client.? v1:172.21.15.125:0/1679475150' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"}]': finished 2021-03-24T21:55:31.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:31 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:55:31.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:31 smithi125 conmon[19936]: -03-24T21:55:31.102678+0000 2021-03-24T21:55:31.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:31 smithi125 conmon[19936]: mon.a (mon.0) 734 : cluster [DBG] osdmap e162: 8 total, 8 up, 6 in 2021-03-24T21:55:31.421 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:31 smithi125 conmon[19936]: 2021-03-24T21:55:31.427 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:31.485 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:31.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:31.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:55:31.591 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:31.606 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:31 smithi142 conmon[20125]: cluster 2021-03-24T21:55:29.899912+0000 osd.7 (osd.7) 36 : cluster 2021-03-24T21:55:31.606 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:31 smithi142 conmon[20125]: [DBG] 6.1c scrub ok 2021-03-24T21:55:31.607 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:31 smithi142 conmon[20125]: cluster 2021-03-24T21:55:30.102188+0000 osd.2 2021-03-24T21:55:31.607 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:31 smithi142 conmon[20125]: (osd.2) 31 : cluster [DBG] 3.0 scrub ok 2021-03-24T21:55:31.608 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:31 smithi142 conmon[20125]: cluster 2021-03-24T21 2021-03-24T21:55:31.608 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:31 smithi142 conmon[20125]: :55:30.165351+0000 osd.1 (osd. 2021-03-24T21:55:31.609 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:31 smithi142 conmon[20125]: 1) 33 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:55:31.609 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:31 smithi142 conmon[20125]: cluster 2021-03-24T21:55:31.609 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:31 smithi142 conmon[20125]: 2021-03-24T21:55:30.165437 2021-03-24T21:55:31.610 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:31 smithi142 conmon[20125]: +0000 osd.1 (osd.1) 34 : cluster [DBG] 2021-03-24T21:55:31.610 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:31 smithi142 conmon[20125]: purged_snaps scrub ok 2021-03-24T21:55:31.610 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:31 smithi142 conmon[20125]: cluster 2021-03-24T21:55:31.611 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:31 smithi142 conmon[20125]: 2021-03-24T21:55:31.084857+0000 osd.4 (osd.4) 46 : 2021-03-24T21:55:31.611 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:31 smithi142 conmon[20125]: cluster [DBG] 4.1a scrub ok 2021-03-24T21:55:31.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:31 smithi142 conmon[20125]: audit 2021-03-24T21:55:31.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:31 smithi142 conmon[20125]: 2021-03-24T21:55:31.102492+0000 mon.a 2021-03-24T21:55:31.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:31 smithi142 conmon[20125]: (mon.0) 733 : audit [INF] from='client.? v1:172.21.15.125:0/1679475150' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"}]': finished 2021-03-24T21:55:31.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:31 smithi142 conmon[20125]: 2021-03-24T21:55:31.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:31 smithi142 conmon[20125]: cluster 2021-03-24T21:55:31.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:31 smithi142 conmon[20125]: 2021-03-24T21:55:31.102678+0000 mon.a 2021-03-24T21:55:31.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:31 smithi142 conmon[20125]: (mon.0) 734 : cluster [DBG] osdmap e162: 8 total, 8 up, 6 in 2021-03-24T21:55:31.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:31 smithi142 conmon[20125]: 2021-03-24T21:55:31.647 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:31.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:31.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:55:31.754 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:31.812 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:31.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:31.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:55:31.919 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:31.974 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:31.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:32.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:55:32.104 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:32.112 INFO:tasks.rados.rados.0.smithi125.stdout:314: expect (ObjNum 123 snap 23 seq_num 123) 2021-03-24T21:55:32.151 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:32.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:32.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:55:32.229 INFO:tasks.rados.rados.0.smithi125.stdout:313: finishing rollback tid 1 to smithi12556470-34 2021-03-24T21:55:32.229 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 34 v 0 (ObjNum 288 snap 0 seq_num 49) dirty dne 2021-03-24T21:55:32.230 INFO:tasks.rados.rados.0.smithi125.stdout:315: expect (ObjNum 141 snap 27 seq_num 141) 2021-03-24T21:55:32.256 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:32.309 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:32.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:32.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:55:32.314 INFO:tasks.rados.rados.0.smithi125.stdout:316: finishing write tid 1 to smithi12556470-17 2021-03-24T21:55:32.315 INFO:tasks.rados.rados.0.smithi125.stdout:316: finishing write tid 2 to smithi12556470-17 2021-03-24T21:55:32.315 INFO:tasks.rados.rados.0.smithi125.stdout:316: finishing write tid 3 to smithi12556470-17 2021-03-24T21:55:32.315 INFO:tasks.rados.rados.0.smithi125.stdout:316: finishing write tid 4 to smithi12556470-17 2021-03-24T21:55:32.315 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 17 v 114 (ObjNum 144 snap 29 seq_num 144) dirty exists 2021-03-24T21:55:32.316 INFO:tasks.rados.rados.0.smithi125.stdout:316: left oid 17 (ObjNum 144 snap 29 seq_num 144) 2021-03-24T21:55:32.316 INFO:tasks.rados.rados.0.smithi125.stdout:312: finishing copy_from to smithi12556470-29 2021-03-24T21:55:32.316 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 29 v 49 (ObjNum 101 snap 20 seq_num 101) dirty exists 2021-03-24T21:55:32.316 INFO:tasks.rados.rados.0.smithi125.stdout:312: finishing copy_from racing read to smithi12556470-29 2021-03-24T21:55:32.316 INFO:tasks.rados.rados.0.smithi125.stdout:312: done (5 left) 2021-03-24T21:55:32.317 INFO:tasks.rados.rados.0.smithi125.stdout:313: done (4 left) 2021-03-24T21:55:32.317 INFO:tasks.rados.rados.0.smithi125.stdout:314: done (3 left) 2021-03-24T21:55:32.317 INFO:tasks.rados.rados.0.smithi125.stdout:315: done (2 left) 2021-03-24T21:55:32.317 INFO:tasks.rados.rados.0.smithi125.stdout:316: done (1 left) 2021-03-24T21:55:32.318 INFO:tasks.rados.rados.0.smithi125.stdout:317: done (0 left) 2021-03-24T21:55:32.318 INFO:tasks.rados.rados.0.smithi125.stdout:318: read oid 38 snap 23 2021-03-24T21:55:32.318 INFO:tasks.rados.rados.0.smithi125.stdout:318: expect (ObjNum 121 snap 23 seq_num 121) 2021-03-24T21:55:32.318 INFO:tasks.rados.rados.0.smithi125.stdout:319: delete oid 26 current snap is 29 2021-03-24T21:55:32.319 INFO:tasks.rados.rados.0.smithi125.stdout:319: done (1 left) 2021-03-24T21:55:32.319 INFO:tasks.rados.rados.0.smithi125.stdout:320: snap_remove snap 27 2021-03-24T21:55:32.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:32 smithi142 conmon[20125]: cluster 2021-03-24T21:55: 2021-03-24T21:55:32.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:32 smithi142 conmon[20125]: 30.728661+0000 mgr.y (mgr.14144) 339 : cluster [DBG] pgmap v404: 148 pgs: 148 active+clean; 123 MiB data, 358 MiB used, 536 GiB / 536 GiB avail; 3.8 MiB/s rd, 439 KiB/s wr, 10 op/s 2021-03-24T21:55:32.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:32 smithi142 conmon[20125]: cluster 2021-03-24T21:55: 2021-03-24T21:55:32.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:32 smithi142 conmon[20125]: 31.118758+0000 osd.2 (osd.2) 32 : cluster [DBG] 5.6 scrub ok 2021-03-24T21:55:32.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:32 smithi142 conmon[20125]: cluster 2021-03-24T21:55:31.561647+0000 osd.5 2021-03-24T21:55:32.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:32 smithi142 conmon[20125]: (osd.5) 46 : cluster [DBG] 4.15 scrub ok 2021-03-24T21:55:32.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:32 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:55:32.429 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:32 smithi142 conmon[20125]: 21:55:31.735379+0000 osd.4 (osd.4) 47 : 2021-03-24T21:55:32.430 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:32 smithi142 conmon[20125]: cluster [DBG] purged_snaps scrub starts 2021-03-24T21:55:32.430 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:32 smithi142 conmon[20125]: cluster 2021-03-24T21:55:31.735621+0000 2021-03-24T21:55:32.430 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:32 smithi142 conmon[20125]: osd.4 (osd.4) 48 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:55:32.430 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:32 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:55:32.431 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:32 smithi142 conmon[20125]: 24T21:55:32.104858+0000 mon.a (mon.0) 735 : cluster 2021-03-24T21:55:32.431 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:32 smithi142 conmon[20125]: [DBG] osdmap e163: 8 total, 8 up, 6 in 2021-03-24T21:55:32.432 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:32.488 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:32.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:32.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:55:32.593 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:32.605 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:32 smithi125 conmon[19936]: cluster 2021-03-24T21:55 2021-03-24T21:55:32.606 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:32 smithi125 conmon[19936]: :30.728661+0000 mgr.y (mgr.14144) 339 : cluster [DBG] pgmap v404: 148 pgs: 148 active+clean; 123 MiB data, 358 MiB used, 536 GiB / 536 GiB avail; 3.8 MiB/s rd, 439 KiB/s wr, 10 op/s 2021-03-24T21:55:32.606 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:32 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:55:32.606 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:32 smithi125 conmon[19936]: 03-24T21:55:31.118758+0000 osd.2 (osd.2) 32 : cluster [DBG] 2021-03-24T21:55:32.607 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:32 smithi125 conmon[19936]: 5.6 scrub ok 2021-03-24T21:55:32.607 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:32 smithi125 conmon[19936]: cluster 2021-03-24T21:55:31 2021-03-24T21:55:32.607 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:32 smithi125 conmon[19936]: .561647+0000 osd.5 (osd.5) 46 : cluster 2021-03-24T21:55:32.608 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:32 smithi125 conmon[19936]: [DBG] 4.15 scrub ok 2021-03-24T21:55:32.608 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:32 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:55:32.608 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:32 smithi125 conmon[19936]: :55:31.735379+0000 osd.4 (osd.4) 2021-03-24T21:55:32.609 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:32 smithi125 conmon[19936]: 47 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:55:32.609 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:32 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:55:32.609 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:32 smithi125 conmon[19936]: :55:31.735621+0000 osd.4 (osd.4) 2021-03-24T21:55:32.610 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:32 smithi125 conmon[19936]: 48 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:55:32.610 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:32 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:55:32.610 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:32 smithi125 conmon[19936]: 24T21:55:32.104858+0000 mon.a (mon. 2021-03-24T21:55:32.611 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:32 smithi125 conmon[19936]: 0) 735 : cluster [DBG] osdmap e163: 8 total, 8 up, 6 in 2021-03-24T21:55:32.612 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:32 smithi125 conmon[24927]: cluster 2021-03-24T21:55:30.728661 2021-03-24T21:55:32.612 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:32 smithi125 conmon[24927]: +0000 mgr.y (mgr.14144) 339 : cluster [DBG] pgmap v404: 148 pgs: 148 active+clean; 123 MiB data, 358 MiB used, 536 GiB / 536 GiB avail; 3.8 MiB/s rd, 439 KiB/s wr, 10 op/s 2021-03-24T21:55:32.612 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:32 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:55:32.613 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:32 smithi125 conmon[24927]: 55:31.118758+0000 osd.2 (osd.2) 32 : cluster [DBG] 5.6 scrub ok 2021-03-24T21:55:32.613 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:32 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:55:32.613 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:32 smithi125 conmon[24927]: 21:55:31.561647+0000 osd.5 (osd.5 2021-03-24T21:55:32.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:32 smithi125 conmon[24927]: ) 46 : cluster [DBG] 4.15 scrub ok 2021-03-24T21:55:32.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:32 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:55:32.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:32 smithi125 conmon[24927]: 24T21:55:31.735379+0000 osd.4 (osd 2021-03-24T21:55:32.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:32 smithi125 conmon[24927]: .4) 47 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:55:32.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:32 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:55:32.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:32 smithi125 conmon[24927]: 21:55:31.735621+0000 osd.4 (osd.4) 48 : 2021-03-24T21:55:32.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:32 smithi125 conmon[24927]: cluster [DBG] purged_snaps scrub ok 2021-03-24T21:55:32.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:32 smithi125 conmon[24927]: cluster 2021-03-24T21:55:32 2021-03-24T21:55:32.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:32 smithi125 conmon[24927]: .104858+0000 mon.a (mon.0) 735 : cluster 2021-03-24T21:55:32.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:32 smithi125 conmon[24927]: [DBG] osdmap e163: 8 total, 8 up, 6 in 2021-03-24T21:55:32.653 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:32.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:32.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:55:32.760 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:32.828 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:32.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:32.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:55:32.934 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:32.994 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:32.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:32.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:55:33.079 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-24T21:55:33.080 INFO:teuthology.orchestra.run.smithi125.stdout: "size": 20, 2021-03-24T21:55:33.081 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 600, 2021-03-24T21:55:33.081 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [ 2021-03-24T21:55:33.081 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.081 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.34352.0:541 4.16 e65/62 4:6b9e929a:::data_log.89:head v 65'5, mlcod=65'3)", 2021-03-24T21:55:33.082 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:51:50.074290+0000", 2021-03-24T21:55:33.082 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 223.00104662699999, 2021-03-24T21:55:33.082 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.017957885, 2021-03-24T21:55:33.082 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:55:33.083 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:55:33.083 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:55:33.083 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.083 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:55:33.083 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.074290+0000", 2021-03-24T21:55:33.084 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:55:33.084 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.084 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.085 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:55:33.085 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.074290+0000", 2021-03-24T21:55:33.085 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999971 2021-03-24T21:55:33.086 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.086 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.086 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:55:33.087 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.074287+0000", 2021-03-24T21:55:33.087 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.1458000000000003e-05 2021-03-24T21:55:33.088 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.088 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.088 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:55:33.088 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.074318+0000", 2021-03-24T21:55:33.089 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.0975000000000001e-05 2021-03-24T21:55:33.089 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.089 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.089 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:55:33.089 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.074349+0000", 2021-03-24T21:55:33.090 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.4476e-05 2021-03-24T21:55:33.090 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.090 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.090 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:55:33.091 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.074364+0000", 2021-03-24T21:55:33.091 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.5336999999999999e-05 2021-03-24T21:55:33.091 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.091 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.091 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:33.092 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.074459+0000", 2021-03-24T21:55:33.092 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.2963999999999998e-05 2021-03-24T21:55:33.092 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.092 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.093 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:55:33.093 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.074492+0000", 2021-03-24T21:55:33.093 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.017691891000000001 2021-03-24T21:55:33.094 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.094 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.094 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:55:33.094 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.092184+0000", 2021-03-24T21:55:33.095 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.3805000000000003e-05 2021-03-24T21:55:33.095 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.095 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.095 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:55:33.095 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.092248+0000", 2021-03-24T21:55:33.096 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.017957885 2021-03-24T21:55:33.096 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:33.096 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:55:33.096 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:33.096 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.097 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.097 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:201 7.c 7:32974111:::smithi12556470-2:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected e107)", 2021-03-24T21:55:33.097 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.781527+0000", 2021-03-24T21:55:33.097 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 56.293809598999999, 2021-03-24T21:55:33.098 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.016494275999999999, 2021-03-24T21:55:33.098 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:55:33.098 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:55:33.098 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:55:33.099 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:55:33.099 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:55:33.099 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 201 2021-03-24T21:55:33.099 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.100 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:55:33.100 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.100 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:55:33.100 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.781527+0000", 2021-03-24T21:55:33.100 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:55:33.101 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.101 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.101 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:55:33.101 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.781527+0000", 2021-03-24T21:55:33.101 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999957 2021-03-24T21:55:33.102 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.102 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.102 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:55:33.102 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.781523+0000", 2021-03-24T21:55:33.103 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.8909000000000001e-05 2021-03-24T21:55:33.103 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.103 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.103 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:55:33.103 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.781542+0000", 2021-03-24T21:55:33.104 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.1047000000000002e-05 2021-03-24T21:55:33.104 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.104 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.104 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:55:33.104 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.781583+0000", 2021-03-24T21:55:33.105 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7564e-05 2021-03-24T21:55:33.105 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.105 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.105 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:55:33.106 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.781600+0000", 2021-03-24T21:55:33.106 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00019036899999999999 2021-03-24T21:55:33.106 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.106 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.106 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:33.107 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.781791+0000", 2021-03-24T21:55:33.107 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000275288 2021-03-24T21:55:33.107 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.107 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.108 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:55:33.108 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.782066+0000", 2021-03-24T21:55:33.108 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0089400980000000005 2021-03-24T21:55:33.109 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.109 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.109 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,7", 2021-03-24T21:55:33.109 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.791006+0000", 2021-03-24T21:55:33.110 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0047708189999999999 2021-03-24T21:55:33.110 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.110 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.110 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:55:33.111 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.795777+0000", 2021-03-24T21:55:33.111 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001420564 2021-03-24T21:55:33.111 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.111 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.111 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:55:33.112 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.797197+0000", 2021-03-24T21:55:33.112 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00056311299999999996 2021-03-24T21:55:33.112 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.112 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.113 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:55:33.113 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.797760+0000", 2021-03-24T21:55:33.113 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000116898 2021-03-24T21:55:33.113 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.114 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.114 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:55:33.114 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.797877+0000", 2021-03-24T21:55:33.114 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000143769 2021-03-24T21:55:33.115 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.115 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.115 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:55:33.115 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.798021+0000", 2021-03-24T21:55:33.115 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.016494275999999999 2021-03-24T21:55:33.116 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:33.116 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:55:33.116 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:33.116 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.116 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.117 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:202 7.c 7:32974111:::smithi12556470-2:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e107)", 2021-03-24T21:55:33.117 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.781643+0000", 2021-03-24T21:55:33.117 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 56.293693887000003, 2021-03-24T21:55:33.117 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.016560772000000001, 2021-03-24T21:55:33.118 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:55:33.118 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T21:55:33.118 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:55:33.118 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:55:33.119 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:55:33.119 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 202 2021-03-24T21:55:33.120 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.120 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:55:33.120 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.121 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:55:33.121 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.781643+0000", 2021-03-24T21:55:33.122 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:55:33.122 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.122 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.123 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:55:33.123 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.781643+0000", 2021-03-24T21:55:33.124 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T21:55:33.124 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.124 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.125 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:55:33.125 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.781641+0000", 2021-03-24T21:55:33.126 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.9519999999999994e-06 2021-03-24T21:55:33.126 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.126 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.127 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:55:33.127 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.781649+0000", 2021-03-24T21:55:33.128 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9220000000000001e-05 2021-03-24T21:55:33.128 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.129 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.129 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:55:33.129 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.781688+0000", 2021-03-24T21:55:33.130 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.4593999999999999e-05 2021-03-24T21:55:33.130 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.131 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.131 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:55:33.131 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.781713+0000", 2021-03-24T21:55:33.132 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00069837999999999996 2021-03-24T21:55:33.132 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.133 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.133 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:33.134 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.782411+0000", 2021-03-24T21:55:33.134 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.3856000000000005e-05 2021-03-24T21:55:33.134 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.135 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.135 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-24T21:55:33.136 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.782495+0000", 2021-03-24T21:55:33.136 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.013149507 2021-03-24T21:55:33.136 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.137 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.137 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:33.138 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.795645+0000", 2021-03-24T21:55:33.138 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.0933000000000006e-05 2021-03-24T21:55:33.139 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.139 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.139 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T21:55:33.140 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.795726+0000", 2021-03-24T21:55:33.140 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0023349199999999999 2021-03-24T21:55:33.141 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.141 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.141 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:33.142 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.798060+0000", 2021-03-24T21:55:33.142 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4214000000000006e-05 2021-03-24T21:55:33.142 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.143 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.143 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:55:33.143 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.798125+0000", 2021-03-24T21:55:33.143 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.8665000000000007e-05 2021-03-24T21:55:33.143 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.144 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.144 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:55:33.144 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.798203+0000", 2021-03-24T21:55:33.144 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.016560772000000001 2021-03-24T21:55:33.145 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:33.145 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:55:33.145 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:33.145 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.145 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.146 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:213 7.3 7:c1eb355b:::smithi12556470-34:head [delete] snapc 2=[2] ondisk+write+peerstat_old+known_if_redirected e108)", 2021-03-24T21:55:33.146 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:37.086973+0000", 2021-03-24T21:55:33.146 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 55.988363624999998, 2021-03-24T21:55:33.146 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.033989375000000002, 2021-03-24T21:55:33.147 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:55:33.147 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:55:33.147 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:55:33.147 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:55:33.147 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:55:33.148 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 213 2021-03-24T21:55:33.148 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.148 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:55:33.148 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.148 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:55:33.149 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:37.086973+0000", 2021-03-24T21:55:33.149 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:55:33.149 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.149 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.150 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:55:33.150 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:37.086973+0000", 2021-03-24T21:55:33.150 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999943 2021-03-24T21:55:33.150 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.151 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.151 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:55:33.151 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:37.086967+0000", 2021-03-24T21:55:33.151 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.2189e-05 2021-03-24T21:55:33.152 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.152 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.152 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:55:33.152 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:37.086989+0000", 2021-03-24T21:55:33.152 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.4728999999999999e-05 2021-03-24T21:55:33.153 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.153 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.153 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:55:33.153 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:37.087024+0000", 2021-03-24T21:55:33.154 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0747e-05 2021-03-24T21:55:33.154 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.154 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.154 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:55:33.154 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:37.087045+0000", 2021-03-24T21:55:33.155 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.6952000000000003e-05 2021-03-24T21:55:33.155 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.155 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.155 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:33.155 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:37.087142+0000", 2021-03-24T21:55:33.156 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.2420000000000003e-06 2021-03-24T21:55:33.156 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.156 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.156 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op must wait for map", 2021-03-24T21:55:33.156 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:37.087149+0000", 2021-03-24T21:55:33.157 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.023165432 2021-03-24T21:55:33.157 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.157 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.157 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:33.158 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:37.110314+0000", 2021-03-24T21:55:33.158 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.3443999999999995e-05 2021-03-24T21:55:33.158 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.158 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.158 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:55:33.159 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:37.110408+0000", 2021-03-24T21:55:33.159 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00021713799999999999 2021-03-24T21:55:33.159 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.159 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.160 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,4", 2021-03-24T21:55:33.160 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:37.110625+0000", 2021-03-24T21:55:33.160 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0049017790000000002 2021-03-24T21:55:33.160 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.160 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.161 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:55:33.161 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:37.115527+0000", 2021-03-24T21:55:33.161 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0022026319999999999 2021-03-24T21:55:33.161 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.162 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.162 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:55:33.162 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:37.117729+0000", 2021-03-24T21:55:33.162 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0031449519999999999 2021-03-24T21:55:33.163 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.163 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.163 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:55:33.163 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:37.120874+0000", 2021-03-24T21:55:33.163 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6164e-05 2021-03-24T21:55:33.164 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.164 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.164 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:55:33.164 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:37.120920+0000", 2021-03-24T21:55:33.164 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.1894000000000002e-05 2021-03-24T21:55:33.165 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.165 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.165 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:55:33.165 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:37.120962+0000", 2021-03-24T21:55:33.165 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.033989375000000002 2021-03-24T21:55:33.166 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:33.166 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:55:33.166 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:33.166 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.167 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.167 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:235 7.c 7:32974111:::smithi12556470-2:head [stat,delete] snapc 3=[3] ondisk+write+known_if_redirected e110)", 2021-03-24T21:55:33.167 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:39.115347+0000", 2021-03-24T21:55:33.167 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 53.959989014000001, 2021-03-24T21:55:33.167 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.025697194999999999, 2021-03-24T21:55:33.168 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:55:33.168 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:55:33.168 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:55:33.168 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:55:33.168 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:55:33.169 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 235 2021-03-24T21:55:33.169 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.169 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:55:33.169 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.170 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:55:33.170 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:39.115347+0000", 2021-03-24T21:55:33.170 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:55:33.170 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.170 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.171 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:55:33.171 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:39.115347+0000", 2021-03-24T21:55:33.171 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999971 2021-03-24T21:55:33.171 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.172 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.172 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:55:33.172 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:39.115345+0000", 2021-03-24T21:55:33.172 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.8789999999999998e-06 2021-03-24T21:55:33.173 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.173 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.173 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:55:33.173 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:39.115355+0000", 2021-03-24T21:55:33.174 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0210000000000001e-05 2021-03-24T21:55:33.174 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.174 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.174 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:55:33.174 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:39.115375+0000", 2021-03-24T21:55:33.175 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.9590000000000007e-06 2021-03-24T21:55:33.175 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.175 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.175 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:55:33.176 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:39.115385+0000", 2021-03-24T21:55:33.176 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.7476000000000001e-05 2021-03-24T21:55:33.176 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.176 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.177 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:33.177 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:39.115472+0000", 2021-03-24T21:55:33.177 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.528e-06 2021-03-24T21:55:33.177 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.177 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.178 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op must wait for map", 2021-03-24T21:55:33.178 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:39.115479+0000", 2021-03-24T21:55:33.178 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011015367 2021-03-24T21:55:33.178 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.179 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.179 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:33.179 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:39.126494+0000", 2021-03-24T21:55:33.179 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.3852999999999998e-05 2021-03-24T21:55:33.179 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.180 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.180 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:55:33.180 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:39.126578+0000", 2021-03-24T21:55:33.180 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00025131600000000002 2021-03-24T21:55:33.180 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.181 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.181 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,7", 2021-03-24T21:55:33.181 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:39.126829+0000", 2021-03-24T21:55:33.181 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.013725329 2021-03-24T21:55:33.182 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.182 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.182 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:55:33.182 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:39.140555+0000", 2021-03-24T21:55:33.183 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.9290999999999998e-05 2021-03-24T21:55:33.183 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.183 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.183 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:55:33.184 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:39.140624+0000", 2021-03-24T21:55:33.184 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00032631000000000001 2021-03-24T21:55:33.184 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.184 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.184 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:55:33.185 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:39.140950+0000", 2021-03-24T21:55:33.185 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1861999999999999e-05 2021-03-24T21:55:33.185 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.185 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.186 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:55:33.186 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:39.141002+0000", 2021-03-24T21:55:33.186 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.2543999999999998e-05 2021-03-24T21:55:33.186 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.187 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.187 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:55:33.187 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:39.141045+0000", 2021-03-24T21:55:33.187 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.025697194999999999 2021-03-24T21:55:33.187 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:33.188 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:55:33.188 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:33.188 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.189 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.189 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:265 7.a 7:5326b13c:::smithi12556470-30:head [write 574806~778913] snapc 8=[8] ondisk+write+known_if_redirected e114)", 2021-03-24T21:55:33.189 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:43.172712+0000", 2021-03-24T21:55:33.189 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 49.902624213999999, 2021-03-24T21:55:33.190 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011375601000000001, 2021-03-24T21:55:33.190 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:55:33.190 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:55:33.190 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:55:33.190 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:55:33.191 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:55:33.191 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 265 2021-03-24T21:55:33.191 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.191 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:55:33.192 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.192 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:55:33.192 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.172712+0000", 2021-03-24T21:55:33.192 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:55:33.193 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.193 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.193 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:55:33.193 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.172712+0000", 2021-03-24T21:55:33.194 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999962 2021-03-24T21:55:33.194 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.194 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.194 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:55:33.194 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.172708+0000", 2021-03-24T21:55:33.195 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00053791699999999995 2021-03-24T21:55:33.195 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.195 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.195 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:55:33.195 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.173246+0000", 2021-03-24T21:55:33.196 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.1007999999999999e-05 2021-03-24T21:55:33.196 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.196 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.196 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:55:33.197 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.173277+0000", 2021-03-24T21:55:33.197 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1088e-05 2021-03-24T21:55:33.197 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.197 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.197 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:55:33.198 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.173288+0000", 2021-03-24T21:55:33.198 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.9979e-05 2021-03-24T21:55:33.198 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.198 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.199 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:33.199 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.173378+0000", 2021-03-24T21:55:33.199 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.0230000000000004e-06 2021-03-24T21:55:33.199 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.199 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.200 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op must wait for map", 2021-03-24T21:55:33.200 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.173385+0000", 2021-03-24T21:55:33.200 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002540077 2021-03-24T21:55:33.200 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.200 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.201 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:33.201 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.175926+0000", 2021-03-24T21:55:33.201 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000119024 2021-03-24T21:55:33.201 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.201 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.202 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:55:33.202 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.176045+0000", 2021-03-24T21:55:33.202 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00036657999999999998 2021-03-24T21:55:33.202 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.203 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.203 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,5", 2021-03-24T21:55:33.203 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.176411+0000", 2021-03-24T21:55:33.203 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0028112760000000001 2021-03-24T21:55:33.204 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.204 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.204 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:55:33.204 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.179222+0000", 2021-03-24T21:55:33.205 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001413531 2021-03-24T21:55:33.205 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.205 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.205 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:55:33.205 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.180636+0000", 2021-03-24T21:55:33.206 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0033107779999999999 2021-03-24T21:55:33.206 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.206 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.206 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:55:33.206 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.183947+0000", 2021-03-24T21:55:33.207 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.5611999999999995e-05 2021-03-24T21:55:33.207 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.207 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.207 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:55:33.208 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.184012+0000", 2021-03-24T21:55:33.208 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.5488999999999995e-05 2021-03-24T21:55:33.208 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.208 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.208 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:55:33.209 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.184088+0000", 2021-03-24T21:55:33.209 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011375601000000001 2021-03-24T21:55:33.209 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:33.209 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:55:33.209 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:33.210 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.210 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.210 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:434 7.1 e123/108 7:86662589:::smithi12556470-19:head v 123'25, mlcod=120'21)", 2021-03-24T21:55:33.211 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:52.103706+0000", 2021-03-24T21:55:33.211 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 40.971630556000001, 2021-03-24T21:55:33.211 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.026400651000000001, 2021-03-24T21:55:33.211 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:55:33.211 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:55:33.212 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:55:33.212 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.212 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:55:33.212 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.103706+0000", 2021-03-24T21:55:33.213 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:55:33.213 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.213 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.214 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:55:33.214 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.103706+0000", 2021-03-24T21:55:33.214 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T21:55:33.215 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.215 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.215 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:55:33.215 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.103704+0000", 2021-03-24T21:55:33.216 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2787e-05 2021-03-24T21:55:33.216 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.216 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.216 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:55:33.216 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.103717+0000", 2021-03-24T21:55:33.217 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.024831961 2021-03-24T21:55:33.217 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.217 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.217 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:55:33.217 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.128549+0000", 2021-03-24T21:55:33.218 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0806e-05 2021-03-24T21:55:33.218 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.218 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.218 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:55:33.219 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.128559+0000", 2021-03-24T21:55:33.219 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2800999999999998e-05 2021-03-24T21:55:33.219 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.219 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.220 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:33.220 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.128612+0000", 2021-03-24T21:55:33.220 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.5789999999999999e-05 2021-03-24T21:55:33.220 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.220 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.221 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:55:33.221 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.128638+0000", 2021-03-24T21:55:33.221 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001432589 2021-03-24T21:55:33.221 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.221 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.222 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:55:33.222 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.130071+0000", 2021-03-24T21:55:33.222 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.6034000000000001e-05 2021-03-24T21:55:33.222 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.223 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.223 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:55:33.223 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.130107+0000", 2021-03-24T21:55:33.223 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:33.226 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.026400651000000001 2021-03-24T21:55:33.227 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:33.227 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:55:33.228 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:33.228 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.229 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.229 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:447 7.1 e123/108 7:8eea9237:::smithi12556470-1:head v 123'34, mlcod=123'32)", 2021-03-24T21:55:33.229 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:52.189948+0000", 2021-03-24T21:55:33.230 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 40.885388442, 2021-03-24T21:55:33.230 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011262179000000001, 2021-03-24T21:55:33.231 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:55:33.231 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:55:33.232 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:55:33.232 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.232 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:55:33.233 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.189948+0000", 2021-03-24T21:55:33.233 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:55:33.233 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.234 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.234 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:55:33.234 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.189948+0000", 2021-03-24T21:55:33.234 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T21:55:33.235 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.235 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.235 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:55:33.235 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.189946+0000", 2021-03-24T21:55:33.235 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4346999999999999e-05 2021-03-24T21:55:33.236 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.236 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.236 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:55:33.236 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.190011+0000", 2021-03-24T21:55:33.236 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9397000000000001e-05 2021-03-24T21:55:33.237 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.237 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.237 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:55:33.237 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.190030+0000", 2021-03-24T21:55:33.238 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.6759999999999996e-06 2021-03-24T21:55:33.238 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.238 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.238 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:55:33.238 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.190039+0000", 2021-03-24T21:55:33.239 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5297999999999999e-05 2021-03-24T21:55:33.239 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.239 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.239 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:33.239 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.190084+0000", 2021-03-24T21:55:33.240 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.4278e-05 2021-03-24T21:55:33.240 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.240 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.240 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:55:33.241 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.190108+0000", 2021-03-24T21:55:33.241 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011048519999999999 2021-03-24T21:55:33.241 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.241 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.241 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:55:33.242 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.201157+0000", 2021-03-24T21:55:33.242 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.3312000000000001e-05 2021-03-24T21:55:33.242 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.242 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.242 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:55:33.243 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.201210+0000", 2021-03-24T21:55:33.243 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011262179000000001 2021-03-24T21:55:33.243 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:33.243 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:55:33.244 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:33.244 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.244 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.244 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:535 7.3 7:cb55abc9:::smithi12556470-6:head [copy-from ver 41] snapc 19=[19,17,16,15] ondisk+write+known_if_redirected e131)", 2021-03-24T21:55:33.245 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:55:00.112890+0000", 2021-03-24T21:55:33.245 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 32.962446272000001, 2021-03-24T21:55:33.245 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.025032469000000002, 2021-03-24T21:55:33.245 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:55:33.246 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:55:33.246 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:55:33.246 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:55:33.246 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:55:33.246 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 535 2021-03-24T21:55:33.247 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.247 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:55:33.247 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.247 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:55:33.247 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.112890+0000", 2021-03-24T21:55:33.248 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:55:33.248 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.248 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.248 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:55:33.248 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.112890+0000", 2021-03-24T21:55:33.249 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999967 2021-03-24T21:55:33.249 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.249 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.249 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:55:33.250 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.112887+0000", 2021-03-24T21:55:33.250 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.5398e-05 2021-03-24T21:55:33.250 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.250 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.250 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:55:33.251 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.112912+0000", 2021-03-24T21:55:33.251 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.9357e-05 2021-03-24T21:55:33.251 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.251 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.251 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:55:33.252 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.112942+0000", 2021-03-24T21:55:33.252 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1905e-05 2021-03-24T21:55:33.252 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.252 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.253 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:55:33.253 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.112953+0000", 2021-03-24T21:55:33.253 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00011492 2021-03-24T21:55:33.253 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.253 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.254 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:33.254 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.113068+0000", 2021-03-24T21:55:33.254 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2971e-05 2021-03-24T21:55:33.255 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.255 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.255 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op must wait for map", 2021-03-24T21:55:33.255 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.113081+0000", 2021-03-24T21:55:33.255 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0036808499999999998 2021-03-24T21:55:33.256 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.256 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.256 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:33.256 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.116762+0000", 2021-03-24T21:55:33.256 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.9623000000000007e-05 2021-03-24T21:55:33.257 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.257 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.257 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:55:33.257 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.116842+0000", 2021-03-24T21:55:33.257 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0094099979999999993 2021-03-24T21:55:33.258 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.258 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.258 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,4", 2021-03-24T21:55:33.258 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.126252+0000", 2021-03-24T21:55:33.259 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0077805369999999997 2021-03-24T21:55:33.259 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.259 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.259 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:55:33.259 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.134032+0000", 2021-03-24T21:55:33.260 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001268124 2021-03-24T21:55:33.260 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.260 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.260 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:55:33.260 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.135300+0000", 2021-03-24T21:55:33.261 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0024497189999999999 2021-03-24T21:55:33.261 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.261 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.261 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:55:33.261 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.137750+0000", 2021-03-24T21:55:33.262 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000100466 2021-03-24T21:55:33.262 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.262 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.262 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:55:33.263 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.137851+0000", 2021-03-24T21:55:33.263 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.2012000000000001e-05 2021-03-24T21:55:33.263 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.263 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.263 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:55:33.264 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.137923+0000", 2021-03-24T21:55:33.264 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.025032469000000002 2021-03-24T21:55:33.264 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:33.264 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:55:33.264 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:33.265 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.265 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.265 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:536 7.3 7:cb55abc9:::smithi12556470-6:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e131)", 2021-03-24T21:55:33.266 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:55:00.112976+0000", 2021-03-24T21:55:33.266 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 32.962360052000001, 2021-03-24T21:55:33.266 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.025211872, 2021-03-24T21:55:33.266 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:55:33.266 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T21:55:33.267 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:55:33.267 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:55:33.267 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:55:33.267 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 536 2021-03-24T21:55:33.267 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.268 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:55:33.268 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.268 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:55:33.268 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.112976+0000", 2021-03-24T21:55:33.268 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:55:33.269 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.269 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.269 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:55:33.269 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.112976+0000", 2021-03-24T21:55:33.270 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T21:55:33.270 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.270 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.270 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:55:33.270 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.112975+0000", 2021-03-24T21:55:33.271 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6469999999999997e-06 2021-03-24T21:55:33.271 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.271 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.271 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:55:33.271 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.112980+0000", 2021-03-24T21:55:33.272 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1405e-05 2021-03-24T21:55:33.272 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.272 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.273 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:55:33.273 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.112991+0000", 2021-03-24T21:55:33.273 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.3589999999999999e-06 2021-03-24T21:55:33.274 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.274 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.275 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:55:33.275 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.112997+0000", 2021-03-24T21:55:33.275 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00014866400000000001 2021-03-24T21:55:33.276 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.276 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.276 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:33.277 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.113145+0000", 2021-03-24T21:55:33.277 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.3260000000000003e-06 2021-03-24T21:55:33.277 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.277 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.277 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting_for_map not empty", 2021-03-24T21:55:33.278 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.113151+0000", 2021-03-24T21:55:33.278 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0038143529999999999 2021-03-24T21:55:33.278 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.278 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.278 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:33.279 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.116965+0000", 2021-03-24T21:55:33.279 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.4128000000000003e-05 2021-03-24T21:55:33.279 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.279 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.280 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-24T21:55:33.280 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.116999+0000", 2021-03-24T21:55:33.280 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.016540909999999999 2021-03-24T21:55:33.280 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.280 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.281 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:33.281 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.133540+0000", 2021-03-24T21:55:33.281 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000122211 2021-03-24T21:55:33.281 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.282 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.282 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T21:55:33.283 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.133662+0000", 2021-03-24T21:55:33.283 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0043142550000000003 2021-03-24T21:55:33.283 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.284 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.284 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:33.285 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.137977+0000", 2021-03-24T21:55:33.285 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000107779 2021-03-24T21:55:33.285 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.286 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.286 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:55:33.287 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.138084+0000", 2021-03-24T21:55:33.287 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000103868 2021-03-24T21:55:33.287 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.288 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.288 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:55:33.289 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:00.138188+0000", 2021-03-24T21:55:33.289 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.025211872 2021-03-24T21:55:33.289 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:33.290 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:55:33.290 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:33.290 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.291 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.291 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:634 7.f e140/99 7:f26cf53d:::smithi12556470-37:head v 140'8, mlcod=107'5)", 2021-03-24T21:55:33.292 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:55:09.384324+0000", 2021-03-24T21:55:33.292 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 23.691012773000001, 2021-03-24T21:55:33.292 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011550596, 2021-03-24T21:55:33.292 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:55:33.292 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:55:33.293 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:55:33.293 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.293 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:55:33.293 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:09.384324+0000", 2021-03-24T21:55:33.294 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:55:33.294 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.294 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.294 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:55:33.295 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:09.384324+0000", 2021-03-24T21:55:33.295 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999962 2021-03-24T21:55:33.296 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.296 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.297 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:55:33.297 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:09.384320+0000", 2021-03-24T21:55:33.297 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0035309830000000001 2021-03-24T21:55:33.298 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.298 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.298 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:55:33.298 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:09.387851+0000", 2021-03-24T21:55:33.299 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.3611000000000001e-05 2021-03-24T21:55:33.299 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.299 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.299 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:55:33.299 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:09.387885+0000", 2021-03-24T21:55:33.300 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5537e-05 2021-03-24T21:55:33.300 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.300 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.300 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:55:33.300 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:09.387900+0000", 2021-03-24T21:55:33.301 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.1958000000000003e-05 2021-03-24T21:55:33.301 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.301 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.301 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:33.302 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:09.387992+0000", 2021-03-24T21:55:33.302 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.7053e-05 2021-03-24T21:55:33.302 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.302 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.302 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:55:33.303 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:09.388019+0000", 2021-03-24T21:55:33.303 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0078007559999999998 2021-03-24T21:55:33.303 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.303 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.304 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:55:33.304 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:09.395820+0000", 2021-03-24T21:55:33.304 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.4463999999999999e-05 2021-03-24T21:55:33.304 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.304 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.305 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:55:33.305 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:09.395874+0000", 2021-03-24T21:55:33.305 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011550596 2021-03-24T21:55:33.305 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:33.306 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:55:33.306 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:33.306 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.307 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.307 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:645 7.c 7:305739dc:::smithi12556470-36:head [append 0~0] snapc 23=[23,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e141)", 2021-03-24T21:55:33.307 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:55:10.121969+0000", 2021-03-24T21:55:33.307 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 22.953367885999999, 2021-03-24T21:55:33.307 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.029700717000000001, 2021-03-24T21:55:33.308 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:55:33.308 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:55:33.308 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:55:33.308 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:55:33.308 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:55:33.309 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 645 2021-03-24T21:55:33.309 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.309 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:55:33.309 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.310 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:55:33.310 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.121969+0000", 2021-03-24T21:55:33.310 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:55:33.310 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.310 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.311 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:55:33.311 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.121969+0000", 2021-03-24T21:55:33.311 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T21:55:33.311 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.311 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.312 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:55:33.312 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.121968+0000", 2021-03-24T21:55:33.312 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.7299999999999996e-06 2021-03-24T21:55:33.312 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.312 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.313 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:55:33.313 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.121972+0000", 2021-03-24T21:55:33.313 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2269999999999993e-06 2021-03-24T21:55:33.313 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.314 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.314 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:55:33.314 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.121980+0000", 2021-03-24T21:55:33.314 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.7979999999999996e-06 2021-03-24T21:55:33.314 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.315 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.315 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:55:33.315 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.121985+0000", 2021-03-24T21:55:33.315 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4379000000000001e-05 2021-03-24T21:55:33.315 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.316 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.316 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:33.316 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122050+0000", 2021-03-24T21:55:33.316 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000171507 2021-03-24T21:55:33.316 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.317 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.317 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:55:33.317 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122221+0000", 2021-03-24T21:55:33.317 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00017259999999999999 2021-03-24T21:55:33.317 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.318 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.318 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,7", 2021-03-24T21:55:33.318 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122394+0000", 2021-03-24T21:55:33.318 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012748140999999999 2021-03-24T21:55:33.319 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.319 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.319 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:55:33.319 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.135142+0000", 2021-03-24T21:55:33.319 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.014595599000000001 2021-03-24T21:55:33.320 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.320 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.320 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:55:33.320 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.149737+0000", 2021-03-24T21:55:33.320 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0017735170000000001 2021-03-24T21:55:33.321 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.321 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.321 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:55:33.321 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.151511+0000", 2021-03-24T21:55:33.321 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.3142999999999999e-05 2021-03-24T21:55:33.322 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.322 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.322 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:55:33.322 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.151604+0000", 2021-03-24T21:55:33.323 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.5141000000000005e-05 2021-03-24T21:55:33.323 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.323 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.323 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:55:33.323 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.151669+0000", 2021-03-24T21:55:33.324 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.029700717000000001 2021-03-24T21:55:33.324 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:33.324 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:55:33.324 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:33.324 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.325 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.325 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:646 7.c 7:305739dc:::smithi12556470-36:head [rollback 15] snapc 23=[23,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e141)", 2021-03-24T21:55:33.325 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:55:10.122086+0000", 2021-03-24T21:55:33.325 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 22.953250217000001, 2021-03-24T21:55:33.325 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.030078549999999999, 2021-03-24T21:55:33.326 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:55:33.326 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:55:33.326 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:55:33.326 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:55:33.327 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:55:33.327 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 646 2021-03-24T21:55:33.327 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.327 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:55:33.327 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.328 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:55:33.328 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122086+0000", 2021-03-24T21:55:33.328 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:55:33.328 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.328 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.329 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:55:33.329 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122086+0000", 2021-03-24T21:55:33.329 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T21:55:33.329 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.329 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.330 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:55:33.330 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122085+0000", 2021-03-24T21:55:33.330 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6999999999999996e-06 2021-03-24T21:55:33.330 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.331 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.331 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:55:33.331 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122091+0000", 2021-03-24T21:55:33.331 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.149e-06 2021-03-24T21:55:33.331 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.332 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.332 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:55:33.332 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122100+0000", 2021-03-24T21:55:33.332 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.4719999999999998e-06 2021-03-24T21:55:33.332 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.333 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.333 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:55:33.333 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122108+0000", 2021-03-24T21:55:33.333 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00083429299999999995 2021-03-24T21:55:33.333 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.334 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.334 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:33.334 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122942+0000", 2021-03-24T21:55:33.334 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000113615 2021-03-24T21:55:33.335 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.335 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.335 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:55:33.335 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.123055+0000", 2021-03-24T21:55:33.335 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00024782800000000001 2021-03-24T21:55:33.336 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.336 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.336 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,7", 2021-03-24T21:55:33.336 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.123303+0000", 2021-03-24T21:55:33.336 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012182396 2021-03-24T21:55:33.337 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.337 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.337 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:55:33.337 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.135486+0000", 2021-03-24T21:55:33.337 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.016421525999999999 2021-03-24T21:55:33.338 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.338 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.338 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:55:33.338 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.151907+0000", 2021-03-24T21:55:33.339 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.3577000000000003e-05 2021-03-24T21:55:33.339 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.339 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.339 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:55:33.339 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.151991+0000", 2021-03-24T21:55:33.340 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.6976000000000001e-05 2021-03-24T21:55:33.340 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.340 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.340 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:55:33.340 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.152088+0000", 2021-03-24T21:55:33.341 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.7041999999999999e-05 2021-03-24T21:55:33.341 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.341 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.341 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:55:33.341 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.152165+0000", 2021-03-24T21:55:33.342 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.030078549999999999 2021-03-24T21:55:33.342 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:33.342 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:55:33.342 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:33.342 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.343 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.343 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:647 7.c 7:305739dc:::smithi12556470-36:head [append 0~0] snapc 23=[23,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e141)", 2021-03-24T21:55:33.343 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:55:10.122139+0000", 2021-03-24T21:55:33.343 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 22.953197876000001, 2021-03-24T21:55:33.343 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.030182674, 2021-03-24T21:55:33.344 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:55:33.344 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:55:33.344 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:55:33.344 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:55:33.345 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:55:33.345 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 647 2021-03-24T21:55:33.345 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.345 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:55:33.345 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.346 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:55:33.346 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122139+0000", 2021-03-24T21:55:33.346 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:55:33.346 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.346 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.347 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:55:33.347 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122139+0000", 2021-03-24T21:55:33.347 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T21:55:33.347 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.347 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.348 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:55:33.348 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122138+0000", 2021-03-24T21:55:33.348 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.0409999999999999e-06 2021-03-24T21:55:33.348 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.349 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.349 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:55:33.349 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122142+0000", 2021-03-24T21:55:33.349 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9560000000000002e-06 2021-03-24T21:55:33.349 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.350 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.350 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:55:33.350 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122148+0000", 2021-03-24T21:55:33.350 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5709999999999999e-06 2021-03-24T21:55:33.350 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.351 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.351 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:55:33.351 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122152+0000", 2021-03-24T21:55:33.351 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0015341910000000001 2021-03-24T21:55:33.351 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.352 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.352 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:33.352 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.123686+0000", 2021-03-24T21:55:33.352 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000308213 2021-03-24T21:55:33.352 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.353 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.353 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:55:33.353 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.123995+0000", 2021-03-24T21:55:33.353 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00017020199999999999 2021-03-24T21:55:33.353 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.354 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.354 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,7", 2021-03-24T21:55:33.354 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124165+0000", 2021-03-24T21:55:33.354 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011385657 2021-03-24T21:55:33.355 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.355 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.355 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:55:33.355 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.135551+0000", 2021-03-24T21:55:33.355 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.016412598 2021-03-24T21:55:33.356 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.356 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.356 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:55:33.356 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.151963+0000", 2021-03-24T21:55:33.356 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00021537100000000001 2021-03-24T21:55:33.357 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.357 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.357 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:55:33.357 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.152179+0000", 2021-03-24T21:55:33.357 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.0764000000000001e-05 2021-03-24T21:55:33.358 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.358 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.358 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:55:33.358 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.152259+0000", 2021-03-24T21:55:33.359 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.1952999999999995e-05 2021-03-24T21:55:33.359 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.359 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.359 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:55:33.359 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.152321+0000", 2021-03-24T21:55:33.360 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.030182674 2021-03-24T21:55:33.360 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:33.360 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:55:33.360 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:33.360 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.361 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.361 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:639 7.7 e141/108 7:ef6d6bee:::smithi12556470-24:head v 141'70, mlcod=141'68)", 2021-03-24T21:55:33.361 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:55:10.122763+0000", 2021-03-24T21:55:33.361 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 22.952573245, 2021-03-24T21:55:33.361 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.029224018000000001, 2021-03-24T21:55:33.362 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:55:33.362 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:55:33.362 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:55:33.362 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.363 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:55:33.363 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122763+0000", 2021-03-24T21:55:33.363 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:55:33.363 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.363 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.364 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:55:33.364 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122763+0000", 2021-03-24T21:55:33.364 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T21:55:33.364 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.364 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.365 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:55:33.365 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122762+0000", 2021-03-24T21:55:33.365 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00019497300000000001 2021-03-24T21:55:33.365 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.365 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.366 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:55:33.366 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122957+0000", 2021-03-24T21:55:33.366 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2621e-05 2021-03-24T21:55:33.366 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.366 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.367 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:55:33.367 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122969+0000", 2021-03-24T21:55:33.367 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.1149999999999994e-06 2021-03-24T21:55:33.367 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.368 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.368 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:55:33.368 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122978+0000", 2021-03-24T21:55:33.368 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.1203000000000003e-05 2021-03-24T21:55:33.368 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.369 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.369 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:33.369 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.123039+0000", 2021-03-24T21:55:33.369 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.6356000000000001e-05 2021-03-24T21:55:33.369 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.370 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.370 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:55:33.370 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.123055+0000", 2021-03-24T21:55:33.370 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.028884112 2021-03-24T21:55:33.370 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.371 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.371 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:55:33.371 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.151939+0000", 2021-03-24T21:55:33.371 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.8056999999999997e-05 2021-03-24T21:55:33.371 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.372 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.372 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:55:33.372 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.151987+0000", 2021-03-24T21:55:33.372 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.029224018000000001 2021-03-24T21:55:33.373 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:33.373 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:55:33.373 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:33.373 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.373 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.374 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:640 7.7 e141/108 7:ef6d6bee:::smithi12556470-24:head v 141'71, mlcod=141'68)", 2021-03-24T21:55:33.374 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:55:10.123846+0000", 2021-03-24T21:55:33.374 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 22.951490954, 2021-03-24T21:55:33.374 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.028164842999999998, 2021-03-24T21:55:33.374 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:55:33.375 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:55:33.375 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:55:33.375 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.375 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:55:33.378 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.123846+0000", 2021-03-24T21:55:33.378 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:55:33.378 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.378 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.379 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:55:33.379 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.123846+0000", 2021-03-24T21:55:33.379 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T21:55:33.379 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.379 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.380 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:55:33.380 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.123844+0000", 2021-03-24T21:55:33.380 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7825e-05 2021-03-24T21:55:33.380 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.380 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.381 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:55:33.381 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.123862+0000", 2021-03-24T21:55:33.381 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.4489999999999999e-06 2021-03-24T21:55:33.381 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.381 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.382 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:55:33.382 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.123870+0000", 2021-03-24T21:55:33.382 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1789999999999999e-06 2021-03-24T21:55:33.382 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.382 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.383 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:55:33.383 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.123876+0000", 2021-03-24T21:55:33.383 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6403999999999997e-05 2021-03-24T21:55:33.383 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.383 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.384 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:33.384 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.123932+0000", 2021-03-24T21:55:33.384 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9221000000000001e-05 2021-03-24T21:55:33.384 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.385 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.385 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:55:33.385 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.123951+0000", 2021-03-24T21:55:33.385 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.028040790999999999 2021-03-24T21:55:33.385 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.386 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.386 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:55:33.386 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.151992+0000", 2021-03-24T21:55:33.386 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.8314000000000001e-05 2021-03-24T21:55:33.386 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.387 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.387 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:55:33.387 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.152010+0000", 2021-03-24T21:55:33.387 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.028164842999999998 2021-03-24T21:55:33.387 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:33.388 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:55:33.388 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:33.388 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.388 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.388 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:871 7.2 7:454f21f6:::smithi12556470-8:head [append 0~0] snapc 36=[36,32,2e] ondisk+write+known_if_redirected e160)", 2021-03-24T21:55:33.389 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:55:29.283295+0000", 2021-03-24T21:55:33.389 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 3.7920414139999998, 2021-03-24T21:55:33.389 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012438489, 2021-03-24T21:55:33.389 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:55:33.389 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:55:33.390 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:55:33.390 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:55:33.390 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:55:33.390 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 871 2021-03-24T21:55:33.391 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.391 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:55:33.391 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.391 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:55:33.391 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:29.283295+0000", 2021-03-24T21:55:33.392 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:55:33.392 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.392 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.392 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:55:33.392 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:29.283295+0000", 2021-03-24T21:55:33.393 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999938 2021-03-24T21:55:33.393 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.393 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.393 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:55:33.393 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:29.283289+0000", 2021-03-24T21:55:33.394 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.1761000000000001e-05 2021-03-24T21:55:33.394 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.394 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.394 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:55:33.394 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:29.283311+0000", 2021-03-24T21:55:33.395 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.3769999999999997e-05 2021-03-24T21:55:33.395 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.395 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.395 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:55:33.396 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:29.283344+0000", 2021-03-24T21:55:33.396 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.4549000000000001e-05 2021-03-24T21:55:33.396 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.396 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.396 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:55:33.397 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:29.283359+0000", 2021-03-24T21:55:33.397 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00016207299999999999 2021-03-24T21:55:33.397 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.397 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.397 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:33.398 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:29.283521+0000", 2021-03-24T21:55:33.398 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2985e-05 2021-03-24T21:55:33.398 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.398 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.398 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op must wait for map", 2021-03-24T21:55:33.399 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:29.283534+0000", 2021-03-24T21:55:33.399 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.005730331 2021-03-24T21:55:33.399 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.399 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.399 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:33.400 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:29.289264+0000", 2021-03-24T21:55:33.400 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000130178 2021-03-24T21:55:33.400 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.400 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.400 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:55:33.401 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:29.289395+0000", 2021-03-24T21:55:33.401 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00026406200000000002 2021-03-24T21:55:33.401 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.401 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.401 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,7", 2021-03-24T21:55:33.402 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:29.289659+0000", 2021-03-24T21:55:33.402 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0058322479999999999 2021-03-24T21:55:33.402 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.402 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.403 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:55:33.403 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:29.295491+0000", 2021-03-24T21:55:33.403 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5411000000000003e-05 2021-03-24T21:55:33.403 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.403 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.404 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:55:33.404 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:29.295536+0000", 2021-03-24T21:55:33.404 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.6691000000000001e-05 2021-03-24T21:55:33.404 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.404 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.405 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:55:33.405 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:29.295623+0000", 2021-03-24T21:55:33.405 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.4311000000000002e-05 2021-03-24T21:55:33.405 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.405 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.406 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:55:33.406 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:29.295677+0000", 2021-03-24T21:55:33.406 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6254e-05 2021-03-24T21:55:33.406 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.407 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.407 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:55:33.407 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:29.295734+0000", 2021-03-24T21:55:33.407 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012438489 2021-03-24T21:55:33.407 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:33.408 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:55:33.408 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:33.408 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.408 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.408 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:872 7.2 7:454f21f6:::smithi12556470-8:head [rollback 2e] snapc 36=[36,32,2e] ondisk+write+known_if_redirected e160)", 2021-03-24T21:55:33.409 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:55:29.283442+0000", 2021-03-24T21:55:33.409 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 3.791894557, 2021-03-24T21:55:33.409 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012462974999999999, 2021-03-24T21:55:33.409 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:55:33.409 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:55:33.410 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:55:33.410 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:55:33.410 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:55:33.410 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 872 2021-03-24T21:55:33.410 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.411 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:55:33.411 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.411 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:55:33.411 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:29.283442+0000", 2021-03-24T21:55:33.412 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:55:33.412 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.412 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.412 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:55:33.412 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:29.283442+0000", 2021-03-24T21:55:33.413 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T21:55:33.413 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.413 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.413 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:55:33.413 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:29.283440+0000", 2021-03-24T21:55:33.414 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.5460999999999999e-05 2021-03-24T21:55:33.414 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.414 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.414 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:55:33.414 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:29.283466+0000", 2021-03-24T21:55:33.415 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1938e-05 2021-03-24T21:55:33.415 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.415 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.415 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:55:33.415 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:29.283478+0000", 2021-03-24T21:55:33.416 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.9868e-05 2021-03-24T21:55:33.416 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.416 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.416 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:55:33.416 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:29.283577+0000", 2021-03-24T21:55:33.417 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000120962 2021-03-24T21:55:33.417 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.417 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.417 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:33.417 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:29.283698+0000", 2021-03-24T21:55:33.418 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5687000000000001e-05 2021-03-24T21:55:33.418 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.418 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.418 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting_for_map not empty", 2021-03-24T21:55:33.419 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:29.283714+0000", 2021-03-24T21:55:33.419 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0065175470000000003 2021-03-24T21:55:33.419 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.419 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.419 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:33.420 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:29.290232+0000", 2021-03-24T21:55:33.420 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.6569000000000005e-05 2021-03-24T21:55:33.420 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.420 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.420 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:55:33.421 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:29.290308+0000", 2021-03-24T21:55:33.421 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00027605199999999999 2021-03-24T21:55:33.421 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.421 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.421 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,7", 2021-03-24T21:55:33.422 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:29.290584+0000", 2021-03-24T21:55:33.422 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0049602279999999997 2021-03-24T21:55:33.422 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.422 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.422 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:55:33.423 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:29.295544+0000", 2021-03-24T21:55:33.423 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9635000000000003e-05 2021-03-24T21:55:33.423 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.423 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.424 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:55:33.424 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:29.295584+0000", 2021-03-24T21:55:33.424 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00020938 2021-03-24T21:55:33.424 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.424 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.425 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:55:33.425 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:29.295793+0000", 2021-03-24T21:55:33.425 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6416000000000003e-05 2021-03-24T21:55:33.425 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.425 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.426 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:55:33.426 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:29.295850+0000", 2021-03-24T21:55:33.426 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5025000000000003e-05 2021-03-24T21:55:33.426 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.426 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.427 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:55:33.427 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:29.295905+0000", 2021-03-24T21:55:33.427 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012462974999999999 2021-03-24T21:55:33.427 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:33.427 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:55:33.428 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:33.428 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.428 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.428 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:873 7.2 7:454f21f6:::smithi12556470-8:head [append 0~0] snapc 36=[36,32,2e] ondisk+write+known_if_redirected e160)", 2021-03-24T21:55:33.428 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:55:29.283730+0000", 2021-03-24T21:55:33.429 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 3.7916065900000002, 2021-03-24T21:55:33.429 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.013019418, 2021-03-24T21:55:33.429 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:55:33.429 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:55:33.430 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:55:33.430 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:55:33.430 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:55:33.430 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 873 2021-03-24T21:55:33.430 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.431 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:55:33.431 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.431 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:55:33.431 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:29.283730+0000", 2021-03-24T21:55:33.431 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:55:33.432 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.432 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.432 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:55:33.432 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:29.283730+0000", 2021-03-24T21:55:33.432 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T21:55:33.433 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.433 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.433 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:55:33.433 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:29.283728+0000", 2021-03-24T21:55:33.433 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.7615999999999997e-05 2021-03-24T21:55:33.434 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.434 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.434 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:55:33.434 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:29.283805+0000", 2021-03-24T21:55:33.435 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.4688000000000001e-05 2021-03-24T21:55:33.435 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.435 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.435 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:55:33.435 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:29.283820+0000", 2021-03-24T21:55:33.436 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.1869999999999997e-06 2021-03-24T21:55:33.436 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.436 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.436 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:55:33.436 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:29.283828+0000", 2021-03-24T21:55:33.437 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.9241999999999996e-05 2021-03-24T21:55:33.437 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.437 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.437 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:33.437 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:29.283928+0000", 2021-03-24T21:55:33.438 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1049999999999999e-05 2021-03-24T21:55:33.438 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.438 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.438 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting_for_map not empty", 2021-03-24T21:55:33.438 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:29.283939+0000", 2021-03-24T21:55:33.439 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011019475000000001 2021-03-24T21:55:33.439 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.439 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.439 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:33.439 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:29.294958+0000", 2021-03-24T21:55:33.440 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.4825000000000001e-05 2021-03-24T21:55:33.440 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.440 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.440 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:55:33.440 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:29.295053+0000", 2021-03-24T21:55:33.441 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000181256 2021-03-24T21:55:33.441 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.441 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.441 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,7", 2021-03-24T21:55:33.442 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:29.295234+0000", 2021-03-24T21:55:33.442 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001031438 2021-03-24T21:55:33.442 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.442 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.442 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:55:33.443 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:29.296266+0000", 2021-03-24T21:55:33.443 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00018317999999999999 2021-03-24T21:55:33.443 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.443 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.443 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:55:33.444 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:29.296449+0000", 2021-03-24T21:55:33.444 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00021558499999999999 2021-03-24T21:55:33.444 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.444 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.444 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:55:33.445 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:29.296664+0000", 2021-03-24T21:55:33.445 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.6736999999999997e-05 2021-03-24T21:55:33.445 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.445 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.446 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:55:33.446 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:29.296701+0000", 2021-03-24T21:55:33.446 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.8180999999999998e-05 2021-03-24T21:55:33.446 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.446 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.447 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:55:33.447 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:29.296749+0000", 2021-03-24T21:55:33.447 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.013019418 2021-03-24T21:55:33.447 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:33.447 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:55:33.448 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:33.448 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.448 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.448 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:878 7.3 7:c1eb355b:::smithi12556470-34:head [rollback 2e] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e162)", 2021-03-24T21:55:33.448 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:55:31.104482+0000", 2021-03-24T21:55:33.449 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 1.970854398, 2021-03-24T21:55:33.449 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.016459621000000001, 2021-03-24T21:55:33.449 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:55:33.449 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T21:55:33.449 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:55:33.450 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:55:33.450 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:55:33.450 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 878 2021-03-24T21:55:33.450 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.451 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:55:33.451 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.451 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:55:33.451 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:31.104482+0000", 2021-03-24T21:55:33.451 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:55:33.452 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.452 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.452 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:55:33.452 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:31.104482+0000", 2021-03-24T21:55:33.452 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999957 2021-03-24T21:55:33.453 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.453 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.453 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:55:33.453 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:31.104478+0000", 2021-03-24T21:55:33.453 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.8773000000000001e-05 2021-03-24T21:55:33.454 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.454 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.454 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:55:33.454 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:31.104496+0000", 2021-03-24T21:55:33.454 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.4781000000000003e-05 2021-03-24T21:55:33.455 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.455 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.455 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:55:33.455 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:31.104551+0000", 2021-03-24T21:55:33.455 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5074e-05 2021-03-24T21:55:33.456 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.456 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.456 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:55:33.456 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:31.104566+0000", 2021-03-24T21:55:33.457 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.8064000000000003e-05 2021-03-24T21:55:33.457 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.457 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.457 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:33.457 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:31.104654+0000", 2021-03-24T21:55:33.458 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5296000000000001e-05 2021-03-24T21:55:33.458 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.458 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.458 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op must wait for map", 2021-03-24T21:55:33.458 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:31.104670+0000", 2021-03-24T21:55:33.459 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0048406339999999999 2021-03-24T21:55:33.459 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.459 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.459 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:55:33.459 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:31.109510+0000", 2021-03-24T21:55:33.460 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00013554999999999999 2021-03-24T21:55:33.460 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.460 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.460 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:55:33.461 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:31.109646+0000", 2021-03-24T21:55:33.461 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011295819 2021-03-24T21:55:33.461 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:55:33.461 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:55:33.461 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:55:33.462 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:31.120942+0000", 2021-03-24T21:55:33.462 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.016459621000000001 2021-03-24T21:55:33.462 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:33.462 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:55:33.462 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:33.463 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:55:33.463 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:55:33.463 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-24T21:55:33.464 INFO:tasks.rados.rados.0.smithi125.stdout:320: done (1 left) 2021-03-24T21:55:33.464 INFO:tasks.rados.rados.0.smithi125.stdout:321: delete oid 35 current snap is 29 2021-03-24T21:55:33.465 INFO:tasks.rados.rados.0.smithi125.stdout:318: expect (ObjNum 121 snap 23 seq_num 121) 2021-03-24T21:55:33.465 INFO:teuthology.orchestra.run.smithi125.stderr:nodeep-scrub is unset 2021-03-24T21:55:33.465 INFO:tasks.rados.rados.0.smithi125.stdout:318: done (1 left) 2021-03-24T21:55:33.466 INFO:tasks.rados.rados.0.smithi125.stdout:321: done (0 left) 2021-03-24T21:55:33.466 INFO:tasks.rados.rados.0.smithi125.stdout:322: snap_create 2021-03-24T21:55:33.476 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:33 smithi125 conmon[19936]: cluster 2021-03-24T21:55: 2021-03-24T21:55:33.476 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:33 smithi125 conmon[19936]: 31.912442+0000 osd.7 (osd.7) 37 : cluster [DBG] 3.11 scrub ok 2021-03-24T21:55:33.477 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:33 smithi125 conmon[19936]: cluster 2021-03-24T21:55:33.477 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:33 smithi125 conmon[19936]: 2021-03-24T21:55:32.214486+0000 2021-03-24T21:55:33.477 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:33 smithi125 conmon[19936]: osd.0 (osd.0) 46 2021-03-24T21:55:33.477 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:33 smithi125 conmon[19936]: : cluster [DBG] 4.1f scrub ok 2021-03-24T21:55:33.477 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:33 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:55:33.478 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:33 smithi125 conmon[19936]: 03-24T21:55:32. 2021-03-24T21:55:33.478 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:33 smithi125 conmon[19936]: 566739+0000 osd.3 (osd.3 2021-03-24T21:55:33.478 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:33 smithi125 conmon[19936]: ) 52 : cluster [DBG] 3.18 scrub ok 2021-03-24T21:55:33.479 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:33 smithi125 conmon[19936]: audit 2021-03-24T21:55:33.479 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:33 smithi125 conmon[19936]: 2021-03-24T21:55:33.479 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:33 smithi125 conmon[19936]: 2021 2021-03-24T21:55:33.479 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:33 smithi125 conmon[19936]: -03-24T21:55:33. 2021-03-24T21:55:33.479 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:33 smithi125 conmon[19936]: 189852+0000 mon.a (mon.0) 736 : audit [INF] from='client.? v1:172.21.15.125:0/1943451891' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:55:33.480 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:33 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:55:33.480 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:33 smithi125 conmon[24927]: T21:55:31.912442+0000 osd.7 (osd.7) 37 : cluster [DBG] 3.11 scrub ok 2021-03-24T21:55:33.480 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:33 smithi125 conmon[24927]: cluster 2021-03-24T21:55:33.481 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:33 smithi125 conmon[24927]: 2021-03-24 2021-03-24T21:55:33.481 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:33 smithi125 conmon[24927]: T21:55:32.214486+0000 osd.0 2021-03-24T21:55:33.481 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:33 smithi125 conmon[24927]: (osd.0) 46 : cluster [DBG] 2021-03-24T21:55:33.481 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:33 smithi125 conmon[24927]: 4.1f scrub ok 2021-03-24T21:55:33.482 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:33 smithi125 conmon[24927]: cluster 2021-03-24T21:55:33.482 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:33 smithi125 conmon[24927]: 2021-03-24T21:55:33.482 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:33 smithi125 conmon[24927]: 2021 2021-03-24T21:55:33.482 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:33 smithi125 conmon[24927]: - 2021-03-24T21:55:33.482 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:33 smithi125 conmon[24927]: 03 2021-03-24T21:55:33.483 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:33 smithi125 conmon[24927]: - 2021-03-24T21:55:33.483 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:33 smithi125 conmon[24927]: 24 2021-03-24T21:55:33.483 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:33 smithi125 conmon[24927]: T 2021-03-24T21:55:33.483 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:33 smithi125 conmon[24927]: 21 2021-03-24T21:55:33.483 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:33 smithi125 conmon[24927]: : 2021-03-24T21:55:33.484 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:33 smithi125 conmon[24927]: 55 2021-03-24T21:55:33.484 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:33 smithi125 conmon[24927]: : 2021-03-24T21:55:33.484 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:33 smithi125 conmon[24927]: 32 2021-03-24T21:55:33.484 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:33 smithi125 conmon[24927]: . 2021-03-24T21:55:33.484 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:33 smithi125 conmon[24927]: 566739 2021-03-24T21:55:33.485 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:33 smithi125 conmon[24927]: +0000 2021-03-24T21:55:33.485 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:33 smithi125 conmon[24927]: osd.3 (osd.3) 2021-03-24T21:55:33.485 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:33 smithi125 conmon[24927]: 52 : cluster [DBG] 3.18 scrub ok 2021-03-24T21:55:33.485 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:33 smithi125 conmon[24927]: audit 2021 2021-03-24T21:55:33.486 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:33 smithi125 conmon[24927]: -03-24T21:55:33 2021-03-24T21:55:33.486 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:33 smithi125 conmon[24927]: .189852+0000 mon.a (mon. 2021-03-24T21:55:33.486 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:33 smithi125 conmon[24927]: 0) 736 : audit [INF] from='client.? v1:172.21.15.125:0/1943451891' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:55:33.486 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:33 smithi125 conmon[24927]: 2021-03-24T21:55:33.518 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:33.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:33.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:55:33.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:33 smithi142 conmon[20125]: cluster 2021-03-24T21:55:33.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:33 smithi142 conmon[20125]: 2021-03-24T21 2021-03-24T21:55:33.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:33 smithi142 conmon[20125]: :55:31.912442+0000 osd.7 (osd.7) 37 : cluster [DBG] 3.11 scrub ok 2021-03-24T21:55:33.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:33 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:55:33.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:33 smithi142 conmon[20125]: 24T21:55:32.214486+0000 osd.0 ( 2021-03-24T21:55:33.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:33 smithi142 conmon[20125]: osd.0) 46 : cluster [DBG] 4.1f scrub ok 2021-03-24T21:55:33.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:33 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:55:33.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:33 smithi142 conmon[20125]: -24T21:55:32.566739+0000 2021-03-24T21:55:33.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:33 smithi142 conmon[20125]: osd.3 (osd.3) 52 : cluster [DBG] 2021-03-24T21:55:33.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:33 smithi142 conmon[20125]: 3.18 scrub ok 2021-03-24T21:55:33.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:33 smithi142 conmon[20125]: audit 2021-03-24T21:55: 2021-03-24T21:55:33.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:33 smithi142 conmon[20125]: 33.189852+0000 mon.a (mon.0) 736 2021-03-24T21:55:33.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:33 smithi142 conmon[20125]: : audit [INF] from='client.? v1:172.21.15.125:0/1943451891' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:55:33.624 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:33.667 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-24T21:55:33.678 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:33.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:33.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:55:33.785 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:33.834 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:33.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:33.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:55:33.945 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:33.996 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:34.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:34.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:55:34.103 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:34.158 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:34.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:34.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:55:34.264 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:34.325 INFO:tasks.rados.rados.0.smithi125.stdout:322: done (0 left) 2021-03-24T21:55:34.325 INFO:tasks.rados.rados.0.smithi125.stdout:323: read oid 12 snap 29 2021-03-24T21:55:34.325 INFO:tasks.rados.rados.0.smithi125.stdout:323: expect deleted 2021-03-24T21:55:34.326 INFO:tasks.rados.rados.0.smithi125.stdout:324: read oid 35 snap -1 2021-03-24T21:55:34.326 INFO:tasks.rados.rados.0.smithi125.stdout:324: expect deleted 2021-03-24T21:55:34.326 INFO:tasks.rados.rados.0.smithi125.stdout:325: read oid 19 snap 23 2021-03-24T21:55:34.326 INFO:tasks.rados.rados.0.smithi125.stdout:325: expect (ObjNum 9 snap 0 seq_num 9) 2021-03-24T21:55:34.327 INFO:tasks.rados.rados.0.smithi125.stdout:326: read oid 24 snap -1 2021-03-24T21:55:34.327 INFO:tasks.rados.rados.0.smithi125.stdout:326: expect (ObjNum 106 snap 21 seq_num 106) 2021-03-24T21:55:34.327 INFO:tasks.rados.rados.0.smithi125.stdout:327: read oid 40 snap -1 2021-03-24T21:55:34.327 INFO:tasks.rados.rados.0.smithi125.stdout:327: expect (ObjNum 132 snap 24 seq_num 132) 2021-03-24T21:55:34.328 INFO:tasks.rados.rados.0.smithi125.stdout:328: write oid 28 current snap is 30 2021-03-24T21:55:34.328 INFO:tasks.rados.rados.0.smithi125.stdout:328: seq_num 147 ranges {470629=472310,1418853=513377} 2021-03-24T21:55:34.329 INFO:tasks.rados.rados.0.smithi125.stdout:328: writing smithi12556470-28 from 470629 to 942939 tid 1 2021-03-24T21:55:34.330 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:34.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:34.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:55:34.336 INFO:tasks.rados.rados.0.smithi125.stdout:328: writing smithi12556470-28 from 1418853 to 1932230 tid 2 2021-03-24T21:55:34.337 INFO:tasks.rados.rados.0.smithi125.stdout:329: read oid 26 snap 23 2021-03-24T21:55:34.337 INFO:tasks.rados.rados.0.smithi125.stdout:329: expect (ObjNum 25 snap 0 seq_num 25) 2021-03-24T21:55:34.337 INFO:tasks.rados.rados.0.smithi125.stdout:330: copy_from oid 17 from oid 44 current snap is 30 2021-03-24T21:55:34.337 INFO:tasks.rados.rados.0.smithi125.stdout:323: done (7 left) 2021-03-24T21:55:34.338 INFO:tasks.rados.rados.0.smithi125.stdout:324: done (6 left) 2021-03-24T21:55:34.338 INFO:tasks.rados.rados.0.smithi125.stdout:331: write oid 10 current snap is 30 2021-03-24T21:55:34.338 INFO:tasks.rados.rados.0.smithi125.stdout:331: seq_num 148 ranges {487436=547559,1760982=657035} 2021-03-24T21:55:34.343 INFO:tasks.rados.rados.0.smithi125.stdout:331: writing smithi12556470-10 from 487436 to 1034995 tid 1 2021-03-24T21:55:34.352 INFO:tasks.rados.rados.0.smithi125.stdout:331: writing smithi12556470-10 from 1760982 to 2418017 tid 2 2021-03-24T21:55:34.352 INFO:tasks.rados.rados.0.smithi125.stdout:332: write (excl) oid 3 current snap is 30 2021-03-24T21:55:34.352 INFO:tasks.rados.rados.0.smithi125.stdout:332: seq_num 149 ranges {504243=622808,1819464=517046} 2021-03-24T21:55:34.358 INFO:tasks.rados.rados.0.smithi125.stdout:332: writing smithi12556470-3 from 504243 to 1127051 tid 1 2021-03-24T21:55:34.366 INFO:tasks.rados.rados.0.smithi125.stdout:332: writing smithi12556470-3 from 1819464 to 2336510 tid 2 2021-03-24T21:55:34.366 INFO:tasks.rados.rados.0.smithi125.stdout:333: read oid 2 snap 29 2021-03-24T21:55:34.366 INFO:tasks.rados.rados.0.smithi125.stdout:333: expect deleted 2021-03-24T21:55:34.366 INFO:tasks.rados.rados.0.smithi125.stdout:334: rollback oid 33 current snap is 30 2021-03-24T21:55:34.367 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 33 to 29 2021-03-24T21:55:34.367 INFO:tasks.rados.rados.0.smithi125.stdout:325: expect (ObjNum 9 snap 0 seq_num 9) 2021-03-24T21:55:34.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:34 smithi142 conmon[20125]: cluster 2021-03-24T21:55:32.729407+0000 mgr.y (mgr.14144) 340 : cluster [DBG] pgmap v407: 148 pgs: 148 active+clean; 123 MiB data, 358 MiB used, 536 GiB / 536 GiB avail; 1.6 MiB/s rd, 197 KiB/s wr, 6 op/s 2021-03-24T21:55:34.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:34 smithi142 conmon[20125]: cluster 2021-03-24T21:55:33.246194+0000 osd.0 (osd.0) 47 : cluster [DBG] 6.9 scrub ok 2021-03-24T21:55:34.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:34 smithi142 conmon[20125]: cluster 2021-03-24T21:55:33.292045+0000 mon.a (mon.0) 737 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T21:55:34.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:34 smithi142 conmon[20125]: cluster 2021-03-24T21:55:33.292110+0000 mon.a (mon.0) 738 : cluster [INF] Cluster is now healthy 2021-03-24T21:55:34.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:34 smithi142 conmon[20125]: audit 2021-03-24T21:55:33.310617+0000 mon.a (mon.0) 739 : audit [INF] from='client.? v1:172.21.15.125:0/1943451891' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T21:55:34.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:34 smithi142 conmon[20125]: cluster 2021-03-24T21:55:33.310694+0000 mon.a (mon.0) 740 : cluster [DBG] osdmap e164: 8 total, 8 up, 6 in 2021-03-24T21:55:34.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:34 smithi142 conmon[20125]: cluster 2021-03-24T21:55:33.533831+0000 osd.3 ( 2021-03-24T21:55:34.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:34 smithi142 conmon[20125]: osd.3) 53 : cluster [DBG] 3.2 scrub ok 2021-03-24T21:55:34.437 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:34.441 INFO:tasks.rados.rados.0.smithi125.stdout:328: finishing write tid 1 to smithi12556470-28 2021-03-24T21:55:34.442 INFO:tasks.rados.rados.0.smithi125.stdout:328: finishing write tid 2 to smithi12556470-28 2021-03-24T21:55:34.442 INFO:tasks.rados.rados.0.smithi125.stdout:328: finishing write tid 3 to smithi12556470-28 2021-03-24T21:55:34.442 INFO:tasks.rados.rados.0.smithi125.stdout:327: expect (ObjNum 132 snap 24 seq_num 132) 2021-03-24T21:55:34.490 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:34.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:34.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:34.544 INFO:tasks.rados.rados.0.smithi125.stdout:328: finishing write tid 4 to smithi12556470-28 2021-03-24T21:55:34.545 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 28 v 95 (ObjNum 147 snap 30 seq_num 147) dirty exists 2021-03-24T21:55:34.545 INFO:tasks.rados.rados.0.smithi125.stdout:328: left oid 28 (ObjNum 147 snap 30 seq_num 147) 2021-03-24T21:55:34.545 INFO:tasks.rados.rados.0.smithi125.stdout:326: expect (ObjNum 106 snap 21 seq_num 106) 2021-03-24T21:55:34.596 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:34.656 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:34.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:34.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:55:34.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:34 smithi125 conmon[19936]: cluster 2021-03-24T21:55:34.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:34 smithi125 conmon[19936]: 2021-03-24T21:55:32.729407+0000 mgr.y 2021-03-24T21:55:34.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:34 smithi125 conmon[19936]: (mgr.14144) 340 : cluster [DBG] pgmap v407: 148 pgs: 148 active+clean; 123 MiB data, 358 MiB used, 536 GiB / 536 GiB avail; 1.6 MiB/s rd, 197 KiB/s wr, 6 op/s 2021-03-24T21:55:34.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:34 smithi125 conmon[19936]: cluster 2021-03-24T21:55:33.246194+0000 osd.0 ( 2021-03-24T21:55:34.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:34 smithi125 conmon[19936]: osd.0) 47 : cluster [DBG] 6.9 scrub ok 2021-03-24T21:55:34.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:34 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:55:34.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:34 smithi125 conmon[19936]: -03-24T21:55:33.292045+0000 mon.a 2021-03-24T21:55:34.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:34 smithi125 conmon[19936]: (mon.0) 737 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T21:55:34.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:34 smithi125 conmon[19936]: 2021-03-24T21:55:34.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:34 smithi125 conmon[19936]: cluster 2021-03-24T21:55:34.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:34 smithi125 conmon[19936]: 2021-03-24T21:55:33.292110+0000 mon.a 2021-03-24T21:55:34.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:34 smithi125 conmon[19936]: (mon.0) 738 : cluster [INF] Cluster is now healthy 2021-03-24T21:55:34.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:34 smithi125 conmon[19936]: audit 2021-03 2021-03-24T21:55:34.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:34 smithi125 conmon[19936]: -24T21:55:33.310617+0000 mon.a (mon. 2021-03-24T21:55:34.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:34 smithi125 conmon[19936]: 0) 739 : audit [INF] from='client.? v1:172.21.15.125:0/1943451891' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T21:55:34.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:34 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:55:34.669 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:34 smithi125 conmon[19936]: :55:33.310694+0000 mon.a (mon.0) 740 : 2021-03-24T21:55:34.669 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:34 smithi125 conmon[19936]: cluster [DBG] osdmap e164: 8 total, 8 up, 6 in 2021-03-24T21:55:34.669 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:34 smithi125 conmon[19936]: cluster 2021-03-24T21:55:33.533831 2021-03-24T21:55:34.670 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:34 smithi125 conmon[19936]: +0000 osd.3 (osd.3) 53 : cluster [DBG] 3.2 scrub ok 2021-03-24T21:55:34.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:34 smithi125 conmon[24927]: cluster 2021-03-24T21:55:34.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:34 smithi125 conmon[24927]: 2021-03-24T21:55:34.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:34 smithi125 conmon[24927]: 2021 2021-03-24T21:55:34.672 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:34 smithi125 conmon[24927]: - 2021-03-24T21:55:34.672 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:34 smithi125 conmon[24927]: 03 2021-03-24T21:55:34.673 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:34 smithi125 conmon[24927]: - 2021-03-24T21:55:34.673 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:34 smithi125 conmon[24927]: 24 2021-03-24T21:55:34.674 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:34 smithi125 conmon[24927]: T 2021-03-24T21:55:34.674 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:34 smithi125 conmon[24927]: 21 2021-03-24T21:55:34.675 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:34 smithi125 conmon[24927]: :55:32.729407+0000 mgr.y ( 2021-03-24T21:55:34.675 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:34 smithi125 conmon[24927]: mgr.14144) 340 : cluster [DBG] pgmap v407: 148 pgs: 148 active+clean; 123 MiB data, 358 MiB used, 536 GiB / 536 GiB avail; 1.6 MiB/s rd, 197 KiB/s wr, 6 op/s 2021-03-24T21:55:34.675 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:34 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:55:34.676 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:34 smithi125 conmon[24927]: 55:33.246194+0000 osd.0 (osd.0) 2021-03-24T21:55:34.676 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:34 smithi125 conmon[24927]: 47 : cluster [DBG] 6.9 scrub ok 2021-03-24T21:55:34.677 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:34 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:55:34.677 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:34 smithi125 conmon[24927]: :55:33.292045+0000 mon.a (mon.0 2021-03-24T21:55:34.678 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:34 smithi125 conmon[24927]: ) 737 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T21:55:34.678 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:34 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:55:34.679 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:34 smithi125 conmon[24927]: :55:33.292110+0000 mon.a (mon.0 2021-03-24T21:55:34.679 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:34 smithi125 conmon[24927]: ) 738 : cluster [INF] Cluster is now healthy 2021-03-24T21:55:34.679 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:34 smithi125 conmon[24927]: audit 2021-03-24T21 2021-03-24T21:55:34.680 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:34 smithi125 conmon[24927]: :55:33.310617+0000 mon.a (mon.0) 2021-03-24T21:55:34.680 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:34 smithi125 conmon[24927]: 739 : audit [INF] from='client.? v1:172.21.15.125:0/1943451891' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T21:55:34.681 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:34 smithi125 conmon[24927]: cluster 2021-03-24T21:55: 2021-03-24T21:55:34.681 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:34 smithi125 conmon[24927]: 33.310694+0000 mon.a (mon.0) 740 : cluster 2021-03-24T21:55:34.682 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:34 smithi125 conmon[24927]: [DBG] osdmap e164: 8 total, 8 up, 6 in 2021-03-24T21:55:34.682 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:34 smithi125 conmon[24927]: cluster 2021-03-24T21:55:33 2021-03-24T21:55:34.683 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:34 smithi125 conmon[24927]: .533831+0000 osd.3 (osd.3) 53 : cluster 2021-03-24T21:55:34.683 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:34 smithi125 conmon[24927]: [DBG] 3.2 scrub ok 2021-03-24T21:55:34.761 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:34.763 INFO:tasks.rados.rados.0.smithi125.stdout:329: expect (ObjNum 25 snap 0 seq_num 25) 2021-03-24T21:55:34.813 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:34.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:34.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:55:34.844 INFO:tasks.rados.rados.0.smithi125.stdout:331: finishing write tid 1 to smithi12556470-10 2021-03-24T21:55:34.845 INFO:tasks.rados.rados.0.smithi125.stdout:331: finishing write tid 2 to smithi12556470-10 2021-03-24T21:55:34.845 INFO:tasks.rados.rados.0.smithi125.stdout:331: finishing write tid 3 to smithi12556470-10 2021-03-24T21:55:34.845 INFO:tasks.rados.rados.0.smithi125.stdout:331: finishing write tid 4 to smithi12556470-10 2021-03-24T21:55:34.845 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 10 v 111 (ObjNum 148 snap 30 seq_num 148) dirty exists 2021-03-24T21:55:34.846 INFO:tasks.rados.rados.0.smithi125.stdout:331: left oid 10 (ObjNum 148 snap 30 seq_num 148) 2021-03-24T21:55:34.846 INFO:tasks.rados.rados.0.smithi125.stdout:330: finishing copy_from to smithi12556470-17 2021-03-24T21:55:34.846 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 17 v 104 (ObjNum 124 snap 23 seq_num 124) dirty exists 2021-03-24T21:55:34.846 INFO:tasks.rados.rados.0.smithi125.stdout:330: finishing copy_from racing read to smithi12556470-17 2021-03-24T21:55:34.847 INFO:tasks.rados.rados.0.smithi125.stdout:332: finishing write tid 1 to smithi12556470-3 2021-03-24T21:55:34.847 INFO:tasks.rados.rados.0.smithi125.stdout:332: finishing write tid 2 to smithi12556470-3 2021-03-24T21:55:34.847 INFO:tasks.rados.rados.0.smithi125.stdout:332: finishing write tid 3 to smithi12556470-3 2021-03-24T21:55:34.847 INFO:tasks.rados.rados.0.smithi125.stdout:332: finishing write tid 4 to smithi12556470-3 2021-03-24T21:55:34.848 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 3 v 54 (ObjNum 149 snap 30 seq_num 149) dirty exists 2021-03-24T21:55:34.848 INFO:tasks.rados.rados.0.smithi125.stdout:332: left oid 3 (ObjNum 149 snap 30 seq_num 149) 2021-03-24T21:55:34.848 INFO:tasks.rados.rados.0.smithi125.stdout:334: finishing rollback tid 0 to smithi12556470-33 2021-03-24T21:55:34.848 INFO:tasks.rados.rados.0.smithi125.stdout:334: finishing rollback tid 1 to smithi12556470-33 2021-03-24T21:55:34.848 INFO:tasks.rados.rados.0.smithi125.stdout:334: finishing rollback tid 2 to smithi12556470-33 2021-03-24T21:55:34.849 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 33 v 115 (ObjNum 63 snap 4 seq_num 63) dirty exists 2021-03-24T21:55:34.849 INFO:tasks.rados.rados.0.smithi125.stdout:325: done (9 left) 2021-03-24T21:55:34.849 INFO:tasks.rados.rados.0.smithi125.stdout:326: done (8 left) 2021-03-24T21:55:34.849 INFO:tasks.rados.rados.0.smithi125.stdout:327: done (7 left) 2021-03-24T21:55:34.850 INFO:tasks.rados.rados.0.smithi125.stdout:328: done (6 left) 2021-03-24T21:55:34.850 INFO:tasks.rados.rados.0.smithi125.stdout:329: done (5 left) 2021-03-24T21:55:34.850 INFO:tasks.rados.rados.0.smithi125.stdout:330: done (4 left) 2021-03-24T21:55:34.850 INFO:tasks.rados.rados.0.smithi125.stdout:331: done (3 left) 2021-03-24T21:55:34.850 INFO:tasks.rados.rados.0.smithi125.stdout:332: done (2 left) 2021-03-24T21:55:34.851 INFO:tasks.rados.rados.0.smithi125.stdout:333: done (1 left) 2021-03-24T21:55:34.851 INFO:tasks.rados.rados.0.smithi125.stdout:334: done (0 left) 2021-03-24T21:55:34.851 INFO:tasks.rados.rados.0.smithi125.stdout:335: copy_from oid 35 from oid 39 current snap is 30 2021-03-24T21:55:34.851 INFO:tasks.rados.rados.0.smithi125.stdout:336: write (excl) oid 15 current snap is 30 2021-03-24T21:55:34.852 INFO:tasks.rados.rados.0.smithi125.stdout:336: seq_num 150 ranges {521050=698057,1877946=643104} 2021-03-24T21:55:34.857 INFO:tasks.rados.rados.0.smithi125.stdout:336: writing smithi12556470-15 from 521050 to 1219107 tid 1 2021-03-24T21:55:34.866 INFO:tasks.rados.rados.0.smithi125.stdout:336: writing smithi12556470-15 from 1877946 to 2521050 tid 2 2021-03-24T21:55:34.867 INFO:tasks.rados.rados.0.smithi125.stdout:337: read oid 1 snap -1 2021-03-24T21:55:34.867 INFO:tasks.rados.rados.0.smithi125.stdout:337: expect (ObjNum 143 snap 27 seq_num 143) 2021-03-24T21:55:34.867 INFO:tasks.rados.rados.0.smithi125.stdout:338: rollback oid 26 current snap is 30 2021-03-24T21:55:34.867 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 26 to 23 2021-03-24T21:55:34.868 INFO:tasks.rados.rados.0.smithi125.stdout:335: finishing copy_from to smithi12556470-35 2021-03-24T21:55:34.868 INFO:tasks.rados.rados.0.smithi125.stdout:335: got expected ENOENT (src dne) 2021-03-24T21:55:34.868 INFO:tasks.rados.rados.0.smithi125.stdout:339: snap_remove snap 29 2021-03-24T21:55:34.919 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:34.981 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:34.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:34.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:55:35.032 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.4 dump_ops_in_flight 2021-03-24T21:55:35.086 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:35.112 INFO:tasks.rados.rados.0.smithi125.stdout:339: done (4 left) 2021-03-24T21:55:35.112 INFO:tasks.rados.rados.0.smithi125.stdout:340: copy_from oid 22 from oid 47 current snap is 30 2021-03-24T21:55:35.112 INFO:tasks.rados.rados.0.smithi125.stdout:341: snap_create 2021-03-24T21:55:35.113 INFO:tasks.rados.rados.0.smithi125.stdout:335: finishing copy_from racing read to smithi12556470-35 2021-03-24T21:55:35.113 INFO:tasks.rados.rados.0.smithi125.stdout:336: finishing write tid 1 to smithi12556470-15 2021-03-24T21:55:35.113 INFO:tasks.rados.rados.0.smithi125.stdout:338: finishing rollback tid 1 to smithi12556470-26 2021-03-24T21:55:35.113 INFO:tasks.rados.rados.0.smithi125.stdout:336: finishing write tid 2 to smithi12556470-15 2021-03-24T21:55:35.114 INFO:tasks.rados.rados.0.smithi125.stdout:336: finishing write tid 3 to smithi12556470-15 2021-03-24T21:55:35.114 INFO:tasks.rados.rados.0.smithi125.stdout:338: finishing rollback tid 2 to smithi12556470-26 2021-03-24T21:55:35.114 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 26 v 88 (ObjNum 25 snap 0 seq_num 25) dirty exists 2021-03-24T21:55:35.114 INFO:tasks.rados.rados.0.smithi125.stdout:336: finishing write tid 4 to smithi12556470-15 2021-03-24T21:55:35.114 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 15 v 92 (ObjNum 150 snap 30 seq_num 150) dirty exists 2021-03-24T21:55:35.115 INFO:tasks.rados.rados.0.smithi125.stdout:336: left oid 15 (ObjNum 150 snap 30 seq_num 150) 2021-03-24T21:55:35.115 INFO:tasks.rados.rados.0.smithi125.stdout:337: expect (ObjNum 143 snap 27 seq_num 143) 2021-03-24T21:55:35.121 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:55:35.149 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:35.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:35.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:35.257 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:35.258 INFO:tasks.rados.rados.0.smithi125.stdout:340: finishing copy_from to smithi12556470-22 2021-03-24T21:55:35.259 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 22 v 31 (ObjNum 125 snap 23 seq_num 125) dirty exists 2021-03-24T21:55:35.259 INFO:tasks.rados.rados.0.smithi125.stdout:340: finishing copy_from racing read to smithi12556470-22 2021-03-24T21:55:35.310 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:35.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:35.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:55:35.356 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T21:55:35.413 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:35.469 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:35.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:35.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:35.573 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:35 smithi125 conmon[24927]: cluster 2021-03-24T21:55:34.321454+0000 mon.a (mon.0) 2021-03-24T21:55:35.573 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:35 smithi125 conmon[24927]: 741 : cluster [DBG] osdmap e165: 8 total, 8 up, 6 in 2021-03-24T21:55:35.574 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:35 smithi125 conmon[24927]: cluster 2021-03-24T21:55:35.574 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:35 smithi125 conmon[24927]: 2021-03-24T21:55:35.103803+0000 mon.a (mon.0) 742 2021-03-24T21:55:35.574 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:35 smithi125 conmon[24927]: : cluster [DBG] osdmap e166: 8 total, 8 up, 6 in 2021-03-24T21:55:35.575 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:35 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T21:55:35.575 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:35 smithi125 conmon[19936]: T21:55:34.321454+0000 mon.a (mon.0) 741 : cluster [DBG] osdmap e165: 8 total, 8 up, 6 in 2021-03-24T21:55:35.575 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:35 smithi125 conmon[19936]: cluster 2021-03-24T21:55: 2021-03-24T21:55:35.576 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:35 smithi125 conmon[19936]: 35.103803+0000 mon.a (mon. 2021-03-24T21:55:35.576 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:35 smithi125 conmon[19936]: 0) 742 : cluster [DBG] 2021-03-24T21:55:35.577 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:35 smithi125 conmon[19936]: osdmap e166: 8 total, 8 up, 6 in 2021-03-24T21:55:35.577 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:35.592 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:35 smithi142 conmon[20125]: cluster 2021-03-24T21:55:34.321454+0000 2021-03-24T21:55:35.593 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:35 smithi142 conmon[20125]: mon.a (mon.0) 741 : cluster [DBG] osdmap e165: 8 total, 8 up, 6 in 2021-03-24T21:55:35.593 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:35 smithi142 conmon[20125]: cluster 2021-03-24T21:55:35.103803+0000 2021-03-24T21:55:35.594 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:35 smithi142 conmon[20125]: mon.a (mon.0) 742 : cluster [DBG] osdmap e166: 8 total, 8 up, 6 in 2021-03-24T21:55:35.639 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:35.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:35.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:55:35.744 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:35.803 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:35.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:35.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:55:35.908 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:35.954 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:35.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:35.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:55:36.059 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:36.107 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:36.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:36.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:55:36.121 INFO:tasks.rados.rados.0.smithi125.stdout:335: done (5 left) 2021-03-24T21:55:36.121 INFO:tasks.rados.rados.0.smithi125.stdout:336: done (4 left) 2021-03-24T21:55:36.122 INFO:tasks.rados.rados.0.smithi125.stdout:337: done (3 left) 2021-03-24T21:55:36.122 INFO:tasks.rados.rados.0.smithi125.stdout:338: done (2 left) 2021-03-24T21:55:36.122 INFO:tasks.rados.rados.0.smithi125.stdout:340: done (1 left) 2021-03-24T21:55:36.122 INFO:tasks.rados.rados.0.smithi125.stdout:341: done (0 left) 2021-03-24T21:55:36.122 INFO:tasks.rados.rados.0.smithi125.stdout:342: copy_from oid 10 from oid 9 current snap is 31 2021-03-24T21:55:36.123 INFO:tasks.rados.rados.0.smithi125.stdout:343: snap_create 2021-03-24T21:55:36.144 INFO:tasks.rados.rados.0.smithi125.stdout:342: finishing copy_from to smithi12556470-10 2021-03-24T21:55:36.144 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 10 v 21 (ObjNum 68 snap 7 seq_num 68) dirty exists 2021-03-24T21:55:36.145 INFO:tasks.rados.rados.0.smithi125.stdout:342: finishing copy_from racing read to smithi12556470-10 2021-03-24T21:55:36.213 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:36.272 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:36.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:36.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:55:36.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:36 smithi142 conmon[20125]: cluster 2021-03-24T21:55:34.730068+0000 mgr.y (mgr.14144) 341 : 2021-03-24T21:55:36.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:36 smithi142 conmon[20125]: cluster [DBG] pgmap v410: 148 pgs: 148 active+clean; 127 MiB data, 374 MiB used, 536 GiB / 536 GiB avail; 4.6 MiB/s rd, 1.4 MiB/s wr, 11 op/s 2021-03-24T21:55:36.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:36 smithi142 conmon[20125]: cluster 2021-03-24T21:55:34.930956+0000 osd.7 2021-03-24T21:55:36.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:36 smithi142 conmon[20125]: (osd.7) 38 : cluster [DBG] 6.5 scrub ok 2021-03-24T21:55:36.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:36 smithi142 conmon[20125]: cluster 2021-03-24T21:55:36.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:36 smithi142 conmon[20125]: 2021-03-24T21:55 2021-03-24T21:55:36.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:36 smithi142 conmon[20125]: :35.512753+0000 osd.5 (osd.5) 47 2021-03-24T21:55:36.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:36 smithi142 conmon[20125]: : cluster [DBG] 5.7 scrub ok 2021-03-24T21:55:36.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:36 smithi142 conmon[20125]: cluster 2021-03-24T21:55:36.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:36 smithi142 conmon[20125]: 2021-03-24T21:55:36.065252 2021-03-24T21:55:36.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:36 smithi142 conmon[20125]: +0000 osd.4 (osd.4) 49 : cluster [DBG] 2021-03-24T21:55:36.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:36 smithi142 conmon[20125]: 4.c scrub ok 2021-03-24T21:55:36.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:36 smithi142 conmon[20125]: cluster 2021-03-24T21:55 2021-03-24T21:55:36.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:36 smithi142 conmon[20125]: :36.118859+0000 mon.a (mon.0) 743 : cluster [DBG] 2021-03-24T21:55:36.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:36 smithi142 conmon[20125]: osdmap e167: 8 total, 8 up, 6 in 2021-03-24T21:55:36.377 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:36.433 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:36.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:36.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:55:36.540 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:36.603 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:36.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:36.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:55:36.627 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:36 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:55:36.627 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:36 smithi125 conmon[24927]: :55:34.730068+0000 mgr.y (mgr.14144) 341 : cluster [DBG] pgmap v410: 148 pgs: 148 active+clean; 127 MiB data, 374 MiB used, 536 GiB / 536 GiB avail; 4.6 MiB/s rd, 1.4 MiB/s wr, 11 op/s 2021-03-24T21:55:36.627 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:36 smithi125 conmon[24927]: cluster 2021-03-24T21:55: 2021-03-24T21:55:36.628 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:36 smithi125 conmon[24927]: 34.930956+0000 osd.7 (osd.7) 38 : cluster 2021-03-24T21:55:36.628 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:36 smithi125 conmon[24927]: [DBG] 6.5 scrub ok 2021-03-24T21:55:36.628 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:36 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:55:36.628 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:36 smithi125 conmon[24927]: :55:35.512753+0000 osd.5 (osd.5) 2021-03-24T21:55:36.628 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:36 smithi125 conmon[24927]: 47 : cluster [DBG] 5.7 scrub ok 2021-03-24T21:55:36.629 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:36 smithi125 conmon[24927]: cluster 2021-03-24T21:55:36.629 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:36 smithi125 conmon[24927]: 2021-03-24T21 2021-03-24T21:55:36.629 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:36 smithi125 conmon[24927]: :55:36.065252+0000 osd.4 (osd.4 2021-03-24T21:55:36.629 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:36 smithi125 conmon[24927]: ) 49 : cluster [DBG] 4.c scrub ok 2021-03-24T21:55:36.630 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:36 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:55:36.630 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:36 smithi125 conmon[24927]: 03-24T21:55:36.118859+0000 mon.a ( 2021-03-24T21:55:36.630 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:36 smithi125 conmon[24927]: mon.0) 743 : cluster [DBG] osdmap e167: 8 total, 8 up, 6 in 2021-03-24T21:55:36.630 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:36 smithi125 conmon[19936]: cluster 2021-03-24T21:55: 2021-03-24T21:55:36.631 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:36 smithi125 conmon[19936]: 34.730068+0000 mgr.y (mgr.14144) 341 : cluster [DBG] pgmap v410: 148 pgs: 148 active+clean; 127 MiB data, 374 MiB used, 536 GiB / 536 GiB avail; 4.6 MiB/s rd, 1.4 MiB/s wr, 11 op/s 2021-03-24T21:55:36.631 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:36 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:55:36.631 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:36 smithi125 conmon[19936]: -24T21:55:34.930956+0000 osd.7 (osd.7) 38 : cluster [DBG] 6.5 scrub ok 2021-03-24T21:55:36.632 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:36 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:55:36.632 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:36 smithi125 conmon[19936]: -24T21:55:35.512753+0000 2021-03-24T21:55:36.632 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:36 smithi125 conmon[19936]: osd.5 (osd.5) 47 : cluster [DBG] 5.7 scrub ok 2021-03-24T21:55:36.632 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:36 smithi125 conmon[19936]: 2021-03-24T21:55:36.632 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:36 smithi125 conmon[19936]: cluster 2021-03-24T21:55:36.633 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:36 smithi125 conmon[19936]: 2021-03-24T21:55:36.065252+0000 osd.4 2021-03-24T21:55:36.633 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:36 smithi125 conmon[19936]: (osd.4) 49 : cluster [DBG] 4.c scrub ok 2021-03-24T21:55:36.633 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:36 smithi125 conmon[19936]: 2021-03-24T21:55:36.633 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:36 smithi125 conmon[19936]: cluster 2021-03-24T21:55:36.634 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:36 smithi125 conmon[19936]: 2021-03-24T21:55:36.118859+0000 2021-03-24T21:55:36.634 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:36 smithi125 conmon[19936]: mon.a (mon.0) 743 : cluster [DBG] 2021-03-24T21:55:36.634 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:36 smithi125 conmon[19936]: osdmap e167: 8 total, 8 up, 6 in 2021-03-24T21:55:36.709 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:36.766 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:36.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:36.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:55:36.872 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:36.917 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T21:55:36.918 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:36.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:36.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:36.937 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T21:55:36.957 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T21:55:36.976 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T21:55:36.999 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T21:55:37.021 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T21:55:37.025 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:37.050 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T21:55:37.075 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd set noscrub 2021-03-24T21:55:37.077 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:37.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:37.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:55:37.081 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T21:55:37.083 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-24T21:55:37.083 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-24T21:55:37.083 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-24T21:55:37.084 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-24T21:55:37.084 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-24T21:55:37.084 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-24T21:55:37.084 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-24T21:55:37.085 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-24T21:55:37.121 INFO:tasks.rados.rados.0.smithi125.stdout:342: done (1 left) 2021-03-24T21:55:37.121 INFO:tasks.rados.rados.0.smithi125.stdout:343: done (0 left) 2021-03-24T21:55:37.121 INFO:tasks.rados.rados.0.smithi125.stdout:344: delete oid 45 current snap is 32 2021-03-24T21:55:37.127 INFO:tasks.rados.rados.0.smithi125.stdout:344: done (0 left) 2021-03-24T21:55:37.127 INFO:tasks.rados.rados.0.smithi125.stdout:345: write oid 19 current snap is 32 2021-03-24T21:55:37.128 INFO:tasks.rados.rados.0.smithi125.stdout:345: seq_num 152 ranges {554664=448555,1478557=664373} 2021-03-24T21:55:37.133 INFO:tasks.rados.rados.0.smithi125.stdout:345: writing smithi12556470-19 from 554664 to 1003219 tid 1 2021-03-24T21:55:37.140 INFO:tasks.rados.rados.0.smithi125.stdout:345: writing smithi12556470-19 from 1478557 to 2142930 tid 2 2021-03-24T21:55:37.140 INFO:tasks.rados.rados.0.smithi125.stdout:346: snap_create 2021-03-24T21:55:37.141 INFO:tasks.rados.rados.0.smithi125.stdout:345: finishing write tid 1 to smithi12556470-19 2021-03-24T21:55:37.145 INFO:tasks.rados.rados.0.smithi125.stdout:345: finishing write tid 2 to smithi12556470-19 2021-03-24T21:55:37.145 INFO:tasks.rados.rados.0.smithi125.stdout:345: finishing write tid 3 to smithi12556470-19 2021-03-24T21:55:37.146 INFO:tasks.rados.rados.0.smithi125.stdout:345: finishing write tid 4 to smithi12556470-19 2021-03-24T21:55:37.146 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 19 v 121 (ObjNum 152 snap 32 seq_num 152) dirty exists 2021-03-24T21:55:37.147 INFO:tasks.rados.rados.0.smithi125.stdout:345: left oid 19 (ObjNum 152 snap 32 seq_num 152) 2021-03-24T21:55:37.182 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:37.237 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:37.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:37.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:55:37.344 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:37.387 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:37.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:37.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:55:37.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:37 smithi125 conmon[19936]: cluster 2021-03-24T21:55:36.236586+0000 osd.0 (osd 2021-03-24T21:55:37.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:37 smithi125 conmon[19936]: .0) 48 : cluster [DBG] 5.5 scrub ok 2021-03-24T21:55:37.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:37 smithi125 conmon[19936]: audit 2021-03-24T21:55:36.886293+0000 mon.c (mon.2) 30 : audit [DBG] from='client.? v1:172.21.15.125:0/3094340538' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:55:37.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:37 smithi125 conmon[19936]: cluster 2021-03-24T21:55:37.076493+0000 osd.4 2021-03-24T21:55:37.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:37 smithi125 conmon[19936]: (osd.4) 50 : cluster [DBG] 4.13 scrub ok 2021-03-24T21:55:37.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:37 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:55:37.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:37 smithi125 conmon[19936]: 03-24 2021-03-24T21:55:37.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:37 smithi125 conmon[19936]: T21:55:37.119659+0000 mon.a (mon.0) 2021-03-24T21:55:37.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:37 smithi125 conmon[19936]: 744 : cluster [DBG] osdmap e168: 8 total, 8 up, 6 in 2021-03-24T21:55:37.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:37 smithi125 conmon[24927]: cluster 2021-03-24T21:55:36.236586+0000 2021-03-24T21:55:37.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:37 smithi125 conmon[24927]: osd.0 (osd.0) 48 : cluster [DBG] 5.5 scrub ok 2021-03-24T21:55:37.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:37 smithi125 conmon[24927]: audit 2021-03-24T21: 2021-03-24T21:55:37.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:37 smithi125 conmon[24927]: 55:36.886293+0000 mon.c (mon.2) 30 : audit [DBG] from='client.? v1:172.21.15.125:0/3094340538' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:55:37.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:37 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:55:37.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:37 smithi125 conmon[24927]: 55:37.076493+0000 osd.4 (osd.4) 50 : 2021-03-24T21:55:37.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:37 smithi125 conmon[24927]: cluster [DBG] 4.13 scrub ok 2021-03-24T21:55:37.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:37 smithi125 conmon[24927]: cluster 2021-03-24T21:55 2021-03-24T21:55:37.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:37 smithi125 conmon[24927]: :37.119659+0000 mon.a (mon.0) 744 : 2021-03-24T21:55:37.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:37 smithi125 conmon[24927]: cluster [DBG] osdmap e168: 8 total, 8 up, 6 in 2021-03-24T21:55:37.494 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:37.506 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:37 smithi142 conmon[20125]: cluster 2021-03-24T21:55:36.236586+0000 osd.0 (osd.0) 48 : cluster [DBG] 5.5 scrub ok 2021-03-24T21:55:37.507 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:37 smithi142 conmon[20125]: audit 2021-03-24T21:55:36.886293+0000 mon.c (mon.2) 30 : audit [DBG] from='client.? v1:172.21.15.125:0/3094340538' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:55:37.507 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:37 smithi142 conmon[20125]: cluster 2021-03-24T21:55:37.507 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:37 smithi142 conmon[20125]: 2021-03-24T21:55:37.076493+0000 2021-03-24T21:55:37.508 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:37 smithi142 conmon[20125]: osd.4 (osd.4) 50 : cluster [DBG] 2021-03-24T21:55:37.508 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:37 smithi142 conmon[20125]: 4.13 scrub ok 2021-03-24T21:55:37.508 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:37 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:55:37.508 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:37 smithi142 conmon[20125]: -24T21:55:37.119659+0000 2021-03-24T21:55:37.508 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:37 smithi142 conmon[20125]: mon.a (mon.0) 744 : cluster [DBG] osdmap e168: 8 total, 8 up, 6 in 2021-03-24T21:55:37.509 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:37 smithi142 conmon[20125]: 2021-03-24T21:55:37.549 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:37.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:37.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:55:37.553 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2021-03-24T21:55:37.554 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T21:55:37.654 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:37.720 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:37.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:37.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:55:37.752 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 7] out_osds: [6, 1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-03-24T21:55:37.752 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-24T21:55:37.753 INFO:tasks.thrashosds.thrasher:fixing pg num pool iscsi 2021-03-24T21:55:37.753 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph pg dump --format=json 2021-03-24T21:55:37.827 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:37.882 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:37.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:37.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:55:37.989 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:38.039 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:38.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:38.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:55:38.124 INFO:tasks.rados.rados.0.smithi125.stdout:345: done (1 left) 2021-03-24T21:55:38.124 INFO:tasks.rados.rados.0.smithi125.stdout:346: done (0 left) 2021-03-24T21:55:38.125 INFO:tasks.rados.rados.0.smithi125.stdout:347: write (excl) oid 46 current snap is 33 2021-03-24T21:55:38.125 INFO:tasks.rados.rados.0.smithi125.stdout:347: seq_num 153 ranges {571471=640157,1653392=524384} 2021-03-24T21:55:38.130 INFO:tasks.rados.rados.0.smithi125.stdout:347: writing smithi12556470-46 from 571471 to 1211628 tid 1 2021-03-24T21:55:38.136 INFO:tasks.rados.rados.0.smithi125.stdout:347: writing smithi12556470-46 from 1653392 to 2177776 tid 2 2021-03-24T21:55:38.137 INFO:tasks.rados.rados.0.smithi125.stdout:348: copy_from oid 40 from oid 14 current snap is 33 2021-03-24T21:55:38.137 INFO:tasks.rados.rados.0.smithi125.stdout:347: finishing write tid 1 to smithi12556470-46 2021-03-24T21:55:38.137 INFO:tasks.rados.rados.0.smithi125.stdout:349: read oid 6 snap -1 2021-03-24T21:55:38.138 INFO:tasks.rados.rados.0.smithi125.stdout:349: expect (ObjNum 126 snap 23 seq_num 126) 2021-03-24T21:55:38.138 INFO:tasks.rados.rados.0.smithi125.stdout:350: rollback oid 39 current snap is 33 2021-03-24T21:55:38.138 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 39 to 30 2021-03-24T21:55:38.138 INFO:tasks.rados.rados.0.smithi125.stdout:351: delete oid 42 current snap is 33 2021-03-24T21:55:38.139 INFO:tasks.rados.rados.0.smithi125.stdout:350: finishing rollback tid 1 to smithi12556470-39 2021-03-24T21:55:38.139 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 39 v 0 (ObjNum 1116673472 snap 21916 seq_num 875967797) dirty dne 2021-03-24T21:55:38.142 INFO:tasks.rados.rados.0.smithi125.stdout:347: finishing write tid 2 to smithi12556470-46 2021-03-24T21:55:38.142 INFO:tasks.rados.rados.0.smithi125.stdout:347: finishing write tid 3 to smithi12556470-46 2021-03-24T21:55:38.142 INFO:tasks.rados.rados.0.smithi125.stdout:347: finishing write tid 4 to smithi12556470-46 2021-03-24T21:55:38.142 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 46 v 53 (ObjNum 153 snap 33 seq_num 153) dirty exists 2021-03-24T21:55:38.143 INFO:tasks.rados.rados.0.smithi125.stdout:347: left oid 46 (ObjNum 153 snap 33 seq_num 153) 2021-03-24T21:55:38.143 INFO:tasks.rados.rados.0.smithi125.stdout:347: done (4 left) 2021-03-24T21:55:38.144 INFO:tasks.rados.rados.0.smithi125.stdout:350: done (3 left) 2021-03-24T21:55:38.144 INFO:tasks.rados.rados.0.smithi125.stdout:351: done (2 left) 2021-03-24T21:55:38.144 INFO:tasks.rados.rados.0.smithi125.stdout:352: snap_create 2021-03-24T21:55:38.144 INFO:tasks.rados.rados.0.smithi125.stdout:348: finishing copy_from to smithi12556470-40 2021-03-24T21:55:38.144 INFO:tasks.rados.rados.0.smithi125.stdout:348: got expected ENOENT (src dne) 2021-03-24T21:55:38.145 INFO:tasks.rados.rados.0.smithi125.stdout:348: finishing copy_from racing read to smithi12556470-40 2021-03-24T21:55:38.145 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:38.146 INFO:tasks.rados.rados.0.smithi125.stdout:349: expect (ObjNum 126 snap 23 seq_num 126) 2021-03-24T21:55:38.157 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:55:38.174 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:55:38.195 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:38.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:38.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:55:38.301 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:38.359 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:38.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:38.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:55:38.464 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:38.521 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:38.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:38.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:55:38.576 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:38 smithi125 conmon[19936]: cluster 2021-03-24T21:55:36.730913+0000 mgr.y (mgr.14144) 342 : cluster [DBG] pgmap v413: 148 pgs: 148 active+clean; 129 MiB data, 377 MiB used, 536 GiB / 536 GiB avail; 6.8 MiB/s rd, 1.7 MiB/s wr, 15 op/s 2021-03-24T21:55:38.576 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:38 smithi125 conmon[19936]: cluster 2021-03-24T21:55:38.045876+0000 osd.4 (osd.4) 51 : cluster [DBG] 3.3 scrub ok 2021-03-24T21:55:38.576 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:38 smithi125 conmon[19936]: cluster 2021-03-24T21:55:38.120718+0000 mon.a (mon.0) 745 : cluster [DBG] osdmap e169: 8 total, 8 up, 6 in 2021-03-24T21:55:38.577 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:38 smithi125 conmon[24927]: cluster 2021-03-24T21:55: 2021-03-24T21:55:38.577 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:38 smithi125 conmon[24927]: 36.730913+0000 mgr.y (mgr.14144) 342 : cluster [DBG] pgmap v413: 148 pgs: 148 active+clean; 129 MiB data, 377 MiB used, 536 GiB / 536 GiB avail; 6.8 MiB/s rd, 1.7 MiB/s wr, 15 op/s 2021-03-24T21:55:38.577 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:38 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:55:38.577 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:38 smithi125 conmon[24927]: :55:38.045876+0000 2021-03-24T21:55:38.578 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:38 smithi125 conmon[24927]: osd.4 (osd.4) 51 2021-03-24T21:55:38.578 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:38 smithi125 conmon[24927]: : cluster [DBG] 3.3 scrub ok 2021-03-24T21:55:38.579 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:38 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:55:38.579 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:38 smithi125 conmon[24927]: -03-24T21:55: 2021-03-24T21:55:38.579 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:38 smithi125 conmon[24927]: 38.120718+0000 mon.a (mon 2021-03-24T21:55:38.580 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:38 smithi125 conmon[24927]: .0) 745 : cluster 2021-03-24T21:55:38.580 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:38 smithi125 conmon[24927]: [DBG] osdmap e169: 8 total, 8 up, 6 in 2021-03-24T21:55:38.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:38 smithi142 conmon[20125]: cluster 2021-03-24T21:55:36.730913+0000 mgr.y (mgr.14144) 342 : cluster [DBG] 2021-03-24T21:55:38.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:38 smithi142 conmon[20125]: pgmap v413: 148 pgs: 148 active+clean; 129 MiB data, 377 MiB used, 536 GiB / 536 GiB avail; 6.8 MiB/s rd, 1.7 MiB/s wr, 15 op/s 2021-03-24T21:55:38.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:38 smithi142 conmon[20125]: cluster 2021-03-24T21:55:38.045876+0000 osd.4 (osd.4 2021-03-24T21:55:38.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:38 smithi142 conmon[20125]: ) 51 : cluster [DBG] 3.3 scrub ok 2021-03-24T21:55:38.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:38 smithi142 conmon[20125]: cluster 2021-03-24T21:55:38.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:38 smithi142 conmon[20125]: 2021-03-24T21:55:38.120718+0000 mon.a ( 2021-03-24T21:55:38.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:38 smithi142 conmon[20125]: mon.0) 745 : cluster [DBG] osdmap e169: 8 total, 8 up, 6 in 2021-03-24T21:55:38.627 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:38.649 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.4 dump_blocked_ops 2021-03-24T21:55:38.686 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:38.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:38.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:55:38.791 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:38.851 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:38.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:38.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:55:38.939 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T21:55:38.958 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:39.020 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:39.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:39.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:55:39.125 INFO:tasks.rados.rados.0.smithi125.stdout:348: done (2 left) 2021-03-24T21:55:39.126 INFO:tasks.rados.rados.0.smithi125.stdout:349: done (1 left) 2021-03-24T21:55:39.126 INFO:tasks.rados.rados.0.smithi125.stdout:352: done (0 left) 2021-03-24T21:55:39.126 INFO:tasks.rados.rados.0.smithi125.stdout:353: read oid 33 snap -1 2021-03-24T21:55:39.126 INFO:tasks.rados.rados.0.smithi125.stdout:353: expect (ObjNum 63 snap 4 seq_num 63) 2021-03-24T21:55:39.127 INFO:tasks.rados.rados.0.smithi125.stdout:354: snap_remove snap 32 2021-03-24T21:55:39.127 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:39.181 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:39.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:39.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:55:39.285 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:39.341 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:39.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:39.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:55:39.434 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:39 smithi142 conmon[20125]: cluster 2021-03-24T21:55:38.294670+0000 osd.0 (osd.0) 49 : cluster [DBG] 3.1f scrub ok 2021-03-24T21:55:39.434 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:39 smithi142 conmon[20125]: cluster 2021-03-24T21:55:38.432913+0000 osd.3 (osd.3) 54 : cluster [DBG] 3.1e scrub ok 2021-03-24T21:55:39.435 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:39 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:55:39.435 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:39 smithi142 conmon[20125]: T21:55:39.123939+0000 mon.a (mon.0) 746 : cluster 2021-03-24T21:55:39.435 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:39 smithi142 conmon[20125]: [DBG] osdmap e170: 8 total, 8 up, 6 in 2021-03-24T21:55:39.446 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:39.502 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:39.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:39.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:39.606 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:39.656 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:39.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:39 smithi125 conmon[19936]: cluster 2021-03-24T21:55:38.294670+0000 osd.0 (osd. 2021-03-24T21:55:39.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:39 smithi125 conmon[19936]: 0) 49 : cluster [DBG] 3.1f scrub ok 2021-03-24T21:55:39.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:39 smithi125 conmon[19936]: cluster 2021-03-24T21:55:38.432913+0000 osd.3 (osd.3) 54 : cluster [DBG] 3.1e scrub ok 2021-03-24T21:55:39.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:39 smithi125 conmon[19936]: cluster 2021-03-24T21:55:39.123939+0000 mon.a (mon.0) 746 : cluster [DBG] osdmap e170: 8 total, 8 up, 6 in 2021-03-24T21:55:39.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:39 smithi125 conmon[24927]: cluster 2021-03-24T21:55: 2021-03-24T21:55:39.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:39 smithi125 conmon[24927]: 38.294670+0000 osd.0 (osd.0) 49 : cluster [DBG] 3.1f scrub ok 2021-03-24T21:55:39.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:39 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:55:39.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:39 smithi125 conmon[24927]: T21:55:38.432913 2021-03-24T21:55:39.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:39 smithi125 conmon[24927]: +0000 osd.3 (osd.3 2021-03-24T21:55:39.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:39 smithi125 conmon[24927]: ) 54 : cluster [DBG] 3.1e scrub ok 2021-03-24T21:55:39.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:39 smithi125 conmon[24927]: 2021-03-24T21:55:39.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:39 smithi125 conmon[24927]: cluster 2021-03-24T21:55:39.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:39 smithi125 conmon[24927]: 2021-03 2021-03-24T21:55:39.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:39 smithi125 conmon[24927]: -24T21:55:39. 2021-03-24T21:55:39.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:39 smithi125 conmon[24927]: 123939+0000 mon.a (mon. 2021-03-24T21:55:39.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:39 smithi125 conmon[24927]: 0) 746 : cluster [DBG] 2021-03-24T21:55:39.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:39 smithi125 conmon[24927]: osdmap e170: 8 total, 8 up, 6 in 2021-03-24T21:55:39.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:39.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:55:39.766 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:39.823 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:39.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:39.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:55:39.930 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:39.996 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:40.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:40.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:55:40.103 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:40.112 INFO:tasks.rados.rados.0.smithi125.stdout:354: done (1 left) 2021-03-24T21:55:40.112 INFO:tasks.rados.rados.0.smithi125.stdout:355: copy_from oid 47 from oid 36 current snap is 34 2021-03-24T21:55:40.113 INFO:tasks.rados.rados.0.smithi125.stdout:356: read oid 12 snap -1 2021-03-24T21:55:40.113 INFO:tasks.rados.rados.0.smithi125.stdout:356: expect deleted 2021-03-24T21:55:40.113 INFO:tasks.rados.rados.0.smithi125.stdout:357: write (excl) oid 26 current snap is 34 2021-03-24T21:55:40.113 INFO:tasks.rados.rados.0.smithi125.stdout:357: seq_num 155 ranges {605085=790655,2170356=434729} 2021-03-24T21:55:40.118 INFO:tasks.rados.rados.0.smithi125.stdout:357: writing smithi12556470-26 from 605085 to 1395740 tid 1 2021-03-24T21:55:40.126 INFO:tasks.rados.rados.0.smithi125.stdout:357: writing smithi12556470-26 from 2170356 to 2605085 tid 2 2021-03-24T21:55:40.130 INFO:tasks.rados.rados.0.smithi125.stdout:353: expect (ObjNum 63 snap 4 seq_num 63) 2021-03-24T21:55:40.148 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-24T21:55:40.152 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:40.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:40.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:55:40.258 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:40.266 INFO:tasks.rados.rados.0.smithi125.stdout:357: finishing write tid 1 to smithi12556470-26 2021-03-24T21:55:40.266 INFO:tasks.rados.rados.0.smithi125.stdout:357: finishing write tid 2 to smithi12556470-26 2021-03-24T21:55:40.266 INFO:tasks.rados.rados.0.smithi125.stdout:355: finishing copy_from to smithi12556470-47 2021-03-24T21:55:40.266 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 47 v 77 (ObjNum 120 snap 23 seq_num 120) dirty exists 2021-03-24T21:55:40.267 INFO:tasks.rados.rados.0.smithi125.stdout:355: finishing copy_from racing read to smithi12556470-47 2021-03-24T21:55:40.267 INFO:tasks.rados.rados.0.smithi125.stdout:357: finishing write tid 3 to smithi12556470-26 2021-03-24T21:55:40.267 INFO:tasks.rados.rados.0.smithi125.stdout:357: finishing write tid 4 to smithi12556470-26 2021-03-24T21:55:40.267 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 26 v 92 (ObjNum 155 snap 34 seq_num 155) dirty exists 2021-03-24T21:55:40.267 INFO:tasks.rados.rados.0.smithi125.stdout:357: left oid 26 (ObjNum 155 snap 34 seq_num 155) 2021-03-24T21:55:40.268 INFO:tasks.rados.rados.0.smithi125.stdout:353: done (3 left) 2021-03-24T21:55:40.268 INFO:tasks.rados.rados.0.smithi125.stdout:355: done (2 left) 2021-03-24T21:55:40.268 INFO:tasks.rados.rados.0.smithi125.stdout:356: done (1 left) 2021-03-24T21:55:40.268 INFO:tasks.rados.rados.0.smithi125.stdout:357: done (0 left) 2021-03-24T21:55:40.268 INFO:tasks.rados.rados.0.smithi125.stdout:358: write oid 34 current snap is 34 2021-03-24T21:55:40.269 INFO:tasks.rados.rados.0.smithi125.stdout:358: seq_num 156 ranges {621892=465904,1712485=671711} 2021-03-24T21:55:40.282 INFO:tasks.rados.rados.0.smithi125.stdout:358: writing smithi12556470-34 from 621892 to 1087796 tid 1 2021-03-24T21:55:40.290 INFO:tasks.rados.rados.0.smithi125.stdout:358: writing smithi12556470-34 from 1712485 to 2384196 tid 2 2021-03-24T21:55:40.291 INFO:tasks.rados.rados.0.smithi125.stdout:359: write oid 47 current snap is 34 2021-03-24T21:55:40.291 INFO:tasks.rados.rados.0.smithi125.stdout:358: finishing write tid 1 to smithi12556470-34 2021-03-24T21:55:40.291 INFO:tasks.rados.rados.0.smithi125.stdout:359: seq_num 157 ranges {638699=541153,1770967=531722} 2021-03-24T21:55:40.297 INFO:tasks.rados.rados.0.smithi125.stdout:359: writing smithi12556470-47 from 638699 to 1179852 tid 1 2021-03-24T21:55:40.304 INFO:tasks.rados.rados.0.smithi125.stdout:359: writing smithi12556470-47 from 1770967 to 2302689 tid 2 2021-03-24T21:55:40.305 INFO:tasks.rados.rados.0.smithi125.stdout:360: snap_create 2021-03-24T21:55:40.305 INFO:tasks.rados.rados.0.smithi125.stdout:358: finishing write tid 2 to smithi12556470-34 2021-03-24T21:55:40.305 INFO:tasks.rados.rados.0.smithi125.stdout:358: finishing write tid 3 to smithi12556470-34 2021-03-24T21:55:40.305 INFO:tasks.rados.rados.0.smithi125.stdout:358: finishing write tid 4 to smithi12556470-34 2021-03-24T21:55:40.305 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 34 v 124 (ObjNum 156 snap 34 seq_num 156) dirty exists 2021-03-24T21:55:40.306 INFO:tasks.rados.rados.0.smithi125.stdout:358: left oid 34 (ObjNum 156 snap 34 seq_num 156) 2021-03-24T21:55:40.306 INFO:tasks.rados.rados.0.smithi125.stdout:359: finishing write tid 1 to smithi12556470-47 2021-03-24T21:55:40.310 INFO:tasks.rados.rados.0.smithi125.stdout:359: finishing write tid 2 to smithi12556470-47 2021-03-24T21:55:40.310 INFO:tasks.rados.rados.0.smithi125.stdout:359: finishing write tid 3 to smithi12556470-47 2021-03-24T21:55:40.311 INFO:tasks.rados.rados.0.smithi125.stdout:359: finishing write tid 4 to smithi12556470-47 2021-03-24T21:55:40.311 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 47 v 80 (ObjNum 157 snap 34 seq_num 157) dirty exists 2021-03-24T21:55:40.311 INFO:tasks.rados.rados.0.smithi125.stdout:359: left oid 47 (ObjNum 157 snap 34 seq_num 157) 2021-03-24T21:55:40.314 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:40.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:40.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:55:40.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:40 smithi142 conmon[20125]: cluster 2021-03-24T21:55:38.731542+0000 mgr.y (mgr.14144) 343 : 2021-03-24T21:55:40.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:40 smithi142 conmon[20125]: cluster [DBG] pgmap v416: 148 pgs: 148 active+clean; 131 MiB data, 382 MiB used, 536 GiB / 536 GiB avail; 11 MiB/s rd, 550 KiB/s wr, 12 op/s 2021-03-24T21:55:40.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:40 smithi142 conmon[20125]: cluster 2021-03-24T21:55:38.773515+0000 osd.7 (osd 2021-03-24T21:55:40.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:40 smithi142 conmon[20125]: .7) 39 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:55:40.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:40 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:55:40.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:40 smithi142 conmon[20125]: 24T21:55:38.773870 2021-03-24T21:55:40.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:40 smithi142 conmon[20125]: +0000 osd.7 (osd.7) 40 : cluster 2021-03-24T21:55:40.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:40 smithi142 conmon[20125]: [DBG] purged_snaps scrub ok 2021-03-24T21:55:40.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:40 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:55:40.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:40 smithi142 conmon[20125]: -24T21:55:38. 2021-03-24T21:55:40.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:40 smithi142 conmon[20125]: 974148+0000 osd.7 (osd.7) 41 : 2021-03-24T21:55:40.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:40 smithi142 conmon[20125]: cluster [DBG] 3.d scrub ok 2021-03-24T21:55:40.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:40 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:55:40.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:40 smithi142 conmon[20125]: 03-24T21:55: 2021-03-24T21:55:40.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:40 smithi142 conmon[20125]: 39.261673+0000 osd.0 (osd.0) 2021-03-24T21:55:40.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:40 smithi142 conmon[20125]: 50 : cluster [DBG] 6.0 scrub ok 2021-03-24T21:55:40.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:40 smithi142 conmon[20125]: cluster 2021-03-24T21:55:40.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:40 smithi142 conmon[20125]: 2021-03-24T21: 2021-03-24T21:55:40.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:40 smithi142 conmon[20125]: 55:39.435259+0000 osd.5 2021-03-24T21:55:40.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:40 smithi142 conmon[20125]: (osd.5) 48 : cluster [DBG] 2021-03-24T21:55:40.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:40 smithi142 conmon[20125]: 6.19 scrub ok 2021-03-24T21:55:40.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:40 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:55:40.373 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:40 smithi142 conmon[20125]: 24T21:55:40.106133 2021-03-24T21:55:40.373 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:40 smithi142 conmon[20125]: +0000 mon.a (mon.0) 747 : cluster 2021-03-24T21:55:40.373 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:40 smithi142 conmon[20125]: [DBG] osdmap e171: 8 total, 8 up, 6 in 2021-03-24T21:55:40.420 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:40.475 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:40.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:40.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:55:40.487 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:40 smithi125 conmon[19936]: cluster 2021-03-24T21:55:38.731542+0000 mgr.y (mgr.14144) 343 : cluster [DBG] pgmap v416: 148 pgs: 148 active+clean; 131 MiB data, 382 MiB used, 536 GiB / 536 GiB avail; 11 MiB/s rd, 550 KiB/s wr, 12 op/s 2021-03-24T21:55:40.487 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:40 smithi125 conmon[19936]: cluster 2021-03-24T21:55:38.773515+0000 osd.7 (osd.7) 39 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:55:40.488 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:40 smithi125 conmon[19936]: cluster 2021-03-24T21:55:38.773870+0000 osd.7 (osd.7) 40 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:55:40.488 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:40 smithi125 conmon[19936]: cluster 2021-03-24T21:55 2021-03-24T21:55:40.488 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:40 smithi125 conmon[19936]: :38.974148+0000 osd.7 (osd.7) 41 : cluster [DBG] 3.d scrub ok 2021-03-24T21:55:40.488 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:40 smithi125 conmon[19936]: cluster 2021-03-24T21:55:39.261673+0000 osd.0 (osd. 2021-03-24T21:55:40.489 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:40 smithi125 conmon[19936]: 0) 50 : cluster [DBG] 6.0 scrub ok 2021-03-24T21:55:40.489 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:40 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:55:40.489 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:40 smithi125 conmon[19936]: 24T21:55:39.435259+0000 osd.5 (osd.5) 48 : cluster [DBG] 6.19 scrub ok 2021-03-24T21:55:40.489 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:40 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:55:40.489 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:40 smithi125 conmon[19936]: 55:40.106133+0000 mon.a (mon.0) 747 : cluster [DBG] osdmap e171: 8 total, 8 up, 6 in 2021-03-24T21:55:40.490 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:40 smithi125 conmon[24927]: cluster 2021-03-24T21:55:38.731542+0000 2021-03-24T21:55:40.490 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:40 smithi125 conmon[24927]: mgr.y (mgr.14144) 343 : cluster [DBG] pgmap v416: 148 pgs: 148 active+clean; 131 MiB data, 382 MiB used, 536 GiB / 536 GiB avail; 11 MiB/s rd, 550 KiB/s wr, 12 op/s 2021-03-24T21:55:40.490 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:40 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:55:40.491 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:40 smithi125 conmon[24927]: -24T21:55:38.773515+0000 osd.7 (osd.7) 39 : cluster [DBG] 2021-03-24T21:55:40.491 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:40 smithi125 conmon[24927]: purged_snaps scrub starts 2021-03-24T21:55:40.491 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:40 smithi125 conmon[24927]: cluster 2021-03-24T21:55:40.491 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:40 smithi125 conmon[24927]: 2021-03-24T21:55:38.773870+0000 osd.7 (osd.7) 2021-03-24T21:55:40.491 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:40 smithi125 conmon[24927]: 40 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:55:40.492 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:40 smithi125 conmon[24927]: cluster 2021-03-24T21:55:38 2021-03-24T21:55:40.492 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:40 smithi125 conmon[24927]: .974148+0000 osd.7 (osd.7) 41 : cluster [DBG] 3.d scrub ok 2021-03-24T21:55:40.492 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:40 smithi125 conmon[24927]: 2021-03-24T21:55:40.492 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:40 smithi125 conmon[24927]: cluster 2021-03-24T21:55:40.493 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:40 smithi125 conmon[24927]: 2021-03-24T21:55:39.261673+0000 osd.0 (osd. 2021-03-24T21:55:40.493 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:40 smithi125 conmon[24927]: 0) 50 : cluster [DBG] 6.0 scrub ok 2021-03-24T21:55:40.493 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:40 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:55:40.493 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:40 smithi125 conmon[24927]: -03-24T21:55:39.435259+0000 osd.5 2021-03-24T21:55:40.493 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:40 smithi125 conmon[24927]: (osd.5) 48 : cluster [DBG] 6.19 scrub ok 2021-03-24T21:55:40.494 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:40 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:55:40.494 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:40 smithi125 conmon[24927]: -24T21:55:40.106133+0000 mon.a (mon.0 2021-03-24T21:55:40.494 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:40 smithi125 conmon[24927]: ) 747 : cluster [DBG] osdmap e171: 8 total, 8 up, 6 in 2021-03-24T21:55:40.545 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:55:40.581 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:40.638 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:40.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:40.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:40.743 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:40.796 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:40.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:40.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:55:40.903 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:40.951 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:40.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:40.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:55:41.021 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2021-03-24T21:55:41.021 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T21:55:41.056 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:41.113 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:41.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:41.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:55:41.137 INFO:tasks.rados.rados.0.smithi125.stdout:358: done (2 left) 2021-03-24T21:55:41.138 INFO:tasks.rados.rados.0.smithi125.stdout:359: done (1 left) 2021-03-24T21:55:41.138 INFO:tasks.rados.rados.0.smithi125.stdout:360: done (0 left) 2021-03-24T21:55:41.138 INFO:tasks.rados.rados.0.smithi125.stdout:361: read oid 45 snap -1 2021-03-24T21:55:41.138 INFO:tasks.rados.rados.0.smithi125.stdout:361: expect deleted 2021-03-24T21:55:41.139 INFO:tasks.rados.rados.0.smithi125.stdout:362: delete oid 42 current snap is 35 2021-03-24T21:55:41.144 INFO:tasks.rados.rados.0.smithi125.stdout:362: done (1 left) 2021-03-24T21:55:41.144 INFO:tasks.rados.rados.0.smithi125.stdout:363: rollback oid 34 current snap is 35 2021-03-24T21:55:41.145 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 34 to 30 2021-03-24T21:55:41.145 INFO:tasks.rados.rados.0.smithi125.stdout:364: write (excl) oid 20 current snap is 35 2021-03-24T21:55:41.145 INFO:tasks.rados.rados.0.smithi125.stdout:364: seq_num 159 ranges {672313=691651,1887931=419038} 2021-03-24T21:55:41.153 INFO:tasks.rados.rados.0.smithi125.stdout:364: writing smithi12556470-20 from 672313 to 1363964 tid 1 2021-03-24T21:55:41.159 INFO:tasks.rados.rados.0.smithi125.stdout:364: writing smithi12556470-20 from 1887931 to 2306969 tid 2 2021-03-24T21:55:41.159 INFO:tasks.rados.rados.0.smithi125.stdout:365: write oid 37 current snap is 35 2021-03-24T21:55:41.159 INFO:tasks.rados.rados.0.smithi125.stdout:365: seq_num 160 ranges {689120=483253,1946413=679049} 2021-03-24T21:55:41.165 INFO:tasks.rados.rados.0.smithi125.stdout:365: writing smithi12556470-37 from 689120 to 1172373 tid 1 2021-03-24T21:55:41.171 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-24T21:55:41.174 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2021-03-24T21:55:41.174 INFO:tasks.rados.rados.0.smithi125.stdout:365: writing smithi12556470-37 from 1946413 to 2625462 tid 2 2021-03-24T21:55:41.175 INFO:tasks.rados.rados.0.smithi125.stdout:361: done (3 left) 2021-03-24T21:55:41.175 INFO:tasks.rados.rados.0.smithi125.stdout:366: write (excl) oid 8 current snap is 35 2021-03-24T21:55:41.175 INFO:tasks.rados.rados.0.smithi125.stdout:363: finishing rollback tid 0 to smithi12556470-34 2021-03-24T21:55:41.175 INFO:tasks.rados.rados.0.smithi125.stdout:366: seq_num 161 ranges {705927=558502,2004895=422707} 2021-03-24T21:55:41.181 INFO:tasks.rados.rados.0.smithi125.stdout:366: writing smithi12556470-8 from 705927 to 1264429 tid 1 2021-03-24T21:55:41.189 INFO:tasks.rados.rados.0.smithi125.stdout:366: writing smithi12556470-8 from 2004895 to 2427602 tid 2 2021-03-24T21:55:41.189 INFO:tasks.rados.rados.0.smithi125.stdout:367: snap_create 2021-03-24T21:55:41.189 INFO:tasks.rados.rados.0.smithi125.stdout:363: finishing rollback tid 1 to smithi12556470-34 2021-03-24T21:55:41.189 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 34 v 127 (ObjNum 1116645632 snap 21916 seq_num 2684356816) dirty dne 2021-03-24T21:55:41.189 INFO:tasks.rados.rados.0.smithi125.stdout:364: finishing write tid 1 to smithi12556470-20 2021-03-24T21:55:41.190 INFO:tasks.rados.rados.0.smithi125.stdout:364: finishing write tid 2 to smithi12556470-20 2021-03-24T21:55:41.190 INFO:tasks.rados.rados.0.smithi125.stdout:364: finishing write tid 3 to smithi12556470-20 2021-03-24T21:55:41.190 INFO:tasks.rados.rados.0.smithi125.stdout:364: finishing write tid 4 to smithi12556470-20 2021-03-24T21:55:41.190 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 20 v 100 (ObjNum 159 snap 35 seq_num 159) dirty exists 2021-03-24T21:55:41.191 INFO:tasks.rados.rados.0.smithi125.stdout:364: left oid 20 (ObjNum 159 snap 35 seq_num 159) 2021-03-24T21:55:41.191 INFO:tasks.rados.rados.0.smithi125.stdout:365: finishing write tid 1 to smithi12556470-37 2021-03-24T21:55:41.191 INFO:tasks.rados.rados.0.smithi125.stdout:365: finishing write tid 2 to smithi12556470-37 2021-03-24T21:55:41.191 INFO:tasks.rados.rados.0.smithi125.stdout:365: finishing write tid 3 to smithi12556470-37 2021-03-24T21:55:41.191 INFO:tasks.rados.rados.0.smithi125.stdout:365: finishing write tid 4 to smithi12556470-37 2021-03-24T21:55:41.192 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 37 v 56 (ObjNum 160 snap 35 seq_num 160) dirty exists 2021-03-24T21:55:41.192 INFO:tasks.rados.rados.0.smithi125.stdout:365: left oid 37 (ObjNum 160 snap 35 seq_num 160) 2021-03-24T21:55:41.192 INFO:tasks.rados.rados.0.smithi125.stdout:366: finishing write tid 1 to smithi12556470-8 2021-03-24T21:55:41.193 INFO:tasks.rados.rados.0.smithi125.stdout:366: finishing write tid 2 to smithi12556470-8 2021-03-24T21:55:41.193 INFO:tasks.rados.rados.0.smithi125.stdout:366: finishing write tid 3 to smithi12556470-8 2021-03-24T21:55:41.194 INFO:tasks.rados.rados.0.smithi125.stdout:366: finishing write tid 4 to smithi12556470-8 2021-03-24T21:55:41.194 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 8 v 30 (ObjNum 161 snap 35 seq_num 161) dirty exists 2021-03-24T21:55:41.194 INFO:tasks.rados.rados.0.smithi125.stdout:366: left oid 8 (ObjNum 161 snap 35 seq_num 161) 2021-03-24T21:55:41.219 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:41.267 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:41.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:41.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:55:41.364 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:41 smithi125 conmon[19936]: cluster 2021-03-24T21:55:40. 2021-03-24T21:55:41.364 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:41 smithi125 conmon[19936]: 456092+0000 osd.5 (osd.5) 49 : cluster [DBG] 5.12 scrub ok 2021-03-24T21:55:41.364 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:41 smithi125 conmon[19936]: cluster 2021-03-24T21:55:41.364 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:41 smithi125 conmon[19936]: 2021-03-24T21:55:41.132826+0000 mon.a (mon.0) 748 : cluster [DBG] 2021-03-24T21:55:41.365 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:41 smithi125 conmon[19936]: osdmap e172: 8 total, 8 up, 6 in 2021-03-24T21:55:41.365 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:41 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:55:41.365 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:41 smithi125 conmon[19936]: 03-24T21:55:41.337115+0000 osd.0 2021-03-24T21:55:41.365 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:41 smithi125 conmon[19936]: (osd.0) 51 : cluster [DBG] 4.1e deep-scrub ok 2021-03-24T21:55:41.373 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:41.424 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:41.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:41.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:41.530 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:41.542 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:41 smithi142 conmon[20125]: cluster 2021-03-24T21:55:40.456092+0000 2021-03-24T21:55:41.543 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:41 smithi142 conmon[20125]: osd.5 (osd.5) 49 : cluster [DBG] 5.12 scrub ok 2021-03-24T21:55:41.543 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:41 smithi142 conmon[20125]: cluster 2021-03-24T21:55:41. 2021-03-24T21:55:41.543 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:41 smithi142 conmon[20125]: 132826+0000 mon.a (mon.0) 748 : cluster [DBG] 2021-03-24T21:55:41.543 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:41 smithi142 conmon[20125]: osdmap e172: 8 total, 8 up, 6 in 2021-03-24T21:55:41.543 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:41 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:55:41.544 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:41 smithi142 conmon[20125]: -24T21:55:41.337115+0000 2021-03-24T21:55:41.544 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:41 smithi142 conmon[20125]: osd.0 (osd.0) 51 : cluster [DBG] 2021-03-24T21:55:41.544 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:41 smithi142 conmon[20125]: 4.1e deep-scrub ok 2021-03-24T21:55:41.582 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:41.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:41.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:55:41.630 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:41 smithi125 conmon[24927]: cluster 2021-03-24T21:55:40 2021-03-24T21:55:41.630 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:41 smithi125 conmon[24927]: .456092+0000 osd.5 (osd.5) 49 : cluster [DBG] 5.12 scrub ok 2021-03-24T21:55:41.630 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:41 smithi125 conmon[24927]: cluster 2021-03-24T21:55:41.631 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:41 smithi125 conmon[24927]: 2021-03-24T21:55:41.132826+0000 mon.a (mon. 2021-03-24T21:55:41.631 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:41 smithi125 conmon[24927]: 0) 748 : cluster [DBG] osdmap e172: 8 total, 8 up, 6 in 2021-03-24T21:55:41.631 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:41 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:55:41.631 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:41 smithi125 conmon[24927]: -03-24T21:55:41.337115 2021-03-24T21:55:41.632 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:41 smithi125 conmon[24927]: +0000 osd.0 (osd.0) 51 : 2021-03-24T21:55:41.632 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:41 smithi125 conmon[24927]: cluster [DBG] 4.1e deep-scrub ok 2021-03-24T21:55:41.686 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:41.746 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:41.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:41.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:41.852 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:41.914 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:41.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:41.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:55:42.021 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:42.082 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:42.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:42.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:55:42.112 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.4 dump_historic_ops 2021-03-24T21:55:42.138 INFO:tasks.rados.rados.0.smithi125.stdout:363: done (4 left) 2021-03-24T21:55:42.139 INFO:tasks.rados.rados.0.smithi125.stdout:364: done (3 left) 2021-03-24T21:55:42.139 INFO:tasks.rados.rados.0.smithi125.stdout:365: done (2 left) 2021-03-24T21:55:42.139 INFO:tasks.rados.rados.0.smithi125.stdout:366: done (1 left) 2021-03-24T21:55:42.139 INFO:tasks.rados.rados.0.smithi125.stdout:367: done (0 left) 2021-03-24T21:55:42.140 INFO:tasks.rados.rados.0.smithi125.stdout:368: delete oid 30 current snap is 36 2021-03-24T21:55:42.144 INFO:tasks.rados.rados.0.smithi125.stdout:368: done (0 left) 2021-03-24T21:55:42.144 INFO:tasks.rados.rados.0.smithi125.stdout:369: snap_remove snap 33 2021-03-24T21:55:42.145 INFO:teuthology.orchestra.run.smithi125.stderr:noscrub is set 2021-03-24T21:55:42.187 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:42.243 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:42.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:42.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:55:42.348 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:42.401 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:42.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:42.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:55:42.418 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T21:55:42.508 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:42.565 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:42.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:42.570 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[24927]: cluster 2021-03-24T21:55 2021-03-24T21:55:42.571 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[24927]: :40.731967+0000 mgr.y (mgr.14144) 344 : cluster [DBG] pgmap v419: 148 pgs: 148 active+clean; 142 MiB data, 415 MiB used, 536 GiB / 536 GiB avail; 18 MiB/s rd, 3.1 MiB/s wr, 24 op/s 2021-03-24T21:55:42.571 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[24927]: audit 2021-03-24T21:55:41.154034+0000 2021-03-24T21:55:42.571 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[24927]: mgr.y (mgr.14144) 345 : audit [DBG] from='client.14865 v1:172.21.15.125:0/2627732576' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T21:55:42.571 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:55:42.572 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[24927]: -03-24T21:55:41.503741+0000 osd.5 ( 2021-03-24T21:55:42.572 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[24927]: osd.5) 50 : cluster [DBG] 6.4 scrub ok 2021-03-24T21:55:42.572 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[24927]: audit 2021- 2021-03-24T21:55:42.572 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[24927]: 03-24T21:55:41.532151+0000 mon.b 2021-03-24T21:55:42.572 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[24927]: (mon.1) 31 : audit [INF] from='client.? v1:172.21.15.125:0/2129167754' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T21:55:42.573 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[24927]: audit 2021 2021-03-24T21:55:42.573 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[24927]: -03-24T21:55:41.532790+0000 2021-03-24T21:55:42.573 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[24927]: mon.a (mon.0) 749 : audit [INF] from='client.24787 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T21:55:42.573 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[24927]: cluster 2021-03-24T21:55:42.574 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[24927]: 2021-03-24T21:55:42.129025+0000 2021-03-24T21:55:42.574 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[24927]: mon.a (mon.0) 750 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:55:42.574 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[24927]: 2021-03-24T21:55:42.574 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[24927]: cluster 2021-03-24T21:55:42 2021-03-24T21:55:42.574 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[24927]: .135498+0000 osd.4 (osd.4) 52 : cluster 2021-03-24T21:55:42.575 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[24927]: [DBG] 3.14 scrub ok 2021-03-24T21:55:42.575 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[24927]: audit 2021-03-24T21: 2021-03-24T21:55:42.575 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[24927]: 55:42.137948+0000 mon.a (mon.0) 2021-03-24T21:55:42.575 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[24927]: 751 : audit [INF] from='client.24787 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T21:55:42.576 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:55:42.576 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[24927]: -24T21:55:42.138104+0000 mon.a ( 2021-03-24T21:55:42.576 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[24927]: mon.0) 752 : cluster [DBG] osdmap e173: 8 total, 8 up, 6 in 2021-03-24T21:55:42.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:55:42.577 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:55:42.578 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[19936]: :55:40.731967+0000 mgr.y (mgr.14144) 344 : cluster [DBG] pgmap v419: 148 pgs: 148 active+clean; 142 MiB data, 415 MiB used, 536 GiB / 536 GiB avail; 18 MiB/s rd, 3.1 MiB/s wr, 24 op/s 2021-03-24T21:55:42.578 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[19936]: audit 2021-03-24T21:55:42.578 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[19936]: 2021-03-24T21:55 2021-03-24T21:55:42.578 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[19936]: :41.154034+0000 mgr.y 2021-03-24T21:55:42.579 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[19936]: (mgr.14144) 345 : audit 2021-03-24T21:55:42.579 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[19936]: [DBG] from='client.14865 v1:172.21.15.125:0/2627732576' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T21:55:42.579 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[19936]: cluster 2021-03-24T21:55:42.579 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[19936]: 2021-03-24T21:55: 2021-03-24T21:55:42.579 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[19936]: 41.503741+0000 osd.5 (osd. 2021-03-24T21:55:42.580 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[19936]: 5) 50 : cluster [DBG] 6.4 scrub ok 2021-03-24T21:55:42.580 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[19936]: 2021-03-24T21:55:42.580 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[19936]: audit 2021-03-24T21:55:42.580 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[19936]: 2021-03-24T21:55 2021-03-24T21:55:42.581 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[19936]: :41.532151+0000 mon.b ( 2021-03-24T21:55:42.581 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[19936]: mon.1) 31 : audit [INF] 2021-03-24T21:55:42.581 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[19936]: from='client.? v1:172.21.15.125:0/2129167754' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T21:55:42.581 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[19936]: audit 2021-03-24T21:55 2021-03-24T21:55:42.582 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[19936]: :41.532790+0000 mon.a (mon.0) 2021-03-24T21:55:42.582 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[19936]: 749 : audit [INF] from='client.24787 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T21:55:42.582 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[19936]: cluster 2021-03-24T21:55:42.582 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[19936]: 2021-03-24T 2021-03-24T21:55:42.583 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[19936]: 21:55:42.129025+0000 mon.a (mon 2021-03-24T21:55:42.583 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[19936]: .0) 750 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:55:42.583 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:55:42.583 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[19936]: 03-24T21:55:42.135498+0000 2021-03-24T21:55:42.583 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[19936]: osd.4 (osd.4) 52 : cluster [DBG] 2021-03-24T21:55:42.584 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[19936]: 3.14 scrub ok 2021-03-24T21:55:42.584 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[19936]: audit 2021-03-24T21:55:42.584 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[19936]: 2021-03-24T21:55:42.137948+0000 2021-03-24T21:55:42.584 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[19936]: mon.a (mon.0) 751 : audit [INF] 2021-03-24T21:55:42.585 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[19936]: from='client.24787 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T21:55:42.585 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[19936]: cluster 2021-03-24T21:55:42.585 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[19936]: 2021-03-24T21:55:42.138104+0000 mon.a (mon.0) 752 : cluster [DBG] 2021-03-24T21:55:42.585 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:42 smithi125 conmon[19936]: osdmap e173: 8 total, 8 up, 6 in 2021-03-24T21:55:42.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:42 smithi142 conmon[20125]: cluster 2021-03-24T21:55:40.731967+0000 mgr.y (mgr.14144 2021-03-24T21:55:42.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:42 smithi142 conmon[20125]: ) 344 : cluster [DBG] pgmap v419: 148 pgs: 148 active+clean; 142 MiB data, 415 MiB used, 536 GiB / 536 GiB avail; 18 MiB/s rd, 3.1 MiB/s wr, 24 op/s 2021-03-24T21:55:42.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:42 smithi142 conmon[20125]: audit 2021-03-24T21:55:41.154034+0000 mgr.y (mgr.14144) 345 : audit [DBG] from='client.14865 v1:172.21.15.125:0/2627732576' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T21:55:42.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:42 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:55:42.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:42 smithi142 conmon[20125]: 21:55:41.503741+0000 osd.5 (osd.5) 50 : cluster [DBG] 6.4 scrub ok 2021-03-24T21:55:42.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:42 smithi142 conmon[20125]: audit 2021-03-24T 2021-03-24T21:55:42.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:42 smithi142 conmon[20125]: 21:55:41.532151+0000 mon.b (mon.1) 31 : audit [INF] from='client.? v1:172.21.15.125:0/2129167754' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T21:55:42.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:42 smithi142 conmon[20125]: audit 2021-03-24T21:55:41 2021-03-24T21:55:42.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:42 smithi142 conmon[20125]: .532790+0000 mon.a (mon.0) 749 : audit [INF] from='client.24787 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T21:55:42.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:42 smithi142 conmon[20125]: cluster 2021-03-24T21:55:42.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:42 smithi142 conmon[20125]: 2021-03-24T21:55:42.129025+0000 mon.a (mon.0) 750 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:55:42.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:42 smithi142 conmon[20125]: cluster 2021-03-24T21:55:42.135498+0000 2021-03-24T21:55:42.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:42 smithi142 conmon[20125]: osd.4 (osd.4) 52 : cluster [DBG] 3.14 scrub ok 2021-03-24T21:55:42.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:42 smithi142 conmon[20125]: audit 2021-03-24T21:55:42 2021-03-24T21:55:42.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:42 smithi142 conmon[20125]: .137948+0000 mon.a (mon.0) 751 : audit [INF] from='client.24787 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T21:55:42.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:42 smithi142 conmon[20125]: cluster 2021-03-24T21 2021-03-24T21:55:42.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:42 smithi142 conmon[20125]: :55:42.138104+0000 mon.a (mon.0) 752 : cluster [DBG] 2021-03-24T21:55:42.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:42 smithi142 conmon[20125]: osdmap e173: 8 total, 8 up, 6 in 2021-03-24T21:55:42.678 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:42.736 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:42.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:42.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:42.841 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:42.894 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:42.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:42.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:43.000 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:43.054 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:43.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:43.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:55:43.143 INFO:tasks.rados.rados.0.smithi125.stdout:369: done (0 left) 2021-03-24T21:55:43.143 INFO:tasks.rados.rados.0.smithi125.stdout:370: copy_from oid 46 from oid 21 current snap is 36 2021-03-24T21:55:43.144 INFO:tasks.rados.rados.0.smithi125.stdout:371: read oid 29 snap -1 2021-03-24T21:55:43.144 INFO:tasks.rados.rados.0.smithi125.stdout:371: expect (ObjNum 101 snap 20 seq_num 101) 2021-03-24T21:55:43.144 INFO:tasks.rados.rados.0.smithi125.stdout:372: write (excl) oid 33 current snap is 36 2021-03-24T21:55:43.144 INFO:tasks.rados.rados.0.smithi125.stdout:372: seq_num 163 ranges {739541=709000,2121859=426376} 2021-03-24T21:55:43.155 INFO:tasks.rados.rados.0.smithi125.stdout:372: writing smithi12556470-33 from 739541 to 1448541 tid 1 2021-03-24T21:55:43.160 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:43.163 INFO:tasks.rados.rados.0.smithi125.stdout:372: writing smithi12556470-33 from 2121859 to 2548235 tid 2 2021-03-24T21:55:43.163 INFO:tasks.rados.rados.0.smithi125.stdout:371: expect (ObjNum 101 snap 20 seq_num 101) 2021-03-24T21:55:43.218 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:43.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:43.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:55:43.329 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:43.335 INFO:tasks.rados.rados.0.smithi125.stdout:370: finishing copy_from to smithi12556470-46 2021-03-24T21:55:43.335 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 46 v 49 (ObjNum 83 snap 10 seq_num 83) dirty exists 2021-03-24T21:55:43.336 INFO:tasks.rados.rados.0.smithi125.stdout:370: finishing copy_from racing read to smithi12556470-46 2021-03-24T21:55:43.336 INFO:tasks.rados.rados.0.smithi125.stdout:372: finishing write tid 1 to smithi12556470-33 2021-03-24T21:55:43.336 INFO:tasks.rados.rados.0.smithi125.stdout:372: finishing write tid 2 to smithi12556470-33 2021-03-24T21:55:43.336 INFO:tasks.rados.rados.0.smithi125.stdout:372: finishing write tid 3 to smithi12556470-33 2021-03-24T21:55:43.337 INFO:tasks.rados.rados.0.smithi125.stdout:372: finishing write tid 4 to smithi12556470-33 2021-03-24T21:55:43.337 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 33 v 125 (ObjNum 163 snap 36 seq_num 163) dirty exists 2021-03-24T21:55:43.337 INFO:tasks.rados.rados.0.smithi125.stdout:372: left oid 33 (ObjNum 163 snap 36 seq_num 163) 2021-03-24T21:55:43.337 INFO:tasks.rados.rados.0.smithi125.stdout:370: done (2 left) 2021-03-24T21:55:43.338 INFO:tasks.rados.rados.0.smithi125.stdout:371: done (1 left) 2021-03-24T21:55:43.338 INFO:tasks.rados.rados.0.smithi125.stdout:372: done (0 left) 2021-03-24T21:55:43.338 INFO:tasks.rados.rados.0.smithi125.stdout:373: read oid 15 snap -1 2021-03-24T21:55:43.338 INFO:tasks.rados.rados.0.smithi125.stdout:373: expect (ObjNum 150 snap 30 seq_num 150) 2021-03-24T21:55:43.338 INFO:tasks.rados.rados.0.smithi125.stdout:374: write oid 39 current snap is 36 2021-03-24T21:55:43.339 INFO:tasks.rados.rados.0.smithi125.stdout:374: seq_num 164 ranges {756348=784249,2063988=686387} 2021-03-24T21:55:43.352 INFO:tasks.rados.rados.0.smithi125.stdout:374: writing smithi12556470-39 from 756348 to 1540597 tid 1 2021-03-24T21:55:43.359 INFO:tasks.rados.rados.0.smithi125.stdout:374: writing smithi12556470-39 from 2063988 to 2750375 tid 2 2021-03-24T21:55:43.359 INFO:tasks.rados.rados.0.smithi125.stdout:375: copy_from oid 22 from oid 26 current snap is 36 2021-03-24T21:55:43.360 INFO:tasks.rados.rados.0.smithi125.stdout:373: expect (ObjNum 150 snap 30 seq_num 150) 2021-03-24T21:55:43.385 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:43.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:43.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:55:43.490 INFO:tasks.rados.rados.0.smithi125.stdout:374: finishing write tid 1 to smithi12556470-39 2021-03-24T21:55:43.491 INFO:tasks.rados.rados.0.smithi125.stdout:374: finishing write tid 2 to smithi12556470-39 2021-03-24T21:55:43.491 INFO:tasks.rados.rados.0.smithi125.stdout:374: finishing write tid 3 to smithi12556470-39 2021-03-24T21:55:43.491 INFO:tasks.rados.rados.0.smithi125.stdout:374: finishing write tid 4 to smithi12556470-39 2021-03-24T21:55:43.492 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 39 v 128 (ObjNum 164 snap 36 seq_num 164) dirty exists 2021-03-24T21:55:43.492 INFO:tasks.rados.rados.0.smithi125.stdout:374: left oid 39 (ObjNum 164 snap 36 seq_num 164) 2021-03-24T21:55:43.492 INFO:tasks.rados.rados.0.smithi125.stdout:375: finishing copy_from to smithi12556470-22 2021-03-24T21:55:43.493 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 22 v 92 (ObjNum 155 snap 34 seq_num 155) dirty exists 2021-03-24T21:55:43.493 INFO:tasks.rados.rados.0.smithi125.stdout:375: finishing copy_from racing read to smithi12556470-22 2021-03-24T21:55:43.493 INFO:tasks.rados.rados.0.smithi125.stdout:373: done (2 left) 2021-03-24T21:55:43.493 INFO:tasks.rados.rados.0.smithi125.stdout:374: done (1 left) 2021-03-24T21:55:43.493 INFO:tasks.rados.rados.0.smithi125.stdout:375: done (0 left) 2021-03-24T21:55:43.494 INFO:tasks.rados.rados.0.smithi125.stdout:376: delete oid 10 current snap is 36 2021-03-24T21:55:43.494 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:43.495 INFO:tasks.rados.rados.0.smithi125.stdout:376: done (0 left) 2021-03-24T21:55:43.495 INFO:tasks.rados.rados.0.smithi125.stdout:377: read oid 15 snap -1 2021-03-24T21:55:43.496 INFO:tasks.rados.rados.0.smithi125.stdout:377: expect (ObjNum 150 snap 30 seq_num 150) 2021-03-24T21:55:43.496 INFO:tasks.rados.rados.0.smithi125.stdout:378: delete oid 36 current snap is 36 2021-03-24T21:55:43.498 INFO:tasks.rados.rados.0.smithi125.stdout:377: expect (ObjNum 150 snap 30 seq_num 150) 2021-03-24T21:55:43.551 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:43.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:43.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:55:43.635 INFO:tasks.rados.rados.0.smithi125.stdout:377: done (1 left) 2021-03-24T21:55:43.635 INFO:tasks.rados.rados.0.smithi125.stdout:378: done (0 left) 2021-03-24T21:55:43.636 INFO:tasks.rados.rados.0.smithi125.stdout:379: rollback oid 36 current snap is 36 2021-03-24T21:55:43.636 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 36 to 30 2021-03-24T21:55:43.636 INFO:tasks.rados.rados.0.smithi125.stdout:380: rollback oid 12 current snap is 36 2021-03-24T21:55:43.636 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 12 to 35 2021-03-24T21:55:43.637 INFO:tasks.rados.rados.0.smithi125.stdout:381: write oid 34 current snap is 36 2021-03-24T21:55:43.637 INFO:tasks.rados.rados.0.smithi125.stdout:381: seq_num 167 ranges {406769=609996,1439434=433714,2303522=503247} 2021-03-24T21:55:43.647 INFO:tasks.rados.rados.0.smithi125.stdout:381: writing smithi12556470-34 from 406769 to 1016765 tid 1 2021-03-24T21:55:43.654 INFO:tasks.rados.rados.0.smithi125.stdout:381: writing smithi12556470-34 from 1439434 to 1873148 tid 2 2021-03-24T21:55:43.659 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:43.661 INFO:tasks.rados.rados.0.smithi125.stdout:381: writing smithi12556470-34 from 2303522 to 2806769 tid 3 2021-03-24T21:55:43.662 INFO:tasks.rados.rados.0.smithi125.stdout:382: read oid 6 snap 35 2021-03-24T21:55:43.662 INFO:tasks.rados.rados.0.smithi125.stdout:382: expect (ObjNum 126 snap 23 seq_num 126) 2021-03-24T21:55:43.663 INFO:tasks.rados.rados.0.smithi125.stdout:383: copy_from oid 14 from oid 13 current snap is 36 2021-03-24T21:55:43.663 INFO:tasks.rados.rados.0.smithi125.stdout:380: finishing rollback tid 1 to smithi12556470-12 2021-03-24T21:55:43.664 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 12 v 112 (ObjNum 1116381296 snap 21916 seq_num 0) dirty dne 2021-03-24T21:55:43.664 INFO:tasks.rados.rados.0.smithi125.stdout:379: finishing rollback tid 1 to smithi12556470-36 2021-03-24T21:55:43.665 INFO:tasks.rados.rados.0.smithi125.stdout:379: finishing rollback tid 2 to smithi12556470-36 2021-03-24T21:55:43.665 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 36 v 98 (ObjNum 120 snap 23 seq_num 120) dirty exists 2021-03-24T21:55:43.665 INFO:tasks.rados.rados.0.smithi125.stdout:381: finishing write tid 1 to smithi12556470-34 2021-03-24T21:55:43.666 INFO:tasks.rados.rados.0.smithi125.stdout:381: finishing write tid 2 to smithi12556470-34 2021-03-24T21:55:43.666 INFO:tasks.rados.rados.0.smithi125.stdout:379: done (4 left) 2021-03-24T21:55:43.667 INFO:tasks.rados.rados.0.smithi125.stdout:380: done (3 left) 2021-03-24T21:55:43.667 INFO:tasks.rados.rados.0.smithi125.stdout:384: snap_remove snap 23 2021-03-24T21:55:43.684 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":420,"stamp":"2021-03-24T21:55:41.129469+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":148625463,"num_objects":382,"num_object_clones":23,"num_object_copies":1146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":382,"num_whiteouts":7,"num_read":2035,"num_read_kb":619208,"num_write":1178,"num_write_kb":239453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":324,"num_bytes_recovered":35484837,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1811,"ondisk_log_size":1811,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":425408,"kb_used_data":343056,"kb_used_omap":0,"kb_used_meta":82304,"kb_avail":562045504,"statfs":{"total":575970213888,"available":575534596096,"internally_reserved":0,"allocated":351289344,"data_stored":396440583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":84279296},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13847042,"num_objects":7,"num_object_clones":7,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":-1,"num_read":71,"num_read_kb":73269,"num_write":27,"num_write_kb":12837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.013566"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":"128","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.116783+0000","last_change":"2021-03-24T21:54:50.292202+0000","last_active":"2021-03-24T21:55:37.116783+0000","last_peered":"2021-03-24T21:55:37.116783+0000","last_clean":"2021-03-24T21:55:37.116783+0000","last_became_active":"2021-03-24T21:54:38.111248+0000","last_became_peered":"2021-03-24T21:54:38.111248+0000","last_unstale":"2021-03-24T21:55:37.116783+0000","last_undegraded":"2021-03-24T21:55:37.116783+0000","last_fullsized":"2021-03-24T21:55:37.116783+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:50.292025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:54:50.292025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"44","reported_epoch":"169","state":"active+clean","last_fresh":"2021-03-24T21:55:38.134863+0000","last_change":"2021-03-24T21:55:10.100067+0000","last_active":"2021-03-24T21:55:38.134863+0000","last_peered":"2021-03-24T21:55:38.134863+0000","last_clean":"2021-03-24T21:55:38.134863+0000","last_became_active":"2021-03-24T21:55:10.099666+0000","last_became_peered":"2021-03-24T21:55:10.099666+0000","last_unstale":"2021-03-24T21:55:38.134863+0000","last_undegraded":"2021-03-24T21:55:38.134863+0000","last_fullsized":"2021-03-24T21:55:38.134863+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:14.710709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:14.710709+0000","last_clean_scrub_stamp":"2021-03-24T21:54:14.710709+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":"125","reported_epoch":"169","state":"active+clean","last_fresh":"2021-03-24T21:55:38.294845+0000","last_change":"2021-03-24T21:55:38.294845+0000","last_active":"2021-03-24T21:55:38.294845+0000","last_peered":"2021-03-24T21:55:38.294845+0000","last_clean":"2021-03-24T21:55:38.294845+0000","last_became_active":"2021-03-24T21:51:45.581000+0000","last_became_peered":"2021-03-24T21:51:45.581000+0000","last_unstale":"2021-03-24T21:55:38.294845+0000","last_undegraded":"2021-03-24T21:55:38.294845+0000","last_fullsized":"2021-03-24T21:55:38.294845+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:38.294685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:38.294685+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"69'15","reported_seq":"208","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.116144+0000","last_change":"2021-03-24T21:54:39.222600+0000","last_active":"2021-03-24T21:55:37.116144+0000","last_peered":"2021-03-24T21:55:37.116144+0000","last_clean":"2021-03-24T21:55:37.116144+0000","last_became_active":"2021-03-24T21:54:38.127426+0000","last_became_peered":"2021-03-24T21:54:38.127426+0000","last_unstale":"2021-03-24T21:55:37.116144+0000","last_undegraded":"2021-03-24T21:55:37.116144+0000","last_fullsized":"2021-03-24T21:55:37.116144+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:53:41.295875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:41.295875+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2076,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"137","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.122408+0000","last_change":"2021-03-24T21:54:43.482408+0000","last_active":"2021-03-24T21:55:37.122408+0000","last_peered":"2021-03-24T21:55:37.122408+0000","last_clean":"2021-03-24T21:55:37.122408+0000","last_became_active":"2021-03-24T21:54:38.130293+0000","last_became_peered":"2021-03-24T21:54:38.130293+0000","last_unstale":"2021-03-24T21:55:37.122408+0000","last_undegraded":"2021-03-24T21:55:37.122408+0000","last_fullsized":"2021-03-24T21:55:37.122408+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:43.482307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:54:43.482307+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"134","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.116430+0000","last_change":"2021-03-24T21:54:38.120961+0000","last_active":"2021-03-24T21:55:37.116430+0000","last_peered":"2021-03-24T21:55:37.116430+0000","last_clean":"2021-03-24T21:55:37.116430+0000","last_became_active":"2021-03-24T21:54:38.120575+0000","last_became_peered":"2021-03-24T21:54:38.120575+0000","last_unstale":"2021-03-24T21:55:37.116430+0000","last_undegraded":"2021-03-24T21:55:37.116430+0000","last_fullsized":"2021-03-24T21:55:37.116430+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:48.329480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:48.329480+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":"143","reported_epoch":"169","state":"active+clean","last_fresh":"2021-03-24T21:55:38.433064+0000","last_change":"2021-03-24T21:55:38.433064+0000","last_active":"2021-03-24T21:55:38.433064+0000","last_peered":"2021-03-24T21:55:38.433064+0000","last_clean":"2021-03-24T21:55:38.433064+0000","last_became_active":"2021-03-24T21:54:38.130163+0000","last_became_peered":"2021-03-24T21:54:38.130163+0000","last_unstale":"2021-03-24T21:55:38.433064+0000","last_undegraded":"2021-03-24T21:55:38.433064+0000","last_fullsized":"2021-03-24T21:55:38.433064+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:38.432918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:38.432918+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"69'20","reported_seq":"158","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.119566+0000","last_change":"2021-03-24T21:54:54.541380+0000","last_active":"2021-03-24T21:55:37.119566+0000","last_peered":"2021-03-24T21:55:37.119566+0000","last_clean":"2021-03-24T21:55:37.119566+0000","last_became_active":"2021-03-24T21:51:47.405776+0000","last_became_peered":"2021-03-24T21:51:47.405776+0000","last_unstale":"2021-03-24T21:55:37.119566+0000","last_undegraded":"2021-03-24T21:55:37.119566+0000","last_fullsized":"2021-03-24T21:55:37.119566+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'20","last_scrub_stamp":"2021-03-24T21:54:54.541183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:54:54.541183+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":22,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"128","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.123977+0000","last_change":"2021-03-24T21:54:49.409465+0000","last_active":"2021-03-24T21:55:37.123977+0000","last_peered":"2021-03-24T21:55:37.123977+0000","last_clean":"2021-03-24T21:55:37.123977+0000","last_became_active":"2021-03-24T21:54:38.127696+0000","last_became_peered":"2021-03-24T21:54:38.127696+0000","last_unstale":"2021-03-24T21:55:37.123977+0000","last_undegraded":"2021-03-24T21:55:37.123977+0000","last_fullsized":"2021-03-24T21:55:37.123977+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:49.409299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:54:49.409299+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"123","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.518914+0000","last_change":"2021-03-24T21:54:44.879188+0000","last_active":"2021-03-24T21:55:37.518914+0000","last_peered":"2021-03-24T21:55:37.518914+0000","last_clean":"2021-03-24T21:55:37.518914+0000","last_became_active":"2021-03-24T21:51:49.413342+0000","last_became_peered":"2021-03-24T21:51:49.413342+0000","last_unstale":"2021-03-24T21:55:37.518914+0000","last_undegraded":"2021-03-24T21:55:37.518914+0000","last_fullsized":"2021-03-24T21:55:37.518914+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:44.878935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:54:44.878935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":"150","reported_epoch":"166","state":"active+clean","last_fresh":"2021-03-24T21:55:35.105251+0000","last_change":"2021-03-24T21:55:10.097750+0000","last_active":"2021-03-24T21:55:35.105251+0000","last_peered":"2021-03-24T21:55:35.105251+0000","last_clean":"2021-03-24T21:55:35.105251+0000","last_became_active":"2021-03-24T21:55:10.097431+0000","last_became_peered":"2021-03-24T21:55:10.097431+0000","last_unstale":"2021-03-24T21:55:35.105251+0000","last_undegraded":"2021-03-24T21:55:35.105251+0000","last_fullsized":"2021-03-24T21:55:35.105251+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:43.877604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:54:43.877604+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"69'13","reported_seq":"147","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.117496+0000","last_change":"2021-03-24T21:55:31.085064+0000","last_active":"2021-03-24T21:55:37.117496+0000","last_peered":"2021-03-24T21:55:37.117496+0000","last_clean":"2021-03-24T21:55:37.117496+0000","last_became_active":"2021-03-24T21:51:47.407488+0000","last_became_peered":"2021-03-24T21:51:47.407488+0000","last_unstale":"2021-03-24T21:55:37.117496+0000","last_undegraded":"2021-03-24T21:55:37.117496+0000","last_fullsized":"2021-03-24T21:55:37.117496+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2021-03-24T21:55:31.084873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:31.084873+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"125","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.518532+0000","last_change":"2021-03-24T21:54:38.122726+0000","last_active":"2021-03-24T21:55:37.518532+0000","last_peered":"2021-03-24T21:55:37.518532+0000","last_clean":"2021-03-24T21:55:37.518532+0000","last_became_active":"2021-03-24T21:54:38.122349+0000","last_became_peered":"2021-03-24T21:54:38.122349+0000","last_unstale":"2021-03-24T21:55:37.518532+0000","last_undegraded":"2021-03-24T21:55:37.518532+0000","last_fullsized":"2021-03-24T21:55:37.518532+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:23.895966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:23.895966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"131","reported_epoch":"169","state":"active+clean","last_fresh":"2021-03-24T21:55:38.135021+0000","last_change":"2021-03-24T21:54:57.741556+0000","last_active":"2021-03-24T21:55:38.135021+0000","last_peered":"2021-03-24T21:55:38.135021+0000","last_clean":"2021-03-24T21:55:38.135021+0000","last_became_active":"2021-03-24T21:54:38.137706+0000","last_became_peered":"2021-03-24T21:54:38.137706+0000","last_unstale":"2021-03-24T21:55:38.135021+0000","last_undegraded":"2021-03-24T21:55:38.135021+0000","last_fullsized":"2021-03-24T21:55:38.135021+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:57.741464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:54:57.741464+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":"107","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.117170+0000","last_change":"2021-03-24T21:55:10.099630+0000","last_active":"2021-03-24T21:55:37.117170+0000","last_peered":"2021-03-24T21:55:37.117170+0000","last_clean":"2021-03-24T21:55:37.117170+0000","last_became_active":"2021-03-24T21:55:10.098790+0000","last_became_peered":"2021-03-24T21:55:10.098790+0000","last_unstale":"2021-03-24T21:55:37.117170+0000","last_undegraded":"2021-03-24T21:55:37.117170+0000","last_fullsized":"2021-03-24T21:55:37.117170+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:09.127715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:54:09.127715+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"91'9","reported_seq":"150","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.117129+0000","last_change":"2021-03-24T21:54:39.157590+0000","last_active":"2021-03-24T21:55:37.117129+0000","last_peered":"2021-03-24T21:55:37.117129+0000","last_clean":"2021-03-24T21:55:37.117129+0000","last_became_active":"2021-03-24T21:54:38.127055+0000","last_became_peered":"2021-03-24T21:54:38.127055+0000","last_unstale":"2021-03-24T21:55:37.117129+0000","last_undegraded":"2021-03-24T21:55:37.117129+0000","last_fullsized":"2021-03-24T21:55:37.117129+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2021-03-24T21:53:33.290000+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2021-03-24T21:53:33.290000+0000","last_clean_scrub_stamp":"2021-03-24T21:53:33.290000+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":8,"num_write":7,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"123","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.118004+0000","last_change":"2021-03-24T21:54:24.199810+0000","last_active":"2021-03-24T21:55:37.118004+0000","last_peered":"2021-03-24T21:55:37.118004+0000","last_clean":"2021-03-24T21:55:37.118004+0000","last_became_active":"2021-03-24T21:54:24.199444+0000","last_became_peered":"2021-03-24T21:54:24.199444+0000","last_unstale":"2021-03-24T21:55:37.118004+0000","last_undegraded":"2021-03-24T21:55:37.118004+0000","last_fullsized":"2021-03-24T21:55:37.118004+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:39.277425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:39.277425+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"77","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.117203+0000","last_change":"2021-03-24T21:54:49.323954+0000","last_active":"2021-03-24T21:55:37.117203+0000","last_peered":"2021-03-24T21:55:37.117203+0000","last_clean":"2021-03-24T21:55:37.117203+0000","last_became_active":"2021-03-24T21:54:38.138620+0000","last_became_peered":"2021-03-24T21:54:38.138620+0000","last_unstale":"2021-03-24T21:55:37.117203+0000","last_undegraded":"2021-03-24T21:55:37.117203+0000","last_fullsized":"2021-03-24T21:55:37.117203+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:49.323789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:54:49.323789+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":"125","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.124751+0000","last_change":"2021-03-24T21:54:43.406949+0000","last_active":"2021-03-24T21:55:37.124751+0000","last_peered":"2021-03-24T21:55:37.124751+0000","last_clean":"2021-03-24T21:55:37.124751+0000","last_became_active":"2021-03-24T21:51:45.381533+0000","last_became_peered":"2021-03-24T21:51:45.381533+0000","last_unstale":"2021-03-24T21:55:37.124751+0000","last_undegraded":"2021-03-24T21:55:37.124751+0000","last_fullsized":"2021-03-24T21:55:37.124751+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:43.406875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:54:43.406875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"69'15","reported_seq":"196","reported_epoch":"166","state":"active+clean","last_fresh":"2021-03-24T21:55:35.105355+0000","last_change":"2021-03-24T21:54:39.063715+0000","last_active":"2021-03-24T21:55:35.105355+0000","last_peered":"2021-03-24T21:55:35.105355+0000","last_clean":"2021-03-24T21:55:35.105355+0000","last_became_active":"2021-03-24T21:54:38.140589+0000","last_became_peered":"2021-03-24T21:54:38.140589+0000","last_unstale":"2021-03-24T21:55:35.105355+0000","last_undegraded":"2021-03-24T21:55:35.105355+0000","last_fullsized":"2021-03-24T21:55:35.105355+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:53:58.587152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:58.587152+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"132","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.122388+0000","last_change":"2021-03-24T21:54:38.144481+0000","last_active":"2021-03-24T21:55:37.122388+0000","last_peered":"2021-03-24T21:55:37.122388+0000","last_clean":"2021-03-24T21:55:37.122388+0000","last_became_active":"2021-03-24T21:54:38.143563+0000","last_became_peered":"2021-03-24T21:54:38.143563+0000","last_unstale":"2021-03-24T21:55:37.122388+0000","last_undegraded":"2021-03-24T21:55:37.122388+0000","last_fullsized":"2021-03-24T21:55:37.122388+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:44.657761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:44.657761+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"121","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.118121+0000","last_change":"2021-03-24T21:55:02.129906+0000","last_active":"2021-03-24T21:55:37.118121+0000","last_peered":"2021-03-24T21:55:37.118121+0000","last_clean":"2021-03-24T21:55:37.118121+0000","last_became_active":"2021-03-24T21:51:49.410920+0000","last_became_peered":"2021-03-24T21:51:49.410920+0000","last_unstale":"2021-03-24T21:55:37.118121+0000","last_undegraded":"2021-03-24T21:55:37.118121+0000","last_fullsized":"2021-03-24T21:55:37.118121+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:02.129719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:55:02.129719+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":"134","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.117479+0000","last_change":"2021-03-24T21:54:38.144167+0000","last_active":"2021-03-24T21:55:37.117479+0000","last_peered":"2021-03-24T21:55:37.117479+0000","last_clean":"2021-03-24T21:55:37.117479+0000","last_became_active":"2021-03-24T21:54:38.143876+0000","last_became_peered":"2021-03-24T21:54:38.143876+0000","last_unstale":"2021-03-24T21:55:37.117479+0000","last_undegraded":"2021-03-24T21:55:37.117479+0000","last_fullsized":"2021-03-24T21:55:37.117479+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:11.286003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:52:54.212459+0000","last_clean_scrub_stamp":"2021-03-24T21:54:11.286003+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"69'16","reported_seq":"178","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.121891+0000","last_change":"2021-03-24T21:54:38.191975+0000","last_active":"2021-03-24T21:55:37.121891+0000","last_peered":"2021-03-24T21:55:37.121891+0000","last_clean":"2021-03-24T21:55:37.121891+0000","last_became_active":"2021-03-24T21:54:38.128246+0000","last_became_peered":"2021-03-24T21:54:38.128246+0000","last_unstale":"2021-03-24T21:55:37.121891+0000","last_undegraded":"2021-03-24T21:55:37.121891+0000","last_fullsized":"2021-03-24T21:55:37.121891+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T21:53:42.611540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:42.611540+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":37,"num_read_kb":20,"num_write":18,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":844,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"120","reported_epoch":"169","state":"active+clean","last_fresh":"2021-03-24T21:55:38.135751+0000","last_change":"2021-03-24T21:55:29.900103+0000","last_active":"2021-03-24T21:55:38.135751+0000","last_peered":"2021-03-24T21:55:38.135751+0000","last_clean":"2021-03-24T21:55:38.135751+0000","last_became_active":"2021-03-24T21:51:51.761492+0000","last_became_peered":"2021-03-24T21:51:51.761492+0000","last_unstale":"2021-03-24T21:55:38.135751+0000","last_undegraded":"2021-03-24T21:55:38.135751+0000","last_fullsized":"2021-03-24T21:55:38.135751+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:29.899923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:55:29.899923+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"102'3","reported_seq":"106","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.118509+0000","last_change":"2021-03-24T21:54:24.283503+0000","last_active":"2021-03-24T21:55:37.118509+0000","last_peered":"2021-03-24T21:55:37.118509+0000","last_clean":"2021-03-24T21:55:37.118509+0000","last_became_active":"2021-03-24T21:54:24.188093+0000","last_became_peered":"2021-03-24T21:54:24.188093+0000","last_unstale":"2021-03-24T21:55:37.118509+0000","last_undegraded":"2021-03-24T21:55:37.118509+0000","last_fullsized":"2021-03-24T21:55:37.118509+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2021-03-24T21:54:15.969746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:54:15.969746+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":"76","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.119062+0000","last_change":"2021-03-24T21:54:48.541576+0000","last_active":"2021-03-24T21:55:37.119062+0000","last_peered":"2021-03-24T21:55:37.119062+0000","last_clean":"2021-03-24T21:55:37.119062+0000","last_became_active":"2021-03-24T21:54:38.130034+0000","last_became_peered":"2021-03-24T21:54:38.130034+0000","last_unstale":"2021-03-24T21:55:37.119062+0000","last_undegraded":"2021-03-24T21:55:37.119062+0000","last_fullsized":"2021-03-24T21:55:37.119062+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:48.541415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:54:48.541415+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"89'27","reported_seq":"156","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.124863+0000","last_change":"2021-03-24T21:53:21.585108+0000","last_active":"2021-03-24T21:55:37.124863+0000","last_peered":"2021-03-24T21:55:37.124863+0000","last_clean":"2021-03-24T21:55:37.124863+0000","last_became_active":"2021-03-24T21:51:47.688995+0000","last_became_peered":"2021-03-24T21:51:47.688995+0000","last_unstale":"2021-03-24T21:55:37.124863+0000","last_undegraded":"2021-03-24T21:55:37.124863+0000","last_fullsized":"2021-03-24T21:55:37.124863+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'27","last_scrub_stamp":"2021-03-24T21:53:21.584999+0000","last_deep_scrub":"89'27","last_deep_scrub_stamp":"2021-03-24T21:53:21.584999+0000","last_clean_scrub_stamp":"2021-03-24T21:53:21.584999+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"77","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.519544+0000","last_change":"2021-03-24T21:54:49.992150+0000","last_active":"2021-03-24T21:55:37.519544+0000","last_peered":"2021-03-24T21:55:37.519544+0000","last_clean":"2021-03-24T21:55:37.519544+0000","last_became_active":"2021-03-24T21:54:38.113849+0000","last_became_peered":"2021-03-24T21:54:38.113849+0000","last_unstale":"2021-03-24T21:55:37.519544+0000","last_undegraded":"2021-03-24T21:55:37.519544+0000","last_fullsized":"2021-03-24T21:55:37.519544+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:49.991966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:54:49.991966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"116","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.125228+0000","last_change":"2021-03-24T21:53:30.660884+0000","last_active":"2021-03-24T21:55:37.125228+0000","last_peered":"2021-03-24T21:55:37.125228+0000","last_clean":"2021-03-24T21:55:37.125228+0000","last_became_active":"2021-03-24T21:51:49.409745+0000","last_became_peered":"2021-03-24T21:51:49.409745+0000","last_unstale":"2021-03-24T21:55:37.125228+0000","last_undegraded":"2021-03-24T21:55:37.125228+0000","last_fullsized":"2021-03-24T21:55:37.125228+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:30.660636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:30.660636+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":"134","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.118711+0000","last_change":"2021-03-24T21:55:32.566923+0000","last_active":"2021-03-24T21:55:37.118711+0000","last_peered":"2021-03-24T21:55:37.118711+0000","last_clean":"2021-03-24T21:55:37.118711+0000","last_became_active":"2021-03-24T21:54:38.145870+0000","last_became_peered":"2021-03-24T21:54:38.145870+0000","last_unstale":"2021-03-24T21:55:37.118711+0000","last_undegraded":"2021-03-24T21:55:37.118711+0000","last_fullsized":"2021-03-24T21:55:37.118711+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:32.566747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:32.566747+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"69'17","reported_seq":"92","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.124673+0000","last_change":"2021-03-24T21:55:32.214663+0000","last_active":"2021-03-24T21:55:37.124673+0000","last_peered":"2021-03-24T21:55:37.124673+0000","last_clean":"2021-03-24T21:55:37.124673+0000","last_became_active":"2021-03-24T21:55:10.101198+0000","last_became_peered":"2021-03-24T21:55:10.101198+0000","last_unstale":"2021-03-24T21:55:37.124673+0000","last_undegraded":"2021-03-24T21:55:37.124673+0000","last_fullsized":"2021-03-24T21:55:37.124673+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'17","last_scrub_stamp":"2021-03-24T21:55:32.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:32.214502+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":37,"num_read_kb":18,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"115","reported_epoch":"169","state":"active+clean","last_fresh":"2021-03-24T21:55:38.135089+0000","last_change":"2021-03-24T21:53:27.877846+0000","last_active":"2021-03-24T21:55:38.135089+0000","last_peered":"2021-03-24T21:55:38.135089+0000","last_clean":"2021-03-24T21:55:38.135089+0000","last_became_active":"2021-03-24T21:51:51.762150+0000","last_became_peered":"2021-03-24T21:51:51.762150+0000","last_unstale":"2021-03-24T21:55:38.135089+0000","last_undegraded":"2021-03-24T21:55:38.135089+0000","last_fullsized":"2021-03-24T21:55:38.135089+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:27.877627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:27.877627+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"83","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.116183+0000","last_change":"2021-03-24T21:55:26.087761+0000","last_active":"2021-03-24T21:55:37.116183+0000","last_peered":"2021-03-24T21:55:37.116183+0000","last_clean":"2021-03-24T21:55:37.116183+0000","last_became_active":"2021-03-24T21:54:24.187568+0000","last_became_peered":"2021-03-24T21:54:24.187568+0000","last_unstale":"2021-03-24T21:55:37.116183+0000","last_undegraded":"2021-03-24T21:55:37.116183+0000","last_fullsized":"2021-03-24T21:55:37.116183+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:26.087584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:55:26.087584+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":"125","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.125601+0000","last_change":"2021-03-24T21:54:48.394336+0000","last_active":"2021-03-24T21:55:37.125601+0000","last_peered":"2021-03-24T21:55:37.125601+0000","last_clean":"2021-03-24T21:55:37.125601+0000","last_became_active":"2021-03-24T21:51:45.379789+0000","last_became_peered":"2021-03-24T21:51:45.379789+0000","last_unstale":"2021-03-24T21:55:37.125601+0000","last_undegraded":"2021-03-24T21:55:37.125601+0000","last_fullsized":"2021-03-24T21:55:37.125601+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:48.394164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:54:48.394164+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"69'12","reported_seq":"238","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.118969+0000","last_change":"2021-03-24T21:54:39.142185+0000","last_active":"2021-03-24T21:55:37.118969+0000","last_peered":"2021-03-24T21:55:37.118969+0000","last_clean":"2021-03-24T21:55:37.118969+0000","last_became_active":"2021-03-24T21:54:38.134132+0000","last_became_peered":"2021-03-24T21:54:38.134132+0000","last_unstale":"2021-03-24T21:55:37.118969+0000","last_undegraded":"2021-03-24T21:55:37.118969+0000","last_fullsized":"2021-03-24T21:55:37.118969+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2021-03-24T21:54:14.574739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:54:14.574739+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":12,"num_write":12,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":3756,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"123","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.122182+0000","last_change":"2021-03-24T21:54:24.204479+0000","last_active":"2021-03-24T21:55:37.122182+0000","last_peered":"2021-03-24T21:55:37.122182+0000","last_clean":"2021-03-24T21:55:37.122182+0000","last_became_active":"2021-03-24T21:54:24.202686+0000","last_became_peered":"2021-03-24T21:54:24.202686+0000","last_unstale":"2021-03-24T21:55:37.122182+0000","last_undegraded":"2021-03-24T21:55:37.122182+0000","last_fullsized":"2021-03-24T21:55:37.122182+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:47.540793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:47.540793+0000","last_clean_scrub_stamp":"2021-03-24T21:53:47.540793+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"131","reported_epoch":"169","state":"active+clean","last_fresh":"2021-03-24T21:55:38.134530+0000","last_change":"2021-03-24T21:55:23.872913+0000","last_active":"2021-03-24T21:55:38.134530+0000","last_peered":"2021-03-24T21:55:38.134530+0000","last_clean":"2021-03-24T21:55:38.134530+0000","last_became_active":"2021-03-24T21:54:24.194423+0000","last_became_peered":"2021-03-24T21:54:24.194423+0000","last_unstale":"2021-03-24T21:55:38.134530+0000","last_undegraded":"2021-03-24T21:55:38.134530+0000","last_fullsized":"2021-03-24T21:55:38.134530+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:23.872742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:55:23.872742+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":"136","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.519632+0000","last_change":"2021-03-24T21:55:26.503047+0000","last_active":"2021-03-24T21:55:37.519632+0000","last_peered":"2021-03-24T21:55:37.519632+0000","last_clean":"2021-03-24T21:55:37.519632+0000","last_became_active":"2021-03-24T21:54:38.121323+0000","last_became_peered":"2021-03-24T21:54:38.121323+0000","last_unstale":"2021-03-24T21:55:37.519632+0000","last_undegraded":"2021-03-24T21:55:37.519632+0000","last_fullsized":"2021-03-24T21:55:37.519632+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:26.502895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:26.502895+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"86'22","reported_seq":"183","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.119323+0000","last_change":"2021-03-24T21:54:51.585204+0000","last_active":"2021-03-24T21:55:37.119323+0000","last_peered":"2021-03-24T21:55:37.119323+0000","last_clean":"2021-03-24T21:55:37.119323+0000","last_became_active":"2021-03-24T21:54:24.188790+0000","last_became_peered":"2021-03-24T21:54:24.188790+0000","last_unstale":"2021-03-24T21:55:37.119323+0000","last_undegraded":"2021-03-24T21:55:37.119323+0000","last_fullsized":"2021-03-24T21:55:37.119323+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'22","last_scrub_stamp":"2021-03-24T21:54:51.585058+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2021-03-24T21:52:54.597505+0000","last_clean_scrub_stamp":"2021-03-24T21:54:51.585058+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":16,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":580,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"128","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.123710+0000","last_change":"2021-03-24T21:54:54.371278+0000","last_active":"2021-03-24T21:55:37.123710+0000","last_peered":"2021-03-24T21:55:37.123710+0000","last_clean":"2021-03-24T21:55:37.123710+0000","last_became_active":"2021-03-24T21:54:38.126662+0000","last_became_peered":"2021-03-24T21:54:38.126662+0000","last_unstale":"2021-03-24T21:55:37.123710+0000","last_undegraded":"2021-03-24T21:55:37.123710+0000","last_fullsized":"2021-03-24T21:55:37.123710+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:54.371106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:54:54.371106+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"125","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.121438+0000","last_change":"2021-03-24T21:54:38.129428+0000","last_active":"2021-03-24T21:55:37.121438+0000","last_peered":"2021-03-24T21:55:37.121438+0000","last_clean":"2021-03-24T21:55:37.121438+0000","last_became_active":"2021-03-24T21:54:38.128067+0000","last_became_peered":"2021-03-24T21:54:38.128067+0000","last_unstale":"2021-03-24T21:55:37.121438+0000","last_undegraded":"2021-03-24T21:55:37.121438+0000","last_fullsized":"2021-03-24T21:55:37.121438+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:31.665434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:31.665434+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":"121","reported_epoch":"169","state":"active+clean","last_fresh":"2021-03-24T21:55:38.136453+0000","last_change":"2021-03-24T21:53:08.841487+0000","last_active":"2021-03-24T21:55:38.136453+0000","last_peered":"2021-03-24T21:55:38.136453+0000","last_clean":"2021-03-24T21:55:38.136453+0000","last_became_active":"2021-03-24T21:51:45.379383+0000","last_became_peered":"2021-03-24T21:51:45.379383+0000","last_unstale":"2021-03-24T21:55:38.136453+0000","last_undegraded":"2021-03-24T21:55:38.136453+0000","last_fullsized":"2021-03-24T21:55:38.136453+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:08.841310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:08.841310+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"69'15","reported_seq":"131","reported_epoch":"166","state":"active+clean","last_fresh":"2021-03-24T21:55:35.104119+0000","last_change":"2021-03-24T21:54:38.182009+0000","last_active":"2021-03-24T21:55:35.104119+0000","last_peered":"2021-03-24T21:55:35.104119+0000","last_clean":"2021-03-24T21:55:35.104119+0000","last_became_active":"2021-03-24T21:54:38.122707+0000","last_became_peered":"2021-03-24T21:54:38.122707+0000","last_unstale":"2021-03-24T21:55:35.104119+0000","last_undegraded":"2021-03-24T21:55:35.104119+0000","last_fullsized":"2021-03-24T21:55:35.104119+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:54:06.651102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:54:06.651102+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":34,"num_read_kb":16,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"119","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.118966+0000","last_change":"2021-03-24T21:55:22.555707+0000","last_active":"2021-03-24T21:55:37.118966+0000","last_peered":"2021-03-24T21:55:37.118966+0000","last_clean":"2021-03-24T21:55:37.118966+0000","last_became_active":"2021-03-24T21:51:51.764871+0000","last_became_peered":"2021-03-24T21:51:51.764871+0000","last_unstale":"2021-03-24T21:55:37.118966+0000","last_undegraded":"2021-03-24T21:55:37.118966+0000","last_fullsized":"2021-03-24T21:55:37.118966+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:22.555624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:23.707914+0000","last_clean_scrub_stamp":"2021-03-24T21:55:22.555624+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"72","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.520379+0000","last_change":"2021-03-24T21:54:38.122046+0000","last_active":"2021-03-24T21:55:37.520379+0000","last_peered":"2021-03-24T21:55:37.520379+0000","last_clean":"2021-03-24T21:55:37.520379+0000","last_became_active":"2021-03-24T21:54:38.120989+0000","last_became_peered":"2021-03-24T21:54:38.120989+0000","last_unstale":"2021-03-24T21:55:37.520379+0000","last_undegraded":"2021-03-24T21:55:37.520379+0000","last_fullsized":"2021-03-24T21:55:37.520379+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:58.783414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:58.783414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":"129","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.117397+0000","last_change":"2021-03-24T21:54:24.181804+0000","last_active":"2021-03-24T21:55:37.117397+0000","last_peered":"2021-03-24T21:55:37.117397+0000","last_clean":"2021-03-24T21:55:37.117397+0000","last_became_active":"2021-03-24T21:54:24.181343+0000","last_became_peered":"2021-03-24T21:54:24.181343+0000","last_unstale":"2021-03-24T21:55:37.117397+0000","last_undegraded":"2021-03-24T21:55:37.117397+0000","last_fullsized":"2021-03-24T21:55:37.117397+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:36.316893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:36.316893+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"69'15","reported_seq":"208","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.118026+0000","last_change":"2021-03-24T21:55:37.076650+0000","last_active":"2021-03-24T21:55:37.118026+0000","last_peered":"2021-03-24T21:55:37.118026+0000","last_clean":"2021-03-24T21:55:37.118026+0000","last_became_active":"2021-03-24T21:54:38.118330+0000","last_became_peered":"2021-03-24T21:54:38.118330+0000","last_unstale":"2021-03-24T21:55:37.118026+0000","last_undegraded":"2021-03-24T21:55:37.118026+0000","last_fullsized":"2021-03-24T21:55:37.118026+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:55:37.076511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:37.076511+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"119","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.124430+0000","last_change":"2021-03-24T21:54:44.392954+0000","last_active":"2021-03-24T21:55:37.124430+0000","last_peered":"2021-03-24T21:55:37.124430+0000","last_clean":"2021-03-24T21:55:37.124430+0000","last_became_active":"2021-03-24T21:51:51.756584+0000","last_became_peered":"2021-03-24T21:51:51.756584+0000","last_unstale":"2021-03-24T21:55:37.124430+0000","last_undegraded":"2021-03-24T21:55:37.124430+0000","last_fullsized":"2021-03-24T21:55:37.124430+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:44.392808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:54:44.392808+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"109'4","reported_seq":"184","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.519859+0000","last_change":"2021-03-24T21:55:29.541436+0000","last_active":"2021-03-24T21:55:37.519859+0000","last_peered":"2021-03-24T21:55:37.519859+0000","last_clean":"2021-03-24T21:55:37.519859+0000","last_became_active":"2021-03-24T21:54:38.130026+0000","last_became_peered":"2021-03-24T21:54:38.130026+0000","last_unstale":"2021-03-24T21:55:37.519859+0000","last_undegraded":"2021-03-24T21:55:37.519859+0000","last_fullsized":"2021-03-24T21:55:37.519859+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'4","last_scrub_stamp":"2021-03-24T21:55:29.541312+0000","last_deep_scrub":"86'3","last_deep_scrub_stamp":"2021-03-24T21:53:08.815163+0000","last_clean_scrub_stamp":"2021-03-24T21:55:29.541312+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":"121","reported_epoch":"169","state":"active+clean","last_fresh":"2021-03-24T21:55:38.135163+0000","last_change":"2021-03-24T21:53:15.868395+0000","last_active":"2021-03-24T21:55:38.135163+0000","last_peered":"2021-03-24T21:55:38.135163+0000","last_clean":"2021-03-24T21:55:38.135163+0000","last_became_active":"2021-03-24T21:51:45.579271+0000","last_became_peered":"2021-03-24T21:51:45.579271+0000","last_unstale":"2021-03-24T21:55:38.135163+0000","last_undegraded":"2021-03-24T21:55:38.135163+0000","last_fullsized":"2021-03-24T21:55:38.135163+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:15.868303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:15.868303+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"69'14","reported_seq":"178","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.122633+0000","last_change":"2021-03-24T21:55:25.582566+0000","last_active":"2021-03-24T21:55:37.122633+0000","last_peered":"2021-03-24T21:55:37.122633+0000","last_clean":"2021-03-24T21:55:37.122633+0000","last_became_active":"2021-03-24T21:54:38.143783+0000","last_became_peered":"2021-03-24T21:54:38.143783+0000","last_unstale":"2021-03-24T21:55:37.122633+0000","last_undegraded":"2021-03-24T21:55:37.122633+0000","last_fullsized":"2021-03-24T21:55:37.122633+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'14","last_scrub_stamp":"2021-03-24T21:55:25.582388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:25.582388+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":27,"num_read_kb":14,"num_write":14,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"119","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.118588+0000","last_change":"2021-03-24T21:54:13.291633+0000","last_active":"2021-03-24T21:55:37.118588+0000","last_peered":"2021-03-24T21:55:37.118588+0000","last_clean":"2021-03-24T21:55:37.118588+0000","last_became_active":"2021-03-24T21:51:51.762876+0000","last_became_peered":"2021-03-24T21:51:51.762876+0000","last_unstale":"2021-03-24T21:55:37.118588+0000","last_undegraded":"2021-03-24T21:55:37.118588+0000","last_fullsized":"2021-03-24T21:55:37.118588+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:13.291461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:54:13.291461+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"86'3","reported_seq":"170","reported_epoch":"169","state":"active+clean","last_fresh":"2021-03-24T21:55:38.134948+0000","last_change":"2021-03-24T21:55:01.577952+0000","last_active":"2021-03-24T21:55:38.134948+0000","last_peered":"2021-03-24T21:55:38.134948+0000","last_clean":"2021-03-24T21:55:38.134948+0000","last_became_active":"2021-03-24T21:51:49.419797+0000","last_became_peered":"2021-03-24T21:51:49.419797+0000","last_unstale":"2021-03-24T21:55:38.134948+0000","last_undegraded":"2021-03-24T21:55:38.134948+0000","last_fullsized":"2021-03-24T21:55:38.134948+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'3","last_scrub_stamp":"2021-03-24T21:55:01.577769+0000","last_deep_scrub":"86'3","last_deep_scrub_stamp":"2021-03-24T21:52:56.678111+0000","last_clean_scrub_stamp":"2021-03-24T21:55:01.577769+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":"125","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.124909+0000","last_change":"2021-03-24T21:54:05.473528+0000","last_active":"2021-03-24T21:55:37.124909+0000","last_peered":"2021-03-24T21:55:37.124909+0000","last_clean":"2021-03-24T21:55:37.124909+0000","last_became_active":"2021-03-24T21:51:45.381516+0000","last_became_peered":"2021-03-24T21:51:45.381516+0000","last_unstale":"2021-03-24T21:55:37.124909+0000","last_undegraded":"2021-03-24T21:55:37.124909+0000","last_fullsized":"2021-03-24T21:55:37.124909+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:05.473348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:05.473348+0000","last_clean_scrub_stamp":"2021-03-24T21:54:05.473348+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"69'13","reported_seq":"149","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.520010+0000","last_change":"2021-03-24T21:55:31.561861+0000","last_active":"2021-03-24T21:55:37.520010+0000","last_peered":"2021-03-24T21:55:37.520010+0000","last_clean":"2021-03-24T21:55:37.520010+0000","last_became_active":"2021-03-24T21:51:47.689288+0000","last_became_peered":"2021-03-24T21:51:47.689288+0000","last_unstale":"2021-03-24T21:55:37.520010+0000","last_undegraded":"2021-03-24T21:55:37.520010+0000","last_fullsized":"2021-03-24T21:55:37.520010+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2021-03-24T21:55:31.561663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:31.561663+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"112","reported_epoch":"166","state":"active+clean","last_fresh":"2021-03-24T21:55:35.105680+0000","last_change":"2021-03-24T21:53:15.623047+0000","last_active":"2021-03-24T21:55:35.105680+0000","last_peered":"2021-03-24T21:55:35.105680+0000","last_clean":"2021-03-24T21:55:35.105680+0000","last_became_active":"2021-03-24T21:51:51.754085+0000","last_became_peered":"2021-03-24T21:51:51.754085+0000","last_unstale":"2021-03-24T21:55:35.105680+0000","last_undegraded":"2021-03-24T21:55:35.105680+0000","last_fullsized":"2021-03-24T21:55:35.105680+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:15.622946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:15.622946+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"130","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.122692+0000","last_change":"2021-03-24T21:54:55.493891+0000","last_active":"2021-03-24T21:55:37.122692+0000","last_peered":"2021-03-24T21:55:37.122692+0000","last_clean":"2021-03-24T21:55:37.122692+0000","last_became_active":"2021-03-24T21:54:38.129390+0000","last_became_peered":"2021-03-24T21:54:38.129390+0000","last_unstale":"2021-03-24T21:55:37.122692+0000","last_undegraded":"2021-03-24T21:55:37.122692+0000","last_fullsized":"2021-03-24T21:55:37.122692+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:55.493713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:55.493713+0000","last_clean_scrub_stamp":"2021-03-24T21:54:55.493713+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":"135","reported_epoch":"169","state":"active+clean","last_fresh":"2021-03-24T21:55:38.134649+0000","last_change":"2021-03-24T21:55:31.912637+0000","last_active":"2021-03-24T21:55:38.134649+0000","last_peered":"2021-03-24T21:55:38.134649+0000","last_clean":"2021-03-24T21:55:38.134649+0000","last_became_active":"2021-03-24T21:54:24.196290+0000","last_became_peered":"2021-03-24T21:54:24.196290+0000","last_unstale":"2021-03-24T21:55:38.134649+0000","last_undegraded":"2021-03-24T21:55:38.134649+0000","last_fullsized":"2021-03-24T21:55:38.134649+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:31.912451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:31.912451+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"69'16","reported_seq":"153","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.125063+0000","last_change":"2021-03-24T21:54:19.588964+0000","last_active":"2021-03-24T21:55:37.125063+0000","last_peered":"2021-03-24T21:55:37.125063+0000","last_clean":"2021-03-24T21:55:37.125063+0000","last_became_active":"2021-03-24T21:51:47.687595+0000","last_became_peered":"2021-03-24T21:51:47.687595+0000","last_unstale":"2021-03-24T21:55:37.125063+0000","last_undegraded":"2021-03-24T21:55:37.125063+0000","last_fullsized":"2021-03-24T21:55:37.125063+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T21:54:19.588797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:54:19.588797+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":38,"num_read_kb":18,"num_write":17,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"124","reported_epoch":"169","state":"active+clean","last_fresh":"2021-03-24T21:55:38.136022+0000","last_change":"2021-03-24T21:54:24.188745+0000","last_active":"2021-03-24T21:55:38.136022+0000","last_peered":"2021-03-24T21:55:38.136022+0000","last_clean":"2021-03-24T21:55:38.136022+0000","last_became_active":"2021-03-24T21:54:24.188386+0000","last_became_peered":"2021-03-24T21:54:24.188386+0000","last_unstale":"2021-03-24T21:55:38.136022+0000","last_undegraded":"2021-03-24T21:55:38.136022+0000","last_fullsized":"2021-03-24T21:55:38.136022+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:24.913238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:24.913238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"132","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.519585+0000","last_change":"2021-03-24T21:55:27.547657+0000","last_active":"2021-03-24T21:55:37.519585+0000","last_peered":"2021-03-24T21:55:37.519585+0000","last_clean":"2021-03-24T21:55:37.519585+0000","last_became_active":"2021-03-24T21:54:38.128409+0000","last_became_peered":"2021-03-24T21:54:38.128409+0000","last_unstale":"2021-03-24T21:55:37.519585+0000","last_undegraded":"2021-03-24T21:55:37.519585+0000","last_fullsized":"2021-03-24T21:55:37.519585+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:27.547493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:55:27.547493+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":"84","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.123899+0000","last_change":"2021-03-24T21:54:52.385672+0000","last_active":"2021-03-24T21:55:37.123899+0000","last_peered":"2021-03-24T21:55:37.123899+0000","last_clean":"2021-03-24T21:55:37.123899+0000","last_became_active":"2021-03-24T21:54:24.201388+0000","last_became_peered":"2021-03-24T21:54:24.201388+0000","last_unstale":"2021-03-24T21:55:37.123899+0000","last_undegraded":"2021-03-24T21:55:37.123899+0000","last_fullsized":"2021-03-24T21:55:37.123899+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:52.385567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:54:52.385567+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"69'7","reported_seq":"148","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.122656+0000","last_change":"2021-03-24T21:54:39.199780+0000","last_active":"2021-03-24T21:55:37.122656+0000","last_peered":"2021-03-24T21:55:37.122656+0000","last_clean":"2021-03-24T21:55:37.122656+0000","last_became_active":"2021-03-24T21:54:38.144098+0000","last_became_peered":"2021-03-24T21:54:38.144098+0000","last_unstale":"2021-03-24T21:55:37.122656+0000","last_undegraded":"2021-03-24T21:55:37.122656+0000","last_fullsized":"2021-03-24T21:55:37.122656+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'7","last_scrub_stamp":"2021-03-24T21:53:13.645928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:13.645928+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":7,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":158,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"163'119","reported_seq":"375","reported_epoch":"169","state":"active+clean","last_fresh":"2021-03-24T21:55:38.142108+0000","last_change":"2021-03-24T21:55:35.119532+0000","last_active":"2021-03-24T21:55:38.142108+0000","last_peered":"2021-03-24T21:55:38.142108+0000","last_clean":"2021-03-24T21:55:38.142108+0000","last_became_active":"2021-03-24T21:54:38.117586+0000","last_became_peered":"2021-03-24T21:54:38.117586+0000","last_unstale":"2021-03-24T21:55:38.142108+0000","last_undegraded":"2021-03-24T21:55:38.142108+0000","last_fullsized":"2021-03-24T21:55:38.142108+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":119,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16903446,"num_objects":7,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":92,"num_read_kb":93146,"num_write":52,"num_write_kb":22900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":6771983,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"36","length":"2"},{"start":"3b","length":"1"}]},{"pgid":"6.a","version":"0'0","reported_seq":"89","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.125279+0000","last_change":"2021-03-24T21:55:10.097929+0000","last_active":"2021-03-24T21:55:37.125279+0000","last_peered":"2021-03-24T21:55:37.125279+0000","last_clean":"2021-03-24T21:55:37.125279+0000","last_became_active":"2021-03-24T21:55:10.097570+0000","last_became_peered":"2021-03-24T21:55:10.097570+0000","last_unstale":"2021-03-24T21:55:37.125279+0000","last_undegraded":"2021-03-24T21:55:37.125279+0000","last_fullsized":"2021-03-24T21:55:37.125279+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:55.067965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:54:55.067965+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"89'4","reported_seq":"126","reported_epoch":"169","state":"active+clean","last_fresh":"2021-03-24T21:55:38.136573+0000","last_change":"2021-03-24T21:53:19.926732+0000","last_active":"2021-03-24T21:55:38.136573+0000","last_peered":"2021-03-24T21:55:38.136573+0000","last_clean":"2021-03-24T21:55:38.136573+0000","last_became_active":"2021-03-24T21:51:45.385110+0000","last_became_peered":"2021-03-24T21:51:45.385110+0000","last_unstale":"2021-03-24T21:55:38.136573+0000","last_undegraded":"2021-03-24T21:55:38.136573+0000","last_fullsized":"2021-03-24T21:55:38.136573+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'4","last_scrub_stamp":"2021-03-24T21:53:19.926637+0000","last_deep_scrub":"89'4","last_deep_scrub_stamp":"2021-03-24T21:53:19.926637+0000","last_clean_scrub_stamp":"2021-03-24T21:53:19.926637+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"97'28","reported_seq":"197","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.518588+0000","last_change":"2021-03-24T21:54:24.220625+0000","last_active":"2021-03-24T21:55:37.518588+0000","last_peered":"2021-03-24T21:55:37.518588+0000","last_clean":"2021-03-24T21:55:37.518588+0000","last_became_active":"2021-03-24T21:54:24.190817+0000","last_became_peered":"2021-03-24T21:54:24.190817+0000","last_unstale":"2021-03-24T21:55:37.518588+0000","last_undegraded":"2021-03-24T21:55:37.518588+0000","last_fullsized":"2021-03-24T21:55:37.518588+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'28","last_scrub_stamp":"2021-03-24T21:54:19.157265+0000","last_deep_scrub":"97'28","last_deep_scrub_stamp":"2021-03-24T21:54:19.157265+0000","last_clean_scrub_stamp":"2021-03-24T21:54:19.157265+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":20,"num_write":19,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"109'4","reported_seq":"196","reported_epoch":"169","state":"active+clean","last_fresh":"2021-03-24T21:55:38.897373+0000","last_change":"2021-03-24T21:54:38.223920+0000","last_active":"2021-03-24T21:55:38.897373+0000","last_peered":"2021-03-24T21:55:38.897373+0000","last_clean":"2021-03-24T21:55:38.897373+0000","last_became_active":"2021-03-24T21:54:38.125497+0000","last_became_peered":"2021-03-24T21:54:38.125497+0000","last_unstale":"2021-03-24T21:55:38.897373+0000","last_undegraded":"2021-03-24T21:55:38.897373+0000","last_fullsized":"2021-03-24T21:55:38.897373+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2021-03-24T21:54:04.706153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:54:04.706153+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"155'51","reported_seq":"253","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.120928+0000","last_change":"2021-03-24T21:55:35.123781+0000","last_active":"2021-03-24T21:55:37.120928+0000","last_peered":"2021-03-24T21:55:37.120928+0000","last_clean":"2021-03-24T21:55:37.120928+0000","last_became_active":"2021-03-24T21:54:22.167632+0000","last_became_peered":"2021-03-24T21:54:22.167632+0000","last_unstale":"2021-03-24T21:55:37.120928+0000","last_undegraded":"2021-03-24T21:55:37.120928+0000","last_fullsized":"2021-03-24T21:55:37.120928+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3882384,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":10,"num_read_kb":6,"num_write":30,"num_write_kb":7401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"36","length":"2"},{"start":"3b","length":"1"}]},{"pgid":"6.b","version":"0'0","reported_seq":"128","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.121633+0000","last_change":"2021-03-24T21:55:24.548918+0000","last_active":"2021-03-24T21:55:37.121633+0000","last_peered":"2021-03-24T21:55:37.121633+0000","last_clean":"2021-03-24T21:55:37.121633+0000","last_became_active":"2021-03-24T21:54:38.129824+0000","last_became_peered":"2021-03-24T21:54:38.129824+0000","last_unstale":"2021-03-24T21:55:37.121633+0000","last_undegraded":"2021-03-24T21:55:37.121633+0000","last_fullsized":"2021-03-24T21:55:37.121633+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:24.548819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:22.724840+0000","last_clean_scrub_stamp":"2021-03-24T21:55:24.548819+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":"135","reported_epoch":"169","state":"active+clean","last_fresh":"2021-03-24T21:55:38.136623+0000","last_change":"2021-03-24T21:54:38.109846+0000","last_active":"2021-03-24T21:55:38.136623+0000","last_peered":"2021-03-24T21:55:38.136623+0000","last_clean":"2021-03-24T21:55:38.136623+0000","last_became_active":"2021-03-24T21:54:38.109439+0000","last_became_peered":"2021-03-24T21:54:38.109439+0000","last_unstale":"2021-03-24T21:55:38.136623+0000","last_undegraded":"2021-03-24T21:55:38.136623+0000","last_fullsized":"2021-03-24T21:55:38.136623+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:51.651164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:51.651164+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"69'15","reported_seq":"174","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.117560+0000","last_change":"2021-03-24T21:54:41.209644+0000","last_active":"2021-03-24T21:55:37.117560+0000","last_peered":"2021-03-24T21:55:37.117560+0000","last_clean":"2021-03-24T21:55:37.117560+0000","last_became_active":"2021-03-24T21:54:38.139863+0000","last_became_peered":"2021-03-24T21:54:38.139863+0000","last_unstale":"2021-03-24T21:55:37.117560+0000","last_undegraded":"2021-03-24T21:55:37.117560+0000","last_fullsized":"2021-03-24T21:55:37.117560+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:53:37.334345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:37.334345+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":19,"num_write":17,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":692,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"128","reported_epoch":"166","state":"active+clean","last_fresh":"2021-03-24T21:55:35.099126+0000","last_change":"2021-03-24T21:55:26.017611+0000","last_active":"2021-03-24T21:55:35.099126+0000","last_peered":"2021-03-24T21:55:35.099126+0000","last_clean":"2021-03-24T21:55:35.099126+0000","last_became_active":"2021-03-24T21:54:38.124317+0000","last_became_peered":"2021-03-24T21:54:38.124317+0000","last_unstale":"2021-03-24T21:55:35.099126+0000","last_undegraded":"2021-03-24T21:55:35.099126+0000","last_fullsized":"2021-03-24T21:55:35.099126+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:26.017426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:55:26.017426+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"166'118","reported_seq":"353","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.124238+0000","last_change":"2021-03-24T21:55:35.126923+0000","last_active":"2021-03-24T21:55:37.124238+0000","last_peered":"2021-03-24T21:55:37.124238+0000","last_clean":"2021-03-24T21:55:37.124238+0000","last_became_active":"2021-03-24T21:54:38.137395+0000","last_became_peered":"2021-03-24T21:54:38.137395+0000","last_unstale":"2021-03-24T21:55:37.124238+0000","last_undegraded":"2021-03-24T21:55:37.124238+0000","last_fullsized":"2021-03-24T21:55:37.124238+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":118,"ondisk_log_size":118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11966518,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":60,"num_read_kb":43168,"num_write":52,"num_write_kb":20244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4396481,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"36","length":"2"},{"start":"3b","length":"1"}]},{"pgid":"6.8","version":"0'0","reported_seq":"120","reported_epoch":"169","state":"active+clean","last_fresh":"2021-03-24T21:55:38.134006+0000","last_change":"2021-03-24T21:54:46.859206+0000","last_active":"2021-03-24T21:55:38.134006+0000","last_peered":"2021-03-24T21:55:38.134006+0000","last_clean":"2021-03-24T21:55:38.134006+0000","last_became_active":"2021-03-24T21:51:51.762513+0000","last_became_peered":"2021-03-24T21:51:51.762513+0000","last_unstale":"2021-03-24T21:55:38.134006+0000","last_undegraded":"2021-03-24T21:55:38.134006+0000","last_fullsized":"2021-03-24T21:55:38.134006+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:46.859034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:54:46.859034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"119","reported_epoch":"166","state":"active+clean","last_fresh":"2021-03-24T21:55:35.103864+0000","last_change":"2021-03-24T21:54:51.202761+0000","last_active":"2021-03-24T21:55:35.103864+0000","last_peered":"2021-03-24T21:55:35.103864+0000","last_clean":"2021-03-24T21:55:35.103864+0000","last_became_active":"2021-03-24T21:51:49.417699+0000","last_became_peered":"2021-03-24T21:51:49.417699+0000","last_unstale":"2021-03-24T21:55:35.103864+0000","last_undegraded":"2021-03-24T21:55:35.103864+0000","last_fullsized":"2021-03-24T21:55:35.103864+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:51.202632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:54:51.202632+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":"135","reported_epoch":"169","state":"active+clean","last_fresh":"2021-03-24T21:55:38.974283+0000","last_change":"2021-03-24T21:55:38.974245+0000","last_active":"2021-03-24T21:55:38.974283+0000","last_peered":"2021-03-24T21:55:38.974283+0000","last_clean":"2021-03-24T21:55:38.974283+0000","last_became_active":"2021-03-24T21:54:24.189299+0000","last_became_peered":"2021-03-24T21:54:24.189299+0000","last_unstale":"2021-03-24T21:55:38.974283+0000","last_undegraded":"2021-03-24T21:55:38.974283+0000","last_fullsized":"2021-03-24T21:55:38.974283+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:38.974155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:38.974155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"69'23","reported_seq":"153","reported_epoch":"169","state":"active+clean","last_fresh":"2021-03-24T21:55:38.135005+0000","last_change":"2021-03-24T21:54:39.420930+0000","last_active":"2021-03-24T21:55:38.135005+0000","last_peered":"2021-03-24T21:55:38.135005+0000","last_clean":"2021-03-24T21:55:38.135005+0000","last_became_active":"2021-03-24T21:54:38.138701+0000","last_became_peered":"2021-03-24T21:54:38.138701+0000","last_unstale":"2021-03-24T21:55:38.135005+0000","last_undegraded":"2021-03-24T21:55:38.135005+0000","last_fullsized":"2021-03-24T21:55:38.135005+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'23","last_scrub_stamp":"2021-03-24T21:54:18.973113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:54:18.973113+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":43,"num_read_kb":24,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":732,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"166'56","reported_seq":"259","reported_epoch":"166","state":"active+clean","last_fresh":"2021-03-24T21:55:35.107251+0000","last_change":"2021-03-24T21:55:35.107251+0000","last_active":"2021-03-24T21:55:35.107251+0000","last_peered":"2021-03-24T21:55:35.107251+0000","last_clean":"2021-03-24T21:55:35.107251+0000","last_became_active":"2021-03-24T21:54:22.166423+0000","last_became_peered":"2021-03-24T21:54:22.166423+0000","last_unstale":"2021-03-24T21:55:35.107251+0000","last_undegraded":"2021-03-24T21:55:35.107251+0000","last_fullsized":"2021-03-24T21:55:35.107251+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7694518,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":13222,"num_write":29,"num_write_kb":8491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"36","length":"2"},{"start":"3b","length":"1"}]},{"pgid":"6.9","version":"0'0","reported_seq":"119","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.124173+0000","last_change":"2021-03-24T21:55:33.246334+0000","last_active":"2021-03-24T21:55:37.124173+0000","last_peered":"2021-03-24T21:55:37.124173+0000","last_clean":"2021-03-24T21:55:37.124173+0000","last_became_active":"2021-03-24T21:51:51.763077+0000","last_became_peered":"2021-03-24T21:51:51.763077+0000","last_unstale":"2021-03-24T21:55:37.124173+0000","last_undegraded":"2021-03-24T21:55:37.124173+0000","last_fullsized":"2021-03-24T21:55:37.124173+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:33.246205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:35.636299+0000","last_clean_scrub_stamp":"2021-03-24T21:55:33.246205+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"119","reported_epoch":"166","state":"active+clean","last_fresh":"2021-03-24T21:55:35.104265+0000","last_change":"2021-03-24T21:54:54.139755+0000","last_active":"2021-03-24T21:55:35.104265+0000","last_peered":"2021-03-24T21:55:35.104265+0000","last_clean":"2021-03-24T21:55:35.104265+0000","last_became_active":"2021-03-24T21:51:49.411921+0000","last_became_peered":"2021-03-24T21:51:49.411921+0000","last_unstale":"2021-03-24T21:55:35.104265+0000","last_undegraded":"2021-03-24T21:55:35.104265+0000","last_fullsized":"2021-03-24T21:55:35.104265+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:54.139655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:54:54.139655+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"115","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.122223+0000","last_change":"2021-03-24T21:55:10.102704+0000","last_active":"2021-03-24T21:55:37.122223+0000","last_peered":"2021-03-24T21:55:37.122223+0000","last_clean":"2021-03-24T21:55:37.122223+0000","last_became_active":"2021-03-24T21:55:10.102360+0000","last_became_peered":"2021-03-24T21:55:10.102360+0000","last_unstale":"2021-03-24T21:55:37.122223+0000","last_undegraded":"2021-03-24T21:55:37.122223+0000","last_fullsized":"2021-03-24T21:55:37.122223+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:00.911020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:54:00.911020+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"69'14","reported_seq":"173","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.124649+0000","last_change":"2021-03-24T21:54:39.837964+0000","last_active":"2021-03-24T21:55:37.124649+0000","last_peered":"2021-03-24T21:55:37.124649+0000","last_clean":"2021-03-24T21:55:37.124649+0000","last_became_active":"2021-03-24T21:54:38.138034+0000","last_became_peered":"2021-03-24T21:54:38.138034+0000","last_unstale":"2021-03-24T21:55:37.124649+0000","last_undegraded":"2021-03-24T21:55:37.124649+0000","last_fullsized":"2021-03-24T21:55:37.124649+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'14","last_scrub_stamp":"2021-03-24T21:53:31.653334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:31.653334+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":16,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":894,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"167'121","reported_seq":"365","reported_epoch":"169","state":"active+clean","last_fresh":"2021-03-24T21:55:38.122599+0000","last_change":"2021-03-24T21:55:35.124391+0000","last_active":"2021-03-24T21:55:38.122599+0000","last_peered":"2021-03-24T21:55:38.122599+0000","last_clean":"2021-03-24T21:55:38.122599+0000","last_became_active":"2021-03-24T21:54:38.124237+0000","last_became_peered":"2021-03-24T21:54:38.124237+0000","last_unstale":"2021-03-24T21:55:38.122599+0000","last_undegraded":"2021-03-24T21:55:38.122599+0000","last_fullsized":"2021-03-24T21:55:38.122599+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":121,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16870744,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":88,"num_read_kb":78401,"num_write":63,"num_write_kb":22665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":5158008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"36","length":"2"},{"start":"3b","length":"1"}]},{"pgid":"6.6","version":"0'0","reported_seq":"119","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.122576+0000","last_change":"2021-03-24T21:54:10.519515+0000","last_active":"2021-03-24T21:55:37.122576+0000","last_peered":"2021-03-24T21:55:37.122576+0000","last_clean":"2021-03-24T21:55:37.122576+0000","last_became_active":"2021-03-24T21:51:51.759514+0000","last_became_peered":"2021-03-24T21:51:51.759514+0000","last_unstale":"2021-03-24T21:55:37.122576+0000","last_undegraded":"2021-03-24T21:55:37.122576+0000","last_fullsized":"2021-03-24T21:55:37.122576+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:10.519411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:52:59.706511+0000","last_clean_scrub_stamp":"2021-03-24T21:54:10.519411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"115","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.125096+0000","last_change":"2021-03-24T21:55:10.100603+0000","last_active":"2021-03-24T21:55:37.125096+0000","last_peered":"2021-03-24T21:55:37.125096+0000","last_clean":"2021-03-24T21:55:37.125096+0000","last_became_active":"2021-03-24T21:55:10.100245+0000","last_became_peered":"2021-03-24T21:55:10.100245+0000","last_unstale":"2021-03-24T21:55:37.125096+0000","last_undegraded":"2021-03-24T21:55:37.125096+0000","last_fullsized":"2021-03-24T21:55:37.125096+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:03.195853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:06.800417+0000","last_clean_scrub_stamp":"2021-03-24T21:55:03.195853+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"89'2","reported_seq":"154","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:38.046278+0000","last_change":"2021-03-24T21:55:38.046157+0000","last_active":"2021-03-24T21:55:38.046278+0000","last_peered":"2021-03-24T21:55:38.046278+0000","last_clean":"2021-03-24T21:55:38.046278+0000","last_became_active":"2021-03-24T21:54:38.144013+0000","last_became_peered":"2021-03-24T21:54:38.144013+0000","last_unstale":"2021-03-24T21:55:38.046278+0000","last_undegraded":"2021-03-24T21:55:38.046278+0000","last_fullsized":"2021-03-24T21:55:38.046278+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2021-03-24T21:55:38.045901+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2021-03-24T21:53:28.219900+0000","last_clean_scrub_stamp":"2021-03-24T21:55:38.045901+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"69'25","reported_seq":"161","reported_epoch":"166","state":"active+clean","last_fresh":"2021-03-24T21:55:35.105458+0000","last_change":"2021-03-24T21:55:29.056321+0000","last_active":"2021-03-24T21:55:35.105458+0000","last_peered":"2021-03-24T21:55:35.105458+0000","last_clean":"2021-03-24T21:55:35.105458+0000","last_became_active":"2021-03-24T21:54:38.141478+0000","last_became_peered":"2021-03-24T21:54:38.141478+0000","last_unstale":"2021-03-24T21:55:35.105458+0000","last_undegraded":"2021-03-24T21:55:35.105458+0000","last_fullsized":"2021-03-24T21:55:35.105458+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'25","last_scrub_stamp":"2021-03-24T21:55:29.056144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:29.056144+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":53,"num_read_kb":30,"num_write":28,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"130","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.125304+0000","last_change":"2021-03-24T21:55:36.236833+0000","last_active":"2021-03-24T21:55:37.125304+0000","last_peered":"2021-03-24T21:55:37.125304+0000","last_clean":"2021-03-24T21:55:37.125304+0000","last_became_active":"2021-03-24T21:54:38.136859+0000","last_became_peered":"2021-03-24T21:54:38.136859+0000","last_unstale":"2021-03-24T21:55:37.125304+0000","last_undegraded":"2021-03-24T21:55:37.125304+0000","last_fullsized":"2021-03-24T21:55:37.125304+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:36.236594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:55:36.236594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"166'97","reported_seq":"332","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.125079+0000","last_change":"2021-03-24T21:55:35.119816+0000","last_active":"2021-03-24T21:55:37.125079+0000","last_peered":"2021-03-24T21:55:37.125079+0000","last_clean":"2021-03-24T21:55:37.125079+0000","last_became_active":"2021-03-24T21:54:38.137466+0000","last_became_peered":"2021-03-24T21:54:38.137466+0000","last_unstale":"2021-03-24T21:55:37.125079+0000","last_undegraded":"2021-03-24T21:55:37.125079+0000","last_fullsized":"2021-03-24T21:55:37.125079+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10773221,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":2,"num_read":66,"num_read_kb":61085,"num_write":49,"num_write_kb":18388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4942492,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"36","length":"2"},{"start":"3b","length":"1"}]},{"pgid":"6.5","version":"0'0","reported_seq":"129","reported_epoch":"169","state":"active+clean","last_fresh":"2021-03-24T21:55:38.136728+0000","last_change":"2021-03-24T21:55:34.931146+0000","last_active":"2021-03-24T21:55:38.136728+0000","last_peered":"2021-03-24T21:55:38.136728+0000","last_clean":"2021-03-24T21:55:38.136728+0000","last_became_active":"2021-03-24T21:54:24.186957+0000","last_became_peered":"2021-03-24T21:54:24.186957+0000","last_unstale":"2021-03-24T21:55:38.136728+0000","last_undegraded":"2021-03-24T21:55:38.136728+0000","last_fullsized":"2021-03-24T21:55:38.136728+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:34.930967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:55:34.930967+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"61'1","reported_seq":"85","reported_epoch":"166","state":"active+clean","last_fresh":"2021-03-24T21:55:35.099367+0000","last_change":"2021-03-24T21:55:30.102312+0000","last_active":"2021-03-24T21:55:35.099367+0000","last_peered":"2021-03-24T21:55:35.099367+0000","last_clean":"2021-03-24T21:55:35.099367+0000","last_became_active":"2021-03-24T21:54:38.123139+0000","last_became_peered":"2021-03-24T21:54:38.123139+0000","last_unstale":"2021-03-24T21:55:35.099367+0000","last_undegraded":"2021-03-24T21:55:35.099367+0000","last_fullsized":"2021-03-24T21:55:35.099367+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2021-03-24T21:55:30.102193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:30.102193+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"127","reported_epoch":"166","state":"active+clean","last_fresh":"2021-03-24T21:55:35.103273+0000","last_change":"2021-03-24T21:53:55.497565+0000","last_active":"2021-03-24T21:55:35.103273+0000","last_peered":"2021-03-24T21:55:35.103273+0000","last_clean":"2021-03-24T21:55:35.103273+0000","last_became_active":"2021-03-24T21:51:40.351000+0000","last_became_peered":"2021-03-24T21:51:40.351000+0000","last_unstale":"2021-03-24T21:55:35.103273+0000","last_undegraded":"2021-03-24T21:55:35.103273+0000","last_fullsized":"2021-03-24T21:55:35.103273+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:55.497382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:39.286101+0000","last_clean_scrub_stamp":"2021-03-24T21:53:55.497382+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"89'22","reported_seq":"160","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.520453+0000","last_change":"2021-03-24T21:55:23.459422+0000","last_active":"2021-03-24T21:55:37.520453+0000","last_peered":"2021-03-24T21:55:37.520453+0000","last_clean":"2021-03-24T21:55:37.520453+0000","last_became_active":"2021-03-24T21:54:38.129459+0000","last_became_peered":"2021-03-24T21:54:38.129459+0000","last_unstale":"2021-03-24T21:55:37.520453+0000","last_undegraded":"2021-03-24T21:55:37.520453+0000","last_fullsized":"2021-03-24T21:55:37.520453+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'22","last_scrub_stamp":"2021-03-24T21:55:23.459251+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2021-03-24T21:53:19.717244+0000","last_clean_scrub_stamp":"2021-03-24T21:55:23.459251+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":36,"num_read_kb":21,"num_write":18,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":644,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"119","reported_epoch":"166","state":"active+clean","last_fresh":"2021-03-24T21:55:35.105856+0000","last_change":"2021-03-24T21:55:31.118872+0000","last_active":"2021-03-24T21:55:35.105856+0000","last_peered":"2021-03-24T21:55:35.105856+0000","last_clean":"2021-03-24T21:55:35.105856+0000","last_became_active":"2021-03-24T21:51:49.418848+0000","last_became_peered":"2021-03-24T21:51:49.418848+0000","last_unstale":"2021-03-24T21:55:35.105856+0000","last_undegraded":"2021-03-24T21:55:35.105856+0000","last_fullsized":"2021-03-24T21:55:35.105856+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:31.118766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:55:31.118766+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"165'88","reported_seq":"302","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.124964+0000","last_change":"2021-03-24T21:55:35.128441+0000","last_active":"2021-03-24T21:55:37.124964+0000","last_peered":"2021-03-24T21:55:37.124964+0000","last_clean":"2021-03-24T21:55:37.124964+0000","last_became_active":"2021-03-24T21:54:22.162608+0000","last_became_peered":"2021-03-24T21:54:22.162608+0000","last_unstale":"2021-03-24T21:55:37.124964+0000","last_undegraded":"2021-03-24T21:55:37.124964+0000","last_fullsized":"2021-03-24T21:55:37.124964+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":88,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12437048,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":68,"num_read_kb":71131,"num_write":36,"num_write_kb":13420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"36","length":"2"},{"start":"3b","length":"1"}]},{"pgid":"6.7","version":"0'0","reported_seq":"121","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.520131+0000","last_change":"2021-03-24T21:54:45.900904+0000","last_active":"2021-03-24T21:55:37.520131+0000","last_peered":"2021-03-24T21:55:37.520131+0000","last_clean":"2021-03-24T21:55:37.520131+0000","last_became_active":"2021-03-24T21:51:51.765236+0000","last_became_peered":"2021-03-24T21:51:51.765236+0000","last_unstale":"2021-03-24T21:55:37.520131+0000","last_undegraded":"2021-03-24T21:55:37.520131+0000","last_fullsized":"2021-03-24T21:55:37.520131+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:45.900726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:54:45.900726+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"149","reported_epoch":"169","state":"active+clean","last_fresh":"2021-03-24T21:55:38.134178+0000","last_change":"2021-03-24T21:54:38.110780+0000","last_active":"2021-03-24T21:55:38.134178+0000","last_peered":"2021-03-24T21:55:38.134178+0000","last_clean":"2021-03-24T21:55:38.134178+0000","last_became_active":"2021-03-24T21:54:38.110327+0000","last_became_peered":"2021-03-24T21:54:38.110327+0000","last_unstale":"2021-03-24T21:55:38.134178+0000","last_undegraded":"2021-03-24T21:55:38.134178+0000","last_fullsized":"2021-03-24T21:55:38.134178+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:30.906502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:49:51.529051+0000","last_clean_scrub_stamp":"2021-03-24T21:53:30.906502+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"134","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.120443+0000","last_change":"2021-03-24T21:55:33.534038+0000","last_active":"2021-03-24T21:55:37.120443+0000","last_peered":"2021-03-24T21:55:37.120443+0000","last_clean":"2021-03-24T21:55:37.120443+0000","last_became_active":"2021-03-24T21:54:24.199349+0000","last_became_peered":"2021-03-24T21:54:24.199349+0000","last_unstale":"2021-03-24T21:55:37.120443+0000","last_undegraded":"2021-03-24T21:55:37.120443+0000","last_fullsized":"2021-03-24T21:55:37.120443+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:33.533840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:33.533840+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"69'15","reported_seq":"140","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.117585+0000","last_change":"2021-03-24T21:55:30.166245+0000","last_active":"2021-03-24T21:55:37.117585+0000","last_peered":"2021-03-24T21:55:37.117585+0000","last_clean":"2021-03-24T21:55:37.117585+0000","last_became_active":"2021-03-24T21:54:24.194711+0000","last_became_peered":"2021-03-24T21:54:24.194711+0000","last_unstale":"2021-03-24T21:55:37.117585+0000","last_undegraded":"2021-03-24T21:55:37.117585+0000","last_fullsized":"2021-03-24T21:55:37.117585+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:55:30.166107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:30.166107+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":26,"num_read_kb":14,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":466,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"122","reported_epoch":"169","state":"active+clean","last_fresh":"2021-03-24T21:55:38.135826+0000","last_change":"2021-03-24T21:54:49.807761+0000","last_active":"2021-03-24T21:55:38.135826+0000","last_peered":"2021-03-24T21:55:38.135826+0000","last_clean":"2021-03-24T21:55:38.135826+0000","last_became_active":"2021-03-24T21:51:49.413251+0000","last_became_peered":"2021-03-24T21:51:49.413251+0000","last_unstale":"2021-03-24T21:55:38.135826+0000","last_undegraded":"2021-03-24T21:55:38.135826+0000","last_fullsized":"2021-03-24T21:55:38.135826+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:49.807585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:54:49.807585+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"155'35","reported_seq":"249","reported_epoch":"169","state":"active+clean","last_fresh":"2021-03-24T21:55:38.135869+0000","last_change":"2021-03-24T21:55:35.109615+0000","last_active":"2021-03-24T21:55:38.135869+0000","last_peered":"2021-03-24T21:55:38.135869+0000","last_clean":"2021-03-24T21:55:38.135869+0000","last_became_active":"2021-03-24T21:54:22.162598+0000","last_became_peered":"2021-03-24T21:54:22.162598+0000","last_unstale":"2021-03-24T21:55:38.135869+0000","last_undegraded":"2021-03-24T21:55:38.135869+0000","last_fullsized":"2021-03-24T21:55:38.135869+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2100875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":16763,"num_write":20,"num_write_kb":8001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"36","length":"2"},{"start":"3b","length":"1"}]},{"pgid":"6.4","version":"0'0","reported_seq":"72","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.519391+0000","last_change":"2021-03-24T21:54:38.122715+0000","last_active":"2021-03-24T21:55:37.519391+0000","last_peered":"2021-03-24T21:55:37.519391+0000","last_clean":"2021-03-24T21:55:37.519391+0000","last_became_active":"2021-03-24T21:54:38.122320+0000","last_became_peered":"2021-03-24T21:54:38.122320+0000","last_unstale":"2021-03-24T21:55:37.519391+0000","last_undegraded":"2021-03-24T21:55:37.519391+0000","last_fullsized":"2021-03-24T21:55:37.519391+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:16.721504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:54:16.721504+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"125","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.123885+0000","last_change":"2021-03-24T21:54:18.594379+0000","last_active":"2021-03-24T21:55:37.123885+0000","last_peered":"2021-03-24T21:55:37.123885+0000","last_clean":"2021-03-24T21:55:37.123885+0000","last_became_active":"2021-03-24T21:51:45.376687+0000","last_became_peered":"2021-03-24T21:51:45.376687+0000","last_unstale":"2021-03-24T21:55:37.123885+0000","last_undegraded":"2021-03-24T21:55:37.123885+0000","last_fullsized":"2021-03-24T21:55:37.123885+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:18.594199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:54:18.594199+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"66'5","reported_seq":"379","reported_epoch":"170","state":"active+clean","last_fresh":"2021-03-24T21:55:39.123266+0000","last_change":"2021-03-24T21:54:38.495832+0000","last_active":"2021-03-24T21:55:39.123266+0000","last_peered":"2021-03-24T21:55:39.123266+0000","last_clean":"2021-03-24T21:55:39.123266+0000","last_became_active":"2021-03-24T21:54:38.125434+0000","last_became_peered":"2021-03-24T21:54:38.125434+0000","last_unstale":"2021-03-24T21:55:39.123266+0000","last_undegraded":"2021-03-24T21:55:39.123266+0000","last_fullsized":"2021-03-24T21:55:39.123266+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2021-03-24T21:54:07.669877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:39.286101+0000","last_clean_scrub_stamp":"2021-03-24T21:54:07.669877+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":232,"num_read_kb":227,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"69'16","reported_seq":"180","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.124547+0000","last_change":"2021-03-24T21:54:57.325888+0000","last_active":"2021-03-24T21:55:37.124547+0000","last_peered":"2021-03-24T21:55:37.124547+0000","last_clean":"2021-03-24T21:55:37.124547+0000","last_became_active":"2021-03-24T21:54:38.121714+0000","last_became_peered":"2021-03-24T21:54:38.121714+0000","last_unstale":"2021-03-24T21:55:37.124547+0000","last_undegraded":"2021-03-24T21:55:37.124547+0000","last_fullsized":"2021-03-24T21:55:37.124547+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T21:54:57.325723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:54:57.325723+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":16,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":630,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"132","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.520925+0000","last_change":"2021-03-24T21:55:35.512941+0000","last_active":"2021-03-24T21:55:37.520925+0000","last_peered":"2021-03-24T21:55:37.520925+0000","last_clean":"2021-03-24T21:55:37.520925+0000","last_became_active":"2021-03-24T21:54:38.128708+0000","last_became_peered":"2021-03-24T21:54:38.128708+0000","last_unstale":"2021-03-24T21:55:37.520925+0000","last_undegraded":"2021-03-24T21:55:37.520925+0000","last_fullsized":"2021-03-24T21:55:37.520925+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:35.512770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:55:35.512770+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"164'20","reported_seq":"241","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.518859+0000","last_change":"2021-03-24T21:55:35.120536+0000","last_active":"2021-03-24T21:55:37.518859+0000","last_peered":"2021-03-24T21:55:37.518859+0000","last_clean":"2021-03-24T21:55:37.518859+0000","last_became_active":"2021-03-24T21:54:38.122313+0000","last_became_peered":"2021-03-24T21:54:38.122313+0000","last_unstale":"2021-03-24T21:55:37.518859+0000","last_undegraded":"2021-03-24T21:55:37.518859+0000","last_fullsized":"2021-03-24T21:55:37.518859+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3008420,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":13,"num_read_kb":13134,"num_write":14,"num_write_kb":3382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1253545,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"36","length":"2"},{"start":"3b","length":"1"}]},{"pgid":"6.1","version":"0'0","reported_seq":"117","reported_epoch":"166","state":"active+clean","last_fresh":"2021-03-24T21:55:35.103925+0000","last_change":"2021-03-24T21:54:38.124075+0000","last_active":"2021-03-24T21:55:35.103925+0000","last_peered":"2021-03-24T21:55:35.103925+0000","last_clean":"2021-03-24T21:55:35.103925+0000","last_became_active":"2021-03-24T21:54:38.122577+0000","last_became_peered":"2021-03-24T21:54:38.122577+0000","last_unstale":"2021-03-24T21:55:35.103925+0000","last_undegraded":"2021-03-24T21:55:35.103925+0000","last_fullsized":"2021-03-24T21:55:35.103925+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:28.691270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:28.691270+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"73","reported_epoch":"166","state":"active+clean","last_fresh":"2021-03-24T21:55:35.104787+0000","last_change":"2021-03-24T21:54:43.286516+0000","last_active":"2021-03-24T21:55:35.104787+0000","last_peered":"2021-03-24T21:55:35.104787+0000","last_clean":"2021-03-24T21:55:35.104787+0000","last_became_active":"2021-03-24T21:54:38.122875+0000","last_became_peered":"2021-03-24T21:54:38.122875+0000","last_unstale":"2021-03-24T21:55:35.104787+0000","last_undegraded":"2021-03-24T21:55:35.104787+0000","last_fullsized":"2021-03-24T21:55:35.104787+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:43.286336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:54:43.286336+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"69'21","reported_seq":"163","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.124860+0000","last_change":"2021-03-24T21:54:10.421436+0000","last_active":"2021-03-24T21:55:37.124860+0000","last_peered":"2021-03-24T21:55:37.124860+0000","last_clean":"2021-03-24T21:55:37.124860+0000","last_became_active":"2021-03-24T21:51:47.686486+0000","last_became_peered":"2021-03-24T21:51:47.686486+0000","last_unstale":"2021-03-24T21:55:37.124860+0000","last_undegraded":"2021-03-24T21:55:37.124860+0000","last_fullsized":"2021-03-24T21:55:37.124860+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'21","last_scrub_stamp":"2021-03-24T21:54:10.421258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:54:10.421258+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":45,"num_read_kb":27,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"79","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.124742+0000","last_change":"2021-03-24T21:54:24.199253+0000","last_active":"2021-03-24T21:55:37.124742+0000","last_peered":"2021-03-24T21:55:37.124742+0000","last_clean":"2021-03-24T21:55:37.124742+0000","last_became_active":"2021-03-24T21:54:24.198895+0000","last_became_peered":"2021-03-24T21:54:24.198895+0000","last_unstale":"2021-03-24T21:55:37.124742+0000","last_undegraded":"2021-03-24T21:55:37.124742+0000","last_fullsized":"2021-03-24T21:55:37.124742+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:23.246320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:23.246320+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"168'121","reported_seq":"380","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.145484+0000","last_change":"2021-03-24T21:55:35.103000+0000","last_active":"2021-03-24T21:55:37.145484+0000","last_peered":"2021-03-24T21:55:37.145484+0000","last_clean":"2021-03-24T21:55:37.145484+0000","last_became_active":"2021-03-24T21:54:38.139355+0000","last_became_peered":"2021-03-24T21:54:38.139355+0000","last_unstale":"2021-03-24T21:55:37.145484+0000","last_undegraded":"2021-03-24T21:55:37.145484+0000","last_fullsized":"2021-03-24T21:55:37.145484+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":121,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14087135,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":97,"num_read_kb":86171,"num_write":67,"num_write_kb":28209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":6977532,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"36","length":"2"},{"start":"3b","length":"1"}]},{"pgid":"6.0","version":"0'0","reported_seq":"114","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.123707+0000","last_change":"2021-03-24T21:53:25.593543+0000","last_active":"2021-03-24T21:55:37.123707+0000","last_peered":"2021-03-24T21:55:37.123707+0000","last_clean":"2021-03-24T21:55:37.123707+0000","last_became_active":"2021-03-24T21:51:51.759627+0000","last_became_peered":"2021-03-24T21:51:51.759627+0000","last_unstale":"2021-03-24T21:55:37.123707+0000","last_undegraded":"2021-03-24T21:55:37.123707+0000","last_fullsized":"2021-03-24T21:55:37.123707+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:25.593358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:25.593358+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"127","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.519882+0000","last_change":"2021-03-24T21:54:20.101633+0000","last_active":"2021-03-24T21:55:37.519882+0000","last_peered":"2021-03-24T21:55:37.519882+0000","last_clean":"2021-03-24T21:55:37.519882+0000","last_became_active":"2021-03-24T21:51:45.583885+0000","last_became_peered":"2021-03-24T21:51:45.583885+0000","last_unstale":"2021-03-24T21:55:37.519882+0000","last_undegraded":"2021-03-24T21:55:37.519882+0000","last_fullsized":"2021-03-24T21:55:37.519882+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:20.101463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:54:20.101463+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"69'16","reported_seq":"141","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.520184+0000","last_change":"2021-03-24T21:54:51.015209+0000","last_active":"2021-03-24T21:55:37.520184+0000","last_peered":"2021-03-24T21:55:37.520184+0000","last_clean":"2021-03-24T21:55:37.520184+0000","last_became_active":"2021-03-24T21:54:38.128745+0000","last_became_peered":"2021-03-24T21:54:38.128745+0000","last_unstale":"2021-03-24T21:55:37.520184+0000","last_undegraded":"2021-03-24T21:55:37.520184+0000","last_fullsized":"2021-03-24T21:55:37.520184+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T21:54:51.015019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:54:51.015019+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":32,"num_read_kb":15,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":940,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"102'4","reported_seq":"178","reported_epoch":"169","state":"active+clean","last_fresh":"2021-03-24T21:55:38.121301+0000","last_change":"2021-03-24T21:54:24.305088+0000","last_active":"2021-03-24T21:55:38.121301+0000","last_peered":"2021-03-24T21:55:38.121301+0000","last_clean":"2021-03-24T21:55:38.121301+0000","last_became_active":"2021-03-24T21:54:24.203310+0000","last_became_peered":"2021-03-24T21:54:24.203310+0000","last_unstale":"2021-03-24T21:55:38.121301+0000","last_undegraded":"2021-03-24T21:55:38.121301+0000","last_fullsized":"2021-03-24T21:55:38.121301+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'3","last_scrub_stamp":"2021-03-24T21:53:26.584678+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-24T21:53:26.584678+0000","last_clean_scrub_stamp":"2021-03-24T21:53:26.584678+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"164'12","reported_seq":"228","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.119514+0000","last_change":"2021-03-24T21:55:35.123407+0000","last_active":"2021-03-24T21:55:37.119514+0000","last_peered":"2021-03-24T21:55:37.119514+0000","last_clean":"2021-03-24T21:55:37.119514+0000","last_became_active":"2021-03-24T21:54:38.143356+0000","last_became_peered":"2021-03-24T21:54:38.143356+0000","last_unstale":"2021-03-24T21:55:37.119514+0000","last_undegraded":"2021-03-24T21:55:37.119514+0000","last_fullsized":"2021-03-24T21:55:37.119514+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3042034,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":3819,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":867724,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"36","length":"2"},{"start":"3b","length":"1"}]},{"pgid":"6.3","version":"0'0","reported_seq":"138","reported_epoch":"169","state":"active+clean","last_fresh":"2021-03-24T21:55:38.135408+0000","last_change":"2021-03-24T21:54:44.933344+0000","last_active":"2021-03-24T21:55:38.135408+0000","last_peered":"2021-03-24T21:55:38.135408+0000","last_clean":"2021-03-24T21:55:38.135408+0000","last_became_active":"2021-03-24T21:54:38.110246+0000","last_became_peered":"2021-03-24T21:54:38.110246+0000","last_unstale":"2021-03-24T21:55:38.135408+0000","last_undegraded":"2021-03-24T21:55:38.135408+0000","last_fullsized":"2021-03-24T21:55:38.135408+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:44.933255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:54:44.933255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"134","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.124647+0000","last_change":"2021-03-24T21:55:28.165657+0000","last_active":"2021-03-24T21:55:37.124647+0000","last_peered":"2021-03-24T21:55:37.124647+0000","last_clean":"2021-03-24T21:55:37.124647+0000","last_became_active":"2021-03-24T21:54:38.134604+0000","last_became_peered":"2021-03-24T21:54:38.134604+0000","last_unstale":"2021-03-24T21:55:37.124647+0000","last_undegraded":"2021-03-24T21:55:37.124647+0000","last_fullsized":"2021-03-24T21:55:37.124647+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:28.165408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:28.165408+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"69'20","reported_seq":"193","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.117020+0000","last_change":"2021-03-24T21:55:00.135489+0000","last_active":"2021-03-24T21:55:37.117020+0000","last_peered":"2021-03-24T21:55:37.117020+0000","last_clean":"2021-03-24T21:55:37.117020+0000","last_became_active":"2021-03-24T21:54:24.189182+0000","last_became_peered":"2021-03-24T21:54:24.189182+0000","last_unstale":"2021-03-24T21:55:37.117020+0000","last_undegraded":"2021-03-24T21:55:37.117020+0000","last_fullsized":"2021-03-24T21:55:37.117020+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'20","last_scrub_stamp":"2021-03-24T21:55:00.135328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:00.135328+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":39,"num_read_kb":20,"num_write":20,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":940,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"109'3","reported_seq":"182","reported_epoch":"169","state":"active+clean","last_fresh":"2021-03-24T21:55:38.133763+0000","last_change":"2021-03-24T21:54:39.228237+0000","last_active":"2021-03-24T21:55:38.133763+0000","last_peered":"2021-03-24T21:55:38.133763+0000","last_clean":"2021-03-24T21:55:38.133763+0000","last_became_active":"2021-03-24T21:54:38.135726+0000","last_became_peered":"2021-03-24T21:54:38.135726+0000","last_unstale":"2021-03-24T21:55:38.133763+0000","last_undegraded":"2021-03-24T21:55:38.133763+0000","last_fullsized":"2021-03-24T21:55:38.133763+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2021-03-24T21:54:17.493861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:54:17.493861+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"162'121","reported_seq":"336","reported_epoch":"169","state":"active+clean","last_fresh":"2021-03-24T21:55:38.142433+0000","last_change":"2021-03-24T21:55:35.120525+0000","last_active":"2021-03-24T21:55:38.142433+0000","last_peered":"2021-03-24T21:55:38.142433+0000","last_clean":"2021-03-24T21:55:38.142433+0000","last_became_active":"2021-03-24T21:54:22.167761+0000","last_became_peered":"2021-03-24T21:54:22.167761+0000","last_unstale":"2021-03-24T21:55:38.142433+0000","last_undegraded":"2021-03-24T21:55:38.142433+0000","last_fullsized":"2021-03-24T21:55:38.142433+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":121,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10790028,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":70,"num_read_kb":60028,"num_write":61,"num_write_kb":28048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"36","length":"2"},{"start":"3b","length":"1"}]},{"pgid":"6.2","version":"0'0","reported_seq":"114","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.117021+0000","last_change":"2021-03-24T21:53:45.349966+0000","last_active":"2021-03-24T21:55:37.117021+0000","last_peered":"2021-03-24T21:55:37.117021+0000","last_clean":"2021-03-24T21:55:37.117021+0000","last_became_active":"2021-03-24T21:51:51.756114+0000","last_became_peered":"2021-03-24T21:51:51.756114+0000","last_unstale":"2021-03-24T21:55:37.117021+0000","last_undegraded":"2021-03-24T21:55:37.117021+0000","last_fullsized":"2021-03-24T21:55:37.117021+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:45.349781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:45.349781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"125","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.122470+0000","last_change":"2021-03-24T21:54:56.508843+0000","last_active":"2021-03-24T21:55:37.122470+0000","last_peered":"2021-03-24T21:55:37.122470+0000","last_clean":"2021-03-24T21:55:37.122470+0000","last_became_active":"2021-03-24T21:51:45.378704+0000","last_became_peered":"2021-03-24T21:51:45.378704+0000","last_unstale":"2021-03-24T21:55:37.122470+0000","last_undegraded":"2021-03-24T21:55:37.122470+0000","last_fullsized":"2021-03-24T21:55:37.122470+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:56.508652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:54:56.508652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"69'22","reported_seq":"163","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.119230+0000","last_change":"2021-03-24T21:55:28.629625+0000","last_active":"2021-03-24T21:55:37.119230+0000","last_peered":"2021-03-24T21:55:37.119230+0000","last_clean":"2021-03-24T21:55:37.119230+0000","last_became_active":"2021-03-24T21:51:47.691459+0000","last_became_peered":"2021-03-24T21:51:47.691459+0000","last_unstale":"2021-03-24T21:55:37.119230+0000","last_undegraded":"2021-03-24T21:55:37.119230+0000","last_fullsized":"2021-03-24T21:55:37.119230+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'22","last_scrub_stamp":"2021-03-24T21:55:28.629443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:28.629443+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":46,"num_read_kb":26,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"121","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.118329+0000","last_change":"2021-03-24T21:55:25.106515+0000","last_active":"2021-03-24T21:55:37.118329+0000","last_peered":"2021-03-24T21:55:37.118329+0000","last_clean":"2021-03-24T21:55:37.118329+0000","last_became_active":"2021-03-24T21:51:49.408761+0000","last_became_peered":"2021-03-24T21:51:49.408761+0000","last_unstale":"2021-03-24T21:55:37.118329+0000","last_undegraded":"2021-03-24T21:55:37.118329+0000","last_fullsized":"2021-03-24T21:55:37.118329+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:25.106440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:25.220436+0000","last_clean_scrub_stamp":"2021-03-24T21:55:25.106440+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"166'94","reported_seq":"301","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.121847+0000","last_change":"2021-03-24T21:55:35.128462+0000","last_active":"2021-03-24T21:55:37.121847+0000","last_peered":"2021-03-24T21:55:37.121847+0000","last_clean":"2021-03-24T21:55:37.121847+0000","last_became_active":"2021-03-24T21:54:24.190419+0000","last_became_peered":"2021-03-24T21:54:24.190419+0000","last_unstale":"2021-03-24T21:55:37.121847+0000","last_undegraded":"2021-03-24T21:55:37.121847+0000","last_fullsized":"2021-03-24T21:55:37.121847+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":94,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8601752,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":28,"num_read_kb":3497,"num_write":50,"num_write_kb":16532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"36","length":"2"},{"start":"3b","length":"1"}]},{"pgid":"6.d","version":"0'0","reported_seq":"130","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.519720+0000","last_change":"2021-03-24T21:55:28.557021+0000","last_active":"2021-03-24T21:55:37.519720+0000","last_peered":"2021-03-24T21:55:37.519720+0000","last_clean":"2021-03-24T21:55:37.519720+0000","last_became_active":"2021-03-24T21:54:38.129469+0000","last_became_peered":"2021-03-24T21:54:38.129469+0000","last_unstale":"2021-03-24T21:55:37.519720+0000","last_undegraded":"2021-03-24T21:55:37.519720+0000","last_fullsized":"2021-03-24T21:55:37.519720+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:28.556844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:55:28.556844+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"65'2","reported_seq":"163","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.118116+0000","last_change":"2021-03-24T21:53:29.268452+0000","last_active":"2021-03-24T21:55:37.118116+0000","last_peered":"2021-03-24T21:55:37.118116+0000","last_clean":"2021-03-24T21:55:37.118116+0000","last_became_active":"2021-03-24T21:51:49.414380+0000","last_became_peered":"2021-03-24T21:51:49.414380+0000","last_unstale":"2021-03-24T21:55:37.118116+0000","last_undegraded":"2021-03-24T21:55:37.118116+0000","last_fullsized":"2021-03-24T21:55:37.118116+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2021-03-24T21:53:29.268273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:29.268273+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":"129","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.119141+0000","last_change":"2021-03-24T21:54:38.135257+0000","last_active":"2021-03-24T21:55:37.119141+0000","last_peered":"2021-03-24T21:55:37.119141+0000","last_clean":"2021-03-24T21:55:37.119141+0000","last_became_active":"2021-03-24T21:54:38.134768+0000","last_became_peered":"2021-03-24T21:54:38.134768+0000","last_unstale":"2021-03-24T21:55:37.119141+0000","last_undegraded":"2021-03-24T21:55:37.119141+0000","last_fullsized":"2021-03-24T21:55:37.119141+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:35.609173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:35.609173+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"69'17","reported_seq":"133","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.122529+0000","last_change":"2021-03-24T21:54:38.505905+0000","last_active":"2021-03-24T21:55:37.122529+0000","last_peered":"2021-03-24T21:55:37.122529+0000","last_clean":"2021-03-24T21:55:37.122529+0000","last_became_active":"2021-03-24T21:54:38.130536+0000","last_became_peered":"2021-03-24T21:54:38.130536+0000","last_unstale":"2021-03-24T21:55:37.122529+0000","last_undegraded":"2021-03-24T21:55:37.122529+0000","last_fullsized":"2021-03-24T21:55:37.122529+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'17","last_scrub_stamp":"2021-03-24T21:54:03.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:54:03.731436+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":18,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":422,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"168'122","reported_seq":"348","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.127615+0000","last_change":"2021-03-24T21:55:35.120351+0000","last_active":"2021-03-24T21:55:37.127615+0000","last_peered":"2021-03-24T21:55:37.127615+0000","last_clean":"2021-03-24T21:55:37.127615+0000","last_became_active":"2021-03-24T21:54:38.123502+0000","last_became_peered":"2021-03-24T21:54:38.123502+0000","last_unstale":"2021-03-24T21:55:37.127615+0000","last_undegraded":"2021-03-24T21:55:37.127615+0000","last_fullsized":"2021-03-24T21:55:37.127615+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":122,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16790094,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":45,"num_read_kb":24714,"num_write":73,"num_write_kb":27950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":5093012,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"36","length":"2"},{"start":"3b","length":"1"}]},{"pgid":"6.c","version":"0'0","reported_seq":"123","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.122104+0000","last_change":"2021-03-24T21:54:24.196940+0000","last_active":"2021-03-24T21:55:37.122104+0000","last_peered":"2021-03-24T21:55:37.122104+0000","last_clean":"2021-03-24T21:55:37.122104+0000","last_became_active":"2021-03-24T21:54:24.196587+0000","last_became_peered":"2021-03-24T21:54:24.196587+0000","last_unstale":"2021-03-24T21:55:37.122104+0000","last_undegraded":"2021-03-24T21:55:37.122104+0000","last_fullsized":"2021-03-24T21:55:37.122104+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:27.701002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:27.701002+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"98","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.118563+0000","last_change":"2021-03-24T21:55:10.102484+0000","last_active":"2021-03-24T21:55:37.118563+0000","last_peered":"2021-03-24T21:55:37.118563+0000","last_clean":"2021-03-24T21:55:37.118563+0000","last_became_active":"2021-03-24T21:55:10.102113+0000","last_became_peered":"2021-03-24T21:55:10.102113+0000","last_unstale":"2021-03-24T21:55:37.118563+0000","last_undegraded":"2021-03-24T21:55:37.118563+0000","last_fullsized":"2021-03-24T21:55:37.118563+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:12.170934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:54:12.170934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"125","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.116693+0000","last_change":"2021-03-24T21:54:53.267409+0000","last_active":"2021-03-24T21:55:37.116693+0000","last_peered":"2021-03-24T21:55:37.116693+0000","last_clean":"2021-03-24T21:55:37.116693+0000","last_became_active":"2021-03-24T21:51:45.579513+0000","last_became_peered":"2021-03-24T21:51:45.579513+0000","last_unstale":"2021-03-24T21:55:37.116693+0000","last_undegraded":"2021-03-24T21:55:37.116693+0000","last_fullsized":"2021-03-24T21:55:37.116693+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:53.267324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:54:53.267324+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"69'15","reported_seq":"208","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.117905+0000","last_change":"2021-03-24T21:54:40.017534+0000","last_active":"2021-03-24T21:55:37.117905+0000","last_peered":"2021-03-24T21:55:37.117905+0000","last_clean":"2021-03-24T21:55:37.117905+0000","last_became_active":"2021-03-24T21:54:38.127461+0000","last_became_peered":"2021-03-24T21:54:38.127461+0000","last_unstale":"2021-03-24T21:55:37.117905+0000","last_undegraded":"2021-03-24T21:55:37.117905+0000","last_fullsized":"2021-03-24T21:55:37.117905+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:54:02.296830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:54:02.296830+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1476,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"155'20","reported_seq":"229","reported_epoch":"166","state":"active+clean","last_fresh":"2021-03-24T21:55:35.106633+0000","last_change":"2021-03-24T21:55:35.106633+0000","last_active":"2021-03-24T21:55:35.106633+0000","last_peered":"2021-03-24T21:55:35.106633+0000","last_clean":"2021-03-24T21:55:35.106633+0000","last_became_active":"2021-03-24T21:54:22.168397+0000","last_became_peered":"2021-03-24T21:54:22.168397+0000","last_unstale":"2021-03-24T21:55:35.106633+0000","last_undegraded":"2021-03-24T21:55:35.106633+0000","last_fullsized":"2021-03-24T21:55:35.106633+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2033647,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":30,"num_read_kb":26379,"num_write":11,"num_write_kb":2188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"36","length":"2"},{"start":"3b","length":"1"}]},{"pgid":"6.f","version":"0'0","reported_seq":"117","reported_epoch":"166","state":"active+clean","last_fresh":"2021-03-24T21:55:35.106343+0000","last_change":"2021-03-24T21:55:24.078895+0000","last_active":"2021-03-24T21:55:35.106343+0000","last_peered":"2021-03-24T21:55:35.106343+0000","last_clean":"2021-03-24T21:55:35.106343+0000","last_became_active":"2021-03-24T21:51:51.765067+0000","last_became_peered":"2021-03-24T21:51:51.765067+0000","last_unstale":"2021-03-24T21:55:35.106343+0000","last_undegraded":"2021-03-24T21:55:35.106343+0000","last_fullsized":"2021-03-24T21:55:35.106343+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:24.078780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:55:24.078780+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"72","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.117627+0000","last_change":"2021-03-24T21:54:38.139233+0000","last_active":"2021-03-24T21:55:37.117627+0000","last_peered":"2021-03-24T21:55:37.117627+0000","last_clean":"2021-03-24T21:55:37.117627+0000","last_became_active":"2021-03-24T21:54:38.138864+0000","last_became_peered":"2021-03-24T21:54:38.138864+0000","last_unstale":"2021-03-24T21:55:37.117627+0000","last_undegraded":"2021-03-24T21:55:37.117627+0000","last_fullsized":"2021-03-24T21:55:37.117627+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:29.735672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:29.735672+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"92","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.117289+0000","last_change":"2021-03-24T21:55:01.169611+0000","last_active":"2021-03-24T21:55:37.117289+0000","last_peered":"2021-03-24T21:55:37.117289+0000","last_clean":"2021-03-24T21:55:37.117289+0000","last_became_active":"2021-03-24T21:54:38.127136+0000","last_became_peered":"2021-03-24T21:54:38.127136+0000","last_unstale":"2021-03-24T21:55:37.117289+0000","last_undegraded":"2021-03-24T21:55:37.117289+0000","last_fullsized":"2021-03-24T21:55:37.117289+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:01.169461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:55:01.169461+0000","last_clean_scrub_stamp":"2021-03-24T21:55:01.169461+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"69'21","reported_seq":"194","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.117306+0000","last_change":"2021-03-24T21:54:56.226932+0000","last_active":"2021-03-24T21:55:37.117306+0000","last_peered":"2021-03-24T21:55:37.117306+0000","last_clean":"2021-03-24T21:55:37.117306+0000","last_became_active":"2021-03-24T21:54:38.144020+0000","last_became_peered":"2021-03-24T21:54:38.144020+0000","last_unstale":"2021-03-24T21:55:37.117306+0000","last_undegraded":"2021-03-24T21:55:37.117306+0000","last_fullsized":"2021-03-24T21:55:37.117306+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'21","last_scrub_stamp":"2021-03-24T21:54:56.226757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:54:56.226757+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":23,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"169'30","reported_seq":"241","reported_epoch":"169","state":"active+clean","last_fresh":"2021-03-24T21:55:38.142131+0000","last_change":"2021-03-24T21:55:35.124087+0000","last_active":"2021-03-24T21:55:38.142131+0000","last_peered":"2021-03-24T21:55:38.142131+0000","last_clean":"2021-03-24T21:55:38.142131+0000","last_became_active":"2021-03-24T21:54:22.160108+0000","last_became_peered":"2021-03-24T21:54:22.160108+0000","last_unstale":"2021-03-24T21:55:38.142131+0000","last_undegraded":"2021-03-24T21:55:38.142131+0000","last_fullsized":"2021-03-24T21:55:38.142131+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7618363,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":27,"num_read_kb":27567,"num_write":14,"num_write_kb":7650,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"30","length":"1"},{"start":"32","length":"1"},{"start":"36","length":"2"},{"start":"3b","length":"1"}]},{"pgid":"6.e","version":"0'0","reported_seq":"123","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.118212+0000","last_change":"2021-03-24T21:54:38.117075+0000","last_active":"2021-03-24T21:55:37.118212+0000","last_peered":"2021-03-24T21:55:37.118212+0000","last_clean":"2021-03-24T21:55:37.118212+0000","last_became_active":"2021-03-24T21:54:38.116786+0000","last_became_peered":"2021-03-24T21:54:38.116786+0000","last_unstale":"2021-03-24T21:55:37.118212+0000","last_undegraded":"2021-03-24T21:55:37.118212+0000","last_fullsized":"2021-03-24T21:55:37.118212+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:40.289984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:40.289984+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"65'2","reported_seq":"166","reported_epoch":"166","state":"active+clean","last_fresh":"2021-03-24T21:55:35.106046+0000","last_change":"2021-03-24T21:54:56.160424+0000","last_active":"2021-03-24T21:55:35.106046+0000","last_peered":"2021-03-24T21:55:35.106046+0000","last_clean":"2021-03-24T21:55:35.106046+0000","last_became_active":"2021-03-24T21:51:49.416182+0000","last_became_peered":"2021-03-24T21:51:49.416182+0000","last_unstale":"2021-03-24T21:55:35.106046+0000","last_undegraded":"2021-03-24T21:55:35.106046+0000","last_fullsized":"2021-03-24T21:55:35.106046+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2021-03-24T21:54:56.160326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:54:56.160326+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"125","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.121282+0000","last_change":"2021-03-24T21:53:57.501519+0000","last_active":"2021-03-24T21:55:37.121282+0000","last_peered":"2021-03-24T21:55:37.121282+0000","last_clean":"2021-03-24T21:55:37.121282+0000","last_became_active":"2021-03-24T21:51:45.375830+0000","last_became_peered":"2021-03-24T21:51:45.375830+0000","last_unstale":"2021-03-24T21:55:37.121282+0000","last_undegraded":"2021-03-24T21:55:37.121282+0000","last_fullsized":"2021-03-24T21:55:37.121282+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:57.501351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:57.501351+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"69'13","reported_seq":"198","reported_epoch":"168","state":"active+clean","last_fresh":"2021-03-24T21:55:37.117914+0000","last_change":"2021-03-24T21:55:36.065455+0000","last_active":"2021-03-24T21:55:37.117914+0000","last_peered":"2021-03-24T21:55:37.117914+0000","last_clean":"2021-03-24T21:55:37.117914+0000","last_became_active":"2021-03-24T21:54:38.127115+0000","last_became_peered":"2021-03-24T21:54:38.127115+0000","last_unstale":"2021-03-24T21:55:37.117914+0000","last_undegraded":"2021-03-24T21:55:37.117914+0000","last_fullsized":"2021-03-24T21:55:37.117914+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2021-03-24T21:55:36.065267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:36.065267+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":13,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":148600227,"num_objects":66,"num_object_clones":23,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":7,"num_read":739,"num_read_kb":618412,"num_write":628,"num_write_kb":239288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":35460777,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":350789632,"data_stored":400274691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1225,"ondisk_log_size":1225,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":232,"num_read_kb":227,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1589,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":5558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1062,"num_read_kb":567,"num_write":542,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":283,"num_bytes_recovered":22082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2027520,"data_stored":72734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":549,"ondisk_log_size":549,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25,"ondisk_log_size":25,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201331,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":81096,"kb_used_data":67968,"kb_used_omap":0,"kb_used_meta":13120,"kb_avail":93664056,"statfs":{"total":95995035648,"available":95911993344,"internally_reserved":0,"allocated":69599232,"data_stored":81115237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":13434880},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:54:47 2021","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.90300000000000002,"15min":0.90000000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.4550000000000001,"5min":2.2240000000000002,"15min":2.2240000000000002},"last":1.121},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.94599999999999995,"15min":0.96399999999999997},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.163,"5min":2.1859999999999999,"15min":2.1859999999999999},"last":0.73599999999999999}]},{"osd":1,"last update":"Wed Mar 24 21:54:47 2021","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":0.88800000000000001,"15min":0.84999999999999998},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.4079999999999999,"5min":1.7090000000000001,"15min":1.7090000000000001},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.238,"5min":2.343,"15min":2.343},"last":0.96599999999999997}]},{"osd":2,"last update":"Wed Mar 24 21:54:47 2021","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":1.0029999999999999,"15min":0.997},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.3069999999999999,"5min":4.944,"15min":4.944},"last":0.76100000000000001},{"interface":"front","average":{"1min":1.0089999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.522,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":0.999}]},{"osd":3,"last update":"Wed Mar 24 21:54:47 2021","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":1.0269999999999999,"15min":1.0389999999999999},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.397,"5min":1.9490000000000001,"15min":1.9490000000000001},"last":1.0669999999999999},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.022,"15min":0.98699999999999999},"min":{"1min":0.71699999999999997,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.6040000000000001,"5min":1.845,"15min":1.845},"last":1.042}]},{"osd":4,"last update":"Wed Mar 24 21:54:47 2021","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":0.95699999999999996,"15min":0.93300000000000005},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.343,"5min":2.2000000000000002,"15min":2.2000000000000002},"last":1.2350000000000001},{"interface":"front","average":{"1min":1.073,"5min":1.0780000000000001,"15min":1.075},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.4359999999999999,"5min":2.1299999999999999,"15min":2.1299999999999999},"last":1.159}]},{"osd":5,"last update":"Wed Mar 24 21:54:47 2021","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.028,"15min":0.98499999999999999},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.5669999999999999,"5min":1.581,"15min":1.581},"last":1.2130000000000001},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.149,"15min":1.1399999999999999},"min":{"1min":0.70199999999999996,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.425,"5min":2.4630000000000001,"15min":2.4630000000000001},"last":1.4159999999999999}]},{"osd":6,"last update":"Wed Mar 24 21:54:47 2021","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.1559999999999999,"15min":1.123},"min":{"1min":0.73999999999999999,"5min":0.62,"15min":0.62},"max":{"1min":1.5269999999999999,"5min":2.5179999999999998,"15min":2.5179999999999998},"last":1.325},{"interface":"front","average":{"1min":1.181,"5min":1.1859999999999999,"15min":1.1759999999999999},"min":{"1min":0.63200000000000001,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.5,"5min":2.3069999999999999,"15min":2.3069999999999999},"last":1.385}]}]},{"osd":6,"up_from":0,"seq":201863462968,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":0,"seq":55834574925,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738448,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":86408,"kb_used_data":72640,"kb_used_omap":0,"kb_used_meta":13760,"kb_avail":93658744,"statfs":{"total":95995035648,"available":95906553856,"internally_reserved":0,"allocated":74383360,"data_stored":79756722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14090240},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Mar 24 21:54:41 2021","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.70199999999999996,"15min":0.60599999999999998},"min":{"1min":0.45400000000000001,"5min":0.314,"15min":0.314},"max":{"1min":4.8040000000000003,"5min":4.8040000000000003,"15min":4.8040000000000003},"last":0.57399999999999995},{"interface":"front","average":{"1min":1.175,"5min":0.79500000000000004,"15min":0.65000000000000002},"min":{"1min":0.46700000000000003,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":8.8989999999999991,"5min":8.8989999999999991,"15min":8.8989999999999991},"last":0.73899999999999999}]},{"osd":2,"last update":"Wed Mar 24 21:54:58 2021","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":0.75700000000000001,"15min":0.68899999999999995},"min":{"1min":0.40899999999999997,"5min":0.314,"15min":0.314},"max":{"1min":5.8810000000000002,"5min":5.8810000000000002,"15min":5.8810000000000002},"last":0.79600000000000004},{"interface":"front","average":{"1min":1.008,"5min":0.82199999999999995,"15min":0.72099999999999997},"min":{"1min":0.47499999999999998,"5min":0.376,"15min":0.376},"max":{"1min":4.766,"5min":4.766,"15min":4.766},"last":0.66300000000000003}]},{"osd":3,"last update":"Wed Mar 24 21:55:21 2021","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.88900000000000001,"15min":0.82799999999999996},"min":{"1min":0.55500000000000005,"5min":0.376,"15min":0.376},"max":{"1min":1.482,"5min":5.9790000000000001,"15min":5.9790000000000001},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.93400000000000005,"5min":0.97799999999999998,"15min":0.94599999999999995},"min":{"1min":0.505,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.3280000000000001,"5min":6.4580000000000002,"15min":6.4580000000000002},"last":0.70399999999999996}]},{"osd":4,"last update":"Wed Mar 24 21:54:41 2021","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.121,"15min":1.107},"min":{"1min":0.73399999999999999,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":5.0330000000000004,"5min":5.0330000000000004,"15min":5.0330000000000004},"last":0.879},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.1080000000000001,"15min":1.04},"min":{"1min":0.85999999999999999,"5min":0.625,"15min":0.625},"max":{"1min":6.1829999999999998,"5min":6.1829999999999998,"15min":6.1829999999999998},"last":0.89700000000000002}]},{"osd":5,"last update":"Wed Mar 24 21:54:58 2021","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":1.0960000000000001,"15min":1.077},"min":{"1min":0.69399999999999995,"5min":0.62,"15min":0.62},"max":{"1min":5.9390000000000001,"5min":5.9390000000000001,"15min":5.9390000000000001},"last":1.302},{"interface":"front","average":{"1min":1.359,"5min":1.218,"15min":1.1899999999999999},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":5.0590000000000002,"5min":5.0590000000000002,"15min":5.0590000000000002},"last":0.77000000000000002}]},{"osd":6,"last update":"Wed Mar 24 21:55:21 2021","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.129,"15min":1.087},"min":{"1min":0.81699999999999995,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.452,"5min":5.9930000000000003,"15min":5.9930000000000003},"last":1.3440000000000001},{"interface":"front","average":{"1min":1.129,"5min":1.248,"15min":1.228},"min":{"1min":0.91800000000000004,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.327,"5min":4.9960000000000004,"15min":4.9960000000000004},"last":0.82499999999999996}]},{"osd":7,"last update":"Wed Mar 24 21:54:46 2021","interfaces":[{"interface":"back","average":{"1min":1.4810000000000001,"5min":1.2370000000000001,"15min":1.1910000000000001},"min":{"1min":0.95199999999999996,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":6.1319999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"last":0.95099999999999996},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.1879999999999999,"15min":1.1459999999999999},"min":{"1min":1.0129999999999999,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":6.0529999999999999,"5min":6.0529999999999999,"15min":6.0529999999999999},"last":1.25}]}]},{"osd":2,"up_from":18,"seq":77309411401,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":57472,"kb_used_data":44728,"kb_used_omap":0,"kb_used_meta":12736,"kb_avail":93687680,"statfs":{"total":95995035648,"available":95936184320,"internally_reserved":0,"allocated":45801472,"data_stored":55515224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":13041664},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:54:57 2021","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.63},"min":{"1min":0.433,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.228,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.74299999999999999,"15min":0.68999999999999995},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.6140000000000001,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":1.022}]},{"osd":1,"last update":"Wed Mar 24 21:54:57 2021","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.77300000000000002,"15min":0.70899999999999996},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5780000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.79300000000000004,"15min":0.73699999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.0840000000000001,"5min":1.6240000000000001,"15min":1.6240000000000001},"last":0.80700000000000005}]},{"osd":3,"last update":"Wed Mar 24 21:55:20 2021","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.85599999999999998,"15min":0.82799999999999996},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.1659999999999999,"5min":1.9890000000000001,"15min":1.9890000000000001},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.91900000000000004,"15min":0.89400000000000002},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.2210000000000001,"5min":2.0950000000000002,"15min":2.0950000000000002},"last":0.97799999999999998}]},{"osd":4,"last update":"Wed Mar 24 21:54:41 2021","interfaces":[{"interface":"back","average":{"1min":1.1279999999999999,"5min":1.0589999999999999,"15min":1.042},"min":{"1min":0.625,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.8169999999999999,"5min":3.3380000000000001,"15min":3.3380000000000001},"last":1.2729999999999999},{"interface":"front","average":{"1min":1.046,"5min":1.0700000000000001,"15min":1.046},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.446,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":1.143}]},{"osd":5,"last update":"Wed Mar 24 21:54:57 2021","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.0509999999999999,"15min":1.0389999999999999},"min":{"1min":0.71199999999999997,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.85899999999999999},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.1599999999999999,"15min":1.1539999999999999},"min":{"1min":0.67200000000000004,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.7490000000000001,"5min":3.1509999999999998,"15min":3.1509999999999998},"last":1.099}]},{"osd":6,"last update":"Wed Mar 24 21:55:20 2021","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.1990000000000001,"15min":1.2430000000000001},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.403,"5min":2.2810000000000001,"15min":2.2810000000000001},"last":1.246},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.1459999999999999,"15min":1.1599999999999999},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.375,"5min":2.016,"15min":2.016},"last":0.95199999999999996}]},{"osd":7,"last update":"Wed Mar 24 21:54:47 2021","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.2150000000000001,"15min":1.2270000000000001},"min":{"1min":0.80900000000000005,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.4750000000000001,"5min":1.6719999999999999,"15min":1.6719999999999999},"last":1.1819999999999999},{"interface":"front","average":{"1min":1.2290000000000001,"5min":1.242,"15min":1.248},"min":{"1min":0.748,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.6759999999999999,"5min":3.5419999999999998,"15min":3.5419999999999998},"last":1.2130000000000001}]}]},{"osd":3,"up_from":26,"seq":111669149765,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":92992,"kb_used_data":75128,"kb_used_omap":0,"kb_used_meta":17856,"kb_avail":93652160,"statfs":{"total":95995035648,"available":95899811840,"internally_reserved":0,"allocated":76931072,"data_stored":86497727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":18284544},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:55:25 2021","interfaces":[{"interface":"back","average":{"1min":1.7450000000000001,"5min":0.85699999999999998,"15min":0.70899999999999996},"min":{"1min":0.42799999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":23.846,"5min":23.846,"15min":23.846},"last":0.86699999999999999},{"interface":"front","average":{"1min":1.819,"5min":0.96399999999999997,"15min":0.79800000000000004},"min":{"1min":0.432,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":23.734999999999999,"5min":23.734999999999999,"15min":23.734999999999999},"last":0.80700000000000005}]},{"osd":1,"last update":"Wed Mar 24 21:55:25 2021","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":0.89800000000000002,"15min":0.79000000000000004},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":14.324,"5min":14.324,"15min":14.324},"last":0.70599999999999996},{"interface":"front","average":{"1min":1.3220000000000001,"5min":0.877,"15min":0.79500000000000004},"min":{"1min":0.45300000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":12.631,"5min":12.631,"15min":12.631},"last":0.67400000000000004}]},{"osd":2,"last update":"Wed Mar 24 21:55:25 2021","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":0.93000000000000005,"15min":0.83099999999999996},"min":{"1min":0.48999999999999999,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":12.510999999999999,"5min":12.510999999999999,"15min":12.510999999999999},"last":1.0469999999999999},{"interface":"front","average":{"1min":1.423,"5min":0.92000000000000004,"15min":0.81299999999999994},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":11.317,"5min":11.317,"15min":11.317},"last":1.345}]},{"osd":4,"last update":"Wed Mar 24 21:54:42 2021","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":0.97899999999999998,"15min":0.96199999999999997},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":2.3090000000000002,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":1.627},{"interface":"front","average":{"1min":1.744,"5min":1.1279999999999999,"15min":0.998},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":14.291,"5min":14.291,"15min":14.291},"last":1.381}]},{"osd":5,"last update":"Wed Mar 24 21:54:58 2021","interfaces":[{"interface":"back","average":{"1min":1.6000000000000001,"5min":1.175,"15min":1.0980000000000001},"min":{"1min":0.67800000000000005,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":12.723000000000001,"5min":12.723000000000001,"15min":12.723000000000001},"last":1.427},{"interface":"front","average":{"1min":1.194,"5min":1.0780000000000001,"15min":1.0900000000000001},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":2.375,"5min":2.375,"15min":2.375},"last":1.5329999999999999}]},{"osd":6,"last update":"Wed Mar 24 21:55:25 2021","interfaces":[{"interface":"back","average":{"1min":1.7989999999999999,"5min":1.2450000000000001,"15min":1.1679999999999999},"min":{"1min":0.68400000000000005,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":14.255000000000001,"5min":14.255000000000001,"15min":14.255000000000001},"last":1.472},{"interface":"front","average":{"1min":1.7490000000000001,"5min":1.2709999999999999,"15min":1.1930000000000001},"min":{"1min":0.73399999999999999,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":12.666,"5min":12.666,"15min":12.666},"last":1.2809999999999999}]},{"osd":7,"last update":"Wed Mar 24 21:54:48 2021","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.129,"15min":1.0940000000000001},"min":{"1min":0.69899999999999995,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":2.3479999999999999,"5min":2.3479999999999999,"15min":2.3479999999999999},"last":1.4510000000000001},{"interface":"front","average":{"1min":1.8999999999999999,"5min":1.3160000000000001,"15min":1.212},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":14.727,"5min":14.727,"15min":14.727},"last":1.5669999999999999}]}]},{"osd":4,"up_from":32,"seq":137438953535,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":68028,"kb_used_data":55156,"kb_used_omap":0,"kb_used_meta":12864,"kb_avail":93677124,"statfs":{"total":95995035648,"available":95925374976,"internally_reserved":0,"allocated":56479744,"data_stored":63219723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":13172736},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:54:38 2021","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.85699999999999998,"15min":0.84499999999999997},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.2030000000000001,"5min":1.268,"15min":1.268},"last":0.56299999999999994},{"interface":"front","average":{"1min":1.2509999999999999,"5min":0.97299999999999998,"15min":0.88900000000000001},"min":{"1min":0.68400000000000005,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":6.3929999999999998,"5min":6.3929999999999998,"15min":6.3929999999999998},"last":0.58799999999999997}]},{"osd":1,"last update":"Wed Mar 24 21:54:38 2021","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.91600000000000004,"15min":0.91700000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.3120000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.89900000000000002,"15min":0.85699999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.873,"5min":1.873,"15min":1.873},"last":0.81299999999999994}]},{"osd":2,"last update":"Wed Mar 24 21:54:38 2021","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":0.94399999999999995,"15min":0.91100000000000003},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":4.4379999999999997,"5min":4.4379999999999997,"15min":4.4379999999999997},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.93899999999999995,"15min":0.91500000000000004},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.28,"5min":1.335,"15min":1.335},"last":0.55400000000000005}]},{"osd":3,"last update":"Wed Mar 24 21:54:38 2021","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.93799999999999994,"15min":0.93700000000000006},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.3240000000000001,"5min":1.601,"15min":1.601},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.96099999999999997,"5min":0.97499999999999998,"15min":0.95399999999999996},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.272,"5min":1.474,"15min":1.474},"last":0.63700000000000001}]},{"osd":5,"last update":"Wed Mar 24 21:55:01 2021","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.93500000000000005,"15min":0.94499999999999995},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.2150000000000001,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.0569999999999999,"15min":1.0109999999999999},"min":{"1min":0.66800000000000004,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.3200000000000001,"5min":2.0760000000000001,"15min":2.0760000000000001},"last":0.50700000000000001}]},{"osd":6,"last update":"Wed Mar 24 21:55:22 2021","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.048,"15min":1.0600000000000001},"min":{"1min":0.58299999999999996,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.405,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":0.49099999999999999},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.0549999999999999,"15min":1.0620000000000001},"min":{"1min":0.74199999999999999,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.224,"5min":1.542,"15min":1.542},"last":0.41099999999999998}]},{"osd":7,"last update":"Wed Mar 24 21:54:46 2021","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":1.155,"15min":1.1830000000000001},"min":{"1min":0.64200000000000002,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.552,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":0.74299999999999999},{"interface":"front","average":{"1min":1.1479999999999999,"5min":1.179,"15min":1.1970000000000001},"min":{"1min":0.73399999999999999,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.4199999999999999,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":0.54600000000000004}]}]},{"osd":5,"up_from":39,"seq":167503724604,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":39412,"kb_used_data":27436,"kb_used_omap":0,"kb_used_meta":11968,"kb_avail":93705740,"statfs":{"total":95995035648,"available":95954677760,"internally_reserved":0,"allocated":28094464,"data_stored":30335950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":12255232},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:55:03 2021","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.872,"15min":0.88300000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.3280000000000001,"5min":2.29,"15min":2.29},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.92000000000000004,"15min":0.89600000000000002},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.272,"5min":3.1779999999999999,"15min":3.1779999999999999},"last":1.036}]},{"osd":1,"last update":"Wed Mar 24 21:55:03 2021","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.92400000000000004,"15min":0.92200000000000004},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.3049999999999999,"5min":2.8690000000000002,"15min":2.8690000000000002},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.92600000000000005,"15min":0.93000000000000005},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.395,"5min":2.0030000000000001,"15min":2.0030000000000001},"last":0.56999999999999995}]},{"osd":2,"last update":"Wed Mar 24 21:55:03 2021","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":0.96899999999999997,"15min":0.92500000000000004},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":6.5199999999999996,"5min":6.5199999999999996,"15min":6.5199999999999996},"last":0.96199999999999997},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.94599999999999995,"15min":0.90700000000000003},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.2030000000000001,"5min":5.8479999999999999,"15min":5.8479999999999999},"last":0.84199999999999997}]},{"osd":3,"last update":"Wed Mar 24 21:55:03 2021","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.95999999999999996,"15min":0.94199999999999995},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.3500000000000001,"5min":3.1259999999999999,"15min":3.1259999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.95699999999999996,"5min":0.98199999999999998,"15min":0.95399999999999996},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.3089999999999999,"5min":2.6850000000000001,"15min":2.6850000000000001},"last":0.86899999999999999}]},{"osd":4,"last update":"Wed Mar 24 21:55:03 2021","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":0.91400000000000003,"15min":0.85499999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.4339999999999999,"5min":2.2320000000000002,"15min":2.2320000000000002},"last":0.68300000000000005},{"interface":"front","average":{"1min":1.0549999999999999,"5min":0.98999999999999999,"15min":0.92200000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.54,"5min":1.6970000000000001,"15min":1.6970000000000001},"last":0.70199999999999996}]},{"osd":6,"last update":"Wed Mar 24 21:55:22 2021","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":0.97199999999999998,"15min":0.89700000000000002},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.3919999999999999,"5min":2.085,"15min":2.085},"last":0.77000000000000002},{"interface":"front","average":{"1min":1.052,"5min":1.048,"15min":0.98499999999999999},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.51,"5min":2.3079999999999998,"15min":2.3079999999999998},"last":0.99099999999999999}]},{"osd":7,"last update":"Wed Mar 24 21:54:45 2021","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.083,"15min":1.0569999999999999},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.6140000000000001,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.90400000000000003},{"interface":"front","average":{"1min":1.169,"5min":1.0580000000000001,"15min":1.002},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.4890000000000001,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":0.88600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":14134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":7188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":15432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":11740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":11460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":299008,"data_stored":10618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":73039872,"data_stored":79222672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7032832,"data_stored":6977532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":44703744,"data_stored":55009713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":75567104,"data_stored":85963702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":55226368,"data_stored":62696247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":26861568,"data_stored":29818891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":68358144,"data_stored":80585934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-03-24T21:55:43.700 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd pool set iscsi pgp_num 3 2021-03-24T21:55:43.718 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:43.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:43.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:55:43.824 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:43.885 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:43.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:43.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:55:43.991 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:44.030 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:55:44.044 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:44.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:44.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:55:44.150 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:44.201 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:44.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:44.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:44.275 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:44 smithi142 conmon[20125]: cluster 2021-03-24T21:55:42. 2021-03-24T21:55:44.275 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:44 smithi142 conmon[20125]: 732797+0000 mgr.y (mgr.14144) 346 : cluster [DBG] pgmap v422: 148 pgs: 148 active+clean; 142 MiB data, 420 MiB used, 536 GiB / 536 GiB avail; 9.3 MiB/s rd, 2.9 MiB/s wr, 16 op/s 2021-03-24T21:55:44.275 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:44 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:55:44.276 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:44 smithi142 conmon[20125]: 21:55:43.141677+0000 mon.a (mon.0) 753 : cluster [DBG] osdmap e174: 8 total, 8 up, 6 in 2021-03-24T21:55:44.310 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:44.359 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:44.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:44.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:55:44.373 INFO:tasks.rados.rados.0.smithi125.stdout:384: done (3 left) 2021-03-24T21:55:44.373 INFO:tasks.rados.rados.0.smithi125.stdout:385: snap_create 2021-03-24T21:55:44.374 INFO:tasks.rados.rados.0.smithi125.stdout:381: finishing write tid 3 to smithi12556470-34 2021-03-24T21:55:44.374 INFO:tasks.rados.rados.0.smithi125.stdout:381: finishing write tid 4 to smithi12556470-34 2021-03-24T21:55:44.374 INFO:tasks.rados.rados.0.smithi125.stdout:381: finishing write tid 5 to smithi12556470-34 2021-03-24T21:55:44.374 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 34 v 132 (ObjNum 167 snap 36 seq_num 167) dirty exists 2021-03-24T21:55:44.375 INFO:tasks.rados.rados.0.smithi125.stdout:381: left oid 34 (ObjNum 167 snap 36 seq_num 167) 2021-03-24T21:55:44.375 INFO:tasks.rados.rados.0.smithi125.stdout:382: expect (ObjNum 126 snap 23 seq_num 126) 2021-03-24T21:55:44.388 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:44 smithi125 conmon[19936]: cluster 2021-03-24T21:55:42. 2021-03-24T21:55:44.388 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:44 smithi125 conmon[19936]: 732797+0000 mgr.y (mgr.14144) 346 : cluster [DBG] pgmap v422: 148 pgs: 148 active+clean; 142 MiB data, 420 MiB used, 536 GiB / 536 GiB avail; 9.3 MiB/s rd, 2.9 MiB/s wr, 16 op/s 2021-03-24T21:55:44.388 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:44 smithi125 conmon[19936]: cluster 2021-03-24T21:55:43.141677 2021-03-24T21:55:44.388 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:44 smithi125 conmon[19936]: +0000 mon.a (mon.0) 753 : cluster [DBG] osdmap e174: 8 total, 8 up, 6 in 2021-03-24T21:55:44.389 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:44 smithi125 conmon[24927]: cluster 2021-03-24T21:55 2021-03-24T21:55:44.389 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:44 smithi125 conmon[24927]: :42.732797+0000 mgr.y (mgr.14144) 346 : cluster [DBG] pgmap v422: 148 pgs: 148 active+clean; 142 MiB data, 420 MiB used, 536 GiB / 536 GiB avail; 9.3 MiB/s rd, 2.9 MiB/s wr, 16 op/s 2021-03-24T21:55:44.389 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:44 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:55:44.390 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:44 smithi125 conmon[24927]: -24T21:55:43.141677+0000 mon.a (mon.0) 753 : cluster [DBG] osdmap e174: 8 total, 8 up, 6 in 2021-03-24T21:55:44.390 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:44 smithi125 conmon[24927]: 2021-03-24T21:55:44.390 INFO:journalctl@ceph.osd.0.smithi125.stdout:Mar 24 21:55:44 smithi125 conmon[28026]: debug 2021-03-24T21:55:44.378+0000 7f61bbc86700 -1 osd.0 pg_epoch: 175 pg[7.4( v 175'102 (0'0,175'102] local-lis/les=108/109 n=5 ec=99/99 lis/c=108/108 les/c/f=109/110/0 sis=108) [0,4,5] r=0 lpr=108 luod=172'100 crt=175'102 lcod 172'99 mlcod 172'99 active+clean+snaptrim trimq=[2e~1] ps=[3b~1,3f~2]] removing snap head 2021-03-24T21:55:44.391 INFO:journalctl@ceph.osd.3.smithi125.stdout:Mar 24 21:55:44 smithi125 conmon[37997]: debug 2021-03-24T21:55:44.374+0000 7f1717bdc700 -1 osd.3 pg_epoch: 175 pg[7.a( v 173'52 (0'0,173'52] local-lis/les=99/100 n=2 ec=99/99 lis/c=99/99 les/c/f=100/100/0 sis=99) [3,2,5] r=0 lpr=99 crt=155'51 lcod 155'51 mlcod 155'51 active+clean+snaptrim trimq=[2e~1] ps=[3b~1,3f~2]] removing snap head 2021-03-24T21:55:44.465 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:44.470 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T21:55:44.474 INFO:tasks.rados.rados.0.smithi125.stdout:383: finishing copy_from to smithi12556470-14 2021-03-24T21:55:44.474 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 14 v 98 (ObjNum 123 snap 23 seq_num 123) dirty exists 2021-03-24T21:55:44.475 INFO:tasks.rados.rados.0.smithi125.stdout:383: finishing copy_from racing read to smithi12556470-14 2021-03-24T21:55:44.496 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T21:55:44.509 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:44.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:44.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:55:44.526 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T21:55:44.542 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2021-03-24T21:55:44.543 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T21:55:44.545 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T21:55:44.554 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 21:55:44 smithi142 conmon[27337]: debug 2021-03-24T21:55:44.554 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 21:55:44 smithi142 conmon[27337]: 2021-03-24T21:55:44.380+0000 7f549af13700 -1 osd.5 pg_epoch: 175 pg[7.0( v 164'20 (0'0,164'20] local-lis/les=108/109 n=2 ec=99/99 lis/c=108/108 les/c/f=109/109/0 sis=108) [5,3,2] r=0 lpr=108 crt=163'19 lcod 163'19 mlcod 163'19 active+clean+snaptrim trimq=[2e~1] ps=[3b~1,3f~2]] removing snap head 2021-03-24T21:55:44.554 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 21:55:44 smithi142 conmon[27337]: 2021-03-24T21:55:44.570 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T21:55:44.593 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T21:55:44.613 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:44.617 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T21:55:44.639 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T21:55:44.641 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-24T21:55:44.641 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-24T21:55:44.641 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-24T21:55:44.642 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-24T21:55:44.642 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-24T21:55:44.642 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-24T21:55:44.642 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-24T21:55:44.643 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-24T21:55:44.657 INFO:journalctl@ceph.osd.3.smithi125.stdout:Mar 24 21:55:44 smithi125 conmon[37997]: debug 2021-03-24T21:55:44.657 INFO:journalctl@ceph.osd.3.smithi125.stdout:Mar 24 21:55:44 smithi125 conmon[37997]: 2021-03-24T21:55:44.382+0000 7f171b3e3700 -1 osd.3 pg_epoch: 175 pg[7.3( v 174'132 (0'0,174'132] local-lis/les=99/100 n=7 ec=99/99 lis/c=99/99 les/c/f=100/100/0 sis=99) [3,2,4] r=0 lpr=99 crt=174'132 lcod 174'131 mlcod 174'131 active+clean+snaptrim trimq=[2e~1] ps=[3b~1,3f~2]] removing snap head 2021-03-24T21:55:44.658 INFO:journalctl@ceph.osd.3.smithi125.stdout:Mar 24 21:55:44 smithi125 conmon[37997]: 2021-03-24T21:55:44.670 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:44.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:44.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:55:44.776 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:44.837 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:44.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:44.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:55:44.941 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:44.991 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:44.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:44.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:55:45.097 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:45.117 INFO:tasks.rados.rados.0.smithi125.stdout:381: done (3 left) 2021-03-24T21:55:45.117 INFO:tasks.rados.rados.0.smithi125.stdout:382: done (2 left) 2021-03-24T21:55:45.117 INFO:tasks.rados.rados.0.smithi125.stdout:383: done (1 left) 2021-03-24T21:55:45.117 INFO:tasks.rados.rados.0.smithi125.stdout:385: done (0 left) 2021-03-24T21:55:45.118 INFO:tasks.rados.rados.0.smithi125.stdout:386: write oid 49 current snap is 37 2021-03-24T21:55:45.118 INFO:tasks.rados.rados.0.smithi125.stdout:386: seq_num 168 ranges {423576=401598,1497916=693725} 2021-03-24T21:55:45.118 INFO:tasks.rados.rados.0.smithi125.stdout:386: writing smithi12556470-49 from 423576 to 825174 tid 1 2021-03-24T21:55:45.126 INFO:tasks.rados.rados.0.smithi125.stdout:386: writing smithi12556470-49 from 1497916 to 2191641 tid 2 2021-03-24T21:55:45.126 INFO:tasks.rados.rados.0.smithi125.stdout:387: delete oid 18 current snap is 37 2021-03-24T21:55:45.126 INFO:tasks.rados.rados.0.smithi125.stdout:386: finishing write tid 1 to smithi12556470-49 2021-03-24T21:55:45.127 INFO:tasks.rados.rados.0.smithi125.stdout:387: done (1 left) 2021-03-24T21:55:45.128 INFO:tasks.rados.rados.0.smithi125.stdout:388: snap_remove snap 35 2021-03-24T21:55:45.150 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:45.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:45.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:45.255 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:45.318 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:45.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:45.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:45.424 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:45.479 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:45.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:45.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:55:45.545 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.5 dump_ops_in_flight 2021-03-24T21:55:45.585 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:45.639 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:45.644 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:45 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:55:45.644 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:45 smithi125 conmon[24927]: 21:55:44.371506+0000 mon.a (mon.0) 754 : cluster [DBG] osdmap e175: 8 total, 8 up, 6 in 2021-03-24T21:55:45.644 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:45 smithi125 conmon[24927]: audit 2021- 2021-03-24T21:55:45.644 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:45 smithi125 conmon[24927]: 03-24T21:55:44.448147+0000 mon.c (mon 2021-03-24T21:55:45.645 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:45 smithi125 conmon[24927]: .2) 31 : audit [DBG] from='client.? v1:172.21.15.125:0/3144440729' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:55:45.645 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:45 smithi125 conmon[24927]: 2021-03-24T21:55:45.645 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:45 smithi125 conmon[24927]: cluster 2021-03-24T21:55:45.645 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:45 smithi125 conmon[24927]: 2021-03-24T21:55: 2021-03-24T21:55:45.646 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:45 smithi125 conmon[24927]: 45.107185+0000 mon.a (mon 2021-03-24T21:55:45.646 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:45 smithi125 conmon[24927]: .0) 755 : cluster [DBG] 2021-03-24T21:55:45.646 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:45 smithi125 conmon[24927]: osdmap e176: 8 total, 8 up, 6 in 2021-03-24T21:55:45.646 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:45 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T21:55:45.647 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:45 smithi125 conmon[19936]: T21:55:44.371506+0000 mon.a (mon.0) 754 : cluster [DBG] osdmap e175: 8 total, 8 up, 6 in 2021-03-24T21:55:45.647 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:45 smithi125 conmon[19936]: audit 2021-03-24T21:55:45.647 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:45 smithi125 conmon[19936]: 2021-03-24T21:55: 2021-03-24T21:55:45.647 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:45 smithi125 conmon[19936]: 44.448147+0000 mon.c (mon 2021-03-24T21:55:45.648 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:45 smithi125 conmon[19936]: .2) 31 : audit [DBG] 2021-03-24T21:55:45.648 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:45 smithi125 conmon[19936]: from='client.? v1:172.21.15.125:0/3144440729' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:55:45.648 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:45 smithi125 conmon[19936]: cluster 2021-03-24T21:55:45.648 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:45 smithi125 conmon[19936]: 2021-03-24T21 2021-03-24T21:55:45.649 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:45 smithi125 conmon[19936]: :55:45.107185+0000 2021-03-24T21:55:45.649 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:45 smithi125 conmon[19936]: mon.a (mon.0) 755 : 2021-03-24T21:55:45.649 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:45 smithi125 conmon[19936]: cluster [DBG] osdmap e176: 8 total, 8 up, 6 in 2021-03-24T21:55:45.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:45.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:45.751 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:45.759 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:45 smithi142 conmon[20125]: cluster 2021-03-24T21:55: 2021-03-24T21:55:45.759 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:45 smithi142 conmon[20125]: 44.371506+0000 mon.a (mon.0) 754 : cluster [DBG] osdmap e175: 8 total, 8 up, 6 in 2021-03-24T21:55:45.759 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:45 smithi142 conmon[20125]: audit 2021-03-24T21:55:44. 2021-03-24T21:55:45.760 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:45 smithi142 conmon[20125]: 448147+0000 mon.c (mon.2) 31 : audit [DBG] from='client.? v1:172.21.15.125:0/3144440729' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:55:45.760 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:45 smithi142 conmon[20125]: cluster 2021-03-24T21:55:45.107185+0000 mon.a (mon. 2021-03-24T21:55:45.760 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:45 smithi142 conmon[20125]: 0) 755 : cluster [DBG] osdmap e176: 8 total, 8 up, 6 in 2021-03-24T21:55:45.795 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:45.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:45.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:55:45.844 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T21:55:45.900 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:45.953 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:45.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:45.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:45.966 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-24T21:55:46.061 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:46.117 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:46.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:46.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:55:46.130 INFO:tasks.rados.rados.0.smithi125.stdout:388: done (1 left) 2021-03-24T21:55:46.130 INFO:tasks.rados.rados.0.smithi125.stdout:389: snap_remove snap 36 2021-03-24T21:55:46.228 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:46.284 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:46.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:46.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:55:46.333 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:55:46.388 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:46.436 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:46.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:46.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:55:46.516 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:46 smithi125 conmon[19936]: cluster 2021-03-24T21:55:44.733358+0000 mgr.y (mgr.14144) 347 : cluster [DBG] pgmap v425: 148 pgs: 148 active+clean; 168 MiB data, 460 MiB used, 536 GiB / 536 GiB avail; 14 MiB/s rd, 4.5 MiB/s wr, 26 op/s 2021-03-24T21:55:46.517 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:46 smithi125 conmon[19936]: cluster 2021-03-24T21:55:46.123318+0000 mon.a (mon.0) 756 : cluster [DBG] osdmap e177: 8 total, 8 up, 6 in 2021-03-24T21:55:46.517 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:46 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:55:46.518 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:46 smithi125 conmon[24927]: 55:44.733358+0000 mgr.y (mgr.14144) 347 : cluster [DBG] pgmap v425: 148 pgs: 148 active+clean; 168 MiB data, 460 MiB used, 536 GiB / 536 GiB avail; 14 MiB/s rd, 4.5 MiB/s wr, 26 op/s 2021-03-24T21:55:46.518 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:46 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:55:46.518 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:46 smithi125 conmon[24927]: -03-24T21:55:46.123318+0000 2021-03-24T21:55:46.518 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:46 smithi125 conmon[24927]: mon.a (mon.0) 756 : cluster 2021-03-24T21:55:46.519 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:46 smithi125 conmon[24927]: [DBG] osdmap e177: 8 total, 8 up, 6 in 2021-03-24T21:55:46.519 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:46 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:55:46.519 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:46 smithi142 conmon[20125]: T21:55:44.733358+0000 mgr.y (mgr.14144) 347 : cluster [DBG] pgmap v425: 148 pgs: 148 active+clean; 168 MiB data, 460 MiB used, 536 GiB / 536 GiB avail; 14 MiB/s rd, 4.5 MiB/s wr, 26 op/s 2021-03-24T21:55:46.520 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:46 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:55:46.520 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:46 smithi142 conmon[20125]: T21:55:46.123318+0000 mon.a (mon.0) 756 : cluster [DBG] osdmap e177: 8 total, 8 up, 6 in 2021-03-24T21:55:46.542 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:46.592 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:46.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:46.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:55:46.698 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:46.751 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:46.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:46.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:55:46.858 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:46.911 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:46.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:46.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:55:47.016 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:47.073 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:47.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:47.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:55:47.131 INFO:teuthology.orchestra.run.smithi125.stderr:set pool 2 pgp_num to 3 2021-03-24T21:55:47.131 INFO:tasks.rados.rados.0.smithi125.stdout:389: done (1 left) 2021-03-24T21:55:47.132 INFO:tasks.rados.rados.0.smithi125.stdout:390: write oid 28 current snap is 37 2021-03-24T21:55:47.132 INFO:tasks.rados.rados.0.smithi125.stdout:386: finishing write tid 2 to smithi12556470-49 2021-03-24T21:55:47.132 INFO:tasks.rados.rados.0.smithi125.stdout:390: seq_num 170 ranges {457190=552096,1614880=581041,2785791=71399} 2021-03-24T21:55:47.139 INFO:tasks.rados.rados.0.smithi125.stdout:390: writing smithi12556470-28 from 457190 to 1009286 tid 1 2021-03-24T21:55:47.146 INFO:tasks.rados.rados.0.smithi125.stdout:390: writing smithi12556470-28 from 1614880 to 2195921 tid 2 2021-03-24T21:55:47.151 INFO:tasks.rados.rados.0.smithi125.stdout:390: writing smithi12556470-28 from 2785791 to 2857190 tid 3 2021-03-24T21:55:47.151 INFO:tasks.rados.rados.0.smithi125.stdout:391: rollback oid 48 current snap is 37 2021-03-24T21:55:47.152 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 48 to 34 2021-03-24T21:55:47.152 INFO:tasks.rados.rados.0.smithi125.stdout:386: finishing write tid 3 to smithi12556470-49 2021-03-24T21:55:47.152 INFO:tasks.rados.rados.0.smithi125.stdout:386: finishing write tid 4 to smithi12556470-49 2021-03-24T21:55:47.152 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 49 v 128 (ObjNum 168 snap 37 seq_num 168) dirty exists 2021-03-24T21:55:47.152 INFO:tasks.rados.rados.0.smithi125.stdout:386: left oid 49 (ObjNum 168 snap 37 seq_num 168) 2021-03-24T21:55:47.153 INFO:tasks.rados.rados.0.smithi125.stdout:390: finishing write tid 1 to smithi12556470-28 2021-03-24T21:55:47.153 INFO:tasks.rados.rados.0.smithi125.stdout:386: done (2 left) 2021-03-24T21:55:47.153 INFO:tasks.rados.rados.0.smithi125.stdout:392: rollback oid 3 current snap is 37 2021-03-24T21:55:47.153 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 3 to 34 2021-03-24T21:55:47.154 INFO:tasks.rados.rados.0.smithi125.stdout:393: read oid 37 snap 34 2021-03-24T21:55:47.154 INFO:tasks.rados.rados.0.smithi125.stdout:393: expect deleted 2021-03-24T21:55:47.154 INFO:tasks.rados.rados.0.smithi125.stdout:394: copy_from oid 7 from oid 15 current snap is 37 2021-03-24T21:55:47.154 INFO:tasks.rados.rados.0.smithi125.stdout:390: finishing write tid 2 to smithi12556470-28 2021-03-24T21:55:47.154 INFO:tasks.rados.rados.0.smithi125.stdout:395: rollback oid 42 current snap is 37 2021-03-24T21:55:47.155 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 42 to 30 2021-03-24T21:55:47.155 INFO:tasks.rados.rados.0.smithi125.stdout:396: delete oid 10 current snap is 37 2021-03-24T21:55:47.159 INFO:tasks.rados.rados.0.smithi125.stdout:393: done (6 left) 2021-03-24T21:55:47.159 INFO:tasks.rados.rados.0.smithi125.stdout:396: done (5 left) 2021-03-24T21:55:47.159 INFO:tasks.rados.rados.0.smithi125.stdout:397: read oid 44 snap 34 2021-03-24T21:55:47.159 INFO:tasks.rados.rados.0.smithi125.stdout:397: expect (ObjNum 124 snap 23 seq_num 124) 2021-03-24T21:55:47.160 INFO:tasks.rados.rados.0.smithi125.stdout:398: read oid 14 snap -1 2021-03-24T21:55:47.160 INFO:tasks.rados.rados.0.smithi125.stdout:398: expect (ObjNum 123 snap 23 seq_num 123) 2021-03-24T21:55:47.160 INFO:tasks.rados.rados.0.smithi125.stdout:399: read oid 23 snap 31 2021-03-24T21:55:47.160 INFO:tasks.rados.rados.0.smithi125.stdout:399: expect (ObjNum 133 snap 24 seq_num 133) 2021-03-24T21:55:47.160 INFO:tasks.rados.rados.0.smithi125.stdout:400: rollback oid 32 current snap is 37 2021-03-24T21:55:47.161 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 32 to 31 2021-03-24T21:55:47.161 INFO:tasks.rados.rados.0.smithi125.stdout:401: rollback oid 39 current snap is 37 2021-03-24T21:55:47.161 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 39 to 34 2021-03-24T21:55:47.161 INFO:tasks.rados.rados.0.smithi125.stdout:402: read oid 13 snap 34 2021-03-24T21:55:47.162 INFO:tasks.rados.rados.0.smithi125.stdout:402: expect (ObjNum 123 snap 23 seq_num 123) 2021-03-24T21:55:47.162 INFO:tasks.rados.rados.0.smithi125.stdout:403: read oid 34 snap -1 2021-03-24T21:55:47.162 INFO:tasks.rados.rados.0.smithi125.stdout:403: expect (ObjNum 167 snap 36 seq_num 167) 2021-03-24T21:55:47.162 INFO:tasks.rados.rados.0.smithi125.stdout:404: delete oid 5 current snap is 37 2021-03-24T21:55:47.162 INFO:tasks.rados.rados.0.smithi125.stdout:391: finishing rollback tid 0 to smithi12556470-48 2021-03-24T21:55:47.163 INFO:tasks.rados.rados.0.smithi125.stdout:391: finishing rollback tid 1 to smithi12556470-48 2021-03-24T21:55:47.163 INFO:tasks.rados.rados.0.smithi125.stdout:391: finishing rollback tid 2 to smithi12556470-48 2021-03-24T21:55:47.164 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 48 v 132 (ObjNum 62 snap 4 seq_num 62) dirty exists 2021-03-24T21:55:47.164 INFO:tasks.rados.rados.0.smithi125.stdout:392: finishing rollback tid 0 to smithi12556470-3 2021-03-24T21:55:47.164 INFO:tasks.rados.rados.0.smithi125.stdout:390: finishing write tid 3 to smithi12556470-28 2021-03-24T21:55:47.164 INFO:tasks.rados.rados.0.smithi125.stdout:390: finishing write tid 4 to smithi12556470-28 2021-03-24T21:55:47.165 INFO:tasks.rados.rados.0.smithi125.stdout:390: finishing write tid 5 to smithi12556470-28 2021-03-24T21:55:47.165 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 28 v 109 (ObjNum 170 snap 37 seq_num 170) dirty exists 2021-03-24T21:55:47.165 INFO:tasks.rados.rados.0.smithi125.stdout:390: left oid 28 (ObjNum 170 snap 37 seq_num 170) 2021-03-24T21:55:47.166 INFO:tasks.rados.rados.0.smithi125.stdout:392: finishing rollback tid 1 to smithi12556470-3 2021-03-24T21:55:47.166 INFO:tasks.rados.rados.0.smithi125.stdout:392: finishing rollback tid 2 to smithi12556470-3 2021-03-24T21:55:47.166 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 3 v 69 (ObjNum 149 snap 30 seq_num 149) dirty exists 2021-03-24T21:55:47.172 INFO:tasks.rados.rados.0.smithi125.stdout:397: expect (ObjNum 124 snap 23 seq_num 124) 2021-03-24T21:55:47.178 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:47.228 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:47.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:47.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:55:47.286 INFO:tasks.rados.rados.0.smithi125.stdout:402: expect (ObjNum 123 snap 23 seq_num 123) 2021-03-24T21:55:47.334 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:47.391 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:47.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:47.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:55:47.404 INFO:tasks.rados.rados.0.smithi125.stdout:398: expect (ObjNum 123 snap 23 seq_num 123) 2021-03-24T21:55:47.497 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:47.510 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:47 smithi142 conmon[20125]: audit 2021-03-24T21: 2021-03-24T21:55:47.511 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:47 smithi142 conmon[20125]: 55:46.979760+0000 mon.a (mon.0) 757 : audit [INF] from='client.? v1:172.21.15.125:0/2020279630' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"}]: dispatch 2021-03-24T21:55:47.511 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:47 smithi142 conmon[20125]: audit 2021-03-24T21:55:47.511 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:47 smithi142 conmon[20125]: 2021-03-24 2021-03-24T21:55:47.511 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:47 smithi142 conmon[20125]: T21:55: 2021-03-24T21:55:47.511 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:47 smithi142 conmon[20125]: 47.124833+0000 2021-03-24T21:55:47.512 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:47 smithi142 conmon[20125]: mon.a (mon. 2021-03-24T21:55:47.512 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:47 smithi142 conmon[20125]: 0) 758 : audit 2021-03-24T21:55:47.512 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:47 smithi142 conmon[20125]: [INF] from='client.? v1:172.21.15.125:0/2020279630' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"}]': finished 2021-03-24T21:55:47.512 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:47 smithi142 conmon[20125]: 2021-03-24T21:55:47.513 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:47 smithi142 conmon[20125]: cluster 2021-03-24T21:55:47.513 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:47 smithi142 conmon[20125]: 2021-03-24 2021-03-24T21:55:47.513 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:47 smithi142 conmon[20125]: T21:55: 2021-03-24T21:55:47.513 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:47 smithi142 conmon[20125]: 47.125065+0000 2021-03-24T21:55:47.513 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:47 smithi142 conmon[20125]: mon.a (mon 2021-03-24T21:55:47.514 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:47 smithi142 conmon[20125]: .0) 759 : 2021-03-24T21:55:47.514 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:47 smithi142 conmon[20125]: cluster [DBG] 2021-03-24T21:55:47.514 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:47 smithi142 conmon[20125]: osdmap e178: 8 total, 8 up, 6 in 2021-03-24T21:55:47.519 INFO:tasks.rados.rados.0.smithi125.stdout:395: finishing rollback tid 1 to smithi12556470-42 2021-03-24T21:55:47.520 INFO:tasks.rados.rados.0.smithi125.stdout:395: finishing rollback tid 2 to smithi12556470-42 2021-03-24T21:55:47.520 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 42 v 71 (ObjNum 91 snap 14 seq_num 91) dirty exists 2021-03-24T21:55:47.520 INFO:tasks.rados.rados.0.smithi125.stdout:400: finishing rollback tid 0 to smithi12556470-32 2021-03-24T21:55:47.520 INFO:tasks.rados.rados.0.smithi125.stdout:403: expect (ObjNum 167 snap 36 seq_num 167) 2021-03-24T21:55:47.548 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:47.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:47.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:55:47.609 INFO:tasks.rados.rados.0.smithi125.stdout:399: expect (ObjNum 133 snap 24 seq_num 133) 2021-03-24T21:55:47.653 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:47.685 INFO:tasks.rados.rados.0.smithi125.stdout:400: finishing rollback tid 1 to smithi12556470-32 2021-03-24T21:55:47.685 INFO:tasks.rados.rados.0.smithi125.stdout:400: finishing rollback tid 2 to smithi12556470-32 2021-03-24T21:55:47.685 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 32 v 128 (ObjNum 108 snap 22 seq_num 108) dirty exists 2021-03-24T21:55:47.685 INFO:tasks.rados.rados.0.smithi125.stdout:401: finishing rollback tid 0 to smithi12556470-39 2021-03-24T21:55:47.686 INFO:tasks.rados.rados.0.smithi125.stdout:394: finishing copy_from to smithi12556470-7 2021-03-24T21:55:47.686 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 7 v 92 (ObjNum 150 snap 30 seq_num 150) dirty exists 2021-03-24T21:55:47.686 INFO:tasks.rados.rados.0.smithi125.stdout:394: finishing copy_from racing read to smithi12556470-7 2021-03-24T21:55:47.686 INFO:tasks.rados.rados.0.smithi125.stdout:401: finishing rollback tid 1 to smithi12556470-39 2021-03-24T21:55:47.686 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 39 v 141 (ObjNum 1116406448 snap 21916 seq_num 1116619072) dirty dne 2021-03-24T21:55:47.687 INFO:tasks.rados.rados.0.smithi125.stdout:390: done (12 left) 2021-03-24T21:55:47.687 INFO:tasks.rados.rados.0.smithi125.stdout:391: done (11 left) 2021-03-24T21:55:47.687 INFO:tasks.rados.rados.0.smithi125.stdout:392: done (10 left) 2021-03-24T21:55:47.687 INFO:tasks.rados.rados.0.smithi125.stdout:394: done (9 left) 2021-03-24T21:55:47.688 INFO:tasks.rados.rados.0.smithi125.stdout:395: done (8 left) 2021-03-24T21:55:47.688 INFO:tasks.rados.rados.0.smithi125.stdout:397: done (7 left) 2021-03-24T21:55:47.688 INFO:tasks.rados.rados.0.smithi125.stdout:398: done (6 left) 2021-03-24T21:55:47.688 INFO:tasks.rados.rados.0.smithi125.stdout:399: done (5 left) 2021-03-24T21:55:47.688 INFO:tasks.rados.rados.0.smithi125.stdout:400: done (4 left) 2021-03-24T21:55:47.689 INFO:tasks.rados.rados.0.smithi125.stdout:401: done (3 left) 2021-03-24T21:55:47.689 INFO:tasks.rados.rados.0.smithi125.stdout:402: done (2 left) 2021-03-24T21:55:47.689 INFO:tasks.rados.rados.0.smithi125.stdout:403: done (1 left) 2021-03-24T21:55:47.689 INFO:tasks.rados.rados.0.smithi125.stdout:404: done (0 left) 2021-03-24T21:55:47.689 INFO:tasks.rados.rados.0.smithi125.stdout:405: rollback oid 16 current snap is 37 2021-03-24T21:55:47.690 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 16 to 31 2021-03-24T21:55:47.690 INFO:tasks.rados.rados.0.smithi125.stdout:406: read oid 24 snap -1 2021-03-24T21:55:47.690 INFO:tasks.rados.rados.0.smithi125.stdout:406: expect (ObjNum 106 snap 21 seq_num 106) 2021-03-24T21:55:47.690 INFO:tasks.rados.rados.0.smithi125.stdout:407: delete oid 23 current snap is 37 2021-03-24T21:55:47.691 INFO:tasks.rados.rados.0.smithi125.stdout:407: done (2 left) 2021-03-24T21:55:47.691 INFO:tasks.rados.rados.0.smithi125.stdout:408: write (excl) oid 41 current snap is 37 2021-03-24T21:55:47.692 INFO:tasks.rados.rados.0.smithi125.stdout:408: seq_num 174 ranges {524418=453092,1732455=588379,2779130=145288} 2021-03-24T21:55:47.700 INFO:tasks.rados.rados.0.smithi125.stdout:408: writing smithi12556470-41 from 524418 to 977510 tid 1 2021-03-24T21:55:47.705 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:47.709 INFO:tasks.rados.rados.0.smithi125.stdout:408: writing smithi12556470-41 from 1732455 to 2320834 tid 2 2021-03-24T21:55:47.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:47.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:47.713 INFO:tasks.rados.rados.0.smithi125.stdout:408: writing smithi12556470-41 from 2779130 to 2924418 tid 3 2021-03-24T21:55:47.713 INFO:tasks.rados.rados.0.smithi125.stdout:409: delete oid 26 current snap is 37 2021-03-24T21:55:47.713 INFO:tasks.rados.rados.0.smithi125.stdout:406: expect (ObjNum 106 snap 21 seq_num 106) 2021-03-24T21:55:47.738 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:47 smithi125 conmon[24927]: audit 2021-03-24T21:55:46.979760+0000 mon.a (mon.0) 757 : audit [INF] from='client.? v1:172.21.15.125:0/2020279630' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"}]: dispatch 2021-03-24T21:55:47.738 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:47 smithi125 conmon[24927]: audit 2021-03-24T21:55:47.124833+0000 mon.a (mon.0) 758 : audit [INF] from='client.? v1:172.21.15.125:0/2020279630' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"}]': finished 2021-03-24T21:55:47.738 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:47 smithi125 conmon[24927]: cluster 2021-03-24T21:55:47.125065+0000 mon.a (mon.0) 759 : cluster [DBG] osdmap e178: 8 total, 8 up, 6 in 2021-03-24T21:55:47.739 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:47 smithi125 conmon[19936]: audit 2021-03-24T21:55:46.979760+0000 mon.a (mon 2021-03-24T21:55:47.739 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:47 smithi125 conmon[19936]: .0) 757 : audit [INF] from='client.? v1:172.21.15.125:0/2020279630' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"}]: dispatch 2021-03-24T21:55:47.739 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:47 smithi125 conmon[19936]: audit 2021-03-24T21:55:47.124833+0000 mon.a (mon.0) 758 : audit [INF] from='client.? v1:172.21.15.125:0/2020279630' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"}]': finished 2021-03-24T21:55:47.740 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:47 smithi125 conmon[19936]: 2021-03-24T21:55:47.740 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:47 smithi125 conmon[19936]: cluster 2021-03-24T21:55:47.125065+0000 mon.a ( 2021-03-24T21:55:47.740 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:47 smithi125 conmon[19936]: mon.0) 759 : cluster [DBG] osdmap e178: 8 total, 8 up, 6 in 2021-03-24T21:55:47.813 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:47.868 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:47.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:47.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:55:47.896 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2021-03-24T21:55:47.896 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2021-03-24T21:55:47.896 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2021-03-24T21:55:47.896 INFO:teuthology.orchestra.run.smithi142.stdout:} 2021-03-24T21:55:47.936 INFO:tasks.rados.rados.0.smithi125.stdout:405: finishing rollback tid 0 to smithi12556470-16 2021-03-24T21:55:47.937 INFO:tasks.rados.rados.0.smithi125.stdout:405: finishing rollback tid 1 to smithi12556470-16 2021-03-24T21:55:47.937 INFO:tasks.rados.rados.0.smithi125.stdout:405: finishing rollback tid 2 to smithi12556470-16 2021-03-24T21:55:47.937 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 16 v 145 (ObjNum 87 snap 11 seq_num 87) dirty exists 2021-03-24T21:55:47.938 INFO:tasks.rados.rados.0.smithi125.stdout:408: finishing write tid 1 to smithi12556470-41 2021-03-24T21:55:47.938 INFO:tasks.rados.rados.0.smithi125.stdout:405: done (3 left) 2021-03-24T21:55:47.938 INFO:tasks.rados.rados.0.smithi125.stdout:406: done (2 left) 2021-03-24T21:55:47.938 INFO:tasks.rados.rados.0.smithi125.stdout:409: done (1 left) 2021-03-24T21:55:47.939 INFO:tasks.rados.rados.0.smithi125.stdout:410: rollback oid 6 current snap is 37 2021-03-24T21:55:47.939 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 6 to 34 2021-03-24T21:55:47.939 INFO:tasks.rados.rados.0.smithi125.stdout:408: finishing write tid 2 to smithi12556470-41 2021-03-24T21:55:47.939 INFO:tasks.rados.rados.0.smithi125.stdout:408: finishing write tid 3 to smithi12556470-41 2021-03-24T21:55:47.939 INFO:tasks.rados.rados.0.smithi125.stdout:408: finishing write tid 4 to smithi12556470-41 2021-03-24T21:55:47.940 INFO:tasks.rados.rados.0.smithi125.stdout:408: finishing write tid 5 to smithi12556470-41 2021-03-24T21:55:47.940 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 41 v 127 (ObjNum 174 snap 37 seq_num 174) dirty exists 2021-03-24T21:55:47.940 INFO:tasks.rados.rados.0.smithi125.stdout:408: left oid 41 (ObjNum 174 snap 37 seq_num 174) 2021-03-24T21:55:47.940 INFO:tasks.rados.rados.0.smithi125.stdout:408: done (1 left) 2021-03-24T21:55:47.941 INFO:tasks.rados.rados.0.smithi125.stdout:411: write oid 5 current snap is 37 2021-03-24T21:55:47.941 INFO:tasks.rados.rados.0.smithi125.stdout:411: seq_num 176 ranges {558032=719943,1849419=592048} 2021-03-24T21:55:47.949 INFO:tasks.rados.rados.0.smithi125.stdout:411: writing smithi12556470-5 from 558032 to 1277975 tid 1 2021-03-24T21:55:47.957 INFO:tasks.rados.rados.0.smithi125.stdout:411: writing smithi12556470-5 from 1849419 to 2441467 tid 2 2021-03-24T21:55:47.957 INFO:tasks.rados.rados.0.smithi125.stdout:412: read oid 46 snap -1 2021-03-24T21:55:47.957 INFO:tasks.rados.rados.0.smithi125.stdout:412: expect (ObjNum 83 snap 10 seq_num 83) 2021-03-24T21:55:47.958 INFO:tasks.rados.rados.0.smithi125.stdout:413: read oid 48 snap -1 2021-03-24T21:55:47.958 INFO:tasks.rados.rados.0.smithi125.stdout:413: expect (ObjNum 62 snap 4 seq_num 62) 2021-03-24T21:55:47.958 INFO:tasks.rados.rados.0.smithi125.stdout:414: read oid 16 snap -1 2021-03-24T21:55:47.958 INFO:tasks.rados.rados.0.smithi125.stdout:414: expect (ObjNum 87 snap 11 seq_num 87) 2021-03-24T21:55:47.959 INFO:tasks.rados.rados.0.smithi125.stdout:410: finishing rollback tid 0 to smithi12556470-6 2021-03-24T21:55:48.067 INFO:tasks.rados.rados.0.smithi125.stdout:415: read oid 14 snap -1 2021-03-24T21:55:48.067 INFO:tasks.rados.rados.0.smithi125.stdout:415: expect (ObjNum 123 snap 23 seq_num 123) 2021-03-24T21:55:48.067 INFO:tasks.rados.rados.0.smithi125.stdout:416: copy_from oid 8 from oid 30 current snap is 37 2021-03-24T21:55:48.068 INFO:tasks.rados.rados.0.smithi125.stdout:410: finishing rollback tid 1 to smithi12556470-6 2021-03-24T21:55:48.068 INFO:tasks.rados.rados.0.smithi125.stdout:410: finishing rollback tid 2 to smithi12556470-6 2021-03-24T21:55:48.068 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 6 v 138 (ObjNum 126 snap 23 seq_num 126) dirty exists 2021-03-24T21:55:48.068 INFO:tasks.rados.rados.0.smithi125.stdout:411: finishing write tid 1 to smithi12556470-5 2021-03-24T21:55:48.068 INFO:tasks.rados.rados.0.smithi125.stdout:410: done (6 left) 2021-03-24T21:55:48.069 INFO:tasks.rados.rados.0.smithi125.stdout:417: copy_from oid 18 from oid 20 current snap is 37 2021-03-24T21:55:48.069 INFO:tasks.rados.rados.0.smithi125.stdout:418: snap_remove snap 30 2021-03-24T21:55:48.069 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:48.129 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:48.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:48.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:55:48.202 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-24T21:55:48.236 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:48.289 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:48.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:48.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:55:48.396 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:48.397 INFO:tasks.rados.rados.0.smithi125.stdout:418: done (7 left) 2021-03-24T21:55:48.398 INFO:tasks.rados.rados.0.smithi125.stdout:419: write (excl) oid 4 current snap is 37 2021-03-24T21:55:48.398 INFO:tasks.rados.rados.0.smithi125.stdout:419: seq_num 177 ranges {574839=795192,1907901=452059} 2021-03-24T21:55:48.409 INFO:tasks.rados.rados.0.smithi125.stdout:419: writing smithi12556470-4 from 574839 to 1370031 tid 1 2021-03-24T21:55:48.417 INFO:tasks.rados.rados.0.smithi125.stdout:419: writing smithi12556470-4 from 1907901 to 2359960 tid 2 2021-03-24T21:55:48.418 INFO:tasks.rados.rados.0.smithi125.stdout:416: finishing copy_from to smithi12556470-8 2021-03-24T21:55:48.418 INFO:tasks.rados.rados.0.smithi125.stdout:416: got expected ENOENT (src dne) 2021-03-24T21:55:48.418 INFO:tasks.rados.rados.0.smithi125.stdout:416: finishing copy_from racing read to smithi12556470-8 2021-03-24T21:55:48.418 INFO:tasks.rados.rados.0.smithi125.stdout:411: finishing write tid 2 to smithi12556470-5 2021-03-24T21:55:48.419 INFO:tasks.rados.rados.0.smithi125.stdout:411: finishing write tid 3 to smithi12556470-5 2021-03-24T21:55:48.419 INFO:tasks.rados.rados.0.smithi125.stdout:411: finishing write tid 4 to smithi12556470-5 2021-03-24T21:55:48.419 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 5 v 130 (ObjNum 176 snap 37 seq_num 176) dirty exists 2021-03-24T21:55:48.419 INFO:tasks.rados.rados.0.smithi125.stdout:411: left oid 5 (ObjNum 176 snap 37 seq_num 176) 2021-03-24T21:55:48.419 INFO:tasks.rados.rados.0.smithi125.stdout:415: expect (ObjNum 123 snap 23 seq_num 123) 2021-03-24T21:55:48.443 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:48 smithi125 conmon[19936]: cluster 2021-03-24T21:55:46.734097+0000 2021-03-24T21:55:48.444 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:48 smithi125 conmon[19936]: mgr.y (mgr.14144) 348 : cluster [DBG] pgmap v428: 148 pgs: 148 active+clean; 166 MiB data, 453 MiB used, 536 GiB / 536 GiB avail; 14 MiB/s rd, 4.5 MiB/s wr, 26 op/s 2021-03-24T21:55:48.444 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:48 smithi125 conmon[24927]: cluster 2021-03-24T21:55:46.734097+0000 mgr.y (mgr.14144) 348 : cluster [DBG] pgmap v428: 148 pgs: 148 active+clean; 166 MiB data, 453 MiB used, 536 GiB / 536 GiB avail; 14 MiB/s rd, 4.5 MiB/s wr, 26 op/s 2021-03-24T21:55:48.447 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:48.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:48.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:55:48.535 INFO:tasks.rados.rados.0.smithi125.stdout:412: expect (ObjNum 83 snap 10 seq_num 83) 2021-03-24T21:55:48.553 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:48.612 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:48.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:48.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:55:48.718 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:48.732 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:48 smithi142 conmon[20125]: cluster 2021-03-24T21:55:48.733 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:48 smithi142 conmon[20125]: 2021-03-24T21:55:48.733 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:48 smithi142 conmon[20125]: 2021 2021-03-24T21:55:48.733 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:48 smithi142 conmon[20125]: - 2021-03-24T21:55:48.734 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:48 smithi142 conmon[20125]: 03 2021-03-24T21:55:48.734 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:48 smithi142 conmon[20125]: - 2021-03-24T21:55:48.734 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:48 smithi142 conmon[20125]: 24 2021-03-24T21:55:48.734 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:48 smithi142 conmon[20125]: T 2021-03-24T21:55:48.735 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:48 smithi142 conmon[20125]: 21 2021-03-24T21:55:48.735 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:48 smithi142 conmon[20125]: : 2021-03-24T21:55:48.735 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:48 smithi142 conmon[20125]: 55 2021-03-24T21:55:48.735 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:48 smithi142 conmon[20125]: : 2021-03-24T21:55:48.736 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:48 smithi142 conmon[20125]: 46 2021-03-24T21:55:48.736 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:48 smithi142 conmon[20125]: . 2021-03-24T21:55:48.736 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:48 smithi142 conmon[20125]: 734097 2021-03-24T21:55:48.736 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:48 smithi142 conmon[20125]: +0000 2021-03-24T21:55:48.737 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:48 smithi142 conmon[20125]: 2021-03-24T21:55:48.737 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:48 smithi142 conmon[20125]: mgr.y 2021-03-24T21:55:48.737 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:48 smithi142 conmon[20125]: ( 2021-03-24T21:55:48.737 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:48 smithi142 conmon[20125]: mgr 2021-03-24T21:55:48.737 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:48 smithi142 conmon[20125]: . 2021-03-24T21:55:48.738 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:48 smithi142 conmon[20125]: 14144 2021-03-24T21:55:48.738 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:48 smithi142 conmon[20125]: ) 2021-03-24T21:55:48.738 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:48 smithi142 conmon[20125]: 348 2021-03-24T21:55:48.738 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:48 smithi142 conmon[20125]: : 2021-03-24T21:55:48.739 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:48 smithi142 conmon[20125]: cluster 2021-03-24T21:55:48.739 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:48 smithi142 conmon[20125]: 2021-03-24T21:55:48.739 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:48 smithi142 conmon[20125]: [DBG] pgmap v428: 148 pgs: 148 active+clean; 166 MiB data, 453 MiB used, 536 GiB / 536 GiB avail; 14 MiB/s rd, 4.5 MiB/s wr, 26 op/s 2021-03-24T21:55:48.757 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:55:48.773 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:48.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:48.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:55:48.794 INFO:tasks.rados.rados.0.smithi125.stdout:413: expect (ObjNum 62 snap 4 seq_num 62) 2021-03-24T21:55:48.878 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:48.900 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.5 dump_blocked_ops 2021-03-24T21:55:48.926 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:48.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:48.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:55:48.939 INFO:tasks.rados.rados.0.smithi125.stdout:417: finishing copy_from to smithi12556470-18 2021-03-24T21:55:48.939 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 18 v 100 (ObjNum 159 snap 35 seq_num 159) dirty exists 2021-03-24T21:55:48.939 INFO:tasks.rados.rados.0.smithi125.stdout:417: finishing copy_from racing read to smithi12556470-18 2021-03-24T21:55:48.939 INFO:tasks.rados.rados.0.smithi125.stdout:411: done (7 left) 2021-03-24T21:55:48.940 INFO:tasks.rados.rados.0.smithi125.stdout:412: done (6 left) 2021-03-24T21:55:48.940 INFO:tasks.rados.rados.0.smithi125.stdout:413: done (5 left) 2021-03-24T21:55:48.940 INFO:tasks.rados.rados.0.smithi125.stdout:415: done (4 left) 2021-03-24T21:55:48.940 INFO:tasks.rados.rados.0.smithi125.stdout:416: done (3 left) 2021-03-24T21:55:48.941 INFO:tasks.rados.rados.0.smithi125.stdout:417: done (2 left) 2021-03-24T21:55:48.941 INFO:tasks.rados.rados.0.smithi125.stdout:420: snap_create 2021-03-24T21:55:48.941 INFO:tasks.rados.rados.0.smithi125.stdout:414: expect (ObjNum 87 snap 11 seq_num 87) 2021-03-24T21:55:49.032 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:49.090 INFO:tasks.rados.rados.0.smithi125.stdout:419: finishing write tid 1 to smithi12556470-4 2021-03-24T21:55:49.091 INFO:tasks.rados.rados.0.smithi125.stdout:419: finishing write tid 2 to smithi12556470-4 2021-03-24T21:55:49.091 INFO:tasks.rados.rados.0.smithi125.stdout:419: finishing write tid 3 to smithi12556470-4 2021-03-24T21:55:49.091 INFO:tasks.rados.rados.0.smithi125.stdout:419: finishing write tid 4 to smithi12556470-4 2021-03-24T21:55:49.092 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 4 v 138 (ObjNum 177 snap 37 seq_num 177) dirty exists 2021-03-24T21:55:49.092 INFO:tasks.rados.rados.0.smithi125.stdout:419: left oid 4 (ObjNum 177 snap 37 seq_num 177) 2021-03-24T21:55:49.093 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:49.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:49.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:55:49.200 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:49.221 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T21:55:49.268 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:49.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:49.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:55:49.375 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:49.416 INFO:tasks.rados.rados.0.smithi125.stdout:414: done (2 left) 2021-03-24T21:55:49.416 INFO:tasks.rados.rados.0.smithi125.stdout:419: done (1 left) 2021-03-24T21:55:49.417 INFO:tasks.rados.rados.0.smithi125.stdout:420: done (0 left) 2021-03-24T21:55:49.417 INFO:tasks.rados.rados.0.smithi125.stdout:421: write oid 13 current snap is 38 2021-03-24T21:55:49.417 INFO:tasks.rados.rados.0.smithi125.stdout:421: seq_num 178 ranges {591646=470441,1566383=595717,2888822=102824} 2021-03-24T21:55:49.417 INFO:teuthology.orchestra.run.smithi125.stderr:nodeep-scrub is set 2021-03-24T21:55:49.419 INFO:tasks.rados.rados.0.smithi125.stdout:421: writing smithi12556470-13 from 591646 to 1062087 tid 1 2021-03-24T21:55:49.425 INFO:tasks.rados.rados.0.smithi125.stdout:421: writing smithi12556470-13 from 1566383 to 2162100 tid 2 2021-03-24T21:55:49.431 INFO:tasks.rados.rados.0.smithi125.stdout:421: writing smithi12556470-13 from 2888822 to 2991646 tid 3 2021-03-24T21:55:49.431 INFO:tasks.rados.rados.0.smithi125.stdout:422: snap_remove snap 37 2021-03-24T21:55:49.435 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:49.437 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:49 smithi125 conmon[19936]: cluster 2021-03-24T21:55:48.395191+0000 mon.a 2021-03-24T21:55:49.438 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:49 smithi125 conmon[19936]: (mon.0) 760 : cluster [DBG] osdmap e179: 8 total, 8 up, 6 in 2021-03-24T21:55:49.438 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:49 smithi125 conmon[19936]: audit 2021-03-24T21:55 2021-03-24T21:55:49.439 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:49 smithi125 conmon[19936]: :49.359132+0000 mon.a (mon.0) 761 : audit [INF] from='client.? v1:172.21.15.125:0/1492582326' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:55:49.439 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:49 smithi125 conmon[24927]: cluster 2021-03-24T21:55: 2021-03-24T21:55:49.439 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:49 smithi125 conmon[24927]: 48.395191+0000 mon.a (mon.0) 760 : cluster [DBG] osdmap e179: 8 total, 8 up, 6 in 2021-03-24T21:55:49.440 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:49 smithi125 conmon[24927]: audit 2021-03-24T21:55: 2021-03-24T21:55:49.440 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:49 smithi125 conmon[24927]: 49.359132+0000 mon.a (mon.0) 761 : audit [INF] 2021-03-24T21:55:49.440 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:49 smithi125 conmon[24927]: from='client.? v1:172.21.15.125:0/1492582326' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:55:49.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:49.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:55:49.542 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:49.600 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:49.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:49.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:55:49.657 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:49 smithi142 conmon[20125]: cluster 2021-03-24T21:55 2021-03-24T21:55:49.658 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:49 smithi142 conmon[20125]: :48.395191+0000 mon.a (mon.0) 760 : cluster [DBG] osdmap e179: 8 total, 8 up, 6 in 2021-03-24T21:55:49.658 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:49 smithi142 conmon[20125]: audit 2021-03-24T 2021-03-24T21:55:49.659 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:49 smithi142 conmon[20125]: 21:55:49.359132+0000 mon.a (mon.0) 761 : audit [INF] from='client.? v1:172.21.15.125:0/1492582326' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:55:49.706 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:49.770 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:49.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:49.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:55:49.875 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:49.927 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:49.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:49.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:55:50.034 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:50.088 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:50.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:50.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:55:50.112 INFO:tasks.rados.rados.0.smithi125.stdout:422: done (1 left) 2021-03-24T21:55:50.112 INFO:tasks.rados.rados.0.smithi125.stdout:423: write oid 25 current snap is 38 2021-03-24T21:55:50.113 INFO:tasks.rados.rados.0.smithi125.stdout:423: seq_num 179 ranges {608453=545690,1624865=455728,2632598=375855} 2021-03-24T21:55:50.118 INFO:tasks.rados.rados.0.smithi125.stdout:423: writing smithi12556470-25 from 608453 to 1154143 tid 1 2021-03-24T21:55:50.124 INFO:tasks.rados.rados.0.smithi125.stdout:423: writing smithi12556470-25 from 1624865 to 2080593 tid 2 2021-03-24T21:55:50.130 INFO:tasks.rados.rados.0.smithi125.stdout:423: writing smithi12556470-25 from 2632598 to 3008453 tid 3 2021-03-24T21:55:50.131 INFO:tasks.rados.rados.0.smithi125.stdout:424: snap_remove snap 34 2021-03-24T21:55:50.195 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:50.242 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:50.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:50.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:55:50.348 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:50.405 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:50.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:50.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:55:50.514 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:50.528 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:50 smithi125 conmon[19936]: cluster 2021-03-24T21:55: 2021-03-24T21:55:50.529 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:50 smithi125 conmon[19936]: 48.734592+0000 mgr.y (mgr.14144) 349 : cluster [DBG] pgmap v431: 148 pgs: 148 active+clean; 163 MiB data, 443 MiB used, 536 GiB / 536 GiB avail; 8.6 MiB/s rd, 772 KiB/s wr, 9 op/s 2021-03-24T21:55:50.529 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:50 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:55:50.529 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:50 smithi125 conmon[19936]: :55:49.399187+0000 mon.a (mon.0) 762 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:55:50.529 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:50 smithi125 conmon[19936]: 2021-03-24T21:55:50.530 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:50 smithi125 conmon[19936]: audit 2021-03 2021-03-24T21:55:50.530 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:50 smithi125 conmon[19936]: -24T21:55:49.411622+0000 mon.a (mon. 2021-03-24T21:55:50.530 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:50 smithi125 conmon[19936]: 0) 763 : audit [INF] from='client.? v1:172.21.15.125:0/1492582326' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T21:55:50.530 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:50 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:55:50.531 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:50 smithi125 conmon[19936]: 21:55:49.411685+0000 mon.a (mon.0 2021-03-24T21:55:50.531 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:50 smithi125 conmon[19936]: ) 764 : cluster [DBG] osdmap e180: 8 total, 8 up, 6 in 2021-03-24T21:55:50.531 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:50 smithi125 conmon[19936]: cluster 2021-03-24T21:55 2021-03-24T21:55:50.531 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:50 smithi125 conmon[19936]: :50.106859+0000 mon.a ( 2021-03-24T21:55:50.532 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:50 smithi125 conmon[19936]: mon.0) 765 : cluster [DBG] osdmap e181: 8 total, 8 up, 6 in 2021-03-24T21:55:50.532 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:50 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:55:50.532 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:50 smithi125 conmon[24927]: :55:48.734592+0000 mgr.y (mgr.14144) 349 : cluster [DBG] pgmap v431: 148 pgs: 148 active+clean; 163 MiB data, 443 MiB used, 536 GiB / 536 GiB avail; 8.6 MiB/s rd, 772 KiB/s wr, 9 op/s 2021-03-24T21:55:50.533 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:50 smithi125 conmon[24927]: cluster 2021-03-24T21:55:50.533 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:50 smithi125 conmon[24927]: 2021-03-24 2021-03-24T21:55:50.533 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:50 smithi125 conmon[24927]: T21:55:49.399187+0000 mon.a (mon 2021-03-24T21:55:50.533 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:50 smithi125 conmon[24927]: .0) 762 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:55:50.534 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:50 smithi125 conmon[24927]: audit 2021- 2021-03-24T21:55:50.534 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:50 smithi125 conmon[24927]: 03-24T21:55:49.411622+0000 2021-03-24T21:55:50.534 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:50 smithi125 conmon[24927]: mon.a (mon.0) 763 : audit [INF] from='client.? v1:172.21.15.125:0/1492582326' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T21:55:50.534 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:50 smithi125 conmon[24927]: cluster 2021-03-24T21:55:50.535 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:50 smithi125 conmon[24927]: 2021-03-24T21:55:49 2021-03-24T21:55:50.535 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:50 smithi125 conmon[24927]: .411685+0000 mon.a (mon.0) 764 2021-03-24T21:55:50.535 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:50 smithi125 conmon[24927]: : cluster [DBG] osdmap e180: 8 total, 8 up, 6 in 2021-03-24T21:55:50.535 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:50 smithi125 conmon[24927]: cluster 2021-03-24T21:55:50.535 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:50 smithi125 conmon[24927]: 2021-03-24T21: 2021-03-24T21:55:50.536 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:50 smithi125 conmon[24927]: 55:50.106859+0000 mon.a 2021-03-24T21:55:50.536 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:50 smithi125 conmon[24927]: (mon.0) 765 : cluster 2021-03-24T21:55:50.536 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:50 smithi125 conmon[24927]: [DBG] osdmap e181: 8 total, 8 up, 6 in 2021-03-24T21:55:50.576 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:50.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:50.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:55:50.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:50 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:55:50.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:50 smithi142 conmon[20125]: T21:55:48.734592+0000 mgr.y (mgr.14144) 349 : cluster [DBG] pgmap v431: 148 pgs: 148 active+clean; 163 MiB data, 443 MiB used, 536 GiB / 536 GiB avail; 8.6 MiB/s rd, 772 KiB/s wr, 9 op/s 2021-03-24T21:55:50.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:50 smithi142 conmon[20125]: cluster 2021-03-24T21:55:50.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:50 smithi142 conmon[20125]: 2021-03-24T21:55: 2021-03-24T21:55:50.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:50 smithi142 conmon[20125]: 49.399187+0000 mon.a (mon 2021-03-24T21:55:50.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:50 smithi142 conmon[20125]: .0) 762 : cluster [WRN] 2021-03-24T21:55:50.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:50 smithi142 conmon[20125]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:55:50.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:50 smithi142 conmon[20125]: audit 2021-03-24T21:55:50.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:50 smithi142 conmon[20125]: 2021-03-24T21:55 2021-03-24T21:55:50.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:50 smithi142 conmon[20125]: :49.411622+0000 mon.a ( 2021-03-24T21:55:50.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:50 smithi142 conmon[20125]: mon.0) 763 : audit [INF] from='client.? v1:172.21.15.125:0/1492582326' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T21:55:50.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:50 smithi142 conmon[20125]: cluster 2021-03-24T21:55:50.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:50 smithi142 conmon[20125]: 2021-03-24T21:55 2021-03-24T21:55:50.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:50 smithi142 conmon[20125]: :49.411685+0000 mon.a ( 2021-03-24T21:55:50.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:50 smithi142 conmon[20125]: mon.0) 764 : cluster [DBG] 2021-03-24T21:55:50.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:50 smithi142 conmon[20125]: osdmap e180: 8 total, 8 up, 6 in 2021-03-24T21:55:50.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:50 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:55:50.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:50 smithi142 conmon[20125]: 24T21:55:50.106859+0000 mon.a ( 2021-03-24T21:55:50.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:50 smithi142 conmon[20125]: mon.0) 765 : cluster [DBG] osdmap e181: 8 total, 8 up, 6 in 2021-03-24T21:55:50.681 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:50.739 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:50.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:50.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:55:50.846 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:50.897 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:50.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:50.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:55:51.003 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:51.069 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:51.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:51.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:51.128 INFO:tasks.rados.rados.0.smithi125.stdout:424: done (2 left) 2021-03-24T21:55:51.129 INFO:tasks.rados.rados.0.smithi125.stdout:425: snap_remove snap 31 2021-03-24T21:55:51.178 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:51.240 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:51.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:51.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:55:51.319 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2021-03-24T21:55:51.320 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2021-03-24T21:55:51.320 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2021-03-24T21:55:51.320 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2021-03-24T21:55:51.320 INFO:teuthology.orchestra.run.smithi142.stdout:} 2021-03-24T21:55:51.325 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T21:55:51.348 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:51.349 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T21:55:51.376 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T21:55:51.398 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T21:55:51.403 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:51.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:51.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:55:51.421 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T21:55:51.443 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T21:55:51.467 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T21:55:51.490 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T21:55:51.492 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-24T21:55:51.492 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-24T21:55:51.492 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-24T21:55:51.492 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-24T21:55:51.492 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-24T21:55:51.493 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-24T21:55:51.493 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-24T21:55:51.493 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-24T21:55:51.509 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:51.558 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:51.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:51.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:51.665 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:51.725 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:51.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:51.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:55:51.831 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:51.887 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:51.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:51.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:55:51.994 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:52.049 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:52.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:52.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:55:52.131 INFO:tasks.rados.rados.0.smithi125.stdout:425: done (2 left) 2021-03-24T21:55:52.131 INFO:tasks.rados.rados.0.smithi125.stdout:426: snap_create 2021-03-24T21:55:52.132 INFO:tasks.rados.rados.0.smithi125.stdout:421: finishing write tid 1 to smithi12556470-13 2021-03-24T21:55:52.132 INFO:tasks.rados.rados.0.smithi125.stdout:421: finishing write tid 2 to smithi12556470-13 2021-03-24T21:55:52.132 INFO:tasks.rados.rados.0.smithi125.stdout:421: finishing write tid 3 to smithi12556470-13 2021-03-24T21:55:52.132 INFO:tasks.rados.rados.0.smithi125.stdout:421: finishing write tid 4 to smithi12556470-13 2021-03-24T21:55:52.132 INFO:tasks.rados.rados.0.smithi125.stdout:421: finishing write tid 5 to smithi12556470-13 2021-03-24T21:55:52.133 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 13 v 145 (ObjNum 178 snap 38 seq_num 178) dirty exists 2021-03-24T21:55:52.133 INFO:tasks.rados.rados.0.smithi125.stdout:421: left oid 13 (ObjNum 178 snap 38 seq_num 178) 2021-03-24T21:55:52.133 INFO:tasks.rados.rados.0.smithi125.stdout:423: finishing write tid 1 to smithi12556470-25 2021-03-24T21:55:52.134 INFO:tasks.rados.rados.0.smithi125.stdout:423: finishing write tid 2 to smithi12556470-25 2021-03-24T21:55:52.134 INFO:tasks.rados.rados.0.smithi125.stdout:423: finishing write tid 3 to smithi12556470-25 2021-03-24T21:55:52.134 INFO:tasks.rados.rados.0.smithi125.stdout:423: finishing write tid 4 to smithi12556470-25 2021-03-24T21:55:52.134 INFO:tasks.rados.rados.0.smithi125.stdout:423: finishing write tid 5 to smithi12556470-25 2021-03-24T21:55:52.135 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 25 v 116 (ObjNum 179 snap 38 seq_num 179) dirty exists 2021-03-24T21:55:52.135 INFO:tasks.rados.rados.0.smithi125.stdout:423: left oid 25 (ObjNum 179 snap 38 seq_num 179) 2021-03-24T21:55:52.156 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:52.206 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:52.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:52.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:55:52.272 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:52 smithi142 conmon[20125]: cluster 2021-03-24T21:55:50.735189+0000 mgr.y (mgr.14144) 350 2021-03-24T21:55:52.272 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:52 smithi142 conmon[20125]: : cluster [DBG] pgmap v434: 148 pgs: 148 active+clean; 139 MiB data, 404 MiB used, 536 GiB / 536 GiB avail; 28 MiB/s rd, 2.9 MiB/s wr, 33 op/s 2021-03-24T21:55:52.272 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:52 smithi142 conmon[20125]: cluster 2021-03-24T21:55:51.127439+0000 mon.a (mon. 2021-03-24T21:55:52.273 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:52 smithi142 conmon[20125]: 0) 766 : cluster [DBG] osdmap e182: 8 total, 8 up, 6 in 2021-03-24T21:55:52.273 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:52 smithi142 conmon[20125]: audit 2021-03-24T21:55:52.273 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:52 smithi142 conmon[20125]: 2021-03-24T21:55:51.297762+0000 mon.c (mon 2021-03-24T21:55:52.273 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:52 smithi142 conmon[20125]: .2) 32 : audit [DBG] from='client.? v1:172.21.15.125:0/1025658892' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:55:52.274 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:52 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:55:52.274 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:52 smithi142 conmon[20125]: -24T21:55:51.835433+0000 osd.2 2021-03-24T21:55:52.274 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:52 smithi142 conmon[20125]: (osd.2) 33 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:55:52.274 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:52 smithi142 conmon[20125]: 2021-03-24T21:55:52.275 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:52 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:55:52.275 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:52 smithi142 conmon[20125]: -03-24T21:55:51.835767+0000 osd.2 ( 2021-03-24T21:55:52.275 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:52 smithi142 conmon[20125]: osd.2) 34 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:55:52.303 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.5 dump_historic_ops 2021-03-24T21:55:52.311 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:52.361 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:52.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:52.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:55:52.410 INFO:journalctl@ceph.osd.0.smithi125.stdout:Mar 24 21:55:52 smithi125 conmon[28026]: debug 2021-03-24T21:55:52.138+0000 7f61b647b700 -1 osd.0 pg_epoch: 183 pg[7.7( v 182'157 (0'0,182'157] local-lis/les=108/109 n=7 ec=99/99 lis/c=108/108 les/c/f=109/109/0 sis=108) [0,3,7] r=0 lpr=108 crt=182'157 lcod 182'155 mlcod 182'155 active+clean+snaptrim trimq=[3e~1] ps=[2e~1,3d~1,42~2,46~1,4a~1]] removing snap head 2021-03-24T21:55:52.411 INFO:journalctl@ceph.osd.0.smithi125.stdout:Mar 24 21:55:52 smithi125 conmon[28026]: debug 2021-03-24T21:55:52.142+0000 7f61b9481700 -1 osd.0 pg_epoch: 183 pg[7.9( v 183'146 (0'0,183'146] local-lis/les=108/109 n=5 ec=99/99 lis/c=108/108 les/c/f=109/109/0 sis=108) [0,4,7] r=0 lpr=108 crt=183'146 lcod 183'144 mlcod 183'144 active+clean+snaptrim trimq=[3e~1] ps=[2e~1,3d~1,42~2,46~1,4a~1]] removing snap head 2021-03-24T21:55:52.411 INFO:journalctl@ceph.osd.0.smithi125.stdout:Mar 24 21:55:52 smithi125 conmon[28026]: debug 2021-03-24T21:55:52.142+0000 7f61bb485700 -1 osd.0 pg_epoch: 183 pg[7.d( v 182'136 (0'0,182'136] local-lis/les=108/109 n=6 ec=99/99 lis/c=108/108 les/c/f=109/109/0 sis=108) [0,3,7] r=0 lpr=108 crt=182'136 lcod 179'134 mlcod 179'134 active+clean+snaptrim trimq=[3e~1] ps=[2e~1,3d~1,42~2,46~1,4a~1]] removing snap head 2021-03-24T21:55:52.412 INFO:journalctl@ceph.osd.0.smithi125.stdout:Mar 24 21:55:52 smithi125 conmon[28026]: debug 2021-03-24T21:55:52.146+0000 7f61bac84700 -1 osd.0 pg_epoch: 183 pg[7.6( v 182'104 (0'0,182'104] local-lis/les=99/100 n=3 ec=99/99 lis/c=99/99 les/c/f=100/100/0 sis=99) [0,7,3] r=0 lpr=99 crt=182'104 lcod 179'102 mlcod 179'102 active+clean+snaptrim trimq=[3e~1] ps=[2e~1,3d~1,42~2,46~1,4a~1]] removing snap head 2021-03-24T21:55:52.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:52 smithi125 conmon[24927]: cluster 2021-03-24T21:55 2021-03-24T21:55:52.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:52 smithi125 conmon[24927]: :50.735189+0000 mgr.y (mgr.14144) 350 : cluster [DBG] pgmap v434: 148 pgs: 148 active+clean; 139 MiB data, 404 MiB used, 536 GiB / 536 GiB avail; 28 MiB/s rd, 2.9 MiB/s wr, 33 op/s 2021-03-24T21:55:52.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:52 smithi125 conmon[24927]: cluster 2021-03-24T21:55:51. 2021-03-24T21:55:52.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:52 smithi125 conmon[24927]: 127439+0000 mon.a (mon.0) 766 : cluster 2021-03-24T21:55:52.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:52 smithi125 conmon[24927]: [DBG] osdmap e182: 8 total, 8 up, 6 in 2021-03-24T21:55:52.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:52 smithi125 conmon[24927]: audit 2021-03-24T21:55: 2021-03-24T21:55:52.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:52 smithi125 conmon[24927]: 51.297762+0000 mon.c (mon.2) 32 2021-03-24T21:55:52.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:52 smithi125 conmon[24927]: : audit [DBG] from='client.? v1:172.21.15.125:0/1025658892' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:55:52.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:52 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:55:52.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:52 smithi125 conmon[24927]: :55:51.835433+0000 osd.2 (osd. 2021-03-24T21:55:52.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:52 smithi125 conmon[24927]: 2) 33 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:55:52.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:52 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:55:52.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:52 smithi125 conmon[24927]: -24T21:55:51.835767 2021-03-24T21:55:52.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:52 smithi125 conmon[24927]: +0000 osd.2 (osd.2) 34 2021-03-24T21:55:52.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:52 smithi125 conmon[24927]: : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:55:52.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:52 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:55:52.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:52 smithi125 conmon[19936]: :55:50.735189+0000 mgr.y (mgr.14144) 350 : cluster [DBG] pgmap v434: 148 pgs: 148 active+clean; 139 MiB data, 404 MiB used, 536 GiB / 536 GiB avail; 28 MiB/s rd, 2.9 MiB/s wr, 33 op/s 2021-03-24T21:55:52.421 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:52 smithi125 conmon[19936]: cluster 2021-03-24T21:55:51.127439+0000 2021-03-24T21:55:52.421 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:52 smithi125 conmon[19936]: mon.a (mon.0) 766 : cluster [DBG] osdmap e182: 8 total, 8 up, 6 in 2021-03-24T21:55:52.422 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:52 smithi125 conmon[19936]: 2021-03-24T21:55:52.422 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:52 smithi125 conmon[19936]: audit 2021-03-24T21:55: 2021-03-24T21:55:52.423 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:52 smithi125 conmon[19936]: 51.297762+0000 mon.c (mon.2) 32 : 2021-03-24T21:55:52.423 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:52 smithi125 conmon[19936]: audit [DBG] from='client.? v1:172.21.15.125:0/1025658892' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:55:52.424 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:52 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T21:55:52.424 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:52 smithi125 conmon[19936]: T21:55:51.835433+0000 osd.2 ( 2021-03-24T21:55:52.425 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:52 smithi125 conmon[19936]: osd.2) 33 : cluster [DBG] 2021-03-24T21:55:52.425 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:52 smithi125 conmon[19936]: purged_snaps scrub starts 2021-03-24T21:55:52.425 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:52 smithi125 conmon[19936]: cluster 2021-03-24T21:55:52.426 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:52 smithi125 conmon[19936]: 2021-03-24T21:55:51 2021-03-24T21:55:52.426 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:52 smithi125 conmon[19936]: .835767+0000 osd.2 (osd.2) 34 : cluster [DBG] 2021-03-24T21:55:52.427 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:52 smithi125 conmon[19936]: purged_snaps scrub ok 2021-03-24T21:55:52.467 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:52.530 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:52.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:52.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:55:52.602 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T21:55:52.637 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:52.691 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:52.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:52.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:55:52.792 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd unset noscrub 2021-03-24T21:55:52.797 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:52.857 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:52.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:52.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:55:52.962 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:53.018 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:53.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:53.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:55:53.095 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:55:53.124 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:53.134 INFO:tasks.rados.rados.0.smithi125.stdout:421: done (2 left) 2021-03-24T21:55:53.134 INFO:tasks.rados.rados.0.smithi125.stdout:423: done (1 left) 2021-03-24T21:55:53.134 INFO:tasks.rados.rados.0.smithi125.stdout:426: done (0 left) 2021-03-24T21:55:53.135 INFO:tasks.rados.rados.0.smithi125.stdout:427: write (excl) oid 36 current snap is 39 2021-03-24T21:55:53.135 INFO:tasks.rados.rados.0.smithi125.stdout:427: seq_num 180 ranges {625260=620939,1683347=599386,2943668=81592} 2021-03-24T21:55:53.138 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:53 smithi125 conmon[19936]: cluster 2021-03-24T21:55:52.130055 2021-03-24T21:55:53.138 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:53 smithi125 conmon[19936]: +0000 mon.a (mon.0) 767 : cluster [DBG] osdmap e183: 8 total, 8 up, 6 in 2021-03-24T21:55:53.141 INFO:tasks.rados.rados.0.smithi125.stdout:427: writing smithi12556470-36 from 625260 to 1246199 tid 1 2021-03-24T21:55:53.147 INFO:tasks.rados.rados.0.smithi125.stdout:427: writing smithi12556470-36 from 1683347 to 2282733 tid 2 2021-03-24T21:55:53.152 INFO:tasks.rados.rados.0.smithi125.stdout:427: writing smithi12556470-36 from 2943668 to 3025260 tid 3 2021-03-24T21:55:53.152 INFO:tasks.rados.rados.0.smithi125.stdout:428: write (excl) oid 47 current snap is 39 2021-03-24T21:55:53.152 INFO:tasks.rados.rados.0.smithi125.stdout:427: finishing write tid 1 to smithi12556470-36 2021-03-24T21:55:53.153 INFO:tasks.rados.rados.0.smithi125.stdout:428: seq_num 181 ranges {642067=696188,2025476=459397,2971091=70976} 2021-03-24T21:55:53.160 INFO:tasks.rados.rados.0.smithi125.stdout:428: writing smithi12556470-47 from 642067 to 1338255 tid 1 2021-03-24T21:55:53.167 INFO:tasks.rados.rados.0.smithi125.stdout:428: writing smithi12556470-47 from 2025476 to 2484873 tid 2 2021-03-24T21:55:53.171 INFO:tasks.rados.rados.0.smithi125.stdout:428: writing smithi12556470-47 from 2971091 to 3042067 tid 3 2021-03-24T21:55:53.172 INFO:tasks.rados.rados.0.smithi125.stdout:429: snap_create 2021-03-24T21:55:53.172 INFO:tasks.rados.rados.0.smithi125.stdout:427: finishing write tid 2 to smithi12556470-36 2021-03-24T21:55:53.172 INFO:tasks.rados.rados.0.smithi125.stdout:427: finishing write tid 3 to smithi12556470-36 2021-03-24T21:55:53.172 INFO:tasks.rados.rados.0.smithi125.stdout:427: finishing write tid 4 to smithi12556470-36 2021-03-24T21:55:53.173 INFO:tasks.rados.rados.0.smithi125.stdout:427: finishing write tid 5 to smithi12556470-36 2021-03-24T21:55:53.173 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 36 v 123 (ObjNum 180 snap 39 seq_num 180) dirty exists 2021-03-24T21:55:53.173 INFO:tasks.rados.rados.0.smithi125.stdout:427: left oid 36 (ObjNum 180 snap 39 seq_num 180) 2021-03-24T21:55:53.173 INFO:tasks.rados.rados.0.smithi125.stdout:428: finishing write tid 1 to smithi12556470-47 2021-03-24T21:55:53.173 INFO:tasks.rados.rados.0.smithi125.stdout:428: finishing write tid 2 to smithi12556470-47 2021-03-24T21:55:53.175 INFO:tasks.rados.rados.0.smithi125.stdout:428: finishing write tid 3 to smithi12556470-47 2021-03-24T21:55:53.175 INFO:tasks.rados.rados.0.smithi125.stdout:428: finishing write tid 4 to smithi12556470-47 2021-03-24T21:55:53.175 INFO:tasks.rados.rados.0.smithi125.stdout:428: finishing write tid 5 to smithi12556470-47 2021-03-24T21:55:53.175 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 47 v 84 (ObjNum 181 snap 39 seq_num 181) dirty exists 2021-03-24T21:55:53.176 INFO:tasks.rados.rados.0.smithi125.stdout:428: left oid 47 (ObjNum 181 snap 39 seq_num 181) 2021-03-24T21:55:53.182 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:53.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:53.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:55:53.287 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:53.348 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:53.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:53.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:55:53.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:53 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:55:53.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:53 smithi142 conmon[20125]: -24T21:55:52.130055+0000 mon.a (mon.0) 2021-03-24T21:55:53.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:53 smithi142 conmon[20125]: 767 : cluster [DBG] 2021-03-24T21:55:53.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:53 smithi142 conmon[20125]: osdmap e183: 8 total, 8 up, 6 in 2021-03-24T21:55:53.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:53 smithi125 conmon[24927]: cluster 2021-03-24T21:55:52.130055+0000 2021-03-24T21:55:53.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:53 smithi125 conmon[24927]: mon.a (mon.0) 767 : cluster [DBG] osdmap e183: 8 total, 8 up, 6 in 2021-03-24T21:55:53.453 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:53.511 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:53.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:53.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:55:53.617 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:53.671 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:53.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:53.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:55:53.724 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 7] out_osds: [6, 1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-03-24T21:55:53.724 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-24T21:55:53.725 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2021-03-24T21:55:53.725 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph pg dump --format=json 2021-03-24T21:55:53.777 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:53.830 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:53.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:53.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:55:53.936 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:53.989 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:53.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:53.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:55:54.094 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:54.144 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:54.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:54.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:55:54.152 INFO:tasks.rados.rados.0.smithi125.stdout:427: done (2 left) 2021-03-24T21:55:54.153 INFO:tasks.rados.rados.0.smithi125.stdout:428: done (1 left) 2021-03-24T21:55:54.153 INFO:tasks.rados.rados.0.smithi125.stdout:429: done (0 left) 2021-03-24T21:55:54.153 INFO:tasks.rados.rados.0.smithi125.stdout:430: write oid 45 current snap is 40 2021-03-24T21:55:54.154 INFO:tasks.rados.rados.0.smithi125.stdout:430: seq_num 182 ranges {658874=771437,2083958=603055} 2021-03-24T21:55:54.161 INFO:tasks.rados.rados.0.smithi125.stdout:430: writing smithi12556470-45 from 658874 to 1430311 tid 1 2021-03-24T21:55:54.169 INFO:tasks.rados.rados.0.smithi125.stdout:430: writing smithi12556470-45 from 2083958 to 2687013 tid 2 2021-03-24T21:55:54.170 INFO:tasks.rados.rados.0.smithi125.stdout:431: read oid 2 snap -1 2021-03-24T21:55:54.170 INFO:tasks.rados.rados.0.smithi125.stdout:431: expect deleted 2021-03-24T21:55:54.170 INFO:tasks.rados.rados.0.smithi125.stdout:432: copy_from oid 5 from oid 18 current snap is 40 2021-03-24T21:55:54.170 INFO:tasks.rados.rados.0.smithi125.stdout:433: write oid 47 current snap is 40 2021-03-24T21:55:54.171 INFO:tasks.rados.rados.0.smithi125.stdout:433: seq_num 183 ranges {675681=563039,1858793=746713,3025937=49744} 2021-03-24T21:55:54.176 INFO:tasks.rados.rados.0.smithi125.stdout:433: writing smithi12556470-47 from 675681 to 1238720 tid 1 2021-03-24T21:55:54.184 INFO:tasks.rados.rados.0.smithi125.stdout:433: writing smithi12556470-47 from 1858793 to 2605506 tid 2 2021-03-24T21:55:54.188 INFO:tasks.rados.rados.0.smithi125.stdout:433: writing smithi12556470-47 from 3025937 to 3075681 tid 3 2021-03-24T21:55:54.188 INFO:tasks.rados.rados.0.smithi125.stdout:434: write (excl) oid 40 current snap is 40 2021-03-24T21:55:54.188 INFO:tasks.rados.rados.0.smithi125.stdout:434: seq_num 184 ranges {692488=638288,1917275=606724} 2021-03-24T21:55:54.195 INFO:tasks.rados.rados.0.smithi125.stdout:434: writing smithi12556470-40 from 692488 to 1330776 tid 1 2021-03-24T21:55:54.203 INFO:tasks.rados.rados.0.smithi125.stdout:434: writing smithi12556470-40 from 1917275 to 2523999 tid 2 2021-03-24T21:55:54.203 INFO:tasks.rados.rados.0.smithi125.stdout:435: copy_from oid 16 from oid 34 current snap is 40 2021-03-24T21:55:54.203 INFO:tasks.rados.rados.0.smithi125.stdout:430: finishing write tid 1 to smithi12556470-45 2021-03-24T21:55:54.203 INFO:tasks.rados.rados.0.smithi125.stdout:431: done (5 left) 2021-03-24T21:55:54.204 INFO:tasks.rados.rados.0.smithi125.stdout:436: write oid 19 current snap is 40 2021-03-24T21:55:54.204 INFO:tasks.rados.rados.0.smithi125.stdout:430: finishing write tid 2 to smithi12556470-45 2021-03-24T21:55:54.204 INFO:tasks.rados.rados.0.smithi125.stdout:430: finishing write tid 3 to smithi12556470-45 2021-03-24T21:55:54.204 INFO:tasks.rados.rados.0.smithi125.stdout:430: finishing write tid 4 to smithi12556470-45 2021-03-24T21:55:54.204 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 45 v 143 (ObjNum 182 snap 40 seq_num 182) dirty exists 2021-03-24T21:55:54.205 INFO:tasks.rados.rados.0.smithi125.stdout:430: left oid 45 (ObjNum 182 snap 40 seq_num 182) 2021-03-24T21:55:54.205 INFO:tasks.rados.rados.0.smithi125.stdout:433: finishing write tid 1 to smithi12556470-47 2021-03-24T21:55:54.205 INFO:tasks.rados.rados.0.smithi125.stdout:433: finishing write tid 2 to smithi12556470-47 2021-03-24T21:55:54.205 INFO:tasks.rados.rados.0.smithi125.stdout:432: finishing copy_from to smithi12556470-5 2021-03-24T21:55:54.205 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 5 v 100 (ObjNum 159 snap 35 seq_num 159) dirty exists 2021-03-24T21:55:54.206 INFO:tasks.rados.rados.0.smithi125.stdout:433: finishing write tid 3 to smithi12556470-47 2021-03-24T21:55:54.206 INFO:tasks.rados.rados.0.smithi125.stdout:433: finishing write tid 4 to smithi12556470-47 2021-03-24T21:55:54.206 INFO:tasks.rados.rados.0.smithi125.stdout:432: finishing copy_from racing read to smithi12556470-5 2021-03-24T21:55:54.206 INFO:tasks.rados.rados.0.smithi125.stdout:433: finishing write tid 5 to smithi12556470-47 2021-03-24T21:55:54.207 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 47 v 88 (ObjNum 183 snap 40 seq_num 183) dirty exists 2021-03-24T21:55:54.207 INFO:tasks.rados.rados.0.smithi125.stdout:433: left oid 47 (ObjNum 183 snap 40 seq_num 183) 2021-03-24T21:55:54.207 INFO:tasks.rados.rados.0.smithi125.stdout:434: finishing write tid 1 to smithi12556470-40 2021-03-24T21:55:54.207 INFO:tasks.rados.rados.0.smithi125.stdout:436: seq_num 185 ranges {709295=713537,1859404=750382} 2021-03-24T21:55:54.210 INFO:tasks.rados.rados.0.smithi125.stdout:436: writing smithi12556470-19 from 709295 to 1422832 tid 1 2021-03-24T21:55:54.218 INFO:tasks.rados.rados.0.smithi125.stdout:436: writing smithi12556470-19 from 1859404 to 2609786 tid 2 2021-03-24T21:55:54.218 INFO:tasks.rados.rados.0.smithi125.stdout:430: done (5 left) 2021-03-24T21:55:54.219 INFO:tasks.rados.rados.0.smithi125.stdout:432: done (4 left) 2021-03-24T21:55:54.219 INFO:tasks.rados.rados.0.smithi125.stdout:433: done (3 left) 2021-03-24T21:55:54.219 INFO:tasks.rados.rados.0.smithi125.stdout:437: write oid 38 current snap is 40 2021-03-24T21:55:54.219 INFO:tasks.rados.rados.0.smithi125.stdout:434: finishing write tid 2 to smithi12556470-40 2021-03-24T21:55:54.220 INFO:tasks.rados.rados.0.smithi125.stdout:437: seq_num 186 ranges {726102=788786,1917886=610393,3108206=17896} 2021-03-24T21:55:54.226 INFO:tasks.rados.rados.0.smithi125.stdout:437: writing smithi12556470-38 from 726102 to 1514888 tid 1 2021-03-24T21:55:54.232 INFO:tasks.rados.rados.0.smithi125.stdout:437: writing smithi12556470-38 from 1917886 to 2528279 tid 2 2021-03-24T21:55:54.236 INFO:tasks.rados.rados.0.smithi125.stdout:437: writing smithi12556470-38 from 3108206 to 3126102 tid 3 2021-03-24T21:55:54.236 INFO:tasks.rados.rados.0.smithi125.stdout:438: copy_from oid 4 from oid 29 current snap is 40 2021-03-24T21:55:54.236 INFO:tasks.rados.rados.0.smithi125.stdout:434: finishing write tid 3 to smithi12556470-40 2021-03-24T21:55:54.236 INFO:tasks.rados.rados.0.smithi125.stdout:434: finishing write tid 4 to smithi12556470-40 2021-03-24T21:55:54.237 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 40 v 138 (ObjNum 184 snap 40 seq_num 184) dirty exists 2021-03-24T21:55:54.237 INFO:tasks.rados.rados.0.smithi125.stdout:434: left oid 40 (ObjNum 184 snap 40 seq_num 184) 2021-03-24T21:55:54.237 INFO:tasks.rados.rados.0.smithi125.stdout:435: finishing copy_from to smithi12556470-16 2021-03-24T21:55:54.237 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 16 v 132 (ObjNum 167 snap 36 seq_num 167) dirty exists 2021-03-24T21:55:54.238 INFO:tasks.rados.rados.0.smithi125.stdout:435: finishing copy_from racing read to smithi12556470-16 2021-03-24T21:55:54.238 INFO:tasks.rados.rados.0.smithi125.stdout:436: finishing write tid 1 to smithi12556470-19 2021-03-24T21:55:54.238 INFO:tasks.rados.rados.0.smithi125.stdout:434: done (4 left) 2021-03-24T21:55:54.238 INFO:tasks.rados.rados.0.smithi125.stdout:435: done (3 left) 2021-03-24T21:55:54.239 INFO:tasks.rados.rados.0.smithi125.stdout:439: snap_remove snap 39 2021-03-24T21:55:54.250 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:54.259 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:54 smithi125 conmon[24927]: cluster 2021-03-24T21:55:52.735838+0000 mgr.y (mgr.14144) 351 : cluster [DBG] pgmap v437: 148 pgs: 148 active+clean; 142 MiB data, 411 MiB used, 536 GiB / 536 GiB avail; 20 MiB/s rd, 2.1 MiB/s wr, 25 op/s 2021-03-24T21:55:54.260 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:54 smithi125 conmon[24927]: cluster 2021-03-24T21:55:53.133043+0000 mon.a (mon.0) 768 : cluster [DBG] osdmap e184: 8 total, 8 up, 6 in 2021-03-24T21:55:54.260 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:54 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:55:54.261 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:54 smithi125 conmon[19936]: 55:52.735838+0000 mgr.y (mgr.14144) 351 : cluster [DBG] pgmap v437: 148 pgs: 148 active+clean; 142 MiB data, 411 MiB used, 536 GiB / 536 GiB avail; 20 MiB/s rd, 2.1 MiB/s wr, 25 op/s 2021-03-24T21:55:54.261 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:54 smithi125 conmon[19936]: cluster 2021-03-24T21:55:54.262 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:54 smithi125 conmon[19936]: 2021-03-24T21:55:53.133043+0000 2021-03-24T21:55:54.262 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:54 smithi125 conmon[19936]: mon.a (mon.0) 768 : cluster 2021-03-24T21:55:54.262 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:54 smithi125 conmon[19936]: [DBG] osdmap e184: 8 total, 8 up, 6 in 2021-03-24T21:55:54.302 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:54.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:54.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:55:54.350 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-24T21:55:54.409 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:54.423 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:54 smithi142 conmon[20125]: cluster 2021-03-24T21:55: 2021-03-24T21:55:54.424 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:54 smithi142 conmon[20125]: 52.735838+0000 mgr.y (mgr.14144) 351 : cluster [DBG] pgmap v437: 148 pgs: 148 active+clean; 142 MiB data, 411 MiB used, 536 GiB / 536 GiB avail; 20 MiB/s rd, 2.1 MiB/s wr, 25 op/s 2021-03-24T21:55:54.424 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:54 smithi142 conmon[20125]: cluster 2021-03-24T21:55:54.425 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:54 smithi142 conmon[20125]: 2021-03-24T21:55:53.133043+0000 mon.a (mon.0) 768 : cluster 2021-03-24T21:55:54.425 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:54 smithi142 conmon[20125]: [DBG] osdmap e184: 8 total, 8 up, 6 in 2021-03-24T21:55:54.456 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:54.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:54.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:54.509 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:55:54.562 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:54.602 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:54.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:54.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:55:54.707 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:54.723 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2021-03-24T21:55:54.724 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T21:55:54.751 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:54.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:54.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:55:54.857 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:54.915 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:54.921 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:55:54.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:54.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:55:55.024 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:55.074 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:55.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:55.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:55.114 INFO:tasks.rados.rados.0.smithi125.stdout:439: done (3 left) 2021-03-24T21:55:55.115 INFO:tasks.rados.rados.0.smithi125.stdout:440: snap_create 2021-03-24T21:55:55.115 INFO:tasks.rados.rados.0.smithi125.stdout:436: finishing write tid 2 to smithi12556470-19 2021-03-24T21:55:55.115 INFO:tasks.rados.rados.0.smithi125.stdout:436: finishing write tid 3 to smithi12556470-19 2021-03-24T21:55:55.116 INFO:tasks.rados.rados.0.smithi125.stdout:436: finishing write tid 4 to smithi12556470-19 2021-03-24T21:55:55.116 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 19 v 150 (ObjNum 185 snap 40 seq_num 185) dirty exists 2021-03-24T21:55:55.117 INFO:tasks.rados.rados.0.smithi125.stdout:436: left oid 19 (ObjNum 185 snap 40 seq_num 185) 2021-03-24T21:55:55.117 INFO:tasks.rados.rados.0.smithi125.stdout:437: finishing write tid 1 to smithi12556470-38 2021-03-24T21:55:55.117 INFO:tasks.rados.rados.0.smithi125.stdout:437: finishing write tid 2 to smithi12556470-38 2021-03-24T21:55:55.118 INFO:tasks.rados.rados.0.smithi125.stdout:437: finishing write tid 3 to smithi12556470-38 2021-03-24T21:55:55.118 INFO:tasks.rados.rados.0.smithi125.stdout:437: finishing write tid 4 to smithi12556470-38 2021-03-24T21:55:55.119 INFO:tasks.rados.rados.0.smithi125.stdout:437: finishing write tid 5 to smithi12556470-38 2021-03-24T21:55:55.119 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 38 v 25 (ObjNum 186 snap 40 seq_num 186) dirty exists 2021-03-24T21:55:55.119 INFO:tasks.rados.rados.0.smithi125.stdout:437: left oid 38 (ObjNum 186 snap 40 seq_num 186) 2021-03-24T21:55:55.120 INFO:tasks.rados.rados.0.smithi125.stdout:438: finishing copy_from to smithi12556470-4 2021-03-24T21:55:55.120 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 4 v 49 (ObjNum 101 snap 20 seq_num 101) dirty exists 2021-03-24T21:55:55.121 INFO:tasks.rados.rados.0.smithi125.stdout:438: finishing copy_from racing read to smithi12556470-4 2021-03-24T21:55:55.187 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:55.243 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:55.247 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:55 smithi142 conmon[20125]: cluster 2021-03-24T21:55:54.150977 2021-03-24T21:55:55.248 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:55 smithi142 conmon[20125]: +0000 mon.a (mon.0) 769 : cluster [DBG] osdmap e185: 8 total, 8 up, 6 in 2021-03-24T21:55:55.248 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:55 smithi142 conmon[20125]: cluster 2021-03-24T21:55:54.813745+0000 osd.5 (osd.5) 51 : cluster [DBG] 2021-03-24T21:55:55.248 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:55 smithi142 conmon[20125]: purged_snaps scrub starts 2021-03-24T21:55:55.248 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:55 smithi142 conmon[20125]: cluster 2021-03-24T21 2021-03-24T21:55:55.249 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:55 smithi142 conmon[20125]: :55:54.813975+0000 osd.5 (osd.5) 52 : cluster 2021-03-24T21:55:55.249 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:55 smithi142 conmon[20125]: [DBG] purged_snaps scrub ok 2021-03-24T21:55:55.249 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:55 smithi142 conmon[20125]: cluster 2021-03-24T21:55:55.250 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:55 smithi142 conmon[20125]: 2021-03-24T21:55:55.112055+0000 mon.a ( 2021-03-24T21:55:55.250 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:55 smithi142 conmon[20125]: mon.0) 770 : cluster [DBG] osdmap e186: 8 total, 8 up, 6 in 2021-03-24T21:55:55.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:55.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:55:55.352 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:55.388 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:55 smithi125 conmon[19936]: cluster 2021-03-24T21:55 2021-03-24T21:55:55.388 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:55 smithi125 conmon[19936]: :54.150977+0000 mon.a (mon.0) 769 : cluster [DBG] osdmap e185: 8 total, 8 up, 6 in 2021-03-24T21:55:55.388 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:55 smithi125 conmon[19936]: cluster 2021-03-24T21:55:55.388 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:55 smithi125 conmon[19936]: 2021-03-24T21:55:54.813745 2021-03-24T21:55:55.389 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:55 smithi125 conmon[19936]: +0000 osd.5 (osd.5) 51 : cluster [DBG] 2021-03-24T21:55:55.389 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:55 smithi125 conmon[19936]: purged_snaps scrub starts 2021-03-24T21:55:55.389 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:55 smithi125 conmon[19936]: cluster 2021-03-24T21:55:55.390 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:55 smithi125 conmon[19936]: 2021-03-24T21:55:54.813975+0000 2021-03-24T21:55:55.390 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:55 smithi125 conmon[19936]: osd.5 (osd.5) 52 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:55:55.390 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:55 smithi125 conmon[19936]: cluster 2021-03-24T21:55 2021-03-24T21:55:55.390 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:55 smithi125 conmon[19936]: :55.112055+0000 mon.a (mon.0) 770 : cluster [DBG] osdmap e186: 8 total, 8 up, 6 in 2021-03-24T21:55:55.391 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:55 smithi125 conmon[24927]: cluster 2021-03-24T21:55 2021-03-24T21:55:55.391 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:55 smithi125 conmon[24927]: :54.150977+0000 mon.a (mon.0) 769 : cluster [DBG] osdmap e185: 8 total, 8 up, 6 in 2021-03-24T21:55:55.391 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:55 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:55:55.391 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:55 smithi125 conmon[24927]: 21:55:54.813745+0000 osd.5 (osd.5) 51 2021-03-24T21:55:55.392 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:55 smithi125 conmon[24927]: : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:55:55.392 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:55 smithi125 conmon[24927]: cluster 2021-03-24T21:55:54. 2021-03-24T21:55:55.392 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:55 smithi125 conmon[24927]: 813975+0000 osd.5 (osd.5) 52 : cluster 2021-03-24T21:55:55.392 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:55 smithi125 conmon[24927]: [DBG] purged_snaps scrub ok 2021-03-24T21:55:55.393 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:55 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:55:55.393 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:55 smithi125 conmon[24927]: T21:55:55.112055+0000 mon.a 2021-03-24T21:55:55.393 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:55 smithi125 conmon[24927]: (mon.0) 770 : cluster [DBG] osdmap e186: 8 total, 8 up, 6 in 2021-03-24T21:55:55.393 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:55 smithi125 conmon[24927]: 2021-03-24T21:55:55.412 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:55.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:55.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:55:55.517 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:55.578 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:55.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:55.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:55:55.682 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:55.736 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.6 dump_ops_in_flight 2021-03-24T21:55:55.744 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:55.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:55.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:55:55.850 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:55.907 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:55.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:55.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:55:56.014 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:56.020 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T21:55:56.063 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:56.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:56.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:55:56.115 INFO:tasks.rados.rados.0.smithi125.stdout:436: done (3 left) 2021-03-24T21:55:56.115 INFO:tasks.rados.rados.0.smithi125.stdout:437: done (2 left) 2021-03-24T21:55:56.116 INFO:tasks.rados.rados.0.smithi125.stdout:438: done (1 left) 2021-03-24T21:55:56.116 INFO:tasks.rados.rados.0.smithi125.stdout:440: done (0 left) 2021-03-24T21:55:56.116 INFO:tasks.rados.rados.0.smithi125.stdout:441: read oid 20 snap -1 2021-03-24T21:55:56.116 INFO:tasks.rados.rados.0.smithi125.stdout:441: expect (ObjNum 159 snap 35 seq_num 159) 2021-03-24T21:55:56.116 INFO:tasks.rados.rados.0.smithi125.stdout:442: rollback oid 43 current snap is 41 2021-03-24T21:55:56.117 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 43 to 40 2021-03-24T21:55:56.117 INFO:tasks.rados.rados.0.smithi125.stdout:443: delete oid 42 current snap is 41 2021-03-24T21:55:56.120 INFO:teuthology.orchestra.run.smithi125.stderr:noscrub is unset 2021-03-24T21:55:56.121 INFO:tasks.rados.rados.0.smithi125.stdout:443: done (2 left) 2021-03-24T21:55:56.122 INFO:tasks.rados.rados.0.smithi125.stdout:444: snap_remove snap 38 2021-03-24T21:55:56.168 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:56.222 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:56.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:56.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:56.255 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:56 smithi125 conmon[24927]: cluster 2021-03-24T21:55 2021-03-24T21:55:56.255 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:56 smithi125 conmon[24927]: :54.736888+0000 mgr.y (mgr.14144) 352 : cluster [DBG] pgmap v440: 148 pgs: 148 active+clean; 114 MiB data, 352 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 1.8 MiB/s wr, 11 op/s 2021-03-24T21:55:56.255 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:56 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T21:55:56.256 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:56 smithi125 conmon[24927]: 24T21:55:55.176547+0000 2021-03-24T21:55:56.256 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:56 smithi125 conmon[24927]: mon.c (mon.2) 33 : audit [INF] 2021-03-24T21:55:56.256 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:56 smithi125 conmon[24927]: from='client.? v1:172.21.15.125:0/1949458087' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T21:55:56.256 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:56 smithi125 conmon[24927]: audit 2021-03-24T21: 2021-03-24T21:55:56.256 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:56 smithi125 conmon[24927]: 55:55.177114+0000 mon.a (mon.0 2021-03-24T21:55:56.257 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:56 smithi125 conmon[24927]: ) 771 : audit [INF] from='client.34820 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T21:55:56.257 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:56 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:55:56.257 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:56 smithi125 conmon[24927]: -03-24T21:55:56.104668 2021-03-24T21:55:56.257 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:56 smithi125 conmon[24927]: +0000 mon.a (mon.0) 772 : cluster 2021-03-24T21:55:56.258 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:56 smithi125 conmon[24927]: [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:55:56.258 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:56 smithi125 conmon[24927]: audit 2021-03-24T21:55:56. 2021-03-24T21:55:56.258 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:56 smithi125 conmon[24927]: 113273+0000 mon.a (mon.0) 773 : audit [INF] from='client.34820 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T21:55:56.258 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:56 smithi125 conmon[24927]: 2021-03-24T21:55:56.259 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:56 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:55:56.259 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:56 smithi125 conmon[24927]: 21:55:56.113452+0000 mon.a (mon.0) 774 : cluster [DBG] osdmap e187: 8 total, 8 up, 6 in 2021-03-24T21:55:56.259 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:56 smithi125 conmon[24927]: 2021-03-24T21:55:56.259 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:56 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:55:56.260 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:56 smithi125 conmon[19936]: :55:54.736888+0000 mgr.y (mgr.14144) 352 : cluster [DBG] pgmap v440: 148 pgs: 148 active+clean; 114 MiB data, 352 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 1.8 MiB/s wr, 11 op/s 2021-03-24T21:55:56.260 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:56 smithi125 conmon[19936]: audit 2021-03-24T21:55:56.260 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:56 smithi125 conmon[19936]: 2021-03-24T21:55:55.176547+0000 mon.c (mon. 2021-03-24T21:55:56.260 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:56 smithi125 conmon[19936]: 2) 33 : audit [INF] from='client.? v1:172.21.15.125:0/1949458087' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T21:55:56.260 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:56 smithi125 conmon[19936]: audit 2021 2021-03-24T21:55:56.261 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:56 smithi125 conmon[19936]: -03-24T21:55:55 2021-03-24T21:55:56.261 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:56 smithi125 conmon[19936]: .177114+0000 mon.a (mon.0) 771 : 2021-03-24T21:55:56.261 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:56 smithi125 conmon[19936]: audit [INF] from='client.34820 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T21:55:56.261 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:56 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T21:55:56.262 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:56 smithi125 conmon[19936]: T21:55:56.104668+0000 mon.a ( 2021-03-24T21:55:56.262 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:56 smithi125 conmon[19936]: mon.0) 772 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:55:56.262 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:56 smithi125 conmon[19936]: audit 2021 2021-03-24T21:55:56.262 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:56 smithi125 conmon[19936]: -03-24T21:55:56. 2021-03-24T21:55:56.262 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:56 smithi125 conmon[19936]: 113273+0000 mon.a (mon.0) 773 : audit [INF] from='client.34820 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T21:55:56.263 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:56 smithi125 conmon[19936]: cluster 2021-03-24T21:55 2021-03-24T21:55:56.263 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:56 smithi125 conmon[19936]: :56.113452+0000 mon.a (mon.0) 774 : cluster 2021-03-24T21:55:56.263 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:56 smithi125 conmon[19936]: [DBG] osdmap e187: 8 total, 8 up, 6 in 2021-03-24T21:55:56.328 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:56.331 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:56 smithi142 conmon[20125]: cluster 2021-03-24T21:55:56.332 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:56 smithi142 conmon[20125]: 2021-03-24T21:55:54.736888+0000 mgr.y (mgr.14144) 352 : cluster [DBG] pgmap v440: 148 pgs: 148 active+clean; 114 MiB data, 352 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 1.8 MiB/s wr, 11 op/s 2021-03-24T21:55:56.332 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:56 smithi142 conmon[20125]: audit 2021-03- 2021-03-24T21:55:56.332 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:56 smithi142 conmon[20125]: 24T21:55:55.176547+0000 mon.c (mon.2) 33 : audit [INF] from='client.? v1:172.21.15.125:0/1949458087' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T21:55:56.332 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:56 smithi142 conmon[20125]: audit 2021 2021-03-24T21:55:56.333 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:56 smithi142 conmon[20125]: -03-24T21:55:55.177114+0000 mon.a (mon.0) 771 : audit [INF] from='client.34820 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T21:55:56.333 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:56 smithi142 conmon[20125]: cluster 2021-03-24T21:55: 2021-03-24T21:55:56.333 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:56 smithi142 conmon[20125]: 56.104668+0000 mon.a (mon.0) 772 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:55:56.333 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:56 smithi142 conmon[20125]: audit 2021-03-24T21:55:56 2021-03-24T21:55:56.334 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:56 smithi142 conmon[20125]: .113273+0000 mon.a (mon.0) 773 : audit [INF] from='client.34820 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T21:55:56.334 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:56 smithi142 conmon[20125]: cluster 2021-03-24T21:55 2021-03-24T21:55:56.334 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:56 smithi142 conmon[20125]: :56.113452+0000 mon.a (mon.0) 774 : cluster [DBG] 2021-03-24T21:55:56.334 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:56 smithi142 conmon[20125]: osdmap e187: 8 total, 8 up, 6 in 2021-03-24T21:55:56.377 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:56.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:56.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:55:56.483 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:56.541 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:56.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:56.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:55:56.649 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:56.705 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:56.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:56.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:56.812 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:56.874 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:56.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:56.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:55:56.980 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:57.042 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:57.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:57.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:55:57.124 INFO:tasks.rados.rados.0.smithi125.stdout:444: done (2 left) 2021-03-24T21:55:57.124 INFO:tasks.rados.rados.0.smithi125.stdout:445: write oid 41 current snap is 41 2021-03-24T21:55:57.125 INFO:tasks.rados.rados.0.smithi125.stdout:445: seq_num 188 ranges {759716=539284,2034850=614062} 2021-03-24T21:55:57.128 INFO:tasks.rados.rados.0.smithi125.stdout:445: writing smithi12556470-41 from 759716 to 1299000 tid 1 2021-03-24T21:55:57.136 INFO:tasks.rados.rados.0.smithi125.stdout:445: writing smithi12556470-41 from 2034850 to 2648912 tid 2 2021-03-24T21:55:57.136 INFO:tasks.rados.rados.0.smithi125.stdout:446: write (excl) oid 27 current snap is 41 2021-03-24T21:55:57.136 INFO:tasks.rados.rados.0.smithi125.stdout:446: seq_num 189 ranges {776523=614533,1976979=757720} 2021-03-24T21:55:57.143 INFO:tasks.rados.rados.0.smithi125.stdout:446: writing smithi12556470-27 from 776523 to 1391056 tid 1 2021-03-24T21:55:57.148 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:57.151 INFO:tasks.rados.rados.0.smithi125.stdout:446: writing smithi12556470-27 from 1976979 to 2734699 tid 2 2021-03-24T21:55:57.152 INFO:tasks.rados.rados.0.smithi125.stdout:447: delete oid 34 current snap is 41 2021-03-24T21:55:57.152 INFO:tasks.rados.rados.0.smithi125.stdout:442: finishing rollback tid 0 to smithi12556470-43 2021-03-24T21:55:57.152 INFO:tasks.rados.rados.0.smithi125.stdout:442: finishing rollback tid 1 to smithi12556470-43 2021-03-24T21:55:57.152 INFO:tasks.rados.rados.0.smithi125.stdout:442: finishing rollback tid 2 to smithi12556470-43 2021-03-24T21:55:57.153 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 43 v 144 (ObjNum 59 snap 4 seq_num 59) dirty exists 2021-03-24T21:55:57.153 INFO:tasks.rados.rados.0.smithi125.stdout:441: expect (ObjNum 159 snap 35 seq_num 159) 2021-03-24T21:55:57.200 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:57.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:57.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:55:57.256 INFO:tasks.rados.rados.0.smithi125.stdout:445: finishing write tid 1 to smithi12556470-41 2021-03-24T21:55:57.257 INFO:tasks.rados.rados.0.smithi125.stdout:445: finishing write tid 2 to smithi12556470-41 2021-03-24T21:55:57.257 INFO:tasks.rados.rados.0.smithi125.stdout:445: finishing write tid 3 to smithi12556470-41 2021-03-24T21:55:57.257 INFO:tasks.rados.rados.0.smithi125.stdout:445: finishing write tid 4 to smithi12556470-41 2021-03-24T21:55:57.257 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 41 v 158 (ObjNum 188 snap 41 seq_num 188) dirty exists 2021-03-24T21:55:57.258 INFO:tasks.rados.rados.0.smithi125.stdout:445: left oid 41 (ObjNum 188 snap 41 seq_num 188) 2021-03-24T21:55:57.258 INFO:tasks.rados.rados.0.smithi125.stdout:446: finishing write tid 1 to smithi12556470-27 2021-03-24T21:55:57.258 INFO:tasks.rados.rados.0.smithi125.stdout:446: finishing write tid 2 to smithi12556470-27 2021-03-24T21:55:57.258 INFO:tasks.rados.rados.0.smithi125.stdout:446: finishing write tid 3 to smithi12556470-27 2021-03-24T21:55:57.259 INFO:tasks.rados.rados.0.smithi125.stdout:446: finishing write tid 4 to smithi12556470-27 2021-03-24T21:55:57.259 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 27 v 161 (ObjNum 189 snap 41 seq_num 189) dirty exists 2021-03-24T21:55:57.259 INFO:tasks.rados.rados.0.smithi125.stdout:446: left oid 27 (ObjNum 189 snap 41 seq_num 189) 2021-03-24T21:55:57.259 INFO:tasks.rados.rados.0.smithi125.stdout:441: done (4 left) 2021-03-24T21:55:57.259 INFO:tasks.rados.rados.0.smithi125.stdout:442: done (3 left) 2021-03-24T21:55:57.260 INFO:tasks.rados.rados.0.smithi125.stdout:445: done (2 left) 2021-03-24T21:55:57.260 INFO:tasks.rados.rados.0.smithi125.stdout:446: done (1 left) 2021-03-24T21:55:57.260 INFO:tasks.rados.rados.0.smithi125.stdout:447: done (0 left) 2021-03-24T21:55:57.260 INFO:tasks.rados.rados.0.smithi125.stdout:448: copy_from oid 2 from oid 16 current snap is 41 2021-03-24T21:55:57.261 INFO:tasks.rados.rados.0.smithi125.stdout:449: read oid 50 snap 40 2021-03-24T21:55:57.261 INFO:tasks.rados.rados.0.smithi125.stdout:449: expect deleted 2021-03-24T21:55:57.261 INFO:tasks.rados.rados.0.smithi125.stdout:450: snap_create 2021-03-24T21:55:57.272 INFO:tasks.rados.rados.0.smithi125.stdout:448: finishing copy_from to smithi12556470-2 2021-03-24T21:55:57.272 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 2 v 132 (ObjNum 167 snap 36 seq_num 167) dirty exists 2021-03-24T21:55:57.273 INFO:tasks.rados.rados.0.smithi125.stdout:448: finishing copy_from racing read to smithi12556470-2 2021-03-24T21:55:57.304 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:57.318 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:57 smithi142 conmon[20125]: cluster 2021-03-24T21:55:56.125015+0000 osd.2 (osd.2) 35 : cluster [DBG] 6.14 scrub ok 2021-03-24T21:55:57.318 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:57 smithi142 conmon[20125]: cluster 2021-03-24T21:55:56.490434+0000 osd.3 (osd.3) 55 : cluster [DBG] 4.17 scrub ok 2021-03-24T21:55:57.318 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:57 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:55:57.318 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:57 smithi142 conmon[20125]: T21:55:57.115538+0000 mon.a (mon.0) 775 : 2021-03-24T21:55:57.319 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:57 smithi142 conmon[20125]: cluster [DBG] osdmap e188: 8 total, 8 up, 6 in 2021-03-24T21:55:57.360 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:57.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:57.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:55:57.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:57 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:55:57.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:57 smithi125 conmon[24927]: 24T21:55:56.125015+0000 osd.2 (osd.2) 35 2021-03-24T21:55:57.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:57 smithi125 conmon[24927]: : cluster [DBG] 6.14 scrub ok 2021-03-24T21:55:57.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:57 smithi125 conmon[24927]: 2021-03-24T21:55:57.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:57 smithi125 conmon[24927]: cluster 2021-03-24T21:55:57.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:57 smithi125 conmon[24927]: 2021-03-24T21: 2021-03-24T21:55:57.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:57 smithi125 conmon[24927]: 55:56.490434+0000 2021-03-24T21:55:57.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:57 smithi125 conmon[24927]: osd.3 (osd.3 2021-03-24T21:55:57.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:57 smithi125 conmon[24927]: ) 55 : cluster 2021-03-24T21:55:57.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:57 smithi125 conmon[24927]: [DBG] 4.17 scrub ok 2021-03-24T21:55:57.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:57 smithi125 conmon[24927]: cluster 2021-03-24T21:55:57.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:57 smithi125 conmon[24927]: 2021-03-24T21 2021-03-24T21:55:57.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:57 smithi125 conmon[24927]: :55:57.115538+0000 2021-03-24T21:55:57.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:57 smithi125 conmon[24927]: mon.a (mon. 2021-03-24T21:55:57.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:57 smithi125 conmon[24927]: 0) 775 : cluster 2021-03-24T21:55:57.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:57 smithi125 conmon[24927]: [DBG] osdmap e188: 8 total, 8 up, 6 in 2021-03-24T21:55:57.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:57 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:55:57.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:57 smithi125 conmon[19936]: :55:56.125015+0000 osd.2 (osd.2) 35 : cluster [DBG] 6.14 scrub ok 2021-03-24T21:55:57.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:57 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:55:57.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:57 smithi125 conmon[19936]: 03-24T21:55:56.490434+0000 osd.3 (osd. 2021-03-24T21:55:57.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:57 smithi125 conmon[19936]: 3) 55 : cluster [DBG] 2021-03-24T21:55:57.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:57 smithi125 conmon[19936]: 4.17 scrub ok 2021-03-24T21:55:57.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:57 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:55:57.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:57 smithi125 conmon[19936]: -03-24T21:55 2021-03-24T21:55:57.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:57 smithi125 conmon[19936]: :57.115538+0000 mon.a 2021-03-24T21:55:57.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:57 smithi125 conmon[19936]: (mon.0) 775 : cluster 2021-03-24T21:55:57.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:57 smithi125 conmon[19936]: [DBG] osdmap e188: 8 total, 8 up, 6 in 2021-03-24T21:55:57.467 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:57.515 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:55:57.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:57.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:55:57.621 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:57.684 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:55:57.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:57.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:55:57.788 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:57.850 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:57.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:57.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:55:57.954 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:57.977 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2021-03-24T21:55:57.977 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T21:55:58.011 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:58.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:58.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:55:58.118 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:58.175 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:58.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:58.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:55:58.184 INFO:tasks.rados.rados.0.smithi125.stdout:448: done (2 left) 2021-03-24T21:55:58.184 INFO:tasks.rados.rados.0.smithi125.stdout:449: done (1 left) 2021-03-24T21:55:58.184 INFO:tasks.rados.rados.0.smithi125.stdout:450: done (0 left) 2021-03-24T21:55:58.184 INFO:tasks.rados.rados.0.smithi125.stdout:451: snap_remove snap 41 2021-03-24T21:55:58.239 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:58 smithi142 conmon[20125]: cluster 2021-03-24T21:55:56.737877+0000 mgr.y (mgr.14144) 353 : cluster [DBG] pgmap v443: 148 pgs: 148 active+clean; 112 MiB data, 351 MiB used, 536 GiB / 536 GiB avail; 3.0 KiB/s rd, 2.2 MiB/s wr, 12 op/s 2021-03-24T21:55:58.240 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:58 smithi142 conmon[20125]: cluster 2021-03-24T21:55:57.107711+0000 osd.7 (osd.7) 42 : cluster [DBG] 3.15 scrub ok 2021-03-24T21:55:58.240 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:58 smithi142 conmon[20125]: cluster 2021-03-24T21:55:57.194602+0000 osd.5 (osd.5) 53 : cluster [DBG] 3.5 scrub ok 2021-03-24T21:55:58.240 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:58 smithi142 conmon[20125]: audit 2021-03-24T21:55:57.275581+0000 mon.a (mon.0) 776 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:55:58.240 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:58 smithi142 conmon[20125]: cluster 2021-03-24T21:55:57.297078+0000 osd.0 (osd.0) 52 : cluster [DBG] 4.b scrub ok 2021-03-24T21:55:58.240 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:58 smithi142 conmon[20125]: audit 2021-03-24T21:55:57.594268+0000 mon.a (mon.0) 777 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:55:58.280 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:58.336 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:55:58.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:58.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:55:58.398 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:58 smithi125 conmon[19936]: cluster 2021-03-24T21:55:56.737877+0000 mgr.y (mgr. 2021-03-24T21:55:58.398 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:58 smithi125 conmon[19936]: 14144) 353 : cluster [DBG] pgmap v443: 148 pgs: 148 active+clean; 112 MiB data, 351 MiB used, 536 GiB / 536 GiB avail; 3.0 KiB/s rd, 2.2 MiB/s wr, 12 op/s 2021-03-24T21:55:58.398 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:58 smithi125 conmon[19936]: cluster 2021-03-24T21:55:57.107711+0000 osd.7 (osd.7) 42 : cluster [DBG] 3.15 scrub ok 2021-03-24T21:55:58.399 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:58 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:55:58.399 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:58 smithi125 conmon[19936]: 03-24T21:55:57.194602+0000 osd.5 (osd.5) 53 : cluster [DBG] 3.5 scrub ok 2021-03-24T21:55:58.399 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:58 smithi125 conmon[19936]: audit 2021-03-24T21:55:57.275581+0000 mon.a (mon.0) 776 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:55:58.399 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:58 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:55:58.399 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:58 smithi125 conmon[19936]: -03-24T21:55:57.297078+0000 osd.0 (osd.0) 52 : cluster [DBG] 4.b scrub ok 2021-03-24T21:55:58.400 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:58 smithi125 conmon[19936]: audit 2021-03 2021-03-24T21:55:58.400 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:58 smithi125 conmon[19936]: -24T21:55:57.594268+0000 mon.a (mon.0) 777 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:55:58.400 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:58 smithi125 conmon[24927]: cluster 2021-03-24T21:55:56.737877+0000 2021-03-24T21:55:58.401 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:58 smithi125 conmon[24927]: mgr.y (mgr.14144) 353 : cluster [DBG] pgmap v443: 148 pgs: 148 active+clean; 112 MiB data, 351 MiB used, 536 GiB / 536 GiB avail; 3.0 KiB/s rd, 2.2 MiB/s wr, 12 op/s 2021-03-24T21:55:58.401 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:58 smithi125 conmon[24927]: cluster 2021-03-24T21:55:57 2021-03-24T21:55:58.401 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:58 smithi125 conmon[24927]: .107711+0000 osd.7 (osd.7) 42 : cluster [DBG] 3.15 scrub ok 2021-03-24T21:55:58.401 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:58 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:55:58.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:58 smithi125 conmon[24927]: 21:55:57.194602+0000 osd.5 (osd.5) 53 : cluster [DBG] 3.5 scrub ok 2021-03-24T21:55:58.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:58 smithi125 conmon[24927]: audit 2021-03-24T21: 2021-03-24T21:55:58.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:58 smithi125 conmon[24927]: 55:57.275581+0000 mon.a (mon.0) 776 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:55:58.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:58 smithi125 conmon[24927]: cluster 2021-03-24T21:55:57.297078 2021-03-24T21:55:58.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:58 smithi125 conmon[24927]: +0000 osd.0 (osd.0) 52 : cluster [DBG] 4.b scrub ok 2021-03-24T21:55:58.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:58 smithi125 conmon[24927]: audit 2021-03-24T21:55:57.594268+0000 2021-03-24T21:55:58.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:58 smithi125 conmon[24927]: mon.a (mon.0) 777 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:55:58.443 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:58.503 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-24T21:55:58.506 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2021-03-24T21:55:58.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:58.508 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:58.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:55:58.609 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:58.663 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T21:55:58.667 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:55:58.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:58.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:55:58.682 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T21:55:58.710 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T21:55:58.730 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T21:55:58.754 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T21:55:58.774 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:58.777 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T21:55:58.800 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T21:55:58.825 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T21:55:58.831 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:58.833 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-24T21:55:58.833 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-24T21:55:58.833 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-24T21:55:58.833 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-24T21:55:58.834 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-24T21:55:58.834 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-24T21:55:58.834 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-24T21:55:58.834 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-24T21:55:58.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:58.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:58.937 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:58.953 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.6 dump_blocked_ops 2021-03-24T21:55:59.000 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:59.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:59.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:59.107 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:59.164 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:55:59.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:59.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:55:59.193 INFO:tasks.rados.rados.0.smithi125.stdout:451: done (0 left) 2021-03-24T21:55:59.194 INFO:tasks.rados.rados.0.smithi125.stdout:452: read oid 35 snap -1 2021-03-24T21:55:59.194 INFO:tasks.rados.rados.0.smithi125.stdout:452: expect deleted 2021-03-24T21:55:59.194 INFO:tasks.rados.rados.0.smithi125.stdout:453: read oid 23 snap -1 2021-03-24T21:55:59.194 INFO:tasks.rados.rados.0.smithi125.stdout:453: expect deleted 2021-03-24T21:55:59.195 INFO:tasks.rados.rados.0.smithi125.stdout:454: snap_remove snap 40 2021-03-24T21:55:59.250 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T21:55:59.271 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:59.282 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:59 smithi125 conmon[19936]: cluster 2021-03-24T21:55: 2021-03-24T21:55:59.283 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:59 smithi125 conmon[19936]: 58.082762+0000 osd.7 (osd.7) 43 : cluster [DBG] 3.13 scrub ok 2021-03-24T21:55:59.283 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:59 smithi125 conmon[19936]: cluster 2021-03-24T21:55:59.283 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:59 smithi125 conmon[19936]: 2021-03-24T21:55 2021-03-24T21:55:59.284 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:59 smithi125 conmon[19936]: :58.182296+0000 mon.a ( 2021-03-24T21:55:59.284 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:59 smithi125 conmon[19936]: mon.0) 778 : cluster 2021-03-24T21:55:59.284 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:59 smithi125 conmon[19936]: [DBG] osdmap e189: 8 total, 8 up, 6 in 2021-03-24T21:55:59.284 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:59 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:55:59.285 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:59 smithi125 conmon[19936]: -03-24T21:55: 2021-03-24T21:55:59.285 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:59 smithi125 conmon[19936]: 58.253770+0000 osd.0 ( 2021-03-24T21:55:59.285 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:59 smithi125 conmon[19936]: osd.0) 53 : cluster 2021-03-24T21:55:59.285 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:59 smithi125 conmon[19936]: [DBG] 5.2 scrub ok 2021-03-24T21:55:59.285 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:59 smithi125 conmon[19936]: audit 2021 2021-03-24T21:55:59.286 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:59 smithi125 conmon[19936]: -03-24T21:55: 2021-03-24T21:55:59.286 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:59 smithi125 conmon[19936]: 58.483488+0000 mgr.y (mgr. 2021-03-24T21:55:59.286 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:59 smithi125 conmon[19936]: 14144) 354 : audit [DBG] from='client.14907 v1:172.21.15.125:0/2000775733' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T21:55:59.286 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:59 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:55:59.287 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:59 smithi125 conmon[19936]: 03-24T21:55:58.636883 2021-03-24T21:55:59.287 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:59 smithi125 conmon[19936]: +0000 mon.c (mon.2) 34 : audit 2021-03-24T21:55:59.287 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:55:59 smithi125 conmon[19936]: [DBG] from='client.? v1:172.21.15.125:0/491285596' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:55:59.287 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:59 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:55:59.288 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:59 smithi125 conmon[24927]: T21:55:58.082762+0000 osd.7 (osd.7) 43 : 2021-03-24T21:55:59.288 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:59 smithi125 conmon[24927]: cluster [DBG] 3.13 scrub ok 2021-03-24T21:55:59.288 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:59 smithi125 conmon[24927]: cluster 2021-03-24T21:55:59.288 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:59 smithi125 conmon[24927]: 2021-03-24T21: 2021-03-24T21:55:59.289 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:59 smithi125 conmon[24927]: 55:58.182296+0000 2021-03-24T21:55:59.289 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:59 smithi125 conmon[24927]: mon.a (mon.0) 2021-03-24T21:55:59.289 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:59 smithi125 conmon[24927]: 778 : cluster [DBG] osdmap e189: 8 total, 8 up, 6 in 2021-03-24T21:55:59.289 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:59 smithi125 conmon[24927]: 2021-03-24T21:55:59.290 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:59 smithi125 conmon[24927]: cluster 2021-03-24T21:55:59.290 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:59 smithi125 conmon[24927]: 2021-03-24T21 2021-03-24T21:55:59.290 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:59 smithi125 conmon[24927]: :55:58.253770 2021-03-24T21:55:59.290 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:59 smithi125 conmon[24927]: +0000 osd.0 (osd.0 2021-03-24T21:55:59.290 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:59 smithi125 conmon[24927]: ) 53 : cluster [DBG] 2021-03-24T21:55:59.291 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:59 smithi125 conmon[24927]: 5.2 scrub ok 2021-03-24T21:55:59.291 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:59 smithi125 conmon[24927]: audit 2021-03-24T21:55:59.291 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:59 smithi125 conmon[24927]: 2021-03-24T21 2021-03-24T21:55:59.291 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:59 smithi125 conmon[24927]: :55:58.483488 2021-03-24T21:55:59.292 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:59 smithi125 conmon[24927]: +0000 mgr.y (mgr.14144 2021-03-24T21:55:59.292 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:59 smithi125 conmon[24927]: ) 354 : audit [DBG] 2021-03-24T21:55:59.292 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:59 smithi125 conmon[24927]: from='client.14907 v1:172.21.15.125:0/2000775733' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T21:55:59.292 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:59 smithi125 conmon[24927]: audit 2021-03-24T21:55:59.292 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:59 smithi125 conmon[24927]: 2021-03-24T21 2021-03-24T21:55:59.293 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:59 smithi125 conmon[24927]: :55:58.636883 2021-03-24T21:55:59.293 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:59 smithi125 conmon[24927]: +0000 mon.c (mon.2 2021-03-24T21:55:59.293 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:59 smithi125 conmon[24927]: ) 34 : audit [DBG] 2021-03-24T21:55:59.293 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:55:59 smithi125 conmon[24927]: from='client.? v1:172.21.15.125:0/491285596' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:55:59.317 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:59.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:59.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:55:59.421 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:59.456 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:59 smithi142 conmon[20125]: cluster 2021-03-24T21:55:58 2021-03-24T21:55:59.456 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:59 smithi142 conmon[20125]: .082762+0000 osd.7 (osd.7) 43 : cluster [DBG] 3.13 scrub ok 2021-03-24T21:55:59.456 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:59 smithi142 conmon[20125]: cluster 2021-03-24T21:55 2021-03-24T21:55:59.457 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:59 smithi142 conmon[20125]: :58.182296+0000 mon.a (mon.0) 778 : cluster [DBG] osdmap e189: 8 total, 8 up, 6 in 2021-03-24T21:55:59.457 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:59 smithi142 conmon[20125]: cluster 2021-03-24T21:55:59.457 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:59 smithi142 conmon[20125]: 2021-03-24T21:55:58.253770+0000 2021-03-24T21:55:59.457 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:59 smithi142 conmon[20125]: osd.0 (osd.0) 53 : cluster [DBG] 5.2 scrub ok 2021-03-24T21:55:59.458 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:59 smithi142 conmon[20125]: audit 2021 2021-03-24T21:55:59.458 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:59 smithi142 conmon[20125]: -03-24T21:55:58.483488+0000 mgr.y 2021-03-24T21:55:59.458 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:59 smithi142 conmon[20125]: (mgr.14144) 354 : audit [DBG] from='client.14907 v1:172.21.15.125:0/2000775733' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T21:55:59.458 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:59 smithi142 conmon[20125]: audit 2021-03-24T21:55:59.458 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:59 smithi142 conmon[20125]: 2021-03-24T21:55:58.636883+0000 2021-03-24T21:55:59.459 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:59 smithi142 conmon[20125]: mon.c (mon.2) 34 : audit [DBG] from='client.? v1:172.21.15.125:0/491285596' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:55:59.459 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:55:59 smithi142 conmon[20125]: 2021-03-24T21:55:59.475 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:59.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:59.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:55:59.581 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:59.637 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:55:59.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:59.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:55:59.742 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:59.794 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:55:59.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:59.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:55:59.867 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-24T21:55:59.901 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:59.958 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:55:59.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:55:59.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:56:00.065 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:00.104 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:00.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:00.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:56:00.111 INFO:tasks.rados.rados.0.smithi125.stdout:454: done (2 left) 2021-03-24T21:56:00.111 INFO:tasks.rados.rados.0.smithi125.stdout:455: write oid 6 current snap is 42 2021-03-24T21:56:00.112 INFO:tasks.rados.rados.0.smithi125.stdout:455: seq_num 191 ranges {410137=481384,1410296=761389,2728968=481169} 2021-03-24T21:56:00.116 INFO:tasks.rados.rados.0.smithi125.stdout:455: writing smithi12556470-6 from 410137 to 891521 tid 1 2021-03-24T21:56:00.124 INFO:tasks.rados.rados.0.smithi125.stdout:455: writing smithi12556470-6 from 1410296 to 2171685 tid 2 2021-03-24T21:56:00.130 INFO:tasks.rados.rados.0.smithi125.stdout:455: writing smithi12556470-6 from 2728968 to 3210137 tid 3 2021-03-24T21:56:00.131 INFO:tasks.rados.rados.0.smithi125.stdout:456: write (excl) oid 17 current snap is 42 2021-03-24T21:56:00.131 INFO:tasks.rados.rados.0.smithi125.stdout:456: seq_num 192 ranges {426944=556633,1468778=621400,2872744=354200} 2021-03-24T21:56:00.136 INFO:tasks.rados.rados.0.smithi125.stdout:456: writing smithi12556470-17 from 426944 to 983577 tid 1 2021-03-24T21:56:00.142 INFO:tasks.rados.rados.0.smithi125.stdout:456: writing smithi12556470-17 from 1468778 to 2090178 tid 2 2021-03-24T21:56:00.149 INFO:tasks.rados.rados.0.smithi125.stdout:456: writing smithi12556470-17 from 2872744 to 3226944 tid 3 2021-03-24T21:56:00.149 INFO:tasks.rados.rados.0.smithi125.stdout:457: read oid 46 snap -1 2021-03-24T21:56:00.149 INFO:tasks.rados.rados.0.smithi125.stdout:457: expect (ObjNum 83 snap 10 seq_num 83) 2021-03-24T21:56:00.149 INFO:tasks.rados.rados.0.smithi125.stdout:458: read oid 49 snap -1 2021-03-24T21:56:00.149 INFO:tasks.rados.rados.0.smithi125.stdout:458: expect (ObjNum 168 snap 37 seq_num 168) 2021-03-24T21:56:00.150 INFO:tasks.rados.rados.0.smithi125.stdout:459: write oid 19 current snap is 42 2021-03-24T21:56:00.150 INFO:tasks.rados.rados.0.smithi125.stdout:459: seq_num 193 ranges {443751=631882,1810907=765058,3067461=176290} 2021-03-24T21:56:00.154 INFO:tasks.rados.rados.0.smithi125.stdout:459: writing smithi12556470-19 from 443751 to 1075633 tid 1 2021-03-24T21:56:00.163 INFO:tasks.rados.rados.0.smithi125.stdout:459: writing smithi12556470-19 from 1810907 to 2575965 tid 2 2021-03-24T21:56:00.167 INFO:tasks.rados.rados.0.smithi125.stdout:459: writing smithi12556470-19 from 3067461 to 3243751 tid 3 2021-03-24T21:56:00.167 INFO:tasks.rados.rados.0.smithi125.stdout:460: rollback oid 33 current snap is 42 2021-03-24T21:56:00.168 INFO:tasks.rados.rados.0.smithi125.stdout:460: done (7 left) 2021-03-24T21:56:00.168 INFO:tasks.rados.rados.0.smithi125.stdout:461: snap_create 2021-03-24T21:56:00.168 INFO:tasks.rados.rados.0.smithi125.stdout:458: expect (ObjNum 168 snap 37 seq_num 168) 2021-03-24T21:56:00.199 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:00 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T21:56:00.199 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:00 smithi125 conmon[19936]: T21:55:58.738701+0000 mgr.y (mgr.14144) 355 : cluster [DBG] pgmap v446: 148 pgs: 148 active+clean; 114 MiB data, 364 MiB used, 536 GiB / 536 GiB avail; 767 B/s rd, 1.9 MiB/s wr, 5 op/s 2021-03-24T21:56:00.200 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:00 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:56:00.200 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:00 smithi125 conmon[19936]: -03-24T21 2021-03-24T21:56:00.200 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:00 smithi125 conmon[19936]: :55:59.032156 2021-03-24T21:56:00.200 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:00 smithi125 conmon[19936]: +0000 osd.4 (osd. 2021-03-24T21:56:00.201 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:00 smithi125 conmon[19936]: 4) 53 : cluster 2021-03-24T21:56:00.201 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:00 smithi125 conmon[19936]: [DBG] 5.e scrub ok 2021-03-24T21:56:00.201 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:00 smithi125 conmon[19936]: cluster 2021-03-24T21:56:00.201 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:00 smithi125 conmon[19936]: 2021-03-24T21: 2021-03-24T21:56:00.201 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:00 smithi125 conmon[19936]: 55:59.192323+0000 mon.a (mon.0 2021-03-24T21:56:00.202 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:00 smithi125 conmon[19936]: ) 779 : cluster [DBG] osdmap e190: 8 total, 8 up, 6 in 2021-03-24T21:56:00.202 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:00 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:56:00.202 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:00 smithi125 conmon[19936]: -03-24T21:55 2021-03-24T21:56:00.202 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:00 smithi125 conmon[19936]: :59.276293+0000 osd.5 ( 2021-03-24T21:56:00.203 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:00 smithi125 conmon[19936]: osd.5) 54 : cluster 2021-03-24T21:56:00.203 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:00 smithi125 conmon[19936]: [DBG] 7.0 scrub ok 2021-03-24T21:56:00.203 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:00 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:56:00.203 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:00 smithi125 conmon[19936]: 24T21:56:00. 2021-03-24T21:56:00.203 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:00 smithi125 conmon[19936]: 108461+0000 mon.a (mon 2021-03-24T21:56:00.204 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:00 smithi125 conmon[19936]: .0) 780 : cluster 2021-03-24T21:56:00.204 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:00 smithi125 conmon[19936]: [DBG] osdmap e191: 8 total, 8 up, 6 in 2021-03-24T21:56:00.212 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:00.259 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:00.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:00.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:56:00.289 INFO:tasks.rados.rados.0.smithi125.stdout:455: finishing write tid 1 to smithi12556470-6 2021-03-24T21:56:00.290 INFO:tasks.rados.rados.0.smithi125.stdout:455: finishing write tid 2 to smithi12556470-6 2021-03-24T21:56:00.290 INFO:tasks.rados.rados.0.smithi125.stdout:457: expect (ObjNum 83 snap 10 seq_num 83) 2021-03-24T21:56:00.365 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:00.371 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":445,"stamp":"2021-03-24T21:55:58.179162+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":118804116,"num_objects":360,"num_object_clones":5,"num_object_copies":1080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":360,"num_whiteouts":1,"num_read":2217,"num_read_kb":792280,"num_write":1301,"num_write_kb":283602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":324,"num_bytes_recovered":35484837,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2162,"ondisk_log_size":2162,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":370964,"kb_used_data":272036,"kb_used_omap":0,"kb_used_meta":98880,"kb_avail":562099948,"statfs":{"total":575970213888,"available":575590346752,"internally_reserved":0,"allocated":278564864,"data_stored":285908364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":101253120},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":562438,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":1,"num_read":12,"num_read_kb":9,"num_write":21,"num_write_kb":8955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.030809"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":"147","reported_epoch":"187","state":"active+clean","last_fresh":"2021-03-24T21:55:56.110580+0000","last_change":"2021-03-24T21:54:50.292202+0000","last_active":"2021-03-24T21:55:56.110580+0000","last_peered":"2021-03-24T21:55:56.110580+0000","last_clean":"2021-03-24T21:55:56.110580+0000","last_became_active":"2021-03-24T21:54:38.111248+0000","last_became_peered":"2021-03-24T21:54:38.111248+0000","last_unstale":"2021-03-24T21:55:56.110580+0000","last_undegraded":"2021-03-24T21:55:56.110580+0000","last_fullsized":"2021-03-24T21:55:56.110580+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:50.292025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:54:50.292025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"59","reported_epoch":"184","state":"active+clean","last_fresh":"2021-03-24T21:55:53.146044+0000","last_change":"2021-03-24T21:55:10.100067+0000","last_active":"2021-03-24T21:55:53.146044+0000","last_peered":"2021-03-24T21:55:53.146044+0000","last_clean":"2021-03-24T21:55:53.146044+0000","last_became_active":"2021-03-24T21:55:10.099666+0000","last_became_peered":"2021-03-24T21:55:10.099666+0000","last_unstale":"2021-03-24T21:55:53.146044+0000","last_undegraded":"2021-03-24T21:55:53.146044+0000","last_fullsized":"2021-03-24T21:55:53.146044+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:14.710709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:14.710709+0000","last_clean_scrub_stamp":"2021-03-24T21:54:14.710709+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":"139","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.137611+0000","last_change":"2021-03-24T21:55:38.294845+0000","last_active":"2021-03-24T21:55:52.137611+0000","last_peered":"2021-03-24T21:55:52.137611+0000","last_clean":"2021-03-24T21:55:52.137611+0000","last_became_active":"2021-03-24T21:51:45.581000+0000","last_became_peered":"2021-03-24T21:51:45.581000+0000","last_unstale":"2021-03-24T21:55:52.137611+0000","last_undegraded":"2021-03-24T21:55:52.137611+0000","last_fullsized":"2021-03-24T21:55:52.137611+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:38.294685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:38.294685+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"69'15","reported_seq":"227","reported_epoch":"187","state":"active+clean","last_fresh":"2021-03-24T21:55:56.110845+0000","last_change":"2021-03-24T21:54:39.222600+0000","last_active":"2021-03-24T21:55:56.110845+0000","last_peered":"2021-03-24T21:55:56.110845+0000","last_clean":"2021-03-24T21:55:56.110845+0000","last_became_active":"2021-03-24T21:54:38.127426+0000","last_became_peered":"2021-03-24T21:54:38.127426+0000","last_unstale":"2021-03-24T21:55:56.110845+0000","last_undegraded":"2021-03-24T21:55:56.110845+0000","last_fullsized":"2021-03-24T21:55:56.110845+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:53:41.295875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:41.295875+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2076,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"152","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.133788+0000","last_change":"2021-03-24T21:54:43.482408+0000","last_active":"2021-03-24T21:55:52.133788+0000","last_peered":"2021-03-24T21:55:52.133788+0000","last_clean":"2021-03-24T21:55:52.133788+0000","last_became_active":"2021-03-24T21:54:38.130293+0000","last_became_peered":"2021-03-24T21:54:38.130293+0000","last_unstale":"2021-03-24T21:55:52.133788+0000","last_undegraded":"2021-03-24T21:55:52.133788+0000","last_fullsized":"2021-03-24T21:55:52.133788+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:43.482307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:54:43.482307+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"153","reported_epoch":"187","state":"active+clean","last_fresh":"2021-03-24T21:55:56.110699+0000","last_change":"2021-03-24T21:54:38.120961+0000","last_active":"2021-03-24T21:55:56.110699+0000","last_peered":"2021-03-24T21:55:56.110699+0000","last_clean":"2021-03-24T21:55:56.110699+0000","last_became_active":"2021-03-24T21:54:38.120575+0000","last_became_peered":"2021-03-24T21:54:38.120575+0000","last_unstale":"2021-03-24T21:55:56.110699+0000","last_undegraded":"2021-03-24T21:55:56.110699+0000","last_fullsized":"2021-03-24T21:55:56.110699+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:48.329480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:48.329480+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":"158","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.130708+0000","last_change":"2021-03-24T21:55:38.433064+0000","last_active":"2021-03-24T21:55:52.130708+0000","last_peered":"2021-03-24T21:55:52.130708+0000","last_clean":"2021-03-24T21:55:52.130708+0000","last_became_active":"2021-03-24T21:54:38.130163+0000","last_became_peered":"2021-03-24T21:54:38.130163+0000","last_unstale":"2021-03-24T21:55:52.130708+0000","last_undegraded":"2021-03-24T21:55:52.130708+0000","last_fullsized":"2021-03-24T21:55:52.130708+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:38.432918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:38.432918+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"69'20","reported_seq":"173","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.129178+0000","last_change":"2021-03-24T21:54:54.541380+0000","last_active":"2021-03-24T21:55:52.129178+0000","last_peered":"2021-03-24T21:55:52.129178+0000","last_clean":"2021-03-24T21:55:52.129178+0000","last_became_active":"2021-03-24T21:51:47.405776+0000","last_became_peered":"2021-03-24T21:51:47.405776+0000","last_unstale":"2021-03-24T21:55:52.129178+0000","last_undegraded":"2021-03-24T21:55:52.129178+0000","last_fullsized":"2021-03-24T21:55:52.129178+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'20","last_scrub_stamp":"2021-03-24T21:54:54.541183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:54:54.541183+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":22,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"142","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.134566+0000","last_change":"2021-03-24T21:54:49.409465+0000","last_active":"2021-03-24T21:55:52.134566+0000","last_peered":"2021-03-24T21:55:52.134566+0000","last_clean":"2021-03-24T21:55:52.134566+0000","last_became_active":"2021-03-24T21:54:38.127696+0000","last_became_peered":"2021-03-24T21:54:38.127696+0000","last_unstale":"2021-03-24T21:55:52.134566+0000","last_undegraded":"2021-03-24T21:55:52.134566+0000","last_fullsized":"2021-03-24T21:55:52.134566+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:49.409299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:54:49.409299+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"138","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.144760+0000","last_change":"2021-03-24T21:54:44.879188+0000","last_active":"2021-03-24T21:55:52.144760+0000","last_peered":"2021-03-24T21:55:52.144760+0000","last_clean":"2021-03-24T21:55:52.144760+0000","last_became_active":"2021-03-24T21:51:49.413342+0000","last_became_peered":"2021-03-24T21:51:49.413342+0000","last_unstale":"2021-03-24T21:55:52.144760+0000","last_undegraded":"2021-03-24T21:55:52.144760+0000","last_fullsized":"2021-03-24T21:55:52.144760+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:44.878935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:54:44.878935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":"170","reported_epoch":"186","state":"active+clean","last_fresh":"2021-03-24T21:55:55.128158+0000","last_change":"2021-03-24T21:55:10.097750+0000","last_active":"2021-03-24T21:55:55.128158+0000","last_peered":"2021-03-24T21:55:55.128158+0000","last_clean":"2021-03-24T21:55:55.128158+0000","last_became_active":"2021-03-24T21:55:10.097431+0000","last_became_peered":"2021-03-24T21:55:10.097431+0000","last_unstale":"2021-03-24T21:55:55.128158+0000","last_undegraded":"2021-03-24T21:55:55.128158+0000","last_fullsized":"2021-03-24T21:55:55.128158+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:43.877604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:54:43.877604+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"69'13","reported_seq":"166","reported_epoch":"187","state":"active+clean","last_fresh":"2021-03-24T21:55:56.110905+0000","last_change":"2021-03-24T21:55:31.085064+0000","last_active":"2021-03-24T21:55:56.110905+0000","last_peered":"2021-03-24T21:55:56.110905+0000","last_clean":"2021-03-24T21:55:56.110905+0000","last_became_active":"2021-03-24T21:51:47.407488+0000","last_became_peered":"2021-03-24T21:51:47.407488+0000","last_unstale":"2021-03-24T21:55:56.110905+0000","last_undegraded":"2021-03-24T21:55:56.110905+0000","last_fullsized":"2021-03-24T21:55:56.110905+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2021-03-24T21:55:31.084873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:31.084873+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"145","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.142259+0000","last_change":"2021-03-24T21:55:39.435565+0000","last_active":"2021-03-24T21:55:52.142259+0000","last_peered":"2021-03-24T21:55:52.142259+0000","last_clean":"2021-03-24T21:55:52.142259+0000","last_became_active":"2021-03-24T21:54:38.122349+0000","last_became_peered":"2021-03-24T21:54:38.122349+0000","last_unstale":"2021-03-24T21:55:52.142259+0000","last_undegraded":"2021-03-24T21:55:52.142259+0000","last_fullsized":"2021-03-24T21:55:52.142259+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:39.435268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:55:39.435268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"146","reported_epoch":"184","state":"active+clean","last_fresh":"2021-03-24T21:55:53.146576+0000","last_change":"2021-03-24T21:54:57.741556+0000","last_active":"2021-03-24T21:55:53.146576+0000","last_peered":"2021-03-24T21:55:53.146576+0000","last_clean":"2021-03-24T21:55:53.146576+0000","last_became_active":"2021-03-24T21:54:38.137706+0000","last_became_peered":"2021-03-24T21:54:38.137706+0000","last_unstale":"2021-03-24T21:55:53.146576+0000","last_undegraded":"2021-03-24T21:55:53.146576+0000","last_fullsized":"2021-03-24T21:55:53.146576+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:57.741464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:54:57.741464+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":"126","reported_epoch":"187","state":"active+clean","last_fresh":"2021-03-24T21:55:56.111023+0000","last_change":"2021-03-24T21:55:10.099630+0000","last_active":"2021-03-24T21:55:56.111023+0000","last_peered":"2021-03-24T21:55:56.111023+0000","last_clean":"2021-03-24T21:55:56.111023+0000","last_became_active":"2021-03-24T21:55:10.098790+0000","last_became_peered":"2021-03-24T21:55:10.098790+0000","last_unstale":"2021-03-24T21:55:56.111023+0000","last_undegraded":"2021-03-24T21:55:56.111023+0000","last_fullsized":"2021-03-24T21:55:56.111023+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:09.127715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:54:09.127715+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"91'9","reported_seq":"169","reported_epoch":"187","state":"active+clean","last_fresh":"2021-03-24T21:55:56.112548+0000","last_change":"2021-03-24T21:54:39.157590+0000","last_active":"2021-03-24T21:55:56.112548+0000","last_peered":"2021-03-24T21:55:56.112548+0000","last_clean":"2021-03-24T21:55:56.112548+0000","last_became_active":"2021-03-24T21:54:38.127055+0000","last_became_peered":"2021-03-24T21:54:38.127055+0000","last_unstale":"2021-03-24T21:55:56.112548+0000","last_undegraded":"2021-03-24T21:55:56.112548+0000","last_fullsized":"2021-03-24T21:55:56.112548+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2021-03-24T21:53:33.290000+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2021-03-24T21:53:33.290000+0000","last_clean_scrub_stamp":"2021-03-24T21:53:33.290000+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":8,"num_write":7,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"142","reported_epoch":"187","state":"active+clean","last_fresh":"2021-03-24T21:55:56.111865+0000","last_change":"2021-03-24T21:54:24.199810+0000","last_active":"2021-03-24T21:55:56.111865+0000","last_peered":"2021-03-24T21:55:56.111865+0000","last_clean":"2021-03-24T21:55:56.111865+0000","last_became_active":"2021-03-24T21:54:24.199444+0000","last_became_peered":"2021-03-24T21:54:24.199444+0000","last_unstale":"2021-03-24T21:55:56.111865+0000","last_undegraded":"2021-03-24T21:55:56.111865+0000","last_fullsized":"2021-03-24T21:55:56.111865+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:39.277425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:39.277425+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"96","reported_epoch":"187","state":"active+clean","last_fresh":"2021-03-24T21:55:56.111276+0000","last_change":"2021-03-24T21:54:49.323954+0000","last_active":"2021-03-24T21:55:56.111276+0000","last_peered":"2021-03-24T21:55:56.111276+0000","last_clean":"2021-03-24T21:55:56.111276+0000","last_became_active":"2021-03-24T21:54:38.138620+0000","last_became_peered":"2021-03-24T21:54:38.138620+0000","last_unstale":"2021-03-24T21:55:56.111276+0000","last_undegraded":"2021-03-24T21:55:56.111276+0000","last_fullsized":"2021-03-24T21:55:56.111276+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:49.323789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:54:49.323789+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":"139","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.133962+0000","last_change":"2021-03-24T21:54:43.406949+0000","last_active":"2021-03-24T21:55:52.133962+0000","last_peered":"2021-03-24T21:55:52.133962+0000","last_clean":"2021-03-24T21:55:52.133962+0000","last_became_active":"2021-03-24T21:51:45.381533+0000","last_became_peered":"2021-03-24T21:51:45.381533+0000","last_unstale":"2021-03-24T21:55:52.133962+0000","last_undegraded":"2021-03-24T21:55:52.133962+0000","last_fullsized":"2021-03-24T21:55:52.133962+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:43.406875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:54:43.406875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"69'15","reported_seq":"216","reported_epoch":"186","state":"active+clean","last_fresh":"2021-03-24T21:55:55.128162+0000","last_change":"2021-03-24T21:54:39.063715+0000","last_active":"2021-03-24T21:55:55.128162+0000","last_peered":"2021-03-24T21:55:55.128162+0000","last_clean":"2021-03-24T21:55:55.128162+0000","last_became_active":"2021-03-24T21:54:38.140589+0000","last_became_peered":"2021-03-24T21:54:38.140589+0000","last_unstale":"2021-03-24T21:55:55.128162+0000","last_undegraded":"2021-03-24T21:55:55.128162+0000","last_fullsized":"2021-03-24T21:55:55.128162+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:53:58.587152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:58.587152+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"147","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.129914+0000","last_change":"2021-03-24T21:54:38.144481+0000","last_active":"2021-03-24T21:55:52.129914+0000","last_peered":"2021-03-24T21:55:52.129914+0000","last_clean":"2021-03-24T21:55:52.129914+0000","last_became_active":"2021-03-24T21:54:38.143563+0000","last_became_peered":"2021-03-24T21:54:38.143563+0000","last_unstale":"2021-03-24T21:55:52.129914+0000","last_undegraded":"2021-03-24T21:55:52.129914+0000","last_fullsized":"2021-03-24T21:55:52.129914+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:44.657761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:44.657761+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"140","reported_epoch":"187","state":"active+clean","last_fresh":"2021-03-24T21:55:56.111613+0000","last_change":"2021-03-24T21:55:02.129906+0000","last_active":"2021-03-24T21:55:56.111613+0000","last_peered":"2021-03-24T21:55:56.111613+0000","last_clean":"2021-03-24T21:55:56.111613+0000","last_became_active":"2021-03-24T21:51:49.410920+0000","last_became_peered":"2021-03-24T21:51:49.410920+0000","last_unstale":"2021-03-24T21:55:56.111613+0000","last_undegraded":"2021-03-24T21:55:56.111613+0000","last_fullsized":"2021-03-24T21:55:56.111613+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:02.129719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:55:02.129719+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":"153","reported_epoch":"187","state":"active+clean","last_fresh":"2021-03-24T21:55:56.111151+0000","last_change":"2021-03-24T21:54:38.144167+0000","last_active":"2021-03-24T21:55:56.111151+0000","last_peered":"2021-03-24T21:55:56.111151+0000","last_clean":"2021-03-24T21:55:56.111151+0000","last_became_active":"2021-03-24T21:54:38.143876+0000","last_became_peered":"2021-03-24T21:54:38.143876+0000","last_unstale":"2021-03-24T21:55:56.111151+0000","last_undegraded":"2021-03-24T21:55:56.111151+0000","last_fullsized":"2021-03-24T21:55:56.111151+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:11.286003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:52:54.212459+0000","last_clean_scrub_stamp":"2021-03-24T21:54:11.286003+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"69'16","reported_seq":"193","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.129453+0000","last_change":"2021-03-24T21:54:38.191975+0000","last_active":"2021-03-24T21:55:52.129453+0000","last_peered":"2021-03-24T21:55:52.129453+0000","last_clean":"2021-03-24T21:55:52.129453+0000","last_became_active":"2021-03-24T21:54:38.128246+0000","last_became_peered":"2021-03-24T21:54:38.128246+0000","last_unstale":"2021-03-24T21:55:52.129453+0000","last_undegraded":"2021-03-24T21:55:52.129453+0000","last_fullsized":"2021-03-24T21:55:52.129453+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T21:53:42.611540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:42.611540+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":37,"num_read_kb":20,"num_write":18,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":844,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"135","reported_epoch":"184","state":"active+clean","last_fresh":"2021-03-24T21:55:53.146322+0000","last_change":"2021-03-24T21:55:29.900103+0000","last_active":"2021-03-24T21:55:53.146322+0000","last_peered":"2021-03-24T21:55:53.146322+0000","last_clean":"2021-03-24T21:55:53.146322+0000","last_became_active":"2021-03-24T21:51:51.761492+0000","last_became_peered":"2021-03-24T21:51:51.761492+0000","last_unstale":"2021-03-24T21:55:53.146322+0000","last_undegraded":"2021-03-24T21:55:53.146322+0000","last_fullsized":"2021-03-24T21:55:53.146322+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:29.899923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:55:29.899923+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"102'3","reported_seq":"129","reported_epoch":"187","state":"active+clean","last_fresh":"2021-03-24T21:55:56.112175+0000","last_change":"2021-03-24T21:54:24.283503+0000","last_active":"2021-03-24T21:55:56.112175+0000","last_peered":"2021-03-24T21:55:56.112175+0000","last_clean":"2021-03-24T21:55:56.112175+0000","last_became_active":"2021-03-24T21:54:24.188093+0000","last_became_peered":"2021-03-24T21:54:24.188093+0000","last_unstale":"2021-03-24T21:55:56.112175+0000","last_undegraded":"2021-03-24T21:55:56.112175+0000","last_fullsized":"2021-03-24T21:55:56.112175+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2021-03-24T21:54:15.969746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:54:15.969746+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":"91","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.128871+0000","last_change":"2021-03-24T21:54:48.541576+0000","last_active":"2021-03-24T21:55:52.128871+0000","last_peered":"2021-03-24T21:55:52.128871+0000","last_clean":"2021-03-24T21:55:52.128871+0000","last_became_active":"2021-03-24T21:54:38.130034+0000","last_became_peered":"2021-03-24T21:54:38.130034+0000","last_unstale":"2021-03-24T21:55:52.128871+0000","last_undegraded":"2021-03-24T21:55:52.128871+0000","last_fullsized":"2021-03-24T21:55:52.128871+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:48.541415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:54:48.541415+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"89'27","reported_seq":"175","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.137871+0000","last_change":"2021-03-24T21:55:41.337301+0000","last_active":"2021-03-24T21:55:52.137871+0000","last_peered":"2021-03-24T21:55:52.137871+0000","last_clean":"2021-03-24T21:55:52.137871+0000","last_became_active":"2021-03-24T21:51:47.688995+0000","last_became_peered":"2021-03-24T21:51:47.688995+0000","last_unstale":"2021-03-24T21:55:52.137871+0000","last_undegraded":"2021-03-24T21:55:52.137871+0000","last_fullsized":"2021-03-24T21:55:52.137871+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'27","last_scrub_stamp":"2021-03-24T21:55:41.337130+0000","last_deep_scrub":"89'27","last_deep_scrub_stamp":"2021-03-24T21:55:41.337130+0000","last_clean_scrub_stamp":"2021-03-24T21:55:41.337130+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"92","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.143940+0000","last_change":"2021-03-24T21:54:49.992150+0000","last_active":"2021-03-24T21:55:52.143940+0000","last_peered":"2021-03-24T21:55:52.143940+0000","last_clean":"2021-03-24T21:55:52.143940+0000","last_became_active":"2021-03-24T21:54:38.113849+0000","last_became_peered":"2021-03-24T21:54:38.113849+0000","last_unstale":"2021-03-24T21:55:52.143940+0000","last_undegraded":"2021-03-24T21:55:52.143940+0000","last_fullsized":"2021-03-24T21:55:52.143940+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:49.991966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:54:49.991966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"130","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.138182+0000","last_change":"2021-03-24T21:53:30.660884+0000","last_active":"2021-03-24T21:55:52.138182+0000","last_peered":"2021-03-24T21:55:52.138182+0000","last_clean":"2021-03-24T21:55:52.138182+0000","last_became_active":"2021-03-24T21:51:49.409745+0000","last_became_peered":"2021-03-24T21:51:49.409745+0000","last_unstale":"2021-03-24T21:55:52.138182+0000","last_undegraded":"2021-03-24T21:55:52.138182+0000","last_fullsized":"2021-03-24T21:55:52.138182+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:30.660636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:30.660636+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":"149","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.128118+0000","last_change":"2021-03-24T21:55:32.566923+0000","last_active":"2021-03-24T21:55:52.128118+0000","last_peered":"2021-03-24T21:55:52.128118+0000","last_clean":"2021-03-24T21:55:52.128118+0000","last_became_active":"2021-03-24T21:54:38.145870+0000","last_became_peered":"2021-03-24T21:54:38.145870+0000","last_unstale":"2021-03-24T21:55:52.128118+0000","last_undegraded":"2021-03-24T21:55:52.128118+0000","last_fullsized":"2021-03-24T21:55:52.128118+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:32.566747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:32.566747+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"69'17","reported_seq":"106","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.137107+0000","last_change":"2021-03-24T21:55:32.214663+0000","last_active":"2021-03-24T21:55:52.137107+0000","last_peered":"2021-03-24T21:55:52.137107+0000","last_clean":"2021-03-24T21:55:52.137107+0000","last_became_active":"2021-03-24T21:55:10.101198+0000","last_became_peered":"2021-03-24T21:55:10.101198+0000","last_unstale":"2021-03-24T21:55:52.137107+0000","last_undegraded":"2021-03-24T21:55:52.137107+0000","last_fullsized":"2021-03-24T21:55:52.137107+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'17","last_scrub_stamp":"2021-03-24T21:55:32.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:32.214502+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":37,"num_read_kb":18,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"130","reported_epoch":"184","state":"active+clean","last_fresh":"2021-03-24T21:55:53.146674+0000","last_change":"2021-03-24T21:53:27.877846+0000","last_active":"2021-03-24T21:55:53.146674+0000","last_peered":"2021-03-24T21:55:53.146674+0000","last_clean":"2021-03-24T21:55:53.146674+0000","last_became_active":"2021-03-24T21:51:51.762150+0000","last_became_peered":"2021-03-24T21:51:51.762150+0000","last_unstale":"2021-03-24T21:55:53.146674+0000","last_undegraded":"2021-03-24T21:55:53.146674+0000","last_fullsized":"2021-03-24T21:55:53.146674+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:27.877627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:27.877627+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"102","reported_epoch":"187","state":"active+clean","last_fresh":"2021-03-24T21:55:56.110220+0000","last_change":"2021-03-24T21:55:26.087761+0000","last_active":"2021-03-24T21:55:56.110220+0000","last_peered":"2021-03-24T21:55:56.110220+0000","last_clean":"2021-03-24T21:55:56.110220+0000","last_became_active":"2021-03-24T21:54:24.187568+0000","last_became_peered":"2021-03-24T21:54:24.187568+0000","last_unstale":"2021-03-24T21:55:56.110220+0000","last_undegraded":"2021-03-24T21:55:56.110220+0000","last_fullsized":"2021-03-24T21:55:56.110220+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:26.087584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:55:26.087584+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":"139","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.137839+0000","last_change":"2021-03-24T21:54:48.394336+0000","last_active":"2021-03-24T21:55:52.137839+0000","last_peered":"2021-03-24T21:55:52.137839+0000","last_clean":"2021-03-24T21:55:52.137839+0000","last_became_active":"2021-03-24T21:51:45.379789+0000","last_became_peered":"2021-03-24T21:51:45.379789+0000","last_unstale":"2021-03-24T21:55:52.137839+0000","last_undegraded":"2021-03-24T21:55:52.137839+0000","last_fullsized":"2021-03-24T21:55:52.137839+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:48.394164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:54:48.394164+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"69'12","reported_seq":"253","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.128182+0000","last_change":"2021-03-24T21:54:39.142185+0000","last_active":"2021-03-24T21:55:52.128182+0000","last_peered":"2021-03-24T21:55:52.128182+0000","last_clean":"2021-03-24T21:55:52.128182+0000","last_became_active":"2021-03-24T21:54:38.134132+0000","last_became_peered":"2021-03-24T21:54:38.134132+0000","last_unstale":"2021-03-24T21:55:52.128182+0000","last_undegraded":"2021-03-24T21:55:52.128182+0000","last_fullsized":"2021-03-24T21:55:52.128182+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2021-03-24T21:54:14.574739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:54:14.574739+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":12,"num_write":12,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":3756,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"138","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.134185+0000","last_change":"2021-03-24T21:54:24.204479+0000","last_active":"2021-03-24T21:55:52.134185+0000","last_peered":"2021-03-24T21:55:52.134185+0000","last_clean":"2021-03-24T21:55:52.134185+0000","last_became_active":"2021-03-24T21:54:24.202686+0000","last_became_peered":"2021-03-24T21:54:24.202686+0000","last_unstale":"2021-03-24T21:55:52.134185+0000","last_undegraded":"2021-03-24T21:55:52.134185+0000","last_fullsized":"2021-03-24T21:55:52.134185+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:47.540793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:47.540793+0000","last_clean_scrub_stamp":"2021-03-24T21:53:47.540793+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"146","reported_epoch":"184","state":"active+clean","last_fresh":"2021-03-24T21:55:53.145884+0000","last_change":"2021-03-24T21:55:23.872913+0000","last_active":"2021-03-24T21:55:53.145884+0000","last_peered":"2021-03-24T21:55:53.145884+0000","last_clean":"2021-03-24T21:55:53.145884+0000","last_became_active":"2021-03-24T21:54:24.194423+0000","last_became_peered":"2021-03-24T21:54:24.194423+0000","last_unstale":"2021-03-24T21:55:53.145884+0000","last_undegraded":"2021-03-24T21:55:53.145884+0000","last_fullsized":"2021-03-24T21:55:53.145884+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:23.872742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:55:23.872742+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":"151","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.143526+0000","last_change":"2021-03-24T21:55:26.503047+0000","last_active":"2021-03-24T21:55:52.143526+0000","last_peered":"2021-03-24T21:55:52.143526+0000","last_clean":"2021-03-24T21:55:52.143526+0000","last_became_active":"2021-03-24T21:54:38.121323+0000","last_became_peered":"2021-03-24T21:54:38.121323+0000","last_unstale":"2021-03-24T21:55:52.143526+0000","last_undegraded":"2021-03-24T21:55:52.143526+0000","last_fullsized":"2021-03-24T21:55:52.143526+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:26.502895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:26.502895+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"86'22","reported_seq":"198","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.129064+0000","last_change":"2021-03-24T21:54:51.585204+0000","last_active":"2021-03-24T21:55:52.129064+0000","last_peered":"2021-03-24T21:55:52.129064+0000","last_clean":"2021-03-24T21:55:52.129064+0000","last_became_active":"2021-03-24T21:54:24.188790+0000","last_became_peered":"2021-03-24T21:54:24.188790+0000","last_unstale":"2021-03-24T21:55:52.129064+0000","last_undegraded":"2021-03-24T21:55:52.129064+0000","last_fullsized":"2021-03-24T21:55:52.129064+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'22","last_scrub_stamp":"2021-03-24T21:54:51.585058+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2021-03-24T21:52:54.597505+0000","last_clean_scrub_stamp":"2021-03-24T21:54:51.585058+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":16,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":580,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"142","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.133133+0000","last_change":"2021-03-24T21:54:54.371278+0000","last_active":"2021-03-24T21:55:52.133133+0000","last_peered":"2021-03-24T21:55:52.133133+0000","last_clean":"2021-03-24T21:55:52.133133+0000","last_became_active":"2021-03-24T21:54:38.126662+0000","last_became_peered":"2021-03-24T21:54:38.126662+0000","last_unstale":"2021-03-24T21:55:52.133133+0000","last_undegraded":"2021-03-24T21:55:52.133133+0000","last_fullsized":"2021-03-24T21:55:52.133133+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:54.371106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:54:54.371106+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"140","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.133696+0000","last_change":"2021-03-24T21:54:38.129428+0000","last_active":"2021-03-24T21:55:52.133696+0000","last_peered":"2021-03-24T21:55:52.133696+0000","last_clean":"2021-03-24T21:55:52.133696+0000","last_became_active":"2021-03-24T21:54:38.128067+0000","last_became_peered":"2021-03-24T21:54:38.128067+0000","last_unstale":"2021-03-24T21:55:52.133696+0000","last_undegraded":"2021-03-24T21:55:52.133696+0000","last_fullsized":"2021-03-24T21:55:52.133696+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:31.665434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:31.665434+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":"136","reported_epoch":"184","state":"active+clean","last_fresh":"2021-03-24T21:55:53.146273+0000","last_change":"2021-03-24T21:53:08.841487+0000","last_active":"2021-03-24T21:55:53.146273+0000","last_peered":"2021-03-24T21:55:53.146273+0000","last_clean":"2021-03-24T21:55:53.146273+0000","last_became_active":"2021-03-24T21:51:45.379383+0000","last_became_peered":"2021-03-24T21:51:45.379383+0000","last_unstale":"2021-03-24T21:55:53.146273+0000","last_undegraded":"2021-03-24T21:55:53.146273+0000","last_fullsized":"2021-03-24T21:55:53.146273+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:08.841310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:08.841310+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"69'15","reported_seq":"151","reported_epoch":"186","state":"active+clean","last_fresh":"2021-03-24T21:55:55.127716+0000","last_change":"2021-03-24T21:54:38.182009+0000","last_active":"2021-03-24T21:55:55.127716+0000","last_peered":"2021-03-24T21:55:55.127716+0000","last_clean":"2021-03-24T21:55:55.127716+0000","last_became_active":"2021-03-24T21:54:38.122707+0000","last_became_peered":"2021-03-24T21:54:38.122707+0000","last_unstale":"2021-03-24T21:55:55.127716+0000","last_undegraded":"2021-03-24T21:55:55.127716+0000","last_fullsized":"2021-03-24T21:55:55.127716+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:54:06.651102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:54:06.651102+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":34,"num_read_kb":16,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"134","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.128823+0000","last_change":"2021-03-24T21:55:22.555707+0000","last_active":"2021-03-24T21:55:52.128823+0000","last_peered":"2021-03-24T21:55:52.128823+0000","last_clean":"2021-03-24T21:55:52.128823+0000","last_became_active":"2021-03-24T21:51:51.764871+0000","last_became_peered":"2021-03-24T21:51:51.764871+0000","last_unstale":"2021-03-24T21:55:52.128823+0000","last_undegraded":"2021-03-24T21:55:52.128823+0000","last_fullsized":"2021-03-24T21:55:52.128823+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:22.555624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:23.707914+0000","last_clean_scrub_stamp":"2021-03-24T21:55:22.555624+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"92","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.144607+0000","last_change":"2021-03-24T21:55:40.456272+0000","last_active":"2021-03-24T21:55:52.144607+0000","last_peered":"2021-03-24T21:55:52.144607+0000","last_clean":"2021-03-24T21:55:52.144607+0000","last_became_active":"2021-03-24T21:54:38.120989+0000","last_became_peered":"2021-03-24T21:54:38.120989+0000","last_unstale":"2021-03-24T21:55:52.144607+0000","last_undegraded":"2021-03-24T21:55:52.144607+0000","last_fullsized":"2021-03-24T21:55:52.144607+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:40.456101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:55:40.456101+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":"153","reported_epoch":"187","state":"active+clean","last_fresh":"2021-03-24T21:55:56.111056+0000","last_change":"2021-03-24T21:55:42.135567+0000","last_active":"2021-03-24T21:55:56.111056+0000","last_peered":"2021-03-24T21:55:56.111056+0000","last_clean":"2021-03-24T21:55:56.111056+0000","last_became_active":"2021-03-24T21:54:24.181343+0000","last_became_peered":"2021-03-24T21:54:24.181343+0000","last_unstale":"2021-03-24T21:55:56.111056+0000","last_undegraded":"2021-03-24T21:55:56.111056+0000","last_fullsized":"2021-03-24T21:55:56.111056+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:42.135502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:42.135502+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"69'15","reported_seq":"227","reported_epoch":"187","state":"active+clean","last_fresh":"2021-03-24T21:55:56.112721+0000","last_change":"2021-03-24T21:55:37.076650+0000","last_active":"2021-03-24T21:55:56.112721+0000","last_peered":"2021-03-24T21:55:56.112721+0000","last_clean":"2021-03-24T21:55:56.112721+0000","last_became_active":"2021-03-24T21:54:38.118330+0000","last_became_peered":"2021-03-24T21:54:38.118330+0000","last_unstale":"2021-03-24T21:55:56.112721+0000","last_undegraded":"2021-03-24T21:55:56.112721+0000","last_fullsized":"2021-03-24T21:55:56.112721+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:55:37.076511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:37.076511+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"133","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.136767+0000","last_change":"2021-03-24T21:54:44.392954+0000","last_active":"2021-03-24T21:55:52.136767+0000","last_peered":"2021-03-24T21:55:52.136767+0000","last_clean":"2021-03-24T21:55:52.136767+0000","last_became_active":"2021-03-24T21:51:51.756584+0000","last_became_peered":"2021-03-24T21:51:51.756584+0000","last_unstale":"2021-03-24T21:55:52.136767+0000","last_undegraded":"2021-03-24T21:55:52.136767+0000","last_fullsized":"2021-03-24T21:55:52.136767+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:44.392808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:54:44.392808+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"109'4","reported_seq":"202","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.144217+0000","last_change":"2021-03-24T21:55:29.541436+0000","last_active":"2021-03-24T21:55:52.144217+0000","last_peered":"2021-03-24T21:55:52.144217+0000","last_clean":"2021-03-24T21:55:52.144217+0000","last_became_active":"2021-03-24T21:54:38.130026+0000","last_became_peered":"2021-03-24T21:54:38.130026+0000","last_unstale":"2021-03-24T21:55:52.144217+0000","last_undegraded":"2021-03-24T21:55:52.144217+0000","last_fullsized":"2021-03-24T21:55:52.144217+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'4","last_scrub_stamp":"2021-03-24T21:55:29.541312+0000","last_deep_scrub":"86'3","last_deep_scrub_stamp":"2021-03-24T21:53:08.815163+0000","last_clean_scrub_stamp":"2021-03-24T21:55:29.541312+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":"136","reported_epoch":"184","state":"active+clean","last_fresh":"2021-03-24T21:55:53.146005+0000","last_change":"2021-03-24T21:53:15.868395+0000","last_active":"2021-03-24T21:55:53.146005+0000","last_peered":"2021-03-24T21:55:53.146005+0000","last_clean":"2021-03-24T21:55:53.146005+0000","last_became_active":"2021-03-24T21:51:45.579271+0000","last_became_peered":"2021-03-24T21:51:45.579271+0000","last_unstale":"2021-03-24T21:55:53.146005+0000","last_undegraded":"2021-03-24T21:55:53.146005+0000","last_fullsized":"2021-03-24T21:55:53.146005+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:15.868303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:15.868303+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"69'14","reported_seq":"193","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.132767+0000","last_change":"2021-03-24T21:55:25.582566+0000","last_active":"2021-03-24T21:55:52.132767+0000","last_peered":"2021-03-24T21:55:52.132767+0000","last_clean":"2021-03-24T21:55:52.132767+0000","last_became_active":"2021-03-24T21:54:38.143783+0000","last_became_peered":"2021-03-24T21:54:38.143783+0000","last_unstale":"2021-03-24T21:55:52.132767+0000","last_undegraded":"2021-03-24T21:55:52.132767+0000","last_fullsized":"2021-03-24T21:55:52.132767+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'14","last_scrub_stamp":"2021-03-24T21:55:25.582388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:25.582388+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":27,"num_read_kb":14,"num_write":14,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"138","reported_epoch":"187","state":"active+clean","last_fresh":"2021-03-24T21:55:56.112276+0000","last_change":"2021-03-24T21:54:13.291633+0000","last_active":"2021-03-24T21:55:56.112276+0000","last_peered":"2021-03-24T21:55:56.112276+0000","last_clean":"2021-03-24T21:55:56.112276+0000","last_became_active":"2021-03-24T21:51:51.762876+0000","last_became_peered":"2021-03-24T21:51:51.762876+0000","last_unstale":"2021-03-24T21:55:56.112276+0000","last_undegraded":"2021-03-24T21:55:56.112276+0000","last_fullsized":"2021-03-24T21:55:56.112276+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:13.291461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:54:13.291461+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"86'3","reported_seq":"188","reported_epoch":"184","state":"active+clean","last_fresh":"2021-03-24T21:55:53.133703+0000","last_change":"2021-03-24T21:55:01.577952+0000","last_active":"2021-03-24T21:55:53.133703+0000","last_peered":"2021-03-24T21:55:53.133703+0000","last_clean":"2021-03-24T21:55:53.133703+0000","last_became_active":"2021-03-24T21:51:49.419797+0000","last_became_peered":"2021-03-24T21:51:49.419797+0000","last_unstale":"2021-03-24T21:55:53.133703+0000","last_undegraded":"2021-03-24T21:55:53.133703+0000","last_fullsized":"2021-03-24T21:55:53.133703+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'3","last_scrub_stamp":"2021-03-24T21:55:01.577769+0000","last_deep_scrub":"86'3","last_deep_scrub_stamp":"2021-03-24T21:52:56.678111+0000","last_clean_scrub_stamp":"2021-03-24T21:55:01.577769+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":"139","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.135278+0000","last_change":"2021-03-24T21:54:05.473528+0000","last_active":"2021-03-24T21:55:52.135278+0000","last_peered":"2021-03-24T21:55:52.135278+0000","last_clean":"2021-03-24T21:55:52.135278+0000","last_became_active":"2021-03-24T21:51:45.381516+0000","last_became_peered":"2021-03-24T21:51:45.381516+0000","last_unstale":"2021-03-24T21:55:52.135278+0000","last_undegraded":"2021-03-24T21:55:52.135278+0000","last_fullsized":"2021-03-24T21:55:52.135278+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:05.473348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:05.473348+0000","last_clean_scrub_stamp":"2021-03-24T21:54:05.473348+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"69'13","reported_seq":"164","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.144481+0000","last_change":"2021-03-24T21:55:31.561861+0000","last_active":"2021-03-24T21:55:52.144481+0000","last_peered":"2021-03-24T21:55:52.144481+0000","last_clean":"2021-03-24T21:55:52.144481+0000","last_became_active":"2021-03-24T21:51:47.689288+0000","last_became_peered":"2021-03-24T21:51:47.689288+0000","last_unstale":"2021-03-24T21:55:52.144481+0000","last_undegraded":"2021-03-24T21:55:52.144481+0000","last_fullsized":"2021-03-24T21:55:52.144481+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2021-03-24T21:55:31.561663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:31.561663+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"137","reported_epoch":"187","state":"active+clean","last_fresh":"2021-03-24T21:55:56.125093+0000","last_change":"2021-03-24T21:55:56.125093+0000","last_active":"2021-03-24T21:55:56.125093+0000","last_peered":"2021-03-24T21:55:56.125093+0000","last_clean":"2021-03-24T21:55:56.125093+0000","last_became_active":"2021-03-24T21:51:51.754085+0000","last_became_peered":"2021-03-24T21:51:51.754085+0000","last_unstale":"2021-03-24T21:55:56.125093+0000","last_undegraded":"2021-03-24T21:55:56.125093+0000","last_fullsized":"2021-03-24T21:55:56.125093+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:56.125019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:55:56.125019+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"145","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.130352+0000","last_change":"2021-03-24T21:54:55.493891+0000","last_active":"2021-03-24T21:55:52.130352+0000","last_peered":"2021-03-24T21:55:52.130352+0000","last_clean":"2021-03-24T21:55:52.130352+0000","last_became_active":"2021-03-24T21:54:38.129390+0000","last_became_peered":"2021-03-24T21:54:38.129390+0000","last_unstale":"2021-03-24T21:55:52.130352+0000","last_undegraded":"2021-03-24T21:55:52.130352+0000","last_fullsized":"2021-03-24T21:55:52.130352+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:55.493713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:55.493713+0000","last_clean_scrub_stamp":"2021-03-24T21:54:55.493713+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":"150","reported_epoch":"184","state":"active+clean","last_fresh":"2021-03-24T21:55:53.145712+0000","last_change":"2021-03-24T21:55:31.912637+0000","last_active":"2021-03-24T21:55:53.145712+0000","last_peered":"2021-03-24T21:55:53.145712+0000","last_clean":"2021-03-24T21:55:53.145712+0000","last_became_active":"2021-03-24T21:54:24.196290+0000","last_became_peered":"2021-03-24T21:54:24.196290+0000","last_unstale":"2021-03-24T21:55:53.145712+0000","last_undegraded":"2021-03-24T21:55:53.145712+0000","last_fullsized":"2021-03-24T21:55:53.145712+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:31.912451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:31.912451+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"69'16","reported_seq":"167","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.137985+0000","last_change":"2021-03-24T21:54:19.588964+0000","last_active":"2021-03-24T21:55:52.137985+0000","last_peered":"2021-03-24T21:55:52.137985+0000","last_clean":"2021-03-24T21:55:52.137985+0000","last_became_active":"2021-03-24T21:51:47.687595+0000","last_became_peered":"2021-03-24T21:51:47.687595+0000","last_unstale":"2021-03-24T21:55:52.137985+0000","last_undegraded":"2021-03-24T21:55:52.137985+0000","last_fullsized":"2021-03-24T21:55:52.137985+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T21:54:19.588797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:54:19.588797+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":38,"num_read_kb":18,"num_write":17,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"139","reported_epoch":"184","state":"active+clean","last_fresh":"2021-03-24T21:55:53.148650+0000","last_change":"2021-03-24T21:54:24.188745+0000","last_active":"2021-03-24T21:55:53.148650+0000","last_peered":"2021-03-24T21:55:53.148650+0000","last_clean":"2021-03-24T21:55:53.148650+0000","last_became_active":"2021-03-24T21:54:24.188386+0000","last_became_peered":"2021-03-24T21:54:24.188386+0000","last_unstale":"2021-03-24T21:55:53.148650+0000","last_undegraded":"2021-03-24T21:55:53.148650+0000","last_fullsized":"2021-03-24T21:55:53.148650+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:24.913238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:24.913238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"147","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.143412+0000","last_change":"2021-03-24T21:55:27.547657+0000","last_active":"2021-03-24T21:55:52.143412+0000","last_peered":"2021-03-24T21:55:52.143412+0000","last_clean":"2021-03-24T21:55:52.143412+0000","last_became_active":"2021-03-24T21:54:38.128409+0000","last_became_peered":"2021-03-24T21:54:38.128409+0000","last_unstale":"2021-03-24T21:55:52.143412+0000","last_undegraded":"2021-03-24T21:55:52.143412+0000","last_fullsized":"2021-03-24T21:55:52.143412+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:27.547493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:55:27.547493+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":"98","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.134394+0000","last_change":"2021-03-24T21:54:52.385672+0000","last_active":"2021-03-24T21:55:52.134394+0000","last_peered":"2021-03-24T21:55:52.134394+0000","last_clean":"2021-03-24T21:55:52.134394+0000","last_became_active":"2021-03-24T21:54:24.201388+0000","last_became_peered":"2021-03-24T21:54:24.201388+0000","last_unstale":"2021-03-24T21:55:52.134394+0000","last_undegraded":"2021-03-24T21:55:52.134394+0000","last_fullsized":"2021-03-24T21:55:52.134394+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:52.385567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:54:52.385567+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"69'7","reported_seq":"163","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.130179+0000","last_change":"2021-03-24T21:54:39.199780+0000","last_active":"2021-03-24T21:55:52.130179+0000","last_peered":"2021-03-24T21:55:52.130179+0000","last_clean":"2021-03-24T21:55:52.130179+0000","last_became_active":"2021-03-24T21:54:38.144098+0000","last_became_peered":"2021-03-24T21:54:38.144098+0000","last_unstale":"2021-03-24T21:55:52.130179+0000","last_undegraded":"2021-03-24T21:55:52.130179+0000","last_fullsized":"2021-03-24T21:55:52.130179+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'7","last_scrub_stamp":"2021-03-24T21:53:13.645928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:13.645928+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":7,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":158,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"183'134","reported_seq":"449","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.146512+0000","last_change":"2021-03-24T21:55:52.146512+0000","last_active":"2021-03-24T21:55:52.146512+0000","last_peered":"2021-03-24T21:55:52.146512+0000","last_clean":"2021-03-24T21:55:52.146512+0000","last_became_active":"2021-03-24T21:54:38.117586+0000","last_became_peered":"2021-03-24T21:54:38.117586+0000","last_unstale":"2021-03-24T21:55:52.146512+0000","last_undegraded":"2021-03-24T21:55:52.146512+0000","last_fullsized":"2021-03-24T21:55:52.146512+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":134,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17462341,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":111,"num_read_kb":113344,"num_write":55,"num_write_kb":24919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":6771983,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2e","length":"1"},{"start":"3d","length":"2"},{"start":"42","length":"2"},{"start":"46","length":"1"},{"start":"4a","length":"1"}]},{"pgid":"6.a","version":"0'0","reported_seq":"103","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.135501+0000","last_change":"2021-03-24T21:55:10.097929+0000","last_active":"2021-03-24T21:55:52.135501+0000","last_peered":"2021-03-24T21:55:52.135501+0000","last_clean":"2021-03-24T21:55:52.135501+0000","last_became_active":"2021-03-24T21:55:10.097570+0000","last_became_peered":"2021-03-24T21:55:10.097570+0000","last_unstale":"2021-03-24T21:55:52.135501+0000","last_undegraded":"2021-03-24T21:55:52.135501+0000","last_fullsized":"2021-03-24T21:55:52.135501+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:55.067965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:54:55.067965+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"89'4","reported_seq":"141","reported_epoch":"184","state":"active+clean","last_fresh":"2021-03-24T21:55:53.148168+0000","last_change":"2021-03-24T21:53:19.926732+0000","last_active":"2021-03-24T21:55:53.148168+0000","last_peered":"2021-03-24T21:55:53.148168+0000","last_clean":"2021-03-24T21:55:53.148168+0000","last_became_active":"2021-03-24T21:51:45.385110+0000","last_became_peered":"2021-03-24T21:51:45.385110+0000","last_unstale":"2021-03-24T21:55:53.148168+0000","last_undegraded":"2021-03-24T21:55:53.148168+0000","last_fullsized":"2021-03-24T21:55:53.148168+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'4","last_scrub_stamp":"2021-03-24T21:53:19.926637+0000","last_deep_scrub":"89'4","last_deep_scrub_stamp":"2021-03-24T21:53:19.926637+0000","last_clean_scrub_stamp":"2021-03-24T21:53:19.926637+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"97'28","reported_seq":"212","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.142168+0000","last_change":"2021-03-24T21:54:24.220625+0000","last_active":"2021-03-24T21:55:52.142168+0000","last_peered":"2021-03-24T21:55:52.142168+0000","last_clean":"2021-03-24T21:55:52.142168+0000","last_became_active":"2021-03-24T21:54:24.190817+0000","last_became_peered":"2021-03-24T21:54:24.190817+0000","last_unstale":"2021-03-24T21:55:52.142168+0000","last_undegraded":"2021-03-24T21:55:52.142168+0000","last_fullsized":"2021-03-24T21:55:52.142168+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'28","last_scrub_stamp":"2021-03-24T21:54:19.157265+0000","last_deep_scrub":"97'28","last_deep_scrub_stamp":"2021-03-24T21:54:19.157265+0000","last_clean_scrub_stamp":"2021-03-24T21:54:19.157265+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":20,"num_write":19,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"109'4","reported_seq":"214","reported_epoch":"184","state":"active+clean","last_fresh":"2021-03-24T21:55:53.910183+0000","last_change":"2021-03-24T21:54:38.223920+0000","last_active":"2021-03-24T21:55:53.910183+0000","last_peered":"2021-03-24T21:55:53.910183+0000","last_clean":"2021-03-24T21:55:53.910183+0000","last_became_active":"2021-03-24T21:54:38.125497+0000","last_became_peered":"2021-03-24T21:54:38.125497+0000","last_unstale":"2021-03-24T21:55:53.910183+0000","last_undegraded":"2021-03-24T21:55:53.910183+0000","last_fullsized":"2021-03-24T21:55:53.910183+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2021-03-24T21:54:04.706153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:54:04.706153+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"178'56","reported_seq":"313","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.131520+0000","last_change":"2021-03-24T21:55:52.131520+0000","last_active":"2021-03-24T21:55:52.131520+0000","last_peered":"2021-03-24T21:55:52.131520+0000","last_clean":"2021-03-24T21:55:52.131520+0000","last_became_active":"2021-03-24T21:54:22.167632+0000","last_became_peered":"2021-03-24T21:54:22.167632+0000","last_unstale":"2021-03-24T21:55:52.131520+0000","last_undegraded":"2021-03-24T21:55:52.131520+0000","last_fullsized":"2021-03-24T21:55:52.131520+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2672313,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":6,"num_write":31,"num_write_kb":10011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2e","length":"1"},{"start":"3d","length":"2"},{"start":"42","length":"2"},{"start":"46","length":"1"},{"start":"4a","length":"1"}]},{"pgid":"6.b","version":"0'0","reported_seq":"143","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.133884+0000","last_change":"2021-03-24T21:55:24.548918+0000","last_active":"2021-03-24T21:55:52.133884+0000","last_peered":"2021-03-24T21:55:52.133884+0000","last_clean":"2021-03-24T21:55:52.133884+0000","last_became_active":"2021-03-24T21:54:38.129824+0000","last_became_peered":"2021-03-24T21:54:38.129824+0000","last_unstale":"2021-03-24T21:55:52.133884+0000","last_undegraded":"2021-03-24T21:55:52.133884+0000","last_fullsized":"2021-03-24T21:55:52.133884+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:24.548819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:22.724840+0000","last_clean_scrub_stamp":"2021-03-24T21:55:24.548819+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":"150","reported_epoch":"184","state":"active+clean","last_fresh":"2021-03-24T21:55:53.147332+0000","last_change":"2021-03-24T21:54:38.109846+0000","last_active":"2021-03-24T21:55:53.147332+0000","last_peered":"2021-03-24T21:55:53.147332+0000","last_clean":"2021-03-24T21:55:53.147332+0000","last_became_active":"2021-03-24T21:54:38.109439+0000","last_became_peered":"2021-03-24T21:54:38.109439+0000","last_unstale":"2021-03-24T21:55:53.147332+0000","last_undegraded":"2021-03-24T21:55:53.147332+0000","last_fullsized":"2021-03-24T21:55:53.147332+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:51.651164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:51.651164+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"69'15","reported_seq":"193","reported_epoch":"187","state":"active+clean","last_fresh":"2021-03-24T21:55:56.110802+0000","last_change":"2021-03-24T21:54:41.209644+0000","last_active":"2021-03-24T21:55:56.110802+0000","last_peered":"2021-03-24T21:55:56.110802+0000","last_clean":"2021-03-24T21:55:56.110802+0000","last_became_active":"2021-03-24T21:54:38.139863+0000","last_became_peered":"2021-03-24T21:54:38.139863+0000","last_unstale":"2021-03-24T21:55:56.110802+0000","last_undegraded":"2021-03-24T21:55:56.110802+0000","last_fullsized":"2021-03-24T21:55:56.110802+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:53:37.334345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:37.334345+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":19,"num_write":17,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":692,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"148","reported_epoch":"186","state":"active+clean","last_fresh":"2021-03-24T21:55:55.127571+0000","last_change":"2021-03-24T21:55:26.017611+0000","last_active":"2021-03-24T21:55:55.127571+0000","last_peered":"2021-03-24T21:55:55.127571+0000","last_clean":"2021-03-24T21:55:55.127571+0000","last_became_active":"2021-03-24T21:54:38.124317+0000","last_became_peered":"2021-03-24T21:54:38.124317+0000","last_unstale":"2021-03-24T21:55:55.127571+0000","last_undegraded":"2021-03-24T21:55:55.127571+0000","last_fullsized":"2021-03-24T21:55:55.127571+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:26.017426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:55:26.017426+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"183'148","reported_seq":"435","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.146936+0000","last_change":"2021-03-24T21:55:52.146936+0000","last_active":"2021-03-24T21:55:52.146936+0000","last_peered":"2021-03-24T21:55:52.146936+0000","last_clean":"2021-03-24T21:55:52.146936+0000","last_became_active":"2021-03-24T21:54:38.137395+0000","last_became_peered":"2021-03-24T21:54:38.137395+0000","last_unstale":"2021-03-24T21:55:52.146936+0000","last_undegraded":"2021-03-24T21:55:52.146936+0000","last_fullsized":"2021-03-24T21:55:52.146936+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":148,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7966518,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":73,"num_read_kb":56272,"num_write":63,"num_write_kb":22687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4396481,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2e","length":"1"},{"start":"3d","length":"2"},{"start":"42","length":"2"},{"start":"46","length":"1"},{"start":"4a","length":"1"}]},{"pgid":"6.8","version":"0'0","reported_seq":"135","reported_epoch":"184","state":"active+clean","last_fresh":"2021-03-24T21:55:53.145242+0000","last_change":"2021-03-24T21:54:46.859206+0000","last_active":"2021-03-24T21:55:53.145242+0000","last_peered":"2021-03-24T21:55:53.145242+0000","last_clean":"2021-03-24T21:55:53.145242+0000","last_became_active":"2021-03-24T21:51:51.762513+0000","last_became_peered":"2021-03-24T21:51:51.762513+0000","last_unstale":"2021-03-24T21:55:53.145242+0000","last_undegraded":"2021-03-24T21:55:53.145242+0000","last_fullsized":"2021-03-24T21:55:53.145242+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:46.859034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:54:46.859034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"139","reported_epoch":"186","state":"active+clean","last_fresh":"2021-03-24T21:55:55.127907+0000","last_change":"2021-03-24T21:54:51.202761+0000","last_active":"2021-03-24T21:55:55.127907+0000","last_peered":"2021-03-24T21:55:55.127907+0000","last_clean":"2021-03-24T21:55:55.127907+0000","last_became_active":"2021-03-24T21:51:49.417699+0000","last_became_peered":"2021-03-24T21:51:49.417699+0000","last_unstale":"2021-03-24T21:55:55.127907+0000","last_undegraded":"2021-03-24T21:55:55.127907+0000","last_fullsized":"2021-03-24T21:55:55.127907+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:51.202632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:54:51.202632+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":"150","reported_epoch":"184","state":"active+clean","last_fresh":"2021-03-24T21:55:53.146544+0000","last_change":"2021-03-24T21:55:38.974245+0000","last_active":"2021-03-24T21:55:53.146544+0000","last_peered":"2021-03-24T21:55:53.146544+0000","last_clean":"2021-03-24T21:55:53.146544+0000","last_became_active":"2021-03-24T21:54:24.189299+0000","last_became_peered":"2021-03-24T21:54:24.189299+0000","last_unstale":"2021-03-24T21:55:53.146544+0000","last_undegraded":"2021-03-24T21:55:53.146544+0000","last_fullsized":"2021-03-24T21:55:53.146544+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:38.974155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:38.974155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"69'23","reported_seq":"168","reported_epoch":"184","state":"active+clean","last_fresh":"2021-03-24T21:55:53.146465+0000","last_change":"2021-03-24T21:54:39.420930+0000","last_active":"2021-03-24T21:55:53.146465+0000","last_peered":"2021-03-24T21:55:53.146465+0000","last_clean":"2021-03-24T21:55:53.146465+0000","last_became_active":"2021-03-24T21:54:38.138701+0000","last_became_peered":"2021-03-24T21:54:38.138701+0000","last_unstale":"2021-03-24T21:55:53.146465+0000","last_undegraded":"2021-03-24T21:55:53.146465+0000","last_fullsized":"2021-03-24T21:55:53.146465+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'23","last_scrub_stamp":"2021-03-24T21:54:18.973113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:54:18.973113+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":43,"num_read_kb":24,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":732,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"187'83","reported_seq":"344","reported_epoch":"187","state":"active+clean","last_fresh":"2021-03-24T21:55:56.118098+0000","last_change":"2021-03-24T21:55:55.128276+0000","last_active":"2021-03-24T21:55:56.118098+0000","last_peered":"2021-03-24T21:55:56.118098+0000","last_clean":"2021-03-24T21:55:56.118098+0000","last_became_active":"2021-03-24T21:54:22.166423+0000","last_became_peered":"2021-03-24T21:54:22.166423+0000","last_unstale":"2021-03-24T21:55:56.118098+0000","last_undegraded":"2021-03-24T21:55:56.118098+0000","last_fullsized":"2021-03-24T21:55:56.118098+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":83,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6033680,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":22,"num_read_kb":13222,"num_write":34,"num_write_kb":8491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"3d","length":"2"},{"start":"42","length":"1"},{"start":"4a","length":"1"},{"start":"4f","length":"1"}]},{"pgid":"6.9","version":"0'0","reported_seq":"133","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.136065+0000","last_change":"2021-03-24T21:55:33.246334+0000","last_active":"2021-03-24T21:55:52.136065+0000","last_peered":"2021-03-24T21:55:52.136065+0000","last_clean":"2021-03-24T21:55:52.136065+0000","last_became_active":"2021-03-24T21:51:51.763077+0000","last_became_peered":"2021-03-24T21:51:51.763077+0000","last_unstale":"2021-03-24T21:55:52.136065+0000","last_undegraded":"2021-03-24T21:55:52.136065+0000","last_fullsized":"2021-03-24T21:55:52.136065+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:33.246205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:35.636299+0000","last_clean_scrub_stamp":"2021-03-24T21:55:33.246205+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"139","reported_epoch":"186","state":"active+clean","last_fresh":"2021-03-24T21:55:55.127808+0000","last_change":"2021-03-24T21:54:54.139755+0000","last_active":"2021-03-24T21:55:55.127808+0000","last_peered":"2021-03-24T21:55:55.127808+0000","last_clean":"2021-03-24T21:55:55.127808+0000","last_became_active":"2021-03-24T21:51:49.411921+0000","last_became_peered":"2021-03-24T21:51:49.411921+0000","last_unstale":"2021-03-24T21:55:55.127808+0000","last_undegraded":"2021-03-24T21:55:55.127808+0000","last_fullsized":"2021-03-24T21:55:55.127808+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:54.139655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:54:54.139655+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"130","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.132013+0000","last_change":"2021-03-24T21:55:10.102704+0000","last_active":"2021-03-24T21:55:52.132013+0000","last_peered":"2021-03-24T21:55:52.132013+0000","last_clean":"2021-03-24T21:55:52.132013+0000","last_became_active":"2021-03-24T21:55:10.102360+0000","last_became_peered":"2021-03-24T21:55:10.102360+0000","last_unstale":"2021-03-24T21:55:52.132013+0000","last_undegraded":"2021-03-24T21:55:52.132013+0000","last_fullsized":"2021-03-24T21:55:52.132013+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:00.911020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:54:00.911020+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"69'14","reported_seq":"187","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.133773+0000","last_change":"2021-03-24T21:54:39.837964+0000","last_active":"2021-03-24T21:55:52.133773+0000","last_peered":"2021-03-24T21:55:52.133773+0000","last_clean":"2021-03-24T21:55:52.133773+0000","last_became_active":"2021-03-24T21:54:38.138034+0000","last_became_peered":"2021-03-24T21:54:38.138034+0000","last_unstale":"2021-03-24T21:55:52.133773+0000","last_undegraded":"2021-03-24T21:55:52.133773+0000","last_fullsized":"2021-03-24T21:55:52.133773+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'14","last_scrub_stamp":"2021-03-24T21:53:31.653334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:53:31.653334+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":16,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":894,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"183'163","reported_seq":"463","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.147118+0000","last_change":"2021-03-24T21:55:52.147118+0000","last_active":"2021-03-24T21:55:52.147118+0000","last_peered":"2021-03-24T21:55:52.147118+0000","last_clean":"2021-03-24T21:55:52.147118+0000","last_became_active":"2021-03-24T21:54:38.124237+0000","last_became_peered":"2021-03-24T21:54:38.124237+0000","last_unstale":"2021-03-24T21:55:52.147118+0000","last_undegraded":"2021-03-24T21:55:52.147118+0000","last_fullsized":"2021-03-24T21:55:52.147118+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":163,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9983292,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":116,"num_read_kb":119735,"num_write":76,"num_write_kb":25212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":5158008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2e","length":"1"},{"start":"3d","length":"2"},{"start":"42","length":"2"},{"start":"46","length":"1"},{"start":"4a","length":"1"}]},{"pgid":"6.6","version":"0'0","reported_seq":"134","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.131123+0000","last_change":"2021-03-24T21:54:10.519515+0000","last_active":"2021-03-24T21:55:52.131123+0000","last_peered":"2021-03-24T21:55:52.131123+0000","last_clean":"2021-03-24T21:55:52.131123+0000","last_became_active":"2021-03-24T21:51:51.759514+0000","last_became_peered":"2021-03-24T21:51:51.759514+0000","last_unstale":"2021-03-24T21:55:52.131123+0000","last_undegraded":"2021-03-24T21:55:52.131123+0000","last_fullsized":"2021-03-24T21:55:52.131123+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:10.519411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:52:59.706511+0000","last_clean_scrub_stamp":"2021-03-24T21:54:10.519411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"129","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.135386+0000","last_change":"2021-03-24T21:55:10.100603+0000","last_active":"2021-03-24T21:55:52.135386+0000","last_peered":"2021-03-24T21:55:52.135386+0000","last_clean":"2021-03-24T21:55:52.135386+0000","last_became_active":"2021-03-24T21:55:10.100245+0000","last_became_peered":"2021-03-24T21:55:10.100245+0000","last_unstale":"2021-03-24T21:55:52.135386+0000","last_undegraded":"2021-03-24T21:55:52.135386+0000","last_fullsized":"2021-03-24T21:55:52.135386+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:03.195853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:06.800417+0000","last_clean_scrub_stamp":"2021-03-24T21:55:03.195853+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"89'2","reported_seq":"173","reported_epoch":"187","state":"active+clean","last_fresh":"2021-03-24T21:55:56.112312+0000","last_change":"2021-03-24T21:55:38.046157+0000","last_active":"2021-03-24T21:55:56.112312+0000","last_peered":"2021-03-24T21:55:56.112312+0000","last_clean":"2021-03-24T21:55:56.112312+0000","last_became_active":"2021-03-24T21:54:38.144013+0000","last_became_peered":"2021-03-24T21:54:38.144013+0000","last_unstale":"2021-03-24T21:55:56.112312+0000","last_undegraded":"2021-03-24T21:55:56.112312+0000","last_fullsized":"2021-03-24T21:55:56.112312+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2021-03-24T21:55:38.045901+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2021-03-24T21:53:28.219900+0000","last_clean_scrub_stamp":"2021-03-24T21:55:38.045901+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"69'25","reported_seq":"181","reported_epoch":"186","state":"active+clean","last_fresh":"2021-03-24T21:55:55.128256+0000","last_change":"2021-03-24T21:55:29.056321+0000","last_active":"2021-03-24T21:55:55.128256+0000","last_peered":"2021-03-24T21:55:55.128256+0000","last_clean":"2021-03-24T21:55:55.128256+0000","last_became_active":"2021-03-24T21:54:38.141478+0000","last_became_peered":"2021-03-24T21:54:38.141478+0000","last_unstale":"2021-03-24T21:55:55.128256+0000","last_undegraded":"2021-03-24T21:55:55.128256+0000","last_fullsized":"2021-03-24T21:55:55.128256+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'25","last_scrub_stamp":"2021-03-24T21:55:29.056144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:29.056144+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":53,"num_read_kb":30,"num_write":28,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"144","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.137098+0000","last_change":"2021-03-24T21:55:36.236833+0000","last_active":"2021-03-24T21:55:52.137098+0000","last_peered":"2021-03-24T21:55:52.137098+0000","last_clean":"2021-03-24T21:55:52.137098+0000","last_became_active":"2021-03-24T21:54:38.136859+0000","last_became_peered":"2021-03-24T21:54:38.136859+0000","last_unstale":"2021-03-24T21:55:52.137098+0000","last_undegraded":"2021-03-24T21:55:52.137098+0000","last_fullsized":"2021-03-24T21:55:52.137098+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:36.236594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:55:36.236594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"183'124","reported_seq":"411","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.142234+0000","last_change":"2021-03-24T21:55:52.142234+0000","last_active":"2021-03-24T21:55:52.142234+0000","last_peered":"2021-03-24T21:55:52.142234+0000","last_clean":"2021-03-24T21:55:52.142234+0000","last_became_active":"2021-03-24T21:54:38.137466+0000","last_became_peered":"2021-03-24T21:54:38.137466+0000","last_unstale":"2021-03-24T21:55:52.142234+0000","last_undegraded":"2021-03-24T21:55:52.142234+0000","last_fullsized":"2021-03-24T21:55:52.142234+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":124,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8537956,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":70,"num_read_kb":61088,"num_write":63,"num_write_kb":21999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4942492,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2e","length":"1"},{"start":"3d","length":"2"},{"start":"42","length":"2"},{"start":"46","length":"1"},{"start":"4a","length":"1"}]},{"pgid":"6.5","version":"0'0","reported_seq":"144","reported_epoch":"184","state":"active+clean","last_fresh":"2021-03-24T21:55:53.148720+0000","last_change":"2021-03-24T21:55:34.931146+0000","last_active":"2021-03-24T21:55:53.148720+0000","last_peered":"2021-03-24T21:55:53.148720+0000","last_clean":"2021-03-24T21:55:53.148720+0000","last_became_active":"2021-03-24T21:54:24.186957+0000","last_became_peered":"2021-03-24T21:54:24.186957+0000","last_unstale":"2021-03-24T21:55:53.148720+0000","last_undegraded":"2021-03-24T21:55:53.148720+0000","last_fullsized":"2021-03-24T21:55:53.148720+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:34.930967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:55:34.930967+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"61'1","reported_seq":"105","reported_epoch":"186","state":"active+clean","last_fresh":"2021-03-24T21:55:55.127749+0000","last_change":"2021-03-24T21:55:30.102312+0000","last_active":"2021-03-24T21:55:55.127749+0000","last_peered":"2021-03-24T21:55:55.127749+0000","last_clean":"2021-03-24T21:55:55.127749+0000","last_became_active":"2021-03-24T21:54:38.123139+0000","last_became_peered":"2021-03-24T21:54:38.123139+0000","last_unstale":"2021-03-24T21:55:55.127749+0000","last_undegraded":"2021-03-24T21:55:55.127749+0000","last_fullsized":"2021-03-24T21:55:55.127749+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2021-03-24T21:55:30.102193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:30.102193+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"147","reported_epoch":"186","state":"active+clean","last_fresh":"2021-03-24T21:55:55.127534+0000","last_change":"2021-03-24T21:53:55.497565+0000","last_active":"2021-03-24T21:55:55.127534+0000","last_peered":"2021-03-24T21:55:55.127534+0000","last_clean":"2021-03-24T21:55:55.127534+0000","last_became_active":"2021-03-24T21:51:40.351000+0000","last_became_peered":"2021-03-24T21:51:40.351000+0000","last_unstale":"2021-03-24T21:55:55.127534+0000","last_undegraded":"2021-03-24T21:55:55.127534+0000","last_fullsized":"2021-03-24T21:55:55.127534+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:55.497382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:39.286101+0000","last_clean_scrub_stamp":"2021-03-24T21:53:55.497382+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"89'22","reported_seq":"175","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.143640+0000","last_change":"2021-03-24T21:55:23.459422+0000","last_active":"2021-03-24T21:55:52.143640+0000","last_peered":"2021-03-24T21:55:52.143640+0000","last_clean":"2021-03-24T21:55:52.143640+0000","last_became_active":"2021-03-24T21:54:38.129459+0000","last_became_peered":"2021-03-24T21:54:38.129459+0000","last_unstale":"2021-03-24T21:55:52.143640+0000","last_undegraded":"2021-03-24T21:55:52.143640+0000","last_fullsized":"2021-03-24T21:55:52.143640+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'22","last_scrub_stamp":"2021-03-24T21:55:23.459251+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2021-03-24T21:53:19.717244+0000","last_clean_scrub_stamp":"2021-03-24T21:55:23.459251+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":36,"num_read_kb":21,"num_write":18,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":644,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"139","reported_epoch":"186","state":"active+clean","last_fresh":"2021-03-24T21:55:55.128448+0000","last_change":"2021-03-24T21:55:31.118872+0000","last_active":"2021-03-24T21:55:55.128448+0000","last_peered":"2021-03-24T21:55:55.128448+0000","last_clean":"2021-03-24T21:55:55.128448+0000","last_became_active":"2021-03-24T21:51:49.418848+0000","last_became_peered":"2021-03-24T21:51:49.418848+0000","last_unstale":"2021-03-24T21:55:55.128448+0000","last_undegraded":"2021-03-24T21:55:55.128448+0000","last_fullsized":"2021-03-24T21:55:55.128448+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:31.118766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:55:31.118766+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"183'106","reported_seq":"375","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.150692+0000","last_change":"2021-03-24T21:55:52.150692+0000","last_active":"2021-03-24T21:55:52.150692+0000","last_peered":"2021-03-24T21:55:52.150692+0000","last_clean":"2021-03-24T21:55:52.150692+0000","last_became_active":"2021-03-24T21:54:22.162608+0000","last_became_peered":"2021-03-24T21:54:22.162608+0000","last_unstale":"2021-03-24T21:55:52.150692+0000","last_undegraded":"2021-03-24T21:55:52.150692+0000","last_fullsized":"2021-03-24T21:55:52.150692+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":106,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3697507,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":80,"num_read_kb":89191,"num_write":41,"num_write_kb":14618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2e","length":"1"},{"start":"3d","length":"2"},{"start":"42","length":"2"},{"start":"46","length":"1"},{"start":"4a","length":"1"}]},{"pgid":"6.7","version":"0'0","reported_seq":"136","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.143448+0000","last_change":"2021-03-24T21:54:45.900904+0000","last_active":"2021-03-24T21:55:52.143448+0000","last_peered":"2021-03-24T21:55:52.143448+0000","last_clean":"2021-03-24T21:55:52.143448+0000","last_became_active":"2021-03-24T21:51:51.765236+0000","last_became_peered":"2021-03-24T21:51:51.765236+0000","last_unstale":"2021-03-24T21:55:52.143448+0000","last_undegraded":"2021-03-24T21:55:52.143448+0000","last_fullsized":"2021-03-24T21:55:52.143448+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:45.900726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:54:45.900726+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"164","reported_epoch":"184","state":"active+clean","last_fresh":"2021-03-24T21:55:53.145631+0000","last_change":"2021-03-24T21:54:38.110780+0000","last_active":"2021-03-24T21:55:53.145631+0000","last_peered":"2021-03-24T21:55:53.145631+0000","last_clean":"2021-03-24T21:55:53.145631+0000","last_became_active":"2021-03-24T21:54:38.110327+0000","last_became_peered":"2021-03-24T21:54:38.110327+0000","last_unstale":"2021-03-24T21:55:53.145631+0000","last_undegraded":"2021-03-24T21:55:53.145631+0000","last_fullsized":"2021-03-24T21:55:53.145631+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:30.906502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:49:51.529051+0000","last_clean_scrub_stamp":"2021-03-24T21:53:30.906502+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"149","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.129586+0000","last_change":"2021-03-24T21:55:33.534038+0000","last_active":"2021-03-24T21:55:52.129586+0000","last_peered":"2021-03-24T21:55:52.129586+0000","last_clean":"2021-03-24T21:55:52.129586+0000","last_became_active":"2021-03-24T21:54:24.199349+0000","last_became_peered":"2021-03-24T21:54:24.199349+0000","last_unstale":"2021-03-24T21:55:52.129586+0000","last_undegraded":"2021-03-24T21:55:52.129586+0000","last_fullsized":"2021-03-24T21:55:52.129586+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:33.533840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:33.533840+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"69'15","reported_seq":"159","reported_epoch":"187","state":"active+clean","last_fresh":"2021-03-24T21:55:56.111440+0000","last_change":"2021-03-24T21:55:30.166245+0000","last_active":"2021-03-24T21:55:56.111440+0000","last_peered":"2021-03-24T21:55:56.111440+0000","last_clean":"2021-03-24T21:55:56.111440+0000","last_became_active":"2021-03-24T21:54:24.194711+0000","last_became_peered":"2021-03-24T21:54:24.194711+0000","last_unstale":"2021-03-24T21:55:56.111440+0000","last_undegraded":"2021-03-24T21:55:56.111440+0000","last_fullsized":"2021-03-24T21:55:56.111440+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:55:30.166107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:30.166107+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":26,"num_read_kb":14,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":466,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"137","reported_epoch":"184","state":"active+clean","last_fresh":"2021-03-24T21:55:53.146413+0000","last_change":"2021-03-24T21:54:49.807761+0000","last_active":"2021-03-24T21:55:53.146413+0000","last_peered":"2021-03-24T21:55:53.146413+0000","last_clean":"2021-03-24T21:55:53.146413+0000","last_became_active":"2021-03-24T21:51:49.413251+0000","last_became_peered":"2021-03-24T21:51:49.413251+0000","last_unstale":"2021-03-24T21:55:53.146413+0000","last_undegraded":"2021-03-24T21:55:53.146413+0000","last_fullsized":"2021-03-24T21:55:53.146413+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:49.807585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:54:49.807585+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"185'58","reported_seq":"327","reported_epoch":"185","state":"active+clean","last_fresh":"2021-03-24T21:55:54.194482+0000","last_change":"2021-03-24T21:55:52.142802+0000","last_active":"2021-03-24T21:55:54.194482+0000","last_peered":"2021-03-24T21:55:54.194482+0000","last_clean":"2021-03-24T21:55:54.194482+0000","last_became_active":"2021-03-24T21:54:22.162598+0000","last_became_peered":"2021-03-24T21:54:22.162598+0000","last_unstale":"2021-03-24T21:55:54.194482+0000","last_undegraded":"2021-03-24T21:55:54.194482+0000","last_fullsized":"2021-03-24T21:55:54.194482+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":58,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5557148,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":16766,"num_write":35,"num_write_kb":13548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2e","length":"1"},{"start":"3d","length":"2"},{"start":"42","length":"2"},{"start":"46","length":"1"},{"start":"4a","length":"1"}]},{"pgid":"6.4","version":"0'0","reported_seq":"92","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.144055+0000","last_change":"2021-03-24T21:55:41.503921+0000","last_active":"2021-03-24T21:55:52.144055+0000","last_peered":"2021-03-24T21:55:52.144055+0000","last_clean":"2021-03-24T21:55:52.144055+0000","last_became_active":"2021-03-24T21:54:38.122320+0000","last_became_peered":"2021-03-24T21:54:38.122320+0000","last_unstale":"2021-03-24T21:55:52.144055+0000","last_undegraded":"2021-03-24T21:55:52.144055+0000","last_fullsized":"2021-03-24T21:55:52.144055+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:41.503758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:55:41.503758+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"139","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.135464+0000","last_change":"2021-03-24T21:54:18.594379+0000","last_active":"2021-03-24T21:55:52.135464+0000","last_peered":"2021-03-24T21:55:52.135464+0000","last_clean":"2021-03-24T21:55:52.135464+0000","last_became_active":"2021-03-24T21:51:45.376687+0000","last_became_peered":"2021-03-24T21:51:45.376687+0000","last_unstale":"2021-03-24T21:55:52.135464+0000","last_undegraded":"2021-03-24T21:55:52.135464+0000","last_fullsized":"2021-03-24T21:55:52.135464+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:18.594199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:54:18.594199+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"66'5","reported_seq":"409","reported_epoch":"185","state":"active+clean","last_fresh":"2021-03-24T21:55:54.168890+0000","last_change":"2021-03-24T21:54:38.495832+0000","last_active":"2021-03-24T21:55:54.168890+0000","last_peered":"2021-03-24T21:55:54.168890+0000","last_clean":"2021-03-24T21:55:54.168890+0000","last_became_active":"2021-03-24T21:54:38.125434+0000","last_became_peered":"2021-03-24T21:54:38.125434+0000","last_unstale":"2021-03-24T21:55:54.168890+0000","last_undegraded":"2021-03-24T21:55:54.168890+0000","last_fullsized":"2021-03-24T21:55:54.168890+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2021-03-24T21:54:07.669877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:39.286101+0000","last_clean_scrub_stamp":"2021-03-24T21:54:07.669877+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":247,"num_read_kb":242,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"69'16","reported_seq":"194","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.136874+0000","last_change":"2021-03-24T21:54:57.325888+0000","last_active":"2021-03-24T21:55:52.136874+0000","last_peered":"2021-03-24T21:55:52.136874+0000","last_clean":"2021-03-24T21:55:52.136874+0000","last_became_active":"2021-03-24T21:54:38.121714+0000","last_became_peered":"2021-03-24T21:54:38.121714+0000","last_unstale":"2021-03-24T21:55:52.136874+0000","last_undegraded":"2021-03-24T21:55:52.136874+0000","last_fullsized":"2021-03-24T21:55:52.136874+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T21:54:57.325723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:54:57.325723+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":16,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":630,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"147","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.143765+0000","last_change":"2021-03-24T21:55:35.512941+0000","last_active":"2021-03-24T21:55:52.143765+0000","last_peered":"2021-03-24T21:55:52.143765+0000","last_clean":"2021-03-24T21:55:52.143765+0000","last_became_active":"2021-03-24T21:54:38.128708+0000","last_became_peered":"2021-03-24T21:54:38.128708+0000","last_unstale":"2021-03-24T21:55:52.143765+0000","last_undegraded":"2021-03-24T21:55:52.143765+0000","last_fullsized":"2021-03-24T21:55:52.143765+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:35.512770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:55:35.512770+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"175'22","reported_seq":"297","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.142953+0000","last_change":"2021-03-24T21:55:52.142953+0000","last_active":"2021-03-24T21:55:52.142953+0000","last_peered":"2021-03-24T21:55:52.142953+0000","last_clean":"2021-03-24T21:55:52.142953+0000","last_became_active":"2021-03-24T21:54:38.122313+0000","last_became_peered":"2021-03-24T21:54:38.122313+0000","last_unstale":"2021-03-24T21:55:52.142953+0000","last_undegraded":"2021-03-24T21:55:52.142953+0000","last_fullsized":"2021-03-24T21:55:52.142953+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":13134,"num_write":14,"num_write_kb":3382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1253545,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2e","length":"1"},{"start":"3d","length":"2"},{"start":"42","length":"2"},{"start":"46","length":"1"},{"start":"4a","length":"1"}]},{"pgid":"6.1","version":"0'0","reported_seq":"137","reported_epoch":"186","state":"active+clean","last_fresh":"2021-03-24T21:55:55.127701+0000","last_change":"2021-03-24T21:54:38.124075+0000","last_active":"2021-03-24T21:55:55.127701+0000","last_peered":"2021-03-24T21:55:55.127701+0000","last_clean":"2021-03-24T21:55:55.127701+0000","last_became_active":"2021-03-24T21:54:38.122577+0000","last_became_peered":"2021-03-24T21:54:38.122577+0000","last_unstale":"2021-03-24T21:55:55.127701+0000","last_undegraded":"2021-03-24T21:55:55.127701+0000","last_fullsized":"2021-03-24T21:55:55.127701+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:28.691270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:28.691270+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"93","reported_epoch":"186","state":"active+clean","last_fresh":"2021-03-24T21:55:55.128064+0000","last_change":"2021-03-24T21:54:43.286516+0000","last_active":"2021-03-24T21:55:55.128064+0000","last_peered":"2021-03-24T21:55:55.128064+0000","last_clean":"2021-03-24T21:55:55.128064+0000","last_became_active":"2021-03-24T21:54:38.122875+0000","last_became_peered":"2021-03-24T21:54:38.122875+0000","last_unstale":"2021-03-24T21:55:55.128064+0000","last_undegraded":"2021-03-24T21:55:55.128064+0000","last_fullsized":"2021-03-24T21:55:55.128064+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:43.286336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:54:43.286336+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"69'21","reported_seq":"177","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.134076+0000","last_change":"2021-03-24T21:54:10.421436+0000","last_active":"2021-03-24T21:55:52.134076+0000","last_peered":"2021-03-24T21:55:52.134076+0000","last_clean":"2021-03-24T21:55:52.134076+0000","last_became_active":"2021-03-24T21:51:47.686486+0000","last_became_peered":"2021-03-24T21:51:47.686486+0000","last_unstale":"2021-03-24T21:55:52.134076+0000","last_undegraded":"2021-03-24T21:55:52.134076+0000","last_fullsized":"2021-03-24T21:55:52.134076+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'21","last_scrub_stamp":"2021-03-24T21:54:10.421258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:54:10.421258+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":45,"num_read_kb":27,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"93","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.135657+0000","last_change":"2021-03-24T21:54:24.199253+0000","last_active":"2021-03-24T21:55:52.135657+0000","last_peered":"2021-03-24T21:55:52.135657+0000","last_clean":"2021-03-24T21:55:52.135657+0000","last_became_active":"2021-03-24T21:54:24.198895+0000","last_became_peered":"2021-03-24T21:54:24.198895+0000","last_unstale":"2021-03-24T21:55:52.135657+0000","last_undegraded":"2021-03-24T21:55:52.135657+0000","last_fullsized":"2021-03-24T21:55:52.135657+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:23.246320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:23.246320+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"188'160","reported_seq":"484","reported_epoch":"188","state":"active+clean","last_fresh":"2021-03-24T21:55:57.127944+0000","last_change":"2021-03-24T21:55:57.128240+0000","last_active":"2021-03-24T21:55:57.127944+0000","last_peered":"2021-03-24T21:55:57.127944+0000","last_clean":"2021-03-24T21:55:57.127944+0000","last_became_active":"2021-03-24T21:54:38.139355+0000","last_became_peered":"2021-03-24T21:54:38.139355+0000","last_unstale":"2021-03-24T21:55:57.127944+0000","last_undegraded":"2021-03-24T21:55:57.127944+0000","last_fullsized":"2021-03-24T21:55:57.127944+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":160,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15075813,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":111,"num_read_kb":101033,"num_write":82,"num_write_kb":35540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":6977532,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3d","length":"2"},{"start":"42","length":"1"},{"start":"4a","length":"1"},{"start":"4e","length":"2"}]},{"pgid":"6.0","version":"0'0","reported_seq":"133","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.133298+0000","last_change":"2021-03-24T21:55:39.261965+0000","last_active":"2021-03-24T21:55:52.133298+0000","last_peered":"2021-03-24T21:55:52.133298+0000","last_clean":"2021-03-24T21:55:52.133298+0000","last_became_active":"2021-03-24T21:51:51.759627+0000","last_became_peered":"2021-03-24T21:51:51.759627+0000","last_unstale":"2021-03-24T21:55:52.133298+0000","last_undegraded":"2021-03-24T21:55:52.133298+0000","last_fullsized":"2021-03-24T21:55:52.133298+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:39.261698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:55:39.261698+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"142","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.144328+0000","last_change":"2021-03-24T21:54:20.101633+0000","last_active":"2021-03-24T21:55:52.144328+0000","last_peered":"2021-03-24T21:55:52.144328+0000","last_clean":"2021-03-24T21:55:52.144328+0000","last_became_active":"2021-03-24T21:51:45.583885+0000","last_became_peered":"2021-03-24T21:51:45.583885+0000","last_unstale":"2021-03-24T21:55:52.144328+0000","last_undegraded":"2021-03-24T21:55:52.144328+0000","last_fullsized":"2021-03-24T21:55:52.144328+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:20.101463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:54:20.101463+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"69'16","reported_seq":"156","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.144518+0000","last_change":"2021-03-24T21:54:51.015209+0000","last_active":"2021-03-24T21:55:52.144518+0000","last_peered":"2021-03-24T21:55:52.144518+0000","last_clean":"2021-03-24T21:55:52.144518+0000","last_became_active":"2021-03-24T21:54:38.128745+0000","last_became_peered":"2021-03-24T21:54:38.128745+0000","last_unstale":"2021-03-24T21:55:52.144518+0000","last_undegraded":"2021-03-24T21:55:52.144518+0000","last_fullsized":"2021-03-24T21:55:52.144518+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T21:54:51.015019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:54:51.015019+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":32,"num_read_kb":15,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":940,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"102'4","reported_seq":"195","reported_epoch":"184","state":"active+clean","last_fresh":"2021-03-24T21:55:53.165216+0000","last_change":"2021-03-24T21:54:24.305088+0000","last_active":"2021-03-24T21:55:53.165216+0000","last_peered":"2021-03-24T21:55:53.165216+0000","last_clean":"2021-03-24T21:55:53.165216+0000","last_became_active":"2021-03-24T21:54:24.203310+0000","last_became_peered":"2021-03-24T21:54:24.203310+0000","last_unstale":"2021-03-24T21:55:53.165216+0000","last_undegraded":"2021-03-24T21:55:53.165216+0000","last_fullsized":"2021-03-24T21:55:53.165216+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'3","last_scrub_stamp":"2021-03-24T21:53:26.584678+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-24T21:53:26.584678+0000","last_clean_scrub_stamp":"2021-03-24T21:53:26.584678+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"183'26","reported_seq":"297","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.138742+0000","last_change":"2021-03-24T21:55:52.138742+0000","last_active":"2021-03-24T21:55:52.138742+0000","last_peered":"2021-03-24T21:55:52.138742+0000","last_clean":"2021-03-24T21:55:52.138742+0000","last_became_active":"2021-03-24T21:54:38.143356+0000","last_became_peered":"2021-03-24T21:54:38.143356+0000","last_unstale":"2021-03-24T21:55:52.138742+0000","last_undegraded":"2021-03-24T21:55:52.138742+0000","last_fullsized":"2021-03-24T21:55:52.138742+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2705927,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":11,"num_write_kb":4778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":867724,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2e","length":"1"},{"start":"3d","length":"2"},{"start":"42","length":"2"},{"start":"46","length":"1"},{"start":"4a","length":"1"}]},{"pgid":"6.3","version":"0'0","reported_seq":"153","reported_epoch":"184","state":"active+clean","last_fresh":"2021-03-24T21:55:53.146254+0000","last_change":"2021-03-24T21:54:44.933344+0000","last_active":"2021-03-24T21:55:53.146254+0000","last_peered":"2021-03-24T21:55:53.146254+0000","last_clean":"2021-03-24T21:55:53.146254+0000","last_became_active":"2021-03-24T21:54:38.110246+0000","last_became_peered":"2021-03-24T21:54:38.110246+0000","last_unstale":"2021-03-24T21:55:53.146254+0000","last_undegraded":"2021-03-24T21:55:53.146254+0000","last_fullsized":"2021-03-24T21:55:53.146254+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:44.933255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:54:44.933255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"148","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.136971+0000","last_change":"2021-03-24T21:55:28.165657+0000","last_active":"2021-03-24T21:55:52.136971+0000","last_peered":"2021-03-24T21:55:52.136971+0000","last_clean":"2021-03-24T21:55:52.136971+0000","last_became_active":"2021-03-24T21:54:38.134604+0000","last_became_peered":"2021-03-24T21:54:38.134604+0000","last_unstale":"2021-03-24T21:55:52.136971+0000","last_undegraded":"2021-03-24T21:55:52.136971+0000","last_fullsized":"2021-03-24T21:55:52.136971+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:28.165408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:28.165408+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"69'20","reported_seq":"212","reported_epoch":"187","state":"active+clean","last_fresh":"2021-03-24T21:55:56.110594+0000","last_change":"2021-03-24T21:55:00.135489+0000","last_active":"2021-03-24T21:55:56.110594+0000","last_peered":"2021-03-24T21:55:56.110594+0000","last_clean":"2021-03-24T21:55:56.110594+0000","last_became_active":"2021-03-24T21:54:24.189182+0000","last_became_peered":"2021-03-24T21:54:24.189182+0000","last_unstale":"2021-03-24T21:55:56.110594+0000","last_undegraded":"2021-03-24T21:55:56.110594+0000","last_fullsized":"2021-03-24T21:55:56.110594+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'20","last_scrub_stamp":"2021-03-24T21:55:00.135328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:00.135328+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":39,"num_read_kb":20,"num_write":20,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":940,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"109'3","reported_seq":"200","reported_epoch":"184","state":"active+clean","last_fresh":"2021-03-24T21:55:53.130538+0000","last_change":"2021-03-24T21:54:39.228237+0000","last_active":"2021-03-24T21:55:53.130538+0000","last_peered":"2021-03-24T21:55:53.130538+0000","last_clean":"2021-03-24T21:55:53.130538+0000","last_became_active":"2021-03-24T21:54:38.135726+0000","last_became_peered":"2021-03-24T21:54:38.135726+0000","last_unstale":"2021-03-24T21:55:53.130538+0000","last_undegraded":"2021-03-24T21:55:53.130538+0000","last_fullsized":"2021-03-24T21:55:53.130538+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2021-03-24T21:54:17.493861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:54:17.493861+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"183'157","reported_seq":"434","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.146222+0000","last_change":"2021-03-24T21:55:52.146222+0000","last_active":"2021-03-24T21:55:52.146222+0000","last_peered":"2021-03-24T21:55:52.146222+0000","last_clean":"2021-03-24T21:55:52.146222+0000","last_became_active":"2021-03-24T21:54:22.167761+0000","last_became_peered":"2021-03-24T21:54:22.167761+0000","last_unstale":"2021-03-24T21:55:52.146222+0000","last_undegraded":"2021-03-24T21:55:52.146222+0000","last_fullsized":"2021-03-24T21:55:52.146222+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":157,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11865709,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":97,"num_read_kb":83337,"num_write":79,"num_write_kb":31814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2e","length":"1"},{"start":"3d","length":"2"},{"start":"42","length":"2"},{"start":"46","length":"1"},{"start":"4a","length":"1"}]},{"pgid":"6.2","version":"0'0","reported_seq":"133","reported_epoch":"187","state":"active+clean","last_fresh":"2021-03-24T21:55:56.110578+0000","last_change":"2021-03-24T21:53:45.349966+0000","last_active":"2021-03-24T21:55:56.110578+0000","last_peered":"2021-03-24T21:55:56.110578+0000","last_clean":"2021-03-24T21:55:56.110578+0000","last_became_active":"2021-03-24T21:51:51.756114+0000","last_became_peered":"2021-03-24T21:51:51.756114+0000","last_unstale":"2021-03-24T21:55:56.110578+0000","last_undegraded":"2021-03-24T21:55:56.110578+0000","last_fullsized":"2021-03-24T21:55:56.110578+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:45.349781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:45.349781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"140","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.130025+0000","last_change":"2021-03-24T21:54:56.508843+0000","last_active":"2021-03-24T21:55:52.130025+0000","last_peered":"2021-03-24T21:55:52.130025+0000","last_clean":"2021-03-24T21:55:52.130025+0000","last_became_active":"2021-03-24T21:51:45.378704+0000","last_became_peered":"2021-03-24T21:51:45.378704+0000","last_unstale":"2021-03-24T21:55:52.130025+0000","last_undegraded":"2021-03-24T21:55:52.130025+0000","last_fullsized":"2021-03-24T21:55:52.130025+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:56.508652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:54:56.508652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"69'22","reported_seq":"178","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.128380+0000","last_change":"2021-03-24T21:55:28.629625+0000","last_active":"2021-03-24T21:55:52.128380+0000","last_peered":"2021-03-24T21:55:52.128380+0000","last_clean":"2021-03-24T21:55:52.128380+0000","last_became_active":"2021-03-24T21:51:47.691459+0000","last_became_peered":"2021-03-24T21:51:47.691459+0000","last_unstale":"2021-03-24T21:55:52.128380+0000","last_undegraded":"2021-03-24T21:55:52.128380+0000","last_fullsized":"2021-03-24T21:55:52.128380+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'22","last_scrub_stamp":"2021-03-24T21:55:28.629443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:28.629443+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":46,"num_read_kb":26,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"140","reported_epoch":"187","state":"active+clean","last_fresh":"2021-03-24T21:55:56.110367+0000","last_change":"2021-03-24T21:55:25.106515+0000","last_active":"2021-03-24T21:55:56.110367+0000","last_peered":"2021-03-24T21:55:56.110367+0000","last_clean":"2021-03-24T21:55:56.110367+0000","last_became_active":"2021-03-24T21:51:49.408761+0000","last_became_peered":"2021-03-24T21:51:49.408761+0000","last_unstale":"2021-03-24T21:55:56.110367+0000","last_undegraded":"2021-03-24T21:55:56.110367+0000","last_fullsized":"2021-03-24T21:55:56.110367+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:25.106440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:25.220436+0000","last_clean_scrub_stamp":"2021-03-24T21:55:25.106440+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"184'123","reported_seq":"387","reported_epoch":"184","state":"active+clean","last_fresh":"2021-03-24T21:55:53.155015+0000","last_change":"2021-03-24T21:55:52.139709+0000","last_active":"2021-03-24T21:55:53.155015+0000","last_peered":"2021-03-24T21:55:53.155015+0000","last_clean":"2021-03-24T21:55:53.155015+0000","last_became_active":"2021-03-24T21:54:24.190419+0000","last_became_peered":"2021-03-24T21:54:24.190419+0000","last_unstale":"2021-03-24T21:55:53.155015+0000","last_undegraded":"2021-03-24T21:55:53.155015+0000","last_fullsized":"2021-03-24T21:55:53.155015+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":123,"ondisk_log_size":123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9021792,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":48,"num_read_kb":25664,"num_write":58,"num_write_kb":20267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2e","length":"1"},{"start":"3d","length":"2"},{"start":"42","length":"2"},{"start":"46","length":"1"},{"start":"4a","length":"1"}]},{"pgid":"6.d","version":"0'0","reported_seq":"145","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.144045+0000","last_change":"2021-03-24T21:55:28.557021+0000","last_active":"2021-03-24T21:55:52.144045+0000","last_peered":"2021-03-24T21:55:52.144045+0000","last_clean":"2021-03-24T21:55:52.144045+0000","last_became_active":"2021-03-24T21:54:38.129469+0000","last_became_peered":"2021-03-24T21:54:38.129469+0000","last_unstale":"2021-03-24T21:55:52.144045+0000","last_undegraded":"2021-03-24T21:55:52.144045+0000","last_fullsized":"2021-03-24T21:55:52.144045+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:28.556844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:55:28.556844+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"65'2","reported_seq":"186","reported_epoch":"187","state":"active+clean","last_fresh":"2021-03-24T21:55:56.111981+0000","last_change":"2021-03-24T21:53:29.268452+0000","last_active":"2021-03-24T21:55:56.111981+0000","last_peered":"2021-03-24T21:55:56.111981+0000","last_clean":"2021-03-24T21:55:56.111981+0000","last_became_active":"2021-03-24T21:51:49.414380+0000","last_became_peered":"2021-03-24T21:51:49.414380+0000","last_unstale":"2021-03-24T21:55:56.111981+0000","last_undegraded":"2021-03-24T21:55:56.111981+0000","last_fullsized":"2021-03-24T21:55:56.111981+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2021-03-24T21:53:29.268273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:29.268273+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":"144","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.128302+0000","last_change":"2021-03-24T21:54:38.135257+0000","last_active":"2021-03-24T21:55:52.128302+0000","last_peered":"2021-03-24T21:55:52.128302+0000","last_clean":"2021-03-24T21:55:52.128302+0000","last_became_active":"2021-03-24T21:54:38.134768+0000","last_became_peered":"2021-03-24T21:54:38.134768+0000","last_unstale":"2021-03-24T21:55:52.128302+0000","last_undegraded":"2021-03-24T21:55:52.128302+0000","last_fullsized":"2021-03-24T21:55:52.128302+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:35.609173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:35.609173+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"69'17","reported_seq":"148","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.130088+0000","last_change":"2021-03-24T21:54:38.505905+0000","last_active":"2021-03-24T21:55:52.130088+0000","last_peered":"2021-03-24T21:55:52.130088+0000","last_clean":"2021-03-24T21:55:52.130088+0000","last_became_active":"2021-03-24T21:54:38.130536+0000","last_became_peered":"2021-03-24T21:54:38.130536+0000","last_unstale":"2021-03-24T21:55:52.130088+0000","last_undegraded":"2021-03-24T21:55:52.130088+0000","last_fullsized":"2021-03-24T21:55:52.130088+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'17","last_scrub_stamp":"2021-03-24T21:54:03.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:54:03.731436+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":18,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":422,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"183'140","reported_seq":"420","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.149435+0000","last_change":"2021-03-24T21:55:52.149435+0000","last_active":"2021-03-24T21:55:52.149435+0000","last_peered":"2021-03-24T21:55:52.149435+0000","last_clean":"2021-03-24T21:55:52.149435+0000","last_became_active":"2021-03-24T21:54:38.123502+0000","last_became_peered":"2021-03-24T21:54:38.123502+0000","last_unstale":"2021-03-24T21:55:52.149435+0000","last_undegraded":"2021-03-24T21:55:52.149435+0000","last_fullsized":"2021-03-24T21:55:52.149435+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":140,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8084134,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":54,"num_read_kb":31460,"num_write":74,"num_write_kb":30495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":5093012,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2e","length":"1"},{"start":"3d","length":"2"},{"start":"42","length":"2"},{"start":"46","length":"1"},{"start":"4a","length":"1"}]},{"pgid":"6.c","version":"0'0","reported_seq":"138","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.131869+0000","last_change":"2021-03-24T21:54:24.196940+0000","last_active":"2021-03-24T21:55:52.131869+0000","last_peered":"2021-03-24T21:55:52.131869+0000","last_clean":"2021-03-24T21:55:52.131869+0000","last_became_active":"2021-03-24T21:54:24.196587+0000","last_became_peered":"2021-03-24T21:54:24.196587+0000","last_unstale":"2021-03-24T21:55:52.131869+0000","last_undegraded":"2021-03-24T21:55:52.131869+0000","last_fullsized":"2021-03-24T21:55:52.131869+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:27.701002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:27.701002+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"117","reported_epoch":"187","state":"active+clean","last_fresh":"2021-03-24T21:55:56.112377+0000","last_change":"2021-03-24T21:55:10.102484+0000","last_active":"2021-03-24T21:55:56.112377+0000","last_peered":"2021-03-24T21:55:56.112377+0000","last_clean":"2021-03-24T21:55:56.112377+0000","last_became_active":"2021-03-24T21:55:10.102113+0000","last_became_peered":"2021-03-24T21:55:10.102113+0000","last_unstale":"2021-03-24T21:55:56.112377+0000","last_undegraded":"2021-03-24T21:55:56.112377+0000","last_fullsized":"2021-03-24T21:55:56.112377+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:12.170934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:54:12.170934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"144","reported_epoch":"187","state":"active+clean","last_fresh":"2021-03-24T21:55:56.110464+0000","last_change":"2021-03-24T21:54:53.267409+0000","last_active":"2021-03-24T21:55:56.110464+0000","last_peered":"2021-03-24T21:55:56.110464+0000","last_clean":"2021-03-24T21:55:56.110464+0000","last_became_active":"2021-03-24T21:51:45.579513+0000","last_became_peered":"2021-03-24T21:51:45.579513+0000","last_unstale":"2021-03-24T21:55:56.110464+0000","last_undegraded":"2021-03-24T21:55:56.110464+0000","last_fullsized":"2021-03-24T21:55:56.110464+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:53.267324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:54:53.267324+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"69'15","reported_seq":"227","reported_epoch":"187","state":"active+clean","last_fresh":"2021-03-24T21:55:56.111778+0000","last_change":"2021-03-24T21:54:40.017534+0000","last_active":"2021-03-24T21:55:56.111778+0000","last_peered":"2021-03-24T21:55:56.111778+0000","last_clean":"2021-03-24T21:55:56.111778+0000","last_became_active":"2021-03-24T21:54:38.127461+0000","last_became_peered":"2021-03-24T21:54:38.127461+0000","last_unstale":"2021-03-24T21:55:56.111778+0000","last_undegraded":"2021-03-24T21:55:56.111778+0000","last_fullsized":"2021-03-24T21:55:56.111778+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:54:02.296830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:54:02.296830+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1476,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"186'27","reported_seq":"300","reported_epoch":"186","state":"active+clean","last_fresh":"2021-03-24T21:55:55.130395+0000","last_change":"2021-03-24T21:55:55.130395+0000","last_active":"2021-03-24T21:55:55.130395+0000","last_peered":"2021-03-24T21:55:55.130395+0000","last_clean":"2021-03-24T21:55:55.130395+0000","last_became_active":"2021-03-24T21:54:22.168397+0000","last_became_peered":"2021-03-24T21:54:22.168397+0000","last_unstale":"2021-03-24T21:55:55.130395+0000","last_undegraded":"2021-03-24T21:55:55.130395+0000","last_fullsized":"2021-03-24T21:55:55.130395+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4030649,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":31,"num_read_kb":26380,"num_write":16,"num_write_kb":3574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"3d","length":"2"},{"start":"42","length":"1"},{"start":"4a","length":"1"},{"start":"4f","length":"1"}]},{"pgid":"6.f","version":"0'0","reported_seq":"137","reported_epoch":"186","state":"active+clean","last_fresh":"2021-03-24T21:55:55.128762+0000","last_change":"2021-03-24T21:55:24.078895+0000","last_active":"2021-03-24T21:55:55.128762+0000","last_peered":"2021-03-24T21:55:55.128762+0000","last_clean":"2021-03-24T21:55:55.128762+0000","last_became_active":"2021-03-24T21:51:51.765067+0000","last_became_peered":"2021-03-24T21:51:51.765067+0000","last_unstale":"2021-03-24T21:55:55.128762+0000","last_undegraded":"2021-03-24T21:55:55.128762+0000","last_fullsized":"2021-03-24T21:55:55.128762+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:24.078780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:55:24.078780+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"91","reported_epoch":"187","state":"active+clean","last_fresh":"2021-03-24T21:55:56.111247+0000","last_change":"2021-03-24T21:54:38.139233+0000","last_active":"2021-03-24T21:55:56.111247+0000","last_peered":"2021-03-24T21:55:56.111247+0000","last_clean":"2021-03-24T21:55:56.111247+0000","last_became_active":"2021-03-24T21:54:38.138864+0000","last_became_peered":"2021-03-24T21:54:38.138864+0000","last_unstale":"2021-03-24T21:55:56.111247+0000","last_undegraded":"2021-03-24T21:55:56.111247+0000","last_fullsized":"2021-03-24T21:55:56.111247+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:29.735672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:53:29.735672+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"111","reported_epoch":"187","state":"active+clean","last_fresh":"2021-03-24T21:55:56.110931+0000","last_change":"2021-03-24T21:55:01.169611+0000","last_active":"2021-03-24T21:55:56.110931+0000","last_peered":"2021-03-24T21:55:56.110931+0000","last_clean":"2021-03-24T21:55:56.110931+0000","last_became_active":"2021-03-24T21:54:38.127136+0000","last_became_peered":"2021-03-24T21:54:38.127136+0000","last_unstale":"2021-03-24T21:55:56.110931+0000","last_undegraded":"2021-03-24T21:55:56.110931+0000","last_fullsized":"2021-03-24T21:55:56.110931+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:01.169461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:55:01.169461+0000","last_clean_scrub_stamp":"2021-03-24T21:55:01.169461+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"69'21","reported_seq":"213","reported_epoch":"187","state":"active+clean","last_fresh":"2021-03-24T21:55:56.111375+0000","last_change":"2021-03-24T21:54:56.226932+0000","last_active":"2021-03-24T21:55:56.111375+0000","last_peered":"2021-03-24T21:55:56.111375+0000","last_clean":"2021-03-24T21:55:56.111375+0000","last_became_active":"2021-03-24T21:54:38.144020+0000","last_became_peered":"2021-03-24T21:54:38.144020+0000","last_unstale":"2021-03-24T21:55:56.111375+0000","last_undegraded":"2021-03-24T21:55:56.111375+0000","last_fullsized":"2021-03-24T21:55:56.111375+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'21","last_scrub_stamp":"2021-03-24T21:54:56.226757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:54:56.226757+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":23,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"183'49","reported_seq":"316","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.149081+0000","last_change":"2021-03-24T21:55:52.149081+0000","last_active":"2021-03-24T21:55:52.149081+0000","last_peered":"2021-03-24T21:55:52.149081+0000","last_clean":"2021-03-24T21:55:52.149081+0000","last_became_active":"2021-03-24T21:54:22.160108+0000","last_became_peered":"2021-03-24T21:54:22.160108+0000","last_unstale":"2021-03-24T21:55:52.149081+0000","last_undegraded":"2021-03-24T21:55:52.149081+0000","last_fullsized":"2021-03-24T21:55:52.149081+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":49,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6084101,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":37,"num_read_kb":40836,"num_write":19,"num_write_kb":12102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2e","length":"1"},{"start":"3d","length":"2"},{"start":"42","length":"2"},{"start":"46","length":"1"},{"start":"4a","length":"1"}]},{"pgid":"6.e","version":"0'0","reported_seq":"142","reported_epoch":"187","state":"active+clean","last_fresh":"2021-03-24T21:55:56.112055+0000","last_change":"2021-03-24T21:54:38.117075+0000","last_active":"2021-03-24T21:55:56.112055+0000","last_peered":"2021-03-24T21:55:56.112055+0000","last_clean":"2021-03-24T21:55:56.112055+0000","last_became_active":"2021-03-24T21:54:38.116786+0000","last_became_peered":"2021-03-24T21:54:38.116786+0000","last_unstale":"2021-03-24T21:55:56.112055+0000","last_undegraded":"2021-03-24T21:55:56.112055+0000","last_fullsized":"2021-03-24T21:55:56.112055+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:40.289984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:53:40.289984+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"65'2","reported_seq":"190","reported_epoch":"186","state":"active+clean","last_fresh":"2021-03-24T21:55:55.128543+0000","last_change":"2021-03-24T21:54:56.160424+0000","last_active":"2021-03-24T21:55:55.128543+0000","last_peered":"2021-03-24T21:55:55.128543+0000","last_clean":"2021-03-24T21:55:55.128543+0000","last_became_active":"2021-03-24T21:51:49.416182+0000","last_became_peered":"2021-03-24T21:51:49.416182+0000","last_unstale":"2021-03-24T21:55:55.128543+0000","last_undegraded":"2021-03-24T21:55:55.128543+0000","last_fullsized":"2021-03-24T21:55:55.128543+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2021-03-24T21:54:56.160326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:54:56.160326+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"140","reported_epoch":"183","state":"active+clean","last_fresh":"2021-03-24T21:55:52.133605+0000","last_change":"2021-03-24T21:53:57.501519+0000","last_active":"2021-03-24T21:55:52.133605+0000","last_peered":"2021-03-24T21:55:52.133605+0000","last_clean":"2021-03-24T21:55:52.133605+0000","last_became_active":"2021-03-24T21:51:45.375830+0000","last_became_peered":"2021-03-24T21:51:45.375830+0000","last_unstale":"2021-03-24T21:55:52.133605+0000","last_undegraded":"2021-03-24T21:55:52.133605+0000","last_fullsized":"2021-03-24T21:55:52.133605+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:53:57.501351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:53:57.501351+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"69'13","reported_seq":"217","reported_epoch":"187","state":"active+clean","last_fresh":"2021-03-24T21:55:56.111501+0000","last_change":"2021-03-24T21:55:36.065455+0000","last_active":"2021-03-24T21:55:56.111501+0000","last_peered":"2021-03-24T21:55:56.111501+0000","last_clean":"2021-03-24T21:55:56.111501+0000","last_became_active":"2021-03-24T21:54:38.127115+0000","last_became_peered":"2021-03-24T21:54:38.127115+0000","last_unstale":"2021-03-24T21:55:56.111501+0000","last_undegraded":"2021-03-24T21:55:56.111501+0000","last_fullsized":"2021-03-24T21:55:56.111501+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2021-03-24T21:55:36.065267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:36.065267+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":13,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":118778880,"num_objects":44,"num_object_clones":5,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":1,"num_read":906,"num_read_kb":791469,"num_write":751,"num_write_kb":283437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":35460777,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":285048832,"data_stored":298586580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1576,"ondisk_log_size":1576,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":247,"num_read_kb":242,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1589,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":5558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1062,"num_read_kb":567,"num_write":542,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":283,"num_bytes_recovered":22082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2027520,"data_stored":72734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":549,"ondisk_log_size":549,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25,"ondisk_log_size":25,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201334,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":64872,"kb_used_data":46880,"kb_used_omap":0,"kb_used_meta":17984,"kb_avail":93680280,"statfs":{"total":95995035648,"available":95928606720,"internally_reserved":0,"allocated":48005120,"data_stored":48531758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":18415616},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:55:49 2021","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":0.97799999999999998,"15min":0.92500000000000004},"min":{"1min":0.50900000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":5.694,"5min":5.694,"15min":5.694},"last":0.67900000000000005},{"interface":"front","average":{"1min":1.0409999999999999,"5min":0.95899999999999996,"15min":0.96899999999999997},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":6.0279999999999996,"5min":6.0279999999999996,"15min":6.0279999999999996},"last":1.0049999999999999}]},{"osd":1,"last update":"Wed Mar 24 21:55:49 2021","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":0.95699999999999996,"15min":0.873},"min":{"1min":0.49099999999999999,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":5.7750000000000004,"5min":5.7750000000000004,"15min":5.7750000000000004},"last":0.79500000000000004},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.006,"15min":0.97999999999999998},"min":{"1min":0.56899999999999995,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":6.0780000000000003,"5min":6.0780000000000003,"15min":6.0780000000000003},"last":0.82399999999999995}]},{"osd":2,"last update":"Wed Mar 24 21:55:49 2021","interfaces":[{"interface":"back","average":{"1min":1.5900000000000001,"5min":1.123,"15min":1.0369999999999999},"min":{"1min":0.50800000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":12.135999999999999,"5min":12.135999999999999,"15min":12.135999999999999},"last":0.91000000000000003},{"interface":"front","average":{"1min":1.2310000000000001,"5min":1.054,"15min":1.026},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":5.9729999999999999,"5min":5.9729999999999999,"15min":5.9729999999999999},"last":0.96299999999999997}]},{"osd":3,"last update":"Wed Mar 24 21:55:49 2021","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.1240000000000001,"15min":1.071},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":12.199,"5min":12.199,"15min":12.199},"last":0.75600000000000001},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.0940000000000001,"15min":1.0109999999999999},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":6.976,"5min":6.976,"15min":6.976},"last":0.874}]},{"osd":4,"last update":"Wed Mar 24 21:55:49 2021","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.036,"15min":0.95999999999999996},"min":{"1min":0.52900000000000003,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":6.2089999999999996,"5min":6.2089999999999996,"15min":6.2089999999999996},"last":1.093},{"interface":"front","average":{"1min":1.6619999999999999,"5min":1.1950000000000001,"15min":1.1140000000000001},"min":{"1min":0.59199999999999997,"5min":0.504,"15min":0.504},"max":{"1min":12.244,"5min":12.244,"15min":12.244},"last":1.0389999999999999}]},{"osd":5,"last update":"Wed Mar 24 21:55:49 2021","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.0980000000000001,"15min":1.008},"min":{"1min":0.64000000000000001,"5min":0.435,"15min":0.435},"max":{"1min":5.8840000000000003,"5min":5.8840000000000003,"15min":5.8840000000000003},"last":1.0640000000000001},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.1970000000000001,"15min":1.1559999999999999},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":6.3300000000000001,"5min":6.3300000000000001,"15min":6.3300000000000001},"last":1.1699999999999999}]},{"osd":6,"last update":"Wed Mar 24 21:55:49 2021","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.2130000000000001,"15min":1.1419999999999999},"min":{"1min":0.67100000000000004,"5min":0.62,"15min":0.62},"max":{"1min":6.1159999999999997,"5min":6.1159999999999997,"15min":6.1159999999999997},"last":1.246},{"interface":"front","average":{"1min":1.786,"5min":1.3089999999999999,"15min":1.2170000000000001},"min":{"1min":0.751,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":12.287000000000001,"5min":12.287000000000001,"15min":12.287000000000001},"last":1.123}]}]},{"osd":6,"up_from":0,"seq":201863462971,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":0,"seq":55834574928,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738451,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":65992,"kb_used_data":47424,"kb_used_omap":0,"kb_used_meta":18560,"kb_avail":93679160,"statfs":{"total":95995035648,"available":95927459840,"internally_reserved":0,"allocated":48562176,"data_stored":48703039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":19005440},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Mar 24 21:55:41 2021","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.71399999999999997,"15min":0.60999999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.171,"5min":4.8040000000000003,"15min":4.8040000000000003},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.81299999999999994,"15min":0.65600000000000003},"min":{"1min":0.42599999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.80000000000000004,"5min":8.8989999999999991,"15min":8.8989999999999991},"last":0.73799999999999999}]},{"osd":2,"last update":"Wed Mar 24 21:54:58 2021","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":0.75700000000000001,"15min":0.68899999999999995},"min":{"1min":0.40899999999999997,"5min":0.314,"15min":0.314},"max":{"1min":5.8810000000000002,"5min":5.8810000000000002,"15min":5.8810000000000002},"last":1},{"interface":"front","average":{"1min":1.008,"5min":0.82199999999999995,"15min":0.72099999999999997},"min":{"1min":0.47499999999999998,"5min":0.376,"15min":0.376},"max":{"1min":4.766,"5min":4.766,"15min":4.766},"last":1.0780000000000001}]},{"osd":3,"last update":"Wed Mar 24 21:55:21 2021","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.88900000000000001,"15min":0.82799999999999996},"min":{"1min":0.55500000000000005,"5min":0.376,"15min":0.376},"max":{"1min":1.482,"5min":5.9790000000000001,"15min":5.9790000000000001},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.93400000000000005,"5min":0.97799999999999998,"15min":0.94599999999999995},"min":{"1min":0.505,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.3280000000000001,"5min":6.4580000000000002,"15min":6.4580000000000002},"last":1.153}]},{"osd":4,"last update":"Wed Mar 24 21:55:41 2021","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":1.101,"15min":1.1000000000000001},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.385,"5min":5.0330000000000004,"15min":5.0330000000000004},"last":1.2},{"interface":"front","average":{"1min":1,"5min":1.1060000000000001,"15min":1.04},"min":{"1min":0.82499999999999996,"5min":0.625,"15min":0.625},"max":{"1min":1.488,"5min":6.1829999999999998,"15min":6.1829999999999998},"last":0.95599999999999996}]},{"osd":5,"last update":"Wed Mar 24 21:54:58 2021","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":1.0960000000000001,"15min":1.077},"min":{"1min":0.69399999999999995,"5min":0.62,"15min":0.62},"max":{"1min":5.9390000000000001,"5min":5.9390000000000001,"15min":5.9390000000000001},"last":0.80300000000000005},{"interface":"front","average":{"1min":1.359,"5min":1.218,"15min":1.1899999999999999},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":5.0590000000000002,"5min":5.0590000000000002,"15min":5.0590000000000002},"last":1.296}]},{"osd":6,"last update":"Wed Mar 24 21:55:21 2021","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.129,"15min":1.087},"min":{"1min":0.81699999999999995,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.452,"5min":5.9930000000000003,"15min":5.9930000000000003},"last":0.92600000000000005},{"interface":"front","average":{"1min":1.129,"5min":1.248,"15min":1.228},"min":{"1min":0.91800000000000004,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.327,"5min":4.9960000000000004,"15min":4.9960000000000004},"last":1.3440000000000001}]},{"osd":7,"last update":"Wed Mar 24 21:55:52 2021","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.2250000000000001,"15min":1.1870000000000001},"min":{"1min":0.85099999999999998,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":2.0129999999999999,"5min":6.1319999999999997,"15min":6.1319999999999997},"last":1.012},{"interface":"front","average":{"1min":1.177,"5min":1.198,"15min":1.149},"min":{"1min":0.70499999999999996,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":2.1099999999999999,"5min":6.0529999999999999,"15min":6.0529999999999999},"last":1.042}]}]},{"osd":2,"up_from":18,"seq":77309411405,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":60828,"kb_used_data":47316,"kb_used_omap":0,"kb_used_meta":13504,"kb_avail":93684324,"statfs":{"total":95995035648,"available":95932747776,"internally_reserved":0,"allocated":48451584,"data_stored":52723729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":13828096},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:54:57 2021","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.63},"min":{"1min":0.433,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.228,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.74299999999999999,"15min":0.68999999999999995},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.6140000000000001,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.71499999999999997}]},{"osd":1,"last update":"Wed Mar 24 21:54:57 2021","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.77300000000000002,"15min":0.70899999999999996},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.5780000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.99199999999999999},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.79300000000000004,"15min":0.73699999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.0840000000000001,"5min":1.6240000000000001,"15min":1.6240000000000001},"last":0.86499999999999999}]},{"osd":3,"last update":"Wed Mar 24 21:55:20 2021","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.85599999999999998,"15min":0.82799999999999996},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.1659999999999999,"5min":1.9890000000000001,"15min":1.9890000000000001},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.91900000000000004,"15min":0.89400000000000002},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.2210000000000001,"5min":2.0950000000000002,"15min":2.0950000000000002},"last":1.1180000000000001}]},{"osd":4,"last update":"Wed Mar 24 21:55:43 2021","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.069,"15min":1.0449999999999999},"min":{"1min":0.72999999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":2.3149999999999999,"5min":3.3380000000000001,"15min":3.3380000000000001},"last":1.3819999999999999},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.077,"15min":1.0489999999999999},"min":{"1min":0.65600000000000003,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":2.1859999999999999,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":0.96499999999999997}]},{"osd":5,"last update":"Wed Mar 24 21:54:57 2021","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.0509999999999999,"15min":1.0389999999999999},"min":{"1min":0.71199999999999997,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.93899999999999995},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.1599999999999999,"15min":1.1539999999999999},"min":{"1min":0.67200000000000004,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.7490000000000001,"5min":3.1509999999999998,"15min":3.1509999999999998},"last":1.429}]},{"osd":6,"last update":"Wed Mar 24 21:55:20 2021","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.1990000000000001,"15min":1.2430000000000001},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.403,"5min":2.2810000000000001,"15min":2.2810000000000001},"last":1.3540000000000001},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.1459999999999999,"15min":1.1599999999999999},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.375,"5min":2.016,"15min":2.016},"last":1.0620000000000001}]},{"osd":7,"last update":"Wed Mar 24 21:55:48 2021","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.232,"15min":1.232},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":4.4880000000000004,"5min":4.4880000000000004,"15min":4.4880000000000004},"last":1.165},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.2330000000000001,"15min":1.2450000000000001},"min":{"1min":0.83099999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.796,"5min":3.5419999999999998,"15min":3.5419999999999998},"last":1.331}]}]},{"osd":3,"up_from":26,"seq":111669149768,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":70856,"kb_used_data":52224,"kb_used_omap":0,"kb_used_meta":18624,"kb_avail":93674296,"statfs":{"total":95995035648,"available":95922479104,"internally_reserved":0,"allocated":53477376,"data_stored":52490176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":19070976},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:55:25 2021","interfaces":[{"interface":"back","average":{"1min":1.7450000000000001,"5min":0.85699999999999998,"15min":0.70899999999999996},"min":{"1min":0.42799999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":23.846,"5min":23.846,"15min":23.846},"last":0.56200000000000006},{"interface":"front","average":{"1min":1.819,"5min":0.96399999999999997,"15min":0.79800000000000004},"min":{"1min":0.432,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":23.734999999999999,"5min":23.734999999999999,"15min":23.734999999999999},"last":0.83299999999999996}]},{"osd":1,"last update":"Wed Mar 24 21:55:25 2021","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":0.89800000000000002,"15min":0.79000000000000004},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":14.324,"5min":14.324,"15min":14.324},"last":0.65300000000000002},{"interface":"front","average":{"1min":1.3220000000000001,"5min":0.877,"15min":0.79500000000000004},"min":{"1min":0.45300000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":12.631,"5min":12.631,"15min":12.631},"last":0.88700000000000001}]},{"osd":2,"last update":"Wed Mar 24 21:55:25 2021","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":0.93000000000000005,"15min":0.83099999999999996},"min":{"1min":0.48999999999999999,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":12.510999999999999,"5min":12.510999999999999,"15min":12.510999999999999},"last":0.72299999999999998},{"interface":"front","average":{"1min":1.423,"5min":0.92000000000000004,"15min":0.81299999999999994},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":11.317,"5min":11.317,"15min":11.317},"last":0.754}]},{"osd":4,"last update":"Wed Mar 24 21:55:46 2021","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":1.002,"15min":0.96999999999999997},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.627,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":0.86299999999999999},{"interface":"front","average":{"1min":1.04,"5min":1.149,"15min":1.0049999999999999},"min":{"1min":0.64100000000000001,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.923,"5min":14.291,"15min":14.291},"last":0.91200000000000003}]},{"osd":5,"last update":"Wed Mar 24 21:54:58 2021","interfaces":[{"interface":"back","average":{"1min":1.6000000000000001,"5min":1.175,"15min":1.0980000000000001},"min":{"1min":0.67800000000000005,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":12.723000000000001,"5min":12.723000000000001,"15min":12.723000000000001},"last":1.0820000000000001},{"interface":"front","average":{"1min":1.194,"5min":1.0780000000000001,"15min":1.0900000000000001},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":2.375,"5min":2.375,"15min":2.375},"last":0.93000000000000005}]},{"osd":6,"last update":"Wed Mar 24 21:55:25 2021","interfaces":[{"interface":"back","average":{"1min":1.7989999999999999,"5min":1.2450000000000001,"15min":1.1679999999999999},"min":{"1min":0.68400000000000005,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":14.255000000000001,"5min":14.255000000000001,"15min":14.255000000000001},"last":0.95499999999999996},{"interface":"front","average":{"1min":1.7490000000000001,"5min":1.2709999999999999,"15min":1.1930000000000001},"min":{"1min":0.73399999999999999,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":12.666,"5min":12.666,"15min":12.666},"last":0.80000000000000004}]},{"osd":7,"last update":"Wed Mar 24 21:55:50 2021","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.167,"15min":1.107},"min":{"1min":0.81899999999999995,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":2.1120000000000001,"5min":2.3479999999999999,"15min":2.3479999999999999},"last":1.0049999999999999},{"interface":"front","average":{"1min":1.272,"5min":1.3380000000000001,"15min":1.2190000000000001},"min":{"1min":0.75800000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":2.0899999999999999,"5min":14.727,"15min":14.727},"last":0.99199999999999999}]}]},{"osd":4,"up_from":32,"seq":137438953539,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":72084,"kb_used_data":54220,"kb_used_omap":0,"kb_used_meta":17856,"kb_avail":93673068,"statfs":{"total":95995035648,"available":95921221632,"internally_reserved":0,"allocated":55521280,"data_stored":59845150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":18284544},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:55:41 2021","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.84199999999999997,"15min":0.83999999999999997},"min":{"1min":0.56299999999999994,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.0649999999999999,"5min":1.268,"15min":1.268},"last":0.93700000000000006},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.98399999999999999,"15min":0.89200000000000002},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.2210000000000001,"5min":6.3929999999999998,"15min":6.3929999999999998},"last":0.89200000000000002}]},{"osd":1,"last update":"Wed Mar 24 21:55:41 2021","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"min":{"1min":0.57899999999999996,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.175,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":1.2130000000000001},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.90000000000000002,"15min":0.85799999999999998},"min":{"1min":0.69199999999999995,"5min":0.436,"15min":0.436},"max":{"1min":1.034,"5min":1.873,"15min":1.873},"last":1.0529999999999999}]},{"osd":2,"last update":"Wed Mar 24 21:55:41 2021","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.95499999999999996,"15min":0.91500000000000004},"min":{"1min":0.61099999999999999,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":2.1520000000000001,"5min":4.4379999999999997,"15min":4.4379999999999997},"last":1.075},{"interface":"front","average":{"1min":1.0589999999999999,"5min":0.96999999999999997,"15min":0.92600000000000005},"min":{"1min":0.55400000000000005,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":2.738,"5min":2.738,"15min":2.738},"last":1.099}]},{"osd":3,"last update":"Wed Mar 24 21:55:41 2021","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":0.96799999999999997,"15min":0.94699999999999995},"min":{"1min":0.60699999999999998,"5min":0.439,"15min":0.439},"max":{"1min":3.2000000000000002,"5min":3.2000000000000002,"15min":3.2000000000000002},"last":0.996},{"interface":"front","average":{"1min":0.94999999999999996,"5min":0.97599999999999998,"15min":0.95399999999999996},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.206,"5min":1.474,"15min":1.474},"last":1.1279999999999999}]},{"osd":5,"last update":"Wed Mar 24 21:55:01 2021","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.93500000000000005,"15min":0.94499999999999995},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.2150000000000001,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":1.0309999999999999},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.0569999999999999,"15min":1.0109999999999999},"min":{"1min":0.66800000000000004,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.3200000000000001,"5min":2.0760000000000001,"15min":2.0760000000000001},"last":1.1659999999999999}]},{"osd":6,"last update":"Wed Mar 24 21:55:22 2021","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.048,"15min":1.0600000000000001},"min":{"1min":0.58299999999999996,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.405,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":1.284},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.0549999999999999,"15min":1.0620000000000001},"min":{"1min":0.74199999999999999,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.224,"5min":1.542,"15min":1.542},"last":1.1910000000000001}]},{"osd":7,"last update":"Wed Mar 24 21:55:47 2021","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.1399999999999999,"15min":1.1779999999999999},"min":{"1min":0.74299999999999999,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.3839999999999999,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.169,"15min":1.194},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.4159999999999999,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":1.258}]}]},{"osd":5,"up_from":39,"seq":167503724607,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36332,"kb_used_data":23972,"kb_used_omap":0,"kb_used_meta":12352,"kb_avail":93708820,"statfs":{"total":95995035648,"available":95957831680,"internally_reserved":0,"allocated":24547328,"data_stored":23614512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":12648448},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:55:03 2021","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.872,"15min":0.88300000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.3280000000000001,"5min":2.29,"15min":2.29},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.92000000000000004,"15min":0.89600000000000002},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.272,"5min":3.1779999999999999,"15min":3.1779999999999999},"last":0.56399999999999995}]},{"osd":1,"last update":"Wed Mar 24 21:55:03 2021","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.92400000000000004,"15min":0.92200000000000004},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.3049999999999999,"5min":2.8690000000000002,"15min":2.8690000000000002},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.92600000000000005,"15min":0.93000000000000005},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.395,"5min":2.0030000000000001,"15min":2.0030000000000001},"last":0.96599999999999997}]},{"osd":2,"last update":"Wed Mar 24 21:55:03 2021","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":0.96899999999999997,"15min":0.92500000000000004},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":6.5199999999999996,"5min":6.5199999999999996,"15min":6.5199999999999996},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.94599999999999995,"15min":0.90700000000000003},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.2030000000000001,"5min":5.8479999999999999,"15min":5.8479999999999999},"last":0.66100000000000003}]},{"osd":3,"last update":"Wed Mar 24 21:55:03 2021","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.95999999999999996,"15min":0.94199999999999995},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.3500000000000001,"5min":3.1259999999999999,"15min":3.1259999999999999},"last":0.88},{"interface":"front","average":{"1min":0.95699999999999996,"5min":0.98199999999999998,"15min":0.95399999999999996},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.3089999999999999,"5min":2.6850000000000001,"15min":2.6850000000000001},"last":0.68799999999999994}]},{"osd":4,"last update":"Wed Mar 24 21:55:03 2021","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":0.91400000000000003,"15min":0.85499999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.4339999999999999,"5min":2.2320000000000002,"15min":2.2320000000000002},"last":1.115},{"interface":"front","average":{"1min":1.0549999999999999,"5min":0.98999999999999999,"15min":0.92200000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.54,"5min":1.6970000000000001,"15min":1.6970000000000001},"last":1.0009999999999999}]},{"osd":6,"last update":"Wed Mar 24 21:55:22 2021","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":0.97199999999999998,"15min":0.89700000000000002},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.3919999999999999,"5min":2.085,"15min":2.085},"last":1.022},{"interface":"front","average":{"1min":1.052,"5min":1.048,"15min":0.98499999999999999},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.51,"5min":2.3079999999999998,"15min":2.3079999999999998},"last":0.73199999999999998}]},{"osd":7,"last update":"Wed Mar 24 21:55:51 2021","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":1.135,"15min":1.0740000000000001},"min":{"1min":0.90400000000000003,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":2.548,"5min":2.548,"15min":2.548},"last":0.86099999999999999},{"interface":"front","average":{"1min":1.2350000000000001,"5min":1.1100000000000001,"15min":1.02},"min":{"1min":0.83999999999999997,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":2.9889999999999999,"5min":2.9889999999999999,"15min":2.9889999999999999},"last":1.075}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":14134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":7188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":15432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":11740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":11460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":299008,"data_stored":10618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":47034368,"data_stored":48064141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7032832,"data_stored":6977532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":47095808,"data_stored":52071620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":51929088,"data_stored":51851303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":54022144,"data_stored":59181938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":23105536,"data_stored":22978769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":54829056,"data_stored":57461277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-03-24T21:56:00.379 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2021-03-24T21:56:00.380 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd pool set unique_pool_0 pg_num 26 2021-03-24T21:56:00.421 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:00.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:00.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:56:00.437 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:56:00.443 INFO:journalctl@ceph.osd.2.smithi125.stdout:Mar 24 21:56:00 smithi125 conmon[34620]: debug 2021-03-24T21:56:00.198+0000 7f47099de700 -1 osd.2 pg_epoch: 191 pg[7.8( v 188'85 (0'0,188'85] local-lis/les=99/100 n=3 ec=99/99 lis/c=99/99 les/c/f=100/100/0 sis=99) [2,7,4] r=0 lpr=99 crt=188'85 lcod 187'83 mlcod 187'83 active+clean+snaptrim trimq=[51~1] ps=[4e~2,53~1]] removing snap head 2021-03-24T21:56:00.444 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:00 smithi125 conmon[24927]: cluster 2021-03-24T21:55:58.738701+0000 mgr.y (mgr.14144) 355 2021-03-24T21:56:00.444 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:00 smithi125 conmon[24927]: : cluster [DBG] pgmap v446: 148 pgs: 148 active+clean; 114 MiB data, 364 MiB used, 536 GiB / 536 GiB avail; 767 B/s rd, 1.9 MiB/s wr, 5 op/s 2021-03-24T21:56:00.444 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:00 smithi125 conmon[24927]: cluster 2021-03-24T21:55:59.032156+0000 osd.4 (osd.4) 53 : cluster [DBG] 5.e scrub ok 2021-03-24T21:56:00.444 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:00 smithi125 conmon[24927]: cluster 2021-03-24T21:55: 2021-03-24T21:56:00.445 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:00 smithi125 conmon[24927]: 59.192323+0000 mon.a (mon.0) 779 : cluster [DBG] osdmap e190: 8 total, 8 up, 6 in 2021-03-24T21:56:00.445 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:00 smithi125 conmon[24927]: cluster 2021-03-24T21:55:59.276293+0000 osd.5 (osd.5) 54 : cluster [DBG] 7.0 scrub ok 2021-03-24T21:56:00.445 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:00 smithi125 conmon[24927]: cluster 2021-03-24T21:56:00.108461+0000 mon.a (mon.0) 780 : cluster [DBG] osdmap e191: 8 total, 8 up, 6 in 2021-03-24T21:56:00.445 INFO:journalctl@ceph.osd.3.smithi125.stdout:Mar 24 21:56:00 smithi125 conmon[37997]: debug 2021-03-24T21:56:00.446 INFO:journalctl@ceph.osd.3.smithi125.stdout:Mar 24 21:56:00 smithi125 conmon[37997]: 2021-03-24T21:56:00.202+0000 7f17173db700 -1 osd.3 pg_epoch: 191 pg[7.3( v 191'165 (0'0,191'165] local-lis/les=99/100 n=6 ec=99/99 lis/c=99/99 les/c/f=100/100/0 sis=99) [3,2,4] r=0 lpr=99 luod=188'163 crt=191'165 lcod 188'161 mlcod 188'161 active+clean+snaptrim trimq=[51~1] ps=[4e~2,53~1]] removing snap head 2021-03-24T21:56:00.446 INFO:journalctl@ceph.osd.3.smithi125.stdout:Mar 24 21:56:00 smithi125 conmon[37997]: 2021-03-24T21:56:00.523 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:00 smithi142 conmon[20125]: cluster 2021-03-24T21:55: 2021-03-24T21:56:00.523 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:00 smithi142 conmon[20125]: 58.738701+0000 mgr.y (mgr.14144) 355 : cluster [DBG] pgmap v446: 148 pgs: 148 active+clean; 114 MiB data, 364 MiB used, 536 GiB / 536 GiB avail; 767 B/s rd, 1.9 MiB/s wr, 5 op/s 2021-03-24T21:56:00.523 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:00 smithi142 conmon[20125]: cluster 2021-03-24T21:56:00.524 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:00 smithi142 conmon[20125]: 2021-03-24T21:55:59.032156+0000 osd.4 ( 2021-03-24T21:56:00.524 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:00 smithi142 conmon[20125]: osd.4) 53 : cluster [DBG] 5.e scrub ok 2021-03-24T21:56:00.524 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:00 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:56:00.525 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:00 smithi142 conmon[20125]: 21:55:59.192323+0000 mon.a (mon. 2021-03-24T21:56:00.525 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:00 smithi142 conmon[20125]: 0) 779 : cluster [DBG] osdmap e190: 8 total, 8 up, 6 in 2021-03-24T21:56:00.525 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:00 smithi142 conmon[20125]: cluster 2021-03-24T21:55:59 2021-03-24T21:56:00.526 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:00 smithi142 conmon[20125]: .276293+0000 osd.5 (osd.5) 54 : cluster [DBG] 7.0 scrub ok 2021-03-24T21:56:00.526 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:00 smithi142 conmon[20125]: cluster 2021-03-24T21 2021-03-24T21:56:00.526 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:00 smithi142 conmon[20125]: :56:00.108461+0000 mon.a (mon.0) 780 : cluster 2021-03-24T21:56:00.527 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:00 smithi142 conmon[20125]: [DBG] osdmap e191: 8 total, 8 up, 6 in 2021-03-24T21:56:00.527 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:00.547 INFO:tasks.rados.rados.0.smithi125.stdout:455: finishing write tid 3 to smithi12556470-6 2021-03-24T21:56:00.547 INFO:tasks.rados.rados.0.smithi125.stdout:455: finishing write tid 4 to smithi12556470-6 2021-03-24T21:56:00.547 INFO:tasks.rados.rados.0.smithi125.stdout:455: finishing write tid 5 to smithi12556470-6 2021-03-24T21:56:00.548 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 6 v 171 (ObjNum 191 snap 42 seq_num 191) dirty exists 2021-03-24T21:56:00.548 INFO:tasks.rados.rados.0.smithi125.stdout:455: left oid 6 (ObjNum 191 snap 42 seq_num 191) 2021-03-24T21:56:00.548 INFO:tasks.rados.rados.0.smithi125.stdout:459: finishing write tid 1 to smithi12556470-19 2021-03-24T21:56:00.548 INFO:tasks.rados.rados.0.smithi125.stdout:456: finishing write tid 1 to smithi12556470-17 2021-03-24T21:56:00.549 INFO:tasks.rados.rados.0.smithi125.stdout:459: finishing write tid 2 to smithi12556470-19 2021-03-24T21:56:00.549 INFO:tasks.rados.rados.0.smithi125.stdout:459: finishing write tid 3 to smithi12556470-19 2021-03-24T21:56:00.549 INFO:tasks.rados.rados.0.smithi125.stdout:459: finishing write tid 4 to smithi12556470-19 2021-03-24T21:56:00.549 INFO:tasks.rados.rados.0.smithi125.stdout:459: finishing write tid 5 to smithi12556470-19 2021-03-24T21:56:00.550 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 19 v 164 (ObjNum 193 snap 42 seq_num 193) dirty exists 2021-03-24T21:56:00.550 INFO:tasks.rados.rados.0.smithi125.stdout:459: left oid 19 (ObjNum 193 snap 42 seq_num 193) 2021-03-24T21:56:00.550 INFO:tasks.rados.rados.0.smithi125.stdout:456: finishing write tid 2 to smithi12556470-17 2021-03-24T21:56:00.550 INFO:tasks.rados.rados.0.smithi125.stdout:456: finishing write tid 3 to smithi12556470-17 2021-03-24T21:56:00.551 INFO:tasks.rados.rados.0.smithi125.stdout:456: finishing write tid 4 to smithi12556470-17 2021-03-24T21:56:00.551 INFO:tasks.rados.rados.0.smithi125.stdout:456: finishing write tid 5 to smithi12556470-17 2021-03-24T21:56:00.551 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 17 v 164 (ObjNum 192 snap 42 seq_num 192) dirty exists 2021-03-24T21:56:00.551 INFO:tasks.rados.rados.0.smithi125.stdout:456: left oid 17 (ObjNum 192 snap 42 seq_num 192) 2021-03-24T21:56:00.578 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:00.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:00.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:56:00.684 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:00.733 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:00.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:00.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:56:00.838 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:00.899 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:00.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:00.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:56:00.930 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:56:01.006 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:01.060 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:01.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:01.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:56:01.117 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2021-03-24T21:56:01.117 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2021-03-24T21:56:01.118 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2021-03-24T21:56:01.118 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2021-03-24T21:56:01.118 INFO:teuthology.orchestra.run.smithi142.stdout:} 2021-03-24T21:56:01.125 INFO:tasks.rados.rados.0.smithi125.stdout:452: done (7 left) 2021-03-24T21:56:01.125 INFO:tasks.rados.rados.0.smithi125.stdout:453: done (6 left) 2021-03-24T21:56:01.125 INFO:tasks.rados.rados.0.smithi125.stdout:455: done (5 left) 2021-03-24T21:56:01.126 INFO:tasks.rados.rados.0.smithi125.stdout:456: done (4 left) 2021-03-24T21:56:01.126 INFO:tasks.rados.rados.0.smithi125.stdout:457: done (3 left) 2021-03-24T21:56:01.126 INFO:tasks.rados.rados.0.smithi125.stdout:458: done (2 left) 2021-03-24T21:56:01.126 INFO:tasks.rados.rados.0.smithi125.stdout:459: done (1 left) 2021-03-24T21:56:01.127 INFO:tasks.rados.rados.0.smithi125.stdout:461: done (0 left) 2021-03-24T21:56:01.127 INFO:tasks.rados.rados.0.smithi125.stdout:462: read oid 23 snap 42 2021-03-24T21:56:01.127 INFO:tasks.rados.rados.0.smithi125.stdout:462: expect deleted 2021-03-24T21:56:01.127 INFO:tasks.rados.rados.0.smithi125.stdout:463: write oid 18 current snap is 43 2021-03-24T21:56:01.127 INFO:tasks.rados.rados.0.smithi125.stdout:463: seq_num 194 ranges {460558=707131,1869389=508716,3094884=165674} 2021-03-24T21:56:01.134 INFO:tasks.rados.rados.0.smithi125.stdout:463: writing smithi12556470-18 from 460558 to 1167689 tid 1 2021-03-24T21:56:01.141 INFO:tasks.rados.rados.0.smithi125.stdout:463: writing smithi12556470-18 from 1869389 to 2378105 tid 2 2021-03-24T21:56:01.147 INFO:tasks.rados.rados.0.smithi125.stdout:463: writing smithi12556470-18 from 3094884 to 3260558 tid 3 2021-03-24T21:56:01.147 INFO:tasks.rados.rados.0.smithi125.stdout:464: rollback oid 34 current snap is 43 2021-03-24T21:56:01.147 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 34 to 42 2021-03-24T21:56:01.147 INFO:tasks.rados.rados.0.smithi125.stdout:463: finishing write tid 1 to smithi12556470-18 2021-03-24T21:56:01.148 INFO:tasks.rados.rados.0.smithi125.stdout:462: done (2 left) 2021-03-24T21:56:01.148 INFO:tasks.rados.rados.0.smithi125.stdout:465: write oid 6 current snap is 43 2021-03-24T21:56:01.148 INFO:tasks.rados.rados.0.smithi125.stdout:463: finishing write tid 2 to smithi12556470-18 2021-03-24T21:56:01.148 INFO:tasks.rados.rados.0.smithi125.stdout:465: seq_num 195 ranges {477365=782380,1927871=768727,3122307=155058} 2021-03-24T21:56:01.153 INFO:tasks.rados.rados.0.smithi125.stdout:465: writing smithi12556470-6 from 477365 to 1259745 tid 1 2021-03-24T21:56:01.161 INFO:tasks.rados.rados.0.smithi125.stdout:465: writing smithi12556470-6 from 1927871 to 2696598 tid 2 2021-03-24T21:56:01.165 INFO:tasks.rados.rados.0.smithi125.stdout:465: writing smithi12556470-6 from 3122307 to 3277365 tid 3 2021-03-24T21:56:01.166 INFO:tasks.rados.rados.0.smithi125.stdout:466: read oid 32 snap 42 2021-03-24T21:56:01.166 INFO:tasks.rados.rados.0.smithi125.stdout:466: expect (ObjNum 108 snap 22 seq_num 108) 2021-03-24T21:56:01.166 INFO:tasks.rados.rados.0.smithi125.stdout:467: write (excl) oid 12 current snap is 43 2021-03-24T21:56:01.166 INFO:tasks.rados.rados.0.smithi125.stdout:464: finishing rollback tid 1 to smithi12556470-34 2021-03-24T21:56:01.167 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 34 v 0 (ObjNum 0 snap 3 seq_num 7) dirty dne 2021-03-24T21:56:01.167 INFO:tasks.rados.rados.0.smithi125.stdout:467: seq_num 196 ranges {494172=457629,1586353=512385,2749730=544442} 2021-03-24T21:56:01.167 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:01.171 INFO:tasks.rados.rados.0.smithi125.stdout:467: writing smithi12556470-12 from 494172 to 951801 tid 1 2021-03-24T21:56:01.177 INFO:tasks.rados.rados.0.smithi125.stdout:467: writing smithi12556470-12 from 1586353 to 2098738 tid 2 2021-03-24T21:56:01.185 INFO:tasks.rados.rados.0.smithi125.stdout:467: writing smithi12556470-12 from 2749730 to 3294172 tid 3 2021-03-24T21:56:01.186 INFO:tasks.rados.rados.0.smithi125.stdout:464: done (4 left) 2021-03-24T21:56:01.186 INFO:tasks.rados.rados.0.smithi125.stdout:468: read oid 38 snap 42 2021-03-24T21:56:01.186 INFO:tasks.rados.rados.0.smithi125.stdout:468: expect (ObjNum 186 snap 40 seq_num 186) 2021-03-24T21:56:01.186 INFO:tasks.rados.rados.0.smithi125.stdout:469: write (excl) oid 34 current snap is 43 2021-03-24T21:56:01.186 INFO:tasks.rados.rados.0.smithi125.stdout:463: finishing write tid 3 to smithi12556470-18 2021-03-24T21:56:01.187 INFO:tasks.rados.rados.0.smithi125.stdout:469: seq_num 197 ranges {510979=532878,1528482=772396,3060800=250179} 2021-03-24T21:56:01.191 INFO:tasks.rados.rados.0.smithi125.stdout:469: writing smithi12556470-34 from 510979 to 1043857 tid 1 2021-03-24T21:56:01.199 INFO:tasks.rados.rados.0.smithi125.stdout:469: writing smithi12556470-34 from 1528482 to 2300878 tid 2 2021-03-24T21:56:01.206 INFO:tasks.rados.rados.0.smithi125.stdout:469: writing smithi12556470-34 from 3060800 to 3310979 tid 3 2021-03-24T21:56:01.206 INFO:tasks.rados.rados.0.smithi125.stdout:470: delete oid 16 current snap is 43 2021-03-24T21:56:01.206 INFO:tasks.rados.rados.0.smithi125.stdout:463: finishing write tid 4 to smithi12556470-18 2021-03-24T21:56:01.206 INFO:tasks.rados.rados.0.smithi125.stdout:463: finishing write tid 5 to smithi12556470-18 2021-03-24T21:56:01.207 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 18 v 104 (ObjNum 194 snap 43 seq_num 194) dirty exists 2021-03-24T21:56:01.207 INFO:tasks.rados.rados.0.smithi125.stdout:463: left oid 18 (ObjNum 194 snap 43 seq_num 194) 2021-03-24T21:56:01.207 INFO:tasks.rados.rados.0.smithi125.stdout:465: finishing write tid 1 to smithi12556470-6 2021-03-24T21:56:01.207 INFO:tasks.rados.rados.0.smithi125.stdout:465: finishing write tid 2 to smithi12556470-6 2021-03-24T21:56:01.208 INFO:tasks.rados.rados.0.smithi125.stdout:465: finishing write tid 3 to smithi12556470-6 2021-03-24T21:56:01.208 INFO:tasks.rados.rados.0.smithi125.stdout:465: finishing write tid 4 to smithi12556470-6 2021-03-24T21:56:01.208 INFO:tasks.rados.rados.0.smithi125.stdout:465: finishing write tid 5 to smithi12556470-6 2021-03-24T21:56:01.208 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 6 v 176 (ObjNum 195 snap 43 seq_num 195) dirty exists 2021-03-24T21:56:01.208 INFO:tasks.rados.rados.0.smithi125.stdout:465: left oid 6 (ObjNum 195 snap 43 seq_num 195) 2021-03-24T21:56:01.209 INFO:tasks.rados.rados.0.smithi125.stdout:467: finishing write tid 1 to smithi12556470-12 2021-03-24T21:56:01.209 INFO:tasks.rados.rados.0.smithi125.stdout:467: finishing write tid 2 to smithi12556470-12 2021-03-24T21:56:01.209 INFO:tasks.rados.rados.0.smithi125.stdout:466: expect (ObjNum 108 snap 22 seq_num 108) 2021-03-24T21:56:01.222 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:01.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:01.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:56:01.327 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:01.338 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:01 smithi125 conmon[19936]: cluster 2021-03-24T21:55:59.472386+0000 osd.3 (osd. 2021-03-24T21:56:01.338 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:01 smithi125 conmon[19936]: 3) 56 : cluster [DBG] 6.c scrub ok 2021-03-24T21:56:01.338 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:01 smithi125 conmon[19936]: cluster 2021-03-24T21:56:01.123161+0000 mon.a (mon.0) 781 : cluster [DBG] osdmap e192: 8 total, 8 up, 6 in 2021-03-24T21:56:01.339 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:01 smithi125 conmon[24927]: cluster 2021-03-24T21:56:01.339 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:01 smithi125 conmon[24927]: 2021-03-24T21:56:01.339 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:01 smithi125 conmon[24927]: 2021-03-24T21 2021-03-24T21:56:01.339 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:01 smithi125 conmon[24927]: :55:59.472386+0000 osd.3 (osd.3) 56 : cluster [DBG] 6.c scrub ok 2021-03-24T21:56:01.340 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:01 smithi125 conmon[24927]: cluster 2021-03-24T21:56:01.340 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:01 smithi125 conmon[24927]: 2021-03-24T21:56:01.123161+0000 2021-03-24T21:56:01.340 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:01 smithi125 conmon[24927]: mon.a (mon.0) 781 : cluster [DBG] osdmap e192: 8 total, 8 up, 6 in 2021-03-24T21:56:01.340 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:01 smithi125 conmon[24927]: 2021-03-24T21:56:01.377 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:01.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:01.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:56:01.391 INFO:tasks.rados.rados.0.smithi125.stdout:467: finishing write tid 3 to smithi12556470-12 2021-03-24T21:56:01.391 INFO:tasks.rados.rados.0.smithi125.stdout:467: finishing write tid 4 to smithi12556470-12 2021-03-24T21:56:01.391 INFO:tasks.rados.rados.0.smithi125.stdout:467: finishing write tid 5 to smithi12556470-12 2021-03-24T21:56:01.392 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 12 v 180 (ObjNum 196 snap 43 seq_num 196) dirty exists 2021-03-24T21:56:01.392 INFO:tasks.rados.rados.0.smithi125.stdout:467: left oid 12 (ObjNum 196 snap 43 seq_num 196) 2021-03-24T21:56:01.392 INFO:tasks.rados.rados.0.smithi125.stdout:468: expect (ObjNum 186 snap 40 seq_num 186) 2021-03-24T21:56:01.483 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:01.527 INFO:tasks.rados.rados.0.smithi125.stdout:469: finishing write tid 1 to smithi12556470-34 2021-03-24T21:56:01.528 INFO:tasks.rados.rados.0.smithi125.stdout:469: finishing write tid 2 to smithi12556470-34 2021-03-24T21:56:01.528 INFO:tasks.rados.rados.0.smithi125.stdout:469: finishing write tid 3 to smithi12556470-34 2021-03-24T21:56:01.528 INFO:tasks.rados.rados.0.smithi125.stdout:469: finishing write tid 4 to smithi12556470-34 2021-03-24T21:56:01.528 INFO:tasks.rados.rados.0.smithi125.stdout:469: finishing write tid 5 to smithi12556470-34 2021-03-24T21:56:01.529 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 34 v 184 (ObjNum 197 snap 43 seq_num 197) dirty exists 2021-03-24T21:56:01.529 INFO:tasks.rados.rados.0.smithi125.stdout:469: left oid 34 (ObjNum 197 snap 43 seq_num 197) 2021-03-24T21:56:01.529 INFO:tasks.rados.rados.0.smithi125.stdout:463: done (6 left) 2021-03-24T21:56:01.529 INFO:tasks.rados.rados.0.smithi125.stdout:465: done (5 left) 2021-03-24T21:56:01.529 INFO:tasks.rados.rados.0.smithi125.stdout:466: done (4 left) 2021-03-24T21:56:01.530 INFO:tasks.rados.rados.0.smithi125.stdout:467: done (3 left) 2021-03-24T21:56:01.530 INFO:tasks.rados.rados.0.smithi125.stdout:468: done (2 left) 2021-03-24T21:56:01.530 INFO:tasks.rados.rados.0.smithi125.stdout:469: done (1 left) 2021-03-24T21:56:01.530 INFO:tasks.rados.rados.0.smithi125.stdout:470: done (0 left) 2021-03-24T21:56:01.531 INFO:tasks.rados.rados.0.smithi125.stdout:471: write oid 47 current snap is 43 2021-03-24T21:56:01.531 INFO:tasks.rados.rados.0.smithi125.stdout:471: seq_num 199 ranges {544593=799729,1761799=776065,2948352=396241} 2021-03-24T21:56:01.535 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:01.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:01.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:56:01.541 INFO:tasks.rados.rados.0.smithi125.stdout:471: writing smithi12556470-47 from 544593 to 1344322 tid 1 2021-03-24T21:56:01.547 INFO:tasks.rados.rados.0.smithi125.stdout:471: writing smithi12556470-47 from 1761799 to 2537864 tid 2 2021-03-24T21:56:01.553 INFO:tasks.rados.rados.0.smithi125.stdout:471: writing smithi12556470-47 from 2948352 to 3344593 tid 3 2021-03-24T21:56:01.553 INFO:tasks.rados.rados.0.smithi125.stdout:472: copy_from oid 10 from oid 7 current snap is 43 2021-03-24T21:56:01.553 INFO:tasks.rados.rados.0.smithi125.stdout:471: finishing write tid 1 to smithi12556470-47 2021-03-24T21:56:01.554 INFO:tasks.rados.rados.0.smithi125.stdout:473: rollback oid 45 current snap is 43 2021-03-24T21:56:01.554 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 45 to 42 2021-03-24T21:56:01.554 INFO:tasks.rados.rados.0.smithi125.stdout:474: read oid 27 snap -1 2021-03-24T21:56:01.554 INFO:tasks.rados.rados.0.smithi125.stdout:474: expect (ObjNum 189 snap 41 seq_num 189) 2021-03-24T21:56:01.555 INFO:tasks.rados.rados.0.smithi125.stdout:475: write (excl) oid 20 current snap is 43 2021-03-24T21:56:01.555 INFO:tasks.rados.rados.0.smithi125.stdout:475: seq_num 200 ranges {561400=474978,1820281=519723,2859422=457579} 2021-03-24T21:56:01.559 INFO:tasks.rados.rados.0.smithi125.stdout:475: writing smithi12556470-20 from 561400 to 1036378 tid 1 2021-03-24T21:56:01.566 INFO:tasks.rados.rados.0.smithi125.stdout:475: writing smithi12556470-20 from 1820281 to 2340004 tid 2 2021-03-24T21:56:01.572 INFO:tasks.rados.rados.0.smithi125.stdout:475: writing smithi12556470-20 from 2859422 to 3317001 tid 3 2021-03-24T21:56:01.572 INFO:tasks.rados.rados.0.smithi125.stdout:476: write oid 13 current snap is 43 2021-03-24T21:56:01.573 INFO:tasks.rados.rados.0.smithi125.stdout:471: finishing write tid 2 to smithi12556470-47 2021-03-24T21:56:01.573 INFO:tasks.rados.rados.0.smithi125.stdout:476: seq_num 201 ranges {578207=550227,1762410=779734,3286845=91362} 2021-03-24T21:56:01.578 INFO:tasks.rados.rados.0.smithi125.stdout:476: writing smithi12556470-13 from 578207 to 1128434 tid 1 2021-03-24T21:56:01.586 INFO:tasks.rados.rados.0.smithi125.stdout:476: writing smithi12556470-13 from 1762410 to 2542144 tid 2 2021-03-24T21:56:01.592 INFO:tasks.rados.rados.0.smithi125.stdout:476: writing smithi12556470-13 from 3286845 to 3378207 tid 3 2021-03-24T21:56:01.592 INFO:tasks.rados.rados.0.smithi125.stdout:477: rollback oid 48 current snap is 43 2021-03-24T21:56:01.592 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 48 to 42 2021-03-24T21:56:01.592 INFO:tasks.rados.rados.0.smithi125.stdout:473: finishing rollback tid 0 to smithi12556470-45 2021-03-24T21:56:01.593 INFO:tasks.rados.rados.0.smithi125.stdout:473: finishing rollback tid 1 to smithi12556470-45 2021-03-24T21:56:01.593 INFO:tasks.rados.rados.0.smithi125.stdout:473: finishing rollback tid 2 to smithi12556470-45 2021-03-24T21:56:01.593 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 45 v 147 (ObjNum 182 snap 40 seq_num 182) dirty exists 2021-03-24T21:56:01.593 INFO:tasks.rados.rados.0.smithi125.stdout:471: finishing write tid 3 to smithi12556470-47 2021-03-24T21:56:01.594 INFO:tasks.rados.rados.0.smithi125.stdout:471: finishing write tid 4 to smithi12556470-47 2021-03-24T21:56:01.594 INFO:tasks.rados.rados.0.smithi125.stdout:473: done (6 left) 2021-03-24T21:56:01.594 INFO:tasks.rados.rados.0.smithi125.stdout:478: write oid 28 current snap is 43 2021-03-24T21:56:01.594 INFO:tasks.rados.rados.0.smithi125.stdout:478: seq_num 202 ranges {595014=625476,1820892=523392,2797915=596476} 2021-03-24T21:56:01.599 INFO:tasks.rados.rados.0.smithi125.stdout:478: writing smithi12556470-28 from 595014 to 1220490 tid 1 2021-03-24T21:56:01.605 INFO:tasks.rados.rados.0.smithi125.stdout:478: writing smithi12556470-28 from 1820892 to 2344284 tid 2 2021-03-24T21:56:01.612 INFO:tasks.rados.rados.0.smithi125.stdout:478: writing smithi12556470-28 from 2797915 to 3394391 tid 3 2021-03-24T21:56:01.612 INFO:tasks.rados.rados.0.smithi125.stdout:479: read oid 29 snap -1 2021-03-24T21:56:01.612 INFO:tasks.rados.rados.0.smithi125.stdout:479: expect (ObjNum 101 snap 20 seq_num 101) 2021-03-24T21:56:01.613 INFO:tasks.rados.rados.0.smithi125.stdout:480: read oid 41 snap 42 2021-03-24T21:56:01.613 INFO:tasks.rados.rados.0.smithi125.stdout:480: expect (ObjNum 188 snap 41 seq_num 188) 2021-03-24T21:56:01.613 INFO:tasks.rados.rados.0.smithi125.stdout:481: write (excl) oid 34 current snap is 43 2021-03-24T21:56:01.613 INFO:tasks.rados.rados.0.smithi125.stdout:471: finishing write tid 5 to smithi12556470-47 2021-03-24T21:56:01.614 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 47 v 92 (ObjNum 199 snap 43 seq_num 199) dirty exists 2021-03-24T21:56:01.614 INFO:tasks.rados.rados.0.smithi125.stdout:471: left oid 47 (ObjNum 199 snap 43 seq_num 199) 2021-03-24T21:56:01.614 INFO:tasks.rados.rados.0.smithi125.stdout:481: seq_num 203 ranges {611821=700725,1879374=783403,3341691=70130} 2021-03-24T21:56:01.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:01 smithi142 conmon[20125]: cluster 2021-03-24T21:55:59.472386+0000 osd.3 ( 2021-03-24T21:56:01.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:01 smithi142 conmon[20125]: osd.3) 56 : cluster [DBG] 6.c scrub ok 2021-03-24T21:56:01.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:01 smithi142 conmon[20125]: cluster 2021-03-24T21:56:01.123161+0000 mon.a (mon.0) 781 : cluster [DBG] osdmap e192: 8 total, 8 up, 6 in 2021-03-24T21:56:01.620 INFO:tasks.rados.rados.0.smithi125.stdout:481: writing smithi12556470-34 from 611821 to 1312546 tid 1 2021-03-24T21:56:01.627 INFO:tasks.rados.rados.0.smithi125.stdout:481: writing smithi12556470-34 from 1879374 to 2662777 tid 2 2021-03-24T21:56:01.632 INFO:tasks.rados.rados.0.smithi125.stdout:481: writing smithi12556470-34 from 3341691 to 3411821 tid 3 2021-03-24T21:56:01.632 INFO:tasks.rados.rados.0.smithi125.stdout:471: done (9 left) 2021-03-24T21:56:01.632 INFO:tasks.rados.rados.0.smithi125.stdout:482: snap_create 2021-03-24T21:56:01.632 INFO:tasks.rados.rados.0.smithi125.stdout:474: expect (ObjNum 189 snap 41 seq_num 189) 2021-03-24T21:56:01.642 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:01.700 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:01.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:01.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:56:01.804 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:01.858 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:01.862 INFO:tasks.rados.rados.0.smithi125.stdout:475: finishing write tid 1 to smithi12556470-20 2021-03-24T21:56:01.862 INFO:tasks.rados.rados.0.smithi125.stdout:472: finishing copy_from to smithi12556470-10 2021-03-24T21:56:01.862 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 10 v 92 (ObjNum 150 snap 30 seq_num 150) dirty exists 2021-03-24T21:56:01.862 INFO:tasks.rados.rados.0.smithi125.stdout:472: finishing copy_from racing read to smithi12556470-10 2021-03-24T21:56:01.863 INFO:tasks.rados.rados.0.smithi125.stdout:475: finishing write tid 2 to smithi12556470-20 2021-03-24T21:56:01.863 INFO:tasks.rados.rados.0.smithi125.stdout:475: finishing write tid 3 to smithi12556470-20 2021-03-24T21:56:01.863 INFO:tasks.rados.rados.0.smithi125.stdout:475: finishing write tid 4 to smithi12556470-20 2021-03-24T21:56:01.863 INFO:tasks.rados.rados.0.smithi125.stdout:475: finishing write tid 5 to smithi12556470-20 2021-03-24T21:56:01.863 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 20 v 129 (ObjNum 200 snap 43 seq_num 200) dirty exists 2021-03-24T21:56:01.864 INFO:tasks.rados.rados.0.smithi125.stdout:475: left oid 20 (ObjNum 200 snap 43 seq_num 200) 2021-03-24T21:56:01.864 INFO:tasks.rados.rados.0.smithi125.stdout:476: finishing write tid 1 to smithi12556470-13 2021-03-24T21:56:01.864 INFO:tasks.rados.rados.0.smithi125.stdout:476: finishing write tid 2 to smithi12556470-13 2021-03-24T21:56:01.864 INFO:tasks.rados.rados.0.smithi125.stdout:476: finishing write tid 3 to smithi12556470-13 2021-03-24T21:56:01.865 INFO:tasks.rados.rados.0.smithi125.stdout:476: finishing write tid 4 to smithi12556470-13 2021-03-24T21:56:01.865 INFO:tasks.rados.rados.0.smithi125.stdout:476: finishing write tid 5 to smithi12556470-13 2021-03-24T21:56:01.865 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 13 v 189 (ObjNum 201 snap 43 seq_num 201) dirty exists 2021-03-24T21:56:01.865 INFO:tasks.rados.rados.0.smithi125.stdout:476: left oid 13 (ObjNum 201 snap 43 seq_num 201) 2021-03-24T21:56:01.865 INFO:tasks.rados.rados.0.smithi125.stdout:477: finishing rollback tid 0 to smithi12556470-48 2021-03-24T21:56:01.866 INFO:tasks.rados.rados.0.smithi125.stdout:477: finishing rollback tid 1 to smithi12556470-48 2021-03-24T21:56:01.866 INFO:tasks.rados.rados.0.smithi125.stdout:477: finishing rollback tid 2 to smithi12556470-48 2021-03-24T21:56:01.866 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 48 v 168 (ObjNum 62 snap 4 seq_num 62) dirty exists 2021-03-24T21:56:01.866 INFO:tasks.rados.rados.0.smithi125.stdout:478: finishing write tid 1 to smithi12556470-28 2021-03-24T21:56:01.867 INFO:tasks.rados.rados.0.smithi125.stdout:478: finishing write tid 2 to smithi12556470-28 2021-03-24T21:56:01.867 INFO:tasks.rados.rados.0.smithi125.stdout:479: expect (ObjNum 101 snap 20 seq_num 101) 2021-03-24T21:56:01.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:01.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:56:01.969 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:02.018 INFO:tasks.rados.rados.0.smithi125.stdout:478: finishing write tid 3 to smithi12556470-28 2021-03-24T21:56:02.018 INFO:tasks.rados.rados.0.smithi125.stdout:478: finishing write tid 4 to smithi12556470-28 2021-03-24T21:56:02.018 INFO:tasks.rados.rados.0.smithi125.stdout:478: finishing write tid 5 to smithi12556470-28 2021-03-24T21:56:02.019 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 28 v 134 (ObjNum 202 snap 43 seq_num 202) dirty exists 2021-03-24T21:56:02.019 INFO:tasks.rados.rados.0.smithi125.stdout:478: left oid 28 (ObjNum 202 snap 43 seq_num 202) 2021-03-24T21:56:02.019 INFO:tasks.rados.rados.0.smithi125.stdout:480: expect (ObjNum 188 snap 41 seq_num 188) 2021-03-24T21:56:02.027 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:02.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:02.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:56:02.045 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.6 dump_historic_ops 2021-03-24T21:56:02.132 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:02.184 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:02.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:02.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:56:02.243 INFO:tasks.rados.rados.0.smithi125.stdout:481: finishing write tid 1 to smithi12556470-34 2021-03-24T21:56:02.243 INFO:tasks.rados.rados.0.smithi125.stdout:481: finishing write tid 2 to smithi12556470-34 2021-03-24T21:56:02.244 INFO:tasks.rados.rados.0.smithi125.stdout:481: finishing write tid 3 to smithi12556470-34 2021-03-24T21:56:02.244 INFO:tasks.rados.rados.0.smithi125.stdout:481: finishing write tid 4 to smithi12556470-34 2021-03-24T21:56:02.244 INFO:tasks.rados.rados.0.smithi125.stdout:481: finishing write tid 5 to smithi12556470-34 2021-03-24T21:56:02.244 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 34 v 193 (ObjNum 203 snap 43 seq_num 203) dirty exists 2021-03-24T21:56:02.245 INFO:tasks.rados.rados.0.smithi125.stdout:481: left oid 34 (ObjNum 203 snap 43 seq_num 203) 2021-03-24T21:56:02.245 INFO:tasks.rados.rados.0.smithi125.stdout:472: done (9 left) 2021-03-24T21:56:02.245 INFO:tasks.rados.rados.0.smithi125.stdout:474: done (8 left) 2021-03-24T21:56:02.245 INFO:tasks.rados.rados.0.smithi125.stdout:475: done (7 left) 2021-03-24T21:56:02.245 INFO:tasks.rados.rados.0.smithi125.stdout:476: done (6 left) 2021-03-24T21:56:02.246 INFO:tasks.rados.rados.0.smithi125.stdout:477: done (5 left) 2021-03-24T21:56:02.246 INFO:tasks.rados.rados.0.smithi125.stdout:478: done (4 left) 2021-03-24T21:56:02.246 INFO:tasks.rados.rados.0.smithi125.stdout:479: done (3 left) 2021-03-24T21:56:02.246 INFO:tasks.rados.rados.0.smithi125.stdout:480: done (2 left) 2021-03-24T21:56:02.247 INFO:tasks.rados.rados.0.smithi125.stdout:481: done (1 left) 2021-03-24T21:56:02.247 INFO:tasks.rados.rados.0.smithi125.stdout:482: done (0 left) 2021-03-24T21:56:02.247 INFO:tasks.rados.rados.0.smithi125.stdout:483: delete oid 24 current snap is 44 2021-03-24T21:56:02.248 INFO:tasks.rados.rados.0.smithi125.stdout:483: done (0 left) 2021-03-24T21:56:02.248 INFO:tasks.rados.rados.0.smithi125.stdout:484: write oid 5 current snap is 44 2021-03-24T21:56:02.248 INFO:tasks.rados.rados.0.smithi125.stdout:484: seq_num 205 ranges {645435=451223,1879985=787072,3280184=165251} 2021-03-24T21:56:02.254 INFO:tasks.rados.rados.0.smithi125.stdout:484: writing smithi12556470-5 from 645435 to 1096658 tid 1 2021-03-24T21:56:02.266 INFO:tasks.rados.rados.0.smithi125.stdout:484: writing smithi12556470-5 from 1879985 to 2667057 tid 2 2021-03-24T21:56:02.271 INFO:tasks.rados.rados.0.smithi125.stdout:484: writing smithi12556470-5 from 3280184 to 3445435 tid 3 2021-03-24T21:56:02.271 INFO:tasks.rados.rados.0.smithi125.stdout:485: read oid 17 snap -1 2021-03-24T21:56:02.271 INFO:tasks.rados.rados.0.smithi125.stdout:485: expect (ObjNum 192 snap 42 seq_num 192) 2021-03-24T21:56:02.272 INFO:tasks.rados.rados.0.smithi125.stdout:486: read oid 6 snap -1 2021-03-24T21:56:02.272 INFO:tasks.rados.rados.0.smithi125.stdout:486: expect (ObjNum 195 snap 43 seq_num 195) 2021-03-24T21:56:02.272 INFO:tasks.rados.rados.0.smithi125.stdout:487: read oid 48 snap -1 2021-03-24T21:56:02.272 INFO:tasks.rados.rados.0.smithi125.stdout:487: expect (ObjNum 62 snap 4 seq_num 62) 2021-03-24T21:56:02.272 INFO:tasks.rados.rados.0.smithi125.stdout:488: write oid 10 current snap is 44 2021-03-24T21:56:02.273 INFO:tasks.rados.rados.0.smithi125.stdout:488: seq_num 206 ranges {662242=642825,2054820=530730,3307607=154635} 2021-03-24T21:56:02.279 INFO:tasks.rados.rados.0.smithi125.stdout:488: writing smithi12556470-10 from 662242 to 1305067 tid 1 2021-03-24T21:56:02.286 INFO:tasks.rados.rados.0.smithi125.stdout:488: writing smithi12556470-10 from 2054820 to 2585550 tid 2 2021-03-24T21:56:02.290 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:02.291 INFO:tasks.rados.rados.0.smithi125.stdout:488: writing smithi12556470-10 from 3307607 to 3462242 tid 3 2021-03-24T21:56:02.292 INFO:tasks.rados.rados.0.smithi125.stdout:489: read oid 43 snap 42 2021-03-24T21:56:02.292 INFO:tasks.rados.rados.0.smithi125.stdout:489: expect (ObjNum 59 snap 4 seq_num 59) 2021-03-24T21:56:02.293 INFO:tasks.rados.rados.0.smithi125.stdout:490: read oid 1 snap -1 2021-03-24T21:56:02.293 INFO:tasks.rados.rados.0.smithi125.stdout:490: expect (ObjNum 143 snap 27 seq_num 143) 2021-03-24T21:56:02.293 INFO:tasks.rados.rados.0.smithi125.stdout:484: finishing write tid 1 to smithi12556470-5 2021-03-24T21:56:02.294 INFO:tasks.rados.rados.0.smithi125.stdout:484: finishing write tid 2 to smithi12556470-5 2021-03-24T21:56:02.294 INFO:tasks.rados.rados.0.smithi125.stdout:486: expect (ObjNum 195 snap 43 seq_num 195) 2021-03-24T21:56:02.333 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:02.336 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:02 smithi125 conmon[19936]: cluster 2021-03-24T21:56:00. 2021-03-24T21:56:02.336 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:02 smithi125 conmon[19936]: 739250+0000 mgr.y (mgr.14144) 356 : cluster [DBG] pgmap v449: 148 pgs: 148 active+clean; 118 MiB data, 386 MiB used, 536 GiB / 536 GiB avail; 2.8 MiB/s rd, 4.4 MiB/s wr, 12 op/s 2021-03-24T21:56:02.337 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:02 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:56:02.337 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:02 smithi125 conmon[19936]: :56:00.986583+0000 osd.7 (osd.7) 44 : cluster [DBG] 6.15 scrub ok 2021-03-24T21:56:02.337 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:02 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:56:02.337 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:02 smithi125 conmon[19936]: -24T21:56:01.124557+0000 osd.2 ( 2021-03-24T21:56:02.338 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:02 smithi125 conmon[19936]: osd.2) 36 : cluster [DBG] 6.1 scrub ok 2021-03-24T21:56:02.338 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:02 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:56:02.338 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:02 smithi125 conmon[19936]: 24T21:56:01.266928+0000 osd.0 (osd 2021-03-24T21:56:02.338 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:02 smithi125 conmon[19936]: .0) 54 : cluster [DBG] 5.3 scrub ok 2021-03-24T21:56:02.339 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:02 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:56:02.339 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:02 smithi125 conmon[19936]: -24T21:56:01.308039+0000 osd.5 ( 2021-03-24T21:56:02.339 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:02 smithi125 conmon[19936]: osd.5) 55 : cluster [DBG] 7.b scrub ok 2021-03-24T21:56:02.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:02.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:56:02.341 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:02 smithi125 conmon[24927]: cluster 2021-03-24T21:56 2021-03-24T21:56:02.341 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:02 smithi125 conmon[24927]: :00.739250+0000 mgr.y (mgr.14144) 356 : cluster [DBG] pgmap v449: 148 pgs: 148 active+clean; 118 MiB data, 386 MiB used, 536 GiB / 536 GiB avail; 2.8 MiB/s rd, 4.4 MiB/s wr, 12 op/s 2021-03-24T21:56:02.341 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:02 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:56:02.342 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:02 smithi125 conmon[24927]: -03-24T21:56:00.986583 2021-03-24T21:56:02.342 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:02 smithi125 conmon[24927]: +0000 osd.7 (osd.7) 44 : cluster 2021-03-24T21:56:02.342 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:02 smithi125 conmon[24927]: [DBG] 6.15 scrub ok 2021-03-24T21:56:02.342 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:02 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:56:02.343 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:02 smithi125 conmon[24927]: -24T21:56:01.124557+0000 2021-03-24T21:56:02.343 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:02 smithi125 conmon[24927]: osd.2 (osd.2) 36 : cluster 2021-03-24T21:56:02.343 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:02 smithi125 conmon[24927]: [DBG] 6.1 scrub ok 2021-03-24T21:56:02.343 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:02 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:56:02.343 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:02 smithi125 conmon[24927]: T21:56:01.266928+0000 osd.0 2021-03-24T21:56:02.344 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:02 smithi125 conmon[24927]: (osd.0) 54 : cluster [DBG] 2021-03-24T21:56:02.344 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:02 smithi125 conmon[24927]: 5.3 scrub ok 2021-03-24T21:56:02.344 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:02 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:56:02.344 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:02 smithi125 conmon[24927]: 03-24T21:56:01.308039+0000 2021-03-24T21:56:02.345 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:02 smithi125 conmon[24927]: osd.5 (osd.5) 55 : cluster [DBG] 2021-03-24T21:56:02.345 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:02 smithi125 conmon[24927]: 7.b scrub ok 2021-03-24T21:56:02.345 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T21:56:02.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:02 smithi142 conmon[20125]: cluster 2021-03-24T21:56:00.739250+0000 mgr.y (mgr.14144) 356 : cluster [DBG] pgmap v449: 148 pgs: 148 active+clean; 118 MiB data, 386 MiB used, 536 GiB / 536 GiB avail; 2.8 MiB/s rd, 4.4 MiB/s wr, 12 op/s 2021-03-24T21:56:02.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:02 smithi142 conmon[20125]: cluster 2021-03-24T21:56:00.986583+0000 osd.7 (osd.7) 44 : cluster [DBG] 6.15 scrub ok 2021-03-24T21:56:02.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:02 smithi142 conmon[20125]: cluster 2021-03-24T21:56:01.124557+0000 osd.2 (osd.2) 36 : cluster [DBG] 6.1 scrub ok 2021-03-24T21:56:02.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:02 smithi142 conmon[20125]: cluster 2021-03-24T21:56:01.266928+0000 osd.0 2021-03-24T21:56:02.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:02 smithi142 conmon[20125]: (osd.0) 54 : cluster [DBG] 5.3 scrub ok 2021-03-24T21:56:02.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:02 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:56:02.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:02 smithi142 conmon[20125]: 21:56:01.308039+0000 osd.5 (osd.5) 55 : cluster [DBG] 7.b scrub ok 2021-03-24T21:56:02.406 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-24T21:56:02.442 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:02.493 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:02.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:02.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:56:02.514 INFO:tasks.rados.rados.0.smithi125.stdout:486: done (6 left) 2021-03-24T21:56:02.514 INFO:tasks.rados.rados.0.smithi125.stdout:491: rollback oid 50 current snap is 44 2021-03-24T21:56:02.514 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 50 to 42 2021-03-24T21:56:02.515 INFO:tasks.rados.rados.0.smithi125.stdout:484: finishing write tid 3 to smithi12556470-5 2021-03-24T21:56:02.515 INFO:tasks.rados.rados.0.smithi125.stdout:484: finishing write tid 4 to smithi12556470-5 2021-03-24T21:56:02.515 INFO:tasks.rados.rados.0.smithi125.stdout:484: finishing write tid 5 to smithi12556470-5 2021-03-24T21:56:02.515 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 5 v 169 (ObjNum 205 snap 44 seq_num 205) dirty exists 2021-03-24T21:56:02.515 INFO:tasks.rados.rados.0.smithi125.stdout:484: left oid 5 (ObjNum 205 snap 44 seq_num 205) 2021-03-24T21:56:02.516 INFO:tasks.rados.rados.0.smithi125.stdout:485: expect (ObjNum 192 snap 42 seq_num 192) 2021-03-24T21:56:02.600 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:02.663 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:02.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:02.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:56:02.684 INFO:tasks.rados.rados.0.smithi125.stdout:487: expect (ObjNum 62 snap 4 seq_num 62) 2021-03-24T21:56:02.767 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:02.819 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:02.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:02.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:56:02.825 INFO:tasks.rados.rados.0.smithi125.stdout:488: finishing write tid 1 to smithi12556470-10 2021-03-24T21:56:02.825 INFO:tasks.rados.rados.0.smithi125.stdout:488: finishing write tid 2 to smithi12556470-10 2021-03-24T21:56:02.826 INFO:tasks.rados.rados.0.smithi125.stdout:484: done (6 left) 2021-03-24T21:56:02.826 INFO:tasks.rados.rados.0.smithi125.stdout:485: done (5 left) 2021-03-24T21:56:02.826 INFO:tasks.rados.rados.0.smithi125.stdout:487: done (4 left) 2021-03-24T21:56:02.826 INFO:tasks.rados.rados.0.smithi125.stdout:492: snap_remove snap 43 2021-03-24T21:56:02.925 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:02.978 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:03.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:03.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:56:03.171 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:03.227 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:03.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:03.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:56:03.232 INFO:tasks.rados.rados.0.smithi125.stdout:492: done (4 left) 2021-03-24T21:56:03.232 INFO:tasks.rados.rados.0.smithi125.stdout:493: write (excl) oid 46 current snap is 44 2021-03-24T21:56:03.232 INFO:tasks.rados.rados.0.smithi125.stdout:493: seq_num 207 ranges {679049=718074,2113302=674388,3335030=144019} 2021-03-24T21:56:03.242 INFO:tasks.rados.rados.0.smithi125.stdout:493: writing smithi12556470-46 from 679049 to 1397123 tid 1 2021-03-24T21:56:03.251 INFO:tasks.rados.rados.0.smithi125.stdout:493: writing smithi12556470-46 from 2113302 to 2787690 tid 2 2021-03-24T21:56:03.255 INFO:tasks.rados.rados.0.smithi125.stdout:493: writing smithi12556470-46 from 3335030 to 3479049 tid 3 2021-03-24T21:56:03.256 INFO:tasks.rados.rados.0.smithi125.stdout:494: delete oid 16 current snap is 44 2021-03-24T21:56:03.256 INFO:tasks.rados.rados.0.smithi125.stdout:488: finishing write tid 3 to smithi12556470-10 2021-03-24T21:56:03.256 INFO:tasks.rados.rados.0.smithi125.stdout:488: finishing write tid 4 to smithi12556470-10 2021-03-24T21:56:03.256 INFO:tasks.rados.rados.0.smithi125.stdout:488: finishing write tid 5 to smithi12556470-10 2021-03-24T21:56:03.256 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 10 v 179 (ObjNum 206 snap 44 seq_num 206) dirty exists 2021-03-24T21:56:03.257 INFO:tasks.rados.rados.0.smithi125.stdout:488: left oid 10 (ObjNum 206 snap 44 seq_num 206) 2021-03-24T21:56:03.257 INFO:tasks.rados.rados.0.smithi125.stdout:490: expect (ObjNum 143 snap 27 seq_num 143) 2021-03-24T21:56:03.333 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:03.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:03 smithi142 conmon[20125]: cluster 2021-03-24T21:56:03.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:03 smithi142 conmon[20125]: 2021-03-24T21 2021-03-24T21:56:03.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:03 smithi142 conmon[20125]: :56:02.021280+0000 osd.4 (osd.4) 54 : cluster [DBG] 4.9 scrub ok 2021-03-24T21:56:03.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:03 smithi142 conmon[20125]: cluster 2021-03-24T21:56:03.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:03 smithi142 conmon[20125]: 2021-03-24T21:56:02.206896+0000 mon.a (mon.0) 782 : cluster [DBG] osdmap e193: 8 total, 8 up, 6 in 2021-03-24T21:56:03.393 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:03.394 INFO:tasks.rados.rados.0.smithi125.stdout:489: expect (ObjNum 59 snap 4 seq_num 59) 2021-03-24T21:56:03.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:03.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:56:03.497 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:03.519 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:03 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:56:03.519 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:03 smithi125 conmon[24927]: 56:02.021280+0000 osd.4 (osd.4) 54 : cluster [DBG] 4.9 scrub ok 2021-03-24T21:56:03.519 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:03 smithi125 conmon[24927]: cluster 2021-03-24T21:56:02. 2021-03-24T21:56:03.519 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:03 smithi125 conmon[24927]: 206896+0000 mon.a (mon.0) 782 : cluster [DBG] 2021-03-24T21:56:03.520 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:03 smithi125 conmon[24927]: osdmap e193: 8 total, 8 up, 6 in 2021-03-24T21:56:03.520 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:03 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:56:03.520 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:03 smithi125 conmon[19936]: :56:02.021280+0000 osd.4 (osd.4) 54 : cluster [DBG] 4.9 scrub ok 2021-03-24T21:56:03.521 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:03 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:56:03.521 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:03 smithi125 conmon[19936]: -24T21:56:02.206896+0000 mon.a 2021-03-24T21:56:03.521 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:03 smithi125 conmon[19936]: (mon.0) 782 : cluster [DBG] osdmap e193: 8 total, 8 up, 6 in 2021-03-24T21:56:03.521 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:03 smithi125 conmon[19936]: 2021-03-24T21:56:03.533 INFO:tasks.rados.rados.0.smithi125.stdout:491: finishing rollback tid 1 to smithi12556470-50 2021-03-24T21:56:03.533 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 50 v 0 (ObjNum 1 snap 32741 seq_num 1116557504) dirty dne 2021-03-24T21:56:03.534 INFO:tasks.rados.rados.0.smithi125.stdout:488: done (5 left) 2021-03-24T21:56:03.534 INFO:tasks.rados.rados.0.smithi125.stdout:489: done (4 left) 2021-03-24T21:56:03.534 INFO:tasks.rados.rados.0.smithi125.stdout:490: done (3 left) 2021-03-24T21:56:03.534 INFO:tasks.rados.rados.0.smithi125.stdout:491: done (2 left) 2021-03-24T21:56:03.534 INFO:tasks.rados.rados.0.smithi125.stdout:494: done (1 left) 2021-03-24T21:56:03.535 INFO:tasks.rados.rados.0.smithi125.stdout:495: snap_remove snap 42 2021-03-24T21:56:03.558 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:56:03.577 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:03.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:03.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:56:03.683 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:03.731 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:03.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:03.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:56:03.837 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:03.942 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:03.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:03.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:56:04.048 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:04.098 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:04.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:04.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:56:04.203 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:04.236 INFO:tasks.rados.rados.0.smithi125.stdout:495: done (1 left) 2021-03-24T21:56:04.237 INFO:tasks.rados.rados.0.smithi125.stdout:496: delete oid 12 current snap is 44 2021-03-24T21:56:04.237 INFO:tasks.rados.rados.0.smithi125.stdout:493: finishing write tid 1 to smithi12556470-46 2021-03-24T21:56:04.237 INFO:tasks.rados.rados.0.smithi125.stdout:493: finishing write tid 2 to smithi12556470-46 2021-03-24T21:56:04.237 INFO:tasks.rados.rados.0.smithi125.stdout:493: finishing write tid 3 to smithi12556470-46 2021-03-24T21:56:04.238 INFO:tasks.rados.rados.0.smithi125.stdout:493: finishing write tid 4 to smithi12556470-46 2021-03-24T21:56:04.238 INFO:tasks.rados.rados.0.smithi125.stdout:493: finishing write tid 5 to smithi12556470-46 2021-03-24T21:56:04.238 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 46 v 60 (ObjNum 207 snap 44 seq_num 207) dirty exists 2021-03-24T21:56:04.238 INFO:tasks.rados.rados.0.smithi125.stdout:493: left oid 46 (ObjNum 207 snap 44 seq_num 207) 2021-03-24T21:56:04.255 INFO:teuthology.orchestra.run.smithi125.stderr:nodeep-scrub is unset 2021-03-24T21:56:04.255 INFO:tasks.rados.rados.0.smithi125.stdout:493: done (1 left) 2021-03-24T21:56:04.256 INFO:tasks.rados.rados.0.smithi125.stdout:496: done (0 left) 2021-03-24T21:56:04.256 INFO:tasks.rados.rados.0.smithi125.stdout:497: snap_create 2021-03-24T21:56:04.273 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:04.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:04.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:56:04.285 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2021-03-24T21:56:04.286 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2021-03-24T21:56:04.286 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2021-03-24T21:56:04.286 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2021-03-24T21:56:04.286 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.287 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.34352.0:91 5.1f 5:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected e65)", 2021-03-24T21:56:04.287 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:51:49.401953+0000", 2021-03-24T21:56:04.287 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 254.87998361699999, 2021-03-24T21:56:04.287 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022343084999999999, 2021-03-24T21:56:04.287 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:56:04.288 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:04.288 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:56:04.288 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.34352", 2021-03-24T21:56:04.288 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/3436348424", 2021-03-24T21:56:04.288 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 91 2021-03-24T21:56:04.289 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.289 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:56:04.289 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.289 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:56:04.290 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.401953+0000", 2021-03-24T21:56:04.290 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:56:04.290 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.290 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.291 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:56:04.291 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.401953+0000", 2021-03-24T21:56:04.291 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999971 2021-03-24T21:56:04.291 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.291 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.292 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:56:04.292 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.401950+0000", 2021-03-24T21:56:04.292 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1336e-05 2021-03-24T21:56:04.292 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.293 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.293 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:56:04.293 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.401962+0000", 2021-03-24T21:56:04.293 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6546999999999999e-05 2021-03-24T21:56:04.293 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.294 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.294 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:56:04.294 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.401978+0000", 2021-03-24T21:56:04.294 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1294000000000001e-05 2021-03-24T21:56:04.295 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.295 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.295 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:56:04.295 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.401990+0000", 2021-03-24T21:56:04.295 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001280642 2021-03-24T21:56:04.296 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.296 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.296 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:56:04.296 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.403270+0000", 2021-03-24T21:56:04.297 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7759000000000002e-05 2021-03-24T21:56:04.297 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.297 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.297 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for peered", 2021-03-24T21:56:04.297 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.403308+0000", 2021-03-24T21:56:04.298 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.016450790999999999 2021-03-24T21:56:04.298 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.298 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.298 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:56:04.298 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.419759+0000", 2021-03-24T21:56:04.299 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013526900000000001 2021-03-24T21:56:04.299 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.299 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.299 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:56:04.300 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.419894+0000", 2021-03-24T21:56:04.300 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00017398199999999999 2021-03-24T21:56:04.300 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.300 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.300 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 4,7", 2021-03-24T21:56:04.301 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.420068+0000", 2021-03-24T21:56:04.301 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00073749500000000001 2021-03-24T21:56:04.301 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.301 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.302 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T21:56:04.302 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.420806+0000", 2021-03-24T21:56:04.302 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0012988640000000001 2021-03-24T21:56:04.302 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.303 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.303 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:04.303 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.422104+0000", 2021-03-24T21:56:04.303 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0021077399999999999 2021-03-24T21:56:04.303 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.304 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.304 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:04.304 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.424212+0000", 2021-03-24T21:56:04.304 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6499000000000002e-05 2021-03-24T21:56:04.305 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.305 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.305 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T21:56:04.305 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.424269+0000", 2021-03-24T21:56:04.305 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7620999999999999e-05 2021-03-24T21:56:04.306 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.306 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.306 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:56:04.306 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.424296+0000", 2021-03-24T21:56:04.307 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022343084999999999 2021-03-24T21:56:04.307 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:04.307 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:56:04.307 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:04.308 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.308 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.308 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.34352.0:423 4.5 4:a204fd0b:::data_log.30:head [call fifo.create_meta in=15b] snapc 0=[] ondisk+write+known_if_redirected e65)", 2021-03-24T21:56:04.308 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:51:49.736485+0000", 2021-03-24T21:56:04.308 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 254.54545180700001, 2021-03-24T21:56:04.309 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0026022409999999999, 2021-03-24T21:56:04.309 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:56:04.309 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:04.309 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:56:04.309 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.34352", 2021-03-24T21:56:04.310 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/3436348424", 2021-03-24T21:56:04.310 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 423 2021-03-24T21:56:04.310 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.310 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:56:04.311 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.311 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:56:04.311 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.736485+0000", 2021-03-24T21:56:04.311 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:56:04.312 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.312 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.312 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:56:04.312 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.736485+0000", 2021-03-24T21:56:04.312 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999971 2021-03-24T21:56:04.313 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.313 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.313 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:56:04.313 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.736482+0000", 2021-03-24T21:56:04.313 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5662e-05 2021-03-24T21:56:04.314 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.314 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.314 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:56:04.314 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.736498+0000", 2021-03-24T21:56:04.315 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8516999999999999e-05 2021-03-24T21:56:04.315 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.315 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.315 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:56:04.316 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.736516+0000", 2021-03-24T21:56:04.316 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0429e-05 2021-03-24T21:56:04.316 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.316 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.316 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:56:04.317 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.736527+0000", 2021-03-24T21:56:04.317 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000120714 2021-03-24T21:56:04.317 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.317 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.317 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:56:04.318 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.736647+0000", 2021-03-24T21:56:04.318 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00019531799999999999 2021-03-24T21:56:04.318 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.318 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.319 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:56:04.319 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.736843+0000", 2021-03-24T21:56:04.319 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00033447000000000001 2021-03-24T21:56:04.319 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.319 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.320 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 4,7", 2021-03-24T21:56:04.320 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.737177+0000", 2021-03-24T21:56:04.320 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00084823400000000001 2021-03-24T21:56:04.320 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.321 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.321 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T21:56:04.321 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.738025+0000", 2021-03-24T21:56:04.321 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00063113899999999996 2021-03-24T21:56:04.321 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.322 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.322 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:04.322 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.738657+0000", 2021-03-24T21:56:04.322 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000267886 2021-03-24T21:56:04.323 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.323 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.323 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:04.323 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.738924+0000", 2021-03-24T21:56:04.323 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.7095000000000005e-05 2021-03-24T21:56:04.324 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.324 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.324 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T21:56:04.324 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.739022+0000", 2021-03-24T21:56:04.325 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.5777000000000004e-05 2021-03-24T21:56:04.325 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.325 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.325 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:56:04.325 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.739087+0000", 2021-03-24T21:56:04.326 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0026022409999999999 2021-03-24T21:56:04.326 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:04.326 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:56:04.326 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:04.327 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.327 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.327 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.34352.0:448 4.a 4:54df8a96:::data_log.42:head [call fifo.get_meta in=12b] snapc 0=[] ondisk+read+known_if_redirected e65)", 2021-03-24T21:56:04.327 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:51:49.801330+0000", 2021-03-24T21:56:04.327 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 254.480606529, 2021-03-24T21:56:04.328 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0029494360000000002, 2021-03-24T21:56:04.328 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:56:04.328 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T21:56:04.328 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:56:04.329 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.34352", 2021-03-24T21:56:04.329 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/3436348424", 2021-03-24T21:56:04.329 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 448 2021-03-24T21:56:04.329 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.330 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:56:04.330 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.330 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:56:04.330 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.801330+0000", 2021-03-24T21:56:04.330 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:56:04.331 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.331 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.331 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:56:04.331 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.801330+0000", 2021-03-24T21:56:04.332 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2021-03-24T21:56:04.332 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.332 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.332 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:56:04.332 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.801329+0000", 2021-03-24T21:56:04.333 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2080000000000003e-06 2021-03-24T21:56:04.333 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.333 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.333 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:56:04.334 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.801336+0000", 2021-03-24T21:56:04.334 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6580999999999999e-05 2021-03-24T21:56:04.334 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.334 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.335 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:56:04.335 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.801352+0000", 2021-03-24T21:56:04.335 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9820000000000002e-06 2021-03-24T21:56:04.335 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.336 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.336 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:56:04.336 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.801359+0000", 2021-03-24T21:56:04.336 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0358000000000001e-05 2021-03-24T21:56:04.336 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.337 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.337 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:56:04.337 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.801410+0000", 2021-03-24T21:56:04.337 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1737999999999997e-05 2021-03-24T21:56:04.337 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.338 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.338 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:56:04.338 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.801471+0000", 2021-03-24T21:56:04.338 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0028082889999999998 2021-03-24T21:56:04.339 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.339 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.339 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:56:04.339 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.804280+0000", 2021-03-24T21:56:04.340 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0029494360000000002 2021-03-24T21:56:04.340 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:04.340 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:56:04.340 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:04.340 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.341 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.341 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.34352.0:495 4.1f 4:fe553feb:::data_log.66:head [call fifo.create_meta in=15b] snapc 0=[] ondisk+write+known_if_redirected e65)", 2021-03-24T21:56:04.341 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:51:49.917847+0000", 2021-03-24T21:56:04.341 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 254.36409030199999, 2021-03-24T21:56:04.342 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.002238402, 2021-03-24T21:56:04.342 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:56:04.342 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:04.342 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:56:04.342 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.34352", 2021-03-24T21:56:04.343 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/3436348424", 2021-03-24T21:56:04.343 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 495 2021-03-24T21:56:04.343 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.343 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:56:04.344 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.344 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:56:04.344 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.917847+0000", 2021-03-24T21:56:04.344 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:56:04.344 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.345 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.345 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:56:04.345 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.917847+0000", 2021-03-24T21:56:04.345 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999962 2021-03-24T21:56:04.346 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.346 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.346 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:56:04.346 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.917843+0000", 2021-03-24T21:56:04.346 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7125e-05 2021-03-24T21:56:04.347 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.347 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.347 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:56:04.347 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.917860+0000", 2021-03-24T21:56:04.348 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0976999999999998e-05 2021-03-24T21:56:04.348 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.348 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.348 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:56:04.348 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.917891+0000", 2021-03-24T21:56:04.349 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4212999999999999e-05 2021-03-24T21:56:04.349 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.349 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.349 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:56:04.349 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.917905+0000", 2021-03-24T21:56:04.350 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000105123 2021-03-24T21:56:04.350 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.350 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.350 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:56:04.351 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.918010+0000", 2021-03-24T21:56:04.351 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00021104000000000001 2021-03-24T21:56:04.351 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.351 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.351 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:56:04.352 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.918221+0000", 2021-03-24T21:56:04.352 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00033568999999999998 2021-03-24T21:56:04.352 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.352 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.353 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1,5", 2021-03-24T21:56:04.353 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.918557+0000", 2021-03-24T21:56:04.353 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00082969900000000004 2021-03-24T21:56:04.353 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.353 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.354 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T21:56:04.354 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.919387+0000", 2021-03-24T21:56:04.354 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00042901700000000002 2021-03-24T21:56:04.354 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.354 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.355 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:04.355 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.919816+0000", 2021-03-24T21:56:04.355 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012908900000000001 2021-03-24T21:56:04.355 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.356 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.356 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:04.356 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.919945+0000", 2021-03-24T21:56:04.356 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0073000000000004e-05 2021-03-24T21:56:04.356 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.357 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.357 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T21:56:04.357 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.920025+0000", 2021-03-24T21:56:04.357 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0105999999999997e-05 2021-03-24T21:56:04.358 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.358 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.358 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:56:04.359 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.920085+0000", 2021-03-24T21:56:04.359 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.002238402 2021-03-24T21:56:04.359 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:04.359 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:56:04.359 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:04.360 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.360 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.360 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.34352.0:496 4.1f 4:fe553feb:::data_log.66:head [call fifo.get_meta in=12b] snapc 0=[] ondisk+read+known_if_redirected e65)", 2021-03-24T21:56:04.360 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:51:49.920311+0000", 2021-03-24T21:56:04.361 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 254.361625949, 2021-03-24T21:56:04.361 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.003022608, 2021-03-24T21:56:04.361 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:56:04.361 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T21:56:04.361 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:56:04.362 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.34352", 2021-03-24T21:56:04.362 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/3436348424", 2021-03-24T21:56:04.362 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 496 2021-03-24T21:56:04.362 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.363 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:56:04.363 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.363 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:56:04.363 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.920311+0000", 2021-03-24T21:56:04.363 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:56:04.364 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.364 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.364 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:56:04.364 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.920311+0000", 2021-03-24T21:56:04.365 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2021-03-24T21:56:04.365 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.365 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.365 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:56:04.365 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.920309+0000", 2021-03-24T21:56:04.366 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3769999999999996e-06 2021-03-24T21:56:04.366 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.366 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.366 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:56:04.366 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.920318+0000", 2021-03-24T21:56:04.367 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4007e-05 2021-03-24T21:56:04.367 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.367 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.367 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:56:04.368 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.920332+0000", 2021-03-24T21:56:04.368 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6730000000000003e-06 2021-03-24T21:56:04.368 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.368 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.368 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:56:04.369 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.920339+0000", 2021-03-24T21:56:04.369 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.4720000000000004e-05 2021-03-24T21:56:04.369 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.369 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.370 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:56:04.370 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.920403+0000", 2021-03-24T21:56:04.370 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013994699999999999 2021-03-24T21:56:04.370 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.371 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.371 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:56:04.371 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.920543+0000", 2021-03-24T21:56:04.371 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0027903419999999999 2021-03-24T21:56:04.372 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.372 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.372 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:56:04.372 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.923334+0000", 2021-03-24T21:56:04.372 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.003022608 2021-03-24T21:56:04.373 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:04.373 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:56:04.373 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:04.373 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.373 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.374 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.34352.0:506 4.1f 4:ffa95a0b:::data_log.71:head [call fifo.get_meta in=12b] snapc 0=[] ondisk+read+known_if_redirected e65)", 2021-03-24T21:56:04.374 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:51:49.948295+0000", 2021-03-24T21:56:04.374 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 254.33364182099999, 2021-03-24T21:56:04.374 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0028968850000000001, 2021-03-24T21:56:04.375 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:56:04.375 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T21:56:04.375 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:56:04.375 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.34352", 2021-03-24T21:56:04.376 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/3436348424", 2021-03-24T21:56:04.376 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 506 2021-03-24T21:56:04.376 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.376 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:56:04.376 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.377 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:56:04.377 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.948295+0000", 2021-03-24T21:56:04.377 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:56:04.377 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.377 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.378 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:56:04.378 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.948295+0000", 2021-03-24T21:56:04.378 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2021-03-24T21:56:04.378 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.379 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.379 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:56:04.379 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.948293+0000", 2021-03-24T21:56:04.379 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.7710000000000006e-06 2021-03-24T21:56:04.379 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.380 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.380 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:56:04.380 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.948301+0000", 2021-03-24T21:56:04.380 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5218999999999999e-05 2021-03-24T21:56:04.381 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.381 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.381 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:56:04.381 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.948316+0000", 2021-03-24T21:56:04.381 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6359999999999997e-06 2021-03-24T21:56:04.382 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.382 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.382 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:56:04.382 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.948323+0000", 2021-03-24T21:56:04.382 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7655000000000003e-05 2021-03-24T21:56:04.383 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.383 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.383 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:56:04.383 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.948381+0000", 2021-03-24T21:56:04.384 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7772000000000002e-05 2021-03-24T21:56:04.384 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.384 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.384 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:56:04.384 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.948438+0000", 2021-03-24T21:56:04.385 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00275348 2021-03-24T21:56:04.385 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.385 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.385 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:56:04.385 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.951192+0000", 2021-03-24T21:56:04.386 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0028968850000000001 2021-03-24T21:56:04.386 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:04.386 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:56:04.386 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:04.387 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.387 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.387 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.34352.0:513 4.a 4:55fa285f:::data_log.75:head [call fifo.create_meta in=15b] snapc 0=[] ondisk+write+known_if_redirected e65)", 2021-03-24T21:56:04.387 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:51:49.968230+0000", 2021-03-24T21:56:04.387 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 254.31370722700001, 2021-03-24T21:56:04.388 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0026831580000000002, 2021-03-24T21:56:04.388 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:56:04.388 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:04.388 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:56:04.388 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.34352", 2021-03-24T21:56:04.389 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/3436348424", 2021-03-24T21:56:04.389 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 513 2021-03-24T21:56:04.389 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.389 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:56:04.390 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.390 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:56:04.390 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.968230+0000", 2021-03-24T21:56:04.390 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:56:04.390 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.391 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.391 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:56:04.391 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.968230+0000", 2021-03-24T21:56:04.391 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999962 2021-03-24T21:56:04.392 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.392 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.392 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:56:04.392 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.968226+0000", 2021-03-24T21:56:04.393 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4802e-05 2021-03-24T21:56:04.393 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.393 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.393 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:56:04.393 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.968241+0000", 2021-03-24T21:56:04.394 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7988e-05 2021-03-24T21:56:04.394 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.394 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.394 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:56:04.395 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.968269+0000", 2021-03-24T21:56:04.395 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1015999999999999e-05 2021-03-24T21:56:04.395 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.395 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.395 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:56:04.396 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.968280+0000", 2021-03-24T21:56:04.396 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012408100000000001 2021-03-24T21:56:04.396 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.396 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.397 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:56:04.397 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.968404+0000", 2021-03-24T21:56:04.397 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00030604699999999999 2021-03-24T21:56:04.397 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.397 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.398 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:56:04.398 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.968710+0000", 2021-03-24T21:56:04.398 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00059409399999999998 2021-03-24T21:56:04.398 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.398 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.399 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1,7", 2021-03-24T21:56:04.399 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.969304+0000", 2021-03-24T21:56:04.399 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00078272499999999996 2021-03-24T21:56:04.399 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.400 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.400 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T21:56:04.400 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.970087+0000", 2021-03-24T21:56:04.400 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00055706500000000003 2021-03-24T21:56:04.400 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.401 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.401 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:04.401 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.970644+0000", 2021-03-24T21:56:04.401 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010976099999999999 2021-03-24T21:56:04.402 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.402 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.402 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:04.402 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.970754+0000", 2021-03-24T21:56:04.402 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.0830999999999998e-05 2021-03-24T21:56:04.403 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.403 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.403 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T21:56:04.403 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.970844+0000", 2021-03-24T21:56:04.404 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.8330999999999993e-05 2021-03-24T21:56:04.404 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.404 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.404 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:56:04.404 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:49.970913+0000", 2021-03-24T21:56:04.405 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0026831580000000002 2021-03-24T21:56:04.405 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:04.405 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:56:04.405 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:04.406 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.406 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.406 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.34352.0:529 4.10 e65/62 4:0958bece:::data_log.83:head v 65'4, mlcod=65'2)", 2021-03-24T21:56:04.406 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:51:50.009770+0000", 2021-03-24T21:56:04.407 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 254.272166535, 2021-03-24T21:56:04.407 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0047180019999999998, 2021-03-24T21:56:04.407 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:56:04.407 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:04.407 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:56:04.408 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.408 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:56:04.408 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:50.009770+0000", 2021-03-24T21:56:04.408 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:56:04.408 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.409 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.409 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:56:04.409 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:50.009770+0000", 2021-03-24T21:56:04.409 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999957 2021-03-24T21:56:04.409 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.410 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.410 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:56:04.410 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:50.009766+0000", 2021-03-24T21:56:04.410 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.074e-05 2021-03-24T21:56:04.411 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.411 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.411 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:56:04.411 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:50.009807+0000", 2021-03-24T21:56:04.411 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5381999999999999e-05 2021-03-24T21:56:04.412 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.412 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.412 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:56:04.412 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:50.009832+0000", 2021-03-24T21:56:04.413 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.5610000000000004e-06 2021-03-24T21:56:04.413 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.413 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.413 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:56:04.413 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:50.009842+0000", 2021-03-24T21:56:04.414 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000153564 2021-03-24T21:56:04.414 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.414 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.414 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:56:04.415 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:50.009995+0000", 2021-03-24T21:56:04.415 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4287999999999999e-05 2021-03-24T21:56:04.415 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.415 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.416 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:56:04.416 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:50.010050+0000", 2021-03-24T21:56:04.416 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0043595109999999999 2021-03-24T21:56:04.416 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.416 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.417 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T21:56:04.417 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:50.014409+0000", 2021-03-24T21:56:04.417 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9079000000000006e-05 2021-03-24T21:56:04.417 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.418 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.418 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:56:04.418 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:50.014488+0000", 2021-03-24T21:56:04.418 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0047180019999999998 2021-03-24T21:56:04.418 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:04.419 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:56:04.419 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:04.419 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.419 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.419 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.34352.0:539 4.10 e65/62 4:09aab3ca:::data_log.88:head v 65'5, mlcod=65'3)", 2021-03-24T21:56:04.420 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:51:50.060695+0000", 2021-03-24T21:56:04.420 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 254.221242176, 2021-03-24T21:56:04.420 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0078237259999999996, 2021-03-24T21:56:04.420 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:56:04.420 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:04.421 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:56:04.421 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.421 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:56:04.421 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:50.060695+0000", 2021-03-24T21:56:04.422 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:56:04.422 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.422 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.422 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:56:04.422 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:50.060695+0000", 2021-03-24T21:56:04.423 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999962 2021-03-24T21:56:04.423 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.423 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.423 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:56:04.424 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:50.060691+0000", 2021-03-24T21:56:04.424 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7368e-05 2021-03-24T21:56:04.424 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.424 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.424 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:56:04.425 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:50.060718+0000", 2021-03-24T21:56:04.425 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2384000000000001e-05 2021-03-24T21:56:04.425 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.425 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.425 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:56:04.426 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:50.060741+0000", 2021-03-24T21:56:04.426 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.009e-05 2021-03-24T21:56:04.426 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.426 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.427 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:56:04.427 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:50.060751+0000", 2021-03-24T21:56:04.427 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1659e-05 2021-03-24T21:56:04.427 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.427 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.428 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:56:04.428 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:50.060813+0000", 2021-03-24T21:56:04.428 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2607999999999997e-05 2021-03-24T21:56:04.428 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.429 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.429 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:56:04.429 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:50.060845+0000", 2021-03-24T21:56:04.429 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.007604474 2021-03-24T21:56:04.429 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.430 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.430 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T21:56:04.430 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:50.068450+0000", 2021-03-24T21:56:04.430 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.8749000000000002e-05 2021-03-24T21:56:04.431 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.431 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.431 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:56:04.431 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:50.068518+0000", 2021-03-24T21:56:04.432 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0078237259999999996 2021-03-24T21:56:04.432 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:04.432 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:56:04.432 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:04.432 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.433 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.433 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.34352.0:556 4.1f 4:fab5cb35:::data_log.96:head [call fifo.get_meta in=12b] snapc 0=[] ondisk+read+known_if_redirected e65)", 2021-03-24T21:56:04.433 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:51:50.119068+0000", 2021-03-24T21:56:04.433 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 254.162868952, 2021-03-24T21:56:04.434 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0033796680000000002, 2021-03-24T21:56:04.434 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:56:04.434 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T21:56:04.434 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:56:04.434 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.34352", 2021-03-24T21:56:04.435 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/3436348424", 2021-03-24T21:56:04.435 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 556 2021-03-24T21:56:04.435 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.435 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:56:04.436 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.436 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:56:04.436 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:50.119068+0000", 2021-03-24T21:56:04.436 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:56:04.436 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.437 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.437 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:56:04.437 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:50.119068+0000", 2021-03-24T21:56:04.437 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2021-03-24T21:56:04.438 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.438 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.438 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:56:04.438 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:50.119066+0000", 2021-03-24T21:56:04.439 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.261e-05 2021-03-24T21:56:04.439 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.439 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.439 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:56:04.439 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:50.119078+0000", 2021-03-24T21:56:04.440 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4312e-05 2021-03-24T21:56:04.440 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.440 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.440 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:56:04.440 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:50.119103+0000", 2021-03-24T21:56:04.441 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5333999999999997e-05 2021-03-24T21:56:04.441 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.441 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.441 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:56:04.442 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:50.119138+0000", 2021-03-24T21:56:04.442 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010066800000000001 2021-03-24T21:56:04.442 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.442 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.442 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:56:04.443 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:50.119239+0000", 2021-03-24T21:56:04.443 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014477000000000001 2021-03-24T21:56:04.443 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.443 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.444 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:56:04.444 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:50.119384+0000", 2021-03-24T21:56:04.444 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0030639909999999999 2021-03-24T21:56:04.444 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.444 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.445 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:56:04.445 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:50.122448+0000", 2021-03-24T21:56:04.445 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0033796680000000002 2021-03-24T21:56:04.445 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:04.446 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:56:04.446 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:04.446 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.446 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.446 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.34352.0:558 4.a 4:50492753:::data_log.97:head [call fifo.get_meta in=12b] snapc 0=[] ondisk+read+known_if_redirected e65)", 2021-03-24T21:56:04.447 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:51:50.125077+0000", 2021-03-24T21:56:04.447 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 254.15685976, 2021-03-24T21:56:04.447 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0033848250000000002, 2021-03-24T21:56:04.447 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:56:04.448 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T21:56:04.448 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:56:04.448 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.34352", 2021-03-24T21:56:04.448 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/3436348424", 2021-03-24T21:56:04.448 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 558 2021-03-24T21:56:04.449 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.449 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:56:04.449 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.449 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:56:04.450 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:50.125077+0000", 2021-03-24T21:56:04.450 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:56:04.450 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.450 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.450 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:56:04.451 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:50.125077+0000", 2021-03-24T21:56:04.451 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2021-03-24T21:56:04.451 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.451 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.452 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:56:04.452 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:50.125075+0000", 2021-03-24T21:56:04.452 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4290000000000001e-06 2021-03-24T21:56:04.452 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.452 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.453 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:56:04.453 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:50.125084+0000", 2021-03-24T21:56:04.453 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.466e-05 2021-03-24T21:56:04.453 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.453 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.454 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:56:04.454 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:50.125098+0000", 2021-03-24T21:56:04.454 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0753e-05 2021-03-24T21:56:04.454 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.455 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.455 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:56:04.455 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:50.125109+0000", 2021-03-24T21:56:04.455 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6709e-05 2021-03-24T21:56:04.455 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.456 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.456 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:56:04.456 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:50.125186+0000", 2021-03-24T21:56:04.456 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015429800000000001 2021-03-24T21:56:04.457 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.457 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.457 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:56:04.457 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:50.125340+0000", 2021-03-24T21:56:04.458 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0031216899999999999 2021-03-24T21:56:04.458 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.458 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.458 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:56:04.458 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:50.128462+0000", 2021-03-24T21:56:04.459 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0033848250000000002 2021-03-24T21:56:04.459 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:04.459 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:56:04.459 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:04.460 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.460 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.460 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.24376.0:23 4.a 4:5127e642:gc::gc.21:head [create,call version.check_conds in=18b,call rgw_gc.rgw_gc_queue_init in=23b,call version.set in=10b] snapc 0=[] ondisk+write+known_if_redirected e69)", 2021-03-24T21:56:04.460 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:51:53.825497+0000", 2021-03-24T21:56:04.460 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 250.456440114, 2021-03-24T21:56:04.461 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.002555266, 2021-03-24T21:56:04.461 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:56:04.461 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:04.461 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:56:04.461 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.24376", 2021-03-24T21:56:04.462 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/3051439499", 2021-03-24T21:56:04.462 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 23 2021-03-24T21:56:04.462 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.462 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:56:04.462 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.463 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:56:04.463 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.825497+0000", 2021-03-24T21:56:04.463 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:56:04.463 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.464 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.464 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:56:04.464 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.825497+0000", 2021-03-24T21:56:04.464 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999971 2021-03-24T21:56:04.464 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.465 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.465 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:56:04.465 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.825494+0000", 2021-03-24T21:56:04.465 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2527000000000001e-05 2021-03-24T21:56:04.465 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.466 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.466 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:56:04.466 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.825506+0000", 2021-03-24T21:56:04.466 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6025e-05 2021-03-24T21:56:04.467 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.467 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.467 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:56:04.467 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.825522+0000", 2021-03-24T21:56:04.467 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2619000000000001e-05 2021-03-24T21:56:04.468 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.468 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.468 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:56:04.468 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.825535+0000", 2021-03-24T21:56:04.469 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0729999999999994e-05 2021-03-24T21:56:04.469 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.469 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.469 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:56:04.469 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.825616+0000", 2021-03-24T21:56:04.521 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016505600000000001 2021-03-24T21:56:04.521 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.521 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.522 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:56:04.522 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.825781+0000", 2021-03-24T21:56:04.522 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00034122599999999998 2021-03-24T21:56:04.522 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.523 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.523 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1,7", 2021-03-24T21:56:04.523 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.826122+0000", 2021-03-24T21:56:04.523 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00072659700000000005 2021-03-24T21:56:04.524 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.524 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.524 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T21:56:04.524 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.826849+0000", 2021-03-24T21:56:04.524 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00046154399999999998 2021-03-24T21:56:04.525 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.525 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.525 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:04.525 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.827310+0000", 2021-03-24T21:56:04.526 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00063453900000000004 2021-03-24T21:56:04.526 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.526 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.526 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:04.526 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.827945+0000", 2021-03-24T21:56:04.527 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7218e-05 2021-03-24T21:56:04.527 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.527 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.527 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T21:56:04.528 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.828002+0000", 2021-03-24T21:56:04.528 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0074999999999997e-05 2021-03-24T21:56:04.528 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.528 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.528 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:56:04.529 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.828052+0000", 2021-03-24T21:56:04.529 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.002555266 2021-03-24T21:56:04.529 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:04.529 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:56:04.529 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:04.530 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.530 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.530 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.24376.0:123 4.a 4:55b0279d:reshard::reshard.0000000009:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e69)", 2021-03-24T21:56:04.530 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:51:53.895346+0000", 2021-03-24T21:56:04.531 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 250.38659058100001, 2021-03-24T21:56:04.531 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.002485391, 2021-03-24T21:56:04.531 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:56:04.531 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:04.531 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:56:04.532 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.24376", 2021-03-24T21:56:04.532 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/3051439499", 2021-03-24T21:56:04.532 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 123 2021-03-24T21:56:04.532 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.533 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:56:04.533 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.533 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:56:04.533 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.895346+0000", 2021-03-24T21:56:04.533 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:56:04.534 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.534 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.534 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:56:04.534 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.895346+0000", 2021-03-24T21:56:04.535 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2021-03-24T21:56:04.535 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.535 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.535 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:56:04.536 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.895345+0000", 2021-03-24T21:56:04.536 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.7560000000000002e-06 2021-03-24T21:56:04.536 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.536 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.536 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:56:04.537 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.895352+0000", 2021-03-24T21:56:04.537 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0829000000000001e-05 2021-03-24T21:56:04.537 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.537 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.538 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:56:04.538 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.895363+0000", 2021-03-24T21:56:04.538 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.1550000000000007e-06 2021-03-24T21:56:04.538 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.539 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.539 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:56:04.539 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.895371+0000", 2021-03-24T21:56:04.539 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081051400000000003 2021-03-24T21:56:04.539 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.540 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.540 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:56:04.540 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.896181+0000", 2021-03-24T21:56:04.540 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016871200000000001 2021-03-24T21:56:04.541 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.541 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.541 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:56:04.541 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.896350+0000", 2021-03-24T21:56:04.541 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00026808600000000001 2021-03-24T21:56:04.542 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.542 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.542 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1,7", 2021-03-24T21:56:04.542 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.896618+0000", 2021-03-24T21:56:04.542 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00068435200000000005 2021-03-24T21:56:04.543 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.543 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.543 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:04.543 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.897302+0000", 2021-03-24T21:56:04.544 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4411e-05 2021-03-24T21:56:04.544 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.544 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.544 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T21:56:04.544 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.897317+0000", 2021-03-24T21:56:04.545 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00041911699999999999 2021-03-24T21:56:04.545 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.545 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.545 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:04.546 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.897736+0000", 2021-03-24T21:56:04.546 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.5018999999999997e-05 2021-03-24T21:56:04.546 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.546 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.547 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T21:56:04.547 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.897781+0000", 2021-03-24T21:56:04.547 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0624000000000002e-05 2021-03-24T21:56:04.547 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.547 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.548 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:56:04.548 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.897832+0000", 2021-03-24T21:56:04.548 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.002485391 2021-03-24T21:56:04.548 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:04.549 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:56:04.549 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:04.549 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.549 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.549 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.24376.0:155 4.5 4:a7f90313:reshard::reshard.0000000012:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e69)", 2021-03-24T21:56:04.550 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:51:53.909138+0000", 2021-03-24T21:56:04.550 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 250.372799168, 2021-03-24T21:56:04.550 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0024562859999999998, 2021-03-24T21:56:04.550 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:56:04.551 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:04.551 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:56:04.551 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.24376", 2021-03-24T21:56:04.551 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/3051439499", 2021-03-24T21:56:04.552 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 155 2021-03-24T21:56:04.552 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.552 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:56:04.552 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.552 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:56:04.553 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.909138+0000", 2021-03-24T21:56:04.553 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:56:04.553 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.553 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.553 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:56:04.554 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.909138+0000", 2021-03-24T21:56:04.554 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2021-03-24T21:56:04.554 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.554 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.555 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:56:04.555 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.909135+0000", 2021-03-24T21:56:04.555 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0755e-05 2021-03-24T21:56:04.555 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.555 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.556 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:56:04.556 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.909146+0000", 2021-03-24T21:56:04.556 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8800999999999998e-05 2021-03-24T21:56:04.556 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.557 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.557 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:56:04.557 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.909165+0000", 2021-03-24T21:56:04.557 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0883000000000001e-05 2021-03-24T21:56:04.557 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.558 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.558 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:56:04.558 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.909176+0000", 2021-03-24T21:56:04.558 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00029039000000000002 2021-03-24T21:56:04.559 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.559 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.559 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:56:04.559 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.909466+0000", 2021-03-24T21:56:04.559 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013261499999999999 2021-03-24T21:56:04.560 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.560 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.560 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:56:04.560 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.909599+0000", 2021-03-24T21:56:04.560 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000299296 2021-03-24T21:56:04.561 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.561 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.561 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 4,7", 2021-03-24T21:56:04.561 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.909898+0000", 2021-03-24T21:56:04.562 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001211059 2021-03-24T21:56:04.562 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.562 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.562 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:04.562 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.911109+0000", 2021-03-24T21:56:04.563 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000242706 2021-03-24T21:56:04.563 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.563 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.563 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T21:56:04.564 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.911352+0000", 2021-03-24T21:56:04.564 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.0080999999999994e-05 2021-03-24T21:56:04.564 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.564 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.564 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:04.565 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.911442+0000", 2021-03-24T21:56:04.565 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9553000000000001e-05 2021-03-24T21:56:04.565 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.565 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.566 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T21:56:04.566 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.911531+0000", 2021-03-24T21:56:04.566 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2521000000000007e-05 2021-03-24T21:56:04.566 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.566 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.567 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:56:04.567 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.911594+0000", 2021-03-24T21:56:04.567 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0024562859999999998 2021-03-24T21:56:04.567 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:04.568 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:56:04.568 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:04.568 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.568 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.568 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.24376.0:287 4.1f 4:fd8d39b6:::obj_delete_at_hint.0000000030:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e69)", 2021-03-24T21:56:04.569 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:51:53.991085+0000", 2021-03-24T21:56:04.569 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 250.290851976, 2021-03-24T21:56:04.569 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0024303039999999999, 2021-03-24T21:56:04.569 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:56:04.569 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:04.570 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:56:04.570 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.24376", 2021-03-24T21:56:04.570 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/3051439499", 2021-03-24T21:56:04.570 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 287 2021-03-24T21:56:04.571 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.571 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:56:04.571 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.571 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:56:04.571 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.991085+0000", 2021-03-24T21:56:04.572 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:56:04.572 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.572 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.572 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:56:04.572 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.991085+0000", 2021-03-24T21:56:04.573 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999971 2021-03-24T21:56:04.573 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.573 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.573 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:56:04.574 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.991082+0000", 2021-03-24T21:56:04.574 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3884e-05 2021-03-24T21:56:04.574 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.574 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.575 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:56:04.575 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.991096+0000", 2021-03-24T21:56:04.575 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1872999999999999e-05 2021-03-24T21:56:04.575 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.575 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.576 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:56:04.576 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.991118+0000", 2021-03-24T21:56:04.576 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4311000000000001e-05 2021-03-24T21:56:04.576 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.576 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.577 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:56:04.577 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.991132+0000", 2021-03-24T21:56:04.577 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010032899999999999 2021-03-24T21:56:04.577 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.578 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.578 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:56:04.578 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.991232+0000", 2021-03-24T21:56:04.578 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016329699999999999 2021-03-24T21:56:04.578 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.579 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.579 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:56:04.579 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.991395+0000", 2021-03-24T21:56:04.579 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000273016 2021-03-24T21:56:04.580 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.580 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.580 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1,5", 2021-03-24T21:56:04.580 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.991669+0000", 2021-03-24T21:56:04.580 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00059263100000000004 2021-03-24T21:56:04.581 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.581 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.581 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T21:56:04.581 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.992261+0000", 2021-03-24T21:56:04.582 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00042415900000000002 2021-03-24T21:56:04.582 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.582 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.582 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:04.582 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.992685+0000", 2021-03-24T21:56:04.583 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00072589499999999995 2021-03-24T21:56:04.583 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.583 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.583 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:04.583 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.993411+0000", 2021-03-24T21:56:04.584 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5755999999999999e-05 2021-03-24T21:56:04.584 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.584 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.584 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T21:56:04.585 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.993467+0000", 2021-03-24T21:56:04.585 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8232999999999997e-05 2021-03-24T21:56:04.585 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.585 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.585 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:56:04.586 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:53.993515+0000", 2021-03-24T21:56:04.586 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0024303039999999999 2021-03-24T21:56:04.586 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:04.586 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:56:04.586 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:04.587 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.587 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.587 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.24376.0:383 4.5 4:a17eb30f:::obj_delete_at_hint.0000000054:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e69)", 2021-03-24T21:56:04.587 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:51:54.103601+0000", 2021-03-24T21:56:04.588 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 250.17833607899999, 2021-03-24T21:56:04.588 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00260355, 2021-03-24T21:56:04.588 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:56:04.588 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:04.588 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:56:04.589 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.24376", 2021-03-24T21:56:04.589 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/3051439499", 2021-03-24T21:56:04.589 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 383 2021-03-24T21:56:04.589 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.589 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:56:04.590 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.590 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:56:04.590 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.103601+0000", 2021-03-24T21:56:04.590 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:56:04.591 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.591 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.591 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:56:04.591 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.103601+0000", 2021-03-24T21:56:04.591 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999943 2021-03-24T21:56:04.592 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.592 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.592 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:56:04.592 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.103595+0000", 2021-03-24T21:56:04.593 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4468e-05 2021-03-24T21:56:04.593 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.593 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.593 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:56:04.593 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.103620+0000", 2021-03-24T21:56:04.594 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2747000000000003e-05 2021-03-24T21:56:04.594 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.594 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.594 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:56:04.595 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.103662+0000", 2021-03-24T21:56:04.595 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8440999999999998e-05 2021-03-24T21:56:04.595 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.595 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.595 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:56:04.596 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.103681+0000", 2021-03-24T21:56:04.596 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014296799999999999 2021-03-24T21:56:04.596 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.596 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.597 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:56:04.597 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.103824+0000", 2021-03-24T21:56:04.597 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00034723499999999999 2021-03-24T21:56:04.597 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.598 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.598 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:56:04.598 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.104171+0000", 2021-03-24T21:56:04.598 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00045475499999999999 2021-03-24T21:56:04.599 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.599 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.599 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 4,7", 2021-03-24T21:56:04.599 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.104626+0000", 2021-03-24T21:56:04.599 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00087004699999999999 2021-03-24T21:56:04.600 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.600 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.600 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:04.600 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.105496+0000", 2021-03-24T21:56:04.601 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4020000000000001e-05 2021-03-24T21:56:04.601 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.601 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.601 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T21:56:04.601 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.105550+0000", 2021-03-24T21:56:04.602 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000349346 2021-03-24T21:56:04.602 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.602 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.602 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:04.603 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.105899+0000", 2021-03-24T21:56:04.603 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014747100000000001 2021-03-24T21:56:04.603 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.603 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.604 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T21:56:04.604 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.106047+0000", 2021-03-24T21:56:04.604 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000157622 2021-03-24T21:56:04.604 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.604 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.605 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:56:04.605 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.106204+0000", 2021-03-24T21:56:04.605 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00260355 2021-03-24T21:56:04.605 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:04.606 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:56:04.606 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:04.606 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.606 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.607 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.24376.0:443 4.5 4:a77c5545:::obj_delete_at_hint.0000000084:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e69)", 2021-03-24T21:56:04.607 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:51:54.253418+0000", 2021-03-24T21:56:04.607 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 250.028518832, 2021-03-24T21:56:04.607 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0022679169999999999, 2021-03-24T21:56:04.607 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:56:04.608 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:04.608 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:56:04.608 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.24376", 2021-03-24T21:56:04.608 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/3051439499", 2021-03-24T21:56:04.608 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 443 2021-03-24T21:56:04.609 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.609 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:56:04.609 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.609 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:56:04.610 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.253418+0000", 2021-03-24T21:56:04.610 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:56:04.610 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.610 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.611 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:56:04.611 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.253418+0000", 2021-03-24T21:56:04.611 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999971 2021-03-24T21:56:04.611 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.611 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.612 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:56:04.612 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.253415+0000", 2021-03-24T21:56:04.612 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4518999999999999e-05 2021-03-24T21:56:04.612 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.612 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.613 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:56:04.613 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.253430+0000", 2021-03-24T21:56:04.613 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6205000000000001e-05 2021-03-24T21:56:04.613 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.614 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.614 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:56:04.614 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.253456+0000", 2021-03-24T21:56:04.614 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2312e-05 2021-03-24T21:56:04.614 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.615 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.615 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:56:04.615 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.253468+0000", 2021-03-24T21:56:04.615 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018132900000000001 2021-03-24T21:56:04.616 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.616 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.616 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:56:04.616 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.253649+0000", 2021-03-24T21:56:04.616 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000221905 2021-03-24T21:56:04.617 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.617 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.617 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:56:04.617 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.253871+0000", 2021-03-24T21:56:04.617 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000352183 2021-03-24T21:56:04.618 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.618 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.618 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 4,7", 2021-03-24T21:56:04.618 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.254223+0000", 2021-03-24T21:56:04.619 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00073342800000000005 2021-03-24T21:56:04.619 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.619 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.619 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T21:56:04.619 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.254957+0000", 2021-03-24T21:56:04.620 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00052811500000000001 2021-03-24T21:56:04.620 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.620 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.620 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:04.620 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.255485+0000", 2021-03-24T21:56:04.621 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3774000000000003e-05 2021-03-24T21:56:04.621 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.621 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.621 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:04.621 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.255549+0000", 2021-03-24T21:56:04.622 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2909000000000003e-05 2021-03-24T21:56:04.622 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.622 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.622 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T21:56:04.622 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.255622+0000", 2021-03-24T21:56:04.623 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.4232000000000004e-05 2021-03-24T21:56:04.623 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.623 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.623 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:56:04.624 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.255686+0000", 2021-03-24T21:56:04.624 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0022679169999999999 2021-03-24T21:56:04.624 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:04.624 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:56:04.624 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:04.625 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.625 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.625 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.24376.0:485 4.a 4:575f3851:::obj_delete_at_hint.0000000105:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e69)", 2021-03-24T21:56:04.625 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:51:54.365303+0000", 2021-03-24T21:56:04.626 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 249.916633792, 2021-03-24T21:56:04.626 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0023663780000000001, 2021-03-24T21:56:04.626 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:56:04.626 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:04.626 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:56:04.627 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.24376", 2021-03-24T21:56:04.627 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/3051439499", 2021-03-24T21:56:04.627 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 485 2021-03-24T21:56:04.627 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.628 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:56:04.628 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.628 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:56:04.628 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.365303+0000", 2021-03-24T21:56:04.628 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:56:04.629 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.629 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.629 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:56:04.629 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.365303+0000", 2021-03-24T21:56:04.630 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999967 2021-03-24T21:56:04.630 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.630 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.630 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:56:04.630 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.365300+0000", 2021-03-24T21:56:04.631 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3397000000000001e-05 2021-03-24T21:56:04.631 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.631 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.631 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:56:04.631 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.365313+0000", 2021-03-24T21:56:04.632 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4878000000000001e-05 2021-03-24T21:56:04.632 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.632 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.632 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:56:04.633 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.365338+0000", 2021-03-24T21:56:04.633 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0018e-05 2021-03-24T21:56:04.633 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.633 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.634 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:56:04.634 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.365348+0000", 2021-03-24T21:56:04.634 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016749399999999999 2021-03-24T21:56:04.634 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.634 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.635 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:56:04.635 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.365515+0000", 2021-03-24T21:56:04.635 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000415599 2021-03-24T21:56:04.635 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.636 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.636 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:56:04.636 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.365931+0000", 2021-03-24T21:56:04.636 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00043708799999999997 2021-03-24T21:56:04.636 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.637 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.637 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1,7", 2021-03-24T21:56:04.637 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.366368+0000", 2021-03-24T21:56:04.637 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00054007299999999996 2021-03-24T21:56:04.638 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.638 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.638 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T21:56:04.638 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.366908+0000", 2021-03-24T21:56:04.638 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00034821 2021-03-24T21:56:04.639 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.639 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.639 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:04.639 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.367256+0000", 2021-03-24T21:56:04.640 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000264419 2021-03-24T21:56:04.640 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.640 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.640 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:04.640 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.367521+0000", 2021-03-24T21:56:04.641 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4135999999999995e-05 2021-03-24T21:56:04.641 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.641 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.641 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T21:56:04.641 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.367605+0000", 2021-03-24T21:56:04.642 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.4611999999999997e-05 2021-03-24T21:56:04.642 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.642 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.642 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:56:04.643 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.367669+0000", 2021-03-24T21:56:04.643 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0023663780000000001 2021-03-24T21:56:04.643 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:04.643 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:56:04.644 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:04.644 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.644 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.644 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.24376.0:503 4.5 4:a1987d85:::obj_delete_at_hint.0000000114:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e69)", 2021-03-24T21:56:04.644 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:51:54.411427+0000", 2021-03-24T21:56:04.645 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 249.87050987500001, 2021-03-24T21:56:04.645 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.002290252, 2021-03-24T21:56:04.645 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:56:04.645 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:04.645 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:56:04.646 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.24376", 2021-03-24T21:56:04.646 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/3051439499", 2021-03-24T21:56:04.646 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 503 2021-03-24T21:56:04.646 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.647 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:56:04.647 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.647 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:56:04.647 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.411427+0000", 2021-03-24T21:56:04.647 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:56:04.648 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.648 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.648 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:56:04.648 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.411427+0000", 2021-03-24T21:56:04.649 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999967 2021-03-24T21:56:04.649 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.649 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.649 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:56:04.649 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.411424+0000", 2021-03-24T21:56:04.650 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6381000000000001e-05 2021-03-24T21:56:04.650 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.650 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.650 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:56:04.651 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.411440+0000", 2021-03-24T21:56:04.651 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3676000000000001e-05 2021-03-24T21:56:04.651 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.651 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.651 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:56:04.652 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.411474+0000", 2021-03-24T21:56:04.652 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1600000000000001e-05 2021-03-24T21:56:04.652 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.652 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.653 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:56:04.653 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.411485+0000", 2021-03-24T21:56:04.653 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6891000000000005e-05 2021-03-24T21:56:04.653 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.653 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.654 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:56:04.654 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.411582+0000", 2021-03-24T21:56:04.654 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00026581899999999998 2021-03-24T21:56:04.654 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.655 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.655 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:56:04.655 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.411848+0000", 2021-03-24T21:56:04.655 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000410596 2021-03-24T21:56:04.655 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.656 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.656 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 4,7", 2021-03-24T21:56:04.656 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.412259+0000", 2021-03-24T21:56:04.656 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00076435200000000004 2021-03-24T21:56:04.656 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.657 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.657 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T21:56:04.657 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.413023+0000", 2021-03-24T21:56:04.657 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00051527300000000001 2021-03-24T21:56:04.658 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.658 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.658 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:04.658 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.413538+0000", 2021-03-24T21:56:04.658 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.4564000000000001e-05 2021-03-24T21:56:04.659 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.659 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.659 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:04.659 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.413603+0000", 2021-03-24T21:56:04.660 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1104000000000003e-05 2021-03-24T21:56:04.660 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.660 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.660 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T21:56:04.660 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.413664+0000", 2021-03-24T21:56:04.661 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.3431999999999999e-05 2021-03-24T21:56:04.661 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.661 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.661 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:56:04.661 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:54.413717+0000", 2021-03-24T21:56:04.662 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.002290252 2021-03-24T21:56:04.662 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:04.662 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:56:04.662 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:04.663 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.663 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.663 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDScrubReserve(1.0 REQUEST e90)", 2021-03-24T21:56:04.663 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:53:30.900885+0000", 2021-03-24T21:56:04.663 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 153.38105155700001, 2021-03-24T21:56:04.664 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0024405429999999999, 2021-03-24T21:56:04.664 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:56:04.664 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T21:56:04.664 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:56:04.665 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.665 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:56:04.665 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:53:30.900885+0000", 2021-03-24T21:56:04.665 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:56:04.665 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.666 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.666 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:56:04.666 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:53:30.900885+0000", 2021-03-24T21:56:04.666 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2021-03-24T21:56:04.667 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.667 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.667 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:56:04.667 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:53:30.900885+0000", 2021-03-24T21:56:04.667 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6560000000000004e-06 2021-03-24T21:56:04.668 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.668 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.668 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:56:04.668 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:53:30.900894+0000", 2021-03-24T21:56:04.668 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0448999999999999e-05 2021-03-24T21:56:04.669 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.669 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.669 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:56:04.669 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:53:30.900925+0000", 2021-03-24T21:56:04.669 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5826999999999999e-05 2021-03-24T21:56:04.670 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.670 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.670 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:56:04.670 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:53:30.900941+0000", 2021-03-24T21:56:04.671 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012462000000000001 2021-03-24T21:56:04.671 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.671 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.671 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:56:04.671 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:53:30.901065+0000", 2021-03-24T21:56:04.672 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2660000000000001e-05 2021-03-24T21:56:04.672 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.672 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.672 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op must wait for map", 2021-03-24T21:56:04.672 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:53:30.901078+0000", 2021-03-24T21:56:04.673 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0021070569999999999 2021-03-24T21:56:04.673 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.673 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.673 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:56:04.674 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:53:30.903185+0000", 2021-03-24T21:56:04.674 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2566e-05 2021-03-24T21:56:04.674 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.674 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.674 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:56:04.675 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:53:30.903217+0000", 2021-03-24T21:56:04.675 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000108348 2021-03-24T21:56:04.675 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:04.675 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:04.676 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:56:04.676 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:53:30.903326+0000", 2021-03-24T21:56:04.676 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0024405429999999999 2021-03-24T21:56:04.676 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:04.676 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:56:04.677 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:04.677 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:04.677 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:56:04.677 INFO:teuthology.orchestra.run.smithi142.stdout:} 2021-03-24T21:56:04.678 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:04.679 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[19936]: cluster 2021-03-24T21:56:04.679 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[19936]: 2021-03-24T21: 2021-03-24T21:56:04.679 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[19936]: 56:02.739836+0000 mgr.y (mgr.14144) 357 : cluster [DBG] pgmap v452: 148 pgs: 148 active+clean; 114 MiB data, 382 MiB used, 536 GiB / 536 GiB avail; 5.3 MiB/s rd, 2.9 MiB/s wr, 11 op/s 2021-03-24T21:56:04.680 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:56:04.680 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[19936]: 56:03.060053+0000 osd.7 (osd.7) 45 : cluster [DBG] 2021-03-24T21:56:04.680 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[19936]: 3.f scrub ok 2021-03-24T21:56:04.680 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[19936]: cluster 2021-03-24T21:56:04.681 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[19936]: 2021-03-24T21: 2021-03-24T21:56:04.681 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[19936]: 56:03.096777+0000 osd.2 (osd.2) 2021-03-24T21:56:04.681 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[19936]: 37 : cluster [DBG] 2.1 scrub ok 2021-03-24T21:56:04.681 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[19936]: cluster 2021-03-24T21:56:04.681 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[19936]: 2021-03-24 2021-03-24T21:56:04.682 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[19936]: T21:56:03.227438+0000 mon.a (mon. 2021-03-24T21:56:04.682 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[19936]: 0) 783 : cluster [DBG] osdmap e194: 8 total, 8 up, 6 in 2021-03-24T21:56:04.682 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[19936]: audit 2021-03-24T21:56:04.682 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[19936]: 2021-03-24T21:56:03.776615+0000 mon.a 2021-03-24T21:56:04.683 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[19936]: (mon.0) 784 : audit [INF] from='client.? v1:172.21.15.125:0/3275940262' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:56:04.683 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:04 smithi142 conmon[20125]: cluster 2021-03-24T21:56:02 2021-03-24T21:56:04.683 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:04 smithi142 conmon[20125]: .739836+0000 mgr.y (mgr.14144) 357 : cluster [DBG] pgmap v452: 148 pgs: 148 active+clean; 114 MiB data, 382 MiB used, 536 GiB / 536 GiB avail; 5.3 MiB/s rd, 2.9 MiB/s wr, 11 op/s 2021-03-24T21:56:04.684 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:04 smithi142 conmon[20125]: cluster 2021-03-24T21:56:03. 2021-03-24T21:56:04.684 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:04 smithi142 conmon[20125]: 060053+0000 osd.7 (osd.7) 45 : cluster [DBG] 3.f scrub ok 2021-03-24T21:56:04.684 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:04 smithi142 conmon[20125]: cluster 2021-03-24T21:56 2021-03-24T21:56:04.684 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:04 smithi142 conmon[20125]: :03.096777+0000 osd.2 (osd.2) 37 : 2021-03-24T21:56:04.684 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:04 smithi142 conmon[20125]: cluster [DBG] 2.1 scrub ok 2021-03-24T21:56:04.685 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:04 smithi142 conmon[20125]: cluster 2021-03-24T21:56 2021-03-24T21:56:04.685 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:04 smithi142 conmon[20125]: :03.227438+0000 mon.a (mon.0) 2021-03-24T21:56:04.685 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:04 smithi142 conmon[20125]: 783 : cluster [DBG] osdmap e194: 8 total, 8 up, 6 in 2021-03-24T21:56:04.685 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:04 smithi142 conmon[20125]: audit 2021-03-24T21: 2021-03-24T21:56:04.685 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:04 smithi142 conmon[20125]: 56:03.776615+0000 mon.a (mon.0) 784 2021-03-24T21:56:04.686 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:04 smithi142 conmon[20125]: : audit [INF] from='client.? v1:172.21.15.125:0/3275940262' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:56:04.686 INFO:journalctl@ceph.osd.0.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[28026]: debug 2021-03-24T21:56:04.250+0000 7f61b647b700 -1 osd.0 pg_epoch: 195 pg[7.7( v 194'184 (0'0,194'184] local-lis/les=108/109 n=6 ec=99/99 lis/c=108/108 les/c/f=109/109/0 sis=108) [0,3,7] r=0 lpr=108 crt=194'183 lcod 194'183 mlcod 194'183 active+clean+snaptrim trimq=[56~1] ps=[4e~2,51~1,53~1,57~1]] removing snap head 2021-03-24T21:56:04.686 INFO:journalctl@ceph.osd.0.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[28026]: debug 2021-03-24T21:56:04.687 INFO:journalctl@ceph.osd.0.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[28026]: 2021-03-24T21:56:04.250+0000 7f61b647b700 -1 osd.0 pg_epoch: 195 pg[7.7( v 195'186 (0'0,195'186] local-lis/les=108/109 n=4 ec=99/99 lis/c=108/108 les/c/f=109/109/0 sis=108) [0,3,7] r=0 lpr=108 luod=194'184 crt=195'186 lcod 194'183 mlcod 194'183 active+clean+snaptrim trimq=[56~1] ps=[4e~2,51~1,53~1,57~1]] removing snap head 2021-03-24T21:56:04.687 INFO:journalctl@ceph.osd.0.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[28026]: 2021-03-24T21:56:04.687 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[24927]: cluster 2021-03-24T21:56:04.688 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[24927]: 2021-03-24T21:56:04.688 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[24927]: 2021 2021-03-24T21:56:04.688 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[24927]: - 2021-03-24T21:56:04.688 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[24927]: 03 2021-03-24T21:56:04.688 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[24927]: - 2021-03-24T21:56:04.689 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[24927]: 24 2021-03-24T21:56:04.689 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[24927]: T 2021-03-24T21:56:04.689 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[24927]: 21 2021-03-24T21:56:04.689 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[24927]: : 2021-03-24T21:56:04.690 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[24927]: 56 2021-03-24T21:56:04.690 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[24927]: : 2021-03-24T21:56:04.690 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[24927]: 02 2021-03-24T21:56:04.690 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[24927]: . 2021-03-24T21:56:04.690 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[24927]: 739836 2021-03-24T21:56:04.691 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[24927]: +0000 2021-03-24T21:56:04.691 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[24927]: 2021-03-24T21:56:04.691 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[24927]: mgr.y 2021-03-24T21:56:04.691 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[24927]: ( 2021-03-24T21:56:04.692 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[24927]: mgr 2021-03-24T21:56:04.692 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[24927]: . 2021-03-24T21:56:04.692 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[24927]: 14144 2021-03-24T21:56:04.692 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[24927]: ) 2021-03-24T21:56:04.692 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[24927]: 357 2021-03-24T21:56:04.693 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[24927]: : 2021-03-24T21:56:04.693 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[24927]: cluster [DBG] pgmap v452: 148 pgs: 148 active+clean; 114 MiB data, 382 MiB used, 536 GiB / 536 GiB avail; 5.3 MiB/s rd, 2.9 MiB/s wr, 11 op/s 2021-03-24T21:56:04.693 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[24927]: cluster 2021-03-24T21:56:03.060053+0000 osd.7 (osd.7) 2021-03-24T21:56:04.693 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[24927]: 45 : cluster [DBG] 3.f scrub ok 2021-03-24T21:56:04.694 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[24927]: cluster 2021-03-24T21:56:03.096777 2021-03-24T21:56:04.694 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[24927]: +0000 osd.2 (osd.2) 37 : cluster [DBG] 2021-03-24T21:56:04.694 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[24927]: 2.1 scrub ok 2021-03-24T21:56:04.694 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[24927]: cluster 2021-03-24T21:56:04.694 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[24927]: 2021-03-24T21:56:03.227438+0000 mon.a 2021-03-24T21:56:04.695 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[24927]: (mon.0) 783 : cluster [DBG] osdmap e194: 8 total, 8 up, 6 in 2021-03-24T21:56:04.695 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[24927]: audit 2021- 2021-03-24T21:56:04.695 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[24927]: 03-24T21:56:03.776615+0000 mon.a 2021-03-24T21:56:04.695 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:04 smithi125 conmon[24927]: (mon.0) 784 : audit [INF] from='client.? v1:172.21.15.125:0/3275940262' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:56:04.745 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:04.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:04.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:56:04.851 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:04.910 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:04.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:04.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:56:05.017 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:05.106 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:05.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:05.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:56:05.117 INFO:tasks.rados.rados.0.smithi125.stdout:497: done (0 left) 2021-03-24T21:56:05.117 INFO:tasks.rados.rados.0.smithi125.stdout:498: snap_create 2021-03-24T21:56:05.121 INFO:teuthology.orchestra.run.smithi125.stderr:set pool 7 pg_num to 26 2021-03-24T21:56:05.213 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:05.235 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.7 dump_ops_in_flight 2021-03-24T21:56:05.268 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:05.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:05.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:56:05.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: cluster 2021-03-24T21:56:04 2021-03-24T21:56:05.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: .042700+0000 osd.7 (osd.7) 46 : cluster [DBG] 6.12 scrub ok 2021-03-24T21:56:05.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:56:05.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: -24T21:56:04.051460+0000 osd.4 ( 2021-03-24T21:56:05.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: osd.4) 55 : cluster [DBG] 2021-03-24T21:56:05.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: 4.1b scrub ok 2021-03-24T21:56:05.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:56:05.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: -03-24T21:56: 2021-03-24T21:56:05.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: 04.173936+0000 osd.5 (osd 2021-03-24T21:56:05.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: .5) 56 : cluster [DBG] 2021-03-24T21:56:05.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: 6.7 scrub ok 2021-03-24T21:56:05.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:56:05.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: -03-24T21:56: 2021-03-24T21:56:05.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: 04.223969+0000 mon.a ( 2021-03-24T21:56:05.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: mon.0) 785 : cluster 2021-03-24T21:56:05.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T21:56:05.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: cluster 2021-03-24T21:56:05.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: 2021-03-24T21: 2021-03-24T21:56:05.373 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: 56:04.224039+0000 mon.a 2021-03-24T21:56:05.373 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: (mon.0) 786 : cluster 2021-03-24T21:56:05.373 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: [INF] Cluster is now healthy 2021-03-24T21:56:05.373 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: audit 2021-03-24T21:56:05.373 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: 2021-03-24T21: 2021-03-24T21:56:05.374 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: 56:04.233805+0000 mon.a 2021-03-24T21:56:05.374 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: (mon.0) 787 : audit 2021-03-24T21:56:05.374 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: [INF] from='client.? v1:172.21.15.125:0/3275940262' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T21:56:05.374 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: cluster 2021-03-24T21:56:05.375 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: 2021-03-24T21:56:04. 2021-03-24T21:56:05.375 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: 233950+0000 mon.a (mon.0) 788 2021-03-24T21:56:05.375 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: : cluster [DBG] osdmap e195: 8 total, 8 up, 6 in 2021-03-24T21:56:05.375 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: audit 2021-03-24T21:56:05.375 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: 2021-03-24T21:56 2021-03-24T21:56:05.376 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: :04.567069+0000 mon.a 2021-03-24T21:56:05.376 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: (mon.0) 789 : audit 2021-03-24T21:56:05.376 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: [INF] from='client.? v1:172.21.15.125:0/122654164' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2021-03-24T21:56:05.376 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: audit 2021 2021-03-24T21:56:05.376 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: -03-24T21:56: 2021-03-24T21:56:05.377 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: 04.759401+0000 mon.a ( 2021-03-24T21:56:05.377 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: mon.0) 790 : audit 2021-03-24T21:56:05.377 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T21:56:05.377 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: audit 2021-03-24T21:56:05.378 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: 2021-03-24T21:56:04.760046 2021-03-24T21:56:05.378 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: +0000 mon.a (mon.0) 791 : audit [INF] 2021-03-24T21:56:05.378 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T21:56:05.378 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: audit 2021 2021-03-24T21:56:05.378 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: -03-24T21:56 2021-03-24T21:56:05.379 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: :05.114246+0000 mon.a 2021-03-24T21:56:05.379 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: (mon.0) 792 : 2021-03-24T21:56:05.379 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: audit [INF] from='client.? v1:172.21.15.125:0/122654164' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2021-03-24T21:56:05.379 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:56:05.380 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: -03-24T21:56 2021-03-24T21:56:05.380 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: :05.114596+0000 mon.a ( 2021-03-24T21:56:05.380 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: mon.0) 793 : cluster 2021-03-24T21:56:05.380 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:05 smithi142 conmon[20125]: [DBG] osdmap e196: 8 total, 8 up, 6 in 2021-03-24T21:56:05.381 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:05.395 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:05 smithi125 conmon[19936]: cluster 2021-03-24T21:56:04.042700+0000 osd.7 (osd.7) 46 : cluster [DBG] 6.12 scrub ok 2021-03-24T21:56:05.395 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:05 smithi125 conmon[19936]: cluster 2021-03-24T21:56:04.051460+0000 osd.4 (osd.4) 55 : cluster [DBG] 4.1b scrub ok 2021-03-24T21:56:05.395 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:05 smithi125 conmon[19936]: cluster 2021-03-24T21:56:04.173936+0000 osd.5 (osd.5) 56 : cluster [DBG] 6.7 scrub ok 2021-03-24T21:56:05.396 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:05 smithi125 conmon[19936]: cluster 2021-03-24T21:56:04.223969+0000 mon.a (mon.0) 785 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T21:56:05.396 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:05 smithi125 conmon[19936]: cluster 2021-03-24T21:56:04 2021-03-24T21:56:05.396 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:05 smithi125 conmon[19936]: .224039+0000 mon.a (mon.0) 786 : 2021-03-24T21:56:05.396 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:05 smithi125 conmon[19936]: cluster [INF] Cluster is now healthy 2021-03-24T21:56:05.397 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:05 smithi125 conmon[19936]: audit 2021-03-24T21 2021-03-24T21:56:05.397 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:05 smithi125 conmon[19936]: :56:04.233805+0000 mon.a (mon.0) 2021-03-24T21:56:05.397 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:05 smithi125 conmon[19936]: 787 : audit [INF] from='client.? v1:172.21.15.125:0/3275940262' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T21:56:05.397 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:05 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:56:05.397 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:05 smithi125 conmon[19936]: 21:56:04.233950+0000 mon.a (mon.0) 788 : 2021-03-24T21:56:05.398 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:05 smithi125 conmon[19936]: cluster [DBG] osdmap e195: 8 total, 8 up, 6 in 2021-03-24T21:56:05.398 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:05 smithi125 conmon[19936]: audit 2021-03-24T21:56:05.398 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:05 smithi125 conmon[19936]: 2021-03-24T21:56:04.567069+0000 mon.a (mon.0) 789 : 2021-03-24T21:56:05.398 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:05 smithi125 conmon[19936]: audit [INF] from='client.? v1:172.21.15.125:0/122654164' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2021-03-24T21:56:05.399 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:05 smithi125 conmon[19936]: audit 2021-03-24T21:56:04.759401+0000 2021-03-24T21:56:05.399 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:05 smithi125 conmon[19936]: mon.a (mon.0) 790 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T21:56:05.399 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:05 smithi125 conmon[19936]: audit 2021-03-24T21:56:04.760046+0000 mon.a (mon. 2021-03-24T21:56:05.399 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:05 smithi125 conmon[19936]: 0) 791 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T21:56:05.399 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:05 smithi125 conmon[19936]: audit 2021-03- 2021-03-24T21:56:05.400 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:05 smithi125 conmon[19936]: 24T21:56:05.114246+0000 mon.a (mon.0 2021-03-24T21:56:05.400 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:05 smithi125 conmon[19936]: ) 792 : audit [INF] from='client.? v1:172.21.15.125:0/122654164' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2021-03-24T21:56:05.400 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:05 smithi125 conmon[19936]: cluster 2021-03-24T21:56:05.114596 2021-03-24T21:56:05.400 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:05 smithi125 conmon[19936]: +0000 mon.a (mon.0) 793 : cluster [DBG] osdmap e196: 8 total, 8 up, 6 in 2021-03-24T21:56:05.401 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:05 smithi125 conmon[24927]: cluster 2021-03-24T21:56:05.401 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:05 smithi125 conmon[24927]: 2021-03-24T21:56:04.042700+0000 osd.7 (osd.7 2021-03-24T21:56:05.401 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:05 smithi125 conmon[24927]: ) 46 : cluster [DBG] 6.12 scrub ok 2021-03-24T21:56:05.401 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:05 smithi125 conmon[24927]: cluster 2021-03-24T21:56:04.051460+0000 osd.4 (osd.4) 55 : cluster [DBG] 4.1b scrub ok 2021-03-24T21:56:05.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:05 smithi125 conmon[24927]: 2021-03-24T21:56:05.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:05 smithi125 conmon[24927]: cluster 2021-03-24T21:56:04.173936+0000 osd.5 (osd.5) 2021-03-24T21:56:05.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:05 smithi125 conmon[24927]: 56 : cluster [DBG] 6.7 scrub ok 2021-03-24T21:56:05.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:05 smithi125 conmon[24927]: cluster 2021-03-24T21:56:04.223969 2021-03-24T21:56:05.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:05 smithi125 conmon[24927]: +0000 mon.a (mon.0) 785 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T21:56:05.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:05 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:56:05.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:05 smithi125 conmon[24927]: T21:56:04.224039+0000 mon.a (mon.0) 786 : cluster [INF] Cluster is now healthy 2021-03-24T21:56:05.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:05 smithi125 conmon[24927]: audit 2021-03-24T21:56:04.233805+0000 mon.a (mon.0) 787 : audit 2021-03-24T21:56:05.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:05 smithi125 conmon[24927]: [INF] from='client.? v1:172.21.15.125:0/3275940262' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T21:56:05.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:05 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:56:05.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:05 smithi125 conmon[24927]: 21:56:04.233950+0000 mon.a (mon.0) 788 : cluster [DBG] osdmap e195: 8 total, 8 up, 6 in 2021-03-24T21:56:05.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:05 smithi125 conmon[24927]: audit 2021-03-24T21:56:04.567069+0000 mon.a (mon.0) 789 : 2021-03-24T21:56:05.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:05 smithi125 conmon[24927]: audit [INF] from='client.? v1:172.21.15.125:0/122654164' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2021-03-24T21:56:05.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:05 smithi125 conmon[24927]: audit 2021-03 2021-03-24T21:56:05.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:05 smithi125 conmon[24927]: -24T21:56:04.759401+0000 mon.a (mon.0) 790 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T21:56:05.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:05 smithi125 conmon[24927]: audit 2021-03-24T21:56:04.760046+0000 mon.a (mon.0) 791 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T21:56:05.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:05 smithi125 conmon[24927]: 2021-03-24T21:56:05.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:05 smithi125 conmon[24927]: audit 2021-03-24T21:56:05 2021-03-24T21:56:05.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:05 smithi125 conmon[24927]: .114246+0000 mon.a (mon.0) 792 : audit [INF] from='client.? v1:172.21.15.125:0/122654164' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2021-03-24T21:56:05.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:05 smithi125 conmon[24927]: cluster 2021-03-24T21:56:05.114596+0000 mon.a (mon. 2021-03-24T21:56:05.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:05 smithi125 conmon[24927]: 0) 793 : cluster [DBG] osdmap e196: 8 total, 8 up, 6 in 2021-03-24T21:56:05.438 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:05.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:05.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:56:05.543 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:05.580 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T21:56:05.605 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:05.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:05.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:56:05.713 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:05.777 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:05.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:05.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:56:05.884 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:05.945 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:05.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:05.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:56:06.050 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:06.102 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:06.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:06.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:56:06.130 INFO:tasks.rados.rados.0.smithi125.stdout:498: done (0 left) 2021-03-24T21:56:06.130 INFO:tasks.rados.rados.0.smithi125.stdout:499: write (excl) oid 16 current snap is 46 2021-03-24T21:56:06.131 INFO:tasks.rados.rados.0.smithi125.stdout:499: seq_num 210 ranges {729470=543821,1772395=538068,2900946=403005} 2021-03-24T21:56:06.141 INFO:tasks.rados.rados.0.smithi125.stdout:499: writing smithi12556470-16 from 729470 to 1273291 tid 1 2021-03-24T21:56:06.148 INFO:tasks.rados.rados.0.smithi125.stdout:499: writing smithi12556470-16 from 1772395 to 2310463 tid 2 2021-03-24T21:56:06.153 INFO:tasks.rados.rados.0.smithi125.stdout:499: writing smithi12556470-16 from 2900946 to 3303951 tid 3 2021-03-24T21:56:06.154 INFO:tasks.rados.rados.0.smithi125.stdout:500: write oid 40 current snap is 46 2021-03-24T21:56:06.154 INFO:tasks.rados.rados.0.smithi125.stdout:499: finishing write tid 1 to smithi12556470-16 2021-03-24T21:56:06.154 INFO:tasks.rados.rados.0.smithi125.stdout:500: seq_num 211 ranges {746277=619070,1830877=681726,2928369=617908} 2021-03-24T21:56:06.161 INFO:tasks.rados.rados.0.smithi125.stdout:500: writing smithi12556470-40 from 746277 to 1365347 tid 1 2021-03-24T21:56:06.168 INFO:tasks.rados.rados.0.smithi125.stdout:500: writing smithi12556470-40 from 1830877 to 2512603 tid 2 2021-03-24T21:56:06.174 INFO:tasks.rados.rados.0.smithi125.stdout:500: writing smithi12556470-40 from 2928369 to 3546277 tid 3 2021-03-24T21:56:06.175 INFO:tasks.rados.rados.0.smithi125.stdout:501: snap_remove snap 45 2021-03-24T21:56:06.208 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:06.267 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:06.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:06.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:56:06.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:06 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:56:06.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:06 smithi142 conmon[20125]: 56:04.740586+0000 mgr.y (mgr.14144) 358 : cluster [DBG] pgmap v455: 148 pgs: 148 active+clean; 132 MiB data, 420 MiB used, 536 GiB / 536 GiB avail; 41 MiB/s rd, 6.2 MiB/s wr, 53 op/s 2021-03-24T21:56:06.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:06 smithi142 conmon[20125]: cluster 2021-03-24T21:56:06.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:06 smithi142 conmon[20125]: 2021-03-24T21:56:05.075662+0000 osd.7 2021-03-24T21:56:06.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:06 smithi142 conmon[20125]: (osd.7) 47 : cluster [DBG] 1.0 scrub ok 2021-03-24T21:56:06.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:06 smithi142 conmon[20125]: 2021-03-24T21:56:06.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:06 smithi142 conmon[20125]: cluster 2021-03-24T21:56:06.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:06 smithi142 conmon[20125]: 2021-03-24T21:56: 2021-03-24T21:56:06.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:06 smithi142 conmon[20125]: 05.210536+0000 osd.5 (osd.5 2021-03-24T21:56:06.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:06 smithi142 conmon[20125]: ) 57 : cluster [DBG] 4.8 scrub ok 2021-03-24T21:56:06.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:06 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:56:06.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:06 smithi142 conmon[20125]: 03-24T21:56:06 2021-03-24T21:56:06.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:06 smithi142 conmon[20125]: .126507+0000 mon.a (mon.0) 794 : 2021-03-24T21:56:06.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:06 smithi142 conmon[20125]: cluster [DBG] osdmap e197: 8 total, 8 up, 6 in 2021-03-24T21:56:06.371 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:06.426 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:06.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:06.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:56:06.533 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:06.591 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:06.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:06.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:56:06.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:06 smithi125 conmon[24927]: cluster 2021-03-24T21:56:04.740586+0000 mgr.y (mgr.14144) 358 : cluster [DBG] pgmap v455: 148 pgs: 148 active+clean; 132 MiB data, 420 MiB used, 536 GiB / 536 GiB avail; 41 MiB/s rd, 6.2 MiB/s wr, 53 op/s 2021-03-24T21:56:06.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:06 smithi125 conmon[24927]: cluster 2021-03-24T21:56:05.075662+0000 osd.7 (osd.7) 47 : cluster [DBG] 1.0 scrub ok 2021-03-24T21:56:06.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:06 smithi125 conmon[24927]: cluster 2021-03-24T21:56:05.210536+0000 osd.5 (osd.5) 57 : cluster [DBG] 4.8 scrub ok 2021-03-24T21:56:06.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:06 smithi125 conmon[24927]: cluster 2021-03-24T21:56:06.126507+0000 mon.a (mon.0) 794 : cluster [DBG] osdmap e197: 8 total, 8 up, 6 in 2021-03-24T21:56:06.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:06 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:56:06.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:06 smithi125 conmon[19936]: :56:04.740586+0000 mgr.y (mgr.14144) 358 : cluster [DBG] pgmap v455: 148 pgs: 148 active+clean; 132 MiB data, 420 MiB used, 536 GiB / 536 GiB avail; 41 MiB/s rd, 6.2 MiB/s wr, 53 op/s 2021-03-24T21:56:06.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:06 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:56:06.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:06 smithi125 conmon[19936]: -24T21:56:05.075662+0000 osd.7 (osd 2021-03-24T21:56:06.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:06 smithi125 conmon[19936]: .7) 47 : cluster [DBG] 1.0 scrub ok 2021-03-24T21:56:06.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:06 smithi125 conmon[19936]: 2021-03-24T21:56:06.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:06 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:56:06.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:06 smithi125 conmon[19936]: 56:05.210536+0000 osd.5 (osd.5) 57 : cluster 2021-03-24T21:56:06.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:06 smithi125 conmon[19936]: [DBG] 4.8 scrub ok 2021-03-24T21:56:06.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:06 smithi125 conmon[19936]: cluster 2021-03-24T21:56:06.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:06 smithi125 conmon[19936]: 2021-03-24T21:56:06.126507 2021-03-24T21:56:06.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:06 smithi125 conmon[19936]: +0000 mon.a (mon.0) 794 : cluster [DBG] 2021-03-24T21:56:06.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:06 smithi125 conmon[19936]: osdmap e197: 8 total, 8 up, 6 in 2021-03-24T21:56:06.697 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:06.756 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:06.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:06.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:56:06.862 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:06.923 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:06.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:06.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:56:07.030 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:07.093 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:07.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:07.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:56:07.138 INFO:tasks.rados.rados.0.smithi125.stdout:501: done (2 left) 2021-03-24T21:56:07.139 INFO:tasks.rados.rados.0.smithi125.stdout:502: snap_remove snap 44 2021-03-24T21:56:07.199 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:07.247 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:07.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:07.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:56:07.304 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:07 smithi125 conmon[19936]: cluster 2021-03-24T21:56:05.966213+0000 osd.4 ( 2021-03-24T21:56:07.305 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:07 smithi125 conmon[19936]: osd.4) 56 : cluster [DBG] 4.18 scrub ok 2021-03-24T21:56:07.305 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:07 smithi125 conmon[19936]: cluster 2021-03-24T21:56:06.042847+0000 osd.7 (osd.7) 48 : cluster [DBG] 3.e scrub ok 2021-03-24T21:56:07.305 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:07 smithi125 conmon[19936]: cluster 2021-03-24T21:56:06.124131+0000 osd.2 2021-03-24T21:56:07.305 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:07 smithi125 conmon[19936]: (osd.2) 38 : cluster [DBG] 4.1c scrub ok 2021-03-24T21:56:07.306 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:07 smithi125 conmon[19936]: audit 2021-03-24T21:56: 2021-03-24T21:56:07.306 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:07 smithi125 conmon[19936]: 06.743296+0000 mon.a (mon.0) 795 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2021-03-24T21:56:07.306 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:07 smithi125 conmon[19936]: 2021-03-24T21:56:07.306 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:07 smithi125 conmon[19936]: audit 2021-03-24 2021-03-24T21:56:07.307 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:07 smithi125 conmon[19936]: T21:56:07.131790+0000 mon.a (mon.0) 796 : audit [INF] 2021-03-24T21:56:07.307 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:07 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2021-03-24T21:56:07.307 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:07 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:56:07.307 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:07 smithi125 conmon[19936]: -24T21:56:07.131967+0000 mon.a (mon.0) 797 : cluster [DBG] 2021-03-24T21:56:07.308 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:07 smithi125 conmon[19936]: osdmap e198: 8 total, 8 up, 6 in 2021-03-24T21:56:07.308 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:07 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:56:07.308 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:07 smithi125 conmon[24927]: T21:56:05.966213+0000 osd.4 (osd.4) 56 : cluster [DBG] 2021-03-24T21:56:07.308 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:07 smithi125 conmon[24927]: 4.18 scrub ok 2021-03-24T21:56:07.309 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:07 smithi125 conmon[24927]: cluster 2021-03-24T21:56:07.309 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:07 smithi125 conmon[24927]: 2021-03-24T21: 2021-03-24T21:56:07.309 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:07 smithi125 conmon[24927]: 56:06.042847+0000 2021-03-24T21:56:07.309 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:07 smithi125 conmon[24927]: osd.7 (osd.7) 2021-03-24T21:56:07.309 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:07 smithi125 conmon[24927]: 48 : cluster [DBG] 3.e scrub ok 2021-03-24T21:56:07.310 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:07 smithi125 conmon[24927]: 2021-03-24T21:56:07.310 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:07 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:56:07.310 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:07 smithi125 conmon[24927]: -03-24T21:56:06. 2021-03-24T21:56:07.310 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:07 smithi125 conmon[24927]: 124131+0000 osd.2 (osd.2) 38 : cluster [DBG] 2021-03-24T21:56:07.311 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:07 smithi125 conmon[24927]: 4.1c scrub ok 2021-03-24T21:56:07.311 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:07 smithi125 conmon[24927]: audit 2021-03-24T21:56:07.311 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:07 smithi125 conmon[24927]: 2021-03-24T21:56:06.743296+0000 2021-03-24T21:56:07.311 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:07 smithi125 conmon[24927]: mon.a (mon.0) 795 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2021-03-24T21:56:07.311 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:07 smithi125 conmon[24927]: 2021-03-24T21:56:07.312 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:07 smithi125 conmon[24927]: audit 2021-03-24T21:56:07.312 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:07 smithi125 conmon[24927]: 2021-03-24T21:56 2021-03-24T21:56:07.312 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:07 smithi125 conmon[24927]: :07.131790+0000 mon.a (mon.0 2021-03-24T21:56:07.312 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:07 smithi125 conmon[24927]: ) 796 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2021-03-24T21:56:07.312 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:07 smithi125 conmon[24927]: cluster 2021-03-24T21:56:07.313 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:07 smithi125 conmon[24927]: 2021-03-24T21:56:07. 2021-03-24T21:56:07.313 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:07 smithi125 conmon[24927]: 131967+0000 mon.a (mon.0 2021-03-24T21:56:07.313 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:07 smithi125 conmon[24927]: ) 797 : cluster [DBG] 2021-03-24T21:56:07.313 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:07 smithi125 conmon[24927]: osdmap e198: 8 total, 8 up, 6 in 2021-03-24T21:56:07.346 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:07 smithi142 conmon[20125]: cluster 2021-03-24T21:56:05.966213+0000 osd.4 (osd.4) 56 : cluster [DBG] 4.18 scrub ok 2021-03-24T21:56:07.347 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:07 smithi142 conmon[20125]: cluster 2021-03-24T21:56:06.042847+0000 osd.7 (osd.7) 48 : cluster [DBG] 3.e scrub ok 2021-03-24T21:56:07.347 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:07 smithi142 conmon[20125]: cluster 2021-03-24T21:56:07.347 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:07 smithi142 conmon[20125]: 2021-03-24T21:56:06.124131+0000 osd.2 (osd. 2021-03-24T21:56:07.347 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:07 smithi142 conmon[20125]: 2) 38 : cluster [DBG] 4.1c scrub ok 2021-03-24T21:56:07.347 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:07 smithi142 conmon[20125]: audit 2021-03-24T21: 2021-03-24T21:56:07.348 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:07 smithi142 conmon[20125]: 56:06.743296+0000 mon.a (mon.0) 795 : audit 2021-03-24T21:56:07.348 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:07 smithi142 conmon[20125]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2021-03-24T21:56:07.348 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:07 smithi142 conmon[20125]: audit 2021-03-24T21:56:07.348 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:07 smithi142 conmon[20125]: 2021-03-24T21:56:07.131790+0000 mon.a (mon. 2021-03-24T21:56:07.349 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:07 smithi142 conmon[20125]: 0) 796 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2021-03-24T21:56:07.349 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:07 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:56:07.349 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:07 smithi142 conmon[20125]: 56:07.131967+0000 mon.a (mon.0) 797 : cluster 2021-03-24T21:56:07.349 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:07 smithi142 conmon[20125]: [DBG] osdmap e198: 8 total, 8 up, 6 in 2021-03-24T21:56:07.353 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:07.404 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:07.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:07.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:56:07.416 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T21:56:07.435 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T21:56:07.453 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T21:56:07.473 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T21:56:07.494 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T21:56:07.509 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:07.515 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T21:56:07.538 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T21:56:07.562 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:07.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:07.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:56:07.570 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T21:56:07.573 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-24T21:56:07.573 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-24T21:56:07.573 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-24T21:56:07.573 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-24T21:56:07.574 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-24T21:56:07.574 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-24T21:56:07.574 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-24T21:56:07.574 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-24T21:56:07.632 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2021-03-24T21:56:07.633 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T21:56:07.668 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:07.727 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:07.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:07.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:56:07.833 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:07.891 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:07.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:07.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:56:07.997 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:08.052 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:08.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:08.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:56:08.140 INFO:tasks.rados.rados.0.smithi125.stdout:502: done (2 left) 2021-03-24T21:56:08.140 INFO:tasks.rados.rados.0.smithi125.stdout:503: copy_from oid 35 from oid 15 current snap is 46 2021-03-24T21:56:08.141 INFO:tasks.rados.rados.0.smithi125.stdout:504: write oid 49 current snap is 46 2021-03-24T21:56:08.141 INFO:tasks.rados.rados.0.smithi125.stdout:504: seq_num 212 ranges {763084=694319,1889359=541737,2955792=541902} 2021-03-24T21:56:08.150 INFO:tasks.rados.rados.0.smithi125.stdout:504: writing smithi12556470-49 from 763084 to 1457403 tid 1 2021-03-24T21:56:08.157 INFO:tasks.rados.rados.0.smithi125.stdout:504: writing smithi12556470-49 from 1889359 to 2431096 tid 2 2021-03-24T21:56:08.160 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:08.164 INFO:tasks.rados.rados.0.smithi125.stdout:504: writing smithi12556470-49 from 2955792 to 3497694 tid 3 2021-03-24T21:56:08.164 INFO:tasks.rados.rados.0.smithi125.stdout:505: write (excl) oid 23 current snap is 46 2021-03-24T21:56:08.164 INFO:tasks.rados.rados.0.smithi125.stdout:505: seq_num 213 ranges {779891=485921,1947841=685395,3383215=196676} 2021-03-24T21:56:08.172 INFO:tasks.rados.rados.0.smithi125.stdout:505: writing smithi12556470-23 from 779891 to 1265812 tid 1 2021-03-24T21:56:08.180 INFO:tasks.rados.rados.0.smithi125.stdout:505: writing smithi12556470-23 from 1947841 to 2633236 tid 2 2021-03-24T21:56:08.185 INFO:tasks.rados.rados.0.smithi125.stdout:505: writing smithi12556470-23 from 3383215 to 3579891 tid 3 2021-03-24T21:56:08.186 INFO:tasks.rados.rados.0.smithi125.stdout:506: delete oid 38 current snap is 46 2021-03-24T21:56:08.186 INFO:tasks.rados.rados.0.smithi125.stdout:499: finishing write tid 2 to smithi12556470-16 2021-03-24T21:56:08.186 INFO:tasks.rados.rados.0.smithi125.stdout:499: finishing write tid 3 to smithi12556470-16 2021-03-24T21:56:08.187 INFO:tasks.rados.rados.0.smithi125.stdout:499: finishing write tid 4 to smithi12556470-16 2021-03-24T21:56:08.187 INFO:tasks.rados.rados.0.smithi125.stdout:499: finishing write tid 5 to smithi12556470-16 2021-03-24T21:56:08.187 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 16 v 192 (ObjNum 210 snap 46 seq_num 210) dirty exists 2021-03-24T21:56:08.187 INFO:tasks.rados.rados.0.smithi125.stdout:499: left oid 16 (ObjNum 210 snap 46 seq_num 210) 2021-03-24T21:56:08.187 INFO:tasks.rados.rados.0.smithi125.stdout:500: finishing write tid 1 to smithi12556470-40 2021-03-24T21:56:08.188 INFO:tasks.rados.rados.0.smithi125.stdout:500: finishing write tid 2 to smithi12556470-40 2021-03-24T21:56:08.188 INFO:tasks.rados.rados.0.smithi125.stdout:500: finishing write tid 3 to smithi12556470-40 2021-03-24T21:56:08.188 INFO:tasks.rados.rados.0.smithi125.stdout:500: finishing write tid 4 to smithi12556470-40 2021-03-24T21:56:08.188 INFO:tasks.rados.rados.0.smithi125.stdout:500: finishing write tid 5 to smithi12556470-40 2021-03-24T21:56:08.188 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 40 v 155 (ObjNum 211 snap 46 seq_num 211) dirty exists 2021-03-24T21:56:08.189 INFO:tasks.rados.rados.0.smithi125.stdout:500: left oid 40 (ObjNum 211 snap 46 seq_num 211) 2021-03-24T21:56:08.189 INFO:tasks.rados.rados.0.smithi125.stdout:503: finishing copy_from to smithi12556470-35 2021-03-24T21:56:08.189 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 35 v 92 (ObjNum 150 snap 30 seq_num 150) dirty exists 2021-03-24T21:56:08.189 INFO:tasks.rados.rados.0.smithi125.stdout:504: finishing write tid 1 to smithi12556470-49 2021-03-24T21:56:08.189 INFO:tasks.rados.rados.0.smithi125.stdout:503: finishing copy_from racing read to smithi12556470-35 2021-03-24T21:56:08.190 INFO:tasks.rados.rados.0.smithi125.stdout:504: finishing write tid 2 to smithi12556470-49 2021-03-24T21:56:08.190 INFO:tasks.rados.rados.0.smithi125.stdout:504: finishing write tid 3 to smithi12556470-49 2021-03-24T21:56:08.190 INFO:tasks.rados.rados.0.smithi125.stdout:504: finishing write tid 4 to smithi12556470-49 2021-03-24T21:56:08.190 INFO:tasks.rados.rados.0.smithi125.stdout:504: finishing write tid 5 to smithi12556470-49 2021-03-24T21:56:08.191 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 49 v 174 (ObjNum 212 snap 46 seq_num 212) dirty exists 2021-03-24T21:56:08.191 INFO:tasks.rados.rados.0.smithi125.stdout:504: left oid 49 (ObjNum 212 snap 46 seq_num 212) 2021-03-24T21:56:08.191 INFO:tasks.rados.rados.0.smithi125.stdout:505: finishing write tid 1 to smithi12556470-23 2021-03-24T21:56:08.191 INFO:tasks.rados.rados.0.smithi125.stdout:505: finishing write tid 2 to smithi12556470-23 2021-03-24T21:56:08.192 INFO:tasks.rados.rados.0.smithi125.stdout:499: done (5 left) 2021-03-24T21:56:08.193 INFO:tasks.rados.rados.0.smithi125.stdout:500: done (4 left) 2021-03-24T21:56:08.193 INFO:tasks.rados.rados.0.smithi125.stdout:503: done (3 left) 2021-03-24T21:56:08.193 INFO:tasks.rados.rados.0.smithi125.stdout:504: done (2 left) 2021-03-24T21:56:08.193 INFO:tasks.rados.rados.0.smithi125.stdout:506: done (1 left) 2021-03-24T21:56:08.194 INFO:tasks.rados.rados.0.smithi125.stdout:507: delete oid 10 current snap is 46 2021-03-24T21:56:08.194 INFO:tasks.rados.rados.0.smithi125.stdout:505: finishing write tid 3 to smithi12556470-23 2021-03-24T21:56:08.194 INFO:tasks.rados.rados.0.smithi125.stdout:505: finishing write tid 4 to smithi12556470-23 2021-03-24T21:56:08.194 INFO:tasks.rados.rados.0.smithi125.stdout:505: finishing write tid 5 to smithi12556470-23 2021-03-24T21:56:08.194 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 23 v 177 (ObjNum 213 snap 46 seq_num 213) dirty exists 2021-03-24T21:56:08.195 INFO:tasks.rados.rados.0.smithi125.stdout:505: left oid 23 (ObjNum 213 snap 46 seq_num 213) 2021-03-24T21:56:08.195 INFO:tasks.rados.rados.0.smithi125.stdout:505: done (1 left) 2021-03-24T21:56:08.195 INFO:tasks.rados.rados.0.smithi125.stdout:507: done (0 left) 2021-03-24T21:56:08.195 INFO:tasks.rados.rados.0.smithi125.stdout:508: copy_from oid 28 from oid 37 current snap is 46 2021-03-24T21:56:08.195 INFO:tasks.rados.rados.0.smithi125.stdout:509: snap_create 2021-03-24T21:56:08.205 INFO:tasks.rados.rados.0.smithi125.stdout:508: finishing copy_from to smithi12556470-28 2021-03-24T21:56:08.206 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 28 v 56 (ObjNum 160 snap 35 seq_num 160) dirty exists 2021-03-24T21:56:08.206 INFO:tasks.rados.rados.0.smithi125.stdout:508: finishing copy_from racing read to smithi12556470-28 2021-03-24T21:56:08.207 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:08.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:08.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:56:08.288 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd set noscrub 2021-03-24T21:56:08.312 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:08.369 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:08.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:08.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:56:08.476 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:08.534 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:08.538 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:08 smithi142 conmon[20125]: cluster 2021-03-24T21:56:06.741356+0000 mgr.y (mgr.14144) 359 : cluster 2021-03-24T21:56:08.538 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:08 smithi142 conmon[20125]: [DBG] pgmap v458: 148 pgs: 148 active+clean; 132 MiB data, 424 MiB used, 536 GiB / 536 GiB avail; 39 MiB/s rd, 6.2 MiB/s wr, 51 op/s 2021-03-24T21:56:08.539 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:08 smithi142 conmon[20125]: cluster 2021-03-24T21:56:07.255491+0000 osd.0 (osd.0) 55 : cluster [DBG] 5.1e scrub ok 2021-03-24T21:56:08.539 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:08 smithi142 conmon[20125]: cluster 2021-03-24T21:56:07.272628+0000 osd.5 (osd.5) 58 : cluster [DBG] 6.1d scrub ok 2021-03-24T21:56:08.539 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:08 smithi142 conmon[20125]: audit 2021-03-24T21:56 2021-03-24T21:56:08.539 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:08 smithi142 conmon[20125]: :07.394119+0000 mon.c (mon.2) 35 2021-03-24T21:56:08.540 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:08 smithi142 conmon[20125]: : audit [DBG] from='client.? v1:172.21.15.125:0/3780463770' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:56:08.540 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:08 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:56:08.540 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:08 smithi142 conmon[20125]: 24T21:56:08.133304+0000 mon.a (mon.0) 2021-03-24T21:56:08.540 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:08 smithi142 conmon[20125]: 798 : cluster [DBG] osdmap e199: 8 total, 8 up, 6 in 2021-03-24T21:56:08.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:08.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:56:08.594 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:56:08.642 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:08.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:08 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:56:08.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:08 smithi125 conmon[24927]: :56:06.741356+0000 mgr.y (mgr.14144) 359 : cluster [DBG] pgmap v458: 148 pgs: 148 active+clean; 132 MiB data, 424 MiB used, 536 GiB / 536 GiB avail; 39 MiB/s rd, 6.2 MiB/s wr, 51 op/s 2021-03-24T21:56:08.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:08 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:56:08.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:08 smithi125 conmon[24927]: -03-24T21:56:07.255491+0000 2021-03-24T21:56:08.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:08 smithi125 conmon[24927]: osd.0 (osd.0) 55 : cluster [DBG] 5.1e scrub ok 2021-03-24T21:56:08.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:08 smithi125 conmon[24927]: 2021-03-24T21:56:08.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:08 smithi125 conmon[24927]: cluster 2021-03-24T21:56: 2021-03-24T21:56:08.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:08 smithi125 conmon[24927]: 07.272628+0000 osd.5 (osd.5) 58 : 2021-03-24T21:56:08.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:08 smithi125 conmon[24927]: cluster [DBG] 6.1d scrub ok 2021-03-24T21:56:08.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:08 smithi125 conmon[24927]: audit 2021-03-24T 2021-03-24T21:56:08.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:08 smithi125 conmon[24927]: 21:56:07.394119+0000 mon.c (mon. 2021-03-24T21:56:08.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:08 smithi125 conmon[24927]: 2) 35 : audit [DBG] from='client.? v1:172.21.15.125:0/3780463770' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:56:08.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:08 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:56:08.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:08 smithi125 conmon[24927]: 24T21:56:08.133304+0000 mon.a ( 2021-03-24T21:56:08.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:08 smithi125 conmon[24927]: mon.0) 798 : cluster [DBG] osdmap e199: 8 total, 8 up, 6 in 2021-03-24T21:56:08.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:08 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:56:08.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:08 smithi125 conmon[19936]: 21:56:06.741356+0000 mgr.y (mgr.14144) 359 : cluster [DBG] pgmap v458: 148 pgs: 148 active+clean; 132 MiB data, 424 MiB used, 536 GiB / 536 GiB avail; 39 MiB/s rd, 6.2 MiB/s wr, 51 op/s 2021-03-24T21:56:08.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:08 smithi125 conmon[19936]: cluster 2021-03-24T21:56: 2021-03-24T21:56:08.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:08 smithi125 conmon[19936]: 07.255491+0000 osd.0 (osd.0) 55 2021-03-24T21:56:08.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:08 smithi125 conmon[19936]: : cluster [DBG] 5.1e scrub ok 2021-03-24T21:56:08.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:08 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:56:08.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:08 smithi125 conmon[19936]: -24T21:56:07.272628+0000 osd.5 2021-03-24T21:56:08.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:08 smithi125 conmon[19936]: (osd.5) 58 : cluster [DBG] 6.1d scrub ok 2021-03-24T21:56:08.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:08 smithi125 conmon[19936]: audit 2021-03-24T21:56:08.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:08 smithi125 conmon[19936]: 2021-03-24T21:56: 2021-03-24T21:56:08.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:08 smithi125 conmon[19936]: 07.394119+0000 mon.c (mon.2) 35 : audit 2021-03-24T21:56:08.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:08 smithi125 conmon[19936]: [DBG] from='client.? v1:172.21.15.125:0/3780463770' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:56:08.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:08 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:56:08.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:08 smithi125 conmon[19936]: 21:56:08.133304+0000 2021-03-24T21:56:08.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:08 smithi125 conmon[19936]: mon.a (mon.0) 798 : 2021-03-24T21:56:08.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:08 smithi125 conmon[19936]: cluster [DBG] osdmap e199: 8 total, 8 up, 6 in 2021-03-24T21:56:08.668 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.7 dump_blocked_ops 2021-03-24T21:56:08.696 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:08.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:08.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:56:08.804 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:08.856 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:08.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:08.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:56:08.961 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:08.990 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T21:56:09.020 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:09.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:09.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:56:09.127 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:09.138 INFO:tasks.rados.rados.0.smithi125.stdout:508: done (1 left) 2021-03-24T21:56:09.138 INFO:tasks.rados.rados.0.smithi125.stdout:509: done (0 left) 2021-03-24T21:56:09.139 INFO:tasks.rados.rados.0.smithi125.stdout:510: write (excl) oid 26 current snap is 47 2021-03-24T21:56:09.139 INFO:tasks.rados.rados.0.smithi125.stdout:510: seq_num 216 ranges {430312=711668,1723287=549075,2781837=703343} 2021-03-24T21:56:09.147 INFO:tasks.rados.rados.0.smithi125.stdout:510: writing smithi12556470-26 from 430312 to 1141980 tid 1 2021-03-24T21:56:09.155 INFO:tasks.rados.rados.0.smithi125.stdout:510: writing smithi12556470-26 from 1723287 to 2272362 tid 2 2021-03-24T21:56:09.163 INFO:tasks.rados.rados.0.smithi125.stdout:510: writing smithi12556470-26 from 2781837 to 3485180 tid 3 2021-03-24T21:56:09.163 INFO:tasks.rados.rados.0.smithi125.stdout:511: read oid 36 snap -1 2021-03-24T21:56:09.164 INFO:tasks.rados.rados.0.smithi125.stdout:511: expect (ObjNum 180 snap 39 seq_num 180) 2021-03-24T21:56:09.164 INFO:tasks.rados.rados.0.smithi125.stdout:512: read oid 8 snap -1 2021-03-24T21:56:09.164 INFO:tasks.rados.rados.0.smithi125.stdout:512: expect (ObjNum 161 snap 35 seq_num 161) 2021-03-24T21:56:09.164 INFO:tasks.rados.rados.0.smithi125.stdout:513: snap_remove snap 46 2021-03-24T21:56:09.186 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:09.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:09.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:56:09.292 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:09.349 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:09.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:09.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:56:09.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:09 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T21:56:09.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:09 smithi125 conmon[19936]: T21:56:07.977114+0000 osd.4 (osd.4) 57 : cluster [DBG] 5.c scrub ok 2021-03-24T21:56:09.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:09 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:56:09.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:09 smithi125 conmon[19936]: 03-24T21:56:08 2021-03-24T21:56:09.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:09 smithi125 conmon[19936]: .744792+0000 mon.a (mon.0 2021-03-24T21:56:09.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:09 smithi125 conmon[19936]: ) 799 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2021-03-24T21:56:09.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:09 smithi125 conmon[19936]: audit 2021 2021-03-24T21:56:09.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:09 smithi125 conmon[19936]: -03-24T21:56:09. 2021-03-24T21:56:09.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:09 smithi125 conmon[19936]: 136676+0000 mon.a (mon.0 2021-03-24T21:56:09.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:09 smithi125 conmon[19936]: ) 800 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2021-03-24T21:56:09.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:09 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:56:09.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:09 smithi125 conmon[19936]: 03-24T21:56:09 2021-03-24T21:56:09.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:09 smithi125 conmon[19936]: .136964+0000 mon.a (mon.0 2021-03-24T21:56:09.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:09 smithi125 conmon[19936]: ) 801 : cluster [DBG] osdmap e200: 8 total, 8 up, 6 in 2021-03-24T21:56:09.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:09 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:56:09.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:09 smithi125 conmon[24927]: T21:56:07.977114+0000 osd.4 (osd.4) 57 : cluster [DBG] 5.c scrub ok 2021-03-24T21:56:09.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:09 smithi125 conmon[24927]: audit 2021-03-24T21:56:09.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:09 smithi125 conmon[24927]: 2021-03-24T21:56: 2021-03-24T21:56:09.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:09 smithi125 conmon[24927]: 08.744792+0000 mon.a (mon 2021-03-24T21:56:09.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:09 smithi125 conmon[24927]: .0) 799 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2021-03-24T21:56:09.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:09 smithi125 conmon[24927]: audit 2021-03-24T21:56:09.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:09 smithi125 conmon[24927]: 2021-03-24T21:56 2021-03-24T21:56:09.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:09 smithi125 conmon[24927]: :09.136676+0000 mon.a (mon.0 2021-03-24T21:56:09.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:09 smithi125 conmon[24927]: ) 800 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2021-03-24T21:56:09.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:09 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:56:09.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:09 smithi125 conmon[24927]: -03-24T21:56:09.136964 2021-03-24T21:56:09.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:09 smithi125 conmon[24927]: +0000 mon.a (mon.0) 801 : cluster 2021-03-24T21:56:09.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:09 smithi125 conmon[24927]: [DBG] osdmap e200: 8 total, 8 up, 6 in 2021-03-24T21:56:09.438 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:09 smithi142 conmon[20125]: cluster 2021-03-24T21:56: 2021-03-24T21:56:09.438 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:09 smithi142 conmon[20125]: 07.977114+0000 osd.4 (osd.4) 57 : cluster [DBG] 5.c scrub ok 2021-03-24T21:56:09.438 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:09 smithi142 conmon[20125]: audit 2021-03-24T21: 2021-03-24T21:56:09.438 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:09 smithi142 conmon[20125]: 56:08.744792+0000 mon.a (mon.0) 799 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2021-03-24T21:56:09.439 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:09 smithi142 conmon[20125]: audit 2021-03-24T21:56:09.136676 2021-03-24T21:56:09.439 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:09 smithi142 conmon[20125]: +0000 mon.a (mon.0) 800 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2021-03-24T21:56:09.439 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:09 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:56:09.439 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:09 smithi142 conmon[20125]: T21:56:09.136964+0000 mon.a (mon.0) 801 : cluster 2021-03-24T21:56:09.440 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:09 smithi142 conmon[20125]: [DBG] osdmap e200: 8 total, 8 up, 6 in 2021-03-24T21:56:09.454 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:09.516 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:09.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:09.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:56:09.623 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:09.684 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:09.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:09.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:56:09.792 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:09.850 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:09.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:09.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:56:09.957 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:10.014 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:10.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:10.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:56:10.120 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:10.130 INFO:tasks.rados.rados.0.smithi125.stdout:513: done (3 left) 2021-03-24T21:56:10.130 INFO:tasks.rados.rados.0.smithi125.stdout:514: read oid 9 snap -1 2021-03-24T21:56:10.130 INFO:tasks.rados.rados.0.smithi125.stdout:514: expect (ObjNum 68 snap 7 seq_num 68) 2021-03-24T21:56:10.131 INFO:tasks.rados.rados.0.smithi125.stdout:515: delete oid 32 current snap is 47 2021-03-24T21:56:10.131 INFO:tasks.rados.rados.0.smithi125.stdout:510: finishing write tid 1 to smithi12556470-26 2021-03-24T21:56:10.131 INFO:tasks.rados.rados.0.smithi125.stdout:510: finishing write tid 2 to smithi12556470-26 2021-03-24T21:56:10.131 INFO:tasks.rados.rados.0.smithi125.stdout:510: finishing write tid 3 to smithi12556470-26 2021-03-24T21:56:10.132 INFO:tasks.rados.rados.0.smithi125.stdout:510: finishing write tid 4 to smithi12556470-26 2021-03-24T21:56:10.132 INFO:tasks.rados.rados.0.smithi125.stdout:510: finishing write tid 5 to smithi12556470-26 2021-03-24T21:56:10.132 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 26 v 110 (ObjNum 216 snap 47 seq_num 216) dirty exists 2021-03-24T21:56:10.132 INFO:tasks.rados.rados.0.smithi125.stdout:510: left oid 26 (ObjNum 216 snap 47 seq_num 216) 2021-03-24T21:56:10.133 INFO:tasks.rados.rados.0.smithi125.stdout:511: expect (ObjNum 180 snap 39 seq_num 180) 2021-03-24T21:56:10.185 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:10.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:10.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:56:10.291 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:10.301 INFO:tasks.rados.rados.0.smithi125.stdout:512: expect (ObjNum 161 snap 35 seq_num 161) 2021-03-24T21:56:10.352 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:10.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:10.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:56:10.458 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:10.459 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-24T21:56:10.467 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:10 smithi125 conmon[19936]: cluster 2021-03-24T21:56:08.742610 2021-03-24T21:56:10.467 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:10 smithi125 conmon[19936]: +0000 mgr.y (mgr.14144) 360 : cluster [DBG] pgmap v461: 158 pgs: 1 peering, 1 activating, 8 unknown, 148 active+clean; 135 MiB data, 415 MiB used, 536 GiB / 536 GiB avail; 255 B/s rd, 1.1 MiB/s wr, 1 op/s 2021-03-24T21:56:10.468 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:10 smithi125 conmon[19936]: cluster 2021-03-24T21:56: 2021-03-24T21:56:10.468 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:10 smithi125 conmon[19936]: 08.985690+0000 osd.4 (osd.4) 58 : cluster [DBG] 6.e scrub ok 2021-03-24T21:56:10.468 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:10 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:56:10.468 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:10 smithi125 conmon[19936]: -24T21:56:09.119556+0000 2021-03-24T21:56:10.469 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:10 smithi125 conmon[19936]: osd.2 (osd.2) 39 : cluster [DBG] 2021-03-24T21:56:10.469 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:10 smithi125 conmon[19936]: 4.12 scrub ok 2021-03-24T21:56:10.469 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:10 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:56:10.469 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:10 smithi125 conmon[19936]: -03-24T21:56:10.118949+0000 mon.a ( 2021-03-24T21:56:10.470 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:10 smithi125 conmon[19936]: mon.0) 802 : cluster [DBG] osdmap e201: 8 total, 8 up, 6 in 2021-03-24T21:56:10.470 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:10 smithi125 conmon[19936]: 2021-03-24T21:56:10.470 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:10 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:56:10.471 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:10 smithi125 conmon[24927]: 56:08.742610+0000 mgr.y (mgr.14144) 360 : cluster [DBG] pgmap v461: 158 pgs: 1 peering, 1 activating, 8 unknown, 148 active+clean; 135 MiB data, 415 MiB used, 536 GiB / 536 GiB avail; 255 B/s rd, 1.1 MiB/s wr, 1 op/s 2021-03-24T21:56:10.471 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:10 smithi125 conmon[24927]: 2021-03-24T21:56:10.471 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:10 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:56:10.471 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:10 smithi125 conmon[24927]: :56:08.985690+0000 osd.4 (osd. 2021-03-24T21:56:10.472 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:10 smithi125 conmon[24927]: 4) 58 : cluster [DBG] 6.e scrub ok 2021-03-24T21:56:10.472 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:10 smithi125 conmon[24927]: 2021-03-24T21:56:10.472 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:10 smithi125 conmon[24927]: cluster 2021-03-24T21:56: 2021-03-24T21:56:10.472 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:10 smithi125 conmon[24927]: 09.119556+0000 osd.2 (osd.2) 39 2021-03-24T21:56:10.473 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:10 smithi125 conmon[24927]: : cluster [DBG] 4.12 scrub ok 2021-03-24T21:56:10.473 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:10 smithi125 conmon[24927]: cluster 2021-03-24T21:56:10. 2021-03-24T21:56:10.473 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:10 smithi125 conmon[24927]: 118949+0000 mon.a (mon.0) 802 : cluster [DBG] 2021-03-24T21:56:10.473 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:10 smithi125 conmon[24927]: osdmap e201: 8 total, 8 up, 6 in 2021-03-24T21:56:10.483 INFO:tasks.rados.rados.0.smithi125.stdout:514: expect (ObjNum 68 snap 7 seq_num 68) 2021-03-24T21:56:10.506 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:10.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:10.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:56:10.563 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:10 smithi142 conmon[20125]: cluster 2021-03-24T21:56: 2021-03-24T21:56:10.563 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:10 smithi142 conmon[20125]: 08.742610+0000 mgr.y (mgr.14144) 360 : cluster [DBG] pgmap v461: 158 pgs: 1 peering, 1 activating, 8 unknown, 148 active+clean; 135 MiB data, 415 MiB used, 536 GiB / 536 GiB avail; 255 B/s rd, 1.1 MiB/s wr, 1 op/s 2021-03-24T21:56:10.564 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:10 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:56:10.564 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:10 smithi142 conmon[20125]: 56:08.985690+0000 osd.4 (osd.4) 58 : cluster [DBG] 6.e scrub ok 2021-03-24T21:56:10.565 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:10 smithi142 conmon[20125]: 2021-03-24T21:56:10.565 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:10 smithi142 conmon[20125]: cluster 2021-03-24T21:56:10.565 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:10 smithi142 conmon[20125]: 2021-03-24T21:56:09.119556+0000 osd.2 2021-03-24T21:56:10.566 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:10 smithi142 conmon[20125]: (osd.2) 39 : cluster [DBG] 2021-03-24T21:56:10.566 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:10 smithi142 conmon[20125]: 4.12 scrub ok 2021-03-24T21:56:10.567 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:10 smithi142 conmon[20125]: cluster 2021-03-24T21:56:10.567 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:10 smithi142 conmon[20125]: 2021-03-24T21:56:10.118949 2021-03-24T21:56:10.568 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:10 smithi142 conmon[20125]: +0000 mon.a (mon.0) 802 : cluster [DBG] osdmap e201: 8 total, 8 up, 6 in 2021-03-24T21:56:10.615 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:10.668 INFO:tasks.rados.rados.0.smithi125.stdout:510: done (4 left) 2021-03-24T21:56:10.668 INFO:tasks.rados.rados.0.smithi125.stdout:511: done (3 left) 2021-03-24T21:56:10.669 INFO:tasks.rados.rados.0.smithi125.stdout:512: done (2 left) 2021-03-24T21:56:10.670 INFO:tasks.rados.rados.0.smithi125.stdout:514: done (1 left) 2021-03-24T21:56:10.670 INFO:tasks.rados.rados.0.smithi125.stdout:515: done (0 left) 2021-03-24T21:56:10.670 INFO:tasks.rados.rados.0.smithi125.stdout:516: rollback oid 1 current snap is 47 2021-03-24T21:56:10.671 INFO:tasks.rados.rados.0.smithi125.stdout:516: done (0 left) 2021-03-24T21:56:10.671 INFO:tasks.rados.rados.0.smithi125.stdout:517: read oid 7 snap -1 2021-03-24T21:56:10.671 INFO:tasks.rados.rados.0.smithi125.stdout:517: expect (ObjNum 150 snap 30 seq_num 150) 2021-03-24T21:56:10.671 INFO:tasks.rados.rados.0.smithi125.stdout:518: read oid 9 snap -1 2021-03-24T21:56:10.671 INFO:tasks.rados.rados.0.smithi125.stdout:518: expect (ObjNum 68 snap 7 seq_num 68) 2021-03-24T21:56:10.672 INFO:tasks.rados.rados.0.smithi125.stdout:519: read oid 12 snap -1 2021-03-24T21:56:10.672 INFO:tasks.rados.rados.0.smithi125.stdout:519: expect deleted 2021-03-24T21:56:10.672 INFO:tasks.rados.rados.0.smithi125.stdout:520: read oid 18 snap -1 2021-03-24T21:56:10.672 INFO:tasks.rados.rados.0.smithi125.stdout:520: expect (ObjNum 194 snap 43 seq_num 194) 2021-03-24T21:56:10.672 INFO:tasks.rados.rados.0.smithi125.stdout:521: snap_create 2021-03-24T21:56:10.674 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:10.678 INFO:tasks.rados.rados.0.smithi125.stdout:517: expect (ObjNum 150 snap 30 seq_num 150) 2021-03-24T21:56:10.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:10.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:56:10.740 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:56:10.781 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:10.815 INFO:tasks.rados.rados.0.smithi125.stdout:518: expect (ObjNum 68 snap 7 seq_num 68) 2021-03-24T21:56:10.841 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:10.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:10.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:56:10.947 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:10.998 INFO:tasks.rados.rados.0.smithi125.stdout:520: expect (ObjNum 194 snap 43 seq_num 194) 2021-03-24T21:56:11.000 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:11.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:11.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:56:11.108 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:11.132 INFO:teuthology.orchestra.run.smithi125.stderr:noscrub is set 2021-03-24T21:56:11.152 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2021-03-24T21:56:11.152 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T21:56:11.156 INFO:tasks.rados.rados.0.smithi125.stdout:517: done (4 left) 2021-03-24T21:56:11.157 INFO:tasks.rados.rados.0.smithi125.stdout:518: done (3 left) 2021-03-24T21:56:11.157 INFO:tasks.rados.rados.0.smithi125.stdout:519: done (2 left) 2021-03-24T21:56:11.157 INFO:tasks.rados.rados.0.smithi125.stdout:520: done (1 left) 2021-03-24T21:56:11.157 INFO:tasks.rados.rados.0.smithi125.stdout:521: done (0 left) 2021-03-24T21:56:11.158 INFO:tasks.rados.rados.0.smithi125.stdout:522: read oid 7 snap 47 2021-03-24T21:56:11.158 INFO:tasks.rados.rados.0.smithi125.stdout:522: expect (ObjNum 150 snap 30 seq_num 150) 2021-03-24T21:56:11.160 INFO:tasks.rados.rados.0.smithi125.stdout:523: rollback oid 27 current snap is 48 2021-03-24T21:56:11.160 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 27 to 47 2021-03-24T21:56:11.160 INFO:tasks.rados.rados.0.smithi125.stdout:524: write (excl) oid 46 current snap is 48 2021-03-24T21:56:11.160 INFO:tasks.rados.rados.0.smithi125.stdout:524: seq_num 218 ranges {463926=462166,1723898=436391,2603977=725887} 2021-03-24T21:56:11.164 INFO:tasks.rados.rados.0.smithi125.stdout:524: writing smithi12556470-46 from 463926 to 926092 tid 1 2021-03-24T21:56:11.167 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:11.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:11.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:56:11.174 INFO:tasks.rados.rados.0.smithi125.stdout:524: writing smithi12556470-46 from 1723898 to 2160289 tid 2 2021-03-24T21:56:11.182 INFO:tasks.rados.rados.0.smithi125.stdout:524: writing smithi12556470-46 from 2603977 to 3329864 tid 3 2021-03-24T21:56:11.183 INFO:tasks.rados.rados.0.smithi125.stdout:525: rollback oid 36 current snap is 48 2021-03-24T21:56:11.183 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 36 to 47 2021-03-24T21:56:11.183 INFO:tasks.rados.rados.0.smithi125.stdout:522: expect (ObjNum 150 snap 30 seq_num 150) 2021-03-24T21:56:11.275 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:11.323 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:11.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:11.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:56:11.347 INFO:tasks.rados.rados.0.smithi125.stdout:523: finishing rollback tid 0 to smithi12556470-27 2021-03-24T21:56:11.347 INFO:tasks.rados.rados.0.smithi125.stdout:523: finishing rollback tid 1 to smithi12556470-27 2021-03-24T21:56:11.348 INFO:tasks.rados.rados.0.smithi125.stdout:523: finishing rollback tid 2 to smithi12556470-27 2021-03-24T21:56:11.348 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 27 v 202 (ObjNum 189 snap 41 seq_num 189) dirty exists 2021-03-24T21:56:11.348 INFO:tasks.rados.rados.0.smithi125.stdout:522: done (3 left) 2021-03-24T21:56:11.348 INFO:tasks.rados.rados.0.smithi125.stdout:523: done (2 left) 2021-03-24T21:56:11.349 INFO:tasks.rados.rados.0.smithi125.stdout:526: delete oid 43 current snap is 48 2021-03-24T21:56:11.349 INFO:tasks.rados.rados.0.smithi125.stdout:524: finishing write tid 1 to smithi12556470-46 2021-03-24T21:56:11.349 INFO:tasks.rados.rados.0.smithi125.stdout:524: finishing write tid 2 to smithi12556470-46 2021-03-24T21:56:11.349 INFO:tasks.rados.rados.0.smithi125.stdout:525: finishing rollback tid 0 to smithi12556470-36 2021-03-24T21:56:11.350 INFO:tasks.rados.rados.0.smithi125.stdout:525: finishing rollback tid 1 to smithi12556470-36 2021-03-24T21:56:11.350 INFO:tasks.rados.rados.0.smithi125.stdout:525: finishing rollback tid 2 to smithi12556470-36 2021-03-24T21:56:11.350 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 36 v 135 (ObjNum 180 snap 39 seq_num 180) dirty exists 2021-03-24T21:56:11.350 INFO:tasks.rados.rados.0.smithi125.stdout:524: finishing write tid 3 to smithi12556470-46 2021-03-24T21:56:11.350 INFO:tasks.rados.rados.0.smithi125.stdout:524: finishing write tid 4 to smithi12556470-46 2021-03-24T21:56:11.351 INFO:tasks.rados.rados.0.smithi125.stdout:524: finishing write tid 5 to smithi12556470-46 2021-03-24T21:56:11.351 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 46 v 64 (ObjNum 218 snap 48 seq_num 218) dirty exists 2021-03-24T21:56:11.351 INFO:tasks.rados.rados.0.smithi125.stdout:524: left oid 46 (ObjNum 218 snap 48 seq_num 218) 2021-03-24T21:56:11.352 INFO:tasks.rados.rados.0.smithi125.stdout:524: done (2 left) 2021-03-24T21:56:11.352 INFO:tasks.rados.rados.0.smithi125.stdout:525: done (1 left) 2021-03-24T21:56:11.352 INFO:tasks.rados.rados.0.smithi125.stdout:526: done (0 left) 2021-03-24T21:56:11.353 INFO:tasks.rados.rados.0.smithi125.stdout:527: rollback oid 27 current snap is 48 2021-03-24T21:56:11.353 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 27 to 47 2021-03-24T21:56:11.353 INFO:tasks.rados.rados.0.smithi125.stdout:528: rollback oid 10 current snap is 48 2021-03-24T21:56:11.353 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 10 to 47 2021-03-24T21:56:11.354 INFO:tasks.rados.rados.0.smithi125.stdout:529: read oid 41 snap 47 2021-03-24T21:56:11.354 INFO:tasks.rados.rados.0.smithi125.stdout:529: expect (ObjNum 188 snap 41 seq_num 188) 2021-03-24T21:56:11.354 INFO:tasks.rados.rados.0.smithi125.stdout:530: snap_create 2021-03-24T21:56:11.355 INFO:tasks.rados.rados.0.smithi125.stdout:528: finishing rollback tid 1 to smithi12556470-10 2021-03-24T21:56:11.355 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 10 v 0 (ObjNum 1112276168 snap 21916 seq_num 1) dirty dne 2021-03-24T21:56:11.356 INFO:tasks.rados.rados.0.smithi125.stdout:527: finishing rollback tid 0 to smithi12556470-27 2021-03-24T21:56:11.356 INFO:tasks.rados.rados.0.smithi125.stdout:527: finishing rollback tid 1 to smithi12556470-27 2021-03-24T21:56:11.356 INFO:tasks.rados.rados.0.smithi125.stdout:527: finishing rollback tid 2 to smithi12556470-27 2021-03-24T21:56:11.356 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 27 v 205 (ObjNum 189 snap 41 seq_num 189) dirty exists 2021-03-24T21:56:11.363 INFO:tasks.rados.rados.0.smithi125.stdout:529: expect (ObjNum 188 snap 41 seq_num 188) 2021-03-24T21:56:11.429 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:11.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:11 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:56:11.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:11 smithi142 conmon[20125]: 56:10.105067+0000 osd.2 (osd.2) 40 : cluster [DBG] 7.e scrub ok 2021-03-24T21:56:11.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:11 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:56:11.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:11 smithi142 conmon[20125]: -03-24T21:56:10.347534+0000 osd.0 (osd.0) 56 : cluster [DBG] 2021-03-24T21:56:11.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:11 smithi142 conmon[20125]: purged_snaps scrub starts 2021-03-24T21:56:11.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:11 smithi142 conmon[20125]: cluster 2021-03-24T21:56 2021-03-24T21:56:11.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:11 smithi142 conmon[20125]: :10.347791+0000 osd.0 (osd.0) 57 2021-03-24T21:56:11.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:11 smithi142 conmon[20125]: : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:56:11.445 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:11 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:56:11.445 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:11 smithi142 conmon[20125]: T21:56:10.423855+0000 osd.3 (osd. 2021-03-24T21:56:11.445 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:11 smithi142 conmon[20125]: 3) 57 : cluster [DBG] 4.1d scrub ok 2021-03-24T21:56:11.445 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:11 smithi142 conmon[20125]: audit 2021 2021-03-24T21:56:11.445 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:11 smithi142 conmon[20125]: -03-24T21:56:10.644370+0000 2021-03-24T21:56:11.446 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:11 smithi142 conmon[20125]: mon.a (mon.0) 803 : audit [INF] from='client.? v1:172.21.15.125:0/4031396176' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T21:56:11.446 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:11 smithi142 conmon[20125]: 2021-03-24T21:56:11.446 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:11 smithi142 conmon[20125]: cluster 2021-03-24T21:56:11.446 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:11 smithi142 conmon[20125]: 2021-03-24T21:56:11.110461+0000 mon.a (mon. 2021-03-24T21:56:11.447 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:11 smithi142 conmon[20125]: 0) 804 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:56:11.447 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:11 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:56:11.447 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:11 smithi142 conmon[20125]: 03-24T21:56:11.115628+0000 2021-03-24T21:56:11.447 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:11 smithi142 conmon[20125]: mon.a (mon.0) 805 : audit [INF] from='client.? v1:172.21.15.125:0/4031396176' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T21:56:11.447 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:11 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:56:11.448 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:11 smithi142 conmon[20125]: -03-24T21:56:11.115767+0000 2021-03-24T21:56:11.448 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:11 smithi142 conmon[20125]: mon.a (mon.0) 806 : cluster [DBG] osdmap e202: 8 total, 8 up, 6 in 2021-03-24T21:56:11.479 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:11.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:11.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:56:11.585 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:11.595 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[19936]: cluster 2021-03-24T21:56:10. 2021-03-24T21:56:11.595 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[19936]: 105067+0000 osd.2 (osd.2) 40 : cluster [DBG] 7.e scrub ok 2021-03-24T21:56:11.596 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[19936]: cluster 2021-03-24T21:56:10.347534 2021-03-24T21:56:11.596 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[19936]: +0000 osd.0 (osd.0) 56 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:56:11.597 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:56:11.597 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[19936]: -24T21:56:10.347791 2021-03-24T21:56:11.597 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[19936]: +0000 osd.0 (osd.0) 57 2021-03-24T21:56:11.597 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[19936]: : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:56:11.598 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:56:11.598 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[19936]: 03-24T21:56:10.423855 2021-03-24T21:56:11.598 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[19936]: +0000 osd.3 (osd.3) 2021-03-24T21:56:11.598 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[19936]: 57 : cluster [DBG] 4.1d scrub ok 2021-03-24T21:56:11.599 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:56:11.599 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[19936]: 03-24T21:56:10. 2021-03-24T21:56:11.599 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[19936]: 644370+0000 mon.a (mon.0 2021-03-24T21:56:11.599 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[19936]: ) 803 : audit [INF] from='client.? v1:172.21.15.125:0/4031396176' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T21:56:11.599 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:56:11.600 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[19936]: 03-24T21:56:11.110461+0000 2021-03-24T21:56:11.600 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[19936]: mon.a (mon.0) 804 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:56:11.600 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[19936]: 2021-03-24T21:56:11.600 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[19936]: audit 2021-03-24T21:56:11.601 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[19936]: 2021-03-24T21:56:11.115628+0000 2021-03-24T21:56:11.601 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[19936]: mon.a (mon.0) 805 : audit 2021-03-24T21:56:11.601 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[19936]: [INF] from='client.? v1:172.21.15.125:0/4031396176' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T21:56:11.601 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:56:11.601 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[19936]: 21:56:11.115767+0000 mon.a (mon 2021-03-24T21:56:11.602 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[19936]: .0) 806 : cluster [DBG] osdmap e202: 8 total, 8 up, 6 in 2021-03-24T21:56:11.602 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: cluster 2021-03-24T21:56:11.603 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: 2021-03-24T21:56:11.603 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: 2021 2021-03-24T21:56:11.603 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: - 2021-03-24T21:56:11.603 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: 03 2021-03-24T21:56:11.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: - 2021-03-24T21:56:11.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: 24 2021-03-24T21:56:11.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: T 2021-03-24T21:56:11.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: 21 2021-03-24T21:56:11.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: : 2021-03-24T21:56:11.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: 56 2021-03-24T21:56:11.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: : 2021-03-24T21:56:11.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: 10 2021-03-24T21:56:11.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: . 2021-03-24T21:56:11.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: 105067 2021-03-24T21:56:11.606 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: +0000 2021-03-24T21:56:11.606 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: 2021-03-24T21:56:11.606 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: osd.2 2021-03-24T21:56:11.606 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: ( 2021-03-24T21:56:11.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: osd 2021-03-24T21:56:11.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: . 2021-03-24T21:56:11.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: 2 2021-03-24T21:56:11.608 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: ) 2021-03-24T21:56:11.608 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: 40 2021-03-24T21:56:11.608 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: : 2021-03-24T21:56:11.608 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: cluster 2021-03-24T21:56:11.608 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: 2021-03-24T21:56:11.609 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: [DBG] 2021-03-24T21:56:11.609 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: 2021-03-24T21:56:11.609 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: 7.e scrub ok 2021-03-24T21:56:11.609 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: 2021-03-24T21:56:11.610 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: cluster 2021-03-24T21:56:11.610 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: 2021-03-24T21:56:11.610 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: 2021 2021-03-24T21:56:11.610 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: - 2021-03-24T21:56:11.611 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: 03 2021-03-24T21:56:11.611 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: - 2021-03-24T21:56:11.611 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: 24 2021-03-24T21:56:11.611 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: T 2021-03-24T21:56:11.611 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: 21 2021-03-24T21:56:11.612 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: : 2021-03-24T21:56:11.612 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: 56 2021-03-24T21:56:11.612 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: : 2021-03-24T21:56:11.612 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: 10 2021-03-24T21:56:11.612 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: . 2021-03-24T21:56:11.613 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: 347534 2021-03-24T21:56:11.613 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: +0000 2021-03-24T21:56:11.613 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: 2021-03-24T21:56:11.613 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: osd.0 2021-03-24T21:56:11.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: ( 2021-03-24T21:56:11.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: osd 2021-03-24T21:56:11.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: . 2021-03-24T21:56:11.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: 0 2021-03-24T21:56:11.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: ) 2021-03-24T21:56:11.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: 56 2021-03-24T21:56:11.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: : 2021-03-24T21:56:11.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: cluster 2021-03-24T21:56:11.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: 2021-03-24T21:56:11.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: [DBG] 2021-03-24T21:56:11.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: 2021-03-24T21:56:11.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: purged_snaps scrub starts 2021-03-24T21:56:11.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: 2021-03-24T21:56:11.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: cluster 2021-03-24T21:56:11.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: 2021-03-24T21:56:11.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: 2021 2021-03-24T21:56:11.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: - 2021-03-24T21:56:11.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: 03 2021-03-24T21:56:11.618 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: - 2021-03-24T21:56:11.618 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: 24 2021-03-24T21:56:11.618 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: T21:56:10. 2021-03-24T21:56:11.618 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: 347791+0000 osd.0 (osd.0) 57 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:56:11.618 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:56:11.619 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: -03-24T21:56:10.423855+0000 osd.3 (osd. 2021-03-24T21:56:11.619 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: 3) 57 : cluster [DBG] 4.1d scrub ok 2021-03-24T21:56:11.619 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: audit 2021- 2021-03-24T21:56:11.619 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: 03-24T21:56:10.644370+0000 mon.a 2021-03-24T21:56:11.620 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: (mon.0) 803 : audit [INF] from='client.? v1:172.21.15.125:0/4031396176' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T21:56:11.620 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:56:11.620 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: -24T21:56:11.110461+0000 mon.a (mon.0) 804 : cluster 2021-03-24T21:56:11.620 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:56:11.620 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: audit 2021-03-24T21:56: 2021-03-24T21:56:11.621 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: 11.115628+0000 mon.a (mon.0) 805 : audit 2021-03-24T21:56:11.621 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: [INF] from='client.? v1:172.21.15.125:0/4031396176' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T21:56:11.621 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: cluster 2021-03-24T21:56:11. 2021-03-24T21:56:11.621 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: 115767+0000 mon.a (mon.0) 806 : cluster 2021-03-24T21:56:11.621 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:11 smithi125 conmon[24927]: [DBG] osdmap e202: 8 total, 8 up, 6 in 2021-03-24T21:56:11.634 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:11.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:11.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:56:11.738 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:11.791 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:11.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:11.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:56:11.896 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:11.954 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:11.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:11.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:56:12.059 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:12.095 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 7] out_osds: [6, 1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-03-24T21:56:12.095 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-24T21:56:12.095 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'reweight-by-utilization'} 2021-03-24T21:56:12.096 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd reweight-by-utilization 1000 0.05 2021-03-24T21:56:12.110 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:12.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:12.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:56:12.117 INFO:tasks.rados.rados.0.smithi125.stdout:527: done (3 left) 2021-03-24T21:56:12.117 INFO:tasks.rados.rados.0.smithi125.stdout:528: done (2 left) 2021-03-24T21:56:12.118 INFO:tasks.rados.rados.0.smithi125.stdout:529: done (1 left) 2021-03-24T21:56:12.118 INFO:tasks.rados.rados.0.smithi125.stdout:530: done (0 left) 2021-03-24T21:56:12.118 INFO:tasks.rados.rados.0.smithi125.stdout:531: read oid 42 snap -1 2021-03-24T21:56:12.119 INFO:tasks.rados.rados.0.smithi125.stdout:531: expect deleted 2021-03-24T21:56:12.119 INFO:tasks.rados.rados.0.smithi125.stdout:532: write oid 3 current snap is 49 2021-03-24T21:56:12.119 INFO:tasks.rados.rados.0.smithi125.stdout:532: seq_num 220 ranges {497540=612664,1840862=440060,3058823=464784} 2021-03-24T21:56:12.129 INFO:tasks.rados.rados.0.smithi125.stdout:532: writing smithi12556470-3 from 497540 to 1110204 tid 1 2021-03-24T21:56:12.136 INFO:tasks.rados.rados.0.smithi125.stdout:532: writing smithi12556470-3 from 1840862 to 2280922 tid 2 2021-03-24T21:56:12.145 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.7 dump_historic_ops 2021-03-24T21:56:12.146 INFO:tasks.rados.rados.0.smithi125.stdout:532: writing smithi12556470-3 from 3058823 to 3523607 tid 3 2021-03-24T21:56:12.146 INFO:tasks.rados.rados.0.smithi125.stdout:533: snap_create 2021-03-24T21:56:12.147 INFO:tasks.rados.rados.0.smithi125.stdout:532: finishing write tid 1 to smithi12556470-3 2021-03-24T21:56:12.147 INFO:tasks.rados.rados.0.smithi125.stdout:532: finishing write tid 2 to smithi12556470-3 2021-03-24T21:56:12.148 INFO:tasks.rados.rados.0.smithi125.stdout:532: finishing write tid 3 to smithi12556470-3 2021-03-24T21:56:12.148 INFO:tasks.rados.rados.0.smithi125.stdout:532: finishing write tid 4 to smithi12556470-3 2021-03-24T21:56:12.149 INFO:tasks.rados.rados.0.smithi125.stdout:532: finishing write tid 5 to smithi12556470-3 2021-03-24T21:56:12.150 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 3 v 92 (ObjNum 220 snap 49 seq_num 220) dirty exists 2021-03-24T21:56:12.150 INFO:tasks.rados.rados.0.smithi125.stdout:532: left oid 3 (ObjNum 220 snap 49 seq_num 220) 2021-03-24T21:56:12.215 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:12.259 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:12.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:12.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:56:12.365 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:12.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:12 smithi142 conmon[20125]: cluster 2021-03-24T21:56:10.743263 2021-03-24T21:56:12.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:12 smithi142 conmon[20125]: +0000 mgr.y (mgr.14144) 361 : cluster [DBG] pgmap v464: 158 pgs: 4 peering, 1 activating, 1 unknown, 152 active+clean; 120 MiB data, 389 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 2.4 MiB/s wr, 7 op/s 2021-03-24T21:56:12.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:12 smithi142 conmon[20125]: cluster 2021-03-24T21 2021-03-24T21:56:12.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:12 smithi142 conmon[20125]: :56:12.116155+0000 2021-03-24T21:56:12.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:12 smithi142 conmon[20125]: mon.a (mon.0) 2021-03-24T21:56:12.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:12 smithi142 conmon[20125]: 807 : cluster [DBG] osdmap e203: 8 total, 8 up, 6 in 2021-03-24T21:56:12.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:12 smithi142 conmon[20125]: 2021-03-24T21:56:12.419 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:12.422 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T21:56:12.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:12.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:56:12.527 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:12.539 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:12 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:56:12.539 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:12 smithi125 conmon[19936]: 56:10.743263+0000 mgr.y (mgr.14144) 361 : cluster [DBG] pgmap v464: 158 pgs: 4 peering, 1 activating, 1 unknown, 152 active+clean; 120 MiB data, 389 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 2.4 MiB/s wr, 7 op/s 2021-03-24T21:56:12.539 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:12 smithi125 conmon[19936]: cluster 2021-03-24T21:56 2021-03-24T21:56:12.540 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:12 smithi125 conmon[19936]: :12.116155+0000 mon.a (mon.0) 807 : cluster 2021-03-24T21:56:12.540 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:12 smithi125 conmon[19936]: [DBG] osdmap e203: 8 total, 8 up, 6 in 2021-03-24T21:56:12.540 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:12 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:56:12.541 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:12 smithi125 conmon[24927]: :56:10.743263+0000 mgr.y (mgr.14144) 361 : cluster [DBG] pgmap v464: 158 pgs: 4 peering, 1 activating, 1 unknown, 152 active+clean; 120 MiB data, 389 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 2.4 MiB/s wr, 7 op/s 2021-03-24T21:56:12.541 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:12 smithi125 conmon[24927]: cluster 2021-03-24T21:56:12.541 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:12 smithi125 conmon[24927]: 2021-03-24T21:56:12 2021-03-24T21:56:12.541 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:12 smithi125 conmon[24927]: .116155+0000 mon.a (mon.0) 807 : 2021-03-24T21:56:12.542 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:12 smithi125 conmon[24927]: cluster [DBG] osdmap e203: 8 total, 8 up, 6 in 2021-03-24T21:56:12.580 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:12.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:12.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:56:12.687 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:12.743 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:12.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:12.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:56:12.849 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:12.907 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:12.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:12.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:56:13.014 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:13.070 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:13.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:13.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:56:13.080 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:56:13.132 INFO:tasks.rados.rados.0.smithi125.stdout:531: done (2 left) 2021-03-24T21:56:13.133 INFO:tasks.rados.rados.0.smithi125.stdout:532: done (1 left) 2021-03-24T21:56:13.133 INFO:tasks.rados.rados.0.smithi125.stdout:533: done (0 left) 2021-03-24T21:56:13.133 INFO:tasks.rados.rados.0.smithi125.stdout:534: snap_remove snap 48 2021-03-24T21:56:13.176 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:13.233 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:13.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:13.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:56:13.339 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:13.395 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:13.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:13.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:56:13.502 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:13.548 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T21:56:13.557 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:13.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:13.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:56:13.570 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T21:56:13.595 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T21:56:13.618 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T21:56:13.643 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T21:56:13.662 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:13.667 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T21:56:13.699 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T21:56:13.721 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T21:56:13.723 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-24T21:56:13.723 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-24T21:56:13.723 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-24T21:56:13.723 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-24T21:56:13.724 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-24T21:56:13.724 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-24T21:56:13.724 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-24T21:56:13.724 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-24T21:56:13.725 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:13.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:13.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:56:13.829 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:13.884 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:13.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:13.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:56:13.988 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:14.044 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:14.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:14.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:56:14.132 INFO:tasks.rados.rados.0.smithi125.stdout:534: done (0 left) 2021-03-24T21:56:14.133 INFO:tasks.rados.rados.0.smithi125.stdout:535: write (excl) oid 28 current snap is 50 2021-03-24T21:56:14.133 INFO:tasks.rados.rados.0.smithi125.stdout:535: seq_num 221 ranges {514347=404266,1499344=700071,2686246=676056} 2021-03-24T21:56:14.139 INFO:tasks.rados.rados.0.smithi125.stdout:535: writing smithi12556470-28 from 514347 to 918613 tid 1 2021-03-24T21:56:14.147 INFO:tasks.rados.rados.0.smithi125.stdout:535: writing smithi12556470-28 from 1499344 to 2199415 tid 2 2021-03-24T21:56:14.150 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:14.154 INFO:tasks.rados.rados.0.smithi125.stdout:535: writing smithi12556470-28 from 2686246 to 3362302 tid 3 2021-03-24T21:56:14.154 INFO:tasks.rados.rados.0.smithi125.stdout:536: delete oid 25 current snap is 50 2021-03-24T21:56:14.156 INFO:tasks.rados.rados.0.smithi125.stdout:536: done (1 left) 2021-03-24T21:56:14.156 INFO:tasks.rados.rados.0.smithi125.stdout:537: snap_remove snap 49 2021-03-24T21:56:14.202 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:14.206 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:14 smithi142 conmon[20125]: cluster 2021-03-24T21:56:12.743756+0000 2021-03-24T21:56:14.206 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:14 smithi142 conmon[20125]: mgr.y (mgr.14144) 362 : cluster [DBG] pgmap v467: 158 pgs: 1 active+undersized+remapped, 4 peering, 1 activating, 1 unknown, 151 active+clean; 115 MiB data, 393 MiB used, 536 GiB / 536 GiB avail; 792 KiB/s, 2 objects/s recovering 2021-03-24T21:56:14.206 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:14 smithi142 conmon[20125]: cluster 2021-03-24T21:56:13.121153+0000 mon.a (mon.0) 808 : cluster [DBG] osdmap e204: 8 total, 8 up, 6 in 2021-03-24T21:56:14.206 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:14 smithi142 conmon[20125]: 2021-03-24T21:56:14.207 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:14 smithi142 conmon[20125]: audit 2021-03-24T 2021-03-24T21:56:14.207 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:14 smithi142 conmon[20125]: 21:56:13.511273+0000 mon.c (mon.2) 36 : 2021-03-24T21:56:14.207 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:14 smithi142 conmon[20125]: audit [DBG] from='client.? v1:172.21.15.125:0/2837032428' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:56:14.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:14.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:56:14.310 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:14.364 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:14.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:14.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:56:14.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:14 smithi125 conmon[19936]: cluster 2021-03-24T21:56:12.743756+0000 mgr.y ( 2021-03-24T21:56:14.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:14 smithi125 conmon[19936]: mgr.14144) 362 : cluster [DBG] pgmap v467: 158 pgs: 1 active+undersized+remapped, 4 peering, 1 activating, 1 unknown, 151 active+clean; 115 MiB data, 393 MiB used, 536 GiB / 536 GiB avail; 792 KiB/s, 2 objects/s recovering 2021-03-24T21:56:14.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:14 smithi125 conmon[19936]: cluster 2021-03-24T21:56:13.121153+0000 mon.a (mon.0) 808 : cluster [DBG] osdmap e204: 8 total, 8 up, 6 in 2021-03-24T21:56:14.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:14 smithi125 conmon[19936]: audit 2021-03-24T21:56:14.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:14 smithi125 conmon[19936]: 2021-03-24T21:56:13.511273+0000 mon.c (mon.2) 36 : audit [DBG] from='client.? v1:172.21.15.125:0/2837032428' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:56:14.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:14 smithi125 conmon[24927]: cluster 2021-03-24T21:56:14.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:14 smithi125 conmon[24927]: 2021-03-24T21: 2021-03-24T21:56:14.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:14 smithi125 conmon[24927]: 56:12.743756+0000 mgr.y (mgr.14144) 362 : cluster [DBG] pgmap v467: 158 pgs: 1 active+undersized+remapped, 4 peering, 1 activating, 1 unknown, 151 active+clean; 115 MiB data, 393 MiB used, 536 GiB / 536 GiB avail; 792 KiB/s, 2 objects/s recovering 2021-03-24T21:56:14.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:14 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:56:14.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:14 smithi125 conmon[24927]: 21:56:13.121153+0000 mon.a (mon.0) 808 : cluster [DBG] osdmap e204: 8 total, 8 up, 6 in 2021-03-24T21:56:14.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:14 smithi125 conmon[24927]: audit 2021-03-24T21:56:14.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:14 smithi125 conmon[24927]: 2021-03-24T21:56:14.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:14 smithi125 conmon[24927]: 2021-03-24T21:56:13 2021-03-24T21:56:14.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:14 smithi125 conmon[24927]: .511273+0000 mon.c (mon.2) 36 : audit [DBG] from='client.? v1:172.21.15.125:0/2837032428' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:56:14.468 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:14.496 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2021-03-24T21:56:14.496 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2021-03-24T21:56:14.497 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2021-03-24T21:56:14.497 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2021-03-24T21:56:14.497 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.497 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.34352.0:3 3.13 3:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected e61)", 2021-03-24T21:56:14.497 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:51:45.387966+0000", 2021-03-24T21:56:14.498 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 269.10536715900002, 2021-03-24T21:56:14.498 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.19192994199999999, 2021-03-24T21:56:14.498 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:56:14.498 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "reached pg", 2021-03-24T21:56:14.499 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:56:14.499 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.34352", 2021-03-24T21:56:14.499 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/3436348424", 2021-03-24T21:56:14.499 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 3 2021-03-24T21:56:14.499 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.500 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:56:14.500 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.500 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:56:14.500 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:45.387966+0000", 2021-03-24T21:56:14.501 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:56:14.501 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.501 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.501 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:56:14.502 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:45.387966+0000", 2021-03-24T21:56:14.502 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2021-03-24T21:56:14.502 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.502 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.502 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:56:14.503 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:45.387964+0000", 2021-03-24T21:56:14.503 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.7370000000000006e-06 2021-03-24T21:56:14.503 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.503 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.504 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:56:14.504 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:45.387972+0000", 2021-03-24T21:56:14.504 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5412000000000001e-05 2021-03-24T21:56:14.504 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.504 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.505 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:56:14.505 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:45.387987+0000", 2021-03-24T21:56:14.505 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.216e-06 2021-03-24T21:56:14.505 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.505 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.506 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:56:14.506 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:45.387994+0000", 2021-03-24T21:56:14.506 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0209000000000004e-05 2021-03-24T21:56:14.506 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.507 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.507 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:56:14.507 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:45.388074+0000", 2021-03-24T21:56:14.507 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2846e-05 2021-03-24T21:56:14.507 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.508 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.508 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for peered", 2021-03-24T21:56:14.508 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:45.388087+0000", 2021-03-24T21:56:14.508 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.19160065900000001 2021-03-24T21:56:14.508 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.509 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.509 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:56:14.509 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:45.579688+0000", 2021-03-24T21:56:14.509 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00020767600000000001 2021-03-24T21:56:14.509 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.510 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.510 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:56:14.510 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:45.579896+0000", 2021-03-24T21:56:14.510 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.19192994199999999 2021-03-24T21:56:14.511 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:14.511 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:56:14.511 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:14.511 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.511 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.512 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.24388.0:166 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected e107)", 2021-03-24T21:56:14.512 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:54:37.427176+0000", 2021-03-24T21:56:14.512 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 97.066156867999993, 2021-03-24T21:56:14.512 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.699168805, 2021-03-24T21:56:14.512 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:56:14.513 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T21:56:14.513 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:56:14.513 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.24388", 2021-03-24T21:56:14.513 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.142:0/626803041", 2021-03-24T21:56:14.513 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 166 2021-03-24T21:56:14.514 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.514 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:56:14.514 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.514 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:56:14.515 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:37.427176+0000", 2021-03-24T21:56:14.515 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:56:14.515 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.515 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.515 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:56:14.516 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:37.427176+0000", 2021-03-24T21:56:14.516 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999957 2021-03-24T21:56:14.516 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.516 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.517 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:56:14.517 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:37.427172+0000", 2021-03-24T21:56:14.517 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8757e-05 2021-03-24T21:56:14.517 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.517 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.518 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:56:14.518 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:37.427190+0000", 2021-03-24T21:56:14.518 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3581999999999998e-05 2021-03-24T21:56:14.518 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.518 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.519 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:56:14.519 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:37.427224+0000", 2021-03-24T21:56:14.519 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5787e-05 2021-03-24T21:56:14.519 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.519 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.520 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:56:14.520 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:37.427240+0000", 2021-03-24T21:56:14.520 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00020074000000000001 2021-03-24T21:56:14.520 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.521 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.521 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:56:14.521 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:37.427440+0000", 2021-03-24T21:56:14.521 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8619000000000002e-05 2021-03-24T21:56:14.521 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.522 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.522 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for peered", 2021-03-24T21:56:14.522 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:37.427469+0000", 2021-03-24T21:56:14.522 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.69859708399999998 2021-03-24T21:56:14.522 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.523 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.523 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:56:14.523 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:38.126066+0000", 2021-03-24T21:56:14.523 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133013 2021-03-24T21:56:14.523 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.524 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.524 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:56:14.524 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:38.126199+0000", 2021-03-24T21:56:14.524 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014562000000000001 2021-03-24T21:56:14.525 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.525 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.525 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:56:14.525 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:38.126345+0000", 2021-03-24T21:56:14.525 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.699168805 2021-03-24T21:56:14.526 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:14.526 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:56:14.526 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:14.526 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.526 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.527 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:250 7.5 7:ad88ec1b:::smithi12556470-47:head [write 557999~703664] snapc 5=[] ondisk+write+known_if_redirected e113)", 2021-03-24T21:56:14.527 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:54:42.157588+0000", 2021-03-24T21:56:14.527 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 92.335745032999995, 2021-03-24T21:56:14.527 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.016509769000000001, 2021-03-24T21:56:14.528 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:56:14.528 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:14.528 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:56:14.528 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T21:56:14.528 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:56:14.529 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 250 2021-03-24T21:56:14.529 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.529 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:56:14.529 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.529 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:56:14.530 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.157588+0000", 2021-03-24T21:56:14.530 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:56:14.530 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.530 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.530 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:56:14.531 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.157588+0000", 2021-03-24T21:56:14.531 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999971 2021-03-24T21:56:14.531 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.531 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.532 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:56:14.532 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.157585+0000", 2021-03-24T21:56:14.532 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00088941400000000005 2021-03-24T21:56:14.532 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.532 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.533 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:56:14.533 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.158474+0000", 2021-03-24T21:56:14.533 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3985000000000002e-05 2021-03-24T21:56:14.533 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.533 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.534 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:56:14.534 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.158498+0000", 2021-03-24T21:56:14.534 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2496e-05 2021-03-24T21:56:14.534 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.534 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.535 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:56:14.535 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.158511+0000", 2021-03-24T21:56:14.535 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010052000000000001 2021-03-24T21:56:14.535 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.536 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.536 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:56:14.536 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.158611+0000", 2021-03-24T21:56:14.536 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4144e-05 2021-03-24T21:56:14.536 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.537 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.537 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op must wait for map", 2021-03-24T21:56:14.537 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.158626+0000", 2021-03-24T21:56:14.537 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.003877751 2021-03-24T21:56:14.537 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.538 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.538 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:56:14.538 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.162503+0000", 2021-03-24T21:56:14.538 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012955200000000001 2021-03-24T21:56:14.539 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.539 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.539 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:56:14.539 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.162633+0000", 2021-03-24T21:56:14.539 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016109300000000001 2021-03-24T21:56:14.540 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.540 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.540 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 0,2", 2021-03-24T21:56:14.540 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.162794+0000", 2021-03-24T21:56:14.540 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0028181040000000001 2021-03-24T21:56:14.541 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.541 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.541 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T21:56:14.541 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.165612+0000", 2021-03-24T21:56:14.541 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001620907 2021-03-24T21:56:14.542 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.542 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.542 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:14.542 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.167233+0000", 2021-03-24T21:56:14.543 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0066663240000000004 2021-03-24T21:56:14.543 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.543 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.543 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:14.543 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.173899+0000", 2021-03-24T21:56:14.544 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000103859 2021-03-24T21:56:14.544 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.544 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.544 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T21:56:14.544 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.174003+0000", 2021-03-24T21:56:14.545 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4363000000000004e-05 2021-03-24T21:56:14.545 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.545 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.545 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:56:14.545 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.174098+0000", 2021-03-24T21:56:14.546 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.016509769000000001 2021-03-24T21:56:14.546 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:14.546 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:56:14.546 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:14.547 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.547 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.547 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:251 7.5 7:ad88ec1b:::smithi12556470-47:head [write 1719003~413684] snapc 5=[] ondisk+write+known_if_redirected e113)", 2021-03-24T21:56:14.547 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:54:42.163381+0000", 2021-03-24T21:56:14.547 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 92.329951476000005, 2021-03-24T21:56:14.548 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011000496, 2021-03-24T21:56:14.548 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:56:14.548 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:14.548 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:56:14.549 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T21:56:14.549 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:56:14.549 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 251 2021-03-24T21:56:14.549 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.549 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:56:14.550 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.550 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:56:14.550 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.163381+0000", 2021-03-24T21:56:14.550 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:56:14.550 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.551 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.551 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:56:14.551 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.163381+0000", 2021-03-24T21:56:14.551 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999971 2021-03-24T21:56:14.552 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.552 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.552 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:56:14.552 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.163378+0000", 2021-03-24T21:56:14.552 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00054152699999999998 2021-03-24T21:56:14.553 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.553 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.553 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:56:14.553 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.163920+0000", 2021-03-24T21:56:14.553 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9369999999999999e-05 2021-03-24T21:56:14.554 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.554 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.554 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:56:14.554 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.163939+0000", 2021-03-24T21:56:14.555 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1433999999999999e-05 2021-03-24T21:56:14.555 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.555 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.555 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:56:14.555 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.163951+0000", 2021-03-24T21:56:14.556 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00063210399999999998 2021-03-24T21:56:14.556 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.556 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.556 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:56:14.556 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.164583+0000", 2021-03-24T21:56:14.557 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000162323 2021-03-24T21:56:14.557 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.557 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.557 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:56:14.557 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.164745+0000", 2021-03-24T21:56:14.558 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00025836400000000002 2021-03-24T21:56:14.558 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.558 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.558 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 0,2", 2021-03-24T21:56:14.559 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.165004+0000", 2021-03-24T21:56:14.559 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001000007 2021-03-24T21:56:14.559 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.559 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.559 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T21:56:14.560 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.166004+0000", 2021-03-24T21:56:14.560 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0026407510000000002 2021-03-24T21:56:14.560 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.560 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.560 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:14.561 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.168644+0000", 2021-03-24T21:56:14.561 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0055347069999999998 2021-03-24T21:56:14.561 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.561 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.562 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:14.562 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.174179+0000", 2021-03-24T21:56:14.562 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.8900000000000005e-05 2021-03-24T21:56:14.562 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.563 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.563 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T21:56:14.563 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.174278+0000", 2021-03-24T21:56:14.563 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000103936 2021-03-24T21:56:14.563 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.564 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.564 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:56:14.564 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.174382+0000", 2021-03-24T21:56:14.564 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011000496 2021-03-24T21:56:14.564 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:14.565 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:56:14.565 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:14.565 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.565 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.565 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.14760.0:260 7.d e114/108 7:bdda3e67:::smithi12556470-22:head v 114'17, mlcod=0'0)", 2021-03-24T21:56:14.566 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:54:43.181710+0000", 2021-03-24T21:56:14.566 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 91.311622306000004, 2021-03-24T21:56:14.566 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0087326069999999999, 2021-03-24T21:56:14.566 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:56:14.567 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:14.567 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:56:14.567 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.567 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:56:14.567 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:43.181710+0000", 2021-03-24T21:56:14.568 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:56:14.568 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.568 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.568 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:56:14.569 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:43.181710+0000", 2021-03-24T21:56:14.569 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2021-03-24T21:56:14.569 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.569 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.569 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:56:14.570 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:43.181709+0000", 2021-03-24T21:56:14.570 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.006109532 2021-03-24T21:56:14.570 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.570 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.570 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:56:14.571 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:43.187819+0000", 2021-03-24T21:56:14.571 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7589000000000002e-05 2021-03-24T21:56:14.571 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.571 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.571 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:56:14.572 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:43.187856+0000", 2021-03-24T21:56:14.572 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1016e-05 2021-03-24T21:56:14.572 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.572 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.572 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:56:14.573 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:43.187877+0000", 2021-03-24T21:56:14.573 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.7049999999999998e-05 2021-03-24T21:56:14.573 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.573 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.574 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:56:14.574 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:43.187924+0000", 2021-03-24T21:56:14.574 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0858e-05 2021-03-24T21:56:14.574 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.574 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.575 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:56:14.575 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:43.187975+0000", 2021-03-24T21:56:14.575 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0023883509999999999 2021-03-24T21:56:14.575 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.575 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.576 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T21:56:14.576 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:43.190363+0000", 2021-03-24T21:56:14.576 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9660999999999995e-05 2021-03-24T21:56:14.576 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.576 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.577 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:56:14.577 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:43.190443+0000", 2021-03-24T21:56:14.577 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0087326069999999999 2021-03-24T21:56:14.577 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:14.577 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:56:14.578 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:14.578 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.578 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.578 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.14760.0:277 7.8 e114/99 7:19ff009c:::smithi12556470-42:head v 114'14, mlcod=110'10)", 2021-03-24T21:56:14.579 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:54:43.207797+0000", 2021-03-24T21:56:14.579 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 91.285535378999995, 2021-03-24T21:56:14.579 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0093171, 2021-03-24T21:56:14.579 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:56:14.579 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:14.580 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:56:14.580 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.580 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:56:14.580 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:43.207797+0000", 2021-03-24T21:56:14.580 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:56:14.581 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.581 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.581 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:56:14.581 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:43.207797+0000", 2021-03-24T21:56:14.582 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2021-03-24T21:56:14.582 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.582 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.582 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:56:14.582 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:43.207797+0000", 2021-03-24T21:56:14.583 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0045723639999999998 2021-03-24T21:56:14.583 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.583 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.583 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:56:14.583 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:43.212369+0000", 2021-03-24T21:56:14.584 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0215000000000001e-05 2021-03-24T21:56:14.584 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.584 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.584 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:56:14.585 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:43.212399+0000", 2021-03-24T21:56:14.585 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1959999999999997e-06 2021-03-24T21:56:14.585 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.585 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.585 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:56:14.586 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:43.212409+0000", 2021-03-24T21:56:14.586 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.8009000000000003e-05 2021-03-24T21:56:14.586 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.586 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.587 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:56:14.587 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:43.212507+0000", 2021-03-24T21:56:14.587 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7835999999999999e-05 2021-03-24T21:56:14.587 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.587 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.588 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:56:14.588 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:43.212534+0000", 2021-03-24T21:56:14.588 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0045320589999999997 2021-03-24T21:56:14.588 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.589 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.589 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T21:56:14.589 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:43.217066+0000", 2021-03-24T21:56:14.589 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8087999999999998e-05 2021-03-24T21:56:14.589 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.590 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.590 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:56:14.590 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:43.217115+0000", 2021-03-24T21:56:14.590 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0093171 2021-03-24T21:56:14.590 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:14.591 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:56:14.591 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:14.591 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.591 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.592 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:431 7.5 7:ad88ec1b:::smithi12556470-47:head [sparse-read 0~2179573,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e123)", 2021-03-24T21:56:14.592 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:54:52.102779+0000", 2021-03-24T21:56:14.592 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 82.390553596000004, 2021-03-24T21:56:14.592 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0091124320000000002, 2021-03-24T21:56:14.592 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:56:14.593 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T21:56:14.593 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:56:14.593 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T21:56:14.593 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:56:14.593 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 431 2021-03-24T21:56:14.594 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.594 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:56:14.594 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.594 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:56:14.595 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:52.102779+0000", 2021-03-24T21:56:14.595 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:56:14.595 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.595 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.595 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:56:14.596 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:52.102779+0000", 2021-03-24T21:56:14.596 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999948 2021-03-24T21:56:14.596 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.596 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.596 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:56:14.597 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:52.102774+0000", 2021-03-24T21:56:14.597 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2546000000000001e-05 2021-03-24T21:56:14.597 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.597 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.598 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:56:14.598 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:52.102796+0000", 2021-03-24T21:56:14.598 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0076000000000001e-05 2021-03-24T21:56:14.598 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.598 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.599 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:56:14.599 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:52.102836+0000", 2021-03-24T21:56:14.599 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6303e-05 2021-03-24T21:56:14.599 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.599 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.600 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:56:14.600 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:52.102853+0000", 2021-03-24T21:56:14.600 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000148763 2021-03-24T21:56:14.600 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.601 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.601 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:56:14.601 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:52.103001+0000", 2021-03-24T21:56:14.601 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4883e-05 2021-03-24T21:56:14.601 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.602 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.602 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op must wait for map", 2021-03-24T21:56:14.602 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:52.103016+0000", 2021-03-24T21:56:14.602 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0060112660000000004 2021-03-24T21:56:14.603 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.603 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.603 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:56:14.603 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:52.109028+0000", 2021-03-24T21:56:14.603 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015945900000000001 2021-03-24T21:56:14.604 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.604 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.604 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:56:14.604 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:52.109187+0000", 2021-03-24T21:56:14.604 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0027045160000000001 2021-03-24T21:56:14.605 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.605 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.605 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:56:14.605 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:52.111892+0000", 2021-03-24T21:56:14.605 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0091124320000000002 2021-03-24T21:56:14.606 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:14.606 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:56:14.606 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:14.606 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.607 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.607 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:432 7.5 7:ad88ec1b:::smithi12556470-47:head [sparse-read 0~2179573] snapc 0=[] ondisk+read+known_if_redirected e123)", 2021-03-24T21:56:14.607 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:54:52.102879+0000", 2021-03-24T21:56:14.607 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 82.390453999000002, 2021-03-24T21:56:14.607 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0095192839999999994, 2021-03-24T21:56:14.607 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:56:14.608 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T21:56:14.608 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:56:14.608 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T21:56:14.608 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:56:14.608 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 432 2021-03-24T21:56:14.609 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.609 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:56:14.609 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.609 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:56:14.610 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:52.102879+0000", 2021-03-24T21:56:14.610 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:56:14.610 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.610 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.610 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:56:14.611 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:52.102879+0000", 2021-03-24T21:56:14.611 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2021-03-24T21:56:14.611 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.611 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.611 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:56:14.612 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:52.102877+0000", 2021-03-24T21:56:14.612 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0169999999999996e-06 2021-03-24T21:56:14.612 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.612 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.612 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:56:14.613 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:52.102885+0000", 2021-03-24T21:56:14.613 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5517000000000001e-05 2021-03-24T21:56:14.613 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.613 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.614 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:56:14.614 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:52.102901+0000", 2021-03-24T21:56:14.614 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2052000000000002e-05 2021-03-24T21:56:14.614 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.614 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.615 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:56:14.615 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:52.102923+0000", 2021-03-24T21:56:14.615 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00023543199999999999 2021-03-24T21:56:14.615 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.616 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.616 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:56:14.616 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:52.103158+0000", 2021-03-24T21:56:14.616 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.507e-06 2021-03-24T21:56:14.616 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.617 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.617 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting_for_map not empty", 2021-03-24T21:56:14.617 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:52.103167+0000", 2021-03-24T21:56:14.617 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0087953590000000009 2021-03-24T21:56:14.618 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.618 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.618 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:56:14.618 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:52.111962+0000", 2021-03-24T21:56:14.618 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016885899999999999 2021-03-24T21:56:14.619 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.619 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.619 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:56:14.619 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:52.112131+0000", 2021-03-24T21:56:14.620 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00026713099999999999 2021-03-24T21:56:14.620 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.620 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.620 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:56:14.621 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:52.112398+0000", 2021-03-24T21:56:14.621 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0095192839999999994 2021-03-24T21:56:14.621 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:14.621 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:56:14.621 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:14.622 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.622 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.622 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:433 7.5 7:ad88ec1b:::smithi12556470-47:head [sparse-read 0~2179573] snapc 0=[] ondisk+read+known_if_redirected e123)", 2021-03-24T21:56:14.622 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:54:52.102966+0000", 2021-03-24T21:56:14.623 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 82.390367003999998, 2021-03-24T21:56:14.623 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0098858420000000006, 2021-03-24T21:56:14.623 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:56:14.623 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T21:56:14.623 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:56:14.624 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T21:56:14.624 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:56:14.624 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 433 2021-03-24T21:56:14.624 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.624 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:56:14.625 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.625 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:56:14.625 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:52.102966+0000", 2021-03-24T21:56:14.625 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:56:14.625 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.626 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.626 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:56:14.626 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:52.102966+0000", 2021-03-24T21:56:14.626 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2021-03-24T21:56:14.627 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.627 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.627 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:56:14.627 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:52.102965+0000", 2021-03-24T21:56:14.627 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5469999999999996e-06 2021-03-24T21:56:14.628 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.628 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.628 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:56:14.628 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:52.102970+0000", 2021-03-24T21:56:14.628 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7566e-05 2021-03-24T21:56:14.629 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.629 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.629 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:56:14.629 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:52.102998+0000", 2021-03-24T21:56:14.629 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0003000000000001e-05 2021-03-24T21:56:14.630 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.630 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.630 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:56:14.630 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:52.103008+0000", 2021-03-24T21:56:14.631 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000197438 2021-03-24T21:56:14.631 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.631 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.631 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:56:14.631 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:52.103205+0000", 2021-03-24T21:56:14.632 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6560000000000003e-06 2021-03-24T21:56:14.632 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.632 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.632 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting_for_map not empty", 2021-03-24T21:56:14.632 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:52.103212+0000", 2021-03-24T21:56:14.633 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0092501110000000001 2021-03-24T21:56:14.633 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.633 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.633 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:56:14.633 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:52.112462+0000", 2021-03-24T21:56:14.634 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015109300000000001 2021-03-24T21:56:14.634 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.634 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.634 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:56:14.635 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:52.112613+0000", 2021-03-24T21:56:14.635 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00023856800000000001 2021-03-24T21:56:14.635 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.635 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.635 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:56:14.636 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:52.112852+0000", 2021-03-24T21:56:14.636 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0098858420000000006 2021-03-24T21:56:14.636 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:14.636 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:56:14.636 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:14.637 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.637 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.637 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:527 7.5 7:ad88ec1b:::smithi12556470-47:head [copy-from ver 24] snapc 17=[17,16,15] ondisk+write+known_if_redirected e130)", 2021-03-24T21:56:14.637 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:54:59.103361+0000", 2021-03-24T21:56:14.637 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 75.389971328000001, 2021-03-24T21:56:14.638 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.019497067, 2021-03-24T21:56:14.638 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:56:14.638 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:14.638 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:56:14.639 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T21:56:14.639 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:56:14.639 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 527 2021-03-24T21:56:14.639 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.639 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:56:14.640 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.640 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:56:14.640 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:59.103361+0000", 2021-03-24T21:56:14.640 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:56:14.640 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.641 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.641 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:56:14.641 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:59.103361+0000", 2021-03-24T21:56:14.641 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2021-03-24T21:56:14.641 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.642 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.642 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:56:14.642 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:59.103359+0000", 2021-03-24T21:56:14.642 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1449999999999997e-06 2021-03-24T21:56:14.643 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.643 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.643 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:56:14.643 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:59.103368+0000", 2021-03-24T21:56:14.643 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4506e-05 2021-03-24T21:56:14.644 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.644 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.644 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:56:14.644 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:59.103383+0000", 2021-03-24T21:56:14.645 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2899999999999999e-06 2021-03-24T21:56:14.645 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.645 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.645 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:56:14.645 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:59.103389+0000", 2021-03-24T21:56:14.646 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8383000000000001e-05 2021-03-24T21:56:14.646 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.646 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.646 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:56:14.647 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:59.103438+0000", 2021-03-24T21:56:14.647 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.2369999999999999e-05 2021-03-24T21:56:14.647 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.647 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.648 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:56:14.648 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:59.103520+0000", 2021-03-24T21:56:14.648 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0056449910000000002 2021-03-24T21:56:14.648 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.648 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.649 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 0,2", 2021-03-24T21:56:14.649 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:59.109165+0000", 2021-03-24T21:56:14.649 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0057482790000000002 2021-03-24T21:56:14.649 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.650 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.650 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T21:56:14.650 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:59.114913+0000", 2021-03-24T21:56:14.650 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00113528 2021-03-24T21:56:14.650 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.651 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.651 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:14.651 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:59.116048+0000", 2021-03-24T21:56:14.651 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0066518710000000002 2021-03-24T21:56:14.651 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.652 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.652 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:14.652 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:59.122700+0000", 2021-03-24T21:56:14.652 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.7063999999999994e-05 2021-03-24T21:56:14.653 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.653 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.653 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T21:56:14.653 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:59.122767+0000", 2021-03-24T21:56:14.653 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1157999999999997e-05 2021-03-24T21:56:14.654 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.654 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.654 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:56:14.654 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:59.122859+0000", 2021-03-24T21:56:14.654 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.019497067 2021-03-24T21:56:14.655 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:14.655 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:56:14.655 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:14.655 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.655 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.656 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:528 7.5 7:ad88ec1b:::smithi12556470-47:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e130)", 2021-03-24T21:56:14.656 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:54:59.103402+0000", 2021-03-24T21:56:14.656 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 75.389930303, 2021-03-24T21:56:14.656 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.019629818, 2021-03-24T21:56:14.656 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:56:14.657 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T21:56:14.657 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:56:14.657 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T21:56:14.657 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:56:14.658 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 528 2021-03-24T21:56:14.658 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.658 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:56:14.658 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.658 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:56:14.659 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:59.103402+0000", 2021-03-24T21:56:14.659 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:56:14.659 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.659 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.659 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:56:14.660 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:59.103402+0000", 2021-03-24T21:56:14.660 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2021-03-24T21:56:14.660 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.660 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.660 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:56:14.661 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:59.103402+0000", 2021-03-24T21:56:14.661 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.732e-06 2021-03-24T21:56:14.661 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.661 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.662 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:56:14.662 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:59.103405+0000", 2021-03-24T21:56:14.662 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2660000000000004e-06 2021-03-24T21:56:14.662 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.662 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.663 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:56:14.663 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:59.103413+0000", 2021-03-24T21:56:14.663 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4159999999999997e-06 2021-03-24T21:56:14.663 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.663 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.664 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:56:14.664 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:59.103417+0000", 2021-03-24T21:56:14.664 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000231597 2021-03-24T21:56:14.664 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.665 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.665 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:56:14.665 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:59.103649+0000", 2021-03-24T21:56:14.665 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.2876e-05 2021-03-24T21:56:14.665 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.665 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.666 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for blocked object", 2021-03-24T21:56:14.666 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:59.103702+0000", 2021-03-24T21:56:14.666 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.010562622000000001 2021-03-24T21:56:14.666 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.667 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.667 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:56:14.667 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:59.114264+0000", 2021-03-24T21:56:14.667 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011914899999999999 2021-03-24T21:56:14.667 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.668 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.668 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for rw locks", 2021-03-24T21:56:14.668 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:59.114383+0000", 2021-03-24T21:56:14.668 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0085108110000000001 2021-03-24T21:56:14.668 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.669 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.669 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:56:14.669 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:59.122894+0000", 2021-03-24T21:56:14.669 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3158999999999996e-05 2021-03-24T21:56:14.669 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.670 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.670 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:56:14.670 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:59.122957+0000", 2021-03-24T21:56:14.670 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.4938999999999995e-05 2021-03-24T21:56:14.671 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.671 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.671 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:56:14.671 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:59.123032+0000", 2021-03-24T21:56:14.671 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.019629818 2021-03-24T21:56:14.672 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:14.672 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:56:14.672 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:14.672 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.673 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.673 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.14760.0:634 7.f e140/99 7:f26cf53d:::smithi12556470-37:head v 140'8, mlcod=107'5)", 2021-03-24T21:56:14.673 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:55:09.384439+0000", 2021-03-24T21:56:14.673 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 65.108894038000003, 2021-03-24T21:56:14.673 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.009244848, 2021-03-24T21:56:14.674 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:56:14.674 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:14.674 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:56:14.674 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.674 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:56:14.675 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:09.384439+0000", 2021-03-24T21:56:14.675 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:56:14.675 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.675 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.675 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:56:14.676 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:09.384439+0000", 2021-03-24T21:56:14.676 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999962 2021-03-24T21:56:14.676 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.676 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.677 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:56:14.677 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:09.384435+0000", 2021-03-24T21:56:14.677 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0047200549999999999 2021-03-24T21:56:14.677 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.677 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.678 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:56:14.678 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:09.389155+0000", 2021-03-24T21:56:14.678 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9009e-05 2021-03-24T21:56:14.678 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.679 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.679 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:56:14.679 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:09.389184+0000", 2021-03-24T21:56:14.679 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2380000000000003e-06 2021-03-24T21:56:14.679 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.680 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.680 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:56:14.680 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:09.389193+0000", 2021-03-24T21:56:14.680 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1938e-05 2021-03-24T21:56:14.680 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.681 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.681 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:56:14.681 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:09.389285+0000", 2021-03-24T21:56:14.681 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3096000000000003e-05 2021-03-24T21:56:14.682 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.682 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.682 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:56:14.682 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:09.389318+0000", 2021-03-24T21:56:14.682 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00431315 2021-03-24T21:56:14.683 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.683 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.683 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T21:56:14.683 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:09.393631+0000", 2021-03-24T21:56:14.683 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.2392000000000003e-05 2021-03-24T21:56:14.684 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.684 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.684 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:56:14.684 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:09.393684+0000", 2021-03-24T21:56:14.685 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.009244848 2021-03-24T21:56:14.685 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:14.685 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:56:14.685 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:14.685 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.686 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.686 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.14760.0:639 7.7 e141/108 7:ef6d6bee:::smithi12556470-24:head v 141'70, mlcod=141'68)", 2021-03-24T21:56:14.686 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:55:10.122599+0000", 2021-03-24T21:56:14.686 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 64.370733955999995, 2021-03-24T21:56:14.687 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.01265399, 2021-03-24T21:56:14.687 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:56:14.687 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:14.687 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:56:14.687 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.688 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:56:14.688 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:10.122599+0000", 2021-03-24T21:56:14.688 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:56:14.688 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.688 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.689 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:56:14.689 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:10.122599+0000", 2021-03-24T21:56:14.689 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2021-03-24T21:56:14.689 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.689 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.690 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:56:14.690 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:10.122597+0000", 2021-03-24T21:56:14.690 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00064497899999999995 2021-03-24T21:56:14.690 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.691 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.691 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:56:14.691 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:10.123242+0000", 2021-03-24T21:56:14.691 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8834e-05 2021-03-24T21:56:14.692 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.692 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.692 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:56:14.692 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:10.123261+0000", 2021-03-24T21:56:14.693 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3110000000000002e-06 2021-03-24T21:56:14.693 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.693 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.693 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:56:14.693 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:10.123267+0000", 2021-03-24T21:56:14.694 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9983999999999999e-05 2021-03-24T21:56:14.694 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.694 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.694 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:56:14.695 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:10.123337+0000", 2021-03-24T21:56:14.695 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3308000000000001e-05 2021-03-24T21:56:14.695 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.695 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.695 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:56:14.696 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:10.123360+0000", 2021-03-24T21:56:14.696 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011832109 2021-03-24T21:56:14.696 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.696 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.696 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T21:56:14.697 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:10.135192+0000", 2021-03-24T21:56:14.697 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0582999999999999e-05 2021-03-24T21:56:14.697 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.697 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.698 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:56:14.698 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:10.135253+0000", 2021-03-24T21:56:14.698 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.01265399 2021-03-24T21:56:14.698 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:14.698 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:56:14.699 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:14.699 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.699 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.699 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.14760.0:645 7.c e141/101 7:305739dc:::smithi12556470-36:head v 141'47, mlcod=134'43)", 2021-03-24T21:56:14.700 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:55:10.122852+0000", 2021-03-24T21:56:14.700 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 64.370480442000002, 2021-03-24T21:56:14.700 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.012177426999999999, 2021-03-24T21:56:14.700 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:56:14.700 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:14.701 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:56:14.701 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.701 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:56:14.701 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:10.122852+0000", 2021-03-24T21:56:14.701 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:56:14.702 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.702 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.702 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:56:14.702 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:10.122852+0000", 2021-03-24T21:56:14.703 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2021-03-24T21:56:14.703 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.703 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.703 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:56:14.703 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:10.122852+0000", 2021-03-24T21:56:14.704 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4273e-05 2021-03-24T21:56:14.704 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.704 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.704 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:56:14.705 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:10.122866+0000", 2021-03-24T21:56:14.705 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5447999999999998e-05 2021-03-24T21:56:14.705 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.705 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.705 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:56:14.706 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:10.122881+0000", 2021-03-24T21:56:14.706 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1198e-05 2021-03-24T21:56:14.706 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.706 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.706 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:56:14.707 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:10.122892+0000", 2021-03-24T21:56:14.707 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1358000000000004e-05 2021-03-24T21:56:14.707 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.707 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.708 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:56:14.708 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:10.122954+0000", 2021-03-24T21:56:14.708 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4607999999999999e-05 2021-03-24T21:56:14.708 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.708 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.709 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:56:14.709 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:10.122978+0000", 2021-03-24T21:56:14.709 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011983791000000001 2021-03-24T21:56:14.709 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.709 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.710 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T21:56:14.710 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:10.134962+0000", 2021-03-24T21:56:14.710 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.7522999999999997e-05 2021-03-24T21:56:14.710 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.711 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.711 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:56:14.711 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:10.135030+0000", 2021-03-24T21:56:14.711 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.012177426999999999 2021-03-24T21:56:14.711 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:14.712 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:56:14.712 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:14.712 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.712 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.712 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.14760.0:640 7.7 e141/108 7:ef6d6bee:::smithi12556470-24:head v 141'71, mlcod=141'68)", 2021-03-24T21:56:14.713 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:55:10.123966+0000", 2021-03-24T21:56:14.713 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 64.369366501000002, 2021-03-24T21:56:14.713 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011333823999999999, 2021-03-24T21:56:14.713 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:56:14.714 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:14.714 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:56:14.714 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.714 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:56:14.714 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:10.123966+0000", 2021-03-24T21:56:14.715 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:56:14.715 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.715 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.715 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:56:14.716 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:10.123966+0000", 2021-03-24T21:56:14.716 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2021-03-24T21:56:14.716 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.716 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.716 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:56:14.717 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:10.123965+0000", 2021-03-24T21:56:14.717 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0403e-05 2021-03-24T21:56:14.717 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.717 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.717 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:56:14.718 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:10.123975+0000", 2021-03-24T21:56:14.718 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2614e-05 2021-03-24T21:56:14.718 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.718 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.718 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:56:14.719 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:10.123988+0000", 2021-03-24T21:56:14.719 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2057e-05 2021-03-24T21:56:14.719 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.719 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.719 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:56:14.720 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:10.124000+0000", 2021-03-24T21:56:14.720 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000183594 2021-03-24T21:56:14.720 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.720 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.721 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:56:14.721 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:10.124184+0000", 2021-03-24T21:56:14.721 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5075e-05 2021-03-24T21:56:14.721 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.721 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.722 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:56:14.722 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:10.124209+0000", 2021-03-24T21:56:14.722 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011050944 2021-03-24T21:56:14.722 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.722 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.723 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T21:56:14.723 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:10.135260+0000", 2021-03-24T21:56:14.723 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0491999999999998e-05 2021-03-24T21:56:14.723 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.724 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.724 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:56:14.724 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:10.135300+0000", 2021-03-24T21:56:14.724 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011333823999999999 2021-03-24T21:56:14.724 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:14.725 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:56:14.725 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:14.725 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.725 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.725 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.14760.0:646 7.c e141/101 7:305739dc:::smithi12556470-36:head v 141'48, mlcod=134'43)", 2021-03-24T21:56:14.726 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:55:10.124320+0000", 2021-03-24T21:56:14.726 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 64.369013226999996, 2021-03-24T21:56:14.726 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011034776, 2021-03-24T21:56:14.726 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:56:14.726 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:14.727 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:56:14.727 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.727 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:56:14.727 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:10.124320+0000", 2021-03-24T21:56:14.728 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:56:14.728 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.728 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.728 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:56:14.728 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:10.124320+0000", 2021-03-24T21:56:14.729 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2021-03-24T21:56:14.729 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.729 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.729 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:56:14.729 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:10.124318+0000", 2021-03-24T21:56:14.730 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9097e-05 2021-03-24T21:56:14.730 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.730 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.730 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:56:14.730 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:10.124337+0000", 2021-03-24T21:56:14.731 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7756000000000001e-05 2021-03-24T21:56:14.731 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.731 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.731 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:56:14.732 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:10.124355+0000", 2021-03-24T21:56:14.732 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6079999999999996e-06 2021-03-24T21:56:14.732 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.732 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.732 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:56:14.733 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:10.124363+0000", 2021-03-24T21:56:14.733 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.7015999999999995e-05 2021-03-24T21:56:14.733 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.733 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.733 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:56:14.734 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:10.124450+0000", 2021-03-24T21:56:14.734 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2881e-05 2021-03-24T21:56:14.734 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.734 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.735 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:56:14.735 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:10.124473+0000", 2021-03-24T21:56:14.735 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.010843952 2021-03-24T21:56:14.735 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.735 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.736 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T21:56:14.736 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:10.135317+0000", 2021-03-24T21:56:14.736 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7005000000000002e-05 2021-03-24T21:56:14.736 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.737 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.737 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:56:14.737 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:10.135354+0000", 2021-03-24T21:56:14.737 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011034776 2021-03-24T21:56:14.737 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:14.738 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:56:14.738 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:14.738 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.738 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.738 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.14760.0:647 7.c e141/101 7:305739dc:::smithi12556470-36:head v 141'49, mlcod=134'43)", 2021-03-24T21:56:14.739 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:55:10.124417+0000", 2021-03-24T21:56:14.739 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 64.368915397999999, 2021-03-24T21:56:14.739 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.010968499, 2021-03-24T21:56:14.739 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:56:14.739 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:14.740 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:56:14.740 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.740 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:56:14.740 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:10.124417+0000", 2021-03-24T21:56:14.740 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:56:14.741 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.741 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.741 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:56:14.741 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:10.124417+0000", 2021-03-24T21:56:14.742 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2021-03-24T21:56:14.742 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.742 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.742 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:56:14.742 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:10.124416+0000", 2021-03-24T21:56:14.743 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2840000000000001e-05 2021-03-24T21:56:14.743 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.743 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.743 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:56:14.743 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:10.124429+0000", 2021-03-24T21:56:14.744 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0944000000000001e-05 2021-03-24T21:56:14.744 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.744 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.744 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:56:14.744 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:10.124440+0000", 2021-03-24T21:56:14.745 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.7989999999999996e-06 2021-03-24T21:56:14.745 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.745 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.745 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:56:14.746 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:10.124447+0000", 2021-03-24T21:56:14.746 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00032511900000000002 2021-03-24T21:56:14.746 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.746 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.746 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:56:14.747 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:10.124772+0000", 2021-03-24T21:56:14.747 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0226000000000002e-05 2021-03-24T21:56:14.747 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.747 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.748 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:56:14.748 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:10.124802+0000", 2021-03-24T21:56:14.748 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0105545 2021-03-24T21:56:14.748 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.749 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.749 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T21:56:14.749 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:10.135357+0000", 2021-03-24T21:56:14.749 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9173999999999999e-05 2021-03-24T21:56:14.750 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.750 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.750 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:56:14.750 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:10.135386+0000", 2021-03-24T21:56:14.751 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.010968499 2021-03-24T21:56:14.751 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:14.751 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:56:14.751 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:14.751 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.752 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.752 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.14760.0:876 7.6 e162/99 7:6ed83fae:::smithi12556470-29:head v 162'78, mlcod=159'75)", 2021-03-24T21:56:14.752 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:55:31.128546+0000", 2021-03-24T21:56:14.753 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 43.364786385000002, 2021-03-24T21:56:14.753 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0087059890000000008, 2021-03-24T21:56:14.753 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:56:14.753 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:14.753 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:56:14.754 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.754 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:56:14.754 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:31.128546+0000", 2021-03-24T21:56:14.754 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:56:14.754 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.755 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.755 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:56:14.755 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:31.128546+0000", 2021-03-24T21:56:14.755 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2021-03-24T21:56:14.756 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.756 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.756 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:56:14.756 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:31.128545+0000", 2021-03-24T21:56:14.756 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034303760000000002 2021-03-24T21:56:14.757 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.757 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.757 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:56:14.757 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:31.131975+0000", 2021-03-24T21:56:14.758 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4629000000000001e-05 2021-03-24T21:56:14.758 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.758 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.758 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:56:14.758 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:31.132000+0000", 2021-03-24T21:56:14.759 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.8309999999999996e-06 2021-03-24T21:56:14.759 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.759 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.759 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:56:14.759 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:31.132009+0000", 2021-03-24T21:56:14.760 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000193303 2021-03-24T21:56:14.760 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.760 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.760 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:56:14.761 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:31.132203+0000", 2021-03-24T21:56:14.761 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8297999999999997e-05 2021-03-24T21:56:14.761 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.761 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.761 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:56:14.762 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:31.132251+0000", 2021-03-24T21:56:14.762 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.004935381 2021-03-24T21:56:14.762 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.762 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.762 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T21:56:14.763 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:31.137186+0000", 2021-03-24T21:56:14.763 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6068000000000005e-05 2021-03-24T21:56:14.763 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.763 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.764 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:56:14.764 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:31.137252+0000", 2021-03-24T21:56:14.764 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0087059890000000008 2021-03-24T21:56:14.764 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:14.764 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:56:14.765 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:14.765 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.765 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.765 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:1285 7.5 7:ad88ec1b:::smithi12556470-47:head [write 2948352~396241] snapc 56=[56] ondisk+write+known_if_redirected e192)", 2021-03-24T21:56:14.765 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:56:01.553407+0000", 2021-03-24T21:56:14.766 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 12.939925946000001, 2021-03-24T21:56:14.766 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0087202960000000006, 2021-03-24T21:56:14.766 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:56:14.766 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:14.767 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:56:14.767 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T21:56:14.767 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:56:14.767 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 1285 2021-03-24T21:56:14.767 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.768 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:56:14.768 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.768 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:56:14.768 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:01.553407+0000", 2021-03-24T21:56:14.769 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:56:14.769 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.769 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.769 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:56:14.769 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:01.553407+0000", 2021-03-24T21:56:14.770 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2021-03-24T21:56:14.770 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.770 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.770 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:56:14.770 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:01.553405+0000", 2021-03-24T21:56:14.771 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00059060500000000001 2021-03-24T21:56:14.771 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.771 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.771 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:56:14.772 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:01.553995+0000", 2021-03-24T21:56:14.772 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7129999999999999e-05 2021-03-24T21:56:14.772 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.772 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.772 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:56:14.773 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:01.554022+0000", 2021-03-24T21:56:14.773 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0997e-05 2021-03-24T21:56:14.773 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.773 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.773 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:56:14.774 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:01.554033+0000", 2021-03-24T21:56:14.774 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3052000000000004e-05 2021-03-24T21:56:14.774 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.774 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.775 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:56:14.775 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:01.554116+0000", 2021-03-24T21:56:14.775 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016363 2021-03-24T21:56:14.775 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.775 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.776 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:56:14.776 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:01.554280+0000", 2021-03-24T21:56:14.776 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00032827000000000003 2021-03-24T21:56:14.776 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.776 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.777 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 0,2", 2021-03-24T21:56:14.777 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:01.554608+0000", 2021-03-24T21:56:14.777 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0026796609999999998 2021-03-24T21:56:14.777 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.778 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.778 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T21:56:14.778 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:01.557288+0000", 2021-03-24T21:56:14.778 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00024691200000000001 2021-03-24T21:56:14.778 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.779 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.779 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:14.779 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:01.557535+0000", 2021-03-24T21:56:14.779 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0044833929999999996 2021-03-24T21:56:14.780 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.780 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.780 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:14.780 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:01.562018+0000", 2021-03-24T21:56:14.780 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.4158000000000005e-05 2021-03-24T21:56:14.781 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.781 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.781 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T21:56:14.781 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:01.562083+0000", 2021-03-24T21:56:14.782 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4620999999999999e-05 2021-03-24T21:56:14.782 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.782 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.782 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:56:14.782 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:01.562127+0000", 2021-03-24T21:56:14.783 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0087202960000000006 2021-03-24T21:56:14.783 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:14.783 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:56:14.783 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:14.783 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.784 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.784 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(osd.0.0:595 7.7 e194/108 7:ef6d6bee:::smithi12556470-24:57 v 194'183, mlcod=193'178)", 2021-03-24T21:56:14.784 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:56:03.249865+0000", 2021-03-24T21:56:14.784 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 11.243468115000001, 2021-03-24T21:56:14.785 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011113328, 2021-03-24T21:56:14.785 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:56:14.785 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:14.785 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:56:14.785 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.786 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:56:14.786 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:03.249865+0000", 2021-03-24T21:56:14.786 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:56:14.786 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.786 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.787 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:56:14.787 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:03.249865+0000", 2021-03-24T21:56:14.787 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2021-03-24T21:56:14.787 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.788 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.788 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:56:14.788 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:03.249864+0000", 2021-03-24T21:56:14.788 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1922e-05 2021-03-24T21:56:14.788 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.789 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.789 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:56:14.789 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:03.249876+0000", 2021-03-24T21:56:14.789 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.8169999999999999e-06 2021-03-24T21:56:14.789 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.790 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.790 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:56:14.790 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:03.249884+0000", 2021-03-24T21:56:14.790 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2599999999999994e-06 2021-03-24T21:56:14.791 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.791 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.791 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:56:14.791 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:03.249893+0000", 2021-03-24T21:56:14.791 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00096562199999999996 2021-03-24T21:56:14.792 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.792 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.792 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:56:14.792 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:03.250858+0000", 2021-03-24T21:56:14.792 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8619000000000003e-05 2021-03-24T21:56:14.793 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.793 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.793 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:56:14.793 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:03.250907+0000", 2021-03-24T21:56:14.793 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.010029877 2021-03-24T21:56:14.794 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.794 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.794 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T21:56:14.794 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:03.260937+0000", 2021-03-24T21:56:14.795 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1041999999999998e-05 2021-03-24T21:56:14.795 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:56:14.795 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:56:14.795 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:56:14.795 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:03.260978+0000", 2021-03-24T21:56:14.796 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011113328 2021-03-24T21:56:14.796 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:14.796 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:56:14.796 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:14.797 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:56:14.797 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:56:14.797 INFO:teuthology.orchestra.run.smithi142.stdout:} 2021-03-24T21:56:14.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:14.799 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:14.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:56:14.900 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:14.955 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:14.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:14.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:56:15.008 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-24T21:56:15.061 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:15.112 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:15.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:15.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:56:15.124 INFO:tasks.rados.rados.0.smithi125.stdout:537: done (1 left) 2021-03-24T21:56:15.125 INFO:tasks.rados.rados.0.smithi125.stdout:538: write oid 11 current snap is 50 2021-03-24T21:56:15.125 INFO:tasks.rados.rados.0.smithi125.stdout:538: seq_num 223 ranges {547961=554764,1616308=703740,2741092=698600} 2021-03-24T21:56:15.131 INFO:tasks.rados.rados.0.smithi125.stdout:538: writing smithi12556470-11 from 547961 to 1102725 tid 1 2021-03-24T21:56:15.142 INFO:tasks.rados.rados.0.smithi125.stdout:538: writing smithi12556470-11 from 1616308 to 2320048 tid 2 2021-03-24T21:56:15.150 INFO:tasks.rados.rados.0.smithi125.stdout:538: writing smithi12556470-11 from 2741092 to 3439692 tid 3 2021-03-24T21:56:15.151 INFO:tasks.rados.rados.0.smithi125.stdout:539: rollback oid 38 current snap is 50 2021-03-24T21:56:15.151 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 38 to 47 2021-03-24T21:56:15.151 INFO:tasks.rados.rados.0.smithi125.stdout:540: copy_from oid 47 from oid 3 current snap is 50 2021-03-24T21:56:15.151 INFO:tasks.rados.rados.0.smithi125.stdout:541: write oid 5 current snap is 50 2021-03-24T21:56:15.152 INFO:tasks.rados.rados.0.smithi125.stdout:541: seq_num 224 ranges {564768=630013,1674790=447398,2768515=626225} 2021-03-24T21:56:15.157 INFO:tasks.rados.rados.0.smithi125.stdout:541: writing smithi12556470-5 from 564768 to 1194781 tid 1 2021-03-24T21:56:15.162 INFO:tasks.rados.rados.0.smithi125.stdout:541: writing smithi12556470-5 from 1674790 to 2122188 tid 2 2021-03-24T21:56:15.170 INFO:tasks.rados.rados.0.smithi125.stdout:541: writing smithi12556470-5 from 2768515 to 3394740 tid 3 2021-03-24T21:56:15.171 INFO:tasks.rados.rados.0.smithi125.stdout:542: write (excl) oid 36 current snap is 50 2021-03-24T21:56:15.171 INFO:tasks.rados.rados.0.smithi125.stdout:535: finishing write tid 1 to smithi12556470-28 2021-03-24T21:56:15.171 INFO:tasks.rados.rados.0.smithi125.stdout:542: seq_num 225 ranges {581575=705262,1733272=707409,3195938=437497} 2021-03-24T21:56:15.177 INFO:tasks.rados.rados.0.smithi125.stdout:542: writing smithi12556470-36 from 581575 to 1286837 tid 1 2021-03-24T21:56:15.185 INFO:tasks.rados.rados.0.smithi125.stdout:542: writing smithi12556470-36 from 1733272 to 2440681 tid 2 2021-03-24T21:56:15.191 INFO:tasks.rados.rados.0.smithi125.stdout:542: writing smithi12556470-36 from 3195938 to 3633435 tid 3 2021-03-24T21:56:15.191 INFO:tasks.rados.rados.0.smithi125.stdout:543: write (excl) oid 22 current snap is 50 2021-03-24T21:56:15.192 INFO:tasks.rados.rados.0.smithi125.stdout:543: seq_num 226 ranges {598382=780511,2075401=451067,3107008=648769} 2021-03-24T21:56:15.199 INFO:tasks.rados.rados.0.smithi125.stdout:543: writing smithi12556470-22 from 598382 to 1378893 tid 1 2021-03-24T21:56:15.206 INFO:tasks.rados.rados.0.smithi125.stdout:543: writing smithi12556470-22 from 2075401 to 2526468 tid 2 2021-03-24T21:56:15.214 INFO:tasks.rados.rados.0.smithi125.stdout:543: writing smithi12556470-22 from 3107008 to 3755777 tid 3 2021-03-24T21:56:15.214 INFO:tasks.rados.rados.0.smithi125.stdout:544: delete oid 6 current snap is 50 2021-03-24T21:56:15.215 INFO:tasks.rados.rados.0.smithi125.stdout:535: finishing write tid 2 to smithi12556470-28 2021-03-24T21:56:15.215 INFO:tasks.rados.rados.0.smithi125.stdout:535: finishing write tid 3 to smithi12556470-28 2021-03-24T21:56:15.215 INFO:tasks.rados.rados.0.smithi125.stdout:535: finishing write tid 4 to smithi12556470-28 2021-03-24T21:56:15.215 INFO:tasks.rados.rados.0.smithi125.stdout:535: finishing write tid 5 to smithi12556470-28 2021-03-24T21:56:15.215 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 28 v 144 (ObjNum 221 snap 50 seq_num 221) dirty exists 2021-03-24T21:56:15.216 INFO:tasks.rados.rados.0.smithi125.stdout:535: left oid 28 (ObjNum 221 snap 50 seq_num 221) 2021-03-24T21:56:15.216 INFO:tasks.rados.rados.0.smithi125.stdout:538: finishing write tid 1 to smithi12556470-11 2021-03-24T21:56:15.216 INFO:tasks.rados.rados.0.smithi125.stdout:538: finishing write tid 2 to smithi12556470-11 2021-03-24T21:56:15.216 INFO:tasks.rados.rados.0.smithi125.stdout:539: finishing rollback tid 1 to smithi12556470-38 2021-03-24T21:56:15.217 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 38 v 0 (ObjNum 2 snap 5 seq_num 0) dirty dne 2021-03-24T21:56:15.217 INFO:tasks.rados.rados.0.smithi125.stdout:538: finishing write tid 3 to smithi12556470-11 2021-03-24T21:56:15.217 INFO:tasks.rados.rados.0.smithi125.stdout:538: finishing write tid 4 to smithi12556470-11 2021-03-24T21:56:15.217 INFO:tasks.rados.rados.0.smithi125.stdout:538: finishing write tid 5 to smithi12556470-11 2021-03-24T21:56:15.217 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 11 v 174 (ObjNum 223 snap 50 seq_num 223) dirty exists 2021-03-24T21:56:15.218 INFO:tasks.rados.rados.0.smithi125.stdout:538: left oid 11 (ObjNum 223 snap 50 seq_num 223) 2021-03-24T21:56:15.218 INFO:tasks.rados.rados.0.smithi125.stdout:541: finishing write tid 1 to smithi12556470-5 2021-03-24T21:56:15.218 INFO:tasks.rados.rados.0.smithi125.stdout:541: finishing write tid 2 to smithi12556470-5 2021-03-24T21:56:15.218 INFO:tasks.rados.rados.0.smithi125.stdout:540: finishing copy_from to smithi12556470-47 2021-03-24T21:56:15.219 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 47 v 92 (ObjNum 220 snap 49 seq_num 220) dirty exists 2021-03-24T21:56:15.219 INFO:tasks.rados.rados.0.smithi125.stdout:541: finishing write tid 3 to smithi12556470-5 2021-03-24T21:56:15.219 INFO:tasks.rados.rados.0.smithi125.stdout:541: finishing write tid 4 to smithi12556470-5 2021-03-24T21:56:15.219 INFO:tasks.rados.rados.0.smithi125.stdout:540: finishing copy_from racing read to smithi12556470-47 2021-03-24T21:56:15.220 INFO:tasks.rados.rados.0.smithi125.stdout:541: finishing write tid 5 to smithi12556470-5 2021-03-24T21:56:15.220 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 5 v 178 (ObjNum 224 snap 50 seq_num 224) dirty exists 2021-03-24T21:56:15.220 INFO:tasks.rados.rados.0.smithi125.stdout:541: left oid 5 (ObjNum 224 snap 50 seq_num 224) 2021-03-24T21:56:15.220 INFO:tasks.rados.rados.0.smithi125.stdout:542: finishing write tid 1 to smithi12556470-36 2021-03-24T21:56:15.221 INFO:tasks.rados.rados.0.smithi125.stdout:542: finishing write tid 2 to smithi12556470-36 2021-03-24T21:56:15.221 INFO:tasks.rados.rados.0.smithi125.stdout:542: finishing write tid 3 to smithi12556470-36 2021-03-24T21:56:15.221 INFO:tasks.rados.rados.0.smithi125.stdout:542: finishing write tid 4 to smithi12556470-36 2021-03-24T21:56:15.221 INFO:tasks.rados.rados.0.smithi125.stdout:542: finishing write tid 5 to smithi12556470-36 2021-03-24T21:56:15.221 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 36 v 139 (ObjNum 225 snap 50 seq_num 225) dirty exists 2021-03-24T21:56:15.222 INFO:tasks.rados.rados.0.smithi125.stdout:542: left oid 36 (ObjNum 225 snap 50 seq_num 225) 2021-03-24T21:56:15.222 INFO:tasks.rados.rados.0.smithi125.stdout:543: finishing write tid 1 to smithi12556470-22 2021-03-24T21:56:15.222 INFO:tasks.rados.rados.0.smithi125.stdout:543: finishing write tid 2 to smithi12556470-22 2021-03-24T21:56:15.222 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:15.224 INFO:tasks.rados.rados.0.smithi125.stdout:535: done (7 left) 2021-03-24T21:56:15.224 INFO:tasks.rados.rados.0.smithi125.stdout:538: done (6 left) 2021-03-24T21:56:15.224 INFO:tasks.rados.rados.0.smithi125.stdout:539: done (5 left) 2021-03-24T21:56:15.224 INFO:tasks.rados.rados.0.smithi125.stdout:540: done (4 left) 2021-03-24T21:56:15.225 INFO:tasks.rados.rados.0.smithi125.stdout:541: done (3 left) 2021-03-24T21:56:15.225 INFO:tasks.rados.rados.0.smithi125.stdout:542: done (2 left) 2021-03-24T21:56:15.225 INFO:tasks.rados.rados.0.smithi125.stdout:544: done (1 left) 2021-03-24T21:56:15.225 INFO:tasks.rados.rados.0.smithi125.stdout:545: copy_from oid 27 from oid 46 current snap is 50 2021-03-24T21:56:15.226 INFO:tasks.rados.rados.0.smithi125.stdout:546: snap_remove snap 47 2021-03-24T21:56:15.253 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:15 smithi125 conmon[24927]: cluster 2021-03-24T21:56:13.586119 2021-03-24T21:56:15.253 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:15 smithi125 conmon[24927]: +0000 osd.3 (osd.3) 58 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:56:15.253 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:15 smithi125 conmon[24927]: cluster 2021-03-24T21:56:13.586419 2021-03-24T21:56:15.254 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:15 smithi125 conmon[24927]: +0000 osd.3 (osd.3) 59 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:56:15.254 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:15 smithi125 conmon[24927]: cluster 2021-03-24T21:56:14. 2021-03-24T21:56:15.254 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:15 smithi125 conmon[24927]: 125616+0000 mon.a (mon.0) 809 : cluster [DBG] osdmap e205: 8 total, 8 up, 6 in 2021-03-24T21:56:15.254 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:15 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:56:15.255 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:15 smithi125 conmon[24927]: 03-24T21:56:15.120529+0000 mon.a 2021-03-24T21:56:15.255 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:15 smithi125 conmon[24927]: (mon.0) 810 : cluster [DBG] osdmap e206: 8 total, 8 up, 6 in 2021-03-24T21:56:15.255 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:15 smithi125 conmon[19936]: cluster 2021-03-24T21:56:13.586119+0000 osd.3 ( 2021-03-24T21:56:15.255 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:15 smithi125 conmon[19936]: osd.3) 58 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:56:15.256 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:15 smithi125 conmon[19936]: cluster 2021-03-24T21:56:13.586419+0000 osd.3 (osd.3) 59 : cluster 2021-03-24T21:56:15.256 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:15 smithi125 conmon[19936]: [DBG] purged_snaps scrub ok 2021-03-24T21:56:15.256 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:15 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T21:56:15.256 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:15 smithi125 conmon[19936]: T21:56:14.125616+0000 mon.a (mon.0) 809 : cluster [DBG] 2021-03-24T21:56:15.256 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:15 smithi125 conmon[19936]: osdmap e205: 8 total, 8 up, 6 in 2021-03-24T21:56:15.257 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:15 smithi125 conmon[19936]: cluster 2021-03-24T21:56:15.120529 2021-03-24T21:56:15.257 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:15 smithi125 conmon[19936]: +0000 mon.a (mon.0) 810 : cluster [DBG] 2021-03-24T21:56:15.257 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:15 smithi125 conmon[19936]: osdmap e206: 8 total, 8 up, 6 in 2021-03-24T21:56:15.275 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:15.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:15.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:56:15.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:15 smithi142 conmon[20125]: cluster 2021-03-24T21:56:13.586119 2021-03-24T21:56:15.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:15 smithi142 conmon[20125]: +0000 osd.3 (osd.3) 58 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:56:15.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:15 smithi142 conmon[20125]: cluster 2021-03-24T21:56:13.586419+0000 osd.3 (osd.3) 59 : cluster 2021-03-24T21:56:15.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:15 smithi142 conmon[20125]: [DBG] purged_snaps scrub ok 2021-03-24T21:56:15.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:15 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:56:15.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:15 smithi142 conmon[20125]: -03-24T21:56:14.125616+0000 mon.a 2021-03-24T21:56:15.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:15 smithi142 conmon[20125]: (mon.0) 809 : cluster [DBG] 2021-03-24T21:56:15.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:15 smithi142 conmon[20125]: osdmap e205: 8 total, 8 up, 6 in 2021-03-24T21:56:15.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:15 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:56:15.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:15 smithi142 conmon[20125]: -03-24 2021-03-24T21:56:15.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:15 smithi142 conmon[20125]: T21:56:15.120529+0000 mon.a ( 2021-03-24T21:56:15.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:15 smithi142 conmon[20125]: mon.0) 810 : cluster [DBG] osdmap e206: 8 total, 8 up, 6 in 2021-03-24T21:56:15.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:15 smithi142 conmon[20125]: 2021-03-24T21:56:15.381 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:15.425 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:56:15.431 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.0 dump_ops_in_flight 2021-03-24T21:56:15.440 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:15.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:15.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:56:15.546 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:15.595 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:15.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:15.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:56:15.699 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:15.716 INFO:teuthology.orchestra.run.smithi125.stdout:moved 0 / 474 (0%) 2021-03-24T21:56:15.717 INFO:teuthology.orchestra.run.smithi125.stdout:avg 79 2021-03-24T21:56:15.717 INFO:teuthology.orchestra.run.smithi125.stdout:stddev 6.60808 -> 6.60808 (expected baseline 8.11377) 2021-03-24T21:56:15.717 INFO:teuthology.orchestra.run.smithi125.stdout:min osd.4 with 71 -> 71 pgs (0.898734 -> 0.898734 * mean) 2021-03-24T21:56:15.717 INFO:teuthology.orchestra.run.smithi125.stdout:max osd.7 with 88 -> 88 pgs (1.11392 -> 1.11392 * mean) 2021-03-24T21:56:15.718 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-24T21:56:15.718 INFO:teuthology.orchestra.run.smithi125.stdout:oload 1000 2021-03-24T21:56:15.718 INFO:teuthology.orchestra.run.smithi125.stdout:max_change 0.05 2021-03-24T21:56:15.718 INFO:teuthology.orchestra.run.smithi125.stdout:max_change_osds 4 2021-03-24T21:56:15.718 INFO:teuthology.orchestra.run.smithi125.stdout:average_utilization 0.0008 2021-03-24T21:56:15.719 INFO:teuthology.orchestra.run.smithi125.stdout:overload_utilization 0.0077 2021-03-24T21:56:15.719 INFO:teuthology.orchestra.run.smithi125.stderr:no change 2021-03-24T21:56:15.745 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:15.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:15.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:56:15.754 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:56:15.851 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:15.914 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:15.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:15.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:56:16.020 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:16.077 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:16.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:16.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:56:16.162 INFO:tasks.rados.rados.0.smithi125.stdout:546: done (2 left) 2021-03-24T21:56:16.162 INFO:tasks.rados.rados.0.smithi125.stdout:547: read oid 43 snap -1 2021-03-24T21:56:16.162 INFO:tasks.rados.rados.0.smithi125.stdout:547: expect deleted 2021-03-24T21:56:16.163 INFO:tasks.rados.rados.0.smithi125.stdout:548: write oid 33 current snap is 50 2021-03-24T21:56:16.163 INFO:tasks.rados.rados.0.smithi125.stdout:548: seq_num 228 ranges {631996=531009,1792365=454736,2761854=671313} 2021-03-24T21:56:16.168 INFO:tasks.rados.rados.0.smithi125.stdout:548: writing smithi12556470-33 from 631996 to 1163005 tid 1 2021-03-24T21:56:16.175 INFO:tasks.rados.rados.0.smithi125.stdout:548: writing smithi12556470-33 from 1792365 to 2247101 tid 2 2021-03-24T21:56:16.182 INFO:tasks.rados.rados.0.smithi125.stdout:548: writing smithi12556470-33 from 2761854 to 3433167 tid 3 2021-03-24T21:56:16.182 INFO:tasks.rados.rados.0.smithi125.stdout:549: read oid 35 snap -1 2021-03-24T21:56:16.183 INFO:tasks.rados.rados.0.smithi125.stdout:549: expect (ObjNum 150 snap 30 seq_num 150) 2021-03-24T21:56:16.183 INFO:tasks.rados.rados.0.smithi125.stdout:550: rollback oid 39 current snap is 50 2021-03-24T21:56:16.183 INFO:tasks.rados.rados.0.smithi125.stdout:550: done (5 left) 2021-03-24T21:56:16.183 INFO:tasks.rados.rados.0.smithi125.stdout:551: read oid 40 snap -1 2021-03-24T21:56:16.184 INFO:tasks.rados.rados.0.smithi125.stdout:551: expect (ObjNum 211 snap 46 seq_num 211) 2021-03-24T21:56:16.184 INFO:tasks.rados.rados.0.smithi125.stdout:552: rollback oid 49 current snap is 50 2021-03-24T21:56:16.184 INFO:tasks.rados.rados.0.smithi125.stdout:552: done (6 left) 2021-03-24T21:56:16.184 INFO:tasks.rados.rados.0.smithi125.stdout:553: copy_from oid 2 from oid 32 current snap is 50 2021-03-24T21:56:16.184 INFO:tasks.rados.rados.0.smithi125.stdout:543: finishing write tid 3 to smithi12556470-22 2021-03-24T21:56:16.185 INFO:tasks.rados.rados.0.smithi125.stdout:543: finishing write tid 4 to smithi12556470-22 2021-03-24T21:56:16.185 INFO:tasks.rados.rados.0.smithi125.stdout:543: finishing write tid 5 to smithi12556470-22 2021-03-24T21:56:16.185 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 22 v 154 (ObjNum 226 snap 50 seq_num 226) dirty exists 2021-03-24T21:56:16.185 INFO:tasks.rados.rados.0.smithi125.stdout:543: left oid 22 (ObjNum 226 snap 50 seq_num 226) 2021-03-24T21:56:16.186 INFO:tasks.rados.rados.0.smithi125.stdout:543: done (6 left) 2021-03-24T21:56:16.186 INFO:tasks.rados.rados.0.smithi125.stdout:554: write (excl) oid 37 current snap is 50 2021-03-24T21:56:16.186 INFO:tasks.rados.rados.0.smithi125.stdout:545: finishing copy_from to smithi12556470-27 2021-03-24T21:56:16.186 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 27 v 64 (ObjNum 218 snap 48 seq_num 218) dirty exists 2021-03-24T21:56:16.189 INFO:tasks.rados.rados.0.smithi125.stdout:554: seq_num 229 ranges {648803=722611,1967200=714747,3421983=426820} 2021-03-24T21:56:16.190 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:16.190 INFO:tasks.rados.rados.0.smithi125.stdout:554: writing smithi12556470-37 from 648803 to 1371414 tid 1 2021-03-24T21:56:16.199 INFO:tasks.rados.rados.0.smithi125.stdout:554: writing smithi12556470-37 from 1967200 to 2681947 tid 2 2021-03-24T21:56:16.207 INFO:tasks.rados.rados.0.smithi125.stdout:554: writing smithi12556470-37 from 3421983 to 3848803 tid 3 2021-03-24T21:56:16.207 INFO:tasks.rados.rados.0.smithi125.stdout:555: copy_from oid 15 from oid 21 current snap is 50 2021-03-24T21:56:16.207 INFO:tasks.rados.rados.0.smithi125.stdout:545: finishing copy_from racing read to smithi12556470-27 2021-03-24T21:56:16.208 INFO:tasks.rados.rados.0.smithi125.stdout:548: finishing write tid 1 to smithi12556470-33 2021-03-24T21:56:16.208 INFO:tasks.rados.rados.0.smithi125.stdout:548: finishing write tid 2 to smithi12556470-33 2021-03-24T21:56:16.208 INFO:tasks.rados.rados.0.smithi125.stdout:548: finishing write tid 3 to smithi12556470-33 2021-03-24T21:56:16.208 INFO:tasks.rados.rados.0.smithi125.stdout:553: finishing copy_from to smithi12556470-2 2021-03-24T21:56:16.209 INFO:tasks.rados.rados.0.smithi125.stdout:553: got expected ENOENT (src dne) 2021-03-24T21:56:16.209 INFO:tasks.rados.rados.0.smithi125.stdout:548: finishing write tid 4 to smithi12556470-33 2021-03-24T21:56:16.209 INFO:tasks.rados.rados.0.smithi125.stdout:545: done (7 left) 2021-03-24T21:56:16.209 INFO:tasks.rados.rados.0.smithi125.stdout:547: done (6 left) 2021-03-24T21:56:16.209 INFO:tasks.rados.rados.0.smithi125.stdout:556: read oid 7 snap -1 2021-03-24T21:56:16.210 INFO:tasks.rados.rados.0.smithi125.stdout:556: expect (ObjNum 150 snap 30 seq_num 150) 2021-03-24T21:56:16.210 INFO:tasks.rados.rados.0.smithi125.stdout:557: snap_create 2021-03-24T21:56:16.210 INFO:tasks.rados.rados.0.smithi125.stdout:548: finishing write tid 5 to smithi12556470-33 2021-03-24T21:56:16.210 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 33 v 198 (ObjNum 228 snap 50 seq_num 228) dirty exists 2021-03-24T21:56:16.211 INFO:tasks.rados.rados.0.smithi125.stdout:548: left oid 33 (ObjNum 228 snap 50 seq_num 228) 2021-03-24T21:56:16.211 INFO:tasks.rados.rados.0.smithi125.stdout:553: finishing copy_from racing read to smithi12556470-2 2021-03-24T21:56:16.211 INFO:tasks.rados.rados.0.smithi125.stdout:549: expect (ObjNum 150 snap 30 seq_num 150) 2021-03-24T21:56:16.243 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:16.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:16.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:56:16.337 INFO:tasks.rados.rados.0.smithi125.stdout:551: expect (ObjNum 211 snap 46 seq_num 211) 2021-03-24T21:56:16.348 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:16.407 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:16.409 INFO:journalctl@ceph.osd.3.smithi125.stdout:Mar 24 21:56:16 smithi125 conmon[37997]: debug 2021-03-24T21:56:16.185+0000 7f1716bda700 -1 osd.3 pg_epoch: 207 pg[7.14( v 206'142 (0'0,206'142] local-lis/les=204/205 n=3 ec=198/99 lis/c=204/204 les/c/f=205/205/0 sis=204) [3,2,5] r=0 lpr=204 crt=206'142 lcod 205'140 mlcod 205'140 active+clean+snaptrim trimq=[60~1] ps=[51~1,53~1,56~2,5a~2,5e~1,61~2]] removing snap head 2021-03-24T21:56:16.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:16 smithi125 conmon[24927]: cluster 2021-03-24T21:56:14.744749+0000 2021-03-24T21:56:16.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:16 smithi125 conmon[24927]: mgr.y (mgr.14144) 363 : cluster [DBG] pgmap v470: 158 pgs: 1 peering, 157 active+clean; 119 MiB data, 420 MiB used, 536 GiB / 536 GiB avail; 27 MiB/s rd, 1.3 MiB/s wr, 26 op/s; 6.7 MiB/s, 17 objects/s recovering 2021-03-24T21:56:16.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:16 smithi125 conmon[19936]: cluster 2021-03-24T21:56 2021-03-24T21:56:16.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:16 smithi125 conmon[19936]: :14.744749+0000 mgr.y (mgr.14144) 363 : cluster [DBG] pgmap v470: 158 pgs: 1 peering, 157 active+clean; 119 MiB data, 420 MiB used, 536 GiB / 536 GiB avail; 27 MiB/s rd, 1.3 MiB/s wr, 26 op/s; 6.7 MiB/s, 17 objects/s recovering 2021-03-24T21:56:16.413 INFO:journalctl@ceph.osd.0.smithi125.stdout:Mar 24 21:56:16 smithi125 conmon[28026]: debug 2021-03-24T21:56:16.193+0000 7f61b847f700 -1 osd.0 pg_epoch: 207 pg[7.13( v 206'200 (0'0,206'200] local-lis/les=203/204 n=3 ec=198/99 lis/c=203/203 les/c/f=204/204/0 sis=203) [0,7,2] r=0 lpr=203 crt=206'200 lcod 195'198 mlcod 195'198 active+clean+snaptrim trimq=[60~1] ps=[51~1,53~1,56~2,5a~2,5e~1,61~2]] removing snap head 2021-03-24T21:56:16.413 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 21:56:16 smithi142 conmon[27337]: debug 2021-03-24T21:56:16.168+0000 7f549d718700 -1 osd.5 pg_epoch: 207 pg[7.b( v 202'162 (0'0,202'162] local-lis/les=108/109 n=6 ec=99/99 lis/c=108/108 les/c/f=109/109/0 sis=108) [5,2,4] r=0 lpr=108 crt=202'162 lcod 201'160 mlcod 201'160 active+clean+snaptrim trimq=[60~1] ps=[51~1,53~1,56~2,5a~2,5e~1,61~2]] removing snap head 2021-03-24T21:56:16.414 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:16 smithi142 conmon[20125]: cluster 2021-03-24T21:56:14 2021-03-24T21:56:16.415 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:16 smithi142 conmon[20125]: .744749+0000 mgr.y (mgr.14144) 363 : cluster [DBG] pgmap v470: 158 pgs: 1 peering, 157 active+clean; 119 MiB data, 420 MiB used, 536 GiB / 536 GiB avail; 27 MiB/s rd, 1.3 MiB/s wr, 26 op/s; 6.7 MiB/s, 17 objects/s recovering 2021-03-24T21:56:16.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:16.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:56:16.518 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:16.556 INFO:tasks.rados.rados.0.smithi125.stdout:554: finishing write tid 1 to smithi12556470-37 2021-03-24T21:56:16.556 INFO:tasks.rados.rados.0.smithi125.stdout:554: finishing write tid 2 to smithi12556470-37 2021-03-24T21:56:16.556 INFO:tasks.rados.rados.0.smithi125.stdout:556: expect (ObjNum 150 snap 30 seq_num 150) 2021-03-24T21:56:16.569 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:16.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:16.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:56:16.675 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:16.683 INFO:tasks.rados.rados.0.smithi125.stdout:554: finishing write tid 3 to smithi12556470-37 2021-03-24T21:56:16.684 INFO:tasks.rados.rados.0.smithi125.stdout:554: finishing write tid 4 to smithi12556470-37 2021-03-24T21:56:16.684 INFO:tasks.rados.rados.0.smithi125.stdout:554: finishing write tid 5 to smithi12556470-37 2021-03-24T21:56:16.684 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 37 v 68 (ObjNum 229 snap 50 seq_num 229) dirty exists 2021-03-24T21:56:16.684 INFO:tasks.rados.rados.0.smithi125.stdout:554: left oid 37 (ObjNum 229 snap 50 seq_num 229) 2021-03-24T21:56:16.685 INFO:tasks.rados.rados.0.smithi125.stdout:555: finishing copy_from to smithi12556470-15 2021-03-24T21:56:16.685 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 15 v 49 (ObjNum 83 snap 10 seq_num 83) dirty exists 2021-03-24T21:56:16.685 INFO:tasks.rados.rados.0.smithi125.stdout:555: finishing copy_from racing read to smithi12556470-15 2021-03-24T21:56:16.733 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:16.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:16.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:56:16.840 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:16.895 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:16.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:16.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:56:17.002 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:17.030 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-24T21:56:17.063 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:17.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:17.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:56:17.169 INFO:tasks.rados.rados.0.smithi125.stdout:548: done (7 left) 2021-03-24T21:56:17.169 INFO:tasks.rados.rados.0.smithi125.stdout:549: done (6 left) 2021-03-24T21:56:17.170 INFO:tasks.rados.rados.0.smithi125.stdout:551: done (5 left) 2021-03-24T21:56:17.170 INFO:tasks.rados.rados.0.smithi125.stdout:553: done (4 left) 2021-03-24T21:56:17.170 INFO:tasks.rados.rados.0.smithi125.stdout:554: done (3 left) 2021-03-24T21:56:17.171 INFO:tasks.rados.rados.0.smithi125.stdout:555: done (2 left) 2021-03-24T21:56:17.171 INFO:tasks.rados.rados.0.smithi125.stdout:556: done (1 left) 2021-03-24T21:56:17.171 INFO:tasks.rados.rados.0.smithi125.stdout:557: done (0 left) 2021-03-24T21:56:17.171 INFO:tasks.rados.rados.0.smithi125.stdout:558: snap_remove snap 50 2021-03-24T21:56:17.172 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:17.231 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:17.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:17.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:56:17.337 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:17.397 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:17.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:17.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:56:17.504 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:17 smithi125 conmon[19936]: audit 2021-03-24T 2021-03-24T21:56:17.504 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:17 smithi125 conmon[19936]: 21:56:15.715284+0000 mgr.y (mgr.14144) 364 : audit [DBG] 2021-03-24T21:56:17.504 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:17 smithi125 conmon[19936]: from='client.14946 v1:172.21.15.125:0/3001108815' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:56:17.504 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:17 smithi125 conmon[19936]: cluster 2021-03-24T21:56:17.505 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:17 smithi125 conmon[19936]: 2021-03-24T21 2021-03-24T21:56:17.505 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:17 smithi125 conmon[19936]: :56:16. 2021-03-24T21:56:17.505 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:17 smithi125 conmon[19936]: 160331+0000 mon.a (mon 2021-03-24T21:56:17.505 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:17 smithi125 conmon[19936]: .0) 811 : 2021-03-24T21:56:17.505 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:17 smithi125 conmon[19936]: cluster [DBG] osdmap e207: 8 total, 8 up, 6 in 2021-03-24T21:56:17.506 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:17 smithi125 conmon[19936]: 2021-03-24T21:56:17.506 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:17 smithi125 conmon[24927]: audit 2021-03 2021-03-24T21:56:17.506 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:17 smithi125 conmon[24927]: -24T21:56:15.715284+0000 2021-03-24T21:56:17.507 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:17 smithi125 conmon[24927]: mgr.y (mgr. 2021-03-24T21:56:17.507 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:17 smithi125 conmon[24927]: 14144) 364 : audit 2021-03-24T21:56:17.507 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:17 smithi125 conmon[24927]: [DBG] from='client.14946 v1:172.21.15.125:0/3001108815' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:56:17.507 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:17 smithi125 conmon[24927]: 2021-03-24T21:56:17.507 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:17 smithi125 conmon[24927]: cluster 2021-03-24T21:56:17.508 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:17 smithi125 conmon[24927]: 2021-03-24T 2021-03-24T21:56:17.508 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:17 smithi125 conmon[24927]: 21:56:16. 2021-03-24T21:56:17.508 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:17 smithi125 conmon[24927]: 160331+0000 mon.a 2021-03-24T21:56:17.508 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:17 smithi125 conmon[24927]: (mon.0) 2021-03-24T21:56:17.508 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:17 smithi125 conmon[24927]: 811 : cluster 2021-03-24T21:56:17.509 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:17 smithi125 conmon[24927]: [DBG] osdmap e207: 8 total, 8 up, 6 in 2021-03-24T21:56:17.509 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:17.522 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:17 smithi142 conmon[20125]: audit 2021-03-24T21:56:15.715284+0000 mgr.y (mgr.14144) 364 : audit [DBG] from='client.14946 v1:172.21.15.125:0/3001108815' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:56:17.523 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:17 smithi142 conmon[20125]: cluster 2021-03-24T21:56:16.160331+0000 mon.a (mon.0) 811 : cluster [DBG] osdmap e207: 8 total, 8 up, 6 in 2021-03-24T21:56:17.564 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:17.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:17.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:56:17.617 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2021-03-24T21:56:17.617 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd reweight-by-utilization 1000 1.0 2021-03-24T21:56:17.670 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:17.722 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:17.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:17.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:56:17.827 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:17.875 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:17.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:17.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:56:17.980 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:18.043 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:18.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:18.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:56:18.151 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:18.181 INFO:tasks.rados.rados.0.smithi125.stdout:558: done (0 left) 2021-03-24T21:56:18.182 INFO:tasks.rados.rados.0.smithi125.stdout:559: rollback oid 10 current snap is 51 2021-03-24T21:56:18.182 INFO:tasks.rados.rados.0.smithi125.stdout:559: done (0 left) 2021-03-24T21:56:18.183 INFO:tasks.rados.rados.0.smithi125.stdout:560: read oid 2 snap -1 2021-03-24T21:56:18.183 INFO:tasks.rados.rados.0.smithi125.stdout:560: expect (ObjNum 167 snap 36 seq_num 167) 2021-03-24T21:56:18.184 INFO:tasks.rados.rados.0.smithi125.stdout:561: write oid 40 current snap is 51 2021-03-24T21:56:18.184 INFO:tasks.rados.rados.0.smithi125.stdout:561: seq_num 230 ranges {665610=797860,1909329=458405,2816700=410210} 2021-03-24T21:56:18.190 INFO:tasks.rados.rados.0.smithi125.stdout:561: writing smithi12556470-40 from 665610 to 1463470 tid 1 2021-03-24T21:56:18.195 INFO:tasks.rados.rados.0.smithi125.stdout:561: writing smithi12556470-40 from 1909329 to 2367734 tid 2 2021-03-24T21:56:18.201 INFO:tasks.rados.rados.0.smithi125.stdout:561: writing smithi12556470-40 from 2816700 to 3226910 tid 3 2021-03-24T21:56:18.201 INFO:tasks.rados.rados.0.smithi125.stdout:562: snap_create 2021-03-24T21:56:18.201 INFO:tasks.rados.rados.0.smithi125.stdout:560: expect (ObjNum 167 snap 36 seq_num 167) 2021-03-24T21:56:18.211 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:18.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:18.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:56:18.294 INFO:tasks.rados.rados.0.smithi125.stdout:561: finishing write tid 1 to smithi12556470-40 2021-03-24T21:56:18.295 INFO:tasks.rados.rados.0.smithi125.stdout:561: finishing write tid 2 to smithi12556470-40 2021-03-24T21:56:18.295 INFO:tasks.rados.rados.0.smithi125.stdout:561: finishing write tid 3 to smithi12556470-40 2021-03-24T21:56:18.295 INFO:tasks.rados.rados.0.smithi125.stdout:561: finishing write tid 4 to smithi12556470-40 2021-03-24T21:56:18.296 INFO:tasks.rados.rados.0.smithi125.stdout:561: finishing write tid 5 to smithi12556470-40 2021-03-24T21:56:18.296 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 40 v 168 (ObjNum 230 snap 51 seq_num 230) dirty exists 2021-03-24T21:56:18.296 INFO:tasks.rados.rados.0.smithi125.stdout:561: left oid 40 (ObjNum 230 snap 51 seq_num 230) 2021-03-24T21:56:18.317 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:18.374 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:18.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:18.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:56:18.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:18 smithi125 conmon[24927]: cluster 2021-03-24T21:56:16 2021-03-24T21:56:18.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:18 smithi125 conmon[24927]: .745646+0000 mgr.y (mgr.14144) 365 : cluster [DBG] pgmap v473: 158 pgs: 1 peering, 157 active+clean; 120 MiB data, 454 MiB used, 536 GiB / 536 GiB avail; 28 MiB/s rd, 2.7 MiB/s wr, 34 op/s; 5.9 MiB/s, 15 objects/s recovering 2021-03-24T21:56:18.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:18 smithi125 conmon[24927]: cluster 2021-03-24T21:56:17 2021-03-24T21:56:18.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:18 smithi125 conmon[24927]: .168374+0000 mon.a (mon.0) 2021-03-24T21:56:18.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:18 smithi125 conmon[24927]: 812 : cluster [DBG] osdmap e208: 8 total, 8 up, 6 in 2021-03-24T21:56:18.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:18 smithi125 conmon[19936]: cluster 2021-03-24T21:56:18.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:18 smithi125 conmon[19936]: 2021-03-24T21 2021-03-24T21:56:18.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:18 smithi125 conmon[19936]: :56:16.745646+0000 mgr.y (mgr.14144) 365 : cluster [DBG] pgmap v473: 158 pgs: 1 peering, 157 active+clean; 120 MiB data, 454 MiB used, 536 GiB / 536 GiB avail; 28 MiB/s rd, 2.7 MiB/s wr, 34 op/s; 5.9 MiB/s, 15 objects/s recovering 2021-03-24T21:56:18.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:18 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:56:18.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:18 smithi125 conmon[19936]: -03-24T21:56:17.168374+0000 mon.a (mon 2021-03-24T21:56:18.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:18 smithi125 conmon[19936]: .0) 812 : cluster [DBG] osdmap e208: 8 total, 8 up, 6 in 2021-03-24T21:56:18.428 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:56:18.480 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:18.536 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:18.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:18.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:56:18.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:18 smithi142 conmon[20125]: cluster 2021-03-24T21:56:16.745646+0000 mgr.y ( 2021-03-24T21:56:18.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:18 smithi142 conmon[20125]: mgr.14144) 365 : cluster [DBG] pgmap v473: 158 pgs: 1 peering, 157 active+clean; 120 MiB data, 454 MiB used, 536 GiB / 536 GiB avail; 28 MiB/s rd, 2.7 MiB/s wr, 34 op/s; 5.9 MiB/s, 15 objects/s recovering 2021-03-24T21:56:18.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:18 smithi142 conmon[20125]: cluster 2021-03-24T21:56:17.168374+0000 mon.a (mon.0) 812 : cluster [DBG] osdmap e208: 8 total, 8 up, 6 in 2021-03-24T21:56:18.643 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:18.702 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:18.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:18.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:56:18.809 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:18.863 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:18.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:18.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:56:18.921 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:56:18.968 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:19.028 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:19.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:19.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:56:19.134 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:19.187 INFO:tasks.rados.rados.0.smithi125.stdout:560: done (2 left) 2021-03-24T21:56:19.187 INFO:tasks.rados.rados.0.smithi125.stdout:561: done (1 left) 2021-03-24T21:56:19.188 INFO:tasks.rados.rados.0.smithi125.stdout:562: done (0 left) 2021-03-24T21:56:19.188 INFO:tasks.rados.rados.0.smithi125.stdout:563: read oid 22 snap 51 2021-03-24T21:56:19.188 INFO:tasks.rados.rados.0.smithi125.stdout:563: expect (ObjNum 226 snap 50 seq_num 226) 2021-03-24T21:56:19.188 INFO:tasks.rados.rados.0.smithi125.stdout:564: rollback oid 7 current snap is 52 2021-03-24T21:56:19.188 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 7 to 51 2021-03-24T21:56:19.189 INFO:tasks.rados.rados.0.smithi125.stdout:565: copy_from oid 28 from oid 40 current snap is 52 2021-03-24T21:56:19.189 INFO:tasks.rados.rados.0.smithi125.stdout:566: copy_from oid 26 from oid 2 current snap is 52 2021-03-24T21:56:19.189 INFO:tasks.rados.rados.0.smithi125.stdout:567: read oid 17 snap -1 2021-03-24T21:56:19.189 INFO:tasks.rados.rados.0.smithi125.stdout:567: expect (ObjNum 192 snap 42 seq_num 192) 2021-03-24T21:56:19.190 INFO:tasks.rados.rados.0.smithi125.stdout:568: rollback oid 50 current snap is 52 2021-03-24T21:56:19.190 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 50 to 51 2021-03-24T21:56:19.190 INFO:tasks.rados.rados.0.smithi125.stdout:569: write (excl) oid 49 current snap is 52 2021-03-24T21:56:19.190 INFO:tasks.rados.rados.0.smithi125.stdout:569: seq_num 231 ranges {682417=473109,1567811=602063,2844123=621482} 2021-03-24T21:56:19.195 INFO:tasks.rados.rados.0.smithi125.stdout:569: writing smithi12556470-49 from 682417 to 1155526 tid 1 2021-03-24T21:56:19.197 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:19.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:19.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:56:19.202 INFO:tasks.rados.rados.0.smithi125.stdout:569: writing smithi12556470-49 from 1567811 to 2169874 tid 2 2021-03-24T21:56:19.212 INFO:tasks.rados.rados.0.smithi125.stdout:569: writing smithi12556470-49 from 2844123 to 3465605 tid 3 2021-03-24T21:56:19.212 INFO:tasks.rados.rados.0.smithi125.stdout:570: snap_create 2021-03-24T21:56:19.212 INFO:tasks.rados.rados.0.smithi125.stdout:564: finishing rollback tid 0 to smithi12556470-7 2021-03-24T21:56:19.212 INFO:tasks.rados.rados.0.smithi125.stdout:564: finishing rollback tid 1 to smithi12556470-7 2021-03-24T21:56:19.213 INFO:tasks.rados.rados.0.smithi125.stdout:568: finishing rollback tid 1 to smithi12556470-50 2021-03-24T21:56:19.213 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 50 v 0 (ObjNum 1 snap 21916 seq_num 1116485552) dirty dne 2021-03-24T21:56:19.213 INFO:tasks.rados.rados.0.smithi125.stdout:563: expect (ObjNum 226 snap 50 seq_num 226) 2021-03-24T21:56:19.303 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:19.316 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:19 smithi142 conmon[20125]: cluster 2021-03-24T21:56:17.428869+0000 osd.6 (osd.6 2021-03-24T21:56:19.317 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:19 smithi142 conmon[20125]: ) 21 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:56:19.317 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:19 smithi142 conmon[20125]: cluster 2021-03-24T21:56:17.429038+0000 osd.6 (osd.6) 22 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:56:19.317 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:19 smithi142 conmon[20125]: cluster 2021-03-24T21:56:18.173683+0000 mon.a (mon.0) 813 : cluster 2021-03-24T21:56:19.318 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:19 smithi142 conmon[20125]: [DBG] osdmap e209: 8 total, 8 up, 6 in 2021-03-24T21:56:19.360 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:19.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:19.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:56:19.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:19 smithi125 conmon[19936]: cluster 2021-03-24T21:56:17.428869+0000 osd.6 (osd. 2021-03-24T21:56:19.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:19 smithi125 conmon[19936]: 6) 21 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:56:19.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:19 smithi125 conmon[19936]: cluster 2021-03-24T21:56:17.429038+0000 osd.6 (osd.6) 22 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:56:19.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:19 smithi125 conmon[19936]: cluster 2021-03-24T21:56: 2021-03-24T21:56:19.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:19 smithi125 conmon[19936]: 18.173683+0000 mon.a (mon.0) 813 : cluster [DBG] osdmap e209: 8 total, 8 up, 6 in 2021-03-24T21:56:19.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:19 smithi125 conmon[24927]: cluster 2021-03-24T21:56:17.428869+0000 osd.6 (osd 2021-03-24T21:56:19.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:19 smithi125 conmon[24927]: .6) 21 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:56:19.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:19 smithi125 conmon[24927]: cluster 2021-03-24T21:56:17.429038+0000 2021-03-24T21:56:19.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:19 smithi125 conmon[24927]: osd.6 (osd.6) 22 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:56:19.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:19 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:56:19.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:19 smithi125 conmon[24927]: 21:56:18.173683+0000 mon.a (mon.0) 813 : cluster 2021-03-24T21:56:19.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:19 smithi125 conmon[24927]: [DBG] osdmap e209: 8 total, 8 up, 6 in 2021-03-24T21:56:19.429 INFO:tasks.rados.rados.0.smithi125.stdout:567: expect (ObjNum 192 snap 42 seq_num 192) 2021-03-24T21:56:19.466 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:19.528 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:19.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:19.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:56:19.589 INFO:tasks.rados.rados.0.smithi125.stdout:564: finishing rollback tid 2 to smithi12556470-7 2021-03-24T21:56:19.589 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 7 v 142 (ObjNum 150 snap 30 seq_num 150) dirty exists 2021-03-24T21:56:19.589 INFO:tasks.rados.rados.0.smithi125.stdout:566: finishing copy_from to smithi12556470-26 2021-03-24T21:56:19.590 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 26 v 132 (ObjNum 167 snap 36 seq_num 167) dirty exists 2021-03-24T21:56:19.590 INFO:tasks.rados.rados.0.smithi125.stdout:566: finishing copy_from racing read to smithi12556470-26 2021-03-24T21:56:19.590 INFO:tasks.rados.rados.0.smithi125.stdout:569: finishing write tid 1 to smithi12556470-49 2021-03-24T21:56:19.590 INFO:tasks.rados.rados.0.smithi125.stdout:569: finishing write tid 2 to smithi12556470-49 2021-03-24T21:56:19.591 INFO:tasks.rados.rados.0.smithi125.stdout:569: finishing write tid 3 to smithi12556470-49 2021-03-24T21:56:19.591 INFO:tasks.rados.rados.0.smithi125.stdout:569: finishing write tid 4 to smithi12556470-49 2021-03-24T21:56:19.591 INFO:tasks.rados.rados.0.smithi125.stdout:569: finishing write tid 5 to smithi12556470-49 2021-03-24T21:56:19.591 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 49 v 179 (ObjNum 231 snap 52 seq_num 231) dirty exists 2021-03-24T21:56:19.591 INFO:tasks.rados.rados.0.smithi125.stdout:569: left oid 49 (ObjNum 231 snap 52 seq_num 231) 2021-03-24T21:56:19.592 INFO:tasks.rados.rados.0.smithi125.stdout:565: finishing copy_from to smithi12556470-28 2021-03-24T21:56:19.592 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 28 v 168 (ObjNum 230 snap 51 seq_num 230) dirty exists 2021-03-24T21:56:19.592 INFO:tasks.rados.rados.0.smithi125.stdout:565: finishing copy_from racing read to smithi12556470-28 2021-03-24T21:56:19.633 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:19.688 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:19.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:19.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:56:19.793 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:19.857 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:19.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:19.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:56:19.963 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:20.034 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:20.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:20.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:56:20.128 INFO:tasks.rados.rados.0.smithi125.stdout:563: done (7 left) 2021-03-24T21:56:20.129 INFO:tasks.rados.rados.0.smithi125.stdout:564: done (6 left) 2021-03-24T21:56:20.129 INFO:tasks.rados.rados.0.smithi125.stdout:565: done (5 left) 2021-03-24T21:56:20.129 INFO:tasks.rados.rados.0.smithi125.stdout:566: done (4 left) 2021-03-24T21:56:20.129 INFO:tasks.rados.rados.0.smithi125.stdout:567: done (3 left) 2021-03-24T21:56:20.130 INFO:tasks.rados.rados.0.smithi125.stdout:568: done (2 left) 2021-03-24T21:56:20.130 INFO:tasks.rados.rados.0.smithi125.stdout:569: done (1 left) 2021-03-24T21:56:20.130 INFO:tasks.rados.rados.0.smithi125.stdout:570: done (0 left) 2021-03-24T21:56:20.130 INFO:tasks.rados.rados.0.smithi125.stdout:571: rollback oid 25 current snap is 53 2021-03-24T21:56:20.131 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 25 to 51 2021-03-24T21:56:20.131 INFO:tasks.rados.rados.0.smithi125.stdout:572: copy_from oid 49 from oid 36 current snap is 53 2021-03-24T21:56:20.131 INFO:tasks.rados.rados.0.smithi125.stdout:573: delete oid 9 current snap is 53 2021-03-24T21:56:20.132 INFO:tasks.rados.rados.0.smithi125.stdout:571: finishing rollback tid 1 to smithi12556470-25 2021-03-24T21:56:20.132 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 25 v 0 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-03-24T21:56:20.132 INFO:teuthology.orchestra.run.smithi125.stderr:nodeep-scrub is set 2021-03-24T21:56:20.140 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:20.144 INFO:tasks.rados.rados.0.smithi125.stdout:571: done (2 left) 2021-03-24T21:56:20.145 INFO:tasks.rados.rados.0.smithi125.stdout:573: done (1 left) 2021-03-24T21:56:20.145 INFO:tasks.rados.rados.0.smithi125.stdout:574: write oid 27 current snap is 53 2021-03-24T21:56:20.145 INFO:tasks.rados.rados.0.smithi125.stdout:574: seq_num 233 ranges {716031=623607,2084775=605732,3298969=617062} 2021-03-24T21:56:20.145 INFO:tasks.rados.rados.0.smithi125.stdout:574: writing smithi12556470-27 from 716031 to 1339638 tid 1 2021-03-24T21:56:20.152 INFO:tasks.rados.rados.0.smithi125.stdout:574: writing smithi12556470-27 from 2084775 to 2690507 tid 2 2021-03-24T21:56:20.161 INFO:tasks.rados.rados.0.smithi125.stdout:574: writing smithi12556470-27 from 3298969 to 3916031 tid 3 2021-03-24T21:56:20.161 INFO:tasks.rados.rados.0.smithi125.stdout:575: copy_from oid 12 from oid 8 current snap is 53 2021-03-24T21:56:20.161 INFO:tasks.rados.rados.0.smithi125.stdout:574: finishing write tid 1 to smithi12556470-27 2021-03-24T21:56:20.162 INFO:tasks.rados.rados.0.smithi125.stdout:574: finishing write tid 2 to smithi12556470-27 2021-03-24T21:56:20.162 INFO:tasks.rados.rados.0.smithi125.stdout:576: rollback oid 23 current snap is 53 2021-03-24T21:56:20.162 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 23 to 52 2021-03-24T21:56:20.162 INFO:tasks.rados.rados.0.smithi125.stdout:577: read oid 34 snap 52 2021-03-24T21:56:20.162 INFO:tasks.rados.rados.0.smithi125.stdout:577: expect (ObjNum 203 snap 43 seq_num 203) 2021-03-24T21:56:20.163 INFO:tasks.rados.rados.0.smithi125.stdout:578: read oid 3 snap -1 2021-03-24T21:56:20.163 INFO:tasks.rados.rados.0.smithi125.stdout:578: expect (ObjNum 220 snap 49 seq_num 220) 2021-03-24T21:56:20.163 INFO:tasks.rados.rados.0.smithi125.stdout:579: read oid 40 snap -1 2021-03-24T21:56:20.163 INFO:tasks.rados.rados.0.smithi125.stdout:579: expect (ObjNum 230 snap 51 seq_num 230) 2021-03-24T21:56:20.163 INFO:tasks.rados.rados.0.smithi125.stdout:580: delete oid 29 current snap is 53 2021-03-24T21:56:20.167 INFO:tasks.rados.rados.0.smithi125.stdout:578: expect (ObjNum 220 snap 49 seq_num 220) 2021-03-24T21:56:20.210 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:20.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:20.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:56:20.315 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:20.328 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:20 smithi125 conmon[19936]: cluster 2021-03-24T21:56:20.329 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:20 smithi125 conmon[19936]: 2021-03-24T21:56:18.746497+0000 mgr.y ( 2021-03-24T21:56:20.329 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:20 smithi125 conmon[19936]: mgr.14144) 366 : cluster [DBG] pgmap v476: 158 pgs: 1 peering, 157 active+clean; 121 MiB data, 435 MiB used, 536 GiB / 536 GiB avail; 3.9 MiB/s rd, 1.2 MiB/s wr, 6 op/s 2021-03-24T21:56:20.329 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:20 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T21:56:20.329 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:20 smithi125 conmon[19936]: T21:56:19.184799+0000 mon.a (mon 2021-03-24T21:56:20.329 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:20 smithi125 conmon[19936]: .0) 814 : cluster [DBG] osdmap e210: 8 total, 8 up, 6 in 2021-03-24T21:56:20.330 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:20 smithi125 conmon[19936]: audit 2021 2021-03-24T21:56:20.330 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:20 smithi125 conmon[19936]: -03-24T21:56 2021-03-24T21:56:20.330 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:20 smithi125 conmon[19936]: :19.198167+0000 mon.a 2021-03-24T21:56:20.330 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:20 smithi125 conmon[19936]: (mon.0) 815 : audit 2021-03-24T21:56:20.331 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:20 smithi125 conmon[19936]: [INF] from='client.? v1:172.21.15.125:0/1993666047' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:56:20.331 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:20 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:56:20.331 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:20 smithi125 conmon[19936]: 03-24T21:56: 2021-03-24T21:56:20.331 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:20 smithi125 conmon[19936]: 20.112451+0000 mon.a (mon. 2021-03-24T21:56:20.331 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:20 smithi125 conmon[19936]: 0) 816 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:56:20.332 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:20 smithi125 conmon[19936]: audit 2021-03-24T21:56:20.332 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:20 smithi125 conmon[19936]: 2021-03 2021-03-24T21:56:20.332 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:20 smithi125 conmon[19936]: -24T21:56:20.126505+0000 mon.a (mon.0) 817 : audit [INF] from='client.? v1:172.21.15.125:0/1993666047' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T21:56:20.332 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:20 smithi125 conmon[19936]: cluster 2021-03-24T21:56:20.333 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:20 smithi125 conmon[19936]: 2021- 2021-03-24T21:56:20.333 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:20 smithi125 conmon[19936]: 03-24T21:56:20.127033+0000 mon.a (mon.0 2021-03-24T21:56:20.333 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:20 smithi125 conmon[19936]: ) 818 : cluster [DBG] osdmap e211: 8 total, 8 up, 6 in 2021-03-24T21:56:20.333 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:20 smithi125 conmon[24927]: cluster 2021-03-24T21:56:18.746497+0000 mgr.y (mgr.14144) 366 : cluster [DBG] pgmap v476: 158 pgs: 1 peering, 157 active+clean; 121 MiB data, 435 MiB used, 536 GiB / 536 GiB avail; 3.9 MiB/s rd, 1.2 MiB/s wr, 6 op/s 2021-03-24T21:56:20.334 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:20 smithi125 conmon[24927]: cluster 2021-03-24T21:56:19.184799+0000 mon.a (mon.0) 814 : cluster [DBG] osdmap e210: 8 total, 8 up, 6 in 2021-03-24T21:56:20.334 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:20 smithi125 conmon[24927]: audit 2021-03-24T21:56:19.198167+0000 mon.a (mon.0) 815 : audit [INF] from='client.? v1:172.21.15.125:0/1993666047' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:56:20.334 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:20 smithi125 conmon[24927]: cluster 2021-03-24T21:56:20.112451+0000 mon.a (mon.0) 816 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:56:20.334 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:20 smithi125 conmon[24927]: audit 2021-03-24T21:56:20.126505+0000 mon.a (mon.0) 817 : audit [INF] from='client.? v1:172.21.15.125:0/1993666047' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T21:56:20.335 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:20 smithi125 conmon[24927]: cluster 2021-03-24T21:56:20.127033+0000 mon.a (mon.0) 818 : cluster [DBG] osdmap e211: 8 total, 8 up, 6 in 2021-03-24T21:56:20.375 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:20.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:20.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:56:20.395 INFO:tasks.rados.rados.0.smithi125.stdout:572: finishing copy_from to smithi12556470-49 2021-03-24T21:56:20.396 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 49 v 139 (ObjNum 225 snap 50 seq_num 225) dirty exists 2021-03-24T21:56:20.396 INFO:tasks.rados.rados.0.smithi125.stdout:572: finishing copy_from racing read to smithi12556470-49 2021-03-24T21:56:20.396 INFO:tasks.rados.rados.0.smithi125.stdout:579: expect (ObjNum 230 snap 51 seq_num 230) 2021-03-24T21:56:20.480 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:20.533 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:20.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:20.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:56:20.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:20 smithi142 conmon[20125]: cluster 2021-03-24T21:56:18 2021-03-24T21:56:20.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:20 smithi142 conmon[20125]: .746497+0000 mgr.y (mgr.14144) 366 : cluster [DBG] pgmap v476: 158 pgs: 1 peering, 157 active+clean; 121 MiB data, 435 MiB used, 536 GiB / 536 GiB avail; 3.9 MiB/s rd, 1.2 MiB/s wr, 6 op/s 2021-03-24T21:56:20.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:20 smithi142 conmon[20125]: cluster 2021-03-24T21:56:19.184799+0000 2021-03-24T21:56:20.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:20 smithi142 conmon[20125]: mon.a (mon.0) 814 : cluster [DBG] osdmap e210: 8 total, 8 up, 6 in 2021-03-24T21:56:20.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:20 smithi142 conmon[20125]: audit 2021-03-24T21:56 2021-03-24T21:56:20.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:20 smithi142 conmon[20125]: :19.198167+0000 mon.a (mon.0) 815 : 2021-03-24T21:56:20.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:20 smithi142 conmon[20125]: audit [INF] from='client.? v1:172.21.15.125:0/1993666047' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:56:20.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:20 smithi142 conmon[20125]: cluster 2021-03-24T21 2021-03-24T21:56:20.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:20 smithi142 conmon[20125]: :56:20.112451+0000 mon.a (mon.0) 2021-03-24T21:56:20.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:20 smithi142 conmon[20125]: 816 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:56:20.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:20 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:56:20.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:20 smithi142 conmon[20125]: 03-24T21:56:20.126505+0000 mon.a 2021-03-24T21:56:20.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:20 smithi142 conmon[20125]: (mon.0) 817 : audit [INF] from='client.? v1:172.21.15.125:0/1993666047' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T21:56:20.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:20 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:56:20.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:20 smithi142 conmon[20125]: 03-24T21:56:20.127033+0000 mon.a ( 2021-03-24T21:56:20.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:20 smithi142 conmon[20125]: mon.0) 818 : cluster [DBG] osdmap e211: 8 total, 8 up, 6 in 2021-03-24T21:56:20.636 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:20.674 INFO:tasks.rados.rados.0.smithi125.stdout:576: finishing rollback tid 0 to smithi12556470-23 2021-03-24T21:56:20.675 INFO:tasks.rados.rados.0.smithi125.stdout:576: finishing rollback tid 1 to smithi12556470-23 2021-03-24T21:56:20.675 INFO:tasks.rados.rados.0.smithi125.stdout:572: done (7 left) 2021-03-24T21:56:20.675 INFO:tasks.rados.rados.0.smithi125.stdout:578: done (6 left) 2021-03-24T21:56:20.675 INFO:tasks.rados.rados.0.smithi125.stdout:579: done (5 left) 2021-03-24T21:56:20.676 INFO:tasks.rados.rados.0.smithi125.stdout:580: done (4 left) 2021-03-24T21:56:20.676 INFO:tasks.rados.rados.0.smithi125.stdout:581: read oid 29 snap 52 2021-03-24T21:56:20.676 INFO:tasks.rados.rados.0.smithi125.stdout:581: expect (ObjNum 101 snap 20 seq_num 101) 2021-03-24T21:56:20.676 INFO:tasks.rados.rados.0.smithi125.stdout:582: snap_remove snap 51 2021-03-24T21:56:20.690 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:20.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:20.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:56:20.796 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:20.845 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-24T21:56:20.846 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-24T21:56:20.846 INFO:teuthology.orchestra.run.smithi125.stdout: "num_ops": 0 2021-03-24T21:56:20.846 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-24T21:56:20.854 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:20.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:20.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:56:20.961 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:21.004 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:21.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:21.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:56:21.110 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:21.159 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:21.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:21.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:56:21.209 INFO:tasks.rados.rados.0.smithi125.stdout:582: done (5 left) 2021-03-24T21:56:21.209 INFO:tasks.rados.rados.0.smithi125.stdout:583: copy_from oid 50 from oid 9 current snap is 53 2021-03-24T21:56:21.209 INFO:tasks.rados.rados.0.smithi125.stdout:584: write oid 10 current snap is 53 2021-03-24T21:56:21.209 INFO:tasks.rados.rados.0.smithi125.stdout:584: seq_num 235 ranges {749645=774105,2085386=609401,3237462=712183} 2021-03-24T21:56:21.221 INFO:tasks.rados.rados.0.smithi125.stdout:584: writing smithi12556470-10 from 749645 to 1523750 tid 1 2021-03-24T21:56:21.229 INFO:tasks.rados.rados.0.smithi125.stdout:584: writing smithi12556470-10 from 2085386 to 2694787 tid 2 2021-03-24T21:56:21.236 INFO:tasks.rados.rados.0.smithi125.stdout:584: writing smithi12556470-10 from 3237462 to 3949645 tid 3 2021-03-24T21:56:21.236 INFO:tasks.rados.rados.0.smithi125.stdout:585: write oid 18 current snap is 53 2021-03-24T21:56:21.236 INFO:tasks.rados.rados.0.smithi125.stdout:576: finishing rollback tid 2 to smithi12556470-23 2021-03-24T21:56:21.237 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 23 v 181 (ObjNum 213 snap 46 seq_num 213) dirty exists 2021-03-24T21:56:21.237 INFO:tasks.rados.rados.0.smithi125.stdout:577: expect (ObjNum 203 snap 43 seq_num 203) 2021-03-24T21:56:21.266 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:21.319 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:21.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:21.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:56:21.399 INFO:tasks.rados.rados.0.smithi125.stdout:574: finishing write tid 3 to smithi12556470-27 2021-03-24T21:56:21.399 INFO:tasks.rados.rados.0.smithi125.stdout:574: finishing write tid 4 to smithi12556470-27 2021-03-24T21:56:21.400 INFO:tasks.rados.rados.0.smithi125.stdout:574: finishing write tid 5 to smithi12556470-27 2021-03-24T21:56:21.400 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 27 v 213 (ObjNum 233 snap 53 seq_num 233) dirty exists 2021-03-24T21:56:21.400 INFO:tasks.rados.rados.0.smithi125.stdout:574: left oid 27 (ObjNum 233 snap 53 seq_num 233) 2021-03-24T21:56:21.400 INFO:tasks.rados.rados.0.smithi125.stdout:575: finishing copy_from to smithi12556470-12 2021-03-24T21:56:21.400 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 12 v 30 (ObjNum 161 snap 35 seq_num 161) dirty exists 2021-03-24T21:56:21.401 INFO:tasks.rados.rados.0.smithi125.stdout:575: finishing copy_from racing read to smithi12556470-12 2021-03-24T21:56:21.401 INFO:tasks.rados.rados.0.smithi125.stdout:581: expect (ObjNum 101 snap 20 seq_num 101) 2021-03-24T21:56:21.425 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:21.473 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:21.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:21.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:56:21.556 INFO:tasks.rados.rados.0.smithi125.stdout:583: finishing copy_from to smithi12556470-50 2021-03-24T21:56:21.556 INFO:tasks.rados.rados.0.smithi125.stdout:583: got expected ENOENT (src dne) 2021-03-24T21:56:21.556 INFO:tasks.rados.rados.0.smithi125.stdout:583: finishing copy_from racing read to smithi12556470-50 2021-03-24T21:56:21.557 INFO:tasks.rados.rados.0.smithi125.stdout:584: finishing write tid 1 to smithi12556470-10 2021-03-24T21:56:21.557 INFO:tasks.rados.rados.0.smithi125.stdout:584: finishing write tid 2 to smithi12556470-10 2021-03-24T21:56:21.557 INFO:tasks.rados.rados.0.smithi125.stdout:584: finishing write tid 3 to smithi12556470-10 2021-03-24T21:56:21.557 INFO:tasks.rados.rados.0.smithi125.stdout:584: finishing write tid 4 to smithi12556470-10 2021-03-24T21:56:21.558 INFO:tasks.rados.rados.0.smithi125.stdout:584: finishing write tid 5 to smithi12556470-10 2021-03-24T21:56:21.558 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 10 v 202 (ObjNum 235 snap 53 seq_num 235) dirty exists 2021-03-24T21:56:21.558 INFO:tasks.rados.rados.0.smithi125.stdout:584: left oid 10 (ObjNum 235 snap 53 seq_num 235) 2021-03-24T21:56:21.558 INFO:tasks.rados.rados.0.smithi125.stdout:585: seq_num 236 ranges {766452=565707,1860221=469412,2981238=594195} 2021-03-24T21:56:21.573 INFO:tasks.rados.rados.0.smithi125.stdout:585: writing smithi12556470-18 from 766452 to 1332159 tid 1 2021-03-24T21:56:21.579 INFO:tasks.rados.rados.0.smithi125.stdout:585: writing smithi12556470-18 from 1860221 to 2329633 tid 2 2021-03-24T21:56:21.579 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:21.586 INFO:tasks.rados.rados.0.smithi125.stdout:585: writing smithi12556470-18 from 2981238 to 3575433 tid 3 2021-03-24T21:56:21.587 INFO:tasks.rados.rados.0.smithi125.stdout:574: done (7 left) 2021-03-24T21:56:21.587 INFO:tasks.rados.rados.0.smithi125.stdout:575: done (6 left) 2021-03-24T21:56:21.587 INFO:tasks.rados.rados.0.smithi125.stdout:576: done (5 left) 2021-03-24T21:56:21.588 INFO:tasks.rados.rados.0.smithi125.stdout:577: done (4 left) 2021-03-24T21:56:21.588 INFO:tasks.rados.rados.0.smithi125.stdout:581: done (3 left) 2021-03-24T21:56:21.588 INFO:tasks.rados.rados.0.smithi125.stdout:583: done (2 left) 2021-03-24T21:56:21.588 INFO:tasks.rados.rados.0.smithi125.stdout:584: done (1 left) 2021-03-24T21:56:21.589 INFO:tasks.rados.rados.0.smithi125.stdout:586: snap_create 2021-03-24T21:56:21.589 INFO:tasks.rados.rados.0.smithi125.stdout:585: finishing write tid 1 to smithi12556470-18 2021-03-24T21:56:21.589 INFO:tasks.rados.rados.0.smithi125.stdout:585: finishing write tid 2 to smithi12556470-18 2021-03-24T21:56:21.592 INFO:tasks.rados.rados.0.smithi125.stdout:585: finishing write tid 3 to smithi12556470-18 2021-03-24T21:56:21.593 INFO:tasks.rados.rados.0.smithi125.stdout:585: finishing write tid 4 to smithi12556470-18 2021-03-24T21:56:21.593 INFO:tasks.rados.rados.0.smithi125.stdout:585: finishing write tid 5 to smithi12556470-18 2021-03-24T21:56:21.593 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 18 v 108 (ObjNum 236 snap 53 seq_num 236) dirty exists 2021-03-24T21:56:21.594 INFO:tasks.rados.rados.0.smithi125.stdout:585: left oid 18 (ObjNum 236 snap 53 seq_num 236) 2021-03-24T21:56:21.631 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:21.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:21.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:56:21.737 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:21.797 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:21.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:21.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:56:21.903 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:21.962 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:21.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:21.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:56:22.069 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:22.123 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:22.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:22.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:56:22.218 INFO:tasks.rados.rados.0.smithi125.stdout:585: done (1 left) 2021-03-24T21:56:22.316 INFO:tasks.rados.rados.0.smithi125.stdout:586: done (0 left) 2021-03-24T21:56:22.316 INFO:tasks.rados.rados.0.smithi125.stdout:587: rollback oid 20 current snap is 54 2021-03-24T21:56:22.317 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 20 to 52 2021-03-24T21:56:22.317 INFO:tasks.rados.rados.0.smithi125.stdout:588: copy_from oid 5 from oid 34 current snap is 54 2021-03-24T21:56:22.317 INFO:tasks.rados.rados.0.smithi125.stdout:589: read oid 26 snap -1 2021-03-24T21:56:22.317 INFO:tasks.rados.rados.0.smithi125.stdout:589: expect (ObjNum 167 snap 36 seq_num 167) 2021-03-24T21:56:22.318 INFO:tasks.rados.rados.0.smithi125.stdout:590: write (excl) oid 40 current snap is 54 2021-03-24T21:56:22.318 INFO:tasks.rados.rados.0.smithi125.stdout:590: seq_num 237 ranges {783259=640956,1918703=613070,3008661=405467,3821939=161320} 2021-03-24T21:56:22.318 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:22.320 INFO:tasks.rados.rados.0.smithi125.stdout:590: writing smithi12556470-40 from 783259 to 1424215 tid 1 2021-03-24T21:56:22.320 INFO:tasks.rados.rados.0.smithi125.stdout:590: writing smithi12556470-40 from 1918703 to 2531773 tid 2 2021-03-24T21:56:22.320 INFO:tasks.rados.rados.0.smithi125.stdout:590: writing smithi12556470-40 from 3008661 to 3414128 tid 3 2021-03-24T21:56:22.321 INFO:tasks.rados.rados.0.smithi125.stdout:590: writing smithi12556470-40 from 3821939 to 3983259 tid 4 2021-03-24T21:56:22.321 INFO:tasks.rados.rados.0.smithi125.stdout:591: delete oid 46 current snap is 54 2021-03-24T21:56:22.321 INFO:tasks.rados.rados.0.smithi125.stdout:587: finishing rollback tid 0 to smithi12556470-20 2021-03-24T21:56:22.321 INFO:tasks.rados.rados.0.smithi125.stdout:587: finishing rollback tid 1 to smithi12556470-20 2021-03-24T21:56:22.321 INFO:tasks.rados.rados.0.smithi125.stdout:587: finishing rollback tid 2 to smithi12556470-20 2021-03-24T21:56:22.322 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 20 v 149 (ObjNum 200 snap 43 seq_num 200) dirty exists 2021-03-24T21:56:22.322 INFO:tasks.rados.rados.0.smithi125.stdout:589: expect (ObjNum 167 snap 36 seq_num 167) 2021-03-24T21:56:22.335 INFO:tasks.rados.rados.0.smithi125.stdout:590: finishing write tid 1 to smithi12556470-40 2021-03-24T21:56:22.335 INFO:tasks.rados.rados.0.smithi125.stdout:590: finishing write tid 2 to smithi12556470-40 2021-03-24T21:56:22.336 INFO:tasks.rados.rados.0.smithi125.stdout:588: finishing copy_from to smithi12556470-5 2021-03-24T21:56:22.336 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 5 v 193 (ObjNum 203 snap 43 seq_num 203) dirty exists 2021-03-24T21:56:22.336 INFO:tasks.rados.rados.0.smithi125.stdout:588: finishing copy_from racing read to smithi12556470-5 2021-03-24T21:56:22.336 INFO:tasks.rados.rados.0.smithi125.stdout:590: finishing write tid 3 to smithi12556470-40 2021-03-24T21:56:22.336 INFO:tasks.rados.rados.0.smithi125.stdout:590: finishing write tid 4 to smithi12556470-40 2021-03-24T21:56:22.337 INFO:tasks.rados.rados.0.smithi125.stdout:590: finishing write tid 5 to smithi12556470-40 2021-03-24T21:56:22.337 INFO:tasks.rados.rados.0.smithi125.stdout:590: finishing write tid 6 to smithi12556470-40 2021-03-24T21:56:22.337 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 40 v 178 (ObjNum 237 snap 54 seq_num 237) dirty exists 2021-03-24T21:56:22.337 INFO:tasks.rados.rados.0.smithi125.stdout:590: left oid 40 (ObjNum 237 snap 54 seq_num 237) 2021-03-24T21:56:22.337 INFO:tasks.rados.rados.0.smithi125.stdout:587: done (4 left) 2021-03-24T21:56:22.338 INFO:tasks.rados.rados.0.smithi125.stdout:588: done (3 left) 2021-03-24T21:56:22.338 INFO:tasks.rados.rados.0.smithi125.stdout:589: done (2 left) 2021-03-24T21:56:22.338 INFO:tasks.rados.rados.0.smithi125.stdout:590: done (1 left) 2021-03-24T21:56:22.338 INFO:tasks.rados.rados.0.smithi125.stdout:591: done (0 left) 2021-03-24T21:56:22.338 INFO:tasks.rados.rados.0.smithi125.stdout:592: write oid 6 current snap is 54 2021-03-24T21:56:22.339 INFO:tasks.rados.rados.0.smithi125.stdout:592: seq_num 239 ranges {416873=791454,1919314=97559} 2021-03-24T21:56:22.348 INFO:tasks.rados.rados.0.smithi125.stdout:592: writing smithi12556470-6 from 416873 to 1208327 tid 1 2021-03-24T21:56:22.352 INFO:tasks.rados.rados.0.smithi125.stdout:592: writing smithi12556470-6 from 1919314 to 2016873 tid 2 2021-03-24T21:56:22.353 INFO:tasks.rados.rados.0.smithi125.stdout:593: rollback oid 36 current snap is 54 2021-03-24T21:56:22.353 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 36 to 52 2021-03-24T21:56:22.353 INFO:tasks.rados.rados.0.smithi125.stdout:594: snap_create 2021-03-24T21:56:22.354 INFO:tasks.rados.rados.0.smithi125.stdout:592: finishing write tid 1 to smithi12556470-6 2021-03-24T21:56:22.355 INFO:tasks.rados.rados.0.smithi125.stdout:592: finishing write tid 2 to smithi12556470-6 2021-03-24T21:56:22.356 INFO:tasks.rados.rados.0.smithi125.stdout:593: finishing rollback tid 0 to smithi12556470-36 2021-03-24T21:56:22.357 INFO:tasks.rados.rados.0.smithi125.stdout:593: finishing rollback tid 1 to smithi12556470-36 2021-03-24T21:56:22.357 INFO:tasks.rados.rados.0.smithi125.stdout:592: finishing write tid 3 to smithi12556470-6 2021-03-24T21:56:22.358 INFO:tasks.rados.rados.0.smithi125.stdout:593: finishing rollback tid 2 to smithi12556470-36 2021-03-24T21:56:22.358 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 36 v 147 (ObjNum 225 snap 50 seq_num 225) dirty exists 2021-03-24T21:56:22.358 INFO:tasks.rados.rados.0.smithi125.stdout:592: finishing write tid 4 to smithi12556470-6 2021-03-24T21:56:22.358 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 6 v 206 (ObjNum 239 snap 54 seq_num 239) dirty exists 2021-03-24T21:56:22.359 INFO:tasks.rados.rados.0.smithi125.stdout:592: left oid 6 (ObjNum 239 snap 54 seq_num 239) 2021-03-24T21:56:22.380 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:22.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:22.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:56:22.478 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:22 smithi125 conmon[19936]: cluster 2021-03-24T21:56:20. 2021-03-24T21:56:22.478 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:22 smithi125 conmon[19936]: 747169+0000 mgr.y (mgr.14144) 367 : cluster [DBG] pgmap v479: 158 pgs: 158 active+clean; 122 MiB data, 418 MiB used, 536 GiB / 536 GiB avail; 15 MiB/s rd, 6.4 MiB/s wr, 28 op/s; 887 KiB/s, 2 objects/s recovering 2021-03-24T21:56:22.479 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:22 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:56:22.479 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:22 smithi125 conmon[19936]: 56:21.207866+0000 mon.a (mon.0) 819 : cluster [DBG] osdmap e212: 8 total, 8 up, 6 in 2021-03-24T21:56:22.479 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:22 smithi125 conmon[24927]: cluster 2021-03-24T21:56:20. 2021-03-24T21:56:22.479 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:22 smithi125 conmon[24927]: 747169+0000 mgr.y (mgr.14144) 367 : cluster [DBG] pgmap v479: 158 pgs: 158 active+clean; 122 MiB data, 418 MiB used, 536 GiB / 536 GiB avail; 15 MiB/s rd, 6.4 MiB/s wr, 28 op/s; 887 KiB/s, 2 objects/s recovering 2021-03-24T21:56:22.480 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:22 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:56:22.480 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:22 smithi125 conmon[24927]: :56:21.207866+0000 mon.a (mon.0) 819 : cluster [DBG] osdmap e212: 8 total, 8 up, 6 in 2021-03-24T21:56:22.487 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:22.538 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:22.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:22.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:56:22.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:22 smithi142 conmon[20125]: cluster 2021-03-24T21:56:20.747169+0000 mgr.y (mgr.14144) 367 : 2021-03-24T21:56:22.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:22 smithi142 conmon[20125]: cluster [DBG] pgmap v479: 158 pgs: 158 active+clean; 122 MiB data, 418 MiB used, 536 GiB / 536 GiB avail; 15 MiB/s rd, 6.4 MiB/s wr, 28 op/s; 887 KiB/s, 2 objects/s recovering 2021-03-24T21:56:22.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:22 smithi142 conmon[20125]: cluster 2021-03-24T21:56:21.207866+0000 mon.a (mon.0) 2021-03-24T21:56:22.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:22 smithi142 conmon[20125]: 819 : cluster [DBG] osdmap e212: 8 total, 8 up, 6 in 2021-03-24T21:56:22.642 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:22.704 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:22.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:22.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:56:22.811 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:22.864 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:22.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:22.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:56:22.969 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:23.026 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:23.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:23.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:56:23.133 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:23.212 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:23.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:23.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:56:23.228 INFO:tasks.rados.rados.0.smithi125.stdout:592: done (2 left) 2021-03-24T21:56:23.229 INFO:tasks.rados.rados.0.smithi125.stdout:593: done (1 left) 2021-03-24T21:56:23.229 INFO:tasks.rados.rados.0.smithi125.stdout:594: done (0 left) 2021-03-24T21:56:23.229 INFO:tasks.rados.rados.0.smithi125.stdout:595: snap_create 2021-03-24T21:56:23.318 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:23.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:23 smithi142 conmon[20125]: cluster 2021-03-24T21:56:22.217038+0000 mon.a (mon.0) 820 : cluster [DBG] osdmap e213: 8 total, 8 up, 6 in 2021-03-24T21:56:23.379 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:23.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:23.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:56:23.484 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:23.493 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:23 smithi125 conmon[24927]: cluster 2021-03-24T21:56:22.217038+0000 mon.a (mon.0) 820 : cluster [DBG] osdmap e213: 8 total, 8 up, 6 in 2021-03-24T21:56:23.493 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:23 smithi125 conmon[19936]: cluster 2021-03-24T21:56:22.217038+0000 mon.a (mon 2021-03-24T21:56:23.494 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:23 smithi125 conmon[19936]: .0) 820 : cluster [DBG] osdmap e213: 8 total, 8 up, 6 in 2021-03-24T21:56:23.512 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T21:56:23.522 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.0 dump_blocked_ops 2021-03-24T21:56:23.533 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T21:56:23.539 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:23.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:23.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:56:23.556 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T21:56:23.579 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T21:56:23.604 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T21:56:23.657 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:23.659 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T21:56:23.659 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T21:56:23.677 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T21:56:23.680 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-24T21:56:23.680 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-24T21:56:23.680 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-24T21:56:23.680 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-24T21:56:23.680 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-24T21:56:23.681 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-24T21:56:23.681 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-24T21:56:23.681 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-24T21:56:23.717 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:23.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:23.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:56:23.823 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:23.884 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:23.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:23.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:56:23.990 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:24.051 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:24.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:24.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:56:24.155 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:24.230 INFO:tasks.rados.rados.0.smithi125.stdout:595: done (0 left) 2021-03-24T21:56:24.231 INFO:tasks.rados.rados.0.smithi125.stdout:596: rollback oid 7 current snap is 56 2021-03-24T21:56:24.231 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 7 to 54 2021-03-24T21:56:24.231 INFO:tasks.rados.rados.0.smithi125.stdout:597: write oid 4 current snap is 56 2021-03-24T21:56:24.231 INFO:tasks.rados.rados.0.smithi125.stdout:597: seq_num 240 ranges {433680=466703,1577796=455884} 2021-03-24T21:56:24.239 INFO:tasks.rados.rados.0.smithi125.stdout:597: writing smithi12556470-4 from 433680 to 900383 tid 1 2021-03-24T21:56:24.248 INFO:tasks.rados.rados.0.smithi125.stdout:597: writing smithi12556470-4 from 1577796 to 2033680 tid 2 2021-03-24T21:56:24.248 INFO:tasks.rados.rados.0.smithi125.stdout:598: rollback oid 25 current snap is 56 2021-03-24T21:56:24.248 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 25 to 54 2021-03-24T21:56:24.248 INFO:tasks.rados.rados.0.smithi125.stdout:596: finishing rollback tid 0 to smithi12556470-7 2021-03-24T21:56:24.249 INFO:tasks.rados.rados.0.smithi125.stdout:596: finishing rollback tid 1 to smithi12556470-7 2021-03-24T21:56:24.249 INFO:tasks.rados.rados.0.smithi125.stdout:596: finishing rollback tid 2 to smithi12556470-7 2021-03-24T21:56:24.249 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 7 v 151 (ObjNum 150 snap 30 seq_num 150) dirty exists 2021-03-24T21:56:24.249 INFO:tasks.rados.rados.0.smithi125.stdout:596: done (2 left) 2021-03-24T21:56:24.249 INFO:tasks.rados.rados.0.smithi125.stdout:599: snap_remove snap 53 2021-03-24T21:56:24.250 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:24.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:24.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:56:24.355 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:24.380 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:24 smithi125 conmon[19936]: cluster 2021-03-24T21:56 2021-03-24T21:56:24.381 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:24 smithi125 conmon[19936]: :22.747931+0000 mgr.y (mgr.14144) 368 : cluster [DBG] pgmap v482: 158 pgs: 158 active+clean; 122 MiB data, 413 MiB used, 536 GiB / 536 GiB avail; 13 MiB/s rd, 5.5 MiB/s wr, 25 op/s; 887 KiB/s, 2 objects/s recovering 2021-03-24T21:56:24.381 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:24 smithi125 conmon[19936]: cluster 2021-03-24T21:56:23.225227+0000 mon.a (mon 2021-03-24T21:56:24.381 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:24 smithi125 conmon[19936]: .0) 821 : cluster [DBG] osdmap e214: 8 total, 8 up, 6 in 2021-03-24T21:56:24.382 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:24 smithi125 conmon[19936]: audit 2021-03-24T21:56:24.382 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:24 smithi125 conmon[19936]: 2021-03-24T21:56:23.486991+0000 mon.a (mon.0) 822 : audit [DBG] from='client.? v1:172.21.15.125:0/469886325' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:56:24.382 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:24 smithi125 conmon[24927]: cluster 2021-03-24T21:56:22.747931 2021-03-24T21:56:24.382 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:24 smithi125 conmon[24927]: +0000 mgr.y (mgr.14144) 368 : cluster [DBG] pgmap v482: 158 pgs: 158 active+clean; 122 MiB data, 413 MiB used, 536 GiB / 536 GiB avail; 13 MiB/s rd, 5.5 MiB/s wr, 25 op/s; 887 KiB/s, 2 objects/s recovering 2021-03-24T21:56:24.383 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:24 smithi125 conmon[24927]: cluster 2021-03-24T21:56:23.225227 2021-03-24T21:56:24.383 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:24 smithi125 conmon[24927]: +0000 mon.a (mon.0) 821 : cluster [DBG] osdmap e214: 8 total, 8 up, 6 in 2021-03-24T21:56:24.384 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:24 smithi125 conmon[24927]: audit 2021-03-24T21:56:24.384 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:24 smithi125 conmon[24927]: 2021 2021-03-24T21:56:24.385 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:24 smithi125 conmon[24927]: -03-24T21:56: 2021-03-24T21:56:24.385 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:24 smithi125 conmon[24927]: 23.486991+0000 mon.a (mon.0) 822 : audit [DBG] from='client.? v1:172.21.15.125:0/469886325' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:56:24.416 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:24.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:24.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:56:24.524 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:24.537 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:24 smithi142 conmon[20125]: cluster 2021-03-24T21:56:22.747931+0000 mgr.y (mgr.14144) 368 : cluster [DBG] pgmap v482: 158 pgs: 158 active+clean; 122 MiB data, 413 MiB used, 536 GiB / 536 GiB avail; 13 MiB/s rd, 5.5 MiB/s wr, 25 op/s; 887 KiB/s, 2 objects/s recovering 2021-03-24T21:56:24.538 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:24 smithi142 conmon[20125]: cluster 2021-03-24T21:56:23. 2021-03-24T21:56:24.538 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:24 smithi142 conmon[20125]: 225227+0000 mon.a (mon.0) 821 : cluster [DBG] osdmap e214: 8 total, 8 up, 6 in 2021-03-24T21:56:24.539 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:24 smithi142 conmon[20125]: audit 2021-03-24T21:56:23.486991+0000 mon.a (mon.0) 822 : audit [DBG] from='client.? v1:172.21.15.125:0/469886325' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:56:24.585 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:24.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:24.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:56:24.691 INFO:teuthology.orchestra.run.smithi125.stdout:moved 0 / 474 (0%) 2021-03-24T21:56:24.707 INFO:teuthology.orchestra.run.smithi125.stdout:avg 79 2021-03-24T21:56:24.708 INFO:teuthology.orchestra.run.smithi125.stdout:stddev 6.60808 -> 6.60808 (expected baseline 8.11377) 2021-03-24T21:56:24.708 INFO:teuthology.orchestra.run.smithi125.stdout:min osd.4 with 71 -> 71 pgs (0.898734 -> 0.898734 * mean) 2021-03-24T21:56:24.708 INFO:teuthology.orchestra.run.smithi125.stdout:max osd.7 with 88 -> 88 pgs (1.11392 -> 1.11392 * mean) 2021-03-24T21:56:24.708 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-24T21:56:24.709 INFO:teuthology.orchestra.run.smithi125.stdout:oload 1000 2021-03-24T21:56:24.709 INFO:teuthology.orchestra.run.smithi125.stdout:max_change 1 2021-03-24T21:56:24.709 INFO:teuthology.orchestra.run.smithi125.stdout:max_change_osds 4 2021-03-24T21:56:24.709 INFO:teuthology.orchestra.run.smithi125.stdout:average_utilization 0.0008 2021-03-24T21:56:24.710 INFO:teuthology.orchestra.run.smithi125.stdout:overload_utilization 0.0081 2021-03-24T21:56:24.710 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:24.711 INFO:teuthology.orchestra.run.smithi125.stderr:no change 2021-03-24T21:56:24.712 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:56:24.769 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:24.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:24.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:56:24.875 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:24.929 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:24.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:24.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:56:25.036 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:25.097 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:25.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:25.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:56:25.128 INFO:tasks.rados.rados.0.smithi125.stdout:599: done (2 left) 2021-03-24T21:56:25.128 INFO:tasks.rados.rados.0.smithi125.stdout:600: snap_create 2021-03-24T21:56:25.129 INFO:tasks.rados.rados.0.smithi125.stdout:598: finishing rollback tid 1 to smithi12556470-25 2021-03-24T21:56:25.129 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 25 v 0 (ObjNum 1116646736 snap 21916 seq_num 1116619840) dirty dne 2021-03-24T21:56:25.129 INFO:tasks.rados.rados.0.smithi125.stdout:597: finishing write tid 1 to smithi12556470-4 2021-03-24T21:56:25.129 INFO:tasks.rados.rados.0.smithi125.stdout:597: finishing write tid 2 to smithi12556470-4 2021-03-24T21:56:25.130 INFO:tasks.rados.rados.0.smithi125.stdout:597: finishing write tid 3 to smithi12556470-4 2021-03-24T21:56:25.130 INFO:tasks.rados.rados.0.smithi125.stdout:597: finishing write tid 4 to smithi12556470-4 2021-03-24T21:56:25.130 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 4 v 178 (ObjNum 240 snap 56 seq_num 240) dirty exists 2021-03-24T21:56:25.130 INFO:tasks.rados.rados.0.smithi125.stdout:597: left oid 4 (ObjNum 240 snap 56 seq_num 240) 2021-03-24T21:56:25.203 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:25.261 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:25.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:25.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:56:25.393 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:25.414 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd unset noscrub 2021-03-24T21:56:25.453 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:25.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:25.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:56:25.559 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:25.573 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:25 smithi142 conmon[20125]: cluster 2021-03-24T21:56:24.229253+0000 mon.a (mon.0) 823 : cluster [DBG] 2021-03-24T21:56:25.573 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:25 smithi142 conmon[20125]: osdmap e215: 8 total, 8 up, 6 in 2021-03-24T21:56:25.573 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:25 smithi142 conmon[20125]: cluster 2021-03-24T21:56:25.124049+0000 mon.a (mon.0) 824 : cluster [DBG] osdmap e216: 8 total, 8 up, 6 in 2021-03-24T21:56:25.615 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:25.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:25.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:56:25.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:25 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:56:25.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:25 smithi125 conmon[19936]: 21:56:24.229253+0000 mon.a (mon.0) 823 : cluster [DBG] osdmap e215: 8 total, 8 up, 6 in 2021-03-24T21:56:25.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:25 smithi125 conmon[19936]: cluster 2021-03-24T21:56:25.124049+0000 2021-03-24T21:56:25.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:25 smithi125 conmon[19936]: mon.a (mon.0) 824 : cluster [DBG] osdmap e216: 8 total, 8 up, 6 in 2021-03-24T21:56:25.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:25 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:56:25.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:25 smithi125 conmon[24927]: 21:56:24.229253+0000 mon.a (mon.0) 823 : cluster [DBG] osdmap e215: 8 total, 8 up, 6 in 2021-03-24T21:56:25.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:25 smithi125 conmon[24927]: cluster 2021-03-24T21:56:25.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:25 smithi125 conmon[24927]: 2021-03 2021-03-24T21:56:25.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:25 smithi125 conmon[24927]: -24T21:56:25. 2021-03-24T21:56:25.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:25 smithi125 conmon[24927]: 124049+0000 mon.a (mon.0 2021-03-24T21:56:25.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:25 smithi125 conmon[24927]: ) 824 : cluster [DBG] 2021-03-24T21:56:25.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:25 smithi125 conmon[24927]: osdmap e216: 8 total, 8 up, 6 in 2021-03-24T21:56:25.721 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:25.777 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:25.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:25.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:56:25.815 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'reweight-by-utilization'} 2021-03-24T21:56:25.816 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd reweight-by-utilization 110 0.05 2021-03-24T21:56:25.882 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:25.941 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:25.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:25.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:56:26.048 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:26.102 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:26.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:26.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:56:26.130 INFO:tasks.rados.rados.0.smithi125.stdout:597: done (2 left) 2021-03-24T21:56:26.131 INFO:tasks.rados.rados.0.smithi125.stdout:598: done (1 left) 2021-03-24T21:56:26.131 INFO:tasks.rados.rados.0.smithi125.stdout:600: done (0 left) 2021-03-24T21:56:26.131 INFO:tasks.rados.rados.0.smithi125.stdout:601: read oid 21 snap -1 2021-03-24T21:56:26.132 INFO:tasks.rados.rados.0.smithi125.stdout:601: expect (ObjNum 83 snap 10 seq_num 83) 2021-03-24T21:56:26.132 INFO:tasks.rados.rados.0.smithi125.stdout:602: write (excl) oid 29 current snap is 57 2021-03-24T21:56:26.132 INFO:tasks.rados.rados.0.smithi125.stdout:602: seq_num 241 ranges {450487=541952,1636278=414209} 2021-03-24T21:56:26.136 INFO:tasks.rados.rados.0.smithi125.stdout:602: writing smithi12556470-29 from 450487 to 992439 tid 1 2021-03-24T21:56:26.142 INFO:tasks.rados.rados.0.smithi125.stdout:602: writing smithi12556470-29 from 1636278 to 2050487 tid 2 2021-03-24T21:56:26.143 INFO:tasks.rados.rados.0.smithi125.stdout:603: snap_remove snap 52 2021-03-24T21:56:26.211 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:26.271 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:26.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:26.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:56:26.377 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:26.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:26 smithi125 conmon[24927]: audit 2021-03-24T21:56:24.689110 2021-03-24T21:56:26.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:26 smithi125 conmon[24927]: +0000 mgr.y (mgr.14144) 369 : audit [DBG] from='client.34886 v1:172.21.15.125:0/4072124617' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:56:26.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:26 smithi125 conmon[24927]: cluster 2021-03-24T21:56:24 2021-03-24T21:56:26.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:26 smithi125 conmon[24927]: .748809+0000 mgr.y (mgr.14144) 370 : cluster [DBG] pgmap v485: 158 pgs: 158 active+clean; 141 MiB data, 457 MiB used, 536 GiB / 536 GiB avail; 20 MiB/s rd, 5.9 MiB/s wr, 27 op/s 2021-03-24T21:56:26.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:26 smithi125 conmon[24927]: cluster 2021-03-24T21:56:26.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:26 smithi125 conmon[24927]: 2021-03-24T21:56:26.128823+0000 mon.a (mon.0) 825 : cluster 2021-03-24T21:56:26.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:26 smithi125 conmon[24927]: [DBG] osdmap e217: 8 total, 8 up, 6 in 2021-03-24T21:56:26.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:26 smithi125 conmon[19936]: audit 2021-03-24T21: 2021-03-24T21:56:26.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:26 smithi125 conmon[19936]: 56:24.689110+0000 mgr.y (mgr.14144) 369 : audit [DBG] from='client.34886 v1:172.21.15.125:0/4072124617' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:56:26.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:26 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:56:26.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:26 smithi125 conmon[19936]: 21:56:24.748809+0000 mgr.y (mgr.14144) 370 : cluster [DBG] pgmap v485: 158 pgs: 158 active+clean; 141 MiB data, 457 MiB used, 536 GiB / 536 GiB avail; 20 MiB/s rd, 5.9 MiB/s wr, 27 op/s 2021-03-24T21:56:26.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:26 smithi125 conmon[19936]: cluster 2021-03-24T21:56:26.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:26 smithi125 conmon[19936]: 2021-03-24T21:56:26.128823+0000 mon.a (mon.0 2021-03-24T21:56:26.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:26 smithi125 conmon[19936]: ) 825 : cluster [DBG] osdmap e217: 8 total, 8 up, 6 in 2021-03-24T21:56:26.433 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:26.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:26.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:56:26.540 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:26.598 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:26.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:26.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:56:26.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:26 smithi142 conmon[20125]: audit 2021-03-24T21:56:24.689110+0000 2021-03-24T21:56:26.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:26 smithi142 conmon[20125]: mgr.y (mgr.14144) 369 : audit [DBG] from='client.34886 v1:172.21.15.125:0/4072124617' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:56:26.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:26 smithi142 conmon[20125]: cluster 2021-03-24T21:56:24.748809+0000 mgr.y (mgr.14144 2021-03-24T21:56:26.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:26 smithi142 conmon[20125]: ) 370 : cluster [DBG] pgmap v485: 158 pgs: 158 active+clean; 141 MiB data, 457 MiB used, 536 GiB / 536 GiB avail; 20 MiB/s rd, 5.9 MiB/s wr, 27 op/s 2021-03-24T21:56:26.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:26 smithi142 conmon[20125]: cluster 2021-03-24T21:56:26.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:26 smithi142 conmon[20125]: 2021-03-24T21:56:26.128823+0000 mon.a (mon.0) 825 : cluster 2021-03-24T21:56:26.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:26 smithi142 conmon[20125]: [DBG] osdmap e217: 8 total, 8 up, 6 in 2021-03-24T21:56:26.672 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:56:26.703 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:26.725 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-24T21:56:26.785 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:26.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:26.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:56:26.893 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:26.914 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:56:26.941 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:26.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:26.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:56:27.047 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:27.051 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:56:27.107 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:27.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:27.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:56:27.132 INFO:tasks.rados.rados.0.smithi125.stdout:603: done (2 left) 2021-03-24T21:56:27.133 INFO:tasks.rados.rados.0.smithi125.stdout:604: delete oid 43 current snap is 57 2021-03-24T21:56:27.133 INFO:tasks.rados.rados.0.smithi125.stdout:601: expect (ObjNum 83 snap 10 seq_num 83) 2021-03-24T21:56:27.159 INFO:journalctl@ceph.osd.0.smithi125.stdout:Mar 24 21:56:27 smithi125 conmon[28026]: debug 2021-03-24T21:56:27.160 INFO:journalctl@ceph.osd.0.smithi125.stdout:Mar 24 21:56:27 smithi125 conmon[28026]: 2021-03-24T21:56:27.133+0000 7f61b647b700 -1 osd.0 pg_epoch: 218 pg[7.f( v 216'71 (0'0,216'71] local-lis/les=99/100 n=3 ec=99/99 lis/c=99/99 les/c/f=100/100/0 sis=99) [0,3,7] r=0 lpr=99 crt=216'71 lcod 213'69 mlcod 213'69 active+clean+snaptrim trimq=[69~1] ps=[66~1,68~1,6b~1]] removing snap head 2021-03-24T21:56:27.160 INFO:journalctl@ceph.osd.0.smithi125.stdout:Mar 24 21:56:27 smithi125 conmon[28026]: 2021-03-24T21:56:27.214 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:27.273 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:27.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:27.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:56:27.367 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 21:56:27 smithi142 conmon[27337]: debug 2021-03-24T21:56:27.135+0000 7f549d718700 -1 osd.5 pg_epoch: 218 pg[7.b( v 218'181 (0'0,218'181] local-lis/les=108/109 n=6 ec=99/99 lis/c=108/108 les/c/f=109/109/0 sis=108) [5,2,4] r=0 lpr=108 luod=216'180 crt=216'180 lcod 213'178 mlcod 213'178 active+clean+snaptrim trimq=[69~1] ps=[66~1,68~1,6b~1]] removing snap head 2021-03-24T21:56:27.381 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:27.392 INFO:tasks.rados.rados.0.smithi125.stdout:602: finishing write tid 1 to smithi12556470-29 2021-03-24T21:56:27.393 INFO:tasks.rados.rados.0.smithi125.stdout:602: finishing write tid 2 to smithi12556470-29 2021-03-24T21:56:27.393 INFO:tasks.rados.rados.0.smithi125.stdout:602: finishing write tid 3 to smithi12556470-29 2021-03-24T21:56:27.394 INFO:tasks.rados.rados.0.smithi125.stdout:602: finishing write tid 4 to smithi12556470-29 2021-03-24T21:56:27.395 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 29 v 113 (ObjNum 241 snap 57 seq_num 241) dirty exists 2021-03-24T21:56:27.396 INFO:tasks.rados.rados.0.smithi125.stdout:602: left oid 29 (ObjNum 241 snap 57 seq_num 241) 2021-03-24T21:56:27.396 INFO:tasks.rados.rados.0.smithi125.stdout:601: done (2 left) 2021-03-24T21:56:27.397 INFO:tasks.rados.rados.0.smithi125.stdout:602: done (1 left) 2021-03-24T21:56:27.397 INFO:tasks.rados.rados.0.smithi125.stdout:604: done (0 left) 2021-03-24T21:56:27.397 INFO:tasks.rados.rados.0.smithi125.stdout:605: rollback oid 28 current snap is 57 2021-03-24T21:56:27.398 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 28 to 56 2021-03-24T21:56:27.398 INFO:tasks.rados.rados.0.smithi125.stdout:606: read oid 21 snap 56 2021-03-24T21:56:27.398 INFO:tasks.rados.rados.0.smithi125.stdout:606: expect (ObjNum 83 snap 10 seq_num 83) 2021-03-24T21:56:27.399 INFO:tasks.rados.rados.0.smithi125.stdout:607: write oid 39 current snap is 57 2021-03-24T21:56:27.399 INFO:tasks.rados.rados.0.smithi125.stdout:607: seq_num 243 ranges {484101=692450,1636889=447212} 2021-03-24T21:56:27.403 INFO:tasks.rados.rados.0.smithi125.stdout:607: writing smithi12556470-39 from 484101 to 1176551 tid 1 2021-03-24T21:56:27.409 INFO:tasks.rados.rados.0.smithi125.stdout:607: writing smithi12556470-39 from 1636889 to 2084101 tid 2 2021-03-24T21:56:27.409 INFO:tasks.rados.rados.0.smithi125.stdout:608: snap_remove snap 55 2021-03-24T21:56:27.446 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:27.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:27.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:56:27.552 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:27.608 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:27.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:27.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:56:27.713 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:27.767 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:27.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:27.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:56:27.874 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:27.931 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:27.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:27.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:56:28.037 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:28.099 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:28.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:28.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:56:28.204 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:28.264 INFO:tasks.rados.rados.0.smithi125.stdout:608: done (3 left) 2021-03-24T21:56:28.264 INFO:tasks.rados.rados.0.smithi125.stdout:609: write (excl) oid 11 current snap is 57 2021-03-24T21:56:28.265 INFO:tasks.rados.rados.0.smithi125.stdout:609: seq_num 244 ranges {500908=484052,1411724=689184} 2021-03-24T21:56:28.265 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:28.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:28.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:56:28.273 INFO:tasks.rados.rados.0.smithi125.stdout:609: writing smithi12556470-11 from 500908 to 984960 tid 1 2021-03-24T21:56:28.281 INFO:tasks.rados.rados.0.smithi125.stdout:609: writing smithi12556470-11 from 1411724 to 2100908 tid 2 2021-03-24T21:56:28.282 INFO:tasks.rados.rados.0.smithi125.stdout:610: snap_remove snap 54 2021-03-24T21:56:28.371 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:28.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:28 smithi125 conmon[24927]: cluster 2021-03-24T21:56:26. 2021-03-24T21:56:28.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:28 smithi125 conmon[24927]: 749380+0000 mgr.y (mgr.14144) 371 : cluster [DBG] pgmap v488: 158 pgs: 158 active+clean; 141 MiB data, 465 MiB used, 536 GiB / 536 GiB avail; 19 MiB/s rd, 5.9 MiB/s wr, 25 op/s 2021-03-24T21:56:28.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:28 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:56:28.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:28 smithi125 conmon[24927]: :56:27.127332+0000 mon.a (mon.0) 826 : cluster [DBG] osdmap e218: 8 total, 8 up, 6 in 2021-03-24T21:56:28.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:28 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:56:28.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:28 smithi125 conmon[19936]: :56:26.749380+0000 mgr.y (mgr.14144) 371 : cluster [DBG] pgmap v488: 158 pgs: 158 active+clean; 141 MiB data, 465 MiB used, 536 GiB / 536 GiB avail; 19 MiB/s rd, 5.9 MiB/s wr, 25 op/s 2021-03-24T21:56:28.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:28 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:56:28.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:28 smithi125 conmon[19936]: -24T21:56:27 2021-03-24T21:56:28.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:28 smithi125 conmon[19936]: .127332+0000 mon.a (mon.0) 2021-03-24T21:56:28.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:28 smithi125 conmon[19936]: 826 : cluster [DBG] osdmap e218: 8 total, 8 up, 6 in 2021-03-24T21:56:28.423 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:28.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:28.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:56:28.530 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:28.584 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:28.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:28.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:56:28.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:28 smithi142 conmon[20125]: cluster 2021-03-24T21:56: 2021-03-24T21:56:28.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:28 smithi142 conmon[20125]: 26.749380+0000 mgr.y (mgr.14144) 371 : cluster [DBG] pgmap v488: 158 pgs: 158 active+clean; 141 MiB data, 465 MiB used, 536 GiB / 536 GiB avail; 19 MiB/s rd, 5.9 MiB/s wr, 25 op/s 2021-03-24T21:56:28.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:28 smithi142 conmon[20125]: cluster 2021-03-24T21:56: 2021-03-24T21:56:28.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:28 smithi142 conmon[20125]: 27.127332+0000 mon.a (mon.0) 826 : cluster [DBG] osdmap e218: 8 total, 8 up, 6 in 2021-03-24T21:56:28.689 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:28.744 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:28.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:28.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:56:28.809 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-24T21:56:28.809 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T21:56:28.852 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:28.903 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:28.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:28.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:56:29.009 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:29.066 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:29.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:29.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:56:29.172 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:29.232 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:29.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:29.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:56:29.284 INFO:tasks.rados.rados.0.smithi125.stdout:610: done (4 left) 2021-03-24T21:56:29.284 INFO:tasks.rados.rados.0.smithi125.stdout:611: delete oid 45 current snap is 57 2021-03-24T21:56:29.285 INFO:tasks.rados.rados.0.smithi125.stdout:605: finishing rollback tid 0 to smithi12556470-28 2021-03-24T21:56:29.285 INFO:tasks.rados.rados.0.smithi125.stdout:605: finishing rollback tid 1 to smithi12556470-28 2021-03-24T21:56:29.285 INFO:tasks.rados.rados.0.smithi125.stdout:605: finishing rollback tid 2 to smithi12556470-28 2021-03-24T21:56:29.285 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 28 v 171 (ObjNum 230 snap 51 seq_num 230) dirty exists 2021-03-24T21:56:29.286 INFO:tasks.rados.rados.0.smithi125.stdout:606: expect (ObjNum 83 snap 10 seq_num 83) 2021-03-24T21:56:29.338 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:29.392 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:29.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:29.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:56:29.497 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:29.510 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:29 smithi125 conmon[19936]: cluster 2021-03-24T21:56: 2021-03-24T21:56:29.511 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:29 smithi125 conmon[19936]: 28.261616+0000 mon.a (mon.0) 827 : cluster [DBG] osdmap e219: 8 total, 8 up, 6 in 2021-03-24T21:56:29.511 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:29 smithi125 conmon[24927]: cluster 2021-03-24T21:56:29.511 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:29 smithi125 conmon[24927]: 2021-03-24T21:56:29.512 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:29 smithi125 conmon[24927]: 2021 2021-03-24T21:56:29.512 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:29 smithi125 conmon[24927]: - 2021-03-24T21:56:29.512 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:29 smithi125 conmon[24927]: 03 2021-03-24T21:56:29.512 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:29 smithi125 conmon[24927]: - 2021-03-24T21:56:29.513 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:29 smithi125 conmon[24927]: 24 2021-03-24T21:56:29.513 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:29 smithi125 conmon[24927]: T 2021-03-24T21:56:29.513 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:29 smithi125 conmon[24927]: 21 2021-03-24T21:56:29.513 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:29 smithi125 conmon[24927]: : 2021-03-24T21:56:29.514 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:29 smithi125 conmon[24927]: 56 2021-03-24T21:56:29.514 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:29 smithi125 conmon[24927]: : 2021-03-24T21:56:29.514 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:29 smithi125 conmon[24927]: 28 2021-03-24T21:56:29.514 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:29 smithi125 conmon[24927]: . 2021-03-24T21:56:29.514 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:29 smithi125 conmon[24927]: 261616 2021-03-24T21:56:29.515 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:29 smithi125 conmon[24927]: +0000 2021-03-24T21:56:29.515 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:29 smithi125 conmon[24927]: 2021-03-24T21:56:29.515 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:29 smithi125 conmon[24927]: mon.a 2021-03-24T21:56:29.515 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:29 smithi125 conmon[24927]: ( 2021-03-24T21:56:29.515 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:29 smithi125 conmon[24927]: mon 2021-03-24T21:56:29.516 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:29 smithi125 conmon[24927]: . 2021-03-24T21:56:29.516 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:29 smithi125 conmon[24927]: 0 2021-03-24T21:56:29.516 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:29 smithi125 conmon[24927]: ) 2021-03-24T21:56:29.516 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:29 smithi125 conmon[24927]: 827 2021-03-24T21:56:29.516 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:29 smithi125 conmon[24927]: : 2021-03-24T21:56:29.517 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:29 smithi125 conmon[24927]: cluster 2021-03-24T21:56:29.517 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:29 smithi125 conmon[24927]: 2021-03-24T21:56:29.517 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:29 smithi125 conmon[24927]: [DBG] 2021-03-24T21:56:29.517 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:29 smithi125 conmon[24927]: 2021-03-24T21:56:29.517 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:29 smithi125 conmon[24927]: osdmap e219: 8 total, 8 up, 6 in 2021-03-24T21:56:29.518 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:29 smithi125 conmon[24927]: 2021-03-24T21:56:29.529 INFO:tasks.rados.rados.0.smithi125.stdout:607: finishing write tid 1 to smithi12556470-39 2021-03-24T21:56:29.529 INFO:tasks.rados.rados.0.smithi125.stdout:607: finishing write tid 2 to smithi12556470-39 2021-03-24T21:56:29.530 INFO:tasks.rados.rados.0.smithi125.stdout:607: finishing write tid 3 to smithi12556470-39 2021-03-24T21:56:29.530 INFO:tasks.rados.rados.0.smithi125.stdout:607: finishing write tid 4 to smithi12556470-39 2021-03-24T21:56:29.530 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 39 v 205 (ObjNum 243 snap 57 seq_num 243) dirty exists 2021-03-24T21:56:29.530 INFO:tasks.rados.rados.0.smithi125.stdout:607: left oid 39 (ObjNum 243 snap 57 seq_num 243) 2021-03-24T21:56:29.531 INFO:tasks.rados.rados.0.smithi125.stdout:605: done (4 left) 2021-03-24T21:56:29.531 INFO:tasks.rados.rados.0.smithi125.stdout:606: done (3 left) 2021-03-24T21:56:29.531 INFO:tasks.rados.rados.0.smithi125.stdout:607: done (2 left) 2021-03-24T21:56:29.531 INFO:tasks.rados.rados.0.smithi125.stdout:611: done (1 left) 2021-03-24T21:56:29.531 INFO:tasks.rados.rados.0.smithi125.stdout:612: write oid 1 current snap is 57 2021-03-24T21:56:29.532 INFO:tasks.rados.rados.0.smithi125.stdout:609: finishing write tid 1 to smithi12556470-11 2021-03-24T21:56:29.532 INFO:tasks.rados.rados.0.smithi125.stdout:609: finishing write tid 2 to smithi12556470-11 2021-03-24T21:56:29.532 INFO:tasks.rados.rados.0.smithi125.stdout:609: finishing write tid 3 to smithi12556470-11 2021-03-24T21:56:29.532 INFO:tasks.rados.rados.0.smithi125.stdout:612: seq_num 246 ranges {534522=634550,1812335=322187} 2021-03-24T21:56:29.542 INFO:tasks.rados.rados.0.smithi125.stdout:612: writing smithi12556470-1 from 534522 to 1169072 tid 1 2021-03-24T21:56:29.549 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:29.551 INFO:tasks.rados.rados.0.smithi125.stdout:612: writing smithi12556470-1 from 1812335 to 2134522 tid 2 2021-03-24T21:56:29.551 INFO:tasks.rados.rados.0.smithi125.stdout:613: read oid 12 snap -1 2021-03-24T21:56:29.551 INFO:tasks.rados.rados.0.smithi125.stdout:613: expect (ObjNum 161 snap 35 seq_num 161) 2021-03-24T21:56:29.551 INFO:tasks.rados.rados.0.smithi125.stdout:614: delete oid 2 current snap is 57 2021-03-24T21:56:29.552 INFO:tasks.rados.rados.0.smithi125.stdout:609: finishing write tid 4 to smithi12556470-11 2021-03-24T21:56:29.552 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 11 v 188 (ObjNum 244 snap 57 seq_num 244) dirty exists 2021-03-24T21:56:29.552 INFO:tasks.rados.rados.0.smithi125.stdout:609: left oid 11 (ObjNum 244 snap 57 seq_num 244) 2021-03-24T21:56:29.552 INFO:tasks.rados.rados.0.smithi125.stdout:612: finishing write tid 1 to smithi12556470-1 2021-03-24T21:56:29.553 INFO:tasks.rados.rados.0.smithi125.stdout:609: done (3 left) 2021-03-24T21:56:29.554 INFO:tasks.rados.rados.0.smithi125.stdout:614: done (2 left) 2021-03-24T21:56:29.554 INFO:tasks.rados.rados.0.smithi125.stdout:615: read oid 4 snap -1 2021-03-24T21:56:29.554 INFO:tasks.rados.rados.0.smithi125.stdout:615: expect (ObjNum 240 snap 56 seq_num 240) 2021-03-24T21:56:29.554 INFO:tasks.rados.rados.0.smithi125.stdout:616: delete oid 13 current snap is 57 2021-03-24T21:56:29.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:29.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:56:29.557 INFO:tasks.rados.rados.0.smithi125.stdout:613: expect (ObjNum 161 snap 35 seq_num 161) 2021-03-24T21:56:29.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:29 smithi142 conmon[20125]: cluster 2021-03-24T21:56 2021-03-24T21:56:29.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:29 smithi142 conmon[20125]: :28.261616+0000 mon.a (mon.0) 827 : cluster [DBG] osdmap e219: 8 total, 8 up, 6 in 2021-03-24T21:56:29.657 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:29.713 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:29.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:29.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:56:29.729 INFO:tasks.rados.rados.0.smithi125.stdout:612: finishing write tid 2 to smithi12556470-1 2021-03-24T21:56:29.729 INFO:tasks.rados.rados.0.smithi125.stdout:612: finishing write tid 3 to smithi12556470-1 2021-03-24T21:56:29.729 INFO:tasks.rados.rados.0.smithi125.stdout:612: finishing write tid 4 to smithi12556470-1 2021-03-24T21:56:29.729 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 1 v 189 (ObjNum 246 snap 57 seq_num 246) dirty exists 2021-03-24T21:56:29.730 INFO:tasks.rados.rados.0.smithi125.stdout:612: left oid 1 (ObjNum 246 snap 57 seq_num 246) 2021-03-24T21:56:29.730 INFO:tasks.rados.rados.0.smithi125.stdout:615: expect (ObjNum 240 snap 56 seq_num 240) 2021-03-24T21:56:29.813 INFO:tasks.rados.rados.0.smithi125.stdout:612: done (3 left) 2021-03-24T21:56:29.813 INFO:tasks.rados.rados.0.smithi125.stdout:613: done (2 left) 2021-03-24T21:56:29.813 INFO:tasks.rados.rados.0.smithi125.stdout:615: done (1 left) 2021-03-24T21:56:29.814 INFO:tasks.rados.rados.0.smithi125.stdout:616: done (0 left) 2021-03-24T21:56:29.814 INFO:tasks.rados.rados.0.smithi125.stdout:617: copy_from oid 17 from oid 14 current snap is 57 2021-03-24T21:56:29.814 INFO:tasks.rados.rados.0.smithi125.stdout:618: copy_from oid 30 from oid 7 current snap is 57 2021-03-24T21:56:29.814 INFO:tasks.rados.rados.0.smithi125.stdout:619: snap_remove snap 56 2021-03-24T21:56:29.820 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:29.874 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:29.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:29.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:56:29.980 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:30.037 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:30.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:30.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:56:30.135 INFO:tasks.rados.rados.0.smithi125.stdout:619: done (2 left) 2021-03-24T21:56:30.136 INFO:tasks.rados.rados.0.smithi125.stdout:620: snap_create 2021-03-24T21:56:30.136 INFO:tasks.rados.rados.0.smithi125.stdout:617: finishing copy_from to smithi12556470-17 2021-03-24T21:56:30.136 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 17 v 98 (ObjNum 123 snap 23 seq_num 123) dirty exists 2021-03-24T21:56:30.136 INFO:tasks.rados.rados.0.smithi125.stdout:617: finishing copy_from racing read to smithi12556470-17 2021-03-24T21:56:30.136 INFO:tasks.rados.rados.0.smithi125.stdout:618: finishing copy_from to smithi12556470-30 2021-03-24T21:56:30.137 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 30 v 151 (ObjNum 150 snap 30 seq_num 150) dirty exists 2021-03-24T21:56:30.137 INFO:tasks.rados.rados.0.smithi125.stdout:618: finishing copy_from racing read to smithi12556470-30 2021-03-24T21:56:30.143 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:30.186 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:30.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:30.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:56:30.292 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:30.351 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:30.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:30.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:56:30.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:30 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T21:56:30.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:30 smithi125 conmon[19936]: T21:56:28.750130+0000 mgr.y (mgr.14144) 372 : cluster [DBG] pgmap v491: 158 pgs: 158 active+clean; 135 MiB data, 450 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 459 KiB/s wr, 2 op/s 2021-03-24T21:56:30.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:30 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:56:30.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:30 smithi125 conmon[19936]: 03-24T21:56:29.271123+0000 mon.a ( 2021-03-24T21:56:30.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:30 smithi125 conmon[19936]: mon.0) 828 : cluster [DBG] osdmap e220: 8 total, 8 up, 6 in 2021-03-24T21:56:30.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:30 smithi125 conmon[19936]: cluster 2021-03-24T21:56:30. 2021-03-24T21:56:30.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:30 smithi125 conmon[19936]: 129379+0000 mon.a (mon.0) 829 : cluster [DBG] 2021-03-24T21:56:30.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:30 smithi125 conmon[19936]: osdmap e221: 8 total, 8 up, 6 in 2021-03-24T21:56:30.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:30 smithi125 conmon[24927]: cluster 2021-03-24T21:56:28.750130+0000 2021-03-24T21:56:30.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:30 smithi125 conmon[24927]: mgr.y (mgr.14144) 372 : cluster [DBG] pgmap v491: 158 pgs: 158 active+clean; 135 MiB data, 450 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 459 KiB/s wr, 2 op/s 2021-03-24T21:56:30.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:30 smithi125 conmon[24927]: cluster 2021-03-24T21:56:29.271123 2021-03-24T21:56:30.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:30 smithi125 conmon[24927]: +0000 mon.a (mon.0) 828 : cluster [DBG] osdmap e220: 8 total, 8 up, 6 in 2021-03-24T21:56:30.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:30 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:56:30.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:30 smithi125 conmon[24927]: 03-24T21:56:30.129379+0000 mon.a (mon.0) 829 : cluster [DBG] osdmap e221: 8 total, 8 up, 6 in 2021-03-24T21:56:30.413 INFO:journalctl@ceph.osd.3.smithi125.stdout:Mar 24 21:56:30 smithi125 conmon[37997]: debug 2021-03-24T21:56:30.414 INFO:journalctl@ceph.osd.3.smithi125.stdout:Mar 24 21:56:30 smithi125 conmon[37997]: 2021-03-24T21:56:30.129+0000 7f17173db700 -1 osd.3 pg_epoch: 221 pg[7.3( v 220'219 (0'0,220'219] local-lis/les=198/199 n=3 ec=99/99 lis/c=198/198 les/c/f=199/199/0 sis=198) [3,2,4] r=0 lpr=198 crt=220'219 lcod 218'217 mlcod 218'217 active+clean+snaptrim trimq=[6f~1] ps=[69~1,6b~3]] removing snap head 2021-03-24T21:56:30.414 INFO:journalctl@ceph.osd.3.smithi125.stdout:Mar 24 21:56:30 smithi125 conmon[37997]: 2021-03-24T21:56:30.414 INFO:journalctl@ceph.osd.3.smithi125.stdout:Mar 24 21:56:30 smithi125 conmon[37997]: debug 2021-03-24T21:56:30.414 INFO:journalctl@ceph.osd.3.smithi125.stdout:Mar 24 21:56:30 smithi125 conmon[37997]: 2021-03-24T21:56:30.133+0000 7f171abe2700 -1 osd.3 pg_epoch: 221 pg[7.c( v 220'165 (0'0,220'165] local-lis/les=101/102 n=5 ec=99/99 lis/c=101/101 les/c/f=102/102/0 sis=101) [3,7,2] r=0 lpr=101 crt=220'165 lcod 220'163 mlcod 220'163 active+clean+snaptrim trimq=[6f~1] ps=[69~1,6b~3]] removing snap head 2021-03-24T21:56:30.414 INFO:journalctl@ceph.osd.3.smithi125.stdout:Mar 24 21:56:30 smithi125 conmon[37997]: 2021-03-24T21:56:30.415 INFO:journalctl@ceph.osd.0.smithi125.stdout:Mar 24 21:56:30 smithi125 conmon[28026]: debug 2021-03-24T21:56:30.415 INFO:journalctl@ceph.osd.0.smithi125.stdout:Mar 24 21:56:30 smithi125 conmon[28026]: 2021-03-24T21:56:30.125+0000 7f61b747d700 -1 osd.0 pg_epoch: 221 pg[7.d( v 220'158 (0'0,220'158] local-lis/les=108/109 n=5 ec=99/99 lis/c=108/108 les/c/f=109/109/0 sis=108) [0,3,7] r=0 lpr=108 crt=220'158 lcod 207'156 mlcod 207'156 active+clean+snaptrim trimq=[6f~1] ps=[69~1,6b~3]] removing snap head 2021-03-24T21:56:30.457 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:30.516 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:30.520 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:30 smithi142 conmon[20125]: cluster 2021-03-24T21:56:28.750130+0000 mgr.y (mgr.14144) 372 : 2021-03-24T21:56:30.520 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:30 smithi142 conmon[20125]: cluster [DBG] pgmap v491: 158 pgs: 158 active+clean; 135 MiB data, 450 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 459 KiB/s wr, 2 op/s 2021-03-24T21:56:30.521 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:30 smithi142 conmon[20125]: cluster 2021-03-24T21:56:29.271123+0000 mon.a (mon 2021-03-24T21:56:30.521 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:30 smithi142 conmon[20125]: .0) 828 : cluster [DBG] osdmap e220: 8 total, 8 up, 6 in 2021-03-24T21:56:30.522 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:30 smithi142 conmon[20125]: cluster 2021-03-24T21:56: 2021-03-24T21:56:30.522 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:30 smithi142 conmon[20125]: 30.129379+0000 mon.a (mon.0) 829 : cluster 2021-03-24T21:56:30.523 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:30 smithi142 conmon[20125]: [DBG] osdmap e221: 8 total, 8 up, 6 in 2021-03-24T21:56:30.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:30.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:56:30.626 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:30.685 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:30.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:30.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:56:30.790 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:30.857 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:30.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:30.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:56:30.963 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:31.015 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:31.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:31.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:56:31.121 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:31.145 INFO:tasks.rados.rados.0.smithi125.stdout:617: done (2 left) 2021-03-24T21:56:31.145 INFO:tasks.rados.rados.0.smithi125.stdout:618: done (1 left) 2021-03-24T21:56:31.145 INFO:tasks.rados.rados.0.smithi125.stdout:620: done (0 left) 2021-03-24T21:56:31.146 INFO:tasks.rados.rados.0.smithi125.stdout:621: delete oid 39 current snap is 58 2021-03-24T21:56:31.148 INFO:teuthology.orchestra.run.smithi125.stderr:noscrub is unset 2021-03-24T21:56:31.175 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:31.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:31.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:56:31.180 INFO:tasks.rados.rados.0.smithi125.stdout:621: done (0 left) 2021-03-24T21:56:31.181 INFO:tasks.rados.rados.0.smithi125.stdout:622: snap_create 2021-03-24T21:56:31.280 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:31.340 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:31.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:31.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:56:31.446 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:31.502 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:31.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:31.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:56:31.609 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:31.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:31 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:56:31.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:31 smithi142 conmon[20125]: 56:30.205776+0000 osd.1 (osd.1) 35 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:56:31.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:31 smithi142 conmon[20125]: cluster 2021-03-24T21:56:31.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:31 smithi142 conmon[20125]: 2021-03-24T21:56:30.205922 2021-03-24T21:56:31.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:31 smithi142 conmon[20125]: +0000 osd.1 (osd.1) 36 : cluster 2021-03-24T21:56:31.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:31 smithi142 conmon[20125]: [DBG] purged_snaps scrub ok 2021-03-24T21:56:31.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:31 smithi142 conmon[20125]: audit 2021-03- 2021-03-24T21:56:31.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:31 smithi142 conmon[20125]: 24T21:56:30.540019+0000 mon.a ( 2021-03-24T21:56:31.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:31 smithi142 conmon[20125]: mon.0) 830 : audit [INF] from='client.? v1:172.21.15.125:0/37370780' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T21:56:31.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:31 smithi142 conmon[20125]: cluster 2021-03-24T21:56:31.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:31 smithi142 conmon[20125]: 2021-03-24T21:56:31. 2021-03-24T21:56:31.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:31 smithi142 conmon[20125]: 135056+0000 mon.a (mon.0) 831 : cluster [WRN] 2021-03-24T21:56:31.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:31 smithi142 conmon[20125]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:56:31.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:31 smithi142 conmon[20125]: audit 2021-03-24T21:56 2021-03-24T21:56:31.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:31 smithi142 conmon[20125]: :31.143537+0000 mon.a (mon.0) 2021-03-24T21:56:31.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:31 smithi142 conmon[20125]: 832 : audit [INF] from='client.? v1:172.21.15.125:0/37370780' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T21:56:31.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:31 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:56:31.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:31 smithi142 conmon[20125]: 21:56:31.143681+0000 mon.a (mon 2021-03-24T21:56:31.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:31 smithi142 conmon[20125]: .0) 833 : cluster [DBG] osdmap e222: 8 total, 8 up, 6 in 2021-03-24T21:56:31.622 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:31 smithi125 conmon[24927]: cluster 2021-03-24T21:56: 2021-03-24T21:56:31.622 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:31 smithi125 conmon[24927]: 30.205776+0000 osd.1 (osd.1) 35 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:56:31.622 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:31 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:56:31.622 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:31 smithi125 conmon[24927]: T21:56:30.205922+0000 osd.1 (osd 2021-03-24T21:56:31.623 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:31 smithi125 conmon[24927]: .1) 36 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:56:31.623 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:31 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T21:56:31.623 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:31 smithi125 conmon[24927]: 24T21:56:30.540019+0000 mon.a (mon 2021-03-24T21:56:31.623 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:31 smithi125 conmon[24927]: .0) 830 : audit [INF] from='client.? v1:172.21.15.125:0/37370780' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T21:56:31.624 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:31 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:56:31.624 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:31 smithi125 conmon[24927]: 21:56:31.135056+0000 mon.a (mon. 2021-03-24T21:56:31.624 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:31 smithi125 conmon[24927]: 0) 831 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:56:31.624 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:31 smithi125 conmon[24927]: audit 2021-03-24T21: 2021-03-24T21:56:31.624 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:31 smithi125 conmon[24927]: 56:31.143537+0000 mon.a (mon.0 2021-03-24T21:56:31.625 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:31 smithi125 conmon[24927]: ) 832 : audit [INF] from='client.? v1:172.21.15.125:0/37370780' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T21:56:31.625 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:31 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:56:31.625 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:31 smithi125 conmon[24927]: :56:31.143681+0000 mon.a (mon.0 2021-03-24T21:56:31.625 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:31 smithi125 conmon[24927]: ) 833 : cluster [DBG] osdmap e222: 8 total, 8 up, 6 in 2021-03-24T21:56:31.626 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:31 smithi125 conmon[19936]: cluster 2021-03-24T21:56:30.205776 2021-03-24T21:56:31.626 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:31 smithi125 conmon[19936]: +0000 osd.1 (osd.1) 35 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:56:31.626 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:31 smithi125 conmon[19936]: cluster 2021-03-24T21:56:30.205922+0000 2021-03-24T21:56:31.626 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:31 smithi125 conmon[19936]: osd.1 (osd.1) 36 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:56:31.627 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:31 smithi125 conmon[19936]: audit 2021-03-24T21:56:30.540019+0000 mon.a 2021-03-24T21:56:31.627 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:31 smithi125 conmon[19936]: (mon.0) 830 : audit [INF] from='client.? v1:172.21.15.125:0/37370780' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T21:56:31.627 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:31 smithi125 conmon[19936]: cluster 2021-03-24T21:56:31.135056+0000 mon.a ( 2021-03-24T21:56:31.627 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:31 smithi125 conmon[19936]: mon.0) 831 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:56:31.627 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:31 smithi125 conmon[19936]: audit 2021-03-24T21: 2021-03-24T21:56:31.628 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:31 smithi125 conmon[19936]: 56:31.143537+0000 mon.a (mon.0) 832 : audit [INF] 2021-03-24T21:56:31.628 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:31 smithi125 conmon[19936]: from='client.? v1:172.21.15.125:0/37370780' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T21:56:31.628 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:31 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:56:31.628 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:31 smithi125 conmon[19936]: 03-24T21:56:31.143681+0000 mon.a (mon.0) 833 : cluster 2021-03-24T21:56:31.629 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:31 smithi125 conmon[19936]: [DBG] osdmap e222: 8 total, 8 up, 6 in 2021-03-24T21:56:31.670 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:31.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:31.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:56:31.775 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:31.833 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:31.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:31.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:56:31.938 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:31.991 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:31.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:31.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:56:32.098 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:32.152 INFO:tasks.rados.rados.0.smithi125.stdout:622: done (0 left) 2021-03-24T21:56:32.153 INFO:tasks.rados.rados.0.smithi125.stdout:623: copy_from oid 1 from oid 20 current snap is 59 2021-03-24T21:56:32.153 INFO:tasks.rados.rados.0.smithi125.stdout:624: read oid 48 snap 58 2021-03-24T21:56:32.153 INFO:tasks.rados.rados.0.smithi125.stdout:624: expect (ObjNum 62 snap 4 seq_num 62) 2021-03-24T21:56:32.154 INFO:tasks.rados.rados.0.smithi125.stdout:625: copy_from oid 45 from oid 30 current snap is 59 2021-03-24T21:56:32.154 INFO:tasks.rados.rados.0.smithi125.stdout:626: rollback oid 14 current snap is 59 2021-03-24T21:56:32.154 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 14 to 57 2021-03-24T21:56:32.155 INFO:tasks.rados.rados.0.smithi125.stdout:627: copy_from oid 27 from oid 19 current snap is 59 2021-03-24T21:56:32.155 INFO:tasks.rados.rados.0.smithi125.stdout:628: read oid 2 snap -1 2021-03-24T21:56:32.155 INFO:tasks.rados.rados.0.smithi125.stdout:628: expect deleted 2021-03-24T21:56:32.155 INFO:tasks.rados.rados.0.smithi125.stdout:629: copy_from oid 24 from oid 17 current snap is 59 2021-03-24T21:56:32.156 INFO:tasks.rados.rados.0.smithi125.stdout:630: rollback oid 5 current snap is 59 2021-03-24T21:56:32.156 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 5 to 57 2021-03-24T21:56:32.156 INFO:tasks.rados.rados.0.smithi125.stdout:631: write oid 8 current snap is 59 2021-03-24T21:56:32.156 INFO:tasks.rados.rados.0.smithi125.stdout:631: seq_num 250 ranges {601750=535546,1929910=271840} 2021-03-24T21:56:32.157 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:32.158 INFO:tasks.rados.rados.0.smithi125.stdout:631: writing smithi12556470-8 from 601750 to 1137296 tid 1 2021-03-24T21:56:32.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:32.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:56:32.165 INFO:tasks.rados.rados.0.smithi125.stdout:631: writing smithi12556470-8 from 1929910 to 2201750 tid 2 2021-03-24T21:56:32.165 INFO:tasks.rados.rados.0.smithi125.stdout:632: read oid 41 snap -1 2021-03-24T21:56:32.166 INFO:tasks.rados.rados.0.smithi125.stdout:632: expect (ObjNum 188 snap 41 seq_num 188) 2021-03-24T21:56:32.166 INFO:tasks.rados.rados.0.smithi125.stdout:633: write oid 13 current snap is 59 2021-03-24T21:56:32.166 INFO:tasks.rados.rados.0.smithi125.stdout:626: finishing rollback tid 0 to smithi12556470-14 2021-03-24T21:56:32.166 INFO:tasks.rados.rados.0.smithi125.stdout:633: seq_num 251 ranges {618557=727148,2104745=113812} 2021-03-24T21:56:32.174 INFO:tasks.rados.rados.0.smithi125.stdout:633: writing smithi12556470-13 from 618557 to 1345705 tid 1 2021-03-24T21:56:32.181 INFO:tasks.rados.rados.0.smithi125.stdout:633: writing smithi12556470-13 from 2104745 to 2218557 tid 2 2021-03-24T21:56:32.181 INFO:tasks.rados.rados.0.smithi125.stdout:634: delete oid 42 current snap is 59 2021-03-24T21:56:32.181 INFO:tasks.rados.rados.0.smithi125.stdout:626: finishing rollback tid 1 to smithi12556470-14 2021-03-24T21:56:32.181 INFO:tasks.rados.rados.0.smithi125.stdout:626: finishing rollback tid 2 to smithi12556470-14 2021-03-24T21:56:32.182 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 14 v 189 (ObjNum 123 snap 23 seq_num 123) dirty exists 2021-03-24T21:56:32.182 INFO:tasks.rados.rados.0.smithi125.stdout:631: finishing write tid 1 to smithi12556470-8 2021-03-24T21:56:32.182 INFO:tasks.rados.rados.0.smithi125.stdout:631: finishing write tid 2 to smithi12556470-8 2021-03-24T21:56:32.182 INFO:tasks.rados.rados.0.smithi125.stdout:631: finishing write tid 3 to smithi12556470-8 2021-03-24T21:56:32.183 INFO:tasks.rados.rados.0.smithi125.stdout:631: finishing write tid 4 to smithi12556470-8 2021-03-24T21:56:32.184 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 8 v 33 (ObjNum 250 snap 59 seq_num 250) dirty exists 2021-03-24T21:56:32.184 INFO:tasks.rados.rados.0.smithi125.stdout:631: left oid 8 (ObjNum 250 snap 59 seq_num 250) 2021-03-24T21:56:32.184 INFO:tasks.rados.rados.0.smithi125.stdout:624: expect (ObjNum 62 snap 4 seq_num 62) 2021-03-24T21:56:32.261 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:32.312 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:32.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:32.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:56:32.332 INFO:tasks.rados.rados.0.smithi125.stdout:624: done (11 left) 2021-03-24T21:56:32.332 INFO:tasks.rados.rados.0.smithi125.stdout:626: done (10 left) 2021-03-24T21:56:32.332 INFO:tasks.rados.rados.0.smithi125.stdout:628: done (9 left) 2021-03-24T21:56:32.332 INFO:tasks.rados.rados.0.smithi125.stdout:631: done (8 left) 2021-03-24T21:56:32.333 INFO:tasks.rados.rados.0.smithi125.stdout:634: done (7 left) 2021-03-24T21:56:32.333 INFO:tasks.rados.rados.0.smithi125.stdout:635: copy_from oid 42 from oid 50 current snap is 59 2021-03-24T21:56:32.333 INFO:tasks.rados.rados.0.smithi125.stdout:625: finishing copy_from to smithi12556470-45 2021-03-24T21:56:32.333 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 45 v 151 (ObjNum 150 snap 30 seq_num 150) dirty exists 2021-03-24T21:56:32.334 INFO:tasks.rados.rados.0.smithi125.stdout:625: finishing copy_from racing read to smithi12556470-45 2021-03-24T21:56:32.334 INFO:tasks.rados.rados.0.smithi125.stdout:633: finishing write tid 1 to smithi12556470-13 2021-03-24T21:56:32.334 INFO:tasks.rados.rados.0.smithi125.stdout:632: expect (ObjNum 188 snap 41 seq_num 188) 2021-03-24T21:56:32.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:32 smithi142 conmon[20125]: cluster 2021-03-24T21:56 2021-03-24T21:56:32.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:32 smithi142 conmon[20125]: :30.750798+0000 mgr.y (mgr.14144) 373 : cluster [DBG] pgmap v494: 158 pgs: 158 active+clean; 124 MiB data, 428 MiB used, 536 GiB / 536 GiB avail; 6.8 MiB/s rd, 738 KiB/s wr, 10 op/s 2021-03-24T21:56:32.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:32 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:56:32.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:32 smithi142 conmon[20125]: 56:32.145838+0000 mon.a ( 2021-03-24T21:56:32.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:32 smithi142 conmon[20125]: mon.0) 834 : cluster [DBG] 2021-03-24T21:56:32.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:32 smithi142 conmon[20125]: osdmap e223: 8 total, 8 up, 6 in 2021-03-24T21:56:32.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:32 smithi125 conmon[19936]: cluster 2021-03-24T21:56:32.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:32 smithi125 conmon[19936]: 2021-03-24T21:56:30 2021-03-24T21:56:32.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:32 smithi125 conmon[19936]: .750798+0000 mgr.y (mgr.14144) 373 : cluster [DBG] pgmap v494: 158 pgs: 158 active+clean; 124 MiB data, 428 MiB used, 536 GiB / 536 GiB avail; 6.8 MiB/s rd, 738 KiB/s wr, 10 op/s 2021-03-24T21:56:32.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:32 smithi125 conmon[19936]: cluster 2021-03-24T21:56:32.145838+0000 mon.a (mon.0) 2021-03-24T21:56:32.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:32 smithi125 conmon[19936]: 834 : cluster [DBG] osdmap e223: 8 total, 8 up, 6 in 2021-03-24T21:56:32.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:32 smithi125 conmon[24927]: cluster 2021-03-24T21:56: 2021-03-24T21:56:32.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:32 smithi125 conmon[24927]: 30.750798+0000 mgr.y (mgr.14144) 373 : cluster [DBG] pgmap v494: 158 pgs: 158 active+clean; 124 MiB data, 428 MiB used, 536 GiB / 536 GiB avail; 6.8 MiB/s rd, 738 KiB/s wr, 10 op/s 2021-03-24T21:56:32.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:32 smithi125 conmon[24927]: cluster 2021-03-24T21:56:32.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:32 smithi125 conmon[24927]: 2021-03-24T21: 2021-03-24T21:56:32.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:32 smithi125 conmon[24927]: 56:32.145838+0000 mon.a (mon. 2021-03-24T21:56:32.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:32 smithi125 conmon[24927]: 0) 834 : cluster [DBG] osdmap e223: 8 total, 8 up, 6 in 2021-03-24T21:56:32.417 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:32.476 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:32.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:32.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:56:32.510 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.0 dump_historic_ops 2021-03-24T21:56:32.546 INFO:tasks.rados.rados.0.smithi125.stdout:630: finishing rollback tid 0 to smithi12556470-5 2021-03-24T21:56:32.546 INFO:tasks.rados.rados.0.smithi125.stdout:630: finishing rollback tid 1 to smithi12556470-5 2021-03-24T21:56:32.546 INFO:tasks.rados.rados.0.smithi125.stdout:630: finishing rollback tid 2 to smithi12556470-5 2021-03-24T21:56:32.546 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 5 v 196 (ObjNum 203 snap 43 seq_num 203) dirty exists 2021-03-24T21:56:32.547 INFO:tasks.rados.rados.0.smithi125.stdout:629: finishing copy_from to smithi12556470-24 2021-03-24T21:56:32.547 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 24 v 98 (ObjNum 123 snap 23 seq_num 123) dirty exists 2021-03-24T21:56:32.547 INFO:tasks.rados.rados.0.smithi125.stdout:629: finishing copy_from racing read to smithi12556470-24 2021-03-24T21:56:32.547 INFO:tasks.rados.rados.0.smithi125.stdout:627: finishing copy_from to smithi12556470-27 2021-03-24T21:56:32.548 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 27 v 164 (ObjNum 193 snap 42 seq_num 193) dirty exists 2021-03-24T21:56:32.548 INFO:tasks.rados.rados.0.smithi125.stdout:627: finishing copy_from racing read to smithi12556470-27 2021-03-24T21:56:32.548 INFO:tasks.rados.rados.0.smithi125.stdout:625: done (7 left) 2021-03-24T21:56:32.548 INFO:tasks.rados.rados.0.smithi125.stdout:627: done (6 left) 2021-03-24T21:56:32.548 INFO:tasks.rados.rados.0.smithi125.stdout:629: done (5 left) 2021-03-24T21:56:32.549 INFO:tasks.rados.rados.0.smithi125.stdout:630: done (4 left) 2021-03-24T21:56:32.549 INFO:tasks.rados.rados.0.smithi125.stdout:632: done (3 left) 2021-03-24T21:56:32.549 INFO:tasks.rados.rados.0.smithi125.stdout:636: copy_from oid 16 from oid 18 current snap is 59 2021-03-24T21:56:32.549 INFO:tasks.rados.rados.0.smithi125.stdout:633: finishing write tid 2 to smithi12556470-13 2021-03-24T21:56:32.549 INFO:tasks.rados.rados.0.smithi125.stdout:633: finishing write tid 3 to smithi12556470-13 2021-03-24T21:56:32.550 INFO:tasks.rados.rados.0.smithi125.stdout:633: finishing write tid 4 to smithi12556470-13 2021-03-24T21:56:32.550 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 13 v 226 (ObjNum 251 snap 59 seq_num 251) dirty exists 2021-03-24T21:56:32.550 INFO:tasks.rados.rados.0.smithi125.stdout:633: left oid 13 (ObjNum 251 snap 59 seq_num 251) 2021-03-24T21:56:32.550 INFO:tasks.rados.rados.0.smithi125.stdout:623: finishing copy_from to smithi12556470-1 2021-03-24T21:56:32.551 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 1 v 149 (ObjNum 200 snap 43 seq_num 200) dirty exists 2021-03-24T21:56:32.551 INFO:tasks.rados.rados.0.smithi125.stdout:623: finishing copy_from racing read to smithi12556470-1 2021-03-24T21:56:32.551 INFO:tasks.rados.rados.0.smithi125.stdout:635: finishing copy_from to smithi12556470-42 2021-03-24T21:56:32.552 INFO:tasks.rados.rados.0.smithi125.stdout:635: got expected ENOENT (src dne) 2021-03-24T21:56:32.552 INFO:tasks.rados.rados.0.smithi125.stdout:635: finishing copy_from racing read to smithi12556470-42 2021-03-24T21:56:32.552 INFO:tasks.rados.rados.0.smithi125.stdout:623: done (3 left) 2021-03-24T21:56:32.552 INFO:tasks.rados.rados.0.smithi125.stdout:633: done (2 left) 2021-03-24T21:56:32.552 INFO:tasks.rados.rados.0.smithi125.stdout:635: done (1 left) 2021-03-24T21:56:32.553 INFO:tasks.rados.rados.0.smithi125.stdout:637: read oid 14 snap 57 2021-03-24T21:56:32.553 INFO:tasks.rados.rados.0.smithi125.stdout:637: expect (ObjNum 123 snap 23 seq_num 123) 2021-03-24T21:56:32.553 INFO:tasks.rados.rados.0.smithi125.stdout:638: delete oid 6 current snap is 59 2021-03-24T21:56:32.554 INFO:tasks.rados.rados.0.smithi125.stdout:638: done (2 left) 2021-03-24T21:56:32.554 INFO:tasks.rados.rados.0.smithi125.stdout:639: snap_remove snap 58 2021-03-24T21:56:32.583 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:32.635 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:32.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:32.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:56:32.741 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:32.792 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:32.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:32.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:56:32.898 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:32.953 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:32.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:32.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:56:32.967 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T21:56:32.989 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T21:56:32.991 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:56:33.008 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T21:56:33.029 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T21:56:33.053 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T21:56:33.058 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:33.083 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T21:56:33.110 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T21:56:33.111 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:33.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:33.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:56:33.139 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T21:56:33.140 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-24T21:56:33.140 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-24T21:56:33.141 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-24T21:56:33.141 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-24T21:56:33.141 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-24T21:56:33.142 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-24T21:56:33.142 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-24T21:56:33.142 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-24T21:56:33.217 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:33.275 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:33.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:33.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:56:33.293 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[19936]: cluster 2021-03-24T21:56: 2021-03-24T21:56:33.293 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[19936]: 32.051509+0000 osd.7 (osd.7) 49 : cluster [DBG] 5.9 scrub ok 2021-03-24T21:56:33.294 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:56:33.294 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[19936]: :56:32.141268+0000 osd.2 (osd.2) 2021-03-24T21:56:33.294 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[19936]: 41 : cluster [DBG] 3.1d scrub ok 2021-03-24T21:56:33.295 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[19936]: cluster 2021-03-24T21:56:33.295 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[19936]: 2021-03-24T21:56:33.295 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[19936]: 2021 2021-03-24T21:56:33.295 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[19936]: - 2021-03-24T21:56:33.295 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[19936]: 03 2021-03-24T21:56:33.296 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[19936]: - 2021-03-24T21:56:33.296 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[19936]: 24 2021-03-24T21:56:33.296 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[19936]: T21 2021-03-24T21:56:33.296 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[19936]: :56:32.279113+0000 osd.0 (osd.0) 2021-03-24T21:56:33.297 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[19936]: 58 : cluster [DBG] 3.12 scrub ok 2021-03-24T21:56:33.297 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:56:33.297 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[19936]: -03-24T21:56: 2021-03-24T21:56:33.297 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[19936]: 32.480106+0000 osd.3 2021-03-24T21:56:33.298 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[19936]: (osd.3) 60 2021-03-24T21:56:33.298 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[19936]: : cluster [DBG] 5.13 scrub ok 2021-03-24T21:56:33.298 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[19936]: 2021-03-24T21:56:33.298 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[19936]: cluster 2021-03-24T21:56:33.299 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[19936]: 2021-03-24T21 2021-03-24T21:56:33.299 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[19936]: :56:32.521932 2021-03-24T21:56:33.299 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[19936]: +0000 osd.4 (osd.4 2021-03-24T21:56:33.304 INFO:tasks.rados.rados.0.smithi125.stdout:639: done (2 left) 2021-03-24T21:56:33.304 INFO:tasks.rados.rados.0.smithi125.stdout:640: delete oid 42 current snap is 59 2021-03-24T21:56:33.305 INFO:tasks.rados.rados.0.smithi125.stdout:637: expect (ObjNum 123 snap 23 seq_num 123) 2021-03-24T21:56:33.382 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:33.417 INFO:tasks.rados.rados.0.smithi125.stdout:636: finishing copy_from to smithi12556470-16 2021-03-24T21:56:33.418 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 16 v 108 (ObjNum 236 snap 53 seq_num 236) dirty exists 2021-03-24T21:56:33.418 INFO:tasks.rados.rados.0.smithi125.stdout:636: finishing copy_from racing read to smithi12556470-16 2021-03-24T21:56:33.418 INFO:tasks.rados.rados.0.smithi125.stdout:636: done (2 left) 2021-03-24T21:56:33.418 INFO:tasks.rados.rados.0.smithi125.stdout:637: done (1 left) 2021-03-24T21:56:33.419 INFO:tasks.rados.rados.0.smithi125.stdout:640: done (0 left) 2021-03-24T21:56:33.419 INFO:tasks.rados.rados.0.smithi125.stdout:641: write oid 48 current snap is 59 2021-03-24T21:56:33.419 INFO:tasks.rados.rados.0.smithi125.stdout:641: seq_num 255 ranges {685785=628144,1822320=463465} 2021-03-24T21:56:33.429 INFO:tasks.rados.rados.0.smithi125.stdout:641: writing smithi12556470-48 from 685785 to 1313929 tid 1 2021-03-24T21:56:33.437 INFO:tasks.rados.rados.0.smithi125.stdout:641: writing smithi12556470-48 from 1822320 to 2285785 tid 2 2021-03-24T21:56:33.437 INFO:tasks.rados.rados.0.smithi125.stdout:642: write oid 19 current snap is 59 2021-03-24T21:56:33.437 INFO:tasks.rados.rados.0.smithi125.stdout:642: seq_num 256 ranges {702592=703393,1880802=421790} 2021-03-24T21:56:33.438 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:33.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:33.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:56:33.445 INFO:tasks.rados.rados.0.smithi125.stdout:642: writing smithi12556470-19 from 702592 to 1405985 tid 1 2021-03-24T21:56:33.450 INFO:tasks.rados.rados.0.smithi125.stdout:642: writing smithi12556470-19 from 1880802 to 2302592 tid 2 2021-03-24T21:56:33.451 INFO:tasks.rados.rados.0.smithi125.stdout:643: snap_remove snap 57 2021-03-24T21:56:33.544 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:33.547 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[19936]: ) 59 : cluster [DBG] 2021-03-24T21:56:33.547 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[19936]: purged_snaps scrub starts 2021-03-24T21:56:33.547 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:56:33.548 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[19936]: 24T21:56:32.522257+0000 osd.4 (osd.4) 2021-03-24T21:56:33.548 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[19936]: 60 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:56:33.548 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[19936]: audit 2021-03-24T21:56:33.548 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[19936]: 2021-03-24T21:56:32 2021-03-24T21:56:33.548 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[19936]: .939679+0000 mon.a (mon.0) 835 2021-03-24T21:56:33.549 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[19936]: : audit [DBG] from='client.? v1:172.21.15.125:0/934828162' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:56:33.550 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[24927]: cluster 2021-03-24T21:56:33.550 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[24927]: 2021-03-24T21:56:33.550 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[24927]: 2021 2021-03-24T21:56:33.550 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[24927]: - 2021-03-24T21:56:33.551 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[24927]: 03 2021-03-24T21:56:33.551 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[24927]: - 2021-03-24T21:56:33.551 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[24927]: 24 2021-03-24T21:56:33.551 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[24927]: T 2021-03-24T21:56:33.551 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[24927]: 21 2021-03-24T21:56:33.552 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[24927]: : 2021-03-24T21:56:33.552 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[24927]: 56 2021-03-24T21:56:33.552 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[24927]: : 2021-03-24T21:56:33.552 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[24927]: 32 2021-03-24T21:56:33.552 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[24927]: . 2021-03-24T21:56:33.553 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[24927]: 051509 2021-03-24T21:56:33.553 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[24927]: +0000 2021-03-24T21:56:33.553 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[24927]: 2021-03-24T21:56:33.553 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[24927]: osd.7 2021-03-24T21:56:33.554 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[24927]: ( 2021-03-24T21:56:33.554 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[24927]: osd 2021-03-24T21:56:33.554 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[24927]: . 2021-03-24T21:56:33.554 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[24927]: 7 2021-03-24T21:56:33.554 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[24927]: ) 2021-03-24T21:56:33.555 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[24927]: 49 2021-03-24T21:56:33.555 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[24927]: : 2021-03-24T21:56:33.555 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[24927]: cluster 2021-03-24T21:56:33.555 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[24927]: 2021-03-24T21:56:33.556 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[24927]: [DBG] 2021-03-24T21:56:33.556 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[24927]: 2021-03-24T21:56:33.556 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[24927]: 5.9 scrub ok 2021-03-24T21:56:33.556 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[24927]: 2021-03-24T21:56:33.556 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[24927]: cluster 2021-03-24T21:56:33.557 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[24927]: 2021-03-24T 2021-03-24T21:56:33.557 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[24927]: 21:56:32.141268+0000 osd.2 (osd.2) 41 : cluster [DBG] 3.1d scrub ok 2021-03-24T21:56:33.557 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[24927]: cluster 2021-03-24T21:56:33.557 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[24927]: 2021-03-24T21:56:33.558 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[24927]: 2021-03-24T21:56:32 2021-03-24T21:56:33.558 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[24927]: .279113+0000 osd.0 (osd.0) 58 : cluster [DBG] 3.12 scrub ok 2021-03-24T21:56:33.558 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[24927]: cluster 2021-03-24T21:56 2021-03-24T21:56:33.558 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[24927]: :32.480106+0000 osd.3 (osd.3) 60 : cluster [DBG] 5.13 scrub ok 2021-03-24T21:56:33.558 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[24927]: cluster 2021-03-24T21:56:33.559 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[24927]: 2021-03-24T21:56:32.521932+0000 osd.4 2021-03-24T21:56:33.559 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[24927]: (osd.4) 59 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:56:33.559 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:56:33.559 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[24927]: :56:32.522257+0000 osd.4 (osd.4) 60 2021-03-24T21:56:33.560 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[24927]: : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:56:33.560 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:33 smithi125 conmon[24927]: audit 2021-03-24T21:56:32.939679+0000 mon.a (mon.0) 835 : audit [DBG] from='client.? v1:172.21.15.125:0/934828162' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:56:33.561 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:33 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:56:33.561 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:33 smithi142 conmon[20125]: 24T21:56:32.051509+0000 osd.7 (osd.7) 49 : cluster 2021-03-24T21:56:33.561 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:33 smithi142 conmon[20125]: [DBG] 5.9 scrub ok 2021-03-24T21:56:33.561 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:33 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:56:33.561 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:33 smithi142 conmon[20125]: -03-24T21:56 2021-03-24T21:56:33.562 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:33 smithi142 conmon[20125]: :32.141268+0000 2021-03-24T21:56:33.562 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:33 smithi142 conmon[20125]: osd.2 (osd. 2021-03-24T21:56:33.562 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:33 smithi142 conmon[20125]: 2) 41 : cluster 2021-03-24T21:56:33.562 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:33 smithi142 conmon[20125]: [DBG] 3.1d scrub ok 2021-03-24T21:56:33.563 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:33 smithi142 conmon[20125]: cluster 2021-03-24T21:56:33.563 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:33 smithi142 conmon[20125]: 2021-03-24T21:56 2021-03-24T21:56:33.563 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:33 smithi142 conmon[20125]: :32.279113+0000 osd.0 ( 2021-03-24T21:56:33.563 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:33 smithi142 conmon[20125]: osd.0) 58 : cluster 2021-03-24T21:56:33.563 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:33 smithi142 conmon[20125]: [DBG] 3.12 scrub ok 2021-03-24T21:56:33.564 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:33 smithi142 conmon[20125]: cluster 2021-03-24T21:56:33.564 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:33 smithi142 conmon[20125]: 2021-03-24T21 2021-03-24T21:56:33.564 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:33 smithi142 conmon[20125]: :56:32.480106+0000 2021-03-24T21:56:33.564 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:33 smithi142 conmon[20125]: osd.3 (osd.3) 60 2021-03-24T21:56:33.564 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:33 smithi142 conmon[20125]: : cluster [DBG] 5.13 scrub ok 2021-03-24T21:56:33.565 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:33 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:56:33.565 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:33 smithi142 conmon[20125]: -03-24T21:56 2021-03-24T21:56:33.565 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:33 smithi142 conmon[20125]: :32.521932+0000 osd.4 2021-03-24T21:56:33.565 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:33 smithi142 conmon[20125]: (osd.4) 59 : 2021-03-24T21:56:33.565 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:33 smithi142 conmon[20125]: cluster [DBG] purged_snaps scrub starts 2021-03-24T21:56:33.566 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:33 smithi142 conmon[20125]: cluster 2021-03-24T21:56:33.566 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:33 smithi142 conmon[20125]: 2021-03-24 2021-03-24T21:56:33.566 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:33 smithi142 conmon[20125]: T21:56:32.522257+0000 2021-03-24T21:56:33.566 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:33 smithi142 conmon[20125]: osd.4 (osd.4) 60 : cluster 2021-03-24T21:56:33.567 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:33 smithi142 conmon[20125]: [DBG] purged_snaps scrub ok 2021-03-24T21:56:33.567 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:33 smithi142 conmon[20125]: audit 2021-03-24T21:56:33.567 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:33 smithi142 conmon[20125]: 2021-03-24 2021-03-24T21:56:33.567 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:33 smithi142 conmon[20125]: T21:56:32 2021-03-24T21:56:33.567 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:33 smithi142 conmon[20125]: .939679+0000 mon.a 2021-03-24T21:56:33.568 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:33 smithi142 conmon[20125]: (mon.0) 2021-03-24T21:56:33.568 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:33 smithi142 conmon[20125]: 835 : audit [DBG] 2021-03-24T21:56:33.568 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:33 smithi142 conmon[20125]: from='client.? v1:172.21.15.125:0/934828162' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:56:33.601 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:33.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:33.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:56:33.706 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:33.766 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:33.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:33.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:56:33.873 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:33.927 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:33.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:33.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:56:34.035 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:34.088 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:34.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:34.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:56:34.196 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:34.250 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:34.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:34.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:56:34.305 INFO:tasks.rados.rados.0.smithi125.stdout:641: finishing write tid 1 to smithi12556470-48 2021-03-24T21:56:34.305 INFO:tasks.rados.rados.0.smithi125.stdout:641: finishing write tid 2 to smithi12556470-48 2021-03-24T21:56:34.305 INFO:tasks.rados.rados.0.smithi125.stdout:641: finishing write tid 3 to smithi12556470-48 2021-03-24T21:56:34.305 INFO:tasks.rados.rados.0.smithi125.stdout:641: finishing write tid 4 to smithi12556470-48 2021-03-24T21:56:34.306 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 48 v 199 (ObjNum 255 snap 59 seq_num 255) dirty exists 2021-03-24T21:56:34.306 INFO:tasks.rados.rados.0.smithi125.stdout:641: left oid 48 (ObjNum 255 snap 59 seq_num 255) 2021-03-24T21:56:34.306 INFO:tasks.rados.rados.0.smithi125.stdout:642: finishing write tid 1 to smithi12556470-19 2021-03-24T21:56:34.306 INFO:tasks.rados.rados.0.smithi125.stdout:642: finishing write tid 2 to smithi12556470-19 2021-03-24T21:56:34.307 INFO:tasks.rados.rados.0.smithi125.stdout:642: finishing write tid 3 to smithi12556470-19 2021-03-24T21:56:34.307 INFO:tasks.rados.rados.0.smithi125.stdout:642: finishing write tid 4 to smithi12556470-19 2021-03-24T21:56:34.307 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 19 v 198 (ObjNum 256 snap 59 seq_num 256) dirty exists 2021-03-24T21:56:34.307 INFO:tasks.rados.rados.0.smithi125.stdout:642: left oid 19 (ObjNum 256 snap 59 seq_num 256) 2021-03-24T21:56:34.307 INFO:tasks.rados.rados.0.smithi125.stdout:641: done (2 left) 2021-03-24T21:56:34.308 INFO:tasks.rados.rados.0.smithi125.stdout:642: done (1 left) 2021-03-24T21:56:34.308 INFO:tasks.rados.rados.0.smithi125.stdout:643: done (0 left) 2021-03-24T21:56:34.308 INFO:tasks.rados.rados.0.smithi125.stdout:644: snap_create 2021-03-24T21:56:34.309 INFO:teuthology.orchestra.run.smithi125.stdout:moved 7 / 474 (1.47679%) 2021-03-24T21:56:34.309 INFO:teuthology.orchestra.run.smithi125.stdout:avg 79 2021-03-24T21:56:34.309 INFO:teuthology.orchestra.run.smithi125.stdout:stddev 6.60808 -> 6.1101 (expected baseline 8.11377) 2021-03-24T21:56:34.309 INFO:teuthology.orchestra.run.smithi125.stdout:min osd.4 with 71 -> 74 pgs (0.898734 -> 0.936709 * mean) 2021-03-24T21:56:34.310 INFO:teuthology.orchestra.run.smithi125.stdout:max osd.7 with 88 -> 89 pgs (1.11392 -> 1.12658 * mean) 2021-03-24T21:56:34.310 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-24T21:56:34.310 INFO:teuthology.orchestra.run.smithi125.stdout:oload 110 2021-03-24T21:56:34.310 INFO:teuthology.orchestra.run.smithi125.stdout:max_change 0.05 2021-03-24T21:56:34.310 INFO:teuthology.orchestra.run.smithi125.stdout:max_change_osds 4 2021-03-24T21:56:34.311 INFO:teuthology.orchestra.run.smithi125.stdout:average_utilization 0.0008 2021-03-24T21:56:34.311 INFO:teuthology.orchestra.run.smithi125.stdout:overload_utilization 0.0008 2021-03-24T21:56:34.311 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2 weight 1.0000 -> 0.9500 2021-03-24T21:56:34.312 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7 weight 1.0000 -> 0.9500 2021-03-24T21:56:34.312 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3 weight 1.0000 -> 0.9500 2021-03-24T21:56:34.357 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:34.412 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:34.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:34.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:56:34.485 INFO:journalctl@ceph.osd.0.smithi125.stdout:Mar 24 21:56:34 smithi125 conmon[28026]: debug 2021-03-24T21:56:34.486 INFO:journalctl@ceph.osd.0.smithi125.stdout:Mar 24 21:56:34 smithi125 conmon[28026]: 2021-03-24T21:56:34.305+0000 7f61bc487700 -1 osd.0 pg_epoch: 225 pg[7.13( v 224'210 (0'0,224'210] local-lis/les=203/204 n=4 ec=198/99 lis/c=203/203 les/c/f=204/204/0 sis=203) [0,7,2] r=0 lpr=203 crt=224'210 lcod 223'208 mlcod 223'208 active+clean+snaptrim trimq=[74~1] ps=[69~1,6b~3,6f~1,75~1]] removing snap head 2021-03-24T21:56:34.486 INFO:journalctl@ceph.osd.0.smithi125.stdout:Mar 24 21:56:34 smithi125 conmon[28026]: 2021-03-24T21:56:34.487 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:34 smithi125 conmon[19936]: cluster 2021-03-24T21:56:32. 2021-03-24T21:56:34.487 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:34 smithi125 conmon[19936]: 751386+0000 mgr.y (mgr.14144) 374 : cluster [DBG] pgmap v497: 158 pgs: 158 active+clean; 124 MiB data, 414 MiB used, 536 GiB / 536 GiB avail; 6.8 MiB/s rd, 278 KiB/s wr, 7 op/s 2021-03-24T21:56:34.487 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:34 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:56:34.488 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:34 smithi125 conmon[19936]: 21:56:33.126460+0000 osd.2 (osd.2) 42 : cluster [DBG] 5.d scrub ok 2021-03-24T21:56:34.488 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:34 smithi125 conmon[19936]: cluster 2021-03-24T21:56:34.488 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:34 smithi125 conmon[19936]: 2021-03-24T21:56:33. 2021-03-24T21:56:34.488 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:34 smithi125 conmon[19936]: 297277+0000 mon.a (mon.0) 836 : cluster [DBG] 2021-03-24T21:56:34.489 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:34 smithi125 conmon[19936]: osdmap e224: 8 total, 8 up, 6 in 2021-03-24T21:56:34.489 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:34 smithi125 conmon[19936]: cluster 2021-03-24T21:56: 2021-03-24T21:56:34.489 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:34 smithi125 conmon[19936]: 33.440612+0000 osd.3 (osd.3) 61 : cluster 2021-03-24T21:56:34.490 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:34 smithi125 conmon[19936]: [DBG] 6.1f scrub ok 2021-03-24T21:56:34.490 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:34 smithi125 conmon[19936]: audit 2021-03-24T21:56 2021-03-24T21:56:34.490 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:34 smithi125 conmon[19936]: :33.458248+0000 mgr.y (mgr.14144) 375 : 2021-03-24T21:56:34.490 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:34 smithi125 conmon[19936]: audit [DBG] from='client.14982 v1:172.21.15.125:0/1236249031' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:56:34.490 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:34 smithi125 conmon[19936]: audit 2021-03- 2021-03-24T21:56:34.491 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:34 smithi125 conmon[19936]: 24T21:56:33.459264+0000 mon.a (mon 2021-03-24T21:56:34.491 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:34 smithi125 conmon[19936]: .0) 837 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'2':'62260','3':'62260','7':'62260'}"}]: dispatch 2021-03-24T21:56:34.491 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:34 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:56:34.491 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:34 smithi125 conmon[19936]: 03-24T21:56:34.092218+0000 osd.4 ( 2021-03-24T21:56:34.492 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:34 smithi125 conmon[19936]: osd.4) 61 : cluster [DBG] 5.18 scrub ok 2021-03-24T21:56:34.492 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:34 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:56:34.492 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:34 smithi125 conmon[24927]: :56:32.751386+0000 mgr.y (mgr.14144) 374 : cluster [DBG] pgmap v497: 158 pgs: 158 active+clean; 124 MiB data, 414 MiB used, 536 GiB / 536 GiB avail; 6.8 MiB/s rd, 278 KiB/s wr, 7 op/s 2021-03-24T21:56:34.492 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:34 smithi125 conmon[24927]: cluster 2021-03-24T21:56:34.493 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:34 smithi125 conmon[24927]: 2021-03-24T21: 2021-03-24T21:56:34.493 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:34 smithi125 conmon[24927]: 56:33.126460+0000 osd.2 (osd.2 2021-03-24T21:56:34.493 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:34 smithi125 conmon[24927]: ) 42 : cluster [DBG] 5.d scrub ok 2021-03-24T21:56:34.494 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:34 smithi125 conmon[24927]: cluster 2021-03-24T21:56:34.494 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:34 smithi125 conmon[24927]: 2021-03-24T21 2021-03-24T21:56:34.494 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:34 smithi125 conmon[24927]: :56:33.297277+0000 mon.a (mon 2021-03-24T21:56:34.494 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:34 smithi125 conmon[24927]: .0) 836 : cluster [DBG] osdmap e224: 8 total, 8 up, 6 in 2021-03-24T21:56:34.495 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:34 smithi125 conmon[24927]: 2021-03-24T21:56:34.495 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:34 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:56:34.495 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:34 smithi125 conmon[24927]: 03-24T21:56:33. 2021-03-24T21:56:34.495 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:34 smithi125 conmon[24927]: 440612+0000 osd.3 (osd.3) 61 2021-03-24T21:56:34.496 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:34 smithi125 conmon[24927]: : cluster [DBG] 6.1f scrub ok 2021-03-24T21:56:34.496 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:34 smithi125 conmon[24927]: audit 2021-03-24 2021-03-24T21:56:34.496 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:34 smithi125 conmon[24927]: T21:56:33.458248+0000 mgr.y (mgr.14144) 375 : audit 2021-03-24T21:56:34.496 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:34 smithi125 conmon[24927]: [DBG] from='client.14982 v1:172.21.15.125:0/1236249031' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:56:34.496 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:34 smithi125 conmon[24927]: audit 2021- 2021-03-24T21:56:34.497 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:34 smithi125 conmon[24927]: 03-24T21:56: 2021-03-24T21:56:34.497 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:34 smithi125 conmon[24927]: 33.459264+0000 mon.a ( 2021-03-24T21:56:34.497 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:34 smithi125 conmon[24927]: mon.0) 837 : audit 2021-03-24T21:56:34.497 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:34 smithi125 conmon[24927]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'2':'62260','3':'62260','7':'62260'}"}]: dispatch 2021-03-24T21:56:34.498 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:34 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:56:34.498 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:34 smithi125 conmon[24927]: -24T21:56:34.092218 2021-03-24T21:56:34.498 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:34 smithi125 conmon[24927]: +0000 osd.4 (osd.4) 61 : 2021-03-24T21:56:34.498 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:34 smithi125 conmon[24927]: cluster [DBG] 5.18 scrub ok 2021-03-24T21:56:34.517 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:34.535 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 24 21:56:34 smithi142 conmon[33921]: debug 2021-03-24T21:56:34.319+0000 7fc7b641e700 -1 osd.7 pg_epoch: 225 pg[7.17( v 223'208 (0'0,223'208] local-lis/les=203/204 n=5 ec=198/99 lis/c=203/203 les/c/f=204/204/0 sis=203) [7,5,2] r=0 lpr=203 crt=223'208 lcod 222'207 mlcod 222'207 active+clean+snaptrim trimq=[74~1] ps=[69~1,6b~3,6f~1,75~1]] removing snap head 2021-03-24T21:56:34.536 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:34 smithi142 conmon[20125]: cluster 2021-03-24T21:56 2021-03-24T21:56:34.536 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:34 smithi142 conmon[20125]: :32.751386+0000 mgr.y (mgr.14144) 374 : cluster [DBG] pgmap v497: 158 pgs: 158 active+clean; 124 MiB data, 414 MiB used, 536 GiB / 536 GiB avail; 6.8 MiB/s rd, 278 KiB/s wr, 7 op/s 2021-03-24T21:56:34.536 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:34 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:56:34.537 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:34 smithi142 conmon[20125]: -03-24T21:56:33.126460+0000 osd.2 (osd.2) 42 : cluster [DBG] 2021-03-24T21:56:34.537 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:34 smithi142 conmon[20125]: 5.d scrub ok 2021-03-24T21:56:34.537 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:34 smithi142 conmon[20125]: cluster 2021-03-24T21:56 2021-03-24T21:56:34.537 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:34 smithi142 conmon[20125]: :33.297277+0000 mon.a (mon.0) 836 2021-03-24T21:56:34.538 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:34 smithi142 conmon[20125]: : cluster [DBG] osdmap e224: 8 total, 8 up, 6 in 2021-03-24T21:56:34.538 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:34 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:56:34.538 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:34 smithi142 conmon[20125]: T21:56:33.440612+0000 osd.3 (osd.3 2021-03-24T21:56:34.538 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:34 smithi142 conmon[20125]: ) 61 : cluster [DBG] 6.1f scrub ok 2021-03-24T21:56:34.538 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:34 smithi142 conmon[20125]: audit 2021-03 2021-03-24T21:56:34.539 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:34 smithi142 conmon[20125]: -24T21:56:33.458248+0000 mgr.y ( 2021-03-24T21:56:34.539 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:34 smithi142 conmon[20125]: mgr.14144) 375 : audit [DBG] from='client.14982 v1:172.21.15.125:0/1236249031' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:56:34.539 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:34 smithi142 conmon[20125]: audit 2021-03 2021-03-24T21:56:34.539 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:34 smithi142 conmon[20125]: -24T21:56:33.459264+0000 mon.a ( 2021-03-24T21:56:34.539 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:34 smithi142 conmon[20125]: mon.0) 837 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'2':'62260','3':'62260','7':'62260'}"}]: dispatch 2021-03-24T21:56:34.540 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:34 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:56:34.540 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:34 smithi142 conmon[20125]: 03-24T21:56:34.092218+0000 osd.4 2021-03-24T21:56:34.540 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:34 smithi142 conmon[20125]: (osd.4) 61 : cluster [DBG] 5.18 scrub ok 2021-03-24T21:56:34.574 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:34.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:34.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:56:34.681 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:34.739 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:34.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:34.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:56:34.808 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-24T21:56:34.845 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:34.914 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:34.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:34.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:56:35.021 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:35.083 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:35.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:35.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:56:35.132 INFO:tasks.rados.rados.0.smithi125.stdout:644: done (0 left) 2021-03-24T21:56:35.132 INFO:tasks.rados.rados.0.smithi125.stdout:645: read oid 10 snap 59 2021-03-24T21:56:35.132 INFO:tasks.rados.rados.0.smithi125.stdout:645: expect (ObjNum 235 snap 53 seq_num 235) 2021-03-24T21:56:35.133 INFO:tasks.rados.rados.0.smithi125.stdout:646: write (excl) oid 49 current snap is 60 2021-03-24T21:56:35.133 INFO:tasks.rados.rados.0.smithi125.stdout:646: seq_num 257 ranges {719399=778642,1939284=380115} 2021-03-24T21:56:35.143 INFO:tasks.rados.rados.0.smithi125.stdout:646: writing smithi12556470-49 from 719399 to 1498041 tid 1 2021-03-24T21:56:35.149 INFO:tasks.rados.rados.0.smithi125.stdout:646: writing smithi12556470-49 from 1939284 to 2319399 tid 2 2021-03-24T21:56:35.149 INFO:tasks.rados.rados.0.smithi125.stdout:647: delete oid 32 current snap is 60 2021-03-24T21:56:35.151 INFO:tasks.rados.rados.0.smithi125.stdout:647: done (2 left) 2021-03-24T21:56:35.151 INFO:tasks.rados.rados.0.smithi125.stdout:648: write oid 44 current snap is 60 2021-03-24T21:56:35.152 INFO:tasks.rados.rados.0.smithi125.stdout:648: seq_num 259 ranges {753013=645493,2056248=296765} 2021-03-24T21:56:35.160 INFO:tasks.rados.rados.0.smithi125.stdout:648: writing smithi12556470-44 from 753013 to 1398506 tid 1 2021-03-24T21:56:35.168 INFO:tasks.rados.rados.0.smithi125.stdout:648: writing smithi12556470-44 from 2056248 to 2353013 tid 2 2021-03-24T21:56:35.168 INFO:tasks.rados.rados.0.smithi125.stdout:649: rollback oid 2 current snap is 60 2021-03-24T21:56:35.169 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 2 to 59 2021-03-24T21:56:35.169 INFO:tasks.rados.rados.0.smithi125.stdout:648: finishing write tid 1 to smithi12556470-44 2021-03-24T21:56:35.169 INFO:tasks.rados.rados.0.smithi125.stdout:650: read oid 20 snap -1 2021-03-24T21:56:35.169 INFO:tasks.rados.rados.0.smithi125.stdout:650: expect (ObjNum 200 snap 43 seq_num 200) 2021-03-24T21:56:35.170 INFO:tasks.rados.rados.0.smithi125.stdout:651: copy_from oid 1 from oid 7 current snap is 60 2021-03-24T21:56:35.170 INFO:tasks.rados.rados.0.smithi125.stdout:652: copy_from oid 45 from oid 43 current snap is 60 2021-03-24T21:56:35.170 INFO:tasks.rados.rados.0.smithi125.stdout:649: finishing rollback tid 1 to smithi12556470-2 2021-03-24T21:56:35.170 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 2 v 0 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-03-24T21:56:35.171 INFO:tasks.rados.rados.0.smithi125.stdout:649: done (6 left) 2021-03-24T21:56:35.171 INFO:tasks.rados.rados.0.smithi125.stdout:653: snap_create 2021-03-24T21:56:35.180 INFO:tasks.rados.rados.0.smithi125.stdout:650: expect (ObjNum 200 snap 43 seq_num 200) 2021-03-24T21:56:35.189 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:35.232 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:35.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:35.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:56:35.340 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:35.375 INFO:tasks.rados.rados.0.smithi125.stdout:652: finishing copy_from to smithi12556470-45 2021-03-24T21:56:35.375 INFO:tasks.rados.rados.0.smithi125.stdout:652: got expected ENOENT (src dne) 2021-03-24T21:56:35.376 INFO:tasks.rados.rados.0.smithi125.stdout:652: finishing copy_from racing read to smithi12556470-45 2021-03-24T21:56:35.376 INFO:tasks.rados.rados.0.smithi125.stdout:648: finishing write tid 2 to smithi12556470-44 2021-03-24T21:56:35.376 INFO:tasks.rados.rados.0.smithi125.stdout:648: finishing write tid 3 to smithi12556470-44 2021-03-24T21:56:35.376 INFO:tasks.rados.rados.0.smithi125.stdout:648: finishing write tid 4 to smithi12556470-44 2021-03-24T21:56:35.376 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 44 v 165 (ObjNum 259 snap 60 seq_num 259) dirty exists 2021-03-24T21:56:35.377 INFO:tasks.rados.rados.0.smithi125.stdout:648: left oid 44 (ObjNum 259 snap 60 seq_num 259) 2021-03-24T21:56:35.377 INFO:tasks.rados.rados.0.smithi125.stdout:645: expect (ObjNum 235 snap 53 seq_num 235) 2021-03-24T21:56:35.403 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:35.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:35.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:56:35.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[19936]: cluster 2021-03-24T21:56:35.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[19936]: 2021-03-24T21:56:35.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[19936]: 2021-03-24T21:56 2021-03-24T21:56:35.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[19936]: :34.002897+0000 osd.7 (osd.7) 50 : cluster [DBG] 5.19 scrub ok 2021-03-24T21:56:35.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:56:35.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[19936]: 03-24T21:56:34.298733+0000 mon.a (mon 2021-03-24T21:56:35.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[19936]: .0) 838 : audit 2021-03-24T21:56:35.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[19936]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'2':'62260','3':'62260','7':'62260'}"}]': finished 2021-03-24T21:56:35.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[19936]: cluster 2021-03-24T21:56:35.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[19936]: 2021-03- 2021-03-24T21:56:35.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[19936]: 24T21:56 2021-03-24T21:56:35.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[19936]: :34.298831 2021-03-24T21:56:35.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[19936]: +0000 mon.a (mon 2021-03-24T21:56:35.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[19936]: .0) 839 : 2021-03-24T21:56:35.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[19936]: cluster [DBG] 2021-03-24T21:56:35.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[19936]: osdmap e225: 8 total, 8 up, 6 in 2021-03-24T21:56:35.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[19936]: cluster 2021-03-24T21:56:35.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[19936]: 2021-03- 2021-03-24T21:56:35.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[19936]: 24T21:56: 2021-03-24T21:56:35.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[19936]: 35.045518+0000 osd.4 2021-03-24T21:56:35.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[19936]: (osd.4) 62 2021-03-24T21:56:35.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[19936]: : cluster [DBG] 6.2 scrub ok 2021-03-24T21:56:35.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[19936]: 2021-03-24T21:56:35.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[19936]: cluster 2021-03-24T21:56:35.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[19936]: 2021-03- 2021-03-24T21:56:35.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[19936]: 24T21: 2021-03-24T21:56:35.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[19936]: 56:35.126554+0000 2021-03-24T21:56:35.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[19936]: mon.a (mon.0 2021-03-24T21:56:35.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[19936]: ) 840 : cluster [DBG] 2021-03-24T21:56:35.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[19936]: osdmap e226: 8 total, 8 up, 6 in 2021-03-24T21:56:35.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: cluster 2021-03-24T21:56:35.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 2021-03-24T21:56:35.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 2021 2021-03-24T21:56:35.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: - 2021-03-24T21:56:35.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 03 2021-03-24T21:56:35.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: - 2021-03-24T21:56:35.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 24 2021-03-24T21:56:35.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: T 2021-03-24T21:56:35.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 21 2021-03-24T21:56:35.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: : 2021-03-24T21:56:35.420 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 56 2021-03-24T21:56:35.420 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: : 2021-03-24T21:56:35.420 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 34 2021-03-24T21:56:35.420 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: . 2021-03-24T21:56:35.420 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 002897 2021-03-24T21:56:35.421 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: +0000 2021-03-24T21:56:35.421 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 2021-03-24T21:56:35.421 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: osd.7 2021-03-24T21:56:35.421 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: ( 2021-03-24T21:56:35.421 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: osd 2021-03-24T21:56:35.422 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: . 2021-03-24T21:56:35.422 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 7 2021-03-24T21:56:35.422 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: ) 2021-03-24T21:56:35.422 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 50 2021-03-24T21:56:35.423 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: : 2021-03-24T21:56:35.423 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: cluster 2021-03-24T21:56:35.423 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 2021-03-24T21:56:35.423 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: [DBG] 2021-03-24T21:56:35.423 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 2021-03-24T21:56:35.424 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 5.19 scrub ok 2021-03-24T21:56:35.424 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 2021-03-24T21:56:35.424 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: audit 2021-03-24T21:56:35.424 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 2021-03-24T21:56:35.425 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 2021 2021-03-24T21:56:35.425 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: - 2021-03-24T21:56:35.425 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 03 2021-03-24T21:56:35.425 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: - 2021-03-24T21:56:35.425 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 24 2021-03-24T21:56:35.426 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: T 2021-03-24T21:56:35.426 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 21 2021-03-24T21:56:35.426 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: : 2021-03-24T21:56:35.426 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 56 2021-03-24T21:56:35.426 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: : 2021-03-24T21:56:35.427 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 34 2021-03-24T21:56:35.427 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: . 2021-03-24T21:56:35.427 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 298733 2021-03-24T21:56:35.427 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: +0000 2021-03-24T21:56:35.428 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 2021-03-24T21:56:35.428 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: mon.a 2021-03-24T21:56:35.428 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: ( 2021-03-24T21:56:35.428 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: mon 2021-03-24T21:56:35.428 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: . 2021-03-24T21:56:35.429 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 0 2021-03-24T21:56:35.429 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: ) 2021-03-24T21:56:35.429 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 838 2021-03-24T21:56:35.429 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: : 2021-03-24T21:56:35.429 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: audit 2021-03-24T21:56:35.430 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 2021-03-24T21:56:35.430 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: [INF] 2021-03-24T21:56:35.430 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 2021-03-24T21:56:35.430 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'2':'62260','3':'62260','7':'62260'}"}]': finished 2021-03-24T21:56:35.430 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 2021-03-24T21:56:35.431 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: cluster 2021-03-24T21:56:35.431 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 2021-03-24T21:56:35.431 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 2021 2021-03-24T21:56:35.431 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: - 2021-03-24T21:56:35.432 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 03 2021-03-24T21:56:35.432 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: - 2021-03-24T21:56:35.432 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 24 2021-03-24T21:56:35.432 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: T 2021-03-24T21:56:35.432 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 21 2021-03-24T21:56:35.433 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: : 2021-03-24T21:56:35.433 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 56 2021-03-24T21:56:35.433 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: : 2021-03-24T21:56:35.433 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 34 2021-03-24T21:56:35.433 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: . 2021-03-24T21:56:35.434 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 298831 2021-03-24T21:56:35.434 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: +0000 2021-03-24T21:56:35.434 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 2021-03-24T21:56:35.434 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: mon.a 2021-03-24T21:56:35.435 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: ( 2021-03-24T21:56:35.435 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: mon 2021-03-24T21:56:35.435 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: . 2021-03-24T21:56:35.435 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 0 2021-03-24T21:56:35.435 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: ) 2021-03-24T21:56:35.436 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 839 2021-03-24T21:56:35.436 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: : 2021-03-24T21:56:35.436 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: cluster 2021-03-24T21:56:35.436 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 2021-03-24T21:56:35.436 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: [DBG] 2021-03-24T21:56:35.437 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 2021-03-24T21:56:35.437 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: osdmap e225: 8 total, 8 up, 6 in 2021-03-24T21:56:35.437 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 2021-03-24T21:56:35.437 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: cluster 2021-03-24T21:56:35.438 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 2021-03-24T21:56:35.438 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 2021 2021-03-24T21:56:35.438 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: - 2021-03-24T21:56:35.438 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 03 2021-03-24T21:56:35.438 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: - 2021-03-24T21:56:35.439 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 24 2021-03-24T21:56:35.439 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: T 2021-03-24T21:56:35.439 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 21 2021-03-24T21:56:35.439 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: : 2021-03-24T21:56:35.440 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 56 2021-03-24T21:56:35.440 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: : 2021-03-24T21:56:35.440 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 35 2021-03-24T21:56:35.440 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: . 2021-03-24T21:56:35.440 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 045518 2021-03-24T21:56:35.441 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: +0000 2021-03-24T21:56:35.441 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 2021-03-24T21:56:35.441 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: osd.4 2021-03-24T21:56:35.441 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: ( 2021-03-24T21:56:35.441 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: osd 2021-03-24T21:56:35.442 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: . 2021-03-24T21:56:35.442 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 4 2021-03-24T21:56:35.442 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: ) 2021-03-24T21:56:35.442 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 62 2021-03-24T21:56:35.443 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: : 2021-03-24T21:56:35.443 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: cluster 2021-03-24T21:56:35.443 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 2021-03-24T21:56:35.443 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: [DBG] 2021-03-24T21:56:35.443 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 2021-03-24T21:56:35.444 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 6.2 scrub ok 2021-03-24T21:56:35.444 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 2021-03-24T21:56:35.444 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: cluster 2021-03-24T21:56:35.444 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 2021-03-24T21:56:35.444 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 2021 2021-03-24T21:56:35.445 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: - 2021-03-24T21:56:35.445 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 03 2021-03-24T21:56:35.445 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: - 2021-03-24T21:56:35.445 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 24 2021-03-24T21:56:35.446 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: T 2021-03-24T21:56:35.446 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 21 2021-03-24T21:56:35.446 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: : 2021-03-24T21:56:35.446 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 56 2021-03-24T21:56:35.446 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: : 2021-03-24T21:56:35.447 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 35 2021-03-24T21:56:35.447 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: . 2021-03-24T21:56:35.447 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 126554 2021-03-24T21:56:35.447 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: +0000 2021-03-24T21:56:35.447 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 2021-03-24T21:56:35.448 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: mon.a 2021-03-24T21:56:35.448 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: ( 2021-03-24T21:56:35.448 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: mon 2021-03-24T21:56:35.448 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: . 2021-03-24T21:56:35.449 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 0 2021-03-24T21:56:35.449 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: ) 2021-03-24T21:56:35.449 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 840 2021-03-24T21:56:35.449 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: : 2021-03-24T21:56:35.449 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: cluster 2021-03-24T21:56:35.450 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 2021-03-24T21:56:35.450 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: [DBG] 2021-03-24T21:56:35.450 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 2021-03-24T21:56:35.450 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: osdmap e226: 8 total, 8 up, 6 in 2021-03-24T21:56:35.451 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:35 smithi125 conmon[24927]: 2021-03-24T21:56:35.502 INFO:tasks.rados.rados.0.smithi125.stdout:646: finishing write tid 1 to smithi12556470-49 2021-03-24T21:56:35.503 INFO:tasks.rados.rados.0.smithi125.stdout:646: finishing write tid 2 to smithi12556470-49 2021-03-24T21:56:35.503 INFO:tasks.rados.rados.0.smithi125.stdout:646: finishing write tid 3 to smithi12556470-49 2021-03-24T21:56:35.504 INFO:tasks.rados.rados.0.smithi125.stdout:646: finishing write tid 4 to smithi12556470-49 2021-03-24T21:56:35.504 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 49 v 207 (ObjNum 257 snap 60 seq_num 257) dirty exists 2021-03-24T21:56:35.504 INFO:tasks.rados.rados.0.smithi125.stdout:646: left oid 49 (ObjNum 257 snap 60 seq_num 257) 2021-03-24T21:56:35.505 INFO:tasks.rados.rados.0.smithi125.stdout:651: finishing copy_from to smithi12556470-1 2021-03-24T21:56:35.505 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 1 v 151 (ObjNum 150 snap 30 seq_num 150) dirty exists 2021-03-24T21:56:35.506 INFO:tasks.rados.rados.0.smithi125.stdout:651: finishing copy_from racing read to smithi12556470-1 2021-03-24T21:56:35.510 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:35.559 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:35.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:35.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:56:35.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:35 smithi142 conmon[20125]: cluster 2021-03-24T21:56 2021-03-24T21:56:35.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:35 smithi142 conmon[20125]: :34.002897+0000 osd.7 (osd.7) 50 : cluster [DBG] 5.19 scrub ok 2021-03-24T21:56:35.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:35 smithi142 conmon[20125]: audit 2021-03-24T21:56:34.298733+0000 mon.a (mon 2021-03-24T21:56:35.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:35 smithi142 conmon[20125]: .0) 838 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'2':'62260','3':'62260','7':'62260'}"}]': finished 2021-03-24T21:56:35.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:35 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:56:35.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:35 smithi142 conmon[20125]: -24T21:56:34.298831+0000 mon.a (mon.0) 839 : cluster [DBG] 2021-03-24T21:56:35.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:35 smithi142 conmon[20125]: osdmap e225: 8 total, 8 up, 6 in 2021-03-24T21:56:35.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:35 smithi142 conmon[20125]: cluster 2021-03-24T21:56 2021-03-24T21:56:35.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:35 smithi142 conmon[20125]: :35.045518+0000 osd.4 (osd.4) 62 : 2021-03-24T21:56:35.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:35 smithi142 conmon[20125]: cluster [DBG] 6.2 scrub ok 2021-03-24T21:56:35.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:35 smithi142 conmon[20125]: cluster 2021-03-24T21:56:35. 2021-03-24T21:56:35.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:35 smithi142 conmon[20125]: 126554+0000 mon.a (mon.0) 840 : cluster [DBG] 2021-03-24T21:56:35.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:35 smithi142 conmon[20125]: osdmap e226: 8 total, 8 up, 6 in 2021-03-24T21:56:35.664 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:35.718 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:35.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:35.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:56:35.824 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:35.878 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:35.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:35.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:56:35.984 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:36.044 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:36.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:36.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:56:36.130 INFO:tasks.rados.rados.0.smithi125.stdout:645: done (6 left) 2021-03-24T21:56:36.130 INFO:tasks.rados.rados.0.smithi125.stdout:646: done (5 left) 2021-03-24T21:56:36.131 INFO:tasks.rados.rados.0.smithi125.stdout:648: done (4 left) 2021-03-24T21:56:36.131 INFO:tasks.rados.rados.0.smithi125.stdout:650: done (3 left) 2021-03-24T21:56:36.131 INFO:tasks.rados.rados.0.smithi125.stdout:651: done (2 left) 2021-03-24T21:56:36.131 INFO:tasks.rados.rados.0.smithi125.stdout:652: done (1 left) 2021-03-24T21:56:36.132 INFO:tasks.rados.rados.0.smithi125.stdout:653: done (0 left) 2021-03-24T21:56:36.132 INFO:tasks.rados.rados.0.smithi125.stdout:654: copy_from oid 1 from oid 42 current snap is 61 2021-03-24T21:56:36.132 INFO:tasks.rados.rados.0.smithi125.stdout:655: read oid 44 snap -1 2021-03-24T21:56:36.132 INFO:tasks.rados.rados.0.smithi125.stdout:655: expect (ObjNum 259 snap 60 seq_num 259) 2021-03-24T21:56:36.132 INFO:tasks.rados.rados.0.smithi125.stdout:656: snap_create 2021-03-24T21:56:36.139 INFO:tasks.rados.rados.0.smithi125.stdout:655: expect (ObjNum 259 snap 60 seq_num 259) 2021-03-24T21:56:36.151 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:36.202 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:36.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:36.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:56:36.221 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-24T21:56:36.251 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'reweight-by-utilization'} 2021-03-24T21:56:36.252 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd reweight-by-utilization 110 0.05 2021-03-24T21:56:36.254 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:56:36.281 INFO:tasks.rados.rados.0.smithi125.stdout:654: finishing copy_from to smithi12556470-1 2021-03-24T21:56:36.281 INFO:tasks.rados.rados.0.smithi125.stdout:654: got expected ENOENT (src dne) 2021-03-24T21:56:36.281 INFO:tasks.rados.rados.0.smithi125.stdout:654: finishing copy_from racing read to smithi12556470-1 2021-03-24T21:56:36.307 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:36.365 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:36.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:36.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:56:36.472 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:36.475 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:36 smithi125 conmon[19936]: cluster 2021-03-24T21:56:34. 2021-03-24T21:56:36.475 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:36 smithi125 conmon[19936]: 751991+0000 mgr.y (mgr.14144) 376 : cluster [DBG] pgmap v500: 158 pgs: 158 active+clean; 124 MiB data, 435 MiB used, 536 GiB / 536 GiB avail; 13 MiB/s rd, 5.9 MiB/s wr, 23 op/s 2021-03-24T21:56:36.475 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:36 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:56:36.476 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:36 smithi125 conmon[19936]: :56:34.991943+0000 osd.7 (osd.7) 51 : cluster [DBG] 2021-03-24T21:56:36.476 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:36 smithi125 conmon[19936]: 4.a scrub ok 2021-03-24T21:56:36.476 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:36 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T21:56:36.476 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:36 smithi125 conmon[19936]: T21:56:36.127172+0000 2021-03-24T21:56:36.477 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:36 smithi125 conmon[19936]: mon.a (mon.0) 841 : cluster [DBG] 2021-03-24T21:56:36.477 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:36 smithi125 conmon[19936]: osdmap e227: 8 total, 8 up, 6 in 2021-03-24T21:56:36.477 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:36 smithi125 conmon[24927]: cluster 2021-03-24T21:56 2021-03-24T21:56:36.478 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:36 smithi125 conmon[24927]: :34.751991+0000 mgr.y (mgr.14144) 376 : cluster [DBG] pgmap v500: 158 pgs: 158 active+clean; 124 MiB data, 435 MiB used, 536 GiB / 536 GiB avail; 13 MiB/s rd, 5.9 MiB/s wr, 23 op/s 2021-03-24T21:56:36.478 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:36 smithi125 conmon[24927]: cluster 2021-03-24T21:56:36.478 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:36 smithi125 conmon[24927]: 2021-03-24T21:56:34 2021-03-24T21:56:36.478 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:36 smithi125 conmon[24927]: .991943+0000 osd.7 (osd. 2021-03-24T21:56:36.479 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:36 smithi125 conmon[24927]: 7) 51 : cluster [DBG] 4.a scrub ok 2021-03-24T21:56:36.479 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:36 smithi125 conmon[24927]: 2021-03-24T21:56:36.479 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:36 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:56:36.480 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:36 smithi125 conmon[24927]: 24T21:56:36.127172 2021-03-24T21:56:36.480 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:36 smithi125 conmon[24927]: +0000 mon.a (mon.0) 841 2021-03-24T21:56:36.480 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:36 smithi125 conmon[24927]: : cluster [DBG] osdmap e227: 8 total, 8 up, 6 in 2021-03-24T21:56:36.526 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:36.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:36.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:56:36.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:36 smithi142 conmon[20125]: cluster 2021-03-24T21:56:34.751991+0000 mgr.y (mgr.14144) 376 2021-03-24T21:56:36.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:36 smithi142 conmon[20125]: : cluster [DBG] pgmap v500: 158 pgs: 158 active+clean; 124 MiB data, 435 MiB used, 536 GiB / 536 GiB avail; 13 MiB/s rd, 5.9 MiB/s wr, 23 op/s 2021-03-24T21:56:36.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:36 smithi142 conmon[20125]: cluster 2021-03-24T21:56:34.991943+0000 osd.7 (osd.7) 51 : cluster 2021-03-24T21:56:36.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:36 smithi142 conmon[20125]: [DBG] 4.a scrub ok 2021-03-24T21:56:36.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:36 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:56:36.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:36 smithi142 conmon[20125]: 24T21:56:36.127172+0000 mon.a (mon.0) 841 2021-03-24T21:56:36.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:36 smithi142 conmon[20125]: : cluster [DBG] osdmap e227: 8 total, 8 up, 6 in 2021-03-24T21:56:36.632 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:36.687 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:36.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:36.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:56:36.704 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:56:36.791 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:36.817 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-24T21:56:36.817 INFO:teuthology.orchestra.run.smithi125.stdout: "size": 20, 2021-03-24T21:56:36.817 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 600, 2021-03-24T21:56:36.818 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [ 2021-03-24T21:56:36.818 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.818 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:215 7.7 7:ec6a359b:::smithi12556470-33:head [write 490771~686315] snapc 3=[3,2] ondisk+write+known_if_redirected e109)", 2021-03-24T21:56:36.819 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:38.120349+0000", 2021-03-24T21:56:36.819 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 118.69031978300001, 2021-03-24T21:56:36.819 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.038313058999999997, 2021-03-24T21:56:36.819 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:56:36.820 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:36.820 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:56:36.820 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:56:36.820 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:56:36.820 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 215 2021-03-24T21:56:36.821 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.821 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:56:36.821 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.821 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:56:36.822 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.120349+0000", 2021-03-24T21:56:36.822 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:56:36.823 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.823 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.824 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:56:36.824 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.120349+0000", 2021-03-24T21:56:36.824 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999971 2021-03-24T21:56:36.825 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.825 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.825 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:56:36.825 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.120346+0000", 2021-03-24T21:56:36.825 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000469061 2021-03-24T21:56:36.826 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.826 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.826 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:56:36.826 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.120815+0000", 2021-03-24T21:56:36.827 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3638999999999999e-05 2021-03-24T21:56:36.827 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.827 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.827 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:56:36.827 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.120829+0000", 2021-03-24T21:56:36.828 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.7680000000000008e-06 2021-03-24T21:56:36.828 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.828 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.829 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:56:36.829 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.120836+0000", 2021-03-24T21:56:36.829 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0045788820000000003 2021-03-24T21:56:36.829 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.829 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.830 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:36.830 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.125415+0000", 2021-03-24T21:56:36.830 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0043762779999999999 2021-03-24T21:56:36.830 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.831 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.831 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for degraded object", 2021-03-24T21:56:36.831 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.129791+0000", 2021-03-24T21:56:36.831 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.022389734000000001 2021-03-24T21:56:36.832 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.832 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.832 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:36.832 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.152181+0000", 2021-03-24T21:56:36.832 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.1767000000000004e-05 2021-03-24T21:56:36.833 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.833 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.833 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:56:36.833 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.152273+0000", 2021-03-24T21:56:36.833 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00026798499999999999 2021-03-24T21:56:36.834 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.834 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.834 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 3,7", 2021-03-24T21:56:36.834 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.152541+0000", 2021-03-24T21:56:36.835 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0035826479999999999 2021-03-24T21:56:36.835 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.836 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.836 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:36.836 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.156124+0000", 2021-03-24T21:56:36.836 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7589999999999999e-05 2021-03-24T21:56:36.837 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.837 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.837 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:56:36.837 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.156141+0000", 2021-03-24T21:56:36.838 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0022837159999999999 2021-03-24T21:56:36.838 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.838 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.838 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:36.838 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.158425+0000", 2021-03-24T21:56:36.839 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.4870999999999995e-05 2021-03-24T21:56:36.839 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.839 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.840 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:56:36.840 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.158500+0000", 2021-03-24T21:56:36.840 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000161822 2021-03-24T21:56:36.840 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.841 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.841 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:56:36.841 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.158662+0000", 2021-03-24T21:56:36.841 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.038313058999999997 2021-03-24T21:56:36.842 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:36.842 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:56:36.842 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:36.843 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.843 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.843 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:216 7.7 7:ec6a359b:::smithi12556470-33:head [write 1885075~406346] snapc 3=[3,2] ondisk+write+known_if_redirected e109)", 2021-03-24T21:56:36.844 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:38.125385+0000", 2021-03-24T21:56:36.844 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 118.685283794, 2021-03-24T21:56:36.844 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.035816903999999997, 2021-03-24T21:56:36.845 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:56:36.845 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:36.845 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:56:36.845 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:56:36.846 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:56:36.846 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 216 2021-03-24T21:56:36.846 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.846 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:56:36.847 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.847 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:56:36.847 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.125385+0000", 2021-03-24T21:56:36.847 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:56:36.847 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.848 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.848 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:56:36.848 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.125385+0000", 2021-03-24T21:56:36.848 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T21:56:36.848 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.849 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.849 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:56:36.849 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.125383+0000", 2021-03-24T21:56:36.850 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000269442 2021-03-24T21:56:36.850 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.850 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.850 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:56:36.851 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.125652+0000", 2021-03-24T21:56:36.851 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.384e-05 2021-03-24T21:56:36.851 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.851 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.852 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:56:36.852 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.125666+0000", 2021-03-24T21:56:36.852 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.1030000000000008e-06 2021-03-24T21:56:36.852 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.853 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.853 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:56:36.853 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.125675+0000", 2021-03-24T21:56:36.853 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0041614060000000003 2021-03-24T21:56:36.854 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.854 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.854 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:36.854 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.129836+0000", 2021-03-24T21:56:36.854 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.8485000000000006e-05 2021-03-24T21:56:36.855 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.855 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.855 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for degraded object", 2021-03-24T21:56:36.855 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.129905+0000", 2021-03-24T21:56:36.856 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.024237214999999999 2021-03-24T21:56:36.856 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.856 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.856 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:36.856 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.154142+0000", 2021-03-24T21:56:36.857 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000120384 2021-03-24T21:56:36.857 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.857 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.857 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:56:36.858 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.154262+0000", 2021-03-24T21:56:36.858 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000222086 2021-03-24T21:56:36.858 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.858 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.858 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 3,7", 2021-03-24T21:56:36.859 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.154485+0000", 2021-03-24T21:56:36.859 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0035572189999999999 2021-03-24T21:56:36.859 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.859 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.860 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:36.860 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.158042+0000", 2021-03-24T21:56:36.860 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00067144500000000001 2021-03-24T21:56:36.860 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.861 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.861 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:36.861 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.158713+0000", 2021-03-24T21:56:36.861 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0022954920000000001 2021-03-24T21:56:36.862 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.862 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.862 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:56:36.863 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.161009+0000", 2021-03-24T21:56:36.863 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.8288000000000002e-05 2021-03-24T21:56:36.864 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.864 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.864 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:56:36.865 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.161067+0000", 2021-03-24T21:56:36.865 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00013443600000000001 2021-03-24T21:56:36.865 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.866 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.866 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:56:36.866 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.161201+0000", 2021-03-24T21:56:36.866 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.035816903999999997 2021-03-24T21:56:36.867 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:36.867 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:56:36.867 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:36.867 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.868 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.868 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:217 7.7 7:ec6a359b:::smithi12556470-33:head [setxattr _header (56) in=7b,truncate 2890771] snapc 3=[3,2] ondisk+write+known_if_redirected e109)", 2021-03-24T21:56:36.869 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:38.125683+0000", 2021-03-24T21:56:36.869 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 118.684985227, 2021-03-24T21:56:36.869 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.035742636000000001, 2021-03-24T21:56:36.869 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:56:36.869 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:36.870 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:56:36.870 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:56:36.871 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:56:36.871 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 217 2021-03-24T21:56:36.871 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.871 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:56:36.871 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.872 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:56:36.872 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.125683+0000", 2021-03-24T21:56:36.872 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:56:36.872 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.873 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.873 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:56:36.873 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.125683+0000", 2021-03-24T21:56:36.873 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T21:56:36.873 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.874 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.874 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:56:36.874 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.125682+0000", 2021-03-24T21:56:36.875 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9900000000000002e-06 2021-03-24T21:56:36.875 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.875 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.875 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:56:36.875 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.125688+0000", 2021-03-24T21:56:36.876 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.5150000000000003e-06 2021-03-24T21:56:36.876 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.876 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.876 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:56:36.877 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.125694+0000", 2021-03-24T21:56:36.877 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.899e-06 2021-03-24T21:56:36.877 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.877 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.877 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:56:36.878 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.125699+0000", 2021-03-24T21:56:36.878 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0042341109999999996 2021-03-24T21:56:36.878 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.878 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.879 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:36.879 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.129933+0000", 2021-03-24T21:56:36.879 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5040000000000002e-05 2021-03-24T21:56:36.879 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.879 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.880 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for degraded object", 2021-03-24T21:56:36.880 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.129988+0000", 2021-03-24T21:56:36.880 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.025401093999999999 2021-03-24T21:56:36.880 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.880 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.881 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:36.881 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.155390+0000", 2021-03-24T21:56:36.882 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.8459e-05 2021-03-24T21:56:36.882 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.882 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.882 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:56:36.883 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.155488+0000", 2021-03-24T21:56:36.883 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00019190900000000001 2021-03-24T21:56:36.883 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.884 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.884 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 3,7", 2021-03-24T21:56:36.884 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.155680+0000", 2021-03-24T21:56:36.884 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0023967860000000001 2021-03-24T21:56:36.885 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.885 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.885 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:36.885 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.158077+0000", 2021-03-24T21:56:36.885 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000672877 2021-03-24T21:56:36.886 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.886 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.886 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:36.886 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.158750+0000", 2021-03-24T21:56:36.887 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0024654239999999999 2021-03-24T21:56:36.887 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.887 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.887 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:56:36.887 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.161215+0000", 2021-03-24T21:56:36.888 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6813e-05 2021-03-24T21:56:36.888 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.888 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.888 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:56:36.889 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.161272+0000", 2021-03-24T21:56:36.889 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00015384700000000001 2021-03-24T21:56:36.889 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.890 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.890 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:56:36.890 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.161426+0000", 2021-03-24T21:56:36.891 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.035742636000000001 2021-03-24T21:56:36.891 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:36.891 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:56:36.891 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:36.892 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.892 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.892 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:218 7.7 7:ec6a359b:::smithi12556470-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e109)", 2021-03-24T21:56:36.892 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:38.125704+0000", 2021-03-24T21:56:36.893 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 118.68496453900001, 2021-03-24T21:56:36.893 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.036079927999999997, 2021-03-24T21:56:36.893 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:56:36.893 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T21:56:36.894 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:56:36.894 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:56:36.894 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:56:36.894 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 218 2021-03-24T21:56:36.894 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.895 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:56:36.895 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.895 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:56:36.895 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.125704+0000", 2021-03-24T21:56:36.896 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:56:36.896 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.896 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.896 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:56:36.896 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.125704+0000", 2021-03-24T21:56:36.897 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999995 2021-03-24T21:56:36.897 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.897 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.897 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:56:36.898 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.125703+0000", 2021-03-24T21:56:36.898 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.0319999999999999e-06 2021-03-24T21:56:36.898 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.898 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.898 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:56:36.899 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.125706+0000", 2021-03-24T21:56:36.899 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.8289999999999996e-06 2021-03-24T21:56:36.899 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.899 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.900 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:56:36.900 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.125712+0000", 2021-03-24T21:56:36.900 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.0029999999999997e-06 2021-03-24T21:56:36.900 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.901 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.901 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:56:36.901 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.125719+0000", 2021-03-24T21:56:36.901 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0042974450000000004 2021-03-24T21:56:36.901 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.902 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.902 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:36.902 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.130017+0000", 2021-03-24T21:56:36.903 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.3440000000000003e-05 2021-03-24T21:56:36.903 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.903 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.903 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for degraded object", 2021-03-24T21:56:36.904 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.130070+0000", 2021-03-24T21:56:36.904 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.025864225000000001 2021-03-24T21:56:36.904 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.904 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.904 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:36.905 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.155934+0000", 2021-03-24T21:56:36.905 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000112889 2021-03-24T21:56:36.905 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.905 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.906 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T21:56:36.906 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.156047+0000", 2021-03-24T21:56:36.906 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0054236689999999999 2021-03-24T21:56:36.906 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.907 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.907 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:36.907 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.161471+0000", 2021-03-24T21:56:36.907 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00011006300000000001 2021-03-24T21:56:36.907 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.908 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.908 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:56:36.908 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.161581+0000", 2021-03-24T21:56:36.908 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00020286999999999999 2021-03-24T21:56:36.908 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.909 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.909 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:56:36.909 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.161784+0000", 2021-03-24T21:56:36.909 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.036079927999999997 2021-03-24T21:56:36.910 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:36.910 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:56:36.910 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:36.910 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.911 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.911 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:260 7.d 7:bdda3e67:::smithi12556470-22:head [copy-from ver 18] snapc 8=[8] ondisk+write+known_if_redirected e114)", 2021-03-24T21:56:36.911 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:43.163489+0000", 2021-03-24T21:56:36.911 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 113.647179357, 2021-03-24T21:56:36.912 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.030508250000000001, 2021-03-24T21:56:36.912 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:56:36.912 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:36.912 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:56:36.913 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:56:36.913 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:56:36.913 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 260 2021-03-24T21:56:36.913 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.914 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:56:36.914 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.914 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:56:36.914 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.163489+0000", 2021-03-24T21:56:36.915 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:56:36.915 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.915 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.916 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:56:36.916 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.163489+0000", 2021-03-24T21:56:36.916 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999952 2021-03-24T21:56:36.916 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.916 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.917 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:56:36.917 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.163484+0000", 2021-03-24T21:56:36.917 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.7425000000000002e-05 2021-03-24T21:56:36.917 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.918 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.918 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:56:36.918 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.163522+0000", 2021-03-24T21:56:36.918 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.0035999999999998e-05 2021-03-24T21:56:36.919 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.919 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.919 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:56:36.919 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.163552+0000", 2021-03-24T21:56:36.919 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0730000000000001e-05 2021-03-24T21:56:36.920 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.920 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.920 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:56:36.920 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.163562+0000", 2021-03-24T21:56:36.920 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.0197000000000004e-05 2021-03-24T21:56:36.921 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.921 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.921 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:36.921 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.163652+0000", 2021-03-24T21:56:36.922 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.204e-05 2021-03-24T21:56:36.922 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.922 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.922 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op must wait for map", 2021-03-24T21:56:36.922 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.163665+0000", 2021-03-24T21:56:36.923 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0052565399999999997 2021-03-24T21:56:36.923 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.923 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.924 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:36.924 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.168921+0000", 2021-03-24T21:56:36.924 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.3563e-05 2021-03-24T21:56:36.924 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.925 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.925 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:56:36.926 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.168995+0000", 2021-03-24T21:56:36.926 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012325943000000001 2021-03-24T21:56:36.926 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.927 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.927 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 3,7", 2021-03-24T21:56:36.928 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.181321+0000", 2021-03-24T21:56:36.928 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0092570110000000008 2021-03-24T21:56:36.928 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.929 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.929 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:36.930 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.190578+0000", 2021-03-24T21:56:36.930 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0026179480000000001 2021-03-24T21:56:36.931 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.931 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.931 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:56:36.931 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.193196+0000", 2021-03-24T21:56:36.932 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00069205999999999998 2021-03-24T21:56:36.932 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.932 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.932 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:36.933 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.193888+0000", 2021-03-24T21:56:36.933 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2537000000000002e-05 2021-03-24T21:56:36.933 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.933 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.933 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:56:36.934 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.193940+0000", 2021-03-24T21:56:36.934 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.7077000000000003e-05 2021-03-24T21:56:36.934 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.934 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.935 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:56:36.935 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.193997+0000", 2021-03-24T21:56:36.935 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.030508250000000001 2021-03-24T21:56:36.935 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:36.935 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:56:36.936 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:36.936 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.936 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.936 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:261 7.d 7:bdda3e67:::smithi12556470-22:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e114)", 2021-03-24T21:56:36.936 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:43.163585+0000", 2021-03-24T21:56:36.937 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 113.647082966, 2021-03-24T21:56:36.937 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.030581255000000002, 2021-03-24T21:56:36.937 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:56:36.937 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T21:56:36.937 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:56:36.938 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:56:36.938 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:56:36.938 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 261 2021-03-24T21:56:36.938 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.939 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:56:36.939 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.939 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:56:36.939 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.163585+0000", 2021-03-24T21:56:36.939 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:56:36.940 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.940 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.940 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:56:36.940 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.163585+0000", 2021-03-24T21:56:36.940 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T21:56:36.941 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.941 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.941 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:56:36.941 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.163584+0000", 2021-03-24T21:56:36.941 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.8089999999999999e-06 2021-03-24T21:56:36.942 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.942 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.942 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:56:36.942 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.163588+0000", 2021-03-24T21:56:36.943 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.0350000000000001e-06 2021-03-24T21:56:36.943 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.943 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.943 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:56:36.943 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.163596+0000", 2021-03-24T21:56:36.944 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.2740000000000001e-06 2021-03-24T21:56:36.944 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.944 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.944 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:56:36.944 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.163601+0000", 2021-03-24T21:56:36.945 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00015019400000000001 2021-03-24T21:56:36.945 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.945 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.945 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:36.945 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.163751+0000", 2021-03-24T21:56:36.946 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.6970000000000005e-06 2021-03-24T21:56:36.946 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.946 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.946 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting_for_map not empty", 2021-03-24T21:56:36.947 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.163760+0000", 2021-03-24T21:56:36.947 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0053697110000000001 2021-03-24T21:56:36.947 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.947 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.947 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:36.948 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.169130+0000", 2021-03-24T21:56:36.948 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.9261000000000002e-05 2021-03-24T21:56:36.948 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.948 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.948 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-24T21:56:36.949 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.169179+0000", 2021-03-24T21:56:36.949 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.01955324 2021-03-24T21:56:36.949 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.949 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.950 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:36.950 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.188733+0000", 2021-03-24T21:56:36.950 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000104594 2021-03-24T21:56:36.950 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.950 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.951 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T21:56:36.951 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.188837+0000", 2021-03-24T21:56:36.951 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.005186459 2021-03-24T21:56:36.951 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.952 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.952 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:36.952 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.194024+0000", 2021-03-24T21:56:36.952 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.5635000000000004e-05 2021-03-24T21:56:36.952 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.953 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.953 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:56:36.953 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.194089+0000", 2021-03-24T21:56:36.953 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.7201999999999997e-05 2021-03-24T21:56:36.953 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.954 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.954 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:56:36.954 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.194167+0000", 2021-03-24T21:56:36.954 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.030581255000000002 2021-03-24T21:56:36.955 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:36.955 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:56:36.955 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:36.955 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.955 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.956 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:426 7.d 7:b2218182:::smithi12556470-21:head [write 460525~690852] snapc 10=[10,f] ondisk+write+known_if_redirected e123)", 2021-03-24T21:56:36.956 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:52.089609+0000", 2021-03-24T21:56:36.956 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 104.721059426, 2021-03-24T21:56:36.956 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.043996557999999998, 2021-03-24T21:56:36.956 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:56:36.957 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:36.957 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:56:36.957 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:56:36.957 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:56:36.958 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 426 2021-03-24T21:56:36.958 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.958 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:56:36.958 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.958 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:56:36.959 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.089609+0000", 2021-03-24T21:56:36.959 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:56:36.959 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.959 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.960 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:56:36.960 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.089609+0000", 2021-03-24T21:56:36.960 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999976 2021-03-24T21:56:36.960 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.960 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.961 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:56:36.961 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.089607+0000", 2021-03-24T21:56:36.961 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00037713800000000001 2021-03-24T21:56:36.961 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.961 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.962 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:56:36.962 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.089984+0000", 2021-03-24T21:56:36.962 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.028226591999999998 2021-03-24T21:56:36.962 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.963 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.963 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:56:36.963 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.118210+0000", 2021-03-24T21:56:36.963 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2758000000000001e-05 2021-03-24T21:56:36.963 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.964 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.964 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:56:36.964 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.118223+0000", 2021-03-24T21:56:36.964 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00053454999999999996 2021-03-24T21:56:36.964 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.965 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.965 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:36.965 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.118758+0000", 2021-03-24T21:56:36.965 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000162077 2021-03-24T21:56:36.966 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.966 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.966 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:56:36.966 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.118920+0000", 2021-03-24T21:56:36.966 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00037629800000000001 2021-03-24T21:56:36.967 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.967 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.967 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 3,7", 2021-03-24T21:56:36.967 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.119296+0000", 2021-03-24T21:56:36.968 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011427837999999999 2021-03-24T21:56:36.968 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.968 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.968 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:36.968 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.130724+0000", 2021-03-24T21:56:36.969 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6586999999999999e-05 2021-03-24T21:56:36.969 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.969 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.969 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:56:36.970 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.130780+0000", 2021-03-24T21:56:36.970 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0026898439999999998 2021-03-24T21:56:36.970 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.970 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.970 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:36.971 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.133470+0000", 2021-03-24T21:56:36.971 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6851000000000001e-05 2021-03-24T21:56:36.971 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.971 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.971 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:56:36.972 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.133537+0000", 2021-03-24T21:56:36.972 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.8347000000000001e-05 2021-03-24T21:56:36.972 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.972 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.973 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:56:36.973 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.133605+0000", 2021-03-24T21:56:36.973 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.043996557999999998 2021-03-24T21:56:36.973 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:36.973 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:56:36.974 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:36.974 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.974 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.974 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:639 7.7 7:ef6d6bee:::smithi12556470-24:head [write 2761543~670951] snapc 23=[23,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e141)", 2021-03-24T21:56:36.975 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:55:10.121478+0000", 2021-03-24T21:56:36.975 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 86.689190612999994, 2021-03-24T21:56:36.975 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.030957241999999999, 2021-03-24T21:56:36.975 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:56:36.975 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:36.976 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:56:36.976 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:56:36.976 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:56:36.976 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 639 2021-03-24T21:56:36.977 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.977 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:56:36.977 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.977 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:56:36.977 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.121478+0000", 2021-03-24T21:56:36.978 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:56:36.978 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.978 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.978 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:56:36.978 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.121478+0000", 2021-03-24T21:56:36.979 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999957 2021-03-24T21:56:36.979 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.979 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.979 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:56:36.980 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.121473+0000", 2021-03-24T21:56:36.980 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000474146 2021-03-24T21:56:36.980 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.980 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.980 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:56:36.981 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.121947+0000", 2021-03-24T21:56:36.981 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.898e-05 2021-03-24T21:56:36.981 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.981 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.981 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:56:36.982 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.121976+0000", 2021-03-24T21:56:36.982 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0725e-05 2021-03-24T21:56:36.982 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.982 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.982 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:56:36.983 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.121987+0000", 2021-03-24T21:56:36.983 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000111301 2021-03-24T21:56:36.983 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.983 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.984 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:36.984 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122098+0000", 2021-03-24T21:56:36.984 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010171900000000001 2021-03-24T21:56:36.984 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.984 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.985 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:56:36.985 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122200+0000", 2021-03-24T21:56:36.985 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000187164 2021-03-24T21:56:36.985 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.985 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.986 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 3,7", 2021-03-24T21:56:36.986 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122387+0000", 2021-03-24T21:56:36.986 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012969125 2021-03-24T21:56:36.986 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.987 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.987 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:36.987 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.135356+0000", 2021-03-24T21:56:36.987 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.016291010000000002 2021-03-24T21:56:36.987 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.988 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.988 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:56:36.988 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.151647+0000", 2021-03-24T21:56:36.988 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00059095299999999996 2021-03-24T21:56:36.988 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.989 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.989 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:36.989 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.152238+0000", 2021-03-24T21:56:36.994 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00011542 2021-03-24T21:56:36.994 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.994 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.994 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:56:36.995 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.152354+0000", 2021-03-24T21:56:36.995 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.1100000000000006e-05 2021-03-24T21:56:36.995 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.995 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.995 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:56:36.996 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.152435+0000", 2021-03-24T21:56:36.996 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.030957241999999999 2021-03-24T21:56:36.996 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:36.996 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:56:36.996 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:36.997 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.997 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:36.997 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:640 7.7 7:ef6d6bee:::smithi12556470-24:head [setxattr _header (57) in=7b,truncate 3781542] snapc 23=[23,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e141)", 2021-03-24T21:56:36.997 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:55:10.122025+0000", 2021-03-24T21:56:36.998 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 86.688643251000002, 2021-03-24T21:56:36.998 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.030689175999999999, 2021-03-24T21:56:36.998 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:56:36.998 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:36.998 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:56:36.999 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:56:36.999 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:56:36.999 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 640 2021-03-24T21:56:36.999 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:36.999 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:56:37.000 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.000 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:56:37.000 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122025+0000", 2021-03-24T21:56:37.000 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:56:37.001 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.001 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.001 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:56:37.001 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122025+0000", 2021-03-24T21:56:37.001 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T21:56:37.002 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.002 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.002 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:56:37.002 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122024+0000", 2021-03-24T21:56:37.002 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.6760000000000002e-06 2021-03-24T21:56:37.003 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.003 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.003 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:56:37.003 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122034+0000", 2021-03-24T21:56:37.004 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7944999999999998e-05 2021-03-24T21:56:37.004 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.004 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.004 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:56:37.004 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122051+0000", 2021-03-24T21:56:37.005 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.1380000000000004e-06 2021-03-24T21:56:37.005 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.005 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.005 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:56:37.005 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122058+0000", 2021-03-24T21:56:37.006 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001360876 2021-03-24T21:56:37.006 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.006 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.006 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:37.007 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.123418+0000", 2021-03-24T21:56:37.007 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000103115 2021-03-24T21:56:37.007 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.007 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.008 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:56:37.008 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.123522+0000", 2021-03-24T21:56:37.008 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000223079 2021-03-24T21:56:37.008 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.008 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.009 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 3,7", 2021-03-24T21:56:37.009 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.123745+0000", 2021-03-24T21:56:37.009 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011662959000000001 2021-03-24T21:56:37.009 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.009 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.010 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:37.010 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.135408+0000", 2021-03-24T21:56:37.010 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.016250840999999999 2021-03-24T21:56:37.010 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.011 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.011 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:56:37.011 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.151658+0000", 2021-03-24T21:56:37.011 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00092373499999999996 2021-03-24T21:56:37.011 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.012 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.012 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:37.012 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.152582+0000", 2021-03-24T21:56:37.012 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.9813000000000004e-05 2021-03-24T21:56:37.012 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.013 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.013 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:56:37.013 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.152652+0000", 2021-03-24T21:56:37.013 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2185000000000002e-05 2021-03-24T21:56:37.014 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.014 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.014 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:56:37.014 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.152714+0000", 2021-03-24T21:56:37.014 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.030689175999999999 2021-03-24T21:56:37.015 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:37.015 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:56:37.015 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:37.015 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.015 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.016 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:641 7.7 7:ef6d6bee:::smithi12556470-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e141)", 2021-03-24T21:56:37.016 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:55:10.122106+0000", 2021-03-24T21:56:37.016 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 86.688562410000003, 2021-03-24T21:56:37.016 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.031131099999999998, 2021-03-24T21:56:37.017 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:56:37.017 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T21:56:37.017 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:56:37.017 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:56:37.017 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:56:37.018 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 641 2021-03-24T21:56:37.018 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.018 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:56:37.018 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.019 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:56:37.019 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122106+0000", 2021-03-24T21:56:37.019 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:56:37.019 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.019 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.020 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:56:37.020 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122106+0000", 2021-03-24T21:56:37.020 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T21:56:37.020 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.020 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.021 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:56:37.021 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122104+0000", 2021-03-24T21:56:37.021 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.7410000000000003e-06 2021-03-24T21:56:37.021 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.022 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.022 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:56:37.022 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122111+0000", 2021-03-24T21:56:37.022 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.6475000000000001e-05 2021-03-24T21:56:37.022 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.023 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.023 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:56:37.023 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122128+0000", 2021-03-24T21:56:37.023 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.3730000000000004e-06 2021-03-24T21:56:37.023 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.024 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.024 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:56:37.024 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122135+0000", 2021-03-24T21:56:37.024 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001905 2021-03-24T21:56:37.025 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.025 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.025 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:37.025 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124040+0000", 2021-03-24T21:56:37.025 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.4141000000000005e-05 2021-03-24T21:56:37.026 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.026 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.026 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T21:56:37.026 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124134+0000", 2021-03-24T21:56:37.027 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.028680985999999999 2021-03-24T21:56:37.027 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.027 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.027 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:37.027 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.152815+0000", 2021-03-24T21:56:37.028 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00019037599999999999 2021-03-24T21:56:37.028 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.028 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.028 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:56:37.028 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.153006+0000", 2021-03-24T21:56:37.029 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00023145500000000001 2021-03-24T21:56:37.029 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.029 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.029 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:56:37.030 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.153237+0000", 2021-03-24T21:56:37.030 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.031131099999999998 2021-03-24T21:56:37.030 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:37.030 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:56:37.030 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:37.031 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.031 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.031 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "MOSDScrubReserve(4.5 REQUEST e161)", 2021-03-24T21:56:37.031 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:55:30.127542+0000", 2021-03-24T21:56:37.032 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 66.683126791999996, 2021-03-24T21:56:37.032 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.033450199999999999, 2021-03-24T21:56:37.032 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:56:37.032 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T21:56:37.032 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:56:37.033 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.033 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:56:37.033 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:30.127542+0000", 2021-03-24T21:56:37.033 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:56:37.033 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.034 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.034 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:56:37.034 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:30.127542+0000", 2021-03-24T21:56:37.034 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999995 2021-03-24T21:56:37.035 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.035 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.035 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:56:37.035 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:30.127541+0000", 2021-03-24T21:56:37.035 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.2640000000000002e-06 2021-03-24T21:56:37.036 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.036 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.036 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:56:37.036 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:30.127545+0000", 2021-03-24T21:56:37.037 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.8475000000000001e-05 2021-03-24T21:56:37.037 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.037 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.037 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:56:37.037 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:30.127574+0000", 2021-03-24T21:56:37.038 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2130000000000001e-05 2021-03-24T21:56:37.038 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.038 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.038 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:56:37.038 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:30.127586+0000", 2021-03-24T21:56:37.039 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.9679999999999996e-05 2021-03-24T21:56:37.039 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.039 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.039 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:37.040 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:30.127666+0000", 2021-03-24T21:56:37.040 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3757000000000001e-05 2021-03-24T21:56:37.040 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.040 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.040 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op must wait for map", 2021-03-24T21:56:37.041 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:30.127679+0000", 2021-03-24T21:56:37.041 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.033272818000000003 2021-03-24T21:56:37.041 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.041 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.041 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:37.042 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:30.160952+0000", 2021-03-24T21:56:37.042 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5781999999999999e-05 2021-03-24T21:56:37.042 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.042 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.043 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:56:37.043 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:30.160968+0000", 2021-03-24T21:56:37.043 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.3696000000000001e-05 2021-03-24T21:56:37.043 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.043 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.044 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:56:37.044 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:30.160992+0000", 2021-03-24T21:56:37.044 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.033450199999999999 2021-03-24T21:56:37.044 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:37.045 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:56:37.045 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:37.045 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.045 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.045 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:876 7.6 7:6ed83fae:::smithi12556470-29:head [copy-from ver 49] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e162)", 2021-03-24T21:56:37.046 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:55:31.107823+0000", 2021-03-24T21:56:37.046 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 65.702845280000005, 2021-03-24T21:56:37.046 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.032579779000000003, 2021-03-24T21:56:37.046 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:56:37.047 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:37.047 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:56:37.047 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:56:37.047 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:56:37.047 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 876 2021-03-24T21:56:37.048 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.048 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:56:37.048 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.048 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:56:37.048 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:31.107823+0000", 2021-03-24T21:56:37.049 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:56:37.049 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.049 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.049 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:56:37.050 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:31.107823+0000", 2021-03-24T21:56:37.050 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999957 2021-03-24T21:56:37.050 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.050 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.050 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:56:37.051 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:31.107819+0000", 2021-03-24T21:56:37.051 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.8042000000000001e-05 2021-03-24T21:56:37.051 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.051 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.052 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:56:37.052 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:31.107837+0000", 2021-03-24T21:56:37.052 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.1486999999999999e-05 2021-03-24T21:56:37.052 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.052 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.053 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:56:37.053 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:31.107868+0000", 2021-03-24T21:56:37.053 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.448e-05 2021-03-24T21:56:37.053 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.053 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.054 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:56:37.054 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:31.107883+0000", 2021-03-24T21:56:37.054 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.5377999999999995e-05 2021-03-24T21:56:37.054 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.055 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.055 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:37.055 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:31.107968+0000", 2021-03-24T21:56:37.055 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3319000000000001e-05 2021-03-24T21:56:37.055 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.056 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.056 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op must wait for map", 2021-03-24T21:56:37.056 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:31.107982+0000", 2021-03-24T21:56:37.056 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0045942719999999999 2021-03-24T21:56:37.056 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.057 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.057 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:37.057 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:31.112576+0000", 2021-03-24T21:56:37.057 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000116446 2021-03-24T21:56:37.058 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.058 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.058 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:56:37.058 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:31.112692+0000", 2021-03-24T21:56:37.058 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.015582362000000001 2021-03-24T21:56:37.059 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.059 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.059 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 3,7", 2021-03-24T21:56:37.059 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:31.128275+0000", 2021-03-24T21:56:37.059 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00917281 2021-03-24T21:56:37.060 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.060 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.060 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:37.060 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:31.137447+0000", 2021-03-24T21:56:37.061 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0021480499999999999 2021-03-24T21:56:37.061 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.061 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.061 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:37.061 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:31.139595+0000", 2021-03-24T21:56:37.062 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00064808499999999996 2021-03-24T21:56:37.062 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.062 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.062 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:56:37.062 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:31.140244+0000", 2021-03-24T21:56:37.063 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.4270000000000006e-05 2021-03-24T21:56:37.063 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.063 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.063 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:56:37.064 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:31.140318+0000", 2021-03-24T21:56:37.064 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.4940999999999999e-05 2021-03-24T21:56:37.064 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.064 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.064 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:56:37.065 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:31.140403+0000", 2021-03-24T21:56:37.065 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.032579779000000003 2021-03-24T21:56:37.065 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:37.065 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:56:37.065 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:37.066 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.066 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.066 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:877 7.6 7:6ed83fae:::smithi12556470-29:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e162)", 2021-03-24T21:56:37.066 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:55:31.107907+0000", 2021-03-24T21:56:37.067 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 65.702761186999993, 2021-03-24T21:56:37.067 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.032708360999999998, 2021-03-24T21:56:37.067 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:56:37.067 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T21:56:37.067 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:56:37.068 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:56:37.068 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:56:37.068 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 877 2021-03-24T21:56:37.068 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.068 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:56:37.069 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.069 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:56:37.069 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:31.107907+0000", 2021-03-24T21:56:37.069 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:56:37.070 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.070 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.070 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:56:37.070 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:31.107907+0000", 2021-03-24T21:56:37.070 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T21:56:37.071 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.071 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.071 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:56:37.071 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:31.107906+0000", 2021-03-24T21:56:37.071 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.8080000000000001e-06 2021-03-24T21:56:37.072 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.072 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.072 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:56:37.072 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:31.107912+0000", 2021-03-24T21:56:37.073 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.9799999999999993e-06 2021-03-24T21:56:37.073 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.073 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.073 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:56:37.073 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:31.107922+0000", 2021-03-24T21:56:37.074 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4679999999999997e-06 2021-03-24T21:56:37.074 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.074 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.074 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:56:37.074 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:31.107928+0000", 2021-03-24T21:56:37.075 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000359544 2021-03-24T21:56:37.075 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.075 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.075 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:37.076 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:31.108288+0000", 2021-03-24T21:56:37.076 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.6840000000000002e-06 2021-03-24T21:56:37.076 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.076 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.076 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting_for_map not empty", 2021-03-24T21:56:37.077 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:31.108296+0000", 2021-03-24T21:56:37.077 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0045667379999999999 2021-03-24T21:56:37.077 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.077 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.077 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:37.078 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:31.112863+0000", 2021-03-24T21:56:37.078 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.9910999999999997e-05 2021-03-24T21:56:37.078 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.078 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.078 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-24T21:56:37.079 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:31.112913+0000", 2021-03-24T21:56:37.079 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.023486591000000001 2021-03-24T21:56:37.079 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.079 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.080 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:37.080 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:31.136400+0000", 2021-03-24T21:56:37.080 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.4000999999999998e-05 2021-03-24T21:56:37.080 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.080 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.081 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T21:56:37.081 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:31.136474+0000", 2021-03-24T21:56:37.081 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0039723190000000002 2021-03-24T21:56:37.081 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.082 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.082 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:37.082 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:31.140446+0000", 2021-03-24T21:56:37.082 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.1848000000000006e-05 2021-03-24T21:56:37.082 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.083 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.083 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:56:37.083 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:31.140538+0000", 2021-03-24T21:56:37.083 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.7707000000000007e-05 2021-03-24T21:56:37.084 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.084 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.084 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:56:37.084 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:31.140615+0000", 2021-03-24T21:56:37.084 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.032708360999999998 2021-03-24T21:56:37.085 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:37.085 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:56:37.085 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:37.085 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.085 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.086 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:947 7.d 7:bdda3e67:::smithi12556470-22:head [copy-from ver 31] snapc 3b=[2e] ondisk+write+known_if_redirected e166)", 2021-03-24T21:56:37.086 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:55:35.107049+0000", 2021-03-24T21:56:37.086 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 61.703619553000003, 2021-03-24T21:56:37.086 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.029247223999999999, 2021-03-24T21:56:37.087 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:56:37.087 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:37.087 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:56:37.087 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:56:37.087 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:56:37.088 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 947 2021-03-24T21:56:37.088 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.088 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:56:37.088 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.089 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:56:37.089 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:35.107049+0000", 2021-03-24T21:56:37.089 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:56:37.089 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.089 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.090 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:56:37.090 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:35.107049+0000", 2021-03-24T21:56:37.090 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999971 2021-03-24T21:56:37.090 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.091 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.091 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:56:37.091 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:35.107046+0000", 2021-03-24T21:56:37.091 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2381999999999999e-05 2021-03-24T21:56:37.091 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.092 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.092 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:56:37.092 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:35.107058+0000", 2021-03-24T21:56:37.092 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.8236999999999998e-05 2021-03-24T21:56:37.092 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.093 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.093 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:56:37.093 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:35.107077+0000", 2021-03-24T21:56:37.093 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.1570000000000003e-06 2021-03-24T21:56:37.094 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.094 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.094 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:56:37.094 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:35.107084+0000", 2021-03-24T21:56:37.094 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.9081999999999999e-05 2021-03-24T21:56:37.095 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.095 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.095 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:37.095 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:35.107163+0000", 2021-03-24T21:56:37.095 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0127e-05 2021-03-24T21:56:37.096 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.096 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.096 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op must wait for map", 2021-03-24T21:56:37.096 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:35.107173+0000", 2021-03-24T21:56:37.097 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0082422009999999993 2021-03-24T21:56:37.097 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.097 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.097 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:37.097 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:35.115415+0000", 2021-03-24T21:56:37.098 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000118127 2021-03-24T21:56:37.098 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.098 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.098 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:56:37.099 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:35.115533+0000", 2021-03-24T21:56:37.099 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0096834090000000005 2021-03-24T21:56:37.099 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.099 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.099 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 3,7", 2021-03-24T21:56:37.100 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:35.125217+0000", 2021-03-24T21:56:37.100 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0047619710000000003 2021-03-24T21:56:37.100 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.100 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.100 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:56:37.101 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:35.129979+0000", 2021-03-24T21:56:37.101 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00034513600000000001 2021-03-24T21:56:37.101 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.101 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.102 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:37.102 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:35.130324+0000", 2021-03-24T21:56:37.102 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0057934889999999998 2021-03-24T21:56:37.102 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.102 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.103 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:37.103 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:35.136117+0000", 2021-03-24T21:56:37.103 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.7144999999999994e-05 2021-03-24T21:56:37.103 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.103 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.104 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:56:37.104 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:35.136195+0000", 2021-03-24T21:56:37.104 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000101461 2021-03-24T21:56:37.104 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.105 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.105 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:56:37.105 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:35.136296+0000", 2021-03-24T21:56:37.105 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.029247223999999999 2021-03-24T21:56:37.105 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:37.106 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:56:37.106 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:37.106 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.106 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.107 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:948 7.d 7:bdda3e67:::smithi12556470-22:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e166)", 2021-03-24T21:56:37.107 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:55:35.107133+0000", 2021-03-24T21:56:37.107 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 61.703535459999998, 2021-03-24T21:56:37.107 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.029474744000000001, 2021-03-24T21:56:37.107 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:56:37.108 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T21:56:37.108 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:56:37.108 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:56:37.108 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:56:37.108 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 948 2021-03-24T21:56:37.109 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.109 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:56:37.109 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.109 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:56:37.110 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:35.107133+0000", 2021-03-24T21:56:37.110 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:56:37.110 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.110 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.110 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:56:37.111 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:35.107133+0000", 2021-03-24T21:56:37.111 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T21:56:37.111 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.111 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.112 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:56:37.112 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:35.107132+0000", 2021-03-24T21:56:37.112 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2887999999999999e-05 2021-03-24T21:56:37.112 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.112 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.113 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:56:37.113 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:35.107144+0000", 2021-03-24T21:56:37.113 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1939e-05 2021-03-24T21:56:37.113 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.113 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.114 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:56:37.114 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:35.107156+0000", 2021-03-24T21:56:37.114 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.1399999999999997e-06 2021-03-24T21:56:37.114 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.115 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.115 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:56:37.115 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:35.107162+0000", 2021-03-24T21:56:37.115 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.8176999999999995e-05 2021-03-24T21:56:37.115 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.116 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.116 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:37.116 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:35.107231+0000", 2021-03-24T21:56:37.116 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6670000000000004e-06 2021-03-24T21:56:37.116 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.117 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.117 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting_for_map not empty", 2021-03-24T21:56:37.117 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:35.107235+0000", 2021-03-24T21:56:37.117 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0084086260000000006 2021-03-24T21:56:37.117 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.118 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.118 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:37.118 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:35.115644+0000", 2021-03-24T21:56:37.118 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.1778e-05 2021-03-24T21:56:37.119 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.119 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.119 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-24T21:56:37.119 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:35.115676+0000", 2021-03-24T21:56:37.119 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012679057000000001 2021-03-24T21:56:37.120 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.120 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.120 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:37.120 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:35.128355+0000", 2021-03-24T21:56:37.120 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000271945 2021-03-24T21:56:37.121 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.121 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.121 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T21:56:37.121 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:35.128627+0000", 2021-03-24T21:56:37.121 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00772618 2021-03-24T21:56:37.122 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.122 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.122 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:37.122 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:35.136353+0000", 2021-03-24T21:56:37.123 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000107327 2021-03-24T21:56:37.123 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.123 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.123 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:56:37.123 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:35.136460+0000", 2021-03-24T21:56:37.124 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00014735599999999999 2021-03-24T21:56:37.124 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.124 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.124 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:56:37.124 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:35.136608+0000", 2021-03-24T21:56:37.125 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.029474744000000001 2021-03-24T21:56:37.125 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:37.125 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:56:37.125 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:37.125 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.126 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.126 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:1235 7.9 7:9b48f357:::smithi12556470-17:head [write 426944~556633,stat] snapc 53=[] ondisk+write+known_if_redirected e191)", 2021-03-24T21:56:37.126 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:56:00.136481+0000", 2021-03-24T21:56:37.126 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 36.674187762000003, 2021-03-24T21:56:37.127 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.081696782999999995, 2021-03-24T21:56:37.127 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:56:37.127 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:37.127 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:56:37.127 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:56:37.128 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:56:37.128 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1235 2021-03-24T21:56:37.128 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.128 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:56:37.129 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.129 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:56:37.129 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.136481+0000", 2021-03-24T21:56:37.129 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:56:37.129 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.130 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.130 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:56:37.130 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.136481+0000", 2021-03-24T21:56:37.130 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999967 2021-03-24T21:56:37.130 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.131 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.131 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:56:37.131 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.136477+0000", 2021-03-24T21:56:37.131 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00036797499999999999 2021-03-24T21:56:37.132 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.132 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.132 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:56:37.132 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.136845+0000", 2021-03-24T21:56:37.132 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.069078045000000005 2021-03-24T21:56:37.133 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.133 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.133 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:56:37.133 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.205923+0000", 2021-03-24T21:56:37.133 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.6849999999999996e-06 2021-03-24T21:56:37.134 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.134 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.134 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:56:37.134 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.205933+0000", 2021-03-24T21:56:37.134 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00082229500000000001 2021-03-24T21:56:37.135 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.135 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.135 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:37.135 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.206755+0000", 2021-03-24T21:56:37.136 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000119758 2021-03-24T21:56:37.136 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.136 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.136 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:56:37.136 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.206875+0000", 2021-03-24T21:56:37.137 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000201941 2021-03-24T21:56:37.137 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.137 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.137 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 4,7", 2021-03-24T21:56:37.137 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.207077+0000", 2021-03-24T21:56:37.138 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0041193269999999999 2021-03-24T21:56:37.138 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.138 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.138 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:56:37.139 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.211196+0000", 2021-03-24T21:56:37.139 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0067061409999999997 2021-03-24T21:56:37.139 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.139 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.139 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:37.140 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.217903+0000", 2021-03-24T21:56:37.140 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00017204899999999999 2021-03-24T21:56:37.140 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.140 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.141 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:37.141 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.218075+0000", 2021-03-24T21:56:37.141 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.8982e-05 2021-03-24T21:56:37.141 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.141 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.142 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:56:37.142 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.218114+0000", 2021-03-24T21:56:37.142 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.3751000000000007e-05 2021-03-24T21:56:37.142 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.142 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.143 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:56:37.143 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.218177+0000", 2021-03-24T21:56:37.143 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.081696782999999995 2021-03-24T21:56:37.143 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:37.144 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:56:37.144 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:37.144 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.144 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.144 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:1532 7.4 7:260d8f69:::smithi12556470-28:head [copy-from ver 168] snapc 68=[68] ondisk+write+known_if_redirected e210)", 2021-03-24T21:56:37.145 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:56:19.186729+0000", 2021-03-24T21:56:37.145 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 17.623939228000001, 2021-03-24T21:56:37.145 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.040612834, 2021-03-24T21:56:37.145 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:56:37.145 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:37.146 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:56:37.146 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:56:37.146 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:56:37.146 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1532 2021-03-24T21:56:37.147 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.147 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:56:37.147 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.147 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:56:37.147 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:19.186729+0000", 2021-03-24T21:56:37.148 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:56:37.148 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.148 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.148 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:56:37.148 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:19.186729+0000", 2021-03-24T21:56:37.149 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T21:56:37.149 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.149 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.149 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:56:37.150 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:19.186728+0000", 2021-03-24T21:56:37.150 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.5259999999999999e-06 2021-03-24T21:56:37.150 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.150 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.150 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:56:37.151 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:19.186735+0000", 2021-03-24T21:56:37.151 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.9740000000000008e-06 2021-03-24T21:56:37.151 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.151 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.152 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:56:37.152 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:19.186744+0000", 2021-03-24T21:56:37.152 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.37e-06 2021-03-24T21:56:37.152 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.152 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.153 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:56:37.153 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:19.186750+0000", 2021-03-24T21:56:37.153 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.6221000000000001e-05 2021-03-24T21:56:37.153 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.153 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.154 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:37.154 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:19.186826+0000", 2021-03-24T21:56:37.154 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2408e-05 2021-03-24T21:56:37.154 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.155 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.155 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op must wait for map", 2021-03-24T21:56:37.155 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:19.186839+0000", 2021-03-24T21:56:37.155 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.004219942 2021-03-24T21:56:37.155 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.156 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.156 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:37.156 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:19.191059+0000", 2021-03-24T21:56:37.156 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000110139 2021-03-24T21:56:37.157 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.157 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.157 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:56:37.157 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:19.191169+0000", 2021-03-24T21:56:37.157 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.01962738 2021-03-24T21:56:37.158 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.158 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.158 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 4,5", 2021-03-24T21:56:37.158 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:19.210796+0000", 2021-03-24T21:56:37.158 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011690872 2021-03-24T21:56:37.159 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.159 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.159 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:56:37.159 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:19.222487+0000", 2021-03-24T21:56:37.159 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0029462009999999999 2021-03-24T21:56:37.160 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.160 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.160 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:37.160 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:19.225433+0000", 2021-03-24T21:56:37.161 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001719139 2021-03-24T21:56:37.161 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.161 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.161 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:37.161 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:19.227152+0000", 2021-03-24T21:56:37.162 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.0577999999999999e-05 2021-03-24T21:56:37.162 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.162 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.162 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:56:37.162 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:19.227243+0000", 2021-03-24T21:56:37.163 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.9079999999999993e-05 2021-03-24T21:56:37.163 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.163 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.163 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:56:37.164 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:19.227342+0000", 2021-03-24T21:56:37.164 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.040612834 2021-03-24T21:56:37.164 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:37.164 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:56:37.164 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:37.165 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.165 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.165 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:1533 7.4 7:260d8f69:::smithi12556470-28:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e210)", 2021-03-24T21:56:37.165 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:56:19.186772+0000", 2021-03-24T21:56:37.165 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 17.623896071000001, 2021-03-24T21:56:37.166 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.040797951999999998, 2021-03-24T21:56:37.166 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:56:37.166 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T21:56:37.166 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:56:37.167 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:56:37.167 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:56:37.167 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1533 2021-03-24T21:56:37.167 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.167 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:56:37.168 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.168 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:56:37.168 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:19.186772+0000", 2021-03-24T21:56:37.168 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:56:37.168 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.169 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.169 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:56:37.169 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:19.186772+0000", 2021-03-24T21:56:37.169 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T21:56:37.169 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.170 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.170 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:56:37.170 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:19.186771+0000", 2021-03-24T21:56:37.170 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.2836999999999998e-05 2021-03-24T21:56:37.171 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.171 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.171 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:56:37.171 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:19.186794+0000", 2021-03-24T21:56:37.171 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1888e-05 2021-03-24T21:56:37.172 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.172 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.172 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:56:37.172 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:19.186806+0000", 2021-03-24T21:56:37.173 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9830000000000001e-06 2021-03-24T21:56:37.173 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.173 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.173 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:56:37.173 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:19.186812+0000", 2021-03-24T21:56:37.174 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.5371000000000006e-05 2021-03-24T21:56:37.174 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.174 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.174 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:37.175 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:19.186908+0000", 2021-03-24T21:56:37.175 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0088999999999999e-05 2021-03-24T21:56:37.175 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.175 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.175 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting_for_map not empty", 2021-03-24T21:56:37.176 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:19.186918+0000", 2021-03-24T21:56:37.176 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0044129579999999998 2021-03-24T21:56:37.176 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.176 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.176 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:37.177 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:19.191331+0000", 2021-03-24T21:56:37.177 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.1284999999999993e-05 2021-03-24T21:56:37.177 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.177 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.177 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-24T21:56:37.178 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:19.191392+0000", 2021-03-24T21:56:37.178 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.026087465000000001 2021-03-24T21:56:37.178 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.178 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.179 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:37.179 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:19.217479+0000", 2021-03-24T21:56:37.179 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000112874 2021-03-24T21:56:37.179 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.179 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.180 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T21:56:37.180 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:19.217592+0000", 2021-03-24T21:56:37.180 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0098013470000000002 2021-03-24T21:56:37.180 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.180 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.181 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:37.181 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:19.227394+0000", 2021-03-24T21:56:37.181 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.8788000000000005e-05 2021-03-24T21:56:37.181 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.182 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.182 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:56:37.182 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:19.227472+0000", 2021-03-24T21:56:37.182 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.7811000000000003e-05 2021-03-24T21:56:37.182 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.183 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.183 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:56:37.183 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:19.227570+0000", 2021-03-24T21:56:37.183 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.040797951999999998 2021-03-24T21:56:37.183 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:37.184 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:56:37.184 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:37.184 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.184 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.185 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:1555 7.13 7:c8d5c1d6:::smithi12556470-12:head [copy-from ver 30] snapc 69=[69,68] ondisk+write+known_if_redirected e211)", 2021-03-24T21:56:37.185 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:56:20.160908+0000", 2021-03-24T21:56:37.185 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 16.649760344000001, 2021-03-24T21:56:37.185 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.029536323, 2021-03-24T21:56:37.185 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:56:37.186 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:37.186 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:56:37.186 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:56:37.186 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:56:37.187 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1555 2021-03-24T21:56:37.187 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.187 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:56:37.187 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.187 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:56:37.188 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:20.160908+0000", 2021-03-24T21:56:37.188 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:56:37.188 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.188 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.188 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:56:37.189 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:20.160908+0000", 2021-03-24T21:56:37.189 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999957 2021-03-24T21:56:37.189 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.189 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.190 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:56:37.190 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:20.160904+0000", 2021-03-24T21:56:37.190 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.6645e-05 2021-03-24T21:56:37.190 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.190 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.191 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:56:37.191 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:20.160930+0000", 2021-03-24T21:56:37.191 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.7725000000000002e-05 2021-03-24T21:56:37.191 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.191 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.192 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:56:37.192 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:20.160968+0000", 2021-03-24T21:56:37.192 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.4076e-05 2021-03-24T21:56:37.192 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.193 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.193 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:56:37.193 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:20.160982+0000", 2021-03-24T21:56:37.193 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2297000000000002e-05 2021-03-24T21:56:37.193 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.194 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.194 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:37.194 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:20.161064+0000", 2021-03-24T21:56:37.194 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0766e-05 2021-03-24T21:56:37.194 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.195 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.195 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op must wait for map", 2021-03-24T21:56:37.195 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:20.161075+0000", 2021-03-24T21:56:37.195 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011188359 2021-03-24T21:56:37.196 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.196 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.196 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:37.196 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:20.172263+0000", 2021-03-24T21:56:37.196 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00016382500000000001 2021-03-24T21:56:37.197 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.197 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.197 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:56:37.197 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:20.172427+0000", 2021-03-24T21:56:37.197 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0089347820000000005 2021-03-24T21:56:37.198 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.198 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.198 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,7", 2021-03-24T21:56:37.198 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:20.181362+0000", 2021-03-24T21:56:37.199 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0059248499999999997 2021-03-24T21:56:37.199 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.199 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.199 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:37.199 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:20.187287+0000", 2021-03-24T21:56:37.200 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0029657009999999998 2021-03-24T21:56:37.200 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.200 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.200 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:37.201 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:20.190253+0000", 2021-03-24T21:56:37.201 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2917e-05 2021-03-24T21:56:37.201 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.201 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.201 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:56:37.202 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:20.190266+0000", 2021-03-24T21:56:37.202 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.9473000000000003e-05 2021-03-24T21:56:37.202 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.202 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.203 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:56:37.203 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:20.190345+0000", 2021-03-24T21:56:37.203 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.9297999999999997e-05 2021-03-24T21:56:37.203 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.203 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.204 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:56:37.204 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:20.190444+0000", 2021-03-24T21:56:37.204 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.029536323 2021-03-24T21:56:37.204 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:37.204 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:56:37.205 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:37.205 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.205 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.205 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:1556 7.13 7:c8d5c1d6:::smithi12556470-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e211)", 2021-03-24T21:56:37.205 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:56:20.161012+0000", 2021-03-24T21:56:37.206 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 16.64965638, 2021-03-24T21:56:37.206 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.029722999999999999, 2021-03-24T21:56:37.206 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:56:37.206 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T21:56:37.207 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:56:37.207 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:56:37.207 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:56:37.207 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1556 2021-03-24T21:56:37.207 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.208 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:56:37.208 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.208 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:56:37.208 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:20.161012+0000", 2021-03-24T21:56:37.208 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:56:37.209 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.209 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.209 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:56:37.209 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:20.161012+0000", 2021-03-24T21:56:37.210 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T21:56:37.210 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.210 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.210 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:56:37.210 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:20.161011+0000", 2021-03-24T21:56:37.211 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2039999999999996e-06 2021-03-24T21:56:37.211 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.211 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.211 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:56:37.212 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:20.161016+0000", 2021-03-24T21:56:37.212 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.8500000000000006e-06 2021-03-24T21:56:37.212 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.212 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.212 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:56:37.213 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:20.161026+0000", 2021-03-24T21:56:37.213 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.3500000000000002e-06 2021-03-24T21:56:37.213 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.213 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.213 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:56:37.214 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:20.161032+0000", 2021-03-24T21:56:37.214 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000102345 2021-03-24T21:56:37.214 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.214 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.215 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:37.215 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:20.161135+0000", 2021-03-24T21:56:37.215 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.3780000000000007e-06 2021-03-24T21:56:37.215 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.215 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.216 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting_for_map not empty", 2021-03-24T21:56:37.216 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:20.161144+0000", 2021-03-24T21:56:37.216 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011450391000000001 2021-03-24T21:56:37.216 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.217 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.217 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:37.217 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:20.172594+0000", 2021-03-24T21:56:37.217 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6276999999999997e-05 2021-03-24T21:56:37.217 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.218 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.218 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-24T21:56:37.218 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:20.172651+0000", 2021-03-24T21:56:37.218 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.014220534 2021-03-24T21:56:37.218 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.219 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.219 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:37.219 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:20.186871+0000", 2021-03-24T21:56:37.219 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000126346 2021-03-24T21:56:37.219 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.220 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.220 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T21:56:37.220 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:20.186998+0000", 2021-03-24T21:56:37.220 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0035011389999999999 2021-03-24T21:56:37.221 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.221 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.221 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:37.221 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:20.190499+0000", 2021-03-24T21:56:37.221 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00011084800000000001 2021-03-24T21:56:37.222 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.222 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.222 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:56:37.222 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:20.190610+0000", 2021-03-24T21:56:37.223 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000125351 2021-03-24T21:56:37.223 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:37.223 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:37.223 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:56:37.223 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:20.190735+0000", 2021-03-24T21:56:37.224 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.029722999999999999 2021-03-24T21:56:37.224 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:37.224 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:56:37.224 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:37.224 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:37.225 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:56:37.225 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-24T21:56:37.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:37.227 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:37.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:56:37.228 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:56:37.228 INFO:tasks.rados.rados.0.smithi125.stdout:654: done (2 left) 2021-03-24T21:56:37.228 INFO:tasks.rados.rados.0.smithi125.stdout:655: done (1 left) 2021-03-24T21:56:37.229 INFO:tasks.rados.rados.0.smithi125.stdout:656: done (0 left) 2021-03-24T21:56:37.229 INFO:tasks.rados.rados.0.smithi125.stdout:657: snap_create 2021-03-24T21:56:37.329 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:37.384 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:37.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:37.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:56:37.490 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:37.539 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:37.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:37.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:56:37.645 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:37.700 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:37.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:37.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:56:37.806 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:37.866 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:37.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:37.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:56:37.973 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:38.028 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:38.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:38.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:56:38.136 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:38.144 INFO:tasks.rados.rados.0.smithi125.stdout:657: done (0 left) 2021-03-24T21:56:38.145 INFO:tasks.rados.rados.0.smithi125.stdout:658: read oid 27 snap -1 2021-03-24T21:56:38.145 INFO:tasks.rados.rados.0.smithi125.stdout:658: expect (ObjNum 193 snap 42 seq_num 193) 2021-03-24T21:56:38.145 INFO:tasks.rados.rados.0.smithi125.stdout:659: write oid 49 current snap is 63 2021-03-24T21:56:38.145 INFO:tasks.rados.rados.0.smithi125.stdout:659: seq_num 260 ranges {769820=720742,2114730=255090} 2021-03-24T21:56:38.148 INFO:tasks.rados.rados.0.smithi125.stdout:659: writing smithi12556470-49 from 769820 to 1490562 tid 1 2021-03-24T21:56:38.155 INFO:tasks.rados.rados.0.smithi125.stdout:659: writing smithi12556470-49 from 2114730 to 2369820 tid 2 2021-03-24T21:56:38.156 INFO:tasks.rados.rados.0.smithi125.stdout:660: delete oid 10 current snap is 63 2021-03-24T21:56:38.164 INFO:tasks.rados.rados.0.smithi125.stdout:658: expect (ObjNum 193 snap 42 seq_num 193) 2021-03-24T21:56:38.191 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:38.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:38.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:56:38.294 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:38.306 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:38 smithi125 conmon[19936]: cluster 2021-03-24T21:56:36.752651+0000 mgr.y (mgr.14144) 377 : 2021-03-24T21:56:38.307 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:38 smithi125 conmon[19936]: cluster [DBG] pgmap v503: 158 pgs: 158 active+clean; 124 MiB data, 439 MiB used, 536 GiB / 536 GiB avail; 11 MiB/s rd, 5.9 MiB/s wr, 21 op/s 2021-03-24T21:56:38.307 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:38 smithi125 conmon[19936]: cluster 2021-03-24T21:56:37.132829+0000 mon.a (mon.0) 842 : cluster [DBG] osdmap e228: 8 total, 8 up, 6 in 2021-03-24T21:56:38.307 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:38 smithi125 conmon[24927]: cluster 2021-03-24T21:56:38.308 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:38 smithi125 conmon[24927]: 2021- 2021-03-24T21:56:38.308 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:38 smithi125 conmon[24927]: 03-24T21:56:36.752651+0000 mgr.y (mgr.14144) 377 : cluster [DBG] pgmap v503: 158 pgs: 158 active+clean; 124 MiB data, 439 MiB used, 536 GiB / 536 GiB avail; 11 MiB/s rd, 5.9 MiB/s wr, 21 op/s 2021-03-24T21:56:38.308 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:38 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:56:38.308 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:38 smithi125 conmon[24927]: 03-24T21:56:37.132829+0000 mon.a (mon.0 2021-03-24T21:56:38.309 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:38 smithi125 conmon[24927]: ) 842 : cluster [DBG] osdmap e228: 8 total, 8 up, 6 in 2021-03-24T21:56:38.339 INFO:tasks.rados.rados.0.smithi125.stdout:659: finishing write tid 1 to smithi12556470-49 2021-03-24T21:56:38.340 INFO:tasks.rados.rados.0.smithi125.stdout:659: finishing write tid 2 to smithi12556470-49 2021-03-24T21:56:38.340 INFO:tasks.rados.rados.0.smithi125.stdout:659: finishing write tid 3 to smithi12556470-49 2021-03-24T21:56:38.340 INFO:tasks.rados.rados.0.smithi125.stdout:659: finishing write tid 4 to smithi12556470-49 2021-03-24T21:56:38.340 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 49 v 213 (ObjNum 260 snap 63 seq_num 260) dirty exists 2021-03-24T21:56:38.340 INFO:tasks.rados.rados.0.smithi125.stdout:659: left oid 49 (ObjNum 260 snap 63 seq_num 260) 2021-03-24T21:56:38.341 INFO:tasks.rados.rados.0.smithi125.stdout:658: done (2 left) 2021-03-24T21:56:38.341 INFO:tasks.rados.rados.0.smithi125.stdout:659: done (1 left) 2021-03-24T21:56:38.341 INFO:tasks.rados.rados.0.smithi125.stdout:660: done (0 left) 2021-03-24T21:56:38.341 INFO:tasks.rados.rados.0.smithi125.stdout:661: rollback oid 17 current snap is 63 2021-03-24T21:56:38.342 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 17 to 60 2021-03-24T21:56:38.342 INFO:tasks.rados.rados.0.smithi125.stdout:662: snap_remove snap 62 2021-03-24T21:56:38.347 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:38.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:38.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:56:38.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:38 smithi142 conmon[20125]: cluster 2021-03-24T21:56:36.752651+0000 mgr.y (mgr. 2021-03-24T21:56:38.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:38 smithi142 conmon[20125]: 14144) 377 : cluster [DBG] pgmap v503: 158 pgs: 158 active+clean; 124 MiB data, 439 MiB used, 536 GiB / 536 GiB avail; 11 MiB/s rd, 5.9 MiB/s wr, 21 op/s 2021-03-24T21:56:38.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:38 smithi142 conmon[20125]: cluster 2021-03-24T21:56:37.132829+0000 mon.a (mon.0) 842 : cluster [DBG] osdmap e228: 8 total, 8 up, 6 in 2021-03-24T21:56:38.452 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:38.497 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:38.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:38.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:56:38.603 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:38.659 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:38.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:38.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:56:38.764 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:38.809 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:38.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:38.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:56:38.915 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:38.976 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:38.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:38.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:56:39.081 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:39.141 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:39.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:39.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:56:39.152 INFO:tasks.rados.rados.0.smithi125.stdout:662: done (1 left) 2021-03-24T21:56:39.152 INFO:tasks.rados.rados.0.smithi125.stdout:663: read oid 13 snap -1 2021-03-24T21:56:39.152 INFO:tasks.rados.rados.0.smithi125.stdout:663: expect (ObjNum 251 snap 59 seq_num 251) 2021-03-24T21:56:39.153 INFO:tasks.rados.rados.0.smithi125.stdout:664: snap_remove snap 61 2021-03-24T21:56:39.161 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: cluster 2021-03-24T21:56:39.161 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 2021-03-24T21:56:39.162 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 2021 2021-03-24T21:56:39.162 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: - 2021-03-24T21:56:39.162 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 03 2021-03-24T21:56:39.162 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: - 2021-03-24T21:56:39.163 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 24 2021-03-24T21:56:39.163 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: T 2021-03-24T21:56:39.163 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 21 2021-03-24T21:56:39.163 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: : 2021-03-24T21:56:39.163 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 56 2021-03-24T21:56:39.164 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: : 2021-03-24T21:56:39.164 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 37 2021-03-24T21:56:39.164 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: . 2021-03-24T21:56:39.164 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 972858 2021-03-24T21:56:39.165 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: +0000 2021-03-24T21:56:39.165 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 2021-03-24T21:56:39.165 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: osd.7 2021-03-24T21:56:39.165 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: ( 2021-03-24T21:56:39.165 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: osd 2021-03-24T21:56:39.166 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: . 2021-03-24T21:56:39.166 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 7 2021-03-24T21:56:39.166 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: ) 2021-03-24T21:56:39.166 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 52 2021-03-24T21:56:39.167 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: : 2021-03-24T21:56:39.167 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: cluster 2021-03-24T21:56:39.167 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 2021-03-24T21:56:39.167 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: [DBG] 2021-03-24T21:56:39.167 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 2021-03-24T21:56:39.168 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 2.0 scrub ok 2021-03-24T21:56:39.168 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 2021-03-24T21:56:39.168 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: cluster 2021-03-24T21:56:39.168 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 2021-03-24T21:56:39.169 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 2021 2021-03-24T21:56:39.169 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: - 2021-03-24T21:56:39.169 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 03 2021-03-24T21:56:39.169 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: - 2021-03-24T21:56:39.169 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 24 2021-03-24T21:56:39.170 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: T 2021-03-24T21:56:39.170 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 21 2021-03-24T21:56:39.170 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: : 2021-03-24T21:56:39.170 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 56 2021-03-24T21:56:39.171 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: : 2021-03-24T21:56:39.171 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 38 2021-03-24T21:56:39.171 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: . 2021-03-24T21:56:39.171 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 075698 2021-03-24T21:56:39.171 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: +0000 2021-03-24T21:56:39.172 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 2021-03-24T21:56:39.247 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:39.261 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:39 smithi142 conmon[20125]: cluster 2021-03-24T21:56:37.972858+0000 osd.7 (osd.7) 2021-03-24T21:56:39.261 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:39 smithi142 conmon[20125]: 52 : cluster [DBG] 2.0 scrub ok 2021-03-24T21:56:39.262 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:39 smithi142 conmon[20125]: cluster 2021-03-24T21:56:38.075698+0000 osd.4 (osd.4) 63 : cluster [DBG] 6.1e scrub ok 2021-03-24T21:56:39.262 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:39 smithi142 conmon[20125]: cluster 2021-03-24T21:56: 2021-03-24T21:56:39.262 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:39 smithi142 conmon[20125]: 38.136268+0000 mon.a (mon.0) 843 : cluster [DBG] osdmap e229: 8 total, 8 up, 6 in 2021-03-24T21:56:39.262 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:39 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:56:39.263 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:39 smithi142 conmon[20125]: -24T21:56:38.144952+0000 osd.2 (osd.2) 43 : cluster 2021-03-24T21:56:39.263 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:39 smithi142 conmon[20125]: [DBG] 3.4 scrub ok 2021-03-24T21:56:39.264 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:39 smithi142 conmon[20125]: cluster 2021-03-24T21:56:38.242088 2021-03-24T21:56:39.264 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:39 smithi142 conmon[20125]: +0000 osd.5 (osd.5) 59 : cluster [DBG] 5.1b scrub ok 2021-03-24T21:56:39.264 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:39 smithi142 conmon[20125]: 2021-03-24T21:56:39.264 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:39 smithi142 conmon[20125]: cluster 2021-03-24T21:56:39.264 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:39 smithi142 conmon[20125]: 2021-03-24T21:56:38.304778+0000 osd.0 (osd.0) 59 : 2021-03-24T21:56:39.265 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:39 smithi142 conmon[20125]: cluster [DBG] 3.1 scrub ok 2021-03-24T21:56:39.266 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.1 dump_ops_in_flight 2021-03-24T21:56:39.304 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:39.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:39.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:56:39.410 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:39.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[24927]: cluster 2021-03-24T21:56:39.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[24927]: 2021-03-24T21:56:39.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[24927]: 2021 2021-03-24T21:56:39.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[24927]: - 2021-03-24T21:56:39.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[24927]: 03 2021-03-24T21:56:39.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[24927]: - 2021-03-24T21:56:39.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[24927]: 24T21:56:37. 2021-03-24T21:56:39.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[24927]: 972858+0000 osd.7 (osd.7) 52 : cluster [DBG] 2.0 scrub ok 2021-03-24T21:56:39.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[24927]: cluster 2021-03-24T21:56:38.075698+0000 osd.4 (osd.4) 63 : cluster [DBG] 6.1e scrub ok 2021-03-24T21:56:39.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[24927]: cluster 2021-03-24T21:56:38.136268+0000 mon.a (mon.0) 843 : cluster [DBG] 2021-03-24T21:56:39.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[24927]: osdmap e229: 8 total, 8 up, 6 in 2021-03-24T21:56:39.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[24927]: cluster 2021-03-24T21:56:38. 2021-03-24T21:56:39.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[24927]: 144952+0000 osd.2 (osd.2) 43 : cluster [DBG] 3.4 scrub ok 2021-03-24T21:56:39.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:56:39.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[24927]: 21:56:38.242088+0000 osd.5 (osd.5) 59 2021-03-24T21:56:39.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[24927]: : cluster [DBG] 5.1b scrub ok 2021-03-24T21:56:39.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[24927]: cluster 2021-03-24T21:56: 2021-03-24T21:56:39.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[24927]: 38.304778+0000 osd.0 (osd.0) 59 : cluster 2021-03-24T21:56:39.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[24927]: [DBG] 3.1 scrub ok 2021-03-24T21:56:39.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: osd.4 2021-03-24T21:56:39.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: ( 2021-03-24T21:56:39.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: osd 2021-03-24T21:56:39.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: . 2021-03-24T21:56:39.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 4 2021-03-24T21:56:39.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: ) 2021-03-24T21:56:39.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 63 2021-03-24T21:56:39.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: : 2021-03-24T21:56:39.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: cluster 2021-03-24T21:56:39.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 2021-03-24T21:56:39.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: [DBG] 2021-03-24T21:56:39.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 2021-03-24T21:56:39.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 6.1e scrub ok 2021-03-24T21:56:39.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 2021-03-24T21:56:39.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: cluster 2021-03-24T21:56:39.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 2021-03-24T21:56:39.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 2021 2021-03-24T21:56:39.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: - 2021-03-24T21:56:39.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 03 2021-03-24T21:56:39.421 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: - 2021-03-24T21:56:39.421 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 24 2021-03-24T21:56:39.421 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: T 2021-03-24T21:56:39.421 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 21 2021-03-24T21:56:39.422 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: : 2021-03-24T21:56:39.422 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 56 2021-03-24T21:56:39.422 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: : 2021-03-24T21:56:39.422 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 38 2021-03-24T21:56:39.423 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: . 2021-03-24T21:56:39.423 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 136268 2021-03-24T21:56:39.423 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: +0000 2021-03-24T21:56:39.423 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 2021-03-24T21:56:39.423 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: mon.a 2021-03-24T21:56:39.424 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: ( 2021-03-24T21:56:39.424 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: mon 2021-03-24T21:56:39.424 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: . 2021-03-24T21:56:39.424 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 0 2021-03-24T21:56:39.425 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: ) 2021-03-24T21:56:39.425 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 843 2021-03-24T21:56:39.425 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: : 2021-03-24T21:56:39.425 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: cluster 2021-03-24T21:56:39.425 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 2021-03-24T21:56:39.426 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: [DBG] 2021-03-24T21:56:39.426 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 2021-03-24T21:56:39.426 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: osdmap e229: 8 total, 8 up, 6 in 2021-03-24T21:56:39.426 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 2021-03-24T21:56:39.427 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: cluster 2021-03-24T21:56:39.427 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 2021-03-24T21:56:39.427 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 2021 2021-03-24T21:56:39.427 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: - 2021-03-24T21:56:39.427 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 03 2021-03-24T21:56:39.428 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: - 2021-03-24T21:56:39.428 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 24 2021-03-24T21:56:39.428 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: T 2021-03-24T21:56:39.428 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 21 2021-03-24T21:56:39.428 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: : 2021-03-24T21:56:39.429 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 56 2021-03-24T21:56:39.429 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: : 2021-03-24T21:56:39.429 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 38 2021-03-24T21:56:39.429 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: . 2021-03-24T21:56:39.430 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 144952 2021-03-24T21:56:39.430 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: +0000 2021-03-24T21:56:39.430 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 2021-03-24T21:56:39.430 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: osd.2 2021-03-24T21:56:39.431 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: ( 2021-03-24T21:56:39.431 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: osd 2021-03-24T21:56:39.431 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: . 2021-03-24T21:56:39.431 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 2 2021-03-24T21:56:39.432 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: ) 2021-03-24T21:56:39.432 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 43 2021-03-24T21:56:39.432 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: : 2021-03-24T21:56:39.432 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: cluster 2021-03-24T21:56:39.433 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 2021-03-24T21:56:39.433 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: [DBG] 2021-03-24T21:56:39.433 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 2021-03-24T21:56:39.433 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 3.4 scrub ok 2021-03-24T21:56:39.434 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 2021-03-24T21:56:39.434 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: cluster 2021-03-24T21:56:39.434 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 2021-03-24T21:56:39.434 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 2021 2021-03-24T21:56:39.434 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: - 2021-03-24T21:56:39.435 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 03 2021-03-24T21:56:39.435 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: - 2021-03-24T21:56:39.435 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 24 2021-03-24T21:56:39.435 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: T 2021-03-24T21:56:39.435 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 21 2021-03-24T21:56:39.436 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: : 2021-03-24T21:56:39.436 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 56 2021-03-24T21:56:39.436 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: : 2021-03-24T21:56:39.436 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 38 2021-03-24T21:56:39.437 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: . 2021-03-24T21:56:39.437 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 242088 2021-03-24T21:56:39.437 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: +0000 2021-03-24T21:56:39.437 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 2021-03-24T21:56:39.438 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: osd.5 2021-03-24T21:56:39.438 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: ( 2021-03-24T21:56:39.438 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: osd 2021-03-24T21:56:39.438 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: . 2021-03-24T21:56:39.439 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 5 2021-03-24T21:56:39.439 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: ) 2021-03-24T21:56:39.439 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 59 2021-03-24T21:56:39.439 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: : 2021-03-24T21:56:39.440 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: cluster 2021-03-24T21:56:39.440 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 2021-03-24T21:56:39.440 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: [DBG] 2021-03-24T21:56:39.440 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 2021-03-24T21:56:39.440 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 5.1b scrub ok 2021-03-24T21:56:39.441 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 2021-03-24T21:56:39.441 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: cluster 2021-03-24T21:56:39.441 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 2021-03-24T21:56:39.441 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 2021 2021-03-24T21:56:39.442 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: - 2021-03-24T21:56:39.442 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 03 2021-03-24T21:56:39.442 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: - 2021-03-24T21:56:39.442 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 24 2021-03-24T21:56:39.442 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: T 2021-03-24T21:56:39.443 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 21 2021-03-24T21:56:39.443 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: : 2021-03-24T21:56:39.443 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 56 2021-03-24T21:56:39.443 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: : 2021-03-24T21:56:39.444 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 38 2021-03-24T21:56:39.444 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: . 2021-03-24T21:56:39.444 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 304778 2021-03-24T21:56:39.444 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: +0000 2021-03-24T21:56:39.444 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 2021-03-24T21:56:39.445 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: osd.0 2021-03-24T21:56:39.445 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: ( 2021-03-24T21:56:39.445 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: osd 2021-03-24T21:56:39.445 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: . 2021-03-24T21:56:39.446 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 0 2021-03-24T21:56:39.446 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: ) 2021-03-24T21:56:39.446 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 59 2021-03-24T21:56:39.446 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: : 2021-03-24T21:56:39.446 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: cluster 2021-03-24T21:56:39.447 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 2021-03-24T21:56:39.447 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: [DBG] 2021-03-24T21:56:39.447 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 2021-03-24T21:56:39.447 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 3.1 scrub ok 2021-03-24T21:56:39.447 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:39 smithi125 conmon[19936]: 2021-03-24T21:56:39.466 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:39.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:39.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:56:39.572 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:39.632 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:39.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:39.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:56:39.739 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:39.791 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:39.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:39.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:56:39.897 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:39.953 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:39.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:39.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:56:40.059 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:40.116 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:40.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:40.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:56:40.137 INFO:tasks.rados.rados.0.smithi125.stdout:664: done (2 left) 2021-03-24T21:56:40.137 INFO:tasks.rados.rados.0.smithi125.stdout:665: read oid 27 snap -1 2021-03-24T21:56:40.138 INFO:tasks.rados.rados.0.smithi125.stdout:665: expect (ObjNum 193 snap 42 seq_num 193) 2021-03-24T21:56:40.138 INFO:tasks.rados.rados.0.smithi125.stdout:666: copy_from oid 26 from oid 8 current snap is 63 2021-03-24T21:56:40.138 INFO:tasks.rados.rados.0.smithi125.stdout:661: finishing rollback tid 0 to smithi12556470-17 2021-03-24T21:56:40.138 INFO:tasks.rados.rados.0.smithi125.stdout:661: finishing rollback tid 1 to smithi12556470-17 2021-03-24T21:56:40.139 INFO:tasks.rados.rados.0.smithi125.stdout:661: finishing rollback tid 2 to smithi12556470-17 2021-03-24T21:56:40.139 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 17 v 202 (ObjNum 123 snap 23 seq_num 123) dirty exists 2021-03-24T21:56:40.139 INFO:tasks.rados.rados.0.smithi125.stdout:663: expect (ObjNum 251 snap 59 seq_num 251) 2021-03-24T21:56:40.219 INFO:tasks.rados.rados.0.smithi125.stdout:665: expect (ObjNum 193 snap 42 seq_num 193) 2021-03-24T21:56:40.223 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:40.276 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:40.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:40.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:56:40.345 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:40 smithi125 conmon[19936]: cluster 2021-03-24T21:56:38.753670+0000 mgr.y (mgr 2021-03-24T21:56:40.346 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:40 smithi125 conmon[19936]: .14144) 378 : cluster [DBG] pgmap v506: 158 pgs: 1 peering, 157 active+clean; 126 MiB data, 438 MiB used, 536 GiB / 536 GiB avail; 275 KiB/s wr, 0 op/s; 1.6 MiB/s, 3 objects/s recovering 2021-03-24T21:56:40.346 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:40 smithi125 conmon[19936]: cluster 2021-03-24T21:56:38.805522+0000 osd.7 (osd.7) 53 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:56:40.346 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:40 smithi125 conmon[19936]: cluster 2021-03-24T21:56:38.805848+0000 osd.7 (osd.7 2021-03-24T21:56:40.347 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:40 smithi125 conmon[19936]: ) 54 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:56:40.347 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:40 smithi125 conmon[19936]: cluster 2021-03-24T21:56:40.347 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:40 smithi125 conmon[19936]: 2021-03-24T21:56:38.970530+0000 osd.7 (osd.7 2021-03-24T21:56:40.347 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:40 smithi125 conmon[19936]: ) 55 : cluster [DBG] 6.8 scrub ok 2021-03-24T21:56:40.347 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:40 smithi125 conmon[19936]: cluster 2021-03-24T21:56: 2021-03-24T21:56:40.348 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:40 smithi125 conmon[19936]: 39.150780+0000 mon.a (mon.0) 844 : cluster [DBG] 2021-03-24T21:56:40.348 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:40 smithi125 conmon[19936]: osdmap e230: 8 total, 8 up, 6 in 2021-03-24T21:56:40.348 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:40 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:56:40.348 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:40 smithi125 conmon[19936]: 03-24T21:56:39.335605+0000 osd.0 (osd.0) 60 : 2021-03-24T21:56:40.348 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:40 smithi125 conmon[19936]: cluster [DBG] 4.3 scrub ok 2021-03-24T21:56:40.349 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:40 smithi125 conmon[19936]: cluster 2021-03-24T21:56:40.349 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:40 smithi125 conmon[19936]: 2021-03-24T21:56:40.129641+0000 mon.a (mon 2021-03-24T21:56:40.349 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:40 smithi125 conmon[19936]: .0) 845 : cluster [DBG] osdmap e231: 8 total, 8 up, 6 in 2021-03-24T21:56:40.350 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:40 smithi125 conmon[24927]: cluster 2021-03-24T21:56:38.753670+0000 mgr.y (mgr.14144 2021-03-24T21:56:40.350 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:40 smithi125 conmon[24927]: ) 378 : cluster [DBG] pgmap v506: 158 pgs: 1 peering, 157 active+clean; 126 MiB data, 438 MiB used, 536 GiB / 536 GiB avail; 275 KiB/s wr, 0 op/s; 1.6 MiB/s, 3 objects/s recovering 2021-03-24T21:56:40.350 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:40 smithi125 conmon[24927]: cluster 2021-03-24T21:56:38.805522+0000 osd.7 (osd.7) 53 : cluster 2021-03-24T21:56:40.350 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:40 smithi125 conmon[24927]: [DBG] purged_snaps scrub starts 2021-03-24T21:56:40.351 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:40 smithi125 conmon[24927]: cluster 2021-03-24T21:56:38.805848+0000 osd.7 (osd 2021-03-24T21:56:40.351 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:40 smithi125 conmon[24927]: .7) 54 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:56:40.351 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:40 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:56:40.351 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:40 smithi125 conmon[24927]: 24T21:56:38.970530+0000 osd.7 (osd.7) 55 : cluster [DBG] 6.8 scrub ok 2021-03-24T21:56:40.351 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:40 smithi125 conmon[24927]: cluster 2021-03-24T21:56 2021-03-24T21:56:40.352 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:40 smithi125 conmon[24927]: :39.150780+0000 mon.a (mon.0) 844 : cluster [DBG] osdmap e230: 8 total, 8 up, 6 in 2021-03-24T21:56:40.352 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:40 smithi125 conmon[24927]: cluster 2021-03-24T21:56 2021-03-24T21:56:40.352 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:40 smithi125 conmon[24927]: :39.335605+0000 osd.0 (osd.0) 60 : cluster [DBG] 4.3 scrub ok 2021-03-24T21:56:40.352 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:40 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:56:40.353 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:40 smithi125 conmon[24927]: :56:40.129641+0000 mon.a (mon.0) 845 : cluster [DBG] osdmap e231: 8 total, 8 up, 6 in 2021-03-24T21:56:40.353 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:40 smithi125 conmon[24927]: 2021-03-24T21:56:40.383 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:40.406 INFO:tasks.rados.rados.0.smithi125.stdout:666: finishing copy_from to smithi12556470-26 2021-03-24T21:56:40.407 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 26 v 33 (ObjNum 250 snap 59 seq_num 250) dirty exists 2021-03-24T21:56:40.407 INFO:tasks.rados.rados.0.smithi125.stdout:666: finishing copy_from racing read to smithi12556470-26 2021-03-24T21:56:40.407 INFO:tasks.rados.rados.0.smithi125.stdout:661: done (3 left) 2021-03-24T21:56:40.407 INFO:tasks.rados.rados.0.smithi125.stdout:663: done (2 left) 2021-03-24T21:56:40.408 INFO:tasks.rados.rados.0.smithi125.stdout:665: done (1 left) 2021-03-24T21:56:40.408 INFO:tasks.rados.rados.0.smithi125.stdout:666: done (0 left) 2021-03-24T21:56:40.408 INFO:tasks.rados.rados.0.smithi125.stdout:667: copy_from oid 19 from oid 45 current snap is 63 2021-03-24T21:56:40.408 INFO:tasks.rados.rados.0.smithi125.stdout:668: copy_from oid 9 from oid 34 current snap is 63 2021-03-24T21:56:40.408 INFO:tasks.rados.rados.0.smithi125.stdout:669: write (excl) oid 47 current snap is 63 2021-03-24T21:56:40.409 INFO:tasks.rados.rados.0.smithi125.stdout:669: seq_num 262 ranges {403434=471240,1315341=400756} 2021-03-24T21:56:40.414 INFO:tasks.rados.rados.0.smithi125.stdout:669: writing smithi12556470-47 from 403434 to 874674 tid 1 2021-03-24T21:56:40.422 INFO:tasks.rados.rados.0.smithi125.stdout:669: writing smithi12556470-47 from 1315341 to 1716097 tid 2 2021-03-24T21:56:40.423 INFO:tasks.rados.rados.0.smithi125.stdout:670: snap_create 2021-03-24T21:56:40.425 INFO:tasks.rados.rados.0.smithi125.stdout:669: finishing write tid 1 to smithi12556470-47 2021-03-24T21:56:40.438 INFO:tasks.rados.rados.0.smithi125.stdout:667: finishing copy_from to smithi12556470-19 2021-03-24T21:56:40.438 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 19 v 151 (ObjNum 150 snap 30 seq_num 150) dirty exists 2021-03-24T21:56:40.439 INFO:tasks.rados.rados.0.smithi125.stdout:667: finishing copy_from racing read to smithi12556470-19 2021-03-24T21:56:40.439 INFO:tasks.rados.rados.0.smithi125.stdout:669: finishing write tid 2 to smithi12556470-47 2021-03-24T21:56:40.439 INFO:tasks.rados.rados.0.smithi125.stdout:668: finishing copy_from to smithi12556470-9 2021-03-24T21:56:40.439 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 9 v 193 (ObjNum 203 snap 43 seq_num 203) dirty exists 2021-03-24T21:56:40.440 INFO:tasks.rados.rados.0.smithi125.stdout:669: finishing write tid 3 to smithi12556470-47 2021-03-24T21:56:40.440 INFO:tasks.rados.rados.0.smithi125.stdout:668: finishing copy_from racing read to smithi12556470-9 2021-03-24T21:56:40.440 INFO:tasks.rados.rados.0.smithi125.stdout:669: finishing write tid 4 to smithi12556470-47 2021-03-24T21:56:40.440 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 47 v 95 (ObjNum 262 snap 63 seq_num 262) dirty exists 2021-03-24T21:56:40.440 INFO:tasks.rados.rados.0.smithi125.stdout:669: left oid 47 (ObjNum 262 snap 63 seq_num 262) 2021-03-24T21:56:40.441 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:40.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:40.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:56:40.544 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:40.602 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:40.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:40.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:56:40.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:40 smithi142 conmon[20125]: cluster 2021-03-24T21:56:40.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:40 smithi142 conmon[20125]: 2021-03-24T21:56:38 2021-03-24T21:56:40.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:40 smithi142 conmon[20125]: .753670+0000 mgr.y (mgr.14144) 378 : cluster [DBG] pgmap v506: 158 pgs: 1 peering, 157 active+clean; 126 MiB data, 438 MiB used, 536 GiB / 536 GiB avail; 275 KiB/s wr, 0 op/s; 1.6 MiB/s, 3 objects/s recovering 2021-03-24T21:56:40.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:40 smithi142 conmon[20125]: cluster 2021-03-24T21:56 2021-03-24T21:56:40.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:40 smithi142 conmon[20125]: :38.805522+0000 osd.7 (osd.7) 53 2021-03-24T21:56:40.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:40 smithi142 conmon[20125]: : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:56:40.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:40 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:56:40.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:40 smithi142 conmon[20125]: -24T21:56:38.805848+0000 osd.7 (osd. 2021-03-24T21:56:40.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:40 smithi142 conmon[20125]: 7) 54 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:56:40.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:40 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:56:40.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:40 smithi142 conmon[20125]: -03-24T21:56:38.970530+0000 osd.7 ( 2021-03-24T21:56:40.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:40 smithi142 conmon[20125]: osd.7) 55 : cluster [DBG] 6.8 scrub ok 2021-03-24T21:56:40.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:40 smithi142 conmon[20125]: cluster 2021-03-24T21:56:40.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:40 smithi142 conmon[20125]: 2021-03-24T21:56:40.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:40 smithi142 conmon[20125]: 2021-03-24T21:56:39 2021-03-24T21:56:40.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:40 smithi142 conmon[20125]: .150780+0000 mon.a (mon.0) 844 : cluster [DBG] osdmap e230: 8 total, 8 up, 6 in 2021-03-24T21:56:40.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:40 smithi142 conmon[20125]: cluster 2021-03-24T21:56:40.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:40 smithi142 conmon[20125]: 2021-03-24T21:56:40.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:40 smithi142 conmon[20125]: 2021-03-24T21:56:39. 2021-03-24T21:56:40.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:40 smithi142 conmon[20125]: 335605+0000 osd.0 (osd.0) 60 : cluster [DBG] 4.3 scrub ok 2021-03-24T21:56:40.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:40 smithi142 conmon[20125]: cluster 2021-03-24T21:56:40.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:40 smithi142 conmon[20125]: 2021-03-24T21:56: 2021-03-24T21:56:40.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:40 smithi142 conmon[20125]: 40.129641+0000 mon.a (mon.0) 845 : cluster [DBG] osdmap e231: 8 total, 8 up, 6 in 2021-03-24T21:56:40.709 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:40.768 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:40.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:40.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:56:40.875 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:40.942 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:40.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:40.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:56:41.047 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:41.102 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:41.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:41.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:56:41.188 INFO:tasks.rados.rados.0.smithi125.stdout:667: done (3 left) 2021-03-24T21:56:41.188 INFO:tasks.rados.rados.0.smithi125.stdout:668: done (2 left) 2021-03-24T21:56:41.189 INFO:tasks.rados.rados.0.smithi125.stdout:669: done (1 left) 2021-03-24T21:56:41.189 INFO:tasks.rados.rados.0.smithi125.stdout:670: done (0 left) 2021-03-24T21:56:41.189 INFO:tasks.rados.rados.0.smithi125.stdout:671: write (excl) oid 24 current snap is 64 2021-03-24T21:56:41.189 INFO:tasks.rados.rados.0.smithi125.stdout:671: seq_num 263 ranges {420241=546489,1373823=544414} 2021-03-24T21:56:41.194 INFO:tasks.rados.rados.0.smithi125.stdout:671: writing smithi12556470-24 from 420241 to 966730 tid 1 2021-03-24T21:56:41.200 INFO:tasks.rados.rados.0.smithi125.stdout:671: writing smithi12556470-24 from 1373823 to 1918237 tid 2 2021-03-24T21:56:41.200 INFO:tasks.rados.rados.0.smithi125.stdout:672: snap_remove snap 60 2021-03-24T21:56:41.207 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:41.258 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:41.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:41.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:56:41.339 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:56:41.365 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:41.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:41 smithi142 conmon[20125]: cluster 2021-03-24T21:56:41.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:41 smithi142 conmon[20125]: 2021-03-24T21:56:41.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:41 smithi142 conmon[20125]: 2021-03-24T21:56:39 2021-03-24T21:56:41.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:41 smithi142 conmon[20125]: .439165+0000 osd.3 (osd.3) 62 : cluster [DBG] 3.8 scrub ok 2021-03-24T21:56:41.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:41 smithi142 conmon[20125]: cluster 2021-03-24T21:56:40.011617+0000 2021-03-24T21:56:41.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:41 smithi142 conmon[20125]: osd.7 (osd.7) 56 : cluster [DBG] 6.3 scrub ok 2021-03-24T21:56:41.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:41 smithi142 conmon[20125]: 2021-03-24T21:56:41.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:41 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:56:41.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:41 smithi142 conmon[20125]: T21:56:40.062294+0000 osd.4 (osd 2021-03-24T21:56:41.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:41 smithi142 conmon[20125]: .4) 64 : cluster [DBG] 4.e scrub ok 2021-03-24T21:56:41.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:41 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:56:41.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:41 smithi142 conmon[20125]: -24T21:56:40.081127+0000 2021-03-24T21:56:41.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:41 smithi142 conmon[20125]: osd.2 (osd.2) 44 : cluster [DBG] 2021-03-24T21:56:41.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:41 smithi142 conmon[20125]: 7.8 scrub ok 2021-03-24T21:56:41.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:41 smithi142 conmon[20125]: cluster 2021-03-24T21:56:41.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:41 smithi142 conmon[20125]: 2021-03-24T21:56:40.276591+0000 osd.5 2021-03-24T21:56:41.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:41 smithi142 conmon[20125]: (osd.5) 60 : cluster [DBG] 7.0 scrub ok 2021-03-24T21:56:41.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:41 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:56:41.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:41 smithi142 conmon[20125]: -24T21:56:40.303193+0000 osd.0 (osd 2021-03-24T21:56:41.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:41 smithi142 conmon[20125]: .0) 61 : cluster [DBG] 4.16 scrub ok 2021-03-24T21:56:41.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:41 smithi142 conmon[20125]: 2021-03-24T21:56:41.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:41 smithi125 conmon[19936]: cluster 2021-03-24T21:56:39.439165+0000 2021-03-24T21:56:41.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:41 smithi125 conmon[19936]: osd.3 (osd.3) 62 : cluster [DBG] 3.8 scrub ok 2021-03-24T21:56:41.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:41 smithi125 conmon[19936]: cluster 2021-03-24T21:56:40.011617+0000 osd.7 (osd.7) 56 : cluster [DBG] 6.3 scrub ok 2021-03-24T21:56:41.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:41 smithi125 conmon[19936]: cluster 2021-03-24T21:56:40.062294+0000 2021-03-24T21:56:41.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:41 smithi125 conmon[19936]: osd.4 (osd.4) 64 : cluster [DBG] 4.e scrub ok 2021-03-24T21:56:41.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:41 smithi125 conmon[19936]: cluster 2021-03-24T21:56:41.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:41 smithi125 conmon[19936]: 2021-03-24T21:56:40.081127+0000 osd.2 (osd 2021-03-24T21:56:41.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:41 smithi125 conmon[19936]: .2) 44 : cluster [DBG] 7.8 scrub ok 2021-03-24T21:56:41.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:41 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:56:41.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:41 smithi125 conmon[19936]: 03-24T21:56:40.276591+0000 osd.5 (osd 2021-03-24T21:56:41.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:41 smithi125 conmon[19936]: .5) 60 : cluster [DBG] 7.0 scrub ok 2021-03-24T21:56:41.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:41 smithi125 conmon[19936]: cluster 2021-03-24T21:56:41.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:41 smithi125 conmon[19936]: 2021-03-24T21:56:40.303193+0000 osd.0 (osd.0) 61 : cluster [DBG] 4.16 scrub ok 2021-03-24T21:56:41.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:41 smithi125 conmon[24927]: cluster 2021-03-24T21:56:39.439165+0000 2021-03-24T21:56:41.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:41 smithi125 conmon[24927]: osd.3 (osd.3) 62 : cluster [DBG] 3.8 scrub ok 2021-03-24T21:56:41.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:41 smithi125 conmon[24927]: cluster 2021-03-24T21:56:40 2021-03-24T21:56:41.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:41 smithi125 conmon[24927]: .011617+0000 osd.7 (osd.7) 56 : cluster [DBG] 6.3 scrub ok 2021-03-24T21:56:41.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:41 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:56:41.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:41 smithi125 conmon[24927]: :56:40.062294+0000 osd.4 (osd.4) 2021-03-24T21:56:41.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:41 smithi125 conmon[24927]: 64 : cluster [DBG] 4.e scrub ok 2021-03-24T21:56:41.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:41 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:56:41.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:41 smithi125 conmon[24927]: -03-24T21:56:40.081127+0000 2021-03-24T21:56:41.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:41 smithi125 conmon[24927]: osd.2 (osd.2) 44 : cluster [DBG] 2021-03-24T21:56:41.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:41 smithi125 conmon[24927]: 7.8 scrub ok 2021-03-24T21:56:41.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:41 smithi125 conmon[24927]: cluster 2021-03-24T21:56: 2021-03-24T21:56:41.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:41 smithi125 conmon[24927]: 40.276591+0000 osd.5 (osd.5) 60 : 2021-03-24T21:56:41.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:41 smithi125 conmon[24927]: cluster [DBG] 7.0 scrub ok 2021-03-24T21:56:41.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:41 smithi125 conmon[24927]: cluster 2021-03-24T21:56 2021-03-24T21:56:41.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:41 smithi125 conmon[24927]: :40.303193+0000 osd.0 (osd.0) 61 2021-03-24T21:56:41.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:41 smithi125 conmon[24927]: : cluster [DBG] 4.16 scrub ok 2021-03-24T21:56:41.417 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:41.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:41.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:56:41.523 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:41.580 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:41.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:41.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:56:41.660 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T21:56:41.681 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T21:56:41.686 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:41.710 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T21:56:41.729 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T21:56:41.741 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:41.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:41.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:56:41.753 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T21:56:41.776 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T21:56:41.798 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T21:56:41.832 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T21:56:41.833 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-24T21:56:41.834 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-24T21:56:41.834 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-24T21:56:41.834 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-24T21:56:41.834 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-24T21:56:41.834 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-24T21:56:41.835 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-24T21:56:41.835 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-24T21:56:41.848 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:41.904 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:41.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:41.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:56:42.012 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:42.057 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:42.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:42.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:56:42.163 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:42.192 INFO:tasks.rados.rados.0.smithi125.stdout:672: done (1 left) 2021-03-24T21:56:42.193 INFO:tasks.rados.rados.0.smithi125.stdout:673: write oid 29 current snap is 64 2021-03-24T21:56:42.193 INFO:tasks.rados.rados.0.smithi125.stdout:671: finishing write tid 1 to smithi12556470-24 2021-03-24T21:56:42.193 INFO:tasks.rados.rados.0.smithi125.stdout:673: seq_num 264 ranges {437048=621738,1832305=404425} 2021-03-24T21:56:42.196 INFO:teuthology.orchestra.run.smithi125.stderr:nodeep-scrub is unset 2021-03-24T21:56:42.205 INFO:tasks.rados.rados.0.smithi125.stdout:673: writing smithi12556470-29 from 437048 to 1058786 tid 1 2021-03-24T21:56:42.212 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:42 smithi125 conmon[19936]: cluster 2021-03-24T21:56:42.212 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:42 smithi125 conmon[19936]: 2021-03-24T21:56:40 2021-03-24T21:56:42.212 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:42 smithi125 conmon[19936]: .458005+0000 osd.3 (osd.3) 63 : cluster [DBG] 6.13 scrub ok 2021-03-24T21:56:42.213 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:42 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:56:42.213 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:42 smithi125 conmon[19936]: -03-24T21:56:40.754354 2021-03-24T21:56:42.213 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:42 smithi125 conmon[19936]: +0000 mgr.y (mgr.14144) 379 2021-03-24T21:56:42.213 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:42 smithi125 conmon[19936]: : cluster [DBG] pgmap v509: 158 pgs: 1 peering, 157 active+clean; 121 MiB data, 437 MiB used, 536 GiB / 536 GiB avail; 11 MiB/s rd, 1.6 MiB/s wr, 15 op/s; 1.6 MiB/s, 4 objects/s recovering 2021-03-24T21:56:42.214 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:42 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:56:42.214 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:42 smithi125 conmon[19936]: -24T21:56:41.181234+0000 mon.a ( 2021-03-24T21:56:42.214 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:42 smithi125 conmon[19936]: mon.0) 846 : cluster [DBG] osdmap e232: 8 total, 8 up, 6 in 2021-03-24T21:56:42.214 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:42 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T21:56:42.215 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:42 smithi125 conmon[19936]: T21:56:41.235135+0000 osd.5 (osd 2021-03-24T21:56:42.215 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:42 smithi125 conmon[19936]: .5) 61 : cluster [DBG] 4.2 scrub ok 2021-03-24T21:56:42.215 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:42 smithi125 conmon[19936]: audit 2021-03 2021-03-24T21:56:42.215 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:42 smithi125 conmon[19936]: -24T21:56:41.559977+0000 mon.c (mon 2021-03-24T21:56:42.215 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:42 smithi125 conmon[19936]: .2) 37 : audit [INF] from='client.? v1:172.21.15.125:0/2864782226' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:56:42.216 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:42 smithi125 conmon[19936]: audit 2021-03-24T 2021-03-24T21:56:42.216 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:42 smithi125 conmon[19936]: 21:56:41.560692+0000 mon.a (mon.0) 847 : 2021-03-24T21:56:42.216 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:42 smithi125 conmon[19936]: audit [INF] from='client.34910 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:56:42.216 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:42 smithi125 conmon[19936]: audit 2021-03-24T21: 2021-03-24T21:56:42.216 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:42 smithi125 conmon[19936]: 56:41.634223+0000 mon.a (mon.0) 848 : audit [DBG] from='client.? v1:172.21.15.125:0/3865129970' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:56:42.217 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:42 smithi125 conmon[19936]: 2021-03-24T21:56:42.217 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:42 smithi125 conmon[24927]: cluster 2021-03-24T21:56:40.458005+0000 osd.3 2021-03-24T21:56:42.217 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:42 smithi125 conmon[24927]: (osd.3) 63 : cluster [DBG] 6.13 scrub ok 2021-03-24T21:56:42.218 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:42 smithi125 conmon[24927]: cluster 2021-03-24T21:56:40. 2021-03-24T21:56:42.218 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:42 smithi125 conmon[24927]: 754354+0000 mgr.y (mgr.14144) 379 : cluster [DBG] pgmap v509: 158 pgs: 1 peering, 157 active+clean; 121 MiB data, 437 MiB used, 536 GiB / 536 GiB avail; 11 MiB/s rd, 1.6 MiB/s wr, 15 op/s; 1.6 MiB/s, 4 objects/s recovering 2021-03-24T21:56:42.218 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:42 smithi125 conmon[24927]: cluster 2021-03-24T21:56: 2021-03-24T21:56:42.218 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:42 smithi125 conmon[24927]: 41.181234+0000 mon.a (mon.0) 846 : cluster 2021-03-24T21:56:42.219 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:42 smithi125 conmon[24927]: [DBG] osdmap e232: 8 total, 8 up, 6 in 2021-03-24T21:56:42.219 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:42 smithi125 conmon[24927]: cluster 2021-03-24T21:56:41. 2021-03-24T21:56:42.219 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:42 smithi125 conmon[24927]: 235135+0000 osd.5 (osd.5) 61 : cluster 2021-03-24T21:56:42.219 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:42 smithi125 conmon[24927]: [DBG] 4.2 scrub ok 2021-03-24T21:56:42.219 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:42 smithi125 conmon[24927]: audit 2021-03-24T21:56:41 2021-03-24T21:56:42.220 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:42 smithi125 conmon[24927]: .559977+0000 mon.c (mon.2) 37 : audit 2021-03-24T21:56:42.220 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:42 smithi125 conmon[24927]: [INF] from='client.? v1:172.21.15.125:0/2864782226' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:56:42.220 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:42 smithi125 conmon[24927]: audit 2021-03-24T21:56:41 2021-03-24T21:56:42.220 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:42 smithi125 conmon[24927]: .560692+0000 mon.a (mon.0) 847 : audit 2021-03-24T21:56:42.220 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:42 smithi125 conmon[24927]: [INF] from='client.34910 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:56:42.221 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:42 smithi125 conmon[24927]: audit 2021-03-24T21:56:41. 2021-03-24T21:56:42.221 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:42 smithi125 conmon[24927]: 634223+0000 mon.a (mon.0) 848 : audit [DBG] 2021-03-24T21:56:42.221 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:42 smithi125 conmon[24927]: from='client.? v1:172.21.15.125:0/3865129970' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:56:42.222 INFO:tasks.rados.rados.0.smithi125.stdout:673: writing smithi12556470-29 from 1832305 to 2236730 tid 2 2021-03-24T21:56:42.223 INFO:tasks.rados.rados.0.smithi125.stdout:674: delete oid 16 current snap is 64 2021-03-24T21:56:42.223 INFO:tasks.rados.rados.0.smithi125.stdout:671: finishing write tid 2 to smithi12556470-24 2021-03-24T21:56:42.223 INFO:tasks.rados.rados.0.smithi125.stdout:671: finishing write tid 3 to smithi12556470-24 2021-03-24T21:56:42.223 INFO:tasks.rados.rados.0.smithi125.stdout:671: finishing write tid 4 to smithi12556470-24 2021-03-24T21:56:42.223 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 24 v 220 (ObjNum 263 snap 64 seq_num 263) dirty exists 2021-03-24T21:56:42.224 INFO:tasks.rados.rados.0.smithi125.stdout:671: left oid 24 (ObjNum 263 snap 64 seq_num 263) 2021-03-24T21:56:42.224 INFO:tasks.rados.rados.0.smithi125.stdout:673: finishing write tid 1 to smithi12556470-29 2021-03-24T21:56:42.224 INFO:tasks.rados.rados.0.smithi125.stdout:671: done (2 left) 2021-03-24T21:56:42.224 INFO:tasks.rados.rados.0.smithi125.stdout:674: done (1 left) 2021-03-24T21:56:42.224 INFO:tasks.rados.rados.0.smithi125.stdout:675: rollback oid 32 current snap is 64 2021-03-24T21:56:42.225 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 32 to 59 2021-03-24T21:56:42.225 INFO:tasks.rados.rados.0.smithi125.stdout:676: write oid 21 current snap is 64 2021-03-24T21:56:42.225 INFO:tasks.rados.rados.0.smithi125.stdout:676: seq_num 266 ranges {470662=772236,1832916=408094} 2021-03-24T21:56:42.227 INFO:tasks.rados.rados.0.smithi125.stdout:676: writing smithi12556470-21 from 470662 to 1242898 tid 1 2021-03-24T21:56:42.233 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:42.234 INFO:tasks.rados.rados.0.smithi125.stdout:676: writing smithi12556470-21 from 1832916 to 2241010 tid 2 2021-03-24T21:56:42.234 INFO:tasks.rados.rados.0.smithi125.stdout:677: snap_create 2021-03-24T21:56:42.234 INFO:tasks.rados.rados.0.smithi125.stdout:675: finishing rollback tid 1 to smithi12556470-32 2021-03-24T21:56:42.235 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 32 v 0 (ObjNum 1953066355 snap 842099048 seq_num 875967797) dirty dne 2021-03-24T21:56:42.235 INFO:tasks.rados.rados.0.smithi125.stdout:673: finishing write tid 2 to smithi12556470-29 2021-03-24T21:56:42.235 INFO:tasks.rados.rados.0.smithi125.stdout:673: finishing write tid 3 to smithi12556470-29 2021-03-24T21:56:42.235 INFO:tasks.rados.rados.0.smithi125.stdout:673: finishing write tid 4 to smithi12556470-29 2021-03-24T21:56:42.236 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 29 v 119 (ObjNum 264 snap 64 seq_num 264) dirty exists 2021-03-24T21:56:42.236 INFO:tasks.rados.rados.0.smithi125.stdout:673: left oid 29 (ObjNum 264 snap 64 seq_num 264) 2021-03-24T21:56:42.236 INFO:tasks.rados.rados.0.smithi125.stdout:676: finishing write tid 1 to smithi12556470-21 2021-03-24T21:56:42.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:42.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:56:42.238 INFO:tasks.rados.rados.0.smithi125.stdout:676: finishing write tid 2 to smithi12556470-21 2021-03-24T21:56:42.238 INFO:tasks.rados.rados.0.smithi125.stdout:676: finishing write tid 3 to smithi12556470-21 2021-03-24T21:56:42.239 INFO:tasks.rados.rados.0.smithi125.stdout:676: finishing write tid 4 to smithi12556470-21 2021-03-24T21:56:42.239 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 21 v 169 (ObjNum 266 snap 64 seq_num 266) dirty exists 2021-03-24T21:56:42.239 INFO:tasks.rados.rados.0.smithi125.stdout:676: left oid 21 (ObjNum 266 snap 64 seq_num 266) 2021-03-24T21:56:42.339 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:42.389 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:42.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:42.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:56:42.495 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:42.558 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:42.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:42.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:56:42.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:42 smithi142 conmon[20125]: cluster 2021-03-24T21:56: 2021-03-24T21:56:42.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:42 smithi142 conmon[20125]: 40.458005+0000 osd.3 (osd.3) 63 : cluster [DBG] 6.13 scrub ok 2021-03-24T21:56:42.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:42 smithi142 conmon[20125]: cluster 2021-03-24T21:56:40. 2021-03-24T21:56:42.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:42 smithi142 conmon[20125]: 754354+0000 mgr.y (mgr.14144) 379 : cluster [DBG] pgmap v509: 158 pgs: 1 peering, 157 active+clean; 121 MiB data, 437 MiB used, 536 GiB / 536 GiB avail; 11 MiB/s rd, 1.6 MiB/s wr, 15 op/s; 1.6 MiB/s, 4 objects/s recovering 2021-03-24T21:56:42.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:42 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:56:42.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:42 smithi142 conmon[20125]: 21:56:41.181234+0000 mon.a (mon.0) 846 : 2021-03-24T21:56:42.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:42 smithi142 conmon[20125]: cluster [DBG] osdmap e232: 8 total, 8 up, 6 in 2021-03-24T21:56:42.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:42 smithi142 conmon[20125]: cluster 2021-03-24T21:56:41.235135+0000 2021-03-24T21:56:42.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:42 smithi142 conmon[20125]: osd.5 (osd.5) 61 : cluster [DBG] 4.2 scrub ok 2021-03-24T21:56:42.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:42 smithi142 conmon[20125]: audit 2021-03- 2021-03-24T21:56:42.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:42 smithi142 conmon[20125]: 24T21:56:41.559977+0000 mon.c (mon.2) 37 2021-03-24T21:56:42.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:42 smithi142 conmon[20125]: : audit [INF] from='client.? v1:172.21.15.125:0/2864782226' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:56:42.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:42 smithi142 conmon[20125]: audit 2021-03-24T21:56:41.560692+0000 2021-03-24T21:56:42.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:42 smithi142 conmon[20125]: mon.a (mon.0) 847 : audit [INF] from='client.34910 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:56:42.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:42 smithi142 conmon[20125]: audit 2021-03-24 2021-03-24T21:56:42.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:42 smithi142 conmon[20125]: T21:56:41.634223+0000 mon.a (mon.0) 848 2021-03-24T21:56:42.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:42 smithi142 conmon[20125]: : audit [DBG] from='client.? v1:172.21.15.125:0/3865129970' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:56:42.663 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:42.719 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:42.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:42.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:56:42.825 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:42.882 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:42.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:42.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:56:42.987 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:43.043 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:43.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:43.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:56:43.150 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:43.199 INFO:tasks.rados.rados.0.smithi125.stdout:673: done (3 left) 2021-03-24T21:56:43.199 INFO:tasks.rados.rados.0.smithi125.stdout:675: done (2 left) 2021-03-24T21:56:43.200 INFO:tasks.rados.rados.0.smithi125.stdout:676: done (1 left) 2021-03-24T21:56:43.200 INFO:tasks.rados.rados.0.smithi125.stdout:677: done (0 left) 2021-03-24T21:56:43.200 INFO:tasks.rados.rados.0.smithi125.stdout:678: read oid 1 snap -1 2021-03-24T21:56:43.200 INFO:tasks.rados.rados.0.smithi125.stdout:678: expect (ObjNum 150 snap 30 seq_num 150) 2021-03-24T21:56:43.201 INFO:tasks.rados.rados.0.smithi125.stdout:679: snap_create 2021-03-24T21:56:43.201 INFO:teuthology.orchestra.run.smithi125.stdout:moved 10 / 474 (2.1097%) 2021-03-24T21:56:43.201 INFO:teuthology.orchestra.run.smithi125.stdout:avg 79 2021-03-24T21:56:43.201 INFO:teuthology.orchestra.run.smithi125.stdout:stddev 6.1101 -> 3.65148 (expected baseline 8.11377) 2021-03-24T21:56:43.202 INFO:teuthology.orchestra.run.smithi125.stdout:min osd.5 with 72 -> 77 pgs (0.911392 -> 0.974684 * mean) 2021-03-24T21:56:43.202 INFO:teuthology.orchestra.run.smithi125.stdout:max osd.7 with 89 -> 83 pgs (1.12658 -> 1.05063 * mean) 2021-03-24T21:56:43.202 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-24T21:56:43.202 INFO:teuthology.orchestra.run.smithi125.stdout:oload 110 2021-03-24T21:56:43.203 INFO:teuthology.orchestra.run.smithi125.stdout:max_change 0.05 2021-03-24T21:56:43.203 INFO:teuthology.orchestra.run.smithi125.stdout:max_change_osds 4 2021-03-24T21:56:43.203 INFO:teuthology.orchestra.run.smithi125.stdout:average_utilization 0.0008 2021-03-24T21:56:43.203 INFO:teuthology.orchestra.run.smithi125.stdout:overload_utilization 0.0009 2021-03-24T21:56:43.203 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7 weight 0.9500 -> 0.9000 2021-03-24T21:56:43.204 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2 weight 0.9500 -> 0.9000 2021-03-24T21:56:43.204 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3 weight 0.9500 -> 0.9000 2021-03-24T21:56:43.212 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:43.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:43.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:56:43.217 INFO:tasks.rados.rados.0.smithi125.stdout:678: expect (ObjNum 150 snap 30 seq_num 150) 2021-03-24T21:56:43.318 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:43.332 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:43 smithi142 conmon[20125]: cluster 2021-03-24T21:56:42.039965+0000 osd.4 (osd.4) 65 2021-03-24T21:56:43.332 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:43 smithi142 conmon[20125]: : cluster [DBG] 5.1f scrub ok 2021-03-24T21:56:43.332 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:43 smithi142 conmon[20125]: cluster 2021-03-24T21:56:42.175454+0000 mon.a (mon.0 2021-03-24T21:56:43.332 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:43 smithi142 conmon[20125]: ) 849 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T21:56:43.333 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:43 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:56:43.333 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:43 smithi142 conmon[20125]: 24T21:56:42.175488 2021-03-24T21:56:43.333 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:43 smithi142 conmon[20125]: +0000 mon.a (mon.0) 850 : 2021-03-24T21:56:43.333 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:43 smithi142 conmon[20125]: cluster [INF] Cluster is now healthy 2021-03-24T21:56:43.333 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:43 smithi142 conmon[20125]: audit 2021 2021-03-24T21:56:43.334 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:43 smithi142 conmon[20125]: -03-24T21:56:42 2021-03-24T21:56:43.334 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:43 smithi142 conmon[20125]: .187614+0000 mon.a (mon.0 2021-03-24T21:56:43.334 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:43 smithi142 conmon[20125]: ) 851 : audit [INF] from='client.34910 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T21:56:43.334 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:43 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:56:43.335 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:43 smithi142 conmon[20125]: 03-24T21:56:42 2021-03-24T21:56:43.335 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:43 smithi142 conmon[20125]: .187706+0000 mon.a (mon.0) 852 : cluster 2021-03-24T21:56:43.335 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:43 smithi142 conmon[20125]: [DBG] osdmap e233: 8 total, 8 up, 6 in 2021-03-24T21:56:43.335 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:43 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:56:43.335 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:43 smithi142 conmon[20125]: 24T21:56:42.209489 2021-03-24T21:56:43.336 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:43 smithi142 conmon[20125]: +0000 osd.5 (osd.5) 62 : 2021-03-24T21:56:43.336 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:43 smithi142 conmon[20125]: cluster [DBG] 5.15 scrub ok 2021-03-24T21:56:43.336 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:43 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:56:43.336 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:43 smithi142 conmon[20125]: -24T21:56:42.254531 2021-03-24T21:56:43.336 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:43 smithi142 conmon[20125]: +0000 osd.0 (osd.0) 62 2021-03-24T21:56:43.337 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:43 smithi142 conmon[20125]: : cluster [DBG] 3.1b scrub ok 2021-03-24T21:56:43.337 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:43 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:56:43.337 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:43 smithi142 conmon[20125]: 03-24T21:56:42. 2021-03-24T21:56:43.337 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:43 smithi142 conmon[20125]: 363797+0000 mgr.y (mgr.14144) 2021-03-24T21:56:43.338 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:43 smithi142 conmon[20125]: 380 : audit [DBG] from='client.34922 v1:172.21.15.125:0/4286714564' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:56:43.338 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:43 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:56:43.338 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:43 smithi142 conmon[20125]: 03-24T21:56:42. 2021-03-24T21:56:43.338 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:43 smithi142 conmon[20125]: 365121+0000 mon.a (mon.0) 853 2021-03-24T21:56:43.338 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:43 smithi142 conmon[20125]: : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'2':'58984','3':'58984','7':'58984'}"}]: dispatch 2021-03-24T21:56:43.379 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:43.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:43.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:56:43.437 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:43 smithi125 conmon[19936]: cluster 2021-03-24T21:56:42.039965+0000 osd.4 (osd.4) 65 : cluster [DBG] 5.1f scrub ok 2021-03-24T21:56:43.438 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:43 smithi125 conmon[19936]: cluster 2021-03-24T21:56:42.175454+0000 mon.a (mon.0) 849 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T21:56:43.438 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:43 smithi125 conmon[19936]: cluster 2021-03-24T21:56:43.438 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:43 smithi125 conmon[19936]: 2021-03-24T21:56:42.175488+0000 mon.a (mon.0) 850 : cluster [INF] Cluster is now healthy 2021-03-24T21:56:43.438 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:43 smithi125 conmon[19936]: audit 2021-03-24T21:56:42.187614+0000 mon.a (mon.0) 851 : audit [INF] from='client.34910 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T21:56:43.439 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:43 smithi125 conmon[19936]: cluster 2021-03-24T21:56:42.187706+0000 mon.a (mon.0) 852 : cluster [DBG] osdmap e233: 8 total, 8 up, 6 in 2021-03-24T21:56:43.439 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:43 smithi125 conmon[19936]: cluster 2021-03-24T21:56:42.209489+0000 osd.5 (osd.5) 62 : cluster [DBG] 5.15 scrub ok 2021-03-24T21:56:43.439 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:43 smithi125 conmon[19936]: cluster 2021-03-24T21:56:42.254531+0000 osd.0 (osd.0) 62 : cluster [DBG] 3.1b scrub ok 2021-03-24T21:56:43.439 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:43 smithi125 conmon[19936]: audit 2021-03-24T21:56:42.363797+0000 mgr.y (mgr.14144) 380 : audit [DBG] from='client.34922 v1:172.21.15.125:0/4286714564' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:56:43.439 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:43 smithi125 conmon[19936]: audit 2021-03-24T21:56:42.365121+0000 mon.a (mon.0) 853 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'2':'58984','3':'58984','7':'58984'}"}]: dispatch 2021-03-24T21:56:43.440 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:43 smithi125 conmon[24927]: cluster 2021-03-24T21:56:42.039965+0000 osd.4 (osd.4) 65 : cluster [DBG] 5.1f scrub ok 2021-03-24T21:56:43.440 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:43 smithi125 conmon[24927]: cluster 2021-03-24T21:56:42.175454+0000 mon.a (mon.0) 849 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T21:56:43.440 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:43 smithi125 conmon[24927]: cluster 2021-03-24T21:56:42.175488+0000 mon.a (mon.0) 850 : cluster [INF] Cluster is now healthy 2021-03-24T21:56:43.440 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:43 smithi125 conmon[24927]: audit 2021-03-24T21:56:42.187614+0000 mon.a (mon.0) 851 : audit [INF] from='client.34910 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T21:56:43.441 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:43 smithi125 conmon[24927]: cluster 2021-03-24T21:56:42.187706+0000 mon.a (mon.0) 852 : cluster [DBG] osdmap e233: 8 total, 8 up, 6 in 2021-03-24T21:56:43.441 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:43 smithi125 conmon[24927]: cluster 2021-03-24T21:56:42.209489+0000 osd.5 (osd.5) 62 : cluster [DBG] 5.15 scrub ok 2021-03-24T21:56:43.441 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:43 smithi125 conmon[24927]: cluster 2021-03-24T21:56:42.254531+0000 osd.0 (osd.0) 62 : cluster [DBG] 3.1b scrub ok 2021-03-24T21:56:43.441 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:43 smithi125 conmon[24927]: audit 2021-03-24T21:56:42.363797+0000 mgr.y (mgr.14144) 380 : audit [DBG] from='client.34922 v1:172.21.15.125:0/4286714564' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:56:43.442 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:43 smithi125 conmon[24927]: audit 2021-03-24T21:56:42.365121+0000 mon.a (mon.0) 853 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'2':'58984','3':'58984','7':'58984'}"}]: dispatch 2021-03-24T21:56:43.485 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:43.538 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:43.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:43.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:56:43.644 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:43.700 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:43.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:43.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:56:43.806 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:43.865 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:43.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:43.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:56:43.971 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:44.026 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:44.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:44.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:56:44.132 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:44.192 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:44.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:44.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:56:44.215 INFO:tasks.rados.rados.0.smithi125.stdout:678: done (1 left) 2021-03-24T21:56:44.215 INFO:tasks.rados.rados.0.smithi125.stdout:679: done (0 left) 2021-03-24T21:56:44.216 INFO:tasks.rados.rados.0.smithi125.stdout:680: snap_create 2021-03-24T21:56:44.277 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:44 smithi125 conmon[19936]: cluster 2021-03-24T21:56:42.412841 2021-03-24T21:56:44.298 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:44.351 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:44.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:44.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:56:44.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:44 smithi125 conmon[24927]: cluster 2021-03-24T21:56:42 2021-03-24T21:56:44.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:44 smithi125 conmon[24927]: .412841+0000 osd.3 (osd.3) 64 : cluster [DBG] 3.b scrub ok 2021-03-24T21:56:44.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:44 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:56:44.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:44 smithi125 conmon[24927]: -24T21:56:42.755033+0000 mgr.y (mgr.14144) 381 2021-03-24T21:56:44.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:44 smithi125 conmon[24927]: : cluster [DBG] pgmap v512: 158 pgs: 1 peering, 157 active+clean; 122 MiB data, 442 MiB used, 536 GiB / 536 GiB avail; 7.7 MiB/s rd, 6 op/s; 234 B/s, 0 objects/s recovering 2021-03-24T21:56:44.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:44 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:56:44.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:44 smithi125 conmon[24927]: 21:56:43.051876+0000 osd.2 2021-03-24T21:56:44.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:44 smithi125 conmon[24927]: (osd.2) 45 : cluster [DBG] 5.a scrub ok 2021-03-24T21:56:44.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:44 smithi125 conmon[24927]: 2021-03-24T21:56:44.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:44 smithi125 conmon[24927]: cluster 2021-03-24T21:56:44.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:44 smithi125 conmon[24927]: 2021-03-24T21:56:43.194142 2021-03-24T21:56:44.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:44 smithi125 conmon[24927]: +0000 osd.5 (osd.5) 63 : cluster [DBG] 2021-03-24T21:56:44.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:44 smithi125 conmon[24927]: 5.7 scrub ok 2021-03-24T21:56:44.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:44 smithi125 conmon[24927]: audit 2021-03-24T21:56:44.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:44 smithi125 conmon[24927]: 2021-03-24T21:56:43. 2021-03-24T21:56:44.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:44 smithi125 conmon[24927]: 198717+0000 mon.a (mon.0) 854 : audit 2021-03-24T21:56:44.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:44 smithi125 conmon[24927]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'2':'58984','3':'58984','7':'58984'}"}]': finished 2021-03-24T21:56:44.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:44 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:56:44.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:44 smithi125 conmon[24927]: :56:43.198794+0000 mon.a (mon.0 2021-03-24T21:56:44.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:44 smithi125 conmon[24927]: ) 855 : cluster [DBG] osdmap e234: 8 total, 8 up, 6 in 2021-03-24T21:56:44.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:44 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:56:44.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:44 smithi125 conmon[24927]: 03-24T21:56:44.018036+0000 osd.2 (osd 2021-03-24T21:56:44.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:44 smithi125 conmon[24927]: .2) 46 : cluster [DBG] 5.b scrub ok 2021-03-24T21:56:44.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:44 smithi125 conmon[19936]: +0000 osd.3 (osd.3) 64 : cluster [DBG] 3.b scrub ok 2021-03-24T21:56:44.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:44 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:56:44.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:44 smithi125 conmon[19936]: 56:42.755033+0000 mgr.y (mgr.14144) 381 : 2021-03-24T21:56:44.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:44 smithi125 conmon[19936]: cluster [DBG] pgmap v512: 158 pgs: 1 peering, 157 active+clean; 122 MiB data, 442 MiB used, 536 GiB / 536 GiB avail; 7.7 MiB/s rd, 6 op/s; 234 B/s, 0 objects/s recovering 2021-03-24T21:56:44.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:44 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:56:44.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:44 smithi125 conmon[19936]: -03-24T21:56:43.051876+0000 osd.2 ( 2021-03-24T21:56:44.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:44 smithi125 conmon[19936]: osd.2) 45 : cluster [DBG] 5.a scrub ok 2021-03-24T21:56:44.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:44 smithi125 conmon[19936]: 2021-03-24T21:56:44.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:44 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T21:56:44.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:44 smithi125 conmon[19936]: T21:56:43.194142+0000 osd.5 (osd. 2021-03-24T21:56:44.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:44 smithi125 conmon[19936]: 5) 63 : cluster [DBG] 5.7 scrub ok 2021-03-24T21:56:44.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:44 smithi125 conmon[19936]: 2021-03-24T21:56:44.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:44 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:56:44.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:44 smithi125 conmon[19936]: 03-24T21:56:43.198717+0000 2021-03-24T21:56:44.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:44 smithi125 conmon[19936]: mon.a (mon.0) 854 : audit 2021-03-24T21:56:44.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:44 smithi125 conmon[19936]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'2':'58984','3':'58984','7':'58984'}"}]': finished 2021-03-24T21:56:44.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:44 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:56:44.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:44 smithi125 conmon[19936]: 03-24T21:56:43.198794+0000 2021-03-24T21:56:44.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:44 smithi125 conmon[19936]: mon.a (mon.0) 855 : cluster 2021-03-24T21:56:44.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:44 smithi125 conmon[19936]: [DBG] osdmap e234: 8 total, 8 up, 6 in 2021-03-24T21:56:44.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:44 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:56:44.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:44 smithi125 conmon[19936]: 03-24T21:56:44.018036+0000 2021-03-24T21:56:44.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:44 smithi125 conmon[19936]: osd.2 (osd.2) 46 : cluster 2021-03-24T21:56:44.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:44 smithi125 conmon[19936]: [DBG] 5.b scrub ok 2021-03-24T21:56:44.456 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:44.470 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:44 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:56:44.470 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:44 smithi142 conmon[20125]: T21:56:42.412841+0000 osd.3 (osd.3) 64 : cluster [DBG] 3.b scrub ok 2021-03-24T21:56:44.471 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:44 smithi142 conmon[20125]: 2021-03-24T21:56:44.471 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:44 smithi142 conmon[20125]: cluster 2021-03-24T21:56:44.471 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:44 smithi142 conmon[20125]: 2021-03-24T21:56:42 2021-03-24T21:56:44.471 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:44 smithi142 conmon[20125]: .755033+0000 mgr.y 2021-03-24T21:56:44.472 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:44 smithi142 conmon[20125]: (mgr.14144) 2021-03-24T21:56:44.472 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:44 smithi142 conmon[20125]: 381 : cluster [DBG] 2021-03-24T21:56:44.472 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:44 smithi142 conmon[20125]: pgmap v512: 158 pgs: 1 peering, 157 active+clean; 122 MiB data, 442 MiB used, 536 GiB / 536 GiB avail; 7.7 MiB/s rd, 6 op/s; 234 B/s, 0 objects/s recovering 2021-03-24T21:56:44.472 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:44 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:56:44.472 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:44 smithi142 conmon[20125]: -03-24T21:56: 2021-03-24T21:56:44.473 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:44 smithi142 conmon[20125]: 43.051876+0000 2021-03-24T21:56:44.473 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:44 smithi142 conmon[20125]: osd.2 (osd.2) 2021-03-24T21:56:44.473 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:44 smithi142 conmon[20125]: 45 : cluster [DBG] 2021-03-24T21:56:44.473 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:44 smithi142 conmon[20125]: 5.a scrub ok 2021-03-24T21:56:44.473 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:44 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:56:44.474 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:44 smithi142 conmon[20125]: -03-24T21: 2021-03-24T21:56:44.474 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:44 smithi142 conmon[20125]: 56:43.194142+0000 2021-03-24T21:56:44.474 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:44 smithi142 conmon[20125]: osd.5 (osd. 2021-03-24T21:56:44.474 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:44 smithi142 conmon[20125]: 5) 63 : cluster 2021-03-24T21:56:44.474 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:44 smithi142 conmon[20125]: [DBG] 5.7 scrub ok 2021-03-24T21:56:44.475 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:44 smithi142 conmon[20125]: audit 2021 2021-03-24T21:56:44.475 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:44 smithi142 conmon[20125]: -03-24T21:56 2021-03-24T21:56:44.475 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:44 smithi142 conmon[20125]: :43.198717+0000 2021-03-24T21:56:44.475 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:44 smithi142 conmon[20125]: mon.a (mon.0 2021-03-24T21:56:44.476 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:44 smithi142 conmon[20125]: ) 854 : audit 2021-03-24T21:56:44.476 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:44 smithi142 conmon[20125]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'2':'58984','3':'58984','7':'58984'}"}]': finished 2021-03-24T21:56:44.476 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:44 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:56:44.476 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:44 smithi142 conmon[20125]: -03-24T21: 2021-03-24T21:56:44.476 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:44 smithi142 conmon[20125]: 56:43.198794+0000 2021-03-24T21:56:44.477 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:44 smithi142 conmon[20125]: mon.a (mon. 2021-03-24T21:56:44.477 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:44 smithi142 conmon[20125]: 0) 855 : cluster 2021-03-24T21:56:44.477 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:44 smithi142 conmon[20125]: [DBG] osdmap e234: 8 total, 8 up, 6 in 2021-03-24T21:56:44.477 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:44 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:56:44.477 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:44 smithi142 conmon[20125]: -03-24T21:56 2021-03-24T21:56:44.478 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:44 smithi142 conmon[20125]: :44.018036+0000 2021-03-24T21:56:44.478 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:44 smithi142 conmon[20125]: osd.2 (osd.2 2021-03-24T21:56:44.478 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:44 smithi142 conmon[20125]: ) 46 : cluster 2021-03-24T21:56:44.478 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:44 smithi142 conmon[20125]: [DBG] 5.b scrub ok 2021-03-24T21:56:44.510 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:44.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:44.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:56:44.619 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:44.674 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:44.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:44.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:56:44.781 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:44.838 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:44.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:44.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:56:44.944 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:44.998 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:45.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:45.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:56:45.105 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:45.133 INFO:tasks.rados.rados.0.smithi125.stdout:680: done (0 left) 2021-03-24T21:56:45.134 INFO:tasks.rados.rados.0.smithi125.stdout:681: snap_create 2021-03-24T21:56:45.160 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:45.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:45.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:56:45.264 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:45.309 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:45.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:45.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:56:45.392 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:45 smithi125 conmon[24927]: cluster 2021-03-24T21:56:43. 2021-03-24T21:56:45.392 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:45 smithi125 conmon[24927]: 970870+0000 osd.4 (osd.4) 66 : cluster [DBG] 5.f scrub ok 2021-03-24T21:56:45.392 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:45 smithi125 conmon[24927]: cluster 2021-03-24T21:56:45.392 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:45 smithi125 conmon[24927]: 2021-03-24T21:56:44.212638+0000 mon.a (mon.0) 856 : cluster 2021-03-24T21:56:45.393 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:45 smithi125 conmon[24927]: [DBG] osdmap e235: 8 total, 8 up, 6 in 2021-03-24T21:56:45.393 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:45 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:56:45.393 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:45 smithi125 conmon[24927]: -03-24T21:56:45.128606+0000 mon.a ( 2021-03-24T21:56:45.393 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:45 smithi125 conmon[24927]: mon.0) 857 : cluster [DBG] osdmap e236: 8 total, 8 up, 6 in 2021-03-24T21:56:45.394 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:45 smithi125 conmon[19936]: cluster 2021-03-24T21:56:43.970870+0000 osd.4 (osd.4) 66 2021-03-24T21:56:45.394 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:45 smithi125 conmon[19936]: : cluster [DBG] 5.f scrub ok 2021-03-24T21:56:45.394 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:45 smithi125 conmon[19936]: cluster 2021-03-24T21:56:44.212638+0000 mon.a (mon.0) 856 : cluster [DBG] osdmap e235: 8 total, 8 up, 6 in 2021-03-24T21:56:45.394 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:45 smithi125 conmon[19936]: cluster 2021-03-24T21:56:45.128606+0000 mon.a (mon.0) 857 : 2021-03-24T21:56:45.395 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:45 smithi125 conmon[19936]: cluster [DBG] osdmap e236: 8 total, 8 up, 6 in 2021-03-24T21:56:45.415 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:45.472 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:45.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:45.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:56:45.507 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2021-03-24T21:56:45.507 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd test-reweight-by-utilization 110 0.05 2021-03-24T21:56:45.576 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:45.589 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:45 smithi142 conmon[20125]: cluster 2021-03-24T21:56 2021-03-24T21:56:45.590 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:45 smithi142 conmon[20125]: :43.970870+0000 osd.4 (osd.4) 66 : cluster [DBG] 5.f scrub ok 2021-03-24T21:56:45.590 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:45 smithi142 conmon[20125]: cluster 2021-03-24T21:56:44 2021-03-24T21:56:45.590 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:45 smithi142 conmon[20125]: .212638+0000 mon.a (mon.0) 856 : cluster [DBG] osdmap e235: 8 total, 8 up, 6 in 2021-03-24T21:56:45.590 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:45 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:56:45.591 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:45 smithi142 conmon[20125]: T21:56:45.128606+0000 mon.a (mon. 2021-03-24T21:56:45.591 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:45 smithi142 conmon[20125]: 0) 857 : cluster [DBG] osdmap e236: 8 total, 8 up, 6 in 2021-03-24T21:56:45.639 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:45.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:45.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:56:45.746 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:45.803 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:45.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:45.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:56:45.861 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:56:45.909 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:45.943 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-24T21:56:45.944 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T21:56:45.960 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:45.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:45.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:56:46.066 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:46.127 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:46.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:46.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:56:46.134 INFO:tasks.rados.rados.0.smithi125.stdout:681: done (0 left) 2021-03-24T21:56:46.134 INFO:tasks.rados.rados.0.smithi125.stdout:682: snap_remove snap 65 2021-03-24T21:56:46.234 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:46.296 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:46.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:46.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:56:46.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:46 smithi142 conmon[20125]: cluster 2021-03-24T21:56: 2021-03-24T21:56:46.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:46 smithi142 conmon[20125]: 44.755879+0000 mgr.y (mgr.14144) 382 : cluster [DBG] pgmap v515: 158 pgs: 7 peering, 151 active+clean; 131 MiB data, 460 MiB used, 536 GiB / 536 GiB avail; 3.3 MiB/s rd, 3 op/s 2021-03-24T21:56:46.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:46 smithi142 conmon[20125]: cluster 2021-03-24T21:56: 2021-03-24T21:56:46.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:46 smithi142 conmon[20125]: 46.133096+0000 mon.a (mon.0) 858 : cluster [DBG] osdmap e237: 8 total, 8 up, 6 in 2021-03-24T21:56:46.402 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:46.463 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:46.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:46.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:56:46.569 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:46.583 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:46 smithi125 conmon[19936]: cluster 2021-03-24T21:56:44.755879+0000 mgr.y 2021-03-24T21:56:46.583 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:46 smithi125 conmon[19936]: (mgr.14144) 382 : cluster [DBG] pgmap v515: 158 pgs: 7 peering, 151 active+clean; 131 MiB data, 460 MiB used, 536 GiB / 536 GiB avail; 3.3 MiB/s rd, 3 op/s 2021-03-24T21:56:46.583 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:46 smithi125 conmon[19936]: cluster 2021-03-24T21:56:46.133096 2021-03-24T21:56:46.584 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:46 smithi125 conmon[19936]: +0000 mon.a (mon.0) 858 : cluster [DBG] osdmap e237: 8 total, 8 up, 6 in 2021-03-24T21:56:46.584 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:46 smithi125 conmon[24927]: cluster 2021-03-24T21:56:44.755879+0000 2021-03-24T21:56:46.584 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:46 smithi125 conmon[24927]: mgr.y (mgr.14144) 382 : cluster [DBG] pgmap v515: 158 pgs: 7 peering, 151 active+clean; 131 MiB data, 460 MiB used, 536 GiB / 536 GiB avail; 3.3 MiB/s rd, 3 op/s 2021-03-24T21:56:46.585 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:46 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:56:46.585 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:46 smithi125 conmon[24927]: 03-24T21:56:46.133096+0000 mon.a (mon.0) 858 2021-03-24T21:56:46.585 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:46 smithi125 conmon[24927]: : cluster [DBG] osdmap e237: 8 total, 8 up, 6 in 2021-03-24T21:56:46.632 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:46.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:46.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:56:46.736 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:46.795 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:46.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:46.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:56:46.901 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:46.961 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:46.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:46.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:56:47.066 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:47.095 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.1 dump_blocked_ops 2021-03-24T21:56:47.123 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:47.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:47.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:56:47.142 INFO:tasks.rados.rados.0.smithi125.stdout:682: done (0 left) 2021-03-24T21:56:47.142 INFO:tasks.rados.rados.0.smithi125.stdout:683: rollback oid 33 current snap is 68 2021-03-24T21:56:47.142 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 33 to 67 2021-03-24T21:56:47.142 INFO:tasks.rados.rados.0.smithi125.stdout:684: copy_from oid 7 from oid 16 current snap is 68 2021-03-24T21:56:47.143 INFO:tasks.rados.rados.0.smithi125.stdout:685: write oid 38 current snap is 68 2021-03-24T21:56:47.143 INFO:tasks.rados.rados.0.smithi125.stdout:685: seq_num 267 ranges {487469=563838,1607751=551752} 2021-03-24T21:56:47.150 INFO:tasks.rados.rados.0.smithi125.stdout:685: writing smithi12556470-38 from 487469 to 1051307 tid 1 2021-03-24T21:56:47.158 INFO:tasks.rados.rados.0.smithi125.stdout:685: writing smithi12556470-38 from 1607751 to 2159503 tid 2 2021-03-24T21:56:47.158 INFO:tasks.rados.rados.0.smithi125.stdout:686: copy_from oid 41 from oid 15 current snap is 68 2021-03-24T21:56:47.158 INFO:tasks.rados.rados.0.smithi125.stdout:684: finishing copy_from to smithi12556470-7 2021-03-24T21:56:47.159 INFO:tasks.rados.rados.0.smithi125.stdout:684: got expected ENOENT (src dne) 2021-03-24T21:56:47.159 INFO:tasks.rados.rados.0.smithi125.stdout:684: finishing copy_from racing read to smithi12556470-7 2021-03-24T21:56:47.160 INFO:tasks.rados.rados.0.smithi125.stdout:684: done (3 left) 2021-03-24T21:56:47.160 INFO:tasks.rados.rados.0.smithi125.stdout:687: write (excl) oid 32 current snap is 68 2021-03-24T21:56:47.161 INFO:tasks.rados.rados.0.smithi125.stdout:687: seq_num 268 ranges {504276=639087,1666233=695410} 2021-03-24T21:56:47.165 INFO:tasks.rados.rados.0.smithi125.stdout:687: writing smithi12556470-32 from 504276 to 1143363 tid 1 2021-03-24T21:56:47.175 INFO:tasks.rados.rados.0.smithi125.stdout:687: writing smithi12556470-32 from 1666233 to 2361643 tid 2 2021-03-24T21:56:47.176 INFO:tasks.rados.rados.0.smithi125.stdout:688: read oid 36 snap 67 2021-03-24T21:56:47.176 INFO:tasks.rados.rados.0.smithi125.stdout:688: expect (ObjNum 225 snap 50 seq_num 225) 2021-03-24T21:56:47.176 INFO:tasks.rados.rados.0.smithi125.stdout:689: rollback oid 29 current snap is 68 2021-03-24T21:56:47.177 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 29 to 64 2021-03-24T21:56:47.177 INFO:tasks.rados.rados.0.smithi125.stdout:685: finishing write tid 1 to smithi12556470-38 2021-03-24T21:56:47.177 INFO:tasks.rados.rados.0.smithi125.stdout:685: finishing write tid 2 to smithi12556470-38 2021-03-24T21:56:47.178 INFO:tasks.rados.rados.0.smithi125.stdout:685: finishing write tid 3 to smithi12556470-38 2021-03-24T21:56:47.178 INFO:tasks.rados.rados.0.smithi125.stdout:685: finishing write tid 4 to smithi12556470-38 2021-03-24T21:56:47.178 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 38 v 34 (ObjNum 267 snap 68 seq_num 267) dirty exists 2021-03-24T21:56:47.179 INFO:tasks.rados.rados.0.smithi125.stdout:685: left oid 38 (ObjNum 267 snap 68 seq_num 267) 2021-03-24T21:56:47.179 INFO:tasks.rados.rados.0.smithi125.stdout:683: finishing rollback tid 0 to smithi12556470-33 2021-03-24T21:56:47.179 INFO:tasks.rados.rados.0.smithi125.stdout:683: finishing rollback tid 1 to smithi12556470-33 2021-03-24T21:56:47.180 INFO:tasks.rados.rados.0.smithi125.stdout:683: finishing rollback tid 2 to smithi12556470-33 2021-03-24T21:56:47.180 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 33 v 228 (ObjNum 228 snap 50 seq_num 228) dirty exists 2021-03-24T21:56:47.180 INFO:tasks.rados.rados.0.smithi125.stdout:683: done (5 left) 2021-03-24T21:56:47.181 INFO:tasks.rados.rados.0.smithi125.stdout:685: done (4 left) 2021-03-24T21:56:47.181 INFO:tasks.rados.rados.0.smithi125.stdout:690: delete oid 2 current snap is 68 2021-03-24T21:56:47.181 INFO:tasks.rados.rados.0.smithi125.stdout:687: finishing write tid 1 to smithi12556470-32 2021-03-24T21:56:47.183 INFO:tasks.rados.rados.0.smithi125.stdout:689: finishing rollback tid 0 to smithi12556470-29 2021-03-24T21:56:47.183 INFO:tasks.rados.rados.0.smithi125.stdout:689: finishing rollback tid 1 to smithi12556470-29 2021-03-24T21:56:47.183 INFO:tasks.rados.rados.0.smithi125.stdout:689: finishing rollback tid 2 to smithi12556470-29 2021-03-24T21:56:47.183 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 29 v 123 (ObjNum 264 snap 64 seq_num 264) dirty exists 2021-03-24T21:56:47.183 INFO:tasks.rados.rados.0.smithi125.stdout:687: finishing write tid 2 to smithi12556470-32 2021-03-24T21:56:47.184 INFO:tasks.rados.rados.0.smithi125.stdout:687: finishing write tid 3 to smithi12556470-32 2021-03-24T21:56:47.184 INFO:tasks.rados.rados.0.smithi125.stdout:686: finishing copy_from to smithi12556470-41 2021-03-24T21:56:47.184 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 41 v 49 (ObjNum 83 snap 10 seq_num 83) dirty exists 2021-03-24T21:56:47.184 INFO:tasks.rados.rados.0.smithi125.stdout:687: finishing write tid 4 to smithi12556470-32 2021-03-24T21:56:47.185 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 32 v 199 (ObjNum 268 snap 68 seq_num 268) dirty exists 2021-03-24T21:56:47.185 INFO:tasks.rados.rados.0.smithi125.stdout:687: left oid 32 (ObjNum 268 snap 68 seq_num 268) 2021-03-24T21:56:47.185 INFO:tasks.rados.rados.0.smithi125.stdout:686: finishing copy_from racing read to smithi12556470-41 2021-03-24T21:56:47.185 INFO:tasks.rados.rados.0.smithi125.stdout:688: expect (ObjNum 225 snap 50 seq_num 225) 2021-03-24T21:56:47.230 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:47.289 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:47.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:47.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:56:47.390 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-24T21:56:47.396 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:47.436 INFO:tasks.rados.rados.0.smithi125.stdout:686: done (4 left) 2021-03-24T21:56:47.436 INFO:tasks.rados.rados.0.smithi125.stdout:687: done (3 left) 2021-03-24T21:56:47.436 INFO:tasks.rados.rados.0.smithi125.stdout:688: done (2 left) 2021-03-24T21:56:47.437 INFO:tasks.rados.rados.0.smithi125.stdout:689: done (1 left) 2021-03-24T21:56:47.437 INFO:tasks.rados.rados.0.smithi125.stdout:690: done (0 left) 2021-03-24T21:56:47.437 INFO:tasks.rados.rados.0.smithi125.stdout:691: read oid 11 snap -1 2021-03-24T21:56:47.438 INFO:tasks.rados.rados.0.smithi125.stdout:691: expect (ObjNum 244 snap 57 seq_num 244) 2021-03-24T21:56:47.438 INFO:tasks.rados.rados.0.smithi125.stdout:692: write oid 38 current snap is 68 2021-03-24T21:56:47.438 INFO:tasks.rados.rados.0.smithi125.stdout:692: seq_num 270 ranges {537890=789585,1783197=699079} 2021-03-24T21:56:47.452 INFO:tasks.rados.rados.0.smithi125.stdout:692: writing smithi12556470-38 from 537890 to 1327475 tid 1 2021-03-24T21:56:47.455 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:47.459 INFO:tasks.rados.rados.0.smithi125.stdout:692: writing smithi12556470-38 from 1783197 to 2482276 tid 2 2021-03-24T21:56:47.459 INFO:tasks.rados.rados.0.smithi125.stdout:693: read oid 27 snap -1 2021-03-24T21:56:47.459 INFO:tasks.rados.rados.0.smithi125.stdout:693: expect (ObjNum 193 snap 42 seq_num 193) 2021-03-24T21:56:47.459 INFO:tasks.rados.rados.0.smithi125.stdout:694: read oid 46 snap -1 2021-03-24T21:56:47.460 INFO:tasks.rados.rados.0.smithi125.stdout:694: expect deleted 2021-03-24T21:56:47.460 INFO:tasks.rados.rados.0.smithi125.stdout:695: delete oid 50 current snap is 68 2021-03-24T21:56:47.460 INFO:tasks.rados.rados.0.smithi125.stdout:691: expect (ObjNum 244 snap 57 seq_num 244) 2021-03-24T21:56:47.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:47.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:56:47.472 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd set noscrub 2021-03-24T21:56:47.549 INFO:tasks.rados.rados.0.smithi125.stdout:692: finishing write tid 1 to smithi12556470-38 2021-03-24T21:56:47.549 INFO:tasks.rados.rados.0.smithi125.stdout:691: done (4 left) 2021-03-24T21:56:47.550 INFO:tasks.rados.rados.0.smithi125.stdout:695: done (3 left) 2021-03-24T21:56:47.550 INFO:tasks.rados.rados.0.smithi125.stdout:696: snap_remove snap 67 2021-03-24T21:56:47.550 INFO:tasks.rados.rados.0.smithi125.stdout:692: finishing write tid 2 to smithi12556470-38 2021-03-24T21:56:47.550 INFO:tasks.rados.rados.0.smithi125.stdout:692: finishing write tid 3 to smithi12556470-38 2021-03-24T21:56:47.551 INFO:tasks.rados.rados.0.smithi125.stdout:692: finishing write tid 4 to smithi12556470-38 2021-03-24T21:56:47.551 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 38 v 37 (ObjNum 270 snap 68 seq_num 270) dirty exists 2021-03-24T21:56:47.551 INFO:tasks.rados.rados.0.smithi125.stdout:692: left oid 38 (ObjNum 270 snap 68 seq_num 270) 2021-03-24T21:56:47.551 INFO:tasks.rados.rados.0.smithi125.stdout:693: expect (ObjNum 193 snap 42 seq_num 193) 2021-03-24T21:56:47.562 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:47.622 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:47.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:47.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:56:47.727 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:47.791 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:47.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:47.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:56:47.897 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:47.958 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:47.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:47.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:56:48.012 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:56:48.065 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:48.066 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:56:48.121 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:56:48.122 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:48.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:48.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:56:48.229 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:48.235 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:48 smithi125 conmon[19936]: cluster 2021-03-24T21:56:46.756745+0000 mgr.y (mgr.14144) 383 : cluster [DBG] pgmap v518: 158 pgs: 7 peering, 151 active+clean; 131 MiB data, 461 MiB used, 536 GiB / 536 GiB avail; 6.4 MiB/s rd, 2.5 MiB/s wr, 11 op/s 2021-03-24T21:56:48.236 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:48 smithi125 conmon[19936]: cluster 2021-03-24T21:56:48.236 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:48 smithi125 conmon[19936]: 2021-03-24T21:56:47.137021+0000 mon.a (mon.0) 859 : cluster [DBG] osdmap e238: 8 total, 8 up, 6 in 2021-03-24T21:56:48.236 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:48 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:56:48.236 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:48 smithi125 conmon[19936]: 21:56:47.168552+0000 osd.5 (osd.5) 64 : cluster [DBG] 4.15 scrub ok 2021-03-24T21:56:48.236 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:48 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:56:48.237 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:48 smithi125 conmon[19936]: -24T21:56:47.285638+0000 2021-03-24T21:56:48.237 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:48 smithi125 conmon[19936]: osd.0 (osd.0) 63 2021-03-24T21:56:48.237 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:48 smithi125 conmon[19936]: : cluster [DBG] 7.d scrub ok 2021-03-24T21:56:48.238 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:48 smithi125 conmon[24927]: cluster 2021-03-24T21:56:46.756745 2021-03-24T21:56:48.238 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:48 smithi125 conmon[24927]: +0000 mgr.y (mgr.14144) 383 : cluster [DBG] pgmap v518: 158 pgs: 7 peering, 151 active+clean; 131 MiB data, 461 MiB used, 536 GiB / 536 GiB avail; 6.4 MiB/s rd, 2.5 MiB/s wr, 11 op/s 2021-03-24T21:56:48.238 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:48 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:56:48.238 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:48 smithi125 conmon[24927]: 24T21:56:47.137021+0000 mon.a (mon.0) 859 : cluster [DBG] osdmap e238: 8 total, 8 up, 6 in 2021-03-24T21:56:48.238 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:48 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:56:48.239 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:48 smithi125 conmon[24927]: -24T21:56:47.168552+0000 osd.5 (osd. 2021-03-24T21:56:48.239 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:48 smithi125 conmon[24927]: 5) 64 : cluster [DBG] 4.15 scrub ok 2021-03-24T21:56:48.239 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:48 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:56:48.239 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:48 smithi125 conmon[24927]: :56:47.285638+0000 osd.0 (osd.0) 63 : cluster 2021-03-24T21:56:48.240 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:48 smithi125 conmon[24927]: [DBG] 7.d scrub ok 2021-03-24T21:56:48.240 INFO:tasks.rados.rados.0.smithi125.stdout:692: done (3 left) 2021-03-24T21:56:48.240 INFO:tasks.rados.rados.0.smithi125.stdout:693: done (2 left) 2021-03-24T21:56:48.240 INFO:tasks.rados.rados.0.smithi125.stdout:694: done (1 left) 2021-03-24T21:56:48.241 INFO:tasks.rados.rados.0.smithi125.stdout:696: done (0 left) 2021-03-24T21:56:48.241 INFO:tasks.rados.rados.0.smithi125.stdout:697: write (excl) oid 18 current snap is 68 2021-03-24T21:56:48.241 INFO:tasks.rados.rados.0.smithi125.stdout:697: seq_num 272 ranges {571504=540083,1783808=702748} 2021-03-24T21:56:48.243 INFO:tasks.rados.rados.0.smithi125.stdout:697: writing smithi12556470-18 from 571504 to 1111587 tid 1 2021-03-24T21:56:48.253 INFO:tasks.rados.rados.0.smithi125.stdout:697: writing smithi12556470-18 from 1783808 to 2486556 tid 2 2021-03-24T21:56:48.253 INFO:tasks.rados.rados.0.smithi125.stdout:698: delete oid 3 current snap is 68 2021-03-24T21:56:48.254 INFO:tasks.rados.rados.0.smithi125.stdout:697: finishing write tid 1 to smithi12556470-18 2021-03-24T21:56:48.255 INFO:tasks.rados.rados.0.smithi125.stdout:698: done (1 left) 2021-03-24T21:56:48.255 INFO:tasks.rados.rados.0.smithi125.stdout:699: read oid 1 snap -1 2021-03-24T21:56:48.256 INFO:tasks.rados.rados.0.smithi125.stdout:699: expect (ObjNum 150 snap 30 seq_num 150) 2021-03-24T21:56:48.257 INFO:tasks.rados.rados.0.smithi125.stdout:700: write (excl) oid 29 current snap is 68 2021-03-24T21:56:48.257 INFO:tasks.rados.rados.0.smithi125.stdout:700: seq_num 274 ranges {605118=406934,1617125=706417} 2021-03-24T21:56:48.263 INFO:tasks.rados.rados.0.smithi125.stdout:700: writing smithi12556470-29 from 605118 to 1012052 tid 1 2021-03-24T21:56:48.271 INFO:tasks.rados.rados.0.smithi125.stdout:700: writing smithi12556470-29 from 1617125 to 2323542 tid 2 2021-03-24T21:56:48.272 INFO:tasks.rados.rados.0.smithi125.stdout:701: snap_remove snap 64 2021-03-24T21:56:48.281 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:48.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:48.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:56:48.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:48 smithi142 conmon[20125]: cluster 2021-03-24T21:56:46.756745+0000 2021-03-24T21:56:48.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:48 smithi142 conmon[20125]: mgr.y (mgr.14144) 383 : cluster [DBG] pgmap v518: 158 pgs: 7 peering, 151 active+clean; 131 MiB data, 461 MiB used, 536 GiB / 536 GiB avail; 6.4 MiB/s rd, 2.5 MiB/s wr, 11 op/s 2021-03-24T21:56:48.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:48 smithi142 conmon[20125]: cluster 2021-03-24T21:56:47.137021+0000 mon.a (mon.0) 859 : cluster [DBG] osdmap e238: 8 total, 8 up, 6 in 2021-03-24T21:56:48.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:48 smithi142 conmon[20125]: cluster 2021-03-24T21:56:47.168552+0000 osd.5 (osd.5) 64 : cluster [DBG] 4.15 scrub ok 2021-03-24T21:56:48.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:48 smithi142 conmon[20125]: cluster 2021-03-24T21:56:47.285638+0000 osd.0 (osd.0 2021-03-24T21:56:48.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:48 smithi142 conmon[20125]: ) 63 : cluster [DBG] 7.d scrub ok 2021-03-24T21:56:48.386 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:48.437 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:48.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:48.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:56:48.543 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:48.606 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:48.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:48.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:56:48.683 INFO:teuthology.orchestra.run.smithi125.stdout:moved 8 / 474 (1.68776%) 2021-03-24T21:56:48.683 INFO:teuthology.orchestra.run.smithi125.stdout:avg 79 2021-03-24T21:56:48.683 INFO:teuthology.orchestra.run.smithi125.stdout:stddev 3.65148 -> 4.20317 (expected baseline 8.11377) 2021-03-24T21:56:48.684 INFO:teuthology.orchestra.run.smithi125.stdout:min osd.2 with 73 -> 71 pgs (0.924051 -> 0.898734 * mean) 2021-03-24T21:56:48.684 INFO:teuthology.orchestra.run.smithi125.stdout:max osd.0 with 83 -> 85 pgs (1.05063 -> 1.07595 * mean) 2021-03-24T21:56:48.684 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-24T21:56:48.684 INFO:teuthology.orchestra.run.smithi125.stdout:oload 110 2021-03-24T21:56:48.685 INFO:teuthology.orchestra.run.smithi125.stdout:max_change 0.05 2021-03-24T21:56:48.685 INFO:teuthology.orchestra.run.smithi125.stdout:max_change_osds 4 2021-03-24T21:56:48.685 INFO:teuthology.orchestra.run.smithi125.stdout:average_utilization 0.0008 2021-03-24T21:56:48.685 INFO:teuthology.orchestra.run.smithi125.stdout:overload_utilization 0.0009 2021-03-24T21:56:48.686 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7 weight 0.9000 -> 0.8500 2021-03-24T21:56:48.686 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2 weight 0.9000 -> 0.8500 2021-03-24T21:56:48.687 INFO:teuthology.orchestra.run.smithi125.stderr:no change 2021-03-24T21:56:48.711 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:48.763 INFO:journalctl@ceph.mgr.y.smithi125.stdout:Mar 24 21:56:48 smithi125 conmon[20260]: debug 2021-03-24T21:56:48.763 INFO:journalctl@ceph.mgr.y.smithi125.stdout:Mar 24 21:56:48 smithi125 conmon[20260]: 2021-03-24T21:56:48.677+0000 7faa8fb8c700 -1 mgr.server reply reply (2) No such file or directory no change 2021-03-24T21:56:48.763 INFO:journalctl@ceph.mgr.y.smithi125.stdout:Mar 24 21:56:48 smithi125 conmon[20260]: 2021-03-24T21:56:48.774 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:48.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:48.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:56:48.879 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:48.924 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:48.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:48.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:56:49.028 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:49.092 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:49.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:49.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:56:49.197 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:49.244 INFO:tasks.rados.rados.0.smithi125.stdout:701: done (3 left) 2021-03-24T21:56:49.244 INFO:tasks.rados.rados.0.smithi125.stdout:702: write (excl) oid 8 current snap is 68 2021-03-24T21:56:49.244 INFO:tasks.rados.rados.0.smithi125.stdout:702: seq_num 275 ranges {621925=482183,1559254=566428} 2021-03-24T21:56:49.252 INFO:tasks.rados.rados.0.smithi125.stdout:702: writing smithi12556470-8 from 621925 to 1104108 tid 1 2021-03-24T21:56:49.258 INFO:tasks.rados.rados.0.smithi125.stdout:702: writing smithi12556470-8 from 1559254 to 2125682 tid 2 2021-03-24T21:56:49.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:49.269 INFO:tasks.rados.rados.0.smithi125.stdout:697: finishing write tid 2 to smithi12556470-18 2021-03-24T21:56:49.270 INFO:tasks.rados.rados.0.smithi125.stdout:697: finishing write tid 3 to smithi12556470-18 2021-03-24T21:56:49.270 INFO:tasks.rados.rados.0.smithi125.stdout:697: finishing write tid 4 to smithi12556470-18 2021-03-24T21:56:49.271 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 18 v 154 (ObjNum 272 snap 68 seq_num 272) dirty exists 2021-03-24T21:56:49.271 INFO:tasks.rados.rados.0.smithi125.stdout:697: left oid 18 (ObjNum 272 snap 68 seq_num 272) 2021-03-24T21:56:49.271 INFO:tasks.rados.rados.0.smithi125.stdout:699: expect (ObjNum 150 snap 30 seq_num 150) 2021-03-24T21:56:49.273 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:49.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:56:49.340 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:49 smithi125 conmon[24927]: cluster 2021-03-24T21:56:49.340 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:49 smithi125 conmon[24927]: 2021-03-24T21: 2021-03-24T21:56:49.340 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:49 smithi125 conmon[24927]: 56:47.920944+0000 osd.4 (osd.4) 67 : cluster [DBG] 3.1c scrub ok 2021-03-24T21:56:49.340 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:49 smithi125 conmon[24927]: cluster 2021-03-24T21:56: 2021-03-24T21:56:49.341 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:49 smithi125 conmon[24927]: 48.144375+0000 osd.5 (osd.5) 65 : cluster 2021-03-24T21:56:49.341 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:49 smithi125 conmon[24927]: [DBG] 5.16 scrub ok 2021-03-24T21:56:49.341 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:49 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:56:49.341 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:49 smithi125 conmon[24927]: 56:48.230922+0000 mon.a (mon.0) 2021-03-24T21:56:49.342 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:49 smithi125 conmon[24927]: 860 : cluster [DBG] osdmap e239: 8 total, 8 up, 6 in 2021-03-24T21:56:49.342 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:49 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:56:49.342 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:49 smithi125 conmon[24927]: 56:48.267706+0000 osd.0 (osd.0) 2021-03-24T21:56:49.342 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:49 smithi125 conmon[24927]: 64 : cluster [DBG] 7.f scrub ok 2021-03-24T21:56:49.343 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:49 smithi125 conmon[19936]: cluster 2021-03-24T21:56:49.343 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:49 smithi125 conmon[19936]: 2021-03-24T21:56:49.343 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:49 smithi125 conmon[19936]: 2021 2021-03-24T21:56:49.343 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:49 smithi125 conmon[19936]: - 2021-03-24T21:56:49.344 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:49 smithi125 conmon[19936]: 03-24T21: 2021-03-24T21:56:49.344 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:49 smithi125 conmon[19936]: 56:47.920944+0000 osd.4 (osd.4) 67 : cluster 2021-03-24T21:56:49.344 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:49 smithi125 conmon[19936]: [DBG] 3.1c scrub ok 2021-03-24T21:56:49.344 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:49 smithi125 conmon[19936]: cluster 2021-03-24T21:56:49.345 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:49 smithi125 conmon[19936]: 2021-03-24T21:56:49.345 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:49 smithi125 conmon[19936]: 2021 2021-03-24T21:56:49.345 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:49 smithi125 conmon[19936]: - 2021-03-24T21:56:49.345 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:49 smithi125 conmon[19936]: 03 2021-03-24T21:56:49.345 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:49 smithi125 conmon[19936]: - 2021-03-24T21:56:49.346 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:49 smithi125 conmon[19936]: 24T21:56:48.144375 2021-03-24T21:56:49.346 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:49 smithi125 conmon[19936]: +0000 osd.5 (osd.5) 65 : cluster [DBG] 5.16 scrub ok 2021-03-24T21:56:49.346 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:49 smithi125 conmon[19936]: 2021-03-24T21:56:49.346 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:49 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:56:49.347 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:49 smithi125 conmon[19936]: -24T21:56:48.230922+0000 mon.a (mon 2021-03-24T21:56:49.347 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:49 smithi125 conmon[19936]: .0) 860 : cluster 2021-03-24T21:56:49.347 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:49 smithi125 conmon[19936]: [DBG] osdmap e239: 8 total, 8 up, 6 in 2021-03-24T21:56:49.347 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:49 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:56:49.348 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:49 smithi125 conmon[19936]: -24T21:56:48.267706+0000 osd.0 ( 2021-03-24T21:56:49.348 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:49 smithi125 conmon[19936]: osd.0) 64 : cluster 2021-03-24T21:56:49.348 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:49 smithi125 conmon[19936]: [DBG] 7.f scrub ok 2021-03-24T21:56:49.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:49 smithi142 conmon[20125]: cluster 2021-03-24T21:56:47.920944+0000 osd.4 (osd.4) 67 : cluster [DBG] 3.1c scrub ok 2021-03-24T21:56:49.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:49 smithi142 conmon[20125]: cluster 2021-03-24T21:56:48.144375+0000 osd.5 (osd.5) 65 : cluster [DBG] 5.16 scrub ok 2021-03-24T21:56:49.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:49 smithi142 conmon[20125]: cluster 2021-03-24T21:56:48.230922+0000 mon.a (mon.0) 860 : cluster [DBG] osdmap e239: 8 total, 8 up, 6 in 2021-03-24T21:56:49.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:49 smithi142 conmon[20125]: cluster 2021-03-24T21:56:48.267706 2021-03-24T21:56:49.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:49 smithi142 conmon[20125]: +0000 osd.0 (osd.0) 64 : cluster [DBG] 7.f scrub ok 2021-03-24T21:56:49.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:49 smithi142 conmon[20125]: 2021-03-24T21:56:49.375 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:49.390 INFO:tasks.rados.rados.0.smithi125.stdout:700: finishing write tid 1 to smithi12556470-29 2021-03-24T21:56:49.390 INFO:tasks.rados.rados.0.smithi125.stdout:700: finishing write tid 2 to smithi12556470-29 2021-03-24T21:56:49.390 INFO:tasks.rados.rados.0.smithi125.stdout:700: finishing write tid 3 to smithi12556470-29 2021-03-24T21:56:49.391 INFO:tasks.rados.rados.0.smithi125.stdout:700: finishing write tid 4 to smithi12556470-29 2021-03-24T21:56:49.391 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 29 v 128 (ObjNum 274 snap 68 seq_num 274) dirty exists 2021-03-24T21:56:49.391 INFO:tasks.rados.rados.0.smithi125.stdout:700: left oid 29 (ObjNum 274 snap 68 seq_num 274) 2021-03-24T21:56:49.391 INFO:tasks.rados.rados.0.smithi125.stdout:702: finishing write tid 1 to smithi12556470-8 2021-03-24T21:56:49.392 INFO:tasks.rados.rados.0.smithi125.stdout:702: finishing write tid 2 to smithi12556470-8 2021-03-24T21:56:49.392 INFO:tasks.rados.rados.0.smithi125.stdout:697: done (3 left) 2021-03-24T21:56:49.392 INFO:tasks.rados.rados.0.smithi125.stdout:699: done (2 left) 2021-03-24T21:56:49.392 INFO:tasks.rados.rados.0.smithi125.stdout:700: done (1 left) 2021-03-24T21:56:49.393 INFO:tasks.rados.rados.0.smithi125.stdout:703: read oid 16 snap 59 2021-03-24T21:56:49.393 INFO:tasks.rados.rados.0.smithi125.stdout:703: expect (ObjNum 236 snap 53 seq_num 236) 2021-03-24T21:56:49.393 INFO:tasks.rados.rados.0.smithi125.stdout:704: write (excl) oid 1 current snap is 68 2021-03-24T21:56:49.393 INFO:tasks.rados.rados.0.smithi125.stdout:702: finishing write tid 3 to smithi12556470-8 2021-03-24T21:56:49.394 INFO:tasks.rados.rados.0.smithi125.stdout:704: seq_num 276 ranges {638732=557432,1617736=710086} 2021-03-24T21:56:49.396 INFO:tasks.rados.rados.0.smithi125.stdout:704: writing smithi12556470-1 from 638732 to 1196164 tid 1 2021-03-24T21:56:49.403 INFO:tasks.rados.rados.0.smithi125.stdout:704: writing smithi12556470-1 from 1617736 to 2327822 tid 2 2021-03-24T21:56:49.403 INFO:tasks.rados.rados.0.smithi125.stdout:705: write (excl) oid 24 current snap is 68 2021-03-24T21:56:49.403 INFO:tasks.rados.rados.0.smithi125.stdout:702: finishing write tid 4 to smithi12556470-8 2021-03-24T21:56:49.404 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 8 v 38 (ObjNum 275 snap 68 seq_num 275) dirty exists 2021-03-24T21:56:49.404 INFO:tasks.rados.rados.0.smithi125.stdout:702: left oid 8 (ObjNum 275 snap 68 seq_num 275) 2021-03-24T21:56:49.404 INFO:tasks.rados.rados.0.smithi125.stdout:705: seq_num 277 ranges {655539=632681,2076218=570097} 2021-03-24T21:56:49.410 INFO:tasks.rados.rados.0.smithi125.stdout:705: writing smithi12556470-24 from 655539 to 1288220 tid 1 2021-03-24T21:56:49.418 INFO:tasks.rados.rados.0.smithi125.stdout:705: writing smithi12556470-24 from 2076218 to 2646315 tid 2 2021-03-24T21:56:49.419 INFO:tasks.rados.rados.0.smithi125.stdout:702: done (3 left) 2021-03-24T21:56:49.419 INFO:tasks.rados.rados.0.smithi125.stdout:706: snap_create 2021-03-24T21:56:49.419 INFO:tasks.rados.rados.0.smithi125.stdout:703: expect (ObjNum 236 snap 53 seq_num 236) 2021-03-24T21:56:49.428 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:49.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:49.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:56:49.533 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:49.585 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:49.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:49.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:56:49.651 INFO:tasks.rados.rados.0.smithi125.stdout:704: finishing write tid 1 to smithi12556470-1 2021-03-24T21:56:49.652 INFO:tasks.rados.rados.0.smithi125.stdout:704: finishing write tid 2 to smithi12556470-1 2021-03-24T21:56:49.652 INFO:tasks.rados.rados.0.smithi125.stdout:704: finishing write tid 3 to smithi12556470-1 2021-03-24T21:56:49.652 INFO:tasks.rados.rados.0.smithi125.stdout:704: finishing write tid 4 to smithi12556470-1 2021-03-24T21:56:49.653 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 1 v 223 (ObjNum 276 snap 68 seq_num 276) dirty exists 2021-03-24T21:56:49.653 INFO:tasks.rados.rados.0.smithi125.stdout:704: left oid 1 (ObjNum 276 snap 68 seq_num 276) 2021-03-24T21:56:49.653 INFO:tasks.rados.rados.0.smithi125.stdout:705: finishing write tid 1 to smithi12556470-24 2021-03-24T21:56:49.653 INFO:tasks.rados.rados.0.smithi125.stdout:705: finishing write tid 2 to smithi12556470-24 2021-03-24T21:56:49.653 INFO:tasks.rados.rados.0.smithi125.stdout:705: finishing write tid 3 to smithi12556470-24 2021-03-24T21:56:49.654 INFO:tasks.rados.rados.0.smithi125.stdout:705: finishing write tid 4 to smithi12556470-24 2021-03-24T21:56:49.654 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 24 v 236 (ObjNum 277 snap 68 seq_num 277) dirty exists 2021-03-24T21:56:49.654 INFO:tasks.rados.rados.0.smithi125.stdout:705: left oid 24 (ObjNum 277 snap 68 seq_num 277) 2021-03-24T21:56:49.689 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:49.751 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:49.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:49.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:56:49.856 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:49.918 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:49.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:49.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:56:50.024 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:50.087 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:50.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:50.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:56:50.134 INFO:tasks.rados.rados.0.smithi125.stdout:703: done (3 left) 2021-03-24T21:56:50.135 INFO:tasks.rados.rados.0.smithi125.stdout:704: done (2 left) 2021-03-24T21:56:50.135 INFO:tasks.rados.rados.0.smithi125.stdout:705: done (1 left) 2021-03-24T21:56:50.135 INFO:tasks.rados.rados.0.smithi125.stdout:706: done (0 left) 2021-03-24T21:56:50.135 INFO:tasks.rados.rados.0.smithi125.stdout:707: write oid 14 current snap is 69 2021-03-24T21:56:50.136 INFO:tasks.rados.rados.0.smithi125.stdout:707: seq_num 278 ranges {672346=707930,2134700=537646} 2021-03-24T21:56:50.143 INFO:tasks.rados.rados.0.smithi125.stdout:707: writing smithi12556470-14 from 672346 to 1380276 tid 1 2021-03-24T21:56:50.151 INFO:tasks.rados.rados.0.smithi125.stdout:707: writing smithi12556470-14 from 2134700 to 2672346 tid 2 2021-03-24T21:56:50.151 INFO:tasks.rados.rados.0.smithi125.stdout:708: read oid 32 snap -1 2021-03-24T21:56:50.152 INFO:tasks.rados.rados.0.smithi125.stdout:708: expect (ObjNum 268 snap 68 seq_num 268) 2021-03-24T21:56:50.152 INFO:tasks.rados.rados.0.smithi125.stdout:709: read oid 39 snap -1 2021-03-24T21:56:50.152 INFO:tasks.rados.rados.0.smithi125.stdout:709: expect deleted 2021-03-24T21:56:50.152 INFO:tasks.rados.rados.0.smithi125.stdout:710: delete oid 26 current snap is 69 2021-03-24T21:56:50.155 INFO:tasks.rados.rados.0.smithi125.stdout:707: finishing write tid 1 to smithi12556470-14 2021-03-24T21:56:50.157 INFO:tasks.rados.rados.0.smithi125.stdout:710: done (3 left) 2021-03-24T21:56:50.157 INFO:tasks.rados.rados.0.smithi125.stdout:711: read oid 24 snap 68 2021-03-24T21:56:50.157 INFO:tasks.rados.rados.0.smithi125.stdout:711: expect (ObjNum 277 snap 68 seq_num 277) 2021-03-24T21:56:50.158 INFO:tasks.rados.rados.0.smithi125.stdout:712: read oid 36 snap -1 2021-03-24T21:56:50.158 INFO:tasks.rados.rados.0.smithi125.stdout:712: expect (ObjNum 225 snap 50 seq_num 225) 2021-03-24T21:56:50.158 INFO:tasks.rados.rados.0.smithi125.stdout:713: snap_create 2021-03-24T21:56:50.161 INFO:tasks.rados.rados.0.smithi125.stdout:707: finishing write tid 2 to smithi12556470-14 2021-03-24T21:56:50.161 INFO:tasks.rados.rados.0.smithi125.stdout:707: finishing write tid 3 to smithi12556470-14 2021-03-24T21:56:50.161 INFO:tasks.rados.rados.0.smithi125.stdout:707: finishing write tid 4 to smithi12556470-14 2021-03-24T21:56:50.162 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 14 v 203 (ObjNum 278 snap 69 seq_num 278) dirty exists 2021-03-24T21:56:50.162 INFO:tasks.rados.rados.0.smithi125.stdout:707: left oid 14 (ObjNum 278 snap 69 seq_num 278) 2021-03-24T21:56:50.164 INFO:tasks.rados.rados.0.smithi125.stdout:708: expect (ObjNum 268 snap 68 seq_num 268) 2021-03-24T21:56:50.192 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:50.244 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:50.245 INFO:tasks.rados.rados.0.smithi125.stdout:712: expect (ObjNum 225 snap 50 seq_num 225) 2021-03-24T21:56:50.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:50.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:56:50.350 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:50.410 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:50.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:50 smithi125 conmon[19936]: audit 2021-03-24T21:56 2021-03-24T21:56:50.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:50 smithi125 conmon[19936]: :48.680625+0000 mgr.y (mgr.14144) 384 : audit [DBG] from='client.34928 v1:172.21.15.125:0/3901984171' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:56:50.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:50 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:56:50.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:50 smithi125 conmon[19936]: -24T21:56:48.757475+0000 mgr.y (mgr.14144) 385 : 2021-03-24T21:56:50.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:50 smithi125 conmon[19936]: cluster [DBG] pgmap v521: 158 pgs: 7 peering, 151 active+clean; 134 MiB data, 474 MiB used, 536 GiB / 536 GiB avail; 2.3 MiB/s rd, 326 KiB/s wr, 4 op/s; 0 B/s, 0 objects/s recovering 2021-03-24T21:56:50.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:50 smithi125 conmon[19936]: cluster 2021-03-24T21:56:50.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:50 smithi125 conmon[19936]: 2021-03-24T21:56:49.241648+0000 mon.a (mon. 2021-03-24T21:56:50.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:50 smithi125 conmon[19936]: 0) 861 : cluster [DBG] osdmap e240: 8 total, 8 up, 6 in 2021-03-24T21:56:50.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:50 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:56:50.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:50 smithi125 conmon[19936]: 24T21:56:49.970064+0000 osd.2 (osd.2) 2021-03-24T21:56:50.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:50 smithi125 conmon[19936]: 47 : cluster [DBG] 6.f scrub ok 2021-03-24T21:56:50.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:50 smithi125 conmon[19936]: cluster 2021-03-24T21:56:50.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:50 smithi125 conmon[19936]: 2021-03-24T21:56:50.132414+0000 2021-03-24T21:56:50.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:50 smithi125 conmon[19936]: mon.a (mon.0) 862 : cluster [DBG] osdmap e241: 8 total, 8 up, 6 in 2021-03-24T21:56:50.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:50 smithi125 conmon[19936]: 2021-03-24T21:56:50.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:50 smithi125 conmon[24927]: audit 2021-03-24T21:56:48 2021-03-24T21:56:50.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:50 smithi125 conmon[24927]: .680625+0000 mgr.y (mgr.14144) 384 : audit [DBG] from='client.34928 v1:172.21.15.125:0/3901984171' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:56:50.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:50 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:56:50.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:50 smithi125 conmon[24927]: 24T21:56:48.757475+0000 mgr.y (mgr.14144) 385 : cluster [DBG] pgmap v521: 158 pgs: 7 peering, 151 active+clean; 134 MiB data, 474 MiB used, 536 GiB / 536 GiB avail; 2.3 MiB/s rd, 326 KiB/s wr, 4 op/s; 0 B/s, 0 objects/s recovering 2021-03-24T21:56:50.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:50 smithi125 conmon[24927]: cluster 2021-03-24T21:56:49.241648 2021-03-24T21:56:50.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:50 smithi125 conmon[24927]: +0000 mon.a (mon.0) 861 : cluster [DBG] osdmap e240: 8 total, 8 up, 6 in 2021-03-24T21:56:50.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:50 smithi125 conmon[24927]: cluster 2021-03-24T21:56 2021-03-24T21:56:50.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:50 smithi125 conmon[24927]: :49.970064+0000 osd.2 (osd.2) 47 : cluster [DBG] 2021-03-24T21:56:50.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:50 smithi125 conmon[24927]: 6.f scrub ok 2021-03-24T21:56:50.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:50 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:56:50.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:50 smithi125 conmon[24927]: 03-24T21:56:50.132414+0000 mon.a ( 2021-03-24T21:56:50.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:50 smithi125 conmon[24927]: mon.0) 862 : cluster [DBG] osdmap e241: 8 total, 8 up, 6 in 2021-03-24T21:56:50.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:50.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:56:50.486 INFO:tasks.rados.rados.0.smithi125.stdout:711: expect (ObjNum 277 snap 68 seq_num 277) 2021-03-24T21:56:50.519 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:50.570 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:50.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:50.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:56:50.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:50 smithi142 conmon[20125]: audit 2021-03-24T21: 2021-03-24T21:56:50.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:50 smithi142 conmon[20125]: 56:48.680625+0000 mgr.y (mgr.14144) 384 : audit [DBG] from='client.34928 v1:172.21.15.125:0/3901984171' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2021-03-24T21:56:50.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:50 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:56:50.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:50 smithi142 conmon[20125]: 03-24T21:56:48.757475+0000 mgr.y (mgr. 2021-03-24T21:56:50.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:50 smithi142 conmon[20125]: 14144) 385 : cluster [DBG] pgmap v521: 158 pgs: 7 peering, 151 active+clean; 134 MiB data, 474 MiB used, 536 GiB / 536 GiB avail; 2.3 MiB/s rd, 326 KiB/s wr, 4 op/s; 0 B/s, 0 objects/s recovering 2021-03-24T21:56:50.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:50 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:56:50.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:50 smithi142 conmon[20125]: -03-24T21:56 2021-03-24T21:56:50.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:50 smithi142 conmon[20125]: :49.241648+0000 mon.a (mon.0 2021-03-24T21:56:50.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:50 smithi142 conmon[20125]: ) 861 : cluster [DBG] osdmap e240: 8 total, 8 up, 6 in 2021-03-24T21:56:50.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:50 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:56:50.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:50 smithi142 conmon[20125]: -03-24T21:56 2021-03-24T21:56:50.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:50 smithi142 conmon[20125]: :49.970064+0000 osd.2 (osd.2 2021-03-24T21:56:50.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:50 smithi142 conmon[20125]: ) 47 : cluster [DBG] 6.f scrub ok 2021-03-24T21:56:50.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:50 smithi142 conmon[20125]: cluster 2021-03-24T21:56:50.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:50 smithi142 conmon[20125]: 2021-03-24T21: 2021-03-24T21:56:50.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:50 smithi142 conmon[20125]: 56:50.132414+0000 mon.a (mon. 2021-03-24T21:56:50.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:50 smithi142 conmon[20125]: 0) 862 : cluster [DBG] osdmap e241: 8 total, 8 up, 6 in 2021-03-24T21:56:50.675 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:50.727 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:50.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:50.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:56:50.833 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:50.886 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:50.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:50.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:56:50.993 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:51.043 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:51.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:51.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:56:51.146 INFO:tasks.rados.rados.0.smithi125.stdout:707: done (5 left) 2021-03-24T21:56:51.147 INFO:tasks.rados.rados.0.smithi125.stdout:708: done (4 left) 2021-03-24T21:56:51.147 INFO:tasks.rados.rados.0.smithi125.stdout:709: done (3 left) 2021-03-24T21:56:51.147 INFO:tasks.rados.rados.0.smithi125.stdout:711: done (2 left) 2021-03-24T21:56:51.148 INFO:tasks.rados.rados.0.smithi125.stdout:712: done (1 left) 2021-03-24T21:56:51.148 INFO:tasks.rados.rados.0.smithi125.stdout:713: done (0 left) 2021-03-24T21:56:51.148 INFO:tasks.rados.rados.0.smithi125.stdout:714: read oid 18 snap -1 2021-03-24T21:56:51.148 INFO:tasks.rados.rados.0.smithi125.stdout:714: expect (ObjNum 272 snap 68 seq_num 272) 2021-03-24T21:56:51.149 INFO:tasks.rados.rados.0.smithi125.stdout:715: write oid 16 current snap is 70 2021-03-24T21:56:51.149 INFO:tasks.rados.rados.0.smithi125.stdout:715: seq_num 280 ranges {705960=458428,1735311=717424} 2021-03-24T21:56:51.149 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:51.155 INFO:tasks.rados.rados.0.smithi125.stdout:715: writing smithi12556470-16 from 705960 to 1164388 tid 1 2021-03-24T21:56:51.163 INFO:tasks.rados.rados.0.smithi125.stdout:715: writing smithi12556470-16 from 1735311 to 2452735 tid 2 2021-03-24T21:56:51.163 INFO:tasks.rados.rados.0.smithi125.stdout:716: write oid 23 current snap is 70 2021-03-24T21:56:51.163 INFO:tasks.rados.rados.0.smithi125.stdout:716: seq_num 281 ranges {722767=533677,1793793=461082} 2021-03-24T21:56:51.170 INFO:tasks.rados.rados.0.smithi125.stdout:716: writing smithi12556470-23 from 722767 to 1256444 tid 1 2021-03-24T21:56:51.175 INFO:tasks.rados.rados.0.smithi125.stdout:716: writing smithi12556470-23 from 1793793 to 2254875 tid 2 2021-03-24T21:56:51.175 INFO:tasks.rados.rados.0.smithi125.stdout:717: delete oid 19 current snap is 70 2021-03-24T21:56:51.190 INFO:tasks.rados.rados.0.smithi125.stdout:714: expect (ObjNum 272 snap 68 seq_num 272) 2021-03-24T21:56:51.200 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:51.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:51.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:56:51.305 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:51.318 INFO:tasks.rados.rados.0.smithi125.stdout:716: finishing write tid 1 to smithi12556470-23 2021-03-24T21:56:51.319 INFO:tasks.rados.rados.0.smithi125.stdout:715: finishing write tid 1 to smithi12556470-16 2021-03-24T21:56:51.319 INFO:tasks.rados.rados.0.smithi125.stdout:715: finishing write tid 2 to smithi12556470-16 2021-03-24T21:56:51.320 INFO:tasks.rados.rados.0.smithi125.stdout:716: finishing write tid 2 to smithi12556470-23 2021-03-24T21:56:51.320 INFO:tasks.rados.rados.0.smithi125.stdout:715: finishing write tid 3 to smithi12556470-16 2021-03-24T21:56:51.321 INFO:tasks.rados.rados.0.smithi125.stdout:716: finishing write tid 3 to smithi12556470-23 2021-03-24T21:56:51.321 INFO:tasks.rados.rados.0.smithi125.stdout:715: finishing write tid 4 to smithi12556470-16 2021-03-24T21:56:51.321 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 16 v 203 (ObjNum 280 snap 70 seq_num 280) dirty exists 2021-03-24T21:56:51.322 INFO:tasks.rados.rados.0.smithi125.stdout:715: left oid 16 (ObjNum 280 snap 70 seq_num 280) 2021-03-24T21:56:51.322 INFO:tasks.rados.rados.0.smithi125.stdout:716: finishing write tid 4 to smithi12556470-23 2021-03-24T21:56:51.323 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 23 v 189 (ObjNum 281 snap 70 seq_num 281) dirty exists 2021-03-24T21:56:51.323 INFO:tasks.rados.rados.0.smithi125.stdout:716: left oid 23 (ObjNum 281 snap 70 seq_num 281) 2021-03-24T21:56:51.324 INFO:tasks.rados.rados.0.smithi125.stdout:714: done (3 left) 2021-03-24T21:56:51.324 INFO:tasks.rados.rados.0.smithi125.stdout:715: done (2 left) 2021-03-24T21:56:51.325 INFO:tasks.rados.rados.0.smithi125.stdout:716: done (1 left) 2021-03-24T21:56:51.325 INFO:tasks.rados.rados.0.smithi125.stdout:717: done (0 left) 2021-03-24T21:56:51.325 INFO:tasks.rados.rados.0.smithi125.stdout:718: read oid 20 snap -1 2021-03-24T21:56:51.326 INFO:tasks.rados.rados.0.smithi125.stdout:718: expect (ObjNum 200 snap 43 seq_num 200) 2021-03-24T21:56:51.326 INFO:tasks.rados.rados.0.smithi125.stdout:719: snap_remove snap 68 2021-03-24T21:56:51.353 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:51.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:51.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:56:51.458 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:51.472 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:51 smithi142 conmon[20125]: cluster 2021-03-24T21:56:49.886046+0000 osd.7 (osd.7) 57 : cluster [DBG] 7.5 deep-scrub ok 2021-03-24T21:56:51.472 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:51 smithi142 conmon[20125]: cluster 2021-03-24T21:56:51.145046+0000 mon.a (mon.0) 863 : cluster [DBG] osdmap e242: 8 total, 8 up, 6 in 2021-03-24T21:56:51.516 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:51.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:51.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:56:51.622 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:51.643 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:51 smithi125 conmon[24927]: cluster 2021-03-24T21:56 2021-03-24T21:56:51.643 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:51 smithi125 conmon[24927]: :49.886046+0000 osd.7 (osd.7) 57 : cluster [DBG] 7.5 deep-scrub ok 2021-03-24T21:56:51.644 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:51 smithi125 conmon[24927]: cluster 2021-03-24T21:56: 2021-03-24T21:56:51.644 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:51 smithi125 conmon[24927]: 51.145046+0000 mon.a (mon.0) 863 : 2021-03-24T21:56:51.644 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:51 smithi125 conmon[24927]: cluster [DBG] osdmap e242: 8 total, 8 up, 6 in 2021-03-24T21:56:51.645 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:51 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:56:51.645 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:51 smithi125 conmon[19936]: -03-24T21:56:49.886046+0000 osd.7 (osd.7) 57 : cluster [DBG] 7.5 deep-scrub ok 2021-03-24T21:56:51.645 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:51 smithi125 conmon[19936]: cluster 2021-03-24T21:56:51.646 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:51 smithi125 conmon[19936]: 2021-03-24T21:56:51.145046+0000 mon.a (mon.0) 863 : cluster 2021-03-24T21:56:51.646 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:51 smithi125 conmon[19936]: [DBG] osdmap e242: 8 total, 8 up, 6 in 2021-03-24T21:56:51.676 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:51.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:51.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:56:51.782 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:51.833 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:51.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:51.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:56:51.939 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:51.996 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:52.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:52.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:56:52.102 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:52.159 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:52.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:52.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:56:52.257 INFO:tasks.rados.rados.0.smithi125.stdout:719: done (1 left) 2021-03-24T21:56:52.257 INFO:tasks.rados.rados.0.smithi125.stdout:720: snap_remove snap 69 2021-03-24T21:56:52.267 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:52.315 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:52.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:52.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:56:52.421 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:52.477 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:52.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:52.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:56:52.584 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:52.594 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[24927]: cluster 2021-03-24T21:56:52.594 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[24927]: 2021-03-24T21:56:52.594 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[24927]: 2021 2021-03-24T21:56:52.594 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[24927]: - 2021-03-24T21:56:52.595 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[24927]: 03 2021-03-24T21:56:52.595 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[24927]: - 2021-03-24T21:56:52.595 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[24927]: 24 2021-03-24T21:56:52.595 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[24927]: T 2021-03-24T21:56:52.595 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[24927]: 21 2021-03-24T21:56:52.596 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[24927]: : 2021-03-24T21:56:52.596 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[24927]: 56 2021-03-24T21:56:52.596 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[24927]: : 2021-03-24T21:56:52.596 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[24927]: 50.758113+0000 mgr.y (mgr.14144) 386 : cluster 2021-03-24T21:56:52.597 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[24927]: [DBG] pgmap v524: 158 pgs: 158 active+clean; 138 MiB data, 478 MiB used, 536 GiB / 536 GiB avail; 13 MiB/s rd, 1.4 MiB/s wr, 14 op/s; 1.2 MiB/s, 8 objects/s recovering 2021-03-24T21:56:52.597 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[24927]: cluster 2021-03-24T21:56:52.597 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[24927]: 2021-03-24T21:56:50.934761+0000 osd.7 (osd.7) 2021-03-24T21:56:52.597 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[24927]: 58 : cluster [DBG] 7.15 scrub ok 2021-03-24T21:56:52.597 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[24927]: cluster 2021-03-24T21:56:52.598 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[24927]: 2021-03-24T21:56:52.598 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[24927]: 2021 2021-03-24T21:56:52.598 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[24927]: - 2021-03-24T21:56:52.598 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[24927]: 03 2021-03-24T21:56:52.598 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[24927]: -24T21:56:51.990755+0000 2021-03-24T21:56:52.599 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[24927]: osd.2 (osd.2 2021-03-24T21:56:52.599 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[24927]: ) 48 : cluster [DBG] 5.8 scrub ok 2021-03-24T21:56:52.599 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[19936]: cluster 2021-03-24T21:56:52.600 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[19936]: 2021-03-24T21:56:52.600 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[19936]: 2021 2021-03-24T21:56:52.600 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[19936]: - 2021-03-24T21:56:52.600 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[19936]: 03 2021-03-24T21:56:52.600 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[19936]: - 2021-03-24T21:56:52.601 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[19936]: 24 2021-03-24T21:56:52.601 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[19936]: T 2021-03-24T21:56:52.601 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[19936]: 21 2021-03-24T21:56:52.601 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[19936]: : 2021-03-24T21:56:52.602 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[19936]: 56 2021-03-24T21:56:52.602 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[19936]: : 2021-03-24T21:56:52.602 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[19936]: 50 2021-03-24T21:56:52.602 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[19936]: . 2021-03-24T21:56:52.602 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[19936]: 758113 2021-03-24T21:56:52.603 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[19936]: +0000 2021-03-24T21:56:52.603 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[19936]: 2021-03-24T21:56:52.603 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[19936]: mgr.y 2021-03-24T21:56:52.603 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[19936]: ( 2021-03-24T21:56:52.603 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[19936]: mgr 2021-03-24T21:56:52.604 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[19936]: . 2021-03-24T21:56:52.604 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[19936]: 14144 2021-03-24T21:56:52.604 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[19936]: ) 2021-03-24T21:56:52.604 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[19936]: 386 2021-03-24T21:56:52.604 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[19936]: : 2021-03-24T21:56:52.605 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[19936]: cluster 2021-03-24T21:56:52.605 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[19936]: 2021-03-24T21:56:52.605 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[19936]: [DBG] 2021-03-24T21:56:52.605 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[19936]: 2021-03-24T21:56:52.606 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[19936]: pgmap v524: 158 pgs: 158 active+clean; 138 MiB data, 478 MiB used, 536 GiB / 536 GiB avail; 13 MiB/s rd, 1.4 MiB/s wr, 14 op/s; 1.2 MiB/s, 8 objects/s recovering 2021-03-24T21:56:52.606 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[19936]: 2021-03-24T21:56:52.606 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[19936]: cluster 2021-03-24T21:56:52.606 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[19936]: 2021-03-24T21:56:50 2021-03-24T21:56:52.607 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[19936]: .934761+0000 osd.7 (osd.7) 58 : cluster [DBG] 7.15 scrub ok 2021-03-24T21:56:52.607 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[19936]: cluster 2021-03-24T21:56:51.990755+0000 osd.2 2021-03-24T21:56:52.607 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:52 smithi125 conmon[19936]: (osd.2) 48 : cluster [DBG] 5.8 scrub ok 2021-03-24T21:56:52.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:52 smithi142 conmon[20125]: cluster 2021-03-24T21:56:50.758113+0000 mgr.y 2021-03-24T21:56:52.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:52 smithi142 conmon[20125]: (mgr.14144) 386 : cluster [DBG] pgmap v524: 158 pgs: 158 active+clean; 138 MiB data, 478 MiB used, 536 GiB / 536 GiB avail; 13 MiB/s rd, 1.4 MiB/s wr, 14 op/s; 1.2 MiB/s, 8 objects/s recovering 2021-03-24T21:56:52.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:52 smithi142 conmon[20125]: cluster 2021-03-24T21:56:50.934761+0000 osd.7 (osd.7) 58 : cluster [DBG] 7.15 scrub ok 2021-03-24T21:56:52.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:52 smithi142 conmon[20125]: cluster 2021-03-24T21:56:51.990755+0000 osd.2 2021-03-24T21:56:52.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:52 smithi142 conmon[20125]: (osd.2) 48 : cluster [DBG] 5.8 scrub ok 2021-03-24T21:56:52.635 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:52.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:52.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:56:52.740 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:52.801 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:52.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:52.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:56:52.908 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:52.974 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:52.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:52.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:56:53.081 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:53.089 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-24T21:56:53.089 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T21:56:53.143 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:53.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:53.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:56:53.250 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:53.256 INFO:tasks.rados.rados.0.smithi125.stdout:720: done (1 left) 2021-03-24T21:56:53.257 INFO:tasks.rados.rados.0.smithi125.stdout:721: rollback oid 45 current snap is 70 2021-03-24T21:56:53.257 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 45 to 59 2021-03-24T21:56:53.257 INFO:tasks.rados.rados.0.smithi125.stdout:718: expect (ObjNum 200 snap 43 seq_num 200) 2021-03-24T21:56:53.301 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:53.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:53.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:56:53.407 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:53.432 INFO:tasks.rados.rados.0.smithi125.stdout:718: done (1 left) 2021-03-24T21:56:53.433 INFO:tasks.rados.rados.0.smithi125.stdout:722: write oid 40 current snap is 70 2021-03-24T21:56:53.433 INFO:tasks.rados.rados.0.smithi125.stdout:721: finishing rollback tid 0 to smithi12556470-45 2021-03-24T21:56:53.433 INFO:tasks.rados.rados.0.smithi125.stdout:722: seq_num 283 ranges {756381=400528,1910757=464751} 2021-03-24T21:56:53.439 INFO:tasks.rados.rados.0.smithi125.stdout:722: writing smithi12556470-40 from 756381 to 1156909 tid 1 2021-03-24T21:56:53.447 INFO:tasks.rados.rados.0.smithi125.stdout:722: writing smithi12556470-40 from 1910757 to 2375508 tid 2 2021-03-24T21:56:53.448 INFO:tasks.rados.rados.0.smithi125.stdout:723: write (excl) oid 2 current snap is 70 2021-03-24T21:56:53.448 INFO:tasks.rados.rados.0.smithi125.stdout:721: finishing rollback tid 1 to smithi12556470-45 2021-03-24T21:56:53.448 INFO:tasks.rados.rados.0.smithi125.stdout:721: finishing rollback tid 2 to smithi12556470-45 2021-03-24T21:56:53.448 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 45 v 173 (ObjNum 150 snap 30 seq_num 150) dirty exists 2021-03-24T21:56:53.449 INFO:tasks.rados.rados.0.smithi125.stdout:722: finishing write tid 1 to smithi12556470-40 2021-03-24T21:56:53.449 INFO:tasks.rados.rados.0.smithi125.stdout:723: seq_num 284 ranges {773188=475777,1969239=724762} 2021-03-24T21:56:53.461 INFO:tasks.rados.rados.0.smithi125.stdout:723: writing smithi12556470-2 from 773188 to 1248965 tid 1 2021-03-24T21:56:53.466 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:53.469 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:53 smithi125 conmon[19936]: cluster 2021-03-24T21:56:51.924955+0000 osd.4 (osd. 2021-03-24T21:56:53.469 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:53 smithi125 conmon[19936]: 4) 68 : cluster [DBG] 6.17 scrub ok 2021-03-24T21:56:53.470 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:53 smithi125 conmon[19936]: cluster 2021-03-24T21:56:51.977290+0000 osd.7 (osd.7) 59 : cluster [DBG] 5.1a deep-scrub ok 2021-03-24T21:56:53.470 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:53 smithi125 conmon[19936]: cluster 2021-03-24T21:56:52.226108+0000 osd.0 (osd.0) 65 : cluster [DBG] 6.10 scrub ok 2021-03-24T21:56:53.470 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:53 smithi125 conmon[19936]: cluster 2021-03-24T21:56:52.254252+0000 mon.a (mon.0) 864 : cluster [DBG] osdmap e243: 8 total, 8 up, 6 in 2021-03-24T21:56:53.470 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:53 smithi125 conmon[19936]: cluster 2021-03-24T21:56:52.703809+0000 osd.2 (osd.2) 49 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:56:53.471 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:53 smithi125 conmon[19936]: cluster 2021-03-24T21:56:52.704242+0000 osd.2 (osd.2) 50 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:56:53.471 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:53 smithi125 conmon[19936]: cluster 2021-03-24T21:56:53.254945+0000 mon.a (mon.0) 865 : cluster [DBG] osdmap e244: 8 total, 8 up, 6 in 2021-03-24T21:56:53.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:53.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:56:53.473 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:53 smithi125 conmon[24927]: cluster 2021-03-24T21:56:51.924955+0000 2021-03-24T21:56:53.473 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:53 smithi125 conmon[24927]: osd.4 (osd.4) 68 : cluster [DBG] 6.17 scrub ok 2021-03-24T21:56:53.473 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:53 smithi125 conmon[24927]: cluster 2021-03-24T21:56 2021-03-24T21:56:53.474 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:53 smithi125 conmon[24927]: :51.977290+0000 osd.7 (osd.7) 59 : cluster [DBG] 5.1a deep-scrub ok 2021-03-24T21:56:53.474 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:53 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:56:53.474 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:53 smithi125 conmon[24927]: :56:52.226108+0000 osd.0 (osd.0) 65 2021-03-24T21:56:53.474 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:53 smithi125 conmon[24927]: : cluster [DBG] 6.10 scrub ok 2021-03-24T21:56:53.475 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:53 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:56:53.475 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:53 smithi125 conmon[24927]: 56:52.254252+0000 mon.a (mon.0) 864 2021-03-24T21:56:53.475 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:53 smithi125 conmon[24927]: : cluster [DBG] osdmap e243: 8 total, 8 up, 6 in 2021-03-24T21:56:53.475 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:53 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:56:53.476 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:53 smithi125 conmon[24927]: 56:52.703809+0000 osd.2 (osd.2) 49 : 2021-03-24T21:56:53.476 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:53 smithi125 conmon[24927]: cluster [DBG] purged_snaps scrub starts 2021-03-24T21:56:53.476 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:53 smithi125 conmon[24927]: cluster 2021-03-24T21:56 2021-03-24T21:56:53.476 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:53 smithi125 conmon[24927]: :52.704242+0000 osd.2 (osd.2) 50 : 2021-03-24T21:56:53.476 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:53 smithi125 conmon[24927]: cluster [DBG] purged_snaps scrub ok 2021-03-24T21:56:53.477 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:53 smithi125 conmon[24927]: cluster 2021-03-24T21:56 2021-03-24T21:56:53.477 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:53 smithi125 conmon[24927]: :53.254945+0000 mon.a (mon.0) 865 : 2021-03-24T21:56:53.477 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:53 smithi125 conmon[24927]: cluster [DBG] osdmap e244: 8 total, 8 up, 6 in 2021-03-24T21:56:53.478 INFO:tasks.rados.rados.0.smithi125.stdout:723: writing smithi12556470-2 from 1969239 to 2694001 tid 2 2021-03-24T21:56:53.479 INFO:tasks.rados.rados.0.smithi125.stdout:721: done (2 left) 2021-03-24T21:56:53.479 INFO:tasks.rados.rados.0.smithi125.stdout:724: delete oid 41 current snap is 70 2021-03-24T21:56:53.479 INFO:tasks.rados.rados.0.smithi125.stdout:722: finishing write tid 2 to smithi12556470-40 2021-03-24T21:56:53.479 INFO:tasks.rados.rados.0.smithi125.stdout:722: finishing write tid 3 to smithi12556470-40 2021-03-24T21:56:53.480 INFO:tasks.rados.rados.0.smithi125.stdout:722: finishing write tid 4 to smithi12556470-40 2021-03-24T21:56:53.480 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 40 v 209 (ObjNum 283 snap 70 seq_num 283) dirty exists 2021-03-24T21:56:53.480 INFO:tasks.rados.rados.0.smithi125.stdout:722: left oid 40 (ObjNum 283 snap 70 seq_num 283) 2021-03-24T21:56:53.480 INFO:tasks.rados.rados.0.smithi125.stdout:723: finishing write tid 1 to smithi12556470-2 2021-03-24T21:56:53.481 INFO:tasks.rados.rados.0.smithi125.stdout:722: done (2 left) 2021-03-24T21:56:53.481 INFO:tasks.rados.rados.0.smithi125.stdout:724: done (1 left) 2021-03-24T21:56:53.481 INFO:tasks.rados.rados.0.smithi125.stdout:725: write (excl) oid 49 current snap is 70 2021-03-24T21:56:53.481 INFO:tasks.rados.rados.0.smithi125.stdout:725: seq_num 286 ranges {406802=626275,1686203=728431} 2021-03-24T21:56:53.482 INFO:tasks.rados.rados.0.smithi125.stdout:725: writing smithi12556470-49 from 406802 to 1033077 tid 1 2021-03-24T21:56:53.488 INFO:tasks.rados.rados.0.smithi125.stdout:725: writing smithi12556470-49 from 1686203 to 2414634 tid 2 2021-03-24T21:56:53.488 INFO:tasks.rados.rados.0.smithi125.stdout:726: read oid 27 snap 59 2021-03-24T21:56:53.488 INFO:tasks.rados.rados.0.smithi125.stdout:726: expect (ObjNum 193 snap 42 seq_num 193) 2021-03-24T21:56:53.488 INFO:tasks.rados.rados.0.smithi125.stdout:727: read oid 17 snap -1 2021-03-24T21:56:53.489 INFO:tasks.rados.rados.0.smithi125.stdout:727: expect (ObjNum 123 snap 23 seq_num 123) 2021-03-24T21:56:53.489 INFO:tasks.rados.rados.0.smithi125.stdout:728: snap_create 2021-03-24T21:56:53.489 INFO:tasks.rados.rados.0.smithi125.stdout:723: finishing write tid 2 to smithi12556470-2 2021-03-24T21:56:53.489 INFO:tasks.rados.rados.0.smithi125.stdout:723: finishing write tid 3 to smithi12556470-2 2021-03-24T21:56:53.490 INFO:tasks.rados.rados.0.smithi125.stdout:723: finishing write tid 4 to smithi12556470-2 2021-03-24T21:56:53.490 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 2 v 171 (ObjNum 284 snap 70 seq_num 284) dirty exists 2021-03-24T21:56:53.490 INFO:tasks.rados.rados.0.smithi125.stdout:723: left oid 2 (ObjNum 284 snap 70 seq_num 284) 2021-03-24T21:56:53.490 INFO:tasks.rados.rados.0.smithi125.stdout:725: finishing write tid 1 to smithi12556470-49 2021-03-24T21:56:53.492 INFO:tasks.rados.rados.0.smithi125.stdout:726: expect (ObjNum 193 snap 42 seq_num 193) 2021-03-24T21:56:53.574 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:53.586 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:53 smithi142 conmon[20125]: cluster 2021-03-24T21:56:51.924955+0000 osd.4 (osd.4) 68 : cluster [DBG] 6.17 scrub ok 2021-03-24T21:56:53.587 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:53 smithi142 conmon[20125]: cluster 2021-03-24T21:56:53.587 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:53 smithi142 conmon[20125]: 2021-03-24T21:56:51.977290+0000 osd.7 (osd.7) 59 : cluster [DBG] 5.1a deep-scrub ok 2021-03-24T21:56:53.587 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:53 smithi142 conmon[20125]: cluster 2021-03-24T21:56:52.226108+0000 osd.0 (osd.0) 65 : cluster [DBG] 6.10 scrub ok 2021-03-24T21:56:53.588 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:53 smithi142 conmon[20125]: cluster 2021-03-24T21:56:52.254252 2021-03-24T21:56:53.588 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:53 smithi142 conmon[20125]: +0000 mon.a (mon.0) 864 : cluster [DBG] osdmap e243: 8 total, 8 up, 6 in 2021-03-24T21:56:53.588 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:53 smithi142 conmon[20125]: cluster 2021-03-24T21:56 2021-03-24T21:56:53.588 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:53 smithi142 conmon[20125]: :52.703809+0000 osd.2 (osd.2) 49 : cluster [DBG] 2021-03-24T21:56:53.588 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:53 smithi142 conmon[20125]: purged_snaps scrub starts 2021-03-24T21:56:53.589 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:53 smithi142 conmon[20125]: cluster 2021-03-24T21:56:52.704242+0000 osd.2 2021-03-24T21:56:53.589 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:53 smithi142 conmon[20125]: (osd.2) 50 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:56:53.589 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:53 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:56:53.589 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:53 smithi142 conmon[20125]: -24T21:56:53.254945+0000 mon.a (mon.0 2021-03-24T21:56:53.590 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:53 smithi142 conmon[20125]: ) 865 : cluster [DBG] osdmap e244: 8 total, 8 up, 6 in 2021-03-24T21:56:53.628 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:53.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:53.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:56:53.680 INFO:tasks.rados.rados.0.smithi125.stdout:727: expect (ObjNum 123 snap 23 seq_num 123) 2021-03-24T21:56:53.733 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:53.784 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:53.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:53.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:56:53.796 INFO:tasks.rados.rados.0.smithi125.stdout:725: finishing write tid 2 to smithi12556470-49 2021-03-24T21:56:53.797 INFO:tasks.rados.rados.0.smithi125.stdout:725: finishing write tid 3 to smithi12556470-49 2021-03-24T21:56:53.797 INFO:tasks.rados.rados.0.smithi125.stdout:725: finishing write tid 4 to smithi12556470-49 2021-03-24T21:56:53.797 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 49 v 227 (ObjNum 286 snap 70 seq_num 286) dirty exists 2021-03-24T21:56:53.797 INFO:tasks.rados.rados.0.smithi125.stdout:725: left oid 49 (ObjNum 286 snap 70 seq_num 286) 2021-03-24T21:56:53.890 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:53.953 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:53.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:53.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:56:54.059 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:54.119 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:56:54.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:54.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:56:54.226 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:54.250 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.1 dump_historic_ops 2021-03-24T21:56:54.287 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:54.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:54.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:56:54.295 INFO:tasks.rados.rados.0.smithi125.stdout:723: done (4 left) 2021-03-24T21:56:54.295 INFO:tasks.rados.rados.0.smithi125.stdout:725: done (3 left) 2021-03-24T21:56:54.295 INFO:tasks.rados.rados.0.smithi125.stdout:726: done (2 left) 2021-03-24T21:56:54.295 INFO:tasks.rados.rados.0.smithi125.stdout:727: done (1 left) 2021-03-24T21:56:54.296 INFO:tasks.rados.rados.0.smithi125.stdout:728: done (0 left) 2021-03-24T21:56:54.296 INFO:tasks.rados.rados.0.smithi125.stdout:729: write (excl) oid 30 current snap is 71 2021-03-24T21:56:54.296 INFO:tasks.rados.rados.0.smithi125.stdout:729: seq_num 287 ranges {423609=701524,1628332=472089,2648987=174622} 2021-03-24T21:56:54.297 INFO:teuthology.orchestra.run.smithi125.stderr:noscrub is set 2021-03-24T21:56:54.302 INFO:tasks.rados.rados.0.smithi125.stdout:729: writing smithi12556470-30 from 423609 to 1125133 tid 1 2021-03-24T21:56:54.309 INFO:tasks.rados.rados.0.smithi125.stdout:729: writing smithi12556470-30 from 1628332 to 2100421 tid 2 2021-03-24T21:56:54.314 INFO:tasks.rados.rados.0.smithi125.stdout:729: writing smithi12556470-30 from 2648987 to 2823609 tid 3 2021-03-24T21:56:54.314 INFO:tasks.rados.rados.0.smithi125.stdout:730: write (excl) oid 3 current snap is 71 2021-03-24T21:56:54.314 INFO:tasks.rados.rados.0.smithi125.stdout:730: seq_num 288 ranges {440416=776773,1686814=732100} 2021-03-24T21:56:54.322 INFO:tasks.rados.rados.0.smithi125.stdout:730: writing smithi12556470-3 from 440416 to 1217189 tid 1 2021-03-24T21:56:54.331 INFO:tasks.rados.rados.0.smithi125.stdout:730: writing smithi12556470-3 from 1686814 to 2418914 tid 2 2021-03-24T21:56:54.331 INFO:tasks.rados.rados.0.smithi125.stdout:731: delete oid 10 current snap is 71 2021-03-24T21:56:54.331 INFO:tasks.rados.rados.0.smithi125.stdout:729: finishing write tid 1 to smithi12556470-30 2021-03-24T21:56:54.332 INFO:tasks.rados.rados.0.smithi125.stdout:729: finishing write tid 2 to smithi12556470-30 2021-03-24T21:56:54.332 INFO:tasks.rados.rados.0.smithi125.stdout:729: finishing write tid 3 to smithi12556470-30 2021-03-24T21:56:54.332 INFO:tasks.rados.rados.0.smithi125.stdout:729: finishing write tid 4 to smithi12556470-30 2021-03-24T21:56:54.332 INFO:tasks.rados.rados.0.smithi125.stdout:729: finishing write tid 5 to smithi12556470-30 2021-03-24T21:56:54.333 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 30 v 158 (ObjNum 287 snap 71 seq_num 287) dirty exists 2021-03-24T21:56:54.333 INFO:tasks.rados.rados.0.smithi125.stdout:729: left oid 30 (ObjNum 287 snap 71 seq_num 287) 2021-03-24T21:56:54.333 INFO:tasks.rados.rados.0.smithi125.stdout:730: finishing write tid 1 to smithi12556470-3 2021-03-24T21:56:54.334 INFO:tasks.rados.rados.0.smithi125.stdout:729: done (2 left) 2021-03-24T21:56:54.334 INFO:tasks.rados.rados.0.smithi125.stdout:731: done (1 left) 2021-03-24T21:56:54.335 INFO:tasks.rados.rados.0.smithi125.stdout:732: read oid 17 snap -1 2021-03-24T21:56:54.335 INFO:tasks.rados.rados.0.smithi125.stdout:732: expect (ObjNum 123 snap 23 seq_num 123) 2021-03-24T21:56:54.335 INFO:tasks.rados.rados.0.smithi125.stdout:733: copy_from oid 50 from oid 42 current snap is 71 2021-03-24T21:56:54.335 INFO:tasks.rados.rados.0.smithi125.stdout:734: rollback oid 24 current snap is 71 2021-03-24T21:56:54.335 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 24 to 59 2021-03-24T21:56:54.336 INFO:tasks.rados.rados.0.smithi125.stdout:735: read oid 27 snap -1 2021-03-24T21:56:54.336 INFO:tasks.rados.rados.0.smithi125.stdout:735: expect (ObjNum 193 snap 42 seq_num 193) 2021-03-24T21:56:54.336 INFO:tasks.rados.rados.0.smithi125.stdout:736: snap_create 2021-03-24T21:56:54.337 INFO:tasks.rados.rados.0.smithi125.stdout:733: finishing copy_from to smithi12556470-50 2021-03-24T21:56:54.337 INFO:tasks.rados.rados.0.smithi125.stdout:733: got expected ENOENT (src dne) 2021-03-24T21:56:54.338 INFO:tasks.rados.rados.0.smithi125.stdout:730: finishing write tid 2 to smithi12556470-3 2021-03-24T21:56:54.338 INFO:tasks.rados.rados.0.smithi125.stdout:733: finishing copy_from racing read to smithi12556470-50 2021-03-24T21:56:54.338 INFO:tasks.rados.rados.0.smithi125.stdout:730: finishing write tid 3 to smithi12556470-3 2021-03-24T21:56:54.338 INFO:tasks.rados.rados.0.smithi125.stdout:730: finishing write tid 4 to smithi12556470-3 2021-03-24T21:56:54.339 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 3 v 103 (ObjNum 288 snap 71 seq_num 288) dirty exists 2021-03-24T21:56:54.339 INFO:tasks.rados.rados.0.smithi125.stdout:730: left oid 3 (ObjNum 288 snap 71 seq_num 288) 2021-03-24T21:56:54.341 INFO:tasks.rados.rados.0.smithi125.stdout:732: expect (ObjNum 123 snap 23 seq_num 123) 2021-03-24T21:56:54.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:54 smithi142 conmon[20125]: cluster 2021-03-24T21:56:52.758809+0000 mgr.y (mgr.14144) 387 : cluster [DBG] pgmap v527: 158 pgs: 158 active+clean; 140 MiB data, 491 MiB used, 536 GiB / 536 GiB avail; 10 MiB/s rd, 1.7 MiB/s wr, 13 op/s; 1.2 MiB/s, 7 objects/s recovering 2021-03-24T21:56:54.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:54 smithi142 conmon[20125]: cluster 2021-03-24T21:56:54.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:54 smithi142 conmon[20125]: 2021-03-24T21:56:53.443544+0000 osd.3 (osd.3) 65 : cluster [DBG] 4.10 scrub ok 2021-03-24T21:56:54.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:54 smithi142 conmon[20125]: audit 2021-03-24T21:56:53.545009+0000 mon.c (mon.2) 38 : audit [INF] from='client.? v1:172.21.15.125:0/4063394782' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T21:56:54.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:54 smithi142 conmon[20125]: audit 2021-03-24T21:56:53.545440+0000 mon.a (mon.0) 866 : audit [INF] from='client.34934 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T21:56:54.395 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:54.441 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T21:56:54.447 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:54 smithi125 conmon[19936]: cluster 2021-03-24T21:56:54.447 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:54 smithi125 conmon[19936]: 2021-03-24T21:56:54.447 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:54 smithi125 conmon[19936]: 2021 2021-03-24T21:56:54.448 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:54 smithi125 conmon[19936]: - 2021-03-24T21:56:54.448 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:54 smithi125 conmon[19936]: 03-24T21:56:52 2021-03-24T21:56:54.448 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:54 smithi125 conmon[19936]: .758809+0000 mgr.y (mgr.14144) 387 : cluster [DBG] pgmap v527: 158 pgs: 158 active+clean; 140 MiB data, 491 MiB used, 536 GiB / 536 GiB avail; 10 MiB/s rd, 1.7 MiB/s wr, 13 op/s; 1.2 MiB/s, 7 objects/s recovering 2021-03-24T21:56:54.448 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:54 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:56:54.448 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:54 smithi125 conmon[19936]: -03-24T21:56:53.443544 2021-03-24T21:56:54.449 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:54 smithi125 conmon[19936]: +0000 osd.3 (osd.3) 65 : 2021-03-24T21:56:54.449 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:54 smithi125 conmon[19936]: cluster [DBG] 4.10 scrub ok 2021-03-24T21:56:54.449 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:54 smithi125 conmon[19936]: audit 2021-03- 2021-03-24T21:56:54.449 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:54 smithi125 conmon[19936]: 24T21:56:53.545009+0000 mon.c 2021-03-24T21:56:54.450 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:54 smithi125 conmon[19936]: (mon.2) 38 : audit [INF] 2021-03-24T21:56:54.450 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:54 smithi125 conmon[19936]: from='client.? v1:172.21.15.125:0/4063394782' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T21:56:54.450 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:54 smithi125 conmon[19936]: audit 2021-03-24T21:56: 2021-03-24T21:56:54.450 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:54 smithi125 conmon[19936]: 53.545440+0000 mon.a (mon.0) 866 : 2021-03-24T21:56:54.451 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:54 smithi125 conmon[19936]: audit [INF] from='client.34934 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T21:56:54.451 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:54 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:56:54.451 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:54 smithi125 conmon[24927]: 24T21:56:52.758809+0000 mgr.y (mgr.14144) 387 : cluster [DBG] pgmap v527: 158 pgs: 158 active+clean; 140 MiB data, 491 MiB used, 536 GiB / 536 GiB avail; 10 MiB/s rd, 1.7 MiB/s wr, 13 op/s; 1.2 MiB/s, 7 objects/s recovering 2021-03-24T21:56:54.452 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:54 smithi125 conmon[24927]: 2021-03-24T21:56:54.452 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:54 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:56:54.452 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:54 smithi125 conmon[24927]: :56:53.443544+0000 osd.3 2021-03-24T21:56:54.452 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:54 smithi125 conmon[24927]: (osd.3) 65 : cluster [DBG] 2021-03-24T21:56:54.452 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:54 smithi125 conmon[24927]: 4.10 scrub ok 2021-03-24T21:56:54.453 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:54 smithi125 conmon[24927]: audit 2021- 2021-03-24T21:56:54.453 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:54 smithi125 conmon[24927]: 03-24T21:56: 2021-03-24T21:56:54.453 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:54 smithi125 conmon[24927]: 53.545009+0000 mon.c (mon 2021-03-24T21:56:54.453 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:54 smithi125 conmon[24927]: .2) 38 : audit [INF] 2021-03-24T21:56:54.454 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:54 smithi125 conmon[24927]: from='client.? v1:172.21.15.125:0/4063394782' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T21:56:54.454 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:54 smithi125 conmon[24927]: audit 2021-03-24 2021-03-24T21:56:54.454 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:54 smithi125 conmon[24927]: T21:56:53.545440 2021-03-24T21:56:54.454 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:54 smithi125 conmon[24927]: +0000 mon.a (mon.0) 2021-03-24T21:56:54.454 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:54 smithi125 conmon[24927]: 866 : audit [INF] from='client.34934 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T21:56:54.455 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:54.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:54.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:56:54.465 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T21:56:54.475 INFO:tasks.rados.rados.0.smithi125.stdout:734: finishing rollback tid 0 to smithi12556470-24 2021-03-24T21:56:54.476 INFO:tasks.rados.rados.0.smithi125.stdout:734: finishing rollback tid 1 to smithi12556470-24 2021-03-24T21:56:54.476 INFO:tasks.rados.rados.0.smithi125.stdout:734: finishing rollback tid 2 to smithi12556470-24 2021-03-24T21:56:54.476 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 24 v 241 (ObjNum 123 snap 23 seq_num 123) dirty exists 2021-03-24T21:56:54.476 INFO:tasks.rados.rados.0.smithi125.stdout:735: expect (ObjNum 193 snap 42 seq_num 193) 2021-03-24T21:56:54.494 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T21:56:54.517 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T21:56:54.542 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T21:56:54.561 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:54.575 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T21:56:54.610 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:54.612 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T21:56:54.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:54.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:56:54.635 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T21:56:54.637 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-24T21:56:54.637 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-24T21:56:54.638 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-24T21:56:54.638 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-24T21:56:54.638 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-24T21:56:54.638 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-24T21:56:54.639 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-24T21:56:54.639 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-24T21:56:54.717 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:54.772 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:54.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:54.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:56:54.878 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:54.935 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:54.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:54.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:56:55.041 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:55.102 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:55.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:55.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:56:55.139 INFO:tasks.rados.rados.0.smithi125.stdout:730: done (5 left) 2021-03-24T21:56:55.139 INFO:tasks.rados.rados.0.smithi125.stdout:732: done (4 left) 2021-03-24T21:56:55.139 INFO:tasks.rados.rados.0.smithi125.stdout:733: done (3 left) 2021-03-24T21:56:55.140 INFO:tasks.rados.rados.0.smithi125.stdout:734: done (2 left) 2021-03-24T21:56:55.140 INFO:tasks.rados.rados.0.smithi125.stdout:735: done (1 left) 2021-03-24T21:56:55.141 INFO:tasks.rados.rados.0.smithi125.stdout:736: done (0 left) 2021-03-24T21:56:55.141 INFO:tasks.rados.rados.0.smithi125.stdout:737: copy_from oid 6 from oid 24 current snap is 72 2021-03-24T21:56:55.141 INFO:tasks.rados.rados.0.smithi125.stdout:738: snap_create 2021-03-24T21:56:55.155 INFO:tasks.rados.rados.0.smithi125.stdout:737: finishing copy_from to smithi12556470-6 2021-03-24T21:56:55.155 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 6 v 241 (ObjNum 123 snap 23 seq_num 123) dirty exists 2021-03-24T21:56:55.155 INFO:tasks.rados.rados.0.smithi125.stdout:737: finishing copy_from racing read to smithi12556470-6 2021-03-24T21:56:55.209 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:55.261 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:55.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:55.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:56:55.366 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:55.428 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:55.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:55.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:56:55.532 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:55.548 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[19936]: cluster 2021-03-24T21:56 2021-03-24T21:56:55.548 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[19936]: :54.046183+0000 osd.5 (osd.5) 66 : cluster [DBG] 4.7 scrub ok 2021-03-24T21:56:55.548 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[19936]: cluster 2021-03-24T21:56:55.549 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[19936]: 2021-03-24T21:56:54.262658+0000 osd.0 (osd.0) 66 : 2021-03-24T21:56:55.549 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[19936]: cluster [DBG] 6.a scrub ok 2021-03-24T21:56:55.549 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T21:56:55.549 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[19936]: T21:56:54.274162+0000 mon.a (mon 2021-03-24T21:56:55.549 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[19936]: .0) 867 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:56:55.550 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[19936]: audit 2021-03 2021-03-24T21:56:55.550 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[19936]: -24T21:56:54.291543+0000 mon.a ( 2021-03-24T21:56:55.550 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[19936]: mon.0) 868 : audit [INF] from='client.34934 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T21:56:55.550 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:56:55.551 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[19936]: 03-24T21:56:54.291660+0000 mon.a ( 2021-03-24T21:56:55.551 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[19936]: mon.0) 869 : cluster [DBG] osdmap e245: 8 total, 8 up, 6 in 2021-03-24T21:56:55.551 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:56:55.551 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[19936]: 03-24T21:56:54.414635+0000 mon.a ( 2021-03-24T21:56:55.551 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[19936]: mon.0) 870 : audit [DBG] from='client.? v1:172.21.15.125:0/2637587333' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:56:55.552 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:56:55.552 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[19936]: -03-24T21:56:55.134686+0000 2021-03-24T21:56:55.552 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[19936]: mon.a (mon.0) 871 : cluster [DBG] osdmap e246: 8 total, 8 up, 6 in 2021-03-24T21:56:55.553 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:56:55.553 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[24927]: 21:56:54.046183+0000 osd.5 (osd.5) 66 : cluster [DBG] 4.7 scrub ok 2021-03-24T21:56:55.553 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[24927]: 2021-03-24T21:56:55.553 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:56:55.554 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[24927]: -03-24T21 2021-03-24T21:56:55.554 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[24927]: :56:54.262658+0000 2021-03-24T21:56:55.554 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[24927]: osd.0 (osd.0) 2021-03-24T21:56:55.554 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[24927]: 66 : cluster [DBG] 6.a scrub ok 2021-03-24T21:56:55.555 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[24927]: 2021-03-24T21:56:55.555 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:56:55.555 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[24927]: 21:56:54.274162+0000 2021-03-24T21:56:55.555 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[24927]: mon.a (mon.0) 2021-03-24T21:56:55.555 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[24927]: 867 : cluster [WRN] 2021-03-24T21:56:55.556 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[24927]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:56:55.556 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[24927]: audit 2021- 2021-03-24T21:56:55.556 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[24927]: 03-24T21:56 2021-03-24T21:56:55.556 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[24927]: :54.291543+0000 2021-03-24T21:56:55.556 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[24927]: mon.a (mon.0) 868 : audit [INF] 2021-03-24T21:56:55.557 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[24927]: from='client.34934 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T21:56:55.557 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:56:55.557 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[24927]: -03-24T21: 2021-03-24T21:56:55.557 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[24927]: 56:54.291660+0000 2021-03-24T21:56:55.558 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[24927]: mon.a (mon.0) 2021-03-24T21:56:55.558 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[24927]: 869 : cluster [DBG] osdmap e245: 8 total, 8 up, 6 in 2021-03-24T21:56:55.558 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[24927]: 2021-03-24T21:56:55.558 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[24927]: audit 2021-03-24 2021-03-24T21:56:55.558 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[24927]: T21:56:54.414635+0000 2021-03-24T21:56:55.559 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[24927]: mon.a (mon.0) 870 : audit [DBG] 2021-03-24T21:56:55.559 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[24927]: from='client.? v1:172.21.15.125:0/2637587333' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:56:55.559 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[24927]: cluster 2021-03-24T21:56 2021-03-24T21:56:55.559 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[24927]: :55.134686+0000 mon.a (mon 2021-03-24T21:56:55.559 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[24927]: .0) 871 : cluster [DBG] 2021-03-24T21:56:55.560 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:55 smithi125 conmon[24927]: osdmap e246: 8 total, 8 up, 6 in 2021-03-24T21:56:55.589 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:55.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:55.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:56:55.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:55 smithi142 conmon[20125]: cluster 2021-03-24T21:56:54.046183+0000 osd.5 (osd.5) 66 : cluster 2021-03-24T21:56:55.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:55 smithi142 conmon[20125]: [DBG] 4.7 scrub ok 2021-03-24T21:56:55.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:55 smithi142 conmon[20125]: cluster 2021-03-24T21:56:54.262658+0000 osd.0 (osd.0) 66 2021-03-24T21:56:55.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:55 smithi142 conmon[20125]: : cluster [DBG] 6.a scrub ok 2021-03-24T21:56:55.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:55 smithi142 conmon[20125]: cluster 2021-03-24T21:56:55.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:55 smithi142 conmon[20125]: 2021-03-24T21:56:54.274162+0000 mon.a 2021-03-24T21:56:55.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:55 smithi142 conmon[20125]: (mon.0) 867 : cluster [WRN] 2021-03-24T21:56:55.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:55 smithi142 conmon[20125]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:56:55.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:55 smithi142 conmon[20125]: audit 2021-03-24T21:56:55.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:55 smithi142 conmon[20125]: 2021-03-24T21:56:54.291543+0000 2021-03-24T21:56:55.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:55 smithi142 conmon[20125]: mon.a (mon.0) 868 : audit 2021-03-24T21:56:55.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:55 smithi142 conmon[20125]: [INF] from='client.34934 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T21:56:55.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:55 smithi142 conmon[20125]: cluster 2021-03-24T21:56:55.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:55 smithi142 conmon[20125]: 2021-03-24T21 2021-03-24T21:56:55.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:55 smithi142 conmon[20125]: :56:54.291660+0000 mon.a (mon 2021-03-24T21:56:55.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:55 smithi142 conmon[20125]: .0) 869 : cluster [DBG] osdmap e245: 8 total, 8 up, 6 in 2021-03-24T21:56:55.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:55 smithi142 conmon[20125]: audit 2021 2021-03-24T21:56:55.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:55 smithi142 conmon[20125]: -03-24T21:56:54. 2021-03-24T21:56:55.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:55 smithi142 conmon[20125]: 414635+0000 mon.a (mon.0) 870 : 2021-03-24T21:56:55.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:55 smithi142 conmon[20125]: audit [DBG] from='client.? v1:172.21.15.125:0/2637587333' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:56:55.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:55 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:56:55.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:55 smithi142 conmon[20125]: 21:56:55.134686+0000 mon.a ( 2021-03-24T21:56:55.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:55 smithi142 conmon[20125]: mon.0) 871 : cluster [DBG] osdmap e246: 8 total, 8 up, 6 in 2021-03-24T21:56:55.695 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:55.751 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:55.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:55.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:56:55.828 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:56:55.857 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:55.909 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:55.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:55.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:56:56.015 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:56.076 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:56.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:56.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:56:56.138 INFO:tasks.rados.rados.0.smithi125.stdout:737: done (1 left) 2021-03-24T21:56:56.138 INFO:tasks.rados.rados.0.smithi125.stdout:738: done (0 left) 2021-03-24T21:56:56.138 INFO:tasks.rados.rados.0.smithi125.stdout:739: write (excl) oid 13 current snap is 73 2021-03-24T21:56:56.139 INFO:tasks.rados.rados.0.smithi125.stdout:739: seq_num 290 ranges {474030=643624,1520131=735769} 2021-03-24T21:56:56.147 INFO:tasks.rados.rados.0.smithi125.stdout:739: writing smithi12556470-13 from 474030 to 1117654 tid 1 2021-03-24T21:56:56.156 INFO:tasks.rados.rados.0.smithi125.stdout:739: writing smithi12556470-13 from 1520131 to 2255900 tid 2 2021-03-24T21:56:56.156 INFO:tasks.rados.rados.0.smithi125.stdout:740: rollback oid 12 current snap is 73 2021-03-24T21:56:56.157 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 12 to 66 2021-03-24T21:56:56.157 INFO:tasks.rados.rados.0.smithi125.stdout:739: finishing write tid 1 to smithi12556470-13 2021-03-24T21:56:56.157 INFO:tasks.rados.rados.0.smithi125.stdout:741: delete oid 49 current snap is 73 2021-03-24T21:56:56.163 INFO:tasks.rados.rados.0.smithi125.stdout:739: finishing write tid 2 to smithi12556470-13 2021-03-24T21:56:56.163 INFO:tasks.rados.rados.0.smithi125.stdout:739: finishing write tid 3 to smithi12556470-13 2021-03-24T21:56:56.164 INFO:tasks.rados.rados.0.smithi125.stdout:739: finishing write tid 4 to smithi12556470-13 2021-03-24T21:56:56.164 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 13 v 234 (ObjNum 290 snap 73 seq_num 290) dirty exists 2021-03-24T21:56:56.164 INFO:tasks.rados.rados.0.smithi125.stdout:739: left oid 13 (ObjNum 290 snap 73 seq_num 290) 2021-03-24T21:56:56.165 INFO:tasks.rados.rados.0.smithi125.stdout:739: done (2 left) 2021-03-24T21:56:56.165 INFO:tasks.rados.rados.0.smithi125.stdout:741: done (1 left) 2021-03-24T21:56:56.165 INFO:tasks.rados.rados.0.smithi125.stdout:742: delete oid 38 current snap is 73 2021-03-24T21:56:56.167 INFO:tasks.rados.rados.0.smithi125.stdout:740: finishing rollback tid 0 to smithi12556470-12 2021-03-24T21:56:56.167 INFO:tasks.rados.rados.0.smithi125.stdout:740: finishing rollback tid 1 to smithi12556470-12 2021-03-24T21:56:56.167 INFO:tasks.rados.rados.0.smithi125.stdout:740: finishing rollback tid 2 to smithi12556470-12 2021-03-24T21:56:56.167 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 12 v 244 (ObjNum 161 snap 35 seq_num 161) dirty exists 2021-03-24T21:56:56.168 INFO:tasks.rados.rados.0.smithi125.stdout:740: done (1 left) 2021-03-24T21:56:56.168 INFO:tasks.rados.rados.0.smithi125.stdout:742: done (0 left) 2021-03-24T21:56:56.168 INFO:tasks.rados.rados.0.smithi125.stdout:743: read oid 5 snap -1 2021-03-24T21:56:56.168 INFO:tasks.rados.rados.0.smithi125.stdout:743: expect (ObjNum 203 snap 43 seq_num 203) 2021-03-24T21:56:56.169 INFO:tasks.rados.rados.0.smithi125.stdout:744: read oid 48 snap 70 2021-03-24T21:56:56.169 INFO:tasks.rados.rados.0.smithi125.stdout:744: expect (ObjNum 255 snap 59 seq_num 255) 2021-03-24T21:56:56.169 INFO:tasks.rados.rados.0.smithi125.stdout:745: write oid 41 current snap is 73 2021-03-24T21:56:56.169 INFO:tasks.rados.rados.0.smithi125.stdout:745: seq_num 293 ranges {524451=469371,1579224=483096,2529878=394573} 2021-03-24T21:56:56.172 INFO:tasks.rados.rados.0.smithi125.stdout:745: writing smithi12556470-41 from 524451 to 993822 tid 1 2021-03-24T21:56:56.178 INFO:tasks.rados.rados.0.smithi125.stdout:745: writing smithi12556470-41 from 1579224 to 2062320 tid 2 2021-03-24T21:56:56.183 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:56.184 INFO:tasks.rados.rados.0.smithi125.stdout:745: writing smithi12556470-41 from 2529878 to 2924451 tid 3 2021-03-24T21:56:56.184 INFO:tasks.rados.rados.0.smithi125.stdout:746: snap_remove snap 59 2021-03-24T21:56:56.237 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:56.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:56.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:56:56.338 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:56 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:56:56.338 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:56 smithi125 conmon[19936]: 56:54.759730+0000 mgr.y (mgr.14144) 388 : cluster [DBG] pgmap v530: 158 pgs: 158 active+clean; 151 MiB data, 520 MiB used, 536 GiB / 536 GiB avail; 19 MiB/s rd, 0 B/s wr, 15 op/s 2021-03-24T21:56:56.338 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:56 smithi125 conmon[19936]: cluster 2021-03-24T21:56:55.697861+0000 osd.5 (osd 2021-03-24T21:56:56.338 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:56 smithi125 conmon[19936]: .5) 67 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:56:56.339 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:56 smithi125 conmon[19936]: cluster 2021-03-24T21:56:56.339 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:56 smithi125 conmon[19936]: 2021-03 2021-03-24T21:56:56.339 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:56 smithi125 conmon[19936]: -24T21:56:55.698478+0000 osd.5 2021-03-24T21:56:56.339 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:56 smithi125 conmon[19936]: (osd.5) 68 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:56:56.340 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:56 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:56:56.340 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:56 smithi125 conmon[19936]: -03-24T21:56:56.135061+0000 mon.a (mon 2021-03-24T21:56:56.340 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:56 smithi125 conmon[19936]: .0) 872 : cluster [DBG] osdmap e247: 8 total, 8 up, 6 in 2021-03-24T21:56:56.341 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:56 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:56:56.341 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:56 smithi125 conmon[24927]: T21:56:54.759730+0000 mgr.y (mgr.14144) 388 : cluster [DBG] pgmap v530: 158 pgs: 158 active+clean; 151 MiB data, 520 MiB used, 536 GiB / 536 GiB avail; 19 MiB/s rd, 0 B/s wr, 15 op/s 2021-03-24T21:56:56.341 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:56 smithi125 conmon[24927]: 2021-03-24T21:56:56.341 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:56 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:56:56.341 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:56 smithi125 conmon[24927]: -03-24T21:56: 2021-03-24T21:56:56.342 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:56 smithi125 conmon[24927]: 55.697861+0000 osd.5 (osd.5) 67 2021-03-24T21:56:56.342 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:56 smithi125 conmon[24927]: : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:56:56.342 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:56 smithi125 conmon[24927]: cluster 2021-03-24T21:56:56.342 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:56 smithi125 conmon[24927]: 2021-03-24T21:56 2021-03-24T21:56:56.343 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:56 smithi125 conmon[24927]: :55.698478+0000 osd.5 (osd.5) 68 2021-03-24T21:56:56.343 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:56 smithi125 conmon[24927]: : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:56:56.343 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:56 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:56:56.343 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:56 smithi125 conmon[24927]: 24T21:56:56.135061 2021-03-24T21:56:56.344 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:56 smithi125 conmon[24927]: +0000 mon.a (mon.0) 872 : cluster 2021-03-24T21:56:56.344 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:56 smithi125 conmon[24927]: [DBG] osdmap e247: 8 total, 8 up, 6 in 2021-03-24T21:56:56.344 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:56.397 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:56.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:56.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:56:56.502 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:56.551 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:56.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:56.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:56:56.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:56 smithi142 conmon[20125]: cluster 2021-03-24T21:56:54.759730+0000 mgr.y (mgr.14144) 388 : cluster 2021-03-24T21:56:56.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:56 smithi142 conmon[20125]: [DBG] pgmap v530: 158 pgs: 158 active+clean; 151 MiB data, 520 MiB used, 536 GiB / 536 GiB avail; 19 MiB/s rd, 0 B/s wr, 15 op/s 2021-03-24T21:56:56.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:56 smithi142 conmon[20125]: cluster 2021-03-24T21:56:55.697861+0000 osd.5 (osd.5) 67 : 2021-03-24T21:56:56.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:56 smithi142 conmon[20125]: cluster [DBG] purged_snaps scrub starts 2021-03-24T21:56:56.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:56 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:56:56.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:56 smithi142 conmon[20125]: -24T21:56:55.698478+0000 osd.5 (osd.5 2021-03-24T21:56:56.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:56 smithi142 conmon[20125]: ) 68 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:56:56.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:56 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:56:56.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:56 smithi142 conmon[20125]: -24T21:56:56.135061+0000 mon.a 2021-03-24T21:56:56.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:56 smithi142 conmon[20125]: (mon.0) 872 : cluster [DBG] osdmap e247: 8 total, 8 up, 6 in 2021-03-24T21:56:56.656 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:56.695 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 7] out_osds: [6, 1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-03-24T21:56:56.695 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-24T21:56:56.695 INFO:tasks.thrashosds.thrasher:Reweighting osd 2 to 0.7242660621692911 2021-03-24T21:56:56.696 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd reweight 2 0.7242660621692911 2021-03-24T21:56:56.714 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:56.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:56.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:56:56.821 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:56.873 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:56.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:56.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:56:56.980 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:57.036 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:57.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:57.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:56:57.142 INFO:tasks.rados.rados.0.smithi125.stdout:746: done (3 left) 2021-03-24T21:56:57.142 INFO:tasks.rados.rados.0.smithi125.stdout:747: snap_create 2021-03-24T21:56:57.143 INFO:tasks.rados.rados.0.smithi125.stdout:743: expect (ObjNum 203 snap 43 seq_num 203) 2021-03-24T21:56:57.143 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:57.196 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:57.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:57.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:56:57.301 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:57.309 INFO:tasks.rados.rados.0.smithi125.stdout:744: expect (ObjNum 255 snap 59 seq_num 255) 2021-03-24T21:56:57.316 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 24 21:56:57 smithi142 conmon[33921]: debug 2021-03-24T21:56:57.155+0000 7fc7b2416700 -1 osd.7 pg_epoch: 248 pg[7.17( v 245'241 (0'0,245'241] local-lis/les=203/204 n=8 ec=198/99 lis/c=203/203 les/c/f=204/204/0 sis=203) [7,5,2] r=0 lpr=203 crt=245'241 lcod 245'240 mlcod 245'240 active+clean+snaptrim trimq=[78~1] ps=[80~1,83~1,87~2]] removing snap head 2021-03-24T21:56:57.334 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:56:57.357 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:57.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:57.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:56:57.440 INFO:tasks.rados.rados.0.smithi125.stdout:745: finishing write tid 1 to smithi12556470-41 2021-03-24T21:56:57.553 INFO:tasks.rados.rados.0.smithi125.stdout:745: finishing write tid 2 to smithi12556470-41 2021-03-24T21:56:57.554 INFO:tasks.rados.rados.0.smithi125.stdout:745: finishing write tid 3 to smithi12556470-41 2021-03-24T21:56:57.554 INFO:tasks.rados.rados.0.smithi125.stdout:745: finishing write tid 4 to smithi12556470-41 2021-03-24T21:56:57.554 INFO:tasks.rados.rados.0.smithi125.stdout:745: finishing write tid 5 to smithi12556470-41 2021-03-24T21:56:57.554 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 41 v 219 (ObjNum 293 snap 73 seq_num 293) dirty exists 2021-03-24T21:56:57.555 INFO:tasks.rados.rados.0.smithi125.stdout:745: left oid 41 (ObjNum 293 snap 73 seq_num 293) 2021-03-24T21:56:57.555 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:57.618 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:57.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:57.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:56:57.725 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:57.780 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:57.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:57.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:56:57.801 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-24T21:56:57.814 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-24T21:56:57.815 INFO:teuthology.orchestra.run.smithi125.stdout: "size": 20, 2021-03-24T21:56:57.815 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 600, 2021-03-24T21:56:57.815 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [ 2021-03-24T21:56:57.815 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.816 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.34352.0:535 4.b e65/62 4:d1061236:::data_log.86:head v 65'3, mlcod=65'1)", 2021-03-24T21:56:57.816 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:51:50.034175+0000", 2021-03-24T21:56:57.816 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 307.77060342499999, 2021-03-24T21:56:57.816 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.017611492999999999, 2021-03-24T21:56:57.817 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:56:57.817 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:57.817 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:56:57.817 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.817 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:56:57.818 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.034175+0000", 2021-03-24T21:56:57.818 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:56:57.818 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.819 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.819 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:56:57.819 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.034175+0000", 2021-03-24T21:56:57.819 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T21:56:57.819 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.820 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.820 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:56:57.820 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.034173+0000", 2021-03-24T21:56:57.821 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3985000000000001e-05 2021-03-24T21:56:57.821 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.821 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.821 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:56:57.821 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.034187+0000", 2021-03-24T21:56:57.822 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7768e-05 2021-03-24T21:56:57.822 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.822 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.822 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:56:57.823 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.034205+0000", 2021-03-24T21:56:57.823 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4919999999999998e-06 2021-03-24T21:56:57.823 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.823 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.823 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:56:57.824 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.034211+0000", 2021-03-24T21:56:57.824 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5782000000000002e-05 2021-03-24T21:56:57.824 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.824 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.824 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:57.825 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.034267+0000", 2021-03-24T21:56:57.825 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.4258000000000003e-05 2021-03-24T21:56:57.825 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.825 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.826 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:56:57.826 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.034301+0000", 2021-03-24T21:56:57.826 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.017442103 2021-03-24T21:56:57.826 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.826 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.827 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:56:57.827 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.051743+0000", 2021-03-24T21:56:57.827 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3061e-05 2021-03-24T21:56:57.827 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.827 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.828 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:56:57.828 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.051786+0000", 2021-03-24T21:56:57.828 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.017611492999999999 2021-03-24T21:56:57.829 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:57.829 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:56:57.829 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:57.829 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.829 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.830 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24376.0:221 4.2 4:479ba34f:::obj_delete_at_hint.0000000021:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e69)", 2021-03-24T21:56:57.830 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:51:53.942625+0000", 2021-03-24T21:56:57.830 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 303.86215378399999, 2021-03-24T21:56:57.830 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0044149480000000001, 2021-03-24T21:56:57.831 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:56:57.831 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:57.831 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:56:57.831 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24376", 2021-03-24T21:56:57.832 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/3051439499", 2021-03-24T21:56:57.832 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 221 2021-03-24T21:56:57.832 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.832 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:56:57.832 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.833 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:56:57.833 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.942625+0000", 2021-03-24T21:56:57.833 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:56:57.833 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.833 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.834 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:56:57.834 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.942625+0000", 2021-03-24T21:56:57.834 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T21:56:57.834 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.835 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.835 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:56:57.835 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.942623+0000", 2021-03-24T21:56:57.835 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.4860000000000003e-06 2021-03-24T21:56:57.835 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.836 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.836 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:56:57.836 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.942632+0000", 2021-03-24T21:56:57.837 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7484000000000001e-05 2021-03-24T21:56:57.837 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.837 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.837 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:56:57.838 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.942650+0000", 2021-03-24T21:56:57.838 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.131e-06 2021-03-24T21:56:57.838 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.838 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.839 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:56:57.839 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.942657+0000", 2021-03-24T21:56:57.839 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.9171999999999993e-05 2021-03-24T21:56:57.839 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.839 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.840 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:57.840 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.942736+0000", 2021-03-24T21:56:57.840 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000181931 2021-03-24T21:56:57.840 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.841 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.841 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:56:57.841 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.942918+0000", 2021-03-24T21:56:57.841 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00019801000000000001 2021-03-24T21:56:57.842 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.842 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.842 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 4,5", 2021-03-24T21:56:57.843 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.943116+0000", 2021-03-24T21:56:57.843 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00048897899999999995 2021-03-24T21:56:57.843 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.843 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.844 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:56:57.844 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.943605+0000", 2021-03-24T21:56:57.844 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000288492 2021-03-24T21:56:57.844 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.845 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.845 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:57.845 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.943893+0000", 2021-03-24T21:56:57.845 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0030618160000000002 2021-03-24T21:56:57.846 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.846 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.846 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:57.846 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.946955+0000", 2021-03-24T21:56:57.847 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.7068999999999999e-05 2021-03-24T21:56:57.847 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.847 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.847 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:56:57.847 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.947002+0000", 2021-03-24T21:56:57.848 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.7311999999999998e-05 2021-03-24T21:56:57.848 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.848 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.849 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:56:57.849 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.947040+0000", 2021-03-24T21:56:57.849 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0044149480000000001 2021-03-24T21:56:57.849 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:57.850 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:56:57.850 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:57.851 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.851 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.851 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:33 7.b 7:d821c111:::smithi12556470-9:head [write 534456~718345,stat] snapc 0=[] ondisk+write+known_if_redirected e107)", 2021-03-24T21:56:57.852 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.193321+0000", 2021-03-24T21:56:57.852 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 141.61145716499999, 2021-03-24T21:56:57.853 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0060392889999999998, 2021-03-24T21:56:57.853 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:56:57.853 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:57.853 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:56:57.854 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:56:57.854 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:56:57.855 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 33 2021-03-24T21:56:57.855 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.855 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:56:57.856 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.856 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:56:57.857 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.193321+0000", 2021-03-24T21:56:57.857 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:56:57.858 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.858 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.859 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:56:57.859 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.193321+0000", 2021-03-24T21:56:57.859 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999976 2021-03-24T21:56:57.860 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.860 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.861 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:56:57.861 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.193319+0000", 2021-03-24T21:56:57.861 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00038596399999999998 2021-03-24T21:56:57.861 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.862 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.862 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:56:57.863 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.193705+0000", 2021-03-24T21:56:57.863 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.8202e-05 2021-03-24T21:56:57.863 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.864 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.864 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:56:57.864 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.193723+0000", 2021-03-24T21:56:57.864 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.6779999999999995e-06 2021-03-24T21:56:57.865 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.865 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.865 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:56:57.865 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.193733+0000", 2021-03-24T21:56:57.866 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.7935000000000005e-05 2021-03-24T21:56:57.866 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.866 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.866 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:57.867 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.193810+0000", 2021-03-24T21:56:57.867 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00016634899999999999 2021-03-24T21:56:57.867 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.867 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.868 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:56:57.868 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.193977+0000", 2021-03-24T21:56:57.868 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00021176300000000001 2021-03-24T21:56:57.868 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.869 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.869 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,5", 2021-03-24T21:56:57.869 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.194189+0000", 2021-03-24T21:56:57.869 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0020706510000000002 2021-03-24T21:56:57.869 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.870 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.870 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:57.870 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.196259+0000", 2021-03-24T21:56:57.870 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.4345000000000002e-05 2021-03-24T21:56:57.871 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.871 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.871 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:56:57.872 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.196284+0000", 2021-03-24T21:56:57.872 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0028126039999999998 2021-03-24T21:56:57.872 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.873 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.873 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:57.874 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.199096+0000", 2021-03-24T21:56:57.874 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.7195999999999997e-05 2021-03-24T21:56:57.874 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.875 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.875 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:56:57.875 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.199173+0000", 2021-03-24T21:56:57.875 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00018708400000000001 2021-03-24T21:56:57.875 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.876 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.876 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:56:57.876 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.199360+0000", 2021-03-24T21:56:57.877 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0060392889999999998 2021-03-24T21:56:57.877 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:57.877 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:56:57.877 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:57.877 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.878 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.878 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:34 7.b 7:d821c111:::smithi12556470-9:head [write 1951503~182953] snapc 0=[] ondisk+write+known_if_redirected e107)", 2021-03-24T21:56:57.878 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.197792+0000", 2021-03-24T21:56:57.878 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 141.606986092, 2021-03-24T21:56:57.879 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0040527949999999997, 2021-03-24T21:56:57.879 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:56:57.879 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:57.879 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:56:57.879 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:56:57.880 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:56:57.880 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 34 2021-03-24T21:56:57.880 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.880 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:56:57.881 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.881 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:56:57.881 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.197792+0000", 2021-03-24T21:56:57.882 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:56:57.882 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.882 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.882 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:56:57.883 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.197792+0000", 2021-03-24T21:56:57.883 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999976 2021-03-24T21:56:57.883 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.883 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.883 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:56:57.884 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.197790+0000", 2021-03-24T21:56:57.884 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.4402999999999997e-05 2021-03-24T21:56:57.884 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.884 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.884 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:56:57.885 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.197884+0000", 2021-03-24T21:56:57.885 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9775999999999998e-05 2021-03-24T21:56:57.885 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.885 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.886 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:56:57.886 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.197904+0000", 2021-03-24T21:56:57.886 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.4100000000000008e-06 2021-03-24T21:56:57.886 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.887 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.887 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:56:57.887 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.197912+0000", 2021-03-24T21:56:57.888 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.2970000000000001e-05 2021-03-24T21:56:57.888 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.888 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.888 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:57.889 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.197985+0000", 2021-03-24T21:56:57.889 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000144194 2021-03-24T21:56:57.889 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.889 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.889 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T21:56:57.890 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.198130+0000", 2021-03-24T21:56:57.890 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0013126240000000001 2021-03-24T21:56:57.890 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.890 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.891 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:57.891 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.199442+0000", 2021-03-24T21:56:57.891 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.276e-05 2021-03-24T21:56:57.891 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.892 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.892 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:56:57.892 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.199535+0000", 2021-03-24T21:56:57.892 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00021443300000000001 2021-03-24T21:56:57.892 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.893 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.893 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,5", 2021-03-24T21:56:57.893 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.199749+0000", 2021-03-24T21:56:57.893 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001662048 2021-03-24T21:56:57.894 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.894 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.894 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:57.894 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.201411+0000", 2021-03-24T21:56:57.894 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.5590000000000006e-06 2021-03-24T21:56:57.895 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.895 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.895 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:56:57.895 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.201420+0000", 2021-03-24T21:56:57.895 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000318401 2021-03-24T21:56:57.896 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.896 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.896 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:57.896 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.201738+0000", 2021-03-24T21:56:57.897 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.8393999999999997e-05 2021-03-24T21:56:57.897 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.897 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.897 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:56:57.897 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.201797+0000", 2021-03-24T21:56:57.898 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.8189999999999998e-05 2021-03-24T21:56:57.898 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.898 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.898 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:56:57.899 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.201845+0000", 2021-03-24T21:56:57.899 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0040527949999999997 2021-03-24T21:56:57.899 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:57.899 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:56:57.899 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:57.900 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.900 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.900 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:35 7.b 7:d821c111:::smithi12556470-9:head [setxattr _header (55) in=7b,truncate 2134456] snapc 0=[] ondisk+write+known_if_redirected e107)", 2021-03-24T21:56:57.900 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.197928+0000", 2021-03-24T21:56:57.901 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 141.606850713, 2021-03-24T21:56:57.901 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0041834120000000001, 2021-03-24T21:56:57.901 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:56:57.901 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:57.902 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:56:57.902 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:56:57.902 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:56:57.903 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 35 2021-03-24T21:56:57.903 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.903 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:56:57.903 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.904 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:56:57.904 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.197928+0000", 2021-03-24T21:56:57.904 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:56:57.904 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.905 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.905 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:56:57.905 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.197928+0000", 2021-03-24T21:56:57.905 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T21:56:57.905 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.906 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.906 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:56:57.906 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.197926+0000", 2021-03-24T21:56:57.907 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.653e-06 2021-03-24T21:56:57.907 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.907 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.907 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:56:57.907 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.197935+0000", 2021-03-24T21:56:57.908 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1636e-05 2021-03-24T21:56:57.908 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.908 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.908 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:56:57.909 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.197946+0000", 2021-03-24T21:56:57.909 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.9549999999999998e-06 2021-03-24T21:56:57.909 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.909 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.909 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:56:57.910 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.197955+0000", 2021-03-24T21:56:57.910 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00023729900000000001 2021-03-24T21:56:57.910 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.910 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.911 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:57.911 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.198193+0000", 2021-03-24T21:56:57.911 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000111729 2021-03-24T21:56:57.911 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.911 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.912 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T21:56:57.912 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.198304+0000", 2021-03-24T21:56:57.912 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0021900470000000001 2021-03-24T21:56:57.912 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.913 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.913 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:57.913 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.200494+0000", 2021-03-24T21:56:57.913 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00020013600000000001 2021-03-24T21:56:57.914 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.914 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.914 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:56:57.914 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.200694+0000", 2021-03-24T21:56:57.915 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00025853600000000002 2021-03-24T21:56:57.915 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.915 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.915 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,5", 2021-03-24T21:56:57.915 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.200953+0000", 2021-03-24T21:56:57.916 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00048642299999999999 2021-03-24T21:56:57.916 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.916 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.916 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:56:57.917 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.201439+0000", 2021-03-24T21:56:57.917 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00047360399999999997 2021-03-24T21:56:57.917 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.917 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.917 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:57.918 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.201913+0000", 2021-03-24T21:56:57.918 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.1973999999999998e-05 2021-03-24T21:56:57.918 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.919 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.919 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:57.919 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.201995+0000", 2021-03-24T21:56:57.920 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.7000999999999999e-05 2021-03-24T21:56:57.920 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.920 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.920 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:56:57.921 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.202052+0000", 2021-03-24T21:56:57.921 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9033999999999998e-05 2021-03-24T21:56:57.921 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.922 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.922 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:56:57.922 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.202111+0000", 2021-03-24T21:56:57.922 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0041834120000000001 2021-03-24T21:56:57.922 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:57.923 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:56:57.923 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:57.923 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.923 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.924 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:36 7.b 7:d821c111:::smithi12556470-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e107)", 2021-03-24T21:56:57.924 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.197999+0000", 2021-03-24T21:56:57.924 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 141.60677965799999, 2021-03-24T21:56:57.924 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0044019999999999997, 2021-03-24T21:56:57.924 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:56:57.925 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T21:56:57.925 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:56:57.925 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:56:57.925 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:56:57.926 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 36 2021-03-24T21:56:57.926 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.926 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:56:57.926 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.927 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:56:57.927 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.197999+0000", 2021-03-24T21:56:57.927 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:56:57.927 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.927 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.928 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:56:57.928 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.197999+0000", 2021-03-24T21:56:57.928 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T21:56:57.929 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.929 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.930 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:56:57.930 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.197997+0000", 2021-03-24T21:56:57.930 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.576e-06 2021-03-24T21:56:57.931 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.931 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.932 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:56:57.932 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.198005+0000", 2021-03-24T21:56:57.933 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2581e-05 2021-03-24T21:56:57.933 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.933 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.934 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:56:57.934 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.198017+0000", 2021-03-24T21:56:57.934 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.6400000000000003e-06 2021-03-24T21:56:57.934 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.935 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.935 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:56:57.935 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.198026+0000", 2021-03-24T21:56:57.936 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00031787599999999998 2021-03-24T21:56:57.936 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.936 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.937 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:57.937 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.198344+0000", 2021-03-24T21:56:57.937 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.3399999999999993e-05 2021-03-24T21:56:57.938 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.938 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.938 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T21:56:57.938 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.198437+0000", 2021-03-24T21:56:57.939 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0028437660000000002 2021-03-24T21:56:57.939 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.939 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.940 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:57.940 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.201281+0000", 2021-03-24T21:56:57.940 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2627999999999996e-05 2021-03-24T21:56:57.941 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.941 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.942 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T21:56:57.942 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.201364+0000", 2021-03-24T21:56:57.942 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000821461 2021-03-24T21:56:57.943 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.943 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.944 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:57.944 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.202185+0000", 2021-03-24T21:56:57.944 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010005699999999999 2021-03-24T21:56:57.945 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.945 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.945 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:56:57.945 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.202285+0000", 2021-03-24T21:56:57.945 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000115427 2021-03-24T21:56:57.946 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.946 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.946 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:56:57.947 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.202401+0000", 2021-03-24T21:56:57.947 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0044019999999999997 2021-03-24T21:56:57.947 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:57.947 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:56:57.947 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:57.948 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.948 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.948 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:53 7.b 7:da3cc97e:::smithi12556470-14:head [write 618491~694590,stat] snapc 0=[] ondisk+write+known_if_redirected e107)", 2021-03-24T21:56:57.948 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.255785+0000", 2021-03-24T21:56:57.949 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 141.548993317, 2021-03-24T21:56:57.949 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0052466329999999997, 2021-03-24T21:56:57.949 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:56:57.949 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:57.950 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:56:57.950 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:56:57.950 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:56:57.950 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 53 2021-03-24T21:56:57.951 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.951 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:56:57.951 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.951 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:56:57.952 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.255785+0000", 2021-03-24T21:56:57.952 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:56:57.952 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.952 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.953 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:56:57.953 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.255785+0000", 2021-03-24T21:56:57.953 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999967 2021-03-24T21:56:57.953 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.953 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.954 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:56:57.954 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.255781+0000", 2021-03-24T21:56:57.954 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00027362600000000001 2021-03-24T21:56:57.954 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.954 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.955 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:56:57.955 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.256055+0000", 2021-03-24T21:56:57.955 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.9417e-05 2021-03-24T21:56:57.955 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.956 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.956 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:56:57.956 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.256084+0000", 2021-03-24T21:56:57.956 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1114000000000001e-05 2021-03-24T21:56:57.956 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.957 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.957 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:56:57.957 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.256096+0000", 2021-03-24T21:56:57.957 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00014862799999999999 2021-03-24T21:56:57.958 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.958 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.958 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:57.958 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.256244+0000", 2021-03-24T21:56:57.958 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000355247 2021-03-24T21:56:57.959 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.959 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.959 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:56:57.959 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.256599+0000", 2021-03-24T21:56:57.960 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00049522699999999999 2021-03-24T21:56:57.960 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.960 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.960 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,5", 2021-03-24T21:56:57.961 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.257095+0000", 2021-03-24T21:56:57.961 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0015160180000000001 2021-03-24T21:56:57.961 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.961 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.961 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:56:57.962 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.258611+0000", 2021-03-24T21:56:57.962 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0010272020000000001 2021-03-24T21:56:57.962 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.962 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.963 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:57.963 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.259638+0000", 2021-03-24T21:56:57.963 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001131791 2021-03-24T21:56:57.963 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.964 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.964 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:57.964 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.260770+0000", 2021-03-24T21:56:57.964 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012263000000000001 2021-03-24T21:56:57.964 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.965 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.965 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:56:57.965 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.260892+0000", 2021-03-24T21:56:57.965 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000139309 2021-03-24T21:56:57.966 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.966 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.966 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:56:57.966 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.261032+0000", 2021-03-24T21:56:57.967 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0052466329999999997 2021-03-24T21:56:57.967 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:57.967 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:56:57.967 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:57.968 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.968 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.969 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:62 7.7 e107/101 7:e7492b26:::smithi12556470-16:head v 107'5, mlcod=107'3)", 2021-03-24T21:56:57.969 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.287402+0000", 2021-03-24T21:56:57.969 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 141.517375899, 2021-03-24T21:56:57.969 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0054094420000000004, 2021-03-24T21:56:57.970 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:56:57.970 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:57.970 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:56:57.970 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.971 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:56:57.971 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.287402+0000", 2021-03-24T21:56:57.971 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:56:57.972 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.972 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.973 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:56:57.973 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.287402+0000", 2021-03-24T21:56:57.973 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999976 2021-03-24T21:56:57.973 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.974 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.974 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:56:57.974 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.287400+0000", 2021-03-24T21:56:57.975 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000213122 2021-03-24T21:56:57.975 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.975 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.975 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:56:57.976 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.287613+0000", 2021-03-24T21:56:57.976 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.8102e-05 2021-03-24T21:56:57.976 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.976 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.976 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:56:57.977 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.287641+0000", 2021-03-24T21:56:57.977 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.8050000000000001e-06 2021-03-24T21:56:57.977 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.977 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.978 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:56:57.978 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.287651+0000", 2021-03-24T21:56:57.978 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9985000000000003e-05 2021-03-24T21:56:57.978 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.979 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.979 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:57.979 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.287711+0000", 2021-03-24T21:56:57.979 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.1588999999999999e-05 2021-03-24T21:56:57.979 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.980 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.980 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:56:57.980 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.287743+0000", 2021-03-24T21:56:57.980 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0050277150000000003 2021-03-24T21:56:57.981 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.981 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.981 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:56:57.981 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.292770+0000", 2021-03-24T21:56:57.982 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.1461000000000002e-05 2021-03-24T21:56:57.982 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.982 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.983 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:56:57.983 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.292812+0000", 2021-03-24T21:56:57.984 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0054094420000000004 2021-03-24T21:56:57.984 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:57.984 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:56:57.985 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:57.985 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.985 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.985 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:63 7.7 e107/101 7:e7492b26:::smithi12556470-16:head v 107'6, mlcod=107'3)", 2021-03-24T21:56:57.985 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.288459+0000", 2021-03-24T21:56:57.986 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 141.51631912799999, 2021-03-24T21:56:57.986 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0043744769999999999, 2021-03-24T21:56:57.986 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:56:57.986 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:57.987 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:56:57.987 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.987 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:56:57.988 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.288459+0000", 2021-03-24T21:56:57.988 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:56:57.988 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.989 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.989 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:56:57.989 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.288459+0000", 2021-03-24T21:56:57.990 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T21:56:57.990 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.991 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.991 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:56:57.991 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.288458+0000", 2021-03-24T21:56:57.991 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.2450000000000004e-06 2021-03-24T21:56:57.992 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.992 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.992 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:56:57.992 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.288467+0000", 2021-03-24T21:56:57.992 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5017e-05 2021-03-24T21:56:57.993 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.993 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.993 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:56:57.993 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.288482+0000", 2021-03-24T21:56:57.994 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.5370000000000003e-06 2021-03-24T21:56:57.994 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.994 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.994 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:56:57.994 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.288488+0000", 2021-03-24T21:56:57.995 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6026999999999998e-05 2021-03-24T21:56:57.995 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.995 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.995 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:57.996 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.288534+0000", 2021-03-24T21:56:57.996 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9232000000000002e-05 2021-03-24T21:56:57.996 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.996 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.996 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:56:57.997 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.288554+0000", 2021-03-24T21:56:57.997 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0042627170000000001 2021-03-24T21:56:57.997 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.997 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.997 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:56:57.998 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.292816+0000", 2021-03-24T21:56:57.998 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7295e-05 2021-03-24T21:56:57.998 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:57.998 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:57.999 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:56:57.999 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.292834+0000", 2021-03-24T21:56:57.999 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0043744769999999999 2021-03-24T21:56:57.999 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:57.999 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:56:58.000 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:58.000 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.000 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.000 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:82 7.d 7:b2218182:::smithi12556470-21:head [write 2020581~315559] snapc 0=[] ondisk+write+known_if_redirected e107)", 2021-03-24T21:56:58.001 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.359676+0000", 2021-03-24T21:56:58.001 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 141.44510185600001, 2021-03-24T21:56:58.001 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0053478029999999999, 2021-03-24T21:56:58.001 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:56:58.001 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:58.002 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:56:58.002 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:56:58.002 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:56:58.002 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 82 2021-03-24T21:56:58.003 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.003 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:56:58.003 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.003 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:56:58.003 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359676+0000", 2021-03-24T21:56:58.004 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:56:58.004 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.004 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.004 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:56:58.005 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359676+0000", 2021-03-24T21:56:58.005 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T21:56:58.005 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.005 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.005 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:56:58.006 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359674+0000", 2021-03-24T21:56:58.006 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000130075 2021-03-24T21:56:58.006 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.006 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.006 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:56:58.007 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359804+0000", 2021-03-24T21:56:58.007 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.2365999999999999e-05 2021-03-24T21:56:58.007 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.007 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.007 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:56:58.008 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359827+0000", 2021-03-24T21:56:58.008 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.6319999999999997e-06 2021-03-24T21:56:58.008 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.008 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.009 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:56:58.009 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359835+0000", 2021-03-24T21:56:58.009 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.2173000000000002e-05 2021-03-24T21:56:58.009 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.009 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.010 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:58.010 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359908+0000", 2021-03-24T21:56:58.010 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00011309199999999999 2021-03-24T21:56:58.010 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.010 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.011 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:56:58.011 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.360021+0000", 2021-03-24T21:56:58.011 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00020120199999999999 2021-03-24T21:56:58.011 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.012 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.012 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,3", 2021-03-24T21:56:58.012 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.360222+0000", 2021-03-24T21:56:58.012 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0013165919999999999 2021-03-24T21:56:58.013 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.013 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.013 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:56:58.013 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.361538+0000", 2021-03-24T21:56:58.013 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00059871600000000001 2021-03-24T21:56:58.014 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.014 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.014 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:58.014 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.362137+0000", 2021-03-24T21:56:58.015 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0027486580000000002 2021-03-24T21:56:58.015 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.015 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.015 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:58.015 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.364886+0000", 2021-03-24T21:56:58.016 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.852e-05 2021-03-24T21:56:58.016 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.016 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.016 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:56:58.016 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.364964+0000", 2021-03-24T21:56:58.017 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9901000000000001e-05 2021-03-24T21:56:58.017 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.017 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.017 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:56:58.018 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.365024+0000", 2021-03-24T21:56:58.018 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0053478029999999999 2021-03-24T21:56:58.018 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:58.018 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:56:58.018 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:58.019 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.019 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.019 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:83 7.d 7:b2218182:::smithi12556470-21:head [setxattr _header (56) in=7b,truncate 2336140] snapc 0=[] ondisk+write+known_if_redirected e107)", 2021-03-24T21:56:58.019 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.359856+0000", 2021-03-24T21:56:58.020 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 141.44492213999999, 2021-03-24T21:56:58.020 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0053737860000000002, 2021-03-24T21:56:58.020 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:56:58.020 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:58.021 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:56:58.021 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:56:58.021 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:56:58.021 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 83 2021-03-24T21:56:58.021 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.022 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:56:58.022 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.022 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:56:58.022 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359856+0000", 2021-03-24T21:56:58.022 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:56:58.023 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.023 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.023 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:56:58.023 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359856+0000", 2021-03-24T21:56:58.024 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T21:56:58.024 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.024 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.024 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:56:58.024 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359854+0000", 2021-03-24T21:56:58.025 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.8180000000000004e-06 2021-03-24T21:56:58.025 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.025 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.025 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:56:58.026 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359862+0000", 2021-03-24T21:56:58.026 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.0979999999999996e-06 2021-03-24T21:56:58.026 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.026 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.026 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:56:58.027 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359869+0000", 2021-03-24T21:56:58.027 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2680000000000002e-06 2021-03-24T21:56:58.027 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.027 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.028 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:56:58.028 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359875+0000", 2021-03-24T21:56:58.028 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00094160000000000001 2021-03-24T21:56:58.028 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.028 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.029 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:58.029 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.360816+0000", 2021-03-24T21:56:58.029 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.3867000000000002e-05 2021-03-24T21:56:58.029 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.029 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.030 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:56:58.030 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.360900+0000", 2021-03-24T21:56:58.030 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00015279599999999999 2021-03-24T21:56:58.030 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.031 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.031 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,3", 2021-03-24T21:56:58.031 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.361053+0000", 2021-03-24T21:56:58.031 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00049272900000000002 2021-03-24T21:56:58.031 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.032 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.032 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:56:58.032 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.361546+0000", 2021-03-24T21:56:58.032 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00077420799999999997 2021-03-24T21:56:58.032 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.033 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.033 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:58.033 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.362320+0000", 2021-03-24T21:56:58.033 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0027734880000000002 2021-03-24T21:56:58.034 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.034 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.034 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:58.034 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.365093+0000", 2021-03-24T21:56:58.035 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.1638999999999997e-05 2021-03-24T21:56:58.035 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.035 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.035 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:56:58.035 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.365155+0000", 2021-03-24T21:56:58.036 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.4994999999999996e-05 2021-03-24T21:56:58.036 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.036 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.036 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:56:58.037 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.365230+0000", 2021-03-24T21:56:58.037 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0053737860000000002 2021-03-24T21:56:58.037 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:58.037 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:56:58.037 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:58.038 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.038 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.038 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:84 7.d 7:b2218182:::smithi12556470-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e107)", 2021-03-24T21:56:58.038 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.359915+0000", 2021-03-24T21:56:58.039 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 141.444863583, 2021-03-24T21:56:58.039 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0056067570000000004, 2021-03-24T21:56:58.039 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:56:58.039 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T21:56:58.039 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:56:58.040 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:56:58.040 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:56:58.040 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 84 2021-03-24T21:56:58.040 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.040 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:56:58.041 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.041 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:56:58.041 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359915+0000", 2021-03-24T21:56:58.041 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:56:58.041 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.042 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.042 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:56:58.042 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359915+0000", 2021-03-24T21:56:58.042 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T21:56:58.043 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.043 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.043 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:56:58.043 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359914+0000", 2021-03-24T21:56:58.043 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.8439999999999997e-06 2021-03-24T21:56:58.044 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.044 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.044 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:56:58.044 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359918+0000", 2021-03-24T21:56:58.044 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0258000000000001e-05 2021-03-24T21:56:58.045 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.045 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.045 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:56:58.045 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359929+0000", 2021-03-24T21:56:58.046 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.8179999999999998e-06 2021-03-24T21:56:58.046 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.046 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.046 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:56:58.046 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359935+0000", 2021-03-24T21:56:58.047 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0013444279999999999 2021-03-24T21:56:58.047 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.047 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.047 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:58.047 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.361280+0000", 2021-03-24T21:56:58.048 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.7156999999999993e-05 2021-03-24T21:56:58.048 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.048 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.048 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T21:56:58.049 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.361357+0000", 2021-03-24T21:56:58.049 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0039437489999999999 2021-03-24T21:56:58.049 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.049 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.049 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:58.050 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.365301+0000", 2021-03-24T21:56:58.050 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000100756 2021-03-24T21:56:58.050 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.050 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.050 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:56:58.051 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.365402+0000", 2021-03-24T21:56:58.051 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00011997099999999999 2021-03-24T21:56:58.051 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.051 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.051 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:56:58.052 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.365522+0000", 2021-03-24T21:56:58.052 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0056067570000000004 2021-03-24T21:56:58.052 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:58.052 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:56:58.053 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:58.053 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.053 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.053 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:121 7.b 7:d841eed8:::smithi12556470-31:head [write 504210~606529,stat] snapc 0=[] ondisk+write+known_if_redirected e107)", 2021-03-24T21:56:58.053 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.487723+0000", 2021-03-24T21:56:58.054 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 141.317055745, 2021-03-24T21:56:58.054 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0043501010000000003, 2021-03-24T21:56:58.054 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:56:58.054 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:58.055 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:56:58.055 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:56:58.055 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:56:58.055 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 121 2021-03-24T21:56:58.055 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.056 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:56:58.056 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.056 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:56:58.056 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.487723+0000", 2021-03-24T21:56:58.056 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:56:58.057 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.057 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.057 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:56:58.057 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.487723+0000", 2021-03-24T21:56:58.058 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999962 2021-03-24T21:56:58.058 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.058 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.058 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:56:58.058 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.487719+0000", 2021-03-24T21:56:58.059 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00036455999999999999 2021-03-24T21:56:58.059 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.059 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.059 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:56:58.059 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.488084+0000", 2021-03-24T21:56:58.060 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.7018000000000001e-05 2021-03-24T21:56:58.060 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.060 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.060 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:56:58.061 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.488111+0000", 2021-03-24T21:56:58.061 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1592e-05 2021-03-24T21:56:58.061 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.061 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.061 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:56:58.062 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.488122+0000", 2021-03-24T21:56:58.062 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.7318999999999996e-05 2021-03-24T21:56:58.062 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.062 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.062 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:58.063 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.488200+0000", 2021-03-24T21:56:58.063 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00019132599999999999 2021-03-24T21:56:58.063 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.063 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.064 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:56:58.064 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.488391+0000", 2021-03-24T21:56:58.064 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00022982699999999999 2021-03-24T21:56:58.064 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.064 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.065 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,5", 2021-03-24T21:56:58.065 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.488621+0000", 2021-03-24T21:56:58.065 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001590948 2021-03-24T21:56:58.065 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.065 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.066 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:56:58.066 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.490212+0000", 2021-03-24T21:56:58.066 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00096796600000000001 2021-03-24T21:56:58.066 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.066 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.067 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:58.067 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.491180+0000", 2021-03-24T21:56:58.067 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00076374199999999998 2021-03-24T21:56:58.067 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.068 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.068 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:58.068 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.491943+0000", 2021-03-24T21:56:58.068 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.4624999999999998e-05 2021-03-24T21:56:58.068 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.069 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.069 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:56:58.069 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.492018+0000", 2021-03-24T21:56:58.069 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.4759000000000002e-05 2021-03-24T21:56:58.070 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.070 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.070 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:56:58.070 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.492073+0000", 2021-03-24T21:56:58.070 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0043501010000000003 2021-03-24T21:56:58.071 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:58.071 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:56:58.071 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:58.071 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.072 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.072 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:124 7.b 7:d841eed8:::smithi12556470-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e107)", 2021-03-24T21:56:58.072 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.494851+0000", 2021-03-24T21:56:58.072 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 141.30992705400001, 2021-03-24T21:56:58.072 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0039240020000000002, 2021-03-24T21:56:58.073 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:56:58.073 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T21:56:58.073 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:56:58.073 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:56:58.074 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:56:58.074 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 124 2021-03-24T21:56:58.074 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.074 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:56:58.074 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.075 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:56:58.075 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.494851+0000", 2021-03-24T21:56:58.075 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:56:58.075 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.075 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.076 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:56:58.076 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.494851+0000", 2021-03-24T21:56:58.076 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T21:56:58.076 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.077 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.077 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:56:58.077 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.494850+0000", 2021-03-24T21:56:58.077 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.8899999999999998e-06 2021-03-24T21:56:58.077 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.078 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.078 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:56:58.078 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.494855+0000", 2021-03-24T21:56:58.078 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.5820000000000004e-06 2021-03-24T21:56:58.079 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.079 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.079 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:56:58.079 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.494863+0000", 2021-03-24T21:56:58.079 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.9119999999999997e-06 2021-03-24T21:56:58.080 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.080 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.080 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:56:58.080 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.494868+0000", 2021-03-24T21:56:58.080 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0023491929999999999 2021-03-24T21:56:58.081 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.081 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.081 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:58.081 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.497217+0000", 2021-03-24T21:56:58.081 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.4723999999999998e-05 2021-03-24T21:56:58.082 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.082 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.082 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T21:56:58.082 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.497292+0000", 2021-03-24T21:56:58.083 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001295278 2021-03-24T21:56:58.083 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.083 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.083 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:58.083 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.498587+0000", 2021-03-24T21:56:58.084 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.5016000000000001e-05 2021-03-24T21:56:58.084 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.084 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.084 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:56:58.084 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.498672+0000", 2021-03-24T21:56:58.085 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010340099999999999 2021-03-24T21:56:58.085 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.085 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.085 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:56:58.086 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.498775+0000", 2021-03-24T21:56:58.086 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0039240020000000002 2021-03-24T21:56:58.086 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:58.086 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:56:58.086 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:58.087 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.087 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.087 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:125 7.b 7:de4c3db5:::smithi12556470-32:head [write 521017~798131,stat] snapc 0=[] ondisk+write+known_if_redirected e107)", 2021-03-24T21:56:58.087 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.502054+0000", 2021-03-24T21:56:58.088 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 141.30272438599999, 2021-03-24T21:56:58.088 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0041060309999999996, 2021-03-24T21:56:58.088 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:56:58.088 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:58.088 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:56:58.089 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:56:58.089 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:56:58.089 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 125 2021-03-24T21:56:58.089 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.089 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:56:58.090 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.090 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:56:58.090 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.502054+0000", 2021-03-24T21:56:58.090 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:56:58.091 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.091 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.091 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:56:58.091 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.502054+0000", 2021-03-24T21:56:58.091 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999976 2021-03-24T21:56:58.092 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.092 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.092 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:56:58.092 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.502052+0000", 2021-03-24T21:56:58.092 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00030999899999999998 2021-03-24T21:56:58.093 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.093 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.093 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:56:58.093 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.502362+0000", 2021-03-24T21:56:58.094 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.1806000000000001e-05 2021-03-24T21:56:58.094 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.094 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.094 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:56:58.094 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.502383+0000", 2021-03-24T21:56:58.095 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.6349999999999995e-06 2021-03-24T21:56:58.095 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.095 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.095 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:56:58.095 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.502392+0000", 2021-03-24T21:56:58.096 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.0788000000000002e-05 2021-03-24T21:56:58.096 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.096 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.096 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:58.096 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.502443+0000", 2021-03-24T21:56:58.097 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00013060400000000001 2021-03-24T21:56:58.097 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.097 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.097 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:56:58.097 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.502573+0000", 2021-03-24T21:56:58.098 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00017257399999999999 2021-03-24T21:56:58.098 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.098 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.098 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,5", 2021-03-24T21:56:58.099 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.502746+0000", 2021-03-24T21:56:58.099 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00132138 2021-03-24T21:56:58.099 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.099 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.099 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:56:58.100 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.504067+0000", 2021-03-24T21:56:58.100 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0011008719999999999 2021-03-24T21:56:58.100 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.100 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.101 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:58.101 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.505168+0000", 2021-03-24T21:56:58.101 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00081040500000000004 2021-03-24T21:56:58.101 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.101 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.102 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:58.102 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.505979+0000", 2021-03-24T21:56:58.102 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.2396000000000001e-05 2021-03-24T21:56:58.102 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.103 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.103 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:56:58.103 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.506071+0000", 2021-03-24T21:56:58.103 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.8867000000000001e-05 2021-03-24T21:56:58.104 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.104 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.104 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:56:58.104 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.506160+0000", 2021-03-24T21:56:58.104 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0041060309999999996 2021-03-24T21:56:58.105 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:58.105 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:56:58.105 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:58.105 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.106 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.106 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:137 7.0 e107/99 7:0ece3580:::smithi12556470-35:head v 107'1, mlcod=0'0)", 2021-03-24T21:56:58.106 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.545040+0000", 2021-03-24T21:56:58.106 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 141.259738712, 2021-03-24T21:56:58.107 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0054328359999999999, 2021-03-24T21:56:58.107 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:56:58.107 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:58.107 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:56:58.107 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.107 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:56:58.108 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.545040+0000", 2021-03-24T21:56:58.108 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:56:58.108 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.108 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.109 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:56:58.109 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.545040+0000", 2021-03-24T21:56:58.109 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999957 2021-03-24T21:56:58.109 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.109 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.110 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:56:58.110 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.545035+0000", 2021-03-24T21:56:58.110 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.004385702 2021-03-24T21:56:58.110 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.110 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.111 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:56:58.111 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.549421+0000", 2021-03-24T21:56:58.111 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.6047999999999999e-05 2021-03-24T21:56:58.111 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.112 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.112 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:56:58.112 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.549447+0000", 2021-03-24T21:56:58.112 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0923e-05 2021-03-24T21:56:58.112 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.113 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.113 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:56:58.113 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.549458+0000", 2021-03-24T21:56:58.113 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.0572000000000002e-05 2021-03-24T21:56:58.113 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.114 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.114 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:58.114 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.549519+0000", 2021-03-24T21:56:58.114 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.2843999999999999e-05 2021-03-24T21:56:58.115 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.115 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.115 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:56:58.115 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.549552+0000", 2021-03-24T21:56:58.115 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00088233300000000001 2021-03-24T21:56:58.116 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.116 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.116 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:56:58.116 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.550434+0000", 2021-03-24T21:56:58.117 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.8571000000000001e-05 2021-03-24T21:56:58.117 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.117 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.117 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:56:58.117 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.550472+0000", 2021-03-24T21:56:58.118 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0054328359999999999 2021-03-24T21:56:58.118 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:58.118 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:56:58.118 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:58.119 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.119 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.119 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:160 7.b 7:da3d170a:::smithi12556470-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e107)", 2021-03-24T21:56:58.119 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.625320+0000", 2021-03-24T21:56:58.120 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 141.179457856, 2021-03-24T21:56:58.120 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0037863419999999998, 2021-03-24T21:56:58.120 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:56:58.120 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T21:56:58.121 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:56:58.121 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:56:58.121 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:56:58.121 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 160 2021-03-24T21:56:58.122 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.122 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:56:58.122 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.122 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:56:58.122 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.625320+0000", 2021-03-24T21:56:58.123 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:56:58.123 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.123 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.123 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:56:58.124 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.625320+0000", 2021-03-24T21:56:58.124 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T21:56:58.124 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.124 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.124 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:56:58.125 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.625319+0000", 2021-03-24T21:56:58.125 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5809999999999996e-06 2021-03-24T21:56:58.125 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.125 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.125 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:56:58.126 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.625325+0000", 2021-03-24T21:56:58.126 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.4336e-05 2021-03-24T21:56:58.126 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.126 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.127 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:56:58.127 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.625339+0000", 2021-03-24T21:56:58.127 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.3259999999999996e-06 2021-03-24T21:56:58.127 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.127 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.128 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:56:58.128 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.625348+0000", 2021-03-24T21:56:58.128 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0021449469999999999 2021-03-24T21:56:58.128 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.129 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.129 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:58.129 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.627492+0000", 2021-03-24T21:56:58.129 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010118100000000001 2021-03-24T21:56:58.129 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.130 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.130 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T21:56:58.130 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.627594+0000", 2021-03-24T21:56:58.130 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001193237 2021-03-24T21:56:58.131 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.131 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.131 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:58.131 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.628787+0000", 2021-03-24T21:56:58.131 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000121074 2021-03-24T21:56:58.132 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.132 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.132 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:56:58.132 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.628908+0000", 2021-03-24T21:56:58.132 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000198884 2021-03-24T21:56:58.133 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.133 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.133 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:56:58.133 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.629107+0000", 2021-03-24T21:56:58.134 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0037863419999999998 2021-03-24T21:56:58.134 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:58.134 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:56:58.134 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:58.134 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.135 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.135 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:178 7.d 7:ba8515f1:::smithi12556470-45:head [write 1875090~648012] snapc 0=[] ondisk+write+known_if_redirected e107)", 2021-03-24T21:56:58.135 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.706281+0000", 2021-03-24T21:56:58.135 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 141.098497428, 2021-03-24T21:56:58.136 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0046999620000000002, 2021-03-24T21:56:58.136 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:56:58.136 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:58.136 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:56:58.137 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:56:58.137 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:56:58.137 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 178 2021-03-24T21:56:58.137 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.137 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:56:58.138 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.138 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:56:58.138 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706281+0000", 2021-03-24T21:56:58.138 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:56:58.139 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.139 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.139 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:56:58.139 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706281+0000", 2021-03-24T21:56:58.139 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999971 2021-03-24T21:56:58.140 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.140 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.140 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:56:58.140 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706278+0000", 2021-03-24T21:56:58.141 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00022174299999999999 2021-03-24T21:56:58.141 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.141 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.141 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:56:58.141 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706500+0000", 2021-03-24T21:56:58.142 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.3224e-05 2021-03-24T21:56:58.142 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.142 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.142 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:56:58.142 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706523+0000", 2021-03-24T21:56:58.143 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7773000000000001e-05 2021-03-24T21:56:58.143 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.143 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.143 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:56:58.144 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706541+0000", 2021-03-24T21:56:58.144 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000101025 2021-03-24T21:56:58.144 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.144 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.144 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:58.145 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706642+0000", 2021-03-24T21:56:58.145 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00019631299999999999 2021-03-24T21:56:58.145 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.145 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.146 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:56:58.146 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706838+0000", 2021-03-24T21:56:58.146 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000349081 2021-03-24T21:56:58.146 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.146 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.147 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,3", 2021-03-24T21:56:58.147 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.707187+0000", 2021-03-24T21:56:58.147 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002206515 2021-03-24T21:56:58.147 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.148 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.148 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:56:58.148 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.709394+0000", 2021-03-24T21:56:58.148 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00031391600000000002 2021-03-24T21:56:58.148 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.149 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.149 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:58.149 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.709708+0000", 2021-03-24T21:56:58.149 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0011238960000000001 2021-03-24T21:56:58.149 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.150 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.150 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:58.150 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.710832+0000", 2021-03-24T21:56:58.150 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.9524000000000006e-05 2021-03-24T21:56:58.151 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.151 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.151 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:56:58.151 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.710911+0000", 2021-03-24T21:56:58.151 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.9733999999999993e-05 2021-03-24T21:56:58.152 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.152 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.152 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:56:58.152 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.710981+0000", 2021-03-24T21:56:58.152 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0046999620000000002 2021-03-24T21:56:58.153 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:58.153 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:56:58.153 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:58.153 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.154 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.154 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:179 7.d 7:ba8515f1:::smithi12556470-45:head [setxattr _header (56) in=7b,truncate 2739508] snapc 0=[] ondisk+write+known_if_redirected e107)", 2021-03-24T21:56:58.154 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.706566+0000", 2021-03-24T21:56:58.154 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 141.098212663, 2021-03-24T21:56:58.154 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.004679417, 2021-03-24T21:56:58.155 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:56:58.155 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:56:58.155 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:56:58.155 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:56:58.156 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:56:58.156 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 179 2021-03-24T21:56:58.156 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.156 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:56:58.156 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.157 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:56:58.157 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706566+0000", 2021-03-24T21:56:58.157 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:56:58.157 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.157 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.158 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:56:58.158 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706566+0000", 2021-03-24T21:56:58.158 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T21:56:58.158 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.159 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.159 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:56:58.159 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706564+0000", 2021-03-24T21:56:58.159 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.3519999999999996e-06 2021-03-24T21:56:58.159 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.160 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.160 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:56:58.160 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706573+0000", 2021-03-24T21:56:58.160 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.4070000000000001e-05 2021-03-24T21:56:58.161 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.161 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.161 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:56:58.161 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706587+0000", 2021-03-24T21:56:58.161 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.7489999999999993e-06 2021-03-24T21:56:58.162 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.162 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.162 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:56:58.162 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706597+0000", 2021-03-24T21:56:58.162 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001623022 2021-03-24T21:56:58.163 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.163 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.163 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:58.163 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.708220+0000", 2021-03-24T21:56:58.164 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00013183199999999999 2021-03-24T21:56:58.164 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.164 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.164 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:56:58.164 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.708352+0000", 2021-03-24T21:56:58.165 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00025474500000000001 2021-03-24T21:56:58.165 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.165 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.165 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,3", 2021-03-24T21:56:58.165 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.708606+0000", 2021-03-24T21:56:58.166 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00080621600000000001 2021-03-24T21:56:58.166 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.166 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.166 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:56:58.167 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.709413+0000", 2021-03-24T21:56:58.167 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00047969100000000003 2021-03-24T21:56:58.167 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.167 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.167 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:58.168 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.709892+0000", 2021-03-24T21:56:58.168 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001199615 2021-03-24T21:56:58.168 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.168 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.168 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:56:58.169 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.711092+0000", 2021-03-24T21:56:58.169 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0001032 2021-03-24T21:56:58.169 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.169 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.169 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:56:58.170 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.711195+0000", 2021-03-24T21:56:58.170 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.9874999999999999e-05 2021-03-24T21:56:58.170 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.170 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.171 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:56:58.171 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.711245+0000", 2021-03-24T21:56:58.171 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.004679417 2021-03-24T21:56:58.171 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:58.172 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:56:58.172 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:58.172 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.172 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.172 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:180 7.d 7:ba8515f1:::smithi12556470-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e107)", 2021-03-24T21:56:58.173 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.706637+0000", 2021-03-24T21:56:58.173 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 141.098141365, 2021-03-24T21:56:58.173 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0049820649999999999, 2021-03-24T21:56:58.173 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:56:58.173 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T21:56:58.174 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:56:58.174 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:56:58.174 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:56:58.174 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 180 2021-03-24T21:56:58.175 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.175 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:56:58.175 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.175 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:56:58.175 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706637+0000", 2021-03-24T21:56:58.176 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:56:58.176 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.176 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.176 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:56:58.177 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706637+0000", 2021-03-24T21:56:58.177 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T21:56:58.177 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.177 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.177 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:56:58.178 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706636+0000", 2021-03-24T21:56:58.178 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5099999999999998e-06 2021-03-24T21:56:58.178 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.178 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.179 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:56:58.179 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706641+0000", 2021-03-24T21:56:58.179 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.6740000000000003e-06 2021-03-24T21:56:58.179 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.179 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.180 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:56:58.180 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706650+0000", 2021-03-24T21:56:58.180 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.7699999999999998e-06 2021-03-24T21:56:58.180 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.181 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.181 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:56:58.181 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706656+0000", 2021-03-24T21:56:58.181 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0023962749999999998 2021-03-24T21:56:58.181 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.182 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.182 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:58.182 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.709052+0000", 2021-03-24T21:56:58.182 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000193926 2021-03-24T21:56:58.182 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.183 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.183 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T21:56:58.183 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.709246+0000", 2021-03-24T21:56:58.183 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002080279 2021-03-24T21:56:58.184 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.184 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.184 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:56:58.184 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.711326+0000", 2021-03-24T21:56:58.185 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00013017399999999999 2021-03-24T21:56:58.185 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.185 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.185 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:56:58.185 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.711456+0000", 2021-03-24T21:56:58.186 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000162767 2021-03-24T21:56:58.186 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:56:58.186 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:56:58.186 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:56:58.186 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.711619+0000", 2021-03-24T21:56:58.187 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0049820649999999999 2021-03-24T21:56:58.187 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:58.187 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:56:58.187 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:58.188 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:56:58.188 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:56:58.188 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-24T21:56:58.188 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-24T21:56:58.189 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:58.193 INFO:tasks.rados.rados.0.smithi125.stdout:743: done (3 left) 2021-03-24T21:56:58.193 INFO:tasks.rados.rados.0.smithi125.stdout:744: done (2 left) 2021-03-24T21:56:58.194 INFO:tasks.rados.rados.0.smithi125.stdout:745: done (1 left) 2021-03-24T21:56:58.194 INFO:tasks.rados.rados.0.smithi125.stdout:747: done (0 left) 2021-03-24T21:56:58.194 INFO:tasks.rados.rados.0.smithi125.stdout:748: read oid 10 snap 72 2021-03-24T21:56:58.194 INFO:tasks.rados.rados.0.smithi125.stdout:748: expect deleted 2021-03-24T21:56:58.195 INFO:tasks.rados.rados.0.smithi125.stdout:749: write oid 26 current snap is 74 2021-03-24T21:56:58.195 INFO:tasks.rados.rados.0.smithi125.stdout:749: seq_num 294 ranges {541258=544620,1637706=626754,2840948=100310} 2021-03-24T21:56:58.195 INFO:tasks.rados.rados.0.smithi125.stdout:749: writing smithi12556470-26 from 541258 to 1085878 tid 1 2021-03-24T21:56:58.195 INFO:tasks.rados.rados.0.smithi125.stdout:749: writing smithi12556470-26 from 1637706 to 2264460 tid 2 2021-03-24T21:56:58.195 INFO:tasks.rados.rados.0.smithi125.stdout:749: writing smithi12556470-26 from 2840948 to 2941258 tid 3 2021-03-24T21:56:58.196 INFO:tasks.rados.rados.0.smithi125.stdout:750: rollback oid 48 current snap is 74 2021-03-24T21:56:58.196 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 48 to 70 2021-03-24T21:56:58.196 INFO:tasks.rados.rados.0.smithi125.stdout:749: finishing write tid 1 to smithi12556470-26 2021-03-24T21:56:58.196 INFO:tasks.rados.rados.0.smithi125.stdout:749: finishing write tid 2 to smithi12556470-26 2021-03-24T21:56:58.197 INFO:tasks.rados.rados.0.smithi125.stdout:748: done (2 left) 2021-03-24T21:56:58.197 INFO:tasks.rados.rados.0.smithi125.stdout:751: rollback oid 42 current snap is 74 2021-03-24T21:56:58.197 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 42 to 70 2021-03-24T21:56:58.197 INFO:tasks.rados.rados.0.smithi125.stdout:752: delete oid 12 current snap is 74 2021-03-24T21:56:58.197 INFO:tasks.rados.rados.0.smithi125.stdout:751: finishing rollback tid 1 to smithi12556470-42 2021-03-24T21:56:58.198 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 42 v 0 (ObjNum 1116674992 snap 21916 seq_num 0) dirty dne 2021-03-24T21:56:58.198 INFO:tasks.rados.rados.0.smithi125.stdout:749: finishing write tid 3 to smithi12556470-26 2021-03-24T21:56:58.198 INFO:tasks.rados.rados.0.smithi125.stdout:750: finishing rollback tid 0 to smithi12556470-48 2021-03-24T21:56:58.198 INFO:tasks.rados.rados.0.smithi125.stdout:750: finishing rollback tid 1 to smithi12556470-48 2021-03-24T21:56:58.199 INFO:tasks.rados.rados.0.smithi125.stdout:750: finishing rollback tid 2 to smithi12556470-48 2021-03-24T21:56:58.199 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 48 v 237 (ObjNum 255 snap 59 seq_num 255) dirty exists 2021-03-24T21:56:58.199 INFO:tasks.rados.rados.0.smithi125.stdout:749: finishing write tid 4 to smithi12556470-26 2021-03-24T21:56:58.199 INFO:tasks.rados.rados.0.smithi125.stdout:750: done (3 left) 2021-03-24T21:56:58.200 INFO:tasks.rados.rados.0.smithi125.stdout:751: done (2 left) 2021-03-24T21:56:58.200 INFO:tasks.rados.rados.0.smithi125.stdout:752: done (1 left) 2021-03-24T21:56:58.200 INFO:tasks.rados.rados.0.smithi125.stdout:753: rollback oid 18 current snap is 74 2021-03-24T21:56:58.200 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 18 to 66 2021-03-24T21:56:58.200 INFO:tasks.rados.rados.0.smithi125.stdout:754: delete oid 3 current snap is 74 2021-03-24T21:56:58.201 INFO:tasks.rados.rados.0.smithi125.stdout:749: finishing write tid 5 to smithi12556470-26 2021-03-24T21:56:58.201 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 26 v 137 (ObjNum 294 snap 74 seq_num 294) dirty exists 2021-03-24T21:56:58.201 INFO:tasks.rados.rados.0.smithi125.stdout:749: left oid 26 (ObjNum 294 snap 74 seq_num 294) 2021-03-24T21:56:58.201 INFO:tasks.rados.rados.0.smithi125.stdout:749: done (2 left) 2021-03-24T21:56:58.202 INFO:tasks.rados.rados.0.smithi125.stdout:754: done (1 left) 2021-03-24T21:56:58.202 INFO:tasks.rados.rados.0.smithi125.stdout:755: write oid 10 current snap is 74 2021-03-24T21:56:58.202 INFO:tasks.rados.rados.0.smithi125.stdout:755: seq_num 297 ranges {591679=486720,1813152=490434} 2021-03-24T21:56:58.202 INFO:tasks.rados.rados.0.smithi125.stdout:755: writing smithi12556470-10 from 591679 to 1078399 tid 1 2021-03-24T21:56:58.202 INFO:tasks.rados.rados.0.smithi125.stdout:755: writing smithi12556470-10 from 1813152 to 2303586 tid 2 2021-03-24T21:56:58.203 INFO:tasks.rados.rados.0.smithi125.stdout:756: read oid 45 snap -1 2021-03-24T21:56:58.203 INFO:tasks.rados.rados.0.smithi125.stdout:756: expect (ObjNum 150 snap 30 seq_num 150) 2021-03-24T21:56:58.203 INFO:tasks.rados.rados.0.smithi125.stdout:757: write (excl) oid 20 current snap is 74 2021-03-24T21:56:58.203 INFO:tasks.rados.rados.0.smithi125.stdout:753: finishing rollback tid 0 to smithi12556470-18 2021-03-24T21:56:58.204 INFO:tasks.rados.rados.0.smithi125.stdout:757: seq_num 298 ranges {608486=561969,1871634=634092,2950640=57846} 2021-03-24T21:56:58.205 INFO:tasks.rados.rados.0.smithi125.stdout:757: writing smithi12556470-20 from 608486 to 1170455 tid 1 2021-03-24T21:56:58.205 INFO:tasks.rados.rados.0.smithi125.stdout:757: writing smithi12556470-20 from 1871634 to 2505726 tid 2 2021-03-24T21:56:58.206 INFO:tasks.rados.rados.0.smithi125.stdout:757: writing smithi12556470-20 from 2950640 to 3008486 tid 3 2021-03-24T21:56:58.207 INFO:tasks.rados.rados.0.smithi125.stdout:758: snap_remove snap 63 2021-03-24T21:56:58.245 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:58.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:58.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:56:58.350 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:58.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:58 smithi125 conmon[19936]: cluster 2021-03-24T21:56:56.760624+0000 mgr.y ( 2021-03-24T21:56:58.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:58 smithi125 conmon[19936]: mgr.14144) 389 : cluster [DBG] pgmap v533: 158 pgs: 158 active+clean; 154 MiB data, 525 MiB used, 536 GiB / 536 GiB avail; 19 MiB/s rd, 0 B/s wr, 15 op/s 2021-03-24T21:56:58.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:58 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T21:56:58.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:58 smithi125 conmon[19936]: T21:56:57.140082+0000 mon.a (mon.0) 873 : cluster [DBG] osdmap e248: 8 total, 8 up, 6 in 2021-03-24T21:56:58.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:58 smithi125 conmon[19936]: 2021-03-24T21:56:58.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:58 smithi125 conmon[19936]: audit 2021-03-24T21: 2021-03-24T21:56:58.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:58 smithi125 conmon[19936]: 56:57.600048+0000 mon.a (mon 2021-03-24T21:56:58.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:58 smithi125 conmon[19936]: .0) 874 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:56:58.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:58 smithi125 conmon[19936]: audit 2021 2021-03-24T21:56:58.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:58 smithi125 conmon[19936]: -03-24T21:56:57. 2021-03-24T21:56:58.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:58 smithi125 conmon[19936]: 926870+0000 mon.a (mon.0) 875 : 2021-03-24T21:56:58.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:58 smithi125 conmon[19936]: audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:56:58.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:58 smithi125 conmon[24927]: cluster 2021-03-24T21:56:56.760624+0000 mgr.y 2021-03-24T21:56:58.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:58 smithi125 conmon[24927]: (mgr.14144) 389 : cluster [DBG] pgmap v533: 158 pgs: 158 active+clean; 154 MiB data, 525 MiB used, 536 GiB / 536 GiB avail; 19 MiB/s rd, 0 B/s wr, 15 op/s 2021-03-24T21:56:58.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:58 smithi125 conmon[24927]: cluster 2021-03-24T21:56: 2021-03-24T21:56:58.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:58 smithi125 conmon[24927]: 57.140082+0000 mon.a (mon.0) 873 : cluster [DBG] osdmap e248: 8 total, 8 up, 6 in 2021-03-24T21:56:58.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:58 smithi125 conmon[24927]: audit 2021-03-24T21:56: 2021-03-24T21:56:58.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:58 smithi125 conmon[24927]: 57.600048+0000 mon.a (mon.0) 874 : audit 2021-03-24T21:56:58.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:58 smithi125 conmon[24927]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:56:58.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:58 smithi125 conmon[24927]: audit 2021-03-24T21:56:57.926870 2021-03-24T21:56:58.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:58 smithi125 conmon[24927]: +0000 mon.a (mon.0) 875 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:56:58.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:58 smithi125 conmon[24927]: 2021-03-24T21:56:58.415 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:56:58.416 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:58 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:56:58.417 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:58 smithi142 conmon[20125]: 21:56:56.760624+0000 mgr.y (mgr.14144) 389 : cluster [DBG] pgmap v533: 158 pgs: 158 active+clean; 154 MiB data, 525 MiB used, 536 GiB / 536 GiB avail; 19 MiB/s rd, 0 B/s wr, 15 op/s 2021-03-24T21:56:58.417 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:58 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:56:58.417 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:58 smithi142 conmon[20125]: 56:57.140082+0000 mon.a (mon.0) 873 : cluster [DBG] osdmap e248: 8 total, 8 up, 6 in 2021-03-24T21:56:58.417 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:58 smithi142 conmon[20125]: 2021-03-24T21:56:58.418 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:58 smithi142 conmon[20125]: audit 2021-03-24T21:56:58.418 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:58 smithi142 conmon[20125]: 2021-03-24T21:56:57.600048+0000 mon.a 2021-03-24T21:56:58.418 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:58 smithi142 conmon[20125]: (mon.0) 874 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:56:58.418 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:58 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:56:58.419 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:58 smithi142 conmon[20125]: 03-24T21:56:57.926870+0000 2021-03-24T21:56:58.419 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:58 smithi142 conmon[20125]: mon.a (mon.0) 875 : audit [INF] 2021-03-24T21:56:58.419 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:58 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:56:58.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:58.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:56:58.521 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:58.577 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:58.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:58.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:56:58.684 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:58.741 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:58.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:58.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:56:58.848 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:58.899 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:58.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:58.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:56:59.005 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:59.059 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:56:59.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:59.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:56:59.150 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:59 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:56:59.151 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:59 smithi125 conmon[24927]: T21:56:58.142028+0000 mon.a (mon.0) 876 : cluster 2021-03-24T21:56:59.151 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:56:59 smithi125 conmon[24927]: [DBG] osdmap e249: 8 total, 8 up, 6 in 2021-03-24T21:56:59.159 INFO:tasks.rados.rados.0.smithi125.stdout:758: done (4 left) 2021-03-24T21:56:59.160 INFO:tasks.rados.rados.0.smithi125.stdout:759: snap_remove snap 73 2021-03-24T21:56:59.166 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:59.209 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:56:59.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:59.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:56:59.314 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:59.374 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:56:59.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:59.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:56:59.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:59 smithi125 conmon[19936]: cluster 2021-03-24T21:56 2021-03-24T21:56:59.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:56:59 smithi125 conmon[19936]: :58.142028+0000 mon.a (mon.0) 876 : cluster [DBG] osdmap e249: 8 total, 8 up, 6 in 2021-03-24T21:56:59.480 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:59.532 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:56:59.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:59.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:56:59.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:59 smithi142 conmon[20125]: cluster 2021-03-24T21:56 2021-03-24T21:56:59.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:56:59 smithi142 conmon[20125]: :58.142028+0000 mon.a (mon.0) 876 : cluster [DBG] osdmap e249: 8 total, 8 up, 6 in 2021-03-24T21:56:59.638 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:59.642 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.2 dump_ops_in_flight 2021-03-24T21:56:59.671 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:56:59.693 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:56:59.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:59.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:56:59.733 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:56:59.800 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:59.848 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:56:59.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:56:59.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:56:59.956 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:00.017 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:00.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:00.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:00.123 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:00.143 INFO:tasks.rados.rados.0.smithi125.stdout:759: done (4 left) 2021-03-24T21:57:00.143 INFO:tasks.rados.rados.0.smithi125.stdout:760: delete oid 41 current snap is 74 2021-03-24T21:57:00.143 INFO:tasks.rados.rados.0.smithi125.stdout:753: finishing rollback tid 1 to smithi12556470-18 2021-03-24T21:57:00.143 INFO:tasks.rados.rados.0.smithi125.stdout:753: finishing rollback tid 2 to smithi12556470-18 2021-03-24T21:57:00.144 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 18 v 161 (ObjNum 236 snap 53 seq_num 236) dirty exists 2021-03-24T21:57:00.144 INFO:tasks.rados.rados.0.smithi125.stdout:755: finishing write tid 1 to smithi12556470-10 2021-03-24T21:57:00.144 INFO:tasks.rados.rados.0.smithi125.stdout:755: finishing write tid 2 to smithi12556470-10 2021-03-24T21:57:00.144 INFO:tasks.rados.rados.0.smithi125.stdout:756: expect (ObjNum 150 snap 30 seq_num 150) 2021-03-24T21:57:00.176 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:00.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:00.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:57:00.281 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:00.284 INFO:tasks.rados.rados.0.smithi125.stdout:755: finishing write tid 3 to smithi12556470-10 2021-03-24T21:57:00.284 INFO:tasks.rados.rados.0.smithi125.stdout:755: finishing write tid 4 to smithi12556470-10 2021-03-24T21:57:00.284 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 10 v 250 (ObjNum 297 snap 74 seq_num 297) dirty exists 2021-03-24T21:57:00.284 INFO:tasks.rados.rados.0.smithi125.stdout:755: left oid 10 (ObjNum 297 snap 74 seq_num 297) 2021-03-24T21:57:00.285 INFO:tasks.rados.rados.0.smithi125.stdout:757: finishing write tid 1 to smithi12556470-20 2021-03-24T21:57:00.285 INFO:tasks.rados.rados.0.smithi125.stdout:757: finishing write tid 2 to smithi12556470-20 2021-03-24T21:57:00.285 INFO:tasks.rados.rados.0.smithi125.stdout:757: finishing write tid 3 to smithi12556470-20 2021-03-24T21:57:00.285 INFO:tasks.rados.rados.0.smithi125.stdout:753: done (4 left) 2021-03-24T21:57:00.285 INFO:tasks.rados.rados.0.smithi125.stdout:755: done (3 left) 2021-03-24T21:57:00.286 INFO:tasks.rados.rados.0.smithi125.stdout:756: done (2 left) 2021-03-24T21:57:00.286 INFO:tasks.rados.rados.0.smithi125.stdout:760: done (1 left) 2021-03-24T21:57:00.286 INFO:tasks.rados.rados.0.smithi125.stdout:761: delete oid 39 current snap is 74 2021-03-24T21:57:00.286 INFO:tasks.rados.rados.0.smithi125.stdout:757: finishing write tid 4 to smithi12556470-20 2021-03-24T21:57:00.287 INFO:tasks.rados.rados.0.smithi125.stdout:757: finishing write tid 5 to smithi12556470-20 2021-03-24T21:57:00.287 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 20 v 160 (ObjNum 298 snap 74 seq_num 298) dirty exists 2021-03-24T21:57:00.287 INFO:tasks.rados.rados.0.smithi125.stdout:757: left oid 20 (ObjNum 298 snap 74 seq_num 298) 2021-03-24T21:57:00.289 INFO:tasks.rados.rados.0.smithi125.stdout:757: done (1 left) 2021-03-24T21:57:00.289 INFO:tasks.rados.rados.0.smithi125.stdout:761: done (0 left) 2021-03-24T21:57:00.289 INFO:tasks.rados.rados.0.smithi125.stdout:762: read oid 17 snap -1 2021-03-24T21:57:00.289 INFO:tasks.rados.rados.0.smithi125.stdout:762: expect (ObjNum 123 snap 23 seq_num 123) 2021-03-24T21:57:00.289 INFO:tasks.rados.rados.0.smithi125.stdout:763: delete oid 25 current snap is 74 2021-03-24T21:57:00.290 INFO:tasks.rados.rados.0.smithi125.stdout:763: done (1 left) 2021-03-24T21:57:00.290 INFO:tasks.rados.rados.0.smithi125.stdout:764: rollback oid 30 current snap is 74 2021-03-24T21:57:00.290 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 30 to 71 2021-03-24T21:57:00.290 INFO:tasks.rados.rados.0.smithi125.stdout:765: read oid 39 snap -1 2021-03-24T21:57:00.291 INFO:tasks.rados.rados.0.smithi125.stdout:765: expect deleted 2021-03-24T21:57:00.291 INFO:tasks.rados.rados.0.smithi125.stdout:766: delete oid 49 current snap is 74 2021-03-24T21:57:00.298 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:00 smithi142 conmon[20125]: cluster 2021-03-24T21:56:58.761684+0000 mgr.y (mgr.14144) 2021-03-24T21:57:00.298 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:00 smithi142 conmon[20125]: 390 : cluster [DBG] pgmap v536: 158 pgs: 158 active+clean; 153 MiB data, 521 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 580 KiB/s wr, 3 op/s 2021-03-24T21:57:00.299 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:00 smithi142 conmon[20125]: cluster 2021-03-24T21:56:59.158368+0000 mon.a (mon.0) 877 : cluster [DBG] osdmap e250: 8 total, 8 up, 6 in 2021-03-24T21:57:00.299 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:00 smithi142 conmon[20125]: cluster 2021-03-24T21:57:00.300 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:00 smithi142 conmon[20125]: 2021-03-24T21:57:00.138860+0000 mon.a (mon.0) 878 : cluster [DBG] osdmap e251: 8 total, 8 up, 6 in 2021-03-24T21:57:00.300 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:00 smithi142 conmon[20125]: 2021-03-24T21:57:00.300 INFO:tasks.rados.rados.0.smithi125.stdout:764: finishing rollback tid 0 to smithi12556470-30 2021-03-24T21:57:00.301 INFO:tasks.rados.rados.0.smithi125.stdout:764: finishing rollback tid 1 to smithi12556470-30 2021-03-24T21:57:00.301 INFO:tasks.rados.rados.0.smithi125.stdout:764: finishing rollback tid 2 to smithi12556470-30 2021-03-24T21:57:00.301 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 30 v 164 (ObjNum 287 snap 71 seq_num 287) dirty exists 2021-03-24T21:57:00.302 INFO:tasks.rados.rados.0.smithi125.stdout:762: expect (ObjNum 123 snap 23 seq_num 123) 2021-03-24T21:57:00.338 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:00.341 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:00 smithi125 conmon[19936]: cluster 2021-03-24T21:56:58.761684+0000 mgr.y (mgr.14144) 390 : 2021-03-24T21:57:00.341 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:00 smithi125 conmon[19936]: cluster [DBG] pgmap v536: 158 pgs: 158 active+clean; 153 MiB data, 521 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 580 KiB/s wr, 3 op/s 2021-03-24T21:57:00.341 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:00 smithi125 conmon[19936]: cluster 2021-03-24T21:56:59.158368+0000 mon.a (mon.0) 877 : cluster [DBG] osdmap e250: 8 total, 8 up, 6 in 2021-03-24T21:57:00.341 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:00 smithi125 conmon[19936]: cluster 2021-03-24T21:57:00.138860+0000 mon.a (mon.0) 878 : 2021-03-24T21:57:00.342 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:00 smithi125 conmon[19936]: cluster [DBG] osdmap e251: 8 total, 8 up, 6 in 2021-03-24T21:57:00.343 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:00 smithi125 conmon[24927]: cluster 2021-03-24T21:56: 2021-03-24T21:57:00.343 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:00 smithi125 conmon[24927]: 58.761684+0000 mgr.y (mgr.14144) 390 : cluster [DBG] pgmap v536: 158 pgs: 158 active+clean; 153 MiB data, 521 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 580 KiB/s wr, 3 op/s 2021-03-24T21:57:00.343 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:00 smithi125 conmon[24927]: cluster 2021-03-24T21:56:59.158368+0000 mon.a (mon.0 2021-03-24T21:57:00.343 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:00 smithi125 conmon[24927]: ) 877 : cluster [DBG] osdmap e250: 8 total, 8 up, 6 in 2021-03-24T21:57:00.344 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:00 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:57:00.344 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:00 smithi125 conmon[24927]: -03-24T21:57:00.138860+0000 mon.a ( 2021-03-24T21:57:00.344 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:00 smithi125 conmon[24927]: mon.0) 878 : cluster [DBG] osdmap e251: 8 total, 8 up, 6 in 2021-03-24T21:57:00.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:00.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:57:00.423 INFO:tasks.rados.rados.0.smithi125.stdout:762: done (3 left) 2021-03-24T21:57:00.424 INFO:tasks.rados.rados.0.smithi125.stdout:764: done (2 left) 2021-03-24T21:57:00.424 INFO:tasks.rados.rados.0.smithi125.stdout:765: done (1 left) 2021-03-24T21:57:00.424 INFO:tasks.rados.rados.0.smithi125.stdout:766: done (0 left) 2021-03-24T21:57:00.424 INFO:tasks.rados.rados.0.smithi125.stdout:767: copy_from oid 19 from oid 11 current snap is 74 2021-03-24T21:57:00.424 INFO:tasks.rados.rados.0.smithi125.stdout:768: rollback oid 44 current snap is 74 2021-03-24T21:57:00.425 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 44 to 66 2021-03-24T21:57:00.425 INFO:tasks.rados.rados.0.smithi125.stdout:769: write (excl) oid 14 current snap is 74 2021-03-24T21:57:00.425 INFO:tasks.rados.rados.0.smithi125.stdout:769: seq_num 303 ranges {692521=538214,1931338=785088} 2021-03-24T21:57:00.430 INFO:tasks.rados.rados.0.smithi125.stdout:769: writing smithi12556470-14 from 692521 to 1230735 tid 1 2021-03-24T21:57:00.440 INFO:tasks.rados.rados.0.smithi125.stdout:769: writing smithi12556470-14 from 1931338 to 2716426 tid 2 2021-03-24T21:57:00.440 INFO:tasks.rados.rados.0.smithi125.stdout:770: write (excl) oid 22 current snap is 74 2021-03-24T21:57:00.441 INFO:tasks.rados.rados.0.smithi125.stdout:768: finishing rollback tid 0 to smithi12556470-44 2021-03-24T21:57:00.441 INFO:tasks.rados.rados.0.smithi125.stdout:770: seq_num 304 ranges {709328=613463,1989820=645099} 2021-03-24T21:57:00.447 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:00.448 INFO:tasks.rados.rados.0.smithi125.stdout:770: writing smithi12556470-22 from 709328 to 1322791 tid 1 2021-03-24T21:57:00.457 INFO:tasks.rados.rados.0.smithi125.stdout:770: writing smithi12556470-22 from 1989820 to 2634919 tid 2 2021-03-24T21:57:00.457 INFO:tasks.rados.rados.0.smithi125.stdout:771: copy_from oid 32 from oid 8 current snap is 74 2021-03-24T21:57:00.458 INFO:tasks.rados.rados.0.smithi125.stdout:768: finishing rollback tid 1 to smithi12556470-44 2021-03-24T21:57:00.458 INFO:tasks.rados.rados.0.smithi125.stdout:768: finishing rollback tid 2 to smithi12556470-44 2021-03-24T21:57:00.458 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 44 v 187 (ObjNum 259 snap 60 seq_num 259) dirty exists 2021-03-24T21:57:00.458 INFO:tasks.rados.rados.0.smithi125.stdout:769: finishing write tid 1 to smithi12556470-14 2021-03-24T21:57:00.459 INFO:tasks.rados.rados.0.smithi125.stdout:768: done (4 left) 2021-03-24T21:57:00.459 INFO:tasks.rados.rados.0.smithi125.stdout:772: rollback oid 13 current snap is 74 2021-03-24T21:57:00.459 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 13 to 70 2021-03-24T21:57:00.459 INFO:tasks.rados.rados.0.smithi125.stdout:767: finishing copy_from to smithi12556470-19 2021-03-24T21:57:00.459 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 19 v 188 (ObjNum 244 snap 57 seq_num 244) dirty exists 2021-03-24T21:57:00.460 INFO:tasks.rados.rados.0.smithi125.stdout:767: finishing copy_from racing read to smithi12556470-19 2021-03-24T21:57:00.460 INFO:tasks.rados.rados.0.smithi125.stdout:769: finishing write tid 2 to smithi12556470-14 2021-03-24T21:57:00.460 INFO:tasks.rados.rados.0.smithi125.stdout:769: finishing write tid 3 to smithi12556470-14 2021-03-24T21:57:00.460 INFO:tasks.rados.rados.0.smithi125.stdout:767: done (4 left) 2021-03-24T21:57:00.461 INFO:tasks.rados.rados.0.smithi125.stdout:773: delete oid 24 current snap is 74 2021-03-24T21:57:00.461 INFO:tasks.rados.rados.0.smithi125.stdout:769: finishing write tid 4 to smithi12556470-14 2021-03-24T21:57:00.461 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 14 v 221 (ObjNum 303 snap 74 seq_num 303) dirty exists 2021-03-24T21:57:00.461 INFO:tasks.rados.rados.0.smithi125.stdout:769: left oid 14 (ObjNum 303 snap 74 seq_num 303) 2021-03-24T21:57:00.461 INFO:tasks.rados.rados.0.smithi125.stdout:770: finishing write tid 1 to smithi12556470-22 2021-03-24T21:57:00.462 INFO:tasks.rados.rados.0.smithi125.stdout:769: done (4 left) 2021-03-24T21:57:00.462 INFO:tasks.rados.rados.0.smithi125.stdout:773: done (3 left) 2021-03-24T21:57:00.463 INFO:tasks.rados.rados.0.smithi125.stdout:774: write oid 4 current snap is 74 2021-03-24T21:57:00.463 INFO:tasks.rados.rados.0.smithi125.stdout:774: seq_num 306 ranges {742942=480314,1823137=648768} 2021-03-24T21:57:00.469 INFO:tasks.rados.rados.0.smithi125.stdout:774: writing smithi12556470-4 from 742942 to 1223256 tid 1 2021-03-24T21:57:00.477 INFO:tasks.rados.rados.0.smithi125.stdout:774: writing smithi12556470-4 from 1823137 to 2471905 tid 2 2021-03-24T21:57:00.478 INFO:tasks.rados.rados.0.smithi125.stdout:775: read oid 35 snap -1 2021-03-24T21:57:00.478 INFO:tasks.rados.rados.0.smithi125.stdout:775: expect (ObjNum 150 snap 30 seq_num 150) 2021-03-24T21:57:00.478 INFO:tasks.rados.rados.0.smithi125.stdout:776: read oid 25 snap 66 2021-03-24T21:57:00.478 INFO:tasks.rados.rados.0.smithi125.stdout:776: expect deleted 2021-03-24T21:57:00.478 INFO:tasks.rados.rados.0.smithi125.stdout:777: read oid 11 snap -1 2021-03-24T21:57:00.479 INFO:tasks.rados.rados.0.smithi125.stdout:777: expect (ObjNum 244 snap 57 seq_num 244) 2021-03-24T21:57:00.479 INFO:tasks.rados.rados.0.smithi125.stdout:778: read oid 3 snap -1 2021-03-24T21:57:00.479 INFO:tasks.rados.rados.0.smithi125.stdout:778: expect deleted 2021-03-24T21:57:00.479 INFO:tasks.rados.rados.0.smithi125.stdout:779: snap_create 2021-03-24T21:57:00.480 INFO:tasks.rados.rados.0.smithi125.stdout:772: finishing rollback tid 0 to smithi12556470-13 2021-03-24T21:57:00.480 INFO:tasks.rados.rados.0.smithi125.stdout:770: finishing write tid 2 to smithi12556470-22 2021-03-24T21:57:00.480 INFO:tasks.rados.rados.0.smithi125.stdout:772: finishing rollback tid 1 to smithi12556470-13 2021-03-24T21:57:00.480 INFO:tasks.rados.rados.0.smithi125.stdout:770: finishing write tid 3 to smithi12556470-22 2021-03-24T21:57:00.480 INFO:tasks.rados.rados.0.smithi125.stdout:772: finishing rollback tid 2 to smithi12556470-13 2021-03-24T21:57:00.481 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 13 v 241 (ObjNum 251 snap 59 seq_num 251) dirty exists 2021-03-24T21:57:00.481 INFO:tasks.rados.rados.0.smithi125.stdout:770: finishing write tid 4 to smithi12556470-22 2021-03-24T21:57:00.481 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 22 v 191 (ObjNum 304 snap 74 seq_num 304) dirty exists 2021-03-24T21:57:00.481 INFO:tasks.rados.rados.0.smithi125.stdout:770: left oid 22 (ObjNum 304 snap 74 seq_num 304) 2021-03-24T21:57:00.482 INFO:tasks.rados.rados.0.smithi125.stdout:774: finishing write tid 1 to smithi12556470-4 2021-03-24T21:57:00.482 INFO:tasks.rados.rados.0.smithi125.stdout:771: finishing copy_from to smithi12556470-32 2021-03-24T21:57:00.482 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 32 v 38 (ObjNum 275 snap 68 seq_num 275) dirty exists 2021-03-24T21:57:00.482 INFO:tasks.rados.rados.0.smithi125.stdout:771: finishing copy_from racing read to smithi12556470-32 2021-03-24T21:57:00.483 INFO:tasks.rados.rados.0.smithi125.stdout:775: expect (ObjNum 150 snap 30 seq_num 150) 2021-03-24T21:57:00.497 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:00.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:00.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:57:00.603 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:00.616 INFO:tasks.rados.rados.0.smithi125.stdout:774: finishing write tid 2 to smithi12556470-4 2021-03-24T21:57:00.616 INFO:tasks.rados.rados.0.smithi125.stdout:774: finishing write tid 3 to smithi12556470-4 2021-03-24T21:57:00.617 INFO:tasks.rados.rados.0.smithi125.stdout:774: finishing write tid 4 to smithi12556470-4 2021-03-24T21:57:00.617 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 4 v 219 (ObjNum 306 snap 74 seq_num 306) dirty exists 2021-03-24T21:57:00.617 INFO:tasks.rados.rados.0.smithi125.stdout:774: left oid 4 (ObjNum 306 snap 74 seq_num 306) 2021-03-24T21:57:00.617 INFO:tasks.rados.rados.0.smithi125.stdout:777: expect (ObjNum 244 snap 57 seq_num 244) 2021-03-24T21:57:00.660 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:00.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:00.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:00.766 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:00.824 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:00.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:00.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:57:00.929 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:00.989 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:00.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:00.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:57:01.039 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:57:01.093 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:01.150 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:01.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:01.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:57:01.176 INFO:tasks.rados.rados.0.smithi125.stdout:770: done (8 left) 2021-03-24T21:57:01.177 INFO:tasks.rados.rados.0.smithi125.stdout:771: done (7 left) 2021-03-24T21:57:01.177 INFO:tasks.rados.rados.0.smithi125.stdout:772: done (6 left) 2021-03-24T21:57:01.177 INFO:tasks.rados.rados.0.smithi125.stdout:774: done (5 left) 2021-03-24T21:57:01.177 INFO:tasks.rados.rados.0.smithi125.stdout:775: done (4 left) 2021-03-24T21:57:01.177 INFO:tasks.rados.rados.0.smithi125.stdout:776: done (3 left) 2021-03-24T21:57:01.178 INFO:tasks.rados.rados.0.smithi125.stdout:777: done (2 left) 2021-03-24T21:57:01.178 INFO:tasks.rados.rados.0.smithi125.stdout:778: done (1 left) 2021-03-24T21:57:01.178 INFO:tasks.rados.rados.0.smithi125.stdout:779: done (0 left) 2021-03-24T21:57:01.178 INFO:tasks.rados.rados.0.smithi125.stdout:780: delete oid 5 current snap is 75 2021-03-24T21:57:01.183 INFO:tasks.rados.rados.0.smithi125.stdout:780: done (0 left) 2021-03-24T21:57:01.183 INFO:tasks.rados.rados.0.smithi125.stdout:781: rollback oid 43 current snap is 75 2021-03-24T21:57:01.184 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 43 to 70 2021-03-24T21:57:01.184 INFO:tasks.rados.rados.0.smithi125.stdout:782: write (excl) oid 28 current snap is 75 2021-03-24T21:57:01.184 INFO:tasks.rados.rados.0.smithi125.stdout:782: seq_num 308 ranges {776556=630812,1823748=652437,3108517=68039} 2021-03-24T21:57:01.185 INFO:teuthology.orchestra.run.smithi125.stderr:reweighted osd.2 to 0.724266 (b969) 2021-03-24T21:57:01.191 INFO:tasks.rados.rados.0.smithi125.stdout:782: writing smithi12556470-28 from 776556 to 1407368 tid 1 2021-03-24T21:57:01.198 INFO:tasks.rados.rados.0.smithi125.stdout:782: writing smithi12556470-28 from 1823748 to 2476185 tid 2 2021-03-24T21:57:01.202 INFO:tasks.rados.rados.0.smithi125.stdout:782: writing smithi12556470-28 from 3108517 to 3176556 tid 3 2021-03-24T21:57:01.202 INFO:tasks.rados.rados.0.smithi125.stdout:783: snap_create 2021-03-24T21:57:01.203 INFO:tasks.rados.rados.0.smithi125.stdout:782: finishing write tid 1 to smithi12556470-28 2021-03-24T21:57:01.203 INFO:tasks.rados.rados.0.smithi125.stdout:782: finishing write tid 2 to smithi12556470-28 2021-03-24T21:57:01.204 INFO:tasks.rados.rados.0.smithi125.stdout:782: finishing write tid 3 to smithi12556470-28 2021-03-24T21:57:01.205 INFO:tasks.rados.rados.0.smithi125.stdout:782: finishing write tid 4 to smithi12556470-28 2021-03-24T21:57:01.205 INFO:tasks.rados.rados.0.smithi125.stdout:782: finishing write tid 5 to smithi12556470-28 2021-03-24T21:57:01.206 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 28 v 175 (ObjNum 308 snap 75 seq_num 308) dirty exists 2021-03-24T21:57:01.206 INFO:tasks.rados.rados.0.smithi125.stdout:782: left oid 28 (ObjNum 308 snap 75 seq_num 308) 2021-03-24T21:57:01.258 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:01.271 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:01 smithi142 conmon[20125]: audit 2021-03-24T21:57:00.449270+0000 mon.a (mon.0 2021-03-24T21:57:01.271 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:01 smithi142 conmon[20125]: ) 879 : audit [INF] from='client.? v1:172.21.15.125:0/2276604061' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 0.7242660621692911}]: dispatch 2021-03-24T21:57:01.312 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:01 smithi125 conmon[19936]: audit 2021-03-24T21:57:00.449270+0000 mon.a (mon.0) 2021-03-24T21:57:01.312 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:01 smithi125 conmon[19936]: 879 : audit [INF] from='client.? v1:172.21.15.125:0/2276604061' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 0.7242660621692911}]: dispatch 2021-03-24T21:57:01.313 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:01 smithi125 conmon[24927]: audit 2021-03-24T21:57:00.449270+0000 mon.a (mon.0 2021-03-24T21:57:01.313 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:01 smithi125 conmon[24927]: ) 879 : audit [INF] from='client.? v1:172.21.15.125:0/2276604061' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 0.7242660621692911}]: dispatch 2021-03-24T21:57:01.314 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:01.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:01.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:57:01.421 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:01.483 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:01.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:01.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:57:01.588 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:01.643 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:01.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:01.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:57:01.749 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:01.806 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:01.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:01.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:57:01.912 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:01.965 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:01.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:01.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:57:02.072 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:02.172 INFO:tasks.rados.rados.0.smithi125.stdout:782: done (2 left) 2021-03-24T21:57:02.173 INFO:tasks.rados.rados.0.smithi125.stdout:783: done (1 left) 2021-03-24T21:57:02.173 INFO:tasks.rados.rados.0.smithi125.stdout:784: read oid 48 snap 75 2021-03-24T21:57:02.173 INFO:tasks.rados.rados.0.smithi125.stdout:784: expect (ObjNum 255 snap 59 seq_num 255) 2021-03-24T21:57:02.173 INFO:tasks.rados.rados.0.smithi125.stdout:785: write (excl) oid 39 current snap is 76 2021-03-24T21:57:02.174 INFO:tasks.rados.rados.0.smithi125.stdout:785: seq_num 309 ranges {793363=706061,2282230=796095} 2021-03-24T21:57:02.174 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:02.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:02.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:57:02.183 INFO:tasks.rados.rados.0.smithi125.stdout:785: writing smithi12556470-39 from 793363 to 1499424 tid 1 2021-03-24T21:57:02.193 INFO:tasks.rados.rados.0.smithi125.stdout:785: writing smithi12556470-39 from 2282230 to 3078325 tid 2 2021-03-24T21:57:02.194 INFO:tasks.rados.rados.0.smithi125.stdout:786: read oid 23 snap -1 2021-03-24T21:57:02.194 INFO:tasks.rados.rados.0.smithi125.stdout:786: expect (ObjNum 281 snap 70 seq_num 281) 2021-03-24T21:57:02.194 INFO:tasks.rados.rados.0.smithi125.stdout:787: read oid 35 snap 66 2021-03-24T21:57:02.195 INFO:tasks.rados.rados.0.smithi125.stdout:787: expect (ObjNum 150 snap 30 seq_num 150) 2021-03-24T21:57:02.195 INFO:tasks.rados.rados.0.smithi125.stdout:788: snap_remove snap 71 2021-03-24T21:57:02.283 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:02.295 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:02 smithi125 conmon[19936]: cluster 2021-03-24T21:57:00.762395+0000 mgr.y (mgr.14144) 2021-03-24T21:57:02.296 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:02 smithi125 conmon[19936]: 391 : cluster [DBG] pgmap v539: 158 pgs: 158 active+clean; 156 MiB data, 512 MiB used, 536 GiB / 536 GiB avail; 15 MiB/s rd, 2.5 MiB/s wr, 25 op/s 2021-03-24T21:57:02.296 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:02 smithi125 conmon[19936]: audit 2021-03-24T21:57:01.167481+0000 mon.a (mon.0) 880 : audit [INF] from='client.? v1:172.21.15.125:0/2276604061' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 0.7242660621692911}]': finished 2021-03-24T21:57:02.296 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:02 smithi125 conmon[19936]: cluster 2021-03-24T21:57:01.167609+0000 mon.a (mon.0) 881 : cluster [DBG] osdmap e252: 8 total, 8 up, 6 in 2021-03-24T21:57:02.296 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:02 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:57:02.297 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:02 smithi125 conmon[19936]: -24T21:57:02.169382+0000 mon.a (mon.0 2021-03-24T21:57:02.297 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:02 smithi125 conmon[19936]: ) 882 : cluster [DBG] osdmap e253: 8 total, 8 up, 6 in 2021-03-24T21:57:02.297 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:02 smithi125 conmon[24927]: cluster 2021-03-24T21:57:02.298 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:02 smithi125 conmon[24927]: 2021-03-24T21:57:02.298 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:02 smithi125 conmon[24927]: 2021 2021-03-24T21:57:02.298 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:02 smithi125 conmon[24927]: - 2021-03-24T21:57:02.298 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:02 smithi125 conmon[24927]: 03 2021-03-24T21:57:02.299 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:02 smithi125 conmon[24927]: - 2021-03-24T21:57:02.299 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:02 smithi125 conmon[24927]: 24 2021-03-24T21:57:02.299 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:02 smithi125 conmon[24927]: T 2021-03-24T21:57:02.299 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:02 smithi125 conmon[24927]: 21 2021-03-24T21:57:02.300 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:02 smithi125 conmon[24927]: : 2021-03-24T21:57:02.300 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:02 smithi125 conmon[24927]: 57:00.762395+0000 2021-03-24T21:57:02.300 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:02 smithi125 conmon[24927]: mgr.y (mgr.14144) 391 : cluster [DBG] pgmap v539: 158 pgs: 158 active+clean; 156 MiB data, 512 MiB used, 536 GiB / 536 GiB avail; 15 MiB/s rd, 2.5 MiB/s wr, 25 op/s 2021-03-24T21:57:02.300 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:02 smithi125 conmon[24927]: audit 2021-03-24T21:57:02.300 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:02 smithi125 conmon[24927]: 2021-03-24T21:57:01 2021-03-24T21:57:02.301 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:02 smithi125 conmon[24927]: .167481+0000 mon.a (mon.0) 880 : audit 2021-03-24T21:57:02.301 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:02 smithi125 conmon[24927]: [INF] from='client.? v1:172.21.15.125:0/2276604061' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 0.7242660621692911}]': finished 2021-03-24T21:57:02.301 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:02 smithi125 conmon[24927]: cluster 2021-03-24T21:57:02.301 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:02 smithi125 conmon[24927]: 2021-03-24T21:57 2021-03-24T21:57:02.302 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:02 smithi125 conmon[24927]: :01.167609+0000 mon.a (mon.0) 881 : 2021-03-24T21:57:02.302 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:02 smithi125 conmon[24927]: cluster [DBG] osdmap e252: 8 total, 8 up, 6 in 2021-03-24T21:57:02.302 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:02 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:57:02.302 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:02 smithi125 conmon[24927]: :57:02.169382+0000 mon.a (mon.0) 882 2021-03-24T21:57:02.303 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:02 smithi125 conmon[24927]: : cluster [DBG] osdmap e253: 8 total, 8 up, 6 in 2021-03-24T21:57:02.339 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:02.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:02.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:57:02.445 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:02.493 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:02.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:02.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:57:02.599 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:02.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:02 smithi142 conmon[20125]: cluster 2021-03-24T21:57:00.762395+0000 mgr.y 2021-03-24T21:57:02.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:02 smithi142 conmon[20125]: (mgr.14144) 391 : cluster [DBG] pgmap v539: 158 pgs: 158 active+clean; 156 MiB data, 512 MiB used, 536 GiB / 536 GiB avail; 15 MiB/s rd, 2.5 MiB/s wr, 25 op/s 2021-03-24T21:57:02.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:02 smithi142 conmon[20125]: audit 2021-03-24T21:57:01.167481+0000 mon.a (mon.0) 880 : audit [INF] from='client.? v1:172.21.15.125:0/2276604061' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 0.7242660621692911}]': finished 2021-03-24T21:57:02.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:02 smithi142 conmon[20125]: cluster 2021-03-24T21:57:01.167609+0000 mon.a (mon.0) 881 2021-03-24T21:57:02.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:02 smithi142 conmon[20125]: : cluster [DBG] osdmap e252: 8 total, 8 up, 6 in 2021-03-24T21:57:02.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:02 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:57:02.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:02 smithi142 conmon[20125]: 57:02.169382+0000 mon.a 2021-03-24T21:57:02.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:02 smithi142 conmon[20125]: (mon.0) 882 : cluster [DBG] 2021-03-24T21:57:02.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:02 smithi142 conmon[20125]: osdmap e253: 8 total, 8 up, 6 in 2021-03-24T21:57:02.657 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:02.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:02.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:02.764 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:02.818 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:02.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:02.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:57:02.925 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:02.982 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:02.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:02.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:57:03.089 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:03.139 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:03.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:03.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:57:03.176 INFO:tasks.rados.rados.0.smithi125.stdout:788: done (5 left) 2021-03-24T21:57:03.177 INFO:tasks.rados.rados.0.smithi125.stdout:789: rollback oid 1 current snap is 76 2021-03-24T21:57:03.177 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 1 to 75 2021-03-24T21:57:03.177 INFO:tasks.rados.rados.0.smithi125.stdout:790: read oid 45 snap -1 2021-03-24T21:57:03.178 INFO:tasks.rados.rados.0.smithi125.stdout:790: expect (ObjNum 150 snap 30 seq_num 150) 2021-03-24T21:57:03.178 INFO:tasks.rados.rados.0.smithi125.stdout:791: write oid 17 current snap is 76 2021-03-24T21:57:03.179 INFO:tasks.rados.rados.0.smithi125.stdout:791: seq_num 310 ranges {410170=781310,1940712=656106,3163363=46807} 2021-03-24T21:57:03.179 INFO:tasks.rados.rados.0.smithi125.stdout:791: writing smithi12556470-17 from 410170 to 1191480 tid 1 2021-03-24T21:57:03.188 INFO:tasks.rados.rados.0.smithi125.stdout:791: writing smithi12556470-17 from 1940712 to 2596818 tid 2 2021-03-24T21:57:03.192 INFO:tasks.rados.rados.0.smithi125.stdout:791: writing smithi12556470-17 from 3163363 to 3210170 tid 3 2021-03-24T21:57:03.192 INFO:tasks.rados.rados.0.smithi125.stdout:792: read oid 37 snap -1 2021-03-24T21:57:03.192 INFO:tasks.rados.rados.0.smithi125.stdout:792: expect (ObjNum 229 snap 50 seq_num 229) 2021-03-24T21:57:03.193 INFO:tasks.rados.rados.0.smithi125.stdout:793: delete oid 30 current snap is 76 2021-03-24T21:57:03.196 INFO:tasks.rados.rados.0.smithi125.stdout:784: expect (ObjNum 255 snap 59 seq_num 255) 2021-03-24T21:57:03.245 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:03.306 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:03.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:03.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:57:03.341 INFO:tasks.rados.rados.0.smithi125.stdout:785: finishing write tid 1 to smithi12556470-39 2021-03-24T21:57:03.342 INFO:tasks.rados.rados.0.smithi125.stdout:784: done (9 left) 2021-03-24T21:57:03.342 INFO:tasks.rados.rados.0.smithi125.stdout:793: done (8 left) 2021-03-24T21:57:03.342 INFO:tasks.rados.rados.0.smithi125.stdout:794: snap_create 2021-03-24T21:57:03.342 INFO:tasks.rados.rados.0.smithi125.stdout:787: expect (ObjNum 150 snap 30 seq_num 150) 2021-03-24T21:57:03.413 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:03.455 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:03.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:03.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:57:03.467 INFO:tasks.rados.rados.0.smithi125.stdout:785: finishing write tid 2 to smithi12556470-39 2021-03-24T21:57:03.467 INFO:tasks.rados.rados.0.smithi125.stdout:785: finishing write tid 3 to smithi12556470-39 2021-03-24T21:57:03.468 INFO:tasks.rados.rados.0.smithi125.stdout:785: finishing write tid 4 to smithi12556470-39 2021-03-24T21:57:03.468 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 39 v 260 (ObjNum 309 snap 76 seq_num 309) dirty exists 2021-03-24T21:57:03.468 INFO:tasks.rados.rados.0.smithi125.stdout:785: left oid 39 (ObjNum 309 snap 76 seq_num 309) 2021-03-24T21:57:03.468 INFO:tasks.rados.rados.0.smithi125.stdout:786: expect (ObjNum 281 snap 70 seq_num 281) 2021-03-24T21:57:03.561 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:03.593 INFO:tasks.rados.rados.0.smithi125.stdout:790: expect (ObjNum 150 snap 30 seq_num 150) 2021-03-24T21:57:03.625 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:03.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:03.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:57:03.714 INFO:tasks.rados.rados.0.smithi125.stdout:789: finishing rollback tid 0 to smithi12556470-1 2021-03-24T21:57:03.714 INFO:tasks.rados.rados.0.smithi125.stdout:789: finishing rollback tid 1 to smithi12556470-1 2021-03-24T21:57:03.714 INFO:tasks.rados.rados.0.smithi125.stdout:791: finishing write tid 1 to smithi12556470-17 2021-03-24T21:57:03.714 INFO:tasks.rados.rados.0.smithi125.stdout:789: finishing rollback tid 2 to smithi12556470-1 2021-03-24T21:57:03.715 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 1 v 254 (ObjNum 276 snap 68 seq_num 276) dirty exists 2021-03-24T21:57:03.715 INFO:tasks.rados.rados.0.smithi125.stdout:791: finishing write tid 2 to smithi12556470-17 2021-03-24T21:57:03.715 INFO:tasks.rados.rados.0.smithi125.stdout:791: finishing write tid 3 to smithi12556470-17 2021-03-24T21:57:03.715 INFO:tasks.rados.rados.0.smithi125.stdout:791: finishing write tid 4 to smithi12556470-17 2021-03-24T21:57:03.716 INFO:tasks.rados.rados.0.smithi125.stdout:791: finishing write tid 5 to smithi12556470-17 2021-03-24T21:57:03.716 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 17 v 235 (ObjNum 310 snap 76 seq_num 310) dirty exists 2021-03-24T21:57:03.716 INFO:tasks.rados.rados.0.smithi125.stdout:791: left oid 17 (ObjNum 310 snap 76 seq_num 310) 2021-03-24T21:57:03.716 INFO:tasks.rados.rados.0.smithi125.stdout:781: finishing rollback tid 1 to smithi12556470-43 2021-03-24T21:57:03.717 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 43 v 0 (ObjNum 1116548352 snap 21916 seq_num 1116680096) dirty dne 2021-03-24T21:57:03.717 INFO:tasks.rados.rados.0.smithi125.stdout:792: expect (ObjNum 229 snap 50 seq_num 229) 2021-03-24T21:57:03.732 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:03.788 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:03.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:03.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:03.893 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:03.948 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:03.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:03.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:04.054 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:04.110 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:04.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:04.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:04.197 INFO:tasks.rados.rados.0.smithi125.stdout:781: done (8 left) 2021-03-24T21:57:04.198 INFO:tasks.rados.rados.0.smithi125.stdout:785: done (7 left) 2021-03-24T21:57:04.198 INFO:tasks.rados.rados.0.smithi125.stdout:786: done (6 left) 2021-03-24T21:57:04.198 INFO:tasks.rados.rados.0.smithi125.stdout:787: done (5 left) 2021-03-24T21:57:04.199 INFO:tasks.rados.rados.0.smithi125.stdout:789: done (4 left) 2021-03-24T21:57:04.199 INFO:tasks.rados.rados.0.smithi125.stdout:790: done (3 left) 2021-03-24T21:57:04.199 INFO:tasks.rados.rados.0.smithi125.stdout:791: done (2 left) 2021-03-24T21:57:04.199 INFO:tasks.rados.rados.0.smithi125.stdout:792: done (1 left) 2021-03-24T21:57:04.199 INFO:tasks.rados.rados.0.smithi125.stdout:794: done (0 left) 2021-03-24T21:57:04.200 INFO:tasks.rados.rados.0.smithi125.stdout:795: read oid 19 snap -1 2021-03-24T21:57:04.200 INFO:tasks.rados.rados.0.smithi125.stdout:795: expect (ObjNum 244 snap 57 seq_num 244) 2021-03-24T21:57:04.200 INFO:tasks.rados.rados.0.smithi125.stdout:796: read oid 42 snap -1 2021-03-24T21:57:04.200 INFO:tasks.rados.rados.0.smithi125.stdout:796: expect deleted 2021-03-24T21:57:04.201 INFO:tasks.rados.rados.0.smithi125.stdout:797: read oid 26 snap 75 2021-03-24T21:57:04.201 INFO:tasks.rados.rados.0.smithi125.stdout:797: expect (ObjNum 294 snap 74 seq_num 294) 2021-03-24T21:57:04.201 INFO:tasks.rados.rados.0.smithi125.stdout:798: rollback oid 3 current snap is 77 2021-03-24T21:57:04.201 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 3 to 70 2021-03-24T21:57:04.201 INFO:tasks.rados.rados.0.smithi125.stdout:799: copy_from oid 33 from oid 43 current snap is 77 2021-03-24T21:57:04.202 INFO:tasks.rados.rados.0.smithi125.stdout:800: copy_from oid 36 from oid 38 current snap is 77 2021-03-24T21:57:04.202 INFO:tasks.rados.rados.0.smithi125.stdout:801: write oid 17 current snap is 77 2021-03-24T21:57:04.202 INFO:tasks.rados.rados.0.smithi125.stdout:801: seq_num 312 ranges {443784=648161,1657676=659775,2818209=425575} 2021-03-24T21:57:04.203 INFO:teuthology.orchestra.run.smithi125.stderr:nodeep-scrub is set 2021-03-24T21:57:04.204 INFO:tasks.rados.rados.0.smithi125.stdout:801: writing smithi12556470-17 from 443784 to 1091945 tid 1 2021-03-24T21:57:04.212 INFO:tasks.rados.rados.0.smithi125.stdout:801: writing smithi12556470-17 from 1657676 to 2317451 tid 2 2021-03-24T21:57:04.217 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:04.218 INFO:tasks.rados.rados.0.smithi125.stdout:801: writing smithi12556470-17 from 2818209 to 3243784 tid 3 2021-03-24T21:57:04.218 INFO:tasks.rados.rados.0.smithi125.stdout:802: snap_remove snap 66 2021-03-24T21:57:04.261 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:04.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:04.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:57:04.368 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:04.377 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:04 smithi125 conmon[19936]: cluster 2021-03-24T21:57: 2021-03-24T21:57:04.378 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:04 smithi125 conmon[19936]: 02.762999+0000 mgr.y (mgr.14144) 392 : cluster [DBG] pgmap v542: 158 pgs: 158 active+clean; 156 MiB data, 509 MiB used, 536 GiB / 536 GiB avail; 17 MiB/s rd, 2.3 MiB/s wr, 26 op/s 2021-03-24T21:57:04.378 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:04 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:57:04.378 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:04 smithi125 conmon[19936]: -24T21:57:03.167811+0000 mon.a (mon.0 2021-03-24T21:57:04.378 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:04 smithi125 conmon[19936]: ) 883 : cluster [DBG] 2021-03-24T21:57:04.379 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:04 smithi125 conmon[19936]: osdmap e254: 8 total, 8 up, 6 in 2021-03-24T21:57:04.379 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:04 smithi125 conmon[19936]: audit 2021-03-24T21:57:04.379 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:04 smithi125 conmon[19936]: 2021-03-24 2021-03-24T21:57:04.379 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:04 smithi125 conmon[19936]: T21 2021-03-24T21:57:04.379 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:04 smithi125 conmon[19936]: :57:04.154181 2021-03-24T21:57:04.380 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:04 smithi125 conmon[19936]: +0000 mon.c (mon. 2021-03-24T21:57:04.380 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:04 smithi125 conmon[19936]: 2) 39 : audit [INF] 2021-03-24T21:57:04.380 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:04 smithi125 conmon[19936]: from='client.? v1:172.21.15.125:0/618445057' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:57:04.380 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:04 smithi125 conmon[19936]: audit 2021-03-24T21:57:04.381 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:04 smithi125 conmon[19936]: 2021-03 2021-03-24T21:57:04.381 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:04 smithi125 conmon[19936]: -24T21:57 2021-03-24T21:57:04.381 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:04 smithi125 conmon[19936]: :04.154728+0000 2021-03-24T21:57:04.381 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:04 smithi125 conmon[19936]: mon.a (mon.0) 2021-03-24T21:57:04.381 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:04 smithi125 conmon[19936]: 884 : audit [INF] from='client.34949 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:57:04.382 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:04 smithi125 conmon[19936]: 2021-03-24T21:57:04.382 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:04 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:57:04.382 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:04 smithi125 conmon[24927]: 57:02.762999+0000 mgr.y (mgr.14144) 392 : cluster [DBG] pgmap v542: 158 pgs: 158 active+clean; 156 MiB data, 509 MiB used, 536 GiB / 536 GiB avail; 17 MiB/s rd, 2.3 MiB/s wr, 26 op/s 2021-03-24T21:57:04.383 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:04 smithi125 conmon[24927]: 2021-03-24T21:57:04.383 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:04 smithi125 conmon[24927]: cluster 2021-03-24T21:57:04.383 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:04 smithi125 conmon[24927]: 2021-03-24T 2021-03-24T21:57:04.383 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:04 smithi125 conmon[24927]: 21:57:03.167811+0000 mon.a ( 2021-03-24T21:57:04.383 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:04 smithi125 conmon[24927]: mon.0) 883 : cluster [DBG] osdmap e254: 8 total, 8 up, 6 in 2021-03-24T21:57:04.384 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:04 smithi125 conmon[24927]: audit 2021-03-24T21:57:04.384 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:04 smithi125 conmon[24927]: 2021-03-24T21:57 2021-03-24T21:57:04.384 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:04 smithi125 conmon[24927]: :04.154181+0000 mon.c (mon.2) 39 2021-03-24T21:57:04.384 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:04 smithi125 conmon[24927]: : audit [INF] from='client.? v1:172.21.15.125:0/618445057' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:57:04.384 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:04 smithi125 conmon[24927]: audit 2021-03-24T 2021-03-24T21:57:04.385 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:04 smithi125 conmon[24927]: 21:57:04.154728+0000 mon.a 2021-03-24T21:57:04.385 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:04 smithi125 conmon[24927]: (mon.0) 884 : audit [INF] from='client.34949 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:57:04.413 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:04.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:04.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:57:04.519 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:04.570 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:04.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:04.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:57:04.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:04 smithi142 conmon[20125]: cluster 2021-03-24T21:57:02.762999+0000 2021-03-24T21:57:04.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:04 smithi142 conmon[20125]: mgr.y (mgr.14144) 392 : cluster [DBG] pgmap v542: 158 pgs: 158 active+clean; 156 MiB data, 509 MiB used, 536 GiB / 536 GiB avail; 17 MiB/s rd, 2.3 MiB/s wr, 26 op/s 2021-03-24T21:57:04.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:04 smithi142 conmon[20125]: cluster 2021-03-24T21:57:03.167811+0000 mon.a (mon. 2021-03-24T21:57:04.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:04 smithi142 conmon[20125]: 0) 883 : cluster [DBG] osdmap e254: 8 total, 8 up, 6 in 2021-03-24T21:57:04.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:04 smithi142 conmon[20125]: audit 2021-03-24T21:57:04.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:04 smithi142 conmon[20125]: 2021-03-24T21:57:04.154181+0000 mon.c (mon.2) 39 2021-03-24T21:57:04.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:04 smithi142 conmon[20125]: : audit [INF] from='client.? v1:172.21.15.125:0/618445057' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:57:04.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:04 smithi142 conmon[20125]: audit 2021-03- 2021-03-24T21:57:04.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:04 smithi142 conmon[20125]: 24T21:57:04.154728+0000 mon.a (mon. 2021-03-24T21:57:04.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:04 smithi142 conmon[20125]: 0) 884 : audit [INF] from='client.34949 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:57:04.676 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:04.738 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:04.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:04.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:57:04.844 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:04.904 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:04.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:04.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:57:05.011 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:05.069 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:05.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:05.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:57:05.157 INFO:tasks.rados.rados.0.smithi125.stdout:802: done (7 left) 2021-03-24T21:57:05.157 INFO:tasks.rados.rados.0.smithi125.stdout:803: write oid 21 current snap is 77 2021-03-24T21:57:05.157 INFO:tasks.rados.rados.0.smithi125.stdout:803: seq_num 313 ranges {460591=723410,1716158=403433,2729279=440140} 2021-03-24T21:57:05.157 INFO:tasks.rados.rados.0.smithi125.stdout:803: writing smithi12556470-21 from 460591 to 1184001 tid 1 2021-03-24T21:57:05.162 INFO:tasks.rados.rados.0.smithi125.stdout:803: writing smithi12556470-21 from 1716158 to 2119591 tid 2 2021-03-24T21:57:05.169 INFO:tasks.rados.rados.0.smithi125.stdout:803: writing smithi12556470-21 from 2729279 to 3169419 tid 3 2021-03-24T21:57:05.170 INFO:tasks.rados.rados.0.smithi125.stdout:804: read oid 11 snap 74 2021-03-24T21:57:05.170 INFO:tasks.rados.rados.0.smithi125.stdout:804: expect (ObjNum 244 snap 57 seq_num 244) 2021-03-24T21:57:05.170 INFO:tasks.rados.rados.0.smithi125.stdout:805: snap_create 2021-03-24T21:57:05.171 INFO:tasks.rados.rados.0.smithi125.stdout:798: finishing rollback tid 1 to smithi12556470-3 2021-03-24T21:57:05.171 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 3 v 107 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-03-24T21:57:05.171 INFO:tasks.rados.rados.0.smithi125.stdout:795: expect (ObjNum 244 snap 57 seq_num 244) 2021-03-24T21:57:05.175 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:05.227 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:05.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:05.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:57:05.255 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T21:57:05.256 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T21:57:05.256 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T21:57:05.256 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T21:57:05.262 INFO:tasks.rados.rados.0.smithi125.stdout:800: finishing copy_from to smithi12556470-36 2021-03-24T21:57:05.263 INFO:tasks.rados.rados.0.smithi125.stdout:800: got expected ENOENT (src dne) 2021-03-24T21:57:05.263 INFO:tasks.rados.rados.0.smithi125.stdout:800: finishing copy_from racing read to smithi12556470-36 2021-03-24T21:57:05.263 INFO:tasks.rados.rados.0.smithi125.stdout:797: expect (ObjNum 294 snap 74 seq_num 294) 2021-03-24T21:57:05.273 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T21:57:05.287 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[19936]: cluster 2021-03-24T21:57:04.185443+0000 mon.a (mon.0) 885 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:57:05.287 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[19936]: audit 2021-03-24T21:57 2021-03-24T21:57:05.287 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[19936]: :04.193506+0000 mon.a (mon.0) 886 : audit [INF] from='client.34949 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T21:57:05.287 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[19936]: cluster 2021-03-24T21:57:04.193817+0000 mon.a (mon.0) 887 : cluster [DBG] osdmap e255: 8 total, 8 up, 6 in 2021-03-24T21:57:05.288 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[19936]: audit 2021 2021-03-24T21:57:05.288 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[19936]: -03-24T21:57:04.769261+0000 mon.a (mon.0) 888 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T21:57:05.288 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[19936]: audit 2021-03-24T21: 2021-03-24T21:57:05.288 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[19936]: 57:04.769843+0000 mon.a (mon.0) 889 : 2021-03-24T21:57:05.289 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[19936]: audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T21:57:05.289 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[19936]: audit 2021-03-24T21:57:05.098084+0000 mon.a (mon.0) 890 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.4", "id": [5, 0]}]: dispatch 2021-03-24T21:57:05.289 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[19936]: audit 2021-03-24T21:57:05.098693+0000 mon.a ( 2021-03-24T21:57:05.289 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[19936]: mon.0) 891 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.6", "id": [7, 4]}]: dispatch 2021-03-24T21:57:05.290 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[19936]: audit 2021-03-24T21:57:05.099063+0000 2021-03-24T21:57:05.290 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[19936]: mon.a (mon.0) 892 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.f", "id": [7, 4]}]: dispatch 2021-03-24T21:57:05.290 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[19936]: audit 2021-03-24T21:57:05.099445 2021-03-24T21:57:05.290 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[19936]: +0000 mon.a (mon.0) 893 : audit [INF] 2021-03-24T21:57:05.290 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.10", "id": [7, 3, 2, 5]}]: dispatch 2021-03-24T21:57:05.291 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[19936]: audit 2021-03-24T21:57:05.291 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[19936]: 2021-03-24T21:57:05.139228+0000 mon.a (mon.0) 2021-03-24T21:57:05.291 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[19936]: 894 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.4", "id": [5, 0]}]': finished 2021-03-24T21:57:05.291 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[19936]: audit 2021-03-24T21:57 2021-03-24T21:57:05.291 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[19936]: :05.141618+0000 mon.a (mon.0) 895 : audit 2021-03-24T21:57:05.292 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[19936]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.6", "id": [7, 4]}]': finished 2021-03-24T21:57:05.292 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[19936]: audit 2021-03-24T21:57:05.141779 2021-03-24T21:57:05.292 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[19936]: +0000 mon.a (mon.0) 896 : audit [INF] 2021-03-24T21:57:05.292 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.f", "id": [7, 4]}]': finished 2021-03-24T21:57:05.292 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[19936]: audit 2021-03-24T21:57:05.293 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[19936]: 2021-03-24T21:57:05.141917+0000 mon.a (mon.0 2021-03-24T21:57:05.293 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[19936]: ) 897 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.10", "id": [7, 3, 2, 5]}]': finished 2021-03-24T21:57:05.293 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[19936]: cluster 2021-03-24T21:57:05. 2021-03-24T21:57:05.293 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[19936]: 142051+0000 mon.a (mon.0) 898 : cluster [DBG] osdmap e256: 8 total, 8 up, 6 in 2021-03-24T21:57:05.293 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[19936]: audit 2021 2021-03-24T21:57:05.294 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[19936]: -03-24T21:57:05.161871+0000 mon.a ( 2021-03-24T21:57:05.294 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[19936]: mon.0) 899 : audit [DBG] from='client.? v1:172.21.15.125:0/1759777245' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:57:05.295 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[24927]: cluster 2021-03-24T21:57:04.185443+0000 mon.a (mon 2021-03-24T21:57:05.295 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[24927]: .0) 885 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:57:05.295 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[24927]: audit 2021-03-24T21:57:05.295 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[24927]: 2021-03-24T21:57:04.193506+0000 mon.a (mon.0) 886 : audit [INF] from='client.34949 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T21:57:05.296 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:57:05.296 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[24927]: 03-24T21:57:04.193817+0000 mon.a (mon.0 2021-03-24T21:57:05.296 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[24927]: ) 887 : cluster [DBG] osdmap e255: 8 total, 8 up, 6 in 2021-03-24T21:57:05.296 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[24927]: audit 2021 2021-03-24T21:57:05.296 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[24927]: -03-24T21:57:04.769261+0000 mon.a (mon.0 2021-03-24T21:57:05.297 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[24927]: ) 888 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T21:57:05.297 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[24927]: audit 2021-03-24 2021-03-24T21:57:05.297 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[24927]: T21:57:04.769843+0000 mon.a (mon.0 2021-03-24T21:57:05.297 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[24927]: ) 889 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T21:57:05.298 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[24927]: audit 2021-03-24T21:57:05.298 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[24927]: 2021-03-24T21: 2021-03-24T21:57:05.298 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[24927]: 57:05.098084+0000 mon.a (mon.0) 2021-03-24T21:57:05.298 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[24927]: 890 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.4", "id": [5, 0]}]: dispatch 2021-03-24T21:57:05.298 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T21:57:05.299 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[24927]: 24T21:57:05.098693+0000 mon.a (mon. 2021-03-24T21:57:05.299 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[24927]: 0) 891 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.6", "id": [7, 4]}]: dispatch 2021-03-24T21:57:05.299 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T21:57:05.299 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[24927]: 24T21:57:05.099063+0000 mon.a (mon.0) 892 : audit 2021-03-24T21:57:05.299 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[24927]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.f", "id": [7, 4]}]: dispatch 2021-03-24T21:57:05.300 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[24927]: audit 2021-03 2021-03-24T21:57:05.300 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[24927]: -24T21:57:05.099445+0000 mon.a ( 2021-03-24T21:57:05.300 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[24927]: mon.0) 893 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.10", "id": [7, 3, 2, 5]}]: dispatch 2021-03-24T21:57:05.300 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[24927]: audit 2021- 2021-03-24T21:57:05.301 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[24927]: 03-24T21:57:05.139228+0000 mon.a ( 2021-03-24T21:57:05.301 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[24927]: mon.0) 894 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.4", "id": [5, 0]}]': finished 2021-03-24T21:57:05.301 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[24927]: audit 2021-03-24T21:57:05.301 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[24927]: 2021-03-24T 2021-03-24T21:57:05.301 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[24927]: 21:57:05.141618+0000 mon.a (mon.0 2021-03-24T21:57:05.302 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[24927]: ) 895 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.6", "id": [7, 4]}]': finished 2021-03-24T21:57:05.302 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[24927]: audit 2021-03-24T21:57: 2021-03-24T21:57:05.302 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[24927]: 05.141779+0000 mon.a (mon.0) 896 : audit [INF] 2021-03-24T21:57:05.302 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[24927]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.f", "id": [7, 4]}]': finished 2021-03-24T21:57:05.303 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[24927]: audit 2021-03-24T21:57: 2021-03-24T21:57:05.303 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[24927]: 05.141917+0000 mon.a (mon.0) 897 : audit 2021-03-24T21:57:05.303 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[24927]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.10", "id": [7, 3, 2, 5]}]': finished 2021-03-24T21:57:05.303 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[24927]: cluster 2021-03-24T21:57:05.303 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[24927]: 2021-03-24T21:57:05 2021-03-24T21:57:05.304 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[24927]: .142051+0000 mon.a (mon.0) 898 : cluster 2021-03-24T21:57:05.304 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[24927]: [DBG] osdmap e256: 8 total, 8 up, 6 in 2021-03-24T21:57:05.304 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T21:57:05.304 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[24927]: 24T21:57:05.161871+0000 mon.a (mon 2021-03-24T21:57:05.304 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:05 smithi125 conmon[24927]: .0) 899 : audit [DBG] from='client.? v1:172.21.15.125:0/1759777245' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:57:05.305 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T21:57:05.322 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T21:57:05.333 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:05.346 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T21:57:05.347 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-24T21:57:05.348 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-24T21:57:05.348 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-24T21:57:05.348 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-24T21:57:05.348 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-24T21:57:05.349 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-24T21:57:05.349 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-24T21:57:05.349 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-24T21:57:05.372 INFO:tasks.rados.rados.0.smithi125.stdout:801: finishing write tid 1 to smithi12556470-17 2021-03-24T21:57:05.372 INFO:tasks.rados.rados.0.smithi125.stdout:801: finishing write tid 2 to smithi12556470-17 2021-03-24T21:57:05.373 INFO:tasks.rados.rados.0.smithi125.stdout:801: finishing write tid 3 to smithi12556470-17 2021-03-24T21:57:05.373 INFO:tasks.rados.rados.0.smithi125.stdout:801: finishing write tid 4 to smithi12556470-17 2021-03-24T21:57:05.373 INFO:tasks.rados.rados.0.smithi125.stdout:801: finishing write tid 5 to smithi12556470-17 2021-03-24T21:57:05.373 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 17 v 240 (ObjNum 312 snap 77 seq_num 312) dirty exists 2021-03-24T21:57:05.374 INFO:tasks.rados.rados.0.smithi125.stdout:801: left oid 17 (ObjNum 312 snap 77 seq_num 312) 2021-03-24T21:57:05.374 INFO:tasks.rados.rados.0.smithi125.stdout:799: finishing copy_from to smithi12556470-33 2021-03-24T21:57:05.374 INFO:tasks.rados.rados.0.smithi125.stdout:799: got expected ENOENT (src dne) 2021-03-24T21:57:05.374 INFO:tasks.rados.rados.0.smithi125.stdout:799: finishing copy_from racing read to smithi12556470-33 2021-03-24T21:57:05.375 INFO:tasks.rados.rados.0.smithi125.stdout:803: finishing write tid 1 to smithi12556470-21 2021-03-24T21:57:05.375 INFO:tasks.rados.rados.0.smithi125.stdout:803: finishing write tid 2 to smithi12556470-21 2021-03-24T21:57:05.375 INFO:tasks.rados.rados.0.smithi125.stdout:803: finishing write tid 3 to smithi12556470-21 2021-03-24T21:57:05.375 INFO:tasks.rados.rados.0.smithi125.stdout:803: finishing write tid 4 to smithi12556470-21 2021-03-24T21:57:05.375 INFO:tasks.rados.rados.0.smithi125.stdout:803: finishing write tid 5 to smithi12556470-21 2021-03-24T21:57:05.376 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 21 v 206 (ObjNum 313 snap 77 seq_num 313) dirty exists 2021-03-24T21:57:05.376 INFO:tasks.rados.rados.0.smithi125.stdout:803: left oid 21 (ObjNum 313 snap 77 seq_num 313) 2021-03-24T21:57:05.376 INFO:tasks.rados.rados.0.smithi125.stdout:804: expect (ObjNum 244 snap 57 seq_num 244) 2021-03-24T21:57:05.390 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:05.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:05.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:57:05.496 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:05.510 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[33921]: debug 2021-03-24T21:57:05.510 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[33921]: 2021-03-24T21:57:05.143+0000 7fc7b9424700 -1 osd.7 pg_epoch: 256 pg[7.19( v 255'240 (0'0,255'240] local-lis/les=204/205 n=6 ec=198/99 lis/c=204/204 les/c/f=205/205/0 sis=204) [7,2,5] r=0 lpr=204 crt=255'240 lcod 255'239 mlcod 255'239 active+clean+snaptrim trimq=[82~1] ps=[7e~1,8c~1,8f~1]] removing snap head 2021-03-24T21:57:05.510 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[33921]: 2021-03-24T21:57:05.511 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[20125]: cluster 2021-03-24T21:57:04.185443+0000 2021-03-24T21:57:05.511 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[20125]: mon.a (mon.0) 885 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:57:05.511 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[20125]: audit 2021-03-24T21:57:04.193506+0000 mon.a ( 2021-03-24T21:57:05.512 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[20125]: mon.0) 886 : audit [INF] from='client.34949 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T21:57:05.512 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[20125]: cluster 2021-03-24T21 2021-03-24T21:57:05.512 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[20125]: :57:04.193817+0000 mon.a (mon. 2021-03-24T21:57:05.512 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[20125]: 0) 887 : cluster [DBG] osdmap e255: 8 total, 8 up, 6 in 2021-03-24T21:57:05.513 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:57:05.513 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[20125]: 03-24T21:57:04.769261+0000 2021-03-24T21:57:05.513 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[20125]: mon.a (mon.0) 888 : audit 2021-03-24T21:57:05.513 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[20125]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T21:57:05.514 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[20125]: audit 2021-03-24T21 2021-03-24T21:57:05.514 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[20125]: :57:04.769843+0000 mon.a (mon. 2021-03-24T21:57:05.514 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[20125]: 0) 889 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T21:57:05.514 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[20125]: audit 2021 2021-03-24T21:57:05.514 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[20125]: -03-24T21:57:05. 2021-03-24T21:57:05.515 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[20125]: 098084+0000 mon.a (mon.0) 890 : 2021-03-24T21:57:05.515 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[20125]: audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.4", "id": [5, 0]}]: dispatch 2021-03-24T21:57:05.515 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:57:05.515 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[20125]: 03-24T21:57:05. 2021-03-24T21:57:05.516 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[20125]: 098693+0000 mon.a (mon.0) 891 : 2021-03-24T21:57:05.516 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[20125]: audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.6", "id": [7, 4]}]: dispatch 2021-03-24T21:57:05.516 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[20125]: audit 2021-03-24T21:57:05.516 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[20125]: 2021-03- 2021-03-24T21:57:05.517 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[20125]: 24T21:57:05.099063+0000 mon.a 2021-03-24T21:57:05.517 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[20125]: (mon.0) 892 : audit [INF] 2021-03-24T21:57:05.517 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.f", "id": [7, 4]}]: dispatch 2021-03-24T21:57:05.517 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[20125]: audit 2021-03-24T 2021-03-24T21:57:05.517 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[20125]: 21:57:05.099445+0000 mon.a ( 2021-03-24T21:57:05.518 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[20125]: mon.0) 893 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.10", "id": [7, 3, 2, 5]}]: dispatch 2021-03-24T21:57:05.518 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[20125]: audit 2021-03-24T21:57:05.518 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[20125]: 2021-03-24T21:57:05. 2021-03-24T21:57:05.518 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[20125]: 139228+0000 mon.a (mon.0) 894 : audit 2021-03-24T21:57:05.518 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[20125]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.4", "id": [5, 0]}]': finished 2021-03-24T21:57:05.519 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[20125]: audit 2021-03-24T21:57:05.141618+0000 mon.a (mon. 2021-03-24T21:57:05.519 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[20125]: 0) 895 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.6", "id": [7, 4]}]': finished 2021-03-24T21:57:05.519 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[20125]: audit 2021 2021-03-24T21:57:05.519 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[20125]: -03-24T21:57:05. 2021-03-24T21:57:05.520 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[20125]: 141779+0000 mon.a (mon.0) 896 : 2021-03-24T21:57:05.520 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[20125]: audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.f", "id": [7, 4]}]': finished 2021-03-24T21:57:05.520 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[20125]: audit 2021-03-24 2021-03-24T21:57:05.520 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[20125]: T21:57:05.141917+0000 mon.a 2021-03-24T21:57:05.520 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[20125]: (mon.0) 897 : audit [INF] 2021-03-24T21:57:05.521 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.10", "id": [7, 3, 2, 5]}]': finished 2021-03-24T21:57:05.521 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[20125]: cluster 2021-03-24T21:57:05.521 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[20125]: 2021-03-24T21:57:05 2021-03-24T21:57:05.521 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[20125]: .142051+0000 mon.a (mon.0) 898 2021-03-24T21:57:05.521 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[20125]: : cluster [DBG] osdmap e256: 8 total, 8 up, 6 in 2021-03-24T21:57:05.522 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[20125]: audit 2021-03 2021-03-24T21:57:05.522 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[20125]: -24T21:57:05.161871+0000 2021-03-24T21:57:05.522 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[20125]: mon.a (mon.0) 899 : audit [DBG] 2021-03-24T21:57:05.522 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:05 smithi142 conmon[20125]: from='client.? v1:172.21.15.125:0/1759777245' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:57:05.554 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:05.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:05.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:57:05.660 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:05.687 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-24T21:57:05.687 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T21:57:05.708 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:05.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:05.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:57:05.813 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:05.867 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:05.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:05.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:57:05.974 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:06.035 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:06.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:06.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:57:06.141 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:06.144 INFO:tasks.rados.rados.0.smithi125.stdout:795: done (9 left) 2021-03-24T21:57:06.145 INFO:tasks.rados.rados.0.smithi125.stdout:796: done (8 left) 2021-03-24T21:57:06.145 INFO:tasks.rados.rados.0.smithi125.stdout:797: done (7 left) 2021-03-24T21:57:06.145 INFO:tasks.rados.rados.0.smithi125.stdout:798: done (6 left) 2021-03-24T21:57:06.145 INFO:tasks.rados.rados.0.smithi125.stdout:799: done (5 left) 2021-03-24T21:57:06.146 INFO:tasks.rados.rados.0.smithi125.stdout:800: done (4 left) 2021-03-24T21:57:06.146 INFO:tasks.rados.rados.0.smithi125.stdout:801: done (3 left) 2021-03-24T21:57:06.146 INFO:tasks.rados.rados.0.smithi125.stdout:803: done (2 left) 2021-03-24T21:57:06.146 INFO:tasks.rados.rados.0.smithi125.stdout:804: done (1 left) 2021-03-24T21:57:06.146 INFO:tasks.rados.rados.0.smithi125.stdout:805: done (0 left) 2021-03-24T21:57:06.147 INFO:tasks.rados.rados.0.smithi125.stdout:806: read oid 38 snap -1 2021-03-24T21:57:06.147 INFO:tasks.rados.rados.0.smithi125.stdout:806: expect deleted 2021-03-24T21:57:06.147 INFO:tasks.rados.rados.0.smithi125.stdout:807: snap_remove snap 72 2021-03-24T21:57:06.197 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:06.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:06.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:57:06.302 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:06.364 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:06.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:06.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:57:06.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:06 smithi125 conmon[19936]: cluster 2021-03-24T21:57:04.763827+0000 mgr.y (mgr.14144) 393 : cluster 2021-03-24T21:57:06.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:06 smithi125 conmon[19936]: [DBG] pgmap v545: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 163 MiB data, 521 MiB used, 536 GiB / 536 GiB avail; 18 MiB/s rd, 3.5 MiB/s wr, 31 op/s; 7/1200 objects misplaced (0.583%); 2.8 MiB/s, 11 objects/s recovering 2021-03-24T21:57:06.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:06 smithi125 conmon[19936]: cluster 2021-03-24T21:57:06.142589+0000 mon.a (mon.0) 900 : cluster [DBG] osdmap e257: 8 total, 8 up, 6 in 2021-03-24T21:57:06.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:06 smithi125 conmon[24927]: cluster 2021-03-24T21:57: 2021-03-24T21:57:06.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:06 smithi125 conmon[24927]: 04.763827+0000 mgr.y (mgr.14144) 393 : cluster [DBG] pgmap v545: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 163 MiB data, 521 MiB used, 536 GiB / 536 GiB avail; 18 MiB/s rd, 3.5 MiB/s wr, 31 op/s; 7/1200 objects misplaced (0.583%); 2.8 MiB/s, 11 objects/s recovering 2021-03-24T21:57:06.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:06 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:57:06.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:06 smithi125 conmon[24927]: T21:57:06.142589+0000 mon.a (mon. 2021-03-24T21:57:06.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:06 smithi125 conmon[24927]: 0) 900 : cluster [DBG] osdmap e257: 8 total, 8 up, 6 in 2021-03-24T21:57:06.470 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:06.525 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:06.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:06.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:57:06.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:06 smithi142 conmon[20125]: cluster 2021-03-24T21:57:04.763827+0000 mgr.y (mgr.14144) 393 : cluster 2021-03-24T21:57:06.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:06 smithi142 conmon[20125]: [DBG] pgmap v545: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 163 MiB data, 521 MiB used, 536 GiB / 536 GiB avail; 18 MiB/s rd, 3.5 MiB/s wr, 31 op/s; 7/1200 objects misplaced (0.583%); 2.8 MiB/s, 11 objects/s recovering 2021-03-24T21:57:06.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:06 smithi142 conmon[20125]: cluster 2021-03-24T21:57:06.142589+0000 mon.a (mon.0) 900 : cluster [DBG] osdmap e257: 8 total, 8 up, 6 in 2021-03-24T21:57:06.632 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:06.687 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:06.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:06.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:57:06.793 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:06.841 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:06.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:06.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:57:06.948 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:06.998 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:07.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:07.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:57:07.104 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:07.147 INFO:tasks.rados.rados.0.smithi125.stdout:807: done (1 left) 2021-03-24T21:57:07.147 INFO:tasks.rados.rados.0.smithi125.stdout:808: read oid 24 snap -1 2021-03-24T21:57:07.147 INFO:tasks.rados.rados.0.smithi125.stdout:808: expect deleted 2021-03-24T21:57:07.148 INFO:tasks.rados.rados.0.smithi125.stdout:809: snap_create 2021-03-24T21:57:07.148 INFO:journalctl@ceph.osd.2.smithi125.stdout:Mar 24 21:57:07 smithi125 conmon[34620]: debug 2021-03-24T21:57:07.136+0000 7f47099de700 -1 osd.2 pg_epoch: 258 pg[7.8( v 256'116 (0'0,256'116] local-lis/les=198/199 n=2 ec=99/99 lis/c=198/198 les/c/f=199/199/0 sis=198) [2,7,4] r=0 lpr=198 crt=256'116 lcod 255'114 mlcod 255'114 active+clean+snaptrim trimq=[8d~1] ps=[7e~1,82~1,8c~1,8f~1]] removing snap head 2021-03-24T21:57:07.167 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:07.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:07.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:57:07.272 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:07.315 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd unset noscrub 2021-03-24T21:57:07.319 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:07.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:07.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:57:07.381 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.2 dump_blocked_ops 2021-03-24T21:57:07.424 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:07.485 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:07.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:07.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:57:07.592 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:07.610 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:57:07.648 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:07.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:07.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:07.669 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:57:07.755 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:07.812 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:07.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:07.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:07.872 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 7] out_osds: [6, 1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-03-24T21:57:07.872 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-24T21:57:07.873 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2021-03-24T21:57:07.873 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph pg dump --format=json 2021-03-24T21:57:07.920 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:07.969 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:07.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:07.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:57:08.074 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:08.130 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:08.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:08.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:08.155 INFO:tasks.rados.rados.0.smithi125.stdout:806: done (2 left) 2021-03-24T21:57:08.155 INFO:tasks.rados.rados.0.smithi125.stdout:808: done (1 left) 2021-03-24T21:57:08.155 INFO:tasks.rados.rados.0.smithi125.stdout:809: done (0 left) 2021-03-24T21:57:08.155 INFO:tasks.rados.rados.0.smithi125.stdout:810: rollback oid 25 current snap is 79 2021-03-24T21:57:08.156 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 25 to 75 2021-03-24T21:57:08.156 INFO:tasks.rados.rados.0.smithi125.stdout:811: rollback oid 19 current snap is 79 2021-03-24T21:57:08.156 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 19 to 70 2021-03-24T21:57:08.156 INFO:tasks.rados.rados.0.smithi125.stdout:812: read oid 39 snap 78 2021-03-24T21:57:08.157 INFO:tasks.rados.rados.0.smithi125.stdout:812: expect (ObjNum 309 snap 76 seq_num 309) 2021-03-24T21:57:08.157 INFO:tasks.rados.rados.0.smithi125.stdout:813: rollback oid 37 current snap is 79 2021-03-24T21:57:08.157 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 37 to 76 2021-03-24T21:57:08.157 INFO:tasks.rados.rados.0.smithi125.stdout:814: read oid 16 snap 77 2021-03-24T21:57:08.158 INFO:tasks.rados.rados.0.smithi125.stdout:814: expect (ObjNum 280 snap 70 seq_num 280) 2021-03-24T21:57:08.158 INFO:tasks.rados.rados.0.smithi125.stdout:815: snap_remove snap 74 2021-03-24T21:57:08.237 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:08.301 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:08.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:08.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:08.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:08 smithi142 conmon[20125]: cluster 2021-03-24T21:57:06.765049+0000 mgr.y (mgr.14144) 394 : cluster [DBG] pgmap v548: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 159 MiB data, 504 MiB used, 536 GiB / 536 GiB avail; 16 MiB/s rd, 3.5 MiB/s wr, 28 op/s; 7/1197 objects misplaced (0.585%); 2.8 MiB/s, 11 objects/s recovering 2021-03-24T21:57:08.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:08 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:57:08.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:08 smithi142 conmon[20125]: 21:57:07.144766+0000 mon.a (mon.0) 901 : cluster [DBG] osdmap e258: 8 total, 8 up, 6 in 2021-03-24T21:57:08.407 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:08.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:08 smithi125 conmon[19936]: cluster 2021-03-24T21:57 2021-03-24T21:57:08.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:08 smithi125 conmon[19936]: :06.765049+0000 mgr.y (mgr.14144) 394 : cluster [DBG] pgmap v548: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 159 MiB data, 504 MiB used, 536 GiB / 536 GiB avail; 16 MiB/s rd, 3.5 MiB/s wr, 28 op/s; 7/1197 objects misplaced (0.585%); 2.8 MiB/s, 11 objects/s recovering 2021-03-24T21:57:08.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:08 smithi125 conmon[19936]: cluster 2021-03-24T21:57:08.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:08 smithi125 conmon[19936]: 2021-03-24T21:57:07 2021-03-24T21:57:08.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:08 smithi125 conmon[19936]: .144766+0000 mon.a (mon.0) 901 2021-03-24T21:57:08.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:08 smithi125 conmon[19936]: : cluster [DBG] osdmap e258: 8 total, 8 up, 6 in 2021-03-24T21:57:08.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:08 smithi125 conmon[24927]: cluster 2021-03-24T21:57 2021-03-24T21:57:08.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:08 smithi125 conmon[24927]: :06.765049+0000 mgr.y (mgr.14144) 394 : cluster [DBG] pgmap v548: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 159 MiB data, 504 MiB used, 536 GiB / 536 GiB avail; 16 MiB/s rd, 3.5 MiB/s wr, 28 op/s; 7/1197 objects misplaced (0.585%); 2.8 MiB/s, 11 objects/s recovering 2021-03-24T21:57:08.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:08 smithi125 conmon[24927]: cluster 2021-03-24T21:57:07. 2021-03-24T21:57:08.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:08 smithi125 conmon[24927]: 144766+0000 mon.a (mon.0) 901 : cluster [DBG] 2021-03-24T21:57:08.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:08 smithi125 conmon[24927]: osdmap e258: 8 total, 8 up, 6 in 2021-03-24T21:57:08.461 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:08.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:08.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:57:08.566 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:08.624 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:08.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:08.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:57:08.732 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:08.787 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:08.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:08.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:08.893 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:08.955 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:08.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:08.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:57:09.062 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:09.117 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:09.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:09.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:57:09.169 INFO:tasks.rados.rados.0.smithi125.stdout:815: done (5 left) 2021-03-24T21:57:09.169 INFO:tasks.rados.rados.0.smithi125.stdout:816: rollback oid 2 current snap is 79 2021-03-24T21:57:09.169 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 2 to 77 2021-03-24T21:57:09.170 INFO:tasks.rados.rados.0.smithi125.stdout:817: read oid 8 snap -1 2021-03-24T21:57:09.170 INFO:tasks.rados.rados.0.smithi125.stdout:817: expect (ObjNum 275 snap 68 seq_num 275) 2021-03-24T21:57:09.170 INFO:tasks.rados.rados.0.smithi125.stdout:818: read oid 36 snap 75 2021-03-24T21:57:09.170 INFO:tasks.rados.rados.0.smithi125.stdout:818: expect (ObjNum 225 snap 50 seq_num 225) 2021-03-24T21:57:09.171 INFO:tasks.rados.rados.0.smithi125.stdout:819: delete oid 43 current snap is 79 2021-03-24T21:57:09.171 INFO:tasks.rados.rados.0.smithi125.stdout:810: finishing rollback tid 1 to smithi12556470-25 2021-03-24T21:57:09.171 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 25 v 0 (ObjNum 1116655552 snap 21916 seq_num 1116654864) dirty dne 2021-03-24T21:57:09.171 INFO:tasks.rados.rados.0.smithi125.stdout:814: expect (ObjNum 280 snap 70 seq_num 280) 2021-03-24T21:57:09.222 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:09.267 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:09.269 INFO:tasks.rados.rados.0.smithi125.stdout:813: finishing rollback tid 0 to smithi12556470-37 2021-03-24T21:57:09.269 INFO:tasks.rados.rados.0.smithi125.stdout:813: finishing rollback tid 1 to smithi12556470-37 2021-03-24T21:57:09.270 INFO:tasks.rados.rados.0.smithi125.stdout:811: finishing rollback tid 0 to smithi12556470-19 2021-03-24T21:57:09.270 INFO:tasks.rados.rados.0.smithi125.stdout:813: finishing rollback tid 2 to smithi12556470-37 2021-03-24T21:57:09.270 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 37 v 77 (ObjNum 229 snap 50 seq_num 229) dirty exists 2021-03-24T21:57:09.270 INFO:tasks.rados.rados.0.smithi125.stdout:811: finishing rollback tid 1 to smithi12556470-19 2021-03-24T21:57:09.271 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 19 v 230 (ObjNum 0 snap 21916 seq_num 1116626304) dirty dne 2021-03-24T21:57:09.271 INFO:tasks.rados.rados.0.smithi125.stdout:812: expect (ObjNum 309 snap 76 seq_num 309) 2021-03-24T21:57:09.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:09.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:57:09.327 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-24T21:57:09.338 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:09 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:57:09.339 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:09 smithi125 conmon[19936]: 21:57:08.153974+0000 mon.a (mon.0) 902 : cluster [DBG] osdmap e259: 8 total, 8 up, 6 in 2021-03-24T21:57:09.339 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:09 smithi125 conmon[24927]: cluster 2021-03-24T21:57:08 2021-03-24T21:57:09.339 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:09 smithi125 conmon[24927]: .153974+0000 mon.a (mon.0) 902 : cluster [DBG] osdmap e259: 8 total, 8 up, 6 in 2021-03-24T21:57:09.373 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:09.376 INFO:tasks.rados.rados.0.smithi125.stdout:817: expect (ObjNum 275 snap 68 seq_num 275) 2021-03-24T21:57:09.427 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:09.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:09.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:57:09.533 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:09.545 INFO:tasks.rados.rados.0.smithi125.stdout:818: expect (ObjNum 225 snap 50 seq_num 225) 2021-03-24T21:57:09.546 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:09 smithi142 conmon[20125]: cluster 2021-03-24T21:57:08.153974+0000 mon.a (mon.0) 902 : cluster [DBG] osdmap e259: 8 total, 8 up, 6 in 2021-03-24T21:57:09.596 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:09.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:09.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:57:09.703 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:09.758 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:09.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:09.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:57:09.779 INFO:tasks.rados.rados.0.smithi125.stdout:816: finishing rollback tid 0 to smithi12556470-2 2021-03-24T21:57:09.779 INFO:tasks.rados.rados.0.smithi125.stdout:810: done (8 left) 2021-03-24T21:57:09.779 INFO:tasks.rados.rados.0.smithi125.stdout:811: done (7 left) 2021-03-24T21:57:09.779 INFO:tasks.rados.rados.0.smithi125.stdout:812: done (6 left) 2021-03-24T21:57:09.780 INFO:tasks.rados.rados.0.smithi125.stdout:813: done (5 left) 2021-03-24T21:57:09.780 INFO:tasks.rados.rados.0.smithi125.stdout:814: done (4 left) 2021-03-24T21:57:09.780 INFO:tasks.rados.rados.0.smithi125.stdout:817: done (3 left) 2021-03-24T21:57:09.780 INFO:tasks.rados.rados.0.smithi125.stdout:818: done (2 left) 2021-03-24T21:57:09.781 INFO:tasks.rados.rados.0.smithi125.stdout:819: done (1 left) 2021-03-24T21:57:09.781 INFO:tasks.rados.rados.0.smithi125.stdout:820: read oid 17 snap -1 2021-03-24T21:57:09.781 INFO:tasks.rados.rados.0.smithi125.stdout:820: expect (ObjNum 312 snap 77 seq_num 312) 2021-03-24T21:57:09.781 INFO:tasks.rados.rados.0.smithi125.stdout:821: snap_remove snap 76 2021-03-24T21:57:09.865 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:09.909 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:09.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:09.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:57:10.015 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:10.067 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:10.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:10.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:57:10.144 INFO:tasks.rados.rados.0.smithi125.stdout:821: done (2 left) 2021-03-24T21:57:10.144 INFO:tasks.rados.rados.0.smithi125.stdout:822: snap_create 2021-03-24T21:57:10.145 INFO:tasks.rados.rados.0.smithi125.stdout:816: finishing rollback tid 1 to smithi12556470-2 2021-03-24T21:57:10.145 INFO:tasks.rados.rados.0.smithi125.stdout:816: finishing rollback tid 2 to smithi12556470-2 2021-03-24T21:57:10.145 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 2 v 175 (ObjNum 284 snap 70 seq_num 284) dirty exists 2021-03-24T21:57:10.145 INFO:tasks.rados.rados.0.smithi125.stdout:820: expect (ObjNum 312 snap 77 seq_num 312) 2021-03-24T21:57:10.174 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:10.215 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:57:10.249 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:10.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:10.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:57:10.285 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:10 smithi125 conmon[19936]: cluster 2021-03-24T21:57:08.766145+0000 2021-03-24T21:57:10.285 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:10 smithi125 conmon[19936]: mgr.y (mgr.14144) 395 : cluster [DBG] pgmap v551: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 153 MiB data, 502 MiB used, 536 GiB / 536 GiB avail; 7.1 MiB/s rd, 6 op/s; 3/1185 objects misplaced (0.253%); 2.2 MiB/s, 1 objects/s recovering 2021-03-24T21:57:10.286 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:10 smithi125 conmon[19936]: cluster 2021-03-24T21:57:09.162081+0000 mon.a (mon.0) 903 : 2021-03-24T21:57:10.286 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:10 smithi125 conmon[19936]: cluster [DBG] osdmap e260: 8 total, 8 up, 6 in 2021-03-24T21:57:10.286 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:10 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:57:10.286 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:10 smithi125 conmon[19936]: 03-24T21:57:10.141637+0000 mon.a (mon.0) 904 : cluster [DBG] osdmap e261: 8 total, 8 up, 6 in 2021-03-24T21:57:10.287 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:10 smithi125 conmon[19936]: 2021-03-24T21:57:10.287 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:10 smithi125 conmon[24927]: cluster 2021-03-24T21:57:08.766145+0000 mgr.y (mgr.14144 2021-03-24T21:57:10.287 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:10 smithi125 conmon[24927]: ) 395 : cluster [DBG] pgmap v551: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 153 MiB data, 502 MiB used, 536 GiB / 536 GiB avail; 7.1 MiB/s rd, 6 op/s; 3/1185 objects misplaced (0.253%); 2.2 MiB/s, 1 objects/s recovering 2021-03-24T21:57:10.287 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:10 smithi125 conmon[24927]: cluster 2021-03-24T21:57:09.162081+0000 mon.a (mon.0) 903 : cluster [DBG] osdmap e260: 8 total, 8 up, 6 in 2021-03-24T21:57:10.288 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:10 smithi125 conmon[24927]: cluster 2021-03-24T21:57:10.141637+0000 mon.a (mon.0) 904 : cluster [DBG] osdmap e261: 8 total, 8 up, 6 in 2021-03-24T21:57:10.290 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:57:10.358 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:10.404 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:10.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:10.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:57:10.510 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:10.524 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:10 smithi142 conmon[20125]: cluster 2021-03-24T21:57:08.766145+0000 mgr.y (mgr.14144) 395 : cluster [DBG] pgmap v551: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 153 MiB data, 502 MiB used, 536 GiB / 536 GiB avail; 7.1 MiB/s rd, 6 op/s; 3/1185 objects misplaced (0.253%); 2.2 MiB/s, 1 objects/s recovering 2021-03-24T21:57:10.524 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:10 smithi142 conmon[20125]: cluster 2021-03-24T21:57:09.162081+0000 mon.a (mon.0) 903 : cluster [DBG] osdmap e260: 8 total, 8 up, 6 in 2021-03-24T21:57:10.524 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:10 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:57:10.524 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:10 smithi142 conmon[20125]: 24T21:57:10.141637+0000 mon.a (mon.0) 2021-03-24T21:57:10.524 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:10 smithi142 conmon[20125]: 904 : cluster [DBG] osdmap e261: 8 total, 8 up, 6 in 2021-03-24T21:57:10.566 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:10.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:10.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:57:10.673 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:10.726 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:10.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:10.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:57:10.832 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:10.885 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:10.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:10.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:57:10.991 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:11.047 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:11.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:11.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:11.149 INFO:tasks.rados.rados.0.smithi125.stdout:816: done (2 left) 2021-03-24T21:57:11.150 INFO:tasks.rados.rados.0.smithi125.stdout:820: done (1 left) 2021-03-24T21:57:11.150 INFO:tasks.rados.rados.0.smithi125.stdout:822: done (0 left) 2021-03-24T21:57:11.150 INFO:tasks.rados.rados.0.smithi125.stdout:823: write (excl) oid 46 current snap is 80 2021-03-24T21:57:11.151 INFO:tasks.rados.rados.0.smithi125.stdout:823: seq_num 315 ranges {494205=473908,1433122=407102,2384125=462684} 2021-03-24T21:57:11.153 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:11.154 INFO:teuthology.orchestra.run.smithi125.stderr:noscrub is unset 2021-03-24T21:57:11.156 INFO:tasks.rados.rados.0.smithi125.stdout:823: writing smithi12556470-46 from 494205 to 968113 tid 1 2021-03-24T21:57:11.162 INFO:tasks.rados.rados.0.smithi125.stdout:823: writing smithi12556470-46 from 1433122 to 1840224 tid 2 2021-03-24T21:57:11.169 INFO:tasks.rados.rados.0.smithi125.stdout:823: writing smithi12556470-46 from 2384125 to 2846809 tid 3 2021-03-24T21:57:11.170 INFO:tasks.rados.rados.0.smithi125.stdout:824: copy_from oid 23 from oid 45 current snap is 80 2021-03-24T21:57:11.170 INFO:tasks.rados.rados.0.smithi125.stdout:825: snap_remove snap 79 2021-03-24T21:57:11.204 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:11.207 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:11 smithi125 conmon[24927]: audit 2021-03-24T21:57:10.588201+0000 mon.c (mon 2021-03-24T21:57:11.207 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:11 smithi125 conmon[24927]: .2) 40 : audit [INF] from='client.? v1:172.21.15.125:0/1300792442' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T21:57:11.207 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:11 smithi125 conmon[24927]: audit 2021-03-24T21:57:10.588751+0000 mon.a (mon.0) 905 2021-03-24T21:57:11.207 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:11 smithi125 conmon[24927]: : audit [INF] from='client.34955 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T21:57:11.208 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:11 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:57:11.208 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:11 smithi125 conmon[24927]: T21:57:11.137023+0000 mon.a (mon.0) 906 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:57:11.208 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:11 smithi125 conmon[24927]: audit 2021-03-24T21:57:11.208 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:11 smithi125 conmon[24927]: 2021-03-24T21:57:11.147970+0000 2021-03-24T21:57:11.208 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:11 smithi125 conmon[24927]: mon.a (mon.0) 907 : audit [INF] from='client.34955 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T21:57:11.209 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:11 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:57:11.209 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:11 smithi125 conmon[24927]: 03-24T21:57:11.148106+0000 mon.a (mon.0) 908 : 2021-03-24T21:57:11.209 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:11 smithi125 conmon[24927]: cluster [DBG] osdmap e262: 8 total, 8 up, 6 in 2021-03-24T21:57:11.209 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:11 smithi125 conmon[19936]: audit 2021-03-24T21:57:11.210 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:11 smithi125 conmon[19936]: 2021-03-24 2021-03-24T21:57:11.210 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:11 smithi125 conmon[19936]: T21:57:10.588201+0000 mon.c (mon.2) 40 : audit [INF] from='client.? v1:172.21.15.125:0/1300792442' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T21:57:11.210 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:11 smithi125 conmon[19936]: audit 2021-03-24T21:57:11.210 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:11 smithi125 conmon[19936]: 2021-03-24T 2021-03-24T21:57:11.211 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:11 smithi125 conmon[19936]: 21:57:10.588751+0000 mon.a (mon.0) 905 : audit [INF] from='client.34955 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T21:57:11.211 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:11 smithi125 conmon[19936]: 2021-03-24T21:57:11.211 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:11 smithi125 conmon[19936]: cluster 2021-03-24T21:57:11.137023+0000 2021-03-24T21:57:11.211 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:11 smithi125 conmon[19936]: mon.a (mon.0) 906 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:57:11.212 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:11 smithi125 conmon[19936]: audit 2021 2021-03-24T21:57:11.212 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:11 smithi125 conmon[19936]: -03-24T21:57:11.147970+0000 mon.a 2021-03-24T21:57:11.212 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:11 smithi125 conmon[19936]: (mon.0) 907 : audit [INF] from='client.34955 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T21:57:11.212 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:11 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:57:11.212 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:11 smithi125 conmon[19936]: 03-24T21:57:11.148106+0000 mon.a ( 2021-03-24T21:57:11.213 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:11 smithi125 conmon[19936]: mon.0) 908 : cluster [DBG] osdmap e262: 8 total, 8 up, 6 in 2021-03-24T21:57:11.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:11.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:57:11.315 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:11.372 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:11.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:11.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:57:11.478 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:11.492 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:11 smithi142 conmon[20125]: audit 2021-03-24T21 2021-03-24T21:57:11.492 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:11 smithi142 conmon[20125]: :57:10.588201+0000 mon.c (mon.2) 40 : audit [INF] from='client.? v1:172.21.15.125:0/1300792442' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T21:57:11.492 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:11 smithi142 conmon[20125]: audit 2021-03-24T21:57:10 2021-03-24T21:57:11.493 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:11 smithi142 conmon[20125]: .588751+0000 mon.a (mon.0) 905 2021-03-24T21:57:11.493 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:11 smithi142 conmon[20125]: : audit [INF] from='client.34955 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T21:57:11.493 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:11 smithi142 conmon[20125]: cluster 2021-03-24T21:57:11.493 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:11 smithi142 conmon[20125]: 2021-03-24T21:57: 2021-03-24T21:57:11.494 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:11 smithi142 conmon[20125]: 11.137023+0000 mon.a (mon.0) 2021-03-24T21:57:11.494 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:11 smithi142 conmon[20125]: 906 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:57:11.494 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:11 smithi142 conmon[20125]: audit 2021-03-24T21:57:11.494 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:11 smithi142 conmon[20125]: 2021-03-24T21:57: 2021-03-24T21:57:11.494 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:11 smithi142 conmon[20125]: 11.147970+0000 mon.a (mon.0) 2021-03-24T21:57:11.495 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:11 smithi142 conmon[20125]: 907 : audit [INF] from='client.34955 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T21:57:11.495 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:11 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:57:11.495 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:11 smithi142 conmon[20125]: -03-24T21:57: 2021-03-24T21:57:11.495 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:11 smithi142 conmon[20125]: 11.148106+0000 mon.a ( 2021-03-24T21:57:11.496 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:11 smithi142 conmon[20125]: mon.0) 908 : 2021-03-24T21:57:11.496 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:11 smithi142 conmon[20125]: cluster [DBG] osdmap e262: 8 total, 8 up, 6 in 2021-03-24T21:57:11.535 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:11.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:11.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:11.641 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:11.696 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:11.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:11.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:11.802 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:11.862 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:11.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:11.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:57:11.969 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:12.033 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:12.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:12.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:57:12.139 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:12.176 INFO:tasks.rados.rados.0.smithi125.stdout:825: done (2 left) 2021-03-24T21:57:12.176 INFO:tasks.rados.rados.0.smithi125.stdout:826: rollback oid 10 current snap is 80 2021-03-24T21:57:12.177 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 10 to 77 2021-03-24T21:57:12.177 INFO:tasks.rados.rados.0.smithi125.stdout:823: finishing write tid 1 to smithi12556470-46 2021-03-24T21:57:12.177 INFO:tasks.rados.rados.0.smithi125.stdout:823: finishing write tid 2 to smithi12556470-46 2021-03-24T21:57:12.177 INFO:tasks.rados.rados.0.smithi125.stdout:823: finishing write tid 3 to smithi12556470-46 2021-03-24T21:57:12.178 INFO:tasks.rados.rados.0.smithi125.stdout:823: finishing write tid 4 to smithi12556470-46 2021-03-24T21:57:12.178 INFO:tasks.rados.rados.0.smithi125.stdout:823: finishing write tid 5 to smithi12556470-46 2021-03-24T21:57:12.178 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 46 v 85 (ObjNum 315 snap 80 seq_num 315) dirty exists 2021-03-24T21:57:12.178 INFO:tasks.rados.rados.0.smithi125.stdout:823: left oid 46 (ObjNum 315 snap 80 seq_num 315) 2021-03-24T21:57:12.179 INFO:tasks.rados.rados.0.smithi125.stdout:824: finishing copy_from to smithi12556470-23 2021-03-24T21:57:12.179 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 23 v 173 (ObjNum 150 snap 30 seq_num 150) dirty exists 2021-03-24T21:57:12.179 INFO:tasks.rados.rados.0.smithi125.stdout:824: finishing copy_from racing read to smithi12556470-23 2021-03-24T21:57:12.179 INFO:tasks.rados.rados.0.smithi125.stdout:823: done (2 left) 2021-03-24T21:57:12.179 INFO:tasks.rados.rados.0.smithi125.stdout:824: done (1 left) 2021-03-24T21:57:12.180 INFO:tasks.rados.rados.0.smithi125.stdout:827: write (excl) oid 2 current snap is 80 2021-03-24T21:57:12.180 INFO:tasks.rados.rados.0.smithi125.stdout:827: seq_num 316 ranges {511012=549157,1775251=550760,3095195=215817} 2021-03-24T21:57:12.188 INFO:tasks.rados.rados.0.smithi125.stdout:827: writing smithi12556470-2 from 511012 to 1060169 tid 1 2021-03-24T21:57:12.196 INFO:tasks.rados.rados.0.smithi125.stdout:827: writing smithi12556470-2 from 1775251 to 2326011 tid 2 2021-03-24T21:57:12.201 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:12.202 INFO:tasks.rados.rados.0.smithi125.stdout:827: writing smithi12556470-2 from 3095195 to 3311012 tid 3 2021-03-24T21:57:12.203 INFO:tasks.rados.rados.0.smithi125.stdout:828: copy_from oid 12 from oid 33 current snap is 80 2021-03-24T21:57:12.203 INFO:tasks.rados.rados.0.smithi125.stdout:826: finishing rollback tid 0 to smithi12556470-10 2021-03-24T21:57:12.203 INFO:tasks.rados.rados.0.smithi125.stdout:826: finishing rollback tid 1 to smithi12556470-10 2021-03-24T21:57:12.203 INFO:tasks.rados.rados.0.smithi125.stdout:826: finishing rollback tid 2 to smithi12556470-10 2021-03-24T21:57:12.204 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 10 v 272 (ObjNum 297 snap 74 seq_num 297) dirty exists 2021-03-24T21:57:12.204 INFO:tasks.rados.rados.0.smithi125.stdout:827: finishing write tid 1 to smithi12556470-2 2021-03-24T21:57:12.204 INFO:tasks.rados.rados.0.smithi125.stdout:827: finishing write tid 2 to smithi12556470-2 2021-03-24T21:57:12.204 INFO:tasks.rados.rados.0.smithi125.stdout:826: done (2 left) 2021-03-24T21:57:12.205 INFO:tasks.rados.rados.0.smithi125.stdout:829: read oid 27 snap 78 2021-03-24T21:57:12.205 INFO:tasks.rados.rados.0.smithi125.stdout:829: expect (ObjNum 193 snap 42 seq_num 193) 2021-03-24T21:57:12.205 INFO:tasks.rados.rados.0.smithi125.stdout:830: rollback oid 50 current snap is 80 2021-03-24T21:57:12.205 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 50 to 78 2021-03-24T21:57:12.205 INFO:tasks.rados.rados.0.smithi125.stdout:831: read oid 45 snap -1 2021-03-24T21:57:12.206 INFO:tasks.rados.rados.0.smithi125.stdout:831: expect (ObjNum 150 snap 30 seq_num 150) 2021-03-24T21:57:12.206 INFO:tasks.rados.rados.0.smithi125.stdout:832: write (excl) oid 36 current snap is 80 2021-03-24T21:57:12.206 INFO:tasks.rados.rados.0.smithi125.stdout:832: seq_num 317 ranges {527819=624406,1833733=410771,2722618=601581} 2021-03-24T21:57:12.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:12.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:57:12.209 INFO:tasks.rados.rados.0.smithi125.stdout:832: writing smithi12556470-36 from 527819 to 1152225 tid 1 2021-03-24T21:57:12.216 INFO:tasks.rados.rados.0.smithi125.stdout:832: writing smithi12556470-36 from 1833733 to 2244504 tid 2 2021-03-24T21:57:12.222 INFO:tasks.rados.rados.0.smithi125.stdout:832: writing smithi12556470-36 from 2722618 to 3324199 tid 3 2021-03-24T21:57:12.223 INFO:tasks.rados.rados.0.smithi125.stdout:833: write oid 44 current snap is 80 2021-03-24T21:57:12.223 INFO:tasks.rados.rados.0.smithi125.stdout:833: seq_num 318 ranges {544626=699655,1892215=554429,3150041=194585} 2021-03-24T21:57:12.229 INFO:tasks.rados.rados.0.smithi125.stdout:833: writing smithi12556470-44 from 544626 to 1244281 tid 1 2021-03-24T21:57:12.236 INFO:tasks.rados.rados.0.smithi125.stdout:833: writing smithi12556470-44 from 1892215 to 2446644 tid 2 2021-03-24T21:57:12.242 INFO:tasks.rados.rados.0.smithi125.stdout:833: writing smithi12556470-44 from 3150041 to 3344626 tid 3 2021-03-24T21:57:12.242 INFO:tasks.rados.rados.0.smithi125.stdout:834: snap_create 2021-03-24T21:57:12.242 INFO:tasks.rados.rados.0.smithi125.stdout:829: expect (ObjNum 193 snap 42 seq_num 193) 2021-03-24T21:57:12.309 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:12.322 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:12 smithi125 conmon[24927]: cluster 2021-03-24T21:57: 2021-03-24T21:57:12.323 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:12 smithi125 conmon[24927]: 10.388578+0000 osd.0 (osd.0) 67 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:57:12.323 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:12 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:57:12.323 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:12 smithi125 conmon[24927]: 03-24T21:57:10.388844 2021-03-24T21:57:12.323 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:12 smithi125 conmon[24927]: +0000 osd.0 (osd.0) 68 : cluster 2021-03-24T21:57:12.324 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:12 smithi125 conmon[24927]: [DBG] purged_snaps scrub ok 2021-03-24T21:57:12.324 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:12 smithi125 conmon[24927]: cluster 2021-03-24T21:57:12.324 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:12 smithi125 conmon[24927]: 2021-03- 2021-03-24T21:57:12.324 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:12 smithi125 conmon[24927]: 24T21:57:10.766681+0000 2021-03-24T21:57:12.324 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:12 smithi125 conmon[24927]: mgr.y (mgr.14144) 396 : cluster [DBG] 2021-03-24T21:57:12.325 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:12 smithi125 conmon[24927]: pgmap v554: 158 pgs: 1 peering, 157 active+clean; 142 MiB data, 493 MiB used, 536 GiB / 536 GiB avail; 13 MiB/s rd, 383 KiB/s wr, 14 op/s; 4.6 MiB/s, 7 objects/s recovering 2021-03-24T21:57:12.325 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:12 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:57:12.325 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:12 smithi125 conmon[24927]: -03-24T21:57:11 2021-03-24T21:57:12.325 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:12 smithi125 conmon[24927]: .229888+0000 osd.0 (osd.0) 69 2021-03-24T21:57:12.326 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:12 smithi125 conmon[24927]: : cluster [DBG] 3.17 scrub ok 2021-03-24T21:57:12.326 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:12 smithi125 conmon[24927]: cluster 2021-03-24T21:57:12.326 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:12 smithi125 conmon[24927]: 2021-03 2021-03-24T21:57:12.326 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:12 smithi125 conmon[24927]: -24T21:57:12.046196+0000 2021-03-24T21:57:12.327 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:12 smithi125 conmon[24927]: osd.2 (osd.2) 51 : cluster 2021-03-24T21:57:12.327 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:12 smithi125 conmon[24927]: [DBG] 3.0 scrub ok 2021-03-24T21:57:12.327 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:12 smithi125 conmon[24927]: cluster 2021-03-24T21:57:12.327 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:12 smithi125 conmon[24927]: 2021-03-24T21 2021-03-24T21:57:12.327 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:12 smithi125 conmon[24927]: :57:12.173373+0000 mon.a (mon 2021-03-24T21:57:12.328 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:12 smithi125 conmon[24927]: .0) 909 : cluster [DBG] osdmap e263: 8 total, 8 up, 6 in 2021-03-24T21:57:12.328 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:12 smithi125 conmon[19936]: cluster 2021-03-24T21:57:10.388578+0000 osd.0 (osd.0) 67 : 2021-03-24T21:57:12.328 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:12 smithi125 conmon[19936]: cluster [DBG] purged_snaps scrub starts 2021-03-24T21:57:12.329 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:12 smithi125 conmon[19936]: cluster 2021-03-24T21:57:10.388844+0000 osd.0 (osd.0) 68 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:57:12.329 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:12 smithi125 conmon[19936]: cluster 2021-03-24T21:57:12.329 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:12 smithi125 conmon[19936]: 2021-03-24T21:57:10.766681+0000 mgr.y (mgr.14144) 396 : cluster [DBG] pgmap v554: 158 pgs: 1 peering, 157 active+clean; 142 MiB data, 493 MiB used, 536 GiB / 536 GiB avail; 13 MiB/s rd, 383 KiB/s wr, 14 op/s; 4.6 MiB/s, 7 objects/s recovering 2021-03-24T21:57:12.329 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:12 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:57:12.329 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:12 smithi125 conmon[19936]: 57:11.229888+0000 osd.0 (osd.0) 69 : cluster [DBG] 2021-03-24T21:57:12.330 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:12 smithi125 conmon[19936]: 3.17 scrub ok 2021-03-24T21:57:12.330 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:12 smithi125 conmon[19936]: cluster 2021-03-24T21:57:12.046196+0000 osd.2 (osd.2 2021-03-24T21:57:12.330 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:12 smithi125 conmon[19936]: ) 51 : cluster [DBG] 3.0 scrub ok 2021-03-24T21:57:12.330 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:12 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:57:12.331 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:12 smithi125 conmon[19936]: -24T21:57:12.173373+0000 mon.a (mon.0) 909 : cluster [DBG] osdmap e263: 8 total, 8 up, 6 in 2021-03-24T21:57:12.358 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:12.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:12.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:57:12.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:12 smithi142 conmon[20125]: cluster 2021-03-24T21:57:12.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:12 smithi142 conmon[20125]: 2021-03-24T21:57:12.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:12 smithi142 conmon[20125]: 2021 2021-03-24T21:57:12.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:12 smithi142 conmon[20125]: - 2021-03-24T21:57:12.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:12 smithi142 conmon[20125]: 03 2021-03-24T21:57:12.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:12 smithi142 conmon[20125]: - 2021-03-24T21:57:12.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:12 smithi142 conmon[20125]: 24T21:57:10 2021-03-24T21:57:12.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:12 smithi142 conmon[20125]: .388578+0000 osd.0 (osd.0) 67 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:57:12.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:12 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:57:12.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:12 smithi142 conmon[20125]: 03-24T21:57:10.388844 2021-03-24T21:57:12.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:12 smithi142 conmon[20125]: +0000 osd.0 (osd 2021-03-24T21:57:12.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:12 smithi142 conmon[20125]: .0) 68 : 2021-03-24T21:57:12.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:12 smithi142 conmon[20125]: cluster [DBG] 2021-03-24T21:57:12.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:12 smithi142 conmon[20125]: purged_snaps scrub ok 2021-03-24T21:57:12.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:12 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:57:12.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:12 smithi142 conmon[20125]: -03-24T21:57:10.766681+0000 2021-03-24T21:57:12.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:12 smithi142 conmon[20125]: mgr.y (mgr. 2021-03-24T21:57:12.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:12 smithi142 conmon[20125]: 14144) 396 : cluster 2021-03-24T21:57:12.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:12 smithi142 conmon[20125]: [DBG] pgmap v554: 158 pgs: 1 peering, 157 active+clean; 142 MiB data, 493 MiB used, 536 GiB / 536 GiB avail; 13 MiB/s rd, 383 KiB/s wr, 14 op/s; 4.6 MiB/s, 7 objects/s recovering 2021-03-24T21:57:12.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:12 smithi142 conmon[20125]: cluster 2021-03-24T21:57:12.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:12 smithi142 conmon[20125]: 2021-03-24T21:57:12.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:12 smithi142 conmon[20125]: 2021 2021-03-24T21:57:12.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:12 smithi142 conmon[20125]: - 2021-03-24T21:57:12.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:12 smithi142 conmon[20125]: 03 2021-03-24T21:57:12.373 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:12 smithi142 conmon[20125]: - 2021-03-24T21:57:12.373 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:12 smithi142 conmon[20125]: 24 2021-03-24T21:57:12.373 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:12 smithi142 conmon[20125]: T21:57:11 2021-03-24T21:57:12.373 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:12 smithi142 conmon[20125]: .229888+0000 osd.0 (osd.0) 69 : cluster [DBG] 2021-03-24T21:57:12.374 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:12 smithi142 conmon[20125]: 3.17 scrub ok 2021-03-24T21:57:12.374 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:12 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:57:12.374 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:12 smithi142 conmon[20125]: -03-24T21:57: 2021-03-24T21:57:12.374 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:12 smithi142 conmon[20125]: 12.046196+0000 osd.2 ( 2021-03-24T21:57:12.374 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:12 smithi142 conmon[20125]: osd.2) 51 : cluster 2021-03-24T21:57:12.375 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:12 smithi142 conmon[20125]: [DBG] 3.0 scrub ok 2021-03-24T21:57:12.375 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:12 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:57:12.375 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:12 smithi142 conmon[20125]: T21:57:12.173373+0000 mon.a (mon 2021-03-24T21:57:12.375 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:12 smithi142 conmon[20125]: .0) 909 : cluster [DBG] osdmap e263: 8 total, 8 up, 6 in 2021-03-24T21:57:12.434 INFO:tasks.rados.rados.0.smithi125.stdout:827: finishing write tid 3 to smithi12556470-2 2021-03-24T21:57:12.435 INFO:tasks.rados.rados.0.smithi125.stdout:830: finishing rollback tid 1 to smithi12556470-50 2021-03-24T21:57:12.435 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 50 v 0 (ObjNum 1116483152 snap 21916 seq_num 1116477424) dirty dne 2021-03-24T21:57:12.435 INFO:tasks.rados.rados.0.smithi125.stdout:831: expect (ObjNum 150 snap 30 seq_num 150) 2021-03-24T21:57:12.463 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:12.517 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:12.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:12.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:12.567 INFO:tasks.rados.rados.0.smithi125.stdout:827: finishing write tid 4 to smithi12556470-2 2021-03-24T21:57:12.567 INFO:tasks.rados.rados.0.smithi125.stdout:827: finishing write tid 5 to smithi12556470-2 2021-03-24T21:57:12.567 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 2 v 181 (ObjNum 316 snap 80 seq_num 316) dirty exists 2021-03-24T21:57:12.568 INFO:tasks.rados.rados.0.smithi125.stdout:827: left oid 2 (ObjNum 316 snap 80 seq_num 316) 2021-03-24T21:57:12.568 INFO:tasks.rados.rados.0.smithi125.stdout:832: finishing write tid 1 to smithi12556470-36 2021-03-24T21:57:12.568 INFO:tasks.rados.rados.0.smithi125.stdout:832: finishing write tid 2 to smithi12556470-36 2021-03-24T21:57:12.569 INFO:tasks.rados.rados.0.smithi125.stdout:832: finishing write tid 3 to smithi12556470-36 2021-03-24T21:57:12.569 INFO:tasks.rados.rados.0.smithi125.stdout:832: finishing write tid 4 to smithi12556470-36 2021-03-24T21:57:12.569 INFO:tasks.rados.rados.0.smithi125.stdout:832: finishing write tid 5 to smithi12556470-36 2021-03-24T21:57:12.569 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 36 v 186 (ObjNum 317 snap 80 seq_num 317) dirty exists 2021-03-24T21:57:12.569 INFO:tasks.rados.rados.0.smithi125.stdout:832: left oid 36 (ObjNum 317 snap 80 seq_num 317) 2021-03-24T21:57:12.570 INFO:tasks.rados.rados.0.smithi125.stdout:828: finishing copy_from to smithi12556470-12 2021-03-24T21:57:12.570 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 12 v 228 (ObjNum 228 snap 50 seq_num 228) dirty exists 2021-03-24T21:57:12.570 INFO:tasks.rados.rados.0.smithi125.stdout:828: finishing copy_from racing read to smithi12556470-12 2021-03-24T21:57:12.570 INFO:tasks.rados.rados.0.smithi125.stdout:833: finishing write tid 1 to smithi12556470-44 2021-03-24T21:57:12.571 INFO:tasks.rados.rados.0.smithi125.stdout:833: finishing write tid 2 to smithi12556470-44 2021-03-24T21:57:12.571 INFO:tasks.rados.rados.0.smithi125.stdout:833: finishing write tid 3 to smithi12556470-44 2021-03-24T21:57:12.571 INFO:tasks.rados.rados.0.smithi125.stdout:833: finishing write tid 4 to smithi12556470-44 2021-03-24T21:57:12.571 INFO:tasks.rados.rados.0.smithi125.stdout:833: finishing write tid 5 to smithi12556470-44 2021-03-24T21:57:12.572 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 44 v 221 (ObjNum 318 snap 80 seq_num 318) dirty exists 2021-03-24T21:57:12.572 INFO:tasks.rados.rados.0.smithi125.stdout:833: left oid 44 (ObjNum 318 snap 80 seq_num 318) 2021-03-24T21:57:12.623 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:12.679 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:12.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:12.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:57:12.705 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-24T21:57:12.705 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T21:57:12.784 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:12.843 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:12.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:12.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:57:12.948 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:13.004 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:13.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:13.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:57:13.110 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:13.164 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:13.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:13.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:57:13.198 INFO:tasks.rados.rados.0.smithi125.stdout:827: done (7 left) 2021-03-24T21:57:13.198 INFO:tasks.rados.rados.0.smithi125.stdout:828: done (6 left) 2021-03-24T21:57:13.198 INFO:tasks.rados.rados.0.smithi125.stdout:829: done (5 left) 2021-03-24T21:57:13.198 INFO:tasks.rados.rados.0.smithi125.stdout:830: done (4 left) 2021-03-24T21:57:13.199 INFO:tasks.rados.rados.0.smithi125.stdout:831: done (3 left) 2021-03-24T21:57:13.199 INFO:tasks.rados.rados.0.smithi125.stdout:832: done (2 left) 2021-03-24T21:57:13.199 INFO:tasks.rados.rados.0.smithi125.stdout:833: done (1 left) 2021-03-24T21:57:13.200 INFO:tasks.rados.rados.0.smithi125.stdout:834: done (0 left) 2021-03-24T21:57:13.200 INFO:tasks.rados.rados.0.smithi125.stdout:835: read oid 1 snap -1 2021-03-24T21:57:13.200 INFO:tasks.rados.rados.0.smithi125.stdout:835: expect (ObjNum 276 snap 68 seq_num 276) 2021-03-24T21:57:13.200 INFO:tasks.rados.rados.0.smithi125.stdout:836: read oid 37 snap 78 2021-03-24T21:57:13.200 INFO:tasks.rados.rados.0.smithi125.stdout:836: expect (ObjNum 229 snap 50 seq_num 229) 2021-03-24T21:57:13.201 INFO:tasks.rados.rados.0.smithi125.stdout:837: read oid 42 snap -1 2021-03-24T21:57:13.201 INFO:tasks.rados.rados.0.smithi125.stdout:837: expect deleted 2021-03-24T21:57:13.201 INFO:tasks.rados.rados.0.smithi125.stdout:838: copy_from oid 50 from oid 29 current snap is 81 2021-03-24T21:57:13.201 INFO:tasks.rados.rados.0.smithi125.stdout:839: rollback oid 32 current snap is 81 2021-03-24T21:57:13.201 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 32 to 80 2021-03-24T21:57:13.202 INFO:tasks.rados.rados.0.smithi125.stdout:840: read oid 40 snap -1 2021-03-24T21:57:13.202 INFO:tasks.rados.rados.0.smithi125.stdout:840: expect (ObjNum 283 snap 70 seq_num 283) 2021-03-24T21:57:13.202 INFO:tasks.rados.rados.0.smithi125.stdout:841: snap_remove snap 70 2021-03-24T21:57:13.271 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:13.327 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:13.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:13.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:57:13.434 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:13.493 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:13.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:13.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:57:13.598 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:13.659 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:13.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:13.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:57:13.764 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:13.821 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:13.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:13.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:13.927 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:13.987 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:13.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:13.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:57:14.093 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:14.154 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:14.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:14.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:14.208 INFO:tasks.rados.rados.0.smithi125.stdout:841: done (6 left) 2021-03-24T21:57:14.209 INFO:tasks.rados.rados.0.smithi125.stdout:842: rollback oid 9 current snap is 81 2021-03-24T21:57:14.209 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 9 to 75 2021-03-24T21:57:14.209 INFO:tasks.rados.rados.0.smithi125.stdout:843: rollback oid 2 current snap is 81 2021-03-24T21:57:14.209 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 2 to 77 2021-03-24T21:57:14.210 INFO:tasks.rados.rados.0.smithi125.stdout:844: read oid 38 snap -1 2021-03-24T21:57:14.210 INFO:tasks.rados.rados.0.smithi125.stdout:844: expect deleted 2021-03-24T21:57:14.210 INFO:tasks.rados.rados.0.smithi125.stdout:845: write oid 4 current snap is 81 2021-03-24T21:57:14.210 INFO:tasks.rados.rados.0.smithi125.stdout:845: seq_num 319 ranges {561433=774904,1950697=414440,2777464=583969} 2021-03-24T21:57:14.214 INFO:tasks.rados.rados.0.smithi125.stdout:845: writing smithi12556470-4 from 561433 to 1336337 tid 1 2021-03-24T21:57:14.221 INFO:tasks.rados.rados.0.smithi125.stdout:845: writing smithi12556470-4 from 1950697 to 2365137 tid 2 2021-03-24T21:57:14.228 INFO:tasks.rados.rados.0.smithi125.stdout:845: writing smithi12556470-4 from 2777464 to 3361433 tid 3 2021-03-24T21:57:14.228 INFO:tasks.rados.rados.0.smithi125.stdout:846: read oid 3 snap -1 2021-03-24T21:57:14.229 INFO:tasks.rados.rados.0.smithi125.stdout:846: expect deleted 2021-03-24T21:57:14.229 INFO:tasks.rados.rados.0.smithi125.stdout:847: read oid 19 snap 75 2021-03-24T21:57:14.229 INFO:tasks.rados.rados.0.smithi125.stdout:847: expect (ObjNum 244 snap 57 seq_num 244) 2021-03-24T21:57:14.229 INFO:tasks.rados.rados.0.smithi125.stdout:848: snap_create 2021-03-24T21:57:14.230 INFO:tasks.rados.rados.0.smithi125.stdout:836: expect (ObjNum 229 snap 50 seq_num 229) 2021-03-24T21:57:14.260 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:14.320 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:14.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:14.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:57:14.345 INFO:journalctl@ceph.osd.2.smithi125.stdout:Mar 24 21:57:14 smithi125 conmon[34620]: debug 2021-03-24T21:57:14.345 INFO:journalctl@ceph.osd.2.smithi125.stdout:Mar 24 21:57:14 smithi125 conmon[34620]: 2021-03-24T21:57:14.212+0000 7f47029d0700 -1 osd.2 pg_epoch: 265 pg[7.e( v 258'43 (0'0,258'43] local-lis/les=99/100 n=2 ec=99/99 lis/c=99/99 les/c/f=100/100/0 sis=99) [2,5,0] r=0 lpr=99 crt=258'43 lcod 254'41 mlcod 254'41 active+clean+snaptrim trimq=[8b~1] ps=[82~1,8c~2,92~1,95~1,9c~1]] removing snap head 2021-03-24T21:57:14.345 INFO:journalctl@ceph.osd.2.smithi125.stdout:Mar 24 21:57:14 smithi125 conmon[34620]: 2021-03-24T21:57:14.345 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:14 smithi125 conmon[19936]: cluster 2021-03-24T21:57:12.514088+0000 2021-03-24T21:57:14.346 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:14 smithi125 conmon[19936]: osd.3 (osd.3) 66 : cluster [DBG] 3.c scrub ok 2021-03-24T21:57:14.346 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:14 smithi125 conmon[19936]: cluster 2021-03-24T21:57 2021-03-24T21:57:14.346 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:14 smithi125 conmon[19936]: :12.767487+0000 mgr.y (mgr.14144) 397 : cluster [DBG] pgmap v557: 158 pgs: 1 peering, 157 active+clean; 139 MiB data, 482 MiB used, 536 GiB / 536 GiB avail; 6.3 MiB/s rd, 383 KiB/s wr, 8 op/s; 2.4 MiB/s, 5 objects/s recovering 2021-03-24T21:57:14.346 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:14 smithi125 conmon[19936]: 2021-03-24T21:57:14.347 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:14 smithi125 conmon[19936]: cluster 2021-03-24T21:57 2021-03-24T21:57:14.347 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:14 smithi125 conmon[19936]: :13.016111+0000 osd.5 (osd.5) 69 : cluster [DBG] 3.16 scrub ok 2021-03-24T21:57:14.347 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:14 smithi125 conmon[19936]: cluster 2021-03-24T21:57:14.347 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:14 smithi125 conmon[19936]: 2021-03-24T21 2021-03-24T21:57:14.347 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:14 smithi125 conmon[19936]: :57:13.194811+0000 mon.a (mon.0 2021-03-24T21:57:14.348 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:14 smithi125 conmon[19936]: ) 910 : cluster [DBG] osdmap e264: 8 total, 8 up, 6 in 2021-03-24T21:57:14.348 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:14 smithi125 conmon[24927]: cluster 2021-03-24T21:57:12.514088+0000 osd.3 (osd.3) 66 : 2021-03-24T21:57:14.348 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:14 smithi125 conmon[24927]: cluster [DBG] 3.c scrub ok 2021-03-24T21:57:14.348 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:14 smithi125 conmon[24927]: cluster 2021-03-24T21:57:12.767487+0000 mgr.y (mgr.14144) 397 : cluster [DBG] pgmap v557: 158 pgs: 1 peering, 157 active+clean; 139 MiB data, 482 MiB used, 536 GiB / 536 GiB avail; 6.3 MiB/s rd, 383 KiB/s wr, 8 op/s; 2.4 MiB/s, 5 objects/s recovering 2021-03-24T21:57:14.349 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:14 smithi125 conmon[24927]: cluster 2021-03-24T21:57:13.016111+0000 osd.5 (osd. 2021-03-24T21:57:14.349 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:14 smithi125 conmon[24927]: 5) 69 : cluster [DBG] 3.16 scrub ok 2021-03-24T21:57:14.349 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:14 smithi125 conmon[24927]: cluster 2021-03-24T21:57:13.194811 2021-03-24T21:57:14.349 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:14 smithi125 conmon[24927]: +0000 mon.a (mon.0) 910 : cluster [DBG] osdmap e264: 8 total, 8 up, 6 in 2021-03-24T21:57:14.349 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:14 smithi125 conmon[24927]: 2021-03-24T21:57:14.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:14 smithi142 conmon[20125]: cluster 2021-03-24T21:57:12.514088+0000 osd.3 (osd.3) 2021-03-24T21:57:14.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:14 smithi142 conmon[20125]: 66 : cluster [DBG] 3.c scrub ok 2021-03-24T21:57:14.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:14 smithi142 conmon[20125]: cluster 2021-03-24T21:57:12.767487+0000 mgr.y (mgr.14144) 397 : cluster [DBG] pgmap v557: 158 pgs: 1 peering, 157 active+clean; 139 MiB data, 482 MiB used, 536 GiB / 536 GiB avail; 6.3 MiB/s rd, 383 KiB/s wr, 8 op/s; 2.4 MiB/s, 5 objects/s recovering 2021-03-24T21:57:14.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:14 smithi142 conmon[20125]: cluster 2021-03-24T21:57:13.016111+0000 osd.5 (osd.5) 69 : cluster [DBG] 3.16 scrub ok 2021-03-24T21:57:14.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:14 smithi142 conmon[20125]: cluster 2021-03-24T21:57:13.194811+0000 mon.a (mon.0) 910 : cluster [DBG] osdmap e264: 8 total, 8 up, 6 in 2021-03-24T21:57:14.368 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 24 21:57:14 smithi142 conmon[33921]: debug 2021-03-24T21:57:14.207+0000 7fc7b541c700 -1 osd.7 pg_epoch: 265 pg[7.19( v 261'256 (0'0,261'256] local-lis/les=204/205 n=3 ec=198/99 lis/c=204/204 les/c/f=205/205/0 sis=204) [7,2,5] r=0 lpr=204 crt=261'256 lcod 260'254 mlcod 260'254 active+clean+snaptrim trimq=[8b~1] ps=[82~1,8c~2,92~1,95~1,9c~1]] removing snap head 2021-03-24T21:57:14.368 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 24 21:57:14 smithi142 conmon[33921]: debug 2021-03-24T21:57:14.368 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 24 21:57:14 smithi142 conmon[33921]: 2021-03-24T21:57:14.215+0000 7fc7b641e700 -1 osd.7 pg_epoch: 265 pg[7.17( v 263'272 (0'0,263'272] local-lis/les=203/204 n=6 ec=198/99 lis/c=203/203 les/c/f=204/204/0 sis=203) [7,5,2] r=0 lpr=203 crt=263'272 lcod 263'271 mlcod 263'271 active+clean+snaptrim trimq=[8b~1] ps=[82~1,8c~2,92~1,95~1,9c~1]] removing snap head 2021-03-24T21:57:14.369 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 24 21:57:14 smithi142 conmon[33921]: 2021-03-24T21:57:14.369 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 24 21:57:14 smithi142 conmon[33921]: debug 2021-03-24T21:57:14.223+0000 7fc7b541c700 -1 osd.7 pg_epoch: 265 pg[7.11( v 265'272 (0'0,265'272] local-lis/les=204/205 n=5 ec=198/99 lis/c=204/204 les/c/f=205/205/0 sis=204) [7,3,2] r=0 lpr=204 crt=265'272 lcod 265'270 mlcod 265'270 active+clean+snaptrim trimq=[8b~1] ps=[82~1,8c~2,92~1,95~1,9c~1]] removing snap head 2021-03-24T21:57:14.402 INFO:tasks.rados.rados.0.smithi125.stdout:835: expect (ObjNum 276 snap 68 seq_num 276) 2021-03-24T21:57:14.427 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:14.469 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:14.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:14.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:57:14.565 INFO:tasks.rados.rados.0.smithi125.stdout:839: finishing rollback tid 0 to smithi12556470-32 2021-03-24T21:57:14.566 INFO:tasks.rados.rados.0.smithi125.stdout:839: finishing rollback tid 1 to smithi12556470-32 2021-03-24T21:57:14.566 INFO:tasks.rados.rados.0.smithi125.stdout:839: finishing rollback tid 2 to smithi12556470-32 2021-03-24T21:57:14.566 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 32 v 241 (ObjNum 275 snap 68 seq_num 275) dirty exists 2021-03-24T21:57:14.567 INFO:tasks.rados.rados.0.smithi125.stdout:840: expect (ObjNum 283 snap 70 seq_num 283) 2021-03-24T21:57:14.575 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:14.629 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:14.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:14.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:57:14.656 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.2 dump_historic_ops 2021-03-24T21:57:14.735 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:14.791 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:14.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:14.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:14.804 INFO:tasks.rados.rados.0.smithi125.stdout:838: finishing copy_from to smithi12556470-50 2021-03-24T21:57:14.804 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 50 v 128 (ObjNum 274 snap 68 seq_num 274) dirty exists 2021-03-24T21:57:14.804 INFO:tasks.rados.rados.0.smithi125.stdout:838: finishing copy_from racing read to smithi12556470-50 2021-03-24T21:57:14.804 INFO:tasks.rados.rados.0.smithi125.stdout:842: finishing rollback tid 0 to smithi12556470-9 2021-03-24T21:57:14.804 INFO:tasks.rados.rados.0.smithi125.stdout:845: finishing write tid 1 to smithi12556470-4 2021-03-24T21:57:14.805 INFO:tasks.rados.rados.0.smithi125.stdout:842: finishing rollback tid 1 to smithi12556470-9 2021-03-24T21:57:14.805 INFO:tasks.rados.rados.0.smithi125.stdout:842: finishing rollback tid 2 to smithi12556470-9 2021-03-24T21:57:14.805 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 9 v 249 (ObjNum 203 snap 43 seq_num 203) dirty exists 2021-03-24T21:57:14.805 INFO:tasks.rados.rados.0.smithi125.stdout:845: finishing write tid 2 to smithi12556470-4 2021-03-24T21:57:14.806 INFO:tasks.rados.rados.0.smithi125.stdout:843: finishing rollback tid 0 to smithi12556470-2 2021-03-24T21:57:14.806 INFO:tasks.rados.rados.0.smithi125.stdout:843: finishing rollback tid 1 to smithi12556470-2 2021-03-24T21:57:14.806 INFO:tasks.rados.rados.0.smithi125.stdout:843: finishing rollback tid 2 to smithi12556470-2 2021-03-24T21:57:14.806 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 2 v 194 (ObjNum 284 snap 70 seq_num 284) dirty exists 2021-03-24T21:57:14.807 INFO:tasks.rados.rados.0.smithi125.stdout:845: finishing write tid 3 to smithi12556470-4 2021-03-24T21:57:14.807 INFO:tasks.rados.rados.0.smithi125.stdout:845: finishing write tid 4 to smithi12556470-4 2021-03-24T21:57:14.807 INFO:tasks.rados.rados.0.smithi125.stdout:845: finishing write tid 5 to smithi12556470-4 2021-03-24T21:57:14.807 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 4 v 241 (ObjNum 319 snap 81 seq_num 319) dirty exists 2021-03-24T21:57:14.807 INFO:tasks.rados.rados.0.smithi125.stdout:845: left oid 4 (ObjNum 319 snap 81 seq_num 319) 2021-03-24T21:57:14.808 INFO:tasks.rados.rados.0.smithi125.stdout:847: expect (ObjNum 244 snap 57 seq_num 244) 2021-03-24T21:57:14.895 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:14.955 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:14.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:14.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:57:14.982 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T21:57:15.000 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T21:57:15.017 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T21:57:15.039 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T21:57:15.061 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:15.066 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T21:57:15.094 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T21:57:15.118 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T21:57:15.119 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:15.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:15.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:15.145 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T21:57:15.147 INFO:tasks.rados.rados.0.smithi125.stdout:835: done (12 left) 2021-03-24T21:57:15.148 INFO:tasks.rados.rados.0.smithi125.stdout:836: done (11 left) 2021-03-24T21:57:15.148 INFO:tasks.rados.rados.0.smithi125.stdout:837: done (10 left) 2021-03-24T21:57:15.148 INFO:tasks.rados.rados.0.smithi125.stdout:838: done (9 left) 2021-03-24T21:57:15.148 INFO:tasks.rados.rados.0.smithi125.stdout:839: done (8 left) 2021-03-24T21:57:15.149 INFO:tasks.rados.rados.0.smithi125.stdout:840: done (7 left) 2021-03-24T21:57:15.149 INFO:tasks.rados.rados.0.smithi125.stdout:842: done (6 left) 2021-03-24T21:57:15.149 INFO:tasks.rados.rados.0.smithi125.stdout:843: done (5 left) 2021-03-24T21:57:15.149 INFO:tasks.rados.rados.0.smithi125.stdout:844: done (4 left) 2021-03-24T21:57:15.149 INFO:tasks.rados.rados.0.smithi125.stdout:845: done (3 left) 2021-03-24T21:57:15.150 INFO:tasks.rados.rados.0.smithi125.stdout:846: done (2 left) 2021-03-24T21:57:15.150 INFO:tasks.rados.rados.0.smithi125.stdout:847: done (1 left) 2021-03-24T21:57:15.150 INFO:tasks.rados.rados.0.smithi125.stdout:848: done (0 left) 2021-03-24T21:57:15.150 INFO:tasks.rados.rados.0.smithi125.stdout:849: read oid 34 snap -1 2021-03-24T21:57:15.150 INFO:tasks.rados.rados.0.smithi125.stdout:849: expect (ObjNum 203 snap 43 seq_num 203) 2021-03-24T21:57:15.151 INFO:tasks.rados.rados.0.smithi125.stdout:850: read oid 35 snap 80 2021-03-24T21:57:15.151 INFO:tasks.rados.rados.0.smithi125.stdout:850: expect (ObjNum 150 snap 30 seq_num 150) 2021-03-24T21:57:15.151 INFO:tasks.rados.rados.0.smithi125.stdout:851: delete oid 24 current snap is 82 2021-03-24T21:57:15.153 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-24T21:57:15.153 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-24T21:57:15.154 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-24T21:57:15.154 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-24T21:57:15.154 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-24T21:57:15.154 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-24T21:57:15.154 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-24T21:57:15.155 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-24T21:57:15.162 INFO:tasks.rados.rados.0.smithi125.stdout:849: expect (ObjNum 203 snap 43 seq_num 203) 2021-03-24T21:57:15.189 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:57:15.220 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:57:15.220 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[19936]: 21:57:13.701134+0000 osd.4 (osd.4) 69 : cluster [DBG] 3.1a scrub ok 2021-03-24T21:57:15.220 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[19936]: cluster 2021-03-24T21:57: 2021-03-24T21:57:15.221 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[19936]: 14.170622+0000 osd.3 (osd. 2021-03-24T21:57:15.221 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[19936]: 3) 67 : cluster [DBG] 2021-03-24T21:57:15.221 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[19936]: purged_snaps scrub starts 2021-03-24T21:57:15.221 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:57:15.222 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[19936]: -03-24T21:57 2021-03-24T21:57:15.222 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[19936]: :14.171071+0000 osd.3 ( 2021-03-24T21:57:15.222 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[19936]: osd.3) 68 : cluster 2021-03-24T21:57:15.222 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[19936]: [DBG] purged_snaps scrub ok 2021-03-24T21:57:15.223 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:57:15.223 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[19936]: -03-24T21:57 2021-03-24T21:57:15.223 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[19936]: :14.199930+0000 mon.a (mon 2021-03-24T21:57:15.223 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[19936]: .0) 911 : cluster [DBG] 2021-03-24T21:57:15.224 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[19936]: osdmap e265: 8 total, 8 up, 6 in 2021-03-24T21:57:15.224 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:57:15.224 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[19936]: -03-24T21: 2021-03-24T21:57:15.224 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[19936]: 57:14.281734+0000 osd.7 2021-03-24T21:57:15.225 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[19936]: (osd.7) 60 : cluster 2021-03-24T21:57:15.225 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[19936]: [DBG] 7.17 scrub ok 2021-03-24T21:57:15.225 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[19936]: audit 2021 2021-03-24T21:57:15.225 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[19936]: -03-24T21:57 2021-03-24T21:57:15.225 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[19936]: :14.959450+0000 mon.c ( 2021-03-24T21:57:15.226 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[19936]: mon.2) 41 : audit 2021-03-24T21:57:15.226 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[19936]: [DBG] from='client.? v1:172.21.15.125:0/100638348' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:57:15.226 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:57:15.227 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[19936]: -03-24T21:57 2021-03-24T21:57:15.227 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[19936]: :15.145583+0000 mon.a (mon 2021-03-24T21:57:15.227 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[19936]: .0) 912 : cluster [DBG] 2021-03-24T21:57:15.228 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[19936]: osdmap e266: 8 total, 8 up, 6 in 2021-03-24T21:57:15.228 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:57:15.228 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[24927]: 24T21:57:13.701134+0000 osd.4 (osd.4) 2021-03-24T21:57:15.228 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[24927]: 69 : cluster [DBG] 3.1a scrub ok 2021-03-24T21:57:15.229 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[24927]: 2021-03-24T21:57:15.229 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[24927]: cluster 2021-03-24T21:57:15.229 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[24927]: 2021-03-24T21:57:15.229 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[24927]: 2021-03- 2021-03-24T21:57:15.230 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[24927]: 24T21:57:14.170622 2021-03-24T21:57:15.230 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[24927]: +0000 osd.3 (osd. 2021-03-24T21:57:15.230 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[24927]: 3) 67 : cluster 2021-03-24T21:57:15.230 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[24927]: [DBG] purged_snaps scrub starts 2021-03-24T21:57:15.231 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:57:15.231 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[24927]: -03-24T21:57:14.171071+0000 2021-03-24T21:57:15.231 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[24927]: osd.3 (osd.3) 68 : cluster [DBG] 2021-03-24T21:57:15.231 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[24927]: purged_snaps scrub ok 2021-03-24T21:57:15.231 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:57:15.232 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[24927]: -03-24T21:57 2021-03-24T21:57:15.232 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[24927]: :14.199930+0000 mon.a ( 2021-03-24T21:57:15.232 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[24927]: mon.0) 911 : cluster 2021-03-24T21:57:15.232 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[24927]: [DBG] osdmap e265: 8 total, 8 up, 6 in 2021-03-24T21:57:15.232 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[24927]: cluster 2021-03-24T21:57:15.233 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[24927]: 2021-03-24T21:57 2021-03-24T21:57:15.233 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[24927]: :14.281734+0000 2021-03-24T21:57:15.233 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[24927]: osd.7 (osd.7) 60 : 2021-03-24T21:57:15.233 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[24927]: cluster [DBG] 7.17 scrub ok 2021-03-24T21:57:15.234 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[24927]: audit 2021-03-24T21:57:15.234 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[24927]: 2021-03-24T21:57 2021-03-24T21:57:15.234 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[24927]: :14.959450+0000 mon.c 2021-03-24T21:57:15.234 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[24927]: (mon.2) 41 2021-03-24T21:57:15.234 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[24927]: : audit [DBG] from='client.? v1:172.21.15.125:0/100638348' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:57:15.235 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:57:15.235 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[24927]: 03-24T21:57 2021-03-24T21:57:15.235 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[24927]: :15.145583+0000 mon.a ( 2021-03-24T21:57:15.235 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[24927]: mon.0) 912 : 2021-03-24T21:57:15.236 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:15 smithi125 conmon[24927]: cluster [DBG] osdmap e266: 8 total, 8 up, 6 in 2021-03-24T21:57:15.236 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:15.296 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:15.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:15.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:57:15.345 INFO:tasks.rados.rados.0.smithi125.stdout:850: expect (ObjNum 150 snap 30 seq_num 150) 2021-03-24T21:57:15.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:15 smithi142 conmon[20125]: cluster 2021-03-24T21:57:13.701134+0000 osd.4 (osd.4) 69 : cluster [DBG] 2021-03-24T21:57:15.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:15 smithi142 conmon[20125]: 3.1a scrub ok 2021-03-24T21:57:15.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:15 smithi142 conmon[20125]: cluster 2021-03-24T21:57:14.170622+0000 osd.3 (osd.3) 67 : cluster [DBG] 2021-03-24T21:57:15.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:15 smithi142 conmon[20125]: purged_snaps scrub starts 2021-03-24T21:57:15.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:15 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:57:15.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:15 smithi142 conmon[20125]: 24T21:57:14.171071+0000 osd.3 (osd.3 2021-03-24T21:57:15.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:15 smithi142 conmon[20125]: ) 68 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:57:15.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:15 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:57:15.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:15 smithi142 conmon[20125]: 24T21:57:14.199930+0000 mon.a ( 2021-03-24T21:57:15.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:15 smithi142 conmon[20125]: mon.0) 911 : cluster [DBG] osdmap e265: 8 total, 8 up, 6 in 2021-03-24T21:57:15.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:15 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:57:15.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:15 smithi142 conmon[20125]: -03-24T21:57:14.281734 2021-03-24T21:57:15.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:15 smithi142 conmon[20125]: +0000 osd.7 (osd.7) 60 : cluster 2021-03-24T21:57:15.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:15 smithi142 conmon[20125]: [DBG] 7.17 scrub ok 2021-03-24T21:57:15.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:15 smithi142 conmon[20125]: audit 2021-03-24T21:57:15.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:15 smithi142 conmon[20125]: 2021-03-24T21:57:14. 2021-03-24T21:57:15.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:15 smithi142 conmon[20125]: 959450+0000 mon.c (mon.2) 41 : audit 2021-03-24T21:57:15.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:15 smithi142 conmon[20125]: [DBG] from='client.? v1:172.21.15.125:0/100638348' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:57:15.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:15 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:57:15.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:15 smithi142 conmon[20125]: 57:15.145583+0000 mon.a (mon.0 2021-03-24T21:57:15.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:15 smithi142 conmon[20125]: ) 912 : cluster [DBG] osdmap e266: 8 total, 8 up, 6 in 2021-03-24T21:57:15.401 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:15.456 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:15.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:15.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:57:15.473 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-24T21:57:15.476 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2021-03-24T21:57:15.479 INFO:tasks.rados.rados.0.smithi125.stdout:849: done (2 left) 2021-03-24T21:57:15.479 INFO:tasks.rados.rados.0.smithi125.stdout:850: done (1 left) 2021-03-24T21:57:15.479 INFO:tasks.rados.rados.0.smithi125.stdout:851: done (0 left) 2021-03-24T21:57:15.479 INFO:tasks.rados.rados.0.smithi125.stdout:852: rollback oid 37 current snap is 82 2021-03-24T21:57:15.479 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 37 to 81 2021-03-24T21:57:15.480 INFO:tasks.rados.rados.0.smithi125.stdout:853: read oid 19 snap 78 2021-03-24T21:57:15.480 INFO:tasks.rados.rados.0.smithi125.stdout:853: expect (ObjNum 244 snap 57 seq_num 244) 2021-03-24T21:57:15.480 INFO:tasks.rados.rados.0.smithi125.stdout:854: read oid 34 snap -1 2021-03-24T21:57:15.480 INFO:tasks.rados.rados.0.smithi125.stdout:854: expect (ObjNum 203 snap 43 seq_num 203) 2021-03-24T21:57:15.480 INFO:tasks.rados.rados.0.smithi125.stdout:855: copy_from oid 31 from oid 50 current snap is 82 2021-03-24T21:57:15.484 INFO:tasks.rados.rados.0.smithi125.stdout:856: rollback oid 7 current snap is 82 2021-03-24T21:57:15.484 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 7 to 77 2021-03-24T21:57:15.485 INFO:tasks.rados.rados.0.smithi125.stdout:857: read oid 27 snap -1 2021-03-24T21:57:15.485 INFO:tasks.rados.rados.0.smithi125.stdout:857: expect (ObjNum 193 snap 42 seq_num 193) 2021-03-24T21:57:15.485 INFO:tasks.rados.rados.0.smithi125.stdout:858: read oid 46 snap 81 2021-03-24T21:57:15.485 INFO:tasks.rados.rados.0.smithi125.stdout:858: expect (ObjNum 315 snap 80 seq_num 315) 2021-03-24T21:57:15.485 INFO:tasks.rados.rados.0.smithi125.stdout:859: write oid 26 current snap is 82 2021-03-24T21:57:15.486 INFO:tasks.rados.rados.0.smithi125.stdout:859: seq_num 321 ranges {595047=641755,1667661=418109,2548663=763022} 2021-03-24T21:57:15.493 INFO:tasks.rados.rados.0.smithi125.stdout:859: writing smithi12556470-26 from 595047 to 1236802 tid 1 2021-03-24T21:57:15.498 INFO:tasks.rados.rados.0.smithi125.stdout:859: writing smithi12556470-26 from 1667661 to 2085770 tid 2 2021-03-24T21:57:15.505 INFO:tasks.rados.rados.0.smithi125.stdout:859: writing smithi12556470-26 from 2548663 to 3311685 tid 3 2021-03-24T21:57:15.505 INFO:tasks.rados.rados.0.smithi125.stdout:860: write (excl) oid 8 current snap is 82 2021-03-24T21:57:15.505 INFO:tasks.rados.rados.0.smithi125.stdout:860: seq_num 322 ranges {611854=717004,2126143=561767,3259733=152121} 2021-03-24T21:57:15.512 INFO:tasks.rados.rados.0.smithi125.stdout:860: writing smithi12556470-8 from 611854 to 1328858 tid 1 2021-03-24T21:57:15.520 INFO:tasks.rados.rados.0.smithi125.stdout:860: writing smithi12556470-8 from 2126143 to 2687910 tid 2 2021-03-24T21:57:15.524 INFO:tasks.rados.rados.0.smithi125.stdout:860: writing smithi12556470-8 from 3259733 to 3411854 tid 3 2021-03-24T21:57:15.525 INFO:tasks.rados.rados.0.smithi125.stdout:861: write oid 48 current snap is 82 2021-03-24T21:57:15.525 INFO:tasks.rados.rados.0.smithi125.stdout:861: seq_num 323 ranges {628661=792253,2184625=421778,3403509=25152} 2021-03-24T21:57:15.532 INFO:tasks.rados.rados.0.smithi125.stdout:861: writing smithi12556470-48 from 628661 to 1420914 tid 1 2021-03-24T21:57:15.539 INFO:tasks.rados.rados.0.smithi125.stdout:861: writing smithi12556470-48 from 2184625 to 2606403 tid 2 2021-03-24T21:57:15.544 INFO:tasks.rados.rados.0.smithi125.stdout:861: writing smithi12556470-48 from 3403509 to 3428661 tid 3 2021-03-24T21:57:15.544 INFO:tasks.rados.rados.0.smithi125.stdout:862: copy_from oid 44 from oid 39 current snap is 82 2021-03-24T21:57:15.544 INFO:tasks.rados.rados.0.smithi125.stdout:853: expect (ObjNum 244 snap 57 seq_num 244) 2021-03-24T21:57:15.563 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:15.610 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:15.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:15.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:57:15.632 INFO:tasks.rados.rados.0.smithi125.stdout:852: finishing rollback tid 0 to smithi12556470-37 2021-03-24T21:57:15.632 INFO:tasks.rados.rados.0.smithi125.stdout:854: expect (ObjNum 203 snap 43 seq_num 203) 2021-03-24T21:57:15.715 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:15.766 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:15.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:15.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:57:15.799 INFO:tasks.rados.rados.0.smithi125.stdout:852: finishing rollback tid 1 to smithi12556470-37 2021-03-24T21:57:15.800 INFO:tasks.rados.rados.0.smithi125.stdout:852: finishing rollback tid 2 to smithi12556470-37 2021-03-24T21:57:15.800 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 37 v 91 (ObjNum 229 snap 50 seq_num 229) dirty exists 2021-03-24T21:57:15.800 INFO:tasks.rados.rados.0.smithi125.stdout:858: expect (ObjNum 315 snap 80 seq_num 315) 2021-03-24T21:57:15.873 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:15.905 INFO:tasks.rados.rados.0.smithi125.stdout:857: expect (ObjNum 193 snap 42 seq_num 193) 2021-03-24T21:57:15.924 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:15.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:15.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:16.031 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:16.078 INFO:tasks.rados.rados.0.smithi125.stdout:856: finishing rollback tid 0 to smithi12556470-7 2021-03-24T21:57:16.079 INFO:tasks.rados.rados.0.smithi125.stdout:856: finishing rollback tid 1 to smithi12556470-7 2021-03-24T21:57:16.079 INFO:tasks.rados.rados.0.smithi125.stdout:856: finishing rollback tid 2 to smithi12556470-7 2021-03-24T21:57:16.079 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 7 v 198 (ObjNum 150 snap 30 seq_num 150) dirty exists 2021-03-24T21:57:16.079 INFO:tasks.rados.rados.0.smithi125.stdout:859: finishing write tid 1 to smithi12556470-26 2021-03-24T21:57:16.080 INFO:tasks.rados.rados.0.smithi125.stdout:859: finishing write tid 2 to smithi12556470-26 2021-03-24T21:57:16.080 INFO:tasks.rados.rados.0.smithi125.stdout:855: finishing copy_from to smithi12556470-31 2021-03-24T21:57:16.080 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 31 v 128 (ObjNum 274 snap 68 seq_num 274) dirty exists 2021-03-24T21:57:16.080 INFO:tasks.rados.rados.0.smithi125.stdout:855: finishing copy_from racing read to smithi12556470-31 2021-03-24T21:57:16.081 INFO:tasks.rados.rados.0.smithi125.stdout:859: finishing write tid 3 to smithi12556470-26 2021-03-24T21:57:16.081 INFO:tasks.rados.rados.0.smithi125.stdout:859: finishing write tid 4 to smithi12556470-26 2021-03-24T21:57:16.081 INFO:tasks.rados.rados.0.smithi125.stdout:859: finishing write tid 5 to smithi12556470-26 2021-03-24T21:57:16.081 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 26 v 141 (ObjNum 321 snap 82 seq_num 321) dirty exists 2021-03-24T21:57:16.081 INFO:tasks.rados.rados.0.smithi125.stdout:859: left oid 26 (ObjNum 321 snap 82 seq_num 321) 2021-03-24T21:57:16.082 INFO:tasks.rados.rados.0.smithi125.stdout:860: finishing write tid 1 to smithi12556470-8 2021-03-24T21:57:16.082 INFO:tasks.rados.rados.0.smithi125.stdout:860: finishing write tid 2 to smithi12556470-8 2021-03-24T21:57:16.082 INFO:tasks.rados.rados.0.smithi125.stdout:860: finishing write tid 3 to smithi12556470-8 2021-03-24T21:57:16.082 INFO:tasks.rados.rados.0.smithi125.stdout:860: finishing write tid 4 to smithi12556470-8 2021-03-24T21:57:16.083 INFO:tasks.rados.rados.0.smithi125.stdout:852: done (10 left) 2021-03-24T21:57:16.083 INFO:tasks.rados.rados.0.smithi125.stdout:853: done (9 left) 2021-03-24T21:57:16.083 INFO:tasks.rados.rados.0.smithi125.stdout:854: done (8 left) 2021-03-24T21:57:16.083 INFO:tasks.rados.rados.0.smithi125.stdout:855: done (7 left) 2021-03-24T21:57:16.083 INFO:tasks.rados.rados.0.smithi125.stdout:856: done (6 left) 2021-03-24T21:57:16.084 INFO:tasks.rados.rados.0.smithi125.stdout:857: done (5 left) 2021-03-24T21:57:16.084 INFO:tasks.rados.rados.0.smithi125.stdout:858: done (4 left) 2021-03-24T21:57:16.084 INFO:tasks.rados.rados.0.smithi125.stdout:859: done (3 left) 2021-03-24T21:57:16.084 INFO:tasks.rados.rados.0.smithi125.stdout:863: read oid 6 snap -1 2021-03-24T21:57:16.084 INFO:tasks.rados.rados.0.smithi125.stdout:863: expect (ObjNum 123 snap 23 seq_num 123) 2021-03-24T21:57:16.085 INFO:tasks.rados.rados.0.smithi125.stdout:864: read oid 4 snap -1 2021-03-24T21:57:16.085 INFO:tasks.rados.rados.0.smithi125.stdout:864: expect (ObjNum 319 snap 81 seq_num 319) 2021-03-24T21:57:16.085 INFO:tasks.rados.rados.0.smithi125.stdout:860: finishing write tid 5 to smithi12556470-8 2021-03-24T21:57:16.085 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 8 v 49 (ObjNum 322 snap 82 seq_num 322) dirty exists 2021-03-24T21:57:16.086 INFO:tasks.rados.rados.0.smithi125.stdout:860: left oid 8 (ObjNum 322 snap 82 seq_num 322) 2021-03-24T21:57:16.086 INFO:tasks.rados.rados.0.smithi125.stdout:861: finishing write tid 1 to smithi12556470-48 2021-03-24T21:57:16.086 INFO:tasks.rados.rados.0.smithi125.stdout:861: finishing write tid 2 to smithi12556470-48 2021-03-24T21:57:16.086 INFO:tasks.rados.rados.0.smithi125.stdout:861: finishing write tid 3 to smithi12556470-48 2021-03-24T21:57:16.086 INFO:tasks.rados.rados.0.smithi125.stdout:865: read oid 13 snap 77 2021-03-24T21:57:16.087 INFO:tasks.rados.rados.0.smithi125.stdout:865: expect (ObjNum 251 snap 59 seq_num 251) 2021-03-24T21:57:16.087 INFO:tasks.rados.rados.0.smithi125.stdout:860: done (5 left) 2021-03-24T21:57:16.087 INFO:tasks.rados.rados.0.smithi125.stdout:866: read oid 31 snap -1 2021-03-24T21:57:16.087 INFO:tasks.rados.rados.0.smithi125.stdout:866: expect (ObjNum 274 snap 68 seq_num 274) 2021-03-24T21:57:16.088 INFO:tasks.rados.rados.0.smithi125.stdout:867: read oid 20 snap -1 2021-03-24T21:57:16.088 INFO:tasks.rados.rados.0.smithi125.stdout:867: expect (ObjNum 298 snap 74 seq_num 298) 2021-03-24T21:57:16.088 INFO:tasks.rados.rados.0.smithi125.stdout:868: write (excl) oid 25 current snap is 82 2021-03-24T21:57:16.089 INFO:tasks.rados.rados.0.smithi125.stdout:868: seq_num 324 ranges {645468=467502,1726754=565436,2798226=596838} 2021-03-24T21:57:16.089 INFO:tasks.rados.rados.0.smithi125.stdout:868: writing smithi12556470-25 from 645468 to 1112970 tid 1 2021-03-24T21:57:16.089 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:16.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:16.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:57:16.094 INFO:tasks.rados.rados.0.smithi125.stdout:868: writing smithi12556470-25 from 1726754 to 2292190 tid 2 2021-03-24T21:57:16.100 INFO:tasks.rados.rados.0.smithi125.stdout:868: writing smithi12556470-25 from 2798226 to 3395064 tid 3 2021-03-24T21:57:16.100 INFO:tasks.rados.rados.0.smithi125.stdout:869: write oid 2 current snap is 82 2021-03-24T21:57:16.101 INFO:tasks.rados.rados.0.smithi125.stdout:861: finishing write tid 4 to smithi12556470-48 2021-03-24T21:57:16.101 INFO:tasks.rados.rados.0.smithi125.stdout:869: seq_num 325 ranges {662275=542751,1785236=425447,2942002=520273} 2021-03-24T21:57:16.106 INFO:tasks.rados.rados.0.smithi125.stdout:869: writing smithi12556470-2 from 662275 to 1205026 tid 1 2021-03-24T21:57:16.112 INFO:tasks.rados.rados.0.smithi125.stdout:869: writing smithi12556470-2 from 1785236 to 2210683 tid 2 2021-03-24T21:57:16.119 INFO:tasks.rados.rados.0.smithi125.stdout:869: writing smithi12556470-2 from 2942002 to 3462275 tid 3 2021-03-24T21:57:16.120 INFO:tasks.rados.rados.0.smithi125.stdout:870: copy_from oid 36 from oid 7 current snap is 82 2021-03-24T21:57:16.120 INFO:tasks.rados.rados.0.smithi125.stdout:861: finishing write tid 5 to smithi12556470-48 2021-03-24T21:57:16.120 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 48 v 279 (ObjNum 323 snap 82 seq_num 323) dirty exists 2021-03-24T21:57:16.120 INFO:tasks.rados.rados.0.smithi125.stdout:861: left oid 48 (ObjNum 323 snap 82 seq_num 323) 2021-03-24T21:57:16.120 INFO:tasks.rados.rados.0.smithi125.stdout:862: finishing copy_from to smithi12556470-44 2021-03-24T21:57:16.121 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 44 v 260 (ObjNum 309 snap 76 seq_num 309) dirty exists 2021-03-24T21:57:16.121 INFO:tasks.rados.rados.0.smithi125.stdout:862: finishing copy_from racing read to smithi12556470-44 2021-03-24T21:57:16.121 INFO:tasks.rados.rados.0.smithi125.stdout:861: done (9 left) 2021-03-24T21:57:16.121 INFO:tasks.rados.rados.0.smithi125.stdout:862: done (8 left) 2021-03-24T21:57:16.122 INFO:tasks.rados.rados.0.smithi125.stdout:871: copy_from oid 19 from oid 16 current snap is 82 2021-03-24T21:57:16.122 INFO:tasks.rados.rados.0.smithi125.stdout:865: expect (ObjNum 251 snap 59 seq_num 251) 2021-03-24T21:57:16.195 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:16.198 INFO:tasks.rados.rados.0.smithi125.stdout:863: expect (ObjNum 123 snap 23 seq_num 123) 2021-03-24T21:57:16.243 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:16.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:16.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:57:16.317 INFO:tasks.rados.rados.0.smithi125.stdout:867: expect (ObjNum 298 snap 74 seq_num 298) 2021-03-24T21:57:16.348 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:16.361 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:16 smithi125 conmon[24927]: cluster 2021-03-24T21:57:14.768167 2021-03-24T21:57:16.362 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:16 smithi125 conmon[24927]: +0000 mgr.y (mgr.14144) 398 : cluster [DBG] pgmap v560: 158 pgs: 158 active+clean; 146 MiB data, 504 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:57:16.362 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:16 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:57:16.362 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:16 smithi125 conmon[24927]: -03-24T21:57:15.071478+0000 osd.2 (osd.2) 52 : cluster 2021-03-24T21:57:16.363 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:16 smithi125 conmon[24927]: [DBG] 4.4 scrub ok 2021-03-24T21:57:16.363 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:16 smithi125 conmon[24927]: audit 2021-03-24T21:57:16.363 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:16 smithi125 conmon[24927]: 2021-03-24T21:57:15.446365+0000 mgr.y (mgr. 2021-03-24T21:57:16.363 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:16 smithi125 conmon[24927]: 14144) 399 : audit [DBG] from='client.34967 v1:172.21.15.125:0/214975654' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T21:57:16.364 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:16 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:57:16.364 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:16 smithi125 conmon[19936]: 24T21:57:14.768167+0000 mgr.y (mgr.14144) 398 : cluster [DBG] pgmap v560: 158 pgs: 158 active+clean; 146 MiB data, 504 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:57:16.365 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:16 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:57:16.365 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:16 smithi125 conmon[19936]: -03-24T21 2021-03-24T21:57:16.365 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:16 smithi125 conmon[19936]: :57:15.071478 2021-03-24T21:57:16.365 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:16 smithi125 conmon[19936]: +0000 osd.2 (osd 2021-03-24T21:57:16.365 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:16 smithi125 conmon[19936]: .2) 52 : cluster [DBG] 2021-03-24T21:57:16.366 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:16 smithi125 conmon[19936]: 4.4 scrub ok 2021-03-24T21:57:16.366 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:16 smithi125 conmon[19936]: audit 2021-03-24T21:57:16.366 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:16 smithi125 conmon[19936]: 2021-03-24 2021-03-24T21:57:16.367 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:16 smithi125 conmon[19936]: T21:57:15 2021-03-24T21:57:16.367 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:16 smithi125 conmon[19936]: .446365+0000 mgr.y (mgr. 2021-03-24T21:57:16.367 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:16 smithi125 conmon[19936]: 14144) 399 : audit [DBG] 2021-03-24T21:57:16.367 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:16 smithi125 conmon[19936]: from='client.34967 v1:172.21.15.125:0/214975654' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T21:57:16.391 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-24T21:57:16.405 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:16.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:16.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:57:16.506 INFO:tasks.rados.rados.0.smithi125.stdout:864: expect (ObjNum 319 snap 81 seq_num 319) 2021-03-24T21:57:16.512 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:16.524 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:16 smithi142 conmon[20125]: cluster 2021-03-24T21:57:14.768167+0000 mgr.y (mgr.14144) 398 : cluster 2021-03-24T21:57:16.524 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:16 smithi142 conmon[20125]: [DBG] pgmap v560: 158 pgs: 158 active+clean; 146 MiB data, 504 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:57:16.524 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:16 smithi142 conmon[20125]: cluster 2021-03-24T21:57:15.071478+0000 osd.2 (osd.2) 52 : cluster [DBG] 4.4 scrub ok 2021-03-24T21:57:16.525 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:16 smithi142 conmon[20125]: 2021-03-24T21:57:16.525 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:16 smithi142 conmon[20125]: audit 2021-03-24 2021-03-24T21:57:16.525 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:16 smithi142 conmon[20125]: T21:57:15.446365+0000 mgr.y (mgr.14144) 399 : audit 2021-03-24T21:57:16.525 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:16 smithi142 conmon[20125]: [DBG] from='client.34967 v1:172.21.15.125:0/214975654' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T21:57:16.564 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:16.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:16.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:16.671 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:16.697 INFO:tasks.rados.rados.0.smithi125.stdout:868: finishing write tid 1 to smithi12556470-25 2021-03-24T21:57:16.697 INFO:tasks.rados.rados.0.smithi125.stdout:863: done (8 left) 2021-03-24T21:57:16.698 INFO:tasks.rados.rados.0.smithi125.stdout:864: done (7 left) 2021-03-24T21:57:16.698 INFO:tasks.rados.rados.0.smithi125.stdout:865: done (6 left) 2021-03-24T21:57:16.698 INFO:tasks.rados.rados.0.smithi125.stdout:867: done (5 left) 2021-03-24T21:57:16.698 INFO:tasks.rados.rados.0.smithi125.stdout:872: snap_remove snap 80 2021-03-24T21:57:16.731 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:16.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:16.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:16.837 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:16.891 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:16.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:16.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:57:16.997 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:17.052 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:17.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:17.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:57:17.159 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:17.214 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:17.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:17.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:57:17.232 INFO:tasks.rados.rados.0.smithi125.stdout:872: done (5 left) 2021-03-24T21:57:17.232 INFO:tasks.rados.rados.0.smithi125.stdout:873: read oid 9 snap -1 2021-03-24T21:57:17.232 INFO:tasks.rados.rados.0.smithi125.stdout:873: expect (ObjNum 203 snap 43 seq_num 203) 2021-03-24T21:57:17.233 INFO:tasks.rados.rados.0.smithi125.stdout:874: copy_from oid 22 from oid 13 current snap is 82 2021-03-24T21:57:17.234 INFO:tasks.rados.rados.0.smithi125.stdout:875: rollback oid 5 current snap is 82 2021-03-24T21:57:17.234 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 5 to 75 2021-03-24T21:57:17.234 INFO:tasks.rados.rados.0.smithi125.stdout:876: read oid 14 snap -1 2021-03-24T21:57:17.234 INFO:tasks.rados.rados.0.smithi125.stdout:876: expect (ObjNum 303 snap 74 seq_num 303) 2021-03-24T21:57:17.235 INFO:tasks.rados.rados.0.smithi125.stdout:877: snap_create 2021-03-24T21:57:17.235 INFO:tasks.rados.rados.0.smithi125.stdout:866: expect (ObjNum 274 snap 68 seq_num 274) 2021-03-24T21:57:17.235 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:57:17.249 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":561,"stamp":"2021-03-24T21:57:15.143617+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":152804239,"num_objects":383,"num_object_clones":24,"num_object_copies":1149,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":383,"num_whiteouts":3,"num_read":3015,"num_read_kb":1653299,"num_write":1671,"num_write_kb":439764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":506,"num_bytes_recovered":100973202,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4499,"ondisk_log_size":4499,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":515784,"kb_used_data":364440,"kb_used_omap":0,"kb_used_meta":151296,"kb_avail":561955128,"statfs":{"total":575970213888,"available":575442051072,"internally_reserved":0,"allocated":373186560,"data_stored":428348864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":154927104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4163269,"num_objects":-2,"num_object_clones":0,"num_object_copies":-6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":-4,"num_read":-22,"num_read_kb":-20747,"num_write":-55,"num_write_kb":-13682,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-22,"num_bytes_recovered":-5318545,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001615"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":"222","reported_epoch":"262","state":"active+clean","last_fresh":"2021-03-24T21:57:11.144897+0000","last_change":"2021-03-24T21:54:50.292202+0000","last_active":"2021-03-24T21:57:11.144897+0000","last_peered":"2021-03-24T21:57:11.144897+0000","last_clean":"2021-03-24T21:57:11.144897+0000","last_became_active":"2021-03-24T21:54:38.111248+0000","last_became_peered":"2021-03-24T21:54:38.111248+0000","last_unstale":"2021-03-24T21:57:11.144897+0000","last_undegraded":"2021-03-24T21:57:11.144897+0000","last_fullsized":"2021-03-24T21:57:11.144897+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:50.292025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:54:50.292025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"144","reported_epoch":"264","state":"active+clean","last_fresh":"2021-03-24T21:57:13.206983+0000","last_change":"2021-03-24T21:56:34.003068+0000","last_active":"2021-03-24T21:57:13.206983+0000","last_peered":"2021-03-24T21:57:13.206983+0000","last_clean":"2021-03-24T21:57:13.206983+0000","last_became_active":"2021-03-24T21:55:10.099666+0000","last_became_peered":"2021-03-24T21:55:10.099666+0000","last_unstale":"2021-03-24T21:57:13.206983+0000","last_undegraded":"2021-03-24T21:57:13.206983+0000","last_fullsized":"2021-03-24T21:57:13.206983+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:34.002906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:14.710709+0000","last_clean_scrub_stamp":"2021-03-24T21:56:34.002906+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":"228","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.214569+0000","last_change":"2021-03-24T21:56:44.230467+0000","last_active":"2021-03-24T21:57:12.214569+0000","last_peered":"2021-03-24T21:57:12.214569+0000","last_clean":"2021-03-24T21:57:12.214569+0000","last_became_active":"2021-03-24T21:56:44.229902+0000","last_became_peered":"2021-03-24T21:56:44.229902+0000","last_unstale":"2021-03-24T21:57:12.214569+0000","last_undegraded":"2021-03-24T21:57:12.214569+0000","last_fullsized":"2021-03-24T21:57:12.214569+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:38.294685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:38.294685+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"69'15","reported_seq":"307","reported_epoch":"262","state":"active+clean","last_fresh":"2021-03-24T21:57:11.144438+0000","last_change":"2021-03-24T21:56:05.966329+0000","last_active":"2021-03-24T21:57:11.144438+0000","last_peered":"2021-03-24T21:57:11.144438+0000","last_clean":"2021-03-24T21:57:11.144438+0000","last_became_active":"2021-03-24T21:54:38.127426+0000","last_became_peered":"2021-03-24T21:54:38.127426+0000","last_unstale":"2021-03-24T21:57:11.144438+0000","last_undegraded":"2021-03-24T21:57:11.144438+0000","last_fullsized":"2021-03-24T21:57:11.144438+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:56:05.966219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:05.966219+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2076,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"231","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.191696+0000","last_change":"2021-03-24T21:54:43.482408+0000","last_active":"2021-03-24T21:57:12.191696+0000","last_peered":"2021-03-24T21:57:12.191696+0000","last_clean":"2021-03-24T21:57:12.191696+0000","last_became_active":"2021-03-24T21:54:38.130293+0000","last_became_peered":"2021-03-24T21:54:38.130293+0000","last_unstale":"2021-03-24T21:57:12.191696+0000","last_undegraded":"2021-03-24T21:57:12.191696+0000","last_fullsized":"2021-03-24T21:57:12.191696+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:43.482307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:54:43.482307+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"233","reported_epoch":"262","state":"active+clean","last_fresh":"2021-03-24T21:57:11.144875+0000","last_change":"2021-03-24T21:56:34.092326+0000","last_active":"2021-03-24T21:57:11.144875+0000","last_peered":"2021-03-24T21:57:11.144875+0000","last_clean":"2021-03-24T21:57:11.144875+0000","last_became_active":"2021-03-24T21:54:38.120575+0000","last_became_peered":"2021-03-24T21:54:38.120575+0000","last_unstale":"2021-03-24T21:57:11.144875+0000","last_undegraded":"2021-03-24T21:57:11.144875+0000","last_fullsized":"2021-03-24T21:57:11.144875+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:34.092227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:34.092227+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":"237","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.192838+0000","last_change":"2021-03-24T21:55:38.433064+0000","last_active":"2021-03-24T21:57:12.192838+0000","last_peered":"2021-03-24T21:57:12.192838+0000","last_clean":"2021-03-24T21:57:12.192838+0000","last_became_active":"2021-03-24T21:54:38.130163+0000","last_became_peered":"2021-03-24T21:54:38.130163+0000","last_unstale":"2021-03-24T21:57:12.192838+0000","last_undegraded":"2021-03-24T21:57:12.192838+0000","last_fullsized":"2021-03-24T21:57:12.192838+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:38.432918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:38.432918+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"69'20","reported_seq":"252","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.191280+0000","last_change":"2021-03-24T21:54:54.541380+0000","last_active":"2021-03-24T21:57:12.191280+0000","last_peered":"2021-03-24T21:57:12.191280+0000","last_clean":"2021-03-24T21:57:12.191280+0000","last_became_active":"2021-03-24T21:51:47.405776+0000","last_became_peered":"2021-03-24T21:51:47.405776+0000","last_unstale":"2021-03-24T21:57:12.191280+0000","last_undegraded":"2021-03-24T21:57:12.191280+0000","last_fullsized":"2021-03-24T21:57:12.191280+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'20","last_scrub_stamp":"2021-03-24T21:54:54.541183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:54:54.541183+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":22,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"265'260","reported_seq":"873","reported_epoch":"265","state":"active+clean","last_fresh":"2021-03-24T21:57:14.217197+0000","last_change":"2021-03-24T21:57:14.217579+0000","last_active":"2021-03-24T21:57:14.217197+0000","last_peered":"2021-03-24T21:57:14.217197+0000","last_clean":"2021-03-24T21:57:14.217197+0000","last_became_active":"2021-03-24T21:56:14.139614+0000","last_became_peered":"2021-03-24T21:56:14.139614+0000","last_unstale":"2021-03-24T21:57:14.217197+0000","last_undegraded":"2021-03-24T21:57:14.217197+0000","last_fullsized":"2021-03-24T21:57:14.217197+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":260,"ondisk_log_size":260,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":124,"num_read_kb":138417,"num_write":69,"num_write_kb":29440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":10280944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"82","length":"1"},{"start":"8b","length":"3"},{"start":"92","length":"1"},{"start":"95","length":"1"},{"start":"9c","length":"1"}]},{"pgid":"6.18","version":"0'0","reported_seq":"222","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.203416+0000","last_change":"2021-03-24T21:54:49.409465+0000","last_active":"2021-03-24T21:57:12.203416+0000","last_peered":"2021-03-24T21:57:12.203416+0000","last_clean":"2021-03-24T21:57:12.203416+0000","last_became_active":"2021-03-24T21:54:38.127696+0000","last_became_peered":"2021-03-24T21:54:38.127696+0000","last_unstale":"2021-03-24T21:57:12.203416+0000","last_undegraded":"2021-03-24T21:57:12.203416+0000","last_fullsized":"2021-03-24T21:57:12.203416+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:49.409299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:54:49.409299+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"225","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.180105+0000","last_change":"2021-03-24T21:56:38.242245+0000","last_active":"2021-03-24T21:57:12.180105+0000","last_peered":"2021-03-24T21:57:12.180105+0000","last_clean":"2021-03-24T21:57:12.180105+0000","last_became_active":"2021-03-24T21:51:49.413342+0000","last_became_peered":"2021-03-24T21:51:49.413342+0000","last_unstale":"2021-03-24T21:57:12.180105+0000","last_undegraded":"2021-03-24T21:57:12.180105+0000","last_fullsized":"2021-03-24T21:57:12.180105+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:38.242096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:38.242096+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":"250","reported_epoch":"261","state":"active+clean","last_fresh":"2021-03-24T21:57:10.144457+0000","last_change":"2021-03-24T21:56:32.141420+0000","last_active":"2021-03-24T21:57:10.144457+0000","last_peered":"2021-03-24T21:57:10.144457+0000","last_clean":"2021-03-24T21:57:10.144457+0000","last_became_active":"2021-03-24T21:55:10.097431+0000","last_became_peered":"2021-03-24T21:55:10.097431+0000","last_unstale":"2021-03-24T21:57:10.144457+0000","last_undegraded":"2021-03-24T21:57:10.144457+0000","last_fullsized":"2021-03-24T21:57:10.144457+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:32.141276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:32.141276+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"69'13","reported_seq":"241","reported_epoch":"262","state":"active+clean","last_fresh":"2021-03-24T21:57:11.145458+0000","last_change":"2021-03-24T21:55:31.085064+0000","last_active":"2021-03-24T21:57:11.145458+0000","last_peered":"2021-03-24T21:57:11.145458+0000","last_clean":"2021-03-24T21:57:11.145458+0000","last_became_active":"2021-03-24T21:51:47.407488+0000","last_became_peered":"2021-03-24T21:51:47.407488+0000","last_unstale":"2021-03-24T21:57:11.145458+0000","last_undegraded":"2021-03-24T21:57:11.145458+0000","last_fullsized":"2021-03-24T21:57:11.145458+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2021-03-24T21:55:31.084873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:31.084873+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"249'90","reported_seq":"229","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.203924+0000","last_change":"2021-03-24T21:57:12.203924+0000","last_active":"2021-03-24T21:57:12.203924+0000","last_peered":"2021-03-24T21:57:12.203924+0000","last_clean":"2021-03-24T21:57:12.203924+0000","last_became_active":"2021-03-24T21:56:10.151682+0000","last_became_peered":"2021-03-24T21:56:10.151682+0000","last_unstale":"2021-03-24T21:57:12.203924+0000","last_undegraded":"2021-03-24T21:57:12.203924+0000","last_fullsized":"2021-03-24T21:57:12.203924+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":90,"ondisk_log_size":90,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252122,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":6611,"num_write":17,"num_write_kb":4246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"82","length":"1"},{"start":"8c","length":"2"},{"start":"92","length":"1"},{"start":"95","length":"1"},{"start":"9c","length":"1"}]},{"pgid":"6.19","version":"0'0","reported_seq":"225","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.172338+0000","last_change":"2021-03-24T21:55:39.435565+0000","last_active":"2021-03-24T21:57:12.172338+0000","last_peered":"2021-03-24T21:57:12.172338+0000","last_clean":"2021-03-24T21:57:12.172338+0000","last_became_active":"2021-03-24T21:54:38.122349+0000","last_became_peered":"2021-03-24T21:54:38.122349+0000","last_unstale":"2021-03-24T21:57:12.172338+0000","last_undegraded":"2021-03-24T21:57:12.172338+0000","last_fullsized":"2021-03-24T21:57:12.172338+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:39.435268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:55:39.435268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"231","reported_epoch":"264","state":"active+clean","last_fresh":"2021-03-24T21:57:13.202299+0000","last_change":"2021-03-24T21:56:51.977473+0000","last_active":"2021-03-24T21:57:13.202299+0000","last_peered":"2021-03-24T21:57:13.202299+0000","last_clean":"2021-03-24T21:57:13.202299+0000","last_became_active":"2021-03-24T21:54:38.137706+0000","last_became_peered":"2021-03-24T21:54:38.137706+0000","last_unstale":"2021-03-24T21:57:13.202299+0000","last_undegraded":"2021-03-24T21:57:13.202299+0000","last_fullsized":"2021-03-24T21:57:13.202299+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:51.977306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:56:51.977306+0000","last_clean_scrub_stamp":"2021-03-24T21:56:51.977306+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":"208","reported_epoch":"262","state":"active+clean","last_fresh":"2021-03-24T21:57:11.146387+0000","last_change":"2021-03-24T21:56:47.921029+0000","last_active":"2021-03-24T21:57:11.146387+0000","last_peered":"2021-03-24T21:57:11.146387+0000","last_clean":"2021-03-24T21:57:11.146387+0000","last_became_active":"2021-03-24T21:55:10.098790+0000","last_became_peered":"2021-03-24T21:55:10.098790+0000","last_unstale":"2021-03-24T21:57:11.146387+0000","last_undegraded":"2021-03-24T21:57:11.146387+0000","last_fullsized":"2021-03-24T21:57:11.146387+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:47.920949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:47.920949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"91'9","reported_seq":"266","reported_epoch":"262","state":"active+clean","last_fresh":"2021-03-24T21:57:11.146120+0000","last_change":"2021-03-24T21:57:02.447431+0000","last_active":"2021-03-24T21:57:11.146120+0000","last_peered":"2021-03-24T21:57:11.146120+0000","last_clean":"2021-03-24T21:57:11.146120+0000","last_became_active":"2021-03-24T21:57:02.184317+0000","last_became_peered":"2021-03-24T21:57:02.184317+0000","last_unstale":"2021-03-24T21:57:11.146120+0000","last_undegraded":"2021-03-24T21:57:11.146120+0000","last_fullsized":"2021-03-24T21:57:11.146120+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2021-03-24T21:56:04.051464+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2021-03-24T21:53:33.290000+0000","last_clean_scrub_stamp":"2021-03-24T21:56:04.051464+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":8,"num_write":7,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"222","reported_epoch":"262","state":"active+clean","last_fresh":"2021-03-24T21:57:11.147733+0000","last_change":"2021-03-24T21:56:38.075890+0000","last_active":"2021-03-24T21:57:11.147733+0000","last_peered":"2021-03-24T21:57:11.147733+0000","last_clean":"2021-03-24T21:57:11.147733+0000","last_became_active":"2021-03-24T21:54:24.199444+0000","last_became_peered":"2021-03-24T21:54:24.199444+0000","last_unstale":"2021-03-24T21:57:11.147733+0000","last_undegraded":"2021-03-24T21:57:11.147733+0000","last_fullsized":"2021-03-24T21:57:11.147733+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:38.075714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:38.075714+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"171","reported_epoch":"262","state":"active+clean","last_fresh":"2021-03-24T21:57:11.145657+0000","last_change":"2021-03-24T21:54:49.323954+0000","last_active":"2021-03-24T21:57:11.145657+0000","last_peered":"2021-03-24T21:57:11.145657+0000","last_clean":"2021-03-24T21:57:11.145657+0000","last_became_active":"2021-03-24T21:54:38.138620+0000","last_became_peered":"2021-03-24T21:54:38.138620+0000","last_unstale":"2021-03-24T21:57:11.145657+0000","last_undegraded":"2021-03-24T21:57:11.145657+0000","last_fullsized":"2021-03-24T21:57:11.145657+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:49.323789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:54:49.323789+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":"224","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.208452+0000","last_change":"2021-03-24T21:56:42.254648+0000","last_active":"2021-03-24T21:57:12.208452+0000","last_peered":"2021-03-24T21:57:12.208452+0000","last_clean":"2021-03-24T21:57:12.208452+0000","last_became_active":"2021-03-24T21:51:45.381533+0000","last_became_peered":"2021-03-24T21:51:45.381533+0000","last_unstale":"2021-03-24T21:57:12.208452+0000","last_undegraded":"2021-03-24T21:57:12.208452+0000","last_fullsized":"2021-03-24T21:57:12.208452+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:42.254536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:42.254536+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"69'15","reported_seq":"104","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.192030+0000","last_change":"2021-03-24T21:56:35.748629+0000","last_active":"2021-03-24T21:57:12.192030+0000","last_peered":"2021-03-24T21:57:12.192030+0000","last_clean":"2021-03-24T21:57:12.192030+0000","last_became_active":"2021-03-24T21:56:35.142209+0000","last_became_peered":"2021-03-24T21:56:35.142209+0000","last_unstale":"2021-03-24T21:57:12.192030+0000","last_undegraded":"2021-03-24T21:57:12.192030+0000","last_fullsized":"2021-03-24T21:57:12.192030+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:56:06.124139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:06.124139+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"231","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.192975+0000","last_change":"2021-03-24T21:56:33.440700+0000","last_active":"2021-03-24T21:57:12.192975+0000","last_peered":"2021-03-24T21:57:12.192975+0000","last_clean":"2021-03-24T21:57:12.192975+0000","last_became_active":"2021-03-24T21:54:38.143563+0000","last_became_peered":"2021-03-24T21:54:38.143563+0000","last_unstale":"2021-03-24T21:57:12.192975+0000","last_undegraded":"2021-03-24T21:57:12.192975+0000","last_fullsized":"2021-03-24T21:57:12.192975+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:33.440618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:33.440618+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"215","reported_epoch":"262","state":"active+clean","last_fresh":"2021-03-24T21:57:11.147224+0000","last_change":"2021-03-24T21:55:02.129906+0000","last_active":"2021-03-24T21:57:11.147224+0000","last_peered":"2021-03-24T21:57:11.147224+0000","last_clean":"2021-03-24T21:57:11.147224+0000","last_became_active":"2021-03-24T21:51:49.410920+0000","last_became_peered":"2021-03-24T21:51:49.410920+0000","last_unstale":"2021-03-24T21:57:11.147224+0000","last_undegraded":"2021-03-24T21:57:11.147224+0000","last_fullsized":"2021-03-24T21:57:11.147224+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:02.129719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:55:02.129719+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":"228","reported_epoch":"262","state":"active+clean","last_fresh":"2021-03-24T21:57:11.145449+0000","last_change":"2021-03-24T21:54:38.144167+0000","last_active":"2021-03-24T21:57:11.145449+0000","last_peered":"2021-03-24T21:57:11.145449+0000","last_clean":"2021-03-24T21:57:11.145449+0000","last_became_active":"2021-03-24T21:54:38.143876+0000","last_became_peered":"2021-03-24T21:54:38.143876+0000","last_unstale":"2021-03-24T21:57:11.145449+0000","last_undegraded":"2021-03-24T21:57:11.145449+0000","last_fullsized":"2021-03-24T21:57:11.145449+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:11.286003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:52:54.212459+0000","last_clean_scrub_stamp":"2021-03-24T21:54:11.286003+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"69'16","reported_seq":"277","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.192539+0000","last_change":"2021-03-24T21:56:10.423931+0000","last_active":"2021-03-24T21:57:12.192539+0000","last_peered":"2021-03-24T21:57:12.192539+0000","last_clean":"2021-03-24T21:57:12.192539+0000","last_became_active":"2021-03-24T21:54:38.128246+0000","last_became_peered":"2021-03-24T21:54:38.128246+0000","last_unstale":"2021-03-24T21:57:12.192539+0000","last_undegraded":"2021-03-24T21:57:12.192539+0000","last_fullsized":"2021-03-24T21:57:12.192539+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T21:56:10.423860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:10.423860+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":37,"num_read_kb":20,"num_write":18,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":844,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"215","reported_epoch":"264","state":"active+clean","last_fresh":"2021-03-24T21:57:13.207150+0000","last_change":"2021-03-24T21:55:29.900103+0000","last_active":"2021-03-24T21:57:13.207150+0000","last_peered":"2021-03-24T21:57:13.207150+0000","last_clean":"2021-03-24T21:57:13.207150+0000","last_became_active":"2021-03-24T21:51:51.761492+0000","last_became_peered":"2021-03-24T21:51:51.761492+0000","last_unstale":"2021-03-24T21:57:13.207150+0000","last_undegraded":"2021-03-24T21:57:13.207150+0000","last_fullsized":"2021-03-24T21:57:13.207150+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:29.899923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:55:29.899923+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"235'4","reported_seq":"237","reported_epoch":"262","state":"active+clean","last_fresh":"2021-03-24T21:57:11.147152+0000","last_change":"2021-03-24T21:56:44.318340+0000","last_active":"2021-03-24T21:57:11.147152+0000","last_peered":"2021-03-24T21:57:11.147152+0000","last_clean":"2021-03-24T21:57:11.147152+0000","last_became_active":"2021-03-24T21:56:44.228196+0000","last_became_peered":"2021-03-24T21:56:44.228196+0000","last_unstale":"2021-03-24T21:57:11.147152+0000","last_undegraded":"2021-03-24T21:57:11.147152+0000","last_fullsized":"2021-03-24T21:57:11.147152+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'3","last_scrub_stamp":"2021-03-24T21:56:42.039976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:42.039976+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":"188","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.190668+0000","last_change":"2021-03-24T21:57:02.182678+0000","last_active":"2021-03-24T21:57:12.190668+0000","last_peered":"2021-03-24T21:57:12.190668+0000","last_clean":"2021-03-24T21:57:12.190668+0000","last_became_active":"2021-03-24T21:57:02.181985+0000","last_became_peered":"2021-03-24T21:57:02.181985+0000","last_unstale":"2021-03-24T21:57:12.190668+0000","last_undegraded":"2021-03-24T21:57:12.190668+0000","last_fullsized":"2021-03-24T21:57:12.190668+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:48.541415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:54:48.541415+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"89'27","reported_seq":"299","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.214038+0000","last_change":"2021-03-24T21:56:45.178898+0000","last_active":"2021-03-24T21:57:12.214038+0000","last_peered":"2021-03-24T21:57:12.214038+0000","last_clean":"2021-03-24T21:57:12.214038+0000","last_became_active":"2021-03-24T21:56:44.230130+0000","last_became_peered":"2021-03-24T21:56:44.230130+0000","last_unstale":"2021-03-24T21:57:12.214038+0000","last_undegraded":"2021-03-24T21:57:12.214038+0000","last_fullsized":"2021-03-24T21:57:12.214038+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'27","last_scrub_stamp":"2021-03-24T21:55:41.337130+0000","last_deep_scrub":"89'27","last_deep_scrub_stamp":"2021-03-24T21:55:41.337130+0000","last_clean_scrub_stamp":"2021-03-24T21:55:41.337130+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":934,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"177","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.183125+0000","last_change":"2021-03-24T21:56:07.272826+0000","last_active":"2021-03-24T21:57:12.183125+0000","last_peered":"2021-03-24T21:57:12.183125+0000","last_clean":"2021-03-24T21:57:12.183125+0000","last_became_active":"2021-03-24T21:54:38.113849+0000","last_became_peered":"2021-03-24T21:54:38.113849+0000","last_unstale":"2021-03-24T21:57:12.183125+0000","last_undegraded":"2021-03-24T21:57:12.183125+0000","last_fullsized":"2021-03-24T21:57:12.183125+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:07.272644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:07.272644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"224","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.214509+0000","last_change":"2021-03-24T21:56:35.141430+0000","last_active":"2021-03-24T21:57:12.214509+0000","last_peered":"2021-03-24T21:57:12.214509+0000","last_clean":"2021-03-24T21:57:12.214509+0000","last_became_active":"2021-03-24T21:56:35.140997+0000","last_became_peered":"2021-03-24T21:56:35.140997+0000","last_unstale":"2021-03-24T21:57:12.214509+0000","last_undegraded":"2021-03-24T21:57:12.214509+0000","last_fullsized":"2021-03-24T21:57:12.214509+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:07.255496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:07.255496+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":"228","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.190405+0000","last_change":"2021-03-24T21:55:32.566923+0000","last_active":"2021-03-24T21:57:12.190405+0000","last_peered":"2021-03-24T21:57:12.190405+0000","last_clean":"2021-03-24T21:57:12.190405+0000","last_became_active":"2021-03-24T21:54:38.145870+0000","last_became_peered":"2021-03-24T21:54:38.145870+0000","last_unstale":"2021-03-24T21:57:12.190405+0000","last_undegraded":"2021-03-24T21:57:12.190405+0000","last_fullsized":"2021-03-24T21:57:12.190405+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:32.566747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:32.566747+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"69'17","reported_seq":"252","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.214086+0000","last_change":"2021-03-24T21:56:45.177087+0000","last_active":"2021-03-24T21:57:12.214086+0000","last_peered":"2021-03-24T21:57:12.214086+0000","last_clean":"2021-03-24T21:57:12.214086+0000","last_became_active":"2021-03-24T21:56:44.229823+0000","last_became_peered":"2021-03-24T21:56:44.229823+0000","last_unstale":"2021-03-24T21:57:12.214086+0000","last_undegraded":"2021-03-24T21:57:12.214086+0000","last_fullsized":"2021-03-24T21:57:12.214086+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'17","last_scrub_stamp":"2021-03-24T21:55:32.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:32.214502+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":37,"num_read_kb":18,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":2100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.13","version":"263'230","reported_seq":"803","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.242510+0000","last_change":"2021-03-24T21:57:12.212997+0000","last_active":"2021-03-24T21:57:12.242510+0000","last_peered":"2021-03-24T21:57:12.242510+0000","last_clean":"2021-03-24T21:57:12.242510+0000","last_became_active":"2021-03-24T21:57:06.153030+0000","last_became_peered":"2021-03-24T21:57:06.153030+0000","last_unstale":"2021-03-24T21:57:12.242510+0000","last_undegraded":"2021-03-24T21:57:12.242510+0000","last_fullsized":"2021-03-24T21:57:12.242510+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":230,"ondisk_log_size":230,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11949777,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":74,"num_read_kb":61662,"num_write":70,"num_write_kb":30484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":4773188,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"82","length":"1"},{"start":"8c","length":"2"},{"start":"92","length":"1"},{"start":"95","length":"1"},{"start":"9c","length":"1"}]},{"pgid":"6.12","version":"0'0","reported_seq":"215","reported_epoch":"264","state":"active+clean","last_fresh":"2021-03-24T21:57:13.202424+0000","last_change":"2021-03-24T21:56:04.042857+0000","last_active":"2021-03-24T21:57:13.202424+0000","last_peered":"2021-03-24T21:57:13.202424+0000","last_clean":"2021-03-24T21:57:13.202424+0000","last_became_active":"2021-03-24T21:51:51.762150+0000","last_became_peered":"2021-03-24T21:51:51.762150+0000","last_unstale":"2021-03-24T21:57:13.202424+0000","last_undegraded":"2021-03-24T21:57:13.202424+0000","last_fullsized":"2021-03-24T21:57:13.202424+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:04.042709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:04.042709+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"177","reported_epoch":"262","state":"active+clean","last_fresh":"2021-03-24T21:57:11.144524+0000","last_change":"2021-03-24T21:55:26.087761+0000","last_active":"2021-03-24T21:57:11.144524+0000","last_peered":"2021-03-24T21:57:11.144524+0000","last_clean":"2021-03-24T21:57:11.144524+0000","last_became_active":"2021-03-24T21:54:24.187568+0000","last_became_peered":"2021-03-24T21:54:24.187568+0000","last_unstale":"2021-03-24T21:57:11.144524+0000","last_undegraded":"2021-03-24T21:57:11.144524+0000","last_fullsized":"2021-03-24T21:57:11.144524+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:26.087584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:55:26.087584+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":"224","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.214764+0000","last_change":"2021-03-24T21:57:11.230052+0000","last_active":"2021-03-24T21:57:12.214764+0000","last_peered":"2021-03-24T21:57:12.214764+0000","last_clean":"2021-03-24T21:57:12.214764+0000","last_became_active":"2021-03-24T21:51:45.379789+0000","last_became_peered":"2021-03-24T21:51:45.379789+0000","last_unstale":"2021-03-24T21:57:12.214764+0000","last_undegraded":"2021-03-24T21:57:12.214764+0000","last_fullsized":"2021-03-24T21:57:12.214764+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:11.229895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:11.229895+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"69'12","reported_seq":"339","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.190474+0000","last_change":"2021-03-24T21:56:53.443666+0000","last_active":"2021-03-24T21:57:12.190474+0000","last_peered":"2021-03-24T21:57:12.190474+0000","last_clean":"2021-03-24T21:57:12.190474+0000","last_became_active":"2021-03-24T21:54:38.134132+0000","last_became_peered":"2021-03-24T21:54:38.134132+0000","last_unstale":"2021-03-24T21:57:12.190474+0000","last_undegraded":"2021-03-24T21:57:12.190474+0000","last_fullsized":"2021-03-24T21:57:12.190474+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2021-03-24T21:56:53.443554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:53.443554+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":12,"num_write":12,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":3756,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.12","version":"183'26","reported_seq":"203","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.204009+0000","last_change":"2021-03-24T21:57:12.204009+0000","last_active":"2021-03-24T21:57:12.204009+0000","last_peered":"2021-03-24T21:57:12.204009+0000","last_clean":"2021-03-24T21:57:12.204009+0000","last_became_active":"2021-03-24T21:56:10.130886+0000","last_became_peered":"2021-03-24T21:56:10.130886+0000","last_unstale":"2021-03-24T21:57:12.204009+0000","last_undegraded":"2021-03-24T21:57:12.204009+0000","last_fullsized":"2021-03-24T21:57:12.204009+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1352964,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":6,"num_write_kb":2389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":433862,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"82","length":"1"},{"start":"8c","length":"2"},{"start":"92","length":"1"},{"start":"95","length":"1"},{"start":"9c","length":"1"}]},{"pgid":"6.13","version":"0'0","reported_seq":"222","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.192013+0000","last_change":"2021-03-24T21:56:40.458139+0000","last_active":"2021-03-24T21:57:12.192013+0000","last_peered":"2021-03-24T21:57:12.192013+0000","last_clean":"2021-03-24T21:57:12.192013+0000","last_became_active":"2021-03-24T21:54:24.202686+0000","last_became_peered":"2021-03-24T21:54:24.202686+0000","last_unstale":"2021-03-24T21:57:12.192013+0000","last_undegraded":"2021-03-24T21:57:12.192013+0000","last_fullsized":"2021-03-24T21:57:12.192013+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:40.458011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:47.540793+0000","last_clean_scrub_stamp":"2021-03-24T21:56:40.458011+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"226","reported_epoch":"264","state":"active+clean","last_fresh":"2021-03-24T21:57:13.202002+0000","last_change":"2021-03-24T21:55:23.872913+0000","last_active":"2021-03-24T21:57:13.202002+0000","last_peered":"2021-03-24T21:57:13.202002+0000","last_clean":"2021-03-24T21:57:13.202002+0000","last_became_active":"2021-03-24T21:54:24.194423+0000","last_became_peered":"2021-03-24T21:54:24.194423+0000","last_unstale":"2021-03-24T21:57:13.202002+0000","last_undegraded":"2021-03-24T21:57:13.202002+0000","last_fullsized":"2021-03-24T21:57:13.202002+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:23.872742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:55:23.872742+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":"236","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:13.016283+0000","last_change":"2021-03-24T21:57:13.016235+0000","last_active":"2021-03-24T21:57:13.016283+0000","last_peered":"2021-03-24T21:57:13.016283+0000","last_clean":"2021-03-24T21:57:13.016283+0000","last_became_active":"2021-03-24T21:54:38.121323+0000","last_became_peered":"2021-03-24T21:54:38.121323+0000","last_unstale":"2021-03-24T21:57:13.016283+0000","last_undegraded":"2021-03-24T21:57:13.016283+0000","last_fullsized":"2021-03-24T21:57:13.016283+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:13.016117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:13.016117+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"86'22","reported_seq":"277","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.190923+0000","last_change":"2021-03-24T21:54:51.585204+0000","last_active":"2021-03-24T21:57:12.190923+0000","last_peered":"2021-03-24T21:57:12.190923+0000","last_clean":"2021-03-24T21:57:12.190923+0000","last_became_active":"2021-03-24T21:54:24.188790+0000","last_became_peered":"2021-03-24T21:54:24.188790+0000","last_unstale":"2021-03-24T21:57:12.190923+0000","last_undegraded":"2021-03-24T21:57:12.190923+0000","last_fullsized":"2021-03-24T21:57:12.190923+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'22","last_scrub_stamp":"2021-03-24T21:54:51.585058+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2021-03-24T21:52:54.597505+0000","last_clean_scrub_stamp":"2021-03-24T21:54:51.585058+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":16,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":580,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"265'274","reported_seq":"861","reported_epoch":"265","state":"active+clean","last_fresh":"2021-03-24T21:57:14.228222+0000","last_change":"2021-03-24T21:57:14.228502+0000","last_active":"2021-03-24T21:57:14.228222+0000","last_peered":"2021-03-24T21:57:14.228222+0000","last_clean":"2021-03-24T21:57:14.228222+0000","last_became_active":"2021-03-24T21:56:14.138730+0000","last_became_peered":"2021-03-24T21:56:14.138730+0000","last_unstale":"2021-03-24T21:57:14.228222+0000","last_undegraded":"2021-03-24T21:57:14.228222+0000","last_fullsized":"2021-03-24T21:57:14.228222+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":274,"ondisk_log_size":274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4260641,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":136,"num_read_kb":136187,"num_write":87,"num_write_kb":38027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":11443110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"82","length":"1"},{"start":"8b","length":"3"},{"start":"92","length":"1"},{"start":"95","length":"1"},{"start":"9c","length":"1"}]},{"pgid":"6.10","version":"0'0","reported_seq":"227","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.202911+0000","last_change":"2021-03-24T21:56:52.226305+0000","last_active":"2021-03-24T21:57:12.202911+0000","last_peered":"2021-03-24T21:57:12.202911+0000","last_clean":"2021-03-24T21:57:12.202911+0000","last_became_active":"2021-03-24T21:54:38.126662+0000","last_became_peered":"2021-03-24T21:54:38.126662+0000","last_unstale":"2021-03-24T21:57:12.202911+0000","last_undegraded":"2021-03-24T21:57:12.202911+0000","last_fullsized":"2021-03-24T21:57:12.202911+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:52.226117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:52.226117+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"224","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.191588+0000","last_change":"2021-03-24T21:56:32.480276+0000","last_active":"2021-03-24T21:57:12.191588+0000","last_peered":"2021-03-24T21:57:12.191588+0000","last_clean":"2021-03-24T21:57:12.191588+0000","last_became_active":"2021-03-24T21:54:38.128067+0000","last_became_peered":"2021-03-24T21:54:38.128067+0000","last_unstale":"2021-03-24T21:57:12.191588+0000","last_undegraded":"2021-03-24T21:57:12.191588+0000","last_fullsized":"2021-03-24T21:57:12.191588+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:32.480115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:32.480115+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":"221","reported_epoch":"264","state":"active+clean","last_fresh":"2021-03-24T21:57:13.205628+0000","last_change":"2021-03-24T21:55:57.107894+0000","last_active":"2021-03-24T21:57:13.205628+0000","last_peered":"2021-03-24T21:57:13.205628+0000","last_clean":"2021-03-24T21:57:13.205628+0000","last_became_active":"2021-03-24T21:51:45.379383+0000","last_became_peered":"2021-03-24T21:51:45.379383+0000","last_unstale":"2021-03-24T21:57:13.205628+0000","last_undegraded":"2021-03-24T21:57:13.205628+0000","last_fullsized":"2021-03-24T21:57:13.205628+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:57.107720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:57.107720+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"69'15","reported_seq":"231","reported_epoch":"261","state":"active+clean","last_fresh":"2021-03-24T21:57:10.140874+0000","last_change":"2021-03-24T21:56:09.119663+0000","last_active":"2021-03-24T21:57:10.140874+0000","last_peered":"2021-03-24T21:57:10.140874+0000","last_clean":"2021-03-24T21:57:10.140874+0000","last_became_active":"2021-03-24T21:54:38.122707+0000","last_became_peered":"2021-03-24T21:54:38.122707+0000","last_unstale":"2021-03-24T21:57:10.140874+0000","last_undegraded":"2021-03-24T21:57:10.140874+0000","last_fullsized":"2021-03-24T21:57:10.140874+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:56:09.119562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:09.119562+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":34,"num_read_kb":16,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.10","version":"199'23","reported_seq":"49","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.179457+0000","last_change":"2021-03-24T21:57:12.179457+0000","last_active":"2021-03-24T21:57:12.179457+0000","last_peered":"2021-03-24T21:57:12.179457+0000","last_clean":"2021-03-24T21:57:12.179457+0000","last_became_active":"2021-03-24T21:57:06.154293+0000","last_became_peered":"2021-03-24T21:57:06.154293+0000","last_unstale":"2021-03-24T21:57:12.179457+0000","last_undegraded":"2021-03-24T21:57:12.179457+0000","last_fullsized":"2021-03-24T21:57:12.179457+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":5,"last_scrub":"175'22","last_scrub_stamp":"2021-03-24T21:55:59.276308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:55:59.276308+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2521050,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":28733,"num_write":8,"num_write_kb":4153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":5668873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"82","length":"1"},{"start":"8c","length":"2"},{"start":"92","length":"1"},{"start":"95","length":"1"},{"start":"9c","length":"1"}]},{"pgid":"6.11","version":"0'0","reported_seq":"213","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.190660+0000","last_change":"2021-03-24T21:55:22.555707+0000","last_active":"2021-03-24T21:57:12.190660+0000","last_peered":"2021-03-24T21:57:12.190660+0000","last_clean":"2021-03-24T21:57:12.190660+0000","last_became_active":"2021-03-24T21:51:51.764871+0000","last_became_peered":"2021-03-24T21:51:51.764871+0000","last_unstale":"2021-03-24T21:57:12.190660+0000","last_undegraded":"2021-03-24T21:57:12.190660+0000","last_fullsized":"2021-03-24T21:57:12.190660+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:22.555624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:23.707914+0000","last_clean_scrub_stamp":"2021-03-24T21:55:22.555624+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"181","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.179523+0000","last_change":"2021-03-24T21:56:35.139626+0000","last_active":"2021-03-24T21:57:12.179523+0000","last_peered":"2021-03-24T21:57:12.179523+0000","last_clean":"2021-03-24T21:57:12.179523+0000","last_became_active":"2021-03-24T21:56:35.139207+0000","last_became_peered":"2021-03-24T21:56:35.139207+0000","last_unstale":"2021-03-24T21:57:12.179523+0000","last_undegraded":"2021-03-24T21:57:12.179523+0000","last_fullsized":"2021-03-24T21:57:12.179523+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:40.456101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:55:40.456101+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":"228","reported_epoch":"262","state":"active+clean","last_fresh":"2021-03-24T21:57:11.146959+0000","last_change":"2021-03-24T21:55:42.135567+0000","last_active":"2021-03-24T21:57:11.146959+0000","last_peered":"2021-03-24T21:57:11.146959+0000","last_clean":"2021-03-24T21:57:11.146959+0000","last_became_active":"2021-03-24T21:54:24.181343+0000","last_became_peered":"2021-03-24T21:54:24.181343+0000","last_unstale":"2021-03-24T21:57:11.146959+0000","last_undegraded":"2021-03-24T21:57:11.146959+0000","last_fullsized":"2021-03-24T21:57:11.146959+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:42.135502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:42.135502+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"69'15","reported_seq":"302","reported_epoch":"262","state":"active+clean","last_fresh":"2021-03-24T21:57:11.146912+0000","last_change":"2021-03-24T21:55:37.076650+0000","last_active":"2021-03-24T21:57:11.146912+0000","last_peered":"2021-03-24T21:57:11.146912+0000","last_clean":"2021-03-24T21:57:11.146912+0000","last_became_active":"2021-03-24T21:54:38.118330+0000","last_became_peered":"2021-03-24T21:54:38.118330+0000","last_unstale":"2021-03-24T21:57:11.146912+0000","last_undegraded":"2021-03-24T21:57:11.146912+0000","last_fullsized":"2021-03-24T21:57:11.146912+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:55:37.076511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:37.076511+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"265'274","reported_seq":"920","reported_epoch":"265","state":"active+clean","last_fresh":"2021-03-24T21:57:14.281851+0000","last_change":"2021-03-24T21:57:14.281851+0000","last_active":"2021-03-24T21:57:14.281851+0000","last_peered":"2021-03-24T21:57:14.281851+0000","last_clean":"2021-03-24T21:57:14.281851+0000","last_became_active":"2021-03-24T21:56:13.132418+0000","last_became_peered":"2021-03-24T21:56:13.132418+0000","last_unstale":"2021-03-24T21:57:14.281851+0000","last_undegraded":"2021-03-24T21:57:14.281851+0000","last_fullsized":"2021-03-24T21:57:14.281851+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":5,"last_scrub":"265'274","last_scrub_stamp":"2021-03-24T21:57:14.281739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:57:14.281739+0000","log_size":274,"ondisk_log_size":274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10017038,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"82","length":"1"},{"start":"8b","length":"3"},{"start":"92","length":"1"},{"start":"95","length":"1"},{"start":"9c","length":"1"}]},{"pgid":"6.16","version":"0'0","reported_seq":"213","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.213749+0000","last_change":"2021-03-24T21:54:44.392954+0000","last_active":"2021-03-24T21:57:12.213749+0000","last_peered":"2021-03-24T21:57:12.213749+0000","last_clean":"2021-03-24T21:57:12.213749+0000","last_became_active":"2021-03-24T21:51:51.756584+0000","last_became_peered":"2021-03-24T21:51:51.756584+0000","last_unstale":"2021-03-24T21:57:12.213749+0000","last_undegraded":"2021-03-24T21:57:12.213749+0000","last_fullsized":"2021-03-24T21:57:12.213749+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:44.392808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:54:44.392808+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"109'4","reported_seq":"303","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.183458+0000","last_change":"2021-03-24T21:56:42.209663+0000","last_active":"2021-03-24T21:57:12.183458+0000","last_peered":"2021-03-24T21:57:12.183458+0000","last_clean":"2021-03-24T21:57:12.183458+0000","last_became_active":"2021-03-24T21:54:38.130026+0000","last_became_peered":"2021-03-24T21:54:38.130026+0000","last_unstale":"2021-03-24T21:57:12.183458+0000","last_undegraded":"2021-03-24T21:57:12.183458+0000","last_fullsized":"2021-03-24T21:57:12.183458+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'4","last_scrub_stamp":"2021-03-24T21:56:42.209497+0000","last_deep_scrub":"86'3","last_deep_scrub_stamp":"2021-03-24T21:53:08.815163+0000","last_clean_scrub_stamp":"2021-03-24T21:56:42.209497+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":"221","reported_epoch":"264","state":"active+clean","last_fresh":"2021-03-24T21:57:13.202702+0000","last_change":"2021-03-24T21:55:58.082943+0000","last_active":"2021-03-24T21:57:13.202702+0000","last_peered":"2021-03-24T21:57:13.202702+0000","last_clean":"2021-03-24T21:57:13.202702+0000","last_became_active":"2021-03-24T21:51:45.579271+0000","last_became_peered":"2021-03-24T21:51:45.579271+0000","last_unstale":"2021-03-24T21:57:13.202702+0000","last_undegraded":"2021-03-24T21:57:13.202702+0000","last_fullsized":"2021-03-24T21:57:13.202702+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:58.082776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:58.082776+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"69'14","reported_seq":"272","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.192265+0000","last_change":"2021-03-24T21:55:25.582566+0000","last_active":"2021-03-24T21:57:12.192265+0000","last_peered":"2021-03-24T21:57:12.192265+0000","last_clean":"2021-03-24T21:57:12.192265+0000","last_became_active":"2021-03-24T21:54:38.143783+0000","last_became_peered":"2021-03-24T21:54:38.143783+0000","last_unstale":"2021-03-24T21:57:12.192265+0000","last_undegraded":"2021-03-24T21:57:12.192265+0000","last_fullsized":"2021-03-24T21:57:12.192265+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'14","last_scrub_stamp":"2021-03-24T21:55:25.582388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:25.582388+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":27,"num_read_kb":14,"num_write":14,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.16","version":"256'136","reported_seq":"675","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.213776+0000","last_change":"2021-03-24T21:57:12.214370+0000","last_active":"2021-03-24T21:57:12.213776+0000","last_peered":"2021-03-24T21:57:12.213776+0000","last_clean":"2021-03-24T21:57:12.213776+0000","last_became_active":"2021-03-24T21:56:13.127048+0000","last_became_peered":"2021-03-24T21:56:13.127048+0000","last_unstale":"2021-03-24T21:57:12.213776+0000","last_undegraded":"2021-03-24T21:57:12.213776+0000","last_fullsized":"2021-03-24T21:57:12.213776+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":136,"ondisk_log_size":136,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":756365,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":58,"num_read_kb":66271,"num_write":36,"num_write_kb":10335,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":3697507,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"82","length":"1"},{"start":"8c","length":"2"},{"start":"92","length":"1"},{"start":"95","length":"1"},{"start":"9c","length":"1"}]},{"pgid":"6.17","version":"0'0","reported_seq":"220","reported_epoch":"262","state":"active+clean","last_fresh":"2021-03-24T21:57:11.147260+0000","last_change":"2021-03-24T21:56:51.925140+0000","last_active":"2021-03-24T21:57:11.147260+0000","last_peered":"2021-03-24T21:57:11.147260+0000","last_clean":"2021-03-24T21:57:11.147260+0000","last_became_active":"2021-03-24T21:51:51.762876+0000","last_became_peered":"2021-03-24T21:51:51.762876+0000","last_unstale":"2021-03-24T21:57:11.147260+0000","last_undegraded":"2021-03-24T21:57:11.147260+0000","last_fullsized":"2021-03-24T21:57:11.147260+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:51.924963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:51.924963+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"235'4","reported_seq":"296","reported_epoch":"264","state":"active+clean","last_fresh":"2021-03-24T21:57:13.203172+0000","last_change":"2021-03-24T21:56:44.227488+0000","last_active":"2021-03-24T21:57:13.203172+0000","last_peered":"2021-03-24T21:57:13.203172+0000","last_clean":"2021-03-24T21:57:13.203172+0000","last_became_active":"2021-03-24T21:56:44.224410+0000","last_became_peered":"2021-03-24T21:56:44.224410+0000","last_unstale":"2021-03-24T21:57:13.203172+0000","last_undegraded":"2021-03-24T21:57:13.203172+0000","last_fullsized":"2021-03-24T21:57:13.203172+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'3","last_scrub_stamp":"2021-03-24T21:55:01.577769+0000","last_deep_scrub":"86'3","last_deep_scrub_stamp":"2021-03-24T21:52:56.678111+0000","last_clean_scrub_stamp":"2021-03-24T21:55:01.577769+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":"224","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.203451+0000","last_change":"2021-03-24T21:56:32.279216+0000","last_active":"2021-03-24T21:57:12.203451+0000","last_peered":"2021-03-24T21:57:12.203451+0000","last_clean":"2021-03-24T21:57:12.203451+0000","last_became_active":"2021-03-24T21:51:45.381516+0000","last_became_peered":"2021-03-24T21:51:45.381516+0000","last_unstale":"2021-03-24T21:57:12.203451+0000","last_undegraded":"2021-03-24T21:57:12.203451+0000","last_fullsized":"2021-03-24T21:57:12.203451+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:32.279119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:05.473348+0000","last_clean_scrub_stamp":"2021-03-24T21:56:32.279119+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"69'13","reported_seq":"249","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.184228+0000","last_change":"2021-03-24T21:56:47.168651+0000","last_active":"2021-03-24T21:57:12.184228+0000","last_peered":"2021-03-24T21:57:12.184228+0000","last_clean":"2021-03-24T21:57:12.184228+0000","last_became_active":"2021-03-24T21:51:47.689288+0000","last_became_peered":"2021-03-24T21:51:47.689288+0000","last_unstale":"2021-03-24T21:57:12.184228+0000","last_undegraded":"2021-03-24T21:57:12.184228+0000","last_fullsized":"2021-03-24T21:57:12.184228+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2021-03-24T21:56:47.168558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:47.168558+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.15","version":"248'81","reported_seq":"642","reported_epoch":"265","state":"active+clean","last_fresh":"2021-03-24T21:57:14.221504+0000","last_change":"2021-03-24T21:57:14.221923+0000","last_active":"2021-03-24T21:57:14.221504+0000","last_peered":"2021-03-24T21:57:14.221504+0000","last_clean":"2021-03-24T21:57:14.221504+0000","last_became_active":"2021-03-24T21:56:10.139551+0000","last_became_peered":"2021-03-24T21:56:10.139551+0000","last_unstale":"2021-03-24T21:57:14.221504+0000","last_undegraded":"2021-03-24T21:57:14.221504+0000","last_fullsized":"2021-03-24T21:57:14.221504+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":5,"last_scrub":"233'79","last_scrub_stamp":"2021-03-24T21:56:50.934781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:56:50.934781+0000","log_size":81,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2403434,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"82","length":"1"},{"start":"8b","length":"3"},{"start":"92","length":"1"},{"start":"95","length":"1"},{"start":"9c","length":"1"}]},{"pgid":"6.14","version":"0'0","reported_seq":"212","reported_epoch":"261","state":"active+clean","last_fresh":"2021-03-24T21:57:10.144120+0000","last_change":"2021-03-24T21:55:56.125093+0000","last_active":"2021-03-24T21:57:10.144120+0000","last_peered":"2021-03-24T21:57:10.144120+0000","last_clean":"2021-03-24T21:57:10.144120+0000","last_became_active":"2021-03-24T21:51:51.754085+0000","last_became_peered":"2021-03-24T21:51:51.754085+0000","last_unstale":"2021-03-24T21:57:10.144120+0000","last_undegraded":"2021-03-24T21:57:10.144120+0000","last_fullsized":"2021-03-24T21:57:10.144120+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:56.125019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:55:56.125019+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"224","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.194797+0000","last_change":"2021-03-24T21:54:55.493891+0000","last_active":"2021-03-24T21:57:12.194797+0000","last_peered":"2021-03-24T21:57:12.194797+0000","last_clean":"2021-03-24T21:57:12.194797+0000","last_became_active":"2021-03-24T21:54:38.129390+0000","last_became_peered":"2021-03-24T21:54:38.129390+0000","last_unstale":"2021-03-24T21:57:12.194797+0000","last_undegraded":"2021-03-24T21:57:12.194797+0000","last_fullsized":"2021-03-24T21:57:12.194797+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:55.493713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:55.493713+0000","last_clean_scrub_stamp":"2021-03-24T21:54:55.493713+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":"230","reported_epoch":"264","state":"active+clean","last_fresh":"2021-03-24T21:57:13.206691+0000","last_change":"2021-03-24T21:55:31.912637+0000","last_active":"2021-03-24T21:57:13.206691+0000","last_peered":"2021-03-24T21:57:13.206691+0000","last_clean":"2021-03-24T21:57:13.206691+0000","last_became_active":"2021-03-24T21:54:24.196290+0000","last_became_peered":"2021-03-24T21:54:24.196290+0000","last_unstale":"2021-03-24T21:57:13.206691+0000","last_undegraded":"2021-03-24T21:57:13.206691+0000","last_fullsized":"2021-03-24T21:57:13.206691+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:31.912451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:31.912451+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"69'16","reported_seq":"252","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.214396+0000","last_change":"2021-03-24T21:56:40.303286+0000","last_active":"2021-03-24T21:57:12.214396+0000","last_peered":"2021-03-24T21:57:12.214396+0000","last_clean":"2021-03-24T21:57:12.214396+0000","last_became_active":"2021-03-24T21:51:47.687595+0000","last_became_peered":"2021-03-24T21:51:47.687595+0000","last_unstale":"2021-03-24T21:57:12.214396+0000","last_undegraded":"2021-03-24T21:57:12.214396+0000","last_fullsized":"2021-03-24T21:57:12.214396+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T21:56:40.303200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:40.303200+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":38,"num_read_kb":18,"num_write":17,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"264'174","reported_seq":"733","reported_epoch":"264","state":"active+clean","last_fresh":"2021-03-24T21:57:13.229560+0000","last_change":"2021-03-24T21:57:12.194893+0000","last_active":"2021-03-24T21:57:13.229560+0000","last_peered":"2021-03-24T21:57:13.229560+0000","last_clean":"2021-03-24T21:57:13.229560+0000","last_became_active":"2021-03-24T21:56:14.139804+0000","last_became_peered":"2021-03-24T21:56:14.139804+0000","last_unstale":"2021-03-24T21:57:13.229560+0000","last_undegraded":"2021-03-24T21:57:13.229560+0000","last_fullsized":"2021-03-24T21:57:13.229560+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":174,"ondisk_log_size":174,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5733006,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":63,"num_read_kb":66061,"num_write":46,"num_write_kb":16333,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":6105417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"82","length":"1"},{"start":"8c","length":"2"},{"start":"92","length":"1"},{"start":"95","length":"1"},{"start":"9c","length":"1"}]},{"pgid":"6.15","version":"0'0","reported_seq":"224","reported_epoch":"264","state":"active+clean","last_fresh":"2021-03-24T21:57:13.205547+0000","last_change":"2021-03-24T21:56:00.986717+0000","last_active":"2021-03-24T21:57:13.205547+0000","last_peered":"2021-03-24T21:57:13.205547+0000","last_clean":"2021-03-24T21:57:13.205547+0000","last_became_active":"2021-03-24T21:54:24.188386+0000","last_became_peered":"2021-03-24T21:54:24.188386+0000","last_unstale":"2021-03-24T21:57:13.205547+0000","last_undegraded":"2021-03-24T21:57:13.205547+0000","last_fullsized":"2021-03-24T21:57:13.205547+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:00.986593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:00.986593+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"232","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.184445+0000","last_change":"2021-03-24T21:56:48.144572+0000","last_active":"2021-03-24T21:57:12.184445+0000","last_peered":"2021-03-24T21:57:12.184445+0000","last_clean":"2021-03-24T21:57:12.184445+0000","last_became_active":"2021-03-24T21:54:38.128409+0000","last_became_peered":"2021-03-24T21:54:38.128409+0000","last_unstale":"2021-03-24T21:57:12.184445+0000","last_undegraded":"2021-03-24T21:57:12.184445+0000","last_fullsized":"2021-03-24T21:57:12.184445+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:48.144391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:48.144391+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":"178","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.203250+0000","last_change":"2021-03-24T21:54:52.385672+0000","last_active":"2021-03-24T21:57:12.203250+0000","last_peered":"2021-03-24T21:57:12.203250+0000","last_clean":"2021-03-24T21:57:12.203250+0000","last_became_active":"2021-03-24T21:54:24.201388+0000","last_became_peered":"2021-03-24T21:54:24.201388+0000","last_unstale":"2021-03-24T21:57:12.203250+0000","last_undegraded":"2021-03-24T21:57:12.203250+0000","last_fullsized":"2021-03-24T21:57:12.203250+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:52.385567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:54:52.385567+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"69'7","reported_seq":"247","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.194667+0000","last_change":"2021-03-24T21:55:56.490518+0000","last_active":"2021-03-24T21:57:12.194667+0000","last_peered":"2021-03-24T21:57:12.194667+0000","last_clean":"2021-03-24T21:57:12.194667+0000","last_became_active":"2021-03-24T21:54:38.144098+0000","last_became_peered":"2021-03-24T21:54:38.144098+0000","last_unstale":"2021-03-24T21:57:12.194667+0000","last_undegraded":"2021-03-24T21:57:12.194667+0000","last_fullsized":"2021-03-24T21:57:12.194667+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'7","last_scrub_stamp":"2021-03-24T21:55:56.490439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:56.490439+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":7,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":158,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"264'241","reported_seq":"902","reported_epoch":"264","state":"active+clean","last_fresh":"2021-03-24T21:57:13.211101+0000","last_change":"2021-03-24T21:57:12.181671+0000","last_active":"2021-03-24T21:57:13.211101+0000","last_peered":"2021-03-24T21:57:13.211101+0000","last_clean":"2021-03-24T21:57:13.211101+0000","last_became_active":"2021-03-24T21:57:05.155400+0000","last_became_peered":"2021-03-24T21:57:05.155400+0000","last_unstale":"2021-03-24T21:57:13.211101+0000","last_undegraded":"2021-03-24T21:57:13.211101+0000","last_fullsized":"2021-03-24T21:57:13.211101+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"191'150","last_scrub_stamp":"2021-03-24T21:56:01.308050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:56:01.308050+0000","log_size":241,"ondisk_log_size":241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18893057,"num_objects":8,"num_object_clones":3,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":201,"num_read_kb":224684,"num_write":102,"num_write_kb":41974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":25174911,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"82","length":"1"},{"start":"8c","length":"2"},{"start":"92","length":"1"},{"start":"95","length":"1"},{"start":"9c","length":"1"}]},{"pgid":"6.a","version":"0'0","reported_seq":"197","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.203628+0000","last_change":"2021-03-24T21:57:02.187469+0000","last_active":"2021-03-24T21:57:12.203628+0000","last_peered":"2021-03-24T21:57:12.203628+0000","last_clean":"2021-03-24T21:57:12.203628+0000","last_became_active":"2021-03-24T21:57:02.187060+0000","last_became_peered":"2021-03-24T21:57:02.187060+0000","last_unstale":"2021-03-24T21:57:12.203628+0000","last_undegraded":"2021-03-24T21:57:12.203628+0000","last_fullsized":"2021-03-24T21:57:12.203628+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:54.262681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:54.262681+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"89'4","reported_seq":"226","reported_epoch":"264","state":"active+clean","last_fresh":"2021-03-24T21:57:13.205524+0000","last_change":"2021-03-24T21:56:03.060260+0000","last_active":"2021-03-24T21:57:13.205524+0000","last_peered":"2021-03-24T21:57:13.205524+0000","last_clean":"2021-03-24T21:57:13.205524+0000","last_became_active":"2021-03-24T21:51:45.385110+0000","last_became_peered":"2021-03-24T21:51:45.385110+0000","last_unstale":"2021-03-24T21:57:13.205524+0000","last_undegraded":"2021-03-24T21:57:13.205524+0000","last_fullsized":"2021-03-24T21:57:13.205524+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'4","last_scrub_stamp":"2021-03-24T21:56:03.060069+0000","last_deep_scrub":"89'4","last_deep_scrub_stamp":"2021-03-24T21:53:19.926637+0000","last_clean_scrub_stamp":"2021-03-24T21:56:03.060069+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"97'28","reported_seq":"297","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.172316+0000","last_change":"2021-03-24T21:56:05.210757+0000","last_active":"2021-03-24T21:57:12.172316+0000","last_peered":"2021-03-24T21:57:12.172316+0000","last_clean":"2021-03-24T21:57:12.172316+0000","last_became_active":"2021-03-24T21:54:24.190817+0000","last_became_peered":"2021-03-24T21:54:24.190817+0000","last_unstale":"2021-03-24T21:57:12.172316+0000","last_undegraded":"2021-03-24T21:57:12.172316+0000","last_fullsized":"2021-03-24T21:57:12.172316+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'28","last_scrub_stamp":"2021-03-24T21:56:05.210553+0000","last_deep_scrub":"97'28","last_deep_scrub_stamp":"2021-03-24T21:54:19.157265+0000","last_clean_scrub_stamp":"2021-03-24T21:56:05.210553+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":20,"num_write":19,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"109'4","reported_seq":"315","reported_epoch":"264","state":"active+clean","last_fresh":"2021-03-24T21:57:13.971632+0000","last_change":"2021-03-24T21:56:32.051794+0000","last_active":"2021-03-24T21:57:13.971632+0000","last_peered":"2021-03-24T21:57:13.971632+0000","last_clean":"2021-03-24T21:57:13.971632+0000","last_became_active":"2021-03-24T21:54:38.125497+0000","last_became_peered":"2021-03-24T21:54:38.125497+0000","last_unstale":"2021-03-24T21:57:13.971632+0000","last_undegraded":"2021-03-24T21:57:13.971632+0000","last_fullsized":"2021-03-24T21:57:13.971632+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'4","last_scrub_stamp":"2021-03-24T21:56:32.051533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:32.051533+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"260'116","reported_seq":"619","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.191458+0000","last_change":"2021-03-24T21:57:12.191458+0000","last_active":"2021-03-24T21:57:12.191458+0000","last_peered":"2021-03-24T21:57:12.191458+0000","last_clean":"2021-03-24T21:57:12.191458+0000","last_became_active":"2021-03-24T21:54:22.167632+0000","last_became_peered":"2021-03-24T21:54:22.167632+0000","last_unstale":"2021-03-24T21:57:12.191458+0000","last_undegraded":"2021-03-24T21:57:12.191458+0000","last_fullsized":"2021-03-24T21:57:12.191458+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":116,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9206505,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":38,"num_read_kb":30602,"num_write":56,"num_write_kb":17950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"82","length":"1"},{"start":"8c","length":"2"},{"start":"92","length":"1"},{"start":"95","length":"1"},{"start":"9c","length":"1"}]},{"pgid":"6.b","version":"0'0","reported_seq":"231","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.191809+0000","last_change":"2021-03-24T21:56:44.228591+0000","last_active":"2021-03-24T21:57:12.191809+0000","last_peered":"2021-03-24T21:57:12.191809+0000","last_clean":"2021-03-24T21:57:12.191809+0000","last_became_active":"2021-03-24T21:56:44.228142+0000","last_became_peered":"2021-03-24T21:56:44.228142+0000","last_unstale":"2021-03-24T21:57:12.191809+0000","last_undegraded":"2021-03-24T21:57:12.191809+0000","last_fullsized":"2021-03-24T21:57:12.191809+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:24.548819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:22.724840+0000","last_clean_scrub_stamp":"2021-03-24T21:55:24.548819+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":"235","reported_epoch":"264","state":"active+clean","last_fresh":"2021-03-24T21:57:13.207247+0000","last_change":"2021-03-24T21:56:06.043071+0000","last_active":"2021-03-24T21:57:13.207247+0000","last_peered":"2021-03-24T21:57:13.207247+0000","last_clean":"2021-03-24T21:57:13.207247+0000","last_became_active":"2021-03-24T21:54:38.109439+0000","last_became_peered":"2021-03-24T21:54:38.109439+0000","last_unstale":"2021-03-24T21:57:13.207247+0000","last_undegraded":"2021-03-24T21:57:13.207247+0000","last_fullsized":"2021-03-24T21:57:13.207247+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:06.042865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:06.042865+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"69'15","reported_seq":"273","reported_epoch":"262","state":"active+clean","last_fresh":"2021-03-24T21:57:11.147005+0000","last_change":"2021-03-24T21:56:02.021432+0000","last_active":"2021-03-24T21:57:11.147005+0000","last_peered":"2021-03-24T21:57:11.147005+0000","last_clean":"2021-03-24T21:57:11.147005+0000","last_became_active":"2021-03-24T21:54:38.139863+0000","last_became_peered":"2021-03-24T21:54:38.139863+0000","last_unstale":"2021-03-24T21:57:11.147005+0000","last_undegraded":"2021-03-24T21:57:11.147005+0000","last_fullsized":"2021-03-24T21:57:11.147005+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:56:02.021288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:02.021288+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":19,"num_write":17,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":692,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"228","reported_epoch":"261","state":"active+clean","last_fresh":"2021-03-24T21:57:10.141597+0000","last_change":"2021-03-24T21:56:51.990882+0000","last_active":"2021-03-24T21:57:10.141597+0000","last_peered":"2021-03-24T21:57:10.141597+0000","last_clean":"2021-03-24T21:57:10.141597+0000","last_became_active":"2021-03-24T21:54:38.124317+0000","last_became_peered":"2021-03-24T21:54:38.124317+0000","last_unstale":"2021-03-24T21:57:10.141597+0000","last_undegraded":"2021-03-24T21:57:10.141597+0000","last_fullsized":"2021-03-24T21:57:10.141597+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:51.990762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:51.990762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"263'203","reported_seq":"755","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.214182+0000","last_change":"2021-03-24T21:57:12.214182+0000","last_active":"2021-03-24T21:57:12.214182+0000","last_peered":"2021-03-24T21:57:12.214182+0000","last_clean":"2021-03-24T21:57:12.214182+0000","last_became_active":"2021-03-24T21:56:08.158313+0000","last_became_peered":"2021-03-24T21:56:08.158313+0000","last_unstale":"2021-03-24T21:57:12.214182+0000","last_undegraded":"2021-03-24T21:57:12.214182+0000","last_fullsized":"2021-03-24T21:57:12.214182+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":203,"ondisk_log_size":203,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10159864,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":61,"num_read_kb":56890,"num_write":51,"num_write_kb":19419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2198240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"82","length":"1"},{"start":"8c","length":"2"},{"start":"92","length":"1"},{"start":"95","length":"1"},{"start":"9c","length":"1"}]},{"pgid":"6.8","version":"0'0","reported_seq":"220","reported_epoch":"264","state":"active+clean","last_fresh":"2021-03-24T21:57:13.201340+0000","last_change":"2021-03-24T21:56:38.970708+0000","last_active":"2021-03-24T21:57:13.201340+0000","last_peered":"2021-03-24T21:57:13.201340+0000","last_clean":"2021-03-24T21:57:13.201340+0000","last_became_active":"2021-03-24T21:51:51.762513+0000","last_became_peered":"2021-03-24T21:51:51.762513+0000","last_unstale":"2021-03-24T21:57:13.201340+0000","last_undegraded":"2021-03-24T21:57:13.201340+0000","last_fullsized":"2021-03-24T21:57:13.201340+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:38.970539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:38.970539+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"23","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.208140+0000","last_change":"2021-03-24T21:57:02.185635+0000","last_active":"2021-03-24T21:57:12.208140+0000","last_peered":"2021-03-24T21:57:12.208140+0000","last_clean":"2021-03-24T21:57:12.208140+0000","last_became_active":"2021-03-24T21:57:02.185281+0000","last_became_peered":"2021-03-24T21:57:02.185281+0000","last_unstale":"2021-03-24T21:57:12.208140+0000","last_undegraded":"2021-03-24T21:57:12.208140+0000","last_fullsized":"2021-03-24T21:57:12.208140+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:44.018043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:44.018043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":"230","reported_epoch":"264","state":"active+clean","last_fresh":"2021-03-24T21:57:13.206015+0000","last_change":"2021-03-24T21:55:38.974245+0000","last_active":"2021-03-24T21:57:13.206015+0000","last_peered":"2021-03-24T21:57:13.206015+0000","last_clean":"2021-03-24T21:57:13.206015+0000","last_became_active":"2021-03-24T21:54:24.189299+0000","last_became_peered":"2021-03-24T21:54:24.189299+0000","last_unstale":"2021-03-24T21:57:13.206015+0000","last_undegraded":"2021-03-24T21:57:13.206015+0000","last_fullsized":"2021-03-24T21:57:13.206015+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:38.974155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:38.974155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"69'23","reported_seq":"253","reported_epoch":"264","state":"active+clean","last_fresh":"2021-03-24T21:57:13.202112+0000","last_change":"2021-03-24T21:56:34.992132+0000","last_active":"2021-03-24T21:57:13.202112+0000","last_peered":"2021-03-24T21:57:13.202112+0000","last_clean":"2021-03-24T21:57:13.202112+0000","last_became_active":"2021-03-24T21:54:38.138701+0000","last_became_peered":"2021-03-24T21:54:38.138701+0000","last_unstale":"2021-03-24T21:57:13.202112+0000","last_undegraded":"2021-03-24T21:57:13.202112+0000","last_fullsized":"2021-03-24T21:57:13.202112+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'23","last_scrub_stamp":"2021-03-24T21:56:34.991954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:34.991954+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":43,"num_read_kb":24,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":732,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"258'118","reported_seq":"628","reported_epoch":"261","state":"active+clean","last_fresh":"2021-03-24T21:57:10.142806+0000","last_change":"2021-03-24T21:57:10.142806+0000","last_active":"2021-03-24T21:57:10.142806+0000","last_peered":"2021-03-24T21:57:10.142806+0000","last_clean":"2021-03-24T21:57:10.142806+0000","last_became_active":"2021-03-24T21:56:08.148045+0000","last_became_peered":"2021-03-24T21:56:08.148045+0000","last_unstale":"2021-03-24T21:57:10.142806+0000","last_undegraded":"2021-03-24T21:57:10.142806+0000","last_fullsized":"2021-03-24T21:57:10.142806+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'96","last_scrub_stamp":"2021-03-24T21:56:40.081137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:56:40.081137+0000","log_size":118,"ondisk_log_size":118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":1474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"82","length":"1"},{"start":"8c","length":"2"},{"start":"92","length":"1"},{"start":"95","length":"1"}]},{"pgid":"6.9","version":"0'0","reported_seq":"222","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.206418+0000","last_change":"2021-03-24T21:57:02.189636+0000","last_active":"2021-03-24T21:57:12.206418+0000","last_peered":"2021-03-24T21:57:12.206418+0000","last_clean":"2021-03-24T21:57:12.206418+0000","last_became_active":"2021-03-24T21:57:02.188645+0000","last_became_peered":"2021-03-24T21:57:02.188645+0000","last_unstale":"2021-03-24T21:57:12.206418+0000","last_undegraded":"2021-03-24T21:57:12.206418+0000","last_fullsized":"2021-03-24T21:57:12.206418+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:33.246205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:35.636299+0000","last_clean_scrub_stamp":"2021-03-24T21:55:33.246205+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"219","reported_epoch":"261","state":"active+clean","last_fresh":"2021-03-24T21:57:10.141087+0000","last_change":"2021-03-24T21:56:43.052062+0000","last_active":"2021-03-24T21:57:10.141087+0000","last_peered":"2021-03-24T21:57:10.141087+0000","last_clean":"2021-03-24T21:57:10.141087+0000","last_became_active":"2021-03-24T21:51:49.411921+0000","last_became_peered":"2021-03-24T21:51:49.411921+0000","last_unstale":"2021-03-24T21:57:10.141087+0000","last_undegraded":"2021-03-24T21:57:10.141087+0000","last_fullsized":"2021-03-24T21:57:10.141087+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:43.051885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:43.051885+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"214","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.514227+0000","last_change":"2021-03-24T21:57:12.514184+0000","last_active":"2021-03-24T21:57:12.514227+0000","last_peered":"2021-03-24T21:57:12.514227+0000","last_clean":"2021-03-24T21:57:12.514227+0000","last_became_active":"2021-03-24T21:55:10.102360+0000","last_became_peered":"2021-03-24T21:55:10.102360+0000","last_unstale":"2021-03-24T21:57:12.514227+0000","last_undegraded":"2021-03-24T21:57:12.514227+0000","last_fullsized":"2021-03-24T21:57:12.514227+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:12.514094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:12.514094+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"69'14","reported_seq":"272","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.208290+0000","last_change":"2021-03-24T21:55:57.297176+0000","last_active":"2021-03-24T21:57:12.208290+0000","last_peered":"2021-03-24T21:57:12.208290+0000","last_clean":"2021-03-24T21:57:12.208290+0000","last_became_active":"2021-03-24T21:54:38.138034+0000","last_became_peered":"2021-03-24T21:54:38.138034+0000","last_unstale":"2021-03-24T21:57:12.208290+0000","last_undegraded":"2021-03-24T21:57:12.208290+0000","last_fullsized":"2021-03-24T21:57:12.208290+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'14","last_scrub_stamp":"2021-03-24T21:55:57.297084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:57.297084+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":16,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":894,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"250'207","reported_seq":"767","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.214988+0000","last_change":"2021-03-24T21:57:12.214988+0000","last_active":"2021-03-24T21:57:12.214988+0000","last_peered":"2021-03-24T21:57:12.214988+0000","last_clean":"2021-03-24T21:57:12.214988+0000","last_became_active":"2021-03-24T21:56:08.157153+0000","last_became_peered":"2021-03-24T21:56:08.157153+0000","last_unstale":"2021-03-24T21:57:12.214988+0000","last_undegraded":"2021-03-24T21:57:12.214988+0000","last_fullsized":"2021-03-24T21:57:12.214988+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":207,"ondisk_log_size":207,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4042116,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":80,"num_read_kb":87094,"num_write":51,"num_write_kb":21606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2579004,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"82","length":"1"},{"start":"8c","length":"2"},{"start":"92","length":"1"},{"start":"95","length":"1"},{"start":"9c","length":"1"}]},{"pgid":"6.6","version":"0'0","reported_seq":"213","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.194465+0000","last_change":"2021-03-24T21:54:10.519515+0000","last_active":"2021-03-24T21:57:12.194465+0000","last_peered":"2021-03-24T21:57:12.194465+0000","last_clean":"2021-03-24T21:57:12.194465+0000","last_became_active":"2021-03-24T21:51:51.759514+0000","last_became_peered":"2021-03-24T21:51:51.759514+0000","last_unstale":"2021-03-24T21:57:12.194465+0000","last_undegraded":"2021-03-24T21:57:12.194465+0000","last_fullsized":"2021-03-24T21:57:12.194465+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:10.519411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:52:59.706511+0000","last_clean_scrub_stamp":"2021-03-24T21:54:10.519411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"209","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.203536+0000","last_change":"2021-03-24T21:55:10.100603+0000","last_active":"2021-03-24T21:57:12.203536+0000","last_peered":"2021-03-24T21:57:12.203536+0000","last_clean":"2021-03-24T21:57:12.203536+0000","last_became_active":"2021-03-24T21:55:10.100245+0000","last_became_peered":"2021-03-24T21:55:10.100245+0000","last_unstale":"2021-03-24T21:57:12.203536+0000","last_undegraded":"2021-03-24T21:57:12.203536+0000","last_fullsized":"2021-03-24T21:57:12.203536+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:03.195853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:06.800417+0000","last_clean_scrub_stamp":"2021-03-24T21:55:03.195853+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"89'2","reported_seq":"248","reported_epoch":"262","state":"active+clean","last_fresh":"2021-03-24T21:57:11.145573+0000","last_change":"2021-03-24T21:55:38.046157+0000","last_active":"2021-03-24T21:57:11.145573+0000","last_peered":"2021-03-24T21:57:11.145573+0000","last_clean":"2021-03-24T21:57:11.145573+0000","last_became_active":"2021-03-24T21:54:38.144013+0000","last_became_peered":"2021-03-24T21:54:38.144013+0000","last_unstale":"2021-03-24T21:57:11.145573+0000","last_undegraded":"2021-03-24T21:57:11.145573+0000","last_fullsized":"2021-03-24T21:57:11.145573+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2021-03-24T21:55:38.045901+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2021-03-24T21:53:28.219900+0000","last_clean_scrub_stamp":"2021-03-24T21:55:38.045901+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"69'25","reported_seq":"256","reported_epoch":"261","state":"active+clean","last_fresh":"2021-03-24T21:57:10.143961+0000","last_change":"2021-03-24T21:55:29.056321+0000","last_active":"2021-03-24T21:57:10.143961+0000","last_peered":"2021-03-24T21:57:10.143961+0000","last_clean":"2021-03-24T21:57:10.143961+0000","last_became_active":"2021-03-24T21:54:38.141478+0000","last_became_peered":"2021-03-24T21:54:38.141478+0000","last_unstale":"2021-03-24T21:57:10.143961+0000","last_undegraded":"2021-03-24T21:57:10.143961+0000","last_fullsized":"2021-03-24T21:57:10.143961+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'25","last_scrub_stamp":"2021-03-24T21:55:29.056144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:29.056144+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":53,"num_read_kb":30,"num_write":28,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"224","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.204288+0000","last_change":"2021-03-24T21:55:36.236833+0000","last_active":"2021-03-24T21:57:12.204288+0000","last_peered":"2021-03-24T21:57:12.204288+0000","last_clean":"2021-03-24T21:57:12.204288+0000","last_became_active":"2021-03-24T21:54:38.136859+0000","last_became_peered":"2021-03-24T21:54:38.136859+0000","last_unstale":"2021-03-24T21:57:12.204288+0000","last_undegraded":"2021-03-24T21:57:12.204288+0000","last_fullsized":"2021-03-24T21:57:12.204288+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:36.236594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:55:36.236594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"260'175","reported_seq":"720","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.205707+0000","last_change":"2021-03-24T21:57:12.205707+0000","last_active":"2021-03-24T21:57:12.205707+0000","last_peered":"2021-03-24T21:57:12.205707+0000","last_clean":"2021-03-24T21:57:12.205707+0000","last_became_active":"2021-03-24T21:56:08.154125+0000","last_became_peered":"2021-03-24T21:56:08.154125+0000","last_unstale":"2021-03-24T21:57:12.205707+0000","last_undegraded":"2021-03-24T21:57:12.205707+0000","last_fullsized":"2021-03-24T21:57:12.205707+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":175,"ondisk_log_size":175,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6704317,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":47,"num_read_kb":36311,"num_write":50,"num_write_kb":22027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2471246,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"82","length":"1"},{"start":"8c","length":"2"},{"start":"92","length":"1"},{"start":"95","length":"1"},{"start":"9c","length":"1"}]},{"pgid":"6.5","version":"0'0","reported_seq":"224","reported_epoch":"264","state":"active+clean","last_fresh":"2021-03-24T21:57:13.206219+0000","last_change":"2021-03-24T21:55:34.931146+0000","last_active":"2021-03-24T21:57:13.206219+0000","last_peered":"2021-03-24T21:57:13.206219+0000","last_clean":"2021-03-24T21:57:13.206219+0000","last_became_active":"2021-03-24T21:54:24.186957+0000","last_became_peered":"2021-03-24T21:54:24.186957+0000","last_unstale":"2021-03-24T21:57:13.206219+0000","last_undegraded":"2021-03-24T21:57:13.206219+0000","last_fullsized":"2021-03-24T21:57:13.206219+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:34.930967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:55:34.930967+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"61'1","reported_seq":"180","reported_epoch":"261","state":"active+clean","last_fresh":"2021-03-24T21:57:10.141797+0000","last_change":"2021-03-24T21:55:30.102312+0000","last_active":"2021-03-24T21:57:10.141797+0000","last_peered":"2021-03-24T21:57:10.141797+0000","last_clean":"2021-03-24T21:57:10.141797+0000","last_became_active":"2021-03-24T21:54:38.123139+0000","last_became_peered":"2021-03-24T21:54:38.123139+0000","last_unstale":"2021-03-24T21:57:10.141797+0000","last_undegraded":"2021-03-24T21:57:10.141797+0000","last_fullsized":"2021-03-24T21:57:10.141797+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2021-03-24T21:55:30.102193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:30.102193+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"227","reported_epoch":"261","state":"active+clean","last_fresh":"2021-03-24T21:57:10.140873+0000","last_change":"2021-03-24T21:56:03.096959+0000","last_active":"2021-03-24T21:57:10.140873+0000","last_peered":"2021-03-24T21:57:10.140873+0000","last_clean":"2021-03-24T21:57:10.140873+0000","last_became_active":"2021-03-24T21:51:40.351000+0000","last_became_peered":"2021-03-24T21:51:40.351000+0000","last_unstale":"2021-03-24T21:57:10.140873+0000","last_undegraded":"2021-03-24T21:57:10.140873+0000","last_fullsized":"2021-03-24T21:57:10.140873+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:03.096786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:39.286101+0000","last_clean_scrub_stamp":"2021-03-24T21:56:03.096786+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"89'22","reported_seq":"260","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.184340+0000","last_change":"2021-03-24T21:56:54.046391+0000","last_active":"2021-03-24T21:57:12.184340+0000","last_peered":"2021-03-24T21:57:12.184340+0000","last_clean":"2021-03-24T21:57:12.184340+0000","last_became_active":"2021-03-24T21:54:38.129459+0000","last_became_peered":"2021-03-24T21:54:38.129459+0000","last_unstale":"2021-03-24T21:57:12.184340+0000","last_undegraded":"2021-03-24T21:57:12.184340+0000","last_fullsized":"2021-03-24T21:57:12.184340+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'22","last_scrub_stamp":"2021-03-24T21:56:54.046194+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2021-03-24T21:53:19.717244+0000","last_clean_scrub_stamp":"2021-03-24T21:56:54.046194+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":36,"num_read_kb":21,"num_write":18,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":644,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"23","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.186386+0000","last_change":"2021-03-24T21:57:02.183062+0000","last_active":"2021-03-24T21:57:12.186386+0000","last_peered":"2021-03-24T21:57:12.186386+0000","last_clean":"2021-03-24T21:57:12.186386+0000","last_became_active":"2021-03-24T21:57:02.182240+0000","last_became_peered":"2021-03-24T21:57:02.182240+0000","last_unstale":"2021-03-24T21:57:12.186386+0000","last_undegraded":"2021-03-24T21:57:12.186386+0000","last_fullsized":"2021-03-24T21:57:12.186386+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:31.118766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:55:31.118766+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"260'132","reported_seq":"742","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.215434+0000","last_change":"2021-03-24T21:57:12.215434+0000","last_active":"2021-03-24T21:57:12.215434+0000","last_peered":"2021-03-24T21:57:12.215434+0000","last_clean":"2021-03-24T21:57:12.215434+0000","last_became_active":"2021-03-24T21:57:09.179214+0000","last_became_peered":"2021-03-24T21:57:09.179214+0000","last_unstale":"2021-03-24T21:57:12.215434+0000","last_undegraded":"2021-03-24T21:57:12.215434+0000","last_fullsized":"2021-03-24T21:57:12.215434+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":132,"ondisk_log_size":132,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4790011,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":59,"num_read_kb":68271,"num_write":33,"num_write_kb":15363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":8481953,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"82","length":"1"},{"start":"8c","length":"2"},{"start":"92","length":"1"},{"start":"95","length":"1"},{"start":"9c","length":"1"}]},{"pgid":"6.7","version":"0'0","reported_seq":"221","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.184122+0000","last_change":"2021-03-24T21:56:04.174062+0000","last_active":"2021-03-24T21:57:12.184122+0000","last_peered":"2021-03-24T21:57:12.184122+0000","last_clean":"2021-03-24T21:57:12.184122+0000","last_became_active":"2021-03-24T21:51:51.765236+0000","last_became_peered":"2021-03-24T21:51:51.765236+0000","last_unstale":"2021-03-24T21:57:12.184122+0000","last_undegraded":"2021-03-24T21:57:12.184122+0000","last_fullsized":"2021-03-24T21:57:12.184122+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:04.173942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:04.173942+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"249","reported_epoch":"264","state":"active+clean","last_fresh":"2021-03-24T21:57:13.201579+0000","last_change":"2021-03-24T21:56:05.075868+0000","last_active":"2021-03-24T21:57:13.201579+0000","last_peered":"2021-03-24T21:57:13.201579+0000","last_clean":"2021-03-24T21:57:13.201579+0000","last_became_active":"2021-03-24T21:54:38.110327+0000","last_became_peered":"2021-03-24T21:54:38.110327+0000","last_unstale":"2021-03-24T21:57:13.201579+0000","last_undegraded":"2021-03-24T21:57:13.201579+0000","last_fullsized":"2021-03-24T21:57:13.201579+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:05.075677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:49:51.529051+0000","last_clean_scrub_stamp":"2021-03-24T21:56:05.075677+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"228","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.190384+0000","last_change":"2021-03-24T21:55:33.534038+0000","last_active":"2021-03-24T21:57:12.190384+0000","last_peered":"2021-03-24T21:57:12.190384+0000","last_clean":"2021-03-24T21:57:12.190384+0000","last_became_active":"2021-03-24T21:54:24.199349+0000","last_became_peered":"2021-03-24T21:54:24.199349+0000","last_unstale":"2021-03-24T21:57:12.190384+0000","last_undegraded":"2021-03-24T21:57:12.190384+0000","last_fullsized":"2021-03-24T21:57:12.190384+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:33.533840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:33.533840+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"69'15","reported_seq":"234","reported_epoch":"262","state":"active+clean","last_fresh":"2021-03-24T21:57:11.146174+0000","last_change":"2021-03-24T21:55:30.166245+0000","last_active":"2021-03-24T21:57:11.146174+0000","last_peered":"2021-03-24T21:57:11.146174+0000","last_clean":"2021-03-24T21:57:11.146174+0000","last_became_active":"2021-03-24T21:54:24.194711+0000","last_became_peered":"2021-03-24T21:54:24.194711+0000","last_unstale":"2021-03-24T21:57:11.146174+0000","last_undegraded":"2021-03-24T21:57:11.146174+0000","last_fullsized":"2021-03-24T21:57:11.146174+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:55:30.166107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:30.166107+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":26,"num_read_kb":14,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":466,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"217","reported_epoch":"264","state":"active+clean","last_fresh":"2021-03-24T21:57:13.205386+0000","last_change":"2021-03-24T21:54:49.807761+0000","last_active":"2021-03-24T21:57:13.205386+0000","last_peered":"2021-03-24T21:57:13.205386+0000","last_clean":"2021-03-24T21:57:13.205386+0000","last_became_active":"2021-03-24T21:51:49.413251+0000","last_became_peered":"2021-03-24T21:51:49.413251+0000","last_unstale":"2021-03-24T21:57:13.205386+0000","last_undegraded":"2021-03-24T21:57:13.205386+0000","last_fullsized":"2021-03-24T21:57:13.205386+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:49.807585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:54:49.807585+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"195'69","reported_seq":"603","reported_epoch":"265","state":"active+clean","last_fresh":"2021-03-24T21:57:14.217665+0000","last_change":"2021-03-24T21:57:14.218036+0000","last_active":"2021-03-24T21:57:14.217665+0000","last_peered":"2021-03-24T21:57:14.217665+0000","last_clean":"2021-03-24T21:57:14.217665+0000","last_became_active":"2021-03-24T21:56:08.152168+0000","last_became_peered":"2021-03-24T21:56:08.152168+0000","last_unstale":"2021-03-24T21:57:14.217665+0000","last_undegraded":"2021-03-24T21:57:14.217665+0000","last_fullsized":"2021-03-24T21:57:14.217665+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'69","last_scrub_stamp":"2021-03-24T21:56:49.886058+0000","last_deep_scrub":"195'69","last_deep_scrub_stamp":"2021-03-24T21:56:49.886058+0000","last_clean_scrub_stamp":"2021-03-24T21:56:49.886058+0000","log_size":69,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"82","length":"1"},{"start":"8b","length":"3"},{"start":"92","length":"1"},{"start":"95","length":"1"},{"start":"9c","length":"1"}]},{"pgid":"6.4","version":"0'0","reported_seq":"17","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.204844+0000","last_change":"2021-03-24T21:57:06.153154+0000","last_active":"2021-03-24T21:57:12.204844+0000","last_peered":"2021-03-24T21:57:12.204844+0000","last_clean":"2021-03-24T21:57:12.204844+0000","last_became_active":"2021-03-24T21:57:06.152772+0000","last_became_peered":"2021-03-24T21:57:06.152772+0000","last_unstale":"2021-03-24T21:57:12.204844+0000","last_undegraded":"2021-03-24T21:57:12.204844+0000","last_fullsized":"2021-03-24T21:57:12.204844+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:41.503758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:55:41.503758+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"224","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.206074+0000","last_change":"2021-03-24T21:56:38.304954+0000","last_active":"2021-03-24T21:57:12.206074+0000","last_peered":"2021-03-24T21:57:12.206074+0000","last_clean":"2021-03-24T21:57:12.206074+0000","last_became_active":"2021-03-24T21:51:45.376687+0000","last_became_peered":"2021-03-24T21:51:45.376687+0000","last_unstale":"2021-03-24T21:57:12.206074+0000","last_undegraded":"2021-03-24T21:57:12.206074+0000","last_fullsized":"2021-03-24T21:57:12.206074+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:38.304792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:38.304792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"66'5","reported_seq":"573","reported_epoch":"264","state":"active+clean","last_fresh":"2021-03-24T21:57:13.430381+0000","last_change":"2021-03-24T21:56:37.973047+0000","last_active":"2021-03-24T21:57:13.430381+0000","last_peered":"2021-03-24T21:57:13.430381+0000","last_clean":"2021-03-24T21:57:13.430381+0000","last_became_active":"2021-03-24T21:54:38.125434+0000","last_became_peered":"2021-03-24T21:54:38.125434+0000","last_unstale":"2021-03-24T21:57:13.430381+0000","last_undegraded":"2021-03-24T21:57:13.430381+0000","last_fullsized":"2021-03-24T21:57:13.430381+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2021-03-24T21:56:37.972874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:39.286101+0000","last_clean_scrub_stamp":"2021-03-24T21:56:37.972874+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":326,"num_read_kb":321,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"69'16","reported_seq":"305","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.213857+0000","last_change":"2021-03-24T21:57:03.051102+0000","last_active":"2021-03-24T21:57:12.213857+0000","last_peered":"2021-03-24T21:57:12.213857+0000","last_clean":"2021-03-24T21:57:12.213857+0000","last_became_active":"2021-03-24T21:57:02.185489+0000","last_became_peered":"2021-03-24T21:57:02.185489+0000","last_unstale":"2021-03-24T21:57:12.213857+0000","last_undegraded":"2021-03-24T21:57:12.213857+0000","last_fullsized":"2021-03-24T21:57:12.213857+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T21:54:57.325723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:54:57.325723+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":16,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1260,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"232","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.184564+0000","last_change":"2021-03-24T21:56:43.194373+0000","last_active":"2021-03-24T21:57:12.184564+0000","last_peered":"2021-03-24T21:57:12.184564+0000","last_clean":"2021-03-24T21:57:12.184564+0000","last_became_active":"2021-03-24T21:54:38.128708+0000","last_became_peered":"2021-03-24T21:54:38.128708+0000","last_unstale":"2021-03-24T21:57:12.184564+0000","last_undegraded":"2021-03-24T21:57:12.184564+0000","last_fullsized":"2021-03-24T21:57:12.184564+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:43.194160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:43.194160+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"175'22","reported_seq":"561","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.179992+0000","last_change":"2021-03-24T21:57:12.179992+0000","last_active":"2021-03-24T21:57:12.179992+0000","last_peered":"2021-03-24T21:57:12.179992+0000","last_clean":"2021-03-24T21:57:12.179992+0000","last_became_active":"2021-03-24T21:56:08.148888+0000","last_became_peered":"2021-03-24T21:56:08.148888+0000","last_unstale":"2021-03-24T21:57:12.179992+0000","last_undegraded":"2021-03-24T21:57:12.179992+0000","last_fullsized":"2021-03-24T21:57:12.179992+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'22","last_scrub_stamp":"2021-03-24T21:56:40.276617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:56:40.276617+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"82","length":"1"},{"start":"8c","length":"2"},{"start":"92","length":"1"},{"start":"95","length":"1"},{"start":"9c","length":"1"}]},{"pgid":"6.1","version":"0'0","reported_seq":"76","reported_epoch":"264","state":"active+clean","last_fresh":"2021-03-24T21:57:13.206297+0000","last_change":"2021-03-24T21:57:02.189486+0000","last_active":"2021-03-24T21:57:13.206297+0000","last_peered":"2021-03-24T21:57:13.206297+0000","last_clean":"2021-03-24T21:57:13.206297+0000","last_became_active":"2021-03-24T21:57:02.189009+0000","last_became_peered":"2021-03-24T21:57:02.189009+0000","last_unstale":"2021-03-24T21:57:13.206297+0000","last_undegraded":"2021-03-24T21:57:13.206297+0000","last_fullsized":"2021-03-24T21:57:13.206297+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:01.124573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:01.124573+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"26","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.179809+0000","last_change":"2021-03-24T21:57:02.189207+0000","last_active":"2021-03-24T21:57:12.179809+0000","last_peered":"2021-03-24T21:57:12.179809+0000","last_clean":"2021-03-24T21:57:12.179809+0000","last_became_active":"2021-03-24T21:57:02.188835+0000","last_became_peered":"2021-03-24T21:57:02.188835+0000","last_unstale":"2021-03-24T21:57:12.179809+0000","last_undegraded":"2021-03-24T21:57:12.179809+0000","last_fullsized":"2021-03-24T21:57:12.179809+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:38.144956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:38.144956+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"69'21","reported_seq":"262","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.208547+0000","last_change":"2021-03-24T21:56:39.335724+0000","last_active":"2021-03-24T21:57:12.208547+0000","last_peered":"2021-03-24T21:57:12.208547+0000","last_clean":"2021-03-24T21:57:12.208547+0000","last_became_active":"2021-03-24T21:51:47.686486+0000","last_became_peered":"2021-03-24T21:51:47.686486+0000","last_unstale":"2021-03-24T21:57:12.208547+0000","last_undegraded":"2021-03-24T21:57:12.208547+0000","last_fullsized":"2021-03-24T21:57:12.208547+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'21","last_scrub_stamp":"2021-03-24T21:56:39.335613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:39.335613+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":45,"num_read_kb":27,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"178","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.203369+0000","last_change":"2021-03-24T21:55:58.253884+0000","last_active":"2021-03-24T21:57:12.203369+0000","last_peered":"2021-03-24T21:57:12.203369+0000","last_clean":"2021-03-24T21:57:12.203369+0000","last_became_active":"2021-03-24T21:54:24.198895+0000","last_became_peered":"2021-03-24T21:54:24.198895+0000","last_unstale":"2021-03-24T21:57:12.203369+0000","last_undegraded":"2021-03-24T21:57:12.203369+0000","last_fullsized":"2021-03-24T21:57:12.203369+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:58.253777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:55:58.253777+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"261'234","reported_seq":"869","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.211985+0000","last_change":"2021-03-24T21:57:12.212332+0000","last_active":"2021-03-24T21:57:12.211985+0000","last_peered":"2021-03-24T21:57:12.211985+0000","last_clean":"2021-03-24T21:57:12.211985+0000","last_became_active":"2021-03-24T21:56:39.160959+0000","last_became_peered":"2021-03-24T21:56:39.160959+0000","last_unstale":"2021-03-24T21:57:12.211985+0000","last_undegraded":"2021-03-24T21:57:12.211985+0000","last_fullsized":"2021-03-24T21:57:12.211985+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":234,"ondisk_log_size":234,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8699491,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":121,"num_read_kb":116703,"num_write":70,"num_write_kb":29218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":10225230,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"82","length":"1"},{"start":"8c","length":"2"},{"start":"92","length":"1"},{"start":"95","length":"1"},{"start":"9c","length":"1"}]},{"pgid":"6.0","version":"0'0","reported_seq":"213","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.203007+0000","last_change":"2021-03-24T21:55:39.261965+0000","last_active":"2021-03-24T21:57:12.203007+0000","last_peered":"2021-03-24T21:57:12.203007+0000","last_clean":"2021-03-24T21:57:12.203007+0000","last_became_active":"2021-03-24T21:51:51.759627+0000","last_became_peered":"2021-03-24T21:51:51.759627+0000","last_unstale":"2021-03-24T21:57:12.203007+0000","last_undegraded":"2021-03-24T21:57:12.203007+0000","last_fullsized":"2021-03-24T21:57:12.203007+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:39.261698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:55:39.261698+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"227","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.183691+0000","last_change":"2021-03-24T21:55:57.194772+0000","last_active":"2021-03-24T21:57:12.183691+0000","last_peered":"2021-03-24T21:57:12.183691+0000","last_clean":"2021-03-24T21:57:12.183691+0000","last_became_active":"2021-03-24T21:51:45.583885+0000","last_became_peered":"2021-03-24T21:51:45.583885+0000","last_unstale":"2021-03-24T21:57:12.183691+0000","last_undegraded":"2021-03-24T21:57:12.183691+0000","last_fullsized":"2021-03-24T21:57:12.183691+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:57.194610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:57.194610+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"69'16","reported_seq":"241","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.173041+0000","last_change":"2021-03-24T21:56:41.235280+0000","last_active":"2021-03-24T21:57:12.173041+0000","last_peered":"2021-03-24T21:57:12.173041+0000","last_clean":"2021-03-24T21:57:12.173041+0000","last_became_active":"2021-03-24T21:54:38.128745+0000","last_became_peered":"2021-03-24T21:54:38.128745+0000","last_unstale":"2021-03-24T21:57:12.173041+0000","last_undegraded":"2021-03-24T21:57:12.173041+0000","last_fullsized":"2021-03-24T21:57:12.173041+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T21:56:41.235143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:41.235143+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":32,"num_read_kb":15,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":940,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"102'4","reported_seq":"296","reported_epoch":"264","state":"active+clean","last_fresh":"2021-03-24T21:57:13.197050+0000","last_change":"2021-03-24T21:56:01.267040+0000","last_active":"2021-03-24T21:57:13.197050+0000","last_peered":"2021-03-24T21:57:13.197050+0000","last_clean":"2021-03-24T21:57:13.197050+0000","last_became_active":"2021-03-24T21:54:24.203310+0000","last_became_peered":"2021-03-24T21:54:24.203310+0000","last_unstale":"2021-03-24T21:57:13.197050+0000","last_undegraded":"2021-03-24T21:57:13.197050+0000","last_fullsized":"2021-03-24T21:57:13.197050+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'4","last_scrub_stamp":"2021-03-24T21:56:01.266936+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-24T21:53:26.584678+0000","last_clean_scrub_stamp":"2021-03-24T21:56:01.266936+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"256'44","reported_seq":"578","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.190879+0000","last_change":"2021-03-24T21:57:12.190879+0000","last_active":"2021-03-24T21:57:12.190879+0000","last_peered":"2021-03-24T21:57:12.190879+0000","last_clean":"2021-03-24T21:57:12.190879+0000","last_became_active":"2021-03-24T21:56:08.148967+0000","last_became_peered":"2021-03-24T21:56:08.148967+0000","last_unstale":"2021-03-24T21:57:12.190879+0000","last_undegraded":"2021-03-24T21:57:12.190879+0000","last_fullsized":"2021-03-24T21:57:12.190879+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1268961,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":17035,"num_write":13,"num_write_kb":4203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":433862,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"82","length":"1"},{"start":"8c","length":"2"},{"start":"92","length":"1"},{"start":"95","length":"1"},{"start":"9c","length":"1"}]},{"pgid":"6.3","version":"0'0","reported_seq":"238","reported_epoch":"264","state":"active+clean","last_fresh":"2021-03-24T21:57:13.202962+0000","last_change":"2021-03-24T21:56:40.011790+0000","last_active":"2021-03-24T21:57:13.202962+0000","last_peered":"2021-03-24T21:57:13.202962+0000","last_clean":"2021-03-24T21:57:13.202962+0000","last_became_active":"2021-03-24T21:54:38.110246+0000","last_became_peered":"2021-03-24T21:54:38.110246+0000","last_unstale":"2021-03-24T21:57:13.202962+0000","last_undegraded":"2021-03-24T21:57:13.202962+0000","last_fullsized":"2021-03-24T21:57:13.202962+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:40.011633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:40.011633+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"237","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.213918+0000","last_change":"2021-03-24T21:56:44.229920+0000","last_active":"2021-03-24T21:57:12.213918+0000","last_peered":"2021-03-24T21:57:12.213918+0000","last_clean":"2021-03-24T21:57:12.213918+0000","last_became_active":"2021-03-24T21:56:44.229469+0000","last_became_peered":"2021-03-24T21:56:44.229469+0000","last_unstale":"2021-03-24T21:57:12.213918+0000","last_undegraded":"2021-03-24T21:57:12.213918+0000","last_fullsized":"2021-03-24T21:57:12.213918+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:28.165408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:28.165408+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"69'20","reported_seq":"287","reported_epoch":"262","state":"active+clean","last_fresh":"2021-03-24T21:57:11.145910+0000","last_change":"2021-03-24T21:55:00.135489+0000","last_active":"2021-03-24T21:57:11.145910+0000","last_peered":"2021-03-24T21:57:11.145910+0000","last_clean":"2021-03-24T21:57:11.145910+0000","last_became_active":"2021-03-24T21:54:24.189182+0000","last_became_peered":"2021-03-24T21:54:24.189182+0000","last_unstale":"2021-03-24T21:57:11.145910+0000","last_undegraded":"2021-03-24T21:57:11.145910+0000","last_fullsized":"2021-03-24T21:57:11.145910+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'20","last_scrub_stamp":"2021-03-24T21:55:00.135328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:00.135328+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":39,"num_read_kb":20,"num_write":20,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":940,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"109'3","reported_seq":"295","reported_epoch":"264","state":"active+clean","last_fresh":"2021-03-24T21:57:13.201681+0000","last_change":"2021-03-24T21:54:39.228237+0000","last_active":"2021-03-24T21:57:13.201681+0000","last_peered":"2021-03-24T21:57:13.201681+0000","last_clean":"2021-03-24T21:57:13.201681+0000","last_became_active":"2021-03-24T21:54:38.135726+0000","last_became_peered":"2021-03-24T21:54:38.135726+0000","last_unstale":"2021-03-24T21:57:13.201681+0000","last_undegraded":"2021-03-24T21:57:13.201681+0000","last_fullsized":"2021-03-24T21:57:13.201681+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2021-03-24T21:54:17.493861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:54:17.493861+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"258'247","reported_seq":"813","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.203018+0000","last_change":"2021-03-24T21:57:12.195187+0000","last_active":"2021-03-24T21:57:12.203018+0000","last_peered":"2021-03-24T21:57:12.203018+0000","last_clean":"2021-03-24T21:57:12.203018+0000","last_became_active":"2021-03-24T21:56:08.148944+0000","last_became_peered":"2021-03-24T21:56:08.148944+0000","last_unstale":"2021-03-24T21:57:12.203018+0000","last_undegraded":"2021-03-24T21:57:12.203018+0000","last_fullsized":"2021-03-24T21:57:12.203018+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":247,"ondisk_log_size":247,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6871586,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":136,"num_read_kb":161152,"num_write":79,"num_write_kb":31931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"82","length":"1"},{"start":"8c","length":"2"},{"start":"92","length":"1"},{"start":"95","length":"1"},{"start":"9c","length":"1"}]},{"pgid":"6.2","version":"0'0","reported_seq":"213","reported_epoch":"262","state":"active+clean","last_fresh":"2021-03-24T21:57:11.145167+0000","last_change":"2021-03-24T21:56:35.045715+0000","last_active":"2021-03-24T21:57:11.145167+0000","last_peered":"2021-03-24T21:57:11.145167+0000","last_clean":"2021-03-24T21:57:11.145167+0000","last_became_active":"2021-03-24T21:51:51.756114+0000","last_became_peered":"2021-03-24T21:51:51.756114+0000","last_unstale":"2021-03-24T21:57:11.145167+0000","last_undegraded":"2021-03-24T21:57:11.145167+0000","last_fullsized":"2021-03-24T21:57:11.145167+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:35.045534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:35.045534+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"219","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.194860+0000","last_change":"2021-03-24T21:54:56.508843+0000","last_active":"2021-03-24T21:57:12.194860+0000","last_peered":"2021-03-24T21:57:12.194860+0000","last_clean":"2021-03-24T21:57:12.194860+0000","last_became_active":"2021-03-24T21:51:45.378704+0000","last_became_peered":"2021-03-24T21:51:45.378704+0000","last_unstale":"2021-03-24T21:57:12.194860+0000","last_undegraded":"2021-03-24T21:57:12.194860+0000","last_fullsized":"2021-03-24T21:57:12.194860+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:56.508652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:54:56.508652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"69'22","reported_seq":"257","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.190825+0000","last_change":"2021-03-24T21:55:28.629625+0000","last_active":"2021-03-24T21:57:12.190825+0000","last_peered":"2021-03-24T21:57:12.190825+0000","last_clean":"2021-03-24T21:57:12.190825+0000","last_became_active":"2021-03-24T21:51:47.691459+0000","last_became_peered":"2021-03-24T21:51:47.691459+0000","last_unstale":"2021-03-24T21:57:12.190825+0000","last_undegraded":"2021-03-24T21:57:12.190825+0000","last_fullsized":"2021-03-24T21:57:12.190825+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'22","last_scrub_stamp":"2021-03-24T21:55:28.629443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:28.629443+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":46,"num_read_kb":26,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"215","reported_epoch":"262","state":"active+clean","last_fresh":"2021-03-24T21:57:11.145981+0000","last_change":"2021-03-24T21:55:25.106515+0000","last_active":"2021-03-24T21:57:11.145981+0000","last_peered":"2021-03-24T21:57:11.145981+0000","last_clean":"2021-03-24T21:57:11.145981+0000","last_became_active":"2021-03-24T21:51:49.408761+0000","last_became_peered":"2021-03-24T21:51:49.408761+0000","last_unstale":"2021-03-24T21:57:11.145981+0000","last_undegraded":"2021-03-24T21:57:11.145981+0000","last_fullsized":"2021-03-24T21:57:11.145981+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:25.106440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:25.220436+0000","last_clean_scrub_stamp":"2021-03-24T21:55:25.106440+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"263'186","reported_seq":"726","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.225636+0000","last_change":"2021-03-24T21:57:12.195092+0000","last_active":"2021-03-24T21:57:12.225636+0000","last_peered":"2021-03-24T21:57:12.225636+0000","last_clean":"2021-03-24T21:57:12.225636+0000","last_became_active":"2021-03-24T21:54:24.190419+0000","last_became_peered":"2021-03-24T21:54:24.190419+0000","last_unstale":"2021-03-24T21:57:12.225636+0000","last_undegraded":"2021-03-24T21:57:12.225636+0000","last_fullsized":"2021-03-24T21:57:12.225636+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":186,"ondisk_log_size":186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15745293,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":130,"num_read_kb":124487,"num_write":90,"num_write_kb":32191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"82","length":"1"},{"start":"8c","length":"2"},{"start":"92","length":"1"},{"start":"95","length":"1"},{"start":"9c","length":"1"}]},{"pgid":"6.d","version":"0'0","reported_seq":"225","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.183339+0000","last_change":"2021-03-24T21:55:28.557021+0000","last_active":"2021-03-24T21:57:12.183339+0000","last_peered":"2021-03-24T21:57:12.183339+0000","last_clean":"2021-03-24T21:57:12.183339+0000","last_became_active":"2021-03-24T21:54:38.129469+0000","last_became_peered":"2021-03-24T21:54:38.129469+0000","last_unstale":"2021-03-24T21:57:12.183339+0000","last_undegraded":"2021-03-24T21:57:12.183339+0000","last_fullsized":"2021-03-24T21:57:12.183339+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:28.556844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:55:28.556844+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"65'2","reported_seq":"281","reported_epoch":"262","state":"active+clean","last_fresh":"2021-03-24T21:57:11.147795+0000","last_change":"2021-03-24T21:55:59.032346+0000","last_active":"2021-03-24T21:57:11.147795+0000","last_peered":"2021-03-24T21:57:11.147795+0000","last_clean":"2021-03-24T21:57:11.147795+0000","last_became_active":"2021-03-24T21:51:49.414380+0000","last_became_peered":"2021-03-24T21:51:49.414380+0000","last_unstale":"2021-03-24T21:57:11.147795+0000","last_undegraded":"2021-03-24T21:57:11.147795+0000","last_fullsized":"2021-03-24T21:57:11.147795+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2021-03-24T21:55:59.032170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:55:59.032170+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":"228","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.190730+0000","last_change":"2021-03-24T21:56:39.439282+0000","last_active":"2021-03-24T21:57:12.190730+0000","last_peered":"2021-03-24T21:57:12.190730+0000","last_clean":"2021-03-24T21:57:12.190730+0000","last_became_active":"2021-03-24T21:54:38.134768+0000","last_became_peered":"2021-03-24T21:54:38.134768+0000","last_unstale":"2021-03-24T21:57:12.190730+0000","last_undegraded":"2021-03-24T21:57:12.190730+0000","last_fullsized":"2021-03-24T21:57:12.190730+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:39.439170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:39.439170+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"69'17","reported_seq":"227","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.194558+0000","last_change":"2021-03-24T21:54:38.505905+0000","last_active":"2021-03-24T21:57:12.194558+0000","last_peered":"2021-03-24T21:57:12.194558+0000","last_clean":"2021-03-24T21:57:12.194558+0000","last_became_active":"2021-03-24T21:54:38.130536+0000","last_became_peered":"2021-03-24T21:54:38.130536+0000","last_unstale":"2021-03-24T21:57:12.194558+0000","last_undegraded":"2021-03-24T21:57:12.194558+0000","last_fullsized":"2021-03-24T21:57:12.194558+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'17","last_scrub_stamp":"2021-03-24T21:54:03.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:54:03.731436+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":18,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":422,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"263'221","reported_seq":"769","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.244411+0000","last_change":"2021-03-24T21:57:12.209708+0000","last_active":"2021-03-24T21:57:12.244411+0000","last_peered":"2021-03-24T21:57:12.244411+0000","last_clean":"2021-03-24T21:57:12.244411+0000","last_became_active":"2021-03-24T21:54:38.123502+0000","last_became_peered":"2021-03-24T21:54:38.123502+0000","last_unstale":"2021-03-24T21:57:12.244411+0000","last_undegraded":"2021-03-24T21:57:12.244411+0000","last_fullsized":"2021-03-24T21:57:12.244411+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'169","last_scrub_stamp":"2021-03-24T21:56:47.285648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:56:47.285648+0000","log_size":221,"ondisk_log_size":221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16470507,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":126,"num_read_kb":117788,"num_write":113,"num_write_kb":42391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":5093012,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"82","length":"1"},{"start":"8c","length":"2"},{"start":"92","length":"1"},{"start":"95","length":"1"},{"start":"9c","length":"1"}]},{"pgid":"6.c","version":"0'0","reported_seq":"222","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.191778+0000","last_change":"2021-03-24T21:55:59.472536+0000","last_active":"2021-03-24T21:57:12.191778+0000","last_peered":"2021-03-24T21:57:12.191778+0000","last_clean":"2021-03-24T21:57:12.191778+0000","last_became_active":"2021-03-24T21:54:24.196587+0000","last_became_peered":"2021-03-24T21:54:24.196587+0000","last_unstale":"2021-03-24T21:57:12.191778+0000","last_undegraded":"2021-03-24T21:57:12.191778+0000","last_fullsized":"2021-03-24T21:57:12.191778+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:59.472393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:55:59.472393+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"197","reported_epoch":"262","state":"active+clean","last_fresh":"2021-03-24T21:57:11.147251+0000","last_change":"2021-03-24T21:56:43.971055+0000","last_active":"2021-03-24T21:57:11.147251+0000","last_peered":"2021-03-24T21:57:11.147251+0000","last_clean":"2021-03-24T21:57:11.147251+0000","last_became_active":"2021-03-24T21:55:10.102113+0000","last_became_peered":"2021-03-24T21:55:10.102113+0000","last_unstale":"2021-03-24T21:57:11.147251+0000","last_undegraded":"2021-03-24T21:57:11.147251+0000","last_fullsized":"2021-03-24T21:57:11.147251+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:43.970879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:43.970879+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"219","reported_epoch":"262","state":"active+clean","last_fresh":"2021-03-24T21:57:11.144834+0000","last_change":"2021-03-24T21:54:53.267409+0000","last_active":"2021-03-24T21:57:11.144834+0000","last_peered":"2021-03-24T21:57:11.144834+0000","last_clean":"2021-03-24T21:57:11.144834+0000","last_became_active":"2021-03-24T21:51:45.579513+0000","last_became_peered":"2021-03-24T21:51:45.579513+0000","last_unstale":"2021-03-24T21:57:11.144834+0000","last_undegraded":"2021-03-24T21:57:11.144834+0000","last_fullsized":"2021-03-24T21:57:11.144834+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:53.267324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:54:53.267324+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"69'15","reported_seq":"307","reported_epoch":"262","state":"active+clean","last_fresh":"2021-03-24T21:57:11.147654+0000","last_change":"2021-03-24T21:56:40.062483+0000","last_active":"2021-03-24T21:57:11.147654+0000","last_peered":"2021-03-24T21:57:11.147654+0000","last_clean":"2021-03-24T21:57:11.147654+0000","last_became_active":"2021-03-24T21:54:38.127461+0000","last_became_peered":"2021-03-24T21:54:38.127461+0000","last_unstale":"2021-03-24T21:57:11.147654+0000","last_undegraded":"2021-03-24T21:57:11.147654+0000","last_fullsized":"2021-03-24T21:57:11.147654+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:56:40.062304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:40.062304+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1476,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"258'43","reported_seq":"556","reported_epoch":"261","state":"active+clean","last_fresh":"2021-03-24T21:57:10.142909+0000","last_change":"2021-03-24T21:57:10.142909+0000","last_active":"2021-03-24T21:57:10.142909+0000","last_peered":"2021-03-24T21:57:10.142909+0000","last_clean":"2021-03-24T21:57:10.142909+0000","last_became_active":"2021-03-24T21:54:22.168397+0000","last_became_peered":"2021-03-24T21:54:22.168397+0000","last_unstale":"2021-03-24T21:57:10.142909+0000","last_undegraded":"2021-03-24T21:57:10.142909+0000","last_fullsized":"2021-03-24T21:57:10.142909+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'30","last_scrub_stamp":"2021-03-24T21:56:10.105084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:56:10.105084+0000","log_size":43,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2537890,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":43,"num_read_kb":36610,"num_write":26,"num_write_kb":6119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"82","length":"1"},{"start":"8c","length":"2"},{"start":"92","length":"1"},{"start":"95","length":"1"}]},{"pgid":"6.f","version":"0'0","reported_seq":"23","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.192728+0000","last_change":"2021-03-24T21:57:02.182962+0000","last_active":"2021-03-24T21:57:12.192728+0000","last_peered":"2021-03-24T21:57:12.192728+0000","last_clean":"2021-03-24T21:57:12.192728+0000","last_became_active":"2021-03-24T21:57:02.181899+0000","last_became_peered":"2021-03-24T21:57:02.181899+0000","last_unstale":"2021-03-24T21:57:12.192728+0000","last_undegraded":"2021-03-24T21:57:12.192728+0000","last_fullsized":"2021-03-24T21:57:12.192728+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:49.970073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:49.970073+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"180","reported_epoch":"262","state":"active+clean","last_fresh":"2021-03-24T21:57:11.147087+0000","last_change":"2021-03-24T21:56:44.230926+0000","last_active":"2021-03-24T21:57:11.147087+0000","last_peered":"2021-03-24T21:57:11.147087+0000","last_clean":"2021-03-24T21:57:11.147087+0000","last_became_active":"2021-03-24T21:56:44.230543+0000","last_became_peered":"2021-03-24T21:56:44.230543+0000","last_unstale":"2021-03-24T21:57:11.147087+0000","last_undegraded":"2021-03-24T21:57:11.147087+0000","last_fullsized":"2021-03-24T21:57:11.147087+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:07.977138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:07.977138+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"186","reported_epoch":"262","state":"active+clean","last_fresh":"2021-03-24T21:57:11.145184+0000","last_change":"2021-03-24T21:55:01.169611+0000","last_active":"2021-03-24T21:57:11.145184+0000","last_peered":"2021-03-24T21:57:11.145184+0000","last_clean":"2021-03-24T21:57:11.145184+0000","last_became_active":"2021-03-24T21:54:38.127136+0000","last_became_peered":"2021-03-24T21:54:38.127136+0000","last_unstale":"2021-03-24T21:57:11.145184+0000","last_undegraded":"2021-03-24T21:57:11.145184+0000","last_fullsized":"2021-03-24T21:57:11.145184+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:01.169461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:55:01.169461+0000","last_clean_scrub_stamp":"2021-03-24T21:55:01.169461+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"69'21","reported_seq":"288","reported_epoch":"262","state":"active+clean","last_fresh":"2021-03-24T21:57:11.145706+0000","last_change":"2021-03-24T21:54:56.226932+0000","last_active":"2021-03-24T21:57:11.145706+0000","last_peered":"2021-03-24T21:57:11.145706+0000","last_clean":"2021-03-24T21:57:11.145706+0000","last_became_active":"2021-03-24T21:54:38.144020+0000","last_became_peered":"2021-03-24T21:54:38.144020+0000","last_unstale":"2021-03-24T21:57:11.145706+0000","last_undegraded":"2021-03-24T21:57:11.145706+0000","last_fullsized":"2021-03-24T21:57:11.145706+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'21","last_scrub_stamp":"2021-03-24T21:54:56.226757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:54:56.226757+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":23,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"262'85","reported_seq":"641","reported_epoch":"264","state":"active+clean","last_fresh":"2021-03-24T21:57:13.198713+0000","last_change":"2021-03-24T21:57:12.215120+0000","last_active":"2021-03-24T21:57:13.198713+0000","last_peered":"2021-03-24T21:57:13.198713+0000","last_clean":"2021-03-24T21:57:13.198713+0000","last_became_active":"2021-03-24T21:57:06.155158+0000","last_became_peered":"2021-03-24T21:57:06.155158+0000","last_unstale":"2021-03-24T21:57:13.198713+0000","last_undegraded":"2021-03-24T21:57:13.198713+0000","last_fullsized":"2021-03-24T21:57:13.198713+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"218'73","last_scrub_stamp":"2021-03-24T21:56:48.267711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:56:48.267711+0000","log_size":85,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7143008,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":69,"num_read_kb":70839,"num_write":42,"num_write_kb":18326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1884965,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"82","length":"1"},{"start":"8c","length":"2"},{"start":"92","length":"1"},{"start":"95","length":"1"},{"start":"9c","length":"1"}]},{"pgid":"6.e","version":"0'0","reported_seq":"231","reported_epoch":"262","state":"active+clean","last_fresh":"2021-03-24T21:57:11.147967+0000","last_change":"2021-03-24T21:57:02.182918+0000","last_active":"2021-03-24T21:57:11.147967+0000","last_peered":"2021-03-24T21:57:11.147967+0000","last_clean":"2021-03-24T21:57:11.147967+0000","last_became_active":"2021-03-24T21:57:02.182551+0000","last_became_peered":"2021-03-24T21:57:02.182551+0000","last_unstale":"2021-03-24T21:57:11.147967+0000","last_undegraded":"2021-03-24T21:57:11.147967+0000","last_fullsized":"2021-03-24T21:57:11.147967+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:08.985694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:08.985694+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"65'2","reported_seq":"285","reported_epoch":"261","state":"active+clean","last_fresh":"2021-03-24T21:57:10.144314+0000","last_change":"2021-03-24T21:56:33.126585+0000","last_active":"2021-03-24T21:57:10.144314+0000","last_peered":"2021-03-24T21:57:10.144314+0000","last_clean":"2021-03-24T21:57:10.144314+0000","last_became_active":"2021-03-24T21:51:49.416182+0000","last_became_peered":"2021-03-24T21:51:49.416182+0000","last_unstale":"2021-03-24T21:57:10.144314+0000","last_undegraded":"2021-03-24T21:57:10.144314+0000","last_fullsized":"2021-03-24T21:57:10.144314+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2021-03-24T21:56:33.126465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:33.126465+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"224","reported_epoch":"263","state":"active+clean","last_fresh":"2021-03-24T21:57:12.191485+0000","last_change":"2021-03-24T21:56:42.413023+0000","last_active":"2021-03-24T21:57:12.191485+0000","last_peered":"2021-03-24T21:57:12.191485+0000","last_clean":"2021-03-24T21:57:12.191485+0000","last_became_active":"2021-03-24T21:51:45.375830+0000","last_became_peered":"2021-03-24T21:51:45.375830+0000","last_unstale":"2021-03-24T21:57:12.191485+0000","last_undegraded":"2021-03-24T21:57:12.191485+0000","last_fullsized":"2021-03-24T21:57:12.191485+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:42.412849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:42.412849+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"69'13","reported_seq":"292","reported_epoch":"262","state":"active+clean","last_fresh":"2021-03-24T21:57:11.147226+0000","last_change":"2021-03-24T21:55:36.065455+0000","last_active":"2021-03-24T21:57:11.147226+0000","last_peered":"2021-03-24T21:57:11.147226+0000","last_clean":"2021-03-24T21:57:11.147226+0000","last_became_active":"2021-03-24T21:54:38.127115+0000","last_became_peered":"2021-03-24T21:54:38.127115+0000","last_unstale":"2021-03-24T21:57:11.147226+0000","last_undegraded":"2021-03-24T21:57:11.147226+0000","last_fullsized":"2021-03-24T21:57:11.147226+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2021-03-24T21:55:36.065267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:36.065267+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":13,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":26,"stat_sum":{"num_bytes":152779003,"num_objects":67,"num_object_clones":24,"num_object_copies":201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":3,"num_read":1626,"num_read_kb":1652410,"num_write":1121,"num_write_kb":439599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":172,"num_bytes_recovered":100945324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":368267264,"data_stored":433403831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3911,"ondisk_log_size":3911,"up":78,"acting":78,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":326,"num_read_kb":321,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1589,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":5558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1061,"num_read_kb":566,"num_write":542,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":322,"num_bytes_recovered":25900,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2027520,"data_stored":72734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":549,"ondisk_log_size":549,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201350,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":91524,"kb_used_data":64572,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93653628,"statfs":{"total":95995035648,"available":95901315072,"internally_reserved":0,"allocated":66121728,"data_stored":73720642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:56:51 2021","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.048,"15min":0.94799999999999995},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":12.222,"5min":12.222,"15min":12.222},"last":0.83799999999999997},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.026,"15min":0.99099999999999999},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":11.506,"5min":11.506,"15min":11.506},"last":0.86099999999999999}]},{"osd":1,"last update":"Wed Mar 24 21:56:51 2021","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":0.98199999999999998,"15min":0.88200000000000001},"min":{"1min":0.57399999999999995,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.645,"5min":5.7750000000000004,"15min":5.7750000000000004},"last":0.93799999999999994},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.98999999999999999,"15min":0.97499999999999998},"min":{"1min":0.65800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.403,"5min":6.0780000000000003,"15min":6.0780000000000003},"last":1.151}]},{"osd":2,"last update":"Wed Mar 24 21:56:51 2021","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.1779999999999999,"15min":1.0549999999999999},"min":{"1min":0.436,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":10.827999999999999,"5min":12.135999999999999,"15min":12.135999999999999},"last":1.0009999999999999},{"interface":"front","average":{"1min":1.4450000000000001,"5min":1.141,"15min":1.0549999999999999},"min":{"1min":0.61799999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":12.178000000000001,"5min":12.178000000000001,"15min":12.178000000000001},"last":0.68799999999999994}]},{"osd":3,"last update":"Wed Mar 24 21:56:51 2021","interfaces":[{"interface":"back","average":{"1min":1.371,"5min":1.1890000000000001,"15min":1.093},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":12.529,"5min":12.529,"15min":12.529},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.198,"15min":1.046},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":12.666,"5min":12.666,"15min":12.666},"last":1.032}]},{"osd":4,"last update":"Wed Mar 24 21:56:51 2021","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":1.0469999999999999,"15min":0.96299999999999997},"min":{"1min":0.54800000000000004,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.6439999999999999,"5min":6.2089999999999996,"15min":6.2089999999999996},"last":1.056},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.1850000000000001,"15min":1.111},"min":{"1min":0.53400000000000003,"5min":0.504,"15min":0.504},"max":{"1min":1.3360000000000001,"5min":12.244,"15min":12.244},"last":1.1739999999999999}]},{"osd":5,"last update":"Wed Mar 24 21:56:51 2021","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.1259999999999999,"15min":1.018},"min":{"1min":0.51500000000000001,"5min":0.435,"15min":0.435},"max":{"1min":1.569,"5min":5.8840000000000003,"15min":5.8840000000000003},"last":1.125},{"interface":"front","average":{"1min":1.131,"5min":1.196,"15min":1.1559999999999999},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.496,"5min":6.3300000000000001,"15min":6.3300000000000001},"last":1.204}]},{"osd":6,"last update":"Wed Mar 24 21:56:51 2021","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.2190000000000001,"15min":1.1439999999999999},"min":{"1min":0.66000000000000003,"5min":0.62,"15min":0.62},"max":{"1min":1.569,"5min":6.1159999999999997,"15min":6.1159999999999997},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.3129999999999999,"15min":1.218},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.4239999999999999,"5min":12.287000000000001,"15min":12.287000000000001},"last":1.238}]}]},{"osd":6,"up_from":0,"seq":201863462986,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":0,"seq":55834574944,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738467,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":90548,"kb_used_data":63916,"kb_used_omap":0,"kb_used_meta":26624,"kb_avail":93654604,"statfs":{"total":95995035648,"available":95902314496,"internally_reserved":0,"allocated":65449984,"data_stored":76965206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27262976},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Mar 24 21:56:43 2021","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.71599999999999997,"15min":0.61599999999999999},"min":{"1min":0.35499999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.2070000000000001,"5min":4.8040000000000003,"15min":4.8040000000000003},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.81699999999999995,"15min":0.66900000000000004},"min":{"1min":0.39800000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.871,"5min":8.8989999999999991,"15min":8.8989999999999991},"last":0.48899999999999999}]},{"osd":2,"last update":"Wed Mar 24 21:57:05 2021","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.76700000000000002,"15min":0.69799999999999995},"min":{"1min":0.43099999999999999,"5min":0.314,"15min":0.314},"max":{"1min":1.962,"5min":5.8810000000000002,"15min":5.8810000000000002},"last":1.1299999999999999},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.84499999999999997,"15min":0.73799999999999999},"min":{"1min":0.61399999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.0069999999999999,"5min":4.766,"15min":4.766},"last":1.1739999999999999}]},{"osd":3,"last update":"Wed Mar 24 21:56:21 2021","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.88400000000000001,"15min":0.82599999999999996},"min":{"1min":0.44600000000000001,"5min":0.376,"15min":0.376},"max":{"1min":1.8560000000000001,"5min":5.9790000000000001,"15min":5.9790000000000001},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.96799999999999997,"15min":0.94299999999999995},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.6279999999999999,"5min":6.4580000000000002,"15min":6.4580000000000002},"last":1.101}]},{"osd":4,"last update":"Wed Mar 24 21:56:43 2021","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":1.0880000000000001,"15min":1.0960000000000001},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.7909999999999999,"5min":5.0330000000000004,"15min":5.0330000000000004},"last":1.2529999999999999},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.1140000000000001,"15min":1.042},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.944,"5min":6.1829999999999998,"15min":6.1829999999999998},"last":0.93500000000000005}]},{"osd":5,"last update":"Wed Mar 24 21:57:05 2021","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":1.0900000000000001,"15min":1.075},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.657,"5min":5.9390000000000001,"15min":5.9390000000000001},"last":0.95199999999999996},{"interface":"front","average":{"1min":1.101,"5min":1.1839999999999999,"15min":1.179},"min":{"1min":0.748,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.4259999999999999,"5min":5.0590000000000002,"15min":5.0590000000000002},"last":1.22}]},{"osd":6,"last update":"Wed Mar 24 21:56:21 2021","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.1299999999999999,"15min":1.0880000000000001},"min":{"1min":0.77500000000000002,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":2.2280000000000002,"5min":5.9930000000000003,"15min":5.9930000000000003},"last":0.98799999999999999},{"interface":"front","average":{"1min":1.087,"5min":1.222,"15min":1.22},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.7549999999999999,"5min":4.9960000000000004,"15min":4.9960000000000004},"last":1.2829999999999999}]},{"osd":7,"last update":"Wed Mar 24 21:56:56 2021","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.2070000000000001,"15min":1.181},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.403,"5min":6.1319999999999997,"15min":6.1319999999999997},"last":1.0329999999999999},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.2030000000000001,"15min":1.151},"min":{"1min":0.77600000000000002,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.9059999999999999,"5min":6.0529999999999999,"15min":6.0529999999999999},"last":0.86899999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411420,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":84044,"kb_used_data":57924,"kb_used_omap":0,"kb_used_meta":26112,"kb_avail":93661108,"statfs":{"total":95995035648,"available":95908974592,"internally_reserved":0,"allocated":59314176,"data_stored":68675926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26738688},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:57:04 2021","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.69899999999999995,"15min":0.64200000000000002},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.49,"5min":1.49,"15min":1.49},"last":1.133},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.751,"15min":0.70099999999999996},"min":{"1min":0.38900000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":2.3119999999999998,"5min":2.3119999999999998,"15min":2.3119999999999998},"last":0.54700000000000004}]},{"osd":1,"last update":"Wed Mar 24 21:57:04 2021","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.77000000000000002,"15min":0.71599999999999997},"min":{"1min":0.44900000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":2.0409999999999999,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":1.2010000000000001},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.745},"min":{"1min":0.41599999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.6479999999999999,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":1.2609999999999999}]},{"osd":3,"last update":"Wed Mar 24 21:56:21 2021","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.88300000000000001,"15min":0.83699999999999997},"min":{"1min":0.56899999999999995,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.7829999999999999,"5min":1.9890000000000001,"15min":1.9890000000000001},"last":1.4870000000000001},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.92000000000000004,"15min":0.89400000000000002},"min":{"1min":0.47399999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.573,"5min":2.0950000000000002,"15min":2.0950000000000002},"last":1.468}]},{"osd":4,"last update":"Wed Mar 24 21:56:46 2021","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.0820000000000001,"15min":1.05},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.8520000000000001,"5min":3.3380000000000001,"15min":3.3380000000000001},"last":1.0920000000000001},{"interface":"front","average":{"1min":1.095,"5min":1.089,"15min":1.0529999999999999},"min":{"1min":0.63200000000000001,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":2.0920000000000001,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":0.76300000000000001}]},{"osd":5,"last update":"Wed Mar 24 21:57:04 2021","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":1.0409999999999999,"15min":1.0349999999999999},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.5609999999999999,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.96099999999999997},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.1599999999999999,"15min":1.1539999999999999},"min":{"1min":0.67500000000000004,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.913,"5min":3.1509999999999998,"15min":3.1509999999999998},"last":0.94099999999999995}]},{"osd":6,"last update":"Wed Mar 24 21:56:21 2021","interfaces":[{"interface":"back","average":{"1min":1.133,"5min":1.173,"15min":1.2350000000000001},"min":{"1min":0.74399999999999999,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":2.149,"5min":2.2810000000000001,"15min":2.2810000000000001},"last":0.82299999999999995},{"interface":"front","average":{"1min":1.069,"5min":1.1259999999999999,"15min":1.153},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.5309999999999999,"5min":2.016,"15min":2.016},"last":1.0209999999999999}]},{"osd":7,"last update":"Wed Mar 24 21:56:50 2021","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.212,"15min":1.226},"min":{"1min":0.73699999999999999,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.714,"5min":4.4880000000000004,"15min":4.4880000000000004},"last":1.077},{"interface":"front","average":{"1min":1.226,"5min":1.228,"15min":1.2430000000000001},"min":{"1min":0.82699999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.9770000000000001,"5min":3.5419999999999998,"15min":3.5419999999999998},"last":1.0429999999999999}]}]},{"osd":3,"up_from":26,"seq":111669149784,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":90484,"kb_used_data":63788,"kb_used_omap":0,"kb_used_meta":26688,"kb_avail":93654668,"statfs":{"total":95995035648,"available":95902380032,"internally_reserved":0,"allocated":65318912,"data_stored":79647027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27328512},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:56:25 2021","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.871,"15min":0.71399999999999997},"min":{"1min":0.36599999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.038,"5min":23.846,"15min":23.846},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.95399999999999996,"15min":0.79500000000000004},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.002,"5min":23.734999999999999,"15min":23.734999999999999},"last":0.72899999999999998}]},{"osd":1,"last update":"Wed Mar 24 21:56:25 2021","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.879,"15min":0.78300000000000003},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.177,"5min":14.324,"15min":14.324},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.87,"15min":0.79300000000000004},"min":{"1min":0.47599999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.1499999999999999,"5min":12.631,"15min":12.631},"last":0.77900000000000003}]},{"osd":2,"last update":"Wed Mar 24 21:56:25 2021","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.92900000000000005,"15min":0.83099999999999996},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.3100000000000001,"5min":12.510999999999999,"15min":12.510999999999999},"last":1.0169999999999999},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.93600000000000005,"15min":0.81799999999999995},"min":{"1min":0.39000000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.345,"5min":11.317,"15min":11.317},"last":0.88400000000000001}]},{"osd":4,"last update":"Wed Mar 24 21:56:46 2021","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.018,"15min":0.97499999999999998},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.696,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.95899999999999996,"5min":1.155,"15min":1.0069999999999999},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.377,"5min":14.291,"15min":14.291},"last":0.96499999999999997}]},{"osd":5,"last update":"Wed Mar 24 21:57:00 2021","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":1.1799999999999999,"15min":1.1000000000000001},"min":{"1min":0.62,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.405,"5min":12.723000000000001,"15min":12.723000000000001},"last":1.105},{"interface":"front","average":{"1min":1.0880000000000001,"5min":1.0900000000000001,"15min":1.0940000000000001},"min":{"1min":0.64700000000000002,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.6100000000000001,"5min":2.375,"15min":2.375},"last":1.121}]},{"osd":6,"last update":"Wed Mar 24 21:56:25 2021","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.232,"15min":1.1639999999999999},"min":{"1min":0.50800000000000001,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.524,"5min":14.255000000000001,"15min":14.255000000000001},"last":1.228},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.258,"15min":1.1879999999999999},"min":{"1min":0.74099999999999999,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.446,"5min":12.666,"15min":12.666},"last":1.3420000000000001}]},{"osd":7,"last update":"Wed Mar 24 21:56:52 2021","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.1890000000000001,"15min":1.1140000000000001},"min":{"1min":0.61899999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.78,"5min":2.3479999999999999,"15min":2.3479999999999999},"last":1.2749999999999999},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.3280000000000001,"15min":1.216},"min":{"1min":0.49099999999999999,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.5029999999999999,"5min":14.727,"15min":14.727},"last":1.179}]}]},{"osd":4,"up_from":32,"seq":137438953554,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":72940,"kb_used_data":47844,"kb_used_omap":0,"kb_used_meta":25088,"kb_avail":93672212,"statfs":{"total":95995035648,"available":95920345088,"internally_reserved":0,"allocated":48992256,"data_stored":54781361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":25690112},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:56:42 2021","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.84199999999999997,"15min":0.83999999999999997},"min":{"1min":0.59999999999999998,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.149,"5min":1.268,"15min":1.268},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.88,"5min":0.99099999999999999,"15min":0.89500000000000002},"min":{"1min":0.62,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.4590000000000001,"5min":6.3929999999999998,"15min":6.3929999999999998},"last":0.94199999999999995}]},{"osd":1,"last update":"Wed Mar 24 21:56:42 2021","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":0.93100000000000005,"15min":0.92200000000000004},"min":{"1min":0.53300000000000003,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.2849999999999999,"5min":1.3120000000000001,"15min":1.4570000000000001},"last":1.034},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.92000000000000004,"15min":0.86399999999999999},"min":{"1min":0.67400000000000004,"5min":0.436,"15min":0.436},"max":{"1min":1.391,"5min":1.873,"15min":1.873},"last":0.877}]},{"osd":2,"last update":"Wed Mar 24 21:56:42 2021","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":0.95799999999999996,"15min":0.91600000000000004},"min":{"1min":0.60099999999999998,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.1719999999999999,"5min":4.4379999999999997,"15min":4.4379999999999997},"last":1.1870000000000001},{"interface":"front","average":{"1min":1.069,"5min":1.0029999999999999,"15min":0.93700000000000006},"min":{"1min":0.67000000000000004,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.502,"5min":2.738,"15min":2.738},"last":1.155}]},{"osd":3,"last update":"Wed Mar 24 21:56:42 2021","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":0.98699999999999999,"15min":0.95299999999999996},"min":{"1min":0.68100000000000005,"5min":0.439,"15min":0.439},"max":{"1min":1.518,"5min":3.2000000000000002,"15min":3.2000000000000002},"last":1.236},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1,"15min":0.96199999999999997},"min":{"1min":0.77200000000000002,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.456,"5min":1.474,"15min":1.474},"last":0.92800000000000005}]},{"osd":5,"last update":"Wed Mar 24 21:57:05 2021","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":0.95999999999999996,"15min":0.95399999999999996},"min":{"1min":0.73899999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.2350000000000001,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":1.139,"5min":1.107,"15min":1.0269999999999999},"min":{"1min":0.83499999999999996,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.405,"5min":2.0760000000000001,"15min":2.0760000000000001},"last":1.0860000000000001}]},{"osd":6,"last update":"Wed Mar 24 21:56:22 2021","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.0589999999999999,"15min":1.0629999999999999},"min":{"1min":0.49099999999999999,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.502,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":1.3280000000000001},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.0720000000000001,"15min":1.0680000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.6020000000000001,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":0.98499999999999999}]},{"osd":7,"last update":"Wed Mar 24 21:56:48 2021","interfaces":[{"interface":"back","average":{"1min":1.2230000000000001,"5min":1.145,"15min":1.179},"min":{"1min":0.92700000000000005,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.6830000000000001,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":1.123},{"interface":"front","average":{"1min":1.272,"5min":1.1819999999999999,"15min":1.198},"min":{"1min":1.0349999999999999,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.635,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":1.2909999999999999}]}]},{"osd":5,"up_from":39,"seq":167503724623,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":86244,"kb_used_data":66396,"kb_used_omap":0,"kb_used_meta":19840,"kb_avail":93658908,"statfs":{"total":95995035648,"available":95906721792,"internally_reserved":0,"allocated":67989504,"data_stored":74558702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20316160},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:57:05 2021","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.84199999999999997,"15min":0.873},"min":{"1min":0.58099999999999996,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.0860000000000001,"5min":2.29,"15min":2.29},"last":0.372},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.89700000000000002,"15min":0.88800000000000001},"min":{"1min":0.51400000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.1870000000000001,"5min":3.1779999999999999,"15min":3.1779999999999999},"last":0.50800000000000001}]},{"osd":1,"last update":"Wed Mar 24 21:57:05 2021","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.89800000000000002,"15min":0.91400000000000003},"min":{"1min":0.49299999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.26,"5min":2.8690000000000002,"15min":2.8690000000000002},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.91800000000000004,"15min":0.92800000000000005},"min":{"1min":0.56000000000000005,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.2749999999999999,"5min":2.0030000000000001,"15min":2.0030000000000001},"last":0.58799999999999997}]},{"osd":2,"last update":"Wed Mar 24 21:57:05 2021","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":0.96899999999999997,"15min":0.92500000000000004},"min":{"1min":0.60199999999999998,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.1160000000000001,"5min":6.5199999999999996,"15min":6.5199999999999996},"last":0.53600000000000003},{"interface":"front","average":{"1min":1.0720000000000001,"5min":0.99199999999999999,"15min":0.92200000000000004},"min":{"1min":0.55800000000000005,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":5.5010000000000003,"5min":5.8479999999999999,"15min":5.8479999999999999},"last":0.65500000000000003}]},{"osd":3,"last update":"Wed Mar 24 21:57:05 2021","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.96199999999999997,"15min":0.94199999999999995},"min":{"1min":0.625,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.3220000000000001,"5min":3.1259999999999999,"15min":3.1259999999999999},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.999,"15min":0.95899999999999996},"min":{"1min":0.54000000000000004,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.2490000000000001,"5min":2.6850000000000001,"15min":2.6850000000000001},"last":0.61299999999999999}]},{"osd":4,"last update":"Wed Mar 24 21:57:05 2021","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.98699999999999999,"15min":0.879},"min":{"1min":0.44900000000000001,"5min":0.376,"15min":0.376},"max":{"1min":1.2410000000000001,"5min":2.2959999999999998,"15min":2.2959999999999998},"last":0.63400000000000001},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.069,"15min":0.94899999999999995},"min":{"1min":0.56100000000000005,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.389,"5min":2.629,"15min":2.629},"last":0.74199999999999999}]},{"osd":6,"last update":"Wed Mar 24 21:56:23 2021","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.024,"15min":0.91400000000000003},"min":{"1min":0.77000000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":2.3879999999999999,"5min":2.3879999999999999,"15min":2.3879999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":1.173,"5min":1.091,"15min":0.999},"min":{"1min":0.71399999999999997,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":2.4660000000000002,"5min":2.4660000000000002,"15min":2.4660000000000002},"last":0.55900000000000005}]},{"osd":7,"last update":"Wed Mar 24 21:56:51 2021","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.145,"15min":1.077},"min":{"1min":0.64700000000000002,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.335,"5min":2.548,"15min":2.548},"last":0.70199999999999996},{"interface":"front","average":{"1min":1.153,"5min":1.145,"15min":1.0309999999999999},"min":{"1min":0.66700000000000004,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.5389999999999999,"5min":2.9889999999999999,"15min":2.9889999999999999},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":14134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":7234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":13448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":352256,"data_stored":12784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":360448,"data_stored":13024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":9948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62939136,"data_stored":75765674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7032832,"data_stored":6977532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57049088,"data_stored":67505397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":62840832,"data_stored":78449504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":46542848,"data_stored":53591499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":68145152,"data_stored":78594862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":63717376,"data_stored":72519363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-03-24T21:57:17.259 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2021-03-24T21:57:17.259 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd pool set unique_pool_0 pg_num 36 2021-03-24T21:57:17.321 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:17.373 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:17.376 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:17 smithi125 conmon[24927]: cluster 2021-03-24T21:57:15.687154+0000 osd.4 (osd.4) 70 : cluster [DBG] 6.1a scrub ok 2021-03-24T21:57:17.376 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:17 smithi125 conmon[24927]: cluster 2021-03-24T21:57:16.076973+0000 osd.5 (osd.5) 70 : cluster [DBG] 6.19 scrub ok 2021-03-24T21:57:17.377 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:17 smithi125 conmon[24927]: cluster 2021-03-24T21:57:16.270869+0000 osd.0 (osd.0) 70 : cluster [DBG] 3.10 scrub ok 2021-03-24T21:57:17.377 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:17 smithi125 conmon[19936]: cluster 2021-03-24T21:57:15.687154+0000 osd.4 ( 2021-03-24T21:57:17.377 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:17 smithi125 conmon[19936]: osd.4) 70 : cluster [DBG] 6.1a scrub ok 2021-03-24T21:57:17.377 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:17 smithi125 conmon[19936]: cluster 2021-03-24T21:57:16.076973+0000 osd.5 (osd.5) 70 : cluster [DBG] 6.19 scrub ok 2021-03-24T21:57:17.378 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:17 smithi125 conmon[19936]: cluster 2021-03-24T21:57:16.270869+0000 osd.0 (osd.0) 70 2021-03-24T21:57:17.378 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:17 smithi125 conmon[19936]: : cluster [DBG] 3.10 scrub ok 2021-03-24T21:57:17.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:17.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:57:17.435 INFO:tasks.rados.rados.0.smithi125.stdout:868: finishing write tid 2 to smithi12556470-25 2021-03-24T21:57:17.435 INFO:tasks.rados.rados.0.smithi125.stdout:868: finishing write tid 3 to smithi12556470-25 2021-03-24T21:57:17.436 INFO:tasks.rados.rados.0.smithi125.stdout:868: finishing write tid 4 to smithi12556470-25 2021-03-24T21:57:17.436 INFO:tasks.rados.rados.0.smithi125.stdout:868: finishing write tid 5 to smithi12556470-25 2021-03-24T21:57:17.436 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 25 v 180 (ObjNum 324 snap 82 seq_num 324) dirty exists 2021-03-24T21:57:17.436 INFO:tasks.rados.rados.0.smithi125.stdout:868: left oid 25 (ObjNum 324 snap 82 seq_num 324) 2021-03-24T21:57:17.436 INFO:tasks.rados.rados.0.smithi125.stdout:869: finishing write tid 1 to smithi12556470-2 2021-03-24T21:57:17.437 INFO:tasks.rados.rados.0.smithi125.stdout:869: finishing write tid 2 to smithi12556470-2 2021-03-24T21:57:17.437 INFO:tasks.rados.rados.0.smithi125.stdout:869: finishing write tid 3 to smithi12556470-2 2021-03-24T21:57:17.437 INFO:tasks.rados.rados.0.smithi125.stdout:869: finishing write tid 4 to smithi12556470-2 2021-03-24T21:57:17.437 INFO:tasks.rados.rados.0.smithi125.stdout:869: finishing write tid 5 to smithi12556470-2 2021-03-24T21:57:17.437 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 2 v 203 (ObjNum 325 snap 82 seq_num 325) dirty exists 2021-03-24T21:57:17.438 INFO:tasks.rados.rados.0.smithi125.stdout:869: left oid 2 (ObjNum 325 snap 82 seq_num 325) 2021-03-24T21:57:17.438 INFO:tasks.rados.rados.0.smithi125.stdout:871: finishing copy_from to smithi12556470-19 2021-03-24T21:57:17.438 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 19 v 203 (ObjNum 280 snap 70 seq_num 280) dirty exists 2021-03-24T21:57:17.438 INFO:tasks.rados.rados.0.smithi125.stdout:871: finishing copy_from racing read to smithi12556470-19 2021-03-24T21:57:17.439 INFO:tasks.rados.rados.0.smithi125.stdout:870: finishing copy_from to smithi12556470-36 2021-03-24T21:57:17.439 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 36 v 198 (ObjNum 150 snap 30 seq_num 150) dirty exists 2021-03-24T21:57:17.439 INFO:tasks.rados.rados.0.smithi125.stdout:870: finishing copy_from racing read to smithi12556470-36 2021-03-24T21:57:17.439 INFO:tasks.rados.rados.0.smithi125.stdout:875: finishing rollback tid 1 to smithi12556470-5 2021-03-24T21:57:17.439 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 5 v 0 (ObjNum 7 snap 5 seq_num 2) dirty dne 2021-03-24T21:57:17.440 INFO:tasks.rados.rados.0.smithi125.stdout:873: expect (ObjNum 203 snap 43 seq_num 203) 2021-03-24T21:57:17.480 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:17.532 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:17.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:17.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:57:17.611 INFO:tasks.rados.rados.0.smithi125.stdout:874: finishing copy_from to smithi12556470-22 2021-03-24T21:57:17.611 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 22 v 241 (ObjNum 251 snap 59 seq_num 251) dirty exists 2021-03-24T21:57:17.611 INFO:tasks.rados.rados.0.smithi125.stdout:874: finishing copy_from racing read to smithi12556470-22 2021-03-24T21:57:17.612 INFO:tasks.rados.rados.0.smithi125.stdout:876: expect (ObjNum 303 snap 74 seq_num 303) 2021-03-24T21:57:17.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:17 smithi142 conmon[20125]: cluster 2021-03-24T21:57 2021-03-24T21:57:17.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:17 smithi142 conmon[20125]: :15.687154+0000 osd.4 (osd.4) 70 : cluster [DBG] 6.1a scrub ok 2021-03-24T21:57:17.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:17 smithi142 conmon[20125]: cluster 2021-03-24T21:57:17.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:17 smithi142 conmon[20125]: 2021-03-24T21: 2021-03-24T21:57:17.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:17 smithi142 conmon[20125]: 57:16.076973+0000 osd.5 ( 2021-03-24T21:57:17.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:17 smithi142 conmon[20125]: osd.5) 70 : cluster [DBG] 6.19 scrub ok 2021-03-24T21:57:17.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:17 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:57:17.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:17 smithi142 conmon[20125]: T21:57:16.270869+0000 osd.0 (osd.0) 70 : 2021-03-24T21:57:17.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:17 smithi142 conmon[20125]: cluster [DBG] 3.10 scrub ok 2021-03-24T21:57:17.638 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:17.683 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:17.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:17.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:57:17.792 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:17.849 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:17.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:17.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:57:17.953 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:18.009 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:18.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:18.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:57:18.116 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:18.171 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:18.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:18.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:57:18.338 INFO:tasks.rados.rados.0.smithi125.stdout:866: done (9 left) 2021-03-24T21:57:18.338 INFO:tasks.rados.rados.0.smithi125.stdout:868: done (8 left) 2021-03-24T21:57:18.338 INFO:tasks.rados.rados.0.smithi125.stdout:869: done (7 left) 2021-03-24T21:57:18.338 INFO:tasks.rados.rados.0.smithi125.stdout:870: done (6 left) 2021-03-24T21:57:18.339 INFO:tasks.rados.rados.0.smithi125.stdout:871: done (5 left) 2021-03-24T21:57:18.339 INFO:tasks.rados.rados.0.smithi125.stdout:873: done (4 left) 2021-03-24T21:57:18.339 INFO:tasks.rados.rados.0.smithi125.stdout:874: done (3 left) 2021-03-24T21:57:18.339 INFO:tasks.rados.rados.0.smithi125.stdout:875: done (2 left) 2021-03-24T21:57:18.339 INFO:tasks.rados.rados.0.smithi125.stdout:876: done (1 left) 2021-03-24T21:57:18.340 INFO:tasks.rados.rados.0.smithi125.stdout:877: done (0 left) 2021-03-24T21:57:18.340 INFO:tasks.rados.rados.0.smithi125.stdout:878: rollback oid 15 current snap is 83 2021-03-24T21:57:18.340 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 15 to 81 2021-03-24T21:57:18.340 INFO:tasks.rados.rados.0.smithi125.stdout:879: delete oid 24 current snap is 83 2021-03-24T21:57:18.341 INFO:tasks.rados.rados.0.smithi125.stdout:878: finishing rollback tid 0 to smithi12556470-15 2021-03-24T21:57:18.341 INFO:tasks.rados.rados.0.smithi125.stdout:878: finishing rollback tid 1 to smithi12556470-15 2021-03-24T21:57:18.341 INFO:tasks.rados.rados.0.smithi125.stdout:878: finishing rollback tid 2 to smithi12556470-15 2021-03-24T21:57:18.341 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 15 v 215 (ObjNum 83 snap 10 seq_num 83) dirty exists 2021-03-24T21:57:18.341 INFO:tasks.rados.rados.0.smithi125.stdout:878: done (1 left) 2021-03-24T21:57:18.342 INFO:tasks.rados.rados.0.smithi125.stdout:879: done (0 left) 2021-03-24T21:57:18.342 INFO:tasks.rados.rados.0.smithi125.stdout:880: write oid 11 current snap is 83 2021-03-24T21:57:18.342 INFO:tasks.rados.rados.0.smithi125.stdout:880: seq_num 327 ranges {695889=409602,1618553=712763,2996848=499041} 2021-03-24T21:57:18.342 INFO:tasks.rados.rados.0.smithi125.stdout:880: writing smithi12556470-11 from 695889 to 1105491 tid 1 2021-03-24T21:57:18.342 INFO:tasks.rados.rados.0.smithi125.stdout:880: writing smithi12556470-11 from 1618553 to 2331316 tid 2 2021-03-24T21:57:18.343 INFO:tasks.rados.rados.0.smithi125.stdout:880: writing smithi12556470-11 from 2996848 to 3495889 tid 3 2021-03-24T21:57:18.343 INFO:tasks.rados.rados.0.smithi125.stdout:881: read oid 44 snap -1 2021-03-24T21:57:18.343 INFO:tasks.rados.rados.0.smithi125.stdout:881: expect (ObjNum 309 snap 76 seq_num 309) 2021-03-24T21:57:18.343 INFO:tasks.rados.rados.0.smithi125.stdout:882: write (excl) oid 49 current snap is 83 2021-03-24T21:57:18.343 INFO:tasks.rados.rados.0.smithi125.stdout:880: finishing write tid 1 to smithi12556470-11 2021-03-24T21:57:18.344 INFO:tasks.rados.rados.0.smithi125.stdout:882: seq_num 328 ranges {712696=484851,1960682=572774,3307918=204778} 2021-03-24T21:57:18.344 INFO:tasks.rados.rados.0.smithi125.stdout:882: writing smithi12556470-49 from 712696 to 1197547 tid 1 2021-03-24T21:57:18.344 INFO:tasks.rados.rados.0.smithi125.stdout:882: writing smithi12556470-49 from 1960682 to 2533456 tid 2 2021-03-24T21:57:18.344 INFO:tasks.rados.rados.0.smithi125.stdout:882: writing smithi12556470-49 from 3307918 to 3512696 tid 3 2021-03-24T21:57:18.344 INFO:tasks.rados.rados.0.smithi125.stdout:883: snap_remove snap 78 2021-03-24T21:57:18.437 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:18.449 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:18 smithi142 conmon[20125]: cluster 2021-03-24T21:57:16.768917+0000 mgr.y (mgr.14144) 400 2021-03-24T21:57:18.449 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:18 smithi142 conmon[20125]: : cluster [DBG] pgmap v562: 158 pgs: 158 active+clean; 143 MiB data, 499 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:57:18.449 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:18 smithi142 conmon[20125]: cluster 2021-03-24T21:57:17.230007+0000 mon.a (mon 2021-03-24T21:57:18.449 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:18 smithi142 conmon[20125]: .0) 913 : cluster [DBG] osdmap e267: 8 total, 8 up, 6 in 2021-03-24T21:57:18.489 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:18.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:18.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:18.526 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:18 smithi125 conmon[24927]: cluster 2021-03-24T21:57 2021-03-24T21:57:18.527 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:18 smithi125 conmon[24927]: :16.768917+0000 mgr.y (mgr.14144) 400 : cluster [DBG] pgmap v562: 158 pgs: 158 active+clean; 143 MiB data, 499 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:57:18.527 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:18 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:57:18.527 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:18 smithi125 conmon[24927]: 21:57:17.230007+0000 mon.a ( 2021-03-24T21:57:18.527 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:18 smithi125 conmon[24927]: mon.0) 913 : cluster [DBG] 2021-03-24T21:57:18.527 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:18 smithi125 conmon[24927]: osdmap e267: 8 total, 8 up, 6 in 2021-03-24T21:57:18.528 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:18 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:57:18.528 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:18 smithi125 conmon[19936]: 57:16.768917+0000 mgr.y (mgr.14144) 400 : cluster [DBG] pgmap v562: 158 pgs: 158 active+clean; 143 MiB data, 499 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:57:18.528 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:18 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:57:18.529 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:18 smithi125 conmon[19936]: 03-24T21:57:17. 2021-03-24T21:57:18.529 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:18 smithi125 conmon[19936]: 230007+0000 mon.a (mon.0) 913 : 2021-03-24T21:57:18.529 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:18 smithi125 conmon[19936]: cluster [DBG] osdmap e267: 8 total, 8 up, 6 in 2021-03-24T21:57:18.596 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:18.647 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:18.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:18.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:57:18.753 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:18.811 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:18.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:18.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:57:18.917 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:18.976 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:18.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:18.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:57:19.080 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:19.140 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:19.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:19.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:57:19.182 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-24T21:57:19.247 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:19.249 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:57:19.272 INFO:tasks.rados.rados.0.smithi125.stdout:883: done (3 left) 2021-03-24T21:57:19.273 INFO:tasks.rados.rados.0.smithi125.stdout:884: read oid 18 snap 77 2021-03-24T21:57:19.273 INFO:tasks.rados.rados.0.smithi125.stdout:884: expect (ObjNum 236 snap 53 seq_num 236) 2021-03-24T21:57:19.273 INFO:tasks.rados.rados.0.smithi125.stdout:885: read oid 2 snap -1 2021-03-24T21:57:19.273 INFO:tasks.rados.rados.0.smithi125.stdout:885: expect (ObjNum 325 snap 82 seq_num 325) 2021-03-24T21:57:19.273 INFO:tasks.rados.rados.0.smithi125.stdout:886: delete oid 21 current snap is 83 2021-03-24T21:57:19.274 INFO:tasks.rados.rados.0.smithi125.stdout:880: finishing write tid 2 to smithi12556470-11 2021-03-24T21:57:19.274 INFO:tasks.rados.rados.0.smithi125.stdout:880: finishing write tid 3 to smithi12556470-11 2021-03-24T21:57:19.274 INFO:tasks.rados.rados.0.smithi125.stdout:880: finishing write tid 4 to smithi12556470-11 2021-03-24T21:57:19.274 INFO:tasks.rados.rados.0.smithi125.stdout:880: finishing write tid 5 to smithi12556470-11 2021-03-24T21:57:19.274 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 11 v 249 (ObjNum 327 snap 83 seq_num 327) dirty exists 2021-03-24T21:57:19.275 INFO:tasks.rados.rados.0.smithi125.stdout:880: left oid 11 (ObjNum 327 snap 83 seq_num 327) 2021-03-24T21:57:19.275 INFO:tasks.rados.rados.0.smithi125.stdout:881: expect (ObjNum 309 snap 76 seq_num 309) 2021-03-24T21:57:19.325 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:19.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:19.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:57:19.393 INFO:tasks.rados.rados.0.smithi125.stdout:882: finishing write tid 1 to smithi12556470-49 2021-03-24T21:57:19.393 INFO:tasks.rados.rados.0.smithi125.stdout:882: finishing write tid 2 to smithi12556470-49 2021-03-24T21:57:19.393 INFO:tasks.rados.rados.0.smithi125.stdout:882: finishing write tid 3 to smithi12556470-49 2021-03-24T21:57:19.393 INFO:tasks.rados.rados.0.smithi125.stdout:882: finishing write tid 4 to smithi12556470-49 2021-03-24T21:57:19.394 INFO:tasks.rados.rados.0.smithi125.stdout:882: finishing write tid 5 to smithi12556470-49 2021-03-24T21:57:19.394 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 49 v 285 (ObjNum 328 snap 83 seq_num 328) dirty exists 2021-03-24T21:57:19.394 INFO:tasks.rados.rados.0.smithi125.stdout:882: left oid 49 (ObjNum 328 snap 83 seq_num 328) 2021-03-24T21:57:19.394 INFO:tasks.rados.rados.0.smithi125.stdout:880: done (5 left) 2021-03-24T21:57:19.394 INFO:tasks.rados.rados.0.smithi125.stdout:881: done (4 left) 2021-03-24T21:57:19.395 INFO:tasks.rados.rados.0.smithi125.stdout:882: done (3 left) 2021-03-24T21:57:19.395 INFO:tasks.rados.rados.0.smithi125.stdout:886: done (2 left) 2021-03-24T21:57:19.395 INFO:tasks.rados.rados.0.smithi125.stdout:887: copy_from oid 33 from oid 20 current snap is 83 2021-03-24T21:57:19.395 INFO:tasks.rados.rados.0.smithi125.stdout:884: expect (ObjNum 236 snap 53 seq_num 236) 2021-03-24T21:57:19.431 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:19.491 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:19.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:19.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:19.565 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:57:19.597 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:19.609 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[19936]: cluster 2021-03-24T21:57:18.199911+0000 2021-03-24T21:57:19.609 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[19936]: osd.7 (osd.7) 61 : cluster [DBG] 5.4 scrub ok 2021-03-24T21:57:19.609 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[19936]: cluster 2021-03-24T21:57:18.238882+0000 mon.a (mon.0 2021-03-24T21:57:19.609 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[19936]: ) 914 : cluster [DBG] osdmap e268: 8 total, 8 up, 6 in 2021-03-24T21:57:19.609 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:57:19.610 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[19936]: 03-24T21:57:18.303671+0000 osd.6 (osd.6) 2021-03-24T21:57:19.610 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[19936]: 23 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:57:19.610 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:57:19.610 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[19936]: 21:57:18.303791+0000 osd.6 (osd. 2021-03-24T21:57:19.611 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[19936]: 6) 24 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:57:19.611 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:19 smithi142 conmon[20125]: cluster 2021-03-24T21:57:18.199911+0000 osd.7 (osd.7) 61 : cluster [DBG] 5.4 scrub ok 2021-03-24T21:57:19.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:19 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:57:19.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:19 smithi142 conmon[20125]: 24T21:57:18.238882+0000 mon.a (mon.0) 914 2021-03-24T21:57:19.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:19 smithi142 conmon[20125]: : cluster [DBG] osdmap e268: 8 total, 8 up, 6 in 2021-03-24T21:57:19.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:19 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:57:19.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:19 smithi142 conmon[20125]: -03-24T21:57:18.303671+0000 osd.6 ( 2021-03-24T21:57:19.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:19 smithi142 conmon[20125]: osd.6) 23 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:57:19.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:19 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:57:19.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:19 smithi142 conmon[20125]: -03-24T21:57:18.303791+0000 osd.6 ( 2021-03-24T21:57:19.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:19 smithi142 conmon[20125]: osd.6) 24 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:57:19.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: cluster 2021-03-24T21:57:19.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: 2021-03-24T21:57:19.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: 2021 2021-03-24T21:57:19.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: - 2021-03-24T21:57:19.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: 03 2021-03-24T21:57:19.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: - 2021-03-24T21:57:19.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: 24 2021-03-24T21:57:19.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: T 2021-03-24T21:57:19.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: 21 2021-03-24T21:57:19.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: : 2021-03-24T21:57:19.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: 57 2021-03-24T21:57:19.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: : 2021-03-24T21:57:19.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: 18 2021-03-24T21:57:19.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: . 2021-03-24T21:57:19.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: 199911 2021-03-24T21:57:19.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: +0000 2021-03-24T21:57:19.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: 2021-03-24T21:57:19.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: osd.7 2021-03-24T21:57:19.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: ( 2021-03-24T21:57:19.618 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: osd 2021-03-24T21:57:19.618 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: . 2021-03-24T21:57:19.618 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: 7 2021-03-24T21:57:19.618 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: ) 2021-03-24T21:57:19.619 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: 61 2021-03-24T21:57:19.619 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: : 2021-03-24T21:57:19.619 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: cluster 2021-03-24T21:57:19.619 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: 2021-03-24T21:57:19.619 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: [DBG] 2021-03-24T21:57:19.620 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: 2021-03-24T21:57:19.620 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: 5.4 scrub ok 2021-03-24T21:57:19.620 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: 2021-03-24T21:57:19.620 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: cluster 2021-03-24T21:57:19.620 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: 2021-03-24T21:57:19.621 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: 2021 2021-03-24T21:57:19.621 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: - 2021-03-24T21:57:19.621 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: 03 2021-03-24T21:57:19.621 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: - 2021-03-24T21:57:19.621 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: 24 2021-03-24T21:57:19.622 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: T 2021-03-24T21:57:19.622 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: 21 2021-03-24T21:57:19.622 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: : 2021-03-24T21:57:19.622 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: 57 2021-03-24T21:57:19.623 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: : 2021-03-24T21:57:19.623 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: 18 2021-03-24T21:57:19.623 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: . 2021-03-24T21:57:19.623 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: 238882 2021-03-24T21:57:19.623 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: +0000 2021-03-24T21:57:19.624 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: 2021-03-24T21:57:19.624 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: mon.a 2021-03-24T21:57:19.624 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: ( 2021-03-24T21:57:19.624 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: mon 2021-03-24T21:57:19.624 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: . 2021-03-24T21:57:19.625 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: 0 2021-03-24T21:57:19.625 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: ) 2021-03-24T21:57:19.625 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: 914 2021-03-24T21:57:19.625 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: : 2021-03-24T21:57:19.625 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: cluster 2021-03-24T21:57:19.626 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: 2021-03-24T21:57:19.626 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: [DBG] 2021-03-24T21:57:19.626 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: 2021-03-24T21:57:19.626 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: osdmap e268: 8 total, 8 up, 6 in 2021-03-24T21:57:19.627 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: 2021-03-24T21:57:19.627 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: cluster 2021-03-24T21:57:19.627 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: 2021-03-24T21:57:19.627 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: 2021 2021-03-24T21:57:19.627 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: - 2021-03-24T21:57:19.627 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: 03 2021-03-24T21:57:19.628 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: - 2021-03-24T21:57:19.628 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: 24T21:57:18.303671 2021-03-24T21:57:19.628 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: +0000 osd.6 (osd.6) 23 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:57:19.628 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: cluster 2021-03-24T21:57:19.628 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: 2021-03-24T21:57:18.303791+0000 osd.6 ( 2021-03-24T21:57:19.629 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:19 smithi125 conmon[24927]: osd.6) 24 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:57:19.641 INFO:tasks.rados.rados.0.smithi125.stdout:885: expect (ObjNum 325 snap 82 seq_num 325) 2021-03-24T21:57:19.653 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:19.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:19.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:57:19.758 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:19.805 INFO:tasks.rados.rados.0.smithi125.stdout:884: done (2 left) 2021-03-24T21:57:19.805 INFO:tasks.rados.rados.0.smithi125.stdout:885: done (1 left) 2021-03-24T21:57:19.805 INFO:tasks.rados.rados.0.smithi125.stdout:888: write (excl) oid 25 current snap is 83 2021-03-24T21:57:19.806 INFO:tasks.rados.rados.0.smithi125.stdout:888: seq_num 330 ranges {746310=635349,2077646=576443,3362764=183546} 2021-03-24T21:57:19.812 INFO:tasks.rados.rados.0.smithi125.stdout:888: writing smithi12556470-25 from 746310 to 1381659 tid 1 2021-03-24T21:57:19.815 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:19.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:19.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:57:19.819 INFO:tasks.rados.rados.0.smithi125.stdout:888: writing smithi12556470-25 from 2077646 to 2654089 tid 2 2021-03-24T21:57:19.825 INFO:tasks.rados.rados.0.smithi125.stdout:888: writing smithi12556470-25 from 3362764 to 3546310 tid 3 2021-03-24T21:57:19.825 INFO:tasks.rados.rados.0.smithi125.stdout:889: copy_from oid 48 from oid 27 current snap is 83 2021-03-24T21:57:19.825 INFO:tasks.rados.rados.0.smithi125.stdout:887: finishing copy_from to smithi12556470-33 2021-03-24T21:57:19.826 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 33 v 160 (ObjNum 298 snap 74 seq_num 298) dirty exists 2021-03-24T21:57:19.826 INFO:tasks.rados.rados.0.smithi125.stdout:887: finishing copy_from racing read to smithi12556470-33 2021-03-24T21:57:19.826 INFO:tasks.rados.rados.0.smithi125.stdout:888: finishing write tid 1 to smithi12556470-25 2021-03-24T21:57:19.826 INFO:tasks.rados.rados.0.smithi125.stdout:888: finishing write tid 2 to smithi12556470-25 2021-03-24T21:57:19.826 INFO:tasks.rados.rados.0.smithi125.stdout:887: done (2 left) 2021-03-24T21:57:19.827 INFO:tasks.rados.rados.0.smithi125.stdout:890: rollback oid 29 current snap is 83 2021-03-24T21:57:19.827 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 29 to 75 2021-03-24T21:57:19.827 INFO:tasks.rados.rados.0.smithi125.stdout:891: write oid 16 current snap is 83 2021-03-24T21:57:19.827 INFO:tasks.rados.rados.0.smithi125.stdout:891: seq_num 331 ranges {763117=710598,2136128=720101,3390187=172930} 2021-03-24T21:57:19.833 INFO:tasks.rados.rados.0.smithi125.stdout:891: writing smithi12556470-16 from 763117 to 1473715 tid 1 2021-03-24T21:57:19.841 INFO:tasks.rados.rados.0.smithi125.stdout:891: writing smithi12556470-16 from 2136128 to 2856229 tid 2 2021-03-24T21:57:19.846 INFO:tasks.rados.rados.0.smithi125.stdout:891: writing smithi12556470-16 from 3390187 to 3563117 tid 3 2021-03-24T21:57:19.846 INFO:tasks.rados.rados.0.smithi125.stdout:892: delete oid 22 current snap is 83 2021-03-24T21:57:19.846 INFO:tasks.rados.rados.0.smithi125.stdout:888: finishing write tid 3 to smithi12556470-25 2021-03-24T21:57:19.847 INFO:tasks.rados.rados.0.smithi125.stdout:888: finishing write tid 4 to smithi12556470-25 2021-03-24T21:57:19.847 INFO:tasks.rados.rados.0.smithi125.stdout:888: finishing write tid 5 to smithi12556470-25 2021-03-24T21:57:19.847 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 25 v 185 (ObjNum 330 snap 83 seq_num 330) dirty exists 2021-03-24T21:57:19.847 INFO:tasks.rados.rados.0.smithi125.stdout:888: left oid 25 (ObjNum 330 snap 83 seq_num 330) 2021-03-24T21:57:19.847 INFO:tasks.rados.rados.0.smithi125.stdout:890: finishing rollback tid 0 to smithi12556470-29 2021-03-24T21:57:19.848 INFO:tasks.rados.rados.0.smithi125.stdout:890: finishing rollback tid 1 to smithi12556470-29 2021-03-24T21:57:19.848 INFO:tasks.rados.rados.0.smithi125.stdout:890: finishing rollback tid 2 to smithi12556470-29 2021-03-24T21:57:19.848 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 29 v 140 (ObjNum 274 snap 68 seq_num 274) dirty exists 2021-03-24T21:57:19.848 INFO:tasks.rados.rados.0.smithi125.stdout:891: finishing write tid 1 to smithi12556470-16 2021-03-24T21:57:19.849 INFO:tasks.rados.rados.0.smithi125.stdout:889: finishing copy_from to smithi12556470-48 2021-03-24T21:57:19.850 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 48 v 164 (ObjNum 193 snap 42 seq_num 193) dirty exists 2021-03-24T21:57:19.850 INFO:tasks.rados.rados.0.smithi125.stdout:889: finishing copy_from racing read to smithi12556470-48 2021-03-24T21:57:19.850 INFO:tasks.rados.rados.0.smithi125.stdout:891: finishing write tid 2 to smithi12556470-16 2021-03-24T21:57:19.850 INFO:tasks.rados.rados.0.smithi125.stdout:891: finishing write tid 3 to smithi12556470-16 2021-03-24T21:57:19.850 INFO:tasks.rados.rados.0.smithi125.stdout:888: done (4 left) 2021-03-24T21:57:19.851 INFO:tasks.rados.rados.0.smithi125.stdout:889: done (3 left) 2021-03-24T21:57:19.851 INFO:tasks.rados.rados.0.smithi125.stdout:890: done (2 left) 2021-03-24T21:57:19.851 INFO:tasks.rados.rados.0.smithi125.stdout:892: done (1 left) 2021-03-24T21:57:19.851 INFO:tasks.rados.rados.0.smithi125.stdout:893: snap_remove snap 77 2021-03-24T21:57:19.920 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:19.975 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:19.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:19.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:57:20.062 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-24T21:57:20.063 INFO:teuthology.orchestra.run.smithi125.stdout: "size": 20, 2021-03-24T21:57:20.063 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 600, 2021-03-24T21:57:20.063 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [ 2021-03-24T21:57:20.063 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.064 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "MOSDPGPush(7.0 109/108 [PushOp(7:0ece3580:::smithi12556470-35:head, version: 107'3, data_included: [571438~623878,1806623~424698], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7:0ece3580:::smithi12556470-35:head@107'3, size: 2571438, copy_subset: [0~2571438], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2231321, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2021-03-24T21:57:20.064 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:38.142023+0000", 2021-03-24T21:57:20.064 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 161.91590072700001, 2021-03-24T21:57:20.064 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.013359005, 2021-03-24T21:57:20.065 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:57:20.065 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T21:57:20.065 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:57:20.065 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.065 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:57:20.066 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.142023+0000", 2021-03-24T21:57:20.066 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:57:20.066 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.066 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.066 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:57:20.067 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.142023+0000", 2021-03-24T21:57:20.067 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T21:57:20.067 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.067 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.067 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:57:20.068 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.142022+0000", 2021-03-24T21:57:20.068 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012155871 2021-03-24T21:57:20.068 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.068 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.068 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:57:20.069 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.154178+0000", 2021-03-24T21:57:20.069 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.9859e-05 2021-03-24T21:57:20.069 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.069 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.069 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:57:20.070 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.154208+0000", 2021-03-24T21:57:20.070 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.4072e-05 2021-03-24T21:57:20.070 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.070 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.071 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:57:20.071 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.154222+0000", 2021-03-24T21:57:20.071 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00023070499999999999 2021-03-24T21:57:20.071 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.071 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.072 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:20.072 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.154452+0000", 2021-03-24T21:57:20.072 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.3880000000000006e-06 2021-03-24T21:57:20.072 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.072 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.073 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:57:20.073 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.154462+0000", 2021-03-24T21:57:20.073 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00092047799999999996 2021-03-24T21:57:20.073 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.074 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.074 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:57:20.074 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:38.155382+0000", 2021-03-24T21:57:20.074 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.013359005 2021-03-24T21:57:20.074 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:20.075 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:57:20.075 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:20.075 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.075 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.076 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:277 7.8 7:19ff009c:::smithi12556470-42:head [copy-from ver 9] snapc 8=[8] ondisk+write+known_if_redirected e114)", 2021-03-24T21:57:20.076 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:43.202214+0000", 2021-03-24T21:57:20.076 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 156.85571002399999, 2021-03-24T21:57:20.076 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.015212942, 2021-03-24T21:57:20.076 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:57:20.077 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:57:20.077 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:57:20.077 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:57:20.077 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:57:20.077 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 277 2021-03-24T21:57:20.078 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.078 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:57:20.078 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.078 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:57:20.078 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.202214+0000", 2021-03-24T21:57:20.079 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:57:20.079 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.079 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.079 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:57:20.079 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.202214+0000", 2021-03-24T21:57:20.080 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999971 2021-03-24T21:57:20.080 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.080 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.080 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:57:20.080 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.202211+0000", 2021-03-24T21:57:20.081 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1984000000000001e-05 2021-03-24T21:57:20.081 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.081 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.081 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:57:20.081 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.202223+0000", 2021-03-24T21:57:20.082 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7306e-05 2021-03-24T21:57:20.082 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.082 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.082 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:57:20.082 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.202240+0000", 2021-03-24T21:57:20.083 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.6490000000000003e-06 2021-03-24T21:57:20.083 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.083 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.083 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:57:20.083 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.202250+0000", 2021-03-24T21:57:20.084 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.6559999999999999e-05 2021-03-24T21:57:20.084 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.084 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.084 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:20.084 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.202327+0000", 2021-03-24T21:57:20.085 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.1945000000000001e-05 2021-03-24T21:57:20.085 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.085 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.085 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:57:20.085 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.202419+0000", 2021-03-24T21:57:20.085 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0048950820000000003 2021-03-24T21:57:20.086 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.086 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.086 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 4,7", 2021-03-24T21:57:20.086 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.207314+0000", 2021-03-24T21:57:20.087 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0050236980000000001 2021-03-24T21:57:20.087 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.087 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.087 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:57:20.087 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.212337+0000", 2021-03-24T21:57:20.087 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0048611870000000003 2021-03-24T21:57:20.088 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.088 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.088 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:57:20.088 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.217199+0000", 2021-03-24T21:57:20.089 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1696000000000001e-05 2021-03-24T21:57:20.089 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.089 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.089 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:57:20.089 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.217250+0000", 2021-03-24T21:57:20.090 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.4931000000000004e-05 2021-03-24T21:57:20.090 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.090 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.090 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:57:20.090 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.217335+0000", 2021-03-24T21:57:20.091 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.1592e-05 2021-03-24T21:57:20.091 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.091 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.091 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:57:20.091 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.217427+0000", 2021-03-24T21:57:20.091 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.015212942 2021-03-24T21:57:20.092 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:20.092 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:57:20.092 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:20.092 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.092 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.093 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:278 7.8 7:19ff009c:::smithi12556470-42:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e114)", 2021-03-24T21:57:20.093 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:43.202267+0000", 2021-03-24T21:57:20.093 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 156.855656469, 2021-03-24T21:57:20.094 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.015574229, 2021-03-24T21:57:20.094 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:57:20.094 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T21:57:20.094 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:57:20.094 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:57:20.095 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:57:20.095 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 278 2021-03-24T21:57:20.095 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.095 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:57:20.096 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.096 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:57:20.096 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.202267+0000", 2021-03-24T21:57:20.096 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:57:20.096 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.097 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.097 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:57:20.097 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.202267+0000", 2021-03-24T21:57:20.097 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T21:57:20.097 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.098 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.098 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:57:20.098 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.202267+0000", 2021-03-24T21:57:20.098 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6749999999999999e-06 2021-03-24T21:57:20.098 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.098 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.099 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:57:20.099 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.202272+0000", 2021-03-24T21:57:20.099 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.8089999999999998e-06 2021-03-24T21:57:20.099 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.100 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.100 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:57:20.100 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.202278+0000", 2021-03-24T21:57:20.100 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.354e-06 2021-03-24T21:57:20.100 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.101 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.101 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:57:20.101 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.202282+0000", 2021-03-24T21:57:20.101 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00032993900000000001 2021-03-24T21:57:20.101 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.102 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.102 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:20.102 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.202611+0000", 2021-03-24T21:57:20.102 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.3274000000000004e-05 2021-03-24T21:57:20.102 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.103 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.103 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-24T21:57:20.103 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.202685+0000", 2021-03-24T21:57:20.103 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0085844809999999997 2021-03-24T21:57:20.103 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.104 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.104 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:20.104 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.211269+0000", 2021-03-24T21:57:20.104 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.8164000000000004e-05 2021-03-24T21:57:20.104 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.104 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.105 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T21:57:20.105 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.211367+0000", 2021-03-24T21:57:20.105 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0061402770000000004 2021-03-24T21:57:20.105 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.105 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.106 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:20.106 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.217508+0000", 2021-03-24T21:57:20.106 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00016125499999999999 2021-03-24T21:57:20.106 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.107 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.107 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:57:20.107 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.217669+0000", 2021-03-24T21:57:20.107 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00017273599999999999 2021-03-24T21:57:20.107 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.108 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.108 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:57:20.108 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.217842+0000", 2021-03-24T21:57:20.108 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.015574229 2021-03-24T21:57:20.108 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:20.109 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:57:20.109 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:20.109 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.109 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.109 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(osd.7.0:28 7.5 e123/99 7:ad88ec1b:::smithi12556470-47:d v 123'21, mlcod=120'18)", 2021-03-24T21:57:20.110 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:52.113893+0000", 2021-03-24T21:57:20.110 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 147.944030459, 2021-03-24T21:57:20.110 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012019541999999999, 2021-03-24T21:57:20.110 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:57:20.111 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:57:20.111 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:57:20.111 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.111 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:57:20.111 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.113893+0000", 2021-03-24T21:57:20.112 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:57:20.112 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.112 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.112 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:57:20.112 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.113893+0000", 2021-03-24T21:57:20.113 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T21:57:20.113 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.113 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.113 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:57:20.113 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.113892+0000", 2021-03-24T21:57:20.114 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3386e-05 2021-03-24T21:57:20.114 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.114 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.114 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:57:20.115 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.113906+0000", 2021-03-24T21:57:20.115 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0048278569999999996 2021-03-24T21:57:20.115 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.115 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.115 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:57:20.116 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.118734+0000", 2021-03-24T21:57:20.116 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.0820000000000005e-06 2021-03-24T21:57:20.116 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.116 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.116 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:57:20.117 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.118743+0000", 2021-03-24T21:57:20.117 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.7781000000000001e-05 2021-03-24T21:57:20.117 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.117 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.117 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:20.118 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.118800+0000", 2021-03-24T21:57:20.118 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.8793e-05 2021-03-24T21:57:20.118 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.118 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.118 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:57:20.119 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.118829+0000", 2021-03-24T21:57:20.119 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0070437140000000004 2021-03-24T21:57:20.119 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.119 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.119 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:57:20.120 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.125873+0000", 2021-03-24T21:57:20.120 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.0057e-05 2021-03-24T21:57:20.120 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.120 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.121 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:57:20.121 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.125913+0000", 2021-03-24T21:57:20.121 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012019541999999999 2021-03-24T21:57:20.121 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:20.121 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:57:20.122 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:20.122 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.122 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.122 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:645 7.c e141/101 7:305739dc:::smithi12556470-36:head v 141'47, mlcod=134'43)", 2021-03-24T21:57:20.122 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:55:10.123986+0000", 2021-03-24T21:57:20.123 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 129.93393753999999, 2021-03-24T21:57:20.123 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.025623269000000001, 2021-03-24T21:57:20.123 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:57:20.123 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:57:20.123 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:57:20.124 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.124 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:57:20.124 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.123986+0000", 2021-03-24T21:57:20.124 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:57:20.125 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.125 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.125 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:57:20.125 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.123986+0000", 2021-03-24T21:57:20.125 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T21:57:20.126 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.126 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.126 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:57:20.126 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.123985+0000", 2021-03-24T21:57:20.126 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2130000000000001e-05 2021-03-24T21:57:20.127 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.127 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.127 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:57:20.127 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.123997+0000", 2021-03-24T21:57:20.127 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.4334e-05 2021-03-24T21:57:20.128 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.128 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.128 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:57:20.128 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124031+0000", 2021-03-24T21:57:20.128 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.3420000000000002e-06 2021-03-24T21:57:20.129 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.129 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.129 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:57:20.129 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124039+0000", 2021-03-24T21:57:20.129 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.0111999999999997e-05 2021-03-24T21:57:20.130 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.130 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.130 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:20.130 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124089+0000", 2021-03-24T21:57:20.131 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.3839e-05 2021-03-24T21:57:20.131 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.131 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.131 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:57:20.131 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124113+0000", 2021-03-24T21:57:20.132 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.025454029999999999 2021-03-24T21:57:20.132 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.132 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.132 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:57:20.132 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.149567+0000", 2021-03-24T21:57:20.133 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.2976000000000003e-05 2021-03-24T21:57:20.133 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.133 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.133 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:57:20.133 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.149610+0000", 2021-03-24T21:57:20.134 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.025623269000000001 2021-03-24T21:57:20.134 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:20.134 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:57:20.134 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:20.135 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.135 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.135 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:646 7.c e141/101 7:305739dc:::smithi12556470-36:head v 141'48, mlcod=134'43)", 2021-03-24T21:57:20.135 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:55:10.124053+0000", 2021-03-24T21:57:20.135 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 129.93387102400001, 2021-03-24T21:57:20.136 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.027636099000000001, 2021-03-24T21:57:20.136 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:57:20.136 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:57:20.136 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:57:20.136 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.137 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:57:20.137 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124053+0000", 2021-03-24T21:57:20.137 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:57:20.137 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.138 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.138 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:57:20.138 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124053+0000", 2021-03-24T21:57:20.138 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T21:57:20.138 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.139 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.139 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:57:20.139 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124052+0000", 2021-03-24T21:57:20.139 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.2480000000000002e-06 2021-03-24T21:57:20.139 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.140 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.140 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:57:20.140 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124061+0000", 2021-03-24T21:57:20.140 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6640000000000003e-06 2021-03-24T21:57:20.141 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.141 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.141 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:57:20.141 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124067+0000", 2021-03-24T21:57:20.141 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9349999999999996e-06 2021-03-24T21:57:20.142 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.142 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.142 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:57:20.142 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124071+0000", 2021-03-24T21:57:20.143 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00051871899999999995 2021-03-24T21:57:20.143 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.143 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.143 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:20.143 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124590+0000", 2021-03-24T21:57:20.144 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2456999999999999e-05 2021-03-24T21:57:20.144 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.144 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.144 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:57:20.144 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124602+0000", 2021-03-24T21:57:20.145 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.027032664000000001 2021-03-24T21:57:20.145 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.145 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.145 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:57:20.146 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.151635+0000", 2021-03-24T21:57:20.146 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.4151999999999999e-05 2021-03-24T21:57:20.146 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.146 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.147 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:57:20.147 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.151689+0000", 2021-03-24T21:57:20.147 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.027636099000000001 2021-03-24T21:57:20.147 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:20.147 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:57:20.148 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:20.148 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.148 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.148 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:647 7.c e141/101 7:305739dc:::smithi12556470-36:head v 141'49, mlcod=134'43)", 2021-03-24T21:57:20.148 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:55:10.124445+0000", 2021-03-24T21:57:20.149 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 129.93347905900001, 2021-03-24T21:57:20.149 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.027276857000000002, 2021-03-24T21:57:20.149 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:57:20.149 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:57:20.149 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:57:20.150 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.150 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:57:20.150 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124445+0000", 2021-03-24T21:57:20.150 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:57:20.151 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.151 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.151 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:57:20.151 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124445+0000", 2021-03-24T21:57:20.151 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T21:57:20.152 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.152 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.152 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:57:20.152 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124444+0000", 2021-03-24T21:57:20.152 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.8339999999999999e-06 2021-03-24T21:57:20.152 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.153 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.153 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:57:20.153 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124451+0000", 2021-03-24T21:57:20.153 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.8640000000000002e-06 2021-03-24T21:57:20.153 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.154 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.154 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:57:20.154 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124460+0000", 2021-03-24T21:57:20.154 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5130000000000004e-06 2021-03-24T21:57:20.154 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.155 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.155 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:57:20.155 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124466+0000", 2021-03-24T21:57:20.155 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00030484700000000002 2021-03-24T21:57:20.155 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.156 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.156 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:20.156 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124771+0000", 2021-03-24T21:57:20.156 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.8258e-05 2021-03-24T21:57:20.156 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.157 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.157 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:57:20.157 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124789+0000", 2021-03-24T21:57:20.157 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.026903249000000001 2021-03-24T21:57:20.157 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.158 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.158 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:57:20.158 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.151692+0000", 2021-03-24T21:57:20.158 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.9533e-05 2021-03-24T21:57:20.158 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.159 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.159 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:57:20.159 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.151722+0000", 2021-03-24T21:57:20.159 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.027276857000000002 2021-03-24T21:57:20.160 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:20.160 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:57:20.160 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:20.160 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.160 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.161 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:1068 7.8 7:16b6db15:::smithi12556470-3:head [rollback 42] snapc 46=[42,3e,3d] ondisk+write+known_if_redirected e178)", 2021-03-24T21:57:20.161 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:55:47.151089+0000", 2021-03-24T21:57:20.161 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 92.906835229999999, 2021-03-24T21:57:20.161 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011797452999999999, 2021-03-24T21:57:20.161 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:57:20.162 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:57:20.162 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:57:20.162 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:57:20.162 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:57:20.162 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1068 2021-03-24T21:57:20.163 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.163 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:57:20.163 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.163 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:57:20.163 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.151089+0000", 2021-03-24T21:57:20.164 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:57:20.164 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.164 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.164 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:57:20.164 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.151089+0000", 2021-03-24T21:57:20.165 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T21:57:20.165 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.165 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.165 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:57:20.165 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.151087+0000", 2021-03-24T21:57:20.166 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.1070000000000001e-06 2021-03-24T21:57:20.166 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.166 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.166 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:57:20.166 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.151094+0000", 2021-03-24T21:57:20.167 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0976e-05 2021-03-24T21:57:20.167 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.167 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.167 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:57:20.167 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.151105+0000", 2021-03-24T21:57:20.168 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.4740000000000005e-06 2021-03-24T21:57:20.168 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.168 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.168 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:57:20.168 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.151113+0000", 2021-03-24T21:57:20.169 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010699799999999999 2021-03-24T21:57:20.169 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.169 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.169 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:20.169 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.151220+0000", 2021-03-24T21:57:20.170 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4570000000000002e-06 2021-03-24T21:57:20.170 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.170 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.170 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting_for_map not empty", 2021-03-24T21:57:20.170 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.151226+0000", 2021-03-24T21:57:20.171 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0068211319999999997 2021-03-24T21:57:20.171 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.171 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.171 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:20.171 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.158048+0000", 2021-03-24T21:57:20.172 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00011455000000000001 2021-03-24T21:57:20.172 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.172 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.172 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:57:20.172 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.158162+0000", 2021-03-24T21:57:20.172 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00023676400000000001 2021-03-24T21:57:20.173 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.173 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.173 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 4,7", 2021-03-24T21:57:20.173 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.158399+0000", 2021-03-24T21:57:20.173 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0021595009999999999 2021-03-24T21:57:20.174 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.174 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.174 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:57:20.174 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.160558+0000", 2021-03-24T21:57:20.174 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002051878 2021-03-24T21:57:20.175 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.175 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.175 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:57:20.175 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.162610+0000", 2021-03-24T21:57:20.175 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000118348 2021-03-24T21:57:20.176 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.176 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.176 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:57:20.176 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.162729+0000", 2021-03-24T21:57:20.176 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.5840000000000004e-05 2021-03-24T21:57:20.177 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.177 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.177 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:57:20.177 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.162824+0000", 2021-03-24T21:57:20.177 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.1639999999999999e-05 2021-03-24T21:57:20.178 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.178 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.178 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:57:20.178 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.162886+0000", 2021-03-24T21:57:20.179 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011797452999999999 2021-03-24T21:57:20.179 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:20.179 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:57:20.179 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:20.179 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.180 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.180 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:1069 7.8 7:16b6db15:::smithi12556470-3:head [append 0~0] snapc 46=[42,3e,3d] ondisk+write+known_if_redirected e178)", 2021-03-24T21:57:20.180 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:55:47.151122+0000", 2021-03-24T21:57:20.180 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 92.906801958000003, 2021-03-24T21:57:20.180 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011921111999999999, 2021-03-24T21:57:20.181 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:57:20.181 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:57:20.181 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:57:20.181 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:57:20.181 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:57:20.182 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1069 2021-03-24T21:57:20.182 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.182 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:57:20.182 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.182 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:57:20.183 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.151122+0000", 2021-03-24T21:57:20.183 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:57:20.183 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.183 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.183 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:57:20.184 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.151122+0000", 2021-03-24T21:57:20.184 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T21:57:20.184 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.184 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.184 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:57:20.185 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.151121+0000", 2021-03-24T21:57:20.185 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.4259999999999996e-06 2021-03-24T21:57:20.185 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.185 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.185 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:57:20.186 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.151126+0000", 2021-03-24T21:57:20.186 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.4299999999999995e-06 2021-03-24T21:57:20.186 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.186 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.187 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:57:20.187 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.151135+0000", 2021-03-24T21:57:20.187 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.8000000000000001e-06 2021-03-24T21:57:20.187 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.187 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.187 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:57:20.188 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.151142+0000", 2021-03-24T21:57:20.188 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000107705 2021-03-24T21:57:20.188 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.188 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.188 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:20.189 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.151249+0000", 2021-03-24T21:57:20.189 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.682e-06 2021-03-24T21:57:20.189 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.189 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.189 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting_for_map not empty", 2021-03-24T21:57:20.190 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.151255+0000", 2021-03-24T21:57:20.190 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0075323079999999997 2021-03-24T21:57:20.190 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.190 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.190 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:20.191 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.158787+0000", 2021-03-24T21:57:20.191 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2745999999999997e-05 2021-03-24T21:57:20.191 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.191 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.191 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:57:20.192 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.158870+0000", 2021-03-24T21:57:20.192 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00016508399999999999 2021-03-24T21:57:20.192 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.192 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.192 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 4,7", 2021-03-24T21:57:20.193 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.159035+0000", 2021-03-24T21:57:20.193 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0035892060000000002 2021-03-24T21:57:20.193 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.193 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.193 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:57:20.194 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.162624+0000", 2021-03-24T21:57:20.194 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.7304000000000005e-05 2021-03-24T21:57:20.194 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.194 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.195 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:57:20.195 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.162692+0000", 2021-03-24T21:57:20.195 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000247958 2021-03-24T21:57:20.195 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.195 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.196 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:57:20.196 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.162940+0000", 2021-03-24T21:57:20.196 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.7689999999999998e-05 2021-03-24T21:57:20.196 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.196 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.197 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:57:20.197 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.162997+0000", 2021-03-24T21:57:20.197 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.562e-05 2021-03-24T21:57:20.197 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.197 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.198 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:57:20.198 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.163043+0000", 2021-03-24T21:57:20.198 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011921111999999999 2021-03-24T21:57:20.198 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:20.198 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:57:20.199 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:20.199 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.199 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.199 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:1075 7.8 7:19ff009c:::smithi12556470-42:head [rollback 3d] snapc 46=[42,3e,3d] ondisk+write+known_if_redirected e178)", 2021-03-24T21:57:20.199 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:55:47.151173+0000", 2021-03-24T21:57:20.200 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 92.906750672000001, 2021-03-24T21:57:20.200 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.014149844, 2021-03-24T21:57:20.200 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:57:20.200 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:57:20.200 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:57:20.201 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:57:20.201 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:57:20.201 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1075 2021-03-24T21:57:20.201 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.201 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:57:20.202 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.202 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:57:20.202 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.151173+0000", 2021-03-24T21:57:20.202 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:57:20.202 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.203 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.203 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:57:20.203 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.151173+0000", 2021-03-24T21:57:20.203 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T21:57:20.203 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.204 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.204 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:57:20.204 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.151172+0000", 2021-03-24T21:57:20.204 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.7420000000000003e-06 2021-03-24T21:57:20.204 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.205 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.205 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:57:20.205 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.151178+0000", 2021-03-24T21:57:20.205 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.147e-05 2021-03-24T21:57:20.205 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.206 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.206 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:57:20.206 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.151189+0000", 2021-03-24T21:57:20.206 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.7680000000000008e-06 2021-03-24T21:57:20.207 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.207 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.207 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:57:20.207 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.151197+0000", 2021-03-24T21:57:20.208 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00021914499999999999 2021-03-24T21:57:20.208 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.208 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.208 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:20.208 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.151416+0000", 2021-03-24T21:57:20.209 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1694e-05 2021-03-24T21:57:20.209 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.209 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.209 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting_for_map not empty", 2021-03-24T21:57:20.209 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.151428+0000", 2021-03-24T21:57:20.210 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.007859497 2021-03-24T21:57:20.210 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.210 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.210 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:20.211 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.159287+0000", 2021-03-24T21:57:20.211 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.5264000000000001e-05 2021-03-24T21:57:20.211 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.211 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.211 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:57:20.212 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.159382+0000", 2021-03-24T21:57:20.212 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00020908999999999999 2021-03-24T21:57:20.212 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.212 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.212 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 4,7", 2021-03-24T21:57:20.213 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.159592+0000", 2021-03-24T21:57:20.213 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0030430000000000001 2021-03-24T21:57:20.213 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.213 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.214 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:57:20.214 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.162635+0000", 2021-03-24T21:57:20.214 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001256271 2021-03-24T21:57:20.214 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.214 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.215 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:57:20.215 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.163891+0000", 2021-03-24T21:57:20.215 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0010054389999999999 2021-03-24T21:57:20.215 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.216 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.216 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:57:20.216 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.164896+0000", 2021-03-24T21:57:20.216 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00028296000000000001 2021-03-24T21:57:20.216 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.217 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.217 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:57:20.217 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.165179+0000", 2021-03-24T21:57:20.217 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000143817 2021-03-24T21:57:20.217 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.218 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.218 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:57:20.218 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.165323+0000", 2021-03-24T21:57:20.218 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.014149844 2021-03-24T21:57:20.219 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:20.219 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:57:20.219 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:20.219 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.219 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.220 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:1076 7.8 7:19ff009c:::smithi12556470-42:head [append 0~0] snapc 46=[42,3e,3d] ondisk+write+known_if_redirected e178)", 2021-03-24T21:57:20.220 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:55:47.151307+0000", 2021-03-24T21:57:20.220 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 92.906616968999998, 2021-03-24T21:57:20.220 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.01424504, 2021-03-24T21:57:20.220 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:57:20.221 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:57:20.221 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:57:20.221 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:57:20.221 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:57:20.221 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1076 2021-03-24T21:57:20.222 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.222 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:57:20.222 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.222 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:57:20.222 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.151307+0000", 2021-03-24T21:57:20.223 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:57:20.223 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.223 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.223 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:57:20.223 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.151307+0000", 2021-03-24T21:57:20.224 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T21:57:20.224 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.224 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.224 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:57:20.224 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.151306+0000", 2021-03-24T21:57:20.225 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2810000000000008e-06 2021-03-24T21:57:20.225 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.225 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.225 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:57:20.225 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.151314+0000", 2021-03-24T21:57:20.226 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3985000000000001e-05 2021-03-24T21:57:20.226 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.226 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.226 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:57:20.226 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.151328+0000", 2021-03-24T21:57:20.227 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.5219999999999996e-06 2021-03-24T21:57:20.227 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.227 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.227 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:57:20.227 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.151337+0000", 2021-03-24T21:57:20.228 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000125938 2021-03-24T21:57:20.228 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.228 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.228 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:20.228 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.151463+0000", 2021-03-24T21:57:20.229 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0497e-05 2021-03-24T21:57:20.229 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.229 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.229 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting_for_map not empty", 2021-03-24T21:57:20.229 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.151474+0000", 2021-03-24T21:57:20.230 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0083861869999999998 2021-03-24T21:57:20.230 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.230 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.230 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:20.230 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.159860+0000", 2021-03-24T21:57:20.231 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.4932000000000006e-05 2021-03-24T21:57:20.231 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.231 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.231 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:57:20.231 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.159945+0000", 2021-03-24T21:57:20.232 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000124358 2021-03-24T21:57:20.232 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.232 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.232 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 4,7", 2021-03-24T21:57:20.232 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.160069+0000", 2021-03-24T21:57:20.233 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0025748390000000002 2021-03-24T21:57:20.233 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.233 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.233 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:57:20.234 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.162644+0000", 2021-03-24T21:57:20.234 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0015101590000000001 2021-03-24T21:57:20.234 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.234 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.234 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:57:20.235 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.164154+0000", 2021-03-24T21:57:20.235 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001249996 2021-03-24T21:57:20.235 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.235 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.236 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:57:20.236 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.165404+0000", 2021-03-24T21:57:20.236 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.5167000000000006e-05 2021-03-24T21:57:20.236 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.236 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.237 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:57:20.237 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.165489+0000", 2021-03-24T21:57:20.237 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2519000000000002e-05 2021-03-24T21:57:20.237 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.237 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.238 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:57:20.238 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:47.165552+0000", 2021-03-24T21:57:20.238 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.01424504 2021-03-24T21:57:20.238 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:20.239 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:57:20.239 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:20.239 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.239 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.239 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:1547 7.11 e211/204 7:8dd64f9e:::smithi12556470-49:head v 211'181, mlcod=210'178)", 2021-03-24T21:57:20.240 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:56:20.149803+0000", 2021-03-24T21:57:20.240 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 59.908120996000001, 2021-03-24T21:57:20.240 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.019109896000000001, 2021-03-24T21:57:20.240 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:57:20.240 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:57:20.241 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:57:20.241 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.241 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:57:20.241 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:20.149803+0000", 2021-03-24T21:57:20.241 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:57:20.242 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.242 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.242 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:57:20.242 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:20.149803+0000", 2021-03-24T21:57:20.243 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T21:57:20.243 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.243 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.243 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:57:20.243 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:20.149802+0000", 2021-03-24T21:57:20.244 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0081225380000000003 2021-03-24T21:57:20.244 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.244 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.244 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:57:20.245 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:20.157924+0000", 2021-03-24T21:57:20.245 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.5210000000000001e-05 2021-03-24T21:57:20.245 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.245 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.245 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:57:20.246 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:20.157950+0000", 2021-03-24T21:57:20.246 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0497e-05 2021-03-24T21:57:20.246 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.246 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.246 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:57:20.247 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:20.157960+0000", 2021-03-24T21:57:20.247 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.3972000000000003e-05 2021-03-24T21:57:20.247 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.247 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.247 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:20.248 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:20.158034+0000", 2021-03-24T21:57:20.248 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.3331000000000003e-05 2021-03-24T21:57:20.248 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.248 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.249 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:57:20.249 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:20.158067+0000", 2021-03-24T21:57:20.249 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.010771836 2021-03-24T21:57:20.249 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.249 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.250 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:57:20.250 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:20.168839+0000", 2021-03-24T21:57:20.250 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.347e-05 2021-03-24T21:57:20.250 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.250 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.251 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:57:20.251 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:20.168913+0000", 2021-03-24T21:57:20.251 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.019109896000000001 2021-03-24T21:57:20.251 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:20.252 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:57:20.252 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:20.252 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.252 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.252 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:1777 7.17 e232/203 7:ef6d6bee:::smithi12556470-24:head v 232'218, mlcod=230'214)", 2021-03-24T21:57:20.253 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:56:41.201926+0000", 2021-03-24T21:57:20.253 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 38.855997539999997, 2021-03-24T21:57:20.253 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.039154920000000003, 2021-03-24T21:57:20.253 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:57:20.253 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:57:20.254 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:57:20.254 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.254 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:57:20.254 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:41.201926+0000", 2021-03-24T21:57:20.255 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:57:20.255 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.255 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.255 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:57:20.255 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:41.201926+0000", 2021-03-24T21:57:20.256 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999995 2021-03-24T21:57:20.256 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.256 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.256 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:57:20.256 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:41.201926+0000", 2021-03-24T21:57:20.257 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00074986499999999995 2021-03-24T21:57:20.257 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.257 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.257 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:57:20.257 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:41.202676+0000", 2021-03-24T21:57:20.258 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5279000000000001e-05 2021-03-24T21:57:20.258 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.258 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.258 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:57:20.258 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:41.202691+0000", 2021-03-24T21:57:20.259 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.8719999999999996e-06 2021-03-24T21:57:20.259 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.259 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.259 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:57:20.260 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:41.202698+0000", 2021-03-24T21:57:20.260 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.9821999999999994e-05 2021-03-24T21:57:20.260 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.260 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.260 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:20.261 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:41.202798+0000", 2021-03-24T21:57:20.261 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.3599000000000001e-05 2021-03-24T21:57:20.261 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.261 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.261 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:57:20.262 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:41.202831+0000", 2021-03-24T21:57:20.262 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.038160301000000001 2021-03-24T21:57:20.262 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.262 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.263 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:57:20.263 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:41.240992+0000", 2021-03-24T21:57:20.263 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.9573000000000005e-05 2021-03-24T21:57:20.263 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.263 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.264 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:57:20.264 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:41.241081+0000", 2021-03-24T21:57:20.264 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.039154920000000003 2021-03-24T21:57:20.264 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:20.264 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:57:20.265 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:20.265 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.265 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.265 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(osd.7.0:167 7.11 e248/204 7:8dd64f9e:::smithi12556470-49:78 v 248'233, mlcod=244'227)", 2021-03-24T21:57:20.265 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:56:57.145705+0000", 2021-03-24T21:57:20.266 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 22.912219205, 2021-03-24T21:57:20.266 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012349828, 2021-03-24T21:57:20.266 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:57:20.266 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:57:20.267 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:57:20.267 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.267 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:57:20.267 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:57.145705+0000", 2021-03-24T21:57:20.267 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:57:20.268 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.268 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.268 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:57:20.268 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:57.145705+0000", 2021-03-24T21:57:20.269 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T21:57:20.269 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.269 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.269 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:57:20.269 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:57.145704+0000", 2021-03-24T21:57:20.270 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.174e-06 2021-03-24T21:57:20.270 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.270 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.270 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:57:20.271 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:57.145712+0000", 2021-03-24T21:57:20.271 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.9009999999999994e-06 2021-03-24T21:57:20.271 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.271 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.271 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:57:20.272 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:57.145720+0000", 2021-03-24T21:57:20.272 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5210000000000002e-06 2021-03-24T21:57:20.272 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.272 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.272 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:57:20.273 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:57.145725+0000", 2021-03-24T21:57:20.273 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0037388790000000001 2021-03-24T21:57:20.273 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.273 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.273 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:20.273 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:57.149464+0000", 2021-03-24T21:57:20.274 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.9209000000000001e-05 2021-03-24T21:57:20.274 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.274 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.274 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:57:20.275 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:57.149493+0000", 2021-03-24T21:57:20.275 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0085010610000000007 2021-03-24T21:57:20.275 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.275 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.275 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:57:20.276 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:57.157995+0000", 2021-03-24T21:57:20.276 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9996e-05 2021-03-24T21:57:20.276 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.276 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.277 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:57:20.277 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:57.158055+0000", 2021-03-24T21:57:20.277 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012349828 2021-03-24T21:57:20.277 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:20.277 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:57:20.278 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:20.278 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.278 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.278 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(osd.3.0:522 7.14 e256/204 7:2cd73231:::smithi12556470-20:8f v 256'169, mlcod=251'165)", 2021-03-24T21:57:20.278 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:57:05.147512+0000", 2021-03-24T21:57:20.279 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 14.910411939999999, 2021-03-24T21:57:20.279 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.01787994, 2021-03-24T21:57:20.279 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:57:20.279 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:57:20.280 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:57:20.280 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.280 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:57:20.280 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.147512+0000", 2021-03-24T21:57:20.280 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:57:20.281 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.281 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.281 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:57:20.281 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.147512+0000", 2021-03-24T21:57:20.281 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T21:57:20.282 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.282 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.282 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:57:20.282 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.147511+0000", 2021-03-24T21:57:20.282 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.1459999999999999e-06 2021-03-24T21:57:20.283 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.283 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.283 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:57:20.283 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.147518+0000", 2021-03-24T21:57:20.283 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.1419999999999993e-06 2021-03-24T21:57:20.284 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.284 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.284 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:57:20.284 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.147526+0000", 2021-03-24T21:57:20.284 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.5629999999999997e-06 2021-03-24T21:57:20.285 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.285 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.285 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:57:20.285 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.147533+0000", 2021-03-24T21:57:20.286 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.4020000000000002e-05 2021-03-24T21:57:20.286 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.286 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.286 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:20.286 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.147577+0000", 2021-03-24T21:57:20.287 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9684999999999999e-05 2021-03-24T21:57:20.287 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.287 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.287 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:57:20.287 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.147597+0000", 2021-03-24T21:57:20.288 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.017761536000000001 2021-03-24T21:57:20.288 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.288 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.288 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:57:20.288 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.165358+0000", 2021-03-24T21:57:20.289 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.3655999999999998e-05 2021-03-24T21:57:20.289 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.289 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.289 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:57:20.289 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.165392+0000", 2021-03-24T21:57:20.290 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.01787994 2021-03-24T21:57:20.290 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:20.290 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:57:20.290 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:20.291 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.291 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.291 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(osd.7.0:209 7.11 e256/204 7:8c1d349c:::smithi12556470-48:8f v 256'256, mlcod=254'253)", 2021-03-24T21:57:20.291 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:57:05.149010+0000", 2021-03-24T21:57:20.291 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 14.908914311, 2021-03-24T21:57:20.292 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.019418392, 2021-03-24T21:57:20.292 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:57:20.292 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:57:20.292 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:57:20.292 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.293 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:57:20.293 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.149010+0000", 2021-03-24T21:57:20.293 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:57:20.293 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.293 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.294 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:57:20.294 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.149010+0000", 2021-03-24T21:57:20.294 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999995 2021-03-24T21:57:20.294 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.294 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.295 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:57:20.295 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.149009+0000", 2021-03-24T21:57:20.295 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.1600000000000003e-06 2021-03-24T21:57:20.295 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.296 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.296 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:57:20.296 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.149015+0000", 2021-03-24T21:57:20.296 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2391999999999999e-05 2021-03-24T21:57:20.296 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.297 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.297 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:57:20.297 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.149028+0000", 2021-03-24T21:57:20.297 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.0729999999999999e-06 2021-03-24T21:57:20.297 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.298 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.298 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:57:20.298 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.149035+0000", 2021-03-24T21:57:20.298 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001466282 2021-03-24T21:57:20.298 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.299 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.299 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:20.299 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.150501+0000", 2021-03-24T21:57:20.299 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.4975000000000001e-05 2021-03-24T21:57:20.299 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.300 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.300 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:57:20.300 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.150526+0000", 2021-03-24T21:57:20.300 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.017882625999999999 2021-03-24T21:57:20.301 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.301 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.301 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:57:20.301 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.168409+0000", 2021-03-24T21:57:20.301 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9357000000000001e-05 2021-03-24T21:57:20.302 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.302 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.302 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:57:20.302 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.168428+0000", 2021-03-24T21:57:20.302 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.019418392 2021-03-24T21:57:20.303 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:20.303 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:57:20.303 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:20.303 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.304 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.304 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(osd.7.0:210 7.19 e256/204 7:9c4f4ea5:::smithi12556470-41:83 v 256'242, mlcod=255'239)", 2021-03-24T21:57:20.304 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:57:05.149040+0000", 2021-03-24T21:57:20.304 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 14.908883771999999, 2021-03-24T21:57:20.304 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.019366148, 2021-03-24T21:57:20.305 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:57:20.305 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:57:20.305 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:57:20.305 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.305 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:57:20.306 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.149040+0000", 2021-03-24T21:57:20.306 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:57:20.306 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.306 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.306 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:57:20.307 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.149040+0000", 2021-03-24T21:57:20.307 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999995 2021-03-24T21:57:20.307 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.307 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.308 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:57:20.308 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.149040+0000", 2021-03-24T21:57:20.308 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.8559999999999996e-06 2021-03-24T21:57:20.308 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.308 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.309 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:57:20.309 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.149049+0000", 2021-03-24T21:57:20.309 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.8119999999999996e-06 2021-03-24T21:57:20.309 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.310 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.310 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:57:20.310 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.149054+0000", 2021-03-24T21:57:20.310 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.349e-06 2021-03-24T21:57:20.310 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.311 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.311 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:57:20.311 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.149058+0000", 2021-03-24T21:57:20.311 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001798797 2021-03-24T21:57:20.311 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.312 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.312 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:20.312 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.150857+0000", 2021-03-24T21:57:20.312 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7183000000000001e-05 2021-03-24T21:57:20.312 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.313 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.313 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:57:20.313 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.150874+0000", 2021-03-24T21:57:20.313 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.017494378000000001 2021-03-24T21:57:20.314 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.314 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.314 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:57:20.314 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.168368+0000", 2021-03-24T21:57:20.314 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.8213000000000002e-05 2021-03-24T21:57:20.315 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.315 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.315 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:57:20.315 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.168406+0000", 2021-03-24T21:57:20.315 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.019366148 2021-03-24T21:57:20.315 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:20.316 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:57:20.316 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:20.316 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.316 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.316 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(osd.7.0:211 7.11 e256/204 7:8eea9237:::smithi12556470-1:83 v 256'258, mlcod=254'253)", 2021-03-24T21:57:20.317 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:57:05.149514+0000", 2021-03-24T21:57:20.317 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 14.908409744, 2021-03-24T21:57:20.317 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.019646961000000001, 2021-03-24T21:57:20.317 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:57:20.317 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:57:20.318 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:57:20.318 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.318 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:57:20.318 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.149514+0000", 2021-03-24T21:57:20.318 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:57:20.319 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.319 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.319 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:57:20.319 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.149514+0000", 2021-03-24T21:57:20.319 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T21:57:20.320 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.320 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.320 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:57:20.320 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.149513+0000", 2021-03-24T21:57:20.320 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.3130000000000004e-06 2021-03-24T21:57:20.321 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.321 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.321 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:57:20.321 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.149520+0000", 2021-03-24T21:57:20.321 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2040000000000002e-06 2021-03-24T21:57:20.322 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.322 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.322 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:57:20.322 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.149526+0000", 2021-03-24T21:57:20.323 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2809999999999999e-06 2021-03-24T21:57:20.323 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.323 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.323 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:57:20.323 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.149531+0000", 2021-03-24T21:57:20.324 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.015713127 2021-03-24T21:57:20.324 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.324 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.324 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:20.324 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.165244+0000", 2021-03-24T21:57:20.325 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.3161999999999999e-05 2021-03-24T21:57:20.325 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.325 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.325 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:57:20.325 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.165268+0000", 2021-03-24T21:57:20.326 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0038786039999999999 2021-03-24T21:57:20.326 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.326 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.326 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:57:20.327 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.169146+0000", 2021-03-24T21:57:20.327 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.4987999999999999e-05 2021-03-24T21:57:20.327 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.327 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.327 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:57:20.328 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.169161+0000", 2021-03-24T21:57:20.328 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.019646961000000001 2021-03-24T21:57:20.328 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:20.328 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:57:20.328 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:20.329 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.329 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.329 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(osd.7.0:212 7.19 e256/204 7:992cb487:::smithi12556470-5:92 v 256'244, mlcod=255'239)", 2021-03-24T21:57:20.329 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:57:05.151829+0000", 2021-03-24T21:57:20.329 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 14.906095374, 2021-03-24T21:57:20.330 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.017315692000000001, 2021-03-24T21:57:20.330 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:57:20.330 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:57:20.330 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:57:20.330 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.331 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:57:20.331 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.151829+0000", 2021-03-24T21:57:20.331 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:57:20.331 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.331 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.332 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:57:20.332 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.151829+0000", 2021-03-24T21:57:20.332 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T21:57:20.332 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.332 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.333 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:57:20.333 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.151828+0000", 2021-03-24T21:57:20.333 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.8889999999999996e-06 2021-03-24T21:57:20.333 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.333 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.334 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:57:20.334 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.151835+0000", 2021-03-24T21:57:20.334 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2439999999999993e-06 2021-03-24T21:57:20.334 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.334 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.335 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:57:20.335 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.151843+0000", 2021-03-24T21:57:20.335 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.8549999999999998e-06 2021-03-24T21:57:20.335 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.336 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.336 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:57:20.336 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.151849+0000", 2021-03-24T21:57:20.336 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.013383232 2021-03-24T21:57:20.336 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.337 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.337 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:20.337 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.165232+0000", 2021-03-24T21:57:20.337 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.2079e-05 2021-03-24T21:57:20.337 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.338 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.338 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:57:20.338 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.165254+0000", 2021-03-24T21:57:20.338 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0038708789999999998 2021-03-24T21:57:20.338 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.339 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.339 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:57:20.339 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.169125+0000", 2021-03-24T21:57:20.339 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9364e-05 2021-03-24T21:57:20.339 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.340 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.340 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:57:20.340 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.169144+0000", 2021-03-24T21:57:20.340 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.017315692000000001 2021-03-24T21:57:20.340 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:20.341 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:57:20.341 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:20.341 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.341 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.341 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:2345 7.11 e269/204 7:8c1d349c:::smithi12556470-48:head v 269'289, mlcod=268'285)", 2021-03-24T21:57:20.342 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:57:19.832048+0000", 2021-03-24T21:57:20.342 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 0.22587604, 2021-03-24T21:57:20.342 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.014660479000000001, 2021-03-24T21:57:20.342 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:57:20.342 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:57:20.343 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:57:20.343 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.343 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:57:20.343 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:19.832048+0000", 2021-03-24T21:57:20.343 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:57:20.344 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.344 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.344 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:57:20.344 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:19.832048+0000", 2021-03-24T21:57:20.344 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T21:57:20.345 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.345 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.345 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:57:20.345 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:19.832046+0000", 2021-03-24T21:57:20.345 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0077522700000000003 2021-03-24T21:57:20.346 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.346 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.346 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:57:20.346 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:19.839798+0000", 2021-03-24T21:57:20.346 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0933000000000001e-05 2021-03-24T21:57:20.347 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.347 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.347 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:57:20.347 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:19.839819+0000", 2021-03-24T21:57:20.348 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2069999999999995e-06 2021-03-24T21:57:20.348 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.348 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.348 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:57:20.348 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:19.839827+0000", 2021-03-24T21:57:20.349 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.8728999999999997e-05 2021-03-24T21:57:20.349 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.349 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.349 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:20.349 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:19.839916+0000", 2021-03-24T21:57:20.350 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1643e-05 2021-03-24T21:57:20.350 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.350 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.350 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:57:20.351 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:19.839968+0000", 2021-03-24T21:57:20.351 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0066941839999999997 2021-03-24T21:57:20.351 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.351 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.351 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:57:20.352 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:19.846662+0000", 2021-03-24T21:57:20.352 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6529999999999997e-05 2021-03-24T21:57:20.352 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:20.352 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:20.352 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:57:20.353 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:19.846708+0000", 2021-03-24T21:57:20.353 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.014660479000000001 2021-03-24T21:57:20.353 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:20.353 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:57:20.353 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:20.354 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:20.354 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:57:20.354 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-24T21:57:20.354 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:20.356 INFO:tasks.rados.rados.0.smithi125.stdout:893: done (1 left) 2021-03-24T21:57:20.356 INFO:tasks.rados.rados.0.smithi125.stdout:894: write oid 8 current snap is 83 2021-03-24T21:57:20.356 INFO:tasks.rados.rados.0.smithi125.stdout:894: seq_num 333 ranges {796731=461096,1736739=723770,2928680=668051} 2021-03-24T21:57:20.357 INFO:tasks.rados.rados.0.smithi125.stdout:894: writing smithi12556470-8 from 796731 to 1257827 tid 1 2021-03-24T21:57:20.357 INFO:tasks.rados.rados.0.smithi125.stdout:894: writing smithi12556470-8 from 1736739 to 2460509 tid 2 2021-03-24T21:57:20.357 INFO:tasks.rados.rados.0.smithi125.stdout:894: writing smithi12556470-8 from 2928680 to 3596731 tid 3 2021-03-24T21:57:20.357 INFO:tasks.rados.rados.0.smithi125.stdout:895: read oid 46 snap 81 2021-03-24T21:57:20.358 INFO:tasks.rados.rados.0.smithi125.stdout:895: expect (ObjNum 315 snap 80 seq_num 315) 2021-03-24T21:57:20.358 INFO:tasks.rados.rados.0.smithi125.stdout:896: snap_remove snap 82 2021-03-24T21:57:20.412 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:20.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:20.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:20.449 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:20 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:57:20.449 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:20 smithi125 conmon[24927]: :57:18.769787+0000 mgr.y (mgr.14144) 401 : cluster [DBG] pgmap v565: 158 pgs: 158 active+clean; 146 MiB data, 494 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:57:20.449 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:20 smithi125 conmon[24927]: cluster 2021-03-24T21:57: 2021-03-24T21:57:20.449 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:20 smithi125 conmon[24927]: 18.801557+0000 osd.4 (osd.4) 71 : cluster [DBG] 4.d scrub ok 2021-03-24T21:57:20.450 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:20 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:57:20.450 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:20 smithi125 conmon[24927]: T21:57:19.252125+0000 mon.a (mon. 2021-03-24T21:57:20.450 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:20 smithi125 conmon[24927]: 0) 915 : cluster [DBG] osdmap e269: 8 total, 8 up, 6 in 2021-03-24T21:57:20.450 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:20 smithi125 conmon[24927]: cluster 2021-03-24T21:57:19 2021-03-24T21:57:20.451 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:20 smithi125 conmon[24927]: .271343+0000 osd.0 (osd.0) 71 : cluster [DBG] 6.18 scrub ok 2021-03-24T21:57:20.451 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:20 smithi125 conmon[24927]: cluster 2021-03-24T21:57 2021-03-24T21:57:20.451 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:20 smithi125 conmon[24927]: :20.146606+0000 mon.a (mon.0) 916 : cluster [DBG] osdmap e270: 8 total, 8 up, 6 in 2021-03-24T21:57:20.451 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:20 smithi125 conmon[19936]: cluster 2021-03-24T21:57 2021-03-24T21:57:20.451 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:20 smithi125 conmon[19936]: :18.769787+0000 mgr.y (mgr.14144) 401 : cluster [DBG] pgmap v565: 158 pgs: 158 active+clean; 146 MiB data, 494 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:57:20.452 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:20 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:57:20.452 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:20 smithi125 conmon[19936]: 03-24T21:57:18.801557+0000 osd.4 (osd.4) 71 : 2021-03-24T21:57:20.452 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:20 smithi125 conmon[19936]: cluster [DBG] 4.d scrub ok 2021-03-24T21:57:20.452 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:20 smithi125 conmon[19936]: cluster 2021-03-24T21:57:20.453 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:20 smithi125 conmon[19936]: 2021-03-24T21:57 2021-03-24T21:57:20.453 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:20 smithi125 conmon[19936]: :19.252125+0000 mon.a (mon.0) 915 : cluster [DBG] 2021-03-24T21:57:20.453 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:20 smithi125 conmon[19936]: osdmap e269: 8 total, 8 up, 6 in 2021-03-24T21:57:20.453 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:20 smithi125 conmon[19936]: cluster 2021-03-24T21:57:20.453 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:20 smithi125 conmon[19936]: 2021-03-24T21:57:19.271343+0000 osd.0 (osd.0) 2021-03-24T21:57:20.453 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:20 smithi125 conmon[19936]: 71 : cluster [DBG] 6.18 scrub ok 2021-03-24T21:57:20.454 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:20 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:57:20.454 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:20 smithi125 conmon[19936]: -03-24T21:57:20.146606+0000 mon.a (mon.0) 916 : cluster [DBG] 2021-03-24T21:57:20.454 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:20 smithi125 conmon[19936]: osdmap e270: 8 total, 8 up, 6 in 2021-03-24T21:57:20.518 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:20.572 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:20.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:20.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:57:20.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:20 smithi142 conmon[20125]: cluster 2021-03-24T21:57:18.769787+0000 2021-03-24T21:57:20.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:20 smithi142 conmon[20125]: mgr.y (mgr.14144) 401 : cluster [DBG] pgmap v565: 158 pgs: 158 active+clean; 146 MiB data, 494 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:57:20.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:20 smithi142 conmon[20125]: cluster 2021-03-24T21:57:18.801557+0000 osd.4 (osd.4) 71 : cluster 2021-03-24T21:57:20.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:20 smithi142 conmon[20125]: [DBG] 4.d scrub ok 2021-03-24T21:57:20.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:20 smithi142 conmon[20125]: cluster 2021-03-24T21:57:19.252125+0000 mon.a (mon.0 2021-03-24T21:57:20.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:20 smithi142 conmon[20125]: ) 915 : cluster [DBG] osdmap e269: 8 total, 8 up, 6 in 2021-03-24T21:57:20.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:20 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:57:20.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:20 smithi142 conmon[20125]: -03-24T21:57:19.271343+0000 osd.0 (osd. 2021-03-24T21:57:20.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:20 smithi142 conmon[20125]: 0) 71 : cluster [DBG] 6.18 scrub ok 2021-03-24T21:57:20.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:20 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:57:20.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:20 smithi142 conmon[20125]: -03-24T21:57:20.146606+0000 2021-03-24T21:57:20.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:20 smithi142 conmon[20125]: mon.a (mon.0) 916 : cluster [DBG] osdmap e270: 8 total, 8 up, 6 in 2021-03-24T21:57:20.678 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:20.737 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:20.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:20.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:57:20.842 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:20.903 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:20.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:20.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:57:21.006 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:21.063 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:21.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:21.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:57:21.167 INFO:tasks.rados.rados.0.smithi125.stdout:896: done (3 left) 2021-03-24T21:57:21.167 INFO:tasks.rados.rados.0.smithi125.stdout:897: snap_remove snap 75 2021-03-24T21:57:21.170 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:21.216 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:21.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:21.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:21.323 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:21.357 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:21 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:57:21.358 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:21 smithi125 conmon[19936]: 21:57:19.767442+0000 osd.4 (osd.4) 72 : cluster [DBG] 5.1c scrub ok 2021-03-24T21:57:21.358 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:21 smithi125 conmon[19936]: cluster 2021-03-24T21:57:20.073638+0000 2021-03-24T21:57:21.358 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:21 smithi125 conmon[19936]: osd.2 (osd.2) 53 : cluster [DBG] 6.14 scrub ok 2021-03-24T21:57:21.358 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:21 smithi125 conmon[19936]: 2021-03-24T21:57:21.359 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:21 smithi125 conmon[19936]: cluster 2021-03-24T21:57:21.359 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:21 smithi125 conmon[19936]: 2021-03-24T21:57:20.215044+0000 2021-03-24T21:57:21.359 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:21 smithi125 conmon[19936]: osd.5 (osd.5) 71 : cluster [DBG] 5.6 scrub ok 2021-03-24T21:57:21.359 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:21 smithi125 conmon[19936]: 2021-03-24T21:57:21.359 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:21 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:57:21.360 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:21 smithi125 conmon[19936]: 57:21.162910+0000 mon.a (mon.0) 2021-03-24T21:57:21.360 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:21 smithi125 conmon[19936]: 917 : cluster [DBG] osdmap e271: 8 total, 8 up, 6 in 2021-03-24T21:57:21.360 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:21 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:57:21.361 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:21 smithi125 conmon[24927]: -24T21:57:19.767442 2021-03-24T21:57:21.361 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:21 smithi125 conmon[24927]: +0000 osd.4 (osd. 2021-03-24T21:57:21.361 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:21 smithi125 conmon[24927]: 4) 72 : cluster 2021-03-24T21:57:21.361 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:21 smithi125 conmon[24927]: [DBG] 5.1c scrub ok 2021-03-24T21:57:21.362 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:21 smithi125 conmon[24927]: cluster 2021-03-24T21:57:21.362 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:21 smithi125 conmon[24927]: 2021-03-24T21 2021-03-24T21:57:21.362 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:21 smithi125 conmon[24927]: :57:20. 2021-03-24T21:57:21.362 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:21 smithi125 conmon[24927]: 073638+0000 osd.2 ( 2021-03-24T21:57:21.363 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:21 smithi125 conmon[24927]: osd.2) 53 2021-03-24T21:57:21.363 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:21 smithi125 conmon[24927]: : cluster [DBG] 6.14 scrub ok 2021-03-24T21:57:21.363 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:21 smithi125 conmon[24927]: 2021-03-24T21:57:21.363 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:21 smithi125 conmon[24927]: cluster 2021-03-24T21:57:21.363 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:21 smithi125 conmon[24927]: 2021-03-24T21: 2021-03-24T21:57:21.364 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:21 smithi125 conmon[24927]: 57:20.215044+0000 2021-03-24T21:57:21.364 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:21 smithi125 conmon[24927]: osd.5 (osd. 2021-03-24T21:57:21.364 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:21 smithi125 conmon[24927]: 5) 71 : 2021-03-24T21:57:21.364 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:21 smithi125 conmon[24927]: cluster [DBG] 5.6 scrub ok 2021-03-24T21:57:21.364 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:21 smithi125 conmon[24927]: 2021-03-24T21:57:21.365 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:21 smithi125 conmon[24927]: cluster 2021-03-24T21:57:21 2021-03-24T21:57:21.365 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:21 smithi125 conmon[24927]: .162910+0000 mon.a (mon.0) 917 : cluster [DBG] osdmap e271: 8 total, 8 up, 6 in 2021-03-24T21:57:21.374 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:21.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:21.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:57:21.481 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:21.536 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:21.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:21.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:57:21.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:21 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:57:21.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:21 smithi142 conmon[20125]: T21:57:19.767442+0000 osd.4 (osd.4) 72 : cluster [DBG] 5.1c scrub ok 2021-03-24T21:57:21.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:21 smithi142 conmon[20125]: cluster 2021-03-24T21:57:20. 2021-03-24T21:57:21.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:21 smithi142 conmon[20125]: 073638+0000 osd.2 (osd.2) 53 : cluster [DBG] 2021-03-24T21:57:21.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:21 smithi142 conmon[20125]: 6.14 scrub ok 2021-03-24T21:57:21.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:21 smithi142 conmon[20125]: cluster 2021-03-24T21:57:20.215044 2021-03-24T21:57:21.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:21 smithi142 conmon[20125]: +0000 osd.5 (osd.5) 71 : cluster [DBG] 2021-03-24T21:57:21.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:21 smithi142 conmon[20125]: 5.6 scrub ok 2021-03-24T21:57:21.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:21 smithi142 conmon[20125]: cluster 2021-03-24T21:57 2021-03-24T21:57:21.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:21 smithi142 conmon[20125]: :21.162910+0000 mon.a (mon.0) 917 : cluster [DBG] osdmap e271: 8 total, 8 up, 6 in 2021-03-24T21:57:21.642 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:21.698 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:21.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:21.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:57:21.805 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:21.864 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:22.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:22.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:57:22.111 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:22.172 INFO:tasks.rados.rados.0.smithi125.stdout:897: done (3 left) 2021-03-24T21:57:22.173 INFO:tasks.rados.rados.0.smithi125.stdout:898: snap_create 2021-03-24T21:57:22.173 INFO:tasks.rados.rados.0.smithi125.stdout:891: finishing write tid 4 to smithi12556470-16 2021-03-24T21:57:22.173 INFO:tasks.rados.rados.0.smithi125.stdout:891: finishing write tid 5 to smithi12556470-16 2021-03-24T21:57:22.173 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 16 v 212 (ObjNum 331 snap 83 seq_num 331) dirty exists 2021-03-24T21:57:22.174 INFO:tasks.rados.rados.0.smithi125.stdout:891: left oid 16 (ObjNum 331 snap 83 seq_num 331) 2021-03-24T21:57:22.174 INFO:tasks.rados.rados.0.smithi125.stdout:894: finishing write tid 1 to smithi12556470-8 2021-03-24T21:57:22.174 INFO:tasks.rados.rados.0.smithi125.stdout:894: finishing write tid 2 to smithi12556470-8 2021-03-24T21:57:22.174 INFO:tasks.rados.rados.0.smithi125.stdout:894: finishing write tid 3 to smithi12556470-8 2021-03-24T21:57:22.174 INFO:tasks.rados.rados.0.smithi125.stdout:894: finishing write tid 4 to smithi12556470-8 2021-03-24T21:57:22.175 INFO:tasks.rados.rados.0.smithi125.stdout:894: finishing write tid 5 to smithi12556470-8 2021-03-24T21:57:22.175 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 8 v 60 (ObjNum 333 snap 83 seq_num 333) dirty exists 2021-03-24T21:57:22.175 INFO:tasks.rados.rados.0.smithi125.stdout:894: left oid 8 (ObjNum 333 snap 83 seq_num 333) 2021-03-24T21:57:22.175 INFO:tasks.rados.rados.0.smithi125.stdout:895: expect (ObjNum 315 snap 80 seq_num 315) 2021-03-24T21:57:22.179 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:22.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:22.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:57:22.284 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:22.292 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:22 smithi125 conmon[19936]: cluster 2021-03-24T21:57:20.770512+0000 mgr.y (mgr.14144) 402 : cluster [DBG] pgmap v568: 158 pgs: 158 active+clean; 155 MiB data, 520 MiB used, 536 GiB / 536 GiB avail; 38 MiB/s rd, 5.7 MiB/s wr, 47 op/s 2021-03-24T21:57:22.293 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:22 smithi125 conmon[19936]: cluster 2021-03-24T21:57:21.260719+0000 osd.0 (osd.0) 72 : cluster [DBG] 7.4 scrub ok 2021-03-24T21:57:22.293 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:22 smithi125 conmon[19936]: cluster 2021-03-24T21:57:22.166107+0000 mon.a (mon.0) 918 : cluster [DBG] osdmap e272: 8 total, 8 up, 6 in 2021-03-24T21:57:22.293 INFO:journalctl@ceph.osd.3.smithi125.stdout:Mar 24 21:57:22 smithi125 conmon[37997]: debug 2021-03-24T21:57:22.294 INFO:journalctl@ceph.osd.3.smithi125.stdout:Mar 24 21:57:22 smithi125 conmon[37997]: 2021-03-24T21:57:22.176+0000 7f1717bdc700 -1 osd.3 pg_epoch: 272 pg[7.a( v 265'120 (0'0,265'120] local-lis/les=99/100 n=3 ec=99/99 lis/c=99/99 les/c/f=100/100/0 sis=99) [3,2,5] r=0 lpr=99 crt=265'120 lcod 265'118 mlcod 265'118 active+clean+snaptrim trimq=[93~1] ps=[8b~1,97~1,99~1,9e~1,a2~1]] removing snap head 2021-03-24T21:57:22.294 INFO:journalctl@ceph.osd.3.smithi125.stdout:Mar 24 21:57:22 smithi125 conmon[37997]: 2021-03-24T21:57:22.294 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:22 smithi125 conmon[24927]: cluster 2021-03-24T21:57:20.770512+0000 mgr.y (mgr.14144) 402 : cluster [DBG] pgmap v568: 158 pgs: 158 active+clean; 155 MiB data, 520 MiB used, 536 GiB / 536 GiB avail; 38 MiB/s rd, 5.7 MiB/s wr, 47 op/s 2021-03-24T21:57:22.295 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:22 smithi125 conmon[24927]: cluster 2021-03-24T21:57:21.260719+0000 osd.0 (osd.0) 72 : cluster [DBG] 7.4 scrub ok 2021-03-24T21:57:22.295 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:22 smithi125 conmon[24927]: cluster 2021-03-24T21:57:22.166107+0000 mon.a (mon.0) 918 : cluster [DBG] osdmap e272: 8 total, 8 up, 6 in 2021-03-24T21:57:22.330 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:22.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:22.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:57:22.435 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:22.454 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:22 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:57:22.455 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:22 smithi142 conmon[20125]: 57:20.770512+0000 mgr.y (mgr.14144) 402 : cluster [DBG] pgmap v568: 158 pgs: 158 active+clean; 155 MiB data, 520 MiB used, 536 GiB / 536 GiB avail; 38 MiB/s rd, 5.7 MiB/s wr, 47 op/s 2021-03-24T21:57:22.455 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:22 smithi142 conmon[20125]: cluster 2021-03-24T21 2021-03-24T21:57:22.455 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:22 smithi142 conmon[20125]: :57:21.260719+0000 2021-03-24T21:57:22.455 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:22 smithi142 conmon[20125]: osd.0 (osd.0) 72 : cluster 2021-03-24T21:57:22.456 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:22 smithi142 conmon[20125]: [DBG] 7.4 scrub ok 2021-03-24T21:57:22.456 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:22 smithi142 conmon[20125]: cluster 2021-03-24T21:57:22.456 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:22 smithi142 conmon[20125]: 2021-03-24T21:57:22 2021-03-24T21:57:22.456 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:22 smithi142 conmon[20125]: .166107+0000 mon.a (mon. 2021-03-24T21:57:22.457 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:22 smithi142 conmon[20125]: 0) 918 : cluster [DBG] 2021-03-24T21:57:22.457 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:22 smithi142 conmon[20125]: osdmap e272: 8 total, 8 up, 6 in 2021-03-24T21:57:22.497 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:22.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:22.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:57:22.603 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:22.801 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:22.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:22.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:57:22.905 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:22.965 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:22.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:22.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:57:23.070 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:23.127 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:23.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:23.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:57:23.173 INFO:tasks.rados.rados.0.smithi125.stdout:891: done (3 left) 2021-03-24T21:57:23.174 INFO:tasks.rados.rados.0.smithi125.stdout:894: done (2 left) 2021-03-24T21:57:23.174 INFO:tasks.rados.rados.0.smithi125.stdout:895: done (1 left) 2021-03-24T21:57:23.175 INFO:tasks.rados.rados.0.smithi125.stdout:898: done (0 left) 2021-03-24T21:57:23.175 INFO:tasks.rados.rados.0.smithi125.stdout:899: read oid 33 snap -1 2021-03-24T21:57:23.176 INFO:tasks.rados.rados.0.smithi125.stdout:899: expect (ObjNum 298 snap 74 seq_num 298) 2021-03-24T21:57:23.176 INFO:tasks.rados.rados.0.smithi125.stdout:900: read oid 14 snap -1 2021-03-24T21:57:23.176 INFO:tasks.rados.rados.0.smithi125.stdout:900: expect (ObjNum 303 snap 74 seq_num 303) 2021-03-24T21:57:23.177 INFO:tasks.rados.rados.0.smithi125.stdout:901: write (excl) oid 21 current snap is 84 2021-03-24T21:57:23.177 INFO:tasks.rados.rados.0.smithi125.stdout:901: seq_num 334 ranges {413538=536345,1395221=583781,2556103=658617} 2021-03-24T21:57:23.181 INFO:teuthology.orchestra.run.smithi125.stderr:nodeep-scrub is unset 2021-03-24T21:57:23.182 INFO:tasks.rados.rados.0.smithi125.stdout:901: writing smithi12556470-21 from 413538 to 949883 tid 1 2021-03-24T21:57:23.188 INFO:tasks.rados.rados.0.smithi125.stdout:901: writing smithi12556470-21 from 1395221 to 1979002 tid 2 2021-03-24T21:57:23.195 INFO:tasks.rados.rados.0.smithi125.stdout:901: writing smithi12556470-21 from 2556103 to 3214720 tid 3 2021-03-24T21:57:23.196 INFO:tasks.rados.rados.0.smithi125.stdout:902: rollback oid 2 current snap is 84 2021-03-24T21:57:23.196 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 2 to 81 2021-03-24T21:57:23.196 INFO:tasks.rados.rados.0.smithi125.stdout:900: expect (ObjNum 303 snap 74 seq_num 303) 2021-03-24T21:57:23.234 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:23.293 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:23.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:23.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:23.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:23 smithi142 conmon[20125]: cluster 2021-03-24T21:57 2021-03-24T21:57:23.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:23 smithi142 conmon[20125]: :21.706118+0000 osd.4 (osd.4) 73 : cluster [DBG] 4.1 scrub ok 2021-03-24T21:57:23.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:23 smithi142 conmon[20125]: cluster 2021-03-24T21:57:23.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:23 smithi142 conmon[20125]: 2021-03-24T21:57:22.158021 2021-03-24T21:57:23.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:23 smithi142 conmon[20125]: +0000 osd.7 (osd.7) 2021-03-24T21:57:23.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:23 smithi142 conmon[20125]: 62 : cluster [DBG] 7.19 scrub ok 2021-03-24T21:57:23.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:23 smithi142 conmon[20125]: 2021-03-24T21:57:23.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:23 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:57:23.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:23 smithi142 conmon[20125]: 03-24T21:57:22 2021-03-24T21:57:23.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:23 smithi142 conmon[20125]: .239421+0000 osd.0 (osd 2021-03-24T21:57:23.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:23 smithi142 conmon[20125]: .0) 73 : cluster 2021-03-24T21:57:23.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:23 smithi142 conmon[20125]: [DBG] 7.7 scrub ok 2021-03-24T21:57:23.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:23 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:57:23.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:23 smithi142 conmon[20125]: 03-24T21:57: 2021-03-24T21:57:23.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:23 smithi142 conmon[20125]: 22.422780+0000 osd.3 ( 2021-03-24T21:57:23.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:23 smithi142 conmon[20125]: osd.3) 69 : cluster 2021-03-24T21:57:23.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:23 smithi142 conmon[20125]: [DBG] 6.6 scrub ok 2021-03-24T21:57:23.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:23 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:57:23.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:23 smithi142 conmon[20125]: 03-24T21:57:22 2021-03-24T21:57:23.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:23 smithi142 conmon[20125]: .561375+0000 mon.c (mon 2021-03-24T21:57:23.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:23 smithi142 conmon[20125]: .2) 42 : audit 2021-03-24T21:57:23.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:23 smithi142 conmon[20125]: [INF] from='client.? v1:172.21.15.125:0/951566320' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:57:23.373 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:23 smithi142 conmon[20125]: audit 2021-03-24T21:57:23.373 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:23 smithi142 conmon[20125]: 2021-03-24T21:57: 2021-03-24T21:57:23.373 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:23 smithi142 conmon[20125]: 22.561947+0000 mon.a (mon 2021-03-24T21:57:23.373 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:23 smithi142 conmon[20125]: .0) 919 : audit 2021-03-24T21:57:23.373 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:23 smithi142 conmon[20125]: [INF] from='client.34973 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:57:23.373 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:23 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:57:23.374 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:23 smithi142 conmon[20125]: -03-24T21:57:23 2021-03-24T21:57:23.374 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:23 smithi142 conmon[20125]: .157101+0000 mon.a ( 2021-03-24T21:57:23.374 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:23 smithi142 conmon[20125]: mon.0) 920 : cluster 2021-03-24T21:57:23.374 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:23 smithi142 conmon[20125]: [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T21:57:23.375 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:23 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:57:23.375 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:23 smithi142 conmon[20125]: 03-24T21:57: 2021-03-24T21:57:23.375 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:23 smithi142 conmon[20125]: 23.157185+0000 mon.a (mon.0) 921 : 2021-03-24T21:57:23.375 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:23 smithi142 conmon[20125]: cluster [INF] Cluster is now healthy 2021-03-24T21:57:23.375 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:23 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:57:23.376 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:23 smithi142 conmon[20125]: 03-24T21:57: 2021-03-24T21:57:23.376 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:23 smithi142 conmon[20125]: 23.170628+0000 mon.a (mon.0 2021-03-24T21:57:23.376 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:23 smithi142 conmon[20125]: ) 922 : audit [INF] 2021-03-24T21:57:23.376 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:23 smithi142 conmon[20125]: from='client.34973 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T21:57:23.376 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:23 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:57:23.377 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:23 smithi142 conmon[20125]: -03-24T21: 2021-03-24T21:57:23.377 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:23 smithi142 conmon[20125]: 57:23.170773 2021-03-24T21:57:23.377 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:23 smithi142 conmon[20125]: +0000 mon.a (mon 2021-03-24T21:57:23.377 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:23 smithi142 conmon[20125]: .0) 923 : 2021-03-24T21:57:23.378 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:23 smithi142 conmon[20125]: cluster [DBG] osdmap e273: 8 total, 8 up, 6 in 2021-03-24T21:57:23.397 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:23.401 INFO:tasks.rados.rados.0.smithi125.stdout:901: finishing write tid 1 to smithi12556470-21 2021-03-24T21:57:23.402 INFO:tasks.rados.rados.0.smithi125.stdout:901: finishing write tid 2 to smithi12556470-21 2021-03-24T21:57:23.402 INFO:tasks.rados.rados.0.smithi125.stdout:899: expect (ObjNum 298 snap 74 seq_num 298) 2021-03-24T21:57:23.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[19936]: cluster 2021-03-24T21:57:21.706118+0000 2021-03-24T21:57:23.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[19936]: osd.4 (osd.4) 73 : cluster [DBG] 4.1 scrub ok 2021-03-24T21:57:23.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:57:23.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[19936]: :57:22.158021+0000 osd.7 (osd.7) 2021-03-24T21:57:23.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[19936]: 62 : cluster [DBG] 7.19 scrub ok 2021-03-24T21:57:23.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[19936]: cluster 2021-03-24T21:57:23.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[19936]: 2021-03-24T21:57 2021-03-24T21:57:23.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[19936]: :22.239421+0000 osd.0 (osd 2021-03-24T21:57:23.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[19936]: .0) 73 : cluster [DBG] 7.7 scrub ok 2021-03-24T21:57:23.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[19936]: cluster 2021-03-24T21:57:23.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[19936]: 2021-03-24T 2021-03-24T21:57:23.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[19936]: 21:57:22.422780+0000 osd.3 (osd.3) 2021-03-24T21:57:23.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[19936]: 69 : cluster [DBG] 6.6 scrub ok 2021-03-24T21:57:23.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[19936]: audit 2021-03-24T21:57:23.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[19936]: 2021-03-24T 2021-03-24T21:57:23.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[19936]: 21:57:22.561375+0000 mon.c (mon.2) 42 : audit [INF] 2021-03-24T21:57:23.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[19936]: from='client.? v1:172.21.15.125:0/951566320' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:57:23.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[19936]: audit 2021-03-24T21:57:22 2021-03-24T21:57:23.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[19936]: .561947+0000 mon.a (mon.0) 919 : audit [INF] from='client.34973 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:57:23.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:57:23.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[19936]: 57:23.157101+0000 mon.a (mon.0) 920 : cluster [INF] 2021-03-24T21:57:23.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[19936]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T21:57:23.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:57:23.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[19936]: -24T21:57:23.157185+0000 mon.a (mon.0) 921 : cluster [INF] Cluster is now healthy 2021-03-24T21:57:23.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[19936]: audit 2021-03-24T21:57:23.170628+0000 2021-03-24T21:57:23.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[19936]: mon.a (mon.0) 922 : audit [INF] from='client.34973 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T21:57:23.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[19936]: cluster 2021-03-24T21:57:23 2021-03-24T21:57:23.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[19936]: .170773+0000 mon.a (mon.0) 923 : cluster [DBG] osdmap e273: 8 total, 8 up, 6 in 2021-03-24T21:57:23.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[24927]: cluster 2021-03-24T21:57:23.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[24927]: 2021-03-24T21 2021-03-24T21:57:23.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[24927]: :57:21.706118+0000 osd.4 (osd.4) 73 : cluster [DBG] 4.1 scrub ok 2021-03-24T21:57:23.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[24927]: cluster 2021-03-24T21:57:23.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[24927]: 2021-03-24 2021-03-24T21:57:23.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[24927]: T21:57:22.158021+0000 osd.7 (osd. 2021-03-24T21:57:23.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[24927]: 7) 62 : cluster [DBG] 7.19 scrub ok 2021-03-24T21:57:23.420 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:57:23.420 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[24927]: T21:57:22.239421+0000 osd.0 (osd. 2021-03-24T21:57:23.420 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[24927]: 0) 73 : cluster [DBG] 7.7 scrub ok 2021-03-24T21:57:23.420 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[24927]: cluster 2021-03-24T21:57: 2021-03-24T21:57:23.421 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[24927]: 22.422780+0000 osd.3 (osd.3) 69 : cluster [DBG] 6.6 scrub ok 2021-03-24T21:57:23.421 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T21:57:23.421 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[24927]: 24T21:57:22.561375+0000 mon.c (mon.2) 2021-03-24T21:57:23.421 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[24927]: 42 : audit [INF] from='client.? v1:172.21.15.125:0/951566320' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:57:23.421 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[24927]: audit 2021-03-24T21:57:23.422 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[24927]: 2021-03-24T21:57:22.561947+0000 mon.a (mon.0) 919 : audit 2021-03-24T21:57:23.422 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[24927]: [INF] from='client.34973 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:57:23.422 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:57:23.422 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[24927]: -03-24T21:57:23.157101+0000 mon.a (mon.0) 920 : cluster [INF] 2021-03-24T21:57:23.423 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[24927]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T21:57:23.423 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:57:23.423 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[24927]: -24T21:57:23.157185+0000 mon.a (mon.0) 921 : cluster [INF] Cluster is now healthy 2021-03-24T21:57:23.423 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[24927]: 2021-03-24T21:57:23.424 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T21:57:23.424 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[24927]: 24T21:57:23.170628+0000 mon.a (mon.0) 922 : audit [INF] 2021-03-24T21:57:23.424 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[24927]: from='client.34973 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T21:57:23.424 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[24927]: cluster 2021-03-24T21:57:23.170773 2021-03-24T21:57:23.424 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:23 smithi125 conmon[24927]: +0000 mon.a (mon.0) 923 : cluster [DBG] osdmap e273: 8 total, 8 up, 6 in 2021-03-24T21:57:23.459 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:23.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:23.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:57:23.535 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.3 dump_ops_in_flight 2021-03-24T21:57:23.564 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:23.599 INFO:tasks.rados.rados.0.smithi125.stdout:901: finishing write tid 3 to smithi12556470-21 2021-03-24T21:57:23.600 INFO:tasks.rados.rados.0.smithi125.stdout:901: finishing write tid 4 to smithi12556470-21 2021-03-24T21:57:23.600 INFO:tasks.rados.rados.0.smithi125.stdout:899: done (3 left) 2021-03-24T21:57:23.600 INFO:tasks.rados.rados.0.smithi125.stdout:900: done (2 left) 2021-03-24T21:57:23.600 INFO:tasks.rados.rados.0.smithi125.stdout:903: read oid 28 snap 83 2021-03-24T21:57:23.600 INFO:tasks.rados.rados.0.smithi125.stdout:903: expect (ObjNum 308 snap 75 seq_num 308) 2021-03-24T21:57:23.601 INFO:tasks.rados.rados.0.smithi125.stdout:904: rollback oid 20 current snap is 84 2021-03-24T21:57:23.601 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 20 to 83 2021-03-24T21:57:23.601 INFO:tasks.rados.rados.0.smithi125.stdout:901: finishing write tid 5 to smithi12556470-21 2021-03-24T21:57:23.601 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 21 v 266 (ObjNum 334 snap 84 seq_num 334) dirty exists 2021-03-24T21:57:23.601 INFO:tasks.rados.rados.0.smithi125.stdout:901: left oid 21 (ObjNum 334 snap 84 seq_num 334) 2021-03-24T21:57:23.602 INFO:tasks.rados.rados.0.smithi125.stdout:902: finishing rollback tid 0 to smithi12556470-2 2021-03-24T21:57:23.602 INFO:tasks.rados.rados.0.smithi125.stdout:902: finishing rollback tid 1 to smithi12556470-2 2021-03-24T21:57:23.602 INFO:tasks.rados.rados.0.smithi125.stdout:902: finishing rollback tid 2 to smithi12556470-2 2021-03-24T21:57:23.602 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 2 v 245 (ObjNum 284 snap 70 seq_num 284) dirty exists 2021-03-24T21:57:23.603 INFO:tasks.rados.rados.0.smithi125.stdout:901: done (3 left) 2021-03-24T21:57:23.603 INFO:tasks.rados.rados.0.smithi125.stdout:902: done (2 left) 2021-03-24T21:57:23.603 INFO:tasks.rados.rados.0.smithi125.stdout:905: snap_create 2021-03-24T21:57:23.604 INFO:tasks.rados.rados.0.smithi125.stdout:904: finishing rollback tid 0 to smithi12556470-20 2021-03-24T21:57:23.604 INFO:tasks.rados.rados.0.smithi125.stdout:904: finishing rollback tid 1 to smithi12556470-20 2021-03-24T21:57:23.604 INFO:tasks.rados.rados.0.smithi125.stdout:904: finishing rollback tid 2 to smithi12556470-20 2021-03-24T21:57:23.605 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 20 v 191 (ObjNum 298 snap 74 seq_num 298) dirty exists 2021-03-24T21:57:23.608 INFO:tasks.rados.rados.0.smithi125.stdout:903: expect (ObjNum 308 snap 75 seq_num 308) 2021-03-24T21:57:23.629 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:23.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:23.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:57:23.736 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:23.789 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:23.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:23.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:57:23.894 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:23.957 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:23.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:23.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:57:24.062 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:24.112 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:24.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:24.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:57:24.219 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:24.279 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:24.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:24.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:57:24.285 INFO:tasks.rados.rados.0.smithi125.stdout:903: done (2 left) 2021-03-24T21:57:24.285 INFO:tasks.rados.rados.0.smithi125.stdout:904: done (1 left) 2021-03-24T21:57:24.286 INFO:tasks.rados.rados.0.smithi125.stdout:905: done (0 left) 2021-03-24T21:57:24.286 INFO:tasks.rados.rados.0.smithi125.stdout:906: snap_create 2021-03-24T21:57:24.289 INFO:teuthology.orchestra.run.smithi125.stderr:set pool 7 pg_num to 36 2021-03-24T21:57:24.371 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:24 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:57:24.372 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:24 smithi125 conmon[19936]: 57:22.669291+0000 osd.4 (osd.4) 74 : cluster [DBG] 7.16 scrub ok 2021-03-24T21:57:24.372 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:24 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T21:57:24.372 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:24 smithi125 conmon[19936]: T21:57:22 2021-03-24T21:57:24.373 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:24 smithi125 conmon[19936]: .771204+0000 mgr.y 2021-03-24T21:57:24.373 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:24 smithi125 conmon[19936]: (mgr.14144) 403 2021-03-24T21:57:24.373 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:24 smithi125 conmon[19936]: : cluster [DBG] pgmap v571: 158 pgs: 158 active+clean; 155 MiB data, 534 MiB used, 536 GiB / 536 GiB avail; 19 MiB/s rd, 4.0 MiB/s wr, 30 op/s 2021-03-24T21:57:24.374 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:24 smithi125 conmon[19936]: 2021-03-24T21:57:24.374 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:24 smithi125 conmon[19936]: cluster 2021-03-24T21:57:24.374 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:24 smithi125 conmon[19936]: 2021-03-24T21 2021-03-24T21:57:24.375 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:24 smithi125 conmon[19936]: :57:23.071385+0000 2021-03-24T21:57:24.375 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:24 smithi125 conmon[19936]: osd.2 (osd.2) 2021-03-24T21:57:24.375 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:24 smithi125 conmon[19936]: 54 : cluster [DBG] 2.1 scrub ok 2021-03-24T21:57:24.376 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:24 smithi125 conmon[19936]: cluster 2021-03-24T21:57:24.376 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:24 smithi125 conmon[19936]: 2021-03-24T21:57:23 2021-03-24T21:57:24.376 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:24 smithi125 conmon[19936]: .176743+0000 osd.5 (osd.5) 72 : cluster [DBG] 6.d scrub ok 2021-03-24T21:57:24.376 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:24 smithi125 conmon[19936]: audit 2021-03 2021-03-24T21:57:24.377 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:24 smithi125 conmon[19936]: -24T21:57:23.847272+0000 mon.a (mon.0) 924 : audit [INF] 2021-03-24T21:57:24.377 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:24 smithi125 conmon[19936]: from='client.? v1:172.21.15.125:0/2424679360' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"}]: dispatch 2021-03-24T21:57:24.377 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:24 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:57:24.377 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:24 smithi125 conmon[24927]: T21:57:22.669291+0000 osd.4 (osd. 2021-03-24T21:57:24.378 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:24 smithi125 conmon[24927]: 4) 74 : cluster [DBG] 2021-03-24T21:57:24.378 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:24 smithi125 conmon[24927]: 7.16 scrub ok 2021-03-24T21:57:24.378 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:24 smithi125 conmon[24927]: cluster 2021-03-24T21:57:24.378 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:24 smithi125 conmon[24927]: 2021-03-24 2021-03-24T21:57:24.378 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:24 smithi125 conmon[24927]: T21:57:22 2021-03-24T21:57:24.379 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:24 smithi125 conmon[24927]: .771204+0000 mgr.y 2021-03-24T21:57:24.379 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:24 smithi125 conmon[24927]: (mgr.14144 2021-03-24T21:57:24.379 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:24 smithi125 conmon[24927]: ) 403 : cluster 2021-03-24T21:57:24.379 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:24 smithi125 conmon[24927]: [DBG] pgmap v571: 158 pgs: 158 active+clean; 155 MiB data, 534 MiB used, 536 GiB / 536 GiB avail; 19 MiB/s rd, 4.0 MiB/s wr, 30 op/s 2021-03-24T21:57:24.379 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:24 smithi125 conmon[24927]: cluster 2021-03-24T21:57:24.380 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:24 smithi125 conmon[24927]: 2021-03-24T21:57:23 2021-03-24T21:57:24.380 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:24 smithi125 conmon[24927]: .071385+0000 osd.2 (osd. 2021-03-24T21:57:24.380 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:24 smithi125 conmon[24927]: 2) 54 : cluster [DBG] 2021-03-24T21:57:24.380 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:24 smithi125 conmon[24927]: 2.1 scrub ok 2021-03-24T21:57:24.380 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:24 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:57:24.381 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:24 smithi125 conmon[24927]: 24T21:57:23 2021-03-24T21:57:24.381 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:24 smithi125 conmon[24927]: . 2021-03-24T21:57:24.381 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:24 smithi125 conmon[24927]: 176743+0000 2021-03-24T21:57:24.381 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:24 smithi125 conmon[24927]: osd.5 (osd.5) 72 : 2021-03-24T21:57:24.381 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:24 smithi125 conmon[24927]: cluster [DBG] 6.d scrub ok 2021-03-24T21:57:24.382 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:24 smithi125 conmon[24927]: audit 2021-03-24T21:57:24.382 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:24 smithi125 conmon[24927]: 2021-03-24T21 2021-03-24T21:57:24.382 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:24 smithi125 conmon[24927]: :57:23.847272 2021-03-24T21:57:24.382 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:24 smithi125 conmon[24927]: +0000 mon.a (mon. 2021-03-24T21:57:24.383 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:24 smithi125 conmon[24927]: 0) 924 : audit 2021-03-24T21:57:24.383 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:24 smithi125 conmon[24927]: [INF] from='client.? v1:172.21.15.125:0/2424679360' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"}]: dispatch 2021-03-24T21:57:24.384 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:24.440 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:24.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:24.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:57:24.546 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:24.560 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:24 smithi142 conmon[20125]: cluster 2021-03-24T21:57:22.669291+0000 osd.4 (osd.4) 74 : cluster 2021-03-24T21:57:24.560 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:24 smithi142 conmon[20125]: [DBG] 7.16 scrub ok 2021-03-24T21:57:24.561 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:24 smithi142 conmon[20125]: cluster 2021-03-24T21:57:22.771204+0000 mgr.y (mgr.14144) 403 2021-03-24T21:57:24.561 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:24 smithi142 conmon[20125]: : cluster [DBG] pgmap v571: 158 pgs: 158 active+clean; 155 MiB data, 534 MiB used, 536 GiB / 536 GiB avail; 19 MiB/s rd, 4.0 MiB/s wr, 30 op/s 2021-03-24T21:57:24.561 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:24 smithi142 conmon[20125]: cluster 2021-03-24T21:57:24.561 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:24 smithi142 conmon[20125]: 2021-03-24T21:57:23.071385+0000 osd.2 2021-03-24T21:57:24.562 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:24 smithi142 conmon[20125]: (osd.2) 54 : cluster [DBG] 2.1 scrub ok 2021-03-24T21:57:24.562 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:24 smithi142 conmon[20125]: 2021-03-24T21:57:24.562 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:24 smithi142 conmon[20125]: cluster 2021-03-24T21:57: 2021-03-24T21:57:24.562 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:24 smithi142 conmon[20125]: 23.176743+0000 osd.5 (osd.5) 72 : 2021-03-24T21:57:24.562 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:24 smithi142 conmon[20125]: cluster [DBG] 6.d scrub ok 2021-03-24T21:57:24.563 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:24 smithi142 conmon[20125]: audit 2021 2021-03-24T21:57:24.563 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:24 smithi142 conmon[20125]: -03-24T21:57:23.847272+0000 mon.a ( 2021-03-24T21:57:24.563 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:24 smithi142 conmon[20125]: mon.0) 924 : audit [INF] 2021-03-24T21:57:24.563 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:24 smithi142 conmon[20125]: from='client.? v1:172.21.15.125:0/2424679360' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"}]: dispatch 2021-03-24T21:57:24.602 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:24.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:24.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:57:24.710 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:24.765 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:24.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:24.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:57:24.872 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:24.928 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:24.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:24.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:25.038 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:25.095 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:25.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:25.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:57:25.152 INFO:tasks.rados.rados.0.smithi125.stdout:906: done (0 left) 2021-03-24T21:57:25.153 INFO:tasks.rados.rados.0.smithi125.stdout:907: write oid 13 current snap is 86 2021-03-24T21:57:25.153 INFO:tasks.rados.rados.0.smithi125.stdout:907: seq_num 335 ranges {430345=727947,1570056=727439,2699879=469889} 2021-03-24T21:57:25.162 INFO:tasks.rados.rados.0.smithi125.stdout:907: writing smithi12556470-13 from 430345 to 1158292 tid 1 2021-03-24T21:57:25.170 INFO:tasks.rados.rados.0.smithi125.stdout:907: writing smithi12556470-13 from 1570056 to 2297495 tid 2 2021-03-24T21:57:25.176 INFO:tasks.rados.rados.0.smithi125.stdout:907: writing smithi12556470-13 from 2699879 to 3169768 tid 3 2021-03-24T21:57:25.176 INFO:tasks.rados.rados.0.smithi125.stdout:908: snap_remove snap 85 2021-03-24T21:57:25.200 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:25.246 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:25.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:25.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:57:25.352 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:25.407 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:25.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:25.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:57:25.440 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:25 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:57:25.441 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:25 smithi125 conmon[19936]: :57:24.209098+0000 osd.7 (osd.7) 63 : cluster [DBG] 7.11 scrub ok 2021-03-24T21:57:25.441 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:25 smithi125 conmon[19936]: audit 2021-03-24 2021-03-24T21:57:25.441 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:25 smithi125 conmon[19936]: T21:57:24.284139+0000 mon.a ( 2021-03-24T21:57:25.441 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:25 smithi125 conmon[19936]: mon.0) 925 : audit [INF] from='client.? v1:172.21.15.125:0/2424679360' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"}]': finished 2021-03-24T21:57:25.442 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:25 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:57:25.442 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:25 smithi125 conmon[19936]: 03-24T21:57:24.284231+0000 mon.a 2021-03-24T21:57:25.442 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:25 smithi125 conmon[19936]: (mon.0) 926 : cluster [DBG] osdmap e274: 8 total, 8 up, 6 in 2021-03-24T21:57:25.442 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:25 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:57:25.443 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:25 smithi125 conmon[19936]: 21:57:24.499082+0000 osd.3 (osd 2021-03-24T21:57:25.443 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:25 smithi125 conmon[19936]: .3) 70 : cluster [DBG] 4.f deep-scrub ok 2021-03-24T21:57:25.443 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:25 smithi125 conmon[19936]: audit 2021 2021-03-24T21:57:25.443 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:25 smithi125 conmon[19936]: -03-24T21:57 2021-03-24T21:57:25.444 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:25 smithi125 conmon[19936]: :24.773921+0000 mon.a (mon 2021-03-24T21:57:25.444 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:25 smithi125 conmon[19936]: .0) 927 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"}]: dispatch 2021-03-24T21:57:25.444 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:25 smithi125 conmon[19936]: audit 2021 2021-03-24T21:57:25.444 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:25 smithi125 conmon[19936]: -03-24T21:57: 2021-03-24T21:57:25.444 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:25 smithi125 conmon[19936]: 25.149255+0000 mon.a (mon.0) 928 2021-03-24T21:57:25.445 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:25 smithi125 conmon[19936]: : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"}]': finished 2021-03-24T21:57:25.445 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:25 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:57:25.445 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:25 smithi125 conmon[19936]: 24T21:57:25.149470+0000 2021-03-24T21:57:25.445 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:25 smithi125 conmon[19936]: mon.a (mon.0) 929 : cluster [DBG] 2021-03-24T21:57:25.446 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:25 smithi125 conmon[19936]: osdmap e275: 8 total, 8 up, 6 in 2021-03-24T21:57:25.446 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:25 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:57:25.446 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:25 smithi125 conmon[24927]: 57:24.209098+0000 osd.7 (osd.7) 63 : cluster [DBG] 7.11 scrub ok 2021-03-24T21:57:25.447 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:25 smithi125 conmon[24927]: audit 2021-03-24T21:57:24 2021-03-24T21:57:25.447 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:25 smithi125 conmon[24927]: .284139+0000 mon.a (mon.0) 925 : audit 2021-03-24T21:57:25.447 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:25 smithi125 conmon[24927]: [INF] from='client.? v1:172.21.15.125:0/2424679360' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"}]': finished 2021-03-24T21:57:25.447 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:25 smithi125 conmon[24927]: cluster 2021-03-24T21:57:24 2021-03-24T21:57:25.448 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:25 smithi125 conmon[24927]: .284231+0000 mon.a (mon.0) 926 : cluster 2021-03-24T21:57:25.448 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:25 smithi125 conmon[24927]: [DBG] osdmap e274: 8 total, 8 up, 6 in 2021-03-24T21:57:25.448 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:25 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:57:25.448 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:25 smithi125 conmon[24927]: T21:57:24.499082+0000 osd.3 (osd. 2021-03-24T21:57:25.448 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:25 smithi125 conmon[24927]: 3) 70 : cluster [DBG] 4.f deep-scrub ok 2021-03-24T21:57:25.449 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:25 smithi125 conmon[24927]: audit 2021- 2021-03-24T21:57:25.449 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:25 smithi125 conmon[24927]: 03-24T21:57:24.773921+0000 mon.a 2021-03-24T21:57:25.449 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:25 smithi125 conmon[24927]: (mon.0) 927 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"}]: dispatch 2021-03-24T21:57:25.449 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:25 smithi125 conmon[24927]: audit 2021- 2021-03-24T21:57:25.450 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:25 smithi125 conmon[24927]: 03-24T21:57:25.149255+0000 mon.a 2021-03-24T21:57:25.450 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:25 smithi125 conmon[24927]: (mon.0) 928 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"}]': finished 2021-03-24T21:57:25.450 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:25 smithi125 conmon[24927]: cluster 2021-03-24T21:57:25.450 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:25 smithi125 conmon[24927]: 2021-03-24T21:57:25.149470 2021-03-24T21:57:25.450 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:25 smithi125 conmon[24927]: +0000 mon.a (mon.0) 929 : cluster [DBG] 2021-03-24T21:57:25.451 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:25 smithi125 conmon[24927]: osdmap e275: 8 total, 8 up, 6 in 2021-03-24T21:57:25.512 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:25.569 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:25.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:25.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:57:25.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:25 smithi142 conmon[20125]: cluster 2021-03-24T21:57 2021-03-24T21:57:25.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:25 smithi142 conmon[20125]: :24.209098+0000 osd.7 (osd.7) 63 : cluster [DBG] 7.11 scrub ok 2021-03-24T21:57:25.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:25 smithi142 conmon[20125]: audit 2021-03-24T21: 2021-03-24T21:57:25.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:25 smithi142 conmon[20125]: 57:24.284139+0000 mon.a (mon.0) 925 : audit [INF] from='client.? v1:172.21.15.125:0/2424679360' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"}]': finished 2021-03-24T21:57:25.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:25 smithi142 conmon[20125]: cluster 2021-03-24T21:57:24.284231 2021-03-24T21:57:25.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:25 smithi142 conmon[20125]: +0000 mon.a (mon.0) 926 : cluster [DBG] osdmap e274: 8 total, 8 up, 6 in 2021-03-24T21:57:25.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:25 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:57:25.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:25 smithi142 conmon[20125]: 57:24.499082+0000 osd.3 (osd.3) 70 : cluster 2021-03-24T21:57:25.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:25 smithi142 conmon[20125]: [DBG] 4.f deep-scrub ok 2021-03-24T21:57:25.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:25 smithi142 conmon[20125]: audit 2021-03-24T21:57:24.773921+0000 mon.a ( 2021-03-24T21:57:25.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:25 smithi142 conmon[20125]: mon.0) 927 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"}]: dispatch 2021-03-24T21:57:25.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:25 smithi142 conmon[20125]: audit 2021-03-24T21:57 2021-03-24T21:57:25.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:25 smithi142 conmon[20125]: :25.149255+0000 mon.a (mon.0) 928 : audit 2021-03-24T21:57:25.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:25 smithi142 conmon[20125]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"}]': finished 2021-03-24T21:57:25.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:25 smithi142 conmon[20125]: cluster 2021-03-24T21:57:25.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:25 smithi142 conmon[20125]: 2021-03-24T21:57:25.149470+0000 mon.a (mon.0) 929 2021-03-24T21:57:25.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:25 smithi142 conmon[20125]: : cluster [DBG] osdmap e275: 8 total, 8 up, 6 in 2021-03-24T21:57:25.677 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:25.737 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:25.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:25.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:57:25.844 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:25.901 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:25.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:25.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:57:26.007 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:26.068 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:26.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:26.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:57:26.163 INFO:tasks.rados.rados.0.smithi125.stdout:908: done (1 left) 2021-03-24T21:57:26.164 INFO:tasks.rados.rados.0.smithi125.stdout:909: write (excl) oid 10 current snap is 86 2021-03-24T21:57:26.164 INFO:tasks.rados.rados.0.smithi125.stdout:909: seq_num 336 ranges {447152=403196,1512185=587450,2727302=681161} 2021-03-24T21:57:26.164 INFO:tasks.rados.rados.0.smithi125.stdout:909: writing smithi12556470-10 from 447152 to 850348 tid 1 2021-03-24T21:57:26.173 INFO:tasks.rados.rados.0.smithi125.stdout:909: writing smithi12556470-10 from 1512185 to 2099635 tid 2 2021-03-24T21:57:26.173 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:26.174 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:57:26.186 INFO:tasks.rados.rados.0.smithi125.stdout:909: writing smithi12556470-10 from 2727302 to 3408463 tid 3 2021-03-24T21:57:26.186 INFO:tasks.rados.rados.0.smithi125.stdout:910: write (excl) oid 22 current snap is 86 2021-03-24T21:57:26.186 INFO:tasks.rados.rados.0.smithi125.stdout:910: seq_num 337 ranges {463959=478445,1570667=731108,3038372=492433} 2021-03-24T21:57:26.193 INFO:tasks.rados.rados.0.smithi125.stdout:910: writing smithi12556470-22 from 463959 to 942404 tid 1 2021-03-24T21:57:26.201 INFO:tasks.rados.rados.0.smithi125.stdout:910: writing smithi12556470-22 from 1570667 to 2301775 tid 2 2021-03-24T21:57:26.209 INFO:tasks.rados.rados.0.smithi125.stdout:910: writing smithi12556470-22 from 3038372 to 3530805 tid 3 2021-03-24T21:57:26.209 INFO:tasks.rados.rados.0.smithi125.stdout:911: snap_remove snap 81 2021-03-24T21:57:26.231 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:26.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:26.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:57:26.244 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T21:57:26.270 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T21:57:26.289 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T21:57:26.317 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T21:57:26.337 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:26.341 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T21:57:26.366 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T21:57:26.388 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:26.391 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:26 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:57:26.391 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:26 smithi125 conmon[24927]: :57:24.772031+0000 mgr.y (mgr.14144) 404 : cluster [DBG] pgmap v574: 158 pgs: 158 active+clean; 155 MiB data, 523 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:57:26.391 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:26 smithi125 conmon[24927]: cluster 2021-03-24T21:57:26.391 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:26 smithi125 conmon[24927]: 2021-03-24T21:57:25.266192+0000 2021-03-24T21:57:26.392 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:26 smithi125 conmon[24927]: osd.0 (osd.0) 74 : cluster [DBG] 7.9 scrub ok 2021-03-24T21:57:26.392 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:26 smithi125 conmon[24927]: 2021-03-24T21:57:26.392 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:26 smithi125 conmon[24927]: cluster 2021-03-24T21:57:25 2021-03-24T21:57:26.392 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:26 smithi125 conmon[24927]: .462933+0000 osd.3 (osd.3) 71 : cluster 2021-03-24T21:57:26.392 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:26 smithi125 conmon[24927]: [DBG] 5.0 scrub ok 2021-03-24T21:57:26.393 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:26 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:57:26.393 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:26 smithi125 conmon[24927]: 21:57:26.153947+0000 mon.a (mon.0 2021-03-24T21:57:26.393 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:26 smithi125 conmon[24927]: ) 930 : cluster [DBG] osdmap e276: 8 total, 8 up, 6 in 2021-03-24T21:57:26.393 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:26 smithi125 conmon[24927]: audit 2021-03 2021-03-24T21:57:26.393 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:26 smithi125 conmon[24927]: -24T21:57:26.212992+0000 mon.a ( 2021-03-24T21:57:26.394 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:26 smithi125 conmon[24927]: mon.0) 931 : audit [DBG] from='client.? v1:172.21.15.125:0/2847493973' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:57:26.394 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:26 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:57:26.395 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:26 smithi125 conmon[19936]: -24T21:57:24.772031+0000 mgr.y (mgr.14144) 404 : cluster [DBG] pgmap v574: 158 pgs: 158 active+clean; 155 MiB data, 523 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:57:26.395 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:26 smithi125 conmon[19936]: cluster 2021-03-24T21:57:26.395 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:26 smithi125 conmon[19936]: 2021-03 2021-03-24T21:57:26.396 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:26 smithi125 conmon[19936]: -24T21:57:25. 2021-03-24T21:57:26.396 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:26 smithi125 conmon[19936]: 266192+0000 osd.0 (osd.0 2021-03-24T21:57:26.396 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:26 smithi125 conmon[19936]: ) 74 : cluster [DBG] 7.9 scrub ok 2021-03-24T21:57:26.397 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:26 smithi125 conmon[19936]: cluster 2021-03-24T21:57:26.397 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:26 smithi125 conmon[19936]: 2021-03-24T21 2021-03-24T21:57:26.397 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:26 smithi125 conmon[19936]: :57:25.462933+0000 2021-03-24T21:57:26.397 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:26 smithi125 conmon[19936]: osd.3 (osd.3) 71 : 2021-03-24T21:57:26.397 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:26 smithi125 conmon[19936]: cluster [DBG] 5.0 scrub ok 2021-03-24T21:57:26.398 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:26 smithi125 conmon[19936]: cluster 2021-03-24T21:57:26.398 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:26 smithi125 conmon[19936]: 2021-03-24T21 2021-03-24T21:57:26.398 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:26 smithi125 conmon[19936]: :57:26.153947+0000 2021-03-24T21:57:26.398 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:26 smithi125 conmon[19936]: mon.a (mon.0) 930 : cluster 2021-03-24T21:57:26.399 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:26 smithi125 conmon[19936]: [DBG] osdmap e276: 8 total, 8 up, 6 in 2021-03-24T21:57:26.399 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:26 smithi125 conmon[19936]: audit 2021-03-24T21:57:26.399 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:26 smithi125 conmon[19936]: 2021-03-24T 2021-03-24T21:57:26.399 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:26 smithi125 conmon[19936]: 21:57:26.212992+0000 2021-03-24T21:57:26.399 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:26 smithi125 conmon[19936]: mon.a (mon.0) 931 : 2021-03-24T21:57:26.400 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:26 smithi125 conmon[19936]: audit [DBG] from='client.? v1:172.21.15.125:0/2847493973' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:57:26.400 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T21:57:26.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:26.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:57:26.423 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T21:57:26.425 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-24T21:57:26.425 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-24T21:57:26.425 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-24T21:57:26.426 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-24T21:57:26.426 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-24T21:57:26.426 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-24T21:57:26.426 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-24T21:57:26.427 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-24T21:57:26.502 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:26.553 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:26.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:26.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:57:26.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:26 smithi142 conmon[20125]: cluster 2021-03-24T21:57:24 2021-03-24T21:57:26.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:26 smithi142 conmon[20125]: .772031+0000 mgr.y (mgr.14144) 404 : cluster [DBG] pgmap v574: 158 pgs: 158 active+clean; 155 MiB data, 523 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:57:26.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:26 smithi142 conmon[20125]: cluster 2021-03-24T21:57:25.266192+0000 osd.0 ( 2021-03-24T21:57:26.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:26 smithi142 conmon[20125]: osd.0) 74 : cluster [DBG] 7.9 scrub ok 2021-03-24T21:57:26.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:26 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:57:26.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:26 smithi142 conmon[20125]: 03-24T21:57:25.462933+0000 osd.3 (osd.3) 71 : cluster [DBG] 2021-03-24T21:57:26.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:26 smithi142 conmon[20125]: 5.0 scrub ok 2021-03-24T21:57:26.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:26 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:57:26.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:26 smithi142 conmon[20125]: -03-24T21:57:26.153947+0000 mon.a (mon.0) 930 : cluster [DBG] 2021-03-24T21:57:26.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:26 smithi142 conmon[20125]: osdmap e276: 8 total, 8 up, 6 in 2021-03-24T21:57:26.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:26 smithi142 conmon[20125]: audit 2021-03- 2021-03-24T21:57:26.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:26 smithi142 conmon[20125]: 24T21:57:26.212992+0000 mon.a (mon.0) 931 : audit [DBG] 2021-03-24T21:57:26.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:26 smithi142 conmon[20125]: from='client.? v1:172.21.15.125:0/2847493973' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:57:26.659 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:26.723 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:26.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:26.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:57:26.829 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:26.892 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:26.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:26.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:57:27.000 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:27.052 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:27.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:27.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:27.157 INFO:tasks.rados.rados.0.smithi125.stdout:911: done (3 left) 2021-03-24T21:57:27.158 INFO:tasks.rados.rados.0.smithi125.stdout:912: read oid 12 snap -1 2021-03-24T21:57:27.158 INFO:tasks.rados.rados.0.smithi125.stdout:912: expect (ObjNum 228 snap 50 seq_num 228) 2021-03-24T21:57:27.158 INFO:tasks.rados.rados.0.smithi125.stdout:913: snap_remove snap 83 2021-03-24T21:57:27.159 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:27.208 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:27.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:27.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:27.317 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:27.374 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:27.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:27.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:57:27.481 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:27.491 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:27 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:57:27.492 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:27 smithi125 conmon[19936]: :57:26.129908+0000 osd.7 (osd.7) 64 : cluster [DBG] 6.1c scrub ok 2021-03-24T21:57:27.492 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:27 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:57:27.492 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:27 smithi125 conmon[19936]: -24T21:57:26.257679+0000 osd.0 (osd.0) 75 : cluster 2021-03-24T21:57:27.492 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:27 smithi125 conmon[19936]: [DBG] 4.6 scrub ok 2021-03-24T21:57:27.493 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:27 smithi125 conmon[19936]: cluster 2021-03-24T21:57:26 2021-03-24T21:57:27.493 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:27 smithi125 conmon[19936]: .417606+0000 osd.3 (osd.3) 72 : cluster 2021-03-24T21:57:27.493 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:27 smithi125 conmon[19936]: [DBG] 3.19 deep-scrub ok 2021-03-24T21:57:27.493 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:27 smithi125 conmon[19936]: cluster 2021-03-24T21:57:27.493 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:27 smithi125 conmon[19936]: 2021-03-24T21:57 2021-03-24T21:57:27.494 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:27 smithi125 conmon[19936]: :26.642874+0000 osd.4 (osd. 2021-03-24T21:57:27.494 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:27 smithi125 conmon[19936]: 4) 75 : cluster [DBG] 5.1d scrub ok 2021-03-24T21:57:27.494 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:27 smithi125 conmon[19936]: audit 2021-03-24T21:57:26 2021-03-24T21:57:27.494 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:27 smithi125 conmon[19936]: .774732+0000 mon.a (mon.0) 932 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"}]: dispatch 2021-03-24T21:57:27.495 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:27 smithi125 conmon[19936]: audit 2021-03-24T21:57:27. 2021-03-24T21:57:27.495 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:27 smithi125 conmon[19936]: 154642+0000 mon.a (mon.0) 933 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"}]': finished 2021-03-24T21:57:27.495 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:27 smithi125 conmon[19936]: 2021-03-24T21:57:27.495 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:27 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T21:57:27.495 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:27 smithi125 conmon[19936]: T21:57:27.155416+0000 mon.a (mon.0) 934 : cluster 2021-03-24T21:57:27.496 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:27 smithi125 conmon[19936]: [DBG] osdmap e277: 8 total, 8 up, 6 in 2021-03-24T21:57:27.496 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:27 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:57:27.496 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:27 smithi125 conmon[24927]: 57:26.129908+0000 osd.7 (osd.7) 64 : cluster [DBG] 6.1c scrub ok 2021-03-24T21:57:27.496 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:27 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:57:27.497 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:27 smithi125 conmon[24927]: 03-24T21:57:26.257679+0000 osd.0 2021-03-24T21:57:27.497 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:27 smithi125 conmon[24927]: (osd.0) 75 : cluster [DBG] 4.6 scrub ok 2021-03-24T21:57:27.497 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:27 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:57:27.497 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:27 smithi125 conmon[24927]: -03-24T21:57:26.417606+0000 osd.3 (osd 2021-03-24T21:57:27.498 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:27 smithi125 conmon[24927]: .3) 72 : cluster [DBG] 3.19 deep-scrub ok 2021-03-24T21:57:27.498 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:27 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:57:27.498 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:27 smithi125 conmon[24927]: 03-24T21:57:26.642874+0000 osd.4 2021-03-24T21:57:27.498 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:27 smithi125 conmon[24927]: (osd.4) 75 : cluster [DBG] 5.1d scrub ok 2021-03-24T21:57:27.499 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:27 smithi125 conmon[24927]: audit 2021 2021-03-24T21:57:27.499 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:27 smithi125 conmon[24927]: -03-24T21:57:26.774732+0000 mon.a 2021-03-24T21:57:27.499 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:27 smithi125 conmon[24927]: (mon.0) 932 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"}]: dispatch 2021-03-24T21:57:27.499 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:27 smithi125 conmon[24927]: audit 2021 2021-03-24T21:57:27.499 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:27 smithi125 conmon[24927]: -03-24T21:57:27.154642+0000 2021-03-24T21:57:27.500 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:27 smithi125 conmon[24927]: mon.a (mon.0) 933 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"}]': finished 2021-03-24T21:57:27.500 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:27 smithi125 conmon[24927]: 2021-03-24T21:57:27.500 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:27 smithi125 conmon[24927]: cluster 2021-03-24T21:57:27.500 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:27 smithi125 conmon[24927]: 2021-03-24T21:57:27.155416+0000 mon.a 2021-03-24T21:57:27.501 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:27 smithi125 conmon[24927]: (mon.0) 934 : cluster [DBG] osdmap e277: 8 total, 8 up, 6 in 2021-03-24T21:57:27.543 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:27.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:27.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:57:27.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:27 smithi142 conmon[20125]: cluster 2021-03-24T21:57:26 2021-03-24T21:57:27.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:27 smithi142 conmon[20125]: .129908+0000 osd.7 (osd.7) 64 : cluster [DBG] 6.1c scrub ok 2021-03-24T21:57:27.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:27 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:57:27.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:27 smithi142 conmon[20125]: T21:57:26.257679+0000 osd.0 (osd.0) 75 : cluster 2021-03-24T21:57:27.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:27 smithi142 conmon[20125]: [DBG] 4.6 scrub ok 2021-03-24T21:57:27.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:27 smithi142 conmon[20125]: cluster 2021-03-24T21:57 2021-03-24T21:57:27.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:27 smithi142 conmon[20125]: :26.417606+0000 osd.3 (osd. 2021-03-24T21:57:27.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:27 smithi142 conmon[20125]: 3) 72 : cluster [DBG] 3.19 deep-scrub ok 2021-03-24T21:57:27.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:27 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:57:27.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:27 smithi142 conmon[20125]: -03-24T21:57:26.642874 2021-03-24T21:57:27.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:27 smithi142 conmon[20125]: +0000 osd.4 (osd.4) 75 : cluster [DBG] 2021-03-24T21:57:27.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:27 smithi142 conmon[20125]: 5.1d scrub ok 2021-03-24T21:57:27.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:27 smithi142 conmon[20125]: audit 2021-03-24T21:57:26 2021-03-24T21:57:27.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:27 smithi142 conmon[20125]: .774732+0000 mon.a (mon.0) 932 : audit 2021-03-24T21:57:27.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:27 smithi142 conmon[20125]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"}]: dispatch 2021-03-24T21:57:27.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:27 smithi142 conmon[20125]: audit 2021-03-24T21:57:27.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:27 smithi142 conmon[20125]: 2021-03-24T21:57 2021-03-24T21:57:27.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:27 smithi142 conmon[20125]: :27.154642+0000 mon.a (mon.0) 933 2021-03-24T21:57:27.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:27 smithi142 conmon[20125]: : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"}]': finished 2021-03-24T21:57:27.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:27 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:57:27.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:27 smithi142 conmon[20125]: 21:57:27.155416+0000 mon.a (mon. 2021-03-24T21:57:27.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:27 smithi142 conmon[20125]: 0) 934 : cluster [DBG] osdmap e277: 8 total, 8 up, 6 in 2021-03-24T21:57:27.650 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:27.713 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:27.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:27.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:57:27.820 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:27.881 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:27.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:27.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:57:27.988 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:28.048 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:28.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:28.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:57:28.062 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd set noscrub 2021-03-24T21:57:28.155 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:28.165 INFO:tasks.rados.rados.0.smithi125.stdout:913: done (4 left) 2021-03-24T21:57:28.166 INFO:tasks.rados.rados.0.smithi125.stdout:914: write oid 1 current snap is 86 2021-03-24T21:57:28.166 INFO:tasks.rados.rados.0.smithi125.stdout:914: seq_num 338 ranges {480766=553694,1629149=591119,2782148=420058} 2021-03-24T21:57:28.173 INFO:tasks.rados.rados.0.smithi125.stdout:914: writing smithi12556470-1 from 480766 to 1034460 tid 1 2021-03-24T21:57:28.180 INFO:tasks.rados.rados.0.smithi125.stdout:914: writing smithi12556470-1 from 1629149 to 2220268 tid 2 2021-03-24T21:57:28.187 INFO:tasks.rados.rados.0.smithi125.stdout:914: writing smithi12556470-1 from 2782148 to 3202206 tid 3 2021-03-24T21:57:28.188 INFO:tasks.rados.rados.0.smithi125.stdout:909: finishing write tid 1 to smithi12556470-10 2021-03-24T21:57:28.188 INFO:tasks.rados.rados.0.smithi125.stdout:909: finishing write tid 2 to smithi12556470-10 2021-03-24T21:57:28.189 INFO:tasks.rados.rados.0.smithi125.stdout:907: finishing write tid 1 to smithi12556470-13 2021-03-24T21:57:28.189 INFO:tasks.rados.rados.0.smithi125.stdout:907: finishing write tid 2 to smithi12556470-13 2021-03-24T21:57:28.189 INFO:tasks.rados.rados.0.smithi125.stdout:907: finishing write tid 3 to smithi12556470-13 2021-03-24T21:57:28.189 INFO:tasks.rados.rados.0.smithi125.stdout:909: finishing write tid 3 to smithi12556470-10 2021-03-24T21:57:28.189 INFO:tasks.rados.rados.0.smithi125.stdout:909: finishing write tid 4 to smithi12556470-10 2021-03-24T21:57:28.190 INFO:tasks.rados.rados.0.smithi125.stdout:909: finishing write tid 5 to smithi12556470-10 2021-03-24T21:57:28.190 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 10 v 295 (ObjNum 336 snap 86 seq_num 336) dirty exists 2021-03-24T21:57:28.190 INFO:tasks.rados.rados.0.smithi125.stdout:909: left oid 10 (ObjNum 336 snap 86 seq_num 336) 2021-03-24T21:57:28.190 INFO:tasks.rados.rados.0.smithi125.stdout:910: finishing write tid 1 to smithi12556470-22 2021-03-24T21:57:28.191 INFO:tasks.rados.rados.0.smithi125.stdout:907: finishing write tid 4 to smithi12556470-13 2021-03-24T21:57:28.191 INFO:tasks.rados.rados.0.smithi125.stdout:907: finishing write tid 5 to smithi12556470-13 2021-03-24T21:57:28.191 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 13 v 254 (ObjNum 335 snap 86 seq_num 335) dirty exists 2021-03-24T21:57:28.191 INFO:tasks.rados.rados.0.smithi125.stdout:907: left oid 13 (ObjNum 335 snap 86 seq_num 335) 2021-03-24T21:57:28.191 INFO:tasks.rados.rados.0.smithi125.stdout:910: finishing write tid 2 to smithi12556470-22 2021-03-24T21:57:28.192 INFO:tasks.rados.rados.0.smithi125.stdout:910: finishing write tid 3 to smithi12556470-22 2021-03-24T21:57:28.192 INFO:tasks.rados.rados.0.smithi125.stdout:910: finishing write tid 4 to smithi12556470-22 2021-03-24T21:57:28.192 INFO:tasks.rados.rados.0.smithi125.stdout:910: finishing write tid 5 to smithi12556470-22 2021-03-24T21:57:28.192 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 22 v 270 (ObjNum 337 snap 86 seq_num 337) dirty exists 2021-03-24T21:57:28.193 INFO:tasks.rados.rados.0.smithi125.stdout:910: left oid 22 (ObjNum 337 snap 86 seq_num 337) 2021-03-24T21:57:28.193 INFO:tasks.rados.rados.0.smithi125.stdout:912: expect (ObjNum 228 snap 50 seq_num 228) 2021-03-24T21:57:28.220 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:28.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:28.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:57:28.326 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:28.376 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:28.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:28.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:57:28.443 INFO:tasks.rados.rados.0.smithi125.stdout:914: finishing write tid 1 to smithi12556470-1 2021-03-24T21:57:28.444 INFO:tasks.rados.rados.0.smithi125.stdout:914: finishing write tid 2 to smithi12556470-1 2021-03-24T21:57:28.444 INFO:tasks.rados.rados.0.smithi125.stdout:907: done (4 left) 2021-03-24T21:57:28.444 INFO:tasks.rados.rados.0.smithi125.stdout:909: done (3 left) 2021-03-24T21:57:28.444 INFO:tasks.rados.rados.0.smithi125.stdout:910: done (2 left) 2021-03-24T21:57:28.445 INFO:tasks.rados.rados.0.smithi125.stdout:912: done (1 left) 2021-03-24T21:57:28.445 INFO:tasks.rados.rados.0.smithi125.stdout:915: snap_remove snap 84 2021-03-24T21:57:28.482 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:28.495 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:28 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:57:28.496 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:28 smithi125 conmon[19936]: 21:57:26.772854+0000 mgr.y (mgr.14144) 405 : cluster [DBG] pgmap v577: 168 pgs: 9 unknown, 159 active+clean; 155 MiB data, 516 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:57:28.496 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:28 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:57:28.496 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:28 smithi125 conmon[19936]: 03-24T21:57 2021-03-24T21:57:28.496 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:28 smithi125 conmon[19936]: :27.105447+0000 osd.2 2021-03-24T21:57:28.497 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:28 smithi125 conmon[19936]: (osd.2) 55 2021-03-24T21:57:28.497 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:28 smithi125 conmon[19936]: : cluster [DBG] 7.e scrub ok 2021-03-24T21:57:28.497 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:28 smithi125 conmon[19936]: 2021-03-24T21:57:28.497 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:28 smithi125 conmon[19936]: cluster 2021-03-24T21:57:28.497 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:28 smithi125 conmon[19936]: 2021-03-24T 2021-03-24T21:57:28.498 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:28 smithi125 conmon[19936]: 21:57:28.158346 2021-03-24T21:57:28.498 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:28 smithi125 conmon[19936]: +0000 mon.a (mon. 2021-03-24T21:57:28.498 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:28 smithi125 conmon[19936]: 0) 935 : cluster 2021-03-24T21:57:28.498 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:28 smithi125 conmon[19936]: [DBG] osdmap e278: 8 total, 8 up, 6 in 2021-03-24T21:57:28.499 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:28 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:57:28.499 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:28 smithi125 conmon[24927]: 57:26.772854+0000 mgr.y (mgr.14144) 405 : cluster [DBG] pgmap v577: 168 pgs: 9 unknown, 159 active+clean; 155 MiB data, 516 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:57:28.499 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:28 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:57:28.499 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:28 smithi125 conmon[24927]: -03-24T21:57:27 2021-03-24T21:57:28.500 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:28 smithi125 conmon[24927]: .105447+0000 osd.2 (osd.2) 55 2021-03-24T21:57:28.500 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:28 smithi125 conmon[24927]: : cluster [DBG] 7.e scrub ok 2021-03-24T21:57:28.500 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:28 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:57:28.500 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:28 smithi125 conmon[24927]: -24T21:57:28.158346+0000 2021-03-24T21:57:28.500 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:28 smithi125 conmon[24927]: mon.a (mon.0) 935 : cluster 2021-03-24T21:57:28.501 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:28 smithi125 conmon[24927]: [DBG] osdmap e278: 8 total, 8 up, 6 in 2021-03-24T21:57:28.541 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:28.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:28.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:57:28.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:28 smithi142 conmon[20125]: cluster 2021-03-24T21:57:26.772854+0000 mgr.y (mgr.14144) 405 : cluster [DBG] pgmap v577: 168 pgs: 9 unknown, 159 active+clean; 155 MiB data, 516 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:57:28.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:28 smithi142 conmon[20125]: cluster 2021-03-24T21:57:27.105447+0000 osd.2 (osd.2) 55 : cluster [DBG] 7.e scrub ok 2021-03-24T21:57:28.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:28 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:57:28.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:28 smithi142 conmon[20125]: T21:57:28.158346+0000 mon.a (mon.0) 935 : cluster [DBG] osdmap e278: 8 total, 8 up, 6 in 2021-03-24T21:57:28.647 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:28.689 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:57:28.705 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:28.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:28.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:57:28.811 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:28.875 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:28.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:28.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:57:28.981 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:29.032 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:29.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:29.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:57:29.139 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:29.162 INFO:tasks.rados.rados.0.smithi125.stdout:915: done (1 left) 2021-03-24T21:57:29.162 INFO:tasks.rados.rados.0.smithi125.stdout:916: rollback oid 45 current snap is 86 2021-03-24T21:57:29.163 INFO:tasks.rados.rados.0.smithi125.stdout:916: done (1 left) 2021-03-24T21:57:29.163 INFO:tasks.rados.rados.0.smithi125.stdout:917: write oid 13 current snap is 86 2021-03-24T21:57:29.163 INFO:tasks.rados.rados.0.smithi125.stdout:917: seq_num 339 ranges {497573=628943,1687631=734777,3093218=604355} 2021-03-24T21:57:29.169 INFO:tasks.rados.rados.0.smithi125.stdout:917: writing smithi12556470-13 from 497573 to 1126516 tid 1 2021-03-24T21:57:29.182 INFO:tasks.rados.rados.0.smithi125.stdout:917: writing smithi12556470-13 from 1687631 to 2422408 tid 2 2021-03-24T21:57:29.191 INFO:tasks.rados.rados.0.smithi125.stdout:917: writing smithi12556470-13 from 3093218 to 3697573 tid 3 2021-03-24T21:57:29.192 INFO:tasks.rados.rados.0.smithi125.stdout:918: snap_create 2021-03-24T21:57:29.192 INFO:tasks.rados.rados.0.smithi125.stdout:914: finishing write tid 3 to smithi12556470-1 2021-03-24T21:57:29.193 INFO:tasks.rados.rados.0.smithi125.stdout:914: finishing write tid 4 to smithi12556470-1 2021-03-24T21:57:29.193 INFO:tasks.rados.rados.0.smithi125.stdout:914: finishing write tid 5 to smithi12556470-1 2021-03-24T21:57:29.193 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 1 v 304 (ObjNum 338 snap 86 seq_num 338) dirty exists 2021-03-24T21:57:29.193 INFO:tasks.rados.rados.0.smithi125.stdout:914: left oid 1 (ObjNum 338 snap 86 seq_num 338) 2021-03-24T21:57:29.194 INFO:tasks.rados.rados.0.smithi125.stdout:917: finishing write tid 1 to smithi12556470-13 2021-03-24T21:57:29.196 INFO:tasks.rados.rados.0.smithi125.stdout:917: finishing write tid 2 to smithi12556470-13 2021-03-24T21:57:29.197 INFO:tasks.rados.rados.0.smithi125.stdout:917: finishing write tid 3 to smithi12556470-13 2021-03-24T21:57:29.198 INFO:tasks.rados.rados.0.smithi125.stdout:917: finishing write tid 4 to smithi12556470-13 2021-03-24T21:57:29.200 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:29.200 INFO:tasks.rados.rados.0.smithi125.stdout:917: finishing write tid 5 to smithi12556470-13 2021-03-24T21:57:29.201 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 13 v 258 (ObjNum 339 snap 86 seq_num 339) dirty exists 2021-03-24T21:57:29.201 INFO:tasks.rados.rados.0.smithi125.stdout:917: left oid 13 (ObjNum 339 snap 86 seq_num 339) 2021-03-24T21:57:29.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:29.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:57:29.312 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:29.373 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:29.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:29.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:57:29.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: cluster 2021-03-24T21:57:29.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: 2021-03-24T21:57:29.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: 2021 2021-03-24T21:57:29.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: - 2021-03-24T21:57:29.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: 03 2021-03-24T21:57:29.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: - 2021-03-24T21:57:29.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: 24 2021-03-24T21:57:29.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: T 2021-03-24T21:57:29.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: 21 2021-03-24T21:57:29.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: : 2021-03-24T21:57:29.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: 57 2021-03-24T21:57:29.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: : 2021-03-24T21:57:29.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: 28 2021-03-24T21:57:29.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: . 2021-03-24T21:57:29.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: 119390 2021-03-24T21:57:29.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: +0000 2021-03-24T21:57:29.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: 2021-03-24T21:57:29.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: osd.5 2021-03-24T21:57:29.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: ( 2021-03-24T21:57:29.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: osd 2021-03-24T21:57:29.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: . 2021-03-24T21:57:29.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: 5 2021-03-24T21:57:29.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: ) 2021-03-24T21:57:29.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: 73 2021-03-24T21:57:29.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: : 2021-03-24T21:57:29.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: cluster 2021-03-24T21:57:29.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: 2021-03-24T21:57:29.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: [DBG] 2021-03-24T21:57:29.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: 2021-03-24T21:57:29.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: 7.10 deep-scrub ok 2021-03-24T21:57:29.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: 2021-03-24T21:57:29.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: cluster 2021-03-24T21:57:29.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: 2021-03-24T21:57:29.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: 2021 2021-03-24T21:57:29.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: - 2021-03-24T21:57:29.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: 03 2021-03-24T21:57:29.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: - 2021-03-24T21:57:29.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: 24 2021-03-24T21:57:29.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: T 2021-03-24T21:57:29.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: 21 2021-03-24T21:57:29.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: : 2021-03-24T21:57:29.421 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: 57 2021-03-24T21:57:29.421 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: : 2021-03-24T21:57:29.421 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: 29 2021-03-24T21:57:29.421 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: . 2021-03-24T21:57:29.422 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: 157923 2021-03-24T21:57:29.422 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: +0000 2021-03-24T21:57:29.422 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: 2021-03-24T21:57:29.422 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: mon.a 2021-03-24T21:57:29.422 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: ( 2021-03-24T21:57:29.423 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: mon 2021-03-24T21:57:29.423 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: . 2021-03-24T21:57:29.423 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: 0 2021-03-24T21:57:29.423 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: ) 2021-03-24T21:57:29.424 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: 936 2021-03-24T21:57:29.424 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: : 2021-03-24T21:57:29.424 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: cluster 2021-03-24T21:57:29.424 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: 2021-03-24T21:57:29.424 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: [DBG] 2021-03-24T21:57:29.425 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: 2021-03-24T21:57:29.425 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: osdmap e279: 8 total, 8 up, 6 in 2021-03-24T21:57:29.425 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[19936]: 2021-03-24T21:57:29.425 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:57:29.426 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[24927]: 21:57:28. 2021-03-24T21:57:29.426 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[24927]: 119390+0000 osd.5 (osd.5) 73 : 2021-03-24T21:57:29.426 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[24927]: cluster [DBG] 7.10 deep-scrub ok 2021-03-24T21:57:29.426 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:57:29.427 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[24927]: -03-24T21:57:29.157923+0000 2021-03-24T21:57:29.427 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[24927]: mon.a (mon.0 2021-03-24T21:57:29.427 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[24927]: ) 936 : 2021-03-24T21:57:29.427 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:29 smithi125 conmon[24927]: cluster [DBG] osdmap e279: 8 total, 8 up, 6 in 2021-03-24T21:57:29.479 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:29.540 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:29.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:29.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:29.562 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-24T21:57:29.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:29 smithi142 conmon[20125]: cluster 2021-03-24T21:57:28.119390+0000 osd.5 (osd.5) 73 : 2021-03-24T21:57:29.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:29 smithi142 conmon[20125]: cluster [DBG] 7.10 deep-scrub ok 2021-03-24T21:57:29.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:29 smithi142 conmon[20125]: cluster 2021-03-24T21:57:29.157923+0000 2021-03-24T21:57:29.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:29 smithi142 conmon[20125]: mon.a (mon.0) 936 : cluster [DBG] osdmap e279: 8 total, 8 up, 6 in 2021-03-24T21:57:29.647 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:29.705 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:29.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:29.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:57:29.813 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:29.866 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:29.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:29.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:57:29.972 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:30.021 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:30.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:30.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:57:30.125 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:30.151 INFO:tasks.rados.rados.0.smithi125.stdout:914: done (2 left) 2021-03-24T21:57:30.152 INFO:tasks.rados.rados.0.smithi125.stdout:917: done (1 left) 2021-03-24T21:57:30.152 INFO:tasks.rados.rados.0.smithi125.stdout:918: done (0 left) 2021-03-24T21:57:30.152 INFO:tasks.rados.rados.0.smithi125.stdout:919: read oid 17 snap -1 2021-03-24T21:57:30.153 INFO:tasks.rados.rados.0.smithi125.stdout:919: expect (ObjNum 312 snap 77 seq_num 312) 2021-03-24T21:57:30.153 INFO:tasks.rados.rados.0.smithi125.stdout:920: snap_remove snap 86 2021-03-24T21:57:30.184 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:30.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:30.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:57:30.289 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:30.340 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-24T21:57:30.341 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T21:57:30.341 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[19936]: cluster 2021-03-24T21:57:30.341 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[19936]: 2021-03-24T21:57:30.342 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[19936]: 2021 2021-03-24T21:57:30.342 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[19936]: - 2021-03-24T21:57:30.342 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[19936]: 03 2021-03-24T21:57:30.342 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[19936]: - 2021-03-24T21:57:30.343 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[19936]: 24 2021-03-24T21:57:30.343 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[19936]: T 2021-03-24T21:57:30.343 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[19936]: 21 2021-03-24T21:57:30.343 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[19936]: : 2021-03-24T21:57:30.343 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[19936]: 57 2021-03-24T21:57:30.344 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[19936]: : 2021-03-24T21:57:30.344 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[19936]: 28 2021-03-24T21:57:30.344 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[19936]: . 2021-03-24T21:57:30.344 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[19936]: 773611 2021-03-24T21:57:30.345 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[19936]: +0000 2021-03-24T21:57:30.345 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[19936]: 2021-03-24T21:57:30.345 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[19936]: mgr.y 2021-03-24T21:57:30.345 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[19936]: ( 2021-03-24T21:57:30.345 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[19936]: mgr 2021-03-24T21:57:30.346 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[19936]: . 2021-03-24T21:57:30.346 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[19936]: 14144 2021-03-24T21:57:30.346 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[19936]: ) 2021-03-24T21:57:30.346 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[19936]: 406 2021-03-24T21:57:30.347 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[19936]: : 2021-03-24T21:57:30.347 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[19936]: cluster 2021-03-24T21:57:30.347 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[19936]: [DBG] pgmap v580: 168 pgs: 2 activating, 1 active+clean+snaptrim, 7 unknown, 158 active+clean; 146 MiB data, 518 MiB used, 536 GiB / 536 GiB avail; 21/1122 objects degraded (1.872%) 2021-03-24T21:57:30.347 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[19936]: cluster 2021-03-24T21:57:30.149705+0000 mon.a ( 2021-03-24T21:57:30.347 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[19936]: mon.0) 937 : cluster [DBG] osdmap e280: 8 total, 8 up, 6 in 2021-03-24T21:57:30.348 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[24927]: cluster 2021-03-24T21:57:30.348 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[24927]: 2021-03-24T21 2021-03-24T21:57:30.348 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[24927]: :57:28.773611+0000 mgr.y (mgr.14144) 406 : cluster [DBG] pgmap v580: 168 pgs: 2 activating, 1 active+clean+snaptrim, 7 unknown, 158 active+clean; 146 MiB data, 518 MiB used, 536 GiB / 536 GiB avail; 21/1122 objects degraded (1.872%) 2021-03-24T21:57:30.349 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[24927]: cluster 2021-03-24T21:57:30.349 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[24927]: 2021-03-24T21:57:30.349 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[24927]: 2021 2021-03-24T21:57:30.349 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[24927]: - 2021-03-24T21:57:30.349 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[24927]: 03 2021-03-24T21:57:30.350 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[24927]: - 2021-03-24T21:57:30.350 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[24927]: 24 2021-03-24T21:57:30.350 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[24927]: T 2021-03-24T21:57:30.350 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[24927]: 21 2021-03-24T21:57:30.351 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[24927]: : 2021-03-24T21:57:30.351 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[24927]: 57 2021-03-24T21:57:30.351 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[24927]: : 2021-03-24T21:57:30.351 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[24927]: 30 2021-03-24T21:57:30.352 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[24927]: . 2021-03-24T21:57:30.352 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[24927]: 149705 2021-03-24T21:57:30.352 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[24927]: +0000 2021-03-24T21:57:30.352 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[24927]: 2021-03-24T21:57:30.352 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[24927]: mon.a 2021-03-24T21:57:30.353 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[24927]: ( 2021-03-24T21:57:30.353 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[24927]: mon 2021-03-24T21:57:30.353 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[24927]: . 2021-03-24T21:57:30.353 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[24927]: 0 2021-03-24T21:57:30.354 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[24927]: ) 2021-03-24T21:57:30.354 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[24927]: 937 2021-03-24T21:57:30.354 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[24927]: : 2021-03-24T21:57:30.354 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[24927]: cluster 2021-03-24T21:57:30.355 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[24927]: 2021-03-24T21:57:30.355 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[24927]: [DBG] 2021-03-24T21:57:30.355 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[24927]: 2021-03-24T21:57:30.355 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[24927]: osdmap e280: 8 total, 8 up, 6 in 2021-03-24T21:57:30.355 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:30 smithi125 conmon[24927]: 2021-03-24T21:57:30.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:30.357 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:30.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:57:30.458 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:30.472 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:30 smithi142 conmon[20125]: cluster 2021-03-24T21:57:28. 2021-03-24T21:57:30.473 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:30 smithi142 conmon[20125]: 773611+0000 mgr.y (mgr.14144) 406 : cluster [DBG] pgmap v580: 168 pgs: 2 activating, 1 active+clean+snaptrim, 7 unknown, 158 active+clean; 146 MiB data, 518 MiB used, 536 GiB / 536 GiB avail; 21/1122 objects degraded (1.872%) 2021-03-24T21:57:30.473 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:30 smithi142 conmon[20125]: cluster 2021-03-24T21:57:30.149705+0000 2021-03-24T21:57:30.473 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:30 smithi142 conmon[20125]: mon.a (mon.0) 937 : cluster [DBG] osdmap e280: 8 total, 8 up, 6 in 2021-03-24T21:57:30.512 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:30.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:30.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:57:30.619 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:30.683 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:30.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:30.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:57:30.788 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:30.834 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:30.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:30.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:57:30.938 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:30.992 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:30.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:30.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:57:31.005 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:57:31.098 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:31.151 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:31.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:31.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:57:31.161 INFO:tasks.rados.rados.0.smithi125.stdout:920: done (1 left) 2021-03-24T21:57:31.161 INFO:tasks.rados.rados.0.smithi125.stdout:921: copy_from oid 22 from oid 28 current snap is 87 2021-03-24T21:57:31.162 INFO:tasks.rados.rados.0.smithi125.stdout:919: expect (ObjNum 312 snap 77 seq_num 312) 2021-03-24T21:57:31.223 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 7] out_osds: [6, 1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-03-24T21:57:31.224 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-24T21:57:31.224 INFO:tasks.thrashosds.thrasher:inject_pause on 4 2021-03-24T21:57:31.224 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2021-03-24T21:57:31.225 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2021-03-24T21:57:31.225 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.4 config set filestore_inject_stall 3 2021-03-24T21:57:31.260 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:31.267 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.3 dump_blocked_ops 2021-03-24T21:57:31.317 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:31.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:31.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:57:31.423 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:31.428 INFO:tasks.rados.rados.0.smithi125.stdout:919: done (1 left) 2021-03-24T21:57:31.428 INFO:tasks.rados.rados.0.smithi125.stdout:922: write oid 17 current snap is 87 2021-03-24T21:57:31.429 INFO:tasks.rados.rados.0.smithi125.stdout:922: seq_num 340 ranges {514380=704192,1629760=478435,2604288=442602} 2021-03-24T21:57:31.435 INFO:tasks.rados.rados.0.smithi125.stdout:922: writing smithi12556470-17 from 514380 to 1218572 tid 1 2021-03-24T21:57:31.442 INFO:tasks.rados.rados.0.smithi125.stdout:922: writing smithi12556470-17 from 1629760 to 2108195 tid 2 2021-03-24T21:57:31.448 INFO:tasks.rados.rados.0.smithi125.stdout:922: writing smithi12556470-17 from 2604288 to 3046890 tid 3 2021-03-24T21:57:31.448 INFO:tasks.rados.rados.0.smithi125.stdout:923: delete oid 15 current snap is 87 2021-03-24T21:57:31.448 INFO:tasks.rados.rados.0.smithi125.stdout:922: finishing write tid 1 to smithi12556470-17 2021-03-24T21:57:31.448 INFO:tasks.rados.rados.0.smithi125.stdout:922: finishing write tid 2 to smithi12556470-17 2021-03-24T21:57:31.450 INFO:tasks.rados.rados.0.smithi125.stdout:923: done (2 left) 2021-03-24T21:57:31.450 INFO:tasks.rados.rados.0.smithi125.stdout:924: read oid 38 snap -1 2021-03-24T21:57:31.451 INFO:tasks.rados.rados.0.smithi125.stdout:924: expect deleted 2021-03-24T21:57:31.451 INFO:tasks.rados.rados.0.smithi125.stdout:925: write (excl) oid 47 current snap is 87 2021-03-24T21:57:31.451 INFO:tasks.rados.rados.0.smithi125.stdout:925: seq_num 342 ranges {547994=454690,1746724=482104,2659134=465146,3605976=142018} 2021-03-24T21:57:31.457 INFO:tasks.rados.rados.0.smithi125.stdout:925: writing smithi12556470-47 from 547994 to 1002684 tid 1 2021-03-24T21:57:31.464 INFO:tasks.rados.rados.0.smithi125.stdout:925: writing smithi12556470-47 from 1746724 to 2228828 tid 2 2021-03-24T21:57:31.469 INFO:tasks.rados.rados.0.smithi125.stdout:925: writing smithi12556470-47 from 2659134 to 3124280 tid 3 2021-03-24T21:57:31.476 INFO:tasks.rados.rados.0.smithi125.stdout:925: writing smithi12556470-47 from 3605976 to 3747994 tid 4 2021-03-24T21:57:31.476 INFO:tasks.rados.rados.0.smithi125.stdout:926: write (excl) oid 2 current snap is 87 2021-03-24T21:57:31.476 INFO:tasks.rados.rados.0.smithi125.stdout:926: seq_num 343 ranges {564801=646292,1921559=742115,3202910=561891} 2021-03-24T21:57:31.483 INFO:tasks.rados.rados.0.smithi125.stdout:926: writing smithi12556470-2 from 564801 to 1211093 tid 1 2021-03-24T21:57:31.483 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:31.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:31.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:57:31.495 INFO:tasks.rados.rados.0.smithi125.stdout:926: writing smithi12556470-2 from 1921559 to 2663674 tid 2 2021-03-24T21:57:31.501 INFO:tasks.rados.rados.0.smithi125.stdout:926: writing smithi12556470-2 from 3202910 to 3764801 tid 3 2021-03-24T21:57:31.502 INFO:tasks.rados.rados.0.smithi125.stdout:927: copy_from oid 35 from oid 34 current snap is 87 2021-03-24T21:57:31.502 INFO:tasks.rados.rados.0.smithi125.stdout:922: finishing write tid 3 to smithi12556470-17 2021-03-24T21:57:31.502 INFO:tasks.rados.rados.0.smithi125.stdout:922: finishing write tid 4 to smithi12556470-17 2021-03-24T21:57:31.503 INFO:tasks.rados.rados.0.smithi125.stdout:922: finishing write tid 5 to smithi12556470-17 2021-03-24T21:57:31.503 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 17 v 266 (ObjNum 340 snap 87 seq_num 340) dirty exists 2021-03-24T21:57:31.503 INFO:tasks.rados.rados.0.smithi125.stdout:922: left oid 17 (ObjNum 340 snap 87 seq_num 340) 2021-03-24T21:57:31.503 INFO:tasks.rados.rados.0.smithi125.stdout:925: finishing write tid 1 to smithi12556470-47 2021-03-24T21:57:31.503 INFO:tasks.rados.rados.0.smithi125.stdout:925: finishing write tid 2 to smithi12556470-47 2021-03-24T21:57:31.504 INFO:tasks.rados.rados.0.smithi125.stdout:925: finishing write tid 3 to smithi12556470-47 2021-03-24T21:57:31.504 INFO:tasks.rados.rados.0.smithi125.stdout:922: done (5 left) 2021-03-24T21:57:31.504 INFO:tasks.rados.rados.0.smithi125.stdout:924: done (4 left) 2021-03-24T21:57:31.504 INFO:tasks.rados.rados.0.smithi125.stdout:928: read oid 37 snap -1 2021-03-24T21:57:31.505 INFO:tasks.rados.rados.0.smithi125.stdout:928: expect (ObjNum 229 snap 50 seq_num 229) 2021-03-24T21:57:31.505 INFO:tasks.rados.rados.0.smithi125.stdout:929: snap_create 2021-03-24T21:57:31.505 INFO:tasks.rados.rados.0.smithi125.stdout:925: finishing write tid 4 to smithi12556470-47 2021-03-24T21:57:31.505 INFO:tasks.rados.rados.0.smithi125.stdout:925: finishing write tid 5 to smithi12556470-47 2021-03-24T21:57:31.505 INFO:tasks.rados.rados.0.smithi125.stdout:925: finishing write tid 6 to smithi12556470-47 2021-03-24T21:57:31.506 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 47 v 100 (ObjNum 342 snap 87 seq_num 342) dirty exists 2021-03-24T21:57:31.506 INFO:tasks.rados.rados.0.smithi125.stdout:925: left oid 47 (ObjNum 342 snap 87 seq_num 342) 2021-03-24T21:57:31.506 INFO:tasks.rados.rados.0.smithi125.stdout:926: finishing write tid 1 to smithi12556470-2 2021-03-24T21:57:31.506 INFO:tasks.rados.rados.0.smithi125.stdout:926: finishing write tid 2 to smithi12556470-2 2021-03-24T21:57:31.507 INFO:tasks.rados.rados.0.smithi125.stdout:926: finishing write tid 3 to smithi12556470-2 2021-03-24T21:57:31.507 INFO:tasks.rados.rados.0.smithi125.stdout:926: finishing write tid 4 to smithi12556470-2 2021-03-24T21:57:31.508 INFO:tasks.rados.rados.0.smithi125.stdout:926: finishing write tid 5 to smithi12556470-2 2021-03-24T21:57:31.508 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 2 v 260 (ObjNum 343 snap 87 seq_num 343) dirty exists 2021-03-24T21:57:31.508 INFO:tasks.rados.rados.0.smithi125.stdout:926: left oid 2 (ObjNum 343 snap 87 seq_num 343) 2021-03-24T21:57:31.510 INFO:tasks.rados.rados.0.smithi125.stdout:928: expect (ObjNum 229 snap 50 seq_num 229) 2021-03-24T21:57:31.539 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T21:57:31.596 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:31.646 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:31.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:31.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:57:31.700 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:57:31.702 INFO:tasks.rados.rados.0.smithi125.stdout:927: finishing copy_from to smithi12556470-35 2021-03-24T21:57:31.702 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 35 v 193 (ObjNum 203 snap 43 seq_num 203) dirty exists 2021-03-24T21:57:31.703 INFO:tasks.rados.rados.0.smithi125.stdout:927: finishing copy_from racing read to smithi12556470-35 2021-03-24T21:57:31.751 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:31.813 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:31.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:31.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:57:31.922 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:31.971 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:31.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:31.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:57:32.076 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:32.135 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:32.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:32.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:57:32.166 INFO:tasks.rados.rados.0.smithi125.stdout:925: done (5 left) 2021-03-24T21:57:32.166 INFO:tasks.rados.rados.0.smithi125.stdout:926: done (4 left) 2021-03-24T21:57:32.167 INFO:tasks.rados.rados.0.smithi125.stdout:927: done (3 left) 2021-03-24T21:57:32.167 INFO:tasks.rados.rados.0.smithi125.stdout:928: done (2 left) 2021-03-24T21:57:32.167 INFO:tasks.rados.rados.0.smithi125.stdout:929: done (1 left) 2021-03-24T21:57:32.167 INFO:tasks.rados.rados.0.smithi125.stdout:930: write (excl) oid 18 current snap is 88 2021-03-24T21:57:32.168 INFO:tasks.rados.rados.0.smithi125.stdout:930: seq_num 344 ranges {581608=721541,1863688=485773,3113980=604043} 2021-03-24T21:57:32.174 INFO:tasks.rados.rados.0.smithi125.stdout:930: writing smithi12556470-18 from 581608 to 1303149 tid 1 2021-03-24T21:57:32.174 INFO:teuthology.orchestra.run.smithi125.stderr:noscrub is set 2021-03-24T21:57:32.180 INFO:tasks.rados.rados.0.smithi125.stdout:930: writing smithi12556470-18 from 1863688 to 2349461 tid 2 2021-03-24T21:57:32.189 INFO:tasks.rados.rados.0.smithi125.stdout:930: writing smithi12556470-18 from 3113980 to 3718023 tid 3 2021-03-24T21:57:32.189 INFO:tasks.rados.rados.0.smithi125.stdout:931: write (excl) oid 16 current snap is 88 2021-03-24T21:57:32.190 INFO:tasks.rados.rados.0.smithi125.stdout:930: finishing write tid 1 to smithi12556470-18 2021-03-24T21:57:32.190 INFO:tasks.rados.rados.0.smithi125.stdout:931: seq_num 345 ranges {598415=796790,1922170=745784,3141403=415315} 2021-03-24T21:57:32.197 INFO:tasks.rados.rados.0.smithi125.stdout:931: writing smithi12556470-16 from 598415 to 1395205 tid 1 2021-03-24T21:57:32.204 INFO:tasks.rados.rados.0.smithi125.stdout:931: writing smithi12556470-16 from 1922170 to 2667954 tid 2 2021-03-24T21:57:32.210 INFO:tasks.rados.rados.0.smithi125.stdout:931: writing smithi12556470-16 from 3141403 to 3556718 tid 3 2021-03-24T21:57:32.210 INFO:tasks.rados.rados.0.smithi125.stdout:932: delete oid 32 current snap is 88 2021-03-24T21:57:32.210 INFO:tasks.rados.rados.0.smithi125.stdout:930: finishing write tid 2 to smithi12556470-18 2021-03-24T21:57:32.211 INFO:tasks.rados.rados.0.smithi125.stdout:921: finishing copy_from to smithi12556470-22 2021-03-24T21:57:32.211 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 22 v 175 (ObjNum 308 snap 75 seq_num 308) dirty exists 2021-03-24T21:57:32.211 INFO:tasks.rados.rados.0.smithi125.stdout:930: finishing write tid 3 to smithi12556470-18 2021-03-24T21:57:32.211 INFO:tasks.rados.rados.0.smithi125.stdout:930: finishing write tid 4 to smithi12556470-18 2021-03-24T21:57:32.212 INFO:tasks.rados.rados.0.smithi125.stdout:921: finishing copy_from racing read to smithi12556470-22 2021-03-24T21:57:32.212 INFO:tasks.rados.rados.0.smithi125.stdout:930: finishing write tid 5 to smithi12556470-18 2021-03-24T21:57:32.212 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 18 v 169 (ObjNum 344 snap 88 seq_num 344) dirty exists 2021-03-24T21:57:32.212 INFO:tasks.rados.rados.0.smithi125.stdout:930: left oid 18 (ObjNum 344 snap 88 seq_num 344) 2021-03-24T21:57:32.212 INFO:tasks.rados.rados.0.smithi125.stdout:931: finishing write tid 1 to smithi12556470-16 2021-03-24T21:57:32.213 INFO:tasks.rados.rados.0.smithi125.stdout:931: finishing write tid 2 to smithi12556470-16 2021-03-24T21:57:32.214 INFO:tasks.rados.rados.0.smithi125.stdout:931: finishing write tid 3 to smithi12556470-16 2021-03-24T21:57:32.214 INFO:tasks.rados.rados.0.smithi125.stdout:931: finishing write tid 4 to smithi12556470-16 2021-03-24T21:57:32.214 INFO:tasks.rados.rados.0.smithi125.stdout:931: finishing write tid 5 to smithi12556470-16 2021-03-24T21:57:32.215 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 16 v 225 (ObjNum 345 snap 88 seq_num 345) dirty exists 2021-03-24T21:57:32.215 INFO:tasks.rados.rados.0.smithi125.stdout:931: left oid 16 (ObjNum 345 snap 88 seq_num 345) 2021-03-24T21:57:32.240 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:32.297 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:32.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:32.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:57:32.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:32 smithi142 conmon[20125]: cluster 2021-03-24T21:57:30.774203+0000 mgr.y (mgr.14144) 407 : cluster 2021-03-24T21:57:32.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:32 smithi142 conmon[20125]: [DBG] pgmap v583: 168 pgs: 5 remapped+peering, 2 activating, 1 active+clean+snaptrim, 2 unknown, 158 active+clean; 135 MiB data, 495 MiB used, 536 GiB / 536 GiB avail; 21/1089 objects degraded (1.928%) 2021-03-24T21:57:32.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:32 smithi142 conmon[20125]: cluster 2021-03-24T21:57:30.830892+0000 2021-03-24T21:57:32.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:32 smithi142 conmon[20125]: osd.1 (osd.1) 37 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:57:32.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:32 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:57:32.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:32 smithi142 conmon[20125]: 24T21:57:30.831068+0000 2021-03-24T21:57:32.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:32 smithi142 conmon[20125]: osd.1 (osd.1) 38 : cluster [DBG] 2021-03-24T21:57:32.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:32 smithi142 conmon[20125]: purged_snaps scrub ok 2021-03-24T21:57:32.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:32 smithi142 conmon[20125]: cluster 2021-03-24T21:57:32.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:32 smithi142 conmon[20125]: 2021-03-24T21:57: 2021-03-24T21:57:32.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:32 smithi142 conmon[20125]: 31.118032+0000 osd.5 (osd.5) 74 2021-03-24T21:57:32.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:32 smithi142 conmon[20125]: : cluster [DBG] 5.12 deep-scrub ok 2021-03-24T21:57:32.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:32 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:57:32.373 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:32 smithi142 conmon[20125]: -24T21:57:31.152617+0000 2021-03-24T21:57:32.373 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:32 smithi142 conmon[20125]: mon.a (mon.0) 938 : cluster 2021-03-24T21:57:32.374 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:32 smithi142 conmon[20125]: [DBG] osdmap e281: 8 total, 8 up, 6 in 2021-03-24T21:57:32.374 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:32 smithi142 conmon[20125]: audit 2021-03-24 2021-03-24T21:57:32.375 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:32 smithi142 conmon[20125]: T21:57:31.955850+0000 2021-03-24T21:57:32.375 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:32 smithi142 conmon[20125]: mon.c (mon.2) 43 : audit [INF] from='client.? v1:172.21.15.125:0/655262657' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T21:57:32.376 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:32 smithi142 conmon[20125]: audit 2021-03-24T21:57:32.376 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:32 smithi142 conmon[20125]: 2021-03-24T21:57 2021-03-24T21:57:32.376 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:32 smithi142 conmon[20125]: :31.956196+0000 mon.a (mon.0) 2021-03-24T21:57:32.377 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:32 smithi142 conmon[20125]: 939 : audit [INF] from='client.34988 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T21:57:32.404 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:32.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:32 smithi125 conmon[19936]: cluster 2021-03-24T21:57:30.774203+0000 mgr.y (mgr.14144) 407 : 2021-03-24T21:57:32.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:32 smithi125 conmon[19936]: cluster [DBG] pgmap v583: 168 pgs: 5 remapped+peering, 2 activating, 1 active+clean+snaptrim, 2 unknown, 158 active+clean; 135 MiB data, 495 MiB used, 536 GiB / 536 GiB avail; 21/1089 objects degraded (1.928%) 2021-03-24T21:57:32.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:32 smithi125 conmon[19936]: cluster 2021-03-24T21:57:30.830892+0000 osd.1 (osd.1) 37 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:57:32.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:32 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:57:32.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:32 smithi125 conmon[19936]: :57:30.831068+0000 osd.1 (osd.1) 38 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:57:32.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:32 smithi125 conmon[19936]: cluster 2021-03-24T21:57:32.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:32 smithi125 conmon[19936]: 2021-03-24T21:57:31.118032+0000 osd.5 (osd.5) 74 : cluster [DBG] 5.12 deep-scrub ok 2021-03-24T21:57:32.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:32 smithi125 conmon[19936]: cluster 2021-03-24T21:57:32.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:32 smithi125 conmon[19936]: 2021-03-24T21:57:31.152617+0000 mon.a (mon.0) 938 : cluster [DBG] osdmap e281: 8 total, 8 up, 6 in 2021-03-24T21:57:32.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:32 smithi125 conmon[19936]: 2021-03-24T21:57:32.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:32 smithi125 conmon[19936]: audit 2021 2021-03-24T21:57:32.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:32 smithi125 conmon[19936]: -03-24T21:57:31.955850+0000 mon.c 2021-03-24T21:57:32.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:32 smithi125 conmon[19936]: (mon.2) 43 : audit [INF] from='client.? v1:172.21.15.125:0/655262657' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T21:57:32.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:32 smithi125 conmon[19936]: 2021-03-24T21:57:32.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:32 smithi125 conmon[19936]: audit 2021-03-24T21:57 2021-03-24T21:57:32.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:32 smithi125 conmon[19936]: :31.956196+0000 mon.a (mon.0) 939 : 2021-03-24T21:57:32.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:32 smithi125 conmon[19936]: audit [INF] from='client.34988 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T21:57:32.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:32 smithi125 conmon[24927]: cluster 2021-03-24T21:57:30.774203+0000 mgr.y 2021-03-24T21:57:32.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:32 smithi125 conmon[24927]: (mgr.14144) 407 : cluster [DBG] pgmap v583: 168 pgs: 5 remapped+peering, 2 activating, 1 active+clean+snaptrim, 2 unknown, 158 active+clean; 135 MiB data, 495 MiB used, 536 GiB / 536 GiB avail; 21/1089 objects degraded (1.928%) 2021-03-24T21:57:32.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:32 smithi125 conmon[24927]: cluster 2021-03-24T21:57:30.830892 2021-03-24T21:57:32.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:32 smithi125 conmon[24927]: +0000 osd.1 (osd.1) 37 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:57:32.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:32 smithi125 conmon[24927]: cluster 2021-03-24T21:57:32.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:32 smithi125 conmon[24927]: 2021-03-24T21:57:30.831068+0000 osd.1 ( 2021-03-24T21:57:32.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:32 smithi125 conmon[24927]: osd.1) 38 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:57:32.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:32 smithi125 conmon[24927]: cluster 2021-03-24T21:57:31.118032+0000 osd.5 (osd. 2021-03-24T21:57:32.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:32 smithi125 conmon[24927]: 5) 74 : cluster [DBG] 5.12 deep-scrub ok 2021-03-24T21:57:32.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:32 smithi125 conmon[24927]: cluster 2021-03-24T21:57:31.152617+0000 2021-03-24T21:57:32.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:32 smithi125 conmon[24927]: mon.a (mon.0) 938 : cluster [DBG] osdmap e281: 8 total, 8 up, 6 in 2021-03-24T21:57:32.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:32 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T21:57:32.421 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:32 smithi125 conmon[24927]: 24T21:57:31.955850+0000 mon.c (mon.2 2021-03-24T21:57:32.421 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:32 smithi125 conmon[24927]: ) 43 : audit [INF] from='client.? v1:172.21.15.125:0/655262657' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T21:57:32.421 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:32 smithi125 conmon[24927]: audit 2021-03-24T21 2021-03-24T21:57:32.422 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:32 smithi125 conmon[24927]: :57:31.956196+0000 mon.a (mon.0) 939 : 2021-03-24T21:57:32.422 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:32 smithi125 conmon[24927]: audit [INF] from='client.34988 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T21:57:32.471 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:32.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:32.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:32.577 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:32.635 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:32.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:32.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:32.741 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:32.796 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:32.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:32.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:57:32.900 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:32.954 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:32.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:32.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:57:33.060 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:33.120 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:33.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:33.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:57:33.182 INFO:tasks.rados.rados.0.smithi125.stdout:921: done (3 left) 2021-03-24T21:57:33.182 INFO:tasks.rados.rados.0.smithi125.stdout:930: done (2 left) 2021-03-24T21:57:33.183 INFO:tasks.rados.rados.0.smithi125.stdout:931: done (1 left) 2021-03-24T21:57:33.183 INFO:tasks.rados.rados.0.smithi125.stdout:932: done (0 left) 2021-03-24T21:57:33.183 INFO:tasks.rados.rados.0.smithi125.stdout:933: snap_create 2021-03-24T21:57:33.225 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:33.280 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:33.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:33.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:57:33.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:33 smithi142 conmon[20125]: cluster 2021-03-24T21:57:32.121478+0000 osd.2 (osd.2) 56 : 2021-03-24T21:57:33.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:33 smithi142 conmon[20125]: cluster [DBG] 7.20 scrub ok 2021-03-24T21:57:33.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:33 smithi142 conmon[20125]: cluster 2021-03-24T21:57:32.147995+0000 2021-03-24T21:57:33.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:33 smithi142 conmon[20125]: mon.a (mon.0) 940 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:57:33.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:33 smithi142 conmon[20125]: audit 2021 2021-03-24T21:57:33.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:33 smithi142 conmon[20125]: -03-24T21:57:32 2021-03-24T21:57:33.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:33 smithi142 conmon[20125]: .165003+0000 mon.a (mon.0) 2021-03-24T21:57:33.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:33 smithi142 conmon[20125]: 941 : audit [INF] from='client.34988 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T21:57:33.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:33 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:57:33.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:33 smithi142 conmon[20125]: -03-24T21:57:32 2021-03-24T21:57:33.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:33 smithi142 conmon[20125]: .165079+0000 mon.a (mon.0) 2021-03-24T21:57:33.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:33 smithi142 conmon[20125]: 942 : cluster [DBG] osdmap e282: 8 total, 8 up, 6 in 2021-03-24T21:57:33.386 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:33.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:33 smithi125 conmon[19936]: cluster 2021-03-24T21:57:32.121478+0000 osd.2 (osd.2) 56 : cluster [DBG] 7.20 scrub ok 2021-03-24T21:57:33.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:33 smithi125 conmon[19936]: cluster 2021-03-24T21:57:32.147995+0000 mon.a (mon.0) 940 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:57:33.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:33 smithi125 conmon[19936]: audit 2021-03-24T21:57:32.165003+0000 mon.a (mon.0) 941 : audit [INF] from='client.34988 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T21:57:33.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:33 smithi125 conmon[19936]: cluster 2021-03-24T21:57:32.165079+0000 mon.a (mon.0) 942 : cluster [DBG] osdmap e282: 8 total, 8 up, 6 in 2021-03-24T21:57:33.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:33 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:57:33.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:33 smithi125 conmon[24927]: :57:32.121478+0000 osd.2 (osd.2) 56 : cluster [DBG] 7.20 scrub ok 2021-03-24T21:57:33.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:33 smithi125 conmon[24927]: cluster 2021-03-24T21:57:33.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:33 smithi125 conmon[24927]: 2021-03-24T21:57:33.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:33 smithi125 conmon[24927]: 2021-03-24T21:57 2021-03-24T21:57:33.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:33 smithi125 conmon[24927]: :32.147995+0000 mon.a (mon.0) 940 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:57:33.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:33 smithi125 conmon[24927]: audit 2021-03-24T21:57:33.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:33 smithi125 conmon[24927]: 2021-03-24T 2021-03-24T21:57:33.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:33 smithi125 conmon[24927]: 21:57:32 2021-03-24T21:57:33.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:33 smithi125 conmon[24927]: .165003+0000 mon.a (mon. 2021-03-24T21:57:33.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:33 smithi125 conmon[24927]: 0) 941 : audit [INF] from='client.34988 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T21:57:33.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:33 smithi125 conmon[24927]: 2021-03-24T21:57:33.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:33 smithi125 conmon[24927]: cluster 2021-03-24T21:57:32.165079+0000 mon.a (mon.0) 942 : cluster [DBG] osdmap e282: 8 total, 8 up, 6 in 2021-03-24T21:57:33.446 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:33.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:33.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:57:33.552 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:33.610 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:33.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:33.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:33.715 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:33.767 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:33.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:33.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:57:33.874 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:33.917 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2021-03-24T21:57:33.917 INFO:teuthology.orchestra.run.smithi142.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2021-03-24T21:57:33.918 INFO:teuthology.orchestra.run.smithi142.stdout:} 2021-03-24T21:57:33.941 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:33.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:33.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:57:34.045 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:34.102 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:34.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:34.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:57:34.193 INFO:tasks.rados.rados.0.smithi125.stdout:933: done (0 left) 2021-03-24T21:57:34.194 INFO:tasks.rados.rados.0.smithi125.stdout:934: read oid 2 snap -1 2021-03-24T21:57:34.194 INFO:tasks.rados.rados.0.smithi125.stdout:934: expect (ObjNum 343 snap 87 seq_num 343) 2021-03-24T21:57:34.194 INFO:tasks.rados.rados.0.smithi125.stdout:935: delete oid 20 current snap is 89 2021-03-24T21:57:34.210 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:34.226 INFO:tasks.rados.rados.0.smithi125.stdout:934: expect (ObjNum 343 snap 87 seq_num 343) 2021-03-24T21:57:34.264 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:34.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:34.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:57:34.348 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:34 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:57:34.349 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:34 smithi125 conmon[19936]: :57:32.774963+0000 mgr.y (mgr.14144) 408 : cluster [DBG] pgmap v586: 168 pgs: 5 remapped+peering, 2 activating, 1 active+clean+snaptrim, 1 unknown, 159 active+clean; 135 MiB data, 487 MiB used, 536 GiB / 536 GiB avail; 18 MiB/s rd, 11 op/s; 21/1089 objects degraded (1.928%); 3.5 MiB/s, 4 objects/s recovering 2021-03-24T21:57:34.349 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:34 smithi125 conmon[19936]: cluster 2021-03-24T21:57:34.349 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:34 smithi125 conmon[19936]: 2021-03-24T21:57:33.168592+0000 mon.a ( 2021-03-24T21:57:34.349 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:34 smithi125 conmon[19936]: mon.0) 943 : cluster [DBG] 2021-03-24T21:57:34.350 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:34 smithi125 conmon[19936]: osdmap e283: 8 total, 8 up, 6 in 2021-03-24T21:57:34.350 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:34 smithi125 conmon[24927]: cluster 2021-03-24T21:57:32.774963+0000 mgr.y 2021-03-24T21:57:34.350 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:34 smithi125 conmon[24927]: (mgr.14144) 408 : cluster [DBG] pgmap v586: 168 pgs: 5 remapped+peering, 2 activating, 1 active+clean+snaptrim, 1 unknown, 159 active+clean; 135 MiB data, 487 MiB used, 536 GiB / 536 GiB avail; 18 MiB/s rd, 11 op/s; 21/1089 objects degraded (1.928%); 3.5 MiB/s, 4 objects/s recovering 2021-03-24T21:57:34.351 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:34 smithi125 conmon[24927]: cluster 2021-03-24T21:57:33 2021-03-24T21:57:34.351 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:34 smithi125 conmon[24927]: .168592+0000 mon.a (mon.0) 943 : cluster [DBG] osdmap e283: 8 total, 8 up, 6 in 2021-03-24T21:57:34.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:34 smithi142 conmon[20125]: cluster 2021-03-24T21:57:32. 2021-03-24T21:57:34.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:34 smithi142 conmon[20125]: 774963+0000 mgr.y (mgr.14144) 408 : cluster [DBG] pgmap v586: 168 pgs: 5 remapped+peering, 2 activating, 1 active+clean+snaptrim, 1 unknown, 159 active+clean; 135 MiB data, 487 MiB used, 536 GiB / 536 GiB avail; 18 MiB/s rd, 11 op/s; 21/1089 objects degraded (1.928%); 3.5 MiB/s, 4 objects/s recovering 2021-03-24T21:57:34.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:34 smithi142 conmon[20125]: cluster 2021-03-24T21:57 2021-03-24T21:57:34.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:34 smithi142 conmon[20125]: :33.168592+0000 mon.a (mon.0) 943 : cluster 2021-03-24T21:57:34.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:34 smithi142 conmon[20125]: [DBG] osdmap e283: 8 total, 8 up, 6 in 2021-03-24T21:57:34.371 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:34.399 INFO:tasks.rados.rados.0.smithi125.stdout:934: done (1 left) 2021-03-24T21:57:34.399 INFO:tasks.rados.rados.0.smithi125.stdout:935: done (0 left) 2021-03-24T21:57:34.400 INFO:tasks.rados.rados.0.smithi125.stdout:936: copy_from oid 46 from oid 43 current snap is 89 2021-03-24T21:57:34.400 INFO:tasks.rados.rados.0.smithi125.stdout:937: read oid 6 snap -1 2021-03-24T21:57:34.400 INFO:tasks.rados.rados.0.smithi125.stdout:937: expect (ObjNum 123 snap 23 seq_num 123) 2021-03-24T21:57:34.400 INFO:tasks.rados.rados.0.smithi125.stdout:938: read oid 15 snap 88 2021-03-24T21:57:34.401 INFO:tasks.rados.rados.0.smithi125.stdout:938: expect deleted 2021-03-24T21:57:34.401 INFO:tasks.rados.rados.0.smithi125.stdout:939: delete oid 24 current snap is 89 2021-03-24T21:57:34.402 INFO:tasks.rados.rados.0.smithi125.stdout:936: finishing copy_from to smithi12556470-46 2021-03-24T21:57:34.402 INFO:tasks.rados.rados.0.smithi125.stdout:936: got expected ENOENT (src dne) 2021-03-24T21:57:34.402 INFO:tasks.rados.rados.0.smithi125.stdout:936: finishing copy_from racing read to smithi12556470-46 2021-03-24T21:57:34.403 INFO:tasks.rados.rados.0.smithi125.stdout:937: expect (ObjNum 123 snap 23 seq_num 123) 2021-03-24T21:57:34.423 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:34.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:34.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:57:34.519 INFO:tasks.rados.rados.0.smithi125.stdout:936: done (3 left) 2021-03-24T21:57:34.520 INFO:tasks.rados.rados.0.smithi125.stdout:937: done (2 left) 2021-03-24T21:57:34.520 INFO:tasks.rados.rados.0.smithi125.stdout:938: done (1 left) 2021-03-24T21:57:34.520 INFO:tasks.rados.rados.0.smithi125.stdout:939: done (0 left) 2021-03-24T21:57:34.520 INFO:tasks.rados.rados.0.smithi125.stdout:940: read oid 19 snap -1 2021-03-24T21:57:34.521 INFO:tasks.rados.rados.0.smithi125.stdout:940: expect (ObjNum 280 snap 70 seq_num 280) 2021-03-24T21:57:34.521 INFO:tasks.rados.rados.0.smithi125.stdout:941: snap_remove snap 87 2021-03-24T21:57:34.527 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:34.583 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:34.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:34.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:57:34.689 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:34.744 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:34.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:34.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:57:34.851 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:34.905 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:34.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:34.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:35.011 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:35.069 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:35.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:35.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:57:35.174 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:35.200 INFO:tasks.rados.rados.0.smithi125.stdout:941: done (1 left) 2021-03-24T21:57:35.201 INFO:tasks.rados.rados.0.smithi125.stdout:942: write oid 9 current snap is 89 2021-03-24T21:57:35.201 INFO:tasks.rados.rados.0.smithi125.stdout:942: seq_num 349 ranges {665643=697786,2039745=753122,3534742=330901} 2021-03-24T21:57:35.205 INFO:tasks.rados.rados.0.smithi125.stdout:942: writing smithi12556470-9 from 665643 to 1363429 tid 1 2021-03-24T21:57:35.206 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:35 smithi125 conmon[19936]: cluster 2021-03-24T21:57:35.206 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:35 smithi125 conmon[19936]: 2021-03-24T21:57:35.206 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:35 smithi125 conmon[19936]: 2021 2021-03-24T21:57:35.207 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:35 smithi125 conmon[19936]: - 2021-03-24T21:57:35.207 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:35 smithi125 conmon[19936]: 03 2021-03-24T21:57:35.207 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:35 smithi125 conmon[19936]: - 2021-03-24T21:57:35.207 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:35 smithi125 conmon[19936]: 24 2021-03-24T21:57:35.208 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:35 smithi125 conmon[19936]: T 2021-03-24T21:57:35.208 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:35 smithi125 conmon[19936]: 21 2021-03-24T21:57:35.208 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:35 smithi125 conmon[19936]: : 2021-03-24T21:57:35.208 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:35 smithi125 conmon[19936]: 57 2021-03-24T21:57:35.208 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:35 smithi125 conmon[19936]: :33.374347+0000 osd.4 (osd. 2021-03-24T21:57:35.209 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:35 smithi125 conmon[19936]: 4) 76 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:57:35.209 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:35 smithi125 conmon[19936]: cluster 2021-03-24T21:57:35.209 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:35 smithi125 conmon[19936]: 2021-03-24T21:57:35.209 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:35 smithi125 conmon[19936]: 2021 2021-03-24T21:57:35.210 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:35 smithi125 conmon[19936]: - 2021-03-24T21:57:35.210 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:35 smithi125 conmon[19936]: 03 2021-03-24T21:57:35.210 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:35 smithi125 conmon[19936]: -24T21:57:33.374848 2021-03-24T21:57:35.210 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:35 smithi125 conmon[19936]: +0000 osd.4 (osd.4) 77 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:57:35.210 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:35 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:57:35.211 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:35 smithi125 conmon[19936]: 03-24T21:57:34.185470+0000 mon.a ( 2021-03-24T21:57:35.211 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:35 smithi125 conmon[19936]: mon.0) 944 : cluster [DBG] osdmap e284: 8 total, 8 up, 6 in 2021-03-24T21:57:35.211 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:35 smithi125 conmon[24927]: cluster 2021-03-24T21:57:33.374347 2021-03-24T21:57:35.211 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:35 smithi125 conmon[24927]: +0000 osd.4 (osd.4) 76 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:57:35.212 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:35 smithi125 conmon[24927]: cluster 2021-03-24T21:57: 2021-03-24T21:57:35.212 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:35 smithi125 conmon[24927]: 33.374848+0000 osd.4 (osd.4) 77 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:57:35.212 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:35 smithi125 conmon[24927]: cluster 2021-03-24T21:57:34. 2021-03-24T21:57:35.212 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:35 smithi125 conmon[24927]: 185470+0000 mon.a (mon.0) 944 : cluster [DBG] 2021-03-24T21:57:35.213 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:35 smithi125 conmon[24927]: osdmap e284: 8 total, 8 up, 6 in 2021-03-24T21:57:35.213 INFO:tasks.rados.rados.0.smithi125.stdout:942: writing smithi12556470-9 from 2039745 to 2792867 tid 2 2021-03-24T21:57:35.220 INFO:tasks.rados.rados.0.smithi125.stdout:942: writing smithi12556470-9 from 3534742 to 3865643 tid 3 2021-03-24T21:57:35.221 INFO:tasks.rados.rados.0.smithi125.stdout:943: read oid 39 snap -1 2021-03-24T21:57:35.221 INFO:tasks.rados.rados.0.smithi125.stdout:943: expect (ObjNum 309 snap 76 seq_num 309) 2021-03-24T21:57:35.222 INFO:tasks.rados.rados.0.smithi125.stdout:944: rollback oid 34 current snap is 89 2021-03-24T21:57:35.222 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 34 to 88 2021-03-24T21:57:35.222 INFO:tasks.rados.rados.0.smithi125.stdout:945: read oid 22 snap -1 2021-03-24T21:57:35.222 INFO:tasks.rados.rados.0.smithi125.stdout:945: expect (ObjNum 308 snap 75 seq_num 308) 2021-03-24T21:57:35.223 INFO:tasks.rados.rados.0.smithi125.stdout:946: write (excl) oid 17 current snap is 89 2021-03-24T21:57:35.223 INFO:tasks.rados.rados.0.smithi125.stdout:946: seq_num 350 ranges {682450=489388,1814580=496780,2878518=788028} 2021-03-24T21:57:35.229 INFO:tasks.rados.rados.0.smithi125.stdout:946: writing smithi12556470-17 from 682450 to 1171838 tid 1 2021-03-24T21:57:35.236 INFO:tasks.rados.rados.0.smithi125.stdout:946: writing smithi12556470-17 from 1814580 to 2311360 tid 2 2021-03-24T21:57:35.239 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:35.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:35.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:57:35.244 INFO:tasks.rados.rados.0.smithi125.stdout:946: writing smithi12556470-17 from 2878518 to 3666546 tid 3 2021-03-24T21:57:35.245 INFO:tasks.rados.rados.0.smithi125.stdout:947: delete oid 35 current snap is 89 2021-03-24T21:57:35.245 INFO:tasks.rados.rados.0.smithi125.stdout:940: expect (ObjNum 280 snap 70 seq_num 280) 2021-03-24T21:57:35.331 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T21:57:35.343 INFO:tasks.rados.rados.0.smithi125.stdout:940: done (6 left) 2021-03-24T21:57:35.344 INFO:tasks.rados.rados.0.smithi125.stdout:947: done (5 left) 2021-03-24T21:57:35.344 INFO:tasks.rados.rados.0.smithi125.stdout:948: rollback oid 4 current snap is 89 2021-03-24T21:57:35.344 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 4 to 88 2021-03-24T21:57:35.344 INFO:tasks.rados.rados.0.smithi125.stdout:942: finishing write tid 1 to smithi12556470-9 2021-03-24T21:57:35.345 INFO:tasks.rados.rados.0.smithi125.stdout:944: finishing rollback tid 0 to smithi12556470-34 2021-03-24T21:57:35.345 INFO:tasks.rados.rados.0.smithi125.stdout:944: finishing rollback tid 1 to smithi12556470-34 2021-03-24T21:57:35.345 INFO:tasks.rados.rados.0.smithi125.stdout:944: finishing rollback tid 2 to smithi12556470-34 2021-03-24T21:57:35.345 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 34 v 253 (ObjNum 203 snap 43 seq_num 203) dirty exists 2021-03-24T21:57:35.346 INFO:tasks.rados.rados.0.smithi125.stdout:945: expect (ObjNum 308 snap 75 seq_num 308) 2021-03-24T21:57:35.346 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:35.359 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T21:57:35.359 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:35 smithi142 conmon[20125]: cluster 2021-03-24T21:57:35.360 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:35 smithi142 conmon[20125]: 2021-03-24T21:57:35.360 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:35 smithi142 conmon[20125]: 2021 2021-03-24T21:57:35.360 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:35 smithi142 conmon[20125]: - 2021-03-24T21:57:35.360 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:35 smithi142 conmon[20125]: 03 2021-03-24T21:57:35.361 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:35 smithi142 conmon[20125]: - 2021-03-24T21:57:35.361 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:35 smithi142 conmon[20125]: 24 2021-03-24T21:57:35.361 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:35 smithi142 conmon[20125]: T 2021-03-24T21:57:35.361 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:35 smithi142 conmon[20125]: 21 2021-03-24T21:57:35.361 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:35 smithi142 conmon[20125]: : 2021-03-24T21:57:35.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:35 smithi142 conmon[20125]: 57 2021-03-24T21:57:35.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:35 smithi142 conmon[20125]: :33.374347+0000 osd.4 (osd.4) 76 : cluster 2021-03-24T21:57:35.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:35 smithi142 conmon[20125]: [DBG] purged_snaps scrub starts 2021-03-24T21:57:35.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:35 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:57:35.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:35 smithi142 conmon[20125]: 21:57:33.374848+0000 osd.4 (osd.4) 77 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:57:35.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:35 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:57:35.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:35 smithi142 conmon[20125]: 21:57:34.185470+0000 mon.a (mon.0) 2021-03-24T21:57:35.363 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:35 smithi142 conmon[20125]: 944 : cluster [DBG] osdmap e284: 8 total, 8 up, 6 in 2021-03-24T21:57:35.379 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T21:57:35.399 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T21:57:35.403 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:35.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:35.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:35.421 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T21:57:35.446 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T21:57:35.469 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T21:57:35.493 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T21:57:35.495 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-24T21:57:35.495 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-24T21:57:35.496 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-24T21:57:35.496 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-24T21:57:35.496 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-24T21:57:35.496 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-24T21:57:35.496 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-24T21:57:35.497 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-24T21:57:35.499 INFO:journalctl@ceph.osd.0.smithi125.stdout:Mar 24 21:57:35 smithi125 conmon[28026]: debug 2021-03-24T21:57:35.499 INFO:journalctl@ceph.osd.0.smithi125.stdout:Mar 24 21:57:35 smithi125 conmon[28026]: 2021-03-24T21:57:35.216+0000 7f61bc487700 -1 osd.0 pg_epoch: 285 pg[7.1b( v 285'282 (0'0,285'282] local-lis/les=282/283 n=4 ec=275/99 lis/c=282/282 les/c/f=283/283/0 sis=282) [0,2,3] r=0 lpr=282 luod=283'279 crt=285'282 lcod 283'277 mlcod 283'277 active+clean+snaptrim trimq=[b0~1] ps=[93~1,97~1,99~1,a0~1,a2~1,a7~3,ae~1]] removing snap head 2021-03-24T21:57:35.499 INFO:journalctl@ceph.osd.0.smithi125.stdout:Mar 24 21:57:35 smithi125 conmon[28026]: 2021-03-24T21:57:35.509 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:35.554 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:35.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:35.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:57:35.632 INFO:tasks.rados.rados.0.smithi125.stdout:942: finishing write tid 2 to smithi12556470-9 2021-03-24T21:57:35.632 INFO:tasks.rados.rados.0.smithi125.stdout:942: finishing write tid 3 to smithi12556470-9 2021-03-24T21:57:35.632 INFO:tasks.rados.rados.0.smithi125.stdout:944: done (5 left) 2021-03-24T21:57:35.632 INFO:tasks.rados.rados.0.smithi125.stdout:945: done (4 left) 2021-03-24T21:57:35.633 INFO:tasks.rados.rados.0.smithi125.stdout:949: write (excl) oid 15 current snap is 89 2021-03-24T21:57:35.633 INFO:tasks.rados.rados.0.smithi125.stdout:942: finishing write tid 4 to smithi12556470-9 2021-03-24T21:57:35.633 INFO:tasks.rados.rados.0.smithi125.stdout:943: expect (ObjNum 309 snap 76 seq_num 309) 2021-03-24T21:57:35.663 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:35.718 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:35.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:35.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:35.740 INFO:tasks.rados.rados.0.smithi125.stdout:946: finishing write tid 1 to smithi12556470-17 2021-03-24T21:57:35.740 INFO:tasks.rados.rados.0.smithi125.stdout:942: finishing write tid 5 to smithi12556470-9 2021-03-24T21:57:35.740 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 9 v 286 (ObjNum 349 snap 89 seq_num 349) dirty exists 2021-03-24T21:57:35.740 INFO:tasks.rados.rados.0.smithi125.stdout:942: left oid 9 (ObjNum 349 snap 89 seq_num 349) 2021-03-24T21:57:35.741 INFO:tasks.rados.rados.0.smithi125.stdout:946: finishing write tid 2 to smithi12556470-17 2021-03-24T21:57:35.741 INFO:tasks.rados.rados.0.smithi125.stdout:949: seq_num 352 ranges {716064=639886,1815191=500449,2817011=526925} 2021-03-24T21:57:35.747 INFO:tasks.rados.rados.0.smithi125.stdout:949: writing smithi12556470-15 from 716064 to 1355950 tid 1 2021-03-24T21:57:35.753 INFO:tasks.rados.rados.0.smithi125.stdout:949: writing smithi12556470-15 from 1815191 to 2315640 tid 2 2021-03-24T21:57:35.759 INFO:tasks.rados.rados.0.smithi125.stdout:949: writing smithi12556470-15 from 2817011 to 3343936 tid 3 2021-03-24T21:57:35.760 INFO:tasks.rados.rados.0.smithi125.stdout:942: done (4 left) 2021-03-24T21:57:35.760 INFO:tasks.rados.rados.0.smithi125.stdout:943: done (3 left) 2021-03-24T21:57:35.760 INFO:tasks.rados.rados.0.smithi125.stdout:950: write (excl) oid 39 current snap is 89 2021-03-24T21:57:35.760 INFO:tasks.rados.rados.0.smithi125.stdout:946: finishing write tid 3 to smithi12556470-17 2021-03-24T21:57:35.761 INFO:tasks.rados.rados.0.smithi125.stdout:950: seq_num 353 ranges {732871=715135,1873673=644107,3244434=688437} 2021-03-24T21:57:35.768 INFO:tasks.rados.rados.0.smithi125.stdout:950: writing smithi12556470-39 from 732871 to 1448006 tid 1 2021-03-24T21:57:35.774 INFO:tasks.rados.rados.0.smithi125.stdout:950: writing smithi12556470-39 from 1873673 to 2517780 tid 2 2021-03-24T21:57:35.783 INFO:tasks.rados.rados.0.smithi125.stdout:950: writing smithi12556470-39 from 3244434 to 3932871 tid 3 2021-03-24T21:57:35.783 INFO:tasks.rados.rados.0.smithi125.stdout:951: write oid 3 current snap is 89 2021-03-24T21:57:35.783 INFO:tasks.rados.rados.0.smithi125.stdout:946: finishing write tid 4 to smithi12556470-17 2021-03-24T21:57:35.784 INFO:tasks.rados.rados.0.smithi125.stdout:946: finishing write tid 5 to smithi12556470-17 2021-03-24T21:57:35.784 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 17 v 271 (ObjNum 350 snap 89 seq_num 350) dirty exists 2021-03-24T21:57:35.784 INFO:tasks.rados.rados.0.smithi125.stdout:946: left oid 17 (ObjNum 350 snap 89 seq_num 350) 2021-03-24T21:57:35.784 INFO:tasks.rados.rados.0.smithi125.stdout:948: finishing rollback tid 0 to smithi12556470-4 2021-03-24T21:57:35.784 INFO:tasks.rados.rados.0.smithi125.stdout:948: finishing rollback tid 1 to smithi12556470-4 2021-03-24T21:57:35.785 INFO:tasks.rados.rados.0.smithi125.stdout:948: finishing rollback tid 2 to smithi12556470-4 2021-03-24T21:57:35.785 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 4 v 273 (ObjNum 319 snap 81 seq_num 319) dirty exists 2021-03-24T21:57:35.785 INFO:tasks.rados.rados.0.smithi125.stdout:951: seq_num 354 ranges {749678=790384,2332155=504118,3271857=549469} 2021-03-24T21:57:35.791 INFO:tasks.rados.rados.0.smithi125.stdout:951: writing smithi12556470-3 from 749678 to 1540062 tid 1 2021-03-24T21:57:35.799 INFO:tasks.rados.rados.0.smithi125.stdout:951: writing smithi12556470-3 from 2332155 to 2836273 tid 2 2021-03-24T21:57:35.805 INFO:tasks.rados.rados.0.smithi125.stdout:951: writing smithi12556470-3 from 3271857 to 3821326 tid 3 2021-03-24T21:57:35.805 INFO:tasks.rados.rados.0.smithi125.stdout:946: done (4 left) 2021-03-24T21:57:35.805 INFO:tasks.rados.rados.0.smithi125.stdout:948: done (3 left) 2021-03-24T21:57:35.805 INFO:tasks.rados.rados.0.smithi125.stdout:952: snap_create 2021-03-24T21:57:35.806 INFO:tasks.rados.rados.0.smithi125.stdout:949: finishing write tid 1 to smithi12556470-15 2021-03-24T21:57:35.806 INFO:tasks.rados.rados.0.smithi125.stdout:949: finishing write tid 2 to smithi12556470-15 2021-03-24T21:57:35.806 INFO:tasks.rados.rados.0.smithi125.stdout:949: finishing write tid 3 to smithi12556470-15 2021-03-24T21:57:35.806 INFO:tasks.rados.rados.0.smithi125.stdout:949: finishing write tid 4 to smithi12556470-15 2021-03-24T21:57:35.807 INFO:tasks.rados.rados.0.smithi125.stdout:949: finishing write tid 5 to smithi12556470-15 2021-03-24T21:57:35.807 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 15 v 264 (ObjNum 352 snap 89 seq_num 352) dirty exists 2021-03-24T21:57:35.807 INFO:tasks.rados.rados.0.smithi125.stdout:949: left oid 15 (ObjNum 352 snap 89 seq_num 352) 2021-03-24T21:57:35.807 INFO:tasks.rados.rados.0.smithi125.stdout:950: finishing write tid 1 to smithi12556470-39 2021-03-24T21:57:35.807 INFO:tasks.rados.rados.0.smithi125.stdout:950: finishing write tid 2 to smithi12556470-39 2021-03-24T21:57:35.808 INFO:tasks.rados.rados.0.smithi125.stdout:950: finishing write tid 3 to smithi12556470-39 2021-03-24T21:57:35.808 INFO:tasks.rados.rados.0.smithi125.stdout:950: finishing write tid 4 to smithi12556470-39 2021-03-24T21:57:35.808 INFO:tasks.rados.rados.0.smithi125.stdout:950: finishing write tid 5 to smithi12556470-39 2021-03-24T21:57:35.808 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 39 v 309 (ObjNum 353 snap 89 seq_num 353) dirty exists 2021-03-24T21:57:35.808 INFO:tasks.rados.rados.0.smithi125.stdout:950: left oid 39 (ObjNum 353 snap 89 seq_num 353) 2021-03-24T21:57:35.809 INFO:tasks.rados.rados.0.smithi125.stdout:951: finishing write tid 1 to smithi12556470-3 2021-03-24T21:57:35.809 INFO:tasks.rados.rados.0.smithi125.stdout:951: finishing write tid 2 to smithi12556470-3 2021-03-24T21:57:35.810 INFO:tasks.rados.rados.0.smithi125.stdout:951: finishing write tid 3 to smithi12556470-3 2021-03-24T21:57:35.810 INFO:tasks.rados.rados.0.smithi125.stdout:951: finishing write tid 4 to smithi12556470-3 2021-03-24T21:57:35.810 INFO:tasks.rados.rados.0.smithi125.stdout:951: finishing write tid 5 to smithi12556470-3 2021-03-24T21:57:35.811 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 3 v 122 (ObjNum 354 snap 89 seq_num 354) dirty exists 2021-03-24T21:57:35.811 INFO:tasks.rados.rados.0.smithi125.stdout:951: left oid 3 (ObjNum 354 snap 89 seq_num 354) 2021-03-24T21:57:35.823 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:35.879 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:35.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:35.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:57:35.986 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:36.045 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:36.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:36.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:57:36.153 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:36.209 INFO:tasks.rados.rados.0.smithi125.stdout:949: done (3 left) 2021-03-24T21:57:36.209 INFO:tasks.rados.rados.0.smithi125.stdout:950: done (2 left) 2021-03-24T21:57:36.209 INFO:tasks.rados.rados.0.smithi125.stdout:951: done (1 left) 2021-03-24T21:57:36.210 INFO:tasks.rados.rados.0.smithi125.stdout:952: done (0 left) 2021-03-24T21:57:36.210 INFO:tasks.rados.rados.0.smithi125.stdout:953: delete oid 8 current snap is 90 2021-03-24T21:57:36.210 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:36.212 INFO:tasks.rados.rados.0.smithi125.stdout:953: done (0 left) 2021-03-24T21:57:36.212 INFO:tasks.rados.rados.0.smithi125.stdout:954: delete oid 12 current snap is 90 2021-03-24T21:57:36.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:36.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:57:36.215 INFO:tasks.rados.rados.0.smithi125.stdout:954: done (0 left) 2021-03-24T21:57:36.215 INFO:tasks.rados.rados.0.smithi125.stdout:955: delete oid 7 current snap is 90 2021-03-24T21:57:36.220 INFO:tasks.rados.rados.0.smithi125.stdout:955: done (0 left) 2021-03-24T21:57:36.220 INFO:tasks.rados.rados.0.smithi125.stdout:956: snap_create 2021-03-24T21:57:36.314 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:36.371 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:36.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:36.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:57:36.478 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:36.482 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:36 smithi125 conmon[24927]: cluster 2021-03-24T21:57:34. 2021-03-24T21:57:36.482 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:36 smithi125 conmon[24927]: 776029+0000 mgr.y (mgr.14144) 409 : cluster [DBG] pgmap v589: 168 pgs: 168 active+clean; 133 MiB data, 515 MiB used, 536 GiB / 536 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 24 op/s; 9.8 MiB/s, 21 objects/s recovering 2021-03-24T21:57:36.483 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:36 smithi125 conmon[24927]: cluster 2021-03-24T21:57 2021-03-24T21:57:36.483 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:36 smithi125 conmon[24927]: :35.188866+0000 mon.a (mon.0) 945 : cluster [DBG] osdmap e285: 8 total, 8 up, 6 in 2021-03-24T21:57:36.483 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:36 smithi125 conmon[24927]: audit 2021-03-24T21:57 2021-03-24T21:57:36.483 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:36 smithi125 conmon[24927]: :35.305918+0000 mon.c (mon.2) 44 : audit [DBG] 2021-03-24T21:57:36.484 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:36 smithi125 conmon[24927]: from='client.? v1:172.21.15.125:0/1770643375' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:57:36.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:36 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T21:57:36.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:36 smithi125 conmon[19936]: T21:57:34.776029+0000 mgr.y (mgr.14144) 409 : cluster [DBG] pgmap v589: 168 pgs: 168 active+clean; 133 MiB data, 515 MiB used, 536 GiB / 536 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 24 op/s; 9.8 MiB/s, 21 objects/s recovering 2021-03-24T21:57:36.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:36 smithi125 conmon[19936]: cluster 2021-03-24T21:57:36.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:36 smithi125 conmon[19936]: 2021-03-24T21: 2021-03-24T21:57:36.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:36 smithi125 conmon[19936]: 57:35.188866+0000 2021-03-24T21:57:36.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:36 smithi125 conmon[19936]: mon.a (mon.0) 945 : 2021-03-24T21:57:36.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:36 smithi125 conmon[19936]: cluster [DBG] osdmap e285: 8 total, 8 up, 6 in 2021-03-24T21:57:36.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:36 smithi125 conmon[19936]: audit 2021-03 2021-03-24T21:57:36.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:36 smithi125 conmon[19936]: -24T21:57:35.305918+0000 mon.c ( 2021-03-24T21:57:36.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:36 smithi125 conmon[19936]: mon.2) 44 : audit [DBG] 2021-03-24T21:57:36.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:36 smithi125 conmon[19936]: from='client.? v1:172.21.15.125:0/1770643375' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:57:36.492 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:36 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:57:36.492 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:36 smithi142 conmon[20125]: 24T21:57:34.776029+0000 mgr.y (mgr.14144) 409 : cluster [DBG] pgmap v589: 168 pgs: 168 active+clean; 133 MiB data, 515 MiB used, 536 GiB / 536 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 24 op/s; 9.8 MiB/s, 21 objects/s recovering 2021-03-24T21:57:36.492 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:36 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:57:36.493 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:36 smithi142 conmon[20125]: -03-24T21:57: 2021-03-24T21:57:36.493 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:36 smithi142 conmon[20125]: 35.188866+0000 mon.a ( 2021-03-24T21:57:36.493 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:36 smithi142 conmon[20125]: mon.0) 945 : cluster 2021-03-24T21:57:36.493 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:36 smithi142 conmon[20125]: [DBG] osdmap e285: 8 total, 8 up, 6 in 2021-03-24T21:57:36.494 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:36 smithi142 conmon[20125]: audit 2021-03-24T21:57:36.494 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:36 smithi142 conmon[20125]: 2021-03-24T21:57: 2021-03-24T21:57:36.494 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:36 smithi142 conmon[20125]: 35.305918+0000 mon.c (mon.2 2021-03-24T21:57:36.494 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:36 smithi142 conmon[20125]: ) 44 : audit [DBG] from='client.? v1:172.21.15.125:0/1770643375' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:57:36.533 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:36.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:36.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:57:36.589 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-24T21:57:36.589 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-24T21:57:36.589 INFO:teuthology.orchestra.run.smithi125.stdout: "complaint_time": 30, 2021-03-24T21:57:36.590 INFO:teuthology.orchestra.run.smithi125.stdout: "num_blocked_ops": 0 2021-03-24T21:57:36.590 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-24T21:57:36.640 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:36.693 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:36.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:36.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:57:36.799 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:36.852 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:36.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:36.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:57:36.958 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:36.977 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-24T21:57:37.017 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:37.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:37.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:57:37.124 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:37.183 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:37.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:37.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:37.215 INFO:tasks.rados.rados.0.smithi125.stdout:956: done (0 left) 2021-03-24T21:57:37.215 INFO:tasks.rados.rados.0.smithi125.stdout:957: copy_from oid 22 from oid 47 current snap is 91 2021-03-24T21:57:37.216 INFO:tasks.rados.rados.0.smithi125.stdout:958: read oid 27 snap 90 2021-03-24T21:57:37.216 INFO:tasks.rados.rados.0.smithi125.stdout:958: expect (ObjNum 193 snap 42 seq_num 193) 2021-03-24T21:57:37.216 INFO:tasks.rados.rados.0.smithi125.stdout:959: write (excl) oid 44 current snap is 91 2021-03-24T21:57:37.216 INFO:tasks.rados.rados.0.smithi125.stdout:959: seq_num 358 ranges {416906=407733,1366083=511456} 2021-03-24T21:57:37.223 INFO:tasks.rados.rados.0.smithi125.stdout:959: writing smithi12556470-44 from 416906 to 824639 tid 1 2021-03-24T21:57:37.229 INFO:tasks.rados.rados.0.smithi125.stdout:959: writing smithi12556470-44 from 1366083 to 1877539 tid 2 2021-03-24T21:57:37.229 INFO:tasks.rados.rados.0.smithi125.stdout:960: write oid 10 current snap is 91 2021-03-24T21:57:37.230 INFO:tasks.rados.rados.0.smithi125.stdout:960: seq_num 359 ranges {433713=482982,1424565=609148} 2021-03-24T21:57:37.235 INFO:tasks.rados.rados.0.smithi125.stdout:960: writing smithi12556470-10 from 433713 to 916695 tid 1 2021-03-24T21:57:37.241 INFO:tasks.rados.rados.0.smithi125.stdout:960: writing smithi12556470-10 from 1424565 to 2033713 tid 2 2021-03-24T21:57:37.242 INFO:tasks.rados.rados.0.smithi125.stdout:961: snap_remove snap 88 2021-03-24T21:57:37.285 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:37 smithi125 conmon[19936]: cluster 2021-03-24T21:57:36.206909+0000 mon.a (mon.0) 946 : cluster [DBG] osdmap e286: 8 total, 8 up, 6 in 2021-03-24T21:57:37.285 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:37 smithi125 conmon[19936]: 2021-03-24T21:57:37.286 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:37 smithi125 conmon[24927]: cluster 2021-03-24T21:57:37.286 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:37 smithi125 conmon[24927]: 2021- 2021-03-24T21:57:37.287 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:37 smithi125 conmon[24927]: 03-24T21:57:36.206909+0000 mon.a (mon.0) 946 : cluster [DBG] osdmap e286: 8 total, 8 up, 6 in 2021-03-24T21:57:37.289 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:37.300 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:37 smithi142 conmon[20125]: cluster 2021-03-24T21:57:36.206909+0000 mon.a 2021-03-24T21:57:37.300 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:37 smithi142 conmon[20125]: (mon.0) 946 : cluster [DBG] osdmap e286: 8 total, 8 up, 6 in 2021-03-24T21:57:37.340 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:37.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:37.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:37.447 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:37.505 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:37.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:37.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:57:37.547 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.3 dump_historic_ops 2021-03-24T21:57:37.570 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:57:37.612 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:37.663 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:37.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:37.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:57:37.767 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:37.826 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:37.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:37.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:37.932 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:37.990 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:37.995 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:57:37.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:37.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:57:38.099 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:38.157 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:38.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:38.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:57:38.231 INFO:tasks.rados.rados.0.smithi125.stdout:961: done (4 left) 2021-03-24T21:57:38.231 INFO:tasks.rados.rados.0.smithi125.stdout:962: snap_remove snap 89 2021-03-24T21:57:38.262 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:38.312 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:38.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:38.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:57:38.410 INFO:journalctl@ceph.osd.3.smithi125.stdout:Mar 24 21:57:38 smithi125 conmon[37997]: debug 2021-03-24T21:57:38.251+0000 7f1716bda700 -1 osd.3 pg_epoch: 288 pg[7.14( v 285'199 (0'0,285'199] local-lis/les=204/205 n=4 ec=198/99 lis/c=204/204 les/c/f=205/205/0 sis=204) [3,2,5] r=0 lpr=204 crt=285'199 lcod 284'197 mlcod 284'197 active+clean+snaptrim trimq=[b1~1] ps=[93~1,97~1,99~1,a0~1,a2~1,a7~3,ae~1,b0~1]] removing snap head 2021-03-24T21:57:38.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:38 smithi125 conmon[19936]: cluster 2021-03-24T21:57:36 2021-03-24T21:57:38.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:38 smithi125 conmon[19936]: .776947+0000 mgr.y (mgr.14144) 410 : cluster [DBG] pgmap v592: 168 pgs: 168 active+clean; 136 MiB data, 525 MiB used, 536 GiB / 536 GiB avail; 14 MiB/s rd, 6.6 MiB/s wr, 30 op/s; 9.7 MiB/s, 21 objects/s recovering 2021-03-24T21:57:38.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:38 smithi125 conmon[19936]: cluster 2021-03-24T21:57:37.211488+0000 mon.a 2021-03-24T21:57:38.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:38 smithi125 conmon[19936]: (mon.0) 947 : cluster [DBG] osdmap e287: 8 total, 8 up, 6 in 2021-03-24T21:57:38.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:38 smithi125 conmon[24927]: cluster 2021-03-24T21:57:36. 2021-03-24T21:57:38.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:38 smithi125 conmon[24927]: 776947+0000 mgr.y (mgr.14144) 410 : cluster [DBG] pgmap v592: 168 pgs: 168 active+clean; 136 MiB data, 525 MiB used, 536 GiB / 536 GiB avail; 14 MiB/s rd, 6.6 MiB/s wr, 30 op/s; 9.7 MiB/s, 21 objects/s recovering 2021-03-24T21:57:38.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:38 smithi125 conmon[24927]: cluster 2021-03-24T21:57:37. 2021-03-24T21:57:38.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:38 smithi125 conmon[24927]: 211488+0000 mon.a (mon.0) 947 : cluster [DBG] osdmap e287: 8 total, 8 up, 6 in 2021-03-24T21:57:38.418 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:38.476 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:38.479 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:38 smithi142 conmon[20125]: cluster 2021-03-24T21:57:36.776947+0000 mgr.y (mgr.14144) 410 : cluster 2021-03-24T21:57:38.479 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:38 smithi142 conmon[20125]: [DBG] pgmap v592: 168 pgs: 168 active+clean; 136 MiB data, 525 MiB used, 536 GiB / 536 GiB avail; 14 MiB/s rd, 6.6 MiB/s wr, 30 op/s; 9.7 MiB/s, 21 objects/s recovering 2021-03-24T21:57:38.479 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:38 smithi142 conmon[20125]: cluster 2021-03-24T21:57:37.211488+0000 mon.a (mon.0) 2021-03-24T21:57:38.480 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:38 smithi142 conmon[20125]: 947 : cluster [DBG] osdmap e287: 8 total, 8 up, 6 in 2021-03-24T21:57:38.480 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 21:57:38 smithi142 conmon[27337]: debug 2021-03-24T21:57:38.480 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 21:57:38 smithi142 conmon[27337]: 2021-03-24T21:57:38.230+0000 7f549af13700 -1 osd.5 pg_epoch: 288 pg[7.10( v 285'27 (0'0,285'27] local-lis/les=256/257 n=2 ec=198/99 lis/c=256/256 les/c/f=257/257/0 sis=256) [5,3,4] r=0 lpr=256 crt=285'27 lcod 281'25 mlcod 281'25 active+clean+snaptrim trimq=[b1~1] ps=[93~1,97~1,99~1,a0~1,a2~1,a7~3,ae~1,b0~1]] removing snap head 2021-03-24T21:57:38.480 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 21:57:38 smithi142 conmon[27337]: 2021-03-24T21:57:38.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:38.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:57:38.583 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:38.643 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:38.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:38.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:57:38.749 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:38.813 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:38.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:38.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:57:38.919 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:38.972 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:38.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:38.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:57:39.078 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:39.127 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:39.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:39.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:57:39.233 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:39.234 INFO:tasks.rados.rados.0.smithi125.stdout:962: done (4 left) 2021-03-24T21:57:39.235 INFO:tasks.rados.rados.0.smithi125.stdout:963: snap_create 2021-03-24T21:57:39.235 INFO:tasks.rados.rados.0.smithi125.stdout:958: expect (ObjNum 193 snap 42 seq_num 193) 2021-03-24T21:57:39.279 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:39.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:39.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:57:39.307 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-24T21:57:39.385 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:39.431 INFO:tasks.rados.rados.0.smithi125.stdout:959: finishing write tid 1 to smithi12556470-44 2021-03-24T21:57:39.431 INFO:tasks.rados.rados.0.smithi125.stdout:959: finishing write tid 2 to smithi12556470-44 2021-03-24T21:57:39.432 INFO:tasks.rados.rados.0.smithi125.stdout:959: finishing write tid 3 to smithi12556470-44 2021-03-24T21:57:39.432 INFO:tasks.rados.rados.0.smithi125.stdout:959: finishing write tid 4 to smithi12556470-44 2021-03-24T21:57:39.432 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 44 v 269 (ObjNum 358 snap 91 seq_num 358) dirty exists 2021-03-24T21:57:39.432 INFO:tasks.rados.rados.0.smithi125.stdout:959: left oid 44 (ObjNum 358 snap 91 seq_num 358) 2021-03-24T21:57:39.432 INFO:tasks.rados.rados.0.smithi125.stdout:960: finishing write tid 1 to smithi12556470-10 2021-03-24T21:57:39.433 INFO:tasks.rados.rados.0.smithi125.stdout:957: finishing copy_from to smithi12556470-22 2021-03-24T21:57:39.433 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 22 v 100 (ObjNum 342 snap 87 seq_num 342) dirty exists 2021-03-24T21:57:39.433 INFO:tasks.rados.rados.0.smithi125.stdout:957: finishing copy_from racing read to smithi12556470-22 2021-03-24T21:57:39.433 INFO:tasks.rados.rados.0.smithi125.stdout:960: finishing write tid 2 to smithi12556470-10 2021-03-24T21:57:39.433 INFO:tasks.rados.rados.0.smithi125.stdout:960: finishing write tid 3 to smithi12556470-10 2021-03-24T21:57:39.434 INFO:tasks.rados.rados.0.smithi125.stdout:960: finishing write tid 4 to smithi12556470-10 2021-03-24T21:57:39.434 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 10 v 313 (ObjNum 359 snap 91 seq_num 359) dirty exists 2021-03-24T21:57:39.434 INFO:tasks.rados.rados.0.smithi125.stdout:960: left oid 10 (ObjNum 359 snap 91 seq_num 359) 2021-03-24T21:57:39.441 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:39.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:39.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:57:39.533 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:39 smithi125 conmon[19936]: cluster 2021-03-24T21:57:39.534 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:39 smithi125 conmon[19936]: 2021-03-24T21:57:39.534 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:39 smithi125 conmon[19936]: 2021 2021-03-24T21:57:39.534 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:39 smithi125 conmon[19936]: - 2021-03-24T21:57:39.534 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:39 smithi125 conmon[19936]: 03 2021-03-24T21:57:39.535 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:39 smithi125 conmon[19936]: - 2021-03-24T21:57:39.535 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:39 smithi125 conmon[19936]: 24 2021-03-24T21:57:39.535 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:39 smithi125 conmon[19936]: T 2021-03-24T21:57:39.535 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:39 smithi125 conmon[19936]: 21 2021-03-24T21:57:39.535 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:39 smithi125 conmon[19936]: : 2021-03-24T21:57:39.536 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:39 smithi125 conmon[19936]: 57 2021-03-24T21:57:39.536 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:39 smithi125 conmon[19936]: :38.229927+0000 mon.a (mon.0) 948 : cluster [DBG] osdmap e288: 8 total, 8 up, 6 in 2021-03-24T21:57:39.536 INFO:journalctl@ceph.osd.0.smithi125.stdout:Mar 24 21:57:39 smithi125 conmon[28026]: debug 2021-03-24T21:57:39.247+0000 7f61b847f700 -1 osd.0 pg_epoch: 289 pg[7.13( v 288'236 (0'0,288'236] local-lis/les=256/257 n=4 ec=198/99 lis/c=256/256 les/c/f=257/257/0 sis=256) [0,7,5] r=0 lpr=256 crt=288'236 lcod 286'234 mlcod 286'234 active+clean+snaptrim trimq=[b3~1] ps=[93~1,97~1,99~1,a0~1,a2~1,a7~3,ae~1,b0~2]] removing snap head 2021-03-24T21:57:39.537 INFO:journalctl@ceph.osd.3.smithi125.stdout:Mar 24 21:57:39 smithi125 conmon[37997]: debug 2021-03-24T21:57:39.255+0000 7f1716bda700 -1 osd.3 pg_epoch: 289 pg[7.c( v 288'268 (0'0,288'268] local-lis/les=275/276 n=2 ec=99/99 lis/c=275/275 les/c/f=276/276/0 sis=275) [3,7,2] r=0 lpr=275 crt=288'268 lcod 286'266 mlcod 286'266 active+clean+snaptrim trimq=[b3~1] ps=[93~1,97~1,99~1,a0~1,a2~1,a7~3,ae~1,b0~2]] removing snap head 2021-03-24T21:57:39.537 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:39 smithi125 conmon[24927]: cluster 2021-03-24T21:57:39.537 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:39 smithi125 conmon[24927]: 2021-03-24T21:57:39.538 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:39 smithi125 conmon[24927]: 2021 2021-03-24T21:57:39.538 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:39 smithi125 conmon[24927]: - 2021-03-24T21:57:39.538 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:39 smithi125 conmon[24927]: 03 2021-03-24T21:57:39.538 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:39 smithi125 conmon[24927]: - 2021-03-24T21:57:39.539 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:39 smithi125 conmon[24927]: 24 2021-03-24T21:57:39.539 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:39 smithi125 conmon[24927]: T 2021-03-24T21:57:39.539 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:39 smithi125 conmon[24927]: 21 2021-03-24T21:57:39.539 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:39 smithi125 conmon[24927]: : 2021-03-24T21:57:39.539 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:39 smithi125 conmon[24927]: 57 2021-03-24T21:57:39.540 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:39 smithi125 conmon[24927]: : 2021-03-24T21:57:39.540 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:39 smithi125 conmon[24927]: 38 2021-03-24T21:57:39.540 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:39 smithi125 conmon[24927]: . 2021-03-24T21:57:39.540 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:39 smithi125 conmon[24927]: 229927 2021-03-24T21:57:39.540 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:39 smithi125 conmon[24927]: +0000 2021-03-24T21:57:39.541 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:39 smithi125 conmon[24927]: 2021-03-24T21:57:39.541 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:39 smithi125 conmon[24927]: mon.a 2021-03-24T21:57:39.541 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:39 smithi125 conmon[24927]: ( 2021-03-24T21:57:39.541 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:39 smithi125 conmon[24927]: mon 2021-03-24T21:57:39.541 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:39 smithi125 conmon[24927]: . 2021-03-24T21:57:39.542 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:39 smithi125 conmon[24927]: 0 2021-03-24T21:57:39.542 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:39 smithi125 conmon[24927]: ) 2021-03-24T21:57:39.542 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:39 smithi125 conmon[24927]: 948 2021-03-24T21:57:39.542 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:39 smithi125 conmon[24927]: : 2021-03-24T21:57:39.542 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:39 smithi125 conmon[24927]: cluster 2021-03-24T21:57:39.543 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:39 smithi125 conmon[24927]: 2021-03-24T21:57:39.543 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:39 smithi125 conmon[24927]: [DBG] 2021-03-24T21:57:39.543 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:39 smithi125 conmon[24927]: 2021-03-24T21:57:39.543 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:39 smithi125 conmon[24927]: osdmap e288: 8 total, 8 up, 6 in 2021-03-24T21:57:39.544 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:39 smithi125 conmon[24927]: 2021-03-24T21:57:39.548 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:39.561 INFO:journalctl@ceph.osd.4.smithi142.stdout:Mar 24 21:57:39 smithi142 conmon[24076]: debug 2021-03-24T21:57:39.562 INFO:journalctl@ceph.osd.4.smithi142.stdout:Mar 24 21:57:39 smithi142 conmon[24076]: 2021-03-24T21:57:39.230+0000 7fc55d62f700 -1 osd.4 pg_epoch: 289 pg[7.22( v 288'70 (0'0,288'70] local-lis/les=277/278 n=1 ec=275/99 lis/c=277/277 les/c/f=278/278/0 sis=277) [4,5,0] r=0 lpr=277 crt=288'70 lcod 286'68 mlcod 286'68 active+clean+snaptrim trimq=[b3~1] ps=[93~1,97~1,99~1,a0~1,a2~1,a7~3,ae~1,b0~2]] removing snap head 2021-03-24T21:57:39.562 INFO:journalctl@ceph.osd.4.smithi142.stdout:Mar 24 21:57:39 smithi142 conmon[24076]: 2021-03-24T21:57:39.562 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:39 smithi142 conmon[20125]: cluster 2021-03-24T21:57:38.229927+0000 2021-03-24T21:57:39.562 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:39 smithi142 conmon[20125]: mon.a (mon.0) 948 : cluster [DBG] osdmap e288: 8 total, 8 up, 6 in 2021-03-24T21:57:39.604 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:39.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:39.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:57:39.711 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:39.771 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:39.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:39.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:57:39.878 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:39.935 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:39.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:39.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:57:39.979 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 7] out_osds: [6, 1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-03-24T21:57:39.979 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-24T21:57:39.980 INFO:tasks.thrashosds.thrasher:Shrinking pool iscsi 2021-03-24T21:57:39.980 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool iscsi by 10 min 8 2021-03-24T21:57:39.981 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph pg dump --format=json 2021-03-24T21:57:40.042 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:40.093 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:40.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:40.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:57:40.161 INFO:tasks.rados.rados.0.smithi125.stdout:957: done (4 left) 2021-03-24T21:57:40.162 INFO:tasks.rados.rados.0.smithi125.stdout:958: done (3 left) 2021-03-24T21:57:40.162 INFO:tasks.rados.rados.0.smithi125.stdout:959: done (2 left) 2021-03-24T21:57:40.162 INFO:tasks.rados.rados.0.smithi125.stdout:960: done (1 left) 2021-03-24T21:57:40.163 INFO:tasks.rados.rados.0.smithi125.stdout:963: done (0 left) 2021-03-24T21:57:40.163 INFO:tasks.rados.rados.0.smithi125.stdout:964: write (excl) oid 15 current snap is 92 2021-03-24T21:57:40.163 INFO:tasks.rados.rados.0.smithi125.stdout:964: seq_num 360 ranges {450520=558231,1483047=515125} 2021-03-24T21:57:40.169 INFO:tasks.rados.rados.0.smithi125.stdout:964: writing smithi12556470-15 from 450520 to 1008751 tid 1 2021-03-24T21:57:40.181 INFO:tasks.rados.rados.0.smithi125.stdout:964: writing smithi12556470-15 from 1483047 to 1998172 tid 2 2021-03-24T21:57:40.181 INFO:tasks.rados.rados.0.smithi125.stdout:965: read oid 42 snap -1 2021-03-24T21:57:40.182 INFO:tasks.rados.rados.0.smithi125.stdout:965: expect deleted 2021-03-24T21:57:40.182 INFO:tasks.rados.rados.0.smithi125.stdout:966: write oid 41 current snap is 92 2021-03-24T21:57:40.182 INFO:tasks.rados.rados.0.smithi125.stdout:964: finishing write tid 1 to smithi12556470-15 2021-03-24T21:57:40.183 INFO:tasks.rados.rados.0.smithi125.stdout:966: seq_num 361 ranges {467327=633480,1825176=242151} 2021-03-24T21:57:40.188 INFO:tasks.rados.rados.0.smithi125.stdout:966: writing smithi12556470-41 from 467327 to 1100807 tid 1 2021-03-24T21:57:40.194 INFO:tasks.rados.rados.0.smithi125.stdout:966: writing smithi12556470-41 from 1825176 to 2067327 tid 2 2021-03-24T21:57:40.194 INFO:tasks.rados.rados.0.smithi125.stdout:967: read oid 43 snap -1 2021-03-24T21:57:40.195 INFO:tasks.rados.rados.0.smithi125.stdout:967: expect deleted 2021-03-24T21:57:40.195 INFO:tasks.rados.rados.0.smithi125.stdout:968: write oid 49 current snap is 92 2021-03-24T21:57:40.195 INFO:tasks.rados.rados.0.smithi125.stdout:968: seq_num 362 ranges {484134=708729,1883658=200476} 2021-03-24T21:57:40.200 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:40.203 INFO:tasks.rados.rados.0.smithi125.stdout:968: writing smithi12556470-49 from 484134 to 1192863 tid 1 2021-03-24T21:57:40.210 INFO:tasks.rados.rados.0.smithi125.stdout:968: writing smithi12556470-49 from 1883658 to 2084134 tid 2 2021-03-24T21:57:40.210 INFO:tasks.rados.rados.0.smithi125.stdout:969: read oid 37 snap -1 2021-03-24T21:57:40.210 INFO:tasks.rados.rados.0.smithi125.stdout:969: expect (ObjNum 229 snap 50 seq_num 229) 2021-03-24T21:57:40.211 INFO:tasks.rados.rados.0.smithi125.stdout:964: finishing write tid 2 to smithi12556470-15 2021-03-24T21:57:40.211 INFO:tasks.rados.rados.0.smithi125.stdout:964: finishing write tid 3 to smithi12556470-15 2021-03-24T21:57:40.211 INFO:tasks.rados.rados.0.smithi125.stdout:964: finishing write tid 4 to smithi12556470-15 2021-03-24T21:57:40.211 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 15 v 274 (ObjNum 360 snap 92 seq_num 360) dirty exists 2021-03-24T21:57:40.212 INFO:tasks.rados.rados.0.smithi125.stdout:964: left oid 15 (ObjNum 360 snap 92 seq_num 360) 2021-03-24T21:57:40.212 INFO:tasks.rados.rados.0.smithi125.stdout:966: finishing write tid 1 to smithi12556470-41 2021-03-24T21:57:40.212 INFO:tasks.rados.rados.0.smithi125.stdout:964: done (5 left) 2021-03-24T21:57:40.212 INFO:tasks.rados.rados.0.smithi125.stdout:965: done (4 left) 2021-03-24T21:57:40.213 INFO:tasks.rados.rados.0.smithi125.stdout:970: rollback oid 26 current snap is 92 2021-03-24T21:57:40.213 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 26 to 90 2021-03-24T21:57:40.213 INFO:tasks.rados.rados.0.smithi125.stdout:967: done (4 left) 2021-03-24T21:57:40.213 INFO:tasks.rados.rados.0.smithi125.stdout:971: copy_from oid 2 from oid 39 current snap is 92 2021-03-24T21:57:40.214 INFO:tasks.rados.rados.0.smithi125.stdout:972: delete oid 45 current snap is 92 2021-03-24T21:57:40.217 INFO:tasks.rados.rados.0.smithi125.stdout:969: expect (ObjNum 229 snap 50 seq_num 229) 2021-03-24T21:57:40.252 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:40.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:40.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:57:40.356 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:40.369 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:40 smithi125 conmon[19936]: cluster 2021-03-24T21:57:38.777871+0000 mgr.y ( 2021-03-24T21:57:40.369 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:40 smithi125 conmon[19936]: mgr.14144) 411 : cluster [DBG] pgmap v595: 168 pgs: 168 active+clean; 131 MiB data, 499 MiB used, 536 GiB / 536 GiB avail; 239 KiB/s wr, 0 op/s 2021-03-24T21:57:40.370 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:40 smithi125 conmon[19936]: cluster 2021-03-24T21:57:38.937640 2021-03-24T21:57:40.370 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:40 smithi125 conmon[19936]: +0000 osd.7 (osd.7) 65 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:57:40.370 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:40 smithi125 conmon[19936]: cluster 2021-03-24T21:57:40.370 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:40 smithi125 conmon[19936]: 2021-03-24T21:57:38.938145+0000 osd.7 (osd.7) 66 : cluster [DBG] 2021-03-24T21:57:40.371 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:40 smithi125 conmon[19936]: purged_snaps scrub ok 2021-03-24T21:57:40.371 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:40 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:57:40.371 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:40 smithi125 conmon[19936]: 03-24T21:57:39.230793+0000 mon.a (mon.0) 2021-03-24T21:57:40.371 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:40 smithi125 conmon[19936]: 949 : cluster [DBG] osdmap e289: 8 total, 8 up, 6 in 2021-03-24T21:57:40.371 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:40 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:57:40.372 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:40 smithi125 conmon[19936]: :57:40.154081+0000 mon.a (mon.0) 950 : cluster 2021-03-24T21:57:40.372 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:40 smithi125 conmon[19936]: [DBG] osdmap e290: 8 total, 8 up, 6 in 2021-03-24T21:57:40.372 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:40 smithi125 conmon[24927]: cluster 2021-03-24T21:57:38.777871+0000 mgr.y (mgr.14144) 411 : cluster [DBG] pgmap v595: 168 pgs: 168 active+clean; 131 MiB data, 499 MiB used, 536 GiB / 536 GiB avail; 239 KiB/s wr, 0 op/s 2021-03-24T21:57:40.373 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:40 smithi125 conmon[24927]: cluster 2021-03-24T21:57:38.937640+0000 osd.7 (osd.7) 65 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:57:40.373 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:40 smithi125 conmon[24927]: cluster 2021-03-24T21:57:38.938145+0000 osd.7 (osd.7) 66 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:57:40.373 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:40 smithi125 conmon[24927]: cluster 2021-03-24T21:57:39.230793+0000 mon.a (mon.0) 949 : cluster [DBG] osdmap e289: 8 total, 8 up, 6 in 2021-03-24T21:57:40.373 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:40 smithi125 conmon[24927]: cluster 2021-03-24T21:57:40.154081+0000 mon.a (mon.0) 950 : cluster [DBG] osdmap e290: 8 total, 8 up, 6 in 2021-03-24T21:57:40.403 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:40.405 INFO:tasks.rados.rados.0.smithi125.stdout:966: finishing write tid 2 to smithi12556470-41 2021-03-24T21:57:40.405 INFO:tasks.rados.rados.0.smithi125.stdout:966: finishing write tid 3 to smithi12556470-41 2021-03-24T21:57:40.405 INFO:tasks.rados.rados.0.smithi125.stdout:966: finishing write tid 4 to smithi12556470-41 2021-03-24T21:57:40.406 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 41 v 276 (ObjNum 361 snap 92 seq_num 361) dirty exists 2021-03-24T21:57:40.406 INFO:tasks.rados.rados.0.smithi125.stdout:966: left oid 41 (ObjNum 361 snap 92 seq_num 361) 2021-03-24T21:57:40.406 INFO:tasks.rados.rados.0.smithi125.stdout:970: finishing rollback tid 0 to smithi12556470-26 2021-03-24T21:57:40.406 INFO:tasks.rados.rados.0.smithi125.stdout:970: finishing rollback tid 1 to smithi12556470-26 2021-03-24T21:57:40.406 INFO:tasks.rados.rados.0.smithi125.stdout:970: finishing rollback tid 2 to smithi12556470-26 2021-03-24T21:57:40.407 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 26 v 151 (ObjNum 321 snap 82 seq_num 321) dirty exists 2021-03-24T21:57:40.407 INFO:tasks.rados.rados.0.smithi125.stdout:968: finishing write tid 1 to smithi12556470-49 2021-03-24T21:57:40.407 INFO:tasks.rados.rados.0.smithi125.stdout:966: done (5 left) 2021-03-24T21:57:40.407 INFO:tasks.rados.rados.0.smithi125.stdout:969: done (4 left) 2021-03-24T21:57:40.408 INFO:tasks.rados.rados.0.smithi125.stdout:970: done (3 left) 2021-03-24T21:57:40.408 INFO:tasks.rados.rados.0.smithi125.stdout:972: done (2 left) 2021-03-24T21:57:40.408 INFO:tasks.rados.rados.0.smithi125.stdout:973: snap_remove snap 90 2021-03-24T21:57:40.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:40.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:57:40.510 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:40.524 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:40 smithi142 conmon[20125]: cluster 2021-03-24T21:57:38.777871+0000 mgr.y (mgr.14144) 411 : cluster 2021-03-24T21:57:40.525 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:40 smithi142 conmon[20125]: [DBG] pgmap v595: 168 pgs: 168 active+clean; 131 MiB data, 499 MiB used, 536 GiB / 536 GiB avail; 239 KiB/s wr, 0 op/s 2021-03-24T21:57:40.525 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:40 smithi142 conmon[20125]: cluster 2021-03-24T21:57:38.937640+0000 osd.7 ( 2021-03-24T21:57:40.525 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:40 smithi142 conmon[20125]: osd.7) 65 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:57:40.525 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:40 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:57:40.526 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:40 smithi142 conmon[20125]: -03-24T21:57:38.938145+0000 2021-03-24T21:57:40.526 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:40 smithi142 conmon[20125]: osd.7 (osd.7) 66 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:57:40.526 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:40 smithi142 conmon[20125]: 2021-03-24T21:57:40.526 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:40 smithi142 conmon[20125]: cluster 2021-03-24T21 2021-03-24T21:57:40.527 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:40 smithi142 conmon[20125]: :57:39.230793+0000 mon.a 2021-03-24T21:57:40.527 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:40 smithi142 conmon[20125]: (mon.0) 949 : cluster 2021-03-24T21:57:40.527 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:40 smithi142 conmon[20125]: [DBG] osdmap e289: 8 total, 8 up, 6 in 2021-03-24T21:57:40.527 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:40 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:57:40.527 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:40 smithi142 conmon[20125]: -03-24T21:57:40 2021-03-24T21:57:40.528 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:40 smithi142 conmon[20125]: .154081+0000 mon.a (mon. 2021-03-24T21:57:40.528 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:40 smithi142 conmon[20125]: 0) 950 : cluster [DBG] 2021-03-24T21:57:40.528 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:40 smithi142 conmon[20125]: osdmap e290: 8 total, 8 up, 6 in 2021-03-24T21:57:40.553 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:57:40.571 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:40.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:40.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:57:40.646 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:57:40.676 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:40.729 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:40.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:40.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:57:40.835 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:40.893 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:40.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:40.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:57:41.000 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:41.010 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-24T21:57:41.010 INFO:teuthology.orchestra.run.smithi125.stdout: "size": 20, 2021-03-24T21:57:41.010 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 600, 2021-03-24T21:57:41.011 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [ 2021-03-24T21:57:41.011 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.011 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:213 7.3 7:c1eb355b:::smithi12556470-34:head [delete] snapc 2=[2] ondisk+write+peerstat_old+known_if_redirected e108)", 2021-03-24T21:57:41.011 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:37.086973+0000", 2021-03-24T21:57:41.011 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 183.91885360800001, 2021-03-24T21:57:41.012 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.033989375000000002, 2021-03-24T21:57:41.012 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:57:41.012 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:57:41.012 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:57:41.013 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:57:41.013 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:57:41.013 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 213 2021-03-24T21:57:41.013 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.013 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:57:41.014 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.014 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:57:41.014 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:37.086973+0000", 2021-03-24T21:57:41.015 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:57:41.015 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.015 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.015 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:57:41.015 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:37.086973+0000", 2021-03-24T21:57:41.016 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999943 2021-03-24T21:57:41.016 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.016 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.016 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:57:41.017 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:37.086967+0000", 2021-03-24T21:57:41.017 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.2189e-05 2021-03-24T21:57:41.017 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.017 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.017 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:57:41.018 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:37.086989+0000", 2021-03-24T21:57:41.018 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.4728999999999999e-05 2021-03-24T21:57:41.018 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.018 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.019 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:57:41.019 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:37.087024+0000", 2021-03-24T21:57:41.019 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0747e-05 2021-03-24T21:57:41.019 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.019 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.020 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:57:41.020 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:37.087045+0000", 2021-03-24T21:57:41.020 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.6952000000000003e-05 2021-03-24T21:57:41.020 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.020 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.021 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:41.021 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:37.087142+0000", 2021-03-24T21:57:41.021 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.2420000000000003e-06 2021-03-24T21:57:41.021 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.022 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.022 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op must wait for map", 2021-03-24T21:57:41.022 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:37.087149+0000", 2021-03-24T21:57:41.022 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.023165432 2021-03-24T21:57:41.022 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.023 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.023 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:41.023 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:37.110314+0000", 2021-03-24T21:57:41.023 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.3443999999999995e-05 2021-03-24T21:57:41.023 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.024 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.024 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:57:41.024 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:37.110408+0000", 2021-03-24T21:57:41.024 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00021713799999999999 2021-03-24T21:57:41.024 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.025 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.025 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,4", 2021-03-24T21:57:41.025 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:37.110625+0000", 2021-03-24T21:57:41.025 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0049017790000000002 2021-03-24T21:57:41.026 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.026 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.026 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:57:41.026 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:37.115527+0000", 2021-03-24T21:57:41.026 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0022026319999999999 2021-03-24T21:57:41.027 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.027 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.027 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:57:41.027 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:37.117729+0000", 2021-03-24T21:57:41.028 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0031449519999999999 2021-03-24T21:57:41.028 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.028 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.028 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:57:41.028 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:37.120874+0000", 2021-03-24T21:57:41.029 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6164e-05 2021-03-24T21:57:41.029 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.029 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.029 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:57:41.029 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:37.120920+0000", 2021-03-24T21:57:41.030 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.1894000000000002e-05 2021-03-24T21:57:41.030 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.030 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.030 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:57:41.031 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:37.120962+0000", 2021-03-24T21:57:41.031 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.033989375000000002 2021-03-24T21:57:41.031 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:41.031 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:57:41.031 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:41.032 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.032 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.032 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:434 7.1 e123/108 7:86662589:::smithi12556470-19:head v 123'25, mlcod=120'21)", 2021-03-24T21:57:41.032 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:52.103706+0000", 2021-03-24T21:57:41.032 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 168.90212053900001, 2021-03-24T21:57:41.033 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.026400651000000001, 2021-03-24T21:57:41.033 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:57:41.033 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:57:41.033 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:57:41.034 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.034 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:57:41.034 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.103706+0000", 2021-03-24T21:57:41.034 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:57:41.034 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.035 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.035 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:57:41.035 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.103706+0000", 2021-03-24T21:57:41.035 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T21:57:41.036 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.036 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.036 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:57:41.036 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.103704+0000", 2021-03-24T21:57:41.036 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2787e-05 2021-03-24T21:57:41.037 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.037 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.037 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:57:41.037 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.103717+0000", 2021-03-24T21:57:41.037 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.024831961 2021-03-24T21:57:41.038 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.038 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.038 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:57:41.038 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.128549+0000", 2021-03-24T21:57:41.039 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0806e-05 2021-03-24T21:57:41.039 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.039 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.039 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:57:41.039 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.128559+0000", 2021-03-24T21:57:41.040 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2800999999999998e-05 2021-03-24T21:57:41.040 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.040 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.040 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:41.041 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.128612+0000", 2021-03-24T21:57:41.041 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.5789999999999999e-05 2021-03-24T21:57:41.041 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.041 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.041 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:57:41.042 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.128638+0000", 2021-03-24T21:57:41.042 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001432589 2021-03-24T21:57:41.042 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.042 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.042 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:57:41.043 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.130071+0000", 2021-03-24T21:57:41.043 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.6034000000000001e-05 2021-03-24T21:57:41.043 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.043 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.043 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:57:41.044 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:52.130107+0000", 2021-03-24T21:57:41.044 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.026400651000000001 2021-03-24T21:57:41.044 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:41.044 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:57:41.045 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:41.045 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.045 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.045 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:645 7.c 7:305739dc:::smithi12556470-36:head [append 0~0] snapc 23=[23,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e141)", 2021-03-24T21:57:41.045 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:55:10.121969+0000", 2021-03-24T21:57:41.046 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 150.883857869, 2021-03-24T21:57:41.046 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.029700717000000001, 2021-03-24T21:57:41.046 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:57:41.046 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:57:41.047 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:57:41.047 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:57:41.047 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:57:41.047 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 645 2021-03-24T21:57:41.047 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.048 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:57:41.048 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.048 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:57:41.048 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.121969+0000", 2021-03-24T21:57:41.048 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:57:41.049 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.049 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.049 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:57:41.049 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.121969+0000", 2021-03-24T21:57:41.049 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T21:57:41.050 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.050 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.050 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:57:41.050 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.121968+0000", 2021-03-24T21:57:41.051 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.7299999999999996e-06 2021-03-24T21:57:41.051 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.051 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.051 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:57:41.051 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.121972+0000", 2021-03-24T21:57:41.053 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2269999999999993e-06 2021-03-24T21:57:41.053 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.053 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.054 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:57:41.054 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.121980+0000", 2021-03-24T21:57:41.054 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.7979999999999996e-06 2021-03-24T21:57:41.054 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.055 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.055 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:57:41.055 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.121985+0000", 2021-03-24T21:57:41.055 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4379000000000001e-05 2021-03-24T21:57:41.055 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.056 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.056 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:41.056 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122050+0000", 2021-03-24T21:57:41.056 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000171507 2021-03-24T21:57:41.056 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.057 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.057 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:57:41.057 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122221+0000", 2021-03-24T21:57:41.057 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00017259999999999999 2021-03-24T21:57:41.058 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.058 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.058 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,7", 2021-03-24T21:57:41.063 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122394+0000", 2021-03-24T21:57:41.063 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012748140999999999 2021-03-24T21:57:41.064 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.064 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.064 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:57:41.064 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.135142+0000", 2021-03-24T21:57:41.065 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.014595599000000001 2021-03-24T21:57:41.065 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.065 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.065 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:57:41.065 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.149737+0000", 2021-03-24T21:57:41.066 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0017735170000000001 2021-03-24T21:57:41.066 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.066 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.066 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:57:41.067 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.151511+0000", 2021-03-24T21:57:41.067 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.3142999999999999e-05 2021-03-24T21:57:41.067 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.067 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.068 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:57:41.068 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.151604+0000", 2021-03-24T21:57:41.068 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.5141000000000005e-05 2021-03-24T21:57:41.068 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.068 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.069 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:57:41.069 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.151669+0000", 2021-03-24T21:57:41.069 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.029700717000000001 2021-03-24T21:57:41.069 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:41.069 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:57:41.070 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:41.070 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.070 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.070 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:646 7.c 7:305739dc:::smithi12556470-36:head [rollback 15] snapc 23=[23,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e141)", 2021-03-24T21:57:41.071 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:55:10.122086+0000", 2021-03-24T21:57:41.071 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 150.88374020000001, 2021-03-24T21:57:41.071 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.030078549999999999, 2021-03-24T21:57:41.071 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:57:41.071 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:57:41.072 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:57:41.072 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:57:41.072 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:57:41.072 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 646 2021-03-24T21:57:41.073 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.073 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:57:41.073 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.073 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:57:41.073 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122086+0000", 2021-03-24T21:57:41.074 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:57:41.074 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.074 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.074 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:57:41.075 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122086+0000", 2021-03-24T21:57:41.075 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T21:57:41.075 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.075 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.075 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:57:41.076 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122085+0000", 2021-03-24T21:57:41.076 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6999999999999996e-06 2021-03-24T21:57:41.076 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.076 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.076 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:57:41.077 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122091+0000", 2021-03-24T21:57:41.077 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.149e-06 2021-03-24T21:57:41.077 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.077 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.077 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:57:41.078 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122100+0000", 2021-03-24T21:57:41.078 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.4719999999999998e-06 2021-03-24T21:57:41.078 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.078 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.079 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:57:41.079 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122108+0000", 2021-03-24T21:57:41.079 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00083429299999999995 2021-03-24T21:57:41.079 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.079 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.080 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:41.080 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122942+0000", 2021-03-24T21:57:41.080 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000113615 2021-03-24T21:57:41.080 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.081 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.081 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:57:41.081 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.123055+0000", 2021-03-24T21:57:41.081 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00024782800000000001 2021-03-24T21:57:41.081 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.082 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.082 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,7", 2021-03-24T21:57:41.082 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.123303+0000", 2021-03-24T21:57:41.082 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012182396 2021-03-24T21:57:41.082 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.083 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.083 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:57:41.083 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.135486+0000", 2021-03-24T21:57:41.083 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.016421525999999999 2021-03-24T21:57:41.084 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.084 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.084 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:57:41.084 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.151907+0000", 2021-03-24T21:57:41.084 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.3577000000000003e-05 2021-03-24T21:57:41.085 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.085 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.085 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:57:41.085 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.151991+0000", 2021-03-24T21:57:41.085 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.6976000000000001e-05 2021-03-24T21:57:41.086 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.086 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.086 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:57:41.086 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.152088+0000", 2021-03-24T21:57:41.087 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.7041999999999999e-05 2021-03-24T21:57:41.087 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.087 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.087 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:57:41.087 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.152165+0000", 2021-03-24T21:57:41.088 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.030078549999999999 2021-03-24T21:57:41.088 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:41.088 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:57:41.088 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:41.088 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.089 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.089 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:647 7.c 7:305739dc:::smithi12556470-36:head [append 0~0] snapc 23=[23,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e141)", 2021-03-24T21:57:41.089 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:55:10.122139+0000", 2021-03-24T21:57:41.089 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 150.88368785899999, 2021-03-24T21:57:41.090 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.030182674, 2021-03-24T21:57:41.090 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:57:41.090 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:57:41.090 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:57:41.090 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:57:41.091 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:57:41.091 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 647 2021-03-24T21:57:41.091 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.091 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:57:41.092 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.092 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:57:41.092 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122139+0000", 2021-03-24T21:57:41.092 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:57:41.092 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.093 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.093 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:57:41.093 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122139+0000", 2021-03-24T21:57:41.093 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T21:57:41.093 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.094 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.094 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:57:41.094 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122138+0000", 2021-03-24T21:57:41.094 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.0409999999999999e-06 2021-03-24T21:57:41.095 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.095 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.095 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:57:41.095 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122142+0000", 2021-03-24T21:57:41.095 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9560000000000002e-06 2021-03-24T21:57:41.096 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.096 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.096 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:57:41.096 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122148+0000", 2021-03-24T21:57:41.096 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5709999999999999e-06 2021-03-24T21:57:41.097 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.097 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.097 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:57:41.097 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122152+0000", 2021-03-24T21:57:41.097 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0015341910000000001 2021-03-24T21:57:41.098 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.098 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.098 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:41.098 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.123686+0000", 2021-03-24T21:57:41.099 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000308213 2021-03-24T21:57:41.099 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.099 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.099 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:57:41.099 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.123995+0000", 2021-03-24T21:57:41.100 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00017020199999999999 2021-03-24T21:57:41.100 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.100 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.100 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,7", 2021-03-24T21:57:41.101 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124165+0000", 2021-03-24T21:57:41.101 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011385657 2021-03-24T21:57:41.101 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.101 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.101 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:57:41.102 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.135551+0000", 2021-03-24T21:57:41.102 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.016412598 2021-03-24T21:57:41.102 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.102 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.103 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:57:41.103 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.151963+0000", 2021-03-24T21:57:41.103 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00021537100000000001 2021-03-24T21:57:41.103 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.103 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.104 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:57:41.104 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.152179+0000", 2021-03-24T21:57:41.104 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.0764000000000001e-05 2021-03-24T21:57:41.104 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.104 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.105 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:57:41.105 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.152259+0000", 2021-03-24T21:57:41.105 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.1952999999999995e-05 2021-03-24T21:57:41.105 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.105 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.106 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:57:41.106 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.152321+0000", 2021-03-24T21:57:41.106 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.030182674 2021-03-24T21:57:41.106 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:41.107 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:57:41.107 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:41.107 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.107 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.107 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:639 7.7 e141/108 7:ef6d6bee:::smithi12556470-24:head v 141'70, mlcod=141'68)", 2021-03-24T21:57:41.108 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:55:10.122763+0000", 2021-03-24T21:57:41.108 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 150.883063228, 2021-03-24T21:57:41.108 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.029224018000000001, 2021-03-24T21:57:41.108 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:57:41.108 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:57:41.109 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:57:41.109 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.109 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:57:41.109 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122763+0000", 2021-03-24T21:57:41.109 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:57:41.110 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.110 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.110 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:57:41.110 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122763+0000", 2021-03-24T21:57:41.110 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T21:57:41.111 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.111 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.111 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:57:41.111 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122762+0000", 2021-03-24T21:57:41.112 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00019497300000000001 2021-03-24T21:57:41.112 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.112 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.112 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:57:41.112 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122957+0000", 2021-03-24T21:57:41.113 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2621e-05 2021-03-24T21:57:41.113 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.113 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.113 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:57:41.113 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122969+0000", 2021-03-24T21:57:41.114 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.1149999999999994e-06 2021-03-24T21:57:41.114 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.114 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.114 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:57:41.114 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.122978+0000", 2021-03-24T21:57:41.115 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.1203000000000003e-05 2021-03-24T21:57:41.115 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.115 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.115 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:41.116 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.123039+0000", 2021-03-24T21:57:41.116 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.6356000000000001e-05 2021-03-24T21:57:41.116 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.116 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.116 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:57:41.117 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.123055+0000", 2021-03-24T21:57:41.117 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.028884112 2021-03-24T21:57:41.117 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.117 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.117 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:57:41.118 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.151939+0000", 2021-03-24T21:57:41.118 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.8056999999999997e-05 2021-03-24T21:57:41.118 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.118 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.119 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:57:41.119 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.151987+0000", 2021-03-24T21:57:41.119 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.029224018000000001 2021-03-24T21:57:41.119 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:41.119 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:57:41.120 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:41.120 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.120 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.120 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:640 7.7 e141/108 7:ef6d6bee:::smithi12556470-24:head v 141'71, mlcod=141'68)", 2021-03-24T21:57:41.120 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:55:10.123846+0000", 2021-03-24T21:57:41.121 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 150.88198093700001, 2021-03-24T21:57:41.121 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.028164842999999998, 2021-03-24T21:57:41.121 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:57:41.121 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:57:41.121 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:57:41.122 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.122 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:57:41.122 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.123846+0000", 2021-03-24T21:57:41.122 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:57:41.123 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.123 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.123 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:57:41.123 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.123846+0000", 2021-03-24T21:57:41.123 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T21:57:41.124 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.124 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.124 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:57:41.124 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.123844+0000", 2021-03-24T21:57:41.124 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7825e-05 2021-03-24T21:57:41.125 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.125 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.125 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:57:41.125 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.123862+0000", 2021-03-24T21:57:41.125 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.4489999999999999e-06 2021-03-24T21:57:41.126 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.126 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.126 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:57:41.126 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.123870+0000", 2021-03-24T21:57:41.127 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1789999999999999e-06 2021-03-24T21:57:41.127 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.127 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.127 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:57:41.127 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.123876+0000", 2021-03-24T21:57:41.128 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6403999999999997e-05 2021-03-24T21:57:41.128 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.128 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.128 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:41.128 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.123932+0000", 2021-03-24T21:57:41.129 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9221000000000001e-05 2021-03-24T21:57:41.129 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.129 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.129 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:57:41.130 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.123951+0000", 2021-03-24T21:57:41.130 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.028040790999999999 2021-03-24T21:57:41.130 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.130 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.130 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:57:41.131 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.151992+0000", 2021-03-24T21:57:41.131 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.8314000000000001e-05 2021-03-24T21:57:41.131 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.131 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.131 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:57:41.132 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.152010+0000", 2021-03-24T21:57:41.132 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.028164842999999998 2021-03-24T21:57:41.132 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:41.132 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:57:41.132 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:41.133 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.133 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.133 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:1230 7.3 7:cb55abc9:::smithi12556470-6:head [write 410137~481384] snapc 53=[] ondisk+write+known_if_redirected e191)", 2021-03-24T21:57:41.133 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:56:00.116990+0000", 2021-03-24T21:57:41.134 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 100.88883619800001, 2021-03-24T21:57:41.134 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.098702171000000005, 2021-03-24T21:57:41.134 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:57:41.134 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:57:41.134 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:57:41.135 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:57:41.135 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:57:41.135 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1230 2021-03-24T21:57:41.135 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.135 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:57:41.136 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.136 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:57:41.136 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.116990+0000", 2021-03-24T21:57:41.136 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:57:41.136 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.137 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.137 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:57:41.137 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.116990+0000", 2021-03-24T21:57:41.137 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999962 2021-03-24T21:57:41.137 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.138 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.138 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:57:41.138 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.116986+0000", 2021-03-24T21:57:41.138 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00030943799999999999 2021-03-24T21:57:41.139 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.139 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.139 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:57:41.139 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.117296+0000", 2021-03-24T21:57:41.139 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5636000000000001e-05 2021-03-24T21:57:41.140 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.140 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.140 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:57:41.140 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.117312+0000", 2021-03-24T21:57:41.140 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.6680000000000007e-06 2021-03-24T21:57:41.141 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.141 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.141 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:57:41.141 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.117320+0000", 2021-03-24T21:57:41.141 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.4054000000000006e-05 2021-03-24T21:57:41.142 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.142 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.142 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:41.142 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.117394+0000", 2021-03-24T21:57:41.143 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2653000000000001e-05 2021-03-24T21:57:41.143 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.143 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.143 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op must wait for map", 2021-03-24T21:57:41.143 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.117407+0000", 2021-03-24T21:57:41.144 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.077794965999999993 2021-03-24T21:57:41.144 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.144 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.144 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:41.145 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.195202+0000", 2021-03-24T21:57:41.145 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00015882200000000001 2021-03-24T21:57:41.145 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.145 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.145 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:57:41.146 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.195361+0000", 2021-03-24T21:57:41.146 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00025800299999999998 2021-03-24T21:57:41.146 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.146 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.146 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,4", 2021-03-24T21:57:41.147 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.195619+0000", 2021-03-24T21:57:41.147 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.010072628 2021-03-24T21:57:41.147 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.147 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.148 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:57:41.148 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.205691+0000", 2021-03-24T21:57:41.148 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002417284 2021-03-24T21:57:41.148 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.148 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.149 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:57:41.149 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.208109+0000", 2021-03-24T21:57:41.149 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0074134020000000004 2021-03-24T21:57:41.149 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.149 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.150 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:57:41.150 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.215522+0000", 2021-03-24T21:57:41.150 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.2187000000000004e-05 2021-03-24T21:57:41.150 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.150 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.151 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:57:41.151 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.215614+0000", 2021-03-24T21:57:41.151 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.8195999999999995e-05 2021-03-24T21:57:41.151 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.151 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.152 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:57:41.152 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.215692+0000", 2021-03-24T21:57:41.152 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.098702171000000005 2021-03-24T21:57:41.152 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:41.152 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:57:41.153 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:41.153 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.153 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.153 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:1231 7.3 7:cb55abc9:::smithi12556470-6:head [write 1410296~761389] snapc 53=[] ondisk+write+known_if_redirected e191)", 2021-03-24T21:57:41.153 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:56:00.125127+0000", 2021-03-24T21:57:41.154 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 100.88069931, 2021-03-24T21:57:41.154 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.090759490999999998, 2021-03-24T21:57:41.154 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:57:41.154 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:57:41.155 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:57:41.155 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:57:41.155 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:57:41.155 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1231 2021-03-24T21:57:41.155 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.155 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:57:41.156 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.156 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:57:41.156 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.125127+0000", 2021-03-24T21:57:41.156 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:57:41.157 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.157 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.157 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:57:41.157 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.125127+0000", 2021-03-24T21:57:41.157 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T21:57:41.158 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.158 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.158 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:57:41.158 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.125125+0000", 2021-03-24T21:57:41.159 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00048102900000000001 2021-03-24T21:57:41.159 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.159 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.159 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:57:41.159 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.125606+0000", 2021-03-24T21:57:41.160 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.070621186000000002 2021-03-24T21:57:41.160 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.160 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.160 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:57:41.160 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.196227+0000", 2021-03-24T21:57:41.161 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7897000000000002e-05 2021-03-24T21:57:41.161 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.161 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.161 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:57:41.161 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.196245+0000", 2021-03-24T21:57:41.162 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001185069 2021-03-24T21:57:41.162 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.162 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.162 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:41.163 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.197430+0000", 2021-03-24T21:57:41.163 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00014529899999999999 2021-03-24T21:57:41.163 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.163 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.163 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:57:41.164 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.197576+0000", 2021-03-24T21:57:41.164 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00027430999999999999 2021-03-24T21:57:41.164 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.164 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.165 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,4", 2021-03-24T21:57:41.165 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.197850+0000", 2021-03-24T21:57:41.165 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0078862050000000003 2021-03-24T21:57:41.165 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.165 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.166 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:57:41.166 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.205736+0000", 2021-03-24T21:57:41.166 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0023783889999999999 2021-03-24T21:57:41.166 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.166 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.167 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:57:41.167 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.208115+0000", 2021-03-24T21:57:41.167 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.007646824 2021-03-24T21:57:41.167 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.168 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.168 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:57:41.168 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.215761+0000", 2021-03-24T21:57:41.168 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.2390000000000003e-05 2021-03-24T21:57:41.168 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.169 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.169 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:57:41.169 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.215834+0000", 2021-03-24T21:57:41.169 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2782999999999999e-05 2021-03-24T21:57:41.169 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.170 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.170 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:57:41.170 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.215887+0000", 2021-03-24T21:57:41.170 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.090759490999999998 2021-03-24T21:57:41.171 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:41.171 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:57:41.171 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:41.171 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.171 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.172 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:1246 7.1 e191/108 7:86662589:::smithi12556470-19:head v 191'161, mlcod=188'158)", 2021-03-24T21:57:41.172 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:56:00.160028+0000", 2021-03-24T21:57:41.172 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 100.845798243, 2021-03-24T21:57:41.172 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.047908946000000001, 2021-03-24T21:57:41.172 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:57:41.173 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:57:41.173 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:57:41.173 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.173 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:57:41.173 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.160028+0000", 2021-03-24T21:57:41.174 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:57:41.174 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.174 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.174 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:57:41.175 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.160028+0000", 2021-03-24T21:57:41.175 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999995 2021-03-24T21:57:41.175 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.175 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.175 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:57:41.176 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.160028+0000", 2021-03-24T21:57:41.176 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00184596 2021-03-24T21:57:41.176 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.176 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.176 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:57:41.177 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.161874+0000", 2021-03-24T21:57:41.177 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.043674593999999997 2021-03-24T21:57:41.177 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.177 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.177 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:57:41.178 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.205548+0000", 2021-03-24T21:57:41.178 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2570000000000007e-06 2021-03-24T21:57:41.178 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.178 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.179 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:57:41.179 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.205556+0000", 2021-03-24T21:57:41.179 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000195749 2021-03-24T21:57:41.179 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.179 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.180 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:41.180 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.205752+0000", 2021-03-24T21:57:41.180 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.7497999999999995e-05 2021-03-24T21:57:41.180 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.180 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.181 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:57:41.181 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.205840+0000", 2021-03-24T21:57:41.181 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0020601600000000001 2021-03-24T21:57:41.181 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.181 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.182 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:57:41.182 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.207900+0000", 2021-03-24T21:57:41.182 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.7367999999999999e-05 2021-03-24T21:57:41.182 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.183 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.183 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:57:41.183 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.207937+0000", 2021-03-24T21:57:41.183 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.047908946000000001 2021-03-24T21:57:41.183 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:41.184 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:57:41.184 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:41.184 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.184 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.184 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:2390 7.23 7:c49de0b4:::smithi12556470-13:head [write 430345~727947] snapc a9=[a9,a8,a7,a0] ondisk+write+known_if_redirected e275)", 2021-03-24T21:57:41.185 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:57:25.173714+0000", 2021-03-24T21:57:41.185 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 15.832112334, 2021-03-24T21:57:41.185 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.018775558, 2021-03-24T21:57:41.185 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:57:41.185 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:57:41.186 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:57:41.186 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:57:41.186 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:57:41.186 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 2390 2021-03-24T21:57:41.187 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.187 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:57:41.187 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.187 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:57:41.187 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.173714+0000", 2021-03-24T21:57:41.188 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:57:41.188 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.188 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.188 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:57:41.189 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.173714+0000", 2021-03-24T21:57:41.189 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T21:57:41.189 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.189 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.189 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:57:41.190 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.173712+0000", 2021-03-24T21:57:41.190 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00044400899999999998 2021-03-24T21:57:41.190 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.190 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.191 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:57:41.191 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.174156+0000", 2021-03-24T21:57:41.191 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0098000000000001e-05 2021-03-24T21:57:41.191 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.191 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.192 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:57:41.192 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.174167+0000", 2021-03-24T21:57:41.192 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.7410000000000003e-06 2021-03-24T21:57:41.192 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.192 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.193 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:57:41.193 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.174173+0000", 2021-03-24T21:57:41.193 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.9721999999999994e-05 2021-03-24T21:57:41.193 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.193 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.194 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:41.194 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.174243+0000", 2021-03-24T21:57:41.194 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3264e-05 2021-03-24T21:57:41.194 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.195 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.195 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T21:57:41.195 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.174256+0000", 2021-03-24T21:57:41.195 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.003266601 2021-03-24T21:57:41.195 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.196 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.196 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:41.196 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.177523+0000", 2021-03-24T21:57:41.196 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00014048700000000001 2021-03-24T21:57:41.196 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.197 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.197 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:57:41.197 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.177663+0000", 2021-03-24T21:57:41.197 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00069449699999999997 2021-03-24T21:57:41.197 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.198 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.198 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,4", 2021-03-24T21:57:41.198 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.178358+0000", 2021-03-24T21:57:41.198 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.005790636 2021-03-24T21:57:41.199 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.199 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.199 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:57:41.199 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.184148+0000", 2021-03-24T21:57:41.199 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00056901200000000003 2021-03-24T21:57:41.200 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.200 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.200 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:57:41.200 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.184717+0000", 2021-03-24T21:57:41.200 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0076483189999999998 2021-03-24T21:57:41.201 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.201 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.201 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:57:41.201 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.192366+0000", 2021-03-24T21:57:41.202 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.1732999999999999e-05 2021-03-24T21:57:41.202 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.202 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.202 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:57:41.202 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.192438+0000", 2021-03-24T21:57:41.203 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2144000000000001e-05 2021-03-24T21:57:41.203 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.203 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.203 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:57:41.203 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.192490+0000", 2021-03-24T21:57:41.204 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.018775558 2021-03-24T21:57:41.204 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:41.204 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:57:41.204 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:41.204 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.205 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.205 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:2391 7.23 7:c49de0b4:::smithi12556470-13:head [write 1570056~727439] snapc a9=[a9,a8,a7,a0] ondisk+write+known_if_redirected e275)", 2021-03-24T21:57:41.205 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:57:25.174190+0000", 2021-03-24T21:57:41.205 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 15.831636254999999, 2021-03-24T21:57:41.205 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.018522293, 2021-03-24T21:57:41.206 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:57:41.206 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:57:41.206 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:57:41.206 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:57:41.206 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:57:41.207 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 2391 2021-03-24T21:57:41.207 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.207 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:57:41.207 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.207 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:57:41.208 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.174190+0000", 2021-03-24T21:57:41.208 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:57:41.208 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.208 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.208 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:57:41.209 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.174190+0000", 2021-03-24T21:57:41.209 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T21:57:41.209 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.209 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.209 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:57:41.209 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.174189+0000", 2021-03-24T21:57:41.210 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000389556 2021-03-24T21:57:41.210 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.210 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.210 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:57:41.210 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.174578+0000", 2021-03-24T21:57:41.211 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4030000000000004e-06 2021-03-24T21:57:41.211 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.211 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.211 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:57:41.211 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.174585+0000", 2021-03-24T21:57:41.212 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1669999999999998e-06 2021-03-24T21:57:41.212 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.212 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.212 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:57:41.212 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.174590+0000", 2021-03-24T21:57:41.213 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1109000000000001e-05 2021-03-24T21:57:41.213 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.213 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.213 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:41.213 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.174641+0000", 2021-03-24T21:57:41.214 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0509000000000001e-05 2021-03-24T21:57:41.214 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.214 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.214 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T21:57:41.215 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.174651+0000", 2021-03-24T21:57:41.215 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0054966999999999 2021-03-24T21:57:41.215 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.215 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.215 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:41.216 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.180148+0000", 2021-03-24T21:57:41.216 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00011527 2021-03-24T21:57:41.216 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.216 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.216 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:57:41.217 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.180263+0000", 2021-03-24T21:57:41.217 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000228352 2021-03-24T21:57:41.217 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.217 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.217 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,4", 2021-03-24T21:57:41.218 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.180492+0000", 2021-03-24T21:57:41.218 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.004233115 2021-03-24T21:57:41.218 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.218 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.219 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:57:41.219 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.184725+0000", 2021-03-24T21:57:41.219 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.006255404 2021-03-24T21:57:41.219 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.219 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.220 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:57:41.220 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.190980+0000", 2021-03-24T21:57:41.220 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0016284769999999999 2021-03-24T21:57:41.220 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.220 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.221 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:57:41.221 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.192609+0000", 2021-03-24T21:57:41.221 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5276999999999999e-05 2021-03-24T21:57:41.221 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.221 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.222 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:57:41.222 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.192664+0000", 2021-03-24T21:57:41.222 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.8547000000000001e-05 2021-03-24T21:57:41.222 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.223 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.223 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:57:41.223 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.192712+0000", 2021-03-24T21:57:41.223 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.018522293 2021-03-24T21:57:41.223 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:41.224 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:57:41.224 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:41.224 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.224 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.224 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:2392 7.23 7:c49de0b4:::smithi12556470-13:head [write 2699879~469889] snapc a9=[a9,a8,a7,a0] ondisk+write+known_if_redirected e275)", 2021-03-24T21:57:41.225 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:57:25.175958+0000", 2021-03-24T21:57:41.225 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 15.829868352, 2021-03-24T21:57:41.225 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.018195899, 2021-03-24T21:57:41.225 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:57:41.225 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:57:41.226 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:57:41.226 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:57:41.226 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:57:41.226 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 2392 2021-03-24T21:57:41.227 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.227 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:57:41.227 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.227 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:57:41.227 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.175958+0000", 2021-03-24T21:57:41.228 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:57:41.228 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.228 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.228 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:57:41.228 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.175958+0000", 2021-03-24T21:57:41.229 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999971 2021-03-24T21:57:41.229 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.229 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.229 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:57:41.229 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.175955+0000", 2021-03-24T21:57:41.230 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00032653800000000002 2021-03-24T21:57:41.230 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.230 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.230 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:57:41.230 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176282+0000", 2021-03-24T21:57:41.231 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0336999999999999e-05 2021-03-24T21:57:41.231 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.231 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.231 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:57:41.231 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176292+0000", 2021-03-24T21:57:41.232 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4629999999999998e-06 2021-03-24T21:57:41.232 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.232 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.232 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:57:41.232 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176299+0000", 2021-03-24T21:57:41.232 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.8623999999999998e-05 2021-03-24T21:57:41.233 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.233 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.233 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:41.233 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176377+0000", 2021-03-24T21:57:41.234 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2301e-05 2021-03-24T21:57:41.234 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.234 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.234 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T21:57:41.234 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176390+0000", 2021-03-24T21:57:41.235 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.005175323 2021-03-24T21:57:41.235 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.235 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.235 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:41.236 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.181565+0000", 2021-03-24T21:57:41.236 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000124189 2021-03-24T21:57:41.236 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.236 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.236 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:57:41.237 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.181689+0000", 2021-03-24T21:57:41.237 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000263744 2021-03-24T21:57:41.237 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.237 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.237 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,4", 2021-03-24T21:57:41.237 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.181953+0000", 2021-03-24T21:57:41.238 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0027792490000000001 2021-03-24T21:57:41.238 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.238 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.238 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:57:41.239 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.184732+0000", 2021-03-24T21:57:41.239 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0063294529999999996 2021-03-24T21:57:41.239 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.239 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.239 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:57:41.240 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.191062+0000", 2021-03-24T21:57:41.240 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0029539750000000002 2021-03-24T21:57:41.240 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.240 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.240 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:57:41.241 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.194016+0000", 2021-03-24T21:57:41.241 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.5142999999999996e-05 2021-03-24T21:57:41.241 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.241 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.241 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:57:41.241 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.194081+0000", 2021-03-24T21:57:41.242 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.3331999999999995e-05 2021-03-24T21:57:41.242 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.242 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.242 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:57:41.243 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.194154+0000", 2021-03-24T21:57:41.243 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.018195899 2021-03-24T21:57:41.243 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:41.243 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:57:41.243 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:41.244 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.244 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.244 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:2393 7.23 7:c49de0b4:::smithi12556470-13:head [setxattr _header (57) in=7b,truncate 3630345] snapc a9=[a9,a8,a7,a0] ondisk+write+known_if_redirected e275)", 2021-03-24T21:57:41.244 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:57:25.176325+0000", 2021-03-24T21:57:41.244 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 15.829501611, 2021-03-24T21:57:41.245 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.024884068, 2021-03-24T21:57:41.245 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:57:41.245 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:57:41.245 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:57:41.245 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:57:41.246 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:57:41.246 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 2393 2021-03-24T21:57:41.246 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.246 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:57:41.247 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.247 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:57:41.247 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176325+0000", 2021-03-24T21:57:41.247 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:57:41.247 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.248 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.248 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:57:41.248 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176325+0000", 2021-03-24T21:57:41.248 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T21:57:41.248 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.249 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.249 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:57:41.249 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176324+0000", 2021-03-24T21:57:41.249 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1580000000000004e-06 2021-03-24T21:57:41.249 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.250 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.250 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:57:41.250 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176329+0000", 2021-03-24T21:57:41.250 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6590000000000001e-06 2021-03-24T21:57:41.251 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.251 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.251 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:57:41.251 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176336+0000", 2021-03-24T21:57:41.251 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1610000000000002e-06 2021-03-24T21:57:41.252 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.252 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.252 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:57:41.252 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176341+0000", 2021-03-24T21:57:41.252 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.3918000000000005e-05 2021-03-24T21:57:41.253 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.253 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.253 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:41.253 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176425+0000", 2021-03-24T21:57:41.253 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0132000000000001e-05 2021-03-24T21:57:41.254 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.254 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.254 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T21:57:41.254 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176435+0000", 2021-03-24T21:57:41.254 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0063475879999999 2021-03-24T21:57:41.255 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.255 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.255 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:41.255 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.182782+0000", 2021-03-24T21:57:41.255 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00028664400000000002 2021-03-24T21:57:41.256 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.256 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.256 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:57:41.256 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.183069+0000", 2021-03-24T21:57:41.257 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00021797799999999999 2021-03-24T21:57:41.257 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.257 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.257 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,4", 2021-03-24T21:57:41.257 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.183287+0000", 2021-03-24T21:57:41.258 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0014522280000000001 2021-03-24T21:57:41.258 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.258 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.258 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:57:41.258 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.184739+0000", 2021-03-24T21:57:41.259 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0095337649999999996 2021-03-24T21:57:41.259 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.259 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.259 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:57:41.260 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.194273+0000", 2021-03-24T21:57:41.260 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0067855449999999996 2021-03-24T21:57:41.260 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.260 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.260 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:57:41.261 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.201059+0000", 2021-03-24T21:57:41.261 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.5068999999999996e-05 2021-03-24T21:57:41.261 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.261 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.261 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:57:41.262 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.201134+0000", 2021-03-24T21:57:41.262 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.5298999999999999e-05 2021-03-24T21:57:41.262 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.262 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.263 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:57:41.263 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.201209+0000", 2021-03-24T21:57:41.263 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.024884068 2021-03-24T21:57:41.263 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:41.263 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:57:41.264 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:41.264 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.264 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.264 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:2394 7.23 7:c49de0b4:::smithi12556470-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e275)", 2021-03-24T21:57:41.264 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:57:25.176373+0000", 2021-03-24T21:57:41.265 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 15.829453853, 2021-03-24T21:57:41.265 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.025101692, 2021-03-24T21:57:41.265 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:57:41.265 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T21:57:41.265 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:57:41.266 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:57:41.266 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:57:41.266 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 2394 2021-03-24T21:57:41.266 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.267 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:57:41.267 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.267 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:57:41.267 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176373+0000", 2021-03-24T21:57:41.267 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:57:41.268 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.268 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.268 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:57:41.268 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176373+0000", 2021-03-24T21:57:41.268 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T21:57:41.269 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.269 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.269 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:57:41.269 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176371+0000", 2021-03-24T21:57:41.270 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.08e-06 2021-03-24T21:57:41.270 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.270 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.270 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:57:41.270 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176378+0000", 2021-03-24T21:57:41.271 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1309e-05 2021-03-24T21:57:41.271 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.271 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.271 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:57:41.271 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176389+0000", 2021-03-24T21:57:41.272 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.8714e-05 2021-03-24T21:57:41.272 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.272 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.272 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:57:41.273 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176418+0000", 2021-03-24T21:57:41.273 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6766000000000004e-05 2021-03-24T21:57:41.273 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.273 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.273 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:41.274 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176485+0000", 2021-03-24T21:57:41.274 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.1589999999999999e-06 2021-03-24T21:57:41.274 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.274 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.274 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T21:57:41.275 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176494+0000", 2021-03-24T21:57:41.275 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0071017390000001 2021-03-24T21:57:41.275 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.275 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.276 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:41.276 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.183596+0000", 2021-03-24T21:57:41.276 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6001999999999996e-05 2021-03-24T21:57:41.276 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.276 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.277 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T21:57:41.277 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.183662+0000", 2021-03-24T21:57:41.277 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.017585719 2021-03-24T21:57:41.277 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.278 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.278 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:41.278 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.201247+0000", 2021-03-24T21:57:41.278 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000106111 2021-03-24T21:57:41.278 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.279 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.279 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:57:41.279 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.201354+0000", 2021-03-24T21:57:41.279 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000120763 2021-03-24T21:57:41.280 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.280 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.280 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:57:41.280 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.201474+0000", 2021-03-24T21:57:41.280 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.025101692 2021-03-24T21:57:41.281 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:41.281 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:57:41.281 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:41.281 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.281 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.282 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:2557 7.11 e290/204 7:8dd64f9e:::smithi12556470-49:head v 290'308, mlcod=278'304)", 2021-03-24T21:57:41.282 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:57:40.204635+0000", 2021-03-24T21:57:41.282 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 0.80119140899999997, 2021-03-24T21:57:41.282 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.047038568000000003, 2021-03-24T21:57:41.283 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:57:41.283 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:57:41.283 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:57:41.283 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.283 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:57:41.284 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.204635+0000", 2021-03-24T21:57:41.284 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:57:41.284 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.284 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.285 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:57:41.285 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.204635+0000", 2021-03-24T21:57:41.285 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999971 2021-03-24T21:57:41.285 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.285 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.286 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:57:41.286 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.204632+0000", 2021-03-24T21:57:41.286 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0012479559999999999 2021-03-24T21:57:41.286 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.287 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.287 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:57:41.287 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.205880+0000", 2021-03-24T21:57:41.287 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0608e-05 2021-03-24T21:57:41.287 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.288 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.288 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:57:41.288 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.205901+0000", 2021-03-24T21:57:41.288 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.25e-06 2021-03-24T21:57:41.288 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.289 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.289 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:57:41.289 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.205908+0000", 2021-03-24T21:57:41.289 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.3885000000000003e-05 2021-03-24T21:57:41.289 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.290 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.290 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:41.290 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.205962+0000", 2021-03-24T21:57:41.290 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.7458999999999999e-05 2021-03-24T21:57:41.291 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.291 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.291 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:57:41.291 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.205989+0000", 2021-03-24T21:57:41.291 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.045622023999999997 2021-03-24T21:57:41.292 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.292 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.292 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:57:41.292 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.251611+0000", 2021-03-24T21:57:41.292 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2255999999999995e-05 2021-03-24T21:57:41.293 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.293 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.293 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:57:41.293 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.251674+0000", 2021-03-24T21:57:41.293 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.047038568000000003 2021-03-24T21:57:41.294 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:41.294 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:57:41.294 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:41.294 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.295 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.295 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:2567 7.c 7:32974111:::smithi12556470-2:head [copy-from ver 309] snapc b7=[b7,b4] ondisk+write+known_if_redirected e290)", 2021-03-24T21:57:41.295 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:57:40.209073+0000", 2021-03-24T21:57:41.295 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 0.79675362999999999, 2021-03-24T21:57:41.295 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.050672957999999997, 2021-03-24T21:57:41.296 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:57:41.296 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:57:41.296 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:57:41.296 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:57:41.296 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:57:41.296 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 2567 2021-03-24T21:57:41.297 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.297 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:57:41.297 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.297 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:57:41.297 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.209073+0000", 2021-03-24T21:57:41.298 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:57:41.298 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.298 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.298 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:57:41.298 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.209073+0000", 2021-03-24T21:57:41.299 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999971 2021-03-24T21:57:41.299 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.299 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.299 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:57:41.299 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.209070+0000", 2021-03-24T21:57:41.300 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1926000000000001e-05 2021-03-24T21:57:41.300 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.300 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.300 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:57:41.300 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.209082+0000", 2021-03-24T21:57:41.301 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.1829999999999999e-05 2021-03-24T21:57:41.301 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.301 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.301 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:57:41.301 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.209104+0000", 2021-03-24T21:57:41.302 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.2329999999999995e-06 2021-03-24T21:57:41.302 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.302 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.302 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:57:41.302 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.209113+0000", 2021-03-24T21:57:41.303 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.7663999999999995e-05 2021-03-24T21:57:41.303 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.303 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.303 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:41.303 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.209181+0000", 2021-03-24T21:57:41.304 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.7707000000000005e-05 2021-03-24T21:57:41.304 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.304 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.304 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:57:41.304 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.209278+0000", 2021-03-24T21:57:41.305 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.017481218 2021-03-24T21:57:41.305 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.305 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.305 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,7", 2021-03-24T21:57:41.305 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.226759+0000", 2021-03-24T21:57:41.306 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0120091 2021-03-24T21:57:41.306 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.306 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.306 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:57:41.306 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.238769+0000", 2021-03-24T21:57:41.307 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018981728999999999 2021-03-24T21:57:41.307 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.307 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.307 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:57:41.307 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.257750+0000", 2021-03-24T21:57:41.308 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001886757 2021-03-24T21:57:41.308 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.308 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.308 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:57:41.308 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.259637+0000", 2021-03-24T21:57:41.309 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5423000000000002e-05 2021-03-24T21:57:41.309 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.309 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.309 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:57:41.309 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.259682+0000", 2021-03-24T21:57:41.310 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.3319999999999997e-05 2021-03-24T21:57:41.310 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.310 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.310 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:57:41.310 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.259746+0000", 2021-03-24T21:57:41.311 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.050672957999999997 2021-03-24T21:57:41.311 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:41.311 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:57:41.311 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:41.311 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.312 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.312 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:2568 7.c 7:32974111:::smithi12556470-2:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e290)", 2021-03-24T21:57:41.312 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:57:40.209133+0000", 2021-03-24T21:57:41.312 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 0.79669348200000001, 2021-03-24T21:57:41.312 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.050908821999999999, 2021-03-24T21:57:41.313 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:57:41.313 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T21:57:41.313 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:57:41.313 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:57:41.313 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:57:41.314 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 2568 2021-03-24T21:57:41.314 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.314 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:57:41.314 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.314 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:57:41.315 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.209133+0000", 2021-03-24T21:57:41.315 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:57:41.315 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.315 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.315 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:57:41.316 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.209133+0000", 2021-03-24T21:57:41.316 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T21:57:41.316 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.316 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.316 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:57:41.317 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.209132+0000", 2021-03-24T21:57:41.317 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6269999999999999e-06 2021-03-24T21:57:41.317 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.317 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.317 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:57:41.318 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.209137+0000", 2021-03-24T21:57:41.318 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.8019999999999992e-06 2021-03-24T21:57:41.318 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.318 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.318 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:57:41.319 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.209145+0000", 2021-03-24T21:57:41.319 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.366e-06 2021-03-24T21:57:41.319 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.319 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.319 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:57:41.320 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.209150+0000", 2021-03-24T21:57:41.320 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00030061699999999998 2021-03-24T21:57:41.320 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.320 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.320 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:41.321 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.209451+0000", 2021-03-24T21:57:41.321 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.1934000000000003e-05 2021-03-24T21:57:41.321 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.321 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.321 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-24T21:57:41.322 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.209482+0000", 2021-03-24T21:57:41.322 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.028425082000000001 2021-03-24T21:57:41.322 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.322 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.323 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:41.323 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.237908+0000", 2021-03-24T21:57:41.323 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.9487999999999993e-05 2021-03-24T21:57:41.323 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.323 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.324 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T21:57:41.324 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.238007+0000", 2021-03-24T21:57:41.324 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.021800834000000002 2021-03-24T21:57:41.324 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.324 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.325 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:41.325 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.259808+0000", 2021-03-24T21:57:41.325 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000114912 2021-03-24T21:57:41.325 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.325 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.326 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:57:41.326 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.259923+0000", 2021-03-24T21:57:41.326 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000119027 2021-03-24T21:57:41.326 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.327 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.327 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:57:41.327 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.260042+0000", 2021-03-24T21:57:41.327 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.050908821999999999 2021-03-24T21:57:41.328 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:41.328 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:57:41.328 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:41.328 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.329 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.329 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:2558 7.11 e290/204 7:8dd64f9e:::smithi12556470-49:head v 290'309, mlcod=278'304)", 2021-03-24T21:57:41.329 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:57:40.209545+0000", 2021-03-24T21:57:41.329 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 0.79628111400000001, 2021-03-24T21:57:41.329 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.047901726999999998, 2021-03-24T21:57:41.330 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:57:41.330 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:57:41.330 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:57:41.330 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.330 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:57:41.331 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.209545+0000", 2021-03-24T21:57:41.331 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:57:41.331 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.331 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.331 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:57:41.332 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.209545+0000", 2021-03-24T21:57:41.332 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T21:57:41.332 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.332 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.332 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:57:41.333 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.209544+0000", 2021-03-24T21:57:41.333 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00028318899999999999 2021-03-24T21:57:41.333 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.333 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.334 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:57:41.334 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.209827+0000", 2021-03-24T21:57:41.334 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.6283999999999999e-05 2021-03-24T21:57:41.334 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.334 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.335 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:57:41.335 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.209843+0000", 2021-03-24T21:57:41.335 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9649999999999996e-06 2021-03-24T21:57:41.335 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.336 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.336 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:57:41.336 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.209849+0000", 2021-03-24T21:57:41.336 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.4833000000000003e-05 2021-03-24T21:57:41.336 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.337 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.337 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:41.337 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.209894+0000", 2021-03-24T21:57:41.337 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.1263e-05 2021-03-24T21:57:41.337 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.338 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.338 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:57:41.338 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.209915+0000", 2021-03-24T21:57:41.338 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.047469086000000001 2021-03-24T21:57:41.338 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.339 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.339 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:57:41.339 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.257384+0000", 2021-03-24T21:57:41.339 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2898000000000007e-05 2021-03-24T21:57:41.339 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.340 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.340 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:57:41.340 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.257447+0000", 2021-03-24T21:57:41.340 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.047901726999999998 2021-03-24T21:57:41.340 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:41.341 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:57:41.341 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:41.341 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.341 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.341 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:2559 7.11 e290/204 7:8dd64f9e:::smithi12556470-49:head v 290'310, mlcod=278'304)", 2021-03-24T21:57:41.342 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:57:40.210626+0000", 2021-03-24T21:57:41.342 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 0.79520048399999999, 2021-03-24T21:57:41.342 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.046846720000000001, 2021-03-24T21:57:41.342 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:57:41.343 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:57:41.343 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:57:41.343 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.343 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:57:41.343 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.210626+0000", 2021-03-24T21:57:41.344 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:57:41.344 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.344 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.344 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:57:41.344 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.210626+0000", 2021-03-24T21:57:41.345 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T21:57:41.345 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.345 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.345 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:57:41.345 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.210625+0000", 2021-03-24T21:57:41.346 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.5619999999999993e-06 2021-03-24T21:57:41.346 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.346 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.346 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:57:41.347 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.210634+0000", 2021-03-24T21:57:41.347 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3704e-05 2021-03-24T21:57:41.347 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.347 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.347 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:57:41.348 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.210648+0000", 2021-03-24T21:57:41.348 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4010000000000002e-06 2021-03-24T21:57:41.348 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.348 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.349 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:57:41.349 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.210654+0000", 2021-03-24T21:57:41.349 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00058815699999999996 2021-03-24T21:57:41.349 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.349 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.350 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:57:41.350 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.211242+0000", 2021-03-24T21:57:41.350 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.4712000000000001e-05 2021-03-24T21:57:41.350 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.351 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.351 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:57:41.351 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.211257+0000", 2021-03-24T21:57:41.351 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.046194720000000002 2021-03-24T21:57:41.351 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.352 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.352 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:57:41.352 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.257452+0000", 2021-03-24T21:57:41.352 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0891999999999999e-05 2021-03-24T21:57:41.352 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:57:41.353 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:57:41.353 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:57:41.353 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.257473+0000", 2021-03-24T21:57:41.353 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.046846720000000001 2021-03-24T21:57:41.354 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:41.354 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:57:41.354 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:41.354 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:57:41.354 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:57:41.355 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-24T21:57:41.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:41.356 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:41.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:57:41.357 INFO:tasks.rados.rados.0.smithi125.stdout:973: done (2 left) 2021-03-24T21:57:41.358 INFO:tasks.rados.rados.0.smithi125.stdout:974: read oid 1 snap -1 2021-03-24T21:57:41.358 INFO:tasks.rados.rados.0.smithi125.stdout:974: expect (ObjNum 338 snap 86 seq_num 338) 2021-03-24T21:57:41.358 INFO:tasks.rados.rados.0.smithi125.stdout:975: delete oid 5 current snap is 92 2021-03-24T21:57:41.358 INFO:tasks.rados.rados.0.smithi125.stdout:968: finishing write tid 2 to smithi12556470-49 2021-03-24T21:57:41.359 INFO:tasks.rados.rados.0.smithi125.stdout:968: finishing write tid 3 to smithi12556470-49 2021-03-24T21:57:41.359 INFO:tasks.rados.rados.0.smithi125.stdout:968: finishing write tid 4 to smithi12556470-49 2021-03-24T21:57:41.359 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 49 v 310 (ObjNum 362 snap 92 seq_num 362) dirty exists 2021-03-24T21:57:41.359 INFO:tasks.rados.rados.0.smithi125.stdout:968: left oid 49 (ObjNum 362 snap 92 seq_num 362) 2021-03-24T21:57:41.360 INFO:tasks.rados.rados.0.smithi125.stdout:971: finishing copy_from to smithi12556470-2 2021-03-24T21:57:41.360 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 2 v 309 (ObjNum 353 snap 89 seq_num 353) dirty exists 2021-03-24T21:57:41.360 INFO:tasks.rados.rados.0.smithi125.stdout:971: finishing copy_from racing read to smithi12556470-2 2021-03-24T21:57:41.361 INFO:teuthology.orchestra.run.smithi125.stderr:nodeep-scrub is set 2021-03-24T21:57:41.361 INFO:tasks.rados.rados.0.smithi125.stdout:974: expect (ObjNum 338 snap 86 seq_num 338) 2021-03-24T21:57:41.365 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:41 smithi125 conmon[19936]: audit 2021-03-24T21:57:40. 2021-03-24T21:57:41.366 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:41 smithi125 conmon[19936]: 866582+0000 mon.c (mon.2) 45 : audit [INF] from='client.? v1:172.21.15.125:0/2818620316' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:57:41.366 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:41 smithi125 conmon[19936]: audit 2021-03-24T21:57 2021-03-24T21:57:41.366 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:41 smithi125 conmon[19936]: :40.867006+0000 mon.a (mon.0) 951 : audit [INF] from='client.35000 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:57:41.367 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:41 smithi125 conmon[24927]: audit 2021-03-24T21:57:40. 2021-03-24T21:57:41.367 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:41 smithi125 conmon[24927]: 866582+0000 mon.c (mon.2) 45 : audit [INF] from='client.? v1:172.21.15.125:0/2818620316' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:57:41.367 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:41 smithi125 conmon[24927]: audit 2021-03-24T21:57:40 2021-03-24T21:57:41.367 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:41 smithi125 conmon[24927]: .867006+0000 mon.a (mon.0) 951 : audit [INF] from='client.35000 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:57:41.458 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:41.507 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:41.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:41.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:57:41.534 INFO:tasks.rados.rados.0.smithi125.stdout:968: done (3 left) 2021-03-24T21:57:41.535 INFO:tasks.rados.rados.0.smithi125.stdout:971: done (2 left) 2021-03-24T21:57:41.535 INFO:tasks.rados.rados.0.smithi125.stdout:974: done (1 left) 2021-03-24T21:57:41.535 INFO:tasks.rados.rados.0.smithi125.stdout:975: done (0 left) 2021-03-24T21:57:41.535 INFO:tasks.rados.rados.0.smithi125.stdout:976: snap_remove snap 91 2021-03-24T21:57:41.613 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:41.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:41 smithi142 conmon[20125]: audit 2021-03-24T21:57:40 2021-03-24T21:57:41.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:41 smithi142 conmon[20125]: .866582+0000 mon.c (mon.2) 45 : audit [INF] from='client.? v1:172.21.15.125:0/2818620316' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:57:41.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:41 smithi142 conmon[20125]: audit 2021-03-24T21: 2021-03-24T21:57:41.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:41 smithi142 conmon[20125]: 57:40.867006+0000 mon.a (mon.0) 951 : audit [INF] from='client.35000 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:57:41.667 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:41.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:41.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:57:41.773 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:41.828 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:41.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:41.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:57:41.934 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:41.992 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:41.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:41.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:57:42.097 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:42.155 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:42.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:42.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:57:42.260 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:42.274 INFO:tasks.rados.rados.0.smithi125.stdout:976: done (0 left) 2021-03-24T21:57:42.275 INFO:tasks.rados.rados.0.smithi125.stdout:977: delete oid 12 current snap is 92 2021-03-24T21:57:42.282 INFO:tasks.rados.rados.0.smithi125.stdout:977: done (0 left) 2021-03-24T21:57:42.283 INFO:tasks.rados.rados.0.smithi125.stdout:978: snap_create 2021-03-24T21:57:42.309 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:42.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:42.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:57:42.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:42 smithi142 conmon[20125]: cluster 2021-03-24T21:57 2021-03-24T21:57:42.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:42 smithi142 conmon[20125]: :40.778449+0000 mgr.y (mgr.14144) 412 : cluster [DBG] pgmap v598: 168 pgs: 168 active+clean; 134 MiB data, 504 MiB used, 536 GiB / 536 GiB avail; 17 MiB/s rd, 3.1 MiB/s wr, 26 op/s 2021-03-24T21:57:42.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:42 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:57:42.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:42 smithi142 conmon[20125]: -24T21:57:41.255747+0000 mon.a (mon.0) 952 : 2021-03-24T21:57:42.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:42 smithi142 conmon[20125]: cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:57:42.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:42 smithi142 conmon[20125]: audit 2021-03-24T21:57: 2021-03-24T21:57:42.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:42 smithi142 conmon[20125]: 41.270930+0000 mon.a (mon.0) 953 2021-03-24T21:57:42.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:42 smithi142 conmon[20125]: : audit [INF] from='client.35000 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T21:57:42.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:42 smithi142 conmon[20125]: cluster 2021-03-24T21 2021-03-24T21:57:42.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:42 smithi142 conmon[20125]: :57:41.271467+0000 mon.a (mon. 2021-03-24T21:57:42.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:42 smithi142 conmon[20125]: 0) 954 : cluster [DBG] osdmap e291: 8 total, 8 up, 6 in 2021-03-24T21:57:42.413 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:42.471 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:42.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:42.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:57:42.527 INFO:journalctl@ceph.osd.0.smithi125.stdout:Mar 24 21:57:42 smithi125 conmon[28026]: debug 2021-03-24T21:57:42.275+0000 7f61b747d700 -1 osd.0 pg_epoch: 292 pg[7.1d( v 291'280 (0'0,291'280] local-lis/les=281/282 n=4 ec=275/99 lis/c=281/281 les/c/f=282/282/0 sis=281) [0,5,7] r=0 lpr=281 crt=291'280 lcod 291'278 mlcod 291'278 active+clean+snaptrim trimq=[b7~1] ps=[b0~2,b3~2]] removing snap head 2021-03-24T21:57:42.528 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:42 smithi125 conmon[19936]: cluster 2021-03-24T21:57:40.778449+0000 mgr.y (mgr.14144) 412 : cluster 2021-03-24T21:57:42.528 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:42 smithi125 conmon[19936]: [DBG] pgmap v598: 168 pgs: 168 active+clean; 134 MiB data, 504 MiB used, 536 GiB / 536 GiB avail; 17 MiB/s rd, 3.1 MiB/s wr, 26 op/s 2021-03-24T21:57:42.528 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:42 smithi125 conmon[19936]: cluster 2021-03-24T21:57:41.255747+0000 mon.a (mon.0) 952 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:57:42.528 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:42 smithi125 conmon[19936]: audit 2021-03-24T21:57:41.270930+0000 mon.a (mon.0) 953 : audit [INF] 2021-03-24T21:57:42.528 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:42 smithi125 conmon[19936]: from='client.35000 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T21:57:42.529 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:42 smithi125 conmon[19936]: cluster 2021-03-24T21:57: 2021-03-24T21:57:42.529 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:42 smithi125 conmon[19936]: 41.271467+0000 mon.a (mon.0) 954 : cluster 2021-03-24T21:57:42.529 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:42 smithi125 conmon[19936]: [DBG] osdmap e291: 8 total, 8 up, 6 in 2021-03-24T21:57:42.529 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:42 smithi125 conmon[24927]: cluster 2021-03-24T21:57:40.778449+0000 mgr.y (mgr 2021-03-24T21:57:42.530 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:42 smithi125 conmon[24927]: .14144) 412 : cluster [DBG] pgmap v598: 168 pgs: 168 active+clean; 134 MiB data, 504 MiB used, 536 GiB / 536 GiB avail; 17 MiB/s rd, 3.1 MiB/s wr, 26 op/s 2021-03-24T21:57:42.530 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:42 smithi125 conmon[24927]: cluster 2021-03-24T21:57:41.255747 2021-03-24T21:57:42.530 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:42 smithi125 conmon[24927]: +0000 mon.a (mon.0) 952 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:57:42.530 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:42 smithi125 conmon[24927]: audit 2021-03-24T21:57:41 2021-03-24T21:57:42.531 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:42 smithi125 conmon[24927]: .270930+0000 mon.a (mon.0) 953 : audit [INF] 2021-03-24T21:57:42.531 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:42 smithi125 conmon[24927]: from='client.35000 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T21:57:42.531 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:42 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:57:42.531 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:42 smithi125 conmon[24927]: -03-24T21:57:41.271467+0000 mon.a ( 2021-03-24T21:57:42.532 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:42 smithi125 conmon[24927]: mon.0) 954 : cluster [DBG] osdmap e291: 8 total, 8 up, 6 in 2021-03-24T21:57:42.577 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:42.642 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:42.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:42.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:57:42.748 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:42.805 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:42.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:42.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:57:42.912 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:42.969 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:42.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:42.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:57:43.079 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:43.142 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:43.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:43.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:57:43.205 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.4 dump_ops_in_flight 2021-03-24T21:57:43.247 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:43.291 INFO:tasks.rados.rados.0.smithi125.stdout:978: done (0 left) 2021-03-24T21:57:43.292 INFO:tasks.rados.rados.0.smithi125.stdout:979: copy_from oid 17 from oid 11 current snap is 93 2021-03-24T21:57:43.292 INFO:tasks.rados.rados.0.smithi125.stdout:980: delete oid 3 current snap is 93 2021-03-24T21:57:43.297 INFO:tasks.rados.rados.0.smithi125.stdout:980: done (1 left) 2021-03-24T21:57:43.298 INFO:tasks.rados.rados.0.smithi125.stdout:981: write (excl) oid 35 current snap is 93 2021-03-24T21:57:43.298 INFO:tasks.rados.rados.0.smithi125.stdout:981: seq_num 367 ranges {568169=401327,1376068=669790} 2021-03-24T21:57:43.306 INFO:tasks.rados.rados.0.smithi125.stdout:981: writing smithi12556470-35 from 568169 to 969496 tid 1 2021-03-24T21:57:43.307 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:43.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:43.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:57:43.313 INFO:tasks.rados.rados.0.smithi125.stdout:981: writing smithi12556470-35 from 1376068 to 2045858 tid 2 2021-03-24T21:57:43.313 INFO:tasks.rados.rados.0.smithi125.stdout:982: copy_from oid 28 from oid 9 current snap is 93 2021-03-24T21:57:43.314 INFO:tasks.rados.rados.0.smithi125.stdout:983: snap_create 2021-03-24T21:57:43.314 INFO:tasks.rados.rados.0.smithi125.stdout:981: finishing write tid 1 to smithi12556470-35 2021-03-24T21:57:43.314 INFO:tasks.rados.rados.0.smithi125.stdout:979: finishing copy_from to smithi12556470-17 2021-03-24T21:57:43.314 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 17 v 249 (ObjNum 327 snap 83 seq_num 327) dirty exists 2021-03-24T21:57:43.315 INFO:tasks.rados.rados.0.smithi125.stdout:979: finishing copy_from racing read to smithi12556470-17 2021-03-24T21:57:43.316 INFO:tasks.rados.rados.0.smithi125.stdout:981: finishing write tid 2 to smithi12556470-35 2021-03-24T21:57:43.316 INFO:tasks.rados.rados.0.smithi125.stdout:981: finishing write tid 3 to smithi12556470-35 2021-03-24T21:57:43.317 INFO:tasks.rados.rados.0.smithi125.stdout:981: finishing write tid 4 to smithi12556470-35 2021-03-24T21:57:43.317 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 35 v 197 (ObjNum 367 snap 93 seq_num 367) dirty exists 2021-03-24T21:57:43.317 INFO:tasks.rados.rados.0.smithi125.stdout:981: left oid 35 (ObjNum 367 snap 93 seq_num 367) 2021-03-24T21:57:43.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:43 smithi142 conmon[20125]: cluster 2021-03-24T21:57:42. 2021-03-24T21:57:43.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:43 smithi142 conmon[20125]: 273675+0000 mon.a (mon.0) 955 : cluster [DBG] osdmap e292: 8 total, 8 up, 6 in 2021-03-24T21:57:43.383 INFO:tasks.rados.rados.0.smithi125.stdout:982: finishing copy_from to smithi12556470-28 2021-03-24T21:57:43.384 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 28 v 286 (ObjNum 349 snap 89 seq_num 349) dirty exists 2021-03-24T21:57:43.384 INFO:tasks.rados.rados.0.smithi125.stdout:982: finishing copy_from racing read to smithi12556470-28 2021-03-24T21:57:43.413 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:43.425 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:43 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:57:43.426 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:43 smithi125 conmon[19936]: 21:57:42.273675+0000 mon.a (mon.0) 955 : cluster [DBG] osdmap e292: 8 total, 8 up, 6 in 2021-03-24T21:57:43.426 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:43 smithi125 conmon[24927]: cluster 2021-03-24T21:57:42.273675+0000 mon.a 2021-03-24T21:57:43.426 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:43 smithi125 conmon[24927]: (mon.0) 955 : cluster [DBG] osdmap e292: 8 total, 8 up, 6 in 2021-03-24T21:57:43.466 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:43.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:43.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:57:43.521 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T21:57:43.572 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:43.625 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:43.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:43.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:57:43.730 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:43.789 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:43.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:43.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:57:43.894 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:43.947 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:43.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:43.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:44.053 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:44.106 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:44.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:44.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:57:44.212 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:44.268 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:44.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:44.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:57:44.300 INFO:tasks.rados.rados.0.smithi125.stdout:979: done (3 left) 2021-03-24T21:57:44.300 INFO:tasks.rados.rados.0.smithi125.stdout:981: done (2 left) 2021-03-24T21:57:44.300 INFO:tasks.rados.rados.0.smithi125.stdout:982: done (1 left) 2021-03-24T21:57:44.300 INFO:tasks.rados.rados.0.smithi125.stdout:983: done (0 left) 2021-03-24T21:57:44.301 INFO:tasks.rados.rados.0.smithi125.stdout:984: copy_from oid 37 from oid 31 current snap is 94 2021-03-24T21:57:44.301 INFO:tasks.rados.rados.0.smithi125.stdout:985: read oid 19 snap -1 2021-03-24T21:57:44.301 INFO:tasks.rados.rados.0.smithi125.stdout:985: expect (ObjNum 280 snap 70 seq_num 280) 2021-03-24T21:57:44.301 INFO:tasks.rados.rados.0.smithi125.stdout:986: snap_create 2021-03-24T21:57:44.310 INFO:tasks.rados.rados.0.smithi125.stdout:985: expect (ObjNum 280 snap 70 seq_num 280) 2021-03-24T21:57:44.373 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:44.404 INFO:tasks.rados.rados.0.smithi125.stdout:984: finishing copy_from to smithi12556470-37 2021-03-24T21:57:44.405 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 37 v 128 (ObjNum 274 snap 68 seq_num 274) dirty exists 2021-03-24T21:57:44.405 INFO:tasks.rados.rados.0.smithi125.stdout:984: finishing copy_from racing read to smithi12556470-37 2021-03-24T21:57:44.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:44 smithi125 conmon[19936]: cluster 2021-03-24T21:57:42.779171+0000 mgr.y (mgr.14144) 413 2021-03-24T21:57:44.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:44 smithi125 conmon[19936]: : cluster [DBG] pgmap v601: 168 pgs: 168 active+clean; 134 MiB data, 504 MiB used, 536 GiB / 536 GiB avail; 17 MiB/s rd, 3.1 MiB/s wr, 26 op/s 2021-03-24T21:57:44.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:44 smithi125 conmon[19936]: cluster 2021-03-24T21:57:43.286814+0000 mon.a (mon.0) 956 : cluster [DBG] osdmap e293: 8 total, 8 up, 6 in 2021-03-24T21:57:44.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:44 smithi125 conmon[24927]: cluster 2021-03-24T21:57:42.779171+0000 mgr.y (mgr.14144) 413 : cluster 2021-03-24T21:57:44.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:44 smithi125 conmon[24927]: [DBG] pgmap v601: 168 pgs: 168 active+clean; 134 MiB data, 504 MiB used, 536 GiB / 536 GiB avail; 17 MiB/s rd, 3.1 MiB/s wr, 26 op/s 2021-03-24T21:57:44.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:44 smithi125 conmon[24927]: cluster 2021-03-24T21:57:43.286814+0000 mon.a (mon.0) 956 : cluster [DBG] osdmap e293: 8 total, 8 up, 6 in 2021-03-24T21:57:44.429 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:44.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:44.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:57:44.536 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:44.549 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:44 smithi142 conmon[20125]: cluster 2021-03-24T21:57:42.779171+0000 2021-03-24T21:57:44.550 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:44 smithi142 conmon[20125]: mgr.y (mgr.14144) 413 : cluster [DBG] pgmap v601: 168 pgs: 168 active+clean; 134 MiB data, 504 MiB used, 536 GiB / 536 GiB avail; 17 MiB/s rd, 3.1 MiB/s wr, 26 op/s 2021-03-24T21:57:44.550 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:44 smithi142 conmon[20125]: cluster 2021-03-24T21:57:44.550 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:44 smithi142 conmon[20125]: 2021-03-24T21 2021-03-24T21:57:44.550 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:44 smithi142 conmon[20125]: :57:43.286814 2021-03-24T21:57:44.551 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:44 smithi142 conmon[20125]: +0000 mon.a ( 2021-03-24T21:57:44.551 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:44 smithi142 conmon[20125]: mon.0) 956 2021-03-24T21:57:44.551 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:44 smithi142 conmon[20125]: : cluster [DBG] osdmap e293: 8 total, 8 up, 6 in 2021-03-24T21:57:44.592 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:44.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:44.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:57:44.698 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:44.751 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:44.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:44.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:57:44.858 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:44.917 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:44.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:44.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:57:45.022 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:45.076 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:45.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:45.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:57:45.160 INFO:tasks.rados.rados.0.smithi125.stdout:984: done (2 left) 2021-03-24T21:57:45.161 INFO:tasks.rados.rados.0.smithi125.stdout:985: done (1 left) 2021-03-24T21:57:45.161 INFO:tasks.rados.rados.0.smithi125.stdout:986: done (0 left) 2021-03-24T21:57:45.161 INFO:tasks.rados.rados.0.smithi125.stdout:987: rollback oid 27 current snap is 95 2021-03-24T21:57:45.161 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 27 to 93 2021-03-24T21:57:45.161 INFO:tasks.rados.rados.0.smithi125.stdout:988: snap_remove snap 94 2021-03-24T21:57:45.182 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:45.245 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:45.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:45.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:45.330 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T21:57:45.349 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:45.351 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T21:57:45.370 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T21:57:45.394 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T21:57:45.403 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:45.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:45.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:45.418 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T21:57:45.432 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:45 smithi125 conmon[19936]: cluster 2021-03-24T21:57:44.297073+0000 mon.a (mon.0) 957 : cluster [DBG] osdmap e294: 8 total, 8 up, 6 in 2021-03-24T21:57:45.432 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:45 smithi125 conmon[19936]: cluster 2021-03-24T21:57:45.156408+0000 mon.a (mon.0) 958 : cluster [DBG] osdmap e295: 8 total, 8 up, 6 in 2021-03-24T21:57:45.433 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:45 smithi125 conmon[19936]: audit 2021-03-24T21:57:45.301655+0000 mon.c (mon.2) 46 : audit [DBG] from='client.? v1:172.21.15.125:0/1135318372' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:57:45.433 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:45 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:57:45.433 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:45 smithi125 conmon[24927]: T21:57:44.297073+0000 mon.a (mon.0) 957 : cluster [DBG] osdmap e294: 8 total, 8 up, 6 in 2021-03-24T21:57:45.433 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:45 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:57:45.434 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:45 smithi125 conmon[24927]: :57:45.156408+0000 mon.a (mon.0) 958 : 2021-03-24T21:57:45.434 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:45 smithi125 conmon[24927]: cluster [DBG] osdmap e295: 8 total, 8 up, 6 in 2021-03-24T21:57:45.434 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:45 smithi125 conmon[24927]: audit 2021-03-24 2021-03-24T21:57:45.434 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:45 smithi125 conmon[24927]: T21:57:45.301655+0000 mon.c ( 2021-03-24T21:57:45.435 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:45 smithi125 conmon[24927]: mon.2) 46 : audit [DBG] from='client.? v1:172.21.15.125:0/1135318372' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:57:45.447 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T21:57:45.471 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T21:57:45.480 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:45 smithi142 conmon[20125]: cluster 2021-03-24T21:57:44.297073+0000 mon.a 2021-03-24T21:57:45.480 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:45 smithi142 conmon[20125]: (mon.0) 957 : cluster [DBG] osdmap e294: 8 total, 8 up, 6 in 2021-03-24T21:57:45.480 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:45 smithi142 conmon[20125]: cluster 2021-03-24T21:57: 2021-03-24T21:57:45.481 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:45 smithi142 conmon[20125]: 45.156408+0000 mon.a (mon.0) 958 : cluster [DBG] osdmap e295: 8 total, 8 up, 6 in 2021-03-24T21:57:45.481 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:45 smithi142 conmon[20125]: audit 2021 2021-03-24T21:57:45.481 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:45 smithi142 conmon[20125]: -03-24T21:57:45 2021-03-24T21:57:45.482 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:45 smithi142 conmon[20125]: .301655+0000 mon.c (mon.2 2021-03-24T21:57:45.482 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:45 smithi142 conmon[20125]: ) 46 : audit [DBG] from='client.? v1:172.21.15.125:0/1135318372' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:57:45.494 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T21:57:45.496 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-24T21:57:45.496 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-24T21:57:45.497 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-24T21:57:45.497 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-24T21:57:45.497 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-24T21:57:45.497 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-24T21:57:45.498 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-24T21:57:45.498 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-24T21:57:45.508 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:45.563 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:45.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:45.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:45.671 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:45.732 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:45.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:45.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:57:45.781 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2021-03-24T21:57:45.781 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T21:57:45.836 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-24T21:57:45.839 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2021-03-24T21:57:45.840 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:45.893 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:45.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:45.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:57:46.000 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:46.062 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:46.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:46.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:57:46.163 INFO:tasks.rados.rados.0.smithi125.stdout:988: done (1 left) 2021-03-24T21:57:46.164 INFO:tasks.rados.rados.0.smithi125.stdout:989: rollback oid 45 current snap is 95 2021-03-24T21:57:46.164 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 45 to 93 2021-03-24T21:57:46.164 INFO:tasks.rados.rados.0.smithi125.stdout:987: finishing rollback tid 0 to smithi12556470-27 2021-03-24T21:57:46.164 INFO:tasks.rados.rados.0.smithi125.stdout:987: finishing rollback tid 1 to smithi12556470-27 2021-03-24T21:57:46.164 INFO:tasks.rados.rados.0.smithi125.stdout:987: finishing rollback tid 2 to smithi12556470-27 2021-03-24T21:57:46.165 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 27 v 259 (ObjNum 193 snap 42 seq_num 193) dirty exists 2021-03-24T21:57:46.165 INFO:tasks.rados.rados.0.smithi125.stdout:987: done (1 left) 2021-03-24T21:57:46.165 INFO:tasks.rados.rados.0.smithi125.stdout:990: write (excl) oid 18 current snap is 95 2021-03-24T21:57:46.165 INFO:tasks.rados.rados.0.smithi125.stdout:990: seq_num 368 ranges {584976=476576,1834550=350426} 2021-03-24T21:57:46.167 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:46.174 INFO:tasks.rados.rados.0.smithi125.stdout:990: writing smithi12556470-18 from 584976 to 1061552 tid 1 2021-03-24T21:57:46.181 INFO:tasks.rados.rados.0.smithi125.stdout:990: writing smithi12556470-18 from 1834550 to 2184976 tid 2 2021-03-24T21:57:46.181 INFO:tasks.rados.rados.0.smithi125.stdout:991: write (excl) oid 20 current snap is 95 2021-03-24T21:57:46.182 INFO:tasks.rados.rados.0.smithi125.stdout:991: seq_num 369 ranges {601783=551825,1776679=425104} 2021-03-24T21:57:46.187 INFO:tasks.rados.rados.0.smithi125.stdout:991: writing smithi12556470-20 from 601783 to 1153608 tid 1 2021-03-24T21:57:46.195 INFO:tasks.rados.rados.0.smithi125.stdout:991: writing smithi12556470-20 from 1776679 to 2201783 tid 2 2021-03-24T21:57:46.195 INFO:tasks.rados.rados.0.smithi125.stdout:992: write oid 26 current snap is 95 2021-03-24T21:57:46.195 INFO:tasks.rados.rados.0.smithi125.stdout:989: finishing rollback tid 1 to smithi12556470-45 2021-03-24T21:57:46.196 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 45 v 0 (ObjNum 1116709328 snap 21916 seq_num 1116712368) dirty dne 2021-03-24T21:57:46.196 INFO:tasks.rados.rados.0.smithi125.stdout:990: finishing write tid 1 to smithi12556470-18 2021-03-24T21:57:46.197 INFO:tasks.rados.rados.0.smithi125.stdout:992: seq_num 370 ranges {618590=627074,1835161=383429} 2021-03-24T21:57:46.201 INFO:tasks.rados.rados.0.smithi125.stdout:992: writing smithi12556470-26 from 618590 to 1245664 tid 1 2021-03-24T21:57:46.207 INFO:tasks.rados.rados.0.smithi125.stdout:992: writing smithi12556470-26 from 1835161 to 2218590 tid 2 2021-03-24T21:57:46.208 INFO:tasks.rados.rados.0.smithi125.stdout:989: done (3 left) 2021-03-24T21:57:46.208 INFO:tasks.rados.rados.0.smithi125.stdout:993: snap_remove snap 92 2021-03-24T21:57:46.209 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd unset noscrub 2021-03-24T21:57:46.217 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:46.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:46.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:57:46.323 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:46.370 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:46.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:46.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:57:46.478 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:46.529 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:46.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:46.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:57:46.589 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:46 smithi125 conmon[19936]: cluster 2021-03-24T21:57: 2021-03-24T21:57:46.589 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:46 smithi125 conmon[19936]: 44.780147+0000 mgr.y (mgr.14144) 414 : cluster [DBG] pgmap v604: 168 pgs: 168 active+clean; 124 MiB data, 459 MiB used, 536 GiB / 536 GiB avail; 8.3 MiB/s rd, 3.4 MiB/s wr, 11 op/s 2021-03-24T21:57:46.590 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:46 smithi125 conmon[19936]: cluster 2021-03-24T21:57: 2021-03-24T21:57:46.590 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:46 smithi125 conmon[19936]: 46.160014+0000 mon.a (mon.0) 959 : cluster [DBG] osdmap e296: 8 total, 8 up, 6 in 2021-03-24T21:57:46.590 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:46 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:57:46.591 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:46 smithi125 conmon[24927]: 57:44.780147+0000 mgr.y (mgr.14144) 414 : cluster [DBG] pgmap v604: 168 pgs: 168 active+clean; 124 MiB data, 459 MiB used, 536 GiB / 536 GiB avail; 8.3 MiB/s rd, 3.4 MiB/s wr, 11 op/s 2021-03-24T21:57:46.591 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:46 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:57:46.591 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:46 smithi125 conmon[24927]: T21:57:46.160014+0000 mon.a (mon. 2021-03-24T21:57:46.591 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:46 smithi125 conmon[24927]: 0) 959 : cluster [DBG] osdmap e296: 8 total, 8 up, 6 in 2021-03-24T21:57:46.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:46 smithi142 conmon[20125]: cluster 2021-03-24T21:57 2021-03-24T21:57:46.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:46 smithi142 conmon[20125]: :44.780147+0000 mgr.y (mgr.14144) 414 : cluster [DBG] pgmap v604: 168 pgs: 168 active+clean; 124 MiB data, 459 MiB used, 536 GiB / 536 GiB avail; 8.3 MiB/s rd, 3.4 MiB/s wr, 11 op/s 2021-03-24T21:57:46.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:46 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:57:46.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:46 smithi142 conmon[20125]: 57:46.160014+0000 mon.a (mon.0) 959 : cluster [DBG] osdmap e296: 8 total, 8 up, 6 in 2021-03-24T21:57:46.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:46 smithi142 conmon[20125]: 2021-03-24T21:57:46.636 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:46.695 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:46.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:46.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:57:46.801 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:46.843 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.4 dump_blocked_ops 2021-03-24T21:57:46.866 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:46.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:46.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:57:46.972 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:47.025 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:47.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:47.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:47.130 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T21:57:47.132 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:47.170 INFO:tasks.rados.rados.0.smithi125.stdout:993: done (3 left) 2021-03-24T21:57:47.171 INFO:tasks.rados.rados.0.smithi125.stdout:994: delete oid 37 current snap is 95 2021-03-24T21:57:47.171 INFO:tasks.rados.rados.0.smithi125.stdout:990: finishing write tid 2 to smithi12556470-18 2021-03-24T21:57:47.171 INFO:tasks.rados.rados.0.smithi125.stdout:990: finishing write tid 3 to smithi12556470-18 2021-03-24T21:57:47.171 INFO:tasks.rados.rados.0.smithi125.stdout:990: finishing write tid 4 to smithi12556470-18 2021-03-24T21:57:47.172 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 18 v 172 (ObjNum 368 snap 95 seq_num 368) dirty exists 2021-03-24T21:57:47.172 INFO:tasks.rados.rados.0.smithi125.stdout:990: left oid 18 (ObjNum 368 snap 95 seq_num 368) 2021-03-24T21:57:47.172 INFO:tasks.rados.rados.0.smithi125.stdout:991: finishing write tid 1 to smithi12556470-20 2021-03-24T21:57:47.172 INFO:tasks.rados.rados.0.smithi125.stdout:991: finishing write tid 2 to smithi12556470-20 2021-03-24T21:57:47.173 INFO:tasks.rados.rados.0.smithi125.stdout:991: finishing write tid 3 to smithi12556470-20 2021-03-24T21:57:47.173 INFO:tasks.rados.rados.0.smithi125.stdout:991: finishing write tid 4 to smithi12556470-20 2021-03-24T21:57:47.173 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 20 v 204 (ObjNum 369 snap 95 seq_num 369) dirty exists 2021-03-24T21:57:47.173 INFO:tasks.rados.rados.0.smithi125.stdout:991: left oid 20 (ObjNum 369 snap 95 seq_num 369) 2021-03-24T21:57:47.173 INFO:tasks.rados.rados.0.smithi125.stdout:992: finishing write tid 1 to smithi12556470-26 2021-03-24T21:57:47.174 INFO:tasks.rados.rados.0.smithi125.stdout:992: finishing write tid 2 to smithi12556470-26 2021-03-24T21:57:47.174 INFO:tasks.rados.rados.0.smithi125.stdout:992: finishing write tid 3 to smithi12556470-26 2021-03-24T21:57:47.174 INFO:tasks.rados.rados.0.smithi125.stdout:992: finishing write tid 4 to smithi12556470-26 2021-03-24T21:57:47.174 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 26 v 159 (ObjNum 370 snap 95 seq_num 370) dirty exists 2021-03-24T21:57:47.175 INFO:tasks.rados.rados.0.smithi125.stdout:992: left oid 26 (ObjNum 370 snap 95 seq_num 370) 2021-03-24T21:57:47.175 INFO:tasks.rados.rados.0.smithi125.stdout:990: done (3 left) 2021-03-24T21:57:47.175 INFO:tasks.rados.rados.0.smithi125.stdout:991: done (2 left) 2021-03-24T21:57:47.176 INFO:tasks.rados.rados.0.smithi125.stdout:992: done (1 left) 2021-03-24T21:57:47.176 INFO:tasks.rados.rados.0.smithi125.stdout:994: done (0 left) 2021-03-24T21:57:47.176 INFO:tasks.rados.rados.0.smithi125.stdout:995: snap_remove snap 93 2021-03-24T21:57:47.197 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:47.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:47.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:47.264 INFO:journalctl@ceph.osd.2.smithi125.stdout:Mar 24 21:57:47 smithi125 conmon[34620]: debug 2021-03-24T21:57:47.264 INFO:journalctl@ceph.osd.2.smithi125.stdout:Mar 24 21:57:47 smithi125 conmon[34620]: 2021-03-24T21:57:47.179+0000 7f47099de700 -1 osd.2 pg_epoch: 297 pg[7.8( v 293'124 (0'0,293'124] local-lis/les=198/199 n=2 ec=99/99 lis/c=198/198 les/c/f=199/199/0 sis=198) [2,7,4] r=0 lpr=198 crt=293'124 lcod 285'122 mlcod 285'122 active+clean+snaptrim trimq=[ba~1] ps=[b1~1,b3~2,b7~1,bc~1]] removing snap head 2021-03-24T21:57:47.302 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:47.310 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:57:47.334 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":605,"stamp":"2021-03-24T21:57:45.152083+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":130096170,"num_objects":359,"num_object_clones":8,"num_object_copies":1077,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":359,"num_whiteouts":0,"num_read":2809,"num_read_kb":1458101,"num_write":1489,"num_write_kb":374153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":539,"num_bytes_recovered":103497219,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6724,"ondisk_log_size":6724,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":504,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":469800,"kb_used_data":298744,"kb_used_omap":0,"kb_used_meta":171008,"kb_avail":562001112,"statfs":{"total":575970213888,"available":575489138688,"internally_reserved":0,"allocated":305913856,"data_stored":296246974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":175112192},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-10869833,"num_objects":-6,"num_object_clones":-4,"num_object_copies":-18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6,"num_whiteouts":-3,"num_read":30,"num_read_kb":34146,"num_write":18,"num_write_kb":13826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.884285"},"pg_stats":[{"pgid":"7.23","version":"281'260","reported_seq":"979","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.293364+0000","last_change":"2021-03-24T21:57:42.293364+0000","last_active":"2021-03-24T21:57:42.293364+0000","last_peered":"2021-03-24T21:57:42.293364+0000","last_clean":"2021-03-24T21:57:42.293364+0000","last_became_active":"2021-03-24T21:57:31.166310+0000","last_became_peered":"2021-03-24T21:57:31.166310+0000","last_unstale":"2021-03-24T21:57:42.293364+0000","last_undegraded":"2021-03-24T21:57:42.293364+0000","last_fullsized":"2021-03-24T21:57:42.293364+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":260,"ondisk_log_size":260,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4243784,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":86,"num_read_kb":104494,"num_write":50,"num_write_kb":19771,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":3418953,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"b0","length":"2"},{"start":"b3","length":"2"},{"start":"b7","length":"1"}]},{"pgid":"7.22","version":"289'72","reported_seq":"99","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.275657+0000","last_change":"2021-03-24T21:57:42.276920+0000","last_active":"2021-03-24T21:57:42.275657+0000","last_peered":"2021-03-24T21:57:42.275657+0000","last_clean":"2021-03-24T21:57:42.275657+0000","last_became_active":"2021-03-24T21:57:28.176845+0000","last_became_peered":"2021-03-24T21:57:28.176845+0000","last_unstale":"2021-03-24T21:57:42.275657+0000","last_undegraded":"2021-03-24T21:57:42.275657+0000","last_fullsized":"2021-03-24T21:57:42.275657+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":72,"ondisk_log_size":72,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":8519,"num_write":13,"num_write_kb":3707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":5774642,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"b0","length":"2"},{"start":"b3","length":"2"},{"start":"b7","length":"1"}]},{"pgid":"4.c","version":"69'13","reported_seq":"321","reported_epoch":"291","state":"active+clean","last_fresh":"2021-03-24T21:57:41.276630+0000","last_change":"2021-03-24T21:55:36.065455+0000","last_active":"2021-03-24T21:57:41.276630+0000","last_peered":"2021-03-24T21:57:41.276630+0000","last_clean":"2021-03-24T21:57:41.276630+0000","last_became_active":"2021-03-24T21:54:38.127115+0000","last_became_peered":"2021-03-24T21:54:38.127115+0000","last_unstale":"2021-03-24T21:57:41.276630+0000","last_undegraded":"2021-03-24T21:57:41.276630+0000","last_fullsized":"2021-03-24T21:57:41.276630+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2021-03-24T21:55:36.065267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:36.065267+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":13,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"253","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.292714+0000","last_change":"2021-03-24T21:56:42.413023+0000","last_active":"2021-03-24T21:57:42.292714+0000","last_peered":"2021-03-24T21:57:42.292714+0000","last_clean":"2021-03-24T21:57:42.292714+0000","last_became_active":"2021-03-24T21:51:45.375830+0000","last_became_peered":"2021-03-24T21:51:45.375830+0000","last_unstale":"2021-03-24T21:57:42.292714+0000","last_undegraded":"2021-03-24T21:57:42.292714+0000","last_fullsized":"2021-03-24T21:57:42.292714+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:42.412849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:42.412849+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"65'2","reported_seq":"320","reported_epoch":"290","state":"active+clean","last_fresh":"2021-03-24T21:57:40.157260+0000","last_change":"2021-03-24T21:56:33.126585+0000","last_active":"2021-03-24T21:57:40.157260+0000","last_peered":"2021-03-24T21:57:40.157260+0000","last_clean":"2021-03-24T21:57:40.157260+0000","last_became_active":"2021-03-24T21:51:49.416182+0000","last_became_peered":"2021-03-24T21:51:49.416182+0000","last_unstale":"2021-03-24T21:57:40.157260+0000","last_undegraded":"2021-03-24T21:57:40.157260+0000","last_fullsized":"2021-03-24T21:57:40.157260+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2021-03-24T21:56:33.126465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:33.126465+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"260","reported_epoch":"291","state":"active+clean","last_fresh":"2021-03-24T21:57:41.278072+0000","last_change":"2021-03-24T21:57:02.182918+0000","last_active":"2021-03-24T21:57:41.278072+0000","last_peered":"2021-03-24T21:57:41.278072+0000","last_clean":"2021-03-24T21:57:41.278072+0000","last_became_active":"2021-03-24T21:57:02.182551+0000","last_became_peered":"2021-03-24T21:57:02.182551+0000","last_unstale":"2021-03-24T21:57:41.278072+0000","last_undegraded":"2021-03-24T21:57:41.278072+0000","last_fullsized":"2021-03-24T21:57:41.278072+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:08.985694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:08.985694+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"277'101","reported_seq":"775","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.278860+0000","last_change":"2021-03-24T21:57:42.278860+0000","last_active":"2021-03-24T21:57:42.278860+0000","last_peered":"2021-03-24T21:57:42.278860+0000","last_clean":"2021-03-24T21:57:42.278860+0000","last_became_active":"2021-03-24T21:57:26.178185+0000","last_became_peered":"2021-03-24T21:57:26.178185+0000","last_unstale":"2021-03-24T21:57:42.278860+0000","last_undegraded":"2021-03-24T21:57:42.278860+0000","last_fullsized":"2021-03-24T21:57:42.278860+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"218'73","last_scrub_stamp":"2021-03-24T21:56:48.267711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:56:48.267711+0000","log_size":101,"ondisk_log_size":101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3571504,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":62,"num_read_kb":83922,"num_write":22,"num_write_kb":9163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":942482,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"b0","length":"2"},{"start":"b3","length":"2"},{"start":"b7","length":"1"}]},{"pgid":"4.d","version":"69'21","reported_seq":"322","reported_epoch":"291","state":"active+clean","last_fresh":"2021-03-24T21:57:41.274971+0000","last_change":"2021-03-24T21:57:18.801773+0000","last_active":"2021-03-24T21:57:41.274971+0000","last_peered":"2021-03-24T21:57:41.274971+0000","last_clean":"2021-03-24T21:57:41.274971+0000","last_became_active":"2021-03-24T21:54:38.144020+0000","last_became_peered":"2021-03-24T21:54:38.144020+0000","last_unstale":"2021-03-24T21:57:41.274971+0000","last_undegraded":"2021-03-24T21:57:41.274971+0000","last_fullsized":"2021-03-24T21:57:41.274971+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'21","last_scrub_stamp":"2021-03-24T21:57:18.801567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:57:18.801567+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":23,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"215","reported_epoch":"291","state":"active+clean","last_fresh":"2021-03-24T21:57:41.273890+0000","last_change":"2021-03-24T21:55:01.169611+0000","last_active":"2021-03-24T21:57:41.273890+0000","last_peered":"2021-03-24T21:57:41.273890+0000","last_clean":"2021-03-24T21:57:41.273890+0000","last_became_active":"2021-03-24T21:54:38.127136+0000","last_became_peered":"2021-03-24T21:54:38.127136+0000","last_unstale":"2021-03-24T21:57:41.273890+0000","last_undegraded":"2021-03-24T21:57:41.273890+0000","last_fullsized":"2021-03-24T21:57:41.273890+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:01.169461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:55:01.169461+0000","last_clean_scrub_stamp":"2021-03-24T21:55:01.169461+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"209","reported_epoch":"291","state":"active+clean","last_fresh":"2021-03-24T21:57:41.275416+0000","last_change":"2021-03-24T21:56:44.230926+0000","last_active":"2021-03-24T21:57:41.275416+0000","last_peered":"2021-03-24T21:57:41.275416+0000","last_clean":"2021-03-24T21:57:41.275416+0000","last_became_active":"2021-03-24T21:56:44.230543+0000","last_became_peered":"2021-03-24T21:56:44.230543+0000","last_unstale":"2021-03-24T21:57:41.275416+0000","last_undegraded":"2021-03-24T21:57:41.275416+0000","last_fullsized":"2021-03-24T21:57:41.275416+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:07.977138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:07.977138+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"52","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.290539+0000","last_change":"2021-03-24T21:57:02.182962+0000","last_active":"2021-03-24T21:57:42.290539+0000","last_peered":"2021-03-24T21:57:42.290539+0000","last_clean":"2021-03-24T21:57:42.290539+0000","last_became_active":"2021-03-24T21:57:02.181899+0000","last_became_peered":"2021-03-24T21:57:02.181899+0000","last_unstale":"2021-03-24T21:57:42.290539+0000","last_undegraded":"2021-03-24T21:57:42.290539+0000","last_fullsized":"2021-03-24T21:57:42.290539+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:49.970073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:49.970073+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"265'45","reported_seq":"669","reported_epoch":"291","state":"active+clean","last_fresh":"2021-03-24T21:57:41.273294+0000","last_change":"2021-03-24T21:57:41.273613+0000","last_active":"2021-03-24T21:57:41.273294+0000","last_peered":"2021-03-24T21:57:41.273294+0000","last_clean":"2021-03-24T21:57:41.273294+0000","last_became_active":"2021-03-24T21:57:26.167896+0000","last_became_peered":"2021-03-24T21:57:26.167896+0000","last_unstale":"2021-03-24T21:57:41.273294+0000","last_undegraded":"2021-03-24T21:57:41.273294+0000","last_fullsized":"2021-03-24T21:57:41.273294+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'45","last_scrub_stamp":"2021-03-24T21:57:27.105464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:57:27.105464+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"b0","length":"2"},{"start":"b3","length":"2"}]},{"pgid":"4.e","version":"69'15","reported_seq":"336","reported_epoch":"291","state":"active+clean","last_fresh":"2021-03-24T21:57:41.276459+0000","last_change":"2021-03-24T21:56:40.062483+0000","last_active":"2021-03-24T21:57:41.276459+0000","last_peered":"2021-03-24T21:57:41.276459+0000","last_clean":"2021-03-24T21:57:41.276459+0000","last_became_active":"2021-03-24T21:54:38.127461+0000","last_became_peered":"2021-03-24T21:54:38.127461+0000","last_unstale":"2021-03-24T21:57:41.276459+0000","last_undegraded":"2021-03-24T21:57:41.276459+0000","last_fullsized":"2021-03-24T21:57:41.276459+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:56:40.062304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:40.062304+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1476,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"248","reported_epoch":"291","state":"active+clean","last_fresh":"2021-03-24T21:57:41.272713+0000","last_change":"2021-03-24T21:54:53.267409+0000","last_active":"2021-03-24T21:57:41.272713+0000","last_peered":"2021-03-24T21:57:41.272713+0000","last_clean":"2021-03-24T21:57:41.272713+0000","last_became_active":"2021-03-24T21:51:45.579513+0000","last_became_peered":"2021-03-24T21:51:45.579513+0000","last_unstale":"2021-03-24T21:57:41.272713+0000","last_undegraded":"2021-03-24T21:57:41.272713+0000","last_fullsized":"2021-03-24T21:57:41.272713+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:53.267324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:54:53.267324+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"226","reported_epoch":"291","state":"active+clean","last_fresh":"2021-03-24T21:57:41.276839+0000","last_change":"2021-03-24T21:56:43.971055+0000","last_active":"2021-03-24T21:57:41.276839+0000","last_peered":"2021-03-24T21:57:41.276839+0000","last_clean":"2021-03-24T21:57:41.276839+0000","last_became_active":"2021-03-24T21:55:10.102113+0000","last_became_peered":"2021-03-24T21:55:10.102113+0000","last_unstale":"2021-03-24T21:57:41.276839+0000","last_undegraded":"2021-03-24T21:57:41.276839+0000","last_fullsized":"2021-03-24T21:57:41.276839+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:43.970879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:43.970879+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"251","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.294154+0000","last_change":"2021-03-24T21:55:59.472536+0000","last_active":"2021-03-24T21:57:42.294154+0000","last_peered":"2021-03-24T21:57:42.294154+0000","last_clean":"2021-03-24T21:57:42.294154+0000","last_became_active":"2021-03-24T21:54:24.196587+0000","last_became_peered":"2021-03-24T21:54:24.196587+0000","last_unstale":"2021-03-24T21:57:42.294154+0000","last_undegraded":"2021-03-24T21:57:42.294154+0000","last_fullsized":"2021-03-24T21:57:42.294154+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:59.472393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:55:59.472393+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"291'275","reported_seq":"925","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.285613+0000","last_change":"2021-03-24T21:57:42.285613+0000","last_active":"2021-03-24T21:57:42.285613+0000","last_peered":"2021-03-24T21:57:42.285613+0000","last_clean":"2021-03-24T21:57:42.285613+0000","last_became_active":"2021-03-24T21:57:26.189692+0000","last_became_peered":"2021-03-24T21:57:26.189692+0000","last_unstale":"2021-03-24T21:57:42.285613+0000","last_undegraded":"2021-03-24T21:57:42.285613+0000","last_fullsized":"2021-03-24T21:57:42.285613+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'169","last_scrub_stamp":"2021-03-24T21:56:47.285648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:56:47.285648+0000","log_size":275,"ondisk_log_size":275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1739574,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":72,"num_read_kb":65135,"num_write":65,"num_write_kb":25607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2546506,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"b0","length":"2"},{"start":"b3","length":"2"},{"start":"b7","length":"1"}]},{"pgid":"4.f","version":"274'26","reported_seq":"270","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.290773+0000","last_change":"2021-03-24T21:57:24.499178+0000","last_active":"2021-03-24T21:57:42.290773+0000","last_peered":"2021-03-24T21:57:42.290773+0000","last_clean":"2021-03-24T21:57:42.290773+0000","last_became_active":"2021-03-24T21:54:38.130536+0000","last_became_peered":"2021-03-24T21:54:38.130536+0000","last_unstale":"2021-03-24T21:57:42.290773+0000","last_undegraded":"2021-03-24T21:57:42.290773+0000","last_fullsized":"2021-03-24T21:57:42.290773+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'26","last_scrub_stamp":"2021-03-24T21:57:24.499090+0000","last_deep_scrub":"274'26","last_deep_scrub_stamp":"2021-03-24T21:57:24.499090+0000","last_clean_scrub_stamp":"2021-03-24T21:57:24.499090+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":18,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":422,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":"257","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.288634+0000","last_change":"2021-03-24T21:56:39.439282+0000","last_active":"2021-03-24T21:57:42.288634+0000","last_peered":"2021-03-24T21:57:42.288634+0000","last_clean":"2021-03-24T21:57:42.288634+0000","last_became_active":"2021-03-24T21:54:38.134768+0000","last_became_peered":"2021-03-24T21:54:38.134768+0000","last_unstale":"2021-03-24T21:57:42.288634+0000","last_undegraded":"2021-03-24T21:57:42.288634+0000","last_fullsized":"2021-03-24T21:57:42.288634+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:39.439170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:39.439170+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"65'2","reported_seq":"316","reported_epoch":"291","state":"active+clean","last_fresh":"2021-03-24T21:57:41.277951+0000","last_change":"2021-03-24T21:55:59.032346+0000","last_active":"2021-03-24T21:57:41.277951+0000","last_peered":"2021-03-24T21:57:41.277951+0000","last_clean":"2021-03-24T21:57:41.277951+0000","last_became_active":"2021-03-24T21:51:49.414380+0000","last_became_peered":"2021-03-24T21:51:49.414380+0000","last_unstale":"2021-03-24T21:57:41.277951+0000","last_undegraded":"2021-03-24T21:57:41.277951+0000","last_fullsized":"2021-03-24T21:57:41.277951+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2021-03-24T21:55:59.032170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:55:59.032170+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"258","reported_epoch":"291","state":"active+clean","last_fresh":"2021-03-24T21:57:41.283970+0000","last_change":"2021-03-24T21:57:23.176879+0000","last_active":"2021-03-24T21:57:41.283970+0000","last_peered":"2021-03-24T21:57:41.283970+0000","last_clean":"2021-03-24T21:57:41.283970+0000","last_became_active":"2021-03-24T21:54:38.129469+0000","last_became_peered":"2021-03-24T21:54:38.129469+0000","last_unstale":"2021-03-24T21:57:41.283970+0000","last_undegraded":"2021-03-24T21:57:41.283970+0000","last_fullsized":"2021-03-24T21:57:41.283970+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:23.176750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:57:23.176750+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"292'284","reported_seq":"922","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.298612+0000","last_change":"2021-03-24T21:57:42.298612+0000","last_active":"2021-03-24T21:57:42.298612+0000","last_peered":"2021-03-24T21:57:42.298612+0000","last_clean":"2021-03-24T21:57:42.298612+0000","last_became_active":"2021-03-24T21:57:26.174174+0000","last_became_peered":"2021-03-24T21:57:26.174174+0000","last_unstale":"2021-03-24T21:57:42.298612+0000","last_undegraded":"2021-03-24T21:57:42.298612+0000","last_fullsized":"2021-03-24T21:57:42.298612+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":284,"ondisk_log_size":284,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":689153,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":85,"num_read_kb":73520,"num_write":69,"num_write_kb":26480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"b0","length":"2"},{"start":"b3","length":"2"},{"start":"b7","length":"1"}]},{"pgid":"5.1","version":"0'0","reported_seq":"244","reported_epoch":"291","state":"active+clean","last_fresh":"2021-03-24T21:57:41.272703+0000","last_change":"2021-03-24T21:55:25.106515+0000","last_active":"2021-03-24T21:57:41.272703+0000","last_peered":"2021-03-24T21:57:41.272703+0000","last_clean":"2021-03-24T21:57:41.272703+0000","last_became_active":"2021-03-24T21:51:49.408761+0000","last_became_peered":"2021-03-24T21:51:49.408761+0000","last_unstale":"2021-03-24T21:57:41.272703+0000","last_undegraded":"2021-03-24T21:57:41.272703+0000","last_fullsized":"2021-03-24T21:57:41.272703+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:25.106440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:25.220436+0000","last_clean_scrub_stamp":"2021-03-24T21:55:25.106440+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"69'22","reported_seq":"286","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.288446+0000","last_change":"2021-03-24T21:55:28.629625+0000","last_active":"2021-03-24T21:57:42.288446+0000","last_peered":"2021-03-24T21:57:42.288446+0000","last_clean":"2021-03-24T21:57:42.288446+0000","last_became_active":"2021-03-24T21:51:47.691459+0000","last_became_peered":"2021-03-24T21:51:47.691459+0000","last_unstale":"2021-03-24T21:57:42.288446+0000","last_undegraded":"2021-03-24T21:57:42.288446+0000","last_fullsized":"2021-03-24T21:57:42.288446+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'22","last_scrub_stamp":"2021-03-24T21:55:28.629443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:28.629443+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":46,"num_read_kb":26,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"248","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.291166+0000","last_change":"2021-03-24T21:54:56.508843+0000","last_active":"2021-03-24T21:57:42.291166+0000","last_peered":"2021-03-24T21:57:42.291166+0000","last_clean":"2021-03-24T21:57:42.291166+0000","last_became_active":"2021-03-24T21:51:45.378704+0000","last_became_peered":"2021-03-24T21:51:45.378704+0000","last_unstale":"2021-03-24T21:57:42.291166+0000","last_undegraded":"2021-03-24T21:57:42.291166+0000","last_fullsized":"2021-03-24T21:57:42.291166+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:56.508652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:54:56.508652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"242","reported_epoch":"291","state":"active+clean","last_fresh":"2021-03-24T21:57:41.273718+0000","last_change":"2021-03-24T21:56:35.045715+0000","last_active":"2021-03-24T21:57:41.273718+0000","last_peered":"2021-03-24T21:57:41.273718+0000","last_clean":"2021-03-24T21:57:41.273718+0000","last_became_active":"2021-03-24T21:51:51.756114+0000","last_became_peered":"2021-03-24T21:51:51.756114+0000","last_unstale":"2021-03-24T21:57:41.273718+0000","last_undegraded":"2021-03-24T21:57:41.273718+0000","last_fullsized":"2021-03-24T21:57:41.273718+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:35.045534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:35.045534+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"288'255","reported_seq":"944","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.293609+0000","last_change":"2021-03-24T21:57:42.293609+0000","last_active":"2021-03-24T21:57:42.293609+0000","last_peered":"2021-03-24T21:57:42.293609+0000","last_clean":"2021-03-24T21:57:42.293609+0000","last_became_active":"2021-03-24T21:57:26.169302+0000","last_became_peered":"2021-03-24T21:57:26.169302+0000","last_unstale":"2021-03-24T21:57:42.293609+0000","last_undegraded":"2021-03-24T21:57:42.293609+0000","last_fullsized":"2021-03-24T21:57:42.293609+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":255,"ondisk_log_size":255,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2764768,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":93,"num_read_kb":117169,"num_write":41,"num_write_kb":15965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"b0","length":"2"},{"start":"b3","length":"2"},{"start":"b7","length":"1"}]},{"pgid":"5.0","version":"109'3","reported_seq":"335","reported_epoch":"293","state":"active+clean","last_fresh":"2021-03-24T21:57:43.289692+0000","last_change":"2021-03-24T21:57:25.463136+0000","last_active":"2021-03-24T21:57:43.289692+0000","last_peered":"2021-03-24T21:57:43.289692+0000","last_clean":"2021-03-24T21:57:43.289692+0000","last_became_active":"2021-03-24T21:54:38.135726+0000","last_became_peered":"2021-03-24T21:54:38.135726+0000","last_unstale":"2021-03-24T21:57:43.289692+0000","last_undegraded":"2021-03-24T21:57:43.289692+0000","last_fullsized":"2021-03-24T21:57:43.289692+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'3","last_scrub_stamp":"2021-03-24T21:57:25.462952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:57:25.462952+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"69'20","reported_seq":"321","reported_epoch":"291","state":"active+clean","last_fresh":"2021-03-24T21:57:41.272847+0000","last_change":"2021-03-24T21:57:21.706231+0000","last_active":"2021-03-24T21:57:41.272847+0000","last_peered":"2021-03-24T21:57:41.272847+0000","last_clean":"2021-03-24T21:57:41.272847+0000","last_became_active":"2021-03-24T21:54:24.189182+0000","last_became_peered":"2021-03-24T21:54:24.189182+0000","last_unstale":"2021-03-24T21:57:41.272847+0000","last_undegraded":"2021-03-24T21:57:41.272847+0000","last_fullsized":"2021-03-24T21:57:41.272847+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'20","last_scrub_stamp":"2021-03-24T21:57:21.706127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:57:21.706127+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":39,"num_read_kb":20,"num_write":20,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":940,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"266","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.277303+0000","last_change":"2021-03-24T21:56:44.229920+0000","last_active":"2021-03-24T21:57:42.277303+0000","last_peered":"2021-03-24T21:57:42.277303+0000","last_clean":"2021-03-24T21:57:42.277303+0000","last_became_active":"2021-03-24T21:56:44.229469+0000","last_became_peered":"2021-03-24T21:56:44.229469+0000","last_unstale":"2021-03-24T21:57:42.277303+0000","last_undegraded":"2021-03-24T21:57:42.277303+0000","last_fullsized":"2021-03-24T21:57:42.277303+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:28.165408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:28.165408+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"266","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.283805+0000","last_change":"2021-03-24T21:56:40.011790+0000","last_active":"2021-03-24T21:57:42.283805+0000","last_peered":"2021-03-24T21:57:42.283805+0000","last_clean":"2021-03-24T21:57:42.283805+0000","last_became_active":"2021-03-24T21:54:38.110246+0000","last_became_peered":"2021-03-24T21:54:38.110246+0000","last_unstale":"2021-03-24T21:57:42.283805+0000","last_undegraded":"2021-03-24T21:57:42.283805+0000","last_fullsized":"2021-03-24T21:57:42.283805+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:40.011633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:40.011633+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"272'64","reported_seq":"703","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.290004+0000","last_change":"2021-03-24T21:57:42.290004+0000","last_active":"2021-03-24T21:57:42.290004+0000","last_peered":"2021-03-24T21:57:42.290004+0000","last_clean":"2021-03-24T21:57:42.290004+0000","last_became_active":"2021-03-24T21:57:26.174158+0000","last_became_peered":"2021-03-24T21:57:26.174158+0000","last_unstale":"2021-03-24T21:57:42.290004+0000","last_undegraded":"2021-03-24T21:57:42.290004+0000","last_fullsized":"2021-03-24T21:57:42.290004+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":64,"ondisk_log_size":64,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1922978,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":8518,"num_write":11,"num_write_kb":3706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":216931,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"b0","length":"2"},{"start":"b3","length":"2"},{"start":"b7","length":"1"}]},{"pgid":"5.3","version":"102'4","reported_seq":"331","reported_epoch":"293","state":"active+clean","last_fresh":"2021-03-24T21:57:43.358799+0000","last_change":"2021-03-24T21:56:01.267040+0000","last_active":"2021-03-24T21:57:43.358799+0000","last_peered":"2021-03-24T21:57:43.358799+0000","last_clean":"2021-03-24T21:57:43.358799+0000","last_became_active":"2021-03-24T21:54:24.203310+0000","last_became_peered":"2021-03-24T21:54:24.203310+0000","last_unstale":"2021-03-24T21:57:43.358799+0000","last_undegraded":"2021-03-24T21:57:43.358799+0000","last_fullsized":"2021-03-24T21:57:43.358799+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'4","last_scrub_stamp":"2021-03-24T21:56:01.266936+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-24T21:53:26.584678+0000","last_clean_scrub_stamp":"2021-03-24T21:56:01.266936+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"69'16","reported_seq":"269","reported_epoch":"291","state":"active+clean","last_fresh":"2021-03-24T21:57:41.282952+0000","last_change":"2021-03-24T21:56:41.235280+0000","last_active":"2021-03-24T21:57:41.282952+0000","last_peered":"2021-03-24T21:57:41.282952+0000","last_clean":"2021-03-24T21:57:41.282952+0000","last_became_active":"2021-03-24T21:54:38.128745+0000","last_became_peered":"2021-03-24T21:54:38.128745+0000","last_unstale":"2021-03-24T21:57:41.282952+0000","last_undegraded":"2021-03-24T21:57:41.282952+0000","last_fullsized":"2021-03-24T21:57:41.282952+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T21:56:41.235143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:41.235143+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":32,"num_read_kb":15,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":940,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"255","reported_epoch":"291","state":"active+clean","last_fresh":"2021-03-24T21:57:41.284255+0000","last_change":"2021-03-24T21:55:57.194772+0000","last_active":"2021-03-24T21:57:41.284255+0000","last_peered":"2021-03-24T21:57:41.284255+0000","last_clean":"2021-03-24T21:57:41.284255+0000","last_became_active":"2021-03-24T21:51:45.583885+0000","last_became_peered":"2021-03-24T21:51:45.583885+0000","last_unstale":"2021-03-24T21:57:41.284255+0000","last_undegraded":"2021-03-24T21:57:41.284255+0000","last_fullsized":"2021-03-24T21:57:41.284255+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:57.194610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:57.194610+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"242","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.275570+0000","last_change":"2021-03-24T21:55:39.261965+0000","last_active":"2021-03-24T21:57:42.275570+0000","last_peered":"2021-03-24T21:57:42.275570+0000","last_clean":"2021-03-24T21:57:42.275570+0000","last_became_active":"2021-03-24T21:51:51.759627+0000","last_became_peered":"2021-03-24T21:51:51.759627+0000","last_unstale":"2021-03-24T21:57:42.275570+0000","last_undegraded":"2021-03-24T21:57:42.275570+0000","last_fullsized":"2021-03-24T21:57:42.275570+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:39.261698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:55:39.261698+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"277'271","reported_seq":"1019","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.275473+0000","last_change":"2021-03-24T21:57:42.277328+0000","last_active":"2021-03-24T21:57:42.275473+0000","last_peered":"2021-03-24T21:57:42.275473+0000","last_clean":"2021-03-24T21:57:42.275473+0000","last_became_active":"2021-03-24T21:57:26.191103+0000","last_became_peered":"2021-03-24T21:57:26.191103+0000","last_unstale":"2021-03-24T21:57:42.275473+0000","last_undegraded":"2021-03-24T21:57:42.275473+0000","last_fullsized":"2021-03-24T21:57:42.275473+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":271,"ondisk_log_size":271,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4667600,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":73,"num_read_kb":70782,"num_write":40,"num_write_kb":17590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":5112615,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"b0","length":"2"},{"start":"b3","length":"2"},{"start":"b7","length":"1"}]},{"pgid":"5.2","version":"0'0","reported_seq":"207","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.276462+0000","last_change":"2021-03-24T21:55:58.253884+0000","last_active":"2021-03-24T21:57:42.276462+0000","last_peered":"2021-03-24T21:57:42.276462+0000","last_clean":"2021-03-24T21:57:42.276462+0000","last_became_active":"2021-03-24T21:54:24.198895+0000","last_became_peered":"2021-03-24T21:54:24.198895+0000","last_unstale":"2021-03-24T21:57:42.276462+0000","last_undegraded":"2021-03-24T21:57:42.276462+0000","last_fullsized":"2021-03-24T21:57:42.276462+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:58.253777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:55:58.253777+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"69'21","reported_seq":"291","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.276931+0000","last_change":"2021-03-24T21:56:39.335724+0000","last_active":"2021-03-24T21:57:42.276931+0000","last_peered":"2021-03-24T21:57:42.276931+0000","last_clean":"2021-03-24T21:57:42.276931+0000","last_became_active":"2021-03-24T21:51:47.686486+0000","last_became_peered":"2021-03-24T21:51:47.686486+0000","last_unstale":"2021-03-24T21:57:42.276931+0000","last_undegraded":"2021-03-24T21:57:42.276931+0000","last_fullsized":"2021-03-24T21:57:42.276931+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'21","last_scrub_stamp":"2021-03-24T21:56:39.335613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:39.335613+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":45,"num_read_kb":27,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"54","reported_epoch":"291","state":"active+clean","last_fresh":"2021-03-24T21:57:41.283549+0000","last_change":"2021-03-24T21:57:02.189207+0000","last_active":"2021-03-24T21:57:41.283549+0000","last_peered":"2021-03-24T21:57:41.283549+0000","last_clean":"2021-03-24T21:57:41.283549+0000","last_became_active":"2021-03-24T21:57:02.188835+0000","last_became_peered":"2021-03-24T21:57:02.188835+0000","last_unstale":"2021-03-24T21:57:41.283549+0000","last_undegraded":"2021-03-24T21:57:41.283549+0000","last_fullsized":"2021-03-24T21:57:41.283549+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:38.144956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:38.144956+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"104","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.282825+0000","last_change":"2021-03-24T21:57:02.189486+0000","last_active":"2021-03-24T21:57:42.282825+0000","last_peered":"2021-03-24T21:57:42.282825+0000","last_clean":"2021-03-24T21:57:42.282825+0000","last_became_active":"2021-03-24T21:57:02.189009+0000","last_became_peered":"2021-03-24T21:57:02.189009+0000","last_unstale":"2021-03-24T21:57:42.282825+0000","last_undegraded":"2021-03-24T21:57:42.282825+0000","last_fullsized":"2021-03-24T21:57:42.282825+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:01.124573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:01.124573+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.0","version":"175'22","reported_seq":"665","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.286881+0000","last_change":"2021-03-24T21:57:42.287239+0000","last_active":"2021-03-24T21:57:42.286881+0000","last_peered":"2021-03-24T21:57:42.286881+0000","last_clean":"2021-03-24T21:57:42.286881+0000","last_became_active":"2021-03-24T21:57:26.169706+0000","last_became_peered":"2021-03-24T21:57:26.169706+0000","last_unstale":"2021-03-24T21:57:42.286881+0000","last_undegraded":"2021-03-24T21:57:42.286881+0000","last_fullsized":"2021-03-24T21:57:42.286881+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'22","last_scrub_stamp":"2021-03-24T21:56:40.276617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:56:40.276617+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"b0","length":"2"},{"start":"b3","length":"2"},{"start":"b7","length":"1"}]},{"pgid":"5.7","version":"0'0","reported_seq":"260","reported_epoch":"291","state":"active+clean","last_fresh":"2021-03-24T21:57:41.286320+0000","last_change":"2021-03-24T21:56:43.194373+0000","last_active":"2021-03-24T21:57:41.286320+0000","last_peered":"2021-03-24T21:57:41.286320+0000","last_clean":"2021-03-24T21:57:41.286320+0000","last_became_active":"2021-03-24T21:54:38.128708+0000","last_became_peered":"2021-03-24T21:54:38.128708+0000","last_unstale":"2021-03-24T21:57:41.286320+0000","last_undegraded":"2021-03-24T21:57:41.286320+0000","last_fullsized":"2021-03-24T21:57:41.286320+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:43.194160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:43.194160+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"69'16","reported_seq":"339","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.278939+0000","last_change":"2021-03-24T21:57:26.257787+0000","last_active":"2021-03-24T21:57:42.278939+0000","last_peered":"2021-03-24T21:57:42.278939+0000","last_clean":"2021-03-24T21:57:42.278939+0000","last_became_active":"2021-03-24T21:57:02.185489+0000","last_became_peered":"2021-03-24T21:57:02.185489+0000","last_unstale":"2021-03-24T21:57:42.278939+0000","last_undegraded":"2021-03-24T21:57:42.278939+0000","last_fullsized":"2021-03-24T21:57:42.278939+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T21:57:26.257686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:57:26.257686+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":16,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1260,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"66'5","reported_seq":"631","reported_epoch":"293","state":"active+clean","last_fresh":"2021-03-24T21:57:43.295121+0000","last_change":"2021-03-24T21:56:37.973047+0000","last_active":"2021-03-24T21:57:43.295121+0000","last_peered":"2021-03-24T21:57:43.295121+0000","last_clean":"2021-03-24T21:57:43.295121+0000","last_became_active":"2021-03-24T21:54:38.125434+0000","last_became_peered":"2021-03-24T21:54:38.125434+0000","last_unstale":"2021-03-24T21:57:43.295121+0000","last_undegraded":"2021-03-24T21:57:43.295121+0000","last_fullsized":"2021-03-24T21:57:43.295121+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2021-03-24T21:56:37.972874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:39.286101+0000","last_clean_scrub_stamp":"2021-03-24T21:56:37.972874+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":356,"num_read_kb":351,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"253","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.275548+0000","last_change":"2021-03-24T21:56:38.304954+0000","last_active":"2021-03-24T21:57:42.275548+0000","last_peered":"2021-03-24T21:57:42.275548+0000","last_clean":"2021-03-24T21:57:42.275548+0000","last_became_active":"2021-03-24T21:51:45.376687+0000","last_became_peered":"2021-03-24T21:51:45.376687+0000","last_unstale":"2021-03-24T21:57:42.275548+0000","last_undegraded":"2021-03-24T21:57:42.275548+0000","last_fullsized":"2021-03-24T21:57:42.275548+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:38.304792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:38.304792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"46","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.277039+0000","last_change":"2021-03-24T21:57:06.153154+0000","last_active":"2021-03-24T21:57:42.277039+0000","last_peered":"2021-03-24T21:57:42.277039+0000","last_clean":"2021-03-24T21:57:42.277039+0000","last_became_active":"2021-03-24T21:57:06.152772+0000","last_became_peered":"2021-03-24T21:57:06.152772+0000","last_unstale":"2021-03-24T21:57:42.277039+0000","last_undegraded":"2021-03-24T21:57:42.277039+0000","last_fullsized":"2021-03-24T21:57:42.277039+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:41.503758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:55:41.503758+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.5","version":"195'69","reported_seq":"695","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.287222+0000","last_change":"2021-03-24T21:57:42.287222+0000","last_active":"2021-03-24T21:57:42.287222+0000","last_peered":"2021-03-24T21:57:42.287222+0000","last_clean":"2021-03-24T21:57:42.287222+0000","last_became_active":"2021-03-24T21:56:08.152168+0000","last_became_peered":"2021-03-24T21:56:08.152168+0000","last_unstale":"2021-03-24T21:57:42.287222+0000","last_undegraded":"2021-03-24T21:57:42.287222+0000","last_fullsized":"2021-03-24T21:57:42.287222+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'69","last_scrub_stamp":"2021-03-24T21:56:49.886058+0000","last_deep_scrub":"195'69","last_deep_scrub_stamp":"2021-03-24T21:56:49.886058+0000","last_clean_scrub_stamp":"2021-03-24T21:56:49.886058+0000","log_size":69,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"b0","length":"2"},{"start":"b3","length":"2"},{"start":"b7","length":"1"}]},{"pgid":"5.4","version":"0'0","reported_seq":"250","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.285022+0000","last_change":"2021-03-24T21:57:18.200083+0000","last_active":"2021-03-24T21:57:42.285022+0000","last_peered":"2021-03-24T21:57:42.285022+0000","last_clean":"2021-03-24T21:57:42.285022+0000","last_became_active":"2021-03-24T21:51:49.413251+0000","last_became_peered":"2021-03-24T21:51:49.413251+0000","last_unstale":"2021-03-24T21:57:42.285022+0000","last_undegraded":"2021-03-24T21:57:42.285022+0000","last_fullsized":"2021-03-24T21:57:42.285022+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:18.199919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:57:18.199919+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"69'15","reported_seq":"263","reported_epoch":"291","state":"active+clean","last_fresh":"2021-03-24T21:57:41.275931+0000","last_change":"2021-03-24T21:55:30.166245+0000","last_active":"2021-03-24T21:57:41.275931+0000","last_peered":"2021-03-24T21:57:41.275931+0000","last_clean":"2021-03-24T21:57:41.275931+0000","last_became_active":"2021-03-24T21:54:24.194711+0000","last_became_peered":"2021-03-24T21:54:24.194711+0000","last_unstale":"2021-03-24T21:57:41.275931+0000","last_undegraded":"2021-03-24T21:57:41.275931+0000","last_fullsized":"2021-03-24T21:57:41.275931+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:55:30.166107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:30.166107+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":26,"num_read_kb":14,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":466,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"257","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.289277+0000","last_change":"2021-03-24T21:55:33.534038+0000","last_active":"2021-03-24T21:57:42.289277+0000","last_peered":"2021-03-24T21:57:42.289277+0000","last_clean":"2021-03-24T21:57:42.289277+0000","last_became_active":"2021-03-24T21:54:24.199349+0000","last_became_peered":"2021-03-24T21:54:24.199349+0000","last_unstale":"2021-03-24T21:57:42.289277+0000","last_undegraded":"2021-03-24T21:57:42.289277+0000","last_fullsized":"2021-03-24T21:57:42.289277+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:33.533840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:33.533840+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"277","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.282594+0000","last_change":"2021-03-24T21:56:05.075868+0000","last_active":"2021-03-24T21:57:42.282594+0000","last_peered":"2021-03-24T21:57:42.282594+0000","last_clean":"2021-03-24T21:57:42.282594+0000","last_became_active":"2021-03-24T21:54:38.110327+0000","last_became_peered":"2021-03-24T21:54:38.110327+0000","last_unstale":"2021-03-24T21:57:42.282594+0000","last_undegraded":"2021-03-24T21:57:42.282594+0000","last_fullsized":"2021-03-24T21:57:42.282594+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:05.075677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:49:51.529051+0000","last_clean_scrub_stamp":"2021-03-24T21:56:05.075677+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"249","reported_epoch":"291","state":"active+clean","last_fresh":"2021-03-24T21:57:41.286003+0000","last_change":"2021-03-24T21:56:04.174062+0000","last_active":"2021-03-24T21:57:41.286003+0000","last_peered":"2021-03-24T21:57:41.286003+0000","last_clean":"2021-03-24T21:57:41.286003+0000","last_became_active":"2021-03-24T21:51:51.765236+0000","last_became_peered":"2021-03-24T21:51:51.765236+0000","last_unstale":"2021-03-24T21:57:41.286003+0000","last_undegraded":"2021-03-24T21:57:41.286003+0000","last_fullsized":"2021-03-24T21:57:41.286003+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:04.173942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:04.173942+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"292'155","reported_seq":"861","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.283935+0000","last_change":"2021-03-24T21:57:42.283935+0000","last_active":"2021-03-24T21:57:42.283935+0000","last_peered":"2021-03-24T21:57:42.283935+0000","last_clean":"2021-03-24T21:57:42.283935+0000","last_became_active":"2021-03-24T21:57:09.179214+0000","last_became_peered":"2021-03-24T21:57:09.179214+0000","last_unstale":"2021-03-24T21:57:42.283935+0000","last_undegraded":"2021-03-24T21:57:42.283935+0000","last_fullsized":"2021-03-24T21:57:42.283935+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":155,"ondisk_log_size":155,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5243800,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":60,"num_read_kb":68272,"num_write":40,"num_write_kb":17145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":8481953,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"b0","length":"2"},{"start":"b3","length":"2"},{"start":"b7","length":"1"}]},{"pgid":"5.6","version":"0'0","reported_seq":"56","reported_epoch":"291","state":"active+clean","last_fresh":"2021-03-24T21:57:41.284022+0000","last_change":"2021-03-24T21:57:20.215284+0000","last_active":"2021-03-24T21:57:41.284022+0000","last_peered":"2021-03-24T21:57:41.284022+0000","last_clean":"2021-03-24T21:57:41.284022+0000","last_became_active":"2021-03-24T21:57:02.182240+0000","last_became_peered":"2021-03-24T21:57:02.182240+0000","last_unstale":"2021-03-24T21:57:41.284022+0000","last_undegraded":"2021-03-24T21:57:41.284022+0000","last_fullsized":"2021-03-24T21:57:41.284022+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:20.215065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:57:20.215065+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"89'22","reported_seq":"288","reported_epoch":"291","state":"active+clean","last_fresh":"2021-03-24T21:57:41.286205+0000","last_change":"2021-03-24T21:56:54.046391+0000","last_active":"2021-03-24T21:57:41.286205+0000","last_peered":"2021-03-24T21:57:41.286205+0000","last_clean":"2021-03-24T21:57:41.286205+0000","last_became_active":"2021-03-24T21:54:38.129459+0000","last_became_peered":"2021-03-24T21:54:38.129459+0000","last_unstale":"2021-03-24T21:57:41.286205+0000","last_undegraded":"2021-03-24T21:57:41.286205+0000","last_fullsized":"2021-03-24T21:57:41.286205+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'22","last_scrub_stamp":"2021-03-24T21:56:54.046194+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2021-03-24T21:53:19.717244+0000","last_clean_scrub_stamp":"2021-03-24T21:56:54.046194+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":36,"num_read_kb":21,"num_write":18,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":644,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"261","reported_epoch":"290","state":"active+clean","last_fresh":"2021-03-24T21:57:40.156025+0000","last_change":"2021-03-24T21:57:23.071573+0000","last_active":"2021-03-24T21:57:40.156025+0000","last_peered":"2021-03-24T21:57:40.156025+0000","last_clean":"2021-03-24T21:57:40.156025+0000","last_became_active":"2021-03-24T21:51:40.351000+0000","last_became_peered":"2021-03-24T21:51:40.351000+0000","last_unstale":"2021-03-24T21:57:40.156025+0000","last_undegraded":"2021-03-24T21:57:40.156025+0000","last_fullsized":"2021-03-24T21:57:40.156025+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:23.071394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:39.286101+0000","last_clean_scrub_stamp":"2021-03-24T21:57:23.071394+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"61'1","reported_seq":"214","reported_epoch":"290","state":"active+clean","last_fresh":"2021-03-24T21:57:40.157123+0000","last_change":"2021-03-24T21:57:12.046385+0000","last_active":"2021-03-24T21:57:40.157123+0000","last_peered":"2021-03-24T21:57:40.157123+0000","last_clean":"2021-03-24T21:57:40.157123+0000","last_became_active":"2021-03-24T21:54:38.123139+0000","last_became_peered":"2021-03-24T21:54:38.123139+0000","last_unstale":"2021-03-24T21:57:40.157123+0000","last_undegraded":"2021-03-24T21:57:40.157123+0000","last_fullsized":"2021-03-24T21:57:40.157123+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2021-03-24T21:57:12.046206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:12.046206+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"252","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.286415+0000","last_change":"2021-03-24T21:55:34.931146+0000","last_active":"2021-03-24T21:57:42.286415+0000","last_peered":"2021-03-24T21:57:42.286415+0000","last_clean":"2021-03-24T21:57:42.286415+0000","last_became_active":"2021-03-24T21:54:24.186957+0000","last_became_peered":"2021-03-24T21:54:24.186957+0000","last_unstale":"2021-03-24T21:57:42.286415+0000","last_undegraded":"2021-03-24T21:57:42.286415+0000","last_fullsized":"2021-03-24T21:57:42.286415+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:34.930967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:55:34.930967+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"293'179","reported_seq":"830","reported_epoch":"293","state":"active+clean","last_fresh":"2021-03-24T21:57:43.382782+0000","last_change":"2021-03-24T21:57:42.278342+0000","last_active":"2021-03-24T21:57:43.382782+0000","last_peered":"2021-03-24T21:57:43.382782+0000","last_clean":"2021-03-24T21:57:43.382782+0000","last_became_active":"2021-03-24T21:56:08.154125+0000","last_became_peered":"2021-03-24T21:56:08.154125+0000","last_unstale":"2021-03-24T21:57:43.382782+0000","last_undegraded":"2021-03-24T21:57:43.382782+0000","last_fullsized":"2021-03-24T21:57:43.382782+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'177","last_scrub_stamp":"2021-03-24T21:57:21.260725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:57:21.260725+0000","log_size":179,"ondisk_log_size":179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7042199,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":9313,"num_write":1,"num_write_kb":3776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"b0","length":"2"},{"start":"b3","length":"2"},{"start":"b7","length":"1"}]},{"pgid":"5.5","version":"0'0","reported_seq":"253","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.279343+0000","last_change":"2021-03-24T21:55:36.236833+0000","last_active":"2021-03-24T21:57:42.279343+0000","last_peered":"2021-03-24T21:57:42.279343+0000","last_clean":"2021-03-24T21:57:42.279343+0000","last_became_active":"2021-03-24T21:54:38.136859+0000","last_became_peered":"2021-03-24T21:54:38.136859+0000","last_unstale":"2021-03-24T21:57:42.279343+0000","last_undegraded":"2021-03-24T21:57:42.279343+0000","last_fullsized":"2021-03-24T21:57:42.279343+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:36.236594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:55:36.236594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"69'25","reported_seq":"290","reported_epoch":"290","state":"active+clean","last_fresh":"2021-03-24T21:57:40.156888+0000","last_change":"2021-03-24T21:57:15.071588+0000","last_active":"2021-03-24T21:57:40.156888+0000","last_peered":"2021-03-24T21:57:40.156888+0000","last_clean":"2021-03-24T21:57:40.156888+0000","last_became_active":"2021-03-24T21:54:38.141478+0000","last_became_peered":"2021-03-24T21:54:38.141478+0000","last_unstale":"2021-03-24T21:57:40.156888+0000","last_undegraded":"2021-03-24T21:57:40.156888+0000","last_fullsized":"2021-03-24T21:57:40.156888+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'25","last_scrub_stamp":"2021-03-24T21:57:15.071486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:57:15.071486+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":53,"num_read_kb":30,"num_write":28,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"89'2","reported_seq":"277","reported_epoch":"291","state":"active+clean","last_fresh":"2021-03-24T21:57:41.274858+0000","last_change":"2021-03-24T21:55:38.046157+0000","last_active":"2021-03-24T21:57:41.274858+0000","last_peered":"2021-03-24T21:57:41.274858+0000","last_clean":"2021-03-24T21:57:41.274858+0000","last_became_active":"2021-03-24T21:54:38.144013+0000","last_became_peered":"2021-03-24T21:54:38.144013+0000","last_unstale":"2021-03-24T21:57:41.274858+0000","last_undegraded":"2021-03-24T21:57:41.274858+0000","last_fullsized":"2021-03-24T21:57:41.274858+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2021-03-24T21:55:38.045901+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2021-03-24T21:53:28.219900+0000","last_clean_scrub_stamp":"2021-03-24T21:55:38.045901+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"238","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.276765+0000","last_change":"2021-03-24T21:55:10.100603+0000","last_active":"2021-03-24T21:57:42.276765+0000","last_peered":"2021-03-24T21:57:42.276765+0000","last_clean":"2021-03-24T21:57:42.276765+0000","last_became_active":"2021-03-24T21:55:10.100245+0000","last_became_peered":"2021-03-24T21:55:10.100245+0000","last_unstale":"2021-03-24T21:57:42.276765+0000","last_undegraded":"2021-03-24T21:57:42.276765+0000","last_fullsized":"2021-03-24T21:57:42.276765+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:03.195853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:06.800417+0000","last_clean_scrub_stamp":"2021-03-24T21:55:03.195853+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"247","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.291540+0000","last_change":"2021-03-24T21:57:22.422901+0000","last_active":"2021-03-24T21:57:42.291540+0000","last_peered":"2021-03-24T21:57:42.291540+0000","last_clean":"2021-03-24T21:57:42.291540+0000","last_became_active":"2021-03-24T21:51:51.759514+0000","last_became_peered":"2021-03-24T21:51:51.759514+0000","last_unstale":"2021-03-24T21:57:42.291540+0000","last_undegraded":"2021-03-24T21:57:42.291540+0000","last_fullsized":"2021-03-24T21:57:42.291540+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:22.422787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:52:59.706511+0000","last_clean_scrub_stamp":"2021-03-24T21:57:22.422787+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"285'227","reported_seq":"890","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.279427+0000","last_change":"2021-03-24T21:57:42.279427+0000","last_active":"2021-03-24T21:57:42.279427+0000","last_peered":"2021-03-24T21:57:42.279427+0000","last_clean":"2021-03-24T21:57:42.279427+0000","last_became_active":"2021-03-24T21:56:08.157153+0000","last_became_peered":"2021-03-24T21:56:08.157153+0000","last_unstale":"2021-03-24T21:57:42.279427+0000","last_undegraded":"2021-03-24T21:57:42.279427+0000","last_fullsized":"2021-03-24T21:57:42.279427+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'218","last_scrub_stamp":"2021-03-24T21:57:22.239425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:57:22.239425+0000","log_size":227,"ondisk_log_size":227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3798415,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"b0","length":"2"},{"start":"b3","length":"2"},{"start":"b7","length":"1"}]},{"pgid":"4.b","version":"69'14","reported_seq":"301","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.276683+0000","last_change":"2021-03-24T21:55:57.297176+0000","last_active":"2021-03-24T21:57:42.276683+0000","last_peered":"2021-03-24T21:57:42.276683+0000","last_clean":"2021-03-24T21:57:42.276683+0000","last_became_active":"2021-03-24T21:54:38.138034+0000","last_became_peered":"2021-03-24T21:54:38.138034+0000","last_unstale":"2021-03-24T21:57:42.276683+0000","last_undegraded":"2021-03-24T21:57:42.276683+0000","last_fullsized":"2021-03-24T21:57:42.276683+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'14","last_scrub_stamp":"2021-03-24T21:55:57.297084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:57.297084+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":16,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":894,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"243","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.294198+0000","last_change":"2021-03-24T21:57:12.514184+0000","last_active":"2021-03-24T21:57:42.294198+0000","last_peered":"2021-03-24T21:57:42.294198+0000","last_clean":"2021-03-24T21:57:42.294198+0000","last_became_active":"2021-03-24T21:55:10.102360+0000","last_became_peered":"2021-03-24T21:55:10.102360+0000","last_unstale":"2021-03-24T21:57:42.294198+0000","last_undegraded":"2021-03-24T21:57:42.294198+0000","last_fullsized":"2021-03-24T21:57:42.294198+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:12.514094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:12.514094+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"248","reported_epoch":"290","state":"active+clean","last_fresh":"2021-03-24T21:57:40.156751+0000","last_change":"2021-03-24T21:56:43.052062+0000","last_active":"2021-03-24T21:57:40.156751+0000","last_peered":"2021-03-24T21:57:40.156751+0000","last_clean":"2021-03-24T21:57:40.156751+0000","last_became_active":"2021-03-24T21:51:49.411921+0000","last_became_peered":"2021-03-24T21:51:49.411921+0000","last_unstale":"2021-03-24T21:57:40.156751+0000","last_undegraded":"2021-03-24T21:57:40.156751+0000","last_fullsized":"2021-03-24T21:57:40.156751+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:43.051885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:43.051885+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"251","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.275772+0000","last_change":"2021-03-24T21:57:02.189636+0000","last_active":"2021-03-24T21:57:42.275772+0000","last_peered":"2021-03-24T21:57:42.275772+0000","last_clean":"2021-03-24T21:57:42.275772+0000","last_became_active":"2021-03-24T21:57:02.188645+0000","last_became_peered":"2021-03-24T21:57:02.188645+0000","last_unstale":"2021-03-24T21:57:42.275772+0000","last_undegraded":"2021-03-24T21:57:42.275772+0000","last_fullsized":"2021-03-24T21:57:42.275772+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:33.246205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:35.636299+0000","last_clean_scrub_stamp":"2021-03-24T21:55:33.246205+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"285'122","reported_seq":"730","reported_epoch":"291","state":"active+clean","last_fresh":"2021-03-24T21:57:41.274556+0000","last_change":"2021-03-24T21:57:41.275050+0000","last_active":"2021-03-24T21:57:41.274556+0000","last_peered":"2021-03-24T21:57:41.274556+0000","last_clean":"2021-03-24T21:57:41.274556+0000","last_became_active":"2021-03-24T21:56:08.148045+0000","last_became_peered":"2021-03-24T21:56:08.148045+0000","last_unstale":"2021-03-24T21:57:41.274556+0000","last_undegraded":"2021-03-24T21:57:41.274556+0000","last_fullsized":"2021-03-24T21:57:41.274556+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'96","last_scrub_stamp":"2021-03-24T21:56:40.081137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:56:40.081137+0000","log_size":122,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3949678,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":11,"num_write_kb":3276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"b0","length":"2"},{"start":"b3","length":"2"}]},{"pgid":"4.a","version":"69'23","reported_seq":"281","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.283251+0000","last_change":"2021-03-24T21:56:34.992132+0000","last_active":"2021-03-24T21:57:42.283251+0000","last_peered":"2021-03-24T21:57:42.283251+0000","last_clean":"2021-03-24T21:57:42.283251+0000","last_became_active":"2021-03-24T21:54:38.138701+0000","last_became_peered":"2021-03-24T21:54:38.138701+0000","last_unstale":"2021-03-24T21:57:42.283251+0000","last_undegraded":"2021-03-24T21:57:42.283251+0000","last_fullsized":"2021-03-24T21:57:42.283251+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'23","last_scrub_stamp":"2021-03-24T21:56:34.991954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:34.991954+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":43,"num_read_kb":24,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":732,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":"258","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.286073+0000","last_change":"2021-03-24T21:55:38.974245+0000","last_active":"2021-03-24T21:57:42.286073+0000","last_peered":"2021-03-24T21:57:42.286073+0000","last_clean":"2021-03-24T21:57:42.286073+0000","last_became_active":"2021-03-24T21:54:24.189299+0000","last_became_peered":"2021-03-24T21:54:24.189299+0000","last_unstale":"2021-03-24T21:57:42.286073+0000","last_undegraded":"2021-03-24T21:57:42.286073+0000","last_fullsized":"2021-03-24T21:57:42.286073+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:38.974155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:38.974155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"52","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.276582+0000","last_change":"2021-03-24T21:57:02.185635+0000","last_active":"2021-03-24T21:57:42.276582+0000","last_peered":"2021-03-24T21:57:42.276582+0000","last_clean":"2021-03-24T21:57:42.276582+0000","last_became_active":"2021-03-24T21:57:02.185281+0000","last_became_peered":"2021-03-24T21:57:02.185281+0000","last_unstale":"2021-03-24T21:57:42.276582+0000","last_undegraded":"2021-03-24T21:57:42.276582+0000","last_fullsized":"2021-03-24T21:57:42.276582+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:44.018043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:44.018043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"248","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.282256+0000","last_change":"2021-03-24T21:56:38.970708+0000","last_active":"2021-03-24T21:57:42.282256+0000","last_peered":"2021-03-24T21:57:42.282256+0000","last_clean":"2021-03-24T21:57:42.282256+0000","last_became_active":"2021-03-24T21:51:51.762513+0000","last_became_peered":"2021-03-24T21:51:51.762513+0000","last_unstale":"2021-03-24T21:57:42.282256+0000","last_undegraded":"2021-03-24T21:57:42.282256+0000","last_fullsized":"2021-03-24T21:57:42.282256+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:38.970539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:38.970539+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.9","version":"272'211","reported_seq":"865","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.276875+0000","last_change":"2021-03-24T21:57:42.276875+0000","last_active":"2021-03-24T21:57:42.276875+0000","last_peered":"2021-03-24T21:57:42.276875+0000","last_clean":"2021-03-24T21:57:42.276875+0000","last_became_active":"2021-03-24T21:56:08.158313+0000","last_became_peered":"2021-03-24T21:56:08.158313+0000","last_unstale":"2021-03-24T21:57:42.276875+0000","last_undegraded":"2021-03-24T21:57:42.276875+0000","last_fullsized":"2021-03-24T21:57:42.276875+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'211","last_scrub_stamp":"2021-03-24T21:57:25.266209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:57:25.266209+0000","log_size":211,"ondisk_log_size":211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2521050,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"b0","length":"2"},{"start":"b3","length":"2"},{"start":"b7","length":"1"}]},{"pgid":"5.8","version":"0'0","reported_seq":"257","reported_epoch":"290","state":"active+clean","last_fresh":"2021-03-24T21:57:40.158044+0000","last_change":"2021-03-24T21:56:51.990882+0000","last_active":"2021-03-24T21:57:40.158044+0000","last_peered":"2021-03-24T21:57:40.158044+0000","last_clean":"2021-03-24T21:57:40.158044+0000","last_became_active":"2021-03-24T21:54:38.124317+0000","last_became_peered":"2021-03-24T21:54:38.124317+0000","last_unstale":"2021-03-24T21:57:40.158044+0000","last_undegraded":"2021-03-24T21:57:40.158044+0000","last_fullsized":"2021-03-24T21:57:40.158044+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:51.990762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:51.990762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"69'15","reported_seq":"302","reported_epoch":"291","state":"active+clean","last_fresh":"2021-03-24T21:57:41.273076+0000","last_change":"2021-03-24T21:56:02.021432+0000","last_active":"2021-03-24T21:57:41.273076+0000","last_peered":"2021-03-24T21:57:41.273076+0000","last_clean":"2021-03-24T21:57:41.273076+0000","last_became_active":"2021-03-24T21:54:38.139863+0000","last_became_peered":"2021-03-24T21:54:38.139863+0000","last_unstale":"2021-03-24T21:57:41.273076+0000","last_undegraded":"2021-03-24T21:57:41.273076+0000","last_fullsized":"2021-03-24T21:57:41.273076+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:56:02.021288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:02.021288+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":19,"num_write":17,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":692,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":"263","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.285099+0000","last_change":"2021-03-24T21:56:06.043071+0000","last_active":"2021-03-24T21:57:42.285099+0000","last_peered":"2021-03-24T21:57:42.285099+0000","last_clean":"2021-03-24T21:57:42.285099+0000","last_became_active":"2021-03-24T21:54:38.109439+0000","last_became_peered":"2021-03-24T21:54:38.109439+0000","last_unstale":"2021-03-24T21:57:42.285099+0000","last_undegraded":"2021-03-24T21:57:42.285099+0000","last_fullsized":"2021-03-24T21:57:42.285099+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:06.042865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:06.042865+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"260","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.292903+0000","last_change":"2021-03-24T21:56:44.228591+0000","last_active":"2021-03-24T21:57:42.292903+0000","last_peered":"2021-03-24T21:57:42.292903+0000","last_clean":"2021-03-24T21:57:42.292903+0000","last_became_active":"2021-03-24T21:56:44.228142+0000","last_became_peered":"2021-03-24T21:56:44.228142+0000","last_unstale":"2021-03-24T21:57:42.292903+0000","last_undegraded":"2021-03-24T21:57:42.292903+0000","last_fullsized":"2021-03-24T21:57:42.292903+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:24.548819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:22.724840+0000","last_clean_scrub_stamp":"2021-03-24T21:55:24.548819+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"272'122","reported_seq":"733","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.290634+0000","last_change":"2021-03-24T21:57:42.290634+0000","last_active":"2021-03-24T21:57:42.290634+0000","last_peered":"2021-03-24T21:57:42.290634+0000","last_clean":"2021-03-24T21:57:42.290634+0000","last_became_active":"2021-03-24T21:57:26.169639+0000","last_became_peered":"2021-03-24T21:57:26.169639+0000","last_unstale":"2021-03-24T21:57:42.290634+0000","last_undegraded":"2021-03-24T21:57:42.290634+0000","last_fullsized":"2021-03-24T21:57:42.290634+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":122,"ondisk_log_size":122,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1983226,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":23,"num_read_kb":22367,"num_write":28,"num_write_kb":8975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"b0","length":"2"},{"start":"b3","length":"2"},{"start":"b7","length":"1"}]},{"pgid":"5.9","version":"109'4","reported_seq":"349","reported_epoch":"293","state":"active+clean","last_fresh":"2021-03-24T21:57:43.296085+0000","last_change":"2021-03-24T21:56:32.051794+0000","last_active":"2021-03-24T21:57:43.296085+0000","last_peered":"2021-03-24T21:57:43.296085+0000","last_clean":"2021-03-24T21:57:43.296085+0000","last_became_active":"2021-03-24T21:54:38.125497+0000","last_became_peered":"2021-03-24T21:54:38.125497+0000","last_unstale":"2021-03-24T21:57:43.296085+0000","last_undegraded":"2021-03-24T21:57:43.296085+0000","last_fullsized":"2021-03-24T21:57:43.296085+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'4","last_scrub_stamp":"2021-03-24T21:56:32.051533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:32.051533+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"97'28","reported_seq":"325","reported_epoch":"291","state":"active+clean","last_fresh":"2021-03-24T21:57:41.282246+0000","last_change":"2021-03-24T21:56:05.210757+0000","last_active":"2021-03-24T21:57:41.282246+0000","last_peered":"2021-03-24T21:57:41.282246+0000","last_clean":"2021-03-24T21:57:41.282246+0000","last_became_active":"2021-03-24T21:54:24.190817+0000","last_became_peered":"2021-03-24T21:54:24.190817+0000","last_unstale":"2021-03-24T21:57:41.282246+0000","last_undegraded":"2021-03-24T21:57:41.282246+0000","last_fullsized":"2021-03-24T21:57:41.282246+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'28","last_scrub_stamp":"2021-03-24T21:56:05.210553+0000","last_deep_scrub":"97'28","last_deep_scrub_stamp":"2021-03-24T21:54:19.157265+0000","last_clean_scrub_stamp":"2021-03-24T21:56:05.210553+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":20,"num_write":19,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"89'4","reported_seq":"254","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.284739+0000","last_change":"2021-03-24T21:56:03.060260+0000","last_active":"2021-03-24T21:57:42.284739+0000","last_peered":"2021-03-24T21:57:42.284739+0000","last_clean":"2021-03-24T21:57:42.284739+0000","last_became_active":"2021-03-24T21:51:45.385110+0000","last_became_peered":"2021-03-24T21:51:45.385110+0000","last_unstale":"2021-03-24T21:57:42.284739+0000","last_undegraded":"2021-03-24T21:57:42.284739+0000","last_fullsized":"2021-03-24T21:57:42.284739+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'4","last_scrub_stamp":"2021-03-24T21:56:03.060069+0000","last_deep_scrub":"89'4","last_deep_scrub_stamp":"2021-03-24T21:53:19.926637+0000","last_clean_scrub_stamp":"2021-03-24T21:56:03.060069+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"226","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.276842+0000","last_change":"2021-03-24T21:57:02.187469+0000","last_active":"2021-03-24T21:57:42.276842+0000","last_peered":"2021-03-24T21:57:42.276842+0000","last_clean":"2021-03-24T21:57:42.276842+0000","last_became_active":"2021-03-24T21:57:02.187060+0000","last_became_peered":"2021-03-24T21:57:02.187060+0000","last_unstale":"2021-03-24T21:57:42.276842+0000","last_undegraded":"2021-03-24T21:57:42.276842+0000","last_fullsized":"2021-03-24T21:57:42.276842+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:54.262681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:54.262681+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"272'275","reported_seq":"1047","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.287714+0000","last_change":"2021-03-24T21:57:42.287997+0000","last_active":"2021-03-24T21:57:42.287714+0000","last_peered":"2021-03-24T21:57:42.287714+0000","last_clean":"2021-03-24T21:57:42.287714+0000","last_became_active":"2021-03-24T21:57:26.170367+0000","last_became_peered":"2021-03-24T21:57:26.170367+0000","last_unstale":"2021-03-24T21:57:42.287714+0000","last_undegraded":"2021-03-24T21:57:42.287714+0000","last_fullsized":"2021-03-24T21:57:42.287714+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"191'150","last_scrub_stamp":"2021-03-24T21:56:01.308050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:56:01.308050+0000","log_size":275,"ondisk_log_size":275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8857338,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":118,"num_read_kb":138390,"num_write":52,"num_write_kb":22259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":12587455,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"b0","length":"2"},{"start":"b3","length":"2"},{"start":"b7","length":"1"}]},{"pgid":"4.17","version":"69'7","reported_seq":"276","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.290904+0000","last_change":"2021-03-24T21:55:56.490518+0000","last_active":"2021-03-24T21:57:42.290904+0000","last_peered":"2021-03-24T21:57:42.290904+0000","last_clean":"2021-03-24T21:57:42.290904+0000","last_became_active":"2021-03-24T21:54:38.144098+0000","last_became_peered":"2021-03-24T21:54:38.144098+0000","last_unstale":"2021-03-24T21:57:42.290904+0000","last_undegraded":"2021-03-24T21:57:42.290904+0000","last_fullsized":"2021-03-24T21:57:42.290904+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'7","last_scrub_stamp":"2021-03-24T21:55:56.490439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:56.490439+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":7,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":158,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":"212","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.275719+0000","last_change":"2021-03-24T21:57:16.270995+0000","last_active":"2021-03-24T21:57:42.275719+0000","last_peered":"2021-03-24T21:57:42.275719+0000","last_clean":"2021-03-24T21:57:42.275719+0000","last_became_active":"2021-03-24T21:54:24.201388+0000","last_became_peered":"2021-03-24T21:54:24.201388+0000","last_unstale":"2021-03-24T21:57:42.275719+0000","last_undegraded":"2021-03-24T21:57:42.275719+0000","last_fullsized":"2021-03-24T21:57:42.275719+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:16.270874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:16.270874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"260","reported_epoch":"291","state":"active+clean","last_fresh":"2021-03-24T21:57:41.283877+0000","last_change":"2021-03-24T21:56:48.144572+0000","last_active":"2021-03-24T21:57:41.283877+0000","last_peered":"2021-03-24T21:57:41.283877+0000","last_clean":"2021-03-24T21:57:41.283877+0000","last_became_active":"2021-03-24T21:54:38.128409+0000","last_became_peered":"2021-03-24T21:54:38.128409+0000","last_unstale":"2021-03-24T21:57:41.283877+0000","last_undegraded":"2021-03-24T21:57:41.283877+0000","last_fullsized":"2021-03-24T21:57:41.283877+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:48.144391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:48.144391+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"252","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.285614+0000","last_change":"2021-03-24T21:56:00.986717+0000","last_active":"2021-03-24T21:57:42.285614+0000","last_peered":"2021-03-24T21:57:42.285614+0000","last_clean":"2021-03-24T21:57:42.285614+0000","last_became_active":"2021-03-24T21:54:24.188386+0000","last_became_peered":"2021-03-24T21:54:24.188386+0000","last_unstale":"2021-03-24T21:57:42.285614+0000","last_undegraded":"2021-03-24T21:57:42.285614+0000","last_fullsized":"2021-03-24T21:57:42.285614+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:00.986593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:00.986593+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.14","version":"288'201","reported_seq":"861","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.295901+0000","last_change":"2021-03-24T21:57:42.295901+0000","last_active":"2021-03-24T21:57:42.295901+0000","last_peered":"2021-03-24T21:57:42.295901+0000","last_clean":"2021-03-24T21:57:42.295901+0000","last_became_active":"2021-03-24T21:56:14.139804+0000","last_became_peered":"2021-03-24T21:56:14.139804+0000","last_unstale":"2021-03-24T21:57:42.295901+0000","last_undegraded":"2021-03-24T21:57:42.295901+0000","last_fullsized":"2021-03-24T21:57:42.295901+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":201,"ondisk_log_size":201,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4664009,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":70864,"num_write":59,"num_write_kb":19290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":6105417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"b0","length":"2"},{"start":"b3","length":"2"},{"start":"b7","length":"1"}]},{"pgid":"4.16","version":"69'16","reported_seq":"281","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.278100+0000","last_change":"2021-03-24T21:56:40.303286+0000","last_active":"2021-03-24T21:57:42.278100+0000","last_peered":"2021-03-24T21:57:42.278100+0000","last_clean":"2021-03-24T21:57:42.278100+0000","last_became_active":"2021-03-24T21:51:47.687595+0000","last_became_peered":"2021-03-24T21:51:47.687595+0000","last_unstale":"2021-03-24T21:57:42.278100+0000","last_undegraded":"2021-03-24T21:57:42.278100+0000","last_fullsized":"2021-03-24T21:57:42.278100+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T21:56:40.303200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:40.303200+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":38,"num_read_kb":18,"num_write":17,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":"260","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.283114+0000","last_change":"2021-03-24T21:57:28.111120+0000","last_active":"2021-03-24T21:57:42.283114+0000","last_peered":"2021-03-24T21:57:42.283114+0000","last_clean":"2021-03-24T21:57:42.283114+0000","last_became_active":"2021-03-24T21:54:24.196290+0000","last_became_peered":"2021-03-24T21:54:24.196290+0000","last_unstale":"2021-03-24T21:57:42.283114+0000","last_undegraded":"2021-03-24T21:57:42.283114+0000","last_fullsized":"2021-03-24T21:57:42.283114+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:31.912451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:31.912451+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"253","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.291084+0000","last_change":"2021-03-24T21:54:55.493891+0000","last_active":"2021-03-24T21:57:42.291084+0000","last_peered":"2021-03-24T21:57:42.291084+0000","last_clean":"2021-03-24T21:57:42.291084+0000","last_became_active":"2021-03-24T21:54:38.129390+0000","last_became_peered":"2021-03-24T21:54:38.129390+0000","last_unstale":"2021-03-24T21:57:42.291084+0000","last_undegraded":"2021-03-24T21:57:42.291084+0000","last_fullsized":"2021-03-24T21:57:42.291084+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:55.493713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:55.493713+0000","last_clean_scrub_stamp":"2021-03-24T21:54:55.493713+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"246","reported_epoch":"290","state":"active+clean","last_fresh":"2021-03-24T21:57:40.157314+0000","last_change":"2021-03-24T21:57:20.073705+0000","last_active":"2021-03-24T21:57:40.157314+0000","last_peered":"2021-03-24T21:57:40.157314+0000","last_clean":"2021-03-24T21:57:40.157314+0000","last_became_active":"2021-03-24T21:51:51.754085+0000","last_became_peered":"2021-03-24T21:51:51.754085+0000","last_unstale":"2021-03-24T21:57:40.157314+0000","last_undegraded":"2021-03-24T21:57:40.157314+0000","last_fullsized":"2021-03-24T21:57:40.157314+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:20.073641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:57:20.073641+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"281'86","reported_seq":"741","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.288062+0000","last_change":"2021-03-24T21:57:42.288062+0000","last_active":"2021-03-24T21:57:42.288062+0000","last_peered":"2021-03-24T21:57:42.288062+0000","last_clean":"2021-03-24T21:57:42.288062+0000","last_became_active":"2021-03-24T21:56:10.139551+0000","last_became_peered":"2021-03-24T21:56:10.139551+0000","last_unstale":"2021-03-24T21:57:42.288062+0000","last_undegraded":"2021-03-24T21:57:42.288062+0000","last_fullsized":"2021-03-24T21:57:42.288062+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":5,"last_scrub":"233'79","last_scrub_stamp":"2021-03-24T21:56:50.934781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:56:50.934781+0000","log_size":86,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3747994,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":1510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"b0","length":"2"},{"start":"b3","length":"2"},{"start":"b7","length":"1"}]},{"pgid":"4.15","version":"69'13","reported_seq":"277","reported_epoch":"291","state":"active+clean","last_fresh":"2021-03-24T21:57:41.284323+0000","last_change":"2021-03-24T21:56:47.168651+0000","last_active":"2021-03-24T21:57:41.284323+0000","last_peered":"2021-03-24T21:57:41.284323+0000","last_clean":"2021-03-24T21:57:41.284323+0000","last_became_active":"2021-03-24T21:51:47.689288+0000","last_became_peered":"2021-03-24T21:51:47.689288+0000","last_unstale":"2021-03-24T21:57:41.284323+0000","last_undegraded":"2021-03-24T21:57:41.284323+0000","last_fullsized":"2021-03-24T21:57:41.284323+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2021-03-24T21:56:47.168558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:47.168558+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":"253","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.276645+0000","last_change":"2021-03-24T21:56:32.279216+0000","last_active":"2021-03-24T21:57:42.276645+0000","last_peered":"2021-03-24T21:57:42.276645+0000","last_clean":"2021-03-24T21:57:42.276645+0000","last_became_active":"2021-03-24T21:51:45.381516+0000","last_became_peered":"2021-03-24T21:51:45.381516+0000","last_unstale":"2021-03-24T21:57:42.276645+0000","last_undegraded":"2021-03-24T21:57:42.276645+0000","last_fullsized":"2021-03-24T21:57:42.276645+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:32.279119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:05.473348+0000","last_clean_scrub_stamp":"2021-03-24T21:56:32.279119+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"235'4","reported_seq":"331","reported_epoch":"293","state":"active+clean","last_fresh":"2021-03-24T21:57:43.291112+0000","last_change":"2021-03-24T21:56:44.227488+0000","last_active":"2021-03-24T21:57:43.291112+0000","last_peered":"2021-03-24T21:57:43.291112+0000","last_clean":"2021-03-24T21:57:43.291112+0000","last_became_active":"2021-03-24T21:56:44.224410+0000","last_became_peered":"2021-03-24T21:56:44.224410+0000","last_unstale":"2021-03-24T21:57:43.291112+0000","last_undegraded":"2021-03-24T21:57:43.291112+0000","last_fullsized":"2021-03-24T21:57:43.291112+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'3","last_scrub_stamp":"2021-03-24T21:55:01.577769+0000","last_deep_scrub":"86'3","last_deep_scrub_stamp":"2021-03-24T21:52:56.678111+0000","last_clean_scrub_stamp":"2021-03-24T21:55:01.577769+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"249","reported_epoch":"291","state":"active+clean","last_fresh":"2021-03-24T21:57:41.276917+0000","last_change":"2021-03-24T21:56:51.925140+0000","last_active":"2021-03-24T21:57:41.276917+0000","last_peered":"2021-03-24T21:57:41.276917+0000","last_clean":"2021-03-24T21:57:41.276917+0000","last_became_active":"2021-03-24T21:51:51.762876+0000","last_became_peered":"2021-03-24T21:51:51.762876+0000","last_unstale":"2021-03-24T21:57:41.276917+0000","last_undegraded":"2021-03-24T21:57:41.276917+0000","last_fullsized":"2021-03-24T21:57:41.276917+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:51.924963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:51.924963+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.16","version":"277'148","reported_seq":"789","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.277420+0000","last_change":"2021-03-24T21:57:42.279087+0000","last_active":"2021-03-24T21:57:42.277420+0000","last_peered":"2021-03-24T21:57:42.277420+0000","last_clean":"2021-03-24T21:57:42.277420+0000","last_became_active":"2021-03-24T21:56:13.127048+0000","last_became_peered":"2021-03-24T21:56:13.127048+0000","last_unstale":"2021-03-24T21:57:42.277420+0000","last_undegraded":"2021-03-24T21:57:42.277420+0000","last_fullsized":"2021-03-24T21:57:42.277420+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'146","last_scrub_stamp":"2021-03-24T21:57:22.669305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:57:22.669305+0000","log_size":148,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2605118,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"b0","length":"2"},{"start":"b3","length":"2"},{"start":"b7","length":"1"}]},{"pgid":"4.14","version":"69'14","reported_seq":"301","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.294439+0000","last_change":"2021-03-24T21:55:25.582566+0000","last_active":"2021-03-24T21:57:42.294439+0000","last_peered":"2021-03-24T21:57:42.294439+0000","last_clean":"2021-03-24T21:57:42.294439+0000","last_became_active":"2021-03-24T21:54:38.143783+0000","last_became_peered":"2021-03-24T21:54:38.143783+0000","last_unstale":"2021-03-24T21:57:42.294439+0000","last_undegraded":"2021-03-24T21:57:42.294439+0000","last_fullsized":"2021-03-24T21:57:42.294439+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'14","last_scrub_stamp":"2021-03-24T21:55:25.582388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:25.582388+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":27,"num_read_kb":14,"num_write":14,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":"249","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.283663+0000","last_change":"2021-03-24T21:55:58.082943+0000","last_active":"2021-03-24T21:57:42.283663+0000","last_peered":"2021-03-24T21:57:42.283663+0000","last_clean":"2021-03-24T21:57:42.283663+0000","last_became_active":"2021-03-24T21:51:45.579271+0000","last_became_peered":"2021-03-24T21:51:45.579271+0000","last_unstale":"2021-03-24T21:57:42.283663+0000","last_undegraded":"2021-03-24T21:57:42.283663+0000","last_fullsized":"2021-03-24T21:57:42.283663+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:58.082776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:58.082776+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"109'4","reported_seq":"337","reported_epoch":"291","state":"active+clean","last_fresh":"2021-03-24T21:57:41.284142+0000","last_change":"2021-03-24T21:56:42.209663+0000","last_active":"2021-03-24T21:57:41.284142+0000","last_peered":"2021-03-24T21:57:41.284142+0000","last_clean":"2021-03-24T21:57:41.284142+0000","last_became_active":"2021-03-24T21:54:38.130026+0000","last_became_peered":"2021-03-24T21:54:38.130026+0000","last_unstale":"2021-03-24T21:57:41.284142+0000","last_undegraded":"2021-03-24T21:57:41.284142+0000","last_fullsized":"2021-03-24T21:57:41.284142+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'4","last_scrub_stamp":"2021-03-24T21:56:42.209497+0000","last_deep_scrub":"86'3","last_deep_scrub_stamp":"2021-03-24T21:53:08.815163+0000","last_clean_scrub_stamp":"2021-03-24T21:56:42.209497+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"242","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.277120+0000","last_change":"2021-03-24T21:54:44.392954+0000","last_active":"2021-03-24T21:57:42.277120+0000","last_peered":"2021-03-24T21:57:42.277120+0000","last_clean":"2021-03-24T21:57:42.277120+0000","last_became_active":"2021-03-24T21:51:51.756584+0000","last_became_peered":"2021-03-24T21:51:51.756584+0000","last_unstale":"2021-03-24T21:57:42.277120+0000","last_undegraded":"2021-03-24T21:57:42.277120+0000","last_fullsized":"2021-03-24T21:57:42.277120+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:44.392808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:54:44.392808+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.17","version":"291'321","reported_seq":"1063","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.285211+0000","last_change":"2021-03-24T21:57:42.285211+0000","last_active":"2021-03-24T21:57:42.285211+0000","last_peered":"2021-03-24T21:57:42.285211+0000","last_clean":"2021-03-24T21:57:42.285211+0000","last_became_active":"2021-03-24T21:56:13.132418+0000","last_became_peered":"2021-03-24T21:56:13.132418+0000","last_unstale":"2021-03-24T21:57:42.285211+0000","last_undegraded":"2021-03-24T21:57:42.285211+0000","last_fullsized":"2021-03-24T21:57:42.285211+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":5,"last_scrub":"265'274","last_scrub_stamp":"2021-03-24T21:57:14.281739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:57:14.281739+0000","log_size":321,"ondisk_log_size":321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8975070,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":25706,"num_write":15,"num_write_kb":7641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"b0","length":"2"},{"start":"b3","length":"2"},{"start":"b7","length":"1"}]},{"pgid":"4.13","version":"69'15","reported_seq":"331","reported_epoch":"291","state":"active+clean","last_fresh":"2021-03-24T21:57:41.276045+0000","last_change":"2021-03-24T21:55:37.076650+0000","last_active":"2021-03-24T21:57:41.276045+0000","last_peered":"2021-03-24T21:57:41.276045+0000","last_clean":"2021-03-24T21:57:41.276045+0000","last_became_active":"2021-03-24T21:54:38.118330+0000","last_became_peered":"2021-03-24T21:54:38.118330+0000","last_unstale":"2021-03-24T21:57:41.276045+0000","last_undegraded":"2021-03-24T21:57:41.276045+0000","last_fullsized":"2021-03-24T21:57:41.276045+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:55:37.076511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:37.076511+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":"257","reported_epoch":"291","state":"active+clean","last_fresh":"2021-03-24T21:57:41.274882+0000","last_change":"2021-03-24T21:55:42.135567+0000","last_active":"2021-03-24T21:57:41.274882+0000","last_peered":"2021-03-24T21:57:41.274882+0000","last_clean":"2021-03-24T21:57:41.274882+0000","last_became_active":"2021-03-24T21:54:24.181343+0000","last_became_peered":"2021-03-24T21:54:24.181343+0000","last_unstale":"2021-03-24T21:57:41.274882+0000","last_undegraded":"2021-03-24T21:57:41.274882+0000","last_fullsized":"2021-03-24T21:57:41.274882+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:42.135502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:42.135502+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"214","reported_epoch":"291","state":"active+clean","last_fresh":"2021-03-24T21:57:41.283006+0000","last_change":"2021-03-24T21:57:31.118235+0000","last_active":"2021-03-24T21:57:41.283006+0000","last_peered":"2021-03-24T21:57:41.283006+0000","last_clean":"2021-03-24T21:57:41.283006+0000","last_became_active":"2021-03-24T21:56:35.139207+0000","last_became_peered":"2021-03-24T21:56:35.139207+0000","last_unstale":"2021-03-24T21:57:41.283006+0000","last_undegraded":"2021-03-24T21:57:41.283006+0000","last_fullsized":"2021-03-24T21:57:41.283006+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:31.118050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:57:31.118050+0000","last_clean_scrub_stamp":"2021-03-24T21:57:31.118050+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"242","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.288461+0000","last_change":"2021-03-24T21:55:22.555707+0000","last_active":"2021-03-24T21:57:42.288461+0000","last_peered":"2021-03-24T21:57:42.288461+0000","last_clean":"2021-03-24T21:57:42.288461+0000","last_became_active":"2021-03-24T21:51:51.764871+0000","last_became_peered":"2021-03-24T21:51:51.764871+0000","last_unstale":"2021-03-24T21:57:42.288461+0000","last_undegraded":"2021-03-24T21:57:42.288461+0000","last_fullsized":"2021-03-24T21:57:42.288461+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:22.555624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:23.707914+0000","last_clean_scrub_stamp":"2021-03-24T21:55:22.555624+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.10","version":"288'29","reported_seq":"159","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.286800+0000","last_change":"2021-03-24T21:57:42.287283+0000","last_active":"2021-03-24T21:57:42.286800+0000","last_peered":"2021-03-24T21:57:42.286800+0000","last_clean":"2021-03-24T21:57:42.286800+0000","last_became_active":"2021-03-24T21:57:06.154293+0000","last_became_peered":"2021-03-24T21:57:06.154293+0000","last_unstale":"2021-03-24T21:57:42.286800+0000","last_undegraded":"2021-03-24T21:57:42.286800+0000","last_fullsized":"2021-03-24T21:57:42.286800+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":5,"last_scrub":"277'24","last_scrub_stamp":"2021-03-24T21:57:28.119420+0000","last_deep_scrub":"277'24","last_deep_scrub_stamp":"2021-03-24T21:57:28.119420+0000","last_clean_scrub_stamp":"2021-03-24T21:57:28.119420+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":2,"num_write_kb":3332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"b0","length":"2"},{"start":"b3","length":"2"},{"start":"b7","length":"1"}]},{"pgid":"4.12","version":"69'15","reported_seq":"260","reported_epoch":"290","state":"active+clean","last_fresh":"2021-03-24T21:57:40.157981+0000","last_change":"2021-03-24T21:56:09.119663+0000","last_active":"2021-03-24T21:57:40.157981+0000","last_peered":"2021-03-24T21:57:40.157981+0000","last_clean":"2021-03-24T21:57:40.157981+0000","last_became_active":"2021-03-24T21:54:38.122707+0000","last_became_peered":"2021-03-24T21:54:38.122707+0000","last_unstale":"2021-03-24T21:57:40.157981+0000","last_undegraded":"2021-03-24T21:57:40.157981+0000","last_fullsized":"2021-03-24T21:57:40.157981+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:56:09.119562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:09.119562+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":34,"num_read_kb":16,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":"249","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.285702+0000","last_change":"2021-03-24T21:55:57.107894+0000","last_active":"2021-03-24T21:57:42.285702+0000","last_peered":"2021-03-24T21:57:42.285702+0000","last_clean":"2021-03-24T21:57:42.285702+0000","last_became_active":"2021-03-24T21:51:45.379383+0000","last_became_peered":"2021-03-24T21:51:45.379383+0000","last_unstale":"2021-03-24T21:57:42.285702+0000","last_undegraded":"2021-03-24T21:57:42.285702+0000","last_fullsized":"2021-03-24T21:57:42.285702+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:57.107720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:57.107720+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"253","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.292769+0000","last_change":"2021-03-24T21:56:32.480276+0000","last_active":"2021-03-24T21:57:42.292769+0000","last_peered":"2021-03-24T21:57:42.292769+0000","last_clean":"2021-03-24T21:57:42.292769+0000","last_became_active":"2021-03-24T21:54:38.128067+0000","last_became_peered":"2021-03-24T21:54:38.128067+0000","last_unstale":"2021-03-24T21:57:42.292769+0000","last_undegraded":"2021-03-24T21:57:42.292769+0000","last_fullsized":"2021-03-24T21:57:42.292769+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:32.480115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:32.480115+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"256","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.275472+0000","last_change":"2021-03-24T21:56:52.226305+0000","last_active":"2021-03-24T21:57:42.275472+0000","last_peered":"2021-03-24T21:57:42.275472+0000","last_clean":"2021-03-24T21:57:42.275472+0000","last_became_active":"2021-03-24T21:54:38.126662+0000","last_became_peered":"2021-03-24T21:54:38.126662+0000","last_unstale":"2021-03-24T21:57:42.275472+0000","last_undegraded":"2021-03-24T21:57:42.275472+0000","last_fullsized":"2021-03-24T21:57:42.275472+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:52.226117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:52.226117+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.11","version":"292'314","reported_seq":"1001","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.290224+0000","last_change":"2021-03-24T21:57:42.290224+0000","last_active":"2021-03-24T21:57:42.290224+0000","last_peered":"2021-03-24T21:57:42.290224+0000","last_clean":"2021-03-24T21:57:42.290224+0000","last_became_active":"2021-03-24T21:56:14.138730+0000","last_became_peered":"2021-03-24T21:56:14.138730+0000","last_unstale":"2021-03-24T21:57:42.290224+0000","last_undegraded":"2021-03-24T21:57:42.290224+0000","last_fullsized":"2021-03-24T21:57:42.290224+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'297","last_scrub_stamp":"2021-03-24T21:57:24.209124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:57:24.209124+0000","log_size":314,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9008651,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":17259,"num_write":9,"num_write_kb":2419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"b0","length":"2"},{"start":"b3","length":"2"},{"start":"b7","length":"1"}]},{"pgid":"4.11","version":"86'22","reported_seq":"306","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.289163+0000","last_change":"2021-03-24T21:54:51.585204+0000","last_active":"2021-03-24T21:57:42.289163+0000","last_peered":"2021-03-24T21:57:42.289163+0000","last_clean":"2021-03-24T21:57:42.289163+0000","last_became_active":"2021-03-24T21:54:24.188790+0000","last_became_peered":"2021-03-24T21:54:24.188790+0000","last_unstale":"2021-03-24T21:57:42.289163+0000","last_undegraded":"2021-03-24T21:57:42.289163+0000","last_fullsized":"2021-03-24T21:57:42.289163+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'22","last_scrub_stamp":"2021-03-24T21:54:51.585058+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2021-03-24T21:52:54.597505+0000","last_clean_scrub_stamp":"2021-03-24T21:54:51.585058+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":16,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":580,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":"264","reported_epoch":"291","state":"active+clean","last_fresh":"2021-03-24T21:57:41.283891+0000","last_change":"2021-03-24T21:57:13.016235+0000","last_active":"2021-03-24T21:57:41.283891+0000","last_peered":"2021-03-24T21:57:41.283891+0000","last_clean":"2021-03-24T21:57:41.283891+0000","last_became_active":"2021-03-24T21:54:38.121323+0000","last_became_peered":"2021-03-24T21:54:38.121323+0000","last_unstale":"2021-03-24T21:57:41.283891+0000","last_undegraded":"2021-03-24T21:57:41.283891+0000","last_fullsized":"2021-03-24T21:57:41.283891+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:13.016117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:13.016117+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"254","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.282845+0000","last_change":"2021-03-24T21:55:23.872913+0000","last_active":"2021-03-24T21:57:42.282845+0000","last_peered":"2021-03-24T21:57:42.282845+0000","last_clean":"2021-03-24T21:57:42.282845+0000","last_became_active":"2021-03-24T21:54:24.194423+0000","last_became_peered":"2021-03-24T21:54:24.194423+0000","last_unstale":"2021-03-24T21:57:42.282845+0000","last_undegraded":"2021-03-24T21:57:42.282845+0000","last_fullsized":"2021-03-24T21:57:42.282845+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:23.872742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:55:23.872742+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"251","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.293090+0000","last_change":"2021-03-24T21:56:40.458139+0000","last_active":"2021-03-24T21:57:42.293090+0000","last_peered":"2021-03-24T21:57:42.293090+0000","last_clean":"2021-03-24T21:57:42.293090+0000","last_became_active":"2021-03-24T21:54:24.202686+0000","last_became_peered":"2021-03-24T21:54:24.202686+0000","last_unstale":"2021-03-24T21:57:42.293090+0000","last_undegraded":"2021-03-24T21:57:42.293090+0000","last_fullsized":"2021-03-24T21:57:42.293090+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:40.458011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:47.540793+0000","last_clean_scrub_stamp":"2021-03-24T21:56:40.458011+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.12","version":"183'26","reported_seq":"300","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.277418+0000","last_change":"2021-03-24T21:57:42.277418+0000","last_active":"2021-03-24T21:57:42.277418+0000","last_peered":"2021-03-24T21:57:42.277418+0000","last_clean":"2021-03-24T21:57:42.277418+0000","last_became_active":"2021-03-24T21:56:10.130886+0000","last_became_peered":"2021-03-24T21:56:10.130886+0000","last_unstale":"2021-03-24T21:57:42.277418+0000","last_undegraded":"2021-03-24T21:57:42.277418+0000","last_fullsized":"2021-03-24T21:57:42.277418+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1352964,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":6,"num_write_kb":2389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":433862,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"b0","length":"2"},{"start":"b3","length":"2"},{"start":"b7","length":"1"}]},{"pgid":"4.10","version":"69'12","reported_seq":"368","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.288112+0000","last_change":"2021-03-24T21:56:53.443666+0000","last_active":"2021-03-24T21:57:42.288112+0000","last_peered":"2021-03-24T21:57:42.288112+0000","last_clean":"2021-03-24T21:57:42.288112+0000","last_became_active":"2021-03-24T21:54:38.134132+0000","last_became_peered":"2021-03-24T21:54:38.134132+0000","last_unstale":"2021-03-24T21:57:42.288112+0000","last_undegraded":"2021-03-24T21:57:42.288112+0000","last_fullsized":"2021-03-24T21:57:42.288112+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2021-03-24T21:56:53.443554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:53.443554+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":12,"num_write":12,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":3756,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":"253","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.278053+0000","last_change":"2021-03-24T21:57:11.230052+0000","last_active":"2021-03-24T21:57:42.278053+0000","last_peered":"2021-03-24T21:57:42.278053+0000","last_clean":"2021-03-24T21:57:42.278053+0000","last_became_active":"2021-03-24T21:51:45.379789+0000","last_became_peered":"2021-03-24T21:51:45.379789+0000","last_unstale":"2021-03-24T21:57:42.278053+0000","last_undegraded":"2021-03-24T21:57:42.278053+0000","last_fullsized":"2021-03-24T21:57:42.278053+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:11.229895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:11.229895+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"206","reported_epoch":"291","state":"active+clean","last_fresh":"2021-03-24T21:57:41.272481+0000","last_change":"2021-03-24T21:55:26.087761+0000","last_active":"2021-03-24T21:57:41.272481+0000","last_peered":"2021-03-24T21:57:41.272481+0000","last_clean":"2021-03-24T21:57:41.272481+0000","last_became_active":"2021-03-24T21:54:24.187568+0000","last_became_peered":"2021-03-24T21:54:24.187568+0000","last_unstale":"2021-03-24T21:57:41.272481+0000","last_undegraded":"2021-03-24T21:57:41.272481+0000","last_fullsized":"2021-03-24T21:57:41.272481+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:26.087584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:55:26.087584+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"243","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.283369+0000","last_change":"2021-03-24T21:56:04.042857+0000","last_active":"2021-03-24T21:57:42.283369+0000","last_peered":"2021-03-24T21:57:42.283369+0000","last_clean":"2021-03-24T21:57:42.283369+0000","last_became_active":"2021-03-24T21:51:51.762150+0000","last_became_peered":"2021-03-24T21:51:51.762150+0000","last_unstale":"2021-03-24T21:57:42.283369+0000","last_undegraded":"2021-03-24T21:57:42.283369+0000","last_fullsized":"2021-03-24T21:57:42.283369+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:04.042709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:04.042709+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.13","version":"292'239","reported_seq":"917","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.282377+0000","last_change":"2021-03-24T21:57:42.278798+0000","last_active":"2021-03-24T21:57:42.282377+0000","last_peered":"2021-03-24T21:57:42.282377+0000","last_clean":"2021-03-24T21:57:42.282377+0000","last_became_active":"2021-03-24T21:57:06.153030+0000","last_became_peered":"2021-03-24T21:57:06.153030+0000","last_unstale":"2021-03-24T21:57:42.282377+0000","last_undegraded":"2021-03-24T21:57:42.282377+0000","last_fullsized":"2021-03-24T21:57:42.282377+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":239,"ondisk_log_size":239,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5411854,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":102,"num_read_kb":100579,"num_write":71,"num_write_kb":30484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":4773188,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"b0","length":"2"},{"start":"b3","length":"2"},{"start":"b7","length":"1"}]},{"pgid":"4.1f","version":"69'17","reported_seq":"281","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.277265+0000","last_change":"2021-03-24T21:56:45.177087+0000","last_active":"2021-03-24T21:57:42.277265+0000","last_peered":"2021-03-24T21:57:42.277265+0000","last_clean":"2021-03-24T21:57:42.277265+0000","last_became_active":"2021-03-24T21:56:44.229823+0000","last_became_peered":"2021-03-24T21:56:44.229823+0000","last_unstale":"2021-03-24T21:57:42.277265+0000","last_undegraded":"2021-03-24T21:57:42.277265+0000","last_fullsized":"2021-03-24T21:57:42.277265+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'17","last_scrub_stamp":"2021-03-24T21:55:32.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:32.214502+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":37,"num_read_kb":18,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":2100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":"257","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.287986+0000","last_change":"2021-03-24T21:55:32.566923+0000","last_active":"2021-03-24T21:57:42.287986+0000","last_peered":"2021-03-24T21:57:42.287986+0000","last_clean":"2021-03-24T21:57:42.287986+0000","last_became_active":"2021-03-24T21:54:38.145870+0000","last_became_peered":"2021-03-24T21:54:38.145870+0000","last_unstale":"2021-03-24T21:57:42.287986+0000","last_undegraded":"2021-03-24T21:57:42.287986+0000","last_fullsized":"2021-03-24T21:57:42.287986+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:32.566747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:32.566747+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"253","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.278309+0000","last_change":"2021-03-24T21:56:35.141430+0000","last_active":"2021-03-24T21:57:42.278309+0000","last_peered":"2021-03-24T21:57:42.278309+0000","last_clean":"2021-03-24T21:57:42.278309+0000","last_became_active":"2021-03-24T21:56:35.140997+0000","last_became_peered":"2021-03-24T21:56:35.140997+0000","last_unstale":"2021-03-24T21:57:42.278309+0000","last_undegraded":"2021-03-24T21:57:42.278309+0000","last_fullsized":"2021-03-24T21:57:42.278309+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:07.255496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:07.255496+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"205","reported_epoch":"291","state":"active+clean","last_fresh":"2021-03-24T21:57:41.283849+0000","last_change":"2021-03-24T21:56:07.272826+0000","last_active":"2021-03-24T21:57:41.283849+0000","last_peered":"2021-03-24T21:57:41.283849+0000","last_clean":"2021-03-24T21:57:41.283849+0000","last_became_active":"2021-03-24T21:54:38.113849+0000","last_became_peered":"2021-03-24T21:54:38.113849+0000","last_unstale":"2021-03-24T21:57:41.283849+0000","last_undegraded":"2021-03-24T21:57:41.283849+0000","last_fullsized":"2021-03-24T21:57:41.283849+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:07.272644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:07.272644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1c","version":"273'245","reported_seq":"898","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.296270+0000","last_change":"2021-03-24T21:57:42.296270+0000","last_active":"2021-03-24T21:57:42.296270+0000","last_peered":"2021-03-24T21:57:42.296270+0000","last_clean":"2021-03-24T21:57:42.296270+0000","last_became_active":"2021-03-24T21:57:31.166900+0000","last_became_peered":"2021-03-24T21:57:31.166900+0000","last_unstale":"2021-03-24T21:57:42.296270+0000","last_undegraded":"2021-03-24T21:57:42.296270+0000","last_fullsized":"2021-03-24T21:57:42.296270+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":245,"ondisk_log_size":245,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8581786,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":70,"num_read_kb":67597,"num_write":52,"num_write_kb":18055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"b0","length":"2"},{"start":"b3","length":"2"},{"start":"b7","length":"1"}]},{"pgid":"4.1e","version":"89'27","reported_seq":"328","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.277628+0000","last_change":"2021-03-24T21:56:45.178898+0000","last_active":"2021-03-24T21:57:42.277628+0000","last_peered":"2021-03-24T21:57:42.277628+0000","last_clean":"2021-03-24T21:57:42.277628+0000","last_became_active":"2021-03-24T21:56:44.230130+0000","last_became_peered":"2021-03-24T21:56:44.230130+0000","last_unstale":"2021-03-24T21:57:42.277628+0000","last_undegraded":"2021-03-24T21:57:42.277628+0000","last_fullsized":"2021-03-24T21:57:42.277628+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'27","last_scrub_stamp":"2021-03-24T21:55:41.337130+0000","last_deep_scrub":"89'27","last_deep_scrub_stamp":"2021-03-24T21:55:41.337130+0000","last_clean_scrub_stamp":"2021-03-24T21:55:41.337130+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":934,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":"222","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.289173+0000","last_change":"2021-03-24T21:57:26.417745+0000","last_active":"2021-03-24T21:57:42.289173+0000","last_peered":"2021-03-24T21:57:42.289173+0000","last_clean":"2021-03-24T21:57:42.289173+0000","last_became_active":"2021-03-24T21:57:02.181985+0000","last_became_peered":"2021-03-24T21:57:02.181985+0000","last_unstale":"2021-03-24T21:57:42.289173+0000","last_undegraded":"2021-03-24T21:57:42.289173+0000","last_fullsized":"2021-03-24T21:57:42.289173+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:26.417613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:57:26.417613+0000","last_clean_scrub_stamp":"2021-03-24T21:57:26.417613+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"235'4","reported_seq":"272","reported_epoch":"291","state":"active+clean","last_fresh":"2021-03-24T21:57:41.276756+0000","last_change":"2021-03-24T21:56:44.318340+0000","last_active":"2021-03-24T21:57:41.276756+0000","last_peered":"2021-03-24T21:57:41.276756+0000","last_clean":"2021-03-24T21:57:41.276756+0000","last_became_active":"2021-03-24T21:56:44.228196+0000","last_became_peered":"2021-03-24T21:56:44.228196+0000","last_unstale":"2021-03-24T21:57:41.276756+0000","last_undegraded":"2021-03-24T21:57:41.276756+0000","last_fullsized":"2021-03-24T21:57:41.276756+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'3","last_scrub_stamp":"2021-03-24T21:56:42.039976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:42.039976+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"248","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.284296+0000","last_change":"2021-03-24T21:57:26.130087+0000","last_active":"2021-03-24T21:57:42.284296+0000","last_peered":"2021-03-24T21:57:42.284296+0000","last_clean":"2021-03-24T21:57:42.284296+0000","last_became_active":"2021-03-24T21:51:51.761492+0000","last_became_peered":"2021-03-24T21:51:51.761492+0000","last_unstale":"2021-03-24T21:57:42.284296+0000","last_undegraded":"2021-03-24T21:57:42.284296+0000","last_fullsized":"2021-03-24T21:57:42.284296+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:26.129924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:57:26.129924+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1d","version":"292'282","reported_seq":"988","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.289053+0000","last_change":"2021-03-24T21:57:42.289053+0000","last_active":"2021-03-24T21:57:42.289053+0000","last_peered":"2021-03-24T21:57:42.289053+0000","last_clean":"2021-03-24T21:57:42.289053+0000","last_became_active":"2021-03-24T21:57:32.177289+0000","last_became_peered":"2021-03-24T21:57:32.177289+0000","last_unstale":"2021-03-24T21:57:42.289053+0000","last_undegraded":"2021-03-24T21:57:42.289053+0000","last_fullsized":"2021-03-24T21:57:42.289053+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"233'169","last_scrub_stamp":"2021-03-24T21:56:47.285648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:56:47.285648+0000","log_size":282,"ondisk_log_size":282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7638864,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":80654,"num_write":69,"num_write_kb":33135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":9406781,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"b0","length":"2"},{"start":"b3","length":"2"},{"start":"b7","length":"1"}]},{"pgid":"4.1d","version":"69'16","reported_seq":"306","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.292346+0000","last_change":"2021-03-24T21:56:10.423931+0000","last_active":"2021-03-24T21:57:42.292346+0000","last_peered":"2021-03-24T21:57:42.292346+0000","last_clean":"2021-03-24T21:57:42.292346+0000","last_became_active":"2021-03-24T21:54:38.128246+0000","last_became_peered":"2021-03-24T21:54:38.128246+0000","last_unstale":"2021-03-24T21:57:42.292346+0000","last_undegraded":"2021-03-24T21:57:42.292346+0000","last_fullsized":"2021-03-24T21:57:42.292346+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T21:56:10.423860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:10.423860+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":37,"num_read_kb":20,"num_write":18,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":844,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":"262","reported_epoch":"291","state":"active+clean","last_fresh":"2021-03-24T21:57:41.275083+0000","last_change":"2021-03-24T21:57:13.701322+0000","last_active":"2021-03-24T21:57:41.275083+0000","last_peered":"2021-03-24T21:57:41.275083+0000","last_clean":"2021-03-24T21:57:41.275083+0000","last_became_active":"2021-03-24T21:54:38.143876+0000","last_became_peered":"2021-03-24T21:54:38.143876+0000","last_unstale":"2021-03-24T21:57:41.275083+0000","last_undegraded":"2021-03-24T21:57:41.275083+0000","last_fullsized":"2021-03-24T21:57:41.275083+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:13.701143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:52:54.212459+0000","last_clean_scrub_stamp":"2021-03-24T21:57:13.701143+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"249","reported_epoch":"291","state":"active+clean","last_fresh":"2021-03-24T21:57:41.275696+0000","last_change":"2021-03-24T21:57:19.767717+0000","last_active":"2021-03-24T21:57:41.275696+0000","last_peered":"2021-03-24T21:57:41.275696+0000","last_clean":"2021-03-24T21:57:41.275696+0000","last_became_active":"2021-03-24T21:51:49.410920+0000","last_became_peered":"2021-03-24T21:51:49.410920+0000","last_unstale":"2021-03-24T21:57:41.275696+0000","last_undegraded":"2021-03-24T21:57:41.275696+0000","last_fullsized":"2021-03-24T21:57:41.275696+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:19.767465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:57:19.767465+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"260","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.290643+0000","last_change":"2021-03-24T21:56:33.440700+0000","last_active":"2021-03-24T21:57:42.290643+0000","last_peered":"2021-03-24T21:57:42.290643+0000","last_clean":"2021-03-24T21:57:42.290643+0000","last_became_active":"2021-03-24T21:54:38.143563+0000","last_became_peered":"2021-03-24T21:54:38.143563+0000","last_unstale":"2021-03-24T21:57:42.290643+0000","last_undegraded":"2021-03-24T21:57:42.290643+0000","last_fullsized":"2021-03-24T21:57:42.290643+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:33.440618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:33.440618+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1e","version":"265'45","reported_seq":"62","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.291922+0000","last_change":"2021-03-24T21:57:42.291922+0000","last_active":"2021-03-24T21:57:42.291922+0000","last_peered":"2021-03-24T21:57:42.291922+0000","last_clean":"2021-03-24T21:57:42.291922+0000","last_became_active":"2021-03-24T21:57:28.182130+0000","last_became_peered":"2021-03-24T21:57:28.182130+0000","last_unstale":"2021-03-24T21:57:42.291922+0000","last_undegraded":"2021-03-24T21:57:42.291922+0000","last_fullsized":"2021-03-24T21:57:42.291922+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":5,"last_scrub":"199'30","last_scrub_stamp":"2021-03-24T21:56:10.105084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:56:10.105084+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":18305,"num_write":13,"num_write_kb":3060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"b0","length":"2"},{"start":"b3","length":"2"},{"start":"b7","length":"1"}]},{"pgid":"4.1c","version":"69'15","reported_seq":"133","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.294311+0000","last_change":"2021-03-24T21:56:35.748629+0000","last_active":"2021-03-24T21:57:42.294311+0000","last_peered":"2021-03-24T21:57:42.294311+0000","last_clean":"2021-03-24T21:57:42.294311+0000","last_became_active":"2021-03-24T21:56:35.142209+0000","last_became_peered":"2021-03-24T21:56:35.142209+0000","last_unstale":"2021-03-24T21:57:42.294311+0000","last_undegraded":"2021-03-24T21:57:42.294311+0000","last_fullsized":"2021-03-24T21:57:42.294311+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:56:06.124139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:06.124139+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":"253","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.276839+0000","last_change":"2021-03-24T21:56:42.254648+0000","last_active":"2021-03-24T21:57:42.276839+0000","last_peered":"2021-03-24T21:57:42.276839+0000","last_clean":"2021-03-24T21:57:42.276839+0000","last_became_active":"2021-03-24T21:51:45.381533+0000","last_became_peered":"2021-03-24T21:51:45.381533+0000","last_unstale":"2021-03-24T21:57:42.276839+0000","last_undegraded":"2021-03-24T21:57:42.276839+0000","last_fullsized":"2021-03-24T21:57:42.276839+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:42.254536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:42.254536+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"205","reported_epoch":"291","state":"active+clean","last_fresh":"2021-03-24T21:57:41.274776+0000","last_change":"2021-03-24T21:57:26.643062+0000","last_active":"2021-03-24T21:57:41.274776+0000","last_peered":"2021-03-24T21:57:41.274776+0000","last_clean":"2021-03-24T21:57:41.274776+0000","last_became_active":"2021-03-24T21:54:38.138620+0000","last_became_peered":"2021-03-24T21:54:38.138620+0000","last_unstale":"2021-03-24T21:57:41.274776+0000","last_undegraded":"2021-03-24T21:57:41.274776+0000","last_fullsized":"2021-03-24T21:57:41.274776+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:26.642883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:57:26.642883+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"251","reported_epoch":"291","state":"active+clean","last_fresh":"2021-03-24T21:57:41.276405+0000","last_change":"2021-03-24T21:56:38.075890+0000","last_active":"2021-03-24T21:57:41.276405+0000","last_peered":"2021-03-24T21:57:41.276405+0000","last_clean":"2021-03-24T21:57:41.276405+0000","last_became_active":"2021-03-24T21:54:24.199444+0000","last_became_peered":"2021-03-24T21:54:24.199444+0000","last_unstale":"2021-03-24T21:57:41.276405+0000","last_undegraded":"2021-03-24T21:57:41.276405+0000","last_fullsized":"2021-03-24T21:57:41.276405+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:38.075714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:38.075714+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1f","version":"272'99","reported_seq":"61","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.277811+0000","last_change":"2021-03-24T21:57:42.278559+0000","last_active":"2021-03-24T21:57:42.277811+0000","last_peered":"2021-03-24T21:57:42.277811+0000","last_clean":"2021-03-24T21:57:42.277811+0000","last_became_active":"2021-03-24T21:57:28.170946+0000","last_became_peered":"2021-03-24T21:57:28.170946+0000","last_unstale":"2021-03-24T21:57:42.277811+0000","last_undegraded":"2021-03-24T21:57:42.277811+0000","last_fullsized":"2021-03-24T21:57:42.277811+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":5,"last_scrub":"218'73","last_scrub_stamp":"2021-03-24T21:56:48.267711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:56:48.267711+0000","log_size":99,"ondisk_log_size":99,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3571504,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":43,"num_read_kb":44484,"num_write":22,"num_write_kb":9163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":942483,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"b0","length":"2"},{"start":"b3","length":"2"},{"start":"b7","length":"1"}]},{"pgid":"4.1b","version":"91'9","reported_seq":"295","reported_epoch":"291","state":"active+clean","last_fresh":"2021-03-24T21:57:41.275760+0000","last_change":"2021-03-24T21:57:02.447431+0000","last_active":"2021-03-24T21:57:41.275760+0000","last_peered":"2021-03-24T21:57:41.275760+0000","last_clean":"2021-03-24T21:57:41.275760+0000","last_became_active":"2021-03-24T21:57:02.184317+0000","last_became_peered":"2021-03-24T21:57:02.184317+0000","last_unstale":"2021-03-24T21:57:41.275760+0000","last_undegraded":"2021-03-24T21:57:41.275760+0000","last_fullsized":"2021-03-24T21:57:41.275760+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2021-03-24T21:56:04.051464+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2021-03-24T21:53:33.290000+0000","last_clean_scrub_stamp":"2021-03-24T21:56:04.051464+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":8,"num_write":7,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":"237","reported_epoch":"291","state":"active+clean","last_fresh":"2021-03-24T21:57:41.274803+0000","last_change":"2021-03-24T21:56:47.921029+0000","last_active":"2021-03-24T21:57:41.274803+0000","last_peered":"2021-03-24T21:57:41.274803+0000","last_clean":"2021-03-24T21:57:41.274803+0000","last_became_active":"2021-03-24T21:55:10.098790+0000","last_became_peered":"2021-03-24T21:55:10.098790+0000","last_unstale":"2021-03-24T21:57:41.274803+0000","last_undegraded":"2021-03-24T21:57:41.274803+0000","last_fullsized":"2021-03-24T21:57:41.274803+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:47.920949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:47.920949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"259","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.283283+0000","last_change":"2021-03-24T21:56:51.977473+0000","last_active":"2021-03-24T21:57:42.283283+0000","last_peered":"2021-03-24T21:57:42.283283+0000","last_clean":"2021-03-24T21:57:42.283283+0000","last_became_active":"2021-03-24T21:54:38.137706+0000","last_became_peered":"2021-03-24T21:54:38.137706+0000","last_unstale":"2021-03-24T21:57:42.283283+0000","last_undegraded":"2021-03-24T21:57:42.283283+0000","last_fullsized":"2021-03-24T21:57:42.283283+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:51.977306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:56:51.977306+0000","last_clean_scrub_stamp":"2021-03-24T21:56:51.977306+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"258","reported_epoch":"291","state":"active+clean","last_fresh":"2021-03-24T21:57:41.282586+0000","last_change":"2021-03-24T21:57:16.077145+0000","last_active":"2021-03-24T21:57:41.282586+0000","last_peered":"2021-03-24T21:57:41.282586+0000","last_clean":"2021-03-24T21:57:41.282586+0000","last_became_active":"2021-03-24T21:54:38.122349+0000","last_became_peered":"2021-03-24T21:54:38.122349+0000","last_unstale":"2021-03-24T21:57:41.282586+0000","last_undegraded":"2021-03-24T21:57:41.282586+0000","last_fullsized":"2021-03-24T21:57:41.282586+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:16.076982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:57:16.076982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.18","version":"249'90","reported_seq":"326","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.277964+0000","last_change":"2021-03-24T21:57:42.277964+0000","last_active":"2021-03-24T21:57:42.277964+0000","last_peered":"2021-03-24T21:57:42.277964+0000","last_clean":"2021-03-24T21:57:42.277964+0000","last_became_active":"2021-03-24T21:56:10.151682+0000","last_became_peered":"2021-03-24T21:56:10.151682+0000","last_unstale":"2021-03-24T21:57:42.277964+0000","last_undegraded":"2021-03-24T21:57:42.277964+0000","last_fullsized":"2021-03-24T21:57:42.277964+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":90,"ondisk_log_size":90,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252122,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":6611,"num_write":17,"num_write_kb":4246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"b0","length":"2"},{"start":"b3","length":"2"},{"start":"b7","length":"1"}]},{"pgid":"4.1a","version":"69'13","reported_seq":"270","reported_epoch":"291","state":"active+clean","last_fresh":"2021-03-24T21:57:41.275196+0000","last_change":"2021-03-24T21:55:31.085064+0000","last_active":"2021-03-24T21:57:41.275196+0000","last_peered":"2021-03-24T21:57:41.275196+0000","last_clean":"2021-03-24T21:57:41.275196+0000","last_became_active":"2021-03-24T21:51:47.407488+0000","last_became_peered":"2021-03-24T21:51:47.407488+0000","last_unstale":"2021-03-24T21:57:41.275196+0000","last_undegraded":"2021-03-24T21:57:41.275196+0000","last_fullsized":"2021-03-24T21:57:41.275196+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2021-03-24T21:55:31.084873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:31.084873+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":"279","reported_epoch":"290","state":"active+clean","last_fresh":"2021-03-24T21:57:40.156428+0000","last_change":"2021-03-24T21:56:32.141420+0000","last_active":"2021-03-24T21:57:40.156428+0000","last_peered":"2021-03-24T21:57:40.156428+0000","last_clean":"2021-03-24T21:57:40.156428+0000","last_became_active":"2021-03-24T21:55:10.097431+0000","last_became_peered":"2021-03-24T21:55:10.097431+0000","last_unstale":"2021-03-24T21:57:40.156428+0000","last_undegraded":"2021-03-24T21:57:40.156428+0000","last_fullsized":"2021-03-24T21:57:40.156428+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:32.141276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:32.141276+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"253","reported_epoch":"291","state":"active+clean","last_fresh":"2021-03-24T21:57:41.283139+0000","last_change":"2021-03-24T21:56:38.242245+0000","last_active":"2021-03-24T21:57:41.283139+0000","last_peered":"2021-03-24T21:57:41.283139+0000","last_clean":"2021-03-24T21:57:41.283139+0000","last_became_active":"2021-03-24T21:51:49.413342+0000","last_became_peered":"2021-03-24T21:51:49.413342+0000","last_unstale":"2021-03-24T21:57:41.283139+0000","last_undegraded":"2021-03-24T21:57:41.283139+0000","last_fullsized":"2021-03-24T21:57:41.283139+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:38.242096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:38.242096+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"256","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.275771+0000","last_change":"2021-03-24T21:57:19.271463+0000","last_active":"2021-03-24T21:57:42.275771+0000","last_peered":"2021-03-24T21:57:42.275771+0000","last_clean":"2021-03-24T21:57:42.275771+0000","last_became_active":"2021-03-24T21:54:38.127696+0000","last_became_peered":"2021-03-24T21:54:38.127696+0000","last_unstale":"2021-03-24T21:57:42.275771+0000","last_undegraded":"2021-03-24T21:57:42.275771+0000","last_fullsized":"2021-03-24T21:57:42.275771+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:19.271351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:57:19.271351+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.19","version":"293'279","reported_seq":"995","reported_epoch":"293","state":"active+clean","last_fresh":"2021-03-24T21:57:43.314276+0000","last_change":"2021-03-24T21:57:42.284781+0000","last_active":"2021-03-24T21:57:43.314276+0000","last_peered":"2021-03-24T21:57:43.314276+0000","last_clean":"2021-03-24T21:57:43.314276+0000","last_became_active":"2021-03-24T21:56:14.139614+0000","last_became_peered":"2021-03-24T21:56:14.139614+0000","last_unstale":"2021-03-24T21:57:43.314276+0000","last_undegraded":"2021-03-24T21:57:43.314276+0000","last_fullsized":"2021-03-24T21:57:43.314276+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":5,"last_scrub":"265'260","last_scrub_stamp":"2021-03-24T21:57:22.158029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:57:22.158029+0000","log_size":279,"ondisk_log_size":279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9445666,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":15631,"num_write":15,"num_write_kb":7593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"b0","length":"2"},{"start":"b3","length":"2"},{"start":"b7","length":"1"}]},{"pgid":"4.19","version":"69'20","reported_seq":"281","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.289293+0000","last_change":"2021-03-24T21:54:54.541380+0000","last_active":"2021-03-24T21:57:42.289293+0000","last_peered":"2021-03-24T21:57:42.289293+0000","last_clean":"2021-03-24T21:57:42.289293+0000","last_became_active":"2021-03-24T21:51:47.405776+0000","last_became_peered":"2021-03-24T21:51:47.405776+0000","last_unstale":"2021-03-24T21:57:42.289293+0000","last_undegraded":"2021-03-24T21:57:42.289293+0000","last_fullsized":"2021-03-24T21:57:42.289293+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'20","last_scrub_stamp":"2021-03-24T21:54:54.541183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:54:54.541183+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":22,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":"266","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.291271+0000","last_change":"2021-03-24T21:55:38.433064+0000","last_active":"2021-03-24T21:57:42.291271+0000","last_peered":"2021-03-24T21:57:42.291271+0000","last_clean":"2021-03-24T21:57:42.291271+0000","last_became_active":"2021-03-24T21:54:38.130163+0000","last_became_peered":"2021-03-24T21:54:38.130163+0000","last_unstale":"2021-03-24T21:57:42.291271+0000","last_undegraded":"2021-03-24T21:57:42.291271+0000","last_fullsized":"2021-03-24T21:57:42.291271+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:38.432918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:38.432918+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"262","reported_epoch":"291","state":"active+clean","last_fresh":"2021-03-24T21:57:41.272763+0000","last_change":"2021-03-24T21:56:34.092326+0000","last_active":"2021-03-24T21:57:41.272763+0000","last_peered":"2021-03-24T21:57:41.272763+0000","last_clean":"2021-03-24T21:57:41.272763+0000","last_became_active":"2021-03-24T21:54:38.120575+0000","last_became_peered":"2021-03-24T21:54:38.120575+0000","last_unstale":"2021-03-24T21:57:41.272763+0000","last_undegraded":"2021-03-24T21:57:41.272763+0000","last_fullsized":"2021-03-24T21:57:41.272763+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:34.092227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:34.092227+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"260","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.292839+0000","last_change":"2021-03-24T21:54:43.482408+0000","last_active":"2021-03-24T21:57:42.292839+0000","last_peered":"2021-03-24T21:57:42.292839+0000","last_clean":"2021-03-24T21:57:42.292839+0000","last_became_active":"2021-03-24T21:54:38.130293+0000","last_became_peered":"2021-03-24T21:54:38.130293+0000","last_unstale":"2021-03-24T21:57:42.292839+0000","last_undegraded":"2021-03-24T21:57:42.292839+0000","last_fullsized":"2021-03-24T21:57:42.292839+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:43.482307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:54:43.482307+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"285'129","reported_seq":"728","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.276889+0000","last_change":"2021-03-24T21:57:42.277357+0000","last_active":"2021-03-24T21:57:42.276889+0000","last_peered":"2021-03-24T21:57:42.276889+0000","last_clean":"2021-03-24T21:57:42.276889+0000","last_became_active":"2021-03-24T21:57:31.167795+0000","last_became_peered":"2021-03-24T21:57:31.167795+0000","last_unstale":"2021-03-24T21:57:42.276889+0000","last_undegraded":"2021-03-24T21:57:42.276889+0000","last_fullsized":"2021-03-24T21:57:42.276889+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":129,"ondisk_log_size":129,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1798382,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":25,"num_read_kb":22368,"num_write":33,"num_write_kb":10745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":3266508,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"b0","length":"2"},{"start":"b3","length":"2"},{"start":"b7","length":"1"}]},{"pgid":"4.18","version":"69'15","reported_seq":"336","reported_epoch":"291","state":"active+clean","last_fresh":"2021-03-24T21:57:41.272244+0000","last_change":"2021-03-24T21:56:05.966329+0000","last_active":"2021-03-24T21:57:41.272244+0000","last_peered":"2021-03-24T21:57:41.272244+0000","last_clean":"2021-03-24T21:57:41.272244+0000","last_became_active":"2021-03-24T21:54:38.127426+0000","last_became_peered":"2021-03-24T21:54:38.127426+0000","last_unstale":"2021-03-24T21:57:41.272244+0000","last_undegraded":"2021-03-24T21:57:41.272244+0000","last_fullsized":"2021-03-24T21:57:41.272244+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:56:05.966219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:05.966219+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2076,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":"257","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.277754+0000","last_change":"2021-03-24T21:56:44.230467+0000","last_active":"2021-03-24T21:57:42.277754+0000","last_peered":"2021-03-24T21:57:42.277754+0000","last_clean":"2021-03-24T21:57:42.277754+0000","last_became_active":"2021-03-24T21:56:44.229902+0000","last_became_peered":"2021-03-24T21:56:44.229902+0000","last_unstale":"2021-03-24T21:57:42.277754+0000","last_undegraded":"2021-03-24T21:57:42.277754+0000","last_fullsized":"2021-03-24T21:57:42.277754+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:38.294685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:38.294685+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"172","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.283441+0000","last_change":"2021-03-24T21:56:34.003068+0000","last_active":"2021-03-24T21:57:42.283441+0000","last_peered":"2021-03-24T21:57:42.283441+0000","last_clean":"2021-03-24T21:57:42.283441+0000","last_became_active":"2021-03-24T21:55:10.099666+0000","last_became_peered":"2021-03-24T21:55:10.099666+0000","last_unstale":"2021-03-24T21:57:42.283441+0000","last_undegraded":"2021-03-24T21:57:42.283441+0000","last_fullsized":"2021-03-24T21:57:42.283441+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:34.002906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:14.710709+0000","last_clean_scrub_stamp":"2021-03-24T21:56:34.002906+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"256","reported_epoch":"291","state":"active+clean","last_fresh":"2021-03-24T21:57:41.273576+0000","last_change":"2021-03-24T21:57:15.687334+0000","last_active":"2021-03-24T21:57:41.273576+0000","last_peered":"2021-03-24T21:57:41.273576+0000","last_clean":"2021-03-24T21:57:41.273576+0000","last_became_active":"2021-03-24T21:54:38.111248+0000","last_became_peered":"2021-03-24T21:54:38.111248+0000","last_unstale":"2021-03-24T21:57:41.273576+0000","last_undegraded":"2021-03-24T21:57:41.273576+0000","last_fullsized":"2021-03-24T21:57:41.273576+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:15.687165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:57:15.687165+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"288'288","reported_seq":"1152","reported_epoch":"293","state":"active+clean","last_fresh":"2021-03-24T21:57:43.354137+0000","last_change":"2021-03-24T21:57:42.278374+0000","last_active":"2021-03-24T21:57:43.354137+0000","last_peered":"2021-03-24T21:57:43.354137+0000","last_clean":"2021-03-24T21:57:43.354137+0000","last_became_active":"2021-03-24T21:57:33.176110+0000","last_became_peered":"2021-03-24T21:57:33.176110+0000","last_unstale":"2021-03-24T21:57:43.354137+0000","last_undegraded":"2021-03-24T21:57:43.354137+0000","last_fullsized":"2021-03-24T21:57:43.354137+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":5,"last_scrub":"191'150","last_scrub_stamp":"2021-03-24T21:56:01.308050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:56:01.308050+0000","log_size":288,"ondisk_log_size":288,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3462325,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":116,"num_read_kb":132716,"num_write":59,"num_write_kb":24002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":28585116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"b0","length":"2"},{"start":"b3","length":"2"},{"start":"b7","length":"1"}]},{"pgid":"7.20","version":"175'22","reported_seq":"321","reported_epoch":"291","state":"active+clean","last_fresh":"2021-03-24T21:57:41.272005+0000","last_change":"2021-03-24T21:57:41.274729+0000","last_active":"2021-03-24T21:57:41.272005+0000","last_peered":"2021-03-24T21:57:41.272005+0000","last_clean":"2021-03-24T21:57:41.272005+0000","last_became_active":"2021-03-24T21:57:28.171436+0000","last_became_peered":"2021-03-24T21:57:28.171436+0000","last_unstale":"2021-03-24T21:57:41.272005+0000","last_undegraded":"2021-03-24T21:57:41.272005+0000","last_fullsized":"2021-03-24T21:57:41.272005+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":6,"last_scrub":"175'22","last_scrub_stamp":"2021-03-24T21:57:32.121490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:57:32.121490+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"b0","length":"2"},{"start":"b3","length":"2"}]},{"pgid":"7.21","version":"288'275","reported_seq":"1014","reported_epoch":"292","state":"active+clean","last_fresh":"2021-03-24T21:57:42.289954+0000","last_change":"2021-03-24T21:57:42.289954+0000","last_active":"2021-03-24T21:57:42.289954+0000","last_peered":"2021-03-24T21:57:42.289954+0000","last_clean":"2021-03-24T21:57:42.289954+0000","last_became_active":"2021-03-24T21:57:32.177566+0000","last_became_peered":"2021-03-24T21:57:32.177566+0000","last_unstale":"2021-03-24T21:57:42.289954+0000","last_undegraded":"2021-03-24T21:57:42.289954+0000","last_fullsized":"2021-03-24T21:57:42.289954+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":275,"ondisk_log_size":275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5559558,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":84002,"num_write":43,"num_write_kb":17590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":10874449,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"b0","length":"2"},{"start":"b3","length":"2"},{"start":"b7","length":"1"}]}],"pool_stats":[{"poolid":7,"num_pg":36,"stat_sum":{"num_bytes":130070934,"num_objects":43,"num_object_clones":8,"num_object_copies":129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":1390,"num_read_kb":1457182,"num_write":939,"num_write_kb":373988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":205,"num_bytes_recovered":103469341,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":294301696,"data_stored":292733435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6127,"ondisk_log_size":6127,"up":108,"acting":108,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":356,"num_read_kb":351,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1589,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":5558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1061,"num_read_kb":566,"num_write":542,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":322,"num_bytes_recovered":25900,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2027520,"data_stored":72734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":558,"ondisk_log_size":558,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201356,"num_pgs":88,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":96652,"kb_used_data":63492,"kb_used_omap":0,"kb_used_meta":33152,"kb_avail":93648500,"statfs":{"total":95995035648,"available":95896064000,"internally_reserved":0,"allocated":65015808,"data_stored":63306447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":33947648},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:56:51 2021","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.048,"15min":0.94799999999999995},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":12.222,"5min":12.222,"15min":12.222},"last":0.49099999999999999},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.026,"15min":0.99099999999999999},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":11.506,"5min":11.506,"15min":11.506},"last":1.044}]},{"osd":1,"last update":"Wed Mar 24 21:56:51 2021","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":0.98199999999999998,"15min":0.88200000000000001},"min":{"1min":0.57399999999999995,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.645,"5min":5.7750000000000004,"15min":5.7750000000000004},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.98999999999999999,"15min":0.97499999999999998},"min":{"1min":0.65800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.403,"5min":6.0780000000000003,"15min":6.0780000000000003},"last":1.569}]},{"osd":2,"last update":"Wed Mar 24 21:56:51 2021","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.1779999999999999,"15min":1.0549999999999999},"min":{"1min":0.436,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":10.827999999999999,"5min":12.135999999999999,"15min":12.135999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":1.4450000000000001,"5min":1.141,"15min":1.0549999999999999},"min":{"1min":0.61799999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":12.178000000000001,"5min":12.178000000000001,"15min":12.178000000000001},"last":0.69499999999999995}]},{"osd":3,"last update":"Wed Mar 24 21:56:51 2021","interfaces":[{"interface":"back","average":{"1min":1.371,"5min":1.1890000000000001,"15min":1.093},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":12.529,"5min":12.529,"15min":12.529},"last":0.56299999999999994},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.198,"15min":1.046},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":12.666,"5min":12.666,"15min":12.666},"last":0.61499999999999999}]},{"osd":4,"last update":"Wed Mar 24 21:56:51 2021","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":1.0469999999999999,"15min":0.96299999999999997},"min":{"1min":0.54800000000000004,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.6439999999999999,"5min":6.2089999999999996,"15min":6.2089999999999996},"last":1.1499999999999999},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.1850000000000001,"15min":1.111},"min":{"1min":0.53400000000000003,"5min":0.504,"15min":0.504},"max":{"1min":1.3360000000000001,"5min":12.244,"15min":12.244},"last":0.63100000000000001}]},{"osd":5,"last update":"Wed Mar 24 21:56:51 2021","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.1259999999999999,"15min":1.018},"min":{"1min":0.51500000000000001,"5min":0.435,"15min":0.435},"max":{"1min":1.569,"5min":5.8840000000000003,"15min":5.8840000000000003},"last":0.77600000000000002},{"interface":"front","average":{"1min":1.131,"5min":1.196,"15min":1.1559999999999999},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.496,"5min":6.3300000000000001,"15min":6.3300000000000001},"last":1.103}]},{"osd":6,"last update":"Wed Mar 24 21:56:51 2021","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.2190000000000001,"15min":1.1439999999999999},"min":{"1min":0.66000000000000003,"5min":0.62,"15min":0.62},"max":{"1min":1.569,"5min":6.1159999999999997,"15min":6.1159999999999997},"last":1.1879999999999999},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.3129999999999999,"15min":1.218},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.4239999999999999,"5min":12.287000000000001,"15min":12.287000000000001},"last":0.66800000000000004}]}]},{"osd":6,"up_from":0,"seq":201863462992,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":0,"seq":55834574950,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738473,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":71588,"kb_used_data":43228,"kb_used_omap":0,"kb_used_meta":28352,"kb_avail":93673564,"statfs":{"total":95995035648,"available":95921729536,"internally_reserved":0,"allocated":44265472,"data_stored":42274519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29032448},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Mar 24 21:56:43 2021","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.71599999999999997,"15min":0.61599999999999999},"min":{"1min":0.35499999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.2070000000000001,"5min":4.8040000000000003,"15min":4.8040000000000003},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.81699999999999995,"15min":0.66900000000000004},"min":{"1min":0.39800000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.871,"5min":8.8989999999999991,"15min":8.8989999999999991},"last":1.069}]},{"osd":2,"last update":"Wed Mar 24 21:57:05 2021","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.76700000000000002,"15min":0.69799999999999995},"min":{"1min":0.43099999999999999,"5min":0.314,"15min":0.314},"max":{"1min":1.962,"5min":5.8810000000000002,"15min":5.8810000000000002},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.84499999999999997,"15min":0.73799999999999999},"min":{"1min":0.61399999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.0069999999999999,"5min":4.766,"15min":4.766},"last":0.66100000000000003}]},{"osd":3,"last update":"Wed Mar 24 21:57:22 2021","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.873,"15min":0.82299999999999995},"min":{"1min":0.48699999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.474,"5min":5.9790000000000001,"15min":5.9790000000000001},"last":1.4099999999999999},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.95999999999999996,"15min":0.93999999999999995},"min":{"1min":0.52100000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.4119999999999999,"5min":6.4580000000000002,"15min":6.4580000000000002},"last":1.119}]},{"osd":4,"last update":"Wed Mar 24 21:56:43 2021","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":1.0880000000000001,"15min":1.0960000000000001},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.7909999999999999,"5min":5.0330000000000004,"15min":5.0330000000000004},"last":0.88600000000000001},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.1140000000000001,"15min":1.042},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.944,"5min":6.1829999999999998,"15min":6.1829999999999998},"last":1.526}]},{"osd":5,"last update":"Wed Mar 24 21:57:05 2021","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":1.0900000000000001,"15min":1.075},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.657,"5min":5.9390000000000001,"15min":5.9390000000000001},"last":0.95599999999999996},{"interface":"front","average":{"1min":1.101,"5min":1.1839999999999999,"15min":1.179},"min":{"1min":0.748,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.4259999999999999,"5min":5.0590000000000002,"15min":5.0590000000000002},"last":1.0009999999999999}]},{"osd":6,"last update":"Wed Mar 24 21:57:22 2021","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.1499999999999999,"15min":1.0940000000000001},"min":{"1min":0.85199999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.7609999999999999,"5min":5.9930000000000003,"15min":5.9930000000000003},"last":1.101},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.212,"15min":1.216},"min":{"1min":0.79900000000000004,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":2.0139999999999998,"5min":4.9960000000000004,"15min":4.9960000000000004},"last":1.1890000000000001}]},{"osd":7,"last update":"Wed Mar 24 21:56:56 2021","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.2070000000000001,"15min":1.181},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.403,"5min":6.1319999999999997,"15min":6.1319999999999997},"last":1.607},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.2030000000000001,"15min":1.151},"min":{"1min":0.77600000000000002,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.9059999999999999,"5min":6.0529999999999999,"15min":6.0529999999999999},"last":1.0229999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411426,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":88564,"kb_used_data":60460,"kb_used_omap":0,"kb_used_meta":28096,"kb_avail":93656588,"statfs":{"total":95995035648,"available":95904346112,"internally_reserved":0,"allocated":61911040,"data_stored":60893600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":28770304},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:57:04 2021","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.69899999999999995,"15min":0.64200000000000002},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.49,"5min":1.49,"15min":1.49},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.751,"15min":0.70099999999999996},"min":{"1min":0.38900000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":2.3119999999999998,"5min":2.3119999999999998,"15min":2.3119999999999998},"last":0.72399999999999998}]},{"osd":1,"last update":"Wed Mar 24 21:57:04 2021","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.77000000000000002,"15min":0.71599999999999997},"min":{"1min":0.44900000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":2.0409999999999999,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.745},"min":{"1min":0.41599999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.6479999999999999,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":0.86399999999999999}]},{"osd":3,"last update":"Wed Mar 24 21:57:21 2021","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.86799999999999999,"15min":0.83799999999999997},"min":{"1min":0.54400000000000004,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.4870000000000001,"5min":1.9890000000000001,"15min":1.9890000000000001},"last":1.1499999999999999},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.90500000000000003,"15min":0.89700000000000002},"min":{"1min":0.45300000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.4950000000000001,"5min":2.0950000000000002,"15min":2.0950000000000002},"last":0.98699999999999999}]},{"osd":4,"last update":"Wed Mar 24 21:56:46 2021","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.0820000000000001,"15min":1.05},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.8520000000000001,"5min":3.3380000000000001,"15min":3.3380000000000001},"last":1.4419999999999999},{"interface":"front","average":{"1min":1.095,"5min":1.089,"15min":1.0529999999999999},"min":{"1min":0.63200000000000001,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":2.0920000000000001,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":1.181}]},{"osd":5,"last update":"Wed Mar 24 21:57:04 2021","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":1.0409999999999999,"15min":1.0349999999999999},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.5609999999999999,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.95799999999999996},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.1599999999999999,"15min":1.1539999999999999},"min":{"1min":0.67500000000000004,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.913,"5min":3.1509999999999998,"15min":3.1509999999999998},"last":1.347}]},{"osd":6,"last update":"Wed Mar 24 21:57:21 2021","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.139,"15min":1.2230000000000001},"min":{"1min":0.70199999999999996,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.5740000000000001,"5min":2.2810000000000001,"15min":2.2810000000000001},"last":1.3759999999999999},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.111,"15min":1.1479999999999999},"min":{"1min":0.77600000000000002,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":2.254,"5min":2.254,"15min":2.254},"last":1.268}]},{"osd":7,"last update":"Wed Mar 24 21:56:50 2021","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.212,"15min":1.226},"min":{"1min":0.73699999999999999,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.714,"5min":4.4880000000000004,"15min":4.4880000000000004},"last":1.228},{"interface":"front","average":{"1min":1.226,"5min":1.228,"15min":1.2430000000000001},"min":{"1min":0.82699999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.9770000000000001,"5min":3.5419999999999998,"15min":3.5419999999999998},"last":1.504}]}]},{"osd":3,"up_from":26,"seq":111669149790,"num_pgs":90,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":85856,"kb_used_data":56920,"kb_used_omap":0,"kb_used_meta":28928,"kb_avail":93659296,"statfs":{"total":95995035648,"available":95907119104,"internally_reserved":0,"allocated":58286080,"data_stored":56597894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29622272},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:57:27 2021","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.87,"15min":0.71399999999999997},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.97399999999999998,"5min":23.846,"15min":23.846},"last":0.501},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.93400000000000005,"15min":0.79300000000000004},"min":{"1min":0.38100000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.131,"5min":23.734999999999999,"15min":23.734999999999999},"last":0.61399999999999999}]},{"osd":1,"last update":"Wed Mar 24 21:57:27 2021","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.86499999999999999,"15min":0.78300000000000003},"min":{"1min":0.39400000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.169,"5min":14.324,"15min":14.324},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.85199999999999998,"15min":0.78900000000000003},"min":{"1min":0.47699999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.995,"5min":12.631,"15min":12.631},"last":0.875}]},{"osd":2,"last update":"Wed Mar 24 21:57:27 2021","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.91200000000000003,"15min":0.82799999999999996},"min":{"1min":0.55300000000000005,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.0169999999999999,"5min":12.510999999999999,"15min":12.510999999999999},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.92700000000000005,"15min":0.82099999999999995},"min":{"1min":0.46300000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.502,"5min":11.317,"15min":11.317},"last":0.54500000000000004}]},{"osd":4,"last update":"Wed Mar 24 21:56:46 2021","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.018,"15min":0.97499999999999998},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.696,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.95899999999999996,"5min":1.155,"15min":1.0069999999999999},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.377,"5min":14.291,"15min":14.291},"last":0.80300000000000005}]},{"osd":5,"last update":"Wed Mar 24 21:57:00 2021","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":1.1799999999999999,"15min":1.1000000000000001},"min":{"1min":0.62,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.405,"5min":12.723000000000001,"15min":12.723000000000001},"last":0.92600000000000005},{"interface":"front","average":{"1min":1.0880000000000001,"5min":1.0900000000000001,"15min":1.0940000000000001},"min":{"1min":0.64700000000000002,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.6100000000000001,"5min":2.375,"15min":2.375},"last":0.94099999999999995}]},{"osd":6,"last update":"Wed Mar 24 21:57:27 2021","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":1.232,"15min":1.1639999999999999},"min":{"1min":0.65800000000000003,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.629,"5min":14.255000000000001,"15min":14.255000000000001},"last":1.004},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.248,"15min":1.1850000000000001},"min":{"1min":0.73299999999999998,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.5029999999999999,"5min":12.666,"15min":12.666},"last":1.2529999999999999}]},{"osd":7,"last update":"Wed Mar 24 21:56:52 2021","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.1890000000000001,"15min":1.1140000000000001},"min":{"1min":0.61899999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.78,"5min":2.3479999999999999,"15min":2.3479999999999999},"last":1.0229999999999999},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.3280000000000001,"15min":1.216},"min":{"1min":0.49099999999999999,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.5029999999999999,"5min":14.727,"15min":14.727},"last":1.0700000000000001}]}]},{"osd":4,"up_from":32,"seq":137438953560,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":56768,"kb_used_data":30136,"kb_used_omap":0,"kb_used_meta":26624,"kb_avail":93688384,"statfs":{"total":95995035648,"available":95936905216,"internally_reserved":0,"allocated":30859264,"data_stored":29278691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27262976},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:56:42 2021","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.84199999999999997,"15min":0.83999999999999997},"min":{"1min":0.59999999999999998,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.149,"5min":1.268,"15min":1.268},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.88,"5min":0.99099999999999999,"15min":0.89500000000000002},"min":{"1min":0.62,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.4590000000000001,"5min":6.3929999999999998,"15min":6.3929999999999998},"last":0.89400000000000002}]},{"osd":1,"last update":"Wed Mar 24 21:56:42 2021","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":0.93100000000000005,"15min":0.92200000000000004},"min":{"1min":0.53300000000000003,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.2849999999999999,"5min":1.3120000000000001,"15min":1.4570000000000001},"last":1.054},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.92000000000000004,"15min":0.86399999999999999},"min":{"1min":0.67400000000000004,"5min":0.436,"15min":0.436},"max":{"1min":1.391,"5min":1.873,"15min":1.873},"last":1.091}]},{"osd":2,"last update":"Wed Mar 24 21:56:42 2021","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":0.95799999999999996,"15min":0.91600000000000004},"min":{"1min":0.60099999999999998,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.1719999999999999,"5min":4.4379999999999997,"15min":4.4379999999999997},"last":0.70299999999999996},{"interface":"front","average":{"1min":1.069,"5min":1.0029999999999999,"15min":0.93700000000000006},"min":{"1min":0.67000000000000004,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.502,"5min":2.738,"15min":2.738},"last":1.143}]},{"osd":3,"last update":"Wed Mar 24 21:56:42 2021","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":0.98699999999999999,"15min":0.95299999999999996},"min":{"1min":0.68100000000000005,"5min":0.439,"15min":0.439},"max":{"1min":1.518,"5min":3.2000000000000002,"15min":3.2000000000000002},"last":0.96899999999999997},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1,"15min":0.96199999999999997},"min":{"1min":0.77200000000000002,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.456,"5min":1.474,"15min":1.474},"last":0.93300000000000005}]},{"osd":5,"last update":"Wed Mar 24 21:57:05 2021","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":0.95999999999999996,"15min":0.95399999999999996},"min":{"1min":0.73899999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.2350000000000001,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":0.996},{"interface":"front","average":{"1min":1.139,"5min":1.107,"15min":1.0269999999999999},"min":{"1min":0.83499999999999996,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.405,"5min":2.0760000000000001,"15min":2.0760000000000001},"last":1.0760000000000001}]},{"osd":6,"last update":"Wed Mar 24 21:57:23 2021","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.081,"15min":1.071},"min":{"1min":0.84299999999999997,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.6859999999999999,"5min":1.6859999999999999,"15min":1.6859999999999999},"last":1.202},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.1020000000000001,"15min":1.0780000000000001},"min":{"1min":0.91400000000000003,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.631,"5min":1.631,"15min":1.631},"last":1.022}]},{"osd":7,"last update":"Wed Mar 24 21:56:48 2021","interfaces":[{"interface":"back","average":{"1min":1.2230000000000001,"5min":1.145,"15min":1.179},"min":{"1min":0.92700000000000005,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.6830000000000001,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":1.179},{"interface":"front","average":{"1min":1.272,"5min":1.1819999999999999,"15min":1.198},"min":{"1min":1.0349999999999999,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.635,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":1.123}]}]},{"osd":5,"up_from":39,"seq":167503724629,"num_pgs":90,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":70372,"kb_used_data":44508,"kb_used_omap":0,"kb_used_meta":25856,"kb_avail":93674780,"statfs":{"total":95995035648,"available":95922974720,"internally_reserved":0,"allocated":45576192,"data_stored":43895823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26476544},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:57:05 2021","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.84199999999999997,"15min":0.873},"min":{"1min":0.58099999999999996,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.0860000000000001,"5min":2.29,"15min":2.29},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.89700000000000002,"15min":0.88800000000000001},"min":{"1min":0.51400000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.1870000000000001,"5min":3.1779999999999999,"15min":3.1779999999999999},"last":1.3200000000000001}]},{"osd":1,"last update":"Wed Mar 24 21:57:05 2021","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.89800000000000002,"15min":0.91400000000000003},"min":{"1min":0.49299999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.26,"5min":2.8690000000000002,"15min":2.8690000000000002},"last":1.0620000000000001},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.91800000000000004,"15min":0.92800000000000005},"min":{"1min":0.56000000000000005,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.2749999999999999,"5min":2.0030000000000001,"15min":2.0030000000000001},"last":1.133}]},{"osd":2,"last update":"Wed Mar 24 21:57:05 2021","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":0.96899999999999997,"15min":0.92500000000000004},"min":{"1min":0.60199999999999998,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.1160000000000001,"5min":6.5199999999999996,"15min":6.5199999999999996},"last":1.272},{"interface":"front","average":{"1min":1.0720000000000001,"5min":0.99199999999999999,"15min":0.92200000000000004},"min":{"1min":0.55800000000000005,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":5.5010000000000003,"5min":5.8479999999999999,"15min":5.8479999999999999},"last":0.96999999999999997}]},{"osd":3,"last update":"Wed Mar 24 21:57:05 2021","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.96199999999999997,"15min":0.94199999999999995},"min":{"1min":0.625,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.3220000000000001,"5min":3.1259999999999999,"15min":3.1259999999999999},"last":1.2030000000000001},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.999,"15min":0.95899999999999996},"min":{"1min":0.54000000000000004,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.2490000000000001,"5min":2.6850000000000001,"15min":2.6850000000000001},"last":1.175}]},{"osd":4,"last update":"Wed Mar 24 21:57:05 2021","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.98699999999999999,"15min":0.879},"min":{"1min":0.44900000000000001,"5min":0.376,"15min":0.376},"max":{"1min":1.2410000000000001,"5min":2.2959999999999998,"15min":2.2959999999999998},"last":1.0169999999999999},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.069,"15min":0.94899999999999995},"min":{"1min":0.56100000000000005,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.389,"5min":2.629,"15min":2.629},"last":1.2230000000000001}]},{"osd":6,"last update":"Wed Mar 24 21:57:26 2021","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":1.0509999999999999,"15min":0.92300000000000004},"min":{"1min":0.67000000000000004,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.3149999999999999,"5min":2.3879999999999999,"15min":2.3879999999999999},"last":1.2470000000000001},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.107,"15min":1.0049999999999999},"min":{"1min":0.55900000000000005,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.458,"5min":2.4660000000000002,"15min":2.4660000000000002},"last":1.3759999999999999}]},{"osd":7,"last update":"Wed Mar 24 21:56:51 2021","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.145,"15min":1.077},"min":{"1min":0.64700000000000002,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.335,"5min":2.548,"15min":2.548},"last":1.1020000000000001},{"interface":"front","average":{"1min":1.153,"5min":1.145,"15min":1.0309999999999999},"min":{"1min":0.66700000000000004,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.5389999999999999,"5min":2.9889999999999999,"15min":2.9889999999999999},"last":1.302}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":14134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":7234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":13448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":352256,"data_stored":12784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":360448,"data_stored":13024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":9948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":41398272,"data_stored":40867543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7032832,"data_stored":6977532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57331712,"data_stored":57411728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":55451648,"data_stored":55192927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28053504,"data_stored":27881353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":42766336,"data_stored":42497464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":62267392,"data_stored":61904888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-03-24T21:57:47.344 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj -7, currently 3 2021-03-24T21:57:47.354 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:47.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:47.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:57:47.395 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:47 smithi142 conmon[20125]: audit 2021-03-24T21:57 2021-03-24T21:57:47.396 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:47 smithi142 conmon[20125]: :45.808500+0000 mgr.y (mgr.14144) 415 : audit [DBG] from='client.15093 v1:172.21.15.125:0/2613517778' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T21:57:47.396 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:47 smithi142 conmon[20125]: cluster 2021-03-24T21 2021-03-24T21:57:47.396 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:47 smithi142 conmon[20125]: :57:47.168185+0000 mon.a (mon.0) 960 : cluster [DBG] osdmap e297: 8 total, 8 up, 6 in 2021-03-24T21:57:47.396 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:47 smithi142 conmon[20125]: 2021-03-24T21:57:47.458 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:47.470 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:47 smithi125 conmon[24927]: audit 2021-03-24T21:57:45.808500+0000 mgr.y (mgr.14144) 415 : audit [DBG] from='client.15093 v1:172.21.15.125:0/2613517778' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T21:57:47.471 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:47 smithi125 conmon[24927]: cluster 2021-03-24T21:57:47.471 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:47 smithi125 conmon[24927]: 2021-03-24T21:57:47.168185+0000 mon.a (mon.0) 960 : cluster [DBG] osdmap e297: 8 total, 8 up, 6 in 2021-03-24T21:57:47.471 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:47 smithi125 conmon[19936]: audit 2021-03-24T21:57:45.808500+0000 2021-03-24T21:57:47.472 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:47 smithi125 conmon[19936]: mgr.y (mgr.14144) 415 : audit [DBG] from='client.15093 v1:172.21.15.125:0/2613517778' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T21:57:47.472 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:47 smithi125 conmon[19936]: cluster 2021-03-24T21:57:47.168185+0000 mon.a (mon.0) 960 : cluster [DBG] osdmap e297: 8 total, 8 up, 6 in 2021-03-24T21:57:47.517 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:47.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:47.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:57:47.624 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:47.680 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:47.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:47.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:57:47.788 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:47.844 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:47.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:47.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:57:47.949 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:48.007 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:48.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:48.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:57:48.112 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:48.169 INFO:tasks.rados.rados.0.smithi125.stdout:995: done (0 left) 2021-03-24T21:57:48.170 INFO:tasks.rados.rados.0.smithi125.stdout:996: snap_create 2021-03-24T21:57:48.171 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:48.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:48.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:57:48.277 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:48.333 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:48.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:48.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:57:48.440 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:48.494 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:48.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:48.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:57:48.601 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:48.614 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:48 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:57:48.615 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:48 smithi125 conmon[19936]: :57:46.781002+0000 mgr.y (mgr.14144) 416 : cluster [DBG] pgmap v607: 168 pgs: 168 active+clean; 124 MiB data, 457 MiB used, 536 GiB / 536 GiB avail; 8.3 MiB/s rd, 3.4 MiB/s wr, 12 op/s 2021-03-24T21:57:48.615 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:48 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:57:48.615 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:48 smithi125 conmon[19936]: 24T21:57:48.165417+0000 mon.a (mon.0) 961 : cluster [DBG] osdmap e298: 8 total, 8 up, 6 in 2021-03-24T21:57:48.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:48 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:57:48.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:48 smithi125 conmon[24927]: :57:46.781002+0000 mgr.y (mgr.14144) 416 : cluster [DBG] pgmap v607: 168 pgs: 168 active+clean; 124 MiB data, 457 MiB used, 536 GiB / 536 GiB avail; 8.3 MiB/s rd, 3.4 MiB/s wr, 12 op/s 2021-03-24T21:57:48.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:48 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:57:48.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:48 smithi125 conmon[24927]: -03-24T21:57:48. 2021-03-24T21:57:48.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:48 smithi125 conmon[24927]: 165417+0000 mon.a (mon.0 2021-03-24T21:57:48.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:48 smithi125 conmon[24927]: ) 961 : cluster [DBG] osdmap e298: 8 total, 8 up, 6 in 2021-03-24T21:57:48.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:48 smithi142 conmon[20125]: cluster 2021-03-24T21:57:46.781002+0000 mgr.y (mgr.14144) 2021-03-24T21:57:48.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:48 smithi142 conmon[20125]: 416 : cluster [DBG] pgmap v607: 168 pgs: 168 active+clean; 124 MiB data, 457 MiB used, 536 GiB / 536 GiB avail; 8.3 MiB/s rd, 3.4 MiB/s wr, 12 op/s 2021-03-24T21:57:48.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:48 smithi142 conmon[20125]: cluster 2021-03-24T21:57:48.165417+0000 mon.a (mon.0) 961 : cluster [DBG] osdmap e298: 8 total, 8 up, 6 in 2021-03-24T21:57:48.662 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:48.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:48.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:57:48.771 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:48.823 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:48.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:48.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:57:48.880 INFO:journalctl@ceph.osd.0.smithi125.stdout:Mar 24 21:57:48 smithi125 conmon[28026]: debug 2021-03-24T21:57:48.827+0000 7f61b647b700 -1 osd.0 pg_epoch: 298 pg[7.f( v 297'106 (0'0,297'106] local-lis/les=275/276 n=1 ec=99/99 lis/c=275/275 les/c/f=276/276/0 sis=275) [0,3,4] r=0 lpr=275 crt=297'106 lcod 297'104 mlcod 297'104 active+clean+snaptrim trimq=[bb~1] ps=[b1~1,b3~2,b7~1,ba~1,bc~1]] removing snap head 2021-03-24T21:57:48.932 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:48.988 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:48.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:48.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:49.095 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:49.148 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:49.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:49.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:57:49.165 INFO:tasks.rados.rados.0.smithi125.stdout:996: done (0 left) 2021-03-24T21:57:49.166 INFO:tasks.rados.rados.0.smithi125.stdout:997: delete oid 14 current snap is 96 2021-03-24T21:57:49.172 INFO:tasks.rados.rados.0.smithi125.stdout:997: done (0 left) 2021-03-24T21:57:49.172 INFO:tasks.rados.rados.0.smithi125.stdout:998: delete oid 43 current snap is 96 2021-03-24T21:57:49.174 INFO:tasks.rados.rados.0.smithi125.stdout:998: done (0 left) 2021-03-24T21:57:49.175 INFO:tasks.rados.rados.0.smithi125.stdout:999: snap_remove snap 95 2021-03-24T21:57:49.241 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2021-03-24T21:57:49.241 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T21:57:49.253 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:49.267 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-24T21:57:49.305 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:49.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:49.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:57:49.410 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:49.462 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:49.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:49.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:57:49.567 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:49.582 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:57:49.616 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:49.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:49.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:49.722 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:49.780 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:49.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:49.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:49.886 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:49.944 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:49.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:49.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:57:50.050 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:50.111 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:50.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:50.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:50.166 INFO:tasks.rados.rados.0.smithi125.stdout:999: done (0 left) 2021-03-24T21:57:50.167 INFO:tasks.rados.rados.0.smithi125.stdout:1000: write (excl) oid 35 current snap is 96 2021-03-24T21:57:50.167 INFO:tasks.rados.rados.0.smithi125.stdout:1000: seq_num 374 ranges {685818=644423,2069089=216729} 2021-03-24T21:57:50.169 INFO:teuthology.orchestra.run.smithi125.stderr:noscrub is unset 2021-03-24T21:57:50.175 INFO:tasks.rados.rados.0.smithi125.stdout:1000: writing smithi12556470-35 from 685818 to 1330241 tid 1 2021-03-24T21:57:50.181 INFO:tasks.rados.rados.0.smithi125.stdout:1000: writing smithi12556470-35 from 2069089 to 2285818 tid 2 2021-03-24T21:57:50.181 INFO:tasks.rados.rados.0.smithi125.stdout:1001: snap_create 2021-03-24T21:57:50.186 INFO:tasks.rados.rados.0.smithi125.stdout:1000: finishing write tid 1 to smithi12556470-35 2021-03-24T21:57:50.191 INFO:tasks.rados.rados.0.smithi125.stdout:1000: finishing write tid 2 to smithi12556470-35 2021-03-24T21:57:50.191 INFO:tasks.rados.rados.0.smithi125.stdout:1000: finishing write tid 3 to smithi12556470-35 2021-03-24T21:57:50.191 INFO:tasks.rados.rados.0.smithi125.stdout:1000: finishing write tid 4 to smithi12556470-35 2021-03-24T21:57:50.191 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 35 v 200 (ObjNum 374 snap 96 seq_num 374) dirty exists 2021-03-24T21:57:50.192 INFO:tasks.rados.rados.0.smithi125.stdout:1000: left oid 35 (ObjNum 374 snap 96 seq_num 374) 2021-03-24T21:57:50.215 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:50.257 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.4 dump_historic_ops 2021-03-24T21:57:50.267 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:50.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:50.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:57:50.358 INFO:journalctl@ceph.osd.0.smithi125.stdout:Mar 24 21:57:50 smithi125 conmon[28026]: debug 2021-03-24T21:57:50.311+0000 7f61b847f700 -1 osd.0 pg_epoch: 300 pg[7.1b( v 299'291 (0'0,299'291] local-lis/les=282/283 n=2 ec=275/99 lis/c=282/282 les/c/f=283/283/0 sis=282) [0,2,3] r=0 lpr=282 crt=299'290 lcod 299'290 mlcod 299'290 active+clean+snaptrim trimq=[c0~1] ps=[ba~3]] removing snap head 2021-03-24T21:57:50.359 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:50 smithi125 conmon[19936]: cluster 2021-03-24T21:57:48.782079 2021-03-24T21:57:50.359 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:50 smithi125 conmon[19936]: +0000 mgr.y (mgr.14144) 417 : cluster [DBG] pgmap v610: 168 pgs: 2 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 164 active+clean; 127 MiB data, 463 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 464 KiB/s wr, 3 op/s 2021-03-24T21:57:50.359 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:50 smithi125 conmon[19936]: cluster 2021-03-24T21:57:49.164633+0000 mon.a (mon.0) 2021-03-24T21:57:50.359 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:50 smithi125 conmon[19936]: 962 : cluster [DBG] osdmap e299: 8 total, 8 up, 6 in 2021-03-24T21:57:50.360 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:50 smithi125 conmon[19936]: audit 2021-03-24T21:57:50.360 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:50 smithi125 conmon[19936]: 2021-03-24T21:57:49.382105+0000 mon.a (mon. 2021-03-24T21:57:50.360 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:50 smithi125 conmon[19936]: 0) 963 : audit [INF] from='client.? v1:172.21.15.125:0/313808972' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T21:57:50.360 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:50 smithi125 conmon[19936]: cluster 2021-03-24T21:57:50.361 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:50 smithi125 conmon[19936]: 2021-03-24T21:57:50.151626+0000 mon.a ( 2021-03-24T21:57:50.361 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:50 smithi125 conmon[19936]: mon.0) 964 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:57:50.361 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:50 smithi125 conmon[19936]: audit 2021-03-24T21:57:50.361 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:50 smithi125 conmon[19936]: 2021-03-24T21:57:50.162554+0000 2021-03-24T21:57:50.361 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:50 smithi125 conmon[19936]: mon.a (mon.0) 965 : audit [INF] from='client.? v1:172.21.15.125:0/313808972' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T21:57:50.362 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:50 smithi125 conmon[19936]: cluster 2021-03-24T21:57:50.362 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:50 smithi125 conmon[19936]: 2021-03-24T21:57:50.162620+0000 2021-03-24T21:57:50.362 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:50 smithi125 conmon[19936]: mon.a (mon.0) 966 : cluster [DBG] osdmap e300: 8 total, 8 up, 6 in 2021-03-24T21:57:50.363 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:50 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:57:50.363 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:50 smithi125 conmon[24927]: 21:57:48.782079+0000 mgr.y (mgr.14144) 417 : cluster [DBG] pgmap v610: 168 pgs: 2 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 164 active+clean; 127 MiB data, 463 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 464 KiB/s wr, 3 op/s 2021-03-24T21:57:50.363 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:50 smithi125 conmon[24927]: cluster 2021-03-24T21:57 2021-03-24T21:57:50.363 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:50 smithi125 conmon[24927]: :49.164633+0000 mon.a (mon.0) 962 : cluster 2021-03-24T21:57:50.364 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:50 smithi125 conmon[24927]: [DBG] osdmap e299: 8 total, 8 up, 6 in 2021-03-24T21:57:50.364 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:50 smithi125 conmon[24927]: audit 2021-03-24T21:57:49. 2021-03-24T21:57:50.365 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:50 smithi125 conmon[24927]: 382105+0000 mon.a (mon.0 2021-03-24T21:57:50.365 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:50 smithi125 conmon[24927]: ) 963 : audit [INF] 2021-03-24T21:57:50.365 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:50 smithi125 conmon[24927]: from='client.? v1:172.21.15.125:0/313808972' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T21:57:50.366 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:50 smithi125 conmon[24927]: cluster 2021-03-24T21:57 2021-03-24T21:57:50.366 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:50 smithi125 conmon[24927]: :50.151626+0000 mon.a (mon.0) 964 2021-03-24T21:57:50.366 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:50 smithi125 conmon[24927]: : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:57:50.367 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:50 smithi125 conmon[24927]: audit 2021-03-24T21:57:50.162554+0000 mon.a (mon.0) 965 : audit [INF] from='client.? v1:172.21.15.125:0/313808972' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T21:57:50.367 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:50 smithi125 conmon[24927]: cluster 2021-03-24T21:57:50.162620+0000 mon.a (mon.0) 966 : cluster [DBG] osdmap e300: 8 total, 8 up, 6 in 2021-03-24T21:57:50.374 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:50.431 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:50.434 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:50 smithi142 conmon[20125]: cluster 2021-03-24T21:57:48.782079 2021-03-24T21:57:50.434 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:50 smithi142 conmon[20125]: +0000 mgr.y (mgr.14144) 417 : cluster [DBG] pgmap v610: 168 pgs: 2 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 164 active+clean; 127 MiB data, 463 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 464 KiB/s wr, 3 op/s 2021-03-24T21:57:50.434 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:50 smithi142 conmon[20125]: cluster 2021-03-24T21:57:49.164633+0000 mon.a (mon.0) 2021-03-24T21:57:50.435 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:50 smithi142 conmon[20125]: 962 : cluster [DBG] osdmap e299: 8 total, 8 up, 6 in 2021-03-24T21:57:50.435 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:50 smithi142 conmon[20125]: audit 2021-03-24T21:57:50.435 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:50 smithi142 conmon[20125]: 2021-03-24T21:57:49.382105+0000 mon.a (mon.0 2021-03-24T21:57:50.435 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:50 smithi142 conmon[20125]: ) 963 : audit [INF] from='client.? v1:172.21.15.125:0/313808972' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T21:57:50.436 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:50 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:57:50.436 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:50 smithi142 conmon[20125]: 03-24T21:57:50.151626+0000 2021-03-24T21:57:50.436 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:50 smithi142 conmon[20125]: mon.a (mon.0) 964 : 2021-03-24T21:57:50.436 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:50 smithi142 conmon[20125]: cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:57:50.436 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:50 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:57:50.437 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:50 smithi142 conmon[20125]: 03-24T21:57:50.162554 2021-03-24T21:57:50.437 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:50 smithi142 conmon[20125]: +0000 mon.a (mon.0) 965 2021-03-24T21:57:50.437 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:50 smithi142 conmon[20125]: : audit [INF] from='client.? v1:172.21.15.125:0/313808972' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T21:57:50.437 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:50 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:57:50.438 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:50 smithi142 conmon[20125]: 03-24T21:57:50.162620+0000 mon.a 2021-03-24T21:57:50.438 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:50 smithi142 conmon[20125]: (mon.0) 966 : cluster 2021-03-24T21:57:50.438 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:50 smithi142 conmon[20125]: [DBG] osdmap e300: 8 total, 8 up, 6 in 2021-03-24T21:57:50.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:50.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:57:50.540 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:50.545 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T21:57:50.594 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:50.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:50.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:57:50.700 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:50.757 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:50.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:50.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:57:50.863 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:50.921 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:50.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:50.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:57:51.027 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:51.085 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:51.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:51.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:57:51.180 INFO:tasks.rados.rados.0.smithi125.stdout:1000: done (1 left) 2021-03-24T21:57:51.181 INFO:tasks.rados.rados.0.smithi125.stdout:1001: done (0 left) 2021-03-24T21:57:51.181 INFO:tasks.rados.rados.0.smithi125.stdout:1002: snap_create 2021-03-24T21:57:51.192 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:51.248 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:51.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:51.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:51.353 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:51.413 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:51.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:51.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:57:51.519 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:51.582 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:51.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:51.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:57:51.687 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:51.741 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:51.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:51.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:51.847 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:51.903 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:51.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:51.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:57:52.013 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:52.068 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:52.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:52.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:57:52.173 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:52.185 INFO:tasks.rados.rados.0.smithi125.stdout:1002: done (0 left) 2021-03-24T21:57:52.185 INFO:tasks.rados.rados.0.smithi125.stdout:1003: write oid 41 current snap is 98 2021-03-24T21:57:52.185 INFO:tasks.rados.rados.0.smithi125.stdout:1003: seq_num 375 ranges {702625=719672,2127571=175054} 2021-03-24T21:57:52.193 INFO:tasks.rados.rados.0.smithi125.stdout:1003: writing smithi12556470-41 from 702625 to 1422297 tid 1 2021-03-24T21:57:52.198 INFO:tasks.rados.rados.0.smithi125.stdout:1003: writing smithi12556470-41 from 2127571 to 2302625 tid 2 2021-03-24T21:57:52.199 INFO:tasks.rados.rados.0.smithi125.stdout:1004: write (excl) oid 32 current snap is 98 2021-03-24T21:57:52.199 INFO:tasks.rados.rados.0.smithi125.stdout:1004: seq_num 376 ranges {719432=794921,2186053=133379} 2021-03-24T21:57:52.207 INFO:tasks.rados.rados.0.smithi125.stdout:1004: writing smithi12556470-32 from 719432 to 1514353 tid 1 2021-03-24T21:57:52.213 INFO:tasks.rados.rados.0.smithi125.stdout:1004: writing smithi12556470-32 from 2186053 to 2319432 tid 2 2021-03-24T21:57:52.213 INFO:tasks.rados.rados.0.smithi125.stdout:1003: finishing write tid 1 to smithi12556470-41 2021-03-24T21:57:52.214 INFO:tasks.rados.rados.0.smithi125.stdout:1003: finishing write tid 2 to smithi12556470-41 2021-03-24T21:57:52.214 INFO:tasks.rados.rados.0.smithi125.stdout:1003: finishing write tid 3 to smithi12556470-41 2021-03-24T21:57:52.214 INFO:tasks.rados.rados.0.smithi125.stdout:1003: finishing write tid 4 to smithi12556470-41 2021-03-24T21:57:52.214 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 41 v 285 (ObjNum 375 snap 98 seq_num 375) dirty exists 2021-03-24T21:57:52.215 INFO:tasks.rados.rados.0.smithi125.stdout:1003: left oid 41 (ObjNum 375 snap 98 seq_num 375) 2021-03-24T21:57:52.215 INFO:tasks.rados.rados.0.smithi125.stdout:1003: done (1 left) 2021-03-24T21:57:52.215 INFO:tasks.rados.rados.0.smithi125.stdout:1005: read oid 43 snap -1 2021-03-24T21:57:52.215 INFO:tasks.rados.rados.0.smithi125.stdout:1005: expect deleted 2021-03-24T21:57:52.216 INFO:tasks.rados.rados.0.smithi125.stdout:1006: snap_remove snap 97 2021-03-24T21:57:52.225 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:52.228 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:52 smithi125 conmon[19936]: cluster 2021-03-24T21:57:50.522339+0000 2021-03-24T21:57:52.228 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:52 smithi125 conmon[19936]: osd.4 (osd.4) 78 : cluster [DBG] 3.a scrub ok 2021-03-24T21:57:52.229 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:52 smithi125 conmon[19936]: cluster 2021-03-24T21:57:50.782944+0000 mgr.y (mgr.14144) 418 : cluster [DBG] 2021-03-24T21:57:52.229 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:52 smithi125 conmon[19936]: pgmap v613: 168 pgs: 2 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 164 active+clean; 118 MiB data, 460 MiB used, 536 GiB / 536 GiB avail; 3.9 MiB/s rd, 1.5 MiB/s wr, 10 op/s 2021-03-24T21:57:52.229 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:52 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:57:52.229 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:52 smithi125 conmon[19936]: 24T21:57:51.037771+0000 osd.7 (osd.7) 67 : cluster [DBG] 3.11 scrub ok 2021-03-24T21:57:52.230 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:52 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:57:52.230 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:52 smithi125 conmon[19936]: 03-24T21:57:51.176806+0000 mon.a (mon 2021-03-24T21:57:52.230 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:52 smithi125 conmon[19936]: .0) 967 : cluster [DBG] osdmap e301: 8 total, 8 up, 6 in 2021-03-24T21:57:52.230 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:52 smithi125 conmon[24927]: cluster 2021-03-24T21:57:50.522339+0000 osd.4 2021-03-24T21:57:52.231 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:52 smithi125 conmon[24927]: (osd.4) 78 : cluster [DBG] 3.a scrub ok 2021-03-24T21:57:52.231 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:52 smithi125 conmon[24927]: cluster 2021-03-24T21:57: 2021-03-24T21:57:52.231 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:52 smithi125 conmon[24927]: 50.782944+0000 mgr.y (mgr.14144) 418 : cluster [DBG] pgmap v613: 168 pgs: 2 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 164 active+clean; 118 MiB data, 460 MiB used, 536 GiB / 536 GiB avail; 3.9 MiB/s rd, 1.5 MiB/s wr, 10 op/s 2021-03-24T21:57:52.231 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:52 smithi125 conmon[24927]: cluster 2021-03-24T21:57:51.037771+0000 osd.7 ( 2021-03-24T21:57:52.231 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:52 smithi125 conmon[24927]: osd.7) 67 : cluster [DBG] 3.11 scrub ok 2021-03-24T21:57:52.232 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:52 smithi125 conmon[24927]: cluster 2021-03-24T21:57:51.176806+0000 mon.a 2021-03-24T21:57:52.232 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:52 smithi125 conmon[24927]: (mon.0) 967 : cluster [DBG] osdmap e301: 8 total, 8 up, 6 in 2021-03-24T21:57:52.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:52.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:57:52.234 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T21:57:52.250 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T21:57:52.276 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:52 smithi142 conmon[20125]: cluster 2021-03-24T21:57:50.522339+0000 osd.4 (osd.4) 78 : cluster 2021-03-24T21:57:52.277 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:52 smithi142 conmon[20125]: [DBG] 3.a scrub ok 2021-03-24T21:57:52.277 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:52 smithi142 conmon[20125]: cluster 2021-03-24T21:57:50.782944+0000 mgr.y (mgr.14144) 418 : cluster [DBG] pgmap v613: 168 pgs: 2 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 164 active+clean; 118 MiB data, 460 MiB used, 536 GiB / 536 GiB avail; 3.9 MiB/s rd, 1.5 MiB/s wr, 10 op/s 2021-03-24T21:57:52.278 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:52 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:57:52.278 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:52 smithi142 conmon[20125]: -03-24T21:57:51.037771+0000 osd.7 (osd.7) 67 : cluster [DBG] 2021-03-24T21:57:52.278 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:52 smithi142 conmon[20125]: 3.11 scrub ok 2021-03-24T21:57:52.279 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:52 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:57:52.279 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:52 smithi142 conmon[20125]: 57:51.176806+0000 mon.a (mon.0) 967 2021-03-24T21:57:52.279 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:52 smithi142 conmon[20125]: : cluster [DBG] osdmap e301: 8 total, 8 up, 6 in 2021-03-24T21:57:52.280 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T21:57:52.303 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T21:57:52.328 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T21:57:52.334 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:52.345 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 7] out_osds: [6, 1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-03-24T21:57:52.346 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-24T21:57:52.346 INFO:tasks.thrashosds.thrasher:Setting osd 7 primary_affinity to 0.794102 2021-03-24T21:57:52.346 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd primary-affinity 7 0.7941021136087902 2021-03-24T21:57:52.355 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T21:57:52.379 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T21:57:52.391 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:52.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:52.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:57:52.400 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T21:57:52.402 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-24T21:57:52.402 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-24T21:57:52.403 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-24T21:57:52.403 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-24T21:57:52.403 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-24T21:57:52.403 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-24T21:57:52.403 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-24T21:57:52.404 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-24T21:57:52.497 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:52.554 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:52.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:52.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:52.564 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2021-03-24T21:57:52.564 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2021-03-24T21:57:52.564 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2021-03-24T21:57:52.564 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2021-03-24T21:57:52.565 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.565 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPushReply(7.1 110/108 [PushReplyOp(7:83e5d499:::smithi12556470-11:head)])", 2021-03-24T21:57:52.565 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:54:40.024517+0000", 2021-03-24T21:57:52.565 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 192.537652646, 2021-03-24T21:57:52.566 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.086802544999999995, 2021-03-24T21:57:52.566 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:57:52.566 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "reached pg", 2021-03-24T21:57:52.566 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:57:52.567 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.567 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:57:52.567 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.024517+0000", 2021-03-24T21:57:52.567 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:57:52.568 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.568 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.568 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:57:52.568 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.024517+0000", 2021-03-24T21:57:52.569 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2021-03-24T21:57:52.569 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.569 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.570 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:57:52.570 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.024515+0000", 2021-03-24T21:57:52.570 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3049999999999998e-06 2021-03-24T21:57:52.570 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.571 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.571 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:57:52.571 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.024522+0000", 2021-03-24T21:57:52.571 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0208999999999999e-05 2021-03-24T21:57:52.571 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.572 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.572 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:57:52.572 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.024542+0000", 2021-03-24T21:57:52.572 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9319999999999999e-06 2021-03-24T21:57:52.573 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.573 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.573 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:57:52.573 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.024549+0000", 2021-03-24T21:57:52.574 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.086712301000000006 2021-03-24T21:57:52.574 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.574 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.574 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:57:52.574 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.111261+0000", 2021-03-24T21:57:52.575 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7893000000000003e-05 2021-03-24T21:57:52.575 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.575 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.576 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:57:52.576 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.111319+0000", 2021-03-24T21:57:52.576 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.086802544999999995 2021-03-24T21:57:52.576 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:57:52.577 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:57:52.577 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:57:52.577 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.577 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.577 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPushReply(4.9 111/108 [PushReplyOp(4:94d01cf7:::data_log.108:head)])", 2021-03-24T21:57:52.578 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:54:40.207652+0000", 2021-03-24T21:57:52.578 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 192.35451695899999, 2021-03-24T21:57:52.578 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.089012543, 2021-03-24T21:57:52.578 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:57:52.579 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "reached pg", 2021-03-24T21:57:52.579 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:57:52.579 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.579 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:57:52.579 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.207652+0000", 2021-03-24T21:57:52.580 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:57:52.580 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.580 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.580 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:57:52.581 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.207652+0000", 2021-03-24T21:57:52.581 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2021-03-24T21:57:52.581 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.581 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.581 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:57:52.582 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.207650+0000", 2021-03-24T21:57:52.582 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.7899999999999994e-06 2021-03-24T21:57:52.582 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.582 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.583 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:57:52.583 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.207660+0000", 2021-03-24T21:57:52.583 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8369e-05 2021-03-24T21:57:52.583 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.584 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.584 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:57:52.584 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.207678+0000", 2021-03-24T21:57:52.585 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2442e-05 2021-03-24T21:57:52.585 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.585 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.586 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:57:52.586 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.207691+0000", 2021-03-24T21:57:52.587 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.088841322 2021-03-24T21:57:52.587 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.587 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.588 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:57:52.588 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.296532+0000", 2021-03-24T21:57:52.589 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013249 2021-03-24T21:57:52.589 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.590 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.590 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:57:52.590 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.296665+0000", 2021-03-24T21:57:52.591 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.089012543 2021-03-24T21:57:52.591 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:57:52.591 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:57:52.592 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:57:52.592 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.592 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.593 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPushReply(7.1 111/108 [PushReplyOp(7:86662589:::smithi12556470-19:head)])", 2021-03-24T21:57:52.593 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:54:40.297084+0000", 2021-03-24T21:57:52.593 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 192.26508474600001, 2021-03-24T21:57:52.594 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.090340742000000002, 2021-03-24T21:57:52.594 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:57:52.594 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "reached pg", 2021-03-24T21:57:52.595 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:57:52.595 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.595 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:57:52.596 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.297084+0000", 2021-03-24T21:57:52.596 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:57:52.596 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.597 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.597 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:57:52.597 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.297084+0000", 2021-03-24T21:57:52.598 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999971 2021-03-24T21:57:52.598 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.599 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.599 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:57:52.599 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.297082+0000", 2021-03-24T21:57:52.599 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3006999999999999e-05 2021-03-24T21:57:52.600 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.600 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.600 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:57:52.601 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.297095+0000", 2021-03-24T21:57:52.601 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8322000000000001e-05 2021-03-24T21:57:52.601 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.601 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.602 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:57:52.602 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.297123+0000", 2021-03-24T21:57:52.602 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9049999999999999e-05 2021-03-24T21:57:52.602 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.602 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.603 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:57:52.603 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.297142+0000", 2021-03-24T21:57:52.603 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.089791975999999996 2021-03-24T21:57:52.603 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.604 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.604 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:57:52.604 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.386934+0000", 2021-03-24T21:57:52.604 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00049109999999999996 2021-03-24T21:57:52.604 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.605 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.605 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:57:52.605 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.387425+0000", 2021-03-24T21:57:52.605 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.090340742000000002 2021-03-24T21:57:52.605 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:57:52.606 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:57:52.606 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:57:52.606 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.606 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.607 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPushReply(4.9 111/108 [PushReplyOp(4:921ecdb1:gc::gc.25:head)])", 2021-03-24T21:57:52.607 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:54:40.483015+0000", 2021-03-24T21:57:52.607 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 192.07915368799999, 2021-03-24T21:57:52.607 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.089336145000000006, 2021-03-24T21:57:52.608 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:57:52.608 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "reached pg", 2021-03-24T21:57:52.608 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:57:52.608 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.608 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:57:52.609 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.483015+0000", 2021-03-24T21:57:52.609 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:57:52.609 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.609 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.610 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:57:52.610 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.483015+0000", 2021-03-24T21:57:52.610 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999971 2021-03-24T21:57:52.610 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.610 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.611 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:57:52.611 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.483012+0000", 2021-03-24T21:57:52.611 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4007e-05 2021-03-24T21:57:52.611 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.612 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.612 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:57:52.612 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.483026+0000", 2021-03-24T21:57:52.612 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9105999999999999e-05 2021-03-24T21:57:52.612 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.613 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.613 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:57:52.613 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.483056+0000", 2021-03-24T21:57:52.613 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1919999999999999e-05 2021-03-24T21:57:52.614 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.614 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.614 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:57:52.615 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.483067+0000", 2021-03-24T21:57:52.615 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.089135437999999997 2021-03-24T21:57:52.615 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.615 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.616 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:57:52.616 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.572203+0000", 2021-03-24T21:57:52.616 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014872799999999999 2021-03-24T21:57:52.616 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.616 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.617 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:57:52.617 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.572352+0000", 2021-03-24T21:57:52.617 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.089336145000000006 2021-03-24T21:57:52.617 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:57:52.618 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:57:52.618 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:57:52.618 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.618 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.618 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPushReply(7.1 111/108 [PushReplyOp(7:8c1d349c:::smithi12556470-48:head)])", 2021-03-24T21:57:52.619 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:54:40.572536+0000", 2021-03-24T21:57:52.619 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 191.98963270499999, 2021-03-24T21:57:52.619 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.090618961999999997, 2021-03-24T21:57:52.619 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:57:52.620 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "reached pg", 2021-03-24T21:57:52.620 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:57:52.620 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.620 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:57:52.621 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.572536+0000", 2021-03-24T21:57:52.621 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:57:52.621 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.621 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.622 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:57:52.622 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.572536+0000", 2021-03-24T21:57:52.622 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999967 2021-03-24T21:57:52.622 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.623 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.623 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:57:52.623 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.572533+0000", 2021-03-24T21:57:52.623 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3864e-05 2021-03-24T21:57:52.623 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.624 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.624 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:57:52.624 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.572547+0000", 2021-03-24T21:57:52.624 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4148999999999999e-05 2021-03-24T21:57:52.624 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.625 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.625 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:57:52.625 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.572571+0000", 2021-03-24T21:57:52.625 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3868e-05 2021-03-24T21:57:52.625 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.626 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.626 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:57:52.626 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.572585+0000", 2021-03-24T21:57:52.626 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.090012964000000001 2021-03-24T21:57:52.627 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.627 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.627 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:57:52.627 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.662598+0000", 2021-03-24T21:57:52.627 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00055724599999999996 2021-03-24T21:57:52.628 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.628 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.628 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:57:52.628 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.663155+0000", 2021-03-24T21:57:52.629 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.090618961999999997 2021-03-24T21:57:52.629 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:57:52.629 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:57:52.629 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:57:52.629 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.630 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.630 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPushReply(4.9 111/108 [PushReplyOp(4:9268b73e:::obj_delete_at_hint.0000000019:head)])", 2021-03-24T21:57:52.630 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:54:40.758844+0000", 2021-03-24T21:57:52.630 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 191.803324965, 2021-03-24T21:57:52.630 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.089171401999999997, 2021-03-24T21:57:52.631 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:57:52.631 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "reached pg", 2021-03-24T21:57:52.631 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:57:52.631 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.632 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:57:52.632 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.758844+0000", 2021-03-24T21:57:52.632 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:57:52.632 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.633 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.633 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:57:52.633 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.758844+0000", 2021-03-24T21:57:52.633 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999967 2021-03-24T21:57:52.633 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.634 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.634 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:57:52.634 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.758841+0000", 2021-03-24T21:57:52.634 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7056000000000001e-05 2021-03-24T21:57:52.635 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.635 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.635 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:57:52.635 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.758858+0000", 2021-03-24T21:57:52.636 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8711e-05 2021-03-24T21:57:52.636 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.636 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.636 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:57:52.637 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.758887+0000", 2021-03-24T21:57:52.637 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5746000000000001e-05 2021-03-24T21:57:52.637 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.637 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.637 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:57:52.638 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.758902+0000", 2021-03-24T21:57:52.638 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.088984598999999998 2021-03-24T21:57:52.638 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.638 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.639 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:57:52.639 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.847887+0000", 2021-03-24T21:57:52.639 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000128644 2021-03-24T21:57:52.639 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.640 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.640 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:57:52.640 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.848016+0000", 2021-03-24T21:57:52.640 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.089171401999999997 2021-03-24T21:57:52.640 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:57:52.641 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:57:52.641 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:57:52.641 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.641 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.642 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPushReply(7.1 111/108 [PushReplyOp(7:8dd64f9e:::smithi12556470-49:head)])", 2021-03-24T21:57:52.642 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:54:40.847956+0000", 2021-03-24T21:57:52.642 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 191.71421324400001, 2021-03-24T21:57:52.642 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.090871619000000001, 2021-03-24T21:57:52.643 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:57:52.643 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "reached pg", 2021-03-24T21:57:52.643 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:57:52.643 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.644 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:57:52.644 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.847956+0000", 2021-03-24T21:57:52.644 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:57:52.644 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.644 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.645 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:57:52.645 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.847956+0000", 2021-03-24T21:57:52.645 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999967 2021-03-24T21:57:52.645 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.646 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.646 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:57:52.647 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.847953+0000", 2021-03-24T21:57:52.647 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5254e-05 2021-03-24T21:57:52.647 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.648 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.648 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:57:52.649 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.847968+0000", 2021-03-24T21:57:52.649 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9885e-05 2021-03-24T21:57:52.650 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.650 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.651 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:57:52.651 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.847988+0000", 2021-03-24T21:57:52.651 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1423999999999999e-05 2021-03-24T21:57:52.652 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.652 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.653 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:57:52.653 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.847999+0000", 2021-03-24T21:57:52.654 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.090269889000000006 2021-03-24T21:57:52.654 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.655 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.655 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:57:52.655 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.938269+0000", 2021-03-24T21:57:52.656 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00055842100000000003 2021-03-24T21:57:52.656 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.657 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.657 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:57:52.658 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.938828+0000", 2021-03-24T21:57:52.658 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.090871619000000001 2021-03-24T21:57:52.659 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:57:52.659 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:57:52.659 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:57:52.660 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.660 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.661 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPushReply(4.9 111/108 [PushReplyOp(4:94441385:::obj_delete_at_hint.0000000039:head)])", 2021-03-24T21:57:52.661 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:54:41.034565+0000", 2021-03-24T21:57:52.662 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 191.52760464900001, 2021-03-24T21:57:52.662 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.089107565999999999, 2021-03-24T21:57:52.663 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:57:52.663 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "reached pg", 2021-03-24T21:57:52.663 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:57:52.664 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.664 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:57:52.665 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:41.034565+0000", 2021-03-24T21:57:52.665 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:57:52.666 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.666 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.667 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:57:52.667 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:41.034565+0000", 2021-03-24T21:57:52.668 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2021-03-24T21:57:52.668 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.668 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.669 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:57:52.669 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:41.034564+0000", 2021-03-24T21:57:52.669 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.4819999999999999e-06 2021-03-24T21:57:52.670 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.670 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.670 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:57:52.671 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:41.034570+0000", 2021-03-24T21:57:52.671 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6356999999999999e-05 2021-03-24T21:57:52.671 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.672 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.672 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:57:52.672 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:41.034587+0000", 2021-03-24T21:57:52.672 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0502999999999999e-05 2021-03-24T21:57:52.673 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.673 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.673 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:57:52.674 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:41.034597+0000", 2021-03-24T21:57:52.674 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.088942807999999998 2021-03-24T21:57:52.674 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.675 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.675 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:57:52.675 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:41.123540+0000", 2021-03-24T21:57:52.676 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000132261 2021-03-24T21:57:52.676 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.676 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.676 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:57:52.677 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:41.123672+0000", 2021-03-24T21:57:52.677 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.089107565999999999 2021-03-24T21:57:52.678 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:57:52.678 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:57:52.678 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:57:52.679 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.679 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.680 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:255 7.1 7:8c1d349c:::smithi12556470-48:head [copy-from ver 6] snapc 5=[] ondisk+write+known_if_redirected e113)", 2021-03-24T21:57:52.680 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:54:42.171264+0000", 2021-03-24T21:57:52.681 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 190.39090535099999, 2021-03-24T21:57:52.681 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.02147785, 2021-03-24T21:57:52.681 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:57:52.682 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:57:52.682 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:57:52.683 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T21:57:52.683 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:57:52.684 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 255 2021-03-24T21:57:52.684 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.684 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:57:52.685 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.685 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:57:52.686 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.171264+0000", 2021-03-24T21:57:52.686 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:57:52.687 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.687 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.688 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:57:52.688 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.171264+0000", 2021-03-24T21:57:52.688 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999962 2021-03-24T21:57:52.689 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.689 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.690 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:57:52.690 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.171260+0000", 2021-03-24T21:57:52.691 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7472000000000001e-05 2021-03-24T21:57:52.691 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.692 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.692 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:57:52.693 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.171278+0000", 2021-03-24T21:57:52.693 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4187000000000001e-05 2021-03-24T21:57:52.694 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.694 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.694 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:57:52.695 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.171302+0000", 2021-03-24T21:57:52.695 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.91e-06 2021-03-24T21:57:52.696 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.696 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.697 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:57:52.697 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.171312+0000", 2021-03-24T21:57:52.698 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010853600000000001 2021-03-24T21:57:52.698 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.698 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.699 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:57:52.699 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.171420+0000", 2021-03-24T21:57:52.700 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000127892 2021-03-24T21:57:52.700 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.700 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.701 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:57:52.701 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.171548+0000", 2021-03-24T21:57:52.702 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011565719 2021-03-24T21:57:52.702 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.703 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.703 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 0,3", 2021-03-24T21:57:52.703 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.183114+0000", 2021-03-24T21:57:52.704 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0061237039999999998 2021-03-24T21:57:52.704 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.704 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.704 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T21:57:52.704 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.189238+0000", 2021-03-24T21:57:52.705 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0027285859999999999 2021-03-24T21:57:52.705 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.705 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.705 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:57:52.706 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.191966+0000", 2021-03-24T21:57:52.706 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00053557599999999996 2021-03-24T21:57:52.706 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.706 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.707 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:57:52.707 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.192502+0000", 2021-03-24T21:57:52.707 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000112501 2021-03-24T21:57:52.707 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.707 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.708 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T21:57:52.708 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.192614+0000", 2021-03-24T21:57:52.708 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012738300000000001 2021-03-24T21:57:52.708 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.709 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.709 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:57:52.709 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.192742+0000", 2021-03-24T21:57:52.709 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.02147785 2021-03-24T21:57:52.709 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:57:52.710 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:57:52.710 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:57:52.710 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.710 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.711 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:256 7.1 7:8c1d349c:::smithi12556470-48:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e113)", 2021-03-24T21:57:52.711 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:54:42.171334+0000", 2021-03-24T21:57:52.711 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 190.39083520899999, 2021-03-24T21:57:52.711 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021795500999999998, 2021-03-24T21:57:52.711 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:57:52.712 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T21:57:52.712 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:57:52.712 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T21:57:52.712 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:57:52.713 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 256 2021-03-24T21:57:52.713 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.713 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:57:52.713 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.714 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:57:52.714 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.171334+0000", 2021-03-24T21:57:52.714 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:57:52.714 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.715 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.715 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:57:52.715 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.171334+0000", 2021-03-24T21:57:52.715 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2021-03-24T21:57:52.716 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.716 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.716 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:57:52.716 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.171333+0000", 2021-03-24T21:57:52.716 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4240000000000003e-06 2021-03-24T21:57:52.717 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.717 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.717 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:57:52.717 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.171337+0000", 2021-03-24T21:57:52.717 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1760000000000004e-06 2021-03-24T21:57:52.718 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.718 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.718 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:57:52.718 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.171344+0000", 2021-03-24T21:57:52.719 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.8359999999999992e-06 2021-03-24T21:57:52.719 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.719 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.719 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:57:52.719 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.171352+0000", 2021-03-24T21:57:52.720 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00043602099999999998 2021-03-24T21:57:52.720 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.720 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.720 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:57:52.721 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.171788+0000", 2021-03-24T21:57:52.721 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000111569 2021-03-24T21:57:52.721 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.721 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.721 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for blocked object", 2021-03-24T21:57:52.722 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.171900+0000", 2021-03-24T21:57:52.722 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.016952834 2021-03-24T21:57:52.722 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.722 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.722 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:57:52.723 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.188853+0000", 2021-03-24T21:57:52.723 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000182533 2021-03-24T21:57:52.723 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.723 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.724 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for rw locks", 2021-03-24T21:57:52.724 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.189035+0000", 2021-03-24T21:57:52.724 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0037762500000000001 2021-03-24T21:57:52.724 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.724 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.725 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:57:52.725 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.192812+0000", 2021-03-24T21:57:52.725 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013949000000000001 2021-03-24T21:57:52.725 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.726 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.726 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:57:52.726 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.192951+0000", 2021-03-24T21:57:52.726 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000178436 2021-03-24T21:57:52.726 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.727 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.727 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:57:52.727 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:42.193129+0000", 2021-03-24T21:57:52.727 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021795500999999998 2021-03-24T21:57:52.727 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:57:52.728 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:57:52.728 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:57:52.728 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.728 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.729 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:434 7.1 7:86662589:::smithi12556470-19:head [delete] snapc 10=[10,f] ondisk+write+peerstat_old+known_if_redirected e123)", 2021-03-24T21:57:52.729 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:54:52.102695+0000", 2021-03-24T21:57:52.729 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 180.45947373499999, 2021-03-24T21:57:52.729 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028155863999999999, 2021-03-24T21:57:52.730 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:57:52.730 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:57:52.730 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:57:52.730 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T21:57:52.730 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:57:52.731 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 434 2021-03-24T21:57:52.731 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.731 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:57:52.731 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.732 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:57:52.732 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:52.102695+0000", 2021-03-24T21:57:52.732 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:57:52.732 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.732 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.733 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:57:52.733 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:52.102695+0000", 2021-03-24T21:57:52.733 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999957 2021-03-24T21:57:52.733 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.733 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.734 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:57:52.734 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:52.102691+0000", 2021-03-24T21:57:52.734 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3886e-05 2021-03-24T21:57:52.734 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.735 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.735 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:57:52.735 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:52.102725+0000", 2021-03-24T21:57:52.735 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5333000000000002e-05 2021-03-24T21:57:52.735 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.736 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.736 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:57:52.736 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:52.102760+0000", 2021-03-24T21:57:52.736 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9732e-05 2021-03-24T21:57:52.736 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.737 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.737 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:57:52.737 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:52.102780+0000", 2021-03-24T21:57:52.737 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00017896599999999999 2021-03-24T21:57:52.737 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.738 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.738 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:57:52.738 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:52.102959+0000", 2021-03-24T21:57:52.738 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00027032400000000001 2021-03-24T21:57:52.739 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.739 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.739 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:57:52.739 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:52.103229+0000", 2021-03-24T21:57:52.740 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000313553 2021-03-24T21:57:52.740 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.740 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.740 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 0,3", 2021-03-24T21:57:52.741 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:52.103543+0000", 2021-03-24T21:57:52.741 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0010644649999999999 2021-03-24T21:57:52.741 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.741 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.741 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T21:57:52.742 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:52.104608+0000", 2021-03-24T21:57:52.742 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023705747999999999 2021-03-24T21:57:52.742 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.742 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.743 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:57:52.743 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:52.128313+0000", 2021-03-24T21:57:52.743 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0024245909999999998 2021-03-24T21:57:52.743 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.743 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.744 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:57:52.744 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:52.130738+0000", 2021-03-24T21:57:52.744 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.4083000000000003e-05 2021-03-24T21:57:52.744 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.744 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.745 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T21:57:52.745 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:52.130802+0000", 2021-03-24T21:57:52.745 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9373000000000003e-05 2021-03-24T21:57:52.745 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.746 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.746 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:57:52.746 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:52.130851+0000", 2021-03-24T21:57:52.746 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028155863999999999 2021-03-24T21:57:52.746 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:57:52.747 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:57:52.747 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:57:52.747 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.747 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.747 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:622 7.1 7:86662589:::smithi12556470-19:head [append 0~0] snapc 20=[20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e140)", 2021-03-24T21:57:52.748 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:55:09.349921+0000", 2021-03-24T21:57:52.748 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 163.212247859, 2021-03-24T21:57:52.748 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.020822875000000001, 2021-03-24T21:57:52.748 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:57:52.748 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:57:52.749 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:57:52.749 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T21:57:52.749 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:57:52.749 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 622 2021-03-24T21:57:52.750 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.750 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:57:52.750 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.750 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:57:52.751 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:09.349921+0000", 2021-03-24T21:57:52.751 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:57:52.751 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.751 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.751 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:57:52.752 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:09.349921+0000", 2021-03-24T21:57:52.752 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999876 2021-03-24T21:57:52.752 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.752 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.753 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:57:52.753 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:09.349909+0000", 2021-03-24T21:57:52.753 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0216e-05 2021-03-24T21:57:52.753 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.753 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.754 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:57:52.754 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:09.349939+0000", 2021-03-24T21:57:52.754 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000848839 2021-03-24T21:57:52.754 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.755 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.755 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:57:52.755 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:09.350788+0000", 2021-03-24T21:57:52.755 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7112000000000002e-05 2021-03-24T21:57:52.755 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.756 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.756 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:57:52.756 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:09.350805+0000", 2021-03-24T21:57:52.756 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0035589469999999998 2021-03-24T21:57:52.757 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.757 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.757 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:57:52.757 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:09.354364+0000", 2021-03-24T21:57:52.758 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00025605199999999999 2021-03-24T21:57:52.758 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.758 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.758 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:57:52.758 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:09.354620+0000", 2021-03-24T21:57:52.759 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00055403199999999998 2021-03-24T21:57:52.759 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.759 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.759 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 0,3", 2021-03-24T21:57:52.759 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:09.355174+0000", 2021-03-24T21:57:52.760 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0054340129999999997 2021-03-24T21:57:52.760 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.760 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.760 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T21:57:52.761 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:09.360608+0000", 2021-03-24T21:57:52.761 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0080520729999999999 2021-03-24T21:57:52.761 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.761 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.762 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:57:52.762 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:09.368660+0000", 2021-03-24T21:57:52.762 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001977211 2021-03-24T21:57:52.762 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.762 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.763 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:57:52.763 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:09.370637+0000", 2021-03-24T21:57:52.763 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.4617999999999997e-05 2021-03-24T21:57:52.763 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.764 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.764 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T21:57:52.764 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:09.370702+0000", 2021-03-24T21:57:52.764 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2231999999999998e-05 2021-03-24T21:57:52.764 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.765 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.765 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:57:52.765 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:09.370744+0000", 2021-03-24T21:57:52.765 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.020822875000000001 2021-03-24T21:57:52.765 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:57:52.766 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:57:52.766 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:57:52.766 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.766 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.767 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:623 7.1 7:86662589:::smithi12556470-19:head [rollback 1c] snapc 20=[20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e140)", 2021-03-24T21:57:52.767 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:55:09.350824+0000", 2021-03-24T21:57:52.767 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 163.21134539900001, 2021-03-24T21:57:52.767 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.020061841, 2021-03-24T21:57:52.767 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:57:52.768 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:57:52.768 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:57:52.768 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T21:57:52.768 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:57:52.769 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 623 2021-03-24T21:57:52.769 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.769 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:57:52.769 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.770 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:57:52.770 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:09.350824+0000", 2021-03-24T21:57:52.770 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:57:52.770 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.770 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.771 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:57:52.771 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:09.350824+0000", 2021-03-24T21:57:52.771 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2021-03-24T21:57:52.771 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.772 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.772 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:57:52.772 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:09.350821+0000", 2021-03-24T21:57:52.772 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1256e-05 2021-03-24T21:57:52.772 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.773 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.773 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:57:52.773 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:09.350832+0000", 2021-03-24T21:57:52.773 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5184000000000001e-05 2021-03-24T21:57:52.774 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.774 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.774 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:57:52.774 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:09.350848+0000", 2021-03-24T21:57:52.774 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0715e-05 2021-03-24T21:57:52.775 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.775 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.775 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:57:52.775 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:09.350858+0000", 2021-03-24T21:57:52.775 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0086539319999999996 2021-03-24T21:57:52.776 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.776 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.776 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:57:52.776 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:09.359512+0000", 2021-03-24T21:57:52.777 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016759899999999999 2021-03-24T21:57:52.777 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.777 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.777 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:57:52.777 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:09.359680+0000", 2021-03-24T21:57:52.778 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00044616099999999999 2021-03-24T21:57:52.778 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.778 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.778 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 0,3", 2021-03-24T21:57:52.779 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:09.360126+0000", 2021-03-24T21:57:52.779 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00068713400000000005 2021-03-24T21:57:52.779 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.779 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.779 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T21:57:52.780 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:09.360813+0000", 2021-03-24T21:57:52.780 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0079851799999999997 2021-03-24T21:57:52.780 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.780 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.781 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:57:52.781 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:09.368798+0000", 2021-03-24T21:57:52.781 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0020075309999999999 2021-03-24T21:57:52.781 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.782 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.782 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:57:52.782 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:09.370806+0000", 2021-03-24T21:57:52.782 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1902000000000001e-05 2021-03-24T21:57:52.783 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.783 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.783 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T21:57:52.783 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:09.370838+0000", 2021-03-24T21:57:52.783 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.7822999999999999e-05 2021-03-24T21:57:52.784 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.784 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.784 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:57:52.784 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:55:09.370886+0000", 2021-03-24T21:57:52.785 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.020061841 2021-03-24T21:57:52.785 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:57:52.785 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:57:52.785 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:57:52.785 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.786 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.786 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:1246 7.1 7:86662589:::smithi12556470-19:head [write 443751~631882] snapc 53=[] ondisk+write+known_if_redirected e191)", 2021-03-24T21:57:52.786 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:56:00.156445+0000", 2021-03-24T21:57:52.786 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 112.40572459800001, 2021-03-24T21:57:52.787 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.060689377000000003, 2021-03-24T21:57:52.787 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:57:52.787 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:57:52.787 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:57:52.788 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T21:57:52.788 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:57:52.788 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 1246 2021-03-24T21:57:52.788 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.788 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:57:52.789 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.789 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:57:52.789 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.156445+0000", 2021-03-24T21:57:52.789 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:57:52.789 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.790 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.790 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:57:52.790 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.156445+0000", 2021-03-24T21:57:52.790 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999952 2021-03-24T21:57:52.791 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.791 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.791 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:57:52.791 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.156440+0000", 2021-03-24T21:57:52.791 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0021389159999999998 2021-03-24T21:57:52.792 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.792 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.792 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:57:52.792 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.158579+0000", 2021-03-24T21:57:52.793 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6568999999999999e-05 2021-03-24T21:57:52.793 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.793 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.793 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:57:52.793 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.158605+0000", 2021-03-24T21:57:52.794 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4353e-05 2021-03-24T21:57:52.794 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.794 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.794 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:57:52.795 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.158620+0000", 2021-03-24T21:57:52.795 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010933 2021-03-24T21:57:52.795 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.795 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.795 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:57:52.796 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.158729+0000", 2021-03-24T21:57:52.796 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016208300000000001 2021-03-24T21:57:52.796 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.796 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.796 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:57:52.797 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.158891+0000", 2021-03-24T21:57:52.797 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00030850199999999999 2021-03-24T21:57:52.797 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.797 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.797 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 0,3", 2021-03-24T21:57:52.798 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.159200+0000", 2021-03-24T21:57:52.798 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.003189473 2021-03-24T21:57:52.798 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.798 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.799 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T21:57:52.799 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.162389+0000", 2021-03-24T21:57:52.799 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.049777185000000002 2021-03-24T21:57:52.799 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.799 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.800 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:57:52.800 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.212166+0000", 2021-03-24T21:57:52.800 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0048629060000000002 2021-03-24T21:57:52.800 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.801 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.801 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:57:52.801 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.217029+0000", 2021-03-24T21:57:52.801 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8270000000000003e-05 2021-03-24T21:57:52.801 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.802 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.802 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T21:57:52.802 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.217088+0000", 2021-03-24T21:57:52.802 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.6387000000000002e-05 2021-03-24T21:57:52.803 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.803 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.803 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:57:52.803 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.217134+0000", 2021-03-24T21:57:52.804 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.060689377000000003 2021-03-24T21:57:52.804 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:57:52.804 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:57:52.804 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:57:52.804 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.805 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.805 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:1247 7.1 7:86662589:::smithi12556470-19:head [write 1810907~765058] snapc 53=[] ondisk+write+known_if_redirected e191)", 2021-03-24T21:57:52.805 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:56:00.164197+0000", 2021-03-24T21:57:52.805 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 112.397972487, 2021-03-24T21:57:52.806 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.054173828, 2021-03-24T21:57:52.806 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:57:52.806 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:57:52.806 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:57:52.807 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T21:57:52.807 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:57:52.807 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 1247 2021-03-24T21:57:52.807 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.808 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:57:52.808 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.808 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:57:52.808 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.164197+0000", 2021-03-24T21:57:52.808 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:57:52.809 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.809 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.809 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:57:52.809 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.164197+0000", 2021-03-24T21:57:52.809 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999962 2021-03-24T21:57:52.810 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.810 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.810 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:57:52.810 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.164193+0000", 2021-03-24T21:57:52.811 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011806989999999999 2021-03-24T21:57:52.811 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.811 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.811 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:57:52.811 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.165374+0000", 2021-03-24T21:57:52.812 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0941000000000001e-05 2021-03-24T21:57:52.812 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.812 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.812 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:57:52.812 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.165405+0000", 2021-03-24T21:57:52.812 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3784999999999999e-05 2021-03-24T21:57:52.813 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.813 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.813 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:57:52.813 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.165418+0000", 2021-03-24T21:57:52.813 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000101259 2021-03-24T21:57:52.814 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.814 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.814 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:57:52.814 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.165520+0000", 2021-03-24T21:57:52.815 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015882 2021-03-24T21:57:52.815 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.815 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.815 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:57:52.815 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.165678+0000", 2021-03-24T21:57:52.816 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000318105 2021-03-24T21:57:52.816 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.816 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.816 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 0,3", 2021-03-24T21:57:52.816 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.165996+0000", 2021-03-24T21:57:52.817 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0042140550000000004 2021-03-24T21:57:52.817 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.817 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.817 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T21:57:52.817 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.170211+0000", 2021-03-24T21:57:52.818 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.042059725999999999 2021-03-24T21:57:52.818 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.818 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.818 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:57:52.819 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.212270+0000", 2021-03-24T21:57:52.819 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0060031010000000003 2021-03-24T21:57:52.819 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.819 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.819 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:57:52.820 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.218273+0000", 2021-03-24T21:57:52.820 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5608e-05 2021-03-24T21:57:52.820 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.820 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.820 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T21:57:52.821 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.218329+0000", 2021-03-24T21:57:52.821 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1533999999999999e-05 2021-03-24T21:57:52.821 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.821 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.821 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:57:52.822 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.218371+0000", 2021-03-24T21:57:52.822 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.054173828 2021-03-24T21:57:52.822 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:57:52.822 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:57:52.823 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:57:52.823 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.823 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.823 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:1248 7.1 7:86662589:::smithi12556470-19:head [write 3067461~176290] snapc 53=[] ondisk+write+known_if_redirected e191)", 2021-03-24T21:57:52.823 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:56:00.167908+0000", 2021-03-24T21:57:52.824 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 112.394260927, 2021-03-24T21:57:52.824 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.050562023999999997, 2021-03-24T21:57:52.824 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:57:52.824 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:57:52.824 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:57:52.825 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T21:57:52.825 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:57:52.825 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 1248 2021-03-24T21:57:52.825 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.825 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:57:52.826 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.826 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:57:52.826 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.167908+0000", 2021-03-24T21:57:52.826 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:57:52.827 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.827 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.827 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:57:52.827 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.167908+0000", 2021-03-24T21:57:52.827 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999971 2021-03-24T21:57:52.827 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.828 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.828 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:57:52.828 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.167905+0000", 2021-03-24T21:57:52.828 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015547299999999999 2021-03-24T21:57:52.828 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.829 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.829 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:57:52.829 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168061+0000", 2021-03-24T21:57:52.829 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9348000000000001e-05 2021-03-24T21:57:52.829 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.830 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.830 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:57:52.830 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168090+0000", 2021-03-24T21:57:52.830 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1722000000000001e-05 2021-03-24T21:57:52.830 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.831 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.831 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:57:52.831 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168102+0000", 2021-03-24T21:57:52.831 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00022216099999999999 2021-03-24T21:57:52.831 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.832 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.832 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:57:52.832 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168324+0000", 2021-03-24T21:57:52.832 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013274000000000001 2021-03-24T21:57:52.832 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.833 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.833 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:57:52.833 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168457+0000", 2021-03-24T21:57:52.833 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000219247 2021-03-24T21:57:52.833 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.834 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.834 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 0,3", 2021-03-24T21:57:52.834 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168676+0000", 2021-03-24T21:57:52.834 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0015439570000000001 2021-03-24T21:57:52.835 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.835 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.835 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T21:57:52.835 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.170220+0000", 2021-03-24T21:57:52.835 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.042126005000000001 2021-03-24T21:57:52.836 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.836 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.836 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:57:52.836 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.212346+0000", 2021-03-24T21:57:52.836 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0060575079999999996 2021-03-24T21:57:52.837 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.837 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.837 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:57:52.837 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.218403+0000", 2021-03-24T21:57:52.837 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7947000000000001e-05 2021-03-24T21:57:52.838 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.838 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.838 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T21:57:52.838 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.218441+0000", 2021-03-24T21:57:52.839 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8948000000000001e-05 2021-03-24T21:57:52.839 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.839 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.839 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:57:52.840 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.218470+0000", 2021-03-24T21:57:52.840 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.050562023999999997 2021-03-24T21:57:52.840 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:57:52.840 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:57:52.840 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:57:52.841 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.841 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.841 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:1249 7.1 7:86662589:::smithi12556470-19:head [setxattr _header (57) in=7b,truncate 3243751] snapc 53=[] ondisk+write+known_if_redirected e191)", 2021-03-24T21:57:52.841 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:56:00.168119+0000", 2021-03-24T21:57:52.841 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 112.39405012899999, 2021-03-24T21:57:52.842 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.052051300000000002, 2021-03-24T21:57:52.842 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:57:52.842 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:57:52.842 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:57:52.842 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T21:57:52.843 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:57:52.843 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 1249 2021-03-24T21:57:52.843 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.843 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:57:52.843 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.844 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:57:52.844 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168119+0000", 2021-03-24T21:57:52.844 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:57:52.844 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.844 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.845 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:57:52.845 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168119+0000", 2021-03-24T21:57:52.845 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2021-03-24T21:57:52.845 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.845 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.846 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:57:52.846 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168118+0000", 2021-03-24T21:57:52.846 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.0740000000000004e-06 2021-03-24T21:57:52.846 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.846 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.847 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:57:52.847 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168125+0000", 2021-03-24T21:57:52.847 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2374e-05 2021-03-24T21:57:52.847 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.847 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.848 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:57:52.848 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168137+0000", 2021-03-24T21:57:52.848 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.5490000000000003e-06 2021-03-24T21:57:52.848 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.848 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.849 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:57:52.849 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168144+0000", 2021-03-24T21:57:52.849 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011986869999999999 2021-03-24T21:57:52.849 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.849 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.850 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:57:52.850 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.169342+0000", 2021-03-24T21:57:52.850 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000160089 2021-03-24T21:57:52.850 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.851 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.851 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:57:52.851 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.169502+0000", 2021-03-24T21:57:52.851 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00031500400000000002 2021-03-24T21:57:52.851 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.852 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.852 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 0,3", 2021-03-24T21:57:52.852 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.169817+0000", 2021-03-24T21:57:52.852 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00070204499999999999 2021-03-24T21:57:52.852 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.853 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.853 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T21:57:52.853 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.170519+0000", 2021-03-24T21:57:52.853 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.041887362999999997 2021-03-24T21:57:52.853 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.854 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.854 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:57:52.854 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.212407+0000", 2021-03-24T21:57:52.854 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0076951889999999998 2021-03-24T21:57:52.855 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.855 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.855 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:57:52.855 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.220102+0000", 2021-03-24T21:57:52.855 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3169e-05 2021-03-24T21:57:52.856 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.856 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.856 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T21:57:52.856 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.220135+0000", 2021-03-24T21:57:52.856 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5286000000000001e-05 2021-03-24T21:57:52.857 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.857 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.857 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:57:52.857 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.220170+0000", 2021-03-24T21:57:52.858 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.052051300000000002 2021-03-24T21:57:52.858 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:57:52.858 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:57:52.858 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:57:52.858 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.859 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.859 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:1250 7.1 7:86662589:::smithi12556470-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e191)", 2021-03-24T21:57:52.859 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:56:00.168170+0000", 2021-03-24T21:57:52.859 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 112.393998827, 2021-03-24T21:57:52.860 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.052145710999999997, 2021-03-24T21:57:52.860 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:57:52.860 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T21:57:52.860 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:57:52.861 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T21:57:52.861 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:57:52.861 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 1250 2021-03-24T21:57:52.861 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.861 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:57:52.862 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.862 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:57:52.862 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168170+0000", 2021-03-24T21:57:52.862 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:57:52.862 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.863 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.863 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:57:52.863 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168170+0000", 2021-03-24T21:57:52.863 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2021-03-24T21:57:52.864 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.864 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.864 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:57:52.864 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168169+0000", 2021-03-24T21:57:52.864 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.5979999999999999e-06 2021-03-24T21:57:52.865 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.865 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.865 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:57:52.865 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168174+0000", 2021-03-24T21:57:52.866 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3064000000000001e-05 2021-03-24T21:57:52.866 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.866 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.866 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:57:52.866 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168187+0000", 2021-03-24T21:57:52.867 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9599999999999997e-06 2021-03-24T21:57:52.867 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.867 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.867 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:57:52.868 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168193+0000", 2021-03-24T21:57:52.868 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0021001650000000002 2021-03-24T21:57:52.868 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.868 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.868 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:57:52.869 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.170293+0000", 2021-03-24T21:57:52.869 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00019255100000000001 2021-03-24T21:57:52.869 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.869 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.869 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for rw locks", 2021-03-24T21:57:52.870 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.170485+0000", 2021-03-24T21:57:52.870 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.049706363000000003 2021-03-24T21:57:52.870 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.870 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.871 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:57:52.871 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.220192+0000", 2021-03-24T21:57:52.871 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9202e-05 2021-03-24T21:57:52.871 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.871 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.872 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:57:52.872 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.220251+0000", 2021-03-24T21:57:52.872 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.5007999999999998e-05 2021-03-24T21:57:52.872 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.873 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.873 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:57:52.873 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.220316+0000", 2021-03-24T21:57:52.873 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.052145710999999997 2021-03-24T21:57:52.873 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:57:52.874 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:57:52.874 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:57:52.874 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.874 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.875 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:1768 7.1 7:86662589:::smithi12556470-19:head [copy-from ver 151] snapc 7b=[79,78] ondisk+write+known_if_redirected e231)", 2021-03-24T21:57:52.875 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:56:40.407353+0000", 2021-03-24T21:57:52.875 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 72.154815804999998, 2021-03-24T21:57:52.875 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031070766999999999, 2021-03-24T21:57:52.875 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:57:52.876 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:57:52.876 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:57:52.876 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T21:57:52.876 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:57:52.876 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 1768 2021-03-24T21:57:52.877 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.877 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:57:52.877 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.877 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:57:52.877 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:40.407353+0000", 2021-03-24T21:57:52.878 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:57:52.878 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.878 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.878 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:57:52.879 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:40.407353+0000", 2021-03-24T21:57:52.879 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999948 2021-03-24T21:57:52.879 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.879 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.879 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:57:52.880 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:40.407348+0000", 2021-03-24T21:57:52.880 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2466000000000001e-05 2021-03-24T21:57:52.880 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.880 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.880 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:57:52.881 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:40.407371+0000", 2021-03-24T21:57:52.881 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1378999999999998e-05 2021-03-24T21:57:52.881 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.881 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.882 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:57:52.882 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:40.407412+0000", 2021-03-24T21:57:52.882 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7003e-05 2021-03-24T21:57:52.882 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.883 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.883 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:57:52.883 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:40.407429+0000", 2021-03-24T21:57:52.883 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00017411799999999999 2021-03-24T21:57:52.883 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.884 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.884 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:57:52.884 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:40.407603+0000", 2021-03-24T21:57:52.884 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00020898999999999999 2021-03-24T21:57:52.884 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.885 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.885 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:57:52.885 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:40.407812+0000", 2021-03-24T21:57:52.885 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0092038569999999993 2021-03-24T21:57:52.886 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.886 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.886 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 0,7", 2021-03-24T21:57:52.886 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:40.417016+0000", 2021-03-24T21:57:52.886 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011321641 2021-03-24T21:57:52.887 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.887 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.887 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:57:52.887 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:40.428338+0000", 2021-03-24T21:57:52.888 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0079992770000000008 2021-03-24T21:57:52.888 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.888 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.888 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:57:52.888 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:40.436337+0000", 2021-03-24T21:57:52.889 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0019174560000000001 2021-03-24T21:57:52.889 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.889 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.889 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T21:57:52.890 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:40.438254+0000", 2021-03-24T21:57:52.890 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9479000000000002e-05 2021-03-24T21:57:52.890 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.890 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.890 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T21:57:52.891 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:40.438344+0000", 2021-03-24T21:57:52.891 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0156999999999998e-05 2021-03-24T21:57:52.891 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.891 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.892 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:57:52.892 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:40.438424+0000", 2021-03-24T21:57:52.892 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031070766999999999 2021-03-24T21:57:52.892 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:57:52.892 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:57:52.893 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:57:52.893 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.893 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.893 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:1769 7.1 7:86662589:::smithi12556470-19:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e231)", 2021-03-24T21:57:52.893 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:56:40.407463+0000", 2021-03-24T21:57:52.894 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 72.154706140000002, 2021-03-24T21:57:52.894 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031191511000000002, 2021-03-24T21:57:52.894 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:57:52.894 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T21:57:52.895 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:57:52.895 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T21:57:52.895 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:57:52.895 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 1769 2021-03-24T21:57:52.896 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.896 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:57:52.896 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.896 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:57:52.896 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:40.407463+0000", 2021-03-24T21:57:52.897 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:57:52.897 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.897 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.897 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:57:52.898 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:40.407463+0000", 2021-03-24T21:57:52.898 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2021-03-24T21:57:52.898 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.898 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.899 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:57:52.899 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:40.407461+0000", 2021-03-24T21:57:52.899 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.4439999999999996e-06 2021-03-24T21:57:52.899 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.899 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.900 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:57:52.900 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:40.407468+0000", 2021-03-24T21:57:52.900 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0879999999999999e-05 2021-03-24T21:57:52.900 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.900 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.901 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:57:52.901 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:40.407479+0000", 2021-03-24T21:57:52.901 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.7030000000000006e-06 2021-03-24T21:57:52.901 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.901 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.902 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:57:52.902 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:40.407486+0000", 2021-03-24T21:57:52.902 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00066091399999999997 2021-03-24T21:57:52.902 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.903 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.903 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:57:52.903 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:40.408147+0000", 2021-03-24T21:57:52.903 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015968800000000001 2021-03-24T21:57:52.903 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.904 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.904 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for blocked object", 2021-03-24T21:57:52.904 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:40.408307+0000", 2021-03-24T21:57:52.904 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.019826705999999999 2021-03-24T21:57:52.904 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.905 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.905 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:57:52.905 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:40.428134+0000", 2021-03-24T21:57:52.905 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014998 2021-03-24T21:57:52.906 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.906 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.906 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for rw locks", 2021-03-24T21:57:52.906 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:40.428284+0000", 2021-03-24T21:57:52.906 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.010192896999999999 2021-03-24T21:57:52.907 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.907 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.907 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:57:52.907 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:40.438477+0000", 2021-03-24T21:57:52.908 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9227000000000001e-05 2021-03-24T21:57:52.908 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.908 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.908 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:57:52.908 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:40.438536+0000", 2021-03-24T21:57:52.909 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000118689 2021-03-24T21:57:52.909 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:57:52.909 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:57:52.909 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:57:52.909 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:40.438655+0000", 2021-03-24T21:57:52.910 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031191511000000002 2021-03-24T21:57:52.910 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:57:52.910 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:57:52.910 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:57:52.910 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:57:52.911 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:57:52.911 INFO:teuthology.orchestra.run.smithi142.stdout:} 2021-03-24T21:57:52.911 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:52.963 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:52.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:53.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:57:53.172 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:53.206 INFO:tasks.rados.rados.0.smithi125.stdout:1006: done (2 left) 2021-03-24T21:57:53.206 INFO:tasks.rados.rados.0.smithi125.stdout:1007: read oid 9 snap 96 2021-03-24T21:57:53.206 INFO:tasks.rados.rados.0.smithi125.stdout:1007: expect (ObjNum 349 snap 89 seq_num 349) 2021-03-24T21:57:53.207 INFO:tasks.rados.rados.0.smithi125.stdout:1008: write oid 17 current snap is 98 2021-03-24T21:57:53.207 INFO:tasks.rados.rados.0.smithi125.stdout:1008: seq_num 377 ranges {736239=470170,1728182=571782} 2021-03-24T21:57:53.214 INFO:tasks.rados.rados.0.smithi125.stdout:1008: writing smithi12556470-17 from 736239 to 1206409 tid 1 2021-03-24T21:57:53.222 INFO:tasks.rados.rados.0.smithi125.stdout:1008: writing smithi12556470-17 from 1728182 to 2299964 tid 2 2021-03-24T21:57:53.222 INFO:tasks.rados.rados.0.smithi125.stdout:1009: read oid 47 snap 96 2021-03-24T21:57:53.222 INFO:tasks.rados.rados.0.smithi125.stdout:1009: expect (ObjNum 342 snap 87 seq_num 342) 2021-03-24T21:57:53.222 INFO:tasks.rados.rados.0.smithi125.stdout:1010: delete oid 10 current snap is 98 2021-03-24T21:57:53.223 INFO:tasks.rados.rados.0.smithi125.stdout:1004: finishing write tid 1 to smithi12556470-32 2021-03-24T21:57:53.223 INFO:tasks.rados.rados.0.smithi125.stdout:1004: finishing write tid 2 to smithi12556470-32 2021-03-24T21:57:53.223 INFO:tasks.rados.rados.0.smithi125.stdout:1004: finishing write tid 3 to smithi12556470-32 2021-03-24T21:57:53.223 INFO:tasks.rados.rados.0.smithi125.stdout:1004: finishing write tid 4 to smithi12556470-32 2021-03-24T21:57:53.223 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 32 v 296 (ObjNum 376 snap 98 seq_num 376) dirty exists 2021-03-24T21:57:53.224 INFO:tasks.rados.rados.0.smithi125.stdout:1004: left oid 32 (ObjNum 376 snap 98 seq_num 376) 2021-03-24T21:57:53.224 INFO:tasks.rados.rados.0.smithi125.stdout:1007: expect (ObjNum 349 snap 89 seq_num 349) 2021-03-24T21:57:53.236 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:53.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:53.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:57:53.341 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:53.348 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:57:53.401 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:53.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:53.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:57:53.470 INFO:tasks.rados.rados.0.smithi125.stdout:1009: expect (ObjNum 342 snap 87 seq_num 342) 2021-03-24T21:57:53.505 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:53.509 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:53 smithi142 conmon[20125]: cluster 2021-03-24T21:57:52.183459+0000 mon.a (mon.0) 968 : cluster [DBG] osdmap e302: 8 total, 8 up, 6 in 2021-03-24T21:57:53.509 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:53 smithi142 conmon[20125]: audit 2021-03-24T21:57:52.192699+0000 mon.c (mon.2) 47 : audit [DBG] from='client.? v1:172.21.15.125:0/225308683' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:57:53.509 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:53 smithi142 conmon[20125]: cluster 2021-03-24T21:57:52.248239+0000 osd.2 (osd.2) 57 : cluster [DBG] 4.12 scrub ok 2021-03-24T21:57:53.517 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:53 smithi125 conmon[19936]: cluster 2021-03-24T21:57: 2021-03-24T21:57:53.517 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:53 smithi125 conmon[19936]: 52.183459+0000 mon.a (mon.0) 968 : cluster [DBG] osdmap e302: 8 total, 8 up, 6 in 2021-03-24T21:57:53.517 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:53 smithi125 conmon[19936]: audit 2021-03-24T21:57: 2021-03-24T21:57:53.517 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:53 smithi125 conmon[19936]: 52.192699+0000 mon.c (mon.2) 47 : audit [DBG] from='client.? v1:172.21.15.125:0/225308683' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:57:53.518 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:53 smithi125 conmon[19936]: 2021-03-24T21:57:53.518 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:53 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:57:53.518 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:53 smithi125 conmon[19936]: 24T21:57:52.248239+0000 osd.2 (osd.2 2021-03-24T21:57:53.518 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:53 smithi125 conmon[19936]: ) 57 : cluster [DBG] 4.12 scrub ok 2021-03-24T21:57:53.519 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:53 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:57:53.519 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:53 smithi125 conmon[24927]: T21:57:52.183459+0000 mon.a (mon.0) 968 : cluster [DBG] 2021-03-24T21:57:53.519 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:53 smithi125 conmon[24927]: osdmap e302: 8 total, 8 up, 6 in 2021-03-24T21:57:53.520 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:53 smithi125 conmon[24927]: audit 2021- 2021-03-24T21:57:53.520 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:53 smithi125 conmon[24927]: 03-24T21:57:52 2021-03-24T21:57:53.520 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:53 smithi125 conmon[24927]: .192699+0000 mon.c (mon.2 2021-03-24T21:57:53.520 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:53 smithi125 conmon[24927]: ) 47 : audit [DBG] from='client.? v1:172.21.15.125:0/225308683' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:57:53.521 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:53 smithi125 conmon[24927]: 2021-03-24T21:57:53.521 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:53 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:57:53.521 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:53 smithi125 conmon[24927]: 24T21:57:52.248239 2021-03-24T21:57:53.521 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:53 smithi125 conmon[24927]: +0000 osd.2 (osd.2) 2021-03-24T21:57:53.522 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:53 smithi125 conmon[24927]: 57 : cluster [DBG] 4.12 scrub ok 2021-03-24T21:57:53.539 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.5 dump_ops_in_flight 2021-03-24T21:57:53.556 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:53.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:53.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:57:53.663 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:53.713 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:53.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:53.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:57:53.758 INFO:tasks.rados.rados.0.smithi125.stdout:1008: finishing write tid 1 to smithi12556470-17 2021-03-24T21:57:53.758 INFO:tasks.rados.rados.0.smithi125.stdout:1008: finishing write tid 2 to smithi12556470-17 2021-03-24T21:57:53.758 INFO:tasks.rados.rados.0.smithi125.stdout:1008: finishing write tid 3 to smithi12556470-17 2021-03-24T21:57:53.759 INFO:tasks.rados.rados.0.smithi125.stdout:1008: finishing write tid 4 to smithi12556470-17 2021-03-24T21:57:53.759 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 17 v 291 (ObjNum 377 snap 98 seq_num 377) dirty exists 2021-03-24T21:57:53.759 INFO:tasks.rados.rados.0.smithi125.stdout:1008: left oid 17 (ObjNum 377 snap 98 seq_num 377) 2021-03-24T21:57:53.759 INFO:tasks.rados.rados.0.smithi125.stdout:1004: done (5 left) 2021-03-24T21:57:53.760 INFO:tasks.rados.rados.0.smithi125.stdout:1005: done (4 left) 2021-03-24T21:57:53.760 INFO:tasks.rados.rados.0.smithi125.stdout:1007: done (3 left) 2021-03-24T21:57:53.760 INFO:tasks.rados.rados.0.smithi125.stdout:1008: done (2 left) 2021-03-24T21:57:53.760 INFO:tasks.rados.rados.0.smithi125.stdout:1009: done (1 left) 2021-03-24T21:57:53.761 INFO:tasks.rados.rados.0.smithi125.stdout:1010: done (0 left) 2021-03-24T21:57:53.761 INFO:tasks.rados.rados.0.smithi125.stdout:1011: rollback oid 15 current snap is 98 2021-03-24T21:57:53.761 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 15 to 96 2021-03-24T21:57:53.761 INFO:tasks.rados.rados.0.smithi125.stdout:1012: copy_from oid 32 from oid 26 current snap is 98 2021-03-24T21:57:53.761 INFO:tasks.rados.rados.0.smithi125.stdout:1013: snap_create 2021-03-24T21:57:53.762 INFO:tasks.rados.rados.0.smithi125.stdout:1011: finishing rollback tid 0 to smithi12556470-15 2021-03-24T21:57:53.763 INFO:tasks.rados.rados.0.smithi125.stdout:1011: finishing rollback tid 1 to smithi12556470-15 2021-03-24T21:57:53.763 INFO:tasks.rados.rados.0.smithi125.stdout:1011: finishing rollback tid 2 to smithi12556470-15 2021-03-24T21:57:53.763 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 15 v 312 (ObjNum 360 snap 92 seq_num 360) dirty exists 2021-03-24T21:57:53.774 INFO:tasks.rados.rados.0.smithi125.stdout:1012: finishing copy_from to smithi12556470-32 2021-03-24T21:57:53.774 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 32 v 159 (ObjNum 370 snap 95 seq_num 370) dirty exists 2021-03-24T21:57:53.774 INFO:tasks.rados.rados.0.smithi125.stdout:1012: finishing copy_from racing read to smithi12556470-32 2021-03-24T21:57:53.782 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-24T21:57:53.818 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:53.842 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T21:57:53.870 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:53.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:53.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:57:53.974 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:54.019 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:54.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:54.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:57:54.124 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:54.185 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:54.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:54.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:57:54.215 INFO:tasks.rados.rados.0.smithi125.stdout:1011: done (2 left) 2021-03-24T21:57:54.215 INFO:tasks.rados.rados.0.smithi125.stdout:1012: done (1 left) 2021-03-24T21:57:54.216 INFO:tasks.rados.rados.0.smithi125.stdout:1013: done (0 left) 2021-03-24T21:57:54.216 INFO:tasks.rados.rados.0.smithi125.stdout:1014: rollback oid 37 current snap is 99 2021-03-24T21:57:54.216 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 37 to 96 2021-03-24T21:57:54.216 INFO:tasks.rados.rados.0.smithi125.stdout:1015: read oid 32 snap -1 2021-03-24T21:57:54.216 INFO:tasks.rados.rados.0.smithi125.stdout:1015: expect (ObjNum 370 snap 95 seq_num 370) 2021-03-24T21:57:54.217 INFO:tasks.rados.rados.0.smithi125.stdout:1016: rollback oid 25 current snap is 99 2021-03-24T21:57:54.217 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 25 to 96 2021-03-24T21:57:54.217 INFO:tasks.rados.rados.0.smithi125.stdout:1017: write (excl) oid 29 current snap is 99 2021-03-24T21:57:54.217 INFO:tasks.rados.rados.0.smithi125.stdout:1017: seq_num 379 ranges {769853=620668,1845146=524707} 2021-03-24T21:57:54.226 INFO:tasks.rados.rados.0.smithi125.stdout:1017: writing smithi12556470-29 from 769853 to 1390521 tid 1 2021-03-24T21:57:54.232 INFO:tasks.rados.rados.0.smithi125.stdout:1017: writing smithi12556470-29 from 1845146 to 2369853 tid 2 2021-03-24T21:57:54.232 INFO:tasks.rados.rados.0.smithi125.stdout:1018: write oid 20 current snap is 99 2021-03-24T21:57:54.232 INFO:tasks.rados.rados.0.smithi125.stdout:1018: seq_num 380 ranges {786660=695917,1903628=435462} 2021-03-24T21:57:54.240 INFO:tasks.rados.rados.0.smithi125.stdout:1018: writing smithi12556470-20 from 786660 to 1482577 tid 1 2021-03-24T21:57:54.246 INFO:tasks.rados.rados.0.smithi125.stdout:1018: writing smithi12556470-20 from 1903628 to 2339090 tid 2 2021-03-24T21:57:54.246 INFO:tasks.rados.rados.0.smithi125.stdout:1019: delete oid 26 current snap is 99 2021-03-24T21:57:54.246 INFO:tasks.rados.rados.0.smithi125.stdout:1014: finishing rollback tid 1 to smithi12556470-37 2021-03-24T21:57:54.247 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 37 v 0 (ObjNum 1116445872 snap 21916 seq_num 3019901136) dirty dne 2021-03-24T21:57:54.247 INFO:tasks.rados.rados.0.smithi125.stdout:1016: finishing rollback tid 0 to smithi12556470-25 2021-03-24T21:57:54.247 INFO:tasks.rados.rados.0.smithi125.stdout:1016: finishing rollback tid 1 to smithi12556470-25 2021-03-24T21:57:54.247 INFO:tasks.rados.rados.0.smithi125.stdout:1015: expect (ObjNum 370 snap 95 seq_num 370) 2021-03-24T21:57:54.291 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:54.338 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:54 smithi142 conmon[20125]: cluster 2021-03-24T21:57:52.376470+0000 2021-03-24T21:57:54.338 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:54 smithi142 conmon[20125]: osd.3 (osd.3) 73 : cluster [DBG] 5.17 scrub ok 2021-03-24T21:57:54.338 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:54 smithi142 conmon[20125]: cluster 2021-03-24T21:57:52.783739+0000 mgr.y (mgr.14144) 419 : cluster [DBG] 2021-03-24T21:57:54.339 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:54 smithi142 conmon[20125]: pgmap v616: 168 pgs: 2 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 164 active+clean; 114 MiB data, 453 MiB used, 536 GiB / 536 GiB avail; 3.9 MiB/s rd, 1.1 MiB/s wr, 7 op/s 2021-03-24T21:57:54.339 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:54 smithi142 conmon[20125]: cluster 2021-03-24T21:57:53.013121+0000 osd.5 (osd.5) 75 : 2021-03-24T21:57:54.339 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:54 smithi142 conmon[20125]: cluster [DBG] 3.5 scrub ok 2021-03-24T21:57:54.339 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:54 smithi142 conmon[20125]: cluster 2021-03-24T21:57:53. 2021-03-24T21:57:54.340 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:54 smithi142 conmon[20125]: 093318+0000 osd.7 (osd.7) 68 : cluster [DBG] 5.10 scrub ok 2021-03-24T21:57:54.340 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:54 smithi142 conmon[20125]: cluster 2021-03-24T21:57:53.201894+0000 mon.a (mon.0 2021-03-24T21:57:54.340 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:54 smithi142 conmon[20125]: ) 969 : cluster [DBG] osdmap e303: 8 total, 8 up, 6 in 2021-03-24T21:57:54.340 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:54 smithi142 conmon[20125]: cluster 2021-03-24T21:57:54.340 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:54 smithi142 conmon[20125]: 2021-03-24T21:57:53.669593+0000 osd.2 (osd.2) 58 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:57:54.341 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:54 smithi142 conmon[20125]: 2021-03-24T21:57:54.341 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:54 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:57:54.341 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:54 smithi142 conmon[20125]: 57:53.669899+0000 osd.2 (osd.2) 59 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:57:54.350 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:54.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:54.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:57:54.379 INFO:tasks.rados.rados.0.smithi125.stdout:1016: finishing rollback tid 2 to smithi12556470-25 2021-03-24T21:57:54.379 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 25 v 208 (ObjNum 330 snap 83 seq_num 330) dirty exists 2021-03-24T21:57:54.379 INFO:tasks.rados.rados.0.smithi125.stdout:1017: finishing write tid 1 to smithi12556470-29 2021-03-24T21:57:54.380 INFO:tasks.rados.rados.0.smithi125.stdout:1017: finishing write tid 2 to smithi12556470-29 2021-03-24T21:57:54.380 INFO:tasks.rados.rados.0.smithi125.stdout:1014: done (5 left) 2021-03-24T21:57:54.380 INFO:tasks.rados.rados.0.smithi125.stdout:1015: done (4 left) 2021-03-24T21:57:54.380 INFO:tasks.rados.rados.0.smithi125.stdout:1016: done (3 left) 2021-03-24T21:57:54.381 INFO:tasks.rados.rados.0.smithi125.stdout:1019: done (2 left) 2021-03-24T21:57:54.381 INFO:tasks.rados.rados.0.smithi125.stdout:1020: write oid 50 current snap is 99 2021-03-24T21:57:54.381 INFO:tasks.rados.rados.0.smithi125.stdout:1017: finishing write tid 3 to smithi12556470-29 2021-03-24T21:57:54.381 INFO:tasks.rados.rados.0.smithi125.stdout:1017: finishing write tid 4 to smithi12556470-29 2021-03-24T21:57:54.382 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 29 v 152 (ObjNum 379 snap 99 seq_num 379) dirty exists 2021-03-24T21:57:54.382 INFO:tasks.rados.rados.0.smithi125.stdout:1017: left oid 29 (ObjNum 379 snap 99 seq_num 379) 2021-03-24T21:57:54.382 INFO:tasks.rados.rados.0.smithi125.stdout:1018: finishing write tid 1 to smithi12556470-20 2021-03-24T21:57:54.382 INFO:tasks.rados.rados.0.smithi125.stdout:1018: finishing write tid 2 to smithi12556470-20 2021-03-24T21:57:54.382 INFO:tasks.rados.rados.0.smithi125.stdout:1018: finishing write tid 3 to smithi12556470-20 2021-03-24T21:57:54.383 INFO:tasks.rados.rados.0.smithi125.stdout:1018: finishing write tid 4 to smithi12556470-20 2021-03-24T21:57:54.383 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 20 v 212 (ObjNum 380 snap 99 seq_num 380) dirty exists 2021-03-24T21:57:54.383 INFO:tasks.rados.rados.0.smithi125.stdout:1018: left oid 20 (ObjNum 380 snap 99 seq_num 380) 2021-03-24T21:57:54.383 INFO:tasks.rados.rados.0.smithi125.stdout:1020: seq_num 382 ranges {420274=562768,1620592=439131} 2021-03-24T21:57:54.390 INFO:tasks.rados.rados.0.smithi125.stdout:1020: writing smithi12556470-50 from 420274 to 983042 tid 1 2021-03-24T21:57:54.399 INFO:tasks.rados.rados.0.smithi125.stdout:1020: writing smithi12556470-50 from 1620592 to 2059723 tid 2 2021-03-24T21:57:54.399 INFO:tasks.rados.rados.0.smithi125.stdout:1017: done (2 left) 2021-03-24T21:57:54.399 INFO:tasks.rados.rados.0.smithi125.stdout:1018: done (1 left) 2021-03-24T21:57:54.400 INFO:tasks.rados.rados.0.smithi125.stdout:1021: copy_from oid 11 from oid 39 current snap is 99 2021-03-24T21:57:54.400 INFO:tasks.rados.rados.0.smithi125.stdout:1020: finishing write tid 1 to smithi12556470-50 2021-03-24T21:57:54.400 INFO:tasks.rados.rados.0.smithi125.stdout:1022: copy_from oid 17 from oid 5 current snap is 99 2021-03-24T21:57:54.400 INFO:tasks.rados.rados.0.smithi125.stdout:1023: rollback oid 29 current snap is 99 2021-03-24T21:57:54.400 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 29 to 98 2021-03-24T21:57:54.401 INFO:tasks.rados.rados.0.smithi125.stdout:1024: write (excl) oid 10 current snap is 99 2021-03-24T21:57:54.401 INFO:tasks.rados.rados.0.smithi125.stdout:1024: seq_num 383 ranges {437081=638017,1679074=582789} 2021-03-24T21:57:54.404 INFO:tasks.rados.rados.0.smithi125.stdout:1024: writing smithi12556470-10 from 437081 to 1075098 tid 1 2021-03-24T21:57:54.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:54 smithi125 conmon[19936]: cluster 2021-03-24T21:57:52.376470+0000 2021-03-24T21:57:54.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:54 smithi125 conmon[19936]: osd.3 (osd.3) 73 : cluster [DBG] 5.17 scrub ok 2021-03-24T21:57:54.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:54 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:57:54.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:54 smithi125 conmon[19936]: 57:52.783739+0000 mgr.y (mgr.14144) 419 : cluster [DBG] pgmap v616: 168 pgs: 2 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 164 active+clean; 114 MiB data, 453 MiB used, 536 GiB / 536 GiB avail; 3.9 MiB/s rd, 1.1 MiB/s wr, 7 op/s 2021-03-24T21:57:54.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:54 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T21:57:54.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:54 smithi125 conmon[19936]: T21:57:53.013121+0000 osd.5 (osd. 2021-03-24T21:57:54.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:54 smithi125 conmon[19936]: 5) 75 : cluster [DBG] 3.5 scrub ok 2021-03-24T21:57:54.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:54 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:57:54.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:54 smithi125 conmon[19936]: 03-24T21:57:53.093318+0000 osd.7 ( 2021-03-24T21:57:54.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:54 smithi125 conmon[19936]: osd.7) 68 : cluster [DBG] 5.10 scrub ok 2021-03-24T21:57:54.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:54 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:57:54.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:54 smithi125 conmon[19936]: -03-24T21:57:53.201894+0000 mon.a ( 2021-03-24T21:57:54.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:54 smithi125 conmon[19936]: mon.0) 969 : cluster [DBG] osdmap e303: 8 total, 8 up, 6 in 2021-03-24T21:57:54.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:54 smithi125 conmon[19936]: cluster 2021-03-24T21:57:54.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:54 smithi125 conmon[19936]: 2021-03-24T21:57:53.669593+0000 osd.2 ( 2021-03-24T21:57:54.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:54 smithi125 conmon[19936]: osd.2) 58 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:57:54.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:54 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T21:57:54.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:54 smithi125 conmon[19936]: T21:57:53.669899+0000 osd.2 (osd.2) 59 : cluster [DBG] 2021-03-24T21:57:54.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:54 smithi125 conmon[19936]: purged_snaps scrub ok 2021-03-24T21:57:54.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:54 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:57:54.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:54 smithi125 conmon[24927]: 21:57:52.376470+0000 osd.3 (osd.3) 73 : cluster [DBG] 5.17 scrub ok 2021-03-24T21:57:54.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:54 smithi125 conmon[24927]: cluster 2021-03-24T21:57:54.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:54 smithi125 conmon[24927]: 2021-03 2021-03-24T21:57:54.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:54 smithi125 conmon[24927]: -24T21:57:52.783739 2021-03-24T21:57:54.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:54 smithi125 conmon[24927]: +0000 mgr.y (mgr.14144) 2021-03-24T21:57:54.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:54 smithi125 conmon[24927]: 419 : cluster [DBG] pgmap v616: 168 pgs: 2 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 164 active+clean; 114 MiB data, 453 MiB used, 536 GiB / 536 GiB avail; 3.9 MiB/s rd, 1.1 MiB/s wr, 7 op/s 2021-03-24T21:57:54.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:54 smithi125 conmon[24927]: cluster 2021-03-24T21:57:54.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:54 smithi125 conmon[24927]: 2021-03-24T21: 2021-03-24T21:57:54.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:54 smithi125 conmon[24927]: 57:53.013121+0000 osd.5 ( 2021-03-24T21:57:54.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:54 smithi125 conmon[24927]: osd.5) 75 : cluster 2021-03-24T21:57:54.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:54 smithi125 conmon[24927]: [DBG] 3.5 scrub ok 2021-03-24T21:57:54.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:54 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:57:54.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:54 smithi125 conmon[24927]: -24T21:57:53 2021-03-24T21:57:54.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:54 smithi125 conmon[24927]: .093318+0000 osd.7 (osd.7 2021-03-24T21:57:54.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:54 smithi125 conmon[24927]: ) 68 : cluster [DBG] 5.10 scrub ok 2021-03-24T21:57:54.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:54 smithi125 conmon[24927]: 2021-03-24T21:57:54.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:54 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:57:54.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:54 smithi125 conmon[24927]: 21:57:53.201894+0000 2021-03-24T21:57:54.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:54 smithi125 conmon[24927]: mon.a (mon.0) 969 : 2021-03-24T21:57:54.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:54 smithi125 conmon[24927]: cluster [DBG] osdmap e303: 8 total, 8 up, 6 in 2021-03-24T21:57:54.420 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:54 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:57:54.420 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:54 smithi125 conmon[24927]: -03-24T21:57: 2021-03-24T21:57:54.420 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:54 smithi125 conmon[24927]: 53.669593+0000 osd.2 (osd 2021-03-24T21:57:54.420 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:54 smithi125 conmon[24927]: .2) 58 : cluster [DBG] 2021-03-24T21:57:54.420 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:54 smithi125 conmon[24927]: purged_snaps scrub starts 2021-03-24T21:57:54.421 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:54 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:57:54.421 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:54 smithi125 conmon[24927]: T21:57:53.669899 2021-03-24T21:57:54.421 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:54 smithi125 conmon[24927]: +0000 osd.2 (osd.2) 59 2021-03-24T21:57:54.421 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:54 smithi125 conmon[24927]: : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:57:54.423 INFO:tasks.rados.rados.0.smithi125.stdout:1024: writing smithi12556470-10 from 1679074 to 2261863 tid 2 2021-03-24T21:57:54.424 INFO:tasks.rados.rados.0.smithi125.stdout:1025: rollback oid 44 current snap is 99 2021-03-24T21:57:54.424 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 44 to 98 2021-03-24T21:57:54.424 INFO:tasks.rados.rados.0.smithi125.stdout:1022: finishing copy_from to smithi12556470-17 2021-03-24T21:57:54.424 INFO:tasks.rados.rados.0.smithi125.stdout:1022: got expected ENOENT (src dne) 2021-03-24T21:57:54.425 INFO:tasks.rados.rados.0.smithi125.stdout:1022: finishing copy_from racing read to smithi12556470-17 2021-03-24T21:57:54.425 INFO:tasks.rados.rados.0.smithi125.stdout:1020: finishing write tid 2 to smithi12556470-50 2021-03-24T21:57:54.425 INFO:tasks.rados.rados.0.smithi125.stdout:1023: finishing rollback tid 0 to smithi12556470-29 2021-03-24T21:57:54.425 INFO:tasks.rados.rados.0.smithi125.stdout:1020: finishing write tid 3 to smithi12556470-50 2021-03-24T21:57:54.425 INFO:tasks.rados.rados.0.smithi125.stdout:1020: finishing write tid 4 to smithi12556470-50 2021-03-24T21:57:54.426 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 50 v 216 (ObjNum 382 snap 99 seq_num 382) dirty exists 2021-03-24T21:57:54.426 INFO:tasks.rados.rados.0.smithi125.stdout:1020: left oid 50 (ObjNum 382 snap 99 seq_num 382) 2021-03-24T21:57:54.426 INFO:tasks.rados.rados.0.smithi125.stdout:1023: finishing rollback tid 1 to smithi12556470-29 2021-03-24T21:57:54.426 INFO:tasks.rados.rados.0.smithi125.stdout:1023: finishing rollback tid 2 to smithi12556470-29 2021-03-24T21:57:54.427 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 29 v 155 (ObjNum 274 snap 68 seq_num 274) dirty exists 2021-03-24T21:57:54.427 INFO:tasks.rados.rados.0.smithi125.stdout:1024: finishing write tid 1 to smithi12556470-10 2021-03-24T21:57:54.427 INFO:tasks.rados.rados.0.smithi125.stdout:1020: done (5 left) 2021-03-24T21:57:54.427 INFO:tasks.rados.rados.0.smithi125.stdout:1022: done (4 left) 2021-03-24T21:57:54.428 INFO:tasks.rados.rados.0.smithi125.stdout:1023: done (3 left) 2021-03-24T21:57:54.428 INFO:tasks.rados.rados.0.smithi125.stdout:1026: read oid 34 snap -1 2021-03-24T21:57:54.428 INFO:tasks.rados.rados.0.smithi125.stdout:1026: expect (ObjNum 203 snap 43 seq_num 203) 2021-03-24T21:57:54.428 INFO:tasks.rados.rados.0.smithi125.stdout:1027: rollback oid 15 current snap is 99 2021-03-24T21:57:54.428 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 15 to 98 2021-03-24T21:57:54.429 INFO:tasks.rados.rados.0.smithi125.stdout:1028: read oid 42 snap -1 2021-03-24T21:57:54.429 INFO:tasks.rados.rados.0.smithi125.stdout:1028: expect deleted 2021-03-24T21:57:54.429 INFO:tasks.rados.rados.0.smithi125.stdout:1029: read oid 40 snap 98 2021-03-24T21:57:54.429 INFO:tasks.rados.rados.0.smithi125.stdout:1029: expect (ObjNum 283 snap 70 seq_num 283) 2021-03-24T21:57:54.430 INFO:tasks.rados.rados.0.smithi125.stdout:1030: copy_from oid 26 from oid 2 current snap is 99 2021-03-24T21:57:54.430 INFO:tasks.rados.rados.0.smithi125.stdout:1031: rollback oid 49 current snap is 99 2021-03-24T21:57:54.430 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 49 to 98 2021-03-24T21:57:54.430 INFO:tasks.rados.rados.0.smithi125.stdout:1032: snap_remove snap 96 2021-03-24T21:57:54.455 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:54.512 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:54.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:54.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:57:54.619 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:54.671 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:54.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:54.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:57:54.779 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:54.791 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:57:54.836 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:54.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:54.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:57:54.942 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:54.997 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:55.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:55.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:57:55.101 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:55.165 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:55.166 INFO:tasks.rados.rados.0.smithi125.stdout:1032: done (9 left) 2021-03-24T21:57:55.167 INFO:tasks.rados.rados.0.smithi125.stdout:1033: rollback oid 46 current snap is 99 2021-03-24T21:57:55.167 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 46 to 98 2021-03-24T21:57:55.167 INFO:tasks.rados.rados.0.smithi125.stdout:1034: write (excl) oid 47 current snap is 99 2021-03-24T21:57:55.167 INFO:tasks.rados.rados.0.smithi125.stdout:1034: seq_num 384 ranges {453888=713266,1737556=442800} 2021-03-24T21:57:55.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:55.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:57:55.179 INFO:tasks.rados.rados.0.smithi125.stdout:1034: writing smithi12556470-47 from 453888 to 1167154 tid 1 2021-03-24T21:57:55.186 INFO:tasks.rados.rados.0.smithi125.stdout:1034: writing smithi12556470-47 from 1737556 to 2180356 tid 2 2021-03-24T21:57:55.186 INFO:tasks.rados.rados.0.smithi125.stdout:1035: read oid 45 snap -1 2021-03-24T21:57:55.187 INFO:tasks.rados.rados.0.smithi125.stdout:1035: expect deleted 2021-03-24T21:57:55.187 INFO:tasks.rados.rados.0.smithi125.stdout:1036: write (excl) oid 17 current snap is 99 2021-03-24T21:57:55.187 INFO:tasks.rados.rados.0.smithi125.stdout:1036: seq_num 385 ranges {470695=788515,1679685=586458} 2021-03-24T21:57:55.194 INFO:tasks.rados.rados.0.smithi125.stdout:1036: writing smithi12556470-17 from 470695 to 1259210 tid 1 2021-03-24T21:57:55.201 INFO:tasks.rados.rados.0.smithi125.stdout:1036: writing smithi12556470-17 from 1679685 to 2266143 tid 2 2021-03-24T21:57:55.202 INFO:tasks.rados.rados.0.smithi125.stdout:1037: rollback oid 18 current snap is 99 2021-03-24T21:57:55.202 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 18 to 98 2021-03-24T21:57:55.202 INFO:tasks.rados.rados.0.smithi125.stdout:1029: expect (ObjNum 283 snap 70 seq_num 283) 2021-03-24T21:57:55.274 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:55.330 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:55.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:55.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:57:55.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:55 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:57:55.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:55 smithi142 conmon[20125]: 57:53.859839+0000 osd.0 (osd.0) 76 : cluster [DBG] 7.12 scrub ok 2021-03-24T21:57:55.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:55 smithi142 conmon[20125]: cluster 2021-03-24T21:57: 2021-03-24T21:57:55.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:55 smithi142 conmon[20125]: 54.211655+0000 mon.a (mon.0) 970 2021-03-24T21:57:55.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:55 smithi142 conmon[20125]: : cluster [DBG] osdmap e304: 8 total, 8 up, 6 in 2021-03-24T21:57:55.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:55 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:57:55.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:55 smithi142 conmon[20125]: -24T21:57:54.241828+0000 2021-03-24T21:57:55.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:55 smithi142 conmon[20125]: osd.2 (osd.2) 60 : cluster [DBG] 2021-03-24T21:57:55.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:55 smithi142 conmon[20125]: 5.d scrub ok 2021-03-24T21:57:55.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:55 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:57:55.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:55 smithi142 conmon[20125]: 57:55.162525+0000 mon.a (mon. 2021-03-24T21:57:55.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:55 smithi142 conmon[20125]: 0) 971 : cluster [DBG] osdmap e305: 8 total, 8 up, 6 in 2021-03-24T21:57:55.369 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-24T21:57:55.435 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:55.477 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: cluster 2021-03-24T21:57:55.478 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: 2021-03-24T21:57:55.478 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: 2021 2021-03-24T21:57:55.478 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: - 2021-03-24T21:57:55.479 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: 03 2021-03-24T21:57:55.479 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: - 2021-03-24T21:57:55.479 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: 24 2021-03-24T21:57:55.479 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: T 2021-03-24T21:57:55.479 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: 21 2021-03-24T21:57:55.480 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: : 2021-03-24T21:57:55.480 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: 57 2021-03-24T21:57:55.480 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: : 2021-03-24T21:57:55.480 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: 53 2021-03-24T21:57:55.481 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: . 2021-03-24T21:57:55.481 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: 859839 2021-03-24T21:57:55.481 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: +0000 2021-03-24T21:57:55.481 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: 2021-03-24T21:57:55.481 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: osd.0 2021-03-24T21:57:55.482 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: ( 2021-03-24T21:57:55.482 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: osd 2021-03-24T21:57:55.482 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: . 2021-03-24T21:57:55.482 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: 0 2021-03-24T21:57:55.483 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: ) 2021-03-24T21:57:55.483 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: 76 2021-03-24T21:57:55.483 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: : 2021-03-24T21:57:55.483 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: cluster 2021-03-24T21:57:55.483 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: 2021-03-24T21:57:55.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: [DBG] 2021-03-24T21:57:55.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: 2021-03-24T21:57:55.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: 7.12 scrub ok 2021-03-24T21:57:55.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: 2021-03-24T21:57:55.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: cluster 2021-03-24T21:57:55.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: 2021-03-24T21:57:55.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: 2021 2021-03-24T21:57:55.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: - 2021-03-24T21:57:55.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: 03 2021-03-24T21:57:55.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: - 2021-03-24T21:57:55.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: 24 2021-03-24T21:57:55.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: T 2021-03-24T21:57:55.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: 21 2021-03-24T21:57:55.487 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: : 2021-03-24T21:57:55.487 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: 57 2021-03-24T21:57:55.487 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: : 2021-03-24T21:57:55.487 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: 54 2021-03-24T21:57:55.487 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: . 2021-03-24T21:57:55.488 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: 211655 2021-03-24T21:57:55.488 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: +0000 2021-03-24T21:57:55.488 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: 2021-03-24T21:57:55.488 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: mon.a 2021-03-24T21:57:55.488 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: ( 2021-03-24T21:57:55.489 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: mon 2021-03-24T21:57:55.489 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: . 2021-03-24T21:57:55.489 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: 0 2021-03-24T21:57:55.489 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: ) 2021-03-24T21:57:55.490 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: 970 2021-03-24T21:57:55.490 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: : 2021-03-24T21:57:55.490 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: cluster 2021-03-24T21:57:55.490 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: 2021-03-24T21:57:55.490 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: [DBG] 2021-03-24T21:57:55.491 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: 2021-03-24T21:57:55.491 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: osdmap e304: 8 total, 8 up, 6 in 2021-03-24T21:57:55.491 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: 2021-03-24T21:57:55.491 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: cluster 2021-03-24T21:57:55.491 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: 2021-03-24T21:57:55.492 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: 2021 2021-03-24T21:57:55.492 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: - 2021-03-24T21:57:55.492 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: 03 2021-03-24T21:57:55.492 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: - 2021-03-24T21:57:55.493 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: 24 2021-03-24T21:57:55.493 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: T 2021-03-24T21:57:55.493 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: 21 2021-03-24T21:57:55.493 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: : 2021-03-24T21:57:55.494 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: 57 2021-03-24T21:57:55.494 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: : 2021-03-24T21:57:55.494 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: 54 2021-03-24T21:57:55.494 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: . 2021-03-24T21:57:55.494 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: 241828 2021-03-24T21:57:55.495 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: +0000 2021-03-24T21:57:55.495 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: 2021-03-24T21:57:55.495 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: osd.2 2021-03-24T21:57:55.495 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: ( 2021-03-24T21:57:55.496 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: osd 2021-03-24T21:57:55.496 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: . 2021-03-24T21:57:55.496 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: 2 2021-03-24T21:57:55.496 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: ) 2021-03-24T21:57:55.497 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: 60 2021-03-24T21:57:55.497 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: : 2021-03-24T21:57:55.497 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: cluster 2021-03-24T21:57:55.497 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: 2021-03-24T21:57:55.498 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: [DBG] 2021-03-24T21:57:55.498 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: 2021-03-24T21:57:55.498 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: 5.d scrub ok 2021-03-24T21:57:55.499 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: 2021-03-24T21:57:55.499 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:57:55.499 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: -03-24T21:57:55.162525+0000 mon.a 2021-03-24T21:57:55.499 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[19936]: (mon.0) 971 : cluster [DBG] osdmap e305: 8 total, 8 up, 6 in 2021-03-24T21:57:55.500 INFO:tasks.rados.rados.0.smithi125.stdout:1024: finishing write tid 2 to smithi12556470-10 2021-03-24T21:57:55.501 INFO:tasks.rados.rados.0.smithi125.stdout:1024: finishing write tid 3 to smithi12556470-10 2021-03-24T21:57:55.501 INFO:tasks.rados.rados.0.smithi125.stdout:1028: done (13 left) 2021-03-24T21:57:55.501 INFO:tasks.rados.rados.0.smithi125.stdout:1029: done (12 left) 2021-03-24T21:57:55.502 INFO:tasks.rados.rados.0.smithi125.stdout:1038: write (excl) oid 50 current snap is 99 2021-03-24T21:57:55.502 INFO:tasks.rados.rados.0.smithi125.stdout:1024: finishing write tid 4 to smithi12556470-10 2021-03-24T21:57:55.502 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 10 v 326 (ObjNum 383 snap 99 seq_num 383) dirty exists 2021-03-24T21:57:55.502 INFO:tasks.rados.rados.0.smithi125.stdout:1024: left oid 10 (ObjNum 383 snap 99 seq_num 383) 2021-03-24T21:57:55.503 INFO:tasks.rados.rados.0.smithi125.stdout:1026: expect (ObjNum 203 snap 43 seq_num 203) 2021-03-24T21:57:55.503 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[24927]: cluster 2021-03-24T21:57:53.859839+0000 2021-03-24T21:57:55.503 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[24927]: osd.0 (osd.0) 76 : cluster [DBG] 7.12 scrub ok 2021-03-24T21:57:55.504 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:57:55.504 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[24927]: 21:57:54.211655+0000 mon.a (mon.0) 970 : cluster [DBG] osdmap e304: 8 total, 8 up, 6 in 2021-03-24T21:57:55.504 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:57:55.504 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[24927]: T21:57:54.241828+0000 osd.2 (osd.2 2021-03-24T21:57:55.504 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[24927]: ) 60 : cluster [DBG] 5.d scrub ok 2021-03-24T21:57:55.505 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:57:55.505 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[24927]: T21:57:55.162525+0000 mon.a (mon.0 2021-03-24T21:57:55.505 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:55 smithi125 conmon[24927]: ) 971 : cluster [DBG] osdmap e305: 8 total, 8 up, 6 in 2021-03-24T21:57:55.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:55.506 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:55.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:57:55.608 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:55.666 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:55.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:55.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:57:55.671 INFO:tasks.rados.rados.0.smithi125.stdout:1031: finishing rollback tid 0 to smithi12556470-49 2021-03-24T21:57:55.672 INFO:tasks.rados.rados.0.smithi125.stdout:1021: finishing copy_from to smithi12556470-11 2021-03-24T21:57:55.672 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 11 v 309 (ObjNum 353 snap 89 seq_num 353) dirty exists 2021-03-24T21:57:55.672 INFO:tasks.rados.rados.0.smithi125.stdout:1025: finishing rollback tid 0 to smithi12556470-44 2021-03-24T21:57:55.672 INFO:tasks.rados.rados.0.smithi125.stdout:1031: finishing rollback tid 1 to smithi12556470-49 2021-03-24T21:57:55.673 INFO:tasks.rados.rados.0.smithi125.stdout:1025: finishing rollback tid 1 to smithi12556470-44 2021-03-24T21:57:55.673 INFO:tasks.rados.rados.0.smithi125.stdout:1025: finishing rollback tid 2 to smithi12556470-44 2021-03-24T21:57:55.673 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 44 v 279 (ObjNum 358 snap 91 seq_num 358) dirty exists 2021-03-24T21:57:55.673 INFO:tasks.rados.rados.0.smithi125.stdout:1038: seq_num 386 ranges {487502=463764,1738167=446469} 2021-03-24T21:57:55.682 INFO:tasks.rados.rados.0.smithi125.stdout:1038: writing smithi12556470-50 from 487502 to 951266 tid 1 2021-03-24T21:57:55.693 INFO:tasks.rados.rados.0.smithi125.stdout:1038: writing smithi12556470-50 from 1738167 to 2184636 tid 2 2021-03-24T21:57:55.694 INFO:tasks.rados.rados.0.smithi125.stdout:1024: done (12 left) 2021-03-24T21:57:55.694 INFO:tasks.rados.rados.0.smithi125.stdout:1025: done (11 left) 2021-03-24T21:57:55.694 INFO:tasks.rados.rados.0.smithi125.stdout:1026: done (10 left) 2021-03-24T21:57:55.694 INFO:tasks.rados.rados.0.smithi125.stdout:1039: write (excl) oid 34 current snap is 99 2021-03-24T21:57:55.695 INFO:tasks.rados.rados.0.smithi125.stdout:1031: finishing rollback tid 2 to smithi12556470-49 2021-03-24T21:57:55.695 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 49 v 318 (ObjNum 362 snap 92 seq_num 362) dirty exists 2021-03-24T21:57:55.695 INFO:tasks.rados.rados.0.smithi125.stdout:1021: finishing copy_from racing read to smithi12556470-11 2021-03-24T21:57:55.696 INFO:tasks.rados.rados.0.smithi125.stdout:1027: finishing rollback tid 0 to smithi12556470-15 2021-03-24T21:57:55.696 INFO:tasks.rados.rados.0.smithi125.stdout:1027: finishing rollback tid 1 to smithi12556470-15 2021-03-24T21:57:55.696 INFO:tasks.rados.rados.0.smithi125.stdout:1027: finishing rollback tid 2 to smithi12556470-15 2021-03-24T21:57:55.696 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 15 v 315 (ObjNum 360 snap 92 seq_num 360) dirty exists 2021-03-24T21:57:55.697 INFO:tasks.rados.rados.0.smithi125.stdout:1030: finishing copy_from to smithi12556470-26 2021-03-24T21:57:55.697 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 26 v 309 (ObjNum 353 snap 89 seq_num 353) dirty exists 2021-03-24T21:57:55.697 INFO:tasks.rados.rados.0.smithi125.stdout:1030: finishing copy_from racing read to smithi12556470-26 2021-03-24T21:57:55.697 INFO:tasks.rados.rados.0.smithi125.stdout:1039: seq_num 387 ranges {504309=539013,1796649=590127} 2021-03-24T21:57:55.698 INFO:tasks.rados.rados.0.smithi125.stdout:1039: writing smithi12556470-34 from 504309 to 1043322 tid 1 2021-03-24T21:57:55.707 INFO:tasks.rados.rados.0.smithi125.stdout:1039: writing smithi12556470-34 from 1796649 to 2386776 tid 2 2021-03-24T21:57:55.707 INFO:tasks.rados.rados.0.smithi125.stdout:1021: done (10 left) 2021-03-24T21:57:55.707 INFO:tasks.rados.rados.0.smithi125.stdout:1027: done (9 left) 2021-03-24T21:57:55.707 INFO:tasks.rados.rados.0.smithi125.stdout:1030: done (8 left) 2021-03-24T21:57:55.708 INFO:tasks.rados.rados.0.smithi125.stdout:1031: done (7 left) 2021-03-24T21:57:55.708 INFO:tasks.rados.rados.0.smithi125.stdout:1040: snap_create 2021-03-24T21:57:55.708 INFO:tasks.rados.rados.0.smithi125.stdout:1034: finishing write tid 1 to smithi12556470-47 2021-03-24T21:57:55.708 INFO:tasks.rados.rados.0.smithi125.stdout:1034: finishing write tid 2 to smithi12556470-47 2021-03-24T21:57:55.708 INFO:tasks.rados.rados.0.smithi125.stdout:1034: finishing write tid 3 to smithi12556470-47 2021-03-24T21:57:55.709 INFO:tasks.rados.rados.0.smithi125.stdout:1034: finishing write tid 4 to smithi12556470-47 2021-03-24T21:57:55.709 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 47 v 103 (ObjNum 384 snap 99 seq_num 384) dirty exists 2021-03-24T21:57:55.709 INFO:tasks.rados.rados.0.smithi125.stdout:1034: left oid 47 (ObjNum 384 snap 99 seq_num 384) 2021-03-24T21:57:55.709 INFO:tasks.rados.rados.0.smithi125.stdout:1036: finishing write tid 1 to smithi12556470-17 2021-03-24T21:57:55.710 INFO:tasks.rados.rados.0.smithi125.stdout:1033: finishing rollback tid 0 to smithi12556470-46 2021-03-24T21:57:55.710 INFO:tasks.rados.rados.0.smithi125.stdout:1033: finishing rollback tid 1 to smithi12556470-46 2021-03-24T21:57:55.710 INFO:tasks.rados.rados.0.smithi125.stdout:1033: finishing rollback tid 2 to smithi12556470-46 2021-03-24T21:57:55.710 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 46 v 132 (ObjNum 315 snap 80 seq_num 315) dirty exists 2021-03-24T21:57:55.711 INFO:tasks.rados.rados.0.smithi125.stdout:1037: finishing rollback tid 0 to smithi12556470-18 2021-03-24T21:57:55.711 INFO:tasks.rados.rados.0.smithi125.stdout:1036: finishing write tid 2 to smithi12556470-17 2021-03-24T21:57:55.712 INFO:tasks.rados.rados.0.smithi125.stdout:1036: finishing write tid 3 to smithi12556470-17 2021-03-24T21:57:55.712 INFO:tasks.rados.rados.0.smithi125.stdout:1037: finishing rollback tid 1 to smithi12556470-18 2021-03-24T21:57:55.712 INFO:tasks.rados.rados.0.smithi125.stdout:1036: finishing write tid 4 to smithi12556470-17 2021-03-24T21:57:55.713 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 17 v 299 (ObjNum 385 snap 99 seq_num 385) dirty exists 2021-03-24T21:57:55.713 INFO:tasks.rados.rados.0.smithi125.stdout:1036: left oid 17 (ObjNum 385 snap 99 seq_num 385) 2021-03-24T21:57:55.713 INFO:tasks.rados.rados.0.smithi125.stdout:1037: finishing rollback tid 2 to smithi12556470-18 2021-03-24T21:57:55.713 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 18 v 175 (ObjNum 368 snap 95 seq_num 368) dirty exists 2021-03-24T21:57:55.714 INFO:tasks.rados.rados.0.smithi125.stdout:1038: finishing write tid 1 to smithi12556470-50 2021-03-24T21:57:55.714 INFO:tasks.rados.rados.0.smithi125.stdout:1038: finishing write tid 2 to smithi12556470-50 2021-03-24T21:57:55.714 INFO:tasks.rados.rados.0.smithi125.stdout:1038: finishing write tid 3 to smithi12556470-50 2021-03-24T21:57:55.715 INFO:tasks.rados.rados.0.smithi125.stdout:1038: finishing write tid 4 to smithi12556470-50 2021-03-24T21:57:55.715 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 50 v 225 (ObjNum 386 snap 99 seq_num 386) dirty exists 2021-03-24T21:57:55.715 INFO:tasks.rados.rados.0.smithi125.stdout:1038: left oid 50 (ObjNum 386 snap 99 seq_num 386) 2021-03-24T21:57:55.715 INFO:tasks.rados.rados.0.smithi125.stdout:1039: finishing write tid 1 to smithi12556470-34 2021-03-24T21:57:55.716 INFO:tasks.rados.rados.0.smithi125.stdout:1039: finishing write tid 2 to smithi12556470-34 2021-03-24T21:57:55.716 INFO:tasks.rados.rados.0.smithi125.stdout:1039: finishing write tid 3 to smithi12556470-34 2021-03-24T21:57:55.717 INFO:tasks.rados.rados.0.smithi125.stdout:1039: finishing write tid 4 to smithi12556470-34 2021-03-24T21:57:55.717 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 34 v 269 (ObjNum 387 snap 99 seq_num 387) dirty exists 2021-03-24T21:57:55.717 INFO:tasks.rados.rados.0.smithi125.stdout:1039: left oid 34 (ObjNum 387 snap 99 seq_num 387) 2021-03-24T21:57:55.772 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:55.823 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:55.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:55.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:57:55.844 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2021-03-24T21:57:55.844 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2021-03-24T21:57:55.845 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2021-03-24T21:57:55.845 INFO:teuthology.orchestra.run.smithi142.stdout:} 2021-03-24T21:57:55.928 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:55.998 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:56.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:56.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:57:56.103 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:56.154 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:56.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:56.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:56.259 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:56.284 INFO:tasks.rados.rados.0.smithi125.stdout:1033: done (7 left) 2021-03-24T21:57:56.284 INFO:tasks.rados.rados.0.smithi125.stdout:1034: done (6 left) 2021-03-24T21:57:56.284 INFO:tasks.rados.rados.0.smithi125.stdout:1035: done (5 left) 2021-03-24T21:57:56.285 INFO:tasks.rados.rados.0.smithi125.stdout:1036: done (4 left) 2021-03-24T21:57:56.285 INFO:tasks.rados.rados.0.smithi125.stdout:1037: done (3 left) 2021-03-24T21:57:56.285 INFO:tasks.rados.rados.0.smithi125.stdout:1038: done (2 left) 2021-03-24T21:57:56.285 INFO:tasks.rados.rados.0.smithi125.stdout:1039: done (1 left) 2021-03-24T21:57:56.286 INFO:tasks.rados.rados.0.smithi125.stdout:1040: done (0 left) 2021-03-24T21:57:56.286 INFO:tasks.rados.rados.0.smithi125.stdout:1041: write (excl) oid 23 current snap is 100 2021-03-24T21:57:56.286 INFO:tasks.rados.rados.0.smithi125.stdout:1041: seq_num 388 ranges {521116=730615,1971484=549632} 2021-03-24T21:57:56.287 INFO:teuthology.orchestra.run.smithi125.stderr:set osd.7 primary-affinity to 0.794102 (8520422) 2021-03-24T21:57:56.292 INFO:tasks.rados.rados.0.smithi125.stdout:1041: writing smithi12556470-23 from 521116 to 1251731 tid 1 2021-03-24T21:57:56.300 INFO:tasks.rados.rados.0.smithi125.stdout:1041: writing smithi12556470-23 from 1971484 to 2521116 tid 2 2021-03-24T21:57:56.300 INFO:tasks.rados.rados.0.smithi125.stdout:1042: write oid 22 current snap is 100 2021-03-24T21:57:56.300 INFO:tasks.rados.rados.0.smithi125.stdout:1041: finishing write tid 1 to smithi12556470-23 2021-03-24T21:57:56.300 INFO:tasks.rados.rados.0.smithi125.stdout:1042: seq_num 389 ranges {537923=405864,1513613=593796} 2021-03-24T21:57:56.305 INFO:tasks.rados.rados.0.smithi125.stdout:1042: writing smithi12556470-22 from 537923 to 943787 tid 1 2021-03-24T21:57:56.313 INFO:tasks.rados.rados.0.smithi125.stdout:1042: writing smithi12556470-22 from 1513613 to 2107409 tid 2 2021-03-24T21:57:56.313 INFO:tasks.rados.rados.0.smithi125.stdout:1043: read oid 10 snap 99 2021-03-24T21:57:56.313 INFO:tasks.rados.rados.0.smithi125.stdout:1043: expect (ObjNum 383 snap 99 seq_num 383) 2021-03-24T21:57:56.313 INFO:tasks.rados.rados.0.smithi125.stdout:1044: rollback oid 5 current snap is 100 2021-03-24T21:57:56.314 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 5 to 99 2021-03-24T21:57:56.314 INFO:tasks.rados.rados.0.smithi125.stdout:1041: finishing write tid 2 to smithi12556470-23 2021-03-24T21:57:56.314 INFO:tasks.rados.rados.0.smithi125.stdout:1045: delete oid 3 current snap is 100 2021-03-24T21:57:56.314 INFO:tasks.rados.rados.0.smithi125.stdout:1041: finishing write tid 3 to smithi12556470-23 2021-03-24T21:57:56.315 INFO:tasks.rados.rados.0.smithi125.stdout:1041: finishing write tid 4 to smithi12556470-23 2021-03-24T21:57:56.315 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 23 v 215 (ObjNum 388 snap 100 seq_num 388) dirty exists 2021-03-24T21:57:56.315 INFO:tasks.rados.rados.0.smithi125.stdout:1041: left oid 23 (ObjNum 388 snap 100 seq_num 388) 2021-03-24T21:57:56.315 INFO:tasks.rados.rados.0.smithi125.stdout:1042: finishing write tid 1 to smithi12556470-22 2021-03-24T21:57:56.316 INFO:tasks.rados.rados.0.smithi125.stdout:1041: done (4 left) 2021-03-24T21:57:56.316 INFO:tasks.rados.rados.0.smithi125.stdout:1045: done (3 left) 2021-03-24T21:57:56.316 INFO:tasks.rados.rados.0.smithi125.stdout:1046: write oid 31 current snap is 100 2021-03-24T21:57:56.317 INFO:tasks.rados.rados.0.smithi125.stdout:1046: seq_num 391 ranges {571537=556362,1630577=597465} 2021-03-24T21:57:56.321 INFO:tasks.rados.rados.0.smithi125.stdout:1046: writing smithi12556470-31 from 571537 to 1127899 tid 1 2021-03-24T21:57:56.328 INFO:tasks.rados.rados.0.smithi125.stdout:1046: writing smithi12556470-31 from 1630577 to 2228042 tid 2 2021-03-24T21:57:56.328 INFO:tasks.rados.rados.0.smithi125.stdout:1047: snap_remove snap 98 2021-03-24T21:57:56.333 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:56.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:56.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:57:56.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:56 smithi142 conmon[20125]: cluster 2021-03-24T21:57:54.381650+0000 osd.3 ( 2021-03-24T21:57:56.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:56 smithi142 conmon[20125]: osd.3) 74 : cluster [DBG] 3.7 scrub ok 2021-03-24T21:57:56.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:56 smithi142 conmon[20125]: cluster 2021-03-24T21:57:54.784401+0000 mgr.y (mgr.14144) 420 : cluster [DBG] pgmap v619: 168 pgs: 168 active+clean; 112 MiB data, 441 MiB used, 536 GiB / 536 GiB avail; 6.7 MiB/s rd, 855 KiB/s wr, 10 op/s 2021-03-24T21:57:56.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:56 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:57:56.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:56 smithi142 conmon[20125]: 03-24T21:57:55.072685+0000 osd.5 (osd.5) 76 : cluster [DBG] 6.7 scrub ok 2021-03-24T21:57:56.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:56 smithi142 conmon[20125]: audit 2021-03-24T21:57:55.625101 2021-03-24T21:57:56.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:56 smithi142 conmon[20125]: +0000 mon.a (mon.0) 972 : audit [INF] from='client.25024 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.7941021136087902}]: dispatch 2021-03-24T21:57:56.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:56 smithi142 conmon[20125]: audit 2021-03-24T21:57:56.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:56 smithi142 conmon[20125]: 2021-03-24T21:57:55.625829+0000 mon.b (mon. 2021-03-24T21:57:56.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:56 smithi142 conmon[20125]: 1) 32 : audit [INF] from='client.? v1:172.21.15.125:0/3848509480' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.7941021136087902}]: dispatch 2021-03-24T21:57:56.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:56 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:57:56.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:56 smithi142 conmon[20125]: -03-24T21:57:55.705689+0000 osd.5 (osd 2021-03-24T21:57:56.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:56 smithi142 conmon[20125]: .5) 77 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:57:56.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:56 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:57:56.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:56 smithi142 conmon[20125]: -03-24T21:57:55.706137+0000 osd.5 (osd 2021-03-24T21:57:56.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:56 smithi142 conmon[20125]: .5) 78 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:57:56.437 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:56.498 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:56.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:56.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:57:56.544 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:56 smithi125 conmon[24927]: cluster 2021-03-24T21:57:54.381650+0000 2021-03-24T21:57:56.544 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:56 smithi125 conmon[24927]: osd.3 (osd.3) 74 : cluster [DBG] 3.7 scrub ok 2021-03-24T21:57:56.544 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:56 smithi125 conmon[24927]: cluster 2021-03-24T21:57:54.784401+0000 mgr.y (mgr.14144) 420 : cluster [DBG] pgmap v619: 168 pgs: 168 active+clean; 112 MiB data, 441 MiB used, 536 GiB / 536 GiB avail; 6.7 MiB/s rd, 855 KiB/s wr, 10 op/s 2021-03-24T21:57:56.545 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:56 smithi125 conmon[24927]: cluster 2021-03-24T21:57:55.072685+0000 2021-03-24T21:57:56.545 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:56 smithi125 conmon[24927]: osd.5 (osd.5) 76 : cluster [DBG] 6.7 scrub ok 2021-03-24T21:57:56.545 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:56 smithi125 conmon[24927]: audit 2021 2021-03-24T21:57:56.545 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:56 smithi125 conmon[24927]: -03-24T21:57:55.625101+0000 2021-03-24T21:57:56.546 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:56 smithi125 conmon[24927]: mon.a (mon.0) 972 : audit [INF] from='client.25024 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.7941021136087902}]: dispatch 2021-03-24T21:57:56.546 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:56 smithi125 conmon[24927]: 2021-03-24T21:57:56.546 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:56 smithi125 conmon[24927]: audit 2021-03-24T21:57:56.546 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:56 smithi125 conmon[24927]: 2021-03-24T21:57:55.625829+0000 mon.b 2021-03-24T21:57:56.546 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:56 smithi125 conmon[24927]: (mon.1) 32 : audit [INF] from='client.? v1:172.21.15.125:0/3848509480' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.7941021136087902}]: dispatch 2021-03-24T21:57:56.547 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:56 smithi125 conmon[24927]: 2021-03-24T21:57:56.547 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:56 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:57:56.547 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:56 smithi125 conmon[24927]: -03-24T21:57:55.705689+0000 osd.5 (osd. 2021-03-24T21:57:56.547 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:56 smithi125 conmon[24927]: 5) 77 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:57:56.548 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:56 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:57:56.548 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:56 smithi125 conmon[24927]: 21:57:55.706137+0000 osd.5 (osd.5 2021-03-24T21:57:56.548 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:56 smithi125 conmon[24927]: ) 78 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:57:56.548 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:56 smithi125 conmon[19936]: cluster 2021-03-24T21:57:54.381650 2021-03-24T21:57:56.549 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:56 smithi125 conmon[19936]: +0000 osd.3 (osd.3) 74 : cluster [DBG] 3.7 scrub ok 2021-03-24T21:57:56.549 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:56 smithi125 conmon[19936]: cluster 2021-03-24T21:57:54.784401+0000 mgr.y ( 2021-03-24T21:57:56.549 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:56 smithi125 conmon[19936]: mgr.14144) 420 : cluster [DBG] pgmap v619: 168 pgs: 168 active+clean; 112 MiB data, 441 MiB used, 536 GiB / 536 GiB avail; 6.7 MiB/s rd, 855 KiB/s wr, 10 op/s 2021-03-24T21:57:56.549 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:56 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:57:56.549 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:56 smithi125 conmon[19936]: 24T21:57:55.072685+0000 osd.5 (osd.5) 76 : cluster [DBG] 6.7 scrub ok 2021-03-24T21:57:56.550 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:56 smithi125 conmon[19936]: 2021-03-24T21:57:56.550 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:56 smithi125 conmon[19936]: audit 2021 2021-03-24T21:57:56.550 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:56 smithi125 conmon[19936]: -03-24T21:57:55.625101+0000 mon.a (mon.0) 2021-03-24T21:57:56.550 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:56 smithi125 conmon[19936]: 972 : audit [INF] from='client.25024 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.7941021136087902}]: dispatch 2021-03-24T21:57:56.551 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:56 smithi125 conmon[19936]: audit 2021-03-24T21:57 2021-03-24T21:57:56.551 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:56 smithi125 conmon[19936]: :55.625829+0000 mon.b (mon.1) 32 : audit 2021-03-24T21:57:56.551 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:56 smithi125 conmon[19936]: [INF] from='client.? v1:172.21.15.125:0/3848509480' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.7941021136087902}]: dispatch 2021-03-24T21:57:56.551 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:56 smithi125 conmon[19936]: cluster 2021-03-24T21:57: 2021-03-24T21:57:56.552 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:56 smithi125 conmon[19936]: 55.705689+0000 osd.5 (osd.5) 77 : cluster 2021-03-24T21:57:56.552 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:56 smithi125 conmon[19936]: [DBG] purged_snaps scrub starts 2021-03-24T21:57:56.552 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:56 smithi125 conmon[19936]: cluster 2021-03-24T21:57 2021-03-24T21:57:56.552 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:56 smithi125 conmon[19936]: :55.706137+0000 osd.5 (osd.5) 78 : 2021-03-24T21:57:56.552 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:56 smithi125 conmon[19936]: cluster [DBG] purged_snaps scrub ok 2021-03-24T21:57:56.603 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:56.657 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:56.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:56.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:57:56.723 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:57:56.762 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:56.806 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:56.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:56.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:57:56.908 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.5 dump_blocked_ops 2021-03-24T21:57:56.915 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:56.972 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:56.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:56.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:57:57.079 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:57.139 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:57.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:57.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:57.199 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T21:57:57.244 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:57.284 INFO:tasks.rados.rados.0.smithi125.stdout:1047: done (4 left) 2021-03-24T21:57:57.284 INFO:tasks.rados.rados.0.smithi125.stdout:1048: rollback oid 2 current snap is 100 2021-03-24T21:57:57.285 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 2 to 99 2021-03-24T21:57:57.285 INFO:tasks.rados.rados.0.smithi125.stdout:1044: finishing rollback tid 1 to smithi12556470-5 2021-03-24T21:57:57.285 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 5 v 0 (ObjNum 1116690320 snap 21916 seq_num 3308612608) dirty dne 2021-03-24T21:57:57.285 INFO:tasks.rados.rados.0.smithi125.stdout:1042: finishing write tid 2 to smithi12556470-22 2021-03-24T21:57:57.286 INFO:tasks.rados.rados.0.smithi125.stdout:1042: finishing write tid 3 to smithi12556470-22 2021-03-24T21:57:57.286 INFO:tasks.rados.rados.0.smithi125.stdout:1042: finishing write tid 4 to smithi12556470-22 2021-03-24T21:57:57.286 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 22 v 286 (ObjNum 389 snap 100 seq_num 389) dirty exists 2021-03-24T21:57:57.286 INFO:tasks.rados.rados.0.smithi125.stdout:1042: left oid 22 (ObjNum 389 snap 100 seq_num 389) 2021-03-24T21:57:57.286 INFO:tasks.rados.rados.0.smithi125.stdout:1043: expect (ObjNum 383 snap 99 seq_num 383) 2021-03-24T21:57:57.326 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:57.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:57.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:57:57.376 INFO:tasks.rados.rados.0.smithi125.stdout:1046: finishing write tid 1 to smithi12556470-31 2021-03-24T21:57:57.376 INFO:tasks.rados.rados.0.smithi125.stdout:1046: finishing write tid 2 to smithi12556470-31 2021-03-24T21:57:57.376 INFO:tasks.rados.rados.0.smithi125.stdout:1046: finishing write tid 3 to smithi12556470-31 2021-03-24T21:57:57.376 INFO:tasks.rados.rados.0.smithi125.stdout:1046: finishing write tid 4 to smithi12556470-31 2021-03-24T21:57:57.377 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 31 v 301 (ObjNum 391 snap 100 seq_num 391) dirty exists 2021-03-24T21:57:57.377 INFO:tasks.rados.rados.0.smithi125.stdout:1046: left oid 31 (ObjNum 391 snap 100 seq_num 391) 2021-03-24T21:57:57.377 INFO:tasks.rados.rados.0.smithi125.stdout:1048: finishing rollback tid 0 to smithi12556470-2 2021-03-24T21:57:57.377 INFO:tasks.rados.rados.0.smithi125.stdout:1048: finishing rollback tid 1 to smithi12556470-2 2021-03-24T21:57:57.377 INFO:tasks.rados.rados.0.smithi125.stdout:1048: finishing rollback tid 2 to smithi12556470-2 2021-03-24T21:57:57.378 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 2 v 318 (ObjNum 353 snap 89 seq_num 353) dirty exists 2021-03-24T21:57:57.378 INFO:tasks.rados.rados.0.smithi125.stdout:1042: done (4 left) 2021-03-24T21:57:57.378 INFO:tasks.rados.rados.0.smithi125.stdout:1043: done (3 left) 2021-03-24T21:57:57.378 INFO:tasks.rados.rados.0.smithi125.stdout:1044: done (2 left) 2021-03-24T21:57:57.379 INFO:tasks.rados.rados.0.smithi125.stdout:1046: done (1 left) 2021-03-24T21:57:57.379 INFO:tasks.rados.rados.0.smithi125.stdout:1048: done (0 left) 2021-03-24T21:57:57.379 INFO:tasks.rados.rados.0.smithi125.stdout:1049: write (excl) oid 50 current snap is 100 2021-03-24T21:57:57.379 INFO:tasks.rados.rados.0.smithi125.stdout:1049: seq_num 392 ranges {588344=631611,1689059=741123} 2021-03-24T21:57:57.387 INFO:tasks.rados.rados.0.smithi125.stdout:1049: writing smithi12556470-50 from 588344 to 1219955 tid 1 2021-03-24T21:57:57.395 INFO:tasks.rados.rados.0.smithi125.stdout:1049: writing smithi12556470-50 from 1689059 to 2430182 tid 2 2021-03-24T21:57:57.395 INFO:tasks.rados.rados.0.smithi125.stdout:1050: read oid 3 snap -1 2021-03-24T21:57:57.395 INFO:tasks.rados.rados.0.smithi125.stdout:1050: expect deleted 2021-03-24T21:57:57.396 INFO:tasks.rados.rados.0.smithi125.stdout:1051: write oid 10 current snap is 100 2021-03-24T21:57:57.396 INFO:tasks.rados.rados.0.smithi125.stdout:1049: finishing write tid 1 to smithi12556470-50 2021-03-24T21:57:57.396 INFO:tasks.rados.rados.0.smithi125.stdout:1051: seq_num 393 ranges {605151=706860,2031188=573963} 2021-03-24T21:57:57.402 INFO:tasks.rados.rados.0.smithi125.stdout:1051: writing smithi12556470-10 from 605151 to 1312011 tid 1 2021-03-24T21:57:57.410 INFO:tasks.rados.rados.0.smithi125.stdout:1051: writing smithi12556470-10 from 2031188 to 2605151 tid 2 2021-03-24T21:57:57.410 INFO:tasks.rados.rados.0.smithi125.stdout:1052: copy_from oid 49 from oid 31 current snap is 100 2021-03-24T21:57:57.411 INFO:tasks.rados.rados.0.smithi125.stdout:1050: done (3 left) 2021-03-24T21:57:57.411 INFO:tasks.rados.rados.0.smithi125.stdout:1053: read oid 36 snap -1 2021-03-24T21:57:57.411 INFO:tasks.rados.rados.0.smithi125.stdout:1053: expect (ObjNum 150 snap 30 seq_num 150) 2021-03-24T21:57:57.411 INFO:tasks.rados.rados.0.smithi125.stdout:1054: rollback oid 21 current snap is 100 2021-03-24T21:57:57.411 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 21 to 99 2021-03-24T21:57:57.412 INFO:tasks.rados.rados.0.smithi125.stdout:1049: finishing write tid 2 to smithi12556470-50 2021-03-24T21:57:57.412 INFO:tasks.rados.rados.0.smithi125.stdout:1049: finishing write tid 3 to smithi12556470-50 2021-03-24T21:57:57.412 INFO:tasks.rados.rados.0.smithi125.stdout:1049: finishing write tid 4 to smithi12556470-50 2021-03-24T21:57:57.412 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 50 v 235 (ObjNum 392 snap 100 seq_num 392) dirty exists 2021-03-24T21:57:57.413 INFO:tasks.rados.rados.0.smithi125.stdout:1049: left oid 50 (ObjNum 392 snap 100 seq_num 392) 2021-03-24T21:57:57.413 INFO:tasks.rados.rados.0.smithi125.stdout:1051: finishing write tid 1 to smithi12556470-10 2021-03-24T21:57:57.413 INFO:tasks.rados.rados.0.smithi125.stdout:1049: done (4 left) 2021-03-24T21:57:57.413 INFO:tasks.rados.rados.0.smithi125.stdout:1055: snap_create 2021-03-24T21:57:57.414 INFO:tasks.rados.rados.0.smithi125.stdout:1054: finishing rollback tid 0 to smithi12556470-21 2021-03-24T21:57:57.414 INFO:tasks.rados.rados.0.smithi125.stdout:1054: finishing rollback tid 1 to smithi12556470-21 2021-03-24T21:57:57.415 INFO:tasks.rados.rados.0.smithi125.stdout:1051: finishing write tid 2 to smithi12556470-10 2021-03-24T21:57:57.416 INFO:tasks.rados.rados.0.smithi125.stdout:1051: finishing write tid 3 to smithi12556470-10 2021-03-24T21:57:57.416 INFO:tasks.rados.rados.0.smithi125.stdout:1051: finishing write tid 4 to smithi12556470-10 2021-03-24T21:57:57.417 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 10 v 332 (ObjNum 393 snap 100 seq_num 393) dirty exists 2021-03-24T21:57:57.417 INFO:tasks.rados.rados.0.smithi125.stdout:1051: left oid 10 (ObjNum 393 snap 100 seq_num 393) 2021-03-24T21:57:57.417 INFO:tasks.rados.rados.0.smithi125.stdout:1054: finishing rollback tid 2 to smithi12556470-21 2021-03-24T21:57:57.417 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 21 v 287 (ObjNum 334 snap 84 seq_num 334) dirty exists 2021-03-24T21:57:57.418 INFO:tasks.rados.rados.0.smithi125.stdout:1053: expect (ObjNum 150 snap 30 seq_num 150) 2021-03-24T21:57:57.431 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:57.468 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:57 smithi142 conmon[20125]: cluster 2021-03-24T21:57:57.469 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:57 smithi142 conmon[20125]: 2021-03-24T21:57:57.469 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:57 smithi142 conmon[20125]: 2021 2021-03-24T21:57:57.469 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:57 smithi142 conmon[20125]: - 2021-03-24T21:57:57.469 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:57 smithi142 conmon[20125]: 03 2021-03-24T21:57:57.470 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:57 smithi142 conmon[20125]: - 2021-03-24T21:57:57.470 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:57 smithi142 conmon[20125]: 24 2021-03-24T21:57:57.470 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:57 smithi142 conmon[20125]: T 2021-03-24T21:57:57.470 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:57 smithi142 conmon[20125]: 21 2021-03-24T21:57:57.471 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:57 smithi142 conmon[20125]: : 2021-03-24T21:57:57.471 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:57 smithi142 conmon[20125]: 57 2021-03-24T21:57:57.471 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:57 smithi142 conmon[20125]: : 2021-03-24T21:57:57.471 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:57 smithi142 conmon[20125]: 55 2021-03-24T21:57:57.471 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:57 smithi142 conmon[20125]: . 2021-03-24T21:57:57.472 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:57 smithi142 conmon[20125]: 592781 2021-03-24T21:57:57.472 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:57 smithi142 conmon[20125]: +0000 osd.4 (osd.4 2021-03-24T21:57:57.472 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:57 smithi142 conmon[20125]: ) 79 : cluster [DBG] 3.9 scrub ok 2021-03-24T21:57:57.472 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:57 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:57:57.473 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:57 smithi142 conmon[20125]: 03-24T21:57:56.065097+0000 osd.7 (osd 2021-03-24T21:57:57.473 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:57 smithi142 conmon[20125]: .7) 69 : cluster [DBG] 6.5 scrub ok 2021-03-24T21:57:57.473 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:57 smithi142 conmon[20125]: audit 2021 2021-03-24T21:57:57.473 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:57 smithi142 conmon[20125]: -03-24T21:57:56.276917+0000 mon.a ( 2021-03-24T21:57:57.473 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:57 smithi142 conmon[20125]: mon.0) 973 : audit [INF] from='client.25024 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.7941021136087902}]': finished 2021-03-24T21:57:57.474 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:57 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:57:57.474 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:57 smithi142 conmon[20125]: 21:57:56.277120+0000 mon.a (mon.0) 2021-03-24T21:57:57.474 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:57 smithi142 conmon[20125]: 974 : cluster [DBG] osdmap e306: 8 total, 8 up, 6 in 2021-03-24T21:57:57.474 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:57 smithi142 conmon[20125]: cluster 2021-03-24T21:57: 2021-03-24T21:57:57.475 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:57 smithi142 conmon[20125]: 57.274072+0000 mon.a (mon.0) 975 : cluster 2021-03-24T21:57:57.475 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:57 smithi142 conmon[20125]: [DBG] osdmap e307: 8 total, 8 up, 6 in 2021-03-24T21:57:57.480 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:57.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:57.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:57:57.551 INFO:tasks.rados.rados.0.smithi125.stdout:1052: finishing copy_from to smithi12556470-49 2021-03-24T21:57:57.551 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 49 v 301 (ObjNum 391 snap 100 seq_num 391) dirty exists 2021-03-24T21:57:57.551 INFO:tasks.rados.rados.0.smithi125.stdout:1052: finishing copy_from racing read to smithi12556470-49 2021-03-24T21:57:57.561 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:57 smithi125 conmon[24927]: cluster 2021-03-24T21:57:55.592781+0000 osd.4 2021-03-24T21:57:57.561 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:57 smithi125 conmon[24927]: (osd.4) 79 : cluster [DBG] 3.9 scrub ok 2021-03-24T21:57:57.561 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:57 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:57:57.562 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:57 smithi125 conmon[24927]: 24 2021-03-24T21:57:57.562 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:57 smithi125 conmon[24927]: T21:57:56.065097 2021-03-24T21:57:57.562 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:57 smithi125 conmon[24927]: +0000 osd.7 (osd.7) 69 : cluster [DBG] 6.5 scrub ok 2021-03-24T21:57:57.563 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:57 smithi125 conmon[24927]: audit 2021-03-24T21:57:56.276917 2021-03-24T21:57:57.563 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:57 smithi125 conmon[24927]: +0000 mon.a (mon.0) 973 : audit [INF] from='client.25024 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.7941021136087902}]': finished 2021-03-24T21:57:57.563 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:57 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:57:57.564 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:57 smithi125 conmon[24927]: -03-24T21:57:56.277120+0000 mon.a (mon 2021-03-24T21:57:57.564 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:57 smithi125 conmon[24927]: .0) 974 : cluster [DBG] osdmap e306: 8 total, 8 up, 6 in 2021-03-24T21:57:57.564 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:57 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:57:57.564 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:57 smithi125 conmon[24927]: 03-24T21:57:57.274072+0000 mon.a (mon 2021-03-24T21:57:57.565 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:57 smithi125 conmon[24927]: .0) 975 : cluster [DBG] osdmap e307: 8 total, 8 up, 6 in 2021-03-24T21:57:57.566 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:57 smithi125 conmon[19936]: cluster 2021-03-24T21:57:57.566 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:57 smithi125 conmon[19936]: 2021-03-24T21:57:57.566 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:57 smithi125 conmon[19936]: 2021 2021-03-24T21:57:57.567 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:57 smithi125 conmon[19936]: - 2021-03-24T21:57:57.567 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:57 smithi125 conmon[19936]: 03 2021-03-24T21:57:57.567 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:57 smithi125 conmon[19936]: -24T21:57:55.592781+0000 2021-03-24T21:57:57.567 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:57 smithi125 conmon[19936]: osd.4 (osd.4) 79 : cluster [DBG] 3.9 scrub ok 2021-03-24T21:57:57.567 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:57 smithi125 conmon[19936]: cluster 2021-03-24T21:57:56.065097+0000 2021-03-24T21:57:57.568 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:57 smithi125 conmon[19936]: osd.7 (osd.7) 69 : cluster [DBG] 6.5 scrub ok 2021-03-24T21:57:57.568 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:57 smithi125 conmon[19936]: audit 2021-03-24T21:57:57.568 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:57 smithi125 conmon[19936]: 2021-03-24T21:57:56.276917 2021-03-24T21:57:57.568 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:57 smithi125 conmon[19936]: +0000 mon.a (mon.0) 973 : audit 2021-03-24T21:57:57.569 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:57 smithi125 conmon[19936]: [INF] from='client.25024 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.7941021136087902}]': finished 2021-03-24T21:57:57.569 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:57 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T21:57:57.569 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:57 smithi125 conmon[19936]: T21:57:56.277120+0000 mon.a ( 2021-03-24T21:57:57.569 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:57 smithi125 conmon[19936]: mon.0) 974 : cluster [DBG] osdmap e306: 8 total, 8 up, 6 in 2021-03-24T21:57:57.569 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:57 smithi125 conmon[19936]: cluster 2021-03-24T21:57:57.570 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:57 smithi125 conmon[19936]: 2021-03-24T21:57:57.274072 2021-03-24T21:57:57.570 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:57 smithi125 conmon[19936]: +0000 mon.a (mon.0) 975 : cluster 2021-03-24T21:57:57.570 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:57 smithi125 conmon[19936]: [DBG] osdmap e307: 8 total, 8 up, 6 in 2021-03-24T21:57:57.585 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:57.642 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:57.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:57.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:57:57.748 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:57.803 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:57:57.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:57.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:57:57.909 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:57.966 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:57:57.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:57.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:57:58.072 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:58.129 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:58.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:58.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:57:58.235 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:58.289 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:57:58.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:58.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:57:58.310 INFO:tasks.rados.rados.0.smithi125.stdout:1051: done (4 left) 2021-03-24T21:57:58.311 INFO:tasks.rados.rados.0.smithi125.stdout:1052: done (3 left) 2021-03-24T21:57:58.311 INFO:tasks.rados.rados.0.smithi125.stdout:1053: done (2 left) 2021-03-24T21:57:58.311 INFO:tasks.rados.rados.0.smithi125.stdout:1054: done (1 left) 2021-03-24T21:57:58.311 INFO:tasks.rados.rados.0.smithi125.stdout:1055: done (0 left) 2021-03-24T21:57:58.311 INFO:tasks.rados.rados.0.smithi125.stdout:1056: snap_create 2021-03-24T21:57:58.314 INFO:teuthology.orchestra.run.smithi125.stderr:nodeep-scrub is unset 2021-03-24T21:57:58.396 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:58.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:58 smithi125 conmon[24927]: cluster 2021-03-24T21:57:56. 2021-03-24T21:57:58.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:58 smithi125 conmon[24927]: 548924+0000 osd.4 (osd.4) 80 : cluster [DBG] 5.11 scrub ok 2021-03-24T21:57:58.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:58 smithi125 conmon[24927]: cluster 2021-03-24T21:57:56 2021-03-24T21:57:58.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:58 smithi125 conmon[24927]: .785079+0000 mgr.y (mgr.14144) 421 : cluster [DBG] pgmap v622: 168 pgs: 2 peering, 166 active+clean; 112 MiB data, 450 MiB used, 536 GiB / 536 GiB avail; 6.7 MiB/s rd, 855 KiB/s wr, 10 op/s 2021-03-24T21:57:58.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:58 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:57:58.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:58 smithi125 conmon[24927]: -03-24T21:57:56.848770+0000 osd.0 (osd.0) 77 : cluster [DBG] 7.18 scrub ok 2021-03-24T21:57:58.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:58 smithi125 conmon[24927]: cluster 2021-03-24T21:57 2021-03-24T21:57:58.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:58 smithi125 conmon[24927]: :57.049853+0000 osd.7 (osd.7) 70 : cluster [DBG] 3.e scrub ok 2021-03-24T21:57:58.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:58 smithi125 conmon[24927]: cluster 2021-03-24T21:57 2021-03-24T21:57:58.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:58 smithi125 conmon[24927]: :57.286906+0000 mon.a (mon.0) 976 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2021-03-24T21:57:58.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:58 smithi125 conmon[24927]: audit 2021-03-24T21 2021-03-24T21:57:58.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:58 smithi125 conmon[24927]: :57:57.932178+0000 mon.a (mon.0) 977 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:57:58.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:58 smithi125 conmon[24927]: 2021-03-24T21:57:58.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:58 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T21:57:58.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:58 smithi125 conmon[24927]: 24T21:57:58.247015+0000 mon.a (mon.0) 978 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:57:58.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:58 smithi125 conmon[24927]: audit 2021-03-24T21 2021-03-24T21:57:58.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:58 smithi125 conmon[24927]: :57:58.267653+0000 mon.a (mon.0) 979 : audit [INF] from='client.? v1:172.21.15.125:0/1210241986' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:57:58.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:58 smithi125 conmon[24927]: 2021-03-24T21:57:58.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:58 smithi125 conmon[19936]: cluster 2021-03-24T21:57:56.548924 2021-03-24T21:57:58.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:58 smithi125 conmon[19936]: +0000 osd.4 (osd.4) 80 : cluster [DBG] 5.11 scrub ok 2021-03-24T21:57:58.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:58 smithi125 conmon[19936]: cluster 2021-03-24T21:57:56.785079+0000 mgr.y ( 2021-03-24T21:57:58.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:58 smithi125 conmon[19936]: mgr.14144) 421 : cluster [DBG] pgmap v622: 168 pgs: 2 peering, 166 active+clean; 112 MiB data, 450 MiB used, 536 GiB / 536 GiB avail; 6.7 MiB/s rd, 855 KiB/s wr, 10 op/s 2021-03-24T21:57:58.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:58 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:57:58.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:58 smithi125 conmon[19936]: -03-24T21:57:56. 2021-03-24T21:57:58.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:58 smithi125 conmon[19936]: 848770+0000 osd.0 (osd.0) 77 : 2021-03-24T21:57:58.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:58 smithi125 conmon[19936]: cluster [DBG] 7.18 scrub ok 2021-03-24T21:57:58.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:58 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:57:58.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:58 smithi125 conmon[19936]: -24T21:57:57.049853+0000 2021-03-24T21:57:58.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:58 smithi125 conmon[19936]: osd.7 (osd.7) 70 : cluster [DBG] 3.e scrub ok 2021-03-24T21:57:58.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:58 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T21:57:58.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:58 smithi125 conmon[19936]: T21:57:57.286906+0000 mon.a (mon 2021-03-24T21:57:58.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:58 smithi125 conmon[19936]: .0) 976 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2021-03-24T21:57:58.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:58 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:57:58.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:58 smithi125 conmon[19936]: 03-24T21:57:57.932178+0000 mon.a 2021-03-24T21:57:58.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:58 smithi125 conmon[19936]: (mon.0) 977 : audit [DBG] 2021-03-24T21:57:58.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:58 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:57:58.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:58 smithi125 conmon[19936]: audit 2021-03-24T21:57:58.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:58 smithi125 conmon[19936]: 2021-03-24T21:57:58.247015+0000 mon.a ( 2021-03-24T21:57:58.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:58 smithi125 conmon[19936]: mon.0) 978 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:57:58.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:58 smithi125 conmon[19936]: audit 2021 2021-03-24T21:57:58.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:58 smithi125 conmon[19936]: -03-24T21:57:58.267653 2021-03-24T21:57:58.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:58 smithi125 conmon[19936]: +0000 mon.a (mon.0) 979 : audit 2021-03-24T21:57:58.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:58 smithi125 conmon[19936]: [INF] from='client.? v1:172.21.15.125:0/1210241986' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:57:58.446 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:58.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:58.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:57:58.551 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:58.585 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:58 smithi142 conmon[20125]: cluster 2021-03-24T21:57:56.548924+0000 osd.4 (osd.4) 80 : cluster [DBG] 5.11 scrub ok 2021-03-24T21:57:58.585 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:58 smithi142 conmon[20125]: cluster 2021-03-24T21:57:56.785079+0000 mgr.y (mgr.14144) 421 : cluster [DBG] pgmap v622: 168 pgs: 2 peering, 166 active+clean; 112 MiB data, 450 MiB used, 536 GiB / 536 GiB avail; 6.7 MiB/s rd, 855 KiB/s wr, 10 op/s 2021-03-24T21:57:58.586 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:58 smithi142 conmon[20125]: cluster 2021-03-24T21:57:56.848770+0000 osd.0 (osd.0) 77 : cluster [DBG] 7.18 scrub ok 2021-03-24T21:57:58.586 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:58 smithi142 conmon[20125]: cluster 2021-03-24T21:57:57.049853+0000 osd.7 (osd.7) 70 : cluster [DBG] 3.e scrub ok 2021-03-24T21:57:58.586 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:58 smithi142 conmon[20125]: 2021-03-24T21:57:58.586 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:58 smithi142 conmon[20125]: cluster 2021-03-24T21:57:57.286906+0000 mon.a (mon.0) 976 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2021-03-24T21:57:58.587 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:58 smithi142 conmon[20125]: audit 2021-03-24T21:57:57.932178+0000 mon.a (mon.0) 977 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:57:58.587 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:58 smithi142 conmon[20125]: audit 2021-03-24T21:57:58.247015+0000 mon.a (mon.0) 978 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:57:58.587 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:58 smithi142 conmon[20125]: audit 2021-03-24T21:57:58.267653+0000 mon.a (mon.0) 979 : audit [INF] from='client.? v1:172.21.15.125:0/1210241986' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:57:58.603 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:58.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:58.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:57:58.710 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:58.768 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:57:58.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:58.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:57:58.873 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:58.930 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:58.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:58.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:57:59.035 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:59.090 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:57:59.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:59.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:57:59.197 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:59.235 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2021-03-24T21:57:59.235 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2021-03-24T21:57:59.235 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2021-03-24T21:57:59.236 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2021-03-24T21:57:59.236 INFO:teuthology.orchestra.run.smithi142.stdout:} 2021-03-24T21:57:59.256 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:59.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:59.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:57:59.317 INFO:tasks.rados.rados.0.smithi125.stdout:1056: done (0 left) 2021-03-24T21:57:59.317 INFO:tasks.rados.rados.0.smithi125.stdout:1057: write oid 14 current snap is 102 2021-03-24T21:57:59.317 INFO:tasks.rados.rados.0.smithi125.stdout:1057: seq_num 394 ranges {621958=782109,2089670=532288} 2021-03-24T21:57:59.320 INFO:tasks.rados.rados.0.smithi125.stdout:1057: writing smithi12556470-14 from 621958 to 1404067 tid 1 2021-03-24T21:57:59.328 INFO:tasks.rados.rados.0.smithi125.stdout:1057: writing smithi12556470-14 from 2089670 to 2621958 tid 2 2021-03-24T21:57:59.328 INFO:tasks.rados.rados.0.smithi125.stdout:1058: write (excl) oid 31 current snap is 102 2021-03-24T21:57:59.328 INFO:tasks.rados.rados.0.smithi125.stdout:1058: seq_num 395 ranges {638765=457358,1748152=604803} 2021-03-24T21:57:59.334 INFO:tasks.rados.rados.0.smithi125.stdout:1058: writing smithi12556470-31 from 638765 to 1096123 tid 1 2021-03-24T21:57:59.343 INFO:tasks.rados.rados.0.smithi125.stdout:1058: writing smithi12556470-31 from 1748152 to 2352955 tid 2 2021-03-24T21:57:59.343 INFO:tasks.rados.rados.0.smithi125.stdout:1059: copy_from oid 41 from oid 17 current snap is 102 2021-03-24T21:57:59.343 INFO:tasks.rados.rados.0.smithi125.stdout:1057: finishing write tid 1 to smithi12556470-14 2021-03-24T21:57:59.344 INFO:tasks.rados.rados.0.smithi125.stdout:1057: finishing write tid 2 to smithi12556470-14 2021-03-24T21:57:59.344 INFO:tasks.rados.rados.0.smithi125.stdout:1057: finishing write tid 3 to smithi12556470-14 2021-03-24T21:57:59.344 INFO:tasks.rados.rados.0.smithi125.stdout:1057: finishing write tid 4 to smithi12556470-14 2021-03-24T21:57:59.344 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 14 v 306 (ObjNum 394 snap 102 seq_num 394) dirty exists 2021-03-24T21:57:59.344 INFO:tasks.rados.rados.0.smithi125.stdout:1057: left oid 14 (ObjNum 394 snap 102 seq_num 394) 2021-03-24T21:57:59.345 INFO:tasks.rados.rados.0.smithi125.stdout:1058: finishing write tid 1 to smithi12556470-31 2021-03-24T21:57:59.345 INFO:tasks.rados.rados.0.smithi125.stdout:1057: done (2 left) 2021-03-24T21:57:59.345 INFO:tasks.rados.rados.0.smithi125.stdout:1060: write oid 43 current snap is 102 2021-03-24T21:57:59.345 INFO:tasks.rados.rados.0.smithi125.stdout:1060: seq_num 396 ranges {655572=648960,1922987=732585} 2021-03-24T21:57:59.349 INFO:tasks.rados.rados.0.smithi125.stdout:1060: writing smithi12556470-43 from 655572 to 1304532 tid 1 2021-03-24T21:57:59.357 INFO:tasks.rados.rados.0.smithi125.stdout:1060: writing smithi12556470-43 from 1922987 to 2655572 tid 2 2021-03-24T21:57:59.357 INFO:tasks.rados.rados.0.smithi125.stdout:1061: read oid 38 snap -1 2021-03-24T21:57:59.358 INFO:tasks.rados.rados.0.smithi125.stdout:1061: expect deleted 2021-03-24T21:57:59.358 INFO:tasks.rados.rados.0.smithi125.stdout:1062: snap_remove snap 101 2021-03-24T21:57:59.365 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:59.417 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:57:59.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:59.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:57:59.523 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:59.525 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:59 smithi125 conmon[24927]: cluster 2021-03-24T21:57:58.133337+0000 osd.5 ( 2021-03-24T21:57:59.525 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:59 smithi125 conmon[24927]: osd.5) 79 : cluster [DBG] 3.d scrub ok 2021-03-24T21:57:59.525 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:59 smithi125 conmon[24927]: cluster 2021-03-24T21:57:58.225152+0000 2021-03-24T21:57:59.525 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:59 smithi125 conmon[24927]: osd.2 (osd.2) 61 : cluster [DBG] 6.8 scrub ok 2021-03-24T21:57:59.526 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:59 smithi125 conmon[24927]: cluster 2021-03-24T21:57:58.301590+0000 mon.a 2021-03-24T21:57:59.526 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:59 smithi125 conmon[24927]: (mon.0) 980 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T21:57:59.526 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:59 smithi125 conmon[24927]: audit 2021-03-24T21:57:58.308297+0000 mon.a (mon.0 2021-03-24T21:57:59.526 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:59 smithi125 conmon[24927]: ) 981 : audit [INF] from='client.? v1:172.21.15.125:0/1210241986' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T21:57:59.527 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:59 smithi125 conmon[24927]: cluster 2021-03-24T21:57:59.527 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:59 smithi125 conmon[24927]: 2021-03-24T21:57:58.308470+0000 mon.a (mon 2021-03-24T21:57:59.527 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:59 smithi125 conmon[24927]: .0) 982 : cluster [DBG] osdmap e308: 8 total, 8 up, 6 in 2021-03-24T21:57:59.527 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:59 smithi125 conmon[24927]: cluster 2021-03-24T21:57:59.528 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:59 smithi125 conmon[24927]: 2021-03-24T21:57:58 2021-03-24T21:57:59.528 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:59 smithi125 conmon[24927]: .491811+0000 osd.4 (osd.4) 81 : cluster [DBG] 3.3 deep-scrub ok 2021-03-24T21:57:59.528 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:59 smithi125 conmon[24927]: audit 2021-03-24T21:57:58. 2021-03-24T21:57:59.528 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:57:59 smithi125 conmon[24927]: 542664+0000 mon.a (mon.0) 983 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:57:59.529 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:59 smithi125 conmon[19936]: cluster 2021-03-24T21:57 2021-03-24T21:57:59.529 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:59 smithi125 conmon[19936]: :58.133337+0000 osd.5 (osd.5) 79 : cluster [DBG] 3.d scrub ok 2021-03-24T21:57:59.529 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:59 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:57:59.529 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:59 smithi125 conmon[19936]: 03-24T21:57:58.225152+0000 osd.2 (osd.2) 61 : cluster [DBG] 6.8 scrub ok 2021-03-24T21:57:59.530 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:59 smithi125 conmon[19936]: cluster 2021-03-24T21:57:59.530 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:59 smithi125 conmon[19936]: 2021-03-24T21:57:58.301590+0000 2021-03-24T21:57:59.530 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:59 smithi125 conmon[19936]: mon.a (mon.0) 980 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T21:57:59.530 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:59 smithi125 conmon[19936]: 2021-03-24T21:57:59.531 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:59 smithi125 conmon[19936]: audit 2021-03-24T21: 2021-03-24T21:57:59.531 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:59 smithi125 conmon[19936]: 57:58.308297+0000 mon.a (mon.0) 2021-03-24T21:57:59.531 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:59 smithi125 conmon[19936]: 981 : audit [INF] from='client.? v1:172.21.15.125:0/1210241986' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T21:57:59.531 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:59 smithi125 conmon[19936]: cluster 2021-03-24T21:57:59.531 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:59 smithi125 conmon[19936]: 2021-03-24T21:57:58.308470+0000 2021-03-24T21:57:59.532 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:59 smithi125 conmon[19936]: mon.a (mon.0) 982 : cluster [DBG] osdmap e308: 8 total, 8 up, 6 in 2021-03-24T21:57:59.532 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:59 smithi125 conmon[19936]: 2021-03-24T21:57:59.532 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:59 smithi125 conmon[19936]: cluster 2021-03-24T21:57:59.532 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:59 smithi125 conmon[19936]: 2021-03-24T21:57:58.491811+0000 2021-03-24T21:57:59.532 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:59 smithi125 conmon[19936]: osd.4 (osd.4) 81 : cluster [DBG] 2021-03-24T21:57:59.533 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:59 smithi125 conmon[19936]: 3.3 deep-scrub ok 2021-03-24T21:57:59.533 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:59 smithi125 conmon[19936]: audit 2021-03-24T21: 2021-03-24T21:57:59.533 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:59 smithi125 conmon[19936]: 57:58.542664+0000 mon.a (mon.0) 983 2021-03-24T21:57:59.533 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:57:59 smithi125 conmon[19936]: : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:57:59.587 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:57:59.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:59.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:57:59.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:59 smithi142 conmon[20125]: cluster 2021-03-24T21:57:58.133337+0000 osd.5 (osd.5) 79 : cluster [DBG] 3.d scrub ok 2021-03-24T21:57:59.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:59 smithi142 conmon[20125]: cluster 2021-03-24T21:57:58.225152+0000 osd.2 (osd.2) 61 : cluster [DBG] 6.8 scrub ok 2021-03-24T21:57:59.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:59 smithi142 conmon[20125]: cluster 2021-03-24T21:57:58.301590+0000 mon.a (mon.0) 980 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T21:57:59.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:59 smithi142 conmon[20125]: audit 2021-03-24T21:57:58.308297+0000 mon.a (mon.0) 981 : audit [INF] from='client.? v1:172.21.15.125:0/1210241986' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T21:57:59.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:59 smithi142 conmon[20125]: cluster 2021-03-24T21:57:58.308470+0000 mon.a (mon. 2021-03-24T21:57:59.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:59 smithi142 conmon[20125]: 0) 982 : cluster [DBG] osdmap e308: 8 total, 8 up, 6 in 2021-03-24T21:57:59.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:59 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:57:59.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:59 smithi142 conmon[20125]: T21:57:58.491811+0000 osd.4 (osd.4) 81 : cluster [DBG] 3.3 deep-scrub ok 2021-03-24T21:57:59.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:59 smithi142 conmon[20125]: audit 2021-03-24T21: 2021-03-24T21:57:59.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:57:59 smithi142 conmon[20125]: 57:58.542664+0000 mon.a (mon.0) 983 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:57:59.694 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:59.754 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:59.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:59.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:57:59.861 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:59.922 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:57:59.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:57:59.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:00.029 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:00.086 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:00.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:00.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:58:00.172 INFO:tasks.rados.rados.0.smithi125.stdout:1062: done (4 left) 2021-03-24T21:58:00.172 INFO:tasks.rados.rados.0.smithi125.stdout:1063: write oid 15 current snap is 102 2021-03-24T21:58:00.172 INFO:tasks.rados.rados.0.smithi125.stdout:1063: seq_num 397 ranges {672379=724209,1865116=608472} 2021-03-24T21:58:00.183 INFO:tasks.rados.rados.0.smithi125.stdout:1063: writing smithi12556470-15 from 672379 to 1396588 tid 1 2021-03-24T21:58:00.191 INFO:tasks.rados.rados.0.smithi125.stdout:1063: writing smithi12556470-15 from 1865116 to 2473588 tid 2 2021-03-24T21:58:00.192 INFO:tasks.rados.rados.0.smithi125.stdout:1064: rollback oid 49 current snap is 102 2021-03-24T21:58:00.192 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 49 to 100 2021-03-24T21:58:00.192 INFO:tasks.rados.rados.0.smithi125.stdout:1058: finishing write tid 2 to smithi12556470-31 2021-03-24T21:58:00.192 INFO:tasks.rados.rados.0.smithi125.stdout:1058: finishing write tid 3 to smithi12556470-31 2021-03-24T21:58:00.193 INFO:tasks.rados.rados.0.smithi125.stdout:1058: finishing write tid 4 to smithi12556470-31 2021-03-24T21:58:00.193 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 31 v 310 (ObjNum 395 snap 102 seq_num 395) dirty exists 2021-03-24T21:58:00.193 INFO:tasks.rados.rados.0.smithi125.stdout:1058: left oid 31 (ObjNum 395 snap 102 seq_num 395) 2021-03-24T21:58:00.193 INFO:tasks.rados.rados.0.smithi125.stdout:1060: finishing write tid 1 to smithi12556470-43 2021-03-24T21:58:00.193 INFO:tasks.rados.rados.0.smithi125.stdout:1059: finishing copy_from to smithi12556470-41 2021-03-24T21:58:00.194 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 41 v 299 (ObjNum 385 snap 99 seq_num 385) dirty exists 2021-03-24T21:58:00.194 INFO:tasks.rados.rados.0.smithi125.stdout:1059: finishing copy_from racing read to smithi12556470-41 2021-03-24T21:58:00.194 INFO:tasks.rados.rados.0.smithi125.stdout:1058: done (5 left) 2021-03-24T21:58:00.194 INFO:tasks.rados.rados.0.smithi125.stdout:1059: done (4 left) 2021-03-24T21:58:00.195 INFO:tasks.rados.rados.0.smithi125.stdout:1065: delete oid 46 current snap is 102 2021-03-24T21:58:00.195 INFO:tasks.rados.rados.0.smithi125.stdout:1060: finishing write tid 2 to smithi12556470-43 2021-03-24T21:58:00.195 INFO:tasks.rados.rados.0.smithi125.stdout:1060: finishing write tid 3 to smithi12556470-43 2021-03-24T21:58:00.195 INFO:tasks.rados.rados.0.smithi125.stdout:1060: finishing write tid 4 to smithi12556470-43 2021-03-24T21:58:00.195 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 43 v 313 (ObjNum 396 snap 102 seq_num 396) dirty exists 2021-03-24T21:58:00.196 INFO:tasks.rados.rados.0.smithi125.stdout:1060: left oid 43 (ObjNum 396 snap 102 seq_num 396) 2021-03-24T21:58:00.196 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:00.200 INFO:tasks.rados.rados.0.smithi125.stdout:1060: done (4 left) 2021-03-24T21:58:00.200 INFO:tasks.rados.rados.0.smithi125.stdout:1061: done (3 left) 2021-03-24T21:58:00.201 INFO:tasks.rados.rados.0.smithi125.stdout:1065: done (2 left) 2021-03-24T21:58:00.201 INFO:tasks.rados.rados.0.smithi125.stdout:1066: copy_from oid 47 from oid 37 current snap is 102 2021-03-24T21:58:00.201 INFO:tasks.rados.rados.0.smithi125.stdout:1067: read oid 35 snap -1 2021-03-24T21:58:00.201 INFO:tasks.rados.rados.0.smithi125.stdout:1067: expect (ObjNum 374 snap 96 seq_num 374) 2021-03-24T21:58:00.202 INFO:tasks.rados.rados.0.smithi125.stdout:1068: snap_remove snap 99 2021-03-24T21:58:00.243 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:00.245 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.5 dump_historic_ops 2021-03-24T21:58:00.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:00.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:58:00.349 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:00.404 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:00.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:00.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:00.503 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:00 smithi125 conmon[24927]: cluster 2021-03-24T21:57:58. 2021-03-24T21:58:00.504 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:00 smithi125 conmon[24927]: 768662+0000 osd.0 (osd.0) 78 : cluster [DBG] 6.16 scrub ok 2021-03-24T21:58:00.504 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:00 smithi125 conmon[24927]: cluster 2021-03-24T21:58:00.504 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:00 smithi125 conmon[24927]: 2021-03-24T21:57:58.785840+0000 mgr.y (mgr.14144) 422 : cluster [DBG] 2021-03-24T21:58:00.504 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:00 smithi125 conmon[24927]: pgmap v625: 168 pgs: 2 peering, 166 active+clean; 112 MiB data, 463 MiB used, 536 GiB / 536 GiB avail; 255 B/s rd, 1.2 MiB/s wr, 2 op/s 2021-03-24T21:58:00.504 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:00 smithi125 conmon[24927]: cluster 2021-03-24T21:57: 2021-03-24T21:58:00.505 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:00 smithi125 conmon[24927]: 59.311314+0000 mon.a (mon.0) 984 : cluster 2021-03-24T21:58:00.505 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:00 smithi125 conmon[24927]: [DBG] osdmap e309: 8 total, 8 up, 6 in 2021-03-24T21:58:00.505 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:00 smithi125 conmon[24927]: cluster 2021-03-24T21:57 2021-03-24T21:58:00.505 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:00 smithi125 conmon[24927]: :59.461840+0000 osd.4 (osd.4) 82 : 2021-03-24T21:58:00.505 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:00 smithi125 conmon[24927]: cluster [DBG] 5.1 scrub ok 2021-03-24T21:58:00.506 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:00 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:58:00.506 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:00 smithi125 conmon[24927]: T21:58:00.168750+0000 mon.a (mon. 2021-03-24T21:58:00.506 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:00 smithi125 conmon[24927]: 0) 985 : cluster [DBG] osdmap e310: 8 total, 8 up, 6 in 2021-03-24T21:58:00.506 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:00 smithi125 conmon[19936]: cluster 2021-03-24T21:57:58 2021-03-24T21:58:00.507 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:00 smithi125 conmon[19936]: .768662+0000 osd.0 (osd.0) 78 : cluster [DBG] 6.16 scrub ok 2021-03-24T21:58:00.507 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:00 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:58:00.507 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:00 smithi125 conmon[19936]: :57:58.785840+0000 mgr.y (mgr.14144) 422 : cluster [DBG] pgmap v625: 168 pgs: 2 peering, 166 active+clean; 112 MiB data, 463 MiB used, 536 GiB / 536 GiB avail; 255 B/s rd, 1.2 MiB/s wr, 2 op/s 2021-03-24T21:58:00.507 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:00 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:58:00.507 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:00 smithi125 conmon[19936]: 24T21:57:59.311314+0000 mon.a (mon 2021-03-24T21:58:00.508 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:00 smithi125 conmon[19936]: .0) 984 : cluster [DBG] osdmap e309: 8 total, 8 up, 6 in 2021-03-24T21:58:00.508 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:00 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:58:00.508 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:00 smithi125 conmon[19936]: -03-24T21:57:59.461840+0000 osd.4 2021-03-24T21:58:00.508 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:00 smithi125 conmon[19936]: (osd.4) 82 : cluster [DBG] 5.1 scrub ok 2021-03-24T21:58:00.508 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:00 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:58:00.509 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:00 smithi125 conmon[19936]: -03-24T21:58:00.168750+0000 mon.a 2021-03-24T21:58:00.509 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:00 smithi125 conmon[19936]: (mon.0) 985 : cluster [DBG] osdmap e310: 8 total, 8 up, 6 in 2021-03-24T21:58:00.510 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:00.521 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:00 smithi142 conmon[20125]: cluster 2021-03-24T21 2021-03-24T21:58:00.521 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:00 smithi142 conmon[20125]: :57:58.768662+0000 osd.0 (osd.0) 78 : cluster [DBG] 6.16 scrub ok 2021-03-24T21:58:00.521 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:00 smithi142 conmon[20125]: cluster 2021-03-24T21:58:00.521 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:00 smithi142 conmon[20125]: 2021-03-24T 2021-03-24T21:58:00.522 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:00 smithi142 conmon[20125]: 21:57:58 2021-03-24T21:58:00.522 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:00 smithi142 conmon[20125]: .785840+0000 mgr.y 2021-03-24T21:58:00.522 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:00 smithi142 conmon[20125]: (mgr.14144) 2021-03-24T21:58:00.522 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:00 smithi142 conmon[20125]: 422 : cluster [DBG] 2021-03-24T21:58:00.522 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:00 smithi142 conmon[20125]: pgmap v625: 168 pgs: 2 peering, 166 active+clean; 112 MiB data, 463 MiB used, 536 GiB / 536 GiB avail; 255 B/s rd, 1.2 MiB/s wr, 2 op/s 2021-03-24T21:58:00.523 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:00 smithi142 conmon[20125]: cluster 2021-03-24T21:58:00.523 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:00 smithi142 conmon[20125]: 2021-03-24T 2021-03-24T21:58:00.523 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:00 smithi142 conmon[20125]: 21:57:59 2021-03-24T21:58:00.523 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:00 smithi142 conmon[20125]: .311314+0000 mon.a 2021-03-24T21:58:00.523 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:00 smithi142 conmon[20125]: (mon.0 2021-03-24T21:58:00.524 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:00 smithi142 conmon[20125]: ) 984 : cluster 2021-03-24T21:58:00.524 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:00 smithi142 conmon[20125]: [DBG] osdmap e309: 8 total, 8 up, 6 in 2021-03-24T21:58:00.524 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:00 smithi142 conmon[20125]: cluster 2021-03-24T21:58:00.524 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:00 smithi142 conmon[20125]: 2021-03-24T 2021-03-24T21:58:00.524 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:00 smithi142 conmon[20125]: 21:57:59 2021-03-24T21:58:00.525 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:00 smithi142 conmon[20125]: .461840+0000 2021-03-24T21:58:00.525 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:00 smithi142 conmon[20125]: osd.4 (osd.4 2021-03-24T21:58:00.525 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:00 smithi142 conmon[20125]: ) 82 : cluster 2021-03-24T21:58:00.525 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:00 smithi142 conmon[20125]: [DBG] 5.1 scrub ok 2021-03-24T21:58:00.525 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:00 smithi142 conmon[20125]: cluster 2021-03-24T21:58:00.168750+0000 2021-03-24T21:58:00.526 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:00 smithi142 conmon[20125]: mon.a (mon.0) 985 : cluster [DBG] osdmap e310: 8 total, 8 up, 6 in 2021-03-24T21:58:00.555 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T21:58:00.563 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:00.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:00.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:58:00.667 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:58:00.724 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:58:00.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:00.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:58:00.771 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T21:58:00.794 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T21:58:00.823 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T21:58:00.830 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:00.855 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T21:58:00.881 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T21:58:00.886 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:00.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:00.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:00.904 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T21:58:00.929 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T21:58:00.952 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T21:58:00.954 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-24T21:58:00.955 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-24T21:58:00.955 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-24T21:58:00.955 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-24T21:58:00.955 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-24T21:58:00.956 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-24T21:58:00.956 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-24T21:58:00.956 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-24T21:58:00.992 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:01.050 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:01.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:01.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:58:01.157 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:01.189 INFO:tasks.rados.rados.0.smithi125.stdout:1068: done (4 left) 2021-03-24T21:58:01.189 INFO:tasks.rados.rados.0.smithi125.stdout:1069: read oid 13 snap -1 2021-03-24T21:58:01.189 INFO:tasks.rados.rados.0.smithi125.stdout:1069: expect (ObjNum 339 snap 86 seq_num 339) 2021-03-24T21:58:01.189 INFO:tasks.rados.rados.0.smithi125.stdout:1070: write (excl) oid 43 current snap is 102 2021-03-24T21:58:01.190 INFO:tasks.rados.rados.0.smithi125.stdout:1070: seq_num 399 ranges {705993=474707,1582080=612141} 2021-03-24T21:58:01.198 INFO:tasks.rados.rados.0.smithi125.stdout:1070: writing smithi12556470-43 from 705993 to 1180700 tid 1 2021-03-24T21:58:01.206 INFO:tasks.rados.rados.0.smithi125.stdout:1070: writing smithi12556470-43 from 1582080 to 2194221 tid 2 2021-03-24T21:58:01.206 INFO:tasks.rados.rados.0.smithi125.stdout:1063: finishing write tid 1 to smithi12556470-15 2021-03-24T21:58:01.206 INFO:tasks.rados.rados.0.smithi125.stdout:1064: finishing rollback tid 0 to smithi12556470-49 2021-03-24T21:58:01.207 INFO:tasks.rados.rados.0.smithi125.stdout:1063: finishing write tid 2 to smithi12556470-15 2021-03-24T21:58:01.207 INFO:tasks.rados.rados.0.smithi125.stdout:1064: finishing rollback tid 1 to smithi12556470-49 2021-03-24T21:58:01.207 INFO:tasks.rados.rados.0.smithi125.stdout:1063: finishing write tid 3 to smithi12556470-15 2021-03-24T21:58:01.207 INFO:tasks.rados.rados.0.smithi125.stdout:1064: finishing rollback tid 2 to smithi12556470-49 2021-03-24T21:58:01.207 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 49 v 328 (ObjNum 391 snap 100 seq_num 391) dirty exists 2021-03-24T21:58:01.208 INFO:tasks.rados.rados.0.smithi125.stdout:1066: finishing copy_from to smithi12556470-47 2021-03-24T21:58:01.208 INFO:tasks.rados.rados.0.smithi125.stdout:1066: got expected ENOENT (src dne) 2021-03-24T21:58:01.208 INFO:tasks.rados.rados.0.smithi125.stdout:1063: finishing write tid 4 to smithi12556470-15 2021-03-24T21:58:01.208 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 15 v 321 (ObjNum 397 snap 102 seq_num 397) dirty exists 2021-03-24T21:58:01.209 INFO:tasks.rados.rados.0.smithi125.stdout:1063: left oid 15 (ObjNum 397 snap 102 seq_num 397) 2021-03-24T21:58:01.209 INFO:tasks.rados.rados.0.smithi125.stdout:1066: finishing copy_from racing read to smithi12556470-47 2021-03-24T21:58:01.209 INFO:tasks.rados.rados.0.smithi125.stdout:1067: expect (ObjNum 374 snap 96 seq_num 374) 2021-03-24T21:58:01.223 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:01.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:01.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:01.311 INFO:tasks.rados.rados.0.smithi125.stdout:1069: expect (ObjNum 339 snap 86 seq_num 339) 2021-03-24T21:58:01.327 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:01.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:01 smithi142 conmon[20125]: cluster 2021-03-24T21:57:59. 2021-03-24T21:58:01.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:01 smithi142 conmon[20125]: 979762+0000 osd.7 (osd.7) 71 : cluster [DBG] 6.12 deep-scrub ok 2021-03-24T21:58:01.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:01 smithi142 conmon[20125]: audit 2021-03-24T21:58:00.746097 2021-03-24T21:58:01.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:01 smithi142 conmon[20125]: +0000 mon.c (mon.2) 48 : audit [DBG] from='client.? v1:172.21.15.125:0/180289416' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:58:01.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:01 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:58:01.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:01 smithi142 conmon[20125]: 21:58:01.186250+0000 mon.a (mon.0) 2021-03-24T21:58:01.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:01 smithi142 conmon[20125]: 986 : cluster [DBG] osdmap e311: 8 total, 8 up, 6 in 2021-03-24T21:58:01.372 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:01.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:01.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:58:01.479 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:58:01.511 INFO:tasks.rados.rados.0.smithi125.stdout:1070: finishing write tid 1 to smithi12556470-43 2021-03-24T21:58:01.512 INFO:tasks.rados.rados.0.smithi125.stdout:1070: finishing write tid 2 to smithi12556470-43 2021-03-24T21:58:01.512 INFO:tasks.rados.rados.0.smithi125.stdout:1070: finishing write tid 3 to smithi12556470-43 2021-03-24T21:58:01.512 INFO:tasks.rados.rados.0.smithi125.stdout:1070: finishing write tid 4 to smithi12556470-43 2021-03-24T21:58:01.512 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 43 v 318 (ObjNum 399 snap 102 seq_num 399) dirty exists 2021-03-24T21:58:01.513 INFO:tasks.rados.rados.0.smithi125.stdout:1070: left oid 43 (ObjNum 399 snap 102 seq_num 399) 2021-03-24T21:58:01.513 INFO:tasks.rados.rados.0.smithi125.stdout:1063: done (5 left) 2021-03-24T21:58:01.513 INFO:tasks.rados.rados.0.smithi125.stdout:1064: done (4 left) 2021-03-24T21:58:01.513 INFO:tasks.rados.rados.0.smithi125.stdout:1066: done (3 left) 2021-03-24T21:58:01.514 INFO:tasks.rados.rados.0.smithi125.stdout:1067: done (2 left) 2021-03-24T21:58:01.514 INFO:tasks.rados.rados.0.smithi125.stdout:1069: done (1 left) 2021-03-24T21:58:01.514 INFO:tasks.rados.rados.0.smithi125.stdout:1070: done (0 left) 2021-03-24T21:58:01.514 INFO:tasks.rados.rados.0.smithi125.stdout:1071: write oid 27 current snap is 102 2021-03-24T21:58:01.515 INFO:tasks.rados.rados.0.smithi125.stdout:1071: seq_num 400 ranges {722800=549956,2040562=682238} 2021-03-24T21:58:01.527 INFO:tasks.rados.rados.0.smithi125.stdout:1071: writing smithi12556470-27 from 722800 to 1272756 tid 1 2021-03-24T21:58:01.535 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:58:01.536 INFO:tasks.rados.rados.0.smithi125.stdout:1071: writing smithi12556470-27 from 2040562 to 2722800 tid 2 2021-03-24T21:58:01.536 INFO:tasks.rados.rados.0.smithi125.stdout:1072: read oid 36 snap -1 2021-03-24T21:58:01.536 INFO:tasks.rados.rados.0.smithi125.stdout:1072: expect (ObjNum 150 snap 30 seq_num 150) 2021-03-24T21:58:01.536 INFO:tasks.rados.rados.0.smithi125.stdout:1073: snap_create 2021-03-24T21:58:01.537 INFO:tasks.rados.rados.0.smithi125.stdout:1071: finishing write tid 1 to smithi12556470-27 2021-03-24T21:58:01.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:01.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:58:01.541 INFO:tasks.rados.rados.0.smithi125.stdout:1072: expect (ObjNum 150 snap 30 seq_num 150) 2021-03-24T21:58:01.642 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:01.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:01 smithi125 conmon[19936]: cluster 2021-03-24T21:57:59. 2021-03-24T21:58:01.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:01 smithi125 conmon[19936]: 979762+0000 osd.7 (osd.7) 71 : cluster [DBG] 6.12 deep-scrub ok 2021-03-24T21:58:01.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:01 smithi125 conmon[19936]: audit 2021-03-24T21:58:00.746097+0000 mon.c (mon.2) 48 : audit 2021-03-24T21:58:01.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:01 smithi125 conmon[19936]: [DBG] from='client.? v1:172.21.15.125:0/180289416' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:58:01.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:01 smithi125 conmon[19936]: cluster 2021-03-24T21:58:01.186250+0000 2021-03-24T21:58:01.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:01 smithi125 conmon[19936]: mon.a (mon.0) 986 : cluster [DBG] osdmap e311: 8 total, 8 up, 6 in 2021-03-24T21:58:01.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:01 smithi125 conmon[24927]: cluster 2021-03-24T21:57 2021-03-24T21:58:01.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:01 smithi125 conmon[24927]: :59.979762+0000 osd.7 (osd.7) 71 : cluster [DBG] 6.12 deep-scrub ok 2021-03-24T21:58:01.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:01 smithi125 conmon[24927]: audit 2021 2021-03-24T21:58:01.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:01 smithi125 conmon[24927]: -03-24T21:58:00.746097+0000 mon.c (mon. 2021-03-24T21:58:01.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:01 smithi125 conmon[24927]: 2) 48 : audit [DBG] from='client.? v1:172.21.15.125:0/180289416' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:58:01.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:01 smithi125 conmon[24927]: cluster 2021-03-24T21:58: 2021-03-24T21:58:01.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:01 smithi125 conmon[24927]: 01.186250+0000 mon.a (mon.0) 986 : cluster [DBG] 2021-03-24T21:58:01.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:01 smithi125 conmon[24927]: osdmap e311: 8 total, 8 up, 6 in 2021-03-24T21:58:01.675 INFO:tasks.rados.rados.0.smithi125.stdout:1071: finishing write tid 2 to smithi12556470-27 2021-03-24T21:58:01.676 INFO:tasks.rados.rados.0.smithi125.stdout:1071: finishing write tid 3 to smithi12556470-27 2021-03-24T21:58:01.676 INFO:tasks.rados.rados.0.smithi125.stdout:1071: finishing write tid 4 to smithi12556470-27 2021-03-24T21:58:01.676 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 27 v 275 (ObjNum 400 snap 102 seq_num 400) dirty exists 2021-03-24T21:58:01.676 INFO:tasks.rados.rados.0.smithi125.stdout:1071: left oid 27 (ObjNum 400 snap 102 seq_num 400) 2021-03-24T21:58:01.690 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:01.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:01.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:01.797 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:01.856 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:01.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:01.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:58:01.964 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:02.017 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:02.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:02.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:58:02.124 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:02.181 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:02.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:02.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:58:02.287 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:02.297 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd set noscrub 2021-03-24T21:58:02.337 INFO:tasks.rados.rados.0.smithi125.stdout:1071: done (2 left) 2021-03-24T21:58:02.337 INFO:tasks.rados.rados.0.smithi125.stdout:1072: done (1 left) 2021-03-24T21:58:02.338 INFO:tasks.rados.rados.0.smithi125.stdout:1073: done (0 left) 2021-03-24T21:58:02.338 INFO:tasks.rados.rados.0.smithi125.stdout:1074: snap_remove snap 100 2021-03-24T21:58:02.349 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:02.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:02.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:02.427 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:02 smithi142 conmon[20125]: cluster 2021-03-24T21:58: 2021-03-24T21:58:02.428 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:02 smithi142 conmon[20125]: 00.786473+0000 mgr.y (mgr.14144) 423 : cluster [DBG] pgmap v628: 168 pgs: 2 peering, 166 active+clean; 116 MiB data, 470 MiB used, 536 GiB / 536 GiB avail; 10 MiB/s rd, 5.1 MiB/s wr, 29 op/s 2021-03-24T21:58:02.453 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:02.502 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:02.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:02.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:58:02.608 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:02.618 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:02 smithi125 conmon[19936]: cluster 2021-03-24T21:58:00 2021-03-24T21:58:02.618 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:02 smithi125 conmon[19936]: .786473+0000 mgr.y (mgr.14144) 423 : cluster [DBG] pgmap v628: 168 pgs: 2 peering, 166 active+clean; 116 MiB data, 470 MiB used, 536 GiB / 536 GiB avail; 10 MiB/s rd, 5.1 MiB/s wr, 29 op/s 2021-03-24T21:58:02.618 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:02 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:58:02.619 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:02 smithi125 conmon[24927]: 24T21:58:00.786473+0000 mgr.y (mgr.14144) 423 2021-03-24T21:58:02.619 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:02 smithi125 conmon[24927]: : cluster [DBG] pgmap v628: 168 pgs: 2 peering, 166 active+clean; 116 MiB data, 470 MiB used, 536 GiB / 536 GiB avail; 10 MiB/s rd, 5.1 MiB/s wr, 29 op/s 2021-03-24T21:58:02.619 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:02 smithi125 conmon[24927]: 2021-03-24T21:58:02.635 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:58:02.663 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:02.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:02.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:02.673 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 7] out_osds: [6, 1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-03-24T21:58:02.673 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-24T21:58:02.674 INFO:tasks.thrashosds.thrasher:inject_pause on 5 2021-03-24T21:58:02.674 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2021-03-24T21:58:02.674 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2021-03-24T21:58:02.675 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.5 config set filestore_inject_stall 3 2021-03-24T21:58:02.719 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2021-03-24T21:58:02.720 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T21:58:02.768 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:02.833 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:02.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:02.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:02.938 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:02.985 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:02.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:02.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:03.090 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:03.146 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:03.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:03.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:58:03.253 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:03.311 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:03.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:03.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:58:03.352 INFO:tasks.rados.rados.0.smithi125.stdout:1074: done (0 left) 2021-03-24T21:58:03.352 INFO:tasks.rados.rados.0.smithi125.stdout:1075: snap_create 2021-03-24T21:58:03.418 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:03.474 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:03.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:03.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:58:03.580 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:03.591 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:03 smithi142 conmon[20125]: cluster 2021-03-24T21:58 2021-03-24T21:58:03.592 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:03 smithi142 conmon[20125]: :01.817476+0000 osd.0 (osd.0) 79 : cluster [DBG] 2.2 scrub ok 2021-03-24T21:58:03.592 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:03 smithi142 conmon[20125]: cluster 2021-03-24T21:58:02. 2021-03-24T21:58:03.592 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:03 smithi142 conmon[20125]: 014892+0000 osd.7 (osd.7) 72 : cluster [DBG] 3.13 scrub ok 2021-03-24T21:58:03.592 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:03 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:58:03.593 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:03 smithi142 conmon[20125]: 24T21:58:02.335695+0000 mon.a (mon.0) 987 : 2021-03-24T21:58:03.593 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:03 smithi142 conmon[20125]: cluster [DBG] osdmap e312: 8 total, 8 up, 6 in 2021-03-24T21:58:03.593 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:03 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:58:03.593 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:03 smithi142 conmon[20125]: T21:58:02.548149+0000 osd.4 (osd.4) 2021-03-24T21:58:03.593 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:03 smithi142 conmon[20125]: 83 : cluster [DBG] 4.13 scrub ok 2021-03-24T21:58:03.639 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:03.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:03.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:58:03.672 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.6 dump_ops_in_flight 2021-03-24T21:58:03.703 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T21:58:03.745 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:03.796 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:03.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:03.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:58:03.901 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:03.910 INFO:journalctl@ceph.osd.0.smithi125.stdout:Mar 24 21:58:03 smithi125 conmon[28026]: debug 2021-03-24T21:58:03.447+0000 7f61ba483700 -1 osd.0 pg_epoch: 313 pg[7.f( v 311'118 (0'0,311'118] local-lis/les=275/276 n=1 ec=99/99 lis/c=275/275 les/c/f=276/276/0 sis=275) [0,3,4] r=0 lpr=275 crt=311'118 lcod 310'116 mlcod 310'116 active+clean+snaptrim trimq=[c9~1] ps=[c2~1,c5~1,c7~1,ca~1]] removing snap head 2021-03-24T21:58:03.911 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:03 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:58:03.911 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:03 smithi125 conmon[19936]: 58:01.817476+0000 osd.0 (osd.0) 79 : cluster [DBG] 2.2 scrub ok 2021-03-24T21:58:03.911 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:03 smithi125 conmon[19936]: cluster 2021-03-24T21:58:03.911 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:03 smithi125 conmon[19936]: 2021-03-24T21:58: 2021-03-24T21:58:03.911 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:03 smithi125 conmon[19936]: 02.014892+0000 osd.7 (osd.7 2021-03-24T21:58:03.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:03 smithi125 conmon[19936]: ) 72 : cluster [DBG] 3.13 scrub ok 2021-03-24T21:58:03.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:03 smithi125 conmon[19936]: cluster 2021-03-24T21:58:03.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:03 smithi125 conmon[19936]: 2021-03-24T21:58 2021-03-24T21:58:03.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:03 smithi125 conmon[19936]: :02.335695+0000 mon.a (mon. 2021-03-24T21:58:03.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:03 smithi125 conmon[19936]: 0) 987 : cluster [DBG] osdmap e312: 8 total, 8 up, 6 in 2021-03-24T21:58:03.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:03 smithi125 conmon[19936]: cluster 2021-03-24T21:58:03.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:03 smithi125 conmon[19936]: 2021- 2021-03-24T21:58:03.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:03 smithi125 conmon[19936]: 03-24T21:58:02. 2021-03-24T21:58:03.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:03 smithi125 conmon[19936]: 548149+0000 osd.4 (osd.4) 83 2021-03-24T21:58:03.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:03 smithi125 conmon[19936]: : cluster [DBG] 4.13 scrub ok 2021-03-24T21:58:03.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:03 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:58:03.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:03 smithi125 conmon[24927]: 24T21:58:01.817476+0000 osd.0 (osd.0) 79 : cluster [DBG] 2021-03-24T21:58:03.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:03 smithi125 conmon[24927]: 2.2 scrub ok 2021-03-24T21:58:03.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:03 smithi125 conmon[24927]: cluster 2021-03-24T21:58:03.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:03 smithi125 conmon[24927]: 2021-03-24T21 2021-03-24T21:58:03.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:03 smithi125 conmon[24927]: :58:02.014892+0000 2021-03-24T21:58:03.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:03 smithi125 conmon[24927]: osd.7 (osd.7) 72 : 2021-03-24T21:58:03.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:03 smithi125 conmon[24927]: cluster [DBG] 3.13 scrub ok 2021-03-24T21:58:03.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:03 smithi125 conmon[24927]: cluster 2021-03-24T21:58:03.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:03 smithi125 conmon[24927]: 2021-03-24T 2021-03-24T21:58:03.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:03 smithi125 conmon[24927]: 21:58:02.335695+0000 2021-03-24T21:58:03.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:03 smithi125 conmon[24927]: mon.a (mon.0) 987 : 2021-03-24T21:58:03.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:03 smithi125 conmon[24927]: cluster [DBG] osdmap e312: 8 total, 8 up, 6 in 2021-03-24T21:58:03.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:03 smithi125 conmon[24927]: cluster 2021-03-24T21:58:03.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:03 smithi125 conmon[24927]: 2021-03-24T21 2021-03-24T21:58:03.917 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:03 smithi125 conmon[24927]: :58:02.548149+0000 2021-03-24T21:58:03.917 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:03 smithi125 conmon[24927]: osd.4 (osd.4) 83 : cluster 2021-03-24T21:58:03.917 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:03 smithi125 conmon[24927]: [DBG] 4.13 scrub ok 2021-03-24T21:58:03.960 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:03.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:03.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:58:04.041 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-24T21:58:04.064 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:04.133 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:04.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:04.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:58:04.208 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T21:58:04.237 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:04.295 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:04.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:04.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:58:04.353 INFO:tasks.rados.rados.0.smithi125.stdout:1075: done (0 left) 2021-03-24T21:58:04.353 INFO:tasks.rados.rados.0.smithi125.stdout:1076: rollback oid 17 current snap is 104 2021-03-24T21:58:04.353 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 17 to 103 2021-03-24T21:58:04.353 INFO:tasks.rados.rados.0.smithi125.stdout:1077: read oid 42 snap -1 2021-03-24T21:58:04.354 INFO:tasks.rados.rados.0.smithi125.stdout:1077: expect deleted 2021-03-24T21:58:04.354 INFO:tasks.rados.rados.0.smithi125.stdout:1078: write oid 29 current snap is 104 2021-03-24T21:58:04.354 INFO:tasks.rados.rados.0.smithi125.stdout:1078: seq_num 401 ranges {739607=625205,1982691=499457} 2021-03-24T21:58:04.361 INFO:tasks.rados.rados.0.smithi125.stdout:1078: writing smithi12556470-29 from 739607 to 1364812 tid 1 2021-03-24T21:58:04.368 INFO:tasks.rados.rados.0.smithi125.stdout:1078: writing smithi12556470-29 from 1982691 to 2482148 tid 2 2021-03-24T21:58:04.368 INFO:tasks.rados.rados.0.smithi125.stdout:1077: done (2 left) 2021-03-24T21:58:04.368 INFO:tasks.rados.rados.0.smithi125.stdout:1079: write (excl) oid 22 current snap is 104 2021-03-24T21:58:04.368 INFO:tasks.rados.rados.0.smithi125.stdout:1079: seq_num 402 ranges {756414=700454,2041173=715241} 2021-03-24T21:58:04.376 INFO:tasks.rados.rados.0.smithi125.stdout:1079: writing smithi12556470-22 from 756414 to 1456868 tid 1 2021-03-24T21:58:04.384 INFO:tasks.rados.rados.0.smithi125.stdout:1079: writing smithi12556470-22 from 2041173 to 2756414 tid 2 2021-03-24T21:58:04.384 INFO:tasks.rados.rados.0.smithi125.stdout:1080: read oid 43 snap -1 2021-03-24T21:58:04.384 INFO:tasks.rados.rados.0.smithi125.stdout:1080: expect (ObjNum 399 snap 102 seq_num 399) 2021-03-24T21:58:04.384 INFO:tasks.rados.rados.0.smithi125.stdout:1081: write oid 18 current snap is 104 2021-03-24T21:58:04.384 INFO:tasks.rados.rados.0.smithi125.stdout:1076: finishing rollback tid 0 to smithi12556470-17 2021-03-24T21:58:04.385 INFO:tasks.rados.rados.0.smithi125.stdout:1081: seq_num 403 ranges {773221=492056,1816008=503126} 2021-03-24T21:58:04.390 INFO:tasks.rados.rados.0.smithi125.stdout:1081: writing smithi12556470-18 from 773221 to 1265277 tid 1 2021-03-24T21:58:04.396 INFO:tasks.rados.rados.0.smithi125.stdout:1081: writing smithi12556470-18 from 1816008 to 2319134 tid 2 2021-03-24T21:58:04.397 INFO:tasks.rados.rados.0.smithi125.stdout:1082: snap_remove snap 102 2021-03-24T21:58:04.400 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:04.455 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:04.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:04.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:04.485 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:58:04.561 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:04.574 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:04 smithi142 conmon[20125]: cluster 2021-03-24T21:58 2021-03-24T21:58:04.574 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:04 smithi142 conmon[20125]: :02.787113+0000 mgr.y (mgr.14144) 424 : cluster [DBG] pgmap v631: 168 pgs: 168 active+clean; 116 MiB data, 475 MiB used, 536 GiB / 536 GiB avail; 10 MiB/s rd, 3.9 MiB/s wr, 26 op/s 2021-03-24T21:58:04.574 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:04 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:58:04.574 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:04 smithi142 conmon[20125]: T21:58:03.332847+0000 mon.a (mon.0) 988 : cluster 2021-03-24T21:58:04.575 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:04 smithi142 conmon[20125]: [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2021-03-24T21:58:04.575 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:04 smithi142 conmon[20125]: cluster 2021-03-24T21:58:03.332934+0000 mon.a 2021-03-24T21:58:04.575 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:04 smithi142 conmon[20125]: (mon.0) 989 : cluster [INF] Cluster is now healthy 2021-03-24T21:58:04.575 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:04 smithi142 conmon[20125]: cluster 2021-03-24T21:58: 2021-03-24T21:58:04.575 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:04 smithi142 conmon[20125]: 03.346935+0000 mon.a (mon.0) 990 : cluster [DBG] osdmap e313: 8 total, 8 up, 6 in 2021-03-24T21:58:04.576 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:04 smithi142 conmon[20125]: 2021-03-24T21:58:04.617 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:04.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:04.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:58:04.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:04 smithi125 conmon[19936]: cluster 2021-03-24T21:58:02.787113+0000 2021-03-24T21:58:04.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:04 smithi125 conmon[19936]: mgr.y (mgr.14144) 424 : cluster [DBG] pgmap v631: 168 pgs: 168 active+clean; 116 MiB data, 475 MiB used, 536 GiB / 536 GiB avail; 10 MiB/s rd, 3.9 MiB/s wr, 26 op/s 2021-03-24T21:58:04.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:04 smithi125 conmon[19936]: cluster 2021-03-24T21:58:04.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:04 smithi125 conmon[19936]: 2021-03-24T21:58:03.332847+0000 mon.a (mon. 2021-03-24T21:58:04.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:04 smithi125 conmon[19936]: 0) 988 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2021-03-24T21:58:04.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:04 smithi125 conmon[19936]: cluster 2021-03-24T21:58:03.332934+0000 mon.a (mon.0) 989 : cluster [INF] Cluster is now healthy 2021-03-24T21:58:04.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:04 smithi125 conmon[19936]: cluster 2021-03-24T21:58:03.346935+0000 mon.a (mon.0) 990 : cluster [DBG] osdmap e313: 8 total, 8 up, 6 in 2021-03-24T21:58:04.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:04 smithi125 conmon[24927]: cluster 2021-03-24T21:58:02.787113+0000 mgr.y (mgr.14144 2021-03-24T21:58:04.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:04 smithi125 conmon[24927]: ) 424 : cluster [DBG] pgmap v631: 168 pgs: 168 active+clean; 116 MiB data, 475 MiB used, 536 GiB / 536 GiB avail; 10 MiB/s rd, 3.9 MiB/s wr, 26 op/s 2021-03-24T21:58:04.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:04 smithi125 conmon[24927]: cluster 2021-03-24T21:58:03.332847+0000 mon.a (mon.0) 988 2021-03-24T21:58:04.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:04 smithi125 conmon[24927]: : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2021-03-24T21:58:04.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:04 smithi125 conmon[24927]: cluster 2021-03-24T21:58:03.332934+0000 mon.a ( 2021-03-24T21:58:04.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:04 smithi125 conmon[24927]: mon.0) 989 : cluster [INF] Cluster is now healthy 2021-03-24T21:58:04.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:04 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:58:04.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:04 smithi125 conmon[24927]: -24T21:58:03.346935+0000 mon.a (mon.0) 990 : 2021-03-24T21:58:04.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:04 smithi125 conmon[24927]: cluster [DBG] osdmap e313: 8 total, 8 up, 6 in 2021-03-24T21:58:04.721 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:04.780 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:04.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:04.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:04.885 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:04.938 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:04.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:04.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:58:05.042 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:05.097 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:05.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:05.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:58:05.178 INFO:tasks.rados.rados.0.smithi125.stdout:1082: done (5 left) 2021-03-24T21:58:05.178 INFO:tasks.rados.rados.0.smithi125.stdout:1083: copy_from oid 27 from oid 24 current snap is 104 2021-03-24T21:58:05.179 INFO:tasks.rados.rados.0.smithi125.stdout:1076: finishing rollback tid 1 to smithi12556470-17 2021-03-24T21:58:05.179 INFO:tasks.rados.rados.0.smithi125.stdout:1076: finishing rollback tid 2 to smithi12556470-17 2021-03-24T21:58:05.179 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 17 v 314 (ObjNum 385 snap 99 seq_num 385) dirty exists 2021-03-24T21:58:05.179 INFO:tasks.rados.rados.0.smithi125.stdout:1078: finishing write tid 1 to smithi12556470-29 2021-03-24T21:58:05.179 INFO:tasks.rados.rados.0.smithi125.stdout:1078: finishing write tid 2 to smithi12556470-29 2021-03-24T21:58:05.180 INFO:tasks.rados.rados.0.smithi125.stdout:1078: finishing write tid 3 to smithi12556470-29 2021-03-24T21:58:05.180 INFO:tasks.rados.rados.0.smithi125.stdout:1078: finishing write tid 4 to smithi12556470-29 2021-03-24T21:58:05.180 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 29 v 163 (ObjNum 401 snap 104 seq_num 401) dirty exists 2021-03-24T21:58:05.180 INFO:tasks.rados.rados.0.smithi125.stdout:1078: left oid 29 (ObjNum 401 snap 104 seq_num 401) 2021-03-24T21:58:05.180 INFO:tasks.rados.rados.0.smithi125.stdout:1079: finishing write tid 1 to smithi12556470-22 2021-03-24T21:58:05.181 INFO:tasks.rados.rados.0.smithi125.stdout:1080: expect (ObjNum 399 snap 102 seq_num 399) 2021-03-24T21:58:05.192 INFO:teuthology.orchestra.run.smithi125.stderr:noscrub is set 2021-03-24T21:58:05.204 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:05.254 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:05.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:05.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:58:05.313 INFO:tasks.rados.rados.0.smithi125.stdout:1079: finishing write tid 2 to smithi12556470-22 2021-03-24T21:58:05.314 INFO:tasks.rados.rados.0.smithi125.stdout:1079: finishing write tid 3 to smithi12556470-22 2021-03-24T21:58:05.314 INFO:tasks.rados.rados.0.smithi125.stdout:1079: finishing write tid 4 to smithi12556470-22 2021-03-24T21:58:05.314 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 22 v 294 (ObjNum 402 snap 104 seq_num 402) dirty exists 2021-03-24T21:58:05.314 INFO:tasks.rados.rados.0.smithi125.stdout:1079: left oid 22 (ObjNum 402 snap 104 seq_num 402) 2021-03-24T21:58:05.315 INFO:tasks.rados.rados.0.smithi125.stdout:1081: finishing write tid 1 to smithi12556470-18 2021-03-24T21:58:05.315 INFO:tasks.rados.rados.0.smithi125.stdout:1081: finishing write tid 2 to smithi12556470-18 2021-03-24T21:58:05.315 INFO:tasks.rados.rados.0.smithi125.stdout:1076: done (5 left) 2021-03-24T21:58:05.315 INFO:tasks.rados.rados.0.smithi125.stdout:1078: done (4 left) 2021-03-24T21:58:05.315 INFO:tasks.rados.rados.0.smithi125.stdout:1079: done (3 left) 2021-03-24T21:58:05.316 INFO:tasks.rados.rados.0.smithi125.stdout:1080: done (2 left) 2021-03-24T21:58:05.316 INFO:tasks.rados.rados.0.smithi125.stdout:1084: snap_remove snap 103 2021-03-24T21:58:05.358 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:05.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:05 smithi142 conmon[20125]: cluster 2021-03-24T21:58:04.012275+0000 2021-03-24T21:58:05.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:05 smithi142 conmon[20125]: osd.7 (osd.7) 73 : cluster [DBG] 6.15 scrub ok 2021-03-24T21:58:05.373 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:05 smithi142 conmon[20125]: cluster 2021-03-24T21:58:04.351766+0000 mon.a (mon.0 2021-03-24T21:58:05.373 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:05 smithi142 conmon[20125]: ) 991 : cluster [DBG] osdmap e314: 8 total, 8 up, 6 in 2021-03-24T21:58:05.373 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:05 smithi142 conmon[20125]: audit 2021-03-24T21:58: 2021-03-24T21:58:05.373 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:05 smithi142 conmon[20125]: 04.721331+0000 mon.c (mon.2) 49 : audit [INF] from='client.? v1:172.21.15.125:0/2277738562' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T21:58:05.373 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:05 smithi142 conmon[20125]: audit 2021-03-24T21:58:04.721829+0000 2021-03-24T21:58:05.374 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:05 smithi142 conmon[20125]: mon.a (mon.0) 992 : audit [INF] from='client.35045 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T21:58:05.374 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:05 smithi142 conmon[20125]: audit 2021-03-24T21:58:04. 2021-03-24T21:58:05.374 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:05 smithi142 conmon[20125]: 779170+0000 mon.a (mon.0) 993 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T21:58:05.374 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:05 smithi142 conmon[20125]: audit 2021-03-24T21:58 2021-03-24T21:58:05.374 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:05 smithi142 conmon[20125]: :04.779631+0000 mon.a (mon.0) 994 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T21:58:05.375 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:05 smithi142 conmon[20125]: 2021-03-24T21:58:05.375 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:05 smithi142 conmon[20125]: cluster 2021-03-24T21 2021-03-24T21:58:05.375 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:05 smithi142 conmon[20125]: :58:05.160666+0000 mon.a (mon.0) 995 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:58:05.375 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:05 smithi142 conmon[20125]: audit 2021-03-24T21:58: 2021-03-24T21:58:05.375 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:05 smithi142 conmon[20125]: 05.172940+0000 mon.a (mon.0) 996 : audit [INF] from='client.35045 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T21:58:05.376 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:05 smithi142 conmon[20125]: 2021-03-24T21:58:05.376 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:05 smithi142 conmon[20125]: cluster 2021-03-24T21 2021-03-24T21:58:05.376 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:05 smithi142 conmon[20125]: :58:05.173210+0000 mon.a (mon.0) 997 : cluster [DBG] osdmap e315: 8 total, 8 up, 6 in 2021-03-24T21:58:05.399 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:05.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:05.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:05.504 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:05.556 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:05.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:05.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:58:05.660 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:05.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[24927]: cluster 2021-03-24T21:58: 2021-03-24T21:58:05.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[24927]: 04.012275+0000 osd.7 (osd.7) 73 : cluster [DBG] 6.15 scrub ok 2021-03-24T21:58:05.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[24927]: cluster 2021-03-24T21:58:04. 2021-03-24T21:58:05.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[24927]: 351766+0000 mon.a (mon.0) 991 : cluster 2021-03-24T21:58:05.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[24927]: [DBG] osdmap e314: 8 total, 8 up, 6 in 2021-03-24T21:58:05.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T21:58:05.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[24927]: 24T21:58:04.721331+0000 2021-03-24T21:58:05.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[24927]: mon.c (mon.2) 49 : audit 2021-03-24T21:58:05.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[24927]: [INF] from='client.? v1:172.21.15.125:0/2277738562' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T21:58:05.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[24927]: audit 2021- 2021-03-24T21:58:05.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[24927]: 03-24T21:58:04.721829+0000 2021-03-24T21:58:05.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[24927]: mon.a (mon.0) 992 : audit [INF] 2021-03-24T21:58:05.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[24927]: from='client.35045 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T21:58:05.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[24927]: audit 2021-03-24T21:58:04.779170+0000 2021-03-24T21:58:05.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[24927]: mon.a (mon.0) 993 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T21:58:05.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[24927]: audit 2021-03-24T21:58 2021-03-24T21:58:05.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[24927]: :04.779631+0000 mon.a (mon.0 2021-03-24T21:58:05.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[24927]: ) 994 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T21:58:05.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:58:05.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[24927]: 03-24T21:58:05.160666+0000 2021-03-24T21:58:05.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[24927]: mon.a (mon.0) 995 : cluster [WRN] 2021-03-24T21:58:05.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[24927]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:58:05.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[24927]: audit 2021 2021-03-24T21:58:05.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[24927]: -03-24T21:58:05.172940+0000 mon.a (mon 2021-03-24T21:58:05.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[24927]: .0) 996 : audit [INF] from='client.35045 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T21:58:05.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:58:05.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[24927]: 21:58:05.173210+0000 mon.a (mon. 2021-03-24T21:58:05.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[24927]: 0) 997 : cluster [DBG] osdmap e315: 8 total, 8 up, 6 in 2021-03-24T21:58:05.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[19936]: cluster 2021-03-24T21:58:04 2021-03-24T21:58:05.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[19936]: .012275+0000 osd.7 (osd.7) 73 : cluster [DBG] 6.15 scrub ok 2021-03-24T21:58:05.669 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[19936]: cluster 2021-03-24T21:58:04.351766 2021-03-24T21:58:05.669 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[19936]: +0000 mon.a (mon.0) 991 : cluster [DBG] osdmap e314: 8 total, 8 up, 6 in 2021-03-24T21:58:05.669 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[19936]: audit 2021 2021-03-24T21:58:05.669 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[19936]: -03-24T21:58:04.721331+0000 mon.c (mon.2) 2021-03-24T21:58:05.669 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[19936]: 49 : audit [INF] from='client.? v1:172.21.15.125:0/2277738562' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T21:58:05.670 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[19936]: audit 2021-03-24 2021-03-24T21:58:05.670 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[19936]: T21:58:04.721829+0000 mon.a (mon. 2021-03-24T21:58:05.670 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[19936]: 0) 992 : audit [INF] from='client.35045 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T21:58:05.670 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:58:05.670 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[19936]: 03-24T21:58:04.779170+0000 2021-03-24T21:58:05.671 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[19936]: mon.a (mon.0) 993 : audit [INF] 2021-03-24T21:58:05.671 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T21:58:05.671 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[19936]: audit 2021-03-24T21:58:05.671 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[19936]: 2021-03-24T21:58: 2021-03-24T21:58:05.672 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[19936]: 04.779631+0000 mon.a (mon.0 2021-03-24T21:58:05.672 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[19936]: ) 994 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T21:58:05.672 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:58:05.672 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[19936]: 03-24T21:58:05 2021-03-24T21:58:05.672 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[19936]: .160666+0000 mon.a (mon.0) 995 : 2021-03-24T21:58:05.673 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[19936]: cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:58:05.673 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[19936]: audit 2021-03- 2021-03-24T21:58:05.673 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[19936]: 24T21:58:05.172940+0000 2021-03-24T21:58:05.673 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[19936]: mon.a (mon.0) 996 : audit [INF] 2021-03-24T21:58:05.673 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[19936]: from='client.35045 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T21:58:05.674 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:58:05.674 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[19936]: 21:58:05.173210+0000 mon.a (mon 2021-03-24T21:58:05.674 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:05 smithi125 conmon[19936]: .0) 997 : cluster [DBG] osdmap e315: 8 total, 8 up, 6 in 2021-03-24T21:58:05.732 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:05.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:05.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:58:05.839 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:05.895 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:05.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:05.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:06.002 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:06.056 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:06.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:06.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:58:06.165 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:06.186 INFO:tasks.rados.rados.0.smithi125.stdout:1084: done (2 left) 2021-03-24T21:58:06.187 INFO:tasks.rados.rados.0.smithi125.stdout:1085: rollback oid 8 current snap is 104 2021-03-24T21:58:06.187 INFO:tasks.rados.rados.0.smithi125.stdout:1085: done (2 left) 2021-03-24T21:58:06.188 INFO:tasks.rados.rados.0.smithi125.stdout:1086: copy_from oid 37 from oid 36 current snap is 104 2021-03-24T21:58:06.188 INFO:tasks.rados.rados.0.smithi125.stdout:1081: finishing write tid 3 to smithi12556470-18 2021-03-24T21:58:06.188 INFO:tasks.rados.rados.0.smithi125.stdout:1081: finishing write tid 4 to smithi12556470-18 2021-03-24T21:58:06.188 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 18 v 178 (ObjNum 403 snap 104 seq_num 403) dirty exists 2021-03-24T21:58:06.189 INFO:tasks.rados.rados.0.smithi125.stdout:1081: left oid 18 (ObjNum 403 snap 104 seq_num 403) 2021-03-24T21:58:06.189 INFO:tasks.rados.rados.0.smithi125.stdout:1083: finishing copy_from to smithi12556470-27 2021-03-24T21:58:06.189 INFO:tasks.rados.rados.0.smithi125.stdout:1083: got expected ENOENT (src dne) 2021-03-24T21:58:06.189 INFO:tasks.rados.rados.0.smithi125.stdout:1083: finishing copy_from racing read to smithi12556470-27 2021-03-24T21:58:06.189 INFO:tasks.rados.rados.0.smithi125.stdout:1081: done (2 left) 2021-03-24T21:58:06.190 INFO:tasks.rados.rados.0.smithi125.stdout:1083: done (1 left) 2021-03-24T21:58:06.190 INFO:tasks.rados.rados.0.smithi125.stdout:1087: write (excl) oid 7 current snap is 104 2021-03-24T21:58:06.190 INFO:tasks.rados.rados.0.smithi125.stdout:1087: seq_num 404 ranges {790028=567305,1874490=763137} 2021-03-24T21:58:06.194 INFO:tasks.rados.rados.0.smithi125.stdout:1087: writing smithi12556470-7 from 790028 to 1357333 tid 1 2021-03-24T21:58:06.204 INFO:tasks.rados.rados.0.smithi125.stdout:1087: writing smithi12556470-7 from 1874490 to 2637627 tid 2 2021-03-24T21:58:06.204 INFO:tasks.rados.rados.0.smithi125.stdout:1088: write (excl) oid 4 current snap is 104 2021-03-24T21:58:06.204 INFO:tasks.rados.rados.0.smithi125.stdout:1088: seq_num 405 ranges {406835=642554,1532972=506795,2772312=34523} 2021-03-24T21:58:06.209 INFO:tasks.rados.rados.0.smithi125.stdout:1088: writing smithi12556470-4 from 406835 to 1049389 tid 1 2021-03-24T21:58:06.215 INFO:tasks.rados.rados.0.smithi125.stdout:1088: writing smithi12556470-4 from 1532972 to 2039767 tid 2 2021-03-24T21:58:06.220 INFO:tasks.rados.rados.0.smithi125.stdout:1088: writing smithi12556470-4 from 2772312 to 2806835 tid 3 2021-03-24T21:58:06.220 INFO:tasks.rados.rados.0.smithi125.stdout:1089: snap_create 2021-03-24T21:58:06.221 INFO:tasks.rados.rados.0.smithi125.stdout:1087: finishing write tid 1 to smithi12556470-7 2021-03-24T21:58:06.221 INFO:tasks.rados.rados.0.smithi125.stdout:1086: finishing copy_from to smithi12556470-37 2021-03-24T21:58:06.221 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 37 v 198 (ObjNum 150 snap 30 seq_num 150) dirty exists 2021-03-24T21:58:06.221 INFO:tasks.rados.rados.0.smithi125.stdout:1086: finishing copy_from racing read to smithi12556470-37 2021-03-24T21:58:06.222 INFO:tasks.rados.rados.0.smithi125.stdout:1087: finishing write tid 2 to smithi12556470-7 2021-03-24T21:58:06.222 INFO:tasks.rados.rados.0.smithi125.stdout:1087: finishing write tid 3 to smithi12556470-7 2021-03-24T21:58:06.222 INFO:tasks.rados.rados.0.smithi125.stdout:1087: finishing write tid 4 to smithi12556470-7 2021-03-24T21:58:06.222 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 7 v 324 (ObjNum 404 snap 104 seq_num 404) dirty exists 2021-03-24T21:58:06.222 INFO:tasks.rados.rados.0.smithi125.stdout:1087: left oid 7 (ObjNum 404 snap 104 seq_num 404) 2021-03-24T21:58:06.223 INFO:tasks.rados.rados.0.smithi125.stdout:1088: finishing write tid 1 to smithi12556470-4 2021-03-24T21:58:06.223 INFO:tasks.rados.rados.0.smithi125.stdout:1088: finishing write tid 2 to smithi12556470-4 2021-03-24T21:58:06.224 INFO:tasks.rados.rados.0.smithi125.stdout:1088: finishing write tid 3 to smithi12556470-4 2021-03-24T21:58:06.224 INFO:tasks.rados.rados.0.smithi125.stdout:1088: finishing write tid 4 to smithi12556470-4 2021-03-24T21:58:06.224 INFO:tasks.rados.rados.0.smithi125.stdout:1088: finishing write tid 5 to smithi12556470-4 2021-03-24T21:58:06.224 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 4 v 279 (ObjNum 405 snap 104 seq_num 405) dirty exists 2021-03-24T21:58:06.225 INFO:tasks.rados.rados.0.smithi125.stdout:1088: left oid 4 (ObjNum 405 snap 104 seq_num 405) 2021-03-24T21:58:06.225 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:06.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:06.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:06.329 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:06.387 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:06.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:06.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:06.493 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:06.551 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:06.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:06.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:06.658 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:58:06.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:06 smithi125 conmon[19936]: cluster 2021-03-24T21:58:04. 2021-03-24T21:58:06.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:06 smithi125 conmon[19936]: 787723+0000 mgr.y (mgr.14144) 425 : cluster [DBG] pgmap v634: 168 pgs: 168 active+clean; 113 MiB data, 451 MiB used, 536 GiB / 536 GiB avail; 7.3 MiB/s rd, 2.4 MiB/s wr, 16 op/s 2021-03-24T21:58:06.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:06 smithi125 conmon[19936]: cluster 2021-03-24T21:58: 2021-03-24T21:58:06.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:06 smithi125 conmon[19936]: 04.965360+0000 osd.7 (osd.7) 74 : cluster [DBG] 3.15 deep-scrub ok 2021-03-24T21:58:06.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:06 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:58:06.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:06 smithi125 conmon[19936]: -24T21:58:06.184566+0000 mon.a (mon. 2021-03-24T21:58:06.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:06 smithi125 conmon[19936]: 0) 998 : cluster [DBG] osdmap e316: 8 total, 8 up, 6 in 2021-03-24T21:58:06.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:06 smithi125 conmon[24927]: cluster 2021-03-24T21:58 2021-03-24T21:58:06.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:06 smithi125 conmon[24927]: :04.787723+0000 mgr.y (mgr.14144) 425 : cluster [DBG] pgmap v634: 168 pgs: 168 active+clean; 113 MiB data, 451 MiB used, 536 GiB / 536 GiB avail; 7.3 MiB/s rd, 2.4 MiB/s wr, 16 op/s 2021-03-24T21:58:06.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:06 smithi125 conmon[24927]: cluster 2021-03-24T21:58:06.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:06 smithi125 conmon[24927]: 2021-03-24T21:58 2021-03-24T21:58:06.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:06 smithi125 conmon[24927]: :04.965360+0000 osd.7 (osd.7) 2021-03-24T21:58:06.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:06 smithi125 conmon[24927]: 74 : cluster [DBG] 3.15 deep-scrub ok 2021-03-24T21:58:06.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:06 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:58:06.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:06 smithi125 conmon[24927]: -03-24T21:58: 2021-03-24T21:58:06.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:06 smithi125 conmon[24927]: 06.184566+0000 mon.a (mon.0) 998 2021-03-24T21:58:06.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:06 smithi125 conmon[24927]: : cluster [DBG] osdmap e316: 8 total, 8 up, 6 in 2021-03-24T21:58:06.715 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:58:06.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:06.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:58:06.788 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:06 smithi142 conmon[20125]: cluster 2021-03-24T21:58:04.787723+0000 mgr.y (mgr. 2021-03-24T21:58:06.788 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:06 smithi142 conmon[20125]: 14144) 425 : cluster [DBG] pgmap v634: 168 pgs: 168 active+clean; 113 MiB data, 451 MiB used, 536 GiB / 536 GiB avail; 7.3 MiB/s rd, 2.4 MiB/s wr, 16 op/s 2021-03-24T21:58:06.789 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:06 smithi142 conmon[20125]: cluster 2021-03-24T21:58:06.789 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:06 smithi142 conmon[20125]: 2021-03-24T21:58:04 2021-03-24T21:58:06.789 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:06 smithi142 conmon[20125]: .965360+0000 osd.7 (osd. 2021-03-24T21:58:06.789 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:06 smithi142 conmon[20125]: 7) 74 : cluster [DBG] 2021-03-24T21:58:06.790 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:06 smithi142 conmon[20125]: 3.15 deep-scrub ok 2021-03-24T21:58:06.790 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:06 smithi142 conmon[20125]: cluster 2021-03-24T21:58:06.790 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:06 smithi142 conmon[20125]: 2021-03-24T21 2021-03-24T21:58:06.790 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:06 smithi142 conmon[20125]: :58:06.184566+0000 2021-03-24T21:58:06.791 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:06 smithi142 conmon[20125]: mon.a (mon.0) 2021-03-24T21:58:06.791 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:06 smithi142 conmon[20125]: 998 : cluster [DBG] osdmap e316: 8 total, 8 up, 6 in 2021-03-24T21:58:06.820 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:06.873 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:06.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:06.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:58:06.884 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2021-03-24T21:58:06.884 INFO:teuthology.orchestra.run.smithi142.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2021-03-24T21:58:06.884 INFO:teuthology.orchestra.run.smithi142.stdout:} 2021-03-24T21:58:06.977 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:07.030 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:07.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:07.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:07.135 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:07.192 INFO:tasks.rados.rados.0.smithi125.stdout:1086: done (3 left) 2021-03-24T21:58:07.193 INFO:tasks.rados.rados.0.smithi125.stdout:1087: done (2 left) 2021-03-24T21:58:07.193 INFO:tasks.rados.rados.0.smithi125.stdout:1088: done (1 left) 2021-03-24T21:58:07.193 INFO:tasks.rados.rados.0.smithi125.stdout:1089: done (0 left) 2021-03-24T21:58:07.193 INFO:tasks.rados.rados.0.smithi125.stdout:1090: rollback oid 44 current snap is 105 2021-03-24T21:58:07.194 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 44 to 104 2021-03-24T21:58:07.194 INFO:tasks.rados.rados.0.smithi125.stdout:1091: read oid 19 snap 104 2021-03-24T21:58:07.194 INFO:tasks.rados.rados.0.smithi125.stdout:1091: expect (ObjNum 280 snap 70 seq_num 280) 2021-03-24T21:58:07.194 INFO:tasks.rados.rados.0.smithi125.stdout:1092: read oid 2 snap -1 2021-03-24T21:58:07.195 INFO:tasks.rados.rados.0.smithi125.stdout:1092: expect (ObjNum 353 snap 89 seq_num 353) 2021-03-24T21:58:07.195 INFO:tasks.rados.rados.0.smithi125.stdout:1093: rollback oid 14 current snap is 105 2021-03-24T21:58:07.195 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 14 to 104 2021-03-24T21:58:07.195 INFO:tasks.rados.rados.0.smithi125.stdout:1094: snap_create 2021-03-24T21:58:07.199 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:07.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:07.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:07.205 INFO:tasks.rados.rados.0.smithi125.stdout:1090: finishing rollback tid 0 to smithi12556470-44 2021-03-24T21:58:07.206 INFO:tasks.rados.rados.0.smithi125.stdout:1091: expect (ObjNum 280 snap 70 seq_num 280) 2021-03-24T21:58:07.302 INFO:tasks.rados.rados.0.smithi125.stdout:1090: finishing rollback tid 1 to smithi12556470-44 2021-03-24T21:58:07.302 INFO:tasks.rados.rados.0.smithi125.stdout:1090: finishing rollback tid 2 to smithi12556470-44 2021-03-24T21:58:07.303 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 44 v 293 (ObjNum 358 snap 91 seq_num 358) dirty exists 2021-03-24T21:58:07.303 INFO:tasks.rados.rados.0.smithi125.stdout:1092: expect (ObjNum 353 snap 89 seq_num 353) 2021-03-24T21:58:07.303 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:07.356 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:07.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:07.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:58:07.441 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T21:58:07.459 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T21:58:07.463 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:58:07.477 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T21:58:07.490 INFO:tasks.rados.rados.0.smithi125.stdout:1093: finishing rollback tid 0 to smithi12556470-14 2021-03-24T21:58:07.490 INFO:tasks.rados.rados.0.smithi125.stdout:1093: finishing rollback tid 1 to smithi12556470-14 2021-03-24T21:58:07.491 INFO:tasks.rados.rados.0.smithi125.stdout:1093: finishing rollback tid 2 to smithi12556470-14 2021-03-24T21:58:07.491 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 14 v 326 (ObjNum 394 snap 102 seq_num 394) dirty exists 2021-03-24T21:58:07.506 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T21:58:07.518 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:58:07.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:07.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:58:07.529 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T21:58:07.551 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T21:58:07.574 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T21:58:07.599 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T21:58:07.601 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-24T21:58:07.601 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-24T21:58:07.602 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-24T21:58:07.602 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-24T21:58:07.602 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-24T21:58:07.602 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-24T21:58:07.602 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-24T21:58:07.603 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-24T21:58:07.624 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:07.678 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:07.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:07.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:58:07.784 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:07.847 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:07.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:07.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:58:07.953 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:08.010 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:08.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:08.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:08.117 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:08.179 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:08.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:08.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:58:08.193 INFO:tasks.rados.rados.0.smithi125.stdout:1090: done (4 left) 2021-03-24T21:58:08.273 INFO:tasks.rados.rados.0.smithi125.stdout:1091: done (3 left) 2021-03-24T21:58:08.273 INFO:tasks.rados.rados.0.smithi125.stdout:1092: done (2 left) 2021-03-24T21:58:08.273 INFO:tasks.rados.rados.0.smithi125.stdout:1093: done (1 left) 2021-03-24T21:58:08.274 INFO:tasks.rados.rados.0.smithi125.stdout:1094: done (0 left) 2021-03-24T21:58:08.274 INFO:tasks.rados.rados.0.smithi125.stdout:1095: read oid 28 snap -1 2021-03-24T21:58:08.274 INFO:tasks.rados.rados.0.smithi125.stdout:1095: expect (ObjNum 349 snap 89 seq_num 349) 2021-03-24T21:58:08.274 INFO:tasks.rados.rados.0.smithi125.stdout:1096: delete oid 35 current snap is 106 2021-03-24T21:58:08.275 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2021-03-24T21:58:08.275 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2021-03-24T21:58:08.275 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2021-03-24T21:58:08.276 INFO:teuthology.orchestra.run.smithi142.stdout:} 2021-03-24T21:58:08.276 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:08 smithi142 conmon[20125]: cluster 2021-03-24T21:58:06 2021-03-24T21:58:08.276 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:08 smithi142 conmon[20125]: .788465+0000 mgr.y (mgr.14144) 426 : cluster [DBG] pgmap v637: 168 pgs: 168 active+clean; 113 MiB data, 451 MiB used, 536 GiB / 536 GiB avail; 7.3 MiB/s rd, 2.4 MiB/s wr, 16 op/s 2021-03-24T21:58:08.277 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:08 smithi142 conmon[20125]: cluster 2021-03-24T21:58:07.190536+0000 mon.a (mon.0 2021-03-24T21:58:08.277 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:08 smithi142 conmon[20125]: ) 999 : cluster [DBG] osdmap e317: 8 total, 8 up, 6 in 2021-03-24T21:58:08.277 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:08 smithi142 conmon[20125]: audit 2021-03-24T21:58:07.415396+0000 2021-03-24T21:58:08.277 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:08 smithi142 conmon[20125]: mon.c (mon.2) 50 : audit [DBG] from='client.? v1:172.21.15.125:0/1772408938' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:58:08.278 INFO:tasks.rados.rados.0.smithi125.stdout:1096: done (1 left) 2021-03-24T21:58:08.279 INFO:tasks.rados.rados.0.smithi125.stdout:1097: rollback oid 48 current snap is 106 2021-03-24T21:58:08.279 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 48 to 104 2021-03-24T21:58:08.279 INFO:tasks.rados.rados.0.smithi125.stdout:1098: delete oid 15 current snap is 106 2021-03-24T21:58:08.279 INFO:tasks.rados.rados.0.smithi125.stdout:1098: done (2 left) 2021-03-24T21:58:08.280 INFO:tasks.rados.rados.0.smithi125.stdout:1099: write oid 38 current snap is 106 2021-03-24T21:58:08.280 INFO:tasks.rados.rados.0.smithi125.stdout:1099: seq_num 408 ranges {457256=468301,1592065=770475,2854581=2675} 2021-03-24T21:58:08.280 INFO:tasks.rados.rados.0.smithi125.stdout:1099: writing smithi12556470-38 from 457256 to 925557 tid 1 2021-03-24T21:58:08.280 INFO:tasks.rados.rados.0.smithi125.stdout:1099: writing smithi12556470-38 from 1592065 to 2362540 tid 2 2021-03-24T21:58:08.280 INFO:tasks.rados.rados.0.smithi125.stdout:1099: writing smithi12556470-38 from 2854581 to 2857256 tid 3 2021-03-24T21:58:08.281 INFO:tasks.rados.rados.0.smithi125.stdout:1100: read oid 26 snap 104 2021-03-24T21:58:08.281 INFO:tasks.rados.rados.0.smithi125.stdout:1100: expect (ObjNum 353 snap 89 seq_num 353) 2021-03-24T21:58:08.281 INFO:tasks.rados.rados.0.smithi125.stdout:1101: copy_from oid 23 from oid 27 current snap is 106 2021-03-24T21:58:08.281 INFO:tasks.rados.rados.0.smithi125.stdout:1095: expect (ObjNum 349 snap 89 seq_num 349) 2021-03-24T21:58:08.285 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:08.338 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:08.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:08.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:58:08.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:08 smithi125 conmon[19936]: cluster 2021-03-24T21:58:06.788465 2021-03-24T21:58:08.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:08 smithi125 conmon[19936]: +0000 mgr.y (mgr.14144) 426 : cluster [DBG] pgmap v637: 168 pgs: 168 active+clean; 113 MiB data, 451 MiB used, 536 GiB / 536 GiB avail; 7.3 MiB/s rd, 2.4 MiB/s wr, 16 op/s 2021-03-24T21:58:08.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:08 smithi125 conmon[19936]: cluster 2021-03-24T21:58:07.190536+0000 mon.a (mon.0) 999 : cluster [DBG] 2021-03-24T21:58:08.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:08 smithi125 conmon[19936]: osdmap e317: 8 total, 8 up, 6 in 2021-03-24T21:58:08.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:08 smithi125 conmon[19936]: audit 2021-03 2021-03-24T21:58:08.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:08 smithi125 conmon[19936]: -24T21:58:07.415396+0000 mon.c (mon.2) 50 2021-03-24T21:58:08.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:08 smithi125 conmon[19936]: : audit [DBG] from='client.? v1:172.21.15.125:0/1772408938' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:58:08.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:08 smithi125 conmon[24927]: cluster 2021-03-24T21:58:06 2021-03-24T21:58:08.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:08 smithi125 conmon[24927]: .788465+0000 mgr.y (mgr.14144) 426 : cluster [DBG] pgmap v637: 168 pgs: 168 active+clean; 113 MiB data, 451 MiB used, 536 GiB / 536 GiB avail; 7.3 MiB/s rd, 2.4 MiB/s wr, 16 op/s 2021-03-24T21:58:08.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:08 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:58:08.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:08 smithi125 conmon[24927]: -24T21:58:07.190536+0000 mon.a (mon.0) 999 : cluster [DBG] osdmap e317: 8 total, 8 up, 6 in 2021-03-24T21:58:08.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:08 smithi125 conmon[24927]: audit 2021-03-24T21 2021-03-24T21:58:08.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:08 smithi125 conmon[24927]: :58:07.415396+0000 mon.c (mon.2) 50 : 2021-03-24T21:58:08.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:08 smithi125 conmon[24927]: audit [DBG] from='client.? v1:172.21.15.125:0/1772408938' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:58:08.445 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:08.451 INFO:tasks.rados.rados.0.smithi125.stdout:1097: finishing rollback tid 0 to smithi12556470-48 2021-03-24T21:58:08.452 INFO:tasks.rados.rados.0.smithi125.stdout:1097: finishing rollback tid 1 to smithi12556470-48 2021-03-24T21:58:08.452 INFO:tasks.rados.rados.0.smithi125.stdout:1097: finishing rollback tid 2 to smithi12556470-48 2021-03-24T21:58:08.452 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 48 v 336 (ObjNum 193 snap 42 seq_num 193) dirty exists 2021-03-24T21:58:08.452 INFO:tasks.rados.rados.0.smithi125.stdout:1099: finishing write tid 1 to smithi12556470-38 2021-03-24T21:58:08.452 INFO:tasks.rados.rados.0.smithi125.stdout:1099: finishing write tid 2 to smithi12556470-38 2021-03-24T21:58:08.453 INFO:tasks.rados.rados.0.smithi125.stdout:1099: finishing write tid 3 to smithi12556470-38 2021-03-24T21:58:08.453 INFO:tasks.rados.rados.0.smithi125.stdout:1099: finishing write tid 4 to smithi12556470-38 2021-03-24T21:58:08.453 INFO:tasks.rados.rados.0.smithi125.stdout:1099: finishing write tid 5 to smithi12556470-38 2021-03-24T21:58:08.453 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 38 v 49 (ObjNum 408 snap 106 seq_num 408) dirty exists 2021-03-24T21:58:08.454 INFO:tasks.rados.rados.0.smithi125.stdout:1099: left oid 38 (ObjNum 408 snap 106 seq_num 408) 2021-03-24T21:58:08.454 INFO:tasks.rados.rados.0.smithi125.stdout:1100: expect (ObjNum 353 snap 89 seq_num 353) 2021-03-24T21:58:08.497 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:08.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:08.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:08.604 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:08.624 INFO:tasks.rados.rados.0.smithi125.stdout:1101: finishing copy_from to smithi12556470-23 2021-03-24T21:58:08.624 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 23 v 275 (ObjNum 400 snap 102 seq_num 400) dirty exists 2021-03-24T21:58:08.624 INFO:tasks.rados.rados.0.smithi125.stdout:1101: finishing copy_from racing read to smithi12556470-23 2021-03-24T21:58:08.624 INFO:tasks.rados.rados.0.smithi125.stdout:1095: done (4 left) 2021-03-24T21:58:08.625 INFO:tasks.rados.rados.0.smithi125.stdout:1097: done (3 left) 2021-03-24T21:58:08.625 INFO:tasks.rados.rados.0.smithi125.stdout:1099: done (2 left) 2021-03-24T21:58:08.625 INFO:tasks.rados.rados.0.smithi125.stdout:1100: done (1 left) 2021-03-24T21:58:08.625 INFO:tasks.rados.rados.0.smithi125.stdout:1101: done (0 left) 2021-03-24T21:58:08.626 INFO:tasks.rados.rados.0.smithi125.stdout:1102: rollback oid 49 current snap is 106 2021-03-24T21:58:08.626 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 49 to 104 2021-03-24T21:58:08.626 INFO:tasks.rados.rados.0.smithi125.stdout:1103: snap_create 2021-03-24T21:58:08.629 INFO:tasks.rados.rados.0.smithi125.stdout:1102: finishing rollback tid 0 to smithi12556470-49 2021-03-24T21:58:08.629 INFO:tasks.rados.rados.0.smithi125.stdout:1102: finishing rollback tid 1 to smithi12556470-49 2021-03-24T21:58:08.630 INFO:tasks.rados.rados.0.smithi125.stdout:1102: finishing rollback tid 2 to smithi12556470-49 2021-03-24T21:58:08.630 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 49 v 340 (ObjNum 391 snap 100 seq_num 391) dirty exists 2021-03-24T21:58:08.656 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:08.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:08.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:58:08.762 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:08.817 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:08.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:08.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:58:08.924 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:08.986 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:08.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:08.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:58:09.046 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-24T21:58:09.091 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:58:09.146 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:58:09.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:09.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:58:09.198 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.6 dump_blocked_ops 2021-03-24T21:58:09.210 INFO:tasks.rados.rados.0.smithi125.stdout:1102: done (1 left) 2021-03-24T21:58:09.210 INFO:tasks.rados.rados.0.smithi125.stdout:1103: done (0 left) 2021-03-24T21:58:09.211 INFO:tasks.rados.rados.0.smithi125.stdout:1104: snap_create 2021-03-24T21:58:09.252 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:09.304 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:09.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:09.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:58:09.338 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:58:09.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:09 smithi142 conmon[20125]: cluster 2021-03-24T21:58:08.190256+0000 mon.a ( 2021-03-24T21:58:09.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:09 smithi142 conmon[20125]: mon.0) 1000 : cluster [DBG] osdmap e318: 8 total, 8 up, 6 in 2021-03-24T21:58:09.409 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:09.458 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:09.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:09.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:58:09.495 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T21:58:09.558 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:09 smithi125 conmon[24927]: cluster 2021-03-24T21:58:08.190256+0000 mon.a (mon.0) 1000 : cluster [DBG] osdmap e318: 8 total, 8 up, 6 in 2021-03-24T21:58:09.559 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:09 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:58:09.559 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:09 smithi125 conmon[19936]: 58:08.190256+0000 mon.a (mon.0) 1000 : cluster [DBG] osdmap e318: 8 total, 8 up, 6 in 2021-03-24T21:58:09.564 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:09.617 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:09.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:09.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:58:09.723 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:09.770 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:09.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:09.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:58:09.877 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:09.934 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:09.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:09.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:58:10.040 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:10.102 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:10.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:10.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:10.180 INFO:tasks.rados.rados.0.smithi125.stdout:1104: done (0 left) 2021-03-24T21:58:10.180 INFO:tasks.rados.rados.0.smithi125.stdout:1105: rollback oid 43 current snap is 108 2021-03-24T21:58:10.181 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 43 to 106 2021-03-24T21:58:10.181 INFO:tasks.rados.rados.0.smithi125.stdout:1106: delete oid 9 current snap is 108 2021-03-24T21:58:10.189 INFO:tasks.rados.rados.0.smithi125.stdout:1105: finishing rollback tid 0 to smithi12556470-43 2021-03-24T21:58:10.193 INFO:tasks.rados.rados.0.smithi125.stdout:1105: finishing rollback tid 1 to smithi12556470-43 2021-03-24T21:58:10.194 INFO:tasks.rados.rados.0.smithi125.stdout:1105: finishing rollback tid 2 to smithi12556470-43 2021-03-24T21:58:10.194 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 43 v 330 (ObjNum 399 snap 102 seq_num 399) dirty exists 2021-03-24T21:58:10.194 INFO:tasks.rados.rados.0.smithi125.stdout:1105: done (1 left) 2021-03-24T21:58:10.195 INFO:tasks.rados.rados.0.smithi125.stdout:1106: done (0 left) 2021-03-24T21:58:10.195 INFO:tasks.rados.rados.0.smithi125.stdout:1107: write (excl) oid 13 current snap is 108 2021-03-24T21:58:10.195 INFO:tasks.rados.rados.0.smithi125.stdout:1107: seq_num 410 ranges {490870=618799,1592676=774144,2793074=97796} 2021-03-24T21:58:10.200 INFO:tasks.rados.rados.0.smithi125.stdout:1107: writing smithi12556470-13 from 490870 to 1109669 tid 1 2021-03-24T21:58:10.207 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:10 smithi125 conmon[24927]: cluster 2021-03-24T21:58:08.789383+0000 mgr.y (mgr.14144) 427 : cluster [DBG] pgmap v640: 168 pgs: 168 active+clean; 114 MiB data, 450 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 518 KiB/s wr, 2 op/s 2021-03-24T21:58:10.207 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:10 smithi125 conmon[24927]: cluster 2021-03-24T21:58:09.207273+0000 mon.a (mon.0) 1001 : cluster [DBG] osdmap e319: 8 total, 8 up, 6 in 2021-03-24T21:58:10.207 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:10 smithi125 conmon[24927]: cluster 2021-03-24T21:58:10.173443+0000 mon.a (mon. 2021-03-24T21:58:10.207 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:10 smithi125 conmon[24927]: 0) 1002 : cluster [DBG] osdmap e320: 8 total, 8 up, 6 in 2021-03-24T21:58:10.208 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:10 smithi125 conmon[19936]: cluster 2021-03-24T21:58:08.789383+0000 2021-03-24T21:58:10.208 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:10 smithi125 conmon[19936]: mgr.y (mgr.14144) 427 : cluster [DBG] pgmap v640: 168 pgs: 168 active+clean; 114 MiB data, 450 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 518 KiB/s wr, 2 op/s 2021-03-24T21:58:10.208 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:10 smithi125 conmon[19936]: cluster 2021-03-24T21:58:09. 2021-03-24T21:58:10.208 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:10 smithi125 conmon[19936]: 207273+0000 mon.a (mon.0) 1001 : cluster [DBG] osdmap e319: 8 total, 8 up, 6 in 2021-03-24T21:58:10.209 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:10 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:58:10.209 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:10 smithi125 conmon[19936]: 03-24T21:58:10.173443+0000 mon.a 2021-03-24T21:58:10.209 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:10 smithi125 conmon[19936]: (mon.0) 1002 : cluster [DBG] osdmap e320: 8 total, 8 up, 6 in 2021-03-24T21:58:10.209 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:10 smithi125 conmon[19936]: 2021-03-24T21:58:10.210 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:10.210 INFO:tasks.rados.rados.0.smithi125.stdout:1107: writing smithi12556470-13 from 1592676 to 2366820 tid 2 2021-03-24T21:58:10.212 INFO:tasks.rados.rados.0.smithi125.stdout:1107: writing smithi12556470-13 from 2793074 to 2890870 tid 3 2021-03-24T21:58:10.212 INFO:tasks.rados.rados.0.smithi125.stdout:1108: read oid 50 snap 107 2021-03-24T21:58:10.212 INFO:tasks.rados.rados.0.smithi125.stdout:1108: expect (ObjNum 392 snap 100 seq_num 392) 2021-03-24T21:58:10.213 INFO:tasks.rados.rados.0.smithi125.stdout:1109: read oid 35 snap 104 2021-03-24T21:58:10.213 INFO:tasks.rados.rados.0.smithi125.stdout:1109: expect (ObjNum 374 snap 96 seq_num 374) 2021-03-24T21:58:10.213 INFO:tasks.rados.rados.0.smithi125.stdout:1110: write oid 37 current snap is 108 2021-03-24T21:58:10.213 INFO:tasks.rados.rados.0.smithi125.stdout:1110: seq_num 411 ranges {507677=410401,1367511=517802,2420497=487180} 2021-03-24T21:58:10.217 INFO:tasks.rados.rados.0.smithi125.stdout:1110: writing smithi12556470-37 from 507677 to 918078 tid 1 2021-03-24T21:58:10.222 INFO:tasks.rados.rados.0.smithi125.stdout:1110: writing smithi12556470-37 from 1367511 to 1885313 tid 2 2021-03-24T21:58:10.228 INFO:tasks.rados.rados.0.smithi125.stdout:1110: writing smithi12556470-37 from 2420497 to 2907677 tid 3 2021-03-24T21:58:10.229 INFO:tasks.rados.rados.0.smithi125.stdout:1111: delete oid 25 current snap is 108 2021-03-24T21:58:10.229 INFO:tasks.rados.rados.0.smithi125.stdout:1109: expect (ObjNum 374 snap 96 seq_num 374) 2021-03-24T21:58:10.263 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:10.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:10.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:58:10.277 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:10 smithi142 conmon[20125]: cluster 2021-03-24T21:58:08 2021-03-24T21:58:10.278 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:10 smithi142 conmon[20125]: .789383+0000 mgr.y (mgr.14144) 427 : cluster [DBG] pgmap v640: 168 pgs: 168 active+clean; 114 MiB data, 450 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 518 KiB/s wr, 2 op/s 2021-03-24T21:58:10.278 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:10 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:58:10.278 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:10 smithi142 conmon[20125]: -03-24T21:58:09.207273+0000 mon.a (mon.0) 1001 : cluster [DBG] 2021-03-24T21:58:10.278 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:10 smithi142 conmon[20125]: osdmap e319: 8 total, 8 up, 6 in 2021-03-24T21:58:10.279 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:10 smithi142 conmon[20125]: cluster 2021-03-24T21:58:10. 2021-03-24T21:58:10.279 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:10 smithi142 conmon[20125]: 173443+0000 mon.a (mon.0) 1002 : cluster 2021-03-24T21:58:10.279 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:10 smithi142 conmon[20125]: [DBG] osdmap e320: 8 total, 8 up, 6 in 2021-03-24T21:58:10.340 INFO:tasks.rados.rados.0.smithi125.stdout:1108: expect (ObjNum 392 snap 100 seq_num 392) 2021-03-24T21:58:10.368 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:10.422 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:10.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:10.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:58:10.528 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:58:10.529 INFO:tasks.rados.rados.0.smithi125.stdout:1107: finishing write tid 1 to smithi12556470-13 2021-03-24T21:58:10.530 INFO:tasks.rados.rados.0.smithi125.stdout:1107: finishing write tid 2 to smithi12556470-13 2021-03-24T21:58:10.530 INFO:tasks.rados.rados.0.smithi125.stdout:1110: finishing write tid 1 to smithi12556470-37 2021-03-24T21:58:10.530 INFO:tasks.rados.rados.0.smithi125.stdout:1110: finishing write tid 2 to smithi12556470-37 2021-03-24T21:58:10.530 INFO:tasks.rados.rados.0.smithi125.stdout:1110: finishing write tid 3 to smithi12556470-37 2021-03-24T21:58:10.531 INFO:tasks.rados.rados.0.smithi125.stdout:1110: finishing write tid 4 to smithi12556470-37 2021-03-24T21:58:10.531 INFO:tasks.rados.rados.0.smithi125.stdout:1110: finishing write tid 5 to smithi12556470-37 2021-03-24T21:58:10.531 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 37 v 202 (ObjNum 411 snap 108 seq_num 411) dirty exists 2021-03-24T21:58:10.531 INFO:tasks.rados.rados.0.smithi125.stdout:1110: left oid 37 (ObjNum 411 snap 108 seq_num 411) 2021-03-24T21:58:10.531 INFO:tasks.rados.rados.0.smithi125.stdout:1107: finishing write tid 3 to smithi12556470-13 2021-03-24T21:58:10.532 INFO:tasks.rados.rados.0.smithi125.stdout:1107: finishing write tid 4 to smithi12556470-13 2021-03-24T21:58:10.532 INFO:tasks.rados.rados.0.smithi125.stdout:1107: finishing write tid 5 to smithi12556470-13 2021-03-24T21:58:10.532 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 13 v 265 (ObjNum 410 snap 108 seq_num 410) dirty exists 2021-03-24T21:58:10.532 INFO:tasks.rados.rados.0.smithi125.stdout:1107: left oid 13 (ObjNum 410 snap 108 seq_num 410) 2021-03-24T21:58:10.532 INFO:tasks.rados.rados.0.smithi125.stdout:1107: done (4 left) 2021-03-24T21:58:10.533 INFO:tasks.rados.rados.0.smithi125.stdout:1108: done (3 left) 2021-03-24T21:58:10.533 INFO:tasks.rados.rados.0.smithi125.stdout:1109: done (2 left) 2021-03-24T21:58:10.533 INFO:tasks.rados.rados.0.smithi125.stdout:1110: done (1 left) 2021-03-24T21:58:10.533 INFO:tasks.rados.rados.0.smithi125.stdout:1111: done (0 left) 2021-03-24T21:58:10.533 INFO:tasks.rados.rados.0.smithi125.stdout:1112: read oid 46 snap -1 2021-03-24T21:58:10.534 INFO:tasks.rados.rados.0.smithi125.stdout:1112: expect deleted 2021-03-24T21:58:10.534 INFO:tasks.rados.rados.0.smithi125.stdout:1113: write (excl) oid 3 current snap is 108 2021-03-24T21:58:10.534 INFO:tasks.rados.rados.0.smithi125.stdout:1113: seq_num 413 ranges {541291=560899,1884475=521471} 2021-03-24T21:58:10.534 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-24T21:58:10.546 INFO:tasks.rados.rados.0.smithi125.stdout:1113: writing smithi12556470-3 from 541291 to 1102190 tid 1 2021-03-24T21:58:10.554 INFO:tasks.rados.rados.0.smithi125.stdout:1113: writing smithi12556470-3 from 1884475 to 2405946 tid 2 2021-03-24T21:58:10.555 INFO:tasks.rados.rados.0.smithi125.stdout:1114: read oid 32 snap -1 2021-03-24T21:58:10.555 INFO:tasks.rados.rados.0.smithi125.stdout:1114: expect (ObjNum 370 snap 95 seq_num 370) 2021-03-24T21:58:10.555 INFO:tasks.rados.rados.0.smithi125.stdout:1115: write (excl) oid 8 current snap is 108 2021-03-24T21:58:10.555 INFO:tasks.rados.rados.0.smithi125.stdout:1115: seq_num 414 ranges {558098=636148,1826604=665129} 2021-03-24T21:58:10.562 INFO:tasks.rados.rados.0.smithi125.stdout:1115: writing smithi12556470-8 from 558098 to 1194246 tid 1 2021-03-24T21:58:10.571 INFO:tasks.rados.rados.0.smithi125.stdout:1115: writing smithi12556470-8 from 1826604 to 2491733 tid 2 2021-03-24T21:58:10.571 INFO:tasks.rados.rados.0.smithi125.stdout:1116: write (excl) oid 28 current snap is 108 2021-03-24T21:58:10.571 INFO:tasks.rados.rados.0.smithi125.stdout:1116: seq_num 415 ranges {574905=711397,1885086=525140,2930189=44716} 2021-03-24T21:58:10.579 INFO:tasks.rados.rados.0.smithi125.stdout:1116: writing smithi12556470-28 from 574905 to 1286302 tid 1 2021-03-24T21:58:10.588 INFO:tasks.rados.rados.0.smithi125.stdout:1116: writing smithi12556470-28 from 1885086 to 2410226 tid 2 2021-03-24T21:58:10.592 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:58:10.595 INFO:tasks.rados.rados.0.smithi125.stdout:1116: writing smithi12556470-28 from 2930189 to 2974905 tid 3 2021-03-24T21:58:10.595 INFO:tasks.rados.rados.0.smithi125.stdout:1117: write (excl) oid 4 current snap is 108 2021-03-24T21:58:10.595 INFO:tasks.rados.rados.0.smithi125.stdout:1117: seq_num 416 ranges {591712=786646,1943568=668798} 2021-03-24T21:58:10.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:10.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:58:10.605 INFO:tasks.rados.rados.0.smithi125.stdout:1117: writing smithi12556470-4 from 591712 to 1378358 tid 1 2021-03-24T21:58:10.613 INFO:tasks.rados.rados.0.smithi125.stdout:1117: writing smithi12556470-4 from 1943568 to 2612366 tid 2 2021-03-24T21:58:10.613 INFO:tasks.rados.rados.0.smithi125.stdout:1112: done (5 left) 2021-03-24T21:58:10.614 INFO:tasks.rados.rados.0.smithi125.stdout:1118: copy_from oid 15 from oid 47 current snap is 108 2021-03-24T21:58:10.614 INFO:tasks.rados.rados.0.smithi125.stdout:1113: finishing write tid 1 to smithi12556470-3 2021-03-24T21:58:10.614 INFO:tasks.rados.rados.0.smithi125.stdout:1119: read oid 48 snap -1 2021-03-24T21:58:10.614 INFO:tasks.rados.rados.0.smithi125.stdout:1119: expect (ObjNum 193 snap 42 seq_num 193) 2021-03-24T21:58:10.614 INFO:tasks.rados.rados.0.smithi125.stdout:1120: copy_from oid 34 from oid 37 current snap is 108 2021-03-24T21:58:10.615 INFO:tasks.rados.rados.0.smithi125.stdout:1114: expect (ObjNum 370 snap 95 seq_num 370) 2021-03-24T21:58:10.698 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:10.751 INFO:tasks.rados.rados.0.smithi125.stdout:1113: finishing write tid 2 to smithi12556470-3 2021-03-24T21:58:10.751 INFO:tasks.rados.rados.0.smithi125.stdout:1113: finishing write tid 3 to smithi12556470-3 2021-03-24T21:58:10.751 INFO:tasks.rados.rados.0.smithi125.stdout:1113: finishing write tid 4 to smithi12556470-3 2021-03-24T21:58:10.751 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 3 v 130 (ObjNum 413 snap 108 seq_num 413) dirty exists 2021-03-24T21:58:10.751 INFO:tasks.rados.rados.0.smithi125.stdout:1113: left oid 3 (ObjNum 413 snap 108 seq_num 413) 2021-03-24T21:58:10.752 INFO:tasks.rados.rados.0.smithi125.stdout:1115: finishing write tid 1 to smithi12556470-8 2021-03-24T21:58:10.752 INFO:tasks.rados.rados.0.smithi125.stdout:1115: finishing write tid 2 to smithi12556470-8 2021-03-24T21:58:10.752 INFO:tasks.rados.rados.0.smithi125.stdout:1113: done (7 left) 2021-03-24T21:58:10.752 INFO:tasks.rados.rados.0.smithi125.stdout:1114: done (6 left) 2021-03-24T21:58:10.753 INFO:tasks.rados.rados.0.smithi125.stdout:1121: write oid 44 current snap is 108 2021-03-24T21:58:10.753 INFO:tasks.rados.rados.0.smithi125.stdout:1115: finishing write tid 3 to smithi12556470-8 2021-03-24T21:58:10.753 INFO:tasks.rados.rados.0.smithi125.stdout:1115: finishing write tid 4 to smithi12556470-8 2021-03-24T21:58:10.753 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 8 v 75 (ObjNum 414 snap 108 seq_num 414) dirty exists 2021-03-24T21:58:10.753 INFO:tasks.rados.rados.0.smithi125.stdout:1115: left oid 8 (ObjNum 414 snap 108 seq_num 414) 2021-03-24T21:58:10.754 INFO:tasks.rados.rados.0.smithi125.stdout:1116: finishing write tid 1 to smithi12556470-28 2021-03-24T21:58:10.754 INFO:tasks.rados.rados.0.smithi125.stdout:1116: finishing write tid 2 to smithi12556470-28 2021-03-24T21:58:10.754 INFO:tasks.rados.rados.0.smithi125.stdout:1121: seq_num 417 ranges {608519=461895,1602050=528809,2585035=423484} 2021-03-24T21:58:10.755 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:10.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:10.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:58:10.758 INFO:tasks.rados.rados.0.smithi125.stdout:1121: writing smithi12556470-44 from 608519 to 1070414 tid 1 2021-03-24T21:58:10.765 INFO:tasks.rados.rados.0.smithi125.stdout:1121: writing smithi12556470-44 from 1602050 to 2130859 tid 2 2021-03-24T21:58:10.772 INFO:tasks.rados.rados.0.smithi125.stdout:1121: writing smithi12556470-44 from 2585035 to 3008519 tid 3 2021-03-24T21:58:10.772 INFO:tasks.rados.rados.0.smithi125.stdout:1115: done (6 left) 2021-03-24T21:58:10.772 INFO:tasks.rados.rados.0.smithi125.stdout:1122: rollback oid 41 current snap is 108 2021-03-24T21:58:10.772 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 41 to 105 2021-03-24T21:58:10.773 INFO:tasks.rados.rados.0.smithi125.stdout:1116: finishing write tid 3 to smithi12556470-28 2021-03-24T21:58:10.773 INFO:tasks.rados.rados.0.smithi125.stdout:1116: finishing write tid 4 to smithi12556470-28 2021-03-24T21:58:10.773 INFO:tasks.rados.rados.0.smithi125.stdout:1116: finishing write tid 5 to smithi12556470-28 2021-03-24T21:58:10.773 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 28 v 290 (ObjNum 415 snap 108 seq_num 415) dirty exists 2021-03-24T21:58:10.773 INFO:tasks.rados.rados.0.smithi125.stdout:1116: left oid 28 (ObjNum 415 snap 108 seq_num 415) 2021-03-24T21:58:10.774 INFO:tasks.rados.rados.0.smithi125.stdout:1117: finishing write tid 1 to smithi12556470-4 2021-03-24T21:58:10.774 INFO:tasks.rados.rados.0.smithi125.stdout:1117: finishing write tid 2 to smithi12556470-4 2021-03-24T21:58:10.774 INFO:tasks.rados.rados.0.smithi125.stdout:1117: finishing write tid 3 to smithi12556470-4 2021-03-24T21:58:10.774 INFO:tasks.rados.rados.0.smithi125.stdout:1117: finishing write tid 4 to smithi12556470-4 2021-03-24T21:58:10.774 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 4 v 283 (ObjNum 416 snap 108 seq_num 416) dirty exists 2021-03-24T21:58:10.775 INFO:tasks.rados.rados.0.smithi125.stdout:1117: left oid 4 (ObjNum 416 snap 108 seq_num 416) 2021-03-24T21:58:10.775 INFO:tasks.rados.rados.0.smithi125.stdout:1116: done (6 left) 2021-03-24T21:58:10.775 INFO:tasks.rados.rados.0.smithi125.stdout:1117: done (5 left) 2021-03-24T21:58:10.775 INFO:tasks.rados.rados.0.smithi125.stdout:1123: write oid 21 current snap is 108 2021-03-24T21:58:10.776 INFO:tasks.rados.rados.0.smithi125.stdout:1119: expect (ObjNum 193 snap 42 seq_num 193) 2021-03-24T21:58:10.846 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:58:10.858 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:10.914 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:10.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:10.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:58:10.976 INFO:tasks.rados.rados.0.smithi125.stdout:1118: finishing copy_from to smithi12556470-15 2021-03-24T21:58:10.976 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 15 v 103 (ObjNum 384 snap 99 seq_num 384) dirty exists 2021-03-24T21:58:10.976 INFO:tasks.rados.rados.0.smithi125.stdout:1118: finishing copy_from racing read to smithi12556470-15 2021-03-24T21:58:10.977 INFO:tasks.rados.rados.0.smithi125.stdout:1120: finishing copy_from to smithi12556470-34 2021-03-24T21:58:10.977 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 34 v 202 (ObjNum 411 snap 108 seq_num 411) dirty exists 2021-03-24T21:58:10.977 INFO:tasks.rados.rados.0.smithi125.stdout:1120: finishing copy_from racing read to smithi12556470-34 2021-03-24T21:58:10.977 INFO:tasks.rados.rados.0.smithi125.stdout:1121: finishing write tid 1 to smithi12556470-44 2021-03-24T21:58:10.977 INFO:tasks.rados.rados.0.smithi125.stdout:1121: finishing write tid 2 to smithi12556470-44 2021-03-24T21:58:10.978 INFO:tasks.rados.rados.0.smithi125.stdout:1123: seq_num 418 ranges {625326=537144,1944179=672467} 2021-03-24T21:58:10.987 INFO:tasks.rados.rados.0.smithi125.stdout:1123: writing smithi12556470-21 from 625326 to 1162470 tid 1 2021-03-24T21:58:10.996 INFO:tasks.rados.rados.0.smithi125.stdout:1123: writing smithi12556470-21 from 1944179 to 2616646 tid 2 2021-03-24T21:58:10.996 INFO:tasks.rados.rados.0.smithi125.stdout:1118: done (5 left) 2021-03-24T21:58:10.996 INFO:tasks.rados.rados.0.smithi125.stdout:1119: done (4 left) 2021-03-24T21:58:10.996 INFO:tasks.rados.rados.0.smithi125.stdout:1120: done (3 left) 2021-03-24T21:58:10.997 INFO:tasks.rados.rados.0.smithi125.stdout:1124: copy_from oid 48 from oid 49 current snap is 108 2021-03-24T21:58:10.997 INFO:tasks.rados.rados.0.smithi125.stdout:1121: finishing write tid 3 to smithi12556470-44 2021-03-24T21:58:10.997 INFO:tasks.rados.rados.0.smithi125.stdout:1125: write oid 40 current snap is 108 2021-03-24T21:58:10.997 INFO:tasks.rados.rados.0.smithi125.stdout:1121: finishing write tid 4 to smithi12556470-44 2021-03-24T21:58:10.997 INFO:tasks.rados.rados.0.smithi125.stdout:1122: finishing rollback tid 0 to smithi12556470-41 2021-03-24T21:58:10.998 INFO:tasks.rados.rados.0.smithi125.stdout:1121: finishing write tid 5 to smithi12556470-44 2021-03-24T21:58:10.998 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 44 v 298 (ObjNum 417 snap 108 seq_num 417) dirty exists 2021-03-24T21:58:10.998 INFO:tasks.rados.rados.0.smithi125.stdout:1121: left oid 44 (ObjNum 417 snap 108 seq_num 417) 2021-03-24T21:58:10.998 INFO:tasks.rados.rados.0.smithi125.stdout:1125: seq_num 419 ranges {642133=728746,2119014=532478} 2021-03-24T21:58:11.003 INFO:tasks.rados.rados.0.smithi125.stdout:1125: writing smithi12556470-40 from 642133 to 1370879 tid 1 2021-03-24T21:58:11.011 INFO:tasks.rados.rados.0.smithi125.stdout:1125: writing smithi12556470-40 from 2119014 to 2651492 tid 2 2021-03-24T21:58:11.012 INFO:tasks.rados.rados.0.smithi125.stdout:1121: done (4 left) 2021-03-24T21:58:11.012 INFO:tasks.rados.rados.0.smithi125.stdout:1126: write (excl) oid 31 current snap is 108 2021-03-24T21:58:11.012 INFO:tasks.rados.rados.0.smithi125.stdout:1122: finishing rollback tid 1 to smithi12556470-41 2021-03-24T21:58:11.012 INFO:tasks.rados.rados.0.smithi125.stdout:1126: seq_num 420 ranges {658940=403995,1777496=676136,2950951=107989} 2021-03-24T21:58:11.017 INFO:tasks.rados.rados.0.smithi125.stdout:1126: writing smithi12556470-31 from 658940 to 1062935 tid 1 2021-03-24T21:58:11.020 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:11.025 INFO:tasks.rados.rados.0.smithi125.stdout:1126: writing smithi12556470-31 from 1777496 to 2453632 tid 2 2021-03-24T21:58:11.030 INFO:tasks.rados.rados.0.smithi125.stdout:1126: writing smithi12556470-31 from 2950951 to 3058940 tid 3 2021-03-24T21:58:11.030 INFO:tasks.rados.rados.0.smithi125.stdout:1127: read oid 45 snap 107 2021-03-24T21:58:11.030 INFO:tasks.rados.rados.0.smithi125.stdout:1127: expect deleted 2021-03-24T21:58:11.031 INFO:tasks.rados.rados.0.smithi125.stdout:1128: copy_from oid 6 from oid 22 current snap is 108 2021-03-24T21:58:11.031 INFO:tasks.rados.rados.0.smithi125.stdout:1122: finishing rollback tid 2 to smithi12556470-41 2021-03-24T21:58:11.031 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 41 v 322 (ObjNum 385 snap 99 seq_num 385) dirty exists 2021-03-24T21:58:11.031 INFO:tasks.rados.rados.0.smithi125.stdout:1123: finishing write tid 1 to smithi12556470-21 2021-03-24T21:58:11.031 INFO:tasks.rados.rados.0.smithi125.stdout:1123: finishing write tid 2 to smithi12556470-21 2021-03-24T21:58:11.032 INFO:tasks.rados.rados.0.smithi125.stdout:1123: finishing write tid 3 to smithi12556470-21 2021-03-24T21:58:11.032 INFO:tasks.rados.rados.0.smithi125.stdout:1123: finishing write tid 4 to smithi12556470-21 2021-03-24T21:58:11.032 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 21 v 302 (ObjNum 418 snap 108 seq_num 418) dirty exists 2021-03-24T21:58:11.032 INFO:tasks.rados.rados.0.smithi125.stdout:1123: left oid 21 (ObjNum 418 snap 108 seq_num 418) 2021-03-24T21:58:11.033 INFO:tasks.rados.rados.0.smithi125.stdout:1125: finishing write tid 1 to smithi12556470-40 2021-03-24T21:58:11.033 INFO:tasks.rados.rados.0.smithi125.stdout:1122: done (6 left) 2021-03-24T21:58:11.033 INFO:tasks.rados.rados.0.smithi125.stdout:1123: done (5 left) 2021-03-24T21:58:11.033 INFO:tasks.rados.rados.0.smithi125.stdout:1129: snap_create 2021-03-24T21:58:11.033 INFO:tasks.rados.rados.0.smithi125.stdout:1125: finishing write tid 2 to smithi12556470-40 2021-03-24T21:58:11.034 INFO:tasks.rados.rados.0.smithi125.stdout:1125: finishing write tid 3 to smithi12556470-40 2021-03-24T21:58:11.034 INFO:tasks.rados.rados.0.smithi125.stdout:1124: finishing copy_from to smithi12556470-48 2021-03-24T21:58:11.034 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 48 v 340 (ObjNum 391 snap 100 seq_num 391) dirty exists 2021-03-24T21:58:11.034 INFO:tasks.rados.rados.0.smithi125.stdout:1125: finishing write tid 4 to smithi12556470-40 2021-03-24T21:58:11.034 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 40 v 336 (ObjNum 419 snap 108 seq_num 419) dirty exists 2021-03-24T21:58:11.035 INFO:tasks.rados.rados.0.smithi125.stdout:1125: left oid 40 (ObjNum 419 snap 108 seq_num 419) 2021-03-24T21:58:11.035 INFO:tasks.rados.rados.0.smithi125.stdout:1124: finishing copy_from racing read to smithi12556470-48 2021-03-24T21:58:11.035 INFO:tasks.rados.rados.0.smithi125.stdout:1126: finishing write tid 1 to smithi12556470-31 2021-03-24T21:58:11.035 INFO:tasks.rados.rados.0.smithi125.stdout:1126: finishing write tid 2 to smithi12556470-31 2021-03-24T21:58:11.036 INFO:tasks.rados.rados.0.smithi125.stdout:1126: finishing write tid 3 to smithi12556470-31 2021-03-24T21:58:11.036 INFO:tasks.rados.rados.0.smithi125.stdout:1126: finishing write tid 4 to smithi12556470-31 2021-03-24T21:58:11.037 INFO:tasks.rados.rados.0.smithi125.stdout:1126: finishing write tid 5 to smithi12556470-31 2021-03-24T21:58:11.037 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 31 v 341 (ObjNum 420 snap 108 seq_num 420) dirty exists 2021-03-24T21:58:11.037 INFO:tasks.rados.rados.0.smithi125.stdout:1126: left oid 31 (ObjNum 420 snap 108 seq_num 420) 2021-03-24T21:58:11.048 INFO:tasks.rados.rados.0.smithi125.stdout:1128: finishing copy_from to smithi12556470-6 2021-03-24T21:58:11.048 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 6 v 294 (ObjNum 402 snap 104 seq_num 402) dirty exists 2021-03-24T21:58:11.048 INFO:tasks.rados.rados.0.smithi125.stdout:1128: finishing copy_from racing read to smithi12556470-6 2021-03-24T21:58:11.070 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:11.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:11.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:11.175 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:11.218 INFO:tasks.rados.rados.0.smithi125.stdout:1124: done (5 left) 2021-03-24T21:58:11.218 INFO:tasks.rados.rados.0.smithi125.stdout:1125: done (4 left) 2021-03-24T21:58:11.218 INFO:tasks.rados.rados.0.smithi125.stdout:1126: done (3 left) 2021-03-24T21:58:11.219 INFO:tasks.rados.rados.0.smithi125.stdout:1127: done (2 left) 2021-03-24T21:58:11.219 INFO:tasks.rados.rados.0.smithi125.stdout:1128: done (1 left) 2021-03-24T21:58:11.219 INFO:tasks.rados.rados.0.smithi125.stdout:1129: done (0 left) 2021-03-24T21:58:11.219 INFO:tasks.rados.rados.0.smithi125.stdout:1130: delete oid 49 current snap is 109 2021-03-24T21:58:11.223 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:11.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:11.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:11.229 INFO:tasks.rados.rados.0.smithi125.stdout:1130: done (0 left) 2021-03-24T21:58:11.229 INFO:tasks.rados.rados.0.smithi125.stdout:1131: snap_create 2021-03-24T21:58:11.329 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:58:11.382 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:58:11.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:11.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:58:11.488 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:11.547 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:11.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:11.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:58:11.564 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2021-03-24T21:58:11.564 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2021-03-24T21:58:11.564 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2021-03-24T21:58:11.565 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2021-03-24T21:58:11.565 INFO:teuthology.orchestra.run.smithi142.stdout:} 2021-03-24T21:58:11.651 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:58:11.702 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:58:11.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:11.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:58:11.808 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:11.870 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:11.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:11.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:58:11.976 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:58:12.035 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:58:12.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:12.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:58:12.141 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:12.200 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:12.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:12.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:12.231 INFO:tasks.rados.rados.0.smithi125.stdout:1131: done (0 left) 2021-03-24T21:58:12.232 INFO:tasks.rados.rados.0.smithi125.stdout:1132: snap_create 2021-03-24T21:58:12.234 INFO:teuthology.orchestra.run.smithi125.stderr:nodeep-scrub is set 2021-03-24T21:58:12.307 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:12.360 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:12.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:12.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:12.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:12 smithi125 conmon[24927]: cluster 2021-03-24T21:58 2021-03-24T21:58:12.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:12 smithi125 conmon[24927]: :10.596406+0000 osd.0 (osd.0) 80 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:58:12.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:12 smithi125 conmon[24927]: cluster 2021-03-24T21:58: 2021-03-24T21:58:12.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:12 smithi125 conmon[24927]: 10.596886+0000 osd.0 (osd.0) 81 : cluster 2021-03-24T21:58:12.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:12 smithi125 conmon[24927]: [DBG] purged_snaps scrub ok 2021-03-24T21:58:12.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:12 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:58:12.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:12 smithi125 conmon[24927]: 21:58:10.789909+0000 mgr.y (mgr. 2021-03-24T21:58:12.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:12 smithi125 conmon[24927]: 14144) 428 : cluster [DBG] pgmap v643: 168 pgs: 168 active+clean; 121 MiB data, 462 MiB used, 536 GiB / 536 GiB avail; 15 MiB/s rd, 2.7 MiB/s wr, 22 op/s 2021-03-24T21:58:12.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:12 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:58:12.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:12 smithi125 conmon[24927]: -03-24T21:58:11.216339+0000 2021-03-24T21:58:12.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:12 smithi125 conmon[24927]: mon.a (mon.0) 1003 : cluster [DBG] osdmap e321: 8 total, 8 up, 6 in 2021-03-24T21:58:12.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:12 smithi125 conmon[24927]: audit 2021- 2021-03-24T21:58:12.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:12 smithi125 conmon[24927]: 03-24T21:58:11.508906+0000 mon.c ( 2021-03-24T21:58:12.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:12 smithi125 conmon[24927]: mon.2) 51 : audit [INF] from='client.? v1:172.21.15.125:0/4283850636' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:58:12.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:12 smithi125 conmon[24927]: audit 2021 2021-03-24T21:58:12.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:12 smithi125 conmon[24927]: -03-24T21:58:11.509536+0000 2021-03-24T21:58:12.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:12 smithi125 conmon[24927]: mon.a (mon.0) 1004 : audit [INF] from='client.35057 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:58:12.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:12 smithi125 conmon[24927]: 2021-03-24T21:58:12.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:12 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:58:12.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:12 smithi125 conmon[19936]: 58:10.596406+0000 osd.0 (osd.0) 80 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:58:12.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:12 smithi125 conmon[19936]: cluster 2021-03-24T21:58:12.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:12 smithi125 conmon[19936]: 2021-03-24T 2021-03-24T21:58:12.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:12 smithi125 conmon[19936]: 21:58:10 2021-03-24T21:58:12.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:12 smithi125 conmon[19936]: .596886+0000 osd.0 2021-03-24T21:58:12.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:12 smithi125 conmon[19936]: (osd.0) 81 2021-03-24T21:58:12.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:12 smithi125 conmon[19936]: : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:58:12.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:12 smithi125 conmon[19936]: 2021-03-24T21:58:12.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:12 smithi125 conmon[19936]: cluster 2021-03-24T21:58:12.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:12 smithi125 conmon[19936]: 2021-03-24 2021-03-24T21:58:12.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:12 smithi125 conmon[19936]: T21:58:10 2021-03-24T21:58:12.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:12 smithi125 conmon[19936]: .789909+0000 mgr.y ( 2021-03-24T21:58:12.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:12 smithi125 conmon[19936]: mgr.14144) 428 : 2021-03-24T21:58:12.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:12 smithi125 conmon[19936]: cluster [DBG] pgmap v643: 168 pgs: 168 active+clean; 121 MiB data, 462 MiB used, 536 GiB / 536 GiB avail; 15 MiB/s rd, 2.7 MiB/s wr, 22 op/s 2021-03-24T21:58:12.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:12 smithi125 conmon[19936]: cluster 2021-03-24T21:58:12.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:12 smithi125 conmon[19936]: 2021-03-24 2021-03-24T21:58:12.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:12 smithi125 conmon[19936]: T21:58: 2021-03-24T21:58:12.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:12 smithi125 conmon[19936]: 11.216339+0000 2021-03-24T21:58:12.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:12 smithi125 conmon[19936]: mon.a (mon.0) 1003 : cluster 2021-03-24T21:58:12.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:12 smithi125 conmon[19936]: [DBG] osdmap e321: 8 total, 8 up, 6 in 2021-03-24T21:58:12.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:12 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:58:12.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:12 smithi125 conmon[19936]: 03-24T21: 2021-03-24T21:58:12.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:12 smithi125 conmon[19936]: 58:11.508906+0000 mon.c ( 2021-03-24T21:58:12.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:12 smithi125 conmon[19936]: mon.2) 51 : audit [INF] from='client.? v1:172.21.15.125:0/4283850636' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:58:12.421 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:12 smithi125 conmon[19936]: 2021-03-24T21:58:12.421 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:12 smithi125 conmon[19936]: audit 2021-03-24T21:58:12.421 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:12 smithi125 conmon[19936]: 2021-03-24T21:58 2021-03-24T21:58:12.421 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:12 smithi125 conmon[19936]: :11.509536+0000 mon.a (mon 2021-03-24T21:58:12.421 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:12 smithi125 conmon[19936]: .0) 1004 : audit [INF] from='client.35057 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:58:12.422 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:12 smithi125 conmon[19936]: 2021-03-24T21:58:12.468 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:12.528 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:12.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:12.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:12.542 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:12 smithi142 conmon[20125]: cluster 2021-03-24T21:58:10. 2021-03-24T21:58:12.543 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:12 smithi142 conmon[20125]: 596406+0000 osd.0 (osd.0) 80 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:58:12.543 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:12 smithi142 conmon[20125]: cluster 2021-03-24T21:58:10.596886+0000 2021-03-24T21:58:12.543 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:12 smithi142 conmon[20125]: osd.0 (osd.0) 81 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:58:12.543 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:12 smithi142 conmon[20125]: cluster 2021-03-24T21:58:10.789909+0000 mgr.y (mgr 2021-03-24T21:58:12.544 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:12 smithi142 conmon[20125]: .14144) 428 : cluster [DBG] pgmap v643: 168 pgs: 168 active+clean; 121 MiB data, 462 MiB used, 536 GiB / 536 GiB avail; 15 MiB/s rd, 2.7 MiB/s wr, 22 op/s 2021-03-24T21:58:12.544 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:12 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:58:12.544 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:12 smithi142 conmon[20125]: -24T21:58:11.216339+0000 mon.a (mon 2021-03-24T21:58:12.544 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:12 smithi142 conmon[20125]: .0) 1003 : cluster [DBG] osdmap e321: 8 total, 8 up, 6 in 2021-03-24T21:58:12.545 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:12 smithi142 conmon[20125]: audit 2021-03- 2021-03-24T21:58:12.545 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:12 smithi142 conmon[20125]: 24T21:58:11.508906+0000 mon.c (mon.2) 51 2021-03-24T21:58:12.545 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:12 smithi142 conmon[20125]: : audit [INF] from='client.? v1:172.21.15.125:0/4283850636' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:58:12.545 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:12 smithi142 conmon[20125]: audit 2021-03-24T21:58:11.509536 2021-03-24T21:58:12.545 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:12 smithi142 conmon[20125]: +0000 mon.a (mon.0) 1004 : audit [INF] from='client.35057 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:58:12.546 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:12 smithi142 conmon[20125]: 2021-03-24T21:58:12.574 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.6 dump_historic_ops 2021-03-24T21:58:12.633 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:12.694 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:12.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:12.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:58:12.800 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:12.856 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:12.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:12.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:58:12.868 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T21:58:12.962 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:58:13.012 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:58:13.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:13.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:58:13.056 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 7] out_osds: [6, 1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-03-24T21:58:13.056 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-24T21:58:13.057 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_0 2021-03-24T21:58:13.057 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph pg dump --format=json 2021-03-24T21:58:13.119 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:13.179 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:13.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:13.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:58:13.243 INFO:tasks.rados.rados.0.smithi125.stdout:1132: done (0 left) 2021-03-24T21:58:13.243 INFO:tasks.rados.rados.0.smithi125.stdout:1133: read oid 40 snap -1 2021-03-24T21:58:13.243 INFO:tasks.rados.rados.0.smithi125.stdout:1133: expect (ObjNum 419 snap 108 seq_num 419) 2021-03-24T21:58:13.244 INFO:tasks.rados.rados.0.smithi125.stdout:1134: write (excl) oid 10 current snap is 111 2021-03-24T21:58:13.244 INFO:tasks.rados.rados.0.smithi125.stdout:1134: seq_num 422 ranges {692554=554493,1778107=679805,2889444=203110} 2021-03-24T21:58:13.245 INFO:tasks.rados.rados.0.smithi125.stdout:1134: writing smithi12556470-10 from 692554 to 1247047 tid 1 2021-03-24T21:58:13.252 INFO:tasks.rados.rados.0.smithi125.stdout:1134: writing smithi12556470-10 from 1778107 to 2457912 tid 2 2021-03-24T21:58:13.256 INFO:tasks.rados.rados.0.smithi125.stdout:1134: writing smithi12556470-10 from 2889444 to 3092554 tid 3 2021-03-24T21:58:13.257 INFO:tasks.rados.rados.0.smithi125.stdout:1135: snap_create 2021-03-24T21:58:13.257 INFO:tasks.rados.rados.0.smithi125.stdout:1133: expect (ObjNum 419 snap 108 seq_num 419) 2021-03-24T21:58:13.287 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:13.289 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:13 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:58:13.290 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:13 smithi125 conmon[19936]: 21:58:12.214109+0000 mon.a (mon.0) 1005 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:58:13.290 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:13 smithi125 conmon[19936]: audit 2021-03- 2021-03-24T21:58:13.290 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:13 smithi125 conmon[19936]: 24T21:58:12.226802+0000 mon.a (mon.0) 1006 : audit [INF] from='client.35057 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T21:58:13.290 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:13 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:58:13.291 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:13 smithi125 conmon[19936]: 21:58:12.226897+0000 mon.a (mon. 2021-03-24T21:58:13.291 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:13 smithi125 conmon[19936]: 0) 1007 : cluster [DBG] osdmap e322: 8 total, 8 up, 6 in 2021-03-24T21:58:13.291 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:13 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:58:13.291 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:13 smithi125 conmon[24927]: :58:12.214109+0000 mon.a (mon.0) 1005 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:58:13.292 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:13 smithi125 conmon[24927]: audit 2021-03-24T21 2021-03-24T21:58:13.292 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:13 smithi125 conmon[24927]: :58:12.226802+0000 mon.a (mon.0 2021-03-24T21:58:13.292 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:13 smithi125 conmon[24927]: ) 1006 : audit [INF] from='client.35057 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T21:58:13.292 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:13 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:58:13.292 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:13 smithi125 conmon[24927]: 03-24T21:58:12.226897+0000 mon.a 2021-03-24T21:58:13.292 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:13 smithi125 conmon[24927]: (mon.0) 1007 : cluster [DBG] osdmap e322: 8 total, 8 up, 6 in 2021-03-24T21:58:13.344 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:13.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:13.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:58:13.375 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:13 smithi142 conmon[20125]: cluster 2021-03-24T21:58 2021-03-24T21:58:13.376 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:13 smithi142 conmon[20125]: :12.214109+0000 mon.a (mon.0) 1005 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:58:13.376 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:13 smithi142 conmon[20125]: audit 2021-03- 2021-03-24T21:58:13.376 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:13 smithi142 conmon[20125]: 24T21:58:12.226802+0000 mon.a (mon.0) 1006 : audit 2021-03-24T21:58:13.376 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:13 smithi142 conmon[20125]: [INF] from='client.35057 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T21:58:13.376 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:13 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:58:13.377 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:13 smithi142 conmon[20125]: -03-24T21:58:12.226897+0000 mon.a (mon 2021-03-24T21:58:13.377 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:13 smithi142 conmon[20125]: .0) 1007 : cluster [DBG] osdmap e322: 8 total, 8 up, 6 in 2021-03-24T21:58:13.383 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:58:13.449 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:58:13.496 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:58:13.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:13.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:58:13.546 INFO:tasks.rados.rados.0.smithi125.stdout:1134: finishing write tid 1 to smithi12556470-10 2021-03-24T21:58:13.546 INFO:tasks.rados.rados.0.smithi125.stdout:1134: finishing write tid 2 to smithi12556470-10 2021-03-24T21:58:13.547 INFO:tasks.rados.rados.0.smithi125.stdout:1134: finishing write tid 3 to smithi12556470-10 2021-03-24T21:58:13.547 INFO:tasks.rados.rados.0.smithi125.stdout:1134: finishing write tid 4 to smithi12556470-10 2021-03-24T21:58:13.547 INFO:tasks.rados.rados.0.smithi125.stdout:1134: finishing write tid 5 to smithi12556470-10 2021-03-24T21:58:13.547 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 10 v 339 (ObjNum 422 snap 111 seq_num 422) dirty exists 2021-03-24T21:58:13.547 INFO:tasks.rados.rados.0.smithi125.stdout:1134: left oid 10 (ObjNum 422 snap 111 seq_num 422) 2021-03-24T21:58:13.603 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:13.657 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:13.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:13.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:13.697 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T21:58:13.721 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T21:58:13.744 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T21:58:13.762 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:13.763 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T21:58:13.793 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T21:58:13.808 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:13.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:13.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:58:13.816 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T21:58:13.851 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T21:58:13.875 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T21:58:13.878 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-24T21:58:13.878 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-24T21:58:13.879 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-24T21:58:13.879 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-24T21:58:13.879 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-24T21:58:13.879 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-24T21:58:13.879 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-24T21:58:13.880 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-24T21:58:13.913 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:58:13.969 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:58:13.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:13.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:58:14.076 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:14.139 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:14.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:14.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:58:14.244 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:58:14.251 INFO:tasks.rados.rados.0.smithi125.stdout:1133: done (2 left) 2021-03-24T21:58:14.251 INFO:tasks.rados.rados.0.smithi125.stdout:1134: done (1 left) 2021-03-24T21:58:14.252 INFO:tasks.rados.rados.0.smithi125.stdout:1135: done (0 left) 2021-03-24T21:58:14.252 INFO:tasks.rados.rados.0.smithi125.stdout:1136: write (excl) oid 48 current snap is 112 2021-03-24T21:58:14.252 INFO:tasks.rados.rados.0.smithi125.stdout:1136: seq_num 423 ranges {709361=629742,1836589=539816,3033220=76141} 2021-03-24T21:58:14.263 INFO:tasks.rados.rados.0.smithi125.stdout:1136: writing smithi12556470-48 from 709361 to 1339103 tid 1 2021-03-24T21:58:14.269 INFO:tasks.rados.rados.0.smithi125.stdout:1136: writing smithi12556470-48 from 1836589 to 2376405 tid 2 2021-03-24T21:58:14.274 INFO:tasks.rados.rados.0.smithi125.stdout:1136: writing smithi12556470-48 from 3033220 to 3109361 tid 3 2021-03-24T21:58:14.275 INFO:tasks.rados.rados.0.smithi125.stdout:1137: read oid 20 snap 105 2021-03-24T21:58:14.275 INFO:tasks.rados.rados.0.smithi125.stdout:1137: expect (ObjNum 380 snap 99 seq_num 380) 2021-03-24T21:58:14.275 INFO:tasks.rados.rados.0.smithi125.stdout:1138: read oid 41 snap -1 2021-03-24T21:58:14.275 INFO:tasks.rados.rados.0.smithi125.stdout:1138: expect (ObjNum 385 snap 99 seq_num 385) 2021-03-24T21:58:14.275 INFO:tasks.rados.rados.0.smithi125.stdout:1136: finishing write tid 1 to smithi12556470-48 2021-03-24T21:58:14.276 INFO:tasks.rados.rados.0.smithi125.stdout:1139: write (excl) oid 28 current snap is 112 2021-03-24T21:58:14.276 INFO:tasks.rados.rados.0.smithi125.stdout:1139: seq_num 424 ranges {726168=704991,1895071=683474} 2021-03-24T21:58:14.283 INFO:tasks.rados.rados.0.smithi125.stdout:1139: writing smithi12556470-28 from 726168 to 1431159 tid 1 2021-03-24T21:58:14.290 INFO:tasks.rados.rados.0.smithi125.stdout:1139: writing smithi12556470-28 from 1895071 to 2578545 tid 2 2021-03-24T21:58:14.290 INFO:tasks.rados.rados.0.smithi125.stdout:1140: write oid 17 current snap is 112 2021-03-24T21:58:14.290 INFO:tasks.rados.rados.0.smithi125.stdout:1140: seq_num 425 ranges {742975=780240,1953553=427132,2971713=171262} 2021-03-24T21:58:14.298 INFO:tasks.rados.rados.0.smithi125.stdout:1140: writing smithi12556470-17 from 742975 to 1523215 tid 1 2021-03-24T21:58:14.301 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:58:14.303 INFO:tasks.rados.rados.0.smithi125.stdout:1140: writing smithi12556470-17 from 1953553 to 2380685 tid 2 2021-03-24T21:58:14.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:14.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:58:14.308 INFO:tasks.rados.rados.0.smithi125.stdout:1140: writing smithi12556470-17 from 2971713 to 3142975 tid 3 2021-03-24T21:58:14.308 INFO:tasks.rados.rados.0.smithi125.stdout:1141: read oid 15 snap 109 2021-03-24T21:58:14.308 INFO:tasks.rados.rados.0.smithi125.stdout:1141: expect (ObjNum 384 snap 99 seq_num 384) 2021-03-24T21:58:14.309 INFO:tasks.rados.rados.0.smithi125.stdout:1142: read oid 31 snap -1 2021-03-24T21:58:14.309 INFO:tasks.rados.rados.0.smithi125.stdout:1142: expect (ObjNum 420 snap 108 seq_num 420) 2021-03-24T21:58:14.309 INFO:tasks.rados.rados.0.smithi125.stdout:1143: snap_remove snap 104 2021-03-24T21:58:14.406 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:14.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:14 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:58:14.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:14 smithi125 conmon[19936]: 58:12.790553+0000 mgr.y (mgr.14144) 429 : cluster [DBG] pgmap v646: 168 pgs: 168 active+clean; 127 MiB data, 475 MiB used, 536 GiB / 536 GiB avail; 15 MiB/s rd, 2.7 MiB/s wr, 22 op/s 2021-03-24T21:58:14.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:14 smithi125 conmon[19936]: cluster 2021-03-24T21:58:14.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:14 smithi125 conmon[19936]: 2021-03- 2021-03-24T21:58:14.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:14 smithi125 conmon[19936]: 24T21:58:13.235408+0000 mon.a (mon 2021-03-24T21:58:14.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:14 smithi125 conmon[19936]: .0) 1008 : cluster [DBG] osdmap e323: 8 total, 8 up, 6 in 2021-03-24T21:58:14.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:14 smithi125 conmon[19936]: audit 2021 2021-03-24T21:58:14.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:14 smithi125 conmon[19936]: -03-24T21:58:13.673914+0000 mon.c 2021-03-24T21:58:14.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:14 smithi125 conmon[19936]: (mon.2) 52 : audit [DBG] from='client.? v1:172.21.15.125:0/3039516635' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:58:14.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:14 smithi125 conmon[19936]: cluster 2021-03-24T21:58:14.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:14 smithi125 conmon[19936]: 2021-03-24T21:58:14.198635+0000 2021-03-24T21:58:14.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:14 smithi125 conmon[19936]: osd.3 (osd.3) 75 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:58:14.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:14 smithi125 conmon[19936]: 2021-03-24T21:58:14.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:14 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:58:14.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:14 smithi125 conmon[19936]: 03-24T21:58:14.199106+0000 osd.3 2021-03-24T21:58:14.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:14 smithi125 conmon[19936]: (osd.3) 76 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:58:14.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:14 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:58:14.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:14 smithi125 conmon[24927]: 58:12.790553+0000 mgr.y (mgr.14144) 429 : cluster [DBG] pgmap v646: 168 pgs: 168 active+clean; 127 MiB data, 475 MiB used, 536 GiB / 536 GiB avail; 15 MiB/s rd, 2.7 MiB/s wr, 22 op/s 2021-03-24T21:58:14.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:14 smithi125 conmon[24927]: cluster 2021-03-24T21:58:13. 2021-03-24T21:58:14.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:14 smithi125 conmon[24927]: 235408+0000 mon.a (mon.0) 1008 : cluster [DBG] 2021-03-24T21:58:14.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:14 smithi125 conmon[24927]: osdmap e323: 8 total, 8 up, 6 in 2021-03-24T21:58:14.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:14 smithi125 conmon[24927]: audit 2021-03-24T21:58: 2021-03-24T21:58:14.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:14 smithi125 conmon[24927]: 13.673914+0000 mon.c (mon.2) 52 : 2021-03-24T21:58:14.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:14 smithi125 conmon[24927]: audit [DBG] from='client.? v1:172.21.15.125:0/3039516635' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:58:14.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:14 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:58:14.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:14 smithi125 conmon[24927]: 21:58:14.198635+0000 osd.3 (osd. 2021-03-24T21:58:14.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:14 smithi125 conmon[24927]: 3) 75 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:58:14.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:14 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:58:14.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:14 smithi125 conmon[24927]: -03-24T21:58:14.199106+0000 2021-03-24T21:58:14.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:14 smithi125 conmon[24927]: osd.3 (osd.3) 76 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:58:14.464 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:14.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:14.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:58:14.569 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:14.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:14 smithi142 conmon[20125]: cluster 2021-03-24T21:58:12.790553+0000 mgr.y (mgr.14144) 429 : 2021-03-24T21:58:14.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:14 smithi142 conmon[20125]: cluster [DBG] pgmap v646: 168 pgs: 168 active+clean; 127 MiB data, 475 MiB used, 536 GiB / 536 GiB avail; 15 MiB/s rd, 2.7 MiB/s wr, 22 op/s 2021-03-24T21:58:14.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:14 smithi142 conmon[20125]: cluster 2021-03-24T21:58:13.235408+0000 2021-03-24T21:58:14.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:14 smithi142 conmon[20125]: mon.a (mon.0) 1008 : cluster [DBG] osdmap e323: 8 total, 8 up, 6 in 2021-03-24T21:58:14.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:14 smithi142 conmon[20125]: audit 2021-03-24T21:58: 2021-03-24T21:58:14.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:14 smithi142 conmon[20125]: 13.673914+0000 mon.c (mon.2) 52 : audit 2021-03-24T21:58:14.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:14 smithi142 conmon[20125]: [DBG] from='client.? v1:172.21.15.125:0/3039516635' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:58:14.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:14 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:58:14.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:14 smithi142 conmon[20125]: 21:58:14.198635+0000 osd.3 (osd.3 2021-03-24T21:58:14.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:14 smithi142 conmon[20125]: ) 75 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:58:14.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:14 smithi142 conmon[20125]: cluster 2021-03-24T21 2021-03-24T21:58:14.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:14 smithi142 conmon[20125]: :58:14.199106+0000 osd.3 (osd.3) 76 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:58:14.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:14 smithi142 conmon[20125]: 2021-03-24T21:58:14.622 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:14.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:14.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:58:14.729 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:14.785 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:14.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:14.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:58:14.890 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:58:14.921 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2021-03-24T21:58:14.921 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T21:58:14.941 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:58:14.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:14.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:58:15.047 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:15.103 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:15.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:15.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:58:15.181 INFO:tasks.rados.rados.0.smithi125.stdout:1143: done (7 left) 2021-03-24T21:58:15.181 INFO:tasks.rados.rados.0.smithi125.stdout:1144: delete oid 6 current snap is 112 2021-03-24T21:58:15.181 INFO:tasks.rados.rados.0.smithi125.stdout:1137: expect (ObjNum 380 snap 99 seq_num 380) 2021-03-24T21:58:15.210 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:15.256 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:15.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:15.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:58:15.299 INFO:tasks.rados.rados.0.smithi125.stdout:1136: finishing write tid 2 to smithi12556470-48 2021-03-24T21:58:15.299 INFO:tasks.rados.rados.0.smithi125.stdout:1136: finishing write tid 3 to smithi12556470-48 2021-03-24T21:58:15.299 INFO:tasks.rados.rados.0.smithi125.stdout:1136: finishing write tid 4 to smithi12556470-48 2021-03-24T21:58:15.300 INFO:tasks.rados.rados.0.smithi125.stdout:1138: expect (ObjNum 385 snap 99 seq_num 385) 2021-03-24T21:58:15.320 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd unset noscrub 2021-03-24T21:58:15.361 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:15.422 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:15.426 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:15 smithi142 conmon[20125]: cluster 2021-03-24T21:58:14.249874+0000 mon.a (mon.0) 1009 : cluster [DBG] osdmap e324: 8 total, 8 up, 6 in 2021-03-24T21:58:15.427 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:15 smithi142 conmon[20125]: 2021-03-24T21:58:15.427 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:15 smithi142 conmon[20125]: cluster 2021-03-24T21:58:15.175274+0000 mon.a (mon.0) 1010 : cluster [DBG] osdmap e325: 8 total, 8 up, 6 in 2021-03-24T21:58:15.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:15.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:58:15.491 INFO:tasks.rados.rados.0.smithi125.stdout:1136: finishing write tid 5 to smithi12556470-48 2021-03-24T21:58:15.491 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 48 v 349 (ObjNum 423 snap 112 seq_num 423) dirty exists 2021-03-24T21:58:15.491 INFO:tasks.rados.rados.0.smithi125.stdout:1136: left oid 48 (ObjNum 423 snap 112 seq_num 423) 2021-03-24T21:58:15.492 INFO:tasks.rados.rados.0.smithi125.stdout:1139: finishing write tid 1 to smithi12556470-28 2021-03-24T21:58:15.492 INFO:tasks.rados.rados.0.smithi125.stdout:1139: finishing write tid 2 to smithi12556470-28 2021-03-24T21:58:15.492 INFO:tasks.rados.rados.0.smithi125.stdout:1139: finishing write tid 3 to smithi12556470-28 2021-03-24T21:58:15.492 INFO:tasks.rados.rados.0.smithi125.stdout:1139: finishing write tid 4 to smithi12556470-28 2021-03-24T21:58:15.493 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 28 v 293 (ObjNum 424 snap 112 seq_num 424) dirty exists 2021-03-24T21:58:15.493 INFO:tasks.rados.rados.0.smithi125.stdout:1139: left oid 28 (ObjNum 424 snap 112 seq_num 424) 2021-03-24T21:58:15.493 INFO:tasks.rados.rados.0.smithi125.stdout:1136: done (7 left) 2021-03-24T21:58:15.493 INFO:tasks.rados.rados.0.smithi125.stdout:1137: done (6 left) 2021-03-24T21:58:15.494 INFO:tasks.rados.rados.0.smithi125.stdout:1138: done (5 left) 2021-03-24T21:58:15.494 INFO:tasks.rados.rados.0.smithi125.stdout:1139: done (4 left) 2021-03-24T21:58:15.494 INFO:tasks.rados.rados.0.smithi125.stdout:1144: done (3 left) 2021-03-24T21:58:15.494 INFO:tasks.rados.rados.0.smithi125.stdout:1145: copy_from oid 37 from oid 38 current snap is 112 2021-03-24T21:58:15.495 INFO:tasks.rados.rados.0.smithi125.stdout:1140: finishing write tid 1 to smithi12556470-17 2021-03-24T21:58:15.495 INFO:tasks.rados.rados.0.smithi125.stdout:1140: finishing write tid 2 to smithi12556470-17 2021-03-24T21:58:15.495 INFO:tasks.rados.rados.0.smithi125.stdout:1141: expect (ObjNum 384 snap 99 seq_num 384) 2021-03-24T21:58:15.530 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:58:15.589 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:58:15.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:15.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:58:15.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:15 smithi125 conmon[24927]: cluster 2021-03-24T21:58:15.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:15 smithi125 conmon[24927]: 2021-03-24T21:58: 2021-03-24T21:58:15.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:15 smithi125 conmon[24927]: 14.249874+0000 mon.a (mon.0) 1009 : cluster [DBG] osdmap e324: 8 total, 8 up, 6 in 2021-03-24T21:58:15.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:15 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:58:15.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:15 smithi125 conmon[24927]: 03-24T21:58:15.175274+0000 mon.a (mon.0) 1010 : cluster [DBG] 2021-03-24T21:58:15.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:15 smithi125 conmon[24927]: osdmap e325: 8 total, 8 up, 6 in 2021-03-24T21:58:15.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:15 smithi125 conmon[19936]: cluster 2021-03-24T21:58:15.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:15 smithi125 conmon[19936]: 2021-03-24T21:58:14.249874+0000 mon.a (mon.0) 1009 : cluster [DBG] osdmap e324: 8 total, 8 up, 6 in 2021-03-24T21:58:15.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:15 smithi125 conmon[19936]: cluster 2021-03-24T21:58:15.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:15 smithi125 conmon[19936]: 2021-03-24T21:58:15.175274+0000 mon.a (mon.0) 1010 : cluster [DBG] 2021-03-24T21:58:15.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:15 smithi125 conmon[19936]: osdmap e325: 8 total, 8 up, 6 in 2021-03-24T21:58:15.696 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:15.739 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:15.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:15.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:58:15.765 INFO:tasks.rados.rados.0.smithi125.stdout:1140: finishing write tid 3 to smithi12556470-17 2021-03-24T21:58:15.765 INFO:tasks.rados.rados.0.smithi125.stdout:1142: expect (ObjNum 420 snap 108 seq_num 420) 2021-03-24T21:58:15.786 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:58:15.846 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:15.899 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:15.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:15.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:58:15.955 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.7 dump_ops_in_flight 2021-03-24T21:58:16.002 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:16.021 INFO:tasks.rados.rados.0.smithi125.stdout:1140: finishing write tid 4 to smithi12556470-17 2021-03-24T21:58:16.021 INFO:tasks.rados.rados.0.smithi125.stdout:1140: finishing write tid 5 to smithi12556470-17 2021-03-24T21:58:16.021 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 17 v 327 (ObjNum 425 snap 112 seq_num 425) dirty exists 2021-03-24T21:58:16.021 INFO:tasks.rados.rados.0.smithi125.stdout:1140: left oid 17 (ObjNum 425 snap 112 seq_num 425) 2021-03-24T21:58:16.021 INFO:tasks.rados.rados.0.smithi125.stdout:1145: finishing copy_from to smithi12556470-37 2021-03-24T21:58:16.022 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 37 v 49 (ObjNum 408 snap 106 seq_num 408) dirty exists 2021-03-24T21:58:16.022 INFO:tasks.rados.rados.0.smithi125.stdout:1145: finishing copy_from racing read to smithi12556470-37 2021-03-24T21:58:16.022 INFO:tasks.rados.rados.0.smithi125.stdout:1140: done (3 left) 2021-03-24T21:58:16.022 INFO:tasks.rados.rados.0.smithi125.stdout:1141: done (2 left) 2021-03-24T21:58:16.022 INFO:tasks.rados.rados.0.smithi125.stdout:1142: done (1 left) 2021-03-24T21:58:16.023 INFO:tasks.rados.rados.0.smithi125.stdout:1145: done (0 left) 2021-03-24T21:58:16.023 INFO:tasks.rados.rados.0.smithi125.stdout:1146: read oid 11 snap 107 2021-03-24T21:58:16.023 INFO:tasks.rados.rados.0.smithi125.stdout:1146: expect (ObjNum 353 snap 89 seq_num 353) 2021-03-24T21:58:16.023 INFO:tasks.rados.rados.0.smithi125.stdout:1147: snap_create 2021-03-24T21:58:16.036 INFO:tasks.rados.rados.0.smithi125.stdout:1146: expect (ObjNum 353 snap 89 seq_num 353) 2021-03-24T21:58:16.054 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:16.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:16.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:16.159 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:16.208 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:16.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:16.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:58:16.262 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T21:58:16.262 INFO:tasks.rados.rados.0.smithi125.stdout:1146: done (1 left) 2021-03-24T21:58:16.262 INFO:tasks.rados.rados.0.smithi125.stdout:1147: done (0 left) 2021-03-24T21:58:16.263 INFO:tasks.rados.rados.0.smithi125.stdout:1148: rollback oid 37 current snap is 113 2021-03-24T21:58:16.263 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 37 to 107 2021-03-24T21:58:16.263 INFO:tasks.rados.rados.0.smithi125.stdout:1149: snap_remove snap 112 2021-03-24T21:58:16.316 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:58:16.373 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:58:16.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:16.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:58:16.417 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-24T21:58:16.420 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2021-03-24T21:58:16.480 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:58:16.534 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:16 smithi142 conmon[20125]: cluster 2021-03-24T21:58:14.791433+0000 mgr.y (mgr.14144) 2021-03-24T21:58:16.534 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:16 smithi142 conmon[20125]: 430 : cluster [DBG] pgmap v649: 168 pgs: 168 active+clean; 156 MiB data, 523 MiB used, 536 GiB / 536 GiB avail; 16 MiB/s rd, 6.9 MiB/s wr, 39 op/s 2021-03-24T21:58:16.538 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:58:16.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:16.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:58:16.576 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:16 smithi125 conmon[19936]: cluster 2021-03-24T21:58:16.576 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:16 smithi125 conmon[19936]: 2021-03-24T21:58:16.576 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:16 smithi125 conmon[19936]: 2021-03-24T21:58:14 2021-03-24T21:58:16.577 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:16 smithi125 conmon[19936]: .791433+0000 mgr.y (mgr.14144) 430 : cluster [DBG] pgmap v649: 168 pgs: 168 active+clean; 156 MiB data, 523 MiB used, 536 GiB / 536 GiB avail; 16 MiB/s rd, 6.9 MiB/s wr, 39 op/s 2021-03-24T21:58:16.577 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:16 smithi125 conmon[24927]: cluster 2021-03-24T21:58:16.577 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:16 smithi125 conmon[24927]: 2021-03-24T 2021-03-24T21:58:16.577 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:16 smithi125 conmon[24927]: 21:58:14.791433+0000 mgr.y (mgr.14144) 430 : cluster [DBG] pgmap v649: 168 pgs: 168 active+clean; 156 MiB data, 523 MiB used, 536 GiB / 536 GiB avail; 16 MiB/s rd, 6.9 MiB/s wr, 39 op/s 2021-03-24T21:58:16.646 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:16.702 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:16.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:16.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:58:16.809 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:16.868 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:16.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:16.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:58:16.974 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:17.031 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:17.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:17.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:58:17.136 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:58:17.190 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:58:17.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:17.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:58:17.263 INFO:tasks.rados.rados.0.smithi125.stdout:1149: done (1 left) 2021-03-24T21:58:17.263 INFO:tasks.rados.rados.0.smithi125.stdout:1150: write (excl) oid 23 current snap is 113 2021-03-24T21:58:17.264 INFO:tasks.rados.rados.0.smithi125.stdout:1148: finishing rollback tid 0 to smithi12556470-37 2021-03-24T21:58:17.264 INFO:tasks.rados.rados.0.smithi125.stdout:1148: finishing rollback tid 1 to smithi12556470-37 2021-03-24T21:58:17.264 INFO:tasks.rados.rados.0.smithi125.stdout:1148: finishing rollback tid 2 to smithi12556470-37 2021-03-24T21:58:17.264 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 37 v 205 (ObjNum 150 snap 30 seq_num 150) dirty exists 2021-03-24T21:58:17.264 INFO:tasks.rados.rados.0.smithi125.stdout:1150: seq_num 427 ranges {776589=647091,2070517=430801,3026559=150030} 2021-03-24T21:58:17.272 INFO:tasks.rados.rados.0.smithi125.stdout:1150: writing smithi12556470-23 from 776589 to 1423680 tid 1 2021-03-24T21:58:17.279 INFO:tasks.rados.rados.0.smithi125.stdout:1150: writing smithi12556470-23 from 2070517 to 2501318 tid 2 2021-03-24T21:58:17.283 INFO:tasks.rados.rados.0.smithi125.stdout:1150: writing smithi12556470-23 from 3026559 to 3176589 tid 3 2021-03-24T21:58:17.283 INFO:tasks.rados.rados.0.smithi125.stdout:1148: done (1 left) 2021-03-24T21:58:17.283 INFO:tasks.rados.rados.0.smithi125.stdout:1151: snap_remove snap 106 2021-03-24T21:58:17.296 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:17.344 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:17.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:17.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:58:17.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:17 smithi142 conmon[20125]: cluster 2021-03-24T21:58:16.260422+0000 mon.a (mon.0) 1011 : cluster [DBG] osdmap e326: 8 total, 8 up, 6 in 2021-03-24T21:58:17.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:17 smithi142 conmon[20125]: audit 2021-03-24T21: 2021-03-24T21:58:17.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:17 smithi142 conmon[20125]: 58:16.390772+0000 mgr.y (mgr.14144) 431 : audit [DBG] from='client.15147 v1:172.21.15.125:0/1940852497' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T21:58:17.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:17 smithi142 conmon[20125]: cluster 2021-03-24T21:58:17.261827+0000 mon.a (mon.0) 1012 : cluster [DBG] osdmap e327: 8 total, 8 up, 6 in 2021-03-24T21:58:17.453 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:17.517 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:17.521 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:17 smithi125 conmon[24927]: cluster 2021-03-24T21:58:17.521 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:17 smithi125 conmon[24927]: 2021-03-24T21:58 2021-03-24T21:58:17.522 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:17 smithi125 conmon[24927]: :16.260422+0000 mon.a (mon.0) 1011 : cluster [DBG] osdmap e326: 8 total, 8 up, 6 in 2021-03-24T21:58:17.522 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:17 smithi125 conmon[24927]: audit 2021-03 2021-03-24T21:58:17.522 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:17 smithi125 conmon[24927]: -24T21:58:16.390772+0000 mgr.y (mgr.14144) 431 2021-03-24T21:58:17.522 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:17 smithi125 conmon[24927]: : audit [DBG] from='client.15147 v1:172.21.15.125:0/1940852497' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T21:58:17.522 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:17 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:58:17.523 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:17 smithi125 conmon[24927]: -24T21:58:17.261827+0000 mon.a (mon 2021-03-24T21:58:17.523 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:17 smithi125 conmon[24927]: .0) 1012 : cluster [DBG] osdmap e327: 8 total, 8 up, 6 in 2021-03-24T21:58:17.523 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:17 smithi125 conmon[19936]: cluster 2021-03-24T21:58:16.260422 2021-03-24T21:58:17.523 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:17 smithi125 conmon[19936]: +0000 mon.a (mon.0) 1011 : cluster [DBG] osdmap e326: 8 total, 8 up, 6 in 2021-03-24T21:58:17.524 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:17 smithi125 conmon[19936]: audit 2021-03-24T21:58:16.390772+0000 2021-03-24T21:58:17.524 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:17 smithi125 conmon[19936]: mgr.y (mgr.14144) 431 : audit [DBG] from='client.15147 v1:172.21.15.125:0/1940852497' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T21:58:17.524 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:17 smithi125 conmon[19936]: cluster 2021-03-24T21:58 2021-03-24T21:58:17.524 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:17 smithi125 conmon[19936]: :17.261827+0000 mon.a (mon.0) 1012 : cluster [DBG] osdmap e327: 8 total, 8 up, 6 in 2021-03-24T21:58:17.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:17.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:58:17.626 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:17.687 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:17.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:17.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:58:17.720 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-24T21:58:17.793 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:17.849 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:17.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:17.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:58:17.954 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:18.006 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:18.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:18.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:58:18.111 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:18.164 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:18.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:18.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:58:18.227 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":651,"stamp":"2021-03-24T21:58:16.254199+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":163687095,"num_objects":389,"num_object_clones":32,"num_object_copies":1167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":389,"num_whiteouts":4,"num_read":3100,"num_read_kb":1702669,"num_write":1721,"num_write_kb":453802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":518,"num_bytes_recovered":103062625,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7163,"ondisk_log_size":7163,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":504,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":535384,"kb_used_data":392232,"kb_used_omap":0,"kb_used_meta":143104,"kb_avail":561935528,"statfs":{"total":575970213888,"available":575421980672,"internally_reserved":0,"allocated":401645568,"data_stored":457937117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":146538496},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30909302,"num_objects":20,"num_object_clones":19,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":2,"num_read":81,"num_read_kb":67525,"num_write":67,"num_write_kb":26158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.030573"},"pg_stats":[{"pgid":"7.23","version":"320'265","reported_seq":"1071","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.226512+0000","last_change":"2021-03-24T21:58:06.192958+0000","last_active":"2021-03-24T21:58:12.226512+0000","last_peered":"2021-03-24T21:58:12.226512+0000","last_clean":"2021-03-24T21:58:12.226512+0000","last_became_active":"2021-03-24T21:57:31.166310+0000","last_became_peered":"2021-03-24T21:57:31.166310+0000","last_unstale":"2021-03-24T21:58:12.226512+0000","last_undegraded":"2021-03-24T21:58:12.226512+0000","last_fullsized":"2021-03-24T21:58:12.226512+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":265,"ondisk_log_size":265,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5734523,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":117123,"num_write":55,"num_write_kb":21228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":3418953,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"cd","length":"1"},{"start":"cf","length":"1"}]},{"pgid":"7.22","version":"320'75","reported_seq":"187","reported_epoch":"321","state":"active+clean","last_fresh":"2021-03-24T21:58:11.536392+0000","last_change":"2021-03-24T21:58:06.210118+0000","last_active":"2021-03-24T21:58:11.536392+0000","last_peered":"2021-03-24T21:58:11.536392+0000","last_clean":"2021-03-24T21:58:11.536392+0000","last_became_active":"2021-03-24T21:57:28.176845+0000","last_became_peered":"2021-03-24T21:57:28.176845+0000","last_unstale":"2021-03-24T21:58:11.536392+0000","last_undegraded":"2021-03-24T21:58:11.536392+0000","last_fullsized":"2021-03-24T21:58:11.536392+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":75,"ondisk_log_size":75,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2958098,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":14,"num_read_kb":8520,"num_write":17,"num_write_kb":4979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":5774642,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"cd","length":"1"},{"start":"cf","length":"1"}]},{"pgid":"4.c","version":"69'13","reported_seq":"351","reported_epoch":"321","state":"active+clean","last_fresh":"2021-03-24T21:58:11.537743+0000","last_change":"2021-03-24T21:55:36.065455+0000","last_active":"2021-03-24T21:58:11.537743+0000","last_peered":"2021-03-24T21:58:11.537743+0000","last_clean":"2021-03-24T21:58:11.537743+0000","last_became_active":"2021-03-24T21:54:38.127115+0000","last_became_peered":"2021-03-24T21:54:38.127115+0000","last_unstale":"2021-03-24T21:58:11.537743+0000","last_undegraded":"2021-03-24T21:58:11.537743+0000","last_fullsized":"2021-03-24T21:58:11.537743+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2021-03-24T21:55:36.065267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:36.065267+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":13,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"283","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.226888+0000","last_change":"2021-03-24T21:56:42.413023+0000","last_active":"2021-03-24T21:58:12.226888+0000","last_peered":"2021-03-24T21:58:12.226888+0000","last_clean":"2021-03-24T21:58:12.226888+0000","last_became_active":"2021-03-24T21:51:45.375830+0000","last_became_peered":"2021-03-24T21:51:45.375830+0000","last_unstale":"2021-03-24T21:58:12.226888+0000","last_undegraded":"2021-03-24T21:58:12.226888+0000","last_fullsized":"2021-03-24T21:58:12.226888+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:42.412849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:42.412849+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"65'2","reported_seq":"361","reported_epoch":"320","state":"active+clean","last_fresh":"2021-03-24T21:58:10.186757+0000","last_change":"2021-03-24T21:57:54.241995+0000","last_active":"2021-03-24T21:58:10.186757+0000","last_peered":"2021-03-24T21:58:10.186757+0000","last_clean":"2021-03-24T21:58:10.186757+0000","last_became_active":"2021-03-24T21:51:49.416182+0000","last_became_peered":"2021-03-24T21:51:49.416182+0000","last_unstale":"2021-03-24T21:58:10.186757+0000","last_undegraded":"2021-03-24T21:58:10.186757+0000","last_fullsized":"2021-03-24T21:58:10.186757+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2021-03-24T21:57:54.241842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:57:54.241842+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"290","reported_epoch":"321","state":"active+clean","last_fresh":"2021-03-24T21:58:11.538634+0000","last_change":"2021-03-24T21:57:02.182918+0000","last_active":"2021-03-24T21:58:11.538634+0000","last_peered":"2021-03-24T21:58:11.538634+0000","last_clean":"2021-03-24T21:58:11.538634+0000","last_became_active":"2021-03-24T21:57:02.182551+0000","last_became_peered":"2021-03-24T21:57:02.182551+0000","last_unstale":"2021-03-24T21:58:11.538634+0000","last_undegraded":"2021-03-24T21:58:11.538634+0000","last_fullsized":"2021-03-24T21:58:11.538634+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:08.985694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:08.985694+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"320'126","reported_seq":"885","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.404819+0000","last_change":"2021-03-24T21:58:06.195520+0000","last_active":"2021-03-24T21:58:12.404819+0000","last_peered":"2021-03-24T21:58:12.404819+0000","last_clean":"2021-03-24T21:58:12.404819+0000","last_became_active":"2021-03-24T21:57:26.178185+0000","last_became_peered":"2021-03-24T21:57:26.178185+0000","last_unstale":"2021-03-24T21:58:12.404819+0000","last_undegraded":"2021-03-24T21:58:12.404819+0000","last_fullsized":"2021-03-24T21:58:12.404819+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"218'73","last_scrub_stamp":"2021-03-24T21:56:48.267711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:56:48.267711+0000","log_size":126,"ondisk_log_size":126,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3936433,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":83923,"num_write":33,"num_write_kb":15553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":942482,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"cd","length":"1"},{"start":"cf","length":"1"}]},{"pgid":"4.d","version":"69'21","reported_seq":"352","reported_epoch":"321","state":"active+clean","last_fresh":"2021-03-24T21:58:11.537466+0000","last_change":"2021-03-24T21:57:18.801773+0000","last_active":"2021-03-24T21:58:11.537466+0000","last_peered":"2021-03-24T21:58:11.537466+0000","last_clean":"2021-03-24T21:58:11.537466+0000","last_became_active":"2021-03-24T21:54:38.144020+0000","last_became_peered":"2021-03-24T21:54:38.144020+0000","last_unstale":"2021-03-24T21:58:11.537466+0000","last_undegraded":"2021-03-24T21:58:11.537466+0000","last_fullsized":"2021-03-24T21:58:11.537466+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'21","last_scrub_stamp":"2021-03-24T21:57:18.801567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:57:18.801567+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":23,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"250","reported_epoch":"321","state":"active+clean","last_fresh":"2021-03-24T21:58:11.536698+0000","last_change":"2021-03-24T21:57:50.522441+0000","last_active":"2021-03-24T21:58:11.536698+0000","last_peered":"2021-03-24T21:58:11.536698+0000","last_clean":"2021-03-24T21:58:11.536698+0000","last_became_active":"2021-03-24T21:54:38.127136+0000","last_became_peered":"2021-03-24T21:54:38.127136+0000","last_unstale":"2021-03-24T21:58:11.536698+0000","last_undegraded":"2021-03-24T21:58:11.536698+0000","last_fullsized":"2021-03-24T21:58:11.536698+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:50.522344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:55:01.169461+0000","last_clean_scrub_stamp":"2021-03-24T21:57:50.522344+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"239","reported_epoch":"321","state":"active+clean","last_fresh":"2021-03-24T21:58:11.537484+0000","last_change":"2021-03-24T21:56:44.230926+0000","last_active":"2021-03-24T21:58:11.537484+0000","last_peered":"2021-03-24T21:58:11.537484+0000","last_clean":"2021-03-24T21:58:11.537484+0000","last_became_active":"2021-03-24T21:56:44.230543+0000","last_became_peered":"2021-03-24T21:56:44.230543+0000","last_unstale":"2021-03-24T21:58:11.537484+0000","last_undegraded":"2021-03-24T21:58:11.537484+0000","last_fullsized":"2021-03-24T21:58:11.537484+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:07.977138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:07.977138+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"82","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.228014+0000","last_change":"2021-03-24T21:57:02.182962+0000","last_active":"2021-03-24T21:58:12.228014+0000","last_peered":"2021-03-24T21:58:12.228014+0000","last_clean":"2021-03-24T21:58:12.228014+0000","last_became_active":"2021-03-24T21:57:02.181899+0000","last_became_peered":"2021-03-24T21:57:02.181899+0000","last_unstale":"2021-03-24T21:58:12.228014+0000","last_undegraded":"2021-03-24T21:58:12.228014+0000","last_fullsized":"2021-03-24T21:58:12.228014+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:49.970073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:49.970073+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"318'49","reported_seq":"762","reported_epoch":"320","state":"active+clean","last_fresh":"2021-03-24T21:58:10.189403+0000","last_change":"2021-03-24T21:58:06.198874+0000","last_active":"2021-03-24T21:58:10.189403+0000","last_peered":"2021-03-24T21:58:10.189403+0000","last_clean":"2021-03-24T21:58:10.189403+0000","last_became_active":"2021-03-24T21:57:26.167896+0000","last_became_peered":"2021-03-24T21:57:26.167896+0000","last_unstale":"2021-03-24T21:58:10.189403+0000","last_undegraded":"2021-03-24T21:58:10.189403+0000","last_fullsized":"2021-03-24T21:58:10.189403+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'45","last_scrub_stamp":"2021-03-24T21:57:27.105464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:57:27.105464+0000","log_size":49,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2857256,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":1214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"cd","length":"1"},{"start":"cf","length":"1"}]},{"pgid":"4.e","version":"69'15","reported_seq":"366","reported_epoch":"321","state":"active+clean","last_fresh":"2021-03-24T21:58:11.538165+0000","last_change":"2021-03-24T21:56:40.062483+0000","last_active":"2021-03-24T21:58:11.538165+0000","last_peered":"2021-03-24T21:58:11.538165+0000","last_clean":"2021-03-24T21:58:11.538165+0000","last_became_active":"2021-03-24T21:54:38.127461+0000","last_became_peered":"2021-03-24T21:54:38.127461+0000","last_unstale":"2021-03-24T21:58:11.538165+0000","last_undegraded":"2021-03-24T21:58:11.538165+0000","last_fullsized":"2021-03-24T21:58:11.538165+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:56:40.062304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:40.062304+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1476,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"283","reported_epoch":"321","state":"active+clean","last_fresh":"2021-03-24T21:58:11.536403+0000","last_change":"2021-03-24T21:57:55.592974+0000","last_active":"2021-03-24T21:58:11.536403+0000","last_peered":"2021-03-24T21:58:11.536403+0000","last_clean":"2021-03-24T21:58:11.536403+0000","last_became_active":"2021-03-24T21:51:45.579513+0000","last_became_peered":"2021-03-24T21:51:45.579513+0000","last_unstale":"2021-03-24T21:58:11.536403+0000","last_undegraded":"2021-03-24T21:58:11.536403+0000","last_fullsized":"2021-03-24T21:58:11.536403+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:55.592795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:55.592795+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"256","reported_epoch":"321","state":"active+clean","last_fresh":"2021-03-24T21:58:11.539300+0000","last_change":"2021-03-24T21:56:43.971055+0000","last_active":"2021-03-24T21:58:11.539300+0000","last_peered":"2021-03-24T21:58:11.539300+0000","last_clean":"2021-03-24T21:58:11.539300+0000","last_became_active":"2021-03-24T21:55:10.102113+0000","last_became_peered":"2021-03-24T21:55:10.102113+0000","last_unstale":"2021-03-24T21:58:11.539300+0000","last_undegraded":"2021-03-24T21:58:11.539300+0000","last_fullsized":"2021-03-24T21:58:11.539300+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:43.970879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:43.970879+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"281","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.227487+0000","last_change":"2021-03-24T21:55:59.472536+0000","last_active":"2021-03-24T21:58:12.227487+0000","last_peered":"2021-03-24T21:58:12.227487+0000","last_clean":"2021-03-24T21:58:12.227487+0000","last_became_active":"2021-03-24T21:54:24.196587+0000","last_became_peered":"2021-03-24T21:54:24.196587+0000","last_unstale":"2021-03-24T21:58:12.227487+0000","last_undegraded":"2021-03-24T21:58:12.227487+0000","last_fullsized":"2021-03-24T21:58:12.227487+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:59.472393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:55:59.472393+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"320'302","reported_seq":"1033","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.404009+0000","last_change":"2021-03-24T21:58:06.197229+0000","last_active":"2021-03-24T21:58:12.404009+0000","last_peered":"2021-03-24T21:58:12.404009+0000","last_clean":"2021-03-24T21:58:12.404009+0000","last_became_active":"2021-03-24T21:57:26.189692+0000","last_became_peered":"2021-03-24T21:57:26.189692+0000","last_unstale":"2021-03-24T21:58:12.404009+0000","last_undegraded":"2021-03-24T21:58:12.404009+0000","last_fullsized":"2021-03-24T21:58:12.404009+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'169","last_scrub_stamp":"2021-03-24T21:56:47.285648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:56:47.285648+0000","log_size":302,"ondisk_log_size":302,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4817549,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":74,"num_read_kb":65137,"num_write":80,"num_write_kb":28172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2546506,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"cd","length":"1"},{"start":"cf","length":"1"}]},{"pgid":"4.f","version":"274'26","reported_seq":"300","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.228621+0000","last_change":"2021-03-24T21:57:24.499178+0000","last_active":"2021-03-24T21:58:12.228621+0000","last_peered":"2021-03-24T21:58:12.228621+0000","last_clean":"2021-03-24T21:58:12.228621+0000","last_became_active":"2021-03-24T21:54:38.130536+0000","last_became_peered":"2021-03-24T21:54:38.130536+0000","last_unstale":"2021-03-24T21:58:12.228621+0000","last_undegraded":"2021-03-24T21:58:12.228621+0000","last_fullsized":"2021-03-24T21:58:12.228621+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'26","last_scrub_stamp":"2021-03-24T21:57:24.499090+0000","last_deep_scrub":"274'26","last_deep_scrub_stamp":"2021-03-24T21:57:24.499090+0000","last_clean_scrub_stamp":"2021-03-24T21:57:24.499090+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":18,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":422,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":"287","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.226179+0000","last_change":"2021-03-24T21:56:39.439282+0000","last_active":"2021-03-24T21:58:12.226179+0000","last_peered":"2021-03-24T21:58:12.226179+0000","last_clean":"2021-03-24T21:58:12.226179+0000","last_became_active":"2021-03-24T21:54:38.134768+0000","last_became_peered":"2021-03-24T21:54:38.134768+0000","last_unstale":"2021-03-24T21:58:12.226179+0000","last_undegraded":"2021-03-24T21:58:12.226179+0000","last_fullsized":"2021-03-24T21:58:12.226179+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:39.439170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:39.439170+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"65'2","reported_seq":"352","reported_epoch":"321","state":"active+clean","last_fresh":"2021-03-24T21:58:11.539492+0000","last_change":"2021-03-24T21:55:59.032346+0000","last_active":"2021-03-24T21:58:11.539492+0000","last_peered":"2021-03-24T21:58:11.539492+0000","last_clean":"2021-03-24T21:58:11.539492+0000","last_became_active":"2021-03-24T21:51:49.414380+0000","last_became_peered":"2021-03-24T21:51:49.414380+0000","last_unstale":"2021-03-24T21:58:11.539492+0000","last_undegraded":"2021-03-24T21:58:11.539492+0000","last_fullsized":"2021-03-24T21:58:11.539492+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2021-03-24T21:55:59.032170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:55:59.032170+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"289","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.409441+0000","last_change":"2021-03-24T21:57:23.176879+0000","last_active":"2021-03-24T21:58:12.409441+0000","last_peered":"2021-03-24T21:58:12.409441+0000","last_clean":"2021-03-24T21:58:12.409441+0000","last_became_active":"2021-03-24T21:54:38.129469+0000","last_became_peered":"2021-03-24T21:54:38.129469+0000","last_unstale":"2021-03-24T21:58:12.409441+0000","last_undegraded":"2021-03-24T21:58:12.409441+0000","last_fullsized":"2021-03-24T21:58:12.409441+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:23.176750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:57:23.176750+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"320'316","reported_seq":"1046","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.227111+0000","last_change":"2021-03-24T21:58:06.189915+0000","last_active":"2021-03-24T21:58:12.227111+0000","last_peered":"2021-03-24T21:58:12.227111+0000","last_clean":"2021-03-24T21:58:12.227111+0000","last_became_active":"2021-03-24T21:57:26.174174+0000","last_became_peered":"2021-03-24T21:57:26.174174+0000","last_unstale":"2021-03-24T21:58:12.227111+0000","last_undegraded":"2021-03-24T21:58:12.227111+0000","last_fullsized":"2021-03-24T21:58:12.227111+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":316,"ondisk_log_size":316,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6554928,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":113518,"num_write":85,"num_write_kb":31481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"cd","length":"1"},{"start":"cf","length":"1"}]},{"pgid":"5.1","version":"0'0","reported_seq":"279","reported_epoch":"321","state":"active+clean","last_fresh":"2021-03-24T21:58:11.536200+0000","last_change":"2021-03-24T21:57:59.462088+0000","last_active":"2021-03-24T21:58:11.536200+0000","last_peered":"2021-03-24T21:58:11.536200+0000","last_clean":"2021-03-24T21:58:11.536200+0000","last_became_active":"2021-03-24T21:51:49.408761+0000","last_became_peered":"2021-03-24T21:51:49.408761+0000","last_unstale":"2021-03-24T21:58:11.536200+0000","last_undegraded":"2021-03-24T21:58:11.536200+0000","last_fullsized":"2021-03-24T21:58:11.536200+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:59.461865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:25.220436+0000","last_clean_scrub_stamp":"2021-03-24T21:57:59.461865+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"69'22","reported_seq":"316","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.226831+0000","last_change":"2021-03-24T21:55:28.629625+0000","last_active":"2021-03-24T21:58:12.226831+0000","last_peered":"2021-03-24T21:58:12.226831+0000","last_clean":"2021-03-24T21:58:12.226831+0000","last_became_active":"2021-03-24T21:51:47.691459+0000","last_became_peered":"2021-03-24T21:51:47.691459+0000","last_unstale":"2021-03-24T21:58:12.226831+0000","last_undegraded":"2021-03-24T21:58:12.226831+0000","last_fullsized":"2021-03-24T21:58:12.226831+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'22","last_scrub_stamp":"2021-03-24T21:55:28.629443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:28.629443+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":46,"num_read_kb":26,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"283","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.228461+0000","last_change":"2021-03-24T21:57:54.381907+0000","last_active":"2021-03-24T21:58:12.228461+0000","last_peered":"2021-03-24T21:58:12.228461+0000","last_clean":"2021-03-24T21:58:12.228461+0000","last_became_active":"2021-03-24T21:51:45.378704+0000","last_became_peered":"2021-03-24T21:51:45.378704+0000","last_unstale":"2021-03-24T21:58:12.228461+0000","last_undegraded":"2021-03-24T21:58:12.228461+0000","last_fullsized":"2021-03-24T21:58:12.228461+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:54.381668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:54.381668+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"272","reported_epoch":"321","state":"active+clean","last_fresh":"2021-03-24T21:58:11.536721+0000","last_change":"2021-03-24T21:56:35.045715+0000","last_active":"2021-03-24T21:58:11.536721+0000","last_peered":"2021-03-24T21:58:11.536721+0000","last_clean":"2021-03-24T21:58:11.536721+0000","last_became_active":"2021-03-24T21:51:51.756114+0000","last_became_peered":"2021-03-24T21:51:51.756114+0000","last_unstale":"2021-03-24T21:58:11.536721+0000","last_undegraded":"2021-03-24T21:58:11.536721+0000","last_fullsized":"2021-03-24T21:58:11.536721+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:35.045534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:35.045534+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"320'279","reported_seq":"1056","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.227368+0000","last_change":"2021-03-24T21:58:06.193804+0000","last_active":"2021-03-24T21:58:12.227368+0000","last_peered":"2021-03-24T21:58:12.227368+0000","last_clean":"2021-03-24T21:58:12.227368+0000","last_became_active":"2021-03-24T21:57:26.169302+0000","last_became_peered":"2021-03-24T21:57:26.169302+0000","last_unstale":"2021-03-24T21:58:12.227368+0000","last_undegraded":"2021-03-24T21:58:12.227368+0000","last_fullsized":"2021-03-24T21:58:12.227368+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":279,"ondisk_log_size":279,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4243982,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":105,"num_read_kb":123787,"num_write":52,"num_write_kb":21114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"cd","length":"1"},{"start":"cf","length":"1"}]},{"pgid":"5.0","version":"109'3","reported_seq":"371","reported_epoch":"323","state":"active+clean","last_fresh":"2021-03-24T21:58:13.440460+0000","last_change":"2021-03-24T21:57:25.463136+0000","last_active":"2021-03-24T21:58:13.440460+0000","last_peered":"2021-03-24T21:58:13.440460+0000","last_clean":"2021-03-24T21:58:13.440460+0000","last_became_active":"2021-03-24T21:54:38.135726+0000","last_became_peered":"2021-03-24T21:54:38.135726+0000","last_unstale":"2021-03-24T21:58:13.440460+0000","last_undegraded":"2021-03-24T21:58:13.440460+0000","last_fullsized":"2021-03-24T21:58:13.440460+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'3","last_scrub_stamp":"2021-03-24T21:57:25.462952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:57:25.462952+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"69'20","reported_seq":"351","reported_epoch":"321","state":"active+clean","last_fresh":"2021-03-24T21:58:11.536399+0000","last_change":"2021-03-24T21:57:21.706231+0000","last_active":"2021-03-24T21:58:11.536399+0000","last_peered":"2021-03-24T21:58:11.536399+0000","last_clean":"2021-03-24T21:58:11.536399+0000","last_became_active":"2021-03-24T21:54:24.189182+0000","last_became_peered":"2021-03-24T21:54:24.189182+0000","last_unstale":"2021-03-24T21:58:11.536399+0000","last_undegraded":"2021-03-24T21:58:11.536399+0000","last_fullsized":"2021-03-24T21:58:11.536399+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'20","last_scrub_stamp":"2021-03-24T21:57:21.706127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:57:21.706127+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":39,"num_read_kb":20,"num_write":20,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":940,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"296","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.401269+0000","last_change":"2021-03-24T21:56:44.229920+0000","last_active":"2021-03-24T21:58:12.401269+0000","last_peered":"2021-03-24T21:58:12.401269+0000","last_clean":"2021-03-24T21:58:12.401269+0000","last_became_active":"2021-03-24T21:56:44.229469+0000","last_became_peered":"2021-03-24T21:56:44.229469+0000","last_unstale":"2021-03-24T21:58:12.401269+0000","last_undegraded":"2021-03-24T21:58:12.401269+0000","last_fullsized":"2021-03-24T21:58:12.401269+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:28.165408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:28.165408+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"297","reported_epoch":"323","state":"active+clean","last_fresh":"2021-03-24T21:58:13.251186+0000","last_change":"2021-03-24T21:56:40.011790+0000","last_active":"2021-03-24T21:58:13.251186+0000","last_peered":"2021-03-24T21:58:13.251186+0000","last_clean":"2021-03-24T21:58:13.251186+0000","last_became_active":"2021-03-24T21:54:38.110246+0000","last_became_peered":"2021-03-24T21:54:38.110246+0000","last_unstale":"2021-03-24T21:58:13.251186+0000","last_undegraded":"2021-03-24T21:58:13.251186+0000","last_fullsized":"2021-03-24T21:58:13.251186+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:40.011633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:40.011633+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"272'64","reported_seq":"787","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.225842+0000","last_change":"2021-03-24T21:58:06.187839+0000","last_active":"2021-03-24T21:58:12.225842+0000","last_peered":"2021-03-24T21:58:12.225842+0000","last_clean":"2021-03-24T21:58:12.225842+0000","last_became_active":"2021-03-24T21:57:26.174158+0000","last_became_peered":"2021-03-24T21:57:26.174158+0000","last_unstale":"2021-03-24T21:58:12.225842+0000","last_undegraded":"2021-03-24T21:58:12.225842+0000","last_fullsized":"2021-03-24T21:58:12.225842+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":64,"ondisk_log_size":64,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1922978,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":8518,"num_write":11,"num_write_kb":3706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":216931,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"cd","length":"1"},{"start":"cf","length":"1"}]},{"pgid":"5.3","version":"102'4","reported_seq":"367","reported_epoch":"323","state":"active+clean","last_fresh":"2021-03-24T21:58:13.246426+0000","last_change":"2021-03-24T21:56:01.267040+0000","last_active":"2021-03-24T21:58:13.246426+0000","last_peered":"2021-03-24T21:58:13.246426+0000","last_clean":"2021-03-24T21:58:13.246426+0000","last_became_active":"2021-03-24T21:54:24.203310+0000","last_became_peered":"2021-03-24T21:54:24.203310+0000","last_unstale":"2021-03-24T21:58:13.246426+0000","last_undegraded":"2021-03-24T21:58:13.246426+0000","last_fullsized":"2021-03-24T21:58:13.246426+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'4","last_scrub_stamp":"2021-03-24T21:56:01.266936+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-24T21:53:26.584678+0000","last_clean_scrub_stamp":"2021-03-24T21:56:01.266936+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"69'16","reported_seq":"300","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.407652+0000","last_change":"2021-03-24T21:56:41.235280+0000","last_active":"2021-03-24T21:58:12.407652+0000","last_peered":"2021-03-24T21:58:12.407652+0000","last_clean":"2021-03-24T21:58:12.407652+0000","last_became_active":"2021-03-24T21:54:38.128745+0000","last_became_peered":"2021-03-24T21:54:38.128745+0000","last_unstale":"2021-03-24T21:58:12.407652+0000","last_undegraded":"2021-03-24T21:58:12.407652+0000","last_fullsized":"2021-03-24T21:58:12.407652+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T21:56:41.235143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:41.235143+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":32,"num_read_kb":15,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":940,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"291","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.409742+0000","last_change":"2021-03-24T21:57:53.013298+0000","last_active":"2021-03-24T21:58:12.409742+0000","last_peered":"2021-03-24T21:58:12.409742+0000","last_clean":"2021-03-24T21:58:12.409742+0000","last_became_active":"2021-03-24T21:51:45.583885+0000","last_became_peered":"2021-03-24T21:51:45.583885+0000","last_unstale":"2021-03-24T21:58:12.409742+0000","last_undegraded":"2021-03-24T21:58:12.409742+0000","last_fullsized":"2021-03-24T21:58:12.409742+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:53.013129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:53.013129+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"272","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.401827+0000","last_change":"2021-03-24T21:55:39.261965+0000","last_active":"2021-03-24T21:58:12.401827+0000","last_peered":"2021-03-24T21:58:12.401827+0000","last_clean":"2021-03-24T21:58:12.401827+0000","last_became_active":"2021-03-24T21:51:51.759627+0000","last_became_peered":"2021-03-24T21:51:51.759627+0000","last_unstale":"2021-03-24T21:58:12.401827+0000","last_undegraded":"2021-03-24T21:58:12.401827+0000","last_fullsized":"2021-03-24T21:58:12.401827+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:39.261698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:55:39.261698+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"307'277","reported_seq":"1110","reported_epoch":"321","state":"active+clean","last_fresh":"2021-03-24T21:58:11.536130+0000","last_change":"2021-03-24T21:58:06.208484+0000","last_active":"2021-03-24T21:58:11.536130+0000","last_peered":"2021-03-24T21:58:11.536130+0000","last_clean":"2021-03-24T21:58:11.536130+0000","last_became_active":"2021-03-24T21:57:26.191103+0000","last_became_peered":"2021-03-24T21:57:26.191103+0000","last_unstale":"2021-03-24T21:58:11.536130+0000","last_undegraded":"2021-03-24T21:58:11.536130+0000","last_fullsized":"2021-03-24T21:58:11.536130+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":277,"ondisk_log_size":277,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5104582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":74,"num_read_kb":70782,"num_write":41,"num_write_kb":21431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":5112615,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"cd","length":"1"},{"start":"cf","length":"1"}]},{"pgid":"5.2","version":"0'0","reported_seq":"237","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.403139+0000","last_change":"2021-03-24T21:55:58.253884+0000","last_active":"2021-03-24T21:58:12.403139+0000","last_peered":"2021-03-24T21:58:12.403139+0000","last_clean":"2021-03-24T21:58:12.403139+0000","last_became_active":"2021-03-24T21:54:24.198895+0000","last_became_peered":"2021-03-24T21:54:24.198895+0000","last_unstale":"2021-03-24T21:58:12.403139+0000","last_undegraded":"2021-03-24T21:58:12.403139+0000","last_fullsized":"2021-03-24T21:58:12.403139+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:58.253777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:55:58.253777+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"69'21","reported_seq":"321","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.404736+0000","last_change":"2021-03-24T21:56:39.335724+0000","last_active":"2021-03-24T21:58:12.404736+0000","last_peered":"2021-03-24T21:58:12.404736+0000","last_clean":"2021-03-24T21:58:12.404736+0000","last_became_active":"2021-03-24T21:51:47.686486+0000","last_became_peered":"2021-03-24T21:51:47.686486+0000","last_unstale":"2021-03-24T21:58:12.404736+0000","last_undegraded":"2021-03-24T21:58:12.404736+0000","last_fullsized":"2021-03-24T21:58:12.404736+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'21","last_scrub_stamp":"2021-03-24T21:56:39.335613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:39.335613+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":45,"num_read_kb":27,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"85","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.406853+0000","last_change":"2021-03-24T21:57:02.189207+0000","last_active":"2021-03-24T21:58:12.406853+0000","last_peered":"2021-03-24T21:58:12.406853+0000","last_clean":"2021-03-24T21:58:12.406853+0000","last_became_active":"2021-03-24T21:57:02.188835+0000","last_became_peered":"2021-03-24T21:57:02.188835+0000","last_unstale":"2021-03-24T21:58:12.406853+0000","last_undegraded":"2021-03-24T21:58:12.406853+0000","last_fullsized":"2021-03-24T21:58:12.406853+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:38.144956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:38.144956+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"135","reported_epoch":"323","state":"active+clean","last_fresh":"2021-03-24T21:58:13.249974+0000","last_change":"2021-03-24T21:57:02.189486+0000","last_active":"2021-03-24T21:58:13.249974+0000","last_peered":"2021-03-24T21:58:13.249974+0000","last_clean":"2021-03-24T21:58:13.249974+0000","last_became_active":"2021-03-24T21:57:02.189009+0000","last_became_peered":"2021-03-24T21:57:02.189009+0000","last_unstale":"2021-03-24T21:58:13.249974+0000","last_undegraded":"2021-03-24T21:58:13.249974+0000","last_fullsized":"2021-03-24T21:58:13.249974+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:01.124573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:01.124573+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.0","version":"175'22","reported_seq":"750","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.404814+0000","last_change":"2021-03-24T21:58:06.188237+0000","last_active":"2021-03-24T21:58:12.404814+0000","last_peered":"2021-03-24T21:58:12.404814+0000","last_clean":"2021-03-24T21:58:12.404814+0000","last_became_active":"2021-03-24T21:57:26.169706+0000","last_became_peered":"2021-03-24T21:57:26.169706+0000","last_unstale":"2021-03-24T21:58:12.404814+0000","last_undegraded":"2021-03-24T21:58:12.404814+0000","last_fullsized":"2021-03-24T21:58:12.404814+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'22","last_scrub_stamp":"2021-03-24T21:56:40.276617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:56:40.276617+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"cd","length":"1"},{"start":"cf","length":"1"}]},{"pgid":"5.7","version":"0'0","reported_seq":"291","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.408521+0000","last_change":"2021-03-24T21:56:43.194373+0000","last_active":"2021-03-24T21:58:12.408521+0000","last_peered":"2021-03-24T21:58:12.408521+0000","last_clean":"2021-03-24T21:58:12.408521+0000","last_became_active":"2021-03-24T21:54:38.128708+0000","last_became_peered":"2021-03-24T21:54:38.128708+0000","last_unstale":"2021-03-24T21:58:12.408521+0000","last_undegraded":"2021-03-24T21:58:12.408521+0000","last_fullsized":"2021-03-24T21:58:12.408521+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:43.194160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:43.194160+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"69'16","reported_seq":"369","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.403364+0000","last_change":"2021-03-24T21:57:26.257787+0000","last_active":"2021-03-24T21:58:12.403364+0000","last_peered":"2021-03-24T21:58:12.403364+0000","last_clean":"2021-03-24T21:58:12.403364+0000","last_became_active":"2021-03-24T21:57:02.185489+0000","last_became_peered":"2021-03-24T21:57:02.185489+0000","last_unstale":"2021-03-24T21:58:12.403364+0000","last_undegraded":"2021-03-24T21:58:12.403364+0000","last_fullsized":"2021-03-24T21:58:12.403364+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T21:57:26.257686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:57:26.257686+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":16,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1260,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"66'5","reported_seq":"692","reported_epoch":"323","state":"active+clean","last_fresh":"2021-03-24T21:58:13.550635+0000","last_change":"2021-03-24T21:56:37.973047+0000","last_active":"2021-03-24T21:58:13.550635+0000","last_peered":"2021-03-24T21:58:13.550635+0000","last_clean":"2021-03-24T21:58:13.550635+0000","last_became_active":"2021-03-24T21:54:38.125434+0000","last_became_peered":"2021-03-24T21:54:38.125434+0000","last_unstale":"2021-03-24T21:58:13.550635+0000","last_undegraded":"2021-03-24T21:58:13.550635+0000","last_fullsized":"2021-03-24T21:58:13.550635+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2021-03-24T21:56:37.972874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:39.286101+0000","last_clean_scrub_stamp":"2021-03-24T21:56:37.972874+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":386,"num_read_kb":381,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"283","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.401887+0000","last_change":"2021-03-24T21:56:38.304954+0000","last_active":"2021-03-24T21:58:12.401887+0000","last_peered":"2021-03-24T21:58:12.401887+0000","last_clean":"2021-03-24T21:58:12.401887+0000","last_became_active":"2021-03-24T21:51:45.376687+0000","last_became_peered":"2021-03-24T21:51:45.376687+0000","last_unstale":"2021-03-24T21:58:12.401887+0000","last_undegraded":"2021-03-24T21:58:12.401887+0000","last_fullsized":"2021-03-24T21:58:12.401887+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:38.304792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:38.304792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"76","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.402628+0000","last_change":"2021-03-24T21:57:06.153154+0000","last_active":"2021-03-24T21:58:12.402628+0000","last_peered":"2021-03-24T21:58:12.402628+0000","last_clean":"2021-03-24T21:58:12.402628+0000","last_became_active":"2021-03-24T21:57:06.152772+0000","last_became_peered":"2021-03-24T21:57:06.152772+0000","last_unstale":"2021-03-24T21:58:12.402628+0000","last_undegraded":"2021-03-24T21:58:12.402628+0000","last_fullsized":"2021-03-24T21:58:12.402628+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:41.503758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:55:41.503758+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.5","version":"195'69","reported_seq":"780","reported_epoch":"323","state":"active+clean","last_fresh":"2021-03-24T21:58:13.251478+0000","last_change":"2021-03-24T21:58:06.202784+0000","last_active":"2021-03-24T21:58:13.251478+0000","last_peered":"2021-03-24T21:58:13.251478+0000","last_clean":"2021-03-24T21:58:13.251478+0000","last_became_active":"2021-03-24T21:56:08.152168+0000","last_became_peered":"2021-03-24T21:56:08.152168+0000","last_unstale":"2021-03-24T21:58:13.251478+0000","last_undegraded":"2021-03-24T21:58:13.251478+0000","last_fullsized":"2021-03-24T21:58:13.251478+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'69","last_scrub_stamp":"2021-03-24T21:56:49.886058+0000","last_deep_scrub":"195'69","last_deep_scrub_stamp":"2021-03-24T21:56:49.886058+0000","last_clean_scrub_stamp":"2021-03-24T21:56:49.886058+0000","log_size":69,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"cd","length":"1"},{"start":"cf","length":"1"}]},{"pgid":"5.4","version":"0'0","reported_seq":"281","reported_epoch":"323","state":"active+clean","last_fresh":"2021-03-24T21:58:13.251505+0000","last_change":"2021-03-24T21:57:18.200083+0000","last_active":"2021-03-24T21:58:13.251505+0000","last_peered":"2021-03-24T21:58:13.251505+0000","last_clean":"2021-03-24T21:58:13.251505+0000","last_became_active":"2021-03-24T21:51:49.413251+0000","last_became_peered":"2021-03-24T21:51:49.413251+0000","last_unstale":"2021-03-24T21:58:13.251505+0000","last_undegraded":"2021-03-24T21:58:13.251505+0000","last_fullsized":"2021-03-24T21:58:13.251505+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:18.199919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:57:18.199919+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"69'15","reported_seq":"293","reported_epoch":"321","state":"active+clean","last_fresh":"2021-03-24T21:58:11.537585+0000","last_change":"2021-03-24T21:55:30.166245+0000","last_active":"2021-03-24T21:58:11.537585+0000","last_peered":"2021-03-24T21:58:11.537585+0000","last_clean":"2021-03-24T21:58:11.537585+0000","last_became_active":"2021-03-24T21:54:24.194711+0000","last_became_peered":"2021-03-24T21:54:24.194711+0000","last_unstale":"2021-03-24T21:58:11.537585+0000","last_undegraded":"2021-03-24T21:58:11.537585+0000","last_fullsized":"2021-03-24T21:58:11.537585+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:55:30.166107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:30.166107+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":26,"num_read_kb":14,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":466,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"287","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.226161+0000","last_change":"2021-03-24T21:55:33.534038+0000","last_active":"2021-03-24T21:58:12.226161+0000","last_peered":"2021-03-24T21:58:12.226161+0000","last_clean":"2021-03-24T21:58:12.226161+0000","last_became_active":"2021-03-24T21:54:24.199349+0000","last_became_peered":"2021-03-24T21:54:24.199349+0000","last_unstale":"2021-03-24T21:58:12.226161+0000","last_undegraded":"2021-03-24T21:58:12.226161+0000","last_fullsized":"2021-03-24T21:58:12.226161+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:33.533840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:33.533840+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"308","reported_epoch":"323","state":"active+clean","last_fresh":"2021-03-24T21:58:13.249604+0000","last_change":"2021-03-24T21:56:05.075868+0000","last_active":"2021-03-24T21:58:13.249604+0000","last_peered":"2021-03-24T21:58:13.249604+0000","last_clean":"2021-03-24T21:58:13.249604+0000","last_became_active":"2021-03-24T21:54:38.110327+0000","last_became_peered":"2021-03-24T21:54:38.110327+0000","last_unstale":"2021-03-24T21:58:13.249604+0000","last_undegraded":"2021-03-24T21:58:13.249604+0000","last_fullsized":"2021-03-24T21:58:13.249604+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:05.075677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:49:51.529051+0000","last_clean_scrub_stamp":"2021-03-24T21:56:05.075677+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"285","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.408082+0000","last_change":"2021-03-24T21:57:55.072899+0000","last_active":"2021-03-24T21:58:12.408082+0000","last_peered":"2021-03-24T21:58:12.408082+0000","last_clean":"2021-03-24T21:58:12.408082+0000","last_became_active":"2021-03-24T21:51:51.765236+0000","last_became_peered":"2021-03-24T21:51:51.765236+0000","last_unstale":"2021-03-24T21:58:12.408082+0000","last_undegraded":"2021-03-24T21:58:12.408082+0000","last_fullsized":"2021-03-24T21:58:12.408082+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:55.072694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:57:55.072694+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"307'170","reported_seq":"964","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.403827+0000","last_change":"2021-03-24T21:58:06.195623+0000","last_active":"2021-03-24T21:58:12.403827+0000","last_peered":"2021-03-24T21:58:12.403827+0000","last_clean":"2021-03-24T21:58:12.403827+0000","last_became_active":"2021-03-24T21:57:09.179214+0000","last_became_peered":"2021-03-24T21:57:09.179214+0000","last_unstale":"2021-03-24T21:58:12.403827+0000","last_undegraded":"2021-03-24T21:58:12.403827+0000","last_fullsized":"2021-03-24T21:58:12.403827+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":170,"ondisk_log_size":170,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5781624,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":70,"num_read_kb":83641,"num_write":46,"num_write_kb":21974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":8481953,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"cd","length":"1"},{"start":"cf","length":"1"}]},{"pgid":"5.6","version":"0'0","reported_seq":"87","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.409577+0000","last_change":"2021-03-24T21:57:20.215284+0000","last_active":"2021-03-24T21:58:12.409577+0000","last_peered":"2021-03-24T21:58:12.409577+0000","last_clean":"2021-03-24T21:58:12.409577+0000","last_became_active":"2021-03-24T21:57:02.182240+0000","last_became_peered":"2021-03-24T21:57:02.182240+0000","last_unstale":"2021-03-24T21:58:12.409577+0000","last_undegraded":"2021-03-24T21:58:12.409577+0000","last_fullsized":"2021-03-24T21:58:12.409577+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:20.215065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:57:20.215065+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"89'22","reported_seq":"319","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.408254+0000","last_change":"2021-03-24T21:56:54.046391+0000","last_active":"2021-03-24T21:58:12.408254+0000","last_peered":"2021-03-24T21:58:12.408254+0000","last_clean":"2021-03-24T21:58:12.408254+0000","last_became_active":"2021-03-24T21:54:38.129459+0000","last_became_peered":"2021-03-24T21:54:38.129459+0000","last_unstale":"2021-03-24T21:58:12.408254+0000","last_undegraded":"2021-03-24T21:58:12.408254+0000","last_fullsized":"2021-03-24T21:58:12.408254+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'22","last_scrub_stamp":"2021-03-24T21:56:54.046194+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2021-03-24T21:53:19.717244+0000","last_clean_scrub_stamp":"2021-03-24T21:56:54.046194+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":36,"num_read_kb":21,"num_write":18,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":644,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"291","reported_epoch":"320","state":"active+clean","last_fresh":"2021-03-24T21:58:10.185209+0000","last_change":"2021-03-24T21:57:23.071573+0000","last_active":"2021-03-24T21:58:10.185209+0000","last_peered":"2021-03-24T21:58:10.185209+0000","last_clean":"2021-03-24T21:58:10.185209+0000","last_became_active":"2021-03-24T21:51:40.351000+0000","last_became_peered":"2021-03-24T21:51:40.351000+0000","last_unstale":"2021-03-24T21:58:10.185209+0000","last_undegraded":"2021-03-24T21:58:10.185209+0000","last_fullsized":"2021-03-24T21:58:10.185209+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:23.071394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:39.286101+0000","last_clean_scrub_stamp":"2021-03-24T21:57:23.071394+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"61'1","reported_seq":"244","reported_epoch":"320","state":"active+clean","last_fresh":"2021-03-24T21:58:10.188888+0000","last_change":"2021-03-24T21:57:12.046385+0000","last_active":"2021-03-24T21:58:10.188888+0000","last_peered":"2021-03-24T21:58:10.188888+0000","last_clean":"2021-03-24T21:58:10.188888+0000","last_became_active":"2021-03-24T21:54:38.123139+0000","last_became_peered":"2021-03-24T21:54:38.123139+0000","last_unstale":"2021-03-24T21:58:10.188888+0000","last_undegraded":"2021-03-24T21:58:10.188888+0000","last_fullsized":"2021-03-24T21:58:10.188888+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2021-03-24T21:57:12.046206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:12.046206+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"288","reported_epoch":"323","state":"active+clean","last_fresh":"2021-03-24T21:58:13.252042+0000","last_change":"2021-03-24T21:57:56.065379+0000","last_active":"2021-03-24T21:58:13.252042+0000","last_peered":"2021-03-24T21:58:13.252042+0000","last_clean":"2021-03-24T21:58:13.252042+0000","last_became_active":"2021-03-24T21:54:24.186957+0000","last_became_peered":"2021-03-24T21:54:24.186957+0000","last_unstale":"2021-03-24T21:58:13.252042+0000","last_undegraded":"2021-03-24T21:58:13.252042+0000","last_fullsized":"2021-03-24T21:58:13.252042+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:56.065113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:57:56.065113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"320'186","reported_seq":"924","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.403102+0000","last_change":"2021-03-24T21:58:06.194517+0000","last_active":"2021-03-24T21:58:12.403102+0000","last_peered":"2021-03-24T21:58:12.403102+0000","last_clean":"2021-03-24T21:58:12.403102+0000","last_became_active":"2021-03-24T21:56:08.154125+0000","last_became_peered":"2021-03-24T21:56:08.154125+0000","last_unstale":"2021-03-24T21:58:12.403102+0000","last_undegraded":"2021-03-24T21:58:12.403102+0000","last_fullsized":"2021-03-24T21:58:12.403102+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'177","last_scrub_stamp":"2021-03-24T21:57:21.260725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:57:21.260725+0000","log_size":186,"ondisk_log_size":186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5146896,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":24422,"num_write":6,"num_write_kb":5028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"cd","length":"1"},{"start":"cf","length":"1"}]},{"pgid":"5.5","version":"0'0","reported_seq":"283","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.403737+0000","last_change":"2021-03-24T21:55:36.236833+0000","last_active":"2021-03-24T21:58:12.403737+0000","last_peered":"2021-03-24T21:58:12.403737+0000","last_clean":"2021-03-24T21:58:12.403737+0000","last_became_active":"2021-03-24T21:54:38.136859+0000","last_became_peered":"2021-03-24T21:54:38.136859+0000","last_unstale":"2021-03-24T21:58:12.403737+0000","last_undegraded":"2021-03-24T21:58:12.403737+0000","last_fullsized":"2021-03-24T21:58:12.403737+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:36.236594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:55:36.236594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"69'25","reported_seq":"320","reported_epoch":"320","state":"active+clean","last_fresh":"2021-03-24T21:58:10.186655+0000","last_change":"2021-03-24T21:57:15.071588+0000","last_active":"2021-03-24T21:58:10.186655+0000","last_peered":"2021-03-24T21:58:10.186655+0000","last_clean":"2021-03-24T21:58:10.186655+0000","last_became_active":"2021-03-24T21:54:38.141478+0000","last_became_peered":"2021-03-24T21:54:38.141478+0000","last_unstale":"2021-03-24T21:58:10.186655+0000","last_undegraded":"2021-03-24T21:58:10.186655+0000","last_fullsized":"2021-03-24T21:58:10.186655+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'25","last_scrub_stamp":"2021-03-24T21:57:15.071486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:57:15.071486+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":53,"num_read_kb":30,"num_write":28,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"89'2","reported_seq":"312","reported_epoch":"321","state":"active+clean","last_fresh":"2021-03-24T21:58:11.537678+0000","last_change":"2021-03-24T21:57:58.492092+0000","last_active":"2021-03-24T21:58:11.537678+0000","last_peered":"2021-03-24T21:58:11.537678+0000","last_clean":"2021-03-24T21:58:11.537678+0000","last_became_active":"2021-03-24T21:54:38.144013+0000","last_became_peered":"2021-03-24T21:54:38.144013+0000","last_unstale":"2021-03-24T21:58:11.537678+0000","last_undegraded":"2021-03-24T21:58:11.537678+0000","last_fullsized":"2021-03-24T21:58:11.537678+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2021-03-24T21:57:58.491830+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2021-03-24T21:57:58.491830+0000","last_clean_scrub_stamp":"2021-03-24T21:57:58.491830+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"273","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.403401+0000","last_change":"2021-03-24T21:58:01.817561+0000","last_active":"2021-03-24T21:58:12.403401+0000","last_peered":"2021-03-24T21:58:12.403401+0000","last_clean":"2021-03-24T21:58:12.403401+0000","last_became_active":"2021-03-24T21:55:10.100245+0000","last_became_peered":"2021-03-24T21:55:10.100245+0000","last_unstale":"2021-03-24T21:58:12.403401+0000","last_undegraded":"2021-03-24T21:58:12.403401+0000","last_fullsized":"2021-03-24T21:58:12.403401+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:01.817480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:06.800417+0000","last_clean_scrub_stamp":"2021-03-24T21:58:01.817480+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"277","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.227448+0000","last_change":"2021-03-24T21:57:22.422901+0000","last_active":"2021-03-24T21:58:12.227448+0000","last_peered":"2021-03-24T21:58:12.227448+0000","last_clean":"2021-03-24T21:58:12.227448+0000","last_became_active":"2021-03-24T21:51:51.759514+0000","last_became_peered":"2021-03-24T21:51:51.759514+0000","last_unstale":"2021-03-24T21:58:12.227448+0000","last_undegraded":"2021-03-24T21:58:12.227448+0000","last_fullsized":"2021-03-24T21:58:12.227448+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:22.422787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:52:59.706511+0000","last_clean_scrub_stamp":"2021-03-24T21:57:22.422787+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"285'227","reported_seq":"974","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.404492+0000","last_change":"2021-03-24T21:58:06.195112+0000","last_active":"2021-03-24T21:58:12.404492+0000","last_peered":"2021-03-24T21:58:12.404492+0000","last_clean":"2021-03-24T21:58:12.404492+0000","last_became_active":"2021-03-24T21:56:08.157153+0000","last_became_peered":"2021-03-24T21:56:08.157153+0000","last_unstale":"2021-03-24T21:58:12.404492+0000","last_undegraded":"2021-03-24T21:58:12.404492+0000","last_fullsized":"2021-03-24T21:58:12.404492+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'218","last_scrub_stamp":"2021-03-24T21:57:22.239425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:57:22.239425+0000","log_size":227,"ondisk_log_size":227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3798415,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"cd","length":"1"},{"start":"cf","length":"1"}]},{"pgid":"4.b","version":"69'14","reported_seq":"331","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.403692+0000","last_change":"2021-03-24T21:55:57.297176+0000","last_active":"2021-03-24T21:58:12.403692+0000","last_peered":"2021-03-24T21:58:12.403692+0000","last_clean":"2021-03-24T21:58:12.403692+0000","last_became_active":"2021-03-24T21:54:38.138034+0000","last_became_peered":"2021-03-24T21:54:38.138034+0000","last_unstale":"2021-03-24T21:58:12.403692+0000","last_undegraded":"2021-03-24T21:58:12.403692+0000","last_fullsized":"2021-03-24T21:58:12.403692+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'14","last_scrub_stamp":"2021-03-24T21:55:57.297084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:57.297084+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":16,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":894,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"273","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.227553+0000","last_change":"2021-03-24T21:57:12.514184+0000","last_active":"2021-03-24T21:58:12.227553+0000","last_peered":"2021-03-24T21:58:12.227553+0000","last_clean":"2021-03-24T21:58:12.227553+0000","last_became_active":"2021-03-24T21:55:10.102360+0000","last_became_peered":"2021-03-24T21:55:10.102360+0000","last_unstale":"2021-03-24T21:58:12.227553+0000","last_undegraded":"2021-03-24T21:58:12.227553+0000","last_fullsized":"2021-03-24T21:58:12.227553+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:12.514094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:12.514094+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"278","reported_epoch":"320","state":"active+clean","last_fresh":"2021-03-24T21:58:10.185456+0000","last_change":"2021-03-24T21:56:43.052062+0000","last_active":"2021-03-24T21:58:10.185456+0000","last_peered":"2021-03-24T21:58:10.185456+0000","last_clean":"2021-03-24T21:58:10.185456+0000","last_became_active":"2021-03-24T21:51:49.411921+0000","last_became_peered":"2021-03-24T21:51:49.411921+0000","last_unstale":"2021-03-24T21:58:10.185456+0000","last_undegraded":"2021-03-24T21:58:10.185456+0000","last_fullsized":"2021-03-24T21:58:10.185456+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:43.051885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:43.051885+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"281","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.402873+0000","last_change":"2021-03-24T21:57:02.189636+0000","last_active":"2021-03-24T21:58:12.402873+0000","last_peered":"2021-03-24T21:58:12.402873+0000","last_clean":"2021-03-24T21:58:12.402873+0000","last_became_active":"2021-03-24T21:57:02.188645+0000","last_became_peered":"2021-03-24T21:57:02.188645+0000","last_unstale":"2021-03-24T21:58:12.402873+0000","last_undegraded":"2021-03-24T21:58:12.402873+0000","last_fullsized":"2021-03-24T21:58:12.402873+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:33.246205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:35.636299+0000","last_clean_scrub_stamp":"2021-03-24T21:55:33.246205+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"320'130","reported_seq":"826","reported_epoch":"320","state":"active+clean","last_fresh":"2021-03-24T21:58:10.559940+0000","last_change":"2021-03-24T21:58:06.202935+0000","last_active":"2021-03-24T21:58:10.559940+0000","last_peered":"2021-03-24T21:58:10.559940+0000","last_clean":"2021-03-24T21:58:10.559940+0000","last_became_active":"2021-03-24T21:56:08.148045+0000","last_became_peered":"2021-03-24T21:56:08.148045+0000","last_unstale":"2021-03-24T21:58:10.559940+0000","last_undegraded":"2021-03-24T21:58:10.559940+0000","last_fullsized":"2021-03-24T21:58:10.559940+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'96","last_scrub_stamp":"2021-03-24T21:56:40.081137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:56:40.081137+0000","log_size":130,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2941291,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":16,"num_write_kb":4334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"cd","length":"1"},{"start":"cf","length":"1"}]},{"pgid":"4.a","version":"69'23","reported_seq":"83","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.407427+0000","last_change":"2021-03-24T21:57:57.287776+0000","last_active":"2021-03-24T21:58:12.407427+0000","last_peered":"2021-03-24T21:58:12.407427+0000","last_clean":"2021-03-24T21:58:12.407427+0000","last_became_active":"2021-03-24T21:57:57.287343+0000","last_became_peered":"2021-03-24T21:57:57.287343+0000","last_unstale":"2021-03-24T21:58:12.407427+0000","last_undegraded":"2021-03-24T21:58:12.407427+0000","last_fullsized":"2021-03-24T21:58:12.407427+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'23","last_scrub_stamp":"2021-03-24T21:56:34.991954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:34.991954+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":43,"num_read_kb":24,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":"86","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.409871+0000","last_change":"2021-03-24T21:57:58.133427+0000","last_active":"2021-03-24T21:58:12.409871+0000","last_peered":"2021-03-24T21:58:12.409871+0000","last_clean":"2021-03-24T21:58:12.409871+0000","last_became_active":"2021-03-24T21:57:57.291071+0000","last_became_peered":"2021-03-24T21:57:57.291071+0000","last_unstale":"2021-03-24T21:58:12.409871+0000","last_undegraded":"2021-03-24T21:58:12.409871+0000","last_fullsized":"2021-03-24T21:58:12.409871+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:58.133341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:58.133341+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"82","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.403243+0000","last_change":"2021-03-24T21:57:02.185635+0000","last_active":"2021-03-24T21:58:12.403243+0000","last_peered":"2021-03-24T21:58:12.403243+0000","last_clean":"2021-03-24T21:58:12.403243+0000","last_became_active":"2021-03-24T21:57:02.185281+0000","last_became_peered":"2021-03-24T21:57:02.185281+0000","last_unstale":"2021-03-24T21:58:12.403243+0000","last_undegraded":"2021-03-24T21:58:12.403243+0000","last_fullsized":"2021-03-24T21:58:12.403243+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:44.018043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:44.018043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"31","reported_epoch":"320","state":"active+clean","last_fresh":"2021-03-24T21:58:10.188997+0000","last_change":"2021-03-24T21:57:58.225287+0000","last_active":"2021-03-24T21:58:10.188997+0000","last_peered":"2021-03-24T21:58:10.188997+0000","last_clean":"2021-03-24T21:58:10.188997+0000","last_became_active":"2021-03-24T21:57:57.296747+0000","last_became_peered":"2021-03-24T21:57:57.296747+0000","last_unstale":"2021-03-24T21:58:10.188997+0000","last_undegraded":"2021-03-24T21:58:10.188997+0000","last_fullsized":"2021-03-24T21:58:10.188997+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:58.225162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:57:58.225162+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"318'221","reported_seq":"959","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.403209+0000","last_change":"2021-03-24T21:58:06.190679+0000","last_active":"2021-03-24T21:58:12.403209+0000","last_peered":"2021-03-24T21:58:12.403209+0000","last_clean":"2021-03-24T21:58:12.403209+0000","last_became_active":"2021-03-24T21:56:08.158313+0000","last_became_peered":"2021-03-24T21:56:08.158313+0000","last_unstale":"2021-03-24T21:58:12.403209+0000","last_undegraded":"2021-03-24T21:58:12.403209+0000","last_fullsized":"2021-03-24T21:58:12.403209+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'211","last_scrub_stamp":"2021-03-24T21:57:25.266209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:57:25.266209+0000","log_size":221,"ondisk_log_size":221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5243916,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":5,"num_write_kb":3910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"cd","length":"1"},{"start":"cf","length":"1"}]},{"pgid":"5.8","version":"0'0","reported_seq":"287","reported_epoch":"320","state":"active+clean","last_fresh":"2021-03-24T21:58:10.188692+0000","last_change":"2021-03-24T21:56:51.990882+0000","last_active":"2021-03-24T21:58:10.188692+0000","last_peered":"2021-03-24T21:58:10.188692+0000","last_clean":"2021-03-24T21:58:10.188692+0000","last_became_active":"2021-03-24T21:54:38.124317+0000","last_became_peered":"2021-03-24T21:54:38.124317+0000","last_unstale":"2021-03-24T21:58:10.188692+0000","last_undegraded":"2021-03-24T21:58:10.188692+0000","last_fullsized":"2021-03-24T21:58:10.188692+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:51.990762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:51.990762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"69'15","reported_seq":"332","reported_epoch":"321","state":"active+clean","last_fresh":"2021-03-24T21:58:11.536601+0000","last_change":"2021-03-24T21:56:02.021432+0000","last_active":"2021-03-24T21:58:11.536601+0000","last_peered":"2021-03-24T21:58:11.536601+0000","last_clean":"2021-03-24T21:58:11.536601+0000","last_became_active":"2021-03-24T21:54:38.139863+0000","last_became_peered":"2021-03-24T21:54:38.139863+0000","last_unstale":"2021-03-24T21:58:11.536601+0000","last_undegraded":"2021-03-24T21:58:11.536601+0000","last_fullsized":"2021-03-24T21:58:11.536601+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:56:02.021288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:02.021288+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":19,"num_write":17,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":692,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":"299","reported_epoch":"323","state":"active+clean","last_fresh":"2021-03-24T21:58:13.253003+0000","last_change":"2021-03-24T21:57:57.050020+0000","last_active":"2021-03-24T21:58:13.253003+0000","last_peered":"2021-03-24T21:58:13.253003+0000","last_clean":"2021-03-24T21:58:13.253003+0000","last_became_active":"2021-03-24T21:54:38.109439+0000","last_became_peered":"2021-03-24T21:54:38.109439+0000","last_unstale":"2021-03-24T21:58:13.253003+0000","last_undegraded":"2021-03-24T21:58:13.253003+0000","last_fullsized":"2021-03-24T21:58:13.253003+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:57.049862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:57.049862+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"290","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.227235+0000","last_change":"2021-03-24T21:56:44.228591+0000","last_active":"2021-03-24T21:58:12.227235+0000","last_peered":"2021-03-24T21:58:12.227235+0000","last_clean":"2021-03-24T21:58:12.227235+0000","last_became_active":"2021-03-24T21:56:44.228142+0000","last_became_peered":"2021-03-24T21:56:44.228142+0000","last_unstale":"2021-03-24T21:58:12.227235+0000","last_undegraded":"2021-03-24T21:58:12.227235+0000","last_fullsized":"2021-03-24T21:58:12.227235+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:24.548819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:22.724840+0000","last_clean_scrub_stamp":"2021-03-24T21:55:24.548819+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"272'122","reported_seq":"817","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.226478+0000","last_change":"2021-03-24T21:58:06.190925+0000","last_active":"2021-03-24T21:58:12.226478+0000","last_peered":"2021-03-24T21:58:12.226478+0000","last_clean":"2021-03-24T21:58:12.226478+0000","last_became_active":"2021-03-24T21:57:26.169639+0000","last_became_peered":"2021-03-24T21:57:26.169639+0000","last_unstale":"2021-03-24T21:58:12.226478+0000","last_undegraded":"2021-03-24T21:58:12.226478+0000","last_fullsized":"2021-03-24T21:58:12.226478+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":122,"ondisk_log_size":122,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1983226,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":23,"num_read_kb":22367,"num_write":28,"num_write_kb":8975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"cd","length":"1"},{"start":"cf","length":"1"}]},{"pgid":"5.9","version":"109'4","reported_seq":"386","reported_epoch":"323","state":"active+clean","last_fresh":"2021-03-24T21:58:14.019172+0000","last_change":"2021-03-24T21:56:32.051794+0000","last_active":"2021-03-24T21:58:14.019172+0000","last_peered":"2021-03-24T21:58:14.019172+0000","last_clean":"2021-03-24T21:58:14.019172+0000","last_became_active":"2021-03-24T21:54:38.125497+0000","last_became_peered":"2021-03-24T21:54:38.125497+0000","last_unstale":"2021-03-24T21:58:14.019172+0000","last_undegraded":"2021-03-24T21:58:14.019172+0000","last_fullsized":"2021-03-24T21:58:14.019172+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'4","last_scrub_stamp":"2021-03-24T21:56:32.051533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:32.051533+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"97'28","reported_seq":"356","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.406775+0000","last_change":"2021-03-24T21:56:05.210757+0000","last_active":"2021-03-24T21:58:12.406775+0000","last_peered":"2021-03-24T21:58:12.406775+0000","last_clean":"2021-03-24T21:58:12.406775+0000","last_became_active":"2021-03-24T21:54:24.190817+0000","last_became_peered":"2021-03-24T21:54:24.190817+0000","last_unstale":"2021-03-24T21:58:12.406775+0000","last_undegraded":"2021-03-24T21:58:12.406775+0000","last_fullsized":"2021-03-24T21:58:12.406775+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'28","last_scrub_stamp":"2021-03-24T21:56:05.210553+0000","last_deep_scrub":"97'28","last_deep_scrub_stamp":"2021-03-24T21:54:19.157265+0000","last_clean_scrub_stamp":"2021-03-24T21:56:05.210553+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":20,"num_write":19,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"89'4","reported_seq":"285","reported_epoch":"323","state":"active+clean","last_fresh":"2021-03-24T21:58:13.252708+0000","last_change":"2021-03-24T21:56:03.060260+0000","last_active":"2021-03-24T21:58:13.252708+0000","last_peered":"2021-03-24T21:58:13.252708+0000","last_clean":"2021-03-24T21:58:13.252708+0000","last_became_active":"2021-03-24T21:51:45.385110+0000","last_became_peered":"2021-03-24T21:51:45.385110+0000","last_unstale":"2021-03-24T21:58:13.252708+0000","last_undegraded":"2021-03-24T21:58:13.252708+0000","last_fullsized":"2021-03-24T21:58:13.252708+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'4","last_scrub_stamp":"2021-03-24T21:56:03.060069+0000","last_deep_scrub":"89'4","last_deep_scrub_stamp":"2021-03-24T21:53:19.926637+0000","last_clean_scrub_stamp":"2021-03-24T21:56:03.060069+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"256","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.404304+0000","last_change":"2021-03-24T21:57:02.187469+0000","last_active":"2021-03-24T21:58:12.404304+0000","last_peered":"2021-03-24T21:58:12.404304+0000","last_clean":"2021-03-24T21:58:12.404304+0000","last_became_active":"2021-03-24T21:57:02.187060+0000","last_became_peered":"2021-03-24T21:57:02.187060+0000","last_unstale":"2021-03-24T21:58:12.404304+0000","last_undegraded":"2021-03-24T21:58:12.404304+0000","last_fullsized":"2021-03-24T21:58:12.404304+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:54.262681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:54.262681+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"272'275","reported_seq":"1132","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.406127+0000","last_change":"2021-03-24T21:58:06.189000+0000","last_active":"2021-03-24T21:58:12.406127+0000","last_peered":"2021-03-24T21:58:12.406127+0000","last_clean":"2021-03-24T21:58:12.406127+0000","last_became_active":"2021-03-24T21:57:26.170367+0000","last_became_peered":"2021-03-24T21:57:26.170367+0000","last_unstale":"2021-03-24T21:58:12.406127+0000","last_undegraded":"2021-03-24T21:58:12.406127+0000","last_fullsized":"2021-03-24T21:58:12.406127+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"191'150","last_scrub_stamp":"2021-03-24T21:56:01.308050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:56:01.308050+0000","log_size":275,"ondisk_log_size":275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8857338,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":118,"num_read_kb":138390,"num_write":52,"num_write_kb":22259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":12587455,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"cd","length":"1"},{"start":"cf","length":"1"}]},{"pgid":"4.17","version":"69'7","reported_seq":"306","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.228725+0000","last_change":"2021-03-24T21:55:56.490518+0000","last_active":"2021-03-24T21:58:12.228725+0000","last_peered":"2021-03-24T21:58:12.228725+0000","last_clean":"2021-03-24T21:58:12.228725+0000","last_became_active":"2021-03-24T21:54:38.144098+0000","last_became_peered":"2021-03-24T21:54:38.144098+0000","last_unstale":"2021-03-24T21:58:12.228725+0000","last_undegraded":"2021-03-24T21:58:12.228725+0000","last_fullsized":"2021-03-24T21:58:12.228725+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'7","last_scrub_stamp":"2021-03-24T21:55:56.490439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:56.490439+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":7,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":158,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":"242","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.402217+0000","last_change":"2021-03-24T21:57:16.270995+0000","last_active":"2021-03-24T21:58:12.402217+0000","last_peered":"2021-03-24T21:58:12.402217+0000","last_clean":"2021-03-24T21:58:12.402217+0000","last_became_active":"2021-03-24T21:54:24.201388+0000","last_became_peered":"2021-03-24T21:54:24.201388+0000","last_unstale":"2021-03-24T21:58:12.402217+0000","last_undegraded":"2021-03-24T21:58:12.402217+0000","last_fullsized":"2021-03-24T21:58:12.402217+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:16.270874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:16.270874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"291","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.409239+0000","last_change":"2021-03-24T21:56:48.144572+0000","last_active":"2021-03-24T21:58:12.409239+0000","last_peered":"2021-03-24T21:58:12.409239+0000","last_clean":"2021-03-24T21:58:12.409239+0000","last_became_active":"2021-03-24T21:54:38.128409+0000","last_became_peered":"2021-03-24T21:54:38.128409+0000","last_unstale":"2021-03-24T21:58:12.409239+0000","last_undegraded":"2021-03-24T21:58:12.409239+0000","last_fullsized":"2021-03-24T21:58:12.409239+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:48.144391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:48.144391+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"288","reported_epoch":"323","state":"active+clean","last_fresh":"2021-03-24T21:58:13.251569+0000","last_change":"2021-03-24T21:58:04.012346+0000","last_active":"2021-03-24T21:58:13.251569+0000","last_peered":"2021-03-24T21:58:13.251569+0000","last_clean":"2021-03-24T21:58:13.251569+0000","last_became_active":"2021-03-24T21:54:24.188386+0000","last_became_peered":"2021-03-24T21:54:24.188386+0000","last_unstale":"2021-03-24T21:58:13.251569+0000","last_undegraded":"2021-03-24T21:58:13.251569+0000","last_fullsized":"2021-03-24T21:58:13.251569+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:04.012279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:58:04.012279+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.14","version":"320'239","reported_seq":"984","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.227252+0000","last_change":"2021-03-24T21:58:06.190279+0000","last_active":"2021-03-24T21:58:12.227252+0000","last_peered":"2021-03-24T21:58:12.227252+0000","last_clean":"2021-03-24T21:58:12.227252+0000","last_became_active":"2021-03-24T21:56:14.139804+0000","last_became_peered":"2021-03-24T21:56:14.139804+0000","last_unstale":"2021-03-24T21:58:12.227252+0000","last_undegraded":"2021-03-24T21:58:12.227252+0000","last_fullsized":"2021-03-24T21:58:12.227252+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":239,"ondisk_log_size":239,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7033895,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":91,"num_read_kb":83359,"num_write":82,"num_write_kb":24561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":6105417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"cd","length":"1"},{"start":"cf","length":"1"}]},{"pgid":"4.16","version":"69'16","reported_seq":"311","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.403956+0000","last_change":"2021-03-24T21:56:40.303286+0000","last_active":"2021-03-24T21:58:12.403956+0000","last_peered":"2021-03-24T21:58:12.403956+0000","last_clean":"2021-03-24T21:58:12.403956+0000","last_became_active":"2021-03-24T21:51:47.687595+0000","last_became_peered":"2021-03-24T21:51:47.687595+0000","last_unstale":"2021-03-24T21:58:12.403956+0000","last_undegraded":"2021-03-24T21:58:12.403956+0000","last_fullsized":"2021-03-24T21:58:12.403956+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T21:56:40.303200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:40.303200+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":38,"num_read_kb":18,"num_write":17,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":"80","reported_epoch":"321","state":"active+clean","last_fresh":"2021-03-24T21:58:11.536653+0000","last_change":"2021-03-24T21:57:57.288460+0000","last_active":"2021-03-24T21:58:11.536653+0000","last_peered":"2021-03-24T21:58:11.536653+0000","last_clean":"2021-03-24T21:58:11.536653+0000","last_became_active":"2021-03-24T21:57:57.287481+0000","last_became_peered":"2021-03-24T21:57:57.287481+0000","last_unstale":"2021-03-24T21:58:11.536653+0000","last_undegraded":"2021-03-24T21:58:11.536653+0000","last_fullsized":"2021-03-24T21:58:11.536653+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:51.037780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:51.037780+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"288","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.228853+0000","last_change":"2021-03-24T21:57:52.376563+0000","last_active":"2021-03-24T21:58:12.228853+0000","last_peered":"2021-03-24T21:58:12.228853+0000","last_clean":"2021-03-24T21:58:12.228853+0000","last_became_active":"2021-03-24T21:54:38.129390+0000","last_became_peered":"2021-03-24T21:54:38.129390+0000","last_unstale":"2021-03-24T21:58:12.228853+0000","last_undegraded":"2021-03-24T21:58:12.228853+0000","last_fullsized":"2021-03-24T21:58:12.228853+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:52.376475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:55.493713+0000","last_clean_scrub_stamp":"2021-03-24T21:57:52.376475+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"276","reported_epoch":"320","state":"active+clean","last_fresh":"2021-03-24T21:58:10.186723+0000","last_change":"2021-03-24T21:57:20.073705+0000","last_active":"2021-03-24T21:58:10.186723+0000","last_peered":"2021-03-24T21:58:10.186723+0000","last_clean":"2021-03-24T21:58:10.186723+0000","last_became_active":"2021-03-24T21:51:51.754085+0000","last_became_peered":"2021-03-24T21:51:51.754085+0000","last_unstale":"2021-03-24T21:58:10.186723+0000","last_undegraded":"2021-03-24T21:58:10.186723+0000","last_fullsized":"2021-03-24T21:58:10.186723+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:20.073641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:57:20.073641+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"307'92","reported_seq":"837","reported_epoch":"323","state":"active+clean","last_fresh":"2021-03-24T21:58:13.251723+0000","last_change":"2021-03-24T21:58:06.203848+0000","last_active":"2021-03-24T21:58:13.251723+0000","last_peered":"2021-03-24T21:58:13.251723+0000","last_clean":"2021-03-24T21:58:13.251723+0000","last_became_active":"2021-03-24T21:56:10.139551+0000","last_became_peered":"2021-03-24T21:56:10.139551+0000","last_unstale":"2021-03-24T21:58:13.251723+0000","last_undegraded":"2021-03-24T21:58:13.251723+0000","last_fullsized":"2021-03-24T21:58:13.251723+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":5,"last_scrub":"233'79","last_scrub_stamp":"2021-03-24T21:56:50.934781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:56:50.934781+0000","log_size":92,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2453888,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":8829,"num_write":10,"num_write_kb":2640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"cd","length":"1"},{"start":"cf","length":"1"}]},{"pgid":"4.15","version":"69'13","reported_seq":"308","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.409981+0000","last_change":"2021-03-24T21:56:47.168651+0000","last_active":"2021-03-24T21:58:12.409981+0000","last_peered":"2021-03-24T21:58:12.409981+0000","last_clean":"2021-03-24T21:58:12.409981+0000","last_became_active":"2021-03-24T21:51:47.689288+0000","last_became_peered":"2021-03-24T21:51:47.689288+0000","last_unstale":"2021-03-24T21:58:12.409981+0000","last_undegraded":"2021-03-24T21:58:12.409981+0000","last_fullsized":"2021-03-24T21:58:12.409981+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2021-03-24T21:56:47.168558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:47.168558+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":"283","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.403375+0000","last_change":"2021-03-24T21:56:32.279216+0000","last_active":"2021-03-24T21:58:12.403375+0000","last_peered":"2021-03-24T21:58:12.403375+0000","last_clean":"2021-03-24T21:58:12.403375+0000","last_became_active":"2021-03-24T21:51:45.381516+0000","last_became_peered":"2021-03-24T21:51:45.381516+0000","last_unstale":"2021-03-24T21:58:12.403375+0000","last_undegraded":"2021-03-24T21:58:12.403375+0000","last_fullsized":"2021-03-24T21:58:12.403375+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:32.279119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:05.473348+0000","last_clean_scrub_stamp":"2021-03-24T21:56:32.279119+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"235'4","reported_seq":"367","reported_epoch":"323","state":"active+clean","last_fresh":"2021-03-24T21:58:13.443256+0000","last_change":"2021-03-24T21:56:44.227488+0000","last_active":"2021-03-24T21:58:13.443256+0000","last_peered":"2021-03-24T21:58:13.443256+0000","last_clean":"2021-03-24T21:58:13.443256+0000","last_became_active":"2021-03-24T21:56:44.224410+0000","last_became_peered":"2021-03-24T21:56:44.224410+0000","last_unstale":"2021-03-24T21:58:13.443256+0000","last_undegraded":"2021-03-24T21:58:13.443256+0000","last_fullsized":"2021-03-24T21:58:13.443256+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'3","last_scrub_stamp":"2021-03-24T21:55:01.577769+0000","last_deep_scrub":"86'3","last_deep_scrub_stamp":"2021-03-24T21:52:56.678111+0000","last_clean_scrub_stamp":"2021-03-24T21:55:01.577769+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"279","reported_epoch":"321","state":"active+clean","last_fresh":"2021-03-24T21:58:11.539473+0000","last_change":"2021-03-24T21:56:51.925140+0000","last_active":"2021-03-24T21:58:11.539473+0000","last_peered":"2021-03-24T21:58:11.539473+0000","last_clean":"2021-03-24T21:58:11.539473+0000","last_became_active":"2021-03-24T21:51:51.762876+0000","last_became_peered":"2021-03-24T21:51:51.762876+0000","last_unstale":"2021-03-24T21:58:11.539473+0000","last_undegraded":"2021-03-24T21:58:11.539473+0000","last_fullsized":"2021-03-24T21:58:11.539473+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:51.924963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:51.924963+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.16","version":"316'167","reported_seq":"892","reported_epoch":"321","state":"active+clean","last_fresh":"2021-03-24T21:58:11.537836+0000","last_change":"2021-03-24T21:58:06.215825+0000","last_active":"2021-03-24T21:58:11.537836+0000","last_peered":"2021-03-24T21:58:11.537836+0000","last_clean":"2021-03-24T21:58:11.537836+0000","last_became_active":"2021-03-24T21:56:13.127048+0000","last_became_peered":"2021-03-24T21:56:13.127048+0000","last_unstale":"2021-03-24T21:58:11.537836+0000","last_undegraded":"2021-03-24T21:58:11.537836+0000","last_fullsized":"2021-03-24T21:58:11.537836+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'146","last_scrub_stamp":"2021-03-24T21:57:22.669305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:57:22.669305+0000","log_size":167,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2739607,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":10,"num_write_kb":2219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"cd","length":"1"},{"start":"cf","length":"1"}]},{"pgid":"4.14","version":"69'14","reported_seq":"331","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.227930+0000","last_change":"2021-03-24T21:55:25.582566+0000","last_active":"2021-03-24T21:58:12.227930+0000","last_peered":"2021-03-24T21:58:12.227930+0000","last_clean":"2021-03-24T21:58:12.227930+0000","last_became_active":"2021-03-24T21:54:38.143783+0000","last_became_peered":"2021-03-24T21:54:38.143783+0000","last_unstale":"2021-03-24T21:58:12.227930+0000","last_undegraded":"2021-03-24T21:58:12.227930+0000","last_fullsized":"2021-03-24T21:58:12.227930+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'14","last_scrub_stamp":"2021-03-24T21:55:25.582388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:25.582388+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":27,"num_read_kb":14,"num_write":14,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":"285","reported_epoch":"323","state":"active+clean","last_fresh":"2021-03-24T21:58:13.251012+0000","last_change":"2021-03-24T21:58:02.015066+0000","last_active":"2021-03-24T21:58:13.251012+0000","last_peered":"2021-03-24T21:58:13.251012+0000","last_clean":"2021-03-24T21:58:13.251012+0000","last_became_active":"2021-03-24T21:51:45.579271+0000","last_became_peered":"2021-03-24T21:51:45.579271+0000","last_unstale":"2021-03-24T21:58:13.251012+0000","last_undegraded":"2021-03-24T21:58:13.251012+0000","last_fullsized":"2021-03-24T21:58:13.251012+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:02.014901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:58:02.014901+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"109'4","reported_seq":"374","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.409637+0000","last_change":"2021-03-24T21:56:42.209663+0000","last_active":"2021-03-24T21:58:12.409637+0000","last_peered":"2021-03-24T21:58:12.409637+0000","last_clean":"2021-03-24T21:58:12.409637+0000","last_became_active":"2021-03-24T21:54:38.130026+0000","last_became_peered":"2021-03-24T21:54:38.130026+0000","last_unstale":"2021-03-24T21:58:12.409637+0000","last_undegraded":"2021-03-24T21:58:12.409637+0000","last_fullsized":"2021-03-24T21:58:12.409637+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'4","last_scrub_stamp":"2021-03-24T21:56:42.209497+0000","last_deep_scrub":"86'3","last_deep_scrub_stamp":"2021-03-24T21:53:08.815163+0000","last_clean_scrub_stamp":"2021-03-24T21:56:42.209497+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"277","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.403166+0000","last_change":"2021-03-24T21:57:58.768856+0000","last_active":"2021-03-24T21:58:12.403166+0000","last_peered":"2021-03-24T21:58:12.403166+0000","last_clean":"2021-03-24T21:58:12.403166+0000","last_became_active":"2021-03-24T21:51:51.756584+0000","last_became_peered":"2021-03-24T21:51:51.756584+0000","last_unstale":"2021-03-24T21:58:12.403166+0000","last_undegraded":"2021-03-24T21:58:12.403166+0000","last_fullsized":"2021-03-24T21:58:12.403166+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:58.768675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:57:58.768675+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.17","version":"323'339","reported_seq":"1170","reported_epoch":"323","state":"active+clean","last_fresh":"2021-03-24T21:58:13.266923+0000","last_change":"2021-03-24T21:58:06.203303+0000","last_active":"2021-03-24T21:58:13.266923+0000","last_peered":"2021-03-24T21:58:13.266923+0000","last_clean":"2021-03-24T21:58:13.266923+0000","last_became_active":"2021-03-24T21:56:13.132418+0000","last_became_peered":"2021-03-24T21:56:13.132418+0000","last_unstale":"2021-03-24T21:58:13.266923+0000","last_undegraded":"2021-03-24T21:58:13.266923+0000","last_fullsized":"2021-03-24T21:58:13.266923+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":5,"last_scrub":"265'274","last_scrub_stamp":"2021-03-24T21:57:14.281739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:57:14.281739+0000","log_size":339,"ondisk_log_size":339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11268209,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":38,"num_read_kb":36426,"num_write":29,"num_write_kb":11492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"cd","length":"1"},{"start":"cf","length":"1"}]},{"pgid":"4.13","version":"69'15","reported_seq":"366","reported_epoch":"321","state":"active+clean","last_fresh":"2021-03-24T21:58:11.538289+0000","last_change":"2021-03-24T21:58:02.548336+0000","last_active":"2021-03-24T21:58:11.538289+0000","last_peered":"2021-03-24T21:58:11.538289+0000","last_clean":"2021-03-24T21:58:11.538289+0000","last_became_active":"2021-03-24T21:54:38.118330+0000","last_became_peered":"2021-03-24T21:54:38.118330+0000","last_unstale":"2021-03-24T21:58:11.538289+0000","last_undegraded":"2021-03-24T21:58:11.538289+0000","last_fullsized":"2021-03-24T21:58:11.538289+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:58:02.548166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:58:02.548166+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":"287","reported_epoch":"321","state":"active+clean","last_fresh":"2021-03-24T21:58:11.537245+0000","last_change":"2021-03-24T21:55:42.135567+0000","last_active":"2021-03-24T21:58:11.537245+0000","last_peered":"2021-03-24T21:58:11.537245+0000","last_clean":"2021-03-24T21:58:11.537245+0000","last_became_active":"2021-03-24T21:54:24.181343+0000","last_became_peered":"2021-03-24T21:54:24.181343+0000","last_unstale":"2021-03-24T21:58:11.537245+0000","last_undegraded":"2021-03-24T21:58:11.537245+0000","last_fullsized":"2021-03-24T21:58:11.537245+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:42.135502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:42.135502+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"245","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.407766+0000","last_change":"2021-03-24T21:57:31.118235+0000","last_active":"2021-03-24T21:58:12.407766+0000","last_peered":"2021-03-24T21:58:12.407766+0000","last_clean":"2021-03-24T21:58:12.407766+0000","last_became_active":"2021-03-24T21:56:35.139207+0000","last_became_peered":"2021-03-24T21:56:35.139207+0000","last_unstale":"2021-03-24T21:58:12.407766+0000","last_undegraded":"2021-03-24T21:58:12.407766+0000","last_fullsized":"2021-03-24T21:58:12.407766+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:31.118050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:57:31.118050+0000","last_clean_scrub_stamp":"2021-03-24T21:57:31.118050+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"272","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.225869+0000","last_change":"2021-03-24T21:55:22.555707+0000","last_active":"2021-03-24T21:58:12.225869+0000","last_peered":"2021-03-24T21:58:12.225869+0000","last_clean":"2021-03-24T21:58:12.225869+0000","last_became_active":"2021-03-24T21:51:51.764871+0000","last_became_peered":"2021-03-24T21:51:51.764871+0000","last_unstale":"2021-03-24T21:58:12.225869+0000","last_undegraded":"2021-03-24T21:58:12.225869+0000","last_fullsized":"2021-03-24T21:58:12.225869+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:22.555624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:23.707914+0000","last_clean_scrub_stamp":"2021-03-24T21:55:22.555624+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.10","version":"318'37","reported_seq":"259","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.403436+0000","last_change":"2021-03-24T21:58:06.188500+0000","last_active":"2021-03-24T21:58:12.403436+0000","last_peered":"2021-03-24T21:58:12.403436+0000","last_clean":"2021-03-24T21:58:12.403436+0000","last_became_active":"2021-03-24T21:57:06.154293+0000","last_became_peered":"2021-03-24T21:57:06.154293+0000","last_unstale":"2021-03-24T21:58:12.403436+0000","last_undegraded":"2021-03-24T21:58:12.403436+0000","last_fullsized":"2021-03-24T21:58:12.403436+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":5,"last_scrub":"277'24","last_scrub_stamp":"2021-03-24T21:57:28.119420+0000","last_deep_scrub":"277'24","last_deep_scrub_stamp":"2021-03-24T21:57:28.119420+0000","last_clean_scrub_stamp":"2021-03-24T21:57:28.119420+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2285818,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":23,"num_read_kb":18238,"num_write":11,"num_write_kb":5221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"cd","length":"1"},{"start":"cf","length":"1"}]},{"pgid":"4.12","version":"69'15","reported_seq":"295","reported_epoch":"320","state":"active+clean","last_fresh":"2021-03-24T21:58:10.185352+0000","last_change":"2021-03-24T21:57:52.248338+0000","last_active":"2021-03-24T21:58:10.185352+0000","last_peered":"2021-03-24T21:58:10.185352+0000","last_clean":"2021-03-24T21:58:10.185352+0000","last_became_active":"2021-03-24T21:54:38.122707+0000","last_became_peered":"2021-03-24T21:54:38.122707+0000","last_unstale":"2021-03-24T21:58:10.185352+0000","last_undegraded":"2021-03-24T21:58:10.185352+0000","last_fullsized":"2021-03-24T21:58:10.185352+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:57:52.248247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:57:52.248247+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":34,"num_read_kb":16,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":"285","reported_epoch":"323","state":"active+clean","last_fresh":"2021-03-24T21:58:13.251634+0000","last_change":"2021-03-24T21:58:04.965522+0000","last_active":"2021-03-24T21:58:13.251634+0000","last_peered":"2021-03-24T21:58:13.251634+0000","last_clean":"2021-03-24T21:58:13.251634+0000","last_became_active":"2021-03-24T21:51:45.379383+0000","last_became_peered":"2021-03-24T21:51:45.379383+0000","last_unstale":"2021-03-24T21:58:13.251634+0000","last_undegraded":"2021-03-24T21:58:13.251634+0000","last_fullsized":"2021-03-24T21:58:13.251634+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:04.965369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:58:04.965369+0000","last_clean_scrub_stamp":"2021-03-24T21:58:04.965369+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"283","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.227015+0000","last_change":"2021-03-24T21:56:32.480276+0000","last_active":"2021-03-24T21:58:12.227015+0000","last_peered":"2021-03-24T21:58:12.227015+0000","last_clean":"2021-03-24T21:58:12.227015+0000","last_became_active":"2021-03-24T21:54:38.128067+0000","last_became_peered":"2021-03-24T21:54:38.128067+0000","last_unstale":"2021-03-24T21:58:12.227015+0000","last_undegraded":"2021-03-24T21:58:12.227015+0000","last_fullsized":"2021-03-24T21:58:12.227015+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:32.480115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:32.480115+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"286","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.401755+0000","last_change":"2021-03-24T21:56:52.226305+0000","last_active":"2021-03-24T21:58:12.401755+0000","last_peered":"2021-03-24T21:58:12.401755+0000","last_clean":"2021-03-24T21:58:12.401755+0000","last_became_active":"2021-03-24T21:54:38.126662+0000","last_became_peered":"2021-03-24T21:54:38.126662+0000","last_unstale":"2021-03-24T21:58:12.401755+0000","last_undegraded":"2021-03-24T21:58:12.401755+0000","last_fullsized":"2021-03-24T21:58:12.401755+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:52.226117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:52.226117+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.11","version":"324'349","reported_seq":"1120","reported_epoch":"324","state":"active+clean","last_fresh":"2021-03-24T21:58:14.281586+0000","last_change":"2021-03-24T21:58:06.200628+0000","last_active":"2021-03-24T21:58:14.281586+0000","last_peered":"2021-03-24T21:58:14.281586+0000","last_clean":"2021-03-24T21:58:14.281586+0000","last_became_active":"2021-03-24T21:56:14.138730+0000","last_became_peered":"2021-03-24T21:56:14.138730+0000","last_unstale":"2021-03-24T21:58:14.281586+0000","last_undegraded":"2021-03-24T21:58:14.281586+0000","last_fullsized":"2021-03-24T21:58:14.281586+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'297","last_scrub_stamp":"2021-03-24T21:57:24.209124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:57:24.209124+0000","log_size":349,"ondisk_log_size":349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13774973,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":22,"num_read_kb":26768,"num_write":25,"num_write_kb":8661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"cd","length":"1"},{"start":"cf","length":"1"}]},{"pgid":"4.11","version":"86'22","reported_seq":"336","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.226607+0000","last_change":"2021-03-24T21:54:51.585204+0000","last_active":"2021-03-24T21:58:12.226607+0000","last_peered":"2021-03-24T21:58:12.226607+0000","last_clean":"2021-03-24T21:58:12.226607+0000","last_became_active":"2021-03-24T21:54:24.188790+0000","last_became_peered":"2021-03-24T21:54:24.188790+0000","last_unstale":"2021-03-24T21:58:12.226607+0000","last_undegraded":"2021-03-24T21:58:12.226607+0000","last_fullsized":"2021-03-24T21:58:12.226607+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'22","last_scrub_stamp":"2021-03-24T21:54:51.585058+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2021-03-24T21:52:54.597505+0000","last_clean_scrub_stamp":"2021-03-24T21:54:51.585058+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":16,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":580,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":"295","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.409366+0000","last_change":"2021-03-24T21:57:13.016235+0000","last_active":"2021-03-24T21:58:12.409366+0000","last_peered":"2021-03-24T21:58:12.409366+0000","last_clean":"2021-03-24T21:58:12.409366+0000","last_became_active":"2021-03-24T21:54:38.121323+0000","last_became_peered":"2021-03-24T21:54:38.121323+0000","last_unstale":"2021-03-24T21:58:12.409366+0000","last_undegraded":"2021-03-24T21:58:12.409366+0000","last_fullsized":"2021-03-24T21:58:12.409366+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:13.016117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:13.016117+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"290","reported_epoch":"323","state":"active+clean","last_fresh":"2021-03-24T21:58:13.250029+0000","last_change":"2021-03-24T21:57:53.093497+0000","last_active":"2021-03-24T21:58:13.250029+0000","last_peered":"2021-03-24T21:58:13.250029+0000","last_clean":"2021-03-24T21:58:13.250029+0000","last_became_active":"2021-03-24T21:54:24.194423+0000","last_became_peered":"2021-03-24T21:54:24.194423+0000","last_unstale":"2021-03-24T21:58:13.250029+0000","last_undegraded":"2021-03-24T21:58:13.250029+0000","last_fullsized":"2021-03-24T21:58:13.250029+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:53.093327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:57:53.093327+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"281","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.227480+0000","last_change":"2021-03-24T21:56:40.458139+0000","last_active":"2021-03-24T21:58:12.227480+0000","last_peered":"2021-03-24T21:58:12.227480+0000","last_clean":"2021-03-24T21:58:12.227480+0000","last_became_active":"2021-03-24T21:54:24.202686+0000","last_became_peered":"2021-03-24T21:54:24.202686+0000","last_unstale":"2021-03-24T21:58:12.227480+0000","last_undegraded":"2021-03-24T21:58:12.227480+0000","last_fullsized":"2021-03-24T21:58:12.227480+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:40.458011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:47.540793+0000","last_clean_scrub_stamp":"2021-03-24T21:56:40.458011+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.12","version":"183'26","reported_seq":"389","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.402135+0000","last_change":"2021-03-24T21:58:06.189999+0000","last_active":"2021-03-24T21:58:12.402135+0000","last_peered":"2021-03-24T21:58:12.402135+0000","last_clean":"2021-03-24T21:58:12.402135+0000","last_became_active":"2021-03-24T21:56:10.130886+0000","last_became_peered":"2021-03-24T21:56:10.130886+0000","last_unstale":"2021-03-24T21:58:12.402135+0000","last_undegraded":"2021-03-24T21:58:12.402135+0000","last_fullsized":"2021-03-24T21:58:12.402135+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":5,"last_scrub":"183'26","last_scrub_stamp":"2021-03-24T21:57:53.859843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:57:53.859843+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"cd","length":"1"},{"start":"cf","length":"1"}]},{"pgid":"4.10","version":"69'12","reported_seq":"398","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.225793+0000","last_change":"2021-03-24T21:56:53.443666+0000","last_active":"2021-03-24T21:58:12.225793+0000","last_peered":"2021-03-24T21:58:12.225793+0000","last_clean":"2021-03-24T21:58:12.225793+0000","last_became_active":"2021-03-24T21:54:38.134132+0000","last_became_peered":"2021-03-24T21:54:38.134132+0000","last_unstale":"2021-03-24T21:58:12.225793+0000","last_undegraded":"2021-03-24T21:58:12.225793+0000","last_fullsized":"2021-03-24T21:58:12.225793+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2021-03-24T21:56:53.443554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:53.443554+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":12,"num_write":12,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":3756,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":"283","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.404920+0000","last_change":"2021-03-24T21:57:11.230052+0000","last_active":"2021-03-24T21:58:12.404920+0000","last_peered":"2021-03-24T21:58:12.404920+0000","last_clean":"2021-03-24T21:58:12.404920+0000","last_became_active":"2021-03-24T21:51:45.379789+0000","last_became_peered":"2021-03-24T21:51:45.379789+0000","last_unstale":"2021-03-24T21:58:12.404920+0000","last_undegraded":"2021-03-24T21:58:12.404920+0000","last_fullsized":"2021-03-24T21:58:12.404920+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:11.229895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:11.229895+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"241","reported_epoch":"321","state":"active+clean","last_fresh":"2021-03-24T21:58:11.536069+0000","last_change":"2021-03-24T21:57:56.549118+0000","last_active":"2021-03-24T21:58:11.536069+0000","last_peered":"2021-03-24T21:58:11.536069+0000","last_clean":"2021-03-24T21:58:11.536069+0000","last_became_active":"2021-03-24T21:54:24.187568+0000","last_became_peered":"2021-03-24T21:54:24.187568+0000","last_unstale":"2021-03-24T21:58:11.536069+0000","last_undegraded":"2021-03-24T21:58:11.536069+0000","last_fullsized":"2021-03-24T21:58:11.536069+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:56.548933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:57:56.548933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"279","reported_epoch":"323","state":"active+clean","last_fresh":"2021-03-24T21:58:13.250656+0000","last_change":"2021-03-24T21:57:59.979943+0000","last_active":"2021-03-24T21:58:13.250656+0000","last_peered":"2021-03-24T21:58:13.250656+0000","last_clean":"2021-03-24T21:58:13.250656+0000","last_became_active":"2021-03-24T21:51:51.762150+0000","last_became_peered":"2021-03-24T21:51:51.762150+0000","last_unstale":"2021-03-24T21:58:13.250656+0000","last_undegraded":"2021-03-24T21:58:13.250656+0000","last_fullsized":"2021-03-24T21:58:13.250656+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:59.979772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:57:59.979772+0000","last_clean_scrub_stamp":"2021-03-24T21:57:59.979772+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.13","version":"320'241","reported_seq":"1003","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.402137+0000","last_change":"2021-03-24T21:58:06.194713+0000","last_active":"2021-03-24T21:58:12.402137+0000","last_peered":"2021-03-24T21:58:12.402137+0000","last_clean":"2021-03-24T21:58:12.402137+0000","last_became_active":"2021-03-24T21:57:06.153030+0000","last_became_peered":"2021-03-24T21:57:06.153030+0000","last_unstale":"2021-03-24T21:58:12.402137+0000","last_undegraded":"2021-03-24T21:58:12.402137+0000","last_fullsized":"2021-03-24T21:58:12.402137+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":241,"ondisk_log_size":241,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8168268,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":103,"num_read_kb":100579,"num_write":72,"num_write_kb":33176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":4773188,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"cd","length":"1"},{"start":"cf","length":"1"}]},{"pgid":"4.1f","version":"69'17","reported_seq":"311","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.404468+0000","last_change":"2021-03-24T21:56:45.177087+0000","last_active":"2021-03-24T21:58:12.404468+0000","last_peered":"2021-03-24T21:58:12.404468+0000","last_clean":"2021-03-24T21:58:12.404468+0000","last_became_active":"2021-03-24T21:56:44.229823+0000","last_became_peered":"2021-03-24T21:56:44.229823+0000","last_unstale":"2021-03-24T21:58:12.404468+0000","last_undegraded":"2021-03-24T21:58:12.404468+0000","last_fullsized":"2021-03-24T21:58:12.404468+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'17","last_scrub_stamp":"2021-03-24T21:55:32.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:32.214502+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":37,"num_read_kb":18,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":2100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":"287","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.225657+0000","last_change":"2021-03-24T21:55:32.566923+0000","last_active":"2021-03-24T21:58:12.225657+0000","last_peered":"2021-03-24T21:58:12.225657+0000","last_clean":"2021-03-24T21:58:12.225657+0000","last_became_active":"2021-03-24T21:54:38.145870+0000","last_became_peered":"2021-03-24T21:54:38.145870+0000","last_unstale":"2021-03-24T21:58:12.225657+0000","last_undegraded":"2021-03-24T21:58:12.225657+0000","last_fullsized":"2021-03-24T21:58:12.225657+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:32.566747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:32.566747+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"283","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.404259+0000","last_change":"2021-03-24T21:56:35.141430+0000","last_active":"2021-03-24T21:58:12.404259+0000","last_peered":"2021-03-24T21:58:12.404259+0000","last_clean":"2021-03-24T21:58:12.404259+0000","last_became_active":"2021-03-24T21:56:35.140997+0000","last_became_peered":"2021-03-24T21:56:35.140997+0000","last_unstale":"2021-03-24T21:58:12.404259+0000","last_undegraded":"2021-03-24T21:58:12.404259+0000","last_fullsized":"2021-03-24T21:58:12.404259+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:07.255496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:07.255496+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"236","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.409271+0000","last_change":"2021-03-24T21:56:07.272826+0000","last_active":"2021-03-24T21:58:12.409271+0000","last_peered":"2021-03-24T21:58:12.409271+0000","last_clean":"2021-03-24T21:58:12.409271+0000","last_became_active":"2021-03-24T21:54:38.113849+0000","last_became_peered":"2021-03-24T21:54:38.113849+0000","last_unstale":"2021-03-24T21:58:12.409271+0000","last_undegraded":"2021-03-24T21:58:12.409271+0000","last_fullsized":"2021-03-24T21:58:12.409271+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:07.272644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:07.272644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1c","version":"273'245","reported_seq":"982","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.227413+0000","last_change":"2021-03-24T21:58:06.193541+0000","last_active":"2021-03-24T21:58:12.227413+0000","last_peered":"2021-03-24T21:58:12.227413+0000","last_clean":"2021-03-24T21:58:12.227413+0000","last_became_active":"2021-03-24T21:57:31.166900+0000","last_became_peered":"2021-03-24T21:57:31.166900+0000","last_unstale":"2021-03-24T21:58:12.227413+0000","last_undegraded":"2021-03-24T21:58:12.227413+0000","last_fullsized":"2021-03-24T21:58:12.227413+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":245,"ondisk_log_size":245,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8581786,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":70,"num_read_kb":67597,"num_write":52,"num_write_kb":18055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"cd","length":"1"},{"start":"cf","length":"1"}]},{"pgid":"4.1e","version":"89'27","reported_seq":"358","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.401012+0000","last_change":"2021-03-24T21:56:45.178898+0000","last_active":"2021-03-24T21:58:12.401012+0000","last_peered":"2021-03-24T21:58:12.401012+0000","last_clean":"2021-03-24T21:58:12.401012+0000","last_became_active":"2021-03-24T21:56:44.230130+0000","last_became_peered":"2021-03-24T21:56:44.230130+0000","last_unstale":"2021-03-24T21:58:12.401012+0000","last_undegraded":"2021-03-24T21:58:12.401012+0000","last_fullsized":"2021-03-24T21:58:12.401012+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'27","last_scrub_stamp":"2021-03-24T21:55:41.337130+0000","last_deep_scrub":"89'27","last_deep_scrub_stamp":"2021-03-24T21:55:41.337130+0000","last_clean_scrub_stamp":"2021-03-24T21:55:41.337130+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":934,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":"252","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.225998+0000","last_change":"2021-03-24T21:57:26.417745+0000","last_active":"2021-03-24T21:58:12.225998+0000","last_peered":"2021-03-24T21:58:12.225998+0000","last_clean":"2021-03-24T21:58:12.225998+0000","last_became_active":"2021-03-24T21:57:02.181985+0000","last_became_peered":"2021-03-24T21:57:02.181985+0000","last_unstale":"2021-03-24T21:58:12.225998+0000","last_undegraded":"2021-03-24T21:58:12.225998+0000","last_fullsized":"2021-03-24T21:58:12.225998+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:26.417613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:57:26.417613+0000","last_clean_scrub_stamp":"2021-03-24T21:57:26.417613+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"235'4","reported_seq":"308","reported_epoch":"321","state":"active+clean","last_fresh":"2021-03-24T21:58:11.539254+0000","last_change":"2021-03-24T21:56:44.318340+0000","last_active":"2021-03-24T21:58:11.539254+0000","last_peered":"2021-03-24T21:58:11.539254+0000","last_clean":"2021-03-24T21:58:11.539254+0000","last_became_active":"2021-03-24T21:56:44.228196+0000","last_became_peered":"2021-03-24T21:56:44.228196+0000","last_unstale":"2021-03-24T21:58:11.539254+0000","last_undegraded":"2021-03-24T21:58:11.539254+0000","last_fullsized":"2021-03-24T21:58:11.539254+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'3","last_scrub_stamp":"2021-03-24T21:56:42.039976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:42.039976+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"26","reported_epoch":"320","state":"active+clean","last_fresh":"2021-03-24T21:58:10.186342+0000","last_change":"2021-03-24T21:57:57.296879+0000","last_active":"2021-03-24T21:58:10.186342+0000","last_peered":"2021-03-24T21:58:10.186342+0000","last_clean":"2021-03-24T21:58:10.186342+0000","last_became_active":"2021-03-24T21:57:57.296495+0000","last_became_peered":"2021-03-24T21:57:57.296495+0000","last_unstale":"2021-03-24T21:58:10.186342+0000","last_undegraded":"2021-03-24T21:58:10.186342+0000","last_fullsized":"2021-03-24T21:58:10.186342+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:26.129924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:57:26.129924+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1d","version":"316'298","reported_seq":"1090","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.403454+0000","last_change":"2021-03-24T21:58:06.211715+0000","last_active":"2021-03-24T21:58:12.403454+0000","last_peered":"2021-03-24T21:58:12.403454+0000","last_clean":"2021-03-24T21:58:12.403454+0000","last_became_active":"2021-03-24T21:57:32.177289+0000","last_became_peered":"2021-03-24T21:57:32.177289+0000","last_unstale":"2021-03-24T21:58:12.403454+0000","last_undegraded":"2021-03-24T21:58:12.403454+0000","last_fullsized":"2021-03-24T21:58:12.403454+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"233'169","last_scrub_stamp":"2021-03-24T21:56:47.285648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:56:47.285648+0000","log_size":298,"ondisk_log_size":298,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6647284,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":85,"num_read_kb":80656,"num_write":77,"num_write_kb":35496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":9406781,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"cd","length":"1"},{"start":"cf","length":"1"}]},{"pgid":"4.1d","version":"69'16","reported_seq":"336","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.226918+0000","last_change":"2021-03-24T21:56:10.423931+0000","last_active":"2021-03-24T21:58:12.226918+0000","last_peered":"2021-03-24T21:58:12.226918+0000","last_clean":"2021-03-24T21:58:12.226918+0000","last_became_active":"2021-03-24T21:54:38.128246+0000","last_became_peered":"2021-03-24T21:54:38.128246+0000","last_unstale":"2021-03-24T21:58:12.226918+0000","last_undegraded":"2021-03-24T21:58:12.226918+0000","last_fullsized":"2021-03-24T21:58:12.226918+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T21:56:10.423860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:10.423860+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":37,"num_read_kb":20,"num_write":18,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":844,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":"292","reported_epoch":"321","state":"active+clean","last_fresh":"2021-03-24T21:58:11.536789+0000","last_change":"2021-03-24T21:57:13.701322+0000","last_active":"2021-03-24T21:58:11.536789+0000","last_peered":"2021-03-24T21:58:11.536789+0000","last_clean":"2021-03-24T21:58:11.536789+0000","last_became_active":"2021-03-24T21:54:38.143876+0000","last_became_peered":"2021-03-24T21:54:38.143876+0000","last_unstale":"2021-03-24T21:58:11.536789+0000","last_undegraded":"2021-03-24T21:58:11.536789+0000","last_fullsized":"2021-03-24T21:58:11.536789+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:13.701143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:52:54.212459+0000","last_clean_scrub_stamp":"2021-03-24T21:57:13.701143+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"279","reported_epoch":"321","state":"active+clean","last_fresh":"2021-03-24T21:58:11.538161+0000","last_change":"2021-03-24T21:57:19.767717+0000","last_active":"2021-03-24T21:58:11.538161+0000","last_peered":"2021-03-24T21:58:11.538161+0000","last_clean":"2021-03-24T21:58:11.538161+0000","last_became_active":"2021-03-24T21:51:49.410920+0000","last_became_peered":"2021-03-24T21:51:49.410920+0000","last_unstale":"2021-03-24T21:58:11.538161+0000","last_undegraded":"2021-03-24T21:58:11.538161+0000","last_fullsized":"2021-03-24T21:58:11.538161+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:19.767465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:57:19.767465+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"290","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.228277+0000","last_change":"2021-03-24T21:56:33.440700+0000","last_active":"2021-03-24T21:58:12.228277+0000","last_peered":"2021-03-24T21:58:12.228277+0000","last_clean":"2021-03-24T21:58:12.228277+0000","last_became_active":"2021-03-24T21:54:38.143563+0000","last_became_peered":"2021-03-24T21:54:38.143563+0000","last_unstale":"2021-03-24T21:58:12.228277+0000","last_undegraded":"2021-03-24T21:58:12.228277+0000","last_fullsized":"2021-03-24T21:58:12.228277+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:33.440618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:33.440618+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1e","version":"265'45","reported_seq":"146","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.226679+0000","last_change":"2021-03-24T21:58:06.193058+0000","last_active":"2021-03-24T21:58:12.226679+0000","last_peered":"2021-03-24T21:58:12.226679+0000","last_clean":"2021-03-24T21:58:12.226679+0000","last_became_active":"2021-03-24T21:57:28.182130+0000","last_became_peered":"2021-03-24T21:57:28.182130+0000","last_unstale":"2021-03-24T21:58:12.226679+0000","last_undegraded":"2021-03-24T21:58:12.226679+0000","last_fullsized":"2021-03-24T21:58:12.226679+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":5,"last_scrub":"199'30","last_scrub_stamp":"2021-03-24T21:56:10.105084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:56:10.105084+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":18305,"num_write":13,"num_write_kb":3060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"cd","length":"1"},{"start":"cf","length":"1"}]},{"pgid":"4.1c","version":"69'15","reported_seq":"163","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.227728+0000","last_change":"2021-03-24T21:56:35.748629+0000","last_active":"2021-03-24T21:58:12.227728+0000","last_peered":"2021-03-24T21:58:12.227728+0000","last_clean":"2021-03-24T21:58:12.227728+0000","last_became_active":"2021-03-24T21:56:35.142209+0000","last_became_peered":"2021-03-24T21:56:35.142209+0000","last_unstale":"2021-03-24T21:58:12.227728+0000","last_undegraded":"2021-03-24T21:58:12.227728+0000","last_fullsized":"2021-03-24T21:58:12.227728+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:56:06.124139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:06.124139+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":"283","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.404092+0000","last_change":"2021-03-24T21:56:42.254648+0000","last_active":"2021-03-24T21:58:12.404092+0000","last_peered":"2021-03-24T21:58:12.404092+0000","last_clean":"2021-03-24T21:58:12.404092+0000","last_became_active":"2021-03-24T21:51:45.381533+0000","last_became_peered":"2021-03-24T21:51:45.381533+0000","last_unstale":"2021-03-24T21:58:12.404092+0000","last_undegraded":"2021-03-24T21:58:12.404092+0000","last_fullsized":"2021-03-24T21:58:12.404092+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:42.254536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:42.254536+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"235","reported_epoch":"321","state":"active+clean","last_fresh":"2021-03-24T21:58:11.537368+0000","last_change":"2021-03-24T21:57:26.643062+0000","last_active":"2021-03-24T21:58:11.537368+0000","last_peered":"2021-03-24T21:58:11.537368+0000","last_clean":"2021-03-24T21:58:11.537368+0000","last_became_active":"2021-03-24T21:54:38.138620+0000","last_became_peered":"2021-03-24T21:54:38.138620+0000","last_unstale":"2021-03-24T21:58:11.537368+0000","last_undegraded":"2021-03-24T21:58:11.537368+0000","last_fullsized":"2021-03-24T21:58:11.537368+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:26.642883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:57:26.642883+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"281","reported_epoch":"321","state":"active+clean","last_fresh":"2021-03-24T21:58:11.538168+0000","last_change":"2021-03-24T21:56:38.075890+0000","last_active":"2021-03-24T21:58:11.538168+0000","last_peered":"2021-03-24T21:58:11.538168+0000","last_clean":"2021-03-24T21:58:11.538168+0000","last_became_active":"2021-03-24T21:54:24.199444+0000","last_became_peered":"2021-03-24T21:54:24.199444+0000","last_unstale":"2021-03-24T21:58:11.538168+0000","last_undegraded":"2021-03-24T21:58:11.538168+0000","last_fullsized":"2021-03-24T21:58:11.538168+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:38.075714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:38.075714+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1f","version":"272'99","reported_seq":"145","reported_epoch":"321","state":"active+clean","last_fresh":"2021-03-24T21:58:11.538343+0000","last_change":"2021-03-24T21:58:06.216279+0000","last_active":"2021-03-24T21:58:11.538343+0000","last_peered":"2021-03-24T21:58:11.538343+0000","last_clean":"2021-03-24T21:58:11.538343+0000","last_became_active":"2021-03-24T21:57:28.170946+0000","last_became_peered":"2021-03-24T21:57:28.170946+0000","last_unstale":"2021-03-24T21:58:11.538343+0000","last_undegraded":"2021-03-24T21:58:11.538343+0000","last_fullsized":"2021-03-24T21:58:11.538343+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":5,"last_scrub":"218'73","last_scrub_stamp":"2021-03-24T21:56:48.267711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:56:48.267711+0000","log_size":99,"ondisk_log_size":99,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3571504,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":43,"num_read_kb":44484,"num_write":22,"num_write_kb":9163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":942483,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"cd","length":"1"},{"start":"cf","length":"1"}]},{"pgid":"4.1b","version":"91'9","reported_seq":"325","reported_epoch":"321","state":"active+clean","last_fresh":"2021-03-24T21:58:11.537914+0000","last_change":"2021-03-24T21:57:02.447431+0000","last_active":"2021-03-24T21:58:11.537914+0000","last_peered":"2021-03-24T21:58:11.537914+0000","last_clean":"2021-03-24T21:58:11.537914+0000","last_became_active":"2021-03-24T21:57:02.184317+0000","last_became_peered":"2021-03-24T21:57:02.184317+0000","last_unstale":"2021-03-24T21:58:11.537914+0000","last_undegraded":"2021-03-24T21:58:11.537914+0000","last_fullsized":"2021-03-24T21:58:11.537914+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2021-03-24T21:56:04.051464+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2021-03-24T21:53:33.290000+0000","last_clean_scrub_stamp":"2021-03-24T21:56:04.051464+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":8,"num_write":7,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":"267","reported_epoch":"321","state":"active+clean","last_fresh":"2021-03-24T21:58:11.538244+0000","last_change":"2021-03-24T21:56:47.921029+0000","last_active":"2021-03-24T21:58:11.538244+0000","last_peered":"2021-03-24T21:58:11.538244+0000","last_clean":"2021-03-24T21:58:11.538244+0000","last_became_active":"2021-03-24T21:55:10.098790+0000","last_became_peered":"2021-03-24T21:55:10.098790+0000","last_unstale":"2021-03-24T21:58:11.538244+0000","last_undegraded":"2021-03-24T21:58:11.538244+0000","last_fullsized":"2021-03-24T21:58:11.538244+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:47.920949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:47.920949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"290","reported_epoch":"323","state":"active+clean","last_fresh":"2021-03-24T21:58:13.250487+0000","last_change":"2021-03-24T21:56:51.977473+0000","last_active":"2021-03-24T21:58:13.250487+0000","last_peered":"2021-03-24T21:58:13.250487+0000","last_clean":"2021-03-24T21:58:13.250487+0000","last_became_active":"2021-03-24T21:54:38.137706+0000","last_became_peered":"2021-03-24T21:54:38.137706+0000","last_unstale":"2021-03-24T21:58:13.250487+0000","last_undegraded":"2021-03-24T21:58:13.250487+0000","last_fullsized":"2021-03-24T21:58:13.250487+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:51.977306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:56:51.977306+0000","last_clean_scrub_stamp":"2021-03-24T21:56:51.977306+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"289","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.404592+0000","last_change":"2021-03-24T21:57:16.077145+0000","last_active":"2021-03-24T21:58:12.404592+0000","last_peered":"2021-03-24T21:58:12.404592+0000","last_clean":"2021-03-24T21:58:12.404592+0000","last_became_active":"2021-03-24T21:54:38.122349+0000","last_became_peered":"2021-03-24T21:54:38.122349+0000","last_unstale":"2021-03-24T21:58:12.404592+0000","last_undegraded":"2021-03-24T21:58:12.404592+0000","last_fullsized":"2021-03-24T21:58:12.404592+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:16.076982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:57:16.076982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.18","version":"249'90","reported_seq":"415","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.401293+0000","last_change":"2021-03-24T21:58:06.189940+0000","last_active":"2021-03-24T21:58:12.401293+0000","last_peered":"2021-03-24T21:58:12.401293+0000","last_clean":"2021-03-24T21:58:12.401293+0000","last_became_active":"2021-03-24T21:56:10.151682+0000","last_became_peered":"2021-03-24T21:56:10.151682+0000","last_unstale":"2021-03-24T21:58:12.401293+0000","last_undegraded":"2021-03-24T21:58:12.401293+0000","last_fullsized":"2021-03-24T21:58:12.401293+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":5,"last_scrub":"249'90","last_scrub_stamp":"2021-03-24T21:57:56.848784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:57:56.848784+0000","log_size":90,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"cd","length":"1"},{"start":"cf","length":"1"}]},{"pgid":"4.1a","version":"69'13","reported_seq":"300","reported_epoch":"321","state":"active+clean","last_fresh":"2021-03-24T21:58:11.535687+0000","last_change":"2021-03-24T21:55:31.085064+0000","last_active":"2021-03-24T21:58:11.535687+0000","last_peered":"2021-03-24T21:58:11.535687+0000","last_clean":"2021-03-24T21:58:11.535687+0000","last_became_active":"2021-03-24T21:51:47.407488+0000","last_became_peered":"2021-03-24T21:51:47.407488+0000","last_unstale":"2021-03-24T21:58:11.535687+0000","last_undegraded":"2021-03-24T21:58:11.535687+0000","last_fullsized":"2021-03-24T21:58:11.535687+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2021-03-24T21:55:31.084873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:31.084873+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":"309","reported_epoch":"320","state":"active+clean","last_fresh":"2021-03-24T21:58:10.186173+0000","last_change":"2021-03-24T21:56:32.141420+0000","last_active":"2021-03-24T21:58:10.186173+0000","last_peered":"2021-03-24T21:58:10.186173+0000","last_clean":"2021-03-24T21:58:10.186173+0000","last_became_active":"2021-03-24T21:55:10.097431+0000","last_became_peered":"2021-03-24T21:55:10.097431+0000","last_unstale":"2021-03-24T21:58:10.186173+0000","last_undegraded":"2021-03-24T21:58:10.186173+0000","last_fullsized":"2021-03-24T21:58:10.186173+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:32.141276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:32.141276+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"284","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.405113+0000","last_change":"2021-03-24T21:56:38.242245+0000","last_active":"2021-03-24T21:58:12.405113+0000","last_peered":"2021-03-24T21:58:12.405113+0000","last_clean":"2021-03-24T21:58:12.405113+0000","last_became_active":"2021-03-24T21:51:49.413342+0000","last_became_peered":"2021-03-24T21:51:49.413342+0000","last_unstale":"2021-03-24T21:58:12.405113+0000","last_undegraded":"2021-03-24T21:58:12.405113+0000","last_fullsized":"2021-03-24T21:58:12.405113+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:38.242096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:38.242096+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"286","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.402446+0000","last_change":"2021-03-24T21:57:19.271463+0000","last_active":"2021-03-24T21:58:12.402446+0000","last_peered":"2021-03-24T21:58:12.402446+0000","last_clean":"2021-03-24T21:58:12.402446+0000","last_became_active":"2021-03-24T21:54:38.127696+0000","last_became_peered":"2021-03-24T21:54:38.127696+0000","last_unstale":"2021-03-24T21:58:12.402446+0000","last_undegraded":"2021-03-24T21:58:12.402446+0000","last_fullsized":"2021-03-24T21:58:12.402446+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:19.271351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:57:19.271351+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.19","version":"324'327","reported_seq":"1130","reported_epoch":"324","state":"active+clean","last_fresh":"2021-03-24T21:58:14.317251+0000","last_change":"2021-03-24T21:58:06.207083+0000","last_active":"2021-03-24T21:58:14.317251+0000","last_peered":"2021-03-24T21:58:14.317251+0000","last_clean":"2021-03-24T21:58:14.317251+0000","last_became_active":"2021-03-24T21:56:14.139614+0000","last_became_peered":"2021-03-24T21:56:14.139614+0000","last_unstale":"2021-03-24T21:58:14.317251+0000","last_undegraded":"2021-03-24T21:58:14.317251+0000","last_fullsized":"2021-03-24T21:58:14.317251+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":5,"last_scrub":"265'260","last_scrub_stamp":"2021-03-24T21:57:22.158029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:57:22.158029+0000","log_size":327,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6821042,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":30,"num_read_kb":27704,"num_write":37,"num_write_kb":14591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"cd","length":"1"},{"start":"cf","length":"1"}]},{"pgid":"4.19","version":"69'20","reported_seq":"311","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.226853+0000","last_change":"2021-03-24T21:54:54.541380+0000","last_active":"2021-03-24T21:58:12.226853+0000","last_peered":"2021-03-24T21:58:12.226853+0000","last_clean":"2021-03-24T21:58:12.226853+0000","last_became_active":"2021-03-24T21:51:47.405776+0000","last_became_peered":"2021-03-24T21:51:47.405776+0000","last_unstale":"2021-03-24T21:58:12.226853+0000","last_undegraded":"2021-03-24T21:58:12.226853+0000","last_fullsized":"2021-03-24T21:58:12.226853+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'20","last_scrub_stamp":"2021-03-24T21:54:54.541183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:54:54.541183+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":22,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":"296","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.227188+0000","last_change":"2021-03-24T21:55:38.433064+0000","last_active":"2021-03-24T21:58:12.227188+0000","last_peered":"2021-03-24T21:58:12.227188+0000","last_clean":"2021-03-24T21:58:12.227188+0000","last_became_active":"2021-03-24T21:54:38.130163+0000","last_became_peered":"2021-03-24T21:54:38.130163+0000","last_unstale":"2021-03-24T21:58:12.227188+0000","last_undegraded":"2021-03-24T21:58:12.227188+0000","last_fullsized":"2021-03-24T21:58:12.227188+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:38.432918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:38.432918+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"292","reported_epoch":"321","state":"active+clean","last_fresh":"2021-03-24T21:58:11.536261+0000","last_change":"2021-03-24T21:56:34.092326+0000","last_active":"2021-03-24T21:58:11.536261+0000","last_peered":"2021-03-24T21:58:11.536261+0000","last_clean":"2021-03-24T21:58:11.536261+0000","last_became_active":"2021-03-24T21:54:38.120575+0000","last_became_peered":"2021-03-24T21:54:38.120575+0000","last_unstale":"2021-03-24T21:58:11.536261+0000","last_undegraded":"2021-03-24T21:58:11.536261+0000","last_fullsized":"2021-03-24T21:58:11.536261+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:34.092227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:34.092227+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"290","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.227125+0000","last_change":"2021-03-24T21:54:43.482408+0000","last_active":"2021-03-24T21:58:12.227125+0000","last_peered":"2021-03-24T21:58:12.227125+0000","last_clean":"2021-03-24T21:58:12.227125+0000","last_became_active":"2021-03-24T21:54:38.130293+0000","last_became_peered":"2021-03-24T21:54:38.130293+0000","last_unstale":"2021-03-24T21:58:12.227125+0000","last_undegraded":"2021-03-24T21:58:12.227125+0000","last_fullsized":"2021-03-24T21:58:12.227125+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:43.482307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:54:43.482307+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"316'151","reported_seq":"833","reported_epoch":"321","state":"active+clean","last_fresh":"2021-03-24T21:58:11.536441+0000","last_change":"2021-03-24T21:58:06.216029+0000","last_active":"2021-03-24T21:58:11.536441+0000","last_peered":"2021-03-24T21:58:11.536441+0000","last_clean":"2021-03-24T21:58:11.536441+0000","last_became_active":"2021-03-24T21:57:31.167795+0000","last_became_peered":"2021-03-24T21:57:31.167795+0000","last_unstale":"2021-03-24T21:58:11.536441+0000","last_undegraded":"2021-03-24T21:58:11.536441+0000","last_fullsized":"2021-03-24T21:58:11.536441+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":151,"ondisk_log_size":151,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":789995,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":28,"num_read_kb":22370,"num_write":43,"num_write_kb":12527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":3266508,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"cd","length":"1"},{"start":"cf","length":"1"}]},{"pgid":"4.18","version":"69'15","reported_seq":"366","reported_epoch":"321","state":"active+clean","last_fresh":"2021-03-24T21:58:11.535970+0000","last_change":"2021-03-24T21:56:05.966329+0000","last_active":"2021-03-24T21:58:11.535970+0000","last_peered":"2021-03-24T21:58:11.535970+0000","last_clean":"2021-03-24T21:58:11.535970+0000","last_became_active":"2021-03-24T21:54:38.127426+0000","last_became_peered":"2021-03-24T21:54:38.127426+0000","last_unstale":"2021-03-24T21:58:11.535970+0000","last_undegraded":"2021-03-24T21:58:11.535970+0000","last_fullsized":"2021-03-24T21:58:11.535970+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:56:05.966219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:05.966219+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2076,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":"287","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.401049+0000","last_change":"2021-03-24T21:56:44.230467+0000","last_active":"2021-03-24T21:58:12.401049+0000","last_peered":"2021-03-24T21:58:12.401049+0000","last_clean":"2021-03-24T21:58:12.401049+0000","last_became_active":"2021-03-24T21:56:44.229902+0000","last_became_peered":"2021-03-24T21:56:44.229902+0000","last_unstale":"2021-03-24T21:58:12.401049+0000","last_undegraded":"2021-03-24T21:58:12.401049+0000","last_fullsized":"2021-03-24T21:58:12.401049+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:38.294685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:38.294685+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"203","reported_epoch":"323","state":"active+clean","last_fresh":"2021-03-24T21:58:13.250645+0000","last_change":"2021-03-24T21:56:34.003068+0000","last_active":"2021-03-24T21:58:13.250645+0000","last_peered":"2021-03-24T21:58:13.250645+0000","last_clean":"2021-03-24T21:58:13.250645+0000","last_became_active":"2021-03-24T21:55:10.099666+0000","last_became_peered":"2021-03-24T21:55:10.099666+0000","last_unstale":"2021-03-24T21:58:13.250645+0000","last_undegraded":"2021-03-24T21:58:13.250645+0000","last_fullsized":"2021-03-24T21:58:13.250645+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:34.002906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:14.710709+0000","last_clean_scrub_stamp":"2021-03-24T21:56:34.002906+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"286","reported_epoch":"321","state":"active+clean","last_fresh":"2021-03-24T21:58:11.536615+0000","last_change":"2021-03-24T21:57:15.687334+0000","last_active":"2021-03-24T21:58:11.536615+0000","last_peered":"2021-03-24T21:58:11.536615+0000","last_clean":"2021-03-24T21:58:11.536615+0000","last_became_active":"2021-03-24T21:54:38.111248+0000","last_became_peered":"2021-03-24T21:54:38.111248+0000","last_unstale":"2021-03-24T21:58:11.536615+0000","last_undegraded":"2021-03-24T21:58:11.536615+0000","last_fullsized":"2021-03-24T21:58:11.536615+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:15.687165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:57:15.687165+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"320'341","reported_seq":"1310","reported_epoch":"323","state":"active+clean","last_fresh":"2021-03-24T21:58:13.245636+0000","last_change":"2021-03-24T21:58:06.194166+0000","last_active":"2021-03-24T21:58:13.245636+0000","last_peered":"2021-03-24T21:58:13.245636+0000","last_clean":"2021-03-24T21:58:13.245636+0000","last_became_active":"2021-03-24T21:57:33.176110+0000","last_became_peered":"2021-03-24T21:57:33.176110+0000","last_unstale":"2021-03-24T21:58:13.245636+0000","last_undegraded":"2021-03-24T21:58:13.245636+0000","last_fullsized":"2021-03-24T21:58:13.245636+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":5,"last_scrub":"191'150","last_scrub_stamp":"2021-03-24T21:56:01.308050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:56:01.308050+0000","log_size":341,"ondisk_log_size":341,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10997274,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":181,"num_read_kb":195827,"num_write":99,"num_write_kb":35333,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":28585116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"cd","length":"1"},{"start":"cf","length":"1"}]},{"pgid":"7.20","version":"175'22","reported_seq":"409","reported_epoch":"320","state":"active+clean","last_fresh":"2021-03-24T21:58:10.184527+0000","last_change":"2021-03-24T21:58:06.201932+0000","last_active":"2021-03-24T21:58:10.184527+0000","last_peered":"2021-03-24T21:58:10.184527+0000","last_clean":"2021-03-24T21:58:10.184527+0000","last_became_active":"2021-03-24T21:57:28.171436+0000","last_became_peered":"2021-03-24T21:57:28.171436+0000","last_unstale":"2021-03-24T21:58:10.184527+0000","last_undegraded":"2021-03-24T21:58:10.184527+0000","last_fullsized":"2021-03-24T21:58:10.184527+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":6,"last_scrub":"175'22","last_scrub_stamp":"2021-03-24T21:57:32.121490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:57:32.121490+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"cd","length":"1"},{"start":"cf","length":"1"}]},{"pgid":"7.21","version":"320'283","reported_seq":"1113","reported_epoch":"322","state":"active+clean","last_fresh":"2021-03-24T21:58:12.226355+0000","last_change":"2021-03-24T21:58:06.191395+0000","last_active":"2021-03-24T21:58:12.226355+0000","last_peered":"2021-03-24T21:58:12.226355+0000","last_clean":"2021-03-24T21:58:12.226355+0000","last_became_active":"2021-03-24T21:57:32.177566+0000","last_became_peered":"2021-03-24T21:57:32.177566+0000","last_unstale":"2021-03-24T21:58:12.226355+0000","last_undegraded":"2021-03-24T21:58:12.226355+0000","last_fullsized":"2021-03-24T21:58:12.226355+0000","mapping_epoch":281,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":283,"ondisk_log_size":283,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6645281,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":104,"num_read_kb":110442,"num_write":52,"num_write_kb":20170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":10874449,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"cd","length":"1"},{"start":"cf","length":"1"}]}],"pool_stats":[{"poolid":7,"num_pg":36,"stat_sum":{"num_bytes":163661859,"num_objects":73,"num_object_clones":32,"num_object_copies":219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":4,"num_read":1651,"num_read_kb":1701720,"num_write":1171,"num_write_kb":453637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":197,"num_bytes_recovered":103035479,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393879552,"data_stored":462456925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6566,"ondisk_log_size":6566,"up":108,"acting":108,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":386,"num_read_kb":381,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1589,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":5558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1061,"num_read_kb":566,"num_write":542,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":309,"num_bytes_recovered":25168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2027520,"data_stored":72734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":558,"ondisk_log_size":558,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201362,"num_pgs":88,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":97208,"kb_used_data":78704,"kb_used_omap":0,"kb_used_meta":18496,"kb_avail":93647944,"statfs":{"total":95995035648,"available":95895494656,"internally_reserved":0,"allocated":80592896,"data_stored":93060604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":18939904},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:57:54 2021","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":1.018,"15min":0.93799999999999994},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":1.1479999999999999,"5min":12.222,"15min":12.222},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.96699999999999997,"5min":1.0249999999999999,"15min":0.99099999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":2.8149999999999999,"5min":11.506,"15min":11.506},"last":0.995}]},{"osd":1,"last update":"Wed Mar 24 21:57:54 2021","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.99199999999999999,"15min":0.88500000000000001},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.476,"5min":5.7750000000000004,"15min":5.7750000000000004},"last":1.216},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.0109999999999999,"15min":0.98199999999999998},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":4.0149999999999997,"5min":6.0780000000000003,"15min":6.0780000000000003},"last":0.879}]},{"osd":2,"last update":"Wed Mar 24 21:57:54 2021","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":1.169,"15min":1.052},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":2.9329999999999998,"5min":12.135999999999999,"15min":12.135999999999999},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.91700000000000004,"5min":1.1220000000000001,"15min":1.048},"min":{"1min":0.54400000000000004,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.8129999999999999,"5min":12.178000000000001,"15min":12.178000000000001},"last":1.143}]},{"osd":3,"last update":"Wed Mar 24 21:57:54 2021","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":1.173,"15min":1.087},"min":{"1min":0.56299999999999994,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.9219999999999999,"5min":12.529,"15min":12.529},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.2,"15min":1.046},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.347,"5min":12.666,"15min":12.666},"last":1.262}]},{"osd":4,"last update":"Wed Mar 24 21:57:54 2021","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":1.0620000000000001,"15min":0.96799999999999997},"min":{"1min":0.629,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.522,"5min":6.2089999999999996,"15min":6.2089999999999996},"last":0.77400000000000002},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.1739999999999999,"15min":1.107},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.671,"5min":12.244,"15min":12.244},"last":1.034}]},{"osd":5,"last update":"Wed Mar 24 21:57:54 2021","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.153,"15min":1.0269999999999999},"min":{"1min":0.64100000000000001,"5min":0.435,"15min":0.435},"max":{"1min":1.4330000000000001,"5min":5.8840000000000003,"15min":5.8840000000000003},"last":1.0660000000000001},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.1910000000000001,"15min":1.1539999999999999},"min":{"1min":0.628,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.629,"5min":6.3300000000000001,"15min":6.3300000000000001},"last":1.097}]},{"osd":6,"last update":"Wed Mar 24 21:57:54 2021","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.232,"15min":1.1479999999999999},"min":{"1min":0.66600000000000004,"5min":0.66000000000000003,"15min":0.62},"max":{"1min":1.577,"5min":6.1159999999999997,"15min":6.1159999999999997},"last":1.1850000000000001},{"interface":"front","average":{"1min":1.117,"5min":1.3029999999999999,"15min":1.2150000000000001},"min":{"1min":0.60499999999999998,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.744,"5min":12.287000000000001,"15min":12.287000000000001},"last":0.97499999999999998}]}]},{"osd":6,"up_from":0,"seq":201863462999,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":0,"seq":55834574956,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738479,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":78216,"kb_used_data":60736,"kb_used_omap":0,"kb_used_meta":17472,"kb_avail":93666936,"statfs":{"total":95995035648,"available":95914942464,"internally_reserved":0,"allocated":62193664,"data_stored":72575194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":17891328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Mar 24 21:57:47 2021","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.70799999999999996,"15min":0.622},"min":{"1min":0.36499999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.054,"5min":4.8040000000000003,"15min":4.8040000000000003},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.82899999999999996,"15min":0.68200000000000005},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":2.734,"5min":8.8989999999999991,"15min":8.8989999999999991},"last":0.59499999999999997}]},{"osd":2,"last update":"Wed Mar 24 21:58:05 2021","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.78100000000000003,"15min":0.70299999999999996},"min":{"1min":0.35399999999999998,"5min":0.34200000000000003,"15min":0.314},"max":{"1min":1.351,"5min":5.8810000000000002,"15min":5.8810000000000002},"last":0.623},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.84599999999999997,"15min":0.749},"min":{"1min":0.50900000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.462,"5min":4.766,"15min":4.766},"last":0.77500000000000002}]},{"osd":3,"last update":"Wed Mar 24 21:57:22 2021","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.873,"15min":0.82299999999999995},"min":{"1min":0.48699999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.474,"5min":5.9790000000000001,"15min":5.9790000000000001},"last":1.014},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.95999999999999996,"15min":0.93999999999999995},"min":{"1min":0.52100000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.4119999999999999,"5min":6.4580000000000002,"15min":6.4580000000000002},"last":0.89600000000000002}]},{"osd":4,"last update":"Wed Mar 24 21:57:47 2021","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":1.089,"15min":1.093},"min":{"1min":0.79300000000000004,"5min":0.436,"15min":0.436},"max":{"1min":1.444,"5min":5.0330000000000004,"15min":5.0330000000000004},"last":1.123},{"interface":"front","average":{"1min":1.056,"5min":1.1120000000000001,"15min":1.046},"min":{"1min":0.57799999999999996,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":2.8180000000000001,"5min":6.1829999999999998,"15min":6.1829999999999998},"last":1.2989999999999999}]},{"osd":5,"last update":"Wed Mar 24 21:58:05 2021","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.1040000000000001,"15min":1.0740000000000001},"min":{"1min":0.64400000000000002,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.603,"5min":5.9390000000000001,"15min":5.9390000000000001},"last":1.1779999999999999},{"interface":"front","average":{"1min":1.103,"5min":1.167,"15min":1.1739999999999999},"min":{"1min":0.75900000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.645,"5min":5.0590000000000002,"15min":5.0590000000000002},"last":1.2609999999999999}]},{"osd":6,"last update":"Wed Mar 24 21:57:22 2021","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.1499999999999999,"15min":1.0940000000000001},"min":{"1min":0.85199999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.7609999999999999,"5min":5.9930000000000003,"15min":5.9930000000000003},"last":1.0800000000000001},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.212,"15min":1.216},"min":{"1min":0.79900000000000004,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":2.0139999999999998,"5min":4.9960000000000004,"15min":4.9960000000000004},"last":1.367}]},{"osd":7,"last update":"Wed Mar 24 21:57:56 2021","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.212,"15min":1.1819999999999999},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":2.8399999999999999,"5min":6.1319999999999997,"15min":6.1319999999999997},"last":1.423},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.206,"15min":1.1519999999999999},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.65800000000000003},"max":{"1min":1.5720000000000001,"5min":6.0529999999999999,"15min":6.0529999999999999},"last":1.242}]}]},{"osd":2,"up_from":18,"seq":77309411432,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":105184,"kb_used_data":71448,"kb_used_omap":0,"kb_used_meta":33728,"kb_avail":93639968,"statfs":{"total":95995035648,"available":95887327232,"internally_reserved":0,"allocated":73162752,"data_stored":84586669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":34537472},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:58:04 2021","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.68600000000000005,"15min":0.64500000000000002},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":1.133,"5min":1.49,"15min":1.49},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.71099999999999997,"15min":0.69699999999999995},"min":{"1min":0.32400000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.93600000000000005,"5min":2.3119999999999998,"15min":2.3119999999999998},"last":0.76500000000000001}]},{"osd":1,"last update":"Wed Mar 24 21:58:04 2021","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.75,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.2010000000000001,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.77500000000000002,"15min":0.75},"min":{"1min":0.36299999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.2609999999999999,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":0.754}]},{"osd":3,"last update":"Wed Mar 24 21:57:21 2021","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.86799999999999999,"15min":0.83799999999999997},"min":{"1min":0.54400000000000004,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.4870000000000001,"5min":1.9890000000000001,"15min":1.9890000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.90500000000000003,"15min":0.89700000000000002},"min":{"1min":0.45300000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.4950000000000001,"5min":2.0950000000000002,"15min":2.0950000000000002},"last":0.70599999999999996}]},{"osd":4,"last update":"Wed Mar 24 21:57:46 2021","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":1.0780000000000001,"15min":1.0489999999999999},"min":{"1min":0.69999999999999996,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.4419999999999999,"5min":3.3380000000000001,"15min":3.3380000000000001},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.96899999999999997,"5min":1.054,"15min":1.048},"min":{"1min":0.66500000000000004,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.181,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":0.89700000000000002}]},{"osd":5,"last update":"Wed Mar 24 21:58:04 2021","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":1.014,"15min":1.0269999999999999},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.163,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.93000000000000005},{"interface":"front","average":{"1min":1.048,"5min":1.125,"15min":1.147},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.4390000000000001,"5min":2.5859999999999999,"15min":3.1509999999999998},"last":1.105}]},{"osd":6,"last update":"Wed Mar 24 21:57:21 2021","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.139,"15min":1.2230000000000001},"min":{"1min":0.70199999999999996,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.5740000000000001,"5min":2.2810000000000001,"15min":2.2810000000000001},"last":0.84499999999999997},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.111,"15min":1.1479999999999999},"min":{"1min":0.77600000000000002,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":2.254,"5min":2.254,"15min":2.254},"last":1.004}]},{"osd":7,"last update":"Wed Mar 24 21:57:55 2021","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.1899999999999999,"15min":1.218},"min":{"1min":0.88300000000000001,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.4079999999999999,"5min":4.4880000000000004,"15min":4.4880000000000004},"last":0.86199999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.2130000000000001,"15min":1.238},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.70199999999999996},"max":{"1min":1.504,"5min":3.5419999999999998,"15min":3.5419999999999998},"last":1.0549999999999999}]}]},{"osd":3,"up_from":26,"seq":111669149796,"num_pgs":90,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":98132,"kb_used_data":79820,"kb_used_omap":0,"kb_used_meta":18304,"kb_avail":93647020,"statfs":{"total":95995035648,"available":95894548480,"internally_reserved":0,"allocated":81735680,"data_stored":97949961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":18743296},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:57:27 2021","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.87,"15min":0.71399999999999997},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.97399999999999998,"5min":23.846,"15min":23.846},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.93400000000000005,"15min":0.79300000000000004},"min":{"1min":0.38100000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.131,"5min":23.734999999999999,"15min":23.734999999999999},"last":0.59899999999999998}]},{"osd":1,"last update":"Wed Mar 24 21:57:27 2021","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.86499999999999999,"15min":0.78300000000000003},"min":{"1min":0.39400000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.169,"5min":14.324,"15min":14.324},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.85199999999999998,"15min":0.78900000000000003},"min":{"1min":0.47699999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.995,"5min":12.631,"15min":12.631},"last":1.0580000000000001}]},{"osd":2,"last update":"Wed Mar 24 21:57:27 2021","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.91200000000000003,"15min":0.82799999999999996},"min":{"1min":0.55300000000000005,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.0169999999999999,"5min":12.510999999999999,"15min":12.510999999999999},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.92700000000000005,"15min":0.82099999999999995},"min":{"1min":0.46300000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.502,"5min":11.317,"15min":11.317},"last":0.65400000000000003}]},{"osd":4,"last update":"Wed Mar 24 21:57:49 2021","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":1.01,"15min":0.97399999999999998},"min":{"1min":0.50800000000000001,"5min":0.443,"15min":0.443},"max":{"1min":1.4670000000000001,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.995,"5min":1.1499999999999999,"15min":1.0109999999999999},"min":{"1min":0.69999999999999996,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.54,"5min":14.291,"15min":14.291},"last":1.022}]},{"osd":5,"last update":"Wed Mar 24 21:58:02 2021","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.1859999999999999,"15min":1.1040000000000001},"min":{"1min":0.73799999999999999,"5min":0.62,"15min":0.54100000000000004},"max":{"1min":1.5840000000000001,"5min":12.723000000000001,"15min":12.723000000000001},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.099,"15min":1.089},"min":{"1min":0.63900000000000001,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.452,"5min":2.375,"15min":2.375},"last":0.98799999999999999}]},{"osd":6,"last update":"Wed Mar 24 21:57:27 2021","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":1.232,"15min":1.1639999999999999},"min":{"1min":0.65800000000000003,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.629,"5min":14.255000000000001,"15min":14.255000000000001},"last":0.81100000000000005},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.248,"15min":1.1850000000000001},"min":{"1min":0.73299999999999998,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.5029999999999999,"5min":12.666,"15min":12.666},"last":1.1359999999999999}]},{"osd":7,"last update":"Wed Mar 24 21:57:53 2021","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":1.181,"15min":1.111},"min":{"1min":0.71399999999999997,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.54,"5min":2.3479999999999999,"15min":2.3479999999999999},"last":1.097},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.319,"15min":1.2130000000000001},"min":{"1min":0.65800000000000003,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.5580000000000001,"5min":14.727,"15min":14.727},"last":0.90700000000000003}]}]},{"osd":4,"up_from":32,"seq":137438953566,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":72096,"kb_used_data":44312,"kb_used_omap":0,"kb_used_meta":27776,"kb_avail":93673056,"statfs":{"total":95995035648,"available":95921209344,"internally_reserved":0,"allocated":45375488,"data_stored":47271434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":28442624},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:57:43 2021","interfaces":[{"interface":"back","average":{"1min":3.1219999999999999,"5min":1.2889999999999999,"15min":0.99199999999999999},"min":{"1min":0.54400000000000004,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":41.472000000000001,"5min":41.472000000000001,"15min":41.472000000000001},"last":0.64900000000000002},{"interface":"front","average":{"1min":3.3700000000000001,"5min":1.468,"15min":1.0629999999999999},"min":{"1min":0.63200000000000001,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":44.185000000000002,"5min":44.185000000000002,"15min":44.185000000000002},"last":0.77600000000000002}]},{"osd":1,"last update":"Wed Mar 24 21:57:43 2021","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.93000000000000005,"15min":0.92300000000000004},"min":{"1min":0.58899999999999997,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.2629999999999999,"5min":1.3120000000000001,"15min":1.4570000000000001},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.89300000000000002,"15min":0.86499999999999999},"min":{"1min":0.46899999999999997,"5min":0.436,"15min":0.436},"max":{"1min":1.159,"5min":1.873,"15min":1.873},"last":0.86099999999999999}]},{"osd":2,"last update":"Wed Mar 24 21:57:43 2021","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":0.96799999999999997,"15min":0.92200000000000004},"min":{"1min":0.69899999999999995,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.667,"5min":4.4379999999999997,"15min":4.4379999999999997},"last":0.90100000000000002},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.008,"15min":0.94799999999999995},"min":{"1min":0.68799999999999994,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":2.0470000000000002,"5min":2.738,"15min":2.738},"last":0.82199999999999995}]},{"osd":3,"last update":"Wed Mar 24 21:57:43 2021","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":1,"15min":0.96099999999999997},"min":{"1min":0.60199999999999998,"5min":0.439,"15min":0.439},"max":{"1min":2.1459999999999999,"5min":3.2000000000000002,"15min":3.2000000000000002},"last":0.96099999999999997},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.98599999999999999,"15min":0.96499999999999997},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.2969999999999999,"5min":1.474,"15min":1.474},"last":0.75900000000000001}]},{"osd":5,"last update":"Wed Mar 24 21:58:08 2021","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":0.94099999999999995,"15min":0.95199999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.419,"5min":1.6579999999999999,"15min":1.9199999999999999},"last":0.93700000000000006},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.0760000000000001,"15min":1.03},"min":{"1min":0.56000000000000005,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.3759999999999999,"5min":1.607,"15min":2.0760000000000001},"last":0.70499999999999996}]},{"osd":6,"last update":"Wed Mar 24 21:57:23 2021","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.081,"15min":1.071},"min":{"1min":0.84299999999999997,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.6859999999999999,"5min":1.6859999999999999,"15min":1.6859999999999999},"last":0.56000000000000005},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.1020000000000001,"15min":1.0780000000000001},"min":{"1min":0.91400000000000003,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.631,"5min":1.631,"15min":1.631},"last":0.91700000000000004}]},{"osd":7,"last update":"Wed Mar 24 21:57:48 2021","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.1439999999999999,"15min":1.179},"min":{"1min":0.81899999999999995,"5min":0.624,"15min":0.58499999999999996},"max":{"1min":1.444,"5min":1.6830000000000001,"15min":2.1099999999999999},"last":0.80800000000000005},{"interface":"front","average":{"1min":1.234,"5min":1.1870000000000001,"15min":1.2},"min":{"1min":0.79400000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.538,"5min":1.702,"15min":2.0009999999999999},"last":0.83999999999999997}]}]},{"osd":5,"up_from":39,"seq":167503724635,"num_pgs":90,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":84548,"kb_used_data":57212,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":93660604,"statfs":{"total":95995035648,"available":95908458496,"internally_reserved":0,"allocated":58585088,"data_stored":62493255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:58:08 2021","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.80700000000000005,"15min":0.86199999999999999},"min":{"1min":0.372,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.0289999999999999,"5min":1.762,"15min":2.29},"last":1.119},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.87,"15min":0.89000000000000001},"min":{"1min":0.45600000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.5800000000000001,"5min":1.5800000000000001,"15min":3.1779999999999999},"last":0.76900000000000002}]},{"osd":1,"last update":"Wed Mar 24 21:58:08 2021","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.86899999999999999,"15min":0.90800000000000003},"min":{"1min":0.45300000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.264,"5min":1.6479999999999999,"15min":2.8690000000000002},"last":1.206},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.89300000000000002,"15min":0.92000000000000004},"min":{"1min":0.41499999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.1930000000000001,"5min":2.0030000000000001,"15min":2.0030000000000001},"last":0.91600000000000004}]},{"osd":2,"last update":"Wed Mar 24 21:58:08 2021","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.95899999999999996,"15min":0.92000000000000004},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.272,"5min":6.5199999999999996,"15min":6.5199999999999996},"last":0.996},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.93400000000000005,"15min":0.92300000000000004},"min":{"1min":0.65500000000000003,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.613,"5min":5.5010000000000003,"15min":5.8479999999999999},"last":0.67600000000000005}]},{"osd":3,"last update":"Wed Mar 24 21:58:08 2021","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":0.95599999999999996,"15min":0.94399999999999995},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.744,"5min":2.2000000000000002,"15min":3.1259999999999999},"last":1.038},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.97199999999999998,"15min":0.95699999999999996},"min":{"1min":0.61299999999999999,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.175,"5min":1.843,"15min":2.6850000000000001},"last":1.26}]},{"osd":4,"last update":"Wed Mar 24 21:58:08 2021","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":1.02,"15min":0.89100000000000001},"min":{"1min":0.56399999999999995,"5min":0.376,"15min":0.376},"max":{"1min":1.6459999999999999,"5min":2.2959999999999998,"15min":2.2959999999999998},"last":0.82299999999999995},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.0880000000000001,"15min":0.96099999999999997},"min":{"1min":0.74199999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.4910000000000001,"5min":2.629,"15min":2.629},"last":0.85199999999999998}]},{"osd":6,"last update":"Wed Mar 24 21:57:26 2021","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":1.0509999999999999,"15min":0.92300000000000004},"min":{"1min":0.67000000000000004,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.3149999999999999,"5min":2.3879999999999999,"15min":2.3879999999999999},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.107,"15min":1.0049999999999999},"min":{"1min":0.55900000000000005,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.458,"5min":2.4660000000000002,"15min":2.4660000000000002},"last":1.179}]},{"osd":7,"last update":"Wed Mar 24 21:57:55 2021","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.1519999999999999,"15min":1.0800000000000001},"min":{"1min":0.61299999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.698,"5min":2.548,"15min":2.548},"last":1.093},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.1719999999999999,"15min":1.04},"min":{"1min":0.71499999999999997,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.389,"5min":2.9889999999999999,"15min":2.9889999999999999},"last":1.23}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":14134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":7234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":13448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":352256,"data_stored":12784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":360448,"data_stored":13024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":9948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":58957824,"data_stored":70955880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7032832,"data_stored":6977532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":70160384,"data_stored":82989114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":78544896,"data_stored":96339666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":42201088,"data_stored":45661742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":56856576,"data_stored":62886659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":80125952,"data_stored":96646332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-03-24T21:58:18.236 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd pool set unique_pool_0 pgp_num 36 2021-03-24T21:58:18.270 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:18.282 INFO:tasks.rados.rados.0.smithi125.stdout:1151: done (1 left) 2021-03-24T21:58:18.283 INFO:tasks.rados.rados.0.smithi125.stdout:1152: write oid 41 current snap is 113 2021-03-24T21:58:18.283 INFO:tasks.rados.rados.0.smithi125.stdout:1152: seq_num 428 ranges {793396=722340,2128999=690812} 2021-03-24T21:58:18.295 INFO:tasks.rados.rados.0.smithi125.stdout:1152: writing smithi12556470-41 from 793396 to 1515736 tid 1 2021-03-24T21:58:18.302 INFO:tasks.rados.rados.0.smithi125.stdout:1152: writing smithi12556470-41 from 2128999 to 2819811 tid 2 2021-03-24T21:58:18.303 INFO:tasks.rados.rados.0.smithi125.stdout:1150: finishing write tid 1 to smithi12556470-23 2021-03-24T21:58:18.303 INFO:tasks.rados.rados.0.smithi125.stdout:1150: finishing write tid 2 to smithi12556470-23 2021-03-24T21:58:18.304 INFO:tasks.rados.rados.0.smithi125.stdout:1150: finishing write tid 3 to smithi12556470-23 2021-03-24T21:58:18.304 INFO:tasks.rados.rados.0.smithi125.stdout:1150: finishing write tid 4 to smithi12556470-23 2021-03-24T21:58:18.304 INFO:tasks.rados.rados.0.smithi125.stdout:1150: finishing write tid 5 to smithi12556470-23 2021-03-24T21:58:18.304 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 23 v 279 (ObjNum 427 snap 113 seq_num 427) dirty exists 2021-03-24T21:58:18.304 INFO:tasks.rados.rados.0.smithi125.stdout:1150: left oid 23 (ObjNum 427 snap 113 seq_num 427) 2021-03-24T21:58:18.305 INFO:tasks.rados.rados.0.smithi125.stdout:1150: done (1 left) 2021-03-24T21:58:18.305 INFO:tasks.rados.rados.0.smithi125.stdout:1153: snap_remove snap 110 2021-03-24T21:58:18.325 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:18.327 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:58:18.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:18.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:58:18.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:18 smithi142 conmon[20125]: cluster 2021-03-24T21:58:16 2021-03-24T21:58:18.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:18 smithi142 conmon[20125]: .792278+0000 mgr.y (mgr.14144) 432 : cluster [DBG] pgmap v652: 168 pgs: 168 active+clean; 156 MiB data, 527 MiB used, 536 GiB / 536 GiB avail; 16 MiB/s rd, 6.4 MiB/s wr, 36 op/s 2021-03-24T21:58:18.384 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2021-03-24T21:58:18.384 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T21:58:18.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:18 smithi125 conmon[19936]: cluster 2021-03-24T21:58:16 2021-03-24T21:58:18.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:18 smithi125 conmon[19936]: .792278+0000 mgr.y (mgr.14144) 432 : cluster [DBG] pgmap v652: 168 pgs: 168 active+clean; 156 MiB data, 527 MiB used, 536 GiB / 536 GiB avail; 16 MiB/s rd, 6.4 MiB/s wr, 36 op/s 2021-03-24T21:58:18.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:18 smithi125 conmon[24927]: cluster 2021-03-24T21:58:16. 2021-03-24T21:58:18.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:18 smithi125 conmon[24927]: 792278+0000 mgr.y (mgr.14144) 432 : cluster [DBG] pgmap v652: 168 pgs: 168 active+clean; 156 MiB data, 527 MiB used, 536 GiB / 536 GiB avail; 16 MiB/s rd, 6.4 MiB/s wr, 36 op/s 2021-03-24T21:58:18.429 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:18.481 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:18.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:18.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:58:18.587 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:18.645 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:18.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:18.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:58:18.651 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:58:18.750 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:18.809 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:18.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:18.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:18.913 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:18.971 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:18.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:18.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:58:19.076 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:19.134 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:19.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:19.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:58:19.240 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:19.299 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:19.303 INFO:tasks.rados.rados.0.smithi125.stdout:1153: done (1 left) 2021-03-24T21:58:19.303 INFO:tasks.rados.rados.0.smithi125.stdout:1154: copy_from oid 45 from oid 20 current snap is 113 2021-03-24T21:58:19.303 INFO:tasks.rados.rados.0.smithi125.stdout:1155: delete oid 3 current snap is 113 2021-03-24T21:58:19.304 INFO:tasks.rados.rados.0.smithi125.stdout:1152: finishing write tid 1 to smithi12556470-41 2021-03-24T21:58:19.304 INFO:tasks.rados.rados.0.smithi125.stdout:1152: finishing write tid 2 to smithi12556470-41 2021-03-24T21:58:19.304 INFO:tasks.rados.rados.0.smithi125.stdout:1152: finishing write tid 3 to smithi12556470-41 2021-03-24T21:58:19.304 INFO:tasks.rados.rados.0.smithi125.stdout:1152: finishing write tid 4 to smithi12556470-41 2021-03-24T21:58:19.304 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 41 v 339 (ObjNum 428 snap 113 seq_num 428) dirty exists 2021-03-24T21:58:19.305 INFO:tasks.rados.rados.0.smithi125.stdout:1152: left oid 41 (ObjNum 428 snap 113 seq_num 428) 2021-03-24T21:58:19.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:19.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:19.308 INFO:tasks.rados.rados.0.smithi125.stdout:1152: done (2 left) 2021-03-24T21:58:19.308 INFO:tasks.rados.rados.0.smithi125.stdout:1155: done (1 left) 2021-03-24T21:58:19.308 INFO:tasks.rados.rados.0.smithi125.stdout:1156: snap_create 2021-03-24T21:58:19.318 INFO:teuthology.orchestra.run.smithi125.stderr:noscrub is unset 2021-03-24T21:58:19.335 INFO:tasks.rados.rados.0.smithi125.stdout:1154: finishing copy_from to smithi12556470-45 2021-03-24T21:58:19.335 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 45 v 212 (ObjNum 380 snap 99 seq_num 380) dirty exists 2021-03-24T21:58:19.335 INFO:tasks.rados.rados.0.smithi125.stdout:1154: finishing copy_from racing read to smithi12556470-45 2021-03-24T21:58:19.341 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.7 dump_blocked_ops 2021-03-24T21:58:19.409 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:19.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:19 smithi125 conmon[19936]: cluster 2021-03-24T21:58:18.280421+0000 mon.a (mon.0) 2021-03-24T21:58:19.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:19 smithi125 conmon[19936]: 1013 : cluster [DBG] osdmap e328: 8 total, 8 up, 6 in 2021-03-24T21:58:19.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:19 smithi125 conmon[19936]: audit 2021-03-24T21:58:18.872637+0000 mon.a (mon.0) 1014 : audit [INF] from='client.? v1:172.21.15.125:0/273197076' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T21:58:19.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:19 smithi125 conmon[24927]: cluster 2021-03-24T21:58:19.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:19 smithi125 conmon[24927]: 2021-03-24T21:58:19.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:19 smithi125 conmon[24927]: 2021-03-24T21: 2021-03-24T21:58:19.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:19 smithi125 conmon[24927]: 58:18.280421+0000 mon.a (mon.0) 1013 : cluster [DBG] osdmap e328: 8 total, 8 up, 6 in 2021-03-24T21:58:19.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:19 smithi125 conmon[24927]: audit 2021- 2021-03-24T21:58:19.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:19 smithi125 conmon[24927]: 03-24T21:58:18.872637+0000 mon.a ( 2021-03-24T21:58:19.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:19 smithi125 conmon[24927]: mon.0) 1014 : audit [INF] from='client.? v1:172.21.15.125:0/273197076' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T21:58:19.459 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:19.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:19.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:19.565 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:19.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:19 smithi142 conmon[20125]: cluster 2021-03-24T21:58:18 2021-03-24T21:58:19.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:19 smithi142 conmon[20125]: .280421+0000 mon.a (mon.0) 1013 : cluster [DBG] osdmap e328: 8 total, 8 up, 6 in 2021-03-24T21:58:19.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:19 smithi142 conmon[20125]: audit 2021-03-24T21:58:18.872637+0000 2021-03-24T21:58:19.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:19 smithi142 conmon[20125]: mon.a (mon.0) 1014 : audit [INF] from='client.? v1:172.21.15.125:0/273197076' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T21:58:19.626 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:19.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:19.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:19.632 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T21:58:19.733 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:19.789 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:19.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:19.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:58:19.893 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:19.947 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:19.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:19.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:58:20.052 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:20.110 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:20.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:20.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:20.184 INFO:tasks.rados.rados.0.smithi125.stdout:1154: done (1 left) 2021-03-24T21:58:20.185 INFO:tasks.rados.rados.0.smithi125.stdout:1156: done (0 left) 2021-03-24T21:58:20.185 INFO:tasks.rados.rados.0.smithi125.stdout:1157: delete oid 37 current snap is 114 2021-03-24T21:58:20.193 INFO:tasks.rados.rados.0.smithi125.stdout:1157: done (0 left) 2021-03-24T21:58:20.193 INFO:tasks.rados.rados.0.smithi125.stdout:1158: read oid 28 snap 107 2021-03-24T21:58:20.193 INFO:tasks.rados.rados.0.smithi125.stdout:1158: expect (ObjNum 349 snap 89 seq_num 349) 2021-03-24T21:58:20.194 INFO:tasks.rados.rados.0.smithi125.stdout:1159: snap_create 2021-03-24T21:58:20.202 INFO:tasks.rados.rados.0.smithi125.stdout:1158: expect (ObjNum 349 snap 89 seq_num 349) 2021-03-24T21:58:20.215 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:20.270 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:20.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:20.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:58:20.375 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:58:20.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:20 smithi125 conmon[19936]: cluster 2021-03-24T21:58:18. 2021-03-24T21:58:20.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:20 smithi125 conmon[19936]: 793420+0000 mgr.y (mgr.14144) 433 : cluster [DBG] pgmap v655: 168 pgs: 168 active+clean; 156 MiB data, 534 MiB used, 536 GiB / 536 GiB avail; 3.7 MiB/s rd, 1 op/s 2021-03-24T21:58:20.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:20 smithi125 conmon[19936]: cluster 2021-03-24T21:58:20.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:20 smithi125 conmon[19936]: 2021-03-24T21:58:19.187266+0000 osd.6 (osd. 2021-03-24T21:58:20.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:20 smithi125 conmon[19936]: 6) 25 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:58:20.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:20 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T21:58:20.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:20 smithi125 conmon[19936]: T21:58:19.187423+0000 osd.6 (osd.6) 2021-03-24T21:58:20.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:20 smithi125 conmon[19936]: 26 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:58:20.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:20 smithi125 conmon[19936]: cluster 2021-03-24T21:58: 2021-03-24T21:58:20.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:20 smithi125 conmon[19936]: 19.288208+0000 mon.a (mon.0) 1015 : cluster [WRN] 2021-03-24T21:58:20.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:20 smithi125 conmon[19936]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:58:20.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:20 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:58:20.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:20 smithi125 conmon[19936]: 03-24T21:58:19.300600+0000 mon.a (mon. 2021-03-24T21:58:20.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:20 smithi125 conmon[19936]: 0) 1016 : audit [INF] from='client.? v1:172.21.15.125:0/273197076' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T21:58:20.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:20 smithi125 conmon[19936]: cluster 2021-03-24T21:58 2021-03-24T21:58:20.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:20 smithi125 conmon[19936]: :19.300682+0000 mon.a (mon.0) 1017 : cluster [DBG] 2021-03-24T21:58:20.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:20 smithi125 conmon[19936]: osdmap e329: 8 total, 8 up, 6 in 2021-03-24T21:58:20.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:20 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:58:20.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:20 smithi125 conmon[19936]: -03-24T21:58:20.182107+0000 mon.a (mon.0) 2021-03-24T21:58:20.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:20 smithi125 conmon[19936]: 1018 : cluster [DBG] osdmap e330: 8 total, 8 up, 6 in 2021-03-24T21:58:20.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:20 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:58:20.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:20 smithi125 conmon[24927]: 58:18.793420+0000 mgr.y (mgr.14144) 433 : cluster [DBG] pgmap v655: 168 pgs: 168 active+clean; 156 MiB data, 534 MiB used, 536 GiB / 536 GiB avail; 3.7 MiB/s rd, 1 op/s 2021-03-24T21:58:20.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:20 smithi125 conmon[24927]: cluster 2021-03-24T21:58:19. 2021-03-24T21:58:20.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:20 smithi125 conmon[24927]: 187266+0000 osd.6 (osd.6) 25 : cluster [DBG] 2021-03-24T21:58:20.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:20 smithi125 conmon[24927]: purged_snaps scrub starts 2021-03-24T21:58:20.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:20 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:58:20.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:20 smithi125 conmon[24927]: 58:19.187423+0000 osd.6 (osd.6) 2021-03-24T21:58:20.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:20 smithi125 conmon[24927]: 26 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:58:20.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:20 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:58:20.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:20 smithi125 conmon[24927]: -03-24T21:58:19.288208+0000 mon.a 2021-03-24T21:58:20.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:20 smithi125 conmon[24927]: (mon.0) 1015 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:58:20.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:20 smithi125 conmon[24927]: audit 2021- 2021-03-24T21:58:20.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:20 smithi125 conmon[24927]: 03-24T21:58:19.300600+0000 mon.a 2021-03-24T21:58:20.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:20 smithi125 conmon[24927]: (mon.0) 1016 : audit [INF] from='client.? v1:172.21.15.125:0/273197076' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T21:58:20.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:20 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:58:20.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:20 smithi125 conmon[24927]: -24T21:58:19.300682+0000 mon.a 2021-03-24T21:58:20.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:20 smithi125 conmon[24927]: (mon.0) 1017 : cluster [DBG] osdmap e329: 8 total, 8 up, 6 in 2021-03-24T21:58:20.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:20 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:58:20.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:20 smithi125 conmon[24927]: 03-24T21:58:20.182107+0000 mon.a 2021-03-24T21:58:20.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:20 smithi125 conmon[24927]: (mon.0) 1018 : cluster [DBG] osdmap e330: 8 total, 8 up, 6 in 2021-03-24T21:58:20.435 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:58:20.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:20.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:58:20.440 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:20 smithi142 conmon[20125]: cluster 2021-03-24T21:58:18.793420+0000 mgr.y (mgr.14144) 2021-03-24T21:58:20.440 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:20 smithi142 conmon[20125]: 433 : cluster [DBG] pgmap v655: 168 pgs: 168 active+clean; 156 MiB data, 534 MiB used, 536 GiB / 536 GiB avail; 3.7 MiB/s rd, 1 op/s 2021-03-24T21:58:20.441 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:20 smithi142 conmon[20125]: cluster 2021-03-24T21:58:19.187266+0000 osd.6 (osd.6) 25 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:58:20.441 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:20 smithi142 conmon[20125]: cluster 2021-03-24T21:58:19.187423+0000 2021-03-24T21:58:20.441 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:20 smithi142 conmon[20125]: osd.6 (osd.6) 26 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:58:20.441 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:20 smithi142 conmon[20125]: cluster 2021-03-24T21:58:19.288208+0000 mon.a (mon.0) 1015 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:58:20.441 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:20 smithi142 conmon[20125]: audit 2021-03-24 2021-03-24T21:58:20.442 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:20 smithi142 conmon[20125]: T21:58:19.300600+0000 mon.a (mon.0) 1016 : audit [INF] from='client.? v1:172.21.15.125:0/273197076' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T21:58:20.442 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:20 smithi142 conmon[20125]: cluster 2021-03-24T21:58:19.300682+0000 mon.a (mon.0 2021-03-24T21:58:20.442 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:20 smithi142 conmon[20125]: ) 1017 : cluster [DBG] osdmap e329: 8 total, 8 up, 6 in 2021-03-24T21:58:20.442 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:20 smithi142 conmon[20125]: cluster 2021-03-24T21:58:20.182107+0000 mon.a (mon.0) 1018 : cluster [DBG] osdmap e330: 8 total, 8 up, 6 in 2021-03-24T21:58:20.541 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:20.596 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:20.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:20.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:58:20.702 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:20.756 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:20.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:20.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:20.863 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:20.922 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:20.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:20.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:58:21.029 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:21.088 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:21.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:21.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:58:21.195 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:58:21.198 INFO:tasks.rados.rados.0.smithi125.stdout:1158: done (1 left) 2021-03-24T21:58:21.198 INFO:tasks.rados.rados.0.smithi125.stdout:1159: done (0 left) 2021-03-24T21:58:21.198 INFO:tasks.rados.rados.0.smithi125.stdout:1160: delete oid 44 current snap is 115 2021-03-24T21:58:21.203 INFO:tasks.rados.rados.0.smithi125.stdout:1160: done (0 left) 2021-03-24T21:58:21.203 INFO:tasks.rados.rados.0.smithi125.stdout:1161: rollback oid 32 current snap is 115 2021-03-24T21:58:21.203 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 32 to 114 2021-03-24T21:58:21.204 INFO:tasks.rados.rados.0.smithi125.stdout:1162: write (excl) oid 24 current snap is 115 2021-03-24T21:58:21.204 INFO:tasks.rados.rados.0.smithi125.stdout:1162: seq_num 432 ranges {460624=623336,1846574=698150,3163674=96950} 2021-03-24T21:58:21.209 INFO:tasks.rados.rados.0.smithi125.stdout:1162: writing smithi12556470-24 from 460624 to 1083960 tid 1 2021-03-24T21:58:21.218 INFO:tasks.rados.rados.0.smithi125.stdout:1162: writing smithi12556470-24 from 1846574 to 2544724 tid 2 2021-03-24T21:58:21.222 INFO:tasks.rados.rados.0.smithi125.stdout:1162: writing smithi12556470-24 from 3163674 to 3260624 tid 3 2021-03-24T21:58:21.222 INFO:tasks.rados.rados.0.smithi125.stdout:1163: write (excl) oid 12 current snap is 115 2021-03-24T21:58:21.222 INFO:tasks.rados.rados.0.smithi125.stdout:1161: finishing rollback tid 0 to smithi12556470-32 2021-03-24T21:58:21.223 INFO:tasks.rados.rados.0.smithi125.stdout:1163: seq_num 433 ranges {477431=698585,1905056=441808,3074744=202687} 2021-03-24T21:58:21.228 INFO:tasks.rados.rados.0.smithi125.stdout:1163: writing smithi12556470-12 from 477431 to 1176016 tid 1 2021-03-24T21:58:21.235 INFO:tasks.rados.rados.0.smithi125.stdout:1163: writing smithi12556470-12 from 1905056 to 2346864 tid 2 2021-03-24T21:58:21.241 INFO:tasks.rados.rados.0.smithi125.stdout:1163: writing smithi12556470-12 from 3074744 to 3277431 tid 3 2021-03-24T21:58:21.241 INFO:tasks.rados.rados.0.smithi125.stdout:1164: copy_from oid 37 from oid 3 current snap is 115 2021-03-24T21:58:21.241 INFO:tasks.rados.rados.0.smithi125.stdout:1161: finishing rollback tid 1 to smithi12556470-32 2021-03-24T21:58:21.242 INFO:tasks.rados.rados.0.smithi125.stdout:1165: rollback oid 6 current snap is 115 2021-03-24T21:58:21.242 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 6 to 107 2021-03-24T21:58:21.242 INFO:tasks.rados.rados.0.smithi125.stdout:1161: finishing rollback tid 2 to smithi12556470-32 2021-03-24T21:58:21.242 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 32 v 363 (ObjNum 370 snap 95 seq_num 370) dirty exists 2021-03-24T21:58:21.243 INFO:tasks.rados.rados.0.smithi125.stdout:1162: finishing write tid 1 to smithi12556470-24 2021-03-24T21:58:21.243 INFO:tasks.rados.rados.0.smithi125.stdout:1161: done (4 left) 2021-03-24T21:58:21.243 INFO:tasks.rados.rados.0.smithi125.stdout:1166: write oid 36 current snap is 115 2021-03-24T21:58:21.243 INFO:tasks.rados.rados.0.smithi125.stdout:1162: finishing write tid 2 to smithi12556470-24 2021-03-24T21:58:21.244 INFO:tasks.rados.rados.0.smithi125.stdout:1162: finishing write tid 3 to smithi12556470-24 2021-03-24T21:58:21.244 INFO:tasks.rados.rados.0.smithi125.stdout:1166: seq_num 434 ranges {494238=490187,1563538=701819,2818520=475718} 2021-03-24T21:58:21.245 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:58:21.246 INFO:tasks.rados.rados.0.smithi125.stdout:1166: writing smithi12556470-36 from 494238 to 984425 tid 1 2021-03-24T21:58:21.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:21.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:58:21.254 INFO:tasks.rados.rados.0.smithi125.stdout:1166: writing smithi12556470-36 from 1563538 to 2265357 tid 2 2021-03-24T21:58:21.260 INFO:tasks.rados.rados.0.smithi125.stdout:1166: writing smithi12556470-36 from 2818520 to 3294238 tid 3 2021-03-24T21:58:21.260 INFO:tasks.rados.rados.0.smithi125.stdout:1167: write oid 50 current snap is 115 2021-03-24T21:58:21.260 INFO:tasks.rados.rados.0.smithi125.stdout:1162: finishing write tid 4 to smithi12556470-24 2021-03-24T21:58:21.261 INFO:tasks.rados.rados.0.smithi125.stdout:1167: seq_num 435 ranges {511045=565436,1622020=445477,2729590=556855} 2021-03-24T21:58:21.266 INFO:tasks.rados.rados.0.smithi125.stdout:1167: writing smithi12556470-50 from 511045 to 1076481 tid 1 2021-03-24T21:58:21.272 INFO:tasks.rados.rados.0.smithi125.stdout:1167: writing smithi12556470-50 from 1622020 to 2067497 tid 2 2021-03-24T21:58:21.279 INFO:tasks.rados.rados.0.smithi125.stdout:1167: writing smithi12556470-50 from 2729590 to 3286445 tid 3 2021-03-24T21:58:21.280 INFO:tasks.rados.rados.0.smithi125.stdout:1168: snap_create 2021-03-24T21:58:21.280 INFO:tasks.rados.rados.0.smithi125.stdout:1162: finishing write tid 5 to smithi12556470-24 2021-03-24T21:58:21.280 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 24 v 349 (ObjNum 432 snap 115 seq_num 432) dirty exists 2021-03-24T21:58:21.281 INFO:tasks.rados.rados.0.smithi125.stdout:1162: left oid 24 (ObjNum 432 snap 115 seq_num 432) 2021-03-24T21:58:21.281 INFO:tasks.rados.rados.0.smithi125.stdout:1163: finishing write tid 1 to smithi12556470-12 2021-03-24T21:58:21.281 INFO:tasks.rados.rados.0.smithi125.stdout:1163: finishing write tid 2 to smithi12556470-12 2021-03-24T21:58:21.281 INFO:tasks.rados.rados.0.smithi125.stdout:1164: finishing copy_from to smithi12556470-37 2021-03-24T21:58:21.281 INFO:tasks.rados.rados.0.smithi125.stdout:1164: got expected ENOENT (src dne) 2021-03-24T21:58:21.282 INFO:tasks.rados.rados.0.smithi125.stdout:1164: finishing copy_from racing read to smithi12556470-37 2021-03-24T21:58:21.282 INFO:tasks.rados.rados.0.smithi125.stdout:1163: finishing write tid 3 to smithi12556470-12 2021-03-24T21:58:21.282 INFO:tasks.rados.rados.0.smithi125.stdout:1163: finishing write tid 4 to smithi12556470-12 2021-03-24T21:58:21.282 INFO:tasks.rados.rados.0.smithi125.stdout:1163: finishing write tid 5 to smithi12556470-12 2021-03-24T21:58:21.282 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 12 v 299 (ObjNum 433 snap 115 seq_num 433) dirty exists 2021-03-24T21:58:21.283 INFO:tasks.rados.rados.0.smithi125.stdout:1163: left oid 12 (ObjNum 433 snap 115 seq_num 433) 2021-03-24T21:58:21.283 INFO:tasks.rados.rados.0.smithi125.stdout:1165: finishing rollback tid 1 to smithi12556470-6 2021-03-24T21:58:21.283 INFO:tasks.rados.rados.0.smithi125.stdout:1165: finishing rollback tid 2 to smithi12556470-6 2021-03-24T21:58:21.283 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 6 v 301 (ObjNum 123 snap 23 seq_num 123) dirty exists 2021-03-24T21:58:21.283 INFO:tasks.rados.rados.0.smithi125.stdout:1166: finishing write tid 1 to smithi12556470-36 2021-03-24T21:58:21.284 INFO:tasks.rados.rados.0.smithi125.stdout:1166: finishing write tid 2 to smithi12556470-36 2021-03-24T21:58:21.284 INFO:tasks.rados.rados.0.smithi125.stdout:1166: finishing write tid 3 to smithi12556470-36 2021-03-24T21:58:21.284 INFO:tasks.rados.rados.0.smithi125.stdout:1166: finishing write tid 4 to smithi12556470-36 2021-03-24T21:58:21.284 INFO:tasks.rados.rados.0.smithi125.stdout:1166: finishing write tid 5 to smithi12556470-36 2021-03-24T21:58:21.284 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 36 v 329 (ObjNum 434 snap 115 seq_num 434) dirty exists 2021-03-24T21:58:21.285 INFO:tasks.rados.rados.0.smithi125.stdout:1166: left oid 36 (ObjNum 434 snap 115 seq_num 434) 2021-03-24T21:58:21.285 INFO:tasks.rados.rados.0.smithi125.stdout:1167: finishing write tid 1 to smithi12556470-50 2021-03-24T21:58:21.285 INFO:tasks.rados.rados.0.smithi125.stdout:1167: finishing write tid 2 to smithi12556470-50 2021-03-24T21:58:21.286 INFO:tasks.rados.rados.0.smithi125.stdout:1167: finishing write tid 3 to smithi12556470-50 2021-03-24T21:58:21.286 INFO:tasks.rados.rados.0.smithi125.stdout:1167: finishing write tid 4 to smithi12556470-50 2021-03-24T21:58:21.286 INFO:tasks.rados.rados.0.smithi125.stdout:1167: finishing write tid 5 to smithi12556470-50 2021-03-24T21:58:21.287 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 50 v 248 (ObjNum 435 snap 115 seq_num 435) dirty exists 2021-03-24T21:58:21.287 INFO:tasks.rados.rados.0.smithi125.stdout:1167: left oid 50 (ObjNum 435 snap 115 seq_num 435) 2021-03-24T21:58:21.308 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:21 smithi125 conmon[24927]: cluster 2021-03-24T21:58:21.308 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:21 smithi125 conmon[24927]: 2021-03-24T21:58:21.308 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:21 smithi125 conmon[24927]: 2021-03-24T21:58: 2021-03-24T21:58:21.308 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:21 smithi125 conmon[24927]: 20.324550+0000 osd.3 (osd.3) 77 : cluster [DBG] 5.14 scrub ok 2021-03-24T21:58:21.309 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:21 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:58:21.309 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:21 smithi125 conmon[24927]: -03-24T21:58:20.372915+0000 osd.2 (osd.2 2021-03-24T21:58:21.309 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:21 smithi125 conmon[24927]: ) 62 : cluster [DBG] 7.8 scrub ok 2021-03-24T21:58:21.309 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:21 smithi125 conmon[24927]: cluster 2021-03-24T21:58:20.702376+0000 2021-03-24T21:58:21.309 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:21 smithi125 conmon[24927]: osd.4 (osd.4) 84 : cluster [DBG] 4.5 scrub ok 2021-03-24T21:58:21.310 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:21 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:58:21.310 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:21 smithi125 conmon[24927]: -24T21:58:21.195759+0000 mon.a (mon.0 2021-03-24T21:58:21.310 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:21 smithi125 conmon[24927]: ) 1019 : cluster [DBG] osdmap e331: 8 total, 8 up, 6 in 2021-03-24T21:58:21.350 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:21.363 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:21 smithi125 conmon[19936]: cluster 2021-03-24T21:58: 2021-03-24T21:58:21.363 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:21 smithi125 conmon[19936]: 20.324550+0000 osd.3 (osd.3) 77 : cluster [DBG] 5.14 scrub ok 2021-03-24T21:58:21.363 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:21 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:58:21.364 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:21 smithi125 conmon[19936]: 24T21:58:20.372915+0000 osd.2 (osd.2) 62 : 2021-03-24T21:58:21.364 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:21 smithi125 conmon[19936]: cluster [DBG] 7.8 scrub ok 2021-03-24T21:58:21.364 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:21 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:58:21.364 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:21 smithi125 conmon[19936]: :58:20.702376+0000 osd.4 (osd.4 2021-03-24T21:58:21.365 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:21 smithi125 conmon[19936]: ) 84 : cluster [DBG] 4.5 scrub ok 2021-03-24T21:58:21.365 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:21 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:58:21.365 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:21 smithi125 conmon[19936]: 21:58:21.195759+0000 mon.a (mon.0) 1019 2021-03-24T21:58:21.365 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:21 smithi125 conmon[19936]: : cluster [DBG] osdmap e331: 8 total, 8 up, 6 in 2021-03-24T21:58:21.405 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:21.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:21.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:58:21.511 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:21.565 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:21.567 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:21 smithi142 conmon[20125]: cluster 2021-03-24T21:58:20.324550+0000 osd.3 2021-03-24T21:58:21.568 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:21 smithi142 conmon[20125]: (osd.3) 77 : cluster [DBG] 5.14 scrub ok 2021-03-24T21:58:21.568 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:21 smithi142 conmon[20125]: cluster 2021-03-24T21:58 2021-03-24T21:58:21.568 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:21 smithi142 conmon[20125]: :20.372915+0000 osd.2 (osd.2) 62 : 2021-03-24T21:58:21.568 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:21 smithi142 conmon[20125]: cluster [DBG] 7.8 scrub ok 2021-03-24T21:58:21.569 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:21 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:58:21.569 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:21 smithi142 conmon[20125]: 03-24T21:58:20.702376+0000 osd.4 2021-03-24T21:58:21.569 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:21 smithi142 conmon[20125]: (osd.4) 84 : cluster [DBG] 4.5 scrub ok 2021-03-24T21:58:21.569 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:21 smithi142 conmon[20125]: cluster 2021-03-24T21:58:21.569 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:21 smithi142 conmon[20125]: 2021- 2021-03-24T21:58:21.570 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:21 smithi142 conmon[20125]: 03-24T21:58:21.195759+0000 2021-03-24T21:58:21.570 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:21 smithi142 conmon[20125]: mon.a (mon.0) 1019 2021-03-24T21:58:21.570 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:21 smithi142 conmon[20125]: : cluster [DBG] osdmap e331: 8 total, 8 up, 6 in 2021-03-24T21:58:21.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:21.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:58:21.672 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:21.736 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:21.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:21.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:58:21.769 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2021-03-24T21:58:21.769 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T21:58:21.843 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:21.892 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:21.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:21.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:21.998 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:22.053 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:22.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:22.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:22.136 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T21:58:22.158 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T21:58:22.159 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:22.180 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T21:58:22.200 INFO:tasks.rados.rados.0.smithi125.stdout:1162: done (6 left) 2021-03-24T21:58:22.201 INFO:tasks.rados.rados.0.smithi125.stdout:1163: done (5 left) 2021-03-24T21:58:22.201 INFO:tasks.rados.rados.0.smithi125.stdout:1164: done (4 left) 2021-03-24T21:58:22.201 INFO:tasks.rados.rados.0.smithi125.stdout:1165: done (3 left) 2021-03-24T21:58:22.201 INFO:tasks.rados.rados.0.smithi125.stdout:1166: done (2 left) 2021-03-24T21:58:22.201 INFO:tasks.rados.rados.0.smithi125.stdout:1167: done (1 left) 2021-03-24T21:58:22.202 INFO:tasks.rados.rados.0.smithi125.stdout:1168: done (0 left) 2021-03-24T21:58:22.202 INFO:tasks.rados.rados.0.smithi125.stdout:1169: copy_from oid 6 from oid 33 current snap is 116 2021-03-24T21:58:22.202 INFO:tasks.rados.rados.0.smithi125.stdout:1170: snap_remove snap 107 2021-03-24T21:58:22.204 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T21:58:22.225 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:22.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:22.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:22.231 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T21:58:22.252 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T21:58:22.277 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T21:58:22.310 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T21:58:22.313 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-24T21:58:22.313 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-24T21:58:22.313 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-24T21:58:22.313 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-24T21:58:22.313 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-24T21:58:22.314 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-24T21:58:22.314 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-24T21:58:22.314 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-24T21:58:22.331 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:22.389 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:22.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:22.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:22.495 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:22.504 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:22 smithi125 conmon[24927]: cluster 2021-03-24T21:58:20.794081+0000 mgr.y 2021-03-24T21:58:22.504 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:22 smithi125 conmon[24927]: (mgr.14144) 434 : cluster [DBG] pgmap v658: 168 pgs: 168 active+clean; 163 MiB data, 544 MiB used, 536 GiB / 536 GiB avail; 11 MiB/s rd, 1.6 MiB/s wr, 15 op/s 2021-03-24T21:58:22.504 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:22 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:58:22.504 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:22 smithi125 conmon[24927]: -24T21:58:20.925593+0000 osd.5 (osd.5) 80 : cluster [DBG] 4.8 scrub ok 2021-03-24T21:58:22.505 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:22 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:58:22.505 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:22 smithi125 conmon[24927]: T21:58:21.039171+0000 osd.7 (osd.7) 75 : cluster [DBG] 1.0 scrub ok 2021-03-24T21:58:22.505 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:22 smithi125 conmon[24927]: audit 2021-03 2021-03-24T21:58:22.505 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:22 smithi125 conmon[24927]: -24T21:58:22.110917+0000 mon.a (mon.0) 2021-03-24T21:58:22.505 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:22 smithi125 conmon[24927]: 1020 : audit [DBG] from='client.? v1:172.21.15.125:0/3196514314' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:58:22.506 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:22 smithi125 conmon[24927]: cluster 2021-03-24T21:58 2021-03-24T21:58:22.506 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:22 smithi125 conmon[24927]: :22.197291+0000 mon.a (mon.0) 1021 : cluster [DBG] osdmap e332: 8 total, 8 up, 6 in 2021-03-24T21:58:22.506 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:22 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:58:22.507 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:22 smithi125 conmon[19936]: 21:58:20.794081+0000 mgr.y (mgr.14144) 434 : cluster [DBG] pgmap v658: 168 pgs: 168 active+clean; 163 MiB data, 544 MiB used, 536 GiB / 536 GiB avail; 11 MiB/s rd, 1.6 MiB/s wr, 15 op/s 2021-03-24T21:58:22.507 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:22 smithi125 conmon[19936]: cluster 2021-03-24T21:58:22.507 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:22 smithi125 conmon[19936]: 2021-03-24T21:58 2021-03-24T21:58:22.507 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:22 smithi125 conmon[19936]: :20.925593+0000 osd.5 ( 2021-03-24T21:58:22.507 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:22 smithi125 conmon[19936]: osd.5) 80 : cluster 2021-03-24T21:58:22.508 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:22 smithi125 conmon[19936]: [DBG] 4.8 scrub ok 2021-03-24T21:58:22.508 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:22 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:58:22.508 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:22 smithi125 conmon[19936]: -03-24T21:58:21.039171+0000 osd.7 2021-03-24T21:58:22.508 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:22 smithi125 conmon[19936]: (osd.7) 75 : cluster [DBG] 1.0 scrub ok 2021-03-24T21:58:22.508 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:22 smithi125 conmon[19936]: audit 2021-03 2021-03-24T21:58:22.509 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:22 smithi125 conmon[19936]: -24T21:58:22.110917+0000 2021-03-24T21:58:22.509 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:22 smithi125 conmon[19936]: mon.a (mon.0) 1020 : audit 2021-03-24T21:58:22.509 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:22 smithi125 conmon[19936]: [DBG] from='client.? v1:172.21.15.125:0/3196514314' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:58:22.509 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:22 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:58:22.509 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:22 smithi125 conmon[19936]: -03-24T21:58:22.197291+0000 mon.a (mon.0 2021-03-24T21:58:22.510 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:22 smithi125 conmon[19936]: ) 1021 : cluster [DBG] osdmap e332: 8 total, 8 up, 6 in 2021-03-24T21:58:22.550 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:22.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:22.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:58:22.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:22 smithi142 conmon[20125]: cluster 2021-03-24T21:58:20.794081+0000 mgr.y (mgr.14144) 434 : cluster [DBG] 2021-03-24T21:58:22.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:22 smithi142 conmon[20125]: pgmap v658: 168 pgs: 168 active+clean; 163 MiB data, 544 MiB used, 536 GiB / 536 GiB avail; 11 MiB/s rd, 1.6 MiB/s wr, 15 op/s 2021-03-24T21:58:22.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:22 smithi142 conmon[20125]: cluster 2021-03-24T21:58:20.925593+0000 osd.5 (osd.5) 80 2021-03-24T21:58:22.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:22 smithi142 conmon[20125]: : cluster [DBG] 4.8 scrub ok 2021-03-24T21:58:22.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:22 smithi142 conmon[20125]: cluster 2021-03-24T21:58:22.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:22 smithi142 conmon[20125]: 2021-03-24T21:58:21.039171 2021-03-24T21:58:22.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:22 smithi142 conmon[20125]: +0000 osd.7 (osd.7) 75 : 2021-03-24T21:58:22.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:22 smithi142 conmon[20125]: cluster [DBG] 1.0 scrub ok 2021-03-24T21:58:22.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:22 smithi142 conmon[20125]: audit 2021-03-24 2021-03-24T21:58:22.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:22 smithi142 conmon[20125]: T21:58:22.110917+0000 mon.a (mon 2021-03-24T21:58:22.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:22 smithi142 conmon[20125]: .0) 1020 : audit [DBG] from='client.? v1:172.21.15.125:0/3196514314' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:58:22.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:22 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:58:22.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:22 smithi142 conmon[20125]: 58:22.197291+0000 mon.a (mon.0) 1021 : cluster 2021-03-24T21:58:22.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:22 smithi142 conmon[20125]: [DBG] osdmap e332: 8 total, 8 up, 6 in 2021-03-24T21:58:22.655 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:22.712 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:22.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:22.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:22.787 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.7 dump_historic_ops 2021-03-24T21:58:22.818 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:22.889 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:22.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:22.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:58:22.996 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:23.043 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:23.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:23.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:58:23.109 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T21:58:23.148 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:23.201 INFO:tasks.rados.rados.0.smithi125.stdout:1170: done (1 left) 2021-03-24T21:58:23.201 INFO:tasks.rados.rados.0.smithi125.stdout:1171: snap_remove snap 109 2021-03-24T21:58:23.213 INFO:teuthology.orchestra.run.smithi125.stderr:set pool 7 pgp_num to 36 2021-03-24T21:58:23.214 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:23.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:23.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:23.321 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:23.375 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:23.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:23.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:58:23.481 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:23.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:23 smithi125 conmon[19936]: audit 2021-03-24T21:58:22. 2021-03-24T21:58:23.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:23 smithi125 conmon[19936]: 623849+0000 mon.c (mon.2) 53 : audit [INF] from='client.? v1:172.21.15.125:0/2773473162' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "36"}]: dispatch 2021-03-24T21:58:23.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:23 smithi125 conmon[19936]: audit 2021-03-24 2021-03-24T21:58:23.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:23 smithi125 conmon[19936]: T21:58:22.624476+0000 mon.a (mon.0) 1022 : audit [INF] from='client.35087 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "36"}]: dispatch 2021-03-24T21:58:23.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:23 smithi125 conmon[19936]: audit 2021-03-24T21: 2021-03-24T21:58:23.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:23 smithi125 conmon[19936]: 58:23.198694+0000 mon.a (mon.0) 1023 : audit 2021-03-24T21:58:23.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:23 smithi125 conmon[19936]: [INF] from='client.35087 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "36"}]': finished 2021-03-24T21:58:23.487 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:23 smithi125 conmon[19936]: cluster 2021-03-24T21:58: 2021-03-24T21:58:23.487 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:23 smithi125 conmon[19936]: 23.199185+0000 mon.a (mon.0) 1024 : cluster [DBG] 2021-03-24T21:58:23.487 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:23 smithi125 conmon[19936]: osdmap e333: 8 total, 8 up, 6 in 2021-03-24T21:58:23.487 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:23 smithi125 conmon[24927]: audit 2021-03-24T21:58:22.623849+0000 mon.c (mon 2021-03-24T21:58:23.488 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:23 smithi125 conmon[24927]: .2) 53 : audit [INF] from='client.? v1:172.21.15.125:0/2773473162' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "36"}]: dispatch 2021-03-24T21:58:23.488 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:23 smithi125 conmon[24927]: audit 2021-03-24T21:58 2021-03-24T21:58:23.488 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:23 smithi125 conmon[24927]: :22.624476+0000 mon.a (mon.0) 1022 : audit [INF] from='client.35087 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "36"}]: dispatch 2021-03-24T21:58:23.488 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:23 smithi125 conmon[24927]: audit 2021-03-24T21 2021-03-24T21:58:23.489 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:23 smithi125 conmon[24927]: :58:23.198694+0000 mon.a (mon. 2021-03-24T21:58:23.489 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:23 smithi125 conmon[24927]: 0) 1023 : audit [INF] from='client.35087 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "36"}]': finished 2021-03-24T21:58:23.489 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:23 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:58:23.489 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:23 smithi125 conmon[24927]: 24T21:58:23.199185+0000 mon.a (mon 2021-03-24T21:58:23.490 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:23 smithi125 conmon[24927]: .0) 1024 : cluster [DBG] osdmap e333: 8 total, 8 up, 6 in 2021-03-24T21:58:23.540 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:23.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:23.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:23.577 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:23 smithi142 conmon[20125]: audit 2021-03-24T21:58: 2021-03-24T21:58:23.578 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:23 smithi142 conmon[20125]: 22.623849+0000 mon.c (mon.2) 53 : audit [INF] from='client.? v1:172.21.15.125:0/2773473162' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "36"}]: dispatch 2021-03-24T21:58:23.578 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:23 smithi142 conmon[20125]: audit 2021-03-24T21:58 2021-03-24T21:58:23.578 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:23 smithi142 conmon[20125]: :22.624476+0000 mon.a (mon.0) 1022 : audit [INF] from='client.35087 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "36"}]: dispatch 2021-03-24T21:58:23.578 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:23 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:58:23.579 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:23 smithi142 conmon[20125]: 03-24T21:58:23.198694+0000 mon.a (mon.0) 1023 : audit [INF] from='client.35087 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "36"}]': finished 2021-03-24T21:58:23.579 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:23 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:58:23.579 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:23 smithi142 conmon[20125]: -03-24T21:58:23.199185+0000 2021-03-24T21:58:23.579 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:23 smithi142 conmon[20125]: mon.a (mon.0) 1024 : cluster 2021-03-24T21:58:23.579 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:23 smithi142 conmon[20125]: [DBG] osdmap e333: 8 total, 8 up, 6 in 2021-03-24T21:58:23.645 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:23.699 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:23.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:23.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:23.743 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-24T21:58:23.806 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:23.865 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:23.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:23.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:23.969 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:24.018 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:24.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:24.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:58:24.123 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:24.182 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:24.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:24.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:24.205 INFO:tasks.rados.rados.0.smithi125.stdout:1171: done (1 left) 2021-03-24T21:58:24.206 INFO:tasks.rados.rados.0.smithi125.stdout:1172: read oid 20 snap -1 2021-03-24T21:58:24.206 INFO:tasks.rados.rados.0.smithi125.stdout:1172: expect (ObjNum 380 snap 99 seq_num 380) 2021-03-24T21:58:24.206 INFO:tasks.rados.rados.0.smithi125.stdout:1173: snap_remove snap 114 2021-03-24T21:58:24.207 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:58:24.288 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:24.342 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:24.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:24.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:58:24.448 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:24.507 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:24.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:24.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:24.569 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:24 smithi142 conmon[20125]: cluster 2021-03-24T21:58: 2021-03-24T21:58:24.569 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:24 smithi142 conmon[20125]: 22.794892+0000 mgr.y (mgr.14144) 435 : cluster [DBG] pgmap v661: 168 pgs: 168 active+clean; 163 MiB data, 550 MiB used, 536 GiB / 536 GiB avail; 6.9 MiB/s rd, 1.6 MiB/s wr, 13 op/s 2021-03-24T21:58:24.570 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:24 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:58:24.570 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:24 smithi142 conmon[20125]: 21:58:22.940850+0000 osd.5 (osd.5) 81 : cluster [DBG] 7.0 scrub ok 2021-03-24T21:58:24.570 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:24 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:58:24.570 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:24 smithi142 conmon[20125]: 58:24.203657+0000 mon.a (mon.0) 1025 : cluster 2021-03-24T21:58:24.570 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:24 smithi142 conmon[20125]: [DBG] osdmap e334: 8 total, 8 up, 6 in 2021-03-24T21:58:24.588 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:24 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:58:24.588 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:24 smithi125 conmon[24927]: 58:22.794892+0000 mgr.y (mgr.14144) 435 : cluster [DBG] pgmap v661: 168 pgs: 168 active+clean; 163 MiB data, 550 MiB used, 536 GiB / 536 GiB avail; 6.9 MiB/s rd, 1.6 MiB/s wr, 13 op/s 2021-03-24T21:58:24.588 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:24 smithi125 conmon[24927]: cluster 2021-03-24T21:58:22. 2021-03-24T21:58:24.588 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:24 smithi125 conmon[24927]: 940850+0000 osd.5 (osd.5) 81 : cluster [DBG] 2021-03-24T21:58:24.589 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:24 smithi125 conmon[24927]: 7.0 scrub ok 2021-03-24T21:58:24.589 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:24 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:58:24.589 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:24 smithi125 conmon[24927]: 58:24.203657+0000 mon.a (mon.0) 2021-03-24T21:58:24.589 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:24 smithi125 conmon[24927]: 1025 : cluster [DBG] osdmap e334: 8 total, 8 up, 6 in 2021-03-24T21:58:24.590 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:24 smithi125 conmon[19936]: cluster 2021-03-24T21:58:22.794892+0000 2021-03-24T21:58:24.590 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:24 smithi125 conmon[19936]: mgr.y (mgr.14144) 435 : cluster [DBG] pgmap v661: 168 pgs: 168 active+clean; 163 MiB data, 550 MiB used, 536 GiB / 536 GiB avail; 6.9 MiB/s rd, 1.6 MiB/s wr, 13 op/s 2021-03-24T21:58:24.590 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:24 smithi125 conmon[19936]: cluster 2021-03-24T21:58:22.940850+0000 osd.5 2021-03-24T21:58:24.590 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:24 smithi125 conmon[19936]: (osd.5) 81 : cluster [DBG] 7.0 scrub ok 2021-03-24T21:58:24.591 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:24 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:58:24.591 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:24 smithi125 conmon[19936]: -03-24T21:58:24. 2021-03-24T21:58:24.591 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:24 smithi125 conmon[19936]: 203657+0000 mon.a (mon.0) 1025 : cluster [DBG] 2021-03-24T21:58:24.591 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:24 smithi125 conmon[19936]: osdmap e334: 8 total, 8 up, 6 in 2021-03-24T21:58:24.613 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:24.673 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:24.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:24.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:24.779 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:24.830 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:24.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:24.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:58:24.937 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:24.999 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:25.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:25.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:58:25.104 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:25.155 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:25.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:25.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:58:25.188 INFO:tasks.rados.rados.0.smithi125.stdout:1173: done (2 left) 2021-03-24T21:58:25.188 INFO:tasks.rados.rados.0.smithi125.stdout:1174: read oid 37 snap -1 2021-03-24T21:58:25.188 INFO:tasks.rados.rados.0.smithi125.stdout:1174: expect deleted 2021-03-24T21:58:25.189 INFO:tasks.rados.rados.0.smithi125.stdout:1175: delete oid 19 current snap is 116 2021-03-24T21:58:25.189 INFO:tasks.rados.rados.0.smithi125.stdout:1169: finishing copy_from to smithi12556470-6 2021-03-24T21:58:25.189 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 6 v 160 (ObjNum 298 snap 74 seq_num 298) dirty exists 2021-03-24T21:58:25.189 INFO:tasks.rados.rados.0.smithi125.stdout:1169: finishing copy_from racing read to smithi12556470-6 2021-03-24T21:58:25.190 INFO:tasks.rados.rados.0.smithi125.stdout:1172: expect (ObjNum 380 snap 99 seq_num 380) 2021-03-24T21:58:25.207 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2021-03-24T21:58:25.207 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2021-03-24T21:58:25.208 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2021-03-24T21:58:25.208 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2021-03-24T21:58:25.208 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.208 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.34352.0:3 3.13 3:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected e61)", 2021-03-24T21:58:25.209 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:51:45.387966+0000", 2021-03-24T21:58:25.209 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 399.817007735, 2021-03-24T21:58:25.209 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.19192994199999999, 2021-03-24T21:58:25.209 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:58:25.210 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "reached pg", 2021-03-24T21:58:25.210 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:58:25.210 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.34352", 2021-03-24T21:58:25.210 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/3436348424", 2021-03-24T21:58:25.210 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 3 2021-03-24T21:58:25.211 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.211 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:58:25.211 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.211 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:58:25.212 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:45.387966+0000", 2021-03-24T21:58:25.212 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:58:25.212 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.212 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.213 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:58:25.213 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:45.387966+0000", 2021-03-24T21:58:25.213 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2021-03-24T21:58:25.213 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.213 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.214 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:58:25.214 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:45.387964+0000", 2021-03-24T21:58:25.214 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.7370000000000006e-06 2021-03-24T21:58:25.214 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.215 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.215 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:58:25.215 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:45.387972+0000", 2021-03-24T21:58:25.215 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5412000000000001e-05 2021-03-24T21:58:25.215 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.216 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.216 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:58:25.216 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:45.387987+0000", 2021-03-24T21:58:25.216 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.216e-06 2021-03-24T21:58:25.216 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.217 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.217 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:58:25.217 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:45.387994+0000", 2021-03-24T21:58:25.217 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0209000000000004e-05 2021-03-24T21:58:25.217 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.218 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.218 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:58:25.218 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:45.388074+0000", 2021-03-24T21:58:25.218 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2846e-05 2021-03-24T21:58:25.219 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.219 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.219 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for peered", 2021-03-24T21:58:25.219 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:45.388087+0000", 2021-03-24T21:58:25.219 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.19160065900000001 2021-03-24T21:58:25.220 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.220 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.220 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:58:25.220 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:45.579688+0000", 2021-03-24T21:58:25.221 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00020767600000000001 2021-03-24T21:58:25.221 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.221 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.221 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:58:25.221 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:45.579896+0000", 2021-03-24T21:58:25.222 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.19192994199999999 2021-03-24T21:58:25.222 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:58:25.222 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:58:25.222 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:58:25.223 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.223 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.223 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.24388.0:166 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected e107)", 2021-03-24T21:58:25.223 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:54:37.427176+0000", 2021-03-24T21:58:25.223 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 227.77779744399999, 2021-03-24T21:58:25.224 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.699168805, 2021-03-24T21:58:25.224 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:58:25.224 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T21:58:25.224 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:58:25.224 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.24388", 2021-03-24T21:58:25.225 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.142:0/626803041", 2021-03-24T21:58:25.225 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 166 2021-03-24T21:58:25.225 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.225 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:58:25.226 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.226 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:58:25.226 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:37.427176+0000", 2021-03-24T21:58:25.226 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:58:25.226 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.227 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.227 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:58:25.227 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:37.427176+0000", 2021-03-24T21:58:25.227 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999957 2021-03-24T21:58:25.228 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.228 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.228 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:58:25.228 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:37.427172+0000", 2021-03-24T21:58:25.228 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8757e-05 2021-03-24T21:58:25.229 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.229 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.229 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:58:25.229 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:37.427190+0000", 2021-03-24T21:58:25.229 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3581999999999998e-05 2021-03-24T21:58:25.230 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.230 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.230 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:58:25.230 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:37.427224+0000", 2021-03-24T21:58:25.231 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5787e-05 2021-03-24T21:58:25.231 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.231 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.231 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:58:25.231 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:37.427240+0000", 2021-03-24T21:58:25.232 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00020074000000000001 2021-03-24T21:58:25.232 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.232 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.232 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:58:25.232 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:37.427440+0000", 2021-03-24T21:58:25.233 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8619000000000002e-05 2021-03-24T21:58:25.233 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.233 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.233 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for peered", 2021-03-24T21:58:25.233 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:37.427469+0000", 2021-03-24T21:58:25.234 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.69859708399999998 2021-03-24T21:58:25.234 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.234 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.234 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:58:25.235 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:38.126066+0000", 2021-03-24T21:58:25.235 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133013 2021-03-24T21:58:25.235 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.235 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.235 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:58:25.236 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:38.126199+0000", 2021-03-24T21:58:25.236 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014562000000000001 2021-03-24T21:58:25.236 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.236 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.236 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:58:25.237 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:38.126345+0000", 2021-03-24T21:58:25.237 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.699168805 2021-03-24T21:58:25.237 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:58:25.237 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:58:25.238 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:58:25.238 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.238 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.238 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:1468 7.15 7:ad88ec1b:::smithi12556470-47:head [copy-from ver 92] snapc 62=[60] ondisk+write+known_if_redirected e206)", 2021-03-24T21:58:25.238 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:56:15.151262+0000", 2021-03-24T21:58:25.239 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 130.053711846, 2021-03-24T21:58:25.239 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027242958000000001, 2021-03-24T21:58:25.239 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:58:25.239 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:58:25.239 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:58:25.240 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T21:58:25.240 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:58:25.240 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 1468 2021-03-24T21:58:25.240 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.241 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:58:25.241 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.241 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:58:25.241 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:15.151262+0000", 2021-03-24T21:58:25.241 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:58:25.242 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.242 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.242 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:58:25.242 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:15.151262+0000", 2021-03-24T21:58:25.242 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999957 2021-03-24T21:58:25.243 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.243 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.243 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:58:25.243 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:15.151257+0000", 2021-03-24T21:58:25.244 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.7110999999999997e-05 2021-03-24T21:58:25.244 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.244 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.244 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:58:25.244 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:15.151304+0000", 2021-03-24T21:58:25.245 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.5647000000000003e-05 2021-03-24T21:58:25.245 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.245 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.245 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:58:25.245 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:15.151380+0000", 2021-03-24T21:58:25.246 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5477999999999999e-05 2021-03-24T21:58:25.246 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.246 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.246 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:58:25.247 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:15.151405+0000", 2021-03-24T21:58:25.247 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.8451999999999999e-05 2021-03-24T21:58:25.247 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.247 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.247 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:58:25.248 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:15.151504+0000", 2021-03-24T21:58:25.248 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1641000000000002e-05 2021-03-24T21:58:25.248 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.248 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.248 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op must wait for map", 2021-03-24T21:58:25.249 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:15.151526+0000", 2021-03-24T21:58:25.249 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0044586779999999998 2021-03-24T21:58:25.249 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.249 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.249 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:58:25.250 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:15.155984+0000", 2021-03-24T21:58:25.250 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000161898 2021-03-24T21:58:25.250 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.250 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.251 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:58:25.251 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:15.156146+0000", 2021-03-24T21:58:25.251 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.010759754999999999 2021-03-24T21:58:25.251 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.251 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.252 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 2,5", 2021-03-24T21:58:25.252 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:15.166906+0000", 2021-03-24T21:58:25.252 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.007121418 2021-03-24T21:58:25.252 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.252 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.253 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:58:25.253 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:15.174027+0000", 2021-03-24T21:58:25.253 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034572380000000001 2021-03-24T21:58:25.253 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.254 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.254 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:58:25.254 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:15.177485+0000", 2021-03-24T21:58:25.254 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00084470900000000002 2021-03-24T21:58:25.254 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.255 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.255 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T21:58:25.255 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:15.178329+0000", 2021-03-24T21:58:25.255 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.1116000000000001e-05 2021-03-24T21:58:25.255 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.256 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.256 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T21:58:25.256 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:15.178410+0000", 2021-03-24T21:58:25.256 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4067000000000006e-05 2021-03-24T21:58:25.257 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.257 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.257 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:58:25.257 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:15.178504+0000", 2021-03-24T21:58:25.257 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027242958000000001 2021-03-24T21:58:25.258 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:58:25.258 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:58:25.258 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:58:25.258 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.258 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.259 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:1469 7.15 7:ad88ec1b:::smithi12556470-47:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e206)", 2021-03-24T21:58:25.259 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:56:15.151434+0000", 2021-03-24T21:58:25.259 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 130.053539489, 2021-03-24T21:58:25.259 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027366415000000002, 2021-03-24T21:58:25.260 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:58:25.260 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T21:58:25.260 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:58:25.260 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T21:58:25.260 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:58:25.261 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 1469 2021-03-24T21:58:25.261 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.261 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:58:25.261 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.261 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:58:25.262 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:15.151434+0000", 2021-03-24T21:58:25.262 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:58:25.262 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.262 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.263 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:58:25.263 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:15.151434+0000", 2021-03-24T21:58:25.263 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2021-03-24T21:58:25.263 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.263 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.264 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:58:25.264 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:15.151432+0000", 2021-03-24T21:58:25.264 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000145919 2021-03-24T21:58:25.264 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.264 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.265 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:58:25.265 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:15.151578+0000", 2021-03-24T21:58:25.265 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7679000000000001e-05 2021-03-24T21:58:25.265 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.265 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.266 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:58:25.266 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:15.151596+0000", 2021-03-24T21:58:25.266 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6449999999999999e-05 2021-03-24T21:58:25.266 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.267 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.267 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:58:25.267 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:15.151622+0000", 2021-03-24T21:58:25.267 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.7301999999999999e-05 2021-03-24T21:58:25.267 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.268 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.268 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:58:25.268 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:15.151710+0000", 2021-03-24T21:58:25.268 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.7120000000000003e-06 2021-03-24T21:58:25.268 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.269 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.269 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting_for_map not empty", 2021-03-24T21:58:25.269 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:15.151716+0000", 2021-03-24T21:58:25.269 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0046931780000000001 2021-03-24T21:58:25.269 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.270 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.270 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:58:25.270 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:15.156409+0000", 2021-03-24T21:58:25.270 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0472e-05 2021-03-24T21:58:25.271 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.271 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.271 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for blocked object", 2021-03-24T21:58:25.271 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:15.156470+0000", 2021-03-24T21:58:25.271 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.017308204000000001 2021-03-24T21:58:25.272 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.272 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.272 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:58:25.272 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:15.173778+0000", 2021-03-24T21:58:25.272 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015794600000000001 2021-03-24T21:58:25.273 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.273 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.273 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for rw locks", 2021-03-24T21:58:25.273 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:15.173936+0000", 2021-03-24T21:58:25.274 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0046358909999999996 2021-03-24T21:58:25.274 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.274 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.274 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:58:25.274 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:15.178572+0000", 2021-03-24T21:58:25.275 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012153400000000001 2021-03-24T21:58:25.275 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.275 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.275 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:58:25.275 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:15.178694+0000", 2021-03-24T21:58:25.276 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000106752 2021-03-24T21:58:25.276 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.276 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.276 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:58:25.277 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:15.178800+0000", 2021-03-24T21:58:25.277 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027366415000000002 2021-03-24T21:58:25.277 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:58:25.277 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:58:25.277 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:58:25.278 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.278 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.278 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:1547 7.11 7:8dd64f9e:::smithi12556470-49:head [copy-from ver 139] snapc 69=[69,68] ondisk+write+known_if_redirected e211)", 2021-03-24T21:58:25.278 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:56:20.129319+0000", 2021-03-24T21:58:25.278 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 125.075654051, 2021-03-24T21:58:25.279 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.040477056999999997, 2021-03-24T21:58:25.279 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:58:25.279 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:58:25.279 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:58:25.280 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T21:58:25.280 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:58:25.280 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 1547 2021-03-24T21:58:25.280 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.280 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:58:25.281 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.281 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:58:25.281 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:20.129319+0000", 2021-03-24T21:58:25.281 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:58:25.281 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.282 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.282 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:58:25.282 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:20.129319+0000", 2021-03-24T21:58:25.282 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999957 2021-03-24T21:58:25.283 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.283 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.283 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:58:25.283 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:20.129315+0000", 2021-03-24T21:58:25.283 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.857e-05 2021-03-24T21:58:25.284 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.284 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.284 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:58:25.284 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:20.129334+0000", 2021-03-24T21:58:25.284 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5431000000000001e-05 2021-03-24T21:58:25.285 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.285 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.285 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:58:25.285 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:20.129359+0000", 2021-03-24T21:58:25.285 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.271e-05 2021-03-24T21:58:25.286 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.286 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.286 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:58:25.286 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:20.129372+0000", 2021-03-24T21:58:25.287 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000104888 2021-03-24T21:58:25.287 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.287 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.287 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:58:25.287 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:20.129477+0000", 2021-03-24T21:58:25.288 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0988999999999999e-05 2021-03-24T21:58:25.288 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.288 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.288 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op must wait for map", 2021-03-24T21:58:25.289 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:20.129488+0000", 2021-03-24T21:58:25.289 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0082642129999999994 2021-03-24T21:58:25.289 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.289 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.289 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:58:25.290 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:20.137752+0000", 2021-03-24T21:58:25.290 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00020204399999999999 2021-03-24T21:58:25.290 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.290 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.291 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:58:25.291 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:20.137954+0000", 2021-03-24T21:58:25.291 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.012045396999999999 2021-03-24T21:58:25.291 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.291 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.292 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 2,3", 2021-03-24T21:58:25.292 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:20.149999+0000", 2021-03-24T21:58:25.292 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0094695879999999993 2021-03-24T21:58:25.292 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.292 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.293 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T21:58:25.293 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:20.159469+0000", 2021-03-24T21:58:25.293 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0098839070000000008 2021-03-24T21:58:25.293 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.294 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.294 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:58:25.294 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:20.169353+0000", 2021-03-24T21:58:25.294 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00024638999999999998 2021-03-24T21:58:25.294 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.295 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.295 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:58:25.295 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:20.169599+0000", 2021-03-24T21:58:25.295 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6977000000000005e-05 2021-03-24T21:58:25.295 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.296 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.296 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T21:58:25.296 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:20.169676+0000", 2021-03-24T21:58:25.296 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012024899999999999 2021-03-24T21:58:25.296 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.297 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.297 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:58:25.297 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:20.169796+0000", 2021-03-24T21:58:25.297 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.040477056999999997 2021-03-24T21:58:25.298 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:58:25.298 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:58:25.298 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:58:25.298 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.298 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.299 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:1548 7.11 7:8dd64f9e:::smithi12556470-49:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e211)", 2021-03-24T21:58:25.299 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:56:20.129395+0000", 2021-03-24T21:58:25.299 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 125.075578722, 2021-03-24T21:58:25.299 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.040731604999999997, 2021-03-24T21:58:25.299 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:58:25.300 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T21:58:25.300 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:58:25.300 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T21:58:25.300 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:58:25.301 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 1548 2021-03-24T21:58:25.301 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.301 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:58:25.301 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.301 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:58:25.302 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:20.129395+0000", 2021-03-24T21:58:25.302 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:58:25.302 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.302 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.303 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:58:25.303 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:20.129395+0000", 2021-03-24T21:58:25.303 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2021-03-24T21:58:25.303 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.303 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.304 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:58:25.304 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:20.129394+0000", 2021-03-24T21:58:25.304 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.2619999999999998e-06 2021-03-24T21:58:25.304 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.304 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.305 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:58:25.305 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:20.129399+0000", 2021-03-24T21:58:25.305 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5998000000000002e-05 2021-03-24T21:58:25.305 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.306 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.306 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:58:25.306 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:20.129415+0000", 2021-03-24T21:58:25.306 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4310000000000002e-06 2021-03-24T21:58:25.306 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.307 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.307 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:58:25.307 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:20.129420+0000", 2021-03-24T21:58:25.307 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00017388599999999999 2021-03-24T21:58:25.307 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.308 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.308 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:58:25.308 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:20.129594+0000", 2021-03-24T21:58:25.308 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3742e-05 2021-03-24T21:58:25.308 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.309 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.309 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting_for_map not empty", 2021-03-24T21:58:25.309 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:20.129608+0000", 2021-03-24T21:58:25.309 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0087234070000000007 2021-03-24T21:58:25.309 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.310 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.310 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:58:25.310 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:20.138331+0000", 2021-03-24T21:58:25.310 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000175859 2021-03-24T21:58:25.311 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.311 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.311 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for blocked object", 2021-03-24T21:58:25.311 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:20.138507+0000", 2021-03-24T21:58:25.311 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.019768038000000002 2021-03-24T21:58:25.312 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.312 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.312 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:58:25.312 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:20.158275+0000", 2021-03-24T21:58:25.313 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000116413 2021-03-24T21:58:25.313 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.313 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.313 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for rw locks", 2021-03-24T21:58:25.313 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:20.158392+0000", 2021-03-24T21:58:25.314 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011478471000000001 2021-03-24T21:58:25.314 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.314 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.314 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:58:25.314 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:20.169870+0000", 2021-03-24T21:58:25.315 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000123504 2021-03-24T21:58:25.315 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.315 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.315 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:58:25.316 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:20.169994+0000", 2021-03-24T21:58:25.316 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000132698 2021-03-24T21:58:25.316 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.316 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.316 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:58:25.317 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:20.170126+0000", 2021-03-24T21:58:25.317 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.040731604999999997 2021-03-24T21:58:25.317 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:58:25.317 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:58:25.318 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:58:25.318 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.318 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.318 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:1591 7.19 7:992cb487:::smithi12556470-5:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e213)", 2021-03-24T21:58:25.318 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:56:22.219253+0000", 2021-03-24T21:58:25.319 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 122.985720118, 2021-03-24T21:58:25.319 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025452993, 2021-03-24T21:58:25.319 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:58:25.319 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T21:58:25.319 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:58:25.320 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T21:58:25.320 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:58:25.320 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 1591 2021-03-24T21:58:25.320 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.321 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:58:25.321 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.321 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:58:25.321 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:22.219253+0000", 2021-03-24T21:58:25.321 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:58:25.322 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.322 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.322 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:58:25.322 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:22.219253+0000", 2021-03-24T21:58:25.323 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2021-03-24T21:58:25.323 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.323 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.323 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:58:25.323 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:22.219251+0000", 2021-03-24T21:58:25.324 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0597e-05 2021-03-24T21:58:25.324 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.324 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.324 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:58:25.324 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:22.219261+0000", 2021-03-24T21:58:25.325 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0409999999999999e-05 2021-03-24T21:58:25.325 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.325 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.325 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:58:25.325 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:22.219282+0000", 2021-03-24T21:58:25.325 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4939e-05 2021-03-24T21:58:25.326 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.326 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.326 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:58:25.326 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:22.219297+0000", 2021-03-24T21:58:25.327 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00082330999999999995 2021-03-24T21:58:25.327 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.327 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.327 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:58:25.327 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:22.220120+0000", 2021-03-24T21:58:25.328 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8739000000000001e-05 2021-03-24T21:58:25.328 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.328 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.328 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting_for_map not empty", 2021-03-24T21:58:25.328 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:22.220139+0000", 2021-03-24T21:58:25.329 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0039958550000000004 2021-03-24T21:58:25.329 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.329 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.329 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:58:25.330 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:22.224135+0000", 2021-03-24T21:58:25.330 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00023615299999999999 2021-03-24T21:58:25.330 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.330 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.330 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for blocked object", 2021-03-24T21:58:25.331 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:22.224371+0000", 2021-03-24T21:58:25.331 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.01466757 2021-03-24T21:58:25.331 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.331 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.332 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:58:25.332 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:22.239038+0000", 2021-03-24T21:58:25.332 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000109604 2021-03-24T21:58:25.332 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.332 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.333 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for rw locks", 2021-03-24T21:58:25.333 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:22.239148+0000", 2021-03-24T21:58:25.333 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0053908719999999997 2021-03-24T21:58:25.333 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.333 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.334 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:58:25.334 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:22.244539+0000", 2021-03-24T21:58:25.334 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.9937000000000002e-05 2021-03-24T21:58:25.334 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.335 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.335 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:58:25.335 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:22.244639+0000", 2021-03-24T21:58:25.335 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.7457000000000001e-05 2021-03-24T21:58:25.335 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.336 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.336 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:58:25.336 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:22.244706+0000", 2021-03-24T21:58:25.336 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025452993 2021-03-24T21:58:25.337 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:58:25.337 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:58:25.337 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:58:25.337 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.337 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.338 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:1777 7.17 7:ef6d6bee:::smithi12556470-24:head [write 420241~546489,stat] snapc 7e=[7e,79,78] ondisk+write+known_if_redirected e232)", 2021-03-24T21:58:25.338 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:56:41.195831+0000", 2021-03-24T21:58:25.338 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 104.00914198, 2021-03-24T21:58:25.338 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.046456559000000001, 2021-03-24T21:58:25.339 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:58:25.339 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:58:25.339 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:58:25.339 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T21:58:25.339 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:58:25.340 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 1777 2021-03-24T21:58:25.340 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.340 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:58:25.340 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.340 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:58:25.341 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.195831+0000", 2021-03-24T21:58:25.341 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:58:25.341 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.341 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.342 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:58:25.342 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.195831+0000", 2021-03-24T21:58:25.342 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999866 2021-03-24T21:58:25.342 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.342 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.343 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:58:25.343 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.195818+0000", 2021-03-24T21:58:25.343 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00086681300000000002 2021-03-24T21:58:25.343 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.343 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.344 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:58:25.344 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.196685+0000", 2021-03-24T21:58:25.344 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5661000000000003e-05 2021-03-24T21:58:25.344 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.344 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.345 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:58:25.345 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.196721+0000", 2021-03-24T21:58:25.345 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5569999999999998e-05 2021-03-24T21:58:25.345 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.346 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.346 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:58:25.346 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.196736+0000", 2021-03-24T21:58:25.346 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000108026 2021-03-24T21:58:25.346 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.347 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.347 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:58:25.347 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.196844+0000", 2021-03-24T21:58:25.347 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1788e-05 2021-03-24T21:58:25.347 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.348 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.348 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op must wait for map", 2021-03-24T21:58:25.348 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.196856+0000", 2021-03-24T21:58:25.348 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0047483960000000002 2021-03-24T21:58:25.348 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.349 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.349 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:58:25.349 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.201604+0000", 2021-03-24T21:58:25.349 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00021254800000000001 2021-03-24T21:58:25.350 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.350 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.350 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:58:25.350 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.201817+0000", 2021-03-24T21:58:25.351 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00043252299999999998 2021-03-24T21:58:25.351 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.351 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.351 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 2,5", 2021-03-24T21:58:25.351 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.202249+0000", 2021-03-24T21:58:25.352 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0031197579999999998 2021-03-24T21:58:25.352 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.352 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.352 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:58:25.352 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.205369+0000", 2021-03-24T21:58:25.353 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.019907871000000001 2021-03-24T21:58:25.353 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.353 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.353 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T21:58:25.354 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.225277+0000", 2021-03-24T21:58:25.354 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.016760265 2021-03-24T21:58:25.354 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.354 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.354 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:58:25.355 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.242037+0000", 2021-03-24T21:58:25.355 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8051000000000001e-05 2021-03-24T21:58:25.355 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.355 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.355 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T21:58:25.356 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.242125+0000", 2021-03-24T21:58:25.356 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000162585 2021-03-24T21:58:25.356 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.356 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.357 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:58:25.357 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.242288+0000", 2021-03-24T21:58:25.357 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.046456559000000001 2021-03-24T21:58:25.357 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:58:25.357 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:58:25.358 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:58:25.358 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.358 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.358 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:1778 7.17 7:ef6d6bee:::smithi12556470-24:head [write 1373823~544414] snapc 7e=[7e,79,78] ondisk+write+known_if_redirected e232)", 2021-03-24T21:58:25.359 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:56:41.201425+0000", 2021-03-24T21:58:25.359 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 104.003548006, 2021-03-24T21:58:25.359 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.044673524999999999, 2021-03-24T21:58:25.359 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:58:25.359 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:58:25.360 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:58:25.360 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T21:58:25.360 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:58:25.360 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 1778 2021-03-24T21:58:25.360 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.361 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:58:25.361 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.361 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:58:25.361 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.201425+0000", 2021-03-24T21:58:25.362 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:58:25.362 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.362 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.362 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:58:25.362 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.201425+0000", 2021-03-24T21:58:25.363 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.99999 2021-03-24T21:58:25.363 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.363 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.363 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:58:25.363 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.201415+0000", 2021-03-24T21:58:25.364 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00088500799999999995 2021-03-24T21:58:25.364 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.364 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.364 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:58:25.364 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.202300+0000", 2021-03-24T21:58:25.365 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6229999999999999e-05 2021-03-24T21:58:25.365 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.365 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.365 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:58:25.365 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.202316+0000", 2021-03-24T21:58:25.366 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3740000000000004e-06 2021-03-24T21:58:25.366 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.366 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.366 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:58:25.366 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.202326+0000", 2021-03-24T21:58:25.367 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0023542160000000001 2021-03-24T21:58:25.367 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.367 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.367 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:58:25.368 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.204680+0000", 2021-03-24T21:58:25.368 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015806899999999999 2021-03-24T21:58:25.368 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.368 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.368 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for rw locks", 2021-03-24T21:58:25.369 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.204838+0000", 2021-03-24T21:58:25.369 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.037561912000000003 2021-03-24T21:58:25.369 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.369 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.369 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:58:25.370 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.242400+0000", 2021-03-24T21:58:25.370 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000117706 2021-03-24T21:58:25.370 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.370 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.370 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:58:25.371 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.242518+0000", 2021-03-24T21:58:25.371 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00030236800000000002 2021-03-24T21:58:25.371 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.371 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.371 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 2,5", 2021-03-24T21:58:25.372 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.242820+0000", 2021-03-24T21:58:25.372 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0023244960000000001 2021-03-24T21:58:25.372 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.372 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.372 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T21:58:25.373 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.245145+0000", 2021-03-24T21:58:25.373 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000461824 2021-03-24T21:58:25.373 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.373 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.373 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:58:25.374 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.245606+0000", 2021-03-24T21:58:25.374 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00032675199999999998 2021-03-24T21:58:25.374 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.374 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.374 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:58:25.375 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.245933+0000", 2021-03-24T21:58:25.375 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010531699999999999 2021-03-24T21:58:25.375 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.375 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.375 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T21:58:25.376 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.246038+0000", 2021-03-24T21:58:25.376 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0384000000000003e-05 2021-03-24T21:58:25.376 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.376 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.377 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:58:25.377 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.246099+0000", 2021-03-24T21:58:25.377 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.044673524999999999 2021-03-24T21:58:25.377 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:58:25.377 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:58:25.378 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:58:25.378 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.378 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.378 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:1779 7.17 7:ef6d6bee:::smithi12556470-24:head [setxattr _header (57) in=7b,truncate 2420241] snapc 7e=[7e,79,78] ondisk+write+known_if_redirected e232)", 2021-03-24T21:58:25.378 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:56:41.202340+0000", 2021-03-24T21:58:25.379 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 104.002632995, 2021-03-24T21:58:25.379 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.044499368999999997, 2021-03-24T21:58:25.379 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:58:25.379 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:58:25.380 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:58:25.380 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T21:58:25.380 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:58:25.380 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 1779 2021-03-24T21:58:25.380 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.381 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:58:25.381 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.381 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:58:25.381 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.202340+0000", 2021-03-24T21:58:25.382 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:58:25.382 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.382 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.382 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:58:25.382 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.202340+0000", 2021-03-24T21:58:25.383 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2021-03-24T21:58:25.383 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.383 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.383 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:58:25.383 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.202339+0000", 2021-03-24T21:58:25.384 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.7700000000000004e-06 2021-03-24T21:58:25.384 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.384 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.384 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:58:25.384 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.202345+0000", 2021-03-24T21:58:25.385 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.5000000000000002e-06 2021-03-24T21:58:25.385 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.385 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.385 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:58:25.385 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.202353+0000", 2021-03-24T21:58:25.386 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.6240000000000001e-06 2021-03-24T21:58:25.386 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.386 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.386 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:58:25.387 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.202358+0000", 2021-03-24T21:58:25.387 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0025531450000000002 2021-03-24T21:58:25.387 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.387 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.387 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:58:25.388 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.204911+0000", 2021-03-24T21:58:25.388 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000116142 2021-03-24T21:58:25.388 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.388 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.388 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for rw locks", 2021-03-24T21:58:25.389 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.205027+0000", 2021-03-24T21:58:25.389 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.039158563 2021-03-24T21:58:25.389 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.389 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.389 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:58:25.390 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.244185+0000", 2021-03-24T21:58:25.390 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000120451 2021-03-24T21:58:25.390 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.390 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.390 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:58:25.391 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.244306+0000", 2021-03-24T21:58:25.391 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00026110700000000001 2021-03-24T21:58:25.391 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.391 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.392 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 2,5", 2021-03-24T21:58:25.392 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.244567+0000", 2021-03-24T21:58:25.392 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00058484200000000002 2021-03-24T21:58:25.392 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.392 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.393 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T21:58:25.393 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.245152+0000", 2021-03-24T21:58:25.393 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00061434100000000002 2021-03-24T21:58:25.393 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.393 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.394 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:58:25.394 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.245766+0000", 2021-03-24T21:58:25.394 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000893928 2021-03-24T21:58:25.394 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.394 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.395 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:58:25.395 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.246660+0000", 2021-03-24T21:58:25.395 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.5969000000000004e-05 2021-03-24T21:58:25.395 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.395 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.396 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T21:58:25.396 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.246736+0000", 2021-03-24T21:58:25.396 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000103649 2021-03-24T21:58:25.396 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.396 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.397 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:58:25.397 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.246840+0000", 2021-03-24T21:58:25.397 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.044499368999999997 2021-03-24T21:58:25.397 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:58:25.397 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:58:25.398 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:58:25.398 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.398 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.398 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:1780 7.17 7:ef6d6bee:::smithi12556470-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e232)", 2021-03-24T21:58:25.398 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:56:41.202362+0000", 2021-03-24T21:58:25.399 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 104.00261091, 2021-03-24T21:58:25.399 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.044960122999999998, 2021-03-24T21:58:25.399 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:58:25.399 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T21:58:25.399 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:58:25.400 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T21:58:25.400 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:58:25.400 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 1780 2021-03-24T21:58:25.400 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.401 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:58:25.401 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.401 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:58:25.401 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.202362+0000", 2021-03-24T21:58:25.401 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:58:25.402 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.402 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.402 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:58:25.402 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.202362+0000", 2021-03-24T21:58:25.403 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2021-03-24T21:58:25.403 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.403 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.403 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:58:25.403 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.202362+0000", 2021-03-24T21:58:25.404 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1719999999999999e-06 2021-03-24T21:58:25.404 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.404 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.404 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:58:25.404 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.202365+0000", 2021-03-24T21:58:25.405 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1340000000000001e-06 2021-03-24T21:58:25.405 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.405 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.405 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:58:25.405 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.202371+0000", 2021-03-24T21:58:25.406 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0439999999999997e-06 2021-03-24T21:58:25.406 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.406 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.406 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:58:25.406 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.202375+0000", 2021-03-24T21:58:25.407 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0027201600000000001 2021-03-24T21:58:25.407 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.407 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.407 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:58:25.407 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.205095+0000", 2021-03-24T21:58:25.408 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018977600000000001 2021-03-24T21:58:25.408 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.408 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.408 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for rw locks", 2021-03-24T21:58:25.408 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.205285+0000", 2021-03-24T21:58:25.409 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.039686143 2021-03-24T21:58:25.409 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.409 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.409 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:58:25.410 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.244971+0000", 2021-03-24T21:58:25.410 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015303199999999999 2021-03-24T21:58:25.410 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.410 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.410 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for rw locks", 2021-03-24T21:58:25.411 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.245124+0000", 2021-03-24T21:58:25.411 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0018001289999999999 2021-03-24T21:58:25.414 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.415 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.415 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:58:25.415 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.246924+0000", 2021-03-24T21:58:25.415 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000103454 2021-03-24T21:58:25.416 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.416 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.416 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:58:25.416 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.247028+0000", 2021-03-24T21:58:25.417 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00029469099999999997 2021-03-24T21:58:25.417 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.417 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.417 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:58:25.417 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.247323+0000", 2021-03-24T21:58:25.418 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.044960122999999998 2021-03-24T21:58:25.418 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:58:25.418 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:58:25.418 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:58:25.418 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.419 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.419 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:2099 7.17 7:ec6a359b:::smithi12556470-33:head [copy-from ver 0] snapc 95=[95,93,92,8d,8b,82] ondisk+write+known_if_redirected e255)", 2021-03-24T21:58:25.419 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:57:04.197620+0000", 2021-03-24T21:58:25.419 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 81.007353281999997, 2021-03-24T21:58:25.419 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.95919054100000001, 2021-03-24T21:58:25.420 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:58:25.420 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T21:58:25.420 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:58:25.420 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T21:58:25.421 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:58:25.421 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 2099 2021-03-24T21:58:25.421 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.421 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:58:25.421 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.422 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:58:25.422 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.197620+0000", 2021-03-24T21:58:25.422 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:58:25.422 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.423 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.423 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:58:25.423 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.197620+0000", 2021-03-24T21:58:25.423 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999962 2021-03-24T21:58:25.423 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.424 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.424 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:58:25.424 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.197616+0000", 2021-03-24T21:58:25.424 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7652999999999999e-05 2021-03-24T21:58:25.424 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.425 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.425 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:58:25.425 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.197634+0000", 2021-03-24T21:58:25.425 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6417000000000001e-05 2021-03-24T21:58:25.426 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.426 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.426 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:58:25.426 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.197660+0000", 2021-03-24T21:58:25.426 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9066e-05 2021-03-24T21:58:25.427 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.427 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.427 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:58:25.427 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.197680+0000", 2021-03-24T21:58:25.427 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00017438800000000001 2021-03-24T21:58:25.428 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.428 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.428 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:58:25.428 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.197854+0000", 2021-03-24T21:58:25.428 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3726e-05 2021-03-24T21:58:25.429 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.429 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.429 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op must wait for map", 2021-03-24T21:58:25.429 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.197878+0000", 2021-03-24T21:58:25.429 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0099756870000000004 2021-03-24T21:58:25.430 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.430 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.430 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:58:25.430 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.207853+0000", 2021-03-24T21:58:25.431 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3523000000000007e-05 2021-03-24T21:58:25.431 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.431 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.431 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:58:25.432 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.207937+0000", 2021-03-24T21:58:25.432 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.948873784 2021-03-24T21:58:25.432 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.432 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.432 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:58:25.433 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:05.156811+0000", 2021-03-24T21:58:25.433 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.95919054100000001 2021-03-24T21:58:25.433 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:58:25.433 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:58:25.433 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:58:25.434 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.434 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.434 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:2100 7.17 7:ec6a359b:::smithi12556470-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e255)", 2021-03-24T21:58:25.434 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:57:04.198076+0000", 2021-03-24T21:58:25.435 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 81.006896908000002, 2021-03-24T21:58:25.435 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.95908168400000005, 2021-03-24T21:58:25.435 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:58:25.435 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T21:58:25.435 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:58:25.436 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T21:58:25.436 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:58:25.436 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 2100 2021-03-24T21:58:25.436 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.436 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:58:25.437 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.437 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:58:25.437 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.198076+0000", 2021-03-24T21:58:25.437 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:58:25.437 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.438 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.438 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:58:25.438 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.198076+0000", 2021-03-24T21:58:25.438 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999971 2021-03-24T21:58:25.438 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.439 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.439 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:58:25.439 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.198074+0000", 2021-03-24T21:58:25.439 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3433e-05 2021-03-24T21:58:25.440 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.440 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.440 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:58:25.440 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.198087+0000", 2021-03-24T21:58:25.440 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.262e-05 2021-03-24T21:58:25.441 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.441 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.441 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:58:25.441 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.198110+0000", 2021-03-24T21:58:25.441 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6405999999999998e-05 2021-03-24T21:58:25.442 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.442 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.442 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:58:25.442 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.198126+0000", 2021-03-24T21:58:25.443 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000109786 2021-03-24T21:58:25.443 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.443 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.443 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:58:25.443 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.198236+0000", 2021-03-24T21:58:25.444 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0522e-05 2021-03-24T21:58:25.444 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.444 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.444 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting_for_map not empty", 2021-03-24T21:58:25.444 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.198246+0000", 2021-03-24T21:58:25.445 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0098265899999999996 2021-03-24T21:58:25.445 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.445 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.445 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:58:25.445 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.208073+0000", 2021-03-24T21:58:25.446 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7707999999999999e-05 2021-03-24T21:58:25.446 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.446 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.446 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for blocked object", 2021-03-24T21:58:25.447 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.208111+0000", 2021-03-24T21:58:25.447 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.94884332199999999 2021-03-24T21:58:25.447 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.447 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.447 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:58:25.448 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:05.156954+0000", 2021-03-24T21:58:25.448 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.8762000000000001e-05 2021-03-24T21:58:25.448 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.448 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.448 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:58:25.449 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:05.157053+0000", 2021-03-24T21:58:25.449 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000105321 2021-03-24T21:58:25.449 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.449 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.449 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:58:25.450 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:05.157158+0000", 2021-03-24T21:58:25.450 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.95908168400000005 2021-03-24T21:58:25.450 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:58:25.450 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:58:25.451 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:58:25.451 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.451 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.451 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:2551 7.19 7:9c4f4ea5:::smithi12556470-41:head [write 1825176~242151] snapc b7=[b7,b4] ondisk+write+known_if_redirected e290)", 2021-03-24T21:58:25.451 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:57:40.195219+0000", 2021-03-24T21:58:25.452 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 45.009754712000003, 2021-03-24T21:58:25.452 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027634101000000001, 2021-03-24T21:58:25.452 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:58:25.452 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:58:25.452 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:58:25.453 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T21:58:25.453 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:58:25.453 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 2551 2021-03-24T21:58:25.453 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.453 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:58:25.454 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.454 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:58:25.454 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.195219+0000", 2021-03-24T21:58:25.454 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:58:25.454 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.455 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.455 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:58:25.455 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.195219+0000", 2021-03-24T21:58:25.455 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999895 2021-03-24T21:58:25.455 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.456 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.456 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:58:25.456 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.195208+0000", 2021-03-24T21:58:25.456 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000237197 2021-03-24T21:58:25.456 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.457 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.457 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:58:25.457 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.195445+0000", 2021-03-24T21:58:25.457 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1628e-05 2021-03-24T21:58:25.458 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.458 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.458 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:58:25.458 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.195467+0000", 2021-03-24T21:58:25.458 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0105999999999999e-05 2021-03-24T21:58:25.459 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.459 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.459 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:58:25.459 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.195477+0000", 2021-03-24T21:58:25.459 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000111798 2021-03-24T21:58:25.460 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.460 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.460 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:58:25.460 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.195589+0000", 2021-03-24T21:58:25.460 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000122922 2021-03-24T21:58:25.461 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.461 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.461 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:58:25.461 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.195712+0000", 2021-03-24T21:58:25.462 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00020389400000000001 2021-03-24T21:58:25.462 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.462 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.462 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 2,5", 2021-03-24T21:58:25.462 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.195916+0000", 2021-03-24T21:58:25.463 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0021605980000000001 2021-03-24T21:58:25.463 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.463 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.463 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:58:25.463 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.198076+0000", 2021-03-24T21:58:25.464 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021071492000000001 2021-03-24T21:58:25.464 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.464 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.464 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T21:58:25.464 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.219148+0000", 2021-03-24T21:58:25.465 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034745430000000001 2021-03-24T21:58:25.465 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.465 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.465 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:58:25.465 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.222622+0000", 2021-03-24T21:58:25.465 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014299399999999999 2021-03-24T21:58:25.466 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.466 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.466 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T21:58:25.466 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.222765+0000", 2021-03-24T21:58:25.467 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.7608999999999994e-05 2021-03-24T21:58:25.467 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.467 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.467 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:58:25.467 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.222853+0000", 2021-03-24T21:58:25.468 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027634101000000001 2021-03-24T21:58:25.468 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:58:25.468 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:58:25.468 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:58:25.468 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.469 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.469 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:2552 7.19 7:9c4f4ea5:::smithi12556470-41:head [setxattr _header (57) in=7b,truncate 2067327] snapc b7=[b7,b4] ondisk+write+known_if_redirected e290)", 2021-03-24T21:58:25.469 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:57:40.195498+0000", 2021-03-24T21:58:25.469 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 45.009475715999997, 2021-03-24T21:58:25.469 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027738336999999998, 2021-03-24T21:58:25.470 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:58:25.470 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:58:25.470 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:58:25.470 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T21:58:25.470 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:58:25.471 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 2552 2021-03-24T21:58:25.471 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.471 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:58:25.471 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.471 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:58:25.472 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.195498+0000", 2021-03-24T21:58:25.472 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:58:25.472 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.472 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.472 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:58:25.473 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.195498+0000", 2021-03-24T21:58:25.473 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2021-03-24T21:58:25.473 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.473 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.473 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:58:25.474 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.195496+0000", 2021-03-24T21:58:25.474 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.7059999999999998e-06 2021-03-24T21:58:25.474 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.474 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.474 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:58:25.475 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.195503+0000", 2021-03-24T21:58:25.475 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1063999999999999e-05 2021-03-24T21:58:25.475 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.475 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.476 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:58:25.476 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.195514+0000", 2021-03-24T21:58:25.476 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0010000000000001e-06 2021-03-24T21:58:25.476 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.476 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.477 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:58:25.477 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.195519+0000", 2021-03-24T21:58:25.477 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0012496709999999999 2021-03-24T21:58:25.477 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.477 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.478 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:58:25.478 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.196769+0000", 2021-03-24T21:58:25.478 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.9450000000000005e-05 2021-03-24T21:58:25.478 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.478 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.479 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:58:25.479 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.196868+0000", 2021-03-24T21:58:25.479 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016830999999999999 2021-03-24T21:58:25.479 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.479 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.480 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 2,5", 2021-03-24T21:58:25.480 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.197037+0000", 2021-03-24T21:58:25.480 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0010866109999999999 2021-03-24T21:58:25.480 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.480 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.481 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:58:25.481 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.198123+0000", 2021-03-24T21:58:25.481 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021033596000000002 2021-03-24T21:58:25.481 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.481 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.482 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T21:58:25.482 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.219157+0000", 2021-03-24T21:58:25.482 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0038287629999999998 2021-03-24T21:58:25.482 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.483 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.483 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:58:25.483 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.222986+0000", 2021-03-24T21:58:25.483 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011129 2021-03-24T21:58:25.483 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.484 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.484 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T21:58:25.484 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.223097+0000", 2021-03-24T21:58:25.484 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000139052 2021-03-24T21:58:25.484 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.485 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.485 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:58:25.485 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.223236+0000", 2021-03-24T21:58:25.485 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027738336999999998 2021-03-24T21:58:25.485 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:58:25.486 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:58:25.486 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:58:25.486 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.486 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.486 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:2553 7.19 7:9c4f4ea5:::smithi12556470-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e290)", 2021-03-24T21:58:25.487 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:57:40.195525+0000", 2021-03-24T21:58:25.487 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 45.009448429999999, 2021-03-24T21:58:25.487 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028274652000000001, 2021-03-24T21:58:25.487 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:58:25.487 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T21:58:25.488 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:58:25.488 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T21:58:25.488 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:58:25.488 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 2553 2021-03-24T21:58:25.488 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.488 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:58:25.489 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.489 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:58:25.489 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.195525+0000", 2021-03-24T21:58:25.489 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:58:25.489 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.490 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.490 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:58:25.490 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.195525+0000", 2021-03-24T21:58:25.490 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2021-03-24T21:58:25.490 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.491 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.491 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:58:25.491 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.195524+0000", 2021-03-24T21:58:25.491 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.932e-06 2021-03-24T21:58:25.492 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.492 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.492 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:58:25.492 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.195527+0000", 2021-03-24T21:58:25.492 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8289999999999996e-06 2021-03-24T21:58:25.493 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.493 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.493 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:58:25.493 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.195533+0000", 2021-03-24T21:58:25.494 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2919999999999997e-06 2021-03-24T21:58:25.494 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.494 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.494 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:58:25.494 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.195537+0000", 2021-03-24T21:58:25.495 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00185119 2021-03-24T21:58:25.495 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.495 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.495 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:58:25.495 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.197389+0000", 2021-03-24T21:58:25.496 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014777599999999999 2021-03-24T21:58:25.496 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.496 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.496 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for rw locks", 2021-03-24T21:58:25.496 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.197536+0000", 2021-03-24T21:58:25.497 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025805392 2021-03-24T21:58:25.497 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.497 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.497 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:58:25.498 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.223342+0000", 2021-03-24T21:58:25.498 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00019874500000000001 2021-03-24T21:58:25.498 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.498 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.498 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:58:25.499 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.223541+0000", 2021-03-24T21:58:25.499 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00025904299999999998 2021-03-24T21:58:25.499 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.499 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.500 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:58:25.500 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.223800+0000", 2021-03-24T21:58:25.500 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028274652000000001 2021-03-24T21:58:25.500 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:58:25.500 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:58:25.501 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:58:25.501 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.501 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.501 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:2557 7.11 7:8dd64f9e:::smithi12556470-49:head [write 484134~708729] snapc b7=[b7,b4] ondisk+write+known_if_redirected e290)", 2021-03-24T21:58:25.501 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:57:40.204419+0000", 2021-03-24T21:58:25.502 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 45.000554317000002, 2021-03-24T21:58:25.502 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.054577542, 2021-03-24T21:58:25.502 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:58:25.502 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:58:25.503 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:58:25.503 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T21:58:25.503 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:58:25.503 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 2557 2021-03-24T21:58:25.504 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.504 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:58:25.504 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.504 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:58:25.504 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.204419+0000", 2021-03-24T21:58:25.505 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:58:25.505 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.505 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.505 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:58:25.505 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.204419+0000", 2021-03-24T21:58:25.506 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2021-03-24T21:58:25.506 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.506 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.506 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:58:25.507 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.204418+0000", 2021-03-24T21:58:25.507 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00064114499999999997 2021-03-24T21:58:25.507 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.507 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.507 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:58:25.508 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.205059+0000", 2021-03-24T21:58:25.508 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3905e-05 2021-03-24T21:58:25.508 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.508 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.509 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:58:25.509 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.205073+0000", 2021-03-24T21:58:25.509 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4770000000000002e-06 2021-03-24T21:58:25.509 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.509 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.510 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:58:25.510 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.205078+0000", 2021-03-24T21:58:25.510 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.0301000000000002e-05 2021-03-24T21:58:25.510 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.511 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.511 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:58:25.511 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.205168+0000", 2021-03-24T21:58:25.511 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010134200000000001 2021-03-24T21:58:25.511 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.512 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.512 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:58:25.512 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.205270+0000", 2021-03-24T21:58:25.512 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000210287 2021-03-24T21:58:25.513 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.513 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.513 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 2,3", 2021-03-24T21:58:25.513 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.205480+0000", 2021-03-24T21:58:25.513 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.017867886999999999 2021-03-24T21:58:25.514 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.514 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.514 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T21:58:25.514 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.223348+0000", 2021-03-24T21:58:25.514 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029882784999999999 2021-03-24T21:58:25.515 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.515 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.515 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:58:25.515 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.253231+0000", 2021-03-24T21:58:25.516 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0055580200000000003 2021-03-24T21:58:25.516 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.516 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.516 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:58:25.516 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.258789+0000", 2021-03-24T21:58:25.517 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010166999999999999 2021-03-24T21:58:25.517 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.517 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.517 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T21:58:25.518 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.258890+0000", 2021-03-24T21:58:25.518 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010621 2021-03-24T21:58:25.518 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.518 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.518 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:58:25.519 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.258997+0000", 2021-03-24T21:58:25.519 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.054577542 2021-03-24T21:58:25.519 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:58:25.519 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:58:25.520 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:58:25.520 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.520 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.520 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:2558 7.11 7:8dd64f9e:::smithi12556470-49:head [write 1883658~200476] snapc b7=[b7,b4] ondisk+write+known_if_redirected e290)", 2021-03-24T21:58:25.520 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:57:40.209863+0000", 2021-03-24T21:58:25.521 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 44.995110857, 2021-03-24T21:58:25.521 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.049555218999999998, 2021-03-24T21:58:25.521 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:58:25.521 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:58:25.522 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:58:25.522 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T21:58:25.522 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:58:25.522 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 2558 2021-03-24T21:58:25.522 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.523 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:58:25.523 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.523 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:58:25.523 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.209863+0000", 2021-03-24T21:58:25.523 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:58:25.524 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.524 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.524 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:58:25.524 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.209863+0000", 2021-03-24T21:58:25.525 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2021-03-24T21:58:25.525 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.525 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.525 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:58:25.525 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.209861+0000", 2021-03-24T21:58:25.526 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00020491199999999999 2021-03-24T21:58:25.526 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.526 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.526 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:58:25.526 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.210066+0000", 2021-03-24T21:58:25.527 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2717e-05 2021-03-24T21:58:25.527 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.527 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.527 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:58:25.527 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.210079+0000", 2021-03-24T21:58:25.528 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.7589999999999997e-06 2021-03-24T21:58:25.528 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.528 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.528 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:58:25.528 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.210083+0000", 2021-03-24T21:58:25.529 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000103578 2021-03-24T21:58:25.529 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.529 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.529 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:58:25.530 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.210187+0000", 2021-03-24T21:58:25.530 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4052e-05 2021-03-24T21:58:25.530 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.530 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.530 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:58:25.531 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.210281+0000", 2021-03-24T21:58:25.531 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000160385 2021-03-24T21:58:25.531 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.531 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.531 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 2,3", 2021-03-24T21:58:25.532 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.210441+0000", 2021-03-24T21:58:25.532 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.012921141000000001 2021-03-24T21:58:25.532 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.532 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.532 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T21:58:25.533 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.223362+0000", 2021-03-24T21:58:25.533 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.035776595000000001 2021-03-24T21:58:25.533 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.533 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.533 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:58:25.534 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.259139+0000", 2021-03-24T21:58:25.534 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.5758e-05 2021-03-24T21:58:25.534 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.534 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.534 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:58:25.535 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.259235+0000", 2021-03-24T21:58:25.535 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000102464 2021-03-24T21:58:25.535 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.535 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.536 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T21:58:25.536 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.259337+0000", 2021-03-24T21:58:25.536 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0425000000000003e-05 2021-03-24T21:58:25.536 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.536 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.537 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:58:25.537 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.259418+0000", 2021-03-24T21:58:25.537 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.049555218999999998 2021-03-24T21:58:25.537 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:58:25.537 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:58:25.538 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:58:25.538 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.538 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.538 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:2559 7.11 7:8dd64f9e:::smithi12556470-49:head [setxattr _header (57) in=7b,truncate 2084134] snapc b7=[b7,b4] ondisk+write+known_if_redirected e290)", 2021-03-24T21:58:25.538 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:57:40.210092+0000", 2021-03-24T21:58:25.539 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 44.994881223, 2021-03-24T21:58:25.539 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.049710130999999998, 2021-03-24T21:58:25.539 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:58:25.539 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:58:25.539 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:58:25.540 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T21:58:25.540 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:58:25.540 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 2559 2021-03-24T21:58:25.540 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.540 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:58:25.541 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.541 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:58:25.541 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.210092+0000", 2021-03-24T21:58:25.541 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:58:25.542 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.542 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.542 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:58:25.542 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.210092+0000", 2021-03-24T21:58:25.542 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2021-03-24T21:58:25.543 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.543 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.543 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:58:25.543 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.210091+0000", 2021-03-24T21:58:25.543 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8589999999999998e-06 2021-03-24T21:58:25.544 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.544 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.544 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:58:25.544 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.210095+0000", 2021-03-24T21:58:25.544 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4030000000000007e-06 2021-03-24T21:58:25.545 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.545 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.545 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:58:25.545 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.210104+0000", 2021-03-24T21:58:25.545 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6840000000000001e-06 2021-03-24T21:58:25.546 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.546 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.546 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:58:25.546 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.210106+0000", 2021-03-24T21:58:25.546 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0010918010000000001 2021-03-24T21:58:25.547 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.547 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.547 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:58:25.547 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.211198+0000", 2021-03-24T21:58:25.547 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015686199999999999 2021-03-24T21:58:25.548 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.548 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.548 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:58:25.548 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.211355+0000", 2021-03-24T21:58:25.548 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00025926900000000001 2021-03-24T21:58:25.549 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.549 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.549 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 2,3", 2021-03-24T21:58:25.549 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.211614+0000", 2021-03-24T21:58:25.549 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011760836 2021-03-24T21:58:25.550 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.550 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.550 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T21:58:25.550 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.223375+0000", 2021-03-24T21:58:25.551 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.036157878999999997 2021-03-24T21:58:25.551 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.551 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.551 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:58:25.551 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.259533+0000", 2021-03-24T21:58:25.551 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2786000000000005e-05 2021-03-24T21:58:25.552 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.552 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.552 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:58:25.552 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.259596+0000", 2021-03-24T21:58:25.553 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000106619 2021-03-24T21:58:25.553 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.553 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.553 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T21:58:25.553 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.259702+0000", 2021-03-24T21:58:25.554 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.9850000000000001e-05 2021-03-24T21:58:25.554 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.554 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.554 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:58:25.554 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.259802+0000", 2021-03-24T21:58:25.555 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.049710130999999998 2021-03-24T21:58:25.555 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:58:25.555 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:58:25.555 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:58:25.555 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.555 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.556 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:2560 7.11 7:8dd64f9e:::smithi12556470-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e290)", 2021-03-24T21:58:25.556 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:57:40.210109+0000", 2021-03-24T21:58:25.556 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 44.994864165000003, 2021-03-24T21:58:25.556 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.050155837000000002, 2021-03-24T21:58:25.557 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T21:58:25.557 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T21:58:25.557 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T21:58:25.557 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T21:58:25.558 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:58:25.558 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 2560 2021-03-24T21:58:25.558 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.558 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T21:58:25.558 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.559 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T21:58:25.559 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.210109+0000", 2021-03-24T21:58:25.559 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T21:58:25.559 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.560 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.560 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T21:58:25.560 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.210109+0000", 2021-03-24T21:58:25.560 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2021-03-24T21:58:25.560 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.561 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.561 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T21:58:25.561 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.210109+0000", 2021-03-24T21:58:25.561 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0779999999999998e-06 2021-03-24T21:58:25.561 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.562 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.562 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T21:58:25.562 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.210111+0000", 2021-03-24T21:58:25.562 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3440000000000001e-06 2021-03-24T21:58:25.562 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.563 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.563 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T21:58:25.563 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.210114+0000", 2021-03-24T21:58:25.563 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6359999999999998e-06 2021-03-24T21:58:25.563 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.564 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.564 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T21:58:25.564 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.210117+0000", 2021-03-24T21:58:25.564 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0018809759999999999 2021-03-24T21:58:25.565 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.565 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.565 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:58:25.565 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.211998+0000", 2021-03-24T21:58:25.565 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.8189999999999996e-05 2021-03-24T21:58:25.566 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.566 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.566 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for rw locks", 2021-03-24T21:58:25.566 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.212066+0000", 2021-03-24T21:58:25.567 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.047829064999999997 2021-03-24T21:58:25.567 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.567 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.567 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T21:58:25.567 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.259895+0000", 2021-03-24T21:58:25.568 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016446700000000001 2021-03-24T21:58:25.568 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.568 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.568 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T21:58:25.568 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.260060+0000", 2021-03-24T21:58:25.569 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000205427 2021-03-24T21:58:25.569 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T21:58:25.569 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T21:58:25.569 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T21:58:25.569 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.260265+0000", 2021-03-24T21:58:25.570 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.050155837000000002 2021-03-24T21:58:25.570 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:58:25.570 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:58:25.570 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:58:25.570 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T21:58:25.571 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T21:58:25.571 INFO:teuthology.orchestra.run.smithi142.stdout:} 2021-03-24T21:58:25.571 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:25.572 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-24T21:58:25.573 INFO:tasks.rados.rados.0.smithi125.stdout:1169: done (3 left) 2021-03-24T21:58:25.573 INFO:tasks.rados.rados.0.smithi125.stdout:1172: done (2 left) 2021-03-24T21:58:25.574 INFO:tasks.rados.rados.0.smithi125.stdout:1174: done (1 left) 2021-03-24T21:58:25.574 INFO:tasks.rados.rados.0.smithi125.stdout:1175: done (0 left) 2021-03-24T21:58:25.574 INFO:tasks.rados.rados.0.smithi125.stdout:1176: snap_remove snap 111 2021-03-24T21:58:25.587 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:25 smithi125 conmon[24927]: cluster 2021-03-24T21:58: 2021-03-24T21:58:25.587 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:25 smithi125 conmon[24927]: 23.865902+0000 osd.0 (osd.0) 82 : cluster [DBG] 5.5 scrub ok 2021-03-24T21:58:25.587 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:25 smithi125 conmon[24927]: cluster 2021-03-24T21:58:25.587 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:25 smithi125 conmon[24927]: 2021-03-24T21:58:23.942025+0000 osd.5 (osd.5) 82 : cluster [DBG] 2021-03-24T21:58:25.588 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:25 smithi125 conmon[24927]: 7.b scrub ok 2021-03-24T21:58:25.588 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:25 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:58:25.588 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:25 smithi125 conmon[24927]: 21:58:24.418884+0000 osd.2 (osd.2) 63 : cluster [DBG] 3.1d scrub ok 2021-03-24T21:58:25.588 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:25 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:58:25.588 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:25 smithi125 conmon[24927]: 21:58:25.181539+0000 mon.a (mon. 2021-03-24T21:58:25.589 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:25 smithi125 conmon[24927]: 0) 1026 : cluster [DBG] osdmap e335: 8 total, 8 up, 6 in 2021-03-24T21:58:25.589 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:25 smithi125 conmon[19936]: cluster 2021-03-24T21:58: 2021-03-24T21:58:25.589 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:25 smithi125 conmon[19936]: 23.865902+0000 osd.0 (osd.0) 82 : cluster [DBG] 5.5 scrub ok 2021-03-24T21:58:25.589 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:25 smithi125 conmon[19936]: cluster 2021-03-24T21:58:25.590 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:25 smithi125 conmon[19936]: 2021-03-24T21:58:23.942025+0000 2021-03-24T21:58:25.590 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:25 smithi125 conmon[19936]: osd.5 (osd.5) 82 : cluster 2021-03-24T21:58:25.590 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:25 smithi125 conmon[19936]: [DBG] 7.b scrub ok 2021-03-24T21:58:25.590 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:25 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:58:25.591 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:25 smithi125 conmon[19936]: 03-24T21:58:24. 2021-03-24T21:58:25.591 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:25 smithi125 conmon[19936]: 418884+0000 osd.2 (osd.2 2021-03-24T21:58:25.591 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:25 smithi125 conmon[19936]: ) 63 : cluster [DBG] 3.1d scrub ok 2021-03-24T21:58:25.591 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:25 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:58:25.591 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:25 smithi125 conmon[19936]: 21:58:25.181539+0000 mon.a (mon.0) 1026 : cluster [DBG] osdmap e335: 8 total, 8 up, 6 in 2021-03-24T21:58:25.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:25 smithi142 conmon[20125]: cluster 2021-03-24T21:58:23.865902+0000 osd.0 (osd.0) 82 : cluster [DBG] 2021-03-24T21:58:25.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:25 smithi142 conmon[20125]: 5.5 scrub ok 2021-03-24T21:58:25.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:25 smithi142 conmon[20125]: cluster 2021-03-24T21:58:23.942025+0000 osd.5 (osd.5) 82 : cluster [DBG] 7.b scrub ok 2021-03-24T21:58:25.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:25 smithi142 conmon[20125]: cluster 2021-03-24T21:58:24.418884+0000 osd.2 (osd.2) 63 : cluster [DBG] 3.1d scrub ok 2021-03-24T21:58:25.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:25 smithi142 conmon[20125]: cluster 2021-03-24T21:58:25.181539+0000 mon.a (mon.0) 1026 : cluster [DBG] osdmap e335: 8 total, 8 up, 6 in 2021-03-24T21:58:25.634 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:25.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:25.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:58:25.740 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:25.787 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:25.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:25.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:25.893 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:25.905 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:58:25.950 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:25.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:25.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:26.057 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:26.115 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:26.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:26.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:26.127 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.0 dump_ops_in_flight 2021-03-24T21:58:26.195 INFO:tasks.rados.rados.0.smithi125.stdout:1176: done (0 left) 2021-03-24T21:58:26.195 INFO:tasks.rados.rados.0.smithi125.stdout:1177: snap_remove snap 105 2021-03-24T21:58:26.220 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:26.267 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:26.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:26.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:26.373 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:26.415 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:26.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:26.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:26.520 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:58:26.579 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:58:26.583 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:26 smithi142 conmon[20125]: cluster 2021-03-24T21:58:24.795770+0000 mgr.y (mgr.14144) 436 : cluster 2021-03-24T21:58:26.584 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:26 smithi142 conmon[20125]: [DBG] pgmap v664: 168 pgs: 168 active+clean; 180 MiB data, 585 MiB used, 536 GiB / 536 GiB avail; 3.7 MiB/s rd, 2.7 MiB/s wr, 12 op/s 2021-03-24T21:58:26.584 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:26 smithi142 conmon[20125]: cluster 2021-03-24T21:58:24.967379+0000 osd.7 (osd.7) 76 : cluster [DBG] 2021-03-24T21:58:26.584 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:26 smithi142 conmon[20125]: 6.1 scrub ok 2021-03-24T21:58:26.585 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:26 smithi142 conmon[20125]: cluster 2021-03-24T21:58:26.193152+0000 mon.a (mon.0) 1027 2021-03-24T21:58:26.585 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:26 smithi142 conmon[20125]: : cluster [DBG] osdmap e336: 8 total, 8 up, 6 in 2021-03-24T21:58:26.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:26.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:58:26.659 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:58:26.687 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:26.739 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:26.742 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:26 smithi125 conmon[24927]: cluster 2021-03-24T21:58:24.795770+0000 2021-03-24T21:58:26.743 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:26 smithi125 conmon[24927]: mgr.y (mgr.14144) 436 : cluster [DBG] pgmap v664: 168 pgs: 168 active+clean; 180 MiB data, 585 MiB used, 536 GiB / 536 GiB avail; 3.7 MiB/s rd, 2.7 MiB/s wr, 12 op/s 2021-03-24T21:58:26.743 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:26 smithi125 conmon[24927]: cluster 2021-03-24T21:58 2021-03-24T21:58:26.743 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:26 smithi125 conmon[24927]: :24.967379+0000 osd.7 (osd.7) 76 : cluster [DBG] 6.1 scrub ok 2021-03-24T21:58:26.743 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:26 smithi125 conmon[24927]: cluster 2021-03-24T21:58:26. 2021-03-24T21:58:26.744 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:26 smithi125 conmon[24927]: 193152+0000 mon.a (mon.0) 1027 : cluster [DBG] osdmap e336: 8 total, 8 up, 6 in 2021-03-24T21:58:26.744 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:26 smithi125 conmon[19936]: cluster 2021-03-24T21:58:24. 2021-03-24T21:58:26.744 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:26 smithi125 conmon[19936]: 795770+0000 mgr.y (mgr.14144) 436 : cluster [DBG] pgmap v664: 168 pgs: 168 active+clean; 180 MiB data, 585 MiB used, 536 GiB / 536 GiB avail; 3.7 MiB/s rd, 2.7 MiB/s wr, 12 op/s 2021-03-24T21:58:26.745 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:26 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:58:26.745 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:26 smithi125 conmon[19936]: :58:24.967379+0000 osd.7 (osd.7) 2021-03-24T21:58:26.745 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:26 smithi125 conmon[19936]: 76 : cluster [DBG] 2021-03-24T21:58:26.745 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:26 smithi125 conmon[19936]: 6.1 scrub ok 2021-03-24T21:58:26.746 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:26 smithi125 conmon[19936]: cluster 2021-03-24T21:58: 2021-03-24T21:58:26.746 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:26 smithi125 conmon[19936]: 26.193152+0000 mon.a (mon.0) 1027 : cluster [DBG] osdmap e336: 8 total, 8 up, 6 in 2021-03-24T21:58:26.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:26.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:58:26.848 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:26.911 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:26.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:26.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:58:27.018 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:27.071 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:27.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:27.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:27.178 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:27.202 INFO:tasks.rados.rados.0.smithi125.stdout:1177: done (0 left) 2021-03-24T21:58:27.202 INFO:tasks.rados.rados.0.smithi125.stdout:1178: read oid 34 snap -1 2021-03-24T21:58:27.202 INFO:tasks.rados.rados.0.smithi125.stdout:1178: expect (ObjNum 411 snap 108 seq_num 411) 2021-03-24T21:58:27.202 INFO:tasks.rados.rados.0.smithi125.stdout:1179: copy_from oid 39 from oid 38 current snap is 116 2021-03-24T21:58:27.203 INFO:tasks.rados.rados.0.smithi125.stdout:1180: read oid 29 snap -1 2021-03-24T21:58:27.203 INFO:tasks.rados.rados.0.smithi125.stdout:1180: expect (ObjNum 401 snap 104 seq_num 401) 2021-03-24T21:58:27.203 INFO:tasks.rados.rados.0.smithi125.stdout:1181: snap_create 2021-03-24T21:58:27.219 INFO:tasks.rados.rados.0.smithi125.stdout:1180: expect (ObjNum 401 snap 104 seq_num 401) 2021-03-24T21:58:27.225 INFO:teuthology.orchestra.run.smithi125.stderr:nodeep-scrub is unset 2021-03-24T21:58:27.251 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:27.253 INFO:journalctl@ceph.osd.3.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[37997]: debug 2021-03-24T21:58:27.254 INFO:journalctl@ceph.osd.3.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[37997]: 2021-03-24T21:58:27.222+0000 7f1716bda700 -1 osd.3 pg_epoch: 337 pg[7.14( v 336'258 (0'0,336'258] local-lis/les=204/205 n=5 ec=198/99 lis/c=204/204 les/c/f=205/205/0 sis=204) [3,2,5] r=0 lpr=204 crt=336'258 lcod 335'256 mlcod 335'256 active+clean+snaptrim trimq=[d3~1] ps=[d4~2,d7~3,e0~1]] removing snap head 2021-03-24T21:58:27.254 INFO:journalctl@ceph.osd.3.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[37997]: 2021-03-24T21:58:27.254 INFO:journalctl@ceph.osd.0.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[28026]: debug 2021-03-24T21:58:27.254 INFO:journalctl@ceph.osd.0.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[28026]: 2021-03-24T21:58:27.214+0000 7f61b847f700 -1 osd.0 pg_epoch: 337 pg[7.1b( v 337'381 (0'0,337'381] local-lis/les=282/283 n=7 ec=275/99 lis/c=282/282 les/c/f=283/283/0 sis=282) [0,2,3] r=0 lpr=282 luod=336'379 crt=337'381 lcod 335'377 mlcod 335'377 active+clean+snaptrim trimq=[d3~1] ps=[d4~2,d7~3,e0~1]] removing snap head 2021-03-24T21:58:27.255 INFO:journalctl@ceph.osd.0.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[28026]: 2021-03-24T21:58:27.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:27.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:58:27.357 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:58:27.392 INFO:tasks.rados.rados.0.smithi125.stdout:1178: expect (ObjNum 411 snap 108 seq_num 411) 2021-03-24T21:58:27.413 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:58:27.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:27.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:58:27.520 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:27.534 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[19936]: cluster 2021-03-24T21:58:25.969847+0000 osd.7 (osd. 2021-03-24T21:58:27.534 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[19936]: 7) 77 : cluster [DBG] 3.f scrub ok 2021-03-24T21:58:27.534 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[19936]: audit 2021-03-24T21 2021-03-24T21:58:27.535 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[19936]: :58:26.394476+0000 mon.c (mon.2) 54 : audit [INF] from='client.? v1:172.21.15.125:0/3506597471' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:58:27.535 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[19936]: audit 2021 2021-03-24T21:58:27.535 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[19936]: -03-24T21:58:26.394856+0000 2021-03-24T21:58:27.535 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[19936]: mon.a (mon.0) 1028 : audit [INF] from='client.35090 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:58:27.535 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[19936]: 2021-03-24T21:58:27.536 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[19936]: cluster 2021-03-24T21:58:26 2021-03-24T21:58:27.536 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[19936]: .464444+0000 osd.2 (osd.2) 64 : cluster 2021-03-24T21:58:27.536 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[19936]: [DBG] 5.a scrub ok 2021-03-24T21:58:27.536 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[19936]: cluster 2021-03-24T21:58:27.537 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[19936]: 2021-03-24T21:58: 2021-03-24T21:58:27.537 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[19936]: 26.697908+0000 osd.4 (osd.4) 85 2021-03-24T21:58:27.537 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[19936]: : cluster [DBG] 4.1a scrub ok 2021-03-24T21:58:27.537 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[19936]: cluster 2021-03-24T21:58:27.537 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[19936]: 2021-03-24T21: 2021-03-24T21:58:27.538 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[19936]: 58:27.187704+0000 mon.a (mon.0) 2021-03-24T21:58:27.538 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[19936]: 1029 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T21:58:27.538 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:58:27.538 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[19936]: 03-24T21:58:27.187813+0000 mon.a 2021-03-24T21:58:27.539 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[19936]: (mon.0) 1030 : cluster [INF] Cluster is now healthy 2021-03-24T21:58:27.539 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[19936]: audit 2021 2021-03-24T21:58:27.539 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[19936]: -03-24T21:58:27.197251+0000 2021-03-24T21:58:27.539 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[19936]: mon.a (mon.0) 1031 : audit [INF] from='client.35090 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T21:58:27.540 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[19936]: 2021-03-24T21:58:27.540 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[19936]: cluster 2021-03-24T21:58:27.540 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[19936]: 2021-03-24T21:58:27.197838+0000 mon.a 2021-03-24T21:58:27.540 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[19936]: (mon.0) 1032 : cluster [DBG] osdmap e337: 8 total, 8 up, 6 in 2021-03-24T21:58:27.541 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:58:27.541 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[24927]: :58:25.969847+0000 osd.7 (osd.7) 77 : cluster [DBG] 3.f scrub ok 2021-03-24T21:58:27.541 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[24927]: audit 2021-03-24T21:58:27.542 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[24927]: 2021-03-24T21:58:26.394476 2021-03-24T21:58:27.542 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[24927]: +0000 mon.c (mon.2) 54 : audit [INF] 2021-03-24T21:58:27.542 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[24927]: from='client.? v1:172.21.15.125:0/3506597471' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:58:27.542 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[24927]: audit 2021- 2021-03-24T21:58:27.543 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[24927]: 03-24T21:58:26.394856+0000 mon.a 2021-03-24T21:58:27.543 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[24927]: (mon.0) 1028 : audit [INF] from='client.35090 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:58:27.543 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[24927]: cluster 2021-03-24T21:58:27.543 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[24927]: 2021-03-24T21:58:26.464444+0000 2021-03-24T21:58:27.543 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[24927]: osd.2 (osd.2) 64 : cluster [DBG] 2021-03-24T21:58:27.544 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[24927]: 5.a scrub ok 2021-03-24T21:58:27.544 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[24927]: cluster 2021-03-24T21:58 2021-03-24T21:58:27.544 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[24927]: :26.697908+0000 osd.4 (osd.4) 85 2021-03-24T21:58:27.544 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[24927]: : cluster [DBG] 4.1a scrub ok 2021-03-24T21:58:27.545 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:58:27.545 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[24927]: 03-24T21:58:27.187704+0000 2021-03-24T21:58:27.545 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[24927]: mon.a (mon.0) 1029 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T21:58:27.545 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[24927]: 2021-03-24T21:58:27.545 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[24927]: cluster 2021-03-24T21:58: 2021-03-24T21:58:27.546 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[24927]: 27.187813+0000 mon.a (mon.0 2021-03-24T21:58:27.546 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[24927]: ) 1030 : cluster [INF] Cluster is now healthy 2021-03-24T21:58:27.546 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[24927]: audit 2021 2021-03-24T21:58:27.547 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[24927]: -03-24T21:58:27.197251+0000 mon.a 2021-03-24T21:58:27.547 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[24927]: (mon.0) 1031 : audit [INF] from='client.35090 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T21:58:27.547 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:58:27.547 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[24927]: 03-24T21:58:27.197838+0000 mon.a 2021-03-24T21:58:27.547 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:27 smithi125 conmon[24927]: (mon.0) 1032 : cluster [DBG] osdmap e337: 8 total, 8 up, 6 in 2021-03-24T21:58:27.551 INFO:tasks.rados.rados.0.smithi125.stdout:1179: finishing copy_from to smithi12556470-39 2021-03-24T21:58:27.552 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 39 v 49 (ObjNum 408 snap 106 seq_num 408) dirty exists 2021-03-24T21:58:27.552 INFO:tasks.rados.rados.0.smithi125.stdout:1179: finishing copy_from racing read to smithi12556470-39 2021-03-24T21:58:27.577 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:27.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:27.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:58:27.616 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 21:58:27 smithi142 conmon[27337]: debug 2021-03-24T21:58:27.202+0000 7f549af13700 -1 osd.5 pg_epoch: 337 pg[7.10( v 325'39 (0'0,325'39] local-lis/les=256/257 n=2 ec=198/99 lis/c=256/256 les/c/f=257/257/0 sis=256) [5,3,4] r=0 lpr=256 crt=325'39 lcod 318'37 mlcod 318'37 active+clean+snaptrim trimq=[d3~1] ps=[d4~2,d7~3,e0~1]] removing snap head 2021-03-24T21:58:27.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:27 smithi142 conmon[20125]: cluster 2021-03-24T21:58:25.969847+0000 osd.7 (osd.7) 77 : cluster [DBG] 2021-03-24T21:58:27.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:27 smithi142 conmon[20125]: 3.f scrub ok 2021-03-24T21:58:27.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:27 smithi142 conmon[20125]: audit 2021-03-24T21:58:26.394476+0000 mon.c (mon 2021-03-24T21:58:27.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:27 smithi142 conmon[20125]: .2) 54 : audit [INF] from='client.? v1:172.21.15.125:0/3506597471' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:58:27.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:27 smithi142 conmon[20125]: audit 2021-03-24T21:58: 2021-03-24T21:58:27.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:27 smithi142 conmon[20125]: 26.394856+0000 mon.a (mon.0) 1028 : 2021-03-24T21:58:27.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:27 smithi142 conmon[20125]: audit [INF] from='client.35090 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:58:27.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:27 smithi142 conmon[20125]: cluster 2021-03-24T21:58 2021-03-24T21:58:27.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:27 smithi142 conmon[20125]: :26.464444+0000 osd.2 (osd.2) 64 2021-03-24T21:58:27.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:27 smithi142 conmon[20125]: : cluster [DBG] 5.a scrub ok 2021-03-24T21:58:27.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:27 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:58:27.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:27 smithi142 conmon[20125]: 58:26.697908+0000 osd.4 (osd.4) 2021-03-24T21:58:27.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:27 smithi142 conmon[20125]: 85 : cluster [DBG] 4.1a scrub ok 2021-03-24T21:58:27.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:27 smithi142 conmon[20125]: cluster 2021-03-24T21 2021-03-24T21:58:27.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:27 smithi142 conmon[20125]: :58:27.187704+0000 mon.a (mon.0 2021-03-24T21:58:27.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:27 smithi142 conmon[20125]: ) 1029 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T21:58:27.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:27 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:58:27.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:27 smithi142 conmon[20125]: 21:58:27.187813+0000 2021-03-24T21:58:27.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:27 smithi142 conmon[20125]: mon.a (mon.0) 1030 : cluster [INF] 2021-03-24T21:58:27.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:27 smithi142 conmon[20125]: Cluster is now healthy 2021-03-24T21:58:27.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:27 smithi142 conmon[20125]: audit 2021-03-24T21:58:27.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:27 smithi142 conmon[20125]: 2021-03-24T21:58:27.197251+0000 2021-03-24T21:58:27.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:27 smithi142 conmon[20125]: mon.a (mon.0) 1031 : audit [INF] 2021-03-24T21:58:27.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:27 smithi142 conmon[20125]: from='client.35090 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T21:58:27.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:27 smithi142 conmon[20125]: cluster 2021-03-24T21:58:27.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:27 smithi142 conmon[20125]: 2021-03-24T21:58:27.197838+0000 mon.a 2021-03-24T21:58:27.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:27 smithi142 conmon[20125]: (mon.0) 1032 : cluster [DBG] osdmap e337: 8 total, 8 up, 6 in 2021-03-24T21:58:27.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:27 smithi142 conmon[20125]: 2021-03-24T21:58:27.681 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:27.729 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:27.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:27.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:58:27.835 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:27.887 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:27.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:27.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:58:27.993 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:28.048 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:28.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:28.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:58:28.155 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:28.202 INFO:tasks.rados.rados.0.smithi125.stdout:1178: done (3 left) 2021-03-24T21:58:28.202 INFO:tasks.rados.rados.0.smithi125.stdout:1179: done (2 left) 2021-03-24T21:58:28.203 INFO:tasks.rados.rados.0.smithi125.stdout:1180: done (1 left) 2021-03-24T21:58:28.203 INFO:tasks.rados.rados.0.smithi125.stdout:1181: done (0 left) 2021-03-24T21:58:28.203 INFO:tasks.rados.rados.0.smithi125.stdout:1182: snap_create 2021-03-24T21:58:28.218 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:28.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:28.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:58:28.323 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:28.380 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:28.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:28.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:58:28.486 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:28.542 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:28.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:28.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:28.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:28 smithi142 conmon[20125]: cluster 2021-03-24T21:58:26.796873 2021-03-24T21:58:28.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:28 smithi142 conmon[20125]: +0000 mgr.y (mgr.14144) 437 : cluster [DBG] pgmap v667: 168 pgs: 168 active+clean; 180 MiB data, 569 MiB used, 536 GiB / 536 GiB avail; 3.7 MiB/s rd, 2.7 MiB/s wr, 12 op/s 2021-03-24T21:58:28.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:28 smithi142 conmon[20125]: cluster 2021-03-24T21:58:26.887933+0000 osd.5 2021-03-24T21:58:28.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:28 smithi142 conmon[20125]: (osd.5) 83 : cluster [DBG] 6.1d scrub ok 2021-03-24T21:58:28.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:28 smithi142 conmon[20125]: cluster 2021-03-24T21:58:27. 2021-03-24T21:58:28.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:28 smithi142 conmon[20125]: 722529+0000 osd.4 (osd.4) 86 : cluster [DBG] 2021-03-24T21:58:28.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:28 smithi142 conmon[20125]: 4.c scrub ok 2021-03-24T21:58:28.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:28 smithi142 conmon[20125]: cluster 2021-03-24T21:58:28 2021-03-24T21:58:28.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:28 smithi142 conmon[20125]: .201081+0000 mon.a (mon.0) 1033 : cluster 2021-03-24T21:58:28.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:28 smithi142 conmon[20125]: [DBG] osdmap e338: 8 total, 8 up, 6 in 2021-03-24T21:58:28.648 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:28.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:28 smithi125 conmon[24927]: cluster 2021-03-24T21:58:26.796873+0000 mgr.y (mgr.14144) 437 2021-03-24T21:58:28.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:28 smithi125 conmon[24927]: : cluster [DBG] pgmap v667: 168 pgs: 168 active+clean; 180 MiB data, 569 MiB used, 536 GiB / 536 GiB avail; 3.7 MiB/s rd, 2.7 MiB/s wr, 12 op/s 2021-03-24T21:58:28.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:28 smithi125 conmon[24927]: cluster 2021-03-24T21:58: 2021-03-24T21:58:28.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:28 smithi125 conmon[24927]: 26.887933+0000 osd.5 (osd.5) 83 : cluster [DBG] 6.1d scrub ok 2021-03-24T21:58:28.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:28 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:58:28.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:28 smithi125 conmon[24927]: T21:58:27.722529+0000 osd.4 (osd.4) 86 2021-03-24T21:58:28.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:28 smithi125 conmon[24927]: : cluster [DBG] 4.c scrub ok 2021-03-24T21:58:28.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:28 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:58:28.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:28 smithi125 conmon[24927]: 03-24T21:58:28.201081+0000 mon.a (mon 2021-03-24T21:58:28.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:28 smithi125 conmon[24927]: .0) 1033 : cluster [DBG] osdmap e338: 8 total, 8 up, 6 in 2021-03-24T21:58:28.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:28 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:58:28.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:28 smithi125 conmon[19936]: 21:58:26.796873+0000 mgr.y (mgr.14144) 437 : cluster [DBG] pgmap v667: 168 pgs: 168 active+clean; 180 MiB data, 569 MiB used, 536 GiB / 536 GiB avail; 3.7 MiB/s rd, 2.7 MiB/s wr, 12 op/s 2021-03-24T21:58:28.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:28 smithi125 conmon[19936]: cluster 2021-03-24T21:58:28.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:28 smithi125 conmon[19936]: 2021-03- 2021-03-24T21:58:28.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:28 smithi125 conmon[19936]: 24T21: 2021-03-24T21:58:28.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:28 smithi125 conmon[19936]: 58:26. 2021-03-24T21:58:28.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:28 smithi125 conmon[19936]: 887933+0000 osd.5 ( 2021-03-24T21:58:28.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:28 smithi125 conmon[19936]: osd.5) 83 2021-03-24T21:58:28.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:28 smithi125 conmon[19936]: : cluster [DBG] 2021-03-24T21:58:28.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:28 smithi125 conmon[19936]: 6.1d scrub ok 2021-03-24T21:58:28.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:28 smithi125 conmon[19936]: cluster 2021-03-24T21:58:28.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:28 smithi125 conmon[19936]: 2021-03 2021-03-24T21:58:28.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:28 smithi125 conmon[19936]: -24T21 2021-03-24T21:58:28.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:28 smithi125 conmon[19936]: :58:27. 2021-03-24T21:58:28.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:28 smithi125 conmon[19936]: 722529+0000 osd.4 2021-03-24T21:58:28.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:28 smithi125 conmon[19936]: (osd.4) 2021-03-24T21:58:28.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:28 smithi125 conmon[19936]: 86 : cluster [DBG] 2021-03-24T21:58:28.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:28 smithi125 conmon[19936]: 4.c scrub ok 2021-03-24T21:58:28.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:28 smithi125 conmon[19936]: cluster 2021-03-24T21:58:28.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:28 smithi125 conmon[19936]: 2021-03 2021-03-24T21:58:28.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:28 smithi125 conmon[19936]: -24T21 2021-03-24T21:58:28.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:28 smithi125 conmon[19936]: :58:28 2021-03-24T21:58:28.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:28 smithi125 conmon[19936]: .201081+0000 mon.a 2021-03-24T21:58:28.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:28 smithi125 conmon[19936]: (mon.0 2021-03-24T21:58:28.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:28 smithi125 conmon[19936]: ) 1033 : cluster 2021-03-24T21:58:28.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:28 smithi125 conmon[19936]: [DBG] osdmap e338: 8 total, 8 up, 6 in 2021-03-24T21:58:28.704 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:28.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:28.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:58:28.810 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:28.864 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:28.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:28.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:28.972 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:58:29.027 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T21:58:29.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:29.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T21:58:29.133 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:29.171 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 7] out_osds: [6, 1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-03-24T21:58:29.171 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-24T21:58:29.172 INFO:tasks.thrashosds.thrasher:Killing osd 5, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2021-03-24T21:58:29.173 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2021-03-24T21:58:29.174 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl stop ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c@osd.5 2021-03-24T21:58:29.193 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:29.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:29.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:58:29.209 INFO:tasks.rados.rados.0.smithi125.stdout:1182: done (0 left) 2021-03-24T21:58:29.209 INFO:tasks.rados.rados.0.smithi125.stdout:1183: rollback oid 32 current snap is 118 2021-03-24T21:58:29.210 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 32 to 116 2021-03-24T21:58:29.210 INFO:tasks.rados.rados.0.smithi125.stdout:1184: read oid 19 snap -1 2021-03-24T21:58:29.210 INFO:tasks.rados.rados.0.smithi125.stdout:1184: expect deleted 2021-03-24T21:58:29.210 INFO:tasks.rados.rados.0.smithi125.stdout:1185: copy_from oid 10 from oid 11 current snap is 118 2021-03-24T21:58:29.211 INFO:tasks.rados.rados.0.smithi125.stdout:1186: snap_remove snap 115 2021-03-24T21:58:29.300 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:29.312 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 21:58:29 smithi142 systemd[1]: Stopping Ceph osd.5 for 3cfa339a-8cea-11eb-90b2-001a4aab830c... 2021-03-24T21:58:29.360 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:29.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:29.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:58:29.379 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T21:58:29.399 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T21:58:29.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:29 smithi125 conmon[19936]: cluster 2021-03-24T21:58:27.920673 2021-03-24T21:58:29.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:29 smithi125 conmon[19936]: +0000 osd.7 (osd.7) 78 : cluster [DBG] 5.19 scrub ok 2021-03-24T21:58:29.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:29 smithi125 conmon[19936]: cluster 2021-03-24T21:58:28.480604+0000 osd.2 (osd.2) 65 : cluster [DBG] 2021-03-24T21:58:29.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:29 smithi125 conmon[19936]: 5.8 scrub ok 2021-03-24T21:58:29.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:29 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:58:29.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:29 smithi125 conmon[19936]: 24T21:58:29.206061+0000 mon.a (mon.0) 1034 : cluster [DBG] osdmap e339: 8 total, 8 up, 6 in 2021-03-24T21:58:29.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:29 smithi125 conmon[24927]: cluster 2021-03-24T21:58:27.920673+0000 osd.7 (osd.7 2021-03-24T21:58:29.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:29 smithi125 conmon[24927]: ) 78 : cluster [DBG] 5.19 scrub ok 2021-03-24T21:58:29.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:29 smithi125 conmon[24927]: cluster 2021-03-24T21:58:28.480604+0000 osd.2 (osd.2) 65 : cluster [DBG] 2021-03-24T21:58:29.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:29 smithi125 conmon[24927]: 5.8 scrub ok 2021-03-24T21:58:29.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:29 smithi125 conmon[24927]: cluster 2021-03-24T21:58:29.206061+0000 2021-03-24T21:58:29.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:29 smithi125 conmon[24927]: mon.a (mon.0) 1034 : cluster [DBG] osdmap e339: 8 total, 8 up, 6 in 2021-03-24T21:58:29.420 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T21:58:29.442 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T21:58:29.463 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T21:58:29.466 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:29.519 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:29.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:29.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:58:29.607 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:29 smithi142 conmon[20125]: cluster 2021-03-24T21:58: 2021-03-24T21:58:29.607 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:29 smithi142 conmon[20125]: 27.920673+0000 osd.7 (osd.7) 78 : cluster [DBG] 5.19 scrub ok 2021-03-24T21:58:29.607 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:29 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:58:29.608 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:29 smithi142 conmon[20125]: -03-24T21:58:28.480604+0000 osd.2 (osd.2) 65 : cluster [DBG] 5.8 scrub ok 2021-03-24T21:58:29.608 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:29 smithi142 conmon[20125]: cluster 2021-03-24T21:58:29.608 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:29 smithi142 conmon[20125]: 2021-03-24T21:58:29.206061 2021-03-24T21:58:29.608 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:29 smithi142 conmon[20125]: +0000 mon.a (mon.0) 1034 : cluster [DBG] osdmap e339: 8 total, 8 up, 6 in 2021-03-24T21:58:29.609 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 21:58:29 smithi142 conmon[27337]: debug 2021-03-24T21:58:29.306+0000 7f54ba01d700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-03-24T21:58:29.609 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 21:58:29 smithi142 conmon[27337]: debug 2021-03-24T21:58:29.306+0000 7f54ba01d700 -1 osd.5 339 *** Got signal Terminated *** 2021-03-24T21:58:29.609 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 21:58:29 smithi142 conmon[27337]: debug 2021-03-24T21:58:29.306+0000 7f54ba01d700 -1 osd.5 339 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-03-24T21:58:29.609 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 21:58:29 smithi142 podman[60508]: 2021-03-24 21:58:29.412917633 +0000 UTC m=+0.197078266 container stop 97264781327ab14946f21d9751e0a39ea24c9db473f401fe3d0a3ff2dc5c9e9c (image=quay.ceph.io/ceph-ci/ceph@sha256:e9db06906b8d0a24f3ef6835ac5aa4b8eea2950a2560bad78f83d4f101eca0db, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Base Image, maintainer=Dimitri Savineau , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.18.0, GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06, RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d, org.label-schema.build-date=20201204, org.label-schema.license=GPLv2) 2021-03-24T21:58:29.610 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 21:58:29 smithi142 podman[60508]: 2021-03-24 21:58:29.509957115 +0000 UTC m=+0.294117661 container died 97264781327ab14946f21d9751e0a39ea24c9db473f401fe3d0a3ff2dc5c9e9c (image=quay.ceph.io/ceph-ci/ceph@sha256:e9db06906b8d0a24f3ef6835ac5aa4b8eea2950a2560bad78f83d4f101eca0db, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5) 2021-03-24T21:58:29.610 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 21:58:29 smithi142 podman[60508]: 97264781327ab14946f21d9751e0a39ea24c9db473f401fe3d0a3ff2dc5c9e9c 2021-03-24T21:58:29.623 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:29.680 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:29.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:29.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:58:29.785 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:29.846 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:29.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:29.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:29.954 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:30.017 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:30.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:30.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:58:30.121 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:30.176 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:30.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:30.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:58:30.193 INFO:tasks.rados.rados.0.smithi125.stdout:1186: done (3 left) 2021-03-24T21:58:30.193 INFO:tasks.rados.rados.0.smithi125.stdout:1187: write oid 12 current snap is 118 2021-03-24T21:58:30.193 INFO:tasks.rados.rados.0.smithi125.stdout:1187: seq_num 437 ranges {544659=715934,1738984=449146,2784436=560223} 2021-03-24T21:58:30.196 INFO:tasks.rados.rados.0.smithi125.stdout:1187: writing smithi12556470-12 from 544659 to 1260593 tid 1 2021-03-24T21:58:30.202 INFO:tasks.rados.rados.0.smithi125.stdout:1187: writing smithi12556470-12 from 1738984 to 2188130 tid 2 2021-03-24T21:58:30.210 INFO:tasks.rados.rados.0.smithi125.stdout:1187: writing smithi12556470-12 from 2784436 to 3344659 tid 3 2021-03-24T21:58:30.211 INFO:tasks.rados.rados.0.smithi125.stdout:1188: read oid 33 snap 113 2021-03-24T21:58:30.211 INFO:tasks.rados.rados.0.smithi125.stdout:1188: expect (ObjNum 298 snap 74 seq_num 298) 2021-03-24T21:58:30.211 INFO:tasks.rados.rados.0.smithi125.stdout:1189: write oid 26 current snap is 118 2021-03-24T21:58:30.211 INFO:tasks.rados.rados.0.smithi125.stdout:1189: seq_num 438 ranges {561466=791183,2081113=592804,3095506=265960} 2021-03-24T21:58:30.218 INFO:tasks.rados.rados.0.smithi125.stdout:1189: writing smithi12556470-26 from 561466 to 1352649 tid 1 2021-03-24T21:58:30.227 INFO:tasks.rados.rados.0.smithi125.stdout:1189: writing smithi12556470-26 from 2081113 to 2673917 tid 2 2021-03-24T21:58:30.232 INFO:tasks.rados.rados.0.smithi125.stdout:1189: writing smithi12556470-26 from 3095506 to 3361466 tid 3 2021-03-24T21:58:30.232 INFO:tasks.rados.rados.0.smithi125.stdout:1190: copy_from oid 30 from oid 23 current snap is 118 2021-03-24T21:58:30.233 INFO:tasks.rados.rados.0.smithi125.stdout:1183: finishing rollback tid 0 to smithi12556470-32 2021-03-24T21:58:30.233 INFO:tasks.rados.rados.0.smithi125.stdout:1183: finishing rollback tid 1 to smithi12556470-32 2021-03-24T21:58:30.233 INFO:tasks.rados.rados.0.smithi125.stdout:1183: finishing rollback tid 2 to smithi12556470-32 2021-03-24T21:58:30.233 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 32 v 393 (ObjNum 370 snap 95 seq_num 370) dirty exists 2021-03-24T21:58:30.233 INFO:tasks.rados.rados.0.smithi125.stdout:1185: finishing copy_from to smithi12556470-10 2021-03-24T21:58:30.234 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 10 v 309 (ObjNum 353 snap 89 seq_num 353) dirty exists 2021-03-24T21:58:30.234 INFO:tasks.rados.rados.0.smithi125.stdout:1185: finishing copy_from racing read to smithi12556470-10 2021-03-24T21:58:30.234 INFO:tasks.rados.rados.0.smithi125.stdout:1183: done (6 left) 2021-03-24T21:58:30.234 INFO:tasks.rados.rados.0.smithi125.stdout:1184: done (5 left) 2021-03-24T21:58:30.235 INFO:tasks.rados.rados.0.smithi125.stdout:1185: done (4 left) 2021-03-24T21:58:30.235 INFO:tasks.rados.rados.0.smithi125.stdout:1191: delete oid 1 current snap is 118 2021-03-24T21:58:30.246 INFO:tasks.rados.rados.0.smithi125.stdout:1191: done (4 left) 2021-03-24T21:58:30.247 INFO:tasks.rados.rados.0.smithi125.stdout:1192: write oid 25 current snap is 118 2021-03-24T21:58:30.247 INFO:tasks.rados.rados.0.smithi125.stdout:1192: seq_num 440 ranges {595080=541681,1798077=596473,3150352=244728} 2021-03-24T21:58:30.253 INFO:tasks.rados.rados.0.smithi125.stdout:1192: writing smithi12556470-25 from 595080 to 1136761 tid 1 2021-03-24T21:58:30.261 INFO:tasks.rados.rados.0.smithi125.stdout:1192: writing smithi12556470-25 from 1798077 to 2394550 tid 2 2021-03-24T21:58:30.267 INFO:tasks.rados.rados.0.smithi125.stdout:1192: writing smithi12556470-25 from 3150352 to 3395080 tid 3 2021-03-24T21:58:30.267 INFO:tasks.rados.rados.0.smithi125.stdout:1193: snap_create 2021-03-24T21:58:30.284 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:30.343 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:30.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:30.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:58:30.449 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:30.463 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03-24T21:58:28.797786+0000 mgr.y (mgr.14144) 438 : cluster [DBG] pgmap v670: 168 pgs: 168 active+clean; 169 MiB data, 557 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:58:30.464 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03-24T21:58:29.298220+0000 osd.3 (osd.3) 78 : cluster [DBG] 6.1b scrub ok 2021-03-24T21:58:30.464 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: audit 2021-03-24T21:58:29.353932+0000 mon.c (mon.2) 55 : audit [DBG] from='client.? v1:172.21.15.125:0/3953067220' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:58:30.464 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: 2021-03-24T21:58:30.464 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:58:30.465 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: 58:29.371779+0000 mon.a (mon.0) 1035 : cluster [DBG] osd.5 reported immediately failed by osd.0 2021-03-24T21:58:30.465 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03-24T21:58:29.371842+0000 mon.a 2021-03-24T21:58:30.465 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: (mon.0) 1036 : cluster [INF] osd.5 failed (root=default,host=smithi142) (connection refused reported by osd.0) 2021-03-24T21:58:30.465 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03-24T21:58:29.372072+0000 mon.a (mon. 2021-03-24T21:58:30.465 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: 0) 1037 : cluster [DBG] osd.5 reported immediately failed by osd.3 2021-03-24T21:58:30.466 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03-24T21:58:30.466 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: 2021-03-24T21:58:29.372304+0000 mon.a (mon.0) 1038 : cluster [DBG] 2021-03-24T21:58:30.466 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: osd.5 reported immediately failed by osd.0 2021-03-24T21:58:30.466 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:58:30.467 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: 58:29.372606+0000 mon.a (mon.0) 1039 : cluster [DBG] osd.5 reported immediately failed by osd.2 2021-03-24T21:58:30.467 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03-24T21:58:29.372774+0000 mon.a ( 2021-03-24T21:58:30.467 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: mon.0) 1040 : cluster [DBG] osd.5 reported immediately failed by osd.3 2021-03-24T21:58:30.467 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03-24T21:58:30.467 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: 2021-03-24T21:58:29.372943+0000 mon.a (mon.0) 1041 : cluster 2021-03-24T21:58:30.468 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: [DBG] osd.5 reported immediately failed by osd.7 2021-03-24T21:58:30.468 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:58:30.468 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: T21:58:29.373086+0000 mon.a (mon.0) 1042 : cluster [DBG] osd.5 reported immediately failed by osd.1 2021-03-24T21:58:30.468 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03-24T21:58:29.373223 2021-03-24T21:58:30.468 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: +0000 mon.a (mon.0) 1043 : cluster [DBG] osd.5 reported immediately failed by osd.7 2021-03-24T21:58:30.469 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:58:30.469 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: 03-24T21:58:29.373389+0000 mon.a (mon.0) 1044 2021-03-24T21:58:30.469 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: : cluster [DBG] osd.5 reported immediately failed by osd.1 2021-03-24T21:58:30.469 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:58:30.470 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: 24T21:58:29.373521+0000 mon.a (mon. 2021-03-24T21:58:30.470 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: 0) 1045 : cluster [DBG] osd.5 reported immediately failed by osd.4 2021-03-24T21:58:30.470 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:58:30.470 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: 58:29.373615+0000 mon.a (mon.0) 1046 : cluster [DBG] osd.5 reported immediately failed by osd.6 2021-03-24T21:58:30.470 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: 2021-03-24T21:58:30.471 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03-24T21:58 2021-03-24T21:58:30.471 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: :29.373787+0000 mon.a (mon.0) 1047 : cluster [DBG] osd.5 reported immediately failed by osd.4 2021-03-24T21:58:30.471 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03-24T21 2021-03-24T21:58:30.471 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: :58:29.373927+0000 mon.a (mon.0) 1048 : cluster [DBG] osd.5 reported immediately failed by osd.6 2021-03-24T21:58:30.471 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: 2021-03-24T21:58:30.472 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03-24T21:58:30.472 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: 2021-03-24T21:58:29.427801+0000 mon.a (mon.0 2021-03-24T21:58:30.472 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: ) 1049 : cluster [DBG] osd.5 reported immediately failed by osd.7 2021-03-24T21:58:30.472 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03-24T21:58:29.429956 2021-03-24T21:58:30.473 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: +0000 mon.a (mon.0) 1050 : cluster [DBG] osd.5 reported immediately failed by osd.4 2021-03-24T21:58:30.473 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:58:30.474 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: 21:58:29.430424+0000 mon.a (mon.0) 1051 : cluster 2021-03-24T21:58:30.474 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: [DBG] osd.5 reported immediately failed by osd.2 2021-03-24T21:58:30.474 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:58:30.474 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: T21:58:29.573436+0000 mon.a (mon.0) 1052 : cluster [DBG] 2021-03-24T21:58:30.475 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: osd.5 reported immediately failed by osd.2 2021-03-24T21:58:30.475 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03-24T21:58:30.475 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: 2021-03-24T21:58:29.573778+0000 mon.a (mon.0) 1053 : cluster [DBG] 2021-03-24T21:58:30.475 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: osd.5 reported immediately failed by osd.2 2021-03-24T21:58:30.475 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03-24T21:58:30.476 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: 2021-03-24T21:58:29.574056+0000 mon.a (mon 2021-03-24T21:58:30.476 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: .0) 1054 : cluster [DBG] osd.5 reported immediately failed by osd.3 2021-03-24T21:58:30.476 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03-24T21:58:29.574351+0000 mon.a (mon.0 2021-03-24T21:58:30.476 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: ) 1055 : cluster [DBG] osd.5 reported immediately failed by osd.4 2021-03-24T21:58:30.476 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:58:30.477 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: 58:29.574669+0000 mon.a (mon.0) 1056 : cluster [DBG] osd.5 reported immediately failed by osd.1 2021-03-24T21:58:30.477 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03-24T21:58:30.477 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: 2021-03-24T21:58:29.574961+0000 mon.a (mon.0) 1057 : cluster [DBG] osd.5 reported immediately failed by osd.0 2021-03-24T21:58:30.477 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03-24T21:58:29.575183+0000 2021-03-24T21:58:30.478 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: mon.a (mon.0) 1058 : cluster [DBG] osd.5 reported immediately failed by osd.3 2021-03-24T21:58:30.478 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03-24T21:58:30.478 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: 2021-03-24T21:58:29.575452+0000 mon.a (mon.0) 1059 : cluster 2021-03-24T21:58:30.478 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: [DBG] osd.5 reported immediately failed by osd.4 2021-03-24T21:58:30.478 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03-24T21:58:29 2021-03-24T21:58:30.479 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: .575593+0000 mon.a (mon.0) 1060 : cluster [DBG] osd.5 reported immediately failed by osd.1 2021-03-24T21:58:30.479 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03-24T21:58:29.575867 2021-03-24T21:58:30.479 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: +0000 mon.a (mon.0) 1061 : cluster [DBG] osd.5 reported immediately failed by osd.0 2021-03-24T21:58:30.479 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03-24T21:58:30.480 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: 2021-03-24T21:58:29.576065+0000 mon.a (mon.0) 1062 : cluster [DBG] osd.5 reported immediately failed by osd.6 2021-03-24T21:58:30.480 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03-24T21:58:29.576255+0000 2021-03-24T21:58:30.480 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: mon.a (mon.0) 1063 : cluster [DBG] osd.5 reported immediately failed by osd.7 2021-03-24T21:58:30.480 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:58:30.480 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: 03-24T21:58:29.576448+0000 mon.a (mon.0) 1064 : cluster [DBG] osd.5 reported immediately failed by osd.6 2021-03-24T21:58:30.481 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03-24T21:58:29.576702+0000 mon.a (mon. 2021-03-24T21:58:30.481 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: 0) 1065 : cluster [DBG] osd.5 reported immediately failed by osd.7 2021-03-24T21:58:30.481 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03-24T21:58:30.481 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: 2021-03-24T21:58:29.629340+0000 mon.a (mon.0) 1066 : cluster [DBG] osd.5 reported immediately failed by osd.7 2021-03-24T21:58:30.481 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03-24T21:58:29.633028+0000 mon.a (mon.0) 1067 : 2021-03-24T21:58:30.482 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster [DBG] osd.5 reported immediately failed by osd.2 2021-03-24T21:58:30.482 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03-24T21:58:29.633203+0000 2021-03-24T21:58:30.482 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: mon.a (mon.0) 1068 : cluster [DBG] osd.5 reported immediately failed by osd.4 2021-03-24T21:58:30.482 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:58:30.483 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: 24T21:58:29.716175+0000 osd.4 (osd.4) 87 : cluster [DBG] 3.14 scrub ok 2021-03-24T21:58:30.483 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03-24T21:58:30.483 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: 2021-03-24T21:58:29.787902+0000 mon.a (mon.0) 1069 : cluster 2021-03-24T21:58:30.483 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: [DBG] osd.5 reported immediately failed by osd.0 2021-03-24T21:58:30.483 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03-24T21:58:29 2021-03-24T21:58:30.484 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: .977547+0000 mon.a (mon.0) 1070 : cluster [DBG] osd.5 reported immediately failed by osd.2 2021-03-24T21:58:30.484 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03-24T21:58:29.978028+0000 mon.a (mon. 2021-03-24T21:58:30.484 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: 0) 1071 : cluster [DBG] osd.5 reported immediately failed by osd.0 2021-03-24T21:58:30.484 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:58:30.484 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: -03-24T21:58:29.978325+0000 mon.a (mon.0) 1072 : cluster 2021-03-24T21:58:30.485 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: [DBG] osd.5 reported immediately failed by osd.2 2021-03-24T21:58:30.485 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03-24T21:58:29. 2021-03-24T21:58:30.485 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: 978607+0000 mon.a (mon.0) 1073 : cluster [DBG] osd.5 reported immediately failed by osd.3 2021-03-24T21:58:30.485 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03-24T21:58:29.979007+0000 mon.a (mon.0) 1074 : 2021-03-24T21:58:30.486 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster [DBG] osd.5 reported immediately failed by osd.4 2021-03-24T21:58:30.486 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03-24T21:58:29. 2021-03-24T21:58:30.486 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: 979450+0000 mon.a (mon.0) 1075 : cluster [DBG] osd.5 reported immediately failed by osd.1 2021-03-24T21:58:30.486 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03-24T21:58:30.486 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: 2021-03-24T21:58:29.979805+0000 mon.a (mon.0) 1076 : cluster [DBG] osd.5 reported immediately failed by osd.0 2021-03-24T21:58:30.487 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:58:30.487 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: T21:58:29.980086+0000 mon.a (mon.0) 1077 : cluster [DBG] osd.5 reported immediately failed by osd.3 2021-03-24T21:58:30.487 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03-24T21:58:29. 2021-03-24T21:58:30.487 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: 980442+0000 mon.a (mon.0) 1078 : cluster [DBG] osd.5 reported immediately failed by osd.6 2021-03-24T21:58:30.487 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03-24T21:58:29.980775+0000 mon.a ( 2021-03-24T21:58:30.488 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: mon.0) 1079 : cluster [DBG] osd.5 reported immediately failed by osd.1 2021-03-24T21:58:30.488 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03-24T21:58:29.980954+0000 mon.a (mon.0) 2021-03-24T21:58:30.488 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: 1080 : cluster [DBG] osd.5 reported immediately failed by osd.7 2021-03-24T21:58:30.488 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:58:30.488 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: -03-24T21:58:29.981137+0000 mon.a (mon.0) 1081 : cluster 2021-03-24T21:58:30.489 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: [DBG] osd.5 reported immediately failed by osd.4 2021-03-24T21:58:30.489 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03-24T21:58:29 2021-03-24T21:58:30.489 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: .981342+0000 mon.a (mon.0) 1082 : cluster [DBG] osd.5 reported immediately failed by osd.6 2021-03-24T21:58:30.489 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03-24T21:58:29.981538+0000 mon.a (mon.0) 1083 2021-03-24T21:58:30.489 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: : cluster [DBG] osd.5 reported immediately failed by osd.7 2021-03-24T21:58:30.490 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03-24T21:58:29.989115+0000 mon.a (mon.0) 1084 : cluster 2021-03-24T21:58:30.490 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: [DBG] osd.5 reported immediately failed by osd.0 2021-03-24T21:58:30.490 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:58:30.490 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: 21:58:30.033445+0000 mon.a (mon.0) 1085 : cluster [DBG] 2021-03-24T21:58:30.491 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: osd.5 reported immediately failed by osd.7 2021-03-24T21:58:30.491 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03-24T21:58:30.037122+0000 2021-03-24T21:58:30.491 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: mon.a (mon.0) 1086 : cluster [DBG] osd.5 reported immediately failed by osd.2 2021-03-24T21:58:30.491 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03-24T21:58:30.037367+0000 mon.a (mon.0) 1087 : cluster [DBG] 2021-03-24T21:58:30.491 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: osd.5 reported immediately failed by osd.4 2021-03-24T21:58:30.492 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:58:30.492 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: 24T21:58:30.110116+0000 mon.a (mon.0) 1088 : cluster [DBG] osd.5 reported immediately failed by osd.3 2021-03-24T21:58:30.492 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021-03-24T21:58:30.176055+0000 mon.a 2021-03-24T21:58:30.492 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: (mon.0) 1089 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2021-03-24T21:58:30.492 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:58:30.493 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:30 smithi142 conmon[20125]: 03-24T21:58:30.192798+0000 mon.a (mon.0) 1090 : cluster [DBG] osdmap e340: 8 total, 7 up, 6 in 2021-03-24T21:58:30.510 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:30.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:30.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:30.549 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021-03-24T21:58: 2021-03-24T21:58:30.550 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: 28.797786+0000 mgr.y (mgr.14144) 438 : cluster [DBG] pgmap v670: 168 pgs: 168 active+clean; 169 MiB data, 557 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:58:30.550 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:58:30.550 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: -24T21:58:29.298220+0000 osd.3 (osd.3) 78 : cluster [DBG] 6.1b scrub ok 2021-03-24T21:58:30.550 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:58:30.551 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: 03-24T21:58:29.353932+0000 mon.c (mon 2021-03-24T21:58:30.551 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: .2) 55 : audit [DBG] from='client.? v1:172.21.15.125:0/3953067220' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:58:30.551 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:58:30.551 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: -24T21:58:29.371779+0000 mon.a ( 2021-03-24T21:58:30.551 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: mon.0) 1035 : cluster [DBG] osd.5 reported immediately failed by osd.0 2021-03-24T21:58:30.552 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:58:30.552 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: -03-24T21:58:29.371842+0000 mon.a 2021-03-24T21:58:30.552 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: (mon.0) 1036 : cluster [INF] osd.5 failed (root=default,host=smithi142) (connection refused reported by osd.0) 2021-03-24T21:58:30.552 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021-03-24T21:58:30.552 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: 2021-03 2021-03-24T21:58:30.553 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: -24T21:58:29.372072+0000 mon.a ( 2021-03-24T21:58:30.553 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: mon.0) 1037 : cluster [DBG] osd.5 reported immediately failed by osd.3 2021-03-24T21:58:30.553 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021-03-24T21:58:30.553 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: 2021-03-24T21:58:29.372304+0000 2021-03-24T21:58:30.553 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: mon.a (mon.0) 1038 : cluster [DBG] osd.5 reported immediately failed by osd.0 2021-03-24T21:58:30.554 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021-03-24T21:58:30.554 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: 2021-03-24T21:58:29.372606+0000 2021-03-24T21:58:30.554 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: mon.a (mon.0) 1039 : cluster [DBG] osd.5 reported immediately failed by osd.2 2021-03-24T21:58:30.554 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: 2021-03-24T21:58:30.555 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021-03-24T21:58:29. 2021-03-24T21:58:30.555 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: 372774+0000 mon.a (mon.0) 1040 : cluster [DBG] 2021-03-24T21:58:30.555 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: osd.5 reported immediately failed by osd.3 2021-03-24T21:58:30.555 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:58:30.555 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: 24T21:58:29.372943+0000 mon.a (mon 2021-03-24T21:58:30.555 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: .0) 1041 : cluster [DBG] osd.5 reported immediately failed by osd.7 2021-03-24T21:58:30.556 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021-03-24T21:58:30.556 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: 2021-03- 2021-03-24T21:58:30.556 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: 24T21:58:29.373086+0000 mon.a (mon. 2021-03-24T21:58:30.556 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: 0) 1042 : cluster [DBG] osd.5 reported immediately failed by osd.1 2021-03-24T21:58:30.556 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021-03-24T21:58:30.557 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: 2021-03-24T21:58:29.373223+0000 2021-03-24T21:58:30.557 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: mon.a (mon.0) 1043 : cluster [DBG] osd.5 reported immediately failed by osd.7 2021-03-24T21:58:30.557 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: 2021-03-24T21:58:30.557 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021-03-24T21:58:29. 2021-03-24T21:58:30.557 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: 373389+0000 mon.a (mon.0) 1044 : cluster [DBG] 2021-03-24T21:58:30.558 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: osd.5 reported immediately failed by osd.1 2021-03-24T21:58:30.558 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021-03-24T21:58 2021-03-24T21:58:30.558 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: :29.373521+0000 mon.a (mon.0) 1045 : 2021-03-24T21:58:30.558 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster [DBG] osd.5 reported immediately failed by osd.4 2021-03-24T21:58:30.559 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021-03-24T21:58:30.559 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: 2021-03-24T21:58:29. 2021-03-24T21:58:30.559 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: 373615+0000 mon.a (mon.0) 1046 : cluster 2021-03-24T21:58:30.559 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: [DBG] osd.5 reported immediately failed by osd.6 2021-03-24T21:58:30.560 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:58:30.560 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: :58:29.373787+0000 mon.a (mon.0) 2021-03-24T21:58:30.560 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: 1047 : cluster [DBG] osd.5 reported immediately failed by osd.4 2021-03-24T21:58:30.560 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:58:30.560 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: 24T21:58:29.373927+0000 mon.a (mon 2021-03-24T21:58:30.561 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: .0) 1048 : cluster [DBG] osd.5 reported immediately failed by osd.6 2021-03-24T21:58:30.561 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:58:30.561 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: -03-24T21:58:29.427801+0000 2021-03-24T21:58:30.561 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: mon.a (mon.0) 1049 : cluster [DBG] osd.5 reported immediately failed by osd.7 2021-03-24T21:58:30.561 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021-03-24T21:58:29.429956+0000 mon.a (mon.0) 1050 : cluster [DBG] osd.5 reported immediately failed by osd.4 2021-03-24T21:58:30.562 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021-03-24T21:58:29.430424+0000 mon.a (mon.0) 1051 : cluster [DBG] osd.5 reported immediately failed by osd.2 2021-03-24T21:58:30.562 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021-03-24T21:58:29.573436+0000 mon.a (mon.0) 1052 : cluster [DBG] osd.5 reported immediately failed by osd.2 2021-03-24T21:58:30.562 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021-03-24T21:58:29.573778+0000 mon.a (mon.0) 1053 : cluster [DBG] osd.5 reported immediately failed by osd.2 2021-03-24T21:58:30.562 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021-03-24T21:58:29.574056+0000 mon.a (mon.0) 1054 : cluster [DBG] osd.5 reported immediately failed by osd.3 2021-03-24T21:58:30.562 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021-03-24T21:58:29.574351+0000 mon.a (mon.0) 1055 : cluster [DBG] osd.5 reported immediately failed by osd.4 2021-03-24T21:58:30.563 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021-03-24T21:58:29.574669+0000 mon.a (mon.0) 1056 : cluster [DBG] osd.5 reported immediately failed by osd.1 2021-03-24T21:58:30.563 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021-03-24T21:58:30.563 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: 2021-03-24T21:58:29.574961+0000 mon.a (mon.0) 1057 : cluster [DBG] osd.5 reported immediately failed by osd.0 2021-03-24T21:58:30.563 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T21:58:30.563 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: T21:58:29.575183+0000 mon.a (mon.0) 1058 : cluster [DBG] 2021-03-24T21:58:30.564 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: osd.5 reported immediately failed by osd.3 2021-03-24T21:58:30.564 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021-03-24T21:58 2021-03-24T21:58:30.564 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: :29.575452+0000 mon.a (mon.0) 1059 : 2021-03-24T21:58:30.564 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster [DBG] osd.5 reported immediately failed by osd.4 2021-03-24T21:58:30.565 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T21:58:30.565 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: T21:58:29.575593+0000 mon.a (mon. 2021-03-24T21:58:30.565 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: 0) 1060 : cluster [DBG] osd.5 reported immediately failed by osd.1 2021-03-24T21:58:30.565 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:58:30.565 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: 03-24T21:58:29.575867+0000 mon.a 2021-03-24T21:58:30.566 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: (mon.0) 1061 : cluster [DBG] osd.5 reported immediately failed by osd.0 2021-03-24T21:58:30.566 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021-03-24T21:58:30.566 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: 2021-03-24T21:58:29.576065+0000 2021-03-24T21:58:30.566 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: mon.a (mon.0) 1062 : cluster [DBG] osd.5 reported immediately failed by osd.6 2021-03-24T21:58:30.566 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021-03-24T21:58:30.567 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: 2021-03-24T21:58:29.576255+0000 2021-03-24T21:58:30.567 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: mon.a (mon.0) 1063 : cluster [DBG] osd.5 reported immediately failed by osd.7 2021-03-24T21:58:30.567 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: 2021-03-24T21:58:30.567 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:58:30.567 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: -03-24T21:58:29.576448+0000 mon.a 2021-03-24T21:58:30.568 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: (mon.0) 1064 : cluster [DBG] osd.5 reported immediately failed by osd.6 2021-03-24T21:58:30.568 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021-03-24T21:58:30.568 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: 2021-03-24T21:58:29.576702+0000 2021-03-24T21:58:30.568 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: mon.a (mon.0) 1065 : cluster [DBG] osd.5 reported immediately failed by osd.7 2021-03-24T21:58:30.568 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:58:30.569 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: -03-24T21:58:29.629340+0000 mon.a 2021-03-24T21:58:30.569 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: (mon.0) 1066 : cluster [DBG] osd.5 reported immediately failed by osd.7 2021-03-24T21:58:30.569 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021-03-24T21:58: 2021-03-24T21:58:30.570 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: 28.797786+0000 mgr.y (mgr.14144) 438 : cluster [DBG] pgmap v670: 168 pgs: 168 active+clean; 169 MiB data, 557 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:58:30.570 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:58:30.570 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: 03-24T21:58:29.298220 2021-03-24T21:58:30.570 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: +0000 osd.3 (osd.3) 78 : cluster [DBG] 6.1b scrub ok 2021-03-24T21:58:30.571 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: audit 2021-03-24T21:58:30.571 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: 2021-03-24T21:58: 2021-03-24T21:58:30.571 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: 29.353932+0000 mon.c (mon.2) 55 2021-03-24T21:58:30.571 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: : audit [DBG] from='client.? v1:172.21.15.125:0/3953067220' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:58:30.571 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:58:30.572 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: -03-24T21:58:29. 2021-03-24T21:58:30.572 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: 371779+0000 mon.a (mon.0) 1035 : cluster [DBG] osd.5 reported immediately failed by osd.0 2021-03-24T21:58:30.572 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:58:30.572 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: 24T21:58:29.371842+0000 2021-03-24T21:58:30.572 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: mon.a (mon.0) 1036 : cluster [INF] osd.5 failed (root=default,host=smithi142) (connection refused reported by osd.0) 2021-03-24T21:58:30.573 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: 2021-03-24T21:58:30.573 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021-03-24T21:58:30.573 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: 2021-03-24T21:58:29.372072+0000 2021-03-24T21:58:30.573 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: mon.a (mon.0) 1037 : cluster [DBG] osd.5 reported immediately failed by osd.3 2021-03-24T21:58:30.573 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:58:30.574 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: -03-24T21:58:29.372304+0000 mon.a 2021-03-24T21:58:30.574 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: (mon.0) 1038 : cluster [DBG] osd.5 reported immediately failed by osd.0 2021-03-24T21:58:30.574 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:58:30.574 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: 21:58:29.372606+0000 mon.a (mon 2021-03-24T21:58:30.575 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: .0) 1039 : cluster [DBG] 2021-03-24T21:58:30.575 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: osd.5 reported immediately failed by osd.2 2021-03-24T21:58:30.575 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:58:30.575 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: 21:58:29.372774+0000 2021-03-24T21:58:30.575 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: mon.a (mon.0) 1040 : cluster [DBG] 2021-03-24T21:58:30.576 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: osd.5 reported immediately failed by osd.3 2021-03-24T21:58:30.576 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:58:30.576 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: 24T21:58:29.372943 2021-03-24T21:58:30.576 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: +0000 mon.a (mon.0) 1041 : cluster [DBG] osd.5 reported immediately failed by osd.7 2021-03-24T21:58:30.576 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:58:30.577 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: T21:58:29.373086+0000 mon.a (mon 2021-03-24T21:58:30.577 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: .0) 1042 : cluster [DBG] osd.5 reported immediately failed by osd.1 2021-03-24T21:58:30.577 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:58:30.577 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: -24T21:58:29.373223+0000 mon.a (mon.0 2021-03-24T21:58:30.578 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: ) 1043 : cluster [DBG] osd.5 reported immediately failed by osd.7 2021-03-24T21:58:30.578 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:58:30.578 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: -03-24T21:58:29.373389+0000 mon.a 2021-03-24T21:58:30.578 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: (mon.0) 1044 : cluster [DBG] osd.5 reported immediately failed by osd.1 2021-03-24T21:58:30.578 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:58:30.579 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: T21:58:29.373521+0000 mon.a (mon.0) 1045 : 2021-03-24T21:58:30.579 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster [DBG] osd.5 reported immediately failed by osd.4 2021-03-24T21:58:30.579 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021-03-24T21:58:30.579 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: 2021-03-24T21:58:29.373615+0000 mon.a (mon.0 2021-03-24T21:58:30.579 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: ) 1046 : cluster [DBG] osd.5 reported immediately failed by osd.6 2021-03-24T21:58:30.580 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:58:30.580 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: :58:29.373787+0000 2021-03-24T21:58:30.581 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021-03-24T21:58:30.581 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: 2021-03-24T21:58:29.633028+0000 2021-03-24T21:58:30.581 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: mon.a (mon.0) 1067 : cluster [DBG] osd.5 reported immediately failed by osd.2 2021-03-24T21:58:30.581 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: 2021-03-24T21:58:30.582 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021-03-24T21:58: 2021-03-24T21:58:30.582 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: 29.633203+0000 mon.a (mon.0) 1068 : 2021-03-24T21:58:30.582 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster [DBG] 2021-03-24T21:58:30.582 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: osd.5 reported immediately failed by osd.4 2021-03-24T21:58:30.582 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021-03-24T21:58:29 2021-03-24T21:58:30.583 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: .716175+0000 osd.4 (osd.4) 87 : cluster 2021-03-24T21:58:30.583 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: [DBG] 3.14 scrub ok 2021-03-24T21:58:30.583 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021-03-24T21:58 2021-03-24T21:58:30.583 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: :29.787902+0000 mon.a (mon.0) 1069 : 2021-03-24T21:58:30.583 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster [DBG] osd.5 reported immediately failed by osd.0 2021-03-24T21:58:30.584 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:58:30.584 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: -24T21:58:29.977547+0000 mon.a ( 2021-03-24T21:58:30.584 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: mon.0) 1070 : cluster [DBG] osd.5 reported immediately failed by osd.2 2021-03-24T21:58:30.584 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:58:30.585 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: -03-24T21:58:29.978028+0000 2021-03-24T21:58:30.585 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: mon.a (mon.0) 1071 : cluster [DBG] osd.5 reported immediately failed by osd.0 2021-03-24T21:58:30.585 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021-03-24T21:58:30.585 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: 2021-03-24T21 2021-03-24T21:58:30.585 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: :58:29.978325+0000 mon.a (mon.0) 1072 : cluster [DBG] 2021-03-24T21:58:30.586 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: osd.5 reported immediately failed by osd.2 2021-03-24T21:58:30.586 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021-03-24T21:58:30.586 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: 2021-03-24T21:58:29.978607 2021-03-24T21:58:30.586 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: +0000 mon.a (mon.0) 1073 : cluster [DBG] 2021-03-24T21:58:30.587 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: osd.5 reported immediately failed by osd.3 2021-03-24T21:58:30.587 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021-03-24T21:58 2021-03-24T21:58:30.587 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: :29.979007+0000 mon.a (mon.0) 1074 : 2021-03-24T21:58:30.587 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster [DBG] osd.5 reported immediately failed by osd.4 2021-03-24T21:58:30.587 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T21:58:30.588 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: T21:58:29.979450+0000 mon.a (mon 2021-03-24T21:58:30.588 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: .0) 1075 : cluster [DBG] osd.5 reported immediately failed by osd.1 2021-03-24T21:58:30.588 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021-03-24T21:58:30.588 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: 2021-03-24T21:58:29.979805+0000 2021-03-24T21:58:30.588 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: mon.a (mon.0) 1076 : cluster [DBG] osd.5 reported immediately failed by osd.0 2021-03-24T21:58:30.589 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021-03-24T21:58:30.589 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: 2021-03-24T21:58:29.980086 2021-03-24T21:58:30.589 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: +0000 mon.a (mon.0) 1077 : cluster [DBG] 2021-03-24T21:58:30.589 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: osd.5 reported immediately failed by osd.3 2021-03-24T21:58:30.589 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021-03-24T21:58: 2021-03-24T21:58:30.590 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: 29.980442+0000 mon.a (mon.0) 1078 : 2021-03-24T21:58:30.590 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster [DBG] osd.5 reported immediately failed by osd.6 2021-03-24T21:58:30.590 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:58:30.590 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: 21:58:29.980775+0000 mon.a (mon.0 2021-03-24T21:58:30.590 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: ) 1079 : cluster [DBG] osd.5 reported immediately failed by osd.1 2021-03-24T21:58:30.591 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:58:30.591 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: 03-24T21:58:29.980954+0000 mon.a 2021-03-24T21:58:30.591 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: (mon.0) 1080 : cluster [DBG] osd.5 reported immediately failed by osd.7 2021-03-24T21:58:30.591 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021-03-24T21:58:30.592 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: 2021-03-24T21:58:29.981137+0000 2021-03-24T21:58:30.592 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: mon.a (mon.0) 1081 : cluster [DBG] 2021-03-24T21:58:30.592 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: osd.5 reported immediately failed by osd.4 2021-03-24T21:58:30.592 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021-03-24T21:58:29 2021-03-24T21:58:30.592 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: .981342+0000 mon.a (mon.0) 1082 : cluster 2021-03-24T21:58:30.593 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: [DBG] osd.5 reported immediately failed by osd.6 2021-03-24T21:58:30.593 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:58:30.593 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: :58:29.981538+0000 mon.a (mon.0 2021-03-24T21:58:30.593 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: ) 1083 : cluster [DBG] osd.5 reported immediately failed by osd.7 2021-03-24T21:58:30.594 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:58:30.594 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: -24T21:58:29.989115+0000 mon.a ( 2021-03-24T21:58:30.594 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: mon.0) 1084 : cluster [DBG] osd.5 reported immediately failed by osd.0 2021-03-24T21:58:30.594 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:58:30.594 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: -03-24T21:58:30.033445+0000 2021-03-24T21:58:30.595 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: mon.a (mon.0) 1085 : cluster [DBG] osd.5 reported immediately failed by osd.7 2021-03-24T21:58:30.595 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021-03-24T21:58:30.595 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: 2021-03-24T21:58:30.037122+0000 2021-03-24T21:58:30.595 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: mon.a (mon.0) 1086 : cluster [DBG] 2021-03-24T21:58:30.595 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: osd.5 reported immediately failed by osd.2 2021-03-24T21:58:30.596 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021-03-24T21:58:30.596 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: 2021-03-24T21:58:30.037367+0000 mon.a (mon 2021-03-24T21:58:30.596 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: .0) 1087 : cluster [DBG] osd.5 reported immediately failed by osd.4 2021-03-24T21:58:30.596 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:58:30.597 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: -24T21:58:30.110116+0000 mon.a ( 2021-03-24T21:58:30.597 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: mon.0) 1088 : cluster [DBG] osd.5 reported immediately failed by osd.3 2021-03-24T21:58:30.597 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:58:30.597 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: 03-24T21:58:30.176055+0000 2021-03-24T21:58:30.597 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: mon.a (mon.0) 1089 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2021-03-24T21:58:30.598 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:58:30.598 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: -03-24T21:58:30.192798+0000 mon.a 2021-03-24T21:58:30.598 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[19936]: (mon.0) 1090 : cluster [DBG] osdmap e340: 8 total, 7 up, 6 in 2021-03-24T21:58:30.599 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: mon.a (mon.0) 1047 : cluster [DBG] osd.5 reported immediately failed by osd.4 2021-03-24T21:58:30.599 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: 2021-03-24T21:58:30.599 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021-03-24T21:58:30.600 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: 2021-03-24T21:58:29.373927+0000 2021-03-24T21:58:30.600 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: mon.a (mon.0) 1048 : cluster [DBG] osd.5 reported immediately failed by osd.6 2021-03-24T21:58:30.600 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:58:30.600 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: 03-24T21:58:29.427801+0000 mon.a (mon.0 2021-03-24T21:58:30.600 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: ) 1049 : cluster [DBG] osd.5 reported immediately failed by osd.7 2021-03-24T21:58:30.601 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021-03-24T21:58:30.601 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: 2021-03-24T21:58:29.429956 2021-03-24T21:58:30.601 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: +0000 mon.a (mon.0) 1050 : cluster [DBG] osd.5 reported immediately failed by osd.4 2021-03-24T21:58:30.601 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021-03-24T21:58:30.601 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: 2021-03-24T21:58: 2021-03-24T21:58:30.602 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: 29.430424+0000 mon.a (mon.0) 1051 2021-03-24T21:58:30.602 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: : cluster [DBG] osd.5 reported immediately failed by osd.2 2021-03-24T21:58:30.602 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021-03-24T21:58:29. 2021-03-24T21:58:30.602 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: 573436+0000 mon.a (mon.0) 1052 : cluster [DBG] 2021-03-24T21:58:30.603 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: osd.5 reported immediately failed by osd.2 2021-03-24T21:58:30.603 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021-03-24T21:58:30.603 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: 2021-03-24T21:58:29. 2021-03-24T21:58:30.603 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: 573778+0000 mon.a (mon.0) 1053 : cluster [DBG] 2021-03-24T21:58:30.603 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: osd.5 reported immediately failed by osd.2 2021-03-24T21:58:30.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:58:30.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: -03-24T21:58:29.574056+0000 2021-03-24T21:58:30.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: mon.a (mon.0) 1054 : cluster 2021-03-24T21:58:30.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: [DBG] osd.5 reported immediately failed by osd.3 2021-03-24T21:58:30.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:58:30.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: 21:58:29.574351+0000 mon.a (mon.0) 1055 : 2021-03-24T21:58:30.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster [DBG] osd.5 reported immediately failed by osd.4 2021-03-24T21:58:30.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:58:30.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: 21:58:29.574669+0000 mon.a ( 2021-03-24T21:58:30.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: mon.0) 1056 : cluster [DBG] osd.5 reported immediately failed by osd.1 2021-03-24T21:58:30.606 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:58:30.606 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: 03-24T21:58:29.574961+0000 mon.a 2021-03-24T21:58:30.606 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: (mon.0) 1057 : cluster [DBG] osd.5 reported immediately failed by osd.0 2021-03-24T21:58:30.606 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:58:30.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: -03-24T21:58: 2021-03-24T21:58:30.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: 29.575183+0000 mon.a (mon.0) 1058 : cluster 2021-03-24T21:58:30.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: [DBG] osd.5 reported immediately failed by osd.3 2021-03-24T21:58:30.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021-03-24T21:58:30.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: 2021-03-24T21:58:29. 2021-03-24T21:58:30.608 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: 575452+0000 mon.a (mon.0) 1059 : cluster [DBG] osd.5 reported immediately failed by osd.4 2021-03-24T21:58:30.608 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: 2021-03-24T21:58:30.608 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:58:30.608 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: 03-24T21:58:29.575593+0000 2021-03-24T21:58:30.609 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: mon.a (mon.0) 1060 : cluster [DBG] osd.5 reported immediately failed by osd.1 2021-03-24T21:58:30.609 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:58:30.609 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: 24T21:58:29.575867+0000 mon.a 2021-03-24T21:58:30.609 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: (mon.0) 1061 : cluster [DBG] 2021-03-24T21:58:30.609 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: osd.5 reported immediately failed by osd.0 2021-03-24T21:58:30.610 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021-03-24T21:58:29 2021-03-24T21:58:30.610 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: .576065+0000 mon.a (mon.0) 1062 : cluster [DBG] osd.5 reported immediately failed by osd.6 2021-03-24T21:58:30.610 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:58:30.610 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: -24T21:58:29.576255 2021-03-24T21:58:30.611 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: +0000 mon.a (mon.0) 1063 : cluster 2021-03-24T21:58:30.611 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: [DBG] osd.5 reported immediately failed by osd.7 2021-03-24T21:58:30.611 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:58:30.611 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: -03-24T21:58: 2021-03-24T21:58:30.611 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: 29.576448+0000 mon.a (mon 2021-03-24T21:58:30.612 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: .0) 1064 : cluster [DBG] osd.5 reported immediately failed by osd.6 2021-03-24T21:58:30.612 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021-03-24T21:58:30.612 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: 2021-03-24T21:58: 2021-03-24T21:58:30.612 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: 29.576702+0000 mon.a (mon.0 2021-03-24T21:58:30.612 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: ) 1065 : cluster [DBG] osd.5 reported immediately failed by osd.7 2021-03-24T21:58:30.613 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:58:30.613 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: 21:58:29.629340+0000 mon.a (mon.0 2021-03-24T21:58:30.613 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: ) 1066 : cluster [DBG] osd.5 reported immediately failed by osd.7 2021-03-24T21:58:30.613 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:58:30.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: :58:29.633028+0000 mon.a (mon. 2021-03-24T21:58:30.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: 0) 1067 : cluster [DBG] osd.5 reported immediately failed by osd.2 2021-03-24T21:58:30.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:58:30.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: :58:29.633203+0000 mon.a (mon.0) 1068 2021-03-24T21:58:30.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: : cluster [DBG] osd.5 reported immediately failed by osd.4 2021-03-24T21:58:30.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021-03-24T21:58:30.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: 2021-03-24T21 2021-03-24T21:58:30.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: :58:29.716175+0000 osd.4 (osd.4) 2021-03-24T21:58:30.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: 87 : cluster [DBG] 3.14 scrub ok 2021-03-24T21:58:30.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021-03-24T21:58: 2021-03-24T21:58:30.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: 29.787902+0000 mon.a (mon. 2021-03-24T21:58:30.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: 0) 1069 : cluster [DBG] osd.5 reported immediately failed by osd.0 2021-03-24T21:58:30.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:58:30.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: T21:58:29.977547+0000 mon.a (mon.0) 1070 : 2021-03-24T21:58:30.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster [DBG] osd.5 reported immediately failed by osd.2 2021-03-24T21:58:30.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:58:30.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: :58:29.978028+0000 2021-03-24T21:58:30.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: mon.a (mon.0) 1071 : cluster [DBG] osd.5 reported immediately failed by osd.0 2021-03-24T21:58:30.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:58:30.618 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: -24T21:58:29.978325+0000 mon.a (mon. 2021-03-24T21:58:30.618 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: 0) 1072 : cluster [DBG] osd.5 reported immediately failed by osd.2 2021-03-24T21:58:30.618 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:58:30.618 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: 03-24T21:58:29.978607+0000 mon.a (mon. 2021-03-24T21:58:30.619 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: 0) 1073 : cluster [DBG] osd.5 reported immediately failed by osd.3 2021-03-24T21:58:30.619 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:58:30.619 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: :58:29.979007+0000 mon.a (mon.0) 1074 : cluster 2021-03-24T21:58:30.619 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: [DBG] osd.5 reported immediately failed by osd.4 2021-03-24T21:58:30.619 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:58:30.620 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: -24T21:58:29.979450+0000 mon.a 2021-03-24T21:58:30.620 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: (mon.0) 1075 : cluster [DBG] osd.5 reported immediately failed by osd.1 2021-03-24T21:58:30.620 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021-03-24T21:58:29.979805+0000 mon.a (mon 2021-03-24T21:58:30.620 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: .0) 1076 : cluster [DBG] osd.5 reported immediately failed by osd.0 2021-03-24T21:58:30.620 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:58:30.621 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: 58:29.980086+0000 mon.a (mon 2021-03-24T21:58:30.621 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: .0) 1077 : cluster [DBG] osd.5 reported immediately failed by osd.3 2021-03-24T21:58:30.621 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021-03-24T21:58: 2021-03-24T21:58:30.621 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: 29.980442+0000 mon.a (mon.0) 1078 : cluster [DBG] 2021-03-24T21:58:30.622 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: osd.5 reported immediately failed by osd.6 2021-03-24T21:58:30.622 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:58:30.622 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: :58:29.980775+0000 mon.a (mon 2021-03-24T21:58:30.622 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: .0) 1079 : cluster [DBG] osd.5 reported immediately failed by osd.1 2021-03-24T21:58:30.622 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:58:30.623 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: -03-24T21:58:29.980954+0000 mon.a 2021-03-24T21:58:30.623 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: (mon.0) 1080 : cluster [DBG] osd.5 reported immediately failed by osd.7 2021-03-24T21:58:30.623 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:58:30.623 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: :58:29.981137+0000 mon.a (mon.0) 1081 : cluster 2021-03-24T21:58:30.624 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: [DBG] osd.5 reported immediately failed by osd.4 2021-03-24T21:58:30.624 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:58:30.624 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: 21:58:29.981342+0000 mon.a (mon.0) 1082 : cluster 2021-03-24T21:58:30.624 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: [DBG] osd.5 reported immediately failed by osd.6 2021-03-24T21:58:30.624 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:58:30.625 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: -24T21:58:29.981538+0000 mon.a (mon.0) 1083 2021-03-24T21:58:30.625 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: : cluster [DBG] osd.5 reported immediately failed by osd.7 2021-03-24T21:58:30.625 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:58:30.625 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: 21:58:29.989115+0000 mon.a (mon 2021-03-24T21:58:30.625 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: .0) 1084 : cluster [DBG] osd.5 reported immediately failed by osd.0 2021-03-24T21:58:30.626 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021-03-24T21:58 2021-03-24T21:58:30.626 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: :30.033445+0000 mon.a (mon 2021-03-24T21:58:30.626 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: .0) 1085 : cluster [DBG] 2021-03-24T21:58:30.626 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: osd.5 reported immediately failed by osd.7 2021-03-24T21:58:30.626 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:58:30.627 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: -03-24T21:58:30.037122+0000 mon.a (mon.0) 1086 : cluster [DBG] 2021-03-24T21:58:30.627 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: osd.5 reported immediately failed by osd.2 2021-03-24T21:58:30.627 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:58:30.627 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: 03-24T21:58:30.037367 2021-03-24T21:58:30.628 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: +0000 mon.a (mon.0) 1087 : cluster [DBG] osd.5 reported immediately failed by osd.4 2021-03-24T21:58:30.628 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:58:30.628 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: 03-24T21:58:30.110116+0000 mon.a (mon.0) 2021-03-24T21:58:30.628 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: 1088 : cluster [DBG] osd.5 reported immediately failed by osd.3 2021-03-24T21:58:30.628 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:58:30.629 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: -03-24T21:58:30.176055+0000 mon.a (mon.0) 1089 : cluster [WRN] 2021-03-24T21:58:30.629 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: Health check failed: 1 osds down (OSD_DOWN) 2021-03-24T21:58:30.629 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: cluster 2021-03-24T21:58:30.629 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: 2021-03-24T21:58:30.192798+0000 2021-03-24T21:58:30.629 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: mon.a (mon.0) 1090 : cluster [DBG] osdmap e340: 8 total, 7 up, 6 in 2021-03-24T21:58:30.630 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:30 smithi125 conmon[24927]: 2021-03-24T21:58:30.630 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:30.691 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:30.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:30.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:58:30.796 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:30.802 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-24T21:58:30.802 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T21:58:30.806 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 21:58:30 smithi142 podman[60548]: 2021-03-24 21:58:30.47463052 +0000 UTC m=+1.083178455 container remove 97264781327ab14946f21d9751e0a39ea24c9db473f401fe3d0a3ff2dc5c9e9c (image=quay.ceph.io/ceph-ci/ceph@sha256:e9db06906b8d0a24f3ef6835ac5aa4b8eea2950a2560bad78f83d4f101eca0db, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5, GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06, RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.18.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20201204, org.label-schema.name=CentOS Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Dimitri Savineau , org.label-schema.vendor=CentOS) 2021-03-24T21:58:30.847 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:30.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:30.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:30.953 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:31.007 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd set noscrub 2021-03-24T21:58:31.013 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:31.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:31.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:31.117 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 21:58:30 smithi142 podman[60712]: 2021-03-24 21:58:30.80641402 +0000 UTC m=+0.243987853 container create 725a07322432ef883c82419badc05794916268d85226edf8a062b5e03fb5bf72 (image=quay.ceph.io/ceph-ci/ceph@sha256:e9db06906b8d0a24f3ef6835ac5aa4b8eea2950a2560bad78f83d4f101eca0db, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06, RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d, maintainer=Dimitri Savineau , org.label-schema.build-date=20201204, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.18.0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2021-03-24T21:58:31.117 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 21:58:30 smithi142 podman[60712]: 2021-03-24 21:58:30.972981039 +0000 UTC m=+0.410554816 container init 725a07322432ef883c82419badc05794916268d85226edf8a062b5e03fb5bf72 (image=quay.ceph.io/ceph-ci/ceph@sha256:e9db06906b8d0a24f3ef6835ac5aa4b8eea2950a2560bad78f83d4f101eca0db, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d, org.label-schema.vendor=CentOS, io.buildah.version=1.18.0, GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06, maintainer=Dimitri Savineau , org.label-schema.name=CentOS Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20201204) 2021-03-24T21:58:31.117 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 21:58:31 smithi142 podman[60712]: 2021-03-24 21:58:31.014629209 +0000 UTC m=+0.452203033 container start 725a07322432ef883c82419badc05794916268d85226edf8a062b5e03fb5bf72 (image=quay.ceph.io/ceph-ci/ceph@sha256:e9db06906b8d0a24f3ef6835ac5aa4b8eea2950a2560bad78f83d4f101eca0db, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Dimitri Savineau , io.buildah.version=1.18.0, GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06, org.label-schema.schema-version=1.0, org.label-schema.build-date=20201204, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Base Image) 2021-03-24T21:58:31.117 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 21:58:31 smithi142 podman[60712]: 2021-03-24 21:58:31.014850146 +0000 UTC m=+0.452424014 container attach 725a07322432ef883c82419badc05794916268d85226edf8a062b5e03fb5bf72 (image=quay.ceph.io/ceph-ci/ceph@sha256:e9db06906b8d0a24f3ef6835ac5aa4b8eea2950a2560bad78f83d4f101eca0db, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Base Image, org.label-schema.build-date=20201204, GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Dimitri Savineau , GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.18.0, RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d) 2021-03-24T21:58:31.119 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:31.171 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:31.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:31.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:58:31.188 INFO:tasks.rados.rados.0.smithi125.stdout:1193: done (5 left) 2021-03-24T21:58:31.188 INFO:tasks.rados.rados.0.smithi125.stdout:1194: read oid 16 snap -1 2021-03-24T21:58:31.189 INFO:tasks.rados.rados.0.smithi125.stdout:1194: expect (ObjNum 345 snap 88 seq_num 345) 2021-03-24T21:58:31.189 INFO:tasks.rados.rados.0.smithi125.stdout:1195: read oid 10 snap 116 2021-03-24T21:58:31.189 INFO:tasks.rados.rados.0.smithi125.stdout:1195: expect (ObjNum 422 snap 111 seq_num 422) 2021-03-24T21:58:31.189 INFO:tasks.rados.rados.0.smithi125.stdout:1196: delete oid 45 current snap is 119 2021-03-24T21:58:31.209 INFO:tasks.rados.rados.0.smithi125.stdout:1192: finishing write tid 1 to smithi12556470-25 2021-03-24T21:58:31.210 INFO:tasks.rados.rados.0.smithi125.stdout:1194: expect (ObjNum 345 snap 88 seq_num 345) 2021-03-24T21:58:31.279 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:31.344 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:31.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:31.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:31.444 INFO:tasks.rados.rados.0.smithi125.stdout:1192: finishing write tid 2 to smithi12556470-25 2021-03-24T21:58:31.446 INFO:tasks.rados.rados.0.smithi125.stdout:1192: finishing write tid 3 to smithi12556470-25 2021-03-24T21:58:31.447 INFO:tasks.rados.rados.0.smithi125.stdout:1192: finishing write tid 4 to smithi12556470-25 2021-03-24T21:58:31.447 INFO:tasks.rados.rados.0.smithi125.stdout:1192: finishing write tid 5 to smithi12556470-25 2021-03-24T21:58:31.447 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 25 v 266 (ObjNum 440 snap 118 seq_num 440) dirty exists 2021-03-24T21:58:31.448 INFO:tasks.rados.rados.0.smithi125.stdout:1192: left oid 25 (ObjNum 440 snap 118 seq_num 440) 2021-03-24T21:58:31.448 INFO:tasks.rados.rados.0.smithi125.stdout:1188: expect (ObjNum 298 snap 74 seq_num 298) 2021-03-24T21:58:31.451 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:31.502 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:31.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:31.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:58:31.608 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:31.616 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 21:58:31 smithi142 podman[60712]: 2021-03-24 21:58:31.414057243 +0000 UTC m=+0.851631077 container died 725a07322432ef883c82419badc05794916268d85226edf8a062b5e03fb5bf72 (image=quay.ceph.io/ceph-ci/ceph@sha256:e9db06906b8d0a24f3ef6835ac5aa4b8eea2950a2560bad78f83d4f101eca0db, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5-deactivate) 2021-03-24T21:58:31.630 INFO:tasks.rados.rados.0.smithi125.stdout:1190: finishing copy_from to smithi12556470-30 2021-03-24T21:58:31.630 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 30 v 279 (ObjNum 427 snap 113 seq_num 427) dirty exists 2021-03-24T21:58:31.630 INFO:tasks.rados.rados.0.smithi125.stdout:1190: finishing copy_from racing read to smithi12556470-30 2021-03-24T21:58:31.631 INFO:tasks.rados.rados.0.smithi125.stdout:1195: expect (ObjNum 422 snap 111 seq_num 422) 2021-03-24T21:58:31.660 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:31.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:31.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:31.766 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:31.791 INFO:tasks.rados.rados.0.smithi125.stdout:1187: finishing write tid 1 to smithi12556470-12 2021-03-24T21:58:31.791 INFO:tasks.rados.rados.0.smithi125.stdout:1187: finishing write tid 2 to smithi12556470-12 2021-03-24T21:58:31.791 INFO:tasks.rados.rados.0.smithi125.stdout:1187: finishing write tid 3 to smithi12556470-12 2021-03-24T21:58:31.791 INFO:tasks.rados.rados.0.smithi125.stdout:1187: finishing write tid 4 to smithi12556470-12 2021-03-24T21:58:31.792 INFO:tasks.rados.rados.0.smithi125.stdout:1187: finishing write tid 5 to smithi12556470-12 2021-03-24T21:58:31.792 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 12 v 305 (ObjNum 437 snap 118 seq_num 437) dirty exists 2021-03-24T21:58:31.792 INFO:tasks.rados.rados.0.smithi125.stdout:1187: left oid 12 (ObjNum 437 snap 118 seq_num 437) 2021-03-24T21:58:31.792 INFO:tasks.rados.rados.0.smithi125.stdout:1189: finishing write tid 1 to smithi12556470-26 2021-03-24T21:58:31.793 INFO:tasks.rados.rados.0.smithi125.stdout:1189: finishing write tid 2 to smithi12556470-26 2021-03-24T21:58:31.793 INFO:tasks.rados.rados.0.smithi125.stdout:1189: finishing write tid 3 to smithi12556470-26 2021-03-24T21:58:31.793 INFO:tasks.rados.rados.0.smithi125.stdout:1189: finishing write tid 4 to smithi12556470-26 2021-03-24T21:58:31.793 INFO:tasks.rados.rados.0.smithi125.stdout:1189: finishing write tid 5 to smithi12556470-26 2021-03-24T21:58:31.793 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 26 v 313 (ObjNum 438 snap 118 seq_num 438) dirty exists 2021-03-24T21:58:31.794 INFO:tasks.rados.rados.0.smithi125.stdout:1189: left oid 26 (ObjNum 438 snap 118 seq_num 438) 2021-03-24T21:58:31.794 INFO:tasks.rados.rados.0.smithi125.stdout:1187: done (7 left) 2021-03-24T21:58:31.794 INFO:tasks.rados.rados.0.smithi125.stdout:1188: done (6 left) 2021-03-24T21:58:31.794 INFO:tasks.rados.rados.0.smithi125.stdout:1189: done (5 left) 2021-03-24T21:58:31.795 INFO:tasks.rados.rados.0.smithi125.stdout:1190: done (4 left) 2021-03-24T21:58:31.795 INFO:tasks.rados.rados.0.smithi125.stdout:1192: done (3 left) 2021-03-24T21:58:31.795 INFO:tasks.rados.rados.0.smithi125.stdout:1194: done (2 left) 2021-03-24T21:58:31.795 INFO:tasks.rados.rados.0.smithi125.stdout:1195: done (1 left) 2021-03-24T21:58:31.796 INFO:tasks.rados.rados.0.smithi125.stdout:1196: done (0 left) 2021-03-24T21:58:31.796 INFO:tasks.rados.rados.0.smithi125.stdout:1197: read oid 49 snap -1 2021-03-24T21:58:31.796 INFO:tasks.rados.rados.0.smithi125.stdout:1197: expect deleted 2021-03-24T21:58:31.796 INFO:tasks.rados.rados.0.smithi125.stdout:1198: write oid 3 current snap is 119 2021-03-24T21:58:31.796 INFO:tasks.rados.rados.0.smithi125.stdout:1198: seq_num 442 ranges {628694=408532,1515041=600142,2805198=552112} 2021-03-24T21:58:31.803 INFO:tasks.rados.rados.0.smithi125.stdout:1198: writing smithi12556470-3 from 628694 to 1037226 tid 1 2021-03-24T21:58:31.809 INFO:tasks.rados.rados.0.smithi125.stdout:1198: writing smithi12556470-3 from 1515041 to 2115183 tid 2 2021-03-24T21:58:31.817 INFO:tasks.rados.rados.0.smithi125.stdout:1198: writing smithi12556470-3 from 2805198 to 3357310 tid 3 2021-03-24T21:58:31.817 INFO:tasks.rados.rados.0.smithi125.stdout:1199: copy_from oid 48 from oid 42 current snap is 119 2021-03-24T21:58:31.817 INFO:tasks.rados.rados.0.smithi125.stdout:1198: finishing write tid 1 to smithi12556470-3 2021-03-24T21:58:31.817 INFO:tasks.rados.rados.0.smithi125.stdout:1198: finishing write tid 2 to smithi12556470-3 2021-03-24T21:58:31.818 INFO:tasks.rados.rados.0.smithi125.stdout:1197: done (2 left) 2021-03-24T21:58:31.818 INFO:tasks.rados.rados.0.smithi125.stdout:1200: copy_from oid 32 from oid 35 current snap is 119 2021-03-24T21:58:31.818 INFO:tasks.rados.rados.0.smithi125.stdout:1201: snap_create 2021-03-24T21:58:31.819 INFO:tasks.rados.rados.0.smithi125.stdout:1200: finishing copy_from to smithi12556470-32 2021-03-24T21:58:31.819 INFO:tasks.rados.rados.0.smithi125.stdout:1200: got expected ENOENT (src dne) 2021-03-24T21:58:31.820 INFO:tasks.rados.rados.0.smithi125.stdout:1200: finishing copy_from racing read to smithi12556470-32 2021-03-24T21:58:31.820 INFO:tasks.rados.rados.0.smithi125.stdout:1199: finishing copy_from to smithi12556470-48 2021-03-24T21:58:31.820 INFO:tasks.rados.rados.0.smithi125.stdout:1199: got expected ENOENT (src dne) 2021-03-24T21:58:31.820 INFO:tasks.rados.rados.0.smithi125.stdout:1199: finishing copy_from racing read to smithi12556470-48 2021-03-24T21:58:31.821 INFO:tasks.rados.rados.0.smithi125.stdout:1198: finishing write tid 3 to smithi12556470-3 2021-03-24T21:58:31.821 INFO:tasks.rados.rados.0.smithi125.stdout:1198: finishing write tid 4 to smithi12556470-3 2021-03-24T21:58:31.822 INFO:tasks.rados.rados.0.smithi125.stdout:1198: finishing write tid 5 to smithi12556470-3 2021-03-24T21:58:31.822 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 3 v 140 (ObjNum 442 snap 119 seq_num 442) dirty exists 2021-03-24T21:58:31.822 INFO:tasks.rados.rados.0.smithi125.stdout:1198: left oid 3 (ObjNum 442 snap 119 seq_num 442) 2021-03-24T21:58:31.823 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:31.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:31.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:58:31.927 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:31.979 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:31.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:31.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:58:32.006 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:58:32.026 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.0 dump_blocked_ops 2021-03-24T21:58:32.084 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:32.146 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:32.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:32.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:32.252 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:32.302 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:32.305 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:32 smithi125 conmon[24927]: cluster 2021-03-24T21:58: 2021-03-24T21:58:32.306 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:32 smithi125 conmon[24927]: 30.678170+0000 osd.4 (osd.4) 88 : cluster [DBG] 4.9 scrub ok 2021-03-24T21:58:32.306 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:32 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:58:32.306 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:32 smithi125 conmon[24927]: 24T21:58:30.798463+0000 mgr.y (mgr. 2021-03-24T21:58:32.306 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:32 smithi125 conmon[24927]: 14144) 439 : cluster [DBG] pgmap v673: 168 pgs: 22 stale+active+clean, 146 active+clean; 145 MiB data, 518 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:58:32.307 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:32 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:58:32.307 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:32 smithi125 conmon[24927]: -03-24T21:58:31.184957+0000 mon.a (mon.0) 1091 : 2021-03-24T21:58:32.307 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:32 smithi125 conmon[24927]: cluster [DBG] osdmap e341: 8 total, 7 up, 6 in 2021-03-24T21:58:32.307 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:32 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:58:32.307 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:32 smithi125 conmon[24927]: T21:58:31.625376+0000 osd.1 (osd. 2021-03-24T21:58:32.308 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:32 smithi125 conmon[24927]: 1) 39 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:58:32.308 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:32 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:58:32.308 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:32 smithi125 conmon[24927]: 03-24T21:58:31.625481+0000 osd.1 ( 2021-03-24T21:58:32.308 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:32 smithi125 conmon[24927]: osd.1) 40 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:58:32.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:32.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:58:32.310 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:32 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:58:32.310 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:32 smithi125 conmon[19936]: 24T21:58:30.678170+0000 osd.4 (osd.4) 88 : cluster [DBG] 4.9 scrub ok 2021-03-24T21:58:32.311 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:32 smithi125 conmon[19936]: cluster 2021-03-24T21:58:32.311 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:32 smithi125 conmon[19936]: 2021-03-24T 2021-03-24T21:58:32.311 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:32 smithi125 conmon[19936]: 21:58:30. 2021-03-24T21:58:32.311 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:32 smithi125 conmon[19936]: 798463+0000 mgr.y ( 2021-03-24T21:58:32.312 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:32 smithi125 conmon[19936]: mgr.14144) 439 2021-03-24T21:58:32.312 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:32 smithi125 conmon[19936]: : cluster [DBG] pgmap v673: 168 pgs: 22 stale+active+clean, 146 active+clean; 145 MiB data, 518 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:58:32.312 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:32 smithi125 conmon[19936]: 2021-03-24T21:58:32.312 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:32 smithi125 conmon[19936]: cluster 2021-03-24T21:58:32.312 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:32 smithi125 conmon[19936]: 2021-03-24T21 2021-03-24T21:58:32.313 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:32 smithi125 conmon[19936]: :58:31.184957+0000 2021-03-24T21:58:32.313 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:32 smithi125 conmon[19936]: mon.a (mon.0) 1091 2021-03-24T21:58:32.313 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:32 smithi125 conmon[19936]: : cluster [DBG] osdmap e341: 8 total, 7 up, 6 in 2021-03-24T21:58:32.313 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:32 smithi125 conmon[19936]: 2021-03-24T21:58:32.313 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:32 smithi125 conmon[19936]: cluster 2021-03-24T21:58:32.314 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:32 smithi125 conmon[19936]: 2021-03-24T 2021-03-24T21:58:32.314 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:32 smithi125 conmon[19936]: 21:58:31.625376 2021-03-24T21:58:32.314 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:32 smithi125 conmon[19936]: +0000 osd.1 (osd. 2021-03-24T21:58:32.314 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:32 smithi125 conmon[19936]: 1) 39 : cluster [DBG] 2021-03-24T21:58:32.315 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:32 smithi125 conmon[19936]: purged_snaps scrub starts 2021-03-24T21:58:32.315 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:32 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:58:32.315 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:32 smithi125 conmon[19936]: 21:58:31.625481+0000 osd.1 (osd.1) 40 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:58:32.318 DEBUG:teuthology.orchestra.run.smithi142:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c@osd.5.service' 2021-03-24T21:58:32.367 INFO:tasks.rados.rados.0.smithi125.stdout:1198: done (3 left) 2021-03-24T21:58:32.368 INFO:tasks.rados.rados.0.smithi125.stdout:1199: done (2 left) 2021-03-24T21:58:32.368 INFO:tasks.rados.rados.0.smithi125.stdout:1200: done (1 left) 2021-03-24T21:58:32.368 INFO:tasks.rados.rados.0.smithi125.stdout:1201: done (0 left) 2021-03-24T21:58:32.368 INFO:tasks.rados.rados.0.smithi125.stdout:1202: write (excl) oid 21 current snap is 120 2021-03-24T21:58:32.369 INFO:tasks.rados.rados.0.smithi125.stdout:1202: seq_num 443 ranges {645501=483781,1573523=460153,2548974=763384} 2021-03-24T21:58:32.374 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:32 smithi142 conmon[20125]: cluster 2021-03-24T21:58:30.678170 2021-03-24T21:58:32.374 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:32 smithi142 conmon[20125]: +0000 osd.4 (osd.4) 88 : cluster [DBG] 4.9 scrub ok 2021-03-24T21:58:32.375 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:32 smithi142 conmon[20125]: cluster 2021-03-24T21:58:30.798463+0000 2021-03-24T21:58:32.375 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:32 smithi142 conmon[20125]: mgr.y (mgr.14144) 439 : cluster [DBG] pgmap v673: 168 pgs: 22 stale+active+clean, 146 active+clean; 145 MiB data, 518 MiB used, 536 GiB / 536 GiB avail 2021-03-24T21:58:32.375 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:32 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:58:32.375 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:32 smithi142 conmon[20125]: -24T21:58:31.184957+0000 mon.a (mon 2021-03-24T21:58:32.376 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:32 smithi142 conmon[20125]: .0) 1091 : cluster [DBG] osdmap e341: 8 total, 7 up, 6 in 2021-03-24T21:58:32.376 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:32 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:58:32.376 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:32 smithi142 conmon[20125]: -24T21:58:31.625376+0000 osd.1 (osd.1) 2021-03-24T21:58:32.376 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:32 smithi142 conmon[20125]: 39 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:58:32.376 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:32 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:58:32.377 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:32 smithi142 conmon[20125]: 03-24T21:58:31.625481+0000 osd.1 (osd 2021-03-24T21:58:32.377 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:32 smithi142 conmon[20125]: .1) 40 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:58:32.377 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 21:58:32 smithi142 podman[60712]: 2021-03-24 21:58:32.30607491 +0000 UTC m=+1.743648736 container remove 725a07322432ef883c82419badc05794916268d85226edf8a062b5e03fb5bf72 (image=quay.ceph.io/ceph-ci/ceph@sha256:e9db06906b8d0a24f3ef6835ac5aa4b8eea2950a2560bad78f83d4f101eca0db, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d, ceph=True, io.buildah.version=1.18.0, org.label-schema.build-date=20201204, org.label-schema.name=CentOS Base Image, maintainer=Dimitri Savineau , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2021-03-24T21:58:32.378 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 21:58:32 smithi142 systemd[1]: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c@osd.5.service: Succeeded. 2021-03-24T21:58:32.378 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 21:58:32 smithi142 systemd[1]: Stopped Ceph osd.5 for 3cfa339a-8cea-11eb-90b2-001a4aab830c. 2021-03-24T21:58:32.378 INFO:tasks.rados.rados.0.smithi125.stdout:1202: writing smithi12556470-21 from 645501 to 1129282 tid 1 2021-03-24T21:58:32.381 INFO:tasks.rados.rados.0.smithi125.stdout:1202: writing smithi12556470-21 from 1573523 to 2033676 tid 2 2021-03-24T21:58:32.391 INFO:tasks.rados.rados.0.smithi125.stdout:1202: writing smithi12556470-21 from 2548974 to 3312358 tid 3 2021-03-24T21:58:32.392 INFO:tasks.rados.rados.0.smithi125.stdout:1203: write (excl) oid 42 current snap is 120 2021-03-24T21:58:32.392 INFO:tasks.rados.rados.0.smithi125.stdout:1203: seq_num 444 ranges {662308=559030,1632005=603811,2860044=602264} 2021-03-24T21:58:32.399 INFO:tasks.rados.rados.0.smithi125.stdout:1203: writing smithi12556470-42 from 662308 to 1221338 tid 1 2021-03-24T21:58:32.406 INFO:tasks.rados.rados.0.smithi125.stdout:1203: writing smithi12556470-42 from 1632005 to 2235816 tid 2 2021-03-24T21:58:32.411 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:32.418 INFO:tasks.rados.rados.0.smithi125.stdout:1203: writing smithi12556470-42 from 2860044 to 3462308 tid 3 2021-03-24T21:58:32.418 INFO:tasks.rados.rados.0.smithi125.stdout:1204: write oid 18 current snap is 120 2021-03-24T21:58:32.418 INFO:tasks.rados.rados.0.smithi125.stdout:1204: seq_num 445 ranges {679115=634279,2090487=463822,3003820=475295} 2021-03-24T21:58:32.422 INFO:tasks.rados.rados.0.smithi125.stdout:1204: writing smithi12556470-18 from 679115 to 1313394 tid 1 2021-03-24T21:58:32.431 INFO:tasks.rados.rados.0.smithi125.stdout:1204: writing smithi12556470-18 from 2090487 to 2554309 tid 2 2021-03-24T21:58:32.437 INFO:tasks.rados.rados.0.smithi125.stdout:1204: writing smithi12556470-18 from 3003820 to 3479115 tid 3 2021-03-24T21:58:32.438 INFO:tasks.rados.rados.0.smithi125.stdout:1205: read oid 27 snap -1 2021-03-24T21:58:32.438 INFO:tasks.rados.rados.0.smithi125.stdout:1205: expect (ObjNum 400 snap 102 seq_num 400) 2021-03-24T21:58:32.438 INFO:tasks.rados.rados.0.smithi125.stdout:1206: snap_create 2021-03-24T21:58:32.440 INFO:tasks.rados.rados.0.smithi125.stdout:1202: finishing write tid 1 to smithi12556470-21 2021-03-24T21:58:32.440 INFO:tasks.rados.rados.0.smithi125.stdout:1204: finishing write tid 1 to smithi12556470-18 2021-03-24T21:58:32.440 INFO:tasks.rados.rados.0.smithi125.stdout:1202: finishing write tid 2 to smithi12556470-21 2021-03-24T21:58:32.441 INFO:tasks.rados.rados.0.smithi125.stdout:1202: finishing write tid 3 to smithi12556470-21 2021-03-24T21:58:32.441 INFO:tasks.rados.rados.0.smithi125.stdout:1202: finishing write tid 4 to smithi12556470-21 2021-03-24T21:58:32.441 INFO:tasks.rados.rados.0.smithi125.stdout:1202: finishing write tid 5 to smithi12556470-21 2021-03-24T21:58:32.441 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 21 v 331 (ObjNum 443 snap 120 seq_num 443) dirty exists 2021-03-24T21:58:32.441 INFO:tasks.rados.rados.0.smithi125.stdout:1202: left oid 21 (ObjNum 443 snap 120 seq_num 443) 2021-03-24T21:58:32.442 INFO:tasks.rados.rados.0.smithi125.stdout:1203: finishing write tid 1 to smithi12556470-42 2021-03-24T21:58:32.444 INFO:tasks.rados.rados.0.smithi125.stdout:1203: finishing write tid 2 to smithi12556470-42 2021-03-24T21:58:32.444 INFO:tasks.rados.rados.0.smithi125.stdout:1203: finishing write tid 3 to smithi12556470-42 2021-03-24T21:58:32.444 INFO:tasks.rados.rados.0.smithi125.stdout:1203: finishing write tid 4 to smithi12556470-42 2021-03-24T21:58:32.445 INFO:tasks.rados.rados.0.smithi125.stdout:1203: finishing write tid 5 to smithi12556470-42 2021-03-24T21:58:32.445 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 42 v 94 (ObjNum 444 snap 120 seq_num 444) dirty exists 2021-03-24T21:58:32.445 INFO:tasks.rados.rados.0.smithi125.stdout:1203: left oid 42 (ObjNum 444 snap 120 seq_num 444) 2021-03-24T21:58:32.447 INFO:tasks.rados.rados.0.smithi125.stdout:1204: finishing write tid 2 to smithi12556470-18 2021-03-24T21:58:32.448 INFO:tasks.rados.rados.0.smithi125.stdout:1204: finishing write tid 3 to smithi12556470-18 2021-03-24T21:58:32.448 INFO:tasks.rados.rados.0.smithi125.stdout:1204: finishing write tid 4 to smithi12556470-18 2021-03-24T21:58:32.449 INFO:tasks.rados.rados.0.smithi125.stdout:1204: finishing write tid 5 to smithi12556470-18 2021-03-24T21:58:32.449 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 18 v 182 (ObjNum 445 snap 120 seq_num 445) dirty exists 2021-03-24T21:58:32.449 INFO:tasks.rados.rados.0.smithi125.stdout:1204: left oid 18 (ObjNum 445 snap 120 seq_num 445) 2021-03-24T21:58:32.453 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-03-24T21:58:32.454 INFO:tasks.cephadm.osd.5:Stopped osd.5 2021-03-24T21:58:32.454 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.5 2021-03-24T21:58:32.455 INFO:tasks.thrashosds.thrasher:remote for osd 5 is ubuntu@smithi142.front.sepia.ceph.com 2021-03-24T21:58:32.455 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell -n osd.5 --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph-objectstore-tool --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op list-pgs 2021-03-24T21:58:32.457 INFO:tasks.rados.rados.0.smithi125.stdout:1205: expect (ObjNum 400 snap 102 seq_num 400) 2021-03-24T21:58:32.509 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:32.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:32.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:58:32.568 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:58:32.620 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:32.675 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/osd.5/config 2021-03-24T21:58:32.685 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:32.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:32.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:58:32.789 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:32.848 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:32.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:32.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:58:32.952 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:33.011 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:33.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:33.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:33.117 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:33.178 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:33.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:33.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:33.284 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:33.340 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:33.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:33.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:58:33.371 INFO:tasks.rados.rados.0.smithi125.stdout:1202: done (4 left) 2021-03-24T21:58:33.372 INFO:tasks.rados.rados.0.smithi125.stdout:1203: done (3 left) 2021-03-24T21:58:33.372 INFO:tasks.rados.rados.0.smithi125.stdout:1204: done (2 left) 2021-03-24T21:58:33.372 INFO:tasks.rados.rados.0.smithi125.stdout:1205: done (1 left) 2021-03-24T21:58:33.373 INFO:tasks.rados.rados.0.smithi125.stdout:1206: done (0 left) 2021-03-24T21:58:33.373 INFO:tasks.rados.rados.0.smithi125.stdout:1207: write oid 10 current snap is 121 2021-03-24T21:58:33.373 INFO:tasks.rados.rados.0.smithi125.stdout:1207: seq_num 446 ranges {695922=709528,2032616=607480,3198537=297385} 2021-03-24T21:58:33.381 INFO:tasks.rados.rados.0.smithi125.stdout:1207: writing smithi12556470-10 from 695922 to 1405450 tid 1 2021-03-24T21:58:33.389 INFO:tasks.rados.rados.0.smithi125.stdout:1207: writing smithi12556470-10 from 2032616 to 2640096 tid 2 2021-03-24T21:58:33.394 INFO:tasks.rados.rados.0.smithi125.stdout:1207: writing smithi12556470-10 from 3198537 to 3495922 tid 3 2021-03-24T21:58:33.395 INFO:tasks.rados.rados.0.smithi125.stdout:1208: read oid 34 snap -1 2021-03-24T21:58:33.396 INFO:tasks.rados.rados.0.smithi125.stdout:1208: expect (ObjNum 411 snap 108 seq_num 411) 2021-03-24T21:58:33.396 INFO:tasks.rados.rados.0.smithi125.stdout:1209: copy_from oid 30 from oid 18 current snap is 121 2021-03-24T21:58:33.396 INFO:tasks.rados.rados.0.smithi125.stdout:1210: copy_from oid 2 from oid 17 current snap is 121 2021-03-24T21:58:33.396 INFO:tasks.rados.rados.0.smithi125.stdout:1211: copy_from oid 33 from oid 38 current snap is 121 2021-03-24T21:58:33.397 INFO:tasks.rados.rados.0.smithi125.stdout:1212: snap_create 2021-03-24T21:58:33.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:33 smithi125 conmon[19936]: cluster 2021-03-24T21:58:31.309073 2021-03-24T21:58:33.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:33 smithi125 conmon[19936]: +0000 osd.3 (osd.3) 79 : cluster [DBG] 4.19 scrub ok 2021-03-24T21:58:33.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:33 smithi125 conmon[19936]: cluster 2021-03-24T21:58:32.362556+0000 mon.a (mon.0 2021-03-24T21:58:33.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:33 smithi125 conmon[19936]: ) 1092 : cluster [DBG] osdmap e342: 8 total, 7 up, 6 in 2021-03-24T21:58:33.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:33 smithi125 conmon[24927]: cluster 2021-03-24T21:58:31.309073+0000 osd.3 (osd.3) 79 : cluster [DBG] 4.19 scrub ok 2021-03-24T21:58:33.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:33 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:58:33.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:33 smithi125 conmon[24927]: 58:32.362556+0000 mon.a (mon 2021-03-24T21:58:33.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:33 smithi125 conmon[24927]: .0) 1092 : cluster [DBG] 2021-03-24T21:58:33.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:33 smithi125 conmon[24927]: osdmap e342: 8 total, 7 up, 6 in 2021-03-24T21:58:33.408 INFO:tasks.rados.rados.0.smithi125.stdout:1207: finishing write tid 1 to smithi12556470-10 2021-03-24T21:58:33.408 INFO:tasks.rados.rados.0.smithi125.stdout:1207: finishing write tid 2 to smithi12556470-10 2021-03-24T21:58:33.408 INFO:tasks.rados.rados.0.smithi125.stdout:1207: finishing write tid 3 to smithi12556470-10 2021-03-24T21:58:33.408 INFO:tasks.rados.rados.0.smithi125.stdout:1207: finishing write tid 4 to smithi12556470-10 2021-03-24T21:58:33.409 INFO:tasks.rados.rados.0.smithi125.stdout:1207: finishing write tid 5 to smithi12556470-10 2021-03-24T21:58:33.409 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 10 v 364 (ObjNum 446 snap 121 seq_num 446) dirty exists 2021-03-24T21:58:33.409 INFO:tasks.rados.rados.0.smithi125.stdout:1207: left oid 10 (ObjNum 446 snap 121 seq_num 446) 2021-03-24T21:58:33.417 INFO:tasks.rados.rados.0.smithi125.stdout:1208: expect (ObjNum 411 snap 108 seq_num 411) 2021-03-24T21:58:33.446 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:33.589 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:33.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:33.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:58:33.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:33 smithi142 conmon[20125]: cluster 2021-03-24T21 2021-03-24T21:58:33.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:33 smithi142 conmon[20125]: :58:31.309073+0000 osd.3 (osd.3) 79 : cluster 2021-03-24T21:58:33.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:33 smithi142 conmon[20125]: [DBG] 4.19 scrub ok 2021-03-24T21:58:33.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:33 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:58:33.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:33 smithi142 conmon[20125]: -03-24T21:58 2021-03-24T21:58:33.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:33 smithi142 conmon[20125]: :32.362556+0000 2021-03-24T21:58:33.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:33 smithi142 conmon[20125]: mon.a (mon.0 2021-03-24T21:58:33.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:33 smithi142 conmon[20125]: ) 1092 : cluster [DBG] 2021-03-24T21:58:33.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:33 smithi142 conmon[20125]: osdmap e342: 8 total, 7 up, 6 in 2021-03-24T21:58:33.654 INFO:tasks.rados.rados.0.smithi125.stdout:1210: finishing copy_from to smithi12556470-2 2021-03-24T21:58:33.654 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 2 v 327 (ObjNum 425 snap 112 seq_num 425) dirty exists 2021-03-24T21:58:33.655 INFO:tasks.rados.rados.0.smithi125.stdout:1210: finishing copy_from racing read to smithi12556470-2 2021-03-24T21:58:33.655 INFO:tasks.rados.rados.0.smithi125.stdout:1211: finishing copy_from to smithi12556470-33 2021-03-24T21:58:33.655 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 33 v 49 (ObjNum 408 snap 106 seq_num 408) dirty exists 2021-03-24T21:58:33.655 INFO:tasks.rados.rados.0.smithi125.stdout:1211: finishing copy_from racing read to smithi12556470-33 2021-03-24T21:58:33.656 INFO:tasks.rados.rados.0.smithi125.stdout:1209: finishing copy_from to smithi12556470-30 2021-03-24T21:58:33.656 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 30 v 182 (ObjNum 445 snap 120 seq_num 445) dirty exists 2021-03-24T21:58:33.656 INFO:tasks.rados.rados.0.smithi125.stdout:1209: finishing copy_from racing read to smithi12556470-30 2021-03-24T21:58:33.702 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:33.752 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:33.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:33.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:33.858 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:33.918 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:33.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:33.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:34.024 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:34.078 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:34.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:34.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:58:34.183 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:34.190 INFO:journalctl@ceph.rgw.foo.a.smithi125.stdout:Mar 24 21:58:34 smithi125 conmon[39595]: debug 2021-03-24T21:58:34.030+0000 7fc212026700 -1 RGWWatcher::handle_error cookie 94195313961216 err (110) Connection timed out 2021-03-24T21:58:34.234 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:34.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:34.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:58:34.317 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:34 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:58:34.318 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:34 smithi125 conmon[19936]: :58:32.799026+0000 mgr.y (mgr.14144) 440 : cluster [DBG] pgmap v676: 168 pgs: 3 active+undersized, 4 active+undersized+degraded, 22 stale+active+clean, 139 active+clean; 145 MiB data, 514 MiB used, 536 GiB / 536 GiB avail; 6.0 MiB/s rd, 1.6 MiB/s wr, 7 op/s; 13/1137 objects degraded (1.143%) 2021-03-24T21:58:34.318 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:34 smithi125 conmon[19936]: cluster 2021-03-24T21:58:33.356211 2021-03-24T21:58:34.318 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:34 smithi125 conmon[19936]: +0000 mon.a (mon.0) 1093 : cluster [WRN] Health check failed: Degraded data redundancy: 13/1137 objects degraded (1.143%), 4 pgs degraded (PG_DEGRADED) 2021-03-24T21:58:34.319 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:34 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:58:34.319 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:34 smithi125 conmon[19936]: 58:33.369395+0000 mon.a (mon.0) 1094 : cluster 2021-03-24T21:58:34.319 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:34 smithi125 conmon[19936]: [DBG] osdmap e343: 8 total, 7 up, 6 in 2021-03-24T21:58:34.319 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:34 smithi125 conmon[24927]: cluster 2021-03-24T21:58:34.320 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:34 smithi125 conmon[24927]: 2021-03-24T21:58:34.320 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:34 smithi125 conmon[24927]: 2021-03-24T21:58:32. 2021-03-24T21:58:34.320 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:34 smithi125 conmon[24927]: 799026+0000 mgr.y (mgr.14144) 440 : cluster [DBG] pgmap v676: 168 pgs: 3 active+undersized, 4 active+undersized+degraded, 22 stale+active+clean, 139 active+clean; 145 MiB data, 514 MiB used, 536 GiB / 536 GiB avail; 6.0 MiB/s rd, 1.6 MiB/s wr, 7 op/s; 13/1137 objects degraded (1.143%) 2021-03-24T21:58:34.320 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:34 smithi125 conmon[24927]: cluster 2021-03-24T21:58:33.356211+0000 mon.a (mon 2021-03-24T21:58:34.320 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:34 smithi125 conmon[24927]: .0) 1093 : cluster [WRN] Health check failed: Degraded data redundancy: 13/1137 objects degraded (1.143%), 4 pgs degraded (PG_DEGRADED) 2021-03-24T21:58:34.321 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:34 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:58:34.321 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:34 smithi125 conmon[24927]: 58:33.369395+0000 mon.a (mon.0) 1094 : cluster 2021-03-24T21:58:34.321 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:34 smithi125 conmon[24927]: [DBG] osdmap e343: 8 total, 7 up, 6 in 2021-03-24T21:58:34.339 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:34.376 INFO:tasks.rados.rados.0.smithi125.stdout:1207: done (5 left) 2021-03-24T21:58:34.377 INFO:tasks.rados.rados.0.smithi125.stdout:1208: done (4 left) 2021-03-24T21:58:34.377 INFO:tasks.rados.rados.0.smithi125.stdout:1209: done (3 left) 2021-03-24T21:58:34.377 INFO:tasks.rados.rados.0.smithi125.stdout:1210: done (2 left) 2021-03-24T21:58:34.377 INFO:tasks.rados.rados.0.smithi125.stdout:1211: done (1 left) 2021-03-24T21:58:34.378 INFO:tasks.rados.rados.0.smithi125.stdout:1212: done (0 left) 2021-03-24T21:58:34.378 INFO:tasks.rados.rados.0.smithi125.stdout:1213: snap_remove snap 120 2021-03-24T21:58:34.405 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T21:58:34.449 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:34.451 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:34 smithi142 conmon[20125]: cluster 2021-03-24T21:58:32.799026+0000 mgr.y (mgr.14144) 440 : cluster [DBG] pgmap v676: 168 pgs: 3 active+undersized, 4 active+undersized+degraded, 22 stale+active+clean, 139 active+clean; 145 MiB data, 514 MiB used, 536 GiB / 536 GiB avail; 6.0 MiB/s rd, 1.6 MiB/s wr, 7 op/s; 13/1137 objects degraded (1.143%) 2021-03-24T21:58:34.452 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:34 smithi142 conmon[20125]: 2021-03-24T21:58:34.452 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:34 smithi142 conmon[20125]: cluster 2021-03-24T21:58:33.356211+0000 mon.a (mon.0) 1093 : cluster 2021-03-24T21:58:34.452 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:34 smithi142 conmon[20125]: [WRN] Health check failed: Degraded data redundancy: 13/1137 objects degraded (1.143%), 4 pgs degraded (PG_DEGRADED) 2021-03-24T21:58:34.452 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:34 smithi142 conmon[20125]: cluster 2021-03-24T21:58:34.453 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:34 smithi142 conmon[20125]: 2021-03-24T21:58:33.369395+0000 mon.a (mon.0) 1094 2021-03-24T21:58:34.453 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:34 smithi142 conmon[20125]: : cluster [DBG] osdmap e343: 8 total, 7 up, 6 in 2021-03-24T21:58:34.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:34.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:58:34.473 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T21:58:34.474 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-24T21:58:34.474 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-24T21:58:34.474 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-24T21:58:34.474 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-24T21:58:34.475 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-24T21:58:34.475 INFO:teuthology.orchestra.run.smithi125.stderr:Error ENXIO: problem getting command descriptions from osd.5 2021-03-24T21:58:34.475 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-24T21:58:34.475 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-24T21:58:34.563 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:34.616 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:34.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:34.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:58:34.724 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:34.764 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:34.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:34.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:58:34.869 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:34.927 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:34.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:34.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:58:35.015 INFO:teuthology.orchestra.run.smithi142.stdout:7.b 2021-03-24T21:58:35.016 INFO:teuthology.orchestra.run.smithi142.stdout:7.6 2021-03-24T21:58:35.016 INFO:teuthology.orchestra.run.smithi142.stdout:7.23 2021-03-24T21:58:35.016 INFO:teuthology.orchestra.run.smithi142.stdout:7.1d 2021-03-24T21:58:35.017 INFO:teuthology.orchestra.run.smithi142.stdout:7.1c 2021-03-24T21:58:35.017 INFO:teuthology.orchestra.run.smithi142.stdout:7.1e 2021-03-24T21:58:35.018 INFO:teuthology.orchestra.run.smithi142.stdout:4.a 2021-03-24T21:58:35.018 INFO:teuthology.orchestra.run.smithi142.stdout:3.17 2021-03-24T21:58:35.019 INFO:teuthology.orchestra.run.smithi142.stdout:6.15 2021-03-24T21:58:35.019 INFO:teuthology.orchestra.run.smithi142.stdout:4.7 2021-03-24T21:58:35.019 INFO:teuthology.orchestra.run.smithi142.stdout:7.0 2021-03-24T21:58:35.020 INFO:teuthology.orchestra.run.smithi142.stdout:2.2 2021-03-24T21:58:35.020 INFO:teuthology.orchestra.run.smithi142.stdout:7.14 2021-03-24T21:58:35.021 INFO:teuthology.orchestra.run.smithi142.stdout:4.3 2021-03-24T21:58:35.021 INFO:teuthology.orchestra.run.smithi142.stdout:4.1e 2021-03-24T21:58:35.021 INFO:teuthology.orchestra.run.smithi142.stdout:4.1 2021-03-24T21:58:35.022 INFO:teuthology.orchestra.run.smithi142.stdout:4.15 2021-03-24T21:58:35.022 INFO:teuthology.orchestra.run.smithi142.stdout:4.2 2021-03-24T21:58:35.023 INFO:teuthology.orchestra.run.smithi142.stdout:4.6 2021-03-24T21:58:35.023 INFO:teuthology.orchestra.run.smithi142.stdout:4.12 2021-03-24T21:58:35.023 INFO:teuthology.orchestra.run.smithi142.stdout:3.6 2021-03-24T21:58:35.024 INFO:teuthology.orchestra.run.smithi142.stdout:7.22 2021-03-24T21:58:35.024 INFO:teuthology.orchestra.run.smithi142.stdout:7.17 2021-03-24T21:58:35.025 INFO:teuthology.orchestra.run.smithi142.stdout:5.13 2021-03-24T21:58:35.025 INFO:teuthology.orchestra.run.smithi142.stdout:6.1a 2021-03-24T21:58:35.025 INFO:teuthology.orchestra.run.smithi142.stdout:5.16 2021-03-24T21:58:35.026 INFO:teuthology.orchestra.run.smithi142.stdout:4.11 2021-03-24T21:58:35.026 INFO:teuthology.orchestra.run.smithi142.stdout:4.1f 2021-03-24T21:58:35.027 INFO:teuthology.orchestra.run.smithi142.stdout:7.18 2021-03-24T21:58:35.027 INFO:teuthology.orchestra.run.smithi142.stdout:4.10 2021-03-24T21:58:35.027 INFO:teuthology.orchestra.run.smithi142.stdout:3.1d 2021-03-24T21:58:35.028 INFO:teuthology.orchestra.run.smithi142.stdout:3.d 2021-03-24T21:58:35.028 INFO:teuthology.orchestra.run.smithi142.stdout:7.4 2021-03-24T21:58:35.028 INFO:teuthology.orchestra.run.smithi142.stdout:3.19 2021-03-24T21:58:35.028 INFO:teuthology.orchestra.run.smithi142.stdout:3.c 2021-03-24T21:58:35.029 INFO:teuthology.orchestra.run.smithi142.stdout:3.1c 2021-03-24T21:58:35.029 INFO:teuthology.orchestra.run.smithi142.stdout:7.e 2021-03-24T21:58:35.029 INFO:teuthology.orchestra.run.smithi142.stdout:7.a 2021-03-24T21:58:35.030 INFO:teuthology.orchestra.run.smithi142.stdout:4.1d 2021-03-24T21:58:35.030 INFO:teuthology.orchestra.run.smithi142.stdout:3.10 2021-03-24T21:58:35.030 INFO:teuthology.orchestra.run.smithi142.stdout:3.5 2021-03-24T21:58:35.031 INFO:teuthology.orchestra.run.smithi142.stdout:4.e 2021-03-24T21:58:35.031 INFO:teuthology.orchestra.run.smithi142.stdout:5.8 2021-03-24T21:58:35.031 INFO:teuthology.orchestra.run.smithi142.stdout:4.13 2021-03-24T21:58:35.031 INFO:teuthology.orchestra.run.smithi142.stdout:3.1f 2021-03-24T21:58:35.032 INFO:teuthology.orchestra.run.smithi142.stdout:3.2 2021-03-24T21:58:35.032 INFO:teuthology.orchestra.run.smithi142.stdout:7.1f 2021-03-24T21:58:35.032 INFO:teuthology.orchestra.run.smithi142.stdout:3.16 2021-03-24T21:58:35.033 INFO:teuthology.orchestra.run.smithi142.stdout:6.b 2021-03-24T21:58:35.033 INFO:teuthology.orchestra.run.smithi142.stdout:4.8 2021-03-24T21:58:35.033 INFO:teuthology.orchestra.run.smithi142.stdout:5.c 2021-03-24T21:58:35.034 INFO:teuthology.orchestra.run.smithi142.stdout:3.4 2021-03-24T21:58:35.034 INFO:teuthology.orchestra.run.smithi142.stdout:6.1f 2021-03-24T21:58:35.034 INFO:teuthology.orchestra.run.smithi142.stdout:3.a 2021-03-24T21:58:35.035 INFO:teuthology.orchestra.run.smithi142.stdout:5.19 2021-03-24T21:58:35.035 INFO:teuthology.orchestra.run.smithi142.stdout:5.1b 2021-03-24T21:58:35.035 INFO:teuthology.orchestra.run.smithi142.stdout:7.10 2021-03-24T21:58:35.036 INFO:teuthology.orchestra.run.smithi142.stdout:5.1d 2021-03-24T21:58:35.036 INFO:teuthology.orchestra.run.smithi142.stdout:5.15 2021-03-24T21:58:35.036 INFO:teuthology.orchestra.run.smithi142.stdout:5.2 2021-03-24T21:58:35.037 INFO:teuthology.orchestra.run.smithi142.stdout:6.a 2021-03-24T21:58:35.037 INFO:teuthology.orchestra.run.smithi142.stdout:6.f 2021-03-24T21:58:35.038 INFO:teuthology.orchestra.run.smithi142.stdout:5.3 2021-03-24T21:58:35.038 INFO:teuthology.orchestra.run.smithi142.stdout:5.4 2021-03-24T21:58:35.038 INFO:teuthology.orchestra.run.smithi142.stdout:5.6 2021-03-24T21:58:35.039 INFO:teuthology.orchestra.run.smithi142.stdout:6.19 2021-03-24T21:58:35.039 INFO:teuthology.orchestra.run.smithi142.stdout:7.21 2021-03-24T21:58:35.040 INFO:teuthology.orchestra.run.smithi142.stdout:5.7 2021-03-24T21:58:35.040 INFO:teuthology.orchestra.run.smithi142.stdout:3.0 2021-03-24T21:58:35.041 INFO:teuthology.orchestra.run.smithi142.stdout:5.b 2021-03-24T21:58:35.041 INFO:teuthology.orchestra.run.smithi142.stdout:5.d 2021-03-24T21:58:35.041 INFO:teuthology.orchestra.run.smithi142.stdout:6.1d 2021-03-24T21:58:35.042 INFO:teuthology.orchestra.run.smithi142.stdout:5.e 2021-03-24T21:58:35.042 INFO:teuthology.orchestra.run.smithi142.stdout:5.f 2021-03-24T21:58:35.043 INFO:teuthology.orchestra.run.smithi142.stdout:6.17 2021-03-24T21:58:35.043 INFO:teuthology.orchestra.run.smithi142.stdout:3.3 2021-03-24T21:58:35.044 INFO:teuthology.orchestra.run.smithi142.stdout:6.10 2021-03-24T21:58:35.044 INFO:teuthology.orchestra.run.smithi142.stdout:6.11 2021-03-24T21:58:35.045 INFO:teuthology.orchestra.run.smithi142.stdout:6.7 2021-03-24T21:58:35.045 INFO:teuthology.orchestra.run.smithi142.stdout:6.1 2021-03-24T21:58:35.046 INFO:teuthology.orchestra.run.smithi142.stdout:7.1a 2021-03-24T21:58:35.046 INFO:teuthology.orchestra.run.smithi142.stdout:6.1c 2021-03-24T21:58:35.046 INFO:teuthology.orchestra.run.smithi142.stdout:7.19 2021-03-24T21:58:35.047 INFO:teuthology.orchestra.run.smithi142.stdout:6.1e 2021-03-24T21:58:35.047 INFO:teuthology.orchestra.run.smithi142.stdout:6.c 2021-03-24T21:58:35.048 INFO:teuthology.orchestra.run.smithi142.stdout:5.12 2021-03-24T21:58:35.048 INFO:teuthology.orchestra.run.smithi142.stdout:6.d 2021-03-24T21:58:35.048 INFO:teuthology.orchestra.run.smithi142.stdout:6.e 2021-03-24T21:58:35.049 INFO:teuthology.orchestra.run.smithi142.stdout:7.13 2021-03-24T21:58:35.049 INFO:teuthology.orchestra.run.smithi142.stdout:7.15 2021-03-24T21:58:35.050 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:35.106 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:35.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:35.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:58:35.195 INFO:tasks.rados.rados.0.smithi125.stdout:1213: done (0 left) 2021-03-24T21:58:35.195 INFO:tasks.rados.rados.0.smithi125.stdout:1214: read oid 34 snap -1 2021-03-24T21:58:35.196 INFO:tasks.rados.rados.0.smithi125.stdout:1214: expect (ObjNum 411 snap 108 seq_num 411) 2021-03-24T21:58:35.196 INFO:tasks.rados.rados.0.smithi125.stdout:1215: rollback oid 28 current snap is 122 2021-03-24T21:58:35.196 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 28 to 121 2021-03-24T21:58:35.196 INFO:tasks.rados.rados.0.smithi125.stdout:1216: read oid 23 snap -1 2021-03-24T21:58:35.196 INFO:tasks.rados.rados.0.smithi125.stdout:1216: expect (ObjNum 427 snap 113 seq_num 427) 2021-03-24T21:58:35.197 INFO:tasks.rados.rados.0.smithi125.stdout:1217: write oid 30 current snap is 122 2021-03-24T21:58:35.197 INFO:tasks.rados.rados.0.smithi125.stdout:1217: seq_num 447 ranges {712729=784777,2091098=467491,3342313=170416} 2021-03-24T21:58:35.206 INFO:tasks.rados.rados.0.smithi125.stdout:1217: writing smithi12556470-30 from 712729 to 1497506 tid 1 2021-03-24T21:58:35.317 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:35.319 INFO:teuthology.orchestra.run.smithi125.stderr:noscrub is set 2021-03-24T21:58:35.320 INFO:tasks.rados.rados.0.smithi125.stdout:1217: writing smithi12556470-30 from 2091098 to 2558589 tid 2 2021-03-24T21:58:35.320 INFO:tasks.rados.rados.0.smithi125.stdout:1217: writing smithi12556470-30 from 3342313 to 3512729 tid 3 2021-03-24T21:58:35.320 INFO:tasks.rados.rados.0.smithi125.stdout:1218: snap_create 2021-03-24T21:58:35.320 INFO:tasks.rados.rados.0.smithi125.stdout:1216: expect (ObjNum 427 snap 113 seq_num 427) 2021-03-24T21:58:35.326 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: cluster 2021-03-24T21:58:35.327 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 2021-03-24T21:58:35.327 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 2021 2021-03-24T21:58:35.327 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: - 2021-03-24T21:58:35.327 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 03 2021-03-24T21:58:35.328 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: - 2021-03-24T21:58:35.328 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 24 2021-03-24T21:58:35.328 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: T 2021-03-24T21:58:35.328 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 21 2021-03-24T21:58:35.329 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: : 2021-03-24T21:58:35.329 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 58 2021-03-24T21:58:35.329 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: : 2021-03-24T21:58:35.329 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 33 2021-03-24T21:58:35.329 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: . 2021-03-24T21:58:35.330 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 622685 2021-03-24T21:58:35.330 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: +0000 2021-03-24T21:58:35.330 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 2021-03-24T21:58:35.330 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: osd.4 2021-03-24T21:58:35.331 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: ( 2021-03-24T21:58:35.331 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: osd 2021-03-24T21:58:35.331 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: . 2021-03-24T21:58:35.331 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 4 2021-03-24T21:58:35.331 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: ) 2021-03-24T21:58:35.332 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 89 2021-03-24T21:58:35.332 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: : 2021-03-24T21:58:35.332 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: cluster 2021-03-24T21:58:35.332 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 2021-03-24T21:58:35.333 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: [DBG] 2021-03-24T21:58:35.333 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 2021-03-24T21:58:35.333 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: purged_snaps scrub starts 2021-03-24T21:58:35.333 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 2021-03-24T21:58:35.333 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: cluster 2021-03-24T21:58:35.334 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 2021-03-24T21:58:35.334 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 2021 2021-03-24T21:58:35.334 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: - 2021-03-24T21:58:35.334 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 03 2021-03-24T21:58:35.335 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: - 2021-03-24T21:58:35.335 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 24 2021-03-24T21:58:35.335 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: T 2021-03-24T21:58:35.335 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 21 2021-03-24T21:58:35.335 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: : 2021-03-24T21:58:35.336 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 58 2021-03-24T21:58:35.336 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: : 2021-03-24T21:58:35.336 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 33 2021-03-24T21:58:35.336 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: . 2021-03-24T21:58:35.336 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 623169 2021-03-24T21:58:35.337 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: +0000 2021-03-24T21:58:35.337 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 2021-03-24T21:58:35.337 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: osd.4 2021-03-24T21:58:35.337 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: ( 2021-03-24T21:58:35.337 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: osd 2021-03-24T21:58:35.338 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: . 2021-03-24T21:58:35.338 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 4 2021-03-24T21:58:35.338 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: ) 2021-03-24T21:58:35.338 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 90 2021-03-24T21:58:35.339 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: : 2021-03-24T21:58:35.339 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: cluster 2021-03-24T21:58:35.339 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 2021-03-24T21:58:35.339 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: [DBG] 2021-03-24T21:58:35.339 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 2021-03-24T21:58:35.340 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: purged_snaps scrub ok 2021-03-24T21:58:35.340 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 2021-03-24T21:58:35.340 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: cluster 2021-03-24T21:58:35.340 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 2021-03-24T21:58:35.341 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 2021 2021-03-24T21:58:35.341 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: - 2021-03-24T21:58:35.341 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 03 2021-03-24T21:58:35.341 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: - 2021-03-24T21:58:35.341 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 24 2021-03-24T21:58:35.342 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: T 2021-03-24T21:58:35.342 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 21 2021-03-24T21:58:35.342 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: : 2021-03-24T21:58:35.342 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 58 2021-03-24T21:58:35.343 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: : 2021-03-24T21:58:35.343 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 34 2021-03-24T21:58:35.343 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: . 2021-03-24T21:58:35.343 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 374996 2021-03-24T21:58:35.343 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: +0000 2021-03-24T21:58:35.344 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 2021-03-24T21:58:35.344 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: mon.a 2021-03-24T21:58:35.344 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: ( 2021-03-24T21:58:35.344 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: mon 2021-03-24T21:58:35.344 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: . 2021-03-24T21:58:35.345 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 0 2021-03-24T21:58:35.345 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: ) 2021-03-24T21:58:35.345 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 1095 2021-03-24T21:58:35.345 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: : 2021-03-24T21:58:35.346 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: cluster 2021-03-24T21:58:35.346 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 2021-03-24T21:58:35.346 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: [DBG] 2021-03-24T21:58:35.346 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 2021-03-24T21:58:35.347 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: osdmap e344: 8 total, 7 up, 6 in 2021-03-24T21:58:35.347 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 2021-03-24T21:58:35.347 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: audit 2021-03-24T21:58:35.348 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 2021-03-24T21:58:35.348 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 2021 2021-03-24T21:58:35.348 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: - 2021-03-24T21:58:35.348 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 03 2021-03-24T21:58:35.348 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: - 2021-03-24T21:58:35.349 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 24 2021-03-24T21:58:35.349 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: T 2021-03-24T21:58:35.349 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 21 2021-03-24T21:58:35.349 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: : 2021-03-24T21:58:35.349 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 58 2021-03-24T21:58:35.350 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: : 2021-03-24T21:58:35.350 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 34 2021-03-24T21:58:35.350 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: . 2021-03-24T21:58:35.350 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 401641 2021-03-24T21:58:35.351 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: +0000 2021-03-24T21:58:35.351 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 2021-03-24T21:58:35.351 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: mon.a 2021-03-24T21:58:35.351 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: ( 2021-03-24T21:58:35.352 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: mon 2021-03-24T21:58:35.352 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: . 2021-03-24T21:58:35.352 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 0 2021-03-24T21:58:35.352 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: ) 2021-03-24T21:58:35.352 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 1096 2021-03-24T21:58:35.353 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: : 2021-03-24T21:58:35.353 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: audit 2021-03-24T21:58:35.353 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 2021-03-24T21:58:35.353 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: [INF] 2021-03-24T21:58:35.353 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 2021-03-24T21:58:35.354 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: from='client.? v1:172.21.15.125:0/925538586' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T21:58:35.354 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 2021-03-24T21:58:35.354 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: cluster 2021-03-24T21:58:35.354 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 2021-03-24T21:58:35.355 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 2021 2021-03-24T21:58:35.355 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: - 2021-03-24T21:58:35.355 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 03 2021-03-24T21:58:35.355 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: - 2021-03-24T21:58:35.355 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 24 2021-03-24T21:58:35.356 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: T 2021-03-24T21:58:35.356 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 21 2021-03-24T21:58:35.356 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: : 2021-03-24T21:58:35.356 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 58 2021-03-24T21:58:35.357 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: : 2021-03-24T21:58:35.357 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 35 2021-03-24T21:58:35.357 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: . 2021-03-24T21:58:35.357 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 179830 2021-03-24T21:58:35.357 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: +0000 2021-03-24T21:58:35.358 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 2021-03-24T21:58:35.358 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: mon.a 2021-03-24T21:58:35.358 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: ( 2021-03-24T21:58:35.358 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: mon 2021-03-24T21:58:35.358 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: . 2021-03-24T21:58:35.359 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 0 2021-03-24T21:58:35.359 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: ) 2021-03-24T21:58:35.359 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 1097 2021-03-24T21:58:35.359 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: : 2021-03-24T21:58:35.360 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: cluster 2021-03-24T21:58:35.360 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 2021-03-24T21:58:35.360 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: [WRN] 2021-03-24T21:58:35.360 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 2021-03-24T21:58:35.360 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:58:35.361 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:35 smithi142 conmon[20125]: 2021-03-24T21:58:35.367 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:35.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:35.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:58:35.471 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:35.480 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[19936]: cluster 2021-03-24T21:58:33. 2021-03-24T21:58:35.481 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[19936]: 622685+0000 osd.4 (osd.4) 89 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:58:35.481 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[19936]: cluster 2021-03-24T21:58:33 2021-03-24T21:58:35.481 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[19936]: .623169+0000 osd.4 (osd.4) 90 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:58:35.481 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:58:35.481 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[19936]: 03-24T21:58:34.374996+0000 2021-03-24T21:58:35.482 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[19936]: mon.a (mon.0) 1095 : cluster [DBG] 2021-03-24T21:58:35.482 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[19936]: osdmap e344: 8 total, 7 up, 6 in 2021-03-24T21:58:35.482 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[19936]: audit 2021-03-24T21:58:35.482 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[19936]: 2021-03-24T21:58:34.401641+0000 2021-03-24T21:58:35.483 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[19936]: mon.a (mon.0) 1096 : audit [INF] 2021-03-24T21:58:35.483 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[19936]: from='client.? v1:172.21.15.125:0/925538586' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T21:58:35.483 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[19936]: cluster 2021-03-24T21:58:35.483 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[19936]: 2021-03-24T21:58:35.179830+0000 mon.a (mon 2021-03-24T21:58:35.483 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[19936]: .0) 1097 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:58:35.484 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: cluster 2021-03-24T21:58:35.484 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 2021-03-24T21:58:35.485 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 2021 2021-03-24T21:58:35.486 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: - 2021-03-24T21:58:35.486 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 03 2021-03-24T21:58:35.486 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: - 2021-03-24T21:58:35.487 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 24 2021-03-24T21:58:35.487 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: T 2021-03-24T21:58:35.488 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 21 2021-03-24T21:58:35.488 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: : 2021-03-24T21:58:35.488 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 58 2021-03-24T21:58:35.489 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: : 2021-03-24T21:58:35.489 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 33 2021-03-24T21:58:35.490 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: . 2021-03-24T21:58:35.490 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 622685 2021-03-24T21:58:35.490 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: +0000 2021-03-24T21:58:35.491 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 2021-03-24T21:58:35.491 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: osd.4 2021-03-24T21:58:35.491 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: ( 2021-03-24T21:58:35.492 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: osd 2021-03-24T21:58:35.492 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: . 2021-03-24T21:58:35.492 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 4 2021-03-24T21:58:35.493 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: ) 2021-03-24T21:58:35.497 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 89 2021-03-24T21:58:35.497 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: : 2021-03-24T21:58:35.497 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: cluster 2021-03-24T21:58:35.498 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 2021-03-24T21:58:35.498 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: [DBG] 2021-03-24T21:58:35.498 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 2021-03-24T21:58:35.498 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: purged_snaps scrub starts 2021-03-24T21:58:35.499 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 2021-03-24T21:58:35.499 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: cluster 2021-03-24T21:58:35.499 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 2021-03-24T21:58:35.500 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 2021 2021-03-24T21:58:35.500 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: - 2021-03-24T21:58:35.500 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 03 2021-03-24T21:58:35.500 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: - 2021-03-24T21:58:35.501 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 24 2021-03-24T21:58:35.501 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: T 2021-03-24T21:58:35.501 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 21 2021-03-24T21:58:35.501 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: : 2021-03-24T21:58:35.502 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 58 2021-03-24T21:58:35.502 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: : 2021-03-24T21:58:35.502 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 33 2021-03-24T21:58:35.502 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: . 2021-03-24T21:58:35.503 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 623169 2021-03-24T21:58:35.503 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: +0000 2021-03-24T21:58:35.503 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 2021-03-24T21:58:35.503 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: osd.4 2021-03-24T21:58:35.503 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: ( 2021-03-24T21:58:35.504 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: osd 2021-03-24T21:58:35.504 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: . 2021-03-24T21:58:35.504 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 4 2021-03-24T21:58:35.504 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: ) 2021-03-24T21:58:35.505 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 90 2021-03-24T21:58:35.505 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: : 2021-03-24T21:58:35.505 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: cluster 2021-03-24T21:58:35.505 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 2021-03-24T21:58:35.506 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: [DBG] 2021-03-24T21:58:35.506 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 2021-03-24T21:58:35.506 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: purged_snaps scrub ok 2021-03-24T21:58:35.506 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 2021-03-24T21:58:35.507 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: cluster 2021-03-24T21:58:35.507 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 2021-03-24T21:58:35.507 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 2021 2021-03-24T21:58:35.507 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: - 2021-03-24T21:58:35.508 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 03 2021-03-24T21:58:35.508 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: - 2021-03-24T21:58:35.508 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 24 2021-03-24T21:58:35.508 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: T 2021-03-24T21:58:35.509 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 21 2021-03-24T21:58:35.509 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: : 2021-03-24T21:58:35.510 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 58 2021-03-24T21:58:35.510 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: : 2021-03-24T21:58:35.510 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 34 2021-03-24T21:58:35.511 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: . 2021-03-24T21:58:35.511 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 374996 2021-03-24T21:58:35.511 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: +0000 2021-03-24T21:58:35.512 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 2021-03-24T21:58:35.512 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: mon.a 2021-03-24T21:58:35.512 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: ( 2021-03-24T21:58:35.512 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: mon 2021-03-24T21:58:35.513 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: . 2021-03-24T21:58:35.513 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 0 2021-03-24T21:58:35.513 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: ) 2021-03-24T21:58:35.513 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 1095 2021-03-24T21:58:35.514 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: : 2021-03-24T21:58:35.514 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: cluster 2021-03-24T21:58:35.514 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 2021-03-24T21:58:35.514 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: [DBG] 2021-03-24T21:58:35.514 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 2021-03-24T21:58:35.515 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: osdmap e344: 8 total, 7 up, 6 in 2021-03-24T21:58:35.515 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 2021-03-24T21:58:35.515 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: audit 2021-03-24T21:58:35.515 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 2021-03-24T21:58:35.516 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 2021 2021-03-24T21:58:35.516 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: - 2021-03-24T21:58:35.516 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 03 2021-03-24T21:58:35.516 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: - 2021-03-24T21:58:35.516 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 24 2021-03-24T21:58:35.517 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: T 2021-03-24T21:58:35.517 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 21 2021-03-24T21:58:35.517 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: : 2021-03-24T21:58:35.518 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 58 2021-03-24T21:58:35.518 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: : 2021-03-24T21:58:35.518 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 34 2021-03-24T21:58:35.519 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: . 2021-03-24T21:58:35.519 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 401641 2021-03-24T21:58:35.519 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: +0000 2021-03-24T21:58:35.519 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 2021-03-24T21:58:35.519 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: mon.a 2021-03-24T21:58:35.520 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: ( 2021-03-24T21:58:35.520 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: mon 2021-03-24T21:58:35.520 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: . 2021-03-24T21:58:35.520 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 0 2021-03-24T21:58:35.520 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: ) 2021-03-24T21:58:35.521 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 1096 2021-03-24T21:58:35.521 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: : 2021-03-24T21:58:35.521 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: audit 2021-03-24T21:58:35.521 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 2021-03-24T21:58:35.522 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: [INF] 2021-03-24T21:58:35.522 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 2021-03-24T21:58:35.522 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: from='client.? v1:172.21.15.125:0/925538586' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T21:58:35.522 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 2021-03-24T21:58:35.523 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: cluster 2021-03-24T21:58:35.523 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 2021-03-24T21:58:35.523 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 2021 2021-03-24T21:58:35.523 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: - 2021-03-24T21:58:35.523 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 03 2021-03-24T21:58:35.524 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: - 2021-03-24T21:58:35.524 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 24 2021-03-24T21:58:35.524 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: T 2021-03-24T21:58:35.524 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 21 2021-03-24T21:58:35.525 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: : 2021-03-24T21:58:35.525 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 58 2021-03-24T21:58:35.525 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: : 2021-03-24T21:58:35.525 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 35 2021-03-24T21:58:35.525 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: . 2021-03-24T21:58:35.526 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 179830 2021-03-24T21:58:35.526 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: +0000 2021-03-24T21:58:35.526 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 2021-03-24T21:58:35.526 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: mon.a 2021-03-24T21:58:35.527 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: ( 2021-03-24T21:58:35.527 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: mon 2021-03-24T21:58:35.527 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: . 2021-03-24T21:58:35.527 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 0 2021-03-24T21:58:35.527 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: ) 2021-03-24T21:58:35.528 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 1097 2021-03-24T21:58:35.528 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: : 2021-03-24T21:58:35.528 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: cluster 2021-03-24T21:58:35.528 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 2021-03-24T21:58:35.529 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: [WRN] 2021-03-24T21:58:35.529 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 2021-03-24T21:58:35.529 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:58:35.529 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:35 smithi125 conmon[24927]: 2021-03-24T21:58:35.530 INFO:tasks.rados.rados.0.smithi125.stdout:1215: finishing rollback tid 0 to smithi12556470-28 2021-03-24T21:58:35.530 INFO:tasks.rados.rados.0.smithi125.stdout:1215: finishing rollback tid 1 to smithi12556470-28 2021-03-24T21:58:35.531 INFO:tasks.rados.rados.0.smithi125.stdout:1215: finishing rollback tid 2 to smithi12556470-28 2021-03-24T21:58:35.531 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 28 v 296 (ObjNum 424 snap 112 seq_num 424) dirty exists 2021-03-24T21:58:35.531 INFO:tasks.rados.rados.0.smithi125.stdout:1214: expect (ObjNum 411 snap 108 seq_num 411) 2021-03-24T21:58:35.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:35.532 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:35.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:58:35.634 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:35.658 INFO:tasks.rados.rados.0.smithi125.stdout:1217: finishing write tid 1 to smithi12556470-30 2021-03-24T21:58:35.658 INFO:tasks.rados.rados.0.smithi125.stdout:1217: finishing write tid 2 to smithi12556470-30 2021-03-24T21:58:35.659 INFO:tasks.rados.rados.0.smithi125.stdout:1217: finishing write tid 3 to smithi12556470-30 2021-03-24T21:58:35.659 INFO:tasks.rados.rados.0.smithi125.stdout:1217: finishing write tid 4 to smithi12556470-30 2021-03-24T21:58:35.660 INFO:tasks.rados.rados.0.smithi125.stdout:1217: finishing write tid 5 to smithi12556470-30 2021-03-24T21:58:35.660 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 30 v 283 (ObjNum 447 snap 122 seq_num 447) dirty exists 2021-03-24T21:58:35.660 INFO:tasks.rados.rados.0.smithi125.stdout:1217: left oid 30 (ObjNum 447 snap 122 seq_num 447) 2021-03-24T21:58:35.680 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:35.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:35.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:58:35.785 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:35.843 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:35.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:35.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:58:35.949 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:36.008 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:36.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:36.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:36.114 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:36.174 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:36.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:36.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:58:36.223 INFO:tasks.rados.rados.0.smithi125.stdout:1214: done (4 left) 2021-03-24T21:58:36.224 INFO:tasks.rados.rados.0.smithi125.stdout:1215: done (3 left) 2021-03-24T21:58:36.224 INFO:tasks.rados.rados.0.smithi125.stdout:1216: done (2 left) 2021-03-24T21:58:36.224 INFO:tasks.rados.rados.0.smithi125.stdout:1217: done (1 left) 2021-03-24T21:58:36.225 INFO:tasks.rados.rados.0.smithi125.stdout:1218: done (0 left) 2021-03-24T21:58:36.225 INFO:tasks.rados.rados.0.smithi125.stdout:1219: write oid 14 current snap is 123 2021-03-24T21:58:36.225 INFO:tasks.rados.rados.0.smithi125.stdout:1219: seq_num 448 ranges {729536=460026,1749580=611149,2853383=452450} 2021-03-24T21:58:36.231 INFO:tasks.rados.rados.0.smithi125.stdout:1219: writing smithi12556470-14 from 729536 to 1189562 tid 1 2021-03-24T21:58:36.239 INFO:tasks.rados.rados.0.smithi125.stdout:1219: writing smithi12556470-14 from 1749580 to 2360729 tid 2 2021-03-24T21:58:36.246 INFO:tasks.rados.rados.0.smithi125.stdout:1219: writing smithi12556470-14 from 2853383 to 3305833 tid 3 2021-03-24T21:58:36.246 INFO:tasks.rados.rados.0.smithi125.stdout:1220: copy_from oid 6 from oid 24 current snap is 123 2021-03-24T21:58:36.246 INFO:tasks.rados.rados.0.smithi125.stdout:1221: read oid 12 snap -1 2021-03-24T21:58:36.247 INFO:tasks.rados.rados.0.smithi125.stdout:1221: expect (ObjNum 437 snap 118 seq_num 437) 2021-03-24T21:58:36.247 INFO:tasks.rados.rados.0.smithi125.stdout:1222: read oid 39 snap -1 2021-03-24T21:58:36.247 INFO:tasks.rados.rados.0.smithi125.stdout:1222: expect (ObjNum 408 snap 106 seq_num 408) 2021-03-24T21:58:36.247 INFO:tasks.rados.rados.0.smithi125.stdout:1223: write oid 20 current snap is 123 2021-03-24T21:58:36.248 INFO:tasks.rados.rados.0.smithi125.stdout:1223: seq_num 449 ranges {746343=651628,1924415=754807,3397159=149184} 2021-03-24T21:58:36.255 INFO:tasks.rados.rados.0.smithi125.stdout:1223: writing smithi12556470-20 from 746343 to 1397971 tid 1 2021-03-24T21:58:36.263 INFO:tasks.rados.rados.0.smithi125.stdout:1223: writing smithi12556470-20 from 1924415 to 2679222 tid 2 2021-03-24T21:58:36.269 INFO:tasks.rados.rados.0.smithi125.stdout:1223: writing smithi12556470-20 from 3397159 to 3546343 tid 3 2021-03-24T21:58:36.270 INFO:tasks.rados.rados.0.smithi125.stdout:1224: snap_remove snap 119 2021-03-24T21:58:36.280 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:36.332 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:36.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:36.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:58:36.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:36 smithi125 conmon[19936]: cluster 2021-03-24T21:58:34.799851+0000 mgr.y (mgr.14144 2021-03-24T21:58:36.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:36 smithi125 conmon[19936]: ) 441 : cluster [DBG] pgmap v679: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 180 MiB data, 552 MiB used, 536 GiB / 536 GiB avail; 46 MiB/s rd, 11 MiB/s wr, 68 op/s; 218/1188 objects degraded (18.350%); 3.1 MiB/s, 0 objects/s recovering 2021-03-24T21:58:36.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:36 smithi125 conmon[19936]: audit 2021-03-24T21:58:35.191932+0000 mon.a (mon.0) 1098 : audit [INF] from='client.? v1:172.21.15.125:0/925538586' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T21:58:36.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:36 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:58:36.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:36 smithi125 conmon[19936]: 21:58:35.192016+0000 mon.a (mon.0) 1099 : cluster [DBG] osdmap e345: 8 total, 7 up, 6 in 2021-03-24T21:58:36.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:36 smithi125 conmon[24927]: cluster 2021-03-24T21:58:36.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:36 smithi125 conmon[24927]: 2021-03-24T21:58:36.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:36 smithi125 conmon[24927]: 2021 2021-03-24T21:58:36.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:36 smithi125 conmon[24927]: - 2021-03-24T21:58:36.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:36 smithi125 conmon[24927]: 03 2021-03-24T21:58:36.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:36 smithi125 conmon[24927]: - 2021-03-24T21:58:36.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:36 smithi125 conmon[24927]: 24 2021-03-24T21:58:36.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:36 smithi125 conmon[24927]: T 2021-03-24T21:58:36.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:36 smithi125 conmon[24927]: 21 2021-03-24T21:58:36.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:36 smithi125 conmon[24927]: : 2021-03-24T21:58:36.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:36 smithi125 conmon[24927]: 58 2021-03-24T21:58:36.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:36 smithi125 conmon[24927]: : 2021-03-24T21:58:36.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:36 smithi125 conmon[24927]: 34 2021-03-24T21:58:36.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:36 smithi125 conmon[24927]: . 2021-03-24T21:58:36.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:36 smithi125 conmon[24927]: 799851 2021-03-24T21:58:36.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:36 smithi125 conmon[24927]: +0000 2021-03-24T21:58:36.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:36 smithi125 conmon[24927]: 2021-03-24T21:58:36.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:36 smithi125 conmon[24927]: mgr.y 2021-03-24T21:58:36.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:36 smithi125 conmon[24927]: ( 2021-03-24T21:58:36.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:36 smithi125 conmon[24927]: mgr 2021-03-24T21:58:36.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:36 smithi125 conmon[24927]: . 2021-03-24T21:58:36.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:36 smithi125 conmon[24927]: 14144 2021-03-24T21:58:36.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:36 smithi125 conmon[24927]: ) 2021-03-24T21:58:36.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:36 smithi125 conmon[24927]: 441 2021-03-24T21:58:36.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:36 smithi125 conmon[24927]: : 2021-03-24T21:58:36.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:36 smithi125 conmon[24927]: cluster 2021-03-24T21:58:36.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:36 smithi125 conmon[24927]: 2021-03-24T21:58:36.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:36 smithi125 conmon[24927]: [DBG] 2021-03-24T21:58:36.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:36 smithi125 conmon[24927]: 2021-03-24T21:58:36.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:36 smithi125 conmon[24927]: pgmap v679: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 180 MiB data, 552 MiB used, 536 GiB / 536 GiB avail; 46 MiB/s rd, 11 MiB/s wr, 68 op/s; 218/1188 objects degraded (18.350%); 3.1 MiB/s, 0 objects/s recovering 2021-03-24T21:58:36.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:36 smithi125 conmon[24927]: 2021-03-24T21:58:36.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:36 smithi125 conmon[24927]: audit 2021-03-24T21:58:36.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:36 smithi125 conmon[24927]: 2021-03-24T21:58:36.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:36 smithi125 conmon[24927]: 2021 2021-03-24T21:58:36.420 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:36 smithi125 conmon[24927]: - 2021-03-24T21:58:36.420 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:36 smithi125 conmon[24927]: 03 2021-03-24T21:58:36.420 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:36 smithi125 conmon[24927]: - 2021-03-24T21:58:36.421 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:36 smithi125 conmon[24927]: 24 2021-03-24T21:58:36.421 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:36 smithi125 conmon[24927]: T 2021-03-24T21:58:36.421 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:36 smithi125 conmon[24927]: 21 2021-03-24T21:58:36.421 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:36 smithi125 conmon[24927]: : 2021-03-24T21:58:36.421 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:36 smithi125 conmon[24927]: 58:35.191932+0000 mon.a (mon.0) 1098 2021-03-24T21:58:36.422 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:36 smithi125 conmon[24927]: : audit [INF] from='client.? v1:172.21.15.125:0/925538586' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T21:58:36.422 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:36 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:58:36.422 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:36 smithi125 conmon[24927]: -03-24T21:58:35.192016+0000 2021-03-24T21:58:36.422 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:36 smithi125 conmon[24927]: mon.a (mon.0) 1099 : cluster [DBG] 2021-03-24T21:58:36.423 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:36 smithi125 conmon[24927]: osdmap e345: 8 total, 7 up, 6 in 2021-03-24T21:58:36.439 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:36.489 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:36.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:36.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:36.597 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:36.611 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:36 smithi142 conmon[20125]: cluster 2021-03-24T21:58: 2021-03-24T21:58:36.611 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:36 smithi142 conmon[20125]: 34.799851+0000 mgr.y (mgr.14144) 441 : cluster [DBG] pgmap v679: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 180 MiB data, 552 MiB used, 536 GiB / 536 GiB avail; 46 MiB/s rd, 11 MiB/s wr, 68 op/s; 218/1188 objects degraded (18.350%); 3.1 MiB/s, 0 objects/s recovering 2021-03-24T21:58:36.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:36 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:58:36.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:36 smithi142 conmon[20125]: 03-24T21:58:35.191932+0000 2021-03-24T21:58:36.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:36 smithi142 conmon[20125]: mon.a (mon.0) 1098 2021-03-24T21:58:36.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:36 smithi142 conmon[20125]: : audit [INF] from='client.? v1:172.21.15.125:0/925538586' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T21:58:36.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:36 smithi142 conmon[20125]: cluster 2021-03-24T21:58:36.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:36 smithi142 conmon[20125]: 2021-03-24T 2021-03-24T21:58:36.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:36 smithi142 conmon[20125]: 21:58:35.192016+0000 2021-03-24T21:58:36.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:36 smithi142 conmon[20125]: mon.a (mon.0) 2021-03-24T21:58:36.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:36 smithi142 conmon[20125]: 1099 : cluster [DBG] osdmap e345: 8 total, 7 up, 6 in 2021-03-24T21:58:36.656 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:36.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:36.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:58:36.752 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell -n osd.5 --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph-objectstore-tool --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op export --pgid 3.3 --file /var/log/ceph/exp.3.3.5 2021-03-24T21:58:36.762 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:36.812 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:36.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:36.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:58:36.918 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:36.957 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/osd.5/config 2021-03-24T21:58:36.965 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:36.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:36.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:58:37.072 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:37.136 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:37.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:37.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:58:37.158 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-03-24T21:58:37.159 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-03-24T21:58:37.243 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:37.251 INFO:tasks.rados.rados.0.smithi125.stdout:1224: done (5 left) 2021-03-24T21:58:37.251 INFO:tasks.rados.rados.0.smithi125.stdout:1225: delete oid 18 current snap is 123 2021-03-24T21:58:37.251 INFO:tasks.rados.rados.0.smithi125.stdout:1219: finishing write tid 1 to smithi12556470-14 2021-03-24T21:58:37.252 INFO:tasks.rados.rados.0.smithi125.stdout:1219: finishing write tid 2 to smithi12556470-14 2021-03-24T21:58:37.252 INFO:tasks.rados.rados.0.smithi125.stdout:1221: expect (ObjNum 437 snap 118 seq_num 437) 2021-03-24T21:58:37.297 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:37.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:37.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:58:37.406 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:37.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:37 smithi125 conmon[24927]: cluster 2021-03-24T21:58 2021-03-24T21:58:37.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:37 smithi125 conmon[24927]: :36.220670+0000 mon.a (mon.0) 1100 : cluster [DBG] osdmap e346: 8 total, 7 up, 6 in 2021-03-24T21:58:37.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:37 smithi125 conmon[19936]: cluster 2021-03-24T21:58:36.220670+0000 mon.a (mon.0 2021-03-24T21:58:37.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:37 smithi125 conmon[19936]: ) 1100 : cluster [DBG] osdmap e346: 8 total, 7 up, 6 in 2021-03-24T21:58:37.451 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:37.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:37.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:58:37.468 INFO:tasks.rados.rados.0.smithi125.stdout:1221: done (5 left) 2021-03-24T21:58:37.468 INFO:tasks.rados.rados.0.smithi125.stdout:1225: done (4 left) 2021-03-24T21:58:37.469 INFO:tasks.rados.rados.0.smithi125.stdout:1226: rollback oid 10 current snap is 123 2021-03-24T21:58:37.469 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 10 to 108 2021-03-24T21:58:37.469 INFO:tasks.rados.rados.0.smithi125.stdout:1227: snap_create 2021-03-24T21:58:37.469 INFO:tasks.rados.rados.0.smithi125.stdout:1223: finishing write tid 1 to smithi12556470-20 2021-03-24T21:58:37.470 INFO:tasks.rados.rados.0.smithi125.stdout:1219: finishing write tid 3 to smithi12556470-14 2021-03-24T21:58:37.470 INFO:tasks.rados.rados.0.smithi125.stdout:1222: expect (ObjNum 408 snap 106 seq_num 408) 2021-03-24T21:58:37.514 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:37 smithi142 conmon[20125]: cluster 2021-03-24T21:58:36.220670+0000 mon.a ( 2021-03-24T21:58:37.514 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:37 smithi142 conmon[20125]: mon.0) 1100 : cluster [DBG] osdmap e346: 8 total, 7 up, 6 in 2021-03-24T21:58:37.557 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:37.570 INFO:tasks.rados.rados.0.smithi125.stdout:1219: finishing write tid 4 to smithi12556470-14 2021-03-24T21:58:37.571 INFO:tasks.rados.rados.0.smithi125.stdout:1219: finishing write tid 5 to smithi12556470-14 2021-03-24T21:58:37.571 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 14 v 400 (ObjNum 448 snap 123 seq_num 448) dirty exists 2021-03-24T21:58:37.571 INFO:tasks.rados.rados.0.smithi125.stdout:1219: left oid 14 (ObjNum 448 snap 123 seq_num 448) 2021-03-24T21:58:37.571 INFO:tasks.rados.rados.0.smithi125.stdout:1223: finishing write tid 2 to smithi12556470-20 2021-03-24T21:58:37.572 INFO:tasks.rados.rados.0.smithi125.stdout:1223: finishing write tid 3 to smithi12556470-20 2021-03-24T21:58:37.572 INFO:tasks.rados.rados.0.smithi125.stdout:1223: finishing write tid 4 to smithi12556470-20 2021-03-24T21:58:37.572 INFO:tasks.rados.rados.0.smithi125.stdout:1223: finishing write tid 5 to smithi12556470-20 2021-03-24T21:58:37.572 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 20 v 271 (ObjNum 449 snap 123 seq_num 449) dirty exists 2021-03-24T21:58:37.573 INFO:tasks.rados.rados.0.smithi125.stdout:1223: left oid 20 (ObjNum 449 snap 123 seq_num 449) 2021-03-24T21:58:37.573 INFO:tasks.rados.rados.0.smithi125.stdout:1220: finishing copy_from to smithi12556470-6 2021-03-24T21:58:37.573 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 6 v 349 (ObjNum 432 snap 115 seq_num 432) dirty exists 2021-03-24T21:58:37.573 INFO:tasks.rados.rados.0.smithi125.stdout:1220: finishing copy_from racing read to smithi12556470-6 2021-03-24T21:58:37.574 INFO:tasks.rados.rados.0.smithi125.stdout:1226: finishing rollback tid 0 to smithi12556470-10 2021-03-24T21:58:37.574 INFO:tasks.rados.rados.0.smithi125.stdout:1226: finishing rollback tid 1 to smithi12556470-10 2021-03-24T21:58:37.574 INFO:tasks.rados.rados.0.smithi125.stdout:1226: finishing rollback tid 2 to smithi12556470-10 2021-03-24T21:58:37.574 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 10 v 370 (ObjNum 393 snap 100 seq_num 393) dirty exists 2021-03-24T21:58:37.612 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:37.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:37.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:58:37.718 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:37.755 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-24T21:58:37.756 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T21:58:37.784 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:37.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:37.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:58:37.896 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:37.949 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:37.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:37.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:58:38.054 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:38.105 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:38.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:38.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:58:38.211 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:38.256 INFO:tasks.rados.rados.0.smithi125.stdout:1219: done (5 left) 2021-03-24T21:58:38.256 INFO:tasks.rados.rados.0.smithi125.stdout:1220: done (4 left) 2021-03-24T21:58:38.256 INFO:tasks.rados.rados.0.smithi125.stdout:1222: done (3 left) 2021-03-24T21:58:38.256 INFO:tasks.rados.rados.0.smithi125.stdout:1223: done (2 left) 2021-03-24T21:58:38.257 INFO:tasks.rados.rados.0.smithi125.stdout:1226: done (1 left) 2021-03-24T21:58:38.257 INFO:tasks.rados.rados.0.smithi125.stdout:1227: done (0 left) 2021-03-24T21:58:38.257 INFO:tasks.rados.rados.0.smithi125.stdout:1228: read oid 32 snap 122 2021-03-24T21:58:38.257 INFO:tasks.rados.rados.0.smithi125.stdout:1228: expect (ObjNum 370 snap 95 seq_num 370) 2021-03-24T21:58:38.257 INFO:tasks.rados.rados.0.smithi125.stdout:1229: write (excl) oid 16 current snap is 124 2021-03-24T21:58:38.258 INFO:tasks.rados.rados.0.smithi125.stdout:1229: seq_num 451 ranges {779957=402126,1925026=758476,3335652=244305} 2021-03-24T21:58:38.265 INFO:tasks.rados.rados.0.smithi125.stdout:1229: writing smithi12556470-16 from 779957 to 1182083 tid 1 2021-03-24T21:58:38.277 INFO:tasks.rados.rados.0.smithi125.stdout:1229: writing smithi12556470-16 from 1925026 to 2683502 tid 2 2021-03-24T21:58:38.285 INFO:tasks.rados.rados.0.smithi125.stdout:1229: writing smithi12556470-16 from 3335652 to 3579957 tid 3 2021-03-24T21:58:38.285 INFO:tasks.rados.rados.0.smithi125.stdout:1230: read oid 21 snap 116 2021-03-24T21:58:38.285 INFO:tasks.rados.rados.0.smithi125.stdout:1230: expect (ObjNum 418 snap 108 seq_num 418) 2021-03-24T21:58:38.286 INFO:tasks.rados.rados.0.smithi125.stdout:1231: write (excl) oid 5 current snap is 124 2021-03-24T21:58:38.286 INFO:tasks.rados.rados.0.smithi125.stdout:1231: seq_num 452 ranges {796764=477375,1983508=618487,3363075=233689} 2021-03-24T21:58:38.289 INFO:tasks.rados.rados.0.smithi125.stdout:1231: writing smithi12556470-5 from 796764 to 1274139 tid 1 2021-03-24T21:58:38.320 INFO:tasks.rados.rados.0.smithi125.stdout:1231: writing smithi12556470-5 from 1983508 to 2601995 tid 2 2021-03-24T21:58:38.327 INFO:tasks.rados.rados.0.smithi125.stdout:1231: writing smithi12556470-5 from 3363075 to 3596764 tid 3 2021-03-24T21:58:38.333 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:38.334 INFO:tasks.rados.rados.0.smithi125.stdout:1229: finishing write tid 1 to smithi12556470-16 2021-03-24T21:58:38.334 INFO:tasks.rados.rados.0.smithi125.stdout:1228: expect (ObjNum 370 snap 95 seq_num 370) 2021-03-24T21:58:38.340 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:38 smithi125 conmon[19936]: cluster 2021-03-24T21:58 2021-03-24T21:58:38.340 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:38 smithi125 conmon[19936]: :36.800986+0000 mgr.y (mgr.14144) 442 : cluster [DBG] pgmap v682: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 184 MiB data, 569 MiB used, 536 GiB / 536 GiB avail; 46 MiB/s rd, 12 MiB/s wr, 69 op/s; 218/1188 objects degraded (18.350%); 3.1 MiB/s, 0 objects/s recovering 2021-03-24T21:58:38.340 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:38 smithi125 conmon[19936]: cluster 2021-03-24T21:58:38.340 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:38 smithi125 conmon[19936]: 2021-03-24T21:58:37.244735 2021-03-24T21:58:38.341 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:38 smithi125 conmon[19936]: +0000 mon.a (mon.0) 1101 2021-03-24T21:58:38.341 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:38 smithi125 conmon[19936]: : cluster [DBG] osdmap e347: 8 total, 7 up, 6 in 2021-03-24T21:58:38.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:38.342 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:38 smithi125 conmon[24927]: cluster 2021-03-24T21:58:36 2021-03-24T21:58:38.342 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:38 smithi125 conmon[24927]: .800986+0000 mgr.y (mgr.14144) 442 : cluster [DBG] pgmap v682: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 184 MiB data, 569 MiB used, 536 GiB / 536 GiB avail; 46 MiB/s rd, 12 MiB/s wr, 69 op/s; 218/1188 objects degraded (18.350%); 3.1 MiB/s, 0 objects/s recovering 2021-03-24T21:58:38.342 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:38 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:58:38.343 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:38 smithi125 conmon[24927]: 03-24T21:58:37.244735+0000 mon.a (mon.0) 1101 : cluster [DBG] osdmap e347: 8 total, 7 up, 6 in 2021-03-24T21:58:38.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:58:38.444 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:38.492 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:38.493 INFO:tasks.rados.rados.0.smithi125.stdout:1230: expect (ObjNum 418 snap 108 seq_num 418) 2021-03-24T21:58:38.495 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:38 smithi142 conmon[20125]: cluster 2021-03-24T21:58:36.800986+0000 mgr.y (mgr.14144) 442 : cluster 2021-03-24T21:58:38.495 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:38 smithi142 conmon[20125]: [DBG] pgmap v682: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 184 MiB data, 569 MiB used, 536 GiB / 536 GiB avail; 46 MiB/s rd, 12 MiB/s wr, 69 op/s; 218/1188 objects degraded (18.350%); 3.1 MiB/s, 0 objects/s recovering 2021-03-24T21:58:38.495 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:38 smithi142 conmon[20125]: cluster 2021-03-24T21:58:37.244735+0000 mon.a (mon.0 2021-03-24T21:58:38.496 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:38 smithi142 conmon[20125]: ) 1101 : cluster [DBG] osdmap e347: 8 total, 7 up, 6 in 2021-03-24T21:58:38.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:38.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:38.598 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:38.646 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:38.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:38.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:38.654 INFO:tasks.rados.rados.0.smithi125.stdout:1229: finishing write tid 2 to smithi12556470-16 2021-03-24T21:58:38.655 INFO:tasks.rados.rados.0.smithi125.stdout:1229: finishing write tid 3 to smithi12556470-16 2021-03-24T21:58:38.655 INFO:tasks.rados.rados.0.smithi125.stdout:1229: finishing write tid 4 to smithi12556470-16 2021-03-24T21:58:38.656 INFO:tasks.rados.rados.0.smithi125.stdout:1228: done (3 left) 2021-03-24T21:58:38.656 INFO:tasks.rados.rados.0.smithi125.stdout:1230: done (2 left) 2021-03-24T21:58:38.656 INFO:tasks.rados.rados.0.smithi125.stdout:1232: read oid 9 snap -1 2021-03-24T21:58:38.657 INFO:tasks.rados.rados.0.smithi125.stdout:1232: expect deleted 2021-03-24T21:58:38.657 INFO:tasks.rados.rados.0.smithi125.stdout:1233: read oid 31 snap -1 2021-03-24T21:58:38.658 INFO:tasks.rados.rados.0.smithi125.stdout:1233: expect (ObjNum 420 snap 108 seq_num 420) 2021-03-24T21:58:38.658 INFO:tasks.rados.rados.0.smithi125.stdout:1234: snap_create 2021-03-24T21:58:38.659 INFO:tasks.rados.rados.0.smithi125.stdout:1229: finishing write tid 5 to smithi12556470-16 2021-03-24T21:58:38.659 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 16 v 232 (ObjNum 451 snap 124 seq_num 451) dirty exists 2021-03-24T21:58:38.659 INFO:tasks.rados.rados.0.smithi125.stdout:1229: left oid 16 (ObjNum 451 snap 124 seq_num 451) 2021-03-24T21:58:38.660 INFO:tasks.rados.rados.0.smithi125.stdout:1231: finishing write tid 1 to smithi12556470-5 2021-03-24T21:58:38.660 INFO:tasks.rados.rados.0.smithi125.stdout:1231: finishing write tid 2 to smithi12556470-5 2021-03-24T21:58:38.661 INFO:tasks.rados.rados.0.smithi125.stdout:1231: finishing write tid 3 to smithi12556470-5 2021-03-24T21:58:38.661 INFO:tasks.rados.rados.0.smithi125.stdout:1231: finishing write tid 4 to smithi12556470-5 2021-03-24T21:58:38.662 INFO:tasks.rados.rados.0.smithi125.stdout:1231: finishing write tid 5 to smithi12556470-5 2021-03-24T21:58:38.662 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 5 v 363 (ObjNum 452 snap 124 seq_num 452) dirty exists 2021-03-24T21:58:38.662 INFO:tasks.rados.rados.0.smithi125.stdout:1231: left oid 5 (ObjNum 452 snap 124 seq_num 452) 2021-03-24T21:58:38.663 INFO:tasks.rados.rados.0.smithi125.stdout:1233: expect (ObjNum 420 snap 108 seq_num 420) 2021-03-24T21:58:38.753 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:38.796 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.0 dump_historic_ops 2021-03-24T21:58:38.814 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:38.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:38.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:58:38.919 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:38.979 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:38.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:38.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:39.086 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:39.113 INFO:teuthology.orchestra.run.smithi142.stderr:Exporting 3.3 info 3.3( v 89'2 (0'0,89'2] local-lis/les=108/109 n=1 ec=60/60 lis/c=108/108 les/c/f=109/112/0 sis=108) 2021-03-24T21:58:39.114 INFO:teuthology.orchestra.run.smithi142.stderr:Read #3:c00c4a6b:::zonegroup_info.62447104-cea8-48da-bd05-4040025bf9f2:head# 2021-03-24T21:58:39.115 INFO:teuthology.orchestra.run.smithi142.stderr:Export successful 2021-03-24T21:58:39.127 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:58:39.137 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:39.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:39.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:39.160 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-24T21:58:39.203 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-24T21:58:39.244 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:39.268 INFO:tasks.rados.rados.0.smithi125.stdout:1229: done (4 left) 2021-03-24T21:58:39.268 INFO:tasks.rados.rados.0.smithi125.stdout:1231: done (3 left) 2021-03-24T21:58:39.268 INFO:tasks.rados.rados.0.smithi125.stdout:1232: done (2 left) 2021-03-24T21:58:39.269 INFO:tasks.rados.rados.0.smithi125.stdout:1233: done (1 left) 2021-03-24T21:58:39.269 INFO:tasks.rados.rados.0.smithi125.stdout:1234: done (0 left) 2021-03-24T21:58:39.269 INFO:tasks.rados.rados.0.smithi125.stdout:1235: read oid 2 snap -1 2021-03-24T21:58:39.270 INFO:tasks.rados.rados.0.smithi125.stdout:1235: expect (ObjNum 425 snap 112 seq_num 425) 2021-03-24T21:58:39.270 INFO:tasks.rados.rados.0.smithi125.stdout:1236: rollback oid 33 current snap is 125 2021-03-24T21:58:39.270 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 33 to 113 2021-03-24T21:58:39.270 INFO:tasks.rados.rados.0.smithi125.stdout:1237: read oid 10 snap -1 2021-03-24T21:58:39.271 INFO:tasks.rados.rados.0.smithi125.stdout:1237: expect (ObjNum 393 snap 100 seq_num 393) 2021-03-24T21:58:39.271 INFO:tasks.rados.rados.0.smithi125.stdout:1238: snap_create 2021-03-24T21:58:39.278 INFO:tasks.rados.rados.0.smithi125.stdout:1236: finishing rollback tid 0 to smithi12556470-33 2021-03-24T21:58:39.285 INFO:tasks.rados.rados.0.smithi125.stdout:1235: expect (ObjNum 425 snap 112 seq_num 425) 2021-03-24T21:58:39.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:39 smithi142 conmon[20125]: cluster 2021-03-24T21:58:38.253476+0000 mon.a (mon.0) 1102 : cluster 2021-03-24T21:58:39.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:39 smithi142 conmon[20125]: [DBG] osdmap e348: 8 total, 7 up, 6 in 2021-03-24T21:58:39.411 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:39.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:39.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:58:39.441 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:39 smithi125 conmon[19936]: cluster 2021-03-24T21:58:38.253476+0000 mon.a (mon.0) 1102 : cluster [DBG] osdmap e348: 8 total, 7 up, 6 in 2021-03-24T21:58:39.441 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:39 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:58:39.441 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:39 smithi125 conmon[24927]: -24T21:58:38.253476+0000 mon.a (mon.0) 1102 : 2021-03-24T21:58:39.441 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:39 smithi125 conmon[24927]: cluster [DBG] osdmap e348: 8 total, 7 up, 6 in 2021-03-24T21:58:39.527 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:39.597 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:39.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:39.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:39.623 INFO:tasks.rados.rados.0.smithi125.stdout:1237: expect (ObjNum 393 snap 100 seq_num 393) 2021-03-24T21:58:39.702 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:39.742 INFO:tasks.rados.rados.0.smithi125.stdout:1236: finishing rollback tid 1 to smithi12556470-33 2021-03-24T21:58:39.743 INFO:tasks.rados.rados.0.smithi125.stdout:1236: finishing rollback tid 2 to smithi12556470-33 2021-03-24T21:58:39.743 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 33 v 374 (ObjNum 298 snap 74 seq_num 298) dirty exists 2021-03-24T21:58:39.754 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:39.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:39.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:58:39.853 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:58:39.860 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:39.913 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:39.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:39.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:58:40.019 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:40.072 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:40.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:40.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:40.178 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:40.191 INFO:tasks.rados.rados.0.smithi125.stdout:1235: done (3 left) 2021-03-24T21:58:40.192 INFO:tasks.rados.rados.0.smithi125.stdout:1236: done (2 left) 2021-03-24T21:58:40.192 INFO:tasks.rados.rados.0.smithi125.stdout:1237: done (1 left) 2021-03-24T21:58:40.192 INFO:tasks.rados.rados.0.smithi125.stdout:1238: done (0 left) 2021-03-24T21:58:40.192 INFO:tasks.rados.rados.0.smithi125.stdout:1239: write (excl) oid 16 current snap is 126 2021-03-24T21:58:40.193 INFO:tasks.rados.rados.0.smithi125.stdout:1239: seq_num 453 ranges {413571=552624,1641990=762145,2990498=425163} 2021-03-24T21:58:40.199 INFO:tasks.rados.rados.0.smithi125.stdout:1239: writing smithi12556470-16 from 413571 to 966195 tid 1 2021-03-24T21:58:40.208 INFO:tasks.rados.rados.0.smithi125.stdout:1239: writing smithi12556470-16 from 1641990 to 2404135 tid 2 2021-03-24T21:58:40.230 INFO:tasks.rados.rados.0.smithi125.stdout:1239: writing smithi12556470-16 from 2990498 to 3415661 tid 3 2021-03-24T21:58:40.231 INFO:tasks.rados.rados.0.smithi125.stdout:1240: rollback oid 5 current snap is 126 2021-03-24T21:58:40.231 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 5 to 108 2021-03-24T21:58:40.231 INFO:tasks.rados.rados.0.smithi125.stdout:1241: write (excl) oid 25 current snap is 126 2021-03-24T21:58:40.232 INFO:tasks.rados.rados.0.smithi125.stdout:1241: seq_num 454 ranges {430378=627873,1700472=622156,3017921=612457} 2021-03-24T21:58:40.254 INFO:tasks.rados.rados.0.smithi125.stdout:1241: writing smithi12556470-25 from 430378 to 1058251 tid 1 2021-03-24T21:58:40.265 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[19936]: cluster 2021-03-24T21:58:38.801973+0000 mgr.y 2021-03-24T21:58:40.265 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[19936]: (mgr.14144) 443 : cluster [DBG] pgmap v685: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 184 MiB data, 569 MiB used, 536 GiB / 536 GiB avail; 255 B/s rd, 381 KiB/s wr, 1 op/s; 219/1191 objects degraded (18.388%) 2021-03-24T21:58:40.265 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[19936]: cluster 2021-03-24T21:58:39.228329+0000 osd.7 (osd.7) 79 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:58:40.266 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[19936]: cluster 2021-03-24T21:58:39 2021-03-24T21:58:40.266 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[19936]: .229014+0000 osd.7 (osd.7) 80 : cluster [DBG] 2021-03-24T21:58:40.266 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[19936]: purged_snaps scrub ok 2021-03-24T21:58:40.267 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[19936]: cluster 2021-03-24T21:58:40.267 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[19936]: 2021-03-24T21 2021-03-24T21:58:40.267 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[19936]: :58:39.247876+0000 mon.a (mon.0) 2021-03-24T21:58:40.267 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[19936]: 1103 : cluster [WRN] Health check update: Degraded data redundancy: 219/1191 objects degraded (18.388%), 40 pgs degraded (PG_DEGRADED) 2021-03-24T21:58:40.268 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[19936]: cluster 2021-03-24T21:58:40.268 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[19936]: 2021-03-24T21:58:39.265836+0000 2021-03-24T21:58:40.268 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[19936]: mon.a (mon.0) 1104 : cluster [DBG] osdmap e349: 8 total, 7 up, 6 in 2021-03-24T21:58:40.268 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[19936]: 2021-03-24T21:58:40.269 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[19936]: cluster 2021-03-24T21:58:40.269 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[19936]: 2021-03-24T21:58 2021-03-24T21:58:40.269 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[19936]: :40.189366+0000 mon.a (mon.0) 1105 2021-03-24T21:58:40.270 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[19936]: : cluster [DBG] osdmap e350: 8 total, 7 up, 6 in 2021-03-24T21:58:40.275 INFO:tasks.rados.rados.0.smithi125.stdout:1241: writing smithi12556470-25 from 1700472 to 2322628 tid 2 2021-03-24T21:58:40.288 INFO:tasks.rados.rados.0.smithi125.stdout:1241: writing smithi12556470-25 from 3017921 to 3630378 tid 3 2021-03-24T21:58:40.289 INFO:tasks.rados.rados.0.smithi125.stdout:1242: snap_create 2021-03-24T21:58:40.289 INFO:tasks.rados.rados.0.smithi125.stdout:1240: finishing rollback tid 0 to smithi12556470-5 2021-03-24T21:58:40.289 INFO:tasks.rados.rados.0.smithi125.stdout:1240: finishing rollback tid 1 to smithi12556470-5 2021-03-24T21:58:40.289 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 5 v 366 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-03-24T21:58:40.289 INFO:tasks.rados.rados.0.smithi125.stdout:1239: finishing write tid 1 to smithi12556470-16 2021-03-24T21:58:40.290 INFO:tasks.rados.rados.0.smithi125.stdout:1239: finishing write tid 2 to smithi12556470-16 2021-03-24T21:58:40.290 INFO:tasks.rados.rados.0.smithi125.stdout:1239: finishing write tid 3 to smithi12556470-16 2021-03-24T21:58:40.290 INFO:tasks.rados.rados.0.smithi125.stdout:1239: finishing write tid 4 to smithi12556470-16 2021-03-24T21:58:40.290 INFO:tasks.rados.rados.0.smithi125.stdout:1239: finishing write tid 5 to smithi12556470-16 2021-03-24T21:58:40.291 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 16 v 237 (ObjNum 453 snap 126 seq_num 453) dirty exists 2021-03-24T21:58:40.291 INFO:tasks.rados.rados.0.smithi125.stdout:1239: left oid 16 (ObjNum 453 snap 126 seq_num 453) 2021-03-24T21:58:40.292 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:40.293 INFO:tasks.rados.rados.0.smithi125.stdout:1241: finishing write tid 1 to smithi12556470-25 2021-03-24T21:58:40.298 INFO:tasks.rados.rados.0.smithi125.stdout:1241: finishing write tid 2 to smithi12556470-25 2021-03-24T21:58:40.299 INFO:tasks.rados.rados.0.smithi125.stdout:1241: finishing write tid 3 to smithi12556470-25 2021-03-24T21:58:40.299 INFO:tasks.rados.rados.0.smithi125.stdout:1241: finishing write tid 4 to smithi12556470-25 2021-03-24T21:58:40.299 INFO:tasks.rados.rados.0.smithi125.stdout:1241: finishing write tid 5 to smithi12556470-25 2021-03-24T21:58:40.299 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 25 v 276 (ObjNum 454 snap 126 seq_num 454) dirty exists 2021-03-24T21:58:40.299 INFO:tasks.rados.rados.0.smithi125.stdout:1241: left oid 25 (ObjNum 454 snap 126 seq_num 454) 2021-03-24T21:58:40.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:40.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:58:40.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:40 smithi142 conmon[20125]: cluster 2021-03-24T21:58:38 2021-03-24T21:58:40.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:40 smithi142 conmon[20125]: .801973+0000 mgr.y (mgr.14144) 443 : cluster [DBG] pgmap v685: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 184 MiB data, 569 MiB used, 536 GiB / 536 GiB avail; 255 B/s rd, 381 KiB/s wr, 1 op/s; 219/1191 objects degraded (18.388%) 2021-03-24T21:58:40.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:40 smithi142 conmon[20125]: cluster 2021-03-24T21 2021-03-24T21:58:40.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:40 smithi142 conmon[20125]: :58:39.228329+0000 osd.7 (osd.7) 79 : cluster [DBG] 2021-03-24T21:58:40.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:40 smithi142 conmon[20125]: purged_snaps scrub starts 2021-03-24T21:58:40.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:40 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:58:40.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:40 smithi142 conmon[20125]: 21:58:39.229014+0000 osd.7 (osd 2021-03-24T21:58:40.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:40 smithi142 conmon[20125]: .7) 80 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:58:40.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:40 smithi142 conmon[20125]: 2021-03-24T21:58:40.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:40 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:58:40.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:40 smithi142 conmon[20125]: 24T21:58:39.247876 2021-03-24T21:58:40.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:40 smithi142 conmon[20125]: +0000 mon.a (mon.0) 1103 : cluster 2021-03-24T21:58:40.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:40 smithi142 conmon[20125]: [WRN] Health check update: Degraded data redundancy: 219/1191 objects degraded (18.388%), 40 pgs degraded (PG_DEGRADED) 2021-03-24T21:58:40.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:40 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:58:40.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:40 smithi142 conmon[20125]: T21:58:39.265836+0000 2021-03-24T21:58:40.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:40 smithi142 conmon[20125]: mon.a (mon.0) 1104 : cluster [DBG] 2021-03-24T21:58:40.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:40 smithi142 conmon[20125]: osdmap e349: 8 total, 7 up, 6 in 2021-03-24T21:58:40.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:40 smithi142 conmon[20125]: cluster 2021-03-24T21:58:40.373 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:40 smithi142 conmon[20125]: 2021-03-24T21:58: 2021-03-24T21:58:40.373 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:40 smithi142 conmon[20125]: 40.189366+0000 mon.a (mon.0) 1105 2021-03-24T21:58:40.373 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:40 smithi142 conmon[20125]: : cluster [DBG] osdmap e350: 8 total, 7 up, 6 in 2021-03-24T21:58:40.403 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:40.464 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:40.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:40.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:40.572 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:40.602 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: cluster 2021-03-24T21:58:40.603 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 2021-03-24T21:58:40.603 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 2021 2021-03-24T21:58:40.603 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: - 2021-03-24T21:58:40.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 03 2021-03-24T21:58:40.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: - 2021-03-24T21:58:40.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 24 2021-03-24T21:58:40.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: T 2021-03-24T21:58:40.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 21 2021-03-24T21:58:40.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: : 2021-03-24T21:58:40.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 58 2021-03-24T21:58:40.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: : 2021-03-24T21:58:40.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 38 2021-03-24T21:58:40.606 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: . 2021-03-24T21:58:40.606 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 801973 2021-03-24T21:58:40.606 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: +0000 2021-03-24T21:58:40.606 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 2021-03-24T21:58:40.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: mgr.y 2021-03-24T21:58:40.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: ( 2021-03-24T21:58:40.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: mgr 2021-03-24T21:58:40.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: . 2021-03-24T21:58:40.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 14144 2021-03-24T21:58:40.608 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: ) 2021-03-24T21:58:40.608 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 443 2021-03-24T21:58:40.608 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: : 2021-03-24T21:58:40.608 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: cluster 2021-03-24T21:58:40.609 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 2021-03-24T21:58:40.609 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: [DBG] 2021-03-24T21:58:40.609 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 2021-03-24T21:58:40.610 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: pgmap v685: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 184 MiB data, 569 MiB used, 536 GiB / 536 GiB avail; 255 B/s rd, 381 KiB/s wr, 1 op/s; 219/1191 objects degraded (18.388%) 2021-03-24T21:58:40.610 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 2021-03-24T21:58:40.610 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: cluster 2021-03-24T21:58:40.610 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 2021-03-24T21:58:40.611 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 2021 2021-03-24T21:58:40.611 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: - 2021-03-24T21:58:40.611 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 03 2021-03-24T21:58:40.611 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: - 2021-03-24T21:58:40.611 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 24 2021-03-24T21:58:40.612 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: T 2021-03-24T21:58:40.612 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 21 2021-03-24T21:58:40.612 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: : 2021-03-24T21:58:40.612 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 58 2021-03-24T21:58:40.613 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: : 2021-03-24T21:58:40.613 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 39 2021-03-24T21:58:40.613 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: . 2021-03-24T21:58:40.613 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 228329 2021-03-24T21:58:40.613 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: +0000 2021-03-24T21:58:40.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 2021-03-24T21:58:40.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: osd.7 2021-03-24T21:58:40.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: ( 2021-03-24T21:58:40.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: osd 2021-03-24T21:58:40.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: . 2021-03-24T21:58:40.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 7 2021-03-24T21:58:40.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: ) 2021-03-24T21:58:40.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 79 2021-03-24T21:58:40.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: : 2021-03-24T21:58:40.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: cluster 2021-03-24T21:58:40.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 2021-03-24T21:58:40.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: [DBG] 2021-03-24T21:58:40.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 2021-03-24T21:58:40.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: purged_snaps scrub starts 2021-03-24T21:58:40.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 2021-03-24T21:58:40.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: cluster 2021-03-24T21:58:40.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 2021-03-24T21:58:40.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 2021 2021-03-24T21:58:40.618 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: - 2021-03-24T21:58:40.618 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 03 2021-03-24T21:58:40.618 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: - 2021-03-24T21:58:40.618 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 24 2021-03-24T21:58:40.619 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: T 2021-03-24T21:58:40.619 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 21 2021-03-24T21:58:40.619 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: : 2021-03-24T21:58:40.620 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 58 2021-03-24T21:58:40.620 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: : 2021-03-24T21:58:40.620 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 39 2021-03-24T21:58:40.620 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: . 2021-03-24T21:58:40.620 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 229014 2021-03-24T21:58:40.621 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: +0000 2021-03-24T21:58:40.621 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 2021-03-24T21:58:40.621 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: osd.7 2021-03-24T21:58:40.621 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: ( 2021-03-24T21:58:40.622 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: osd 2021-03-24T21:58:40.622 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: . 2021-03-24T21:58:40.622 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 7 2021-03-24T21:58:40.622 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: ) 2021-03-24T21:58:40.622 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 80 2021-03-24T21:58:40.623 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: : 2021-03-24T21:58:40.624 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: cluster 2021-03-24T21:58:40.624 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 2021-03-24T21:58:40.624 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: [DBG] 2021-03-24T21:58:40.624 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 2021-03-24T21:58:40.625 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: purged_snaps scrub ok 2021-03-24T21:58:40.625 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 2021-03-24T21:58:40.625 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: cluster 2021-03-24T21:58:40.625 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 2021-03-24T21:58:40.625 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 2021 2021-03-24T21:58:40.626 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: - 2021-03-24T21:58:40.626 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 03 2021-03-24T21:58:40.626 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: - 2021-03-24T21:58:40.626 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 24 2021-03-24T21:58:40.627 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: T 2021-03-24T21:58:40.627 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 21 2021-03-24T21:58:40.627 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: : 2021-03-24T21:58:40.627 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 58 2021-03-24T21:58:40.627 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: : 2021-03-24T21:58:40.628 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 39 2021-03-24T21:58:40.628 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: . 2021-03-24T21:58:40.628 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 247876 2021-03-24T21:58:40.628 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: +0000 2021-03-24T21:58:40.629 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 2021-03-24T21:58:40.629 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: mon.a 2021-03-24T21:58:40.629 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: ( 2021-03-24T21:58:40.629 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: mon 2021-03-24T21:58:40.629 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: . 2021-03-24T21:58:40.630 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 0 2021-03-24T21:58:40.630 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: ) 2021-03-24T21:58:40.631 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 1103 2021-03-24T21:58:40.631 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: : 2021-03-24T21:58:40.631 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: cluster 2021-03-24T21:58:40.631 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 2021-03-24T21:58:40.631 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: [WRN] 2021-03-24T21:58:40.632 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 2021-03-24T21:58:40.632 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: Health check update: Degraded data redundancy: 219/1191 objects degraded (18.388%), 40 pgs degraded (PG_DEGRADED) 2021-03-24T21:58:40.632 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 2021-03-24T21:58:40.632 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: cluster 2021-03-24T21:58:40.632 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 2021-03-24T21:58:40.633 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 2021 2021-03-24T21:58:40.633 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: - 2021-03-24T21:58:40.633 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 03 2021-03-24T21:58:40.633 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: - 2021-03-24T21:58:40.634 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 24 2021-03-24T21:58:40.634 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: T 2021-03-24T21:58:40.634 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 21 2021-03-24T21:58:40.634 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: : 2021-03-24T21:58:40.634 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 58 2021-03-24T21:58:40.635 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: : 2021-03-24T21:58:40.635 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 39 2021-03-24T21:58:40.635 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: . 2021-03-24T21:58:40.635 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 265836 2021-03-24T21:58:40.635 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: +0000 2021-03-24T21:58:40.636 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 2021-03-24T21:58:40.636 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: mon.a 2021-03-24T21:58:40.636 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: ( 2021-03-24T21:58:40.636 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: mon 2021-03-24T21:58:40.637 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: . 2021-03-24T21:58:40.637 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 0 2021-03-24T21:58:40.637 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: ) 2021-03-24T21:58:40.637 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 1104 2021-03-24T21:58:40.637 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: : 2021-03-24T21:58:40.638 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: cluster 2021-03-24T21:58:40.638 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 2021-03-24T21:58:40.638 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: [DBG] 2021-03-24T21:58:40.638 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 2021-03-24T21:58:40.639 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: osdmap e349: 8 total, 7 up, 6 in 2021-03-24T21:58:40.639 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 2021-03-24T21:58:40.639 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: cluster 2021-03-24T21:58:40.639 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 2021-03-24T21:58:40.639 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 2021 2021-03-24T21:58:40.640 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: - 2021-03-24T21:58:40.640 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 03 2021-03-24T21:58:40.640 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: - 2021-03-24T21:58:40.641 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 24 2021-03-24T21:58:40.641 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: T 2021-03-24T21:58:40.641 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 21 2021-03-24T21:58:40.641 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: : 2021-03-24T21:58:40.641 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 58 2021-03-24T21:58:40.642 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: : 2021-03-24T21:58:40.642 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 40 2021-03-24T21:58:40.642 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: . 2021-03-24T21:58:40.642 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 189366 2021-03-24T21:58:40.643 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: +0000 2021-03-24T21:58:40.643 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 2021-03-24T21:58:40.643 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: mon.a 2021-03-24T21:58:40.643 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: ( 2021-03-24T21:58:40.644 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: mon 2021-03-24T21:58:40.644 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: . 2021-03-24T21:58:40.644 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 0 2021-03-24T21:58:40.644 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: ) 2021-03-24T21:58:40.644 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 1105 2021-03-24T21:58:40.645 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: : 2021-03-24T21:58:40.645 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: cluster 2021-03-24T21:58:40.645 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 2021-03-24T21:58:40.646 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: [DBG] 2021-03-24T21:58:40.646 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 2021-03-24T21:58:40.646 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: osdmap e350: 8 total, 7 up, 6 in 2021-03-24T21:58:40.646 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:40 smithi125 conmon[24927]: 2021-03-24T21:58:40.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:40.648 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:40.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:58:40.656 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell -n osd.5 --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph-objectstore-tool --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --force --op remove --pgid 3.3 2021-03-24T21:58:40.749 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:40.754 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:58:40.819 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:40.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:40.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:58:40.858 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/osd.5/config 2021-03-24T21:58:40.923 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:40.979 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:40.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:40.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:41.084 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:41.145 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:41.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:41.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:41.250 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:41.268 INFO:tasks.rados.rados.0.smithi125.stdout:1239: done (3 left) 2021-03-24T21:58:41.268 INFO:tasks.rados.rados.0.smithi125.stdout:1240: done (2 left) 2021-03-24T21:58:41.268 INFO:tasks.rados.rados.0.smithi125.stdout:1241: done (1 left) 2021-03-24T21:58:41.268 INFO:tasks.rados.rados.0.smithi125.stdout:1242: done (0 left) 2021-03-24T21:58:41.269 INFO:tasks.rados.rados.0.smithi125.stdout:1243: snap_create 2021-03-24T21:58:41.304 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:41.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:41.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:58:41.409 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:41.510 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:41.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:41.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:58:41.615 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:41.673 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:41.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:41.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:58:41.780 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:41.834 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:41.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:41.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:41.939 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:42.001 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:42.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:42.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:42.108 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:42.166 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:42.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:42.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:42.272 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:42.273 INFO:tasks.rados.rados.0.smithi125.stdout:1243: done (0 left) 2021-03-24T21:58:42.273 INFO:tasks.rados.rados.0.smithi125.stdout:1244: copy_from oid 40 from oid 35 current snap is 128 2021-03-24T21:58:42.274 INFO:tasks.rados.rados.0.smithi125.stdout:1245: rollback oid 13 current snap is 128 2021-03-24T21:58:42.274 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 13 to 121 2021-03-24T21:58:42.274 INFO:tasks.rados.rados.0.smithi125.stdout:1246: copy_from oid 14 from oid 8 current snap is 128 2021-03-24T21:58:42.274 INFO:tasks.rados.rados.0.smithi125.stdout:1247: snap_remove snap 113 2021-03-24T21:58:42.277 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:42 smithi142 conmon[20125]: cluster 2021-03-24T21:58:40.802774+0000 mgr.y (mgr.14144 2021-03-24T21:58:42.278 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:42 smithi142 conmon[20125]: ) 444 : cluster [DBG] pgmap v688: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 202 MiB data, 593 MiB used, 536 GiB / 536 GiB avail; 24 MiB/s rd, 2.5 MiB/s wr, 28 op/s; 226/1218 objects degraded (18.555%) 2021-03-24T21:58:42.278 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:42 smithi142 conmon[20125]: cluster 2021-03-24T21:58:41.265507+0000 mon.a (mon.0) 1106 : cluster [DBG] osdmap e351: 8 total, 7 up, 6 in 2021-03-24T21:58:42.417 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:42.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:42.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:42.523 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:42.531 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: cluster 2021-03-24T21:58:42.532 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: 2021-03-24T21:58:42.532 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: 2021 2021-03-24T21:58:42.532 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: - 2021-03-24T21:58:42.532 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: 03 2021-03-24T21:58:42.532 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: - 2021-03-24T21:58:42.533 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: 24 2021-03-24T21:58:42.533 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: T 2021-03-24T21:58:42.533 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: 21 2021-03-24T21:58:42.533 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: : 2021-03-24T21:58:42.534 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: 58 2021-03-24T21:58:42.534 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: : 2021-03-24T21:58:42.534 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: 40 2021-03-24T21:58:42.534 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: . 2021-03-24T21:58:42.535 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: 802774 2021-03-24T21:58:42.535 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: +0000 2021-03-24T21:58:42.535 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: 2021-03-24T21:58:42.535 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: mgr.y 2021-03-24T21:58:42.535 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: ( 2021-03-24T21:58:42.536 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: mgr 2021-03-24T21:58:42.536 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: . 2021-03-24T21:58:42.536 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: 14144 2021-03-24T21:58:42.536 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: ) 2021-03-24T21:58:42.537 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: 444 2021-03-24T21:58:42.537 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: : 2021-03-24T21:58:42.537 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: cluster 2021-03-24T21:58:42.537 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: 2021-03-24T21:58:42.537 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: [DBG] 2021-03-24T21:58:42.538 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: 2021-03-24T21:58:42.538 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: pgmap v688: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 202 MiB data, 593 MiB used, 536 GiB / 536 GiB avail; 24 MiB/s rd, 2.5 MiB/s wr, 28 op/s; 226/1218 objects degraded (18.555%) 2021-03-24T21:58:42.538 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: 2021-03-24T21:58:42.538 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: cluster 2021-03-24T21:58:42.539 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: 2021-03-24T21:58:42.539 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: 2021 2021-03-24T21:58:42.539 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: - 2021-03-24T21:58:42.539 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: 03 2021-03-24T21:58:42.539 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: - 2021-03-24T21:58:42.540 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: 24 2021-03-24T21:58:42.540 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: T 2021-03-24T21:58:42.540 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: 21 2021-03-24T21:58:42.540 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: : 2021-03-24T21:58:42.541 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: 58 2021-03-24T21:58:42.541 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: : 2021-03-24T21:58:42.541 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: 41 2021-03-24T21:58:42.541 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: . 2021-03-24T21:58:42.541 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: 265507 2021-03-24T21:58:42.542 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: +0000 2021-03-24T21:58:42.542 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: 2021-03-24T21:58:42.542 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: mon.a 2021-03-24T21:58:42.542 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: ( 2021-03-24T21:58:42.542 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: mon 2021-03-24T21:58:42.543 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: . 2021-03-24T21:58:42.543 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: 0 2021-03-24T21:58:42.543 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: ) 2021-03-24T21:58:42.543 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: 1106 2021-03-24T21:58:42.544 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: : 2021-03-24T21:58:42.544 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: cluster 2021-03-24T21:58:42.544 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: 2021-03-24T21:58:42.544 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: [DBG] 2021-03-24T21:58:42.544 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: 2021-03-24T21:58:42.545 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: osdmap e351: 8 total, 7 up, 6 in 2021-03-24T21:58:42.545 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[19936]: 2021-03-24T21:58:42.545 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: cluster 2021-03-24T21:58:42.546 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: 2021-03-24T21:58:42.546 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: 2021 2021-03-24T21:58:42.546 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: - 2021-03-24T21:58:42.546 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: 03 2021-03-24T21:58:42.547 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: - 2021-03-24T21:58:42.547 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: 24 2021-03-24T21:58:42.547 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: T 2021-03-24T21:58:42.547 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: 21 2021-03-24T21:58:42.547 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: : 2021-03-24T21:58:42.548 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: 58 2021-03-24T21:58:42.548 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: : 2021-03-24T21:58:42.548 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: 40 2021-03-24T21:58:42.548 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: . 2021-03-24T21:58:42.548 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: 802774 2021-03-24T21:58:42.549 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: +0000 2021-03-24T21:58:42.549 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: 2021-03-24T21:58:42.549 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: mgr.y 2021-03-24T21:58:42.549 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: ( 2021-03-24T21:58:42.550 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: mgr 2021-03-24T21:58:42.550 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: . 2021-03-24T21:58:42.550 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: 14144 2021-03-24T21:58:42.550 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: ) 2021-03-24T21:58:42.550 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: 444 2021-03-24T21:58:42.551 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: : 2021-03-24T21:58:42.551 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: cluster 2021-03-24T21:58:42.551 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: 2021-03-24T21:58:42.551 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: [DBG] 2021-03-24T21:58:42.552 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: 2021-03-24T21:58:42.552 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: pgmap v688: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 202 MiB data, 593 MiB used, 536 GiB / 536 GiB avail; 24 MiB/s rd, 2.5 MiB/s wr, 28 op/s; 226/1218 objects degraded (18.555%) 2021-03-24T21:58:42.552 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: 2021-03-24T21:58:42.552 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: cluster 2021-03-24T21:58:42.552 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: 2021-03-24T21:58:42.553 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: 2021 2021-03-24T21:58:42.553 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: - 2021-03-24T21:58:42.553 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: 03 2021-03-24T21:58:42.553 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: - 2021-03-24T21:58:42.553 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: 24 2021-03-24T21:58:42.554 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: T 2021-03-24T21:58:42.554 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: 21 2021-03-24T21:58:42.554 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: : 2021-03-24T21:58:42.554 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: 58 2021-03-24T21:58:42.554 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: : 2021-03-24T21:58:42.555 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: 41 2021-03-24T21:58:42.555 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: . 2021-03-24T21:58:42.555 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: 265507 2021-03-24T21:58:42.555 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: +0000 2021-03-24T21:58:42.555 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: 2021-03-24T21:58:42.556 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: mon.a 2021-03-24T21:58:42.556 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: ( 2021-03-24T21:58:42.556 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: mon 2021-03-24T21:58:42.556 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: . 2021-03-24T21:58:42.557 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: 0 2021-03-24T21:58:42.557 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: ) 2021-03-24T21:58:42.557 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: 1106 2021-03-24T21:58:42.557 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: : 2021-03-24T21:58:42.557 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: cluster 2021-03-24T21:58:42.557 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: 2021-03-24T21:58:42.558 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: [DBG] 2021-03-24T21:58:42.558 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: 2021-03-24T21:58:42.558 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: osdmap e351: 8 total, 7 up, 6 in 2021-03-24T21:58:42.558 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:42 smithi125 conmon[24927]: 2021-03-24T21:58:42.565 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:42.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:42.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:58:42.671 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:42.678 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-24T21:58:42.678 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T21:58:42.727 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:42.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:42.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:42.833 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:42.892 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:42.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:42.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:58:42.897 INFO:teuthology.orchestra.run.smithi142.stdout: marking collection for removal 2021-03-24T21:58:42.897 INFO:teuthology.orchestra.run.smithi142.stdout:setting '_remove' omap key 2021-03-24T21:58:42.897 INFO:teuthology.orchestra.run.smithi142.stdout:finish_remove_pgs 3.3_head removing 3.3 2021-03-24T21:58:42.897 INFO:teuthology.orchestra.run.smithi142.stdout:Remove successful 2021-03-24T21:58:42.997 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:43.058 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:43.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:43.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:58:43.164 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:43.220 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:43.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:43.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:58:43.277 INFO:tasks.rados.rados.0.smithi125.stdout:1247: done (3 left) 2021-03-24T21:58:43.277 INFO:tasks.rados.rados.0.smithi125.stdout:1248: copy_from oid 27 from oid 45 current snap is 128 2021-03-24T21:58:43.277 INFO:tasks.rados.rados.0.smithi125.stdout:1244: finishing copy_from to smithi12556470-40 2021-03-24T21:58:43.277 INFO:tasks.rados.rados.0.smithi125.stdout:1244: got expected ENOENT (src dne) 2021-03-24T21:58:43.277 INFO:tasks.rados.rados.0.smithi125.stdout:1244: finishing copy_from racing read to smithi12556470-40 2021-03-24T21:58:43.278 INFO:tasks.rados.rados.0.smithi125.stdout:1245: finishing rollback tid 0 to smithi12556470-13 2021-03-24T21:58:43.278 INFO:tasks.rados.rados.0.smithi125.stdout:1245: finishing rollback tid 1 to smithi12556470-13 2021-03-24T21:58:43.278 INFO:tasks.rados.rados.0.smithi125.stdout:1246: finishing copy_from to smithi12556470-14 2021-03-24T21:58:43.278 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 14 v 75 (ObjNum 414 snap 108 seq_num 414) dirty exists 2021-03-24T21:58:43.279 INFO:tasks.rados.rados.0.smithi125.stdout:1245: finishing rollback tid 2 to smithi12556470-13 2021-03-24T21:58:43.279 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 13 v 277 (ObjNum 410 snap 108 seq_num 410) dirty exists 2021-03-24T21:58:43.279 INFO:tasks.rados.rados.0.smithi125.stdout:1246: finishing copy_from racing read to smithi12556470-14 2021-03-24T21:58:43.279 INFO:tasks.rados.rados.0.smithi125.stdout:1244: done (3 left) 2021-03-24T21:58:43.279 INFO:tasks.rados.rados.0.smithi125.stdout:1245: done (2 left) 2021-03-24T21:58:43.280 INFO:tasks.rados.rados.0.smithi125.stdout:1246: done (1 left) 2021-03-24T21:58:43.280 INFO:tasks.rados.rados.0.smithi125.stdout:1249: copy_from oid 18 from oid 47 current snap is 128 2021-03-24T21:58:43.280 INFO:tasks.rados.rados.0.smithi125.stdout:1250: read oid 50 snap -1 2021-03-24T21:58:43.280 INFO:tasks.rados.rados.0.smithi125.stdout:1250: expect (ObjNum 435 snap 115 seq_num 435) 2021-03-24T21:58:43.281 INFO:tasks.rados.rados.0.smithi125.stdout:1251: read oid 48 snap 116 2021-03-24T21:58:43.281 INFO:tasks.rados.rados.0.smithi125.stdout:1251: expect (ObjNum 423 snap 112 seq_num 423) 2021-03-24T21:58:43.281 INFO:tasks.rados.rados.0.smithi125.stdout:1252: read oid 25 snap 117 2021-03-24T21:58:43.281 INFO:tasks.rados.rados.0.smithi125.stdout:1252: expect deleted 2021-03-24T21:58:43.281 INFO:tasks.rados.rados.0.smithi125.stdout:1253: snap_remove snap 118 2021-03-24T21:58:43.327 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:43.379 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:43.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:43.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:58:43.485 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:43.493 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[24927]: cluster 2021-03-24T21:58:43.494 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[24927]: 2021-03-24T21:58:43.494 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[24927]: 2021 2021-03-24T21:58:43.494 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[24927]: - 2021-03-24T21:58:43.494 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[24927]: 03 2021-03-24T21:58:43.495 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[24927]: - 2021-03-24T21:58:43.495 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[24927]: 24 2021-03-24T21:58:43.495 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[24927]: T 2021-03-24T21:58:43.495 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[24927]: 21 2021-03-24T21:58:43.495 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[24927]: : 2021-03-24T21:58:43.496 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[24927]: 58 2021-03-24T21:58:43.496 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[24927]: : 2021-03-24T21:58:43.496 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[24927]: 42 2021-03-24T21:58:43.496 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[24927]: . 2021-03-24T21:58:43.496 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[24927]: 268710 2021-03-24T21:58:43.497 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[24927]: +0000 2021-03-24T21:58:43.497 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[24927]: 2021-03-24T21:58:43.497 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[24927]: mon.a 2021-03-24T21:58:43.497 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[24927]: ( 2021-03-24T21:58:43.498 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[24927]: mon 2021-03-24T21:58:43.498 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[24927]: . 2021-03-24T21:58:43.498 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[24927]: 0 2021-03-24T21:58:43.498 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[24927]: ) 2021-03-24T21:58:43.498 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[24927]: 1107 2021-03-24T21:58:43.499 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[24927]: : 2021-03-24T21:58:43.499 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[24927]: cluster 2021-03-24T21:58:43.499 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[24927]: 2021-03-24T21:58:43.499 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[24927]: [DBG] 2021-03-24T21:58:43.500 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[24927]: 2021-03-24T21:58:43.500 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[24927]: osdmap e352: 8 total, 7 up, 6 in 2021-03-24T21:58:43.500 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[24927]: 2021-03-24T21:58:43.501 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[19936]: cluster 2021-03-24T21:58:43.501 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[19936]: 2021-03-24T21:58:43.501 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[19936]: 2021 2021-03-24T21:58:43.501 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[19936]: - 2021-03-24T21:58:43.502 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[19936]: 03 2021-03-24T21:58:43.502 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[19936]: - 2021-03-24T21:58:43.502 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[19936]: 24 2021-03-24T21:58:43.502 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[19936]: T 2021-03-24T21:58:43.503 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[19936]: 21 2021-03-24T21:58:43.503 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[19936]: : 2021-03-24T21:58:43.503 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[19936]: 58 2021-03-24T21:58:43.503 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[19936]: : 2021-03-24T21:58:43.503 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[19936]: 42 2021-03-24T21:58:43.504 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[19936]: . 2021-03-24T21:58:43.504 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[19936]: 268710 2021-03-24T21:58:43.504 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[19936]: +0000 2021-03-24T21:58:43.504 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[19936]: 2021-03-24T21:58:43.504 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[19936]: mon.a 2021-03-24T21:58:43.505 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[19936]: ( 2021-03-24T21:58:43.505 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[19936]: mon 2021-03-24T21:58:43.505 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[19936]: . 2021-03-24T21:58:43.505 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[19936]: 0 2021-03-24T21:58:43.505 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[19936]: ) 2021-03-24T21:58:43.506 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[19936]: 1107 2021-03-24T21:58:43.506 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[19936]: : 2021-03-24T21:58:43.506 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[19936]: cluster 2021-03-24T21:58:43.506 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[19936]: 2021-03-24T21:58:43.507 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[19936]: [DBG] 2021-03-24T21:58:43.507 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[19936]: 2021-03-24T21:58:43.507 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[19936]: osdmap e352: 8 total, 7 up, 6 in 2021-03-24T21:58:43.507 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:43 smithi125 conmon[19936]: 2021-03-24T21:58:43.541 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:43.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:43.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:43.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:43 smithi142 conmon[20125]: cluster 2021-03-24T21:58:43.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:43 smithi142 conmon[20125]: 2021-03-24T21:58:43.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:43 smithi142 conmon[20125]: 2021 2021-03-24T21:58:43.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:43 smithi142 conmon[20125]: - 2021-03-24T21:58:43.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:43 smithi142 conmon[20125]: 03 2021-03-24T21:58:43.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:43 smithi142 conmon[20125]: - 2021-03-24T21:58:43.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:43 smithi142 conmon[20125]: 24 2021-03-24T21:58:43.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:43 smithi142 conmon[20125]: T 2021-03-24T21:58:43.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:43 smithi142 conmon[20125]: 21 2021-03-24T21:58:43.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:43 smithi142 conmon[20125]: : 2021-03-24T21:58:43.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:43 smithi142 conmon[20125]: 58 2021-03-24T21:58:43.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:43 smithi142 conmon[20125]: : 2021-03-24T21:58:43.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:43 smithi142 conmon[20125]: 42 2021-03-24T21:58:43.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:43 smithi142 conmon[20125]: . 2021-03-24T21:58:43.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:43 smithi142 conmon[20125]: 268710 2021-03-24T21:58:43.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:43 smithi142 conmon[20125]: +0000 2021-03-24T21:58:43.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:43 smithi142 conmon[20125]: 2021-03-24T21:58:43.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:43 smithi142 conmon[20125]: mon.a 2021-03-24T21:58:43.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:43 smithi142 conmon[20125]: ( 2021-03-24T21:58:43.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:43 smithi142 conmon[20125]: mon 2021-03-24T21:58:43.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:43 smithi142 conmon[20125]: . 2021-03-24T21:58:43.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:43 smithi142 conmon[20125]: 0 2021-03-24T21:58:43.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:43 smithi142 conmon[20125]: ) 2021-03-24T21:58:43.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:43 smithi142 conmon[20125]: 1107 2021-03-24T21:58:43.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:43 smithi142 conmon[20125]: : 2021-03-24T21:58:43.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:43 smithi142 conmon[20125]: cluster 2021-03-24T21:58:43.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:43 smithi142 conmon[20125]: 2021-03-24T21:58:43.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:43 smithi142 conmon[20125]: [DBG] 2021-03-24T21:58:43.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:43 smithi142 conmon[20125]: 2021-03-24T21:58:43.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:43 smithi142 conmon[20125]: osdmap e352: 8 total, 7 up, 6 in 2021-03-24T21:58:43.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:43 smithi142 conmon[20125]: 2021-03-24T21:58:43.648 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:43.695 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:43.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:43.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:58:43.799 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:43.858 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:43.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:43.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:58:43.964 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:44.018 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:44.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:44.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:44.124 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:44.186 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:44.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:44.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:58:44.278 INFO:tasks.rados.rados.0.smithi125.stdout:1253: done (5 left) 2021-03-24T21:58:44.278 INFO:tasks.rados.rados.0.smithi125.stdout:1254: write (excl) oid 14 current snap is 128 2021-03-24T21:58:44.278 INFO:tasks.rados.rados.0.smithi125.stdout:1254: seq_num 455 ranges {447185=703122,1642601=765814,2928991=447707} 2021-03-24T21:58:44.287 INFO:tasks.rados.rados.0.smithi125.stdout:1254: writing smithi12556470-14 from 447185 to 1150307 tid 1 2021-03-24T21:58:44.292 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:44.306 INFO:tasks.rados.rados.0.smithi125.stdout:1254: writing smithi12556470-14 from 1642601 to 2408415 tid 2 2021-03-24T21:58:44.317 INFO:tasks.rados.rados.0.smithi125.stdout:1254: writing smithi12556470-14 from 2928991 to 3376698 tid 3 2021-03-24T21:58:44.318 INFO:tasks.rados.rados.0.smithi125.stdout:1255: delete oid 36 current snap is 128 2021-03-24T21:58:44.318 INFO:tasks.rados.rados.0.smithi125.stdout:1251: expect (ObjNum 423 snap 112 seq_num 423) 2021-03-24T21:58:44.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:44 smithi142 conmon[20125]: cluster 2021-03-24T21:58:42.803391+0000 mgr.y ( 2021-03-24T21:58:44.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:44 smithi142 conmon[20125]: mgr.14144) 445 : cluster [DBG] pgmap v691: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 202 MiB data, 597 MiB used, 536 GiB / 536 GiB avail; 24 MiB/s rd, 2.5 MiB/s wr, 28 op/s; 226/1218 objects degraded (18.555%) 2021-03-24T21:58:44.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:44 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:58:44.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:44 smithi142 conmon[20125]: -24T21:58:43.271251 2021-03-24T21:58:44.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:44 smithi142 conmon[20125]: +0000 mon.a (mon. 2021-03-24T21:58:44.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:44 smithi142 conmon[20125]: 0) 1108 : cluster 2021-03-24T21:58:44.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:44 smithi142 conmon[20125]: [DBG] osdmap e353: 8 total, 7 up, 6 in 2021-03-24T21:58:44.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:44 smithi142 conmon[20125]: cluster 2021-03-24T21:58:44.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:44 smithi142 conmon[20125]: 2021-03-24 2021-03-24T21:58:44.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:44 smithi142 conmon[20125]: T21:58:44 2021-03-24T21:58:44.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:44 smithi142 conmon[20125]: .273813+0000 mon.a 2021-03-24T21:58:44.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:44 smithi142 conmon[20125]: (mon.0) 2021-03-24T21:58:44.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:44 smithi142 conmon[20125]: 1109 : cluster [DBG] 2021-03-24T21:58:44.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:44 smithi142 conmon[20125]: osdmap e354: 8 total, 7 up, 6 in 2021-03-24T21:58:44.437 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:44.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:44.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:58:44.472 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell -n osd.5 --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.3.3.5 2021-03-24T21:58:44.514 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.1 dump_ops_in_flight 2021-03-24T21:58:44.543 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:44.552 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:44 smithi125 conmon[19936]: cluster 2021-03-24T21:58:44.553 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:44 smithi125 conmon[19936]: 2021-03-24T21:58:44.553 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:44 smithi125 conmon[19936]: 2021 2021-03-24T21:58:44.553 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:44 smithi125 conmon[19936]: - 2021-03-24T21:58:44.553 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:44 smithi125 conmon[19936]: 03 2021-03-24T21:58:44.554 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:44 smithi125 conmon[19936]: - 2021-03-24T21:58:44.554 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:44 smithi125 conmon[19936]: 24 2021-03-24T21:58:44.554 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:44 smithi125 conmon[19936]: T21 2021-03-24T21:58:44.555 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:44 smithi125 conmon[19936]: :58:42.803391+0000 mgr.y (mgr.14144) 445 : cluster [DBG] pgmap v691: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 202 MiB data, 597 MiB used, 536 GiB / 536 GiB avail; 24 MiB/s rd, 2.5 MiB/s wr, 28 op/s; 226/1218 objects degraded (18.555%) 2021-03-24T21:58:44.555 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:44 smithi125 conmon[19936]: cluster 2021-03-24T21:58:44.555 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:44 smithi125 conmon[19936]: 2021-03-24T21:58:44.555 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:44 smithi125 conmon[19936]: 2021-03-24T21:58: 2021-03-24T21:58:44.555 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:44 smithi125 conmon[19936]: 43.271251+0000 mon.a (mon.0) 1108 : cluster [DBG] osdmap e353: 8 total, 7 up, 6 in 2021-03-24T21:58:44.556 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:44 smithi125 conmon[19936]: cluster 2021-03-24T21:58 2021-03-24T21:58:44.556 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:44 smithi125 conmon[19936]: :44.273813+0000 mon.a (mon.0) 1109 : cluster [DBG] 2021-03-24T21:58:44.556 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:44 smithi125 conmon[19936]: osdmap e354: 8 total, 7 up, 6 in 2021-03-24T21:58:44.557 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:44 smithi125 conmon[24927]: cluster 2021-03-24T21:58:42.803391+0000 mgr.y ( 2021-03-24T21:58:44.557 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:44 smithi125 conmon[24927]: mgr.14144) 445 : cluster [DBG] pgmap v691: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 202 MiB data, 597 MiB used, 536 GiB / 536 GiB avail; 24 MiB/s rd, 2.5 MiB/s wr, 28 op/s; 226/1218 objects degraded (18.555%) 2021-03-24T21:58:44.557 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:44 smithi125 conmon[24927]: cluster 2021-03-24T21:58:43.271251+0000 mon.a (mon.0) 2021-03-24T21:58:44.557 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:44 smithi125 conmon[24927]: 1108 : cluster [DBG] osdmap e353: 8 total, 7 up, 6 in 2021-03-24T21:58:44.557 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:44 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:58:44.558 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:44 smithi125 conmon[24927]: 21:58:44.273813+0000 mon.a (mon.0) 1109 : cluster [DBG] osdmap e354: 8 total, 7 up, 6 in 2021-03-24T21:58:44.622 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:44.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:44.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:58:44.636 INFO:tasks.rados.rados.0.smithi125.stdout:1249: finishing copy_from to smithi12556470-18 2021-03-24T21:58:44.637 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 18 v 103 (ObjNum 384 snap 99 seq_num 384) dirty exists 2021-03-24T21:58:44.637 INFO:tasks.rados.rados.0.smithi125.stdout:1249: finishing copy_from racing read to smithi12556470-18 2021-03-24T21:58:44.637 INFO:tasks.rados.rados.0.smithi125.stdout:1250: expect (ObjNum 435 snap 115 seq_num 435) 2021-03-24T21:58:44.688 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/osd.5/config 2021-03-24T21:58:44.723 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T21:58:44.730 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:44.746 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T21:58:44.772 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T21:58:44.788 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:44.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:44.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:44.796 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T21:58:44.818 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T21:58:44.838 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T21:58:44.860 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T21:58:44.861 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-24T21:58:44.861 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-24T21:58:44.862 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-24T21:58:44.862 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-24T21:58:44.862 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-24T21:58:44.862 INFO:teuthology.orchestra.run.smithi125.stderr:Error ENXIO: problem getting command descriptions from osd.5 2021-03-24T21:58:44.863 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-24T21:58:44.863 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-24T21:58:44.893 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:44.904 INFO:tasks.rados.rados.0.smithi125.stdout:1248: finishing copy_from to smithi12556470-27 2021-03-24T21:58:44.905 INFO:tasks.rados.rados.0.smithi125.stdout:1248: got expected ENOENT (src dne) 2021-03-24T21:58:44.905 INFO:tasks.rados.rados.0.smithi125.stdout:1248: finishing copy_from racing read to smithi12556470-27 2021-03-24T21:58:44.905 INFO:tasks.rados.rados.0.smithi125.stdout:1248: done (6 left) 2021-03-24T21:58:44.905 INFO:tasks.rados.rados.0.smithi125.stdout:1249: done (5 left) 2021-03-24T21:58:44.906 INFO:tasks.rados.rados.0.smithi125.stdout:1250: done (4 left) 2021-03-24T21:58:44.906 INFO:tasks.rados.rados.0.smithi125.stdout:1251: done (3 left) 2021-03-24T21:58:44.906 INFO:tasks.rados.rados.0.smithi125.stdout:1252: done (2 left) 2021-03-24T21:58:44.906 INFO:tasks.rados.rados.0.smithi125.stdout:1255: done (1 left) 2021-03-24T21:58:44.907 INFO:tasks.rados.rados.0.smithi125.stdout:1256: copy_from oid 11 from oid 22 current snap is 128 2021-03-24T21:58:44.907 INFO:tasks.rados.rados.0.smithi125.stdout:1254: finishing write tid 1 to smithi12556470-14 2021-03-24T21:58:44.907 INFO:tasks.rados.rados.0.smithi125.stdout:1254: finishing write tid 2 to smithi12556470-14 2021-03-24T21:58:44.907 INFO:tasks.rados.rados.0.smithi125.stdout:1254: finishing write tid 3 to smithi12556470-14 2021-03-24T21:58:44.908 INFO:tasks.rados.rados.0.smithi125.stdout:1254: finishing write tid 4 to smithi12556470-14 2021-03-24T21:58:44.908 INFO:tasks.rados.rados.0.smithi125.stdout:1254: finishing write tid 5 to smithi12556470-14 2021-03-24T21:58:44.908 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 14 v 412 (ObjNum 455 snap 128 seq_num 455) dirty exists 2021-03-24T21:58:44.908 INFO:tasks.rados.rados.0.smithi125.stdout:1254: left oid 14 (ObjNum 455 snap 128 seq_num 455) 2021-03-24T21:58:44.909 INFO:tasks.rados.rados.0.smithi125.stdout:1254: done (1 left) 2021-03-24T21:58:44.909 INFO:tasks.rados.rados.0.smithi125.stdout:1257: snap_create 2021-03-24T21:58:44.921 INFO:tasks.rados.rados.0.smithi125.stdout:1256: finishing copy_from to smithi12556470-11 2021-03-24T21:58:44.922 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 11 v 294 (ObjNum 402 snap 104 seq_num 402) dirty exists 2021-03-24T21:58:44.922 INFO:tasks.rados.rados.0.smithi125.stdout:1256: finishing copy_from racing read to smithi12556470-11 2021-03-24T21:58:44.941 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:44.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:44.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:58:44.966 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:58:45.047 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:45.102 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:45.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:45.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:58:45.208 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:45.259 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:45.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:45.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:58:45.295 INFO:tasks.rados.rados.0.smithi125.stdout:1256: done (1 left) 2021-03-24T21:58:45.296 INFO:tasks.rados.rados.0.smithi125.stdout:1257: done (0 left) 2021-03-24T21:58:45.296 INFO:tasks.rados.rados.0.smithi125.stdout:1258: write (excl) oid 32 current snap is 129 2021-03-24T21:58:45.296 INFO:tasks.rados.rados.0.smithi125.stdout:1258: seq_num 457 ranges {480799=569973,1475918=769483,2700190=586604} 2021-03-24T21:58:45.299 INFO:teuthology.orchestra.run.smithi125.stderr:nodeep-scrub is set 2021-03-24T21:58:45.302 INFO:tasks.rados.rados.0.smithi125.stdout:1258: writing smithi12556470-32 from 480799 to 1050772 tid 1 2021-03-24T21:58:45.310 INFO:tasks.rados.rados.0.smithi125.stdout:1258: writing smithi12556470-32 from 1475918 to 2245401 tid 2 2021-03-24T21:58:45.317 INFO:tasks.rados.rados.0.smithi125.stdout:1258: writing smithi12556470-32 from 2700190 to 3286794 tid 3 2021-03-24T21:58:45.317 INFO:tasks.rados.rados.0.smithi125.stdout:1259: read oid 34 snap -1 2021-03-24T21:58:45.317 INFO:tasks.rados.rados.0.smithi125.stdout:1259: expect (ObjNum 411 snap 108 seq_num 411) 2021-03-24T21:58:45.318 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:45 smithi125 conmon[19936]: audit 2021-03-24T21 2021-03-24T21:58:45.318 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:45 smithi125 conmon[19936]: :58:44.696616+0000 mon.c (mon.2) 56 : audit [DBG] from='client.? v1:172.21.15.125:0/3782665137' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:58:45.318 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:45 smithi125 conmon[19936]: audit 2021-03-24T21:58:45.318 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:45 smithi125 conmon[19936]: 2021-03-24T21:58: 2021-03-24T21:58:45.319 INFO:tasks.rados.rados.0.smithi125.stdout:1260: write (excl) oid 25 current snap is 129 2021-03-24T21:58:45.319 INFO:tasks.rados.rados.0.smithi125.stdout:1260: seq_num 458 ranges {497606=645222,1934400=629494,3243966=453640} 2021-03-24T21:58:45.324 INFO:tasks.rados.rados.0.smithi125.stdout:1260: writing smithi12556470-25 from 497606 to 1142828 tid 1 2021-03-24T21:58:45.333 INFO:tasks.rados.rados.0.smithi125.stdout:1260: writing smithi12556470-25 from 1934400 to 2563894 tid 2 2021-03-24T21:58:45.341 INFO:tasks.rados.rados.0.smithi125.stdout:1260: writing smithi12556470-25 from 3243966 to 3697606 tid 3 2021-03-24T21:58:45.341 INFO:tasks.rados.rados.0.smithi125.stdout:1261: read oid 15 snap -1 2021-03-24T21:58:45.341 INFO:tasks.rados.rados.0.smithi125.stdout:1261: expect (ObjNum 384 snap 99 seq_num 384) 2021-03-24T21:58:45.342 INFO:tasks.rados.rados.0.smithi125.stdout:1262: read oid 26 snap -1 2021-03-24T21:58:45.342 INFO:tasks.rados.rados.0.smithi125.stdout:1262: expect (ObjNum 438 snap 118 seq_num 438) 2021-03-24T21:58:45.342 INFO:tasks.rados.rados.0.smithi125.stdout:1263: delete oid 44 current snap is 129 2021-03-24T21:58:45.342 INFO:tasks.rados.rados.0.smithi125.stdout:1258: finishing write tid 1 to smithi12556470-32 2021-03-24T21:58:45.360 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:45 smithi142 conmon[20125]: audit 2021-03-24T 2021-03-24T21:58:45.360 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:45 smithi142 conmon[20125]: 21:58:44.696616+0000 mon.c (mon.2) 56 : audit [DBG] from='client.? v1:172.21.15.125:0/3782665137' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:58:45.360 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:45 smithi142 conmon[20125]: audit 2021-03- 2021-03-24T21:58:45.360 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:45 smithi142 conmon[20125]: 24T21:58:45.167871+0000 2021-03-24T21:58:45.360 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:45 smithi142 conmon[20125]: mon.a (mon.0) 1110 : audit 2021-03-24T21:58:45.361 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:45 smithi142 conmon[20125]: [INF] from='client.? v1:172.21.15.125:0/885361269' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:58:45.361 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:45 smithi142 conmon[20125]: cluster 2021-03-24T21:58:45.361 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:45 smithi142 conmon[20125]: 2021-03-24T 2021-03-24T21:58:45.361 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:45 smithi142 conmon[20125]: 21:58:45.182888+0000 mon.a ( 2021-03-24T21:58:45.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:45 smithi142 conmon[20125]: mon.0) 1111 : cluster [WRN] Health check update: Degraded data redundancy: 226/1218 objects degraded (18.555%), 40 pgs degraded (PG_DEGRADED) 2021-03-24T21:58:45.362 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:45 smithi142 conmon[20125]: 2021-03-24T21:58:45.362 INFO:tasks.rados.rados.0.smithi125.stdout:1259: expect (ObjNum 411 snap 108 seq_num 411) 2021-03-24T21:58:45.364 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:45.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:45 smithi125 conmon[24927]: audit 2021-03-24T21:58: 2021-03-24T21:58:45.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:45 smithi125 conmon[24927]: 44.696616+0000 mon.c (mon.2) 56 : audit [DBG] from='client.? v1:172.21.15.125:0/3782665137' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:58:45.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:45 smithi125 conmon[24927]: audit 2021-03-24T21 2021-03-24T21:58:45.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:45 smithi125 conmon[24927]: :58:45.167871+0000 mon.a (mon.0) 1110 : audit [INF] from='client.? v1:172.21.15.125:0/885361269' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:58:45.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:45 smithi125 conmon[24927]: cluster 2021-03-24T21:58:45.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:45 smithi125 conmon[24927]: 2021-03-24T21:58:45.182888+0000 mon.a (mon.0) 1111 : cluster [WRN] Health check update: Degraded data redundancy: 226/1218 objects degraded (18.555%), 40 pgs degraded (PG_DEGRADED) 2021-03-24T21:58:45.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:45 smithi125 conmon[19936]: 45.167871+0000 mon.a (mon.0 2021-03-24T21:58:45.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:45 smithi125 conmon[19936]: ) 1110 : audit [INF] from='client.? v1:172.21.15.125:0/885361269' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:58:45.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:45 smithi125 conmon[19936]: cluster 2021-03-24T21:58:45.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:45 smithi125 conmon[19936]: 2021-03-24T21:58: 2021-03-24T21:58:45.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:45 smithi125 conmon[19936]: 45.182888+0000 mon.a (mon. 2021-03-24T21:58:45.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:45 smithi125 conmon[19936]: 0) 1111 : cluster [WRN] 2021-03-24T21:58:45.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:45 smithi125 conmon[19936]: Health check update: Degraded data redundancy: 226/1218 objects degraded (18.555%), 40 pgs degraded (PG_DEGRADED) 2021-03-24T21:58:45.421 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:45.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:45.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:58:45.530 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:45.546 INFO:tasks.rados.rados.0.smithi125.stdout:1261: expect (ObjNum 384 snap 99 seq_num 384) 2021-03-24T21:58:45.582 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:45.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:45.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:58:45.688 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:45.741 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:45.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:45.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:58:45.833 INFO:tasks.rados.rados.0.smithi125.stdout:1260: finishing write tid 1 to smithi12556470-25 2021-03-24T21:58:45.833 INFO:tasks.rados.rados.0.smithi125.stdout:1258: finishing write tid 2 to smithi12556470-32 2021-03-24T21:58:45.833 INFO:tasks.rados.rados.0.smithi125.stdout:1258: finishing write tid 3 to smithi12556470-32 2021-03-24T21:58:45.834 INFO:tasks.rados.rados.0.smithi125.stdout:1258: finishing write tid 4 to smithi12556470-32 2021-03-24T21:58:45.834 INFO:tasks.rados.rados.0.smithi125.stdout:1258: finishing write tid 5 to smithi12556470-32 2021-03-24T21:58:45.834 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 32 v 419 (ObjNum 457 snap 129 seq_num 457) dirty exists 2021-03-24T21:58:45.834 INFO:tasks.rados.rados.0.smithi125.stdout:1258: left oid 32 (ObjNum 457 snap 129 seq_num 457) 2021-03-24T21:58:45.835 INFO:tasks.rados.rados.0.smithi125.stdout:1260: finishing write tid 2 to smithi12556470-25 2021-03-24T21:58:45.835 INFO:tasks.rados.rados.0.smithi125.stdout:1260: finishing write tid 3 to smithi12556470-25 2021-03-24T21:58:45.835 INFO:tasks.rados.rados.0.smithi125.stdout:1260: finishing write tid 4 to smithi12556470-25 2021-03-24T21:58:45.835 INFO:tasks.rados.rados.0.smithi125.stdout:1260: finishing write tid 5 to smithi12556470-25 2021-03-24T21:58:45.835 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 25 v 281 (ObjNum 458 snap 129 seq_num 458) dirty exists 2021-03-24T21:58:45.836 INFO:tasks.rados.rados.0.smithi125.stdout:1260: left oid 25 (ObjNum 458 snap 129 seq_num 458) 2021-03-24T21:58:45.836 INFO:tasks.rados.rados.0.smithi125.stdout:1262: expect (ObjNum 438 snap 118 seq_num 438) 2021-03-24T21:58:45.847 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:45.897 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:45.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:45.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:46.001 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:46.017 INFO:tasks.rados.rados.0.smithi125.stdout:1258: done (5 left) 2021-03-24T21:58:46.017 INFO:tasks.rados.rados.0.smithi125.stdout:1259: done (4 left) 2021-03-24T21:58:46.017 INFO:tasks.rados.rados.0.smithi125.stdout:1260: done (3 left) 2021-03-24T21:58:46.017 INFO:tasks.rados.rados.0.smithi125.stdout:1261: done (2 left) 2021-03-24T21:58:46.018 INFO:tasks.rados.rados.0.smithi125.stdout:1262: done (1 left) 2021-03-24T21:58:46.018 INFO:tasks.rados.rados.0.smithi125.stdout:1263: done (0 left) 2021-03-24T21:58:46.018 INFO:tasks.rados.rados.0.smithi125.stdout:1264: copy_from oid 12 from oid 31 current snap is 129 2021-03-24T21:58:46.018 INFO:tasks.rados.rados.0.smithi125.stdout:1265: write (excl) oid 41 current snap is 129 2021-03-24T21:58:46.018 INFO:tasks.rados.rados.0.smithi125.stdout:1265: seq_num 460 ranges {531220=795720,1935011=516810,3066106=420420} 2021-03-24T21:58:46.028 INFO:tasks.rados.rados.0.smithi125.stdout:1265: writing smithi12556470-41 from 531220 to 1326940 tid 1 2021-03-24T21:58:46.032 INFO:tasks.rados.rados.0.smithi125.stdout:1265: writing smithi12556470-41 from 1935011 to 2451821 tid 2 2021-03-24T21:58:46.038 INFO:tasks.rados.rados.0.smithi125.stdout:1265: writing smithi12556470-41 from 3066106 to 3486526 tid 3 2021-03-24T21:58:46.038 INFO:tasks.rados.rados.0.smithi125.stdout:1266: read oid 1 snap 117 2021-03-24T21:58:46.039 INFO:tasks.rados.rados.0.smithi125.stdout:1266: expect (ObjNum 338 snap 86 seq_num 338) 2021-03-24T21:58:46.039 INFO:tasks.rados.rados.0.smithi125.stdout:1267: copy_from oid 46 from oid 43 current snap is 129 2021-03-24T21:58:46.039 INFO:tasks.rados.rados.0.smithi125.stdout:1265: finishing write tid 1 to smithi12556470-41 2021-03-24T21:58:46.039 INFO:tasks.rados.rados.0.smithi125.stdout:1268: copy_from oid 24 from oid 42 current snap is 129 2021-03-24T21:58:46.040 INFO:tasks.rados.rados.0.smithi125.stdout:1269: snap_remove snap 126 2021-03-24T21:58:46.060 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:46.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:46.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:58:46.166 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:46.224 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:46.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:46.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:58:46.313 INFO:tasks.rados.rados.0.smithi125.stdout:1269: done (5 left) 2021-03-24T21:58:46.313 INFO:tasks.rados.rados.0.smithi125.stdout:1270: write (excl) oid 32 current snap is 129 2021-03-24T21:58:46.314 INFO:tasks.rados.rados.0.smithi125.stdout:1270: seq_num 461 ranges {548027=470969,1593493=776821,3093529=631692} 2021-03-24T21:58:46.316 INFO:tasks.rados.rados.0.smithi125.stdout:1270: writing smithi12556470-32 from 548027 to 1018996 tid 1 2021-03-24T21:58:46.326 INFO:tasks.rados.rados.0.smithi125.stdout:1270: writing smithi12556470-32 from 1593493 to 2370314 tid 2 2021-03-24T21:58:46.329 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:46.334 INFO:tasks.rados.rados.0.smithi125.stdout:1270: writing smithi12556470-32 from 3093529 to 3725221 tid 3 2021-03-24T21:58:46.335 INFO:tasks.rados.rados.0.smithi125.stdout:1271: write oid 40 current snap is 129 2021-03-24T21:58:46.335 INFO:tasks.rados.rados.0.smithi125.stdout:1265: finishing write tid 2 to smithi12556470-41 2021-03-24T21:58:46.336 INFO:tasks.rados.rados.0.smithi125.stdout:1264: finishing copy_from to smithi12556470-12 2021-03-24T21:58:46.336 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 12 v 341 (ObjNum 420 snap 108 seq_num 420) dirty exists 2021-03-24T21:58:46.336 INFO:tasks.rados.rados.0.smithi125.stdout:1264: finishing copy_from racing read to smithi12556470-12 2021-03-24T21:58:46.336 INFO:tasks.rados.rados.0.smithi125.stdout:1267: finishing copy_from to smithi12556470-46 2021-03-24T21:58:46.337 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 46 v 330 (ObjNum 399 snap 102 seq_num 399) dirty exists 2021-03-24T21:58:46.337 INFO:tasks.rados.rados.0.smithi125.stdout:1267: finishing copy_from racing read to smithi12556470-46 2021-03-24T21:58:46.337 INFO:tasks.rados.rados.0.smithi125.stdout:1265: finishing write tid 3 to smithi12556470-41 2021-03-24T21:58:46.337 INFO:tasks.rados.rados.0.smithi125.stdout:1265: finishing write tid 4 to smithi12556470-41 2021-03-24T21:58:46.338 INFO:tasks.rados.rados.0.smithi125.stdout:1265: finishing write tid 5 to smithi12556470-41 2021-03-24T21:58:46.338 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 41 v 371 (ObjNum 460 snap 129 seq_num 460) dirty exists 2021-03-24T21:58:46.338 INFO:tasks.rados.rados.0.smithi125.stdout:1265: left oid 41 (ObjNum 460 snap 129 seq_num 460) 2021-03-24T21:58:46.338 INFO:tasks.rados.rados.0.smithi125.stdout:1266: expect (ObjNum 338 snap 86 seq_num 338) 2021-03-24T21:58:46.377 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:46.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:46.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:58:46.437 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:46 smithi142 conmon[20125]: cluster 2021-03-24T21:58:44.804251+0000 mgr.y (mgr.14144) 446 : cluster 2021-03-24T21:58:46.437 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:46 smithi142 conmon[20125]: [DBG] pgmap v694: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 206 MiB data, 610 MiB used, 536 GiB / 536 GiB avail; 11 MiB/s rd, 1.6 MiB/s wr, 13 op/s; 229/1230 objects degraded (18.618%) 2021-03-24T21:58:46.437 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:46 smithi142 conmon[20125]: cluster 2021-03-24T21:58:45.281300+0000 mon.a (mon.0) 2021-03-24T21:58:46.437 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:46 smithi142 conmon[20125]: 1112 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:58:46.438 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:46 smithi142 conmon[20125]: audit 2021-03-24T21:58:45. 2021-03-24T21:58:46.438 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:46 smithi142 conmon[20125]: 293652+0000 mon.a (mon.0) 1113 : audit [INF] from='client.? v1:172.21.15.125:0/885361269' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T21:58:46.438 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:46 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:58:46.438 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:46 smithi142 conmon[20125]: -03-24T21:58:45.293707+0000 mon.a (mon.0) 1114 : cluster [DBG] osdmap e355: 8 total, 7 up, 6 in 2021-03-24T21:58:46.484 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:46.532 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:46.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:46.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:58:46.611 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:46 smithi125 conmon[19936]: cluster 2021-03-24T21:58:44.804251+0000 mgr.y (mgr. 2021-03-24T21:58:46.612 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:46 smithi125 conmon[19936]: 14144) 446 : cluster [DBG] pgmap v694: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 206 MiB data, 610 MiB used, 536 GiB / 536 GiB avail; 11 MiB/s rd, 1.6 MiB/s wr, 13 op/s; 229/1230 objects degraded (18.618%) 2021-03-24T21:58:46.612 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:46 smithi125 conmon[19936]: cluster 2021-03-24T21:58:45.281300+0000 2021-03-24T21:58:46.612 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:46 smithi125 conmon[19936]: mon.a (mon.0) 1112 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:58:46.612 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:46 smithi125 conmon[19936]: audit 2021-03-24T 2021-03-24T21:58:46.612 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:46 smithi125 conmon[19936]: 21:58:45.293652+0000 mon.a (mon.0) 1113 : audit [INF] from='client.? v1:172.21.15.125:0/885361269' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T21:58:46.613 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:46 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:58:46.613 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:46 smithi125 conmon[19936]: 58:45.293707+0000 mon.a (mon.0) 1114 : cluster [DBG] 2021-03-24T21:58:46.613 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:46 smithi125 conmon[19936]: osdmap e355: 8 total, 7 up, 6 in 2021-03-24T21:58:46.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:46 smithi125 conmon[24927]: cluster 2021-03-24T21:58:44 2021-03-24T21:58:46.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:46 smithi125 conmon[24927]: .804251+0000 mgr.y (mgr.14144) 446 : cluster [DBG] pgmap v694: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 206 MiB data, 610 MiB used, 536 GiB / 536 GiB avail; 11 MiB/s rd, 1.6 MiB/s wr, 13 op/s; 229/1230 objects degraded (18.618%) 2021-03-24T21:58:46.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:46 smithi125 conmon[24927]: cluster 2021-03-24T21:58:46.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:46 smithi125 conmon[24927]: 2021-03-24T21:58:45.281300 2021-03-24T21:58:46.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:46 smithi125 conmon[24927]: +0000 mon.a (mon.0) 1112 : cluster [WRN] 2021-03-24T21:58:46.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:46 smithi125 conmon[24927]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:58:46.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:46 smithi125 conmon[24927]: audit 2021-03-24T21: 2021-03-24T21:58:46.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:46 smithi125 conmon[24927]: 58:45.293652+0000 mon.a (mon.0) 2021-03-24T21:58:46.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:46 smithi125 conmon[24927]: 1113 : audit [INF] from='client.? v1:172.21.15.125:0/885361269' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T21:58:46.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:46 smithi125 conmon[24927]: cluster 2021-03-24T21:58 2021-03-24T21:58:46.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:46 smithi125 conmon[24927]: :45.293707+0000 mon.a (mon.0) 1114 2021-03-24T21:58:46.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:46 smithi125 conmon[24927]: : cluster [DBG] osdmap e355: 8 total, 7 up, 6 in 2021-03-24T21:58:46.621 INFO:tasks.rados.rados.0.smithi125.stdout:1268: finishing copy_from to smithi12556470-24 2021-03-24T21:58:46.621 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 24 v 94 (ObjNum 444 snap 120 seq_num 444) dirty exists 2021-03-24T21:58:46.622 INFO:tasks.rados.rados.0.smithi125.stdout:1268: finishing copy_from racing read to smithi12556470-24 2021-03-24T21:58:46.622 INFO:tasks.rados.rados.0.smithi125.stdout:1271: seq_num 462 ranges {564834=546218,1651975=520479,2720952=559317} 2021-03-24T21:58:46.630 INFO:tasks.rados.rados.0.smithi125.stdout:1271: writing smithi12556470-40 from 564834 to 1111052 tid 1 2021-03-24T21:58:46.638 INFO:tasks.rados.rados.0.smithi125.stdout:1271: writing smithi12556470-40 from 1651975 to 2172454 tid 2 2021-03-24T21:58:46.638 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:46.649 INFO:tasks.rados.rados.0.smithi125.stdout:1271: writing smithi12556470-40 from 2720952 to 3280269 tid 3 2021-03-24T21:58:46.649 INFO:tasks.rados.rados.0.smithi125.stdout:1264: done (6 left) 2021-03-24T21:58:46.649 INFO:tasks.rados.rados.0.smithi125.stdout:1265: done (5 left) 2021-03-24T21:58:46.650 INFO:tasks.rados.rados.0.smithi125.stdout:1266: done (4 left) 2021-03-24T21:58:46.650 INFO:tasks.rados.rados.0.smithi125.stdout:1267: done (3 left) 2021-03-24T21:58:46.650 INFO:tasks.rados.rados.0.smithi125.stdout:1268: done (2 left) 2021-03-24T21:58:46.650 INFO:tasks.rados.rados.0.smithi125.stdout:1272: snap_create 2021-03-24T21:58:46.650 INFO:tasks.rados.rados.0.smithi125.stdout:1270: finishing write tid 1 to smithi12556470-32 2021-03-24T21:58:46.651 INFO:tasks.rados.rados.0.smithi125.stdout:1270: finishing write tid 2 to smithi12556470-32 2021-03-24T21:58:46.651 INFO:tasks.rados.rados.0.smithi125.stdout:1270: finishing write tid 3 to smithi12556470-32 2021-03-24T21:58:46.651 INFO:tasks.rados.rados.0.smithi125.stdout:1270: finishing write tid 4 to smithi12556470-32 2021-03-24T21:58:46.651 INFO:tasks.rados.rados.0.smithi125.stdout:1270: finishing write tid 5 to smithi12556470-32 2021-03-24T21:58:46.652 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 32 v 427 (ObjNum 461 snap 129 seq_num 461) dirty exists 2021-03-24T21:58:46.652 INFO:tasks.rados.rados.0.smithi125.stdout:1270: left oid 32 (ObjNum 461 snap 129 seq_num 461) 2021-03-24T21:58:46.652 INFO:tasks.rados.rados.0.smithi125.stdout:1271: finishing write tid 1 to smithi12556470-40 2021-03-24T21:58:46.652 INFO:tasks.rados.rados.0.smithi125.stdout:1271: finishing write tid 2 to smithi12556470-40 2021-03-24T21:58:46.653 INFO:tasks.rados.rados.0.smithi125.stdout:1271: finishing write tid 3 to smithi12556470-40 2021-03-24T21:58:46.654 INFO:tasks.rados.rados.0.smithi125.stdout:1271: finishing write tid 4 to smithi12556470-40 2021-03-24T21:58:46.654 INFO:tasks.rados.rados.0.smithi125.stdout:1271: finishing write tid 5 to smithi12556470-40 2021-03-24T21:58:46.654 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 40 v 432 (ObjNum 462 snap 129 seq_num 462) dirty exists 2021-03-24T21:58:46.654 INFO:tasks.rados.rados.0.smithi125.stdout:1271: left oid 40 (ObjNum 462 snap 129 seq_num 462) 2021-03-24T21:58:46.665 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-03-24T21:58:46.667 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-03-24T21:58:46.681 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:46.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:46.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:46.786 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:46.847 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:46.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:46.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:58:46.954 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:47.001 INFO:teuthology.orchestra.run.smithi142.stdout:get_pg_num_history pg_num_history pg_num_history(e339 pg_nums {1={19=1},2={56=3},3={60=32},4={62=32},5={64=32},6={66=32},7={99=16,198=26,275=36}} deleted_pools ) 2021-03-24T21:58:47.001 INFO:teuthology.orchestra.run.smithi142.stdout:Importing pgid 3.3 2021-03-24T21:58:47.002 INFO:teuthology.orchestra.run.smithi142.stdout:Write #3:c00c4a6b:::zonegroup_info.62447104-cea8-48da-bd05-4040025bf9f2:head# 2021-03-24T21:58:47.002 INFO:teuthology.orchestra.run.smithi142.stdout:snapset 0=[]:{} 2021-03-24T21:58:47.002 INFO:teuthology.orchestra.run.smithi142.stdout:write_pg epoch 308 info 3.3( v 89'2 (0'0,89'2] local-lis/les=108/109 n=1 ec=60/60 lis/c=108/108 les/c/f=109/112/0 sis=108) 2021-03-24T21:58:47.003 INFO:teuthology.orchestra.run.smithi142.stdout:Import successful 2021-03-24T21:58:47.011 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:47.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:47.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:58:47.117 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:47.173 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:47.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:47.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:47.278 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:47.315 INFO:tasks.rados.rados.0.smithi125.stdout:1270: done (2 left) 2021-03-24T21:58:47.316 INFO:tasks.rados.rados.0.smithi125.stdout:1271: done (1 left) 2021-03-24T21:58:47.316 INFO:tasks.rados.rados.0.smithi125.stdout:1272: done (0 left) 2021-03-24T21:58:47.316 INFO:tasks.rados.rados.0.smithi125.stdout:1273: write (excl) oid 28 current snap is 130 2021-03-24T21:58:47.316 INFO:tasks.rados.rados.0.smithi125.stdout:1273: seq_num 463 ranges {581641=621467,1994104=780490,3432022=349619} 2021-03-24T21:58:47.324 INFO:tasks.rados.rados.0.smithi125.stdout:1273: writing smithi12556470-28 from 581641 to 1203108 tid 1 2021-03-24T21:58:47.334 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:47.336 INFO:tasks.rados.rados.0.smithi125.stdout:1273: writing smithi12556470-28 from 1994104 to 2774594 tid 2 2021-03-24T21:58:47.343 INFO:tasks.rados.rados.0.smithi125.stdout:1273: writing smithi12556470-28 from 3432022 to 3781641 tid 3 2021-03-24T21:58:47.343 INFO:tasks.rados.rados.0.smithi125.stdout:1274: rollback oid 20 current snap is 130 2021-03-24T21:58:47.343 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 20 to 128 2021-03-24T21:58:47.343 INFO:tasks.rados.rados.0.smithi125.stdout:1275: write (excl) oid 1 current snap is 130 2021-03-24T21:58:47.344 INFO:tasks.rados.rados.0.smithi125.stdout:1275: seq_num 464 ranges {598448=413069,1768939=524148,2775798=581861,3781488=16960} 2021-03-24T21:58:47.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:47.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:58:47.349 INFO:tasks.rados.rados.0.smithi125.stdout:1275: writing smithi12556470-1 from 598448 to 1011517 tid 1 2021-03-24T21:58:47.357 INFO:tasks.rados.rados.0.smithi125.stdout:1275: writing smithi12556470-1 from 1768939 to 2293087 tid 2 2021-03-24T21:58:47.364 INFO:tasks.rados.rados.0.smithi125.stdout:1275: writing smithi12556470-1 from 2775798 to 3357659 tid 3 2021-03-24T21:58:47.367 INFO:tasks.rados.rados.0.smithi125.stdout:1275: writing smithi12556470-1 from 3781488 to 3798448 tid 4 2021-03-24T21:58:47.368 INFO:tasks.rados.rados.0.smithi125.stdout:1276: rollback oid 30 current snap is 130 2021-03-24T21:58:47.368 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 30 to 116 2021-03-24T21:58:47.368 INFO:tasks.rados.rados.0.smithi125.stdout:1277: delete oid 47 current snap is 130 2021-03-24T21:58:47.369 INFO:tasks.rados.rados.0.smithi125.stdout:1277: done (4 left) 2021-03-24T21:58:47.370 INFO:tasks.rados.rados.0.smithi125.stdout:1278: snap_create 2021-03-24T21:58:47.374 INFO:tasks.rados.rados.0.smithi125.stdout:1275: finishing write tid 1 to smithi12556470-1 2021-03-24T21:58:47.381 INFO:tasks.rados.rados.0.smithi125.stdout:1273: finishing write tid 1 to smithi12556470-28 2021-03-24T21:58:47.385 INFO:tasks.rados.rados.0.smithi125.stdout:1273: finishing write tid 2 to smithi12556470-28 2021-03-24T21:58:47.385 INFO:tasks.rados.rados.0.smithi125.stdout:1273: finishing write tid 3 to smithi12556470-28 2021-03-24T21:58:47.386 INFO:tasks.rados.rados.0.smithi125.stdout:1275: finishing write tid 2 to smithi12556470-1 2021-03-24T21:58:47.386 INFO:tasks.rados.rados.0.smithi125.stdout:1273: finishing write tid 4 to smithi12556470-28 2021-03-24T21:58:47.389 INFO:tasks.rados.rados.0.smithi125.stdout:1275: finishing write tid 3 to smithi12556470-1 2021-03-24T21:58:47.390 INFO:tasks.rados.rados.0.smithi125.stdout:1275: finishing write tid 4 to smithi12556470-1 2021-03-24T21:58:47.390 INFO:tasks.rados.rados.0.smithi125.stdout:1275: finishing write tid 5 to smithi12556470-1 2021-03-24T21:58:47.390 INFO:tasks.rados.rados.0.smithi125.stdout:1273: finishing write tid 5 to smithi12556470-28 2021-03-24T21:58:47.391 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 28 v 300 (ObjNum 463 snap 130 seq_num 463) dirty exists 2021-03-24T21:58:47.391 INFO:tasks.rados.rados.0.smithi125.stdout:1273: left oid 28 (ObjNum 463 snap 130 seq_num 463) 2021-03-24T21:58:47.391 INFO:tasks.rados.rados.0.smithi125.stdout:1275: finishing write tid 6 to smithi12556470-1 2021-03-24T21:58:47.391 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 1 v 380 (ObjNum 464 snap 130 seq_num 464) dirty exists 2021-03-24T21:58:47.391 INFO:tasks.rados.rados.0.smithi125.stdout:1275: left oid 1 (ObjNum 464 snap 130 seq_num 464) 2021-03-24T21:58:47.392 INFO:tasks.rados.rados.0.smithi125.stdout:1274: finishing rollback tid 0 to smithi12556470-20 2021-03-24T21:58:47.393 INFO:tasks.rados.rados.0.smithi125.stdout:1274: finishing rollback tid 1 to smithi12556470-20 2021-03-24T21:58:47.393 INFO:tasks.rados.rados.0.smithi125.stdout:1274: finishing rollback tid 2 to smithi12556470-20 2021-03-24T21:58:47.393 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 20 v 285 (ObjNum 449 snap 123 seq_num 449) dirty exists 2021-03-24T21:58:47.397 INFO:tasks.rados.rados.0.smithi125.stdout:1276: finishing rollback tid 0 to smithi12556470-30 2021-03-24T21:58:47.409 INFO:tasks.rados.rados.0.smithi125.stdout:1276: finishing rollback tid 1 to smithi12556470-30 2021-03-24T21:58:47.409 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 30 v 285 (ObjNum 1116564000 snap 21916 seq_num 248) dirty dne 2021-03-24T21:58:47.448 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:47.455 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:47 smithi125 conmon[24927]: cluster 2021-03-24T21:58:46.302514+0000 2021-03-24T21:58:47.456 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:47 smithi125 conmon[24927]: mon.a (mon.0) 1115 : cluster [DBG] osdmap e356: 8 total, 7 up, 6 in 2021-03-24T21:58:47.456 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:47 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:58:47.457 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:47 smithi125 conmon[19936]: 21:58:46.302514+0000 mon.a (mon.0) 1115 : cluster [DBG] osdmap e356: 8 total, 7 up, 6 in 2021-03-24T21:58:47.540 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:47.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:47.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:58:47.562 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:47 smithi142 conmon[20125]: cluster 2021-03-24T21:58:47.563 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:47 smithi142 conmon[20125]: 2021-03-24T21:58 2021-03-24T21:58:47.563 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:47 smithi142 conmon[20125]: :46.302514+0000 mon.a (mon.0) 1115 : cluster [DBG] osdmap e356: 8 total, 7 up, 6 in 2021-03-24T21:58:47.645 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:47.695 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:47.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:47.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:58:47.800 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:47.855 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:47.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:47.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:58:47.960 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:48.016 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:48.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:48.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:58:48.121 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:48.174 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:48.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:48.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:48.281 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:48.322 INFO:tasks.rados.rados.0.smithi125.stdout:1273: done (4 left) 2021-03-24T21:58:48.322 INFO:tasks.rados.rados.0.smithi125.stdout:1274: done (3 left) 2021-03-24T21:58:48.322 INFO:tasks.rados.rados.0.smithi125.stdout:1275: done (2 left) 2021-03-24T21:58:48.323 INFO:tasks.rados.rados.0.smithi125.stdout:1276: done (1 left) 2021-03-24T21:58:48.323 INFO:tasks.rados.rados.0.smithi125.stdout:1278: done (0 left) 2021-03-24T21:58:48.323 INFO:tasks.rados.rados.0.smithi125.stdout:1279: rollback oid 48 current snap is 131 2021-03-24T21:58:48.323 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 48 to 129 2021-03-24T21:58:48.324 INFO:tasks.rados.rados.0.smithi125.stdout:1280: rollback oid 2 current snap is 131 2021-03-24T21:58:48.324 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 2 to 129 2021-03-24T21:58:48.324 INFO:tasks.rados.rados.0.smithi125.stdout:1281: write (excl) oid 45 current snap is 131 2021-03-24T21:58:48.324 INFO:tasks.rados.rados.0.smithi125.stdout:1281: seq_num 466 ranges {632062=563567,1885903=527817,2830644=720758} 2021-03-24T21:58:48.341 INFO:tasks.rados.rados.0.smithi125.stdout:1281: writing smithi12556470-45 from 632062 to 1195629 tid 1 2021-03-24T21:58:48.342 INFO:tasks.rados.rados.0.smithi125.stdout:1281: writing smithi12556470-45 from 1885903 to 2413720 tid 2 2021-03-24T21:58:48.350 INFO:tasks.rados.rados.0.smithi125.stdout:1281: writing smithi12556470-45 from 2830644 to 3551402 tid 3 2021-03-24T21:58:48.351 INFO:tasks.rados.rados.0.smithi125.stdout:1279: finishing rollback tid 0 to smithi12556470-48 2021-03-24T21:58:48.351 INFO:tasks.rados.rados.0.smithi125.stdout:1279: finishing rollback tid 1 to smithi12556470-48 2021-03-24T21:58:48.351 INFO:tasks.rados.rados.0.smithi125.stdout:1279: finishing rollback tid 2 to smithi12556470-48 2021-03-24T21:58:48.351 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 48 v 384 (ObjNum 423 snap 112 seq_num 423) dirty exists 2021-03-24T21:58:48.352 INFO:tasks.rados.rados.0.smithi125.stdout:1281: finishing write tid 1 to smithi12556470-45 2021-03-24T21:58:48.352 INFO:tasks.rados.rados.0.smithi125.stdout:1280: finishing rollback tid 0 to smithi12556470-2 2021-03-24T21:58:48.352 INFO:tasks.rados.rados.0.smithi125.stdout:1280: finishing rollback tid 1 to smithi12556470-2 2021-03-24T21:58:48.352 INFO:tasks.rados.rados.0.smithi125.stdout:1279: done (2 left) 2021-03-24T21:58:48.352 INFO:tasks.rados.rados.0.smithi125.stdout:1282: write oid 34 current snap is 131 2021-03-24T21:58:48.353 INFO:tasks.rados.rados.0.smithi125.stdout:1282: seq_num 467 ranges {648869=638816,1828032=787828,3141714=532030} 2021-03-24T21:58:48.359 INFO:tasks.rados.rados.0.smithi125.stdout:1282: writing smithi12556470-34 from 648869 to 1287685 tid 1 2021-03-24T21:58:48.370 INFO:tasks.rados.rados.0.smithi125.stdout:1282: writing smithi12556470-34 from 1828032 to 2615860 tid 2 2021-03-24T21:58:48.378 INFO:tasks.rados.rados.0.smithi125.stdout:1282: writing smithi12556470-34 from 3141714 to 3673744 tid 3 2021-03-24T21:58:48.380 INFO:tasks.rados.rados.0.smithi125.stdout:1281: finishing write tid 2 to smithi12556470-45 2021-03-24T21:58:48.380 INFO:tasks.rados.rados.0.smithi125.stdout:1280: finishing rollback tid 2 to smithi12556470-2 2021-03-24T21:58:48.380 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 2 v 355 (ObjNum 425 snap 112 seq_num 425) dirty exists 2021-03-24T21:58:48.380 INFO:tasks.rados.rados.0.smithi125.stdout:1281: finishing write tid 3 to smithi12556470-45 2021-03-24T21:58:48.380 INFO:tasks.rados.rados.0.smithi125.stdout:1281: finishing write tid 4 to smithi12556470-45 2021-03-24T21:58:48.381 INFO:tasks.rados.rados.0.smithi125.stdout:1281: finishing write tid 5 to smithi12556470-45 2021-03-24T21:58:48.381 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 45 v 305 (ObjNum 466 snap 131 seq_num 466) dirty exists 2021-03-24T21:58:48.381 INFO:tasks.rados.rados.0.smithi125.stdout:1281: left oid 45 (ObjNum 466 snap 131 seq_num 466) 2021-03-24T21:58:48.381 INFO:tasks.rados.rados.0.smithi125.stdout:1282: finishing write tid 1 to smithi12556470-34 2021-03-24T21:58:48.382 INFO:tasks.rados.rados.0.smithi125.stdout:1282: finishing write tid 2 to smithi12556470-34 2021-03-24T21:58:48.382 INFO:tasks.rados.rados.0.smithi125.stdout:1280: done (2 left) 2021-03-24T21:58:48.382 INFO:tasks.rados.rados.0.smithi125.stdout:1281: done (1 left) 2021-03-24T21:58:48.382 INFO:tasks.rados.rados.0.smithi125.stdout:1283: read oid 21 snap -1 2021-03-24T21:58:48.383 INFO:tasks.rados.rados.0.smithi125.stdout:1283: expect (ObjNum 443 snap 120 seq_num 443) 2021-03-24T21:58:48.383 INFO:tasks.rados.rados.0.smithi125.stdout:1284: rollback oid 27 current snap is 131 2021-03-24T21:58:48.383 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 27 to 125 2021-03-24T21:58:48.383 INFO:tasks.rados.rados.0.smithi125.stdout:1285: snap_create 2021-03-24T21:58:48.387 INFO:tasks.rados.rados.0.smithi125.stdout:1282: finishing write tid 3 to smithi12556470-34 2021-03-24T21:58:48.389 INFO:tasks.rados.rados.0.smithi125.stdout:1282: finishing write tid 4 to smithi12556470-34 2021-03-24T21:58:48.390 INFO:tasks.rados.rados.0.smithi125.stdout:1282: finishing write tid 5 to smithi12556470-34 2021-03-24T21:58:48.391 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 34 v 292 (ObjNum 467 snap 131 seq_num 467) dirty exists 2021-03-24T21:58:48.391 INFO:tasks.rados.rados.0.smithi125.stdout:1282: left oid 34 (ObjNum 467 snap 131 seq_num 467) 2021-03-24T21:58:48.391 INFO:tasks.rados.rados.0.smithi125.stdout:1284: finishing rollback tid 0 to smithi12556470-27 2021-03-24T21:58:48.397 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:48.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:48.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:58:48.405 INFO:tasks.rados.rados.0.smithi125.stdout:1284: finishing rollback tid 1 to smithi12556470-27 2021-03-24T21:58:48.405 INFO:tasks.rados.rados.0.smithi125.stdout:1284: finishing rollback tid 2 to smithi12556470-27 2021-03-24T21:58:48.405 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 27 v 296 (ObjNum 400 snap 102 seq_num 400) dirty exists 2021-03-24T21:58:48.413 INFO:tasks.rados.rados.0.smithi125.stdout:1283: expect (ObjNum 443 snap 120 seq_num 443) 2021-03-24T21:58:48.503 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:48.511 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:48 smithi125 conmon[19936]: cluster 2021-03-24T21:58:46.805446+0000 2021-03-24T21:58:48.512 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:48 smithi125 conmon[19936]: mgr.y (mgr.14144) 447 : cluster [DBG] pgmap v697: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 206 MiB data, 619 MiB used, 536 GiB / 536 GiB avail; 11 MiB/s rd, 1.6 MiB/s wr, 13 op/s; 229/1230 objects degraded (18.618%) 2021-03-24T21:58:48.512 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:48 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:58:48.512 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:48 smithi125 conmon[19936]: 21:58:47.314124+0000 mon.a (mon.0) 1116 : cluster [DBG] osdmap e357: 8 total, 7 up, 6 in 2021-03-24T21:58:48.513 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:48 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:58:48.513 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:48 smithi125 conmon[24927]: 24T21:58:46.805446+0000 mgr.y (mgr.14144) 447 : cluster [DBG] pgmap v697: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 206 MiB data, 619 MiB used, 536 GiB / 536 GiB avail; 11 MiB/s rd, 1.6 MiB/s wr, 13 op/s; 229/1230 objects degraded (18.618%) 2021-03-24T21:58:48.513 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:48 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:58:48.513 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:48 smithi125 conmon[24927]: 24T21:58:47.314124+0000 2021-03-24T21:58:48.513 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:48 smithi125 conmon[24927]: mon.a (mon.0) 1116 : cluster [DBG] osdmap e357: 8 total, 7 up, 6 in 2021-03-24T21:58:48.514 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:48 smithi125 conmon[24927]: 2021-03-24T21:58:48.558 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-24T21:58:48.558 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-24T21:58:48.558 INFO:teuthology.orchestra.run.smithi125.stdout: "num_ops": 0 2021-03-24T21:58:48.559 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-24T21:58:48.578 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:48 smithi142 conmon[20125]: cluster 2021-03-24T21:58 2021-03-24T21:58:48.578 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:48 smithi142 conmon[20125]: :46.805446+0000 mgr.y (mgr.14144) 447 : cluster [DBG] pgmap v697: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 206 MiB data, 619 MiB used, 536 GiB / 536 GiB avail; 11 MiB/s rd, 1.6 MiB/s wr, 13 op/s; 229/1230 objects degraded (18.618%) 2021-03-24T21:58:48.578 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:48 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:58:48.578 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:48 smithi142 conmon[20125]: -24T21:58:47.314124+0000 mon.a (mon.0) 1116 2021-03-24T21:58:48.579 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:48 smithi142 conmon[20125]: : cluster [DBG] osdmap e357: 8 total, 7 up, 6 in 2021-03-24T21:58:48.595 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:48.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:48.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:58:48.600 DEBUG:teuthology.orchestra.run.smithi142:> sudo rm -f /var/log/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/exp.3.3.5 2021-03-24T21:58:48.669 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-24T21:58:48.700 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:48.751 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:48.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:48.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:48.857 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:48.909 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:48.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:48.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:58:48.936 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd unset noscrub 2021-03-24T21:58:49.016 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:49.073 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:49.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:49.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:49.178 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:49.234 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:49.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:49.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:58:49.330 INFO:tasks.rados.rados.0.smithi125.stdout:1282: done (3 left) 2021-03-24T21:58:49.331 INFO:tasks.rados.rados.0.smithi125.stdout:1283: done (2 left) 2021-03-24T21:58:49.331 INFO:tasks.rados.rados.0.smithi125.stdout:1284: done (1 left) 2021-03-24T21:58:49.331 INFO:tasks.rados.rados.0.smithi125.stdout:1285: done (0 left) 2021-03-24T21:58:49.332 INFO:tasks.rados.rados.0.smithi125.stdout:1286: write oid 22 current snap is 132 2021-03-24T21:58:49.332 INFO:tasks.rados.rados.0.smithi125.stdout:1286: seq_num 468 ranges {665676=714065,1886514=531486,3169137=696539} 2021-03-24T21:58:49.338 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:49.341 INFO:tasks.rados.rados.0.smithi125.stdout:1286: writing smithi12556470-22 from 665676 to 1379741 tid 1 2021-03-24T21:58:49.348 INFO:tasks.rados.rados.0.smithi125.stdout:1286: writing smithi12556470-22 from 1886514 to 2418000 tid 2 2021-03-24T21:58:49.358 INFO:tasks.rados.rados.0.smithi125.stdout:1286: writing smithi12556470-22 from 3169137 to 3865676 tid 3 2021-03-24T21:58:49.358 INFO:tasks.rados.rados.0.smithi125.stdout:1287: read oid 1 snap -1 2021-03-24T21:58:49.359 INFO:tasks.rados.rados.0.smithi125.stdout:1287: expect (ObjNum 464 snap 130 seq_num 464) 2021-03-24T21:58:49.359 INFO:tasks.rados.rados.0.smithi125.stdout:1288: snap_create 2021-03-24T21:58:49.377 INFO:tasks.rados.rados.0.smithi125.stdout:1286: finishing write tid 1 to smithi12556470-22 2021-03-24T21:58:49.377 INFO:tasks.rados.rados.0.smithi125.stdout:1286: finishing write tid 2 to smithi12556470-22 2021-03-24T21:58:49.378 INFO:tasks.rados.rados.0.smithi125.stdout:1286: finishing write tid 3 to smithi12556470-22 2021-03-24T21:58:49.378 INFO:tasks.rados.rados.0.smithi125.stdout:1286: finishing write tid 4 to smithi12556470-22 2021-03-24T21:58:49.378 INFO:tasks.rados.rados.0.smithi125.stdout:1286: finishing write tid 5 to smithi12556470-22 2021-03-24T21:58:49.378 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 22 v 310 (ObjNum 468 snap 132 seq_num 468) dirty exists 2021-03-24T21:58:49.379 INFO:tasks.rados.rados.0.smithi125.stdout:1286: left oid 22 (ObjNum 468 snap 132 seq_num 468) 2021-03-24T21:58:49.385 INFO:tasks.rados.rados.0.smithi125.stdout:1287: expect (ObjNum 464 snap 130 seq_num 464) 2021-03-24T21:58:49.437 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:49.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:49.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:58:49.473 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.1 dump_blocked_ops 2021-03-24T21:58:49.495 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:58:49.517 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:49 smithi125 conmon[19936]: cluster 2021-03-24T21:58:48.319505 2021-03-24T21:58:49.517 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:49 smithi125 conmon[19936]: +0000 mon.a (mon.0) 1117 : cluster [DBG] osdmap e358: 8 total, 7 up, 6 in 2021-03-24T21:58:49.518 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:49 smithi125 conmon[24927]: cluster 2021-03-24T21:58:48. 2021-03-24T21:58:49.518 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:49 smithi125 conmon[24927]: 319505+0000 mon.a (mon.0) 1117 : cluster [DBG] osdmap e358: 8 total, 7 up, 6 in 2021-03-24T21:58:49.535 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:58:49.548 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:49.591 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:49.595 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:49 smithi142 conmon[20125]: cluster 2021-03-24T21:58 2021-03-24T21:58:49.595 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:49 smithi142 conmon[20125]: :48.319505+0000 mon.a (mon.0) 1117 : cluster [DBG] osdmap e358: 8 total, 7 up, 6 in 2021-03-24T21:58:49.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:49.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:49.697 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:49.751 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:49.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:49.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:58:49.858 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:49.914 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:49.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:49.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:58:50.020 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:50.080 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:50.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:50.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:58:50.141 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:58:50.186 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:50.197 INFO:tasks.rados.rados.0.smithi125.stdout:1286: done (2 left) 2021-03-24T21:58:50.198 INFO:tasks.rados.rados.0.smithi125.stdout:1287: done (1 left) 2021-03-24T21:58:50.198 INFO:tasks.rados.rados.0.smithi125.stdout:1288: done (0 left) 2021-03-24T21:58:50.198 INFO:tasks.rados.rados.0.smithi125.stdout:1289: delete oid 2 current snap is 133 2021-03-24T21:58:50.232 INFO:tasks.rados.rados.0.smithi125.stdout:1289: done (0 left) 2021-03-24T21:58:50.233 INFO:tasks.rados.rados.0.smithi125.stdout:1290: copy_from oid 27 from oid 19 current snap is 133 2021-03-24T21:58:50.233 INFO:tasks.rados.rados.0.smithi125.stdout:1291: write (excl) oid 44 current snap is 133 2021-03-24T21:58:50.233 INFO:tasks.rados.rados.0.smithi125.stdout:1291: seq_num 470 ranges {699290=464563,1603478=535155,2823983=765846} 2021-03-24T21:58:50.237 INFO:tasks.rados.rados.0.smithi125.stdout:1291: writing smithi12556470-44 from 699290 to 1163853 tid 1 2021-03-24T21:58:50.244 INFO:tasks.rados.rados.0.smithi125.stdout:1291: writing smithi12556470-44 from 1603478 to 2138633 tid 2 2021-03-24T21:58:50.254 INFO:tasks.rados.rados.0.smithi125.stdout:1291: writing smithi12556470-44 from 2823983 to 3589829 tid 3 2021-03-24T21:58:50.255 INFO:tasks.rados.rados.0.smithi125.stdout:1290: finishing copy_from to smithi12556470-27 2021-03-24T21:58:50.255 INFO:tasks.rados.rados.0.smithi125.stdout:1290: got expected ENOENT (src dne) 2021-03-24T21:58:50.255 INFO:tasks.rados.rados.0.smithi125.stdout:1290: finishing copy_from racing read to smithi12556470-27 2021-03-24T21:58:50.256 INFO:tasks.rados.rados.0.smithi125.stdout:1290: done (1 left) 2021-03-24T21:58:50.256 INFO:tasks.rados.rados.0.smithi125.stdout:1292: read oid 26 snap -1 2021-03-24T21:58:50.256 INFO:tasks.rados.rados.0.smithi125.stdout:1292: expect (ObjNum 438 snap 118 seq_num 438) 2021-03-24T21:58:50.256 INFO:tasks.rados.rados.0.smithi125.stdout:1293: read oid 42 snap 130 2021-03-24T21:58:50.257 INFO:tasks.rados.rados.0.smithi125.stdout:1293: expect (ObjNum 444 snap 120 seq_num 444) 2021-03-24T21:58:50.257 INFO:tasks.rados.rados.0.smithi125.stdout:1294: copy_from oid 48 from oid 40 current snap is 133 2021-03-24T21:58:50.257 INFO:tasks.rados.rados.0.smithi125.stdout:1295: read oid 24 snap 132 2021-03-24T21:58:50.257 INFO:tasks.rados.rados.0.smithi125.stdout:1295: expect (ObjNum 444 snap 120 seq_num 444) 2021-03-24T21:58:50.257 INFO:tasks.rados.rados.0.smithi125.stdout:1296: snap_remove snap 125 2021-03-24T21:58:50.280 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:50.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:50.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:58:50.386 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:50.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:50 smithi125 conmon[19936]: cluster 2021-03-24T21:58:48.806321+0000 2021-03-24T21:58:50.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:50 smithi125 conmon[19936]: mgr.y (mgr.14144) 448 : cluster [DBG] pgmap v700: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 211 MiB data, 611 MiB used, 536 GiB / 536 GiB avail; 0 B/s rd, 1.2 MiB/s wr, 0 op/s; 229/1233 objects degraded (18.573%) 2021-03-24T21:58:50.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:50 smithi125 conmon[19936]: cluster 2021-03-24T21:58:49.328989+0000 mon.a (mon.0) 1118 : cluster [DBG] osdmap e359: 8 total, 7 up, 6 in 2021-03-24T21:58:50.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:50 smithi125 conmon[19936]: cluster 2021-03-24T21:58 2021-03-24T21:58:50.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:50 smithi125 conmon[19936]: :50.187476+0000 mon.a (mon.0) 1119 2021-03-24T21:58:50.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:50 smithi125 conmon[19936]: : cluster [WRN] Health check update: Degraded data redundancy: 229/1233 objects degraded (18.573%), 40 pgs degraded (PG_DEGRADED) 2021-03-24T21:58:50.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:50 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T21:58:50.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:50 smithi125 conmon[19936]: T21:58:50.193818+0000 mon.a ( 2021-03-24T21:58:50.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:50 smithi125 conmon[19936]: mon.0) 1120 : cluster [DBG] osdmap e360: 8 total, 7 up, 6 in 2021-03-24T21:58:50.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:50 smithi125 conmon[24927]: cluster 2021-03-24T21:58:48.806321+0000 2021-03-24T21:58:50.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:50 smithi125 conmon[24927]: mgr.y (mgr.14144) 448 : cluster [DBG] pgmap v700: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 211 MiB data, 611 MiB used, 536 GiB / 536 GiB avail; 0 B/s rd, 1.2 MiB/s wr, 0 op/s; 229/1233 objects degraded (18.573%) 2021-03-24T21:58:50.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:50 smithi125 conmon[24927]: cluster 2021-03-24T21:58:49 2021-03-24T21:58:50.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:50 smithi125 conmon[24927]: .328989+0000 mon.a (mon.0) 1118 : cluster [DBG] osdmap e359: 8 total, 7 up, 6 in 2021-03-24T21:58:50.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:50 smithi125 conmon[24927]: cluster 2021-03-24T21:58:50 2021-03-24T21:58:50.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:50 smithi125 conmon[24927]: .187476+0000 mon.a (mon.0) 1119 : cluster [WRN] 2021-03-24T21:58:50.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:50 smithi125 conmon[24927]: Health check update: Degraded data redundancy: 229/1233 objects degraded (18.573%), 40 pgs degraded (PG_DEGRADED) 2021-03-24T21:58:50.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:50 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:58:50.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:50 smithi125 conmon[24927]: 03-24T21:58:50.193818+0000 mon.a (mon.0) 1120 : cluster 2021-03-24T21:58:50.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:50 smithi125 conmon[24927]: [DBG] osdmap e360: 8 total, 7 up, 6 in 2021-03-24T21:58:50.434 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:50.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:50.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:58:50.538 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:50.549 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:50 smithi142 conmon[20125]: cluster 2021-03-24T21:58:48. 2021-03-24T21:58:50.549 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:50 smithi142 conmon[20125]: 806321+0000 mgr.y (mgr.14144) 448 : cluster [DBG] pgmap v700: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 211 MiB data, 611 MiB used, 536 GiB / 536 GiB avail; 0 B/s rd, 1.2 MiB/s wr, 0 op/s; 229/1233 objects degraded (18.573%) 2021-03-24T21:58:50.550 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:50 smithi142 conmon[20125]: cluster 2021-03-24T21:58:49.328989+0000 mon.a (mon.0) 1118 : 2021-03-24T21:58:50.550 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:50 smithi142 conmon[20125]: cluster [DBG] osdmap e359: 8 total, 7 up, 6 in 2021-03-24T21:58:50.550 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:50 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:58:50.550 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:50 smithi142 conmon[20125]: -24T21:58:50.187476+0000 mon.a (mon.0) 2021-03-24T21:58:50.551 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:50 smithi142 conmon[20125]: 1119 : cluster [WRN] Health check update: Degraded data redundancy: 229/1233 objects degraded (18.573%), 40 pgs degraded (PG_DEGRADED) 2021-03-24T21:58:50.551 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:50 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:58:50.551 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:50 smithi142 conmon[20125]: 58:50.193818+0000 mon.a (mon.0) 2021-03-24T21:58:50.551 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:50 smithi142 conmon[20125]: 1120 : cluster [DBG] osdmap e360: 8 total, 7 up, 6 in 2021-03-24T21:58:50.590 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:50.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:50.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:58:50.695 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:50.746 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:50.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:50.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:58:50.852 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:50.908 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:50.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:50.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:51.015 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:51.071 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:51.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:51.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:51.178 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:51.214 INFO:tasks.rados.rados.0.smithi125.stdout:1296: done (5 left) 2021-03-24T21:58:51.215 INFO:tasks.rados.rados.0.smithi125.stdout:1297: rollback oid 3 current snap is 133 2021-03-24T21:58:51.215 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 3 to 130 2021-03-24T21:58:51.215 INFO:tasks.rados.rados.0.smithi125.stdout:1298: write (excl) oid 46 current snap is 133 2021-03-24T21:58:51.215 INFO:tasks.rados.rados.0.smithi125.stdout:1298: seq_num 471 ranges {716097=539812,1945607=795166,3535053=381044} 2021-03-24T21:58:51.221 INFO:tasks.rados.rados.0.smithi125.stdout:1298: writing smithi12556470-46 from 716097 to 1255909 tid 1 2021-03-24T21:58:51.234 INFO:tasks.rados.rados.0.smithi125.stdout:1298: writing smithi12556470-46 from 1945607 to 2740773 tid 2 2021-03-24T21:58:51.242 INFO:tasks.rados.rados.0.smithi125.stdout:1298: writing smithi12556470-46 from 3535053 to 3916097 tid 3 2021-03-24T21:58:51.243 INFO:tasks.rados.rados.0.smithi125.stdout:1299: snap_remove snap 124 2021-03-24T21:58:51.260 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:51.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:51.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:58:51.368 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:51.443 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:51.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:51.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:58:51.553 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:51.599 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:51.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:51.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:51.705 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:51.764 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:51.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:51.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:51.872 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:51.930 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:51.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:51.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:58:52.036 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:52.096 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:52.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:52.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:58:52.199 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:52.219 INFO:tasks.rados.rados.0.smithi125.stdout:1299: done (7 left) 2021-03-24T21:58:52.219 INFO:tasks.rados.rados.0.smithi125.stdout:1300: write oid 25 current snap is 133 2021-03-24T21:58:52.219 INFO:tasks.rados.rados.0.smithi125.stdout:1300: seq_num 472 ranges {732904=731414,2120442=538824,3278829=504743} 2021-03-24T21:58:52.230 INFO:tasks.rados.rados.0.smithi125.stdout:1300: writing smithi12556470-25 from 732904 to 1464318 tid 1 2021-03-24T21:58:52.239 INFO:tasks.rados.rados.0.smithi125.stdout:1300: writing smithi12556470-25 from 2120442 to 2659266 tid 2 2021-03-24T21:58:52.246 INFO:tasks.rados.rados.0.smithi125.stdout:1300: writing smithi12556470-25 from 3278829 to 3783572 tid 3 2021-03-24T21:58:52.249 INFO:tasks.rados.rados.0.smithi125.stdout:1293: expect (ObjNum 444 snap 120 seq_num 444) 2021-03-24T21:58:52.339 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:52.348 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:52 smithi125 conmon[24927]: cluster 2021-03-24T21:58 2021-03-24T21:58:52.348 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:52 smithi125 conmon[24927]: :50.807372+0000 mgr.y (mgr.14144) 449 : cluster [DBG] pgmap v703: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 240 MiB data, 657 MiB used, 536 GiB / 536 GiB avail; 16 MiB/s rd, 7.7 MiB/s wr, 34 op/s; 237/1281 objects degraded (18.501%) 2021-03-24T21:58:52.348 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:52 smithi125 conmon[24927]: cluster 2021-03-24T21:58:51.210309+0000 mon.a (mon.0 2021-03-24T21:58:52.348 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:52 smithi125 conmon[24927]: ) 1121 : cluster [DBG] osdmap e361: 8 total, 7 up, 6 in 2021-03-24T21:58:52.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:52.349 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:52 smithi125 conmon[19936]: cluster 2021-03-24T21:58:50.807372 2021-03-24T21:58:52.350 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:52 smithi125 conmon[19936]: +0000 mgr.y (mgr.14144) 449 : cluster [DBG] pgmap v703: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 240 MiB data, 657 MiB used, 536 GiB / 536 GiB avail; 16 MiB/s rd, 7.7 MiB/s wr, 34 op/s; 237/1281 objects degraded (18.501%) 2021-03-24T21:58:52.350 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:52 smithi125 conmon[19936]: cluster 2021-03-24T21:58:51.210309+0000 mon.a ( 2021-03-24T21:58:52.350 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:52 smithi125 conmon[19936]: mon.0) 1121 : cluster [DBG] osdmap e361: 8 total, 7 up, 6 in 2021-03-24T21:58:52.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:58:52.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:52 smithi142 conmon[20125]: cluster 2021-03-24T21:58:50.807372+0000 mgr.y (mgr.14144) 449 : 2021-03-24T21:58:52.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:52 smithi142 conmon[20125]: cluster [DBG] pgmap v703: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 240 MiB data, 657 MiB used, 536 GiB / 536 GiB avail; 16 MiB/s rd, 7.7 MiB/s wr, 34 op/s; 237/1281 objects degraded (18.501%) 2021-03-24T21:58:52.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:52 smithi142 conmon[20125]: cluster 2021-03-24T21:58: 2021-03-24T21:58:52.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:52 smithi142 conmon[20125]: 51.210309+0000 mon.a (mon.0) 1121 : cluster [DBG] osdmap e361: 8 total, 7 up, 6 in 2021-03-24T21:58:52.407 INFO:tasks.rados.rados.0.smithi125.stdout:1295: expect (ObjNum 444 snap 120 seq_num 444) 2021-03-24T21:58:52.452 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:52.515 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:52.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:52.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:58:52.575 INFO:tasks.rados.rados.0.smithi125.stdout:1292: expect (ObjNum 438 snap 118 seq_num 438) 2021-03-24T21:58:52.621 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:52.674 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:52.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:52.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:52.751 INFO:tasks.rados.rados.0.smithi125.stdout:1291: finishing write tid 1 to smithi12556470-44 2021-03-24T21:58:52.751 INFO:tasks.rados.rados.0.smithi125.stdout:1294: finishing copy_from to smithi12556470-48 2021-03-24T21:58:52.751 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 48 v 432 (ObjNum 462 snap 129 seq_num 462) dirty exists 2021-03-24T21:58:52.752 INFO:tasks.rados.rados.0.smithi125.stdout:1294: finishing copy_from racing read to smithi12556470-48 2021-03-24T21:58:52.752 INFO:tasks.rados.rados.0.smithi125.stdout:1291: finishing write tid 2 to smithi12556470-44 2021-03-24T21:58:52.752 INFO:tasks.rados.rados.0.smithi125.stdout:1291: finishing write tid 3 to smithi12556470-44 2021-03-24T21:58:52.752 INFO:tasks.rados.rados.0.smithi125.stdout:1291: finishing write tid 4 to smithi12556470-44 2021-03-24T21:58:52.753 INFO:tasks.rados.rados.0.smithi125.stdout:1291: finishing write tid 5 to smithi12556470-44 2021-03-24T21:58:52.753 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 44 v 344 (ObjNum 470 snap 133 seq_num 470) dirty exists 2021-03-24T21:58:52.753 INFO:tasks.rados.rados.0.smithi125.stdout:1291: left oid 44 (ObjNum 470 snap 133 seq_num 470) 2021-03-24T21:58:52.753 INFO:tasks.rados.rados.0.smithi125.stdout:1297: finishing rollback tid 0 to smithi12556470-3 2021-03-24T21:58:52.754 INFO:tasks.rados.rados.0.smithi125.stdout:1297: finishing rollback tid 1 to smithi12556470-3 2021-03-24T21:58:52.754 INFO:tasks.rados.rados.0.smithi125.stdout:1297: finishing rollback tid 2 to smithi12556470-3 2021-03-24T21:58:52.754 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 3 v 144 (ObjNum 442 snap 119 seq_num 442) dirty exists 2021-03-24T21:58:52.754 INFO:tasks.rados.rados.0.smithi125.stdout:1298: finishing write tid 1 to smithi12556470-46 2021-03-24T21:58:52.754 INFO:tasks.rados.rados.0.smithi125.stdout:1298: finishing write tid 2 to smithi12556470-46 2021-03-24T21:58:52.755 INFO:tasks.rados.rados.0.smithi125.stdout:1298: finishing write tid 3 to smithi12556470-46 2021-03-24T21:58:52.755 INFO:tasks.rados.rados.0.smithi125.stdout:1298: finishing write tid 4 to smithi12556470-46 2021-03-24T21:58:52.755 INFO:tasks.rados.rados.0.smithi125.stdout:1298: finishing write tid 5 to smithi12556470-46 2021-03-24T21:58:52.755 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 46 v 334 (ObjNum 471 snap 133 seq_num 471) dirty exists 2021-03-24T21:58:52.756 INFO:tasks.rados.rados.0.smithi125.stdout:1298: left oid 46 (ObjNum 471 snap 133 seq_num 471) 2021-03-24T21:58:52.756 INFO:tasks.rados.rados.0.smithi125.stdout:1300: finishing write tid 1 to smithi12556470-25 2021-03-24T21:58:52.756 INFO:tasks.rados.rados.0.smithi125.stdout:1300: finishing write tid 2 to smithi12556470-25 2021-03-24T21:58:52.756 INFO:tasks.rados.rados.0.smithi125.stdout:1300: finishing write tid 3 to smithi12556470-25 2021-03-24T21:58:52.756 INFO:tasks.rados.rados.0.smithi125.stdout:1291: done (7 left) 2021-03-24T21:58:52.757 INFO:tasks.rados.rados.0.smithi125.stdout:1292: done (6 left) 2021-03-24T21:58:52.757 INFO:tasks.rados.rados.0.smithi125.stdout:1293: done (5 left) 2021-03-24T21:58:52.757 INFO:tasks.rados.rados.0.smithi125.stdout:1294: done (4 left) 2021-03-24T21:58:52.757 INFO:tasks.rados.rados.0.smithi125.stdout:1295: done (3 left) 2021-03-24T21:58:52.758 INFO:tasks.rados.rados.0.smithi125.stdout:1297: done (2 left) 2021-03-24T21:58:52.758 INFO:tasks.rados.rados.0.smithi125.stdout:1298: done (1 left) 2021-03-24T21:58:52.758 INFO:tasks.rados.rados.0.smithi125.stdout:1301: copy_from oid 5 from oid 38 current snap is 133 2021-03-24T21:58:52.758 INFO:tasks.rados.rados.0.smithi125.stdout:1300: finishing write tid 4 to smithi12556470-25 2021-03-24T21:58:52.758 INFO:tasks.rados.rados.0.smithi125.stdout:1300: finishing write tid 5 to smithi12556470-25 2021-03-24T21:58:52.759 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 25 v 290 (ObjNum 472 snap 133 seq_num 472) dirty exists 2021-03-24T21:58:52.759 INFO:tasks.rados.rados.0.smithi125.stdout:1300: left oid 25 (ObjNum 472 snap 133 seq_num 472) 2021-03-24T21:58:52.759 INFO:tasks.rados.rados.0.smithi125.stdout:1300: done (1 left) 2021-03-24T21:58:52.759 INFO:tasks.rados.rados.0.smithi125.stdout:1302: read oid 31 snap -1 2021-03-24T21:58:52.760 INFO:tasks.rados.rados.0.smithi125.stdout:1302: expect (ObjNum 420 snap 108 seq_num 420) 2021-03-24T21:58:52.760 INFO:tasks.rados.rados.0.smithi125.stdout:1303: copy_from oid 20 from oid 16 current snap is 133 2021-03-24T21:58:52.760 INFO:tasks.rados.rados.0.smithi125.stdout:1304: snap_create 2021-03-24T21:58:52.760 INFO:tasks.rados.rados.0.smithi125.stdout:1302: expect (ObjNum 420 snap 108 seq_num 420) 2021-03-24T21:58:52.780 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:52.825 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:52.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:52.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:58:52.934 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:52.992 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:52.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:52.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:58:53.021 INFO:tasks.rados.rados.0.smithi125.stdout:1301: finishing copy_from to smithi12556470-5 2021-03-24T21:58:53.022 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 5 v 49 (ObjNum 408 snap 106 seq_num 408) dirty exists 2021-03-24T21:58:53.022 INFO:tasks.rados.rados.0.smithi125.stdout:1301: finishing copy_from racing read to smithi12556470-5 2021-03-24T21:58:53.024 INFO:tasks.rados.rados.0.smithi125.stdout:1303: finishing copy_from to smithi12556470-20 2021-03-24T21:58:53.024 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 20 v 237 (ObjNum 453 snap 126 seq_num 453) dirty exists 2021-03-24T21:58:53.025 INFO:tasks.rados.rados.0.smithi125.stdout:1303: finishing copy_from racing read to smithi12556470-20 2021-03-24T21:58:53.097 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:53.162 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:53.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:53.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:58:53.219 INFO:tasks.rados.rados.0.smithi125.stdout:1301: done (3 left) 2021-03-24T21:58:53.220 INFO:tasks.rados.rados.0.smithi125.stdout:1302: done (2 left) 2021-03-24T21:58:53.220 INFO:tasks.rados.rados.0.smithi125.stdout:1303: done (1 left) 2021-03-24T21:58:53.220 INFO:tasks.rados.rados.0.smithi125.stdout:1304: done (0 left) 2021-03-24T21:58:53.220 INFO:tasks.rados.rados.0.smithi125.stdout:1305: snap_remove snap 131 2021-03-24T21:58:53.269 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:53.340 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:53.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:53.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:58:53.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:53 smithi142 conmon[20125]: cluster 2021-03-24T21:58: 2021-03-24T21:58:53.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:53 smithi142 conmon[20125]: 52.216051+0000 mon.a (mon.0) 1122 : cluster [DBG] osdmap e362: 8 total, 7 up, 6 in 2021-03-24T21:58:53.447 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:53.459 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:53 smithi125 conmon[19936]: cluster 2021-03-24T21:58:53.459 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:53 smithi125 conmon[19936]: 2021-03- 2021-03-24T21:58:53.459 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:53 smithi125 conmon[19936]: 24T21:58:52.216051+0000 mon.a (mon.0) 1122 : cluster [DBG] osdmap e362: 8 total, 7 up, 6 in 2021-03-24T21:58:53.460 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:53 smithi125 conmon[24927]: cluster 2021-03-24T21:58:52.216051+0000 mon.a (mon.0) 1122 : cluster [DBG] osdmap e362: 8 total, 7 up, 6 in 2021-03-24T21:58:53.491 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:53.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:53.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:53.596 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:53.616 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 7] out_osds: [6, 1] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 6, 7] 2021-03-24T21:58:53.617 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-24T21:58:53.617 INFO:tasks.thrashosds.thrasher:Growing pool .rgw.root 2021-03-24T21:58:53.617 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph pg dump --format=json 2021-03-24T21:58:53.648 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:53.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:53.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:53.753 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:53.758 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T21:58:53.783 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T21:58:53.808 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:53.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:53.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:58:53.817 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T21:58:53.837 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T21:58:53.862 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T21:58:53.888 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T21:58:53.913 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:53.928 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T21:58:53.928 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-24T21:58:53.928 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-24T21:58:53.929 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-24T21:58:53.929 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-24T21:58:53.929 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-24T21:58:53.929 INFO:teuthology.orchestra.run.smithi125.stderr:Error ENXIO: problem getting command descriptions from osd.5 2021-03-24T21:58:53.930 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-24T21:58:53.930 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-24T21:58:53.965 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:53.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:53.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:58:54.066 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-24T21:58:54.066 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T21:58:54.072 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:54.120 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:54.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:54.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:58:54.219 INFO:tasks.rados.rados.0.smithi125.stdout:1305: done (0 left) 2021-03-24T21:58:54.219 INFO:tasks.rados.rados.0.smithi125.stdout:1306: snap_remove snap 121 2021-03-24T21:58:54.225 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:54.226 INFO:teuthology.orchestra.run.smithi125.stderr:noscrub is unset 2021-03-24T21:58:54.291 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:54.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:54.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:58:54.400 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:54.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:54 smithi125 conmon[19936]: cluster 2021-03-24T21:58:52 2021-03-24T21:58:54.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:54 smithi125 conmon[19936]: .808208+0000 mgr.y (mgr.14144) 450 : cluster [DBG] pgmap v706: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 240 MiB data, 668 MiB used, 536 GiB / 536 GiB avail; 16 MiB/s rd, 6.5 MiB/s wr, 34 op/s; 237/1284 objects degraded (18.458%) 2021-03-24T21:58:54.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:54 smithi125 conmon[19936]: cluster 2021-03-24T21:58:54.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:54 smithi125 conmon[19936]: 2021-03-24T21:58:53.217407+0000 mon.a (mon.0) 1123 : cluster [DBG] osdmap e363: 8 total, 7 up, 6 in 2021-03-24T21:58:54.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:54 smithi125 conmon[19936]: audit 2021-03-24T21:58:53.655838+0000 mon.c (mon.2) 57 : audit [INF] from='client.? v1:172.21.15.125:0/3416419716' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T21:58:54.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:54 smithi125 conmon[19936]: audit 2021-03-24T21:58:53.656185+0000 mon.a (mon.0) 1124 : audit [INF] from='client.35123 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T21:58:54.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:54 smithi125 conmon[19936]: audit 2021-03-24T21:58:53.736987+0000 mon.c (mon.2) 58 : audit [DBG] from='client.? v1:172.21.15.125:0/4033924085' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:58:54.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:54 smithi125 conmon[19936]: cluster 2021-03-24T21:58:54.212146+0000 mon.a (mon.0) 1125 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:58:54.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:54 smithi125 conmon[19936]: audit 2021-03-24T21:58:54.217116+0000 mon.a (mon.0) 1126 : audit [INF] from='client.35123 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T21:58:54.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:54 smithi125 conmon[19936]: cluster 2021-03-24T21:58:54.217198+0000 mon.a (mon.0) 1127 : cluster [DBG] osdmap e364: 8 total, 7 up, 6 in 2021-03-24T21:58:54.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:54 smithi125 conmon[24927]: cluster 2021-03-24T21:58:54.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:54 smithi125 conmon[24927]: 2021-03-24T21:58:54.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:54 smithi125 conmon[24927]: 2021-03-24T21:58: 2021-03-24T21:58:54.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:54 smithi125 conmon[24927]: 52.808208+0000 mgr.y (mgr.14144) 450 : cluster [DBG] pgmap v706: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 240 MiB data, 668 MiB used, 536 GiB / 536 GiB avail; 16 MiB/s rd, 6.5 MiB/s wr, 34 op/s; 237/1284 objects degraded (18.458%) 2021-03-24T21:58:54.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:54 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:58:54.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:54 smithi125 conmon[24927]: 03-24T21:58:53.217407+0000 2021-03-24T21:58:54.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:54 smithi125 conmon[24927]: mon.a (mon.0) 1123 : cluster [DBG] 2021-03-24T21:58:54.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:54 smithi125 conmon[24927]: osdmap e363: 8 total, 7 up, 6 in 2021-03-24T21:58:54.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:54 smithi125 conmon[24927]: audit 2021-03-24 2021-03-24T21:58:54.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:54 smithi125 conmon[24927]: T21:58:53.655838+0000 mon.c ( 2021-03-24T21:58:54.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:54 smithi125 conmon[24927]: mon.2) 57 : audit [INF] from='client.? v1:172.21.15.125:0/3416419716' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T21:58:54.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:54 smithi125 conmon[24927]: audit 2021-03-24T21:58:54.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:54 smithi125 conmon[24927]: 2021-03-24T21:58:54.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:54 smithi125 conmon[24927]: 2021 2021-03-24T21:58:54.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:54 smithi125 conmon[24927]: - 2021-03-24T21:58:54.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:54 smithi125 conmon[24927]: 03 2021-03-24T21:58:54.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:54 smithi125 conmon[24927]: - 2021-03-24T21:58:54.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:54 smithi125 conmon[24927]: 24T21:58:53.656185+0000 2021-03-24T21:58:54.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:54 smithi125 conmon[24927]: mon.a (mon.0) 1124 : audit [INF] from='client.35123 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T21:58:54.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:54 smithi125 conmon[24927]: audit 2021-03-24T21:58:54.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:54 smithi125 conmon[24927]: 2021-03-24T21 2021-03-24T21:58:54.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:54 smithi125 conmon[24927]: :58:53.736987+0000 mon.c (mon. 2021-03-24T21:58:54.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:54 smithi125 conmon[24927]: 2) 58 : audit [DBG] from='client.? v1:172.21.15.125:0/4033924085' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:58:54.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:54 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:58:54.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:54 smithi125 conmon[24927]: 03-24T21:58:54.212146+0000 mon.a 2021-03-24T21:58:54.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:54 smithi125 conmon[24927]: (mon.0) 1125 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:58:54.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:54 smithi125 conmon[24927]: 2021-03-24T21:58:54.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:54 smithi125 conmon[24927]: audit 2021-03-24T21 2021-03-24T21:58:54.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:54 smithi125 conmon[24927]: :58:54.217116+0000 mon.a (mon.0 2021-03-24T21:58:54.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:54 smithi125 conmon[24927]: ) 1126 : audit [INF] from='client.35123 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T21:58:54.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:54 smithi125 conmon[24927]: cluster 2021-03-24T21:58:54.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:54 smithi125 conmon[24927]: 2021-03- 2021-03-24T21:58:54.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:54 smithi125 conmon[24927]: 24T21:58:54.217198+0000 mon.a (mon 2021-03-24T21:58:54.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:54 smithi125 conmon[24927]: .0) 1127 : cluster [DBG] osdmap e364: 8 total, 7 up, 6 in 2021-03-24T21:58:54.467 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:54.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:54.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:58:54.573 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:54.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: cluster 2021-03-24T21:58:54.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 2021-03-24T21:58:54.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 2021 2021-03-24T21:58:54.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: - 2021-03-24T21:58:54.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 03 2021-03-24T21:58:54.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: - 2021-03-24T21:58:54.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 24 2021-03-24T21:58:54.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: T 2021-03-24T21:58:54.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 21 2021-03-24T21:58:54.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: : 2021-03-24T21:58:54.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 58 2021-03-24T21:58:54.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: : 2021-03-24T21:58:54.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 52 2021-03-24T21:58:54.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: . 2021-03-24T21:58:54.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 808208 2021-03-24T21:58:54.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: +0000 2021-03-24T21:58:54.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 2021-03-24T21:58:54.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: mgr.y 2021-03-24T21:58:54.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: ( 2021-03-24T21:58:54.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: mgr 2021-03-24T21:58:54.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: . 2021-03-24T21:58:54.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 14144 2021-03-24T21:58:54.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: ) 2021-03-24T21:58:54.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 450 2021-03-24T21:58:54.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: : 2021-03-24T21:58:54.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: cluster 2021-03-24T21:58:54.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 2021-03-24T21:58:54.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: [DBG] 2021-03-24T21:58:54.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 2021-03-24T21:58:54.625 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: pgmap v706: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 240 MiB data, 668 MiB used, 536 GiB / 536 GiB avail; 16 MiB/s rd, 6.5 MiB/s wr, 34 op/s; 237/1284 objects degraded (18.458%) 2021-03-24T21:58:54.625 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 2021-03-24T21:58:54.625 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: cluster 2021-03-24T21:58:54.625 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 2021-03-24T21:58:54.626 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 2021 2021-03-24T21:58:54.626 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: - 2021-03-24T21:58:54.626 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 03 2021-03-24T21:58:54.626 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: - 2021-03-24T21:58:54.626 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 24 2021-03-24T21:58:54.627 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: T 2021-03-24T21:58:54.627 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 21 2021-03-24T21:58:54.627 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: : 2021-03-24T21:58:54.627 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 58 2021-03-24T21:58:54.628 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: : 2021-03-24T21:58:54.628 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 53 2021-03-24T21:58:54.628 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: . 2021-03-24T21:58:54.628 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 217407 2021-03-24T21:58:54.628 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: +0000 2021-03-24T21:58:54.629 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 2021-03-24T21:58:54.629 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: mon.a 2021-03-24T21:58:54.629 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: ( 2021-03-24T21:58:54.629 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: mon 2021-03-24T21:58:54.629 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: . 2021-03-24T21:58:54.630 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 0 2021-03-24T21:58:54.630 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: ) 2021-03-24T21:58:54.630 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 1123 2021-03-24T21:58:54.630 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: : 2021-03-24T21:58:54.631 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: cluster 2021-03-24T21:58:54.631 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 2021-03-24T21:58:54.631 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: [DBG] 2021-03-24T21:58:54.631 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 2021-03-24T21:58:54.631 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: osdmap e363: 8 total, 7 up, 6 in 2021-03-24T21:58:54.632 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 2021-03-24T21:58:54.632 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: audit 2021-03-24T21:58:54.632 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 2021-03-24T21:58:54.632 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 2021 2021-03-24T21:58:54.632 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: - 2021-03-24T21:58:54.633 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 03 2021-03-24T21:58:54.633 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: - 2021-03-24T21:58:54.633 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 24 2021-03-24T21:58:54.633 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: T 2021-03-24T21:58:54.634 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 21 2021-03-24T21:58:54.634 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: : 2021-03-24T21:58:54.634 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 58 2021-03-24T21:58:54.634 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: : 2021-03-24T21:58:54.634 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 53 2021-03-24T21:58:54.635 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: . 2021-03-24T21:58:54.635 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 655838 2021-03-24T21:58:54.635 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: +0000 2021-03-24T21:58:54.635 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 2021-03-24T21:58:54.635 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: mon.c 2021-03-24T21:58:54.636 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: ( 2021-03-24T21:58:54.636 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: mon 2021-03-24T21:58:54.636 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: . 2021-03-24T21:58:54.636 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 2 2021-03-24T21:58:54.637 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: ) 2021-03-24T21:58:54.637 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 57 2021-03-24T21:58:54.638 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: : 2021-03-24T21:58:54.638 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: audit 2021-03-24T21:58:54.639 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 2021-03-24T21:58:54.639 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: [INF] 2021-03-24T21:58:54.639 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 2021-03-24T21:58:54.639 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: from='client.? v1:172.21.15.125:0/3416419716' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T21:58:54.640 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 2021-03-24T21:58:54.640 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: audit 2021-03-24T21:58:54.640 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 2021-03-24T21:58:54.640 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 2021 2021-03-24T21:58:54.640 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: - 2021-03-24T21:58:54.641 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 03 2021-03-24T21:58:54.641 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: - 2021-03-24T21:58:54.641 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 24 2021-03-24T21:58:54.641 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: T 2021-03-24T21:58:54.642 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 21 2021-03-24T21:58:54.642 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: : 2021-03-24T21:58:54.642 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 58 2021-03-24T21:58:54.642 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: : 2021-03-24T21:58:54.642 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 53 2021-03-24T21:58:54.643 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: . 2021-03-24T21:58:54.643 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 656185 2021-03-24T21:58:54.643 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: +0000 2021-03-24T21:58:54.643 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 2021-03-24T21:58:54.644 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: mon.a 2021-03-24T21:58:54.644 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: ( 2021-03-24T21:58:54.644 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: mon 2021-03-24T21:58:54.644 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: . 2021-03-24T21:58:54.644 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 0 2021-03-24T21:58:54.645 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: ) 2021-03-24T21:58:54.645 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 1124 2021-03-24T21:58:54.645 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: : 2021-03-24T21:58:54.645 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: audit 2021-03-24T21:58:54.646 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 2021-03-24T21:58:54.646 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: [INF] 2021-03-24T21:58:54.646 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 2021-03-24T21:58:54.646 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: from='client.35123 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T21:58:54.646 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 2021-03-24T21:58:54.647 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: audit 2021-03-24T21:58:54.647 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 2021-03-24T21:58:54.647 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 2021 2021-03-24T21:58:54.647 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: - 2021-03-24T21:58:54.648 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 03 2021-03-24T21:58:54.648 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: - 2021-03-24T21:58:54.648 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 24 2021-03-24T21:58:54.648 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: T 2021-03-24T21:58:54.648 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 21 2021-03-24T21:58:54.649 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: : 2021-03-24T21:58:54.649 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 58 2021-03-24T21:58:54.649 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: : 2021-03-24T21:58:54.649 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 53 2021-03-24T21:58:54.649 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: . 2021-03-24T21:58:54.650 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 736987 2021-03-24T21:58:54.650 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: +0000 2021-03-24T21:58:54.650 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 2021-03-24T21:58:54.650 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: mon.c 2021-03-24T21:58:54.651 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: ( 2021-03-24T21:58:54.651 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: mon 2021-03-24T21:58:54.651 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: . 2021-03-24T21:58:54.651 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 2 2021-03-24T21:58:54.651 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: ) 2021-03-24T21:58:54.652 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 58 2021-03-24T21:58:54.652 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: : 2021-03-24T21:58:54.652 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: audit 2021-03-24T21:58:54.652 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 2021-03-24T21:58:54.653 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: [DBG] 2021-03-24T21:58:54.653 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 2021-03-24T21:58:54.653 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: from='client.? v1:172.21.15.125:0/4033924085' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:58:54.653 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 2021-03-24T21:58:54.653 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: cluster 2021-03-24T21:58:54.654 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 2021-03-24T21:58:54.654 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 2021 2021-03-24T21:58:54.654 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: - 2021-03-24T21:58:54.654 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 03 2021-03-24T21:58:54.655 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: - 2021-03-24T21:58:54.655 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 24 2021-03-24T21:58:54.655 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: T 2021-03-24T21:58:54.655 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 21 2021-03-24T21:58:54.655 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: : 2021-03-24T21:58:54.656 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 58 2021-03-24T21:58:54.656 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: : 2021-03-24T21:58:54.656 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 54 2021-03-24T21:58:54.656 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: . 2021-03-24T21:58:54.656 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 212146 2021-03-24T21:58:54.657 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: +0000 2021-03-24T21:58:54.657 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 2021-03-24T21:58:54.657 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: mon.a 2021-03-24T21:58:54.657 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: ( 2021-03-24T21:58:54.658 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: mon 2021-03-24T21:58:54.658 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: . 2021-03-24T21:58:54.658 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 0 2021-03-24T21:58:54.658 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: ) 2021-03-24T21:58:54.658 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 1125 2021-03-24T21:58:54.659 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: : 2021-03-24T21:58:54.659 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: cluster 2021-03-24T21:58:54.659 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 2021-03-24T21:58:54.659 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: [WRN] 2021-03-24T21:58:54.659 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 2021-03-24T21:58:54.660 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:58:54.660 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 2021-03-24T21:58:54.660 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: audit 2021-03-24T21:58:54.660 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 2021-03-24T21:58:54.660 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 2021 2021-03-24T21:58:54.661 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: - 2021-03-24T21:58:54.661 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 03 2021-03-24T21:58:54.661 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: - 2021-03-24T21:58:54.661 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 24 2021-03-24T21:58:54.661 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: T 2021-03-24T21:58:54.662 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 21 2021-03-24T21:58:54.662 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: : 2021-03-24T21:58:54.662 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 58 2021-03-24T21:58:54.662 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: : 2021-03-24T21:58:54.662 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 54 2021-03-24T21:58:54.663 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: . 2021-03-24T21:58:54.663 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 217116 2021-03-24T21:58:54.663 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: +0000 2021-03-24T21:58:54.663 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 2021-03-24T21:58:54.664 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: mon.a 2021-03-24T21:58:54.664 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: ( 2021-03-24T21:58:54.664 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: mon 2021-03-24T21:58:54.664 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: . 2021-03-24T21:58:54.664 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 0 2021-03-24T21:58:54.665 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: ) 2021-03-24T21:58:54.665 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 1126 2021-03-24T21:58:54.665 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: : 2021-03-24T21:58:54.665 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: audit 2021-03-24T21:58:54.665 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 2021-03-24T21:58:54.666 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: [INF] 2021-03-24T21:58:54.666 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 2021-03-24T21:58:54.666 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: from='client.35123 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T21:58:54.666 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 2021-03-24T21:58:54.667 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: cluster 2021-03-24T21:58:54.667 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 2021-03-24T21:58:54.667 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 2021 2021-03-24T21:58:54.667 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: - 2021-03-24T21:58:54.667 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 03 2021-03-24T21:58:54.668 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: - 2021-03-24T21:58:54.668 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 24 2021-03-24T21:58:54.668 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: T 2021-03-24T21:58:54.668 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 21 2021-03-24T21:58:54.669 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: : 2021-03-24T21:58:54.669 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 58 2021-03-24T21:58:54.669 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: : 2021-03-24T21:58:54.670 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 54 2021-03-24T21:58:54.670 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: . 2021-03-24T21:58:54.670 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 217198 2021-03-24T21:58:54.670 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: +0000 2021-03-24T21:58:54.670 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 2021-03-24T21:58:54.671 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: mon.a 2021-03-24T21:58:54.671 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: ( 2021-03-24T21:58:54.671 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: mon 2021-03-24T21:58:54.671 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: . 2021-03-24T21:58:54.672 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 0 2021-03-24T21:58:54.672 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: ) 2021-03-24T21:58:54.672 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 1127 2021-03-24T21:58:54.672 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: : 2021-03-24T21:58:54.672 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: cluster 2021-03-24T21:58:54.673 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 2021-03-24T21:58:54.673 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: [DBG] 2021-03-24T21:58:54.673 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 2021-03-24T21:58:54.673 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: osdmap e364: 8 total, 7 up, 6 in 2021-03-24T21:58:54.673 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:54 smithi142 conmon[20125]: 2021-03-24T21:58:54.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:54.674 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:54.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:58:54.776 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:54.839 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:54.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:54.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:54.946 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:55.005 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:55.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:55.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:58:55.109 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:55.165 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:55.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:55.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:58:55.233 INFO:tasks.rados.rados.0.smithi125.stdout:1306: done (0 left) 2021-03-24T21:58:55.234 INFO:tasks.rados.rados.0.smithi125.stdout:1307: write oid 1 current snap is 134 2021-03-24T21:58:55.234 INFO:tasks.rados.rados.0.smithi125.stdout:1307: seq_num 473 ranges {749711=406663,1778924=682482,3189899=716015} 2021-03-24T21:58:55.235 INFO:tasks.rados.rados.0.smithi125.stdout:1307: writing smithi12556470-1 from 749711 to 1156374 tid 1 2021-03-24T21:58:55.244 INFO:tasks.rados.rados.0.smithi125.stdout:1307: writing smithi12556470-1 from 1778924 to 2461406 tid 2 2021-03-24T21:58:55.254 INFO:tasks.rados.rados.0.smithi125.stdout:1307: writing smithi12556470-1 from 3189899 to 3905914 tid 3 2021-03-24T21:58:55.254 INFO:tasks.rados.rados.0.smithi125.stdout:1308: snap_create 2021-03-24T21:58:55.261 INFO:tasks.rados.rados.0.smithi125.stdout:1307: finishing write tid 1 to smithi12556470-1 2021-03-24T21:58:55.267 INFO:tasks.rados.rados.0.smithi125.stdout:1307: finishing write tid 2 to smithi12556470-1 2021-03-24T21:58:55.268 INFO:tasks.rados.rados.0.smithi125.stdout:1307: finishing write tid 3 to smithi12556470-1 2021-03-24T21:58:55.268 INFO:tasks.rados.rados.0.smithi125.stdout:1307: finishing write tid 4 to smithi12556470-1 2021-03-24T21:58:55.268 INFO:tasks.rados.rados.0.smithi125.stdout:1307: finishing write tid 5 to smithi12556470-1 2021-03-24T21:58:55.268 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 1 v 436 (ObjNum 473 snap 134 seq_num 473) dirty exists 2021-03-24T21:58:55.269 INFO:tasks.rados.rados.0.smithi125.stdout:1307: left oid 1 (ObjNum 473 snap 134 seq_num 473) 2021-03-24T21:58:55.270 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:55.365 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:55.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:55.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:55.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:55 smithi125 conmon[24927]: cluster 2021-03-24T21:58:54.590467+0000 2021-03-24T21:58:55.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:55 smithi125 conmon[24927]: osd.2 (osd.2) 66 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:58:55.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:55 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:58:55.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:55 smithi125 conmon[24927]: -03-24T21:58:54.590877+0000 osd.2 (osd.2) 67 : cluster [DBG] 2021-03-24T21:58:55.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:55 smithi125 conmon[24927]: purged_snaps scrub ok 2021-03-24T21:58:55.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:55 smithi125 conmon[24927]: cluster 2021-03-24T21:58:55.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:55 smithi125 conmon[24927]: 2021-03-24T21:58:54.693589+0000 osd.2 (osd.2) 2021-03-24T21:58:55.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:55 smithi125 conmon[24927]: 68 : cluster [DBG] 4.4 scrub ok 2021-03-24T21:58:55.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:55 smithi125 conmon[24927]: cluster 2021-03-24T21:58 2021-03-24T21:58:55.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:55 smithi125 conmon[24927]: :55.189207+0000 mon.a (mon.0) 1128 2021-03-24T21:58:55.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:55 smithi125 conmon[24927]: : cluster [WRN] Health check update: Degraded data redundancy: 237/1284 objects degraded (18.458%), 40 pgs degraded (PG_DEGRADED) 2021-03-24T21:58:55.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:55 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:58:55.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:55 smithi125 conmon[24927]: :58:55.219472+0000 mon.a (mon.0 2021-03-24T21:58:55.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:55 smithi125 conmon[24927]: ) 1129 : cluster [DBG] osdmap e365: 8 total, 7 up, 6 in 2021-03-24T21:58:55.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:55 smithi125 conmon[19936]: cluster 2021-03-24T21:58:55.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:55 smithi125 conmon[19936]: 2021-03-24T21:58:55.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:55 smithi125 conmon[19936]: 2021 2021-03-24T21:58:55.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:55 smithi125 conmon[19936]: - 2021-03-24T21:58:55.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:55 smithi125 conmon[19936]: 03 2021-03-24T21:58:55.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:55 smithi125 conmon[19936]: - 2021-03-24T21:58:55.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:55 smithi125 conmon[19936]: 24 2021-03-24T21:58:55.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:55 smithi125 conmon[19936]: T 2021-03-24T21:58:55.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:55 smithi125 conmon[19936]: 21 2021-03-24T21:58:55.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:55 smithi125 conmon[19936]: : 2021-03-24T21:58:55.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:55 smithi125 conmon[19936]: 58 2021-03-24T21:58:55.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:55 smithi125 conmon[19936]: : 2021-03-24T21:58:55.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:55 smithi125 conmon[19936]: 54 2021-03-24T21:58:55.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:55 smithi125 conmon[19936]: . 2021-03-24T21:58:55.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:55 smithi125 conmon[19936]: 590467 2021-03-24T21:58:55.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:55 smithi125 conmon[19936]: +0000 2021-03-24T21:58:55.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:55 smithi125 conmon[19936]: 2021-03-24T21:58:55.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:55 smithi125 conmon[19936]: osd.2 2021-03-24T21:58:55.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:55 smithi125 conmon[19936]: ( 2021-03-24T21:58:55.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:55 smithi125 conmon[19936]: osd 2021-03-24T21:58:55.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:55 smithi125 conmon[19936]: . 2021-03-24T21:58:55.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:55 smithi125 conmon[19936]: 2 2021-03-24T21:58:55.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:55 smithi125 conmon[19936]: ) 2021-03-24T21:58:55.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:55 smithi125 conmon[19936]: 66 2021-03-24T21:58:55.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:55 smithi125 conmon[19936]: : 2021-03-24T21:58:55.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:55 smithi125 conmon[19936]: cluster 2021-03-24T21:58:55.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:55 smithi125 conmon[19936]: 2021-03-24T21:58:55.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:55 smithi125 conmon[19936]: [DBG] 2021-03-24T21:58:55.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:55 smithi125 conmon[19936]: 2021-03-24T21:58:55.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:55 smithi125 conmon[19936]: purged_snaps scrub starts 2021-03-24T21:58:55.421 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:55 smithi125 conmon[19936]: 2021-03-24T21:58:55.421 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:55 smithi125 conmon[19936]: cluster 2021-03-24T21:58:55.421 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:55 smithi125 conmon[19936]: 2021-03-24T21:58:55.421 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:55 smithi125 conmon[19936]: 2021-03-24T21:58:54.590877+0000 osd.2 (osd.2) 67 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:58:55.422 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:55 smithi125 conmon[19936]: cluster 2021-03-24T21:58:54.693589+0000 osd.2 (osd.2) 68 : cluster [DBG] 4.4 scrub ok 2021-03-24T21:58:55.422 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:55 smithi125 conmon[19936]: cluster 2021-03-24T21:58:55.189207+0000 mon.a (mon.0) 1128 : cluster [WRN] Health check update: Degraded data redundancy: 237/1284 objects degraded (18.458%), 40 pgs degraded (PG_DEGRADED) 2021-03-24T21:58:55.422 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:55 smithi125 conmon[19936]: cluster 2021-03-24T21:58:55.219472+0000 mon.a (mon.0) 1129 : cluster [DBG] osdmap e365: 8 total, 7 up, 6 in 2021-03-24T21:58:55.472 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:55.480 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: cluster 2021-03-24T21:58:55.481 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 2021-03-24T21:58:55.481 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 2021 2021-03-24T21:58:55.481 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: - 2021-03-24T21:58:55.481 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 03 2021-03-24T21:58:55.481 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: - 2021-03-24T21:58:55.482 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 24 2021-03-24T21:58:55.482 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: T 2021-03-24T21:58:55.482 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 21 2021-03-24T21:58:55.482 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: : 2021-03-24T21:58:55.483 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 58 2021-03-24T21:58:55.483 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: : 2021-03-24T21:58:55.483 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 54 2021-03-24T21:58:55.483 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: . 2021-03-24T21:58:55.483 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 590467 2021-03-24T21:58:55.484 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: +0000 2021-03-24T21:58:55.484 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 2021-03-24T21:58:55.484 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: osd.2 2021-03-24T21:58:55.484 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: ( 2021-03-24T21:58:55.485 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: osd 2021-03-24T21:58:55.485 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: . 2021-03-24T21:58:55.485 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 2 2021-03-24T21:58:55.485 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: ) 2021-03-24T21:58:55.485 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 66 2021-03-24T21:58:55.486 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: : 2021-03-24T21:58:55.486 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: cluster 2021-03-24T21:58:55.486 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 2021-03-24T21:58:55.486 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: [DBG] 2021-03-24T21:58:55.487 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 2021-03-24T21:58:55.487 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: purged_snaps scrub starts 2021-03-24T21:58:55.487 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 2021-03-24T21:58:55.487 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: cluster 2021-03-24T21:58:55.487 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 2021-03-24T21:58:55.488 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 2021 2021-03-24T21:58:55.488 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: - 2021-03-24T21:58:55.488 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 03 2021-03-24T21:58:55.488 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: - 2021-03-24T21:58:55.489 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 24 2021-03-24T21:58:55.489 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: T 2021-03-24T21:58:55.489 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 21 2021-03-24T21:58:55.489 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: : 2021-03-24T21:58:55.489 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 58 2021-03-24T21:58:55.490 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: : 2021-03-24T21:58:55.490 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 54 2021-03-24T21:58:55.490 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: . 2021-03-24T21:58:55.490 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 590877 2021-03-24T21:58:55.491 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: +0000 2021-03-24T21:58:55.491 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 2021-03-24T21:58:55.491 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: osd.2 2021-03-24T21:58:55.491 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: ( 2021-03-24T21:58:55.491 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: osd 2021-03-24T21:58:55.492 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: . 2021-03-24T21:58:55.492 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 2 2021-03-24T21:58:55.492 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: ) 2021-03-24T21:58:55.492 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 67 2021-03-24T21:58:55.493 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: : 2021-03-24T21:58:55.493 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: cluster 2021-03-24T21:58:55.493 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 2021-03-24T21:58:55.493 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: [DBG] 2021-03-24T21:58:55.494 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 2021-03-24T21:58:55.494 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: purged_snaps scrub ok 2021-03-24T21:58:55.494 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 2021-03-24T21:58:55.494 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: cluster 2021-03-24T21:58:55.495 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 2021-03-24T21:58:55.495 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 2021 2021-03-24T21:58:55.495 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: - 2021-03-24T21:58:55.495 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 03 2021-03-24T21:58:55.495 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: - 2021-03-24T21:58:55.496 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 24 2021-03-24T21:58:55.496 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: T 2021-03-24T21:58:55.496 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 21 2021-03-24T21:58:55.496 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: : 2021-03-24T21:58:55.497 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 58 2021-03-24T21:58:55.497 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: : 2021-03-24T21:58:55.497 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 54 2021-03-24T21:58:55.497 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: . 2021-03-24T21:58:55.497 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 693589 2021-03-24T21:58:55.498 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: +0000 2021-03-24T21:58:55.498 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 2021-03-24T21:58:55.498 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: osd.2 2021-03-24T21:58:55.498 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: ( 2021-03-24T21:58:55.499 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: osd 2021-03-24T21:58:55.499 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: . 2021-03-24T21:58:55.499 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 2 2021-03-24T21:58:55.499 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: ) 2021-03-24T21:58:55.499 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 68 2021-03-24T21:58:55.500 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: : 2021-03-24T21:58:55.500 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: cluster 2021-03-24T21:58:55.500 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 2021-03-24T21:58:55.500 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: [DBG] 2021-03-24T21:58:55.500 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 2021-03-24T21:58:55.501 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 4.4 scrub ok 2021-03-24T21:58:55.501 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 2021-03-24T21:58:55.501 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: cluster 2021-03-24T21:58:55.502 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 2021-03-24T21:58:55.502 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 2021 2021-03-24T21:58:55.502 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: - 2021-03-24T21:58:55.502 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 03 2021-03-24T21:58:55.503 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: - 2021-03-24T21:58:55.503 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 24 2021-03-24T21:58:55.503 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: T 2021-03-24T21:58:55.503 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 21 2021-03-24T21:58:55.503 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: : 2021-03-24T21:58:55.504 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 58 2021-03-24T21:58:55.504 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: : 2021-03-24T21:58:55.504 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 55 2021-03-24T21:58:55.504 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: . 2021-03-24T21:58:55.505 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 189207 2021-03-24T21:58:55.505 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: +0000 2021-03-24T21:58:55.505 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 2021-03-24T21:58:55.505 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: mon.a 2021-03-24T21:58:55.506 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: ( 2021-03-24T21:58:55.506 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: mon 2021-03-24T21:58:55.506 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: . 2021-03-24T21:58:55.506 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 0 2021-03-24T21:58:55.507 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: ) 2021-03-24T21:58:55.507 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 1128 2021-03-24T21:58:55.507 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: : 2021-03-24T21:58:55.507 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: cluster 2021-03-24T21:58:55.508 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 2021-03-24T21:58:55.508 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: [WRN] 2021-03-24T21:58:55.508 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 2021-03-24T21:58:55.508 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: Health check update: Degraded data redundancy: 237/1284 objects degraded (18.458%), 40 pgs degraded (PG_DEGRADED) 2021-03-24T21:58:55.508 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 2021-03-24T21:58:55.509 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: cluster 2021-03-24T21:58:55.509 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 2021-03-24T21:58:55.509 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 2021 2021-03-24T21:58:55.509 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: - 2021-03-24T21:58:55.509 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 03 2021-03-24T21:58:55.510 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: - 2021-03-24T21:58:55.510 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 24 2021-03-24T21:58:55.510 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: T 2021-03-24T21:58:55.510 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 21 2021-03-24T21:58:55.511 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: : 2021-03-24T21:58:55.511 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 58 2021-03-24T21:58:55.511 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: : 2021-03-24T21:58:55.511 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 55 2021-03-24T21:58:55.511 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: . 2021-03-24T21:58:55.512 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 219472 2021-03-24T21:58:55.512 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: +0000 2021-03-24T21:58:55.512 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 2021-03-24T21:58:55.512 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: mon.a 2021-03-24T21:58:55.513 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: ( 2021-03-24T21:58:55.513 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: mon 2021-03-24T21:58:55.513 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: . 2021-03-24T21:58:55.513 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 0 2021-03-24T21:58:55.513 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: ) 2021-03-24T21:58:55.514 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 1129 2021-03-24T21:58:55.514 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: : 2021-03-24T21:58:55.514 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: cluster 2021-03-24T21:58:55.514 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 2021-03-24T21:58:55.515 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: [DBG] 2021-03-24T21:58:55.515 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 2021-03-24T21:58:55.515 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: osdmap e365: 8 total, 7 up, 6 in 2021-03-24T21:58:55.515 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:55 smithi142 conmon[20125]: 2021-03-24T21:58:55.516 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:55.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:55.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:55.618 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:55.672 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:55.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:55.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:55.779 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:55.829 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:55.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:55.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:58:55.937 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:55.999 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:56.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:56.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:58:56.104 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:56.166 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:56.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:56.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:58:56.257 INFO:tasks.rados.rados.0.smithi125.stdout:1307: done (1 left) 2021-03-24T21:58:56.257 INFO:tasks.rados.rados.0.smithi125.stdout:1308: done (0 left) 2021-03-24T21:58:56.257 INFO:tasks.rados.rados.0.smithi125.stdout:1309: read oid 7 snap -1 2021-03-24T21:58:56.258 INFO:tasks.rados.rados.0.smithi125.stdout:1309: expect (ObjNum 404 snap 104 seq_num 404) 2021-03-24T21:58:56.258 INFO:tasks.rados.rados.0.smithi125.stdout:1310: read oid 14 snap -1 2021-03-24T21:58:56.258 INFO:tasks.rados.rados.0.smithi125.stdout:1310: expect (ObjNum 455 snap 128 seq_num 455) 2021-03-24T21:58:56.258 INFO:tasks.rados.rados.0.smithi125.stdout:1311: read oid 3 snap -1 2021-03-24T21:58:56.258 INFO:tasks.rados.rados.0.smithi125.stdout:1311: expect (ObjNum 442 snap 119 seq_num 442) 2021-03-24T21:58:56.259 INFO:tasks.rados.rados.0.smithi125.stdout:1312: rollback oid 8 current snap is 135 2021-03-24T21:58:56.259 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 8 to 128 2021-03-24T21:58:56.259 INFO:tasks.rados.rados.0.smithi125.stdout:1313: snap_create 2021-03-24T21:58:56.265 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:56 smithi125 conmon[19936]: cluster 2021-03-24T21:58:54.809045+0000 mgr.y (mgr.14144) 451 : cluster [DBG] pgmap v709: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 259 MiB data, 690 MiB used, 536 GiB / 536 GiB avail; 14 MiB/s rd, 4.2 MiB/s wr, 20 op/s; 240/1299 objects degraded (18.476%) 2021-03-24T21:58:56.265 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:56 smithi125 conmon[19936]: cluster 2021-03-24T21:58:54.826526+0000 osd.7 (osd.7) 81 : cluster [DBG] 6.3 scrub ok 2021-03-24T21:58:56.265 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:56 smithi125 conmon[24927]: cluster 2021-03-24T21:58:56.266 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:56 smithi125 conmon[24927]: 2021-03-24T21:58:54.809045 2021-03-24T21:58:56.266 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:56 smithi125 conmon[24927]: +0000 mgr.y (mgr.14144) 451 : cluster [DBG] pgmap v709: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 259 MiB data, 690 MiB used, 536 GiB / 536 GiB avail; 14 MiB/s rd, 4.2 MiB/s wr, 20 op/s; 240/1299 objects degraded (18.476%) 2021-03-24T21:58:56.266 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:56 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:58:56.266 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:56 smithi125 conmon[24927]: -03-24T21:58:54.826526+0000 osd.7 (osd.7) 81 : cluster [DBG] 2021-03-24T21:58:56.267 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:56 smithi125 conmon[24927]: 6.3 scrub ok 2021-03-24T21:58:56.271 INFO:tasks.rados.rados.0.smithi125.stdout:1309: expect (ObjNum 404 snap 104 seq_num 404) 2021-03-24T21:58:56.272 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:56.330 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:56.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:56.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:56.403 INFO:tasks.rados.rados.0.smithi125.stdout:1311: expect (ObjNum 442 snap 119 seq_num 442) 2021-03-24T21:58:56.435 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:56.493 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:56.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:56.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:56.549 INFO:tasks.rados.rados.0.smithi125.stdout:1310: expect (ObjNum 455 snap 128 seq_num 455) 2021-03-24T21:58:56.599 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:56.609 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:56 smithi142 conmon[20125]: cluster 2021-03-24T21:58:56.610 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:56 smithi142 conmon[20125]: 2021-03-24T21:58:56.610 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:56 smithi142 conmon[20125]: 2021 2021-03-24T21:58:56.610 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:56 smithi142 conmon[20125]: - 2021-03-24T21:58:56.611 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:56 smithi142 conmon[20125]: 03 2021-03-24T21:58:56.611 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:56 smithi142 conmon[20125]: - 2021-03-24T21:58:56.611 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:56 smithi142 conmon[20125]: 24 2021-03-24T21:58:56.611 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:56 smithi142 conmon[20125]: T 2021-03-24T21:58:56.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:56 smithi142 conmon[20125]: 21 2021-03-24T21:58:56.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:56 smithi142 conmon[20125]: :58:54.809045+0000 mgr.y ( 2021-03-24T21:58:56.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:56 smithi142 conmon[20125]: mgr.14144) 451 : cluster [DBG] pgmap v709: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 259 MiB data, 690 MiB used, 536 GiB / 536 GiB avail; 14 MiB/s rd, 4.2 MiB/s wr, 20 op/s; 240/1299 objects degraded (18.476%) 2021-03-24T21:58:56.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:56 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:58:56.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:56 smithi142 conmon[20125]: 21:58:54.826526+0000 osd.7 (osd. 2021-03-24T21:58:56.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:56 smithi142 conmon[20125]: 7) 81 : cluster [DBG] 6.3 scrub ok 2021-03-24T21:58:56.645 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:56.646 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-03-24T21:58:56.647 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-03-24T21:58:56.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:56.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:56.719 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.1 dump_historic_ops 2021-03-24T21:58:56.724 INFO:tasks.rados.rados.0.smithi125.stdout:1312: finishing rollback tid 0 to smithi12556470-8 2021-03-24T21:58:56.724 INFO:tasks.rados.rados.0.smithi125.stdout:1312: finishing rollback tid 1 to smithi12556470-8 2021-03-24T21:58:56.724 INFO:tasks.rados.rados.0.smithi125.stdout:1312: finishing rollback tid 2 to smithi12556470-8 2021-03-24T21:58:56.725 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 8 v 79 (ObjNum 414 snap 108 seq_num 414) dirty exists 2021-03-24T21:58:56.750 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:56.805 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:56.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:56.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:58:56.811 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:58:56.911 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:56.959 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:56.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:56.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:58:57.004 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:58:57.064 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:57.113 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:57.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:57.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:57.256 INFO:tasks.rados.rados.0.smithi125.stdout:1309: done (4 left) 2021-03-24T21:58:57.257 INFO:tasks.rados.rados.0.smithi125.stdout:1310: done (3 left) 2021-03-24T21:58:57.257 INFO:tasks.rados.rados.0.smithi125.stdout:1311: done (2 left) 2021-03-24T21:58:57.257 INFO:tasks.rados.rados.0.smithi125.stdout:1312: done (1 left) 2021-03-24T21:58:57.257 INFO:tasks.rados.rados.0.smithi125.stdout:1313: done (0 left) 2021-03-24T21:58:57.258 INFO:tasks.rados.rados.0.smithi125.stdout:1314: write (excl) oid 47 current snap is 136 2021-03-24T21:58:57.258 INFO:tasks.rados.rados.0.smithi125.stdout:1314: seq_num 474 ranges {766518=481912,1837406=542493,2933675=527287,3876584=89934} 2021-03-24T21:58:57.264 INFO:tasks.rados.rados.0.smithi125.stdout:1314: writing smithi12556470-47 from 766518 to 1248430 tid 1 2021-03-24T21:58:57.271 INFO:tasks.rados.rados.0.smithi125.stdout:1314: writing smithi12556470-47 from 1837406 to 2379899 tid 2 2021-03-24T21:58:57.277 INFO:tasks.rados.rados.0.smithi125.stdout:1314: writing smithi12556470-47 from 2933675 to 3460962 tid 3 2021-03-24T21:58:57.280 INFO:tasks.rados.rados.0.smithi125.stdout:1314: writing smithi12556470-47 from 3876584 to 3966518 tid 4 2021-03-24T21:58:57.280 INFO:tasks.rados.rados.0.smithi125.stdout:1315: delete oid 38 current snap is 136 2021-03-24T21:58:57.280 INFO:tasks.rados.rados.0.smithi125.stdout:1314: finishing write tid 1 to smithi12556470-47 2021-03-24T21:58:57.281 INFO:tasks.rados.rados.0.smithi125.stdout:1314: finishing write tid 2 to smithi12556470-47 2021-03-24T21:58:57.281 INFO:tasks.rados.rados.0.smithi125.stdout:1314: finishing write tid 3 to smithi12556470-47 2021-03-24T21:58:57.283 INFO:tasks.rados.rados.0.smithi125.stdout:1314: finishing write tid 4 to smithi12556470-47 2021-03-24T21:58:57.283 INFO:tasks.rados.rados.0.smithi125.stdout:1315: done (1 left) 2021-03-24T21:58:57.283 INFO:tasks.rados.rados.0.smithi125.stdout:1316: rollback oid 43 current snap is 136 2021-03-24T21:58:57.283 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 43 to 117 2021-03-24T21:58:57.284 INFO:tasks.rados.rados.0.smithi125.stdout:1317: write (excl) oid 49 current snap is 136 2021-03-24T21:58:57.284 INFO:tasks.rados.rados.0.smithi125.stdout:1317: seq_num 476 ranges {400132=632410,1438017=546162} 2021-03-24T21:58:57.290 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:57.294 INFO:tasks.rados.rados.0.smithi125.stdout:1317: writing smithi12556470-49 from 400132 to 1032542 tid 1 2021-03-24T21:58:57.302 INFO:tasks.rados.rados.0.smithi125.stdout:1317: writing smithi12556470-49 from 1438017 to 1984179 tid 2 2021-03-24T21:58:57.303 INFO:tasks.rados.rados.0.smithi125.stdout:1318: read oid 6 snap 130 2021-03-24T21:58:57.303 INFO:tasks.rados.rados.0.smithi125.stdout:1318: expect (ObjNum 432 snap 115 seq_num 432) 2021-03-24T21:58:57.303 INFO:tasks.rados.rados.0.smithi125.stdout:1319: write (excl) oid 25 current snap is 136 2021-03-24T21:58:57.303 INFO:tasks.rados.rados.0.smithi125.stdout:1319: seq_num 477 ranges {416939=707659,1896499=120440} 2021-03-24T21:58:57.310 INFO:tasks.rados.rados.0.smithi125.stdout:1319: writing smithi12556470-25 from 416939 to 1124598 tid 1 2021-03-24T21:58:57.316 INFO:tasks.rados.rados.0.smithi125.stdout:1319: writing smithi12556470-25 from 1896499 to 2016939 tid 2 2021-03-24T21:58:57.316 INFO:tasks.rados.rados.0.smithi125.stdout:1320: snap_create 2021-03-24T21:58:57.322 INFO:tasks.rados.rados.0.smithi125.stdout:1314: finishing write tid 5 to smithi12556470-47 2021-03-24T21:58:57.322 INFO:tasks.rados.rados.0.smithi125.stdout:1314: finishing write tid 6 to smithi12556470-47 2021-03-24T21:58:57.323 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 47 v 109 (ObjNum 474 snap 136 seq_num 474) dirty exists 2021-03-24T21:58:57.323 INFO:tasks.rados.rados.0.smithi125.stdout:1314: left oid 47 (ObjNum 474 snap 136 seq_num 474) 2021-03-24T21:58:57.323 INFO:tasks.rados.rados.0.smithi125.stdout:1316: finishing rollback tid 0 to smithi12556470-43 2021-03-24T21:58:57.323 INFO:tasks.rados.rados.0.smithi125.stdout:1316: finishing rollback tid 1 to smithi12556470-43 2021-03-24T21:58:57.329 INFO:tasks.rados.rados.0.smithi125.stdout:1319: finishing write tid 1 to smithi12556470-25 2021-03-24T21:58:57.329 INFO:tasks.rados.rados.0.smithi125.stdout:1318: expect (ObjNum 432 snap 115 seq_num 432) 2021-03-24T21:58:57.344 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:57 smithi125 conmon[19936]: cluster 2021-03-24T21:58:55.660728 2021-03-24T21:58:57.344 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:57 smithi125 conmon[19936]: +0000 osd.4 (osd.4) 91 : cluster [DBG] 4.1b scrub ok 2021-03-24T21:58:57.344 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:57 smithi125 conmon[19936]: cluster 2021-03-24T21:58:56.251534+0000 mon.a (mon.0 2021-03-24T21:58:57.344 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:57 smithi125 conmon[19936]: ) 1130 : cluster [DBG] osdmap e366: 8 total, 7 up, 6 in 2021-03-24T21:58:57.345 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:57 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:58:57.345 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:57 smithi125 conmon[19936]: 03-24T21:58:56.473502+0000 2021-03-24T21:58:57.345 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:57 smithi125 conmon[19936]: osd.3 (osd.3) 80 : cluster 2021-03-24T21:58:57.345 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:57 smithi125 conmon[19936]: [DBG] 4.0 scrub ok 2021-03-24T21:58:57.345 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:57 smithi125 conmon[19936]: cluster 2021-03-24T21:58:57.346 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:57 smithi125 conmon[19936]: 2021- 2021-03-24T21:58:57.346 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:57 smithi125 conmon[19936]: 03-24T21:58:57 2021-03-24T21:58:57.346 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:57 smithi125 conmon[19936]: .254457+0000 mon.a (mon. 2021-03-24T21:58:57.346 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:57 smithi125 conmon[19936]: 0) 1131 : cluster [DBG] 2021-03-24T21:58:57.346 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:57 smithi125 conmon[19936]: osdmap e367: 8 total, 7 up, 6 in 2021-03-24T21:58:57.347 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:57 smithi125 conmon[24927]: cluster 2021-03-24T21:58:55.660728+0000 osd.4 2021-03-24T21:58:57.347 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:57 smithi125 conmon[24927]: (osd.4) 91 : cluster [DBG] 4.1b scrub ok 2021-03-24T21:58:57.347 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:57 smithi125 conmon[24927]: cluster 2021-03-24T21:58:56.251534+0000 2021-03-24T21:58:57.347 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:57 smithi125 conmon[24927]: mon.a (mon.0) 1130 : cluster [DBG] osdmap e366: 8 total, 7 up, 6 in 2021-03-24T21:58:57.348 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:57 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:58:57.348 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:57 smithi125 conmon[24927]: 58:56.473502+0000 osd.3 (osd.3) 80 : cluster 2021-03-24T21:58:57.348 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:57 smithi125 conmon[24927]: [DBG] 4.0 scrub ok 2021-03-24T21:58:57.348 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:57 smithi125 conmon[24927]: cluster 2021-03-24T21:58:57.348 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:57 smithi125 conmon[24927]: 2021-03-24T21:58:57.254457+0000 mon.a ( 2021-03-24T21:58:57.349 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:57 smithi125 conmon[24927]: mon.0) 1131 : cluster [DBG] osdmap e367: 8 total, 7 up, 6 in 2021-03-24T21:58:57.421 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:57.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:57.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:57.509 INFO:tasks.rados.rados.0.smithi125.stdout:1316: finishing rollback tid 2 to smithi12556470-43 2021-03-24T21:58:57.509 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 43 v 454 (ObjNum 399 snap 102 seq_num 399) dirty exists 2021-03-24T21:58:57.510 INFO:tasks.rados.rados.0.smithi125.stdout:1317: finishing write tid 1 to smithi12556470-49 2021-03-24T21:58:57.510 INFO:tasks.rados.rados.0.smithi125.stdout:1319: finishing write tid 2 to smithi12556470-25 2021-03-24T21:58:57.510 INFO:tasks.rados.rados.0.smithi125.stdout:1319: finishing write tid 3 to smithi12556470-25 2021-03-24T21:58:57.510 INFO:tasks.rados.rados.0.smithi125.stdout:1319: finishing write tid 4 to smithi12556470-25 2021-03-24T21:58:57.511 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 25 v 296 (ObjNum 477 snap 136 seq_num 477) dirty exists 2021-03-24T21:58:57.511 INFO:tasks.rados.rados.0.smithi125.stdout:1319: left oid 25 (ObjNum 477 snap 136 seq_num 477) 2021-03-24T21:58:57.511 INFO:tasks.rados.rados.0.smithi125.stdout:1317: finishing write tid 2 to smithi12556470-49 2021-03-24T21:58:57.511 INFO:tasks.rados.rados.0.smithi125.stdout:1317: finishing write tid 3 to smithi12556470-49 2021-03-24T21:58:57.512 INFO:tasks.rados.rados.0.smithi125.stdout:1317: finishing write tid 4 to smithi12556470-49 2021-03-24T21:58:57.512 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 49 v 439 (ObjNum 476 snap 136 seq_num 476) dirty exists 2021-03-24T21:58:57.512 INFO:tasks.rados.rados.0.smithi125.stdout:1317: left oid 49 (ObjNum 476 snap 136 seq_num 476) 2021-03-24T21:58:57.530 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:57.540 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:57 smithi142 conmon[20125]: cluster 2021-03-24T21:58:55.660728+0000 2021-03-24T21:58:57.540 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:57 smithi142 conmon[20125]: osd.4 (osd.4) 91 : cluster [DBG] 4.1b scrub ok 2021-03-24T21:58:57.540 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:57 smithi142 conmon[20125]: cluster 2021-03-24T21:58:56.251534+0000 mon.a (mon.0) 1130 : cluster [DBG] osdmap e366: 8 total, 7 up, 6 in 2021-03-24T21:58:57.541 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:57 smithi142 conmon[20125]: cluster 2021-03-24T21:58:56.473502+0000 osd.3 (osd.3) 80 2021-03-24T21:58:57.541 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:57 smithi142 conmon[20125]: : cluster [DBG] 4.0 scrub ok 2021-03-24T21:58:57.541 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:57 smithi142 conmon[20125]: cluster 2021-03-24T21:58: 2021-03-24T21:58:57.541 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:57 smithi142 conmon[20125]: 57.254457+0000 mon.a (mon.0) 1131 : cluster 2021-03-24T21:58:57.541 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:57 smithi142 conmon[20125]: [DBG] osdmap e367: 8 total, 7 up, 6 in 2021-03-24T21:58:57.576 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:58:57.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:57.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:58:57.682 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:57.725 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:57.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:57.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:58:57.829 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:57.884 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:57.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:57.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:58:57.989 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:58.045 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:58.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:58.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:58:58.228 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:58.279 INFO:tasks.rados.rados.0.smithi125.stdout:1314: done (5 left) 2021-03-24T21:58:58.280 INFO:tasks.rados.rados.0.smithi125.stdout:1316: done (4 left) 2021-03-24T21:58:58.280 INFO:tasks.rados.rados.0.smithi125.stdout:1317: done (3 left) 2021-03-24T21:58:58.280 INFO:tasks.rados.rados.0.smithi125.stdout:1318: done (2 left) 2021-03-24T21:58:58.280 INFO:tasks.rados.rados.0.smithi125.stdout:1319: done (1 left) 2021-03-24T21:58:58.281 INFO:tasks.rados.rados.0.smithi125.stdout:1320: done (0 left) 2021-03-24T21:58:58.281 INFO:tasks.rados.rados.0.smithi125.stdout:1321: rollback oid 10 current snap is 137 2021-03-24T21:58:58.281 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 10 to 136 2021-03-24T21:58:58.281 INFO:tasks.rados.rados.0.smithi125.stdout:1322: snap_create 2021-03-24T21:58:58.291 INFO:tasks.rados.rados.0.smithi125.stdout:1321: finishing rollback tid 0 to smithi12556470-10 2021-03-24T21:58:58.291 INFO:tasks.rados.rados.0.smithi125.stdout:1321: finishing rollback tid 1 to smithi12556470-10 2021-03-24T21:58:58.292 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:58.292 INFO:tasks.rados.rados.0.smithi125.stdout:1321: finishing rollback tid 2 to smithi12556470-10 2021-03-24T21:58:58.293 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 10 v 380 (ObjNum 393 snap 100 seq_num 393) dirty exists 2021-03-24T21:58:58.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:58.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:58.380 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:58 smithi125 conmon[19936]: cluster 2021-03-24T21:58:56. 2021-03-24T21:58:58.380 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:58 smithi125 conmon[19936]: 648030+0000 osd.4 (osd.4) 92 : cluster [DBG] 7.1 scrub ok 2021-03-24T21:58:58.380 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:58 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:58:58.381 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:58 smithi125 conmon[19936]: 21:58:56.809942+0000 mgr.y (mgr 2021-03-24T21:58:58.381 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:58 smithi125 conmon[19936]: .14144) 452 : cluster 2021-03-24T21:58:58.381 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:58 smithi125 conmon[19936]: [DBG] pgmap v712: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 259 MiB data, 700 MiB used, 536 GiB / 536 GiB avail; 18 MiB/s rd, 4.2 MiB/s wr, 22 op/s; 240/1299 objects degraded (18.476%) 2021-03-24T21:58:58.381 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:58 smithi125 conmon[19936]: cluster 2021-03-24T21:58:57.435348+0000 osd.3 (osd.3) 81 : cluster [DBG] 4.14 scrub ok 2021-03-24T21:58:58.382 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:58 smithi125 conmon[24927]: cluster 2021-03-24T21:58: 2021-03-24T21:58:58.382 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:58 smithi125 conmon[24927]: 56.648030+0000 osd.4 (osd.4) 92 : cluster [DBG] 7.1 scrub ok 2021-03-24T21:58:58.382 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:58 smithi125 conmon[24927]: cluster 2021-03-24T21:58:58.382 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:58 smithi125 conmon[24927]: 2021-03-24T21:58:56.809942+0000 mgr.y (mgr.14144 2021-03-24T21:58:58.383 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:58 smithi125 conmon[24927]: ) 452 : cluster [DBG] pgmap v712: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 259 MiB data, 700 MiB used, 536 GiB / 536 GiB avail; 18 MiB/s rd, 4.2 MiB/s wr, 22 op/s; 240/1299 objects degraded (18.476%) 2021-03-24T21:58:58.383 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:58 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:58:58.383 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:58 smithi125 conmon[24927]: -03-24T21:58:57.435348+0000 osd.3 (osd.3) 81 : cluster [DBG] 4.14 scrub ok 2021-03-24T21:58:58.399 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:58.445 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:58.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:58.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:58:58.551 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:58.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:58 smithi142 conmon[20125]: cluster 2021-03-24T21:58:56 2021-03-24T21:58:58.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:58 smithi142 conmon[20125]: .648030+0000 osd.4 (osd.4) 92 : cluster [DBG] 7.1 scrub ok 2021-03-24T21:58:58.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:58 smithi142 conmon[20125]: cluster 2021-03-24T21:58:56 2021-03-24T21:58:58.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:58 smithi142 conmon[20125]: .809942+0000 mgr.y (mgr.14144) 452 : cluster [DBG] pgmap v712: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 259 MiB data, 700 MiB used, 536 GiB / 536 GiB avail; 18 MiB/s rd, 4.2 MiB/s wr, 22 op/s; 240/1299 objects degraded (18.476%) 2021-03-24T21:58:58.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:58 smithi142 conmon[20125]: cluster 2021-03-24T21:58:57.435348+0000 osd.3 ( 2021-03-24T21:58:58.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:58 smithi142 conmon[20125]: osd.3) 81 : cluster [DBG] 4.14 scrub ok 2021-03-24T21:58:58.626 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:58.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:58.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:58.648 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-24T21:58:58.734 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:58.785 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:58:58.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:58.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:58:58.798 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-24T21:58:58.890 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:59.010 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:59.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:59.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:59.118 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:59.171 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:58:59.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:59.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:58:59.277 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:59.287 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:59 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:58:59.293 INFO:tasks.rados.rados.0.smithi125.stdout:1321: done (1 left) 2021-03-24T21:58:59.294 INFO:tasks.rados.rados.0.smithi125.stdout:1322: done (0 left) 2021-03-24T21:58:59.294 INFO:tasks.rados.rados.0.smithi125.stdout:1323: rollback oid 20 current snap is 138 2021-03-24T21:58:59.294 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 20 to 130 2021-03-24T21:58:59.295 INFO:tasks.rados.rados.0.smithi125.stdout:1324: rollback oid 29 current snap is 138 2021-03-24T21:58:59.295 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 29 to 127 2021-03-24T21:58:59.295 INFO:tasks.rados.rados.0.smithi125.stdout:1325: write (excl) oid 37 current snap is 138 2021-03-24T21:58:59.295 INFO:tasks.rados.rados.0.smithi125.stdout:1325: seq_num 478 ranges {433746=782908,1954981=78765} 2021-03-24T21:58:59.303 INFO:tasks.rados.rados.0.smithi125.stdout:1325: writing smithi12556470-37 from 433746 to 1216654 tid 1 2021-03-24T21:58:59.309 INFO:tasks.rados.rados.0.smithi125.stdout:1325: writing smithi12556470-37 from 1954981 to 2033746 tid 2 2021-03-24T21:58:59.311 INFO:tasks.rados.rados.0.smithi125.stdout:1323: finishing rollback tid 0 to smithi12556470-20 2021-03-24T21:58:59.311 INFO:tasks.rados.rados.0.smithi125.stdout:1323: finishing rollback tid 1 to smithi12556470-20 2021-03-24T21:58:59.311 INFO:tasks.rados.rados.0.smithi125.stdout:1323: finishing rollback tid 2 to smithi12556470-20 2021-03-24T21:58:59.312 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 20 v 300 (ObjNum 449 snap 123 seq_num 449) dirty exists 2021-03-24T21:58:59.312 INFO:tasks.rados.rados.0.smithi125.stdout:1323: done (2 left) 2021-03-24T21:58:59.312 INFO:tasks.rados.rados.0.smithi125.stdout:1326: write oid 2 current snap is 138 2021-03-24T21:58:59.312 INFO:tasks.rados.rados.0.smithi125.stdout:1326: seq_num 479 ranges {450553=574510,1613463=437090} 2021-03-24T21:58:59.332 INFO:tasks.rados.rados.0.smithi125.stdout:1326: writing smithi12556470-2 from 450553 to 1025063 tid 1 2021-03-24T21:58:59.347 INFO:tasks.rados.rados.0.smithi125.stdout:1326: writing smithi12556470-2 from 1613463 to 2050553 tid 2 2021-03-24T21:58:59.349 INFO:tasks.rados.rados.0.smithi125.stdout:1324: finishing rollback tid 0 to smithi12556470-29 2021-03-24T21:58:59.349 INFO:tasks.rados.rados.0.smithi125.stdout:1324: finishing rollback tid 1 to smithi12556470-29 2021-03-24T21:58:59.349 INFO:tasks.rados.rados.0.smithi125.stdout:1324: finishing rollback tid 2 to smithi12556470-29 2021-03-24T21:58:59.349 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 29 v 171 (ObjNum 401 snap 104 seq_num 401) dirty exists 2021-03-24T21:58:59.350 INFO:tasks.rados.rados.0.smithi125.stdout:1325: finishing write tid 1 to smithi12556470-37 2021-03-24T21:58:59.350 INFO:tasks.rados.rados.0.smithi125.stdout:1325: finishing write tid 2 to smithi12556470-37 2021-03-24T21:58:59.350 INFO:tasks.rados.rados.0.smithi125.stdout:1325: finishing write tid 3 to smithi12556470-37 2021-03-24T21:58:59.350 INFO:tasks.rados.rados.0.smithi125.stdout:1325: finishing write tid 4 to smithi12556470-37 2021-03-24T21:58:59.351 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 37 v 337 (ObjNum 478 snap 138 seq_num 478) dirty exists 2021-03-24T21:58:59.351 INFO:tasks.rados.rados.0.smithi125.stdout:1325: left oid 37 (ObjNum 478 snap 138 seq_num 478) 2021-03-24T21:58:59.351 INFO:tasks.rados.rados.0.smithi125.stdout:1326: finishing write tid 1 to smithi12556470-2 2021-03-24T21:58:59.351 INFO:tasks.rados.rados.0.smithi125.stdout:1324: done (2 left) 2021-03-24T21:58:59.352 INFO:tasks.rados.rados.0.smithi125.stdout:1325: done (1 left) 2021-03-24T21:58:59.352 INFO:tasks.rados.rados.0.smithi125.stdout:1327: read oid 29 snap -1 2021-03-24T21:58:59.352 INFO:tasks.rados.rados.0.smithi125.stdout:1327: expect (ObjNum 401 snap 104 seq_num 401) 2021-03-24T21:58:59.352 INFO:tasks.rados.rados.0.smithi125.stdout:1328: read oid 11 snap 129 2021-03-24T21:58:59.352 INFO:tasks.rados.rados.0.smithi125.stdout:1328: expect (ObjNum 402 snap 104 seq_num 402) 2021-03-24T21:58:59.353 INFO:tasks.rados.rados.0.smithi125.stdout:1329: delete oid 34 current snap is 138 2021-03-24T21:58:59.361 INFO:tasks.rados.rados.0.smithi125.stdout:1326: finishing write tid 2 to smithi12556470-2 2021-03-24T21:58:59.361 INFO:tasks.rados.rados.0.smithi125.stdout:1326: finishing write tid 3 to smithi12556470-2 2021-03-24T21:58:59.361 INFO:tasks.rados.rados.0.smithi125.stdout:1326: finishing write tid 4 to smithi12556470-2 2021-03-24T21:58:59.362 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 2 v 374 (ObjNum 479 snap 138 seq_num 479) dirty exists 2021-03-24T21:58:59.362 INFO:tasks.rados.rados.0.smithi125.stdout:1326: left oid 2 (ObjNum 479 snap 138 seq_num 479) 2021-03-24T21:58:59.365 INFO:tasks.rados.rados.0.smithi125.stdout:1326: done (3 left) 2021-03-24T21:58:59.365 INFO:tasks.rados.rados.0.smithi125.stdout:1329: done (2 left) 2021-03-24T21:58:59.365 INFO:tasks.rados.rados.0.smithi125.stdout:1330: read oid 7 snap -1 2021-03-24T21:58:59.366 INFO:tasks.rados.rados.0.smithi125.stdout:1330: expect (ObjNum 404 snap 104 seq_num 404) 2021-03-24T21:58:59.366 INFO:tasks.rados.rados.0.smithi125.stdout:1331: delete oid 6 current snap is 138 2021-03-24T21:58:59.373 INFO:tasks.rados.rados.0.smithi125.stdout:1330: expect (ObjNum 404 snap 104 seq_num 404) 2021-03-24T21:58:59.413 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:58:59.417 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:58:59.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:59.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:58:59.453 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:58:59.531 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:59.542 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:59 smithi125 conmon[19936]: 21:58:58.033855+0000 osd.0 (osd.0) 83 : cluster [DBG] 6.4 scrub ok 2021-03-24T21:58:59.600 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:59 smithi125 conmon[19936]: cluster 2021-03-24T21:58:58.277120+0000 2021-03-24T21:58:59.601 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:59 smithi125 conmon[19936]: mon.a (mon.0) 1132 : cluster [DBG] osdmap e368: 8 total, 7 up, 6 in 2021-03-24T21:58:59.601 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:59 smithi125 conmon[19936]: audit 2021-03-24T21:58:59.601 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:59 smithi125 conmon[19936]: 2021-03- 2021-03-24T21:58:59.601 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:59 smithi125 conmon[19936]: 24T21:58:58.546780+0000 mon.a (mon 2021-03-24T21:58:59.602 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:59 smithi125 conmon[19936]: .0) 1133 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:58:59.602 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:59 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:58:59.602 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:59 smithi125 conmon[19936]: 03-24T21:58:58.834601+0000 mon.a 2021-03-24T21:58:59.602 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:59 smithi125 conmon[19936]: (mon.0) 1134 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:58:59.602 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:59 smithi125 conmon[19936]: audit 2021-03-24T21:58:59.603 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:59 smithi125 conmon[19936]: 2021-03-24T21:58:59.128454 2021-03-24T21:58:59.603 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:59 smithi125 conmon[19936]: +0000 mon.a (mon.0) 1135 : audit 2021-03-24T21:58:59.603 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:58:59 smithi125 conmon[19936]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:58:59.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:59 smithi125 conmon[24927]: cluster 2021-03-24T21:58:58.033855+0000 osd.0 (osd.0) 83 : cluster [DBG] 6.4 scrub ok 2021-03-24T21:58:59.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:59 smithi125 conmon[24927]: cluster 2021-03-24T21:58:58.277120+0000 mon.a (mon.0) 1132 : cluster [DBG] osdmap e368: 8 total, 7 up, 6 in 2021-03-24T21:58:59.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:59 smithi125 conmon[24927]: audit 2021-03-24T21:58:58.546780+0000 mon.a (mon.0) 1133 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:58:59.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:59 smithi125 conmon[24927]: audit 2021-03-24T21:58:58.834601+0000 mon.a (mon.0) 1134 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:58:59.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:58:59 smithi125 conmon[24927]: audit 2021-03-24T21:58:59.128454+0000 mon.a (mon.0) 1135 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:58:59.605 INFO:tasks.rados.rados.0.smithi125.stdout:1327: expect (ObjNum 401 snap 104 seq_num 401) 2021-03-24T21:58:59.605 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:58:59.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:59.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:58:59.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:59 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:58:59.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:59 smithi142 conmon[20125]: 58:58.033855+0000 osd.0 (osd.0) 83 : cluster [DBG] 6.4 scrub ok 2021-03-24T21:58:59.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:59 smithi142 conmon[20125]: cluster 2021-03-24T21:58:59.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:59 smithi142 conmon[20125]: 2021-03-24T21:58:58. 2021-03-24T21:58:59.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:59 smithi142 conmon[20125]: 277120+0000 mon.a (mon.0) 2021-03-24T21:58:59.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:59 smithi142 conmon[20125]: 1132 : cluster [DBG] osdmap e368: 8 total, 7 up, 6 in 2021-03-24T21:58:59.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:59 smithi142 conmon[20125]: audit 2021-03-24T21:58:59.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:59 smithi142 conmon[20125]: 2021-03-24T21:58 2021-03-24T21:58:59.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:59 smithi142 conmon[20125]: :58.546780+0000 mon.a ( 2021-03-24T21:58:59.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:59 smithi142 conmon[20125]: mon.0) 1133 : audit [DBG] 2021-03-24T21:58:59.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:59 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:58:59.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:59 smithi142 conmon[20125]: audit 2021-03-24T21:58: 2021-03-24T21:58:59.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:59 smithi142 conmon[20125]: 58.834601+0000 mon.a (mon.0) 1134 2021-03-24T21:58:59.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:59 smithi142 conmon[20125]: : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:58:59.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:59 smithi142 conmon[20125]: audit 2021-03-24T21 2021-03-24T21:58:59.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:59 smithi142 conmon[20125]: :58:59.128454+0000 mon.a (mon. 2021-03-24T21:58:59.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:58:59 smithi142 conmon[20125]: 0) 1135 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T21:58:59.710 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:59.756 INFO:tasks.rados.rados.0.smithi125.stdout:1328: expect (ObjNum 402 snap 104 seq_num 402) 2021-03-24T21:58:59.764 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:58:59.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:59.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:58:59.869 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:59.889 INFO:tasks.rados.rados.0.smithi125.stdout:1327: done (3 left) 2021-03-24T21:58:59.890 INFO:tasks.rados.rados.0.smithi125.stdout:1328: done (2 left) 2021-03-24T21:58:59.890 INFO:tasks.rados.rados.0.smithi125.stdout:1330: done (1 left) 2021-03-24T21:58:59.890 INFO:tasks.rados.rados.0.smithi125.stdout:1331: done (0 left) 2021-03-24T21:58:59.890 INFO:tasks.rados.rados.0.smithi125.stdout:1332: delete oid 16 current snap is 138 2021-03-24T21:58:59.893 INFO:tasks.rados.rados.0.smithi125.stdout:1332: done (0 left) 2021-03-24T21:58:59.893 INFO:tasks.rados.rados.0.smithi125.stdout:1333: write oid 21 current snap is 138 2021-03-24T21:58:59.893 INFO:tasks.rados.rados.0.smithi125.stdout:1333: seq_num 483 ranges {517781=475506,1731038=386743} 2021-03-24T21:58:59.904 INFO:tasks.rados.rados.0.smithi125.stdout:1333: writing smithi12556470-21 from 517781 to 993287 tid 1 2021-03-24T21:58:59.912 INFO:tasks.rados.rados.0.smithi125.stdout:1333: writing smithi12556470-21 from 1731038 to 2117781 tid 2 2021-03-24T21:58:59.912 INFO:tasks.rados.rados.0.smithi125.stdout:1334: snap_remove snap 127 2021-03-24T21:58:59.924 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:58:59.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:58:59.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:58:59.950 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-24T21:58:59.953 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2021-03-24T21:59:00.031 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:00.087 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:00.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:00.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:00.194 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:00.219 INFO:tasks.rados.rados.0.smithi125.stdout:1334: done (1 left) 2021-03-24T21:59:00.219 INFO:tasks.rados.rados.0.smithi125.stdout:1335: rollback oid 22 current snap is 138 2021-03-24T21:59:00.220 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 22 to 136 2021-03-24T21:59:00.220 INFO:tasks.rados.rados.0.smithi125.stdout:1333: finishing write tid 1 to smithi12556470-21 2021-03-24T21:59:00.220 INFO:tasks.rados.rados.0.smithi125.stdout:1333: finishing write tid 2 to smithi12556470-21 2021-03-24T21:59:00.220 INFO:tasks.rados.rados.0.smithi125.stdout:1333: finishing write tid 3 to smithi12556470-21 2021-03-24T21:59:00.221 INFO:tasks.rados.rados.0.smithi125.stdout:1333: finishing write tid 4 to smithi12556470-21 2021-03-24T21:59:00.221 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 21 v 348 (ObjNum 483 snap 138 seq_num 483) dirty exists 2021-03-24T21:59:00.221 INFO:tasks.rados.rados.0.smithi125.stdout:1333: left oid 21 (ObjNum 483 snap 138 seq_num 483) 2021-03-24T21:59:00.221 INFO:tasks.rados.rados.0.smithi125.stdout:1333: done (1 left) 2021-03-24T21:59:00.222 INFO:tasks.rados.rados.0.smithi125.stdout:1336: read oid 45 snap -1 2021-03-24T21:59:00.222 INFO:tasks.rados.rados.0.smithi125.stdout:1336: expect (ObjNum 466 snap 131 seq_num 466) 2021-03-24T21:59:00.222 INFO:tasks.rados.rados.0.smithi125.stdout:1337: read oid 44 snap -1 2021-03-24T21:59:00.222 INFO:tasks.rados.rados.0.smithi125.stdout:1337: expect (ObjNum 470 snap 133 seq_num 470) 2021-03-24T21:59:00.222 INFO:tasks.rados.rados.0.smithi125.stdout:1338: copy_from oid 40 from oid 21 current snap is 138 2021-03-24T21:59:00.223 INFO:tasks.rados.rados.0.smithi125.stdout:1339: copy_from oid 16 from oid 10 current snap is 138 2021-03-24T21:59:00.223 INFO:tasks.rados.rados.0.smithi125.stdout:1340: write (excl) oid 6 current snap is 138 2021-03-24T21:59:00.223 INFO:tasks.rados.rados.0.smithi125.stdout:1340: seq_num 484 ranges {534588=550755,1789520=345068} 2021-03-24T21:59:00.223 INFO:tasks.rados.rados.0.smithi125.stdout:1340: writing smithi12556470-6 from 534588 to 1085343 tid 1 2021-03-24T21:59:00.224 INFO:tasks.rados.rados.0.smithi125.stdout:1340: writing smithi12556470-6 from 1789520 to 2134588 tid 2 2021-03-24T21:59:00.224 INFO:tasks.rados.rados.0.smithi125.stdout:1341: read oid 37 snap -1 2021-03-24T21:59:00.224 INFO:tasks.rados.rados.0.smithi125.stdout:1341: expect (ObjNum 478 snap 138 seq_num 478) 2021-03-24T21:59:00.224 INFO:tasks.rados.rados.0.smithi125.stdout:1342: snap_remove snap 122 2021-03-24T21:59:00.297 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:00.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:00.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:59:00.387 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[19936]: cluster 2021-03-24T21:58:58.719282+0000 2021-03-24T21:59:00.388 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[19936]: osd.4 (osd.4) 93 : cluster [DBG] 4.18 scrub ok 2021-03-24T21:59:00.388 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[19936]: cluster 2021-03-24T21:58:58.813754+0000 mgr.y (mgr.14144) 453 : cluster [DBG] pgmap v715: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 258 MiB data, 700 MiB used, 536 GiB / 536 GiB avail; 241/1302 objects degraded (18.510%) 2021-03-24T21:59:00.388 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[19936]: 2021-03-24T21:59:00.388 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:59:00.389 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[19936]: :58:58.817994+0000 osd.7 (osd.7) 82 : cluster 2021-03-24T21:59:00.389 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[19936]: [DBG] 2.0 scrub ok 2021-03-24T21:59:00.389 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:59:00.389 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[19936]: :58:59.289971+0000 mon.a (mon.0 2021-03-24T21:59:00.390 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[19936]: ) 1136 : cluster [DBG] osdmap e369: 8 total, 7 up, 6 in 2021-03-24T21:59:00.390 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:59:00.390 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[19936]: -24T21:58:59.862015+0000 2021-03-24T21:59:00.390 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[19936]: osd.2 (osd.2) 69 : cluster [DBG] 6.14 scrub ok 2021-03-24T21:59:00.391 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[19936]: 2021-03-24T21:59:00.391 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[19936]: cluster 2021-03-24T21:59:00.391 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[19936]: 2021-03-24T21:59:00.191070+0000 2021-03-24T21:59:00.391 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[19936]: mon.a (mon.0) 1137 : cluster [WRN] 2021-03-24T21:59:00.391 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[19936]: Health check update: Degraded data redundancy: 241/1302 objects degraded (18.510%), 40 pgs degraded (PG_DEGRADED) 2021-03-24T21:59:00.392 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[19936]: cluster 2021-03-24T21:59 2021-03-24T21:59:00.392 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[19936]: :00.200998+0000 mon.a (mon.0) 2021-03-24T21:59:00.392 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[19936]: 1138 : cluster [DBG] osdmap e370: 8 total, 7 up, 6 in 2021-03-24T21:59:00.393 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: cluster 2021-03-24T21:59:00.393 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 2021-03-24T21:59:00.393 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 2021 2021-03-24T21:59:00.394 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: - 2021-03-24T21:59:00.394 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 03 2021-03-24T21:59:00.394 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: - 2021-03-24T21:59:00.394 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 24 2021-03-24T21:59:00.394 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: T 2021-03-24T21:59:00.395 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 21 2021-03-24T21:59:00.395 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: : 2021-03-24T21:59:00.395 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 58 2021-03-24T21:59:00.395 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: : 2021-03-24T21:59:00.395 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 58 2021-03-24T21:59:00.396 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: . 2021-03-24T21:59:00.396 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 719282 2021-03-24T21:59:00.396 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: +0000 2021-03-24T21:59:00.397 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 2021-03-24T21:59:00.397 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: osd.4 2021-03-24T21:59:00.397 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: ( 2021-03-24T21:59:00.397 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: osd 2021-03-24T21:59:00.397 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: . 2021-03-24T21:59:00.398 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 4 2021-03-24T21:59:00.398 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: ) 2021-03-24T21:59:00.398 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 93 2021-03-24T21:59:00.398 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: : 2021-03-24T21:59:00.398 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: cluster 2021-03-24T21:59:00.399 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 2021-03-24T21:59:00.399 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: [DBG] 2021-03-24T21:59:00.399 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 2021-03-24T21:59:00.399 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 4.18 scrub ok 2021-03-24T21:59:00.399 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 2021-03-24T21:59:00.400 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: cluster 2021-03-24T21:59:00.400 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 2021-03-24T21:59:00.400 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 2021 2021-03-24T21:59:00.400 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: - 2021-03-24T21:59:00.401 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 03 2021-03-24T21:59:00.401 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: - 2021-03-24T21:59:00.401 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 24 2021-03-24T21:59:00.401 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: T 2021-03-24T21:59:00.401 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 21 2021-03-24T21:59:00.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: : 2021-03-24T21:59:00.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 58 2021-03-24T21:59:00.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: : 2021-03-24T21:59:00.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 58 2021-03-24T21:59:00.402 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: . 2021-03-24T21:59:00.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 813754 2021-03-24T21:59:00.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: +0000 2021-03-24T21:59:00.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 2021-03-24T21:59:00.403 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: mgr.y 2021-03-24T21:59:00.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: ( 2021-03-24T21:59:00.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: mgr 2021-03-24T21:59:00.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: . 2021-03-24T21:59:00.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 14144 2021-03-24T21:59:00.404 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: ) 2021-03-24T21:59:00.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 453 2021-03-24T21:59:00.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: : 2021-03-24T21:59:00.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: cluster 2021-03-24T21:59:00.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 2021-03-24T21:59:00.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: [DBG] 2021-03-24T21:59:00.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 2021-03-24T21:59:00.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: pgmap v715: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 258 MiB data, 700 MiB used, 536 GiB / 536 GiB avail; 241/1302 objects degraded (18.510%) 2021-03-24T21:59:00.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 2021-03-24T21:59:00.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: cluster 2021-03-24T21:59:00.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 2021-03-24T21:59:00.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 2021 2021-03-24T21:59:00.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: - 2021-03-24T21:59:00.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 03 2021-03-24T21:59:00.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: - 2021-03-24T21:59:00.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 24 2021-03-24T21:59:00.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: T 2021-03-24T21:59:00.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 21 2021-03-24T21:59:00.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: : 2021-03-24T21:59:00.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 58 2021-03-24T21:59:00.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: : 2021-03-24T21:59:00.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 58 2021-03-24T21:59:00.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: . 2021-03-24T21:59:00.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 817994 2021-03-24T21:59:00.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: +0000 2021-03-24T21:59:00.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 2021-03-24T21:59:00.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: osd.7 2021-03-24T21:59:00.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: ( 2021-03-24T21:59:00.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: osd 2021-03-24T21:59:00.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: . 2021-03-24T21:59:00.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 7 2021-03-24T21:59:00.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: ) 2021-03-24T21:59:00.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 82 2021-03-24T21:59:00.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: : 2021-03-24T21:59:00.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: cluster 2021-03-24T21:59:00.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 2021-03-24T21:59:00.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: [DBG] 2021-03-24T21:59:00.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 2021-03-24T21:59:00.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 2.0 scrub ok 2021-03-24T21:59:00.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 2021-03-24T21:59:00.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: cluster 2021-03-24T21:59:00.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 2021-03-24T21:59:00.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 2021 2021-03-24T21:59:00.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: - 2021-03-24T21:59:00.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 03 2021-03-24T21:59:00.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: - 2021-03-24T21:59:00.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 24 2021-03-24T21:59:00.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: T 2021-03-24T21:59:00.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 21 2021-03-24T21:59:00.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: : 2021-03-24T21:59:00.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 58 2021-03-24T21:59:00.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: : 2021-03-24T21:59:00.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 59 2021-03-24T21:59:00.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: . 2021-03-24T21:59:00.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 289971 2021-03-24T21:59:00.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: +0000 2021-03-24T21:59:00.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 2021-03-24T21:59:00.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: mon.a 2021-03-24T21:59:00.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: ( 2021-03-24T21:59:00.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: mon 2021-03-24T21:59:00.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: . 2021-03-24T21:59:00.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 0 2021-03-24T21:59:00.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: ) 2021-03-24T21:59:00.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 1136 2021-03-24T21:59:00.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: : 2021-03-24T21:59:00.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: cluster 2021-03-24T21:59:00.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 2021-03-24T21:59:00.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: [DBG] 2021-03-24T21:59:00.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 2021-03-24T21:59:00.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: osdmap e369: 8 total, 7 up, 6 in 2021-03-24T21:59:00.420 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 2021-03-24T21:59:00.420 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: cluster 2021-03-24T21:59:00.420 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 2021-03-24T21:59:00.420 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 2021 2021-03-24T21:59:00.420 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: - 2021-03-24T21:59:00.421 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 03 2021-03-24T21:59:00.421 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: - 2021-03-24T21:59:00.421 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 24 2021-03-24T21:59:00.421 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: T 2021-03-24T21:59:00.421 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 21 2021-03-24T21:59:00.422 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: : 2021-03-24T21:59:00.422 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 58 2021-03-24T21:59:00.422 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: : 2021-03-24T21:59:00.422 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 59 2021-03-24T21:59:00.423 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: . 2021-03-24T21:59:00.423 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 862015 2021-03-24T21:59:00.423 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: +0000 2021-03-24T21:59:00.423 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 2021-03-24T21:59:00.423 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: osd.2 2021-03-24T21:59:00.424 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: ( 2021-03-24T21:59:00.424 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: osd 2021-03-24T21:59:00.424 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: . 2021-03-24T21:59:00.424 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 2 2021-03-24T21:59:00.424 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: ) 2021-03-24T21:59:00.425 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 69 2021-03-24T21:59:00.425 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: : 2021-03-24T21:59:00.425 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: cluster 2021-03-24T21:59:00.425 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 2021-03-24T21:59:00.425 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: [DBG] 2021-03-24T21:59:00.426 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 2021-03-24T21:59:00.426 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 6.14 scrub ok 2021-03-24T21:59:00.426 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 2021-03-24T21:59:00.426 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: cluster 2021-03-24T21:59:00.427 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 2021-03-24T21:59:00.427 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 2021 2021-03-24T21:59:00.427 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: - 2021-03-24T21:59:00.427 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 03 2021-03-24T21:59:00.427 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: - 2021-03-24T21:59:00.428 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 24 2021-03-24T21:59:00.428 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: T 2021-03-24T21:59:00.428 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 21 2021-03-24T21:59:00.428 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: : 2021-03-24T21:59:00.428 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 59 2021-03-24T21:59:00.429 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: : 2021-03-24T21:59:00.429 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 00 2021-03-24T21:59:00.429 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: . 2021-03-24T21:59:00.429 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 191070 2021-03-24T21:59:00.429 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: +0000 2021-03-24T21:59:00.430 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 2021-03-24T21:59:00.430 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: mon.a 2021-03-24T21:59:00.430 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: ( 2021-03-24T21:59:00.430 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: mon 2021-03-24T21:59:00.431 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: . 2021-03-24T21:59:00.431 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 0 2021-03-24T21:59:00.431 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: ) 2021-03-24T21:59:00.431 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 1137 2021-03-24T21:59:00.431 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: : 2021-03-24T21:59:00.432 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: cluster 2021-03-24T21:59:00.432 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 2021-03-24T21:59:00.432 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: [WRN] 2021-03-24T21:59:00.432 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 2021-03-24T21:59:00.432 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: Health check update: Degraded data redundancy: 241/1302 objects degraded (18.510%), 40 pgs degraded (PG_DEGRADED) 2021-03-24T21:59:00.433 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 2021-03-24T21:59:00.433 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: cluster 2021-03-24T21:59:00.433 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 2021-03-24T21:59:00.433 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 2021 2021-03-24T21:59:00.433 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: - 2021-03-24T21:59:00.434 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 03 2021-03-24T21:59:00.434 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: - 2021-03-24T21:59:00.434 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 24 2021-03-24T21:59:00.434 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: T 2021-03-24T21:59:00.434 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 21 2021-03-24T21:59:00.435 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: : 2021-03-24T21:59:00.435 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 59 2021-03-24T21:59:00.435 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: : 2021-03-24T21:59:00.435 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 00 2021-03-24T21:59:00.435 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: . 2021-03-24T21:59:00.436 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 200998 2021-03-24T21:59:00.436 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: +0000 2021-03-24T21:59:00.436 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 2021-03-24T21:59:00.436 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: mon.a 2021-03-24T21:59:00.437 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: ( 2021-03-24T21:59:00.437 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: mon 2021-03-24T21:59:00.437 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: . 2021-03-24T21:59:00.437 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 0 2021-03-24T21:59:00.437 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: ) 2021-03-24T21:59:00.438 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 1138 2021-03-24T21:59:00.438 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: : 2021-03-24T21:59:00.438 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: cluster 2021-03-24T21:59:00.438 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 2021-03-24T21:59:00.439 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: [DBG] 2021-03-24T21:59:00.439 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 2021-03-24T21:59:00.439 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: osdmap e370: 8 total, 7 up, 6 in 2021-03-24T21:59:00.439 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:00 smithi125 conmon[24927]: 2021-03-24T21:59:00.440 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:00.484 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:00.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:00.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:59:00.588 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:00.601 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:00 smithi142 conmon[20125]: cluster 2021-03-24T21:58:58.719282+0000 osd.4 2021-03-24T21:59:00.602 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:00 smithi142 conmon[20125]: (osd.4) 93 : cluster [DBG] 4.18 scrub ok 2021-03-24T21:59:00.602 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:00 smithi142 conmon[20125]: cluster 2021-03-24T21:58:58.813754+0000 mgr.y (mgr.14144) 453 : cluster [DBG] pgmap v715: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 258 MiB data, 700 MiB used, 536 GiB / 536 GiB avail; 241/1302 objects degraded (18.510%) 2021-03-24T21:59:00.602 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:00 smithi142 conmon[20125]: cluster 2021-03-24T21:58:58.817994 2021-03-24T21:59:00.602 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:00 smithi142 conmon[20125]: +0000 osd.7 (osd.7) 82 : cluster [DBG] 2.0 scrub ok 2021-03-24T21:59:00.603 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:00 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:59:00.604 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:00 smithi142 conmon[20125]: -03-24T21:58:59.289971+0000 mon.a ( 2021-03-24T21:59:00.604 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:00 smithi142 conmon[20125]: mon.0) 1136 : cluster [DBG] osdmap e369: 8 total, 7 up, 6 in 2021-03-24T21:59:00.604 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:00 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:59:00.604 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:00 smithi142 conmon[20125]: -24T21:58:59.862015+0000 osd.2 (osd 2021-03-24T21:59:00.605 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:00 smithi142 conmon[20125]: .2) 69 : cluster [DBG] 6.14 scrub ok 2021-03-24T21:59:00.605 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:00 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:59:00.605 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:00 smithi142 conmon[20125]: -24T21:59:00.191070+0000 mon.a ( 2021-03-24T21:59:00.605 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:00 smithi142 conmon[20125]: mon.0) 1137 : cluster [WRN] Health check update: Degraded data redundancy: 241/1302 objects degraded (18.510%), 40 pgs degraded (PG_DEGRADED) 2021-03-24T21:59:00.605 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:00 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:59:00.606 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:00 smithi142 conmon[20125]: 24T21:59:00.200998+0000 mon.a (mon. 2021-03-24T21:59:00.606 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:00 smithi142 conmon[20125]: 0) 1138 : cluster [DBG] osdmap e370: 8 total, 7 up, 6 in 2021-03-24T21:59:00.639 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:00.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:00.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:59:00.754 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:00.806 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:00.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:00.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:00.913 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:00.974 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:00.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:00.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:59:01.078 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:01.135 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:01.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:01.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:59:01.214 INFO:tasks.rados.rados.0.smithi125.stdout:1342: done (7 left) 2021-03-24T21:59:01.214 INFO:tasks.rados.rados.0.smithi125.stdout:1343: copy_from oid 26 from oid 48 current snap is 138 2021-03-24T21:59:01.214 INFO:tasks.rados.rados.0.smithi125.stdout:1344: read oid 29 snap -1 2021-03-24T21:59:01.215 INFO:tasks.rados.rados.0.smithi125.stdout:1344: expect (ObjNum 401 snap 104 seq_num 401) 2021-03-24T21:59:01.215 INFO:tasks.rados.rados.0.smithi125.stdout:1345: copy_from oid 7 from oid 46 current snap is 138 2021-03-24T21:59:01.215 INFO:tasks.rados.rados.0.smithi125.stdout:1346: snap_create 2021-03-24T21:59:01.215 INFO:tasks.rados.rados.0.smithi125.stdout:1336: expect (ObjNum 466 snap 131 seq_num 466) 2021-03-24T21:59:01.242 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:01.304 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:01 smithi142 conmon[20125]: audit 2021-03-24T21:58:59.927177+0000 2021-03-24T21:59:01.304 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:01 smithi142 conmon[20125]: mgr.y (mgr.14144) 454 : audit [DBG] from='client.35135 v1:172.21.15.125:0/2138294430' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T21:59:01.304 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:01 smithi142 conmon[20125]: cluster 2021-03-24T21:59:00.033456+0000 osd.0 (osd.0) 84 : cluster [DBG] 2021-03-24T21:59:01.304 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:01 smithi142 conmon[20125]: 6.9 scrub ok 2021-03-24T21:59:01.305 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:01 smithi142 conmon[20125]: cluster 2021-03-24T21 2021-03-24T21:59:01.305 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:01 smithi142 conmon[20125]: :59:00.393736+0000 osd.3 (osd.3) 82 : cluster 2021-03-24T21:59:01.305 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:01 smithi142 conmon[20125]: [DBG] 3.18 scrub ok 2021-03-24T21:59:01.305 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:01 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:59:01.306 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:01 smithi142 conmon[20125]: 59:01.204834+0000 mon.a (mon.0 2021-03-24T21:59:01.306 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:01 smithi142 conmon[20125]: ) 1139 : cluster [DBG] osdmap e371: 8 total, 7 up, 6 in 2021-03-24T21:59:01.350 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:01.357 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[24927]: audit 2021-03-24T21:58:59.927177 2021-03-24T21:59:01.358 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[24927]: +0000 mgr.y (mgr.14144) 454 : audit [DBG] from='client.35135 v1:172.21.15.125:0/2138294430' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T21:59:01.358 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:59:01.358 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[24927]: 24T21:59:00.033456+0000 osd.0 (osd.0) 84 : cluster [DBG] 2021-03-24T21:59:01.358 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[24927]: 6.9 scrub ok 2021-03-24T21:59:01.359 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:59:01.359 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[24927]: 03-24T21:59:00.393736+0000 osd.3 (osd.3 2021-03-24T21:59:01.359 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[24927]: ) 82 : cluster [DBG] 3.18 scrub ok 2021-03-24T21:59:01.359 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[24927]: cluster 2021-03-24T21:59 2021-03-24T21:59:01.359 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[24927]: :01.204834+0000 mon.a (mon.0) 1139 2021-03-24T21:59:01.360 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[24927]: : cluster [DBG] osdmap e371: 8 total, 7 up, 6 in 2021-03-24T21:59:01.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:01.361 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: audit 2021-03-24T21:59:01.361 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 2021-03-24T21:59:01.361 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 2021 2021-03-24T21:59:01.361 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: - 2021-03-24T21:59:01.362 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 03 2021-03-24T21:59:01.362 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: - 2021-03-24T21:59:01.362 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 24 2021-03-24T21:59:01.362 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: T 2021-03-24T21:59:01.363 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 21 2021-03-24T21:59:01.363 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: : 2021-03-24T21:59:01.363 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 58 2021-03-24T21:59:01.363 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: : 2021-03-24T21:59:01.363 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 59 2021-03-24T21:59:01.364 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: . 2021-03-24T21:59:01.364 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 927177 2021-03-24T21:59:01.364 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: +0000 2021-03-24T21:59:01.364 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 2021-03-24T21:59:01.364 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: mgr.y 2021-03-24T21:59:01.365 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: ( 2021-03-24T21:59:01.365 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: mgr 2021-03-24T21:59:01.365 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: . 2021-03-24T21:59:01.365 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 14144 2021-03-24T21:59:01.366 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: ) 2021-03-24T21:59:01.366 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 454 2021-03-24T21:59:01.366 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: : 2021-03-24T21:59:01.366 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: audit 2021-03-24T21:59:01.366 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 2021-03-24T21:59:01.367 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: [DBG] 2021-03-24T21:59:01.367 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 2021-03-24T21:59:01.367 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: from='client.35135 v1:172.21.15.125:0/2138294430' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T21:59:01.367 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 2021-03-24T21:59:01.367 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: cluster 2021-03-24T21:59:01.368 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 2021-03-24T21:59:01.368 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 2021 2021-03-24T21:59:01.368 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: - 2021-03-24T21:59:01.368 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 03 2021-03-24T21:59:01.369 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: - 2021-03-24T21:59:01.369 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 24 2021-03-24T21:59:01.369 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: T 2021-03-24T21:59:01.369 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 21 2021-03-24T21:59:01.369 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: : 2021-03-24T21:59:01.370 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 59 2021-03-24T21:59:01.370 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: : 2021-03-24T21:59:01.370 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 00 2021-03-24T21:59:01.370 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: . 2021-03-24T21:59:01.370 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 033456 2021-03-24T21:59:01.371 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: +0000 2021-03-24T21:59:01.371 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 2021-03-24T21:59:01.371 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: osd.0 2021-03-24T21:59:01.371 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: ( 2021-03-24T21:59:01.371 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: osd 2021-03-24T21:59:01.372 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: . 2021-03-24T21:59:01.372 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 0 2021-03-24T21:59:01.372 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: ) 2021-03-24T21:59:01.372 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 84 2021-03-24T21:59:01.373 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: : 2021-03-24T21:59:01.373 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: cluster 2021-03-24T21:59:01.373 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 2021-03-24T21:59:01.373 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: [DBG] 2021-03-24T21:59:01.373 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 2021-03-24T21:59:01.374 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 6.9 scrub ok 2021-03-24T21:59:01.374 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 2021-03-24T21:59:01.374 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: cluster 2021-03-24T21:59:01.374 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 2021-03-24T21:59:01.374 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 2021 2021-03-24T21:59:01.375 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: - 2021-03-24T21:59:01.375 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 03 2021-03-24T21:59:01.375 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: - 2021-03-24T21:59:01.375 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 24 2021-03-24T21:59:01.375 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: T 2021-03-24T21:59:01.376 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 21 2021-03-24T21:59:01.376 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: : 2021-03-24T21:59:01.376 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 59 2021-03-24T21:59:01.376 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: : 2021-03-24T21:59:01.376 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 00 2021-03-24T21:59:01.377 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: . 2021-03-24T21:59:01.377 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 393736 2021-03-24T21:59:01.377 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: +0000 2021-03-24T21:59:01.377 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 2021-03-24T21:59:01.377 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: osd.3 2021-03-24T21:59:01.378 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: ( 2021-03-24T21:59:01.378 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: osd 2021-03-24T21:59:01.378 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: . 2021-03-24T21:59:01.378 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 3 2021-03-24T21:59:01.379 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: ) 2021-03-24T21:59:01.379 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 82 2021-03-24T21:59:01.379 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: : 2021-03-24T21:59:01.379 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: cluster 2021-03-24T21:59:01.379 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 2021-03-24T21:59:01.380 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: [DBG] 2021-03-24T21:59:01.380 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 2021-03-24T21:59:01.380 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 3.18 scrub ok 2021-03-24T21:59:01.380 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 2021-03-24T21:59:01.380 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: cluster 2021-03-24T21:59:01.381 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 2021-03-24T21:59:01.381 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 2021 2021-03-24T21:59:01.381 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: - 2021-03-24T21:59:01.381 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 03 2021-03-24T21:59:01.381 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: - 2021-03-24T21:59:01.382 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 24 2021-03-24T21:59:01.382 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: T 2021-03-24T21:59:01.382 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 21 2021-03-24T21:59:01.382 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: : 2021-03-24T21:59:01.383 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 59 2021-03-24T21:59:01.383 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: : 2021-03-24T21:59:01.383 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 01 2021-03-24T21:59:01.383 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: . 2021-03-24T21:59:01.383 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 204834 2021-03-24T21:59:01.384 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: +0000 2021-03-24T21:59:01.384 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 2021-03-24T21:59:01.384 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: mon.a 2021-03-24T21:59:01.384 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: ( 2021-03-24T21:59:01.384 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: mon 2021-03-24T21:59:01.385 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: . 2021-03-24T21:59:01.385 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 0 2021-03-24T21:59:01.385 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: ) 2021-03-24T21:59:01.385 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 1139 2021-03-24T21:59:01.385 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: : 2021-03-24T21:59:01.386 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: cluster 2021-03-24T21:59:01.386 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 2021-03-24T21:59:01.386 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: [DBG] 2021-03-24T21:59:01.386 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 2021-03-24T21:59:01.386 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: osdmap e371: 8 total, 7 up, 6 in 2021-03-24T21:59:01.387 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:01 smithi125 conmon[19936]: 2021-03-24T21:59:01.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:01.397 INFO:tasks.rados.rados.0.smithi125.stdout:1337: expect (ObjNum 470 snap 133 seq_num 470) 2021-03-24T21:59:01.474 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-24T21:59:01.475 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T21:59:01.489 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:01.536 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:01.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:01.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:01.575 INFO:tasks.rados.rados.0.smithi125.stdout:1335: finishing rollback tid 0 to smithi12556470-22 2021-03-24T21:59:01.575 INFO:tasks.rados.rados.0.smithi125.stdout:1335: finishing rollback tid 1 to smithi12556470-22 2021-03-24T21:59:01.576 INFO:tasks.rados.rados.0.smithi125.stdout:1335: finishing rollback tid 2 to smithi12556470-22 2021-03-24T21:59:01.576 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 22 v 314 (ObjNum 468 snap 132 seq_num 468) dirty exists 2021-03-24T21:59:01.576 INFO:tasks.rados.rados.0.smithi125.stdout:1340: finishing write tid 1 to smithi12556470-6 2021-03-24T21:59:01.576 INFO:tasks.rados.rados.0.smithi125.stdout:1341: expect (ObjNum 478 snap 138 seq_num 478) 2021-03-24T21:59:01.622 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":716,"stamp":"2021-03-24T21:58:59.285543+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":271756138,"num_objects":436,"num_object_clones":74,"num_object_copies":1308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":242,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":436,"num_whiteouts":7,"num_read":3461,"num_read_kb":2083663,"num_write":1928,"num_write_kb":546144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":481,"num_bytes_recovered":98261234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7829,"ondisk_log_size":7829,"up":414,"acting":414,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":504,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":720584,"kb_used_data":587352,"kb_used_omap":0,"kb_used_meta":133184,"kb_avail":561750328,"statfs":{"total":575970213888,"available":575232335872,"internally_reserved":0,"allocated":601448448,"data_stored":746874045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":136380416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-343612,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":-46,"num_read_kb":-30656,"num_write":-34,"num_write_kb":-23313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-5112615,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.069214"},"pg_stats":[{"pgid":"7.23","version":"352'277","reported_seq":"1172","reported_epoch":"367","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:58:57.306117+0000","last_change":"2021-03-24T21:58:31.229425+0000","last_active":"2021-03-24T21:58:57.306117+0000","last_peered":"2021-03-24T21:58:57.306117+0000","last_clean":"2021-03-24T21:58:29.207011+0000","last_became_active":"2021-03-24T21:58:31.229425+0000","last_became_peered":"2021-03-24T21:58:31.229425+0000","last_unstale":"2021-03-24T21:58:57.306117+0000","last_undegraded":"2021-03-24T21:58:31.197028+0000","last_fullsized":"2021-03-24T21:58:31.196881+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":277,"ondisk_log_size":277,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"stat_sum":{"num_bytes":3437081,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":117123,"num_write":57,"num_write_kb":21228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":3418953,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.22","version":"366'79","reported_seq":"281","reported_epoch":"366","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:58:56.287139+0000","last_change":"2021-03-24T21:58:31.208603+0000","last_active":"2021-03-24T21:58:56.287139+0000","last_peered":"2021-03-24T21:58:56.287139+0000","last_clean":"2021-03-24T21:58:29.213948+0000","last_became_active":"2021-03-24T21:58:31.208603+0000","last_became_peered":"2021-03-24T21:58:31.208603+0000","last_unstale":"2021-03-24T21:58:56.287139+0000","last_undegraded":"2021-03-24T21:58:31.191893+0000","last_fullsized":"2021-03-24T21:58:31.191490+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":79,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"stat_sum":{"num_bytes":2958098,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":8520,"num_write":19,"num_write_kb":4979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":5774642,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"69'13","reported_seq":"401","reported_epoch":"366","state":"active+clean","last_fresh":"2021-03-24T21:58:56.264603+0000","last_change":"2021-03-24T21:58:27.722715+0000","last_active":"2021-03-24T21:58:56.264603+0000","last_peered":"2021-03-24T21:58:56.264603+0000","last_clean":"2021-03-24T21:58:56.264603+0000","last_became_active":"2021-03-24T21:54:38.127115+0000","last_became_peered":"2021-03-24T21:54:38.127115+0000","last_unstale":"2021-03-24T21:58:56.264603+0000","last_undegraded":"2021-03-24T21:58:56.264603+0000","last_fullsized":"2021-03-24T21:58:56.264603+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2021-03-24T21:58:27.722546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:58:27.722546+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":13,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"328","reported_epoch":"367","state":"active+clean","last_fresh":"2021-03-24T21:58:57.306472+0000","last_change":"2021-03-24T21:56:42.413023+0000","last_active":"2021-03-24T21:58:57.306472+0000","last_peered":"2021-03-24T21:58:57.306472+0000","last_clean":"2021-03-24T21:58:57.306472+0000","last_became_active":"2021-03-24T21:51:45.375830+0000","last_became_peered":"2021-03-24T21:51:45.375830+0000","last_unstale":"2021-03-24T21:58:57.306472+0000","last_undegraded":"2021-03-24T21:58:57.306472+0000","last_fullsized":"2021-03-24T21:58:57.306472+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:42.412849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:42.412849+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"343'5","reported_seq":"423","reported_epoch":"365","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:58:55.226064+0000","last_change":"2021-03-24T21:58:31.205289+0000","last_active":"2021-03-24T21:58:55.226064+0000","last_peered":"2021-03-24T21:58:55.226064+0000","last_clean":"2021-03-24T21:58:29.217483+0000","last_became_active":"2021-03-24T21:58:31.205289+0000","last_became_peered":"2021-03-24T21:58:31.205289+0000","last_unstale":"2021-03-24T21:58:55.226064+0000","last_undegraded":"2021-03-24T21:58:31.194525+0000","last_fullsized":"2021-03-24T21:58:31.194414+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2021-03-24T21:57:54.241842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:57:54.241842+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[{"shards":"2,7","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"341","reported_epoch":"366","state":"active+undersized","last_fresh":"2021-03-24T21:58:56.270534+0000","last_change":"2021-03-24T21:58:31.202498+0000","last_active":"2021-03-24T21:58:56.270534+0000","last_peered":"2021-03-24T21:58:56.270534+0000","last_clean":"2021-03-24T21:58:29.218000+0000","last_became_active":"2021-03-24T21:58:31.202498+0000","last_became_peered":"2021-03-24T21:58:31.202498+0000","last_unstale":"2021-03-24T21:58:56.270534+0000","last_undegraded":"2021-03-24T21:58:56.270534+0000","last_fullsized":"2021-03-24T21:58:31.194729+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:08.985694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:08.985694+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"364'160","reported_seq":"1043","reported_epoch":"367","state":"active+clean","last_fresh":"2021-03-24T21:58:57.283623+0000","last_change":"2021-03-24T21:58:55.258824+0000","last_active":"2021-03-24T21:58:57.283623+0000","last_peered":"2021-03-24T21:58:57.283623+0000","last_clean":"2021-03-24T21:58:57.283623+0000","last_became_active":"2021-03-24T21:57:26.178185+0000","last_became_peered":"2021-03-24T21:57:26.178185+0000","last_unstale":"2021-03-24T21:58:57.283623+0000","last_undegraded":"2021-03-24T21:58:57.283623+0000","last_fullsized":"2021-03-24T21:58:57.283623+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"218'73","last_scrub_stamp":"2021-03-24T21:56:48.267711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:56:48.267711+0000","log_size":160,"ondisk_log_size":160,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8123972,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":71,"num_read_kb":83924,"num_write":43,"num_write_kb":22665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":942482,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"df","length":"1"},{"start":"e1","length":"1"},{"start":"ea","length":"4"},{"start":"f2","length":"3"},{"start":"fc","length":"1"}]},{"pgid":"4.d","version":"69'21","reported_seq":"397","reported_epoch":"366","state":"active+clean","last_fresh":"2021-03-24T21:58:56.264257+0000","last_change":"2021-03-24T21:57:18.801773+0000","last_active":"2021-03-24T21:58:56.264257+0000","last_peered":"2021-03-24T21:58:56.264257+0000","last_clean":"2021-03-24T21:58:56.264257+0000","last_became_active":"2021-03-24T21:54:38.144020+0000","last_became_peered":"2021-03-24T21:54:38.144020+0000","last_unstale":"2021-03-24T21:58:56.264257+0000","last_undegraded":"2021-03-24T21:58:56.264257+0000","last_fullsized":"2021-03-24T21:58:56.264257+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'21","last_scrub_stamp":"2021-03-24T21:57:18.801567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:57:18.801567+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":23,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"301","reported_epoch":"366","state":"active+undersized","last_fresh":"2021-03-24T21:58:56.266301+0000","last_change":"2021-03-24T21:58:31.223148+0000","last_active":"2021-03-24T21:58:56.266301+0000","last_peered":"2021-03-24T21:58:56.266301+0000","last_clean":"2021-03-24T21:58:29.214275+0000","last_became_active":"2021-03-24T21:58:31.223148+0000","last_became_peered":"2021-03-24T21:58:31.223148+0000","last_unstale":"2021-03-24T21:58:56.266301+0000","last_undegraded":"2021-03-24T21:58:56.266301+0000","last_fullsized":"2021-03-24T21:58:31.188534+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:50.522344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:55:01.169461+0000","last_clean_scrub_stamp":"2021-03-24T21:57:50.522344+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"290","reported_epoch":"366","state":"active+undersized","last_fresh":"2021-03-24T21:58:56.264156+0000","last_change":"2021-03-24T21:58:31.207750+0000","last_active":"2021-03-24T21:58:56.264156+0000","last_peered":"2021-03-24T21:58:56.264156+0000","last_clean":"2021-03-24T21:58:29.215104+0000","last_became_active":"2021-03-24T21:58:31.207750+0000","last_became_peered":"2021-03-24T21:58:31.207750+0000","last_unstale":"2021-03-24T21:58:56.264156+0000","last_undegraded":"2021-03-24T21:58:56.264156+0000","last_fullsized":"2021-03-24T21:58:31.192794+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:07.977138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:07.977138+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"133","reported_epoch":"367","state":"active+undersized","last_fresh":"2021-03-24T21:58:57.297096+0000","last_change":"2021-03-24T21:58:31.228194+0000","last_active":"2021-03-24T21:58:57.297096+0000","last_peered":"2021-03-24T21:58:57.297096+0000","last_clean":"2021-03-24T21:58:29.208259+0000","last_became_active":"2021-03-24T21:58:31.228194+0000","last_became_peered":"2021-03-24T21:58:31.228194+0000","last_unstale":"2021-03-24T21:58:57.297096+0000","last_undegraded":"2021-03-24T21:58:57.297096+0000","last_fullsized":"2021-03-24T21:58:31.194809+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:49.970073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:49.970073+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"318'49","reported_seq":"856","reported_epoch":"365","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:58:55.225574+0000","last_change":"2021-03-24T21:58:31.205434+0000","last_active":"2021-03-24T21:58:55.225574+0000","last_peered":"2021-03-24T21:58:55.225574+0000","last_clean":"2021-03-24T21:58:29.217512+0000","last_became_active":"2021-03-24T21:58:31.205434+0000","last_became_peered":"2021-03-24T21:58:31.205434+0000","last_unstale":"2021-03-24T21:58:55.225574+0000","last_undegraded":"2021-03-24T21:58:31.193957+0000","last_fullsized":"2021-03-24T21:58:31.193806+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'45","last_scrub_stamp":"2021-03-24T21:57:27.105464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:57:27.105464+0000","log_size":49,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"stat_sum":{"num_bytes":2857256,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":1214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"69'15","reported_seq":"417","reported_epoch":"366","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:58:56.270219+0000","last_change":"2021-03-24T21:58:31.223443+0000","last_active":"2021-03-24T21:58:56.270219+0000","last_peered":"2021-03-24T21:58:56.270219+0000","last_clean":"2021-03-24T21:58:29.217662+0000","last_became_active":"2021-03-24T21:58:31.223443+0000","last_became_peered":"2021-03-24T21:58:31.223443+0000","last_unstale":"2021-03-24T21:58:56.270219+0000","last_undegraded":"2021-03-24T21:58:31.189044+0000","last_fullsized":"2021-03-24T21:58:31.188912+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:56:40.062304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:40.062304+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1476,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":9}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"328","reported_epoch":"366","state":"active+clean","last_fresh":"2021-03-24T21:58:56.261393+0000","last_change":"2021-03-24T21:57:55.592974+0000","last_active":"2021-03-24T21:58:56.261393+0000","last_peered":"2021-03-24T21:58:56.261393+0000","last_clean":"2021-03-24T21:58:56.261393+0000","last_became_active":"2021-03-24T21:51:45.579513+0000","last_became_peered":"2021-03-24T21:51:45.579513+0000","last_unstale":"2021-03-24T21:58:56.261393+0000","last_undegraded":"2021-03-24T21:58:56.261393+0000","last_fullsized":"2021-03-24T21:58:56.261393+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:55.592795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:55.592795+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"307","reported_epoch":"366","state":"active+undersized","last_fresh":"2021-03-24T21:58:56.271296+0000","last_change":"2021-03-24T21:58:31.200145+0000","last_active":"2021-03-24T21:58:56.271296+0000","last_peered":"2021-03-24T21:58:56.271296+0000","last_clean":"2021-03-24T21:58:29.218572+0000","last_became_active":"2021-03-24T21:58:31.200145+0000","last_became_peered":"2021-03-24T21:58:31.200145+0000","last_unstale":"2021-03-24T21:58:56.271296+0000","last_undegraded":"2021-03-24T21:58:56.271296+0000","last_fullsized":"2021-03-24T21:58:31.196922+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:43.970879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:43.970879+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"332","reported_epoch":"367","state":"active+undersized","last_fresh":"2021-03-24T21:58:57.305190+0000","last_change":"2021-03-24T21:58:31.207098+0000","last_active":"2021-03-24T21:58:57.305190+0000","last_peered":"2021-03-24T21:58:57.305190+0000","last_clean":"2021-03-24T21:58:29.207688+0000","last_became_active":"2021-03-24T21:58:31.207098+0000","last_became_peered":"2021-03-24T21:58:31.207098+0000","last_unstale":"2021-03-24T21:58:57.305190+0000","last_undegraded":"2021-03-24T21:58:57.305190+0000","last_fullsized":"2021-03-24T21:58:31.188164+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:59.472393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:55:59.472393+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"360'344","reported_seq":"1201","reported_epoch":"367","state":"active+clean","last_fresh":"2021-03-24T21:58:57.283194+0000","last_change":"2021-03-24T21:58:55.236638+0000","last_active":"2021-03-24T21:58:57.283194+0000","last_peered":"2021-03-24T21:58:57.283194+0000","last_clean":"2021-03-24T21:58:57.283194+0000","last_became_active":"2021-03-24T21:57:26.189692+0000","last_became_peered":"2021-03-24T21:57:26.189692+0000","last_unstale":"2021-03-24T21:58:57.283194+0000","last_undegraded":"2021-03-24T21:58:57.283194+0000","last_fullsized":"2021-03-24T21:58:57.283194+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'169","last_scrub_stamp":"2021-03-24T21:56:47.285648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:56:47.285648+0000","log_size":344,"ondisk_log_size":344,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7882726,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":95,"num_read_kb":91229,"num_write":91,"num_write_kb":31566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2546506,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"df","length":"1"},{"start":"e1","length":"1"},{"start":"ea","length":"4"},{"start":"f2","length":"3"},{"start":"fc","length":"1"}]},{"pgid":"4.f","version":"274'26","reported_seq":"345","reported_epoch":"367","state":"active+clean","last_fresh":"2021-03-24T21:58:57.297456+0000","last_change":"2021-03-24T21:57:24.499178+0000","last_active":"2021-03-24T21:58:57.297456+0000","last_peered":"2021-03-24T21:58:57.297456+0000","last_clean":"2021-03-24T21:58:57.297456+0000","last_became_active":"2021-03-24T21:54:38.130536+0000","last_became_peered":"2021-03-24T21:54:38.130536+0000","last_unstale":"2021-03-24T21:58:57.297456+0000","last_undegraded":"2021-03-24T21:58:57.297456+0000","last_fullsized":"2021-03-24T21:58:57.297456+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'26","last_scrub_stamp":"2021-03-24T21:57:24.499090+0000","last_deep_scrub":"274'26","last_deep_scrub_stamp":"2021-03-24T21:57:24.499090+0000","last_clean_scrub_stamp":"2021-03-24T21:57:24.499090+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":18,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":422,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":"332","reported_epoch":"367","state":"active+clean","last_fresh":"2021-03-24T21:58:57.290787+0000","last_change":"2021-03-24T21:56:39.439282+0000","last_active":"2021-03-24T21:58:57.290787+0000","last_peered":"2021-03-24T21:58:57.290787+0000","last_clean":"2021-03-24T21:58:57.290787+0000","last_became_active":"2021-03-24T21:54:38.134768+0000","last_became_peered":"2021-03-24T21:54:38.134768+0000","last_unstale":"2021-03-24T21:58:57.290787+0000","last_undegraded":"2021-03-24T21:58:57.290787+0000","last_fullsized":"2021-03-24T21:58:57.290787+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:39.439170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:39.439170+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"343'3","reported_seq":"412","reported_epoch":"366","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:58:56.270423+0000","last_change":"2021-03-24T21:58:31.208494+0000","last_active":"2021-03-24T21:58:56.270423+0000","last_peered":"2021-03-24T21:58:56.270423+0000","last_clean":"2021-03-24T21:58:29.217762+0000","last_became_active":"2021-03-24T21:58:31.208494+0000","last_became_peered":"2021-03-24T21:58:31.208494+0000","last_unstale":"2021-03-24T21:58:56.270423+0000","last_undegraded":"2021-03-24T21:58:31.194480+0000","last_fullsized":"2021-03-24T21:58:31.194360+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2021-03-24T21:55:59.032170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:55:59.032170+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"92","reported_epoch":"367","state":"active+undersized","last_fresh":"2021-03-24T21:58:57.284106+0000","last_change":"2021-03-24T21:58:31.200312+0000","last_active":"2021-03-24T21:58:57.284106+0000","last_peered":"2021-03-24T21:58:57.284106+0000","last_clean":"2021-03-24T21:54:36.087752+0000","last_became_active":"2021-03-24T21:58:31.200312+0000","last_became_peered":"2021-03-24T21:58:31.200312+0000","last_unstale":"2021-03-24T21:58:57.284106+0000","last_undegraded":"2021-03-24T21:58:57.284106+0000","last_fullsized":"2021-03-24T21:58:31.192375+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:23.176750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:57:23.176750+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.c","version":"365'371","reported_seq":"1230","reported_epoch":"367","state":"active+clean","last_fresh":"2021-03-24T21:58:57.305696+0000","last_change":"2021-03-24T21:58:55.263562+0000","last_active":"2021-03-24T21:58:57.305696+0000","last_peered":"2021-03-24T21:58:57.305696+0000","last_clean":"2021-03-24T21:58:57.305696+0000","last_became_active":"2021-03-24T21:57:26.174174+0000","last_became_peered":"2021-03-24T21:57:26.174174+0000","last_unstale":"2021-03-24T21:58:57.305696+0000","last_undegraded":"2021-03-24T21:58:57.305696+0000","last_fullsized":"2021-03-24T21:58:57.305696+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":371,"ondisk_log_size":371,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8990718,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":2,"num_read":152,"num_read_kb":153685,"num_write":95,"num_write_kb":36181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"df","length":"1"},{"start":"e1","length":"1"},{"start":"ea","length":"4"},{"start":"f2","length":"3"},{"start":"fc","length":"1"}]},{"pgid":"5.1","version":"0'0","reported_seq":"324","reported_epoch":"366","state":"active+clean","last_fresh":"2021-03-24T21:58:56.261166+0000","last_change":"2021-03-24T21:57:59.462088+0000","last_active":"2021-03-24T21:58:56.261166+0000","last_peered":"2021-03-24T21:58:56.261166+0000","last_clean":"2021-03-24T21:58:56.261166+0000","last_became_active":"2021-03-24T21:51:49.408761+0000","last_became_peered":"2021-03-24T21:51:49.408761+0000","last_unstale":"2021-03-24T21:58:56.261166+0000","last_undegraded":"2021-03-24T21:58:56.261166+0000","last_fullsized":"2021-03-24T21:58:56.261166+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:59.461865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:25.220436+0000","last_clean_scrub_stamp":"2021-03-24T21:57:59.461865+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"69'22","reported_seq":"366","reported_epoch":"367","state":"active+clean","last_fresh":"2021-03-24T21:58:57.290850+0000","last_change":"2021-03-24T21:58:56.473659+0000","last_active":"2021-03-24T21:58:57.290850+0000","last_peered":"2021-03-24T21:58:57.290850+0000","last_clean":"2021-03-24T21:58:57.290850+0000","last_became_active":"2021-03-24T21:51:47.691459+0000","last_became_peered":"2021-03-24T21:51:47.691459+0000","last_unstale":"2021-03-24T21:58:57.290850+0000","last_undegraded":"2021-03-24T21:58:57.290850+0000","last_fullsized":"2021-03-24T21:58:57.290850+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'22","last_scrub_stamp":"2021-03-24T21:58:56.473510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:58:56.473510+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":46,"num_read_kb":26,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"328","reported_epoch":"367","state":"active+clean","last_fresh":"2021-03-24T21:58:57.297403+0000","last_change":"2021-03-24T21:57:54.381907+0000","last_active":"2021-03-24T21:58:57.297403+0000","last_peered":"2021-03-24T21:58:57.297403+0000","last_clean":"2021-03-24T21:58:57.297403+0000","last_became_active":"2021-03-24T21:51:45.378704+0000","last_became_peered":"2021-03-24T21:51:45.378704+0000","last_unstale":"2021-03-24T21:58:57.297403+0000","last_undegraded":"2021-03-24T21:58:57.297403+0000","last_fullsized":"2021-03-24T21:58:57.297403+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:54.381668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:54.381668+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"317","reported_epoch":"366","state":"active+clean","last_fresh":"2021-03-24T21:58:56.266202+0000","last_change":"2021-03-24T21:56:35.045715+0000","last_active":"2021-03-24T21:58:56.266202+0000","last_peered":"2021-03-24T21:58:56.266202+0000","last_clean":"2021-03-24T21:58:56.266202+0000","last_became_active":"2021-03-24T21:51:51.756114+0000","last_became_peered":"2021-03-24T21:51:51.756114+0000","last_unstale":"2021-03-24T21:58:56.266202+0000","last_undegraded":"2021-03-24T21:58:56.266202+0000","last_fullsized":"2021-03-24T21:58:56.266202+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:35.045534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:35.045534+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"365'308","reported_seq":"1223","reported_epoch":"367","state":"active+clean","last_fresh":"2021-03-24T21:58:57.311061+0000","last_change":"2021-03-24T21:58:55.254806+0000","last_active":"2021-03-24T21:58:57.311061+0000","last_peered":"2021-03-24T21:58:57.311061+0000","last_clean":"2021-03-24T21:58:57.311061+0000","last_became_active":"2021-03-24T21:57:26.169302+0000","last_became_peered":"2021-03-24T21:57:26.169302+0000","last_unstale":"2021-03-24T21:58:57.311061+0000","last_undegraded":"2021-03-24T21:58:57.311061+0000","last_fullsized":"2021-03-24T21:58:57.311061+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":308,"ondisk_log_size":308,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4107509,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":149,"num_read_kb":182543,"num_write":59,"num_write_kb":23028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"df","length":"1"},{"start":"e1","length":"1"},{"start":"ea","length":"4"},{"start":"f2","length":"3"},{"start":"fc","length":"1"}]},{"pgid":"5.0","version":"109'3","reported_seq":"425","reported_epoch":"368","state":"active+clean","last_fresh":"2021-03-24T21:58:58.711670+0000","last_change":"2021-03-24T21:57:25.463136+0000","last_active":"2021-03-24T21:58:58.711670+0000","last_peered":"2021-03-24T21:58:58.711670+0000","last_clean":"2021-03-24T21:58:58.711670+0000","last_became_active":"2021-03-24T21:54:38.135726+0000","last_became_peered":"2021-03-24T21:54:38.135726+0000","last_unstale":"2021-03-24T21:58:58.711670+0000","last_undegraded":"2021-03-24T21:58:58.711670+0000","last_fullsized":"2021-03-24T21:58:58.711670+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'3","last_scrub_stamp":"2021-03-24T21:57:25.462952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:57:25.462952+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"69'20","reported_seq":"402","reported_epoch":"366","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:58:56.261667+0000","last_change":"2021-03-24T21:58:31.207847+0000","last_active":"2021-03-24T21:58:56.261667+0000","last_peered":"2021-03-24T21:58:56.261667+0000","last_clean":"2021-03-24T21:58:29.215347+0000","last_became_active":"2021-03-24T21:58:31.207847+0000","last_became_peered":"2021-03-24T21:58:31.207847+0000","last_unstale":"2021-03-24T21:58:56.261667+0000","last_undegraded":"2021-03-24T21:58:31.193521+0000","last_fullsized":"2021-03-24T21:58:31.193193+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'20","last_scrub_stamp":"2021-03-24T21:57:21.706127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:57:21.706127+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":39,"num_read_kb":20,"num_write":20,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":940,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":13}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"347","reported_epoch":"367","state":"active+undersized","last_fresh":"2021-03-24T21:58:57.285736+0000","last_change":"2021-03-24T21:58:31.208601+0000","last_active":"2021-03-24T21:58:57.285736+0000","last_peered":"2021-03-24T21:58:57.285736+0000","last_clean":"2021-03-24T21:58:29.213618+0000","last_became_active":"2021-03-24T21:58:31.208601+0000","last_became_peered":"2021-03-24T21:58:31.208601+0000","last_unstale":"2021-03-24T21:58:57.285736+0000","last_undegraded":"2021-03-24T21:58:57.285736+0000","last_fullsized":"2021-03-24T21:58:31.190336+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:28.165408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:28.165408+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"336","reported_epoch":"363","state":"active+clean","last_fresh":"2021-03-24T21:58:53.504097+0000","last_change":"2021-03-24T21:56:40.011790+0000","last_active":"2021-03-24T21:58:53.504097+0000","last_peered":"2021-03-24T21:58:53.504097+0000","last_clean":"2021-03-24T21:58:53.504097+0000","last_became_active":"2021-03-24T21:54:38.110246+0000","last_became_peered":"2021-03-24T21:54:38.110246+0000","last_unstale":"2021-03-24T21:58:53.504097+0000","last_undegraded":"2021-03-24T21:58:53.504097+0000","last_fullsized":"2021-03-24T21:58:53.504097+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:40.011633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:40.011633+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"272'64","reported_seq":"912","reported_epoch":"367","state":"active+clean","last_fresh":"2021-03-24T21:58:57.292767+0000","last_change":"2021-03-24T21:58:55.236822+0000","last_active":"2021-03-24T21:58:57.292767+0000","last_peered":"2021-03-24T21:58:57.292767+0000","last_clean":"2021-03-24T21:58:57.292767+0000","last_became_active":"2021-03-24T21:57:26.174158+0000","last_became_peered":"2021-03-24T21:57:26.174158+0000","last_unstale":"2021-03-24T21:58:57.292767+0000","last_undegraded":"2021-03-24T21:58:57.292767+0000","last_fullsized":"2021-03-24T21:58:57.292767+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":64,"ondisk_log_size":64,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1922978,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":8518,"num_write":11,"num_write_kb":3706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":216931,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"df","length":"1"},{"start":"e1","length":"1"},{"start":"ea","length":"4"},{"start":"f2","length":"3"},{"start":"fc","length":"1"}]},{"pgid":"5.3","version":"343'5","reported_seq":"426","reported_epoch":"367","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:58:57.285077+0000","last_change":"2021-03-24T21:58:31.223346+0000","last_active":"2021-03-24T21:58:57.285077+0000","last_peered":"2021-03-24T21:58:57.285077+0000","last_clean":"2021-03-24T21:58:29.216175+0000","last_became_active":"2021-03-24T21:58:31.223346+0000","last_became_peered":"2021-03-24T21:58:31.223346+0000","last_unstale":"2021-03-24T21:58:57.285077+0000","last_undegraded":"2021-03-24T21:58:31.193746+0000","last_fullsized":"2021-03-24T21:58:31.193622+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'4","last_scrub_stamp":"2021-03-24T21:56:01.266936+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-24T21:53:26.584678+0000","last_clean_scrub_stamp":"2021-03-24T21:56:01.266936+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"69'16","reported_seq":"75","reported_epoch":"366","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:58:56.266725+0000","last_change":"2021-03-24T21:58:31.209014+0000","last_active":"2021-03-24T21:58:56.266725+0000","last_peered":"2021-03-24T21:58:56.266725+0000","last_clean":"2021-03-24T21:51:54.404218+0000","last_became_active":"2021-03-24T21:58:31.209014+0000","last_became_peered":"2021-03-24T21:58:31.209014+0000","last_unstale":"2021-03-24T21:58:56.266725+0000","last_undegraded":"2021-03-24T21:58:31.194887+0000","last_fullsized":"2021-03-24T21:58:31.194702+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T21:56:41.235143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:41.235143+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":32,"num_read_kb":15,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":11}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"36","reported_epoch":"367","state":"active+undersized","last_fresh":"2021-03-24T21:58:57.316625+0000","last_change":"2021-03-24T21:58:31.210981+0000","last_active":"2021-03-24T21:58:57.316625+0000","last_peered":"2021-03-24T21:58:57.316625+0000","last_clean":"2021-03-24T21:51:44.337765+0000","last_became_active":"2021-03-24T21:58:31.210981+0000","last_became_peered":"2021-03-24T21:58:31.210981+0000","last_unstale":"2021-03-24T21:58:57.316625+0000","last_undegraded":"2021-03-24T21:58:57.316625+0000","last_fullsized":"2021-03-24T21:58:31.196029+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:53.013129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:53.013129+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"317","reported_epoch":"367","state":"active+clean","last_fresh":"2021-03-24T21:58:57.282844+0000","last_change":"2021-03-24T21:55:39.261965+0000","last_active":"2021-03-24T21:58:57.282844+0000","last_peered":"2021-03-24T21:58:57.282844+0000","last_clean":"2021-03-24T21:58:57.282844+0000","last_became_active":"2021-03-24T21:51:51.759627+0000","last_became_peered":"2021-03-24T21:51:51.759627+0000","last_unstale":"2021-03-24T21:58:57.282844+0000","last_undegraded":"2021-03-24T21:58:57.282844+0000","last_fullsized":"2021-03-24T21:58:57.282844+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:39.261698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:55:39.261698+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"365'287","reported_seq":"1254","reported_epoch":"366","state":"active+clean","last_fresh":"2021-03-24T21:58:56.648191+0000","last_change":"2021-03-24T21:58:56.648146+0000","last_active":"2021-03-24T21:58:56.648191+0000","last_peered":"2021-03-24T21:58:56.648191+0000","last_clean":"2021-03-24T21:58:56.648191+0000","last_became_active":"2021-03-24T21:57:26.191103+0000","last_became_peered":"2021-03-24T21:57:26.191103+0000","last_unstale":"2021-03-24T21:58:56.648191+0000","last_undegraded":"2021-03-24T21:58:56.648191+0000","last_fullsized":"2021-03-24T21:58:56.648191+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"365'287","last_scrub_stamp":"2021-03-24T21:58:56.648040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:58:56.648040+0000","log_size":287,"ondisk_log_size":287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6689285,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"df","length":"1"},{"start":"e1","length":"1"},{"start":"ea","length":"4"},{"start":"f2","length":"3"},{"start":"fc","length":"1"}]},{"pgid":"5.2","version":"0'0","reported_seq":"288","reported_epoch":"367","state":"active+undersized","last_fresh":"2021-03-24T21:58:57.284117+0000","last_change":"2021-03-24T21:58:31.199463+0000","last_active":"2021-03-24T21:58:57.284117+0000","last_peered":"2021-03-24T21:58:57.284117+0000","last_clean":"2021-03-24T21:58:29.212689+0000","last_became_active":"2021-03-24T21:58:31.199463+0000","last_became_peered":"2021-03-24T21:58:31.199463+0000","last_unstale":"2021-03-24T21:58:57.284117+0000","last_undegraded":"2021-03-24T21:58:57.284117+0000","last_fullsized":"2021-03-24T21:58:31.192367+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:58.253777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:55:58.253777+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"69'21","reported_seq":"372","reported_epoch":"367","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:58:57.284921+0000","last_change":"2021-03-24T21:58:31.209599+0000","last_active":"2021-03-24T21:58:57.284921+0000","last_peered":"2021-03-24T21:58:57.284921+0000","last_clean":"2021-03-24T21:58:29.216071+0000","last_became_active":"2021-03-24T21:58:31.209599+0000","last_became_peered":"2021-03-24T21:58:31.209599+0000","last_unstale":"2021-03-24T21:58:57.284921+0000","last_undegraded":"2021-03-24T21:58:31.193410+0000","last_fullsized":"2021-03-24T21:58:31.193294+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'21","last_scrub_stamp":"2021-03-24T21:56:39.335613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:39.335613+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":45,"num_read_kb":27,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":10}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"42","reported_epoch":"367","state":"active+undersized","last_fresh":"2021-03-24T21:58:57.286270+0000","last_change":"2021-03-24T21:58:31.199174+0000","last_active":"2021-03-24T21:58:57.286270+0000","last_peered":"2021-03-24T21:58:57.286270+0000","last_clean":"2021-03-24T21:51:44.337765+0000","last_became_active":"2021-03-24T21:58:31.199174+0000","last_became_peered":"2021-03-24T21:58:31.199174+0000","last_unstale":"2021-03-24T21:58:57.286270+0000","last_undegraded":"2021-03-24T21:58:57.286270+0000","last_fullsized":"2021-03-24T21:58:31.189176+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:38.144956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:38.144956+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"185","reported_epoch":"363","state":"active+undersized","last_fresh":"2021-03-24T21:58:53.503504+0000","last_change":"2021-03-24T21:58:31.224919+0000","last_active":"2021-03-24T21:58:53.503504+0000","last_peered":"2021-03-24T21:58:53.503504+0000","last_clean":"2021-03-24T21:58:29.200711+0000","last_became_active":"2021-03-24T21:58:31.224919+0000","last_became_peered":"2021-03-24T21:58:31.224919+0000","last_unstale":"2021-03-24T21:58:53.503504+0000","last_undegraded":"2021-03-24T21:58:53.503504+0000","last_fullsized":"2021-03-24T21:58:31.196827+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:24.967391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:58:24.967391+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.0","version":"175'22","reported_seq":"295","reported_epoch":"367","state":"active+undersized","last_fresh":"2021-03-24T21:58:57.290113+0000","last_change":"2021-03-24T21:58:31.197067+0000","last_active":"2021-03-24T21:58:57.290113+0000","last_peered":"2021-03-24T21:58:57.290113+0000","last_clean":"2021-03-24T21:55:44.382770+0000","last_became_active":"2021-03-24T21:58:31.197067+0000","last_became_peered":"2021-03-24T21:58:31.197067+0000","last_unstale":"2021-03-24T21:58:57.290113+0000","last_undegraded":"2021-03-24T21:58:57.290113+0000","last_fullsized":"2021-03-24T21:58:31.184568+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'22","last_scrub_stamp":"2021-03-24T21:58:22.940875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:58:22.940875+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3283,"num_write":3,"num_write_kb":845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":313386,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"94","reported_epoch":"367","state":"active+undersized","last_fresh":"2021-03-24T21:58:57.285260+0000","last_change":"2021-03-24T21:58:31.218358+0000","last_active":"2021-03-24T21:58:57.285260+0000","last_peered":"2021-03-24T21:58:57.285260+0000","last_clean":"2021-03-24T21:54:36.088853+0000","last_became_active":"2021-03-24T21:58:31.218358+0000","last_became_peered":"2021-03-24T21:58:31.218358+0000","last_unstale":"2021-03-24T21:58:57.285260+0000","last_undegraded":"2021-03-24T21:58:57.285260+0000","last_fullsized":"2021-03-24T21:58:31.191469+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:43.194160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:43.194160+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"69'16","reported_seq":"420","reported_epoch":"367","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:58:57.285484+0000","last_change":"2021-03-24T21:58:31.208049+0000","last_active":"2021-03-24T21:58:57.285484+0000","last_peered":"2021-03-24T21:58:57.285484+0000","last_clean":"2021-03-24T21:58:29.213527+0000","last_became_active":"2021-03-24T21:58:31.208049+0000","last_became_peered":"2021-03-24T21:58:31.208049+0000","last_unstale":"2021-03-24T21:58:57.285484+0000","last_undegraded":"2021-03-24T21:58:31.189256+0000","last_fullsized":"2021-03-24T21:58:31.189131+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T21:57:26.257686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:57:26.257686+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":16,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1260,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":10}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"66'5","reported_seq":"771","reported_epoch":"363","state":"active+clean","last_fresh":"2021-03-24T21:58:53.621722+0000","last_change":"2021-03-24T21:56:37.973047+0000","last_active":"2021-03-24T21:58:53.621722+0000","last_peered":"2021-03-24T21:58:53.621722+0000","last_clean":"2021-03-24T21:58:53.621722+0000","last_became_active":"2021-03-24T21:54:38.125434+0000","last_became_peered":"2021-03-24T21:54:38.125434+0000","last_unstale":"2021-03-24T21:58:53.621722+0000","last_undegraded":"2021-03-24T21:58:53.621722+0000","last_fullsized":"2021-03-24T21:58:53.621722+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2021-03-24T21:56:37.972874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:39.286101+0000","last_clean_scrub_stamp":"2021-03-24T21:56:37.972874+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":426,"num_read_kb":421,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"328","reported_epoch":"367","state":"active+clean","last_fresh":"2021-03-24T21:58:57.282464+0000","last_change":"2021-03-24T21:56:38.304954+0000","last_active":"2021-03-24T21:58:57.282464+0000","last_peered":"2021-03-24T21:58:57.282464+0000","last_clean":"2021-03-24T21:58:57.282464+0000","last_became_active":"2021-03-24T21:51:45.376687+0000","last_became_peered":"2021-03-24T21:51:45.376687+0000","last_unstale":"2021-03-24T21:58:57.282464+0000","last_undegraded":"2021-03-24T21:58:57.282464+0000","last_fullsized":"2021-03-24T21:58:57.282464+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:38.304792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:38.304792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"126","reported_epoch":"367","state":"active+clean","last_fresh":"2021-03-24T21:58:58.034282+0000","last_change":"2021-03-24T21:58:58.034141+0000","last_active":"2021-03-24T21:58:58.034282+0000","last_peered":"2021-03-24T21:58:58.034282+0000","last_clean":"2021-03-24T21:58:58.034282+0000","last_became_active":"2021-03-24T21:57:06.152772+0000","last_became_peered":"2021-03-24T21:57:06.152772+0000","last_unstale":"2021-03-24T21:58:58.034282+0000","last_undegraded":"2021-03-24T21:58:58.034282+0000","last_fullsized":"2021-03-24T21:58:58.034282+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:58.033876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:58:58.033876+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.5","version":"195'69","reported_seq":"895","reported_epoch":"364","state":"active+clean","last_fresh":"2021-03-24T21:58:54.264938+0000","last_change":"2021-03-24T21:58:54.265980+0000","last_active":"2021-03-24T21:58:54.264938+0000","last_peered":"2021-03-24T21:58:54.264938+0000","last_clean":"2021-03-24T21:58:54.264938+0000","last_became_active":"2021-03-24T21:56:08.152168+0000","last_became_peered":"2021-03-24T21:56:08.152168+0000","last_unstale":"2021-03-24T21:58:54.264938+0000","last_undegraded":"2021-03-24T21:58:54.264938+0000","last_fullsized":"2021-03-24T21:58:54.264938+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'69","last_scrub_stamp":"2021-03-24T21:56:49.886058+0000","last_deep_scrub":"195'69","last_deep_scrub_stamp":"2021-03-24T21:56:49.886058+0000","last_clean_scrub_stamp":"2021-03-24T21:56:49.886058+0000","log_size":69,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"df","length":"1"},{"start":"e1","length":"1"},{"start":"ea","length":"3"},{"start":"f2","length":"3"},{"start":"fc","length":"1"}]},{"pgid":"5.4","version":"0'0","reported_seq":"326","reported_epoch":"363","state":"active+undersized","last_fresh":"2021-03-24T21:58:53.504348+0000","last_change":"2021-03-24T21:58:31.209266+0000","last_active":"2021-03-24T21:58:53.504348+0000","last_peered":"2021-03-24T21:58:53.504348+0000","last_clean":"2021-03-24T21:58:29.202947+0000","last_became_active":"2021-03-24T21:58:31.209266+0000","last_became_peered":"2021-03-24T21:58:31.209266+0000","last_unstale":"2021-03-24T21:58:53.504348+0000","last_undegraded":"2021-03-24T21:58:53.504348+0000","last_fullsized":"2021-03-24T21:58:31.199943+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:18.199919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:57:18.199919+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"69'15","reported_seq":"343","reported_epoch":"366","state":"active+clean","last_fresh":"2021-03-24T21:58:56.264643+0000","last_change":"2021-03-24T21:58:20.702582+0000","last_active":"2021-03-24T21:58:56.264643+0000","last_peered":"2021-03-24T21:58:56.264643+0000","last_clean":"2021-03-24T21:58:56.264643+0000","last_became_active":"2021-03-24T21:54:24.194711+0000","last_became_peered":"2021-03-24T21:54:24.194711+0000","last_unstale":"2021-03-24T21:58:56.264643+0000","last_undegraded":"2021-03-24T21:58:56.264643+0000","last_fullsized":"2021-03-24T21:58:56.264643+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:58:20.702393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:58:20.702393+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":26,"num_read_kb":14,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":466,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"338","reported_epoch":"367","state":"active+undersized","last_fresh":"2021-03-24T21:58:57.292927+0000","last_change":"2021-03-24T21:58:31.207564+0000","last_active":"2021-03-24T21:58:57.292927+0000","last_peered":"2021-03-24T21:58:57.292927+0000","last_clean":"2021-03-24T21:58:29.207031+0000","last_became_active":"2021-03-24T21:58:31.207564+0000","last_became_peered":"2021-03-24T21:58:31.207564+0000","last_unstale":"2021-03-24T21:58:57.292927+0000","last_undegraded":"2021-03-24T21:58:57.292927+0000","last_fullsized":"2021-03-24T21:58:31.185319+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:33.533840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:33.533840+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"352","reported_epoch":"363","state":"active+clean","last_fresh":"2021-03-24T21:58:53.502727+0000","last_change":"2021-03-24T21:58:21.039376+0000","last_active":"2021-03-24T21:58:53.502727+0000","last_peered":"2021-03-24T21:58:53.502727+0000","last_clean":"2021-03-24T21:58:53.502727+0000","last_became_active":"2021-03-24T21:54:38.110327+0000","last_became_peered":"2021-03-24T21:54:38.110327+0000","last_unstale":"2021-03-24T21:58:53.502727+0000","last_undegraded":"2021-03-24T21:58:53.502727+0000","last_fullsized":"2021-03-24T21:58:53.502727+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:21.039180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:49:51.529051+0000","last_clean_scrub_stamp":"2021-03-24T21:58:21.039180+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"36","reported_epoch":"367","state":"active+undersized","last_fresh":"2021-03-24T21:58:57.297577+0000","last_change":"2021-03-24T21:58:31.229327+0000","last_active":"2021-03-24T21:58:57.297577+0000","last_peered":"2021-03-24T21:58:57.297577+0000","last_clean":"2021-03-24T21:51:50.731137+0000","last_became_active":"2021-03-24T21:58:31.229327+0000","last_became_peered":"2021-03-24T21:58:31.229327+0000","last_unstale":"2021-03-24T21:58:57.297577+0000","last_undegraded":"2021-03-24T21:58:57.297577+0000","last_fullsized":"2021-03-24T21:58:31.194892+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:55.072694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:57:55.072694+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.6","version":"341'175","reported_seq":"1065","reported_epoch":"367","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:58:57.285917+0000","last_change":"2021-03-24T21:58:31.222554+0000","last_active":"2021-03-24T21:58:57.285917+0000","last_peered":"2021-03-24T21:58:57.285917+0000","last_clean":"2021-03-24T21:58:29.213936+0000","last_became_active":"2021-03-24T21:58:31.222554+0000","last_became_peered":"2021-03-24T21:58:31.222554+0000","last_unstale":"2021-03-24T21:58:57.285917+0000","last_undegraded":"2021-03-24T21:58:31.192024+0000","last_fullsized":"2021-03-24T21:58:31.191882+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":175,"ondisk_log_size":175,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"stat_sum":{"num_bytes":7431571,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":89,"num_read_kb":116480,"num_write":51,"num_write_kb":23586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":8481953,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"34","reported_epoch":"363","state":"active+undersized","last_fresh":"2021-03-24T21:58:53.505024+0000","last_change":"2021-03-24T21:58:31.204715+0000","last_active":"2021-03-24T21:58:53.505024+0000","last_peered":"2021-03-24T21:58:53.505024+0000","last_clean":"2021-03-24T21:51:48.384232+0000","last_became_active":"2021-03-24T21:58:31.204715+0000","last_became_peered":"2021-03-24T21:58:31.204715+0000","last_unstale":"2021-03-24T21:58:53.505024+0000","last_undegraded":"2021-03-24T21:58:53.505024+0000","last_fullsized":"2021-03-24T21:58:31.199993+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:20.215065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:57:20.215065+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"89'22","reported_seq":"103","reported_epoch":"367","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:58:57.285117+0000","last_change":"2021-03-24T21:58:31.207163+0000","last_active":"2021-03-24T21:58:57.285117+0000","last_peered":"2021-03-24T21:58:57.285117+0000","last_clean":"2021-03-24T21:53:19.701158+0000","last_became_active":"2021-03-24T21:58:31.207163+0000","last_became_peered":"2021-03-24T21:58:31.207163+0000","last_unstale":"2021-03-24T21:58:57.285117+0000","last_undegraded":"2021-03-24T21:58:31.191478+0000","last_fullsized":"2021-03-24T21:58:31.191363+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'22","last_scrub_stamp":"2021-03-24T21:56:54.046194+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2021-03-24T21:53:19.717244+0000","last_clean_scrub_stamp":"2021-03-24T21:56:54.046194+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":36,"num_read_kb":21,"num_write":18,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":7}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"336","reported_epoch":"365","state":"active+clean","last_fresh":"2021-03-24T21:58:55.221583+0000","last_change":"2021-03-24T21:57:23.071573+0000","last_active":"2021-03-24T21:58:55.221583+0000","last_peered":"2021-03-24T21:58:55.221583+0000","last_clean":"2021-03-24T21:58:55.221583+0000","last_became_active":"2021-03-24T21:51:40.351000+0000","last_became_peered":"2021-03-24T21:51:40.351000+0000","last_unstale":"2021-03-24T21:58:55.221583+0000","last_undegraded":"2021-03-24T21:58:55.221583+0000","last_fullsized":"2021-03-24T21:58:55.221583+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:23.071394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:39.286101+0000","last_clean_scrub_stamp":"2021-03-24T21:57:23.071394+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"61'1","reported_seq":"295","reported_epoch":"365","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:58:55.222075+0000","last_change":"2021-03-24T21:58:31.201261+0000","last_active":"2021-03-24T21:58:55.222075+0000","last_peered":"2021-03-24T21:58:55.222075+0000","last_clean":"2021-03-24T21:58:29.216757+0000","last_became_active":"2021-03-24T21:58:31.201261+0000","last_became_peered":"2021-03-24T21:58:31.201261+0000","last_unstale":"2021-03-24T21:58:55.222075+0000","last_undegraded":"2021-03-24T21:58:31.191545+0000","last_fullsized":"2021-03-24T21:58:31.191379+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2021-03-24T21:57:12.046206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:12.046206+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"327","reported_epoch":"363","state":"active+clean","last_fresh":"2021-03-24T21:58:53.504615+0000","last_change":"2021-03-24T21:57:56.065379+0000","last_active":"2021-03-24T21:58:53.504615+0000","last_peered":"2021-03-24T21:58:53.504615+0000","last_clean":"2021-03-24T21:58:53.504615+0000","last_became_active":"2021-03-24T21:54:24.186957+0000","last_became_peered":"2021-03-24T21:54:24.186957+0000","last_unstale":"2021-03-24T21:58:53.504615+0000","last_undegraded":"2021-03-24T21:58:53.504615+0000","last_fullsized":"2021-03-24T21:58:53.504615+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:56.065113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:57:56.065113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"357'213","reported_seq":"1043","reported_epoch":"367","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:58:57.284608+0000","last_change":"2021-03-24T21:58:31.208413+0000","last_active":"2021-03-24T21:58:57.284608+0000","last_peered":"2021-03-24T21:58:57.284608+0000","last_clean":"2021-03-24T21:58:29.212922+0000","last_became_active":"2021-03-24T21:58:31.208413+0000","last_became_peered":"2021-03-24T21:58:31.208413+0000","last_unstale":"2021-03-24T21:58:57.284608+0000","last_undegraded":"2021-03-24T21:58:31.190092+0000","last_fullsized":"2021-03-24T21:58:31.189951+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'177","last_scrub_stamp":"2021-03-24T21:57:21.260725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:57:21.260725+0000","log_size":213,"ondisk_log_size":213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"stat_sum":{"num_bytes":6572063,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":30,"num_read_kb":39532,"num_write":17,"num_write_kb":8097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":4}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"333","reported_epoch":"367","state":"active+clean","last_fresh":"2021-03-24T21:58:57.282975+0000","last_change":"2021-03-24T21:58:23.866009+0000","last_active":"2021-03-24T21:58:57.282975+0000","last_peered":"2021-03-24T21:58:57.282975+0000","last_clean":"2021-03-24T21:58:57.282975+0000","last_became_active":"2021-03-24T21:54:38.136859+0000","last_became_peered":"2021-03-24T21:54:38.136859+0000","last_unstale":"2021-03-24T21:58:57.282975+0000","last_undegraded":"2021-03-24T21:58:57.282975+0000","last_fullsized":"2021-03-24T21:58:57.282975+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:23.865909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:58:23.865909+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"69'25","reported_seq":"370","reported_epoch":"365","state":"active+clean","last_fresh":"2021-03-24T21:58:55.226562+0000","last_change":"2021-03-24T21:58:54.693769+0000","last_active":"2021-03-24T21:58:55.226562+0000","last_peered":"2021-03-24T21:58:55.226562+0000","last_clean":"2021-03-24T21:58:55.226562+0000","last_became_active":"2021-03-24T21:54:38.141478+0000","last_became_peered":"2021-03-24T21:54:38.141478+0000","last_unstale":"2021-03-24T21:58:55.226562+0000","last_undegraded":"2021-03-24T21:58:55.226562+0000","last_fullsized":"2021-03-24T21:58:55.226562+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'25","last_scrub_stamp":"2021-03-24T21:58:54.693601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:58:54.693601+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":53,"num_read_kb":30,"num_write":28,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"89'2","reported_seq":"363","reported_epoch":"366","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:58:56.261965+0000","last_change":"2021-03-24T21:58:31.207923+0000","last_active":"2021-03-24T21:58:56.261965+0000","last_peered":"2021-03-24T21:58:56.261965+0000","last_clean":"2021-03-24T21:58:29.214732+0000","last_became_active":"2021-03-24T21:58:31.207923+0000","last_became_peered":"2021-03-24T21:58:31.207923+0000","last_unstale":"2021-03-24T21:58:56.261965+0000","last_undegraded":"2021-03-24T21:58:31.192809+0000","last_fullsized":"2021-03-24T21:58:31.192644+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2021-03-24T21:57:58.491830+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2021-03-24T21:57:58.491830+0000","last_clean_scrub_stamp":"2021-03-24T21:57:58.491830+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"324","reported_epoch":"367","state":"active+undersized","last_fresh":"2021-03-24T21:58:57.284247+0000","last_change":"2021-03-24T21:58:31.223020+0000","last_active":"2021-03-24T21:58:57.284247+0000","last_peered":"2021-03-24T21:58:57.284247+0000","last_clean":"2021-03-24T21:58:29.212707+0000","last_became_active":"2021-03-24T21:58:31.223020+0000","last_became_peered":"2021-03-24T21:58:31.223020+0000","last_unstale":"2021-03-24T21:58:57.284247+0000","last_undegraded":"2021-03-24T21:58:57.284247+0000","last_fullsized":"2021-03-24T21:58:31.188845+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:01.817480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:06.800417+0000","last_clean_scrub_stamp":"2021-03-24T21:58:01.817480+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"322","reported_epoch":"367","state":"active+clean","last_fresh":"2021-03-24T21:58:57.298300+0000","last_change":"2021-03-24T21:57:22.422901+0000","last_active":"2021-03-24T21:58:57.298300+0000","last_peered":"2021-03-24T21:58:57.298300+0000","last_clean":"2021-03-24T21:58:57.298300+0000","last_became_active":"2021-03-24T21:51:51.759514+0000","last_became_peered":"2021-03-24T21:51:51.759514+0000","last_unstale":"2021-03-24T21:58:57.298300+0000","last_undegraded":"2021-03-24T21:58:57.298300+0000","last_fullsized":"2021-03-24T21:58:57.298300+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:22.422787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:52:59.706511+0000","last_clean_scrub_stamp":"2021-03-24T21:57:22.422787+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"365'247","reported_seq":"1123","reported_epoch":"367","state":"active+clean","last_fresh":"2021-03-24T21:58:57.283487+0000","last_change":"2021-03-24T21:58:55.262711+0000","last_active":"2021-03-24T21:58:57.283487+0000","last_peered":"2021-03-24T21:58:57.283487+0000","last_clean":"2021-03-24T21:58:57.283487+0000","last_became_active":"2021-03-24T21:56:08.157153+0000","last_became_peered":"2021-03-24T21:56:08.157153+0000","last_unstale":"2021-03-24T21:58:57.283487+0000","last_undegraded":"2021-03-24T21:58:57.283487+0000","last_fullsized":"2021-03-24T21:58:57.283487+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'218","last_scrub_stamp":"2021-03-24T21:57:22.239425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:57:22.239425+0000","log_size":247,"ondisk_log_size":247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6231512,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":7093,"num_write":15,"num_write_kb":4988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"df","length":"1"},{"start":"e1","length":"1"},{"start":"ea","length":"4"},{"start":"f2","length":"3"},{"start":"fc","length":"1"}]},{"pgid":"4.b","version":"69'14","reported_seq":"376","reported_epoch":"367","state":"active+clean","last_fresh":"2021-03-24T21:58:57.284675+0000","last_change":"2021-03-24T21:55:57.297176+0000","last_active":"2021-03-24T21:58:57.284675+0000","last_peered":"2021-03-24T21:58:57.284675+0000","last_clean":"2021-03-24T21:58:57.284675+0000","last_became_active":"2021-03-24T21:54:38.138034+0000","last_became_peered":"2021-03-24T21:54:38.138034+0000","last_unstale":"2021-03-24T21:58:57.284675+0000","last_undegraded":"2021-03-24T21:58:57.284675+0000","last_fullsized":"2021-03-24T21:58:57.284675+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'14","last_scrub_stamp":"2021-03-24T21:55:57.297084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:57.297084+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":16,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":894,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"324","reported_epoch":"367","state":"active+undersized","last_fresh":"2021-03-24T21:58:57.305313+0000","last_change":"2021-03-24T21:58:31.229412+0000","last_active":"2021-03-24T21:58:57.305313+0000","last_peered":"2021-03-24T21:58:57.305313+0000","last_clean":"2021-03-24T21:58:29.207714+0000","last_became_active":"2021-03-24T21:58:31.229412+0000","last_became_peered":"2021-03-24T21:58:31.229412+0000","last_unstale":"2021-03-24T21:58:57.305313+0000","last_undegraded":"2021-03-24T21:58:57.305313+0000","last_fullsized":"2021-03-24T21:58:31.188427+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:12.514094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:12.514094+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"328","reported_epoch":"365","state":"active+clean","last_fresh":"2021-03-24T21:58:55.224959+0000","last_change":"2021-03-24T21:58:26.464636+0000","last_active":"2021-03-24T21:58:55.224959+0000","last_peered":"2021-03-24T21:58:55.224959+0000","last_clean":"2021-03-24T21:58:55.224959+0000","last_became_active":"2021-03-24T21:51:49.411921+0000","last_became_peered":"2021-03-24T21:51:49.411921+0000","last_unstale":"2021-03-24T21:58:55.224959+0000","last_undegraded":"2021-03-24T21:58:55.224959+0000","last_fullsized":"2021-03-24T21:58:55.224959+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:26.464456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:58:26.464456+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"326","reported_epoch":"367","state":"active+clean","last_fresh":"2021-03-24T21:58:57.283462+0000","last_change":"2021-03-24T21:57:02.189636+0000","last_active":"2021-03-24T21:58:57.283462+0000","last_peered":"2021-03-24T21:58:57.283462+0000","last_clean":"2021-03-24T21:58:57.283462+0000","last_became_active":"2021-03-24T21:57:02.188645+0000","last_became_peered":"2021-03-24T21:57:02.188645+0000","last_unstale":"2021-03-24T21:58:57.283462+0000","last_undegraded":"2021-03-24T21:58:57.283462+0000","last_fullsized":"2021-03-24T21:58:57.283462+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:33.246205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:35.636299+0000","last_clean_scrub_stamp":"2021-03-24T21:55:33.246205+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"365'150","reported_seq":"978","reported_epoch":"366","state":"active+clean","last_fresh":"2021-03-24T21:58:56.266221+0000","last_change":"2021-03-24T21:58:55.228101+0000","last_active":"2021-03-24T21:58:56.266221+0000","last_peered":"2021-03-24T21:58:56.266221+0000","last_clean":"2021-03-24T21:58:56.266221+0000","last_became_active":"2021-03-24T21:56:08.148045+0000","last_became_peered":"2021-03-24T21:56:08.148045+0000","last_unstale":"2021-03-24T21:58:56.266221+0000","last_undegraded":"2021-03-24T21:58:56.266221+0000","last_fullsized":"2021-03-24T21:58:56.266221+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"329'132","last_scrub_stamp":"2021-03-24T21:58:20.372929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:58:20.372929+0000","log_size":150,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6369985,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":14929,"num_write":24,"num_write_kb":5860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"df","length":"1"},{"start":"e1","length":"1"},{"start":"ea","length":"4"},{"start":"f2","length":"3"},{"start":"fc","length":"1"}]},{"pgid":"4.a","version":"69'23","reported_seq":"91","reported_epoch":"367","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:58:57.285730+0000","last_change":"2021-03-24T21:58:31.206644+0000","last_active":"2021-03-24T21:58:57.285730+0000","last_peered":"2021-03-24T21:58:57.285730+0000","last_clean":"2021-03-24T21:51:54.474291+0000","last_became_active":"2021-03-24T21:58:31.206644+0000","last_became_peered":"2021-03-24T21:58:31.206644+0000","last_unstale":"2021-03-24T21:58:57.285730+0000","last_undegraded":"2021-03-24T21:58:31.191416+0000","last_fullsized":"2021-03-24T21:58:31.191299+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'23","last_scrub_stamp":"2021-03-24T21:56:34.991954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:34.991954+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":43,"num_read_kb":24,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":13}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":"92","reported_epoch":"367","state":"active+undersized","last_fresh":"2021-03-24T21:58:57.316576+0000","last_change":"2021-03-24T21:58:31.206125+0000","last_active":"2021-03-24T21:58:57.316576+0000","last_peered":"2021-03-24T21:58:57.316576+0000","last_clean":"2021-03-24T21:54:22.149668+0000","last_became_active":"2021-03-24T21:58:31.206125+0000","last_became_peered":"2021-03-24T21:58:31.206125+0000","last_unstale":"2021-03-24T21:58:57.316576+0000","last_undegraded":"2021-03-24T21:58:57.316576+0000","last_fullsized":"2021-03-24T21:58:31.195695+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:58.133341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:58.133341+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"133","reported_epoch":"367","state":"active+undersized","last_fresh":"2021-03-24T21:58:57.284430+0000","last_change":"2021-03-24T21:58:31.209196+0000","last_active":"2021-03-24T21:58:57.284430+0000","last_peered":"2021-03-24T21:58:57.284430+0000","last_clean":"2021-03-24T21:58:29.215889+0000","last_became_active":"2021-03-24T21:58:31.209196+0000","last_became_peered":"2021-03-24T21:58:31.209196+0000","last_unstale":"2021-03-24T21:58:57.284430+0000","last_undegraded":"2021-03-24T21:58:57.284430+0000","last_fullsized":"2021-03-24T21:58:31.192460+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:44.018043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:44.018043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"76","reported_epoch":"365","state":"active+clean","last_fresh":"2021-03-24T21:58:55.222199+0000","last_change":"2021-03-24T21:57:58.225287+0000","last_active":"2021-03-24T21:58:55.222199+0000","last_peered":"2021-03-24T21:58:55.222199+0000","last_clean":"2021-03-24T21:58:55.222199+0000","last_became_active":"2021-03-24T21:57:57.296747+0000","last_became_peered":"2021-03-24T21:57:57.296747+0000","last_unstale":"2021-03-24T21:58:55.222199+0000","last_undegraded":"2021-03-24T21:58:55.222199+0000","last_fullsized":"2021-03-24T21:58:55.222199+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:58.225162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:57:58.225162+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"337'240","reported_seq":"1107","reported_epoch":"367","state":"active+clean","last_fresh":"2021-03-24T21:58:57.283607+0000","last_change":"2021-03-24T21:58:55.241875+0000","last_active":"2021-03-24T21:58:57.283607+0000","last_peered":"2021-03-24T21:58:57.283607+0000","last_clean":"2021-03-24T21:58:57.283607+0000","last_became_active":"2021-03-24T21:56:08.158313+0000","last_became_peered":"2021-03-24T21:56:08.158313+0000","last_unstale":"2021-03-24T21:58:57.283607+0000","last_undegraded":"2021-03-24T21:58:57.283607+0000","last_fullsized":"2021-03-24T21:58:57.283607+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'211","last_scrub_stamp":"2021-03-24T21:57:25.266209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:57:25.266209+0000","log_size":240,"ondisk_log_size":240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4254481,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":15224,"num_write":10,"num_write_kb":5110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"df","length":"1"},{"start":"e1","length":"1"},{"start":"ea","length":"4"},{"start":"f2","length":"3"},{"start":"fc","length":"1"}]},{"pgid":"5.8","version":"0'0","reported_seq":"343","reported_epoch":"365","state":"active+undersized","last_fresh":"2021-03-24T21:58:55.221865+0000","last_change":"2021-03-24T21:58:31.198862+0000","last_active":"2021-03-24T21:58:55.221865+0000","last_peered":"2021-03-24T21:58:55.221865+0000","last_clean":"2021-03-24T21:58:29.216464+0000","last_became_active":"2021-03-24T21:58:31.198862+0000","last_became_peered":"2021-03-24T21:58:31.198862+0000","last_unstale":"2021-03-24T21:58:55.221865+0000","last_undegraded":"2021-03-24T21:58:55.221865+0000","last_fullsized":"2021-03-24T21:58:31.190814+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:28.480625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:58:28.480625+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"69'15","reported_seq":"382","reported_epoch":"366","state":"active+clean","last_fresh":"2021-03-24T21:58:56.262089+0000","last_change":"2021-03-24T21:58:30.678342+0000","last_active":"2021-03-24T21:58:56.262089+0000","last_peered":"2021-03-24T21:58:56.262089+0000","last_clean":"2021-03-24T21:58:56.262089+0000","last_became_active":"2021-03-24T21:54:38.139863+0000","last_became_peered":"2021-03-24T21:54:38.139863+0000","last_unstale":"2021-03-24T21:58:56.262089+0000","last_undegraded":"2021-03-24T21:58:56.262089+0000","last_fullsized":"2021-03-24T21:58:56.262089+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:58:30.678185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:58:30.678185+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":19,"num_write":17,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":692,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":"338","reported_epoch":"363","state":"active+clean","last_fresh":"2021-03-24T21:58:53.504806+0000","last_change":"2021-03-24T21:57:57.050020+0000","last_active":"2021-03-24T21:58:53.504806+0000","last_peered":"2021-03-24T21:58:53.504806+0000","last_clean":"2021-03-24T21:58:53.504806+0000","last_became_active":"2021-03-24T21:54:38.109439+0000","last_became_peered":"2021-03-24T21:54:38.109439+0000","last_unstale":"2021-03-24T21:58:53.504806+0000","last_undegraded":"2021-03-24T21:58:53.504806+0000","last_fullsized":"2021-03-24T21:58:53.504806+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:57.049862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:57.049862+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"341","reported_epoch":"367","state":"active+undersized","last_fresh":"2021-03-24T21:58:57.310869+0000","last_change":"2021-03-24T21:58:31.229441+0000","last_active":"2021-03-24T21:58:57.310869+0000","last_peered":"2021-03-24T21:58:57.310869+0000","last_clean":"2021-03-24T21:58:29.208467+0000","last_became_active":"2021-03-24T21:58:31.229441+0000","last_became_peered":"2021-03-24T21:58:31.229441+0000","last_unstale":"2021-03-24T21:58:57.310869+0000","last_undegraded":"2021-03-24T21:58:57.310869+0000","last_fullsized":"2021-03-24T21:58:31.197372+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:24.548819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:22.724840+0000","last_clean_scrub_stamp":"2021-03-24T21:55:24.548819+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"357'133","reported_seq":"918","reported_epoch":"367","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:58:57.293051+0000","last_change":"2021-03-24T21:58:31.226756+0000","last_active":"2021-03-24T21:58:57.293051+0000","last_peered":"2021-03-24T21:58:57.293051+0000","last_clean":"2021-03-24T21:58:29.207481+0000","last_became_active":"2021-03-24T21:58:31.196478+0000","last_became_peered":"2021-03-24T21:58:31.196478+0000","last_unstale":"2021-03-24T21:58:57.293051+0000","last_undegraded":"2021-03-24T21:58:31.226366+0000","last_fullsized":"2021-03-24T21:58:31.185724+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":133,"ondisk_log_size":133,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"stat_sum":{"num_bytes":10061614,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":26,"num_read_kb":22368,"num_write":37,"num_write_kb":16867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"109'4","reported_seq":"433","reported_epoch":"363","state":"active+clean","last_fresh":"2021-03-24T21:58:54.051217+0000","last_change":"2021-03-24T21:56:32.051794+0000","last_active":"2021-03-24T21:58:54.051217+0000","last_peered":"2021-03-24T21:58:54.051217+0000","last_clean":"2021-03-24T21:58:54.051217+0000","last_became_active":"2021-03-24T21:54:38.125497+0000","last_became_peered":"2021-03-24T21:54:38.125497+0000","last_unstale":"2021-03-24T21:58:54.051217+0000","last_undegraded":"2021-03-24T21:58:54.051217+0000","last_fullsized":"2021-03-24T21:58:54.051217+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'4","last_scrub_stamp":"2021-03-24T21:56:32.051533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:32.051533+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"97'28","reported_seq":"111","reported_epoch":"363","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:58:53.502908+0000","last_change":"2021-03-24T21:58:31.202742+0000","last_active":"2021-03-24T21:58:53.502908+0000","last_peered":"2021-03-24T21:58:53.502908+0000","last_clean":"2021-03-24T21:54:19.144778+0000","last_became_active":"2021-03-24T21:58:31.202742+0000","last_became_peered":"2021-03-24T21:58:31.202742+0000","last_unstale":"2021-03-24T21:58:53.502908+0000","last_undegraded":"2021-03-24T21:58:31.193830+0000","last_fullsized":"2021-03-24T21:58:31.193649+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'28","last_scrub_stamp":"2021-03-24T21:58:20.925603+0000","last_deep_scrub":"97'28","last_deep_scrub_stamp":"2021-03-24T21:54:19.157265+0000","last_clean_scrub_stamp":"2021-03-24T21:58:20.925603+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":20,"num_write":19,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[{"shards":"4,7","objects":10}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"89'4","reported_seq":"329","reported_epoch":"363","state":"active+clean","last_fresh":"2021-03-24T21:58:53.505386+0000","last_change":"2021-03-24T21:58:25.970055+0000","last_active":"2021-03-24T21:58:53.505386+0000","last_peered":"2021-03-24T21:58:53.505386+0000","last_clean":"2021-03-24T21:58:53.505386+0000","last_became_active":"2021-03-24T21:51:45.385110+0000","last_became_peered":"2021-03-24T21:51:45.385110+0000","last_unstale":"2021-03-24T21:58:53.505386+0000","last_undegraded":"2021-03-24T21:58:53.505386+0000","last_fullsized":"2021-03-24T21:58:53.505386+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'4","last_scrub_stamp":"2021-03-24T21:58:25.969865+0000","last_deep_scrub":"89'4","last_deep_scrub_stamp":"2021-03-24T21:53:19.926637+0000","last_clean_scrub_stamp":"2021-03-24T21:58:25.969865+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"307","reported_epoch":"367","state":"active+undersized","last_fresh":"2021-03-24T21:58:57.284328+0000","last_change":"2021-03-24T21:58:31.222499+0000","last_active":"2021-03-24T21:58:57.284328+0000","last_peered":"2021-03-24T21:58:57.284328+0000","last_clean":"2021-03-24T21:58:29.212838+0000","last_became_active":"2021-03-24T21:58:31.222499+0000","last_became_peered":"2021-03-24T21:58:31.222499+0000","last_unstale":"2021-03-24T21:58:57.284328+0000","last_undegraded":"2021-03-24T21:58:57.284328+0000","last_fullsized":"2021-03-24T21:58:31.188323+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:54.262681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:54.262681+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"272'275","reported_seq":"1005","reported_epoch":"366","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:58:56.261621+0000","last_change":"2021-03-24T21:58:31.208031+0000","last_active":"2021-03-24T21:58:56.261621+0000","last_peered":"2021-03-24T21:58:56.261621+0000","last_clean":"2021-03-24T21:57:22.173331+0000","last_became_active":"2021-03-24T21:58:31.208031+0000","last_became_peered":"2021-03-24T21:58:31.208031+0000","last_unstale":"2021-03-24T21:58:56.261621+0000","last_undegraded":"2021-03-24T21:58:31.193214+0000","last_fullsized":"2021-03-24T21:58:31.192237+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'275","last_scrub_stamp":"2021-03-24T21:58:23.942036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:58:23.942036+0000","log_size":275,"ondisk_log_size":275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"stat_sum":{"num_bytes":8857338,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":114,"num_read_kb":132714,"num_write":52,"num_write_kb":22259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":12587455,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":3}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"69'7","reported_seq":"351","reported_epoch":"367","state":"active+clean","last_fresh":"2021-03-24T21:58:57.297513+0000","last_change":"2021-03-24T21:55:56.490518+0000","last_active":"2021-03-24T21:58:57.297513+0000","last_peered":"2021-03-24T21:58:57.297513+0000","last_clean":"2021-03-24T21:58:57.297513+0000","last_became_active":"2021-03-24T21:54:38.144098+0000","last_became_peered":"2021-03-24T21:54:38.144098+0000","last_unstale":"2021-03-24T21:58:57.297513+0000","last_undegraded":"2021-03-24T21:58:57.297513+0000","last_fullsized":"2021-03-24T21:58:57.297513+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'7","last_scrub_stamp":"2021-03-24T21:55:56.490439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:56.490439+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":7,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":158,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":"293","reported_epoch":"367","state":"active+undersized","last_fresh":"2021-03-24T21:58:57.285377+0000","last_change":"2021-03-24T21:58:31.222518+0000","last_active":"2021-03-24T21:58:57.285377+0000","last_peered":"2021-03-24T21:58:57.285377+0000","last_clean":"2021-03-24T21:58:29.212134+0000","last_became_active":"2021-03-24T21:58:31.222518+0000","last_became_peered":"2021-03-24T21:58:31.222518+0000","last_unstale":"2021-03-24T21:58:57.285377+0000","last_undegraded":"2021-03-24T21:58:57.285377+0000","last_fullsized":"2021-03-24T21:58:31.192669+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:16.270874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:16.270874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"94","reported_epoch":"367","state":"active+undersized","last_fresh":"2021-03-24T21:58:57.298192+0000","last_change":"2021-03-24T21:58:31.202521+0000","last_active":"2021-03-24T21:58:57.298192+0000","last_peered":"2021-03-24T21:58:57.298192+0000","last_clean":"2021-03-24T21:54:36.088165+0000","last_became_active":"2021-03-24T21:58:31.202521+0000","last_became_peered":"2021-03-24T21:58:31.202521+0000","last_unstale":"2021-03-24T21:58:57.298192+0000","last_undegraded":"2021-03-24T21:58:57.298192+0000","last_fullsized":"2021-03-24T21:58:31.187228+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:48.144391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:48.144391+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"333","reported_epoch":"363","state":"active+undersized","last_fresh":"2021-03-24T21:58:53.504001+0000","last_change":"2021-03-24T21:58:31.203277+0000","last_active":"2021-03-24T21:58:53.504001+0000","last_peered":"2021-03-24T21:58:53.504001+0000","last_clean":"2021-03-24T21:58:29.203193+0000","last_became_active":"2021-03-24T21:58:31.203277+0000","last_became_peered":"2021-03-24T21:58:31.203277+0000","last_unstale":"2021-03-24T21:58:53.504001+0000","last_undegraded":"2021-03-24T21:58:53.504001+0000","last_fullsized":"2021-03-24T21:58:31.196438+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:04.012279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:58:04.012279+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.14","version":"367'296","reported_seq":"1139","reported_epoch":"367","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:58:57.329910+0000","last_change":"2021-03-24T21:58:31.195219+0000","last_active":"2021-03-24T21:58:57.329910+0000","last_peered":"2021-03-24T21:58:57.329910+0000","last_clean":"2021-03-24T21:58:29.206770+0000","last_became_active":"2021-03-24T21:58:31.195219+0000","last_became_peered":"2021-03-24T21:58:31.195219+0000","last_unstale":"2021-03-24T21:58:57.329910+0000","last_undegraded":"2021-03-24T21:58:31.186610+0000","last_fullsized":"2021-03-24T21:58:31.186105+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":296,"ondisk_log_size":296,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"stat_sum":{"num_bytes":20846660,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":128,"num_read_kb":117448,"num_write":119,"num_write_kb":38551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":6105417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":11}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"69'16","reported_seq":"356","reported_epoch":"367","state":"active+clean","last_fresh":"2021-03-24T21:58:57.286062+0000","last_change":"2021-03-24T21:56:40.303286+0000","last_active":"2021-03-24T21:58:57.286062+0000","last_peered":"2021-03-24T21:58:57.286062+0000","last_clean":"2021-03-24T21:58:57.286062+0000","last_became_active":"2021-03-24T21:51:47.687595+0000","last_became_peered":"2021-03-24T21:51:47.687595+0000","last_unstale":"2021-03-24T21:58:57.286062+0000","last_undegraded":"2021-03-24T21:58:57.286062+0000","last_fullsized":"2021-03-24T21:58:57.286062+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T21:56:40.303200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:40.303200+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":38,"num_read_kb":18,"num_write":17,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":"125","reported_epoch":"366","state":"active+clean","last_fresh":"2021-03-24T21:58:56.261778+0000","last_change":"2021-03-24T21:57:57.288460+0000","last_active":"2021-03-24T21:58:56.261778+0000","last_peered":"2021-03-24T21:58:56.261778+0000","last_clean":"2021-03-24T21:58:56.261778+0000","last_became_active":"2021-03-24T21:57:57.287481+0000","last_became_peered":"2021-03-24T21:57:57.287481+0000","last_unstale":"2021-03-24T21:58:56.261778+0000","last_undegraded":"2021-03-24T21:58:56.261778+0000","last_fullsized":"2021-03-24T21:58:56.261778+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:51.037780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:51.037780+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"333","reported_epoch":"367","state":"active+clean","last_fresh":"2021-03-24T21:58:57.297653+0000","last_change":"2021-03-24T21:57:52.376563+0000","last_active":"2021-03-24T21:58:57.297653+0000","last_peered":"2021-03-24T21:58:57.297653+0000","last_clean":"2021-03-24T21:58:57.297653+0000","last_became_active":"2021-03-24T21:54:38.129390+0000","last_became_peered":"2021-03-24T21:54:38.129390+0000","last_unstale":"2021-03-24T21:58:57.297653+0000","last_undegraded":"2021-03-24T21:58:57.297653+0000","last_fullsized":"2021-03-24T21:58:57.297653+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:52.376475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:55.493713+0000","last_clean_scrub_stamp":"2021-03-24T21:57:52.376475+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"321","reported_epoch":"365","state":"active+clean","last_fresh":"2021-03-24T21:58:55.226655+0000","last_change":"2021-03-24T21:57:20.073705+0000","last_active":"2021-03-24T21:58:55.226655+0000","last_peered":"2021-03-24T21:58:55.226655+0000","last_clean":"2021-03-24T21:58:55.226655+0000","last_became_active":"2021-03-24T21:51:51.754085+0000","last_became_peered":"2021-03-24T21:51:51.754085+0000","last_unstale":"2021-03-24T21:58:55.226655+0000","last_undegraded":"2021-03-24T21:58:55.226655+0000","last_fullsized":"2021-03-24T21:58:55.226655+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:20.073641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:57:20.073641+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"357'94","reported_seq":"923","reported_epoch":"363","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:58:53.504327+0000","last_change":"2021-03-24T21:58:31.209906+0000","last_active":"2021-03-24T21:58:53.504327+0000","last_peered":"2021-03-24T21:58:53.504327+0000","last_clean":"2021-03-24T21:58:29.203501+0000","last_became_active":"2021-03-24T21:58:31.209906+0000","last_became_peered":"2021-03-24T21:58:31.209906+0000","last_unstale":"2021-03-24T21:58:53.504327+0000","last_undegraded":"2021-03-24T21:58:31.197019+0000","last_fullsized":"2021-03-24T21:58:31.196878+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":5,"last_scrub":"233'79","last_scrub_stamp":"2021-03-24T21:56:50.934781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:56:50.934781+0000","log_size":94,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":9,"stat_sum":{"num_bytes":2453888,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":12,"num_read_kb":8829,"num_write":11,"num_write_kb":2640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":2}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"69'13","reported_seq":"68","reported_epoch":"363","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:58:53.504694+0000","last_change":"2021-03-24T21:58:31.204551+0000","last_active":"2021-03-24T21:58:53.504694+0000","last_peered":"2021-03-24T21:58:53.504694+0000","last_clean":"2021-03-24T21:51:54.397987+0000","last_became_active":"2021-03-24T21:58:31.204551+0000","last_became_peered":"2021-03-24T21:58:31.204551+0000","last_unstale":"2021-03-24T21:58:53.504694+0000","last_undegraded":"2021-03-24T21:58:31.199681+0000","last_fullsized":"2021-03-24T21:58:31.199516+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2021-03-24T21:56:47.168558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:47.168558+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":14,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[{"shards":"3,7","objects":8}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":"328","reported_epoch":"367","state":"active+clean","last_fresh":"2021-03-24T21:58:57.284195+0000","last_change":"2021-03-24T21:56:32.279216+0000","last_active":"2021-03-24T21:58:57.284195+0000","last_peered":"2021-03-24T21:58:57.284195+0000","last_clean":"2021-03-24T21:58:57.284195+0000","last_became_active":"2021-03-24T21:51:45.381516+0000","last_became_peered":"2021-03-24T21:51:45.381516+0000","last_unstale":"2021-03-24T21:58:57.284195+0000","last_undegraded":"2021-03-24T21:58:57.284195+0000","last_fullsized":"2021-03-24T21:58:57.284195+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:32.279119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:05.473348+0000","last_clean_scrub_stamp":"2021-03-24T21:56:32.279119+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"235'4","reported_seq":"426","reported_epoch":"368","state":"active+clean","last_fresh":"2021-03-24T21:58:58.712474+0000","last_change":"2021-03-24T21:58:20.324676+0000","last_active":"2021-03-24T21:58:58.712474+0000","last_peered":"2021-03-24T21:58:58.712474+0000","last_clean":"2021-03-24T21:58:58.712474+0000","last_became_active":"2021-03-24T21:56:44.224410+0000","last_became_peered":"2021-03-24T21:56:44.224410+0000","last_unstale":"2021-03-24T21:58:58.712474+0000","last_undegraded":"2021-03-24T21:58:58.712474+0000","last_fullsized":"2021-03-24T21:58:58.712474+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"235'4","last_scrub_stamp":"2021-03-24T21:58:20.324557+0000","last_deep_scrub":"86'3","last_deep_scrub_stamp":"2021-03-24T21:52:56.678111+0000","last_clean_scrub_stamp":"2021-03-24T21:58:20.324557+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"330","reported_epoch":"366","state":"active+undersized","last_fresh":"2021-03-24T21:58:56.271529+0000","last_change":"2021-03-24T21:58:31.200854+0000","last_active":"2021-03-24T21:58:56.271529+0000","last_peered":"2021-03-24T21:58:56.271529+0000","last_clean":"2021-03-24T21:58:29.218648+0000","last_became_active":"2021-03-24T21:58:31.200854+0000","last_became_peered":"2021-03-24T21:58:31.200854+0000","last_unstale":"2021-03-24T21:58:56.271529+0000","last_undegraded":"2021-03-24T21:58:56.271529+0000","last_fullsized":"2021-03-24T21:58:31.197192+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:51.924963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:51.924963+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.16","version":"316'167","reported_seq":"1020","reported_epoch":"366","state":"active+clean","last_fresh":"2021-03-24T21:58:56.270011+0000","last_change":"2021-03-24T21:58:55.234935+0000","last_active":"2021-03-24T21:58:56.270011+0000","last_peered":"2021-03-24T21:58:56.270011+0000","last_clean":"2021-03-24T21:58:56.270011+0000","last_became_active":"2021-03-24T21:56:13.127048+0000","last_became_peered":"2021-03-24T21:56:13.127048+0000","last_unstale":"2021-03-24T21:58:56.270011+0000","last_undegraded":"2021-03-24T21:58:56.270011+0000","last_fullsized":"2021-03-24T21:58:56.270011+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'146","last_scrub_stamp":"2021-03-24T21:57:22.669305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:57:22.669305+0000","log_size":167,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2739607,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":8860,"num_write":10,"num_write_kb":2219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"df","length":"1"},{"start":"e1","length":"1"},{"start":"ea","length":"4"},{"start":"f2","length":"3"},{"start":"fc","length":"1"}]},{"pgid":"4.14","version":"69'14","reported_seq":"381","reported_epoch":"367","state":"active+clean","last_fresh":"2021-03-24T21:58:57.435674+0000","last_change":"2021-03-24T21:58:57.435463+0000","last_active":"2021-03-24T21:58:57.435674+0000","last_peered":"2021-03-24T21:58:57.435674+0000","last_clean":"2021-03-24T21:58:57.435674+0000","last_became_active":"2021-03-24T21:54:38.143783+0000","last_became_peered":"2021-03-24T21:54:38.143783+0000","last_unstale":"2021-03-24T21:58:57.435674+0000","last_undegraded":"2021-03-24T21:58:57.435674+0000","last_fullsized":"2021-03-24T21:58:57.435674+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'14","last_scrub_stamp":"2021-03-24T21:58:57.435358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:58:57.435358+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":27,"num_read_kb":14,"num_write":14,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":"324","reported_epoch":"363","state":"active+clean","last_fresh":"2021-03-24T21:58:53.503764+0000","last_change":"2021-03-24T21:58:02.015066+0000","last_active":"2021-03-24T21:58:53.503764+0000","last_peered":"2021-03-24T21:58:53.503764+0000","last_clean":"2021-03-24T21:58:53.503764+0000","last_became_active":"2021-03-24T21:51:45.579271+0000","last_became_peered":"2021-03-24T21:51:45.579271+0000","last_unstale":"2021-03-24T21:58:53.503764+0000","last_undegraded":"2021-03-24T21:58:53.503764+0000","last_fullsized":"2021-03-24T21:58:53.503764+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:02.014901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:58:02.014901+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"343'5","reported_seq":"90","reported_epoch":"367","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:58:57.285473+0000","last_change":"2021-03-24T21:58:31.223133+0000","last_active":"2021-03-24T21:58:57.285473+0000","last_peered":"2021-03-24T21:58:57.285473+0000","last_clean":"2021-03-24T21:53:08.809615+0000","last_became_active":"2021-03-24T21:58:31.223133+0000","last_became_peered":"2021-03-24T21:58:31.223133+0000","last_unstale":"2021-03-24T21:58:57.285473+0000","last_undegraded":"2021-03-24T21:58:31.192057+0000","last_fullsized":"2021-03-24T21:58:31.191934+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'4","last_scrub_stamp":"2021-03-24T21:56:42.209497+0000","last_deep_scrub":"86'3","last_deep_scrub_stamp":"2021-03-24T21:53:08.815163+0000","last_clean_scrub_stamp":"2021-03-24T21:56:42.209497+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"322","reported_epoch":"367","state":"active+clean","last_fresh":"2021-03-24T21:58:57.283199+0000","last_change":"2021-03-24T21:57:58.768856+0000","last_active":"2021-03-24T21:58:57.283199+0000","last_peered":"2021-03-24T21:58:57.283199+0000","last_clean":"2021-03-24T21:58:57.283199+0000","last_became_active":"2021-03-24T21:51:51.756584+0000","last_became_peered":"2021-03-24T21:51:51.756584+0000","last_unstale":"2021-03-24T21:58:57.283199+0000","last_undegraded":"2021-03-24T21:58:57.283199+0000","last_fullsized":"2021-03-24T21:58:57.283199+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:58.768675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:57:58.768675+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.17","version":"355'376","reported_seq":"1307","reported_epoch":"363","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:58:53.505245+0000","last_change":"2021-03-24T21:58:31.208787+0000","last_active":"2021-03-24T21:58:53.505245+0000","last_peered":"2021-03-24T21:58:53.505245+0000","last_clean":"2021-03-24T21:58:29.232381+0000","last_became_active":"2021-03-24T21:58:31.208787+0000","last_became_peered":"2021-03-24T21:58:31.208787+0000","last_unstale":"2021-03-24T21:58:53.505245+0000","last_undegraded":"2021-03-24T21:58:31.195157+0000","last_fullsized":"2021-03-24T21:58:31.194997+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":5,"last_scrub":"265'274","last_scrub_stamp":"2021-03-24T21:57:14.281739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:57:14.281739+0000","log_size":376,"ondisk_log_size":376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":9,"stat_sum":{"num_bytes":34866554,"num_objects":12,"num_object_clones":8,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":84,"num_read_kb":92013,"num_write":47,"num_write_kb":27261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":12}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"69'15","reported_seq":"417","reported_epoch":"366","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:58:56.262571+0000","last_change":"2021-03-24T21:58:31.200974+0000","last_active":"2021-03-24T21:58:56.262571+0000","last_peered":"2021-03-24T21:58:56.262571+0000","last_clean":"2021-03-24T21:58:29.215183+0000","last_became_active":"2021-03-24T21:58:31.200974+0000","last_became_peered":"2021-03-24T21:58:31.200974+0000","last_unstale":"2021-03-24T21:58:56.262571+0000","last_undegraded":"2021-03-24T21:58:31.193428+0000","last_fullsized":"2021-03-24T21:58:31.193298+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:58:02.548166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:58:02.548166+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[{"shards":"2,4","objects":9}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":"337","reported_epoch":"366","state":"active+clean","last_fresh":"2021-03-24T21:58:56.263888+0000","last_change":"2021-03-24T21:58:29.716360+0000","last_active":"2021-03-24T21:58:56.263888+0000","last_peered":"2021-03-24T21:58:56.263888+0000","last_clean":"2021-03-24T21:58:56.263888+0000","last_became_active":"2021-03-24T21:54:24.181343+0000","last_became_peered":"2021-03-24T21:54:24.181343+0000","last_unstale":"2021-03-24T21:58:56.263888+0000","last_undegraded":"2021-03-24T21:58:56.263888+0000","last_fullsized":"2021-03-24T21:58:56.263888+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:29.716186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:58:29.716186+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"165","reported_epoch":"363","state":"active+undersized","last_fresh":"2021-03-24T21:58:53.503093+0000","last_change":"2021-03-24T21:58:31.205120+0000","last_active":"2021-03-24T21:58:53.503093+0000","last_peered":"2021-03-24T21:58:53.503093+0000","last_clean":"2021-03-24T21:56:33.303502+0000","last_became_active":"2021-03-24T21:58:31.205120+0000","last_became_peered":"2021-03-24T21:58:31.205120+0000","last_unstale":"2021-03-24T21:58:53.503093+0000","last_undegraded":"2021-03-24T21:58:53.503093+0000","last_fullsized":"2021-03-24T21:58:31.195804+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:31.118050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:57:31.118050+0000","last_clean_scrub_stamp":"2021-03-24T21:57:31.118050+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"323","reported_epoch":"367","state":"active+undersized","last_fresh":"2021-03-24T21:58:57.296683+0000","last_change":"2021-03-24T21:58:31.219635+0000","last_active":"2021-03-24T21:58:57.296683+0000","last_peered":"2021-03-24T21:58:57.296683+0000","last_clean":"2021-03-24T21:58:29.204869+0000","last_became_active":"2021-03-24T21:58:31.219635+0000","last_became_peered":"2021-03-24T21:58:31.219635+0000","last_unstale":"2021-03-24T21:58:57.296683+0000","last_undegraded":"2021-03-24T21:58:57.296683+0000","last_fullsized":"2021-03-24T21:58:31.184834+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:22.555624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:23.707914+0000","last_clean_scrub_stamp":"2021-03-24T21:55:22.555624+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.10","version":"337'41","reported_seq":"348","reported_epoch":"367","state":"active+undersized","last_fresh":"2021-03-24T21:58:57.290583+0000","last_change":"2021-03-24T21:58:31.206788+0000","last_active":"2021-03-24T21:58:57.290583+0000","last_peered":"2021-03-24T21:58:57.290583+0000","last_clean":"2021-03-24T21:58:27.206026+0000","last_became_active":"2021-03-24T21:58:31.206788+0000","last_became_peered":"2021-03-24T21:58:31.206788+0000","last_unstale":"2021-03-24T21:58:57.290583+0000","last_undegraded":"2021-03-24T21:58:57.290583+0000","last_fullsized":"2021-03-24T21:58:31.185192+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":5,"last_scrub":"277'24","last_scrub_stamp":"2021-03-24T21:57:28.119420+0000","last_deep_scrub":"277'24","last_deep_scrub_stamp":"2021-03-24T21:57:28.119420+0000","last_clean_scrub_stamp":"2021-03-24T21:57:28.119420+0000","log_size":41,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":23,"num_read_kb":18238,"num_write":11,"num_write_kb":5221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"69'15","reported_seq":"346","reported_epoch":"365","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:58:55.224794+0000","last_change":"2021-03-24T21:58:31.205390+0000","last_active":"2021-03-24T21:58:55.224794+0000","last_peered":"2021-03-24T21:58:55.224794+0000","last_clean":"2021-03-24T21:58:29.216491+0000","last_became_active":"2021-03-24T21:58:31.205390+0000","last_became_peered":"2021-03-24T21:58:31.205390+0000","last_unstale":"2021-03-24T21:58:55.224794+0000","last_undegraded":"2021-03-24T21:58:31.194807+0000","last_fullsized":"2021-03-24T21:58:31.190627+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:57:52.248247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:57:52.248247+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":34,"num_read_kb":16,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":10}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":"324","reported_epoch":"363","state":"active+clean","last_fresh":"2021-03-24T21:58:53.504121+0000","last_change":"2021-03-24T21:58:04.965522+0000","last_active":"2021-03-24T21:58:53.504121+0000","last_peered":"2021-03-24T21:58:53.504121+0000","last_clean":"2021-03-24T21:58:53.504121+0000","last_became_active":"2021-03-24T21:51:45.379383+0000","last_became_peered":"2021-03-24T21:51:45.379383+0000","last_unstale":"2021-03-24T21:58:53.504121+0000","last_undegraded":"2021-03-24T21:58:53.504121+0000","last_fullsized":"2021-03-24T21:58:53.504121+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:04.965369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:58:04.965369+0000","last_clean_scrub_stamp":"2021-03-24T21:58:04.965369+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"334","reported_epoch":"367","state":"active+undersized","last_fresh":"2021-03-24T21:58:57.306567+0000","last_change":"2021-03-24T21:58:31.231202+0000","last_active":"2021-03-24T21:58:57.306567+0000","last_peered":"2021-03-24T21:58:57.306567+0000","last_clean":"2021-03-24T21:58:29.207905+0000","last_became_active":"2021-03-24T21:58:31.231202+0000","last_became_peered":"2021-03-24T21:58:31.231202+0000","last_unstale":"2021-03-24T21:58:57.306567+0000","last_undegraded":"2021-03-24T21:58:57.306567+0000","last_fullsized":"2021-03-24T21:58:31.201837+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:32.480115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:32.480115+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"337","reported_epoch":"367","state":"active+undersized","last_fresh":"2021-03-24T21:58:57.282594+0000","last_change":"2021-03-24T21:58:31.223636+0000","last_active":"2021-03-24T21:58:57.282594+0000","last_peered":"2021-03-24T21:58:57.282594+0000","last_clean":"2021-03-24T21:58:29.211869+0000","last_became_active":"2021-03-24T21:58:31.223636+0000","last_became_peered":"2021-03-24T21:58:31.223636+0000","last_unstale":"2021-03-24T21:58:57.282594+0000","last_undegraded":"2021-03-24T21:58:57.282594+0000","last_fullsized":"2021-03-24T21:58:31.193002+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:52.226117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:52.226117+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.11","version":"364'392","reported_seq":"1283","reported_epoch":"364","state":"active+clean","last_fresh":"2021-03-24T21:58:54.314504+0000","last_change":"2021-03-24T21:58:54.314865+0000","last_active":"2021-03-24T21:58:54.314504+0000","last_peered":"2021-03-24T21:58:54.314504+0000","last_clean":"2021-03-24T21:58:54.314504+0000","last_became_active":"2021-03-24T21:56:14.138730+0000","last_became_peered":"2021-03-24T21:56:14.138730+0000","last_unstale":"2021-03-24T21:58:54.314504+0000","last_undegraded":"2021-03-24T21:58:54.314504+0000","last_fullsized":"2021-03-24T21:58:54.314504+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'297","last_scrub_stamp":"2021-03-24T21:57:24.209124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:57:24.209124+0000","log_size":392,"ondisk_log_size":392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18094504,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":54,"num_read_kb":74280,"num_write":35,"num_write_kb":13840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"df","length":"1"},{"start":"e1","length":"1"},{"start":"ea","length":"3"},{"start":"f2","length":"3"},{"start":"fc","length":"1"}]},{"pgid":"4.11","version":"86'22","reported_seq":"387","reported_epoch":"367","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:58:57.309575+0000","last_change":"2021-03-24T21:58:31.202532+0000","last_active":"2021-03-24T21:58:57.309575+0000","last_peered":"2021-03-24T21:58:57.309575+0000","last_clean":"2021-03-24T21:58:29.206035+0000","last_became_active":"2021-03-24T21:58:31.202532+0000","last_became_peered":"2021-03-24T21:58:31.202532+0000","last_unstale":"2021-03-24T21:58:57.309575+0000","last_undegraded":"2021-03-24T21:58:31.187011+0000","last_fullsized":"2021-03-24T21:58:31.186847+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'22","last_scrub_stamp":"2021-03-24T21:54:51.585058+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2021-03-24T21:52:54.597505+0000","last_clean_scrub_stamp":"2021-03-24T21:54:51.585058+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":16,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":580,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":8}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":"98","reported_epoch":"367","state":"active+undersized","last_fresh":"2021-03-24T21:58:57.297846+0000","last_change":"2021-03-24T21:58:31.202328+0000","last_active":"2021-03-24T21:58:57.297846+0000","last_peered":"2021-03-24T21:58:57.297846+0000","last_clean":"2021-03-24T21:54:36.088286+0000","last_became_active":"2021-03-24T21:58:31.202328+0000","last_became_peered":"2021-03-24T21:58:31.202328+0000","last_unstale":"2021-03-24T21:58:57.297846+0000","last_undegraded":"2021-03-24T21:58:57.297846+0000","last_fullsized":"2021-03-24T21:58:31.186585+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:13.016117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:13.016117+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"329","reported_epoch":"363","state":"active+clean","last_fresh":"2021-03-24T21:58:53.503058+0000","last_change":"2021-03-24T21:57:53.093497+0000","last_active":"2021-03-24T21:58:53.503058+0000","last_peered":"2021-03-24T21:58:53.503058+0000","last_clean":"2021-03-24T21:58:53.503058+0000","last_became_active":"2021-03-24T21:54:24.194423+0000","last_became_peered":"2021-03-24T21:54:24.194423+0000","last_unstale":"2021-03-24T21:58:53.503058+0000","last_undegraded":"2021-03-24T21:58:53.503058+0000","last_fullsized":"2021-03-24T21:58:53.503058+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:53.093327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:57:53.093327+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"326","reported_epoch":"367","state":"active+clean","last_fresh":"2021-03-24T21:58:57.311156+0000","last_change":"2021-03-24T21:56:40.458139+0000","last_active":"2021-03-24T21:58:57.311156+0000","last_peered":"2021-03-24T21:58:57.311156+0000","last_clean":"2021-03-24T21:58:57.311156+0000","last_became_active":"2021-03-24T21:54:24.202686+0000","last_became_peered":"2021-03-24T21:54:24.202686+0000","last_unstale":"2021-03-24T21:58:57.311156+0000","last_undegraded":"2021-03-24T21:58:57.311156+0000","last_fullsized":"2021-03-24T21:58:57.311156+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:40.458011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:47.540793+0000","last_clean_scrub_stamp":"2021-03-24T21:56:40.458011+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.12","version":"183'26","reported_seq":"514","reported_epoch":"367","state":"active+clean","last_fresh":"2021-03-24T21:58:57.282810+0000","last_change":"2021-03-24T21:58:55.258304+0000","last_active":"2021-03-24T21:58:57.282810+0000","last_peered":"2021-03-24T21:58:57.282810+0000","last_clean":"2021-03-24T21:58:57.282810+0000","last_became_active":"2021-03-24T21:56:10.130886+0000","last_became_peered":"2021-03-24T21:56:10.130886+0000","last_unstale":"2021-03-24T21:58:57.282810+0000","last_undegraded":"2021-03-24T21:58:57.282810+0000","last_fullsized":"2021-03-24T21:58:57.282810+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":5,"last_scrub":"183'26","last_scrub_stamp":"2021-03-24T21:57:53.859843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:57:53.859843+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"df","length":"1"},{"start":"e1","length":"1"},{"start":"ea","length":"4"},{"start":"f2","length":"3"},{"start":"fc","length":"1"}]},{"pgid":"4.10","version":"69'12","reported_seq":"449","reported_epoch":"367","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:58:57.290436+0000","last_change":"2021-03-24T21:58:31.206182+0000","last_active":"2021-03-24T21:58:57.290436+0000","last_peered":"2021-03-24T21:58:57.290436+0000","last_clean":"2021-03-24T21:58:29.204663+0000","last_became_active":"2021-03-24T21:58:31.206182+0000","last_became_peered":"2021-03-24T21:58:31.206182+0000","last_unstale":"2021-03-24T21:58:57.290436+0000","last_undegraded":"2021-03-24T21:58:31.184913+0000","last_fullsized":"2021-03-24T21:58:31.184684+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2021-03-24T21:56:53.443554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:53.443554+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":12,"num_write":12,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":3756,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":10}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":"334","reported_epoch":"367","state":"active+undersized","last_fresh":"2021-03-24T21:58:57.284975+0000","last_change":"2021-03-24T21:58:31.222882+0000","last_active":"2021-03-24T21:58:57.284975+0000","last_peered":"2021-03-24T21:58:57.284975+0000","last_clean":"2021-03-24T21:58:29.214071+0000","last_became_active":"2021-03-24T21:58:31.222882+0000","last_became_peered":"2021-03-24T21:58:31.222882+0000","last_unstale":"2021-03-24T21:58:57.284975+0000","last_undegraded":"2021-03-24T21:58:57.284975+0000","last_fullsized":"2021-03-24T21:58:31.191156+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:11.229895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:11.229895+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"286","reported_epoch":"366","state":"active+clean","last_fresh":"2021-03-24T21:58:56.260813+0000","last_change":"2021-03-24T21:57:56.549118+0000","last_active":"2021-03-24T21:58:56.260813+0000","last_peered":"2021-03-24T21:58:56.260813+0000","last_clean":"2021-03-24T21:58:56.260813+0000","last_became_active":"2021-03-24T21:54:24.187568+0000","last_became_peered":"2021-03-24T21:54:24.187568+0000","last_unstale":"2021-03-24T21:58:56.260813+0000","last_undegraded":"2021-03-24T21:58:56.260813+0000","last_fullsized":"2021-03-24T21:58:56.260813+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:56.548933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:57:56.548933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"318","reported_epoch":"363","state":"active+clean","last_fresh":"2021-03-24T21:58:53.503428+0000","last_change":"2021-03-24T21:57:59.979943+0000","last_active":"2021-03-24T21:58:53.503428+0000","last_peered":"2021-03-24T21:58:53.503428+0000","last_clean":"2021-03-24T21:58:53.503428+0000","last_became_active":"2021-03-24T21:51:51.762150+0000","last_became_peered":"2021-03-24T21:51:51.762150+0000","last_unstale":"2021-03-24T21:58:53.503428+0000","last_undegraded":"2021-03-24T21:58:53.503428+0000","last_fullsized":"2021-03-24T21:58:53.503428+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:59.979772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:57:59.979772+0000","last_clean_scrub_stamp":"2021-03-24T21:57:59.979772+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.13","version":"355'274","reported_seq":"1133","reported_epoch":"367","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:58:57.322621+0000","last_change":"2021-03-24T21:58:31.208850+0000","last_active":"2021-03-24T21:58:57.322621+0000","last_peered":"2021-03-24T21:58:57.322621+0000","last_clean":"2021-03-24T21:58:29.214984+0000","last_became_active":"2021-03-24T21:58:31.208850+0000","last_became_peered":"2021-03-24T21:58:31.208850+0000","last_unstale":"2021-03-24T21:58:57.322621+0000","last_undegraded":"2021-03-24T21:58:31.188986+0000","last_fullsized":"2021-03-24T21:58:31.188796+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":274,"ondisk_log_size":274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"stat_sum":{"num_bytes":22499052,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":128,"num_read_kb":130045,"num_write":87,"num_write_kb":45287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":4773188,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":8}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"69'17","reported_seq":"362","reported_epoch":"367","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:58:57.284553+0000","last_change":"2021-03-24T21:58:31.200099+0000","last_active":"2021-03-24T21:58:57.284553+0000","last_peered":"2021-03-24T21:58:57.284553+0000","last_clean":"2021-03-24T21:58:29.213514+0000","last_became_active":"2021-03-24T21:58:31.200099+0000","last_became_peered":"2021-03-24T21:58:31.200099+0000","last_unstale":"2021-03-24T21:58:57.284553+0000","last_undegraded":"2021-03-24T21:58:31.189311+0000","last_fullsized":"2021-03-24T21:58:31.189194+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'17","last_scrub_stamp":"2021-03-24T21:55:32.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:32.214502+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":37,"num_read_kb":18,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":2100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":11}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":"332","reported_epoch":"367","state":"active+clean","last_fresh":"2021-03-24T21:58:57.290355+0000","last_change":"2021-03-24T21:55:32.566923+0000","last_active":"2021-03-24T21:58:57.290355+0000","last_peered":"2021-03-24T21:58:57.290355+0000","last_clean":"2021-03-24T21:58:57.290355+0000","last_became_active":"2021-03-24T21:54:38.145870+0000","last_became_peered":"2021-03-24T21:54:38.145870+0000","last_unstale":"2021-03-24T21:58:57.290355+0000","last_undegraded":"2021-03-24T21:58:57.290355+0000","last_fullsized":"2021-03-24T21:58:57.290355+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:32.566747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:32.566747+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"328","reported_epoch":"367","state":"active+clean","last_fresh":"2021-03-24T21:58:57.286195+0000","last_change":"2021-03-24T21:56:35.141430+0000","last_active":"2021-03-24T21:58:57.286195+0000","last_peered":"2021-03-24T21:58:57.286195+0000","last_clean":"2021-03-24T21:58:57.286195+0000","last_became_active":"2021-03-24T21:56:35.140997+0000","last_became_peered":"2021-03-24T21:56:35.140997+0000","last_unstale":"2021-03-24T21:58:57.286195+0000","last_undegraded":"2021-03-24T21:58:57.286195+0000","last_fullsized":"2021-03-24T21:58:57.286195+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:07.255496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:07.255496+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"38","reported_epoch":"366","state":"active+undersized","last_fresh":"2021-03-24T21:58:56.263334+0000","last_change":"2021-03-24T21:58:31.198325+0000","last_active":"2021-03-24T21:58:56.263334+0000","last_peered":"2021-03-24T21:58:56.263334+0000","last_clean":"2021-03-24T21:51:50.731137+0000","last_became_active":"2021-03-24T21:58:31.198325+0000","last_became_peered":"2021-03-24T21:58:31.198325+0000","last_unstale":"2021-03-24T21:58:56.263334+0000","last_undegraded":"2021-03-24T21:58:56.263334+0000","last_fullsized":"2021-03-24T21:58:31.193484+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:26.887939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:58:26.887939+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"273'245","reported_seq":"1072","reported_epoch":"367","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:58:57.305034+0000","last_change":"2021-03-24T21:58:31.206732+0000","last_active":"2021-03-24T21:58:57.305034+0000","last_peered":"2021-03-24T21:58:57.305034+0000","last_clean":"2021-03-24T21:58:29.207340+0000","last_became_active":"2021-03-24T21:58:31.206732+0000","last_became_peered":"2021-03-24T21:58:31.206732+0000","last_unstale":"2021-03-24T21:58:57.305034+0000","last_undegraded":"2021-03-24T21:58:31.188200+0000","last_fullsized":"2021-03-24T21:58:31.188045+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":245,"ondisk_log_size":245,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"stat_sum":{"num_bytes":8581786,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":70,"num_read_kb":67597,"num_write":52,"num_write_kb":18055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"89'27","reported_seq":"409","reported_epoch":"367","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:58:57.297997+0000","last_change":"2021-03-24T21:58:31.228394+0000","last_active":"2021-03-24T21:58:57.297997+0000","last_peered":"2021-03-24T21:58:57.297997+0000","last_clean":"2021-03-24T21:58:29.213813+0000","last_became_active":"2021-03-24T21:58:31.228394+0000","last_became_peered":"2021-03-24T21:58:31.228394+0000","last_unstale":"2021-03-24T21:58:57.297997+0000","last_undegraded":"2021-03-24T21:58:31.192365+0000","last_fullsized":"2021-03-24T21:58:31.192251+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'27","last_scrub_stamp":"2021-03-24T21:55:41.337130+0000","last_deep_scrub":"89'27","last_deep_scrub_stamp":"2021-03-24T21:55:41.337130+0000","last_clean_scrub_stamp":"2021-03-24T21:55:41.337130+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":934,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":11}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":"303","reported_epoch":"367","state":"active+undersized","last_fresh":"2021-03-24T21:58:57.309366+0000","last_change":"2021-03-24T21:58:31.207653+0000","last_active":"2021-03-24T21:58:57.309366+0000","last_peered":"2021-03-24T21:58:57.309366+0000","last_clean":"2021-03-24T21:58:29.205110+0000","last_became_active":"2021-03-24T21:58:31.207653+0000","last_became_peered":"2021-03-24T21:58:31.207653+0000","last_unstale":"2021-03-24T21:58:57.309366+0000","last_undegraded":"2021-03-24T21:58:57.309366+0000","last_fullsized":"2021-03-24T21:58:31.185073+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:26.417613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:57:26.417613+0000","last_clean_scrub_stamp":"2021-03-24T21:57:26.417613+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"235'4","reported_seq":"362","reported_epoch":"366","state":"active+clean","last_fresh":"2021-03-24T21:58:56.271174+0000","last_change":"2021-03-24T21:56:44.318340+0000","last_active":"2021-03-24T21:58:56.271174+0000","last_peered":"2021-03-24T21:58:56.271174+0000","last_clean":"2021-03-24T21:58:56.271174+0000","last_became_active":"2021-03-24T21:56:44.228196+0000","last_became_peered":"2021-03-24T21:56:44.228196+0000","last_unstale":"2021-03-24T21:58:56.271174+0000","last_undegraded":"2021-03-24T21:58:56.271174+0000","last_fullsized":"2021-03-24T21:58:56.271174+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'3","last_scrub_stamp":"2021-03-24T21:56:42.039976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:42.039976+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"77","reported_epoch":"365","state":"active+undersized","last_fresh":"2021-03-24T21:58:55.226468+0000","last_change":"2021-03-24T21:58:31.205748+0000","last_active":"2021-03-24T21:58:55.226468+0000","last_peered":"2021-03-24T21:58:55.226468+0000","last_clean":"2021-03-24T21:58:29.217060+0000","last_became_active":"2021-03-24T21:58:31.205748+0000","last_became_peered":"2021-03-24T21:58:31.205748+0000","last_unstale":"2021-03-24T21:58:55.226468+0000","last_undegraded":"2021-03-24T21:58:55.226468+0000","last_fullsized":"2021-03-24T21:58:31.191168+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:26.129924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:57:26.129924+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1d","version":"359'310","reported_seq":"1194","reported_epoch":"367","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:58:57.283827+0000","last_change":"2021-03-24T21:58:31.203265+0000","last_active":"2021-03-24T21:58:57.283827+0000","last_peered":"2021-03-24T21:58:57.283827+0000","last_clean":"2021-03-24T21:58:29.213086+0000","last_became_active":"2021-03-24T21:58:31.203265+0000","last_became_peered":"2021-03-24T21:58:31.203265+0000","last_unstale":"2021-03-24T21:58:57.283827+0000","last_undegraded":"2021-03-24T21:58:31.188863+0000","last_fullsized":"2021-03-24T21:58:31.188701+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"233'169","last_scrub_stamp":"2021-03-24T21:56:47.285648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:56:47.285648+0000","log_size":310,"ondisk_log_size":310,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"stat_sum":{"num_bytes":15220819,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":90,"num_read_kb":80658,"num_write":89,"num_write_kb":41497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":9406781,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":5}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1d","version":"69'16","reported_seq":"387","reported_epoch":"367","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:58:57.316769+0000","last_change":"2021-03-24T21:58:31.229186+0000","last_active":"2021-03-24T21:58:57.316769+0000","last_peered":"2021-03-24T21:58:57.316769+0000","last_clean":"2021-03-24T21:58:29.206116+0000","last_became_active":"2021-03-24T21:58:31.229186+0000","last_became_peered":"2021-03-24T21:58:31.229186+0000","last_unstale":"2021-03-24T21:58:57.316769+0000","last_undegraded":"2021-03-24T21:58:31.196334+0000","last_fullsized":"2021-03-24T21:58:31.196221+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T21:56:10.423860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:10.423860+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":37,"num_read_kb":20,"num_write":18,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":844,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":9}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":"337","reported_epoch":"366","state":"active+clean","last_fresh":"2021-03-24T21:58:56.266517+0000","last_change":"2021-03-24T21:57:13.701322+0000","last_active":"2021-03-24T21:58:56.266517+0000","last_peered":"2021-03-24T21:58:56.266517+0000","last_clean":"2021-03-24T21:58:56.266517+0000","last_became_active":"2021-03-24T21:54:38.143876+0000","last_became_peered":"2021-03-24T21:54:38.143876+0000","last_unstale":"2021-03-24T21:58:56.266517+0000","last_undegraded":"2021-03-24T21:58:56.266517+0000","last_fullsized":"2021-03-24T21:58:56.266517+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:13.701143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:52:54.212459+0000","last_clean_scrub_stamp":"2021-03-24T21:57:13.701143+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"324","reported_epoch":"366","state":"active+clean","last_fresh":"2021-03-24T21:58:56.264871+0000","last_change":"2021-03-24T21:57:19.767717+0000","last_active":"2021-03-24T21:58:56.264871+0000","last_peered":"2021-03-24T21:58:56.264871+0000","last_clean":"2021-03-24T21:58:56.264871+0000","last_became_active":"2021-03-24T21:51:49.410920+0000","last_became_peered":"2021-03-24T21:51:49.410920+0000","last_unstale":"2021-03-24T21:58:56.264871+0000","last_undegraded":"2021-03-24T21:58:56.264871+0000","last_fullsized":"2021-03-24T21:58:56.264871+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:19.767465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:57:19.767465+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"341","reported_epoch":"367","state":"active+undersized","last_fresh":"2021-03-24T21:58:57.297295+0000","last_change":"2021-03-24T21:58:31.196650+0000","last_active":"2021-03-24T21:58:57.297295+0000","last_peered":"2021-03-24T21:58:57.297295+0000","last_clean":"2021-03-24T21:58:29.208496+0000","last_became_active":"2021-03-24T21:58:31.196650+0000","last_became_peered":"2021-03-24T21:58:31.196650+0000","last_unstale":"2021-03-24T21:58:57.297295+0000","last_undegraded":"2021-03-24T21:58:57.297295+0000","last_fullsized":"2021-03-24T21:58:31.188039+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:33.440618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:33.440618+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1e","version":"265'45","reported_seq":"236","reported_epoch":"367","state":"active+undersized","last_fresh":"2021-03-24T21:58:57.297819+0000","last_change":"2021-03-24T21:58:31.206881+0000","last_active":"2021-03-24T21:58:57.297819+0000","last_peered":"2021-03-24T21:58:57.297819+0000","last_clean":"2021-03-24T21:58:29.205698+0000","last_became_active":"2021-03-24T21:58:31.206881+0000","last_became_peered":"2021-03-24T21:58:31.206881+0000","last_unstale":"2021-03-24T21:58:57.297819+0000","last_undegraded":"2021-03-24T21:58:57.297819+0000","last_fullsized":"2021-03-24T21:58:31.185389+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":5,"last_scrub":"199'30","last_scrub_stamp":"2021-03-24T21:56:10.105084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:56:10.105084+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":18305,"num_write":13,"num_write_kb":3060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"69'15","reported_seq":"208","reported_epoch":"367","state":"active+clean","last_fresh":"2021-03-24T21:58:57.305564+0000","last_change":"2021-03-24T21:56:35.748629+0000","last_active":"2021-03-24T21:58:57.305564+0000","last_peered":"2021-03-24T21:58:57.305564+0000","last_clean":"2021-03-24T21:58:57.305564+0000","last_became_active":"2021-03-24T21:56:35.142209+0000","last_became_peered":"2021-03-24T21:56:35.142209+0000","last_unstale":"2021-03-24T21:58:57.305564+0000","last_undegraded":"2021-03-24T21:58:57.305564+0000","last_fullsized":"2021-03-24T21:58:57.305564+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:56:06.124139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:06.124139+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":"328","reported_epoch":"367","state":"active+clean","last_fresh":"2021-03-24T21:58:57.284818+0000","last_change":"2021-03-24T21:56:42.254648+0000","last_active":"2021-03-24T21:58:57.284818+0000","last_peered":"2021-03-24T21:58:57.284818+0000","last_clean":"2021-03-24T21:58:57.284818+0000","last_became_active":"2021-03-24T21:51:45.381533+0000","last_became_peered":"2021-03-24T21:51:45.381533+0000","last_unstale":"2021-03-24T21:58:57.284818+0000","last_undegraded":"2021-03-24T21:58:57.284818+0000","last_fullsized":"2021-03-24T21:58:57.284818+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:42.254536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:42.254536+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"286","reported_epoch":"366","state":"active+undersized","last_fresh":"2021-03-24T21:58:56.263897+0000","last_change":"2021-03-24T21:58:31.208452+0000","last_active":"2021-03-24T21:58:56.263897+0000","last_peered":"2021-03-24T21:58:56.263897+0000","last_clean":"2021-03-24T21:58:29.217323+0000","last_became_active":"2021-03-24T21:58:31.208452+0000","last_became_peered":"2021-03-24T21:58:31.208452+0000","last_unstale":"2021-03-24T21:58:56.263897+0000","last_undegraded":"2021-03-24T21:58:56.263897+0000","last_fullsized":"2021-03-24T21:58:31.193967+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:26.642883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:57:26.642883+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"332","reported_epoch":"366","state":"active+undersized","last_fresh":"2021-03-24T21:58:56.270225+0000","last_change":"2021-03-24T21:58:31.223458+0000","last_active":"2021-03-24T21:58:56.270225+0000","last_peered":"2021-03-24T21:58:56.270225+0000","last_clean":"2021-03-24T21:58:29.217708+0000","last_became_active":"2021-03-24T21:58:31.223458+0000","last_became_peered":"2021-03-24T21:58:31.223458+0000","last_unstale":"2021-03-24T21:58:56.270225+0000","last_undegraded":"2021-03-24T21:58:56.270225+0000","last_fullsized":"2021-03-24T21:58:31.189856+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:38.075714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:38.075714+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1f","version":"272'99","reported_seq":"235","reported_epoch":"366","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:58:56.270360+0000","last_change":"2021-03-24T21:58:31.223352+0000","last_active":"2021-03-24T21:58:56.270360+0000","last_peered":"2021-03-24T21:58:56.270360+0000","last_clean":"2021-03-24T21:58:29.217518+0000","last_became_active":"2021-03-24T21:58:31.223352+0000","last_became_peered":"2021-03-24T21:58:31.223352+0000","last_unstale":"2021-03-24T21:58:56.270360+0000","last_undegraded":"2021-03-24T21:58:31.189248+0000","last_fullsized":"2021-03-24T21:58:31.188821+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":5,"last_scrub":"218'73","last_scrub_stamp":"2021-03-24T21:56:48.267711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:56:48.267711+0000","log_size":99,"ondisk_log_size":99,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"stat_sum":{"num_bytes":3571504,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":43,"num_read_kb":44484,"num_write":22,"num_write_kb":9163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":942483,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"91'9","reported_seq":"375","reported_epoch":"366","state":"active+clean","last_fresh":"2021-03-24T21:58:56.262194+0000","last_change":"2021-03-24T21:58:55.660850+0000","last_active":"2021-03-24T21:58:56.262194+0000","last_peered":"2021-03-24T21:58:56.262194+0000","last_clean":"2021-03-24T21:58:56.262194+0000","last_became_active":"2021-03-24T21:57:02.184317+0000","last_became_peered":"2021-03-24T21:57:02.184317+0000","last_unstale":"2021-03-24T21:58:56.262194+0000","last_undegraded":"2021-03-24T21:58:56.262194+0000","last_fullsized":"2021-03-24T21:58:56.262194+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2021-03-24T21:58:55.660737+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2021-03-24T21:53:33.290000+0000","last_clean_scrub_stamp":"2021-03-24T21:58:55.660737+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":8,"num_write":7,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":"318","reported_epoch":"366","state":"active+undersized","last_fresh":"2021-03-24T21:58:56.263535+0000","last_change":"2021-03-24T21:58:31.202563+0000","last_active":"2021-03-24T21:58:56.263535+0000","last_peered":"2021-03-24T21:58:56.263535+0000","last_clean":"2021-03-24T21:58:29.214992+0000","last_became_active":"2021-03-24T21:58:31.202563+0000","last_became_peered":"2021-03-24T21:58:31.202563+0000","last_unstale":"2021-03-24T21:58:56.263535+0000","last_undegraded":"2021-03-24T21:58:56.263535+0000","last_fullsized":"2021-03-24T21:58:31.192438+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:47.920949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:47.920949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"329","reported_epoch":"363","state":"active+clean","last_fresh":"2021-03-24T21:58:53.503289+0000","last_change":"2021-03-24T21:56:51.977473+0000","last_active":"2021-03-24T21:58:53.503289+0000","last_peered":"2021-03-24T21:58:53.503289+0000","last_clean":"2021-03-24T21:58:53.503289+0000","last_became_active":"2021-03-24T21:54:38.137706+0000","last_became_peered":"2021-03-24T21:54:38.137706+0000","last_unstale":"2021-03-24T21:58:53.503289+0000","last_undegraded":"2021-03-24T21:58:53.503289+0000","last_fullsized":"2021-03-24T21:58:53.503289+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:51.977306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:56:51.977306+0000","last_clean_scrub_stamp":"2021-03-24T21:56:51.977306+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"92","reported_epoch":"367","state":"active+undersized","last_fresh":"2021-03-24T21:58:57.296667+0000","last_change":"2021-03-24T21:58:31.196725+0000","last_active":"2021-03-24T21:58:57.296667+0000","last_peered":"2021-03-24T21:58:57.296667+0000","last_clean":"2021-03-24T21:54:36.085927+0000","last_became_active":"2021-03-24T21:58:31.196725+0000","last_became_peered":"2021-03-24T21:58:31.196725+0000","last_unstale":"2021-03-24T21:58:57.296667+0000","last_undegraded":"2021-03-24T21:58:57.296667+0000","last_fullsized":"2021-03-24T21:58:31.184385+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:16.076982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:57:16.076982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.18","version":"342'94","reported_seq":"514","reported_epoch":"367","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:58:57.282097+0000","last_change":"2021-03-24T21:58:32.440753+0000","last_active":"2021-03-24T21:58:57.282097+0000","last_peered":"2021-03-24T21:58:57.282097+0000","last_clean":"2021-03-24T21:58:29.211604+0000","last_became_active":"2021-03-24T21:58:31.208262+0000","last_became_peered":"2021-03-24T21:58:31.208262+0000","last_unstale":"2021-03-24T21:58:57.282097+0000","last_undegraded":"2021-03-24T21:58:32.436918+0000","last_fullsized":"2021-03-24T21:58:31.193361+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":5,"last_scrub":"249'90","last_scrub_stamp":"2021-03-24T21:57:56.848784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:57:56.848784+0000","log_size":94,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"stat_sum":{"num_bytes":3462308,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":5177,"num_write":5,"num_write_kb":1725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"69'13","reported_seq":"350","reported_epoch":"366","state":"active+clean","last_fresh":"2021-03-24T21:58:56.266431+0000","last_change":"2021-03-24T21:58:26.698101+0000","last_active":"2021-03-24T21:58:56.266431+0000","last_peered":"2021-03-24T21:58:56.266431+0000","last_clean":"2021-03-24T21:58:56.266431+0000","last_became_active":"2021-03-24T21:51:47.407488+0000","last_became_peered":"2021-03-24T21:51:47.407488+0000","last_unstale":"2021-03-24T21:58:56.266431+0000","last_undegraded":"2021-03-24T21:58:56.266431+0000","last_fullsized":"2021-03-24T21:58:56.266431+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2021-03-24T21:58:26.697918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:58:26.697918+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":"365","reported_epoch":"365","state":"active+undersized","last_fresh":"2021-03-24T21:58:55.225553+0000","last_change":"2021-03-24T21:58:31.205041+0000","last_active":"2021-03-24T21:58:55.225553+0000","last_peered":"2021-03-24T21:58:55.225553+0000","last_clean":"2021-03-24T21:58:29.217058+0000","last_became_active":"2021-03-24T21:58:31.205041+0000","last_became_peered":"2021-03-24T21:58:31.205041+0000","last_unstale":"2021-03-24T21:58:55.225553+0000","last_undegraded":"2021-03-24T21:58:55.225553+0000","last_fullsized":"2021-03-24T21:58:31.191384+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:24.418893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:58:24.418893+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"36","reported_epoch":"367","state":"active+undersized","last_fresh":"2021-03-24T21:58:57.284166+0000","last_change":"2021-03-24T21:58:31.208343+0000","last_active":"2021-03-24T21:58:57.284166+0000","last_peered":"2021-03-24T21:58:57.284166+0000","last_clean":"2021-03-24T21:51:48.384232+0000","last_became_active":"2021-03-24T21:58:31.208343+0000","last_became_peered":"2021-03-24T21:58:31.208343+0000","last_unstale":"2021-03-24T21:58:57.284166+0000","last_undegraded":"2021-03-24T21:58:57.284166+0000","last_fullsized":"2021-03-24T21:58:31.191979+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:38.242096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:38.242096+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"331","reported_epoch":"367","state":"active+clean","last_fresh":"2021-03-24T21:58:57.283579+0000","last_change":"2021-03-24T21:57:19.271463+0000","last_active":"2021-03-24T21:58:57.283579+0000","last_peered":"2021-03-24T21:58:57.283579+0000","last_clean":"2021-03-24T21:58:57.283579+0000","last_became_active":"2021-03-24T21:54:38.127696+0000","last_became_peered":"2021-03-24T21:54:38.127696+0000","last_unstale":"2021-03-24T21:58:57.283579+0000","last_undegraded":"2021-03-24T21:58:57.283579+0000","last_fullsized":"2021-03-24T21:58:57.283579+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:19.271351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:57:19.271351+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.19","version":"362'372","reported_seq":"1254","reported_epoch":"363","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:58:53.503090+0000","last_change":"2021-03-24T21:58:31.209291+0000","last_active":"2021-03-24T21:58:53.503090+0000","last_peered":"2021-03-24T21:58:53.503090+0000","last_clean":"2021-03-24T21:58:29.200505+0000","last_became_active":"2021-03-24T21:58:31.209291+0000","last_became_peered":"2021-03-24T21:58:31.209291+0000","last_unstale":"2021-03-24T21:58:53.503090+0000","last_undegraded":"2021-03-24T21:58:31.197311+0000","last_fullsized":"2021-03-24T21:58:31.197197+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":5,"last_scrub":"265'260","last_scrub_stamp":"2021-03-24T21:57:22.158029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:57:22.158029+0000","log_size":372,"ondisk_log_size":372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":9,"stat_sum":{"num_bytes":17039443,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":36,"num_read_kb":27707,"num_write":54,"num_write_kb":21757,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":7}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"69'20","reported_seq":"361","reported_epoch":"367","state":"active+clean","last_fresh":"2021-03-24T21:58:57.309788+0000","last_change":"2021-03-24T21:58:31.309185+0000","last_active":"2021-03-24T21:58:57.309788+0000","last_peered":"2021-03-24T21:58:57.309788+0000","last_clean":"2021-03-24T21:58:57.309788+0000","last_became_active":"2021-03-24T21:51:47.405776+0000","last_became_peered":"2021-03-24T21:51:47.405776+0000","last_unstale":"2021-03-24T21:58:57.309788+0000","last_undegraded":"2021-03-24T21:58:57.309788+0000","last_fullsized":"2021-03-24T21:58:57.309788+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'20","last_scrub_stamp":"2021-03-24T21:58:31.309079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:58:31.309079+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":22,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":"341","reported_epoch":"367","state":"active+clean","last_fresh":"2021-03-24T21:58:57.298140+0000","last_change":"2021-03-24T21:55:38.433064+0000","last_active":"2021-03-24T21:58:57.298140+0000","last_peered":"2021-03-24T21:58:57.298140+0000","last_clean":"2021-03-24T21:58:57.298140+0000","last_became_active":"2021-03-24T21:54:38.130163+0000","last_became_peered":"2021-03-24T21:54:38.130163+0000","last_unstale":"2021-03-24T21:58:57.298140+0000","last_undegraded":"2021-03-24T21:58:57.298140+0000","last_fullsized":"2021-03-24T21:58:57.298140+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:38.432918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:38.432918+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"337","reported_epoch":"366","state":"active+clean","last_fresh":"2021-03-24T21:58:56.261208+0000","last_change":"2021-03-24T21:56:34.092326+0000","last_active":"2021-03-24T21:58:56.261208+0000","last_peered":"2021-03-24T21:58:56.261208+0000","last_clean":"2021-03-24T21:58:56.261208+0000","last_became_active":"2021-03-24T21:54:38.120575+0000","last_became_peered":"2021-03-24T21:54:38.120575+0000","last_unstale":"2021-03-24T21:58:56.261208+0000","last_undegraded":"2021-03-24T21:58:56.261208+0000","last_fullsized":"2021-03-24T21:58:56.261208+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:34.092227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:34.092227+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"340","reported_epoch":"367","state":"active+clean","last_fresh":"2021-03-24T21:58:57.306703+0000","last_change":"2021-03-24T21:58:29.298430+0000","last_active":"2021-03-24T21:58:57.306703+0000","last_peered":"2021-03-24T21:58:57.306703+0000","last_clean":"2021-03-24T21:58:57.306703+0000","last_became_active":"2021-03-24T21:54:38.130293+0000","last_became_peered":"2021-03-24T21:54:38.130293+0000","last_unstale":"2021-03-24T21:58:57.306703+0000","last_undegraded":"2021-03-24T21:58:57.306703+0000","last_fullsized":"2021-03-24T21:58:57.306703+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:29.298229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:58:29.298229+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"353'159","reported_seq":"932","reported_epoch":"366","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:58:56.261178+0000","last_change":"2021-03-24T21:58:31.203294+0000","last_active":"2021-03-24T21:58:56.261178+0000","last_peered":"2021-03-24T21:58:56.261178+0000","last_clean":"2021-03-24T21:58:29.213731+0000","last_became_active":"2021-03-24T21:58:31.203294+0000","last_became_peered":"2021-03-24T21:58:31.203294+0000","last_unstale":"2021-03-24T21:58:56.261178+0000","last_undegraded":"2021-03-24T21:58:31.193672+0000","last_fullsized":"2021-03-24T21:58:31.193468+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":159,"ondisk_log_size":159,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"stat_sum":{"num_bytes":5047878,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":30,"num_read_kb":22371,"num_write":50,"num_write_kb":16462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":3266508,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":3}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"69'15","reported_seq":"411","reported_epoch":"366","state":"active+clean","last_fresh":"2021-03-24T21:58:56.260755+0000","last_change":"2021-03-24T21:56:05.966329+0000","last_active":"2021-03-24T21:58:56.260755+0000","last_peered":"2021-03-24T21:58:56.260755+0000","last_clean":"2021-03-24T21:58:56.260755+0000","last_became_active":"2021-03-24T21:54:38.127426+0000","last_became_peered":"2021-03-24T21:54:38.127426+0000","last_unstale":"2021-03-24T21:58:56.260755+0000","last_undegraded":"2021-03-24T21:58:56.260755+0000","last_fullsized":"2021-03-24T21:58:56.260755+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:56:05.966219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:05.966219+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2076,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":"339","reported_epoch":"367","state":"active+undersized","last_fresh":"2021-03-24T21:58:57.297979+0000","last_change":"2021-03-24T21:58:31.216651+0000","last_active":"2021-03-24T21:58:57.297979+0000","last_peered":"2021-03-24T21:58:57.297979+0000","last_clean":"2021-03-24T21:58:29.786750+0000","last_became_active":"2021-03-24T21:58:31.216651+0000","last_became_peered":"2021-03-24T21:58:31.216651+0000","last_unstale":"2021-03-24T21:58:57.297979+0000","last_undegraded":"2021-03-24T21:58:57.297979+0000","last_fullsized":"2021-03-24T21:58:31.191007+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:38.294685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:38.294685+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"253","reported_epoch":"363","state":"active+undersized","last_fresh":"2021-03-24T21:58:53.504143+0000","last_change":"2021-03-24T21:58:31.208824+0000","last_active":"2021-03-24T21:58:53.504143+0000","last_peered":"2021-03-24T21:58:53.504143+0000","last_clean":"2021-03-24T21:58:29.201408+0000","last_became_active":"2021-03-24T21:58:31.208824+0000","last_became_peered":"2021-03-24T21:58:31.208824+0000","last_unstale":"2021-03-24T21:58:53.504143+0000","last_undegraded":"2021-03-24T21:58:53.504143+0000","last_fullsized":"2021-03-24T21:58:31.195827+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:27.920682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:14.710709+0000","last_clean_scrub_stamp":"2021-03-24T21:58:27.920682+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"337","reported_epoch":"366","state":"active+undersized","last_fresh":"2021-03-24T21:58:56.266002+0000","last_change":"2021-03-24T21:58:31.202629+0000","last_active":"2021-03-24T21:58:56.266002+0000","last_peered":"2021-03-24T21:58:56.266002+0000","last_clean":"2021-03-24T21:58:29.214118+0000","last_became_active":"2021-03-24T21:58:31.202629+0000","last_became_peered":"2021-03-24T21:58:31.202629+0000","last_unstale":"2021-03-24T21:58:56.266002+0000","last_undegraded":"2021-03-24T21:58:56.266002+0000","last_fullsized":"2021-03-24T21:58:31.194110+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:15.687165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:57:15.687165+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"367'454","reported_seq":"1556","reported_epoch":"367","state":"active+clean","last_fresh":"2021-03-24T21:58:57.322818+0000","last_change":"2021-03-24T21:58:55.285448+0000","last_active":"2021-03-24T21:58:57.322818+0000","last_peered":"2021-03-24T21:58:57.322818+0000","last_clean":"2021-03-24T21:58:57.322818+0000","last_became_active":"2021-03-24T21:57:33.176110+0000","last_became_peered":"2021-03-24T21:57:33.176110+0000","last_unstale":"2021-03-24T21:58:57.322818+0000","last_undegraded":"2021-03-24T21:58:57.322818+0000","last_fullsized":"2021-03-24T21:58:57.322818+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":5,"last_scrub":"191'150","last_scrub_stamp":"2021-03-24T21:56:01.308050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:56:01.308050+0000","log_size":454,"ondisk_log_size":454,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15368875,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":234,"num_read_kb":259055,"num_write":131,"num_write_kb":46892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":28585116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"df","length":"1"},{"start":"e1","length":"1"},{"start":"ea","length":"4"},{"start":"f2","length":"3"},{"start":"fc","length":"1"}]},{"pgid":"7.20","version":"175'22","reported_seq":"534","reported_epoch":"365","state":"active+clean","last_fresh":"2021-03-24T21:58:55.222845+0000","last_change":"2021-03-24T21:58:55.222845+0000","last_active":"2021-03-24T21:58:55.222845+0000","last_peered":"2021-03-24T21:58:55.222845+0000","last_clean":"2021-03-24T21:58:55.222845+0000","last_became_active":"2021-03-24T21:57:28.171436+0000","last_became_peered":"2021-03-24T21:57:28.171436+0000","last_unstale":"2021-03-24T21:58:55.222845+0000","last_undegraded":"2021-03-24T21:58:55.222845+0000","last_fullsized":"2021-03-24T21:58:55.222845+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":6,"last_scrub":"175'22","last_scrub_stamp":"2021-03-24T21:57:32.121490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:57:32.121490+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"df","length":"1"},{"start":"e1","length":"1"},{"start":"ea","length":"4"},{"start":"f2","length":"3"},{"start":"fc","length":"1"}]},{"pgid":"7.21","version":"337'297","reported_seq":"1215","reported_epoch":"367","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:58:57.311607+0000","last_change":"2021-03-24T21:58:31.203128+0000","last_active":"2021-03-24T21:58:57.311607+0000","last_peered":"2021-03-24T21:58:57.311607+0000","last_clean":"2021-03-24T21:58:29.205714+0000","last_became_active":"2021-03-24T21:58:31.203128+0000","last_became_peered":"2021-03-24T21:58:31.203128+0000","last_unstale":"2021-03-24T21:58:57.311607+0000","last_undegraded":"2021-03-24T21:58:31.186929+0000","last_fullsized":"2021-03-24T21:58:31.185968+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":297,"ondisk_log_size":297,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"stat_sum":{"num_bytes":5189837,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":105,"num_read_kb":110442,"num_write":53,"num_write_kb":20170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":10874449,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":36,"stat_sum":{"num_bytes":271730902,"num_objects":120,"num_object_clones":74,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":77,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":120,"num_whiteouts":7,"num_read":1973,"num_read_kb":2082675,"num_write":1378,"num_write_kb":545979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":189,"num_bytes_recovered":98236250,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":598134784,"data_stored":749015039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7226,"ondisk_log_size":7226,"up":87,"acting":87,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":426,"num_read_kb":421,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":8,"acting":8,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1589,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":5558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":80,"acting":80,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":159,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1060,"num_read_kb":565,"num_write":542,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":281,"num_bytes_recovered":23006,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2027520,"data_stored":72734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":558,"ondisk_log_size":558,"up":80,"acting":80,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":33,"ondisk_log_size":33,"up":78,"acting":78,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":78,"acting":78,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201370,"num_pgs":88,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":161816,"kb_used_data":137232,"kb_used_omap":0,"kb_used_meta":24576,"kb_avail":93583336,"statfs":{"total":95995035648,"available":95829336064,"internally_reserved":0,"allocated":140525568,"data_stored":166821356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":25165824},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:57:54 2021","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":1.018,"15min":0.93799999999999994},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":1.1479999999999999,"5min":12.222,"15min":12.222},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.96699999999999997,"5min":1.0249999999999999,"15min":0.99099999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":2.8149999999999999,"5min":11.506,"15min":11.506},"last":0.54000000000000004}]},{"osd":1,"last update":"Wed Mar 24 21:57:54 2021","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.99199999999999999,"15min":0.88500000000000001},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.476,"5min":5.7750000000000004,"15min":5.7750000000000004},"last":0.496},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.0109999999999999,"15min":0.98199999999999998},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":4.0149999999999997,"5min":6.0780000000000003,"15min":6.0780000000000003},"last":0.60799999999999998}]},{"osd":2,"last update":"Wed Mar 24 21:57:54 2021","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":1.169,"15min":1.052},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":2.9329999999999998,"5min":12.135999999999999,"15min":12.135999999999999},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.91700000000000004,"5min":1.1220000000000001,"15min":1.048},"min":{"1min":0.54400000000000004,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.8129999999999999,"5min":12.178000000000001,"15min":12.178000000000001},"last":0.57299999999999995}]},{"osd":3,"last update":"Wed Mar 24 21:57:54 2021","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":1.173,"15min":1.087},"min":{"1min":0.56299999999999994,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.9219999999999999,"5min":12.529,"15min":12.529},"last":1.607},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.2,"15min":1.046},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.347,"5min":12.666,"15min":12.666},"last":0.72799999999999998}]},{"osd":4,"last update":"Wed Mar 24 21:57:54 2021","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":1.0620000000000001,"15min":0.96799999999999997},"min":{"1min":0.629,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.522,"5min":6.2089999999999996,"15min":6.2089999999999996},"last":0.68200000000000005},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.1739999999999999,"15min":1.107},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.671,"5min":12.244,"15min":12.244},"last":1.1639999999999999}]},{"osd":5,"last update":"Wed Mar 24 21:57:54 2021","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.153,"15min":1.0269999999999999},"min":{"1min":0.64100000000000001,"5min":0.435,"15min":0.435},"max":{"1min":1.4330000000000001,"5min":5.8840000000000003,"15min":5.8840000000000003},"last":1.2430000000000001},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.1910000000000001,"15min":1.1539999999999999},"min":{"1min":0.628,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.629,"5min":6.3300000000000001,"15min":6.3300000000000001},"last":1.1799999999999999}]},{"osd":6,"last update":"Wed Mar 24 21:57:54 2021","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.232,"15min":1.1479999999999999},"min":{"1min":0.66600000000000004,"5min":0.66000000000000003,"15min":0.62},"max":{"1min":1.577,"5min":6.1159999999999997,"15min":6.1159999999999997},"last":0.64800000000000002},{"interface":"front","average":{"1min":1.117,"5min":1.3029999999999999,"15min":1.2150000000000001},"min":{"1min":0.60499999999999998,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.744,"5min":12.287000000000001,"15min":12.287000000000001},"last":1.1080000000000001}]}]},{"osd":6,"up_from":0,"seq":201863463007,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":0,"seq":55834574964,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738488,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":112932,"kb_used_data":93532,"kb_used_omap":0,"kb_used_meta":19392,"kb_avail":93632220,"statfs":{"total":95995035648,"available":95879393280,"internally_reserved":0,"allocated":95776768,"data_stored":118205147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":19857408},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Mar 24 21:58:47 2021","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.69099999999999995,"15min":0.629},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.99099999999999999,"5min":4.8040000000000003,"15min":4.8040000000000003},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.83899999999999997,"15min":0.69699999999999995},"min":{"1min":0.47199999999999998,"5min":0.374,"15min":0.374},"max":{"1min":2.5179999999999998,"5min":8.8989999999999991,"15min":8.8989999999999991},"last":1.079}]},{"osd":2,"last update":"Wed Mar 24 21:58:05 2021","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.78100000000000003,"15min":0.70299999999999996},"min":{"1min":0.35399999999999998,"5min":0.34200000000000003,"15min":0.314},"max":{"1min":1.351,"5min":5.8810000000000002,"15min":5.8810000000000002},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.84599999999999997,"15min":0.749},"min":{"1min":0.50900000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.462,"5min":4.766,"15min":4.766},"last":0.61699999999999999}]},{"osd":3,"last update":"Wed Mar 24 21:58:23 2021","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":0.91400000000000003,"15min":0.83899999999999997},"min":{"1min":0.49299999999999999,"5min":0.434,"15min":0.376},"max":{"1min":2.79,"5min":5.9790000000000001,"15min":5.9790000000000001},"last":1.149},{"interface":"front","average":{"1min":1.036,"5min":0.97699999999999998,"15min":0.94699999999999995},"min":{"1min":0.60099999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":2.3180000000000001,"5min":6.4580000000000002,"15min":6.4580000000000002},"last":1.0580000000000001}]},{"osd":4,"last update":"Wed Mar 24 21:58:47 2021","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.085,"15min":1.091},"min":{"1min":0.55300000000000005,"5min":0.436,"15min":0.436},"max":{"1min":2.496,"5min":5.0330000000000004,"15min":5.0330000000000004},"last":1.04},{"interface":"front","average":{"1min":1.073,"5min":1.1160000000000001,"15min":1.05},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":2.2629999999999999,"5min":6.1829999999999998,"15min":6.1829999999999998},"last":1.1679999999999999}]},{"osd":5,"last update":"Wed Mar 24 21:58:05 2021","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.1040000000000001,"15min":1.0740000000000001},"min":{"1min":0.64400000000000002,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.603,"5min":5.9390000000000001,"15min":5.9390000000000001},"last":0.88900000000000001},{"interface":"front","average":{"1min":1.103,"5min":1.167,"15min":1.1739999999999999},"min":{"1min":0.75900000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.645,"5min":5.0590000000000002,"15min":5.0590000000000002},"last":1.0189999999999999}]},{"osd":6,"last update":"Wed Mar 24 21:58:23 2021","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.173,"15min":1.1020000000000001},"min":{"1min":0.755,"5min":0.755,"15min":0.72699999999999998},"max":{"1min":1.6839999999999999,"5min":5.9930000000000003,"15min":5.9930000000000003},"last":0.77800000000000002},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.2110000000000001,"15min":1.218},"min":{"1min":0.78000000000000003,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":2.21,"5min":4.9960000000000004,"15min":4.9960000000000004},"last":1.0149999999999999}]},{"osd":7,"last update":"Wed Mar 24 21:57:56 2021","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.212,"15min":1.1819999999999999},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":2.8399999999999999,"5min":6.1319999999999997,"15min":6.1319999999999997},"last":1.123},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.206,"15min":1.1519999999999999},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.65800000000000003},"max":{"1min":1.5720000000000001,"5min":6.0529999999999999,"15min":6.0529999999999999},"last":0.75600000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411441,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":146420,"kb_used_data":126828,"kb_used_omap":0,"kb_used_meta":19584,"kb_avail":93598732,"statfs":{"total":95995035648,"available":95845101568,"internally_reserved":0,"allocated":129871872,"data_stored":169040316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20054016},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:58:04 2021","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.68600000000000005,"15min":0.64500000000000002},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":1.133,"5min":1.49,"15min":1.49},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.71099999999999997,"15min":0.69699999999999995},"min":{"1min":0.32400000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.93600000000000005,"5min":2.3119999999999998,"15min":2.3119999999999998},"last":0.68200000000000005}]},{"osd":1,"last update":"Wed Mar 24 21:58:04 2021","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.75,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.2010000000000001,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.77500000000000002,"15min":0.75},"min":{"1min":0.36299999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.2609999999999999,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":0.88600000000000001}]},{"osd":3,"last update":"Wed Mar 24 21:58:22 2021","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.86699999999999999,"15min":0.84099999999999997},"min":{"1min":0.36799999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.9610000000000001,"5min":1.9610000000000001,"15min":1.9890000000000001},"last":0.503},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.89600000000000002,"15min":0.89900000000000002},"min":{"1min":0.38300000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.8859999999999999,"5min":1.8859999999999999,"15min":2.0950000000000002},"last":0.754}]},{"osd":4,"last update":"Wed Mar 24 21:58:48 2021","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":1.0600000000000001,"15min":1.0449999999999999},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":2.5390000000000001,"5min":2.5390000000000001,"15min":3.3380000000000001},"last":0.71699999999999997},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.0589999999999999,"15min":1.054},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":2.3119999999999998,"5min":2.3119999999999998,"15min":2.3119999999999998},"last":1.006}]},{"osd":5,"last update":"Wed Mar 24 21:58:04 2021","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":1.014,"15min":1.0269999999999999},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.163,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":1.4419999999999999},{"interface":"front","average":{"1min":1.048,"5min":1.125,"15min":1.147},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.4390000000000001,"5min":2.5859999999999999,"15min":3.1509999999999998},"last":1.552}]},{"osd":6,"last update":"Wed Mar 24 21:58:22 2021","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.1279999999999999,"15min":1.2170000000000001},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":2.4529999999999998,"5min":2.4529999999999998,"15min":2.4529999999999998},"last":0.97199999999999998},{"interface":"front","average":{"1min":1.161,"5min":1.115,"15min":1.1479999999999999},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":2.6379999999999999,"5min":2.6379999999999999,"15min":2.6379999999999999},"last":0.79800000000000004}]},{"osd":7,"last update":"Wed Mar 24 21:57:55 2021","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.1899999999999999,"15min":1.218},"min":{"1min":0.88300000000000001,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.4079999999999999,"5min":4.4880000000000004,"15min":4.4880000000000004},"last":0.93200000000000005},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.2130000000000001,"15min":1.238},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.70199999999999996},"max":{"1min":1.504,"5min":3.5419999999999998,"15min":3.5419999999999998},"last":0.85299999999999998}]}]},{"osd":3,"up_from":26,"seq":111669149805,"num_pgs":90,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":132872,"kb_used_data":112384,"kb_used_omap":0,"kb_used_meta":20480,"kb_avail":93612280,"statfs":{"total":95995035648,"available":95858974720,"internally_reserved":0,"allocated":115081216,"data_stored":148216210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20971520},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:58:29 2021","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.871,"15min":0.71299999999999997},"min":{"1min":0.38800000000000001,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.93799999999999994,"5min":23.846,"15min":23.846},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.92500000000000004,"15min":0.79100000000000004},"min":{"1min":0.38100000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.345,"5min":23.734999999999999,"15min":23.734999999999999},"last":0.54700000000000004}]},{"osd":1,"last update":"Wed Mar 24 21:58:29 2021","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.85299999999999998,"15min":0.77700000000000002},"min":{"1min":0.38900000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.88700000000000001,"5min":14.324,"15min":14.324},"last":0.629},{"interface":"front","average":{"1min":0.754,"5min":0.85499999999999998,"15min":0.78800000000000003},"min":{"1min":0.40100000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.109,"5min":12.631,"15min":12.631},"last":0.77600000000000002}]},{"osd":2,"last update":"Wed Mar 24 21:58:29 2021","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.90300000000000002,"15min":0.82699999999999996},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.446,"5min":12.510999999999999,"15min":12.510999999999999},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.92900000000000005,"15min":0.82099999999999995},"min":{"1min":0.46500000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.099,"5min":11.317,"15min":11.317},"last":0.69199999999999995}]},{"osd":4,"last update":"Wed Mar 24 21:58:50 2021","interfaces":[{"interface":"back","average":{"1min":1.137,"5min":1.05,"15min":0.98599999999999999},"min":{"1min":0.70299999999999996,"5min":0.443,"15min":0.443},"max":{"1min":2.601,"5min":2.601,"15min":2.601},"last":1.1950000000000001},{"interface":"front","average":{"1min":0.98399999999999999,"5min":1.1439999999999999,"15min":1.014},"min":{"1min":0.66300000000000003,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.6299999999999999,"5min":14.291,"15min":14.291},"last":0.84799999999999998}]},{"osd":5,"last update":"Wed Mar 24 21:58:02 2021","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.1859999999999999,"15min":1.1040000000000001},"min":{"1min":0.73799999999999999,"5min":0.62,"15min":0.54100000000000004},"max":{"1min":1.5840000000000001,"5min":12.723000000000001,"15min":12.723000000000001},"last":1.0449999999999999},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.099,"15min":1.089},"min":{"1min":0.63900000000000001,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.452,"5min":2.375,"15min":2.375},"last":1.4850000000000001}]},{"osd":6,"last update":"Wed Mar 24 21:58:29 2021","interfaces":[{"interface":"back","average":{"1min":1,"5min":1.2170000000000001,"15min":1.155},"min":{"1min":0.68999999999999995,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.4590000000000001,"5min":14.255000000000001,"15min":14.255000000000001},"last":0.92500000000000004},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.2450000000000001,"15min":1.1839999999999999},"min":{"1min":0.64800000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.7,"5min":12.666,"15min":12.666},"last":1.0229999999999999}]},{"osd":7,"last update":"Wed Mar 24 21:58:55 2021","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.204,"15min":1.1240000000000001},"min":{"1min":0.66700000000000004,"5min":0.61899999999999999,"15min":0.61499999999999999},"max":{"1min":2.633,"5min":2.633,"15min":2.633},"last":1.252},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.2949999999999999,"15min":1.2070000000000001},"min":{"1min":0.73799999999999999,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.7769999999999999,"5min":14.727,"15min":14.727},"last":1.006}]}]},{"osd":4,"up_from":32,"seq":137438953575,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":68792,"kb_used_data":51760,"kb_used_omap":0,"kb_used_meta":17024,"kb_avail":93676360,"statfs":{"total":95995035648,"available":95924592640,"internally_reserved":0,"allocated":53002240,"data_stored":73308245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":17432576},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:58:46 2021","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":1.2769999999999999,"15min":0.98799999999999999},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.113,"5min":41.472000000000001,"15min":41.472000000000001},"last":0.98799999999999999},{"interface":"front","average":{"1min":0.84399999999999997,"5min":1.4490000000000001,"15min":1.0629999999999999},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.3700000000000001,"5min":44.185000000000002,"15min":44.185000000000002},"last":0.96899999999999997}]},{"osd":1,"last update":"Wed Mar 24 21:58:46 2021","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.91800000000000004,"15min":0.91800000000000004},"min":{"1min":0.5,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.194,"5min":1.3120000000000001,"15min":1.4570000000000001},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.89000000000000001,"15min":0.86499999999999999},"min":{"1min":0.55400000000000005,"5min":0.436,"15min":0.436},"max":{"1min":1.1699999999999999,"5min":1.873,"15min":1.873},"last":1.2230000000000001}]},{"osd":2,"last update":"Wed Mar 24 21:58:46 2021","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.95099999999999996,"15min":0.91700000000000004},"min":{"1min":0.52800000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.232,"5min":4.4379999999999997,"15min":4.4379999999999997},"last":1.0489999999999999},{"interface":"front","average":{"1min":0.86599999999999999,"5min":1.0049999999999999,"15min":0.94599999999999995},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.21,"5min":2.738,"15min":2.738},"last":1.0189999999999999}]},{"osd":3,"last update":"Wed Mar 24 21:58:46 2021","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":1.0289999999999999,"15min":0.96399999999999997},"min":{"1min":0.45200000000000001,"5min":0.439,"15min":0.439},"max":{"1min":1.96,"5min":3.2000000000000002,"15min":3.2000000000000002},"last":0.92600000000000005},{"interface":"front","average":{"1min":1.085,"5min":1.0089999999999999,"15min":0.97399999999999998},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":4.1189999999999998,"5min":4.1189999999999998,"15min":4.1189999999999998},"last":1.274}]},{"osd":5,"last update":"Wed Mar 24 21:58:08 2021","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":0.94099999999999995,"15min":0.95199999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.419,"5min":1.6579999999999999,"15min":1.9199999999999999},"last":0.98999999999999999},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.0760000000000001,"15min":1.03},"min":{"1min":0.56000000000000005,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.3759999999999999,"5min":1.607,"15min":2.0760000000000001},"last":1.327}]},{"osd":6,"last update":"Wed Mar 24 21:58:28 2021","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":1.069,"15min":1.0649999999999999},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.3220000000000001,"5min":1.6859999999999999,"15min":1.6859999999999999},"last":1.125},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.101,"15min":1.0740000000000001},"min":{"1min":0.41399999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.284,"5min":1.631,"15min":1.631},"last":0.82899999999999996}]},{"osd":7,"last update":"Wed Mar 24 21:58:49 2021","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":1.1299999999999999,"15min":1.165},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.3700000000000001,"5min":1.6830000000000001,"15min":2.1099999999999999},"last":1.0920000000000001},{"interface":"front","average":{"1min":0.999,"5min":1.161,"15min":1.1859999999999999},"min":{"1min":0.57599999999999996,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.3779999999999999,"5min":1.635,"15min":2.0009999999999999},"last":0.80000000000000004}]}]},{"osd":5,"up_from":39,"seq":167503724638,"num_pgs":90,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":97752,"kb_used_data":65616,"kb_used_omap":0,"kb_used_meta":32128,"kb_avail":93647400,"statfs":{"total":95995035648,"available":95894937600,"internally_reserved":0,"allocated":67190784,"data_stored":71282771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32899072},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:58:08 2021","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.80700000000000005,"15min":0.86199999999999999},"min":{"1min":0.372,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.0289999999999999,"5min":1.762,"15min":2.29},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.87,"15min":0.89000000000000001},"min":{"1min":0.45600000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.5800000000000001,"5min":1.5800000000000001,"15min":3.1779999999999999},"last":0.79800000000000004}]},{"osd":1,"last update":"Wed Mar 24 21:58:08 2021","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.86899999999999999,"15min":0.90800000000000003},"min":{"1min":0.45300000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.264,"5min":1.6479999999999999,"15min":2.8690000000000002},"last":0.89700000000000002},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.89300000000000002,"15min":0.92000000000000004},"min":{"1min":0.41499999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.1930000000000001,"5min":2.0030000000000001,"15min":2.0030000000000001},"last":1.081}]},{"osd":2,"last update":"Wed Mar 24 21:58:08 2021","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.95899999999999996,"15min":0.92000000000000004},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.272,"5min":6.5199999999999996,"15min":6.5199999999999996},"last":0.93799999999999994},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.93400000000000005,"15min":0.92300000000000004},"min":{"1min":0.65500000000000003,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.613,"5min":5.5010000000000003,"15min":5.8479999999999999},"last":1.1200000000000001}]},{"osd":3,"last update":"Wed Mar 24 21:58:08 2021","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":0.95599999999999996,"15min":0.94399999999999995},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.744,"5min":2.2000000000000002,"15min":3.1259999999999999},"last":1.2430000000000001},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.97199999999999998,"15min":0.95699999999999996},"min":{"1min":0.61299999999999999,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.175,"5min":1.843,"15min":2.6850000000000001},"last":0.96299999999999997}]},{"osd":4,"last update":"Wed Mar 24 21:58:08 2021","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":1.02,"15min":0.89100000000000001},"min":{"1min":0.56399999999999995,"5min":0.376,"15min":0.376},"max":{"1min":1.6459999999999999,"5min":2.2959999999999998,"15min":2.2959999999999998},"last":1.335},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.0880000000000001,"15min":0.96099999999999997},"min":{"1min":0.74199999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.4910000000000001,"5min":2.629,"15min":2.629},"last":1.1559999999999999}]},{"osd":6,"last update":"Wed Mar 24 21:58:27 2021","interfaces":[{"interface":"back","average":{"1min":1.115,"5min":1.069,"15min":0.93999999999999995},"min":{"1min":0.63600000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.821,"5min":2.3879999999999999,"15min":2.3879999999999999},"last":1.3999999999999999},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.103,"15min":1.016},"min":{"1min":0.70899999999999996,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.645,"5min":2.4660000000000002,"15min":2.4660000000000002},"last":1.1799999999999999}]},{"osd":7,"last update":"Wed Mar 24 21:57:55 2021","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.1519999999999999,"15min":1.0800000000000001},"min":{"1min":0.61299999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.698,"5min":2.548,"15min":2.548},"last":1.218},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.1719999999999999,"15min":1.04},"min":{"1min":0.71499999999999997,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.389,"5min":2.9889999999999999,"15min":2.9889999999999999},"last":1.5049999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":14134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":7234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":13448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":352256,"data_stored":12784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":360448,"data_stored":13024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":9948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":94273536,"data_stored":117563509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7032832,"data_stored":6977532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":128614400,"data_stored":168427591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":113623040,"data_stored":147583639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":51560448,"data_stored":72676277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":63827968,"data_stored":69565904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":139202560,"data_stored":166220587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-03-24T21:59:01.635 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2021-03-24T21:59:01.636 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd pool set .rgw.root pg_num 42 2021-03-24T21:59:01.641 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:01.641 INFO:tasks.rados.rados.0.smithi125.stdout:1340: finishing write tid 2 to smithi12556470-6 2021-03-24T21:59:01.642 INFO:tasks.rados.rados.0.smithi125.stdout:1340: finishing write tid 3 to smithi12556470-6 2021-03-24T21:59:01.642 INFO:tasks.rados.rados.0.smithi125.stdout:1340: finishing write tid 4 to smithi12556470-6 2021-03-24T21:59:01.642 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 6 v 353 (ObjNum 484 snap 138 seq_num 484) dirty exists 2021-03-24T21:59:01.642 INFO:tasks.rados.rados.0.smithi125.stdout:1340: left oid 6 (ObjNum 484 snap 138 seq_num 484) 2021-03-24T21:59:01.643 INFO:tasks.rados.rados.0.smithi125.stdout:1339: finishing copy_from to smithi12556470-16 2021-03-24T21:59:01.643 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 16 v 380 (ObjNum 393 snap 100 seq_num 393) dirty exists 2021-03-24T21:59:01.643 INFO:tasks.rados.rados.0.smithi125.stdout:1339: finishing copy_from racing read to smithi12556470-16 2021-03-24T21:59:01.643 INFO:tasks.rados.rados.0.smithi125.stdout:1338: finishing copy_from to smithi12556470-40 2021-03-24T21:59:01.644 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 40 v 348 (ObjNum 483 snap 138 seq_num 483) dirty exists 2021-03-24T21:59:01.644 INFO:tasks.rados.rados.0.smithi125.stdout:1338: finishing copy_from racing read to smithi12556470-40 2021-03-24T21:59:01.644 INFO:tasks.rados.rados.0.smithi125.stdout:1344: expect (ObjNum 401 snap 104 seq_num 401) 2021-03-24T21:59:01.705 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:01.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:01.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:59:01.803 INFO:tasks.rados.rados.0.smithi125.stdout:1345: finishing copy_from to smithi12556470-7 2021-03-24T21:59:01.804 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 7 v 334 (ObjNum 471 snap 133 seq_num 471) dirty exists 2021-03-24T21:59:01.804 INFO:tasks.rados.rados.0.smithi125.stdout:1345: finishing copy_from racing read to smithi12556470-7 2021-03-24T21:59:01.804 INFO:tasks.rados.rados.0.smithi125.stdout:1343: finishing copy_from to smithi12556470-26 2021-03-24T21:59:01.804 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 26 v 432 (ObjNum 462 snap 129 seq_num 462) dirty exists 2021-03-24T21:59:01.805 INFO:tasks.rados.rados.0.smithi125.stdout:1343: finishing copy_from racing read to smithi12556470-26 2021-03-24T21:59:01.811 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:01.870 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:01.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:01.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:59:01.975 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:02.028 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:02.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:02.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:59:02.135 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:02.208 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:02.209 INFO:tasks.rados.rados.0.smithi125.stdout:1335: done (10 left) 2021-03-24T21:59:02.209 INFO:tasks.rados.rados.0.smithi125.stdout:1336: done (9 left) 2021-03-24T21:59:02.210 INFO:tasks.rados.rados.0.smithi125.stdout:1337: done (8 left) 2021-03-24T21:59:02.210 INFO:tasks.rados.rados.0.smithi125.stdout:1338: done (7 left) 2021-03-24T21:59:02.210 INFO:tasks.rados.rados.0.smithi125.stdout:1339: done (6 left) 2021-03-24T21:59:02.210 INFO:tasks.rados.rados.0.smithi125.stdout:1340: done (5 left) 2021-03-24T21:59:02.211 INFO:tasks.rados.rados.0.smithi125.stdout:1341: done (4 left) 2021-03-24T21:59:02.211 INFO:tasks.rados.rados.0.smithi125.stdout:1343: done (3 left) 2021-03-24T21:59:02.211 INFO:tasks.rados.rados.0.smithi125.stdout:1344: done (2 left) 2021-03-24T21:59:02.211 INFO:tasks.rados.rados.0.smithi125.stdout:1345: done (1 left) 2021-03-24T21:59:02.212 INFO:tasks.rados.rados.0.smithi125.stdout:1346: done (0 left) 2021-03-24T21:59:02.212 INFO:tasks.rados.rados.0.smithi125.stdout:1347: delete oid 13 current snap is 139 2021-03-24T21:59:02.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:02.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:59:02.221 INFO:tasks.rados.rados.0.smithi125.stdout:1347: done (0 left) 2021-03-24T21:59:02.222 INFO:tasks.rados.rados.0.smithi125.stdout:1348: snap_remove snap 132 2021-03-24T21:59:02.317 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:02.357 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:02.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:02.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:59:02.462 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:02.472 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:02 smithi142 conmon[20125]: cluster 2021-03-24T21:59:00.814732+0000 mgr.y 2021-03-24T21:59:02.472 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:02 smithi142 conmon[20125]: (mgr.14144) 455 : cluster [DBG] pgmap v718: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 267 MiB data, 708 MiB used, 536 GiB / 536 GiB avail; 243/1314 objects degraded (18.493%) 2021-03-24T21:59:02.473 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:02 smithi142 conmon[20125]: cluster 2021-03-24T21:59:01.806754+0000 osd.2 (osd.2) 70 : cluster [DBG] 2021-03-24T21:59:02.473 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:02 smithi142 conmon[20125]: 2.1 scrub ok 2021-03-24T21:59:02.473 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:02 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:59:02.473 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:02 smithi142 conmon[20125]: 21:59:02.206825+0000 mon.a (mon.0) 1140 : cluster [DBG] osdmap e372: 8 total, 7 up, 6 in 2021-03-24T21:59:02.474 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:02 smithi142 conmon[20125]: 2021-03-24T21:59:02.506 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:02.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:02.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:02.610 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:02.621 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:59:02.622 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[19936]: 59:00.814732+0000 mgr.y (mgr.14144) 455 : cluster [DBG] pgmap v718: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 267 MiB data, 708 MiB used, 536 GiB / 536 GiB avail; 243/1314 objects degraded (18.493%) 2021-03-24T21:59:02.622 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:59:02.622 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[19936]: 21:59:01.806754+0000 2021-03-24T21:59:02.623 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[19936]: osd.2 (osd.2) 70 2021-03-24T21:59:02.623 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[19936]: : cluster [DBG] 2.1 scrub ok 2021-03-24T21:59:02.623 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[19936]: cluster 2021-03-24T21:59:02.623 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[19936]: 2021-03-24T21:59: 2021-03-24T21:59:02.623 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[19936]: 02.206825+0000 mon.a (mon 2021-03-24T21:59:02.624 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[19936]: .0) 1140 : cluster [DBG] 2021-03-24T21:59:02.624 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[19936]: osdmap e372: 8 total, 7 up, 6 in 2021-03-24T21:59:02.625 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: cluster 2021-03-24T21:59:02.625 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: 2021-03-24T21:59:02.625 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: 2021 2021-03-24T21:59:02.625 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: - 2021-03-24T21:59:02.625 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: 03 2021-03-24T21:59:02.626 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: - 2021-03-24T21:59:02.626 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: 24 2021-03-24T21:59:02.626 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: T 2021-03-24T21:59:02.626 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: 21 2021-03-24T21:59:02.627 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: : 2021-03-24T21:59:02.627 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: 59 2021-03-24T21:59:02.627 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: : 2021-03-24T21:59:02.627 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: 00 2021-03-24T21:59:02.627 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: . 2021-03-24T21:59:02.628 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: 814732 2021-03-24T21:59:02.628 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: +0000 2021-03-24T21:59:02.628 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: 2021-03-24T21:59:02.628 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: mgr.y 2021-03-24T21:59:02.629 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: ( 2021-03-24T21:59:02.629 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: mgr 2021-03-24T21:59:02.629 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: . 2021-03-24T21:59:02.629 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: 14144 2021-03-24T21:59:02.629 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: ) 2021-03-24T21:59:02.630 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: 455 2021-03-24T21:59:02.630 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: : 2021-03-24T21:59:02.631 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: cluster 2021-03-24T21:59:02.631 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: 2021-03-24T21:59:02.631 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: [DBG] 2021-03-24T21:59:02.632 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: 2021-03-24T21:59:02.632 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: pgmap v718: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 267 MiB data, 708 MiB used, 536 GiB / 536 GiB avail; 243/1314 objects degraded (18.493%) 2021-03-24T21:59:02.632 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: 2021-03-24T21:59:02.633 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: cluster 2021-03-24T21:59:02.633 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: 2021-03-24T21:59:02.634 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: 2021 2021-03-24T21:59:02.634 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: - 2021-03-24T21:59:02.635 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: 03 2021-03-24T21:59:02.635 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: - 2021-03-24T21:59:02.635 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: 24 2021-03-24T21:59:02.636 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: T 2021-03-24T21:59:02.636 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: 21 2021-03-24T21:59:02.637 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: : 2021-03-24T21:59:02.637 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: 59 2021-03-24T21:59:02.637 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: : 2021-03-24T21:59:02.638 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: 01 2021-03-24T21:59:02.638 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: . 2021-03-24T21:59:02.639 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: 806754 2021-03-24T21:59:02.639 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: +0000 2021-03-24T21:59:02.639 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: 2021-03-24T21:59:02.640 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: osd.2 2021-03-24T21:59:02.640 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: ( 2021-03-24T21:59:02.641 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: osd 2021-03-24T21:59:02.641 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: . 2021-03-24T21:59:02.642 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: 2 2021-03-24T21:59:02.642 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: ) 2021-03-24T21:59:02.642 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: 70 2021-03-24T21:59:02.643 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: : 2021-03-24T21:59:02.643 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: cluster 2021-03-24T21:59:02.644 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: 2021-03-24T21:59:02.644 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: [DBG] 2021-03-24T21:59:02.645 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: 2021-03-24T21:59:02.645 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: 2.1 scrub ok 2021-03-24T21:59:02.646 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: 2021-03-24T21:59:02.646 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: cluster 2021-03-24T21:59:02.646 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: 2021-03-24T21:59:02.647 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: 2021 2021-03-24T21:59:02.647 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: - 2021-03-24T21:59:02.648 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: 03 2021-03-24T21:59:02.648 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: - 2021-03-24T21:59:02.649 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: 24 2021-03-24T21:59:02.649 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: T 2021-03-24T21:59:02.650 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: 21 2021-03-24T21:59:02.650 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: : 2021-03-24T21:59:02.650 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: 59 2021-03-24T21:59:02.651 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: : 2021-03-24T21:59:02.651 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: 02 2021-03-24T21:59:02.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: . 2021-03-24T21:59:02.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: 206825 2021-03-24T21:59:02.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: +0000 2021-03-24T21:59:02.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: 2021-03-24T21:59:02.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: mon.a 2021-03-24T21:59:02.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: ( 2021-03-24T21:59:02.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: mon 2021-03-24T21:59:02.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: . 2021-03-24T21:59:02.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: 0 2021-03-24T21:59:02.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: ) 2021-03-24T21:59:02.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: 1140 2021-03-24T21:59:02.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: : 2021-03-24T21:59:02.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: cluster 2021-03-24T21:59:02.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: 2021-03-24T21:59:02.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: [DBG] 2021-03-24T21:59:02.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: 2021-03-24T21:59:02.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: osdmap e372: 8 total, 7 up, 6 in 2021-03-24T21:59:02.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:02 smithi125 conmon[24927]: 2021-03-24T21:59:02.665 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:02.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:02.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:59:02.772 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:02.841 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:02.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:02.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:02.950 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:02.995 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:02.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:02.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:59:03.102 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:03.150 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:03.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:03.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:59:03.211 INFO:tasks.rados.rados.0.smithi125.stdout:1348: done (0 left) 2021-03-24T21:59:03.211 INFO:tasks.rados.rados.0.smithi125.stdout:1349: delete oid 43 current snap is 139 2021-03-24T21:59:03.226 INFO:tasks.rados.rados.0.smithi125.stdout:1349: done (0 left) 2021-03-24T21:59:03.226 INFO:tasks.rados.rados.0.smithi125.stdout:1350: read oid 5 snap -1 2021-03-24T21:59:03.226 INFO:tasks.rados.rados.0.smithi125.stdout:1350: expect (ObjNum 408 snap 106 seq_num 408) 2021-03-24T21:59:03.227 INFO:tasks.rados.rados.0.smithi125.stdout:1351: snap_remove snap 130 2021-03-24T21:59:03.254 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:03.391 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:03.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:03.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:03.417 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.2 dump_ops_in_flight 2021-03-24T21:59:03.501 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:03.562 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:03.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:03.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:59:03.670 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:03.720 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:03.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:03.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:03.823 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:03.883 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:03.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:03.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:59:03.988 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:04.052 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:04.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:04.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:04.098 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:59:04.162 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:04.221 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:04.222 INFO:tasks.rados.rados.0.smithi125.stdout:1351: done (1 left) 2021-03-24T21:59:04.222 INFO:tasks.rados.rados.0.smithi125.stdout:1352: read oid 42 snap -1 2021-03-24T21:59:04.223 INFO:tasks.rados.rados.0.smithi125.stdout:1352: expect (ObjNum 444 snap 120 seq_num 444) 2021-03-24T21:59:04.223 INFO:tasks.rados.rados.0.smithi125.stdout:1353: write (excl) oid 6 current snap is 139 2021-03-24T21:59:04.223 INFO:tasks.rados.rados.0.smithi125.stdout:1353: seq_num 487 ranges {585009=492855,1564966=620043} 2021-03-24T21:59:04.231 INFO:tasks.rados.rados.0.smithi125.stdout:1353: writing smithi12556470-6 from 585009 to 1077864 tid 1 2021-03-24T21:59:04.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:04.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:59:04.239 INFO:tasks.rados.rados.0.smithi125.stdout:1353: writing smithi12556470-6 from 1564966 to 2185009 tid 2 2021-03-24T21:59:04.243 INFO:tasks.rados.rados.0.smithi125.stdout:1350: expect (ObjNum 408 snap 106 seq_num 408) 2021-03-24T21:59:04.339 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:04.397 INFO:tasks.rados.rados.0.smithi125.stdout:1350: done (2 left) 2021-03-24T21:59:04.397 INFO:tasks.rados.rados.0.smithi125.stdout:1354: snap_create 2021-03-24T21:59:04.398 INFO:tasks.rados.rados.0.smithi125.stdout:1352: expect (ObjNum 444 snap 120 seq_num 444) 2021-03-24T21:59:04.403 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:59:04.470 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:04.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:04.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:04.478 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[24927]: cluster 2021-03-24T21:59:02.668709+0000 osd.4 ( 2021-03-24T21:59:04.478 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[24927]: osd.4) 94 : cluster [DBG] 5.18 scrub ok 2021-03-24T21:59:04.478 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[24927]: cluster 2021-03-24T21:59:02 2021-03-24T21:59:04.478 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[24927]: .815683+0000 mgr.y (mgr.14144) 456 : cluster [DBG] 2021-03-24T21:59:04.479 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[24927]: pgmap v721: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 267 MiB data, 715 MiB used, 536 GiB / 536 GiB avail; 9.9 MiB/s rd, 1.3 MiB/s wr, 15 op/s; 244/1317 objects degraded (18.527%) 2021-03-24T21:59:04.479 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:59:04.479 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[24927]: -24T21:59:03.208933+0000 mon.a (mon.0) 1141 : cluster [DBG] 2021-03-24T21:59:04.479 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[24927]: osdmap e373: 8 total, 7 up, 6 in 2021-03-24T21:59:04.479 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:59:04.480 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[24927]: -24T21:59:03.758906+0000 osd.2 (osd.2) 71 : cluster [DBG] 2021-03-24T21:59:04.480 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[24927]: 7.20 scrub ok 2021-03-24T21:59:04.482 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: cluster 2021-03-24T21:59:04.482 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 2021-03-24T21:59:04.483 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 2021 2021-03-24T21:59:04.483 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: - 2021-03-24T21:59:04.483 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 03 2021-03-24T21:59:04.483 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: - 2021-03-24T21:59:04.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 24 2021-03-24T21:59:04.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: T 2021-03-24T21:59:04.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 21 2021-03-24T21:59:04.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: : 2021-03-24T21:59:04.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 59 2021-03-24T21:59:04.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: : 2021-03-24T21:59:04.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 02 2021-03-24T21:59:04.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: . 2021-03-24T21:59:04.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 668709 2021-03-24T21:59:04.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: +0000 2021-03-24T21:59:04.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 2021-03-24T21:59:04.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: osd.4 2021-03-24T21:59:04.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: ( 2021-03-24T21:59:04.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: osd 2021-03-24T21:59:04.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: . 2021-03-24T21:59:04.487 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 4 2021-03-24T21:59:04.487 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: ) 2021-03-24T21:59:04.487 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 94 2021-03-24T21:59:04.487 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: : 2021-03-24T21:59:04.487 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: cluster 2021-03-24T21:59:04.488 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 2021-03-24T21:59:04.488 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: [DBG] 2021-03-24T21:59:04.488 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 2021-03-24T21:59:04.488 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 5.18 scrub ok 2021-03-24T21:59:04.489 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 2021-03-24T21:59:04.489 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: cluster 2021-03-24T21:59:04.489 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 2021-03-24T21:59:04.489 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 2021 2021-03-24T21:59:04.489 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: - 2021-03-24T21:59:04.490 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 03 2021-03-24T21:59:04.490 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: - 2021-03-24T21:59:04.490 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 24 2021-03-24T21:59:04.490 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: T 2021-03-24T21:59:04.491 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 21 2021-03-24T21:59:04.491 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: : 2021-03-24T21:59:04.491 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 59 2021-03-24T21:59:04.491 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: : 2021-03-24T21:59:04.491 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 02 2021-03-24T21:59:04.492 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: . 2021-03-24T21:59:04.492 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 815683 2021-03-24T21:59:04.492 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: +0000 2021-03-24T21:59:04.492 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 2021-03-24T21:59:04.493 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: mgr.y 2021-03-24T21:59:04.493 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: ( 2021-03-24T21:59:04.493 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: mgr 2021-03-24T21:59:04.493 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: . 2021-03-24T21:59:04.493 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 14144 2021-03-24T21:59:04.494 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: ) 2021-03-24T21:59:04.494 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 456 2021-03-24T21:59:04.494 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: : 2021-03-24T21:59:04.494 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: cluster 2021-03-24T21:59:04.495 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 2021-03-24T21:59:04.495 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: [DBG] 2021-03-24T21:59:04.495 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 2021-03-24T21:59:04.495 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: pgmap v721: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 267 MiB data, 715 MiB used, 536 GiB / 536 GiB avail; 9.9 MiB/s rd, 1.3 MiB/s wr, 15 op/s; 244/1317 objects degraded (18.527%) 2021-03-24T21:59:04.495 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 2021-03-24T21:59:04.496 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: cluster 2021-03-24T21:59:04.496 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 2021-03-24T21:59:04.496 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 2021 2021-03-24T21:59:04.496 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: - 2021-03-24T21:59:04.497 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 03 2021-03-24T21:59:04.497 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: - 2021-03-24T21:59:04.497 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 24 2021-03-24T21:59:04.497 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: T 2021-03-24T21:59:04.497 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 21 2021-03-24T21:59:04.498 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: : 2021-03-24T21:59:04.498 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 59 2021-03-24T21:59:04.498 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: : 2021-03-24T21:59:04.498 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 03 2021-03-24T21:59:04.498 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: . 2021-03-24T21:59:04.499 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 208933 2021-03-24T21:59:04.499 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: +0000 2021-03-24T21:59:04.499 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 2021-03-24T21:59:04.499 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: mon.a 2021-03-24T21:59:04.500 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: ( 2021-03-24T21:59:04.500 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: mon 2021-03-24T21:59:04.500 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: . 2021-03-24T21:59:04.500 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 0 2021-03-24T21:59:04.500 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: ) 2021-03-24T21:59:04.501 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 1141 2021-03-24T21:59:04.501 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: : 2021-03-24T21:59:04.501 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: cluster 2021-03-24T21:59:04.502 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 2021-03-24T21:59:04.502 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: [DBG] 2021-03-24T21:59:04.502 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 2021-03-24T21:59:04.502 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: osdmap e373: 8 total, 7 up, 6 in 2021-03-24T21:59:04.502 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 2021-03-24T21:59:04.503 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: cluster 2021-03-24T21:59:04.503 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 2021-03-24T21:59:04.503 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 2021 2021-03-24T21:59:04.503 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: - 2021-03-24T21:59:04.503 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 03 2021-03-24T21:59:04.504 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: - 2021-03-24T21:59:04.504 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 24 2021-03-24T21:59:04.504 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: T 2021-03-24T21:59:04.504 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 21 2021-03-24T21:59:04.505 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: : 2021-03-24T21:59:04.505 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 59 2021-03-24T21:59:04.505 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: : 2021-03-24T21:59:04.505 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 03 2021-03-24T21:59:04.505 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: . 2021-03-24T21:59:04.506 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 758906 2021-03-24T21:59:04.506 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: +0000 2021-03-24T21:59:04.506 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 2021-03-24T21:59:04.506 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: osd.2 2021-03-24T21:59:04.506 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: ( 2021-03-24T21:59:04.507 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: osd 2021-03-24T21:59:04.507 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: . 2021-03-24T21:59:04.507 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 2 2021-03-24T21:59:04.507 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: ) 2021-03-24T21:59:04.507 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 71 2021-03-24T21:59:04.508 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: : 2021-03-24T21:59:04.508 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: cluster 2021-03-24T21:59:04.508 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 2021-03-24T21:59:04.508 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: [DBG] 2021-03-24T21:59:04.508 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 2021-03-24T21:59:04.509 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 7.20 scrub ok 2021-03-24T21:59:04.509 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:04 smithi125 conmon[19936]: 2021-03-24T21:59:04.579 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:04.592 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: cluster 2021-03-24T21:59:04.593 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 2021-03-24T21:59:04.593 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 2021 2021-03-24T21:59:04.593 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: - 2021-03-24T21:59:04.593 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 03 2021-03-24T21:59:04.593 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: - 2021-03-24T21:59:04.594 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 24 2021-03-24T21:59:04.594 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: T 2021-03-24T21:59:04.594 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 21 2021-03-24T21:59:04.594 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: : 2021-03-24T21:59:04.595 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 59 2021-03-24T21:59:04.595 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: : 2021-03-24T21:59:04.595 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 02 2021-03-24T21:59:04.595 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: . 2021-03-24T21:59:04.596 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 668709 2021-03-24T21:59:04.596 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: +0000 2021-03-24T21:59:04.596 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 2021-03-24T21:59:04.596 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: osd.4 2021-03-24T21:59:04.596 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: ( 2021-03-24T21:59:04.597 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: osd 2021-03-24T21:59:04.597 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: . 2021-03-24T21:59:04.597 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 4 2021-03-24T21:59:04.597 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: ) 2021-03-24T21:59:04.598 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 94 2021-03-24T21:59:04.598 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: : 2021-03-24T21:59:04.598 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: cluster 2021-03-24T21:59:04.598 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 2021-03-24T21:59:04.598 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: [DBG] 2021-03-24T21:59:04.599 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 2021-03-24T21:59:04.599 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 5.18 scrub ok 2021-03-24T21:59:04.599 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 2021-03-24T21:59:04.599 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: cluster 2021-03-24T21:59:04.600 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 2021-03-24T21:59:04.600 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 2021 2021-03-24T21:59:04.600 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: - 2021-03-24T21:59:04.600 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 03 2021-03-24T21:59:04.600 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: - 2021-03-24T21:59:04.601 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 24 2021-03-24T21:59:04.601 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: T 2021-03-24T21:59:04.601 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 21 2021-03-24T21:59:04.601 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: : 2021-03-24T21:59:04.602 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 59 2021-03-24T21:59:04.602 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: : 2021-03-24T21:59:04.602 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 02 2021-03-24T21:59:04.602 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: . 2021-03-24T21:59:04.602 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 815683 2021-03-24T21:59:04.603 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: +0000 2021-03-24T21:59:04.603 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 2021-03-24T21:59:04.603 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: mgr.y 2021-03-24T21:59:04.603 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: ( 2021-03-24T21:59:04.604 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: mgr 2021-03-24T21:59:04.604 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: . 2021-03-24T21:59:04.604 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 14144 2021-03-24T21:59:04.604 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: ) 2021-03-24T21:59:04.604 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 456 2021-03-24T21:59:04.605 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: : 2021-03-24T21:59:04.605 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: cluster 2021-03-24T21:59:04.605 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 2021-03-24T21:59:04.605 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: [DBG] 2021-03-24T21:59:04.606 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 2021-03-24T21:59:04.606 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: pgmap v721: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 267 MiB data, 715 MiB used, 536 GiB / 536 GiB avail; 9.9 MiB/s rd, 1.3 MiB/s wr, 15 op/s; 244/1317 objects degraded (18.527%) 2021-03-24T21:59:04.606 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 2021-03-24T21:59:04.606 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: cluster 2021-03-24T21:59:04.606 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 2021-03-24T21:59:04.607 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 2021 2021-03-24T21:59:04.607 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: - 2021-03-24T21:59:04.607 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 03 2021-03-24T21:59:04.607 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: - 2021-03-24T21:59:04.608 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 24 2021-03-24T21:59:04.608 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: T 2021-03-24T21:59:04.608 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 21 2021-03-24T21:59:04.608 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: : 2021-03-24T21:59:04.608 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 59 2021-03-24T21:59:04.609 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: : 2021-03-24T21:59:04.609 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 03 2021-03-24T21:59:04.609 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: . 2021-03-24T21:59:04.609 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 208933 2021-03-24T21:59:04.609 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: +0000 2021-03-24T21:59:04.610 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 2021-03-24T21:59:04.610 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: mon.a 2021-03-24T21:59:04.610 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: ( 2021-03-24T21:59:04.610 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: mon 2021-03-24T21:59:04.611 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: . 2021-03-24T21:59:04.611 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 0 2021-03-24T21:59:04.611 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: ) 2021-03-24T21:59:04.611 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 1141 2021-03-24T21:59:04.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: : 2021-03-24T21:59:04.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: cluster 2021-03-24T21:59:04.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 2021-03-24T21:59:04.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: [DBG] 2021-03-24T21:59:04.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 2021-03-24T21:59:04.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: osdmap e373: 8 total, 7 up, 6 in 2021-03-24T21:59:04.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 2021-03-24T21:59:04.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: cluster 2021-03-24T21:59:04.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 2021-03-24T21:59:04.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 2021 2021-03-24T21:59:04.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: - 2021-03-24T21:59:04.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 03 2021-03-24T21:59:04.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: - 2021-03-24T21:59:04.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 24 2021-03-24T21:59:04.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: T 2021-03-24T21:59:04.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 21 2021-03-24T21:59:04.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: : 2021-03-24T21:59:04.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 59 2021-03-24T21:59:04.615 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: : 2021-03-24T21:59:04.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 03 2021-03-24T21:59:04.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: . 2021-03-24T21:59:04.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 758906 2021-03-24T21:59:04.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: +0000 2021-03-24T21:59:04.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 2021-03-24T21:59:04.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: osd.2 2021-03-24T21:59:04.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: ( 2021-03-24T21:59:04.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: osd 2021-03-24T21:59:04.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: . 2021-03-24T21:59:04.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 2 2021-03-24T21:59:04.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: ) 2021-03-24T21:59:04.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 71 2021-03-24T21:59:04.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: : 2021-03-24T21:59:04.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: cluster 2021-03-24T21:59:04.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 2021-03-24T21:59:04.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: [DBG] 2021-03-24T21:59:04.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 2021-03-24T21:59:04.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 7.20 scrub ok 2021-03-24T21:59:04.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:04 smithi142 conmon[20125]: 2021-03-24T21:59:04.620 INFO:tasks.rados.rados.0.smithi125.stdout:1353: finishing write tid 1 to smithi12556470-6 2021-03-24T21:59:04.620 INFO:tasks.rados.rados.0.smithi125.stdout:1353: finishing write tid 2 to smithi12556470-6 2021-03-24T21:59:04.621 INFO:tasks.rados.rados.0.smithi125.stdout:1353: finishing write tid 3 to smithi12556470-6 2021-03-24T21:59:04.621 INFO:tasks.rados.rados.0.smithi125.stdout:1353: finishing write tid 4 to smithi12556470-6 2021-03-24T21:59:04.621 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 6 v 356 (ObjNum 487 snap 139 seq_num 487) dirty exists 2021-03-24T21:59:04.621 INFO:tasks.rados.rados.0.smithi125.stdout:1353: left oid 6 (ObjNum 487 snap 139 seq_num 487) 2021-03-24T21:59:04.637 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:04.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:04.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:04.742 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:04.791 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:04.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:04.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:59:04.860 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T21:59:04.880 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T21:59:04.896 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:04.901 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T21:59:04.923 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T21:59:04.946 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:04.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:04.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:04.955 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T21:59:04.984 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T21:59:05.008 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T21:59:05.009 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-24T21:59:05.009 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-24T21:59:05.009 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-24T21:59:05.010 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-24T21:59:05.010 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-24T21:59:05.010 INFO:teuthology.orchestra.run.smithi125.stderr:Error ENXIO: problem getting command descriptions from osd.5 2021-03-24T21:59:05.010 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-24T21:59:05.011 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-24T21:59:05.050 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:05.111 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:05.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:05.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:59:05.218 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:05.237 INFO:tasks.rados.rados.0.smithi125.stdout:1352: done (2 left) 2021-03-24T21:59:05.237 INFO:tasks.rados.rados.0.smithi125.stdout:1353: done (1 left) 2021-03-24T21:59:05.238 INFO:tasks.rados.rados.0.smithi125.stdout:1354: done (0 left) 2021-03-24T21:59:05.238 INFO:tasks.rados.rados.0.smithi125.stdout:1355: read oid 9 snap -1 2021-03-24T21:59:05.238 INFO:tasks.rados.rados.0.smithi125.stdout:1355: expect deleted 2021-03-24T21:59:05.238 INFO:tasks.rados.rados.0.smithi125.stdout:1356: write oid 6 current snap is 140 2021-03-24T21:59:05.239 INFO:tasks.rados.rados.0.smithi125.stdout:1356: seq_num 488 ranges {601816=568104,1623448=451823} 2021-03-24T21:59:05.240 INFO:tasks.rados.rados.0.smithi125.stdout:1356: writing smithi12556470-6 from 601816 to 1169920 tid 1 2021-03-24T21:59:05.247 INFO:tasks.rados.rados.0.smithi125.stdout:1356: writing smithi12556470-6 from 1623448 to 2075271 tid 2 2021-03-24T21:59:05.247 INFO:tasks.rados.rados.0.smithi125.stdout:1356: finishing write tid 1 to smithi12556470-6 2021-03-24T21:59:05.247 INFO:tasks.rados.rados.0.smithi125.stdout:1355: done (1 left) 2021-03-24T21:59:05.248 INFO:tasks.rados.rados.0.smithi125.stdout:1357: rollback oid 9 current snap is 140 2021-03-24T21:59:05.248 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 9 to 108 2021-03-24T21:59:05.248 INFO:tasks.rados.rados.0.smithi125.stdout:1358: write (excl) oid 38 current snap is 140 2021-03-24T21:59:05.248 INFO:tasks.rados.rados.0.smithi125.stdout:1358: seq_num 489 ranges {618623=643353,1681930=536693} 2021-03-24T21:59:05.255 INFO:tasks.rados.rados.0.smithi125.stdout:1358: writing smithi12556470-38 from 618623 to 1261976 tid 1 2021-03-24T21:59:05.262 INFO:tasks.rados.rados.0.smithi125.stdout:1358: writing smithi12556470-38 from 1681930 to 2218623 tid 2 2021-03-24T21:59:05.263 INFO:tasks.rados.rados.0.smithi125.stdout:1357: finishing rollback tid 1 to smithi12556470-9 2021-03-24T21:59:05.264 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 9 v 0 (ObjNum 1116535408 snap 21916 seq_num 1116418064) dirty dne 2021-03-24T21:59:05.264 INFO:tasks.rados.rados.0.smithi125.stdout:1356: finishing write tid 2 to smithi12556470-6 2021-03-24T21:59:05.265 INFO:tasks.rados.rados.0.smithi125.stdout:1356: finishing write tid 3 to smithi12556470-6 2021-03-24T21:59:05.265 INFO:tasks.rados.rados.0.smithi125.stdout:1356: finishing write tid 4 to smithi12556470-6 2021-03-24T21:59:05.266 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 6 v 359 (ObjNum 488 snap 140 seq_num 488) dirty exists 2021-03-24T21:59:05.266 INFO:tasks.rados.rados.0.smithi125.stdout:1356: left oid 6 (ObjNum 488 snap 140 seq_num 488) 2021-03-24T21:59:05.267 INFO:tasks.rados.rados.0.smithi125.stdout:1356: done (2 left) 2021-03-24T21:59:05.267 INFO:tasks.rados.rados.0.smithi125.stdout:1357: done (1 left) 2021-03-24T21:59:05.267 INFO:tasks.rados.rados.0.smithi125.stdout:1359: snap_create 2021-03-24T21:59:05.268 INFO:tasks.rados.rados.0.smithi125.stdout:1358: finishing write tid 1 to smithi12556470-38 2021-03-24T21:59:05.277 INFO:tasks.rados.rados.0.smithi125.stdout:1358: finishing write tid 2 to smithi12556470-38 2021-03-24T21:59:05.277 INFO:tasks.rados.rados.0.smithi125.stdout:1358: finishing write tid 3 to smithi12556470-38 2021-03-24T21:59:05.281 INFO:tasks.rados.rados.0.smithi125.stdout:1358: finishing write tid 4 to smithi12556470-38 2021-03-24T21:59:05.281 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 38 v 54 (ObjNum 489 snap 140 seq_num 489) dirty exists 2021-03-24T21:59:05.281 INFO:tasks.rados.rados.0.smithi125.stdout:1358: left oid 38 (ObjNum 489 snap 140 seq_num 489) 2021-03-24T21:59:05.293 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:05 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:59:05.293 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:05 smithi125 conmon[24927]: :59:03.626245+0000 2021-03-24T21:59:05.294 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:05 smithi125 conmon[24927]: osd.4 (osd.4) 95 : cluster 2021-03-24T21:59:05.294 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:05 smithi125 conmon[24927]: [DBG] 6.2 scrub ok 2021-03-24T21:59:05.294 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:05 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:59:05.294 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:05 smithi125 conmon[24927]: 03-24T21:59:04. 2021-03-24T21:59:05.294 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:05 smithi125 conmon[24927]: 077560+0000 osd.0 (osd. 2021-03-24T21:59:05.295 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:05 smithi125 conmon[24927]: 0) 85 : cluster [DBG] 6.0 scrub ok 2021-03-24T21:59:05.295 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:05 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:59:05.295 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:05 smithi125 conmon[24927]: 21:59:04.211443+0000 mon.a (mon.0) 1142 : cluster [DBG] osdmap e374: 8 total, 7 up, 6 in 2021-03-24T21:59:05.295 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:05 smithi125 conmon[24927]: audit 2021 2021-03-24T21:59:05.296 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:05 smithi125 conmon[24927]: -03-24T21:59:04.783926+0000 mon.a (mon.0) 1143 : 2021-03-24T21:59:05.296 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:05 smithi125 conmon[24927]: audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T21:59:05.296 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:05 smithi125 conmon[24927]: audit 2021- 2021-03-24T21:59:05.296 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:05 smithi125 conmon[24927]: 03-24T21:59:04.784134+0000 mon.a (mon.0) 1144 : audit 2021-03-24T21:59:05.296 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:05 smithi125 conmon[24927]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T21:59:05.297 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:05 smithi125 conmon[24927]: audit 2021-03-24T21:59:05.297 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:05 smithi125 conmon[24927]: 2021-03-24T21:59:04.836030 2021-03-24T21:59:05.297 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:05 smithi125 conmon[24927]: +0000 mon.a (mon.0) 1145 : 2021-03-24T21:59:05.297 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:05 smithi125 conmon[24927]: audit [DBG] from='client.? v1:172.21.15.125:0/1554155333' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:59:05.297 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:05 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:59:05.298 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:05 smithi125 conmon[24927]: -24T21:59:05.194082+0000 mon.a ( 2021-03-24T21:59:05.298 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:05 smithi125 conmon[24927]: mon.0) 1146 : cluster [WRN] Health check update: Degraded data redundancy: 244/1317 objects degraded (18.527%), 40 pgs degraded (PG_DEGRADED) 2021-03-24T21:59:05.298 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:05 smithi125 conmon[19936]: cluster 2021-03-24T21:59: 2021-03-24T21:59:05.299 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:05 smithi125 conmon[19936]: 03.626245+0000 osd.4 (osd.4) 95 : cluster [DBG] 6.2 scrub ok 2021-03-24T21:59:05.299 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:05 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:59:05.299 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:05 smithi125 conmon[19936]: :59:04.077560+0000 osd.0 (osd.0) 85 : cluster [DBG] 2021-03-24T21:59:05.299 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:05 smithi125 conmon[19936]: 6.0 scrub ok 2021-03-24T21:59:05.300 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:05 smithi125 conmon[19936]: cluster 2021-03-24T21:59: 2021-03-24T21:59:05.300 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:05 smithi125 conmon[19936]: 04.211443+0000 mon.a (mon.0) 1142 2021-03-24T21:59:05.300 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:05 smithi125 conmon[19936]: : cluster [DBG] osdmap e374: 8 total, 7 up, 6 in 2021-03-24T21:59:05.300 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:05 smithi125 conmon[19936]: audit 2021-03-24 2021-03-24T21:59:05.300 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:05 smithi125 conmon[19936]: T21:59:04.783926+0000 mon.a (mon 2021-03-24T21:59:05.301 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:05 smithi125 conmon[19936]: .0) 1143 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T21:59:05.301 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:05 smithi125 conmon[19936]: audit 2021-03- 2021-03-24T21:59:05.301 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:05 smithi125 conmon[19936]: 24T21:59:04.784134+0000 mon.a (mon.0 2021-03-24T21:59:05.301 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:05 smithi125 conmon[19936]: ) 1144 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T21:59:05.301 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:05 smithi125 conmon[19936]: audit 2021-03-24T21:59:04.836030+0000 2021-03-24T21:59:05.302 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:05 smithi125 conmon[19936]: mon.a (mon.0) 1145 : audit [DBG] from='client.? v1:172.21.15.125:0/1554155333' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:59:05.302 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:05 smithi125 conmon[19936]: cluster 2021-03-24T21:59: 2021-03-24T21:59:05.302 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:05 smithi125 conmon[19936]: 05.194082+0000 mon.a (mon.0) 1146 : cluster [WRN] 2021-03-24T21:59:05.302 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:05 smithi125 conmon[19936]: Health check update: Degraded data redundancy: 244/1317 objects degraded (18.527%), 40 pgs degraded (PG_DEGRADED) 2021-03-24T21:59:05.322 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:05.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:05.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:05.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:05 smithi142 conmon[20125]: cluster 2021-03-24T21:59:03.626245+0000 osd.4 (osd.4) 95 : cluster 2021-03-24T21:59:05.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:05 smithi142 conmon[20125]: [DBG] 6.2 scrub ok 2021-03-24T21:59:05.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:05 smithi142 conmon[20125]: cluster 2021-03-24T21:59:04.077560+0000 osd.0 (osd.0 2021-03-24T21:59:05.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:05 smithi142 conmon[20125]: ) 85 : cluster [DBG] 6.0 scrub ok 2021-03-24T21:59:05.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:05 smithi142 conmon[20125]: cluster 2021-03-24T21:59:05.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:05 smithi142 conmon[20125]: 2021-03-24T21:59:04.211443+0000 mon.a 2021-03-24T21:59:05.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:05 smithi142 conmon[20125]: (mon.0) 1142 : cluster [DBG] osdmap e374: 8 total, 7 up, 6 in 2021-03-24T21:59:05.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:05 smithi142 conmon[20125]: 2021-03-24T21:59:05.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:05 smithi142 conmon[20125]: audit 2021 2021-03-24T21:59:05.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:05 smithi142 conmon[20125]: -03-24T21:59:04.783926+0000 mon.a ( 2021-03-24T21:59:05.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:05 smithi142 conmon[20125]: mon.0) 1143 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T21:59:05.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:05 smithi142 conmon[20125]: audit 2021-03 2021-03-24T21:59:05.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:05 smithi142 conmon[20125]: -24T21:59:04.784134+0000 2021-03-24T21:59:05.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:05 smithi142 conmon[20125]: mon.a (mon.0) 1144 : audit [INF] 2021-03-24T21:59:05.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:05 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T21:59:05.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:05 smithi142 conmon[20125]: audit 2021-03-24T21:59:05.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:05 smithi142 conmon[20125]: 2021-03-24T21:59:04.836030+0000 mon.a 2021-03-24T21:59:05.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:05 smithi142 conmon[20125]: (mon.0) 1145 : audit [DBG] from='client.? v1:172.21.15.125:0/1554155333' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:59:05.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:05 smithi142 conmon[20125]: 2021-03-24T21:59:05.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:05 smithi142 conmon[20125]: cluster 2021-03-24T21:59:05.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:05 smithi142 conmon[20125]: 2021-03-24T21:59:05.194082+0000 2021-03-24T21:59:05.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:05 smithi142 conmon[20125]: mon.a (mon.0) 1146 : cluster [WRN] 2021-03-24T21:59:05.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:05 smithi142 conmon[20125]: Health check update: Degraded data redundancy: 244/1317 objects degraded (18.527%), 40 pgs degraded (PG_DEGRADED) 2021-03-24T21:59:05.431 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:05.486 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:05.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:05.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:05.593 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:05.641 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:05.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:05.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:59:05.744 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:05.796 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:05.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:05.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:05.900 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:05.952 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:05.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:05.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:59:06.059 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:06.114 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:06.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:06.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:59:06.221 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:06.244 INFO:teuthology.orchestra.run.smithi125.stderr:nodeep-scrub is unset 2021-03-24T21:59:06.246 INFO:tasks.rados.rados.0.smithi125.stdout:1358: done (1 left) 2021-03-24T21:59:06.246 INFO:tasks.rados.rados.0.smithi125.stdout:1359: done (0 left) 2021-03-24T21:59:06.246 INFO:tasks.rados.rados.0.smithi125.stdout:1360: delete oid 34 current snap is 141 2021-03-24T21:59:06.301 INFO:tasks.rados.rados.0.smithi125.stdout:1360: done (0 left) 2021-03-24T21:59:06.302 INFO:tasks.rados.rados.0.smithi125.stdout:1361: rollback oid 43 current snap is 141 2021-03-24T21:59:06.302 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 43 to 123 2021-03-24T21:59:06.302 INFO:tasks.rados.rados.0.smithi125.stdout:1362: delete oid 1 current snap is 141 2021-03-24T21:59:06.317 INFO:tasks.rados.rados.0.smithi125.stdout:1362: done (1 left) 2021-03-24T21:59:06.317 INFO:tasks.rados.rados.0.smithi125.stdout:1363: write (excl) oid 37 current snap is 141 2021-03-24T21:59:06.317 INFO:tasks.rados.rados.0.smithi125.stdout:1363: seq_num 492 ranges {669044=469100,1741023=459161} 2021-03-24T21:59:06.320 INFO:tasks.rados.rados.0.smithi125.stdout:1363: writing smithi12556470-37 from 669044 to 1138144 tid 1 2021-03-24T21:59:06.331 INFO:tasks.rados.rados.0.smithi125.stdout:1363: writing smithi12556470-37 from 1741023 to 2200184 tid 2 2021-03-24T21:59:06.332 INFO:tasks.rados.rados.0.smithi125.stdout:1364: copy_from oid 36 from oid 9 current snap is 141 2021-03-24T21:59:06.332 INFO:tasks.rados.rados.0.smithi125.stdout:1361: finishing rollback tid 1 to smithi12556470-43 2021-03-24T21:59:06.332 INFO:tasks.rados.rados.0.smithi125.stdout:1361: finishing rollback tid 2 to smithi12556470-43 2021-03-24T21:59:06.332 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 43 v 484 (ObjNum 399 snap 102 seq_num 399) dirty exists 2021-03-24T21:59:06.333 INFO:tasks.rados.rados.0.smithi125.stdout:1361: done (2 left) 2021-03-24T21:59:06.333 INFO:tasks.rados.rados.0.smithi125.stdout:1365: delete oid 5 current snap is 141 2021-03-24T21:59:06.333 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:06.334 INFO:tasks.rados.rados.0.smithi125.stdout:1363: finishing write tid 1 to smithi12556470-37 2021-03-24T21:59:06.337 INFO:tasks.rados.rados.0.smithi125.stdout:1365: done (2 left) 2021-03-24T21:59:06.337 INFO:tasks.rados.rados.0.smithi125.stdout:1366: snap_create 2021-03-24T21:59:06.339 INFO:tasks.rados.rados.0.smithi125.stdout:1363: finishing write tid 2 to smithi12556470-37 2021-03-24T21:59:06.339 INFO:tasks.rados.rados.0.smithi125.stdout:1363: finishing write tid 3 to smithi12556470-37 2021-03-24T21:59:06.340 INFO:tasks.rados.rados.0.smithi125.stdout:1363: finishing write tid 4 to smithi12556470-37 2021-03-24T21:59:06.340 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 37 v 340 (ObjNum 492 snap 141 seq_num 492) dirty exists 2021-03-24T21:59:06.340 INFO:tasks.rados.rados.0.smithi125.stdout:1363: left oid 37 (ObjNum 492 snap 141 seq_num 492) 2021-03-24T21:59:06.341 INFO:tasks.rados.rados.0.smithi125.stdout:1364: finishing copy_from to smithi12556470-36 2021-03-24T21:59:06.341 INFO:tasks.rados.rados.0.smithi125.stdout:1364: got expected ENOENT (src dne) 2021-03-24T21:59:06.341 INFO:tasks.rados.rados.0.smithi125.stdout:1364: finishing copy_from racing read to smithi12556470-36 2021-03-24T21:59:06.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:06.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:59:06.444 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:06.453 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: cluster 2021-03-24T21:59:06.454 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 2021-03-24T21:59:06.454 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 2021 2021-03-24T21:59:06.454 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: - 2021-03-24T21:59:06.454 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 03 2021-03-24T21:59:06.454 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: - 2021-03-24T21:59:06.455 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 24 2021-03-24T21:59:06.455 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: T 2021-03-24T21:59:06.455 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 21 2021-03-24T21:59:06.455 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: : 2021-03-24T21:59:06.456 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 59 2021-03-24T21:59:06.456 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: : 2021-03-24T21:59:06.456 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 04 2021-03-24T21:59:06.456 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: . 2021-03-24T21:59:06.456 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 625950 2021-03-24T21:59:06.457 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: +0000 2021-03-24T21:59:06.457 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 2021-03-24T21:59:06.457 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: osd.4 2021-03-24T21:59:06.457 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: ( 2021-03-24T21:59:06.457 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: osd 2021-03-24T21:59:06.458 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: . 2021-03-24T21:59:06.458 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 4 2021-03-24T21:59:06.458 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: ) 2021-03-24T21:59:06.458 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 96 2021-03-24T21:59:06.459 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: : 2021-03-24T21:59:06.459 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: cluster 2021-03-24T21:59:06.459 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 2021-03-24T21:59:06.459 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: [DBG] 2021-03-24T21:59:06.459 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 2021-03-24T21:59:06.460 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 5.1f scrub ok 2021-03-24T21:59:06.460 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 2021-03-24T21:59:06.460 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: cluster 2021-03-24T21:59:06.460 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 2021-03-24T21:59:06.460 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 2021 2021-03-24T21:59:06.461 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: - 2021-03-24T21:59:06.461 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 03 2021-03-24T21:59:06.461 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: - 2021-03-24T21:59:06.461 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 24 2021-03-24T21:59:06.462 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: T 2021-03-24T21:59:06.462 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 21 2021-03-24T21:59:06.462 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: : 2021-03-24T21:59:06.462 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 59 2021-03-24T21:59:06.462 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: : 2021-03-24T21:59:06.463 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 04 2021-03-24T21:59:06.463 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: . 2021-03-24T21:59:06.463 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 816740 2021-03-24T21:59:06.463 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: +0000 2021-03-24T21:59:06.463 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 2021-03-24T21:59:06.464 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: mgr.y 2021-03-24T21:59:06.464 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: ( 2021-03-24T21:59:06.464 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: mgr 2021-03-24T21:59:06.464 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: . 2021-03-24T21:59:06.464 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 14144 2021-03-24T21:59:06.465 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: ) 2021-03-24T21:59:06.465 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 457 2021-03-24T21:59:06.465 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: : 2021-03-24T21:59:06.465 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: cluster 2021-03-24T21:59:06.465 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 2021-03-24T21:59:06.466 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: [DBG] 2021-03-24T21:59:06.466 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 2021-03-24T21:59:06.466 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: pgmap v724: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 289 MiB data, 755 MiB used, 536 GiB / 536 GiB avail; 19 MiB/s rd, 3.8 MiB/s wr, 29 op/s; 249/1347 objects degraded (18.486%) 2021-03-24T21:59:06.466 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 2021-03-24T21:59:06.467 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: cluster 2021-03-24T21:59:06.467 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 2021-03-24T21:59:06.467 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 2021 2021-03-24T21:59:06.467 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: - 2021-03-24T21:59:06.467 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 03 2021-03-24T21:59:06.468 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: - 2021-03-24T21:59:06.468 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 24 2021-03-24T21:59:06.468 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: T 2021-03-24T21:59:06.468 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 21 2021-03-24T21:59:06.468 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: : 2021-03-24T21:59:06.469 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 59 2021-03-24T21:59:06.469 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: : 2021-03-24T21:59:06.469 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 05 2021-03-24T21:59:06.469 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: . 2021-03-24T21:59:06.469 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 058507 2021-03-24T21:59:06.470 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: +0000 2021-03-24T21:59:06.470 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 2021-03-24T21:59:06.470 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: osd.0 2021-03-24T21:59:06.470 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: ( 2021-03-24T21:59:06.470 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: osd 2021-03-24T21:59:06.471 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: . 2021-03-24T21:59:06.471 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 0 2021-03-24T21:59:06.471 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: ) 2021-03-24T21:59:06.471 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 86 2021-03-24T21:59:06.471 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: : 2021-03-24T21:59:06.472 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: cluster 2021-03-24T21:59:06.473 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 2021-03-24T21:59:06.473 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: [DBG] 2021-03-24T21:59:06.473 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 2021-03-24T21:59:06.473 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 4.b scrub ok 2021-03-24T21:59:06.474 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 2021-03-24T21:59:06.474 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: cluster 2021-03-24T21:59:06.474 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 2021-03-24T21:59:06.474 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 2021 2021-03-24T21:59:06.474 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: - 2021-03-24T21:59:06.475 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 03 2021-03-24T21:59:06.475 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: - 2021-03-24T21:59:06.475 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 24 2021-03-24T21:59:06.475 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: T 2021-03-24T21:59:06.476 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 21 2021-03-24T21:59:06.476 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: : 2021-03-24T21:59:06.476 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 59 2021-03-24T21:59:06.476 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: : 2021-03-24T21:59:06.476 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 05 2021-03-24T21:59:06.477 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: . 2021-03-24T21:59:06.477 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 229541 2021-03-24T21:59:06.477 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: +0000 2021-03-24T21:59:06.477 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 2021-03-24T21:59:06.478 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: mon.a 2021-03-24T21:59:06.478 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: ( 2021-03-24T21:59:06.478 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: mon 2021-03-24T21:59:06.478 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: . 2021-03-24T21:59:06.478 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 0 2021-03-24T21:59:06.479 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: ) 2021-03-24T21:59:06.479 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 1147 2021-03-24T21:59:06.479 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: : 2021-03-24T21:59:06.479 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: cluster 2021-03-24T21:59:06.480 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 2021-03-24T21:59:06.480 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: [DBG] 2021-03-24T21:59:06.480 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 2021-03-24T21:59:06.480 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: osdmap e375: 8 total, 7 up, 6 in 2021-03-24T21:59:06.480 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 2021-03-24T21:59:06.481 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: audit 2021-03-24T21:59:06.481 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 2021-03-24T21:59:06.481 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 2021 2021-03-24T21:59:06.481 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: - 2021-03-24T21:59:06.482 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 03 2021-03-24T21:59:06.482 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: - 2021-03-24T21:59:06.482 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 24 2021-03-24T21:59:06.482 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: T 2021-03-24T21:59:06.483 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 21 2021-03-24T21:59:06.483 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: : 2021-03-24T21:59:06.483 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 59 2021-03-24T21:59:06.483 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: : 2021-03-24T21:59:06.483 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 05 2021-03-24T21:59:06.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: . 2021-03-24T21:59:06.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 556842 2021-03-24T21:59:06.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: +0000 2021-03-24T21:59:06.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 2021-03-24T21:59:06.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: mon.c 2021-03-24T21:59:06.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: ( 2021-03-24T21:59:06.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: mon 2021-03-24T21:59:06.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: . 2021-03-24T21:59:06.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 2 2021-03-24T21:59:06.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: ) 2021-03-24T21:59:06.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 59 2021-03-24T21:59:06.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: : 2021-03-24T21:59:06.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: audit 2021-03-24T21:59:06.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 2021-03-24T21:59:06.487 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: [INF] 2021-03-24T21:59:06.487 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 2021-03-24T21:59:06.487 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: from='client.? v1:172.21.15.125:0/3376785030' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:59:06.487 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 2021-03-24T21:59:06.487 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: audit 2021-03-24T21:59:06.488 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 2021-03-24T21:59:06.488 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 2021 2021-03-24T21:59:06.488 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: - 2021-03-24T21:59:06.488 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 03 2021-03-24T21:59:06.489 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: - 2021-03-24T21:59:06.489 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 24 2021-03-24T21:59:06.489 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: T 2021-03-24T21:59:06.489 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 21 2021-03-24T21:59:06.489 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: : 2021-03-24T21:59:06.490 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 59 2021-03-24T21:59:06.490 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: : 2021-03-24T21:59:06.492 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 05 2021-03-24T21:59:06.492 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: . 2021-03-24T21:59:06.492 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 557206 2021-03-24T21:59:06.492 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: +0000 2021-03-24T21:59:06.493 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 2021-03-24T21:59:06.493 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: mon.a 2021-03-24T21:59:06.493 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: ( 2021-03-24T21:59:06.493 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: mon 2021-03-24T21:59:06.494 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: . 2021-03-24T21:59:06.494 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 0 2021-03-24T21:59:06.494 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: ) 2021-03-24T21:59:06.494 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 1148 2021-03-24T21:59:06.495 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: : 2021-03-24T21:59:06.495 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: audit 2021-03-24T21:59:06.496 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 2021-03-24T21:59:06.496 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: [INF] 2021-03-24T21:59:06.496 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 2021-03-24T21:59:06.497 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: from='client.35147 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:59:06.497 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[19936]: 2021-03-24T21:59:06.498 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: cluster 2021-03-24T21:59:06.498 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 2021-03-24T21:59:06.498 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 2021 2021-03-24T21:59:06.498 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: - 2021-03-24T21:59:06.499 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 03 2021-03-24T21:59:06.499 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: - 2021-03-24T21:59:06.499 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 24 2021-03-24T21:59:06.499 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: T 2021-03-24T21:59:06.499 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 21 2021-03-24T21:59:06.500 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: : 2021-03-24T21:59:06.500 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 59 2021-03-24T21:59:06.500 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: : 2021-03-24T21:59:06.500 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 04 2021-03-24T21:59:06.501 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: . 2021-03-24T21:59:06.501 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 625950 2021-03-24T21:59:06.501 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: +0000 2021-03-24T21:59:06.501 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 2021-03-24T21:59:06.501 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: osd.4 2021-03-24T21:59:06.502 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: ( 2021-03-24T21:59:06.502 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: osd 2021-03-24T21:59:06.502 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: . 2021-03-24T21:59:06.502 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 4 2021-03-24T21:59:06.503 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: ) 2021-03-24T21:59:06.503 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 96 2021-03-24T21:59:06.503 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: : 2021-03-24T21:59:06.503 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: cluster 2021-03-24T21:59:06.503 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 2021-03-24T21:59:06.504 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: [DBG] 2021-03-24T21:59:06.504 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 2021-03-24T21:59:06.504 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 5.1f scrub ok 2021-03-24T21:59:06.504 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 2021-03-24T21:59:06.504 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: cluster 2021-03-24T21:59:06.505 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 2021-03-24T21:59:06.505 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 2021 2021-03-24T21:59:06.505 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: - 2021-03-24T21:59:06.505 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 03 2021-03-24T21:59:06.505 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: - 2021-03-24T21:59:06.506 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 24 2021-03-24T21:59:06.506 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: T 2021-03-24T21:59:06.506 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 21 2021-03-24T21:59:06.506 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: : 2021-03-24T21:59:06.507 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 59 2021-03-24T21:59:06.507 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: : 2021-03-24T21:59:06.507 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 04 2021-03-24T21:59:06.507 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: . 2021-03-24T21:59:06.507 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 816740 2021-03-24T21:59:06.508 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: +0000 2021-03-24T21:59:06.508 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 2021-03-24T21:59:06.508 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: mgr.y 2021-03-24T21:59:06.508 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: ( 2021-03-24T21:59:06.508 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: mgr 2021-03-24T21:59:06.509 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: . 2021-03-24T21:59:06.509 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 14144 2021-03-24T21:59:06.509 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: ) 2021-03-24T21:59:06.509 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 457 2021-03-24T21:59:06.510 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: : 2021-03-24T21:59:06.510 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: cluster 2021-03-24T21:59:06.510 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 2021-03-24T21:59:06.510 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: [DBG] 2021-03-24T21:59:06.510 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 2021-03-24T21:59:06.511 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: pgmap v724: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 289 MiB data, 755 MiB used, 536 GiB / 536 GiB avail; 19 MiB/s rd, 3.8 MiB/s wr, 29 op/s; 249/1347 objects degraded (18.486%) 2021-03-24T21:59:06.511 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 2021-03-24T21:59:06.511 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: cluster 2021-03-24T21:59:06.511 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 2021-03-24T21:59:06.511 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 2021 2021-03-24T21:59:06.512 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: - 2021-03-24T21:59:06.512 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 03 2021-03-24T21:59:06.512 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: - 2021-03-24T21:59:06.512 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 24 2021-03-24T21:59:06.512 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: T 2021-03-24T21:59:06.513 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 21 2021-03-24T21:59:06.513 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: : 2021-03-24T21:59:06.513 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 59 2021-03-24T21:59:06.513 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: : 2021-03-24T21:59:06.514 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 05 2021-03-24T21:59:06.514 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: . 2021-03-24T21:59:06.514 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 058507 2021-03-24T21:59:06.514 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: +0000 2021-03-24T21:59:06.514 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 2021-03-24T21:59:06.515 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: osd.0 2021-03-24T21:59:06.515 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: ( 2021-03-24T21:59:06.515 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: osd 2021-03-24T21:59:06.515 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: . 2021-03-24T21:59:06.515 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 0 2021-03-24T21:59:06.516 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: ) 2021-03-24T21:59:06.516 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 86 2021-03-24T21:59:06.516 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: : 2021-03-24T21:59:06.516 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: cluster 2021-03-24T21:59:06.517 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 2021-03-24T21:59:06.517 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: [DBG] 2021-03-24T21:59:06.517 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 2021-03-24T21:59:06.517 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 4.b scrub ok 2021-03-24T21:59:06.517 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 2021-03-24T21:59:06.518 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: cluster 2021-03-24T21:59:06.518 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 2021-03-24T21:59:06.518 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 2021 2021-03-24T21:59:06.518 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: - 2021-03-24T21:59:06.518 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 03 2021-03-24T21:59:06.519 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: - 2021-03-24T21:59:06.519 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 24 2021-03-24T21:59:06.519 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: T 2021-03-24T21:59:06.519 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 21 2021-03-24T21:59:06.520 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: : 2021-03-24T21:59:06.520 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 59 2021-03-24T21:59:06.520 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: : 2021-03-24T21:59:06.520 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 05 2021-03-24T21:59:06.520 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: . 2021-03-24T21:59:06.521 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 229541 2021-03-24T21:59:06.521 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: +0000 2021-03-24T21:59:06.521 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 2021-03-24T21:59:06.521 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: mon.a 2021-03-24T21:59:06.521 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: ( 2021-03-24T21:59:06.522 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: mon 2021-03-24T21:59:06.522 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: . 2021-03-24T21:59:06.522 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 0 2021-03-24T21:59:06.522 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: ) 2021-03-24T21:59:06.523 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 1147 2021-03-24T21:59:06.523 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: : 2021-03-24T21:59:06.523 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: cluster 2021-03-24T21:59:06.523 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 2021-03-24T21:59:06.523 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: [DBG] 2021-03-24T21:59:06.524 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 2021-03-24T21:59:06.524 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: osdmap e375: 8 total, 7 up, 6 in 2021-03-24T21:59:06.524 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 2021-03-24T21:59:06.524 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: audit 2021-03-24T21:59:06.524 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 2021-03-24T21:59:06.525 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 2021 2021-03-24T21:59:06.525 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: - 2021-03-24T21:59:06.525 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 03 2021-03-24T21:59:06.525 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: - 2021-03-24T21:59:06.525 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 24 2021-03-24T21:59:06.526 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: T 2021-03-24T21:59:06.526 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 21 2021-03-24T21:59:06.526 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: : 2021-03-24T21:59:06.526 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 59 2021-03-24T21:59:06.526 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: : 2021-03-24T21:59:06.527 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 05 2021-03-24T21:59:06.527 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: . 2021-03-24T21:59:06.527 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 556842 2021-03-24T21:59:06.527 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: +0000 2021-03-24T21:59:06.528 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 2021-03-24T21:59:06.528 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: mon.c 2021-03-24T21:59:06.528 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: ( 2021-03-24T21:59:06.528 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: mon 2021-03-24T21:59:06.528 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: . 2021-03-24T21:59:06.529 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 2 2021-03-24T21:59:06.529 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: ) 2021-03-24T21:59:06.529 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 59 2021-03-24T21:59:06.529 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: : 2021-03-24T21:59:06.529 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: audit 2021-03-24T21:59:06.530 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 2021-03-24T21:59:06.530 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: [INF] 2021-03-24T21:59:06.530 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 2021-03-24T21:59:06.530 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: from='client.? v1:172.21.15.125:0/3376785030' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:59:06.530 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 2021-03-24T21:59:06.531 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: audit 2021-03-24T21:59:06.531 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 2021-03-24T21:59:06.531 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 2021 2021-03-24T21:59:06.531 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: - 2021-03-24T21:59:06.531 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 03 2021-03-24T21:59:06.532 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: - 2021-03-24T21:59:06.532 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 24 2021-03-24T21:59:06.532 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: T 2021-03-24T21:59:06.532 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 21 2021-03-24T21:59:06.533 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: : 2021-03-24T21:59:06.533 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 59 2021-03-24T21:59:06.533 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: : 2021-03-24T21:59:06.533 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 05 2021-03-24T21:59:06.533 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: . 2021-03-24T21:59:06.534 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 557206 2021-03-24T21:59:06.534 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: +0000 2021-03-24T21:59:06.534 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 2021-03-24T21:59:06.534 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: mon.a 2021-03-24T21:59:06.535 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: ( 2021-03-24T21:59:06.535 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: mon 2021-03-24T21:59:06.535 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: . 2021-03-24T21:59:06.535 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 0 2021-03-24T21:59:06.536 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: ) 2021-03-24T21:59:06.536 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 1148 2021-03-24T21:59:06.536 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: : 2021-03-24T21:59:06.536 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: audit 2021-03-24T21:59:06.536 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 2021-03-24T21:59:06.537 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: [INF] 2021-03-24T21:59:06.537 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 2021-03-24T21:59:06.537 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: from='client.35147 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:59:06.537 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:06 smithi125 conmon[24927]: 2021-03-24T21:59:06.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:06.539 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:06.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:06.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:06 smithi142 conmon[20125]: cluster 2021-03-24T21:59:04.625950+0000 osd.4 (osd.4 2021-03-24T21:59:06.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:06 smithi142 conmon[20125]: ) 96 : cluster [DBG] 5.1f scrub ok 2021-03-24T21:59:06.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:06 smithi142 conmon[20125]: cluster 2021-03-24T21:59:04.816740+0000 mgr.y (mgr.14144) 457 : cluster [DBG] pgmap v724: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 289 MiB data, 755 MiB used, 536 GiB / 536 GiB avail; 19 MiB/s rd, 3.8 MiB/s wr, 29 op/s; 249/1347 objects degraded (18.486%) 2021-03-24T21:59:06.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:06 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:59:06.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:06 smithi142 conmon[20125]: 21:59:05.058507+0000 osd.0 (osd.0) 86 : cluster [DBG] 4.b scrub ok 2021-03-24T21:59:06.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:06 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:59:06.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:06 smithi142 conmon[20125]: -03-24T21:59:05.229541+0000 mon.a (mon.0) 1147 : cluster 2021-03-24T21:59:06.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:06 smithi142 conmon[20125]: [DBG] osdmap e375: 8 total, 7 up, 6 in 2021-03-24T21:59:06.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:06 smithi142 conmon[20125]: audit 2021-03-24T21:59:05.556842+0000 2021-03-24T21:59:06.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:06 smithi142 conmon[20125]: mon.c (mon.2) 59 : audit [INF] from='client.? v1:172.21.15.125:0/3376785030' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:59:06.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:06 smithi142 conmon[20125]: 2021-03-24T21:59:06.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:06 smithi142 conmon[20125]: audit 2021 2021-03-24T21:59:06.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:06 smithi142 conmon[20125]: -03-24T21:59:05.557206+0000 mon.a (mon.0) 1148 : 2021-03-24T21:59:06.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:06 smithi142 conmon[20125]: audit [INF] from='client.35147 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:59:06.641 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:06.692 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-03-24T21:59:06.693 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-03-24T21:59:06.694 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:06.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:06.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:59:06.800 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:06.856 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:06.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:06.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:59:06.963 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:07.024 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:07.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:07.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:59:07.130 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:07.192 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:07.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:07.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:07.277 INFO:tasks.rados.rados.0.smithi125.stdout:1363: done (2 left) 2021-03-24T21:59:07.278 INFO:tasks.rados.rados.0.smithi125.stdout:1364: done (1 left) 2021-03-24T21:59:07.278 INFO:tasks.rados.rados.0.smithi125.stdout:1366: done (0 left) 2021-03-24T21:59:07.278 INFO:tasks.rados.rados.0.smithi125.stdout:1367: rollback oid 27 current snap is 142 2021-03-24T21:59:07.278 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 27 to 117 2021-03-24T21:59:07.279 INFO:tasks.rados.rados.0.smithi125.stdout:1368: read oid 44 snap -1 2021-03-24T21:59:07.279 INFO:tasks.rados.rados.0.smithi125.stdout:1368: expect (ObjNum 470 snap 133 seq_num 470) 2021-03-24T21:59:07.279 INFO:tasks.rados.rados.0.smithi125.stdout:1369: write (excl) oid 49 current snap is 142 2021-03-24T21:59:07.279 INFO:tasks.rados.rados.0.smithi125.stdout:1369: seq_num 494 ranges {702658=619598,1857987=444671} 2021-03-24T21:59:07.281 INFO:tasks.rados.rados.0.smithi125.stdout:1369: writing smithi12556470-49 from 702658 to 1322256 tid 1 2021-03-24T21:59:07.288 INFO:tasks.rados.rados.0.smithi125.stdout:1369: writing smithi12556470-49 from 1857987 to 2302658 tid 2 2021-03-24T21:59:07.289 INFO:tasks.rados.rados.0.smithi125.stdout:1370: read oid 10 snap 128 2021-03-24T21:59:07.289 INFO:tasks.rados.rados.0.smithi125.stdout:1370: expect (ObjNum 393 snap 100 seq_num 393) 2021-03-24T21:59:07.289 INFO:tasks.rados.rados.0.smithi125.stdout:1371: write oid 11 current snap is 142 2021-03-24T21:59:07.289 INFO:tasks.rados.rados.0.smithi125.stdout:1367: finishing rollback tid 0 to smithi12556470-27 2021-03-24T21:59:07.290 INFO:tasks.rados.rados.0.smithi125.stdout:1367: finishing rollback tid 1 to smithi12556470-27 2021-03-24T21:59:07.290 INFO:tasks.rados.rados.0.smithi125.stdout:1367: finishing rollback tid 2 to smithi12556470-27 2021-03-24T21:59:07.290 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 27 v 329 (ObjNum 400 snap 102 seq_num 400) dirty exists 2021-03-24T21:59:07.290 INFO:tasks.rados.rados.0.smithi125.stdout:1371: seq_num 495 ranges {719465=411200,1632822=686643} 2021-03-24T21:59:07.296 INFO:tasks.rados.rados.0.smithi125.stdout:1371: writing smithi12556470-11 from 719465 to 1130665 tid 1 2021-03-24T21:59:07.297 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:07.305 INFO:tasks.rados.rados.0.smithi125.stdout:1371: writing smithi12556470-11 from 1632822 to 2319465 tid 2 2021-03-24T21:59:07.306 INFO:tasks.rados.rados.0.smithi125.stdout:1367: done (4 left) 2021-03-24T21:59:07.306 INFO:tasks.rados.rados.0.smithi125.stdout:1372: copy_from oid 34 from oid 22 current snap is 142 2021-03-24T21:59:07.313 INFO:tasks.rados.rados.0.smithi125.stdout:1370: expect (ObjNum 393 snap 100 seq_num 393) 2021-03-24T21:59:07.374 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:07.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:07.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:59:07.462 INFO:tasks.rados.rados.0.smithi125.stdout:1369: finishing write tid 1 to smithi12556470-49 2021-03-24T21:59:07.463 INFO:tasks.rados.rados.0.smithi125.stdout:1370: done (4 left) 2021-03-24T21:59:07.463 INFO:tasks.rados.rados.0.smithi125.stdout:1373: copy_from oid 47 from oid 45 current snap is 142 2021-03-24T21:59:07.463 INFO:tasks.rados.rados.0.smithi125.stdout:1368: expect (ObjNum 470 snap 133 seq_num 470) 2021-03-24T21:59:07.482 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:07.492 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:07 smithi142 conmon[20125]: cluster 2021-03-24T21:59:05.405779 2021-03-24T21:59:07.493 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:07 smithi142 conmon[20125]: +0000 osd.3 (osd.3) 83 : cluster [DBG] 3.1e scrub ok 2021-03-24T21:59:07.493 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:07 smithi142 conmon[20125]: cluster 2021-03-24T21:59: 2021-03-24T21:59:07.493 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:07 smithi142 conmon[20125]: 05.674540+0000 osd.4 (osd.4) 97 : cluster [DBG] 4.d scrub ok 2021-03-24T21:59:07.493 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:07 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:59:07.494 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:07 smithi142 conmon[20125]: -24T21:59:06.226094+0000 2021-03-24T21:59:07.494 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:07 smithi142 conmon[20125]: mon.a (mon.0) 1149 : cluster [INF] 2021-03-24T21:59:07.494 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:07 smithi142 conmon[20125]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T21:59:07.494 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:07 smithi142 conmon[20125]: audit 2021-03-24T21:59:07.495 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:07 smithi142 conmon[20125]: 2021-03-24T21:59:06.235689+0000 mon.a 2021-03-24T21:59:07.495 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:07 smithi142 conmon[20125]: (mon.0) 1150 : audit [INF] from='client.35147 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T21:59:07.495 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:07 smithi142 conmon[20125]: 2021-03-24T21:59:07.495 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:07 smithi142 conmon[20125]: cluster 2021-03-24T21:59:07.495 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:07 smithi142 conmon[20125]: 2021-03-24T 2021-03-24T21:59:07.496 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:07 smithi142 conmon[20125]: 21:59:06.235864+0000 mon.a (mon 2021-03-24T21:59:07.496 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:07 smithi142 conmon[20125]: .0) 1151 : cluster [DBG] osdmap e376: 8 total, 7 up, 6 in 2021-03-24T21:59:07.496 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:07 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:59:07.496 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:07 smithi142 conmon[20125]: -03-24T21:59:06.367162 2021-03-24T21:59:07.497 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:07 smithi142 conmon[20125]: +0000 osd.3 (osd.3) 84 : cluster [DBG] 2021-03-24T21:59:07.497 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:07 smithi142 conmon[20125]: 4.17 scrub ok 2021-03-24T21:59:07.523 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:07 smithi125 conmon[24927]: cluster 2021-03-24T21:59: 2021-03-24T21:59:07.524 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:07 smithi125 conmon[24927]: 05.405779+0000 osd.3 (osd.3) 83 : cluster [DBG] 3.1e scrub ok 2021-03-24T21:59:07.524 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:07 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:59:07.524 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:07 smithi125 conmon[24927]: 03-24T21:59:05.674540+0000 osd.4 (osd.4) 97 : cluster [DBG] 2021-03-24T21:59:07.524 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:07 smithi125 conmon[24927]: 4.d scrub ok 2021-03-24T21:59:07.525 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:07 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:59:07.525 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:07 smithi125 conmon[24927]: 24T21:59:06.226094+0000 2021-03-24T21:59:07.525 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:07 smithi125 conmon[24927]: mon.a (mon.0) 1149 : cluster [INF] 2021-03-24T21:59:07.525 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:07 smithi125 conmon[24927]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T21:59:07.525 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:07 smithi125 conmon[24927]: audit 2021-03-24T21: 2021-03-24T21:59:07.526 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:07 smithi125 conmon[24927]: 59:06.235689+0000 mon.a (mon.0 2021-03-24T21:59:07.526 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:07 smithi125 conmon[24927]: ) 1150 : audit [INF] from='client.35147 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T21:59:07.526 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:07 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:59:07.526 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:07 smithi125 conmon[24927]: 21:59:06.235864+0000 mon.a (mon 2021-03-24T21:59:07.527 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:07 smithi125 conmon[24927]: .0) 1151 : cluster [DBG] osdmap e376: 8 total, 7 up, 6 in 2021-03-24T21:59:07.527 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:07 smithi125 conmon[24927]: cluster 2021-03-24T21:59:07.527 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:07 smithi125 conmon[24927]: 2021-03-24T21 2021-03-24T21:59:07.527 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:07 smithi125 conmon[24927]: :59:06.367162+0000 osd.3 (osd.3 2021-03-24T21:59:07.528 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:07 smithi125 conmon[24927]: ) 84 : cluster [DBG] 4.17 scrub ok 2021-03-24T21:59:07.528 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:07 smithi125 conmon[19936]: cluster 2021-03-24T21:59:05.405779+0000 2021-03-24T21:59:07.528 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:07 smithi125 conmon[19936]: osd.3 (osd.3) 83 : cluster [DBG] 3.1e scrub ok 2021-03-24T21:59:07.529 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:07 smithi125 conmon[19936]: cluster 2021-03-24T21:59:05.674540+0000 osd.4 (osd.4 2021-03-24T21:59:07.529 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:07 smithi125 conmon[19936]: ) 97 : cluster [DBG] 4.d scrub ok 2021-03-24T21:59:07.529 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:07 smithi125 conmon[19936]: cluster 2021-03-24T21:59 2021-03-24T21:59:07.529 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:07 smithi125 conmon[19936]: :06.226094+0000 mon.a (mon.0) 1149 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T21:59:07.530 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:07 smithi125 conmon[19936]: 2021-03-24T21:59:07.530 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:07 smithi125 conmon[19936]: audit 2021-03-24T21:59: 2021-03-24T21:59:07.530 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:07 smithi125 conmon[19936]: 06.235689+0000 mon.a (mon.0) 1150 : audit 2021-03-24T21:59:07.530 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:07 smithi125 conmon[19936]: [INF] from='client.35147 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T21:59:07.531 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:07 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:59:07.531 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:07 smithi125 conmon[19936]: 24T21:59:06.235864+0000 mon.a (mon 2021-03-24T21:59:07.531 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:07 smithi125 conmon[19936]: .0) 1151 : cluster [DBG] osdmap e376: 8 total, 7 up, 6 in 2021-03-24T21:59:07.531 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:07 smithi125 conmon[19936]: cluster 2021-03-24T21:59:07.531 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:07 smithi125 conmon[19936]: 2021-03-24 2021-03-24T21:59:07.532 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:07 smithi125 conmon[19936]: T21:59:06.367162+0000 osd.3 ( 2021-03-24T21:59:07.532 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:07 smithi125 conmon[19936]: osd.3) 84 : cluster [DBG] 4.17 scrub ok 2021-03-24T21:59:07.532 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:07.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:07.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:59:07.614 INFO:tasks.rados.rados.0.smithi125.stdout:1369: finishing write tid 2 to smithi12556470-49 2021-03-24T21:59:07.614 INFO:tasks.rados.rados.0.smithi125.stdout:1369: finishing write tid 3 to smithi12556470-49 2021-03-24T21:59:07.614 INFO:tasks.rados.rados.0.smithi125.stdout:1369: finishing write tid 4 to smithi12556470-49 2021-03-24T21:59:07.615 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 49 v 443 (ObjNum 494 snap 142 seq_num 494) dirty exists 2021-03-24T21:59:07.615 INFO:tasks.rados.rados.0.smithi125.stdout:1369: left oid 49 (ObjNum 494 snap 142 seq_num 494) 2021-03-24T21:59:07.615 INFO:tasks.rados.rados.0.smithi125.stdout:1371: finishing write tid 1 to smithi12556470-11 2021-03-24T21:59:07.615 INFO:tasks.rados.rados.0.smithi125.stdout:1371: finishing write tid 2 to smithi12556470-11 2021-03-24T21:59:07.615 INFO:tasks.rados.rados.0.smithi125.stdout:1371: finishing write tid 3 to smithi12556470-11 2021-03-24T21:59:07.616 INFO:tasks.rados.rados.0.smithi125.stdout:1371: finishing write tid 4 to smithi12556470-11 2021-03-24T21:59:07.616 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 11 v 312 (ObjNum 495 snap 142 seq_num 495) dirty exists 2021-03-24T21:59:07.616 INFO:tasks.rados.rados.0.smithi125.stdout:1371: left oid 11 (ObjNum 495 snap 142 seq_num 495) 2021-03-24T21:59:07.616 INFO:tasks.rados.rados.0.smithi125.stdout:1372: finishing copy_from to smithi12556470-34 2021-03-24T21:59:07.617 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 34 v 314 (ObjNum 468 snap 132 seq_num 468) dirty exists 2021-03-24T21:59:07.617 INFO:tasks.rados.rados.0.smithi125.stdout:1372: finishing copy_from racing read to smithi12556470-34 2021-03-24T21:59:07.617 INFO:tasks.rados.rados.0.smithi125.stdout:1368: done (4 left) 2021-03-24T21:59:07.617 INFO:tasks.rados.rados.0.smithi125.stdout:1369: done (3 left) 2021-03-24T21:59:07.618 INFO:tasks.rados.rados.0.smithi125.stdout:1371: done (2 left) 2021-03-24T21:59:07.618 INFO:tasks.rados.rados.0.smithi125.stdout:1372: done (1 left) 2021-03-24T21:59:07.618 INFO:tasks.rados.rados.0.smithi125.stdout:1374: read oid 40 snap 140 2021-03-24T21:59:07.618 INFO:tasks.rados.rados.0.smithi125.stdout:1374: expect (ObjNum 483 snap 138 seq_num 483) 2021-03-24T21:59:07.618 INFO:tasks.rados.rados.0.smithi125.stdout:1375: read oid 19 snap -1 2021-03-24T21:59:07.619 INFO:tasks.rados.rados.0.smithi125.stdout:1375: expect deleted 2021-03-24T21:59:07.619 INFO:tasks.rados.rados.0.smithi125.stdout:1376: rollback oid 26 current snap is 142 2021-03-24T21:59:07.619 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 26 to 136 2021-03-24T21:59:07.619 INFO:tasks.rados.rados.0.smithi125.stdout:1373: finishing copy_from to smithi12556470-47 2021-03-24T21:59:07.620 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 47 v 305 (ObjNum 466 snap 131 seq_num 466) dirty exists 2021-03-24T21:59:07.620 INFO:tasks.rados.rados.0.smithi125.stdout:1373: finishing copy_from racing read to smithi12556470-47 2021-03-24T21:59:07.620 INFO:tasks.rados.rados.0.smithi125.stdout:1373: done (3 left) 2021-03-24T21:59:07.620 INFO:tasks.rados.rados.0.smithi125.stdout:1377: copy_from oid 36 from oid 41 current snap is 142 2021-03-24T21:59:07.620 INFO:tasks.rados.rados.0.smithi125.stdout:1378: read oid 24 snap -1 2021-03-24T21:59:07.621 INFO:tasks.rados.rados.0.smithi125.stdout:1378: expect (ObjNum 444 snap 120 seq_num 444) 2021-03-24T21:59:07.621 INFO:tasks.rados.rados.0.smithi125.stdout:1379: snap_create 2021-03-24T21:59:07.621 INFO:tasks.rados.rados.0.smithi125.stdout:1374: expect (ObjNum 483 snap 138 seq_num 483) 2021-03-24T21:59:07.635 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:07.678 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:07.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:07.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:59:07.756 INFO:tasks.rados.rados.0.smithi125.stdout:1376: finishing rollback tid 0 to smithi12556470-26 2021-03-24T21:59:07.757 INFO:tasks.rados.rados.0.smithi125.stdout:1376: finishing rollback tid 1 to smithi12556470-26 2021-03-24T21:59:07.757 INFO:tasks.rados.rados.0.smithi125.stdout:1376: finishing rollback tid 2 to smithi12556470-26 2021-03-24T21:59:07.758 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 26 v 435 (ObjNum 438 snap 118 seq_num 438) dirty exists 2021-03-24T21:59:07.758 INFO:tasks.rados.rados.0.smithi125.stdout:1378: expect (ObjNum 444 snap 120 seq_num 444) 2021-03-24T21:59:07.786 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:07.838 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:07.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:07.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:59:07.866 INFO:tasks.rados.rados.0.smithi125.stdout:1377: finishing copy_from to smithi12556470-36 2021-03-24T21:59:07.866 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 36 v 371 (ObjNum 460 snap 129 seq_num 460) dirty exists 2021-03-24T21:59:07.866 INFO:tasks.rados.rados.0.smithi125.stdout:1377: finishing copy_from racing read to smithi12556470-36 2021-03-24T21:59:07.944 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:08.000 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:08.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:08.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:08.105 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:08.170 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:08.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:08.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:08.276 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:08.286 INFO:tasks.rados.rados.0.smithi125.stdout:1374: done (5 left) 2021-03-24T21:59:08.286 INFO:tasks.rados.rados.0.smithi125.stdout:1375: done (4 left) 2021-03-24T21:59:08.286 INFO:tasks.rados.rados.0.smithi125.stdout:1376: done (3 left) 2021-03-24T21:59:08.287 INFO:tasks.rados.rados.0.smithi125.stdout:1377: done (2 left) 2021-03-24T21:59:08.287 INFO:tasks.rados.rados.0.smithi125.stdout:1378: done (1 left) 2021-03-24T21:59:08.287 INFO:tasks.rados.rados.0.smithi125.stdout:1379: done (0 left) 2021-03-24T21:59:08.287 INFO:tasks.rados.rados.0.smithi125.stdout:1380: read oid 40 snap -1 2021-03-24T21:59:08.287 INFO:tasks.rados.rados.0.smithi125.stdout:1380: expect (ObjNum 483 snap 138 seq_num 483) 2021-03-24T21:59:08.288 INFO:tasks.rados.rados.0.smithi125.stdout:1381: copy_from oid 1 from oid 49 current snap is 143 2021-03-24T21:59:08.288 INFO:tasks.rados.rados.0.smithi125.stdout:1382: read oid 37 snap -1 2021-03-24T21:59:08.288 INFO:tasks.rados.rados.0.smithi125.stdout:1382: expect (ObjNum 492 snap 141 seq_num 492) 2021-03-24T21:59:08.288 INFO:tasks.rados.rados.0.smithi125.stdout:1383: rollback oid 39 current snap is 143 2021-03-24T21:59:08.289 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 39 to 129 2021-03-24T21:59:08.289 INFO:tasks.rados.rados.0.smithi125.stdout:1384: snap_create 2021-03-24T21:59:08.292 INFO:tasks.rados.rados.0.smithi125.stdout:1380: expect (ObjNum 483 snap 138 seq_num 483) 2021-03-24T21:59:08.365 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:08.366 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:08 smithi125 conmon[19936]: cluster 2021-03-24T21:59:06 2021-03-24T21:59:08.366 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:08 smithi125 conmon[19936]: .817835+0000 mgr.y (mgr.14144) 458 : cluster [DBG] pgmap v727: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 291 MiB data, 760 MiB used, 536 GiB / 536 GiB avail; 19 MiB/s rd, 4.1 MiB/s wr, 30 op/s; 249/1347 objects degraded (18.486%) 2021-03-24T21:59:08.366 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:08 smithi125 conmon[19936]: cluster 2021-03-24T21:59:08.366 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:08 smithi125 conmon[19936]: 2021-03-24T21:59:06.906690+0000 osd.7 (osd.7) 83 : cluster [DBG] 5.9 scrub ok 2021-03-24T21:59:08.367 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:08 smithi125 conmon[19936]: cluster 2021-03-24T21:59:07. 2021-03-24T21:59:08.367 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:08 smithi125 conmon[19936]: 266154+0000 mon.a (mon.0) 1152 : cluster [DBG] osdmap e377: 8 total, 7 up, 6 in 2021-03-24T21:59:08.367 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:08 smithi125 conmon[19936]: cluster 2021-03-24T21:59: 2021-03-24T21:59:08.367 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:08 smithi125 conmon[19936]: 07.373681+0000 osd.3 (osd.3) 85 : cluster [DBG] 2021-03-24T21:59:08.368 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:08 smithi125 conmon[19936]: 7.3 scrub ok 2021-03-24T21:59:08.368 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:08 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:59:08.368 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:08 smithi125 conmon[24927]: 21:59:06.817835+0000 mgr.y (mgr.14144) 458 : cluster [DBG] pgmap v727: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 291 MiB data, 760 MiB used, 536 GiB / 536 GiB avail; 19 MiB/s rd, 4.1 MiB/s wr, 30 op/s; 249/1347 objects degraded (18.486%) 2021-03-24T21:59:08.368 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:08 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:59:08.369 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:08 smithi125 conmon[24927]: -03-24T21:59:06.906690+0000 osd.7 (osd.7 2021-03-24T21:59:08.369 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:08 smithi125 conmon[24927]: ) 83 : cluster [DBG] 5.9 scrub ok 2021-03-24T21:59:08.369 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:08 smithi125 conmon[24927]: cluster 2021-03-24T21:59 2021-03-24T21:59:08.369 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:08 smithi125 conmon[24927]: :07.266154+0000 mon.a (mon.0) 1152 : cluster 2021-03-24T21:59:08.369 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:08 smithi125 conmon[24927]: [DBG] osdmap e377: 8 total, 7 up, 6 in 2021-03-24T21:59:08.370 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:08 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:59:08.370 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:08 smithi125 conmon[24927]: 21:59:07.373681+0000 osd.3 (osd. 2021-03-24T21:59:08.370 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:08 smithi125 conmon[24927]: 3) 85 : cluster [DBG] 7.3 scrub ok 2021-03-24T21:59:08.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:08.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:59:08.467 INFO:tasks.rados.rados.0.smithi125.stdout:1383: finishing rollback tid 0 to smithi12556470-39 2021-03-24T21:59:08.467 INFO:tasks.rados.rados.0.smithi125.stdout:1383: finishing rollback tid 1 to smithi12556470-39 2021-03-24T21:59:08.467 INFO:tasks.rados.rados.0.smithi125.stdout:1383: finishing rollback tid 2 to smithi12556470-39 2021-03-24T21:59:08.468 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 39 v 384 (ObjNum 408 snap 106 seq_num 408) dirty exists 2021-03-24T21:59:08.468 INFO:tasks.rados.rados.0.smithi125.stdout:1382: expect (ObjNum 492 snap 141 seq_num 492) 2021-03-24T21:59:08.472 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:08.544 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:08.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:08.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:59:08.596 INFO:tasks.rados.rados.0.smithi125.stdout:1381: finishing copy_from to smithi12556470-1 2021-03-24T21:59:08.597 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 1 v 443 (ObjNum 494 snap 142 seq_num 494) dirty exists 2021-03-24T21:59:08.597 INFO:tasks.rados.rados.0.smithi125.stdout:1381: finishing copy_from racing read to smithi12556470-1 2021-03-24T21:59:08.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:08 smithi142 conmon[20125]: cluster 2021-03-24T21:59 2021-03-24T21:59:08.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:08 smithi142 conmon[20125]: :06.817835+0000 mgr.y (mgr.14144) 458 : cluster [DBG] pgmap v727: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 291 MiB data, 760 MiB used, 536 GiB / 536 GiB avail; 19 MiB/s rd, 4.1 MiB/s wr, 30 op/s; 249/1347 objects degraded (18.486%) 2021-03-24T21:59:08.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:08 smithi142 conmon[20125]: cluster 2021-03-24T21:59:08.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:08 smithi142 conmon[20125]: 2021-03-24T21:59:06.906690+0000 osd.7 (osd.7) 83 : cluster [DBG] 2021-03-24T21:59:08.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:08 smithi142 conmon[20125]: 5.9 scrub ok 2021-03-24T21:59:08.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:08 smithi142 conmon[20125]: cluster 2021-03-24T21:59:08.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:08 smithi142 conmon[20125]: 2021-03-24T21:59: 2021-03-24T21:59:08.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:08 smithi142 conmon[20125]: 07.266154+0000 mon.a (mon.0) 1152 : 2021-03-24T21:59:08.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:08 smithi142 conmon[20125]: cluster [DBG] osdmap e377: 8 total, 7 up, 6 in 2021-03-24T21:59:08.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:08 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:59:08.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:08 smithi142 conmon[20125]: 03-24T21:59:07.373681+0000 osd.3 ( 2021-03-24T21:59:08.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:08 smithi142 conmon[20125]: osd.3) 85 : cluster [DBG] 7.3 scrub ok 2021-03-24T21:59:08.650 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:08.694 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-24T21:59:08.702 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:08.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:08.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:08.807 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:08.862 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:08.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:08.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:08.967 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:09.017 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:09.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:09.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:59:09.123 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:09.171 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:09.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:09.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:59:09.277 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:09.289 INFO:tasks.rados.rados.0.smithi125.stdout:1380: done (4 left) 2021-03-24T21:59:09.289 INFO:tasks.rados.rados.0.smithi125.stdout:1381: done (3 left) 2021-03-24T21:59:09.289 INFO:tasks.rados.rados.0.smithi125.stdout:1382: done (2 left) 2021-03-24T21:59:09.290 INFO:tasks.rados.rados.0.smithi125.stdout:1383: done (1 left) 2021-03-24T21:59:09.290 INFO:tasks.rados.rados.0.smithi125.stdout:1384: done (0 left) 2021-03-24T21:59:09.290 INFO:tasks.rados.rados.0.smithi125.stdout:1385: read oid 24 snap 137 2021-03-24T21:59:09.291 INFO:tasks.rados.rados.0.smithi125.stdout:1385: expect (ObjNum 444 snap 120 seq_num 444) 2021-03-24T21:59:09.291 INFO:tasks.rados.rados.0.smithi125.stdout:1386: snap_create 2021-03-24T21:59:09.292 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:59:09.334 INFO:tasks.rados.rados.0.smithi125.stdout:1385: expect (ObjNum 444 snap 120 seq_num 444) 2021-03-24T21:59:09.388 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:09.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:09.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:09.456 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:09 smithi125 conmon[24927]: cluster 2021-03-24T21:59:08.017862+0000 2021-03-24T21:59:09.457 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:09 smithi125 conmon[24927]: osd.0 (osd.0) 87 : cluster [DBG] 7.f scrub ok 2021-03-24T21:59:09.457 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:09 smithi125 conmon[24927]: cluster 2021-03-24T21:59:08.278283+0000 mon.a ( 2021-03-24T21:59:09.457 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:09 smithi125 conmon[24927]: mon.0) 1153 : cluster [DBG] osdmap e378: 8 total, 7 up, 6 in 2021-03-24T21:59:09.458 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:09 smithi125 conmon[24927]: cluster 2021-03-24T21:59: 2021-03-24T21:59:09.458 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:09 smithi125 conmon[24927]: 08.406177+0000 osd.3 (osd.3) 86 : cluster [DBG] 7.2 deep-scrub ok 2021-03-24T21:59:09.458 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:09 smithi125 conmon[19936]: cluster 2021-03-24T21:59:08.017862+0000 osd.0 (osd.0) 87 : cluster [DBG] 7.f scrub ok 2021-03-24T21:59:09.459 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:09 smithi125 conmon[19936]: cluster 2021-03-24T21:59:08.278283+0000 mon.a (mon.0) 1153 : cluster [DBG] osdmap e378: 8 total, 7 up, 6 in 2021-03-24T21:59:09.459 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:09 smithi125 conmon[19936]: cluster 2021-03-24T21:59:08.406177+0000 osd.3 (osd.3) 86 : cluster [DBG] 7.2 deep-scrub ok 2021-03-24T21:59:09.511 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:09.558 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:09.562 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:09 smithi142 conmon[20125]: cluster 2021-03-24T21:59:08 2021-03-24T21:59:09.562 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:09 smithi142 conmon[20125]: .017862+0000 osd.0 (osd.0) 87 : cluster [DBG] 7.f scrub ok 2021-03-24T21:59:09.562 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:09 smithi142 conmon[20125]: cluster 2021-03-24T21:59:09.563 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:09 smithi142 conmon[20125]: 2021-03-24T21:59:08 2021-03-24T21:59:09.563 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:09 smithi142 conmon[20125]: .278283+0000 mon.a (mon.0) 1153 2021-03-24T21:59:09.563 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:09 smithi142 conmon[20125]: : cluster [DBG] osdmap e378: 8 total, 7 up, 6 in 2021-03-24T21:59:09.563 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:09 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:59:09.563 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:09 smithi142 conmon[20125]: -03-24T21:59:08.406177+0000 osd.3 2021-03-24T21:59:09.564 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:09 smithi142 conmon[20125]: (osd.3) 86 : cluster 2021-03-24T21:59:09.564 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:09 smithi142 conmon[20125]: [DBG] 7.2 deep-scrub ok 2021-03-24T21:59:09.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:09.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:59:09.666 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:09.716 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:09.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:09.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:59:09.820 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:09.872 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:09.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:09.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:59:09.979 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:10.024 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:10.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:10.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:10.110 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-24T21:59:10.111 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-24T21:59:10.111 INFO:teuthology.orchestra.run.smithi125.stdout: "num_ops": 0 2021-03-24T21:59:10.111 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-24T21:59:10.129 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:10.179 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:10.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:10.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:10.217 INFO:tasks.rados.rados.0.smithi125.stdout:1385: done (1 left) 2021-03-24T21:59:10.217 INFO:tasks.rados.rados.0.smithi125.stdout:1386: done (0 left) 2021-03-24T21:59:10.218 INFO:tasks.rados.rados.0.smithi125.stdout:1387: write (excl) oid 19 current snap is 145 2021-03-24T21:59:10.218 INFO:tasks.rados.rados.0.smithi125.stdout:1387: seq_num 496 ranges {736272=486449,1974951=361321} 2021-03-24T21:59:10.218 INFO:tasks.rados.rados.0.smithi125.stdout:1387: writing smithi12556470-19 from 736272 to 1222721 tid 1 2021-03-24T21:59:10.223 INFO:tasks.rados.rados.0.smithi125.stdout:1387: writing smithi12556470-19 from 1974951 to 2336272 tid 2 2021-03-24T21:59:10.223 INFO:tasks.rados.rados.0.smithi125.stdout:1388: snap_remove snap 144 2021-03-24T21:59:10.224 INFO:teuthology.orchestra.run.smithi125.stderr:set pool 3 pg_num to 42 2021-03-24T21:59:10.286 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:10.350 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:10.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:10.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:59:10.455 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:10.497 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:10.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:10.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:59:10.589 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[19936]: cluster 2021-03-24T21:59:08.818758+0000 2021-03-24T21:59:10.589 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[19936]: mgr.y (mgr.14144) 459 : cluster [DBG] pgmap v730: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 292 MiB data, 766 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 556 KiB/s wr, 2 op/s; 249/1350 objects degraded (18.444%) 2021-03-24T21:59:10.590 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[19936]: cluster 2021-03-24T21:59:08.942239 2021-03-24T21:59:10.590 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[19936]: +0000 osd.7 (osd.7) 84 : cluster [DBG] 5.1a scrub ok 2021-03-24T21:59:10.590 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:59:10.591 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[19936]: 03-24T21:59:09.287158+0000 mon.a 2021-03-24T21:59:10.591 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[19936]: (mon.0) 1154 : cluster [DBG] osdmap e379: 8 total, 7 up, 6 in 2021-03-24T21:59:10.591 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:59:10.592 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[19936]: 03-24T21:59:09.981164+0000 2021-03-24T21:59:10.592 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[19936]: mon.c (mon.2) 60 : audit [INF] 2021-03-24T21:59:10.593 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[19936]: from='client.? v1:172.21.15.125:0/1407951140' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "42"}]: dispatch 2021-03-24T21:59:10.593 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[19936]: audit 2021-03-24T21:59: 2021-03-24T21:59:10.593 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[19936]: 09.981570+0000 mon.a (mon.0) 1155 2021-03-24T21:59:10.593 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[19936]: : audit [INF] from='client.35153 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "42"}]: dispatch 2021-03-24T21:59:10.594 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T21:59:10.594 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[19936]: T21:59:10.195315+0000 mon.a ( 2021-03-24T21:59:10.594 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[19936]: mon.0) 1156 : cluster [WRN] Health check update: Degraded data redundancy: 249/1350 objects degraded (18.444%), 40 pgs degraded (PG_DEGRADED) 2021-03-24T21:59:10.594 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:59:10.595 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[19936]: 03-24T21:59:10.203909+0000 2021-03-24T21:59:10.595 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[19936]: mon.a (mon.0) 1157 : audit [INF] 2021-03-24T21:59:10.595 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[19936]: from='client.35153 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "42"}]': finished 2021-03-24T21:59:10.595 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[19936]: cluster 2021-03-24T21:59 2021-03-24T21:59:10.596 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[19936]: :10.204012+0000 mon.a (mon.0) 2021-03-24T21:59:10.596 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[19936]: 1158 : cluster [DBG] osdmap e380: 8 total, 7 up, 6 in 2021-03-24T21:59:10.597 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: cluster 2021-03-24T21:59:10.597 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 2021-03-24T21:59:10.597 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 2021 2021-03-24T21:59:10.597 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: - 2021-03-24T21:59:10.598 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 03 2021-03-24T21:59:10.598 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: - 2021-03-24T21:59:10.598 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 24 2021-03-24T21:59:10.598 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: T 2021-03-24T21:59:10.599 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 21 2021-03-24T21:59:10.599 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: : 2021-03-24T21:59:10.599 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 59 2021-03-24T21:59:10.599 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: : 2021-03-24T21:59:10.600 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 08 2021-03-24T21:59:10.600 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: . 2021-03-24T21:59:10.600 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 818758 2021-03-24T21:59:10.600 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: +0000 2021-03-24T21:59:10.600 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 2021-03-24T21:59:10.601 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: mgr.y 2021-03-24T21:59:10.601 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: ( 2021-03-24T21:59:10.601 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: mgr 2021-03-24T21:59:10.601 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: . 2021-03-24T21:59:10.602 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 14144 2021-03-24T21:59:10.602 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: ) 2021-03-24T21:59:10.602 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 459 2021-03-24T21:59:10.602 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: : 2021-03-24T21:59:10.603 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: cluster 2021-03-24T21:59:10.603 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 2021-03-24T21:59:10.603 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: [DBG] 2021-03-24T21:59:10.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 2021-03-24T21:59:10.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: pgmap v730: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 292 MiB data, 766 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 556 KiB/s wr, 2 op/s; 249/1350 objects degraded (18.444%) 2021-03-24T21:59:10.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 2021-03-24T21:59:10.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: cluster 2021-03-24T21:59:10.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 2021-03-24T21:59:10.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 2021 2021-03-24T21:59:10.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: - 2021-03-24T21:59:10.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 03 2021-03-24T21:59:10.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: - 2021-03-24T21:59:10.606 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 24 2021-03-24T21:59:10.606 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: T 2021-03-24T21:59:10.606 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 21 2021-03-24T21:59:10.606 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: : 2021-03-24T21:59:10.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 59 2021-03-24T21:59:10.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: : 2021-03-24T21:59:10.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 08 2021-03-24T21:59:10.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: . 2021-03-24T21:59:10.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 942239 2021-03-24T21:59:10.608 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: +0000 2021-03-24T21:59:10.608 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 2021-03-24T21:59:10.608 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: osd.7 2021-03-24T21:59:10.608 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: ( 2021-03-24T21:59:10.609 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: osd 2021-03-24T21:59:10.609 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: . 2021-03-24T21:59:10.609 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 7 2021-03-24T21:59:10.609 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: ) 2021-03-24T21:59:10.609 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 84 2021-03-24T21:59:10.610 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: : 2021-03-24T21:59:10.610 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: cluster 2021-03-24T21:59:10.610 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 2021-03-24T21:59:10.610 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: [DBG] 2021-03-24T21:59:10.611 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 2021-03-24T21:59:10.611 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 5.1a scrub ok 2021-03-24T21:59:10.611 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 2021-03-24T21:59:10.611 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: cluster 2021-03-24T21:59:10.611 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 2021-03-24T21:59:10.612 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 2021 2021-03-24T21:59:10.612 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: - 2021-03-24T21:59:10.612 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 03 2021-03-24T21:59:10.612 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: - 2021-03-24T21:59:10.613 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 24 2021-03-24T21:59:10.613 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: T 2021-03-24T21:59:10.613 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 21 2021-03-24T21:59:10.613 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: : 2021-03-24T21:59:10.613 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 59 2021-03-24T21:59:10.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: : 2021-03-24T21:59:10.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 09 2021-03-24T21:59:10.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: . 2021-03-24T21:59:10.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 287158 2021-03-24T21:59:10.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: +0000 2021-03-24T21:59:10.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 2021-03-24T21:59:10.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: mon.a 2021-03-24T21:59:10.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: ( 2021-03-24T21:59:10.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: mon 2021-03-24T21:59:10.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: . 2021-03-24T21:59:10.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 0 2021-03-24T21:59:10.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: ) 2021-03-24T21:59:10.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 1154 2021-03-24T21:59:10.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: : 2021-03-24T21:59:10.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: cluster 2021-03-24T21:59:10.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 2021-03-24T21:59:10.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: [DBG] 2021-03-24T21:59:10.618 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 2021-03-24T21:59:10.618 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: osdmap e379: 8 total, 7 up, 6 in 2021-03-24T21:59:10.618 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 2021-03-24T21:59:10.619 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: audit 2021-03-24T21:59:10.619 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 2021-03-24T21:59:10.619 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 2021 2021-03-24T21:59:10.619 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: - 2021-03-24T21:59:10.620 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 03 2021-03-24T21:59:10.620 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: - 2021-03-24T21:59:10.620 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 24 2021-03-24T21:59:10.620 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: T 2021-03-24T21:59:10.620 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 21 2021-03-24T21:59:10.621 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: : 2021-03-24T21:59:10.621 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 59 2021-03-24T21:59:10.621 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: : 2021-03-24T21:59:10.621 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 09 2021-03-24T21:59:10.622 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: . 2021-03-24T21:59:10.622 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 981164 2021-03-24T21:59:10.622 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: +0000 2021-03-24T21:59:10.622 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 2021-03-24T21:59:10.623 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: mon.c 2021-03-24T21:59:10.623 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: ( 2021-03-24T21:59:10.623 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: mon 2021-03-24T21:59:10.623 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: . 2021-03-24T21:59:10.623 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 2 2021-03-24T21:59:10.624 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: ) 2021-03-24T21:59:10.624 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 60 2021-03-24T21:59:10.624 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: : 2021-03-24T21:59:10.625 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: audit 2021-03-24T21:59:10.625 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 2021-03-24T21:59:10.625 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: [INF] 2021-03-24T21:59:10.625 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 2021-03-24T21:59:10.626 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: from='client.? v1:172.21.15.125:0/1407951140' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "42"}]: dispatch 2021-03-24T21:59:10.626 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 2021-03-24T21:59:10.626 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: audit 2021-03-24T21:59:10.626 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 2021-03-24T21:59:10.626 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 2021 2021-03-24T21:59:10.627 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: - 2021-03-24T21:59:10.627 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 03 2021-03-24T21:59:10.627 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: - 2021-03-24T21:59:10.627 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 24 2021-03-24T21:59:10.628 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: T 2021-03-24T21:59:10.628 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 21 2021-03-24T21:59:10.628 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: : 2021-03-24T21:59:10.628 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 59 2021-03-24T21:59:10.628 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: : 2021-03-24T21:59:10.629 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 09 2021-03-24T21:59:10.629 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: . 2021-03-24T21:59:10.629 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 981570 2021-03-24T21:59:10.629 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: +0000 2021-03-24T21:59:10.629 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 2021-03-24T21:59:10.630 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: mon.a 2021-03-24T21:59:10.630 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: ( 2021-03-24T21:59:10.630 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: mon 2021-03-24T21:59:10.630 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: . 2021-03-24T21:59:10.631 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 0 2021-03-24T21:59:10.631 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: ) 2021-03-24T21:59:10.631 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 1155 2021-03-24T21:59:10.631 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: : 2021-03-24T21:59:10.631 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: audit 2021-03-24T21:59:10.632 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 2021-03-24T21:59:10.632 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: [INF] 2021-03-24T21:59:10.632 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 2021-03-24T21:59:10.632 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: from='client.35153 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "42"}]: dispatch 2021-03-24T21:59:10.632 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 2021-03-24T21:59:10.633 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: cluster 2021-03-24T21:59:10.633 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 2021-03-24T21:59:10.633 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 2021 2021-03-24T21:59:10.633 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: - 2021-03-24T21:59:10.634 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 03 2021-03-24T21:59:10.634 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: - 2021-03-24T21:59:10.634 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 24 2021-03-24T21:59:10.634 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: T 2021-03-24T21:59:10.634 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 21 2021-03-24T21:59:10.635 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: : 2021-03-24T21:59:10.635 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 59 2021-03-24T21:59:10.635 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: : 2021-03-24T21:59:10.636 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 10 2021-03-24T21:59:10.636 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: . 2021-03-24T21:59:10.636 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 195315 2021-03-24T21:59:10.636 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: +0000 2021-03-24T21:59:10.637 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 2021-03-24T21:59:10.637 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: mon.a 2021-03-24T21:59:10.637 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: ( 2021-03-24T21:59:10.637 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: mon 2021-03-24T21:59:10.637 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: . 2021-03-24T21:59:10.638 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 0 2021-03-24T21:59:10.638 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: ) 2021-03-24T21:59:10.638 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 1156 2021-03-24T21:59:10.638 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: : 2021-03-24T21:59:10.639 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: cluster 2021-03-24T21:59:10.639 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 2021-03-24T21:59:10.639 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: [WRN] 2021-03-24T21:59:10.639 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 2021-03-24T21:59:10.640 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: Health check update: Degraded data redundancy: 249/1350 objects degraded (18.444%), 40 pgs degraded (PG_DEGRADED) 2021-03-24T21:59:10.640 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 2021-03-24T21:59:10.640 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: audit 2021-03-24T21:59:10.640 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 2021-03-24T21:59:10.640 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 2021 2021-03-24T21:59:10.641 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: - 2021-03-24T21:59:10.641 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 03 2021-03-24T21:59:10.641 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: - 2021-03-24T21:59:10.641 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 24 2021-03-24T21:59:10.642 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: T 2021-03-24T21:59:10.642 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 21 2021-03-24T21:59:10.642 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: : 2021-03-24T21:59:10.642 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 59 2021-03-24T21:59:10.643 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: : 2021-03-24T21:59:10.643 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 10 2021-03-24T21:59:10.643 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: . 2021-03-24T21:59:10.643 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 203909 2021-03-24T21:59:10.643 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: +0000 2021-03-24T21:59:10.644 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 2021-03-24T21:59:10.644 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: mon.a 2021-03-24T21:59:10.644 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: ( 2021-03-24T21:59:10.644 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: mon 2021-03-24T21:59:10.644 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: . 2021-03-24T21:59:10.645 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 0 2021-03-24T21:59:10.645 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: ) 2021-03-24T21:59:10.645 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 1157 2021-03-24T21:59:10.646 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: : 2021-03-24T21:59:10.646 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: audit 2021-03-24T21:59:10.646 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 2021-03-24T21:59:10.646 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: [INF] 2021-03-24T21:59:10.646 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 2021-03-24T21:59:10.647 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: from='client.35153 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "42"}]': finished 2021-03-24T21:59:10.647 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 2021-03-24T21:59:10.647 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: cluster 2021-03-24T21:59:10.647 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 2021-03-24T21:59:10.647 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 2021 2021-03-24T21:59:10.648 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: - 2021-03-24T21:59:10.648 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 03 2021-03-24T21:59:10.648 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: - 2021-03-24T21:59:10.648 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 24 2021-03-24T21:59:10.649 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: T 2021-03-24T21:59:10.649 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 21 2021-03-24T21:59:10.649 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: : 2021-03-24T21:59:10.649 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 59 2021-03-24T21:59:10.649 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: : 2021-03-24T21:59:10.650 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 10 2021-03-24T21:59:10.650 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: . 2021-03-24T21:59:10.650 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 204012 2021-03-24T21:59:10.650 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: +0000 2021-03-24T21:59:10.651 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 2021-03-24T21:59:10.651 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: mon.a 2021-03-24T21:59:10.651 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: ( 2021-03-24T21:59:10.651 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: mon 2021-03-24T21:59:10.651 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: . 2021-03-24T21:59:10.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 0 2021-03-24T21:59:10.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: ) 2021-03-24T21:59:10.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 1158 2021-03-24T21:59:10.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: : 2021-03-24T21:59:10.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: cluster 2021-03-24T21:59:10.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 2021-03-24T21:59:10.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: [DBG] 2021-03-24T21:59:10.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 2021-03-24T21:59:10.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: osdmap e380: 8 total, 7 up, 6 in 2021-03-24T21:59:10.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:10 smithi125 conmon[24927]: 2021-03-24T21:59:10.655 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:10.656 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:10 smithi142 conmon[20125]: cluster 2021-03-24T21:59:08.818758+0000 mgr.y (mgr. 2021-03-24T21:59:10.657 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:10 smithi142 conmon[20125]: 14144) 459 : cluster [DBG] pgmap v730: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 292 MiB data, 766 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 556 KiB/s wr, 2 op/s; 249/1350 objects degraded (18.444%) 2021-03-24T21:59:10.657 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:10 smithi142 conmon[20125]: cluster 2021-03-24T21:59:08.942239+0000 osd.7 (osd.7) 84 : cluster [DBG] 5.1a scrub ok 2021-03-24T21:59:10.657 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:10 smithi142 conmon[20125]: cluster 2021-03-24T21 2021-03-24T21:59:10.657 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:10 smithi142 conmon[20125]: :59:09.287158+0000 mon.a (mon.0) 2021-03-24T21:59:10.658 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:10 smithi142 conmon[20125]: 1154 : cluster [DBG] osdmap e379: 8 total, 7 up, 6 in 2021-03-24T21:59:10.658 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:10 smithi142 conmon[20125]: audit 2021-03 2021-03-24T21:59:10.658 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:10 smithi142 conmon[20125]: -24T21:59:09.981164+0000 mon.c ( 2021-03-24T21:59:10.658 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:10 smithi142 conmon[20125]: mon.2) 60 : audit [INF] from='client.? v1:172.21.15.125:0/1407951140' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "42"}]: dispatch 2021-03-24T21:59:10.659 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:10 smithi142 conmon[20125]: audit 2021 2021-03-24T21:59:10.659 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:10 smithi142 conmon[20125]: -03-24T21:59:09.981570+0000 2021-03-24T21:59:10.659 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:10 smithi142 conmon[20125]: mon.a (mon.0) 1155 : audit [INF] 2021-03-24T21:59:10.659 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:10 smithi142 conmon[20125]: from='client.35153 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "42"}]: dispatch 2021-03-24T21:59:10.660 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:10 smithi142 conmon[20125]: cluster 2021-03-24T21:59 2021-03-24T21:59:10.660 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:10 smithi142 conmon[20125]: :10.195315+0000 mon.a (mon.0) 2021-03-24T21:59:10.660 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:10 smithi142 conmon[20125]: 1156 : cluster [WRN] Health check update: Degraded data redundancy: 249/1350 objects degraded (18.444%), 40 pgs degraded (PG_DEGRADED) 2021-03-24T21:59:10.660 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:10 smithi142 conmon[20125]: audit 2021-03-24 2021-03-24T21:59:10.660 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:10 smithi142 conmon[20125]: T21:59:10.203909+0000 mon.a ( 2021-03-24T21:59:10.661 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:10 smithi142 conmon[20125]: mon.0) 1157 : audit [INF] from='client.35153 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "42"}]': finished 2021-03-24T21:59:10.661 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:10 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:59:10.661 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:10 smithi142 conmon[20125]: 03-24T21:59:10.204012+0000 2021-03-24T21:59:10.661 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:10 smithi142 conmon[20125]: mon.a (mon.0) 1158 : cluster [DBG] 2021-03-24T21:59:10.662 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:10 smithi142 conmon[20125]: osdmap e380: 8 total, 7 up, 6 in 2021-03-24T21:59:10.718 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:10.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:10.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:59:10.825 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:10.886 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:10.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:10.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:59:10.993 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:11.052 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:11.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:11.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:59:11.069 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd set noscrub 2021-03-24T21:59:11.159 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:11.207 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:11.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:11.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:59:11.237 INFO:tasks.rados.rados.0.smithi125.stdout:1388: done (1 left) 2021-03-24T21:59:11.237 INFO:tasks.rados.rados.0.smithi125.stdout:1389: write oid 1 current snap is 145 2021-03-24T21:59:11.237 INFO:tasks.rados.rados.0.smithi125.stdout:1389: seq_num 497 ranges {753079=561698,2033433=319646} 2021-03-24T21:59:11.238 INFO:tasks.rados.rados.0.smithi125.stdout:1389: writing smithi12556470-1 from 753079 to 1314777 tid 1 2021-03-24T21:59:11.249 INFO:tasks.rados.rados.0.smithi125.stdout:1389: writing smithi12556470-1 from 2033433 to 2353079 tid 2 2021-03-24T21:59:11.249 INFO:tasks.rados.rados.0.smithi125.stdout:1387: finishing write tid 1 to smithi12556470-19 2021-03-24T21:59:11.250 INFO:tasks.rados.rados.0.smithi125.stdout:1387: finishing write tid 2 to smithi12556470-19 2021-03-24T21:59:11.250 INFO:tasks.rados.rados.0.smithi125.stdout:1387: finishing write tid 3 to smithi12556470-19 2021-03-24T21:59:11.250 INFO:tasks.rados.rados.0.smithi125.stdout:1387: finishing write tid 4 to smithi12556470-19 2021-03-24T21:59:11.250 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 19 v 300 (ObjNum 496 snap 145 seq_num 496) dirty exists 2021-03-24T21:59:11.250 INFO:tasks.rados.rados.0.smithi125.stdout:1387: left oid 19 (ObjNum 496 snap 145 seq_num 496) 2021-03-24T21:59:11.251 INFO:tasks.rados.rados.0.smithi125.stdout:1389: finishing write tid 1 to smithi12556470-1 2021-03-24T21:59:11.251 INFO:tasks.rados.rados.0.smithi125.stdout:1387: done (1 left) 2021-03-24T21:59:11.251 INFO:tasks.rados.rados.0.smithi125.stdout:1390: snap_create 2021-03-24T21:59:11.260 INFO:tasks.rados.rados.0.smithi125.stdout:1389: finishing write tid 2 to smithi12556470-1 2021-03-24T21:59:11.264 INFO:tasks.rados.rados.0.smithi125.stdout:1389: finishing write tid 3 to smithi12556470-1 2021-03-24T21:59:11.264 INFO:tasks.rados.rados.0.smithi125.stdout:1389: finishing write tid 4 to smithi12556470-1 2021-03-24T21:59:11.264 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 1 v 446 (ObjNum 497 snap 145 seq_num 497) dirty exists 2021-03-24T21:59:11.265 INFO:tasks.rados.rados.0.smithi125.stdout:1389: left oid 1 (ObjNum 497 snap 145 seq_num 497) 2021-03-24T21:59:11.313 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:11.317 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: cluster 2021-03-24T21:59:11.317 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: 2021-03-24T21:59:11.318 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: 2021 2021-03-24T21:59:11.318 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: - 2021-03-24T21:59:11.318 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: 03 2021-03-24T21:59:11.318 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: - 2021-03-24T21:59:11.319 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: 24 2021-03-24T21:59:11.319 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: T 2021-03-24T21:59:11.319 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: 21 2021-03-24T21:59:11.319 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: : 2021-03-24T21:59:11.320 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: 59 2021-03-24T21:59:11.320 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: : 2021-03-24T21:59:11.320 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: 09 2021-03-24T21:59:11.320 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: . 2021-03-24T21:59:11.321 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: 930378 2021-03-24T21:59:11.321 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: +0000 2021-03-24T21:59:11.321 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: 2021-03-24T21:59:11.321 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: osd.7 2021-03-24T21:59:11.321 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: ( 2021-03-24T21:59:11.322 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: osd 2021-03-24T21:59:11.322 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: . 2021-03-24T21:59:11.322 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: 7 2021-03-24T21:59:11.322 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: ) 2021-03-24T21:59:11.323 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: 85 2021-03-24T21:59:11.323 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: : 2021-03-24T21:59:11.323 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: cluster 2021-03-24T21:59:11.323 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: 2021-03-24T21:59:11.323 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: [DBG] 2021-03-24T21:59:11.324 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: 2021-03-24T21:59:11.324 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: 7.5 scrub ok 2021-03-24T21:59:11.324 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: 2021-03-24T21:59:11.324 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: audit 2021-03-24T21:59:11.325 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: 2021-03-24T21:59:11.325 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: 2021 2021-03-24T21:59:11.325 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: - 2021-03-24T21:59:11.325 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: 03 2021-03-24T21:59:11.326 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: - 2021-03-24T21:59:11.326 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: 24 2021-03-24T21:59:11.326 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: T 2021-03-24T21:59:11.326 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: 21 2021-03-24T21:59:11.327 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: : 2021-03-24T21:59:11.327 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: 59 2021-03-24T21:59:11.327 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: : 2021-03-24T21:59:11.327 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: 10 2021-03-24T21:59:11.327 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: . 2021-03-24T21:59:11.328 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: 821874 2021-03-24T21:59:11.328 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: +0000 2021-03-24T21:59:11.328 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: 2021-03-24T21:59:11.328 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: mon.a 2021-03-24T21:59:11.329 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: ( 2021-03-24T21:59:11.329 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: mon 2021-03-24T21:59:11.329 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: . 2021-03-24T21:59:11.330 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: 0 2021-03-24T21:59:11.330 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: ) 2021-03-24T21:59:11.330 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: 1159 2021-03-24T21:59:11.330 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: : 2021-03-24T21:59:11.331 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: audit 2021-03-24T21:59:11.331 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: 2021-03-24T21:59:11.331 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: [INF] 2021-03-24T21:59:11.331 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: 2021-03-24T21:59:11.331 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "42"}]: dispatch 2021-03-24T21:59:11.332 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: 2021-03-24T21:59:11.332 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: audit 2021-03-24T21:59:11.332 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: 2021-03-24T21:59:11.332 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: 2021 2021-03-24T21:59:11.333 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: - 2021-03-24T21:59:11.334 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:11 smithi125 conmon[24927]: cluster 2021-03-24T21:59:09.930378+0000 2021-03-24T21:59:11.334 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:11 smithi125 conmon[24927]: osd.7 (osd.7) 85 : cluster [DBG] 7.5 scrub ok 2021-03-24T21:59:11.334 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:11 smithi125 conmon[24927]: audit 2021-03-24T21:59:10.821874+0000 mon.a (mon.0) 1159 2021-03-24T21:59:11.334 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:11 smithi125 conmon[24927]: : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "42"}]: dispatch 2021-03-24T21:59:11.335 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:11 smithi125 conmon[24927]: audit 2021-03-24T21:59: 2021-03-24T21:59:11.335 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:11 smithi125 conmon[24927]: 11.222631+0000 mon.a (mon.0) 1160 : audit 2021-03-24T21:59:11.335 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:11 smithi125 conmon[24927]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "42"}]': finished 2021-03-24T21:59:11.335 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:11 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:59:11.336 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:11 smithi125 conmon[24927]: -24T21:59:11.222746+0000 mon.a (mon 2021-03-24T21:59:11.336 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:11 smithi125 conmon[24927]: .0) 1161 : cluster [DBG] osdmap e381: 8 total, 7 up, 6 in 2021-03-24T21:59:11.337 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:11 smithi125 conmon[19936]: cluster 2021-03-24T21:59:09.930378+0000 2021-03-24T21:59:11.337 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:11 smithi125 conmon[19936]: osd.7 (osd.7) 85 : cluster [DBG] 7.5 scrub ok 2021-03-24T21:59:11.337 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:11 smithi125 conmon[19936]: audit 2021-03-24T21:59:10.821874+0000 mon.a (mon.0) 1159 2021-03-24T21:59:11.338 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:11 smithi125 conmon[19936]: : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "42"}]: dispatch 2021-03-24T21:59:11.338 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:11 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:59:11.338 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:11 smithi125 conmon[19936]: 03-24T21:59:11.222631+0000 2021-03-24T21:59:11.338 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:11 smithi125 conmon[19936]: mon.a (mon.0) 1160 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "42"}]': finished 2021-03-24T21:59:11.338 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:11 smithi125 conmon[19936]: 2021-03-24T21:59:11.339 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:11 smithi125 conmon[19936]: cluster 2021-03-24T21:59: 2021-03-24T21:59:11.339 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:11 smithi125 conmon[19936]: 11.222746+0000 mon.a (mon.0) 1161 : 2021-03-24T21:59:11.339 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:11 smithi125 conmon[19936]: cluster [DBG] osdmap e381: 8 total, 7 up, 6 in 2021-03-24T21:59:11.441 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:11.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:11.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:59:11.545 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:11.596 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:11.600 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: 03 2021-03-24T21:59:11.600 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: - 2021-03-24T21:59:11.600 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: 24 2021-03-24T21:59:11.600 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: T 2021-03-24T21:59:11.601 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: 21 2021-03-24T21:59:11.601 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: :59:11.222631+0000 mon.a (mon 2021-03-24T21:59:11.601 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: .0) 1160 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "42"}]': finished 2021-03-24T21:59:11.601 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:59:11.602 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: -24T21:59:11.222746+0000 mon.a (mon.0) 2021-03-24T21:59:11.602 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:11 smithi142 conmon[20125]: 1161 : cluster [DBG] osdmap e381: 8 total, 7 up, 6 in 2021-03-24T21:59:11.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:11.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:11.704 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:11.761 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:11.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:11.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:59:11.866 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:11.932 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:11.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:11.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:11.995 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.2 dump_blocked_ops 2021-03-24T21:59:12.038 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:12.098 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:12.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:12.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:59:12.203 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:12.227 INFO:tasks.rados.rados.0.smithi125.stdout:1389: done (1 left) 2021-03-24T21:59:12.228 INFO:tasks.rados.rados.0.smithi125.stdout:1390: done (0 left) 2021-03-24T21:59:12.228 INFO:tasks.rados.rados.0.smithi125.stdout:1391: delete oid 26 current snap is 146 2021-03-24T21:59:12.281 INFO:tasks.rados.rados.0.smithi125.stdout:1391: done (0 left) 2021-03-24T21:59:12.281 INFO:tasks.rados.rados.0.smithi125.stdout:1392: delete oid 5 current snap is 146 2021-03-24T21:59:12.284 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:12.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:12.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:12.297 INFO:tasks.rados.rados.0.smithi125.stdout:1392: done (0 left) 2021-03-24T21:59:12.297 INFO:tasks.rados.rados.0.smithi125.stdout:1393: rollback oid 35 current snap is 146 2021-03-24T21:59:12.297 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 35 to 139 2021-03-24T21:59:12.298 INFO:tasks.rados.rados.0.smithi125.stdout:1394: write (excl) oid 21 current snap is 146 2021-03-24T21:59:12.298 INFO:tasks.rados.rados.0.smithi125.stdout:1394: seq_num 500 ranges {403500=787445,1692526=473837} 2021-03-24T21:59:12.301 INFO:tasks.rados.rados.0.smithi125.stdout:1394: writing smithi12556470-21 from 403500 to 1190945 tid 1 2021-03-24T21:59:12.318 INFO:tasks.rados.rados.0.smithi125.stdout:1394: writing smithi12556470-21 from 1692526 to 2166363 tid 2 2021-03-24T21:59:12.318 INFO:tasks.rados.rados.0.smithi125.stdout:1393: finishing rollback tid 1 to smithi12556470-35 2021-03-24T21:59:12.318 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 35 v 0 (ObjNum 1116508576 snap 21916 seq_num 3154118864) dirty dne 2021-03-24T21:59:12.319 INFO:tasks.rados.rados.0.smithi125.stdout:1394: finishing write tid 1 to smithi12556470-21 2021-03-24T21:59:12.319 INFO:tasks.rados.rados.0.smithi125.stdout:1393: done (1 left) 2021-03-24T21:59:12.319 INFO:tasks.rados.rados.0.smithi125.stdout:1395: copy_from oid 16 from oid 18 current snap is 146 2021-03-24T21:59:12.319 INFO:tasks.rados.rados.0.smithi125.stdout:1396: write oid 28 current snap is 146 2021-03-24T21:59:12.319 INFO:tasks.rados.rados.0.smithi125.stdout:1396: seq_num 501 ranges {420307=462694,1351008=617495} 2021-03-24T21:59:12.323 INFO:tasks.rados.rados.0.smithi125.stdout:1396: writing smithi12556470-28 from 420307 to 883001 tid 1 2021-03-24T21:59:12.331 INFO:tasks.rados.rados.0.smithi125.stdout:1396: writing smithi12556470-28 from 1351008 to 1968503 tid 2 2021-03-24T21:59:12.332 INFO:tasks.rados.rados.0.smithi125.stdout:1396: finishing write tid 1 to smithi12556470-28 2021-03-24T21:59:12.332 INFO:tasks.rados.rados.0.smithi125.stdout:1397: delete oid 12 current snap is 146 2021-03-24T21:59:12.340 INFO:tasks.rados.rados.0.smithi125.stdout:1394: finishing write tid 2 to smithi12556470-21 2021-03-24T21:59:12.340 INFO:tasks.rados.rados.0.smithi125.stdout:1394: finishing write tid 3 to smithi12556470-21 2021-03-24T21:59:12.340 INFO:tasks.rados.rados.0.smithi125.stdout:1394: finishing write tid 4 to smithi12556470-21 2021-03-24T21:59:12.341 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 21 v 360 (ObjNum 500 snap 146 seq_num 500) dirty exists 2021-03-24T21:59:12.341 INFO:tasks.rados.rados.0.smithi125.stdout:1394: left oid 21 (ObjNum 500 snap 146 seq_num 500) 2021-03-24T21:59:12.342 INFO:tasks.rados.rados.0.smithi125.stdout:1396: finishing write tid 2 to smithi12556470-28 2021-03-24T21:59:12.342 INFO:tasks.rados.rados.0.smithi125.stdout:1396: finishing write tid 3 to smithi12556470-28 2021-03-24T21:59:12.343 INFO:tasks.rados.rados.0.smithi125.stdout:1396: finishing write tid 4 to smithi12556470-28 2021-03-24T21:59:12.343 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 28 v 303 (ObjNum 501 snap 146 seq_num 501) dirty exists 2021-03-24T21:59:12.343 INFO:tasks.rados.rados.0.smithi125.stdout:1396: left oid 28 (ObjNum 501 snap 146 seq_num 501) 2021-03-24T21:59:12.349 INFO:tasks.rados.rados.0.smithi125.stdout:1395: finishing copy_from to smithi12556470-16 2021-03-24T21:59:12.349 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 16 v 103 (ObjNum 384 snap 99 seq_num 384) dirty exists 2021-03-24T21:59:12.349 INFO:tasks.rados.rados.0.smithi125.stdout:1395: finishing copy_from racing read to smithi12556470-16 2021-03-24T21:59:12.354 INFO:tasks.rados.rados.0.smithi125.stdout:1394: done (3 left) 2021-03-24T21:59:12.354 INFO:tasks.rados.rados.0.smithi125.stdout:1395: done (2 left) 2021-03-24T21:59:12.354 INFO:tasks.rados.rados.0.smithi125.stdout:1396: done (1 left) 2021-03-24T21:59:12.354 INFO:tasks.rados.rados.0.smithi125.stdout:1397: done (0 left) 2021-03-24T21:59:12.355 INFO:tasks.rados.rados.0.smithi125.stdout:1398: read oid 14 snap -1 2021-03-24T21:59:12.355 INFO:tasks.rados.rados.0.smithi125.stdout:1398: expect (ObjNum 455 snap 128 seq_num 455) 2021-03-24T21:59:12.355 INFO:tasks.rados.rados.0.smithi125.stdout:1399: write (excl) oid 34 current snap is 146 2021-03-24T21:59:12.355 INFO:tasks.rados.rados.0.smithi125.stdout:1399: seq_num 503 ranges {453921=729545,1584325=621164} 2021-03-24T21:59:12.365 INFO:tasks.rados.rados.0.smithi125.stdout:1399: writing smithi12556470-34 from 453921 to 1183466 tid 1 2021-03-24T21:59:12.378 INFO:tasks.rados.rados.0.smithi125.stdout:1399: writing smithi12556470-34 from 1584325 to 2205489 tid 2 2021-03-24T21:59:12.378 INFO:tasks.rados.rados.0.smithi125.stdout:1399: finishing write tid 1 to smithi12556470-34 2021-03-24T21:59:12.378 INFO:tasks.rados.rados.0.smithi125.stdout:1400: read oid 15 snap -1 2021-03-24T21:59:12.378 INFO:tasks.rados.rados.0.smithi125.stdout:1400: expect (ObjNum 384 snap 99 seq_num 384) 2021-03-24T21:59:12.379 INFO:tasks.rados.rados.0.smithi125.stdout:1401: write (excl) oid 50 current snap is 146 2021-03-24T21:59:12.379 INFO:tasks.rados.rados.0.smithi125.stdout:1401: seq_num 504 ranges {470728=404794,1526454=481175} 2021-03-24T21:59:12.386 INFO:tasks.rados.rados.0.smithi125.stdout:1401: writing smithi12556470-50 from 470728 to 875522 tid 1 2021-03-24T21:59:12.393 INFO:tasks.rados.rados.0.smithi125.stdout:1401: writing smithi12556470-50 from 1526454 to 2007629 tid 2 2021-03-24T21:59:12.394 INFO:tasks.rados.rados.0.smithi125.stdout:1400: expect (ObjNum 384 snap 99 seq_num 384) 2021-03-24T21:59:12.396 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:12.470 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:12.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:12.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:59:12.577 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:12.585 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:12 smithi125 conmon[19936]: cluster 2021-03-24T21:59:10.819750+0000 mgr.y (mgr. 2021-03-24T21:59:12.585 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:12 smithi125 conmon[19936]: 14144) 460 : cluster [DBG] pgmap v733: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 317 MiB data, 797 MiB used, 536 GiB / 536 GiB avail; 255/1383 objects degraded (18.438%) 2021-03-24T21:59:12.585 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:12 smithi125 conmon[19936]: cluster 2021-03-24T21:59:10.882032+0000 osd.0 (osd.0) 88 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:59:12.586 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:12 smithi125 conmon[19936]: cluster 2021-03-24T21:59:10. 2021-03-24T21:59:12.586 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:12 smithi125 conmon[19936]: 883007+0000 osd.0 (osd.0) 89 : cluster 2021-03-24T21:59:12.586 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:12 smithi125 conmon[19936]: [DBG] purged_snaps scrub ok 2021-03-24T21:59:12.586 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:12 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:59:12.587 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:12 smithi125 conmon[19936]: :59:12.225699+0000 mon.a (mon.0 2021-03-24T21:59:12.587 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:12 smithi125 conmon[19936]: ) 1162 : cluster [DBG] osdmap e382: 8 total, 7 up, 6 in 2021-03-24T21:59:12.587 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:12 smithi125 conmon[24927]: cluster 2021-03-24T21:59:10.819750+0000 2021-03-24T21:59:12.588 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:12 smithi125 conmon[24927]: mgr.y (mgr.14144) 460 : cluster [DBG] pgmap v733: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 317 MiB data, 797 MiB used, 536 GiB / 536 GiB avail; 255/1383 objects degraded (18.438%) 2021-03-24T21:59:12.588 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:12 smithi125 conmon[24927]: cluster 2021-03-24T21:59:10. 2021-03-24T21:59:12.588 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:12 smithi125 conmon[24927]: 882032+0000 osd.0 (osd.0) 88 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:59:12.589 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:12 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:59:12.589 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:12 smithi125 conmon[24927]: :59:10.883007+0000 osd.0 (osd. 2021-03-24T21:59:12.589 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:12 smithi125 conmon[24927]: 0) 89 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:59:12.589 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:12 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:59:12.590 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:12 smithi125 conmon[24927]: 24T21:59:12.225699+0000 mon.a 2021-03-24T21:59:12.590 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:12 smithi125 conmon[24927]: (mon.0) 1162 : cluster [DBG] osdmap e382: 8 total, 7 up, 6 in 2021-03-24T21:59:12.590 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:12 smithi125 conmon[24927]: 2021-03-24T21:59:12.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: cluster 2021-03-24T21:59:12.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 2021-03-24T21:59:12.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 2021 2021-03-24T21:59:12.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: - 2021-03-24T21:59:12.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 03 2021-03-24T21:59:12.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: - 2021-03-24T21:59:12.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 24 2021-03-24T21:59:12.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: T 2021-03-24T21:59:12.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 21 2021-03-24T21:59:12.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: : 2021-03-24T21:59:12.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 59 2021-03-24T21:59:12.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: : 2021-03-24T21:59:12.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 10 2021-03-24T21:59:12.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: . 2021-03-24T21:59:12.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 819750 2021-03-24T21:59:12.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: +0000 2021-03-24T21:59:12.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 2021-03-24T21:59:12.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: mgr.y 2021-03-24T21:59:12.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: ( 2021-03-24T21:59:12.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: mgr 2021-03-24T21:59:12.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: . 2021-03-24T21:59:12.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 14144 2021-03-24T21:59:12.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: ) 2021-03-24T21:59:12.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 460 2021-03-24T21:59:12.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: : 2021-03-24T21:59:12.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: cluster 2021-03-24T21:59:12.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 2021-03-24T21:59:12.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: [DBG] 2021-03-24T21:59:12.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 2021-03-24T21:59:12.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: pgmap v733: 168 pgs: 50 active+undersized, 40 active+undersized+degraded, 78 active+clean; 317 MiB data, 797 MiB used, 536 GiB / 536 GiB avail; 255/1383 objects degraded (18.438%) 2021-03-24T21:59:12.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 2021-03-24T21:59:12.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: cluster 2021-03-24T21:59:12.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 2021-03-24T21:59:12.625 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 2021 2021-03-24T21:59:12.625 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: - 2021-03-24T21:59:12.625 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 03 2021-03-24T21:59:12.625 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: - 2021-03-24T21:59:12.626 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 24 2021-03-24T21:59:12.626 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: T 2021-03-24T21:59:12.626 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 21 2021-03-24T21:59:12.626 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: : 2021-03-24T21:59:12.626 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 59 2021-03-24T21:59:12.627 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: : 2021-03-24T21:59:12.627 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 10 2021-03-24T21:59:12.627 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: . 2021-03-24T21:59:12.627 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 882032 2021-03-24T21:59:12.627 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: +0000 2021-03-24T21:59:12.628 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 2021-03-24T21:59:12.628 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: osd.0 2021-03-24T21:59:12.628 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: ( 2021-03-24T21:59:12.628 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: osd 2021-03-24T21:59:12.628 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: . 2021-03-24T21:59:12.629 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 0 2021-03-24T21:59:12.629 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: ) 2021-03-24T21:59:12.629 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 88 2021-03-24T21:59:12.629 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: : 2021-03-24T21:59:12.629 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: cluster 2021-03-24T21:59:12.630 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 2021-03-24T21:59:12.630 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: [DBG] 2021-03-24T21:59:12.630 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 2021-03-24T21:59:12.630 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: purged_snaps scrub starts 2021-03-24T21:59:12.630 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 2021-03-24T21:59:12.631 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: cluster 2021-03-24T21:59:12.631 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 2021-03-24T21:59:12.632 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 2021 2021-03-24T21:59:12.632 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: - 2021-03-24T21:59:12.632 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 03 2021-03-24T21:59:12.633 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: - 2021-03-24T21:59:12.633 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 24 2021-03-24T21:59:12.633 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: T 2021-03-24T21:59:12.633 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 21 2021-03-24T21:59:12.633 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: : 2021-03-24T21:59:12.634 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 59 2021-03-24T21:59:12.634 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: : 2021-03-24T21:59:12.634 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 10 2021-03-24T21:59:12.634 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: . 2021-03-24T21:59:12.635 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 883007 2021-03-24T21:59:12.635 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: +0000 2021-03-24T21:59:12.635 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 2021-03-24T21:59:12.635 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: osd.0 2021-03-24T21:59:12.635 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: ( 2021-03-24T21:59:12.636 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: osd 2021-03-24T21:59:12.636 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: . 2021-03-24T21:59:12.636 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 0 2021-03-24T21:59:12.636 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: ) 2021-03-24T21:59:12.636 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 89 2021-03-24T21:59:12.637 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: : 2021-03-24T21:59:12.637 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: cluster 2021-03-24T21:59:12.638 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 2021-03-24T21:59:12.638 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: [DBG] 2021-03-24T21:59:12.638 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 2021-03-24T21:59:12.639 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: purged_snaps scrub ok 2021-03-24T21:59:12.639 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 2021-03-24T21:59:12.640 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: cluster 2021-03-24T21:59:12.640 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 2021-03-24T21:59:12.640 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 2021 2021-03-24T21:59:12.641 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: - 2021-03-24T21:59:12.641 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 03 2021-03-24T21:59:12.642 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: - 2021-03-24T21:59:12.642 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 24 2021-03-24T21:59:12.643 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: T 2021-03-24T21:59:12.643 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 21 2021-03-24T21:59:12.643 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: : 2021-03-24T21:59:12.644 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 59 2021-03-24T21:59:12.644 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: : 2021-03-24T21:59:12.645 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 12 2021-03-24T21:59:12.645 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: . 2021-03-24T21:59:12.645 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 225699 2021-03-24T21:59:12.646 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: +0000 2021-03-24T21:59:12.646 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 2021-03-24T21:59:12.647 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: mon.a 2021-03-24T21:59:12.647 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: ( 2021-03-24T21:59:12.647 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: mon 2021-03-24T21:59:12.647 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: . 2021-03-24T21:59:12.648 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 0 2021-03-24T21:59:12.648 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: ) 2021-03-24T21:59:12.648 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 1162 2021-03-24T21:59:12.649 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: : 2021-03-24T21:59:12.649 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: cluster 2021-03-24T21:59:12.649 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 2021-03-24T21:59:12.650 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: [DBG] 2021-03-24T21:59:12.650 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 2021-03-24T21:59:12.650 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: osdmap e382: 8 total, 7 up, 6 in 2021-03-24T21:59:12.651 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:12 smithi142 conmon[20125]: 2021-03-24T21:59:12.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:12.652 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:12.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:12.688 INFO:tasks.rados.rados.0.smithi125.stdout:1399: finishing write tid 2 to smithi12556470-34 2021-03-24T21:59:12.688 INFO:tasks.rados.rados.0.smithi125.stdout:1399: finishing write tid 3 to smithi12556470-34 2021-03-24T21:59:12.688 INFO:tasks.rados.rados.0.smithi125.stdout:1399: finishing write tid 4 to smithi12556470-34 2021-03-24T21:59:12.689 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 34 v 334 (ObjNum 503 snap 146 seq_num 503) dirty exists 2021-03-24T21:59:12.689 INFO:tasks.rados.rados.0.smithi125.stdout:1399: left oid 34 (ObjNum 503 snap 146 seq_num 503) 2021-03-24T21:59:12.690 INFO:tasks.rados.rados.0.smithi125.stdout:1398: expect (ObjNum 455 snap 128 seq_num 455) 2021-03-24T21:59:12.754 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:12.784 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:59:12.798 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:59:12.814 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:12.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:12.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:59:12.866 INFO:tasks.rados.rados.0.smithi125.stdout:1401: finishing write tid 1 to smithi12556470-50 2021-03-24T21:59:12.866 INFO:tasks.rados.rados.0.smithi125.stdout:1401: finishing write tid 2 to smithi12556470-50 2021-03-24T21:59:12.866 INFO:tasks.rados.rados.0.smithi125.stdout:1398: done (3 left) 2021-03-24T21:59:12.867 INFO:tasks.rados.rados.0.smithi125.stdout:1399: done (2 left) 2021-03-24T21:59:12.867 INFO:tasks.rados.rados.0.smithi125.stdout:1400: done (1 left) 2021-03-24T21:59:12.867 INFO:tasks.rados.rados.0.smithi125.stdout:1402: rollback oid 33 current snap is 146 2021-03-24T21:59:12.867 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 33 to 116 2021-03-24T21:59:12.867 INFO:tasks.rados.rados.0.smithi125.stdout:1401: finishing write tid 3 to smithi12556470-50 2021-03-24T21:59:12.868 INFO:tasks.rados.rados.0.smithi125.stdout:1401: finishing write tid 4 to smithi12556470-50 2021-03-24T21:59:12.868 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 50 v 304 (ObjNum 504 snap 146 seq_num 504) dirty exists 2021-03-24T21:59:12.868 INFO:tasks.rados.rados.0.smithi125.stdout:1401: left oid 50 (ObjNum 504 snap 146 seq_num 504) 2021-03-24T21:59:12.868 INFO:tasks.rados.rados.0.smithi125.stdout:1401: done (1 left) 2021-03-24T21:59:12.868 INFO:tasks.rados.rados.0.smithi125.stdout:1403: write (excl) oid 20 current snap is 146 2021-03-24T21:59:12.869 INFO:tasks.rados.rados.0.smithi125.stdout:1403: seq_num 505 ranges {487535=480043,1584936=624833} 2021-03-24T21:59:12.871 INFO:tasks.rados.rados.0.smithi125.stdout:1403: writing smithi12556470-20 from 487535 to 967578 tid 1 2021-03-24T21:59:12.880 INFO:tasks.rados.rados.0.smithi125.stdout:1403: writing smithi12556470-20 from 1584936 to 2209769 tid 2 2021-03-24T21:59:12.881 INFO:tasks.rados.rados.0.smithi125.stdout:1404: read oid 1 snap -1 2021-03-24T21:59:12.881 INFO:tasks.rados.rados.0.smithi125.stdout:1404: expect (ObjNum 497 snap 145 seq_num 497) 2021-03-24T21:59:12.881 INFO:tasks.rados.rados.0.smithi125.stdout:1405: snap_create 2021-03-24T21:59:12.881 INFO:tasks.rados.rados.0.smithi125.stdout:1402: finishing rollback tid 0 to smithi12556470-33 2021-03-24T21:59:12.882 INFO:tasks.rados.rados.0.smithi125.stdout:1402: finishing rollback tid 1 to smithi12556470-33 2021-03-24T21:59:12.882 INFO:tasks.rados.rados.0.smithi125.stdout:1402: finishing rollback tid 2 to smithi12556470-33 2021-03-24T21:59:12.882 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 33 v 388 (ObjNum 298 snap 74 seq_num 298) dirty exists 2021-03-24T21:59:12.882 INFO:tasks.rados.rados.0.smithi125.stdout:1403: finishing write tid 1 to smithi12556470-20 2021-03-24T21:59:12.883 INFO:tasks.rados.rados.0.smithi125.stdout:1403: finishing write tid 2 to smithi12556470-20 2021-03-24T21:59:12.883 INFO:tasks.rados.rados.0.smithi125.stdout:1403: finishing write tid 3 to smithi12556470-20 2021-03-24T21:59:12.884 INFO:tasks.rados.rados.0.smithi125.stdout:1403: finishing write tid 4 to smithi12556470-20 2021-03-24T21:59:12.884 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 20 v 308 (ObjNum 505 snap 146 seq_num 505) dirty exists 2021-03-24T21:59:12.884 INFO:tasks.rados.rados.0.smithi125.stdout:1403: left oid 20 (ObjNum 505 snap 146 seq_num 505) 2021-03-24T21:59:12.892 INFO:tasks.rados.rados.0.smithi125.stdout:1404: expect (ObjNum 497 snap 145 seq_num 497) 2021-03-24T21:59:12.919 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:12.966 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:12.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:12.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:59:13.071 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:13.126 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:13.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:13.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:13.232 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:13.280 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:13.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:13.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:59:13.326 INFO:tasks.rados.rados.0.smithi125.stdout:1402: done (3 left) 2021-03-24T21:59:13.326 INFO:tasks.rados.rados.0.smithi125.stdout:1403: done (2 left) 2021-03-24T21:59:13.327 INFO:tasks.rados.rados.0.smithi125.stdout:1404: done (1 left) 2021-03-24T21:59:13.327 INFO:tasks.rados.rados.0.smithi125.stdout:1405: done (0 left) 2021-03-24T21:59:13.327 INFO:tasks.rados.rados.0.smithi125.stdout:1406: read oid 6 snap 142 2021-03-24T21:59:13.327 INFO:tasks.rados.rados.0.smithi125.stdout:1406: expect (ObjNum 488 snap 140 seq_num 488) 2021-03-24T21:59:13.327 INFO:tasks.rados.rados.0.smithi125.stdout:1407: snap_remove snap 138 2021-03-24T21:59:13.385 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:13.443 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:13.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:13.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:13.470 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:13 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T21:59:13.470 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:13 smithi125 conmon[19936]: T21:59:11.936645+0000 osd.7 (osd.7) 86 : cluster [DBG] 7.11 scrub ok 2021-03-24T21:59:13.471 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:13 smithi125 conmon[19936]: cluster 2021-03-24T21:59:12.370446+0000 osd.3 (osd.3) 87 : cluster [DBG] 7.c scrub ok 2021-03-24T21:59:13.471 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:13 smithi125 conmon[19936]: audit 2021-03-24T21:59:12.821822+0000 mon.a (mon.0) 1163 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "42"}]: dispatch 2021-03-24T21:59:13.471 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:13 smithi125 conmon[24927]: cluster 2021-03-24T21:59:11.936645+0000 osd.7 (osd.7) 86 : cluster [DBG] 7.11 scrub ok 2021-03-24T21:59:13.472 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:13 smithi125 conmon[24927]: cluster 2021-03-24T21:59:12.370446+0000 osd.3 (osd.3) 87 : cluster [DBG] 7.c scrub ok 2021-03-24T21:59:13.472 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:13 smithi125 conmon[24927]: audit 2021-03-24T21:59:12.821822+0000 mon.a (mon.0) 1163 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "42"}]: dispatch 2021-03-24T21:59:13.536 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T21:59:13.550 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:13.574 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T21:59:13.606 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:13.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:13.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:59:13.615 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T21:59:13.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:13 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:59:13.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:13 smithi142 conmon[20125]: 21:59:11.936645+0000 osd.7 (osd.7) 86 : cluster [DBG] 7.11 scrub ok 2021-03-24T21:59:13.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:13 smithi142 conmon[20125]: cluster 2021-03-24T21 2021-03-24T21:59:13.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:13 smithi142 conmon[20125]: :59:12.370446+0000 osd.3 (osd.3) 87 : cluster [DBG] 7.c scrub ok 2021-03-24T21:59:13.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:13 smithi142 conmon[20125]: audit 2021-03-24T21:59:12 2021-03-24T21:59:13.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:13 smithi142 conmon[20125]: .821822+0000 mon.a (mon.0) 1163 : audit 2021-03-24T21:59:13.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:13 smithi142 conmon[20125]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "42"}]: dispatch 2021-03-24T21:59:13.646 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T21:59:13.684 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T21:59:13.715 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:13.717 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T21:59:13.743 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T21:59:13.744 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-24T21:59:13.744 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-24T21:59:13.744 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-24T21:59:13.745 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-24T21:59:13.745 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-24T21:59:13.745 INFO:teuthology.orchestra.run.smithi125.stderr:Error ENXIO: problem getting command descriptions from osd.5 2021-03-24T21:59:13.745 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-24T21:59:13.745 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-24T21:59:13.759 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:13.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:13.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:13.863 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:13.918 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:13.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:13.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:59:14.025 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:14.081 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:14.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:14.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:59:14.187 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:14.249 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:14.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:14.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:59:14.355 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:14.361 INFO:tasks.rados.rados.0.smithi125.stdout:1407: done (1 left) 2021-03-24T21:59:14.361 INFO:tasks.rados.rados.0.smithi125.stdout:1408: write (excl) oid 2 current snap is 147 2021-03-24T21:59:14.362 INFO:tasks.rados.rados.0.smithi125.stdout:1408: seq_num 506 ranges {504342=555292,1643418=484844} 2021-03-24T21:59:14.362 INFO:tasks.rados.rados.0.smithi125.stdout:1408: writing smithi12556470-2 from 504342 to 1059634 tid 1 2021-03-24T21:59:14.367 INFO:tasks.rados.rados.0.smithi125.stdout:1408: writing smithi12556470-2 from 1643418 to 2128262 tid 2 2021-03-24T21:59:14.368 INFO:tasks.rados.rados.0.smithi125.stdout:1406: expect (ObjNum 488 snap 140 seq_num 488) 2021-03-24T21:59:14.533 INFO:tasks.rados.rados.0.smithi125.stdout:1406: done (1 left) 2021-03-24T21:59:14.534 INFO:tasks.rados.rados.0.smithi125.stdout:1409: read oid 4 snap -1 2021-03-24T21:59:14.534 INFO:tasks.rados.rados.0.smithi125.stdout:1409: expect (ObjNum 416 snap 108 seq_num 416) 2021-03-24T21:59:14.534 INFO:tasks.rados.rados.0.smithi125.stdout:1410: write (excl) oid 17 current snap is 147 2021-03-24T21:59:14.535 INFO:tasks.rados.rados.0.smithi125.stdout:1410: seq_num 507 ranges {521149=630541,1701900=628502} 2021-03-24T21:59:14.535 INFO:tasks.rados.rados.0.smithi125.stdout:1410: writing smithi12556470-17 from 521149 to 1151690 tid 1 2021-03-24T21:59:14.542 INFO:tasks.rados.rados.0.smithi125.stdout:1410: writing smithi12556470-17 from 1701900 to 2330402 tid 2 2021-03-24T21:59:14.544 INFO:tasks.rados.rados.0.smithi125.stdout:1411: read oid 46 snap -1 2021-03-24T21:59:14.545 INFO:tasks.rados.rados.0.smithi125.stdout:1411: expect (ObjNum 471 snap 133 seq_num 471) 2021-03-24T21:59:14.545 INFO:tasks.rados.rados.0.smithi125.stdout:1412: snap_remove snap 145 2021-03-24T21:59:14.553 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:14.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:14.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:59:14.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:14 smithi142 conmon[20125]: cluster 2021-03-24T21:59:12.679635+0000 osd.4 2021-03-24T21:59:14.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:14 smithi142 conmon[20125]: (osd.4) 98 : cluster [DBG] 3.1a scrub ok 2021-03-24T21:59:14.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:14 smithi142 conmon[20125]: cluster 2021-03-24T21:59:12.820475+0000 mgr.y (mgr.14144) 461 : cluster [DBG] pgmap v736: 178 pgs: 1 peering, 10 unknown, 50 active+undersized, 39 active+undersized+degraded, 78 active+clean; 317 MiB data, 811 MiB used, 536 GiB / 536 GiB avail; 254/1380 objects degraded (18.406%) 2021-03-24T21:59:14.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:14 smithi142 conmon[20125]: cluster 2021-03-24T21:59:12.939215+0000 osd.7 (osd 2021-03-24T21:59:14.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:14 smithi142 conmon[20125]: .7) 87 : cluster [DBG] 3.e scrub ok 2021-03-24T21:59:14.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:14 smithi142 conmon[20125]: audit 2021- 2021-03-24T21:59:14.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:14 smithi142 conmon[20125]: 03-24T21:59:13.321111 2021-03-24T21:59:14.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:14 smithi142 conmon[20125]: +0000 mon.a (mon.0) 1164 : audit 2021-03-24T21:59:14.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:14 smithi142 conmon[20125]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "42"}]': finished 2021-03-24T21:59:14.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:14 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:59:14.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:14 smithi142 conmon[20125]: 59:13.321354+0000 mon.a (mon. 2021-03-24T21:59:14.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:14 smithi142 conmon[20125]: 0) 1165 : cluster [DBG] osdmap e383: 8 total, 7 up, 6 in 2021-03-24T21:59:14.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:14 smithi142 conmon[20125]: audit 2021-03 2021-03-24T21:59:14.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:14 smithi142 conmon[20125]: -24T21:59:13.443480+0000 2021-03-24T21:59:14.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:14 smithi142 conmon[20125]: mon.b (mon.1) 33 : audit 2021-03-24T21:59:14.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:14 smithi142 conmon[20125]: [DBG] from='client.? v1:172.21.15.125:0/1662696351' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:59:14.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[19936]: cluster 2021-03-24T21:59:12.679635+0000 osd.4 ( 2021-03-24T21:59:14.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[19936]: osd.4) 98 : cluster [DBG] 3.1a scrub ok 2021-03-24T21:59:14.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[19936]: cluster 2021-03-24T21:59:12.820475+0000 mgr.y (mgr.14144) 461 : cluster [DBG] pgmap v736: 178 pgs: 1 peering, 10 unknown, 50 active+undersized, 39 active+undersized+degraded, 78 active+clean; 317 MiB data, 811 MiB used, 536 GiB / 536 GiB avail; 254/1380 objects degraded (18.406%) 2021-03-24T21:59:14.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[19936]: cluster 2021-03-24T21:59:12.939215+0000 2021-03-24T21:59:14.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[19936]: osd.7 (osd.7) 87 : cluster [DBG] 2021-03-24T21:59:14.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[19936]: 3.e scrub ok 2021-03-24T21:59:14.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[19936]: audit 2021 2021-03-24T21:59:14.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[19936]: -03-24T21:59:13.321111+0000 mon.a ( 2021-03-24T21:59:14.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[19936]: mon.0) 1164 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "42"}]': finished 2021-03-24T21:59:14.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:59:14.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[19936]: 24T21:59:13.321354+0000 mon.a ( 2021-03-24T21:59:14.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[19936]: mon.0) 1165 : cluster [DBG] osdmap e383: 8 total, 7 up, 6 in 2021-03-24T21:59:14.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[19936]: audit 2021-03- 2021-03-24T21:59:14.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[19936]: 24T21:59:13.443480+0000 mon.b ( 2021-03-24T21:59:14.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[19936]: mon.1) 33 : audit [DBG] from='client.? v1:172.21.15.125:0/1662696351' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:59:14.664 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:14.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: cluster 2021-03-24T21:59:14.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 2021-03-24T21:59:14.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 2021 2021-03-24T21:59:14.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: - 2021-03-24T21:59:14.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 03 2021-03-24T21:59:14.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: - 2021-03-24T21:59:14.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 24 2021-03-24T21:59:14.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: T 2021-03-24T21:59:14.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 21 2021-03-24T21:59:14.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: : 2021-03-24T21:59:14.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 59 2021-03-24T21:59:14.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: : 2021-03-24T21:59:14.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 12 2021-03-24T21:59:14.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: . 2021-03-24T21:59:14.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 679635 2021-03-24T21:59:14.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: +0000 2021-03-24T21:59:14.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 2021-03-24T21:59:14.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: osd.4 2021-03-24T21:59:14.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: ( 2021-03-24T21:59:14.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: osd 2021-03-24T21:59:14.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: . 2021-03-24T21:59:14.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 4 2021-03-24T21:59:14.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: ) 2021-03-24T21:59:14.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 98 2021-03-24T21:59:14.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: : 2021-03-24T21:59:14.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: cluster 2021-03-24T21:59:14.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 2021-03-24T21:59:14.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: [DBG] 2021-03-24T21:59:14.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 2021-03-24T21:59:14.672 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 3.1a scrub ok 2021-03-24T21:59:14.672 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 2021-03-24T21:59:14.672 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: cluster 2021-03-24T21:59:14.672 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 2021-03-24T21:59:14.673 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 2021 2021-03-24T21:59:14.673 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: - 2021-03-24T21:59:14.673 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 03 2021-03-24T21:59:14.673 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: - 2021-03-24T21:59:14.673 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 24 2021-03-24T21:59:14.674 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: T 2021-03-24T21:59:14.674 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 21 2021-03-24T21:59:14.674 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: : 2021-03-24T21:59:14.674 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 59 2021-03-24T21:59:14.675 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: : 2021-03-24T21:59:14.675 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 12 2021-03-24T21:59:14.675 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: . 2021-03-24T21:59:14.675 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 820475 2021-03-24T21:59:14.675 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: +0000 2021-03-24T21:59:14.676 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 2021-03-24T21:59:14.676 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: mgr.y 2021-03-24T21:59:14.676 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: ( 2021-03-24T21:59:14.676 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: mgr 2021-03-24T21:59:14.676 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: . 2021-03-24T21:59:14.677 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 14144 2021-03-24T21:59:14.677 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: ) 2021-03-24T21:59:14.677 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 461 2021-03-24T21:59:14.677 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: : 2021-03-24T21:59:14.677 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: cluster 2021-03-24T21:59:14.678 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 2021-03-24T21:59:14.678 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: [DBG] 2021-03-24T21:59:14.678 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 2021-03-24T21:59:14.678 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: pgmap v736: 178 pgs: 1 peering, 10 unknown, 50 active+undersized, 39 active+undersized+degraded, 78 active+clean; 317 MiB data, 811 MiB used, 536 GiB / 536 GiB avail; 254/1380 objects degraded (18.406%) 2021-03-24T21:59:14.678 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 2021-03-24T21:59:14.679 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: cluster 2021-03-24T21:59:14.679 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 2021-03-24T21:59:14.679 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 2021 2021-03-24T21:59:14.679 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: - 2021-03-24T21:59:14.680 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 03 2021-03-24T21:59:14.680 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: - 2021-03-24T21:59:14.680 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 24 2021-03-24T21:59:14.680 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: T 2021-03-24T21:59:14.680 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 21 2021-03-24T21:59:14.680 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: : 2021-03-24T21:59:14.681 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 59 2021-03-24T21:59:14.681 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: : 2021-03-24T21:59:14.681 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 12 2021-03-24T21:59:14.681 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: . 2021-03-24T21:59:14.681 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 939215 2021-03-24T21:59:14.682 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: +0000 2021-03-24T21:59:14.682 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 2021-03-24T21:59:14.682 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: osd.7 2021-03-24T21:59:14.682 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: ( 2021-03-24T21:59:14.683 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: osd 2021-03-24T21:59:14.683 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: . 2021-03-24T21:59:14.683 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 7 2021-03-24T21:59:14.683 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: ) 2021-03-24T21:59:14.683 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 87 2021-03-24T21:59:14.684 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: : 2021-03-24T21:59:14.684 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: cluster 2021-03-24T21:59:14.684 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 2021-03-24T21:59:14.684 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: [DBG] 2021-03-24T21:59:14.685 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 2021-03-24T21:59:14.685 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 3.e scrub ok 2021-03-24T21:59:14.685 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 2021-03-24T21:59:14.685 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: audit 2021-03-24T21:59:14.685 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 2021-03-24T21:59:14.686 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 2021 2021-03-24T21:59:14.686 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: - 2021-03-24T21:59:14.687 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 03 2021-03-24T21:59:14.687 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: - 2021-03-24T21:59:14.687 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 24 2021-03-24T21:59:14.688 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: T 2021-03-24T21:59:14.688 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 21 2021-03-24T21:59:14.689 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: : 2021-03-24T21:59:14.689 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 59 2021-03-24T21:59:14.689 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: : 2021-03-24T21:59:14.690 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 13 2021-03-24T21:59:14.690 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: . 2021-03-24T21:59:14.691 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 321111 2021-03-24T21:59:14.691 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: +0000 2021-03-24T21:59:14.692 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 2021-03-24T21:59:14.692 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: mon.a 2021-03-24T21:59:14.692 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: ( 2021-03-24T21:59:14.693 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: mon 2021-03-24T21:59:14.693 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: . 2021-03-24T21:59:14.694 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 0 2021-03-24T21:59:14.694 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: ) 2021-03-24T21:59:14.695 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 1164 2021-03-24T21:59:14.695 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: : 2021-03-24T21:59:14.696 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: audit 2021-03-24T21:59:14.696 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 2021-03-24T21:59:14.696 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: [INF] 2021-03-24T21:59:14.697 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 2021-03-24T21:59:14.697 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "42"}]': finished 2021-03-24T21:59:14.698 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 2021-03-24T21:59:14.698 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: cluster 2021-03-24T21:59:14.699 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 2021-03-24T21:59:14.699 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 2021 2021-03-24T21:59:14.700 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: - 2021-03-24T21:59:14.700 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 03 2021-03-24T21:59:14.701 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: - 2021-03-24T21:59:14.701 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 24 2021-03-24T21:59:14.702 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: T 2021-03-24T21:59:14.704 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 21 2021-03-24T21:59:14.705 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: : 2021-03-24T21:59:14.705 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 59 2021-03-24T21:59:14.706 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: : 2021-03-24T21:59:14.706 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 13 2021-03-24T21:59:14.707 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: . 2021-03-24T21:59:14.707 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 321354 2021-03-24T21:59:14.707 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: +0000 2021-03-24T21:59:14.707 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 2021-03-24T21:59:14.708 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: mon.a 2021-03-24T21:59:14.708 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: ( 2021-03-24T21:59:14.708 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: mon 2021-03-24T21:59:14.709 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: . 2021-03-24T21:59:14.709 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 0 2021-03-24T21:59:14.709 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: ) 2021-03-24T21:59:14.710 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 1165 2021-03-24T21:59:14.710 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: : 2021-03-24T21:59:14.710 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: cluster 2021-03-24T21:59:14.710 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 2021-03-24T21:59:14.711 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: [DBG] 2021-03-24T21:59:14.711 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 2021-03-24T21:59:14.711 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: osdmap e383: 8 total, 7 up, 6 in 2021-03-24T21:59:14.712 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 2021-03-24T21:59:14.712 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: audit 2021-03-24T21:59:14.712 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 2021-03-24T21:59:14.713 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 2021 2021-03-24T21:59:14.713 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: - 2021-03-24T21:59:14.713 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 03 2021-03-24T21:59:14.714 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: - 2021-03-24T21:59:14.714 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 24 2021-03-24T21:59:14.714 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: T 2021-03-24T21:59:14.715 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 21 2021-03-24T21:59:14.715 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: : 2021-03-24T21:59:14.715 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 59 2021-03-24T21:59:14.716 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: : 2021-03-24T21:59:14.716 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 13 2021-03-24T21:59:14.716 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: . 2021-03-24T21:59:14.717 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 443480 2021-03-24T21:59:14.717 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: +0000 2021-03-24T21:59:14.718 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 2021-03-24T21:59:14.718 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: mon.b 2021-03-24T21:59:14.718 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: ( 2021-03-24T21:59:14.719 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: mon 2021-03-24T21:59:14.719 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: . 2021-03-24T21:59:14.720 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 1 2021-03-24T21:59:14.720 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: ) 2021-03-24T21:59:14.721 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 33 2021-03-24T21:59:14.721 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: : 2021-03-24T21:59:14.722 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: audit 2021-03-24T21:59:14.722 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 2021-03-24T21:59:14.722 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: [DBG] 2021-03-24T21:59:14.723 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 2021-03-24T21:59:14.723 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: from='client.? v1:172.21.15.125:0/1662696351' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:59:14.724 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:14 smithi125 conmon[24927]: 2021-03-24T21:59:14.777 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:14.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:14.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:14.883 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:14.944 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:14.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:14.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:59:15.050 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:15.112 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:15.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:15.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:59:15.219 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:15.280 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:15.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:15.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:59:15.345 INFO:tasks.rados.rados.0.smithi125.stdout:1412: done (4 left) 2021-03-24T21:59:15.346 INFO:tasks.rados.rados.0.smithi125.stdout:1413: copy_from oid 16 from oid 24 current snap is 147 2021-03-24T21:59:15.346 INFO:tasks.rados.rados.0.smithi125.stdout:1414: delete oid 37 current snap is 147 2021-03-24T21:59:15.346 INFO:tasks.rados.rados.0.smithi125.stdout:1408: finishing write tid 1 to smithi12556470-2 2021-03-24T21:59:15.346 INFO:tasks.rados.rados.0.smithi125.stdout:1408: finishing write tid 2 to smithi12556470-2 2021-03-24T21:59:15.347 INFO:tasks.rados.rados.0.smithi125.stdout:1408: finishing write tid 3 to smithi12556470-2 2021-03-24T21:59:15.347 INFO:tasks.rados.rados.0.smithi125.stdout:1408: finishing write tid 4 to smithi12556470-2 2021-03-24T21:59:15.347 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 2 v 397 (ObjNum 506 snap 147 seq_num 506) dirty exists 2021-03-24T21:59:15.347 INFO:tasks.rados.rados.0.smithi125.stdout:1408: left oid 2 (ObjNum 506 snap 147 seq_num 506) 2021-03-24T21:59:15.347 INFO:tasks.rados.rados.0.smithi125.stdout:1409: expect (ObjNum 416 snap 108 seq_num 416) 2021-03-24T21:59:15.387 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:15.445 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:15.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:15.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:15.525 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[19936]: cluster 2021-03-24T21:59:15.525 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[19936]: 2021-03-24T21:59:14.319790+0000 osd.3 (osd.3) 88 : cluster [DBG] 4.1c scrub ok 2021-03-24T21:59:15.526 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[19936]: cluster 2021-03-24T21:59:14.338215+0000 mon.a (mon.0) 1166 : cluster [DBG] osdmap e384: 8 total, 7 up, 6 in 2021-03-24T21:59:15.526 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[19936]: cluster 2021-03-24T21:59:14.444734+0000 osd.3 (osd.3) 89 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:59:15.526 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[19936]: cluster 2021-03-24T21:59:14.445209+0000 osd.3 (osd.3) 90 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:59:15.526 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[19936]: cluster 2021-03-24T21:59:15.197113+0000 mon.a (mon.0) 1167 : cluster [WRN] Health check update: Degraded data redundancy: 254/1380 objects degraded (18.406%), 39 pgs degraded (PG_DEGRADED) 2021-03-24T21:59:15.527 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[19936]: cluster 2021-03-24T21:59:15.335753+0000 mon.a (mon.0) 1168 : cluster [DBG] osdmap e385: 8 total, 7 up, 6 in 2021-03-24T21:59:15.527 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: cluster 2021-03-24T21:59:15.527 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 2021-03-24T21:59:15.528 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 2021 2021-03-24T21:59:15.528 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: - 2021-03-24T21:59:15.528 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 03 2021-03-24T21:59:15.528 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: - 2021-03-24T21:59:15.528 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 24 2021-03-24T21:59:15.529 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: T 2021-03-24T21:59:15.529 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 21 2021-03-24T21:59:15.529 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: : 2021-03-24T21:59:15.529 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 59 2021-03-24T21:59:15.530 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: : 2021-03-24T21:59:15.530 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 14 2021-03-24T21:59:15.530 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: . 2021-03-24T21:59:15.530 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 319790 2021-03-24T21:59:15.530 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: +0000 2021-03-24T21:59:15.531 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 2021-03-24T21:59:15.531 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: osd.3 2021-03-24T21:59:15.531 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: ( 2021-03-24T21:59:15.531 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: osd 2021-03-24T21:59:15.532 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: . 2021-03-24T21:59:15.532 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 3 2021-03-24T21:59:15.532 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: ) 2021-03-24T21:59:15.532 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 88 2021-03-24T21:59:15.532 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: : 2021-03-24T21:59:15.533 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: cluster 2021-03-24T21:59:15.533 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 2021-03-24T21:59:15.533 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: [DBG] 2021-03-24T21:59:15.533 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 2021-03-24T21:59:15.534 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 4.1c scrub ok 2021-03-24T21:59:15.534 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 2021-03-24T21:59:15.534 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: cluster 2021-03-24T21:59:15.534 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 2021-03-24T21:59:15.534 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 2021 2021-03-24T21:59:15.535 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: - 2021-03-24T21:59:15.535 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 03 2021-03-24T21:59:15.535 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: - 2021-03-24T21:59:15.535 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 24 2021-03-24T21:59:15.535 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: T 2021-03-24T21:59:15.536 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 21 2021-03-24T21:59:15.536 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: : 2021-03-24T21:59:15.536 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 59 2021-03-24T21:59:15.536 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: : 2021-03-24T21:59:15.536 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 14 2021-03-24T21:59:15.537 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: . 2021-03-24T21:59:15.537 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 338215 2021-03-24T21:59:15.537 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: +0000 2021-03-24T21:59:15.537 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 2021-03-24T21:59:15.538 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: mon.a 2021-03-24T21:59:15.538 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: ( 2021-03-24T21:59:15.538 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: mon 2021-03-24T21:59:15.538 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: . 2021-03-24T21:59:15.538 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 0 2021-03-24T21:59:15.539 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: ) 2021-03-24T21:59:15.539 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 1166 2021-03-24T21:59:15.539 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: : 2021-03-24T21:59:15.539 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: cluster 2021-03-24T21:59:15.540 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 2021-03-24T21:59:15.540 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: [DBG] 2021-03-24T21:59:15.540 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 2021-03-24T21:59:15.540 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: osdmap e384: 8 total, 7 up, 6 in 2021-03-24T21:59:15.540 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 2021-03-24T21:59:15.541 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: cluster 2021-03-24T21:59:15.541 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 2021-03-24T21:59:15.541 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 2021 2021-03-24T21:59:15.541 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: - 2021-03-24T21:59:15.541 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 03 2021-03-24T21:59:15.542 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: - 2021-03-24T21:59:15.542 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 24 2021-03-24T21:59:15.542 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: T 2021-03-24T21:59:15.542 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 21 2021-03-24T21:59:15.542 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: : 2021-03-24T21:59:15.543 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 59 2021-03-24T21:59:15.543 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: : 2021-03-24T21:59:15.543 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 14 2021-03-24T21:59:15.543 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: . 2021-03-24T21:59:15.544 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 444734 2021-03-24T21:59:15.544 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: +0000 2021-03-24T21:59:15.544 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 2021-03-24T21:59:15.544 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: osd.3 2021-03-24T21:59:15.544 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: ( 2021-03-24T21:59:15.545 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: osd 2021-03-24T21:59:15.545 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: . 2021-03-24T21:59:15.545 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 3 2021-03-24T21:59:15.545 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: ) 2021-03-24T21:59:15.545 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 89 2021-03-24T21:59:15.546 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: : 2021-03-24T21:59:15.546 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: cluster 2021-03-24T21:59:15.546 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 2021-03-24T21:59:15.546 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: [DBG] 2021-03-24T21:59:15.546 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 2021-03-24T21:59:15.547 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: purged_snaps scrub starts 2021-03-24T21:59:15.547 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 2021-03-24T21:59:15.547 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: cluster 2021-03-24T21:59:15.547 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 2021-03-24T21:59:15.548 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 2021 2021-03-24T21:59:15.548 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: - 2021-03-24T21:59:15.548 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 03 2021-03-24T21:59:15.548 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: - 2021-03-24T21:59:15.548 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 24 2021-03-24T21:59:15.549 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: T 2021-03-24T21:59:15.549 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 21 2021-03-24T21:59:15.549 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: : 2021-03-24T21:59:15.549 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 59 2021-03-24T21:59:15.549 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: : 2021-03-24T21:59:15.550 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 14 2021-03-24T21:59:15.550 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: . 2021-03-24T21:59:15.550 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 445209 2021-03-24T21:59:15.550 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: +0000 2021-03-24T21:59:15.551 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 2021-03-24T21:59:15.551 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: osd.3 2021-03-24T21:59:15.551 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: ( 2021-03-24T21:59:15.551 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: osd 2021-03-24T21:59:15.551 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: . 2021-03-24T21:59:15.552 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 3 2021-03-24T21:59:15.552 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: ) 2021-03-24T21:59:15.552 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 90 2021-03-24T21:59:15.552 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: : 2021-03-24T21:59:15.552 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: cluster 2021-03-24T21:59:15.553 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 2021-03-24T21:59:15.553 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: [DBG] 2021-03-24T21:59:15.553 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 2021-03-24T21:59:15.553 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: purged_snaps scrub ok 2021-03-24T21:59:15.554 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 2021-03-24T21:59:15.554 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: cluster 2021-03-24T21:59:15.554 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 2021-03-24T21:59:15.554 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 2021 2021-03-24T21:59:15.554 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: - 2021-03-24T21:59:15.555 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 03 2021-03-24T21:59:15.555 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: - 2021-03-24T21:59:15.555 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 24 2021-03-24T21:59:15.555 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: T 2021-03-24T21:59:15.555 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 21 2021-03-24T21:59:15.556 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: : 2021-03-24T21:59:15.556 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 59 2021-03-24T21:59:15.556 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: : 2021-03-24T21:59:15.556 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 15 2021-03-24T21:59:15.556 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: . 2021-03-24T21:59:15.557 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 197113 2021-03-24T21:59:15.557 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: +0000 2021-03-24T21:59:15.557 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 2021-03-24T21:59:15.557 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: mon.a 2021-03-24T21:59:15.557 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: ( 2021-03-24T21:59:15.558 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: mon 2021-03-24T21:59:15.558 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: . 2021-03-24T21:59:15.558 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 0 2021-03-24T21:59:15.558 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: ) 2021-03-24T21:59:15.558 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 1167 2021-03-24T21:59:15.559 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: : 2021-03-24T21:59:15.559 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: cluster 2021-03-24T21:59:15.559 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 2021-03-24T21:59:15.559 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: [WRN] 2021-03-24T21:59:15.560 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 2021-03-24T21:59:15.560 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: Health check update: Degraded data redundancy: 254/1380 objects degraded (18.406%), 39 pgs degraded (PG_DEGRADED) 2021-03-24T21:59:15.560 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 2021-03-24T21:59:15.560 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: cluster 2021-03-24T21:59:15.560 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 2021-03-24T21:59:15.561 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 2021 2021-03-24T21:59:15.561 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: - 2021-03-24T21:59:15.561 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 03 2021-03-24T21:59:15.561 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: - 2021-03-24T21:59:15.561 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 24 2021-03-24T21:59:15.562 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: T 2021-03-24T21:59:15.562 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 21 2021-03-24T21:59:15.562 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: : 2021-03-24T21:59:15.562 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 59 2021-03-24T21:59:15.563 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: : 2021-03-24T21:59:15.563 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 15 2021-03-24T21:59:15.563 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: . 2021-03-24T21:59:15.563 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 335753 2021-03-24T21:59:15.563 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: +0000 2021-03-24T21:59:15.564 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 2021-03-24T21:59:15.564 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: mon.a 2021-03-24T21:59:15.564 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: ( 2021-03-24T21:59:15.564 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: mon 2021-03-24T21:59:15.564 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: . 2021-03-24T21:59:15.565 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 0 2021-03-24T21:59:15.565 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: ) 2021-03-24T21:59:15.565 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 1168 2021-03-24T21:59:15.565 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: : 2021-03-24T21:59:15.565 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: cluster 2021-03-24T21:59:15.566 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 2021-03-24T21:59:15.566 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: [DBG] 2021-03-24T21:59:15.566 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 2021-03-24T21:59:15.566 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: osdmap e385: 8 total, 7 up, 6 in 2021-03-24T21:59:15.567 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:15 smithi125 conmon[24927]: 2021-03-24T21:59:15.567 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:15.575 INFO:tasks.rados.rados.0.smithi125.stdout:1410: finishing write tid 1 to smithi12556470-17 2021-03-24T21:59:15.575 INFO:tasks.rados.rados.0.smithi125.stdout:1411: expect (ObjNum 471 snap 133 seq_num 471) 2021-03-24T21:59:15.610 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:15.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:15.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:59:15.717 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:15.731 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: cluster 2021-03-24T21:59:15.731 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 2021-03-24T21:59:15.732 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 2021 2021-03-24T21:59:15.732 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: - 2021-03-24T21:59:15.732 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 03 2021-03-24T21:59:15.732 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: - 2021-03-24T21:59:15.732 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 24 2021-03-24T21:59:15.733 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: T 2021-03-24T21:59:15.733 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 21 2021-03-24T21:59:15.733 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: : 2021-03-24T21:59:15.733 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 59 2021-03-24T21:59:15.734 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: : 2021-03-24T21:59:15.734 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 14 2021-03-24T21:59:15.734 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: . 2021-03-24T21:59:15.734 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 319790 2021-03-24T21:59:15.734 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: +0000 2021-03-24T21:59:15.735 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 2021-03-24T21:59:15.735 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: osd.3 2021-03-24T21:59:15.735 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: ( 2021-03-24T21:59:15.735 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: osd 2021-03-24T21:59:15.736 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: . 2021-03-24T21:59:15.736 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 3 2021-03-24T21:59:15.736 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: ) 2021-03-24T21:59:15.736 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 88 2021-03-24T21:59:15.736 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: : 2021-03-24T21:59:15.737 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: cluster 2021-03-24T21:59:15.737 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 2021-03-24T21:59:15.737 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: [DBG] 2021-03-24T21:59:15.737 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 2021-03-24T21:59:15.737 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 4.1c scrub ok 2021-03-24T21:59:15.738 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 2021-03-24T21:59:15.738 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: cluster 2021-03-24T21:59:15.738 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 2021-03-24T21:59:15.738 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 2021 2021-03-24T21:59:15.739 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: - 2021-03-24T21:59:15.739 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 03 2021-03-24T21:59:15.739 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: - 2021-03-24T21:59:15.739 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 24 2021-03-24T21:59:15.739 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: T 2021-03-24T21:59:15.740 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 21 2021-03-24T21:59:15.740 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: : 2021-03-24T21:59:15.740 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 59 2021-03-24T21:59:15.740 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: : 2021-03-24T21:59:15.740 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 14 2021-03-24T21:59:15.741 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: . 2021-03-24T21:59:15.741 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 338215 2021-03-24T21:59:15.741 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: +0000 2021-03-24T21:59:15.741 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 2021-03-24T21:59:15.741 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: mon.a 2021-03-24T21:59:15.742 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: ( 2021-03-24T21:59:15.742 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: mon 2021-03-24T21:59:15.742 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: . 2021-03-24T21:59:15.742 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 0 2021-03-24T21:59:15.743 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: ) 2021-03-24T21:59:15.743 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 1166 2021-03-24T21:59:15.743 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: : 2021-03-24T21:59:15.743 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: cluster 2021-03-24T21:59:15.743 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 2021-03-24T21:59:15.744 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: [DBG] 2021-03-24T21:59:15.744 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 2021-03-24T21:59:15.744 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: osdmap e384: 8 total, 7 up, 6 in 2021-03-24T21:59:15.744 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 2021-03-24T21:59:15.744 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: cluster 2021-03-24T21:59:15.745 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 2021-03-24T21:59:15.745 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 2021 2021-03-24T21:59:15.745 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: - 2021-03-24T21:59:15.745 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 03 2021-03-24T21:59:15.746 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: - 2021-03-24T21:59:15.746 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 24 2021-03-24T21:59:15.746 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: T 2021-03-24T21:59:15.746 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 21 2021-03-24T21:59:15.746 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: : 2021-03-24T21:59:15.747 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 59 2021-03-24T21:59:15.747 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: : 2021-03-24T21:59:15.747 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 14 2021-03-24T21:59:15.747 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: . 2021-03-24T21:59:15.747 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 444734 2021-03-24T21:59:15.748 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: +0000 2021-03-24T21:59:15.748 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 2021-03-24T21:59:15.748 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: osd.3 2021-03-24T21:59:15.748 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: ( 2021-03-24T21:59:15.748 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: osd 2021-03-24T21:59:15.749 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: . 2021-03-24T21:59:15.749 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 3 2021-03-24T21:59:15.749 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: ) 2021-03-24T21:59:15.749 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 89 2021-03-24T21:59:15.749 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: : 2021-03-24T21:59:15.750 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: cluster 2021-03-24T21:59:15.750 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 2021-03-24T21:59:15.750 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: [DBG] 2021-03-24T21:59:15.750 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 2021-03-24T21:59:15.751 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: purged_snaps scrub starts 2021-03-24T21:59:15.751 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 2021-03-24T21:59:15.751 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: cluster 2021-03-24T21:59:15.752 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 2021-03-24T21:59:15.752 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 2021 2021-03-24T21:59:15.752 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: - 2021-03-24T21:59:15.752 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 03 2021-03-24T21:59:15.753 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: - 2021-03-24T21:59:15.753 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 24 2021-03-24T21:59:15.753 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: T 2021-03-24T21:59:15.753 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 21 2021-03-24T21:59:15.753 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: : 2021-03-24T21:59:15.754 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 59 2021-03-24T21:59:15.754 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: : 2021-03-24T21:59:15.754 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 14 2021-03-24T21:59:15.754 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: . 2021-03-24T21:59:15.754 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 445209 2021-03-24T21:59:15.755 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: +0000 2021-03-24T21:59:15.755 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 2021-03-24T21:59:15.755 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: osd.3 2021-03-24T21:59:15.755 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: ( 2021-03-24T21:59:15.756 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: osd 2021-03-24T21:59:15.756 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: . 2021-03-24T21:59:15.756 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 3 2021-03-24T21:59:15.756 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: ) 2021-03-24T21:59:15.756 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 90 2021-03-24T21:59:15.757 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: : 2021-03-24T21:59:15.757 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: cluster 2021-03-24T21:59:15.757 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 2021-03-24T21:59:15.757 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: [DBG] 2021-03-24T21:59:15.757 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 2021-03-24T21:59:15.758 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: purged_snaps scrub ok 2021-03-24T21:59:15.758 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 2021-03-24T21:59:15.758 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: cluster 2021-03-24T21:59:15.758 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 2021-03-24T21:59:15.759 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 2021 2021-03-24T21:59:15.759 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: - 2021-03-24T21:59:15.759 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 03 2021-03-24T21:59:15.759 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: - 2021-03-24T21:59:15.759 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 24 2021-03-24T21:59:15.760 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: T 2021-03-24T21:59:15.760 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 21 2021-03-24T21:59:15.760 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: : 2021-03-24T21:59:15.760 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 59 2021-03-24T21:59:15.760 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: : 2021-03-24T21:59:15.761 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 15 2021-03-24T21:59:15.761 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: . 2021-03-24T21:59:15.761 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 197113 2021-03-24T21:59:15.761 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: +0000 2021-03-24T21:59:15.761 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 2021-03-24T21:59:15.762 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: mon.a 2021-03-24T21:59:15.762 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: ( 2021-03-24T21:59:15.762 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: mon 2021-03-24T21:59:15.762 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: . 2021-03-24T21:59:15.763 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 0 2021-03-24T21:59:15.763 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: ) 2021-03-24T21:59:15.763 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 1167 2021-03-24T21:59:15.763 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: : 2021-03-24T21:59:15.763 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: cluster 2021-03-24T21:59:15.764 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 2021-03-24T21:59:15.764 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: [WRN] 2021-03-24T21:59:15.764 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 2021-03-24T21:59:15.764 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: Health check update: Degraded data redundancy: 254/1380 objects degraded (18.406%), 39 pgs degraded (PG_DEGRADED) 2021-03-24T21:59:15.764 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 2021-03-24T21:59:15.765 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: cluster 2021-03-24T21:59:15.765 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 2021-03-24T21:59:15.765 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 2021 2021-03-24T21:59:15.765 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: - 2021-03-24T21:59:15.765 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 03 2021-03-24T21:59:15.766 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: - 2021-03-24T21:59:15.766 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 24 2021-03-24T21:59:15.766 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: T 2021-03-24T21:59:15.766 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 21 2021-03-24T21:59:15.767 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: : 2021-03-24T21:59:15.767 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 59 2021-03-24T21:59:15.767 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: : 2021-03-24T21:59:15.767 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 15 2021-03-24T21:59:15.767 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: . 2021-03-24T21:59:15.768 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 335753 2021-03-24T21:59:15.768 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: +0000 2021-03-24T21:59:15.768 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 2021-03-24T21:59:15.768 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: mon.a 2021-03-24T21:59:15.768 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: ( 2021-03-24T21:59:15.769 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: mon 2021-03-24T21:59:15.769 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: . 2021-03-24T21:59:15.769 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 0 2021-03-24T21:59:15.769 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: ) 2021-03-24T21:59:15.769 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 1168 2021-03-24T21:59:15.770 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: : 2021-03-24T21:59:15.770 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: cluster 2021-03-24T21:59:15.770 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 2021-03-24T21:59:15.770 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: [DBG] 2021-03-24T21:59:15.771 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 2021-03-24T21:59:15.771 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: osdmap e385: 8 total, 7 up, 6 in 2021-03-24T21:59:15.771 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:15 smithi142 conmon[20125]: 2021-03-24T21:59:15.772 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:15.773 INFO:tasks.rados.rados.0.smithi125.stdout:1410: finishing write tid 2 to smithi12556470-17 2021-03-24T21:59:15.773 INFO:tasks.rados.rados.0.smithi125.stdout:1410: finishing write tid 3 to smithi12556470-17 2021-03-24T21:59:15.773 INFO:tasks.rados.rados.0.smithi125.stdout:1410: finishing write tid 4 to smithi12556470-17 2021-03-24T21:59:15.773 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 17 v 379 (ObjNum 507 snap 147 seq_num 507) dirty exists 2021-03-24T21:59:15.773 INFO:tasks.rados.rados.0.smithi125.stdout:1410: left oid 17 (ObjNum 507 snap 147 seq_num 507) 2021-03-24T21:59:15.774 INFO:tasks.rados.rados.0.smithi125.stdout:1413: finishing copy_from to smithi12556470-16 2021-03-24T21:59:15.774 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 16 v 94 (ObjNum 444 snap 120 seq_num 444) dirty exists 2021-03-24T21:59:15.774 INFO:tasks.rados.rados.0.smithi125.stdout:1413: finishing copy_from racing read to smithi12556470-16 2021-03-24T21:59:15.774 INFO:tasks.rados.rados.0.smithi125.stdout:1408: done (5 left) 2021-03-24T21:59:15.775 INFO:tasks.rados.rados.0.smithi125.stdout:1409: done (4 left) 2021-03-24T21:59:15.775 INFO:tasks.rados.rados.0.smithi125.stdout:1410: done (3 left) 2021-03-24T21:59:15.775 INFO:tasks.rados.rados.0.smithi125.stdout:1411: done (2 left) 2021-03-24T21:59:15.775 INFO:tasks.rados.rados.0.smithi125.stdout:1413: done (1 left) 2021-03-24T21:59:15.775 INFO:tasks.rados.rados.0.smithi125.stdout:1414: done (0 left) 2021-03-24T21:59:15.776 INFO:tasks.rados.rados.0.smithi125.stdout:1415: rollback oid 35 current snap is 147 2021-03-24T21:59:15.776 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 35 to 137 2021-03-24T21:59:15.776 INFO:tasks.rados.rados.0.smithi125.stdout:1416: snap_remove snap 117 2021-03-24T21:59:15.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:15.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:15.878 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:15.929 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:15.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:15.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:16.036 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:16.091 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:16.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:16.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:59:16.198 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:16.254 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:16.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:16.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:59:16.360 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:16.399 INFO:tasks.rados.rados.0.smithi125.stdout:1416: done (1 left) 2021-03-24T21:59:16.400 INFO:tasks.rados.rados.0.smithi125.stdout:1417: read oid 28 snap -1 2021-03-24T21:59:16.400 INFO:tasks.rados.rados.0.smithi125.stdout:1417: expect (ObjNum 501 snap 146 seq_num 501) 2021-03-24T21:59:16.400 INFO:tasks.rados.rados.0.smithi125.stdout:1418: delete oid 8 current snap is 147 2021-03-24T21:59:16.401 INFO:tasks.rados.rados.0.smithi125.stdout:1415: finishing rollback tid 1 to smithi12556470-35 2021-03-24T21:59:16.401 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 35 v 0 (ObjNum 1118656688 snap 21916 seq_num 3019901136) dirty dne 2021-03-24T21:59:16.412 INFO:tasks.rados.rados.0.smithi125.stdout:1417: expect (ObjNum 501 snap 146 seq_num 501) 2021-03-24T21:59:16.459 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:16.464 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:16 smithi142 conmon[20125]: cluster 2021-03-24T21:59:14.821685 2021-03-24T21:59:16.464 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:16 smithi142 conmon[20125]: +0000 mgr.y (mgr.14144) 462 : cluster [DBG] pgmap v739: 178 pgs: 1 active+clean+snaptrim, 1 activating, 1 remapped+peering, 9 peering, 51 active+undersized, 38 active+undersized+degraded, 77 active+clean; 338 MiB data, 833 MiB used, 536 GiB / 536 GiB avail; 259/1422 objects degraded (18.214%) 2021-03-24T21:59:16.464 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:16 smithi142 conmon[20125]: cluster 2021-03-24T21:59:15.076143+0000 2021-03-24T21:59:16.464 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:16 smithi142 conmon[20125]: osd.0 (osd.0) 90 : cluster [DBG] 7.d scrub ok 2021-03-24T21:59:16.465 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:16 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:59:16.465 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:16 smithi142 conmon[20125]: T21:59:15.289438+0000 osd.3 (osd.3) 91 : cluster [DBG] 6.13 scrub ok 2021-03-24T21:59:16.465 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:16 smithi142 conmon[20125]: cluster 2021-03-24T21:59:15. 2021-03-24T21:59:16.465 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:16 smithi142 conmon[20125]: 365769+0000 mon.a (mon.0) 1169 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2021-03-24T21:59:16.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:16.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:16.562 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-03-24T21:59:16.633 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-03-24T21:59:16.634 INFO:tasks.rados.rados.0.smithi125.stdout:1415: done (2 left) 2021-03-24T21:59:16.634 INFO:tasks.rados.rados.0.smithi125.stdout:1417: done (1 left) 2021-03-24T21:59:16.634 INFO:tasks.rados.rados.0.smithi125.stdout:1418: done (0 left) 2021-03-24T21:59:16.635 INFO:tasks.rados.rados.0.smithi125.stdout:1419: rollback oid 27 current snap is 147 2021-03-24T21:59:16.635 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 27 to 137 2021-03-24T21:59:16.635 INFO:tasks.rados.rados.0.smithi125.stdout:1420: copy_from oid 47 from oid 9 current snap is 147 2021-03-24T21:59:16.635 INFO:tasks.rados.rados.0.smithi125.stdout:1421: snap_remove snap 108 2021-03-24T21:59:16.636 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:16.644 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:16 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:59:16.644 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:16 smithi125 conmon[19936]: 59:14.821685+0000 mgr.y (mgr.14144) 462 : cluster [DBG] pgmap v739: 178 pgs: 1 active+clean+snaptrim, 1 activating, 1 remapped+peering, 9 peering, 51 active+undersized, 38 active+undersized+degraded, 77 active+clean; 338 MiB data, 833 MiB used, 536 GiB / 536 GiB avail; 259/1422 objects degraded (18.214%) 2021-03-24T21:59:16.644 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:16 smithi125 conmon[19936]: cluster 2021-03-24T21:59:16.645 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:16 smithi125 conmon[19936]: 2021-03-24T21:59:15.076143+0000 osd.0 (osd.0) 90 : 2021-03-24T21:59:16.645 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:16 smithi125 conmon[19936]: cluster [DBG] 7.d scrub ok 2021-03-24T21:59:16.645 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:16 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:59:16.645 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:16 smithi125 conmon[19936]: 03-24T21:59:15.289438+0000 osd.3 2021-03-24T21:59:16.646 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:16 smithi125 conmon[19936]: (osd.3) 91 : cluster [DBG] 6.13 scrub ok 2021-03-24T21:59:16.646 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:16 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:59:16.646 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:16 smithi125 conmon[19936]: 03-24T21:59:15.365769+0000 mon.a ( 2021-03-24T21:59:16.646 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:16 smithi125 conmon[19936]: mon.0) 1169 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2021-03-24T21:59:16.647 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:16 smithi125 conmon[24927]: cluster 2021-03-24T21:59:14 2021-03-24T21:59:16.647 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:16 smithi125 conmon[24927]: .821685+0000 mgr.y (mgr.14144) 462 : cluster [DBG] pgmap v739: 178 pgs: 1 active+clean+snaptrim, 1 activating, 1 remapped+peering, 9 peering, 51 active+undersized, 38 active+undersized+degraded, 77 active+clean; 338 MiB data, 833 MiB used, 536 GiB / 536 GiB avail; 259/1422 objects degraded (18.214%) 2021-03-24T21:59:16.647 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:16 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:59:16.647 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:16 smithi125 conmon[24927]: -03-24T21:59:15.076143+0000 osd.0 (osd.0) 90 : 2021-03-24T21:59:16.648 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:16 smithi125 conmon[24927]: cluster [DBG] 7.d scrub ok 2021-03-24T21:59:16.648 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:16 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:59:16.648 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:16 smithi125 conmon[24927]: -24T21:59:15.289438+0000 osd.3 (osd.3) 2021-03-24T21:59:16.648 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:16 smithi125 conmon[24927]: 91 : cluster [DBG] 6.13 scrub ok 2021-03-24T21:59:16.648 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:16 smithi125 conmon[24927]: cluster 2021-03-24T21:59:16.649 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:16 smithi125 conmon[24927]: 2021-03-24T21:59:15.365769+0000 mon.a (mon 2021-03-24T21:59:16.649 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:16 smithi125 conmon[24927]: .0) 1169 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2021-03-24T21:59:16.691 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:16.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:16.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:59:16.796 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:16.848 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:16.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:16.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:59:16.955 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:17.053 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:17.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:17.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:59:17.126 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-24T21:59:17.127 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T21:59:17.160 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:17.221 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:17.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:17.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:59:17.328 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:17.388 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:17.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:17.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:17.407 INFO:tasks.rados.rados.0.smithi125.stdout:1421: done (2 left) 2021-03-24T21:59:17.407 INFO:tasks.rados.rados.0.smithi125.stdout:1422: read oid 7 snap -1 2021-03-24T21:59:17.408 INFO:tasks.rados.rados.0.smithi125.stdout:1422: expect (ObjNum 471 snap 133 seq_num 471) 2021-03-24T21:59:17.408 INFO:tasks.rados.rados.0.smithi125.stdout:1423: snap_create 2021-03-24T21:59:17.409 INFO:tasks.rados.rados.0.smithi125.stdout:1420: finishing copy_from to smithi12556470-47 2021-03-24T21:59:17.409 INFO:tasks.rados.rados.0.smithi125.stdout:1420: got expected ENOENT (src dne) 2021-03-24T21:59:17.410 INFO:tasks.rados.rados.0.smithi125.stdout:1420: finishing copy_from racing read to smithi12556470-47 2021-03-24T21:59:17.410 INFO:tasks.rados.rados.0.smithi125.stdout:1419: finishing rollback tid 0 to smithi12556470-27 2021-03-24T21:59:17.411 INFO:tasks.rados.rados.0.smithi125.stdout:1419: finishing rollback tid 1 to smithi12556470-27 2021-03-24T21:59:17.411 INFO:tasks.rados.rados.0.smithi125.stdout:1419: finishing rollback tid 2 to smithi12556470-27 2021-03-24T21:59:17.411 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 27 v 346 (ObjNum 400 snap 102 seq_num 400) dirty exists 2021-03-24T21:59:17.423 INFO:teuthology.orchestra.run.smithi125.stderr:noscrub is set 2021-03-24T21:59:17.453 INFO:tasks.rados.rados.0.smithi125.stdout:1422: expect (ObjNum 471 snap 133 seq_num 471) 2021-03-24T21:59:17.458 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:17 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:59:17.459 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:17 smithi125 conmon[24927]: 03-24T21:59:15.949532+0000 osd.7 (osd.7) 88 : cluster [DBG] 2021-03-24T21:59:17.459 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:17 smithi125 conmon[24927]: 3.13 scrub ok 2021-03-24T21:59:17.459 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:17 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:59:17.459 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:17 smithi125 conmon[24927]: 21:59:16.396532+0000 mon.a 2021-03-24T21:59:17.459 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:17 smithi125 conmon[24927]: (mon.0) 1170 : cluster 2021-03-24T21:59:17.460 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:17 smithi125 conmon[24927]: [DBG] osdmap e386: 8 total, 7 up, 6 in 2021-03-24T21:59:17.462 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:17 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T21:59:17.462 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:17 smithi125 conmon[24927]: 24T21:59:17.036919+0000 2021-03-24T21:59:17.463 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:17 smithi125 conmon[24927]: mon.a (mon.0) 2021-03-24T21:59:17.463 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:17 smithi125 conmon[24927]: 1171 : audit [INF] from='client.? v1:172.21.15.125:0/3013362618' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T21:59:17.463 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:17 smithi125 conmon[19936]: cluster 2021-03-24T21:59:15.949532+0000 2021-03-24T21:59:17.464 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:17 smithi125 conmon[19936]: osd.7 (osd.7) 88 : cluster [DBG] 3.13 scrub ok 2021-03-24T21:59:17.464 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:17 smithi125 conmon[19936]: cluster 2021-03-24T21:59:16.396532+0000 2021-03-24T21:59:17.464 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:17 smithi125 conmon[19936]: mon.a (mon.0) 1170 : cluster [DBG] osdmap e386: 8 total, 7 up, 6 in 2021-03-24T21:59:17.464 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:17 smithi125 conmon[19936]: audit 2021-03-24T21:59:17. 2021-03-24T21:59:17.465 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:17 smithi125 conmon[19936]: 036919+0000 mon.a (mon.0) 1171 : audit [INF] 2021-03-24T21:59:17.465 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:17 smithi125 conmon[19936]: from='client.? v1:172.21.15.125:0/3013362618' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T21:59:17.493 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:17.544 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:17.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:17.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:17.649 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:17.704 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:17.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:17.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:17.714 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 7] out_osds: [6, 1] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 6, 7] 2021-03-24T21:59:17.715 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-24T21:59:17.716 INFO:tasks.thrashosds.thrasher:Shrinking pool default.rgw.log 2021-03-24T21:59:17.716 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool default.rgw.log by 10 min 8 2021-03-24T21:59:17.717 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph pg dump --format=json 2021-03-24T21:59:17.810 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:17.861 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:17.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:17.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:59:17.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:17 smithi142 conmon[20125]: cluster 2021-03-24T21:59:15.949532+0000 osd.7 (osd.7) 88 : cluster [DBG] 3.13 scrub ok 2021-03-24T21:59:17.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:17 smithi142 conmon[20125]: cluster 2021-03-24T21:59:16.396532+0000 mon.a (mon.0) 1170 : cluster [DBG] osdmap e386: 8 total, 7 up, 6 in 2021-03-24T21:59:17.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:17 smithi142 conmon[20125]: audit 2021 2021-03-24T21:59:17.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:17 smithi142 conmon[20125]: -03-24T21:59:17.036919+0000 mon.a ( 2021-03-24T21:59:17.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:17 smithi142 conmon[20125]: mon.0) 1171 : audit [INF] from='client.? v1:172.21.15.125:0/3013362618' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T21:59:17.967 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:18.018 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:18.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:18.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:59:18.124 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:18.179 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:18.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:18.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:59:18.284 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:18.341 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:18.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:18.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:59:18.420 INFO:tasks.rados.rados.0.smithi125.stdout:1419: done (3 left) 2021-03-24T21:59:18.420 INFO:tasks.rados.rados.0.smithi125.stdout:1420: done (2 left) 2021-03-24T21:59:18.421 INFO:tasks.rados.rados.0.smithi125.stdout:1422: done (1 left) 2021-03-24T21:59:18.421 INFO:tasks.rados.rados.0.smithi125.stdout:1423: done (0 left) 2021-03-24T21:59:18.421 INFO:tasks.rados.rados.0.smithi125.stdout:1424: write oid 41 current snap is 148 2021-03-24T21:59:18.421 INFO:tasks.rados.rados.0.smithi125.stdout:1424: seq_num 510 ranges {571570=572641,1877346=694224} 2021-03-24T21:59:18.429 INFO:tasks.rados.rados.0.smithi125.stdout:1424: writing smithi12556470-41 from 571570 to 1144211 tid 1 2021-03-24T21:59:18.438 INFO:tasks.rados.rados.0.smithi125.stdout:1424: writing smithi12556470-41 from 1877346 to 2571570 tid 2 2021-03-24T21:59:18.439 INFO:tasks.rados.rados.0.smithi125.stdout:1425: snap_create 2021-03-24T21:59:18.446 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:18.450 INFO:tasks.rados.rados.0.smithi125.stdout:1424: finishing write tid 1 to smithi12556470-41 2021-03-24T21:59:18.450 INFO:tasks.rados.rados.0.smithi125.stdout:1424: finishing write tid 2 to smithi12556470-41 2021-03-24T21:59:18.451 INFO:tasks.rados.rados.0.smithi125.stdout:1424: finishing write tid 3 to smithi12556470-41 2021-03-24T21:59:18.451 INFO:tasks.rados.rados.0.smithi125.stdout:1424: finishing write tid 4 to smithi12556470-41 2021-03-24T21:59:18.452 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 41 v 383 (ObjNum 510 snap 148 seq_num 510) dirty exists 2021-03-24T21:59:18.452 INFO:tasks.rados.rados.0.smithi125.stdout:1424: left oid 41 (ObjNum 510 snap 148 seq_num 510) 2021-03-24T21:59:18.505 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:18.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:18.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:59:18.610 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:18.635 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-24T21:59:18.654 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:18.657 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:18 smithi142 conmon[20125]: cluster 2021-03-24T21:59:16.822524+0000 mgr.y (mgr.14144) 463 2021-03-24T21:59:18.657 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:18 smithi142 conmon[20125]: : cluster [DBG] pgmap v742: 178 pgs: 1 active+clean+snaptrim, 1 activating, 1 remapped+peering, 8 peering, 52 active+undersized, 38 active+undersized+degraded, 77 active+clean; 338 MiB data, 840 MiB used, 536 GiB / 536 GiB avail; 259/1422 objects degraded (18.214%) 2021-03-24T21:59:18.657 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:18 smithi142 conmon[20125]: cluster 2021-03-24T21:59:17.040617+0000 osd.0 (osd.0) 91 : cluster [DBG] 2021-03-24T21:59:18.658 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:18 smithi142 conmon[20125]: 5.1e deep-scrub ok 2021-03-24T21:59:18.658 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:18 smithi142 conmon[20125]: cluster 2021-03-24T21:59:17.264081+0000 osd.3 (osd.3) 92 2021-03-24T21:59:18.658 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:18 smithi142 conmon[20125]: : cluster [DBG] 3.b scrub ok 2021-03-24T21:59:18.658 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:18 smithi142 conmon[20125]: cluster 2021-03-24T21:59:18.658 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:18 smithi142 conmon[20125]: 2021-03-24T21:59:17.390547+0000 mon.a ( 2021-03-24T21:59:18.659 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:18 smithi142 conmon[20125]: mon.0) 1172 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:59:18.659 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:18 smithi142 conmon[20125]: audit 2021-03- 2021-03-24T21:59:18.659 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:18 smithi142 conmon[20125]: 24T21:59:17.403052+0000 mon.a (mon.0) 1173 : audit [INF] 2021-03-24T21:59:18.659 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:18 smithi142 conmon[20125]: from='client.? v1:172.21.15.125:0/3013362618' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T21:59:18.660 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:18 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:59:18.660 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:18 smithi142 conmon[20125]: -24T21:59:17.403148+0000 mon.a (mon.0) 1174 : cluster [DBG] osdmap e387: 8 total, 7 up, 6 in 2021-03-24T21:59:18.660 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:18 smithi142 conmon[20125]: 2021-03-24T21:59:18.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:18.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:59:18.683 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:18 smithi125 conmon[19936]: cluster 2021-03-24T21:59:16.822524 2021-03-24T21:59:18.683 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:18 smithi125 conmon[19936]: +0000 mgr.y (mgr.14144) 463 : cluster [DBG] pgmap v742: 178 pgs: 1 active+clean+snaptrim, 1 activating, 1 remapped+peering, 8 peering, 52 active+undersized, 38 active+undersized+degraded, 77 active+clean; 338 MiB data, 840 MiB used, 536 GiB / 536 GiB avail; 259/1422 objects degraded (18.214%) 2021-03-24T21:59:18.684 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:18 smithi125 conmon[19936]: cluster 2021-03-24T21:59:17. 2021-03-24T21:59:18.684 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:18 smithi125 conmon[19936]: 040617+0000 osd.0 (osd.0) 91 : cluster 2021-03-24T21:59:18.684 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:18 smithi125 conmon[19936]: [DBG] 5.1e deep-scrub ok 2021-03-24T21:59:18.684 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:18 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:59:18.684 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:18 smithi125 conmon[19936]: :59:17.264081+0000 osd.3 (osd. 2021-03-24T21:59:18.685 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:18 smithi125 conmon[19936]: 3) 92 : cluster [DBG] 3.b scrub ok 2021-03-24T21:59:18.685 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:18 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:59:18.685 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:18 smithi125 conmon[19936]: 03-24T21:59:17.390547+0000 2021-03-24T21:59:18.685 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:18 smithi125 conmon[19936]: mon.a (mon.0) 1172 : cluster [WRN] 2021-03-24T21:59:18.686 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:18 smithi125 conmon[19936]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:59:18.686 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:18 smithi125 conmon[19936]: audit 2021-03-24T21:59: 2021-03-24T21:59:18.686 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:18 smithi125 conmon[19936]: 17.403052+0000 mon.a (mon. 2021-03-24T21:59:18.686 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:18 smithi125 conmon[19936]: 0) 1173 : audit [INF] from='client.? v1:172.21.15.125:0/3013362618' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T21:59:18.687 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:18 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:59:18.687 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:18 smithi125 conmon[19936]: 03-24T21:59:17.403148+0000 2021-03-24T21:59:18.687 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:18 smithi125 conmon[19936]: mon.a (mon.0) 1174 : cluster [DBG] 2021-03-24T21:59:18.687 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:18 smithi125 conmon[19936]: osdmap e387: 8 total, 7 up, 6 in 2021-03-24T21:59:18.688 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:18 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:59:18.688 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:18 smithi125 conmon[24927]: 21:59:16.822524+0000 mgr.y (mgr.14144) 463 : cluster 2021-03-24T21:59:18.688 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:18 smithi125 conmon[24927]: [DBG] pgmap v742: 178 pgs: 1 active+clean+snaptrim, 1 activating, 1 remapped+peering, 8 peering, 52 active+undersized, 38 active+undersized+degraded, 77 active+clean; 338 MiB data, 840 MiB used, 536 GiB / 536 GiB avail; 259/1422 objects degraded (18.214%) 2021-03-24T21:59:18.688 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:18 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:59:18.689 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:18 smithi125 conmon[24927]: 03-24T21:59:17.040617+0000 osd.0 (osd. 2021-03-24T21:59:18.689 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:18 smithi125 conmon[24927]: 0) 91 : cluster [DBG] 5.1e deep-scrub ok 2021-03-24T21:59:18.689 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:18 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:59:18.689 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:18 smithi125 conmon[24927]: 03-24T21:59:17.264081+0000 2021-03-24T21:59:18.690 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:18 smithi125 conmon[24927]: osd.3 (osd.3) 92 : cluster 2021-03-24T21:59:18.690 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:18 smithi125 conmon[24927]: [DBG] 3.b scrub ok 2021-03-24T21:59:18.690 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:18 smithi125 conmon[24927]: cluster 2021-03-24T21:59:18.690 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:18 smithi125 conmon[24927]: 2021-03-24 2021-03-24T21:59:18.690 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:18 smithi125 conmon[24927]: T21:59:17.390547+0000 2021-03-24T21:59:18.691 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:18 smithi125 conmon[24927]: mon.a (mon.0) 1172 : 2021-03-24T21:59:18.691 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:18 smithi125 conmon[24927]: cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:59:18.691 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:18 smithi125 conmon[24927]: audit 2021-03-24T21:59:18.691 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:18 smithi125 conmon[24927]: 2021-03-24T21:59:17 2021-03-24T21:59:18.692 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:18 smithi125 conmon[24927]: .403052+0000 mon.a (mon.0) 2021-03-24T21:59:18.692 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:18 smithi125 conmon[24927]: 1173 : audit [INF] from='client.? v1:172.21.15.125:0/3013362618' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T21:59:18.692 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:18 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:59:18.692 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:18 smithi125 conmon[24927]: -24T21:59:17.403148+0000 2021-03-24T21:59:18.692 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:18 smithi125 conmon[24927]: mon.a (mon.0) 1174 : 2021-03-24T21:59:18.693 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:18 smithi125 conmon[24927]: cluster [DBG] osdmap e387: 8 total, 7 up, 6 in 2021-03-24T21:59:18.762 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:18.820 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:18.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:18.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:18.926 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:18.984 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:18.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:18.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:59:19.032 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.2 dump_historic_ops 2021-03-24T21:59:19.089 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:19.107 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:59:19.145 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:19.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:19.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:59:19.174 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:59:19.251 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:19.306 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:19.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:19.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:19.413 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:19.429 INFO:tasks.rados.rados.0.smithi125.stdout:1424: done (1 left) 2021-03-24T21:59:19.430 INFO:tasks.rados.rados.0.smithi125.stdout:1425: done (0 left) 2021-03-24T21:59:19.430 INFO:tasks.rados.rados.0.smithi125.stdout:1426: read oid 23 snap -1 2021-03-24T21:59:19.430 INFO:tasks.rados.rados.0.smithi125.stdout:1426: expect (ObjNum 427 snap 113 seq_num 427) 2021-03-24T21:59:19.430 INFO:tasks.rados.rados.0.smithi125.stdout:1427: read oid 8 snap -1 2021-03-24T21:59:19.430 INFO:tasks.rados.rados.0.smithi125.stdout:1427: expect deleted 2021-03-24T21:59:19.431 INFO:tasks.rados.rados.0.smithi125.stdout:1428: snap_create 2021-03-24T21:59:19.436 INFO:tasks.rados.rados.0.smithi125.stdout:1426: expect (ObjNum 427 snap 113 seq_num 427) 2021-03-24T21:59:19.478 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:19.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:19.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:19.580 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:19 smithi125 conmon[19936]: cluster 2021-03-24T21:59:19.580 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:19 smithi125 conmon[19936]: 2021-03-24T21:59:19.581 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:19 smithi125 conmon[19936]: 2021 2021-03-24T21:59:19.581 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:19 smithi125 conmon[19936]: - 2021-03-24T21:59:19.581 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:19 smithi125 conmon[19936]: 03 2021-03-24T21:59:19.581 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:19 smithi125 conmon[19936]: - 2021-03-24T21:59:19.582 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:19 smithi125 conmon[19936]: 24 2021-03-24T21:59:19.582 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:19 smithi125 conmon[19936]: T 2021-03-24T21:59:19.582 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:19 smithi125 conmon[19936]: 21 2021-03-24T21:59:19.582 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:19 smithi125 conmon[19936]: : 2021-03-24T21:59:19.583 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:19 smithi125 conmon[19936]: 59 2021-03-24T21:59:19.583 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:19 smithi125 conmon[19936]: : 2021-03-24T21:59:19.583 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:19 smithi125 conmon[19936]: 18 2021-03-24T21:59:19.583 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:19 smithi125 conmon[19936]: . 2021-03-24T21:59:19.583 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:19 smithi125 conmon[19936]: 417309 2021-03-24T21:59:19.584 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:19 smithi125 conmon[19936]: +0000 2021-03-24T21:59:19.584 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:19 smithi125 conmon[19936]: 2021-03-24T21:59:19.584 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:19 smithi125 conmon[19936]: mon.a 2021-03-24T21:59:19.584 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:19 smithi125 conmon[19936]: (mon.0) 1175 : cluster 2021-03-24T21:59:19.585 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:19 smithi125 conmon[19936]: [DBG] osdmap e388: 8 total, 7 up, 6 in 2021-03-24T21:59:19.585 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:19 smithi125 conmon[24927]: cluster 2021-03-24T21:59:18.417309+0000 mon.a (mon.0) 1175 : cluster [DBG] osdmap e388: 8 total, 7 up, 6 in 2021-03-24T21:59:19.585 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:19.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:19 smithi142 conmon[20125]: cluster 2021-03-24T21:59:18.417309+0000 mon.a (mon.0) 1175 : cluster [DBG] osdmap e388: 8 total, 7 up, 6 in 2021-03-24T21:59:19.623 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:59:19.656 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:19.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:19.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:19.764 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:19.816 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:19.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:19.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:19.921 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:19.970 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:19.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:19.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:59:20.075 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:20.128 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:20.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:20.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:59:20.213 INFO:tasks.rados.rados.0.smithi125.stdout:1426: done (2 left) 2021-03-24T21:59:20.213 INFO:tasks.rados.rados.0.smithi125.stdout:1427: done (1 left) 2021-03-24T21:59:20.213 INFO:tasks.rados.rados.0.smithi125.stdout:1428: done (0 left) 2021-03-24T21:59:20.214 INFO:tasks.rados.rados.0.smithi125.stdout:1429: snap_remove snap 123 2021-03-24T21:59:20.235 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:20.295 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:20.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:20.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:59:20.402 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:20.463 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:20.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:20.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:59:20.569 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:20.631 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:20.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:20.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:59:20.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[24927]: cluster 2021-03-24T21:59:20.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[24927]: 2021-03-24T21 2021-03-24T21:59:20.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[24927]: :59:18.823163+0000 mgr.y (mgr.14144) 464 : cluster [DBG] pgmap v745: 178 pgs: 1 active+clean+snaptrim, 1 activating, 1 remapped+peering, 8 peering, 52 active+undersized, 38 active+undersized+degraded, 77 active+clean; 338 MiB data, 835 MiB used, 536 GiB / 536 GiB avail; 0 B/s wr, 0 op/s; 260/1425 objects degraded (18.246%) 2021-03-24T21:59:20.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:59:20.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[24927]: -24T21:59:19.427715+0000 mon.a 2021-03-24T21:59:20.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[24927]: (mon.0) 1176 : cluster [DBG] osdmap e389: 8 total, 7 up, 6 in 2021-03-24T21:59:20.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[24927]: 2021-03-24T21:59:20.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:59:20.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[24927]: -24T21:59:20.107134+0000 osd.6 (osd. 2021-03-24T21:59:20.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[24927]: 6) 27 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:59:20.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[24927]: cluster 2021-03-24T21:59:20. 2021-03-24T21:59:20.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[24927]: 107223+0000 osd.6 (osd.6) 28 : cluster 2021-03-24T21:59:20.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[24927]: [DBG] purged_snaps scrub ok 2021-03-24T21:59:20.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[24927]: cluster 2021-03-24T21:59:20.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[24927]: 2021-03-24T21:59:20.199890+0000 mon.a 2021-03-24T21:59:20.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[24927]: (mon.0) 1177 : cluster [WRN] Health check update: Degraded data redundancy: 260/1425 objects degraded (18.246%), 38 pgs degraded (PG_DEGRADED) 2021-03-24T21:59:20.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:59:20.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[24927]: -24T21:59:20.208772+0000 mon.a 2021-03-24T21:59:20.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[24927]: (mon.0) 1178 : cluster [DBG] osdmap e390: 8 total, 7 up, 6 in 2021-03-24T21:59:20.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[24927]: 2021-03-24T21:59:20.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: cluster 2021-03-24T21:59:20.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 2021-03-24T21:59:20.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 2021 2021-03-24T21:59:20.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: - 2021-03-24T21:59:20.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 03 2021-03-24T21:59:20.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: - 2021-03-24T21:59:20.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 24 2021-03-24T21:59:20.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: T 2021-03-24T21:59:20.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 21 2021-03-24T21:59:20.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: : 2021-03-24T21:59:20.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 59 2021-03-24T21:59:20.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: : 2021-03-24T21:59:20.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 18 2021-03-24T21:59:20.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: . 2021-03-24T21:59:20.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 823163 2021-03-24T21:59:20.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: +0000 2021-03-24T21:59:20.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 2021-03-24T21:59:20.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: mgr.y 2021-03-24T21:59:20.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: ( 2021-03-24T21:59:20.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: mgr 2021-03-24T21:59:20.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: . 2021-03-24T21:59:20.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 14144 2021-03-24T21:59:20.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: ) 2021-03-24T21:59:20.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 464 2021-03-24T21:59:20.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: : 2021-03-24T21:59:20.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: cluster 2021-03-24T21:59:20.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 2021-03-24T21:59:20.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: [DBG] 2021-03-24T21:59:20.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 2021-03-24T21:59:20.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: pgmap v745: 178 pgs: 1 active+clean+snaptrim, 1 activating, 1 remapped+peering, 8 peering, 52 active+undersized, 38 active+undersized+degraded, 77 active+clean; 338 MiB data, 835 MiB used, 536 GiB / 536 GiB avail; 0 B/s wr, 0 op/s; 260/1425 objects degraded (18.246%) 2021-03-24T21:59:20.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 2021-03-24T21:59:20.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: cluster 2021-03-24T21:59:20.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 2021-03-24T21:59:20.669 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 2021 2021-03-24T21:59:20.669 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: - 2021-03-24T21:59:20.669 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 03 2021-03-24T21:59:20.669 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: - 2021-03-24T21:59:20.670 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 24 2021-03-24T21:59:20.670 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: T 2021-03-24T21:59:20.670 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 21 2021-03-24T21:59:20.670 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: : 2021-03-24T21:59:20.670 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 59 2021-03-24T21:59:20.671 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: : 2021-03-24T21:59:20.671 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 19 2021-03-24T21:59:20.671 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: . 2021-03-24T21:59:20.671 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 427715 2021-03-24T21:59:20.672 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: +0000 2021-03-24T21:59:20.672 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 2021-03-24T21:59:20.672 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: mon.a 2021-03-24T21:59:20.672 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: ( 2021-03-24T21:59:20.672 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: mon 2021-03-24T21:59:20.673 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: . 2021-03-24T21:59:20.673 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 0 2021-03-24T21:59:20.673 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: ) 2021-03-24T21:59:20.673 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 1176 2021-03-24T21:59:20.674 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: : 2021-03-24T21:59:20.674 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: cluster 2021-03-24T21:59:20.674 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 2021-03-24T21:59:20.674 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: [DBG] 2021-03-24T21:59:20.675 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 2021-03-24T21:59:20.675 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: osdmap e389: 8 total, 7 up, 6 in 2021-03-24T21:59:20.675 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 2021-03-24T21:59:20.675 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: cluster 2021-03-24T21:59:20.675 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 2021-03-24T21:59:20.676 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 2021 2021-03-24T21:59:20.676 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: - 2021-03-24T21:59:20.676 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 03 2021-03-24T21:59:20.676 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: - 2021-03-24T21:59:20.676 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 24 2021-03-24T21:59:20.677 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: T 2021-03-24T21:59:20.677 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 21 2021-03-24T21:59:20.677 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: : 2021-03-24T21:59:20.677 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 59 2021-03-24T21:59:20.678 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: : 2021-03-24T21:59:20.678 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 20 2021-03-24T21:59:20.678 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: . 2021-03-24T21:59:20.678 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 107134 2021-03-24T21:59:20.678 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: +0000 2021-03-24T21:59:20.679 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 2021-03-24T21:59:20.679 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: osd.6 2021-03-24T21:59:20.679 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: ( 2021-03-24T21:59:20.679 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: osd 2021-03-24T21:59:20.680 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: . 2021-03-24T21:59:20.680 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 6 2021-03-24T21:59:20.680 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: ) 2021-03-24T21:59:20.681 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 27 2021-03-24T21:59:20.681 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: : 2021-03-24T21:59:20.681 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: cluster 2021-03-24T21:59:20.681 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 2021-03-24T21:59:20.682 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: [DBG] 2021-03-24T21:59:20.682 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 2021-03-24T21:59:20.682 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: purged_snaps scrub starts 2021-03-24T21:59:20.682 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 2021-03-24T21:59:20.683 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: cluster 2021-03-24T21:59:20.683 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 2021-03-24T21:59:20.683 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 2021 2021-03-24T21:59:20.683 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: - 2021-03-24T21:59:20.684 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 03 2021-03-24T21:59:20.684 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: - 2021-03-24T21:59:20.684 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 24 2021-03-24T21:59:20.684 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: T 2021-03-24T21:59:20.684 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 21 2021-03-24T21:59:20.685 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: : 2021-03-24T21:59:20.685 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 59 2021-03-24T21:59:20.685 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: : 2021-03-24T21:59:20.685 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 20 2021-03-24T21:59:20.686 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: . 2021-03-24T21:59:20.686 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 107223 2021-03-24T21:59:20.686 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: +0000 2021-03-24T21:59:20.686 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 2021-03-24T21:59:20.686 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: osd.6 2021-03-24T21:59:20.687 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: ( 2021-03-24T21:59:20.687 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: osd 2021-03-24T21:59:20.687 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: . 2021-03-24T21:59:20.687 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 6 2021-03-24T21:59:20.688 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: ) 2021-03-24T21:59:20.688 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 28 2021-03-24T21:59:20.688 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: : 2021-03-24T21:59:20.688 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: cluster 2021-03-24T21:59:20.688 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 2021-03-24T21:59:20.689 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: [DBG] 2021-03-24T21:59:20.689 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 2021-03-24T21:59:20.689 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: purged_snaps scrub ok 2021-03-24T21:59:20.689 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 2021-03-24T21:59:20.689 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: cluster 2021-03-24T21:59:20.690 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 2021-03-24T21:59:20.690 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 2021 2021-03-24T21:59:20.690 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: - 2021-03-24T21:59:20.690 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 03 2021-03-24T21:59:20.691 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: - 2021-03-24T21:59:20.691 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 24 2021-03-24T21:59:20.691 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: T 2021-03-24T21:59:20.691 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 21 2021-03-24T21:59:20.691 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: : 2021-03-24T21:59:20.692 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 59 2021-03-24T21:59:20.692 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: : 2021-03-24T21:59:20.692 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 20 2021-03-24T21:59:20.692 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: . 2021-03-24T21:59:20.693 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 199890 2021-03-24T21:59:20.693 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: +0000 2021-03-24T21:59:20.693 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 2021-03-24T21:59:20.693 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: mon.a 2021-03-24T21:59:20.693 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: ( 2021-03-24T21:59:20.694 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: mon 2021-03-24T21:59:20.694 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: . 2021-03-24T21:59:20.694 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 0 2021-03-24T21:59:20.694 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: ) 2021-03-24T21:59:20.695 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 1177 2021-03-24T21:59:20.695 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: : 2021-03-24T21:59:20.695 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: cluster 2021-03-24T21:59:20.695 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 2021-03-24T21:59:20.695 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: [WRN] 2021-03-24T21:59:20.696 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 2021-03-24T21:59:20.696 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: Health check update: Degraded data redundancy: 260/1425 objects degraded (18.246%), 38 pgs degraded (PG_DEGRADED) 2021-03-24T21:59:20.696 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 2021-03-24T21:59:20.696 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: cluster 2021-03-24T21:59:20.697 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 2021-03-24T21:59:20.697 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 2021 2021-03-24T21:59:20.697 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: - 2021-03-24T21:59:20.697 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 03 2021-03-24T21:59:20.697 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: - 2021-03-24T21:59:20.698 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 24 2021-03-24T21:59:20.698 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: T 2021-03-24T21:59:20.698 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 21 2021-03-24T21:59:20.698 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: : 2021-03-24T21:59:20.699 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 59 2021-03-24T21:59:20.699 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: : 2021-03-24T21:59:20.699 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 20 2021-03-24T21:59:20.699 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: . 2021-03-24T21:59:20.699 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 208772 2021-03-24T21:59:20.700 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: +0000 2021-03-24T21:59:20.700 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 2021-03-24T21:59:20.700 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: mon.a 2021-03-24T21:59:20.700 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: ( 2021-03-24T21:59:20.701 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: mon 2021-03-24T21:59:20.701 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: . 2021-03-24T21:59:20.701 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 0 2021-03-24T21:59:20.701 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: ) 2021-03-24T21:59:20.701 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 1178 2021-03-24T21:59:20.702 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: : 2021-03-24T21:59:20.702 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: cluster 2021-03-24T21:59:20.702 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 2021-03-24T21:59:20.702 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: [DBG] 2021-03-24T21:59:20.703 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 2021-03-24T21:59:20.703 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: osdmap e390: 8 total, 7 up, 6 in 2021-03-24T21:59:20.703 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:20 smithi125 conmon[19936]: 2021-03-24T21:59:20.737 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:20.752 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:20 smithi142 conmon[20125]: cluster 2021-03-24T21:59:18 2021-03-24T21:59:20.753 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:20 smithi142 conmon[20125]: .823163+0000 mgr.y (mgr.14144) 464 : cluster [DBG] pgmap v745: 178 pgs: 1 active+clean+snaptrim, 1 activating, 1 remapped+peering, 8 peering, 52 active+undersized, 38 active+undersized+degraded, 77 active+clean; 338 MiB data, 835 MiB used, 536 GiB / 536 GiB avail; 0 B/s wr, 0 op/s; 260/1425 objects degraded (18.246%) 2021-03-24T21:59:20.753 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:20 smithi142 conmon[20125]: cluster 2021-03-24T21:59:19.427715+0000 2021-03-24T21:59:20.753 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:20 smithi142 conmon[20125]: mon.a (mon.0) 1176 : cluster [DBG] osdmap e389: 8 total, 7 up, 6 in 2021-03-24T21:59:20.754 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:20 smithi142 conmon[20125]: cluster 2021-03-24T21:59 2021-03-24T21:59:20.754 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:20 smithi142 conmon[20125]: :20.107134+0000 osd.6 (osd.6) 2021-03-24T21:59:20.754 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:20 smithi142 conmon[20125]: 27 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:59:20.754 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:20 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:59:20.755 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:20 smithi142 conmon[20125]: -24T21:59:20.107223 2021-03-24T21:59:20.755 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:20 smithi142 conmon[20125]: +0000 osd.6 (osd.6) 2021-03-24T21:59:20.755 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:20 smithi142 conmon[20125]: 28 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:59:20.755 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:20 smithi142 conmon[20125]: cluster 2021-03-24T21:59:20.756 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:20 smithi142 conmon[20125]: 2021-03-24T21: 2021-03-24T21:59:20.756 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:20 smithi142 conmon[20125]: 59:20.199890+0000 mon.a (mon.0 2021-03-24T21:59:20.756 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:20 smithi142 conmon[20125]: ) 1177 : cluster [WRN] Health check update: Degraded data redundancy: 260/1425 objects degraded (18.246%), 38 pgs degraded (PG_DEGRADED) 2021-03-24T21:59:20.756 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:20 smithi142 conmon[20125]: cluster 2021-03-24T21:59 2021-03-24T21:59:20.756 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:20 smithi142 conmon[20125]: :20.208772+0000 mon.a (mon.0) 1178 2021-03-24T21:59:20.757 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:20 smithi142 conmon[20125]: : cluster [DBG] osdmap e390: 8 total, 7 up, 6 in 2021-03-24T21:59:20.802 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:20.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:20.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:59:20.910 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:20.964 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:20.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:20.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:59:21.070 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:21.074 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-24T21:59:21.147 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:21.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:21.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:59:21.217 INFO:tasks.rados.rados.0.smithi125.stdout:1429: done (0 left) 2021-03-24T21:59:21.218 INFO:tasks.rados.rados.0.smithi125.stdout:1430: read oid 26 snap -1 2021-03-24T21:59:21.218 INFO:tasks.rados.rados.0.smithi125.stdout:1430: expect deleted 2021-03-24T21:59:21.218 INFO:tasks.rados.rados.0.smithi125.stdout:1431: read oid 13 snap -1 2021-03-24T21:59:21.219 INFO:tasks.rados.rados.0.smithi125.stdout:1431: expect deleted 2021-03-24T21:59:21.219 INFO:tasks.rados.rados.0.smithi125.stdout:1432: copy_from oid 17 from oid 46 current snap is 150 2021-03-24T21:59:21.219 INFO:tasks.rados.rados.0.smithi125.stdout:1433: delete oid 14 current snap is 150 2021-03-24T21:59:21.256 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:21.257 INFO:tasks.rados.rados.0.smithi125.stdout:1431: done (3 left) 2021-03-24T21:59:21.258 INFO:tasks.rados.rados.0.smithi125.stdout:1433: done (2 left) 2021-03-24T21:59:21.258 INFO:tasks.rados.rados.0.smithi125.stdout:1434: copy_from oid 48 from oid 11 current snap is 150 2021-03-24T21:59:21.258 INFO:tasks.rados.rados.0.smithi125.stdout:1435: read oid 38 snap -1 2021-03-24T21:59:21.259 INFO:tasks.rados.rados.0.smithi125.stdout:1435: expect (ObjNum 489 snap 140 seq_num 489) 2021-03-24T21:59:21.259 INFO:tasks.rados.rados.0.smithi125.stdout:1436: delete oid 15 current snap is 150 2021-03-24T21:59:21.269 INFO:tasks.rados.rados.0.smithi125.stdout:1435: expect (ObjNum 489 snap 140 seq_num 489) 2021-03-24T21:59:21.321 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:21.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:21.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:59:21.393 INFO:tasks.rados.rados.0.smithi125.stdout:1432: finishing copy_from to smithi12556470-17 2021-03-24T21:59:21.393 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 17 v 334 (ObjNum 471 snap 133 seq_num 471) dirty exists 2021-03-24T21:59:21.394 INFO:tasks.rados.rados.0.smithi125.stdout:1432: finishing copy_from racing read to smithi12556470-17 2021-03-24T21:59:21.394 INFO:tasks.rados.rados.0.smithi125.stdout:1434: finishing copy_from to smithi12556470-48 2021-03-24T21:59:21.394 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 48 v 312 (ObjNum 495 snap 142 seq_num 495) dirty exists 2021-03-24T21:59:21.394 INFO:tasks.rados.rados.0.smithi125.stdout:1434: finishing copy_from racing read to smithi12556470-48 2021-03-24T21:59:21.395 INFO:tasks.rados.rados.0.smithi125.stdout:1430: done (4 left) 2021-03-24T21:59:21.395 INFO:tasks.rados.rados.0.smithi125.stdout:1432: done (3 left) 2021-03-24T21:59:21.395 INFO:tasks.rados.rados.0.smithi125.stdout:1434: done (2 left) 2021-03-24T21:59:21.395 INFO:tasks.rados.rados.0.smithi125.stdout:1435: done (1 left) 2021-03-24T21:59:21.396 INFO:tasks.rados.rados.0.smithi125.stdout:1436: done (0 left) 2021-03-24T21:59:21.396 INFO:tasks.rados.rados.0.smithi125.stdout:1437: rollback oid 5 current snap is 150 2021-03-24T21:59:21.396 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 5 to 148 2021-03-24T21:59:21.396 INFO:tasks.rados.rados.0.smithi125.stdout:1438: rollback oid 48 current snap is 150 2021-03-24T21:59:21.397 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 48 to 142 2021-03-24T21:59:21.397 INFO:tasks.rados.rados.0.smithi125.stdout:1439: snap_remove snap 133 2021-03-24T21:59:21.430 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:21.493 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:21.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:21.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:21.602 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:21.658 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:21.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:21.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:59:21.765 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:21.819 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:21.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:21.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:21.925 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:21.980 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:21.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:21.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:59:22.086 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:22.143 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:22.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:22.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:59:22.224 INFO:tasks.rados.rados.0.smithi125.stdout:1439: done (2 left) 2021-03-24T21:59:22.224 INFO:tasks.rados.rados.0.smithi125.stdout:1440: snap_create 2021-03-24T21:59:22.224 INFO:tasks.rados.rados.0.smithi125.stdout:1437: finishing rollback tid 1 to smithi12556470-5 2021-03-24T21:59:22.224 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 5 v 374 (ObjNum 1116602224 snap 21916 seq_num 1117819152) dirty dne 2021-03-24T21:59:22.225 INFO:tasks.rados.rados.0.smithi125.stdout:1438: finishing rollback tid 0 to smithi12556470-48 2021-03-24T21:59:22.225 INFO:tasks.rados.rados.0.smithi125.stdout:1438: finishing rollback tid 1 to smithi12556470-48 2021-03-24T21:59:22.225 INFO:tasks.rados.rados.0.smithi125.stdout:1438: finishing rollback tid 2 to smithi12556470-48 2021-03-24T21:59:22.225 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 48 v 449 (ObjNum 462 snap 129 seq_num 462) dirty exists 2021-03-24T21:59:22.250 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:22.310 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:22.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:22.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:59:22.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:22 smithi142 conmon[20125]: cluster 2021-03-24T21:59:20 2021-03-24T21:59:22.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:22 smithi142 conmon[20125]: .823869+0000 mgr.y (mgr.14144) 465 : cluster [DBG] pgmap v748: 178 pgs: 4 active+clean+remapped, 52 active+undersized, 38 active+undersized+degraded, 84 active+clean; 337 MiB data, 829 MiB used, 536 GiB / 536 GiB avail; 262/1425 objects degraded (18.386%); 1/1425 objects misplaced (0.070%) 2021-03-24T21:59:22.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:22 smithi142 conmon[20125]: cluster 2021-03-24T21:59:22.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:22 smithi142 conmon[20125]: 2021-03-24T21:59:21 2021-03-24T21:59:22.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:22 smithi142 conmon[20125]: .214059+0000 mon.a (mon.0 2021-03-24T21:59:22.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:22 smithi142 conmon[20125]: ) 1179 : cluster [DBG] osdmap e391: 8 total, 7 up, 6 in 2021-03-24T21:59:22.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:22 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:59:22.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:22 smithi142 conmon[20125]: -03-24T21:59:21 2021-03-24T21:59:22.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:22 smithi142 conmon[20125]: .424003+0000 mon.a (mon.0 2021-03-24T21:59:22.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:22 smithi142 conmon[20125]: ) 1180 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2021-03-24T21:59:22.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:22 smithi142 conmon[20125]: 2021-03-24T21:59:22.423 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:22.484 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:22.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:22.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:59:22.589 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:22.643 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:22.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:22.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:22.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:22 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:59:22.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:22 smithi125 conmon[19936]: 24T21:59:20.823869+0000 mgr.y (mgr.14144) 465 : cluster [DBG] pgmap v748: 178 pgs: 4 active+clean+remapped, 52 active+undersized, 38 active+undersized+degraded, 84 active+clean; 337 MiB data, 829 MiB used, 536 GiB / 536 GiB avail; 262/1425 objects degraded (18.386%); 1/1425 objects misplaced (0.070%) 2021-03-24T21:59:22.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:22 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:59:22.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:22 smithi125 conmon[19936]: -03-24T21:59:21. 2021-03-24T21:59:22.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:22 smithi125 conmon[19936]: 214059+0000 mon.a (mon.0) 1179 : cluster [DBG] 2021-03-24T21:59:22.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:22 smithi125 conmon[19936]: osdmap e391: 8 total, 7 up, 6 in 2021-03-24T21:59:22.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:22 smithi125 conmon[19936]: cluster 2021-03-24T21:59:22.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:22 smithi125 conmon[19936]: 2021-03-24T21:59:21. 2021-03-24T21:59:22.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:22 smithi125 conmon[19936]: 424003+0000 mon.a (mon.0) 1180 : cluster 2021-03-24T21:59:22.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:22 smithi125 conmon[19936]: [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2021-03-24T21:59:22.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:22 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:59:22.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:22 smithi125 conmon[24927]: 59:20.823869+0000 mgr.y (mgr.14144) 465 : cluster [DBG] pgmap v748: 178 pgs: 4 active+clean+remapped, 52 active+undersized, 38 active+undersized+degraded, 84 active+clean; 337 MiB data, 829 MiB used, 536 GiB / 536 GiB avail; 262/1425 objects degraded (18.386%); 1/1425 objects misplaced (0.070%) 2021-03-24T21:59:22.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:22 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:59:22.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:22 smithi125 conmon[24927]: -03-24T21:59:21 2021-03-24T21:59:22.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:22 smithi125 conmon[24927]: .214059+0000 mon.a (mon. 2021-03-24T21:59:22.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:22 smithi125 conmon[24927]: 0) 1179 : cluster [DBG] osdmap e391: 8 total, 7 up, 6 in 2021-03-24T21:59:22.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:22 smithi125 conmon[24927]: 2021-03-24T21:59:22.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:22 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:59:22.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:22 smithi125 conmon[24927]: -03-24T21:59: 2021-03-24T21:59:22.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:22 smithi125 conmon[24927]: 21.424003+0000 mon.a (mon 2021-03-24T21:59:22.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:22 smithi125 conmon[24927]: .0) 1180 : cluster [INF] 2021-03-24T21:59:22.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:22 smithi125 conmon[24927]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2021-03-24T21:59:22.748 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:22.809 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:22.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:22.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:22.917 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:22.975 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:22.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:22.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:23.081 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:23.142 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:23.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:23.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:59:23.233 INFO:tasks.rados.rados.0.smithi125.stdout:1437: done (2 left) 2021-03-24T21:59:23.234 INFO:tasks.rados.rados.0.smithi125.stdout:1438: done (1 left) 2021-03-24T21:59:23.234 INFO:tasks.rados.rados.0.smithi125.stdout:1440: done (0 left) 2021-03-24T21:59:23.234 INFO:tasks.rados.rados.0.smithi125.stdout:1441: snap_remove snap 146 2021-03-24T21:59:23.235 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:59:23.249 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:23.273 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:23 smithi125 conmon[24927]: cluster 2021-03-24T21:59:22. 2021-03-24T21:59:23.273 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:23 smithi125 conmon[24927]: 221152+0000 mon.a (mon.0) 1181 : cluster [DBG] osdmap e392: 8 total, 7 up, 6 in 2021-03-24T21:59:23.274 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:23 smithi125 conmon[19936]: cluster 2021-03-24T21:59:22.221152 2021-03-24T21:59:23.274 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:23 smithi125 conmon[19936]: +0000 mon.a (mon.0) 1181 : cluster [DBG] osdmap e392: 8 total, 7 up, 6 in 2021-03-24T21:59:23.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:23 smithi142 conmon[20125]: cluster 2021-03-24T21:59:22 2021-03-24T21:59:23.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:23 smithi142 conmon[20125]: .221152+0000 mon.a (mon.0) 1181 : cluster [DBG] osdmap e392: 8 total, 7 up, 6 in 2021-03-24T21:59:23.367 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:23.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:23.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:59:23.473 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:23.515 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-24T21:59:23.520 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2021-03-24T21:59:23.523 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:23.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:23.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:59:23.629 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:23.687 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:23.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:23.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:23.707 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T21:59:23.747 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T21:59:23.785 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T21:59:23.794 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:23.805 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T21:59:23.826 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T21:59:23.849 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:23.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:23.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:59:23.856 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T21:59:23.883 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T21:59:23.884 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-24T21:59:23.884 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-24T21:59:23.884 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-24T21:59:23.885 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-24T21:59:23.885 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-24T21:59:23.885 INFO:teuthology.orchestra.run.smithi125.stderr:Error ENXIO: problem getting command descriptions from osd.5 2021-03-24T21:59:23.886 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-24T21:59:23.886 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-24T21:59:23.956 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:24.004 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:24.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:24.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:59:24.109 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:24.163 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:24.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:24.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:24.251 INFO:tasks.rados.rados.0.smithi125.stdout:1441: done (0 left) 2021-03-24T21:59:24.252 INFO:tasks.rados.rados.0.smithi125.stdout:1442: delete oid 35 current snap is 151 2021-03-24T21:59:24.269 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:24.273 INFO:tasks.rados.rados.0.smithi125.stdout:1442: done (0 left) 2021-03-24T21:59:24.274 INFO:tasks.rados.rados.0.smithi125.stdout:1443: rollback oid 30 current snap is 151 2021-03-24T21:59:24.274 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 30 to 139 2021-03-24T21:59:24.281 INFO:tasks.rados.rados.0.smithi125.stdout:1444: write oid 50 current snap is 151 2021-03-24T21:59:24.281 INFO:tasks.rados.rados.0.smithi125.stdout:1444: seq_num 514 ranges {638798=473637,1594921=783167} 2021-03-24T21:59:24.281 INFO:tasks.rados.rados.0.smithi125.stdout:1444: writing smithi12556470-50 from 638798 to 1112435 tid 1 2021-03-24T21:59:24.297 INFO:tasks.rados.rados.0.smithi125.stdout:1444: writing smithi12556470-50 from 1594921 to 2378088 tid 2 2021-03-24T21:59:24.298 INFO:tasks.rados.rados.0.smithi125.stdout:1443: finishing rollback tid 1 to smithi12556470-30 2021-03-24T21:59:24.298 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 30 v 285 (ObjNum 1116508576 snap 21916 seq_num 3154118864) dirty dne 2021-03-24T21:59:24.299 INFO:tasks.rados.rados.0.smithi125.stdout:1443: done (1 left) 2021-03-24T21:59:24.299 INFO:tasks.rados.rados.0.smithi125.stdout:1445: write oid 24 current snap is 151 2021-03-24T21:59:24.299 INFO:tasks.rados.rados.0.smithi125.stdout:1445: seq_num 515 ranges {655605=548886,1653403=643178} 2021-03-24T21:59:24.313 INFO:tasks.rados.rados.0.smithi125.stdout:1445: writing smithi12556470-24 from 655605 to 1204491 tid 1 2021-03-24T21:59:24.327 INFO:tasks.rados.rados.0.smithi125.stdout:1445: writing smithi12556470-24 from 1653403 to 2296581 tid 2 2021-03-24T21:59:24.327 INFO:tasks.rados.rados.0.smithi125.stdout:1444: finishing write tid 1 to smithi12556470-50 2021-03-24T21:59:24.327 INFO:tasks.rados.rados.0.smithi125.stdout:1446: snap_create 2021-03-24T21:59:24.329 INFO:tasks.rados.rados.0.smithi125.stdout:1445: finishing write tid 1 to smithi12556470-24 2021-03-24T21:59:24.337 INFO:tasks.rados.rados.0.smithi125.stdout:1444: finishing write tid 2 to smithi12556470-50 2021-03-24T21:59:24.337 INFO:tasks.rados.rados.0.smithi125.stdout:1444: finishing write tid 3 to smithi12556470-50 2021-03-24T21:59:24.338 INFO:tasks.rados.rados.0.smithi125.stdout:1444: finishing write tid 4 to smithi12556470-50 2021-03-24T21:59:24.338 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 50 v 312 (ObjNum 514 snap 151 seq_num 514) dirty exists 2021-03-24T21:59:24.338 INFO:tasks.rados.rados.0.smithi125.stdout:1444: left oid 50 (ObjNum 514 snap 151 seq_num 514) 2021-03-24T21:59:24.345 INFO:tasks.rados.rados.0.smithi125.stdout:1445: finishing write tid 2 to smithi12556470-24 2021-03-24T21:59:24.345 INFO:tasks.rados.rados.0.smithi125.stdout:1445: finishing write tid 3 to smithi12556470-24 2021-03-24T21:59:24.346 INFO:tasks.rados.rados.0.smithi125.stdout:1445: finishing write tid 4 to smithi12556470-24 2021-03-24T21:59:24.346 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 24 v 392 (ObjNum 515 snap 151 seq_num 515) dirty exists 2021-03-24T21:59:24.346 INFO:tasks.rados.rados.0.smithi125.stdout:1445: left oid 24 (ObjNum 515 snap 151 seq_num 515) 2021-03-24T21:59:24.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:24 smithi125 conmon[19936]: cluster 2021-03-24T21:59:22.824848+0000 mgr.y (mgr 2021-03-24T21:59:24.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:24 smithi125 conmon[19936]: .14144) 466 : cluster [DBG] pgmap v751: 178 pgs: 4 active+clean+remapped, 52 active+undersized, 38 active+undersized+degraded, 84 active+clean; 334 MiB data, 825 MiB used, 536 GiB / 536 GiB avail; 262/1422 objects degraded (18.425%); 1/1422 objects misplaced (0.070%) 2021-03-24T21:59:24.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:24 smithi125 conmon[19936]: cluster 2021-03-24T21:59:23.231410+0000 mon.a (mon.0) 1182 : cluster [DBG] osdmap e393: 8 total, 7 up, 6 in 2021-03-24T21:59:24.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:24 smithi125 conmon[19936]: audit 2021-03-24T 2021-03-24T21:59:24.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:24 smithi125 conmon[19936]: 21:59:23.490608+0000 mgr.y (mgr.14144) 467 : audit [DBG] from='client.35168 v1:172.21.15.125:0/2403748969' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T21:59:24.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:24 smithi125 conmon[19936]: audit 2021-03-24T21:59:24.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:24 smithi125 conmon[19936]: 2021-03-24T21:59:23.675168+0000 mon.c (mon.2 2021-03-24T21:59:24.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:24 smithi125 conmon[19936]: ) 61 : audit [DBG] from='client.? v1:172.21.15.125:0/3844587342' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:59:24.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:24 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:59:24.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:24 smithi125 conmon[24927]: 59:22.824848+0000 mgr.y (mgr.14144) 466 : cluster [DBG] pgmap v751: 178 pgs: 4 active+clean+remapped, 52 active+undersized, 38 active+undersized+degraded, 84 active+clean; 334 MiB data, 825 MiB used, 536 GiB / 536 GiB avail; 262/1422 objects degraded (18.425%); 1/1422 objects misplaced (0.070%) 2021-03-24T21:59:24.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:24 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:59:24.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:24 smithi125 conmon[24927]: :59:23.231410+0000 mon.a (mon.0) 2021-03-24T21:59:24.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:24 smithi125 conmon[24927]: 1182 : cluster [DBG] osdmap e393: 8 total, 7 up, 6 in 2021-03-24T21:59:24.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:24 smithi125 conmon[24927]: audit 2021-03-24T21:59:24.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:24 smithi125 conmon[24927]: 2021-03- 2021-03-24T21:59:24.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:24 smithi125 conmon[24927]: 24T21:59:23.490608+0000 mgr.y ( 2021-03-24T21:59:24.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:24 smithi125 conmon[24927]: mgr.14144) 467 : audit [DBG] from='client.35168 v1:172.21.15.125:0/2403748969' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T21:59:24.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:24 smithi125 conmon[24927]: audit 2021-03-24T21:59:23.675168+0000 mon.c (mon.2) 61 : audit [DBG] from='client.? v1:172.21.15.125:0/3844587342' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:59:24.427 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:24.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:24.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:59:24.535 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:24.580 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:24.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:24.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:59:24.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:24 smithi142 conmon[20125]: cluster 2021-03-24T21 2021-03-24T21:59:24.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:24 smithi142 conmon[20125]: :59:22.824848+0000 mgr.y (mgr.14144) 466 : cluster [DBG] pgmap v751: 178 pgs: 4 active+clean+remapped, 52 active+undersized, 38 active+undersized+degraded, 84 active+clean; 334 MiB data, 825 MiB used, 536 GiB / 536 GiB avail; 262/1422 objects degraded (18.425%); 1/1422 objects misplaced (0.070%) 2021-03-24T21:59:24.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:24 smithi142 conmon[20125]: cluster 2021-03-24T21:59:24.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:24 smithi142 conmon[20125]: 2021-03-24T21:59:23.231410+0000 mon.a (mon. 2021-03-24T21:59:24.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:24 smithi142 conmon[20125]: 0) 1182 : cluster [DBG] osdmap e393: 8 total, 7 up, 6 in 2021-03-24T21:59:24.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:24 smithi142 conmon[20125]: audit 2021-03- 2021-03-24T21:59:24.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:24 smithi142 conmon[20125]: 24T21:59:23.490608+0000 mgr.y (mgr 2021-03-24T21:59:24.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:24 smithi142 conmon[20125]: .14144) 467 : audit [DBG] from='client.35168 v1:172.21.15.125:0/2403748969' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T21:59:24.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:24 smithi142 conmon[20125]: audit 2021-03-24 2021-03-24T21:59:24.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:24 smithi142 conmon[20125]: T21:59:23.675168+0000 mon.c (mon 2021-03-24T21:59:24.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:24 smithi142 conmon[20125]: .2) 61 : audit [DBG] from='client.? v1:172.21.15.125:0/3844587342' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:59:24.685 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:24.738 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:24.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:24.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:24.843 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:24.895 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:24.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:24.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:25.002 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:25.062 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:25.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:25.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:59:25.169 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:25.223 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:25.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:25.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:59:25.247 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-24T21:59:25.247 INFO:teuthology.orchestra.run.smithi125.stdout: "size": 20, 2021-03-24T21:59:25.247 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 600, 2021-03-24T21:59:25.247 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [ 2021-03-24T21:59:25.248 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.248 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:277 7.8 7:19ff009c:::smithi12556470-42:head [copy-from ver 9] snapc 8=[8] ondisk+write+known_if_redirected e114)", 2021-03-24T21:59:25.248 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:43.202214+0000", 2021-03-24T21:59:25.248 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 281.86784089999998, 2021-03-24T21:59:25.249 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.015212942, 2021-03-24T21:59:25.249 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:59:25.249 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:59:25.249 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:59:25.250 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:59:25.250 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:59:25.250 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 277 2021-03-24T21:59:25.250 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.251 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:59:25.251 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.251 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:59:25.251 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.202214+0000", 2021-03-24T21:59:25.251 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:59:25.252 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.252 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.252 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:59:25.252 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.202214+0000", 2021-03-24T21:59:25.253 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999971 2021-03-24T21:59:25.253 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.253 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.253 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:59:25.254 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.202211+0000", 2021-03-24T21:59:25.254 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1984000000000001e-05 2021-03-24T21:59:25.254 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.255 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.255 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:59:25.255 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.202223+0000", 2021-03-24T21:59:25.255 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7306e-05 2021-03-24T21:59:25.255 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.256 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.256 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:59:25.256 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.202240+0000", 2021-03-24T21:59:25.256 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.6490000000000003e-06 2021-03-24T21:59:25.257 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.257 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.257 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:59:25.257 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.202250+0000", 2021-03-24T21:59:25.257 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.6559999999999999e-05 2021-03-24T21:59:25.258 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.258 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.258 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:25.258 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.202327+0000", 2021-03-24T21:59:25.259 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.1945000000000001e-05 2021-03-24T21:59:25.259 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.259 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.259 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:59:25.259 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.202419+0000", 2021-03-24T21:59:25.260 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0048950820000000003 2021-03-24T21:59:25.260 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.260 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.260 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 4,7", 2021-03-24T21:59:25.260 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.207314+0000", 2021-03-24T21:59:25.261 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0050236980000000001 2021-03-24T21:59:25.261 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.261 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.261 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:59:25.262 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.212337+0000", 2021-03-24T21:59:25.262 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0048611870000000003 2021-03-24T21:59:25.262 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.262 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.263 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:59:25.263 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.217199+0000", 2021-03-24T21:59:25.263 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1696000000000001e-05 2021-03-24T21:59:25.263 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.263 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.264 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:59:25.264 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.217250+0000", 2021-03-24T21:59:25.264 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.4931000000000004e-05 2021-03-24T21:59:25.264 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.264 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.265 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:59:25.265 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.217335+0000", 2021-03-24T21:59:25.265 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.1592e-05 2021-03-24T21:59:25.266 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.266 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.267 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:59:25.267 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.217427+0000", 2021-03-24T21:59:25.267 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.015212942 2021-03-24T21:59:25.268 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:25.268 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:59:25.268 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:25.268 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.269 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.269 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:278 7.8 7:19ff009c:::smithi12556470-42:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e114)", 2021-03-24T21:59:25.269 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:43.202267+0000", 2021-03-24T21:59:25.269 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 281.86778734500001, 2021-03-24T21:59:25.269 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.015574229, 2021-03-24T21:59:25.270 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:59:25.270 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T21:59:25.270 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:59:25.271 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:59:25.271 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:59:25.271 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 278 2021-03-24T21:59:25.271 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.271 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:59:25.272 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.272 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:59:25.272 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.202267+0000", 2021-03-24T21:59:25.273 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:59:25.273 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.273 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.273 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:59:25.273 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.202267+0000", 2021-03-24T21:59:25.274 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T21:59:25.274 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.274 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.274 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:59:25.274 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.202267+0000", 2021-03-24T21:59:25.275 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6749999999999999e-06 2021-03-24T21:59:25.275 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.275 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.275 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:59:25.276 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.202272+0000", 2021-03-24T21:59:25.276 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.8089999999999998e-06 2021-03-24T21:59:25.276 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.276 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.277 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:59:25.277 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.202278+0000", 2021-03-24T21:59:25.277 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.354e-06 2021-03-24T21:59:25.277 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.278 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.278 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:59:25.278 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.202282+0000", 2021-03-24T21:59:25.278 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00032993900000000001 2021-03-24T21:59:25.278 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.279 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.279 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:25.279 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.202611+0000", 2021-03-24T21:59:25.279 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.3274000000000004e-05 2021-03-24T21:59:25.279 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.280 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.280 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-24T21:59:25.280 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.202685+0000", 2021-03-24T21:59:25.280 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0085844809999999997 2021-03-24T21:59:25.281 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.281 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.281 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:25.281 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.211269+0000", 2021-03-24T21:59:25.281 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.8164000000000004e-05 2021-03-24T21:59:25.282 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.282 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.282 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T21:59:25.282 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.211367+0000", 2021-03-24T21:59:25.282 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0061402770000000004 2021-03-24T21:59:25.283 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.283 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.283 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:25.283 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.217508+0000", 2021-03-24T21:59:25.284 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00016125499999999999 2021-03-24T21:59:25.284 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.284 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.284 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:59:25.284 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.217669+0000", 2021-03-24T21:59:25.285 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00017273599999999999 2021-03-24T21:59:25.285 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.285 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.285 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:59:25.285 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:43.217842+0000", 2021-03-24T21:59:25.286 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.015574229 2021-03-24T21:59:25.286 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:25.286 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:59:25.286 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:25.287 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.287 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.287 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:645 7.c e141/101 7:305739dc:::smithi12556470-36:head v 141'47, mlcod=134'43)", 2021-03-24T21:59:25.287 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:55:10.123986+0000", 2021-03-24T21:59:25.287 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 254.946068416, 2021-03-24T21:59:25.288 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.025623269000000001, 2021-03-24T21:59:25.288 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:59:25.288 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:59:25.288 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:59:25.289 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.289 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:59:25.289 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.123986+0000", 2021-03-24T21:59:25.289 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:59:25.289 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.290 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.290 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:59:25.290 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.123986+0000", 2021-03-24T21:59:25.290 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T21:59:25.290 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.291 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.291 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:59:25.291 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.123985+0000", 2021-03-24T21:59:25.291 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2130000000000001e-05 2021-03-24T21:59:25.292 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.292 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.292 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:59:25.292 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.123997+0000", 2021-03-24T21:59:25.292 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.4334e-05 2021-03-24T21:59:25.293 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.293 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.293 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:59:25.293 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124031+0000", 2021-03-24T21:59:25.294 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.3420000000000002e-06 2021-03-24T21:59:25.294 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.294 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.294 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:59:25.294 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124039+0000", 2021-03-24T21:59:25.295 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.0111999999999997e-05 2021-03-24T21:59:25.295 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.295 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.295 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:25.295 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124089+0000", 2021-03-24T21:59:25.296 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.3839e-05 2021-03-24T21:59:25.296 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.296 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.296 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:59:25.297 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124113+0000", 2021-03-24T21:59:25.297 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.025454029999999999 2021-03-24T21:59:25.297 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.297 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.297 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:59:25.298 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.149567+0000", 2021-03-24T21:59:25.298 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.2976000000000003e-05 2021-03-24T21:59:25.298 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.298 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.298 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:59:25.299 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.149610+0000", 2021-03-24T21:59:25.299 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.025623269000000001 2021-03-24T21:59:25.299 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:25.299 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:59:25.300 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:25.300 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.300 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.300 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:646 7.c e141/101 7:305739dc:::smithi12556470-36:head v 141'48, mlcod=134'43)", 2021-03-24T21:59:25.300 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:55:10.124053+0000", 2021-03-24T21:59:25.301 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 254.9460019, 2021-03-24T21:59:25.301 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.027636099000000001, 2021-03-24T21:59:25.301 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:59:25.301 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:59:25.302 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:59:25.302 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.302 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:59:25.302 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124053+0000", 2021-03-24T21:59:25.302 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:59:25.303 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.303 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.303 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:59:25.303 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124053+0000", 2021-03-24T21:59:25.303 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T21:59:25.304 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.304 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.304 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:59:25.304 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124052+0000", 2021-03-24T21:59:25.305 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.2480000000000002e-06 2021-03-24T21:59:25.305 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.305 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.305 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:59:25.305 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124061+0000", 2021-03-24T21:59:25.306 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6640000000000003e-06 2021-03-24T21:59:25.306 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.306 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.306 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:59:25.307 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124067+0000", 2021-03-24T21:59:25.307 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9349999999999996e-06 2021-03-24T21:59:25.307 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.307 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.307 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:59:25.308 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124071+0000", 2021-03-24T21:59:25.308 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00051871899999999995 2021-03-24T21:59:25.308 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.308 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.309 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:25.309 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124590+0000", 2021-03-24T21:59:25.309 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2456999999999999e-05 2021-03-24T21:59:25.309 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.309 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.310 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:59:25.310 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124602+0000", 2021-03-24T21:59:25.310 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.027032664000000001 2021-03-24T21:59:25.310 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.311 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.311 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:59:25.311 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.151635+0000", 2021-03-24T21:59:25.311 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.4151999999999999e-05 2021-03-24T21:59:25.312 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.312 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.312 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:59:25.312 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.151689+0000", 2021-03-24T21:59:25.312 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.027636099000000001 2021-03-24T21:59:25.313 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:25.313 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:59:25.313 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:25.313 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.314 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.314 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:647 7.c e141/101 7:305739dc:::smithi12556470-36:head v 141'49, mlcod=134'43)", 2021-03-24T21:59:25.314 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:55:10.124445+0000", 2021-03-24T21:59:25.314 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 254.94560993499999, 2021-03-24T21:59:25.315 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.027276857000000002, 2021-03-24T21:59:25.315 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:59:25.315 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:59:25.315 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:59:25.315 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.316 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:59:25.316 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124445+0000", 2021-03-24T21:59:25.316 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:59:25.316 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.316 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.317 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:59:25.317 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124445+0000", 2021-03-24T21:59:25.317 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T21:59:25.317 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.317 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.318 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:59:25.318 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124444+0000", 2021-03-24T21:59:25.318 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.8339999999999999e-06 2021-03-24T21:59:25.318 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.319 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.319 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:59:25.319 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124451+0000", 2021-03-24T21:59:25.319 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.8640000000000002e-06 2021-03-24T21:59:25.320 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.320 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.320 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:59:25.320 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124460+0000", 2021-03-24T21:59:25.320 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5130000000000004e-06 2021-03-24T21:59:25.321 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.321 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.321 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:59:25.321 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124466+0000", 2021-03-24T21:59:25.322 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00030484700000000002 2021-03-24T21:59:25.322 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.322 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.322 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:25.322 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124771+0000", 2021-03-24T21:59:25.323 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.8258e-05 2021-03-24T21:59:25.323 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.323 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.323 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:59:25.324 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.124789+0000", 2021-03-24T21:59:25.324 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.026903249000000001 2021-03-24T21:59:25.324 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.324 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.324 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:59:25.325 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.151692+0000", 2021-03-24T21:59:25.325 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.9533e-05 2021-03-24T21:59:25.325 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.325 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.326 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:59:25.326 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:55:10.151722+0000", 2021-03-24T21:59:25.326 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.027276857000000002 2021-03-24T21:59:25.326 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:25.326 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:59:25.327 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:25.327 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.327 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.327 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:1547 7.11 e211/204 7:8dd64f9e:::smithi12556470-49:head v 211'181, mlcod=210'178)", 2021-03-24T21:59:25.328 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:56:20.149803+0000", 2021-03-24T21:59:25.328 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 184.92025187199999, 2021-03-24T21:59:25.328 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.019109896000000001, 2021-03-24T21:59:25.328 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:59:25.328 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:59:25.329 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:59:25.329 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.329 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:59:25.329 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:20.149803+0000", 2021-03-24T21:59:25.329 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:59:25.330 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.330 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.330 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:59:25.330 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:20.149803+0000", 2021-03-24T21:59:25.331 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T21:59:25.331 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.331 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.331 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:59:25.331 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:20.149802+0000", 2021-03-24T21:59:25.332 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0081225380000000003 2021-03-24T21:59:25.332 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.332 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.332 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:59:25.332 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:20.157924+0000", 2021-03-24T21:59:25.333 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.5210000000000001e-05 2021-03-24T21:59:25.333 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.333 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.333 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:59:25.333 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:20.157950+0000", 2021-03-24T21:59:25.334 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0497e-05 2021-03-24T21:59:25.334 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.334 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.334 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:59:25.335 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:20.157960+0000", 2021-03-24T21:59:25.335 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.3972000000000003e-05 2021-03-24T21:59:25.335 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.335 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.335 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:25.336 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:20.158034+0000", 2021-03-24T21:59:25.336 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.3331000000000003e-05 2021-03-24T21:59:25.336 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.336 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.337 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:59:25.337 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:20.158067+0000", 2021-03-24T21:59:25.337 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.010771836 2021-03-24T21:59:25.337 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.337 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.338 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:59:25.338 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:20.168839+0000", 2021-03-24T21:59:25.338 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.347e-05 2021-03-24T21:59:25.338 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.339 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.339 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:59:25.339 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:20.168913+0000", 2021-03-24T21:59:25.339 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.019109896000000001 2021-03-24T21:59:25.339 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:25.340 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:59:25.340 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:25.340 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.340 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.340 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:1777 7.17 e232/203 7:ef6d6bee:::smithi12556470-24:head v 232'218, mlcod=230'214)", 2021-03-24T21:59:25.341 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:56:41.201926+0000", 2021-03-24T21:59:25.341 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 163.86812841599999, 2021-03-24T21:59:25.341 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.039154920000000003, 2021-03-24T21:59:25.341 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:59:25.342 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:59:25.342 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:59:25.342 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.342 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:59:25.342 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:41.201926+0000", 2021-03-24T21:59:25.343 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:59:25.343 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.343 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.343 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:59:25.343 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:41.201926+0000", 2021-03-24T21:59:25.344 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999995 2021-03-24T21:59:25.344 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.344 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.344 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:59:25.345 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:41.201926+0000", 2021-03-24T21:59:25.345 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00074986499999999995 2021-03-24T21:59:25.345 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.345 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.345 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:59:25.346 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:41.202676+0000", 2021-03-24T21:59:25.346 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5279000000000001e-05 2021-03-24T21:59:25.346 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.346 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.347 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:59:25.347 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:41.202691+0000", 2021-03-24T21:59:25.347 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.8719999999999996e-06 2021-03-24T21:59:25.347 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.347 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.348 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:59:25.348 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:41.202698+0000", 2021-03-24T21:59:25.348 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.9821999999999994e-05 2021-03-24T21:59:25.348 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.348 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.349 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:25.349 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:41.202798+0000", 2021-03-24T21:59:25.349 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.3599000000000001e-05 2021-03-24T21:59:25.349 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.349 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.350 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:59:25.350 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:41.202831+0000", 2021-03-24T21:59:25.350 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.038160301000000001 2021-03-24T21:59:25.350 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.351 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.351 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:59:25.351 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:41.240992+0000", 2021-03-24T21:59:25.351 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.9573000000000005e-05 2021-03-24T21:59:25.352 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.352 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.352 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:59:25.352 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:41.241081+0000", 2021-03-24T21:59:25.352 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.039154920000000003 2021-03-24T21:59:25.353 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:25.353 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:59:25.353 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:25.354 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.354 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.354 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(osd.3.0:522 7.14 e256/204 7:2cd73231:::smithi12556470-20:8f v 256'169, mlcod=251'165)", 2021-03-24T21:59:25.354 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:57:05.147512+0000", 2021-03-24T21:59:25.354 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 139.922542816, 2021-03-24T21:59:25.355 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.01787994, 2021-03-24T21:59:25.355 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:59:25.355 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:59:25.355 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:59:25.356 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.356 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:59:25.356 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.147512+0000", 2021-03-24T21:59:25.356 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:59:25.356 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.357 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.357 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:59:25.357 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.147512+0000", 2021-03-24T21:59:25.357 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T21:59:25.357 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.358 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.358 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:59:25.358 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.147511+0000", 2021-03-24T21:59:25.358 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.1459999999999999e-06 2021-03-24T21:59:25.359 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.359 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.359 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:59:25.359 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.147518+0000", 2021-03-24T21:59:25.359 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.1419999999999993e-06 2021-03-24T21:59:25.360 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.360 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.360 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:59:25.360 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.147526+0000", 2021-03-24T21:59:25.360 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.5629999999999997e-06 2021-03-24T21:59:25.361 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.361 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.361 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:59:25.361 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.147533+0000", 2021-03-24T21:59:25.362 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.4020000000000002e-05 2021-03-24T21:59:25.362 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.362 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.362 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:25.362 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.147577+0000", 2021-03-24T21:59:25.363 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9684999999999999e-05 2021-03-24T21:59:25.363 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.363 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.363 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:59:25.363 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.147597+0000", 2021-03-24T21:59:25.364 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.017761536000000001 2021-03-24T21:59:25.364 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.364 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.364 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:59:25.365 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.165358+0000", 2021-03-24T21:59:25.365 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.3655999999999998e-05 2021-03-24T21:59:25.365 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.365 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.365 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:59:25.366 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.165392+0000", 2021-03-24T21:59:25.366 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.01787994 2021-03-24T21:59:25.366 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:25.366 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:59:25.367 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:25.367 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.367 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.367 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(osd.7.0:209 7.11 e256/204 7:8c1d349c:::smithi12556470-48:8f v 256'256, mlcod=254'253)", 2021-03-24T21:59:25.368 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:57:05.149010+0000", 2021-03-24T21:59:25.368 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 139.921045187, 2021-03-24T21:59:25.368 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.019418392, 2021-03-24T21:59:25.368 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:59:25.368 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:59:25.369 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:59:25.369 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.369 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:59:25.369 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.149010+0000", 2021-03-24T21:59:25.370 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:59:25.370 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.370 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.370 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:59:25.370 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.149010+0000", 2021-03-24T21:59:25.371 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999995 2021-03-24T21:59:25.371 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.371 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.371 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:59:25.372 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.149009+0000", 2021-03-24T21:59:25.372 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.1600000000000003e-06 2021-03-24T21:59:25.372 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.372 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.372 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:59:25.373 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.149015+0000", 2021-03-24T21:59:25.373 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2391999999999999e-05 2021-03-24T21:59:25.373 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.373 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.373 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:59:25.374 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.149028+0000", 2021-03-24T21:59:25.374 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.0729999999999999e-06 2021-03-24T21:59:25.374 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.374 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.375 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:59:25.375 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.149035+0000", 2021-03-24T21:59:25.375 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001466282 2021-03-24T21:59:25.375 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.375 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.376 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:25.376 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.150501+0000", 2021-03-24T21:59:25.376 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.4975000000000001e-05 2021-03-24T21:59:25.376 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.376 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.377 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:59:25.377 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.150526+0000", 2021-03-24T21:59:25.377 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.017882625999999999 2021-03-24T21:59:25.377 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.378 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.378 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:59:25.378 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.168409+0000", 2021-03-24T21:59:25.378 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9357000000000001e-05 2021-03-24T21:59:25.378 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.379 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.379 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:59:25.379 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.168428+0000", 2021-03-24T21:59:25.379 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.019418392 2021-03-24T21:59:25.380 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:25.380 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:59:25.380 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:25.380 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.380 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.381 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(osd.7.0:210 7.19 e256/204 7:9c4f4ea5:::smithi12556470-41:83 v 256'242, mlcod=255'239)", 2021-03-24T21:59:25.381 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:57:05.149040+0000", 2021-03-24T21:59:25.381 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 139.92101464800001, 2021-03-24T21:59:25.381 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.019366148, 2021-03-24T21:59:25.381 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:59:25.382 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:59:25.382 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:59:25.382 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.382 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:59:25.383 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.149040+0000", 2021-03-24T21:59:25.383 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:59:25.383 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.383 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.383 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:59:25.384 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.149040+0000", 2021-03-24T21:59:25.384 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999995 2021-03-24T21:59:25.384 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.384 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.384 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:59:25.385 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.149040+0000", 2021-03-24T21:59:25.385 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.8559999999999996e-06 2021-03-24T21:59:25.385 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.385 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.386 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:59:25.386 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.149049+0000", 2021-03-24T21:59:25.386 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.8119999999999996e-06 2021-03-24T21:59:25.386 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.387 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.387 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:59:25.387 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.149054+0000", 2021-03-24T21:59:25.387 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.349e-06 2021-03-24T21:59:25.387 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.388 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.388 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:59:25.388 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.149058+0000", 2021-03-24T21:59:25.388 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001798797 2021-03-24T21:59:25.388 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.389 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.389 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:25.389 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.150857+0000", 2021-03-24T21:59:25.389 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7183000000000001e-05 2021-03-24T21:59:25.390 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.390 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.390 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:59:25.390 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.150874+0000", 2021-03-24T21:59:25.390 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.017494378000000001 2021-03-24T21:59:25.391 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.391 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.391 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:59:25.391 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.168368+0000", 2021-03-24T21:59:25.392 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.8213000000000002e-05 2021-03-24T21:59:25.392 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.392 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.392 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:59:25.393 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.168406+0000", 2021-03-24T21:59:25.393 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.019366148 2021-03-24T21:59:25.393 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:25.393 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:59:25.393 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:25.394 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.394 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.394 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(osd.7.0:211 7.11 e256/204 7:8eea9237:::smithi12556470-1:83 v 256'258, mlcod=254'253)", 2021-03-24T21:59:25.394 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:57:05.149514+0000", 2021-03-24T21:59:25.394 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 139.92054062, 2021-03-24T21:59:25.395 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.019646961000000001, 2021-03-24T21:59:25.395 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:59:25.395 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:59:25.395 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:59:25.396 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.396 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:59:25.396 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.149514+0000", 2021-03-24T21:59:25.396 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:59:25.396 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.397 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.397 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:59:25.397 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.149514+0000", 2021-03-24T21:59:25.397 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T21:59:25.397 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.398 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.398 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:59:25.398 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.149513+0000", 2021-03-24T21:59:25.398 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.3130000000000004e-06 2021-03-24T21:59:25.399 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.399 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.399 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:59:25.399 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.149520+0000", 2021-03-24T21:59:25.400 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2040000000000002e-06 2021-03-24T21:59:25.400 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.400 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.400 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:59:25.400 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.149526+0000", 2021-03-24T21:59:25.401 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2809999999999999e-06 2021-03-24T21:59:25.401 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.401 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.401 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:59:25.401 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.149531+0000", 2021-03-24T21:59:25.402 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.015713127 2021-03-24T21:59:25.402 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.402 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.402 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:25.403 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.165244+0000", 2021-03-24T21:59:25.403 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.3161999999999999e-05 2021-03-24T21:59:25.403 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.403 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.403 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:59:25.404 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.165268+0000", 2021-03-24T21:59:25.404 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0038786039999999999 2021-03-24T21:59:25.404 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.404 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.404 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:59:25.405 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.169146+0000", 2021-03-24T21:59:25.405 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.4987999999999999e-05 2021-03-24T21:59:25.405 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.405 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.406 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:59:25.406 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.169161+0000", 2021-03-24T21:59:25.406 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.019646961000000001 2021-03-24T21:59:25.406 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:25.406 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:59:25.407 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:25.407 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.407 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.407 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(osd.7.0:212 7.19 e256/204 7:992cb487:::smithi12556470-5:92 v 256'244, mlcod=255'239)", 2021-03-24T21:59:25.407 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:57:05.151829+0000", 2021-03-24T21:59:25.408 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 139.91822625, 2021-03-24T21:59:25.408 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.017315692000000001, 2021-03-24T21:59:25.408 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:59:25.408 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:59:25.409 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:59:25.409 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.409 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:59:25.409 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.151829+0000", 2021-03-24T21:59:25.409 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:59:25.410 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.410 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.410 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:59:25.410 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.151829+0000", 2021-03-24T21:59:25.410 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T21:59:25.411 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.411 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.411 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:59:25.411 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.151828+0000", 2021-03-24T21:59:25.412 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.8889999999999996e-06 2021-03-24T21:59:25.412 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.412 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.412 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:59:25.412 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.151835+0000", 2021-03-24T21:59:25.413 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2439999999999993e-06 2021-03-24T21:59:25.413 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.413 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.413 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:59:25.414 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.151843+0000", 2021-03-24T21:59:25.414 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.8549999999999998e-06 2021-03-24T21:59:25.414 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.414 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.414 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:59:25.415 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.151849+0000", 2021-03-24T21:59:25.415 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.013383232 2021-03-24T21:59:25.415 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.415 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.416 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:25.416 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.165232+0000", 2021-03-24T21:59:25.416 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.2079e-05 2021-03-24T21:59:25.416 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.416 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.417 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:59:25.417 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.165254+0000", 2021-03-24T21:59:25.417 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0038708789999999998 2021-03-24T21:59:25.417 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.418 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.418 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:59:25.418 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.169125+0000", 2021-03-24T21:59:25.418 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9364e-05 2021-03-24T21:59:25.418 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.419 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.419 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:59:25.419 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:05.169144+0000", 2021-03-24T21:59:25.419 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.017315692000000001 2021-03-24T21:59:25.419 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:25.420 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:59:25.420 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:25.420 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.420 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.420 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:2345 7.11 e269/204 7:8c1d349c:::smithi12556470-48:head v 269'289, mlcod=268'285)", 2021-03-24T21:59:25.421 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:57:19.832048+0000", 2021-03-24T21:59:25.421 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 125.238006916, 2021-03-24T21:59:25.421 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.014660479000000001, 2021-03-24T21:59:25.421 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:59:25.422 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:59:25.422 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:59:25.422 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.422 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:59:25.423 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:19.832048+0000", 2021-03-24T21:59:25.423 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:59:25.423 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.423 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.423 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:59:25.424 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:19.832048+0000", 2021-03-24T21:59:25.424 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T21:59:25.424 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.424 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.424 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:59:25.425 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:19.832046+0000", 2021-03-24T21:59:25.425 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0077522700000000003 2021-03-24T21:59:25.425 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.425 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.426 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:59:25.426 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:19.839798+0000", 2021-03-24T21:59:25.426 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0933000000000001e-05 2021-03-24T21:59:25.426 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.426 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.427 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:59:25.427 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:19.839819+0000", 2021-03-24T21:59:25.427 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2069999999999995e-06 2021-03-24T21:59:25.427 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.427 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.428 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:59:25.428 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:19.839827+0000", 2021-03-24T21:59:25.428 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.8728999999999997e-05 2021-03-24T21:59:25.428 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.429 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.429 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:25.429 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:19.839916+0000", 2021-03-24T21:59:25.429 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1643e-05 2021-03-24T21:59:25.429 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.430 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.430 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:59:25.430 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:19.839968+0000", 2021-03-24T21:59:25.430 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0066941839999999997 2021-03-24T21:59:25.430 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.431 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.431 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:59:25.431 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:19.846662+0000", 2021-03-24T21:59:25.431 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6529999999999997e-05 2021-03-24T21:59:25.432 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.432 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.432 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:59:25.432 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:19.846708+0000", 2021-03-24T21:59:25.432 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.014660479000000001 2021-03-24T21:59:25.433 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:25.433 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:59:25.433 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:25.433 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.433 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.434 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:2557 7.11 e290/204 7:8dd64f9e:::smithi12556470-49:head v 290'308, mlcod=278'304)", 2021-03-24T21:59:25.434 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:57:40.204468+0000", 2021-03-24T21:59:25.434 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 104.865586924, 2021-03-24T21:59:25.434 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.052820996000000002, 2021-03-24T21:59:25.435 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:59:25.435 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:59:25.435 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:59:25.435 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.435 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:59:25.436 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.204468+0000", 2021-03-24T21:59:25.436 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:59:25.436 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.436 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.436 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:59:25.437 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.204468+0000", 2021-03-24T21:59:25.437 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999952 2021-03-24T21:59:25.437 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.437 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.438 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:59:25.438 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.204463+0000", 2021-03-24T21:59:25.438 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00085829000000000005 2021-03-24T21:59:25.438 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.438 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.439 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:59:25.439 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.205321+0000", 2021-03-24T21:59:25.439 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.4329999999999998e-05 2021-03-24T21:59:25.439 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.439 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.440 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:59:25.440 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.205356+0000", 2021-03-24T21:59:25.440 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2873e-05 2021-03-24T21:59:25.440 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.441 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.441 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:59:25.441 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.205369+0000", 2021-03-24T21:59:25.441 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.8431000000000007e-05 2021-03-24T21:59:25.441 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.442 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.442 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:25.442 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.205467+0000", 2021-03-24T21:59:25.442 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.0049000000000002e-05 2021-03-24T21:59:25.443 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.443 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.443 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:59:25.443 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.205507+0000", 2021-03-24T21:59:25.443 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.051707234999999997 2021-03-24T21:59:25.444 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.444 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.444 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:59:25.444 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.257214+0000", 2021-03-24T21:59:25.444 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.4583000000000001e-05 2021-03-24T21:59:25.445 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.445 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.445 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:59:25.445 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.257289+0000", 2021-03-24T21:59:25.446 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.052820996000000002 2021-03-24T21:59:25.446 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:25.446 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:59:25.446 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:25.446 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.447 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.447 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:2558 7.11 e290/204 7:8dd64f9e:::smithi12556470-49:head v 290'309, mlcod=278'304)", 2021-03-24T21:59:25.447 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:57:40.209426+0000", 2021-03-24T21:59:25.447 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 104.86062921200001, 2021-03-24T21:59:25.447 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.047890466999999999, 2021-03-24T21:59:25.448 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:59:25.448 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:59:25.448 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:59:25.448 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.449 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:59:25.449 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.209426+0000", 2021-03-24T21:59:25.449 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:59:25.449 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.449 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.450 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:59:25.450 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.209426+0000", 2021-03-24T21:59:25.450 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T21:59:25.450 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.450 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.451 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:59:25.451 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.209424+0000", 2021-03-24T21:59:25.451 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00021029499999999999 2021-03-24T21:59:25.451 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.452 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.452 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:59:25.452 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.209634+0000", 2021-03-24T21:59:25.452 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9154999999999999e-05 2021-03-24T21:59:25.452 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.453 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.453 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:59:25.453 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.209653+0000", 2021-03-24T21:59:25.453 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.9319999999999999e-06 2021-03-24T21:59:25.454 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.454 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.454 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:59:25.454 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.209660+0000", 2021-03-24T21:59:25.454 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6637e-05 2021-03-24T21:59:25.455 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.455 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.455 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:25.455 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.209717+0000", 2021-03-24T21:59:25.455 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0486e-05 2021-03-24T21:59:25.456 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.456 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.456 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:59:25.456 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.209737+0000", 2021-03-24T21:59:25.457 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.047558077999999997 2021-03-24T21:59:25.457 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.457 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.457 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:59:25.457 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.257295+0000", 2021-03-24T21:59:25.458 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0899000000000001e-05 2021-03-24T21:59:25.458 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.458 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.458 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:59:25.459 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.257316+0000", 2021-03-24T21:59:25.459 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.047890466999999999 2021-03-24T21:59:25.459 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:25.459 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:59:25.459 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:25.460 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.460 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.460 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:2559 7.11 e290/204 7:8dd64f9e:::smithi12556470-49:head v 290'310, mlcod=278'304)", 2021-03-24T21:59:25.460 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:57:40.210587+0000", 2021-03-24T21:59:25.460 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 104.859467536, 2021-03-24T21:59:25.461 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.046757605000000001, 2021-03-24T21:59:25.461 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:59:25.461 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:59:25.461 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:59:25.462 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.462 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:59:25.462 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.210587+0000", 2021-03-24T21:59:25.462 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:59:25.462 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.463 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.463 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:59:25.463 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.210587+0000", 2021-03-24T21:59:25.463 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T21:59:25.464 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.464 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.464 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:59:25.464 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.210585+0000", 2021-03-24T21:59:25.464 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.3989999999999993e-06 2021-03-24T21:59:25.465 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.465 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.465 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:59:25.465 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.210595+0000", 2021-03-24T21:59:25.466 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2702e-05 2021-03-24T21:59:25.466 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.466 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.466 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:59:25.466 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.210607+0000", 2021-03-24T21:59:25.467 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.3879999999999999e-06 2021-03-24T21:59:25.467 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.467 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.467 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:59:25.468 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.210613+0000", 2021-03-24T21:59:25.468 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.1072000000000002e-05 2021-03-24T21:59:25.468 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.468 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.468 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:25.469 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.210654+0000", 2021-03-24T21:59:25.469 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.5398999999999999e-05 2021-03-24T21:59:25.469 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.469 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.469 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:59:25.470 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.210679+0000", 2021-03-24T21:59:25.470 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.046641477000000001 2021-03-24T21:59:25.470 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.470 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.471 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:59:25.471 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.257321+0000", 2021-03-24T21:59:25.471 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.4128999999999999e-05 2021-03-24T21:59:25.471 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.471 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.472 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:59:25.472 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.257345+0000", 2021-03-24T21:59:25.472 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.046757605000000001 2021-03-24T21:59:25.472 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:25.472 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:59:25.473 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:25.473 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.473 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.473 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:2567 7.c e290/275 7:32974111:::smithi12556470-2:head v 290'276, mlcod=290'273)", 2021-03-24T21:59:25.473 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:57:40.226884+0000", 2021-03-24T21:59:25.474 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 104.843170988, 2021-03-24T21:59:25.474 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.030407558000000001, 2021-03-24T21:59:25.474 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:59:25.474 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:59:25.475 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:59:25.475 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.475 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:59:25.475 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.226884+0000", 2021-03-24T21:59:25.475 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:59:25.476 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.476 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.476 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:59:25.476 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.226884+0000", 2021-03-24T21:59:25.476 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999971 2021-03-24T21:59:25.477 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.477 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.477 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:59:25.477 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.226881+0000", 2021-03-24T21:59:25.477 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0017965539999999999 2021-03-24T21:59:25.478 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.478 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.478 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:59:25.478 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.228677+0000", 2021-03-24T21:59:25.479 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.3691999999999999e-05 2021-03-24T21:59:25.479 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.479 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.479 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:59:25.479 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.228701+0000", 2021-03-24T21:59:25.480 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.2380000000000003e-06 2021-03-24T21:59:25.480 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.480 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.480 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:59:25.480 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.228710+0000", 2021-03-24T21:59:25.481 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.9072999999999994e-05 2021-03-24T21:59:25.481 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.481 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.481 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:25.482 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.228779+0000", 2021-03-24T21:59:25.482 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3399000000000002e-05 2021-03-24T21:59:25.482 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.482 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.482 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:59:25.483 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.228823+0000", 2021-03-24T21:59:25.483 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.028418450000000001 2021-03-24T21:59:25.483 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.483 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.483 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:59:25.484 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.257241+0000", 2021-03-24T21:59:25.484 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.0179000000000002e-05 2021-03-24T21:59:25.484 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.484 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.485 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:59:25.485 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.257291+0000", 2021-03-24T21:59:25.485 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.030407558000000001 2021-03-24T21:59:25.485 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:25.485 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:59:25.486 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:25.486 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.486 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.486 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(osd.7.0:328 7.19 e305/204 7:9b48f357:::smithi12556470-17:c3 v 305'295, mlcod=303'290)", 2021-03-24T21:59:25.486 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:57:55.175676+0000", 2021-03-24T21:59:25.487 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 89.894379068000006, 2021-03-24T21:59:25.487 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.014762582999999999, 2021-03-24T21:59:25.487 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:59:25.487 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:59:25.487 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:59:25.488 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.488 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:59:25.488 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:55.175676+0000", 2021-03-24T21:59:25.488 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:59:25.489 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.489 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.489 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:59:25.489 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:55.175676+0000", 2021-03-24T21:59:25.489 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T21:59:25.490 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.490 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.490 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:59:25.490 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:55.175674+0000", 2021-03-24T21:59:25.490 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.8054999999999999e-05 2021-03-24T21:59:25.491 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.491 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.491 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:59:25.491 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:55.175692+0000", 2021-03-24T21:59:25.491 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2050000000000002e-06 2021-03-24T21:59:25.492 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.492 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.492 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:59:25.492 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:55.175701+0000", 2021-03-24T21:59:25.493 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.421e-06 2021-03-24T21:59:25.493 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.493 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.493 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:59:25.493 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:55.175707+0000", 2021-03-24T21:59:25.494 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3652999999999997e-05 2021-03-24T21:59:25.494 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.494 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.494 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:25.495 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:55.175751+0000", 2021-03-24T21:59:25.495 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9961000000000001e-05 2021-03-24T21:59:25.495 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.495 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.495 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:59:25.496 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:55.175771+0000", 2021-03-24T21:59:25.496 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.014611031999999999 2021-03-24T21:59:25.496 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.496 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.496 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:59:25.497 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:55.190382+0000", 2021-03-24T21:59:25.497 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6721999999999997e-05 2021-03-24T21:59:25.497 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.497 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.498 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:59:25.498 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:55.190438+0000", 2021-03-24T21:59:25.498 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.014762582999999999 2021-03-24T21:59:25.498 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:25.498 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:59:25.499 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:25.499 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.499 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.499 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:2915 7.14 e320/204 7:2db4b4db:::smithi12556470-25:head v 320'239, mlcod=307'235)", 2021-03-24T21:59:25.499 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:58:10.236819+0000", 2021-03-24T21:59:25.500 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 74.833235905999999, 2021-03-24T21:59:25.500 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018306156000000001, 2021-03-24T21:59:25.500 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:59:25.500 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:59:25.501 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:59:25.501 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.501 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:59:25.501 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:10.236819+0000", 2021-03-24T21:59:25.501 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:59:25.502 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.502 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.502 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:59:25.502 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:10.236819+0000", 2021-03-24T21:59:25.502 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999995 2021-03-24T21:59:25.503 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.503 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.503 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:59:25.503 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:10.236818+0000", 2021-03-24T21:59:25.503 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.0869999999999996e-06 2021-03-24T21:59:25.504 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.504 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.504 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:59:25.504 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:10.236827+0000", 2021-03-24T21:59:25.504 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.3009999999999997e-05 2021-03-24T21:59:25.505 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.505 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.505 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:59:25.505 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:10.236860+0000", 2021-03-24T21:59:25.506 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2979000000000001e-05 2021-03-24T21:59:25.506 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.506 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.506 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:59:25.506 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:10.236873+0000", 2021-03-24T21:59:25.507 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012326899999999999 2021-03-24T21:59:25.507 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.507 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.507 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:25.507 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:10.236997+0000", 2021-03-24T21:59:25.508 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.7005000000000001e-05 2021-03-24T21:59:25.508 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.508 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.508 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:59:25.509 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:10.237044+0000", 2021-03-24T21:59:25.509 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018022888000000001 2021-03-24T21:59:25.509 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.509 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.509 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:59:25.510 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:10.255067+0000", 2021-03-24T21:59:25.510 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.8437999999999999e-05 2021-03-24T21:59:25.510 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.510 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.511 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:59:25.511 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:10.255125+0000", 2021-03-24T21:59:25.511 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018306156000000001 2021-03-24T21:59:25.511 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:25.511 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:59:25.512 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:25.512 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.512 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.512 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:3411 7.11 e360/204 7:8c1d349c:::smithi12556470-48:head v 360'386, mlcod=358'383)", 2021-03-24T21:59:25.513 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:58:50.291627+0000", 2021-03-24T21:59:25.513 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 34.778427393000001, 2021-03-24T21:59:25.513 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.014809836, 2021-03-24T21:59:25.513 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:59:25.513 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:59:25.514 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:59:25.514 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.514 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:59:25.514 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:50.291627+0000", 2021-03-24T21:59:25.514 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:59:25.515 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.515 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.515 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:59:25.515 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:50.291627+0000", 2021-03-24T21:59:25.516 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999995 2021-03-24T21:59:25.516 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.516 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.516 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:59:25.517 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:50.291627+0000", 2021-03-24T21:59:25.517 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0062836460000000004 2021-03-24T21:59:25.517 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.517 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.517 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:59:25.518 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:50.297911+0000", 2021-03-24T21:59:25.518 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.5154999999999999e-05 2021-03-24T21:59:25.518 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.518 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.518 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:59:25.519 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:50.297936+0000", 2021-03-24T21:59:25.519 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.3556999999999997e-05 2021-03-24T21:59:25.519 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.519 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.519 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:59:25.520 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:50.297969+0000", 2021-03-24T21:59:25.520 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6843999999999999e-05 2021-03-24T21:59:25.520 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.520 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.521 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:25.521 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:50.298036+0000", 2021-03-24T21:59:25.521 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.7399e-05 2021-03-24T21:59:25.521 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.521 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.522 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:59:25.522 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:50.298064+0000", 2021-03-24T21:59:25.522 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0083184680000000007 2021-03-24T21:59:25.522 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.522 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.523 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:59:25.523 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:50.306382+0000", 2021-03-24T21:59:25.523 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5145000000000001e-05 2021-03-24T21:59:25.523 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:25.524 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:25.524 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:59:25.524 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:50.306437+0000", 2021-03-24T21:59:25.524 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.014809836 2021-03-24T21:59:25.524 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:25.525 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:59:25.525 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:25.525 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:25.525 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:59:25.525 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-24T21:59:25.526 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:25.527 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:25 smithi142 conmon[20125]: cluster 2021-03-24T21:59:24.240417+0000 mon.a (mon.0) 1183 : cluster [DBG] osdmap e394: 8 total, 7 up, 6 in 2021-03-24T21:59:25.527 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:25 smithi142 conmon[20125]: cluster 2021-03-24T21:59:25.202360+0000 mon.a (mon.0) 1184 : cluster [WRN] Health check update: Degraded data redundancy: 262/1422 objects degraded (18.425%), 38 pgs degraded (PG_DEGRADED) 2021-03-24T21:59:25.528 INFO:tasks.rados.rados.0.smithi125.stdout:1444: done (2 left) 2021-03-24T21:59:25.528 INFO:tasks.rados.rados.0.smithi125.stdout:1445: done (1 left) 2021-03-24T21:59:25.528 INFO:tasks.rados.rados.0.smithi125.stdout:1446: done (0 left) 2021-03-24T21:59:25.529 INFO:tasks.rados.rados.0.smithi125.stdout:1447: snap_create 2021-03-24T21:59:25.579 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:25.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:25.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:59:25.650 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:25 smithi125 conmon[24927]: cluster 2021-03-24T21:59: 2021-03-24T21:59:25.650 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:25 smithi125 conmon[24927]: 24.240417+0000 mon.a (mon.0) 1183 : cluster [DBG] osdmap e394: 8 total, 7 up, 6 in 2021-03-24T21:59:25.651 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:25 smithi125 conmon[24927]: cluster 2021-03-24T21:59 2021-03-24T21:59:25.651 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:25 smithi125 conmon[24927]: :25.202360+0000 mon.a (mon.0) 1184 : cluster [WRN] Health check update: Degraded data redundancy: 262/1422 objects degraded (18.425%), 38 pgs degraded (PG_DEGRADED) 2021-03-24T21:59:25.651 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:25 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:59:25.651 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:25 smithi125 conmon[19936]: 59:24.240417+0000 mon.a (mon.0) 1183 : cluster [DBG] osdmap e394: 8 total, 7 up, 6 in 2021-03-24T21:59:25.652 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:25 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T21:59:25.652 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:25 smithi125 conmon[19936]: T21:59:25.202360+0000 2021-03-24T21:59:25.652 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:25 smithi125 conmon[19936]: mon.a (mon.0) 1184 : 2021-03-24T21:59:25.652 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:25 smithi125 conmon[19936]: cluster [WRN] Health check update: Degraded data redundancy: 262/1422 objects degraded (18.425%), 38 pgs degraded (PG_DEGRADED) 2021-03-24T21:59:25.684 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:25.739 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:25.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:25.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:59:25.845 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:25.907 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:25.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:25.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:59:25.921 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":752,"stamp":"2021-03-24T21:59:23.228026+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":350739738,"num_objects":474,"num_object_clones":107,"num_object_copies":1422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":262,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":474,"num_whiteouts":7,"num_read":3229,"num_read_kb":1796465,"num_write":1739,"num_write_kb":469307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":474,"num_bytes_recovered":94555393,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8214,"ondisk_log_size":8214,"up":440,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":534,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":844668,"kb_used_data":698124,"kb_used_omap":0,"kb_used_meta":146496,"kb_avail":561626244,"statfs":{"total":575970213888,"available":575105273856,"internally_reserved":0,"allocated":714878976,"data_stored":914898025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":150011904},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2941291,"num_objects":-1,"num_object_clones":-1,"num_object_copies":-3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":9,"num_read_kb":9825,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.805439"},"pg_stats":[{"pgid":"3.29","version":"0'0","reported_seq":"14","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.408738+0000","last_change":"2021-03-24T21:59:14.374618+0000","last_active":"2021-03-24T21:59:17.408738+0000","last_peered":"2021-03-24T21:59:17.408738+0000","last_clean":"2021-03-24T21:59:17.408738+0000","last_became_active":"2021-03-24T21:59:14.374095+0000","last_became_peered":"2021-03-24T21:59:14.374095+0000","last_unstale":"2021-03-24T21:59:17.408738+0000","last_undegraded":"2021-03-24T21:59:17.408738+0000","last_fullsized":"2021-03-24T21:59:17.408738+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:55.592795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:55.592795+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":"370","reported_epoch":"387","state":"active+clean+remapped","last_fresh":"2021-03-24T21:59:17.408727+0000","last_change":"2021-03-24T21:59:15.357350+0000","last_active":"2021-03-24T21:59:17.408727+0000","last_peered":"2021-03-24T21:59:17.408727+0000","last_clean":"2021-03-24T21:59:17.408727+0000","last_became_active":"2021-03-24T21:59:15.356790+0000","last_became_peered":"2021-03-24T21:59:15.356790+0000","last_unstale":"2021-03-24T21:59:17.408727+0000","last_undegraded":"2021-03-24T21:59:17.408727+0000","last_fullsized":"2021-03-24T21:59:17.408727+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:39.439170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:39.439170+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2,4],"avail_no_missing":["3","2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"89'2","reported_seq":"55","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.409082+0000","last_change":"2021-03-24T21:59:14.374798+0000","last_active":"2021-03-24T21:59:17.409082+0000","last_peered":"2021-03-24T21:59:17.409082+0000","last_clean":"2021-03-24T21:59:17.409082+0000","last_became_active":"2021-03-24T21:59:14.374418+0000","last_became_peered":"2021-03-24T21:59:14.374418+0000","last_unstale":"2021-03-24T21:59:17.409082+0000","last_undegraded":"2021-03-24T21:59:17.409082+0000","last_fullsized":"2021-03-24T21:59:17.409082+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":6,"last_scrub":"89'2","last_scrub_stamp":"2021-03-24T21:57:58.491830+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2021-03-24T21:57:58.491830+0000","last_clean_scrub_stamp":"2021-03-24T21:57:58.491830+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":179,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":"373","reported_epoch":"387","state":"active+clean+remapped","last_fresh":"2021-03-24T21:59:17.421174+0000","last_change":"2021-03-24T21:59:15.397856+0000","last_active":"2021-03-24T21:59:17.421174+0000","last_peered":"2021-03-24T21:59:17.421174+0000","last_clean":"2021-03-24T21:59:17.421174+0000","last_became_active":"2021-03-24T21:59:15.397227+0000","last_became_peered":"2021-03-24T21:59:15.397227+0000","last_unstale":"2021-03-24T21:59:17.421174+0000","last_undegraded":"2021-03-24T21:59:17.421174+0000","last_fullsized":"2021-03-24T21:59:17.421174+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:33.533840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:33.533840+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0,4],"avail_no_missing":["3","0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":"15","reported_epoch":"387","state":"active+clean+remapped","last_fresh":"2021-03-24T21:59:17.413434+0000","last_change":"2021-03-24T21:59:15.381648+0000","last_active":"2021-03-24T21:59:17.413434+0000","last_peered":"2021-03-24T21:59:17.413434+0000","last_clean":"2021-03-24T21:59:17.413434+0000","last_became_active":"2021-03-24T21:59:15.381150+0000","last_became_peered":"2021-03-24T21:59:15.381150+0000","last_unstale":"2021-03-24T21:59:17.413434+0000","last_undegraded":"2021-03-24T21:59:17.413434+0000","last_fullsized":"2021-03-24T21:59:17.413434+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:38.304792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:38.304792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3,0],"avail_no_missing":["7","0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"61'1","reported_seq":"79","reported_epoch":"387","state":"active+clean+remapped","last_fresh":"2021-03-24T21:59:17.407314+0000","last_change":"2021-03-24T21:59:15.412246+0000","last_active":"2021-03-24T21:59:17.407314+0000","last_peered":"2021-03-24T21:59:17.407314+0000","last_clean":"2021-03-24T21:59:17.407314+0000","last_became_active":"2021-03-24T21:59:15.355186+0000","last_became_peered":"2021-03-24T21:59:15.355186+0000","last_unstale":"2021-03-24T21:59:17.407314+0000","last_undegraded":"2021-03-24T21:59:17.407314+0000","last_fullsized":"2021-03-24T21:59:17.407314+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":6,"last_scrub":"61'1","last_scrub_stamp":"2021-03-24T21:57:12.046206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:12.046206+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":437,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7,2],"avail_no_missing":["0","2","7"],"object_location_counts":[{"shards":"0,2,7","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":"365","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.438585+0000","last_change":"2021-03-24T21:59:14.370910+0000","last_active":"2021-03-24T21:59:17.438585+0000","last_peered":"2021-03-24T21:59:17.438585+0000","last_clean":"2021-03-24T21:59:17.438585+0000","last_became_active":"2021-03-24T21:59:14.370506+0000","last_became_peered":"2021-03-24T21:59:14.370506+0000","last_unstale":"2021-03-24T21:59:17.438585+0000","last_undegraded":"2021-03-24T21:59:17.438585+0000","last_fullsized":"2021-03-24T21:59:17.438585+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:54.381668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:54.381668+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.23","version":"372'279","reported_seq":"1193","reported_epoch":"387","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:59:17.454564+0000","last_change":"2021-03-24T21:58:31.229425+0000","last_active":"2021-03-24T21:59:17.454564+0000","last_peered":"2021-03-24T21:59:17.454564+0000","last_clean":"2021-03-24T21:58:29.207011+0000","last_became_active":"2021-03-24T21:58:31.229425+0000","last_became_peered":"2021-03-24T21:58:31.229425+0000","last_unstale":"2021-03-24T21:59:17.454564+0000","last_undegraded":"2021-03-24T21:58:31.197028+0000","last_fullsized":"2021-03-24T21:58:31.196881+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":279,"ondisk_log_size":279,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":19,"stat_sum":{"num_bytes":3437081,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":97,"num_read_kb":117123,"num_write":58,"num_write_kb":21228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":3418953,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":"14","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.418102+0000","last_change":"2021-03-24T21:59:14.353817+0000","last_active":"2021-03-24T21:59:17.418102+0000","last_peered":"2021-03-24T21:59:17.418102+0000","last_clean":"2021-03-24T21:59:17.418102+0000","last_became_active":"2021-03-24T21:59:14.353387+0000","last_became_peered":"2021-03-24T21:59:14.353387+0000","last_unstale":"2021-03-24T21:59:17.418102+0000","last_undegraded":"2021-03-24T21:59:17.418102+0000","last_fullsized":"2021-03-24T21:59:17.418102+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:28.165408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:28.165408+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.22","version":"386'81","reported_seq":"307","reported_epoch":"391","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:59:21.211870+0000","last_change":"2021-03-24T21:58:31.208603+0000","last_active":"2021-03-24T21:59:21.211870+0000","last_peered":"2021-03-24T21:59:21.211870+0000","last_clean":"2021-03-24T21:58:29.213948+0000","last_became_active":"2021-03-24T21:58:31.208603+0000","last_became_peered":"2021-03-24T21:58:31.208603+0000","last_unstale":"2021-03-24T21:59:21.211870+0000","last_undegraded":"2021-03-24T21:58:31.191893+0000","last_fullsized":"2021-03-24T21:58:31.191490+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":81,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":21,"stat_sum":{"num_bytes":2958098,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":14,"num_read_kb":8520,"num_write":20,"num_write_kb":4979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":5774642,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":3}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"69'13","reported_seq":"426","reported_epoch":"391","state":"active+clean","last_fresh":"2021-03-24T21:59:21.214541+0000","last_change":"2021-03-24T21:58:27.722715+0000","last_active":"2021-03-24T21:59:21.214541+0000","last_peered":"2021-03-24T21:59:21.214541+0000","last_clean":"2021-03-24T21:59:21.214541+0000","last_became_active":"2021-03-24T21:54:38.127115+0000","last_became_peered":"2021-03-24T21:54:38.127115+0000","last_unstale":"2021-03-24T21:59:21.214541+0000","last_undegraded":"2021-03-24T21:59:21.214541+0000","last_fullsized":"2021-03-24T21:59:21.214541+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2021-03-24T21:58:27.722546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:58:27.722546+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":13,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"353","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.453506+0000","last_change":"2021-03-24T21:59:17.264251+0000","last_active":"2021-03-24T21:59:17.453506+0000","last_peered":"2021-03-24T21:59:17.453506+0000","last_clean":"2021-03-24T21:59:17.453506+0000","last_became_active":"2021-03-24T21:51:45.375830+0000","last_became_peered":"2021-03-24T21:51:45.375830+0000","last_unstale":"2021-03-24T21:59:17.453506+0000","last_undegraded":"2021-03-24T21:59:17.453506+0000","last_fullsized":"2021-03-24T21:59:17.453506+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:17.264089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:59:17.264089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"343'5","reported_seq":"451","reported_epoch":"390","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:59:20.214268+0000","last_change":"2021-03-24T21:58:31.205289+0000","last_active":"2021-03-24T21:59:20.214268+0000","last_peered":"2021-03-24T21:59:20.214268+0000","last_clean":"2021-03-24T21:58:29.217483+0000","last_became_active":"2021-03-24T21:58:31.205289+0000","last_became_peered":"2021-03-24T21:58:31.205289+0000","last_unstale":"2021-03-24T21:59:20.214268+0000","last_undegraded":"2021-03-24T21:58:31.194525+0000","last_fullsized":"2021-03-24T21:58:31.194414+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2021-03-24T21:57:54.241842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:57:54.241842+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[{"shards":"2,7","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"366","reported_epoch":"391","state":"active+undersized","last_fresh":"2021-03-24T21:59:21.217175+0000","last_change":"2021-03-24T21:58:31.202498+0000","last_active":"2021-03-24T21:59:21.217175+0000","last_peered":"2021-03-24T21:59:21.217175+0000","last_clean":"2021-03-24T21:58:29.218000+0000","last_became_active":"2021-03-24T21:58:31.202498+0000","last_became_peered":"2021-03-24T21:58:31.202498+0000","last_unstale":"2021-03-24T21:59:21.217175+0000","last_undegraded":"2021-03-24T21:59:21.217175+0000","last_fullsized":"2021-03-24T21:58:31.194729+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:08.985694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:08.985694+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"387'177","reported_seq":"1131","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.495016+0000","last_change":"2021-03-24T21:59:17.495016+0000","last_active":"2021-03-24T21:59:17.495016+0000","last_peered":"2021-03-24T21:59:17.495016+0000","last_clean":"2021-03-24T21:59:17.495016+0000","last_became_active":"2021-03-24T21:57:26.178185+0000","last_became_peered":"2021-03-24T21:57:26.178185+0000","last_unstale":"2021-03-24T21:59:17.495016+0000","last_undegraded":"2021-03-24T21:59:17.495016+0000","last_fullsized":"2021-03-24T21:59:17.495016+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'171","last_scrub_stamp":"2021-03-24T21:59:08.017874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:59:08.017874+0000","log_size":177,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8247162,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":16,"num_read_kb":19232,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"d6","length":"1"},{"start":"df","length":"1"},{"start":"e1","length":"1"},{"start":"e8","length":"1"},{"start":"ea","length":"4"},{"start":"ef","length":"1"},{"start":"f2","length":"4"},{"start":"fb","length":"3"},{"start":"109","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"}]},{"pgid":"4.d","version":"69'21","reported_seq":"427","reported_epoch":"391","state":"active+clean","last_fresh":"2021-03-24T21:59:21.214193+0000","last_change":"2021-03-24T21:59:05.674664+0000","last_active":"2021-03-24T21:59:21.214193+0000","last_peered":"2021-03-24T21:59:21.214193+0000","last_clean":"2021-03-24T21:59:21.214193+0000","last_became_active":"2021-03-24T21:54:38.144020+0000","last_became_peered":"2021-03-24T21:54:38.144020+0000","last_unstale":"2021-03-24T21:59:21.214193+0000","last_undegraded":"2021-03-24T21:59:21.214193+0000","last_fullsized":"2021-03-24T21:59:21.214193+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'21","last_scrub_stamp":"2021-03-24T21:59:05.674550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:59:05.674550+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":23,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"326","reported_epoch":"391","state":"active+undersized","last_fresh":"2021-03-24T21:59:21.212654+0000","last_change":"2021-03-24T21:58:31.223148+0000","last_active":"2021-03-24T21:59:21.212654+0000","last_peered":"2021-03-24T21:59:21.212654+0000","last_clean":"2021-03-24T21:58:29.214275+0000","last_became_active":"2021-03-24T21:58:31.223148+0000","last_became_peered":"2021-03-24T21:58:31.223148+0000","last_unstale":"2021-03-24T21:59:21.212654+0000","last_undegraded":"2021-03-24T21:59:21.212654+0000","last_fullsized":"2021-03-24T21:58:31.188534+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:50.522344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:55:01.169461+0000","last_clean_scrub_stamp":"2021-03-24T21:57:50.522344+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"315","reported_epoch":"391","state":"active+undersized","last_fresh":"2021-03-24T21:59:21.214395+0000","last_change":"2021-03-24T21:58:31.207750+0000","last_active":"2021-03-24T21:59:21.214395+0000","last_peered":"2021-03-24T21:59:21.214395+0000","last_clean":"2021-03-24T21:58:29.215104+0000","last_became_active":"2021-03-24T21:58:31.207750+0000","last_became_peered":"2021-03-24T21:58:31.207750+0000","last_unstale":"2021-03-24T21:59:21.214395+0000","last_undegraded":"2021-03-24T21:59:21.214395+0000","last_fullsized":"2021-03-24T21:58:31.192794+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:07.977138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:07.977138+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"153","reported_epoch":"387","state":"active+undersized","last_fresh":"2021-03-24T21:59:17.437942+0000","last_change":"2021-03-24T21:58:31.228194+0000","last_active":"2021-03-24T21:59:17.437942+0000","last_peered":"2021-03-24T21:59:17.437942+0000","last_clean":"2021-03-24T21:58:29.208259+0000","last_became_active":"2021-03-24T21:58:31.228194+0000","last_became_peered":"2021-03-24T21:58:31.228194+0000","last_unstale":"2021-03-24T21:59:17.437942+0000","last_undegraded":"2021-03-24T21:59:17.437942+0000","last_fullsized":"2021-03-24T21:58:31.194809+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:49.970073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:49.970073+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"375'54","reported_seq":"887","reported_epoch":"391","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:59:21.262137+0000","last_change":"2021-03-24T21:58:31.205434+0000","last_active":"2021-03-24T21:59:21.262137+0000","last_peered":"2021-03-24T21:59:21.262137+0000","last_clean":"2021-03-24T21:58:29.217512+0000","last_became_active":"2021-03-24T21:58:31.205434+0000","last_became_peered":"2021-03-24T21:58:31.205434+0000","last_unstale":"2021-03-24T21:59:21.262137+0000","last_undegraded":"2021-03-24T21:58:31.193957+0000","last_fullsized":"2021-03-24T21:58:31.193806+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'45","last_scrub_stamp":"2021-03-24T21:57:27.105464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:57:27.105464+0000","log_size":54,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":20,"stat_sum":{"num_bytes":5075879,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9827,"num_write":10,"num_write_kb":2368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"69'15","reported_seq":"442","reported_epoch":"391","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:59:21.214700+0000","last_change":"2021-03-24T21:58:31.223443+0000","last_active":"2021-03-24T21:59:21.214700+0000","last_peered":"2021-03-24T21:59:21.214700+0000","last_clean":"2021-03-24T21:58:29.217662+0000","last_became_active":"2021-03-24T21:58:31.223443+0000","last_became_peered":"2021-03-24T21:58:31.223443+0000","last_unstale":"2021-03-24T21:59:21.214700+0000","last_undegraded":"2021-03-24T21:58:31.189044+0000","last_fullsized":"2021-03-24T21:58:31.188912+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:56:40.062304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:40.062304+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1476,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":9}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"362","reported_epoch":"391","state":"active+clean","last_fresh":"2021-03-24T21:59:21.211342+0000","last_change":"2021-03-24T21:59:12.267536+0000","last_active":"2021-03-24T21:59:21.211342+0000","last_peered":"2021-03-24T21:59:21.211342+0000","last_clean":"2021-03-24T21:59:21.211342+0000","last_became_active":"2021-03-24T21:59:12.267158+0000","last_became_peered":"2021-03-24T21:59:12.267158+0000","last_unstale":"2021-03-24T21:59:21.211342+0000","last_undegraded":"2021-03-24T21:59:21.211342+0000","last_fullsized":"2021-03-24T21:59:21.211342+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:55.592795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:55.592795+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"332","reported_epoch":"391","state":"active+undersized","last_fresh":"2021-03-24T21:59:21.217058+0000","last_change":"2021-03-24T21:58:31.200145+0000","last_active":"2021-03-24T21:59:21.217058+0000","last_peered":"2021-03-24T21:59:21.217058+0000","last_clean":"2021-03-24T21:58:29.218572+0000","last_became_active":"2021-03-24T21:58:31.200145+0000","last_became_peered":"2021-03-24T21:58:31.200145+0000","last_unstale":"2021-03-24T21:59:21.217058+0000","last_undegraded":"2021-03-24T21:59:21.217058+0000","last_fullsized":"2021-03-24T21:58:31.196922+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:43.970879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:43.970879+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"352","reported_epoch":"387","state":"active+undersized","last_fresh":"2021-03-24T21:59:17.439634+0000","last_change":"2021-03-24T21:58:31.207098+0000","last_active":"2021-03-24T21:59:17.439634+0000","last_peered":"2021-03-24T21:59:17.439634+0000","last_clean":"2021-03-24T21:58:29.207688+0000","last_became_active":"2021-03-24T21:58:31.207098+0000","last_became_peered":"2021-03-24T21:58:31.207098+0000","last_unstale":"2021-03-24T21:59:17.439634+0000","last_undegraded":"2021-03-24T21:59:17.439634+0000","last_fullsized":"2021-03-24T21:58:31.188164+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:59.472393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:55:59.472393+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"387'370","reported_seq":"1294","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.481202+0000","last_change":"2021-03-24T21:59:17.481202+0000","last_active":"2021-03-24T21:59:17.481202+0000","last_peered":"2021-03-24T21:59:17.481202+0000","last_clean":"2021-03-24T21:59:17.481202+0000","last_became_active":"2021-03-24T21:57:26.189692+0000","last_became_peered":"2021-03-24T21:57:26.189692+0000","last_unstale":"2021-03-24T21:59:17.481202+0000","last_undegraded":"2021-03-24T21:59:17.481202+0000","last_fullsized":"2021-03-24T21:59:17.481202+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"384'362","last_scrub_stamp":"2021-03-24T21:59:15.076161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:59:15.076161+0000","log_size":370,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11125745,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"d6","length":"1"},{"start":"df","length":"1"},{"start":"e1","length":"1"},{"start":"e8","length":"1"},{"start":"ea","length":"4"},{"start":"ef","length":"1"},{"start":"f2","length":"4"},{"start":"fb","length":"3"},{"start":"109","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"}]},{"pgid":"4.f","version":"274'26","reported_seq":"365","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.438491+0000","last_change":"2021-03-24T21:57:24.499178+0000","last_active":"2021-03-24T21:59:17.438491+0000","last_peered":"2021-03-24T21:59:17.438491+0000","last_clean":"2021-03-24T21:59:17.438491+0000","last_became_active":"2021-03-24T21:54:38.130536+0000","last_became_peered":"2021-03-24T21:54:38.130536+0000","last_unstale":"2021-03-24T21:59:17.438491+0000","last_undegraded":"2021-03-24T21:59:17.438491+0000","last_fullsized":"2021-03-24T21:59:17.438491+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'26","last_scrub_stamp":"2021-03-24T21:57:24.499090+0000","last_deep_scrub":"274'26","last_deep_scrub_stamp":"2021-03-24T21:57:24.499090+0000","last_clean_scrub_stamp":"2021-03-24T21:57:24.499090+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":18,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":422,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":"361","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.410042+0000","last_change":"2021-03-24T21:59:12.287392+0000","last_active":"2021-03-24T21:59:17.410042+0000","last_peered":"2021-03-24T21:59:17.410042+0000","last_clean":"2021-03-24T21:59:17.410042+0000","last_became_active":"2021-03-24T21:59:12.277682+0000","last_became_peered":"2021-03-24T21:59:12.277682+0000","last_unstale":"2021-03-24T21:59:17.410042+0000","last_undegraded":"2021-03-24T21:59:17.410042+0000","last_fullsized":"2021-03-24T21:59:17.410042+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:39.439170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:39.439170+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"343'3","reported_seq":"442","reported_epoch":"391","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:59:21.215149+0000","last_change":"2021-03-24T21:58:31.208494+0000","last_active":"2021-03-24T21:59:21.215149+0000","last_peered":"2021-03-24T21:59:21.215149+0000","last_clean":"2021-03-24T21:58:29.217762+0000","last_became_active":"2021-03-24T21:58:31.208494+0000","last_became_peered":"2021-03-24T21:58:31.208494+0000","last_unstale":"2021-03-24T21:59:21.215149+0000","last_undegraded":"2021-03-24T21:58:31.194480+0000","last_fullsized":"2021-03-24T21:58:31.194360+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2021-03-24T21:55:59.032170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:55:59.032170+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"112","reported_epoch":"387","state":"active+undersized","last_fresh":"2021-03-24T21:59:17.411860+0000","last_change":"2021-03-24T21:58:31.200312+0000","last_active":"2021-03-24T21:59:17.411860+0000","last_peered":"2021-03-24T21:59:17.411860+0000","last_clean":"2021-03-24T21:54:36.087752+0000","last_became_active":"2021-03-24T21:58:31.200312+0000","last_became_peered":"2021-03-24T21:58:31.200312+0000","last_unstale":"2021-03-24T21:59:17.411860+0000","last_undegraded":"2021-03-24T21:59:17.411860+0000","last_fullsized":"2021-03-24T21:58:31.192375+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:23.176750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:57:23.176750+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.c","version":"387'411","reported_seq":"1336","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.495124+0000","last_change":"2021-03-24T21:59:17.495124+0000","last_active":"2021-03-24T21:59:17.495124+0000","last_peered":"2021-03-24T21:59:17.495124+0000","last_clean":"2021-03-24T21:59:17.495124+0000","last_became_active":"2021-03-24T21:57:26.174174+0000","last_became_peered":"2021-03-24T21:57:26.174174+0000","last_unstale":"2021-03-24T21:59:17.495124+0000","last_undegraded":"2021-03-24T21:59:17.495124+0000","last_fullsized":"2021-03-24T21:59:17.495124+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'393","last_scrub_stamp":"2021-03-24T21:59:12.370455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:59:12.370455+0000","log_size":411,"ondisk_log_size":411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26728086,"num_objects":9,"num_object_clones":5,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":17,"num_read_kb":22500,"num_write":4,"num_write_kb":1017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d6","length":"1"},{"start":"df","length":"1"},{"start":"e1","length":"1"},{"start":"e8","length":"1"},{"start":"ea","length":"4"},{"start":"ef","length":"1"},{"start":"f2","length":"4"},{"start":"fb","length":"3"},{"start":"109","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"}]},{"pgid":"5.1","version":"0'0","reported_seq":"349","reported_epoch":"391","state":"active+clean","last_fresh":"2021-03-24T21:59:21.211283+0000","last_change":"2021-03-24T21:57:59.462088+0000","last_active":"2021-03-24T21:59:21.211283+0000","last_peered":"2021-03-24T21:59:21.211283+0000","last_clean":"2021-03-24T21:59:21.211283+0000","last_became_active":"2021-03-24T21:51:49.408761+0000","last_became_peered":"2021-03-24T21:51:49.408761+0000","last_unstale":"2021-03-24T21:59:21.211283+0000","last_undegraded":"2021-03-24T21:59:21.211283+0000","last_fullsized":"2021-03-24T21:59:21.211283+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:59.461865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:25.220436+0000","last_clean_scrub_stamp":"2021-03-24T21:57:59.461865+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"69'22","reported_seq":"386","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.410124+0000","last_change":"2021-03-24T21:58:56.473659+0000","last_active":"2021-03-24T21:59:17.410124+0000","last_peered":"2021-03-24T21:59:17.410124+0000","last_clean":"2021-03-24T21:59:17.410124+0000","last_became_active":"2021-03-24T21:51:47.691459+0000","last_became_peered":"2021-03-24T21:51:47.691459+0000","last_unstale":"2021-03-24T21:59:17.410124+0000","last_undegraded":"2021-03-24T21:59:17.410124+0000","last_fullsized":"2021-03-24T21:59:17.410124+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'22","last_scrub_stamp":"2021-03-24T21:58:56.473510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:58:56.473510+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":46,"num_read_kb":26,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"357","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.438386+0000","last_change":"2021-03-24T21:59:12.288369+0000","last_active":"2021-03-24T21:59:17.438386+0000","last_peered":"2021-03-24T21:59:17.438386+0000","last_clean":"2021-03-24T21:59:17.438386+0000","last_became_active":"2021-03-24T21:59:12.287655+0000","last_became_peered":"2021-03-24T21:59:12.287655+0000","last_unstale":"2021-03-24T21:59:17.438386+0000","last_undegraded":"2021-03-24T21:59:17.438386+0000","last_fullsized":"2021-03-24T21:59:17.438386+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:54.381668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:54.381668+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"347","reported_epoch":"391","state":"active+clean","last_fresh":"2021-03-24T21:59:21.212330+0000","last_change":"2021-03-24T21:59:03.626326+0000","last_active":"2021-03-24T21:59:21.212330+0000","last_peered":"2021-03-24T21:59:21.212330+0000","last_clean":"2021-03-24T21:59:21.212330+0000","last_became_active":"2021-03-24T21:51:51.756114+0000","last_became_peered":"2021-03-24T21:51:51.756114+0000","last_unstale":"2021-03-24T21:59:21.212330+0000","last_undegraded":"2021-03-24T21:59:21.212330+0000","last_fullsized":"2021-03-24T21:59:21.212330+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:03.626250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:59:03.626250+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"387'350","reported_seq":"1319","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.481150+0000","last_change":"2021-03-24T21:59:17.481150+0000","last_active":"2021-03-24T21:59:17.481150+0000","last_peered":"2021-03-24T21:59:17.481150+0000","last_clean":"2021-03-24T21:59:17.481150+0000","last_became_active":"2021-03-24T21:57:26.169302+0000","last_became_peered":"2021-03-24T21:57:26.169302+0000","last_unstale":"2021-03-24T21:59:17.481150+0000","last_undegraded":"2021-03-24T21:59:17.481150+0000","last_fullsized":"2021-03-24T21:59:17.481150+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'330","last_scrub_stamp":"2021-03-24T21:59:07.373692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:59:07.373692+0000","log_size":350,"ondisk_log_size":350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13214698,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":6,"num_write_kb":1320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d6","length":"1"},{"start":"df","length":"1"},{"start":"e1","length":"1"},{"start":"e8","length":"1"},{"start":"ea","length":"4"},{"start":"ef","length":"1"},{"start":"f2","length":"4"},{"start":"fb","length":"3"},{"start":"109","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"}]},{"pgid":"5.0","version":"109'3","reported_seq":"449","reported_epoch":"388","state":"active+clean","last_fresh":"2021-03-24T21:59:18.415716+0000","last_change":"2021-03-24T21:57:25.463136+0000","last_active":"2021-03-24T21:59:18.415716+0000","last_peered":"2021-03-24T21:59:18.415716+0000","last_clean":"2021-03-24T21:59:18.415716+0000","last_became_active":"2021-03-24T21:54:38.135726+0000","last_became_peered":"2021-03-24T21:54:38.135726+0000","last_unstale":"2021-03-24T21:59:18.415716+0000","last_undegraded":"2021-03-24T21:59:18.415716+0000","last_fullsized":"2021-03-24T21:59:18.415716+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'3","last_scrub_stamp":"2021-03-24T21:57:25.462952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:57:25.462952+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"69'20","reported_seq":"427","reported_epoch":"391","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:59:21.211686+0000","last_change":"2021-03-24T21:58:31.207847+0000","last_active":"2021-03-24T21:59:21.211686+0000","last_peered":"2021-03-24T21:59:21.211686+0000","last_clean":"2021-03-24T21:58:29.215347+0000","last_became_active":"2021-03-24T21:58:31.207847+0000","last_became_peered":"2021-03-24T21:58:31.207847+0000","last_unstale":"2021-03-24T21:59:21.211686+0000","last_undegraded":"2021-03-24T21:58:31.193521+0000","last_fullsized":"2021-03-24T21:58:31.193193+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'20","last_scrub_stamp":"2021-03-24T21:57:21.706127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:57:21.706127+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":39,"num_read_kb":20,"num_write":20,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":940,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":13}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"373","reported_epoch":"387","state":"active+undersized","last_fresh":"2021-03-24T21:59:17.412959+0000","last_change":"2021-03-24T21:59:12.277943+0000","last_active":"2021-03-24T21:59:17.412959+0000","last_peered":"2021-03-24T21:59:17.412959+0000","last_clean":"2021-03-24T21:58:29.213618+0000","last_became_active":"2021-03-24T21:59:12.277943+0000","last_became_peered":"2021-03-24T21:59:12.277943+0000","last_unstale":"2021-03-24T21:59:17.412959+0000","last_undegraded":"2021-03-24T21:59:17.412959+0000","last_fullsized":"2021-03-24T21:59:12.271114+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:28.165408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:28.165408+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"365","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.415512+0000","last_change":"2021-03-24T21:58:54.826704+0000","last_active":"2021-03-24T21:59:17.415512+0000","last_peered":"2021-03-24T21:59:17.415512+0000","last_clean":"2021-03-24T21:59:17.415512+0000","last_became_active":"2021-03-24T21:54:38.110246+0000","last_became_peered":"2021-03-24T21:54:38.110246+0000","last_unstale":"2021-03-24T21:59:17.415512+0000","last_undegraded":"2021-03-24T21:59:17.415512+0000","last_fullsized":"2021-03-24T21:59:17.415512+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:54.826534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:58:54.826534+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"272'64","reported_seq":"973","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.454080+0000","last_change":"2021-03-24T21:59:17.454080+0000","last_active":"2021-03-24T21:59:17.454080+0000","last_peered":"2021-03-24T21:59:17.454080+0000","last_clean":"2021-03-24T21:59:17.454080+0000","last_became_active":"2021-03-24T21:57:26.174158+0000","last_became_peered":"2021-03-24T21:57:26.174158+0000","last_unstale":"2021-03-24T21:59:17.454080+0000","last_undegraded":"2021-03-24T21:59:17.454080+0000","last_fullsized":"2021-03-24T21:59:17.454080+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'64","last_scrub_stamp":"2021-03-24T21:59:08.406188+0000","last_deep_scrub":"272'64","last_deep_scrub_stamp":"2021-03-24T21:59:08.406188+0000","last_clean_scrub_stamp":"2021-03-24T21:59:08.406188+0000","log_size":64,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d6","length":"1"},{"start":"df","length":"1"},{"start":"e1","length":"1"},{"start":"e8","length":"1"},{"start":"ea","length":"4"},{"start":"ef","length":"1"},{"start":"f2","length":"4"},{"start":"fb","length":"3"},{"start":"109","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"}]},{"pgid":"5.3","version":"343'5","reported_seq":"450","reported_epoch":"387","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:59:17.409451+0000","last_change":"2021-03-24T21:58:31.223346+0000","last_active":"2021-03-24T21:59:17.409451+0000","last_peered":"2021-03-24T21:59:17.409451+0000","last_clean":"2021-03-24T21:58:29.216175+0000","last_became_active":"2021-03-24T21:58:31.223346+0000","last_became_peered":"2021-03-24T21:58:31.223346+0000","last_unstale":"2021-03-24T21:59:17.409451+0000","last_undegraded":"2021-03-24T21:58:31.193746+0000","last_fullsized":"2021-03-24T21:58:31.193622+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'4","last_scrub_stamp":"2021-03-24T21:56:01.266936+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-24T21:53:26.584678+0000","last_clean_scrub_stamp":"2021-03-24T21:56:01.266936+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"69'16","reported_seq":"100","reported_epoch":"391","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:59:21.213672+0000","last_change":"2021-03-24T21:58:31.209014+0000","last_active":"2021-03-24T21:59:21.213672+0000","last_peered":"2021-03-24T21:59:21.213672+0000","last_clean":"2021-03-24T21:51:54.404218+0000","last_became_active":"2021-03-24T21:58:31.209014+0000","last_became_peered":"2021-03-24T21:58:31.209014+0000","last_unstale":"2021-03-24T21:59:21.213672+0000","last_undegraded":"2021-03-24T21:58:31.194887+0000","last_fullsized":"2021-03-24T21:58:31.194702+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T21:56:41.235143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:41.235143+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":32,"num_read_kb":15,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":11}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"62","reported_epoch":"387","state":"active+undersized","last_fresh":"2021-03-24T21:59:17.423304+0000","last_change":"2021-03-24T21:59:12.277198+0000","last_active":"2021-03-24T21:59:17.423304+0000","last_peered":"2021-03-24T21:59:17.423304+0000","last_clean":"2021-03-24T21:51:44.337765+0000","last_became_active":"2021-03-24T21:59:12.277198+0000","last_became_peered":"2021-03-24T21:59:12.277198+0000","last_unstale":"2021-03-24T21:59:17.423304+0000","last_undegraded":"2021-03-24T21:59:17.423304+0000","last_fullsized":"2021-03-24T21:59:12.271242+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:53.013129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:53.013129+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"342","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.407326+0000","last_change":"2021-03-24T21:59:04.077747+0000","last_active":"2021-03-24T21:59:17.407326+0000","last_peered":"2021-03-24T21:59:17.407326+0000","last_clean":"2021-03-24T21:59:17.407326+0000","last_became_active":"2021-03-24T21:51:51.759627+0000","last_became_peered":"2021-03-24T21:51:51.759627+0000","last_unstale":"2021-03-24T21:59:17.407326+0000","last_undegraded":"2021-03-24T21:59:17.407326+0000","last_fullsized":"2021-03-24T21:59:17.407326+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:04.077569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:59:04.077569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"392'305","reported_seq":"1345","reported_epoch":"392","state":"active+clean","last_fresh":"2021-03-24T21:59:22.315740+0000","last_change":"2021-03-24T21:59:22.315951+0000","last_active":"2021-03-24T21:59:22.315740+0000","last_peered":"2021-03-24T21:59:22.315740+0000","last_clean":"2021-03-24T21:59:22.315740+0000","last_became_active":"2021-03-24T21:57:26.191103+0000","last_became_peered":"2021-03-24T21:57:26.191103+0000","last_unstale":"2021-03-24T21:59:22.315740+0000","last_undegraded":"2021-03-24T21:59:22.315740+0000","last_fullsized":"2021-03-24T21:59:22.315740+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"365'287","last_scrub_stamp":"2021-03-24T21:58:56.648040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:58:56.648040+0000","log_size":305,"ondisk_log_size":305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7787128,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":16157,"num_write":4,"num_write_kb":1073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d6","length":"1"},{"start":"df","length":"1"},{"start":"e1","length":"1"},{"start":"e8","length":"1"},{"start":"ea","length":"4"},{"start":"ef","length":"1"},{"start":"f1","length":"5"},{"start":"fb","length":"3"},{"start":"100","length":"1"},{"start":"109","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"}]},{"pgid":"5.2","version":"0'0","reported_seq":"308","reported_epoch":"387","state":"active+undersized","last_fresh":"2021-03-24T21:59:17.409063+0000","last_change":"2021-03-24T21:58:31.199463+0000","last_active":"2021-03-24T21:59:17.409063+0000","last_peered":"2021-03-24T21:59:17.409063+0000","last_clean":"2021-03-24T21:58:29.212689+0000","last_became_active":"2021-03-24T21:58:31.199463+0000","last_became_peered":"2021-03-24T21:58:31.199463+0000","last_unstale":"2021-03-24T21:59:17.409063+0000","last_undegraded":"2021-03-24T21:59:17.409063+0000","last_fullsized":"2021-03-24T21:58:31.192367+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:58.253777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:55:58.253777+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"69'21","reported_seq":"392","reported_epoch":"387","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:59:17.409332+0000","last_change":"2021-03-24T21:58:31.209599+0000","last_active":"2021-03-24T21:59:17.409332+0000","last_peered":"2021-03-24T21:59:17.409332+0000","last_clean":"2021-03-24T21:58:29.216071+0000","last_became_active":"2021-03-24T21:58:31.209599+0000","last_became_peered":"2021-03-24T21:58:31.209599+0000","last_unstale":"2021-03-24T21:59:17.409332+0000","last_undegraded":"2021-03-24T21:58:31.193410+0000","last_fullsized":"2021-03-24T21:58:31.193294+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'21","last_scrub_stamp":"2021-03-24T21:56:39.335613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:39.335613+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":45,"num_read_kb":27,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":10}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"68","reported_epoch":"387","state":"active+undersized","last_fresh":"2021-03-24T21:59:17.410784+0000","last_change":"2021-03-24T21:59:12.275932+0000","last_active":"2021-03-24T21:59:17.410784+0000","last_peered":"2021-03-24T21:59:17.410784+0000","last_clean":"2021-03-24T21:51:44.337765+0000","last_became_active":"2021-03-24T21:59:12.275932+0000","last_became_peered":"2021-03-24T21:59:12.275932+0000","last_unstale":"2021-03-24T21:59:17.410784+0000","last_undegraded":"2021-03-24T21:59:17.410784+0000","last_fullsized":"2021-03-24T21:59:12.267098+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:38.144956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:38.144956+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"209","reported_epoch":"387","state":"active+undersized","last_fresh":"2021-03-24T21:59:17.414664+0000","last_change":"2021-03-24T21:58:31.224919+0000","last_active":"2021-03-24T21:59:17.414664+0000","last_peered":"2021-03-24T21:59:17.414664+0000","last_clean":"2021-03-24T21:58:29.200711+0000","last_became_active":"2021-03-24T21:58:31.224919+0000","last_became_peered":"2021-03-24T21:58:31.224919+0000","last_unstale":"2021-03-24T21:59:17.414664+0000","last_undegraded":"2021-03-24T21:59:17.414664+0000","last_fullsized":"2021-03-24T21:58:31.196827+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:24.967391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:58:24.967391+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.0","version":"175'22","reported_seq":"315","reported_epoch":"387","state":"active+undersized","last_fresh":"2021-03-24T21:59:17.409073+0000","last_change":"2021-03-24T21:58:31.197067+0000","last_active":"2021-03-24T21:59:17.409073+0000","last_peered":"2021-03-24T21:59:17.409073+0000","last_clean":"2021-03-24T21:55:44.382770+0000","last_became_active":"2021-03-24T21:58:31.197067+0000","last_became_peered":"2021-03-24T21:58:31.197067+0000","last_unstale":"2021-03-24T21:59:17.409073+0000","last_undegraded":"2021-03-24T21:59:17.409073+0000","last_fullsized":"2021-03-24T21:58:31.184568+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'22","last_scrub_stamp":"2021-03-24T21:58:22.940875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:58:22.940875+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":19,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3283,"num_write":3,"num_write_kb":845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":313386,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"114","reported_epoch":"387","state":"active+undersized","last_fresh":"2021-03-24T21:59:17.418083+0000","last_change":"2021-03-24T21:58:31.218358+0000","last_active":"2021-03-24T21:59:17.418083+0000","last_peered":"2021-03-24T21:59:17.418083+0000","last_clean":"2021-03-24T21:54:36.088853+0000","last_became_active":"2021-03-24T21:58:31.218358+0000","last_became_peered":"2021-03-24T21:58:31.218358+0000","last_unstale":"2021-03-24T21:59:17.418083+0000","last_undegraded":"2021-03-24T21:59:17.418083+0000","last_fullsized":"2021-03-24T21:58:31.191469+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:43.194160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:43.194160+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"69'16","reported_seq":"440","reported_epoch":"387","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:59:17.411488+0000","last_change":"2021-03-24T21:58:31.208049+0000","last_active":"2021-03-24T21:59:17.411488+0000","last_peered":"2021-03-24T21:59:17.411488+0000","last_clean":"2021-03-24T21:58:29.213527+0000","last_became_active":"2021-03-24T21:58:31.208049+0000","last_became_peered":"2021-03-24T21:58:31.208049+0000","last_unstale":"2021-03-24T21:59:17.411488+0000","last_undegraded":"2021-03-24T21:58:31.189256+0000","last_fullsized":"2021-03-24T21:58:31.189131+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T21:57:26.257686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:57:26.257686+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":16,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1260,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":10}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"66'5","reported_seq":"825","reported_epoch":"388","state":"active+clean","last_fresh":"2021-03-24T21:59:18.434974+0000","last_change":"2021-03-24T21:58:58.818153+0000","last_active":"2021-03-24T21:59:18.434974+0000","last_peered":"2021-03-24T21:59:18.434974+0000","last_clean":"2021-03-24T21:59:18.434974+0000","last_became_active":"2021-03-24T21:54:38.125434+0000","last_became_peered":"2021-03-24T21:54:38.125434+0000","last_unstale":"2021-03-24T21:59:18.434974+0000","last_undegraded":"2021-03-24T21:59:18.434974+0000","last_fullsized":"2021-03-24T21:59:18.434974+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2021-03-24T21:58:58.818001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:39.286101+0000","last_clean_scrub_stamp":"2021-03-24T21:58:58.818001+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":451,"num_read_kb":446,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"357","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.407163+0000","last_change":"2021-03-24T21:59:12.281839+0000","last_active":"2021-03-24T21:59:17.407163+0000","last_peered":"2021-03-24T21:59:17.407163+0000","last_clean":"2021-03-24T21:59:17.407163+0000","last_became_active":"2021-03-24T21:59:12.281282+0000","last_became_peered":"2021-03-24T21:59:12.281282+0000","last_unstale":"2021-03-24T21:59:17.407163+0000","last_undegraded":"2021-03-24T21:59:17.407163+0000","last_fullsized":"2021-03-24T21:59:17.407163+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:38.304792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:38.304792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"146","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.409561+0000","last_change":"2021-03-24T21:58:58.034141+0000","last_active":"2021-03-24T21:59:17.409561+0000","last_peered":"2021-03-24T21:59:17.409561+0000","last_clean":"2021-03-24T21:59:17.409561+0000","last_became_active":"2021-03-24T21:57:06.152772+0000","last_became_peered":"2021-03-24T21:57:06.152772+0000","last_unstale":"2021-03-24T21:59:17.409561+0000","last_undegraded":"2021-03-24T21:59:17.409561+0000","last_fullsized":"2021-03-24T21:59:17.409561+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:58.033876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:58:58.033876+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.5","version":"195'69","reported_seq":"964","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.419656+0000","last_change":"2021-03-24T21:59:17.419656+0000","last_active":"2021-03-24T21:59:17.419656+0000","last_peered":"2021-03-24T21:59:17.419656+0000","last_clean":"2021-03-24T21:59:17.419656+0000","last_became_active":"2021-03-24T21:56:08.152168+0000","last_became_peered":"2021-03-24T21:56:08.152168+0000","last_unstale":"2021-03-24T21:59:17.419656+0000","last_undegraded":"2021-03-24T21:59:17.419656+0000","last_fullsized":"2021-03-24T21:59:17.419656+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'69","last_scrub_stamp":"2021-03-24T21:59:09.930390+0000","last_deep_scrub":"195'69","last_deep_scrub_stamp":"2021-03-24T21:56:49.886058+0000","last_clean_scrub_stamp":"2021-03-24T21:59:09.930390+0000","log_size":69,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d6","length":"1"},{"start":"df","length":"1"},{"start":"e1","length":"1"},{"start":"e8","length":"1"},{"start":"ea","length":"4"},{"start":"ef","length":"1"},{"start":"f2","length":"4"},{"start":"fb","length":"3"},{"start":"109","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"}]},{"pgid":"5.4","version":"0'0","reported_seq":"350","reported_epoch":"387","state":"active+undersized","last_fresh":"2021-03-24T21:59:17.416313+0000","last_change":"2021-03-24T21:58:31.209266+0000","last_active":"2021-03-24T21:59:17.416313+0000","last_peered":"2021-03-24T21:59:17.416313+0000","last_clean":"2021-03-24T21:58:29.202947+0000","last_became_active":"2021-03-24T21:58:31.209266+0000","last_became_peered":"2021-03-24T21:58:31.209266+0000","last_unstale":"2021-03-24T21:59:17.416313+0000","last_undegraded":"2021-03-24T21:59:17.416313+0000","last_fullsized":"2021-03-24T21:58:31.199943+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:18.199919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:57:18.199919+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"69'15","reported_seq":"368","reported_epoch":"391","state":"active+clean","last_fresh":"2021-03-24T21:59:21.214618+0000","last_change":"2021-03-24T21:58:20.702582+0000","last_active":"2021-03-24T21:59:21.214618+0000","last_peered":"2021-03-24T21:59:21.214618+0000","last_clean":"2021-03-24T21:59:21.214618+0000","last_became_active":"2021-03-24T21:54:24.194711+0000","last_became_peered":"2021-03-24T21:54:24.194711+0000","last_unstale":"2021-03-24T21:59:21.214618+0000","last_undegraded":"2021-03-24T21:59:21.214618+0000","last_fullsized":"2021-03-24T21:59:21.214618+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:58:20.702393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:58:20.702393+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":26,"num_read_kb":14,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":466,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"364","reported_epoch":"387","state":"active+undersized","last_fresh":"2021-03-24T21:59:17.421820+0000","last_change":"2021-03-24T21:59:12.276341+0000","last_active":"2021-03-24T21:59:17.421820+0000","last_peered":"2021-03-24T21:59:17.421820+0000","last_clean":"2021-03-24T21:58:29.207031+0000","last_became_active":"2021-03-24T21:59:12.276341+0000","last_became_peered":"2021-03-24T21:59:12.276341+0000","last_unstale":"2021-03-24T21:59:17.421820+0000","last_undegraded":"2021-03-24T21:59:17.421820+0000","last_fullsized":"2021-03-24T21:59:12.272786+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:33.533840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:33.533840+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"376","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.412758+0000","last_change":"2021-03-24T21:58:21.039376+0000","last_active":"2021-03-24T21:59:17.412758+0000","last_peered":"2021-03-24T21:59:17.412758+0000","last_clean":"2021-03-24T21:59:17.412758+0000","last_became_active":"2021-03-24T21:54:38.110327+0000","last_became_peered":"2021-03-24T21:54:38.110327+0000","last_unstale":"2021-03-24T21:59:17.412758+0000","last_undegraded":"2021-03-24T21:59:17.412758+0000","last_fullsized":"2021-03-24T21:59:17.412758+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:21.039180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:49:51.529051+0000","last_clean_scrub_stamp":"2021-03-24T21:58:21.039180+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"56","reported_epoch":"387","state":"active+undersized","last_fresh":"2021-03-24T21:59:17.438775+0000","last_change":"2021-03-24T21:58:31.229327+0000","last_active":"2021-03-24T21:59:17.438775+0000","last_peered":"2021-03-24T21:59:17.438775+0000","last_clean":"2021-03-24T21:51:50.731137+0000","last_became_active":"2021-03-24T21:58:31.229327+0000","last_became_peered":"2021-03-24T21:58:31.229327+0000","last_unstale":"2021-03-24T21:59:17.438775+0000","last_undegraded":"2021-03-24T21:59:17.438775+0000","last_fullsized":"2021-03-24T21:58:31.194892+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:55.072694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:57:55.072694+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.6","version":"382'183","reported_seq":"1091","reported_epoch":"387","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:59:17.413474+0000","last_change":"2021-03-24T21:58:31.222554+0000","last_active":"2021-03-24T21:59:17.413474+0000","last_peered":"2021-03-24T21:59:17.413474+0000","last_clean":"2021-03-24T21:58:29.213936+0000","last_became_active":"2021-03-24T21:58:31.222554+0000","last_became_peered":"2021-03-24T21:58:31.222554+0000","last_unstale":"2021-03-24T21:59:17.413474+0000","last_undegraded":"2021-03-24T21:58:31.192024+0000","last_fullsized":"2021-03-24T21:58:31.191882+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":183,"ondisk_log_size":183,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":19,"stat_sum":{"num_bytes":14557871,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":91,"num_read_kb":116480,"num_write":55,"num_write_kb":27263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":8481953,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":5}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"58","reported_epoch":"387","state":"active+undersized","last_fresh":"2021-03-24T21:59:17.418193+0000","last_change":"2021-03-24T21:58:31.204715+0000","last_active":"2021-03-24T21:59:17.418193+0000","last_peered":"2021-03-24T21:59:17.418193+0000","last_clean":"2021-03-24T21:51:48.384232+0000","last_became_active":"2021-03-24T21:58:31.204715+0000","last_became_peered":"2021-03-24T21:58:31.204715+0000","last_unstale":"2021-03-24T21:59:17.418193+0000","last_undegraded":"2021-03-24T21:59:17.418193+0000","last_fullsized":"2021-03-24T21:58:31.199993+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:20.215065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:57:20.215065+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"89'22","reported_seq":"123","reported_epoch":"387","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:59:17.417983+0000","last_change":"2021-03-24T21:58:31.207163+0000","last_active":"2021-03-24T21:59:17.417983+0000","last_peered":"2021-03-24T21:59:17.417983+0000","last_clean":"2021-03-24T21:53:19.701158+0000","last_became_active":"2021-03-24T21:58:31.207163+0000","last_became_peered":"2021-03-24T21:58:31.207163+0000","last_unstale":"2021-03-24T21:59:17.417983+0000","last_undegraded":"2021-03-24T21:58:31.191478+0000","last_fullsized":"2021-03-24T21:58:31.191363+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'22","last_scrub_stamp":"2021-03-24T21:56:54.046194+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2021-03-24T21:53:19.717244+0000","last_clean_scrub_stamp":"2021-03-24T21:56:54.046194+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":36,"num_read_kb":21,"num_write":18,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":7}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"364","reported_epoch":"390","state":"active+clean","last_fresh":"2021-03-24T21:59:20.216738+0000","last_change":"2021-03-24T21:59:01.806818+0000","last_active":"2021-03-24T21:59:20.216738+0000","last_peered":"2021-03-24T21:59:20.216738+0000","last_clean":"2021-03-24T21:59:20.216738+0000","last_became_active":"2021-03-24T21:51:40.351000+0000","last_became_peered":"2021-03-24T21:51:40.351000+0000","last_unstale":"2021-03-24T21:59:20.216738+0000","last_undegraded":"2021-03-24T21:59:20.216738+0000","last_fullsized":"2021-03-24T21:59:20.216738+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:01.806758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:39.286101+0000","last_clean_scrub_stamp":"2021-03-24T21:59:01.806758+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"61'1","reported_seq":"324","reported_epoch":"390","state":"active+undersized","last_fresh":"2021-03-24T21:59:20.210780+0000","last_change":"2021-03-24T21:59:12.274512+0000","last_active":"2021-03-24T21:59:20.210780+0000","last_peered":"2021-03-24T21:59:20.210780+0000","last_clean":"2021-03-24T21:58:29.216757+0000","last_became_active":"2021-03-24T21:59:12.274512+0000","last_became_peered":"2021-03-24T21:59:12.274512+0000","last_unstale":"2021-03-24T21:59:20.210780+0000","last_undegraded":"2021-03-24T21:59:20.210780+0000","last_fullsized":"2021-03-24T21:59:12.266630+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2021-03-24T21:57:12.046206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:12.046206+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":436,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"351","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.419157+0000","last_change":"2021-03-24T21:57:56.065379+0000","last_active":"2021-03-24T21:59:17.419157+0000","last_peered":"2021-03-24T21:59:17.419157+0000","last_clean":"2021-03-24T21:59:17.419157+0000","last_became_active":"2021-03-24T21:54:24.186957+0000","last_became_peered":"2021-03-24T21:54:24.186957+0000","last_unstale":"2021-03-24T21:59:17.419157+0000","last_undegraded":"2021-03-24T21:59:17.419157+0000","last_fullsized":"2021-03-24T21:59:17.419157+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:56.065113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:57:56.065113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"382'217","reported_seq":"1070","reported_epoch":"387","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:59:17.408678+0000","last_change":"2021-03-24T21:58:31.208413+0000","last_active":"2021-03-24T21:59:17.408678+0000","last_peered":"2021-03-24T21:59:17.408678+0000","last_clean":"2021-03-24T21:58:29.212922+0000","last_became_active":"2021-03-24T21:58:31.208413+0000","last_became_peered":"2021-03-24T21:58:31.208413+0000","last_unstale":"2021-03-24T21:59:17.408678+0000","last_undegraded":"2021-03-24T21:58:31.190092+0000","last_fullsized":"2021-03-24T21:58:31.189951+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'177","last_scrub_stamp":"2021-03-24T21:57:21.260725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:57:21.260725+0000","log_size":217,"ondisk_log_size":217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":19,"stat_sum":{"num_bytes":7652252,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":40,"num_read_kb":51357,"num_write":21,"num_write_kb":9153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":5}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"353","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.411199+0000","last_change":"2021-03-24T21:58:23.866009+0000","last_active":"2021-03-24T21:59:17.411199+0000","last_peered":"2021-03-24T21:59:17.411199+0000","last_clean":"2021-03-24T21:59:17.411199+0000","last_became_active":"2021-03-24T21:54:38.136859+0000","last_became_peered":"2021-03-24T21:54:38.136859+0000","last_unstale":"2021-03-24T21:59:17.411199+0000","last_undegraded":"2021-03-24T21:59:17.411199+0000","last_fullsized":"2021-03-24T21:59:17.411199+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:23.865909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:58:23.865909+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"69'25","reported_seq":"393","reported_epoch":"390","state":"active+clean","last_fresh":"2021-03-24T21:59:20.209686+0000","last_change":"2021-03-24T21:58:54.693769+0000","last_active":"2021-03-24T21:59:20.209686+0000","last_peered":"2021-03-24T21:59:20.209686+0000","last_clean":"2021-03-24T21:59:20.209686+0000","last_became_active":"2021-03-24T21:54:38.141478+0000","last_became_peered":"2021-03-24T21:54:38.141478+0000","last_unstale":"2021-03-24T21:59:20.209686+0000","last_undegraded":"2021-03-24T21:59:20.209686+0000","last_fullsized":"2021-03-24T21:59:20.209686+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'25","last_scrub_stamp":"2021-03-24T21:58:54.693601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:58:54.693601+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":53,"num_read_kb":30,"num_write":28,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"89'2","reported_seq":"395","reported_epoch":"391","state":"active+undersized","last_fresh":"2021-03-24T21:59:21.212831+0000","last_change":"2021-03-24T21:59:12.279790+0000","last_active":"2021-03-24T21:59:21.212831+0000","last_peered":"2021-03-24T21:59:21.212831+0000","last_clean":"2021-03-24T21:58:29.214732+0000","last_became_active":"2021-03-24T21:59:12.279790+0000","last_became_peered":"2021-03-24T21:59:12.279790+0000","last_unstale":"2021-03-24T21:59:21.212831+0000","last_undegraded":"2021-03-24T21:59:21.212831+0000","last_fullsized":"2021-03-24T21:59:12.232229+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2021-03-24T21:57:58.491830+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2021-03-24T21:57:58.491830+0000","last_clean_scrub_stamp":"2021-03-24T21:57:58.491830+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":179,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"344","reported_epoch":"387","state":"active+undersized","last_fresh":"2021-03-24T21:59:17.409214+0000","last_change":"2021-03-24T21:58:31.223020+0000","last_active":"2021-03-24T21:59:17.409214+0000","last_peered":"2021-03-24T21:59:17.409214+0000","last_clean":"2021-03-24T21:58:29.212707+0000","last_became_active":"2021-03-24T21:58:31.223020+0000","last_became_peered":"2021-03-24T21:58:31.223020+0000","last_unstale":"2021-03-24T21:59:17.409214+0000","last_undegraded":"2021-03-24T21:59:17.409214+0000","last_fullsized":"2021-03-24T21:58:31.188845+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:01.817480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:06.800417+0000","last_clean_scrub_stamp":"2021-03-24T21:58:01.817480+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"342","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.439513+0000","last_change":"2021-03-24T21:57:22.422901+0000","last_active":"2021-03-24T21:59:17.439513+0000","last_peered":"2021-03-24T21:59:17.439513+0000","last_clean":"2021-03-24T21:59:17.439513+0000","last_became_active":"2021-03-24T21:51:51.759514+0000","last_became_peered":"2021-03-24T21:51:51.759514+0000","last_unstale":"2021-03-24T21:59:17.439513+0000","last_undegraded":"2021-03-24T21:59:17.439513+0000","last_fullsized":"2021-03-24T21:59:17.439513+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:22.422787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:52:59.706511+0000","last_clean_scrub_stamp":"2021-03-24T21:57:22.422787+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"387'270","reported_seq":"1202","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.471094+0000","last_change":"2021-03-24T21:59:17.471094+0000","last_active":"2021-03-24T21:59:17.471094+0000","last_peered":"2021-03-24T21:59:17.471094+0000","last_clean":"2021-03-24T21:59:17.471094+0000","last_became_active":"2021-03-24T21:56:08.157153+0000","last_became_peered":"2021-03-24T21:56:08.157153+0000","last_unstale":"2021-03-24T21:59:17.471094+0000","last_undegraded":"2021-03-24T21:59:17.471094+0000","last_fullsized":"2021-03-24T21:59:17.471094+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'218","last_scrub_stamp":"2021-03-24T21:57:22.239425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:57:22.239425+0000","log_size":270,"ondisk_log_size":270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14752859,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":17,"num_read_kb":7093,"num_write":19,"num_write_kb":13312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"d6","length":"1"},{"start":"df","length":"1"},{"start":"e1","length":"1"},{"start":"e8","length":"1"},{"start":"ea","length":"4"},{"start":"ef","length":"1"},{"start":"f2","length":"4"},{"start":"fb","length":"3"},{"start":"109","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"}]},{"pgid":"4.b","version":"69'14","reported_seq":"401","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.409214+0000","last_change":"2021-03-24T21:59:05.058605+0000","last_active":"2021-03-24T21:59:17.409214+0000","last_peered":"2021-03-24T21:59:17.409214+0000","last_clean":"2021-03-24T21:59:17.409214+0000","last_became_active":"2021-03-24T21:54:38.138034+0000","last_became_peered":"2021-03-24T21:54:38.138034+0000","last_unstale":"2021-03-24T21:59:17.409214+0000","last_undegraded":"2021-03-24T21:59:17.409214+0000","last_fullsized":"2021-03-24T21:59:17.409214+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'14","last_scrub_stamp":"2021-03-24T21:59:05.058513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:59:05.058513+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":16,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":894,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"344","reported_epoch":"387","state":"active+undersized","last_fresh":"2021-03-24T21:59:17.439760+0000","last_change":"2021-03-24T21:58:31.229412+0000","last_active":"2021-03-24T21:59:17.439760+0000","last_peered":"2021-03-24T21:59:17.439760+0000","last_clean":"2021-03-24T21:58:29.207714+0000","last_became_active":"2021-03-24T21:58:31.229412+0000","last_became_peered":"2021-03-24T21:58:31.229412+0000","last_unstale":"2021-03-24T21:59:17.439760+0000","last_undegraded":"2021-03-24T21:59:17.439760+0000","last_fullsized":"2021-03-24T21:58:31.188427+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:12.514094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:12.514094+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"351","reported_epoch":"390","state":"active+clean","last_fresh":"2021-03-24T21:59:20.211362+0000","last_change":"2021-03-24T21:58:26.464636+0000","last_active":"2021-03-24T21:59:20.211362+0000","last_peered":"2021-03-24T21:59:20.211362+0000","last_clean":"2021-03-24T21:59:20.211362+0000","last_became_active":"2021-03-24T21:51:49.411921+0000","last_became_peered":"2021-03-24T21:51:49.411921+0000","last_unstale":"2021-03-24T21:59:20.211362+0000","last_undegraded":"2021-03-24T21:59:20.211362+0000","last_fullsized":"2021-03-24T21:59:20.211362+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:26.464456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:58:26.464456+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"351","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.408257+0000","last_change":"2021-03-24T21:59:00.033527+0000","last_active":"2021-03-24T21:59:17.408257+0000","last_peered":"2021-03-24T21:59:17.408257+0000","last_clean":"2021-03-24T21:59:17.408257+0000","last_became_active":"2021-03-24T21:57:02.188645+0000","last_became_peered":"2021-03-24T21:57:02.188645+0000","last_unstale":"2021-03-24T21:59:17.408257+0000","last_undegraded":"2021-03-24T21:59:17.408257+0000","last_fullsized":"2021-03-24T21:59:17.408257+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:00.033461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:35.636299+0000","last_clean_scrub_stamp":"2021-03-24T21:59:00.033461+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"391'162","reported_seq":"1053","reported_epoch":"391","state":"active+clean","last_fresh":"2021-03-24T21:59:21.221984+0000","last_change":"2021-03-24T21:59:21.222345+0000","last_active":"2021-03-24T21:59:21.221984+0000","last_peered":"2021-03-24T21:59:21.221984+0000","last_clean":"2021-03-24T21:59:21.221984+0000","last_became_active":"2021-03-24T21:56:08.148045+0000","last_became_peered":"2021-03-24T21:56:08.148045+0000","last_unstale":"2021-03-24T21:59:21.221984+0000","last_undegraded":"2021-03-24T21:59:21.221984+0000","last_fullsized":"2021-03-24T21:59:21.221984+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"329'132","last_scrub_stamp":"2021-03-24T21:58:20.372929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:58:20.372929+0000","log_size":162,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3428694,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":14929,"num_write":24,"num_write_kb":5860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"d6","length":"1"},{"start":"df","length":"1"},{"start":"e1","length":"1"},{"start":"e8","length":"1"},{"start":"ea","length":"4"},{"start":"ef","length":"1"},{"start":"f1","length":"5"},{"start":"fb","length":"3"},{"start":"109","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"}]},{"pgid":"4.a","version":"69'23","reported_seq":"111","reported_epoch":"387","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:59:17.408838+0000","last_change":"2021-03-24T21:58:31.206644+0000","last_active":"2021-03-24T21:59:17.408838+0000","last_peered":"2021-03-24T21:59:17.408838+0000","last_clean":"2021-03-24T21:51:54.474291+0000","last_became_active":"2021-03-24T21:58:31.206644+0000","last_became_peered":"2021-03-24T21:58:31.206644+0000","last_unstale":"2021-03-24T21:59:17.408838+0000","last_undegraded":"2021-03-24T21:58:31.191416+0000","last_fullsized":"2021-03-24T21:58:31.191299+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'23","last_scrub_stamp":"2021-03-24T21:56:34.991954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:34.991954+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":43,"num_read_kb":24,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":13}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":"112","reported_epoch":"387","state":"active+undersized","last_fresh":"2021-03-24T21:59:17.423267+0000","last_change":"2021-03-24T21:58:31.206125+0000","last_active":"2021-03-24T21:59:17.423267+0000","last_peered":"2021-03-24T21:59:17.423267+0000","last_clean":"2021-03-24T21:54:22.149668+0000","last_became_active":"2021-03-24T21:58:31.206125+0000","last_became_peered":"2021-03-24T21:58:31.206125+0000","last_unstale":"2021-03-24T21:59:17.423267+0000","last_undegraded":"2021-03-24T21:59:17.423267+0000","last_fullsized":"2021-03-24T21:58:31.195695+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:58.133341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:58.133341+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"153","reported_epoch":"387","state":"active+undersized","last_fresh":"2021-03-24T21:59:17.409036+0000","last_change":"2021-03-24T21:58:31.209196+0000","last_active":"2021-03-24T21:59:17.409036+0000","last_peered":"2021-03-24T21:59:17.409036+0000","last_clean":"2021-03-24T21:58:29.215889+0000","last_became_active":"2021-03-24T21:58:31.209196+0000","last_became_peered":"2021-03-24T21:58:31.209196+0000","last_unstale":"2021-03-24T21:59:17.409036+0000","last_undegraded":"2021-03-24T21:59:17.409036+0000","last_fullsized":"2021-03-24T21:58:31.192460+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:44.018043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:44.018043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"99","reported_epoch":"390","state":"active+clean","last_fresh":"2021-03-24T21:59:20.211073+0000","last_change":"2021-03-24T21:57:58.225287+0000","last_active":"2021-03-24T21:59:20.211073+0000","last_peered":"2021-03-24T21:59:20.211073+0000","last_clean":"2021-03-24T21:59:20.211073+0000","last_became_active":"2021-03-24T21:57:57.296747+0000","last_became_peered":"2021-03-24T21:57:57.296747+0000","last_unstale":"2021-03-24T21:59:20.211073+0000","last_undegraded":"2021-03-24T21:59:20.211073+0000","last_fullsized":"2021-03-24T21:59:20.211073+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:58.225162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:57:58.225162+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"387'242","reported_seq":"1165","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.424316+0000","last_change":"2021-03-24T21:59:17.424316+0000","last_active":"2021-03-24T21:59:17.424316+0000","last_peered":"2021-03-24T21:59:17.424316+0000","last_clean":"2021-03-24T21:59:17.424316+0000","last_became_active":"2021-03-24T21:56:08.158313+0000","last_became_peered":"2021-03-24T21:56:08.158313+0000","last_unstale":"2021-03-24T21:59:17.424316+0000","last_undegraded":"2021-03-24T21:59:17.424316+0000","last_fullsized":"2021-03-24T21:59:17.424316+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'211","last_scrub_stamp":"2021-03-24T21:57:25.266209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:57:25.266209+0000","log_size":242,"ondisk_log_size":242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3176589,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":14,"num_read_kb":15224,"num_write":10,"num_write_kb":5110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"d6","length":"1"},{"start":"df","length":"1"},{"start":"e1","length":"1"},{"start":"e8","length":"1"},{"start":"ea","length":"4"},{"start":"ef","length":"1"},{"start":"f2","length":"4"},{"start":"fb","length":"3"},{"start":"109","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"}]},{"pgid":"5.8","version":"0'0","reported_seq":"366","reported_epoch":"390","state":"active+undersized","last_fresh":"2021-03-24T21:59:20.210520+0000","last_change":"2021-03-24T21:58:31.198862+0000","last_active":"2021-03-24T21:59:20.210520+0000","last_peered":"2021-03-24T21:59:20.210520+0000","last_clean":"2021-03-24T21:58:29.216464+0000","last_became_active":"2021-03-24T21:58:31.198862+0000","last_became_peered":"2021-03-24T21:58:31.198862+0000","last_unstale":"2021-03-24T21:59:20.210520+0000","last_undegraded":"2021-03-24T21:59:20.210520+0000","last_fullsized":"2021-03-24T21:58:31.190814+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:28.480625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:58:28.480625+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"69'15","reported_seq":"407","reported_epoch":"391","state":"active+clean","last_fresh":"2021-03-24T21:59:21.212359+0000","last_change":"2021-03-24T21:58:30.678342+0000","last_active":"2021-03-24T21:59:21.212359+0000","last_peered":"2021-03-24T21:59:21.212359+0000","last_clean":"2021-03-24T21:59:21.212359+0000","last_became_active":"2021-03-24T21:54:38.139863+0000","last_became_peered":"2021-03-24T21:54:38.139863+0000","last_unstale":"2021-03-24T21:59:21.212359+0000","last_undegraded":"2021-03-24T21:59:21.212359+0000","last_fullsized":"2021-03-24T21:59:21.212359+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:58:30.678185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:58:30.678185+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":19,"num_write":17,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":692,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":"367","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.417894+0000","last_change":"2021-03-24T21:59:12.939396+0000","last_active":"2021-03-24T21:59:17.417894+0000","last_peered":"2021-03-24T21:59:17.417894+0000","last_clean":"2021-03-24T21:59:17.417894+0000","last_became_active":"2021-03-24T21:54:38.109439+0000","last_became_peered":"2021-03-24T21:54:38.109439+0000","last_unstale":"2021-03-24T21:59:17.417894+0000","last_undegraded":"2021-03-24T21:59:17.417894+0000","last_fullsized":"2021-03-24T21:59:17.417894+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:12.939226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:59:12.939226+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"361","reported_epoch":"387","state":"active+undersized","last_fresh":"2021-03-24T21:59:17.455299+0000","last_change":"2021-03-24T21:58:31.229441+0000","last_active":"2021-03-24T21:59:17.455299+0000","last_peered":"2021-03-24T21:59:17.455299+0000","last_clean":"2021-03-24T21:58:29.208467+0000","last_became_active":"2021-03-24T21:58:31.229441+0000","last_became_peered":"2021-03-24T21:58:31.229441+0000","last_unstale":"2021-03-24T21:59:17.455299+0000","last_undegraded":"2021-03-24T21:59:17.455299+0000","last_fullsized":"2021-03-24T21:58:31.197372+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:24.548819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:22.724840+0000","last_clean_scrub_stamp":"2021-03-24T21:55:24.548819+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"357'133","reported_seq":"938","reported_epoch":"387","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:59:17.422071+0000","last_change":"2021-03-24T21:58:31.226756+0000","last_active":"2021-03-24T21:59:17.422071+0000","last_peered":"2021-03-24T21:59:17.422071+0000","last_clean":"2021-03-24T21:58:29.207481+0000","last_became_active":"2021-03-24T21:58:31.196478+0000","last_became_peered":"2021-03-24T21:58:31.196478+0000","last_unstale":"2021-03-24T21:59:17.422071+0000","last_undegraded":"2021-03-24T21:58:31.226366+0000","last_fullsized":"2021-03-24T21:58:31.185724+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":133,"ondisk_log_size":133,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":19,"stat_sum":{"num_bytes":10061614,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":26,"num_read_kb":22368,"num_write":37,"num_write_kb":16867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"109'4","reported_seq":"467","reported_epoch":"388","state":"active+clean","last_fresh":"2021-03-24T21:59:18.444046+0000","last_change":"2021-03-24T21:59:06.906875+0000","last_active":"2021-03-24T21:59:18.444046+0000","last_peered":"2021-03-24T21:59:18.444046+0000","last_clean":"2021-03-24T21:59:18.444046+0000","last_became_active":"2021-03-24T21:54:38.125497+0000","last_became_peered":"2021-03-24T21:54:38.125497+0000","last_unstale":"2021-03-24T21:59:18.444046+0000","last_undegraded":"2021-03-24T21:59:18.444046+0000","last_fullsized":"2021-03-24T21:59:18.444046+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'4","last_scrub_stamp":"2021-03-24T21:59:06.906705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:59:06.906705+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"97'28","reported_seq":"135","reported_epoch":"387","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:59:17.412969+0000","last_change":"2021-03-24T21:58:31.202742+0000","last_active":"2021-03-24T21:59:17.412969+0000","last_peered":"2021-03-24T21:59:17.412969+0000","last_clean":"2021-03-24T21:54:19.144778+0000","last_became_active":"2021-03-24T21:58:31.202742+0000","last_became_peered":"2021-03-24T21:58:31.202742+0000","last_unstale":"2021-03-24T21:59:17.412969+0000","last_undegraded":"2021-03-24T21:58:31.193830+0000","last_fullsized":"2021-03-24T21:58:31.193649+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'28","last_scrub_stamp":"2021-03-24T21:58:20.925603+0000","last_deep_scrub":"97'28","last_deep_scrub_stamp":"2021-03-24T21:54:19.157265+0000","last_clean_scrub_stamp":"2021-03-24T21:58:20.925603+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":20,"num_write":19,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[{"shards":"4,7","objects":10}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"89'4","reported_seq":"353","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.417493+0000","last_change":"2021-03-24T21:58:25.970055+0000","last_active":"2021-03-24T21:59:17.417493+0000","last_peered":"2021-03-24T21:59:17.417493+0000","last_clean":"2021-03-24T21:59:17.417493+0000","last_became_active":"2021-03-24T21:51:45.385110+0000","last_became_peered":"2021-03-24T21:51:45.385110+0000","last_unstale":"2021-03-24T21:59:17.417493+0000","last_undegraded":"2021-03-24T21:59:17.417493+0000","last_fullsized":"2021-03-24T21:59:17.417493+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'4","last_scrub_stamp":"2021-03-24T21:58:25.969865+0000","last_deep_scrub":"89'4","last_deep_scrub_stamp":"2021-03-24T21:53:19.926637+0000","last_clean_scrub_stamp":"2021-03-24T21:58:25.969865+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"327","reported_epoch":"387","state":"active+undersized","last_fresh":"2021-03-24T21:59:17.409367+0000","last_change":"2021-03-24T21:58:31.222499+0000","last_active":"2021-03-24T21:59:17.409367+0000","last_peered":"2021-03-24T21:59:17.409367+0000","last_clean":"2021-03-24T21:58:29.212838+0000","last_became_active":"2021-03-24T21:58:31.222499+0000","last_became_peered":"2021-03-24T21:58:31.222499+0000","last_unstale":"2021-03-24T21:59:17.409367+0000","last_undegraded":"2021-03-24T21:59:17.409367+0000","last_fullsized":"2021-03-24T21:58:31.188323+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:54.262681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:54.262681+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"272'275","reported_seq":"1030","reported_epoch":"391","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:59:21.211622+0000","last_change":"2021-03-24T21:58:31.208031+0000","last_active":"2021-03-24T21:59:21.211622+0000","last_peered":"2021-03-24T21:59:21.211622+0000","last_clean":"2021-03-24T21:57:22.173331+0000","last_became_active":"2021-03-24T21:58:31.208031+0000","last_became_peered":"2021-03-24T21:58:31.208031+0000","last_unstale":"2021-03-24T21:59:21.211622+0000","last_undegraded":"2021-03-24T21:58:31.193214+0000","last_fullsized":"2021-03-24T21:58:31.192237+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'275","last_scrub_stamp":"2021-03-24T21:58:23.942036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:58:23.942036+0000","log_size":275,"ondisk_log_size":275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":21,"stat_sum":{"num_bytes":8857338,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":114,"num_read_kb":132714,"num_write":52,"num_write_kb":22259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":12587455,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":3}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"69'7","reported_seq":"376","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.438681+0000","last_change":"2021-03-24T21:59:06.367269+0000","last_active":"2021-03-24T21:59:17.438681+0000","last_peered":"2021-03-24T21:59:17.438681+0000","last_clean":"2021-03-24T21:59:17.438681+0000","last_became_active":"2021-03-24T21:54:38.144098+0000","last_became_peered":"2021-03-24T21:54:38.144098+0000","last_unstale":"2021-03-24T21:59:17.438681+0000","last_undegraded":"2021-03-24T21:59:17.438681+0000","last_fullsized":"2021-03-24T21:59:17.438681+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'7","last_scrub_stamp":"2021-03-24T21:59:06.367172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:59:06.367172+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":7,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":158,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":"313","reported_epoch":"387","state":"active+undersized","last_fresh":"2021-03-24T21:59:17.407535+0000","last_change":"2021-03-24T21:58:31.222518+0000","last_active":"2021-03-24T21:59:17.407535+0000","last_peered":"2021-03-24T21:59:17.407535+0000","last_clean":"2021-03-24T21:58:29.212134+0000","last_became_active":"2021-03-24T21:58:31.222518+0000","last_became_peered":"2021-03-24T21:58:31.222518+0000","last_unstale":"2021-03-24T21:59:17.407535+0000","last_undegraded":"2021-03-24T21:59:17.407535+0000","last_fullsized":"2021-03-24T21:58:31.192669+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:16.270874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:16.270874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"114","reported_epoch":"387","state":"active+undersized","last_fresh":"2021-03-24T21:59:17.439327+0000","last_change":"2021-03-24T21:58:31.202521+0000","last_active":"2021-03-24T21:59:17.439327+0000","last_peered":"2021-03-24T21:59:17.439327+0000","last_clean":"2021-03-24T21:54:36.088165+0000","last_became_active":"2021-03-24T21:58:31.202521+0000","last_became_peered":"2021-03-24T21:58:31.202521+0000","last_unstale":"2021-03-24T21:59:17.439327+0000","last_undegraded":"2021-03-24T21:59:17.439327+0000","last_fullsized":"2021-03-24T21:58:31.187228+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:48.144391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:48.144391+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"357","reported_epoch":"387","state":"active+undersized","last_fresh":"2021-03-24T21:59:17.418346+0000","last_change":"2021-03-24T21:58:31.203277+0000","last_active":"2021-03-24T21:59:17.418346+0000","last_peered":"2021-03-24T21:59:17.418346+0000","last_clean":"2021-03-24T21:58:29.203193+0000","last_became_active":"2021-03-24T21:58:31.203277+0000","last_became_peered":"2021-03-24T21:58:31.203277+0000","last_unstale":"2021-03-24T21:59:17.418346+0000","last_undegraded":"2021-03-24T21:59:17.418346+0000","last_fullsized":"2021-03-24T21:58:31.196438+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:04.012279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:58:04.012279+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.14","version":"382'308","reported_seq":"1170","reported_epoch":"387","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:59:17.439145+0000","last_change":"2021-03-24T21:58:31.195219+0000","last_active":"2021-03-24T21:59:17.439145+0000","last_peered":"2021-03-24T21:59:17.439145+0000","last_clean":"2021-03-24T21:58:29.206770+0000","last_became_active":"2021-03-24T21:58:31.195219+0000","last_became_peered":"2021-03-24T21:58:31.195219+0000","last_unstale":"2021-03-24T21:59:17.439145+0000","last_undegraded":"2021-03-24T21:58:31.186610+0000","last_fullsized":"2021-03-24T21:58:31.186105+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":308,"ondisk_log_size":308,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":19,"stat_sum":{"num_bytes":26383848,"num_objects":14,"num_object_clones":11,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":132,"num_read_kb":117450,"num_write":129,"num_write_kb":40497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":6105417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":14}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"69'16","reported_seq":"376","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.413667+0000","last_change":"2021-03-24T21:56:40.303286+0000","last_active":"2021-03-24T21:59:17.413667+0000","last_peered":"2021-03-24T21:59:17.413667+0000","last_clean":"2021-03-24T21:59:17.413667+0000","last_became_active":"2021-03-24T21:51:47.687595+0000","last_became_peered":"2021-03-24T21:51:47.687595+0000","last_unstale":"2021-03-24T21:59:17.413667+0000","last_undegraded":"2021-03-24T21:59:17.413667+0000","last_fullsized":"2021-03-24T21:59:17.413667+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T21:56:40.303200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:40.303200+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":38,"num_read_kb":18,"num_write":17,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":"150","reported_epoch":"391","state":"active+clean","last_fresh":"2021-03-24T21:59:21.212300+0000","last_change":"2021-03-24T21:57:57.288460+0000","last_active":"2021-03-24T21:59:21.212300+0000","last_peered":"2021-03-24T21:59:21.212300+0000","last_clean":"2021-03-24T21:59:21.212300+0000","last_became_active":"2021-03-24T21:57:57.287481+0000","last_became_peered":"2021-03-24T21:57:57.287481+0000","last_unstale":"2021-03-24T21:59:21.212300+0000","last_undegraded":"2021-03-24T21:59:21.212300+0000","last_fullsized":"2021-03-24T21:59:21.212300+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:51.037780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:51.037780+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"353","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.438901+0000","last_change":"2021-03-24T21:57:52.376563+0000","last_active":"2021-03-24T21:59:17.438901+0000","last_peered":"2021-03-24T21:59:17.438901+0000","last_clean":"2021-03-24T21:59:17.438901+0000","last_became_active":"2021-03-24T21:54:38.129390+0000","last_became_peered":"2021-03-24T21:54:38.129390+0000","last_unstale":"2021-03-24T21:59:17.438901+0000","last_undegraded":"2021-03-24T21:59:17.438901+0000","last_fullsized":"2021-03-24T21:59:17.438901+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:52.376475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:55.493713+0000","last_clean_scrub_stamp":"2021-03-24T21:57:52.376475+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"349","reported_epoch":"390","state":"active+clean","last_fresh":"2021-03-24T21:59:20.207674+0000","last_change":"2021-03-24T21:58:59.862111+0000","last_active":"2021-03-24T21:59:20.207674+0000","last_peered":"2021-03-24T21:59:20.207674+0000","last_clean":"2021-03-24T21:59:20.207674+0000","last_became_active":"2021-03-24T21:51:51.754085+0000","last_became_peered":"2021-03-24T21:51:51.754085+0000","last_unstale":"2021-03-24T21:59:20.207674+0000","last_undegraded":"2021-03-24T21:59:20.207674+0000","last_fullsized":"2021-03-24T21:59:20.207674+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:59.862021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:58:59.862021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"377'101","reported_seq":"956","reported_epoch":"387","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:59:17.418618+0000","last_change":"2021-03-24T21:58:31.209906+0000","last_active":"2021-03-24T21:59:17.418618+0000","last_peered":"2021-03-24T21:59:17.418618+0000","last_clean":"2021-03-24T21:58:29.203501+0000","last_became_active":"2021-03-24T21:58:31.209906+0000","last_became_peered":"2021-03-24T21:58:31.209906+0000","last_unstale":"2021-03-24T21:59:17.418618+0000","last_undegraded":"2021-03-24T21:58:31.197019+0000","last_fullsized":"2021-03-24T21:58:31.196878+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":5,"last_scrub":"233'79","last_scrub_stamp":"2021-03-24T21:56:50.934781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:56:50.934781+0000","log_size":101,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":19,"stat_sum":{"num_bytes":10252468,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":8830,"num_write":18,"num_write_kb":7987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":3}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"69'13","reported_seq":"92","reported_epoch":"387","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:59:17.419025+0000","last_change":"2021-03-24T21:58:31.204551+0000","last_active":"2021-03-24T21:59:17.419025+0000","last_peered":"2021-03-24T21:59:17.419025+0000","last_clean":"2021-03-24T21:51:54.397987+0000","last_became_active":"2021-03-24T21:58:31.204551+0000","last_became_peered":"2021-03-24T21:58:31.204551+0000","last_unstale":"2021-03-24T21:59:17.419025+0000","last_undegraded":"2021-03-24T21:58:31.199681+0000","last_fullsized":"2021-03-24T21:58:31.199516+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2021-03-24T21:56:47.168558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:47.168558+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":14,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[{"shards":"3,7","objects":8}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":"348","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.409170+0000","last_change":"2021-03-24T21:56:32.279216+0000","last_active":"2021-03-24T21:59:17.409170+0000","last_peered":"2021-03-24T21:59:17.409170+0000","last_clean":"2021-03-24T21:59:17.409170+0000","last_became_active":"2021-03-24T21:51:45.381516+0000","last_became_peered":"2021-03-24T21:51:45.381516+0000","last_unstale":"2021-03-24T21:59:17.409170+0000","last_undegraded":"2021-03-24T21:59:17.409170+0000","last_fullsized":"2021-03-24T21:59:17.409170+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:32.279119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:05.473348+0000","last_clean_scrub_stamp":"2021-03-24T21:56:32.279119+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"235'4","reported_seq":"450","reported_epoch":"388","state":"active+clean","last_fresh":"2021-03-24T21:59:18.418844+0000","last_change":"2021-03-24T21:58:20.324676+0000","last_active":"2021-03-24T21:59:18.418844+0000","last_peered":"2021-03-24T21:59:18.418844+0000","last_clean":"2021-03-24T21:59:18.418844+0000","last_became_active":"2021-03-24T21:56:44.224410+0000","last_became_peered":"2021-03-24T21:56:44.224410+0000","last_unstale":"2021-03-24T21:59:18.418844+0000","last_undegraded":"2021-03-24T21:59:18.418844+0000","last_fullsized":"2021-03-24T21:59:18.418844+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"235'4","last_scrub_stamp":"2021-03-24T21:58:20.324557+0000","last_deep_scrub":"86'3","last_deep_scrub_stamp":"2021-03-24T21:52:56.678111+0000","last_clean_scrub_stamp":"2021-03-24T21:58:20.324557+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"355","reported_epoch":"391","state":"active+undersized","last_fresh":"2021-03-24T21:59:21.217505+0000","last_change":"2021-03-24T21:58:31.200854+0000","last_active":"2021-03-24T21:59:21.217505+0000","last_peered":"2021-03-24T21:59:21.217505+0000","last_clean":"2021-03-24T21:58:29.218648+0000","last_became_active":"2021-03-24T21:58:31.200854+0000","last_became_peered":"2021-03-24T21:58:31.200854+0000","last_unstale":"2021-03-24T21:59:21.217505+0000","last_undegraded":"2021-03-24T21:59:21.217505+0000","last_fullsized":"2021-03-24T21:58:31.197192+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:51.924963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:51.924963+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.16","version":"392'187","reported_seq":"1114","reported_epoch":"392","state":"active+clean","last_fresh":"2021-03-24T21:59:22.324124+0000","last_change":"2021-03-24T21:59:22.324546+0000","last_active":"2021-03-24T21:59:22.324124+0000","last_peered":"2021-03-24T21:59:22.324124+0000","last_clean":"2021-03-24T21:59:22.324124+0000","last_became_active":"2021-03-24T21:56:13.127048+0000","last_became_peered":"2021-03-24T21:56:13.127048+0000","last_unstale":"2021-03-24T21:59:22.324124+0000","last_undegraded":"2021-03-24T21:59:22.324124+0000","last_fullsized":"2021-03-24T21:59:22.324124+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'146","last_scrub_stamp":"2021-03-24T21:57:22.669305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:57:22.669305+0000","log_size":187,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2739607,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":27,"num_read_kb":26576,"num_write":12,"num_write_kb":2219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d6","length":"1"},{"start":"df","length":"1"},{"start":"e1","length":"1"},{"start":"e8","length":"1"},{"start":"ea","length":"4"},{"start":"ef","length":"1"},{"start":"f1","length":"5"},{"start":"fb","length":"3"},{"start":"100","length":"1"},{"start":"109","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"}]},{"pgid":"4.14","version":"69'14","reported_seq":"401","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.440201+0000","last_change":"2021-03-24T21:58:57.435463+0000","last_active":"2021-03-24T21:59:17.440201+0000","last_peered":"2021-03-24T21:59:17.440201+0000","last_clean":"2021-03-24T21:59:17.440201+0000","last_became_active":"2021-03-24T21:54:38.143783+0000","last_became_peered":"2021-03-24T21:54:38.143783+0000","last_unstale":"2021-03-24T21:59:17.440201+0000","last_undegraded":"2021-03-24T21:59:17.440201+0000","last_fullsized":"2021-03-24T21:59:17.440201+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'14","last_scrub_stamp":"2021-03-24T21:58:57.435358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:58:57.435358+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":27,"num_read_kb":14,"num_write":14,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":"353","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.415320+0000","last_change":"2021-03-24T21:59:15.949641+0000","last_active":"2021-03-24T21:59:17.415320+0000","last_peered":"2021-03-24T21:59:17.415320+0000","last_clean":"2021-03-24T21:59:17.415320+0000","last_became_active":"2021-03-24T21:51:45.579271+0000","last_became_peered":"2021-03-24T21:51:45.579271+0000","last_unstale":"2021-03-24T21:59:17.415320+0000","last_undegraded":"2021-03-24T21:59:17.415320+0000","last_fullsized":"2021-03-24T21:59:17.415320+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:15.949538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:59:15.949538+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"343'5","reported_seq":"114","reported_epoch":"387","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:59:17.411597+0000","last_change":"2021-03-24T21:58:31.223133+0000","last_active":"2021-03-24T21:59:17.411597+0000","last_peered":"2021-03-24T21:59:17.411597+0000","last_clean":"2021-03-24T21:53:08.809615+0000","last_became_active":"2021-03-24T21:58:31.223133+0000","last_became_peered":"2021-03-24T21:58:31.223133+0000","last_unstale":"2021-03-24T21:59:17.411597+0000","last_undegraded":"2021-03-24T21:58:31.192057+0000","last_fullsized":"2021-03-24T21:58:31.191934+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'4","last_scrub_stamp":"2021-03-24T21:56:42.209497+0000","last_deep_scrub":"86'3","last_deep_scrub_stamp":"2021-03-24T21:53:08.815163+0000","last_clean_scrub_stamp":"2021-03-24T21:56:42.209497+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"342","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.411364+0000","last_change":"2021-03-24T21:57:58.768856+0000","last_active":"2021-03-24T21:59:17.411364+0000","last_peered":"2021-03-24T21:59:17.411364+0000","last_clean":"2021-03-24T21:59:17.411364+0000","last_became_active":"2021-03-24T21:51:51.756584+0000","last_became_peered":"2021-03-24T21:51:51.756584+0000","last_unstale":"2021-03-24T21:59:17.411364+0000","last_undegraded":"2021-03-24T21:59:17.411364+0000","last_fullsized":"2021-03-24T21:59:17.411364+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:58.768675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:57:58.768675+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.17","version":"382'388","reported_seq":"1351","reported_epoch":"387","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:59:17.416813+0000","last_change":"2021-03-24T21:58:31.208787+0000","last_active":"2021-03-24T21:59:17.416813+0000","last_peered":"2021-03-24T21:59:17.416813+0000","last_clean":"2021-03-24T21:58:29.232381+0000","last_became_active":"2021-03-24T21:58:31.208787+0000","last_became_peered":"2021-03-24T21:58:31.208787+0000","last_unstale":"2021-03-24T21:59:17.416813+0000","last_undegraded":"2021-03-24T21:58:31.195157+0000","last_fullsized":"2021-03-24T21:58:31.194997+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":5,"last_scrub":"265'274","last_scrub_stamp":"2021-03-24T21:57:14.281739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:57:14.281739+0000","log_size":388,"ondisk_log_size":388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":19,"stat_sum":{"num_bytes":37875040,"num_objects":15,"num_object_clones":11,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":107,"num_read_kb":124358,"num_write":53,"num_write_kb":27261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":15}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"69'15","reported_seq":"442","reported_epoch":"391","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:59:21.213416+0000","last_change":"2021-03-24T21:58:31.200974+0000","last_active":"2021-03-24T21:59:21.213416+0000","last_peered":"2021-03-24T21:59:21.213416+0000","last_clean":"2021-03-24T21:58:29.215183+0000","last_became_active":"2021-03-24T21:58:31.200974+0000","last_became_peered":"2021-03-24T21:58:31.200974+0000","last_unstale":"2021-03-24T21:59:21.213416+0000","last_undegraded":"2021-03-24T21:58:31.193428+0000","last_fullsized":"2021-03-24T21:58:31.193298+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:58:02.548166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:58:02.548166+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[{"shards":"2,4","objects":9}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":"362","reported_epoch":"391","state":"active+clean","last_fresh":"2021-03-24T21:59:21.214291+0000","last_change":"2021-03-24T21:58:29.716360+0000","last_active":"2021-03-24T21:59:21.214291+0000","last_peered":"2021-03-24T21:59:21.214291+0000","last_clean":"2021-03-24T21:59:21.214291+0000","last_became_active":"2021-03-24T21:54:24.181343+0000","last_became_peered":"2021-03-24T21:54:24.181343+0000","last_unstale":"2021-03-24T21:59:21.214291+0000","last_undegraded":"2021-03-24T21:59:21.214291+0000","last_fullsized":"2021-03-24T21:59:21.214291+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:29.716186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:58:29.716186+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"189","reported_epoch":"387","state":"active+undersized","last_fresh":"2021-03-24T21:59:17.414635+0000","last_change":"2021-03-24T21:58:31.205120+0000","last_active":"2021-03-24T21:59:17.414635+0000","last_peered":"2021-03-24T21:59:17.414635+0000","last_clean":"2021-03-24T21:56:33.303502+0000","last_became_active":"2021-03-24T21:58:31.205120+0000","last_became_peered":"2021-03-24T21:58:31.205120+0000","last_unstale":"2021-03-24T21:59:17.414635+0000","last_undegraded":"2021-03-24T21:59:17.414635+0000","last_fullsized":"2021-03-24T21:58:31.195804+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:31.118050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:57:31.118050+0000","last_clean_scrub_stamp":"2021-03-24T21:57:31.118050+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"343","reported_epoch":"387","state":"active+undersized","last_fresh":"2021-03-24T21:59:17.422150+0000","last_change":"2021-03-24T21:58:31.219635+0000","last_active":"2021-03-24T21:59:17.422150+0000","last_peered":"2021-03-24T21:59:17.422150+0000","last_clean":"2021-03-24T21:58:29.204869+0000","last_became_active":"2021-03-24T21:58:31.219635+0000","last_became_peered":"2021-03-24T21:58:31.219635+0000","last_unstale":"2021-03-24T21:59:17.422150+0000","last_undegraded":"2021-03-24T21:59:17.422150+0000","last_fullsized":"2021-03-24T21:58:31.184834+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:22.555624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:23.707914+0000","last_clean_scrub_stamp":"2021-03-24T21:55:22.555624+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.10","version":"385'43","reported_seq":"370","reported_epoch":"387","state":"active+undersized","last_fresh":"2021-03-24T21:59:17.409721+0000","last_change":"2021-03-24T21:58:31.206788+0000","last_active":"2021-03-24T21:59:17.409721+0000","last_peered":"2021-03-24T21:59:17.409721+0000","last_clean":"2021-03-24T21:58:27.206026+0000","last_became_active":"2021-03-24T21:58:31.206788+0000","last_became_peered":"2021-03-24T21:58:31.206788+0000","last_unstale":"2021-03-24T21:59:17.409721+0000","last_undegraded":"2021-03-24T21:59:17.409721+0000","last_fullsized":"2021-03-24T21:58:31.185192+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":5,"last_scrub":"277'24","last_scrub_stamp":"2021-03-24T21:57:28.119420+0000","last_deep_scrub":"277'24","last_deep_scrub_stamp":"2021-03-24T21:57:28.119420+0000","last_clean_scrub_stamp":"2021-03-24T21:57:28.119420+0000","log_size":43,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":19,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":23,"num_read_kb":18238,"num_write":11,"num_write_kb":5221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"69'15","reported_seq":"369","reported_epoch":"390","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:59:20.211118+0000","last_change":"2021-03-24T21:58:31.205390+0000","last_active":"2021-03-24T21:59:20.211118+0000","last_peered":"2021-03-24T21:59:20.211118+0000","last_clean":"2021-03-24T21:58:29.216491+0000","last_became_active":"2021-03-24T21:58:31.205390+0000","last_became_peered":"2021-03-24T21:58:31.205390+0000","last_unstale":"2021-03-24T21:59:20.211118+0000","last_undegraded":"2021-03-24T21:58:31.194807+0000","last_fullsized":"2021-03-24T21:58:31.190627+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:57:52.248247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:57:52.248247+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":34,"num_read_kb":16,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":10}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":"348","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.418466+0000","last_change":"2021-03-24T21:58:04.965522+0000","last_active":"2021-03-24T21:59:17.418466+0000","last_peered":"2021-03-24T21:59:17.418466+0000","last_clean":"2021-03-24T21:59:17.418466+0000","last_became_active":"2021-03-24T21:51:45.379383+0000","last_became_peered":"2021-03-24T21:51:45.379383+0000","last_unstale":"2021-03-24T21:59:17.418466+0000","last_undegraded":"2021-03-24T21:59:17.418466+0000","last_fullsized":"2021-03-24T21:59:17.418466+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:04.965369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:58:04.965369+0000","last_clean_scrub_stamp":"2021-03-24T21:58:04.965369+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"354","reported_epoch":"387","state":"active+undersized","last_fresh":"2021-03-24T21:59:17.453588+0000","last_change":"2021-03-24T21:58:31.231202+0000","last_active":"2021-03-24T21:59:17.453588+0000","last_peered":"2021-03-24T21:59:17.453588+0000","last_clean":"2021-03-24T21:58:29.207905+0000","last_became_active":"2021-03-24T21:58:31.231202+0000","last_became_peered":"2021-03-24T21:58:31.231202+0000","last_unstale":"2021-03-24T21:59:17.453588+0000","last_undegraded":"2021-03-24T21:59:17.453588+0000","last_fullsized":"2021-03-24T21:58:31.201837+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:32.480115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:32.480115+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"357","reported_epoch":"387","state":"active+undersized","last_fresh":"2021-03-24T21:59:17.407150+0000","last_change":"2021-03-24T21:58:31.223636+0000","last_active":"2021-03-24T21:59:17.407150+0000","last_peered":"2021-03-24T21:59:17.407150+0000","last_clean":"2021-03-24T21:58:29.211869+0000","last_became_active":"2021-03-24T21:58:31.223636+0000","last_became_peered":"2021-03-24T21:58:31.223636+0000","last_unstale":"2021-03-24T21:59:17.407150+0000","last_undegraded":"2021-03-24T21:59:17.407150+0000","last_fullsized":"2021-03-24T21:58:31.193002+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:52.226117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:52.226117+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.11","version":"387'439","reported_seq":"1400","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.474757+0000","last_change":"2021-03-24T21:59:17.474757+0000","last_active":"2021-03-24T21:59:17.474757+0000","last_peered":"2021-03-24T21:59:17.474757+0000","last_clean":"2021-03-24T21:59:17.474757+0000","last_became_active":"2021-03-24T21:56:14.138730+0000","last_became_peered":"2021-03-24T21:56:14.138730+0000","last_unstale":"2021-03-24T21:59:17.474757+0000","last_undegraded":"2021-03-24T21:59:17.474757+0000","last_fullsized":"2021-03-24T21:59:17.474757+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":5,"last_scrub":"381'425","last_scrub_stamp":"2021-03-24T21:59:11.936662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:59:11.936662+0000","log_size":439,"ondisk_log_size":439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22450769,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":70,"num_read_kb":85778,"num_write":54,"num_write_kb":20909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d6","length":"1"},{"start":"df","length":"1"},{"start":"e1","length":"1"},{"start":"e8","length":"1"},{"start":"ea","length":"4"},{"start":"ef","length":"1"},{"start":"f2","length":"4"},{"start":"fb","length":"3"},{"start":"109","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"}]},{"pgid":"4.11","version":"86'22","reported_seq":"407","reported_epoch":"387","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:59:17.422561+0000","last_change":"2021-03-24T21:58:31.202532+0000","last_active":"2021-03-24T21:59:17.422561+0000","last_peered":"2021-03-24T21:59:17.422561+0000","last_clean":"2021-03-24T21:58:29.206035+0000","last_became_active":"2021-03-24T21:58:31.202532+0000","last_became_peered":"2021-03-24T21:58:31.202532+0000","last_unstale":"2021-03-24T21:59:17.422561+0000","last_undegraded":"2021-03-24T21:58:31.187011+0000","last_fullsized":"2021-03-24T21:58:31.186847+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'22","last_scrub_stamp":"2021-03-24T21:54:51.585058+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2021-03-24T21:52:54.597505+0000","last_clean_scrub_stamp":"2021-03-24T21:54:51.585058+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":16,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":580,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":8}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":"118","reported_epoch":"387","state":"active+undersized","last_fresh":"2021-03-24T21:59:17.423505+0000","last_change":"2021-03-24T21:58:31.202328+0000","last_active":"2021-03-24T21:59:17.423505+0000","last_peered":"2021-03-24T21:59:17.423505+0000","last_clean":"2021-03-24T21:54:36.088286+0000","last_became_active":"2021-03-24T21:58:31.202328+0000","last_became_peered":"2021-03-24T21:58:31.202328+0000","last_unstale":"2021-03-24T21:59:17.423505+0000","last_undegraded":"2021-03-24T21:59:17.423505+0000","last_fullsized":"2021-03-24T21:58:31.186585+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:13.016117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:13.016117+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"353","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.413192+0000","last_change":"2021-03-24T21:57:53.093497+0000","last_active":"2021-03-24T21:59:17.413192+0000","last_peered":"2021-03-24T21:59:17.413192+0000","last_clean":"2021-03-24T21:59:17.413192+0000","last_became_active":"2021-03-24T21:54:24.194423+0000","last_became_peered":"2021-03-24T21:54:24.194423+0000","last_unstale":"2021-03-24T21:59:17.413192+0000","last_undegraded":"2021-03-24T21:59:17.413192+0000","last_fullsized":"2021-03-24T21:59:17.413192+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:53.093327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:57:53.093327+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"351","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.455080+0000","last_change":"2021-03-24T21:59:15.289579+0000","last_active":"2021-03-24T21:59:17.455080+0000","last_peered":"2021-03-24T21:59:17.455080+0000","last_clean":"2021-03-24T21:59:17.455080+0000","last_became_active":"2021-03-24T21:54:24.202686+0000","last_became_peered":"2021-03-24T21:54:24.202686+0000","last_unstale":"2021-03-24T21:59:17.455080+0000","last_undegraded":"2021-03-24T21:59:17.455080+0000","last_fullsized":"2021-03-24T21:59:17.455080+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:15.289444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:47.540793+0000","last_clean_scrub_stamp":"2021-03-24T21:59:15.289444+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.12","version":"183'26","reported_seq":"570","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.410367+0000","last_change":"2021-03-24T21:59:17.410367+0000","last_active":"2021-03-24T21:59:17.410367+0000","last_peered":"2021-03-24T21:59:17.410367+0000","last_clean":"2021-03-24T21:59:17.410367+0000","last_became_active":"2021-03-24T21:56:10.130886+0000","last_became_peered":"2021-03-24T21:56:10.130886+0000","last_unstale":"2021-03-24T21:59:17.410367+0000","last_undegraded":"2021-03-24T21:59:17.410367+0000","last_fullsized":"2021-03-24T21:59:17.410367+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":5,"last_scrub":"183'26","last_scrub_stamp":"2021-03-24T21:57:53.859843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:57:53.859843+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"d6","length":"1"},{"start":"df","length":"1"},{"start":"e1","length":"1"},{"start":"e8","length":"1"},{"start":"ea","length":"4"},{"start":"ef","length":"1"},{"start":"f2","length":"4"},{"start":"fb","length":"3"},{"start":"109","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"}]},{"pgid":"4.10","version":"69'12","reported_seq":"469","reported_epoch":"387","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:59:17.409444+0000","last_change":"2021-03-24T21:58:31.206182+0000","last_active":"2021-03-24T21:59:17.409444+0000","last_peered":"2021-03-24T21:59:17.409444+0000","last_clean":"2021-03-24T21:58:29.204663+0000","last_became_active":"2021-03-24T21:58:31.206182+0000","last_became_peered":"2021-03-24T21:58:31.206182+0000","last_unstale":"2021-03-24T21:59:17.409444+0000","last_undegraded":"2021-03-24T21:58:31.184913+0000","last_fullsized":"2021-03-24T21:58:31.184684+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2021-03-24T21:56:53.443554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:53.443554+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":12,"num_write":12,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":3756,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":10}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":"354","reported_epoch":"387","state":"active+undersized","last_fresh":"2021-03-24T21:59:17.417831+0000","last_change":"2021-03-24T21:58:31.222882+0000","last_active":"2021-03-24T21:59:17.417831+0000","last_peered":"2021-03-24T21:59:17.417831+0000","last_clean":"2021-03-24T21:58:29.214071+0000","last_became_active":"2021-03-24T21:58:31.222882+0000","last_became_peered":"2021-03-24T21:58:31.222882+0000","last_unstale":"2021-03-24T21:59:17.417831+0000","last_undegraded":"2021-03-24T21:59:17.417831+0000","last_fullsized":"2021-03-24T21:58:31.191156+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:11.229895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:11.229895+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"311","reported_epoch":"391","state":"active+clean","last_fresh":"2021-03-24T21:59:21.210887+0000","last_change":"2021-03-24T21:57:56.549118+0000","last_active":"2021-03-24T21:59:21.210887+0000","last_peered":"2021-03-24T21:59:21.210887+0000","last_clean":"2021-03-24T21:59:21.210887+0000","last_became_active":"2021-03-24T21:54:24.187568+0000","last_became_peered":"2021-03-24T21:54:24.187568+0000","last_unstale":"2021-03-24T21:59:21.210887+0000","last_undegraded":"2021-03-24T21:59:21.210887+0000","last_fullsized":"2021-03-24T21:59:21.210887+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:56.548933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:57:56.548933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"342","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.416480+0000","last_change":"2021-03-24T21:57:59.979943+0000","last_active":"2021-03-24T21:59:17.416480+0000","last_peered":"2021-03-24T21:59:17.416480+0000","last_clean":"2021-03-24T21:59:17.416480+0000","last_became_active":"2021-03-24T21:51:51.762150+0000","last_became_peered":"2021-03-24T21:51:51.762150+0000","last_unstale":"2021-03-24T21:59:17.416480+0000","last_undegraded":"2021-03-24T21:59:17.416480+0000","last_fullsized":"2021-03-24T21:59:17.416480+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:59.979772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:57:59.979772+0000","last_clean_scrub_stamp":"2021-03-24T21:57:59.979772+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.13","version":"382'289","reported_seq":"1170","reported_epoch":"387","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:59:17.408502+0000","last_change":"2021-03-24T21:58:31.208850+0000","last_active":"2021-03-24T21:59:17.408502+0000","last_peered":"2021-03-24T21:59:17.408502+0000","last_clean":"2021-03-24T21:58:29.214984+0000","last_became_active":"2021-03-24T21:58:31.208850+0000","last_became_peered":"2021-03-24T21:58:31.208850+0000","last_unstale":"2021-03-24T21:59:17.408502+0000","last_undegraded":"2021-03-24T21:58:31.188986+0000","last_fullsized":"2021-03-24T21:58:31.188796+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":289,"ondisk_log_size":289,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":19,"stat_sum":{"num_bytes":26783272,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":143,"num_read_kb":142958,"num_write":101,"num_write_kb":48247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":4773188,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":12}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"69'17","reported_seq":"382","reported_epoch":"387","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:59:17.416901+0000","last_change":"2021-03-24T21:58:31.200099+0000","last_active":"2021-03-24T21:59:17.416901+0000","last_peered":"2021-03-24T21:59:17.416901+0000","last_clean":"2021-03-24T21:58:29.213514+0000","last_became_active":"2021-03-24T21:58:31.200099+0000","last_became_peered":"2021-03-24T21:58:31.200099+0000","last_unstale":"2021-03-24T21:59:17.416901+0000","last_undegraded":"2021-03-24T21:58:31.189311+0000","last_fullsized":"2021-03-24T21:58:31.189194+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'17","last_scrub_stamp":"2021-03-24T21:55:32.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:55:32.214502+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":37,"num_read_kb":18,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":2100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":11}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":"357","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.409324+0000","last_change":"2021-03-24T21:59:00.393853+0000","last_active":"2021-03-24T21:59:17.409324+0000","last_peered":"2021-03-24T21:59:17.409324+0000","last_clean":"2021-03-24T21:59:17.409324+0000","last_became_active":"2021-03-24T21:54:38.145870+0000","last_became_peered":"2021-03-24T21:54:38.145870+0000","last_unstale":"2021-03-24T21:59:17.409324+0000","last_undegraded":"2021-03-24T21:59:17.409324+0000","last_fullsized":"2021-03-24T21:59:17.409324+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:00.393742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:59:00.393742+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"353","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.413826+0000","last_change":"2021-03-24T21:59:17.040729+0000","last_active":"2021-03-24T21:59:17.413826+0000","last_peered":"2021-03-24T21:59:17.413826+0000","last_clean":"2021-03-24T21:59:17.413826+0000","last_became_active":"2021-03-24T21:56:35.140997+0000","last_became_peered":"2021-03-24T21:56:35.140997+0000","last_unstale":"2021-03-24T21:59:17.413826+0000","last_undegraded":"2021-03-24T21:59:17.413826+0000","last_fullsized":"2021-03-24T21:59:17.413826+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:17.040623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:59:17.040623+0000","last_clean_scrub_stamp":"2021-03-24T21:59:17.040623+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"63","reported_epoch":"391","state":"active+undersized","last_fresh":"2021-03-24T21:59:21.213148+0000","last_change":"2021-03-24T21:58:31.198325+0000","last_active":"2021-03-24T21:59:21.213148+0000","last_peered":"2021-03-24T21:59:21.213148+0000","last_clean":"2021-03-24T21:51:50.731137+0000","last_became_active":"2021-03-24T21:58:31.198325+0000","last_became_peered":"2021-03-24T21:58:31.198325+0000","last_unstale":"2021-03-24T21:59:21.213148+0000","last_undegraded":"2021-03-24T21:59:21.213148+0000","last_fullsized":"2021-03-24T21:58:31.193484+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:26.887939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:58:26.887939+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"273'245","reported_seq":"1092","reported_epoch":"387","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:59:17.439459+0000","last_change":"2021-03-24T21:58:31.206732+0000","last_active":"2021-03-24T21:59:17.439459+0000","last_peered":"2021-03-24T21:59:17.439459+0000","last_clean":"2021-03-24T21:58:29.207340+0000","last_became_active":"2021-03-24T21:58:31.206732+0000","last_became_peered":"2021-03-24T21:58:31.206732+0000","last_unstale":"2021-03-24T21:59:17.439459+0000","last_undegraded":"2021-03-24T21:58:31.188200+0000","last_fullsized":"2021-03-24T21:58:31.188045+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":245,"ondisk_log_size":245,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":19,"stat_sum":{"num_bytes":8581786,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":70,"num_read_kb":67597,"num_write":52,"num_write_kb":18055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"89'27","reported_seq":"429","reported_epoch":"387","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:59:17.413231+0000","last_change":"2021-03-24T21:58:31.228394+0000","last_active":"2021-03-24T21:59:17.413231+0000","last_peered":"2021-03-24T21:59:17.413231+0000","last_clean":"2021-03-24T21:58:29.213813+0000","last_became_active":"2021-03-24T21:58:31.228394+0000","last_became_peered":"2021-03-24T21:58:31.228394+0000","last_unstale":"2021-03-24T21:59:17.413231+0000","last_undegraded":"2021-03-24T21:58:31.192365+0000","last_fullsized":"2021-03-24T21:58:31.192251+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'27","last_scrub_stamp":"2021-03-24T21:55:41.337130+0000","last_deep_scrub":"89'27","last_deep_scrub_stamp":"2021-03-24T21:55:41.337130+0000","last_clean_scrub_stamp":"2021-03-24T21:55:41.337130+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":934,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":11}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":"323","reported_epoch":"387","state":"active+undersized","last_fresh":"2021-03-24T21:59:17.422286+0000","last_change":"2021-03-24T21:58:31.207653+0000","last_active":"2021-03-24T21:59:17.422286+0000","last_peered":"2021-03-24T21:59:17.422286+0000","last_clean":"2021-03-24T21:58:29.205110+0000","last_became_active":"2021-03-24T21:58:31.207653+0000","last_became_peered":"2021-03-24T21:58:31.207653+0000","last_unstale":"2021-03-24T21:59:17.422286+0000","last_undegraded":"2021-03-24T21:59:17.422286+0000","last_fullsized":"2021-03-24T21:58:31.185073+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:26.417613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:57:26.417613+0000","last_clean_scrub_stamp":"2021-03-24T21:57:26.417613+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"235'4","reported_seq":"397","reported_epoch":"391","state":"active+clean","last_fresh":"2021-03-24T21:59:21.216774+0000","last_change":"2021-03-24T21:59:04.626032+0000","last_active":"2021-03-24T21:59:21.216774+0000","last_peered":"2021-03-24T21:59:21.216774+0000","last_clean":"2021-03-24T21:59:21.216774+0000","last_became_active":"2021-03-24T21:56:44.228196+0000","last_became_peered":"2021-03-24T21:56:44.228196+0000","last_unstale":"2021-03-24T21:59:21.216774+0000","last_undegraded":"2021-03-24T21:59:21.216774+0000","last_fullsized":"2021-03-24T21:59:21.216774+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"235'4","last_scrub_stamp":"2021-03-24T21:59:04.625954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:59:04.625954+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"100","reported_epoch":"390","state":"active+undersized","last_fresh":"2021-03-24T21:59:20.209389+0000","last_change":"2021-03-24T21:58:31.205748+0000","last_active":"2021-03-24T21:59:20.209389+0000","last_peered":"2021-03-24T21:59:20.209389+0000","last_clean":"2021-03-24T21:58:29.217060+0000","last_became_active":"2021-03-24T21:58:31.205748+0000","last_became_peered":"2021-03-24T21:58:31.205748+0000","last_unstale":"2021-03-24T21:59:20.209389+0000","last_undegraded":"2021-03-24T21:59:20.209389+0000","last_fullsized":"2021-03-24T21:58:31.191168+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:26.129924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:57:26.129924+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1d","version":"370'314","reported_seq":"1222","reported_epoch":"387","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:59:17.410893+0000","last_change":"2021-03-24T21:58:31.203265+0000","last_active":"2021-03-24T21:59:17.410893+0000","last_peered":"2021-03-24T21:59:17.410893+0000","last_clean":"2021-03-24T21:58:29.213086+0000","last_became_active":"2021-03-24T21:58:31.203265+0000","last_became_peered":"2021-03-24T21:58:31.203265+0000","last_unstale":"2021-03-24T21:59:17.410893+0000","last_undegraded":"2021-03-24T21:58:31.188863+0000","last_fullsized":"2021-03-24T21:58:31.188701+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":5,"last_scrub":"233'169","last_scrub_stamp":"2021-03-24T21:56:47.285648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:56:47.285648+0000","log_size":314,"ondisk_log_size":314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":19,"stat_sum":{"num_bytes":15220819,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":97,"num_read_kb":85972,"num_write":91,"num_write_kb":41497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":9406781,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":6}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1d","version":"69'16","reported_seq":"407","reported_epoch":"387","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:59:17.423520+0000","last_change":"2021-03-24T21:58:31.229186+0000","last_active":"2021-03-24T21:59:17.423520+0000","last_peered":"2021-03-24T21:59:17.423520+0000","last_clean":"2021-03-24T21:58:29.206116+0000","last_became_active":"2021-03-24T21:58:31.229186+0000","last_became_peered":"2021-03-24T21:58:31.229186+0000","last_unstale":"2021-03-24T21:59:17.423520+0000","last_undegraded":"2021-03-24T21:58:31.196334+0000","last_fullsized":"2021-03-24T21:58:31.196221+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T21:56:10.423860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:10.423860+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":37,"num_read_kb":20,"num_write":18,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":844,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":9}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":"367","reported_epoch":"391","state":"active+clean","last_fresh":"2021-03-24T21:59:21.213683+0000","last_change":"2021-03-24T21:59:12.679897+0000","last_active":"2021-03-24T21:59:21.213683+0000","last_peered":"2021-03-24T21:59:21.213683+0000","last_clean":"2021-03-24T21:59:21.213683+0000","last_became_active":"2021-03-24T21:54:38.143876+0000","last_became_peered":"2021-03-24T21:54:38.143876+0000","last_unstale":"2021-03-24T21:59:21.213683+0000","last_undegraded":"2021-03-24T21:59:21.213683+0000","last_fullsized":"2021-03-24T21:59:21.213683+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:12.679659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:52:54.212459+0000","last_clean_scrub_stamp":"2021-03-24T21:59:12.679659+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"349","reported_epoch":"391","state":"active+clean","last_fresh":"2021-03-24T21:59:21.214786+0000","last_change":"2021-03-24T21:57:19.767717+0000","last_active":"2021-03-24T21:59:21.214786+0000","last_peered":"2021-03-24T21:59:21.214786+0000","last_clean":"2021-03-24T21:59:21.214786+0000","last_became_active":"2021-03-24T21:51:49.410920+0000","last_became_peered":"2021-03-24T21:51:49.410920+0000","last_unstale":"2021-03-24T21:59:21.214786+0000","last_undegraded":"2021-03-24T21:59:21.214786+0000","last_fullsized":"2021-03-24T21:59:21.214786+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:19.767465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:57:19.767465+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"361","reported_epoch":"387","state":"active+undersized","last_fresh":"2021-03-24T21:59:17.438181+0000","last_change":"2021-03-24T21:58:31.196650+0000","last_active":"2021-03-24T21:59:17.438181+0000","last_peered":"2021-03-24T21:59:17.438181+0000","last_clean":"2021-03-24T21:58:29.208496+0000","last_became_active":"2021-03-24T21:58:31.196650+0000","last_became_peered":"2021-03-24T21:58:31.196650+0000","last_unstale":"2021-03-24T21:59:17.438181+0000","last_undegraded":"2021-03-24T21:59:17.438181+0000","last_fullsized":"2021-03-24T21:58:31.188039+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:33.440618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:33.440618+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1e","version":"265'45","reported_seq":"256","reported_epoch":"387","state":"active+undersized","last_fresh":"2021-03-24T21:59:17.422865+0000","last_change":"2021-03-24T21:58:31.206881+0000","last_active":"2021-03-24T21:59:17.422865+0000","last_peered":"2021-03-24T21:59:17.422865+0000","last_clean":"2021-03-24T21:58:29.205698+0000","last_became_active":"2021-03-24T21:58:31.206881+0000","last_became_peered":"2021-03-24T21:58:31.206881+0000","last_unstale":"2021-03-24T21:59:17.422865+0000","last_undegraded":"2021-03-24T21:59:17.422865+0000","last_fullsized":"2021-03-24T21:58:31.185389+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":5,"last_scrub":"199'30","last_scrub_stamp":"2021-03-24T21:56:10.105084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:56:10.105084+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":19,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":18305,"num_write":13,"num_write_kb":3060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"69'15","reported_seq":"233","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.439986+0000","last_change":"2021-03-24T21:59:14.320000+0000","last_active":"2021-03-24T21:59:17.439986+0000","last_peered":"2021-03-24T21:59:17.439986+0000","last_clean":"2021-03-24T21:59:17.439986+0000","last_became_active":"2021-03-24T21:56:35.142209+0000","last_became_peered":"2021-03-24T21:56:35.142209+0000","last_unstale":"2021-03-24T21:59:17.439986+0000","last_undegraded":"2021-03-24T21:59:17.439986+0000","last_fullsized":"2021-03-24T21:59:17.439986+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:59:14.319800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:59:14.319800+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":"348","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.409319+0000","last_change":"2021-03-24T21:56:42.254648+0000","last_active":"2021-03-24T21:59:17.409319+0000","last_peered":"2021-03-24T21:59:17.409319+0000","last_clean":"2021-03-24T21:59:17.409319+0000","last_became_active":"2021-03-24T21:51:45.381533+0000","last_became_peered":"2021-03-24T21:51:45.381533+0000","last_unstale":"2021-03-24T21:59:17.409319+0000","last_undegraded":"2021-03-24T21:59:17.409319+0000","last_fullsized":"2021-03-24T21:59:17.409319+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:42.254536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:42.254536+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"311","reported_epoch":"391","state":"active+undersized","last_fresh":"2021-03-24T21:59:21.213746+0000","last_change":"2021-03-24T21:58:31.208452+0000","last_active":"2021-03-24T21:59:21.213746+0000","last_peered":"2021-03-24T21:59:21.213746+0000","last_clean":"2021-03-24T21:58:29.217323+0000","last_became_active":"2021-03-24T21:58:31.208452+0000","last_became_peered":"2021-03-24T21:58:31.208452+0000","last_unstale":"2021-03-24T21:59:21.213746+0000","last_undegraded":"2021-03-24T21:59:21.213746+0000","last_fullsized":"2021-03-24T21:58:31.193967+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:26.642883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:57:26.642883+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"357","reported_epoch":"391","state":"active+undersized","last_fresh":"2021-03-24T21:59:21.214784+0000","last_change":"2021-03-24T21:58:31.223458+0000","last_active":"2021-03-24T21:59:21.214784+0000","last_peered":"2021-03-24T21:59:21.214784+0000","last_clean":"2021-03-24T21:58:29.217708+0000","last_became_active":"2021-03-24T21:58:31.223458+0000","last_became_peered":"2021-03-24T21:58:31.223458+0000","last_unstale":"2021-03-24T21:59:21.214784+0000","last_undegraded":"2021-03-24T21:59:21.214784+0000","last_fullsized":"2021-03-24T21:58:31.189856+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:38.075714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:38.075714+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1f","version":"272'99","reported_seq":"260","reported_epoch":"391","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:59:21.213199+0000","last_change":"2021-03-24T21:58:31.223352+0000","last_active":"2021-03-24T21:59:21.213199+0000","last_peered":"2021-03-24T21:59:21.213199+0000","last_clean":"2021-03-24T21:58:29.217518+0000","last_became_active":"2021-03-24T21:58:31.223352+0000","last_became_peered":"2021-03-24T21:58:31.223352+0000","last_unstale":"2021-03-24T21:59:21.213199+0000","last_undegraded":"2021-03-24T21:58:31.189248+0000","last_fullsized":"2021-03-24T21:58:31.188821+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":5,"last_scrub":"218'73","last_scrub_stamp":"2021-03-24T21:56:48.267711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:56:48.267711+0000","log_size":99,"ondisk_log_size":99,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":21,"stat_sum":{"num_bytes":3571504,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":43,"num_read_kb":44484,"num_write":22,"num_write_kb":9163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":942483,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"91'9","reported_seq":"400","reported_epoch":"391","state":"active+clean","last_fresh":"2021-03-24T21:59:21.213228+0000","last_change":"2021-03-24T21:58:55.660850+0000","last_active":"2021-03-24T21:59:21.213228+0000","last_peered":"2021-03-24T21:59:21.213228+0000","last_clean":"2021-03-24T21:59:21.213228+0000","last_became_active":"2021-03-24T21:57:02.184317+0000","last_became_peered":"2021-03-24T21:57:02.184317+0000","last_unstale":"2021-03-24T21:59:21.213228+0000","last_undegraded":"2021-03-24T21:59:21.213228+0000","last_fullsized":"2021-03-24T21:59:21.213228+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2021-03-24T21:58:55.660737+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2021-03-24T21:53:33.290000+0000","last_clean_scrub_stamp":"2021-03-24T21:58:55.660737+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":8,"num_write":7,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":"343","reported_epoch":"391","state":"active+undersized","last_fresh":"2021-03-24T21:59:21.213782+0000","last_change":"2021-03-24T21:58:31.202563+0000","last_active":"2021-03-24T21:59:21.213782+0000","last_peered":"2021-03-24T21:59:21.213782+0000","last_clean":"2021-03-24T21:58:29.214992+0000","last_became_active":"2021-03-24T21:58:31.202563+0000","last_became_peered":"2021-03-24T21:58:31.202563+0000","last_unstale":"2021-03-24T21:59:21.213782+0000","last_undegraded":"2021-03-24T21:59:21.213782+0000","last_fullsized":"2021-03-24T21:58:31.192438+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:47.920949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:47.920949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"358","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.415929+0000","last_change":"2021-03-24T21:59:08.942411+0000","last_active":"2021-03-24T21:59:17.415929+0000","last_peered":"2021-03-24T21:59:17.415929+0000","last_clean":"2021-03-24T21:59:17.415929+0000","last_became_active":"2021-03-24T21:54:38.137706+0000","last_became_peered":"2021-03-24T21:54:38.137706+0000","last_unstale":"2021-03-24T21:59:17.415929+0000","last_undegraded":"2021-03-24T21:59:17.415929+0000","last_fullsized":"2021-03-24T21:59:17.415929+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:08.942247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:56:51.977306+0000","last_clean_scrub_stamp":"2021-03-24T21:59:08.942247+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"112","reported_epoch":"387","state":"active+undersized","last_fresh":"2021-03-24T21:59:17.422007+0000","last_change":"2021-03-24T21:58:31.196725+0000","last_active":"2021-03-24T21:59:17.422007+0000","last_peered":"2021-03-24T21:59:17.422007+0000","last_clean":"2021-03-24T21:54:36.085927+0000","last_became_active":"2021-03-24T21:58:31.196725+0000","last_became_peered":"2021-03-24T21:58:31.196725+0000","last_unstale":"2021-03-24T21:59:17.422007+0000","last_undegraded":"2021-03-24T21:59:17.422007+0000","last_fullsized":"2021-03-24T21:58:31.184385+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:16.076982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:57:16.076982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.18","version":"342'94","reported_seq":"537","reported_epoch":"387","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:59:17.406845+0000","last_change":"2021-03-24T21:58:32.440753+0000","last_active":"2021-03-24T21:59:17.406845+0000","last_peered":"2021-03-24T21:59:17.406845+0000","last_clean":"2021-03-24T21:58:29.211604+0000","last_became_active":"2021-03-24T21:58:31.208262+0000","last_became_peered":"2021-03-24T21:58:31.208262+0000","last_unstale":"2021-03-24T21:59:17.406845+0000","last_undegraded":"2021-03-24T21:58:32.436918+0000","last_fullsized":"2021-03-24T21:58:31.193361+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":5,"last_scrub":"249'90","last_scrub_stamp":"2021-03-24T21:57:56.848784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:57:56.848784+0000","log_size":94,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":19,"stat_sum":{"num_bytes":3462308,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":17,"num_read_kb":15393,"num_write":5,"num_write_kb":1725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"69'13","reported_seq":"375","reported_epoch":"391","state":"active+clean","last_fresh":"2021-03-24T21:59:21.212615+0000","last_change":"2021-03-24T21:58:26.698101+0000","last_active":"2021-03-24T21:59:21.212615+0000","last_peered":"2021-03-24T21:59:21.212615+0000","last_clean":"2021-03-24T21:59:21.212615+0000","last_became_active":"2021-03-24T21:51:47.407488+0000","last_became_peered":"2021-03-24T21:51:47.407488+0000","last_unstale":"2021-03-24T21:59:21.212615+0000","last_undegraded":"2021-03-24T21:59:21.212615+0000","last_fullsized":"2021-03-24T21:59:21.212615+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2021-03-24T21:58:26.697918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:58:26.697918+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":"388","reported_epoch":"390","state":"active+undersized","last_fresh":"2021-03-24T21:59:20.213145+0000","last_change":"2021-03-24T21:58:31.205041+0000","last_active":"2021-03-24T21:59:20.213145+0000","last_peered":"2021-03-24T21:59:20.213145+0000","last_clean":"2021-03-24T21:58:29.217058+0000","last_became_active":"2021-03-24T21:58:31.205041+0000","last_became_peered":"2021-03-24T21:58:31.205041+0000","last_unstale":"2021-03-24T21:59:20.213145+0000","last_undegraded":"2021-03-24T21:59:20.213145+0000","last_fullsized":"2021-03-24T21:58:31.191384+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:24.418893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:58:24.418893+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"56","reported_epoch":"387","state":"active+undersized","last_fresh":"2021-03-24T21:59:17.408852+0000","last_change":"2021-03-24T21:58:31.208343+0000","last_active":"2021-03-24T21:59:17.408852+0000","last_peered":"2021-03-24T21:59:17.408852+0000","last_clean":"2021-03-24T21:51:48.384232+0000","last_became_active":"2021-03-24T21:58:31.208343+0000","last_became_peered":"2021-03-24T21:58:31.208343+0000","last_unstale":"2021-03-24T21:59:17.408852+0000","last_undegraded":"2021-03-24T21:59:17.408852+0000","last_fullsized":"2021-03-24T21:58:31.191979+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:38.242096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:38.242096+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"351","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.407642+0000","last_change":"2021-03-24T21:57:19.271463+0000","last_active":"2021-03-24T21:59:17.407642+0000","last_peered":"2021-03-24T21:59:17.407642+0000","last_clean":"2021-03-24T21:59:17.407642+0000","last_became_active":"2021-03-24T21:54:38.127696+0000","last_became_peered":"2021-03-24T21:54:38.127696+0000","last_unstale":"2021-03-24T21:59:17.407642+0000","last_undegraded":"2021-03-24T21:59:17.407642+0000","last_fullsized":"2021-03-24T21:59:17.407642+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:19.271351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:57:19.271351+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.19","version":"388'383","reported_seq":"1292","reported_epoch":"388","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:59:18.451765+0000","last_change":"2021-03-24T21:58:31.209291+0000","last_active":"2021-03-24T21:59:18.451765+0000","last_peered":"2021-03-24T21:59:18.451765+0000","last_clean":"2021-03-24T21:58:29.200505+0000","last_became_active":"2021-03-24T21:58:31.209291+0000","last_became_peered":"2021-03-24T21:58:31.209291+0000","last_unstale":"2021-03-24T21:59:18.451765+0000","last_undegraded":"2021-03-24T21:58:31.197311+0000","last_fullsized":"2021-03-24T21:58:31.197197+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":5,"last_scrub":"265'260","last_scrub_stamp":"2021-03-24T21:57:22.158029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:57:22.158029+0000","log_size":383,"ondisk_log_size":383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":19,"stat_sum":{"num_bytes":19565351,"num_objects":10,"num_object_clones":7,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":47,"num_read_kb":38877,"num_write":63,"num_write_kb":24225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":10}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"69'20","reported_seq":"381","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.422772+0000","last_change":"2021-03-24T21:58:31.309185+0000","last_active":"2021-03-24T21:59:17.422772+0000","last_peered":"2021-03-24T21:59:17.422772+0000","last_clean":"2021-03-24T21:59:17.422772+0000","last_became_active":"2021-03-24T21:51:47.405776+0000","last_became_peered":"2021-03-24T21:51:47.405776+0000","last_unstale":"2021-03-24T21:59:17.422772+0000","last_undegraded":"2021-03-24T21:59:17.422772+0000","last_fullsized":"2021-03-24T21:59:17.422772+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'20","last_scrub_stamp":"2021-03-24T21:58:31.309079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:58:31.309079+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":22,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":"366","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.439239+0000","last_change":"2021-03-24T21:59:05.405888+0000","last_active":"2021-03-24T21:59:17.439239+0000","last_peered":"2021-03-24T21:59:17.439239+0000","last_clean":"2021-03-24T21:59:17.439239+0000","last_became_active":"2021-03-24T21:54:38.130163+0000","last_became_peered":"2021-03-24T21:54:38.130163+0000","last_unstale":"2021-03-24T21:59:17.439239+0000","last_undegraded":"2021-03-24T21:59:17.439239+0000","last_fullsized":"2021-03-24T21:59:17.439239+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:05.405785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:59:05.405785+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"367","reported_epoch":"391","state":"active+clean","last_fresh":"2021-03-24T21:59:21.211184+0000","last_change":"2021-03-24T21:59:02.668868+0000","last_active":"2021-03-24T21:59:21.211184+0000","last_peered":"2021-03-24T21:59:21.211184+0000","last_clean":"2021-03-24T21:59:21.211184+0000","last_became_active":"2021-03-24T21:54:38.120575+0000","last_became_peered":"2021-03-24T21:54:38.120575+0000","last_unstale":"2021-03-24T21:59:21.211184+0000","last_undegraded":"2021-03-24T21:59:21.211184+0000","last_fullsized":"2021-03-24T21:59:21.211184+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:02.668717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:59:02.668717+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"360","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.453707+0000","last_change":"2021-03-24T21:58:29.298430+0000","last_active":"2021-03-24T21:59:17.453707+0000","last_peered":"2021-03-24T21:59:17.453707+0000","last_clean":"2021-03-24T21:59:17.453707+0000","last_became_active":"2021-03-24T21:54:38.130293+0000","last_became_peered":"2021-03-24T21:54:38.130293+0000","last_unstale":"2021-03-24T21:59:17.453707+0000","last_undegraded":"2021-03-24T21:59:17.453707+0000","last_fullsized":"2021-03-24T21:59:17.453707+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:29.298229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:58:29.298229+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"353'159","reported_seq":"958","reported_epoch":"391","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:59:21.211228+0000","last_change":"2021-03-24T21:58:31.203294+0000","last_active":"2021-03-24T21:59:21.211228+0000","last_peered":"2021-03-24T21:59:21.211228+0000","last_clean":"2021-03-24T21:58:29.213731+0000","last_became_active":"2021-03-24T21:58:31.203294+0000","last_became_peered":"2021-03-24T21:58:31.203294+0000","last_unstale":"2021-03-24T21:59:21.211228+0000","last_undegraded":"2021-03-24T21:58:31.193672+0000","last_fullsized":"2021-03-24T21:58:31.193468+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":159,"ondisk_log_size":159,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":21,"stat_sum":{"num_bytes":5047878,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":30,"num_read_kb":22371,"num_write":50,"num_write_kb":16462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":3266508,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":3}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"69'15","reported_seq":"441","reported_epoch":"391","state":"active+clean","last_fresh":"2021-03-24T21:59:21.210866+0000","last_change":"2021-03-24T21:58:58.719478+0000","last_active":"2021-03-24T21:59:21.210866+0000","last_peered":"2021-03-24T21:59:21.210866+0000","last_clean":"2021-03-24T21:59:21.210866+0000","last_became_active":"2021-03-24T21:54:38.127426+0000","last_became_peered":"2021-03-24T21:54:38.127426+0000","last_unstale":"2021-03-24T21:59:21.210866+0000","last_undegraded":"2021-03-24T21:59:21.210866+0000","last_fullsized":"2021-03-24T21:59:21.210866+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:58:58.719296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:58:58.719296+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2076,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":"359","reported_epoch":"387","state":"active+undersized","last_fresh":"2021-03-24T21:59:17.417588+0000","last_change":"2021-03-24T21:58:31.216651+0000","last_active":"2021-03-24T21:59:17.417588+0000","last_peered":"2021-03-24T21:59:17.417588+0000","last_clean":"2021-03-24T21:58:29.786750+0000","last_became_active":"2021-03-24T21:58:31.216651+0000","last_became_peered":"2021-03-24T21:58:31.216651+0000","last_unstale":"2021-03-24T21:59:17.417588+0000","last_undegraded":"2021-03-24T21:59:17.417588+0000","last_fullsized":"2021-03-24T21:58:31.191007+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:38.294685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:38.294685+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"277","reported_epoch":"387","state":"active+undersized","last_fresh":"2021-03-24T21:59:17.415265+0000","last_change":"2021-03-24T21:58:31.208824+0000","last_active":"2021-03-24T21:59:17.415265+0000","last_peered":"2021-03-24T21:59:17.415265+0000","last_clean":"2021-03-24T21:58:29.201408+0000","last_became_active":"2021-03-24T21:58:31.208824+0000","last_became_peered":"2021-03-24T21:58:31.208824+0000","last_unstale":"2021-03-24T21:59:17.415265+0000","last_undegraded":"2021-03-24T21:59:17.415265+0000","last_fullsized":"2021-03-24T21:58:31.195827+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:27.920682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:14.710709+0000","last_clean_scrub_stamp":"2021-03-24T21:58:27.920682+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"362","reported_epoch":"391","state":"active+undersized","last_fresh":"2021-03-24T21:59:21.212009+0000","last_change":"2021-03-24T21:58:31.202629+0000","last_active":"2021-03-24T21:59:21.212009+0000","last_peered":"2021-03-24T21:59:21.212009+0000","last_clean":"2021-03-24T21:58:29.214118+0000","last_became_active":"2021-03-24T21:58:31.202629+0000","last_became_peered":"2021-03-24T21:58:31.202629+0000","last_unstale":"2021-03-24T21:59:21.212009+0000","last_undegraded":"2021-03-24T21:59:21.212009+0000","last_fullsized":"2021-03-24T21:58:31.194110+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:15.687165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:57:15.687165+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"387'502","reported_seq":"1660","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.424101+0000","last_change":"2021-03-24T21:59:17.424101+0000","last_active":"2021-03-24T21:59:17.424101+0000","last_peered":"2021-03-24T21:59:17.424101+0000","last_clean":"2021-03-24T21:59:17.424101+0000","last_became_active":"2021-03-24T21:57:33.176110+0000","last_became_peered":"2021-03-24T21:57:33.176110+0000","last_unstale":"2021-03-24T21:59:17.424101+0000","last_undegraded":"2021-03-24T21:59:17.424101+0000","last_fullsized":"2021-03-24T21:59:17.424101+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":5,"last_scrub":"191'150","last_scrub_stamp":"2021-03-24T21:56:01.308050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:56:01.308050+0000","log_size":502,"ondisk_log_size":502,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20192649,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":262,"num_read_kb":293477,"num_write":134,"num_write_kb":48961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":28585116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"d6","length":"1"},{"start":"df","length":"1"},{"start":"e1","length":"1"},{"start":"e8","length":"1"},{"start":"ea","length":"4"},{"start":"ef","length":"1"},{"start":"f2","length":"4"},{"start":"fb","length":"3"},{"start":"109","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"}]},{"pgid":"3.24","version":"0'0","reported_seq":"14","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.441437+0000","last_change":"2021-03-24T21:59:14.369914+0000","last_active":"2021-03-24T21:59:17.441437+0000","last_peered":"2021-03-24T21:59:17.441437+0000","last_clean":"2021-03-24T21:59:17.441437+0000","last_became_active":"2021-03-24T21:59:14.369606+0000","last_became_peered":"2021-03-24T21:59:14.369606+0000","last_unstale":"2021-03-24T21:59:17.441437+0000","last_undegraded":"2021-03-24T21:59:17.441437+0000","last_fullsized":"2021-03-24T21:59:17.441437+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:38.144956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:38.144956+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"175'22","reported_seq":"602","reported_epoch":"391","state":"active+clean","last_fresh":"2021-03-24T21:59:21.215475+0000","last_change":"2021-03-24T21:59:21.216150+0000","last_active":"2021-03-24T21:59:21.215475+0000","last_peered":"2021-03-24T21:59:21.215475+0000","last_clean":"2021-03-24T21:59:21.215475+0000","last_became_active":"2021-03-24T21:57:28.171436+0000","last_became_peered":"2021-03-24T21:57:28.171436+0000","last_unstale":"2021-03-24T21:59:21.215475+0000","last_undegraded":"2021-03-24T21:59:21.215475+0000","last_fullsized":"2021-03-24T21:59:21.215475+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":6,"last_scrub":"175'22","last_scrub_stamp":"2021-03-24T21:59:03.758935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:59:03.758935+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"d6","length":"1"},{"start":"df","length":"1"},{"start":"e1","length":"1"},{"start":"e8","length":"1"},{"start":"ea","length":"4"},{"start":"ef","length":"1"},{"start":"f1","length":"5"},{"start":"fb","length":"3"},{"start":"109","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"}]},{"pgid":"3.25","version":"0'0","reported_seq":"70","reported_epoch":"387","state":"active+clean","last_fresh":"2021-03-24T21:59:17.422752+0000","last_change":"2021-03-24T21:59:14.371982+0000","last_active":"2021-03-24T21:59:17.422752+0000","last_peered":"2021-03-24T21:59:17.422752+0000","last_clean":"2021-03-24T21:59:17.422752+0000","last_became_active":"2021-03-24T21:59:14.371649+0000","last_became_peered":"2021-03-24T21:59:14.371649+0000","last_unstale":"2021-03-24T21:59:17.422752+0000","last_undegraded":"2021-03-24T21:59:17.422752+0000","last_fullsized":"2021-03-24T21:59:17.422752+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:53.013129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:53.013129+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.21","version":"380'300","reported_seq":"1242","reported_epoch":"387","state":"active+undersized+degraded","last_fresh":"2021-03-24T21:59:17.422542+0000","last_change":"2021-03-24T21:58:31.203128+0000","last_active":"2021-03-24T21:59:17.422542+0000","last_peered":"2021-03-24T21:59:17.422542+0000","last_clean":"2021-03-24T21:58:29.205714+0000","last_became_active":"2021-03-24T21:58:31.203128+0000","last_became_peered":"2021-03-24T21:58:31.203128+0000","last_unstale":"2021-03-24T21:59:17.422542+0000","last_undegraded":"2021-03-24T21:58:31.186929+0000","last_fullsized":"2021-03-24T21:58:31.185968+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":300,"ondisk_log_size":300,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":19,"stat_sum":{"num_bytes":7526109,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":117,"num_read_kb":127979,"num_write":57,"num_write_kb":20999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":10874449,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":36,"stat_sum":{"num_bytes":350714502,"num_objects":158,"num_object_clones":107,"num_object_copies":474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":99,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":7,"num_read":1715,"num_read_kb":1795451,"num_write":1189,"num_write_kb":469142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":94530331,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":712544256,"data_stored":921790576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7608,"ondisk_log_size":7608,"up":87,"acting":87,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":451,"num_read_kb":446,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":8,"acting":8,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":3,"num_pg":42,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1667,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":6431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":106,"acting":110,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":159,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1060,"num_read_kb":565,"num_write":542,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":281,"num_bytes_recovered":23006,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2027520,"data_stored":72734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":558,"ondisk_log_size":558,"up":80,"acting":80,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":33,"ondisk_log_size":33,"up":78,"acting":78,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":78,"acting":78,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201375,"num_pgs":92,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":193660,"kb_used_data":167860,"kb_used_omap":0,"kb_used_meta":25792,"kb_avail":93551492,"statfs":{"total":95995035648,"available":95796727808,"internally_reserved":0,"allocated":171888640,"data_stored":208351150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26411008},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:58:58 2021","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.97499999999999998,"15min":0.92800000000000005},"min":{"1min":0.38600000000000001,"5min":0.312,"15min":0.312},"max":{"1min":1.0329999999999999,"5min":12.222,"15min":12.222},"last":0.51800000000000002},{"interface":"front","average":{"1min":1.575,"5min":1.1459999999999999,"15min":1.0309999999999999},"min":{"1min":0.54000000000000004,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":7.1609999999999996,"5min":11.506,"15min":11.506},"last":0.61299999999999999}]},{"osd":1,"last update":"Wed Mar 24 21:58:58 2021","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.97299999999999998,"15min":0.88500000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.216,"5min":5.7750000000000004,"15min":5.7750000000000004},"last":0.44},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.98299999999999998,"15min":0.97699999999999998},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.327,"5min":6.0780000000000003,"15min":6.0780000000000003},"last":0.53600000000000003}]},{"osd":2,"last update":"Wed Mar 24 21:58:58 2021","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":1.117,"15min":1.044},"min":{"1min":0.39000000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.9510000000000001,"5min":12.135999999999999,"15min":12.135999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":1.26,"5min":1.1719999999999999,"15min":1.0649999999999999},"min":{"1min":0.50700000000000001,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":7.0460000000000003,"5min":12.178000000000001,"15min":12.178000000000001},"last":0.58899999999999997}]},{"osd":3,"last update":"Wed Mar 24 21:58:58 2021","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":1.2290000000000001,"15min":1.105},"min":{"1min":0.61899999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":7.2800000000000002,"5min":12.529,"15min":12.529},"last":0.82999999999999996},{"interface":"front","average":{"1min":1.0780000000000001,"5min":1.194,"15min":1.054},"min":{"1min":0.59599999999999997,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":3.012,"5min":12.666,"15min":12.666},"last":0.64500000000000002}]},{"osd":4,"last update":"Wed Mar 24 21:58:58 2021","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":1.052,"15min":0.97099999999999997},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.635,"5min":6.2089999999999996,"15min":6.2089999999999996},"last":0.89000000000000001},{"interface":"front","average":{"1min":1.002,"5min":1.155,"15min":1.1020000000000001},"min":{"1min":0.55700000000000005,"5min":0.435,"15min":0.435},"max":{"1min":1.3839999999999999,"5min":12.244,"15min":12.244},"last":0.67000000000000004}]},{"osd":5,"last update":"Wed Mar 24 21:57:54 2021","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.153,"15min":1.0269999999999999},"min":{"1min":0.64100000000000001,"5min":0.435,"15min":0.435},"max":{"1min":1.4330000000000001,"5min":5.8840000000000003,"15min":5.8840000000000003},"last":1.2430000000000001},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.1910000000000001,"15min":1.1539999999999999},"min":{"1min":0.628,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.629,"5min":6.3300000000000001,"15min":6.3300000000000001},"last":1.1799999999999999}]},{"osd":6,"last update":"Wed Mar 24 21:58:58 2021","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.1919999999999999,"15min":1.145},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.448,"5min":6.1159999999999997,"15min":6.1159999999999997},"last":0.65500000000000003},{"interface":"front","average":{"1min":1.0409999999999999,"5min":1.2629999999999999,"15min":1.206},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.587,"5min":12.287000000000001,"15min":12.287000000000001},"last":0.629}]}]},{"osd":6,"up_from":0,"seq":201863463012,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":0,"seq":55834574969,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738492,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":136020,"kb_used_data":111564,"kb_used_omap":0,"kb_used_meta":24448,"kb_avail":93609132,"statfs":{"total":95995035648,"available":95855751168,"internally_reserved":0,"allocated":114241536,"data_stored":147047996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":25034752},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Mar 24 21:58:47 2021","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.69099999999999995,"15min":0.629},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.99099999999999999,"5min":4.8040000000000003,"15min":4.8040000000000003},"last":0.748},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.83899999999999997,"15min":0.69699999999999995},"min":{"1min":0.47199999999999998,"5min":0.374,"15min":0.374},"max":{"1min":2.5179999999999998,"5min":8.8989999999999991,"15min":8.8989999999999991},"last":0.93200000000000005}]},{"osd":2,"last update":"Wed Mar 24 21:59:05 2021","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.81799999999999995,"15min":0.71799999999999997},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":4.4939999999999998,"5min":5.8810000000000002,"15min":5.8810000000000002},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.84599999999999997,"15min":0.75700000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":2.5499999999999998,"5min":4.766,"15min":4.766},"last":0.66500000000000004}]},{"osd":3,"last update":"Wed Mar 24 21:58:23 2021","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":0.91400000000000003,"15min":0.83899999999999997},"min":{"1min":0.49299999999999999,"5min":0.434,"15min":0.376},"max":{"1min":2.79,"5min":5.9790000000000001,"15min":5.9790000000000001},"last":0.79800000000000004},{"interface":"front","average":{"1min":1.036,"5min":0.97699999999999998,"15min":0.94699999999999995},"min":{"1min":0.60099999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":2.3180000000000001,"5min":6.4580000000000002,"15min":6.4580000000000002},"last":0.873}]},{"osd":4,"last update":"Wed Mar 24 21:58:47 2021","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.085,"15min":1.091},"min":{"1min":0.55300000000000005,"5min":0.436,"15min":0.436},"max":{"1min":2.496,"5min":5.0330000000000004,"15min":5.0330000000000004},"last":0.81399999999999995},{"interface":"front","average":{"1min":1.073,"5min":1.1160000000000001,"15min":1.05},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":2.2629999999999999,"5min":6.1829999999999998,"15min":6.1829999999999998},"last":0.84099999999999997}]},{"osd":5,"last update":"Wed Mar 24 21:58:05 2021","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.1040000000000001,"15min":1.0740000000000001},"min":{"1min":0.64400000000000002,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.603,"5min":5.9390000000000001,"15min":5.9390000000000001},"last":0.88900000000000001},{"interface":"front","average":{"1min":1.103,"5min":1.167,"15min":1.1739999999999999},"min":{"1min":0.75900000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.645,"5min":5.0590000000000002,"15min":5.0590000000000002},"last":1.0189999999999999}]},{"osd":6,"last update":"Wed Mar 24 21:58:23 2021","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.173,"15min":1.1020000000000001},"min":{"1min":0.755,"5min":0.755,"15min":0.72699999999999998},"max":{"1min":1.6839999999999999,"5min":5.9930000000000003,"15min":5.9930000000000003},"last":0.98099999999999998},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.2110000000000001,"15min":1.218},"min":{"1min":0.78000000000000003,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":2.21,"5min":4.9960000000000004,"15min":4.9960000000000004},"last":0.94899999999999995}]},{"osd":7,"last update":"Wed Mar 24 21:59:00 2021","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.218,"15min":1.1870000000000001},"min":{"1min":0.61099999999999999,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":2.3559999999999999,"5min":6.1319999999999997,"15min":6.1319999999999997},"last":1.04},{"interface":"front","average":{"1min":1.276,"5min":1.2290000000000001,"15min":1.1619999999999999},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":4.4500000000000002,"5min":6.0529999999999999,"15min":6.0529999999999999},"last":1.081}]}]},{"osd":2,"up_from":18,"seq":77309411446,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":175652,"kb_used_data":155036,"kb_used_omap":0,"kb_used_meta":20608,"kb_avail":93569500,"statfs":{"total":95995035648,"available":95815168000,"internally_reserved":0,"allocated":158756864,"data_stored":214093346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21102592},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:59:04 2021","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.68500000000000005,"15min":0.65200000000000002},"min":{"1min":0.34399999999999997,"5min":0.307,"15min":0.307},"max":{"1min":1.655,"5min":1.655,"15min":1.655},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.747,"15min":0.71099999999999997},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2.9340000000000002,"5min":2.9340000000000002,"15min":2.9340000000000002},"last":0.61699999999999999}]},{"osd":1,"last update":"Wed Mar 24 21:59:04 2021","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.74099999999999999,"15min":0.71999999999999997},"min":{"1min":0.34200000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.7090000000000001,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76400000000000001,"15min":0.753},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.827,"5min":1.827,"15min":1.827},"last":0.86199999999999999}]},{"osd":3,"last update":"Wed Mar 24 21:58:22 2021","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.86699999999999999,"15min":0.84099999999999997},"min":{"1min":0.36799999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.9610000000000001,"5min":1.9610000000000001,"15min":1.9890000000000001},"last":1.143},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.89600000000000002,"15min":0.89900000000000002},"min":{"1min":0.38300000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.8859999999999999,"5min":1.8859999999999999,"15min":2.0950000000000002},"last":1.5169999999999999}]},{"osd":4,"last update":"Wed Mar 24 21:58:48 2021","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":1.0600000000000001,"15min":1.0449999999999999},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":2.5390000000000001,"5min":2.5390000000000001,"15min":3.3380000000000001},"last":1.2829999999999999},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.0589999999999999,"15min":1.054},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":2.3119999999999998,"5min":2.3119999999999998,"15min":2.3119999999999998},"last":1.3200000000000001}]},{"osd":5,"last update":"Wed Mar 24 21:58:04 2021","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":1.014,"15min":1.0269999999999999},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.163,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":1.4419999999999999},{"interface":"front","average":{"1min":1.048,"5min":1.125,"15min":1.147},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.4390000000000001,"5min":2.5859999999999999,"15min":3.1509999999999998},"last":1.552}]},{"osd":6,"last update":"Wed Mar 24 21:58:22 2021","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.1279999999999999,"15min":1.2170000000000001},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":2.4529999999999998,"5min":2.4529999999999998,"15min":2.4529999999999998},"last":1.425},{"interface":"front","average":{"1min":1.161,"5min":1.115,"15min":1.1479999999999999},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":2.6379999999999999,"5min":2.6379999999999999,"15min":2.6379999999999999},"last":1.1879999999999999}]},{"osd":7,"last update":"Wed Mar 24 21:58:58 2021","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":1.1990000000000001,"15min":1.2170000000000001},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":2.5659999999999998,"5min":4.4880000000000004,"15min":4.4880000000000004},"last":1.05},{"interface":"front","average":{"1min":1.1990000000000001,"5min":1.2070000000000001,"15min":1.2350000000000001},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":2.2909999999999999,"5min":2.2909999999999999,"15min":3.5419999999999998},"last":1.3620000000000001}]}]},{"osd":3,"up_from":26,"seq":111669149809,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":163932,"kb_used_data":138260,"kb_used_omap":0,"kb_used_meta":25664,"kb_avail":93581220,"statfs":{"total":95995035648,"available":95827169280,"internally_reserved":0,"allocated":141578240,"data_stored":179818896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26279936},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:58:29 2021","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.871,"15min":0.71299999999999997},"min":{"1min":0.38800000000000001,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.93799999999999994,"5min":23.846,"15min":23.846},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.92500000000000004,"15min":0.79100000000000004},"min":{"1min":0.38100000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.345,"5min":23.734999999999999,"15min":23.734999999999999},"last":0.58199999999999996}]},{"osd":1,"last update":"Wed Mar 24 21:58:29 2021","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.85299999999999998,"15min":0.77700000000000002},"min":{"1min":0.38900000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.88700000000000001,"5min":14.324,"15min":14.324},"last":0.378},{"interface":"front","average":{"1min":0.754,"5min":0.85499999999999998,"15min":0.78800000000000003},"min":{"1min":0.40100000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.109,"5min":12.631,"15min":12.631},"last":0.45500000000000002}]},{"osd":2,"last update":"Wed Mar 24 21:58:29 2021","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.90300000000000002,"15min":0.82699999999999996},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.446,"5min":12.510999999999999,"15min":12.510999999999999},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.92900000000000005,"15min":0.82099999999999995},"min":{"1min":0.46500000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.099,"5min":11.317,"15min":11.317},"last":0.56499999999999995}]},{"osd":4,"last update":"Wed Mar 24 21:58:50 2021","interfaces":[{"interface":"back","average":{"1min":1.137,"5min":1.05,"15min":0.98599999999999999},"min":{"1min":0.70299999999999996,"5min":0.443,"15min":0.443},"max":{"1min":2.601,"5min":2.601,"15min":2.601},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.98399999999999999,"5min":1.1439999999999999,"15min":1.014},"min":{"1min":0.66300000000000003,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.6299999999999999,"5min":14.291,"15min":14.291},"last":0.32900000000000001}]},{"osd":5,"last update":"Wed Mar 24 21:58:02 2021","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.1859999999999999,"15min":1.1040000000000001},"min":{"1min":0.73799999999999999,"5min":0.62,"15min":0.54100000000000004},"max":{"1min":1.5840000000000001,"5min":12.723000000000001,"15min":12.723000000000001},"last":1.0449999999999999},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.099,"15min":1.089},"min":{"1min":0.63900000000000001,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.452,"5min":2.375,"15min":2.375},"last":1.4850000000000001}]},{"osd":6,"last update":"Wed Mar 24 21:58:29 2021","interfaces":[{"interface":"back","average":{"1min":1,"5min":1.2170000000000001,"15min":1.155},"min":{"1min":0.68999999999999995,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.4590000000000001,"5min":14.255000000000001,"15min":14.255000000000001},"last":0.40000000000000002},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.2450000000000001,"15min":1.1839999999999999},"min":{"1min":0.64800000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.7,"5min":12.666,"15min":12.666},"last":0.66300000000000003}]},{"osd":7,"last update":"Wed Mar 24 21:58:55 2021","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.204,"15min":1.1240000000000001},"min":{"1min":0.66700000000000004,"5min":0.61899999999999999,"15min":0.61499999999999999},"max":{"1min":2.633,"5min":2.633,"15min":2.633},"last":0.76000000000000001},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.2949999999999999,"15min":1.2070000000000001},"min":{"1min":0.73799999999999999,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.7769999999999999,"5min":14.727,"15min":14.727},"last":0.52000000000000002}]}]},{"osd":4,"up_from":32,"seq":137438953580,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":77652,"kb_used_data":59788,"kb_used_omap":0,"kb_used_meta":17856,"kb_avail":93667500,"statfs":{"total":95995035648,"available":95915520000,"internally_reserved":0,"allocated":61222912,"data_stored":94303866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":18284544},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:58:46 2021","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":1.2769999999999999,"15min":0.98799999999999999},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.113,"5min":41.472000000000001,"15min":41.472000000000001},"last":1.601},{"interface":"front","average":{"1min":0.84399999999999997,"5min":1.4490000000000001,"15min":1.0629999999999999},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.3700000000000001,"5min":44.185000000000002,"15min":44.185000000000002},"last":0.39000000000000001}]},{"osd":1,"last update":"Wed Mar 24 21:58:46 2021","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.91800000000000004,"15min":0.91800000000000004},"min":{"1min":0.5,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.194,"5min":1.3120000000000001,"15min":1.4570000000000001},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.89000000000000001,"15min":0.86499999999999999},"min":{"1min":0.55400000000000005,"5min":0.436,"15min":0.436},"max":{"1min":1.1699999999999999,"5min":1.873,"15min":1.873},"last":0.81299999999999994}]},{"osd":2,"last update":"Wed Mar 24 21:58:46 2021","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.95099999999999996,"15min":0.91700000000000004},"min":{"1min":0.52800000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.232,"5min":4.4379999999999997,"15min":4.4379999999999997},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.86599999999999999,"5min":1.0049999999999999,"15min":0.94599999999999995},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.21,"5min":2.738,"15min":2.738},"last":0.40799999999999997}]},{"osd":3,"last update":"Wed Mar 24 21:58:46 2021","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":1.0289999999999999,"15min":0.96399999999999997},"min":{"1min":0.45200000000000001,"5min":0.439,"15min":0.439},"max":{"1min":1.96,"5min":3.2000000000000002,"15min":3.2000000000000002},"last":0.89800000000000002},{"interface":"front","average":{"1min":1.085,"5min":1.0089999999999999,"15min":0.97399999999999998},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":4.1189999999999998,"5min":4.1189999999999998,"15min":4.1189999999999998},"last":1.03}]},{"osd":5,"last update":"Wed Mar 24 21:58:08 2021","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":0.94099999999999995,"15min":0.95199999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.419,"5min":1.6579999999999999,"15min":1.9199999999999999},"last":0.98999999999999999},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.0760000000000001,"15min":1.03},"min":{"1min":0.56000000000000005,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.3759999999999999,"5min":1.607,"15min":2.0760000000000001},"last":1.327}]},{"osd":6,"last update":"Wed Mar 24 21:58:28 2021","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":1.069,"15min":1.0649999999999999},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.3220000000000001,"5min":1.6859999999999999,"15min":1.6859999999999999},"last":0.48899999999999999},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.101,"15min":1.0740000000000001},"min":{"1min":0.41399999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.284,"5min":1.631,"15min":1.631},"last":0.99299999999999999}]},{"osd":7,"last update":"Wed Mar 24 21:58:49 2021","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":1.1299999999999999,"15min":1.165},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.3700000000000001,"5min":1.6830000000000001,"15min":2.1099999999999999},"last":0.93999999999999995},{"interface":"front","average":{"1min":0.999,"5min":1.161,"15min":1.1859999999999999},"min":{"1min":0.57599999999999996,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.3779999999999999,"5min":1.635,"15min":2.0009999999999999},"last":0.59899999999999998}]}]},{"osd":5,"up_from":39,"seq":167503724638,"num_pgs":90,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":97752,"kb_used_data":65616,"kb_used_omap":0,"kb_used_meta":32128,"kb_avail":93647400,"statfs":{"total":95995035648,"available":95894937600,"internally_reserved":0,"allocated":67190784,"data_stored":71282771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32899072},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:58:08 2021","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.80700000000000005,"15min":0.86199999999999999},"min":{"1min":0.372,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.0289999999999999,"5min":1.762,"15min":2.29},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.87,"15min":0.89000000000000001},"min":{"1min":0.45600000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.5800000000000001,"5min":1.5800000000000001,"15min":3.1779999999999999},"last":0.79800000000000004}]},{"osd":1,"last update":"Wed Mar 24 21:58:08 2021","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.86899999999999999,"15min":0.90800000000000003},"min":{"1min":0.45300000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.264,"5min":1.6479999999999999,"15min":2.8690000000000002},"last":0.89700000000000002},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.89300000000000002,"15min":0.92000000000000004},"min":{"1min":0.41499999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.1930000000000001,"5min":2.0030000000000001,"15min":2.0030000000000001},"last":1.081}]},{"osd":2,"last update":"Wed Mar 24 21:58:08 2021","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.95899999999999996,"15min":0.92000000000000004},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.272,"5min":6.5199999999999996,"15min":6.5199999999999996},"last":0.93799999999999994},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.93400000000000005,"15min":0.92300000000000004},"min":{"1min":0.65500000000000003,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.613,"5min":5.5010000000000003,"15min":5.8479999999999999},"last":1.1200000000000001}]},{"osd":3,"last update":"Wed Mar 24 21:58:08 2021","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":0.95599999999999996,"15min":0.94399999999999995},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.744,"5min":2.2000000000000002,"15min":3.1259999999999999},"last":1.2430000000000001},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.97199999999999998,"15min":0.95699999999999996},"min":{"1min":0.61299999999999999,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.175,"5min":1.843,"15min":2.6850000000000001},"last":0.96299999999999997}]},{"osd":4,"last update":"Wed Mar 24 21:58:08 2021","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":1.02,"15min":0.89100000000000001},"min":{"1min":0.56399999999999995,"5min":0.376,"15min":0.376},"max":{"1min":1.6459999999999999,"5min":2.2959999999999998,"15min":2.2959999999999998},"last":1.335},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.0880000000000001,"15min":0.96099999999999997},"min":{"1min":0.74199999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.4910000000000001,"5min":2.629,"15min":2.629},"last":1.1559999999999999}]},{"osd":6,"last update":"Wed Mar 24 21:58:27 2021","interfaces":[{"interface":"back","average":{"1min":1.115,"5min":1.069,"15min":0.93999999999999995},"min":{"1min":0.63600000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.821,"5min":2.3879999999999999,"15min":2.3879999999999999},"last":1.3999999999999999},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.103,"15min":1.016},"min":{"1min":0.70899999999999996,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.645,"5min":2.4660000000000002,"15min":2.4660000000000002},"last":1.1799999999999999}]},{"osd":7,"last update":"Wed Mar 24 21:57:55 2021","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.1519999999999999,"15min":1.0800000000000001},"min":{"1min":0.61299999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.698,"5min":2.548,"15min":2.548},"last":1.218},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.1719999999999999,"15min":1.04},"min":{"1min":0.71499999999999997,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.389,"5min":2.9889999999999999,"15min":2.9889999999999999},"last":1.5049999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":14134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":7234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":13448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":352256,"data_stored":12784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":360448,"data_stored":13024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":9948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":112504832,"data_stored":146269354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7032832,"data_stored":6977532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":159576064,"data_stored":218874529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":139874304,"data_stored":179042263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":59473920,"data_stored":93491228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":63827968,"data_stored":69565904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":170254336,"data_stored":207569766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-03-24T21:59:25.931 INFO:tasks.cephadm.ceph_manager.ceph:decrease pool size by 10 2021-03-24T21:59:25.932 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd pool set default.rgw.log pg_num 22 2021-03-24T21:59:26.015 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:26.068 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:26.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:26.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:59:26.174 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:26.233 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:26.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:26.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:26.267 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:26 smithi125 conmon[19936]: cluster 2021-03-24T21:59:24.825982+0000 2021-03-24T21:59:26.268 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:26 smithi125 conmon[19936]: mgr.y (mgr.14144) 468 : cluster [DBG] pgmap v754: 178 pgs: 4 active+clean+remapped, 52 active+undersized, 38 active+undersized+degraded, 84 active+clean; 339 MiB data, 825 MiB used, 536 GiB / 536 GiB avail; 6.9 MiB/s rd, 1.8 MiB/s wr, 9 op/s; 264/1431 objects degraded (18.449%); 1/1431 objects misplaced (0.070%) 2021-03-24T21:59:26.268 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:26 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T21:59:26.268 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:26 smithi125 conmon[19936]: T21:59:25.265602+0000 mon.a (mon.0) 1185 : cluster [DBG] osdmap e395: 8 total, 7 up, 6 in 2021-03-24T21:59:26.269 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:26 smithi125 conmon[24927]: cluster 2021-03-24T21:59:24.825982 2021-03-24T21:59:26.269 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:26 smithi125 conmon[24927]: +0000 mgr.y (mgr.14144) 468 : cluster [DBG] pgmap v754: 178 pgs: 4 active+clean+remapped, 52 active+undersized, 38 active+undersized+degraded, 84 active+clean; 339 MiB data, 825 MiB used, 536 GiB / 536 GiB avail; 6.9 MiB/s rd, 1.8 MiB/s wr, 9 op/s; 264/1431 objects degraded (18.449%); 1/1431 objects misplaced (0.070%) 2021-03-24T21:59:26.269 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:26 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:59:26.269 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:26 smithi125 conmon[24927]: 24T21:59:25.265602+0000 mon.a (mon.0) 1185 : cluster [DBG] osdmap e395: 8 total, 7 up, 6 in 2021-03-24T21:59:26.277 INFO:tasks.rados.rados.0.smithi125.stdout:1447: done (0 left) 2021-03-24T21:59:26.277 INFO:tasks.rados.rados.0.smithi125.stdout:1448: delete oid 40 current snap is 153 2021-03-24T21:59:26.316 INFO:tasks.rados.rados.0.smithi125.stdout:1448: done (0 left) 2021-03-24T21:59:26.316 INFO:tasks.rados.rados.0.smithi125.stdout:1449: snap_remove snap 134 2021-03-24T21:59:26.342 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:26.386 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:26.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:26.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:26.491 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:26.542 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:26.545 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:26 smithi142 conmon[20125]: cluster 2021-03-24T21:59:24.825982+0000 mgr.y (mgr.14144) 468 : cluster 2021-03-24T21:59:26.546 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:26 smithi142 conmon[20125]: [DBG] pgmap v754: 178 pgs: 4 active+clean+remapped, 52 active+undersized, 38 active+undersized+degraded, 84 active+clean; 339 MiB data, 825 MiB used, 536 GiB / 536 GiB avail; 6.9 MiB/s rd, 1.8 MiB/s wr, 9 op/s; 264/1431 objects degraded (18.449%); 1/1431 objects misplaced (0.070%) 2021-03-24T21:59:26.546 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:26 smithi142 conmon[20125]: cluster 2021-03-24T21:59:25.265602+0000 mon.a (mon.0) 2021-03-24T21:59:26.546 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:26 smithi142 conmon[20125]: 1185 : cluster [DBG] osdmap e395: 8 total, 7 up, 6 in 2021-03-24T21:59:26.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:26.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:59:26.649 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:26.700 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:26.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:26.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:59:26.805 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:26.856 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:26.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:26.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:59:26.963 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:27.006 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-03-24T21:59:27.007 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-03-24T21:59:27.017 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:27.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:27.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:27.055 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:59:27.077 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.3 dump_ops_in_flight 2021-03-24T21:59:27.124 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:27.169 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:27.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:27.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:59:27.275 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:27.284 INFO:tasks.rados.rados.0.smithi125.stdout:1449: done (0 left) 2021-03-24T21:59:27.284 INFO:tasks.rados.rados.0.smithi125.stdout:1450: snap_create 2021-03-24T21:59:27.354 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:27.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:27.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:59:27.467 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:27.512 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:27.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:27.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:59:27.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:27 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:59:27.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:27 smithi142 conmon[20125]: 59:26.271351+0000 mon.a (mon.0) 1186 : cluster [DBG] osdmap e396: 8 total, 7 up, 6 in 2021-03-24T21:59:27.617 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:27.627 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:27 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:59:27.628 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:27 smithi125 conmon[24927]: 59:26.271351+0000 mon.a (mon.0) 1186 : cluster [DBG] osdmap e396: 8 total, 7 up, 6 in 2021-03-24T21:59:27.628 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:27 smithi125 conmon[19936]: cluster 2021-03-24T21:59:26. 2021-03-24T21:59:27.628 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:27 smithi125 conmon[19936]: 271351+0000 mon.a (mon.0) 1186 : cluster [DBG] osdmap e396: 8 total, 7 up, 6 in 2021-03-24T21:59:27.659 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:27.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:27.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:27.766 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:27.828 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:27.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:27.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:59:27.935 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:27.994 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:28.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:28.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:59:28.102 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:28.158 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:28.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:28.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:59:28.265 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:28.286 INFO:tasks.rados.rados.0.smithi125.stdout:1450: done (0 left) 2021-03-24T21:59:28.287 INFO:tasks.rados.rados.0.smithi125.stdout:1451: rollback oid 39 current snap is 154 2021-03-24T21:59:28.287 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 39 to 137 2021-03-24T21:59:28.287 INFO:tasks.rados.rados.0.smithi125.stdout:1452: copy_from oid 28 from oid 41 current snap is 154 2021-03-24T21:59:28.288 INFO:tasks.rados.rados.0.smithi125.stdout:1453: write oid 31 current snap is 154 2021-03-24T21:59:28.288 INFO:tasks.rados.rados.0.smithi125.stdout:1453: seq_num 517 ranges {689219=415737,1770367=646847} 2021-03-24T21:59:28.293 INFO:tasks.rados.rados.0.smithi125.stdout:1453: writing smithi12556470-31 from 689219 to 1104956 tid 1 2021-03-24T21:59:28.302 INFO:tasks.rados.rados.0.smithi125.stdout:1453: writing smithi12556470-31 from 1770367 to 2417214 tid 2 2021-03-24T21:59:28.302 INFO:tasks.rados.rados.0.smithi125.stdout:1454: read oid 38 snap -1 2021-03-24T21:59:28.302 INFO:tasks.rados.rados.0.smithi125.stdout:1454: expect (ObjNum 489 snap 140 seq_num 489) 2021-03-24T21:59:28.303 INFO:tasks.rados.rados.0.smithi125.stdout:1455: write oid 29 current snap is 154 2021-03-24T21:59:28.303 INFO:tasks.rados.rados.0.smithi125.stdout:1455: seq_num 518 ranges {706026=490986,1828849=790505} 2021-03-24T21:59:28.320 INFO:tasks.rados.rados.0.smithi125.stdout:1455: writing smithi12556470-29 from 706026 to 1197012 tid 1 2021-03-24T21:59:28.330 INFO:tasks.rados.rados.0.smithi125.stdout:1455: writing smithi12556470-29 from 1828849 to 2619354 tid 2 2021-03-24T21:59:28.333 INFO:tasks.rados.rados.0.smithi125.stdout:1451: finishing rollback tid 0 to smithi12556470-39 2021-03-24T21:59:28.333 INFO:tasks.rados.rados.0.smithi125.stdout:1451: finishing rollback tid 1 to smithi12556470-39 2021-03-24T21:59:28.334 INFO:tasks.rados.rados.0.smithi125.stdout:1451: finishing rollback tid 2 to smithi12556470-39 2021-03-24T21:59:28.334 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 39 v 396 (ObjNum 408 snap 106 seq_num 408) dirty exists 2021-03-24T21:59:28.334 INFO:tasks.rados.rados.0.smithi125.stdout:1454: expect (ObjNum 489 snap 140 seq_num 489) 2021-03-24T21:59:28.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: cluster 2021-03-24T21:59:28.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: 2021-03-24T21:59:28.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: 2021 2021-03-24T21:59:28.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: - 2021-03-24T21:59:28.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: 03 2021-03-24T21:59:28.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: - 2021-03-24T21:59:28.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: 24 2021-03-24T21:59:28.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: T 2021-03-24T21:59:28.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: 21 2021-03-24T21:59:28.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: : 2021-03-24T21:59:28.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: 59 2021-03-24T21:59:28.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: : 2021-03-24T21:59:28.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: 26 2021-03-24T21:59:28.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: . 2021-03-24T21:59:28.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: 827115 2021-03-24T21:59:28.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: +0000 2021-03-24T21:59:28.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: 2021-03-24T21:59:28.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: mgr.y 2021-03-24T21:59:28.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: ( 2021-03-24T21:59:28.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: mgr 2021-03-24T21:59:28.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: . 2021-03-24T21:59:28.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: 14144 2021-03-24T21:59:28.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: ) 2021-03-24T21:59:28.373 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: 469 2021-03-24T21:59:28.373 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: : 2021-03-24T21:59:28.373 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: cluster 2021-03-24T21:59:28.373 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: 2021-03-24T21:59:28.374 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: [DBG] 2021-03-24T21:59:28.374 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: 2021-03-24T21:59:28.375 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: pgmap v757: 178 pgs: 4 active+clean+remapped, 52 active+undersized, 38 active+undersized+degraded, 84 active+clean; 339 MiB data, 829 MiB used, 536 GiB / 536 GiB avail; 4.5 MiB/s rd, 1.8 MiB/s wr, 7 op/s; 264/1431 objects degraded (18.449%); 1/1431 objects misplaced (0.070%) 2021-03-24T21:59:28.375 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: 2021-03-24T21:59:28.376 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: cluster 2021-03-24T21:59:28.376 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: 2021-03-24T21:59:28.376 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: 2021 2021-03-24T21:59:28.377 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: - 2021-03-24T21:59:28.377 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: 03 2021-03-24T21:59:28.378 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: - 2021-03-24T21:59:28.378 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: 24 2021-03-24T21:59:28.378 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: T 2021-03-24T21:59:28.379 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: 21 2021-03-24T21:59:28.379 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: : 2021-03-24T21:59:28.379 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: 59 2021-03-24T21:59:28.379 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: : 2021-03-24T21:59:28.380 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: 27 2021-03-24T21:59:28.380 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: . 2021-03-24T21:59:28.380 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: 279560 2021-03-24T21:59:28.380 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: +0000 2021-03-24T21:59:28.380 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: 2021-03-24T21:59:28.381 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: mon.a 2021-03-24T21:59:28.381 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: ( 2021-03-24T21:59:28.381 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: mon 2021-03-24T21:59:28.381 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: . 2021-03-24T21:59:28.382 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: 0 2021-03-24T21:59:28.382 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: ) 2021-03-24T21:59:28.382 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: 1187 2021-03-24T21:59:28.382 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: : 2021-03-24T21:59:28.383 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: cluster 2021-03-24T21:59:28.383 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: 2021-03-24T21:59:28.383 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: [DBG] 2021-03-24T21:59:28.383 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: 2021-03-24T21:59:28.384 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: osdmap e397: 8 total, 7 up, 6 in 2021-03-24T21:59:28.384 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:28 smithi142 conmon[20125]: 2021-03-24T21:59:28.397 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:28.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:28.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:28.425 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:59:28.450 INFO:tasks.rados.rados.0.smithi125.stdout:1451: done (4 left) 2021-03-24T21:59:28.450 INFO:tasks.rados.rados.0.smithi125.stdout:1454: done (3 left) 2021-03-24T21:59:28.450 INFO:tasks.rados.rados.0.smithi125.stdout:1456: copy_from oid 20 from oid 2 current snap is 154 2021-03-24T21:59:28.451 INFO:tasks.rados.rados.0.smithi125.stdout:1457: write oid 37 current snap is 154 2021-03-24T21:59:28.451 INFO:tasks.rados.rados.0.smithi125.stdout:1457: seq_num 519 ranges {722833=566235,1887331=650516} 2021-03-24T21:59:28.458 INFO:tasks.rados.rados.0.smithi125.stdout:1457: writing smithi12556470-37 from 722833 to 1289068 tid 1 2021-03-24T21:59:28.466 INFO:tasks.rados.rados.0.smithi125.stdout:1457: writing smithi12556470-37 from 1887331 to 2537847 tid 2 2021-03-24T21:59:28.468 INFO:tasks.rados.rados.0.smithi125.stdout:1453: finishing write tid 1 to smithi12556470-31 2021-03-24T21:59:28.469 INFO:tasks.rados.rados.0.smithi125.stdout:1453: finishing write tid 2 to smithi12556470-31 2021-03-24T21:59:28.469 INFO:tasks.rados.rados.0.smithi125.stdout:1453: finishing write tid 3 to smithi12556470-31 2021-03-24T21:59:28.469 INFO:tasks.rados.rados.0.smithi125.stdout:1453: finishing write tid 4 to smithi12556470-31 2021-03-24T21:59:28.469 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 31 v 532 (ObjNum 517 snap 154 seq_num 517) dirty exists 2021-03-24T21:59:28.469 INFO:tasks.rados.rados.0.smithi125.stdout:1453: left oid 31 (ObjNum 517 snap 154 seq_num 517) 2021-03-24T21:59:28.470 INFO:tasks.rados.rados.0.smithi125.stdout:1452: finishing copy_from to smithi12556470-28 2021-03-24T21:59:28.470 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 28 v 383 (ObjNum 510 snap 148 seq_num 510) dirty exists 2021-03-24T21:59:28.470 INFO:tasks.rados.rados.0.smithi125.stdout:1455: finishing write tid 1 to smithi12556470-29 2021-03-24T21:59:28.470 INFO:tasks.rados.rados.0.smithi125.stdout:1455: finishing write tid 2 to smithi12556470-29 2021-03-24T21:59:28.471 INFO:tasks.rados.rados.0.smithi125.stdout:1455: finishing write tid 3 to smithi12556470-29 2021-03-24T21:59:28.471 INFO:tasks.rados.rados.0.smithi125.stdout:1455: finishing write tid 4 to smithi12556470-29 2021-03-24T21:59:28.471 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 29 v 193 (ObjNum 518 snap 154 seq_num 518) dirty exists 2021-03-24T21:59:28.471 INFO:tasks.rados.rados.0.smithi125.stdout:1455: left oid 29 (ObjNum 518 snap 154 seq_num 518) 2021-03-24T21:59:28.471 INFO:tasks.rados.rados.0.smithi125.stdout:1452: finishing copy_from racing read to smithi12556470-28 2021-03-24T21:59:28.472 INFO:tasks.rados.rados.0.smithi125.stdout:1452: done (4 left) 2021-03-24T21:59:28.472 INFO:tasks.rados.rados.0.smithi125.stdout:1453: done (3 left) 2021-03-24T21:59:28.472 INFO:tasks.rados.rados.0.smithi125.stdout:1455: done (2 left) 2021-03-24T21:59:28.472 INFO:tasks.rados.rados.0.smithi125.stdout:1458: snap_remove snap 152 2021-03-24T21:59:28.511 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:28.521 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:28 smithi125 conmon[19936]: cluster 2021-03-24T21:59:28.521 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:28 smithi125 conmon[19936]: 2021-03-24T21:59:28.522 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:28 smithi125 conmon[19936]: 2021 2021-03-24T21:59:28.522 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:28 smithi125 conmon[19936]: - 2021-03-24T21:59:28.522 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:28 smithi125 conmon[19936]: 03 2021-03-24T21:59:28.522 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:28 smithi125 conmon[19936]: - 2021-03-24T21:59:28.523 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:28 smithi125 conmon[19936]: 24 2021-03-24T21:59:28.523 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:28 smithi125 conmon[19936]: T 2021-03-24T21:59:28.523 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:28 smithi125 conmon[19936]: 21 2021-03-24T21:59:28.523 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:28 smithi125 conmon[19936]: : 2021-03-24T21:59:28.524 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:28 smithi125 conmon[19936]: 59 2021-03-24T21:59:28.524 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:28 smithi125 conmon[19936]: : 2021-03-24T21:59:28.524 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:28 smithi125 conmon[19936]: 26 2021-03-24T21:59:28.524 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:28 smithi125 conmon[19936]: . 2021-03-24T21:59:28.525 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:28 smithi125 conmon[19936]: 827115 2021-03-24T21:59:28.525 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:28 smithi125 conmon[19936]: +0000 2021-03-24T21:59:28.525 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:28 smithi125 conmon[19936]: 2021-03-24T21:59:28.525 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:28 smithi125 conmon[19936]: mgr.y 2021-03-24T21:59:28.526 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:28 smithi125 conmon[19936]: ( 2021-03-24T21:59:28.526 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:28 smithi125 conmon[19936]: mgr 2021-03-24T21:59:28.526 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:28 smithi125 conmon[19936]: . 2021-03-24T21:59:28.526 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:28 smithi125 conmon[19936]: 14144) 469 : cluster [DBG] pgmap v757: 178 pgs: 4 active+clean+remapped, 52 active+undersized, 38 active+undersized+degraded, 84 active+clean; 339 MiB data, 829 MiB used, 536 GiB / 536 GiB avail; 4.5 MiB/s rd, 1.8 MiB/s wr, 7 op/s; 264/1431 objects degraded (18.449%); 1/1431 objects misplaced (0.070%) 2021-03-24T21:59:28.526 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:28 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:59:28.527 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:28 smithi125 conmon[19936]: 24T21:59:27.279560+0000 mon.a (mon.0) 1187 : cluster [DBG] 2021-03-24T21:59:28.527 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:28 smithi125 conmon[19936]: osdmap e397: 8 total, 7 up, 6 in 2021-03-24T21:59:28.528 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:28 smithi125 conmon[24927]: cluster 2021-03-24T21:59:26.827115+0000 mgr.y 2021-03-24T21:59:28.528 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:28 smithi125 conmon[24927]: (mgr.14144) 469 : cluster [DBG] pgmap v757: 178 pgs: 4 active+clean+remapped, 52 active+undersized, 38 active+undersized+degraded, 84 active+clean; 339 MiB data, 829 MiB used, 536 GiB / 536 GiB avail; 4.5 MiB/s rd, 1.8 MiB/s wr, 7 op/s; 264/1431 objects degraded (18.449%); 1/1431 objects misplaced (0.070%) 2021-03-24T21:59:28.529 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:28 smithi125 conmon[24927]: cluster 2021-03-24T21:59:27.279560+0000 mon.a (mon.0) 1187 : 2021-03-24T21:59:28.529 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:28 smithi125 conmon[24927]: cluster [DBG] osdmap e397: 8 total, 7 up, 6 in 2021-03-24T21:59:28.554 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:28.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:28.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:59:28.658 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:28.713 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:28.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:28.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:28.819 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:28.873 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:28.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:28.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:28.979 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:29.008 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-24T21:59:29.039 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:29.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:29.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:59:29.145 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:29.206 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:29.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:29.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:59:29.310 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:29.323 INFO:tasks.rados.rados.0.smithi125.stdout:1458: done (2 left) 2021-03-24T21:59:29.323 INFO:tasks.rados.rados.0.smithi125.stdout:1459: write oid 9 current snap is 154 2021-03-24T21:59:29.324 INFO:tasks.rados.rados.0.smithi125.stdout:1459: seq_num 520 ranges {739640=641484,1829460=794174} 2021-03-24T21:59:29.334 INFO:tasks.rados.rados.0.smithi125.stdout:1459: writing smithi12556470-9 from 739640 to 1381124 tid 1 2021-03-24T21:59:29.341 INFO:tasks.rados.rados.0.smithi125.stdout:1459: writing smithi12556470-9 from 1829460 to 2623634 tid 2 2021-03-24T21:59:29.341 INFO:tasks.rados.rados.0.smithi125.stdout:1457: finishing write tid 1 to smithi12556470-37 2021-03-24T21:59:29.341 INFO:tasks.rados.rados.0.smithi125.stdout:1456: finishing copy_from to smithi12556470-20 2021-03-24T21:59:29.342 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 20 v 397 (ObjNum 506 snap 147 seq_num 506) dirty exists 2021-03-24T21:59:29.342 INFO:tasks.rados.rados.0.smithi125.stdout:1456: finishing copy_from racing read to smithi12556470-20 2021-03-24T21:59:29.342 INFO:tasks.rados.rados.0.smithi125.stdout:1457: finishing write tid 2 to smithi12556470-37 2021-03-24T21:59:29.342 INFO:tasks.rados.rados.0.smithi125.stdout:1457: finishing write tid 3 to smithi12556470-37 2021-03-24T21:59:29.343 INFO:tasks.rados.rados.0.smithi125.stdout:1457: finishing write tid 4 to smithi12556470-37 2021-03-24T21:59:29.343 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 37 v 344 (ObjNum 519 snap 154 seq_num 519) dirty exists 2021-03-24T21:59:29.343 INFO:tasks.rados.rados.0.smithi125.stdout:1457: left oid 37 (ObjNum 519 snap 154 seq_num 519) 2021-03-24T21:59:29.343 INFO:tasks.rados.rados.0.smithi125.stdout:1456: done (2 left) 2021-03-24T21:59:29.344 INFO:tasks.rados.rados.0.smithi125.stdout:1457: done (1 left) 2021-03-24T21:59:29.344 INFO:tasks.rados.rados.0.smithi125.stdout:1460: delete oid 3 current snap is 154 2021-03-24T21:59:29.345 INFO:tasks.rados.rados.0.smithi125.stdout:1460: done (1 left) 2021-03-24T21:59:29.345 INFO:tasks.rados.rados.0.smithi125.stdout:1461: write oid 1 current snap is 154 2021-03-24T21:59:29.345 INFO:tasks.rados.rados.0.smithi125.stdout:1461: seq_num 522 ranges {773254=791982,2346424=426830} 2021-03-24T21:59:29.363 INFO:tasks.rados.rados.0.smithi125.stdout:1461: writing smithi12556470-1 from 773254 to 1565236 tid 1 2021-03-24T21:59:29.378 INFO:tasks.rados.rados.0.smithi125.stdout:1461: writing smithi12556470-1 from 2346424 to 2773254 tid 2 2021-03-24T21:59:29.378 INFO:tasks.rados.rados.0.smithi125.stdout:1462: read oid 3 snap 141 2021-03-24T21:59:29.379 INFO:tasks.rados.rados.0.smithi125.stdout:1462: expect (ObjNum 442 snap 119 seq_num 442) 2021-03-24T21:59:29.379 INFO:tasks.rados.rados.0.smithi125.stdout:1463: delete oid 4 current snap is 154 2021-03-24T21:59:29.385 INFO:tasks.rados.rados.0.smithi125.stdout:1459: finishing write tid 1 to smithi12556470-9 2021-03-24T21:59:29.386 INFO:tasks.rados.rados.0.smithi125.stdout:1459: finishing write tid 2 to smithi12556470-9 2021-03-24T21:59:29.386 INFO:tasks.rados.rados.0.smithi125.stdout:1459: finishing write tid 3 to smithi12556470-9 2021-03-24T21:59:29.386 INFO:tasks.rados.rados.0.smithi125.stdout:1459: finishing write tid 4 to smithi12556470-9 2021-03-24T21:59:29.386 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 9 v 539 (ObjNum 520 snap 154 seq_num 520) dirty exists 2021-03-24T21:59:29.387 INFO:tasks.rados.rados.0.smithi125.stdout:1459: left oid 9 (ObjNum 520 snap 154 seq_num 520) 2021-03-24T21:59:29.387 INFO:tasks.rados.rados.0.smithi125.stdout:1461: finishing write tid 1 to smithi12556470-1 2021-03-24T21:59:29.387 INFO:tasks.rados.rados.0.smithi125.stdout:1459: done (3 left) 2021-03-24T21:59:29.387 INFO:tasks.rados.rados.0.smithi125.stdout:1463: done (2 left) 2021-03-24T21:59:29.388 INFO:tasks.rados.rados.0.smithi125.stdout:1464: write oid 39 current snap is 154 2021-03-24T21:59:29.388 INFO:tasks.rados.rados.0.smithi125.stdout:1464: seq_num 524 ranges {406868=542480,1547035=401512,2690354=116514} 2021-03-24T21:59:29.399 INFO:tasks.rados.rados.0.smithi125.stdout:1464: writing smithi12556470-39 from 406868 to 949348 tid 1 2021-03-24T21:59:29.416 INFO:tasks.rados.rados.0.smithi125.stdout:1464: writing smithi12556470-39 from 1547035 to 1948547 tid 2 2021-03-24T21:59:29.429 INFO:tasks.rados.rados.0.smithi125.stdout:1464: writing smithi12556470-39 from 2690354 to 2806868 tid 3 2021-03-24T21:59:29.442 INFO:tasks.rados.rados.0.smithi125.stdout:1461: finishing write tid 2 to smithi12556470-1 2021-03-24T21:59:29.442 INFO:tasks.rados.rados.0.smithi125.stdout:1461: finishing write tid 3 to smithi12556470-1 2021-03-24T21:59:29.443 INFO:tasks.rados.rados.0.smithi125.stdout:1461: finishing write tid 4 to smithi12556470-1 2021-03-24T21:59:29.443 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 1 v 460 (ObjNum 522 snap 154 seq_num 522) dirty exists 2021-03-24T21:59:29.443 INFO:tasks.rados.rados.0.smithi125.stdout:1461: left oid 1 (ObjNum 522 snap 154 seq_num 522) 2021-03-24T21:59:29.444 INFO:tasks.rados.rados.0.smithi125.stdout:1462: expect (ObjNum 442 snap 119 seq_num 442) 2021-03-24T21:59:29.444 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:29.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:29.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:59:29.551 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:29 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:59:29.551 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:29 smithi125 conmon[19936]: 21:59:28.283049+0000 mon.a (mon.0) 1188 : cluster [DBG] osdmap e398: 8 total, 7 up, 6 in 2021-03-24T21:59:29.552 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:29 smithi125 conmon[24927]: cluster 2021-03-24T21:59:28. 2021-03-24T21:59:29.552 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:29 smithi125 conmon[24927]: 283049+0000 mon.a (mon.0) 1188 : cluster [DBG] osdmap e398: 8 total, 7 up, 6 in 2021-03-24T21:59:29.553 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:29.613 INFO:tasks.rados.rados.0.smithi125.stdout:1464: finishing write tid 1 to smithi12556470-39 2021-03-24T21:59:29.614 INFO:tasks.rados.rados.0.smithi125.stdout:1464: finishing write tid 2 to smithi12556470-39 2021-03-24T21:59:29.614 INFO:tasks.rados.rados.0.smithi125.stdout:1461: done (2 left) 2021-03-24T21:59:29.614 INFO:tasks.rados.rados.0.smithi125.stdout:1462: done (1 left) 2021-03-24T21:59:29.615 INFO:tasks.rados.rados.0.smithi125.stdout:1465: read oid 15 snap -1 2021-03-24T21:59:29.615 INFO:tasks.rados.rados.0.smithi125.stdout:1465: expect deleted 2021-03-24T21:59:29.615 INFO:tasks.rados.rados.0.smithi125.stdout:1466: write oid 14 current snap is 154 2021-03-24T21:59:29.615 INFO:tasks.rados.rados.0.smithi125.stdout:1464: finishing write tid 3 to smithi12556470-39 2021-03-24T21:59:29.616 INFO:tasks.rados.rados.0.smithi125.stdout:1466: seq_num 525 ranges {423675=734082,1721870=545170,2717777=105898} 2021-03-24T21:59:29.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:29 smithi142 conmon[20125]: cluster 2021-03-24T21:59 2021-03-24T21:59:29.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:29 smithi142 conmon[20125]: :28.283049+0000 mon.a (mon.0) 1188 : cluster [DBG] osdmap e398: 8 total, 7 up, 6 in 2021-03-24T21:59:29.621 INFO:tasks.rados.rados.0.smithi125.stdout:1466: writing smithi12556470-14 from 423675 to 1157757 tid 1 2021-03-24T21:59:29.624 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:29.628 INFO:tasks.rados.rados.0.smithi125.stdout:1466: writing smithi12556470-14 from 1721870 to 2267040 tid 2 2021-03-24T21:59:29.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:29.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:59:29.633 INFO:tasks.rados.rados.0.smithi125.stdout:1466: writing smithi12556470-14 from 2717777 to 2823675 tid 3 2021-03-24T21:59:29.633 INFO:tasks.rados.rados.0.smithi125.stdout:1467: delete oid 2 current snap is 154 2021-03-24T21:59:29.633 INFO:tasks.rados.rados.0.smithi125.stdout:1464: finishing write tid 4 to smithi12556470-39 2021-03-24T21:59:29.633 INFO:tasks.rados.rados.0.smithi125.stdout:1464: finishing write tid 5 to smithi12556470-39 2021-03-24T21:59:29.634 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 39 v 400 (ObjNum 524 snap 154 seq_num 524) dirty exists 2021-03-24T21:59:29.634 INFO:tasks.rados.rados.0.smithi125.stdout:1464: left oid 39 (ObjNum 524 snap 154 seq_num 524) 2021-03-24T21:59:29.634 INFO:tasks.rados.rados.0.smithi125.stdout:1466: finishing write tid 1 to smithi12556470-14 2021-03-24T21:59:29.634 INFO:tasks.rados.rados.0.smithi125.stdout:1466: finishing write tid 2 to smithi12556470-14 2021-03-24T21:59:29.637 INFO:tasks.rados.rados.0.smithi125.stdout:1464: done (3 left) 2021-03-24T21:59:29.637 INFO:tasks.rados.rados.0.smithi125.stdout:1465: done (2 left) 2021-03-24T21:59:29.637 INFO:tasks.rados.rados.0.smithi125.stdout:1467: done (1 left) 2021-03-24T21:59:29.638 INFO:tasks.rados.rados.0.smithi125.stdout:1468: snap_create 2021-03-24T21:59:29.640 INFO:tasks.rados.rados.0.smithi125.stdout:1466: finishing write tid 3 to smithi12556470-14 2021-03-24T21:59:29.641 INFO:tasks.rados.rados.0.smithi125.stdout:1466: finishing write tid 4 to smithi12556470-14 2021-03-24T21:59:29.642 INFO:tasks.rados.rados.0.smithi125.stdout:1466: finishing write tid 5 to smithi12556470-14 2021-03-24T21:59:29.642 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 14 v 543 (ObjNum 525 snap 154 seq_num 525) dirty exists 2021-03-24T21:59:29.643 INFO:tasks.rados.rados.0.smithi125.stdout:1466: left oid 14 (ObjNum 525 snap 154 seq_num 525) 2021-03-24T21:59:29.715 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:59:29.731 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:29.783 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:29.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:29.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:29.890 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:29.948 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:29.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:29.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:30.053 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:30.099 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:30.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:30.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:59:30.204 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:30.220 INFO:tasks.rados.rados.0.smithi125.stdout:1466: done (1 left) 2021-03-24T21:59:30.220 INFO:tasks.rados.rados.0.smithi125.stdout:1468: done (0 left) 2021-03-24T21:59:30.220 INFO:tasks.rados.rados.0.smithi125.stdout:1469: read oid 18 snap 136 2021-03-24T21:59:30.221 INFO:tasks.rados.rados.0.smithi125.stdout:1469: expect (ObjNum 384 snap 99 seq_num 384) 2021-03-24T21:59:30.221 INFO:tasks.rados.rados.0.smithi125.stdout:1470: write oid 7 current snap is 155 2021-03-24T21:59:30.221 INFO:tasks.rados.rados.0.smithi125.stdout:1470: seq_num 527 ranges {457289=484580,1438834=548839,2772623=84666} 2021-03-24T21:59:30.227 INFO:tasks.rados.rados.0.smithi125.stdout:1470: writing smithi12556470-7 from 457289 to 941869 tid 1 2021-03-24T21:59:30.230 INFO:teuthology.orchestra.run.smithi125.stderr:nodeep-scrub is set 2021-03-24T21:59:30.234 INFO:tasks.rados.rados.0.smithi125.stdout:1470: writing smithi12556470-7 from 1438834 to 1987673 tid 2 2021-03-24T21:59:30.241 INFO:tasks.rados.rados.0.smithi125.stdout:1470: writing smithi12556470-7 from 2772623 to 2857289 tid 3 2021-03-24T21:59:30.242 INFO:tasks.rados.rados.0.smithi125.stdout:1471: copy_from oid 6 from oid 26 current snap is 155 2021-03-24T21:59:30.242 INFO:tasks.rados.rados.0.smithi125.stdout:1472: read oid 21 snap -1 2021-03-24T21:59:30.242 INFO:tasks.rados.rados.0.smithi125.stdout:1472: expect (ObjNum 500 snap 146 seq_num 500) 2021-03-24T21:59:30.243 INFO:tasks.rados.rados.0.smithi125.stdout:1473: read oid 34 snap -1 2021-03-24T21:59:30.243 INFO:tasks.rados.rados.0.smithi125.stdout:1473: expect (ObjNum 503 snap 146 seq_num 503) 2021-03-24T21:59:30.243 INFO:tasks.rados.rados.0.smithi125.stdout:1474: rollback oid 14 current snap is 155 2021-03-24T21:59:30.243 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 14 to 141 2021-03-24T21:59:30.243 INFO:tasks.rados.rados.0.smithi125.stdout:1475: write oid 17 current snap is 155 2021-03-24T21:59:30.244 INFO:tasks.rados.rados.0.smithi125.stdout:1475: seq_num 528 ranges {474096=559829,1780963=408850,2800046=74050} 2021-03-24T21:59:30.249 INFO:tasks.rados.rados.0.smithi125.stdout:1475: writing smithi12556470-17 from 474096 to 1033925 tid 1 2021-03-24T21:59:30.257 INFO:tasks.rados.rados.0.smithi125.stdout:1475: writing smithi12556470-17 from 1780963 to 2189813 tid 2 2021-03-24T21:59:30.261 INFO:tasks.rados.rados.0.smithi125.stdout:1475: writing smithi12556470-17 from 2800046 to 2874096 tid 3 2021-03-24T21:59:30.261 INFO:tasks.rados.rados.0.smithi125.stdout:1476: copy_from oid 32 from oid 23 current snap is 155 2021-03-24T21:59:30.262 INFO:tasks.rados.rados.0.smithi125.stdout:1477: delete oid 35 current snap is 155 2021-03-24T21:59:30.262 INFO:tasks.rados.rados.0.smithi125.stdout:1469: expect (ObjNum 384 snap 99 seq_num 384) 2021-03-24T21:59:30.301 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:30.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:30.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:30.338 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: cluster 2021-03-24T21:59:30.339 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: cluster 2021-03-24T21:59:30.339 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 2021-03-24T21:59:30.339 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 2021 2021-03-24T21:59:30.340 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: - 2021-03-24T21:59:30.340 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 03 2021-03-24T21:59:30.340 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: - 2021-03-24T21:59:30.340 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 24 2021-03-24T21:59:30.341 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: T 2021-03-24T21:59:30.341 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 21 2021-03-24T21:59:30.341 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: : 2021-03-24T21:59:30.341 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 59 2021-03-24T21:59:30.342 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: : 2021-03-24T21:59:30.342 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 28 2021-03-24T21:59:30.342 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: . 2021-03-24T21:59:30.342 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 828143 2021-03-24T21:59:30.343 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: +0000 2021-03-24T21:59:30.343 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 2021-03-24T21:59:30.343 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: mgr.y 2021-03-24T21:59:30.343 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: ( 2021-03-24T21:59:30.344 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: mgr 2021-03-24T21:59:30.344 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: . 2021-03-24T21:59:30.344 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 14144 2021-03-24T21:59:30.344 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: ) 2021-03-24T21:59:30.344 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 470 2021-03-24T21:59:30.345 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: : 2021-03-24T21:59:30.345 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: cluster 2021-03-24T21:59:30.345 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 2021-03-24T21:59:30.345 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: [DBG] 2021-03-24T21:59:30.346 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 2021-03-24T21:59:30.346 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: pgmap v760: 178 pgs: 4 active+clean+remapped, 52 active+undersized, 38 active+undersized+degraded, 84 active+clean; 339 MiB data, 829 MiB used, 536 GiB / 536 GiB avail; 264/1431 objects degraded (18.449%); 1/1431 objects misplaced (0.070%) 2021-03-24T21:59:30.346 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 2021-03-24T21:59:30.347 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: cluster 2021-03-24T21:59:30.347 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 2021-03-24T21:59:30.347 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 2021 2021-03-24T21:59:30.347 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: - 2021-03-24T21:59:30.347 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 03 2021-03-24T21:59:30.348 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: - 2021-03-24T21:59:30.348 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 24 2021-03-24T21:59:30.348 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: T 2021-03-24T21:59:30.348 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 21 2021-03-24T21:59:30.349 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: : 2021-03-24T21:59:30.349 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 59 2021-03-24T21:59:30.349 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: : 2021-03-24T21:59:30.349 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 29 2021-03-24T21:59:30.349 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: . 2021-03-24T21:59:30.350 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 310624 2021-03-24T21:59:30.350 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: +0000 2021-03-24T21:59:30.350 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 2021-03-24T21:59:30.350 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: mon.a 2021-03-24T21:59:30.351 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: ( 2021-03-24T21:59:30.351 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: mon 2021-03-24T21:59:30.351 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: . 2021-03-24T21:59:30.351 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 0 2021-03-24T21:59:30.352 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: ) 2021-03-24T21:59:30.352 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 1189 2021-03-24T21:59:30.352 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: : 2021-03-24T21:59:30.352 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: cluster 2021-03-24T21:59:30.352 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 2021-03-24T21:59:30.353 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: [DBG] 2021-03-24T21:59:30.353 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 2021-03-24T21:59:30.353 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: osdmap e399: 8 total, 7 up, 6 in 2021-03-24T21:59:30.353 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 2021-03-24T21:59:30.354 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: audit 2021-03-24T21:59:30.354 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 2021-03-24T21:59:30.354 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 2021 2021-03-24T21:59:30.354 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: - 2021-03-24T21:59:30.355 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 03 2021-03-24T21:59:30.355 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: - 2021-03-24T21:59:30.355 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 24 2021-03-24T21:59:30.355 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: T 2021-03-24T21:59:30.355 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 21 2021-03-24T21:59:30.356 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: : 2021-03-24T21:59:30.356 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 59 2021-03-24T21:59:30.356 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: : 2021-03-24T21:59:30.356 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 29 2021-03-24T21:59:30.357 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: . 2021-03-24T21:59:30.357 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 974208 2021-03-24T21:59:30.357 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: +0000 2021-03-24T21:59:30.357 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 2021-03-24T21:59:30.357 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: mon.c 2021-03-24T21:59:30.358 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: ( 2021-03-24T21:59:30.358 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: mon 2021-03-24T21:59:30.358 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: . 2021-03-24T21:59:30.358 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 2 2021-03-24T21:59:30.359 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: ) 2021-03-24T21:59:30.359 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 62 2021-03-24T21:59:30.360 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: : 2021-03-24T21:59:30.360 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: audit 2021-03-24T21:59:30.361 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 2021-03-24T21:59:30.361 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: [INF] 2021-03-24T21:59:30.361 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 2021-03-24T21:59:30.361 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: from='client.? v1:172.21.15.125:0/2059877721' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:59:30.362 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 2021-03-24T21:59:30.362 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: audit 2021-03-24T21:59:30.362 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 2021-03-24T21:59:30.362 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 2021 2021-03-24T21:59:30.363 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: - 2021-03-24T21:59:30.363 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 03 2021-03-24T21:59:30.363 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: - 2021-03-24T21:59:30.363 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 24 2021-03-24T21:59:30.364 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: T 2021-03-24T21:59:30.364 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 21 2021-03-24T21:59:30.364 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: : 2021-03-24T21:59:30.364 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 59 2021-03-24T21:59:30.365 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: : 2021-03-24T21:59:30.365 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 29 2021-03-24T21:59:30.365 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: . 2021-03-24T21:59:30.365 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 974606 2021-03-24T21:59:30.366 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: +0000 2021-03-24T21:59:30.366 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 2021-03-24T21:59:30.366 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: mon.a 2021-03-24T21:59:30.366 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: ( 2021-03-24T21:59:30.366 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: mon 2021-03-24T21:59:30.367 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: . 2021-03-24T21:59:30.367 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 0 2021-03-24T21:59:30.367 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: ) 2021-03-24T21:59:30.368 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 1190 2021-03-24T21:59:30.368 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: : 2021-03-24T21:59:30.368 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: audit 2021-03-24T21:59:30.368 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 2021-03-24T21:59:30.368 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: [INF] 2021-03-24T21:59:30.369 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 2021-03-24T21:59:30.369 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: from='client.35180 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:59:30.369 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 2021-03-24T21:59:30.369 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: cluster 2021-03-24T21:59:30.370 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 2021-03-24T21:59:30.370 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 2021 2021-03-24T21:59:30.370 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: - 2021-03-24T21:59:30.371 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 03 2021-03-24T21:59:30.371 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: - 2021-03-24T21:59:30.371 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 24 2021-03-24T21:59:30.371 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: T 2021-03-24T21:59:30.371 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 21 2021-03-24T21:59:30.372 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: : 2021-03-24T21:59:30.372 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 59 2021-03-24T21:59:30.372 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: : 2021-03-24T21:59:30.372 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 30 2021-03-24T21:59:30.373 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: . 2021-03-24T21:59:30.373 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 203547 2021-03-24T21:59:30.373 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: +0000 2021-03-24T21:59:30.373 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 2021-03-24T21:59:30.373 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: mon.a 2021-03-24T21:59:30.374 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: ( 2021-03-24T21:59:30.374 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: mon 2021-03-24T21:59:30.374 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: . 2021-03-24T21:59:30.374 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 0 2021-03-24T21:59:30.374 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: ) 2021-03-24T21:59:30.375 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 1191 2021-03-24T21:59:30.375 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: : 2021-03-24T21:59:30.375 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: cluster 2021-03-24T21:59:30.375 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 2021-03-24T21:59:30.376 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: [WRN] 2021-03-24T21:59:30.376 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 2021-03-24T21:59:30.376 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: Health check update: Degraded data redundancy: 264/1431 objects degraded (18.449%), 38 pgs degraded (PG_DEGRADED) 2021-03-24T21:59:30.376 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 2021-03-24T21:59:30.376 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: cluster 2021-03-24T21:59:30.377 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 2021-03-24T21:59:30.377 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 2021 2021-03-24T21:59:30.377 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: - 2021-03-24T21:59:30.377 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 03 2021-03-24T21:59:30.378 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: - 2021-03-24T21:59:30.378 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 24 2021-03-24T21:59:30.378 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: T 2021-03-24T21:59:30.378 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 21 2021-03-24T21:59:30.378 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: : 2021-03-24T21:59:30.379 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 59 2021-03-24T21:59:30.379 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: : 2021-03-24T21:59:30.379 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 30 2021-03-24T21:59:30.379 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: . 2021-03-24T21:59:30.379 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 204445 2021-03-24T21:59:30.380 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: +0000 2021-03-24T21:59:30.380 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 2021-03-24T21:59:30.406 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:30.473 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:30.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:30.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:59:30.513 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 2021-03-24T21:59:30.513 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 2021 2021-03-24T21:59:30.513 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: - 2021-03-24T21:59:30.514 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 03 2021-03-24T21:59:30.514 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: - 2021-03-24T21:59:30.514 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 24 2021-03-24T21:59:30.514 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: T 2021-03-24T21:59:30.514 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 21 2021-03-24T21:59:30.515 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: : 2021-03-24T21:59:30.515 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 59 2021-03-24T21:59:30.515 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: : 2021-03-24T21:59:30.515 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 28 2021-03-24T21:59:30.516 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: . 2021-03-24T21:59:30.516 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 828143 2021-03-24T21:59:30.516 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: +0000 2021-03-24T21:59:30.516 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 2021-03-24T21:59:30.516 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: mgr.y 2021-03-24T21:59:30.517 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: ( 2021-03-24T21:59:30.517 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: mgr 2021-03-24T21:59:30.517 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: . 2021-03-24T21:59:30.517 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 14144 2021-03-24T21:59:30.518 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: ) 2021-03-24T21:59:30.518 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 470 2021-03-24T21:59:30.518 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: : 2021-03-24T21:59:30.518 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: cluster 2021-03-24T21:59:30.518 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 2021-03-24T21:59:30.519 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: [DBG] 2021-03-24T21:59:30.519 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 2021-03-24T21:59:30.519 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: pgmap v760: 178 pgs: 4 active+clean+remapped, 52 active+undersized, 38 active+undersized+degraded, 84 active+clean; 339 MiB data, 829 MiB used, 536 GiB / 536 GiB avail; 264/1431 objects degraded (18.449%); 1/1431 objects misplaced (0.070%) 2021-03-24T21:59:30.519 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 2021-03-24T21:59:30.520 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: cluster 2021-03-24T21:59:30.520 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 2021-03-24T21:59:30.520 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 2021 2021-03-24T21:59:30.520 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: - 2021-03-24T21:59:30.521 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 03 2021-03-24T21:59:30.521 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: - 2021-03-24T21:59:30.521 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 24 2021-03-24T21:59:30.521 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: T 2021-03-24T21:59:30.521 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 21 2021-03-24T21:59:30.522 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: : 2021-03-24T21:59:30.522 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 59 2021-03-24T21:59:30.522 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: : 2021-03-24T21:59:30.522 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 29 2021-03-24T21:59:30.523 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: . 2021-03-24T21:59:30.523 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 310624 2021-03-24T21:59:30.523 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: +0000 2021-03-24T21:59:30.523 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 2021-03-24T21:59:30.523 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: mon.a 2021-03-24T21:59:30.524 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: ( 2021-03-24T21:59:30.524 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: mon 2021-03-24T21:59:30.524 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: . 2021-03-24T21:59:30.524 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 0 2021-03-24T21:59:30.524 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: ) 2021-03-24T21:59:30.525 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 1189 2021-03-24T21:59:30.525 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: : 2021-03-24T21:59:30.525 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: cluster 2021-03-24T21:59:30.525 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 2021-03-24T21:59:30.526 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: [DBG] 2021-03-24T21:59:30.526 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 2021-03-24T21:59:30.526 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: osdmap e399: 8 total, 7 up, 6 in 2021-03-24T21:59:30.526 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 2021-03-24T21:59:30.526 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: audit 2021-03-24T21:59:30.527 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 2021-03-24T21:59:30.527 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 2021 2021-03-24T21:59:30.527 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: - 2021-03-24T21:59:30.527 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 03 2021-03-24T21:59:30.528 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: - 2021-03-24T21:59:30.528 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 24 2021-03-24T21:59:30.528 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: T 2021-03-24T21:59:30.528 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 21 2021-03-24T21:59:30.528 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: : 2021-03-24T21:59:30.529 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 59 2021-03-24T21:59:30.529 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: : 2021-03-24T21:59:30.529 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 29 2021-03-24T21:59:30.529 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: . 2021-03-24T21:59:30.530 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 974208 2021-03-24T21:59:30.530 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: +0000 2021-03-24T21:59:30.530 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 2021-03-24T21:59:30.530 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: mon.c 2021-03-24T21:59:30.531 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: ( 2021-03-24T21:59:30.531 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: mon 2021-03-24T21:59:30.531 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: . 2021-03-24T21:59:30.531 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 2 2021-03-24T21:59:30.532 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: ) 2021-03-24T21:59:30.532 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 62 2021-03-24T21:59:30.532 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: : 2021-03-24T21:59:30.533 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: mon.a 2021-03-24T21:59:30.533 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: ( 2021-03-24T21:59:30.533 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: mon 2021-03-24T21:59:30.533 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: . 2021-03-24T21:59:30.534 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 0 2021-03-24T21:59:30.534 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: ) 2021-03-24T21:59:30.534 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 1192 2021-03-24T21:59:30.534 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: : 2021-03-24T21:59:30.535 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: cluster 2021-03-24T21:59:30.535 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 2021-03-24T21:59:30.535 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: [WRN] 2021-03-24T21:59:30.536 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 2021-03-24T21:59:30.536 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:59:30.537 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 2021-03-24T21:59:30.537 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: audit 2021-03-24T21:59:30.537 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 2021-03-24T21:59:30.538 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 2021 2021-03-24T21:59:30.538 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: - 2021-03-24T21:59:30.539 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 03 2021-03-24T21:59:30.539 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: - 2021-03-24T21:59:30.540 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 24 2021-03-24T21:59:30.540 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: T 2021-03-24T21:59:30.540 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 21 2021-03-24T21:59:30.540 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: : 2021-03-24T21:59:30.541 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 59 2021-03-24T21:59:30.541 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: : 2021-03-24T21:59:30.541 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 30 2021-03-24T21:59:30.541 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: . 2021-03-24T21:59:30.542 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 215245 2021-03-24T21:59:30.542 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: +0000 2021-03-24T21:59:30.542 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 2021-03-24T21:59:30.542 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: mon.a 2021-03-24T21:59:30.543 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: ( 2021-03-24T21:59:30.543 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: mon 2021-03-24T21:59:30.543 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: . 2021-03-24T21:59:30.543 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 0 2021-03-24T21:59:30.544 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: ) 2021-03-24T21:59:30.544 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 1193 2021-03-24T21:59:30.544 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: : 2021-03-24T21:59:30.544 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: audit 2021-03-24T21:59:30.545 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 2021-03-24T21:59:30.545 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: [INF] 2021-03-24T21:59:30.545 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 2021-03-24T21:59:30.545 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: from='client.35180 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T21:59:30.546 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 2021-03-24T21:59:30.546 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: cluster 2021-03-24T21:59:30.546 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 2021-03-24T21:59:30.546 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 2021 2021-03-24T21:59:30.547 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: - 2021-03-24T21:59:30.547 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 03 2021-03-24T21:59:30.547 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: - 2021-03-24T21:59:30.547 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 24 2021-03-24T21:59:30.547 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: T 2021-03-24T21:59:30.548 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 21 2021-03-24T21:59:30.548 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: : 2021-03-24T21:59:30.548 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 59 2021-03-24T21:59:30.548 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: : 2021-03-24T21:59:30.549 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 30 2021-03-24T21:59:30.549 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: . 2021-03-24T21:59:30.549 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 215480 2021-03-24T21:59:30.549 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: +0000 2021-03-24T21:59:30.549 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 2021-03-24T21:59:30.550 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: mon.a 2021-03-24T21:59:30.550 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: ( 2021-03-24T21:59:30.550 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: mon 2021-03-24T21:59:30.550 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: . 2021-03-24T21:59:30.551 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 0 2021-03-24T21:59:30.551 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: ) 2021-03-24T21:59:30.551 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 1194 2021-03-24T21:59:30.551 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: : 2021-03-24T21:59:30.551 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: cluster 2021-03-24T21:59:30.552 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 2021-03-24T21:59:30.552 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: [DBG] 2021-03-24T21:59:30.552 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 2021-03-24T21:59:30.552 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: osdmap e400: 8 total, 7 up, 6 in 2021-03-24T21:59:30.553 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[24927]: 2021-03-24T21:59:30.553 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: audit 2021-03-24T21:59:30.554 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 2021-03-24T21:59:30.554 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: [INF] 2021-03-24T21:59:30.554 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 2021-03-24T21:59:30.555 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: from='client.? v1:172.21.15.125:0/2059877721' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:59:30.555 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 2021-03-24T21:59:30.555 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: audit 2021-03-24T21:59:30.555 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 2021-03-24T21:59:30.556 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 2021 2021-03-24T21:59:30.556 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: - 2021-03-24T21:59:30.556 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 03 2021-03-24T21:59:30.556 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: - 2021-03-24T21:59:30.556 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 24 2021-03-24T21:59:30.557 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: T 2021-03-24T21:59:30.557 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 21 2021-03-24T21:59:30.557 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: : 2021-03-24T21:59:30.557 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 59 2021-03-24T21:59:30.558 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: : 2021-03-24T21:59:30.558 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 29 2021-03-24T21:59:30.558 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: . 2021-03-24T21:59:30.558 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 974606 2021-03-24T21:59:30.559 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: +0000 2021-03-24T21:59:30.559 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 2021-03-24T21:59:30.559 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: mon.a 2021-03-24T21:59:30.559 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: ( 2021-03-24T21:59:30.559 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: mon 2021-03-24T21:59:30.560 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: . 2021-03-24T21:59:30.560 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 0 2021-03-24T21:59:30.560 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: ) 2021-03-24T21:59:30.560 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 1190 2021-03-24T21:59:30.561 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: : 2021-03-24T21:59:30.561 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: audit 2021-03-24T21:59:30.561 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 2021-03-24T21:59:30.561 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: [INF] 2021-03-24T21:59:30.561 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 2021-03-24T21:59:30.562 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: from='client.35180 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:59:30.562 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 2021-03-24T21:59:30.562 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: cluster 2021-03-24T21:59:30.562 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 2021-03-24T21:59:30.563 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 2021 2021-03-24T21:59:30.563 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: - 2021-03-24T21:59:30.563 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 03 2021-03-24T21:59:30.563 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: - 2021-03-24T21:59:30.563 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 24 2021-03-24T21:59:30.564 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: T 2021-03-24T21:59:30.564 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 21 2021-03-24T21:59:30.564 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: : 2021-03-24T21:59:30.564 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 59 2021-03-24T21:59:30.564 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: : 2021-03-24T21:59:30.565 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 30 2021-03-24T21:59:30.565 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: . 2021-03-24T21:59:30.565 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 203547 2021-03-24T21:59:30.565 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: +0000 2021-03-24T21:59:30.566 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 2021-03-24T21:59:30.566 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: mon.a 2021-03-24T21:59:30.566 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: ( 2021-03-24T21:59:30.566 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: mon 2021-03-24T21:59:30.566 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: . 2021-03-24T21:59:30.567 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 0 2021-03-24T21:59:30.567 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: ) 2021-03-24T21:59:30.567 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 1191 2021-03-24T21:59:30.568 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: : 2021-03-24T21:59:30.568 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: cluster 2021-03-24T21:59:30.568 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 2021-03-24T21:59:30.569 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: [WRN] 2021-03-24T21:59:30.569 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 2021-03-24T21:59:30.569 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: Health check update: Degraded data redundancy: 264/1431 objects degraded (18.449%), 38 pgs degraded (PG_DEGRADED) 2021-03-24T21:59:30.570 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 2021-03-24T21:59:30.570 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: cluster 2021-03-24T21:59:30.571 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 2021-03-24T21:59:30.571 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 2021 2021-03-24T21:59:30.572 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: - 2021-03-24T21:59:30.572 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 03 2021-03-24T21:59:30.573 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: - 2021-03-24T21:59:30.573 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 24 2021-03-24T21:59:30.573 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: T 2021-03-24T21:59:30.574 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 21 2021-03-24T21:59:30.574 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: : 2021-03-24T21:59:30.575 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 59 2021-03-24T21:59:30.575 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: : 2021-03-24T21:59:30.575 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 30 2021-03-24T21:59:30.576 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: . 2021-03-24T21:59:30.576 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 204445 2021-03-24T21:59:30.577 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: +0000 2021-03-24T21:59:30.577 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 2021-03-24T21:59:30.577 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: mon.a 2021-03-24T21:59:30.577 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: ( 2021-03-24T21:59:30.578 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: mon 2021-03-24T21:59:30.578 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: . 2021-03-24T21:59:30.578 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 0 2021-03-24T21:59:30.578 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: ) 2021-03-24T21:59:30.579 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 1192 2021-03-24T21:59:30.579 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: : 2021-03-24T21:59:30.579 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: cluster 2021-03-24T21:59:30.579 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 2021-03-24T21:59:30.580 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: [WRN] 2021-03-24T21:59:30.580 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 2021-03-24T21:59:30.580 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:59:30.580 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 2021-03-24T21:59:30.580 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: audit 2021-03-24T21:59:30.581 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 2021-03-24T21:59:30.581 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 2021 2021-03-24T21:59:30.581 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: - 2021-03-24T21:59:30.581 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 03 2021-03-24T21:59:30.582 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: - 2021-03-24T21:59:30.582 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 24 2021-03-24T21:59:30.582 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: T 2021-03-24T21:59:30.582 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 21 2021-03-24T21:59:30.583 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: : 2021-03-24T21:59:30.583 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 59 2021-03-24T21:59:30.583 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: : 2021-03-24T21:59:30.583 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 30 2021-03-24T21:59:30.583 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: . 2021-03-24T21:59:30.584 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 215245 2021-03-24T21:59:30.584 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: +0000 2021-03-24T21:59:30.584 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 2021-03-24T21:59:30.584 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: mon.a 2021-03-24T21:59:30.584 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: ( 2021-03-24T21:59:30.585 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: mon 2021-03-24T21:59:30.585 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: . 2021-03-24T21:59:30.585 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 0 2021-03-24T21:59:30.585 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: ) 2021-03-24T21:59:30.586 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 1193 2021-03-24T21:59:30.586 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: : 2021-03-24T21:59:30.586 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: audit 2021-03-24T21:59:30.586 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 2021-03-24T21:59:30.587 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: [INF] 2021-03-24T21:59:30.587 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 2021-03-24T21:59:30.587 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: from='client.35180 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T21:59:30.587 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 2021-03-24T21:59:30.588 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: cluster 2021-03-24T21:59:30.588 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 2021-03-24T21:59:30.588 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 2021 2021-03-24T21:59:30.588 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: - 2021-03-24T21:59:30.588 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 03 2021-03-24T21:59:30.589 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: - 2021-03-24T21:59:30.589 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 24 2021-03-24T21:59:30.589 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: T 2021-03-24T21:59:30.589 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 21 2021-03-24T21:59:30.590 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: : 2021-03-24T21:59:30.590 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 59 2021-03-24T21:59:30.590 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: : 2021-03-24T21:59:30.590 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 30 2021-03-24T21:59:30.591 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: . 2021-03-24T21:59:30.591 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 215480 2021-03-24T21:59:30.591 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: +0000 2021-03-24T21:59:30.591 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 2021-03-24T21:59:30.592 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: mon.a 2021-03-24T21:59:30.592 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: ( 2021-03-24T21:59:30.592 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: mon 2021-03-24T21:59:30.592 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: . 2021-03-24T21:59:30.593 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 0 2021-03-24T21:59:30.593 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: ) 2021-03-24T21:59:30.593 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 1194 2021-03-24T21:59:30.593 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: : 2021-03-24T21:59:30.594 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: cluster 2021-03-24T21:59:30.594 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 2021-03-24T21:59:30.594 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: [DBG] 2021-03-24T21:59:30.594 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 2021-03-24T21:59:30.595 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: osdmap e400: 8 total, 7 up, 6 in 2021-03-24T21:59:30.595 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:30 smithi125 conmon[19936]: 2021-03-24T21:59:30.595 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:30.596 INFO:tasks.rados.rados.0.smithi125.stdout:1473: expect (ObjNum 503 snap 146 seq_num 503) 2021-03-24T21:59:30.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:30 smithi142 conmon[20125]: cluster 2021-03-24T21:59:28. 2021-03-24T21:59:30.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:30 smithi142 conmon[20125]: 828143+0000 mgr.y (mgr.14144) 470 : cluster [DBG] pgmap v760: 178 pgs: 4 active+clean+remapped, 52 active+undersized, 38 active+undersized+degraded, 84 active+clean; 339 MiB data, 829 MiB used, 536 GiB / 536 GiB avail; 264/1431 objects degraded (18.449%); 1/1431 objects misplaced (0.070%) 2021-03-24T21:59:30.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:30 smithi142 conmon[20125]: cluster 2021-03-24T21:59:29. 2021-03-24T21:59:30.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:30 smithi142 conmon[20125]: 310624+0000 mon.a (mon.0) 1189 : cluster [DBG] osdmap e399: 8 total, 7 up, 6 in 2021-03-24T21:59:30.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:30 smithi142 conmon[20125]: audit 2021-03-24T 2021-03-24T21:59:30.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:30 smithi142 conmon[20125]: 21:59:29.974208+0000 mon.c (mon 2021-03-24T21:59:30.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:30 smithi142 conmon[20125]: .2) 62 : audit [INF] from='client.? v1:172.21.15.125:0/2059877721' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:59:30.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:30 smithi142 conmon[20125]: audit 2021-03-24 2021-03-24T21:59:30.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:30 smithi142 conmon[20125]: T21:59:29.974606+0000 mon.a ( 2021-03-24T21:59:30.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:30 smithi142 conmon[20125]: mon.0) 1190 : audit [INF] from='client.35180 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:59:30.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:30 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:59:30.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:30 smithi142 conmon[20125]: T21:59:30.203547+0000 mon.a ( 2021-03-24T21:59:30.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:30 smithi142 conmon[20125]: mon.0) 1191 : cluster [WRN] Health check update: Degraded data redundancy: 264/1431 objects degraded (18.449%), 38 pgs degraded (PG_DEGRADED) 2021-03-24T21:59:30.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:30 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:59:30.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:30 smithi142 conmon[20125]: 03-24T21:59:30.204445+0000 2021-03-24T21:59:30.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:30 smithi142 conmon[20125]: mon.a (mon.0) 1192 : cluster [WRN] 2021-03-24T21:59:30.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:30 smithi142 conmon[20125]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:59:30.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:30 smithi142 conmon[20125]: audit 2021-03-24T21:59:30.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:30 smithi142 conmon[20125]: 2021-03-24T21:59:30.215245+0000 mon.a 2021-03-24T21:59:30.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:30 smithi142 conmon[20125]: (mon.0) 1193 : audit [INF] from='client.35180 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T21:59:30.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:30 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:59:30.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:30 smithi142 conmon[20125]: 03-24T21:59:30.215480+0000 2021-03-24T21:59:30.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:30 smithi142 conmon[20125]: mon.a (mon.0) 1194 : cluster [DBG] 2021-03-24T21:59:30.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:30 smithi142 conmon[20125]: osdmap e400: 8 total, 7 up, 6 in 2021-03-24T21:59:30.637 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:30.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:30.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:59:30.742 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:30.776 INFO:tasks.rados.rados.0.smithi125.stdout:1472: expect (ObjNum 500 snap 146 seq_num 500) 2021-03-24T21:59:30.796 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:30.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:30.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:30.903 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:30.928 INFO:tasks.rados.rados.0.smithi125.stdout:1474: finishing rollback tid 0 to smithi12556470-14 2021-03-24T21:59:30.929 INFO:tasks.rados.rados.0.smithi125.stdout:1474: finishing rollback tid 1 to smithi12556470-14 2021-03-24T21:59:30.929 INFO:tasks.rados.rados.0.smithi125.stdout:1475: finishing write tid 1 to smithi12556470-17 2021-03-24T21:59:30.929 INFO:tasks.rados.rados.0.smithi125.stdout:1471: finishing copy_from to smithi12556470-6 2021-03-24T21:59:30.929 INFO:tasks.rados.rados.0.smithi125.stdout:1471: got expected ENOENT (src dne) 2021-03-24T21:59:30.930 INFO:tasks.rados.rados.0.smithi125.stdout:1471: finishing copy_from racing read to smithi12556470-6 2021-03-24T21:59:30.930 INFO:tasks.rados.rados.0.smithi125.stdout:1475: finishing write tid 2 to smithi12556470-17 2021-03-24T21:59:30.930 INFO:tasks.rados.rados.0.smithi125.stdout:1475: finishing write tid 3 to smithi12556470-17 2021-03-24T21:59:30.931 INFO:tasks.rados.rados.0.smithi125.stdout:1475: finishing write tid 4 to smithi12556470-17 2021-03-24T21:59:30.931 INFO:tasks.rados.rados.0.smithi125.stdout:1475: finishing write tid 5 to smithi12556470-17 2021-03-24T21:59:30.931 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 17 v 391 (ObjNum 528 snap 155 seq_num 528) dirty exists 2021-03-24T21:59:30.931 INFO:tasks.rados.rados.0.smithi125.stdout:1475: left oid 17 (ObjNum 528 snap 155 seq_num 528) 2021-03-24T21:59:30.932 INFO:tasks.rados.rados.0.smithi125.stdout:1470: finishing write tid 1 to smithi12556470-7 2021-03-24T21:59:30.932 INFO:tasks.rados.rados.0.smithi125.stdout:1470: finishing write tid 2 to smithi12556470-7 2021-03-24T21:59:30.932 INFO:tasks.rados.rados.0.smithi125.stdout:1470: finishing write tid 3 to smithi12556470-7 2021-03-24T21:59:30.933 INFO:tasks.rados.rados.0.smithi125.stdout:1470: finishing write tid 4 to smithi12556470-7 2021-03-24T21:59:30.933 INFO:tasks.rados.rados.0.smithi125.stdout:1470: finishing write tid 5 to smithi12556470-7 2021-03-24T21:59:30.934 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 7 v 438 (ObjNum 527 snap 155 seq_num 527) dirty exists 2021-03-24T21:59:30.934 INFO:tasks.rados.rados.0.smithi125.stdout:1470: left oid 7 (ObjNum 527 snap 155 seq_num 527) 2021-03-24T21:59:30.934 INFO:tasks.rados.rados.0.smithi125.stdout:1474: finishing rollback tid 2 to smithi12556470-14 2021-03-24T21:59:30.935 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 14 v 547 (ObjNum 455 snap 128 seq_num 455) dirty exists 2021-03-24T21:59:30.935 INFO:tasks.rados.rados.0.smithi125.stdout:1476: finishing copy_from to smithi12556470-32 2021-03-24T21:59:30.936 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 32 v 279 (ObjNum 427 snap 113 seq_num 427) dirty exists 2021-03-24T21:59:30.936 INFO:tasks.rados.rados.0.smithi125.stdout:1476: finishing copy_from racing read to smithi12556470-32 2021-03-24T21:59:30.937 INFO:tasks.rados.rados.0.smithi125.stdout:1469: done (8 left) 2021-03-24T21:59:30.937 INFO:tasks.rados.rados.0.smithi125.stdout:1470: done (7 left) 2021-03-24T21:59:30.937 INFO:tasks.rados.rados.0.smithi125.stdout:1471: done (6 left) 2021-03-24T21:59:30.938 INFO:tasks.rados.rados.0.smithi125.stdout:1472: done (5 left) 2021-03-24T21:59:30.938 INFO:tasks.rados.rados.0.smithi125.stdout:1473: done (4 left) 2021-03-24T21:59:30.939 INFO:tasks.rados.rados.0.smithi125.stdout:1474: done (3 left) 2021-03-24T21:59:30.939 INFO:tasks.rados.rados.0.smithi125.stdout:1475: done (2 left) 2021-03-24T21:59:30.939 INFO:tasks.rados.rados.0.smithi125.stdout:1476: done (1 left) 2021-03-24T21:59:30.940 INFO:tasks.rados.rados.0.smithi125.stdout:1477: done (0 left) 2021-03-24T21:59:30.940 INFO:tasks.rados.rados.0.smithi125.stdout:1478: snap_create 2021-03-24T21:59:30.968 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:30.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:30.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:59:31.072 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:31.131 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:31.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:31.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:31.238 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:31.287 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:31.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:31.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:59:31.339 INFO:tasks.rados.rados.0.smithi125.stdout:1478: done (0 left) 2021-03-24T21:59:31.339 INFO:tasks.rados.rados.0.smithi125.stdout:1479: rollback oid 30 current snap is 156 2021-03-24T21:59:31.339 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 30 to 128 2021-03-24T21:59:31.340 INFO:tasks.rados.rados.0.smithi125.stdout:1480: snap_create 2021-03-24T21:59:31.347 INFO:tasks.rados.rados.0.smithi125.stdout:1479: finishing rollback tid 1 to smithi12556470-30 2021-03-24T21:59:31.347 INFO:tasks.rados.rados.0.smithi125.stdout:1479: finishing rollback tid 2 to smithi12556470-30 2021-03-24T21:59:31.347 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 30 v 287 (ObjNum 447 snap 122 seq_num 447) dirty exists 2021-03-24T21:59:31.392 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:31.441 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:31.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:31.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:59:31.549 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:31.600 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:31.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:31.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:31.705 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:31.757 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:31.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:31.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:59:31.862 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:31.912 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:31.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:31.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:32.018 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:32.073 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:32.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:32.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:32.178 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:32.237 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:32.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:32.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:59:32.342 INFO:tasks.rados.rados.0.smithi125.stdout:1479: done (1 left) 2021-03-24T21:59:32.342 INFO:tasks.rados.rados.0.smithi125.stdout:1480: done (0 left) 2021-03-24T21:59:32.342 INFO:tasks.rados.rados.0.smithi125.stdout:1481: read oid 34 snap -1 2021-03-24T21:59:32.343 INFO:tasks.rados.rados.0.smithi125.stdout:1481: expect (ObjNum 503 snap 146 seq_num 503) 2021-03-24T21:59:32.343 INFO:tasks.rados.rados.0.smithi125.stdout:1482: write (excl) oid 12 current snap is 157 2021-03-24T21:59:32.343 INFO:tasks.rados.rados.0.smithi125.stdout:1482: seq_num 530 ranges {507710=710327,1897927=412519,2854892=52818} 2021-03-24T21:59:32.344 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:32.351 INFO:tasks.rados.rados.0.smithi125.stdout:1482: writing smithi12556470-12 from 507710 to 1218037 tid 1 2021-03-24T21:59:32.353 INFO:teuthology.orchestra.run.smithi125.stderr:set pool 4 pg_num to 22 2021-03-24T21:59:32.358 INFO:tasks.rados.rados.0.smithi125.stdout:1482: writing smithi12556470-12 from 1897927 to 2310446 tid 2 2021-03-24T21:59:32.362 INFO:tasks.rados.rados.0.smithi125.stdout:1482: writing smithi12556470-12 from 2854892 to 2907710 tid 3 2021-03-24T21:59:32.363 INFO:tasks.rados.rados.0.smithi125.stdout:1483: delete oid 17 current snap is 157 2021-03-24T21:59:32.363 INFO:tasks.rados.rados.0.smithi125.stdout:1481: expect (ObjNum 503 snap 146 seq_num 503) 2021-03-24T21:59:32.417 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:32.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:32.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:59:32.438 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:32 smithi125 conmon[19936]: cluster 2021-03-24T21:59:30.829129+0000 mgr.y (mgr.14144) 471 : cluster [DBG] 2021-03-24T21:59:32.439 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:32 smithi125 conmon[19936]: pgmap v763: 178 pgs: 4 active+clean+remapped, 52 active+undersized, 38 active+undersized+degraded, 84 active+clean; 347 MiB data, 827 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 2.1 MiB/s wr, 6 op/s; 268/1446 objects degraded (18.534%); 1/1446 objects misplaced (0.069%) 2021-03-24T21:59:32.439 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:32 smithi125 conmon[19936]: cluster 2021-03-24T21:59:31.337095+0000 mon.a (mon.0) 1195 : cluster [DBG] osdmap e401: 8 total, 7 up, 6 in 2021-03-24T21:59:32.440 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:32 smithi125 conmon[19936]: audit 2021-03-24T21:59 2021-03-24T21:59:32.440 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:32 smithi125 conmon[19936]: :31.624923+0000 mon.a (mon.0) 1196 : audit [INF] from='client.25168 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num", "val": "22"}]: dispatch 2021-03-24T21:59:32.440 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:32 smithi125 conmon[19936]: audit 2021-03-24T21:59: 2021-03-24T21:59:32.440 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:32 smithi125 conmon[19936]: 31.626392+0000 mon.b (mon.1) 34 : audit [INF] 2021-03-24T21:59:32.440 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:32 smithi125 conmon[19936]: from='client.? v1:172.21.15.125:0/4256720089' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num", "val": "22"}]: dispatch 2021-03-24T21:59:32.441 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:32 smithi125 conmon[24927]: cluster 2021-03-24T21:59: 2021-03-24T21:59:32.441 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:32 smithi125 conmon[24927]: 30.829129+0000 mgr.y (mgr.14144) 471 : cluster [DBG] pgmap v763: 178 pgs: 4 active+clean+remapped, 52 active+undersized, 38 active+undersized+degraded, 84 active+clean; 347 MiB data, 827 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 2.1 MiB/s wr, 6 op/s; 268/1446 objects degraded (18.534%); 1/1446 objects misplaced (0.069%) 2021-03-24T21:59:32.441 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:32 smithi125 conmon[24927]: cluster 2021-03-24T21:59:31.337095+0000 mon.a (mon 2021-03-24T21:59:32.442 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:32 smithi125 conmon[24927]: .0) 1195 : cluster [DBG] osdmap e401: 8 total, 7 up, 6 in 2021-03-24T21:59:32.442 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:32 smithi125 conmon[24927]: audit 2021-03-24T21:59:32.442 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:32 smithi125 conmon[24927]: 2021-03-24T21:59:31.624923+0000 mon.a (mon.0) 1196 2021-03-24T21:59:32.442 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:32 smithi125 conmon[24927]: : audit [INF] from='client.25168 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num", "val": "22"}]: dispatch 2021-03-24T21:59:32.442 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:32 smithi125 conmon[24927]: audit 2021- 2021-03-24T21:59:32.443 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:32 smithi125 conmon[24927]: 03-24T21:59:31.626392+0000 mon.b (mon.1) 34 : audit 2021-03-24T21:59:32.443 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:32 smithi125 conmon[24927]: [INF] from='client.? v1:172.21.15.125:0/4256720089' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num", "val": "22"}]: dispatch 2021-03-24T21:59:32.522 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:32.529 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:32 smithi142 conmon[20125]: cluster 2021-03-24T21:59:32.530 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:32 smithi142 conmon[20125]: 2021-03-24T21:59:32.530 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:32 smithi142 conmon[20125]: 2021 2021-03-24T21:59:32.530 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:32 smithi142 conmon[20125]: - 2021-03-24T21:59:32.530 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:32 smithi142 conmon[20125]: 03- 2021-03-24T21:59:32.531 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:32 smithi142 conmon[20125]: 24T21:59:30.829129+0000 mgr.y (mgr.14144) 471 : cluster [DBG] pgmap v763: 178 pgs: 4 active+clean+remapped, 52 active+undersized, 38 active+undersized+degraded, 84 active+clean; 347 MiB data, 827 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 2.1 MiB/s wr, 6 op/s; 268/1446 objects degraded (18.534%); 1/1446 objects misplaced (0.069%) 2021-03-24T21:59:32.531 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:32 smithi142 conmon[20125]: 2021-03-24T21:59:32.531 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:32 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:59:32.531 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:32 smithi142 conmon[20125]: -03-24T21:59:31.337095+0000 2021-03-24T21:59:32.532 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:32 smithi142 conmon[20125]: mon.a (mon.0) 1195 : cluster [DBG] osdmap e401: 8 total, 7 up, 6 in 2021-03-24T21:59:32.532 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:32 smithi142 conmon[20125]: 2021-03-24T21:59:32.532 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:32 smithi142 conmon[20125]: audit 2021-03-24T21:59:32.532 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:32 smithi142 conmon[20125]: 2021-03-24T21:59:31.624923+0000 mon.a (mon 2021-03-24T21:59:32.532 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:32 smithi142 conmon[20125]: .0) 1196 : audit [INF] from='client.25168 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num", "val": "22"}]: dispatch 2021-03-24T21:59:32.533 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:32 smithi142 conmon[20125]: audit 2021-03-24T 2021-03-24T21:59:32.533 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:32 smithi142 conmon[20125]: 21:59:31.626392+0000 mon.b (mon.1 2021-03-24T21:59:32.533 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:32 smithi142 conmon[20125]: ) 34 : audit [INF] from='client.? v1:172.21.15.125:0/4256720089' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num", "val": "22"}]: dispatch 2021-03-24T21:59:32.567 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:32.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:32.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:32.606 INFO:tasks.rados.rados.0.smithi125.stdout:1482: finishing write tid 1 to smithi12556470-12 2021-03-24T21:59:32.706 INFO:tasks.rados.rados.0.smithi125.stdout:1482: finishing write tid 2 to smithi12556470-12 2021-03-24T21:59:32.706 INFO:tasks.rados.rados.0.smithi125.stdout:1482: finishing write tid 3 to smithi12556470-12 2021-03-24T21:59:32.707 INFO:tasks.rados.rados.0.smithi125.stdout:1482: finishing write tid 4 to smithi12556470-12 2021-03-24T21:59:32.707 INFO:tasks.rados.rados.0.smithi125.stdout:1482: finishing write tid 5 to smithi12556470-12 2021-03-24T21:59:32.707 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 12 v 364 (ObjNum 530 snap 157 seq_num 530) dirty exists 2021-03-24T21:59:32.707 INFO:tasks.rados.rados.0.smithi125.stdout:1482: left oid 12 (ObjNum 530 snap 157 seq_num 530) 2021-03-24T21:59:32.708 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:32.710 INFO:tasks.rados.rados.0.smithi125.stdout:1481: done (2 left) 2021-03-24T21:59:32.710 INFO:tasks.rados.rados.0.smithi125.stdout:1482: done (1 left) 2021-03-24T21:59:32.710 INFO:tasks.rados.rados.0.smithi125.stdout:1483: done (0 left) 2021-03-24T21:59:32.711 INFO:tasks.rados.rados.0.smithi125.stdout:1484: read oid 19 snap 143 2021-03-24T21:59:32.711 INFO:tasks.rados.rados.0.smithi125.stdout:1484: expect deleted 2021-03-24T21:59:32.711 INFO:tasks.rados.rados.0.smithi125.stdout:1485: delete oid 17 current snap is 157 2021-03-24T21:59:32.712 INFO:tasks.rados.rados.0.smithi125.stdout:1484: done (1 left) 2021-03-24T21:59:32.712 INFO:tasks.rados.rados.0.smithi125.stdout:1485: done (0 left) 2021-03-24T21:59:32.712 INFO:tasks.rados.rados.0.smithi125.stdout:1486: snap_create 2021-03-24T21:59:32.763 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:32.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:32.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:59:32.868 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:32.929 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:32.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:32.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:59:33.037 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:33.099 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:33.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:33.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:59:33.206 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:33.260 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:33.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:33.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:59:33.366 INFO:tasks.rados.rados.0.smithi125.stdout:1486: done (0 left) 2021-03-24T21:59:33.367 INFO:tasks.rados.rados.0.smithi125.stdout:1487: snap_create 2021-03-24T21:59:33.368 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:33.529 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:33.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:33.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:33.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:33 smithi142 conmon[20125]: cluster 2021-03-24T21:59:31.831955+0000 osd.1 (osd.1) 41 : cluster 2021-03-24T21:59:33.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:33 smithi142 conmon[20125]: [DBG] purged_snaps scrub starts 2021-03-24T21:59:33.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:33 smithi142 conmon[20125]: cluster 2021-03-24T21:59:31.832106+0000 osd.1 (osd. 2021-03-24T21:59:33.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:33 smithi142 conmon[20125]: 1) 42 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:59:33.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:33 smithi142 conmon[20125]: audit 2021-03-24T21:59 2021-03-24T21:59:33.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:33 smithi142 conmon[20125]: :32.339444+0000 mon.a (mon.0) 1197 : 2021-03-24T21:59:33.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:33 smithi142 conmon[20125]: audit [INF] from='client.25168 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num", "val": "22"}]': finished 2021-03-24T21:59:33.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:33 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:59:33.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:33 smithi142 conmon[20125]: 59:32.339580+0000 mon.a (mon.0) 2021-03-24T21:59:33.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:33 smithi142 conmon[20125]: 1198 : cluster [DBG] osdmap e402: 8 total, 7 up, 6 in 2021-03-24T21:59:33.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:33 smithi142 conmon[20125]: audit 2021-03- 2021-03-24T21:59:33.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:33 smithi142 conmon[20125]: 24T21:59:32.832171+0000 mon.a ( 2021-03-24T21:59:33.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:33 smithi142 conmon[20125]: mon.0) 1199 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pgp_num_actual", "val": "22"}]: dispatch 2021-03-24T21:59:33.639 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:33.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:33 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:59:33.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:33 smithi125 conmon[24927]: 59:31.831955+0000 osd.1 (osd.1) 41 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:59:33.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:33 smithi125 conmon[24927]: cluster 2021-03-24T21:59 2021-03-24T21:59:33.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:33 smithi125 conmon[24927]: :31.832106+0000 osd.1 (osd.1) 42 : 2021-03-24T21:59:33.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:33 smithi125 conmon[24927]: cluster [DBG] purged_snaps scrub ok 2021-03-24T21:59:33.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:33 smithi125 conmon[24927]: audit 2021-03 2021-03-24T21:59:33.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:33 smithi125 conmon[24927]: -24T21:59:32.339444+0000 mon.a ( 2021-03-24T21:59:33.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:33 smithi125 conmon[24927]: mon.0) 1197 : audit [INF] from='client.25168 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num", "val": "22"}]': finished 2021-03-24T21:59:33.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:33 smithi125 conmon[24927]: cluster 2021-03-24T21:59:33.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:33 smithi125 conmon[24927]: 2021-03-24T21:59:32.339580+0000 2021-03-24T21:59:33.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:33 smithi125 conmon[24927]: mon.a (mon.0) 1198 : cluster [DBG] osdmap e402: 8 total, 7 up, 6 in 2021-03-24T21:59:33.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:33 smithi125 conmon[24927]: 2021-03-24T21:59:33.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:33 smithi125 conmon[24927]: audit 2021-03-24T21:59:32 2021-03-24T21:59:33.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:33 smithi125 conmon[24927]: .832171+0000 mon.a (mon.0) 1199 : audit 2021-03-24T21:59:33.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:33 smithi125 conmon[24927]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pgp_num_actual", "val": "22"}]: dispatch 2021-03-24T21:59:33.657 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:33 smithi125 conmon[19936]: cluster 2021-03-24T21:59: 2021-03-24T21:59:33.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:33 smithi125 conmon[19936]: 31.831955+0000 osd.1 (osd.1) 41 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:59:33.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:33 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:59:33.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:33 smithi125 conmon[19936]: 21:59:31.832106+0000 2021-03-24T21:59:33.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:33 smithi125 conmon[19936]: osd.1 (osd.1) 42 : 2021-03-24T21:59:33.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:33 smithi125 conmon[19936]: cluster [DBG] purged_snaps scrub ok 2021-03-24T21:59:33.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:33 smithi125 conmon[19936]: audit 2021-03-24T21:59:33.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:33 smithi125 conmon[19936]: 2021-03 2021-03-24T21:59:33.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:33 smithi125 conmon[19936]: -24T21:59:32.339444+0000 mon.a 2021-03-24T21:59:33.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:33 smithi125 conmon[19936]: (mon.0) 1197 : audit 2021-03-24T21:59:33.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:33 smithi125 conmon[19936]: [INF] from='client.25168 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num", "val": "22"}]': finished 2021-03-24T21:59:33.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:33 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:59:33.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:33 smithi125 conmon[19936]: -03-24T21:59:32 2021-03-24T21:59:33.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:33 smithi125 conmon[19936]: .339580+0000 mon.a (mon. 2021-03-24T21:59:33.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:33 smithi125 conmon[19936]: 0) 1198 : cluster [DBG] osdmap e402: 8 total, 7 up, 6 in 2021-03-24T21:59:33.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:33 smithi125 conmon[19936]: 2021-03-24T21:59:33.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:33 smithi125 conmon[19936]: audit 2021 2021-03-24T21:59:33.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:33 smithi125 conmon[19936]: -03-24T21:59: 2021-03-24T21:59:33.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:33 smithi125 conmon[19936]: 32.832171+0000 mon.a (mon 2021-03-24T21:59:33.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:33 smithi125 conmon[19936]: .0) 1199 : audit [INF] 2021-03-24T21:59:33.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:33 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pgp_num_actual", "val": "22"}]: dispatch 2021-03-24T21:59:33.691 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:33.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:33.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:59:33.798 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:33.856 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:33.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:33.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:59:33.962 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:34.017 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:34.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:34.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:59:34.124 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:34.183 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:34.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:34.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:59:34.289 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:34.347 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:34.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:34.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:59:34.369 INFO:tasks.rados.rados.0.smithi125.stdout:1487: done (0 left) 2021-03-24T21:59:34.369 INFO:tasks.rados.rados.0.smithi125.stdout:1488: read oid 31 snap -1 2021-03-24T21:59:34.370 INFO:tasks.rados.rados.0.smithi125.stdout:1488: expect (ObjNum 517 snap 154 seq_num 517) 2021-03-24T21:59:34.370 INFO:tasks.rados.rados.0.smithi125.stdout:1489: copy_from oid 2 from oid 26 current snap is 159 2021-03-24T21:59:34.370 INFO:tasks.rados.rados.0.smithi125.stdout:1490: read oid 3 snap -1 2021-03-24T21:59:34.370 INFO:tasks.rados.rados.0.smithi125.stdout:1490: expect deleted 2021-03-24T21:59:34.371 INFO:tasks.rados.rados.0.smithi125.stdout:1491: read oid 46 snap -1 2021-03-24T21:59:34.371 INFO:tasks.rados.rados.0.smithi125.stdout:1491: expect (ObjNum 471 snap 133 seq_num 471) 2021-03-24T21:59:34.371 INFO:tasks.rados.rados.0.smithi125.stdout:1492: rollback oid 29 current snap is 159 2021-03-24T21:59:34.371 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 29 to 136 2021-03-24T21:59:34.372 INFO:tasks.rados.rados.0.smithi125.stdout:1493: snap_remove snap 154 2021-03-24T21:59:34.453 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:34.461 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:34 smithi142 conmon[20125]: cluster 2021-03-24T21:59:32 2021-03-24T21:59:34.461 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:34 smithi142 conmon[20125]: .830208+0000 mgr.y (mgr.14144) 472 : cluster [DBG] pgmap v766: 178 pgs: 4 active+clean+remapped, 52 active+undersized, 38 active+undersized+degraded, 84 active+clean; 347 MiB data, 845 MiB used, 536 GiB / 536 GiB avail; 6.0 MiB/s rd, 2.1 MiB/s wr, 11 op/s; 268/1449 objects degraded (18.496%); 1/1449 objects misplaced (0.069%) 2021-03-24T21:59:34.462 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:34 smithi142 conmon[20125]: audit 2021-03-24T21:59:33.362263+0000 2021-03-24T21:59:34.462 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:34 smithi142 conmon[20125]: mon.a (mon.0) 1200 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pgp_num_actual", "val": "22"}]': finished 2021-03-24T21:59:34.462 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:34 smithi142 conmon[20125]: cluster 2021-03-24T21:59:34.462 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:34 smithi142 conmon[20125]: 2021-03-24T21:59:33.362386+0000 mon.a ( 2021-03-24T21:59:34.462 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:34 smithi142 conmon[20125]: mon.0) 1201 : cluster [DBG] osdmap e403: 8 total, 7 up, 6 in 2021-03-24T21:59:34.555 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:34.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:34.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:34.641 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-24T21:59:34.641 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-24T21:59:34.642 INFO:teuthology.orchestra.run.smithi125.stdout: "num_ops": 0 2021-03-24T21:59:34.642 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-24T21:59:34.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:34 smithi125 conmon[24927]: cluster 2021-03-24T21:59:32.830208+0000 mgr.y ( 2021-03-24T21:59:34.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:34 smithi125 conmon[24927]: mgr.14144) 472 : cluster [DBG] pgmap v766: 178 pgs: 4 active+clean+remapped, 52 active+undersized, 38 active+undersized+degraded, 84 active+clean; 347 MiB data, 845 MiB used, 536 GiB / 536 GiB avail; 6.0 MiB/s rd, 2.1 MiB/s wr, 11 op/s; 268/1449 objects degraded (18.496%); 1/1449 objects misplaced (0.069%) 2021-03-24T21:59:34.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:34 smithi125 conmon[24927]: audit 2021-03-24T21:59:33.362263+0000 mon.a ( 2021-03-24T21:59:34.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:34 smithi125 conmon[24927]: mon.0) 1200 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pgp_num_actual", "val": "22"}]': finished 2021-03-24T21:59:34.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:34 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:59:34.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:34 smithi125 conmon[24927]: 21:59:33.362386+0000 mon.a (mon.0) 1201 : cluster [DBG] osdmap e403: 8 total, 7 up, 6 in 2021-03-24T21:59:34.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:34 smithi125 conmon[19936]: cluster 2021-03-24T21:59:34.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:34 smithi125 conmon[19936]: 2021-03-24T21:59:34.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:34 smithi125 conmon[19936]: 2021 2021-03-24T21:59:34.657 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:34 smithi125 conmon[19936]: - 2021-03-24T21:59:34.657 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:34 smithi125 conmon[19936]: 03 2021-03-24T21:59:34.657 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:34 smithi125 conmon[19936]: - 2021-03-24T21:59:34.657 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:34 smithi125 conmon[19936]: 24 2021-03-24T21:59:34.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:34 smithi125 conmon[19936]: T 2021-03-24T21:59:34.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:34 smithi125 conmon[19936]: 21 2021-03-24T21:59:34.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:34 smithi125 conmon[19936]: : 2021-03-24T21:59:34.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:34 smithi125 conmon[19936]: 59 2021-03-24T21:59:34.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:34 smithi125 conmon[19936]: : 2021-03-24T21:59:34.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:34 smithi125 conmon[19936]: 32 2021-03-24T21:59:34.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:34 smithi125 conmon[19936]: . 2021-03-24T21:59:34.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:34 smithi125 conmon[19936]: 830208 2021-03-24T21:59:34.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:34 smithi125 conmon[19936]: +0000 mgr.y (mgr.14144) 472 : cluster [DBG] pgmap v766: 178 pgs: 4 active+clean+remapped, 52 active+undersized, 38 active+undersized+degraded, 84 active+clean; 347 MiB data, 845 MiB used, 536 GiB / 536 GiB avail; 6.0 MiB/s rd, 2.1 MiB/s wr, 11 op/s; 268/1449 objects degraded (18.496%); 1/1449 objects misplaced (0.069%) 2021-03-24T21:59:34.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:34 smithi125 conmon[19936]: 2021-03-24T21:59:34.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:34 smithi125 conmon[19936]: audit 2021-03-24T21:59:34.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:34 smithi125 conmon[19936]: 2021-03-24T21:59:33.362263+0000 mon.a 2021-03-24T21:59:34.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:34 smithi125 conmon[19936]: (mon.0) 1200 : audit [INF] 2021-03-24T21:59:34.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:34 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pgp_num_actual", "val": "22"}]': finished 2021-03-24T21:59:34.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:34 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:59:34.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:34 smithi125 conmon[19936]: -24T21:59:33. 2021-03-24T21:59:34.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:34 smithi125 conmon[19936]: 362386+0000 mon.a (mon.0) 2021-03-24T21:59:34.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:34 smithi125 conmon[19936]: 1201 : cluster [DBG] osdmap e403: 8 total, 7 up, 6 in 2021-03-24T21:59:34.662 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:34.710 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:34.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:34.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:59:34.810 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd unset noscrub 2021-03-24T21:59:34.815 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:34.867 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:34.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:34.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:59:34.971 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:35.018 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:35.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:35.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:59:35.123 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:35.173 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:35.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:35.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:59:35.279 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:35.335 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:35.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:35.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:35.377 INFO:tasks.rados.rados.0.smithi125.stdout:1493: done (5 left) 2021-03-24T21:59:35.377 INFO:tasks.rados.rados.0.smithi125.stdout:1494: snap_create 2021-03-24T21:59:35.378 INFO:tasks.rados.rados.0.smithi125.stdout:1488: expect (ObjNum 517 snap 154 seq_num 517) 2021-03-24T21:59:35.439 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:35.557 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:35.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:35.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:59:35.571 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:35 smithi125 conmon[24927]: cluster 2021-03-24T21:59:34 2021-03-24T21:59:35.571 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:35 smithi125 conmon[24927]: .366827+0000 mon.a (mon.0) 1202 : cluster [DBG] osdmap e404: 8 total, 7 up, 6 in 2021-03-24T21:59:35.572 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:35 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:59:35.572 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:35 smithi125 conmon[24927]: 59:34.555220+0000 osd.4 (osd.4) 99 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:59:35.572 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:35 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:59:35.572 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:35 smithi125 conmon[24927]: 03-24T21:59:34.555578+0000 2021-03-24T21:59:35.573 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:35 smithi125 conmon[24927]: osd.4 (osd.4) 100 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:59:35.573 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:35 smithi125 conmon[24927]: 2021-03-24T21:59:35.573 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:35 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:59:35.573 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:35 smithi125 conmon[24927]: -03-24T21:59:35.207000+0000 mon.a (mon.0) 2021-03-24T21:59:35.574 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:35 smithi125 conmon[24927]: 1203 : cluster [WRN] Health check update: Degraded data redundancy: 268/1449 objects degraded (18.496%), 38 pgs degraded, 84 pgs undersized (PG_DEGRADED) 2021-03-24T21:59:35.574 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:35 smithi125 conmon[24927]: cluster 2021-03-24T21:59:35.367780+0000 mon.a (mon.0) 1204 : cluster 2021-03-24T21:59:35.574 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:35 smithi125 conmon[24927]: [DBG] osdmap e405: 8 total, 7 up, 6 in 2021-03-24T21:59:35.575 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:35 smithi125 conmon[19936]: cluster 2021-03-24T21:59:34.366827 2021-03-24T21:59:35.575 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:35 smithi125 conmon[19936]: +0000 mon.a (mon.0) 1202 : cluster [DBG] osdmap e404: 8 total, 7 up, 6 in 2021-03-24T21:59:35.575 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:35 smithi125 conmon[19936]: cluster 2021-03-24T21:59:34.555220+0000 osd.4 (osd.4) 99 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:59:35.575 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:35 smithi125 conmon[19936]: cluster 2021-03-24T21:59:34.555578 2021-03-24T21:59:35.575 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:35 smithi125 conmon[19936]: +0000 osd.4 (osd.4) 100 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:59:35.576 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:35 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:59:35.576 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:35 smithi125 conmon[19936]: :59:35.207000+0000 mon.a (mon.0) 1203 : cluster 2021-03-24T21:59:35.576 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:35 smithi125 conmon[19936]: [WRN] Health check update: Degraded data redundancy: 268/1449 objects degraded (18.496%), 38 pgs degraded, 84 pgs undersized (PG_DEGRADED) 2021-03-24T21:59:35.577 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:35 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T21:59:35.577 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:35 smithi125 conmon[19936]: 24T21:59:35.367780+0000 mon.a (mon.0) 1204 : cluster [DBG] osdmap e405: 8 total, 7 up, 6 in 2021-03-24T21:59:35.672 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:35.691 INFO:tasks.rados.rados.0.smithi125.stdout:1492: finishing rollback tid 0 to smithi12556470-29 2021-03-24T21:59:35.691 INFO:tasks.rados.rados.0.smithi125.stdout:1492: finishing rollback tid 1 to smithi12556470-29 2021-03-24T21:59:35.691 INFO:tasks.rados.rados.0.smithi125.stdout:1492: finishing rollback tid 2 to smithi12556470-29 2021-03-24T21:59:35.692 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 29 v 199 (ObjNum 401 snap 104 seq_num 401) dirty exists 2021-03-24T21:59:35.692 INFO:tasks.rados.rados.0.smithi125.stdout:1489: finishing copy_from to smithi12556470-2 2021-03-24T21:59:35.692 INFO:tasks.rados.rados.0.smithi125.stdout:1489: got expected ENOENT (src dne) 2021-03-24T21:59:35.692 INFO:tasks.rados.rados.0.smithi125.stdout:1489: finishing copy_from racing read to smithi12556470-2 2021-03-24T21:59:35.692 INFO:tasks.rados.rados.0.smithi125.stdout:1491: expect (ObjNum 471 snap 133 seq_num 471) 2021-03-24T21:59:35.720 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:35.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:35.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:59:35.826 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:35.837 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:35 smithi142 conmon[20125]: cluster 2021-03-24T21:59:34.366827+0000 mon.a ( 2021-03-24T21:59:35.837 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:35 smithi142 conmon[20125]: mon.0) 1202 : cluster [DBG] osdmap e404: 8 total, 7 up, 6 in 2021-03-24T21:59:35.838 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:35 smithi142 conmon[20125]: cluster 2021-03-24T21:59:34.555220+0000 osd.4 (osd.4) 99 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:59:35.838 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:35 smithi142 conmon[20125]: cluster 2021-03-24T21:59:35.838 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:35 smithi142 conmon[20125]: 2021-03-24T21:59:34.555578+0000 osd.4 (osd 2021-03-24T21:59:35.838 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:35 smithi142 conmon[20125]: .4) 100 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:59:35.839 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:35 smithi142 conmon[20125]: cluster 2021-03-24T21:59:35.839 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:35 smithi142 conmon[20125]: 2021-03-24T21:59:35.207000+0000 mon.a 2021-03-24T21:59:35.839 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:35 smithi142 conmon[20125]: (mon.0) 1203 : cluster [WRN] Health check update: Degraded data redundancy: 268/1449 objects degraded (18.496%), 38 pgs degraded, 84 pgs undersized (PG_DEGRADED) 2021-03-24T21:59:35.839 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:35 smithi142 conmon[20125]: cluster 2021-03-24T21:59:35.840 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:35 smithi142 conmon[20125]: 2021-03-24T21:59:35.367780+0000 2021-03-24T21:59:35.840 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:35 smithi142 conmon[20125]: mon.a (mon.0) 1204 : cluster [DBG] osdmap e405: 8 total, 7 up, 6 in 2021-03-24T21:59:35.840 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:35 smithi142 conmon[20125]: 2021-03-24T21:59:35.865 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.3 dump_blocked_ops 2021-03-24T21:59:35.872 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:35.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:35.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:35.948 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:59:35.976 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:36.018 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:36.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:36.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:59:36.063 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T21:59:36.099 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T21:59:36.116 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T21:59:36.123 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:36.146 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T21:59:36.168 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T21:59:36.169 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:36.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:36.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:59:36.201 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T21:59:36.235 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T21:59:36.235 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-24T21:59:36.235 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-24T21:59:36.236 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-24T21:59:36.236 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-24T21:59:36.236 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-24T21:59:36.236 INFO:teuthology.orchestra.run.smithi125.stderr:Error ENXIO: problem getting command descriptions from osd.5 2021-03-24T21:59:36.237 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-24T21:59:36.237 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-24T21:59:36.273 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:36.324 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:36.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:36.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:36.382 INFO:tasks.rados.rados.0.smithi125.stdout:1488: done (5 left) 2021-03-24T21:59:36.383 INFO:tasks.rados.rados.0.smithi125.stdout:1489: done (4 left) 2021-03-24T21:59:36.383 INFO:tasks.rados.rados.0.smithi125.stdout:1490: done (3 left) 2021-03-24T21:59:36.383 INFO:tasks.rados.rados.0.smithi125.stdout:1491: done (2 left) 2021-03-24T21:59:36.383 INFO:tasks.rados.rados.0.smithi125.stdout:1492: done (1 left) 2021-03-24T21:59:36.383 INFO:tasks.rados.rados.0.smithi125.stdout:1494: done (0 left) 2021-03-24T21:59:36.384 INFO:tasks.rados.rados.0.smithi125.stdout:1495: copy_from oid 21 from oid 46 current snap is 160 2021-03-24T21:59:36.384 INFO:tasks.rados.rados.0.smithi125.stdout:1496: snap_create 2021-03-24T21:59:36.386 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:36 smithi142 conmon[20125]: cluster 2021-03-24T21:59:34.831584+0000 mgr.y (mgr.14144) 473 : 2021-03-24T21:59:36.386 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:36 smithi142 conmon[20125]: cluster [DBG] pgmap v769: 178 pgs: 4 peering, 4 active+clean+remapped, 52 active+undersized, 37 active+undersized+degraded, 81 active+clean; 370 MiB data, 880 MiB used, 536 GiB / 536 GiB avail; 16 MiB/s rd, 3.1 MiB/s wr, 9 op/s; 260/1437 objects degraded (18.093%); 1/1437 objects misplaced (0.070%) 2021-03-24T21:59:36.386 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:36 smithi142 conmon[20125]: cluster 2021-03-24T21:59:35.370517+0000 mon.a (mon.0) 1205 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2021-03-24T21:59:36.387 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:36 smithi142 conmon[20125]: audit 2021-03-24T21:59:36.036686+0000 mon.c (mon.2) 63 : audit [DBG] from='client.? v1:172.21.15.125:0/3276424646' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:59:36.427 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:36 smithi125 conmon[19936]: cluster 2021-03-24T21:59:34.831584+0000 2021-03-24T21:59:36.427 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:36 smithi125 conmon[19936]: mgr.y (mgr.14144) 473 : cluster [DBG] pgmap v769: 178 pgs: 4 peering, 4 active+clean+remapped, 52 active+undersized, 37 active+undersized+degraded, 81 active+clean; 370 MiB data, 880 MiB used, 536 GiB / 536 GiB avail; 16 MiB/s rd, 3.1 MiB/s wr, 9 op/s; 260/1437 objects degraded (18.093%); 1/1437 objects misplaced (0.070%) 2021-03-24T21:59:36.428 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:36 smithi125 conmon[19936]: cluster 2021-03-24T21:59:35.370517+0000 mon.a (mon.0) 1205 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2021-03-24T21:59:36.428 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:36 smithi125 conmon[19936]: audit 2021-03-24T21:59:36.036686+0000 mon.c (mon. 2021-03-24T21:59:36.428 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:36 smithi125 conmon[19936]: 2) 63 : audit [DBG] from='client.? v1:172.21.15.125:0/3276424646' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:59:36.428 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:36 smithi125 conmon[24927]: cluster 2021-03-24T21:59:36.429 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:36 smithi125 conmon[24927]: 2021-03-24T21:59:36.429 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:36 smithi125 conmon[24927]: 2021 2021-03-24T21:59:36.429 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:36 smithi125 conmon[24927]: - 2021-03-24T21:59:36.429 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:36 smithi125 conmon[24927]: 03 2021-03-24T21:59:36.430 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:36 smithi125 conmon[24927]: -24T21:59:34.831584 2021-03-24T21:59:36.430 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:36 smithi125 conmon[24927]: +0000 mgr.y (mgr.14144) 473 : cluster [DBG] pgmap v769: 178 pgs: 4 peering, 4 active+clean+remapped, 52 active+undersized, 37 active+undersized+degraded, 81 active+clean; 370 MiB data, 880 MiB used, 536 GiB / 536 GiB avail; 16 MiB/s rd, 3.1 MiB/s wr, 9 op/s; 260/1437 objects degraded (18.093%); 1/1437 objects misplaced (0.070%) 2021-03-24T21:59:36.430 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:36 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:59:36.430 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:36 smithi125 conmon[24927]: 59:35.370517+0000 mon.a (mon.0) 1205 : cluster 2021-03-24T21:59:36.431 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:36 smithi125 conmon[24927]: [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2021-03-24T21:59:36.431 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:36 smithi125 conmon[24927]: audit 2021-03-24T21: 2021-03-24T21:59:36.431 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:36 smithi125 conmon[24927]: 59:36.036686+0000 mon.c (mon.2) 63 : audit 2021-03-24T21:59:36.431 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:36 smithi125 conmon[24927]: [DBG] from='client.? v1:172.21.15.125:0/3276424646' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:59:36.432 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:36.433 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:59:36.456 INFO:tasks.rados.rados.0.smithi125.stdout:1495: finishing copy_from to smithi12556470-21 2021-03-24T21:59:36.457 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 21 v 334 (ObjNum 471 snap 133 seq_num 471) dirty exists 2021-03-24T21:59:36.457 INFO:tasks.rados.rados.0.smithi125.stdout:1495: finishing copy_from racing read to smithi12556470-21 2021-03-24T21:59:36.525 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:36.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:36.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:36.639 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:36.679 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:36.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:36.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:59:36.786 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:36.833 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:36.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:36.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:59:36.940 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:36.990 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:36.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:36.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:59:37.097 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:37.153 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:37.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:37.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:59:37.258 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:37.312 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:37.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:37.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:59:37.388 INFO:tasks.rados.rados.0.smithi125.stdout:1495: done (1 left) 2021-03-24T21:59:37.389 INFO:tasks.rados.rados.0.smithi125.stdout:1496: done (0 left) 2021-03-24T21:59:37.389 INFO:tasks.rados.rados.0.smithi125.stdout:1497: read oid 24 snap -1 2021-03-24T21:59:37.389 INFO:tasks.rados.rados.0.smithi125.stdout:1497: expect (ObjNum 515 snap 151 seq_num 515) 2021-03-24T21:59:37.389 INFO:tasks.rados.rados.0.smithi125.stdout:1498: read oid 50 snap -1 2021-03-24T21:59:37.390 INFO:tasks.rados.rados.0.smithi125.stdout:1498: expect (ObjNum 514 snap 151 seq_num 514) 2021-03-24T21:59:37.390 INFO:tasks.rados.rados.0.smithi125.stdout:1499: delete oid 13 current snap is 161 2021-03-24T21:59:37.417 INFO:tasks.rados.rados.0.smithi125.stdout:1498: expect (ObjNum 514 snap 151 seq_num 514) 2021-03-24T21:59:37.419 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:37.537 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:37.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:37.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:59:37.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[24927]: cluster 2021-03-24T21:59:37.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[24927]: 2021-03-24T21:59:37.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[24927]: 2021 2021-03-24T21:59:37.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[24927]: - 2021-03-24T21:59:37.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[24927]: 03 2021-03-24T21:59:37.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[24927]: - 2021-03-24T21:59:37.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[24927]: 24 2021-03-24T21:59:37.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[24927]: T 2021-03-24T21:59:37.618 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[24927]: 21 2021-03-24T21:59:37.618 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[24927]: : 2021-03-24T21:59:37.618 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[24927]: 59 2021-03-24T21:59:37.618 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[24927]: : 2021-03-24T21:59:37.618 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[24927]: 36 2021-03-24T21:59:37.619 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[24927]: . 2021-03-24T21:59:37.619 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[24927]: 378535 2021-03-24T21:59:37.619 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[24927]: +0000 2021-03-24T21:59:37.619 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[24927]: 2021-03-24T21:59:37.620 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[24927]: mon.a 2021-03-24T21:59:37.620 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[24927]: ( 2021-03-24T21:59:37.620 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[24927]: mon 2021-03-24T21:59:37.620 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[24927]: . 2021-03-24T21:59:37.621 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[24927]: 0 2021-03-24T21:59:37.621 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[24927]: ) 2021-03-24T21:59:37.621 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[24927]: 1206 2021-03-24T21:59:37.621 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[24927]: : 2021-03-24T21:59:37.621 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[24927]: cluster 2021-03-24T21:59:37.622 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[24927]: [DBG] osdmap e406: 8 total, 7 up, 6 in 2021-03-24T21:59:37.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:37 smithi142 conmon[20125]: cluster 2021-03-24T21:59:36.378535+0000 mon.a ( 2021-03-24T21:59:37.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:37 smithi142 conmon[20125]: mon.0) 1206 : cluster [DBG] osdmap e406: 8 total, 7 up, 6 in 2021-03-24T21:59:37.623 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[19936]: cluster 2021-03-24T21:59:37.623 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[19936]: 2021-03-24T21:59:37.623 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[19936]: 2021 2021-03-24T21:59:37.624 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[19936]: - 2021-03-24T21:59:37.624 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[19936]: 03 2021-03-24T21:59:37.624 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[19936]: - 2021-03-24T21:59:37.624 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[19936]: 24 2021-03-24T21:59:37.625 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[19936]: T 2021-03-24T21:59:37.625 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[19936]: 21 2021-03-24T21:59:37.625 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[19936]: : 2021-03-24T21:59:37.625 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[19936]: 59 2021-03-24T21:59:37.625 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[19936]: : 2021-03-24T21:59:37.626 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[19936]: 36 2021-03-24T21:59:37.626 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[19936]: . 2021-03-24T21:59:37.626 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[19936]: 378535 2021-03-24T21:59:37.626 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[19936]: +0000 2021-03-24T21:59:37.627 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[19936]: 2021-03-24T21:59:37.627 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[19936]: mon.a 2021-03-24T21:59:37.627 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[19936]: ( 2021-03-24T21:59:37.627 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[19936]: mon 2021-03-24T21:59:37.627 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[19936]: . 2021-03-24T21:59:37.628 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[19936]: 0 2021-03-24T21:59:37.628 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[19936]: ) 2021-03-24T21:59:37.628 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[19936]: 1206 2021-03-24T21:59:37.628 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[19936]: : 2021-03-24T21:59:37.629 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[19936]: cluster 2021-03-24T21:59:37.629 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[19936]: 2021-03-24T21:59:37.629 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[19936]: [DBG] 2021-03-24T21:59:37.629 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[19936]: 2021-03-24T21:59:37.629 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[19936]: osdmap e406: 8 total, 7 up, 6 in 2021-03-24T21:59:37.630 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:37 smithi125 conmon[19936]: 2021-03-24T21:59:37.648 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:37.699 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:37.700 INFO:tasks.rados.rados.0.smithi125.stdout:1498: done (2 left) 2021-03-24T21:59:37.700 INFO:tasks.rados.rados.0.smithi125.stdout:1499: done (1 left) 2021-03-24T21:59:37.700 INFO:tasks.rados.rados.0.smithi125.stdout:1500: snap_create 2021-03-24T21:59:37.701 INFO:tasks.rados.rados.0.smithi125.stdout:1497: expect (ObjNum 515 snap 151 seq_num 515) 2021-03-24T21:59:37.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:37.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:59:37.807 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:37.857 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:37.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:37.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:37.964 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:38.021 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:38.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:38.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:59:38.127 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:38.185 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:38.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:38.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:59:38.222 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-03-24T21:59:38.223 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-03-24T21:59:38.292 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:38.348 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:38.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:38.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:59:38.422 INFO:tasks.rados.rados.0.smithi125.stdout:1497: done (1 left) 2021-03-24T21:59:38.422 INFO:tasks.rados.rados.0.smithi125.stdout:1500: done (0 left) 2021-03-24T21:59:38.422 INFO:tasks.rados.rados.0.smithi125.stdout:1501: read oid 22 snap -1 2021-03-24T21:59:38.423 INFO:tasks.rados.rados.0.smithi125.stdout:1501: expect (ObjNum 468 snap 132 seq_num 468) 2021-03-24T21:59:38.423 INFO:tasks.rados.rados.0.smithi125.stdout:1502: read oid 23 snap -1 2021-03-24T21:59:38.423 INFO:tasks.rados.rados.0.smithi125.stdout:1502: expect (ObjNum 427 snap 113 seq_num 427) 2021-03-24T21:59:38.424 INFO:tasks.rados.rados.0.smithi125.stdout:1503: read oid 30 snap 155 2021-03-24T21:59:38.424 INFO:tasks.rados.rados.0.smithi125.stdout:1503: expect deleted 2021-03-24T21:59:38.424 INFO:tasks.rados.rados.0.smithi125.stdout:1504: write (excl) oid 18 current snap is 162 2021-03-24T21:59:38.424 INFO:tasks.rados.rados.0.smithi125.stdout:1504: seq_num 534 ranges {574938=727676,1731855=703504,2848231=126707} 2021-03-24T21:59:38.425 INFO:tasks.rados.rados.0.smithi125.stdout:1504: writing smithi12556470-18 from 574938 to 1302614 tid 1 2021-03-24T21:59:38.431 INFO:tasks.rados.rados.0.smithi125.stdout:1504: writing smithi12556470-18 from 1731855 to 2435359 tid 2 2021-03-24T21:59:38.441 INFO:tasks.rados.rados.0.smithi125.stdout:1504: writing smithi12556470-18 from 2848231 to 2974938 tid 3 2021-03-24T21:59:38.441 INFO:tasks.rados.rados.0.smithi125.stdout:1505: delete oid 39 current snap is 162 2021-03-24T21:59:38.445 INFO:tasks.rados.rados.0.smithi125.stdout:1502: expect (ObjNum 427 snap 113 seq_num 427) 2021-03-24T21:59:38.454 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:38.586 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:38 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:59:38.586 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:38 smithi125 conmon[24927]: 59:36.832362+0000 mgr.y (mgr.14144) 474 : cluster [DBG] pgmap v772: 178 pgs: 4 peering, 4 active+clean+remapped, 52 active+undersized, 37 active+undersized+degraded, 81 active+clean; 370 MiB data, 879 MiB used, 536 GiB / 536 GiB avail; 3.1 MiB/s wr, 4 op/s; 260/1437 objects degraded (18.093%); 1/1437 objects misplaced (0.070%) 2021-03-24T21:59:38.586 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:38 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T21:59:38.587 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:38 smithi125 conmon[24927]: 21:59:37.382827+0000 2021-03-24T21:59:38.587 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:38 smithi125 conmon[24927]: mon.a (mon.0) 1207 : cluster 2021-03-24T21:59:38.587 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:38 smithi125 conmon[24927]: [DBG] osdmap e407: 8 total, 7 up, 6 in 2021-03-24T21:59:38.588 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:38 smithi125 conmon[19936]: cluster 2021-03-24T21:59:36. 2021-03-24T21:59:38.588 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:38 smithi125 conmon[19936]: 832362+0000 mgr.y (mgr.14144) 474 : cluster [DBG] pgmap v772: 178 pgs: 4 peering, 4 active+clean+remapped, 52 active+undersized, 37 active+undersized+degraded, 81 active+clean; 370 MiB data, 879 MiB used, 536 GiB / 536 GiB avail; 3.1 MiB/s wr, 4 op/s; 260/1437 objects degraded (18.093%); 1/1437 objects misplaced (0.070%) 2021-03-24T21:59:38.588 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:38 smithi125 conmon[19936]: cluster 2021-03-24T21:59:37. 2021-03-24T21:59:38.588 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:38 smithi125 conmon[19936]: 382827+0000 mon.a (mon.0) 1207 : cluster 2021-03-24T21:59:38.589 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:38 smithi125 conmon[19936]: [DBG] osdmap e407: 8 total, 7 up, 6 in 2021-03-24T21:59:38.613 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:38.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:38.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:59:38.699 INFO:tasks.rados.rados.0.smithi125.stdout:1504: finishing write tid 1 to smithi12556470-18 2021-03-24T21:59:38.699 INFO:tasks.rados.rados.0.smithi125.stdout:1501: expect (ObjNum 468 snap 132 seq_num 468) 2021-03-24T21:59:38.723 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:38.766 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:38.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:38.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:59:38.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:38 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:59:38.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:38 smithi142 conmon[20125]: 59:36.832362+0000 mgr.y (mgr.14144) 474 : cluster [DBG] pgmap v772: 178 pgs: 4 peering, 4 active+clean+remapped, 52 active+undersized, 37 active+undersized+degraded, 81 active+clean; 370 MiB data, 879 MiB used, 536 GiB / 536 GiB avail; 3.1 MiB/s wr, 4 op/s; 260/1437 objects degraded (18.093%); 1/1437 objects misplaced (0.070%) 2021-03-24T21:59:38.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:38 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:59:38.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:38 smithi142 conmon[20125]: 03-24T21:59:37.382827+0000 mon.a (mon.0) 1207 2021-03-24T21:59:38.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:38 smithi142 conmon[20125]: : cluster [DBG] osdmap e407: 8 total, 7 up, 6 in 2021-03-24T21:59:38.870 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:38.915 INFO:tasks.rados.rados.0.smithi125.stdout:1504: finishing write tid 2 to smithi12556470-18 2021-03-24T21:59:38.915 INFO:tasks.rados.rados.0.smithi125.stdout:1504: finishing write tid 3 to smithi12556470-18 2021-03-24T21:59:38.915 INFO:tasks.rados.rados.0.smithi125.stdout:1504: finishing write tid 4 to smithi12556470-18 2021-03-24T21:59:38.916 INFO:tasks.rados.rados.0.smithi125.stdout:1504: finishing write tid 5 to smithi12556470-18 2021-03-24T21:59:38.916 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 18 v 187 (ObjNum 534 snap 162 seq_num 534) dirty exists 2021-03-24T21:59:38.916 INFO:tasks.rados.rados.0.smithi125.stdout:1504: left oid 18 (ObjNum 534 snap 162 seq_num 534) 2021-03-24T21:59:38.916 INFO:tasks.rados.rados.0.smithi125.stdout:1501: done (4 left) 2021-03-24T21:59:38.917 INFO:tasks.rados.rados.0.smithi125.stdout:1502: done (3 left) 2021-03-24T21:59:38.917 INFO:tasks.rados.rados.0.smithi125.stdout:1503: done (2 left) 2021-03-24T21:59:38.917 INFO:tasks.rados.rados.0.smithi125.stdout:1504: done (1 left) 2021-03-24T21:59:38.917 INFO:tasks.rados.rados.0.smithi125.stdout:1505: done (0 left) 2021-03-24T21:59:38.918 INFO:tasks.rados.rados.0.smithi125.stdout:1506: snap_remove snap 140 2021-03-24T21:59:38.922 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:38.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:38.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:59:39.028 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:39.082 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:39.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:39.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:39.187 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:39.241 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:39.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:39.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:39.347 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:39.404 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:39.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:39.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:59:39.419 INFO:tasks.rados.rados.0.smithi125.stdout:1506: done (0 left) 2021-03-24T21:59:39.419 INFO:tasks.rados.rados.0.smithi125.stdout:1507: rollback oid 10 current snap is 162 2021-03-24T21:59:39.419 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 10 to 161 2021-03-24T21:59:39.420 INFO:tasks.rados.rados.0.smithi125.stdout:1508: snap_remove snap 141 2021-03-24T21:59:39.510 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:39.517 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:39 smithi125 conmon[19936]: cluster 2021-03-24T21:59: 2021-03-24T21:59:39.518 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:39 smithi125 conmon[19936]: 38.409129+0000 mon.a (mon.0) 1208 : cluster [DBG] osdmap e408: 8 total, 7 up, 6 in 2021-03-24T21:59:39.521 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:39 smithi125 conmon[24927]: cluster 2021-03-24T21:59:38.409129+0000 mon.a (mon.0) 1208 : cluster [DBG] osdmap e408: 8 total, 7 up, 6 in 2021-03-24T21:59:39.592 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 7] out_osds: [6, 1] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 6, 7] 2021-03-24T21:59:39.592 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-24T21:59:39.593 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [0, 2, 3, 4, 5, 7] 2021-03-24T21:59:39.593 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd out 7 2021-03-24T21:59:39.619 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:39.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:39.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:59:39.730 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:39.772 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:39.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:39.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:59:39.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:39 smithi142 conmon[20125]: cluster 2021-03-24T21:59:38.409129+0000 mon.a (mon.0) 1208 : cluster [DBG] osdmap e408: 8 total, 7 up, 6 in 2021-03-24T21:59:39.876 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:39.920 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:39.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:39.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:59:40.024 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:40.072 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:59:40.074 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:40.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:40.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:59:40.179 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:40.224 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-24T21:59:40.225 INFO:tasks.rados.rados.0.smithi125.stdout:1508: done (1 left) 2021-03-24T21:59:40.226 INFO:tasks.rados.rados.0.smithi125.stdout:1509: rollback oid 16 current snap is 162 2021-03-24T21:59:40.226 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 16 to 139 2021-03-24T21:59:40.226 INFO:tasks.rados.rados.0.smithi125.stdout:1510: snap_create 2021-03-24T21:59:40.226 INFO:tasks.rados.rados.0.smithi125.stdout:1507: finishing rollback tid 0 to smithi12556470-10 2021-03-24T21:59:40.226 INFO:tasks.rados.rados.0.smithi125.stdout:1507: finishing rollback tid 1 to smithi12556470-10 2021-03-24T21:59:40.227 INFO:tasks.rados.rados.0.smithi125.stdout:1507: finishing rollback tid 2 to smithi12556470-10 2021-03-24T21:59:40.227 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 10 v 406 (ObjNum 393 snap 100 seq_num 393) dirty exists 2021-03-24T21:59:40.243 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:40.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:40.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:59:40.257 INFO:tasks.rados.rados.0.smithi125.stdout:1509: finishing rollback tid 0 to smithi12556470-16 2021-03-24T21:59:40.261 INFO:tasks.rados.rados.0.smithi125.stdout:1509: finishing rollback tid 1 to smithi12556470-16 2021-03-24T21:59:40.261 INFO:tasks.rados.rados.0.smithi125.stdout:1509: finishing rollback tid 2 to smithi12556470-16 2021-03-24T21:59:40.262 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 16 v 383 (ObjNum 393 snap 100 seq_num 393) dirty exists 2021-03-24T21:59:40.357 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:40.400 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:40.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:40.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:59:40.488 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-24T21:59:40.489 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T21:59:40.504 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:40.553 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:40.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:40.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:59:40.579 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[24927]: cluster 2021-03-24T21:59:38.833420 2021-03-24T21:59:40.579 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[24927]: +0000 mgr.y (mgr.14144) 475 : cluster [DBG] pgmap v775: 178 pgs: 4 peering, 5 active+clean+remapped, 52 active+undersized, 35 active+undersized+degraded, 82 active+clean; 372 MiB data, 879 MiB used, 536 GiB / 536 GiB avail; 0 B/s wr, 0 op/s; 240/1440 objects degraded (16.667%); 12/1440 objects misplaced (0.833%); 222 B/s, 2 objects/s recovering 2021-03-24T21:59:40.580 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:59:40.580 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[24927]: 59:39.344607+0000 osd.7 (osd.7) 2021-03-24T21:59:40.581 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[24927]: 89 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:59:40.581 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[24927]: cluster 2021-03-24T21:59:39.345283+0000 osd.7 (osd.7) 90 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:59:40.582 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[24927]: cluster 2021-03-24T21:59:39.416803+0000 mon.a (mon.0) 1209 : cluster [DBG] osdmap e409: 8 total, 7 up, 6 in 2021-03-24T21:59:40.582 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[24927]: cluster 2021-03-24T21:59:40.208909+0000 mon.a (mon.0) 1210 : cluster [WRN] Health check update: Degraded data redundancy: 240/1440 objects degraded (16.667%), 35 pgs degraded, 81 pgs undersized (PG_DEGRADED) 2021-03-24T21:59:40.582 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[24927]: cluster 2021-03-24T21:59:40.221767+0000 mon.a (mon.0) 1211 : cluster [DBG] osdmap e410: 8 total, 7 up, 6 in 2021-03-24T21:59:40.583 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[24927]: audit 2021-03-24T21:59:40.340788+0000 mon.c (mon.2) 64 : audit [INF] from='client.? v1:172.21.15.125:0/768534752' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T21:59:40.583 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[24927]: audit 2021-03-24T21:59:40.341398+0000 mon.a (mon.0) 1212 : audit [INF] from='client.35198 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T21:59:40.584 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: cluster 2021-03-24T21:59:40.585 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 2021-03-24T21:59:40.585 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 2021 2021-03-24T21:59:40.585 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: - 2021-03-24T21:59:40.586 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 03 2021-03-24T21:59:40.586 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: - 2021-03-24T21:59:40.587 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 24 2021-03-24T21:59:40.587 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: T 2021-03-24T21:59:40.587 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 21 2021-03-24T21:59:40.588 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: : 2021-03-24T21:59:40.588 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 59 2021-03-24T21:59:40.588 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: : 2021-03-24T21:59:40.589 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 38 2021-03-24T21:59:40.589 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: . 2021-03-24T21:59:40.589 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 833420 2021-03-24T21:59:40.589 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: +0000 2021-03-24T21:59:40.590 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 2021-03-24T21:59:40.590 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: mgr.y 2021-03-24T21:59:40.590 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: ( 2021-03-24T21:59:40.590 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: mgr 2021-03-24T21:59:40.591 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: . 2021-03-24T21:59:40.591 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 14144 2021-03-24T21:59:40.591 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: ) 2021-03-24T21:59:40.591 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 475 2021-03-24T21:59:40.591 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: : 2021-03-24T21:59:40.592 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: cluster 2021-03-24T21:59:40.592 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 2021-03-24T21:59:40.592 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: [DBG] 2021-03-24T21:59:40.592 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 2021-03-24T21:59:40.593 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: pgmap v775: 178 pgs: 4 peering, 5 active+clean+remapped, 52 active+undersized, 35 active+undersized+degraded, 82 active+clean; 372 MiB data, 879 MiB used, 536 GiB / 536 GiB avail; 0 B/s wr, 0 op/s; 240/1440 objects degraded (16.667%); 12/1440 objects misplaced (0.833%); 222 B/s, 2 objects/s recovering 2021-03-24T21:59:40.593 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 2021-03-24T21:59:40.593 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: cluster 2021-03-24T21:59:40.593 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 2021-03-24T21:59:40.594 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 2021 2021-03-24T21:59:40.594 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: - 2021-03-24T21:59:40.594 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 03 2021-03-24T21:59:40.594 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: - 2021-03-24T21:59:40.594 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 24 2021-03-24T21:59:40.595 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: T 2021-03-24T21:59:40.595 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 21 2021-03-24T21:59:40.595 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: : 2021-03-24T21:59:40.595 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 59 2021-03-24T21:59:40.595 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: : 2021-03-24T21:59:40.596 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 39 2021-03-24T21:59:40.596 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: . 2021-03-24T21:59:40.596 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 344607 2021-03-24T21:59:40.596 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: +0000 2021-03-24T21:59:40.596 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 2021-03-24T21:59:40.597 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: osd.7 2021-03-24T21:59:40.597 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: ( 2021-03-24T21:59:40.597 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: osd 2021-03-24T21:59:40.597 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: . 2021-03-24T21:59:40.598 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 7 2021-03-24T21:59:40.598 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: ) 2021-03-24T21:59:40.598 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 89 2021-03-24T21:59:40.599 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: : 2021-03-24T21:59:40.599 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: cluster 2021-03-24T21:59:40.599 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 2021-03-24T21:59:40.599 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: [DBG] 2021-03-24T21:59:40.600 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 2021-03-24T21:59:40.600 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: purged_snaps scrub starts 2021-03-24T21:59:40.600 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 2021-03-24T21:59:40.600 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: cluster 2021-03-24T21:59:40.600 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 2021-03-24T21:59:40.601 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 2021 2021-03-24T21:59:40.601 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: - 2021-03-24T21:59:40.601 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 03 2021-03-24T21:59:40.601 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: - 2021-03-24T21:59:40.601 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 24 2021-03-24T21:59:40.602 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: T 2021-03-24T21:59:40.602 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 21 2021-03-24T21:59:40.602 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: : 2021-03-24T21:59:40.602 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 59 2021-03-24T21:59:40.603 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: : 2021-03-24T21:59:40.603 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 39 2021-03-24T21:59:40.603 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: . 2021-03-24T21:59:40.603 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 345283 2021-03-24T21:59:40.604 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: +0000 2021-03-24T21:59:40.604 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 2021-03-24T21:59:40.604 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: osd.7 2021-03-24T21:59:40.604 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: ( 2021-03-24T21:59:40.605 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: osd 2021-03-24T21:59:40.605 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: . 2021-03-24T21:59:40.605 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 7 2021-03-24T21:59:40.605 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: ) 2021-03-24T21:59:40.605 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 90 2021-03-24T21:59:40.606 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: : 2021-03-24T21:59:40.606 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: cluster 2021-03-24T21:59:40.606 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 2021-03-24T21:59:40.606 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: [DBG] 2021-03-24T21:59:40.606 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 2021-03-24T21:59:40.607 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: purged_snaps scrub ok 2021-03-24T21:59:40.607 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 2021-03-24T21:59:40.607 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: cluster 2021-03-24T21:59:40.607 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 2021-03-24T21:59:40.608 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 2021 2021-03-24T21:59:40.608 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: - 2021-03-24T21:59:40.608 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 03 2021-03-24T21:59:40.608 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: - 2021-03-24T21:59:40.608 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 24 2021-03-24T21:59:40.609 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: T 2021-03-24T21:59:40.609 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 21 2021-03-24T21:59:40.609 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: : 2021-03-24T21:59:40.609 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 59 2021-03-24T21:59:40.609 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: : 2021-03-24T21:59:40.610 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 39 2021-03-24T21:59:40.610 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: . 2021-03-24T21:59:40.610 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 416803 2021-03-24T21:59:40.610 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: +0000 2021-03-24T21:59:40.610 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 2021-03-24T21:59:40.611 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: mon.a 2021-03-24T21:59:40.611 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: ( 2021-03-24T21:59:40.611 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: mon 2021-03-24T21:59:40.611 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: . 2021-03-24T21:59:40.612 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 0 2021-03-24T21:59:40.612 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: ) 2021-03-24T21:59:40.612 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 1209 2021-03-24T21:59:40.612 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: : 2021-03-24T21:59:40.612 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: cluster 2021-03-24T21:59:40.613 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 2021-03-24T21:59:40.613 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: [DBG] 2021-03-24T21:59:40.613 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 2021-03-24T21:59:40.613 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: osdmap e409: 8 total, 7 up, 6 in 2021-03-24T21:59:40.613 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 2021-03-24T21:59:40.614 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: cluster 2021-03-24T21:59:40.614 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 2021-03-24T21:59:40.614 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 2021 2021-03-24T21:59:40.614 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: - 2021-03-24T21:59:40.614 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 03 2021-03-24T21:59:40.615 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: - 2021-03-24T21:59:40.615 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 24 2021-03-24T21:59:40.615 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: T 2021-03-24T21:59:40.615 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 21 2021-03-24T21:59:40.616 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: : 2021-03-24T21:59:40.616 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 59 2021-03-24T21:59:40.616 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: : 2021-03-24T21:59:40.616 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 40 2021-03-24T21:59:40.616 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: . 2021-03-24T21:59:40.617 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 208909 2021-03-24T21:59:40.617 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: +0000 2021-03-24T21:59:40.617 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 2021-03-24T21:59:40.617 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: mon.a 2021-03-24T21:59:40.617 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: ( 2021-03-24T21:59:40.618 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: mon 2021-03-24T21:59:40.618 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: . 2021-03-24T21:59:40.618 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 0 2021-03-24T21:59:40.618 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: ) 2021-03-24T21:59:40.619 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 1210 2021-03-24T21:59:40.619 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: : 2021-03-24T21:59:40.619 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: cluster 2021-03-24T21:59:40.619 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 2021-03-24T21:59:40.619 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: [WRN] 2021-03-24T21:59:40.620 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 2021-03-24T21:59:40.620 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: Health check update: Degraded data redundancy: 240/1440 objects degraded (16.667%), 35 pgs degraded, 81 pgs undersized (PG_DEGRADED) 2021-03-24T21:59:40.620 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:59:40.620 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: :59:40.221767+0000 mon.a (mon.0 2021-03-24T21:59:40.620 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: ) 1211 : cluster [DBG] osdmap e410: 8 total, 7 up, 6 in 2021-03-24T21:59:40.621 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: audit 2021- 2021-03-24T21:59:40.621 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: 03-24T21:59:40.340788+0000 mon.c ( 2021-03-24T21:59:40.621 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: mon.2) 64 : audit [INF] from='client.? v1:172.21.15.125:0/768534752' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T21:59:40.621 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: audit 2021 2021-03-24T21:59:40.621 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: -03-24T21:59:40.341398+0000 2021-03-24T21:59:40.622 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: mon.a (mon.0) 1212 : audit [INF] 2021-03-24T21:59:40.622 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:40 smithi125 conmon[19936]: from='client.35198 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T21:59:40.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:40 smithi142 conmon[20125]: cluster 2021-03-24T21:59:38 2021-03-24T21:59:40.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:40 smithi142 conmon[20125]: .833420+0000 mgr.y (mgr.14144) 475 : cluster [DBG] pgmap v775: 178 pgs: 4 peering, 5 active+clean+remapped, 52 active+undersized, 35 active+undersized+degraded, 82 active+clean; 372 MiB data, 879 MiB used, 536 GiB / 536 GiB avail; 0 B/s wr, 0 op/s; 240/1440 objects degraded (16.667%); 12/1440 objects misplaced (0.833%); 222 B/s, 2 objects/s recovering 2021-03-24T21:59:40.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:40 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:59:40.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:40 smithi142 conmon[20125]: 21:59:39.344607+0000 osd.7 (osd.7) 89 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:59:40.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:40 smithi142 conmon[20125]: cluster 2021-03-24T21:59:39.345283+0000 osd.7 (osd 2021-03-24T21:59:40.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:40 smithi142 conmon[20125]: .7) 90 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:59:40.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:40 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:59:40.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:40 smithi142 conmon[20125]: 59:39.416803+0000 mon.a (mon.0) 1209 : cluster 2021-03-24T21:59:40.625 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:40 smithi142 conmon[20125]: [DBG] osdmap e409: 8 total, 7 up, 6 in 2021-03-24T21:59:40.625 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:40 smithi142 conmon[20125]: cluster 2021-03-24T21:59:40.625 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:40 smithi142 conmon[20125]: 2021-03-24T21:59:40.208909+0000 mon.a (mon.0) 1210 2021-03-24T21:59:40.625 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:40 smithi142 conmon[20125]: : cluster [WRN] Health check update: Degraded data redundancy: 240/1440 objects degraded (16.667%), 35 pgs degraded, 81 pgs undersized (PG_DEGRADED) 2021-03-24T21:59:40.625 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:40 smithi142 conmon[20125]: cluster 2021-03-24T21:59:40.221767+0000 2021-03-24T21:59:40.626 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:40 smithi142 conmon[20125]: mon.a (mon.0) 1211 : cluster [DBG] osdmap e410: 8 total, 7 up, 6 in 2021-03-24T21:59:40.626 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:40 smithi142 conmon[20125]: audit 2021-03-24 2021-03-24T21:59:40.626 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:40 smithi142 conmon[20125]: T21:59:40.340788+0000 mon.c (mon.2) 2021-03-24T21:59:40.626 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:40 smithi142 conmon[20125]: 64 : audit [INF] from='client.? v1:172.21.15.125:0/768534752' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T21:59:40.626 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:40 smithi142 conmon[20125]: audit 2021-03-24T21:59:40.341398+0000 2021-03-24T21:59:40.627 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:40 smithi142 conmon[20125]: mon.a (mon.0) 1212 : audit [INF] 2021-03-24T21:59:40.627 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:40 smithi142 conmon[20125]: from='client.35198 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T21:59:40.659 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:40.720 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:40.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:40.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:59:40.826 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:40.874 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:40.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:40.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:59:40.980 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:41.034 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:41.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:41.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:59:41.140 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:41.203 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:41.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:41.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:59:41.231 INFO:tasks.rados.rados.0.smithi125.stdout:1507: done (2 left) 2021-03-24T21:59:41.232 INFO:tasks.rados.rados.0.smithi125.stdout:1509: done (1 left) 2021-03-24T21:59:41.232 INFO:tasks.rados.rados.0.smithi125.stdout:1510: done (0 left) 2021-03-24T21:59:41.232 INFO:tasks.rados.rados.0.smithi125.stdout:1511: snap_create 2021-03-24T21:59:41.266 INFO:teuthology.orchestra.run.smithi125.stderr:noscrub is unset 2021-03-24T21:59:41.310 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:41.398 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:41.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:41.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:41.505 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:41.554 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:41.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:41.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:41.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: audit 2021-03-24T21:59:41.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 2021-03-24T21:59:41.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 2021 2021-03-24T21:59:41.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: - 2021-03-24T21:59:41.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 03 2021-03-24T21:59:41.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: - 2021-03-24T21:59:41.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 24 2021-03-24T21:59:41.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: T 2021-03-24T21:59:41.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 21 2021-03-24T21:59:41.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: : 2021-03-24T21:59:41.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 59 2021-03-24T21:59:41.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: : 2021-03-24T21:59:41.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 40 2021-03-24T21:59:41.657 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: . 2021-03-24T21:59:41.657 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 837922 2021-03-24T21:59:41.657 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: +0000 2021-03-24T21:59:41.657 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 2021-03-24T21:59:41.657 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: mon.a 2021-03-24T21:59:41.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: ( 2021-03-24T21:59:41.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: mon 2021-03-24T21:59:41.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: . 2021-03-24T21:59:41.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 0 2021-03-24T21:59:41.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: ) 2021-03-24T21:59:41.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 1213 2021-03-24T21:59:41.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: : 2021-03-24T21:59:41.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: audit 2021-03-24T21:59:41.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 2021-03-24T21:59:41.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: [INF] 2021-03-24T21:59:41.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 2021-03-24T21:59:41.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "31"}]: dispatch 2021-03-24T21:59:41.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 2021-03-24T21:59:41.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: cluster 2021-03-24T21:59:41.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 2021-03-24T21:59:41.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 2021 2021-03-24T21:59:41.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: - 2021-03-24T21:59:41.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 03 2021-03-24T21:59:41.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: - 2021-03-24T21:59:41.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 24 2021-03-24T21:59:41.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: T 2021-03-24T21:59:41.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 21 2021-03-24T21:59:41.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: : 2021-03-24T21:59:41.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 59 2021-03-24T21:59:41.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: : 2021-03-24T21:59:41.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 41 2021-03-24T21:59:41.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: . 2021-03-24T21:59:41.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 211465 2021-03-24T21:59:41.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: +0000 2021-03-24T21:59:41.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 2021-03-24T21:59:41.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: mon.a 2021-03-24T21:59:41.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: ( 2021-03-24T21:59:41.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: mon 2021-03-24T21:59:41.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: . 2021-03-24T21:59:41.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 0 2021-03-24T21:59:41.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: ) 2021-03-24T21:59:41.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 1214 2021-03-24T21:59:41.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: : 2021-03-24T21:59:41.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: cluster 2021-03-24T21:59:41.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 2021-03-24T21:59:41.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: [WRN] 2021-03-24T21:59:41.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 2021-03-24T21:59:41.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:59:41.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 2021-03-24T21:59:41.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: audit 2021-03-24T21:59:41.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 2021-03-24T21:59:41.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 2021 2021-03-24T21:59:41.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: - 2021-03-24T21:59:41.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 03 2021-03-24T21:59:41.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: - 2021-03-24T21:59:41.669 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 24 2021-03-24T21:59:41.669 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: T 2021-03-24T21:59:41.669 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 21 2021-03-24T21:59:41.669 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: : 2021-03-24T21:59:41.669 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 59 2021-03-24T21:59:41.670 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: : 2021-03-24T21:59:41.670 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 41 2021-03-24T21:59:41.670 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: . 2021-03-24T21:59:41.670 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 222802 2021-03-24T21:59:41.671 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: +0000 2021-03-24T21:59:41.671 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 2021-03-24T21:59:41.671 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: mon.a 2021-03-24T21:59:41.671 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: ( 2021-03-24T21:59:41.672 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: mon 2021-03-24T21:59:41.672 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: . 2021-03-24T21:59:41.672 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 0 2021-03-24T21:59:41.672 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: ) 2021-03-24T21:59:41.672 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 1215 2021-03-24T21:59:41.673 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: : 2021-03-24T21:59:41.673 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: audit 2021-03-24T21:59:41.673 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 2021-03-24T21:59:41.674 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 2021 2021-03-24T21:59:41.674 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: - 2021-03-24T21:59:41.674 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 03 2021-03-24T21:59:41.675 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: - 2021-03-24T21:59:41.675 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 24 2021-03-24T21:59:41.675 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: T 2021-03-24T21:59:41.675 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 21 2021-03-24T21:59:41.675 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: : 2021-03-24T21:59:41.676 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 59 2021-03-24T21:59:41.676 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: : 2021-03-24T21:59:41.676 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 40 2021-03-24T21:59:41.676 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: . 2021-03-24T21:59:41.677 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 837922 2021-03-24T21:59:41.677 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: +0000 2021-03-24T21:59:41.677 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 2021-03-24T21:59:41.677 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: mon.a 2021-03-24T21:59:41.677 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: ( 2021-03-24T21:59:41.678 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: mon 2021-03-24T21:59:41.678 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: . 2021-03-24T21:59:41.678 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 0 2021-03-24T21:59:41.678 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: ) 2021-03-24T21:59:41.678 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 1213 2021-03-24T21:59:41.679 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: : 2021-03-24T21:59:41.679 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: audit 2021-03-24T21:59:41.679 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 2021-03-24T21:59:41.679 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: [INF] 2021-03-24T21:59:41.679 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 2021-03-24T21:59:41.680 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "31"}]: dispatch 2021-03-24T21:59:41.680 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 2021-03-24T21:59:41.680 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: cluster 2021-03-24T21:59:41.680 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 2021-03-24T21:59:41.680 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 2021 2021-03-24T21:59:41.681 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: - 2021-03-24T21:59:41.681 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 03 2021-03-24T21:59:41.681 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: - 2021-03-24T21:59:41.681 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 24 2021-03-24T21:59:41.681 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: T 2021-03-24T21:59:41.682 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 21 2021-03-24T21:59:41.682 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: : 2021-03-24T21:59:41.682 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 59 2021-03-24T21:59:41.682 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: : 2021-03-24T21:59:41.683 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 41 2021-03-24T21:59:41.683 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: . 2021-03-24T21:59:41.683 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 211465 2021-03-24T21:59:41.683 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: +0000 2021-03-24T21:59:41.683 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 2021-03-24T21:59:41.684 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: mon.a 2021-03-24T21:59:41.684 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: ( 2021-03-24T21:59:41.684 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: mon 2021-03-24T21:59:41.684 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: . 2021-03-24T21:59:41.684 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 0 2021-03-24T21:59:41.685 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: ) 2021-03-24T21:59:41.685 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 1214 2021-03-24T21:59:41.685 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: : 2021-03-24T21:59:41.685 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: cluster 2021-03-24T21:59:41.685 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 2021-03-24T21:59:41.686 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: [WRN] 2021-03-24T21:59:41.686 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 2021-03-24T21:59:41.686 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:59:41.686 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 2021-03-24T21:59:41.687 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: audit 2021-03-24T21:59:41.687 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 2021-03-24T21:59:41.687 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 2021 2021-03-24T21:59:41.687 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: - 2021-03-24T21:59:41.687 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 03 2021-03-24T21:59:41.688 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: - 2021-03-24T21:59:41.688 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 24 2021-03-24T21:59:41.688 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: T 2021-03-24T21:59:41.688 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 21 2021-03-24T21:59:41.688 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: : 2021-03-24T21:59:41.689 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 59 2021-03-24T21:59:41.689 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: : 2021-03-24T21:59:41.689 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 41 2021-03-24T21:59:41.689 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: . 2021-03-24T21:59:41.689 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 222802 2021-03-24T21:59:41.690 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: +0000 2021-03-24T21:59:41.690 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 2021-03-24T21:59:41.690 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: mon.a 2021-03-24T21:59:41.690 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: ( 2021-03-24T21:59:41.690 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: mon 2021-03-24T21:59:41.691 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: . 2021-03-24T21:59:41.691 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 0 2021-03-24T21:59:41.691 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: ) 2021-03-24T21:59:41.691 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 1215 2021-03-24T21:59:41.692 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: : 2021-03-24T21:59:41.692 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: audit 2021-03-24T21:59:41.692 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 2021-03-24T21:59:41.692 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: [INF] 2021-03-24T21:59:41.693 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 2021-03-24T21:59:41.693 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: from='client.35198 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T21:59:41.693 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 2021-03-24T21:59:41.693 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: audit 2021-03-24T21:59:41.694 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 2021-03-24T21:59:41.694 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 2021 2021-03-24T21:59:41.694 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: - 2021-03-24T21:59:41.694 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 03 2021-03-24T21:59:41.694 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: - 2021-03-24T21:59:41.695 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 24 2021-03-24T21:59:41.695 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: T 2021-03-24T21:59:41.695 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 21 2021-03-24T21:59:41.695 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: : 2021-03-24T21:59:41.696 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 59 2021-03-24T21:59:41.696 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: : 2021-03-24T21:59:41.696 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 41 2021-03-24T21:59:41.696 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: . 2021-03-24T21:59:41.696 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 222970 2021-03-24T21:59:41.697 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: +0000 2021-03-24T21:59:41.697 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 2021-03-24T21:59:41.697 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: mon.a 2021-03-24T21:59:41.697 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: ( 2021-03-24T21:59:41.697 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: mon 2021-03-24T21:59:41.698 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: . 2021-03-24T21:59:41.698 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 0 2021-03-24T21:59:41.698 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: ) 2021-03-24T21:59:41.698 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 1216 2021-03-24T21:59:41.699 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: : 2021-03-24T21:59:41.699 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: audit 2021-03-24T21:59:41.699 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 2021-03-24T21:59:41.699 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: [INF] 2021-03-24T21:59:41.699 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 2021-03-24T21:59:41.700 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "31"}]': finished 2021-03-24T21:59:41.700 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 2021-03-24T21:59:41.700 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: cluster 2021-03-24T21:59:41.700 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 2021-03-24T21:59:41.700 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 2021 2021-03-24T21:59:41.701 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: - 2021-03-24T21:59:41.701 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 03 2021-03-24T21:59:41.701 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: - 2021-03-24T21:59:41.701 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 24 2021-03-24T21:59:41.701 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: T 2021-03-24T21:59:41.702 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 21 2021-03-24T21:59:41.702 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: : 2021-03-24T21:59:41.702 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 59 2021-03-24T21:59:41.702 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: : 2021-03-24T21:59:41.703 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 41 2021-03-24T21:59:41.703 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: . 2021-03-24T21:59:41.703 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 223065 2021-03-24T21:59:41.703 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: +0000 2021-03-24T21:59:41.703 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 2021-03-24T21:59:41.704 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: mon.a 2021-03-24T21:59:41.704 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: ( 2021-03-24T21:59:41.704 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: mon 2021-03-24T21:59:41.704 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: . 2021-03-24T21:59:41.704 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 0 2021-03-24T21:59:41.705 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: ) 2021-03-24T21:59:41.705 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 1217 2021-03-24T21:59:41.705 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: : 2021-03-24T21:59:41.705 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: cluster 2021-03-24T21:59:41.705 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 2021-03-24T21:59:41.706 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: [DBG] 2021-03-24T21:59:41.706 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 2021-03-24T21:59:41.706 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: osdmap e411: 8 total, 7 up, 6 in 2021-03-24T21:59:41.706 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[19936]: 2021-03-24T21:59:41.707 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: audit 2021-03-24T21:59:41.707 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 2021-03-24T21:59:41.707 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: [INF] 2021-03-24T21:59:41.707 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 2021-03-24T21:59:41.708 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: from='client.35198 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T21:59:41.708 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 2021-03-24T21:59:41.708 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: audit 2021-03-24T21:59:41.708 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 2021-03-24T21:59:41.709 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 2021 2021-03-24T21:59:41.709 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: - 2021-03-24T21:59:41.709 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 03 2021-03-24T21:59:41.709 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: - 2021-03-24T21:59:41.709 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 24 2021-03-24T21:59:41.710 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: T 2021-03-24T21:59:41.710 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 21 2021-03-24T21:59:41.710 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: : 2021-03-24T21:59:41.710 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 59 2021-03-24T21:59:41.710 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: : 2021-03-24T21:59:41.711 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 41 2021-03-24T21:59:41.711 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: . 2021-03-24T21:59:41.711 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 222970 2021-03-24T21:59:41.711 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: +0000 2021-03-24T21:59:41.712 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 2021-03-24T21:59:41.712 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: mon.a 2021-03-24T21:59:41.712 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: ( 2021-03-24T21:59:41.712 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: mon 2021-03-24T21:59:41.712 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: . 2021-03-24T21:59:41.713 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 0 2021-03-24T21:59:41.713 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: ) 2021-03-24T21:59:41.713 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 1216 2021-03-24T21:59:41.713 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: : 2021-03-24T21:59:41.713 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: audit 2021-03-24T21:59:41.714 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 2021-03-24T21:59:41.714 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: [INF] 2021-03-24T21:59:41.714 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 2021-03-24T21:59:41.714 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "31"}]': finished 2021-03-24T21:59:41.715 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 2021-03-24T21:59:41.715 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: cluster 2021-03-24T21:59:41.715 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 2021-03-24T21:59:41.715 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 2021 2021-03-24T21:59:41.715 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: - 2021-03-24T21:59:41.716 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 03 2021-03-24T21:59:41.716 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: - 2021-03-24T21:59:41.716 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 24 2021-03-24T21:59:41.716 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: T 2021-03-24T21:59:41.716 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 21 2021-03-24T21:59:41.717 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: : 2021-03-24T21:59:41.717 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 59 2021-03-24T21:59:41.717 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: : 2021-03-24T21:59:41.717 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 41 2021-03-24T21:59:41.718 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: . 2021-03-24T21:59:41.718 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 223065 2021-03-24T21:59:41.718 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: +0000 2021-03-24T21:59:41.718 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 2021-03-24T21:59:41.718 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: mon.a 2021-03-24T21:59:41.719 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: ( 2021-03-24T21:59:41.719 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: mon 2021-03-24T21:59:41.719 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: . 2021-03-24T21:59:41.719 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 0 2021-03-24T21:59:41.719 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: ) 2021-03-24T21:59:41.720 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 1217 2021-03-24T21:59:41.720 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: : 2021-03-24T21:59:41.720 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: cluster 2021-03-24T21:59:41.720 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 2021-03-24T21:59:41.720 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: [DBG] 2021-03-24T21:59:41.721 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 2021-03-24T21:59:41.721 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: osdmap e411: 8 total, 7 up, 6 in 2021-03-24T21:59:41.721 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:41 smithi125 conmon[24927]: 2021-03-24T21:59:41.721 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:59:41.722 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:41.769 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:41.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:41.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:59:41.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:41 smithi142 conmon[20125]: audit 2021-03-24T21:59:40 2021-03-24T21:59:41.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:41 smithi142 conmon[20125]: .837922+0000 mon.a (mon.0) 1213 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "31"}]: dispatch 2021-03-24T21:59:41.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:41 smithi142 conmon[20125]: cluster 2021-03-24T21:59: 2021-03-24T21:59:41.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:41 smithi142 conmon[20125]: 41.211465+0000 mon.a (mon.0) 1214 : cluster 2021-03-24T21:59:41.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:41 smithi142 conmon[20125]: [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T21:59:41.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:41 smithi142 conmon[20125]: audit 2021 2021-03-24T21:59:41.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:41 smithi142 conmon[20125]: -03-24T21:59:41.222802+0000 2021-03-24T21:59:41.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:41 smithi142 conmon[20125]: mon.a (mon.0) 1215 : audit [INF] 2021-03-24T21:59:41.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:41 smithi142 conmon[20125]: from='client.35198 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T21:59:41.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:41 smithi142 conmon[20125]: audit 2021-03-24T21:59 2021-03-24T21:59:41.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:41 smithi142 conmon[20125]: :41.222970+0000 mon.a (mon.0) 1216 : 2021-03-24T21:59:41.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:41 smithi142 conmon[20125]: audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "31"}]': finished 2021-03-24T21:59:41.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:41 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:59:41.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:41 smithi142 conmon[20125]: -03-24T21:59: 2021-03-24T21:59:41.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:41 smithi142 conmon[20125]: 41.223065+0000 mon.a (mon. 2021-03-24T21:59:41.872 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:41 smithi142 conmon[20125]: 0) 1217 : cluster [DBG] osdmap e411: 8 total, 7 up, 6 in 2021-03-24T21:59:41.872 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:41 smithi142 conmon[20125]: 2021-03-24T21:59:41.876 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:41.931 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:41.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:41.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:41.948 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.3 dump_historic_ops 2021-03-24T21:59:42.038 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:42.088 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:42.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:42.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:42.194 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:42.229 INFO:tasks.rados.rados.0.smithi125.stdout:1511: done (0 left) 2021-03-24T21:59:42.229 INFO:tasks.rados.rados.0.smithi125.stdout:1512: delete oid 5 current snap is 164 2021-03-24T21:59:42.238 INFO:tasks.rados.rados.0.smithi125.stdout:1512: done (0 left) 2021-03-24T21:59:42.238 INFO:tasks.rados.rados.0.smithi125.stdout:1513: snap_create 2021-03-24T21:59:42.251 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:42.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:42.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:59:42.368 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:42.413 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:42.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:42.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:59:42.517 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:42.526 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:42 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:59:42.526 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:42 smithi125 conmon[19936]: :59:40.835666+0000 mgr.y (mgr.14144) 476 : cluster [DBG] pgmap v778: 178 pgs: 9 active+clean+remapped, 52 active+undersized, 35 active+undersized+degraded, 82 active+clean; 373 MiB data, 890 MiB used, 536 GiB / 536 GiB avail; 21 MiB/s rd, 958 KiB/s wr, 34 op/s; 241/1485 objects degraded (16.229%); 45/1485 objects misplaced (3.030%); 1.2 KiB/s, 13 objects/s recovering 2021-03-24T21:59:42.527 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:42 smithi125 conmon[19936]: cluster 2021-03-24T21:59:42.527 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:42 smithi125 conmon[19936]: 2021-03-24T21:59: 2021-03-24T21:59:42.527 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:42 smithi125 conmon[19936]: 41.414897+0000 mon.a (mon 2021-03-24T21:59:42.527 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:42 smithi125 conmon[19936]: .0) 1218 : cluster [INF] 2021-03-24T21:59:42.528 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:42 smithi125 conmon[19936]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2021-03-24T21:59:42.528 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:42 smithi125 conmon[19936]: cluster 2021 2021-03-24T21:59:42.528 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:42 smithi125 conmon[19936]: -03-24T21:59: 2021-03-24T21:59:42.528 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:42 smithi125 conmon[19936]: 42.220283+0000 mon.a (mon 2021-03-24T21:59:42.528 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:42 smithi125 conmon[19936]: .0) 1219 : cluster [DBG] 2021-03-24T21:59:42.529 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:42 smithi125 conmon[19936]: osdmap e412: 8 total, 7 up, 6 in 2021-03-24T21:59:42.529 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:42 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:59:42.529 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:42 smithi125 conmon[24927]: T21:59:40.835666+0000 mgr.y (mgr.14144) 476 : cluster [DBG] 2021-03-24T21:59:42.529 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:42 smithi125 conmon[24927]: pgmap v778: 178 pgs: 9 active+clean+remapped, 52 active+undersized, 35 active+undersized+degraded, 82 active+clean; 373 MiB data, 890 MiB used, 536 GiB / 536 GiB avail; 21 MiB/s rd, 958 KiB/s wr, 34 op/s; 241/1485 objects degraded (16.229%); 45/1485 objects misplaced (3.030%); 1.2 KiB/s, 13 objects/s recovering 2021-03-24T21:59:42.530 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:42 smithi125 conmon[24927]: cluster 2021-03-24T21:59:42.530 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:42 smithi125 conmon[24927]: 2021-03- 2021-03-24T21:59:42.530 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:42 smithi125 conmon[24927]: 24T21:59:41.414897+0000 mon.a (mon.0) 1218 2021-03-24T21:59:42.530 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:42 smithi125 conmon[24927]: : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2021-03-24T21:59:42.531 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:42 smithi125 conmon[24927]: cluster 2021-03-24T21:59:42.531 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:42 smithi125 conmon[24927]: 2021-03- 2021-03-24T21:59:42.531 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:42 smithi125 conmon[24927]: 24T21:59:42.220283+0000 mon.a (mon.0) 1219 : cluster [DBG] osdmap e412: 8 total, 7 up, 6 in 2021-03-24T21:59:42.581 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:42.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:42.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:59:42.690 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:42.734 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:42.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:42.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:59:42.839 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:42.851 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:42 smithi142 conmon[20125]: cluster 2021-03-24T21:59: 2021-03-24T21:59:42.851 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:42 smithi142 conmon[20125]: 40.835666+0000 mgr.y (mgr.14144) 476 : cluster [DBG] pgmap v778: 178 pgs: 9 active+clean+remapped, 52 active+undersized, 35 active+undersized+degraded, 82 active+clean; 373 MiB data, 890 MiB used, 536 GiB / 536 GiB avail; 21 MiB/s rd, 958 KiB/s wr, 34 op/s; 241/1485 objects degraded (16.229%); 45/1485 objects misplaced (3.030%); 1.2 KiB/s, 13 objects/s recovering 2021-03-24T21:59:42.851 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:42 smithi142 conmon[20125]: cluster 2021-03-24T21:59:42.852 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:42 smithi142 conmon[20125]: 2021-03-24T21 2021-03-24T21:59:42.852 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:42 smithi142 conmon[20125]: :59:41.414897 2021-03-24T21:59:42.852 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:42 smithi142 conmon[20125]: +0000 mon.a (mon. 2021-03-24T21:59:42.852 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:42 smithi142 conmon[20125]: 0) 1218 : cluster 2021-03-24T21:59:42.852 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:42 smithi142 conmon[20125]: [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2021-03-24T21:59:42.853 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:42 smithi142 conmon[20125]: cluster 2021-03-24T21:59:42.853 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:42 smithi142 conmon[20125]: 2021-03-24T21 2021-03-24T21:59:42.853 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:42 smithi142 conmon[20125]: :59:42.220283 2021-03-24T21:59:42.853 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:42 smithi142 conmon[20125]: +0000 mon.a (mon. 2021-03-24T21:59:42.854 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:42 smithi142 conmon[20125]: 0) 1219 : cluster 2021-03-24T21:59:42.854 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:42 smithi142 conmon[20125]: [DBG] osdmap e412: 8 total, 7 up, 6 in 2021-03-24T21:59:42.894 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:42.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:42.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:59:43.001 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:43.060 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:43.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:43.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:59:43.165 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:43.227 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:43.227 INFO:tasks.rados.rados.0.smithi125.stdout:1513: done (0 left) 2021-03-24T21:59:43.228 INFO:tasks.rados.rados.0.smithi125.stdout:1514: read oid 18 snap -1 2021-03-24T21:59:43.228 INFO:tasks.rados.rados.0.smithi125.stdout:1514: expect (ObjNum 534 snap 162 seq_num 534) 2021-03-24T21:59:43.228 INFO:tasks.rados.rados.0.smithi125.stdout:1515: snap_create 2021-03-24T21:59:43.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:43.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:59:43.249 INFO:tasks.rados.rados.0.smithi125.stdout:1514: expect (ObjNum 534 snap 162 seq_num 534) 2021-03-24T21:59:43.333 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:43.389 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:43.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:43.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:59:43.497 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:43.535 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:43 smithi125 conmon[19936]: cluster 2021-03-24T21:59:41.839843+0000 osd.7 2021-03-24T21:59:43.535 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:43 smithi125 conmon[19936]: (osd.7) 91 : cluster [DBG] 5.10 scrub ok 2021-03-24T21:59:43.535 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:43 smithi125 conmon[19936]: cluster 2021-03-24T21:59:42.813878+0000 osd.2 (osd.2) 72 : cluster [DBG] 6.8 scrub ok 2021-03-24T21:59:43.536 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:43 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:59:43.536 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:43 smithi125 conmon[19936]: 21:59:43.223654+0000 mon.a (mon.0) 1220 : cluster [DBG] osdmap e413: 8 total, 7 up, 6 in 2021-03-24T21:59:43.536 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:43 smithi125 conmon[24927]: cluster 2021-03-24T21:59:41.839843 2021-03-24T21:59:43.536 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:43 smithi125 conmon[24927]: +0000 osd.7 (osd.7) 91 : cluster [DBG] 5.10 scrub ok 2021-03-24T21:59:43.537 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:43 smithi125 conmon[24927]: cluster 2021-03-24T21:59:42.813878+0000 osd.2 (osd.2) 72 : cluster [DBG] 6.8 scrub ok 2021-03-24T21:59:43.537 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:43 smithi125 conmon[24927]: cluster 2021-03-24T21:59:43.223654+0000 mon.a (mon.0) 1220 : cluster [DBG] osdmap e413: 8 total, 7 up, 6 in 2021-03-24T21:59:43.548 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:43.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:43.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:59:43.654 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:43.704 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:59:43.709 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:43.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:43.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:59:43.815 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:43.829 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:43 smithi142 conmon[20125]: cluster 2021-03-24T21 2021-03-24T21:59:43.830 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:43 smithi142 conmon[20125]: :59:41.839843+0000 osd.7 (osd.7) 91 : cluster [DBG] 5.10 scrub ok 2021-03-24T21:59:43.830 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:43 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:59:43.831 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:43 smithi142 conmon[20125]: 24T21:59:42.813878+0000 osd.2 (osd.2) 72 : cluster [DBG] 6.8 scrub ok 2021-03-24T21:59:43.831 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:43 smithi142 conmon[20125]: 2021-03-24T21:59:43.832 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:43 smithi142 conmon[20125]: cluster 2021-03-24T21:59:43.832 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:43 smithi142 conmon[20125]: 2021-03-24T21:59:43 2021-03-24T21:59:43.832 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:43 smithi142 conmon[20125]: .223654+0000 mon.a (mon.0 2021-03-24T21:59:43.833 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:43 smithi142 conmon[20125]: ) 1220 : cluster [DBG] osdmap e413: 8 total, 7 up, 6 in 2021-03-24T21:59:43.870 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:43.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:43.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:59:43.977 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:44.029 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:44.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:44.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:59:44.141 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:44.203 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:44.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:44.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:59:44.230 INFO:tasks.rados.rados.0.smithi125.stdout:1514: done (1 left) 2021-03-24T21:59:44.230 INFO:tasks.rados.rados.0.smithi125.stdout:1515: done (0 left) 2021-03-24T21:59:44.231 INFO:tasks.rados.rados.0.smithi125.stdout:1516: snap_remove snap 153 2021-03-24T21:59:44.310 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:44.354 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:44.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:44.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:59:44.460 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:44.509 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:44.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:44.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:44.613 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:44.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:44 smithi142 conmon[20125]: cluster 2021-03-24T21:59: 2021-03-24T21:59:44.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:44 smithi142 conmon[20125]: 42.837183+0000 mgr.y (mgr.14144) 477 : cluster [DBG] pgmap v781: 178 pgs: 9 active+clean+remapped, 52 active+undersized, 35 active+undersized+degraded, 82 active+clean; 373 MiB data, 891 MiB used, 536 GiB / 536 GiB avail; 21 MiB/s rd, 958 KiB/s wr, 34 op/s; 241/1485 objects degraded (16.229%); 45/1485 objects misplaced (3.030%); 1.0 KiB/s, 10 objects/s recovering 2021-03-24T21:59:44.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:44 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:59:44.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:44 smithi142 conmon[20125]: -03-24T21:59:42.879238+0000 osd.7 (osd. 2021-03-24T21:59:44.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:44 smithi142 conmon[20125]: 7) 92 : cluster [DBG] 3.26 scrub ok 2021-03-24T21:59:44.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:44 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:59:44.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:44 smithi142 conmon[20125]: -03-24T21:59:44. 2021-03-24T21:59:44.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:44 smithi142 conmon[20125]: 226385+0000 mon.a (mon.0) 1221 : 2021-03-24T21:59:44.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:44 smithi142 conmon[20125]: cluster [DBG] osdmap e414: 8 total, 7 up, 6 in 2021-03-24T21:59:44.658 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:44.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:44.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:59:44.765 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:44.779 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:44 smithi125 conmon[19936]: cluster 2021-03-24T21:59:42. 2021-03-24T21:59:44.780 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:44 smithi125 conmon[19936]: 837183+0000 mgr.y (mgr.14144) 477 : cluster [DBG] pgmap v781: 178 pgs: 9 active+clean+remapped, 52 active+undersized, 35 active+undersized+degraded, 82 active+clean; 373 MiB data, 891 MiB used, 536 GiB / 536 GiB avail; 21 MiB/s rd, 958 KiB/s wr, 34 op/s; 241/1485 objects degraded (16.229%); 45/1485 objects misplaced (3.030%); 1.0 KiB/s, 10 objects/s recovering 2021-03-24T21:59:44.780 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:44 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:59:44.780 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:44 smithi125 conmon[19936]: 59:42.879238+0000 osd.7 (osd.7) 92 : cluster [DBG] 3.26 scrub ok 2021-03-24T21:59:44.780 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:44 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:59:44.781 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:44 smithi125 conmon[19936]: 03-24T21:59:44.226385+0000 2021-03-24T21:59:44.781 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:44 smithi125 conmon[19936]: mon.a (mon.0) 1221 : cluster [DBG] 2021-03-24T21:59:44.781 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:44 smithi125 conmon[19936]: osdmap e414: 8 total, 7 up, 6 in 2021-03-24T21:59:44.781 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:44 smithi125 conmon[24927]: cluster 2021-03-24T21:59:42.837183+0000 2021-03-24T21:59:44.782 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:44 smithi125 conmon[24927]: mgr.y (mgr.14144) 477 : cluster [DBG] pgmap v781: 178 pgs: 9 active+clean+remapped, 52 active+undersized, 35 active+undersized+degraded, 82 active+clean; 373 MiB data, 891 MiB used, 536 GiB / 536 GiB avail; 21 MiB/s rd, 958 KiB/s wr, 34 op/s; 241/1485 objects degraded (16.229%); 45/1485 objects misplaced (3.030%); 1.0 KiB/s, 10 objects/s recovering 2021-03-24T21:59:44.782 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:44 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:59:44.782 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:44 smithi125 conmon[24927]: :59:42.879238+0000 osd.7 (osd.7) 92 : cluster [DBG] 3.26 scrub ok 2021-03-24T21:59:44.782 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:44 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:59:44.783 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:44 smithi125 conmon[24927]: 24T21:59:44.226385+0000 mon.a (mon 2021-03-24T21:59:44.783 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:44 smithi125 conmon[24927]: .0) 1221 : cluster [DBG] osdmap e414: 8 total, 7 up, 6 in 2021-03-24T21:59:44.821 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:44.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:44.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:59:44.929 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:44.991 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:44.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:44.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:59:45.098 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:45.146 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:45.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:45.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:59:45.248 INFO:tasks.rados.rados.0.smithi125.stdout:1516: done (0 left) 2021-03-24T21:59:45.248 INFO:tasks.rados.rados.0.smithi125.stdout:1517: read oid 20 snap -1 2021-03-24T21:59:45.249 INFO:tasks.rados.rados.0.smithi125.stdout:1517: expect (ObjNum 506 snap 147 seq_num 506) 2021-03-24T21:59:45.249 INFO:tasks.rados.rados.0.smithi125.stdout:1518: write (excl) oid 37 current snap is 166 2021-03-24T21:59:45.249 INFO:tasks.rados.rados.0.smithi125.stdout:1518: seq_num 537 ranges {625359=553423,1790948=567184,2930500=94859} 2021-03-24T21:59:45.251 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:45.255 INFO:tasks.rados.rados.0.smithi125.stdout:1518: writing smithi12556470-37 from 625359 to 1178782 tid 1 2021-03-24T21:59:45.262 INFO:tasks.rados.rados.0.smithi125.stdout:1518: writing smithi12556470-37 from 1790948 to 2358132 tid 2 2021-03-24T21:59:45.267 INFO:tasks.rados.rados.0.smithi125.stdout:1518: writing smithi12556470-37 from 2930500 to 3025359 tid 3 2021-03-24T21:59:45.267 INFO:tasks.rados.rados.0.smithi125.stdout:1519: rollback oid 1 current snap is 166 2021-03-24T21:59:45.267 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 1 to 149 2021-03-24T21:59:45.267 INFO:tasks.rados.rados.0.smithi125.stdout:1520: delete oid 19 current snap is 166 2021-03-24T21:59:45.293 INFO:tasks.rados.rados.0.smithi125.stdout:1517: expect (ObjNum 506 snap 147 seq_num 506) 2021-03-24T21:59:45.362 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:45.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:45.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:45.467 INFO:tasks.rados.rados.0.smithi125.stdout:1518: finishing write tid 1 to smithi12556470-37 2021-03-24T21:59:45.467 INFO:tasks.rados.rados.0.smithi125.stdout:1519: finishing rollback tid 0 to smithi12556470-1 2021-03-24T21:59:45.468 INFO:tasks.rados.rados.0.smithi125.stdout:1519: finishing rollback tid 1 to smithi12556470-1 2021-03-24T21:59:45.468 INFO:tasks.rados.rados.0.smithi125.stdout:1519: finishing rollback tid 2 to smithi12556470-1 2021-03-24T21:59:45.468 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 1 v 476 (ObjNum 497 snap 145 seq_num 497) dirty exists 2021-03-24T21:59:45.468 INFO:tasks.rados.rados.0.smithi125.stdout:1518: finishing write tid 2 to smithi12556470-37 2021-03-24T21:59:45.469 INFO:tasks.rados.rados.0.smithi125.stdout:1518: finishing write tid 3 to smithi12556470-37 2021-03-24T21:59:45.469 INFO:tasks.rados.rados.0.smithi125.stdout:1518: finishing write tid 4 to smithi12556470-37 2021-03-24T21:59:45.469 INFO:tasks.rados.rados.0.smithi125.stdout:1518: finishing write tid 5 to smithi12556470-37 2021-03-24T21:59:45.469 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 37 v 348 (ObjNum 537 snap 166 seq_num 537) dirty exists 2021-03-24T21:59:45.469 INFO:tasks.rados.rados.0.smithi125.stdout:1518: left oid 37 (ObjNum 537 snap 166 seq_num 537) 2021-03-24T21:59:45.470 INFO:tasks.rados.rados.0.smithi125.stdout:1517: done (3 left) 2021-03-24T21:59:45.470 INFO:tasks.rados.rados.0.smithi125.stdout:1518: done (2 left) 2021-03-24T21:59:45.470 INFO:tasks.rados.rados.0.smithi125.stdout:1519: done (1 left) 2021-03-24T21:59:45.470 INFO:tasks.rados.rados.0.smithi125.stdout:1520: done (0 left) 2021-03-24T21:59:45.471 INFO:tasks.rados.rados.0.smithi125.stdout:1521: snap_remove snap 163 2021-03-24T21:59:45.471 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:45.516 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:45.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:45.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:59:45.621 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:45.633 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: cluster 2021-03-24T21:59:45.633 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: 2021-03-24T21:59:45.633 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: 2021 2021-03-24T21:59:45.633 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: - 2021-03-24T21:59:45.634 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: 03 2021-03-24T21:59:45.634 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: - 2021-03-24T21:59:45.634 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: 24 2021-03-24T21:59:45.634 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: T 2021-03-24T21:59:45.635 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: 21 2021-03-24T21:59:45.635 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: : 2021-03-24T21:59:45.635 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: 59 2021-03-24T21:59:45.635 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: : 2021-03-24T21:59:45.636 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: 43 2021-03-24T21:59:45.636 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: . 2021-03-24T21:59:45.636 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: 929576 2021-03-24T21:59:45.636 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: +0000 2021-03-24T21:59:45.637 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: 2021-03-24T21:59:45.637 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: osd.0 2021-03-24T21:59:45.637 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: ( 2021-03-24T21:59:45.638 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: osd 2021-03-24T21:59:45.638 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: . 2021-03-24T21:59:45.638 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: 0 2021-03-24T21:59:45.638 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: ) 2021-03-24T21:59:45.638 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: 92 2021-03-24T21:59:45.639 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: : 2021-03-24T21:59:45.639 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: cluster 2021-03-24T21:59:45.639 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: 2021-03-24T21:59:45.639 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: [DBG] 2021-03-24T21:59:45.640 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: 2021-03-24T21:59:45.640 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: 7.1b scrub ok 2021-03-24T21:59:45.640 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: 2021-03-24T21:59:45.640 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: cluster 2021-03-24T21:59:45.641 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: 2021-03-24T21:59:45.641 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: 2021 2021-03-24T21:59:45.642 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: - 2021-03-24T21:59:45.642 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: 03-24T21:59:44.665230+0000 2021-03-24T21:59:45.642 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: osd.4 (osd.4) 101 : cluster [DBG] 4.1e scrub ok 2021-03-24T21:59:45.643 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: cluster 2021-03-24T21:59:45.643 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: 2021-03-24T21:59:44.828971+0000 osd.2 (osd.2) 73 : cluster 2021-03-24T21:59:45.644 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: [DBG] 7.8 scrub ok 2021-03-24T21:59:45.644 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: cluster 2021-03-24T21:59:45. 2021-03-24T21:59:45.645 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: 210244+0000 mon.a (mon.0) 1222 : cluster [WRN] Health check update: Degraded data redundancy: 241/1485 objects degraded (16.229%), 35 pgs degraded, 81 pgs undersized (PG_DEGRADED) 2021-03-24T21:59:45.645 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: 2021-03-24T21:59:45.645 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:59:45.646 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: 03-24T21:59:45.239220+0000 mon.a (mon.0) 2021-03-24T21:59:45.646 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: 1223 : cluster [DBG] osdmap e415: 8 total, 7 up, 6 in 2021-03-24T21:59:45.647 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: audit 2021-03-24T 2021-03-24T21:59:45.647 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: 21:59:45.378706+0000 mon.c (mon.2 2021-03-24T21:59:45.648 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: ) 65 : audit [INF] from='client.? v1:172.21.15.125:0/158475072' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2021-03-24T21:59:45.648 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: audit 2021- 2021-03-24T21:59:45.648 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: 03-24T21:59:45.379096+0000 2021-03-24T21:59:45.649 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: mon.a (mon.0) 1224 : audit 2021-03-24T21:59:45.649 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: [INF] from='client.35204 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2021-03-24T21:59:45.650 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T21:59:45.650 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: T21:59:45.379226+0000 mon.a ( 2021-03-24T21:59:45.651 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: mon.0) 1225 : cluster [INF] 2021-03-24T21:59:45.651 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[24927]: Client client.admin marked osd.7 out, while it was still marked up 2021-03-24T21:59:45.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: debug 2021-03-24T21:59:45.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 2021-03-24T21:59:45.373+0000 7f2118128700 -1 mon.a@0(leader).osd e415 definitely_dead 0 2021-03-24T21:59:45.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 2021-03-24T21:59:45.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: cluster 2021-03-24T21:59:45.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 2021-03-24T21:59:45.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 2021 2021-03-24T21:59:45.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: - 2021-03-24T21:59:45.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 03 2021-03-24T21:59:45.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: - 2021-03-24T21:59:45.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 24 2021-03-24T21:59:45.657 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: T 2021-03-24T21:59:45.657 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 21 2021-03-24T21:59:45.657 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: : 2021-03-24T21:59:45.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 59 2021-03-24T21:59:45.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: : 2021-03-24T21:59:45.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 43 2021-03-24T21:59:45.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: . 2021-03-24T21:59:45.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 929576 2021-03-24T21:59:45.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: +0000 2021-03-24T21:59:45.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 2021-03-24T21:59:45.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: osd.0 2021-03-24T21:59:45.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: ( 2021-03-24T21:59:45.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: osd 2021-03-24T21:59:45.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: . 2021-03-24T21:59:45.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 0 2021-03-24T21:59:45.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: ) 2021-03-24T21:59:45.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 92 2021-03-24T21:59:45.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: : 2021-03-24T21:59:45.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: cluster 2021-03-24T21:59:45.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 2021-03-24T21:59:45.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: [DBG] 2021-03-24T21:59:45.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 2021-03-24T21:59:45.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 7.1b scrub ok 2021-03-24T21:59:45.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 2021-03-24T21:59:45.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: cluster 2021-03-24T21:59:45.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 2021-03-24T21:59:45.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 2021 2021-03-24T21:59:45.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: - 2021-03-24T21:59:45.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 03 2021-03-24T21:59:45.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: - 2021-03-24T21:59:45.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 24 2021-03-24T21:59:45.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: T 2021-03-24T21:59:45.669 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 21 2021-03-24T21:59:45.669 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: : 2021-03-24T21:59:45.669 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 59 2021-03-24T21:59:45.670 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: : 2021-03-24T21:59:45.671 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 44 2021-03-24T21:59:45.671 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: . 2021-03-24T21:59:45.671 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 665230 2021-03-24T21:59:45.672 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: +0000 2021-03-24T21:59:45.672 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 2021-03-24T21:59:45.673 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: osd.4 2021-03-24T21:59:45.673 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: ( 2021-03-24T21:59:45.673 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: osd 2021-03-24T21:59:45.674 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: . 2021-03-24T21:59:45.674 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 4 2021-03-24T21:59:45.674 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: ) 2021-03-24T21:59:45.674 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 101 2021-03-24T21:59:45.675 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: : 2021-03-24T21:59:45.675 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: cluster 2021-03-24T21:59:45.675 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 2021-03-24T21:59:45.675 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: [DBG] 2021-03-24T21:59:45.676 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 2021-03-24T21:59:45.676 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 4.1e scrub ok 2021-03-24T21:59:45.676 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 2021-03-24T21:59:45.676 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: cluster 2021-03-24T21:59:45.677 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 2021-03-24T21:59:45.677 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 2021 2021-03-24T21:59:45.677 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: - 2021-03-24T21:59:45.677 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 03 2021-03-24T21:59:45.677 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: - 2021-03-24T21:59:45.678 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 24 2021-03-24T21:59:45.678 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: T 2021-03-24T21:59:45.678 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 21 2021-03-24T21:59:45.678 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: : 2021-03-24T21:59:45.679 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 59 2021-03-24T21:59:45.679 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: : 2021-03-24T21:59:45.679 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 44 2021-03-24T21:59:45.679 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: . 2021-03-24T21:59:45.679 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 828971 2021-03-24T21:59:45.680 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: +0000 2021-03-24T21:59:45.680 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 2021-03-24T21:59:45.680 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: osd.2 2021-03-24T21:59:45.680 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: ( 2021-03-24T21:59:45.681 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: osd 2021-03-24T21:59:45.681 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: . 2021-03-24T21:59:45.681 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 2 2021-03-24T21:59:45.681 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: ) 2021-03-24T21:59:45.682 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 73 2021-03-24T21:59:45.682 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: : 2021-03-24T21:59:45.682 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: cluster 2021-03-24T21:59:45.682 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 2021-03-24T21:59:45.682 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: [DBG] 2021-03-24T21:59:45.683 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 2021-03-24T21:59:45.683 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 7.8 scrub ok 2021-03-24T21:59:45.683 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 2021-03-24T21:59:45.683 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: cluster 2021-03-24T21:59:45.684 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 2021-03-24T21:59:45.684 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 2021 2021-03-24T21:59:45.685 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: - 2021-03-24T21:59:45.685 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 03 2021-03-24T21:59:45.685 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: - 2021-03-24T21:59:45.686 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 24 2021-03-24T21:59:45.686 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: T 2021-03-24T21:59:45.687 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 21 2021-03-24T21:59:45.687 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: : 2021-03-24T21:59:45.687 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 59 2021-03-24T21:59:45.688 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: : 2021-03-24T21:59:45.688 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 45 2021-03-24T21:59:45.688 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: . 2021-03-24T21:59:45.688 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 210244 2021-03-24T21:59:45.688 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: +0000 2021-03-24T21:59:45.689 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 2021-03-24T21:59:45.689 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: mon.a 2021-03-24T21:59:45.689 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: ( 2021-03-24T21:59:45.689 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: mon 2021-03-24T21:59:45.690 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: . 2021-03-24T21:59:45.690 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 0 2021-03-24T21:59:45.690 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: ) 2021-03-24T21:59:45.690 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 1222 2021-03-24T21:59:45.691 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: : 2021-03-24T21:59:45.691 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: cluster 2021-03-24T21:59:45.691 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 2021-03-24T21:59:45.691 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: [WRN] 2021-03-24T21:59:45.691 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 2021-03-24T21:59:45.692 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: Health check update: Degraded data redundancy: 241/1485 objects degraded (16.229%), 35 pgs degraded, 81 pgs undersized (PG_DEGRADED) 2021-03-24T21:59:45.692 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 2021-03-24T21:59:45.692 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: cluster 2021-03-24T21:59:45.692 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 2021-03-24T21:59:45.692 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 2021 2021-03-24T21:59:45.693 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: - 2021-03-24T21:59:45.693 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 03 2021-03-24T21:59:45.693 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: - 2021-03-24T21:59:45.693 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 24 2021-03-24T21:59:45.694 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: T 2021-03-24T21:59:45.694 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 21 2021-03-24T21:59:45.694 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: : 2021-03-24T21:59:45.694 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 59 2021-03-24T21:59:45.695 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: : 2021-03-24T21:59:45.695 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 45 2021-03-24T21:59:45.695 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: . 2021-03-24T21:59:45.695 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 239220 2021-03-24T21:59:45.696 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: +0000 2021-03-24T21:59:45.696 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 2021-03-24T21:59:45.696 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: mon.a 2021-03-24T21:59:45.696 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: ( 2021-03-24T21:59:45.697 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: mon 2021-03-24T21:59:45.697 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: . 2021-03-24T21:59:45.697 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 0 2021-03-24T21:59:45.697 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: ) 2021-03-24T21:59:45.697 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 1223 2021-03-24T21:59:45.698 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: : 2021-03-24T21:59:45.698 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: cluster 2021-03-24T21:59:45.698 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 2021-03-24T21:59:45.699 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: [DBG] 2021-03-24T21:59:45.699 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 2021-03-24T21:59:45.699 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: osdmap e415: 8 total, 7 up, 6 in 2021-03-24T21:59:45.699 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 2021-03-24T21:59:45.700 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: audit 2021-03-24T21:59:45.700 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 2021-03-24T21:59:45.700 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 2021 2021-03-24T21:59:45.700 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: - 2021-03-24T21:59:45.700 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 03 2021-03-24T21:59:45.701 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: - 2021-03-24T21:59:45.701 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 24 2021-03-24T21:59:45.701 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: T 2021-03-24T21:59:45.701 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 21 2021-03-24T21:59:45.701 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: : 2021-03-24T21:59:45.702 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 59 2021-03-24T21:59:45.702 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: : 2021-03-24T21:59:45.702 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 45 2021-03-24T21:59:45.702 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: . 2021-03-24T21:59:45.703 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 378706 2021-03-24T21:59:45.703 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: +0000 2021-03-24T21:59:45.703 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 2021-03-24T21:59:45.703 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: mon.c 2021-03-24T21:59:45.703 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: ( 2021-03-24T21:59:45.704 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: mon 2021-03-24T21:59:45.704 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: . 2021-03-24T21:59:45.704 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 2 2021-03-24T21:59:45.704 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: ) 2021-03-24T21:59:45.705 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 65 2021-03-24T21:59:45.705 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: : 2021-03-24T21:59:45.705 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: audit 2021-03-24T21:59:45.705 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 2021-03-24T21:59:45.705 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: [INF] 2021-03-24T21:59:45.706 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 2021-03-24T21:59:45.706 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: from='client.? v1:172.21.15.125:0/158475072' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2021-03-24T21:59:45.706 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 2021-03-24T21:59:45.706 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: audit 2021-03-24T21:59:45.706 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 2021-03-24T21:59:45.707 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 2021-03-24T21: 2021-03-24T21:59:45.707 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: 59:45.379096+0000 mon.a (mon.0) 1224 : audit [INF] from='client.35204 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2021-03-24T21:59:45.707 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: cluster 2021-03-24T21:59:45.379226+0000 mon.a 2021-03-24T21:59:45.707 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:45 smithi125 conmon[19936]: (mon.0) 1225 : cluster [INF] Client client.admin marked osd.7 out, while it was still marked up 2021-03-24T21:59:45.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:45.709 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:45.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:59:45.716 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-24T21:59:45.811 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:45.813 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:45 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:59:45.865 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:45.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:45.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:59:45.970 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:46.019 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:46.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:46.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:59:46.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:45 smithi142 conmon[20125]: 59:43.929576+0000 osd.0 (osd.0) 92 : cluster [DBG] 7.1b scrub ok 2021-03-24T21:59:46.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:45 smithi142 conmon[20125]: cluster 2021-03-24T21:59:46.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:45 smithi142 conmon[20125]: 2021-03-24T21 2021-03-24T21:59:46.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:45 smithi142 conmon[20125]: :59:44.665230 2021-03-24T21:59:46.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:45 smithi142 conmon[20125]: +0000 osd.4 (osd.4 2021-03-24T21:59:46.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:45 smithi142 conmon[20125]: ) 101 : cluster [DBG] 2021-03-24T21:59:46.119 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:45 smithi142 conmon[20125]: 4.1e scrub ok 2021-03-24T21:59:46.119 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:45 smithi142 conmon[20125]: cluster 2021-03-24T21:59:46.119 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:45 smithi142 conmon[20125]: 2021-03-24T 2021-03-24T21:59:46.119 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:45 smithi142 conmon[20125]: 21:59:44.828971 2021-03-24T21:59:46.120 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:45 smithi142 conmon[20125]: +0000 osd.2 (osd 2021-03-24T21:59:46.120 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:45 smithi142 conmon[20125]: .2) 73 : cluster 2021-03-24T21:59:46.120 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:45 smithi142 conmon[20125]: [DBG] 7.8 scrub ok 2021-03-24T21:59:46.120 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:45 smithi142 conmon[20125]: cluster 2021-03-24T21:59:46.120 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:45 smithi142 conmon[20125]: 2021-03-24T 2021-03-24T21:59:46.121 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:45 smithi142 conmon[20125]: 21:59:45. 2021-03-24T21:59:46.121 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:45 smithi142 conmon[20125]: 210244+0000 mon.a (mon 2021-03-24T21:59:46.121 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:45 smithi142 conmon[20125]: .0) 1222 : cluster 2021-03-24T21:59:46.121 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:45 smithi142 conmon[20125]: [WRN] Health check update: Degraded data redundancy: 241/1485 objects degraded (16.229%), 35 pgs degraded, 81 pgs undersized (PG_DEGRADED) 2021-03-24T21:59:46.121 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:45 smithi142 conmon[20125]: cluster 2021-03-24T21:59:46.122 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:45 smithi142 conmon[20125]: 2021-03-24T 2021-03-24T21:59:46.122 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:45 smithi142 conmon[20125]: 21:59:45. 2021-03-24T21:59:46.122 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:45 smithi142 conmon[20125]: 239220+0000 mon.a (mon 2021-03-24T21:59:46.122 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:45 smithi142 conmon[20125]: .0) 1223 : cluster 2021-03-24T21:59:46.123 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:45 smithi142 conmon[20125]: [DBG] osdmap e415: 8 total, 7 up, 6 in 2021-03-24T21:59:46.123 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:45 smithi142 conmon[20125]: audit 2021-03-24T21:59:46.123 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:45 smithi142 conmon[20125]: 2021-03-24T21 2021-03-24T21:59:46.123 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:45 smithi142 conmon[20125]: :59:45. 2021-03-24T21:59:46.124 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:45 smithi142 conmon[20125]: 378706+0000 mon.c (mon 2021-03-24T21:59:46.124 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:45 smithi142 conmon[20125]: .2) 65 : audit 2021-03-24T21:59:46.124 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:45 smithi142 conmon[20125]: [INF] from='client.? v1:172.21.15.125:0/158475072' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2021-03-24T21:59:46.124 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:45 smithi142 conmon[20125]: audit 2021-03-24T21:59:46.124 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:45 smithi142 conmon[20125]: 2021-03-24T 2021-03-24T21:59:46.125 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:45 smithi142 conmon[20125]: 21:59:45.379096+0000 mon.a ( 2021-03-24T21:59:46.125 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:45 smithi142 conmon[20125]: mon.0) 1224 : 2021-03-24T21:59:46.125 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:45 smithi142 conmon[20125]: audit [INF] from='client.35204 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2021-03-24T21:59:46.125 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:45 smithi142 conmon[20125]: cluster 2021-03-24T21:59:46.126 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:45 smithi142 conmon[20125]: 2021-03-24T 2021-03-24T21:59:46.126 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:45 smithi142 conmon[20125]: 21:59:45. 2021-03-24T21:59:46.126 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:45 smithi142 conmon[20125]: 379226+0000 mon.a ( 2021-03-24T21:59:46.126 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:45 smithi142 conmon[20125]: mon.0) 1225 : 2021-03-24T21:59:46.127 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:45 smithi142 conmon[20125]: cluster [INF] Client client.admin marked osd.7 out, while it was still marked up 2021-03-24T21:59:46.127 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:45 smithi142 conmon[20125]: 2021-03-24T21:59:46.127 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:46.185 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:46.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:46.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:46.246 INFO:tasks.rados.rados.0.smithi125.stdout:1521: done (0 left) 2021-03-24T21:59:46.247 INFO:tasks.rados.rados.0.smithi125.stdout:1522: snap_create 2021-03-24T21:59:46.266 INFO:teuthology.orchestra.run.smithi125.stderr:marked out osd.7. 2021-03-24T21:59:46.291 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:46.369 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:46.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:46.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:46.479 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:46.525 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:46.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:46.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:59:46.563 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:46 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:59:46.563 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:46 smithi125 conmon[24927]: :59:44.838485+0000 mgr.y (mgr.14144) 478 : cluster [DBG] pgmap v784: 177 pgs: 1 peering, 9 active+clean+remapped, 52 active+undersized, 35 active+undersized+degraded, 80 active+clean; 385 MiB data, 893 MiB used, 536 GiB / 536 GiB avail; 243/1503 objects degraded (16.168%); 45/1503 objects misplaced (2.994%) 2021-03-24T21:59:46.563 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:46 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:59:46.564 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:46 smithi125 conmon[24927]: -03-24T21 2021-03-24T21:59:46.564 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:46 smithi125 conmon[24927]: :59:45.663642+0000 2021-03-24T21:59:46.564 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:46 smithi125 conmon[24927]: osd.4 (osd.4) 102 : 2021-03-24T21:59:46.564 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:46 smithi125 conmon[24927]: cluster [DBG] 4.1d scrub ok 2021-03-24T21:59:46.564 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:46 smithi125 conmon[24927]: audit 2021-03-24T21:59:46.565 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:46 smithi125 conmon[24927]: 2021-03-24T21: 2021-03-24T21:59:46.565 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:46 smithi125 conmon[24927]: 59:46.241554+0000 mon.a 2021-03-24T21:59:46.565 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:46 smithi125 conmon[24927]: (mon.0) 1226 : 2021-03-24T21:59:46.565 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:46 smithi125 conmon[24927]: audit [INF] from='client.35204 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2021-03-24T21:59:46.566 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:46 smithi125 conmon[24927]: cluster 2021-03-24T21:59:46.566 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:46 smithi125 conmon[24927]: 2021-03-24T21: 2021-03-24T21:59:46.566 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:46 smithi125 conmon[24927]: 59:46.241803+0000 mon.a 2021-03-24T21:59:46.566 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:46 smithi125 conmon[24927]: (mon.0) 1227 : cluster 2021-03-24T21:59:46.566 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:46 smithi125 conmon[24927]: [DBG] osdmap e416: 8 total, 7 up, 5 in 2021-03-24T21:59:46.567 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:46 smithi125 conmon[19936]: cluster 2021-03-24T21:59:44.838485+0000 2021-03-24T21:59:46.567 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:46 smithi125 conmon[19936]: mgr.y (mgr.14144) 478 : cluster [DBG] pgmap v784: 177 pgs: 1 peering, 9 active+clean+remapped, 52 active+undersized, 35 active+undersized+degraded, 80 active+clean; 385 MiB data, 893 MiB used, 536 GiB / 536 GiB avail; 243/1503 objects degraded (16.168%); 45/1503 objects misplaced (2.994%) 2021-03-24T21:59:46.567 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:46 smithi125 conmon[19936]: cluster 2021-03-24T21:59:45.663642+0000 osd.4 (osd.4) 102 : cluster [DBG] 4.1d scrub ok 2021-03-24T21:59:46.567 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:46 smithi125 conmon[19936]: audit 2021-03-24T21:59:46.568 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:46 smithi125 conmon[19936]: 2021-03-24T21:59:46.241554+0000 mon.a (mon.0) 1226 : audit [INF] from='client.35204 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2021-03-24T21:59:46.568 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:46 smithi125 conmon[19936]: cluster 2021- 2021-03-24T21:59:46.568 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:46 smithi125 conmon[19936]: 03-24T21:59:46.241803+0000 mon.a (mon.0) 1227 : cluster [DBG] osdmap e416: 8 total, 7 up, 5 in 2021-03-24T21:59:46.632 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:46.685 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:46.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:46.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:46.752 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T21:59:46.772 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T21:59:46.792 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:46.803 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T21:59:46.806 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:46 smithi142 conmon[20125]: cluster 2021-03-24T21:59:44.838485+0000 mgr.y (mgr.14144) 478 2021-03-24T21:59:46.806 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:46 smithi142 conmon[20125]: : cluster [DBG] pgmap v784: 177 pgs: 1 peering, 9 active+clean+remapped, 52 active+undersized, 35 active+undersized+degraded, 80 active+clean; 385 MiB data, 893 MiB used, 536 GiB / 536 GiB avail; 243/1503 objects degraded (16.168%); 45/1503 objects misplaced (2.994%) 2021-03-24T21:59:46.806 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:46 smithi142 conmon[20125]: cluster 2021-03-24T21:59:45.663642+0000 osd.4 (osd.4) 102 : cluster [DBG] 4.1d scrub ok 2021-03-24T21:59:46.806 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:46 smithi142 conmon[20125]: audit 2021-03-24T21:59 2021-03-24T21:59:46.807 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:46 smithi142 conmon[20125]: :46.241554+0000 mon.a (mon.0) 1226 : audit [INF] from='client.35204 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2021-03-24T21:59:46.807 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:46 smithi142 conmon[20125]: cluster 2021-03-24T21:59:46.807 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:46 smithi142 conmon[20125]: 2021-03-24T21:59:46.241803+0000 mon.a (mon 2021-03-24T21:59:46.807 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:46 smithi142 conmon[20125]: .0) 1227 : cluster [DBG] osdmap e416: 8 total, 7 up, 5 in 2021-03-24T21:59:46.824 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T21:59:46.847 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:46.849 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T21:59:46.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:46.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:46.872 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T21:59:46.900 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T21:59:46.901 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-24T21:59:46.901 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-24T21:59:46.901 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-24T21:59:46.901 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-24T21:59:46.902 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-24T21:59:46.902 INFO:teuthology.orchestra.run.smithi125.stderr:Error ENXIO: problem getting command descriptions from osd.5 2021-03-24T21:59:46.902 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-24T21:59:46.902 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-24T21:59:46.953 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:47.014 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:47.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:47.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:59:47.120 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:47.173 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:47.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:47.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:59:47.265 INFO:tasks.rados.rados.0.smithi125.stdout:1522: done (0 left) 2021-03-24T21:59:47.266 INFO:tasks.rados.rados.0.smithi125.stdout:1523: write (excl) oid 6 current snap is 167 2021-03-24T21:59:47.266 INFO:tasks.rados.rados.0.smithi125.stdout:1523: seq_num 539 ranges {658973=703921,1907912=570853,2985346=73627} 2021-03-24T21:59:47.266 INFO:tasks.rados.rados.0.smithi125.stdout:1523: writing smithi12556470-6 from 658973 to 1362894 tid 1 2021-03-24T21:59:47.271 INFO:tasks.rados.rados.0.smithi125.stdout:1523: writing smithi12556470-6 from 1907912 to 2478765 tid 2 2021-03-24T21:59:47.274 INFO:tasks.rados.rados.0.smithi125.stdout:1523: writing smithi12556470-6 from 2985346 to 3058973 tid 3 2021-03-24T21:59:47.275 INFO:tasks.rados.rados.0.smithi125.stdout:1524: write (excl) oid 2 current snap is 167 2021-03-24T21:59:47.275 INFO:tasks.rados.rados.0.smithi125.stdout:1524: seq_num 540 ranges {675780=495523,1682747=714511,3012769=63011} 2021-03-24T21:59:47.279 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:47.282 INFO:tasks.rados.rados.0.smithi125.stdout:1524: writing smithi12556470-2 from 675780 to 1171303 tid 1 2021-03-24T21:59:47.289 INFO:tasks.rados.rados.0.smithi125.stdout:1524: writing smithi12556470-2 from 1682747 to 2397258 tid 2 2021-03-24T21:59:47.293 INFO:tasks.rados.rados.0.smithi125.stdout:1524: writing smithi12556470-2 from 3012769 to 3075780 tid 3 2021-03-24T21:59:47.294 INFO:tasks.rados.rados.0.smithi125.stdout:1525: delete oid 22 current snap is 167 2021-03-24T21:59:47.334 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:47.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:47.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:47.441 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:47.488 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:47.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:47.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:59:47.592 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:47.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:47 smithi142 conmon[20125]: cluster 2021-03-24T21:59:46.375136+0000 osd.3 (osd.3 2021-03-24T21:59:47.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:47 smithi142 conmon[20125]: ) 93 : cluster [DBG] 5.0 scrub ok 2021-03-24T21:59:47.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:47 smithi142 conmon[20125]: audit 2021-03-24T21:59:46.723856+0000 mon.a (mon.0) 1228 : audit [DBG] from='client.? v1:172.21.15.125:0/318159345' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:59:47.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:47 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:59:47.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:47 smithi142 conmon[20125]: 24T21:59:47.246355+0000 mon.a (mon.0) 1229 : cluster [DBG] 2021-03-24T21:59:47.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:47 smithi142 conmon[20125]: osdmap e417: 8 total, 7 up, 5 in 2021-03-24T21:59:47.655 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:47.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:47.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:59:47.761 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:47.775 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:47 smithi125 conmon[19936]: cluster 2021-03-24T21:59:46. 2021-03-24T21:59:47.775 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:47 smithi125 conmon[19936]: 375136+0000 osd.3 (osd.3) 93 : cluster [DBG] 5.0 scrub ok 2021-03-24T21:59:47.775 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:47 smithi125 conmon[19936]: audit 2021-03-24T21: 2021-03-24T21:59:47.776 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:47 smithi125 conmon[19936]: 59:46.723856+0000 mon.a (mon.0) 1228 : audit [DBG] from='client.? v1:172.21.15.125:0/318159345' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:59:47.776 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:47 smithi125 conmon[19936]: cluster 2021-03-24T21:59:47.776 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:47 smithi125 conmon[19936]: 2021-03-24T21:59:47.246355+0000 mon.a ( 2021-03-24T21:59:47.776 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:47 smithi125 conmon[19936]: mon.0) 1229 : cluster [DBG] osdmap e417: 8 total, 7 up, 5 in 2021-03-24T21:59:47.777 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:47 smithi125 conmon[24927]: cluster 2021-03-24T21:59: 2021-03-24T21:59:47.777 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:47 smithi125 conmon[24927]: 46.375136+0000 osd.3 (osd.3) 93 : cluster [DBG] 5.0 scrub ok 2021-03-24T21:59:47.777 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:47 smithi125 conmon[24927]: audit 2021-03 2021-03-24T21:59:47.777 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:47 smithi125 conmon[24927]: -24T21:59:46.723856+0000 mon.a (mon.0) 1228 2021-03-24T21:59:47.778 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:47 smithi125 conmon[24927]: : audit [DBG] from='client.? v1:172.21.15.125:0/318159345' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:59:47.778 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:47 smithi125 conmon[24927]: cluster 2021-03-24T21:59:47. 2021-03-24T21:59:47.778 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:47 smithi125 conmon[24927]: 246355+0000 mon.a (mon.0) 1229 : cluster [DBG] 2021-03-24T21:59:47.778 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:47 smithi125 conmon[24927]: osdmap e417: 8 total, 7 up, 5 in 2021-03-24T21:59:47.816 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:47.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:47.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:59:47.924 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:47.958 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:59:47.981 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:47.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:47.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:59:48.085 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:48.144 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:48.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:48.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:48.249 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:48.361 INFO:tasks.rados.rados.0.smithi125.stdout:1524: finishing write tid 1 to smithi12556470-2 2021-03-24T21:59:48.362 INFO:tasks.rados.rados.0.smithi125.stdout:1523: finishing write tid 1 to smithi12556470-6 2021-03-24T21:59:48.362 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:48.369 INFO:tasks.rados.rados.0.smithi125.stdout:1525: done (2 left) 2021-03-24T21:59:48.370 INFO:tasks.rados.rados.0.smithi125.stdout:1526: read oid 20 snap -1 2021-03-24T21:59:48.370 INFO:tasks.rados.rados.0.smithi125.stdout:1526: expect (ObjNum 506 snap 147 seq_num 506) 2021-03-24T21:59:48.371 INFO:tasks.rados.rados.0.smithi125.stdout:1527: write oid 7 current snap is 167 2021-03-24T21:59:48.371 INFO:tasks.rados.rados.0.smithi125.stdout:1527: seq_num 542 ranges {709394=646021,2083358=718180} 2021-03-24T21:59:48.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:48.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:59:48.375 INFO:tasks.rados.rados.0.smithi125.stdout:1527: writing smithi12556470-7 from 709394 to 1355415 tid 1 2021-03-24T21:59:48.385 INFO:tasks.rados.rados.0.smithi125.stdout:1527: writing smithi12556470-7 from 2083358 to 2801538 tid 2 2021-03-24T21:59:48.386 INFO:tasks.rados.rados.0.smithi125.stdout:1528: rollback oid 27 current snap is 167 2021-03-24T21:59:48.387 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 27 to 143 2021-03-24T21:59:48.388 INFO:tasks.rados.rados.0.smithi125.stdout:1529: read oid 30 snap -1 2021-03-24T21:59:48.391 INFO:tasks.rados.rados.0.smithi125.stdout:1529: expect (ObjNum 447 snap 122 seq_num 447) 2021-03-24T21:59:48.391 INFO:tasks.rados.rados.0.smithi125.stdout:1530: write oid 3 current snap is 167 2021-03-24T21:59:48.392 INFO:tasks.rados.rados.0.smithi125.stdout:1530: seq_num 543 ranges {726201=721270,2141840=578191} 2021-03-24T21:59:48.393 INFO:tasks.rados.rados.0.smithi125.stdout:1530: writing smithi12556470-3 from 726201 to 1447471 tid 1 2021-03-24T21:59:48.403 INFO:tasks.rados.rados.0.smithi125.stdout:1530: writing smithi12556470-3 from 2141840 to 2720031 tid 2 2021-03-24T21:59:48.404 INFO:tasks.rados.rados.0.smithi125.stdout:1531: read oid 36 snap -1 2021-03-24T21:59:48.405 INFO:tasks.rados.rados.0.smithi125.stdout:1531: expect (ObjNum 460 snap 129 seq_num 460) 2021-03-24T21:59:48.405 INFO:tasks.rados.rados.0.smithi125.stdout:1532: read oid 15 snap -1 2021-03-24T21:59:48.406 INFO:tasks.rados.rados.0.smithi125.stdout:1532: expect deleted 2021-03-24T21:59:48.407 INFO:tasks.rados.rados.0.smithi125.stdout:1533: rollback oid 34 current snap is 167 2021-03-24T21:59:48.407 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 34 to 160 2021-03-24T21:59:48.418 INFO:tasks.rados.rados.0.smithi125.stdout:1523: finishing write tid 2 to smithi12556470-6 2021-03-24T21:59:48.418 INFO:tasks.rados.rados.0.smithi125.stdout:1523: finishing write tid 3 to smithi12556470-6 2021-03-24T21:59:48.419 INFO:tasks.rados.rados.0.smithi125.stdout:1526: expect (ObjNum 506 snap 147 seq_num 506) 2021-03-24T21:59:48.454 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:48 smithi125 conmon[19936]: cluster 2021-03-24T21:59:46.839918+0000 mgr.y (mgr. 2021-03-24T21:59:48.455 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:48 smithi125 conmon[19936]: 14144) 479 : cluster [DBG] pgmap v787: 177 pgs: 16 peering, 9 active+clean+remapped, 49 active+undersized, 31 active+undersized+degraded, 72 active+clean; 387 MiB data, 689 MiB used, 446 GiB / 447 GiB avail; 207/1503 objects degraded (13.772%); 45/1503 objects misplaced (2.994%) 2021-03-24T21:59:48.455 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:48 smithi125 conmon[19936]: cluster 2021-03-24T21:59:46.941614+0000 osd.0 (osd.0) 93 : cluster [DBG] 3.1 scrub ok 2021-03-24T21:59:48.457 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:48 smithi125 conmon[19936]: cluster 2021-03-24T21:59:48.457 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:48 smithi125 conmon[19936]: 2021-03-24T21:59:47.442580+0000 mon.a (mon.0) 1230 : cluster [WRN] Health check failed: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2021-03-24T21:59:48.458 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:48 smithi125 conmon[19936]: cluster 2021-03-24T21:59:48.460 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:48 smithi125 conmon[19936]: 2021-03-24T21:59: 2021-03-24T21:59:48.460 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:48 smithi125 conmon[19936]: 48.246246+0000 mon.a (mon.0) 1231 : cluster 2021-03-24T21:59:48.461 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:48 smithi125 conmon[19936]: [DBG] osdmap e418: 8 total, 7 up, 5 in 2021-03-24T21:59:48.476 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:48.530 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:48.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:48.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:48.542 INFO:tasks.rados.rados.0.smithi125.stdout:1524: finishing write tid 2 to smithi12556470-2 2021-03-24T21:59:48.544 INFO:tasks.rados.rados.0.smithi125.stdout:1523: finishing write tid 4 to smithi12556470-6 2021-03-24T21:59:48.545 INFO:tasks.rados.rados.0.smithi125.stdout:1523: finishing write tid 5 to smithi12556470-6 2021-03-24T21:59:48.545 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 6 v 368 (ObjNum 539 snap 167 seq_num 539) dirty exists 2021-03-24T21:59:48.545 INFO:tasks.rados.rados.0.smithi125.stdout:1523: left oid 6 (ObjNum 539 snap 167 seq_num 539) 2021-03-24T21:59:48.545 INFO:tasks.rados.rados.0.smithi125.stdout:1524: finishing write tid 3 to smithi12556470-2 2021-03-24T21:59:48.546 INFO:tasks.rados.rados.0.smithi125.stdout:1524: finishing write tid 4 to smithi12556470-2 2021-03-24T21:59:48.546 INFO:tasks.rados.rados.0.smithi125.stdout:1524: finishing write tid 5 to smithi12556470-2 2021-03-24T21:59:48.546 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 2 v 460 (ObjNum 540 snap 167 seq_num 540) dirty exists 2021-03-24T21:59:48.546 INFO:tasks.rados.rados.0.smithi125.stdout:1524: left oid 2 (ObjNum 540 snap 167 seq_num 540) 2021-03-24T21:59:48.547 INFO:tasks.rados.rados.0.smithi125.stdout:1527: finishing write tid 1 to smithi12556470-7 2021-03-24T21:59:48.547 INFO:tasks.rados.rados.0.smithi125.stdout:1523: done (9 left) 2021-03-24T21:59:48.547 INFO:tasks.rados.rados.0.smithi125.stdout:1524: done (8 left) 2021-03-24T21:59:48.547 INFO:tasks.rados.rados.0.smithi125.stdout:1526: done (7 left) 2021-03-24T21:59:48.547 INFO:tasks.rados.rados.0.smithi125.stdout:1534: snap_create 2021-03-24T21:59:48.548 INFO:tasks.rados.rados.0.smithi125.stdout:1528: finishing rollback tid 0 to smithi12556470-27 2021-03-24T21:59:48.548 INFO:tasks.rados.rados.0.smithi125.stdout:1529: expect (ObjNum 447 snap 122 seq_num 447) 2021-03-24T21:59:48.564 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:48 smithi125 conmon[24927]: cluster 2021-03-24T21:59:46. 2021-03-24T21:59:48.565 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:48 smithi125 conmon[24927]: 839918+0000 mgr.y (mgr.14144) 479 : cluster [DBG] pgmap v787: 177 pgs: 16 peering, 9 active+clean+remapped, 49 active+undersized, 31 active+undersized+degraded, 72 active+clean; 387 MiB data, 689 MiB used, 446 GiB / 447 GiB avail; 207/1503 objects degraded (13.772%); 45/1503 objects misplaced (2.994%) 2021-03-24T21:59:48.565 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:48 smithi125 conmon[24927]: cluster 2021-03-24T21:59:48.566 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:48 smithi125 conmon[24927]: 2021-03-24T21:59:46.941614+0000 osd.0 (osd.0 2021-03-24T21:59:48.566 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:48 smithi125 conmon[24927]: ) 93 : cluster [DBG] 3.1 scrub ok 2021-03-24T21:59:48.566 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:48 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:59:48.567 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:48 smithi125 conmon[24927]: -24T21:59:47.442580+0000 2021-03-24T21:59:48.568 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:48 smithi125 conmon[24927]: mon.a (mon.0) 1230 : cluster [WRN] Health check failed: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2021-03-24T21:59:48.568 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:48 smithi125 conmon[24927]: cluster 2021-03-24T21:59 2021-03-24T21:59:48.568 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:48 smithi125 conmon[24927]: :48.246246+0000 mon.a (mon.0) 1231 : cluster [DBG] osdmap e418: 8 total, 7 up, 5 in 2021-03-24T21:59:48.639 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:48.687 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:48.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:48.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:59:48.793 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:48.817 INFO:tasks.rados.rados.0.smithi125.stdout:1527: finishing write tid 2 to smithi12556470-7 2021-03-24T21:59:48.818 INFO:tasks.rados.rados.0.smithi125.stdout:1528: finishing rollback tid 1 to smithi12556470-27 2021-03-24T21:59:48.818 INFO:tasks.rados.rados.0.smithi125.stdout:1528: finishing rollback tid 2 to smithi12556470-27 2021-03-24T21:59:48.818 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 27 v 372 (ObjNum 400 snap 102 seq_num 400) dirty exists 2021-03-24T21:59:48.818 INFO:tasks.rados.rados.0.smithi125.stdout:1527: finishing write tid 3 to smithi12556470-7 2021-03-24T21:59:48.819 INFO:tasks.rados.rados.0.smithi125.stdout:1527: finishing write tid 4 to smithi12556470-7 2021-03-24T21:59:48.819 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 7 v 464 (ObjNum 542 snap 167 seq_num 542) dirty exists 2021-03-24T21:59:48.819 INFO:tasks.rados.rados.0.smithi125.stdout:1527: left oid 7 (ObjNum 542 snap 167 seq_num 542) 2021-03-24T21:59:48.819 INFO:tasks.rados.rados.0.smithi125.stdout:1530: finishing write tid 1 to smithi12556470-3 2021-03-24T21:59:48.820 INFO:tasks.rados.rados.0.smithi125.stdout:1530: finishing write tid 2 to smithi12556470-3 2021-03-24T21:59:48.820 INFO:tasks.rados.rados.0.smithi125.stdout:1530: finishing write tid 3 to smithi12556470-3 2021-03-24T21:59:48.820 INFO:tasks.rados.rados.0.smithi125.stdout:1530: finishing write tid 4 to smithi12556470-3 2021-03-24T21:59:48.820 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 3 v 173 (ObjNum 543 snap 167 seq_num 543) dirty exists 2021-03-24T21:59:48.821 INFO:tasks.rados.rados.0.smithi125.stdout:1530: left oid 3 (ObjNum 543 snap 167 seq_num 543) 2021-03-24T21:59:48.821 INFO:tasks.rados.rados.0.smithi125.stdout:1533: finishing rollback tid 0 to smithi12556470-34 2021-03-24T21:59:48.821 INFO:tasks.rados.rados.0.smithi125.stdout:1533: finishing rollback tid 1 to smithi12556470-34 2021-03-24T21:59:48.821 INFO:tasks.rados.rados.0.smithi125.stdout:1533: finishing rollback tid 2 to smithi12556470-34 2021-03-24T21:59:48.821 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 34 v 376 (ObjNum 503 snap 146 seq_num 503) dirty exists 2021-03-24T21:59:48.822 INFO:tasks.rados.rados.0.smithi125.stdout:1531: expect (ObjNum 460 snap 129 seq_num 460) 2021-03-24T21:59:48.841 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:48.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:48.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:59:48.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:48 smithi142 conmon[20125]: cluster 2021-03-24T21: 2021-03-24T21:59:48.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:48 smithi142 conmon[20125]: 59:46.839918+0000 mgr.y (mgr.14144) 479 : cluster [DBG] pgmap v787: 177 pgs: 16 peering, 9 active+clean+remapped, 49 active+undersized, 31 active+undersized+degraded, 72 active+clean; 387 MiB data, 689 MiB used, 446 GiB / 447 GiB avail; 207/1503 objects degraded (13.772%); 45/1503 objects misplaced (2.994%) 2021-03-24T21:59:48.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:48 smithi142 conmon[20125]: cluster 2021-03-24T21:59:48.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:48 smithi142 conmon[20125]: 2021-03-24T21:59:46. 2021-03-24T21:59:48.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:48 smithi142 conmon[20125]: 941614+0000 osd.0 (osd 2021-03-24T21:59:48.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:48 smithi142 conmon[20125]: .0) 93 : cluster [DBG] 2021-03-24T21:59:48.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:48 smithi142 conmon[20125]: 3.1 scrub ok 2021-03-24T21:59:48.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:48 smithi142 conmon[20125]: cluster 2021-03-24T21:59:48.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:48 smithi142 conmon[20125]: 2021-03-24T 2021-03-24T21:59:48.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:48 smithi142 conmon[20125]: 21:59:47.442580 2021-03-24T21:59:48.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:48 smithi142 conmon[20125]: +0000 mon.a (mon.0) 2021-03-24T21:59:48.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:48 smithi142 conmon[20125]: 1230 : cluster [WRN] Health check failed: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2021-03-24T21:59:48.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:48 smithi142 conmon[20125]: cluster 2021-03-24T21:59:48.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:48 smithi142 conmon[20125]: 2021-03-24T21: 2021-03-24T21:59:48.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:48 smithi142 conmon[20125]: 59:48.246246+0000 2021-03-24T21:59:48.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:48 smithi142 conmon[20125]: mon.a (mon.0) 2021-03-24T21:59:48.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:48 smithi142 conmon[20125]: 1231 : cluster [DBG] 2021-03-24T21:59:48.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:48 smithi142 conmon[20125]: osdmap e418: 8 total, 7 up, 5 in 2021-03-24T21:59:48.946 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:48.994 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:48.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:48.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:59:49.099 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:49.147 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:49.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:49.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:59:49.239 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-24T21:59:49.239 INFO:teuthology.orchestra.run.smithi125.stdout: "size": 20, 2021-03-24T21:59:49.239 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 600, 2021-03-24T21:59:49.240 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [ 2021-03-24T21:59:49.240 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.240 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:1230 7.3 7:cb55abc9:::smithi12556470-6:head [write 410137~481384] snapc 53=[] ondisk+write+known_if_redirected e191)", 2021-03-24T21:59:49.241 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:56:00.116990+0000", 2021-03-24T21:59:49.241 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 229.06685426300001, 2021-03-24T21:59:49.241 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.098702171000000005, 2021-03-24T21:59:49.241 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:59:49.242 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:59:49.242 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:59:49.242 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:59:49.242 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:59:49.243 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1230 2021-03-24T21:59:49.243 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.243 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:59:49.243 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.243 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:59:49.244 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.116990+0000", 2021-03-24T21:59:49.244 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:59:49.244 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.244 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.245 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:59:49.245 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.116990+0000", 2021-03-24T21:59:49.245 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999962 2021-03-24T21:59:49.245 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.246 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.246 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:59:49.246 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.116986+0000", 2021-03-24T21:59:49.246 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00030943799999999999 2021-03-24T21:59:49.246 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.247 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.247 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:59:49.247 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.117296+0000", 2021-03-24T21:59:49.247 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5636000000000001e-05 2021-03-24T21:59:49.248 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.248 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.248 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:59:49.248 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.117312+0000", 2021-03-24T21:59:49.249 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.6680000000000007e-06 2021-03-24T21:59:49.249 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.249 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.249 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:59:49.250 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.117320+0000", 2021-03-24T21:59:49.250 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.4054000000000006e-05 2021-03-24T21:59:49.250 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.250 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.251 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:49.251 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.117394+0000", 2021-03-24T21:59:49.251 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2653000000000001e-05 2021-03-24T21:59:49.251 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.252 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.252 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op must wait for map", 2021-03-24T21:59:49.252 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.117407+0000", 2021-03-24T21:59:49.252 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.077794965999999993 2021-03-24T21:59:49.253 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.253 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.253 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:49.253 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.195202+0000", 2021-03-24T21:59:49.253 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00015882200000000001 2021-03-24T21:59:49.254 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.254 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.254 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:59:49.254 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.195361+0000", 2021-03-24T21:59:49.255 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00025800299999999998 2021-03-24T21:59:49.255 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.255 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.255 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,4", 2021-03-24T21:59:49.256 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.195619+0000", 2021-03-24T21:59:49.256 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.010072628 2021-03-24T21:59:49.256 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.256 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.256 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:59:49.257 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.205691+0000", 2021-03-24T21:59:49.257 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002417284 2021-03-24T21:59:49.257 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.257 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.257 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:59:49.258 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.208109+0000", 2021-03-24T21:59:49.258 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0074134020000000004 2021-03-24T21:59:49.258 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.259 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.259 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:59:49.259 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.215522+0000", 2021-03-24T21:59:49.259 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.2187000000000004e-05 2021-03-24T21:59:49.260 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.260 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.260 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:59:49.260 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.215614+0000", 2021-03-24T21:59:49.261 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.8195999999999995e-05 2021-03-24T21:59:49.261 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.261 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.261 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:59:49.262 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.215692+0000", 2021-03-24T21:59:49.262 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.098702171000000005 2021-03-24T21:59:49.262 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:49.263 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:59:49.263 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:49.263 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.263 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.264 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:1231 7.3 7:cb55abc9:::smithi12556470-6:head [write 1410296~761389] snapc 53=[] ondisk+write+known_if_redirected e191)", 2021-03-24T21:59:49.264 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:56:00.125127+0000", 2021-03-24T21:59:49.264 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 229.05871737499999, 2021-03-24T21:59:49.264 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.090759490999999998, 2021-03-24T21:59:49.265 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:59:49.265 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:59:49.265 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:59:49.265 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:59:49.265 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:59:49.266 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1231 2021-03-24T21:59:49.266 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.266 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:59:49.266 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.267 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:59:49.267 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.125127+0000", 2021-03-24T21:59:49.267 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:59:49.267 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.267 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.268 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:59:49.268 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.125127+0000", 2021-03-24T21:59:49.268 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T21:59:49.268 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.269 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.269 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:59:49.269 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.125125+0000", 2021-03-24T21:59:49.269 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00048102900000000001 2021-03-24T21:59:49.270 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.270 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.270 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:59:49.270 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.125606+0000", 2021-03-24T21:59:49.271 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.070621186000000002 2021-03-24T21:59:49.271 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.271 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.271 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:59:49.272 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.196227+0000", 2021-03-24T21:59:49.272 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7897000000000002e-05 2021-03-24T21:59:49.272 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.273 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.273 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:59:49.273 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.196245+0000", 2021-03-24T21:59:49.273 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001185069 2021-03-24T21:59:49.273 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.274 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.274 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:49.274 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.197430+0000", 2021-03-24T21:59:49.274 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00014529899999999999 2021-03-24T21:59:49.275 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.275 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.275 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:59:49.275 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.197576+0000", 2021-03-24T21:59:49.275 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00027430999999999999 2021-03-24T21:59:49.276 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.276 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.276 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,4", 2021-03-24T21:59:49.276 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.197850+0000", 2021-03-24T21:59:49.277 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0078862050000000003 2021-03-24T21:59:49.277 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.277 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.277 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:59:49.277 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.205736+0000", 2021-03-24T21:59:49.278 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0023783889999999999 2021-03-24T21:59:49.278 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.278 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.279 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:59:49.279 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.208115+0000", 2021-03-24T21:59:49.280 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.007646824 2021-03-24T21:59:49.280 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.280 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.281 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:59:49.281 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.215761+0000", 2021-03-24T21:59:49.282 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.2390000000000003e-05 2021-03-24T21:59:49.282 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.283 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.283 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:59:49.283 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.215834+0000", 2021-03-24T21:59:49.284 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2782999999999999e-05 2021-03-24T21:59:49.284 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.285 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.285 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:59:49.286 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.215887+0000", 2021-03-24T21:59:49.286 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.090759490999999998 2021-03-24T21:59:49.286 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:49.287 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:59:49.287 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:49.288 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.288 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.288 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:2390 7.23 7:c49de0b4:::smithi12556470-13:head [write 430345~727947] snapc a9=[a9,a8,a7,a0] ondisk+write+known_if_redirected e275)", 2021-03-24T21:59:49.289 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:57:25.173714+0000", 2021-03-24T21:59:49.289 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 144.01013039899999, 2021-03-24T21:59:49.290 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.018775558, 2021-03-24T21:59:49.290 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:59:49.291 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:59:49.291 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:59:49.291 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:59:49.292 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:59:49.292 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 2390 2021-03-24T21:59:49.293 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.293 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:59:49.294 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.294 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:59:49.294 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.173714+0000", 2021-03-24T21:59:49.295 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:59:49.295 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.296 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.296 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:59:49.297 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.173714+0000", 2021-03-24T21:59:49.297 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T21:59:49.297 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.298 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.298 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:59:49.299 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.173712+0000", 2021-03-24T21:59:49.300 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00044400899999999998 2021-03-24T21:59:49.300 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.300 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.301 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:59:49.301 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.174156+0000", 2021-03-24T21:59:49.301 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0098000000000001e-05 2021-03-24T21:59:49.302 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.302 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.303 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:59:49.303 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.174167+0000", 2021-03-24T21:59:49.303 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.7410000000000003e-06 2021-03-24T21:59:49.303 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.304 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.304 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:59:49.304 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.174173+0000", 2021-03-24T21:59:49.304 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.9721999999999994e-05 2021-03-24T21:59:49.305 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.305 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.306 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:49.306 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.174243+0000", 2021-03-24T21:59:49.306 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3264e-05 2021-03-24T21:59:49.307 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.307 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.308 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T21:59:49.308 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.174256+0000", 2021-03-24T21:59:49.309 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.003266601 2021-03-24T21:59:49.309 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.309 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.310 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:49.310 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.177523+0000", 2021-03-24T21:59:49.311 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00014048700000000001 2021-03-24T21:59:49.311 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.312 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.312 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:59:49.312 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.177663+0000", 2021-03-24T21:59:49.313 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00069449699999999997 2021-03-24T21:59:49.313 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.314 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.314 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,4", 2021-03-24T21:59:49.315 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.178358+0000", 2021-03-24T21:59:49.315 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.005790636 2021-03-24T21:59:49.315 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.316 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.316 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:59:49.316 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.184148+0000", 2021-03-24T21:59:49.317 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00056901200000000003 2021-03-24T21:59:49.317 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.318 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.318 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:59:49.319 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.184717+0000", 2021-03-24T21:59:49.319 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0076483189999999998 2021-03-24T21:59:49.319 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.320 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.320 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:59:49.321 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.192366+0000", 2021-03-24T21:59:49.322 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.1732999999999999e-05 2021-03-24T21:59:49.322 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.322 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.323 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:59:49.323 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.192438+0000", 2021-03-24T21:59:49.324 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2144000000000001e-05 2021-03-24T21:59:49.324 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.325 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.325 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:59:49.325 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.192490+0000", 2021-03-24T21:59:49.326 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.018775558 2021-03-24T21:59:49.326 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:49.327 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:59:49.327 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:49.328 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.328 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.329 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:2391 7.23 7:c49de0b4:::smithi12556470-13:head [write 1570056~727439] snapc a9=[a9,a8,a7,a0] ondisk+write+known_if_redirected e275)", 2021-03-24T21:59:49.329 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:57:25.174190+0000", 2021-03-24T21:59:49.329 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 144.00965432000001, 2021-03-24T21:59:49.330 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.018522293, 2021-03-24T21:59:49.330 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:59:49.330 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:59:49.331 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:59:49.331 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:59:49.331 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:59:49.332 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 2391 2021-03-24T21:59:49.332 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.332 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:59:49.333 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.333 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:59:49.333 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.174190+0000", 2021-03-24T21:59:49.334 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:59:49.334 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.334 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.334 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:59:49.335 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.174190+0000", 2021-03-24T21:59:49.335 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T21:59:49.335 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.335 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.335 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:59:49.336 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.174189+0000", 2021-03-24T21:59:49.336 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000389556 2021-03-24T21:59:49.336 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.336 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.337 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:59:49.337 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.174578+0000", 2021-03-24T21:59:49.337 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4030000000000004e-06 2021-03-24T21:59:49.337 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.337 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.338 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:59:49.338 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.174585+0000", 2021-03-24T21:59:49.338 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1669999999999998e-06 2021-03-24T21:59:49.338 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.339 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.339 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:59:49.339 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.174590+0000", 2021-03-24T21:59:49.339 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1109000000000001e-05 2021-03-24T21:59:49.339 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.340 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.340 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:49.340 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.174641+0000", 2021-03-24T21:59:49.340 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0509000000000001e-05 2021-03-24T21:59:49.341 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.341 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.341 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T21:59:49.341 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.174651+0000", 2021-03-24T21:59:49.342 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0054966999999999 2021-03-24T21:59:49.342 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.342 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.342 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:49.343 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.180148+0000", 2021-03-24T21:59:49.343 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00011527 2021-03-24T21:59:49.343 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.343 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.344 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:59:49.344 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.180263+0000", 2021-03-24T21:59:49.344 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000228352 2021-03-24T21:59:49.344 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.345 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.345 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,4", 2021-03-24T21:59:49.345 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.180492+0000", 2021-03-24T21:59:49.345 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.004233115 2021-03-24T21:59:49.345 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.346 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.346 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:59:49.346 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.184725+0000", 2021-03-24T21:59:49.346 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.006255404 2021-03-24T21:59:49.347 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.347 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.347 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:59:49.347 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.190980+0000", 2021-03-24T21:59:49.347 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0016284769999999999 2021-03-24T21:59:49.348 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.348 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.348 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:59:49.348 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.192609+0000", 2021-03-24T21:59:49.349 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5276999999999999e-05 2021-03-24T21:59:49.349 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.349 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.349 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:59:49.349 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.192664+0000", 2021-03-24T21:59:49.350 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.8547000000000001e-05 2021-03-24T21:59:49.350 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.350 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.350 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:59:49.351 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.192712+0000", 2021-03-24T21:59:49.351 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.018522293 2021-03-24T21:59:49.351 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:49.351 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:59:49.352 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:49.352 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.352 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.352 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:2392 7.23 7:c49de0b4:::smithi12556470-13:head [write 2699879~469889] snapc a9=[a9,a8,a7,a0] ondisk+write+known_if_redirected e275)", 2021-03-24T21:59:49.352 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:57:25.175958+0000", 2021-03-24T21:59:49.353 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 144.00788641700001, 2021-03-24T21:59:49.353 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.018195899, 2021-03-24T21:59:49.353 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:59:49.353 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:59:49.354 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:59:49.354 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:59:49.354 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:59:49.354 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 2392 2021-03-24T21:59:49.355 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.355 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:59:49.355 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.355 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:59:49.356 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.175958+0000", 2021-03-24T21:59:49.356 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:59:49.356 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.356 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.356 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:59:49.357 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.175958+0000", 2021-03-24T21:59:49.357 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999971 2021-03-24T21:59:49.357 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.357 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.358 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:59:49.358 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.175955+0000", 2021-03-24T21:59:49.358 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00032653800000000002 2021-03-24T21:59:49.358 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.359 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.359 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:59:49.359 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176282+0000", 2021-03-24T21:59:49.359 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0336999999999999e-05 2021-03-24T21:59:49.359 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.360 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.360 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:59:49.360 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176292+0000", 2021-03-24T21:59:49.360 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4629999999999998e-06 2021-03-24T21:59:49.360 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.361 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.361 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:59:49.361 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176299+0000", 2021-03-24T21:59:49.361 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.8623999999999998e-05 2021-03-24T21:59:49.362 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.362 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.362 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:49.362 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176377+0000", 2021-03-24T21:59:49.363 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2301e-05 2021-03-24T21:59:49.363 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.363 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.363 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T21:59:49.363 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176390+0000", 2021-03-24T21:59:49.364 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.005175323 2021-03-24T21:59:49.364 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.364 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.364 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:49.364 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.181565+0000", 2021-03-24T21:59:49.365 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000124189 2021-03-24T21:59:49.365 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.365 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.365 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:59:49.366 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.181689+0000", 2021-03-24T21:59:49.366 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000263744 2021-03-24T21:59:49.366 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.366 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.366 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,4", 2021-03-24T21:59:49.367 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.181953+0000", 2021-03-24T21:59:49.367 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0027792490000000001 2021-03-24T21:59:49.367 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.367 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.368 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:59:49.368 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.184732+0000", 2021-03-24T21:59:49.368 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0063294529999999996 2021-03-24T21:59:49.368 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.368 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.369 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:59:49.369 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.191062+0000", 2021-03-24T21:59:49.369 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0029539750000000002 2021-03-24T21:59:49.369 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.370 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.370 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:59:49.370 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.194016+0000", 2021-03-24T21:59:49.370 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.5142999999999996e-05 2021-03-24T21:59:49.371 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.371 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.371 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:59:49.371 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.194081+0000", 2021-03-24T21:59:49.371 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.3331999999999995e-05 2021-03-24T21:59:49.372 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.372 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.372 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:59:49.372 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.194154+0000", 2021-03-24T21:59:49.373 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.018195899 2021-03-24T21:59:49.373 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:49.373 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:59:49.373 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:49.373 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.374 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.374 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:2393 7.23 7:c49de0b4:::smithi12556470-13:head [setxattr _header (57) in=7b,truncate 3630345] snapc a9=[a9,a8,a7,a0] ondisk+write+known_if_redirected e275)", 2021-03-24T21:59:49.374 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:57:25.176325+0000", 2021-03-24T21:59:49.374 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 144.00751967599999, 2021-03-24T21:59:49.375 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.024884068, 2021-03-24T21:59:49.375 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:59:49.375 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:59:49.375 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:59:49.375 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:59:49.376 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:59:49.376 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 2393 2021-03-24T21:59:49.376 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.376 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:59:49.377 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.377 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:59:49.377 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176325+0000", 2021-03-24T21:59:49.377 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:59:49.377 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.378 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.378 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:59:49.378 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176325+0000", 2021-03-24T21:59:49.378 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T21:59:49.379 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.379 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.379 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:59:49.379 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176324+0000", 2021-03-24T21:59:49.379 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1580000000000004e-06 2021-03-24T21:59:49.380 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.380 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.380 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:59:49.380 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176329+0000", 2021-03-24T21:59:49.381 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6590000000000001e-06 2021-03-24T21:59:49.381 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.381 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.381 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:59:49.381 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176336+0000", 2021-03-24T21:59:49.382 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1610000000000002e-06 2021-03-24T21:59:49.382 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.382 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.382 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:59:49.382 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176341+0000", 2021-03-24T21:59:49.383 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.3918000000000005e-05 2021-03-24T21:59:49.383 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.383 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.383 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:49.384 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176425+0000", 2021-03-24T21:59:49.384 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0132000000000001e-05 2021-03-24T21:59:49.384 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.384 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.384 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T21:59:49.385 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176435+0000", 2021-03-24T21:59:49.385 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0063475879999999 2021-03-24T21:59:49.385 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.385 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.386 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:49.386 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.182782+0000", 2021-03-24T21:59:49.386 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00028664400000000002 2021-03-24T21:59:49.386 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.387 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.387 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:59:49.387 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.183069+0000", 2021-03-24T21:59:49.387 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00021797799999999999 2021-03-24T21:59:49.388 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.388 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.388 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,4", 2021-03-24T21:59:49.388 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.183287+0000", 2021-03-24T21:59:49.389 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0014522280000000001 2021-03-24T21:59:49.389 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.389 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.389 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:59:49.390 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.184739+0000", 2021-03-24T21:59:49.390 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0095337649999999996 2021-03-24T21:59:49.390 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.390 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.391 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:59:49.391 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.194273+0000", 2021-03-24T21:59:49.391 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0067855449999999996 2021-03-24T21:59:49.391 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.391 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.392 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:59:49.392 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.201059+0000", 2021-03-24T21:59:49.392 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.5068999999999996e-05 2021-03-24T21:59:49.392 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.393 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.393 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:59:49.393 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.201134+0000", 2021-03-24T21:59:49.393 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.5298999999999999e-05 2021-03-24T21:59:49.394 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.394 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.394 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:59:49.394 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.201209+0000", 2021-03-24T21:59:49.394 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.024884068 2021-03-24T21:59:49.395 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:49.395 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:59:49.395 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:49.395 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.396 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.396 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:2394 7.23 7:c49de0b4:::smithi12556470-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e275)", 2021-03-24T21:59:49.396 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:57:25.176373+0000", 2021-03-24T21:59:49.396 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 144.00747191799999, 2021-03-24T21:59:49.396 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.025101692, 2021-03-24T21:59:49.397 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:59:49.397 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T21:59:49.397 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:59:49.397 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:59:49.398 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:59:49.398 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 2394 2021-03-24T21:59:49.398 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.398 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:59:49.399 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.399 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:59:49.399 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176373+0000", 2021-03-24T21:59:49.399 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:59:49.399 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.400 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.400 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:59:49.400 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176373+0000", 2021-03-24T21:59:49.400 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T21:59:49.401 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.401 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.401 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:59:49.401 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176371+0000", 2021-03-24T21:59:49.402 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.08e-06 2021-03-24T21:59:49.402 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.402 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.402 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:59:49.403 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176378+0000", 2021-03-24T21:59:49.403 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1309e-05 2021-03-24T21:59:49.403 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.403 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.403 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:59:49.404 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176389+0000", 2021-03-24T21:59:49.404 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.8714e-05 2021-03-24T21:59:49.404 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.404 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.405 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:59:49.405 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176418+0000", 2021-03-24T21:59:49.405 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6766000000000004e-05 2021-03-24T21:59:49.405 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.405 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.406 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:49.406 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176485+0000", 2021-03-24T21:59:49.406 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.1589999999999999e-06 2021-03-24T21:59:49.406 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.407 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.407 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T21:59:49.407 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176494+0000", 2021-03-24T21:59:49.407 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0071017390000001 2021-03-24T21:59:49.407 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.408 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.408 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:49.408 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.183596+0000", 2021-03-24T21:59:49.408 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6001999999999996e-05 2021-03-24T21:59:49.408 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.409 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.409 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T21:59:49.409 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.183662+0000", 2021-03-24T21:59:49.409 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.017585719 2021-03-24T21:59:49.410 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.410 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.410 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:49.410 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.201247+0000", 2021-03-24T21:59:49.410 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000106111 2021-03-24T21:59:49.411 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.411 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.411 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:59:49.411 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.201354+0000", 2021-03-24T21:59:49.412 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000120763 2021-03-24T21:59:49.412 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.412 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.412 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:59:49.412 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.201474+0000", 2021-03-24T21:59:49.413 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.025101692 2021-03-24T21:59:49.413 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:49.413 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:59:49.413 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:49.413 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.414 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.414 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:3829 7.c 7:31cb714a:::smithi12556470-7:head [write 457289~484580] snapc 125=[125,123,120,11e,11b,11a,119,110,10f,10e,10d,10c,106,105,104,fa,f8,e7] ondisk+write+known_if_redirected e400)", 2021-03-24T21:59:49.414 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:30.226743+0000", 2021-03-24T21:59:49.414 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 18.957101937000001, 2021-03-24T21:59:49.415 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.099094491000000007, 2021-03-24T21:59:49.415 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:59:49.415 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:59:49.415 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:59:49.416 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:59:49.416 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:59:49.416 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3829 2021-03-24T21:59:49.416 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.416 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:59:49.417 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.417 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:59:49.417 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.226743+0000", 2021-03-24T21:59:49.417 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:59:49.417 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.418 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.418 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:59:49.418 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.226743+0000", 2021-03-24T21:59:49.418 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999943 2021-03-24T21:59:49.419 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.419 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.419 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:59:49.419 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.226737+0000", 2021-03-24T21:59:49.419 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00035636400000000002 2021-03-24T21:59:49.420 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.420 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.420 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:59:49.420 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.227093+0000", 2021-03-24T21:59:49.421 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.0479000000000001e-05 2021-03-24T21:59:49.421 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.421 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.421 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:59:49.421 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.227124+0000", 2021-03-24T21:59:49.422 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2363e-05 2021-03-24T21:59:49.422 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.422 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.423 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:59:49.423 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.227136+0000", 2021-03-24T21:59:49.423 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4978e-05 2021-03-24T21:59:49.423 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.424 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.424 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:49.424 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.227201+0000", 2021-03-24T21:59:49.424 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1637000000000001e-05 2021-03-24T21:59:49.425 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.425 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.425 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op must wait for map", 2021-03-24T21:59:49.425 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.227213+0000", 2021-03-24T21:59:49.426 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.042208045 2021-03-24T21:59:49.426 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.426 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.426 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:49.426 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.269421+0000", 2021-03-24T21:59:49.427 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012722699999999999 2021-03-24T21:59:49.427 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.427 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.427 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:59:49.428 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.269548+0000", 2021-03-24T21:59:49.428 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00027209000000000001 2021-03-24T21:59:49.428 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.428 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.429 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,7", 2021-03-24T21:59:49.429 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.269820+0000", 2021-03-24T21:59:49.429 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.026839100000000001 2021-03-24T21:59:49.430 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.430 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.430 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:59:49.431 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.296659+0000", 2021-03-24T21:59:49.431 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.025150059999999998 2021-03-24T21:59:49.431 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.431 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.432 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:59:49.432 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.321809+0000", 2021-03-24T21:59:49.432 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0037085629999999998 2021-03-24T21:59:49.432 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.432 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.433 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:59:49.433 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.325518+0000", 2021-03-24T21:59:49.433 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000132884 2021-03-24T21:59:49.433 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.434 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.434 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:59:49.434 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.325651+0000", 2021-03-24T21:59:49.435 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00018655899999999999 2021-03-24T21:59:49.435 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.435 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.435 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:59:49.436 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.325837+0000", 2021-03-24T21:59:49.436 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.099094491000000007 2021-03-24T21:59:49.436 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:49.436 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:59:49.436 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:49.437 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.437 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.437 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:3830 7.c 7:31cb714a:::smithi12556470-7:head [write 1438834~548839] snapc 125=[125,123,120,11e,11b,11a,119,110,10f,10e,10d,10c,106,105,104,fa,f8,e7] ondisk+write+known_if_redirected e400)", 2021-03-24T21:59:49.437 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:30.234020+0000", 2021-03-24T21:59:49.437 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 18.949824206999999, 2021-03-24T21:59:49.438 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.092194254000000003, 2021-03-24T21:59:49.438 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:59:49.438 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:59:49.438 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:59:49.439 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:59:49.439 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:59:49.440 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3830 2021-03-24T21:59:49.440 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.441 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:59:49.441 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:49.444 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.444 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:59:49.444 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.234020+0000", 2021-03-24T21:59:49.444 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:59:49.445 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.445 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.445 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:59:49.445 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.234020+0000", 2021-03-24T21:59:49.445 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999971 2021-03-24T21:59:49.446 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.446 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.446 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:59:49.446 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.234017+0000", 2021-03-24T21:59:49.447 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00037994999999999999 2021-03-24T21:59:49.447 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.447 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.447 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:59:49.447 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.234397+0000", 2021-03-24T21:59:49.448 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.7203e-05 2021-03-24T21:59:49.448 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.448 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.448 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:59:49.449 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.234425+0000", 2021-03-24T21:59:49.449 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1429e-05 2021-03-24T21:59:49.449 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.449 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.449 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:59:49.450 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.234436+0000", 2021-03-24T21:59:49.450 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.2726999999999997e-05 2021-03-24T21:59:49.450 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.450 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.451 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:49.451 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.234509+0000", 2021-03-24T21:59:49.451 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1219e-05 2021-03-24T21:59:49.451 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.451 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.452 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting_for_map not empty", 2021-03-24T21:59:49.452 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.234520+0000", 2021-03-24T21:59:49.452 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.036768780000000001 2021-03-24T21:59:49.452 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.452 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.453 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:49.453 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.271289+0000", 2021-03-24T21:59:49.453 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00011443299999999999 2021-03-24T21:59:49.453 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.454 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.454 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:59:49.454 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.271403+0000", 2021-03-24T21:59:49.454 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000217584 2021-03-24T21:59:49.454 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.455 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.455 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,7", 2021-03-24T21:59:49.455 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.271621+0000", 2021-03-24T21:59:49.455 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.025077184999999998 2021-03-24T21:59:49.456 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.456 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.456 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:59:49.456 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.296698+0000", 2021-03-24T21:59:49.456 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.028701053000000001 2021-03-24T21:59:49.457 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.457 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.457 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:59:49.457 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.325399+0000", 2021-03-24T21:59:49.457 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00046768100000000001 2021-03-24T21:59:49.458 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.458 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.458 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:59:49.459 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.325867+0000", 2021-03-24T21:59:49.459 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00018980099999999999 2021-03-24T21:59:49.460 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.460 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.460 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:59:49.460 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.326057+0000", 2021-03-24T21:59:49.461 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00015802000000000001 2021-03-24T21:59:49.461 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.461 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.461 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:59:49.462 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.326215+0000", 2021-03-24T21:59:49.462 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.092194254000000003 2021-03-24T21:59:49.462 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:49.462 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:59:49.463 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:49.463 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.463 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.463 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:3831 7.c 7:31cb714a:::smithi12556470-7:head [write 2772623~84666] snapc 125=[125,123,120,11e,11b,11a,119,110,10f,10e,10d,10c,106,105,104,fa,f8,e7] ondisk+write+known_if_redirected e400)", 2021-03-24T21:59:49.464 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:30.241057+0000", 2021-03-24T21:59:49.464 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 18.942787607, 2021-03-24T21:59:49.464 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.085535926999999998, 2021-03-24T21:59:49.464 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:59:49.465 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:59:49.465 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:59:49.465 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:59:49.465 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:59:49.466 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3831 2021-03-24T21:59:49.466 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.466 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:59:49.466 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.467 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:59:49.467 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.241057+0000", 2021-03-24T21:59:49.467 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:59:49.467 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.467 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.468 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:59:49.468 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.241057+0000", 2021-03-24T21:59:49.468 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999962 2021-03-24T21:59:49.468 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.469 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.469 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:59:49.469 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.241053+0000", 2021-03-24T21:59:49.469 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000103366 2021-03-24T21:59:49.469 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.470 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.470 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:59:49.470 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.241156+0000", 2021-03-24T21:59:49.470 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.9848e-05 2021-03-24T21:59:49.471 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.471 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.471 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:59:49.471 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.241186+0000", 2021-03-24T21:59:49.472 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.4022000000000001e-05 2021-03-24T21:59:49.472 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.472 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.472 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:59:49.472 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.241200+0000", 2021-03-24T21:59:49.473 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00016864399999999999 2021-03-24T21:59:49.473 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.473 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.473 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:49.474 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.241369+0000", 2021-03-24T21:59:49.474 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.7857999999999997e-05 2021-03-24T21:59:49.474 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.474 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.475 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting_for_map not empty", 2021-03-24T21:59:49.475 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.241467+0000", 2021-03-24T21:59:49.475 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.044281963000000001 2021-03-24T21:59:49.475 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.476 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.476 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:49.476 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.285749+0000", 2021-03-24T21:59:49.477 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00020886 2021-03-24T21:59:49.477 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.478 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.478 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:59:49.478 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.285958+0000", 2021-03-24T21:59:49.479 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00037323800000000002 2021-03-24T21:59:49.479 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.479 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.480 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,7", 2021-03-24T21:59:49.480 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.286331+0000", 2021-03-24T21:59:49.480 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.016125222000000002 2021-03-24T21:59:49.481 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.481 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.481 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:59:49.482 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.302456+0000", 2021-03-24T21:59:49.482 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.022986532 2021-03-24T21:59:49.483 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.483 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.483 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:59:49.484 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.325443+0000", 2021-03-24T21:59:49.484 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00079908900000000001 2021-03-24T21:59:49.484 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.485 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.485 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:59:49.485 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.326242+0000", 2021-03-24T21:59:49.486 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00018786399999999999 2021-03-24T21:59:49.486 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.487 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.487 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:59:49.487 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.326430+0000", 2021-03-24T21:59:49.488 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00016322400000000001 2021-03-24T21:59:49.488 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.488 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.488 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:59:49.489 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.326593+0000", 2021-03-24T21:59:49.489 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.085535926999999998 2021-03-24T21:59:49.489 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:49.489 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:59:49.490 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:49.490 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.490 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.490 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:3832 7.c 7:31cb714a:::smithi12556470-7:head [setxattr _header (57) in=7b,truncate 2857289] snapc 125=[125,123,120,11e,11b,11a,119,110,10f,10e,10d,10c,106,105,104,fa,f8,e7] ondisk+write+known_if_redirected e400)", 2021-03-24T21:59:49.490 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:30.241220+0000", 2021-03-24T21:59:49.491 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 18.942624756000001, 2021-03-24T21:59:49.491 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.085821088000000004, 2021-03-24T21:59:49.491 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:59:49.491 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:59:49.492 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:59:49.492 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:59:49.492 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:59:49.492 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3832 2021-03-24T21:59:49.493 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.493 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:59:49.493 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.493 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:59:49.494 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.241220+0000", 2021-03-24T21:59:49.494 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:59:49.494 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.494 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.494 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:59:49.495 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.241220+0000", 2021-03-24T21:59:49.495 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T21:59:49.495 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.495 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.495 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:59:49.496 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.241219+0000", 2021-03-24T21:59:49.496 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2539999999999998e-06 2021-03-24T21:59:49.496 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.496 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.497 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:59:49.497 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.241225+0000", 2021-03-24T21:59:49.497 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.0859999999999996e-06 2021-03-24T21:59:49.497 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.498 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.498 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:59:49.498 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.241232+0000", 2021-03-24T21:59:49.498 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1019999999999996e-06 2021-03-24T21:59:49.498 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.499 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.499 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:59:49.499 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.241237+0000", 2021-03-24T21:59:49.499 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00029782700000000002 2021-03-24T21:59:49.500 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.500 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.500 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:49.500 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.241535+0000", 2021-03-24T21:59:49.501 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.4120000000000007e-06 2021-03-24T21:59:49.501 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.501 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.501 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting_for_map not empty", 2021-03-24T21:59:49.502 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.241545+0000", 2021-03-24T21:59:49.502 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.045443089999999998 2021-03-24T21:59:49.502 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.502 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.503 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:49.503 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.286988+0000", 2021-03-24T21:59:49.503 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010259699999999999 2021-03-24T21:59:49.503 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.503 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.504 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:59:49.504 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.287090+0000", 2021-03-24T21:59:49.504 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00020753699999999999 2021-03-24T21:59:49.504 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.505 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.505 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,7", 2021-03-24T21:59:49.505 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.287298+0000", 2021-03-24T21:59:49.505 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.015207745999999999 2021-03-24T21:59:49.506 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.506 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.506 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:59:49.506 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.302505+0000", 2021-03-24T21:59:49.507 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.022981549 2021-03-24T21:59:49.507 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.507 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.507 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:59:49.507 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.325487+0000", 2021-03-24T21:59:49.508 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001131197 2021-03-24T21:59:49.508 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.508 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.508 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:59:49.509 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.326618+0000", 2021-03-24T21:59:49.509 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00018641 2021-03-24T21:59:49.509 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.509 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.509 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:59:49.510 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.326805+0000", 2021-03-24T21:59:49.510 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000236197 2021-03-24T21:59:49.510 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.510 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.511 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:59:49.511 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.327041+0000", 2021-03-24T21:59:49.511 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.085821088000000004 2021-03-24T21:59:49.511 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:49.511 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:59:49.512 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:49.512 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.512 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.512 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:3833 7.c 7:31cb714a:::smithi12556470-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e400)", 2021-03-24T21:59:49.513 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:30.241243+0000", 2021-03-24T21:59:49.513 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 18.942601870000001, 2021-03-24T21:59:49.513 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.086069089000000001, 2021-03-24T21:59:49.513 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:59:49.513 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T21:59:49.514 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:59:49.514 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:59:49.514 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:59:49.515 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3833 2021-03-24T21:59:49.515 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.515 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:59:49.515 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.515 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:59:49.516 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.241243+0000", 2021-03-24T21:59:49.516 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:59:49.516 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.516 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.517 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:59:49.517 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.241243+0000", 2021-03-24T21:59:49.517 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999995 2021-03-24T21:59:49.517 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.517 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.518 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:59:49.518 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.241242+0000", 2021-03-24T21:59:49.518 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.3799999999999998e-06 2021-03-24T21:59:49.518 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.519 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.519 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:59:49.519 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.241245+0000", 2021-03-24T21:59:49.519 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0127e-05 2021-03-24T21:59:49.519 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.520 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.520 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:59:49.520 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.241256+0000", 2021-03-24T21:59:49.520 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2660000000000001e-06 2021-03-24T21:59:49.521 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.521 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.521 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:59:49.521 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.241261+0000", 2021-03-24T21:59:49.521 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00038992699999999998 2021-03-24T21:59:49.522 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.522 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.522 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:49.522 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.241651+0000", 2021-03-24T21:59:49.522 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0039e-05 2021-03-24T21:59:49.523 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.523 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.523 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting_for_map not empty", 2021-03-24T21:59:49.524 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.241661+0000", 2021-03-24T21:59:49.524 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.047469484999999999 2021-03-24T21:59:49.524 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.524 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.525 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:49.525 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.289130+0000", 2021-03-24T21:59:49.525 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.8741999999999998e-05 2021-03-24T21:59:49.526 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.526 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.527 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T21:59:49.527 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.289229+0000", 2021-03-24T21:59:49.527 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.037842530999999999 2021-03-24T21:59:49.528 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.528 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.529 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:49.529 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.327072+0000", 2021-03-24T21:59:49.530 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000106603 2021-03-24T21:59:49.530 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.531 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.531 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:59:49.532 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.327178+0000", 2021-03-24T21:59:49.532 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00013354600000000001 2021-03-24T21:59:49.532 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.533 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.533 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:59:49.534 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.327312+0000", 2021-03-24T21:59:49.534 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.086069089000000001 2021-03-24T21:59:49.535 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:49.535 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:59:49.535 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:49.536 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.536 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.537 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:3844 7.1b e400/282 7:da3cc97e:::smithi12556470-14:head v 400'547, mlcod=399'542)", 2021-03-24T21:59:49.537 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:30.263220+0000", 2021-03-24T21:59:49.538 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 18.920624093000001, 2021-03-24T21:59:49.538 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.068552193999999997, 2021-03-24T21:59:49.539 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:59:49.539 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:59:49.540 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:59:49.540 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.541 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:59:49.541 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.263220+0000", 2021-03-24T21:59:49.541 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:59:49.542 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.542 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.543 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:59:49.543 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.263220+0000", 2021-03-24T21:59:49.544 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999976 2021-03-24T21:59:49.544 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.545 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.545 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:59:49.546 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.263218+0000", 2021-03-24T21:59:49.546 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.4243999999999999e-05 2021-03-24T21:59:49.547 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.547 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.547 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:59:49.548 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.263232+0000", 2021-03-24T21:59:49.548 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5965999999999999e-05 2021-03-24T21:59:49.549 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.549 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.550 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:59:49.550 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.263248+0000", 2021-03-24T21:59:49.551 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.6860000000000006e-06 2021-03-24T21:59:49.551 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.551 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.552 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:59:49.552 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.263256+0000", 2021-03-24T21:59:49.553 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.017319199 2021-03-24T21:59:49.553 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.554 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.554 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:49.554 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.280575+0000", 2021-03-24T21:59:49.555 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.5725000000000001e-05 2021-03-24T21:59:49.555 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.556 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.556 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:59:49.557 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.280611+0000", 2021-03-24T21:59:49.557 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.051060853000000003 2021-03-24T21:59:49.558 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.558 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.559 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:59:49.559 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.331672+0000", 2021-03-24T21:59:49.560 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000101043 2021-03-24T21:59:49.560 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.560 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.561 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:59:49.561 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.331773+0000", 2021-03-24T21:59:49.562 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.068552193999999997 2021-03-24T21:59:49.562 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:49.563 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:59:49.563 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:49.563 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.564 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.564 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:3950 7.c 7:32974111:::smithi12556470-2:head [write 675780~495523,stat] snapc 136=[136,133,132,130,12d,12c,12b,129,128,127,126,120,11e,11b,11a,119,110,10f,10c,106,105,104,fa,f8,e7] ondisk+write+known_if_redirected e417)", 2021-03-24T21:59:49.565 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:47.297626+0000", 2021-03-24T21:59:49.565 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 1.886218427, 2021-03-24T21:59:49.566 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0613052380000001, 2021-03-24T21:59:49.566 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:59:49.567 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:59:49.567 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:59:49.568 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:59:49.568 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:59:49.568 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3950 2021-03-24T21:59:49.569 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.569 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:59:49.570 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.570 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:59:49.571 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.297626+0000", 2021-03-24T21:59:49.571 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:59:49.572 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.572 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.572 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:59:49.573 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.297626+0000", 2021-03-24T21:59:49.573 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999976 2021-03-24T21:59:49.574 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.574 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.574 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:59:49.574 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.297624+0000", 2021-03-24T21:59:49.575 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00032374599999999997 2021-03-24T21:59:49.575 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.575 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.575 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:59:49.576 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.297948+0000", 2021-03-24T21:59:49.576 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1829e-05 2021-03-24T21:59:49.576 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.576 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.576 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:59:49.577 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.297959+0000", 2021-03-24T21:59:49.577 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.9770000000000003e-06 2021-03-24T21:59:49.577 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.577 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.578 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:59:49.578 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.297966+0000", 2021-03-24T21:59:49.578 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001568081 2021-03-24T21:59:49.578 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.578 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.579 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:49.579 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299534+0000", 2021-03-24T21:59:49.579 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.7069999999999998e-06 2021-03-24T21:59:49.579 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.579 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.580 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T21:59:49.580 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299543+0000", 2021-03-24T21:59:49.580 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.02708646 2021-03-24T21:59:49.580 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.581 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.581 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:49.581 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.326630+0000", 2021-03-24T21:59:49.581 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000180722 2021-03-24T21:59:49.581 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.582 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.582 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:59:49.582 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.326810+0000", 2021-03-24T21:59:49.582 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00025251599999999999 2021-03-24T21:59:49.583 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.583 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.583 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,2", 2021-03-24T21:59:49.583 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.327063+0000", 2021-03-24T21:59:49.583 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0041170629999999998 2021-03-24T21:59:49.584 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.584 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.584 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:59:49.584 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.331180+0000", 2021-03-24T21:59:49.585 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.024822364999999999 2021-03-24T21:59:49.585 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.585 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.585 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:59:49.585 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.356002+0000", 2021-03-24T21:59:49.586 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000346561 2021-03-24T21:59:49.586 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.586 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.586 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:59:49.586 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.356349+0000", 2021-03-24T21:59:49.587 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00048905499999999996 2021-03-24T21:59:49.587 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.587 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.587 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:59:49.588 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.356838+0000", 2021-03-24T21:59:49.588 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0020934259999999998 2021-03-24T21:59:49.588 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.588 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.588 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:59:49.589 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.358931+0000", 2021-03-24T21:59:49.589 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0613052380000001 2021-03-24T21:59:49.589 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:49.589 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:59:49.589 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:49.590 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.590 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.590 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:3951 7.c 7:32974111:::smithi12556470-2:head [write 1682747~714511] snapc 136=[136,133,132,130,12d,12c,12b,129,128,127,126,120,11e,11b,11a,119,110,10f,10c,106,105,104,fa,f8,e7] ondisk+write+known_if_redirected e417)", 2021-03-24T21:59:49.590 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:47.298949+0000", 2021-03-24T21:59:49.591 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 1.8848954870000001, 2021-03-24T21:59:49.591 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0693593290000001, 2021-03-24T21:59:49.591 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:59:49.591 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:59:49.591 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:59:49.592 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:59:49.592 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:59:49.592 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3951 2021-03-24T21:59:49.592 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.593 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:59:49.593 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.593 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:59:49.593 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.298949+0000", 2021-03-24T21:59:49.593 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:59:49.594 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.594 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.594 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:59:49.594 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.298949+0000", 2021-03-24T21:59:49.594 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T21:59:49.595 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.595 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.595 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:59:49.595 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.298948+0000", 2021-03-24T21:59:49.596 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000412256 2021-03-24T21:59:49.596 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.596 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.596 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:59:49.596 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299360+0000", 2021-03-24T21:59:49.597 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1399999999999999e-05 2021-03-24T21:59:49.597 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.597 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.597 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:59:49.597 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299371+0000", 2021-03-24T21:59:49.598 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.3679999999999996e-06 2021-03-24T21:59:49.598 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.598 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.598 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:59:49.599 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299379+0000", 2021-03-24T21:59:49.599 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00048704199999999998 2021-03-24T21:59:49.599 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.599 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.599 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:49.600 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299866+0000", 2021-03-24T21:59:49.600 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.277e-06 2021-03-24T21:59:49.600 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.600 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.601 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T21:59:49.601 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299873+0000", 2021-03-24T21:59:49.601 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0281777940000001 2021-03-24T21:59:49.601 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.602 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.602 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:49.602 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.328051+0000", 2021-03-24T21:59:49.602 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000151185 2021-03-24T21:59:49.603 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.603 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.603 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T21:59:49.603 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.328202+0000", 2021-03-24T21:59:49.604 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.030788554999999999 2021-03-24T21:59:49.604 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.604 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.604 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:49.604 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.358991+0000", 2021-03-24T21:59:49.605 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012337099999999999 2021-03-24T21:59:49.605 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.605 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.605 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:59:49.606 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.359114+0000", 2021-03-24T21:59:49.606 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00022504400000000001 2021-03-24T21:59:49.606 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.606 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.606 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,2", 2021-03-24T21:59:49.607 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.359339+0000", 2021-03-24T21:59:49.607 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0044580849999999997 2021-03-24T21:59:49.607 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.607 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.608 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:59:49.608 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.363797+0000", 2021-03-24T21:59:49.608 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0031527479999999999 2021-03-24T21:59:49.608 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.608 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.609 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:59:49.609 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.366950+0000", 2021-03-24T21:59:49.609 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001068685 2021-03-24T21:59:49.609 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.610 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.610 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:59:49.610 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.368018+0000", 2021-03-24T21:59:49.610 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.4264999999999995e-05 2021-03-24T21:59:49.610 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.611 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.611 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:59:49.611 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.368093+0000", 2021-03-24T21:59:49.611 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00021562 2021-03-24T21:59:49.611 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.612 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.612 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:59:49.612 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.368308+0000", 2021-03-24T21:59:49.612 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0693593290000001 2021-03-24T21:59:49.613 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:49.613 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:59:49.613 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:49.613 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.613 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.614 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:3952 7.c 7:32974111:::smithi12556470-2:head [write 3012769~63011] snapc 136=[136,133,132,130,12d,12c,12b,129,128,127,126,120,11e,11b,11a,119,110,10f,10c,106,105,104,fa,f8,e7] ondisk+write+known_if_redirected e417)", 2021-03-24T21:59:49.614 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:47.299385+0000", 2021-03-24T21:59:49.614 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 1.884459141, 2021-03-24T21:59:49.614 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.070508668, 2021-03-24T21:59:49.615 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:59:49.615 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:59:49.615 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:59:49.615 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:59:49.615 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:59:49.616 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3952 2021-03-24T21:59:49.616 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.616 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:59:49.616 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.616 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:59:49.617 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299385+0000", 2021-03-24T21:59:49.617 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:59:49.617 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.617 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.618 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:59:49.618 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299385+0000", 2021-03-24T21:59:49.618 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T21:59:49.618 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.619 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.619 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:59:49.619 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299384+0000", 2021-03-24T21:59:49.619 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.6712000000000001e-05 2021-03-24T21:59:49.619 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.620 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.620 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:59:49.620 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299411+0000", 2021-03-24T21:59:49.620 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6420000000000001e-06 2021-03-24T21:59:49.621 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.621 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.621 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:59:49.621 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299418+0000", 2021-03-24T21:59:49.621 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6809999999999997e-06 2021-03-24T21:59:49.622 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.622 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.622 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:59:49.622 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299422+0000", 2021-03-24T21:59:49.623 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000472101 2021-03-24T21:59:49.623 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.623 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.623 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:49.623 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299895+0000", 2021-03-24T21:59:49.624 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.0889999999999996e-06 2021-03-24T21:59:49.624 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.624 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.624 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T21:59:49.624 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299901+0000", 2021-03-24T21:59:49.625 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0283333750000001 2021-03-24T21:59:49.625 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.625 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.625 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:49.626 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.328234+0000", 2021-03-24T21:59:49.626 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000104591 2021-03-24T21:59:49.626 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.626 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.626 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T21:59:49.627 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.328339+0000", 2021-03-24T21:59:49.627 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.033718655 2021-03-24T21:59:49.627 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.627 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.628 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:49.628 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.362057+0000", 2021-03-24T21:59:49.628 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00014753499999999999 2021-03-24T21:59:49.628 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.629 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.629 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:59:49.629 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.362205+0000", 2021-03-24T21:59:49.629 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000219259 2021-03-24T21:59:49.629 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.630 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.630 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,2", 2021-03-24T21:59:49.630 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.362424+0000", 2021-03-24T21:59:49.630 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001410356 2021-03-24T21:59:49.631 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.631 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.631 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:59:49.631 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.363834+0000", 2021-03-24T21:59:49.631 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.003124342 2021-03-24T21:59:49.632 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.632 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.632 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:59:49.632 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.366959+0000", 2021-03-24T21:59:49.632 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0014798490000000001 2021-03-24T21:59:49.633 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.633 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.633 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:59:49.633 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.368439+0000", 2021-03-24T21:59:49.634 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001231179 2021-03-24T21:59:49.634 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.634 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.634 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:59:49.635 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.369670+0000", 2021-03-24T21:59:49.635 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00022420900000000001 2021-03-24T21:59:49.635 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.635 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.635 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:59:49.636 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.369894+0000", 2021-03-24T21:59:49.636 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.070508668 2021-03-24T21:59:49.636 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:49.636 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:59:49.636 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:49.637 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.637 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.637 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:3953 7.c 7:32974111:::smithi12556470-2:head [setxattr _header (57) in=7b,truncate 3075780] snapc 136=[136,133,132,130,12d,12c,12b,129,128,127,126,120,11e,11b,11a,119,110,10f,10c,106,105,104,fa,f8,e7] ondisk+write+known_if_redirected e417)", 2021-03-24T21:59:49.637 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:47.299427+0000", 2021-03-24T21:59:49.638 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 1.884417373, 2021-03-24T21:59:49.638 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0708905799999999, 2021-03-24T21:59:49.638 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:59:49.638 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T21:59:49.639 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:59:49.639 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:59:49.639 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:59:49.639 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3953 2021-03-24T21:59:49.639 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.640 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:59:49.640 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.640 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:59:49.640 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299427+0000", 2021-03-24T21:59:49.641 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:59:49.641 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.641 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.641 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:59:49.641 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299427+0000", 2021-03-24T21:59:49.642 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999995 2021-03-24T21:59:49.642 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.642 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.642 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:59:49.643 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299427+0000", 2021-03-24T21:59:49.643 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.4259999999999999e-06 2021-03-24T21:59:49.643 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.643 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.643 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:59:49.644 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299430+0000", 2021-03-24T21:59:49.644 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1189999999999996e-06 2021-03-24T21:59:49.644 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.644 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.645 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:59:49.645 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299435+0000", 2021-03-24T21:59:49.645 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9140000000000001e-06 2021-03-24T21:59:49.645 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.645 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.646 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:59:49.646 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299439+0000", 2021-03-24T21:59:49.646 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00048008799999999999 2021-03-24T21:59:49.646 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.647 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.647 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:49.647 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299919+0000", 2021-03-24T21:59:49.647 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.835e-06 2021-03-24T21:59:49.647 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.648 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.648 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T21:59:49.648 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299925+0000", 2021-03-24T21:59:49.648 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.028441138 2021-03-24T21:59:49.649 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.649 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.649 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:49.649 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.328366+0000", 2021-03-24T21:59:49.649 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010274500000000001 2021-03-24T21:59:49.650 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.650 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.650 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T21:59:49.650 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.328469+0000", 2021-03-24T21:59:49.651 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.034522276999999997 2021-03-24T21:59:49.651 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.651 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.651 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:49.651 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.362991+0000", 2021-03-24T21:59:49.652 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000114469 2021-03-24T21:59:49.652 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.652 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.652 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:59:49.653 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.363106+0000", 2021-03-24T21:59:49.653 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00021205599999999999 2021-03-24T21:59:49.653 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.653 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.653 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,2", 2021-03-24T21:59:49.654 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.363318+0000", 2021-03-24T21:59:49.654 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00096840900000000004 2021-03-24T21:59:49.654 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.654 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.655 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:59:49.655 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.364286+0000", 2021-03-24T21:59:49.655 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002681143 2021-03-24T21:59:49.655 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.655 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.656 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T21:59:49.656 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.366967+0000", 2021-03-24T21:59:49.656 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002998559 2021-03-24T21:59:49.656 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.657 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.657 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T21:59:49.657 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.369966+0000", 2021-03-24T21:59:49.657 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012207299999999999 2021-03-24T21:59:49.657 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.658 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.658 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T21:59:49.658 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.370088+0000", 2021-03-24T21:59:49.658 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00022991299999999999 2021-03-24T21:59:49.658 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.659 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.659 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:59:49.659 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.370318+0000", 2021-03-24T21:59:49.659 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0708905799999999 2021-03-24T21:59:49.660 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:49.660 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:59:49.660 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:49.660 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.660 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.661 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:3954 7.c 7:32974111:::smithi12556470-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e417)", 2021-03-24T21:59:49.661 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:47.299443+0000", 2021-03-24T21:59:49.661 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 1.88440169, 2021-03-24T21:59:49.661 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.071832683, 2021-03-24T21:59:49.662 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:59:49.662 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T21:59:49.662 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T21:59:49.662 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T21:59:49.663 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T21:59:49.663 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3954 2021-03-24T21:59:49.663 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.663 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:59:49.663 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.664 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:59:49.664 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299443+0000", 2021-03-24T21:59:49.664 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:59:49.664 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.664 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.665 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:59:49.665 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299443+0000", 2021-03-24T21:59:49.665 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999995 2021-03-24T21:59:49.665 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.666 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.666 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:59:49.666 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299442+0000", 2021-03-24T21:59:49.666 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.2239999999999998e-06 2021-03-24T21:59:49.666 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.667 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.667 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:59:49.667 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299445+0000", 2021-03-24T21:59:49.667 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.3809999999999998e-06 2021-03-24T21:59:49.668 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.668 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.668 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:59:49.668 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299450+0000", 2021-03-24T21:59:49.668 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9160000000000003e-06 2021-03-24T21:59:49.669 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.669 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.669 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:59:49.669 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299454+0000", 2021-03-24T21:59:49.670 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00049169700000000003 2021-03-24T21:59:49.670 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.670 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.670 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:49.670 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299946+0000", 2021-03-24T21:59:49.671 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.003e-06 2021-03-24T21:59:49.671 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.671 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.671 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T21:59:49.672 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299952+0000", 2021-03-24T21:59:49.672 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0285410699999999 2021-03-24T21:59:49.672 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.672 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.673 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:49.673 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.328493+0000", 2021-03-24T21:59:49.673 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0015757410000000001 2021-03-24T21:59:49.673 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.673 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.674 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T21:59:49.674 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.330068+0000", 2021-03-24T21:59:49.674 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.033581221000000001 2021-03-24T21:59:49.674 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.675 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.675 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:49.675 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.363650+0000", 2021-03-24T21:59:49.675 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010419700000000001 2021-03-24T21:59:49.675 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.676 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.676 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T21:59:49.676 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.363754+0000", 2021-03-24T21:59:49.676 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0070060599999999997 2021-03-24T21:59:49.677 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.677 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.677 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:49.677 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.370760+0000", 2021-03-24T21:59:49.678 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00020776099999999999 2021-03-24T21:59:49.678 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.678 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.678 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T21:59:49.678 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.370968+0000", 2021-03-24T21:59:49.679 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00030779100000000002 2021-03-24T21:59:49.679 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.679 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.679 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:59:49.680 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.371275+0000", 2021-03-24T21:59:49.680 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.071832683 2021-03-24T21:59:49.680 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:49.680 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:59:49.680 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:49.681 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.681 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.681 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "MOSDPGPushReply(4.11 418/417 [PushReplyOp(4:898f825a:::data_log.11:head)])", 2021-03-24T21:59:49.681 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:48.611078+0000", 2021-03-24T21:59:49.682 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 0.57276695099999997, 2021-03-24T21:59:49.682 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.089347641000000005, 2021-03-24T21:59:49.682 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:59:49.682 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "reached pg", 2021-03-24T21:59:49.683 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:59:49.683 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.683 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:59:49.683 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.611078+0000", 2021-03-24T21:59:49.683 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:59:49.684 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.684 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.684 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:59:49.684 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.611078+0000", 2021-03-24T21:59:49.685 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999971 2021-03-24T21:59:49.685 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.685 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.685 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:59:49.685 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.611075+0000", 2021-03-24T21:59:49.686 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2187999999999999e-05 2021-03-24T21:59:49.686 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.686 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.686 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:59:49.687 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.611087+0000", 2021-03-24T21:59:49.687 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.2677e-05 2021-03-24T21:59:49.687 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.687 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.687 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:59:49.688 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.611110+0000", 2021-03-24T21:59:49.688 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.55e-06 2021-03-24T21:59:49.688 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.688 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.689 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:59:49.689 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.611119+0000", 2021-03-24T21:59:49.689 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.089152789999999996 2021-03-24T21:59:49.689 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.690 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.690 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:49.690 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.700272+0000", 2021-03-24T21:59:49.690 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00015309399999999999 2021-03-24T21:59:49.690 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.691 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.691 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:59:49.691 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.700425+0000", 2021-03-24T21:59:49.691 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.089347641000000005 2021-03-24T21:59:49.692 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:49.692 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:59:49.692 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:49.692 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.693 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.693 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "MOSDPGPushReply(4.11 418/417 [PushReplyOp(4:89dabe60:::data_log.118:head)])", 2021-03-24T21:59:49.693 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:48.873036+0000", 2021-03-24T21:59:49.693 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 0.310808474, 2021-03-24T21:59:49.693 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.175769548, 2021-03-24T21:59:49.694 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T21:59:49.694 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "reached pg", 2021-03-24T21:59:49.694 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T21:59:49.694 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.695 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T21:59:49.695 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.873036+0000", 2021-03-24T21:59:49.695 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T21:59:49.695 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.695 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.696 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T21:59:49.696 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.873036+0000", 2021-03-24T21:59:49.696 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T21:59:49.696 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.696 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.697 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T21:59:49.697 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.873035+0000", 2021-03-24T21:59:49.697 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.0559999999999999e-06 2021-03-24T21:59:49.697 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.698 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.698 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T21:59:49.698 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.873042+0000", 2021-03-24T21:59:49.698 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2588e-05 2021-03-24T21:59:49.698 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.699 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.699 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T21:59:49.699 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.873054+0000", 2021-03-24T21:59:49.699 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5929999999999996e-06 2021-03-24T21:59:49.700 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.700 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.700 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T21:59:49.700 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.873060+0000", 2021-03-24T21:59:49.700 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.17545031799999999 2021-03-24T21:59:49.701 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.701 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.701 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T21:59:49.701 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:49.048510+0000", 2021-03-24T21:59:49.702 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000295309 2021-03-24T21:59:49.702 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T21:59:49.702 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T21:59:49.702 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T21:59:49.702 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:49.048806+0000", 2021-03-24T21:59:49.703 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.175769548 2021-03-24T21:59:49.703 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:49.703 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:59:49.703 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:49.704 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T21:59:49.704 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T21:59:49.704 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-24T21:59:49.705 INFO:tasks.rados.rados.0.smithi125.stdout:1527: done (7 left) 2021-03-24T21:59:49.705 INFO:tasks.rados.rados.0.smithi125.stdout:1528: done (6 left) 2021-03-24T21:59:49.706 INFO:tasks.rados.rados.0.smithi125.stdout:1529: done (5 left) 2021-03-24T21:59:49.706 INFO:tasks.rados.rados.0.smithi125.stdout:1530: done (4 left) 2021-03-24T21:59:49.706 INFO:tasks.rados.rados.0.smithi125.stdout:1531: done (3 left) 2021-03-24T21:59:49.706 INFO:tasks.rados.rados.0.smithi125.stdout:1532: done (2 left) 2021-03-24T21:59:49.707 INFO:tasks.rados.rados.0.smithi125.stdout:1533: done (1 left) 2021-03-24T21:59:49.707 INFO:tasks.rados.rados.0.smithi125.stdout:1534: done (0 left) 2021-03-24T21:59:49.707 INFO:tasks.rados.rados.0.smithi125.stdout:1535: copy_from oid 31 from oid 9 current snap is 168 2021-03-24T21:59:49.707 INFO:tasks.rados.rados.0.smithi125.stdout:1536: write (excl) oid 4 current snap is 168 2021-03-24T21:59:49.707 INFO:tasks.rados.rados.0.smithi125.stdout:1536: seq_num 544 ranges {743008=796519,2200322=721849} 2021-03-24T21:59:49.708 INFO:tasks.rados.rados.0.smithi125.stdout:1536: writing smithi12556470-4 from 743008 to 1539527 tid 1 2021-03-24T21:59:49.708 INFO:tasks.rados.rados.0.smithi125.stdout:1536: writing smithi12556470-4 from 2200322 to 2922171 tid 2 2021-03-24T21:59:49.708 INFO:tasks.rados.rados.0.smithi125.stdout:1537: read oid 43 snap -1 2021-03-24T21:59:49.708 INFO:tasks.rados.rados.0.smithi125.stdout:1537: expect (ObjNum 399 snap 102 seq_num 399) 2021-03-24T21:59:49.709 INFO:tasks.rados.rados.0.smithi125.stdout:1538: snap_remove snap 162 2021-03-24T21:59:49.709 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-03-24T21:59:49.710 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-03-24T21:59:49.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:49.712 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:49.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:59:49.813 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:49.861 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:49.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:49.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:59:49.966 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:50.034 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:50.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:50.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:59:50.139 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:50.167 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.4 dump_ops_in_flight 2021-03-24T21:59:50.192 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:50.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:50.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:59:50.225 INFO:tasks.rados.rados.0.smithi125.stdout:1538: done (3 left) 2021-03-24T21:59:50.226 INFO:tasks.rados.rados.0.smithi125.stdout:1539: write (excl) oid 47 current snap is 168 2021-03-24T21:59:50.226 INFO:tasks.rados.rados.0.smithi125.stdout:1539: seq_num 545 ranges {759815=471768,1742451=465507,2917178=242637} 2021-03-24T21:59:50.233 INFO:tasks.rados.rados.0.smithi125.stdout:1539: writing smithi12556470-47 from 759815 to 1231583 tid 1 2021-03-24T21:59:50.239 INFO:tasks.rados.rados.0.smithi125.stdout:1539: writing smithi12556470-47 from 1742451 to 2207958 tid 2 2021-03-24T21:59:50.245 INFO:tasks.rados.rados.0.smithi125.stdout:1539: writing smithi12556470-47 from 2917178 to 3159815 tid 3 2021-03-24T21:59:50.245 INFO:tasks.rados.rados.0.smithi125.stdout:1540: snap_create 2021-03-24T21:59:50.253 INFO:tasks.rados.rados.0.smithi125.stdout:1536: finishing write tid 1 to smithi12556470-4 2021-03-24T21:59:50.253 INFO:tasks.rados.rados.0.smithi125.stdout:1537: expect (ObjNum 399 snap 102 seq_num 399) 2021-03-24T21:59:50.297 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:50.357 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:50.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:50.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:50.367 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:50 smithi125 conmon[24927]: cluster 2021-03-24T21:59:48.840973+0000 2021-03-24T21:59:50.367 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:50 smithi125 conmon[24927]: mgr.y (mgr.14144) 480 : cluster [DBG] pgmap v790: 177 pgs: 18 remapped+peering, 16 peering, 8 active+clean+remapped, 43 active+undersized, 2 active+recovering, 29 active+undersized+degraded, 61 active+clean; 387 MiB data, 691 MiB used, 446 GiB / 447 GiB avail; 4.7 MiB/s rd, 1.9 MiB/s wr, 9 op/s; 193/1503 objects degraded (12.841%); 40/1503 objects misplaced (2.661%); 144 B/s, 2 objects/s recovering 2021-03-24T21:59:50.367 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:50 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:59:50.367 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:50 smithi125 conmon[24927]: 59:49.248133+0000 mon.a (mon.0) 1232 : cluster [DBG] osdmap e419: 8 total, 7 up, 5 in 2021-03-24T21:59:50.368 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:50 smithi125 conmon[24927]: cluster 2021-03-24T21 2021-03-24T21:59:50.368 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:50 smithi125 conmon[24927]: :59:50.212992+0000 mon.a (mon.0) 1233 : cluster [WRN] Health check update: Degraded data redundancy: 193/1503 objects degraded (12.841%), 29 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2021-03-24T21:59:50.368 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:50 smithi125 conmon[24927]: cluster 2021-03-24T21:59:50. 2021-03-24T21:59:50.369 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:50 smithi125 conmon[24927]: 218733+0000 mon.a (mon.0) 1234 : cluster [DBG] osdmap e420: 8 total, 7 up, 5 in 2021-03-24T21:59:50.369 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:50 smithi125 conmon[19936]: cluster 2021-03-24T21:59:48.840973+0000 2021-03-24T21:59:50.369 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:50 smithi125 conmon[19936]: mgr.y (mgr.14144) 480 : cluster [DBG] pgmap v790: 177 pgs: 18 remapped+peering, 16 peering, 8 active+clean+remapped, 43 active+undersized, 2 active+recovering, 29 active+undersized+degraded, 61 active+clean; 387 MiB data, 691 MiB used, 446 GiB / 447 GiB avail; 4.7 MiB/s rd, 1.9 MiB/s wr, 9 op/s; 193/1503 objects degraded (12.841%); 40/1503 objects misplaced (2.661%); 144 B/s, 2 objects/s recovering 2021-03-24T21:59:50.370 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:50 smithi125 conmon[19936]: cluster 2021-03-24T21:59:49.248133+0000 mon.a (mon.0) 1232 : cluster [DBG] osdmap e419: 8 total, 7 up, 5 in 2021-03-24T21:59:50.370 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:50 smithi125 conmon[19936]: 2021-03-24T21:59:50.370 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:50 smithi125 conmon[19936]: cluster 2021-03-24T21 2021-03-24T21:59:50.370 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:50 smithi125 conmon[19936]: :59:50.212992+0000 mon.a (mon.0) 1233 2021-03-24T21:59:50.371 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:50 smithi125 conmon[19936]: : cluster [WRN] Health check update: Degraded data redundancy: 193/1503 objects degraded (12.841%), 29 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2021-03-24T21:59:50.371 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:50 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T21:59:50.371 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:50 smithi125 conmon[19936]: -24T21:59:50.218733+0000 2021-03-24T21:59:50.372 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:50 smithi125 conmon[19936]: mon.a (mon.0) 1234 : cluster [DBG] 2021-03-24T21:59:50.372 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:50 smithi125 conmon[19936]: osdmap e420: 8 total, 7 up, 5 in 2021-03-24T21:59:50.421 INFO:tasks.rados.rados.0.smithi125.stdout:1536: finishing write tid 2 to smithi12556470-4 2021-03-24T21:59:50.421 INFO:tasks.rados.rados.0.smithi125.stdout:1536: finishing write tid 3 to smithi12556470-4 2021-03-24T21:59:50.422 INFO:tasks.rados.rados.0.smithi125.stdout:1536: finishing write tid 4 to smithi12556470-4 2021-03-24T21:59:50.422 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 4 v 307 (ObjNum 544 snap 168 seq_num 544) dirty exists 2021-03-24T21:59:50.422 INFO:tasks.rados.rados.0.smithi125.stdout:1536: left oid 4 (ObjNum 544 snap 168 seq_num 544) 2021-03-24T21:59:50.423 INFO:tasks.rados.rados.0.smithi125.stdout:1535: finishing copy_from to smithi12556470-31 2021-03-24T21:59:50.423 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 31 v 539 (ObjNum 520 snap 154 seq_num 520) dirty exists 2021-03-24T21:59:50.424 INFO:tasks.rados.rados.0.smithi125.stdout:1535: finishing copy_from racing read to smithi12556470-31 2021-03-24T21:59:50.424 INFO:tasks.rados.rados.0.smithi125.stdout:1539: finishing write tid 1 to smithi12556470-47 2021-03-24T21:59:50.425 INFO:tasks.rados.rados.0.smithi125.stdout:1539: finishing write tid 2 to smithi12556470-47 2021-03-24T21:59:50.425 INFO:tasks.rados.rados.0.smithi125.stdout:1539: finishing write tid 3 to smithi12556470-47 2021-03-24T21:59:50.425 INFO:tasks.rados.rados.0.smithi125.stdout:1539: finishing write tid 4 to smithi12556470-47 2021-03-24T21:59:50.425 INFO:tasks.rados.rados.0.smithi125.stdout:1539: finishing write tid 5 to smithi12556470-47 2021-03-24T21:59:50.426 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 47 v 309 (ObjNum 545 snap 168 seq_num 545) dirty exists 2021-03-24T21:59:50.426 INFO:tasks.rados.rados.0.smithi125.stdout:1539: left oid 47 (ObjNum 545 snap 168 seq_num 545) 2021-03-24T21:59:50.463 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:50.473 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:50 smithi142 conmon[20125]: cluster 2021-03-24T21:59:48.840973+0000 2021-03-24T21:59:50.473 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:50 smithi142 conmon[20125]: mgr.y (mgr.14144) 480 : cluster [DBG] pgmap v790: 177 pgs: 18 remapped+peering, 16 peering, 8 active+clean+remapped, 43 active+undersized, 2 active+recovering, 29 active+undersized+degraded, 61 active+clean; 387 MiB data, 691 MiB used, 446 GiB / 447 GiB avail; 4.7 MiB/s rd, 1.9 MiB/s wr, 9 op/s; 193/1503 objects degraded (12.841%); 40/1503 objects misplaced (2.661%); 144 B/s, 2 objects/s recovering 2021-03-24T21:59:50.474 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:50 smithi142 conmon[20125]: cluster 2021-03-24T21:59:49.248133+0000 mon.a (mon.0) 1232 : cluster [DBG] osdmap e419: 8 total, 7 up, 5 in 2021-03-24T21:59:50.474 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:50 smithi142 conmon[20125]: cluster 2021-03-24T21:59:50.474 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:50 smithi142 conmon[20125]: 2021-03-24T21:59:50.212992+0000 mon.a (mon.0) 1233 : 2021-03-24T21:59:50.474 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:50 smithi142 conmon[20125]: cluster [WRN] Health check update: Degraded data redundancy: 193/1503 objects degraded (12.841%), 29 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2021-03-24T21:59:50.475 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:50 smithi142 conmon[20125]: cluster 2021-03-24T21:59:50 2021-03-24T21:59:50.475 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:50 smithi142 conmon[20125]: .218733+0000 mon.a (mon.0) 1234 : cluster 2021-03-24T21:59:50.475 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:50 smithi142 conmon[20125]: [DBG] osdmap e420: 8 total, 7 up, 5 in 2021-03-24T21:59:50.481 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T21:59:50.513 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:50.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:50.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:59:50.617 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:50.667 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:50.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:50.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:50.772 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:50.828 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:50.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:50.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:59:50.933 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:50.981 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:50.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:50.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:59:51.086 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:51.141 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:51.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:51.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:51.225 INFO:tasks.rados.rados.0.smithi125.stdout:1535: done (4 left) 2021-03-24T21:59:51.225 INFO:tasks.rados.rados.0.smithi125.stdout:1536: done (3 left) 2021-03-24T21:59:51.226 INFO:tasks.rados.rados.0.smithi125.stdout:1537: done (2 left) 2021-03-24T21:59:51.226 INFO:tasks.rados.rados.0.smithi125.stdout:1539: done (1 left) 2021-03-24T21:59:51.226 INFO:tasks.rados.rados.0.smithi125.stdout:1540: done (0 left) 2021-03-24T21:59:51.226 INFO:tasks.rados.rados.0.smithi125.stdout:1541: snap_create 2021-03-24T21:59:51.245 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:51.311 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:51.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:51.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:59:51.417 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:51.461 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:51.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:51.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:59:51.566 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:51.614 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:51.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:51.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:59:51.711 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-24T21:59:51.720 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:51.766 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:51.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:51.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:51.870 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:51.927 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:51.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:51.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:52.030 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:59:52.032 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:52.078 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:52.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:52.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:59:52.182 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:52.234 INFO:tasks.rados.rados.0.smithi125.stdout:1541: done (0 left) 2021-03-24T21:59:52.235 INFO:tasks.rados.rados.0.smithi125.stdout:1542: copy_from oid 7 from oid 29 current snap is 170 2021-03-24T21:59:52.235 INFO:tasks.rados.rados.0.smithi125.stdout:1543: copy_from oid 34 from oid 9 current snap is 170 2021-03-24T21:59:52.236 INFO:tasks.rados.rados.0.smithi125.stdout:1544: delete oid 20 current snap is 170 2021-03-24T21:59:52.241 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:52.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:52.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:52.250 INFO:teuthology.orchestra.run.smithi125.stderr:nodeep-scrub is unset 2021-03-24T21:59:52.266 INFO:tasks.rados.rados.0.smithi125.stdout:1544: done (2 left) 2021-03-24T21:59:52.267 INFO:tasks.rados.rados.0.smithi125.stdout:1545: delete oid 2 current snap is 170 2021-03-24T21:59:52.295 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: cluster 2021-03-24T21:59:52.295 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 2021-03-24T21:59:52.296 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 2021 2021-03-24T21:59:52.296 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: - 2021-03-24T21:59:52.297 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 03 2021-03-24T21:59:52.297 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: - 2021-03-24T21:59:52.297 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 24 2021-03-24T21:59:52.298 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: T 2021-03-24T21:59:52.298 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 21 2021-03-24T21:59:52.299 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: : 2021-03-24T21:59:52.299 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 59 2021-03-24T21:59:52.300 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: : 2021-03-24T21:59:52.300 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 50 2021-03-24T21:59:52.300 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: . 2021-03-24T21:59:52.301 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 842337 2021-03-24T21:59:52.301 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: +0000 2021-03-24T21:59:52.302 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 2021-03-24T21:59:52.302 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: mgr.y 2021-03-24T21:59:52.303 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: ( 2021-03-24T21:59:52.303 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: mgr 2021-03-24T21:59:52.304 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: . 2021-03-24T21:59:52.304 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 14144 2021-03-24T21:59:52.305 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: ) 2021-03-24T21:59:52.305 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 481 2021-03-24T21:59:52.306 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: : 2021-03-24T21:59:52.306 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: cluster 2021-03-24T21:59:52.306 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 2021-03-24T21:59:52.307 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: [DBG] 2021-03-24T21:59:52.307 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 2021-03-24T21:59:52.308 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: pgmap v793: 177 pgs: 18 remapped+peering, 15 peering, 24 active+clean+remapped, 2 active+recovering+undersized+remapped, 36 active+undersized, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering, 18 active+undersized+degraded, 1 active+recovering+degraded+remapped, 8 active+recovery_wait+degraded+remapped, 47 active+clean; 395 MiB data, 696 MiB used, 446 GiB / 447 GiB avail; 1.2 MiB/s wr, 6 op/s; 268/1527 objects degraded (17.551%); 222/1527 objects misplaced (14.538%); 125 B/s, 5 objects/s recovering 2021-03-24T21:59:52.308 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 2021-03-24T21:59:52.309 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: cluster 2021-03-24T21:59:52.309 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 2021-03-24T21:59:52.310 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 2021 2021-03-24T21:59:52.310 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: - 2021-03-24T21:59:52.310 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 03 2021-03-24T21:59:52.311 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: - 2021-03-24T21:59:52.311 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 24 2021-03-24T21:59:52.312 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: T 2021-03-24T21:59:52.312 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 21 2021-03-24T21:59:52.312 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: : 2021-03-24T21:59:52.313 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 59 2021-03-24T21:59:52.313 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: : 2021-03-24T21:59:52.314 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 51 2021-03-24T21:59:52.314 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: . 2021-03-24T21:59:52.315 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 223259 2021-03-24T21:59:52.315 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: +0000 2021-03-24T21:59:52.316 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 2021-03-24T21:59:52.316 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: mon.a 2021-03-24T21:59:52.316 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: ( 2021-03-24T21:59:52.317 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: mon 2021-03-24T21:59:52.317 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: . 2021-03-24T21:59:52.318 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 0 2021-03-24T21:59:52.318 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: ) 2021-03-24T21:59:52.319 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 1235 2021-03-24T21:59:52.319 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: : 2021-03-24T21:59:52.320 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: cluster 2021-03-24T21:59:52.320 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 2021-03-24T21:59:52.320 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: [DBG] 2021-03-24T21:59:52.321 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 2021-03-24T21:59:52.321 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: osdmap e421: 8 total, 7 up, 5 in 2021-03-24T21:59:52.322 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 2021-03-24T21:59:52.322 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: audit 2021-03-24T21:59:52.323 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 2021-03-24T21:59:52.323 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 2021 2021-03-24T21:59:52.324 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: - 2021-03-24T21:59:52.324 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 03 2021-03-24T21:59:52.324 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: - 2021-03-24T21:59:52.324 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 24 2021-03-24T21:59:52.324 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: T 2021-03-24T21:59:52.325 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 21 2021-03-24T21:59:52.325 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: : 2021-03-24T21:59:52.325 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 59 2021-03-24T21:59:52.325 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: : 2021-03-24T21:59:52.326 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 52 2021-03-24T21:59:52.326 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: . 2021-03-24T21:59:52.326 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 066082 2021-03-24T21:59:52.326 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: +0000 2021-03-24T21:59:52.326 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 2021-03-24T21:59:52.327 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: mon.c 2021-03-24T21:59:52.327 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: ( 2021-03-24T21:59:52.327 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: mon 2021-03-24T21:59:52.327 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: . 2021-03-24T21:59:52.327 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 2 2021-03-24T21:59:52.328 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: ) 2021-03-24T21:59:52.328 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 66 2021-03-24T21:59:52.328 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: : 2021-03-24T21:59:52.328 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: audit 2021-03-24T21:59:52.329 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 2021-03-24T21:59:52.329 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: [INF] 2021-03-24T21:59:52.329 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 2021-03-24T21:59:52.329 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: from='client.? v1:172.21.15.125:0/3055724187' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:59:52.330 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 2021-03-24T21:59:52.330 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: audit 2021-03-24T21:59:52.330 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 2021-03-24T21:59:52.330 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 2021 2021-03-24T21:59:52.331 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: - 2021-03-24T21:59:52.331 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 03 2021-03-24T21:59:52.331 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: - 2021-03-24T21:59:52.331 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 24 2021-03-24T21:59:52.331 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: T 2021-03-24T21:59:52.332 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 21 2021-03-24T21:59:52.332 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: : 2021-03-24T21:59:52.332 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 59 2021-03-24T21:59:52.332 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: : 2021-03-24T21:59:52.333 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 52 2021-03-24T21:59:52.333 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: . 2021-03-24T21:59:52.333 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 066560 2021-03-24T21:59:52.333 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: +0000 2021-03-24T21:59:52.333 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 2021-03-24T21:59:52.334 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: mon.a 2021-03-24T21:59:52.334 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: ( 2021-03-24T21:59:52.334 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: mon 2021-03-24T21:59:52.334 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: . 2021-03-24T21:59:52.335 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 0 2021-03-24T21:59:52.335 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: ) 2021-03-24T21:59:52.336 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 1236 2021-03-24T21:59:52.336 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: : 2021-03-24T21:59:52.336 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: audit 2021-03-24T21:59:52.336 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 2021-03-24T21:59:52.337 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: [INF] 2021-03-24T21:59:52.337 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 2021-03-24T21:59:52.337 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: from='client.35216 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:59:52.337 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[19936]: 2021-03-24T21:59:52.338 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: cluster 2021-03-24T21:59:52.338 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 2021-03-24T21:59:52.339 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 2021 2021-03-24T21:59:52.339 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: - 2021-03-24T21:59:52.339 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 03 2021-03-24T21:59:52.339 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: - 2021-03-24T21:59:52.339 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 24 2021-03-24T21:59:52.340 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: T 2021-03-24T21:59:52.340 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 21 2021-03-24T21:59:52.340 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: : 2021-03-24T21:59:52.340 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 59 2021-03-24T21:59:52.341 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: : 2021-03-24T21:59:52.341 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 50 2021-03-24T21:59:52.341 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: . 2021-03-24T21:59:52.341 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 842337 2021-03-24T21:59:52.342 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: +0000 2021-03-24T21:59:52.342 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 2021-03-24T21:59:52.342 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: mgr.y 2021-03-24T21:59:52.343 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: ( 2021-03-24T21:59:52.343 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: mgr 2021-03-24T21:59:52.344 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: . 2021-03-24T21:59:52.344 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 14144 2021-03-24T21:59:52.344 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: ) 2021-03-24T21:59:52.344 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 481 2021-03-24T21:59:52.345 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: : 2021-03-24T21:59:52.345 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: cluster 2021-03-24T21:59:52.345 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 2021-03-24T21:59:52.345 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: [DBG] 2021-03-24T21:59:52.346 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 2021-03-24T21:59:52.346 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: pgmap v793: 177 pgs: 18 remapped+peering, 15 peering, 24 active+clean+remapped, 2 active+recovering+undersized+remapped, 36 active+undersized, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering, 18 active+undersized+degraded, 1 active+recovering+degraded+remapped, 8 active+recovery_wait+degraded+remapped, 47 active+clean; 395 MiB data, 696 MiB used, 446 GiB / 447 GiB avail; 1.2 MiB/s wr, 6 op/s; 268/1527 objects degraded (17.551%); 222/1527 objects misplaced (14.538%); 125 B/s, 5 objects/s recovering 2021-03-24T21:59:52.346 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 2021-03-24T21:59:52.346 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: cluster 2021-03-24T21:59:52.347 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 2021-03-24T21:59:52.347 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 2021 2021-03-24T21:59:52.347 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: - 2021-03-24T21:59:52.347 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 03 2021-03-24T21:59:52.347 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: - 2021-03-24T21:59:52.348 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 24 2021-03-24T21:59:52.348 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: T 2021-03-24T21:59:52.348 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 21 2021-03-24T21:59:52.349 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: : 2021-03-24T21:59:52.349 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 59 2021-03-24T21:59:52.349 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: : 2021-03-24T21:59:52.349 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 51 2021-03-24T21:59:52.349 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: . 2021-03-24T21:59:52.350 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 223259 2021-03-24T21:59:52.350 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: +0000 2021-03-24T21:59:52.350 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 2021-03-24T21:59:52.350 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: mon.a 2021-03-24T21:59:52.350 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: ( 2021-03-24T21:59:52.351 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: mon 2021-03-24T21:59:52.351 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: . 2021-03-24T21:59:52.351 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 0 2021-03-24T21:59:52.352 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: ) 2021-03-24T21:59:52.352 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 1235 2021-03-24T21:59:52.352 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: : 2021-03-24T21:59:52.352 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: cluster 2021-03-24T21:59:52.353 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 2021-03-24T21:59:52.353 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: [DBG] 2021-03-24T21:59:52.353 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 2021-03-24T21:59:52.353 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: osdmap e421: 8 total, 7 up, 5 in 2021-03-24T21:59:52.353 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 2021-03-24T21:59:52.354 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: audit 2021-03-24T21:59:52.354 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 2021-03-24T21:59:52.354 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 2021 2021-03-24T21:59:52.354 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: - 2021-03-24T21:59:52.354 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 03 2021-03-24T21:59:52.355 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: - 2021-03-24T21:59:52.355 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 24 2021-03-24T21:59:52.355 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: T 2021-03-24T21:59:52.355 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 21 2021-03-24T21:59:52.356 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: : 2021-03-24T21:59:52.356 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 59 2021-03-24T21:59:52.356 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: : 2021-03-24T21:59:52.356 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 52 2021-03-24T21:59:52.356 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: . 2021-03-24T21:59:52.357 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 066082 2021-03-24T21:59:52.357 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: +0000 2021-03-24T21:59:52.357 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 2021-03-24T21:59:52.357 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: mon.c 2021-03-24T21:59:52.358 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: ( 2021-03-24T21:59:52.358 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: mon 2021-03-24T21:59:52.358 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: . 2021-03-24T21:59:52.358 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 2 2021-03-24T21:59:52.358 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: ) 2021-03-24T21:59:52.359 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 66 2021-03-24T21:59:52.359 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: : 2021-03-24T21:59:52.359 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: audit 2021-03-24T21:59:52.359 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 2021-03-24T21:59:52.360 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: [INF] 2021-03-24T21:59:52.360 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 2021-03-24T21:59:52.360 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: from='client.? v1:172.21.15.125:0/3055724187' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:59:52.360 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 2021-03-24T21:59:52.360 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: audit 2021-03-24T21:59:52.361 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 2021-03-24T21:59:52.361 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 2021 2021-03-24T21:59:52.361 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: - 2021-03-24T21:59:52.361 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 03 2021-03-24T21:59:52.362 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: - 2021-03-24T21:59:52.362 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 24 2021-03-24T21:59:52.362 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: T 2021-03-24T21:59:52.362 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 21 2021-03-24T21:59:52.363 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: : 2021-03-24T21:59:52.363 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 59 2021-03-24T21:59:52.363 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: : 2021-03-24T21:59:52.363 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 52 2021-03-24T21:59:52.363 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: . 2021-03-24T21:59:52.364 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 066560 2021-03-24T21:59:52.364 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: +0000 2021-03-24T21:59:52.364 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 2021-03-24T21:59:52.365 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: mon.a 2021-03-24T21:59:52.365 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: ( 2021-03-24T21:59:52.365 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: mon 2021-03-24T21:59:52.365 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: . 2021-03-24T21:59:52.365 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 0 2021-03-24T21:59:52.366 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: ) 2021-03-24T21:59:52.366 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 1236 2021-03-24T21:59:52.366 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: : 2021-03-24T21:59:52.366 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: audit 2021-03-24T21:59:52.366 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 2021-03-24T21:59:52.367 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: [INF] 2021-03-24T21:59:52.367 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 2021-03-24T21:59:52.367 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: from='client.35216 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:59:52.367 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:52 smithi125 conmon[24927]: 2021-03-24T21:59:52.369 INFO:tasks.rados.rados.0.smithi125.stdout:1542: finishing copy_from to smithi12556470-7 2021-03-24T21:59:52.369 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 7 v 199 (ObjNum 401 snap 104 seq_num 401) dirty exists 2021-03-24T21:59:52.370 INFO:tasks.rados.rados.0.smithi125.stdout:1542: finishing copy_from racing read to smithi12556470-7 2021-03-24T21:59:52.370 INFO:tasks.rados.rados.0.smithi125.stdout:1542: done (2 left) 2021-03-24T21:59:52.370 INFO:tasks.rados.rados.0.smithi125.stdout:1545: done (1 left) 2021-03-24T21:59:52.370 INFO:tasks.rados.rados.0.smithi125.stdout:1546: write (excl) oid 6 current snap is 170 2021-03-24T21:59:52.371 INFO:tasks.rados.rados.0.smithi125.stdout:1546: seq_num 548 ranges {410236=413868,1234250=729187,2715800=423063} 2021-03-24T21:59:52.371 INFO:tasks.rados.rados.0.smithi125.stdout:1546: writing smithi12556470-6 from 410236 to 824104 tid 1 2021-03-24T21:59:52.371 INFO:tasks.rados.rados.0.smithi125.stdout:1546: writing smithi12556470-6 from 1234250 to 1963437 tid 2 2021-03-24T21:59:52.371 INFO:tasks.rados.rados.0.smithi125.stdout:1546: writing smithi12556470-6 from 2715800 to 3138863 tid 3 2021-03-24T21:59:52.371 INFO:tasks.rados.rados.0.smithi125.stdout:1547: snap_create 2021-03-24T21:59:52.372 INFO:tasks.rados.rados.0.smithi125.stdout:1543: finishing copy_from to smithi12556470-34 2021-03-24T21:59:52.372 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 34 v 539 (ObjNum 520 snap 154 seq_num 520) dirty exists 2021-03-24T21:59:52.372 INFO:tasks.rados.rados.0.smithi125.stdout:1543: finishing copy_from racing read to smithi12556470-34 2021-03-24T21:59:52.372 INFO:tasks.rados.rados.0.smithi125.stdout:1546: finishing write tid 1 to smithi12556470-6 2021-03-24T21:59:52.373 INFO:tasks.rados.rados.0.smithi125.stdout:1546: finishing write tid 2 to smithi12556470-6 2021-03-24T21:59:52.373 INFO:tasks.rados.rados.0.smithi125.stdout:1546: finishing write tid 3 to smithi12556470-6 2021-03-24T21:59:52.373 INFO:tasks.rados.rados.0.smithi125.stdout:1546: finishing write tid 4 to smithi12556470-6 2021-03-24T21:59:52.373 INFO:tasks.rados.rados.0.smithi125.stdout:1546: finishing write tid 5 to smithi12556470-6 2021-03-24T21:59:52.374 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 6 v 372 (ObjNum 548 snap 170 seq_num 548) dirty exists 2021-03-24T21:59:52.374 INFO:tasks.rados.rados.0.smithi125.stdout:1546: left oid 6 (ObjNum 548 snap 170 seq_num 548) 2021-03-24T21:59:52.374 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:52.449 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:52.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:52.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:52.492 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:52 smithi142 conmon[20125]: cluster 2021-03-24T21:59:50.842337+0000 mgr.y ( 2021-03-24T21:59:52.493 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:52 smithi142 conmon[20125]: mgr.14144) 481 : cluster [DBG] pgmap v793: 177 pgs: 18 remapped+peering, 15 peering, 24 active+clean+remapped, 2 active+recovering+undersized+remapped, 36 active+undersized, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering, 18 active+undersized+degraded, 1 active+recovering+degraded+remapped, 8 active+recovery_wait+degraded+remapped, 47 active+clean; 395 MiB data, 696 MiB used, 446 GiB / 447 GiB avail; 1.2 MiB/s wr, 6 op/s; 268/1527 objects degraded (17.551%); 222/1527 objects misplaced (14.538%); 125 B/s, 5 objects/s recovering 2021-03-24T21:59:52.493 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:52 smithi142 conmon[20125]: cluster 2021-03-24T21:59:51.223259+0000 mon.a (mon.0) 1235 : cluster [DBG] osdmap e421: 8 total, 7 up, 5 in 2021-03-24T21:59:52.493 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:52 smithi142 conmon[20125]: audit 2021-03-24T21:59:52.066082+0000 mon.c (mon.2) 66 : audit [INF] 2021-03-24T21:59:52.494 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:52 smithi142 conmon[20125]: from='client.? v1:172.21.15.125:0/3055724187' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:59:52.494 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:52 smithi142 conmon[20125]: audit 2021-03-24T21:59:52.494 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:52 smithi142 conmon[20125]: 2021-03-24T21:59:52.066560+0000 mon.a (mon.0) 1236 2021-03-24T21:59:52.494 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:52 smithi142 conmon[20125]: : audit [INF] from='client.35216 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T21:59:52.555 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:52.599 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:52.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:52.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:59:52.704 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:52.740 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2021-03-24T21:59:52.740 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2021-03-24T21:59:52.741 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2021-03-24T21:59:52.741 INFO:teuthology.orchestra.run.smithi142.stdout:} 2021-03-24T21:59:52.751 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:52.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:52.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:59:52.856 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:52.909 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:52.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:52.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:59:53.013 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:53.058 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:53.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:53.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:53.164 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:53.217 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:53.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:53.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:59:53.251 INFO:tasks.rados.rados.0.smithi125.stdout:1543: done (2 left) 2021-03-24T21:59:53.251 INFO:tasks.rados.rados.0.smithi125.stdout:1546: done (1 left) 2021-03-24T21:59:53.251 INFO:tasks.rados.rados.0.smithi125.stdout:1547: done (0 left) 2021-03-24T21:59:53.252 INFO:tasks.rados.rados.0.smithi125.stdout:1548: copy_from oid 46 from oid 1 current snap is 171 2021-03-24T21:59:53.252 INFO:tasks.rados.rados.0.smithi125.stdout:1549: delete oid 8 current snap is 171 2021-03-24T21:59:53.262 INFO:tasks.rados.rados.0.smithi125.stdout:1549: done (1 left) 2021-03-24T21:59:53.262 INFO:tasks.rados.rados.0.smithi125.stdout:1550: read oid 45 snap -1 2021-03-24T21:59:53.263 INFO:tasks.rados.rados.0.smithi125.stdout:1550: expect (ObjNum 466 snap 131 seq_num 466) 2021-03-24T21:59:53.263 INFO:tasks.rados.rados.0.smithi125.stdout:1551: write (excl) oid 17 current snap is 171 2021-03-24T21:59:53.263 INFO:tasks.rados.rados.0.smithi125.stdout:1551: seq_num 550 ranges {443850=564366,1634861=732856,2770646=473204} 2021-03-24T21:59:53.264 INFO:tasks.rados.rados.0.smithi125.stdout:1551: writing smithi12556470-17 from 443850 to 1008216 tid 1 2021-03-24T21:59:53.267 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5] out_osds: [6, 1, 7] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 6, 7] 2021-03-24T21:59:53.267 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-24T21:59:53.268 INFO:tasks.thrashosds.thrasher:Reviving osd 5 2021-03-24T21:59:53.268 INFO:tasks.cephadm.osd.5:Restarting osd.5 (starting--it wasn't running)... 2021-03-24T21:59:53.269 DEBUG:teuthology.orchestra.run.smithi142:osd.5> sudo journalctl -f -n 0 -u ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c@osd.5.service 2021-03-24T21:59:53.271 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl start ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c@osd.5 2021-03-24T21:59:53.273 INFO:tasks.rados.rados.0.smithi125.stdout:1551: writing smithi12556470-17 from 1634861 to 2367717 tid 2 2021-03-24T21:59:53.277 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:53 smithi142 conmon[20125]: cluster 2021-03-24T21:59:53.277 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:53 smithi142 conmon[20125]: 2021-03-24T21:59:53.277 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:53 smithi142 conmon[20125]: 2021 2021-03-24T21:59:53.278 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:53 smithi142 conmon[20125]: -03-24T21:59 2021-03-24T21:59:53.278 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:53 smithi142 conmon[20125]: :52.219904+0000 mon.a (mon.0) 1237 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T21:59:53.278 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:53 smithi142 conmon[20125]: audit 2021-03 2021-03-24T21:59:53.278 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:53 smithi142 conmon[20125]: -24T21:59:52.227833+0000 mon.a (mon.0) 1238 : audit [INF] 2021-03-24T21:59:53.279 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:53 smithi142 conmon[20125]: from='client.35216 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T21:59:53.279 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:53 smithi142 conmon[20125]: cluster 2021-03-24T21:59:53.279 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:53 smithi142 conmon[20125]: 2021-03 2021-03-24T21:59:53.279 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:53 smithi142 conmon[20125]: -24T21:59:52.228016 2021-03-24T21:59:53.280 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:53 smithi142 conmon[20125]: +0000 mon.a (mon.0) 1239 : 2021-03-24T21:59:53.280 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:53 smithi142 conmon[20125]: cluster [DBG] osdmap e422: 8 total, 7 up, 5 in 2021-03-24T21:59:53.281 INFO:tasks.rados.rados.0.smithi125.stdout:1551: writing smithi12556470-17 from 2770646 to 3243850 tid 3 2021-03-24T21:59:53.281 INFO:tasks.rados.rados.0.smithi125.stdout:1552: copy_from oid 14 from oid 43 current snap is 171 2021-03-24T21:59:53.282 INFO:journalctl@ceph.osd.5.smithi142.stdout:-- Logs begin at Wed 2021-03-03 23:51:07 UTC. -- 2021-03-24T21:59:53.292 INFO:tasks.rados.rados.0.smithi125.stdout:1550: expect (ObjNum 466 snap 131 seq_num 466) 2021-03-24T21:59:53.322 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:53.369 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:53.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:53.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:59:53.449 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[19936]: cluster 2021-03-24T21:59:52.219904+0000 2021-03-24T21:59:53.450 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[19936]: mon.a (mon.0) 1237 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T21:59:53.450 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[19936]: audit 2021-03-24T21:59:52.227833+0000 mon.a (mon.0) 1238 : audit 2021-03-24T21:59:53.451 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[19936]: [INF] from='client.35216 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T21:59:53.451 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T21:59:53.451 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[19936]: 21:59:52.228016+0000 mon.a (mon.0) 1239 : cluster [DBG] 2021-03-24T21:59:53.451 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[19936]: osdmap e422: 8 total, 7 up, 5 in 2021-03-24T21:59:53.454 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: cluster 2021-03-24T21:59:53.454 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: 2021-03-24T21:59:53.454 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: 2021 2021-03-24T21:59:53.455 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: - 2021-03-24T21:59:53.455 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: 03 2021-03-24T21:59:53.455 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: - 2021-03-24T21:59:53.455 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: 24 2021-03-24T21:59:53.456 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: T 2021-03-24T21:59:53.456 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: 21 2021-03-24T21:59:53.456 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: : 2021-03-24T21:59:53.456 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: 59 2021-03-24T21:59:53.456 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: : 2021-03-24T21:59:53.457 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: 52 2021-03-24T21:59:53.457 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: . 2021-03-24T21:59:53.457 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: 219904 2021-03-24T21:59:53.457 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: +0000 2021-03-24T21:59:53.458 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: 2021-03-24T21:59:53.458 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: mon.a 2021-03-24T21:59:53.458 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: ( 2021-03-24T21:59:53.458 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: mon 2021-03-24T21:59:53.458 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: . 2021-03-24T21:59:53.459 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: 0 2021-03-24T21:59:53.459 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: ) 2021-03-24T21:59:53.459 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: 1237 2021-03-24T21:59:53.459 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: : 2021-03-24T21:59:53.460 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: cluster 2021-03-24T21:59:53.460 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: 2021-03-24T21:59:53.460 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: [INF] 2021-03-24T21:59:53.460 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: 2021-03-24T21:59:53.460 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T21:59:53.461 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: 2021-03-24T21:59:53.461 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: audit 2021-03-24T21:59:53.461 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: 2021-03-24T21:59:53.461 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: 2021 2021-03-24T21:59:53.461 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: - 2021-03-24T21:59:53.462 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: 03 2021-03-24T21:59:53.462 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: - 2021-03-24T21:59:53.462 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: 24T21:59:52 2021-03-24T21:59:53.462 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: .227833+0000 mon.a (mon.0) 1238 : audit [INF] from='client.35216 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T21:59:53.463 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: cluster 2021-03-24T21:59:53.463 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: 2021-03-24T21:59:53.463 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: 2021 2021-03-24T21:59:53.463 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: - 2021-03-24T21:59:53.463 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: 03 2021-03-24T21:59:53.464 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: - 2021-03-24T21:59:53.464 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: 24 2021-03-24T21:59:53.464 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: T 2021-03-24T21:59:53.464 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: 21 2021-03-24T21:59:53.464 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: : 2021-03-24T21:59:53.465 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: 59 2021-03-24T21:59:53.465 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: : 2021-03-24T21:59:53.465 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: 52 2021-03-24T21:59:53.465 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: . 2021-03-24T21:59:53.466 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: 228016 2021-03-24T21:59:53.466 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: +0000 2021-03-24T21:59:53.466 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: 2021-03-24T21:59:53.466 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: mon.a 2021-03-24T21:59:53.467 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: ( 2021-03-24T21:59:53.467 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: mon 2021-03-24T21:59:53.467 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: . 2021-03-24T21:59:53.468 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: 0 2021-03-24T21:59:53.468 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: ) 2021-03-24T21:59:53.468 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: 1239 2021-03-24T21:59:53.468 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: : 2021-03-24T21:59:53.468 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: cluster 2021-03-24T21:59:53.469 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: 2021-03-24T21:59:53.469 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: [DBG] 2021-03-24T21:59:53.469 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: 2021-03-24T21:59:53.469 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: osdmap e422: 8 total, 7 up, 5 in 2021-03-24T21:59:53.469 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:53 smithi125 conmon[24927]: 2021-03-24T21:59:53.471 INFO:tasks.rados.rados.0.smithi125.stdout:1551: finishing write tid 1 to smithi12556470-17 2021-03-24T21:59:53.471 INFO:tasks.rados.rados.0.smithi125.stdout:1551: finishing write tid 2 to smithi12556470-17 2021-03-24T21:59:53.471 INFO:tasks.rados.rados.0.smithi125.stdout:1551: finishing write tid 3 to smithi12556470-17 2021-03-24T21:59:53.472 INFO:tasks.rados.rados.0.smithi125.stdout:1551: finishing write tid 4 to smithi12556470-17 2021-03-24T21:59:53.472 INFO:tasks.rados.rados.0.smithi125.stdout:1551: finishing write tid 5 to smithi12556470-17 2021-03-24T21:59:53.472 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 17 v 399 (ObjNum 550 snap 171 seq_num 550) dirty exists 2021-03-24T21:59:53.472 INFO:tasks.rados.rados.0.smithi125.stdout:1551: left oid 17 (ObjNum 550 snap 171 seq_num 550) 2021-03-24T21:59:53.472 INFO:tasks.rados.rados.0.smithi125.stdout:1548: finishing copy_from to smithi12556470-46 2021-03-24T21:59:53.473 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 46 v 476 (ObjNum 497 snap 145 seq_num 497) dirty exists 2021-03-24T21:59:53.473 INFO:tasks.rados.rados.0.smithi125.stdout:1548: finishing copy_from racing read to smithi12556470-46 2021-03-24T21:59:53.473 INFO:tasks.rados.rados.0.smithi125.stdout:1552: finishing copy_from to smithi12556470-14 2021-03-24T21:59:53.473 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 14 v 484 (ObjNum 399 snap 102 seq_num 399) dirty exists 2021-03-24T21:59:53.474 INFO:tasks.rados.rados.0.smithi125.stdout:1552: finishing copy_from racing read to smithi12556470-14 2021-03-24T21:59:53.474 INFO:tasks.rados.rados.0.smithi125.stdout:1548: done (3 left) 2021-03-24T21:59:53.474 INFO:tasks.rados.rados.0.smithi125.stdout:1550: done (2 left) 2021-03-24T21:59:53.474 INFO:tasks.rados.rados.0.smithi125.stdout:1551: done (1 left) 2021-03-24T21:59:53.474 INFO:tasks.rados.rados.0.smithi125.stdout:1552: done (0 left) 2021-03-24T21:59:53.475 INFO:tasks.rados.rados.0.smithi125.stdout:1553: copy_from oid 47 from oid 21 current snap is 171 2021-03-24T21:59:53.475 INFO:tasks.rados.rados.0.smithi125.stdout:1554: read oid 25 snap -1 2021-03-24T21:59:53.475 INFO:tasks.rados.rados.0.smithi125.stdout:1554: expect (ObjNum 477 snap 136 seq_num 477) 2021-03-24T21:59:53.475 INFO:tasks.rados.rados.0.smithi125.stdout:1555: snap_remove snap 160 2021-03-24T21:59:53.476 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:53.523 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:53.527 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 21:59:53 smithi142 systemd[1]: Starting Ceph osd.5 for 3cfa339a-8cea-11eb-90b2-001a4aab830c... 2021-03-24T21:59:53.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:53.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:59:53.629 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:53.676 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:53.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:53.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:53.783 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:53.786 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.4 dump_blocked_ops 2021-03-24T21:59:53.829 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:53.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:53.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:53.934 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:53.991 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:53.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:53.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:59:54.085 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 21:59:54 smithi142 podman[66255]: 2021-03-24 21:59:54.001083774 +0000 UTC m=+0.196482366 container create 91d89347ed66f412025247252966e53e6e3220cbda2990d9cb0d0f95056bb241 (image=quay.ceph.io/ceph-ci/ceph@sha256:e9db06906b8d0a24f3ef6835ac5aa4b8eea2950a2560bad78f83d4f101eca0db, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5-activate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20201204, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Dimitri Savineau , RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06, GIT_CLEAN=True, org.label-schema.name=CentOS Base Image, io.buildah.version=1.18.0, GIT_BRANCH=HEAD) 2021-03-24T21:59:54.096 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:54.147 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:54.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:54.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:59:54.195 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T21:59:54.253 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:54.266 INFO:tasks.rados.rados.0.smithi125.stdout:1554: expect (ObjNum 477 snap 136 seq_num 477) 2021-03-24T21:59:54.366 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 21:59:54 smithi142 podman[66255]: 2021-03-24 21:59:54.15099195 +0000 UTC m=+0.346390562 container init 91d89347ed66f412025247252966e53e6e3220cbda2990d9cb0d0f95056bb241 (image=quay.ceph.io/ceph-ci/ceph@sha256:e9db06906b8d0a24f3ef6835ac5aa4b8eea2950a2560bad78f83d4f101eca0db, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5-activate, io.buildah.version=1.18.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d, org.label-schema.vendor=CentOS, GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06, GIT_CLEAN=True, org.label-schema.build-date=20201204, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Base Image, maintainer=Dimitri Savineau , org.label-schema.license=GPLv2) 2021-03-24T21:59:54.367 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 21:59:54 smithi142 podman[66255]: 2021-03-24 21:59:54.184338276 +0000 UTC m=+0.379736919 container start 91d89347ed66f412025247252966e53e6e3220cbda2990d9cb0d0f95056bb241 (image=quay.ceph.io/ceph-ci/ceph@sha256:e9db06906b8d0a24f3ef6835ac5aa4b8eea2950a2560bad78f83d4f101eca0db, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5-activate, GIT_CLEAN=True, io.buildah.version=1.18.0, RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d, maintainer=Dimitri Savineau , org.label-schema.name=CentOS Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20201204, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2021-03-24T21:59:54.367 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 21:59:54 smithi142 podman[66255]: 2021-03-24 21:59:54.184613825 +0000 UTC m=+0.380012463 container attach 91d89347ed66f412025247252966e53e6e3220cbda2990d9cb0d0f95056bb241 (image=quay.ceph.io/ceph-ci/ceph@sha256:e9db06906b8d0a24f3ef6835ac5aa4b8eea2950a2560bad78f83d4f101eca0db, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5-activate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20201204, maintainer=Dimitri Savineau , org.label-schema.license=GPLv2, RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d, CEPH_POINT_RELEASE=, io.buildah.version=1.18.0, org.label-schema.vendor=CentOS, GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06, GIT_CLEAN=True) 2021-03-24T21:59:54.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:54 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T21:59:54.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:54 smithi142 conmon[20125]: 24T21:59:52.843814+0000 mgr.y (mgr.14144) 482 : cluster 2021-03-24T21:59:54.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:54 smithi142 conmon[20125]: [DBG] pgmap v796: 177 pgs: 18 remapped+peering, 32 active+clean+remapped, 2 active+recovering+undersized+remapped, 35 active+undersized, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering, 18 active+undersized+degraded, 1 active+recovering+degraded+remapped, 8 active+recovery_wait+degraded+remapped, 54 active+clean; 387 MiB data, 704 MiB used, 446 GiB / 447 GiB avail; 1.8 MiB/s wr, 8 op/s; 281/1515 objects degraded (18.548%); 297/1515 objects misplaced (19.604%); 11 MiB/s, 22 objects/s recovering 2021-03-24T21:59:54.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:54 smithi142 conmon[20125]: cluster 2021-03-24T21:59:54.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:54 smithi142 conmon[20125]: 2021-03-24T 2021-03-24T21:59:54.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:54 smithi142 conmon[20125]: 21:59:53 2021-03-24T21:59:54.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:54 smithi142 conmon[20125]: .235673+0000 2021-03-24T21:59:54.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:54 smithi142 conmon[20125]: mon.a (mon.0 2021-03-24T21:59:54.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:54 smithi142 conmon[20125]: ) 1240 : cluster 2021-03-24T21:59:54.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:54 smithi142 conmon[20125]: [WRN] Health check update: Reduced data availability: 6 pgs inactive, 7 pgs peering (PG_AVAILABILITY) 2021-03-24T21:59:54.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:54 smithi142 conmon[20125]: cluster 2021-03-24T21:59:54.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:54 smithi142 conmon[20125]: 2021-03-24 2021-03-24T21:59:54.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:54 smithi142 conmon[20125]: T21:59: 2021-03-24T21:59:54.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:54 smithi142 conmon[20125]: 53.247153+0000 2021-03-24T21:59:54.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:54 smithi142 conmon[20125]: mon.a (mon. 2021-03-24T21:59:54.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:54 smithi142 conmon[20125]: 0) 1241 : 2021-03-24T21:59:54.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:54 smithi142 conmon[20125]: cluster [DBG] osdmap e423: 8 total, 7 up, 5 in 2021-03-24T21:59:54.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:54 smithi142 conmon[20125]: 2021-03-24T21:59:54.383 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:54.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:54.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:54.422 INFO:tasks.rados.rados.0.smithi125.stdout:1553: finishing copy_from to smithi12556470-47 2021-03-24T21:59:54.423 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 47 v 334 (ObjNum 471 snap 133 seq_num 471) dirty exists 2021-03-24T21:59:54.423 INFO:tasks.rados.rados.0.smithi125.stdout:1553: finishing copy_from racing read to smithi12556470-47 2021-03-24T21:59:54.423 INFO:tasks.rados.rados.0.smithi125.stdout:1553: done (2 left) 2021-03-24T21:59:54.424 INFO:tasks.rados.rados.0.smithi125.stdout:1554: done (1 left) 2021-03-24T21:59:54.424 INFO:tasks.rados.rados.0.smithi125.stdout:1555: done (0 left) 2021-03-24T21:59:54.424 INFO:tasks.rados.rados.0.smithi125.stdout:1556: copy_from oid 3 from oid 25 current snap is 171 2021-03-24T21:59:54.424 INFO:tasks.rados.rados.0.smithi125.stdout:1557: read oid 36 snap -1 2021-03-24T21:59:54.425 INFO:tasks.rados.rados.0.smithi125.stdout:1557: expect (ObjNum 460 snap 129 seq_num 460) 2021-03-24T21:59:54.425 INFO:tasks.rados.rados.0.smithi125.stdout:1558: delete oid 50 current snap is 171 2021-03-24T21:59:54.429 INFO:tasks.rados.rados.0.smithi125.stdout:1557: expect (ObjNum 460 snap 129 seq_num 460) 2021-03-24T21:59:54.496 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:54.551 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:54.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:54.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:59:54.575 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: cluster 2021-03-24T21:59:54.576 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: 2021-03-24T21:59:54.576 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: 2021 2021-03-24T21:59:54.577 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: - 2021-03-24T21:59:54.577 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: 03 2021-03-24T21:59:54.577 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: - 2021-03-24T21:59:54.577 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: 24 2021-03-24T21:59:54.578 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: T 2021-03-24T21:59:54.578 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: 21 2021-03-24T21:59:54.578 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: : 2021-03-24T21:59:54.579 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: 59 2021-03-24T21:59:54.579 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: : 2021-03-24T21:59:54.579 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: 52 2021-03-24T21:59:54.580 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: . 2021-03-24T21:59:54.580 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: 843814 2021-03-24T21:59:54.580 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: +0000 2021-03-24T21:59:54.581 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: 2021-03-24T21:59:54.581 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: mgr.y 2021-03-24T21:59:54.581 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: ( 2021-03-24T21:59:54.582 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: mgr 2021-03-24T21:59:54.582 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: . 2021-03-24T21:59:54.582 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: 14144 2021-03-24T21:59:54.583 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: ) 2021-03-24T21:59:54.583 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: 482 2021-03-24T21:59:54.583 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: : 2021-03-24T21:59:54.583 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: cluster 2021-03-24T21:59:54.584 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: 2021-03-24T21:59:54.584 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: [DBG] 2021-03-24T21:59:54.584 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: 2021-03-24T21:59:54.585 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: pgmap v796: 177 pgs: 18 remapped+peering, 32 active+clean+remapped, 2 active+recovering+undersized+remapped, 35 active+undersized, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering, 18 active+undersized+degraded, 1 active+recovering+degraded+remapped, 8 active+recovery_wait+degraded+remapped, 54 active+clean; 387 MiB data, 704 MiB used, 446 GiB / 447 GiB avail; 1.8 MiB/s wr, 8 op/s; 281/1515 objects degraded (18.548%); 297/1515 objects misplaced (19.604%); 11 MiB/s, 22 objects/s recovering 2021-03-24T21:59:54.585 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: 2021-03-24T21:59:54.585 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: cluster 2021-03-24T21:59:54.586 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: 2021-03-24T21:59:54.586 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: 2021 2021-03-24T21:59:54.587 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: - 2021-03-24T21:59:54.587 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: 03 2021-03-24T21:59:54.588 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: - 2021-03-24T21:59:54.588 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: 24 2021-03-24T21:59:54.588 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: T 2021-03-24T21:59:54.589 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: 21 2021-03-24T21:59:54.589 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: : 2021-03-24T21:59:54.590 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: 59 2021-03-24T21:59:54.590 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: : 2021-03-24T21:59:54.591 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: 53 2021-03-24T21:59:54.591 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: . 2021-03-24T21:59:54.591 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: 235673 2021-03-24T21:59:54.592 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: +0000 2021-03-24T21:59:54.592 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: 2021-03-24T21:59:54.593 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: mon.a 2021-03-24T21:59:54.593 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: ( 2021-03-24T21:59:54.594 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: mon 2021-03-24T21:59:54.594 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: . 2021-03-24T21:59:54.595 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: 0 2021-03-24T21:59:54.595 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: ) 2021-03-24T21:59:54.596 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: 1240 2021-03-24T21:59:54.596 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: : 2021-03-24T21:59:54.596 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: cluster 2021-03-24T21:59:54.597 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: 2021-03-24T21:59:54.597 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: [WRN] Health check update: Reduced data availability: 6 pgs inactive, 7 pgs peering (PG_AVAILABILITY) 2021-03-24T21:59:54.598 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: cluster 2021-03-24T21:59:53.247153+0000 mon.a (mon. 2021-03-24T21:59:54.598 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[24927]: 0) 1241 : cluster [DBG] osdmap e423: 8 total, 7 up, 5 in 2021-03-24T21:59:54.598 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[19936]: cluster 2021-03-24T21:59:52.843814+0000 mgr.y ( 2021-03-24T21:59:54.599 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[19936]: mgr.14144) 482 : cluster [DBG] pgmap v796: 177 pgs: 18 remapped+peering, 32 active+clean+remapped, 2 active+recovering+undersized+remapped, 35 active+undersized, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering, 18 active+undersized+degraded, 1 active+recovering+degraded+remapped, 8 active+recovery_wait+degraded+remapped, 54 active+clean; 387 MiB data, 704 MiB used, 446 GiB / 447 GiB avail; 1.8 MiB/s wr, 8 op/s; 281/1515 objects degraded (18.548%); 297/1515 objects misplaced (19.604%); 11 MiB/s, 22 objects/s recovering 2021-03-24T21:59:54.599 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[19936]: cluster 2021-03-24T21:59:53.235673+0000 mon.a (mon.0) 1240 : cluster [WRN] Health check update: Reduced data availability: 6 pgs inactive, 7 pgs peering (PG_AVAILABILITY) 2021-03-24T21:59:54.599 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[19936]: cluster 2021-03-24T21: 2021-03-24T21:59:54.599 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:54 smithi125 conmon[19936]: 59:53.247153+0000 mon.a (mon.0) 1241 : cluster [DBG] osdmap e423: 8 total, 7 up, 5 in 2021-03-24T21:59:54.655 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:54.705 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:54.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:54.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:59:54.721 INFO:tasks.rados.rados.0.smithi125.stdout:1556: finishing copy_from to smithi12556470-3 2021-03-24T21:59:54.721 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 3 v 296 (ObjNum 477 snap 136 seq_num 477) dirty exists 2021-03-24T21:59:54.722 INFO:tasks.rados.rados.0.smithi125.stdout:1556: finishing copy_from racing read to smithi12556470-3 2021-03-24T21:59:54.722 INFO:tasks.rados.rados.0.smithi125.stdout:1556: done (2 left) 2021-03-24T21:59:54.722 INFO:tasks.rados.rados.0.smithi125.stdout:1557: done (1 left) 2021-03-24T21:59:54.722 INFO:tasks.rados.rados.0.smithi125.stdout:1558: done (0 left) 2021-03-24T21:59:54.723 INFO:tasks.rados.rados.0.smithi125.stdout:1559: snap_create 2021-03-24T21:59:54.803 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 21:59:54 smithi142 bash[66255]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2021-03-24T21:59:54.803 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 21:59:54 smithi142 bash[66255]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-5 --no-mon-config 2021-03-24T21:59:54.804 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 21:59:54 smithi142 bash[66255]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5/block 2021-03-24T21:59:54.804 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 21:59:54 smithi142 bash[66255]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-5/block 2021-03-24T21:59:54.804 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 21:59:54 smithi142 bash[66255]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2021-03-24T21:59:54.804 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 21:59:54 smithi142 bash[66255]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2021-03-24T21:59:54.805 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 21:59:54 smithi142 bash[66255]: --> ceph-volume lvm activate successful for osd ID: 5 2021-03-24T21:59:54.809 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:54.865 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:54.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:54.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:54.972 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:55.029 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:55.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:55.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:59:55.105 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T21:59:55.127 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T21:59:55.136 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:55.159 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T21:59:55.189 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:55.190 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T21:59:55.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:55.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:59:55.227 INFO:tasks.rados.rados.0.smithi125.stdout:1559: done (0 left) 2021-03-24T21:59:55.228 INFO:tasks.rados.rados.0.smithi125.stdout:1560: read oid 7 snap 171 2021-03-24T21:59:55.228 INFO:tasks.rados.rados.0.smithi125.stdout:1560: expect (ObjNum 401 snap 104 seq_num 401) 2021-03-24T21:59:55.228 INFO:tasks.rados.rados.0.smithi125.stdout:1561: rollback oid 1 current snap is 172 2021-03-24T21:59:55.228 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 1 to 142 2021-03-24T21:59:55.229 INFO:tasks.rados.rados.0.smithi125.stdout:1562: read oid 23 snap -1 2021-03-24T21:59:55.229 INFO:tasks.rados.rados.0.smithi125.stdout:1562: expect (ObjNum 427 snap 113 seq_num 427) 2021-03-24T21:59:55.229 INFO:tasks.rados.rados.0.smithi125.stdout:1563: copy_from oid 45 from oid 43 current snap is 172 2021-03-24T21:59:55.229 INFO:tasks.rados.rados.0.smithi125.stdout:1564: read oid 42 snap -1 2021-03-24T21:59:55.230 INFO:tasks.rados.rados.0.smithi125.stdout:1564: expect (ObjNum 444 snap 120 seq_num 444) 2021-03-24T21:59:55.230 INFO:tasks.rados.rados.0.smithi125.stdout:1565: delete oid 3 current snap is 172 2021-03-24T21:59:55.239 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T21:59:55.241 INFO:tasks.rados.rados.0.smithi125.stdout:1561: finishing rollback tid 0 to smithi12556470-1 2021-03-24T21:59:55.244 INFO:tasks.rados.rados.0.smithi125.stdout:1561: finishing rollback tid 1 to smithi12556470-1 2021-03-24T21:59:55.244 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 1 v 479 (ObjNum 1953066355 snap 842099048 seq_num 875967797) dirty dne 2021-03-24T21:59:55.245 INFO:tasks.rados.rados.0.smithi125.stdout:1561: done (5 left) 2021-03-24T21:59:55.245 INFO:tasks.rados.rados.0.smithi125.stdout:1565: done (4 left) 2021-03-24T21:59:55.245 INFO:tasks.rados.rados.0.smithi125.stdout:1566: snap_remove snap 169 2021-03-24T21:59:55.296 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:55.301 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T21:59:55.320 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 21:59:54 smithi142 podman[66255]: 2021-03-24 21:59:54.836671695 +0000 UTC m=+1.032070346 container died 91d89347ed66f412025247252966e53e6e3220cbda2990d9cb0d0f95056bb241 (image=quay.ceph.io/ceph-ci/ceph@sha256:e9db06906b8d0a24f3ef6835ac5aa4b8eea2950a2560bad78f83d4f101eca0db, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5-activate) 2021-03-24T21:59:55.351 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T21:59:55.352 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-24T21:59:55.352 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-24T21:59:55.352 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-24T21:59:55.352 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-24T21:59:55.353 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-24T21:59:55.353 INFO:teuthology.orchestra.run.smithi125.stderr:Error ENXIO: problem getting command descriptions from osd.5 2021-03-24T21:59:55.353 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-24T21:59:55.353 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-24T21:59:55.421 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:55.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:55.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:59:55.507 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd set noscrub 2021-03-24T21:59:55.526 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:55.530 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[24927]: cluster 2021-03-24T21: 2021-03-24T21:59:55.531 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[24927]: 59:54.259576+0000 mon.a (mon.0) 1242 : cluster [DBG] osdmap e424: 8 total, 7 up, 5 in 2021-03-24T21:59:55.531 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[24927]: audit 2021 2021-03-24T21:59:55.532 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[24927]: -03-24T21:59:55.080723+0000 2021-03-24T21:59:55.532 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[24927]: mon.a (mon.0) 1243 : audit [DBG] 2021-03-24T21:59:55.532 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[24927]: from='client.? v1:172.21.15.125:0/1179148725' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:59:55.532 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[24927]: cluster 2021 2021-03-24T21:59:55.532 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[24927]: -03-24T21:59:55.214991+0000 2021-03-24T21:59:55.533 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[24927]: mon.a (mon.0) 1244 : cluster [WRN] 2021-03-24T21:59:55.533 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[24927]: Health check update: Degraded data redundancy: 281/1515 objects degraded (18.548%), 33 pgs degraded, 47 pgs undersized (PG_DEGRADED) 2021-03-24T21:59:55.533 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[24927]: cluster 2021- 2021-03-24T21:59:55.533 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[24927]: 03-24T21:59:55.222921+0000 2021-03-24T21:59:55.534 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[24927]: mon.a (mon.0) 1245 : cluster [DBG] 2021-03-24T21:59:55.534 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[24927]: osdmap e425: 8 total, 7 up, 5 in 2021-03-24T21:59:55.534 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: cluster 2021-03-24T21:59:55.535 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 2021-03-24T21:59:55.535 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 2021 2021-03-24T21:59:55.535 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: - 2021-03-24T21:59:55.535 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 03 2021-03-24T21:59:55.536 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: - 2021-03-24T21:59:55.536 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 24 2021-03-24T21:59:55.536 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: T 2021-03-24T21:59:55.536 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 21 2021-03-24T21:59:55.537 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: : 2021-03-24T21:59:55.537 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 59 2021-03-24T21:59:55.537 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: : 2021-03-24T21:59:55.537 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 54 2021-03-24T21:59:55.537 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: . 2021-03-24T21:59:55.538 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 259576 2021-03-24T21:59:55.538 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: +0000 2021-03-24T21:59:55.538 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 2021-03-24T21:59:55.538 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: mon.a 2021-03-24T21:59:55.539 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: ( 2021-03-24T21:59:55.539 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: mon 2021-03-24T21:59:55.539 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: . 2021-03-24T21:59:55.539 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 0 2021-03-24T21:59:55.540 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: ) 2021-03-24T21:59:55.540 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 1242 2021-03-24T21:59:55.540 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: : 2021-03-24T21:59:55.540 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: cluster 2021-03-24T21:59:55.541 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 2021-03-24T21:59:55.541 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: [DBG] 2021-03-24T21:59:55.541 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 2021-03-24T21:59:55.542 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: osdmap e424: 8 total, 7 up, 5 in 2021-03-24T21:59:55.542 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 2021-03-24T21:59:55.542 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: audit 2021-03-24T21:59:55.543 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 2021-03-24T21:59:55.543 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 2021 2021-03-24T21:59:55.543 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: - 2021-03-24T21:59:55.544 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 03 2021-03-24T21:59:55.544 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: - 2021-03-24T21:59:55.544 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 24 2021-03-24T21:59:55.544 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: T 2021-03-24T21:59:55.545 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 21 2021-03-24T21:59:55.545 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: : 2021-03-24T21:59:55.545 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 59 2021-03-24T21:59:55.545 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: : 2021-03-24T21:59:55.546 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 55 2021-03-24T21:59:55.546 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: . 2021-03-24T21:59:55.546 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 080723 2021-03-24T21:59:55.546 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: +0000 2021-03-24T21:59:55.547 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 2021-03-24T21:59:55.547 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: mon.a 2021-03-24T21:59:55.547 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: ( 2021-03-24T21:59:55.547 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: mon 2021-03-24T21:59:55.548 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: . 2021-03-24T21:59:55.548 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 0 2021-03-24T21:59:55.548 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: ) 2021-03-24T21:59:55.548 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 1243 2021-03-24T21:59:55.548 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: : 2021-03-24T21:59:55.549 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: audit 2021-03-24T21:59:55.549 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 2021-03-24T21:59:55.549 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: [DBG] 2021-03-24T21:59:55.550 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 2021-03-24T21:59:55.550 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: from='client.? v1:172.21.15.125:0/1179148725' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:59:55.550 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 2021-03-24T21:59:55.550 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: cluster 2021-03-24T21:59:55.551 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 2021-03-24T21:59:55.551 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 2021 2021-03-24T21:59:55.551 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: - 2021-03-24T21:59:55.552 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 03 2021-03-24T21:59:55.552 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: - 2021-03-24T21:59:55.553 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 24 2021-03-24T21:59:55.554 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: T 2021-03-24T21:59:55.554 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 21 2021-03-24T21:59:55.554 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: : 2021-03-24T21:59:55.554 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 59 2021-03-24T21:59:55.555 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: : 2021-03-24T21:59:55.555 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 55 2021-03-24T21:59:55.555 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: . 2021-03-24T21:59:55.556 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 214991 2021-03-24T21:59:55.556 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: +0000 2021-03-24T21:59:55.556 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 2021-03-24T21:59:55.556 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: mon.a 2021-03-24T21:59:55.556 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: ( 2021-03-24T21:59:55.557 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: mon 2021-03-24T21:59:55.557 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: . 2021-03-24T21:59:55.557 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 0 2021-03-24T21:59:55.557 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: ) 2021-03-24T21:59:55.558 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 1244 2021-03-24T21:59:55.558 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: : 2021-03-24T21:59:55.558 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: cluster 2021-03-24T21:59:55.558 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 2021-03-24T21:59:55.559 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: [WRN] 2021-03-24T21:59:55.559 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 2021-03-24T21:59:55.559 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: Health check update: Degraded data redundancy: 281/1515 objects degraded (18.548%), 33 pgs degraded, 47 pgs undersized (PG_DEGRADED) 2021-03-24T21:59:55.559 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 2021-03-24T21:59:55.559 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: cluster 2021-03-24T21:59:55.560 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 2021-03-24T21:59:55.560 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 2021 2021-03-24T21:59:55.560 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: - 2021-03-24T21:59:55.560 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 03 2021-03-24T21:59:55.561 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: - 2021-03-24T21:59:55.561 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 24 2021-03-24T21:59:55.561 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: T 2021-03-24T21:59:55.561 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 21 2021-03-24T21:59:55.561 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: : 2021-03-24T21:59:55.562 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 59 2021-03-24T21:59:55.562 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: : 2021-03-24T21:59:55.562 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 55 2021-03-24T21:59:55.563 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: . 2021-03-24T21:59:55.563 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 222921 2021-03-24T21:59:55.563 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: +0000 2021-03-24T21:59:55.563 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 2021-03-24T21:59:55.564 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: mon.a 2021-03-24T21:59:55.564 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: ( 2021-03-24T21:59:55.564 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: mon 2021-03-24T21:59:55.564 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: . 2021-03-24T21:59:55.564 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 0 2021-03-24T21:59:55.565 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: ) 2021-03-24T21:59:55.565 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 1245 2021-03-24T21:59:55.565 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: : 2021-03-24T21:59:55.565 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: cluster 2021-03-24T21:59:55.565 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 2021-03-24T21:59:55.566 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: [DBG] 2021-03-24T21:59:55.566 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 2021-03-24T21:59:55.566 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: osdmap e425: 8 total, 7 up, 5 in 2021-03-24T21:59:55.566 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:55 smithi125 conmon[19936]: 2021-03-24T21:59:55.584 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:55.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:55.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:59:55.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:55 smithi142 conmon[20125]: cluster 2021-03-24T21:59:54.259576+0000 mon.a 2021-03-24T21:59:55.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:55 smithi142 conmon[20125]: (mon.0) 1242 : cluster [DBG] osdmap e424: 8 total, 7 up, 5 in 2021-03-24T21:59:55.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:55 smithi142 conmon[20125]: audit 2021-03-24T21:59:55.080723+0000 mon.a (mon.0) 1243 2021-03-24T21:59:55.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:55 smithi142 conmon[20125]: : audit [DBG] from='client.? v1:172.21.15.125:0/1179148725' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T21:59:55.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:55 smithi142 conmon[20125]: cluster 2021-03-24T21:59:55.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:55 smithi142 conmon[20125]: 2021-03-24T21:59:55.214991+0000 mon.a (mon 2021-03-24T21:59:55.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:55 smithi142 conmon[20125]: .0) 1244 : cluster [WRN] Health check update: Degraded data redundancy: 281/1515 objects degraded (18.548%), 33 pgs degraded, 47 pgs undersized (PG_DEGRADED) 2021-03-24T21:59:55.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:55 smithi142 conmon[20125]: cluster 2021- 2021-03-24T21:59:55.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:55 smithi142 conmon[20125]: 03-24T21:59:55.222921+0000 2021-03-24T21:59:55.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:55 smithi142 conmon[20125]: mon.a (mon.0) 1245 : cluster [DBG] 2021-03-24T21:59:55.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:55 smithi142 conmon[20125]: osdmap e425: 8 total, 7 up, 5 in 2021-03-24T21:59:55.688 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:55.746 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:55.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:55.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:59:55.851 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:55.901 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:55.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:55.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:59:56.007 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:56.056 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:56.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:56.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:56.161 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:56.207 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:56.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:56.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:59:56.241 INFO:tasks.rados.rados.0.smithi125.stdout:1566: done (4 left) 2021-03-24T21:59:56.242 INFO:tasks.rados.rados.0.smithi125.stdout:1567: write oid 34 current snap is 172 2021-03-24T21:59:56.242 INFO:tasks.rados.rados.0.smithi125.stdout:1567: seq_num 553 ranges {494271=790113,1693954=480183,2620209=512129} 2021-03-24T21:59:56.249 INFO:tasks.rados.rados.0.smithi125.stdout:1567: writing smithi12556470-34 from 494271 to 1284384 tid 1 2021-03-24T21:59:56.255 INFO:tasks.rados.rados.0.smithi125.stdout:1567: writing smithi12556470-34 from 1693954 to 2174137 tid 2 2021-03-24T21:59:56.262 INFO:tasks.rados.rados.0.smithi125.stdout:1567: writing smithi12556470-34 from 2620209 to 3132338 tid 3 2021-03-24T21:59:56.263 INFO:tasks.rados.rados.0.smithi125.stdout:1568: read oid 40 snap -1 2021-03-24T21:59:56.263 INFO:tasks.rados.rados.0.smithi125.stdout:1568: expect deleted 2021-03-24T21:59:56.263 INFO:tasks.rados.rados.0.smithi125.stdout:1569: copy_from oid 36 from oid 31 current snap is 172 2021-03-24T21:59:56.263 INFO:tasks.rados.rados.0.smithi125.stdout:1560: expect (ObjNum 401 snap 104 seq_num 401) 2021-03-24T21:59:56.312 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:56.422 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:56.428 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:56 smithi125 conmon[19936]: cluster 2021-03-24T21:59:54.845043+0000 mgr.y ( 2021-03-24T21:59:56.428 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:56 smithi125 conmon[19936]: mgr.14144) 483 : cluster [DBG] pgmap v799: 177 pgs: 53 active+clean+remapped, 2 active+recovering+undersized+remapped, 35 active+undersized, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 18 active+undersized+degraded, 1 active+recovering+degraded+remapped, 5 active+recovery_wait+degraded+remapped, 57 active+clean; 402 MiB data, 779 MiB used, 446 GiB / 447 GiB avail; 8.4 MiB/s rd, 4.5 MiB/s wr, 16 op/s; 263/1536 objects degraded (17.122%); 321/1536 objects misplaced (20.898%); 15 MiB/s, 49 objects/s recovering 2021-03-24T21:59:56.429 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:56 smithi125 conmon[19936]: cluster 2021-03-24T21:59:55.252620+0000 mon.a (mon.0) 1246 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 7 pgs peering) 2021-03-24T21:59:56.429 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:56 smithi125 conmon[19936]: cluster 2021-03-24T21:59:55.500216+0000 osd.2 (osd.2) 74 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:59:56.429 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:56 smithi125 conmon[19936]: cluster 2021-03-24T21:59:55.500610+0000 osd.2 (osd. 2021-03-24T21:59:56.429 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:56 smithi125 conmon[19936]: 2) 75 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:59:56.430 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:56 smithi125 conmon[19936]: cluster 2021-03-24T21:59: 2021-03-24T21:59:56.430 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:56 smithi125 conmon[19936]: 56.238679+0000 mon.a (mon.0) 1247 : cluster 2021-03-24T21:59:56.430 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:56 smithi125 conmon[19936]: [DBG] osdmap e426: 8 total, 7 up, 5 in 2021-03-24T21:59:56.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:56.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:59:56.432 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:56 smithi125 conmon[24927]: cluster 2021-03-24T21:59:54.845043 2021-03-24T21:59:56.432 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:56 smithi125 conmon[24927]: +0000 mgr.y (mgr.14144) 483 : cluster [DBG] pgmap v799: 177 pgs: 53 active+clean+remapped, 2 active+recovering+undersized+remapped, 35 active+undersized, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 18 active+undersized+degraded, 1 active+recovering+degraded+remapped, 5 active+recovery_wait+degraded+remapped, 57 active+clean; 402 MiB data, 779 MiB used, 446 GiB / 447 GiB avail; 8.4 MiB/s rd, 4.5 MiB/s wr, 16 op/s; 263/1536 objects degraded (17.122%); 321/1536 objects misplaced (20.898%); 15 MiB/s, 49 objects/s recovering 2021-03-24T21:59:56.432 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:56 smithi125 conmon[24927]: cluster 2021-03-24T21:59:55.252620+0000 mon.a (mon.0) 1246 : cluster [INF] 2021-03-24T21:59:56.433 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:56 smithi125 conmon[24927]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 7 pgs peering) 2021-03-24T21:59:56.433 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:56 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T21:59:56.433 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:56 smithi125 conmon[24927]: 24T21:59:55.500216+0000 osd.2 (osd.2) 2021-03-24T21:59:56.433 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:56 smithi125 conmon[24927]: 74 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:59:56.434 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:56 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T21:59:56.434 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:56 smithi125 conmon[24927]: -24T21:59:55.500610+0000 osd.2 2021-03-24T21:59:56.434 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:56 smithi125 conmon[24927]: (osd.2) 75 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:59:56.434 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:56 smithi125 conmon[24927]: 2021-03-24T21:59:56.435 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:56 smithi125 conmon[24927]: cluster 2021-03-24T21:59 2021-03-24T21:59:56.435 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:56 smithi125 conmon[24927]: :56.238679+0000 mon.a (mon.0) 2021-03-24T21:59:56.435 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:56 smithi125 conmon[24927]: 1247 : cluster [DBG] osdmap e426: 8 total, 7 up, 5 in 2021-03-24T21:59:56.472 INFO:tasks.rados.rados.0.smithi125.stdout:1564: expect (ObjNum 444 snap 120 seq_num 444) 2021-03-24T21:59:56.532 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:56.550 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 21:59:56 smithi142 podman[66255]: 2021-03-24 21:59:56.292324862 +0000 UTC m=+2.487723411 container remove 91d89347ed66f412025247252966e53e6e3220cbda2990d9cb0d0f95056bb241 (image=quay.ceph.io/ceph-ci/ceph@sha256:e9db06906b8d0a24f3ef6835ac5aa4b8eea2950a2560bad78f83d4f101eca0db, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5-activate, GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06, maintainer=Dimitri Savineau , org.label-schema.vendor=CentOS, io.buildah.version=1.18.0, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20201204, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Base Image) 2021-03-24T21:59:56.550 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:56 smithi142 conmon[20125]: cluster 2021-03-24T21:59:56.551 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:56 smithi142 conmon[20125]: 2021- 2021-03-24T21:59:56.551 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:56 smithi142 conmon[20125]: 03-24T21:59:54.845043+0000 mgr.y (mgr.14144) 483 : cluster [DBG] pgmap v799: 177 pgs: 53 active+clean+remapped, 2 active+recovering+undersized+remapped, 35 active+undersized, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 18 active+undersized+degraded, 1 active+recovering+degraded+remapped, 5 active+recovery_wait+degraded+remapped, 57 active+clean; 402 MiB data, 779 MiB used, 446 GiB / 447 GiB avail; 8.4 MiB/s rd, 4.5 MiB/s wr, 16 op/s; 263/1536 objects degraded (17.122%); 321/1536 objects misplaced (20.898%); 15 MiB/s, 49 objects/s recovering 2021-03-24T21:59:56.551 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:56 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T21:59:56.551 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:56 smithi142 conmon[20125]: 21:59:55.252620+0000 mon.a (mon.0) 1246 : 2021-03-24T21:59:56.552 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:56 smithi142 conmon[20125]: cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 7 pgs peering) 2021-03-24T21:59:56.552 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:56 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:59:56.553 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:56 smithi142 conmon[20125]: T21:59:55.500216+0000 osd.2 (osd.2 2021-03-24T21:59:56.553 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:56 smithi142 conmon[20125]: ) 74 : cluster [DBG] purged_snaps scrub starts 2021-03-24T21:59:56.553 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:56 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T21:59:56.553 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:56 smithi142 conmon[20125]: T21:59:55.500610+0000 osd.2 (osd.2 2021-03-24T21:59:56.554 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:56 smithi142 conmon[20125]: ) 75 : cluster [DBG] purged_snaps scrub ok 2021-03-24T21:59:56.554 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:56 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T21:59:56.554 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:56 smithi142 conmon[20125]: -24T21:59:56.238679+0000 mon.a (mon 2021-03-24T21:59:56.554 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:56 smithi142 conmon[20125]: .0) 1247 : cluster [DBG] osdmap e426: 8 total, 7 up, 5 in 2021-03-24T21:59:56.578 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-03-24T21:59:56.579 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-03-24T21:59:56.582 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:59:56.586 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:56.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:56.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:59:56.595 INFO:tasks.rados.rados.0.smithi125.stdout:1562: expect (ObjNum 427 snap 113 seq_num 427) 2021-03-24T21:59:56.692 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:56.750 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:56.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:56.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:56.802 INFO:tasks.rados.rados.0.smithi125.stdout:1563: finishing copy_from to smithi12556470-45 2021-03-24T21:59:56.802 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 45 v 484 (ObjNum 399 snap 102 seq_num 399) dirty exists 2021-03-24T21:59:56.802 INFO:tasks.rados.rados.0.smithi125.stdout:1563: finishing copy_from racing read to smithi12556470-45 2021-03-24T21:59:56.803 INFO:tasks.rados.rados.0.smithi125.stdout:1560: done (6 left) 2021-03-24T21:59:56.803 INFO:tasks.rados.rados.0.smithi125.stdout:1562: done (5 left) 2021-03-24T21:59:56.803 INFO:tasks.rados.rados.0.smithi125.stdout:1563: done (4 left) 2021-03-24T21:59:56.803 INFO:tasks.rados.rados.0.smithi125.stdout:1564: done (3 left) 2021-03-24T21:59:56.804 INFO:tasks.rados.rados.0.smithi125.stdout:1570: read oid 41 snap -1 2021-03-24T21:59:56.804 INFO:tasks.rados.rados.0.smithi125.stdout:1570: expect (ObjNum 510 snap 148 seq_num 510) 2021-03-24T21:59:56.804 INFO:tasks.rados.rados.0.smithi125.stdout:1571: read oid 10 snap -1 2021-03-24T21:59:56.804 INFO:tasks.rados.rados.0.smithi125.stdout:1571: expect (ObjNum 393 snap 100 seq_num 393) 2021-03-24T21:59:56.804 INFO:tasks.rados.rados.0.smithi125.stdout:1572: copy_from oid 27 from oid 16 current snap is 172 2021-03-24T21:59:56.805 INFO:tasks.rados.rados.0.smithi125.stdout:1573: delete oid 7 current snap is 172 2021-03-24T21:59:56.805 INFO:tasks.rados.rados.0.smithi125.stdout:1567: finishing write tid 1 to smithi12556470-34 2021-03-24T21:59:56.805 INFO:tasks.rados.rados.0.smithi125.stdout:1567: finishing write tid 2 to smithi12556470-34 2021-03-24T21:59:56.805 INFO:tasks.rados.rados.0.smithi125.stdout:1567: finishing write tid 3 to smithi12556470-34 2021-03-24T21:59:56.806 INFO:tasks.rados.rados.0.smithi125.stdout:1567: finishing write tid 4 to smithi12556470-34 2021-03-24T21:59:56.806 INFO:tasks.rados.rados.0.smithi125.stdout:1567: finishing write tid 5 to smithi12556470-34 2021-03-24T21:59:56.806 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 34 v 543 (ObjNum 553 snap 172 seq_num 553) dirty exists 2021-03-24T21:59:56.806 INFO:tasks.rados.rados.0.smithi125.stdout:1567: left oid 34 (ObjNum 553 snap 172 seq_num 553) 2021-03-24T21:59:56.806 INFO:tasks.rados.rados.0.smithi125.stdout:1569: finishing copy_from to smithi12556470-36 2021-03-24T21:59:56.807 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 36 v 539 (ObjNum 520 snap 154 seq_num 520) dirty exists 2021-03-24T21:59:56.807 INFO:tasks.rados.rados.0.smithi125.stdout:1569: finishing copy_from racing read to smithi12556470-36 2021-03-24T21:59:56.807 INFO:tasks.rados.rados.0.smithi125.stdout:1567: done (6 left) 2021-03-24T21:59:56.808 INFO:tasks.rados.rados.0.smithi125.stdout:1568: done (5 left) 2021-03-24T21:59:56.808 INFO:tasks.rados.rados.0.smithi125.stdout:1569: done (4 left) 2021-03-24T21:59:56.808 INFO:tasks.rados.rados.0.smithi125.stdout:1573: done (3 left) 2021-03-24T21:59:56.808 INFO:tasks.rados.rados.0.smithi125.stdout:1574: write oid 1 current snap is 172 2021-03-24T21:59:56.809 INFO:tasks.rados.rados.0.smithi125.stdout:1574: seq_num 555 ranges {527885=656964,1927271=483852,3191408=136477} 2021-03-24T21:59:56.809 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2021-03-24T21:59:56.810 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T21:59:56.811 INFO:tasks.rados.rados.0.smithi125.stdout:1574: writing smithi12556470-1 from 527885 to 1184849 tid 1 2021-03-24T21:59:56.818 INFO:tasks.rados.rados.0.smithi125.stdout:1574: writing smithi12556470-1 from 1927271 to 2411123 tid 2 2021-03-24T21:59:56.824 INFO:tasks.rados.rados.0.smithi125.stdout:1574: writing smithi12556470-1 from 3191408 to 3327885 tid 3 2021-03-24T21:59:56.824 INFO:tasks.rados.rados.0.smithi125.stdout:1575: write (excl) oid 22 current snap is 172 2021-03-24T21:59:56.824 INFO:tasks.rados.rados.0.smithi125.stdout:1575: seq_num 556 ranges {544692=732213,1985753=743863,3335184=9508} 2021-03-24T21:59:56.832 INFO:tasks.rados.rados.0.smithi125.stdout:1575: writing smithi12556470-22 from 544692 to 1276905 tid 1 2021-03-24T21:59:56.841 INFO:tasks.rados.rados.0.smithi125.stdout:1575: writing smithi12556470-22 from 1985753 to 2729616 tid 2 2021-03-24T21:59:56.845 INFO:tasks.rados.rados.0.smithi125.stdout:1575: writing smithi12556470-22 from 3335184 to 3344692 tid 3 2021-03-24T21:59:56.846 INFO:tasks.rados.rados.0.smithi125.stdout:1576: read oid 50 snap 128 2021-03-24T21:59:56.846 INFO:tasks.rados.rados.0.smithi125.stdout:1576: expect (ObjNum 435 snap 115 seq_num 435) 2021-03-24T21:59:56.846 INFO:tasks.rados.rados.0.smithi125.stdout:1577: read oid 24 snap 165 2021-03-24T21:59:56.846 INFO:tasks.rados.rados.0.smithi125.stdout:1577: expect (ObjNum 515 snap 151 seq_num 515) 2021-03-24T21:59:56.846 INFO:tasks.rados.rados.0.smithi125.stdout:1578: delete oid 49 current snap is 172 2021-03-24T21:59:56.847 INFO:tasks.rados.rados.0.smithi125.stdout:1571: expect (ObjNum 393 snap 100 seq_num 393) 2021-03-24T21:59:56.856 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:56.908 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:56.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:56.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:56.997 INFO:tasks.rados.rados.0.smithi125.stdout:1574: finishing write tid 1 to smithi12556470-1 2021-03-24T21:59:56.998 INFO:tasks.rados.rados.0.smithi125.stdout:1572: finishing copy_from to smithi12556470-27 2021-03-24T21:59:56.998 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 27 v 383 (ObjNum 393 snap 100 seq_num 393) dirty exists 2021-03-24T21:59:56.998 INFO:tasks.rados.rados.0.smithi125.stdout:1572: finishing copy_from racing read to smithi12556470-27 2021-03-24T21:59:56.998 INFO:tasks.rados.rados.0.smithi125.stdout:1574: finishing write tid 2 to smithi12556470-1 2021-03-24T21:59:56.999 INFO:tasks.rados.rados.0.smithi125.stdout:1574: finishing write tid 3 to smithi12556470-1 2021-03-24T21:59:56.999 INFO:tasks.rados.rados.0.smithi125.stdout:1574: finishing write tid 4 to smithi12556470-1 2021-03-24T21:59:56.999 INFO:tasks.rados.rados.0.smithi125.stdout:1574: finishing write tid 5 to smithi12556470-1 2021-03-24T21:59:56.999 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 1 v 483 (ObjNum 555 snap 172 seq_num 555) dirty exists 2021-03-24T21:59:56.999 INFO:tasks.rados.rados.0.smithi125.stdout:1574: left oid 1 (ObjNum 555 snap 172 seq_num 555) 2021-03-24T21:59:57.000 INFO:tasks.rados.rados.0.smithi125.stdout:1575: finishing write tid 1 to smithi12556470-22 2021-03-24T21:59:57.000 INFO:tasks.rados.rados.0.smithi125.stdout:1575: finishing write tid 2 to smithi12556470-22 2021-03-24T21:59:57.000 INFO:tasks.rados.rados.0.smithi125.stdout:1575: finishing write tid 3 to smithi12556470-22 2021-03-24T21:59:57.000 INFO:tasks.rados.rados.0.smithi125.stdout:1575: finishing write tid 4 to smithi12556470-22 2021-03-24T21:59:57.001 INFO:tasks.rados.rados.0.smithi125.stdout:1575: finishing write tid 5 to smithi12556470-22 2021-03-24T21:59:57.001 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 22 v 488 (ObjNum 556 snap 172 seq_num 556) dirty exists 2021-03-24T21:59:57.001 INFO:tasks.rados.rados.0.smithi125.stdout:1575: left oid 22 (ObjNum 556 snap 172 seq_num 556) 2021-03-24T21:59:57.001 INFO:tasks.rados.rados.0.smithi125.stdout:1576: expect (ObjNum 435 snap 115 seq_num 435) 2021-03-24T21:59:57.013 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:57.059 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:57.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:57.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:57.075 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 21:59:56 smithi142 podman[66712]: 2021-03-24 21:59:56.867217963 +0000 UTC m=+0.344093122 container create a1b2ec26c85d1a9908b76a3702d8d989a024aa1578d51f03de80747c615d29d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:e9db06906b8d0a24f3ef6835ac5aa4b8eea2950a2560bad78f83d4f101eca0db, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5, GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20201204, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.18.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d, maintainer=Dimitri Savineau , CEPH_POINT_RELEASE=) 2021-03-24T21:59:57.075 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 21:59:57 smithi142 podman[66712]: 2021-03-24 21:59:57.000571768 +0000 UTC m=+0.477446978 container init a1b2ec26c85d1a9908b76a3702d8d989a024aa1578d51f03de80747c615d29d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:e9db06906b8d0a24f3ef6835ac5aa4b8eea2950a2560bad78f83d4f101eca0db, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06, RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20201204, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Dimitri Savineau , org.label-schema.name=CentOS Base Image, io.buildah.version=1.18.0) 2021-03-24T21:59:57.076 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 21:59:57 smithi142 podman[66712]: 2021-03-24 21:59:57.075736666 +0000 UTC m=+0.552611891 container start a1b2ec26c85d1a9908b76a3702d8d989a024aa1578d51f03de80747c615d29d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:e9db06906b8d0a24f3ef6835ac5aa4b8eea2950a2560bad78f83d4f101eca0db, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Dimitri Savineau , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d, io.buildah.version=1.18.0, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20201204, GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06, org.label-schema.name=CentOS Base Image) 2021-03-24T21:59:57.081 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.5 dump_ops_in_flight 2021-03-24T21:59:57.164 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:57.210 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:57.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:57.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:59:57.293 INFO:tasks.rados.rados.0.smithi125.stdout:1577: expect (ObjNum 515 snap 151 seq_num 515) 2021-03-24T21:59:57.317 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:57.367 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 21:59:57 smithi142 bash[66712]: a1b2ec26c85d1a9908b76a3702d8d989a024aa1578d51f03de80747c615d29d9 2021-03-24T21:59:57.368 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 21:59:57 smithi142 systemd[1]: Started Ceph osd.5 for 3cfa339a-8cea-11eb-90b2-001a4aab830c. 2021-03-24T21:59:57.368 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 21:59:57 smithi142 conmon[66800]: debug 2021-03-24T21:59:57.364+0000 7f22596f7080 -1 Falling back to public interface 2021-03-24T21:59:57.442 INFO:tasks.rados.rados.0.smithi125.stdout:1570: expect (ObjNum 510 snap 148 seq_num 510) 2021-03-24T21:59:57.445 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T21:59:57.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:57.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T21:59:57.549 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:57.596 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:57.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:57.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:59:57.651 INFO:tasks.rados.rados.0.smithi125.stdout:1570: done (7 left) 2021-03-24T21:59:57.651 INFO:tasks.rados.rados.0.smithi125.stdout:1571: done (6 left) 2021-03-24T21:59:57.651 INFO:tasks.rados.rados.0.smithi125.stdout:1572: done (5 left) 2021-03-24T21:59:57.652 INFO:tasks.rados.rados.0.smithi125.stdout:1574: done (4 left) 2021-03-24T21:59:57.652 INFO:tasks.rados.rados.0.smithi125.stdout:1575: done (3 left) 2021-03-24T21:59:57.652 INFO:tasks.rados.rados.0.smithi125.stdout:1576: done (2 left) 2021-03-24T21:59:57.652 INFO:tasks.rados.rados.0.smithi125.stdout:1577: done (1 left) 2021-03-24T21:59:57.653 INFO:tasks.rados.rados.0.smithi125.stdout:1578: done (0 left) 2021-03-24T21:59:57.653 INFO:tasks.rados.rados.0.smithi125.stdout:1579: snap_remove snap 137 2021-03-24T21:59:57.702 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:57.756 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:57.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:57.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:57.862 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:57.864 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T21:59:57.891 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.4 dump_historic_ops 2021-03-24T21:59:57.909 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:57.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:57.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:59:58.014 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:58.068 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:58.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:58.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:59:58.172 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:58.217 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T21:59:58.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:58.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T21:59:58.284 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:58 smithi142 conmon[20125]: cluster 2021-03-24T21:59: 2021-03-24T21:59:58.284 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:58 smithi142 conmon[20125]: 56.846065+0000 mgr.y (mgr.14144) 484 : cluster [DBG] pgmap v802: 177 pgs: 1 remapped+peering, 53 active+clean+remapped, 2 active+recovering+undersized+remapped, 35 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 18 active+undersized+degraded, 1 active+recovering+degraded+remapped, 5 active+recovery_wait+degraded+remapped, 57 active+clean; 411 MiB data, 797 MiB used, 446 GiB / 447 GiB avail; 8.4 MiB/s rd, 5.7 MiB/s wr, 17 op/s; 250/1545 objects degraded (16.181%); 279/1545 objects misplaced (18.058%); 10 MiB/s, 40 objects/s recovering 2021-03-24T21:59:58.285 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:58 smithi142 conmon[20125]: cluster 2021 2021-03-24T21:59:58.285 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:58 smithi142 conmon[20125]: -03-24T21:59:57.242042+0000 mon.a (mon.0) 1248 : cluster [DBG] osdmap e427: 8 total, 7 up, 5 in 2021-03-24T21:59:58.285 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:58 smithi142 conmon[20125]: 2021-03-24T21:59:58.288 INFO:tasks.rados.rados.0.smithi125.stdout:1579: done (0 left) 2021-03-24T21:59:58.288 INFO:tasks.rados.rados.0.smithi125.stdout:1580: write (excl) oid 33 current snap is 172 2021-03-24T21:59:58.288 INFO:tasks.rados.rados.0.smithi125.stdout:1580: seq_num 558 ranges {578306=482711,1586364=631179,2757324=484842} 2021-03-24T21:59:58.296 INFO:tasks.rados.rados.0.smithi125.stdout:1580: writing smithi12556470-33 from 578306 to 1061017 tid 1 2021-03-24T21:59:58.304 INFO:tasks.rados.rados.0.smithi125.stdout:1580: writing smithi12556470-33 from 1586364 to 2217543 tid 2 2021-03-24T21:59:58.311 INFO:tasks.rados.rados.0.smithi125.stdout:1580: writing smithi12556470-33 from 2757324 to 3242166 tid 3 2021-03-24T21:59:58.312 INFO:tasks.rados.rados.0.smithi125.stdout:1580: finishing write tid 1 to smithi12556470-33 2021-03-24T21:59:58.312 INFO:tasks.rados.rados.0.smithi125.stdout:1581: write (excl) oid 6 current snap is 172 2021-03-24T21:59:58.312 INFO:tasks.rados.rados.0.smithi125.stdout:1581: seq_num 559 ranges {595113=557960,1644846=491190,2784747=610366} 2021-03-24T21:59:58.320 INFO:tasks.rados.rados.0.smithi125.stdout:1581: writing smithi12556470-6 from 595113 to 1153073 tid 1 2021-03-24T21:59:58.322 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:58.327 INFO:tasks.rados.rados.0.smithi125.stdout:1581: writing smithi12556470-6 from 1644846 to 2136036 tid 2 2021-03-24T21:59:58.335 INFO:tasks.rados.rados.0.smithi125.stdout:1581: writing smithi12556470-6 from 2784747 to 3395113 tid 3 2021-03-24T21:59:58.337 INFO:tasks.rados.rados.0.smithi125.stdout:1580: finishing write tid 2 to smithi12556470-33 2021-03-24T21:59:58.337 INFO:tasks.rados.rados.0.smithi125.stdout:1580: finishing write tid 3 to smithi12556470-33 2021-03-24T21:59:58.337 INFO:tasks.rados.rados.0.smithi125.stdout:1580: finishing write tid 4 to smithi12556470-33 2021-03-24T21:59:58.337 INFO:tasks.rados.rados.0.smithi125.stdout:1580: finishing write tid 5 to smithi12556470-33 2021-03-24T21:59:58.338 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 33 v 451 (ObjNum 558 snap 172 seq_num 558) dirty exists 2021-03-24T21:59:58.338 INFO:tasks.rados.rados.0.smithi125.stdout:1580: left oid 33 (ObjNum 558 snap 172 seq_num 558) 2021-03-24T21:59:58.338 INFO:tasks.rados.rados.0.smithi125.stdout:1580: done (1 left) 2021-03-24T21:59:58.338 INFO:tasks.rados.rados.0.smithi125.stdout:1582: rollback oid 28 current snap is 172 2021-03-24T21:59:58.339 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 28 to 149 2021-03-24T21:59:58.339 INFO:tasks.rados.rados.0.smithi125.stdout:1583: copy_from oid 20 from oid 48 current snap is 172 2021-03-24T21:59:58.339 INFO:tasks.rados.rados.0.smithi125.stdout:1584: delete oid 44 current snap is 172 2021-03-24T21:59:58.355 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T21:59:58.362 INFO:tasks.rados.rados.0.smithi125.stdout:1582: finishing rollback tid 0 to smithi12556470-28 2021-03-24T21:59:58.362 INFO:tasks.rados.rados.0.smithi125.stdout:1582: finishing rollback tid 1 to smithi12556470-28 2021-03-24T21:59:58.364 INFO:tasks.rados.rados.0.smithi125.stdout:1582: finishing rollback tid 2 to smithi12556470-28 2021-03-24T21:59:58.364 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 28 v 386 (ObjNum 501 snap 146 seq_num 501) dirty exists 2021-03-24T21:59:58.369 INFO:tasks.rados.rados.0.smithi125.stdout:1581: finishing write tid 1 to smithi12556470-6 2021-03-24T21:59:58.373 INFO:tasks.rados.rados.0.smithi125.stdout:1581: finishing write tid 2 to smithi12556470-6 2021-03-24T21:59:58.374 INFO:tasks.rados.rados.0.smithi125.stdout:1581: finishing write tid 3 to smithi12556470-6 2021-03-24T21:59:58.374 INFO:tasks.rados.rados.0.smithi125.stdout:1581: finishing write tid 4 to smithi12556470-6 2021-03-24T21:59:58.375 INFO:tasks.rados.rados.0.smithi125.stdout:1581: finishing write tid 5 to smithi12556470-6 2021-03-24T21:59:58.375 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 6 v 376 (ObjNum 559 snap 172 seq_num 559) dirty exists 2021-03-24T21:59:58.375 INFO:tasks.rados.rados.0.smithi125.stdout:1581: left oid 6 (ObjNum 559 snap 172 seq_num 559) 2021-03-24T21:59:58.380 INFO:tasks.rados.rados.0.smithi125.stdout:1581: done (3 left) 2021-03-24T21:59:58.380 INFO:tasks.rados.rados.0.smithi125.stdout:1582: done (2 left) 2021-03-24T21:59:58.380 INFO:tasks.rados.rados.0.smithi125.stdout:1584: done (1 left) 2021-03-24T21:59:58.381 INFO:tasks.rados.rados.0.smithi125.stdout:1585: write oid 6 current snap is 172 2021-03-24T21:59:58.381 INFO:tasks.rados.rados.0.smithi125.stdout:1585: seq_num 561 ranges {628727=708458,2045457=494859,3123240=305487} 2021-03-24T21:59:58.388 INFO:tasks.rados.rados.0.smithi125.stdout:1585: writing smithi12556470-6 from 628727 to 1337185 tid 1 2021-03-24T21:59:58.396 INFO:tasks.rados.rados.0.smithi125.stdout:1585: writing smithi12556470-6 from 2045457 to 2540316 tid 2 2021-03-24T21:59:58.405 INFO:tasks.rados.rados.0.smithi125.stdout:1585: writing smithi12556470-6 from 3123240 to 3428727 tid 3 2021-03-24T21:59:58.405 INFO:tasks.rados.rados.0.smithi125.stdout:1585: finishing write tid 1 to smithi12556470-6 2021-03-24T21:59:58.406 INFO:tasks.rados.rados.0.smithi125.stdout:1586: snap_create 2021-03-24T21:59:58.408 INFO:tasks.rados.rados.0.smithi125.stdout:1585: finishing write tid 2 to smithi12556470-6 2021-03-24T21:59:58.409 INFO:tasks.rados.rados.0.smithi125.stdout:1585: finishing write tid 3 to smithi12556470-6 2021-03-24T21:59:58.413 INFO:tasks.rados.rados.0.smithi125.stdout:1585: finishing write tid 4 to smithi12556470-6 2021-03-24T21:59:58.414 INFO:tasks.rados.rados.0.smithi125.stdout:1585: finishing write tid 5 to smithi12556470-6 2021-03-24T21:59:58.414 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 6 v 380 (ObjNum 561 snap 172 seq_num 561) dirty exists 2021-03-24T21:59:58.415 INFO:tasks.rados.rados.0.smithi125.stdout:1585: left oid 6 (ObjNum 561 snap 172 seq_num 561) 2021-03-24T21:59:58.430 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:58.436 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:58 smithi125 conmon[19936]: cluster 2021-03-24T21:59:56.846065+0000 2021-03-24T21:59:58.437 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:58 smithi125 conmon[19936]: mgr.y (mgr.14144) 484 : cluster [DBG] pgmap v802: 177 pgs: 1 remapped+peering, 53 active+clean+remapped, 2 active+recovering+undersized+remapped, 35 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 18 active+undersized+degraded, 1 active+recovering+degraded+remapped, 5 active+recovery_wait+degraded+remapped, 57 active+clean; 411 MiB data, 797 MiB used, 446 GiB / 447 GiB avail; 8.4 MiB/s rd, 5.7 MiB/s wr, 17 op/s; 250/1545 objects degraded (16.181%); 279/1545 objects misplaced (18.058%); 10 MiB/s, 40 objects/s recovering 2021-03-24T21:59:58.437 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:58 smithi125 conmon[19936]: cluster 2021-03-24T21:59:58.437 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:58 smithi125 conmon[19936]: 2021-03-24T21:59:57.242042+0000 mon.a (mon.0 2021-03-24T21:59:58.438 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:58 smithi125 conmon[19936]: ) 1248 : cluster [DBG] osdmap e427: 8 total, 7 up, 5 in 2021-03-24T21:59:58.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:58.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:59:58.440 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:58 smithi125 conmon[24927]: cluster 2021-03-24T21:59:56. 2021-03-24T21:59:58.440 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:58 smithi125 conmon[24927]: 846065+0000 mgr.y (mgr.14144) 484 : cluster [DBG] pgmap v802: 177 pgs: 1 remapped+peering, 53 active+clean+remapped, 2 active+recovering+undersized+remapped, 35 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 18 active+undersized+degraded, 1 active+recovering+degraded+remapped, 5 active+recovery_wait+degraded+remapped, 57 active+clean; 411 MiB data, 797 MiB used, 446 GiB / 447 GiB avail; 8.4 MiB/s rd, 5.7 MiB/s wr, 17 op/s; 250/1545 objects degraded (16.181%); 279/1545 objects misplaced (18.058%); 10 MiB/s, 40 objects/s recovering 2021-03-24T21:59:58.440 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:58 smithi125 conmon[24927]: cluster 2021-03-24T21:59:57. 2021-03-24T21:59:58.440 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:58 smithi125 conmon[24927]: 242042+0000 mon.a (mon.0) 1248 : cluster [DBG] osdmap e427: 8 total, 7 up, 5 in 2021-03-24T21:59:58.540 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:58.580 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-24T21:59:58.597 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:58.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:58.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:58.702 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:58.749 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:58.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:58.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:59:58.854 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:58.898 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T21:59:58.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:58.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T21:59:58.904 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T21:59:59.005 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:59.060 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:59.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:59.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:59.167 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:59.227 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T21:59:59.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:59.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T21:59:59.296 INFO:tasks.rados.rados.0.smithi125.stdout:1585: done (2 left) 2021-03-24T21:59:59.296 INFO:tasks.rados.rados.0.smithi125.stdout:1586: done (1 left) 2021-03-24T21:59:59.296 INFO:tasks.rados.rados.0.smithi125.stdout:1587: snap_remove snap 165 2021-03-24T21:59:59.322 INFO:teuthology.orchestra.run.smithi125.stderr:noscrub is set 2021-03-24T21:59:59.332 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:59.344 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: cluster 2021-03-24T21:59:58.284897+0000 2021-03-24T21:59:59.345 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: mon.a (mon.0) 1249 : cluster [DBG] osdmap e428: 8 total, 7 up, 5 in 2021-03-24T21:59:59.345 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: audit 2021-03-24T21:59:58.937077+0000 mon.c (mon 2021-03-24T21:59:59.345 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: .2) 67 : audit [INF] from='client.? v1:172.21.15.125:0/2826100500' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T21:59:59.345 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: audit 2021-03-24T21:59:59.346 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: 2021-03-24T21:59:59.346 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: 2021 2021-03-24T21:59:59.346 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: - 2021-03-24T21:59:59.347 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: 03 2021-03-24T21:59:59.347 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: - 2021-03-24T21:59:59.347 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: 24 2021-03-24T21:59:59.347 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: T 2021-03-24T21:59:59.347 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: 21 2021-03-24T21:59:59.348 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: : 2021-03-24T21:59:59.348 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: 59 2021-03-24T21:59:59.348 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: : 2021-03-24T21:59:59.348 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: 58 2021-03-24T21:59:59.348 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: . 2021-03-24T21:59:59.349 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: 937529 2021-03-24T21:59:59.349 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: +0000 2021-03-24T21:59:59.350 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: 2021-03-24T21:59:59.350 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: mon.a 2021-03-24T21:59:59.351 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: ( 2021-03-24T21:59:59.351 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: mon 2021-03-24T21:59:59.351 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: . 2021-03-24T21:59:59.352 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: 0 2021-03-24T21:59:59.352 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: ) 2021-03-24T21:59:59.352 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: 1250 2021-03-24T21:59:59.353 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: : 2021-03-24T21:59:59.353 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: audit 2021-03-24T21:59:59.353 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: 2021-03-24T21:59:59.353 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: [INF] 2021-03-24T21:59:59.354 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: 2021-03-24T21:59:59.354 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: from='client.35231 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T21:59:59.354 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: 2021-03-24T21:59:59.354 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: audit 2021-03-24T21:59:59.355 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: 2021-03-24T21:59:59.355 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: 2021 2021-03-24T21:59:59.355 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: - 2021-03-24T21:59:59.355 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: 03 2021-03-24T21:59:59.356 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: - 2021-03-24T21:59:59.356 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: 24 2021-03-24T21:59:59.356 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: T 2021-03-24T21:59:59.356 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: 21 2021-03-24T21:59:59.356 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: : 2021-03-24T21:59:59.357 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: 59 2021-03-24T21:59:59.357 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: : 2021-03-24T21:59:59.357 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: 59 2021-03-24T21:59:59.357 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: . 2021-03-24T21:59:59.358 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: 131987 2021-03-24T21:59:59.358 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: +0000 2021-03-24T21:59:59.358 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: 2021-03-24T21:59:59.358 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: mon.a 2021-03-24T21:59:59.359 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: ( 2021-03-24T21:59:59.359 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: mon 2021-03-24T21:59:59.359 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: . 2021-03-24T21:59:59.359 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: 0 2021-03-24T21:59:59.359 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: ) 2021-03-24T21:59:59.360 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: 1251 2021-03-24T21:59:59.360 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: : 2021-03-24T21:59:59.360 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: audit 2021-03-24T21:59:59.360 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: 2021-03-24T21:59:59.361 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: [DBG] 2021-03-24T21:59:59.361 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: 2021-03-24T21:59:59.361 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:59:59.361 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[24927]: 2021-03-24T21:59:59.362 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[19936]: cluster 2021-03-24T21:59:58.284897+0000 mon.a (mon.0) 1249 : cluster [DBG] osdmap e428: 8 total, 7 up, 5 in 2021-03-24T21:59:59.362 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[19936]: audit 2021-03-24T21:59:58.937077+0000 mon.c (mon.2) 67 : audit [INF] from='client.? v1:172.21.15.125:0/2826100500' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T21:59:59.363 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[19936]: audit 2021-03-24T21:59:58.937529+0000 mon.a (mon.0) 1250 : audit [INF] from='client.35231 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T21:59:59.363 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 21:59:59 smithi125 conmon[19936]: audit 2021-03-24T21:59:59.131987+0000 mon.a (mon.0) 1251 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:59:59.431 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T21:59:59.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:59.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T21:59:59.536 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:59.547 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:59 smithi142 conmon[20125]: cluster 2021-03-24T21:59:58.284897+0000 mon.a (mon.0) 1249 : cluster [DBG] osdmap e428: 8 total, 7 up, 5 in 2021-03-24T21:59:59.547 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:59 smithi142 conmon[20125]: audit 2021-03-24T21:59:58.937077+0000 mon.c (mon.2) 67 : audit [INF] from='client.? v1:172.21.15.125:0/2826100500' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T21:59:59.547 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:59 smithi142 conmon[20125]: audit 2021-03-24T21:59:58.937529+0000 mon.a (mon.0) 1250 : audit [INF] from='client.35231 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T21:59:59.547 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 21:59:59 smithi142 conmon[20125]: audit 2021-03-24T21:59:59.131987+0000 mon.a (mon.0) 1251 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T21:59:59.583 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T21:59:59.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:59.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T21:59:59.689 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:59.737 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:59.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:59.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:59:59.843 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:59.887 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T21:59:59.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:59:59.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T21:59:59.991 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:00.044 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:00.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:00.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:00:00.058 INFO:teuthology.orchestra.run.smithi142.stderr:no valid command found; 10 closest matches: 2021-03-24T22:00:00.058 INFO:teuthology.orchestra.run.smithi142.stderr:0 2021-03-24T22:00:00.058 INFO:teuthology.orchestra.run.smithi142.stderr:1 2021-03-24T22:00:00.058 INFO:teuthology.orchestra.run.smithi142.stderr:2 2021-03-24T22:00:00.059 INFO:teuthology.orchestra.run.smithi142.stderr:abort 2021-03-24T22:00:00.059 INFO:teuthology.orchestra.run.smithi142.stderr:assert 2021-03-24T22:00:00.059 INFO:teuthology.orchestra.run.smithi142.stderr:config diff 2021-03-24T22:00:00.059 INFO:teuthology.orchestra.run.smithi142.stderr:config diff get 2021-03-24T22:00:00.060 INFO:teuthology.orchestra.run.smithi142.stderr:config get 2021-03-24T22:00:00.060 INFO:teuthology.orchestra.run.smithi142.stderr:config help [] 2021-03-24T22:00:00.060 INFO:teuthology.orchestra.run.smithi142.stderr:config set ... 2021-03-24T22:00:00.060 INFO:teuthology.orchestra.run.smithi142.stderr:admin_socket: invalid command 2021-03-24T22:00:00.150 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:00.198 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:00.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:00.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:00:00.225 INFO:tasks.rados.rados.0.smithi125.stdout:1587: done (1 left) 2021-03-24T22:00:00.225 INFO:tasks.rados.rados.0.smithi125.stdout:1588: snap_remove snap 164 2021-03-24T22:00:00.305 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:00.329 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03-24T21:59:58.846705 2021-03-24T22:00:00.330 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: +0000 mgr.y (mgr.14144) 485 : cluster [DBG] pgmap v805: 177 pgs: 1 remapped+peering, 54 active+clean+remapped, 1 active+recovering+undersized+remapped, 35 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 18 active+undersized+degraded, 1 active+recovering+degraded+remapped, 4 active+recovery_wait+degraded+remapped, 58 active+clean; 411 MiB data, 834 MiB used, 446 GiB / 447 GiB avail; 255 B/s rd, 1.8 MiB/s wr, 3 op/s; 240/1545 objects degraded (15.534%); 275/1545 objects misplaced (17.799%); 6.9 MiB/s, 14 objects/s recovering 2021-03-24T22:00:00.330 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03-24T21:59:59.282839+0000 mon.a (mon.0) 1252 : 2021-03-24T22:00:00.330 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:00:00.331 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: audit 2021-03-24 2021-03-24T22:00:00.331 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: T21:59:59.293445+0000 mon.a (mon.0) 1253 : 2021-03-24T22:00:00.331 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: audit [INF] from='client.35231 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T22:00:00.331 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03-24T21 2021-03-24T22:00:00.331 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: :59:59.293508+0000 mon.a (mon.0 2021-03-24T22:00:00.332 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: ) 1254 : cluster [DBG] osdmap e429: 8 total, 7 up, 5 in 2021-03-24T22:00:00.332 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: audit 2021-03 2021-03-24T22:00:00.332 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: -24T21:59:59.546748+0000 mon.a 2021-03-24T22:00:00.332 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: (mon.0) 1255 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:00:00.332 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:00:00.333 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: -03-24T22:00:00.000227 2021-03-24T22:00:00.333 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: +0000 mon.a (mon.0) 1256 : cluster [WRN] 2021-03-24T22:00:00.333 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: Health detail: HEALTH_WARN noscrub flag(s) set; 1 osds down; Degraded data redundancy: 240/1545 objects degraded (15.534%), 28 pgs degraded, 47 pgs undersized 2021-03-24T22:00:00.333 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03-24T22:00: 2021-03-24T22:00:00.334 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 00.000296+0000 mon.a (mon.0) 1257 2021-03-24T22:00:00.334 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: : cluster [WRN] [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2021-03-24T22:00:00.334 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:00:00.334 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 22:00:00.000331+0000 mon.a (mon. 2021-03-24T22:00:00.335 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 0) 1258 : cluster [WRN] [WRN] OSD_DOWN: 1 osds down 2021-03-24T22:00:00.335 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:00:00.335 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: -24T22:00:00.000362+0000 2021-03-24T22:00:00.335 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: mon.a (mon.0) 1259 : cluster [WRN] osd.5 (root=default,host=smithi142) is down 2021-03-24T22:00:00.336 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 2021-03-24T22:00:00.336 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03-24T22:00:00.336 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 2021-03- 2021-03-24T22:00:00.336 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 24T22:00:00.000388+0000 mon.a 2021-03-24T22:00:00.337 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: (mon.0) 1260 : cluster [WRN] [WRN] PG_DEGRADED: Degraded data redundancy: 240/1545 objects degraded (15.534%), 28 pgs degraded, 47 pgs undersized 2021-03-24T22:00:00.337 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03-24T22:00:00.337 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 2021-03-24T 2021-03-24T22:00:00.337 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 22:00:00.000422+0000 mon.a (mon. 2021-03-24T22:00:00.337 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 0) 1261 : cluster [WRN] pg 2.2 is stuck undersized for 87s, current state active+undersized, last acting [0,3] 2021-03-24T22:00:00.338 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03-24T22:00:00.338 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 2021-03-24T22 2021-03-24T22:00:00.338 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: :00:00.000453+0000 mon.a (mon.0 2021-03-24T22:00:00.339 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: ) 1262 : cluster [WRN] pg 3.a is stuck undersized for 87s, current state active+undersized, last acting [4,3] 2021-03-24T22:00:00.339 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03-24T22:00:00.339 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 2021-03-24T22: 2021-03-24T22:00:00.339 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 00:00.000485+0000 mon.a (mon.0 2021-03-24T22:00:00.340 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: ) 1263 : cluster [WRN] pg 3.c is stuck undersized for 87s, current state active+undersized, last acting [3,0] 2021-03-24T22:00:00.340 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03-24T22:00:00.341 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 2021-03-24T22:00:00. 2021-03-24T22:00:00.341 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 000531+0000 mon.a (mon.0) 1264 : cluster 2021-03-24T22:00:00.341 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: [WRN] pg 3.10 is stuck undersized for 87s, current state active+undersized, last acting [0,3] 2021-03-24T22:00:00.341 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03-24T22: 2021-03-24T22:00:00.342 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 00:00.000558+0000 mon.a (mon.0 2021-03-24T22:00:00.342 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: ) 1265 : cluster [WRN] pg 3.17 is stuck undersized for 87s, current state active+undersized, last acting [0,3] 2021-03-24T22:00:00.342 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:00:00.342 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: -24T22:00:00.000594+0000 mon.a 2021-03-24T22:00:00.342 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: (mon.0) 1266 : cluster [WRN] pg 3.19 is stuck undersized for 87s, current state active+undersized, last acting [3,4] 2021-03-24T22:00:00.343 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03-24T22:00:00.343 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 2021-03-24T22:00: 2021-03-24T22:00:00.343 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 00.000627+0000 mon.a (mon.0) 1267 : 2021-03-24T22:00:00.343 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster [WRN] pg 4.1 is stuck undersized for 87s, current state active+undersized+degraded, last acting [4,0] 2021-03-24T22:00:00.344 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:00:00.344 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: -24T22:00:00.000657+0000 2021-03-24T22:00:00.344 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: mon.a (mon.0) 1268 : cluster [WRN] 2021-03-24T22:00:00.344 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: pg 4.2 is stuck undersized for 87s, current state active+undersized+degraded, last acting [4,0] 2021-03-24T22:00:00.344 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:00:00.345 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: -24T22:00:00.000685+0000 mon.a 2021-03-24T22:00:00.345 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: (mon.0) 1269 : cluster [WRN] pg 4.6 is active+recovery_wait+degraded+remapped, acting [0,4,7] 2021-03-24T22:00:00.345 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 2021-03-24T22:00:00.345 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03-24T22:00:00.345 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 2021-03-24T22:00:00.000714+0000 2021-03-24T22:00:00.346 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: mon.a (mon.0) 1270 : cluster [WRN] pg 4.7 is active+recovering+degraded+remapped, acting [0,3,7] 2021-03-24T22:00:00.346 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 2021-03-24T22:00:00.346 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03-24T22:00:00.346 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 2021-03-24T22:00:00.000740+0000 2021-03-24T22:00:00.346 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: mon.a (mon.0) 1271 : cluster [WRN] pg 4.e is stuck undersized for 87s, current state active+undersized+degraded, last acting [4,3] 2021-03-24T22:00:00.347 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 2021-03-24T22:00:00.347 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03-24T22:00:00.347 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 2021-03-24T22:00:00.000772+0000 mon.a 2021-03-24T22:00:00.347 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: (mon.0) 1272 : cluster [WRN] pg 4.10 is stuck undersized for 87s, current state active+undersized+degraded, last acting [3,4] 2021-03-24T22:00:00.348 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:00:00.348 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 03-24T22:00:00.000800+0000 2021-03-24T22:00:00.348 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: mon.a (mon.0) 1273 : cluster [WRN] 2021-03-24T22:00:00.348 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: pg 4.12 is stuck undersized for 87s, current state active+undersized+degraded, last acting [2,0] 2021-03-24T22:00:00.361 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021-03-24T21:59:58.846705 2021-03-24T22:00:00.362 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: +0000 mgr.y (mgr.14144) 485 : cluster [DBG] pgmap v805: 177 pgs: 1 remapped+peering, 54 active+clean+remapped, 1 active+recovering+undersized+remapped, 35 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 18 active+undersized+degraded, 1 active+recovering+degraded+remapped, 4 active+recovery_wait+degraded+remapped, 58 active+clean; 411 MiB data, 834 MiB used, 446 GiB / 447 GiB avail; 255 B/s rd, 1.8 MiB/s wr, 3 op/s; 240/1545 objects degraded (15.534%); 275/1545 objects misplaced (17.799%); 6.9 MiB/s, 14 objects/s recovering 2021-03-24T22:00:00.362 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021-03-24T21:59:59.282839+0000 mon.a (mon.0 2021-03-24T22:00:00.362 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: ) 1252 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:00:00.363 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: audit 2021-03 2021-03-24T22:00:00.363 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: -24T21:59:59.293445+0000 mon.a (mon.0 2021-03-24T22:00:00.363 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: ) 1253 : audit [INF] from='client.35231 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T22:00:00.363 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:00:00.363 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: -03-24T21:59:59.293508 2021-03-24T22:00:00.364 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: +0000 mon.a (mon.0) 1254 : cluster 2021-03-24T22:00:00.364 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: [DBG] osdmap e429: 8 total, 7 up, 5 in 2021-03-24T22:00:00.364 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: audit 2021 2021-03-24T22:00:00.364 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: -03-24T21:59:59.546748 2021-03-24T22:00:00.364 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: +0000 mon.a (mon.0) 1255 : audit 2021-03-24T22:00:00.365 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:00:00.365 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:00:00.365 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 03-24T22:00:00.000227 2021-03-24T22:00:00.365 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: +0000 mon.a (mon.0) 1256 : cluster 2021-03-24T22:00:00.365 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: [WRN] Health detail: HEALTH_WARN noscrub flag(s) set; 1 osds down; Degraded data redundancy: 240/1545 objects degraded (15.534%), 28 pgs degraded, 47 pgs undersized 2021-03-24T22:00:00.366 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:00:00.366 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 03-24T22:00:00.000296+0000 2021-03-24T22:00:00.366 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: mon.a (mon.0) 1257 : cluster [WRN] 2021-03-24T22:00:00.366 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2021-03-24T22:00:00.367 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:00:00.367 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 24T22:00:00.000331+0000 mon.a ( 2021-03-24T22:00:00.367 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: mon.0) 1258 : cluster [WRN] [WRN] OSD_DOWN: 1 osds down 2021-03-24T22:00:00.367 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 2021-03-24T22:00:00.368 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021-03-24T22 2021-03-24T22:00:00.368 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: :00:00.000362+0000 mon.a (mon 2021-03-24T22:00:00.368 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: .0) 1259 : cluster [WRN] osd.5 (root=default,host=smithi142) is down 2021-03-24T22:00:00.368 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T22:00:00.369 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: -24T22:00:00.000388+0000 2021-03-24T22:00:00.369 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: mon.a (mon.0) 1260 : cluster [WRN] 2021-03-24T22:00:00.369 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: [WRN] PG_DEGRADED: Degraded data redundancy: 240/1545 objects degraded (15.534%), 28 pgs degraded, 47 pgs undersized 2021-03-24T22:00:00.369 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021-03-24T22:00:00.369 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 2021-03-24T22:00:00.000422 2021-03-24T22:00:00.370 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: +0000 mon.a (mon.0) 1261 : cluster 2021-03-24T22:00:00.370 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: [WRN] pg 2.2 is stuck undersized for 87s, current state active+undersized, last acting [0,3] 2021-03-24T22:00:00.370 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:00:00.370 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 03-24T22:00:00.000453 2021-03-24T22:00:00.370 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: +0000 mon.a (mon.0) 1262 : cluster 2021-03-24T22:00:00.371 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: [WRN] pg 3.a is stuck undersized for 87s, current state active+undersized, last acting [4,3] 2021-03-24T22:00:00.372 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:00:00.372 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: -03-24T22:00:00.000485 2021-03-24T22:00:00.373 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: +0000 mon.a (mon.0) 1263 : cluster 2021-03-24T22:00:00.373 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: [WRN] pg 3.c is stuck undersized for 87s, current state active+undersized, last acting [3,0] 2021-03-24T22:00:00.373 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:00:00.373 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: -03-24T22:00:00. 2021-03-24T22:00:00.374 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 000531+0000 mon.a (mon.0) 1264 : 2021-03-24T22:00:00.374 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster [WRN] pg 3.10 is stuck undersized for 87s, current state active+undersized, last acting [0,3] 2021-03-24T22:00:00.374 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:00:00.374 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: -03-24T22:00:00. 2021-03-24T22:00:00.375 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 000558+0000 mon.a (mon.0) 1265 : cluster 2021-03-24T22:00:00.375 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: [WRN] pg 3.17 is stuck undersized for 87s, current state active+undersized, last acting [0,3] 2021-03-24T22:00:00.375 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:00:00.375 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 03-24T22:00:00.000594 2021-03-24T22:00:00.376 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: +0000 mon.a (mon.0) 1266 : cluster 2021-03-24T22:00:00.376 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: [WRN] pg 3.19 is stuck undersized for 87s, current state active+undersized, last acting [3,4] 2021-03-24T22:00:00.376 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:00:00.376 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 03-24T22: 2021-03-24T22:00:00.377 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 00:00 2021-03-24T22:00:00.377 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: .000627+0000 mon.a (mon.0) 1267 2021-03-24T22:00:00.377 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: : cluster [WRN] pg 4.1 is stuck undersized for 87s, current state active+undersized+degraded, last acting [4,0] 2021-03-24T22:00:00.377 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021-03-24T22:00:00.377 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 2021-03-24T22:00:00 2021-03-24T22:00:00.378 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: .000657+0000 mon.a (mon.0) 1268 2021-03-24T22:00:00.378 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: : cluster [WRN] pg 4.2 is stuck undersized for 87s, current state active+undersized+degraded, last acting [4,0] 2021-03-24T22:00:00.378 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:00:00.378 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 00:00.000685+0000 mon.a (mon.0 2021-03-24T22:00:00.379 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: ) 1269 : cluster [WRN] pg 4.6 is active+recovery_wait+degraded+remapped, acting [0,4,7] 2021-03-24T22:00:00.379 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021-03-24T22:00:00.379 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 2021-03-24T22:00:00 2021-03-24T22:00:00.379 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: .000714+0000 mon.a (mon.0) 1270 2021-03-24T22:00:00.379 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: : cluster [WRN] pg 4.7 is active+recovering+degraded+remapped, acting [0,3,7] 2021-03-24T22:00:00.380 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021-03-24T22:00:00.380 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 2021-03-24T22:00:00 2021-03-24T22:00:00.380 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: .000740+0000 mon.a (mon.0) 1271 2021-03-24T22:00:00.381 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: : cluster [WRN] pg 4.e is stuck undersized for 87s, current state active+undersized+degraded, last acting [4,3] 2021-03-24T22:00:00.381 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:00:00.382 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: -03-24T22:00:00. 2021-03-24T22:00:00.382 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 000772+0000 mon.a (mon.0) 1272 : 2021-03-24T22:00:00.382 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster [WRN] pg 4.10 is stuck undersized for 87s, current state active+undersized+degraded, last acting [3,4] 2021-03-24T22:00:00.382 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021-03-24T22:00:00.382 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 2021-03-24T22:00:00. 2021-03-24T22:00:00.383 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 000800+0000 mon.a (mon.0) 1273 : cluster 2021-03-24T22:00:00.383 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: [WRN] pg 4.12 is stuck undersized for 87s, current state active+undersized+degraded, last acting [2,0] 2021-03-24T22:00:00.383 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:00:00.383 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: -03-24T22:00:00. 2021-03-24T22:00:00.384 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 000829+0000 mon.a (mon.0) 1274 : cluster 2021-03-24T22:00:00.384 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: [WRN] pg 4.13 is stuck undersized for 87s, current state active+undersized+degraded, last acting [4,2] 2021-03-24T22:00:00.384 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:00:00.384 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: -03-24T22:00:00. 2021-03-24T22:00:00.385 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.385 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 2021-03-24T22:00:00.385 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 2021 2021-03-24T22:00:00.386 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: - 2021-03-24T22:00:00.386 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 03 2021-03-24T22:00:00.386 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: - 2021-03-24T22:00:00.386 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 24 2021-03-24T22:00:00.386 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: T 2021-03-24T22:00:00.387 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 21 2021-03-24T22:00:00.387 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: : 2021-03-24T22:00:00.387 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 59 2021-03-24T22:00:00.387 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: : 2021-03-24T22:00:00.387 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 58 2021-03-24T22:00:00.388 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: . 2021-03-24T22:00:00.388 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 846705 2021-03-24T22:00:00.388 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: +0000 2021-03-24T22:00:00.388 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 2021-03-24T22:00:00.389 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: mgr.y 2021-03-24T22:00:00.389 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: ( 2021-03-24T22:00:00.389 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: mgr 2021-03-24T22:00:00.389 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: . 2021-03-24T22:00:00.389 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 14144 2021-03-24T22:00:00.390 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: ) 2021-03-24T22:00:00.390 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 485 2021-03-24T22:00:00.390 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: : 2021-03-24T22:00:00.390 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.391 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 2021-03-24T22:00:00.391 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: [DBG] 2021-03-24T22:00:00.391 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 2021-03-24T22:00:00.391 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: pgmap v805: 177 pgs: 1 remapped+peering, 54 active+clean+remapped, 1 active+recovering+undersized+remapped, 35 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 18 active+undersized+degraded, 1 active+recovering+degraded+remapped, 4 active+recovery_wait+degraded+remapped, 58 active+clean; 411 MiB data, 834 MiB used, 446 GiB / 447 GiB avail; 255 B/s rd, 1.8 MiB/s wr, 3 op/s; 240/1545 objects degraded (15.534%); 275/1545 objects misplaced (17.799%); 6.9 MiB/s, 14 objects/s recovering 2021-03-24T22:00:00.391 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 2021-03-24T22:00:00.392 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.392 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 2021-03-24T22:00:00.392 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 2021 2021-03-24T22:00:00.392 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: - 2021-03-24T22:00:00.392 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 03 2021-03-24T22:00:00.393 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: - 2021-03-24T22:00:00.393 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 24 2021-03-24T22:00:00.393 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: T 2021-03-24T22:00:00.393 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 21 2021-03-24T22:00:00.394 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: : 2021-03-24T22:00:00.394 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 59 2021-03-24T22:00:00.394 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: : 2021-03-24T22:00:00.394 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 59 2021-03-24T22:00:00.394 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: . 2021-03-24T22:00:00.395 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 282839 2021-03-24T22:00:00.395 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: +0000 2021-03-24T22:00:00.395 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 2021-03-24T22:00:00.395 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: mon.a 2021-03-24T22:00:00.396 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 000861+0000 mon.a (mon.0) 1275 : 2021-03-24T22:00:00.396 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster [WRN] pg 4.15 is active+recovery_wait+degraded+remapped, acting [3,0,7] 2021-03-24T22:00:00.396 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021-03-24T22:00:00.397 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 2021-03-24T22:00:00. 2021-03-24T22:00:00.397 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 000894+0000 mon.a (mon.0) 1276 2021-03-24T22:00:00.397 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: : cluster [WRN] pg 4.16 is active+recovery_wait+degraded+remapped, acting [0,4,7] 2021-03-24T22:00:00.397 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:00:00.397 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 03-24T22:00:00.000930+0000 mon.a (mon.0) 2021-03-24T22:00:00.398 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 1277 : cluster [WRN] pg 4.17 is active+recovery_wait+degraded+remapped, acting [0,3,7] 2021-03-24T22:00:00.398 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021-03-24T22:00:00.398 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 2021-03-24T22:00:00 2021-03-24T22:00:00.398 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: .000964+0000 mon.a (mon.0) 1278 2021-03-24T22:00:00.398 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: : cluster [WRN] pg 5.2 is stuck undersized for 87s, current state active+undersized, last acting [0,2] 2021-03-24T22:00:00.399 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:00:00.399 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 03-24T22:00:00.001004 2021-03-24T22:00:00.399 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: +0000 mon.a (mon.0) 1279 : cluster 2021-03-24T22:00:00.399 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: [WRN] pg 5.3 is stuck undersized for 87s, current state active+undersized+degraded, last acting [0,3] 2021-03-24T22:00:00.400 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:00:00.400 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: -03-24T22:00:00.001033 2021-03-24T22:00:00.400 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: +0000 mon.a (mon.0) 1280 : cluster 2021-03-24T22:00:00.400 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: [WRN] pg 5.7 is stuck undersized for 87s, current state active+undersized, last acting [0,4] 2021-03-24T22:00:00.400 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:00:00.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: -03-24T22:00:00. 2021-03-24T22:00:00.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 001049+0000 mon.a (mon.0) 1281 : cluster 2021-03-24T22:00:00.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: [WRN] pg 5.8 is stuck undersized for 87s, current state active+undersized, last acting [2,0] 2021-03-24T22:00:00.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:00:00.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: -03-24T22:00:00. 2021-03-24T22:00:00.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 001062+0000 mon.a (mon.0) 1282 : 2021-03-24T22:00:00.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster [WRN] pg 5.c is stuck undersized for 87s, current state active+undersized, last acting [4,0] 2021-03-24T22:00:00.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:00:00.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: -03-24T22:00:00. 2021-03-24T22:00:00.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 001080+0000 mon.a (mon.0) 1283 : 2021-03-24T22:00:00.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster [WRN] pg 5.e is stuck undersized for 87s, current state active+undersized+degraded, last acting [4,0] 2021-03-24T22:00:00.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:00:00.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: -03-24T22:00:00.001104 2021-03-24T22:00:00.403 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: +0000 mon.a (mon.0) 1284 : 2021-03-24T22:00:00.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster [WRN] pg 5.f is stuck undersized for 87s, current state active+undersized, last acting [4,2] 2021-03-24T22:00:00.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021-03-24T22:00:00.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 2021-03-24T22:00:00.001134 2021-03-24T22:00:00.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: +0000 mon.a (mon.0) 1285 : cluster [WRN] pg 5.13 is stuck undersized for 87s, current state active+undersized, last acting [3,0] 2021-03-24T22:00:00.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:00:00.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: -03-24T22:00:00. 2021-03-24T22:00:00.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 001159+0000 mon.a (mon.0) 1286 : cluster 2021-03-24T22:00:00.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: [WRN] pg 5.15 is stuck undersized for 87s, current state active+undersized+degraded, last acting [0,3] 2021-03-24T22:00:00.405 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021-03-24T22:00:00.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 2021-03-24T22:00:00 2021-03-24T22:00:00.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: .001184+0000 mon.a (mon.0) 1287 2021-03-24T22:00:00.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: : cluster [WRN] pg 5.1d is stuck undersized for 87s, current state active+undersized, last acting [4,0] 2021-03-24T22:00:00.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:00:00.406 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: -03-24T22:00:00.001207 2021-03-24T22:00:00.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: +0000 mon.a (mon.0) 1288 : cluster 2021-03-24T22:00:00.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: [WRN] pg 6.7 is stuck undersized for 87s, current state active+undersized, last acting [3,4] 2021-03-24T22:00:00.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:00:00.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 03-24T22:00:00.001233 2021-03-24T22:00:00.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: +0000 mon.a (mon.0) 1289 : cluster [WRN] pg 6.a is stuck undersized for 87s, current state active+undersized, last acting [0,3] 2021-03-24T22:00:00.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:00:00.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: T22:00:00.001259+0000 mon.a ( 2021-03-24T22:00:00.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: mon.0) 1290 : cluster [WRN] 2021-03-24T22:00:00.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: pg 6.b is stuck undersized for 87s, current state active+undersized, last acting [3,4] 2021-03-24T22:00:00.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T22:00:00.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: -24T22:00:00.001283+0000 2021-03-24T22:00:00.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: mon.a (mon.0) 1291 : cluster [WRN] 2021-03-24T22:00:00.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: pg 6.d is stuck undersized for 87s, current state active+undersized, last acting [0,2] 2021-03-24T22:00:00.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:00:00.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 24T22:00:00.001308+0000 mon.a 2021-03-24T22:00:00.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: (mon.0) 1292 : cluster [WRN] 2021-03-24T22:00:00.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: pg 6.f is stuck undersized for 87s, current state active+undersized, last acting [3,4] 2021-03-24T22:00:00.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021-03-24T22:00:00.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 2021-03-24T22:00:00.001328+0000 2021-03-24T22:00:00.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: mon.a (mon.0) 1293 : cluster [WRN] 2021-03-24T22:00:00.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: pg 6.10 is stuck undersized for 87s, current state active+undersized, last acting [0,3] 2021-03-24T22:00:00.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021-03-24T22:00:00.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 2021-03-24T22:00:00.001352+0000 mon.a (mon. 2021-03-24T22:00:00.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 0) 1294 : cluster [WRN] pg 6.11 is stuck undersized for 87s, current state active+undersized, last acting [3,0] 2021-03-24T22:00:00.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:00:00.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 24T22:00:00.001376+0000 mon.a 2021-03-24T22:00:00.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: (mon.0) 1295 : cluster [WRN] 2021-03-24T22:00:00.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: pg 6.17 is stuck undersized for 87s, current state active+undersized, last acting [4,2] 2021-03-24T22:00:00.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021-03-24T22:00:00.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 2021-03-24T22:00:00.001401 2021-03-24T22:00:00.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: +0000 mon.a (mon.0) 1296 : cluster 2021-03-24T22:00:00.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: [WRN] pg 6.1d is stuck undersized for 87s, current state active+undersized, last acting [4,2] 2021-03-24T22:00:00.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:00:00.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 00:00.001424+0000 mon.a (mon. 2021-03-24T22:00:00.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 0) 1297 : cluster [WRN] pg 6.1f is stuck undersized for 87s, current state active+undersized, last acting [3,2] 2021-03-24T22:00:00.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:00:00.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: -03-24T22:00:00.001448 2021-03-24T22:00:00.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: +0000 mon.a (mon.0) 1298 : cluster 2021-03-24T22:00:00.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: [WRN] pg 7.0 is stuck undersized for 87s, current state active+undersized, last acting [3,2] 2021-03-24T22:00:00.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021-03-24T22 2021-03-24T22:00:00.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: :00:00.001471+0000 mon.a ( 2021-03-24T22:00:00.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: mon.0) 1299 : cluster [WRN] pg 7.4 is stuck undersized for 87s, current state active+undersized+degraded, last acting [0,4] 2021-03-24T22:00:00.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021-03-24T22:00:00.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 2021-03-24T22:00:00. 2021-03-24T22:00:00.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 001495+0000 mon.a (mon.0) 1300 : 2021-03-24T22:00:00.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster [WRN] pg 7.6 is stuck undersized for 87s, current state active+undersized+degraded, last acting [0,4] 2021-03-24T22:00:00.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:00:00.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 24T22:00:00.001520+0000 2021-03-24T22:00:00.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: mon.a (mon.0) 1301 : cluster [WRN] 2021-03-24T22:00:00.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: pg 7.7 is active+recovery_wait+undersized+degraded+remapped, acting [0,3] 2021-03-24T22:00:00.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:00:00.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: T22:00:00.001544+0000 2021-03-24T22:00:00.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: mon.a (mon.0) 1302 : cluster [WRN] pg 7.a is stuck undersized for 87s, current state active+undersized+degraded, last acting [3,2] 2021-03-24T22:00:00.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021-03-24T22:00:00.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 2021 2021-03-24T22:00:00.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: - 2021-03-24T22:00:00.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 03 2021-03-24T22:00:00.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: - 2021-03-24T22:00:00.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 24 2021-03-24T22:00:00.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: T 2021-03-24T22:00:00.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 22 2021-03-24T22:00:00.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: : 2021-03-24T22:00:00.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 00 2021-03-24T22:00:00.421 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: : 2021-03-24T22:00:00.421 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 00 2021-03-24T22:00:00.421 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: . 2021-03-24T22:00:00.421 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 001569 2021-03-24T22:00:00.421 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: +0000 2021-03-24T22:00:00.422 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 2021-03-24T22:00:00.422 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: mon.a 2021-03-24T22:00:00.422 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: ( 2021-03-24T22:00:00.422 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: mon 2021-03-24T22:00:00.423 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: . 2021-03-24T22:00:00.423 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 0 2021-03-24T22:00:00.423 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: ) 2021-03-24T22:00:00.423 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 1303 2021-03-24T22:00:00.423 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: : 2021-03-24T22:00:00.424 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021-03-24T22:00:00.424 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 2021-03-24T22:00:00.424 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: [WRN] 2021-03-24T22:00:00.424 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 2021-03-24T22:00:00.424 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: pg 7.b is stuck undersized for 87s, current state active+undersized+degraded, last acting [4,0] 2021-03-24T22:00:00.425 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 2021-03-24T22:00:00.425 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021-03-24T22:00:00.425 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 2021-03-24T22:00:00.425 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 2021 2021-03-24T22:00:00.425 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: - 2021-03-24T22:00:00.426 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 03 2021-03-24T22:00:00.426 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: - 2021-03-24T22:00:00.426 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 24 2021-03-24T22:00:00.426 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: T 2021-03-24T22:00:00.427 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 22 2021-03-24T22:00:00.427 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: : 2021-03-24T22:00:00.427 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 00 2021-03-24T22:00:00.427 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: : 2021-03-24T22:00:00.427 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 00 2021-03-24T22:00:00.428 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: . 2021-03-24T22:00:00.428 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 001590 2021-03-24T22:00:00.428 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: +0000 2021-03-24T22:00:00.428 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 2021-03-24T22:00:00.428 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: mon.a 2021-03-24T22:00:00.429 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: ( 2021-03-24T22:00:00.429 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: mon 2021-03-24T22:00:00.429 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: . 2021-03-24T22:00:00.429 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 0 2021-03-24T22:00:00.430 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: ) 2021-03-24T22:00:00.430 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 1304 2021-03-24T22:00:00.430 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: : 2021-03-24T22:00:00.430 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021-03-24T22:00:00.430 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 2021-03-24T22:00:00.431 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: [WRN] 2021-03-24T22:00:00.431 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 2021-03-24T22:00:00.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:00.432 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:00.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:00:00.534 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:00.580 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: pg 7.c is active+recovery_wait+undersized+degraded+remapped, acting [3,2] 2021-03-24T22:00:00.580 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 2021-03-24T22:00:00.580 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021-03-24T22:00:00.580 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 2021-03-24T22:00:00.581 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 2021 2021-03-24T22:00:00.581 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: - 2021-03-24T22:00:00.581 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 03 2021-03-24T22:00:00.581 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: - 2021-03-24T22:00:00.582 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 24 2021-03-24T22:00:00.582 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: T 2021-03-24T22:00:00.582 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 22 2021-03-24T22:00:00.582 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: : 2021-03-24T22:00:00.582 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 00 2021-03-24T22:00:00.583 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: : 2021-03-24T22:00:00.583 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 00 2021-03-24T22:00:00.583 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: . 2021-03-24T22:00:00.583 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 001614 2021-03-24T22:00:00.584 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: +0000 2021-03-24T22:00:00.584 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 2021-03-24T22:00:00.584 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: mon.a 2021-03-24T22:00:00.584 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: ( 2021-03-24T22:00:00.584 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: mon 2021-03-24T22:00:00.585 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: . 2021-03-24T22:00:00.585 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 0 2021-03-24T22:00:00.585 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: ) 2021-03-24T22:00:00.585 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 1305 2021-03-24T22:00:00.585 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: : 2021-03-24T22:00:00.586 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021-03-24T22:00:00.586 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 2021-03-24T22:00:00.586 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: [WRN] 2021-03-24T22:00:00.586 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 2021-03-24T22:00:00.587 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: pg 7.e is stuck undersized for 87s, current state active+undersized+degraded, last acting [2,0] 2021-03-24T22:00:00.587 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 2021-03-24T22:00:00.587 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021-03-24T22:00:00.587 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 2021-03-24T22:00:00.587 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 2021 2021-03-24T22:00:00.588 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: - 2021-03-24T22:00:00.588 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 03 2021-03-24T22:00:00.588 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: - 2021-03-24T22:00:00.588 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 24 2021-03-24T22:00:00.588 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: T 2021-03-24T22:00:00.589 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 22 2021-03-24T22:00:00.589 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: : 2021-03-24T22:00:00.589 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 00 2021-03-24T22:00:00.589 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: : 2021-03-24T22:00:00.589 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 00 2021-03-24T22:00:00.590 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: . 2021-03-24T22:00:00.590 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 001640 2021-03-24T22:00:00.590 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: +0000 2021-03-24T22:00:00.590 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 2021-03-24T22:00:00.591 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: mon.a 2021-03-24T22:00:00.591 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: ( 2021-03-24T22:00:00.591 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: mon 2021-03-24T22:00:00.591 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: . 2021-03-24T22:00:00.591 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 0 2021-03-24T22:00:00.592 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: ) 2021-03-24T22:00:00.592 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 1306 2021-03-24T22:00:00.592 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: : 2021-03-24T22:00:00.592 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021-03-24T22:00:00.593 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 2021-03-24T22:00:00.593 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: [WRN] 2021-03-24T22:00:00.593 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 2021-03-24T22:00:00.593 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: pg 7.13 is active+recovery_wait+undersized+degraded+remapped, acting [0,7] 2021-03-24T22:00:00.593 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 2021-03-24T22:00:00.594 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021-03-24T22:00:00.594 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 2021-03-24T22:00:00.594 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 2021 2021-03-24T22:00:00.594 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: - 2021-03-24T22:00:00.594 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 03 2021-03-24T22:00:00.595 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: - 2021-03-24T22:00:00.595 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 24 2021-03-24T22:00:00.595 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: T 2021-03-24T22:00:00.595 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 22 2021-03-24T22:00:00.595 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: : 2021-03-24T22:00:00.596 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 00 2021-03-24T22:00:00.596 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: : 2021-03-24T22:00:00.596 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 00 2021-03-24T22:00:00.596 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: . 2021-03-24T22:00:00.597 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 001664 2021-03-24T22:00:00.597 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: +0000 2021-03-24T22:00:00.597 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 2021-03-24T22:00:00.597 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: mon.a 2021-03-24T22:00:00.597 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: ( 2021-03-24T22:00:00.598 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: mon 2021-03-24T22:00:00.598 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: . 2021-03-24T22:00:00.598 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 0 2021-03-24T22:00:00.599 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: ( 2021-03-24T22:00:00.599 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: mon 2021-03-24T22:00:00.599 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: . 2021-03-24T22:00:00.599 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 0 2021-03-24T22:00:00.600 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: ) 2021-03-24T22:00:00.600 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 1252 2021-03-24T22:00:00.600 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: : 2021-03-24T22:00:00.600 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.601 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 2021-03-24T22:00:00.601 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: [WRN] 2021-03-24T22:00:00.601 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 2021-03-24T22:00:00.601 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:00:00.601 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 2021-03-24T22:00:00.602 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: audit 2021-03-24T22:00:00.602 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 2021-03-24T22:00:00.602 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 2021 2021-03-24T22:00:00.602 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: - 2021-03-24T22:00:00.602 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 03 2021-03-24T22:00:00.603 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: - 2021-03-24T22:00:00.603 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 24 2021-03-24T22:00:00.603 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: T 2021-03-24T22:00:00.603 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 21 2021-03-24T22:00:00.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: : 2021-03-24T22:00:00.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 59 2021-03-24T22:00:00.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: : 2021-03-24T22:00:00.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 59 2021-03-24T22:00:00.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: . 2021-03-24T22:00:00.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 293445 2021-03-24T22:00:00.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: +0000 2021-03-24T22:00:00.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 2021-03-24T22:00:00.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: mon.a 2021-03-24T22:00:00.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: ( 2021-03-24T22:00:00.606 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: mon 2021-03-24T22:00:00.606 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: . 2021-03-24T22:00:00.606 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 0 2021-03-24T22:00:00.606 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: ) 2021-03-24T22:00:00.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 1253 2021-03-24T22:00:00.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: : 2021-03-24T22:00:00.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: audit 2021-03-24T22:00:00.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 2021-03-24T22:00:00.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: [INF] 2021-03-24T22:00:00.608 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 2021-03-24T22:00:00.608 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: from='client.35231 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T22:00:00.608 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 2021-03-24T22:00:00.608 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.608 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 2021-03-24T22:00:00.609 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 2021 2021-03-24T22:00:00.609 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: - 2021-03-24T22:00:00.609 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 03 2021-03-24T22:00:00.609 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: - 2021-03-24T22:00:00.609 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 24 2021-03-24T22:00:00.610 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: T 2021-03-24T22:00:00.610 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 21 2021-03-24T22:00:00.610 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: : 2021-03-24T22:00:00.610 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 59 2021-03-24T22:00:00.611 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: : 2021-03-24T22:00:00.611 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 59 2021-03-24T22:00:00.611 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: . 2021-03-24T22:00:00.611 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 293508 2021-03-24T22:00:00.612 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: +0000 2021-03-24T22:00:00.612 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 2021-03-24T22:00:00.612 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: mon.a 2021-03-24T22:00:00.612 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: ( 2021-03-24T22:00:00.612 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: mon 2021-03-24T22:00:00.613 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: . 2021-03-24T22:00:00.613 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 0 2021-03-24T22:00:00.613 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: ) 2021-03-24T22:00:00.613 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 1254 2021-03-24T22:00:00.613 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: : 2021-03-24T22:00:00.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 2021-03-24T22:00:00.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: [DBG] 2021-03-24T22:00:00.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 2021-03-24T22:00:00.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: osdmap e429: 8 total, 7 up, 5 in 2021-03-24T22:00:00.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 2021-03-24T22:00:00.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: audit 2021-03-24T22:00:00.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 2021-03-24T22:00:00.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 2021 2021-03-24T22:00:00.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: - 2021-03-24T22:00:00.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 03 2021-03-24T22:00:00.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: - 2021-03-24T22:00:00.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 24 2021-03-24T22:00:00.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: T 2021-03-24T22:00:00.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 21 2021-03-24T22:00:00.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: : 2021-03-24T22:00:00.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 59 2021-03-24T22:00:00.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: : 2021-03-24T22:00:00.618 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 59 2021-03-24T22:00:00.618 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: ) 2021-03-24T22:00:00.618 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 1307 2021-03-24T22:00:00.619 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: : 2021-03-24T22:00:00.619 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021-03-24T22:00:00.619 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 2021-03-24T22:00:00.619 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: [WRN] 2021-03-24T22:00:00.620 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 2021-03-24T22:00:00.620 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: pg 7.14 is stuck undersized for 87s, current state active+undersized+degraded, last acting [3,2] 2021-03-24T22:00:00.620 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 2021-03-24T22:00:00.620 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021-03-24T22:00:00.620 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 2021-03-24T22:00:00.621 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 2021 2021-03-24T22:00:00.621 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: - 2021-03-24T22:00:00.621 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 03 2021-03-24T22:00:00.621 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: - 2021-03-24T22:00:00.621 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 24 2021-03-24T22:00:00.622 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: T 2021-03-24T22:00:00.622 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 22 2021-03-24T22:00:00.622 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: : 2021-03-24T22:00:00.622 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 00 2021-03-24T22:00:00.623 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: : 2021-03-24T22:00:00.623 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 00 2021-03-24T22:00:00.623 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: . 2021-03-24T22:00:00.623 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: 001687 2021-03-24T22:00:00.623 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: +0000 mon.a (mon.0) 1308 : cluster [WRN] pg 7.1d is active+recovery_wait+undersized+degraded+remapped, acting [0,7] 2021-03-24T22:00:00.624 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021-03-24T22:00:00.001711+0000 mon.a (mon.0) 1309 : cluster [WRN] pg 7.1e is stuck undersized for 87s, current state active+undersized, last acting [3,4] 2021-03-24T22:00:00.624 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021-03-24T22:00:00.001735+0000 mon.a (mon.0) 1310 : cluster [WRN] pg 7.22 is stuck undersized for 87s, current state active+undersized+degraded, last acting [4,0] 2021-03-24T22:00:00.624 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021-03-24T22:00:00.001758+0000 mon.a (mon.0) 1311 : cluster [WRN] pg 7.23 is stuck undersized for 87s, current state active+undersized+degraded, last acting [3,4] 2021-03-24T22:00:00.624 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021-03-24T22:00:00.216825+0000 mon.a (mon.0) 1312 : cluster [WRN] Health check update: Degraded data redundancy: 240/1545 objects degraded (15.534%), 28 pgs degraded, 47 pgs undersized (PG_DEGRADED) 2021-03-24T22:00:00.624 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[19936]: cluster 2021-03-24T22:00:00.223507+0000 mon.a (mon.0) 1313 : cluster [DBG] osdmap e430: 8 total, 7 up, 5 in 2021-03-24T22:00:00.626 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: . 2021-03-24T22:00:00.626 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 546748 2021-03-24T22:00:00.626 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: +0000 2021-03-24T22:00:00.626 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 2021-03-24T22:00:00.627 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: mon.a 2021-03-24T22:00:00.627 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: ( 2021-03-24T22:00:00.627 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: mon 2021-03-24T22:00:00.627 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: . 2021-03-24T22:00:00.627 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 0 2021-03-24T22:00:00.628 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: ) 2021-03-24T22:00:00.628 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 1255 2021-03-24T22:00:00.628 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: : 2021-03-24T22:00:00.628 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: audit 2021-03-24T22:00:00.629 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 2021-03-24T22:00:00.629 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: [INF] 2021-03-24T22:00:00.629 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 2021-03-24T22:00:00.629 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:00:00.629 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 2021-03-24T22:00:00.630 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.630 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 2021-03-24T22:00:00.630 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 2021 2021-03-24T22:00:00.630 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: - 2021-03-24T22:00:00.631 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 03 2021-03-24T22:00:00.631 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: - 2021-03-24T22:00:00.631 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 24 2021-03-24T22:00:00.631 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: T 2021-03-24T22:00:00.631 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 22 2021-03-24T22:00:00.632 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: : 2021-03-24T22:00:00.632 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 00 2021-03-24T22:00:00.632 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: : 2021-03-24T22:00:00.632 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 00 2021-03-24T22:00:00.632 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: . 2021-03-24T22:00:00.633 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 000227 2021-03-24T22:00:00.633 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: +0000 2021-03-24T22:00:00.633 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 2021-03-24T22:00:00.633 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: mon.a 2021-03-24T22:00:00.634 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: ( 2021-03-24T22:00:00.634 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: mon 2021-03-24T22:00:00.634 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: . 2021-03-24T22:00:00.634 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 0 2021-03-24T22:00:00.634 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: ) 2021-03-24T22:00:00.635 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 1256 2021-03-24T22:00:00.635 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: : 2021-03-24T22:00:00.635 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.635 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 2021-03-24T22:00:00.635 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: [WRN] 2021-03-24T22:00:00.636 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 2021-03-24T22:00:00.636 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: Health detail: HEALTH_WARN noscrub flag(s) set; 1 osds down; Degraded data redundancy: 240/1545 objects degraded (15.534%), 28 pgs degraded, 47 pgs undersized 2021-03-24T22:00:00.636 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 2021-03-24T22:00:00.636 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.637 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 2021-03-24T22:00:00.637 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 2021 2021-03-24T22:00:00.637 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: - 2021-03-24T22:00:00.637 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 03 2021-03-24T22:00:00.637 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: - 2021-03-24T22:00:00.638 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 24 2021-03-24T22:00:00.638 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: T 2021-03-24T22:00:00.638 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 22 2021-03-24T22:00:00.638 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: : 2021-03-24T22:00:00.638 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 00 2021-03-24T22:00:00.639 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: : 2021-03-24T22:00:00.639 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 00 2021-03-24T22:00:00.639 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: . 2021-03-24T22:00:00.639 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 000296 2021-03-24T22:00:00.640 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: +0000 2021-03-24T22:00:00.640 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 2021-03-24T22:00:00.640 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: mon.a (mon.0) 1257 : cluster [WRN] [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2021-03-24T22:00:00.640 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.000331+0000 mon.a (mon.0) 1258 : cluster [WRN] [WRN] OSD_DOWN: 1 osds down 2021-03-24T22:00:00.640 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.000362+0000 2021-03-24T22:00:00.641 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: mon.a (mon.0) 1259 : cluster [WRN] osd.5 (root=default,host=smithi142) is down 2021-03-24T22:00:00.641 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.000388+0000 mon.a (mon.0) 1260 : cluster [WRN] [WRN] PG_DEGRADED: Degraded data redundancy: 240/1545 objects degraded (15.534%), 28 pgs degraded, 47 pgs undersized 2021-03-24T22:00:00.641 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.000422+0000 mon.a (mon.0) 1261 : cluster [WRN] pg 2.2 is stuck undersized for 87s, current state active+undersized, last acting [0,3] 2021-03-24T22:00:00.641 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.000453+0000 mon.a (mon.0) 1262 : cluster [WRN] pg 3.a is stuck undersized for 87s, current state active+undersized, last acting [4,3] 2021-03-24T22:00:00.641 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.000485+0000 mon.a (mon.0) 1263 : cluster [WRN] pg 3.c is stuck undersized for 87s, current state active+undersized, last acting [3,0] 2021-03-24T22:00:00.642 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.000531+0000 mon.a (mon.0) 1264 : cluster [WRN] pg 3.10 is stuck undersized for 87s, current state active+undersized, last acting [0,3] 2021-03-24T22:00:00.642 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.000558+0000 mon.a (mon.0) 1265 : cluster [WRN] pg 3.17 is stuck undersized for 87s, current state active+undersized, last acting [0,3] 2021-03-24T22:00:00.642 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.000594+0000 mon.a (mon.0) 1266 : cluster [WRN] pg 3.19 is stuck undersized for 87s, current state active+undersized, last acting [3,4] 2021-03-24T22:00:00.642 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.000627+0000 mon.a (mon.0) 1267 : cluster [WRN] pg 4.1 is stuck undersized for 87s, current state active+undersized+degraded, last acting [4,0] 2021-03-24T22:00:00.643 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.000657+0000 mon.a (mon.0) 1268 : cluster [WRN] pg 4.2 is stuck undersized for 87s, current state active+undersized+degraded, last acting [4,0] 2021-03-24T22:00:00.643 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.000685+0000 mon.a (mon.0) 1269 : cluster [WRN] pg 4.6 is active+recovery_wait+degraded+remapped, acting [0,4,7] 2021-03-24T22:00:00.643 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.000714+0000 mon.a (mon.0) 1270 : cluster [WRN] pg 4.7 is active+recovering+degraded+remapped, acting [0,3,7] 2021-03-24T22:00:00.643 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.000740+0000 mon.a (mon.0) 1271 : cluster [WRN] pg 4.e is stuck undersized for 87s, current state active+undersized+degraded, last acting [4,3] 2021-03-24T22:00:00.643 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.000772+0000 mon.a (mon.0) 1272 : cluster [WRN] pg 4.10 is stuck undersized for 87s, current state active+undersized+degraded, last acting [3,4] 2021-03-24T22:00:00.644 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.000800+0000 mon.a (mon.0) 1273 : cluster [WRN] pg 4.12 is stuck undersized for 87s, current state active+undersized+degraded, last acting [2,0] 2021-03-24T22:00:00.644 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.000829+0000 mon.a (mon.0) 1274 : cluster [WRN] pg 4.13 is stuck undersized for 87s, current state active+undersized+degraded, last acting [4,2] 2021-03-24T22:00:00.644 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.000861+0000 mon.a (mon.0) 1275 : cluster [WRN] pg 4.15 is active+recovery_wait+degraded+remapped, acting [3,0,7] 2021-03-24T22:00:00.644 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.000894+0000 mon.a (mon.0) 1276 : cluster [WRN] pg 4.16 is active+recovery_wait+degraded+remapped, acting [0,4,7] 2021-03-24T22:00:00.645 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.000930+0000 mon.a (mon.0) 1277 : cluster [WRN] pg 4.17 is active+recovery_wait+degraded+remapped, acting [0,3,7] 2021-03-24T22:00:00.645 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.000964+0000 mon.a (mon.0) 1278 : cluster [WRN] pg 5.2 is stuck undersized for 87s, current state active+undersized, last acting [0,2] 2021-03-24T22:00:00.645 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.001004+0000 mon.a (mon.0) 1279 : cluster [WRN] pg 5.3 is stuck undersized for 87s, current state active+undersized+degraded, last acting [0,3] 2021-03-24T22:00:00.645 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.001033+0000 mon.a (mon.0) 1280 : cluster [WRN] pg 5.7 is stuck undersized for 87s, current state active+undersized, last acting [0,4] 2021-03-24T22:00:00.645 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:00:00.646 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: 24T22:00:00.001049+0000 mon.a (mon.0) 1281 : cluster [WRN] pg 5.8 is stuck undersized for 87s, current state active+undersized, last acting [2,0] 2021-03-24T22:00:00.646 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.001062+0000 mon.a (mon.0) 1282 : cluster [WRN] pg 5.c is stuck undersized for 87s, current state active+undersized, last acting [4,0] 2021-03-24T22:00:00.646 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.001080+0000 mon.a (mon.0) 1283 : cluster [WRN] pg 5.e is stuck undersized for 87s, current state active+undersized+degraded, last acting [4,0] 2021-03-24T22:00:00.646 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.001104+0000 mon.a (mon.0) 1284 : cluster [WRN] pg 5.f is stuck undersized for 87s, current state active+undersized, last acting [4,2] 2021-03-24T22:00:00.646 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.001134+0000 mon.a (mon.0) 1285 : cluster [WRN] pg 5.13 is stuck undersized for 87s, current state active+undersized, last acting [3,0] 2021-03-24T22:00:00.647 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.001159+0000 mon.a (mon.0) 1286 : cluster [WRN] pg 5.15 is stuck undersized for 87s, current state active+undersized+degraded, last acting [0,3] 2021-03-24T22:00:00.647 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.001184+0000 mon.a (mon.0) 1287 : cluster [WRN] pg 5.1d is stuck undersized for 87s, current state active+undersized, last acting [4,0] 2021-03-24T22:00:00.647 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.001207+0000 mon.a (mon.0) 1288 : cluster [WRN] pg 6.7 is stuck undersized for 87s, current state active+undersized, last acting [3,4] 2021-03-24T22:00:00.647 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.001233+0000 mon.a (mon.0) 1289 : cluster [WRN] pg 6.a is stuck undersized for 87s, current state active+undersized, last acting [0,3] 2021-03-24T22:00:00.648 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.001259+0000 mon.a (mon.0) 1290 : cluster [WRN] pg 6.b is stuck undersized for 87s, current state active+undersized, last acting [3,4] 2021-03-24T22:00:00.648 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.001283+0000 mon.a (mon.0) 1291 : cluster [WRN] pg 6.d is stuck undersized for 87s, current state active+undersized, last acting [0,2] 2021-03-24T22:00:00.648 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.001308+0000 mon.a (mon.0) 1292 : cluster [WRN] pg 6.f is stuck undersized for 87s, current state active+undersized, last acting [3,4] 2021-03-24T22:00:00.648 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.001328+0000 mon.a (mon.0) 1293 : cluster [WRN] pg 6.10 is stuck undersized for 87s, current state active+undersized, last acting [0,3] 2021-03-24T22:00:00.648 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.001352+0000 mon.a (mon.0) 1294 : cluster [WRN] pg 6.11 is stuck undersized for 87s, current state active+undersized, last acting [3,0] 2021-03-24T22:00:00.649 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.001376+0000 mon.a (mon.0) 1295 : cluster [WRN] pg 6.17 is stuck undersized for 87s, current state active+undersized, last acting [4,2] 2021-03-24T22:00:00.649 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.001401+0000 mon.a (mon.0) 1296 : cluster [WRN] pg 6.1d is stuck undersized for 87s, current state active+undersized, last acting [4,2] 2021-03-24T22:00:00.649 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.001424+0000 mon.a (mon.0) 1297 : cluster [WRN] pg 6.1f is stuck undersized for 87s, current state active+undersized, last acting [3,2] 2021-03-24T22:00:00.649 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.001448+0000 mon.a (mon.0) 1298 : cluster [WRN] pg 7.0 is stuck undersized for 87s, current state active+undersized, last acting [3,2] 2021-03-24T22:00:00.649 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.001471+0000 mon.a (mon.0) 1299 : cluster [WRN] pg 7.4 is stuck undersized for 87s, current state active+undersized+degraded, last acting [0,4] 2021-03-24T22:00:00.650 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.001495+0000 mon.a (mon.0) 1300 : cluster [WRN] pg 7.6 is stuck undersized for 87s, current state active+undersized+degraded, last acting [0,4] 2021-03-24T22:00:00.650 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.001520+0000 mon.a (mon.0) 1301 : cluster [WRN] pg 7.7 is active+recovery_wait+undersized+degraded+remapped, acting [0,3] 2021-03-24T22:00:00.650 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.001544+0000 mon.a (mon.0) 1302 : cluster [WRN] pg 7.a is stuck undersized for 87s, current state active+undersized+degraded, last acting [3,2] 2021-03-24T22:00:00.650 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.001569+0000 mon.a (mon.0) 1303 : cluster [WRN] pg 7.b is stuck undersized for 87s, current state active+undersized+degraded, last acting [4,0] 2021-03-24T22:00:00.651 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.001590+0000 mon.a (mon.0) 1304 : cluster [WRN] pg 7.c is active+recovery_wait+undersized+degraded+remapped, acting [3,2] 2021-03-24T22:00:00.651 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.001614+0000 mon.a (mon.0) 1305 : cluster [WRN] pg 7.e is stuck undersized for 87s, current state active+undersized+degraded, last acting [2,0] 2021-03-24T22:00:00.651 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.001640+0000 mon.a (mon.0) 1306 : cluster [WRN] pg 7.13 is active+recovery_wait+undersized+degraded+remapped, acting [0,7] 2021-03-24T22:00:00.651 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.001664+0000 mon.a (mon.0) 1307 : cluster [WRN] pg 7.14 is stuck undersized for 87s, current state active+undersized+degraded, last acting [3,2] 2021-03-24T22:00:00.651 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.001687+0000 mon.a (mon.0) 1308 : cluster [WRN] pg 7.1d is active+recovery_wait+undersized+degraded+remapped, acting [0,7] 2021-03-24T22:00:00.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.001711+0000 mon.a (mon.0) 1309 : cluster [WRN] pg 7.1e is stuck undersized for 87s, current state active+undersized, last acting [3,4] 2021-03-24T22:00:00.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.001735+0000 mon.a (mon.0) 1310 : cluster [WRN] pg 7.22 is stuck undersized for 87s, current state active+undersized+degraded, last acting [4,0] 2021-03-24T22:00:00.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.001758+0000 mon.a (mon.0) 1311 : cluster [WRN] pg 7.23 is stuck undersized for 87s, current state active+undersized+degraded, last acting [3,4] 2021-03-24T22:00:00.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.216825+0000 mon.a (mon.0) 1312 : cluster [WRN] Health check update: Degraded data redundancy: 240/1545 objects degraded (15.534%), 28 pgs degraded, 47 pgs undersized (PG_DEGRADED) 2021-03-24T22:00:00.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:00.223507+0000 mon.a (mon.0) 1313 : cluster [DBG] osdmap e430: 8 total, 7 up, 5 in 2021-03-24T22:00:00.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:00.654 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:00.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:00.655 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03-24T22:00 2021-03-24T22:00:00.656 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: :00.000829+0000 mon.a (mon.0) 1274 2021-03-24T22:00:00.656 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: : cluster [WRN] pg 4.13 is stuck undersized for 87s, current state active+undersized+degraded, last acting [4,2] 2021-03-24T22:00:00.656 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:00:00.656 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: T22:00:00.000861+0000 mon.a ( 2021-03-24T22:00:00.656 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: mon.0) 1275 : cluster [WRN] pg 4.15 is active+recovery_wait+degraded+remapped, acting [3,0,7] 2021-03-24T22:00:00.657 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03-24T22:00:00.657 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 2021-03-24T 2021-03-24T22:00:00.657 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 22:00:00.000894+0000 mon.a (mon 2021-03-24T22:00:00.657 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: .0) 1276 : cluster [WRN] pg 4.16 is active+recovery_wait+degraded+remapped, acting [0,4,7] 2021-03-24T22:00:00.658 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:00:00.658 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 03-24T22:00:00.000930+0000 2021-03-24T22:00:00.658 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: mon.a (mon.0) 1277 : cluster [WRN] 2021-03-24T22:00:00.658 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: pg 4.17 is active+recovery_wait+degraded+remapped, acting [0,3,7] 2021-03-24T22:00:00.659 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03-24T22:00:00. 2021-03-24T22:00:00.659 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 000964+0000 mon.a (mon.0) 1278 : cluster 2021-03-24T22:00:00.659 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: [WRN] pg 5.2 is stuck undersized for 87s, current state active+undersized, last acting [0,2] 2021-03-24T22:00:00.659 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03-24T22:00 2021-03-24T22:00:00.659 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: :00.001004+0000 mon.a (mon.0) 1279 2021-03-24T22:00:00.660 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: : cluster [WRN] pg 5.3 is stuck undersized for 87s, current state active+undersized+degraded, last acting [0,3] 2021-03-24T22:00:00.660 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:00:00.660 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: T22:00:00.001033+0000 mon.a (mon 2021-03-24T22:00:00.660 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: .0) 1280 : cluster [WRN] pg 5.7 is stuck undersized for 87s, current state active+undersized, last acting [0,4] 2021-03-24T22:00:00.660 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:00:00.661 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: -03-24T22:00:00.001049+0000 2021-03-24T22:00:00.661 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: mon.a (mon.0) 1281 : cluster [WRN] 2021-03-24T22:00:00.661 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: pg 5.8 is stuck undersized for 87s, current state active+undersized, last acting [2,0] 2021-03-24T22:00:00.661 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03-24T22:00 2021-03-24T22:00:00.662 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: :00.001062+0000 mon.a (mon.0) 2021-03-24T22:00:00.662 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 1282 : cluster [WRN] pg 5.c is stuck undersized for 87s, current state active+undersized, last acting [4,0] 2021-03-24T22:00:00.662 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:00:00.662 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 24T22:00:00.001080+0000 mon.a ( 2021-03-24T22:00:00.663 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: mon.0) 1283 : cluster [WRN] pg 5.e is stuck undersized for 87s, current state active+undersized+degraded, last acting [4,0] 2021-03-24T22:00:00.663 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:00:00.663 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 03-24T22:00:00.001104+0000 2021-03-24T22:00:00.663 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: mon.a (mon.0) 1284 : cluster [WRN] 2021-03-24T22:00:00.663 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: pg 5.f is stuck undersized for 87s, current state active+undersized, last acting [4,2] 2021-03-24T22:00:00.664 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03-24T22:00 2021-03-24T22:00:00.664 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: :00.001134+0000 mon.a (mon.0) 2021-03-24T22:00:00.664 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 1285 : cluster [WRN] pg 5.13 is stuck undersized for 87s, current state active+undersized, last acting [3,0] 2021-03-24T22:00:00.664 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:00:00.665 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 24T22:00:00.001159+0000 mon.a 2021-03-24T22:00:00.665 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: (mon.0) 1286 : cluster [WRN] pg 5.15 is stuck undersized for 87s, current state active+undersized+degraded, last acting [0,3] 2021-03-24T22:00:00.665 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:00:00.665 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 03-24T22:00:00.001184+0000 2021-03-24T22:00:00.665 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: mon.a (mon.0) 1287 : cluster [WRN] 2021-03-24T22:00:00.666 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: pg 5.1d is stuck undersized for 87s, current state active+undersized, last acting [4,0] 2021-03-24T22:00:00.666 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:00:00.666 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: -24T22:00:00.001207+0000 mon.a (mon.0 2021-03-24T22:00:00.666 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: ) 1288 : cluster [WRN] pg 6.7 is stuck undersized for 87s, current state active+undersized, last acting [3,4] 2021-03-24T22:00:00.667 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:00:00.667 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 22:00:00.001233+0000 mon.a (mon 2021-03-24T22:00:00.667 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: .0) 1289 : cluster [WRN] pg 6.a is stuck undersized for 87s, current state active+undersized, last acting [0,3] 2021-03-24T22:00:00.667 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03-24T22:00:00.667 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 2021-03-24T22:00:00. 2021-03-24T22:00:00.668 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 001259+0000 mon.a (mon.0) 1290 : cluster 2021-03-24T22:00:00.668 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: [WRN] pg 6.b is stuck undersized for 87s, current state active+undersized, last acting [3,4] 2021-03-24T22:00:00.668 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:00:00.668 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: T22:00:00.001283+0000 mon.a ( 2021-03-24T22:00:00.668 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: mon.0) 1291 : cluster [WRN] pg 6.d is stuck undersized for 87s, current state active+undersized, last acting [0,2] 2021-03-24T22:00:00.669 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03-24T22:00:00.669 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 2021-03-24T22:00:00. 2021-03-24T22:00:00.669 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 001308+0000 mon.a (mon.0) 1292 : cluster 2021-03-24T22:00:00.669 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: [WRN] pg 6.f is stuck undersized for 87s, current state active+undersized, last acting [3,4] 2021-03-24T22:00:00.670 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:00:00.670 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 24T22:00:00.001328+0000 mon.a ( 2021-03-24T22:00:00.670 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: mon.0) 1293 : cluster [WRN] pg 6.10 is stuck undersized for 87s, current state active+undersized, last acting [0,3] 2021-03-24T22:00:00.670 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03-24T22:00:00.670 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 2021-03-24T22:00:00 2021-03-24T22:00:00.671 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: .001352+0000 mon.a (mon.0) 1294 : 2021-03-24T22:00:00.671 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster [WRN] pg 6.11 is stuck undersized for 87s, current state active+undersized, last acting [3,0] 2021-03-24T22:00:00.671 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:00:00.671 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 22:00:00.001376+0000 mon.a (mon. 2021-03-24T22:00:00.671 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 0) 1295 : cluster [WRN] pg 6.17 is stuck undersized for 87s, current state active+undersized, last acting [4,2] 2021-03-24T22:00:00.672 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:00:00.672 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 03-24T22:00:00.001401+0000 2021-03-24T22:00:00.672 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: mon.a (mon.0) 1296 : cluster [WRN] 2021-03-24T22:00:00.672 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: pg 6.1d is stuck undersized for 87s, current state active+undersized, last acting [4,2] 2021-03-24T22:00:00.673 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03-24T22:00:00.001424+0000 2021-03-24T22:00:00.673 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: mon.a (mon.0) 1297 : cluster [WRN] 2021-03-24T22:00:00.673 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: pg 6.1f is stuck undersized for 87s, current state active+undersized, last acting [3,2] 2021-03-24T22:00:00.673 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03-24T22:00:00.673 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 2021-03-24T22:00: 2021-03-24T22:00:00.674 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 00.001448+0000 mon.a (mon.0) 1298 2021-03-24T22:00:00.674 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: : cluster [WRN] pg 7.0 is stuck undersized for 87s, current state active+undersized, last acting [3,2] 2021-03-24T22:00:00.674 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03-24T22:00:00.674 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 2021-03-24T22:00: 2021-03-24T22:00:00.675 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 00.001471+0000 mon.a (mon.0) 1299 2021-03-24T22:00:00.675 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: : cluster [WRN] pg 7.4 is stuck undersized for 87s, current state active+undersized+degraded, last acting [0,4] 2021-03-24T22:00:00.675 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03-24T22 2021-03-24T22:00:00.675 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: :00:00.001495+0000 mon.a (mon.0 2021-03-24T22:00:00.675 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: ) 1300 : cluster [WRN] pg 7.6 is stuck undersized for 87s, current state active+undersized+degraded, last acting [0,4] 2021-03-24T22:00:00.676 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03-24T22:00:00.676 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 2021-03-24T22:00:00 2021-03-24T22:00:00.676 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: .001520+0000 mon.a (mon.0) 1301 : cluster 2021-03-24T22:00:00.676 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: [WRN] pg 7.7 is active+recovery_wait+undersized+degraded+remapped, acting [0,3] 2021-03-24T22:00:00.677 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03-24T22 2021-03-24T22:00:00.677 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: :00:00.001544+0000 mon.a (mon. 2021-03-24T22:00:00.677 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 0) 1302 : cluster [WRN] pg 7.a is stuck undersized for 87s, current state active+undersized+degraded, last acting [3,2] 2021-03-24T22:00:00.677 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03-24T22:00:00.677 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 2021-03-24T22: 2021-03-24T22:00:00.678 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 00:00.001569+0000 mon.a (mon.0 2021-03-24T22:00:00.678 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: ) 1303 : cluster [WRN] pg 7.b is stuck undersized for 87s, current state active+undersized+degraded, last acting [4,0] 2021-03-24T22:00:00.678 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:00:00.678 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: -03-24T22:00:00.001590 2021-03-24T22:00:00.678 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: +0000 mon.a (mon.0) 1304 : cluster 2021-03-24T22:00:00.679 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: [WRN] pg 7.c is active+recovery_wait+undersized+degraded+remapped, acting [3,2] 2021-03-24T22:00:00.679 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:00:00.679 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: 24T22:00:00.001614+0000 mon.a (mon.0) 1305 : cluster [WRN] pg 7.e is stuck undersized for 87s, current state active+undersized+degraded, last acting [2,0] 2021-03-24T22:00:00.679 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03-24T22:00:00.001640+0000 mon.a (mon.0) 1306 : cluster [WRN] pg 7.13 is active+recovery_wait+undersized+degraded+remapped, acting [0,7] 2021-03-24T22:00:00.680 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03-24T22:00:00.001664+0000 mon.a (mon.0) 1307 : cluster [WRN] pg 7.14 is stuck undersized for 87s, current state active+undersized+degraded, last acting [3,2] 2021-03-24T22:00:00.680 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03-24T22:00:00.001687+0000 mon.a (mon.0) 1308 : cluster [WRN] pg 7.1d is active+recovery_wait+undersized+degraded+remapped, acting [0,7] 2021-03-24T22:00:00.680 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03-24T22:00:00.001711+0000 mon.a (mon.0) 1309 : cluster [WRN] pg 7.1e is stuck undersized for 87s, current state active+undersized, last acting [3,4] 2021-03-24T22:00:00.680 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03-24T22:00:00.001735+0000 mon.a (mon.0) 1310 : cluster [WRN] pg 7.22 is stuck undersized for 87s, current state active+undersized+degraded, last acting [4,0] 2021-03-24T22:00:00.680 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03-24T22:00:00.001758+0000 mon.a (mon.0) 1311 : cluster [WRN] pg 7.23 is stuck undersized for 87s, current state active+undersized+degraded, last acting [3,4] 2021-03-24T22:00:00.681 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03-24T22:00:00.216825+0000 mon.a (mon.0) 1312 : cluster [WRN] Health check update: Degraded data redundancy: 240/1545 objects degraded (15.534%), 28 pgs degraded, 47 pgs undersized (PG_DEGRADED) 2021-03-24T22:00:00.681 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[20125]: cluster 2021-03-24T22:00:00.223507+0000 mon.a (mon.0) 1313 : cluster [DBG] osdmap e430: 8 total, 7 up, 5 in 2021-03-24T22:00:00.756 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:00.810 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:00.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:00.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:00.866 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 22:00:00 smithi142 conmon[66800]: debug 2021-03-24T22:00:00.600+0000 7f22596f7080 -1 osd.5 339 log_to_monitors {default=true} 2021-03-24T22:00:00.916 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:00.977 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:00.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:00.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:01.083 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:01.137 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:01.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:01.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:00:01.238 INFO:tasks.rados.rados.0.smithi125.stdout:1588: done (1 left) 2021-03-24T22:00:01.238 INFO:tasks.rados.rados.0.smithi125.stdout:1589: write oid 27 current snap is 173 2021-03-24T22:00:01.239 INFO:tasks.rados.rados.0.smithi125.stdout:1589: seq_num 562 ranges {645534=783707,2103939=638517,3150663=294871} 2021-03-24T22:00:01.244 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:01.253 INFO:tasks.rados.rados.0.smithi125.stdout:1589: writing smithi12556470-27 from 645534 to 1429241 tid 1 2021-03-24T22:00:01.262 INFO:tasks.rados.rados.0.smithi125.stdout:1589: writing smithi12556470-27 from 2103939 to 2742456 tid 2 2021-03-24T22:00:01.271 INFO:tasks.rados.rados.0.smithi125.stdout:1589: writing smithi12556470-27 from 3150663 to 3445534 tid 3 2021-03-24T22:00:01.272 INFO:tasks.rados.rados.0.smithi125.stdout:1583: finishing copy_from to smithi12556470-20 2021-03-24T22:00:01.273 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 20 v 449 (ObjNum 462 snap 129 seq_num 462) dirty exists 2021-03-24T22:00:01.273 INFO:tasks.rados.rados.0.smithi125.stdout:1583: finishing copy_from racing read to smithi12556470-20 2021-03-24T22:00:01.273 INFO:tasks.rados.rados.0.smithi125.stdout:1583: done (1 left) 2021-03-24T22:00:01.273 INFO:tasks.rados.rados.0.smithi125.stdout:1590: rollback oid 1 current snap is 173 2021-03-24T22:00:01.274 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 1 to 136 2021-03-24T22:00:01.274 INFO:tasks.rados.rados.0.smithi125.stdout:1591: read oid 46 snap -1 2021-03-24T22:00:01.274 INFO:tasks.rados.rados.0.smithi125.stdout:1591: expect (ObjNum 497 snap 145 seq_num 497) 2021-03-24T22:00:01.274 INFO:tasks.rados.rados.0.smithi125.stdout:1592: write oid 18 current snap is 173 2021-03-24T22:00:01.274 INFO:tasks.rados.rados.0.smithi125.stdout:1592: seq_num 563 ranges {662341=575309,1878774=498528,2894439=457555} 2021-03-24T22:00:01.287 INFO:tasks.rados.rados.0.smithi125.stdout:1592: writing smithi12556470-18 from 662341 to 1237650 tid 1 2021-03-24T22:00:01.295 INFO:tasks.rados.rados.0.smithi125.stdout:1592: writing smithi12556470-18 from 1878774 to 2377302 tid 2 2021-03-24T22:00:01.302 INFO:tasks.rados.rados.0.smithi125.stdout:1592: writing smithi12556470-18 from 2894439 to 3351994 tid 3 2021-03-24T22:00:01.305 INFO:tasks.rados.rados.0.smithi125.stdout:1590: finishing rollback tid 0 to smithi12556470-1 2021-03-24T22:00:01.305 INFO:tasks.rados.rados.0.smithi125.stdout:1590: finishing rollback tid 1 to smithi12556470-1 2021-03-24T22:00:01.306 INFO:tasks.rados.rados.0.smithi125.stdout:1591: expect (ObjNum 497 snap 145 seq_num 497) 2021-03-24T22:00:01.361 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:01.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:01.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:00:01.466 INFO:tasks.rados.rados.0.smithi125.stdout:1590: finishing rollback tid 2 to smithi12556470-1 2021-03-24T22:00:01.528 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 1 v 507 (ObjNum 473 snap 134 seq_num 473) dirty exists 2021-03-24T22:00:01.529 INFO:tasks.rados.rados.0.smithi125.stdout:1589: finishing write tid 1 to smithi12556470-27 2021-03-24T22:00:01.529 INFO:tasks.rados.rados.0.smithi125.stdout:1589: finishing write tid 2 to smithi12556470-27 2021-03-24T22:00:01.530 INFO:tasks.rados.rados.0.smithi125.stdout:1589: finishing write tid 3 to smithi12556470-27 2021-03-24T22:00:01.530 INFO:tasks.rados.rados.0.smithi125.stdout:1589: finishing write tid 4 to smithi12556470-27 2021-03-24T22:00:01.530 INFO:tasks.rados.rados.0.smithi125.stdout:1589: finishing write tid 5 to smithi12556470-27 2021-03-24T22:00:01.531 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 27 v 547 (ObjNum 562 snap 173 seq_num 562) dirty exists 2021-03-24T22:00:01.531 INFO:tasks.rados.rados.0.smithi125.stdout:1589: left oid 27 (ObjNum 562 snap 173 seq_num 562) 2021-03-24T22:00:01.531 INFO:tasks.rados.rados.0.smithi125.stdout:1592: finishing write tid 1 to smithi12556470-18 2021-03-24T22:00:01.531 INFO:tasks.rados.rados.0.smithi125.stdout:1592: finishing write tid 2 to smithi12556470-18 2021-03-24T22:00:01.532 INFO:tasks.rados.rados.0.smithi125.stdout:1592: finishing write tid 3 to smithi12556470-18 2021-03-24T22:00:01.532 INFO:tasks.rados.rados.0.smithi125.stdout:1592: finishing write tid 4 to smithi12556470-18 2021-03-24T22:00:01.532 INFO:tasks.rados.rados.0.smithi125.stdout:1592: finishing write tid 5 to smithi12556470-18 2021-03-24T22:00:01.532 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 18 v 191 (ObjNum 563 snap 173 seq_num 563) dirty exists 2021-03-24T22:00:01.532 INFO:tasks.rados.rados.0.smithi125.stdout:1592: left oid 18 (ObjNum 563 snap 173 seq_num 563) 2021-03-24T22:00:01.533 INFO:tasks.rados.rados.0.smithi125.stdout:1589: done (3 left) 2021-03-24T22:00:01.533 INFO:tasks.rados.rados.0.smithi125.stdout:1590: done (2 left) 2021-03-24T22:00:01.533 INFO:tasks.rados.rados.0.smithi125.stdout:1591: done (1 left) 2021-03-24T22:00:01.533 INFO:tasks.rados.rados.0.smithi125.stdout:1592: done (0 left) 2021-03-24T22:00:01.534 INFO:tasks.rados.rados.0.smithi125.stdout:1593: read oid 37 snap -1 2021-03-24T22:00:01.534 INFO:tasks.rados.rados.0.smithi125.stdout:1593: expect (ObjNum 537 snap 166 seq_num 537) 2021-03-24T22:00:01.534 INFO:tasks.rados.rados.0.smithi125.stdout:1594: write oid 12 current snap is 173 2021-03-24T22:00:01.534 INFO:tasks.rados.rados.0.smithi125.stdout:1594: seq_num 564 ranges {679148=650558,1937256=642186,3321862=157286} 2021-03-24T22:00:01.535 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:01.537 INFO:tasks.rados.rados.0.smithi125.stdout:1594: writing smithi12556470-12 from 679148 to 1329706 tid 1 2021-03-24T22:00:01.538 INFO:tasks.rados.rados.0.smithi125.stdout:1594: writing smithi12556470-12 from 1937256 to 2579442 tid 2 2021-03-24T22:00:01.538 INFO:tasks.rados.rados.0.smithi125.stdout:1594: writing smithi12556470-12 from 3321862 to 3479148 tid 3 2021-03-24T22:00:01.539 INFO:tasks.rados.rados.0.smithi125.stdout:1595: read oid 32 snap -1 2021-03-24T22:00:01.539 INFO:tasks.rados.rados.0.smithi125.stdout:1595: expect (ObjNum 427 snap 113 seq_num 427) 2021-03-24T22:00:01.539 INFO:tasks.rados.rados.0.smithi125.stdout:1596: write oid 6 current snap is 173 2021-03-24T22:00:01.539 INFO:tasks.rados.rados.0.smithi125.stdout:1596: seq_num 565 ranges {695955=725807,1879385=502197,2949285=480099} 2021-03-24T22:00:01.540 INFO:tasks.rados.rados.0.smithi125.stdout:1596: writing smithi12556470-6 from 695955 to 1421762 tid 1 2021-03-24T22:00:01.540 INFO:tasks.rados.rados.0.smithi125.stdout:1596: writing smithi12556470-6 from 1879385 to 2381582 tid 2 2021-03-24T22:00:01.540 INFO:tasks.rados.rados.0.smithi125.stdout:1596: writing smithi12556470-6 from 2949285 to 3429384 tid 3 2021-03-24T22:00:01.540 INFO:tasks.rados.rados.0.smithi125.stdout:1597: snap_create 2021-03-24T22:00:01.541 INFO:tasks.rados.rados.0.smithi125.stdout:1593: expect (ObjNum 537 snap 166 seq_num 537) 2021-03-24T22:00:01.550 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[24927]: audit 2021-03-24T22:00:00.608636 2021-03-24T22:00:01.551 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[24927]: +0000 mon.a (mon.0) 1314 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-03-24T22:00:01.551 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[24927]: audit 2021-03 2021-03-24T22:00:01.551 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[24927]: -24T22:00:00.610252+0000 mon.b (mon.1) 35 : 2021-03-24T22:00:01.551 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[24927]: audit [INF] from='osd.5 v1:172.21.15.142:6804/3992668882' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-03-24T22:00:01.552 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[24927]: audit 2021-03-24T22:00: 2021-03-24T22:00:01.552 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[24927]: 01.234634+0000 mon.a (mon.0) 1315 2021-03-24T22:00:01.552 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[24927]: : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2021-03-24T22:00:01.552 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:00:01.553 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[24927]: 00:01.234860+0000 mon.a (mon.0 2021-03-24T22:00:01.553 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[24927]: ) 1316 : cluster [DBG] osdmap e431: 8 total, 7 up, 5 in 2021-03-24T22:00:01.553 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[24927]: audit 2021-03-24T 2021-03-24T22:00:01.553 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[24927]: 22:00:01.237744+0000 mon.a (mon 2021-03-24T22:00:01.553 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[24927]: .0) 1317 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2021-03-24T22:00:01.554 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T22:00:01.554 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[24927]: 24T22:00:01.239095+0000 mon.b 2021-03-24T22:00:01.554 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[24927]: (mon.1) 36 : audit [INF] from='osd.5 v1:172.21.15.142:6804/3992668882' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2021-03-24T22:00:01.555 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[24927]: 2021-03-24T22:00:01.556 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: audit 2021-03-24T22:00:01.556 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 2021-03-24T22:00:01.556 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 2021 2021-03-24T22:00:01.557 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: - 2021-03-24T22:00:01.557 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 03 2021-03-24T22:00:01.557 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: - 2021-03-24T22:00:01.557 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 24 2021-03-24T22:00:01.557 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: T 2021-03-24T22:00:01.558 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 22 2021-03-24T22:00:01.558 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: : 2021-03-24T22:00:01.558 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 00 2021-03-24T22:00:01.558 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: : 2021-03-24T22:00:01.559 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 00 2021-03-24T22:00:01.559 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: . 2021-03-24T22:00:01.559 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 608636 2021-03-24T22:00:01.560 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: +0000 2021-03-24T22:00:01.560 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 2021-03-24T22:00:01.560 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: mon.a 2021-03-24T22:00:01.561 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: ( 2021-03-24T22:00:01.561 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: mon 2021-03-24T22:00:01.562 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: . 2021-03-24T22:00:01.562 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 0 2021-03-24T22:00:01.562 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: ) 2021-03-24T22:00:01.563 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 1314 2021-03-24T22:00:01.563 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: : 2021-03-24T22:00:01.564 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: audit 2021-03-24T22:00:01.564 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 2021-03-24T22:00:01.564 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: [INF] 2021-03-24T22:00:01.565 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 2021-03-24T22:00:01.565 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-03-24T22:00:01.566 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 2021-03-24T22:00:01.566 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: audit 2021-03-24T22:00:01.567 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 2021-03-24T22:00:01.567 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 2021 2021-03-24T22:00:01.567 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: - 2021-03-24T22:00:01.568 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 03 2021-03-24T22:00:01.568 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: - 2021-03-24T22:00:01.569 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 24 2021-03-24T22:00:01.569 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: T 2021-03-24T22:00:01.570 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 22 2021-03-24T22:00:01.570 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: : 2021-03-24T22:00:01.570 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 00 2021-03-24T22:00:01.571 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: : 2021-03-24T22:00:01.571 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 00 2021-03-24T22:00:01.572 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: . 2021-03-24T22:00:01.572 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 610252 2021-03-24T22:00:01.572 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: +0000 2021-03-24T22:00:01.573 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 2021-03-24T22:00:01.573 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: mon.b 2021-03-24T22:00:01.574 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: ( 2021-03-24T22:00:01.574 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: mon 2021-03-24T22:00:01.575 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: . 2021-03-24T22:00:01.575 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 1 2021-03-24T22:00:01.575 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: ) 2021-03-24T22:00:01.576 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 35 2021-03-24T22:00:01.576 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: : 2021-03-24T22:00:01.577 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: audit 2021-03-24T22:00:01.577 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 2021-03-24T22:00:01.578 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: [INF] 2021-03-24T22:00:01.578 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 2021-03-24T22:00:01.578 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: from='osd.5 v1:172.21.15.142:6804/3992668882' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-03-24T22:00:01.579 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 2021-03-24T22:00:01.579 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: audit 2021-03-24T22:00:01.580 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 2021-03-24T22:00:01.580 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 2021 2021-03-24T22:00:01.580 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: - 2021-03-24T22:00:01.581 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 03 2021-03-24T22:00:01.581 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: - 2021-03-24T22:00:01.582 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 24 2021-03-24T22:00:01.582 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: T 2021-03-24T22:00:01.583 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 22 2021-03-24T22:00:01.583 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: : 2021-03-24T22:00:01.583 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 00 2021-03-24T22:00:01.584 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: : 2021-03-24T22:00:01.584 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 01 2021-03-24T22:00:01.585 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: . 2021-03-24T22:00:01.585 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 234634 2021-03-24T22:00:01.586 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: +0000 2021-03-24T22:00:01.586 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 2021-03-24T22:00:01.586 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: mon.a 2021-03-24T22:00:01.587 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: ( 2021-03-24T22:00:01.587 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: mon 2021-03-24T22:00:01.588 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: . 2021-03-24T22:00:01.588 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 0 2021-03-24T22:00:01.588 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: ) 2021-03-24T22:00:01.589 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 1315 2021-03-24T22:00:01.589 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: : 2021-03-24T22:00:01.590 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: audit 2021-03-24T22:00:01.590 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 2021-03-24T22:00:01.591 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: [INF] 2021-03-24T22:00:01.591 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 2021-03-24T22:00:01.591 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2021-03-24T22:00:01.592 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 2021-03-24T22:00:01.592 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: cluster 2021-03-24T22:00:01.593 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 2021-03-24T22:00:01.593 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 2021 2021-03-24T22:00:01.594 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: - 2021-03-24T22:00:01.594 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 03 2021-03-24T22:00:01.594 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: - 2021-03-24T22:00:01.595 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 24 2021-03-24T22:00:01.595 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: T 2021-03-24T22:00:01.596 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 22 2021-03-24T22:00:01.596 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: : 2021-03-24T22:00:01.597 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 00 2021-03-24T22:00:01.597 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: : 2021-03-24T22:00:01.597 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 01 2021-03-24T22:00:01.598 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: . 2021-03-24T22:00:01.598 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 234860 2021-03-24T22:00:01.599 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: +0000 2021-03-24T22:00:01.599 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 2021-03-24T22:00:01.600 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: mon.a 2021-03-24T22:00:01.600 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: ( 2021-03-24T22:00:01.600 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: mon 2021-03-24T22:00:01.601 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: . 2021-03-24T22:00:01.601 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 0 2021-03-24T22:00:01.602 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: ) 2021-03-24T22:00:01.602 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 1316 2021-03-24T22:00:01.602 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: : 2021-03-24T22:00:01.603 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: cluster 2021-03-24T22:00:01.603 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 2021-03-24T22:00:01.604 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: [DBG] 2021-03-24T22:00:01.604 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 2021-03-24T22:00:01.605 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: osdmap e431: 8 total, 7 up, 5 in 2021-03-24T22:00:01.605 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 2021-03-24T22:00:01.605 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: audit 2021-03-24T22:00:01.606 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 2021-03-24T22:00:01.606 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 2021 2021-03-24T22:00:01.607 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: - 2021-03-24T22:00:01.607 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 03 2021-03-24T22:00:01.607 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: - 2021-03-24T22:00:01.608 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 24 2021-03-24T22:00:01.608 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: T 2021-03-24T22:00:01.609 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 22 2021-03-24T22:00:01.609 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: : 2021-03-24T22:00:01.609 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 00 2021-03-24T22:00:01.610 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: : 2021-03-24T22:00:01.610 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 01 2021-03-24T22:00:01.611 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: . 2021-03-24T22:00:01.611 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 237744 2021-03-24T22:00:01.612 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: +0000 2021-03-24T22:00:01.612 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 2021-03-24T22:00:01.612 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: mon.a 2021-03-24T22:00:01.613 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: ( 2021-03-24T22:00:01.613 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: mon 2021-03-24T22:00:01.614 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: . 2021-03-24T22:00:01.614 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 0 2021-03-24T22:00:01.614 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: ) 2021-03-24T22:00:01.615 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 1317 2021-03-24T22:00:01.615 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: : 2021-03-24T22:00:01.616 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: audit 2021-03-24T22:00:01.616 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 2021-03-24T22:00:01.617 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: [INF] 2021-03-24T22:00:01.617 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 2021-03-24T22:00:01.617 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2021-03-24T22:00:01.618 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 2021-03-24T22:00:01.618 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: audit 2021-03-24T22:00:01.619 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 2021-03-24T22:00:01.619 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 2021 2021-03-24T22:00:01.620 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: - 2021-03-24T22:00:01.620 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 03 2021-03-24T22:00:01.620 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: - 2021-03-24T22:00:01.621 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 24 2021-03-24T22:00:01.621 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: T 2021-03-24T22:00:01.622 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 22 2021-03-24T22:00:01.622 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: : 2021-03-24T22:00:01.623 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 00 2021-03-24T22:00:01.623 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: : 2021-03-24T22:00:01.623 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 01 2021-03-24T22:00:01.624 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: . 2021-03-24T22:00:01.624 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 239095 2021-03-24T22:00:01.624 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: +0000 2021-03-24T22:00:01.625 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 2021-03-24T22:00:01.625 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: mon.b 2021-03-24T22:00:01.625 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: ( 2021-03-24T22:00:01.625 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: mon 2021-03-24T22:00:01.625 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: . 2021-03-24T22:00:01.626 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 1 2021-03-24T22:00:01.626 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: ) 2021-03-24T22:00:01.626 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 36 2021-03-24T22:00:01.626 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: : 2021-03-24T22:00:01.627 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: audit 2021-03-24T22:00:01.627 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 2021-03-24T22:00:01.627 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: [INF] 2021-03-24T22:00:01.627 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 2021-03-24T22:00:01.627 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: from='osd.5 v1:172.21.15.142:6804/3992668882' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2021-03-24T22:00:01.628 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:01 smithi125 conmon[19936]: 2021-03-24T22:00:01.629 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:01 smithi142 conmon[20125]: audit 2021-03-24T22:00:01.629 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:01 smithi142 conmon[20125]: 2021-03-24T22:00:01.629 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:01 smithi142 conmon[20125]: 2021 2021-03-24T22:00:01.629 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:01 smithi142 conmon[20125]: - 2021-03-24T22:00:01.630 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:01 smithi142 conmon[20125]: 03 2021-03-24T22:00:01.630 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:01 smithi142 conmon[20125]: - 2021-03-24T22:00:01.630 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:01 smithi142 conmon[20125]: 24 2021-03-24T22:00:01.630 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:01 smithi142 conmon[20125]: T 2021-03-24T22:00:01.630 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:01 smithi142 conmon[20125]: 22 2021-03-24T22:00:01.631 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:01 smithi142 conmon[20125]: : 2021-03-24T22:00:01.631 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:01 smithi142 conmon[20125]: 00 2021-03-24T22:00:01.631 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:01 smithi142 conmon[20125]: : 2021-03-24T22:00:01.631 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:01 smithi142 conmon[20125]: 00 2021-03-24T22:00:01.632 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:01 smithi142 conmon[20125]: . 2021-03-24T22:00:01.632 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:01 smithi142 conmon[20125]: 608636 2021-03-24T22:00:01.632 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:01 smithi142 conmon[20125]: +0000 mon.a (mon.0) 1314 : audit 2021-03-24T22:00:01.632 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:01 smithi142 conmon[20125]: [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-03-24T22:00:01.632 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:01 smithi142 conmon[20125]: audit 2021-03-24T22:00:00. 2021-03-24T22:00:01.633 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:01 smithi142 conmon[20125]: 610252+0000 mon.b (mon.1) 35 : audit 2021-03-24T22:00:01.633 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:01 smithi142 conmon[20125]: [INF] from='osd.5 v1:172.21.15.142:6804/3992668882' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-03-24T22:00:01.633 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:01 smithi142 conmon[20125]: audit 2021-03-24T22 2021-03-24T22:00:01.633 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:01 smithi142 conmon[20125]: :00:01.234634+0000 mon.a (mon.0) 2021-03-24T22:00:01.634 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:01 smithi142 conmon[20125]: 1315 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2021-03-24T22:00:01.634 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:01 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:00:01.634 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:01 smithi142 conmon[20125]: 22:00:01.234860+0000 mon.a (mon 2021-03-24T22:00:01.634 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:01 smithi142 conmon[20125]: .0) 1316 : cluster [DBG] osdmap e431: 8 total, 7 up, 5 in 2021-03-24T22:00:01.634 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:01 smithi142 conmon[20125]: audit 2021-03-24T22:00:01.237744+0000 mon.a 2021-03-24T22:00:01.635 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:01 smithi142 conmon[20125]: (mon.0) 1317 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2021-03-24T22:00:01.635 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:01 smithi142 conmon[20125]: audit 2021-03-24 2021-03-24T22:00:01.635 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:01 smithi142 conmon[20125]: T22:00:01.239095+0000 mon.b (mon. 2021-03-24T22:00:01.635 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:01 smithi142 conmon[20125]: 1) 36 : audit [INF] from='osd.5 v1:172.21.15.142:6804/3992668882' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2021-03-24T22:00:01.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:01.636 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:01.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:00:01.651 INFO:tasks.rados.rados.0.smithi125.stdout:1594: finishing write tid 1 to smithi12556470-12 2021-03-24T22:00:01.652 INFO:tasks.rados.rados.0.smithi125.stdout:1594: finishing write tid 2 to smithi12556470-12 2021-03-24T22:00:01.652 INFO:tasks.rados.rados.0.smithi125.stdout:1594: finishing write tid 3 to smithi12556470-12 2021-03-24T22:00:01.652 INFO:tasks.rados.rados.0.smithi125.stdout:1594: finishing write tid 4 to smithi12556470-12 2021-03-24T22:00:01.652 INFO:tasks.rados.rados.0.smithi125.stdout:1595: expect (ObjNum 427 snap 113 seq_num 427) 2021-03-24T22:00:01.738 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:01.799 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:01.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:01.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:00:01.841 DEBUG:teuthology.orchestra.run:got remote process result: 22 2021-03-24T22:00:01.842 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-5, ['dump_ops_in_flight'] 2021-03-24T22:00:01.849 INFO:tasks.rados.rados.0.smithi125.stdout:1594: finishing write tid 5 to smithi12556470-12 2021-03-24T22:00:01.850 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 12 v 384 (ObjNum 564 snap 173 seq_num 564) dirty exists 2021-03-24T22:00:01.850 INFO:tasks.rados.rados.0.smithi125.stdout:1594: left oid 12 (ObjNum 564 snap 173 seq_num 564) 2021-03-24T22:00:01.850 INFO:tasks.rados.rados.0.smithi125.stdout:1596: finishing write tid 1 to smithi12556470-6 2021-03-24T22:00:01.850 INFO:tasks.rados.rados.0.smithi125.stdout:1596: finishing write tid 2 to smithi12556470-6 2021-03-24T22:00:01.850 INFO:tasks.rados.rados.0.smithi125.stdout:1596: finishing write tid 3 to smithi12556470-6 2021-03-24T22:00:01.851 INFO:tasks.rados.rados.0.smithi125.stdout:1596: finishing write tid 4 to smithi12556470-6 2021-03-24T22:00:01.851 INFO:tasks.rados.rados.0.smithi125.stdout:1596: finishing write tid 5 to smithi12556470-6 2021-03-24T22:00:01.851 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 6 v 388 (ObjNum 565 snap 173 seq_num 565) dirty exists 2021-03-24T22:00:01.851 INFO:tasks.rados.rados.0.smithi125.stdout:1596: left oid 6 (ObjNum 565 snap 173 seq_num 565) 2021-03-24T22:00:01.903 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:01.961 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:01.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:01.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:00:02.067 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:02.130 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:02.132 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T22:00:02.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:02.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:00:02.151 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T22:00:02.172 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T22:00:02.191 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T22:00:02.216 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T22:00:02.236 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:02.247 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T22:00:02.247 INFO:tasks.rados.rados.0.smithi125.stdout:1593: done (4 left) 2021-03-24T22:00:02.248 INFO:tasks.rados.rados.0.smithi125.stdout:1594: done (3 left) 2021-03-24T22:00:02.248 INFO:tasks.rados.rados.0.smithi125.stdout:1595: done (2 left) 2021-03-24T22:00:02.248 INFO:tasks.rados.rados.0.smithi125.stdout:1596: done (1 left) 2021-03-24T22:00:02.248 INFO:tasks.rados.rados.0.smithi125.stdout:1597: done (0 left) 2021-03-24T22:00:02.249 INFO:tasks.rados.rados.0.smithi125.stdout:1598: write (excl) oid 19 current snap is 174 2021-03-24T22:00:02.249 INFO:tasks.rados.rados.0.smithi125.stdout:1598: seq_num 566 ranges {712762=401056,1537867=645855,2860355=652407} 2021-03-24T22:00:02.249 INFO:tasks.rados.rados.0.smithi125.stdout:1598: writing smithi12556470-19 from 712762 to 1113818 tid 1 2021-03-24T22:00:02.256 INFO:tasks.rados.rados.0.smithi125.stdout:1598: writing smithi12556470-19 from 1537867 to 2183722 tid 2 2021-03-24T22:00:02.265 INFO:tasks.rados.rados.0.smithi125.stdout:1598: writing smithi12556470-19 from 2860355 to 3512762 tid 3 2021-03-24T22:00:02.267 INFO:tasks.rados.rados.0.smithi125.stdout:1599: snap_remove snap 143 2021-03-24T22:00:02.268 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2021-03-24T22:00:02.268 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2021-03-24T22:00:02.268 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2021-03-24T22:00:02.269 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2021-03-24T22:00:02.269 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.269 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPushReply(7.1 110/108 [PushReplyOp(7:83e5d499:::smithi12556470-11:head)])", 2021-03-24T22:00:02.269 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:54:40.024517+0000", 2021-03-24T22:00:02.269 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 322.23560340900002, 2021-03-24T22:00:02.270 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.086802544999999995, 2021-03-24T22:00:02.270 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:02.270 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "reached pg", 2021-03-24T22:00:02.270 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:02.271 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.271 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:02.271 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.024517+0000", 2021-03-24T22:00:02.272 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:02.272 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.273 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.273 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:02.274 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.024517+0000", 2021-03-24T22:00:02.274 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2021-03-24T22:00:02.275 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.275 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.275 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:02.276 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.024515+0000", 2021-03-24T22:00:02.276 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3049999999999998e-06 2021-03-24T22:00:02.277 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.277 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.278 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:02.278 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.024522+0000", 2021-03-24T22:00:02.278 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0208999999999999e-05 2021-03-24T22:00:02.279 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.279 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.280 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:02.280 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.024542+0000", 2021-03-24T22:00:02.280 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9319999999999999e-06 2021-03-24T22:00:02.281 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.281 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.282 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:02.282 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.024549+0000", 2021-03-24T22:00:02.282 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.086712301000000006 2021-03-24T22:00:02.283 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.283 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.283 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:02.283 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.111261+0000", 2021-03-24T22:00:02.284 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7893000000000003e-05 2021-03-24T22:00:02.284 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.284 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.285 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:02.285 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.111319+0000", 2021-03-24T22:00:02.285 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.086802544999999995 2021-03-24T22:00:02.285 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:02.285 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:02.286 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:02.286 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.287 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.287 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPushReply(4.9 111/108 [PushReplyOp(4:94d01cf7:::data_log.108:head)])", 2021-03-24T22:00:02.287 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:54:40.207652+0000", 2021-03-24T22:00:02.287 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 322.05246772200002, 2021-03-24T22:00:02.287 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.089012543, 2021-03-24T22:00:02.288 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:02.288 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "reached pg", 2021-03-24T22:00:02.288 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:02.288 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.289 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:02.289 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.207652+0000", 2021-03-24T22:00:02.289 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:02.289 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.290 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.290 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:02.290 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.207652+0000", 2021-03-24T22:00:02.290 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2021-03-24T22:00:02.291 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.291 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.291 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:02.291 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.207650+0000", 2021-03-24T22:00:02.292 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.7899999999999994e-06 2021-03-24T22:00:02.292 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.292 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.292 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:02.292 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.207660+0000", 2021-03-24T22:00:02.293 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8369e-05 2021-03-24T22:00:02.293 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.293 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.293 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:02.294 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.207678+0000", 2021-03-24T22:00:02.294 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2442e-05 2021-03-24T22:00:02.294 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.294 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.294 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:02.295 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.207691+0000", 2021-03-24T22:00:02.295 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.088841322 2021-03-24T22:00:02.295 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.295 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.296 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:02.296 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.296532+0000", 2021-03-24T22:00:02.296 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013249 2021-03-24T22:00:02.296 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.297 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.297 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:02.297 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.296665+0000", 2021-03-24T22:00:02.297 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.089012543 2021-03-24T22:00:02.297 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:02.298 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:02.298 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:02.298 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.298 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.299 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPushReply(7.1 111/108 [PushReplyOp(7:86662589:::smithi12556470-19:head)])", 2021-03-24T22:00:02.299 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:54:40.297084+0000", 2021-03-24T22:00:02.299 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 321.96303550900001, 2021-03-24T22:00:02.299 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.090340742000000002, 2021-03-24T22:00:02.300 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:02.300 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "reached pg", 2021-03-24T22:00:02.300 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:02.300 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.300 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:02.301 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.297084+0000", 2021-03-24T22:00:02.301 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:02.301 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.301 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.302 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:02.302 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.297084+0000", 2021-03-24T22:00:02.302 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999971 2021-03-24T22:00:02.302 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.302 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.303 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:02.303 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.297082+0000", 2021-03-24T22:00:02.303 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3006999999999999e-05 2021-03-24T22:00:02.303 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.304 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.304 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:02.304 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.297095+0000", 2021-03-24T22:00:02.304 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8322000000000001e-05 2021-03-24T22:00:02.304 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.305 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.305 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:02.305 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.297123+0000", 2021-03-24T22:00:02.305 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9049999999999999e-05 2021-03-24T22:00:02.306 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.306 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.306 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:02.306 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.297142+0000", 2021-03-24T22:00:02.307 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.089791975999999996 2021-03-24T22:00:02.307 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.307 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.307 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:02.308 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.386934+0000", 2021-03-24T22:00:02.308 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00049109999999999996 2021-03-24T22:00:02.308 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.308 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.309 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:02.309 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.387425+0000", 2021-03-24T22:00:02.309 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.090340742000000002 2021-03-24T22:00:02.309 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:02.309 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:02.310 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:02.310 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.310 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.310 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPushReply(4.9 111/108 [PushReplyOp(4:921ecdb1:gc::gc.25:head)])", 2021-03-24T22:00:02.311 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:54:40.483015+0000", 2021-03-24T22:00:02.311 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 321.77710445100001, 2021-03-24T22:00:02.311 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.089336145000000006, 2021-03-24T22:00:02.311 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:02.312 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "reached pg", 2021-03-24T22:00:02.312 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:02.312 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.313 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:02.313 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.483015+0000", 2021-03-24T22:00:02.314 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:02.314 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.314 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.315 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:02.315 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.483015+0000", 2021-03-24T22:00:02.316 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999971 2021-03-24T22:00:02.316 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.316 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.317 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:02.317 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.483012+0000", 2021-03-24T22:00:02.318 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4007e-05 2021-03-24T22:00:02.318 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.318 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.319 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:02.319 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.483026+0000", 2021-03-24T22:00:02.320 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9105999999999999e-05 2021-03-24T22:00:02.320 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.320 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.321 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:02.321 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.483056+0000", 2021-03-24T22:00:02.322 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1919999999999999e-05 2021-03-24T22:00:02.322 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.322 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.323 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:02.323 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.483067+0000", 2021-03-24T22:00:02.324 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.089135437999999997 2021-03-24T22:00:02.324 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.324 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.325 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:02.325 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.572203+0000", 2021-03-24T22:00:02.325 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014872799999999999 2021-03-24T22:00:02.325 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.325 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.326 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:02.326 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.572352+0000", 2021-03-24T22:00:02.326 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.089336145000000006 2021-03-24T22:00:02.326 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:02.327 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:02.327 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:02.327 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.327 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.328 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPushReply(7.1 111/108 [PushReplyOp(7:8c1d349c:::smithi12556470-48:head)])", 2021-03-24T22:00:02.328 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:54:40.572536+0000", 2021-03-24T22:00:02.328 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 321.68758346800001, 2021-03-24T22:00:02.328 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.090618961999999997, 2021-03-24T22:00:02.329 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:02.329 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "reached pg", 2021-03-24T22:00:02.329 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:02.329 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.329 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:02.330 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.572536+0000", 2021-03-24T22:00:02.330 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:02.330 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.330 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.331 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:02.331 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.572536+0000", 2021-03-24T22:00:02.331 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999967 2021-03-24T22:00:02.331 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.331 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.332 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:02.332 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.572533+0000", 2021-03-24T22:00:02.332 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3864e-05 2021-03-24T22:00:02.332 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.333 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.333 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:02.333 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.572547+0000", 2021-03-24T22:00:02.333 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4148999999999999e-05 2021-03-24T22:00:02.333 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.334 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.334 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:02.334 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.572571+0000", 2021-03-24T22:00:02.334 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3868e-05 2021-03-24T22:00:02.335 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.335 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.335 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:02.335 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.572585+0000", 2021-03-24T22:00:02.336 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.090012964000000001 2021-03-24T22:00:02.336 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.336 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.336 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:02.336 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.662598+0000", 2021-03-24T22:00:02.337 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00055724599999999996 2021-03-24T22:00:02.337 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.337 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.337 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:02.338 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.663155+0000", 2021-03-24T22:00:02.338 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.090618961999999997 2021-03-24T22:00:02.338 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:02.338 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:02.339 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:02.339 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.339 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.339 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPushReply(4.9 111/108 [PushReplyOp(4:9268b73e:::obj_delete_at_hint.0000000019:head)])", 2021-03-24T22:00:02.340 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:54:40.758844+0000", 2021-03-24T22:00:02.340 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 321.501275728, 2021-03-24T22:00:02.340 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.089171401999999997, 2021-03-24T22:00:02.341 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:02.341 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "reached pg", 2021-03-24T22:00:02.341 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:02.342 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.342 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:02.342 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.758844+0000", 2021-03-24T22:00:02.342 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:02.343 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.343 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.343 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:02.343 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.758844+0000", 2021-03-24T22:00:02.343 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999967 2021-03-24T22:00:02.344 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.344 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.344 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:02.344 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.758841+0000", 2021-03-24T22:00:02.345 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7056000000000001e-05 2021-03-24T22:00:02.345 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.345 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.345 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:02.346 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.758858+0000", 2021-03-24T22:00:02.346 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8711e-05 2021-03-24T22:00:02.346 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.347 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.347 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:02.347 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.758887+0000", 2021-03-24T22:00:02.347 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5746000000000001e-05 2021-03-24T22:00:02.348 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.348 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.348 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:02.348 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.758902+0000", 2021-03-24T22:00:02.349 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.088984598999999998 2021-03-24T22:00:02.349 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.349 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.349 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:02.349 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.847887+0000", 2021-03-24T22:00:02.350 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000128644 2021-03-24T22:00:02.350 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.350 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.350 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:02.351 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.848016+0000", 2021-03-24T22:00:02.351 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.089171401999999997 2021-03-24T22:00:02.351 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:02.351 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:02.351 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:02.352 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.352 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.352 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPushReply(7.1 111/108 [PushReplyOp(7:8dd64f9e:::smithi12556470-49:head)])", 2021-03-24T22:00:02.352 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:54:40.847956+0000", 2021-03-24T22:00:02.353 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 321.412164007, 2021-03-24T22:00:02.353 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.090871619000000001, 2021-03-24T22:00:02.353 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:02.353 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "reached pg", 2021-03-24T22:00:02.353 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:02.354 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.354 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:02.354 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.847956+0000", 2021-03-24T22:00:02.354 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:02.355 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.355 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.355 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:02.355 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.847956+0000", 2021-03-24T22:00:02.355 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999967 2021-03-24T22:00:02.356 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.356 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.356 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:02.356 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.847953+0000", 2021-03-24T22:00:02.357 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5254e-05 2021-03-24T22:00:02.357 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.357 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.357 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:02.357 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.847968+0000", 2021-03-24T22:00:02.358 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9885e-05 2021-03-24T22:00:02.358 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.358 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.358 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:02.359 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.847988+0000", 2021-03-24T22:00:02.359 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1423999999999999e-05 2021-03-24T22:00:02.359 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.359 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.359 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:02.360 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.847999+0000", 2021-03-24T22:00:02.360 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.090269889000000006 2021-03-24T22:00:02.360 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.360 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.360 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:02.361 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.938269+0000", 2021-03-24T22:00:02.361 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00055842100000000003 2021-03-24T22:00:02.361 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.361 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.362 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:02.362 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.938828+0000", 2021-03-24T22:00:02.362 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.090871619000000001 2021-03-24T22:00:02.362 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:02.363 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:02.363 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:02.363 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.363 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.363 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPushReply(4.9 111/108 [PushReplyOp(4:94441385:::obj_delete_at_hint.0000000039:head)])", 2021-03-24T22:00:02.364 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:54:41.034565+0000", 2021-03-24T22:00:02.364 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 321.22555541200001, 2021-03-24T22:00:02.364 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.089107565999999999, 2021-03-24T22:00:02.364 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:02.365 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "reached pg", 2021-03-24T22:00:02.365 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:02.365 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.365 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:02.365 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:41.034565+0000", 2021-03-24T22:00:02.366 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:02.366 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.366 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.366 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:02.366 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:41.034565+0000", 2021-03-24T22:00:02.367 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2021-03-24T22:00:02.367 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.367 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.368 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:02.368 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:41.034564+0000", 2021-03-24T22:00:02.368 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.4819999999999999e-06 2021-03-24T22:00:02.369 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.369 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.369 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:02.369 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:41.034570+0000", 2021-03-24T22:00:02.370 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6356999999999999e-05 2021-03-24T22:00:02.370 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.370 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.370 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:02.371 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:41.034587+0000", 2021-03-24T22:00:02.371 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0502999999999999e-05 2021-03-24T22:00:02.371 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.371 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.371 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:02.372 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:41.034597+0000", 2021-03-24T22:00:02.372 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.088942807999999998 2021-03-24T22:00:02.372 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.372 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.373 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:02.373 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:41.123540+0000", 2021-03-24T22:00:02.373 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000132261 2021-03-24T22:00:02.373 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.374 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.374 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:02.374 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:41.123672+0000", 2021-03-24T22:00:02.374 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.089107565999999999 2021-03-24T22:00:02.374 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:02.375 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:02.375 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:02.375 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.375 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.376 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:1246 7.1 7:86662589:::smithi12556470-19:head [write 443751~631882] snapc 53=[] ondisk+write+known_if_redirected e191)", 2021-03-24T22:00:02.376 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:56:00.156445+0000", 2021-03-24T22:00:02.376 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 242.103675361, 2021-03-24T22:00:02.376 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.060689377000000003, 2021-03-24T22:00:02.376 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:02.377 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:00:02.377 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:00:02.377 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:00:02.377 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:00:02.377 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 1246 2021-03-24T22:00:02.378 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.378 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:02.378 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.378 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:02.379 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.156445+0000", 2021-03-24T22:00:02.379 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:02.379 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.379 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.379 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:02.380 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.156445+0000", 2021-03-24T22:00:02.380 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999952 2021-03-24T22:00:02.380 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.380 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.380 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:02.381 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.156440+0000", 2021-03-24T22:00:02.381 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0021389159999999998 2021-03-24T22:00:02.381 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.381 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.382 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:02.382 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.158579+0000", 2021-03-24T22:00:02.382 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6568999999999999e-05 2021-03-24T22:00:02.382 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.383 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.383 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:02.383 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.158605+0000", 2021-03-24T22:00:02.383 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4353e-05 2021-03-24T22:00:02.383 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.384 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.384 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:02.384 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.158620+0000", 2021-03-24T22:00:02.384 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010933 2021-03-24T22:00:02.385 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.385 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.385 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:02.385 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.158729+0000", 2021-03-24T22:00:02.385 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016208300000000001 2021-03-24T22:00:02.386 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.386 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.386 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:00:02.386 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.158891+0000", 2021-03-24T22:00:02.387 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00030850199999999999 2021-03-24T22:00:02.387 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.387 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.387 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 0,3", 2021-03-24T22:00:02.387 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.159200+0000", 2021-03-24T22:00:02.388 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.003189473 2021-03-24T22:00:02.388 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.388 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.388 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T22:00:02.388 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.162389+0000", 2021-03-24T22:00:02.389 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.049777185000000002 2021-03-24T22:00:02.389 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.389 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.389 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:02.390 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.212166+0000", 2021-03-24T22:00:02.390 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0048629060000000002 2021-03-24T22:00:02.390 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.390 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.391 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:02.391 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.217029+0000", 2021-03-24T22:00:02.391 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8270000000000003e-05 2021-03-24T22:00:02.391 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.392 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.392 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:00:02.392 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.217088+0000", 2021-03-24T22:00:02.392 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.6387000000000002e-05 2021-03-24T22:00:02.393 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.393 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.393 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:02.393 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.217134+0000", 2021-03-24T22:00:02.393 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.060689377000000003 2021-03-24T22:00:02.394 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:02.394 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:02.394 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:02.394 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.395 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.395 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:1247 7.1 7:86662589:::smithi12556470-19:head [write 1810907~765058] snapc 53=[] ondisk+write+known_if_redirected e191)", 2021-03-24T22:00:02.395 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:56:00.164197+0000", 2021-03-24T22:00:02.395 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 242.09592325, 2021-03-24T22:00:02.395 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.054173828, 2021-03-24T22:00:02.396 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:02.396 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:00:02.396 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:00:02.396 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:00:02.396 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:00:02.397 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 1247 2021-03-24T22:00:02.397 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.397 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:02.397 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.398 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:02.398 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.164197+0000", 2021-03-24T22:00:02.398 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:02.398 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.398 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.399 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:02.399 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.164197+0000", 2021-03-24T22:00:02.399 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999962 2021-03-24T22:00:02.399 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.400 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.400 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:02.400 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.164193+0000", 2021-03-24T22:00:02.400 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011806989999999999 2021-03-24T22:00:02.400 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.401 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.401 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:02.401 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.165374+0000", 2021-03-24T22:00:02.401 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0941000000000001e-05 2021-03-24T22:00:02.401 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.402 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.402 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:02.402 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.165405+0000", 2021-03-24T22:00:02.402 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3784999999999999e-05 2021-03-24T22:00:02.403 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.403 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.403 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:02.403 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.165418+0000", 2021-03-24T22:00:02.403 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000101259 2021-03-24T22:00:02.404 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.404 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.404 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:02.404 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.165520+0000", 2021-03-24T22:00:02.405 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015882 2021-03-24T22:00:02.405 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.405 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.405 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:00:02.405 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.165678+0000", 2021-03-24T22:00:02.406 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000318105 2021-03-24T22:00:02.406 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.406 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.406 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 0,3", 2021-03-24T22:00:02.407 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.165996+0000", 2021-03-24T22:00:02.407 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0042140550000000004 2021-03-24T22:00:02.407 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.408 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.408 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T22:00:02.408 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.170211+0000", 2021-03-24T22:00:02.408 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.042059725999999999 2021-03-24T22:00:02.408 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.409 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.409 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:02.409 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.212270+0000", 2021-03-24T22:00:02.409 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0060031010000000003 2021-03-24T22:00:02.409 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.410 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.410 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:02.410 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.218273+0000", 2021-03-24T22:00:02.410 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5608e-05 2021-03-24T22:00:02.411 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.411 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.411 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:00:02.411 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.218329+0000", 2021-03-24T22:00:02.412 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1533999999999999e-05 2021-03-24T22:00:02.412 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.412 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.412 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:02.412 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.218371+0000", 2021-03-24T22:00:02.413 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.054173828 2021-03-24T22:00:02.413 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:02.413 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:02.413 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:02.414 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.414 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.414 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:1248 7.1 7:86662589:::smithi12556470-19:head [write 3067461~176290] snapc 53=[] ondisk+write+known_if_redirected e191)", 2021-03-24T22:00:02.414 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:56:00.167908+0000", 2021-03-24T22:00:02.414 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 242.09221169, 2021-03-24T22:00:02.415 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.050562023999999997, 2021-03-24T22:00:02.415 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:02.415 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:00:02.415 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:00:02.416 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:00:02.416 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:00:02.416 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 1248 2021-03-24T22:00:02.416 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.416 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:02.417 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.417 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:02.417 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.167908+0000", 2021-03-24T22:00:02.417 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:02.418 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.418 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.418 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:02.418 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.167908+0000", 2021-03-24T22:00:02.418 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999971 2021-03-24T22:00:02.419 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.419 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.419 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:02.419 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.167905+0000", 2021-03-24T22:00:02.420 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015547299999999999 2021-03-24T22:00:02.420 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.420 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.420 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:02.420 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168061+0000", 2021-03-24T22:00:02.421 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9348000000000001e-05 2021-03-24T22:00:02.421 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.421 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.421 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:02.421 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168090+0000", 2021-03-24T22:00:02.422 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1722000000000001e-05 2021-03-24T22:00:02.422 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.422 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.422 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:02.423 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168102+0000", 2021-03-24T22:00:02.423 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00022216099999999999 2021-03-24T22:00:02.423 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.423 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.424 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:02.424 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168324+0000", 2021-03-24T22:00:02.424 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013274000000000001 2021-03-24T22:00:02.424 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.424 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.425 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:00:02.425 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168457+0000", 2021-03-24T22:00:02.425 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000219247 2021-03-24T22:00:02.425 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.425 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.426 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 0,3", 2021-03-24T22:00:02.426 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168676+0000", 2021-03-24T22:00:02.426 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0015439570000000001 2021-03-24T22:00:02.426 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.427 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.427 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T22:00:02.427 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.170220+0000", 2021-03-24T22:00:02.427 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.042126005000000001 2021-03-24T22:00:02.427 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.428 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.428 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:02.428 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.212346+0000", 2021-03-24T22:00:02.428 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0060575079999999996 2021-03-24T22:00:02.428 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.429 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.429 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:02.429 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.218403+0000", 2021-03-24T22:00:02.429 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7947000000000001e-05 2021-03-24T22:00:02.430 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.430 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.430 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:00:02.430 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.218441+0000", 2021-03-24T22:00:02.430 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8948000000000001e-05 2021-03-24T22:00:02.431 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.431 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.431 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:02.431 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.218470+0000", 2021-03-24T22:00:02.432 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.050562023999999997 2021-03-24T22:00:02.432 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:02.432 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:02.432 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:02.432 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.433 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.433 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:1249 7.1 7:86662589:::smithi12556470-19:head [setxattr _header (57) in=7b,truncate 3243751] snapc 53=[] ondisk+write+known_if_redirected e191)", 2021-03-24T22:00:02.433 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:56:00.168119+0000", 2021-03-24T22:00:02.433 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 242.09200089199999, 2021-03-24T22:00:02.434 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.052051300000000002, 2021-03-24T22:00:02.434 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:02.434 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:00:02.434 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:00:02.434 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:00:02.435 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:00:02.435 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 1249 2021-03-24T22:00:02.435 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.435 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:02.436 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.436 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:02.436 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168119+0000", 2021-03-24T22:00:02.436 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:02.436 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.437 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.437 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:02.437 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168119+0000", 2021-03-24T22:00:02.438 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2021-03-24T22:00:02.438 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.439 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.439 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:02.439 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168118+0000", 2021-03-24T22:00:02.439 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.0740000000000004e-06 2021-03-24T22:00:02.439 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.440 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.440 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:02.440 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168125+0000", 2021-03-24T22:00:02.440 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2374e-05 2021-03-24T22:00:02.441 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.441 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.441 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:02.441 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168137+0000", 2021-03-24T22:00:02.442 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.5490000000000003e-06 2021-03-24T22:00:02.442 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.442 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.442 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:02.443 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168144+0000", 2021-03-24T22:00:02.443 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011986869999999999 2021-03-24T22:00:02.443 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.443 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.443 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:02.444 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.169342+0000", 2021-03-24T22:00:02.444 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000160089 2021-03-24T22:00:02.444 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.444 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.444 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:00:02.445 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.169502+0000", 2021-03-24T22:00:02.445 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00031500400000000002 2021-03-24T22:00:02.445 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.445 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.446 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 0,3", 2021-03-24T22:00:02.446 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.169817+0000", 2021-03-24T22:00:02.447 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00070204499999999999 2021-03-24T22:00:02.447 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.447 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.447 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T22:00:02.448 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.170519+0000", 2021-03-24T22:00:02.448 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.041887362999999997 2021-03-24T22:00:02.448 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.448 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.449 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:02.449 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.212407+0000", 2021-03-24T22:00:02.449 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0076951889999999998 2021-03-24T22:00:02.449 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.449 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.450 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:02.450 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.220102+0000", 2021-03-24T22:00:02.450 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3169e-05 2021-03-24T22:00:02.450 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.451 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.451 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:00:02.451 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.220135+0000", 2021-03-24T22:00:02.451 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5286000000000001e-05 2021-03-24T22:00:02.451 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.452 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.452 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:02.452 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.220170+0000", 2021-03-24T22:00:02.452 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.052051300000000002 2021-03-24T22:00:02.452 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:02.453 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:02.453 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:02.453 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.454 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.454 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:1250 7.1 7:86662589:::smithi12556470-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e191)", 2021-03-24T22:00:02.454 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:56:00.168170+0000", 2021-03-24T22:00:02.454 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 242.09194959000001, 2021-03-24T22:00:02.454 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.052145710999999997, 2021-03-24T22:00:02.455 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:02.455 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:00:02.455 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:00:02.455 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:00:02.561 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:00:02.562 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 1250 2021-03-24T22:00:02.562 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.562 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:02.562 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.563 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:02.563 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168170+0000", 2021-03-24T22:00:02.563 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:02.563 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.564 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.564 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:02.564 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168170+0000", 2021-03-24T22:00:02.564 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2021-03-24T22:00:02.565 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.565 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.565 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:02.566 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168169+0000", 2021-03-24T22:00:02.566 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.5979999999999999e-06 2021-03-24T22:00:02.566 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.567 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.567 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:02.567 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168174+0000", 2021-03-24T22:00:02.568 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3064000000000001e-05 2021-03-24T22:00:02.568 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.568 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.568 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:02.569 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168187+0000", 2021-03-24T22:00:02.569 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9599999999999997e-06 2021-03-24T22:00:02.569 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.569 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.570 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:02.570 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168193+0000", 2021-03-24T22:00:02.570 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0021001650000000002 2021-03-24T22:00:02.571 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.571 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.571 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:02.571 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.170293+0000", 2021-03-24T22:00:02.571 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00019255100000000001 2021-03-24T22:00:02.572 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.572 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.572 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for rw locks", 2021-03-24T22:00:02.573 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.170485+0000", 2021-03-24T22:00:02.573 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.049706363000000003 2021-03-24T22:00:02.573 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.573 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.574 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:02.574 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.220192+0000", 2021-03-24T22:00:02.574 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9202e-05 2021-03-24T22:00:02.574 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.575 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.575 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:00:02.575 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.220251+0000", 2021-03-24T22:00:02.575 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.5007999999999998e-05 2021-03-24T22:00:02.576 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.576 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.576 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:02.576 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.220316+0000", 2021-03-24T22:00:02.577 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.052145710999999997 2021-03-24T22:00:02.577 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:02.577 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:02.577 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:02.577 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.578 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.578 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.14760.0:2899 7.23 e320/280 7:c49de0b4:::smithi12556470-13:head v 320'262, mlcod=279'258)", 2021-03-24T22:00:02.578 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:58:10.202989+0000", 2021-03-24T22:00:02.578 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 112.057131399, 2021-03-24T22:00:02.579 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.037682814000000002, 2021-03-24T22:00:02.579 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:02.579 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:00:02.580 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:02.580 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.580 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:02.580 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:10.202989+0000", 2021-03-24T22:00:02.581 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:02.581 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.581 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.581 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:02.582 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:10.202989+0000", 2021-03-24T22:00:02.582 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2021-03-24T22:00:02.582 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.582 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.583 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:02.583 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:10.202988+0000", 2021-03-24T22:00:02.583 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0025096099999999998 2021-03-24T22:00:02.583 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.584 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.584 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:02.584 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:10.205498+0000", 2021-03-24T22:00:02.585 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0431e-05 2021-03-24T22:00:02.585 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.585 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.586 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:02.586 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:10.205538+0000", 2021-03-24T22:00:02.586 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6042999999999999e-05 2021-03-24T22:00:02.586 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.587 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.587 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:02.587 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:10.205554+0000", 2021-03-24T22:00:02.587 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018305200000000001 2021-03-24T22:00:02.588 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.588 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.588 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:02.588 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:10.205737+0000", 2021-03-24T22:00:02.589 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.7670999999999995e-05 2021-03-24T22:00:02.589 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.589 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.589 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:00:02.590 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:10.205815+0000", 2021-03-24T22:00:02.590 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.034783873 2021-03-24T22:00:02.590 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.590 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.590 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:00:02.591 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:10.240599+0000", 2021-03-24T22:00:02.591 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2775999999999996e-05 2021-03-24T22:00:02.591 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.592 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.592 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:02.592 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:10.240671+0000", 2021-03-24T22:00:02.592 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.037682814000000002 2021-03-24T22:00:02.593 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:02.593 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:02.593 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:02.593 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.594 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.594 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:3787 7.16 7:6ed83fae:::smithi12556470-29:head [write 706026~490986] snapc 123=[123,121,120,11e,11b,11a,119,110,10f,10e,10d,10c,106,105,104,fa,f8,e7] ondisk+write+known_if_redirected e398)", 2021-03-24T22:00:02.594 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:59:28.322064+0000", 2021-03-24T22:00:02.594 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 33.938055681000002, 2021-03-24T22:00:02.595 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.042790763000000002, 2021-03-24T22:00:02.595 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:02.595 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:00:02.595 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:00:02.596 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:00:02.596 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:00:02.596 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 3787 2021-03-24T22:00:02.596 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.597 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:02.597 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.597 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:02.597 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.322064+0000", 2021-03-24T22:00:02.598 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:02.598 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.598 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.599 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:02.599 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.322064+0000", 2021-03-24T22:00:02.599 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999967 2021-03-24T22:00:02.599 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.600 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.600 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:02.600 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.322061+0000", 2021-03-24T22:00:02.600 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00063717000000000005 2021-03-24T22:00:02.601 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.601 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.601 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:02.601 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.322698+0000", 2021-03-24T22:00:02.602 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3450000000000001e-05 2021-03-24T22:00:02.602 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.602 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.602 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:02.603 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.322721+0000", 2021-03-24T22:00:02.603 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3859999999999996e-06 2021-03-24T22:00:02.603 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.604 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.604 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:02.604 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.322731+0000", 2021-03-24T22:00:02.604 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.5216999999999995e-05 2021-03-24T22:00:02.605 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.605 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.605 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:02.605 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.322806+0000", 2021-03-24T22:00:02.606 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000140579 2021-03-24T22:00:02.606 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.606 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.606 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:00:02.607 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.322947+0000", 2021-03-24T22:00:02.607 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00037403799999999999 2021-03-24T22:00:02.607 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.607 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.608 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 0,7", 2021-03-24T22:00:02.608 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.323321+0000", 2021-03-24T22:00:02.608 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0021549210000000002 2021-03-24T22:00:02.608 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.609 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.609 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T22:00:02.609 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.325476+0000", 2021-03-24T22:00:02.609 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00047763900000000003 2021-03-24T22:00:02.610 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.610 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.610 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:02.610 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.325953+0000", 2021-03-24T22:00:02.611 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038753673000000002 2021-03-24T22:00:02.611 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.611 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.611 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:02.612 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.364707+0000", 2021-03-24T22:00:02.612 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0543000000000005e-05 2021-03-24T22:00:02.612 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.613 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.613 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:00:02.613 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.364787+0000", 2021-03-24T22:00:02.613 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.7587000000000002e-05 2021-03-24T22:00:02.613 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.614 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.614 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:02.614 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.364855+0000", 2021-03-24T22:00:02.614 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.042790763000000002 2021-03-24T22:00:02.615 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:02.615 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:02.615 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:02.616 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.616 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.616 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:3788 7.16 7:6ed83fae:::smithi12556470-29:head [write 1828849~790505] snapc 123=[123,121,120,11e,11b,11a,119,110,10f,10e,10d,10c,106,105,104,fa,f8,e7] ondisk+write+known_if_redirected e398)", 2021-03-24T22:00:02.616 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:59:28.331962+0000", 2021-03-24T22:00:02.617 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 33.928157833, 2021-03-24T22:00:02.617 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033086802999999998, 2021-03-24T22:00:02.617 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:02.617 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:00:02.618 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:00:02.618 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:00:02.618 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:00:02.619 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 3788 2021-03-24T22:00:02.619 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.619 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:02.619 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.620 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:02.620 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.331962+0000", 2021-03-24T22:00:02.620 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:02.620 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.621 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.621 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:02.621 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.331962+0000", 2021-03-24T22:00:02.621 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2021-03-24T22:00:02.622 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.622 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.622 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:02.622 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.331960+0000", 2021-03-24T22:00:02.623 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00082868500000000001 2021-03-24T22:00:02.623 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.623 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.623 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:02.624 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.332789+0000", 2021-03-24T22:00:02.624 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8278999999999999e-05 2021-03-24T22:00:02.624 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.624 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.625 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:02.625 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.332807+0000", 2021-03-24T22:00:02.625 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3250000000000008e-06 2021-03-24T22:00:02.625 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.626 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.626 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:02.626 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.332816+0000", 2021-03-24T22:00:02.626 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.3894999999999999e-05 2021-03-24T22:00:02.627 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.627 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.627 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:02.627 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.332859+0000", 2021-03-24T22:00:02.628 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013111500000000001 2021-03-24T22:00:02.628 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.628 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.628 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:00:02.629 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.332991+0000", 2021-03-24T22:00:02.629 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00026257500000000003 2021-03-24T22:00:02.629 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.629 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.630 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 0,7", 2021-03-24T22:00:02.630 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.333253+0000", 2021-03-24T22:00:02.630 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0023521100000000001 2021-03-24T22:00:02.630 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.631 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.631 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T22:00:02.631 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.335605+0000", 2021-03-24T22:00:02.631 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00094053500000000005 2021-03-24T22:00:02.632 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.632 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.632 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:02.632 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.336546+0000", 2021-03-24T22:00:02.633 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028386855999999999 2021-03-24T22:00:02.633 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.633 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.633 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:02.634 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.364933+0000", 2021-03-24T22:00:02.634 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0581999999999997e-05 2021-03-24T22:00:02.634 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.634 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.635 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:00:02.635 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.364993+0000", 2021-03-24T22:00:02.635 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5719e-05 2021-03-24T22:00:02.635 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.636 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.636 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:02.636 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.365049+0000", 2021-03-24T22:00:02.636 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033086802999999998 2021-03-24T22:00:02.637 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:02.637 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:02.637 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:02.637 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.638 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.638 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:3789 7.16 7:6ed83fae:::smithi12556470-29:head [setxattr _header (58) in=7b,truncate 2706026] snapc 123=[123,121,120,11e,11b,11a,119,110,10f,10e,10d,10c,106,105,104,fa,f8,e7] ondisk+write+known_if_redirected e398)", 2021-03-24T22:00:02.638 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:59:28.332833+0000", 2021-03-24T22:00:02.639 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 33.927286770999999, 2021-03-24T22:00:02.639 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032412495, 2021-03-24T22:00:02.639 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:02.639 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:00:02.640 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:00:02.640 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:00:02.640 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:00:02.640 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 3789 2021-03-24T22:00:02.641 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.641 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:02.641 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.641 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:02.642 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.332833+0000", 2021-03-24T22:00:02.642 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:02.642 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.643 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.643 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:02.643 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.332833+0000", 2021-03-24T22:00:02.643 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2021-03-24T22:00:02.644 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.644 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.644 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:02.644 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.332832+0000", 2021-03-24T22:00:02.645 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7080000000000002e-06 2021-03-24T22:00:02.645 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.645 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.645 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:02.645 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.332838+0000", 2021-03-24T22:00:02.646 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.8630000000000008e-06 2021-03-24T22:00:02.646 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.646 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.646 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:02.647 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.332846+0000", 2021-03-24T22:00:02.647 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.7400000000000004e-06 2021-03-24T22:00:02.647 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.647 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.648 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:02.648 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.332850+0000", 2021-03-24T22:00:02.648 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0016531409999999999 2021-03-24T22:00:02.648 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.649 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.649 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:02.649 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.334503+0000", 2021-03-24T22:00:02.650 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012792700000000001 2021-03-24T22:00:02.650 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.650 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.650 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:00:02.651 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.334631+0000", 2021-03-24T22:00:02.651 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00022061100000000001 2021-03-24T22:00:02.651 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.652 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.652 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 0,7", 2021-03-24T22:00:02.652 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.334852+0000", 2021-03-24T22:00:02.652 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00076157799999999995 2021-03-24T22:00:02.653 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.653 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.653 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T22:00:02.653 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.335614+0000", 2021-03-24T22:00:02.654 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00097286900000000001 2021-03-24T22:00:02.654 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.654 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.655 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:02.655 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.336586+0000", 2021-03-24T22:00:02.655 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028520177000000001 2021-03-24T22:00:02.655 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.656 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.656 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:02.656 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.365107+0000", 2021-03-24T22:00:02.656 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3498999999999996e-05 2021-03-24T22:00:02.657 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.657 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.657 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:00:02.658 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.365170+0000", 2021-03-24T22:00:02.658 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.5605999999999994e-05 2021-03-24T22:00:02.658 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.659 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.659 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:02.659 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.365246+0000", 2021-03-24T22:00:02.659 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032412495 2021-03-24T22:00:02.660 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:02.660 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:02.660 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:02.661 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.661 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.661 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:3790 7.16 7:6ed83fae:::smithi12556470-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e398)", 2021-03-24T22:00:02.661 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:59:28.332856+0000", 2021-03-24T22:00:02.661 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 33.927264416, 2021-03-24T22:00:02.662 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032682089999999997, 2021-03-24T22:00:02.662 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:02.662 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:00:02.662 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:00:02.663 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:00:02.663 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:00:02.663 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 3790 2021-03-24T22:00:02.663 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.664 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:02.664 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.664 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:02.665 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.332856+0000", 2021-03-24T22:00:02.665 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:02.665 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.665 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.666 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:02.666 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.332856+0000", 2021-03-24T22:00:02.666 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2021-03-24T22:00:02.667 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.667 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.667 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:02.668 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.332855+0000", 2021-03-24T22:00:02.668 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7269999999999999e-06 2021-03-24T22:00:02.668 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.669 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.669 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:02.669 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.332858+0000", 2021-03-24T22:00:02.669 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7060000000000001e-06 2021-03-24T22:00:02.670 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.670 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.670 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:02.670 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.332863+0000", 2021-03-24T22:00:02.671 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.437e-06 2021-03-24T22:00:02.671 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.671 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.671 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:02.672 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.332868+0000", 2021-03-24T22:00:02.672 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.002257074 2021-03-24T22:00:02.672 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.672 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.672 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:02.673 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.335125+0000", 2021-03-24T22:00:02.673 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9114000000000001e-05 2021-03-24T22:00:02.673 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.673 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.674 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for rw locks", 2021-03-24T22:00:02.674 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.335214+0000", 2021-03-24T22:00:02.674 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030079662 2021-03-24T22:00:02.674 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.675 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.675 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:02.675 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.365294+0000", 2021-03-24T22:00:02.675 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.5464999999999995e-05 2021-03-24T22:00:02.676 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.676 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.676 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:00:02.677 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.365389+0000", 2021-03-24T22:00:02.677 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014837699999999999 2021-03-24T22:00:02.677 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.677 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.678 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:02.678 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:28.365538+0000", 2021-03-24T22:00:02.678 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032682089999999997 2021-03-24T22:00:02.678 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:02.679 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:02.679 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:02.679 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.679 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.680 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(4.3 422/417 [PushOp(4:c5eee8e2:::data_log.59:head, version: 65'2, data_included: [0~156], data_size: 156, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:c5eee8e2:::data_log.59:head@65'2, size: 156, copy_subset: [0~156], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:156, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2021-03-24T22:00:02.680 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:59:52.286731+0000", 2021-03-24T22:00:02.680 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 9.9733892829999995, 2021-03-24T22:00:02.681 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.16378574300000001, 2021-03-24T22:00:02.681 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:02.681 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:00:02.681 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:02.682 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.682 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:02.682 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:52.286731+0000", 2021-03-24T22:00:02.683 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:02.683 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.683 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.683 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:02.684 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:52.286731+0000", 2021-03-24T22:00:02.684 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999971 2021-03-24T22:00:02.684 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.684 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.685 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:02.685 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:52.286728+0000", 2021-03-24T22:00:02.685 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3193e-05 2021-03-24T22:00:02.685 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.686 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.686 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:02.686 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:52.286751+0000", 2021-03-24T22:00:02.687 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1464e-05 2021-03-24T22:00:02.687 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.687 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.687 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:02.688 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:52.286773+0000", 2021-03-24T22:00:02.688 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3377000000000001e-05 2021-03-24T22:00:02.688 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.688 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.689 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:02.689 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:52.286786+0000", 2021-03-24T22:00:02.689 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.16356628100000001 2021-03-24T22:00:02.689 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.690 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.690 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:02.690 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:52.450352+0000", 2021-03-24T22:00:02.690 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4530000000000001e-06 2021-03-24T22:00:02.691 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.691 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.691 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:00:02.691 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:52.450358+0000", 2021-03-24T22:00:02.692 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015859899999999999 2021-03-24T22:00:02.692 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.692 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.693 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:02.693 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:52.450516+0000", 2021-03-24T22:00:02.693 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.16378574300000001 2021-03-24T22:00:02.693 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:02.693 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:02.694 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:02.699 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.699 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.699 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(4.6 424/417 [PushOp(4:620d8b71:::data_log.99:head, version: 65'1, data_included: [0~156], data_size: 156, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:620d8b71:::data_log.99:head@65'1, size: 156, copy_subset: [0~156], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:156, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2021-03-24T22:00:02.700 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:59:54.302766+0000", 2021-03-24T22:00:02.700 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 7.9573540210000004, 2021-03-24T22:00:02.700 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.13019102099999999, 2021-03-24T22:00:02.701 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:02.701 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:00:02.701 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:02.702 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.702 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:02.702 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:54.302766+0000", 2021-03-24T22:00:02.702 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:02.703 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.703 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.703 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:02.703 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:54.302766+0000", 2021-03-24T22:00:02.704 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2021-03-24T22:00:02.704 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.704 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.705 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:02.705 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:54.302764+0000", 2021-03-24T22:00:02.705 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7988000000000001e-05 2021-03-24T22:00:02.705 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.706 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.706 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:02.706 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:54.302782+0000", 2021-03-24T22:00:02.706 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3334000000000001e-05 2021-03-24T22:00:02.707 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.707 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.707 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:02.708 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:54.302806+0000", 2021-03-24T22:00:02.708 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7897000000000002e-05 2021-03-24T22:00:02.708 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.709 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.709 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:02.709 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:54.302824+0000", 2021-03-24T22:00:02.710 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.129927036 2021-03-24T22:00:02.710 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.710 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.710 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:02.711 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:54.432751+0000", 2021-03-24T22:00:02.711 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.7220000000000002e-06 2021-03-24T22:00:02.711 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.712 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.712 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:00:02.712 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:54.432757+0000", 2021-03-24T22:00:02.713 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00019955999999999999 2021-03-24T22:00:02.713 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:02.713 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:02.714 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:02.714 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:54.432957+0000", 2021-03-24T22:00:02.714 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.13019102099999999 2021-03-24T22:00:02.715 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:02.715 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:02.715 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:02.716 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:02.716 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:02.716 INFO:teuthology.orchestra.run.smithi142.stdout:} 2021-03-24T22:00:02.717 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:02 smithi142 conmon[20125]: cluster 2021-03-24T22:00:00.847916+0000 mgr.y (mgr.14144) 486 : cluster [DBG] pgmap v808: 177 pgs: 2 peering, 1 remapped+peering, 60 active+clean+remapped, 35 active+undersized, 3 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 58 active+clean; 429 MiB data, 879 MiB used, 446 GiB / 447 GiB avail; 18 MiB/s rd, 3.8 MiB/s wr, 24 op/s; 143/1575 objects degraded (9.079%); 233/1575 objects misplaced (14.794%); 19 MiB/s, 43 objects/s recovering 2021-03-24T22:00:02.718 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:02 smithi142 conmon[20125]: 2021-03-24T22:00:02.718 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:02 smithi142 conmon[20125]: audit 2021-03-24T22:00:02.104582+0000 mon.c (mon.2) 68 : audit [DBG] from='client.? v1:172.21.15.125:0/4081542210' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:00:02.718 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:02 smithi142 conmon[20125]: cluster 2021-03-24T22:00:02.227381+0000 mon.a (mon.0) 1318 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2021-03-24T22:00:02.718 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:02 smithi142 conmon[20125]: cluster 2021-03-24T22:00:02.237228+0000 mon.a (mon.0) 1319 : cluster [INF] osd.5 v1:172.21.15.142:6804/3992668882 boot 2021-03-24T22:00:02.719 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:02 smithi142 conmon[20125]: cluster 2021-03-24T22:00:02.719 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:02 smithi142 conmon[20125]: 2021-03- 2021-03-24T22:00:02.720 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:02 smithi142 conmon[20125]: 24T22:00:02.237321+0000 mon.a (mon.0) 1320 2021-03-24T22:00:02.720 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:02 smithi142 conmon[20125]: : cluster [DBG] osdmap e432: 8 total, 8 up, 5 in 2021-03-24T22:00:02.721 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:02 smithi142 conmon[20125]: audit 2021-03- 2021-03-24T22:00:02.721 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:02 smithi142 conmon[20125]: 24T22:00:02.238739+0000 mon.a (mon.0) 2021-03-24T22:00:02.721 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:02 smithi142 conmon[20125]: 1321 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-03-24T22:00:02.722 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T22:00:02.722 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-24T22:00:02.722 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-24T22:00:02.723 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-24T22:00:02.723 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-24T22:00:02.723 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-24T22:00:02.723 INFO:teuthology.orchestra.run.smithi125.stderr:Error ENXIO: problem getting command descriptions from osd.5 2021-03-24T22:00:02.724 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-24T22:00:02.724 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-24T22:00:02.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:02.726 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:02.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:00:02.727 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:02 smithi125 conmon[19936]: cluster 2021-03-24T22:00:00.847916+0000 2021-03-24T22:00:02.727 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:02 smithi125 conmon[19936]: mgr.y (mgr.14144) 486 : cluster [DBG] pgmap v808: 177 pgs: 2 peering, 1 remapped+peering, 60 active+clean+remapped, 35 active+undersized, 3 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 58 active+clean; 429 MiB data, 879 MiB used, 446 GiB / 447 GiB avail; 18 MiB/s rd, 3.8 MiB/s wr, 24 op/s; 143/1575 objects degraded (9.079%); 233/1575 objects misplaced (14.794%); 19 MiB/s, 43 objects/s recovering 2021-03-24T22:00:02.728 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:02 smithi125 conmon[19936]: audit 2021-03-24T22:00:02.104582+0000 mon.c (mon. 2021-03-24T22:00:02.728 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:02 smithi125 conmon[19936]: 2) 68 : audit [DBG] from='client.? v1:172.21.15.125:0/4081542210' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:00:02.728 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:02 smithi125 conmon[19936]: cluster 2021-03-24T22:00:02. 2021-03-24T22:00:02.728 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:02 smithi125 conmon[19936]: 227381+0000 mon.a (mon.0) 1318 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2021-03-24T22:00:02.729 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:02 smithi125 conmon[19936]: cluster 2021-03-24T22:00:02.729 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:02 smithi125 conmon[19936]: 2021-03-24T22:00:02.237228+0000 mon.a (mon.0) 1319 : cluster 2021-03-24T22:00:02.729 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:02 smithi125 conmon[19936]: [INF] osd.5 v1:172.21.15.142:6804/3992668882 boot 2021-03-24T22:00:02.730 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:02 smithi125 conmon[19936]: cluster 2021-03-24T22:00:02.730 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:02 smithi125 conmon[19936]: 2021-03-24T 2021-03-24T22:00:02.730 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:02 smithi125 conmon[19936]: 22:00:02.237321+0000 mon.a 2021-03-24T22:00:02.730 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:02 smithi125 conmon[19936]: (mon.0) 1320 : cluster [DBG] 2021-03-24T22:00:02.731 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:02 smithi125 conmon[19936]: osdmap e432: 8 total, 8 up, 5 in 2021-03-24T22:00:02.731 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:02 smithi125 conmon[19936]: audit 2021-03-24T22:00 2021-03-24T22:00:02.732 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:02 smithi125 conmon[19936]: :02.238739+0000 mon.a (mon.0) 1321 2021-03-24T22:00:02.732 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:02 smithi125 conmon[19936]: : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-03-24T22:00:02.733 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:02 smithi125 conmon[24927]: cluster 2021-03-24T22:00: 2021-03-24T22:00:02.733 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:02 smithi125 conmon[24927]: 00.847916+0000 mgr.y (mgr.14144) 486 : cluster [DBG] pgmap v808: 177 pgs: 2 peering, 1 remapped+peering, 60 active+clean+remapped, 35 active+undersized, 3 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 58 active+clean; 429 MiB data, 879 MiB used, 446 GiB / 447 GiB avail; 18 MiB/s rd, 3.8 MiB/s wr, 24 op/s; 143/1575 objects degraded (9.079%); 233/1575 objects misplaced (14.794%); 19 MiB/s, 43 objects/s recovering 2021-03-24T22:00:02.733 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:02 smithi125 conmon[24927]: audit 2021-03-24T22:00:02.734 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:02 smithi125 conmon[24927]: 2021-03-24T22:00:02.104582+0000 mon.c (mon.2 2021-03-24T22:00:02.734 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:02 smithi125 conmon[24927]: ) 68 : audit [DBG] from='client.? v1:172.21.15.125:0/4081542210' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:00:02.734 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:02 smithi125 conmon[24927]: cluster 2021-03-24T22:00 2021-03-24T22:00:02.734 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:02 smithi125 conmon[24927]: :02.227381+0000 mon.a (mon.0) 1318 : cluster [INF] 2021-03-24T22:00:02.735 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:02 smithi125 conmon[24927]: Health check cleared: OSD_DOWN (was: 1 osds down) 2021-03-24T22:00:02.735 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:02 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:00:02.735 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:02 smithi125 conmon[24927]: 03-24T22:00:02.237228+0000 mon.a (mon.0) 1319 : cluster 2021-03-24T22:00:02.736 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:02 smithi125 conmon[24927]: [INF] osd.5 v1:172.21.15.142:6804/3992668882 boot 2021-03-24T22:00:02.736 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:02 smithi125 conmon[24927]: cluster 2021-03-24T22:00:02.736 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:02 smithi125 conmon[24927]: 2021-03-24T22:00:02.237321 2021-03-24T22:00:02.737 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:02 smithi125 conmon[24927]: +0000 mon.a (mon.0) 1320 : cluster [DBG] osdmap e432: 8 total, 8 up, 5 in 2021-03-24T22:00:02.737 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:02 smithi125 conmon[24927]: audit 2021-03 2021-03-24T22:00:02.737 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:02 smithi125 conmon[24927]: -24T22:00:02.238739+0000 mon.a (mon. 2021-03-24T22:00:02.737 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:02 smithi125 conmon[24927]: 0) 1321 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-03-24T22:00:02.827 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:02.883 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:02.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:02.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:00:02.989 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:03.045 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:03.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:03.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:00:03.151 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:03.215 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:03.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:03.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:00:03.255 INFO:tasks.rados.rados.0.smithi125.stdout:1599: done (1 left) 2021-03-24T22:00:03.255 INFO:tasks.rados.rados.0.smithi125.stdout:1600: rollback oid 8 current snap is 174 2021-03-24T22:00:03.255 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 8 to 147 2021-03-24T22:00:03.256 INFO:tasks.rados.rados.0.smithi125.stdout:1601: write (excl) oid 18 current snap is 174 2021-03-24T22:00:03.256 INFO:tasks.rados.rados.0.smithi125.stdout:1601: seq_num 567 ranges {729569=476305,1996349=505866,3004131=525438} 2021-03-24T22:00:03.256 INFO:tasks.rados.rados.0.smithi125.stdout:1601: writing smithi12556470-18 from 729569 to 1205874 tid 1 2021-03-24T22:00:03.263 INFO:tasks.rados.rados.0.smithi125.stdout:1601: writing smithi12556470-18 from 1996349 to 2502215 tid 2 2021-03-24T22:00:03.270 INFO:tasks.rados.rados.0.smithi125.stdout:1601: writing smithi12556470-18 from 3004131 to 3529569 tid 3 2021-03-24T22:00:03.270 INFO:tasks.rados.rados.0.smithi125.stdout:1602: delete oid 45 current snap is 174 2021-03-24T22:00:03.299 INFO:tasks.rados.rados.0.smithi125.stdout:1600: finishing rollback tid 1 to smithi12556470-8 2021-03-24T22:00:03.300 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 8 v 81 (ObjNum 0 snap 32741 seq_num 1116859504) dirty dne 2021-03-24T22:00:03.322 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:03.371 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:03.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:03.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:00:03.440 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.6 dump_ops_in_flight 2021-03-24T22:00:03.493 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-03-24T22:00:03.495 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-03-24T22:00:03.540 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:03.594 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:03.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:03.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:00:03.698 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:03.700 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-24T22:00:03.745 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T22:00:03.751 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:03.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:03.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:00:03.857 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:03.912 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:03.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:03.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:00:03.980 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:00:04.017 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:04.090 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:04.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:04.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:00:04.195 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:04.275 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:04.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:04.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:00:04.318 INFO:tasks.rados.rados.0.smithi125.stdout:1601: finishing write tid 1 to smithi12556470-18 2021-03-24T22:00:04.321 INFO:tasks.rados.rados.0.smithi125.stdout:1598: finishing write tid 1 to smithi12556470-19 2021-03-24T22:00:04.326 INFO:tasks.rados.rados.0.smithi125.stdout:1598: finishing write tid 2 to smithi12556470-19 2021-03-24T22:00:04.327 INFO:tasks.rados.rados.0.smithi125.stdout:1601: finishing write tid 2 to smithi12556470-18 2021-03-24T22:00:04.328 INFO:tasks.rados.rados.0.smithi125.stdout:1601: finishing write tid 3 to smithi12556470-18 2021-03-24T22:00:04.328 INFO:tasks.rados.rados.0.smithi125.stdout:1601: finishing write tid 4 to smithi12556470-18 2021-03-24T22:00:04.328 INFO:tasks.rados.rados.0.smithi125.stdout:1601: finishing write tid 5 to smithi12556470-18 2021-03-24T22:00:04.328 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 18 v 195 (ObjNum 567 snap 174 seq_num 567) dirty exists 2021-03-24T22:00:04.329 INFO:tasks.rados.rados.0.smithi125.stdout:1601: left oid 18 (ObjNum 567 snap 174 seq_num 567) 2021-03-24T22:00:04.329 INFO:tasks.rados.rados.0.smithi125.stdout:1598: finishing write tid 3 to smithi12556470-19 2021-03-24T22:00:04.329 INFO:tasks.rados.rados.0.smithi125.stdout:1598: finishing write tid 4 to smithi12556470-19 2021-03-24T22:00:04.329 INFO:tasks.rados.rados.0.smithi125.stdout:1598: finishing write tid 5 to smithi12556470-19 2021-03-24T22:00:04.329 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 19 v 323 (ObjNum 566 snap 174 seq_num 566) dirty exists 2021-03-24T22:00:04.330 INFO:tasks.rados.rados.0.smithi125.stdout:1598: left oid 19 (ObjNum 566 snap 174 seq_num 566) 2021-03-24T22:00:04.338 INFO:tasks.rados.rados.0.smithi125.stdout:1598: done (3 left) 2021-03-24T22:00:04.338 INFO:tasks.rados.rados.0.smithi125.stdout:1600: done (2 left) 2021-03-24T22:00:04.339 INFO:tasks.rados.rados.0.smithi125.stdout:1601: done (1 left) 2021-03-24T22:00:04.339 INFO:tasks.rados.rados.0.smithi125.stdout:1602: done (0 left) 2021-03-24T22:00:04.339 INFO:tasks.rados.rados.0.smithi125.stdout:1603: copy_from oid 14 from oid 37 current snap is 174 2021-03-24T22:00:04.339 INFO:tasks.rados.rados.0.smithi125.stdout:1604: delete oid 49 current snap is 174 2021-03-24T22:00:04.340 INFO:tasks.rados.rados.0.smithi125.stdout:1604: done (1 left) 2021-03-24T22:00:04.340 INFO:tasks.rados.rados.0.smithi125.stdout:1605: delete oid 31 current snap is 174 2021-03-24T22:00:04.354 INFO:tasks.rados.rados.0.smithi125.stdout:1603: finishing copy_from to smithi12556470-14 2021-03-24T22:00:04.354 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 14 v 348 (ObjNum 537 snap 166 seq_num 537) dirty exists 2021-03-24T22:00:04.355 INFO:tasks.rados.rados.0.smithi125.stdout:1603: finishing copy_from racing read to smithi12556470-14 2021-03-24T22:00:04.355 INFO:tasks.rados.rados.0.smithi125.stdout:1603: done (1 left) 2021-03-24T22:00:04.355 INFO:tasks.rados.rados.0.smithi125.stdout:1605: done (0 left) 2021-03-24T22:00:04.356 INFO:tasks.rados.rados.0.smithi125.stdout:1606: read oid 18 snap -1 2021-03-24T22:00:04.356 INFO:tasks.rados.rados.0.smithi125.stdout:1606: expect (ObjNum 567 snap 174 seq_num 567) 2021-03-24T22:00:04.356 INFO:tasks.rados.rados.0.smithi125.stdout:1607: read oid 45 snap -1 2021-03-24T22:00:04.357 INFO:tasks.rados.rados.0.smithi125.stdout:1607: expect deleted 2021-03-24T22:00:04.357 INFO:tasks.rados.rados.0.smithi125.stdout:1608: write (excl) oid 33 current snap is 174 2021-03-24T22:00:04.358 INFO:tasks.rados.rados.0.smithi125.stdout:1608: seq_num 571 ranges {796797=493654,1830277=796851,3397470=199327} 2021-03-24T22:00:04.363 INFO:tasks.rados.rados.0.smithi125.stdout:1608: writing smithi12556470-33 from 796797 to 1290451 tid 1 2021-03-24T22:00:04.372 INFO:tasks.rados.rados.0.smithi125.stdout:1608: writing smithi12556470-33 from 1830277 to 2627128 tid 2 2021-03-24T22:00:04.379 INFO:tasks.rados.rados.0.smithi125.stdout:1608: writing smithi12556470-33 from 3397470 to 3596797 tid 3 2021-03-24T22:00:04.379 INFO:tasks.rados.rados.0.smithi125.stdout:1609: snap_create 2021-03-24T22:00:04.380 INFO:tasks.rados.rados.0.smithi125.stdout:1606: expect (ObjNum 567 snap 174 seq_num 567) 2021-03-24T22:00:04.382 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:04.418 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:04 smithi142 conmon[20125]: cluster 2021-03-24T22:00:04.418 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:04 smithi142 conmon[20125]: 2021-03-24T22:00:04.418 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:04 smithi142 conmon[20125]: 2021 2021-03-24T22:00:04.419 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:04 smithi142 conmon[20125]: - 2021-03-24T22:00:04.419 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:04 smithi142 conmon[20125]: 03 2021-03-24T22:00:04.420 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:04 smithi142 conmon[20125]: - 2021-03-24T22:00:04.420 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:04 smithi142 conmon[20125]: 24 2021-03-24T22:00:04.421 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:04 smithi142 conmon[20125]: T 2021-03-24T22:00:04.421 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:04 smithi142 conmon[20125]: 22 2021-03-24T22:00:04.422 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:04 smithi142 conmon[20125]: : 2021-03-24T22:00:04.422 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:04 smithi142 conmon[20125]: 00 2021-03-24T22:00:04.422 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:04 smithi142 conmon[20125]: :01.558641+0000 osd.5 (osd.5) 1 2021-03-24T22:00:04.423 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:04 smithi142 conmon[20125]: : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:00:04.423 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:04 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:00:04.424 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:04 smithi142 conmon[20125]: 03-24T22:00:01.559922+0000 osd.5 (osd.5) 2 : cluster [DBG] 2021-03-24T22:00:04.424 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:04 smithi142 conmon[20125]: purged_snaps scrub ok 2021-03-24T22:00:04.425 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:04 smithi142 conmon[20125]: cluster 2021-03-24T22:00:02 2021-03-24T22:00:04.425 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:04 smithi142 conmon[20125]: .849146+0000 mgr.y (mgr.14144) 487 : cluster 2021-03-24T22:00:04.425 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:04 smithi142 conmon[20125]: [DBG] pgmap v811: 177 pgs: 2 peering, 61 active+clean+remapped, 35 active+undersized, 3 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 58 active+clean; 425 MiB data, 884 MiB used, 446 GiB / 447 GiB avail; 26 MiB/s rd, 4.1 MiB/s wr, 32 op/s; 143/1569 objects degraded (9.114%); 244/1569 objects misplaced (15.551%); 18 MiB/s, 39 objects/s recovering 2021-03-24T22:00:04.426 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:04 smithi142 conmon[20125]: cluster 2021-03-24T22:00:04.426 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:04 smithi142 conmon[20125]: 2021-03-24T22:00:03.241320+0000 mon.a (mon.0 2021-03-24T22:00:04.427 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:04 smithi142 conmon[20125]: ) 1322 : cluster [DBG] osdmap e433: 8 total, 8 up, 5 in 2021-03-24T22:00:04.427 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:04 smithi142 conmon[20125]: cluster 2021-03-24T22:00:03 2021-03-24T22:00:04.428 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:04 smithi142 conmon[20125]: .250046+0000 osd.3 (osd.3) 94 : cluster [DBG] 4.f starting backfill to osd.5 from (69'17,69'17] MIN to 274'26 2021-03-24T22:00:04.428 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:04 smithi142 conmon[20125]: 2021-03-24T22:00:04.435 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:04.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:04.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:00:04.467 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:04 smithi125 conmon[24927]: cluster 2021-03-24T22:00:01. 2021-03-24T22:00:04.467 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:04 smithi125 conmon[24927]: 558641+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:00:04.467 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:04 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T22:00:04.468 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:04 smithi125 conmon[24927]: T22:00:01.559922+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:00:04.468 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:04 smithi125 conmon[24927]: 2021-03-24T22:00:04.468 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:04 smithi125 conmon[24927]: cluster 2021-03-24T22:00:04.469 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:04 smithi125 conmon[24927]: 2021-03-24T22:00:02.849146+0000 mgr.y (mgr 2021-03-24T22:00:04.469 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:04 smithi125 conmon[24927]: .14144) 487 : cluster [DBG] pgmap v811: 177 pgs: 2 peering, 61 active+clean+remapped, 35 active+undersized, 3 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 58 active+clean; 425 MiB data, 884 MiB used, 446 GiB / 447 GiB avail; 26 MiB/s rd, 4.1 MiB/s wr, 32 op/s; 143/1569 objects degraded (9.114%); 244/1569 objects misplaced (15.551%); 18 MiB/s, 39 objects/s recovering 2021-03-24T22:00:04.469 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:04 smithi125 conmon[24927]: cluster 2021-03-24T22:00: 2021-03-24T22:00:04.470 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:04 smithi125 conmon[24927]: 03.241320+0000 mon.a (mon.0) 1322 : cluster 2021-03-24T22:00:04.470 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:04 smithi125 conmon[24927]: [DBG] osdmap e433: 8 total, 8 up, 5 in 2021-03-24T22:00:04.470 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:04 smithi125 conmon[24927]: cluster 2021-03-24T22:00:04.471 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:04 smithi125 conmon[24927]: 2021-03-24T22:00:03.250046+0000 osd.3 (osd. 2021-03-24T22:00:04.471 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:04 smithi125 conmon[24927]: 3) 94 : cluster [DBG] 4.f starting backfill to osd.5 from (69'17,69'17] MIN to 274'26 2021-03-24T22:00:04.472 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:04 smithi125 conmon[19936]: cluster 2021-03-24T22:00:04.472 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:04 smithi125 conmon[19936]: 2021-03-24T22:00:04.473 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:04 smithi125 conmon[19936]: 2021 2021-03-24T22:00:04.473 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:04 smithi125 conmon[19936]: - 2021-03-24T22:00:04.473 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:04 smithi125 conmon[19936]: 03 2021-03-24T22:00:04.473 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:04 smithi125 conmon[19936]: - 2021-03-24T22:00:04.474 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:04 smithi125 conmon[19936]: 24T22:00:01.558641 2021-03-24T22:00:04.474 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:04 smithi125 conmon[19936]: +0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:00:04.474 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:04 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:00:04.475 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:04 smithi125 conmon[19936]: -03-24T22:00:01.559922+0000 osd.5 2021-03-24T22:00:04.475 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:04 smithi125 conmon[19936]: (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:00:04.475 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:04 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:00:04.476 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:04 smithi125 conmon[19936]: 03-24T22:00:02.849146+0000 mgr.y (mgr. 2021-03-24T22:00:04.476 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:04 smithi125 conmon[19936]: 14144) 487 : cluster [DBG] pgmap v811: 177 pgs: 2 peering, 61 active+clean+remapped, 35 active+undersized, 3 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 58 active+clean; 425 MiB data, 884 MiB used, 446 GiB / 447 GiB avail; 26 MiB/s rd, 4.1 MiB/s wr, 32 op/s; 143/1569 objects degraded (9.114%); 244/1569 objects misplaced (15.551%); 18 MiB/s, 39 objects/s recovering 2021-03-24T22:00:04.476 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:04 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:00:04.477 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:04 smithi125 conmon[19936]: T22:00:03.241320+0000 mon.a (mon.0) 1322 : 2021-03-24T22:00:04.477 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:04 smithi125 conmon[19936]: cluster [DBG] osdmap e433: 8 total, 8 up, 5 in 2021-03-24T22:00:04.477 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:04 smithi125 conmon[19936]: cluster 2021-03-24T22:00 2021-03-24T22:00:04.478 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:04 smithi125 conmon[19936]: :03.250046+0000 osd.3 (osd.3) 94 : 2021-03-24T22:00:04.478 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:04 smithi125 conmon[19936]: cluster [DBG] 4.f starting backfill to osd.5 from (69'17,69'17] MIN to 274'26 2021-03-24T22:00:04.541 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:04.600 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:04.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:04.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:00:04.676 INFO:tasks.rados.rados.0.smithi125.stdout:1608: finishing write tid 1 to smithi12556470-33 2021-03-24T22:00:04.676 INFO:tasks.rados.rados.0.smithi125.stdout:1608: finishing write tid 2 to smithi12556470-33 2021-03-24T22:00:04.677 INFO:tasks.rados.rados.0.smithi125.stdout:1608: finishing write tid 3 to smithi12556470-33 2021-03-24T22:00:04.677 INFO:tasks.rados.rados.0.smithi125.stdout:1608: finishing write tid 4 to smithi12556470-33 2021-03-24T22:00:04.678 INFO:tasks.rados.rados.0.smithi125.stdout:1608: finishing write tid 5 to smithi12556470-33 2021-03-24T22:00:04.678 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 33 v 460 (ObjNum 571 snap 174 seq_num 571) dirty exists 2021-03-24T22:00:04.679 INFO:tasks.rados.rados.0.smithi125.stdout:1608: left oid 33 (ObjNum 571 snap 174 seq_num 571) 2021-03-24T22:00:04.705 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:04.762 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:04.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:04.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:04.869 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:04.924 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:04.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:04.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:00:05.032 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:05.097 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:05.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:05.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:00:05.203 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:05.229 INFO:tasks.rados.rados.0.smithi125.stdout:1606: done (3 left) 2021-03-24T22:00:05.229 INFO:tasks.rados.rados.0.smithi125.stdout:1607: done (2 left) 2021-03-24T22:00:05.229 INFO:tasks.rados.rados.0.smithi125.stdout:1608: done (1 left) 2021-03-24T22:00:05.229 INFO:tasks.rados.rados.0.smithi125.stdout:1609: done (0 left) 2021-03-24T22:00:05.230 INFO:tasks.rados.rados.0.smithi125.stdout:1610: snap_remove snap 171 2021-03-24T22:00:05.278 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:05.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:05.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:00:05.388 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:05.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[19936]: cluster 2021-03-24T22:00:05.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[19936]: 2021-03- 2021-03-24T22:00:05.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[19936]: 24T22:00:04.236218+0000 mon.a (mon.0) 1323 : cluster [DBG] osdmap e434: 8 total, 8 up, 5 in 2021-03-24T22:00:05.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[19936]: audit 2021-03-24T22:00 2021-03-24T22:00:05.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[19936]: :04.791244+0000 mon.a (mon.0) 1324 : audit 2021-03-24T22:00:05.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[19936]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T22:00:05.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[19936]: audit 2021-03-24T22:00:05.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[19936]: 2021-03-24T22:00:04.791632+0000 mon.a (mon. 2021-03-24T22:00:05.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[19936]: 0) 1325 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T22:00:05.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:00:05.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[19936]: 00:05.219988+0000 mon.a (mon.0 2021-03-24T22:00:05.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[19936]: ) 1326 : cluster [WRN] Health check update: Degraded data redundancy: 143/1569 objects degraded (9.114%), 21 pgs degraded, 47 pgs undersized (PG_DEGRADED) 2021-03-24T22:00:05.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[19936]: cluster 2021-03-24T22 2021-03-24T22:00:05.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[19936]: :00:05.225795+0000 mon.a (mon.0 2021-03-24T22:00:05.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[19936]: ) 1327 : cluster [DBG] osdmap e435: 8 total, 8 up, 5 in 2021-03-24T22:00:05.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: cluster 2021-03-24T22:00:05.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 2021-03-24T22:00:05.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 2021 2021-03-24T22:00:05.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: - 2021-03-24T22:00:05.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 03 2021-03-24T22:00:05.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: - 2021-03-24T22:00:05.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 24 2021-03-24T22:00:05.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: T 2021-03-24T22:00:05.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 22 2021-03-24T22:00:05.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: : 2021-03-24T22:00:05.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 00 2021-03-24T22:00:05.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: : 2021-03-24T22:00:05.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 04 2021-03-24T22:00:05.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: . 2021-03-24T22:00:05.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 236218 2021-03-24T22:00:05.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: +0000 2021-03-24T22:00:05.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 2021-03-24T22:00:05.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: mon.a 2021-03-24T22:00:05.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: ( 2021-03-24T22:00:05.420 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: mon 2021-03-24T22:00:05.420 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: . 2021-03-24T22:00:05.420 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 0 2021-03-24T22:00:05.420 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: ) 2021-03-24T22:00:05.421 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 1323 2021-03-24T22:00:05.421 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: : 2021-03-24T22:00:05.421 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: cluster 2021-03-24T22:00:05.421 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 2021-03-24T22:00:05.421 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: [DBG] 2021-03-24T22:00:05.422 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 2021-03-24T22:00:05.422 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: osdmap e434: 8 total, 8 up, 5 in 2021-03-24T22:00:05.422 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 2021-03-24T22:00:05.422 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: audit 2021-03-24T22:00:05.423 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 2021-03-24T22:00:05.423 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 2021 2021-03-24T22:00:05.423 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: - 2021-03-24T22:00:05.423 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 03 2021-03-24T22:00:05.424 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: - 2021-03-24T22:00:05.424 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 24 2021-03-24T22:00:05.424 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: T 2021-03-24T22:00:05.424 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 22 2021-03-24T22:00:05.425 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: : 2021-03-24T22:00:05.425 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 00 2021-03-24T22:00:05.425 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: : 2021-03-24T22:00:05.425 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 04 2021-03-24T22:00:05.425 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: . 2021-03-24T22:00:05.426 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 791244 2021-03-24T22:00:05.426 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: +0000 2021-03-24T22:00:05.427 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 2021-03-24T22:00:05.427 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: mon.a 2021-03-24T22:00:05.427 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: ( 2021-03-24T22:00:05.427 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: mon 2021-03-24T22:00:05.428 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: . 2021-03-24T22:00:05.428 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 0 2021-03-24T22:00:05.428 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: ) 2021-03-24T22:00:05.428 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 1324 2021-03-24T22:00:05.429 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: : 2021-03-24T22:00:05.429 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: audit 2021-03-24T22:00:05.430 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 2021-03-24T22:00:05.430 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: [INF] 2021-03-24T22:00:05.431 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 2021-03-24T22:00:05.431 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T22:00:05.431 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 2021-03-24T22:00:05.432 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: audit 2021-03-24T22:00:05.432 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 2021-03-24T22:00:05.433 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 2021 2021-03-24T22:00:05.433 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: - 2021-03-24T22:00:05.434 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 03 2021-03-24T22:00:05.434 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: - 2021-03-24T22:00:05.434 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 24 2021-03-24T22:00:05.435 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: T 2021-03-24T22:00:05.435 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 22 2021-03-24T22:00:05.436 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: : 2021-03-24T22:00:05.436 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 00 2021-03-24T22:00:05.437 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: : 2021-03-24T22:00:05.437 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 04 2021-03-24T22:00:05.438 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: . 2021-03-24T22:00:05.438 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 791632 2021-03-24T22:00:05.439 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: +0000 2021-03-24T22:00:05.439 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 2021-03-24T22:00:05.440 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: mon.a 2021-03-24T22:00:05.440 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: ( 2021-03-24T22:00:05.440 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: mon 2021-03-24T22:00:05.441 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: . 2021-03-24T22:00:05.441 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 0 2021-03-24T22:00:05.442 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: ) 2021-03-24T22:00:05.442 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 1325 2021-03-24T22:00:05.443 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: : 2021-03-24T22:00:05.443 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: audit 2021-03-24T22:00:05.443 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 2021-03-24T22:00:05.444 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: [INF] 2021-03-24T22:00:05.444 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 2021-03-24T22:00:05.445 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T22:00:05.445 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 2021-03-24T22:00:05.446 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: cluster 2021-03-24T22:00:05.446 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 2021-03-24T22:00:05.447 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 2021 2021-03-24T22:00:05.447 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: - 2021-03-24T22:00:05.447 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 03 2021-03-24T22:00:05.448 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: - 2021-03-24T22:00:05.448 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 24 2021-03-24T22:00:05.449 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: T 2021-03-24T22:00:05.449 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 22 2021-03-24T22:00:05.450 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: : 2021-03-24T22:00:05.451 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 00 2021-03-24T22:00:05.451 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: : 2021-03-24T22:00:05.451 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 05 2021-03-24T22:00:05.452 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: . 2021-03-24T22:00:05.452 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 219988 2021-03-24T22:00:05.453 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: +0000 2021-03-24T22:00:05.453 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 2021-03-24T22:00:05.453 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: mon.a 2021-03-24T22:00:05.454 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: ( 2021-03-24T22:00:05.454 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: mon 2021-03-24T22:00:05.454 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: . 2021-03-24T22:00:05.455 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 0 2021-03-24T22:00:05.455 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: ) 2021-03-24T22:00:05.455 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 1326 2021-03-24T22:00:05.455 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: : 2021-03-24T22:00:05.456 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: cluster 2021-03-24T22:00:05.456 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 2021-03-24T22:00:05.457 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: [WRN] 2021-03-24T22:00:05.457 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 2021-03-24T22:00:05.457 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: Health check update: Degraded data redundancy: 143/1569 objects degraded (9.114%), 21 pgs degraded, 47 pgs undersized (PG_DEGRADED) 2021-03-24T22:00:05.458 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 2021-03-24T22:00:05.458 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: cluster 2021-03-24T22:00:05.458 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 2021-03-24T22:00:05.459 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 2021 2021-03-24T22:00:05.459 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: - 2021-03-24T22:00:05.459 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 03 2021-03-24T22:00:05.459 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: - 2021-03-24T22:00:05.460 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 24 2021-03-24T22:00:05.460 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: T 2021-03-24T22:00:05.460 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 22 2021-03-24T22:00:05.460 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: : 2021-03-24T22:00:05.461 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 00 2021-03-24T22:00:05.461 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: : 2021-03-24T22:00:05.462 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 05 2021-03-24T22:00:05.462 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: . 2021-03-24T22:00:05.463 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 225795 2021-03-24T22:00:05.463 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: +0000 2021-03-24T22:00:05.463 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 2021-03-24T22:00:05.464 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: mon.a 2021-03-24T22:00:05.464 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: ( 2021-03-24T22:00:05.465 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: mon 2021-03-24T22:00:05.465 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: . 2021-03-24T22:00:05.465 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 0 2021-03-24T22:00:05.466 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: ) 2021-03-24T22:00:05.466 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 1327 2021-03-24T22:00:05.467 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: : 2021-03-24T22:00:05.467 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: cluster 2021-03-24T22:00:05.467 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 2021-03-24T22:00:05.468 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: [DBG] 2021-03-24T22:00:05.468 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 2021-03-24T22:00:05.468 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: osdmap e435: 8 total, 8 up, 5 in 2021-03-24T22:00:05.468 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:05 smithi125 conmon[24927]: 2021-03-24T22:00:05.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:05.470 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:05.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:00:05.499 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-24T22:00:05.572 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:05.573 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:05 smithi142 conmon[20125]: cluster 2021-03-24T22:00:04.236218 2021-03-24T22:00:05.574 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:05 smithi142 conmon[20125]: +0000 mon.a (mon.0) 1323 : cluster [DBG] osdmap e434: 8 total, 8 up, 5 in 2021-03-24T22:00:05.574 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:05 smithi142 conmon[20125]: audit 2021-03-24T22:00:04.791244+0000 mon.a ( 2021-03-24T22:00:05.574 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:05 smithi142 conmon[20125]: mon.0) 1324 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T22:00:05.575 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:05 smithi142 conmon[20125]: audit 2021-03-24T22:00:04.791632+0000 mon.a 2021-03-24T22:00:05.575 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:05 smithi142 conmon[20125]: (mon.0) 1325 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T22:00:05.575 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:05 smithi142 conmon[20125]: cluster 2021-03-24T22:00:05.575 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:05 smithi142 conmon[20125]: 2021-03-24T22:00:05 2021-03-24T22:00:05.575 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:05 smithi142 conmon[20125]: .219988+0000 mon.a (mon.0) 1326 : cluster [WRN] 2021-03-24T22:00:05.576 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:05 smithi142 conmon[20125]: Health check update: Degraded data redundancy: 143/1569 objects degraded (9.114%), 21 pgs degraded, 47 pgs undersized (PG_DEGRADED) 2021-03-24T22:00:05.576 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:05 smithi142 conmon[20125]: cluster 2021-03-24T22:00: 2021-03-24T22:00:05.576 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:05 smithi142 conmon[20125]: 05.225795+0000 mon.a (mon.0) 1327 : cluster [DBG] 2021-03-24T22:00:05.576 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:05 smithi142 conmon[20125]: osdmap e435: 8 total, 8 up, 5 in 2021-03-24T22:00:05.636 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:05.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:05.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:05.741 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:05.791 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:05.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:05.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:00:05.830 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2021-03-24T22:00:05.831 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2021-03-24T22:00:05.831 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2021-03-24T22:00:05.831 INFO:teuthology.orchestra.run.smithi142.stdout:} 2021-03-24T22:00:05.898 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:05.945 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:05.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:05.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:00:05.971 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:00:06.049 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:06.095 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:06.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:06.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:00:06.199 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:06.240 INFO:tasks.rados.rados.0.smithi125.stdout:1610: done (0 left) 2021-03-24T22:00:06.241 INFO:tasks.rados.rados.0.smithi125.stdout:1611: read oid 28 snap -1 2021-03-24T22:00:06.241 INFO:tasks.rados.rados.0.smithi125.stdout:1611: expect (ObjNum 501 snap 146 seq_num 501) 2021-03-24T22:00:06.242 INFO:tasks.rados.rados.0.smithi125.stdout:1612: write (excl) oid 13 current snap is 175 2021-03-24T22:00:06.242 INFO:tasks.rados.rados.0.smithi125.stdout:1612: seq_num 572 ranges {413604=568903,1488759=656862,2624893=591709} 2021-03-24T22:00:06.247 INFO:tasks.rados.rados.0.smithi125.stdout:1612: writing smithi12556470-13 from 413604 to 982507 tid 1 2021-03-24T22:00:06.266 INFO:tasks.rados.rados.0.smithi125.stdout:1612: writing smithi12556470-13 from 1488759 to 2145621 tid 2 2021-03-24T22:00:06.269 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:06.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:06.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:00:06.279 INFO:tasks.rados.rados.0.smithi125.stdout:1612: writing smithi12556470-13 from 2624893 to 3216602 tid 3 2021-03-24T22:00:06.280 INFO:tasks.rados.rados.0.smithi125.stdout:1613: read oid 7 snap -1 2021-03-24T22:00:06.280 INFO:tasks.rados.rados.0.smithi125.stdout:1613: expect deleted 2021-03-24T22:00:06.280 INFO:tasks.rados.rados.0.smithi125.stdout:1614: read oid 31 snap -1 2021-03-24T22:00:06.281 INFO:tasks.rados.rados.0.smithi125.stdout:1614: expect deleted 2021-03-24T22:00:06.281 INFO:tasks.rados.rados.0.smithi125.stdout:1615: write (excl) oid 43 current snap is 175 2021-03-24T22:00:06.281 INFO:tasks.rados.rados.0.smithi125.stdout:1615: seq_num 573 ranges {430411=644152,1830888=400520,2935963=402981} 2021-03-24T22:00:06.287 INFO:tasks.rados.rados.0.smithi125.stdout:1615: writing smithi12556470-43 from 430411 to 1074563 tid 1 2021-03-24T22:00:06.294 INFO:tasks.rados.rados.0.smithi125.stdout:1615: writing smithi12556470-43 from 1830888 to 2231408 tid 2 2021-03-24T22:00:06.302 INFO:tasks.rados.rados.0.smithi125.stdout:1615: writing smithi12556470-43 from 2935963 to 3338944 tid 3 2021-03-24T22:00:06.302 INFO:tasks.rados.rados.0.smithi125.stdout:1616: snap_remove snap 147 2021-03-24T22:00:06.331 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[19936]: cluster 2021-03-24T22:00:04.850066+0000 mgr.y (mgr.14144) 488 : cluster [DBG] pgmap v814: 177 pgs: 3 active+recovering+degraded, 19 remapped+peering, 37 peering, 19 active+clean+remapped, 11 active+undersized, 1 active+remapped+backfill_wait, 5 active+undersized+degraded, 82 active+clean; 428 MiB data, 970 MiB used, 446 GiB / 447 GiB avail; 18 MiB/s rd, 2.8 MiB/s wr, 21 op/s; 38/1566 objects degraded (2.427%); 104/1566 objects misplaced (6.641%); 19 MiB/s, 12 objects/s recovering 2021-03-24T22:00:06.331 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[19936]: cluster 2021-03-24T22:00:06.234858+0000 mon.a (mon.0) 1328 : cluster [DBG] osdmap e436: 8 total, 8 up, 5 in 2021-03-24T22:00:06.332 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: cluster 2021-03-24T22:00:06.332 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: 2021-03-24T22:00:06.333 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: 2021 2021-03-24T22:00:06.333 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: - 2021-03-24T22:00:06.333 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: 03 2021-03-24T22:00:06.334 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: - 2021-03-24T22:00:06.334 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: 24 2021-03-24T22:00:06.335 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: T 2021-03-24T22:00:06.335 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: 22 2021-03-24T22:00:06.336 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: : 2021-03-24T22:00:06.336 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: 00 2021-03-24T22:00:06.336 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: : 2021-03-24T22:00:06.337 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: 04 2021-03-24T22:00:06.337 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: . 2021-03-24T22:00:06.338 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: 850066 2021-03-24T22:00:06.339 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: +0000 2021-03-24T22:00:06.339 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: 2021-03-24T22:00:06.340 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: mgr.y 2021-03-24T22:00:06.340 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: ( 2021-03-24T22:00:06.341 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: mgr 2021-03-24T22:00:06.341 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: . 2021-03-24T22:00:06.342 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: 14144 2021-03-24T22:00:06.342 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: ) 2021-03-24T22:00:06.342 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: 488 2021-03-24T22:00:06.343 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: : 2021-03-24T22:00:06.343 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: cluster 2021-03-24T22:00:06.344 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: 2021-03-24T22:00:06.344 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: [DBG] 2021-03-24T22:00:06.345 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: 2021-03-24T22:00:06.345 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: pgmap v814: 177 pgs: 3 active+recovering+degraded, 19 remapped+peering, 37 peering, 19 active+clean+remapped, 11 active+undersized, 1 active+remapped+backfill_wait, 5 active+undersized+degraded, 82 active+clean; 428 MiB data, 970 MiB used, 446 GiB / 447 GiB avail; 18 MiB/s rd, 2.8 MiB/s wr, 21 op/s; 38/1566 objects degraded (2.427%); 104/1566 objects misplaced (6.641%); 19 MiB/s, 12 objects/s recovering 2021-03-24T22:00:06.346 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: 2021-03-24T22:00:06.346 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: cluster 2021-03-24T22:00:06.346 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: 2021-03-24T22:00:06.347 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: 2021 2021-03-24T22:00:06.347 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: - 2021-03-24T22:00:06.348 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: 03 2021-03-24T22:00:06.348 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: - 2021-03-24T22:00:06.349 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: 24 2021-03-24T22:00:06.349 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: T 2021-03-24T22:00:06.349 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: 22 2021-03-24T22:00:06.350 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: : 2021-03-24T22:00:06.350 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: 00 2021-03-24T22:00:06.351 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: : 2021-03-24T22:00:06.351 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: 06 2021-03-24T22:00:06.352 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: . 2021-03-24T22:00:06.352 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: 234858 2021-03-24T22:00:06.353 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: +0000 2021-03-24T22:00:06.353 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: 2021-03-24T22:00:06.353 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: mon.a 2021-03-24T22:00:06.354 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: ( 2021-03-24T22:00:06.354 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: mon 2021-03-24T22:00:06.355 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: . 2021-03-24T22:00:06.355 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: 0 2021-03-24T22:00:06.355 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: ) 2021-03-24T22:00:06.356 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: 1328 2021-03-24T22:00:06.356 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: : 2021-03-24T22:00:06.357 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: cluster 2021-03-24T22:00:06.357 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: 2021-03-24T22:00:06.358 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: [DBG] 2021-03-24T22:00:06.358 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: 2021-03-24T22:00:06.358 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: osdmap e436: 8 total, 8 up, 5 in 2021-03-24T22:00:06.359 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:06 smithi125 conmon[24927]: 2021-03-24T22:00:06.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:06 smithi142 conmon[20125]: cluster 2021-03-24T22:00:04.850066+0000 2021-03-24T22:00:06.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:06 smithi142 conmon[20125]: mgr.y (mgr.14144) 488 : cluster [DBG] pgmap v814: 177 pgs: 3 active+recovering+degraded, 19 remapped+peering, 37 peering, 19 active+clean+remapped, 11 active+undersized, 1 active+remapped+backfill_wait, 5 active+undersized+degraded, 82 active+clean; 428 MiB data, 970 MiB used, 446 GiB / 447 GiB avail; 18 MiB/s rd, 2.8 MiB/s wr, 21 op/s; 38/1566 objects degraded (2.427%); 104/1566 objects misplaced (6.641%); 19 MiB/s, 12 objects/s recovering 2021-03-24T22:00:06.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:06 smithi142 conmon[20125]: cluster 2021-03-24T22:00:06.234858+0000 mon.a (mon.0) 1328 : cluster [DBG] osdmap e436: 8 total, 8 up, 5 in 2021-03-24T22:00:06.379 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:06.437 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:06.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:06.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:00:06.543 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:06.606 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:06.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:06.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:00:06.713 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:06.756 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:06.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:06.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:00:06.831 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.6 dump_blocked_ops 2021-03-24T22:00:06.844 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.5 dump_ops_in_flight 2021-03-24T22:00:06.861 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:06.909 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:06.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:06.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:00:07.014 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:07.063 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:07.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:07.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:00:07.142 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T22:00:07.169 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:07.209 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T22:00:07.219 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:07.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:07.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:00:07.275 INFO:tasks.rados.rados.0.smithi125.stdout:1616: done (5 left) 2021-03-24T22:00:07.276 INFO:tasks.rados.rados.0.smithi125.stdout:1617: copy_from oid 49 from oid 26 current snap is 175 2021-03-24T22:00:07.276 INFO:tasks.rados.rados.0.smithi125.stdout:1618: snap_remove snap 161 2021-03-24T22:00:07.279 INFO:teuthology.orchestra.run.smithi125.stderr:nodeep-scrub is set 2021-03-24T22:00:07.323 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:07.368 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:07.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:07.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:00:07.473 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:07.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:07 smithi125 conmon[19936]: audit 2021-03-24 2021-03-24T22:00:07.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:07 smithi125 conmon[19936]: T22:00:06.300820+0000 mon.a (mon.0) 1329 : audit [INF] from='client.? v1:172.21.15.125:0/3268927874' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:00:07.487 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:07 smithi125 conmon[24927]: audit 2021- 2021-03-24T22:00:07.487 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:07 smithi125 conmon[24927]: 03-24T22:00:06.300820+0000 2021-03-24T22:00:07.487 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:07 smithi125 conmon[24927]: mon.a (mon.0) 2021-03-24T22:00:07.487 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:07 smithi125 conmon[24927]: 1329 : audit [INF] 2021-03-24T22:00:07.488 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:07 smithi125 conmon[24927]: from='client.? v1:172.21.15.125:0/3268927874' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:00:07.519 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:07.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:07.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:07.567 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:07 smithi142 conmon[20125]: audit 2021-03-24T22:00:06.300820+0000 mon.a (mon.0) 1329 2021-03-24T22:00:07.568 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:07 smithi142 conmon[20125]: : audit [INF] from='client.? v1:172.21.15.125:0/3268927874' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:00:07.625 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:07.679 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:07.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:07.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:00:07.785 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:07.833 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:07.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:07.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:00:07.937 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:07.986 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:07.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:07.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:00:08.091 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:08.140 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:08.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:08.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:00:08.247 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:08.282 INFO:tasks.rados.rados.0.smithi125.stdout:1618: done (6 left) 2021-03-24T22:00:08.282 INFO:tasks.rados.rados.0.smithi125.stdout:1619: write oid 10 current snap is 175 2021-03-24T22:00:08.282 INFO:tasks.rados.rados.0.smithi125.stdout:1619: seq_num 574 ranges {447218=719401,1889370=660531,2963386=614253} 2021-03-24T22:00:08.291 INFO:tasks.rados.rados.0.smithi125.stdout:1619: writing smithi12556470-10 from 447218 to 1166619 tid 1 2021-03-24T22:00:08.300 INFO:tasks.rados.rados.0.smithi125.stdout:1619: writing smithi12556470-10 from 1889370 to 2549901 tid 2 2021-03-24T22:00:08.307 INFO:tasks.rados.rados.0.smithi125.stdout:1619: writing smithi12556470-10 from 2963386 to 3577639 tid 3 2021-03-24T22:00:08.307 INFO:tasks.rados.rados.0.smithi125.stdout:1620: write (excl) oid 35 current snap is 175 2021-03-24T22:00:08.307 INFO:tasks.rados.rados.0.smithi125.stdout:1620: seq_num 575 ranges {464025=794650,1947852=404189,2990809=425525} 2021-03-24T22:00:08.315 INFO:tasks.rados.rados.0.smithi125.stdout:1620: writing smithi12556470-35 from 464025 to 1258675 tid 1 2021-03-24T22:00:08.321 INFO:tasks.rados.rados.0.smithi125.stdout:1620: writing smithi12556470-35 from 1947852 to 2352041 tid 2 2021-03-24T22:00:08.329 INFO:tasks.rados.rados.0.smithi125.stdout:1620: writing smithi12556470-35 from 2990809 to 3416334 tid 3 2021-03-24T22:00:08.329 INFO:tasks.rados.rados.0.smithi125.stdout:1621: snap_create 2021-03-24T22:00:08.329 INFO:tasks.rados.rados.0.smithi125.stdout:1611: expect (ObjNum 501 snap 146 seq_num 501) 2021-03-24T22:00:08.330 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:08.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:08.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:08.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:08 smithi142 conmon[20125]: cluster 2021-03-24T22:00: 2021-03-24T22:00:08.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:08 smithi142 conmon[20125]: 06.851111+0000 mgr.y (mgr.14144) 489 : cluster [DBG] pgmap v817: 177 pgs: 1 active+recovery_wait+degraded, 3 active+recovering+degraded, 15 remapped+peering, 37 peering, 11 active+clean+wait, 1 active+recovering, 1 active+remapped+backfill_wait, 108 active+clean; 428 MiB data, 1.0 GiB used, 446 GiB / 447 GiB avail; 9.3 MiB/s rd, 2.8 MiB/s wr, 15 op/s; 30/1566 objects degraded (1.916%); 24/1566 objects misplaced (1.533%); 27 MiB/s, 18 objects/s recovering 2021-03-24T22:00:08.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:08 smithi142 conmon[20125]: cluster 2021-03-24T22: 2021-03-24T22:00:08.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:08 smithi142 conmon[20125]: 00:07.259292+0000 mon.a (mon.0) 1330 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:00:08.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:08 smithi142 conmon[20125]: audit 2021-03- 2021-03-24T22:00:08.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:08 smithi142 conmon[20125]: 24T22:00:07.271572+0000 mon.a (mon.0) 1331 : audit [INF] from='client.? v1:172.21.15.125:0/3268927874' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T22:00:08.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:08 smithi142 conmon[20125]: cluster 2021-03-24T22: 2021-03-24T22:00:08.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:08 smithi142 conmon[20125]: 00:07.271683+0000 mon.a (mon.0) 1332 : cluster [DBG] 2021-03-24T22:00:08.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:08 smithi142 conmon[20125]: osdmap e437: 8 total, 8 up, 5 in 2021-03-24T22:00:08.442 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:08.481 INFO:tasks.rados.rados.0.smithi125.stdout:1612: finishing write tid 1 to smithi12556470-13 2021-03-24T22:00:08.481 INFO:tasks.rados.rados.0.smithi125.stdout:1612: finishing write tid 2 to smithi12556470-13 2021-03-24T22:00:08.482 INFO:tasks.rados.rados.0.smithi125.stdout:1612: finishing write tid 3 to smithi12556470-13 2021-03-24T22:00:08.482 INFO:tasks.rados.rados.0.smithi125.stdout:1612: finishing write tid 4 to smithi12556470-13 2021-03-24T22:00:08.482 INFO:tasks.rados.rados.0.smithi125.stdout:1612: finishing write tid 5 to smithi12556470-13 2021-03-24T22:00:08.483 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 13 v 288 (ObjNum 572 snap 175 seq_num 572) dirty exists 2021-03-24T22:00:08.483 INFO:tasks.rados.rados.0.smithi125.stdout:1612: left oid 13 (ObjNum 572 snap 175 seq_num 572) 2021-03-24T22:00:08.483 INFO:tasks.rados.rados.0.smithi125.stdout:1615: finishing write tid 1 to smithi12556470-43 2021-03-24T22:00:08.483 INFO:tasks.rados.rados.0.smithi125.stdout:1615: finishing write tid 2 to smithi12556470-43 2021-03-24T22:00:08.484 INFO:tasks.rados.rados.0.smithi125.stdout:1615: finishing write tid 3 to smithi12556470-43 2021-03-24T22:00:08.484 INFO:tasks.rados.rados.0.smithi125.stdout:1615: finishing write tid 4 to smithi12556470-43 2021-03-24T22:00:08.485 INFO:tasks.rados.rados.0.smithi125.stdout:1615: finishing write tid 5 to smithi12556470-43 2021-03-24T22:00:08.486 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 43 v 624 (ObjNum 573 snap 175 seq_num 573) dirty exists 2021-03-24T22:00:08.486 INFO:tasks.rados.rados.0.smithi125.stdout:1615: left oid 43 (ObjNum 573 snap 175 seq_num 573) 2021-03-24T22:00:08.486 INFO:tasks.rados.rados.0.smithi125.stdout:1617: finishing copy_from to smithi12556470-49 2021-03-24T22:00:08.487 INFO:tasks.rados.rados.0.smithi125.stdout:1617: got expected ENOENT (src dne) 2021-03-24T22:00:08.487 INFO:tasks.rados.rados.0.smithi125.stdout:1617: finishing copy_from racing read to smithi12556470-49 2021-03-24T22:00:08.488 INFO:tasks.rados.rados.0.smithi125.stdout:1619: finishing write tid 1 to smithi12556470-10 2021-03-24T22:00:08.488 INFO:tasks.rados.rados.0.smithi125.stdout:1619: finishing write tid 2 to smithi12556470-10 2021-03-24T22:00:08.488 INFO:tasks.rados.rados.0.smithi125.stdout:1619: finishing write tid 3 to smithi12556470-10 2021-03-24T22:00:08.489 INFO:tasks.rados.rados.0.smithi125.stdout:1619: finishing write tid 4 to smithi12556470-10 2021-03-24T22:00:08.489 INFO:tasks.rados.rados.0.smithi125.stdout:1619: finishing write tid 5 to smithi12556470-10 2021-03-24T22:00:08.489 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 10 v 489 (ObjNum 574 snap 175 seq_num 574) dirty exists 2021-03-24T22:00:08.490 INFO:tasks.rados.rados.0.smithi125.stdout:1619: left oid 10 (ObjNum 574 snap 175 seq_num 574) 2021-03-24T22:00:08.490 INFO:tasks.rados.rados.0.smithi125.stdout:1620: finishing write tid 1 to smithi12556470-35 2021-03-24T22:00:08.490 INFO:tasks.rados.rados.0.smithi125.stdout:1620: finishing write tid 2 to smithi12556470-35 2021-03-24T22:00:08.490 INFO:tasks.rados.rados.0.smithi125.stdout:1620: finishing write tid 3 to smithi12556470-35 2021-03-24T22:00:08.491 INFO:tasks.rados.rados.0.smithi125.stdout:1620: finishing write tid 4 to smithi12556470-35 2021-03-24T22:00:08.491 INFO:tasks.rados.rados.0.smithi125.stdout:1620: finishing write tid 5 to smithi12556470-35 2021-03-24T22:00:08.491 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 35 v 205 (ObjNum 575 snap 175 seq_num 575) dirty exists 2021-03-24T22:00:08.492 INFO:tasks.rados.rados.0.smithi125.stdout:1620: left oid 35 (ObjNum 575 snap 175 seq_num 575) 2021-03-24T22:00:08.493 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:08.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:08.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:00:08.596 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:08.657 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:08.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:08 smithi125 conmon[24927]: cluster 2021-03-24T22:00:08.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:08 smithi125 conmon[24927]: 2021-03-24T22:00:08.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:08 smithi125 conmon[24927]: 2021-03-24T22:00:06.851111+0000 mgr.y (mgr.14144) 489 : 2021-03-24T22:00:08.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:08 smithi125 conmon[24927]: cluster [DBG] pgmap v817: 177 pgs: 1 active+recovery_wait+degraded, 3 active+recovering+degraded, 15 remapped+peering, 37 peering, 11 active+clean+wait, 1 active+recovering, 1 active+remapped+backfill_wait, 108 active+clean; 428 MiB data, 1.0 GiB used, 446 GiB / 447 GiB avail; 9.3 MiB/s rd, 2.8 MiB/s wr, 15 op/s; 30/1566 objects degraded (1.916%); 24/1566 objects misplaced (1.533%); 27 MiB/s, 18 objects/s recovering 2021-03-24T22:00:08.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:08 smithi125 conmon[24927]: cluster 2021-03-24T22:00:07.259292+0000 mon.a (mon.0) 1330 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:00:08.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:08 smithi125 conmon[24927]: audit 2021-03-24T22:00:08.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:08 smithi125 conmon[24927]: 2021-03-24T22:00:07.271572+0000 mon.a (mon.0) 1331 : audit [INF] from='client.? v1:172.21.15.125:0/3268927874' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T22:00:08.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:08 smithi125 conmon[24927]: cluster 2021-03-24T22:00:07.271683+0000 mon.a (mon.0) 1332 : cluster [DBG] 2021-03-24T22:00:08.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:08 smithi125 conmon[24927]: osdmap e437: 8 total, 8 up, 5 in 2021-03-24T22:00:08.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:08 smithi125 conmon[19936]: cluster 2021-03-24T22:00:06.851111+0000 mgr.y 2021-03-24T22:00:08.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:08 smithi125 conmon[19936]: (mgr.14144) 489 : cluster [DBG] pgmap v817: 177 pgs: 1 active+recovery_wait+degraded, 3 active+recovering+degraded, 15 remapped+peering, 37 peering, 11 active+clean+wait, 1 active+recovering, 1 active+remapped+backfill_wait, 108 active+clean; 428 MiB data, 1.0 GiB used, 446 GiB / 447 GiB avail; 9.3 MiB/s rd, 2.8 MiB/s wr, 15 op/s; 30/1566 objects degraded (1.916%); 24/1566 objects misplaced (1.533%); 27 MiB/s, 18 objects/s recovering 2021-03-24T22:00:08.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:08 smithi125 conmon[19936]: cluster 2021-03-24T22:00:07.259292+0000 mon.a (mon.0 2021-03-24T22:00:08.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:08 smithi125 conmon[19936]: ) 1330 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:00:08.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:08 smithi125 conmon[19936]: audit 2021 2021-03-24T22:00:08.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:08 smithi125 conmon[19936]: -03-24T22:00:07.271572 2021-03-24T22:00:08.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:08 smithi125 conmon[19936]: +0000 mon.a (mon.0) 1331 : 2021-03-24T22:00:08.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:08 smithi125 conmon[19936]: audit [INF] from='client.? v1:172.21.15.125:0/3268927874' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T22:00:08.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:08 smithi125 conmon[19936]: cluster 2021-03-24T22:00:08.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:08 smithi125 conmon[19936]: 2021-03-24T22:00:07 2021-03-24T22:00:08.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:08 smithi125 conmon[19936]: .271683+0000 mon.a (mon.0) 2021-03-24T22:00:08.669 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:08 smithi125 conmon[19936]: 1332 : cluster [DBG] osdmap e437: 8 total, 8 up, 5 in 2021-03-24T22:00:08.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:08.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:00:08.771 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:08.824 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:08.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:08.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:00:08.931 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:08.979 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:08.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:08.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:09.086 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:09.096 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T22:00:09.119 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T22:00:09.137 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T22:00:09.148 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:09.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:09.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:00:09.158 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T22:00:09.189 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T22:00:09.212 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T22:00:09.236 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T22:00:09.256 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:09.267 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T22:00:09.269 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-24T22:00:09.269 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-24T22:00:09.269 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-24T22:00:09.270 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-24T22:00:09.270 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-24T22:00:09.270 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-24T22:00:09.270 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-24T22:00:09.271 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-24T22:00:09.292 INFO:tasks.rados.rados.0.smithi125.stdout:1611: done (8 left) 2021-03-24T22:00:09.292 INFO:tasks.rados.rados.0.smithi125.stdout:1612: done (7 left) 2021-03-24T22:00:09.292 INFO:tasks.rados.rados.0.smithi125.stdout:1613: done (6 left) 2021-03-24T22:00:09.293 INFO:tasks.rados.rados.0.smithi125.stdout:1614: done (5 left) 2021-03-24T22:00:09.293 INFO:tasks.rados.rados.0.smithi125.stdout:1615: done (4 left) 2021-03-24T22:00:09.294 INFO:tasks.rados.rados.0.smithi125.stdout:1617: done (3 left) 2021-03-24T22:00:09.294 INFO:tasks.rados.rados.0.smithi125.stdout:1619: done (2 left) 2021-03-24T22:00:09.294 INFO:tasks.rados.rados.0.smithi125.stdout:1620: done (1 left) 2021-03-24T22:00:09.294 INFO:tasks.rados.rados.0.smithi125.stdout:1621: done (0 left) 2021-03-24T22:00:09.294 INFO:tasks.rados.rados.0.smithi125.stdout:1622: write (excl) oid 6 current snap is 176 2021-03-24T22:00:09.295 INFO:tasks.rados.rados.0.smithi125.stdout:1622: seq_num 576 ranges {480832=469899,1606334=664200,3018232=662600} 2021-03-24T22:00:09.298 INFO:tasks.rados.rados.0.smithi125.stdout:1622: writing smithi12556470-6 from 480832 to 950731 tid 1 2021-03-24T22:00:09.306 INFO:tasks.rados.rados.0.smithi125.stdout:1622: writing smithi12556470-6 from 1606334 to 2270534 tid 2 2021-03-24T22:00:09.314 INFO:tasks.rados.rados.0.smithi125.stdout:1622: writing smithi12556470-6 from 3018232 to 3680832 tid 3 2021-03-24T22:00:09.315 INFO:tasks.rados.rados.0.smithi125.stdout:1623: write oid 31 current snap is 176 2021-03-24T22:00:09.315 INFO:tasks.rados.rados.0.smithi125.stdout:1623: seq_num 577 ranges {497639=545148,1548463=407858,2529302=564422,3528143=169496} 2021-03-24T22:00:09.316 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:09.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:09.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:00:09.320 INFO:tasks.rados.rados.0.smithi125.stdout:1623: writing smithi12556470-31 from 497639 to 1042787 tid 1 2021-03-24T22:00:09.325 INFO:tasks.rados.rados.0.smithi125.stdout:1623: writing smithi12556470-31 from 1548463 to 1956321 tid 2 2021-03-24T22:00:09.333 INFO:tasks.rados.rados.0.smithi125.stdout:1623: writing smithi12556470-31 from 2529302 to 3093724 tid 3 2021-03-24T22:00:09.337 INFO:tasks.rados.rados.0.smithi125.stdout:1623: writing smithi12556470-31 from 3528143 to 3697639 tid 4 2021-03-24T22:00:09.337 INFO:tasks.rados.rados.0.smithi125.stdout:1624: delete oid 42 current snap is 176 2021-03-24T22:00:09.337 INFO:tasks.rados.rados.0.smithi125.stdout:1622: finishing write tid 1 to smithi12556470-6 2021-03-24T22:00:09.338 INFO:tasks.rados.rados.0.smithi125.stdout:1622: finishing write tid 2 to smithi12556470-6 2021-03-24T22:00:09.338 INFO:tasks.rados.rados.0.smithi125.stdout:1622: finishing write tid 3 to smithi12556470-6 2021-03-24T22:00:09.339 INFO:tasks.rados.rados.0.smithi125.stdout:1622: finishing write tid 4 to smithi12556470-6 2021-03-24T22:00:09.339 INFO:tasks.rados.rados.0.smithi125.stdout:1622: finishing write tid 5 to smithi12556470-6 2021-03-24T22:00:09.339 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 6 v 392 (ObjNum 576 snap 176 seq_num 576) dirty exists 2021-03-24T22:00:09.340 INFO:tasks.rados.rados.0.smithi125.stdout:1622: left oid 6 (ObjNum 576 snap 176 seq_num 576) 2021-03-24T22:00:09.340 INFO:tasks.rados.rados.0.smithi125.stdout:1623: finishing write tid 1 to smithi12556470-31 2021-03-24T22:00:09.341 INFO:tasks.rados.rados.0.smithi125.stdout:1623: finishing write tid 2 to smithi12556470-31 2021-03-24T22:00:09.341 INFO:tasks.rados.rados.0.smithi125.stdout:1623: finishing write tid 3 to smithi12556470-31 2021-03-24T22:00:09.345 INFO:tasks.rados.rados.0.smithi125.stdout:1623: finishing write tid 4 to smithi12556470-31 2021-03-24T22:00:09.345 INFO:tasks.rados.rados.0.smithi125.stdout:1622: done (2 left) 2021-03-24T22:00:09.346 INFO:tasks.rados.rados.0.smithi125.stdout:1624: done (1 left) 2021-03-24T22:00:09.346 INFO:tasks.rados.rados.0.smithi125.stdout:1625: rollback oid 4 current snap is 176 2021-03-24T22:00:09.346 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 4 to 174 2021-03-24T22:00:09.346 INFO:tasks.rados.rados.0.smithi125.stdout:1626: write (excl) oid 23 current snap is 176 2021-03-24T22:00:09.347 INFO:tasks.rados.rados.0.smithi125.stdout:1626: seq_num 579 ranges {531253=411999,1381780=411527,2300501=586966,3376974=354279} 2021-03-24T22:00:09.347 INFO:tasks.rados.rados.0.smithi125.stdout:1626: writing smithi12556470-23 from 531253 to 943252 tid 1 2021-03-24T22:00:09.353 INFO:tasks.rados.rados.0.smithi125.stdout:1626: writing smithi12556470-23 from 1381780 to 1793307 tid 2 2021-03-24T22:00:09.359 INFO:tasks.rados.rados.0.smithi125.stdout:1626: writing smithi12556470-23 from 2300501 to 2887467 tid 3 2021-03-24T22:00:09.365 INFO:tasks.rados.rados.0.smithi125.stdout:1626: writing smithi12556470-23 from 3376974 to 3731253 tid 4 2021-03-24T22:00:09.365 INFO:tasks.rados.rados.0.smithi125.stdout:1627: write oid 8 current snap is 176 2021-03-24T22:00:09.365 INFO:tasks.rados.rados.0.smithi125.stdout:1623: finishing write tid 5 to smithi12556470-31 2021-03-24T22:00:09.366 INFO:tasks.rados.rados.0.smithi125.stdout:1623: finishing write tid 6 to smithi12556470-31 2021-03-24T22:00:09.366 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 31 v 645 (ObjNum 577 snap 176 seq_num 577) dirty exists 2021-03-24T22:00:09.366 INFO:tasks.rados.rados.0.smithi125.stdout:1623: left oid 31 (ObjNum 577 snap 176 seq_num 577) 2021-03-24T22:00:09.367 INFO:tasks.rados.rados.0.smithi125.stdout:1625: finishing rollback tid 0 to smithi12556470-4 2021-03-24T22:00:09.367 INFO:tasks.rados.rados.0.smithi125.stdout:1625: finishing rollback tid 1 to smithi12556470-4 2021-03-24T22:00:09.367 INFO:tasks.rados.rados.0.smithi125.stdout:1625: finishing rollback tid 2 to smithi12556470-4 2021-03-24T22:00:09.367 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 4 v 335 (ObjNum 544 snap 168 seq_num 544) dirty exists 2021-03-24T22:00:09.367 INFO:tasks.rados.rados.0.smithi125.stdout:1626: finishing write tid 1 to smithi12556470-23 2021-03-24T22:00:09.368 INFO:tasks.rados.rados.0.smithi125.stdout:1626: finishing write tid 2 to smithi12556470-23 2021-03-24T22:00:09.368 INFO:tasks.rados.rados.0.smithi125.stdout:1626: finishing write tid 3 to smithi12556470-23 2021-03-24T22:00:09.368 INFO:tasks.rados.rados.0.smithi125.stdout:1627: seq_num 580 ranges {548060=487248,1440262=671538,2844277=798238} 2021-03-24T22:00:09.370 INFO:tasks.rados.rados.0.smithi125.stdout:1627: writing smithi12556470-8 from 548060 to 1035308 tid 1 2021-03-24T22:00:09.377 INFO:tasks.rados.rados.0.smithi125.stdout:1627: writing smithi12556470-8 from 1440262 to 2111800 tid 2 2021-03-24T22:00:09.385 INFO:tasks.rados.rados.0.smithi125.stdout:1627: writing smithi12556470-8 from 2844277 to 3642515 tid 3 2021-03-24T22:00:09.386 INFO:tasks.rados.rados.0.smithi125.stdout:1623: done (3 left) 2021-03-24T22:00:09.386 INFO:tasks.rados.rados.0.smithi125.stdout:1625: done (2 left) 2021-03-24T22:00:09.386 INFO:tasks.rados.rados.0.smithi125.stdout:1628: snap_remove snap 136 2021-03-24T22:00:09.420 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:09.475 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:09.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:09.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:00:09.581 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:09.594 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:09 smithi125 conmon[24927]: cluster 2021-03-24T22:00:09.595 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:09 smithi125 conmon[24927]: 2021-03 2021-03-24T22:00:09.595 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:09 smithi125 conmon[24927]: -24T22:00:08.276047+0000 mon.a (mon.0) 1333 : cluster [DBG] osdmap e438: 8 total, 8 up, 5 in 2021-03-24T22:00:09.596 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:09 smithi125 conmon[24927]: audit 2021 2021-03-24T22:00:09.596 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:09 smithi125 conmon[24927]: -03-24T22:00:08.852719+0000 mon.a ( 2021-03-24T22:00:09.597 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:09 smithi125 conmon[24927]: mon.0) 1334 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "30"}]: dispatch 2021-03-24T22:00:09.598 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:09 smithi125 conmon[24927]: audit 2021- 2021-03-24T22:00:09.598 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:09 smithi125 conmon[24927]: 03-24T22:00:09.067885+0000 mon.c 2021-03-24T22:00:09.598 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:09 smithi125 conmon[24927]: (mon.2) 69 : audit [DBG] from='client.? v1:172.21.15.125:0/2984826543' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:00:09.599 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:09 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:00:09.599 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:09 smithi125 conmon[19936]: 03-24T22:00:08.276047+0000 mon.a (mon.0) 2021-03-24T22:00:09.599 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:09 smithi125 conmon[19936]: 1333 : cluster [DBG] 2021-03-24T22:00:09.600 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:09 smithi125 conmon[19936]: osdmap e438: 8 total, 8 up, 5 in 2021-03-24T22:00:09.600 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:09 smithi125 conmon[19936]: audit 2021-03-24T22:00:09.600 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:09 smithi125 conmon[19936]: 2021 2021-03-24T22:00:09.600 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:09 smithi125 conmon[19936]: -03-24T22:00: 2021-03-24T22:00:09.601 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:09 smithi125 conmon[19936]: 08.852719+0000 2021-03-24T22:00:09.601 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:09 smithi125 conmon[19936]: mon.a (mon. 2021-03-24T22:00:09.601 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:09 smithi125 conmon[19936]: 0) 1334 : audit 2021-03-24T22:00:09.601 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:09 smithi125 conmon[19936]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "30"}]: dispatch 2021-03-24T22:00:09.601 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:09 smithi125 conmon[19936]: audit 2021- 2021-03-24T22:00:09.602 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:09 smithi125 conmon[19936]: 03-24T22:00:09.067885+0000 mon.c ( 2021-03-24T22:00:09.602 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:09 smithi125 conmon[19936]: mon.2) 69 : audit 2021-03-24T22:00:09.602 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:09 smithi125 conmon[19936]: [DBG] from='client.? v1:172.21.15.125:0/2984826543' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:00:09.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:09 smithi142 conmon[20125]: cluster 2021-03-24T22:00:08.276047+0000 2021-03-24T22:00:09.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:09 smithi142 conmon[20125]: mon.a (mon.0) 1333 : cluster [DBG] osdmap e438: 8 total, 8 up, 5 in 2021-03-24T22:00:09.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:09 smithi142 conmon[20125]: audit 2021-03-24T22:00:08.852719+0000 mon.a (mon.0) 1334 : audit 2021-03-24T22:00:09.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:09 smithi142 conmon[20125]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "30"}]: dispatch 2021-03-24T22:00:09.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:09 smithi142 conmon[20125]: audit 2021-03-24T 2021-03-24T22:00:09.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:09 smithi142 conmon[20125]: 22:00:09.067885+0000 mon.c (mon.2) 69 : audit [DBG] from='client.? v1:172.21.15.125:0/2984826543' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:00:09.640 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:09.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:09.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:09.746 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:09.802 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:09.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:09.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:00:09.907 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:09.965 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:09.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:09.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:00:10.072 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:10.131 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:10.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:10.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:00:10.139 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2021-03-24T22:00:10.139 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T22:00:10.236 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:10.237 INFO:tasks.rados.rados.0.smithi125.stdout:1628: done (2 left) 2021-03-24T22:00:10.238 INFO:tasks.rados.rados.0.smithi125.stdout:1629: read oid 13 snap -1 2021-03-24T22:00:10.238 INFO:tasks.rados.rados.0.smithi125.stdout:1629: expect (ObjNum 572 snap 175 seq_num 572) 2021-03-24T22:00:10.238 INFO:tasks.rados.rados.0.smithi125.stdout:1630: delete oid 10 current snap is 176 2021-03-24T22:00:10.239 INFO:tasks.rados.rados.0.smithi125.stdout:1626: finishing write tid 4 to smithi12556470-23 2021-03-24T22:00:10.239 INFO:tasks.rados.rados.0.smithi125.stdout:1626: finishing write tid 5 to smithi12556470-23 2021-03-24T22:00:10.239 INFO:tasks.rados.rados.0.smithi125.stdout:1626: finishing write tid 6 to smithi12556470-23 2021-03-24T22:00:10.240 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 23 v 284 (ObjNum 579 snap 176 seq_num 579) dirty exists 2021-03-24T22:00:10.240 INFO:tasks.rados.rados.0.smithi125.stdout:1626: left oid 23 (ObjNum 579 snap 176 seq_num 579) 2021-03-24T22:00:10.240 INFO:tasks.rados.rados.0.smithi125.stdout:1627: finishing write tid 1 to smithi12556470-8 2021-03-24T22:00:10.241 INFO:tasks.rados.rados.0.smithi125.stdout:1627: finishing write tid 2 to smithi12556470-8 2021-03-24T22:00:10.241 INFO:tasks.rados.rados.0.smithi125.stdout:1627: finishing write tid 3 to smithi12556470-8 2021-03-24T22:00:10.242 INFO:tasks.rados.rados.0.smithi125.stdout:1627: finishing write tid 4 to smithi12556470-8 2021-03-24T22:00:10.243 INFO:tasks.rados.rados.0.smithi125.stdout:1627: finishing write tid 5 to smithi12556470-8 2021-03-24T22:00:10.243 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 8 v 91 (ObjNum 580 snap 176 seq_num 580) dirty exists 2021-03-24T22:00:10.244 INFO:tasks.rados.rados.0.smithi125.stdout:1627: left oid 8 (ObjNum 580 snap 176 seq_num 580) 2021-03-24T22:00:10.258 INFO:tasks.rados.rados.0.smithi125.stdout:1629: expect (ObjNum 572 snap 175 seq_num 572) 2021-03-24T22:00:10.301 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:10.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:10.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:10.360 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2021-03-24T22:00:10.360 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T22:00:10.388 INFO:tasks.rados.rados.0.smithi125.stdout:1626: done (3 left) 2021-03-24T22:00:10.388 INFO:tasks.rados.rados.0.smithi125.stdout:1627: done (2 left) 2021-03-24T22:00:10.388 INFO:tasks.rados.rados.0.smithi125.stdout:1629: done (1 left) 2021-03-24T22:00:10.389 INFO:tasks.rados.rados.0.smithi125.stdout:1630: done (0 left) 2021-03-24T22:00:10.389 INFO:tasks.rados.rados.0.smithi125.stdout:1631: rollback oid 5 current snap is 176 2021-03-24T22:00:10.389 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 5 to 148 2021-03-24T22:00:10.389 INFO:tasks.rados.rados.0.smithi125.stdout:1632: write oid 29 current snap is 176 2021-03-24T22:00:10.389 INFO:tasks.rados.rados.0.smithi125.stdout:1632: seq_num 582 ranges {581674=637746,1840873=558854,3066417=537135} 2021-03-24T22:00:10.398 INFO:tasks.rados.rados.0.smithi125.stdout:1632: writing smithi12556470-29 from 581674 to 1219420 tid 1 2021-03-24T22:00:10.404 INFO:tasks.rados.rados.0.smithi125.stdout:1632: writing smithi12556470-29 from 1840873 to 2399727 tid 2 2021-03-24T22:00:10.406 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:10.412 INFO:tasks.rados.rados.0.smithi125.stdout:1632: writing smithi12556470-29 from 3066417 to 3603552 tid 3 2021-03-24T22:00:10.412 INFO:tasks.rados.rados.0.smithi125.stdout:1631: finishing rollback tid 1 to smithi12556470-5 2021-03-24T22:00:10.413 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 5 v 374 (ObjNum 1953066355 snap 842099048 seq_num 875967797) dirty dne 2021-03-24T22:00:10.413 INFO:tasks.rados.rados.0.smithi125.stdout:1633: read oid 10 snap -1 2021-03-24T22:00:10.413 INFO:tasks.rados.rados.0.smithi125.stdout:1633: expect deleted 2021-03-24T22:00:10.413 INFO:tasks.rados.rados.0.smithi125.stdout:1631: done (2 left) 2021-03-24T22:00:10.414 INFO:tasks.rados.rados.0.smithi125.stdout:1634: read oid 47 snap -1 2021-03-24T22:00:10.414 INFO:tasks.rados.rados.0.smithi125.stdout:1634: expect (ObjNum 471 snap 133 seq_num 471) 2021-03-24T22:00:10.414 INFO:tasks.rados.rados.0.smithi125.stdout:1635: read oid 37 snap 149 2021-03-24T22:00:10.415 INFO:tasks.rados.rados.0.smithi125.stdout:1635: expect deleted 2021-03-24T22:00:10.415 INFO:tasks.rados.rados.0.smithi125.stdout:1636: delete oid 41 current snap is 176 2021-03-24T22:00:10.415 INFO:tasks.rados.rados.0.smithi125.stdout:1632: finishing write tid 1 to smithi12556470-29 2021-03-24T22:00:10.415 INFO:tasks.rados.rados.0.smithi125.stdout:1632: finishing write tid 2 to smithi12556470-29 2021-03-24T22:00:10.416 INFO:tasks.rados.rados.0.smithi125.stdout:1635: done (4 left) 2021-03-24T22:00:10.417 INFO:tasks.rados.rados.0.smithi125.stdout:1636: done (3 left) 2021-03-24T22:00:10.417 INFO:tasks.rados.rados.0.smithi125.stdout:1637: delete oid 50 current snap is 176 2021-03-24T22:00:10.417 INFO:tasks.rados.rados.0.smithi125.stdout:1632: finishing write tid 3 to smithi12556470-29 2021-03-24T22:00:10.418 INFO:tasks.rados.rados.0.smithi125.stdout:1632: finishing write tid 4 to smithi12556470-29 2021-03-24T22:00:10.419 INFO:tasks.rados.rados.0.smithi125.stdout:1632: finishing write tid 5 to smithi12556470-29 2021-03-24T22:00:10.419 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 29 v 220 (ObjNum 582 snap 176 seq_num 582) dirty exists 2021-03-24T22:00:10.420 INFO:tasks.rados.rados.0.smithi125.stdout:1632: left oid 29 (ObjNum 582 snap 176 seq_num 582) 2021-03-24T22:00:10.420 INFO:tasks.rados.rados.0.smithi125.stdout:1632: done (3 left) 2021-03-24T22:00:10.420 INFO:tasks.rados.rados.0.smithi125.stdout:1633: done (2 left) 2021-03-24T22:00:10.420 INFO:tasks.rados.rados.0.smithi125.stdout:1637: done (1 left) 2021-03-24T22:00:10.420 INFO:tasks.rados.rados.0.smithi125.stdout:1638: read oid 23 snap 170 2021-03-24T22:00:10.421 INFO:tasks.rados.rados.0.smithi125.stdout:1638: expect (ObjNum 427 snap 113 seq_num 427) 2021-03-24T22:00:10.421 INFO:tasks.rados.rados.0.smithi125.stdout:1639: copy_from oid 8 from oid 26 current snap is 176 2021-03-24T22:00:10.421 INFO:tasks.rados.rados.0.smithi125.stdout:1640: snap_create 2021-03-24T22:00:10.422 INFO:tasks.rados.rados.0.smithi125.stdout:1639: finishing copy_from to smithi12556470-8 2021-03-24T22:00:10.422 INFO:tasks.rados.rados.0.smithi125.stdout:1639: got expected ENOENT (src dne) 2021-03-24T22:00:10.423 INFO:tasks.rados.rados.0.smithi125.stdout:1639: finishing copy_from racing read to smithi12556470-8 2021-03-24T22:00:10.423 INFO:tasks.rados.rados.0.smithi125.stdout:1634: expect (ObjNum 471 snap 133 seq_num 471) 2021-03-24T22:00:10.456 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:10.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:10.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:00:10.560 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:10.574 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:10 smithi142 conmon[20125]: cluster 2021-03-24T22: 2021-03-24T22:00:10.574 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:10 smithi142 conmon[20125]: 00:08.851712+0000 mgr.y (mgr.14144) 490 : cluster [DBG] pgmap v820: 177 pgs: 1 active+recovery_wait+degraded, 3 active+recovering+degraded, 1 remapped+peering, 24 peering, 11 active+clean+wait, 2 active+recovering, 1 active+remapped+backfill_wait, 134 active+clean; 429 MiB data, 1.0 GiB used, 446 GiB / 447 GiB avail; 3.2 MiB/s rd, 733 KiB/s wr, 5 op/s; 30/1569 objects degraded (1.912%); 25/1569 objects misplaced (1.593%); 12 MiB/s, 12 objects/s recovering 2021-03-24T22:00:10.574 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:10 smithi142 conmon[20125]: audit 2021-03-24T22:00:09.289450+0000 mon.a (mon 2021-03-24T22:00:10.575 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:10 smithi142 conmon[20125]: .0) 1335 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "30"}]': finished 2021-03-24T22:00:10.575 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:10 smithi142 conmon[20125]: cluster 2021-03-24T22:00:10.575 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:10 smithi142 conmon[20125]: 2021-03-24T 2021-03-24T22:00:10.576 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:10 smithi142 conmon[20125]: 22:00:09.289558+0000 mon.a (mon. 2021-03-24T22:00:10.576 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:10 smithi142 conmon[20125]: 0) 1336 : cluster [DBG] osdmap e439: 8 total, 8 up, 5 in 2021-03-24T22:00:10.576 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:10 smithi142 conmon[20125]: cluster 2021-03-24T22:00:10.576 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:10 smithi142 conmon[20125]: 2021-03-24T22:00:10.222158+0000 2021-03-24T22:00:10.577 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:10 smithi142 conmon[20125]: mon.a (mon.0) 1337 : cluster [WRN] Health check update: Degraded data redundancy: 30/1569 objects degraded (1.912%), 4 pgs degraded (PG_DEGRADED) 2021-03-24T22:00:10.577 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:10 smithi142 conmon[20125]: 2021-03-24T22:00:10.577 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:10 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:00:10.577 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:10 smithi142 conmon[20125]: -03-24T22:00:10.232353+0000 2021-03-24T22:00:10.578 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:10 smithi142 conmon[20125]: mon.a (mon.0) 1338 : cluster [DBG] 2021-03-24T22:00:10.578 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:10 smithi142 conmon[20125]: osdmap e440: 8 total, 8 up, 5 in 2021-03-24T22:00:10.613 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:10.614 INFO:tasks.rados.rados.0.smithi125.stdout:1638: expect (ObjNum 427 snap 113 seq_num 427) 2021-03-24T22:00:10.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:10.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:00:10.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:10 smithi125 conmon[19936]: cluster 2021-03-24T22:00:08.851712+0000 mgr.y (mgr.14144) 490 : cluster [DBG] pgmap v820: 177 pgs: 1 active+recovery_wait+degraded, 3 active+recovering+degraded, 1 remapped+peering, 24 peering, 11 active+clean+wait, 2 active+recovering, 1 active+remapped+backfill_wait, 134 active+clean; 429 MiB data, 1.0 GiB used, 446 GiB / 447 GiB avail; 3.2 MiB/s rd, 733 KiB/s wr, 5 op/s; 30/1569 objects degraded (1.912%); 25/1569 objects misplaced (1.593%); 12 MiB/s, 12 objects/s recovering 2021-03-24T22:00:10.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:10 smithi125 conmon[19936]: audit 2021-03-24T22:00:09.289450+0000 mon.a (mon.0) 1335 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "30"}]': finished 2021-03-24T22:00:10.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:10 smithi125 conmon[19936]: cluster 2021-03-24T22:00 2021-03-24T22:00:10.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:10 smithi125 conmon[19936]: :09.289558+0000 mon.a (mon.0) 1336 : cluster [DBG] osdmap e439: 8 total, 8 up, 5 in 2021-03-24T22:00:10.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:10 smithi125 conmon[19936]: cluster 2021-03-24T22:00:10.222158+0000 mon.a (mon.0) 1337 : cluster [WRN] Health check update: Degraded data redundancy: 30/1569 objects degraded (1.912%), 4 pgs degraded (PG_DEGRADED) 2021-03-24T22:00:10.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:10 smithi125 conmon[19936]: cluster 2021-03-24T22:00:10.232353+0000 mon.a ( 2021-03-24T22:00:10.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:10 smithi125 conmon[19936]: mon.0) 1338 : cluster [DBG] osdmap e440: 8 total, 8 up, 5 in 2021-03-24T22:00:10.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:10 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T22:00:10.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:10 smithi125 conmon[24927]: -24T22:00:08.851712+0000 mgr.y (mgr 2021-03-24T22:00:10.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:10 smithi125 conmon[24927]: .14144) 490 : 2021-03-24T22:00:10.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:10 smithi125 conmon[24927]: cluster [DBG] pgmap v820: 177 pgs: 1 active+recovery_wait+degraded, 3 active+recovering+degraded, 1 remapped+peering, 24 peering, 11 active+clean+wait, 2 active+recovering, 1 active+remapped+backfill_wait, 134 active+clean; 429 MiB data, 1.0 GiB used, 446 GiB / 447 GiB avail; 3.2 MiB/s rd, 733 KiB/s wr, 5 op/s; 30/1569 objects degraded (1.912%); 25/1569 objects misplaced (1.593%); 12 MiB/s, 12 objects/s recovering 2021-03-24T22:00:10.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:10 smithi125 conmon[24927]: 2021-03-24T22:00:10.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:10 smithi125 conmon[24927]: audit 2021-03-24T22:00:10.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:10 smithi125 conmon[24927]: 2021- 2021-03-24T22:00:10.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:10 smithi125 conmon[24927]: 03-24T22:00 2021-03-24T22:00:10.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:10 smithi125 conmon[24927]: :09.289450+0000 2021-03-24T22:00:10.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:10 smithi125 conmon[24927]: mon.a (mon. 2021-03-24T22:00:10.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:10 smithi125 conmon[24927]: 0) 1335 : audit 2021-03-24T22:00:10.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:10 smithi125 conmon[24927]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "30"}]': finished 2021-03-24T22:00:10.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:10 smithi125 conmon[24927]: cluster 2021-03-24T22:00:10.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:10 smithi125 conmon[24927]: 2021 2021-03-24T22:00:10.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:10 smithi125 conmon[24927]: -03-24T22: 2021-03-24T22:00:10.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:10 smithi125 conmon[24927]: 00:09.289558 2021-03-24T22:00:10.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:10 smithi125 conmon[24927]: +0000 mon.a (mon 2021-03-24T22:00:10.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:10 smithi125 conmon[24927]: .0) 1336 : 2021-03-24T22:00:10.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:10 smithi125 conmon[24927]: cluster [DBG] osdmap e439: 8 total, 8 up, 5 in 2021-03-24T22:00:10.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:10 smithi125 conmon[24927]: 2021-03-24T22:00:10.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:10 smithi125 conmon[24927]: cluster 2021-03-24T22:00:10.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:10 smithi125 conmon[24927]: 2021 2021-03-24T22:00:10.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:10 smithi125 conmon[24927]: -03-24T22: 2021-03-24T22:00:10.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:10 smithi125 conmon[24927]: 00:10.222158 2021-03-24T22:00:10.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:10 smithi125 conmon[24927]: +0000 mon.a (mon 2021-03-24T22:00:10.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:10 smithi125 conmon[24927]: .0) 1337 : cluster 2021-03-24T22:00:10.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:10 smithi125 conmon[24927]: [WRN] Health check update: Degraded data redundancy: 30/1569 objects degraded (1.912%), 4 pgs degraded (PG_DEGRADED) 2021-03-24T22:00:10.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:10 smithi125 conmon[24927]: cluster 2021-03-24T22:00:10.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:10 smithi125 conmon[24927]: 2021-03-24T22:00:10.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:10 smithi125 conmon[24927]: 2021 2021-03-24T22:00:10.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:10 smithi125 conmon[24927]: - 2021-03-24T22:00:10.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:10 smithi125 conmon[24927]: 03-24T 2021-03-24T22:00:10.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:10 smithi125 conmon[24927]: 22:00:10.232353+0000 mon.a 2021-03-24T22:00:10.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:10 smithi125 conmon[24927]: (mon.0) 2021-03-24T22:00:10.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:10 smithi125 conmon[24927]: 1338 : cluster [DBG] 2021-03-24T22:00:10.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:10 smithi125 conmon[24927]: osdmap e440: 8 total, 8 up, 5 in 2021-03-24T22:00:10.696 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd unset noscrub 2021-03-24T22:00:10.719 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:10.771 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:10.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:10.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:00:10.875 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:10.933 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:10.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:10.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:00:11.016 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:00:11.039 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:11.093 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:11.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:11.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:00:11.199 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:11.256 INFO:tasks.rados.rados.0.smithi125.stdout:1634: done (3 left) 2021-03-24T22:00:11.256 INFO:tasks.rados.rados.0.smithi125.stdout:1638: done (2 left) 2021-03-24T22:00:11.256 INFO:tasks.rados.rados.0.smithi125.stdout:1639: done (1 left) 2021-03-24T22:00:11.257 INFO:tasks.rados.rados.0.smithi125.stdout:1640: done (0 left) 2021-03-24T22:00:11.257 INFO:tasks.rados.rados.0.smithi125.stdout:1641: delete oid 24 current snap is 177 2021-03-24T22:00:11.260 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:11.263 INFO:tasks.rados.rados.0.smithi125.stdout:1641: done (0 left) 2021-03-24T22:00:11.263 INFO:tasks.rados.rados.0.smithi125.stdout:1642: snap_remove snap 148 2021-03-24T22:00:11.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:11.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:11.367 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:11.417 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:11.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:11.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:00:11.523 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:11.583 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:11.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:11.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:00:11.687 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:11.743 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:11.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:11.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:11.849 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:11.857 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.6 dump_historic_ops 2021-03-24T22:00:11.900 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:11.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:11.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:00:12.006 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:12.057 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:12.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:12.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:00:12.167 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:12.201 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T22:00:12.216 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:12.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:12.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:00:12.262 INFO:tasks.rados.rados.0.smithi125.stdout:1642: done (0 left) 2021-03-24T22:00:12.262 INFO:tasks.rados.rados.0.smithi125.stdout:1643: read oid 44 snap -1 2021-03-24T22:00:12.262 INFO:tasks.rados.rados.0.smithi125.stdout:1643: expect deleted 2021-03-24T22:00:12.263 INFO:tasks.rados.rados.0.smithi125.stdout:1644: read oid 8 snap 151 2021-03-24T22:00:12.263 INFO:tasks.rados.rados.0.smithi125.stdout:1644: expect deleted 2021-03-24T22:00:12.263 INFO:tasks.rados.rados.0.smithi125.stdout:1645: rollback oid 30 current snap is 177 2021-03-24T22:00:12.264 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 30 to 150 2021-03-24T22:00:12.264 INFO:tasks.rados.rados.0.smithi125.stdout:1646: snap_remove snap 173 2021-03-24T22:00:12.322 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:12.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:12 smithi142 conmon[20125]: cluster 2021-03-24T22: 2021-03-24T22:00:12.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:12 smithi142 conmon[20125]: 00:10.852254+0000 mgr.y (mgr.14144) 491 : cluster [DBG] pgmap v823: 177 pgs: 1 active+recovery_wait+degraded, 11 active+clean+wait, 1 active+undersized+remapped, 2 active+recovering, 162 active+clean; 437 MiB data, 1.0 GiB used, 446 GiB / 447 GiB avail; 4.9 MiB/s rd, 2.3 MiB/s wr, 12 op/s; 19/1569 objects degraded (1.211%); 5/1569 objects misplaced (0.319%); 22 MiB/s, 33 objects/s recovering 2021-03-24T22:00:12.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:12 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:00:12.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:12 smithi142 conmon[20125]: 03-24T22:00 2021-03-24T22:00:12.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:12 smithi142 conmon[20125]: :11.250433+0000 mon.a 2021-03-24T22:00:12.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:12 smithi142 conmon[20125]: (mon.0) 1339 : 2021-03-24T22:00:12.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:12 smithi142 conmon[20125]: cluster [DBG] osdmap e441: 8 total, 8 up, 5 in 2021-03-24T22:00:12.374 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:12.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:12.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:00:12.442 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-24T22:00:12.480 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:12.507 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:12 smithi125 conmon[19936]: cluster 2021-03-24T22:00 2021-03-24T22:00:12.507 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:12 smithi125 conmon[19936]: :10.852254+0000 mgr.y (mgr.14144) 491 : cluster [DBG] pgmap v823: 177 pgs: 1 active+recovery_wait+degraded, 11 active+clean+wait, 1 active+undersized+remapped, 2 active+recovering, 162 active+clean; 437 MiB data, 1.0 GiB used, 446 GiB / 447 GiB avail; 4.9 MiB/s rd, 2.3 MiB/s wr, 12 op/s; 19/1569 objects degraded (1.211%); 5/1569 objects misplaced (0.319%); 22 MiB/s, 33 objects/s recovering 2021-03-24T22:00:12.508 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:12 smithi125 conmon[19936]: cluster 2021-03-24T22:00:12.508 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:12 smithi125 conmon[19936]: 2021-03-24T22:00:11.250433+0000 2021-03-24T22:00:12.508 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:12 smithi125 conmon[19936]: mon.a (mon.0) 1339 : cluster [DBG] osdmap e441: 8 total, 8 up, 5 in 2021-03-24T22:00:12.508 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:12 smithi125 conmon[19936]: 2021-03-24T22:00:12.509 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:12 smithi125 conmon[24927]: cluster 2021-03-24T22:00 2021-03-24T22:00:12.509 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:12 smithi125 conmon[24927]: :10.852254+0000 mgr.y (mgr.14144) 491 : cluster [DBG] pgmap v823: 177 pgs: 1 active+recovery_wait+degraded, 11 active+clean+wait, 1 active+undersized+remapped, 2 active+recovering, 162 active+clean; 437 MiB data, 1.0 GiB used, 446 GiB / 447 GiB avail; 4.9 MiB/s rd, 2.3 MiB/s wr, 12 op/s; 19/1569 objects degraded (1.211%); 5/1569 objects misplaced (0.319%); 22 MiB/s, 33 objects/s recovering 2021-03-24T22:00:12.509 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:12 smithi125 conmon[24927]: cluster 2021-03-24T22:00:11. 2021-03-24T22:00:12.509 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:12 smithi125 conmon[24927]: 250433+0000 mon.a (mon.0) 1339 : cluster [DBG] 2021-03-24T22:00:12.510 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:12 smithi125 conmon[24927]: osdmap e441: 8 total, 8 up, 5 in 2021-03-24T22:00:12.540 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:12.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:12.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:00:12.646 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:12.701 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:12.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:12.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:00:12.811 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:12.862 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:12.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:12.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:12.965 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:00:12.968 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:13.018 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:13.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:13.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:13.122 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:13.177 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:13.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:13.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:00:13.284 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:13.286 INFO:tasks.rados.rados.0.smithi125.stdout:1646: done (3 left) 2021-03-24T22:00:13.286 INFO:tasks.rados.rados.0.smithi125.stdout:1647: write (excl) oid 13 current snap is 177 2021-03-24T22:00:13.286 INFO:tasks.rados.rados.0.smithi125.stdout:1647: seq_num 586 ranges {648902=655095,2074801=566192,3176109=672793} 2021-03-24T22:00:13.287 INFO:tasks.rados.rados.0.smithi125.stdout:1647: writing smithi12556470-13 from 648902 to 1303997 tid 1 2021-03-24T22:00:13.290 INFO:teuthology.orchestra.run.smithi125.stderr:noscrub is unset 2021-03-24T22:00:13.295 INFO:tasks.rados.rados.0.smithi125.stdout:1647: writing smithi12556470-13 from 2074801 to 2640993 tid 2 2021-03-24T22:00:13.303 INFO:tasks.rados.rados.0.smithi125.stdout:1647: writing smithi12556470-13 from 3176109 to 3848902 tid 3 2021-03-24T22:00:13.303 INFO:tasks.rados.rados.0.smithi125.stdout:1648: delete oid 3 current snap is 177 2021-03-24T22:00:13.303 INFO:tasks.rados.rados.0.smithi125.stdout:1645: finishing rollback tid 0 to smithi12556470-30 2021-03-24T22:00:13.304 INFO:tasks.rados.rados.0.smithi125.stdout:1645: finishing rollback tid 1 to smithi12556470-30 2021-03-24T22:00:13.304 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 30 v 289 (ObjNum 1 snap 32741 seq_num 1117600496) dirty dne 2021-03-24T22:00:13.304 INFO:tasks.rados.rados.0.smithi125.stdout:1647: finishing write tid 1 to smithi12556470-13 2021-03-24T22:00:13.304 INFO:tasks.rados.rados.0.smithi125.stdout:1647: finishing write tid 2 to smithi12556470-13 2021-03-24T22:00:13.305 INFO:tasks.rados.rados.0.smithi125.stdout:1643: done (4 left) 2021-03-24T22:00:13.305 INFO:tasks.rados.rados.0.smithi125.stdout:1644: done (3 left) 2021-03-24T22:00:13.306 INFO:tasks.rados.rados.0.smithi125.stdout:1645: done (2 left) 2021-03-24T22:00:13.306 INFO:tasks.rados.rados.0.smithi125.stdout:1648: done (1 left) 2021-03-24T22:00:13.306 INFO:tasks.rados.rados.0.smithi125.stdout:1649: rollback oid 3 current snap is 177 2021-03-24T22:00:13.306 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 3 to 139 2021-03-24T22:00:13.307 INFO:tasks.rados.rados.0.smithi125.stdout:1650: write oid 18 current snap is 177 2021-03-24T22:00:13.307 INFO:tasks.rados.rados.0.smithi125.stdout:1650: seq_num 588 ranges {682516=405593,1791765=569861,2830955=721120} 2021-03-24T22:00:13.311 INFO:tasks.rados.rados.0.smithi125.stdout:1650: writing smithi12556470-18 from 682516 to 1088109 tid 1 2021-03-24T22:00:13.318 INFO:tasks.rados.rados.0.smithi125.stdout:1650: writing smithi12556470-18 from 1791765 to 2361626 tid 2 2021-03-24T22:00:13.326 INFO:tasks.rados.rados.0.smithi125.stdout:1650: writing smithi12556470-18 from 2830955 to 3552075 tid 3 2021-03-24T22:00:13.327 INFO:tasks.rados.rados.0.smithi125.stdout:1651: rollback oid 2 current snap is 177 2021-03-24T22:00:13.328 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 2 to 151 2021-03-24T22:00:13.328 INFO:tasks.rados.rados.0.smithi125.stdout:1647: finishing write tid 3 to smithi12556470-13 2021-03-24T22:00:13.328 INFO:tasks.rados.rados.0.smithi125.stdout:1647: finishing write tid 4 to smithi12556470-13 2021-03-24T22:00:13.328 INFO:tasks.rados.rados.0.smithi125.stdout:1647: finishing write tid 5 to smithi12556470-13 2021-03-24T22:00:13.328 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 13 v 295 (ObjNum 586 snap 177 seq_num 586) dirty exists 2021-03-24T22:00:13.329 INFO:tasks.rados.rados.0.smithi125.stdout:1647: left oid 13 (ObjNum 586 snap 177 seq_num 586) 2021-03-24T22:00:13.329 INFO:tasks.rados.rados.0.smithi125.stdout:1649: finishing rollback tid 1 to smithi12556470-3 2021-03-24T22:00:13.329 INFO:tasks.rados.rados.0.smithi125.stdout:1649: finishing rollback tid 2 to smithi12556470-3 2021-03-24T22:00:13.329 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 3 v 299 (ObjNum 442 snap 119 seq_num 442) dirty exists 2021-03-24T22:00:13.329 INFO:tasks.rados.rados.0.smithi125.stdout:1650: finishing write tid 1 to smithi12556470-18 2021-03-24T22:00:13.330 INFO:tasks.rados.rados.0.smithi125.stdout:1650: finishing write tid 2 to smithi12556470-18 2021-03-24T22:00:13.330 INFO:tasks.rados.rados.0.smithi125.stdout:1647: done (3 left) 2021-03-24T22:00:13.330 INFO:tasks.rados.rados.0.smithi125.stdout:1649: done (2 left) 2021-03-24T22:00:13.330 INFO:tasks.rados.rados.0.smithi125.stdout:1652: rollback oid 26 current snap is 177 2021-03-24T22:00:13.331 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 26 to 157 2021-03-24T22:00:13.331 INFO:tasks.rados.rados.0.smithi125.stdout:1653: delete oid 46 current snap is 177 2021-03-24T22:00:13.332 INFO:tasks.rados.rados.0.smithi125.stdout:1652: finishing rollback tid 1 to smithi12556470-26 2021-03-24T22:00:13.332 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 26 v 436 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-03-24T22:00:13.332 INFO:tasks.rados.rados.0.smithi125.stdout:1652: done (3 left) 2021-03-24T22:00:13.332 INFO:tasks.rados.rados.0.smithi125.stdout:1653: done (2 left) 2021-03-24T22:00:13.332 INFO:tasks.rados.rados.0.smithi125.stdout:1654: snap_remove snap 116 2021-03-24T22:00:13.338 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:13.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:13.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:00:13.442 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:13.453 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:13 smithi142 conmon[20125]: cluster 2021-03-24T22:00:13.453 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:13 smithi142 conmon[20125]: 2021-03-24T 2021-03-24T22:00:13.454 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:13 smithi142 conmon[20125]: 22:00:11.902916+0000 osd.0 (osd.0) 94 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:00:13.454 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:13 smithi142 conmon[20125]: cluster 2021-03-24T22 2021-03-24T22:00:13.454 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:13 smithi142 conmon[20125]: :00:11.903420+0000 osd.0 (osd.0) 95 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:00:13.454 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:13 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:00:13.454 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:13 smithi142 conmon[20125]: 24T22:00:12.258420+0000 mon.a (mon.0) 1340 : cluster [DBG] osdmap e442: 8 total, 8 up, 5 in 2021-03-24T22:00:13.455 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:13 smithi142 conmon[20125]: audit 2021-03-24T22:00:12. 2021-03-24T22:00:13.455 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:13 smithi142 conmon[20125]: 855248+0000 mon.a (mon.0) 1341 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "29"}]: dispatch 2021-03-24T22:00:13.455 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:13 smithi142 conmon[20125]: audit 2021 2021-03-24T22:00:13.455 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:13 smithi142 conmon[20125]: -03-24T22:00:13.195651+0000 mon.a (mon.0 2021-03-24T22:00:13.456 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:13 smithi142 conmon[20125]: ) 1342 : audit [INF] from='client.? v1:172.21.15.125:0/2759362218' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T22:00:13.495 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:13.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:13.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:00:13.535 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:13 smithi125 conmon[19936]: cluster 2021-03-24T22:00:13.535 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:13 smithi125 conmon[19936]: 2021-03-24T22:00:11.902916+0000 2021-03-24T22:00:13.535 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:13 smithi125 conmon[19936]: osd.0 (osd.0) 94 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:00:13.536 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:13 smithi125 conmon[19936]: cluster 2021-03-24T22:00:13.536 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:13 smithi125 conmon[19936]: 2021-03-24T22:00:11.903420+0000 osd.0 (osd.0) 95 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:00:13.536 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:13 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T22:00:13.536 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:13 smithi125 conmon[19936]: 22:00:12.258420+0000 mon.a (mon.0) 1340 2021-03-24T22:00:13.537 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:13 smithi125 conmon[19936]: : cluster [DBG] osdmap e442: 8 total, 8 up, 5 in 2021-03-24T22:00:13.537 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:13 smithi125 conmon[19936]: audit 2021- 2021-03-24T22:00:13.537 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:13 smithi125 conmon[19936]: 03-24T22:00:12.855248+0000 mon.a (mon.0) 1341 : audit 2021-03-24T22:00:13.537 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:13 smithi125 conmon[19936]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "29"}]: dispatch 2021-03-24T22:00:13.538 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:13 smithi125 conmon[19936]: audit 2021-03-24 2021-03-24T22:00:13.538 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:13 smithi125 conmon[19936]: T22:00:13.195651+0000 mon.a (mon.0) 1342 : audit [INF] from='client.? v1:172.21.15.125:0/2759362218' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T22:00:13.538 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:13 smithi125 conmon[24927]: cluster 2021-03-24T22:00 2021-03-24T22:00:13.539 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:13 smithi125 conmon[24927]: :11.902916+0000 osd.0 (osd.0) 94 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:00:13.539 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:13 smithi125 conmon[24927]: cluster 2021-03-24T22:00:13.539 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:13 smithi125 conmon[24927]: 2021-03-24T22:00 2021-03-24T22:00:13.539 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:13 smithi125 conmon[24927]: :11.903420+0000 osd.0 (osd.0) 95 2021-03-24T22:00:13.540 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:13 smithi125 conmon[24927]: : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:00:13.540 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:13 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:00:13.540 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:13 smithi125 conmon[24927]: 03-24T22:00:12.258420+0000 mon.a (mon.0) 1340 2021-03-24T22:00:13.540 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:13 smithi125 conmon[24927]: : cluster [DBG] osdmap e442: 8 total, 8 up, 5 in 2021-03-24T22:00:13.541 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:13 smithi125 conmon[24927]: audit 2021- 2021-03-24T22:00:13.541 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:13 smithi125 conmon[24927]: 03-24T22:00:12.855248+0000 mon.a 2021-03-24T22:00:13.541 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:13 smithi125 conmon[24927]: (mon.0) 1341 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "29"}]: dispatch 2021-03-24T22:00:13.541 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:13 smithi125 conmon[24927]: audit 2021-03-24T22:00:13.541 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:13 smithi125 conmon[24927]: 2021-03-24T22:00:13.195651+0000 2021-03-24T22:00:13.542 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:13 smithi125 conmon[24927]: mon.a (mon.0) 1342 : audit [INF] 2021-03-24T22:00:13.542 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:13 smithi125 conmon[24927]: from='client.? v1:172.21.15.125:0/2759362218' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T22:00:13.601 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:13.664 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:13.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:13.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:00:13.769 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:13.826 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:13.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:13.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:00:13.933 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:13.994 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:13.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:13.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:00:14.100 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:14.133 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2021-03-24T22:00:14.133 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T22:00:14.158 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:14.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:14.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:14.264 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:14.284 INFO:tasks.rados.rados.0.smithi125.stdout:1654: done (2 left) 2021-03-24T22:00:14.284 INFO:tasks.rados.rados.0.smithi125.stdout:1655: rollback oid 16 current snap is 177 2021-03-24T22:00:14.285 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 16 to 129 2021-03-24T22:00:14.285 INFO:tasks.rados.rados.0.smithi125.stdout:1656: read oid 26 snap -1 2021-03-24T22:00:14.285 INFO:tasks.rados.rados.0.smithi125.stdout:1656: expect deleted 2021-03-24T22:00:14.285 INFO:tasks.rados.rados.0.smithi125.stdout:1657: write (excl) oid 21 current snap is 177 2021-03-24T22:00:14.286 INFO:tasks.rados.rados.0.smithi125.stdout:1657: seq_num 590 ranges {716130=556091,1792376=573530,2769448=460017,3821956=94174} 2021-03-24T22:00:14.286 INFO:tasks.rados.rados.0.smithi125.stdout:1657: writing smithi12556470-21 from 716130 to 1272221 tid 1 2021-03-24T22:00:14.295 INFO:tasks.rados.rados.0.smithi125.stdout:1657: writing smithi12556470-21 from 1792376 to 2365906 tid 2 2021-03-24T22:00:14.300 INFO:tasks.rados.rados.0.smithi125.stdout:1657: writing smithi12556470-21 from 2769448 to 3229465 tid 3 2021-03-24T22:00:14.305 INFO:tasks.rados.rados.0.smithi125.stdout:1657: writing smithi12556470-21 from 3821956 to 3916130 tid 4 2021-03-24T22:00:14.306 INFO:tasks.rados.rados.0.smithi125.stdout:1651: finishing rollback tid 1 to smithi12556470-2 2021-03-24T22:00:14.306 INFO:tasks.rados.rados.0.smithi125.stdout:1651: finishing rollback tid 2 to smithi12556470-2 2021-03-24T22:00:14.307 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 2 v 542 (ObjNum 506 snap 147 seq_num 506) dirty exists 2021-03-24T22:00:14.307 INFO:tasks.rados.rados.0.smithi125.stdout:1650: finishing write tid 3 to smithi12556470-18 2021-03-24T22:00:14.308 INFO:tasks.rados.rados.0.smithi125.stdout:1650: finishing write tid 4 to smithi12556470-18 2021-03-24T22:00:14.308 INFO:tasks.rados.rados.0.smithi125.stdout:1650: finishing write tid 5 to smithi12556470-18 2021-03-24T22:00:14.309 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 18 v 203 (ObjNum 588 snap 177 seq_num 588) dirty exists 2021-03-24T22:00:14.309 INFO:tasks.rados.rados.0.smithi125.stdout:1650: left oid 18 (ObjNum 588 snap 177 seq_num 588) 2021-03-24T22:00:14.309 INFO:tasks.rados.rados.0.smithi125.stdout:1655: finishing rollback tid 0 to smithi12556470-16 2021-03-24T22:00:14.310 INFO:tasks.rados.rados.0.smithi125.stdout:1655: finishing rollback tid 1 to smithi12556470-16 2021-03-24T22:00:14.310 INFO:tasks.rados.rados.0.smithi125.stdout:1657: finishing write tid 1 to smithi12556470-21 2021-03-24T22:00:14.311 INFO:tasks.rados.rados.0.smithi125.stdout:1650: done (4 left) 2021-03-24T22:00:14.311 INFO:tasks.rados.rados.0.smithi125.stdout:1651: done (3 left) 2021-03-24T22:00:14.311 INFO:tasks.rados.rados.0.smithi125.stdout:1656: done (2 left) 2021-03-24T22:00:14.311 INFO:tasks.rados.rados.0.smithi125.stdout:1658: write (excl) oid 19 current snap is 177 2021-03-24T22:00:14.312 INFO:tasks.rados.rados.0.smithi125.stdout:1658: seq_num 591 ranges {732937=631340,1850858=433541,2796871=671289,3888195=44742} 2021-03-24T22:00:14.313 INFO:tasks.rados.rados.0.smithi125.stdout:1658: writing smithi12556470-19 from 732937 to 1364277 tid 1 2021-03-24T22:00:14.319 INFO:tasks.rados.rados.0.smithi125.stdout:1658: writing smithi12556470-19 from 1850858 to 2284399 tid 2 2021-03-24T22:00:14.324 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:14.326 INFO:tasks.rados.rados.0.smithi125.stdout:1658: writing smithi12556470-19 from 2796871 to 3468160 tid 3 2021-03-24T22:00:14.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:14.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:00:14.329 INFO:tasks.rados.rados.0.smithi125.stdout:1658: writing smithi12556470-19 from 3888195 to 3932937 tid 4 2021-03-24T22:00:14.329 INFO:tasks.rados.rados.0.smithi125.stdout:1659: write (excl) oid 7 current snap is 177 2021-03-24T22:00:14.330 INFO:tasks.rados.rados.0.smithi125.stdout:1655: finishing rollback tid 2 to smithi12556470-16 2021-03-24T22:00:14.330 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 16 v 386 (ObjNum 453 snap 126 seq_num 453) dirty exists 2021-03-24T22:00:14.330 INFO:tasks.rados.rados.0.smithi125.stdout:1659: seq_num 592 ranges {749744=706589,1909340=577199,3224294=482561} 2021-03-24T22:00:14.337 INFO:tasks.rados.rados.0.smithi125.stdout:1659: writing smithi12556470-7 from 749744 to 1456333 tid 1 2021-03-24T22:00:14.343 INFO:tasks.rados.rados.0.smithi125.stdout:1659: writing smithi12556470-7 from 1909340 to 2486539 tid 2 2021-03-24T22:00:14.351 INFO:tasks.rados.rados.0.smithi125.stdout:1659: writing smithi12556470-7 from 3224294 to 3706855 tid 3 2021-03-24T22:00:14.351 INFO:tasks.rados.rados.0.smithi125.stdout:1655: done (3 left) 2021-03-24T22:00:14.351 INFO:tasks.rados.rados.0.smithi125.stdout:1660: rollback oid 2 current snap is 177 2021-03-24T22:00:14.352 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 2 to 159 2021-03-24T22:00:14.352 INFO:tasks.rados.rados.0.smithi125.stdout:1657: finishing write tid 2 to smithi12556470-21 2021-03-24T22:00:14.352 INFO:tasks.rados.rados.0.smithi125.stdout:1657: finishing write tid 3 to smithi12556470-21 2021-03-24T22:00:14.352 INFO:tasks.rados.rados.0.smithi125.stdout:1657: finishing write tid 4 to smithi12556470-21 2021-03-24T22:00:14.352 INFO:tasks.rados.rados.0.smithi125.stdout:1657: finishing write tid 5 to smithi12556470-21 2021-03-24T22:00:14.353 INFO:tasks.rados.rados.0.smithi125.stdout:1657: finishing write tid 6 to smithi12556470-21 2021-03-24T22:00:14.353 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 21 v 410 (ObjNum 590 snap 177 seq_num 590) dirty exists 2021-03-24T22:00:14.353 INFO:tasks.rados.rados.0.smithi125.stdout:1657: left oid 21 (ObjNum 590 snap 177 seq_num 590) 2021-03-24T22:00:14.353 INFO:tasks.rados.rados.0.smithi125.stdout:1658: finishing write tid 1 to smithi12556470-19 2021-03-24T22:00:14.354 INFO:tasks.rados.rados.0.smithi125.stdout:1658: finishing write tid 2 to smithi12556470-19 2021-03-24T22:00:14.354 INFO:tasks.rados.rados.0.smithi125.stdout:1661: read oid 1 snap -1 2021-03-24T22:00:14.354 INFO:tasks.rados.rados.0.smithi125.stdout:1661: expect (ObjNum 473 snap 134 seq_num 473) 2021-03-24T22:00:14.354 INFO:tasks.rados.rados.0.smithi125.stdout:1657: done (4 left) 2021-03-24T22:00:14.354 INFO:tasks.rados.rados.0.smithi125.stdout:1662: read oid 43 snap -1 2021-03-24T22:00:14.355 INFO:tasks.rados.rados.0.smithi125.stdout:1662: expect (ObjNum 573 snap 175 seq_num 573) 2021-03-24T22:00:14.355 INFO:tasks.rados.rados.0.smithi125.stdout:1663: snap_remove snap 139 2021-03-24T22:00:14.429 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:14.442 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:14 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:00:14.442 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:14 smithi142 conmon[20125]: T22:00:12.853018+0000 mgr.y (mgr.14144) 492 : cluster [DBG] pgmap v826: 176 pgs: 11 active+clean+wait, 1 active+undersized+remapped, 1 active+recovering, 163 active+clean; 443 MiB data, 1.1 GiB used, 446 GiB / 447 GiB avail; 2.8 MiB/s wr, 3 op/s; 1/1551 objects misplaced (0.064%); 24 MiB/s, 30 objects/s recovering 2021-03-24T22:00:14.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:14 smithi142 conmon[20125]: cluster 2021-03-24T22:00:14.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:14 smithi142 conmon[20125]: 2021-03 2021-03-24T22:00:14.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:14 smithi142 conmon[20125]: -24T22:00:13. 2021-03-24T22:00:14.443 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:14 smithi142 conmon[20125]: 252380+0000 mon.a (mon. 2021-03-24T22:00:14.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:14 smithi142 conmon[20125]: 0) 1343 : cluster 2021-03-24T22:00:14.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:14 smithi142 conmon[20125]: [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 19/1569 objects degraded (1.211%), 1 pg degraded) 2021-03-24T22:00:14.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:14 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:00:14.444 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:14 smithi142 conmon[20125]: 03-24T22:00:13. 2021-03-24T22:00:14.445 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:14 smithi142 conmon[20125]: 253217+0000 mon.a (mon. 2021-03-24T22:00:14.445 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:14 smithi142 conmon[20125]: 0) 1344 : cluster 2021-03-24T22:00:14.445 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:14 smithi142 conmon[20125]: [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:00:14.445 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:14 smithi142 conmon[20125]: audit 2021-03-24T22:00:14.446 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:14 smithi142 conmon[20125]: 2021-03 2021-03-24T22:00:14.446 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:14 smithi142 conmon[20125]: -24T22:00:13 2021-03-24T22:00:14.446 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:14 smithi142 conmon[20125]: .265849+0000 mon.a ( 2021-03-24T22:00:14.446 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:14 smithi142 conmon[20125]: mon.0) 1345 : audit 2021-03-24T22:00:14.446 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:14 smithi142 conmon[20125]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "29"}]': finished 2021-03-24T22:00:14.447 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:14 smithi142 conmon[20125]: audit 2021- 2021-03-24T22:00:14.447 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:14 smithi142 conmon[20125]: 03-24T22:00:13.265969+0000 mon.a ( 2021-03-24T22:00:14.447 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:14 smithi142 conmon[20125]: mon.0) 1346 : audit [INF] 2021-03-24T22:00:14.447 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:14 smithi142 conmon[20125]: from='client.? v1:172.21.15.125:0/2759362218' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T22:00:14.448 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:14 smithi142 conmon[20125]: cluster 2021-03-24T22:00:13.266307+0000 2021-03-24T22:00:14.448 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:14 smithi142 conmon[20125]: mon.a (mon.0) 1347 : cluster [DBG] osdmap e443: 8 total, 8 up, 5 in 2021-03-24T22:00:14.490 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:14.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:14.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:00:14.595 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:14.626 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:14 smithi125 conmon[19936]: cluster 2021-03-24T22:00:12.853018 2021-03-24T22:00:14.627 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:14 smithi125 conmon[19936]: +0000 mgr.y (mgr.14144) 492 : cluster [DBG] pgmap v826: 176 pgs: 11 active+clean+wait, 1 active+undersized+remapped, 1 active+recovering, 163 active+clean; 443 MiB data, 1.1 GiB used, 446 GiB / 447 GiB avail; 2.8 MiB/s wr, 3 op/s; 1/1551 objects misplaced (0.064%); 24 MiB/s, 30 objects/s recovering 2021-03-24T22:00:14.627 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:14 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:00:14.627 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:14 smithi125 conmon[19936]: T22:00:13.252380+0000 mon.a (mon.0) 1343 2021-03-24T22:00:14.628 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:14 smithi125 conmon[19936]: : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 19/1569 objects degraded (1.211%), 1 pg degraded) 2021-03-24T22:00:14.628 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:14 smithi125 conmon[19936]: cluster 2021-03-24T22:00:13.253217 2021-03-24T22:00:14.628 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:14 smithi125 conmon[19936]: +0000 mon.a (mon.0) 1344 : cluster [WRN] 2021-03-24T22:00:14.628 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:14 smithi125 conmon[19936]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:00:14.628 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:14 smithi125 conmon[19936]: audit 2021-03-24T22: 2021-03-24T22:00:14.629 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:14 smithi125 conmon[19936]: 00:13.265849+0000 mon.a (mon.0) 1345 2021-03-24T22:00:14.629 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:14 smithi125 conmon[19936]: : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "29"}]': finished 2021-03-24T22:00:14.629 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:14 smithi125 conmon[19936]: audit 2021 2021-03-24T22:00:14.629 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:14 smithi125 conmon[19936]: -03-24T22:00:13 2021-03-24T22:00:14.630 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:14 smithi125 conmon[19936]: .265969+0000 mon.a (mon. 2021-03-24T22:00:14.630 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:14 smithi125 conmon[19936]: 0) 1346 : audit [INF] from='client.? v1:172.21.15.125:0/2759362218' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T22:00:14.630 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:14 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:00:14.630 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:14 smithi125 conmon[19936]: 24T22:00:13.266307+0000 mon.a 2021-03-24T22:00:14.631 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:14 smithi125 conmon[19936]: (mon.0) 1347 : cluster [DBG] osdmap e443: 8 total, 8 up, 5 in 2021-03-24T22:00:14.631 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:14 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:00:14.631 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:14 smithi125 conmon[24927]: 24T22:00:12.853018+0000 mgr.y (mgr.14144) 492 : cluster [DBG] pgmap v826: 176 pgs: 11 active+clean+wait, 1 active+undersized+remapped, 1 active+recovering, 163 active+clean; 443 MiB data, 1.1 GiB used, 446 GiB / 447 GiB avail; 2.8 MiB/s wr, 3 op/s; 1/1551 objects misplaced (0.064%); 24 MiB/s, 30 objects/s recovering 2021-03-24T22:00:14.632 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:14 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:00:14.632 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:14 smithi125 conmon[24927]: -03-24T22:00:13.252380+0000 mon.a 2021-03-24T22:00:14.632 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:14 smithi125 conmon[24927]: (mon.0) 1343 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 19/1569 objects degraded (1.211%), 1 pg degraded) 2021-03-24T22:00:14.632 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:14 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:00:14.632 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:14 smithi125 conmon[24927]: 03-24T22:00:13.253217+0000 2021-03-24T22:00:14.633 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:14 smithi125 conmon[24927]: mon.a (mon.0) 1344 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:00:14.633 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:14 smithi125 conmon[24927]: 2021-03-24T22:00:14.633 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:14 smithi125 conmon[24927]: audit 2021-03-24T22:00:14.634 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:14 smithi125 conmon[24927]: 2021-03-24T22:00:13.265849+0000 mon.a (mon.0 2021-03-24T22:00:14.634 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:14 smithi125 conmon[24927]: ) 1345 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "29"}]': finished 2021-03-24T22:00:14.634 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:14 smithi125 conmon[24927]: audit 2021-03-24 2021-03-24T22:00:14.635 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:14 smithi125 conmon[24927]: T22:00:13.265969+0000 mon.a 2021-03-24T22:00:14.635 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:14 smithi125 conmon[24927]: (mon.0) 1346 : audit [INF] from='client.? v1:172.21.15.125:0/2759362218' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T22:00:14.635 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:14 smithi125 conmon[24927]: 2021-03-24T22:00:14.635 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:14 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:00:14.636 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:14 smithi125 conmon[24927]: -03-24T22:00:13.266307+0000 mon.a ( 2021-03-24T22:00:14.636 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:14 smithi125 conmon[24927]: mon.0) 1347 : cluster [DBG] osdmap e443: 8 total, 8 up, 5 in 2021-03-24T22:00:14.654 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:14.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:14.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:00:14.759 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:14.819 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:14.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:14.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:00:14.925 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:14.980 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:14.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:14.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:00:15.086 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:15.148 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:15.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:15.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:00:15.161 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.7 dump_ops_in_flight 2021-03-24T22:00:15.244 INFO:tasks.rados.rados.0.smithi125.stdout:1663: done (5 left) 2021-03-24T22:00:15.245 INFO:tasks.rados.rados.0.smithi125.stdout:1664: read oid 9 snap -1 2021-03-24T22:00:15.245 INFO:tasks.rados.rados.0.smithi125.stdout:1664: expect (ObjNum 520 snap 154 seq_num 520) 2021-03-24T22:00:15.245 INFO:tasks.rados.rados.0.smithi125.stdout:1665: snap_remove snap 128 2021-03-24T22:00:15.252 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:15.304 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:15.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:15.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:00:15.322 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[24927]: cluster 2021-03-24T22:00:15.323 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[24927]: 2021-03-24T22:00:15.323 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[24927]: 2021 2021-03-24T22:00:15.323 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[24927]: - 2021-03-24T22:00:15.323 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[24927]: 03 2021-03-24T22:00:15.324 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[24927]: - 2021-03-24T22:00:15.324 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[24927]: 24 2021-03-24T22:00:15.324 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[24927]: T 2021-03-24T22:00:15.324 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[24927]: 22 2021-03-24T22:00:15.325 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[24927]: : 2021-03-24T22:00:15.325 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[24927]: 00 2021-03-24T22:00:15.325 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[24927]: : 2021-03-24T22:00:15.325 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[24927]: 13 2021-03-24T22:00:15.325 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[24927]: .347145+0000 osd.3 (osd.3) 2021-03-24T22:00:15.326 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[24927]: 95 : cluster [DBG] 4.11 scrub ok 2021-03-24T22:00:15.326 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[24927]: cluster 2021-03-24T22:00:15.327 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[24927]: 2021-03-24T22:00:13.624928+0000 osd.0 (osd.0) 96 : cluster [DBG] 3.1f scrub ok 2021-03-24T22:00:15.327 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:00:15.327 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[24927]: 24T22:00:13.834275+0000 osd.4 (osd 2021-03-24T22:00:15.327 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[24927]: .4) 103 : cluster [DBG] 6.e scrub ok 2021-03-24T22:00:15.328 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T22:00:15.328 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[24927]: T22:00:14.276231+0000 mon.a (mon 2021-03-24T22:00:15.328 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[24927]: .0) 1348 : cluster [DBG] osdmap e444: 8 total, 8 up, 5 in 2021-03-24T22:00:15.328 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[24927]: audit 2021-03-24 2021-03-24T22:00:15.328 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[24927]: T22:00:14.786934+0000 mon.a (mon.0 2021-03-24T22:00:15.329 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[24927]: ) 1349 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:00:15.329 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:00:15.329 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[24927]: 00:15.237887+0000 mon.a (mon.0) 1350 2021-03-24T22:00:15.329 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[24927]: : cluster [DBG] osdmap e445: 8 total, 8 up, 5 in 2021-03-24T22:00:15.330 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[19936]: cluster 2021-03-24T22:00:15.330 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[19936]: 2021-03-24T22:00:15.330 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[19936]: 2021 2021-03-24T22:00:15.331 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[19936]: - 2021-03-24T22:00:15.331 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[19936]: 03 2021-03-24T22:00:15.331 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[19936]: - 2021-03-24T22:00:15.331 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[19936]: 24 2021-03-24T22:00:15.331 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[19936]: T 2021-03-24T22:00:15.332 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[19936]: 22 2021-03-24T22:00:15.332 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[19936]: : 2021-03-24T22:00:15.332 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[19936]: 00 2021-03-24T22:00:15.332 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[19936]: : 2021-03-24T22:00:15.332 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[19936]: 13 2021-03-24T22:00:15.333 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[19936]: . 2021-03-24T22:00:15.333 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[19936]: 347145 2021-03-24T22:00:15.333 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[19936]: +0000 2021-03-24T22:00:15.333 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[19936]: 2021-03-24T22:00:15.334 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[19936]: osd.3 2021-03-24T22:00:15.334 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[19936]: ( 2021-03-24T22:00:15.334 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[19936]: osd 2021-03-24T22:00:15.334 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[19936]: . 2021-03-24T22:00:15.335 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[19936]: 3 2021-03-24T22:00:15.335 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[19936]: ) 2021-03-24T22:00:15.335 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[19936]: 95 2021-03-24T22:00:15.335 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[19936]: : 2021-03-24T22:00:15.335 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[19936]: cluster 2021-03-24T22:00:15.336 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[19936]: 2021-03-24T22:00:15.336 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[19936]: [DBG] 2021-03-24T22:00:15.336 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[19936]: 2021-03-24T22:00:15.337 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[19936]: 4.11 scrub ok 2021-03-24T22:00:15.337 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[19936]: 2021-03-24T22:00:15.337 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:00:15.337 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[19936]: 00:13.624928+0000 osd.0 (osd.0) 96 : cluster [DBG] 3.1f scrub ok 2021-03-24T22:00:15.337 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:00:15.338 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[19936]: 03-24T22:00:13.834275+0000 osd.4 (osd.4) 103 : cluster [DBG] 6.e scrub ok 2021-03-24T22:00:15.338 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[19936]: cluster 2021-03-24T22:00:14.276231+0000 mon.a ( 2021-03-24T22:00:15.338 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[19936]: mon.0) 1348 : cluster [DBG] osdmap e444: 8 total, 8 up, 5 in 2021-03-24T22:00:15.338 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[19936]: audit 2021-03-24T22:00 2021-03-24T22:00:15.339 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[19936]: :14.786934+0000 mon.a (mon.0) 1349 : audit [INF] 2021-03-24T22:00:15.339 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:00:15.339 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[19936]: cluster 2021-03-24T22:00:15.339 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[19936]: 2021-03-24T22:00:15.237887+0000 mon.a (mon.0 2021-03-24T22:00:15.339 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:15 smithi125 conmon[19936]: ) 1350 : cluster [DBG] osdmap e445: 8 total, 8 up, 5 in 2021-03-24T22:00:15.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:15 smithi142 conmon[20125]: cluster 2021-03-24T22:00:13. 2021-03-24T22:00:15.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:15 smithi142 conmon[20125]: 347145+0000 osd.3 (osd.3) 95 : cluster [DBG] 4.11 scrub ok 2021-03-24T22:00:15.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:15 smithi142 conmon[20125]: cluster 2021-03-24T22:00:13.624928+0000 osd.0 (osd.0) 96 : cluster 2021-03-24T22:00:15.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:15 smithi142 conmon[20125]: [DBG] 3.1f scrub ok 2021-03-24T22:00:15.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:15 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:00:15.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:15 smithi142 conmon[20125]: 24T22:00:13.834275+0000 osd.4 (osd.4) 103 : cluster 2021-03-24T22:00:15.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:15 smithi142 conmon[20125]: [DBG] 6.e scrub ok 2021-03-24T22:00:15.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:15 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:00:15.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:15 smithi142 conmon[20125]: T22:00:14.276231+0000 mon.a (mon. 2021-03-24T22:00:15.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:15 smithi142 conmon[20125]: 0) 1348 : cluster [DBG] osdmap e444: 8 total, 8 up, 5 in 2021-03-24T22:00:15.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:15 smithi142 conmon[20125]: audit 2021-03-24 2021-03-24T22:00:15.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:15 smithi142 conmon[20125]: T22:00:14.786934+0000 mon.a (mon.0 2021-03-24T22:00:15.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:15 smithi142 conmon[20125]: ) 1349 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:00:15.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:15 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:00:15.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:15 smithi142 conmon[20125]: 22:00:15.237887+0000 mon.a (mon.0 2021-03-24T22:00:15.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:15 smithi142 conmon[20125]: ) 1350 : cluster [DBG] osdmap e445: 8 total, 8 up, 5 in 2021-03-24T22:00:15.410 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:15.463 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:15.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:15.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:00:15.476 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T22:00:15.569 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:15.621 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:15.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:15.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:00:15.730 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:15.777 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:15.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:15.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:00:15.872 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T22:00:15.883 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:15.894 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T22:00:15.917 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T22:00:15.935 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:15.938 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T22:00:15.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:15.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:15.958 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T22:00:15.980 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T22:00:16.004 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T22:00:16.028 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T22:00:16.030 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-24T22:00:16.030 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-24T22:00:16.031 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-24T22:00:16.031 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-24T22:00:16.031 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-24T22:00:16.031 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-24T22:00:16.032 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-24T22:00:16.032 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-24T22:00:16.043 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:16.090 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:16.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:16.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:00:16.196 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:16.245 INFO:tasks.rados.rados.0.smithi125.stdout:1665: done (6 left) 2021-03-24T22:00:16.246 INFO:tasks.rados.rados.0.smithi125.stdout:1666: read oid 37 snap 166 2021-03-24T22:00:16.247 INFO:tasks.rados.rados.0.smithi125.stdout:1666: expect (ObjNum 537 snap 166 seq_num 537) 2021-03-24T22:00:16.253 INFO:tasks.rados.rados.0.smithi125.stdout:1667: rollback oid 16 current snap is 177 2021-03-24T22:00:16.254 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 16 to 149 2021-03-24T22:00:16.254 INFO:tasks.rados.rados.0.smithi125.stdout:1668: write oid 13 current snap is 177 2021-03-24T22:00:16.254 INFO:tasks.rados.rados.0.smithi125.stdout:1668: seq_num 593 ranges {766551=498191,1684175=720857,2851717=693833} 2021-03-24T22:00:16.255 INFO:tasks.rados.rados.0.smithi125.stdout:1668: writing smithi12556470-13 from 766551 to 1264742 tid 1 2021-03-24T22:00:16.259 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:16.262 INFO:tasks.rados.rados.0.smithi125.stdout:1668: writing smithi12556470-13 from 1684175 to 2405032 tid 2 2021-03-24T22:00:16.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:16.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:00:16.269 INFO:tasks.rados.rados.0.smithi125.stdout:1668: writing smithi12556470-13 from 2851717 to 3545550 tid 3 2021-03-24T22:00:16.270 INFO:tasks.rados.rados.0.smithi125.stdout:1669: snap_remove snap 155 2021-03-24T22:00:16.366 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:16.418 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:16.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:16.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:16.482 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:16 smithi142 conmon[20125]: cluster 2021-03-24T22:00: 2021-03-24T22:00:16.526 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:16.538 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[24927]: cluster 2021-03-24T22:00:14. 2021-03-24T22:00:16.539 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[24927]: 616760+0000 osd.0 (osd.0) 97 : cluster [DBG] 5.2 scrub ok 2021-03-24T22:00:16.539 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T22:00:16.540 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[24927]: -24T22:00:14.786796+0000 osd.3 (osd.3) 96 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:00:16.540 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:00:16.540 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[24927]: 03-24T22:00:14.787429+0000 osd.3 ( 2021-03-24T22:00:16.540 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[24927]: osd.3) 97 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:00:16.541 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[24927]: cluster 2021-03-24T22:00:16.541 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[24927]: 2021-03-24T22:00:14.853980+0000 mgr.y (mgr.14144) 2021-03-24T22:00:16.541 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[24927]: 493 : cluster [DBG] pgmap v829: 176 pgs: 11 active+clean+wait, 165 active+clean; 460 MiB data, 1.1 GiB used, 446 GiB / 447 GiB avail; 9.2 MiB/s rd, 3.4 MiB/s wr, 24 op/s; 6.5 MiB/s, 5 objects/s recovering 2021-03-24T22:00:16.542 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[24927]: cluster 2021-03-24T22:00:16.542 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[24927]: 2021-03-24T22:00:14.879299+0000 osd.4 (osd.4) 104 2021-03-24T22:00:16.542 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[24927]: : cluster [DBG] 5.e scrub ok 2021-03-24T22:00:16.543 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[24927]: audit 2021 2021-03-24T22:00:16.543 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[24927]: -03-24T22:00:15.845299+0000 mon.c (mon.2) 70 2021-03-24T22:00:16.543 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[24927]: : audit [DBG] from='client.? v1:172.21.15.125:0/2430439855' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:00:16.544 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[24927]: cluster 2021-03-24T22:00:16.240108+0000 mon.a 2021-03-24T22:00:16.544 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[24927]: (mon.0) 1351 : cluster [DBG] osdmap e446: 8 total, 8 up, 5 in 2021-03-24T22:00:16.544 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[19936]: cluster 2021-03-24T22:00:16.545 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[19936]: 2021-03-24T22:00:16.545 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[19936]: 2021 2021-03-24T22:00:16.545 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[19936]: - 2021-03-24T22:00:16.546 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[19936]: 03 2021-03-24T22:00:16.546 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[19936]: - 2021-03-24T22:00:16.546 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[19936]: 24 2021-03-24T22:00:16.546 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[19936]: T 2021-03-24T22:00:16.547 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[19936]: 22 2021-03-24T22:00:16.547 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[19936]: : 2021-03-24T22:00:16.547 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[19936]: 00 2021-03-24T22:00:16.548 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[19936]: : 2021-03-24T22:00:16.548 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[19936]: 14 2021-03-24T22:00:16.548 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[19936]: . 2021-03-24T22:00:16.548 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[19936]: 616760 2021-03-24T22:00:16.549 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[19936]: +0000 2021-03-24T22:00:16.549 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[19936]: osd.0 (osd.0) 97 : 2021-03-24T22:00:16.549 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[19936]: cluster [DBG] 5.2 scrub ok 2021-03-24T22:00:16.550 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[19936]: cluster 2021-03-24T22:00 2021-03-24T22:00:16.550 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[19936]: :14.786796+0000 osd.3 (osd.3) 96 2021-03-24T22:00:16.550 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[19936]: : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:00:16.550 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:00:16.551 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[19936]: -03-24T22:00:14.787429+0000 osd.3 ( 2021-03-24T22:00:16.551 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[19936]: osd.3) 97 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:00:16.551 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:00:16.551 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[19936]: -03-24T22:00:14.853980+0000 mgr.y 2021-03-24T22:00:16.552 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[19936]: (mgr.14144) 493 : cluster [DBG] pgmap v829: 176 pgs: 11 active+clean+wait, 165 active+clean; 460 MiB data, 1.1 GiB used, 446 GiB / 447 GiB avail; 9.2 MiB/s rd, 3.4 MiB/s wr, 24 op/s; 6.5 MiB/s, 5 objects/s recovering 2021-03-24T22:00:16.552 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:00:16.552 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[19936]: 03-24T22:00:14.879299+0000 osd.4 2021-03-24T22:00:16.553 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[19936]: (osd.4) 104 : cluster [DBG] 5.e scrub ok 2021-03-24T22:00:16.553 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[19936]: audit 2021 2021-03-24T22:00:16.553 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[19936]: -03-24T22:00:15.845299+0000 mon.c 2021-03-24T22:00:16.553 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[19936]: (mon.2) 70 : audit [DBG] 2021-03-24T22:00:16.554 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[19936]: from='client.? v1:172.21.15.125:0/2430439855' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:00:16.554 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:00:16.554 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[19936]: T22:00:16.240108+0000 mon.a (mon. 2021-03-24T22:00:16.554 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:16 smithi125 conmon[19936]: 0) 1351 : cluster [DBG] osdmap e446: 8 total, 8 up, 5 in 2021-03-24T22:00:16.586 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:16.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:16.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:16.693 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:16.749 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:16.754 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:16 smithi142 conmon[20125]: 14.616760+0000 osd.0 (osd.0) 97 : cluster [DBG] 5.2 scrub ok 2021-03-24T22:00:16.754 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:16 smithi142 conmon[20125]: cluster 2021-03-24T22:00 2021-03-24T22:00:16.754 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:16 smithi142 conmon[20125]: :14.786796+0000 osd.3 (osd.3) 96 : cluster 2021-03-24T22:00:16.754 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:16 smithi142 conmon[20125]: [DBG] purged_snaps scrub starts 2021-03-24T22:00:16.755 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:16 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:00:16.755 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:16 smithi142 conmon[20125]: -03-24T22:00:14 2021-03-24T22:00:16.755 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:16 smithi142 conmon[20125]: .787429+0000 osd.3 (osd.3) 2021-03-24T22:00:16.755 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:16 smithi142 conmon[20125]: 97 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:00:16.756 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:16 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:00:16.756 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:16 smithi142 conmon[20125]: 03-24T22:00:14. 2021-03-24T22:00:16.756 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:16 smithi142 conmon[20125]: 853980+0000 mgr.y (mgr.14144) 493 : 2021-03-24T22:00:16.756 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:16 smithi142 conmon[20125]: cluster [DBG] pgmap v829: 176 pgs: 11 active+clean+wait, 165 active+clean; 460 MiB data, 1.1 GiB used, 446 GiB / 447 GiB avail; 9.2 MiB/s rd, 3.4 MiB/s wr, 24 op/s; 6.5 MiB/s, 5 objects/s recovering 2021-03-24T22:00:16.757 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:16 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:00:16.757 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:16 smithi142 conmon[20125]: T22:00:14.879299+0000 2021-03-24T22:00:16.757 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:16 smithi142 conmon[20125]: osd.4 (osd.4) 104 : cluster 2021-03-24T22:00:16.757 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:16 smithi142 conmon[20125]: [DBG] 5.e scrub ok 2021-03-24T22:00:16.757 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:16 smithi142 conmon[20125]: audit 2021-03-24T22: 2021-03-24T22:00:16.758 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:16 smithi142 conmon[20125]: 00:15.845299+0000 mon.c (mon.2) 2021-03-24T22:00:16.758 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:16 smithi142 conmon[20125]: 70 : audit [DBG] from='client.? v1:172.21.15.125:0/2430439855' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:00:16.758 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:16 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:00:16.758 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:16 smithi142 conmon[20125]: 24T22:00:16.240108 2021-03-24T22:00:16.759 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:16 smithi142 conmon[20125]: +0000 mon.a (mon.0) 2021-03-24T22:00:16.759 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:16 smithi142 conmon[20125]: 1351 : cluster [DBG] osdmap e446: 8 total, 8 up, 5 in 2021-03-24T22:00:16.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:16.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:00:16.862 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:16.891 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5] out_osds: [6, 1, 7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2021-03-24T22:00:16.892 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-24T22:00:16.892 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd dump -f json-pretty 2021-03-24T22:00:16.921 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:16.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:16.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:00:17.026 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:17.087 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:17.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:17.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:17.192 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:17.248 INFO:tasks.rados.rados.0.smithi125.stdout:1669: done (9 left) 2021-03-24T22:00:17.249 INFO:tasks.rados.rados.0.smithi125.stdout:1670: snap_create 2021-03-24T22:00:17.249 INFO:tasks.rados.rados.0.smithi125.stdout:1658: finishing write tid 3 to smithi12556470-19 2021-03-24T22:00:17.249 INFO:tasks.rados.rados.0.smithi125.stdout:1658: finishing write tid 4 to smithi12556470-19 2021-03-24T22:00:17.249 INFO:tasks.rados.rados.0.smithi125.stdout:1658: finishing write tid 5 to smithi12556470-19 2021-03-24T22:00:17.250 INFO:tasks.rados.rados.0.smithi125.stdout:1658: finishing write tid 6 to smithi12556470-19 2021-03-24T22:00:17.250 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 19 v 351 (ObjNum 591 snap 177 seq_num 591) dirty exists 2021-03-24T22:00:17.250 INFO:tasks.rados.rados.0.smithi125.stdout:1658: left oid 19 (ObjNum 591 snap 177 seq_num 591) 2021-03-24T22:00:17.251 INFO:tasks.rados.rados.0.smithi125.stdout:1659: finishing write tid 1 to smithi12556470-7 2021-03-24T22:00:17.251 INFO:tasks.rados.rados.0.smithi125.stdout:1659: finishing write tid 2 to smithi12556470-7 2021-03-24T22:00:17.251 INFO:tasks.rados.rados.0.smithi125.stdout:1659: finishing write tid 3 to smithi12556470-7 2021-03-24T22:00:17.252 INFO:tasks.rados.rados.0.smithi125.stdout:1659: finishing write tid 4 to smithi12556470-7 2021-03-24T22:00:17.252 INFO:tasks.rados.rados.0.smithi125.stdout:1659: finishing write tid 5 to smithi12556470-7 2021-03-24T22:00:17.252 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 7 v 546 (ObjNum 592 snap 177 seq_num 592) dirty exists 2021-03-24T22:00:17.253 INFO:tasks.rados.rados.0.smithi125.stdout:1659: left oid 7 (ObjNum 592 snap 177 seq_num 592) 2021-03-24T22:00:17.253 INFO:tasks.rados.rados.0.smithi125.stdout:1660: finishing rollback tid 0 to smithi12556470-2 2021-03-24T22:00:17.253 INFO:tasks.rados.rados.0.smithi125.stdout:1661: expect (ObjNum 473 snap 134 seq_num 473) 2021-03-24T22:00:17.258 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:17.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:17.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:00:17.324 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-24T22:00:17.363 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:17.371 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:00:17.414 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:17.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:17.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:00:17.471 INFO:tasks.rados.rados.0.smithi125.stdout:1660: finishing rollback tid 1 to smithi12556470-2 2021-03-24T22:00:17.472 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 2 v 548 (ObjNum 1 snap 21916 seq_num 1117546192) dirty dne 2021-03-24T22:00:17.472 INFO:tasks.rados.rados.0.smithi125.stdout:1662: expect (ObjNum 573 snap 175 seq_num 573) 2021-03-24T22:00:17.520 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:17.534 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:17 smithi142 conmon[20125]: audit 2021-03-24T22:00:16.857418+0000 2021-03-24T22:00:17.534 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:17 smithi142 conmon[20125]: mon.a (mon.0) 1352 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "28"}]: dispatch 2021-03-24T22:00:17.534 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:17 smithi142 conmon[20125]: audit 2021-03-24T22:00:17.242550+0000 mon.a 2021-03-24T22:00:17.535 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:17 smithi142 conmon[20125]: (mon.0) 1353 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "28"}]': finished 2021-03-24T22:00:17.535 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:17 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:00:17.535 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:17 smithi142 conmon[20125]: -03-24T22:00:17. 2021-03-24T22:00:17.535 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:17 smithi142 conmon[20125]: 242693+0000 mon.a (mon.0 2021-03-24T22:00:17.535 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:17 smithi142 conmon[20125]: ) 1354 : cluster [DBG] osdmap e447: 8 total, 8 up, 5 in 2021-03-24T22:00:17.548 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2021-03-24T22:00:17.549 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2021-03-24T22:00:17.549 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2021-03-24T22:00:17.550 INFO:teuthology.orchestra.run.smithi142.stdout:} 2021-03-24T22:00:17.572 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:17.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:17.615 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:17 smithi125 conmon[19936]: audit 2021-03-24T22:00:16.857418+0000 2021-03-24T22:00:17.615 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:17 smithi125 conmon[19936]: mon.a (mon.0) 1352 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "28"}]: dispatch 2021-03-24T22:00:17.615 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:17 smithi125 conmon[19936]: audit 2021-03-24T22:00:17.242550+0000 mon.a (mon.0) 1353 : audit [INF] 2021-03-24T22:00:17.616 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:17 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "28"}]': finished 2021-03-24T22:00:17.616 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:17 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:00:17.616 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:17 smithi125 conmon[19936]: 00:17.242693+0000 mon.a (mon.0) 1354 : cluster [DBG] osdmap e447: 8 total, 8 up, 5 in 2021-03-24T22:00:17.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:17 smithi125 conmon[24927]: audit 2021-03-24T22:00:16.857418+0000 mon.a (mon.0) 1352 : audit 2021-03-24T22:00:17.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:17 smithi125 conmon[24927]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "28"}]: dispatch 2021-03-24T22:00:17.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:17 smithi125 conmon[24927]: audit 2021-03-24T22:00:17.242550+0000 mon.a 2021-03-24T22:00:17.618 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:17 smithi125 conmon[24927]: (mon.0) 1353 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "28"}]': finished 2021-03-24T22:00:17.618 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:17 smithi125 conmon[24927]: cluster 2021-03-24T22:00:17.618 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:17 smithi125 conmon[24927]: 2021-03-24T22:00:17.242693+0000 mon.a (mon.0) 1354 : 2021-03-24T22:00:17.618 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:17 smithi125 conmon[24927]: cluster [DBG] osdmap e447: 8 total, 8 up, 5 in 2021-03-24T22:00:17.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:00:17.692 INFO:tasks.rados.rados.0.smithi125.stdout:1664: expect (ObjNum 520 snap 154 seq_num 520) 2021-03-24T22:00:17.721 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:17.770 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:17.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:17.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:00:17.794 INFO:tasks.rados.rados.0.smithi125.stdout:1666: expect (ObjNum 537 snap 166 seq_num 537) 2021-03-24T22:00:17.864 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:00:17.875 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:17.921 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:17.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:17.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:17.944 INFO:tasks.rados.rados.0.smithi125.stdout:1667: finishing rollback tid 0 to smithi12556470-16 2021-03-24T22:00:17.945 INFO:tasks.rados.rados.0.smithi125.stdout:1667: finishing rollback tid 1 to smithi12556470-16 2021-03-24T22:00:17.945 INFO:tasks.rados.rados.0.smithi125.stdout:1667: finishing rollback tid 2 to smithi12556470-16 2021-03-24T22:00:17.945 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 16 v 389 (ObjNum 444 snap 120 seq_num 444) dirty exists 2021-03-24T22:00:17.946 INFO:tasks.rados.rados.0.smithi125.stdout:1668: finishing write tid 1 to smithi12556470-13 2021-03-24T22:00:17.946 INFO:tasks.rados.rados.0.smithi125.stdout:1668: finishing write tid 2 to smithi12556470-13 2021-03-24T22:00:17.946 INFO:tasks.rados.rados.0.smithi125.stdout:1668: finishing write tid 3 to smithi12556470-13 2021-03-24T22:00:17.947 INFO:tasks.rados.rados.0.smithi125.stdout:1668: finishing write tid 4 to smithi12556470-13 2021-03-24T22:00:17.947 INFO:tasks.rados.rados.0.smithi125.stdout:1668: finishing write tid 5 to smithi12556470-13 2021-03-24T22:00:17.947 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 13 v 301 (ObjNum 593 snap 177 seq_num 593) dirty exists 2021-03-24T22:00:17.948 INFO:tasks.rados.rados.0.smithi125.stdout:1668: left oid 13 (ObjNum 593 snap 177 seq_num 593) 2021-03-24T22:00:18.026 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:18.088 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:18.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:18.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:00:18.250 INFO:tasks.rados.rados.0.smithi125.stdout:1658: done (9 left) 2021-03-24T22:00:18.264 INFO:tasks.rados.rados.0.smithi125.stdout:1659: done (8 left) 2021-03-24T22:00:18.264 INFO:tasks.rados.rados.0.smithi125.stdout:1660: done (7 left) 2021-03-24T22:00:18.264 INFO:tasks.rados.rados.0.smithi125.stdout:1661: done (6 left) 2021-03-24T22:00:18.265 INFO:tasks.rados.rados.0.smithi125.stdout:1662: done (5 left) 2021-03-24T22:00:18.265 INFO:tasks.rados.rados.0.smithi125.stdout:1664: done (4 left) 2021-03-24T22:00:18.265 INFO:tasks.rados.rados.0.smithi125.stdout:1666: done (3 left) 2021-03-24T22:00:18.265 INFO:tasks.rados.rados.0.smithi125.stdout:1667: done (2 left) 2021-03-24T22:00:18.265 INFO:tasks.rados.rados.0.smithi125.stdout:1668: done (1 left) 2021-03-24T22:00:18.266 INFO:tasks.rados.rados.0.smithi125.stdout:1670: done (0 left) 2021-03-24T22:00:18.266 INFO:tasks.rados.rados.0.smithi125.stdout:1671: rollback oid 38 current snap is 178 2021-03-24T22:00:18.266 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 38 to 170 2021-03-24T22:00:18.266 INFO:tasks.rados.rados.0.smithi125.stdout:1672: snap_create 2021-03-24T22:00:18.267 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:18.268 INFO:tasks.rados.rados.0.smithi125.stdout:1671: finishing rollback tid 0 to smithi12556470-38 2021-03-24T22:00:18.269 INFO:tasks.rados.rados.0.smithi125.stdout:1671: finishing rollback tid 1 to smithi12556470-38 2021-03-24T22:00:18.269 INFO:tasks.rados.rados.0.smithi125.stdout:1671: finishing rollback tid 2 to smithi12556470-38 2021-03-24T22:00:18.269 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 38 v 64 (ObjNum 489 snap 140 seq_num 489) dirty exists 2021-03-24T22:00:18.317 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:18.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:18.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:18.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:18 smithi142 conmon[20125]: cluster 2021-03-24T22:00:16.563321+0000 osd.0 (osd.0) 98 : cluster [DBG] 5.3 scrub ok 2021-03-24T22:00:18.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:18 smithi142 conmon[20125]: cluster 2021-03-24T22:00:16.855080+0000 mgr.y (mgr.14144) 494 : cluster [DBG] pgmap v832: 175 pgs: 1 active+clean+wait, 174 active+clean; 449 MiB data, 1.1 GiB used, 446 GiB / 447 GiB avail; 5.5 MiB/s rd, 2.6 MiB/s wr, 16 op/s; 33 B/s, 0 objects/s recovering 2021-03-24T22:00:18.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:18 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:00:18.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:18 smithi142 conmon[20125]: -03-24T22:00:17.991491+0000 osd.2 (osd.2) 76 : cluster [DBG] 7.15 scrub ok 2021-03-24T22:00:18.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:18 smithi142 conmon[20125]: cluster 2021-03-24T22:00:18.245413 2021-03-24T22:00:18.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:18 smithi142 conmon[20125]: +0000 mon.a (mon.0) 1355 : cluster [DBG] osdmap e448: 8 total, 8 up, 5 in 2021-03-24T22:00:18.423 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:18.481 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.7 dump_blocked_ops 2021-03-24T22:00:18.489 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:18.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:18.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:00:18.597 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:18.651 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:18.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:18.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:00:18.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:18 smithi125 conmon[19936]: cluster 2021-03-24T22:00:16.563321 2021-03-24T22:00:18.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:18 smithi125 conmon[19936]: +0000 osd.0 (osd.0) 98 : cluster [DBG] 5.3 scrub ok 2021-03-24T22:00:18.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:18 smithi125 conmon[19936]: cluster 2021-03-24T22:00:16.855080+0000 mgr.y (mgr.14144) 494 : cluster [DBG] 2021-03-24T22:00:18.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:18 smithi125 conmon[19936]: pgmap v832: 175 pgs: 1 active+clean+wait, 174 active+clean; 449 MiB data, 1.1 GiB used, 446 GiB / 447 GiB avail; 5.5 MiB/s rd, 2.6 MiB/s wr, 16 op/s; 33 B/s, 0 objects/s recovering 2021-03-24T22:00:18.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:18 smithi125 conmon[19936]: cluster 2021-03-24T22 2021-03-24T22:00:18.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:18 smithi125 conmon[19936]: :00:17.991491+0000 osd.2 (osd.2) 76 : cluster 2021-03-24T22:00:18.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:18 smithi125 conmon[19936]: [DBG] 7.15 scrub ok 2021-03-24T22:00:18.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:18 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:00:18.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:18 smithi125 conmon[19936]: T22:00:18.245413+0000 mon.a (mon 2021-03-24T22:00:18.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:18 smithi125 conmon[19936]: .0) 1355 : cluster 2021-03-24T22:00:18.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:18 smithi125 conmon[19936]: [DBG] osdmap e448: 8 total, 8 up, 5 in 2021-03-24T22:00:18.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:18 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T22:00:18.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:18 smithi125 conmon[24927]: 22:00:16.563321+0000 osd.0 (osd.0) 98 : cluster 2021-03-24T22:00:18.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:18 smithi125 conmon[24927]: [DBG] 5.3 scrub ok 2021-03-24T22:00:18.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:18 smithi125 conmon[24927]: cluster 2021-03-24T22:00:18.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:18 smithi125 conmon[24927]: 2021-03-24T22 2021-03-24T22:00:18.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:18 smithi125 conmon[24927]: :00:16.855080 2021-03-24T22:00:18.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:18 smithi125 conmon[24927]: +0000 mgr.y (mgr. 2021-03-24T22:00:18.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:18 smithi125 conmon[24927]: 14144) 494 : cluster 2021-03-24T22:00:18.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:18 smithi125 conmon[24927]: [DBG] pgmap v832: 175 pgs: 1 active+clean+wait, 174 active+clean; 449 MiB data, 1.1 GiB used, 446 GiB / 447 GiB avail; 5.5 MiB/s rd, 2.6 MiB/s wr, 16 op/s; 33 B/s, 0 objects/s recovering 2021-03-24T22:00:18.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:18 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:00:18.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:18 smithi125 conmon[24927]: -03-24T22:00:17.991491 2021-03-24T22:00:18.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:18 smithi125 conmon[24927]: +0000 osd.2 (osd.2 2021-03-24T22:00:18.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:18 smithi125 conmon[24927]: ) 76 : cluster [DBG] 2021-03-24T22:00:18.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:18 smithi125 conmon[24927]: 7.15 scrub ok 2021-03-24T22:00:18.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:18 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:00:18.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:18 smithi125 conmon[24927]: -03-24T22:00: 2021-03-24T22:00:18.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:18 smithi125 conmon[24927]: 18.245413+0000 mon.a 2021-03-24T22:00:18.672 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:18 smithi125 conmon[24927]: (mon.0) 1355 2021-03-24T22:00:18.672 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:18 smithi125 conmon[24927]: : cluster [DBG] osdmap e448: 8 total, 8 up, 5 in 2021-03-24T22:00:18.672 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:18 smithi125 conmon[24927]: 2021-03-24T22:00:18.761 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:18.777 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T22:00:18.822 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:18.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:18.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:00:18.928 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:19.070 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:19.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:19.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:00:19.176 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:19.251 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:19.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:19.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:00:19.259 INFO:tasks.rados.rados.0.smithi125.stdout:1671: done (1 left) 2021-03-24T22:00:19.259 INFO:tasks.rados.rados.0.smithi125.stdout:1672: done (0 left) 2021-03-24T22:00:19.259 INFO:tasks.rados.rados.0.smithi125.stdout:1673: rollback oid 12 current snap is 179 2021-03-24T22:00:19.260 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 12 to 150 2021-03-24T22:00:19.260 INFO:tasks.rados.rados.0.smithi125.stdout:1674: read oid 4 snap -1 2021-03-24T22:00:19.260 INFO:tasks.rados.rados.0.smithi125.stdout:1674: expect (ObjNum 544 snap 168 seq_num 544) 2021-03-24T22:00:19.260 INFO:tasks.rados.rados.0.smithi125.stdout:1675: rollback oid 46 current snap is 179 2021-03-24T22:00:19.260 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 46 to 175 2021-03-24T22:00:19.261 INFO:tasks.rados.rados.0.smithi125.stdout:1676: copy_from oid 7 from oid 18 current snap is 179 2021-03-24T22:00:19.261 INFO:tasks.rados.rados.0.smithi125.stdout:1677: write oid 14 current snap is 179 2021-03-24T22:00:19.261 INFO:tasks.rados.rados.0.smithi125.stdout:1677: seq_num 594 ranges {783358=573440,2026304=580868,3279140=621458} 2021-03-24T22:00:19.263 INFO:tasks.rados.rados.0.smithi125.stdout:1677: writing smithi12556470-14 from 783358 to 1356798 tid 1 2021-03-24T22:00:19.271 INFO:tasks.rados.rados.0.smithi125.stdout:1677: writing smithi12556470-14 from 2026304 to 2607172 tid 2 2021-03-24T22:00:19.279 INFO:tasks.rados.rados.0.smithi125.stdout:1677: writing smithi12556470-14 from 3279140 to 3900598 tid 3 2021-03-24T22:00:19.279 INFO:tasks.rados.rados.0.smithi125.stdout:1678: write oid 1 current snap is 179 2021-03-24T22:00:19.279 INFO:tasks.rados.rados.0.smithi125.stdout:1673: finishing rollback tid 0 to smithi12556470-12 2021-03-24T22:00:19.279 INFO:tasks.rados.rados.0.smithi125.stdout:1678: seq_num 595 ranges {400165=648689,1684786=315379} 2021-03-24T22:00:19.285 INFO:tasks.rados.rados.0.smithi125.stdout:1678: writing smithi12556470-1 from 400165 to 1048854 tid 1 2021-03-24T22:00:19.291 INFO:tasks.rados.rados.0.smithi125.stdout:1678: writing smithi12556470-1 from 1684786 to 2000165 tid 2 2021-03-24T22:00:19.291 INFO:tasks.rados.rados.0.smithi125.stdout:1679: snap_remove snap 157 2021-03-24T22:00:19.318 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-24T22:00:19.363 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:19.419 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:19.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:19.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:00:19.524 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:19.592 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:19.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:19.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:00:19.698 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:19.764 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:19.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:19.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:19.877 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:19.935 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:19.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:19.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:00:20.046 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:20.101 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:20.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:20.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:00:20.208 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:20.241 INFO:tasks.rados.rados.0.smithi125.stdout:1679: done (6 left) 2021-03-24T22:00:20.242 INFO:tasks.rados.rados.0.smithi125.stdout:1680: delete oid 19 current snap is 179 2021-03-24T22:00:20.243 INFO:tasks.rados.rados.0.smithi125.stdout:1673: finishing rollback tid 1 to smithi12556470-12 2021-03-24T22:00:20.243 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 12 v 394 (ObjNum 1116857136 snap 21916 seq_num 1116520064) dirty dne 2021-03-24T22:00:20.243 INFO:tasks.rados.rados.0.smithi125.stdout:1675: finishing rollback tid 1 to smithi12556470-46 2021-03-24T22:00:20.245 INFO:tasks.rados.rados.0.smithi125.stdout:1675: finishing rollback tid 2 to smithi12556470-46 2021-03-24T22:00:20.245 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 46 v 479 (ObjNum 497 snap 145 seq_num 497) dirty exists 2021-03-24T22:00:20.246 INFO:tasks.rados.rados.0.smithi125.stdout:1677: finishing write tid 1 to smithi12556470-14 2021-03-24T22:00:20.247 INFO:tasks.rados.rados.0.smithi125.stdout:1674: expect (ObjNum 544 snap 168 seq_num 544) 2021-03-24T22:00:20.288 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:00:20.289 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:20.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:20.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:00:20.388 INFO:tasks.rados.rados.0.smithi125.stdout:1673: done (6 left) 2021-03-24T22:00:20.388 INFO:tasks.rados.rados.0.smithi125.stdout:1674: done (5 left) 2021-03-24T22:00:20.389 INFO:tasks.rados.rados.0.smithi125.stdout:1675: done (4 left) 2021-03-24T22:00:20.389 INFO:tasks.rados.rados.0.smithi125.stdout:1680: done (3 left) 2021-03-24T22:00:20.389 INFO:tasks.rados.rados.0.smithi125.stdout:1681: write oid 13 current snap is 179 2021-03-24T22:00:20.389 INFO:tasks.rados.rados.0.smithi125.stdout:1677: finishing write tid 2 to smithi12556470-14 2021-03-24T22:00:20.390 INFO:tasks.rados.rados.0.smithi125.stdout:1681: seq_num 597 ranges {433779=799187,1801750=232029} 2021-03-24T22:00:20.394 INFO:tasks.rados.rados.0.smithi125.stdout:1681: writing smithi12556470-13 from 433779 to 1232966 tid 1 2021-03-24T22:00:20.395 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:20.401 INFO:tasks.rados.rados.0.smithi125.stdout:1681: writing smithi12556470-13 from 1801750 to 2033779 tid 2 2021-03-24T22:00:20.401 INFO:tasks.rados.rados.0.smithi125.stdout:1682: delete oid 35 current snap is 179 2021-03-24T22:00:20.401 INFO:tasks.rados.rados.0.smithi125.stdout:1678: finishing write tid 1 to smithi12556470-1 2021-03-24T22:00:20.401 INFO:tasks.rados.rados.0.smithi125.stdout:1676: finishing copy_from to smithi12556470-7 2021-03-24T22:00:20.402 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 7 v 203 (ObjNum 588 snap 177 seq_num 588) dirty exists 2021-03-24T22:00:20.402 INFO:tasks.rados.rados.0.smithi125.stdout:1677: finishing write tid 3 to smithi12556470-14 2021-03-24T22:00:20.402 INFO:tasks.rados.rados.0.smithi125.stdout:1677: finishing write tid 4 to smithi12556470-14 2021-03-24T22:00:20.402 INFO:tasks.rados.rados.0.smithi125.stdout:1676: finishing copy_from racing read to smithi12556470-7 2021-03-24T22:00:20.402 INFO:tasks.rados.rados.0.smithi125.stdout:1677: finishing write tid 5 to smithi12556470-14 2021-03-24T22:00:20.403 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 14 v 710 (ObjNum 594 snap 179 seq_num 594) dirty exists 2021-03-24T22:00:20.403 INFO:tasks.rados.rados.0.smithi125.stdout:1677: left oid 14 (ObjNum 594 snap 179 seq_num 594) 2021-03-24T22:00:20.403 INFO:tasks.rados.rados.0.smithi125.stdout:1678: finishing write tid 2 to smithi12556470-1 2021-03-24T22:00:20.403 INFO:tasks.rados.rados.0.smithi125.stdout:1678: finishing write tid 3 to smithi12556470-1 2021-03-24T22:00:20.404 INFO:tasks.rados.rados.0.smithi125.stdout:1678: finishing write tid 4 to smithi12556470-1 2021-03-24T22:00:20.404 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 1 v 560 (ObjNum 595 snap 179 seq_num 595) dirty exists 2021-03-24T22:00:20.404 INFO:tasks.rados.rados.0.smithi125.stdout:1678: left oid 1 (ObjNum 595 snap 179 seq_num 595) 2021-03-24T22:00:20.405 INFO:tasks.rados.rados.0.smithi125.stdout:1681: finishing write tid 1 to smithi12556470-13 2021-03-24T22:00:20.405 INFO:tasks.rados.rados.0.smithi125.stdout:1681: finishing write tid 2 to smithi12556470-13 2021-03-24T22:00:20.405 INFO:tasks.rados.rados.0.smithi125.stdout:1676: done (4 left) 2021-03-24T22:00:20.406 INFO:tasks.rados.rados.0.smithi125.stdout:1677: done (3 left) 2021-03-24T22:00:20.406 INFO:tasks.rados.rados.0.smithi125.stdout:1678: done (2 left) 2021-03-24T22:00:20.406 INFO:tasks.rados.rados.0.smithi125.stdout:1682: done (1 left) 2021-03-24T22:00:20.406 INFO:tasks.rados.rados.0.smithi125.stdout:1683: snap_remove snap 142 2021-03-24T22:00:20.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:20 smithi125 conmon[19936]: cluster 2021-03-24T22:00:18.322171+0000 osd.3 2021-03-24T22:00:20.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:20 smithi125 conmon[19936]: (osd.3) 98 : cluster [DBG] 6.b scrub ok 2021-03-24T22:00:20.407 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:20 smithi125 conmon[19936]: cluster 2021-03-24T22:00:18.855770+0000 mgr.y (mgr.14144) 495 : cluster [DBG] pgmap v835: 175 pgs: 2 peering, 1 active+clean+wait, 172 active+clean; 443 MiB data, 1.1 GiB used, 446 GiB / 447 GiB avail; 255 B/s rd, 415 KiB/s wr, 1 op/s 2021-03-24T22:00:20.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:20 smithi125 conmon[19936]: cluster 2021-03-24T22:00:19.248831+0000 mon.a ( 2021-03-24T22:00:20.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:20 smithi125 conmon[19936]: mon.0) 1356 : cluster [DBG] osdmap e449: 8 total, 8 up, 5 in 2021-03-24T22:00:20.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:20 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:00:20.408 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:20 smithi125 conmon[19936]: T22:00:20.238274+0000 mon.a (mon.0 2021-03-24T22:00:20.409 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:20 smithi125 conmon[19936]: ) 1357 : cluster [DBG] osdmap e450: 8 total, 8 up, 5 in 2021-03-24T22:00:20.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:20 smithi125 conmon[24927]: cluster 2021-03-24T22:00:18.322171+0000 osd.3 (osd 2021-03-24T22:00:20.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:20 smithi125 conmon[24927]: .3) 98 : cluster [DBG] 6.b scrub ok 2021-03-24T22:00:20.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:20 smithi125 conmon[24927]: cluster 2021-03-24T22:00:18.855770+0000 mgr.y (mgr.14144) 495 : 2021-03-24T22:00:20.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:20 smithi125 conmon[24927]: cluster [DBG] pgmap v835: 175 pgs: 2 peering, 1 active+clean+wait, 172 active+clean; 443 MiB data, 1.1 GiB used, 446 GiB / 447 GiB avail; 255 B/s rd, 415 KiB/s wr, 1 op/s 2021-03-24T22:00:20.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:20 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:00:20.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:20 smithi125 conmon[24927]: 24T22:00:19.248831+0000 mon.a (mon.0) 1356 : 2021-03-24T22:00:20.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:20 smithi125 conmon[24927]: cluster [DBG] osdmap e449: 8 total, 8 up, 5 in 2021-03-24T22:00:20.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:20 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T22:00:20.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:20 smithi125 conmon[24927]: 22:00:20.238274+0000 mon.a (mon. 2021-03-24T22:00:20.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:20 smithi125 conmon[24927]: 0) 1357 : cluster [DBG] osdmap e450: 8 total, 8 up, 5 in 2021-03-24T22:00:20.438 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:20.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:20.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:20.482 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-24T22:00:20.483 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-24T22:00:20.483 INFO:teuthology.orchestra.run.smithi125.stdout: "epoch": 450, 2021-03-24T22:00:20.483 INFO:teuthology.orchestra.run.smithi125.stdout: "fsid": "3cfa339a-8cea-11eb-90b2-001a4aab830c", 2021-03-24T22:00:20.483 INFO:teuthology.orchestra.run.smithi125.stdout: "created": "2021-03-24T21:46:19.163312+0000", 2021-03-24T22:00:20.484 INFO:teuthology.orchestra.run.smithi125.stdout: "modified": "2021-03-24T22:00:20.230419+0000", 2021-03-24T22:00:20.484 INFO:teuthology.orchestra.run.smithi125.stdout: "last_up_change": "2021-03-24T22:00:02.225213+0000", 2021-03-24T22:00:20.484 INFO:teuthology.orchestra.run.smithi125.stdout: "last_in_change": "2021-03-24T21:59:46.230295+0000", 2021-03-24T22:00:20.484 INFO:teuthology.orchestra.run.smithi125.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-03-24T22:00:20.485 INFO:teuthology.orchestra.run.smithi125.stdout: "flags_num": 5804032, 2021-03-24T22:00:20.485 INFO:teuthology.orchestra.run.smithi125.stdout: "flags_set": [ 2021-03-24T22:00:20.485 INFO:teuthology.orchestra.run.smithi125.stdout: "nodeep-scrub", 2021-03-24T22:00:20.485 INFO:teuthology.orchestra.run.smithi125.stdout: "pglog_hardlimit", 2021-03-24T22:00:20.486 INFO:teuthology.orchestra.run.smithi125.stdout: "purged_snapdirs", 2021-03-24T22:00:20.486 INFO:teuthology.orchestra.run.smithi125.stdout: "recovery_deletes", 2021-03-24T22:00:20.486 INFO:teuthology.orchestra.run.smithi125.stdout: "sortbitwise" 2021-03-24T22:00:20.486 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2021-03-24T22:00:20.487 INFO:teuthology.orchestra.run.smithi125.stdout: "crush_version": 19, 2021-03-24T22:00:20.487 INFO:teuthology.orchestra.run.smithi125.stdout: "full_ratio": 0.94999998807907104, 2021-03-24T22:00:20.487 INFO:teuthology.orchestra.run.smithi125.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-03-24T22:00:20.487 INFO:teuthology.orchestra.run.smithi125.stdout: "nearfull_ratio": 0.85000002384185791, 2021-03-24T22:00:20.487 INFO:teuthology.orchestra.run.smithi125.stdout: "cluster_snapshot": "", 2021-03-24T22:00:20.488 INFO:teuthology.orchestra.run.smithi125.stdout: "pool_max": 7, 2021-03-24T22:00:20.488 INFO:teuthology.orchestra.run.smithi125.stdout: "max_osd": 8, 2021-03-24T22:00:20.488 INFO:teuthology.orchestra.run.smithi125.stdout: "require_min_compat_client": "luminous", 2021-03-24T22:00:20.488 INFO:teuthology.orchestra.run.smithi125.stdout: "min_compat_client": "luminous", 2021-03-24T22:00:20.489 INFO:teuthology.orchestra.run.smithi125.stdout: "require_osd_release": "pacific", 2021-03-24T22:00:20.489 INFO:teuthology.orchestra.run.smithi125.stdout: "pools": [ 2021-03-24T22:00:20.489 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.489 INFO:teuthology.orchestra.run.smithi125.stdout: "pool": 1, 2021-03-24T22:00:20.489 INFO:teuthology.orchestra.run.smithi125.stdout: "pool_name": "device_health_metrics", 2021-03-24T22:00:20.490 INFO:teuthology.orchestra.run.smithi125.stdout: "create_time": "2021-03-24T21:49:51.001357+0000", 2021-03-24T22:00:20.490 INFO:teuthology.orchestra.run.smithi125.stdout: "flags": 1, 2021-03-24T22:00:20.490 INFO:teuthology.orchestra.run.smithi125.stdout: "flags_names": "hashpspool", 2021-03-24T22:00:20.490 INFO:teuthology.orchestra.run.smithi125.stdout: "type": 1, 2021-03-24T22:00:20.491 INFO:teuthology.orchestra.run.smithi125.stdout: "size": 3, 2021-03-24T22:00:20.491 INFO:teuthology.orchestra.run.smithi125.stdout: "min_size": 2, 2021-03-24T22:00:20.491 INFO:teuthology.orchestra.run.smithi125.stdout: "crush_rule": 0, 2021-03-24T22:00:20.492 INFO:teuthology.orchestra.run.smithi125.stdout: "peering_crush_bucket_count": 0, 2021-03-24T22:00:20.492 INFO:teuthology.orchestra.run.smithi125.stdout: "peering_crush_bucket_target": 0, 2021-03-24T22:00:20.492 INFO:teuthology.orchestra.run.smithi125.stdout: "peering_crush_bucket_barrier": 0, 2021-03-24T22:00:20.492 INFO:teuthology.orchestra.run.smithi125.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-24T22:00:20.493 INFO:teuthology.orchestra.run.smithi125.stdout: "object_hash": 2, 2021-03-24T22:00:20.493 INFO:teuthology.orchestra.run.smithi125.stdout: "pg_autoscale_mode": "off", 2021-03-24T22:00:20.493 INFO:teuthology.orchestra.run.smithi125.stdout: "pg_num": 1, 2021-03-24T22:00:20.493 INFO:teuthology.orchestra.run.smithi125.stdout: "pg_placement_num": 1, 2021-03-24T22:00:20.493 INFO:teuthology.orchestra.run.smithi125.stdout: "pg_placement_num_target": 1, 2021-03-24T22:00:20.494 INFO:teuthology.orchestra.run.smithi125.stdout: "pg_num_target": 1, 2021-03-24T22:00:20.494 INFO:teuthology.orchestra.run.smithi125.stdout: "pg_num_pending": 1, 2021-03-24T22:00:20.494 INFO:teuthology.orchestra.run.smithi125.stdout: "last_pg_merge_meta": { 2021-03-24T22:00:20.495 INFO:teuthology.orchestra.run.smithi125.stdout: "source_pgid": "0.0", 2021-03-24T22:00:20.495 INFO:teuthology.orchestra.run.smithi125.stdout: "ready_epoch": 0, 2021-03-24T22:00:20.495 INFO:teuthology.orchestra.run.smithi125.stdout: "last_epoch_started": 0, 2021-03-24T22:00:20.495 INFO:teuthology.orchestra.run.smithi125.stdout: "last_epoch_clean": 0, 2021-03-24T22:00:20.496 INFO:teuthology.orchestra.run.smithi125.stdout: "source_version": "0'0", 2021-03-24T22:00:20.496 INFO:teuthology.orchestra.run.smithi125.stdout: "target_version": "0'0" 2021-03-24T22:00:20.496 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.496 INFO:teuthology.orchestra.run.smithi125.stdout: "last_change": "21", 2021-03-24T22:00:20.497 INFO:teuthology.orchestra.run.smithi125.stdout: "last_force_op_resend": "0", 2021-03-24T22:00:20.497 INFO:teuthology.orchestra.run.smithi125.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-24T22:00:20.497 INFO:teuthology.orchestra.run.smithi125.stdout: "last_force_op_resend_preluminous": "0", 2021-03-24T22:00:20.497 INFO:teuthology.orchestra.run.smithi125.stdout: "auid": 0, 2021-03-24T22:00:20.497 INFO:teuthology.orchestra.run.smithi125.stdout: "snap_mode": "selfmanaged", 2021-03-24T22:00:20.498 INFO:teuthology.orchestra.run.smithi125.stdout: "snap_seq": 0, 2021-03-24T22:00:20.498 INFO:teuthology.orchestra.run.smithi125.stdout: "snap_epoch": 0, 2021-03-24T22:00:20.498 INFO:teuthology.orchestra.run.smithi125.stdout: "pool_snaps": [], 2021-03-24T22:00:20.498 INFO:teuthology.orchestra.run.smithi125.stdout: "removed_snaps": "[]", 2021-03-24T22:00:20.498 INFO:teuthology.orchestra.run.smithi125.stdout: "quota_max_bytes": 0, 2021-03-24T22:00:20.499 INFO:teuthology.orchestra.run.smithi125.stdout: "quota_max_objects": 0, 2021-03-24T22:00:20.499 INFO:teuthology.orchestra.run.smithi125.stdout: "tiers": [], 2021-03-24T22:00:20.499 INFO:teuthology.orchestra.run.smithi125.stdout: "tier_of": -1, 2021-03-24T22:00:20.499 INFO:teuthology.orchestra.run.smithi125.stdout: "read_tier": -1, 2021-03-24T22:00:20.500 INFO:teuthology.orchestra.run.smithi125.stdout: "write_tier": -1, 2021-03-24T22:00:20.500 INFO:teuthology.orchestra.run.smithi125.stdout: "cache_mode": "none", 2021-03-24T22:00:20.500 INFO:teuthology.orchestra.run.smithi125.stdout: "target_max_bytes": 0, 2021-03-24T22:00:20.500 INFO:teuthology.orchestra.run.smithi125.stdout: "target_max_objects": 0, 2021-03-24T22:00:20.501 INFO:teuthology.orchestra.run.smithi125.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-24T22:00:20.501 INFO:teuthology.orchestra.run.smithi125.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-24T22:00:20.501 INFO:teuthology.orchestra.run.smithi125.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-24T22:00:20.501 INFO:teuthology.orchestra.run.smithi125.stdout: "cache_min_flush_age": 0, 2021-03-24T22:00:20.502 INFO:teuthology.orchestra.run.smithi125.stdout: "cache_min_evict_age": 0, 2021-03-24T22:00:20.502 INFO:teuthology.orchestra.run.smithi125.stdout: "erasure_code_profile": "", 2021-03-24T22:00:20.502 INFO:teuthology.orchestra.run.smithi125.stdout: "hit_set_params": { 2021-03-24T22:00:20.502 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "none" 2021-03-24T22:00:20.503 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.503 INFO:teuthology.orchestra.run.smithi125.stdout: "hit_set_period": 0, 2021-03-24T22:00:20.503 INFO:teuthology.orchestra.run.smithi125.stdout: "hit_set_count": 0, 2021-03-24T22:00:20.504 INFO:teuthology.orchestra.run.smithi125.stdout: "use_gmt_hitset": true, 2021-03-24T22:00:20.504 INFO:teuthology.orchestra.run.smithi125.stdout: "min_read_recency_for_promote": 0, 2021-03-24T22:00:20.504 INFO:teuthology.orchestra.run.smithi125.stdout: "min_write_recency_for_promote": 0, 2021-03-24T22:00:20.504 INFO:teuthology.orchestra.run.smithi125.stdout: "hit_set_grade_decay_rate": 0, 2021-03-24T22:00:20.505 INFO:teuthology.orchestra.run.smithi125.stdout: "hit_set_search_last_n": 0, 2021-03-24T22:00:20.505 INFO:teuthology.orchestra.run.smithi125.stdout: "grade_table": [], 2021-03-24T22:00:20.505 INFO:teuthology.orchestra.run.smithi125.stdout: "stripe_width": 0, 2021-03-24T22:00:20.506 INFO:teuthology.orchestra.run.smithi125.stdout: "expected_num_objects": 0, 2021-03-24T22:00:20.506 INFO:teuthology.orchestra.run.smithi125.stdout: "fast_read": false, 2021-03-24T22:00:20.506 INFO:teuthology.orchestra.run.smithi125.stdout: "options": { 2021-03-24T22:00:20.507 INFO:teuthology.orchestra.run.smithi125.stdout: "pg_num_min": 1 2021-03-24T22:00:20.507 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.507 INFO:teuthology.orchestra.run.smithi125.stdout: "application_metadata": { 2021-03-24T22:00:20.507 INFO:teuthology.orchestra.run.smithi125.stdout: "mgr_devicehealth": {} 2021-03-24T22:00:20.508 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.508 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.508 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.508 INFO:teuthology.orchestra.run.smithi125.stdout: "pool": 2, 2021-03-24T22:00:20.509 INFO:teuthology.orchestra.run.smithi125.stdout: "pool_name": "iscsi", 2021-03-24T22:00:20.509 INFO:teuthology.orchestra.run.smithi125.stdout: "create_time": "2021-03-24T21:51:39.190981+0000", 2021-03-24T22:00:20.509 INFO:teuthology.orchestra.run.smithi125.stdout: "flags": 1, 2021-03-24T22:00:20.509 INFO:teuthology.orchestra.run.smithi125.stdout: "flags_names": "hashpspool", 2021-03-24T22:00:20.509 INFO:teuthology.orchestra.run.smithi125.stdout: "type": 1, 2021-03-24T22:00:20.510 INFO:teuthology.orchestra.run.smithi125.stdout: "size": 3, 2021-03-24T22:00:20.510 INFO:teuthology.orchestra.run.smithi125.stdout: "min_size": 2, 2021-03-24T22:00:20.510 INFO:teuthology.orchestra.run.smithi125.stdout: "crush_rule": 0, 2021-03-24T22:00:20.510 INFO:teuthology.orchestra.run.smithi125.stdout: "peering_crush_bucket_count": 0, 2021-03-24T22:00:20.511 INFO:teuthology.orchestra.run.smithi125.stdout: "peering_crush_bucket_target": 0, 2021-03-24T22:00:20.511 INFO:teuthology.orchestra.run.smithi125.stdout: "peering_crush_bucket_barrier": 0, 2021-03-24T22:00:20.511 INFO:teuthology.orchestra.run.smithi125.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-24T22:00:20.512 INFO:teuthology.orchestra.run.smithi125.stdout: "object_hash": 2, 2021-03-24T22:00:20.512 INFO:teuthology.orchestra.run.smithi125.stdout: "pg_autoscale_mode": "off", 2021-03-24T22:00:20.512 INFO:teuthology.orchestra.run.smithi125.stdout: "pg_num": 3, 2021-03-24T22:00:20.512 INFO:teuthology.orchestra.run.smithi125.stdout: "pg_placement_num": 3, 2021-03-24T22:00:20.513 INFO:teuthology.orchestra.run.smithi125.stdout: "pg_placement_num_target": 3, 2021-03-24T22:00:20.513 INFO:teuthology.orchestra.run.smithi125.stdout: "pg_num_target": 3, 2021-03-24T22:00:20.513 INFO:teuthology.orchestra.run.smithi125.stdout: "pg_num_pending": 3, 2021-03-24T22:00:20.514 INFO:teuthology.orchestra.run.smithi125.stdout: "last_pg_merge_meta": { 2021-03-24T22:00:20.514 INFO:teuthology.orchestra.run.smithi125.stdout: "source_pgid": "0.0", 2021-03-24T22:00:20.514 INFO:teuthology.orchestra.run.smithi125.stdout: "ready_epoch": 0, 2021-03-24T22:00:20.514 INFO:teuthology.orchestra.run.smithi125.stdout: "last_epoch_started": 0, 2021-03-24T22:00:20.515 INFO:teuthology.orchestra.run.smithi125.stdout: "last_epoch_clean": 0, 2021-03-24T22:00:20.515 INFO:teuthology.orchestra.run.smithi125.stdout: "source_version": "0'0", 2021-03-24T22:00:20.515 INFO:teuthology.orchestra.run.smithi125.stdout: "target_version": "0'0" 2021-03-24T22:00:20.516 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.516 INFO:teuthology.orchestra.run.smithi125.stdout: "last_change": "178", 2021-03-24T22:00:20.516 INFO:teuthology.orchestra.run.smithi125.stdout: "last_force_op_resend": "0", 2021-03-24T22:00:20.517 INFO:teuthology.orchestra.run.smithi125.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-24T22:00:20.517 INFO:teuthology.orchestra.run.smithi125.stdout: "last_force_op_resend_preluminous": "0", 2021-03-24T22:00:20.518 INFO:teuthology.orchestra.run.smithi125.stdout: "auid": 0, 2021-03-24T22:00:20.518 INFO:teuthology.orchestra.run.smithi125.stdout: "snap_mode": "selfmanaged", 2021-03-24T22:00:20.519 INFO:teuthology.orchestra.run.smithi125.stdout: "snap_seq": 0, 2021-03-24T22:00:20.519 INFO:teuthology.orchestra.run.smithi125.stdout: "snap_epoch": 0, 2021-03-24T22:00:20.519 INFO:teuthology.orchestra.run.smithi125.stdout: "pool_snaps": [], 2021-03-24T22:00:20.520 INFO:teuthology.orchestra.run.smithi125.stdout: "removed_snaps": "[]", 2021-03-24T22:00:20.520 INFO:teuthology.orchestra.run.smithi125.stdout: "quota_max_bytes": 0, 2021-03-24T22:00:20.520 INFO:teuthology.orchestra.run.smithi125.stdout: "quota_max_objects": 0, 2021-03-24T22:00:20.520 INFO:teuthology.orchestra.run.smithi125.stdout: "tiers": [], 2021-03-24T22:00:20.521 INFO:teuthology.orchestra.run.smithi125.stdout: "tier_of": -1, 2021-03-24T22:00:20.521 INFO:teuthology.orchestra.run.smithi125.stdout: "read_tier": -1, 2021-03-24T22:00:20.521 INFO:teuthology.orchestra.run.smithi125.stdout: "write_tier": -1, 2021-03-24T22:00:20.521 INFO:teuthology.orchestra.run.smithi125.stdout: "cache_mode": "none", 2021-03-24T22:00:20.522 INFO:teuthology.orchestra.run.smithi125.stdout: "target_max_bytes": 0, 2021-03-24T22:00:20.522 INFO:teuthology.orchestra.run.smithi125.stdout: "target_max_objects": 0, 2021-03-24T22:00:20.522 INFO:teuthology.orchestra.run.smithi125.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-24T22:00:20.522 INFO:teuthology.orchestra.run.smithi125.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-24T22:00:20.523 INFO:teuthology.orchestra.run.smithi125.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-24T22:00:20.523 INFO:teuthology.orchestra.run.smithi125.stdout: "cache_min_flush_age": 0, 2021-03-24T22:00:20.523 INFO:teuthology.orchestra.run.smithi125.stdout: "cache_min_evict_age": 0, 2021-03-24T22:00:20.523 INFO:teuthology.orchestra.run.smithi125.stdout: "erasure_code_profile": "", 2021-03-24T22:00:20.524 INFO:teuthology.orchestra.run.smithi125.stdout: "hit_set_params": { 2021-03-24T22:00:20.524 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "none" 2021-03-24T22:00:20.524 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.524 INFO:teuthology.orchestra.run.smithi125.stdout: "hit_set_period": 0, 2021-03-24T22:00:20.525 INFO:teuthology.orchestra.run.smithi125.stdout: "hit_set_count": 0, 2021-03-24T22:00:20.525 INFO:teuthology.orchestra.run.smithi125.stdout: "use_gmt_hitset": true, 2021-03-24T22:00:20.525 INFO:teuthology.orchestra.run.smithi125.stdout: "min_read_recency_for_promote": 0, 2021-03-24T22:00:20.525 INFO:teuthology.orchestra.run.smithi125.stdout: "min_write_recency_for_promote": 0, 2021-03-24T22:00:20.526 INFO:teuthology.orchestra.run.smithi125.stdout: "hit_set_grade_decay_rate": 0, 2021-03-24T22:00:20.526 INFO:teuthology.orchestra.run.smithi125.stdout: "hit_set_search_last_n": 0, 2021-03-24T22:00:20.526 INFO:teuthology.orchestra.run.smithi125.stdout: "grade_table": [], 2021-03-24T22:00:20.526 INFO:teuthology.orchestra.run.smithi125.stdout: "stripe_width": 0, 2021-03-24T22:00:20.527 INFO:teuthology.orchestra.run.smithi125.stdout: "expected_num_objects": 0, 2021-03-24T22:00:20.527 INFO:teuthology.orchestra.run.smithi125.stdout: "fast_read": false, 2021-03-24T22:00:20.527 INFO:teuthology.orchestra.run.smithi125.stdout: "options": {}, 2021-03-24T22:00:20.527 INFO:teuthology.orchestra.run.smithi125.stdout: "application_metadata": { 2021-03-24T22:00:20.527 INFO:teuthology.orchestra.run.smithi125.stdout: "rbd": {} 2021-03-24T22:00:20.528 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.528 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.528 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.528 INFO:teuthology.orchestra.run.smithi125.stdout: "pool": 3, 2021-03-24T22:00:20.529 INFO:teuthology.orchestra.run.smithi125.stdout: "pool_name": ".rgw.root", 2021-03-24T22:00:20.529 INFO:teuthology.orchestra.run.smithi125.stdout: "create_time": "2021-03-24T21:51:44.159145+0000", 2021-03-24T22:00:20.529 INFO:teuthology.orchestra.run.smithi125.stdout: "flags": 1, 2021-03-24T22:00:20.529 INFO:teuthology.orchestra.run.smithi125.stdout: "flags_names": "hashpspool", 2021-03-24T22:00:20.530 INFO:teuthology.orchestra.run.smithi125.stdout: "type": 1, 2021-03-24T22:00:20.530 INFO:teuthology.orchestra.run.smithi125.stdout: "size": 3, 2021-03-24T22:00:20.530 INFO:teuthology.orchestra.run.smithi125.stdout: "min_size": 2, 2021-03-24T22:00:20.530 INFO:teuthology.orchestra.run.smithi125.stdout: "crush_rule": 0, 2021-03-24T22:00:20.531 INFO:teuthology.orchestra.run.smithi125.stdout: "peering_crush_bucket_count": 0, 2021-03-24T22:00:20.531 INFO:teuthology.orchestra.run.smithi125.stdout: "peering_crush_bucket_target": 0, 2021-03-24T22:00:20.531 INFO:teuthology.orchestra.run.smithi125.stdout: "peering_crush_bucket_barrier": 0, 2021-03-24T22:00:20.532 INFO:teuthology.orchestra.run.smithi125.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-24T22:00:20.532 INFO:teuthology.orchestra.run.smithi125.stdout: "object_hash": 2, 2021-03-24T22:00:20.532 INFO:teuthology.orchestra.run.smithi125.stdout: "pg_autoscale_mode": "off", 2021-03-24T22:00:20.532 INFO:teuthology.orchestra.run.smithi125.stdout: "pg_num": 42, 2021-03-24T22:00:20.532 INFO:teuthology.orchestra.run.smithi125.stdout: "pg_placement_num": 42, 2021-03-24T22:00:20.533 INFO:teuthology.orchestra.run.smithi125.stdout: "pg_placement_num_target": 42, 2021-03-24T22:00:20.533 INFO:teuthology.orchestra.run.smithi125.stdout: "pg_num_target": 42, 2021-03-24T22:00:20.533 INFO:teuthology.orchestra.run.smithi125.stdout: "pg_num_pending": 42, 2021-03-24T22:00:20.534 INFO:teuthology.orchestra.run.smithi125.stdout: "last_pg_merge_meta": { 2021-03-24T22:00:20.534 INFO:teuthology.orchestra.run.smithi125.stdout: "source_pgid": "0.0", 2021-03-24T22:00:20.535 INFO:teuthology.orchestra.run.smithi125.stdout: "ready_epoch": 0, 2021-03-24T22:00:20.535 INFO:teuthology.orchestra.run.smithi125.stdout: "last_epoch_started": 0, 2021-03-24T22:00:20.535 INFO:teuthology.orchestra.run.smithi125.stdout: "last_epoch_clean": 0, 2021-03-24T22:00:20.536 INFO:teuthology.orchestra.run.smithi125.stdout: "source_version": "0'0", 2021-03-24T22:00:20.536 INFO:teuthology.orchestra.run.smithi125.stdout: "target_version": "0'0" 2021-03-24T22:00:20.536 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.536 INFO:teuthology.orchestra.run.smithi125.stdout: "last_change": "383", 2021-03-24T22:00:20.537 INFO:teuthology.orchestra.run.smithi125.stdout: "last_force_op_resend": "0", 2021-03-24T22:00:20.537 INFO:teuthology.orchestra.run.smithi125.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-24T22:00:20.537 INFO:teuthology.orchestra.run.smithi125.stdout: "last_force_op_resend_preluminous": "381", 2021-03-24T22:00:20.537 INFO:teuthology.orchestra.run.smithi125.stdout: "auid": 0, 2021-03-24T22:00:20.538 INFO:teuthology.orchestra.run.smithi125.stdout: "snap_mode": "selfmanaged", 2021-03-24T22:00:20.538 INFO:teuthology.orchestra.run.smithi125.stdout: "snap_seq": 0, 2021-03-24T22:00:20.538 INFO:teuthology.orchestra.run.smithi125.stdout: "snap_epoch": 0, 2021-03-24T22:00:20.538 INFO:teuthology.orchestra.run.smithi125.stdout: "pool_snaps": [], 2021-03-24T22:00:20.539 INFO:teuthology.orchestra.run.smithi125.stdout: "removed_snaps": "[]", 2021-03-24T22:00:20.539 INFO:teuthology.orchestra.run.smithi125.stdout: "quota_max_bytes": 0, 2021-03-24T22:00:20.539 INFO:teuthology.orchestra.run.smithi125.stdout: "quota_max_objects": 0, 2021-03-24T22:00:20.539 INFO:teuthology.orchestra.run.smithi125.stdout: "tiers": [], 2021-03-24T22:00:20.539 INFO:teuthology.orchestra.run.smithi125.stdout: "tier_of": -1, 2021-03-24T22:00:20.540 INFO:teuthology.orchestra.run.smithi125.stdout: "read_tier": -1, 2021-03-24T22:00:20.540 INFO:teuthology.orchestra.run.smithi125.stdout: "write_tier": -1, 2021-03-24T22:00:20.540 INFO:teuthology.orchestra.run.smithi125.stdout: "cache_mode": "none", 2021-03-24T22:00:20.540 INFO:teuthology.orchestra.run.smithi125.stdout: "target_max_bytes": 0, 2021-03-24T22:00:20.540 INFO:teuthology.orchestra.run.smithi125.stdout: "target_max_objects": 0, 2021-03-24T22:00:20.541 INFO:teuthology.orchestra.run.smithi125.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-24T22:00:20.541 INFO:teuthology.orchestra.run.smithi125.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-24T22:00:20.541 INFO:teuthology.orchestra.run.smithi125.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-24T22:00:20.541 INFO:teuthology.orchestra.run.smithi125.stdout: "cache_min_flush_age": 0, 2021-03-24T22:00:20.542 INFO:teuthology.orchestra.run.smithi125.stdout: "cache_min_evict_age": 0, 2021-03-24T22:00:20.542 INFO:teuthology.orchestra.run.smithi125.stdout: "erasure_code_profile": "", 2021-03-24T22:00:20.542 INFO:teuthology.orchestra.run.smithi125.stdout: "hit_set_params": { 2021-03-24T22:00:20.542 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "none" 2021-03-24T22:00:20.542 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.543 INFO:teuthology.orchestra.run.smithi125.stdout: "hit_set_period": 0, 2021-03-24T22:00:20.543 INFO:teuthology.orchestra.run.smithi125.stdout: "hit_set_count": 0, 2021-03-24T22:00:20.543 INFO:teuthology.orchestra.run.smithi125.stdout: "use_gmt_hitset": true, 2021-03-24T22:00:20.543 INFO:teuthology.orchestra.run.smithi125.stdout: "min_read_recency_for_promote": 0, 2021-03-24T22:00:20.544 INFO:teuthology.orchestra.run.smithi125.stdout: "min_write_recency_for_promote": 0, 2021-03-24T22:00:20.544 INFO:teuthology.orchestra.run.smithi125.stdout: "hit_set_grade_decay_rate": 0, 2021-03-24T22:00:20.544 INFO:teuthology.orchestra.run.smithi125.stdout: "hit_set_search_last_n": 0, 2021-03-24T22:00:20.544 INFO:teuthology.orchestra.run.smithi125.stdout: "grade_table": [], 2021-03-24T22:00:20.545 INFO:teuthology.orchestra.run.smithi125.stdout: "stripe_width": 0, 2021-03-24T22:00:20.545 INFO:teuthology.orchestra.run.smithi125.stdout: "expected_num_objects": 0, 2021-03-24T22:00:20.545 INFO:teuthology.orchestra.run.smithi125.stdout: "fast_read": false, 2021-03-24T22:00:20.546 INFO:teuthology.orchestra.run.smithi125.stdout: "options": {}, 2021-03-24T22:00:20.546 INFO:teuthology.orchestra.run.smithi125.stdout: "application_metadata": { 2021-03-24T22:00:20.547 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw": {} 2021-03-24T22:00:20.547 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.548 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.548 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.548 INFO:teuthology.orchestra.run.smithi125.stdout: "pool": 4, 2021-03-24T22:00:20.548 INFO:teuthology.orchestra.run.smithi125.stdout: "pool_name": "default.rgw.log", 2021-03-24T22:00:20.549 INFO:teuthology.orchestra.run.smithi125.stdout: "create_time": "2021-03-24T21:51:46.090924+0000", 2021-03-24T22:00:20.549 INFO:teuthology.orchestra.run.smithi125.stdout: "flags": 1, 2021-03-24T22:00:20.549 INFO:teuthology.orchestra.run.smithi125.stdout: "flags_names": "hashpspool", 2021-03-24T22:00:20.550 INFO:teuthology.orchestra.run.smithi125.stdout: "type": 1, 2021-03-24T22:00:20.550 INFO:teuthology.orchestra.run.smithi125.stdout: "size": 3, 2021-03-24T22:00:20.550 INFO:teuthology.orchestra.run.smithi125.stdout: "min_size": 2, 2021-03-24T22:00:20.550 INFO:teuthology.orchestra.run.smithi125.stdout: "crush_rule": 0, 2021-03-24T22:00:20.551 INFO:teuthology.orchestra.run.smithi125.stdout: "peering_crush_bucket_count": 0, 2021-03-24T22:00:20.551 INFO:teuthology.orchestra.run.smithi125.stdout: "peering_crush_bucket_target": 0, 2021-03-24T22:00:20.551 INFO:teuthology.orchestra.run.smithi125.stdout: "peering_crush_bucket_barrier": 0, 2021-03-24T22:00:20.551 INFO:teuthology.orchestra.run.smithi125.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-24T22:00:20.551 INFO:teuthology.orchestra.run.smithi125.stdout: "object_hash": 2, 2021-03-24T22:00:20.552 INFO:teuthology.orchestra.run.smithi125.stdout: "pg_autoscale_mode": "off", 2021-03-24T22:00:20.552 INFO:teuthology.orchestra.run.smithi125.stdout: "pg_num": 28, 2021-03-24T22:00:20.552 INFO:teuthology.orchestra.run.smithi125.stdout: "pg_placement_num": 22, 2021-03-24T22:00:20.552 INFO:teuthology.orchestra.run.smithi125.stdout: "pg_placement_num_target": 22, 2021-03-24T22:00:20.552 INFO:teuthology.orchestra.run.smithi125.stdout: "pg_num_target": 22, 2021-03-24T22:00:20.553 INFO:teuthology.orchestra.run.smithi125.stdout: "pg_num_pending": 28, 2021-03-24T22:00:20.553 INFO:teuthology.orchestra.run.smithi125.stdout: "last_pg_merge_meta": { 2021-03-24T22:00:20.553 INFO:teuthology.orchestra.run.smithi125.stdout: "source_pgid": "4.1c", 2021-03-24T22:00:20.553 INFO:teuthology.orchestra.run.smithi125.stdout: "ready_epoch": 449, 2021-03-24T22:00:20.554 INFO:teuthology.orchestra.run.smithi125.stdout: "last_epoch_started": 448, 2021-03-24T22:00:20.554 INFO:teuthology.orchestra.run.smithi125.stdout: "last_epoch_clean": 448, 2021-03-24T22:00:20.554 INFO:teuthology.orchestra.run.smithi125.stdout: "source_version": "69'15", 2021-03-24T22:00:20.554 INFO:teuthology.orchestra.run.smithi125.stdout: "target_version": "69'13" 2021-03-24T22:00:20.555 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.555 INFO:teuthology.orchestra.run.smithi125.stdout: "last_change": "449", 2021-03-24T22:00:20.555 INFO:teuthology.orchestra.run.smithi125.stdout: "last_force_op_resend": "0", 2021-03-24T22:00:20.555 INFO:teuthology.orchestra.run.smithi125.stdout: "last_force_op_resend_prenautilus": "449", 2021-03-24T22:00:20.555 INFO:teuthology.orchestra.run.smithi125.stdout: "last_force_op_resend_preluminous": "447", 2021-03-24T22:00:20.556 INFO:teuthology.orchestra.run.smithi125.stdout: "auid": 0, 2021-03-24T22:00:20.556 INFO:teuthology.orchestra.run.smithi125.stdout: "snap_mode": "selfmanaged", 2021-03-24T22:00:20.556 INFO:teuthology.orchestra.run.smithi125.stdout: "snap_seq": 0, 2021-03-24T22:00:20.557 INFO:teuthology.orchestra.run.smithi125.stdout: "snap_epoch": 0, 2021-03-24T22:00:20.557 INFO:teuthology.orchestra.run.smithi125.stdout: "pool_snaps": [], 2021-03-24T22:00:20.557 INFO:teuthology.orchestra.run.smithi125.stdout: "removed_snaps": "[]", 2021-03-24T22:00:20.557 INFO:teuthology.orchestra.run.smithi125.stdout: "quota_max_bytes": 0, 2021-03-24T22:00:20.557 INFO:teuthology.orchestra.run.smithi125.stdout: "quota_max_objects": 0, 2021-03-24T22:00:20.558 INFO:teuthology.orchestra.run.smithi125.stdout: "tiers": [], 2021-03-24T22:00:20.558 INFO:teuthology.orchestra.run.smithi125.stdout: "tier_of": -1, 2021-03-24T22:00:20.558 INFO:teuthology.orchestra.run.smithi125.stdout: "read_tier": -1, 2021-03-24T22:00:20.559 INFO:teuthology.orchestra.run.smithi125.stdout: "write_tier": -1, 2021-03-24T22:00:20.559 INFO:teuthology.orchestra.run.smithi125.stdout: "cache_mode": "none", 2021-03-24T22:00:20.559 INFO:teuthology.orchestra.run.smithi125.stdout: "target_max_bytes": 0, 2021-03-24T22:00:20.559 INFO:teuthology.orchestra.run.smithi125.stdout: "target_max_objects": 0, 2021-03-24T22:00:20.559 INFO:teuthology.orchestra.run.smithi125.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-24T22:00:20.560 INFO:teuthology.orchestra.run.smithi125.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-24T22:00:20.560 INFO:teuthology.orchestra.run.smithi125.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-24T22:00:20.560 INFO:teuthology.orchestra.run.smithi125.stdout: "cache_min_flush_age": 0, 2021-03-24T22:00:20.560 INFO:teuthology.orchestra.run.smithi125.stdout: "cache_min_evict_age": 0, 2021-03-24T22:00:20.561 INFO:teuthology.orchestra.run.smithi125.stdout: "erasure_code_profile": "", 2021-03-24T22:00:20.561 INFO:teuthology.orchestra.run.smithi125.stdout: "hit_set_params": { 2021-03-24T22:00:20.561 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "none" 2021-03-24T22:00:20.561 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.561 INFO:teuthology.orchestra.run.smithi125.stdout: "hit_set_period": 0, 2021-03-24T22:00:20.562 INFO:teuthology.orchestra.run.smithi125.stdout: "hit_set_count": 0, 2021-03-24T22:00:20.562 INFO:teuthology.orchestra.run.smithi125.stdout: "use_gmt_hitset": true, 2021-03-24T22:00:20.562 INFO:teuthology.orchestra.run.smithi125.stdout: "min_read_recency_for_promote": 0, 2021-03-24T22:00:20.562 INFO:teuthology.orchestra.run.smithi125.stdout: "min_write_recency_for_promote": 0, 2021-03-24T22:00:20.563 INFO:teuthology.orchestra.run.smithi125.stdout: "hit_set_grade_decay_rate": 0, 2021-03-24T22:00:20.563 INFO:teuthology.orchestra.run.smithi125.stdout: "hit_set_search_last_n": 0, 2021-03-24T22:00:20.563 INFO:teuthology.orchestra.run.smithi125.stdout: "grade_table": [], 2021-03-24T22:00:20.563 INFO:teuthology.orchestra.run.smithi125.stdout: "stripe_width": 0, 2021-03-24T22:00:20.564 INFO:teuthology.orchestra.run.smithi125.stdout: "expected_num_objects": 0, 2021-03-24T22:00:20.564 INFO:teuthology.orchestra.run.smithi125.stdout: "fast_read": false, 2021-03-24T22:00:20.564 INFO:teuthology.orchestra.run.smithi125.stdout: "options": {}, 2021-03-24T22:00:20.564 INFO:teuthology.orchestra.run.smithi125.stdout: "application_metadata": { 2021-03-24T22:00:20.564 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw": {} 2021-03-24T22:00:20.565 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.565 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.565 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.565 INFO:teuthology.orchestra.run.smithi125.stdout: "pool": 5, 2021-03-24T22:00:20.566 INFO:teuthology.orchestra.run.smithi125.stdout: "pool_name": "default.rgw.control", 2021-03-24T22:00:20.566 INFO:teuthology.orchestra.run.smithi125.stdout: "create_time": "2021-03-24T21:51:47.842521+0000", 2021-03-24T22:00:20.566 INFO:teuthology.orchestra.run.smithi125.stdout: "flags": 1, 2021-03-24T22:00:20.567 INFO:teuthology.orchestra.run.smithi125.stdout: "flags_names": "hashpspool", 2021-03-24T22:00:20.567 INFO:teuthology.orchestra.run.smithi125.stdout: "type": 1, 2021-03-24T22:00:20.567 INFO:teuthology.orchestra.run.smithi125.stdout: "size": 3, 2021-03-24T22:00:20.567 INFO:teuthology.orchestra.run.smithi125.stdout: "min_size": 2, 2021-03-24T22:00:20.568 INFO:teuthology.orchestra.run.smithi125.stdout: "crush_rule": 0, 2021-03-24T22:00:20.568 INFO:teuthology.orchestra.run.smithi125.stdout: "peering_crush_bucket_count": 0, 2021-03-24T22:00:20.568 INFO:teuthology.orchestra.run.smithi125.stdout: "peering_crush_bucket_target": 0, 2021-03-24T22:00:20.568 INFO:teuthology.orchestra.run.smithi125.stdout: "peering_crush_bucket_barrier": 0, 2021-03-24T22:00:20.568 INFO:teuthology.orchestra.run.smithi125.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-24T22:00:20.569 INFO:teuthology.orchestra.run.smithi125.stdout: "object_hash": 2, 2021-03-24T22:00:20.569 INFO:teuthology.orchestra.run.smithi125.stdout: "pg_autoscale_mode": "off", 2021-03-24T22:00:20.569 INFO:teuthology.orchestra.run.smithi125.stdout: "pg_num": 32, 2021-03-24T22:00:20.570 INFO:teuthology.orchestra.run.smithi125.stdout: "pg_placement_num": 32, 2021-03-24T22:00:20.570 INFO:teuthology.orchestra.run.smithi125.stdout: "pg_placement_num_target": 32, 2021-03-24T22:00:20.570 INFO:teuthology.orchestra.run.smithi125.stdout: "pg_num_target": 32, 2021-03-24T22:00:20.570 INFO:teuthology.orchestra.run.smithi125.stdout: "pg_num_pending": 32, 2021-03-24T22:00:20.571 INFO:teuthology.orchestra.run.smithi125.stdout: "last_pg_merge_meta": { 2021-03-24T22:00:20.571 INFO:teuthology.orchestra.run.smithi125.stdout: "source_pgid": "0.0", 2021-03-24T22:00:20.571 INFO:teuthology.orchestra.run.smithi125.stdout: "ready_epoch": 0, 2021-03-24T22:00:20.571 INFO:teuthology.orchestra.run.smithi125.stdout: "last_epoch_started": 0, 2021-03-24T22:00:20.571 INFO:teuthology.orchestra.run.smithi125.stdout: "last_epoch_clean": 0, 2021-03-24T22:00:20.572 INFO:teuthology.orchestra.run.smithi125.stdout: "source_version": "0'0", 2021-03-24T22:00:20.572 INFO:teuthology.orchestra.run.smithi125.stdout: "target_version": "0'0" 2021-03-24T22:00:20.572 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.572 INFO:teuthology.orchestra.run.smithi125.stdout: "last_change": "66", 2021-03-24T22:00:20.573 INFO:teuthology.orchestra.run.smithi125.stdout: "last_force_op_resend": "0", 2021-03-24T22:00:20.573 INFO:teuthology.orchestra.run.smithi125.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-24T22:00:20.573 INFO:teuthology.orchestra.run.smithi125.stdout: "last_force_op_resend_preluminous": "0", 2021-03-24T22:00:20.573 INFO:teuthology.orchestra.run.smithi125.stdout: "auid": 0, 2021-03-24T22:00:20.573 INFO:teuthology.orchestra.run.smithi125.stdout: "snap_mode": "selfmanaged", 2021-03-24T22:00:20.574 INFO:teuthology.orchestra.run.smithi125.stdout: "snap_seq": 0, 2021-03-24T22:00:20.574 INFO:teuthology.orchestra.run.smithi125.stdout: "snap_epoch": 0, 2021-03-24T22:00:20.574 INFO:teuthology.orchestra.run.smithi125.stdout: "pool_snaps": [], 2021-03-24T22:00:20.574 INFO:teuthology.orchestra.run.smithi125.stdout: "removed_snaps": "[]", 2021-03-24T22:00:20.575 INFO:teuthology.orchestra.run.smithi125.stdout: "quota_max_bytes": 0, 2021-03-24T22:00:20.575 INFO:teuthology.orchestra.run.smithi125.stdout: "quota_max_objects": 0, 2021-03-24T22:00:20.575 INFO:teuthology.orchestra.run.smithi125.stdout: "tiers": [], 2021-03-24T22:00:20.575 INFO:teuthology.orchestra.run.smithi125.stdout: "tier_of": -1, 2021-03-24T22:00:20.575 INFO:teuthology.orchestra.run.smithi125.stdout: "read_tier": -1, 2021-03-24T22:00:20.576 INFO:teuthology.orchestra.run.smithi125.stdout: "write_tier": -1, 2021-03-24T22:00:20.576 INFO:teuthology.orchestra.run.smithi125.stdout: "cache_mode": "none", 2021-03-24T22:00:20.576 INFO:teuthology.orchestra.run.smithi125.stdout: "target_max_bytes": 0, 2021-03-24T22:00:20.576 INFO:teuthology.orchestra.run.smithi125.stdout: "target_max_objects": 0, 2021-03-24T22:00:20.576 INFO:teuthology.orchestra.run.smithi125.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-24T22:00:20.577 INFO:teuthology.orchestra.run.smithi125.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-24T22:00:20.577 INFO:teuthology.orchestra.run.smithi125.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-24T22:00:20.577 INFO:teuthology.orchestra.run.smithi125.stdout: "cache_min_flush_age": 0, 2021-03-24T22:00:20.577 INFO:teuthology.orchestra.run.smithi125.stdout: "cache_min_evict_age": 0, 2021-03-24T22:00:20.578 INFO:teuthology.orchestra.run.smithi125.stdout: "erasure_code_profile": "", 2021-03-24T22:00:20.578 INFO:teuthology.orchestra.run.smithi125.stdout: "hit_set_params": { 2021-03-24T22:00:20.578 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "none" 2021-03-24T22:00:20.578 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.579 INFO:teuthology.orchestra.run.smithi125.stdout: "hit_set_period": 0, 2021-03-24T22:00:20.579 INFO:teuthology.orchestra.run.smithi125.stdout: "hit_set_count": 0, 2021-03-24T22:00:20.579 INFO:teuthology.orchestra.run.smithi125.stdout: "use_gmt_hitset": true, 2021-03-24T22:00:20.579 INFO:teuthology.orchestra.run.smithi125.stdout: "min_read_recency_for_promote": 0, 2021-03-24T22:00:20.580 INFO:teuthology.orchestra.run.smithi125.stdout: "min_write_recency_for_promote": 0, 2021-03-24T22:00:20.580 INFO:teuthology.orchestra.run.smithi125.stdout: "hit_set_grade_decay_rate": 0, 2021-03-24T22:00:20.580 INFO:teuthology.orchestra.run.smithi125.stdout: "hit_set_search_last_n": 0, 2021-03-24T22:00:20.580 INFO:teuthology.orchestra.run.smithi125.stdout: "grade_table": [], 2021-03-24T22:00:20.581 INFO:teuthology.orchestra.run.smithi125.stdout: "stripe_width": 0, 2021-03-24T22:00:20.581 INFO:teuthology.orchestra.run.smithi125.stdout: "expected_num_objects": 0, 2021-03-24T22:00:20.581 INFO:teuthology.orchestra.run.smithi125.stdout: "fast_read": false, 2021-03-24T22:00:20.581 INFO:teuthology.orchestra.run.smithi125.stdout: "options": {}, 2021-03-24T22:00:20.581 INFO:teuthology.orchestra.run.smithi125.stdout: "application_metadata": { 2021-03-24T22:00:20.582 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw": {} 2021-03-24T22:00:20.582 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.582 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.582 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.583 INFO:teuthology.orchestra.run.smithi125.stdout: "pool": 6, 2021-03-24T22:00:20.583 INFO:teuthology.orchestra.run.smithi125.stdout: "pool_name": "default.rgw.meta", 2021-03-24T22:00:20.583 INFO:teuthology.orchestra.run.smithi125.stdout: "create_time": "2021-03-24T21:51:50.730906+0000", 2021-03-24T22:00:20.583 INFO:teuthology.orchestra.run.smithi125.stdout: "flags": 1, 2021-03-24T22:00:20.583 INFO:teuthology.orchestra.run.smithi125.stdout: "flags_names": "hashpspool", 2021-03-24T22:00:20.584 INFO:teuthology.orchestra.run.smithi125.stdout: "type": 1, 2021-03-24T22:00:20.584 INFO:teuthology.orchestra.run.smithi125.stdout: "size": 3, 2021-03-24T22:00:20.584 INFO:teuthology.orchestra.run.smithi125.stdout: "min_size": 2, 2021-03-24T22:00:20.584 INFO:teuthology.orchestra.run.smithi125.stdout: "crush_rule": 0, 2021-03-24T22:00:20.585 INFO:teuthology.orchestra.run.smithi125.stdout: "peering_crush_bucket_count": 0, 2021-03-24T22:00:20.585 INFO:teuthology.orchestra.run.smithi125.stdout: "peering_crush_bucket_target": 0, 2021-03-24T22:00:20.585 INFO:teuthology.orchestra.run.smithi125.stdout: "peering_crush_bucket_barrier": 0, 2021-03-24T22:00:20.585 INFO:teuthology.orchestra.run.smithi125.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-24T22:00:20.586 INFO:teuthology.orchestra.run.smithi125.stdout: "object_hash": 2, 2021-03-24T22:00:20.586 INFO:teuthology.orchestra.run.smithi125.stdout: "pg_autoscale_mode": "off", 2021-03-24T22:00:20.586 INFO:teuthology.orchestra.run.smithi125.stdout: "pg_num": 32, 2021-03-24T22:00:20.586 INFO:teuthology.orchestra.run.smithi125.stdout: "pg_placement_num": 32, 2021-03-24T22:00:20.587 INFO:teuthology.orchestra.run.smithi125.stdout: "pg_placement_num_target": 32, 2021-03-24T22:00:20.587 INFO:teuthology.orchestra.run.smithi125.stdout: "pg_num_target": 32, 2021-03-24T22:00:20.587 INFO:teuthology.orchestra.run.smithi125.stdout: "pg_num_pending": 32, 2021-03-24T22:00:20.587 INFO:teuthology.orchestra.run.smithi125.stdout: "last_pg_merge_meta": { 2021-03-24T22:00:20.587 INFO:teuthology.orchestra.run.smithi125.stdout: "source_pgid": "0.0", 2021-03-24T22:00:20.588 INFO:teuthology.orchestra.run.smithi125.stdout: "ready_epoch": 0, 2021-03-24T22:00:20.588 INFO:teuthology.orchestra.run.smithi125.stdout: "last_epoch_started": 0, 2021-03-24T22:00:20.588 INFO:teuthology.orchestra.run.smithi125.stdout: "last_epoch_clean": 0, 2021-03-24T22:00:20.588 INFO:teuthology.orchestra.run.smithi125.stdout: "source_version": "0'0", 2021-03-24T22:00:20.588 INFO:teuthology.orchestra.run.smithi125.stdout: "target_version": "0'0" 2021-03-24T22:00:20.589 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.589 INFO:teuthology.orchestra.run.smithi125.stdout: "last_change": "69", 2021-03-24T22:00:20.589 INFO:teuthology.orchestra.run.smithi125.stdout: "last_force_op_resend": "0", 2021-03-24T22:00:20.589 INFO:teuthology.orchestra.run.smithi125.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-24T22:00:20.589 INFO:teuthology.orchestra.run.smithi125.stdout: "last_force_op_resend_preluminous": "0", 2021-03-24T22:00:20.590 INFO:teuthology.orchestra.run.smithi125.stdout: "auid": 0, 2021-03-24T22:00:20.590 INFO:teuthology.orchestra.run.smithi125.stdout: "snap_mode": "selfmanaged", 2021-03-24T22:00:20.590 INFO:teuthology.orchestra.run.smithi125.stdout: "snap_seq": 0, 2021-03-24T22:00:20.591 INFO:teuthology.orchestra.run.smithi125.stdout: "snap_epoch": 0, 2021-03-24T22:00:20.591 INFO:teuthology.orchestra.run.smithi125.stdout: "pool_snaps": [], 2021-03-24T22:00:20.591 INFO:teuthology.orchestra.run.smithi125.stdout: "removed_snaps": "[]", 2021-03-24T22:00:20.591 INFO:teuthology.orchestra.run.smithi125.stdout: "quota_max_bytes": 0, 2021-03-24T22:00:20.592 INFO:teuthology.orchestra.run.smithi125.stdout: "quota_max_objects": 0, 2021-03-24T22:00:20.592 INFO:teuthology.orchestra.run.smithi125.stdout: "tiers": [], 2021-03-24T22:00:20.592 INFO:teuthology.orchestra.run.smithi125.stdout: "tier_of": -1, 2021-03-24T22:00:20.592 INFO:teuthology.orchestra.run.smithi125.stdout: "read_tier": -1, 2021-03-24T22:00:20.593 INFO:teuthology.orchestra.run.smithi125.stdout: "write_tier": -1, 2021-03-24T22:00:20.593 INFO:teuthology.orchestra.run.smithi125.stdout: "cache_mode": "none", 2021-03-24T22:00:20.593 INFO:teuthology.orchestra.run.smithi125.stdout: "target_max_bytes": 0, 2021-03-24T22:00:20.593 INFO:teuthology.orchestra.run.smithi125.stdout: "target_max_objects": 0, 2021-03-24T22:00:20.594 INFO:teuthology.orchestra.run.smithi125.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-24T22:00:20.594 INFO:teuthology.orchestra.run.smithi125.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-24T22:00:20.594 INFO:teuthology.orchestra.run.smithi125.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-24T22:00:20.594 INFO:teuthology.orchestra.run.smithi125.stdout: "cache_min_flush_age": 0, 2021-03-24T22:00:20.594 INFO:teuthology.orchestra.run.smithi125.stdout: "cache_min_evict_age": 0, 2021-03-24T22:00:20.595 INFO:teuthology.orchestra.run.smithi125.stdout: "erasure_code_profile": "", 2021-03-24T22:00:20.595 INFO:teuthology.orchestra.run.smithi125.stdout: "hit_set_params": { 2021-03-24T22:00:20.595 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "none" 2021-03-24T22:00:20.595 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.595 INFO:teuthology.orchestra.run.smithi125.stdout: "hit_set_period": 0, 2021-03-24T22:00:20.596 INFO:teuthology.orchestra.run.smithi125.stdout: "hit_set_count": 0, 2021-03-24T22:00:20.596 INFO:teuthology.orchestra.run.smithi125.stdout: "use_gmt_hitset": true, 2021-03-24T22:00:20.596 INFO:teuthology.orchestra.run.smithi125.stdout: "min_read_recency_for_promote": 0, 2021-03-24T22:00:20.597 INFO:teuthology.orchestra.run.smithi125.stdout: "min_write_recency_for_promote": 0, 2021-03-24T22:00:20.597 INFO:teuthology.orchestra.run.smithi125.stdout: "hit_set_grade_decay_rate": 0, 2021-03-24T22:00:20.597 INFO:teuthology.orchestra.run.smithi125.stdout: "hit_set_search_last_n": 0, 2021-03-24T22:00:20.597 INFO:teuthology.orchestra.run.smithi125.stdout: "grade_table": [], 2021-03-24T22:00:20.597 INFO:teuthology.orchestra.run.smithi125.stdout: "stripe_width": 0, 2021-03-24T22:00:20.598 INFO:teuthology.orchestra.run.smithi125.stdout: "expected_num_objects": 0, 2021-03-24T22:00:20.598 INFO:teuthology.orchestra.run.smithi125.stdout: "fast_read": false, 2021-03-24T22:00:20.598 INFO:teuthology.orchestra.run.smithi125.stdout: "options": { 2021-03-24T22:00:20.598 INFO:teuthology.orchestra.run.smithi125.stdout: "pg_autoscale_bias": 4, 2021-03-24T22:00:20.598 INFO:teuthology.orchestra.run.smithi125.stdout: "pg_num_min": 8 2021-03-24T22:00:20.599 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.599 INFO:teuthology.orchestra.run.smithi125.stdout: "application_metadata": { 2021-03-24T22:00:20.599 INFO:teuthology.orchestra.run.smithi125.stdout: "rgw": {} 2021-03-24T22:00:20.599 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.600 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.600 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.600 INFO:teuthology.orchestra.run.smithi125.stdout: "pool": 7, 2021-03-24T22:00:20.600 INFO:teuthology.orchestra.run.smithi125.stdout: "pool_name": "unique_pool_0", 2021-03-24T22:00:20.601 INFO:teuthology.orchestra.run.smithi125.stdout: "create_time": "2021-03-24T21:54:21.087744+0000", 2021-03-24T22:00:20.601 INFO:teuthology.orchestra.run.smithi125.stdout: "flags": 8193, 2021-03-24T22:00:20.602 INFO:teuthology.orchestra.run.smithi125.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2021-03-24T22:00:20.602 INFO:teuthology.orchestra.run.smithi125.stdout: "type": 1, 2021-03-24T22:00:20.602 INFO:teuthology.orchestra.run.smithi125.stdout: "size": 3, 2021-03-24T22:00:20.603 INFO:teuthology.orchestra.run.smithi125.stdout: "min_size": 2, 2021-03-24T22:00:20.603 INFO:teuthology.orchestra.run.smithi125.stdout: "crush_rule": 0, 2021-03-24T22:00:20.604 INFO:teuthology.orchestra.run.smithi125.stdout: "peering_crush_bucket_count": 0, 2021-03-24T22:00:20.604 INFO:teuthology.orchestra.run.smithi125.stdout: "peering_crush_bucket_target": 0, 2021-03-24T22:00:20.605 INFO:teuthology.orchestra.run.smithi125.stdout: "peering_crush_bucket_barrier": 0, 2021-03-24T22:00:20.605 INFO:teuthology.orchestra.run.smithi125.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-24T22:00:20.605 INFO:teuthology.orchestra.run.smithi125.stdout: "object_hash": 2, 2021-03-24T22:00:20.606 INFO:teuthology.orchestra.run.smithi125.stdout: "pg_autoscale_mode": "off", 2021-03-24T22:00:20.606 INFO:teuthology.orchestra.run.smithi125.stdout: "pg_num": 36, 2021-03-24T22:00:20.607 INFO:teuthology.orchestra.run.smithi125.stdout: "pg_placement_num": 36, 2021-03-24T22:00:20.607 INFO:teuthology.orchestra.run.smithi125.stdout: "pg_placement_num_target": 36, 2021-03-24T22:00:20.607 INFO:teuthology.orchestra.run.smithi125.stdout: "pg_num_target": 36, 2021-03-24T22:00:20.608 INFO:teuthology.orchestra.run.smithi125.stdout: "pg_num_pending": 36, 2021-03-24T22:00:20.608 INFO:teuthology.orchestra.run.smithi125.stdout: "last_pg_merge_meta": { 2021-03-24T22:00:20.609 INFO:teuthology.orchestra.run.smithi125.stdout: "source_pgid": "0.0", 2021-03-24T22:00:20.609 INFO:teuthology.orchestra.run.smithi125.stdout: "ready_epoch": 0, 2021-03-24T22:00:20.609 INFO:teuthology.orchestra.run.smithi125.stdout: "last_epoch_started": 0, 2021-03-24T22:00:20.610 INFO:teuthology.orchestra.run.smithi125.stdout: "last_epoch_clean": 0, 2021-03-24T22:00:20.610 INFO:teuthology.orchestra.run.smithi125.stdout: "source_version": "0'0", 2021-03-24T22:00:20.610 INFO:teuthology.orchestra.run.smithi125.stdout: "target_version": "0'0" 2021-03-24T22:00:20.611 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.611 INFO:teuthology.orchestra.run.smithi125.stdout: "last_change": "450", 2021-03-24T22:00:20.611 INFO:teuthology.orchestra.run.smithi125.stdout: "last_force_op_resend": "0", 2021-03-24T22:00:20.612 INFO:teuthology.orchestra.run.smithi125.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-24T22:00:20.612 INFO:teuthology.orchestra.run.smithi125.stdout: "last_force_op_resend_preluminous": "275", 2021-03-24T22:00:20.612 INFO:teuthology.orchestra.run.smithi125.stdout: "auid": 0, 2021-03-24T22:00:20.613 INFO:teuthology.orchestra.run.smithi125.stdout: "snap_mode": "selfmanaged", 2021-03-24T22:00:20.613 INFO:teuthology.orchestra.run.smithi125.stdout: "snap_seq": 340, 2021-03-24T22:00:20.613 INFO:teuthology.orchestra.run.smithi125.stdout: "snap_epoch": 450, 2021-03-24T22:00:20.613 INFO:teuthology.orchestra.run.smithi125.stdout: "pool_snaps": [], 2021-03-24T22:00:20.613 INFO:teuthology.orchestra.run.smithi125.stdout: "removed_snaps": "[]", 2021-03-24T22:00:20.614 INFO:teuthology.orchestra.run.smithi125.stdout: "quota_max_bytes": 0, 2021-03-24T22:00:20.614 INFO:teuthology.orchestra.run.smithi125.stdout: "quota_max_objects": 0, 2021-03-24T22:00:20.614 INFO:teuthology.orchestra.run.smithi125.stdout: "tiers": [], 2021-03-24T22:00:20.614 INFO:teuthology.orchestra.run.smithi125.stdout: "tier_of": -1, 2021-03-24T22:00:20.615 INFO:teuthology.orchestra.run.smithi125.stdout: "read_tier": -1, 2021-03-24T22:00:20.615 INFO:teuthology.orchestra.run.smithi125.stdout: "write_tier": -1, 2021-03-24T22:00:20.615 INFO:teuthology.orchestra.run.smithi125.stdout: "cache_mode": "none", 2021-03-24T22:00:20.615 INFO:teuthology.orchestra.run.smithi125.stdout: "target_max_bytes": 0, 2021-03-24T22:00:20.616 INFO:teuthology.orchestra.run.smithi125.stdout: "target_max_objects": 0, 2021-03-24T22:00:20.616 INFO:teuthology.orchestra.run.smithi125.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-24T22:00:20.616 INFO:teuthology.orchestra.run.smithi125.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-24T22:00:20.616 INFO:teuthology.orchestra.run.smithi125.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-24T22:00:20.617 INFO:teuthology.orchestra.run.smithi125.stdout: "cache_min_flush_age": 0, 2021-03-24T22:00:20.617 INFO:teuthology.orchestra.run.smithi125.stdout: "cache_min_evict_age": 0, 2021-03-24T22:00:20.617 INFO:teuthology.orchestra.run.smithi125.stdout: "erasure_code_profile": "", 2021-03-24T22:00:20.617 INFO:teuthology.orchestra.run.smithi125.stdout: "hit_set_params": { 2021-03-24T22:00:20.617 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "none" 2021-03-24T22:00:20.618 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.618 INFO:teuthology.orchestra.run.smithi125.stdout: "hit_set_period": 0, 2021-03-24T22:00:20.618 INFO:teuthology.orchestra.run.smithi125.stdout: "hit_set_count": 0, 2021-03-24T22:00:20.618 INFO:teuthology.orchestra.run.smithi125.stdout: "use_gmt_hitset": true, 2021-03-24T22:00:20.619 INFO:teuthology.orchestra.run.smithi125.stdout: "min_read_recency_for_promote": 0, 2021-03-24T22:00:20.619 INFO:teuthology.orchestra.run.smithi125.stdout: "min_write_recency_for_promote": 0, 2021-03-24T22:00:20.619 INFO:teuthology.orchestra.run.smithi125.stdout: "hit_set_grade_decay_rate": 0, 2021-03-24T22:00:20.619 INFO:teuthology.orchestra.run.smithi125.stdout: "hit_set_search_last_n": 0, 2021-03-24T22:00:20.619 INFO:teuthology.orchestra.run.smithi125.stdout: "grade_table": [], 2021-03-24T22:00:20.620 INFO:teuthology.orchestra.run.smithi125.stdout: "stripe_width": 0, 2021-03-24T22:00:20.620 INFO:teuthology.orchestra.run.smithi125.stdout: "expected_num_objects": 0, 2021-03-24T22:00:20.620 INFO:teuthology.orchestra.run.smithi125.stdout: "fast_read": false, 2021-03-24T22:00:20.620 INFO:teuthology.orchestra.run.smithi125.stdout: "options": {}, 2021-03-24T22:00:20.621 INFO:teuthology.orchestra.run.smithi125.stdout: "application_metadata": { 2021-03-24T22:00:20.621 INFO:teuthology.orchestra.run.smithi125.stdout: "rados": {} 2021-03-24T22:00:20.621 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.621 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.621 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2021-03-24T22:00:20.622 INFO:teuthology.orchestra.run.smithi125.stdout: "osds": [ 2021-03-24T22:00:20.622 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.622 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": 0, 2021-03-24T22:00:20.623 INFO:teuthology.orchestra.run.smithi125.stdout: "uuid": "1f8f9a69-4b85-4054-8b11-36d35574494c", 2021-03-24T22:00:20.623 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2021-03-24T22:00:20.623 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2021-03-24T22:00:20.623 INFO:teuthology.orchestra.run.smithi125.stdout: "weight": 1, 2021-03-24T22:00:20.624 INFO:teuthology.orchestra.run.smithi125.stdout: "primary_affinity": 1, 2021-03-24T22:00:20.624 INFO:teuthology.orchestra.run.smithi125.stdout: "last_clean_begin": 0, 2021-03-24T22:00:20.625 INFO:teuthology.orchestra.run.smithi125.stdout: "last_clean_end": 0, 2021-03-24T22:00:20.625 INFO:teuthology.orchestra.run.smithi125.stdout: "up_from": 8, 2021-03-24T22:00:20.625 INFO:teuthology.orchestra.run.smithi125.stdout: "up_thru": 439, 2021-03-24T22:00:20.626 INFO:teuthology.orchestra.run.smithi125.stdout: "down_at": 0, 2021-03-24T22:00:20.626 INFO:teuthology.orchestra.run.smithi125.stdout: "lost_at": 0, 2021-03-24T22:00:20.626 INFO:teuthology.orchestra.run.smithi125.stdout: "public_addrs": { 2021-03-24T22:00:20.627 INFO:teuthology.orchestra.run.smithi125.stdout: "addrvec": [ 2021-03-24T22:00:20.627 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.627 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2021-03-24T22:00:20.627 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "172.21.15.125:6801", 2021-03-24T22:00:20.628 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 1197472172 2021-03-24T22:00:20.628 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.628 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:20.628 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.628 INFO:teuthology.orchestra.run.smithi125.stdout: "cluster_addrs": { 2021-03-24T22:00:20.629 INFO:teuthology.orchestra.run.smithi125.stdout: "addrvec": [ 2021-03-24T22:00:20.629 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.629 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2021-03-24T22:00:20.629 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "172.21.15.125:6802", 2021-03-24T22:00:20.629 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 1197472172 2021-03-24T22:00:20.630 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.630 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:20.630 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.630 INFO:teuthology.orchestra.run.smithi125.stdout: "heartbeat_back_addrs": { 2021-03-24T22:00:20.631 INFO:teuthology.orchestra.run.smithi125.stdout: "addrvec": [ 2021-03-24T22:00:20.631 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.631 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2021-03-24T22:00:20.631 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "172.21.15.125:6804", 2021-03-24T22:00:20.632 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 1197472172 2021-03-24T22:00:20.632 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.632 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:20.632 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.633 INFO:teuthology.orchestra.run.smithi125.stdout: "heartbeat_front_addrs": { 2021-03-24T22:00:20.633 INFO:teuthology.orchestra.run.smithi125.stdout: "addrvec": [ 2021-03-24T22:00:20.633 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.633 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2021-03-24T22:00:20.633 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "172.21.15.125:6803", 2021-03-24T22:00:20.634 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 1197472172 2021-03-24T22:00:20.634 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.634 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:20.634 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.634 INFO:teuthology.orchestra.run.smithi125.stdout: "public_addr": "172.21.15.125:6801/1197472172", 2021-03-24T22:00:20.635 INFO:teuthology.orchestra.run.smithi125.stdout: "cluster_addr": "172.21.15.125:6802/1197472172", 2021-03-24T22:00:20.635 INFO:teuthology.orchestra.run.smithi125.stdout: "heartbeat_back_addr": "172.21.15.125:6804/1197472172", 2021-03-24T22:00:20.635 INFO:teuthology.orchestra.run.smithi125.stdout: "heartbeat_front_addr": "172.21.15.125:6803/1197472172", 2021-03-24T22:00:20.635 INFO:teuthology.orchestra.run.smithi125.stdout: "state": [ 2021-03-24T22:00:20.636 INFO:teuthology.orchestra.run.smithi125.stdout: "exists", 2021-03-24T22:00:20.636 INFO:teuthology.orchestra.run.smithi125.stdout: "up" 2021-03-24T22:00:20.636 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:20.636 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.637 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.637 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": 1, 2021-03-24T22:00:20.637 INFO:teuthology.orchestra.run.smithi125.stdout: "uuid": "48db0723-5806-4e5c-9919-f3d7f52ad169", 2021-03-24T22:00:20.637 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2021-03-24T22:00:20.637 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 0, 2021-03-24T22:00:20.638 INFO:teuthology.orchestra.run.smithi125.stdout: "weight": 0, 2021-03-24T22:00:20.638 INFO:teuthology.orchestra.run.smithi125.stdout: "primary_affinity": 1, 2021-03-24T22:00:20.638 INFO:teuthology.orchestra.run.smithi125.stdout: "last_clean_begin": 0, 2021-03-24T22:00:20.638 INFO:teuthology.orchestra.run.smithi125.stdout: "last_clean_end": 0, 2021-03-24T22:00:20.639 INFO:teuthology.orchestra.run.smithi125.stdout: "up_from": 13, 2021-03-24T22:00:20.639 INFO:teuthology.orchestra.run.smithi125.stdout: "up_thru": 101, 2021-03-24T22:00:20.639 INFO:teuthology.orchestra.run.smithi125.stdout: "down_at": 0, 2021-03-24T22:00:20.639 INFO:teuthology.orchestra.run.smithi125.stdout: "lost_at": 0, 2021-03-24T22:00:20.640 INFO:teuthology.orchestra.run.smithi125.stdout: "public_addrs": { 2021-03-24T22:00:20.640 INFO:teuthology.orchestra.run.smithi125.stdout: "addrvec": [ 2021-03-24T22:00:20.640 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.641 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2021-03-24T22:00:20.641 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "172.21.15.125:6805", 2021-03-24T22:00:20.641 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 113845288 2021-03-24T22:00:20.641 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.641 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:20.642 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.642 INFO:teuthology.orchestra.run.smithi125.stdout: "cluster_addrs": { 2021-03-24T22:00:20.642 INFO:teuthology.orchestra.run.smithi125.stdout: "addrvec": [ 2021-03-24T22:00:20.642 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.643 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2021-03-24T22:00:20.643 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "172.21.15.125:6806", 2021-03-24T22:00:20.643 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 113845288 2021-03-24T22:00:20.643 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.644 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:20.644 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.644 INFO:teuthology.orchestra.run.smithi125.stdout: "heartbeat_back_addrs": { 2021-03-24T22:00:20.644 INFO:teuthology.orchestra.run.smithi125.stdout: "addrvec": [ 2021-03-24T22:00:20.645 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.645 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2021-03-24T22:00:20.645 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "172.21.15.125:6808", 2021-03-24T22:00:20.645 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 113845288 2021-03-24T22:00:20.645 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.646 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:20.646 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.646 INFO:teuthology.orchestra.run.smithi125.stdout: "heartbeat_front_addrs": { 2021-03-24T22:00:20.646 INFO:teuthology.orchestra.run.smithi125.stdout: "addrvec": [ 2021-03-24T22:00:20.647 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.647 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2021-03-24T22:00:20.647 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "172.21.15.125:6807", 2021-03-24T22:00:20.647 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 113845288 2021-03-24T22:00:20.647 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.648 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:20.648 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.648 INFO:teuthology.orchestra.run.smithi125.stdout: "public_addr": "172.21.15.125:6805/113845288", 2021-03-24T22:00:20.648 INFO:teuthology.orchestra.run.smithi125.stdout: "cluster_addr": "172.21.15.125:6806/113845288", 2021-03-24T22:00:20.649 INFO:teuthology.orchestra.run.smithi125.stdout: "heartbeat_back_addr": "172.21.15.125:6808/113845288", 2021-03-24T22:00:20.649 INFO:teuthology.orchestra.run.smithi125.stdout: "heartbeat_front_addr": "172.21.15.125:6807/113845288", 2021-03-24T22:00:20.649 INFO:teuthology.orchestra.run.smithi125.stdout: "state": [ 2021-03-24T22:00:20.649 INFO:teuthology.orchestra.run.smithi125.stdout: "exists", 2021-03-24T22:00:20.650 INFO:teuthology.orchestra.run.smithi125.stdout: "up" 2021-03-24T22:00:20.650 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:20.650 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.650 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.651 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": 2, 2021-03-24T22:00:20.651 INFO:teuthology.orchestra.run.smithi125.stdout: "uuid": "d6a361f8-4e4d-4f46-8ab9-01bf71dbb2c8", 2021-03-24T22:00:20.651 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2021-03-24T22:00:20.651 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2021-03-24T22:00:20.652 INFO:teuthology.orchestra.run.smithi125.stdout: "weight": 0.7242584228515625, 2021-03-24T22:00:20.652 INFO:teuthology.orchestra.run.smithi125.stdout: "primary_affinity": 1, 2021-03-24T22:00:20.653 INFO:teuthology.orchestra.run.smithi125.stdout: "last_clean_begin": 0, 2021-03-24T22:00:20.653 INFO:teuthology.orchestra.run.smithi125.stdout: "last_clean_end": 0, 2021-03-24T22:00:20.654 INFO:teuthology.orchestra.run.smithi125.stdout: "up_from": 18, 2021-03-24T22:00:20.654 INFO:teuthology.orchestra.run.smithi125.stdout: "up_thru": 433, 2021-03-24T22:00:20.654 INFO:teuthology.orchestra.run.smithi125.stdout: "down_at": 0, 2021-03-24T22:00:20.655 INFO:teuthology.orchestra.run.smithi125.stdout: "lost_at": 0, 2021-03-24T22:00:20.655 INFO:teuthology.orchestra.run.smithi125.stdout: "public_addrs": { 2021-03-24T22:00:20.656 INFO:teuthology.orchestra.run.smithi125.stdout: "addrvec": [ 2021-03-24T22:00:20.656 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.657 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2021-03-24T22:00:20.657 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "172.21.15.125:6809", 2021-03-24T22:00:20.657 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 246939163 2021-03-24T22:00:20.657 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.658 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:20.658 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.658 INFO:teuthology.orchestra.run.smithi125.stdout: "cluster_addrs": { 2021-03-24T22:00:20.658 INFO:teuthology.orchestra.run.smithi125.stdout: "addrvec": [ 2021-03-24T22:00:20.659 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.659 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2021-03-24T22:00:20.659 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "172.21.15.125:6810", 2021-03-24T22:00:20.659 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 246939163 2021-03-24T22:00:20.659 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.660 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:20.660 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.660 INFO:teuthology.orchestra.run.smithi125.stdout: "heartbeat_back_addrs": { 2021-03-24T22:00:20.660 INFO:teuthology.orchestra.run.smithi125.stdout: "addrvec": [ 2021-03-24T22:00:20.660 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.661 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2021-03-24T22:00:20.661 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "172.21.15.125:6812", 2021-03-24T22:00:20.661 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 246939163 2021-03-24T22:00:20.661 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.662 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:20.662 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.662 INFO:teuthology.orchestra.run.smithi125.stdout: "heartbeat_front_addrs": { 2021-03-24T22:00:20.662 INFO:teuthology.orchestra.run.smithi125.stdout: "addrvec": [ 2021-03-24T22:00:20.663 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.663 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2021-03-24T22:00:20.663 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "172.21.15.125:6811", 2021-03-24T22:00:20.664 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 246939163 2021-03-24T22:00:20.664 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.664 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:20.665 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.665 INFO:teuthology.orchestra.run.smithi125.stdout: "public_addr": "172.21.15.125:6809/246939163", 2021-03-24T22:00:20.666 INFO:teuthology.orchestra.run.smithi125.stdout: "cluster_addr": "172.21.15.125:6810/246939163", 2021-03-24T22:00:20.666 INFO:teuthology.orchestra.run.smithi125.stdout: "heartbeat_back_addr": "172.21.15.125:6812/246939163", 2021-03-24T22:00:20.667 INFO:teuthology.orchestra.run.smithi125.stdout: "heartbeat_front_addr": "172.21.15.125:6811/246939163", 2021-03-24T22:00:20.667 INFO:teuthology.orchestra.run.smithi125.stdout: "state": [ 2021-03-24T22:00:20.668 INFO:teuthology.orchestra.run.smithi125.stdout: "exists", 2021-03-24T22:00:20.668 INFO:teuthology.orchestra.run.smithi125.stdout: "up" 2021-03-24T22:00:20.669 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:20.669 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.669 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.670 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": 3, 2021-03-24T22:00:20.670 INFO:teuthology.orchestra.run.smithi125.stdout: "uuid": "716a41ce-7a28-400f-bf75-4c1ad99c7f94", 2021-03-24T22:00:20.670 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2021-03-24T22:00:20.671 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2021-03-24T22:00:20.671 INFO:teuthology.orchestra.run.smithi125.stdout: "weight": 0.9000244140625, 2021-03-24T22:00:20.672 INFO:teuthology.orchestra.run.smithi125.stdout: "primary_affinity": 1, 2021-03-24T22:00:20.672 INFO:teuthology.orchestra.run.smithi125.stdout: "last_clean_begin": 0, 2021-03-24T22:00:20.672 INFO:teuthology.orchestra.run.smithi125.stdout: "last_clean_end": 0, 2021-03-24T22:00:20.672 INFO:teuthology.orchestra.run.smithi125.stdout: "up_from": 26, 2021-03-24T22:00:20.673 INFO:teuthology.orchestra.run.smithi125.stdout: "up_thru": 434, 2021-03-24T22:00:20.673 INFO:teuthology.orchestra.run.smithi125.stdout: "down_at": 0, 2021-03-24T22:00:20.673 INFO:teuthology.orchestra.run.smithi125.stdout: "lost_at": 0, 2021-03-24T22:00:20.673 INFO:teuthology.orchestra.run.smithi125.stdout: "public_addrs": { 2021-03-24T22:00:20.673 INFO:teuthology.orchestra.run.smithi125.stdout: "addrvec": [ 2021-03-24T22:00:20.674 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.674 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2021-03-24T22:00:20.674 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "172.21.15.125:6813", 2021-03-24T22:00:20.674 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 2209636173 2021-03-24T22:00:20.675 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.675 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:20.675 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.675 INFO:teuthology.orchestra.run.smithi125.stdout: "cluster_addrs": { 2021-03-24T22:00:20.675 INFO:teuthology.orchestra.run.smithi125.stdout: "addrvec": [ 2021-03-24T22:00:20.676 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.676 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2021-03-24T22:00:20.676 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "172.21.15.125:6814", 2021-03-24T22:00:20.676 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 2209636173 2021-03-24T22:00:20.677 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.677 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:20.677 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.677 INFO:teuthology.orchestra.run.smithi125.stdout: "heartbeat_back_addrs": { 2021-03-24T22:00:20.677 INFO:teuthology.orchestra.run.smithi125.stdout: "addrvec": [ 2021-03-24T22:00:20.678 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.678 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2021-03-24T22:00:20.678 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "172.21.15.125:6816", 2021-03-24T22:00:20.678 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 2209636173 2021-03-24T22:00:20.678 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.679 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:20.679 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.679 INFO:teuthology.orchestra.run.smithi125.stdout: "heartbeat_front_addrs": { 2021-03-24T22:00:20.679 INFO:teuthology.orchestra.run.smithi125.stdout: "addrvec": [ 2021-03-24T22:00:20.680 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.680 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2021-03-24T22:00:20.680 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "172.21.15.125:6815", 2021-03-24T22:00:20.680 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 2209636173 2021-03-24T22:00:20.680 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.681 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:20.681 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.681 INFO:teuthology.orchestra.run.smithi125.stdout: "public_addr": "172.21.15.125:6813/2209636173", 2021-03-24T22:00:20.681 INFO:teuthology.orchestra.run.smithi125.stdout: "cluster_addr": "172.21.15.125:6814/2209636173", 2021-03-24T22:00:20.682 INFO:teuthology.orchestra.run.smithi125.stdout: "heartbeat_back_addr": "172.21.15.125:6816/2209636173", 2021-03-24T22:00:20.682 INFO:teuthology.orchestra.run.smithi125.stdout: "heartbeat_front_addr": "172.21.15.125:6815/2209636173", 2021-03-24T22:00:20.682 INFO:teuthology.orchestra.run.smithi125.stdout: "state": [ 2021-03-24T22:00:20.682 INFO:teuthology.orchestra.run.smithi125.stdout: "exists", 2021-03-24T22:00:20.682 INFO:teuthology.orchestra.run.smithi125.stdout: "up" 2021-03-24T22:00:20.683 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:20.683 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.683 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.683 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": 4, 2021-03-24T22:00:20.684 INFO:teuthology.orchestra.run.smithi125.stdout: "uuid": "f9bdd8d2-b292-49da-ad4f-99682122f543", 2021-03-24T22:00:20.684 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2021-03-24T22:00:20.684 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2021-03-24T22:00:20.684 INFO:teuthology.orchestra.run.smithi125.stdout: "weight": 1, 2021-03-24T22:00:20.684 INFO:teuthology.orchestra.run.smithi125.stdout: "primary_affinity": 1, 2021-03-24T22:00:20.685 INFO:teuthology.orchestra.run.smithi125.stdout: "last_clean_begin": 0, 2021-03-24T22:00:20.685 INFO:teuthology.orchestra.run.smithi125.stdout: "last_clean_end": 0, 2021-03-24T22:00:20.685 INFO:teuthology.orchestra.run.smithi125.stdout: "up_from": 32, 2021-03-24T22:00:20.685 INFO:teuthology.orchestra.run.smithi125.stdout: "up_thru": 449, 2021-03-24T22:00:20.685 INFO:teuthology.orchestra.run.smithi125.stdout: "down_at": 0, 2021-03-24T22:00:20.686 INFO:teuthology.orchestra.run.smithi125.stdout: "lost_at": 0, 2021-03-24T22:00:20.686 INFO:teuthology.orchestra.run.smithi125.stdout: "public_addrs": { 2021-03-24T22:00:20.687 INFO:teuthology.orchestra.run.smithi125.stdout: "addrvec": [ 2021-03-24T22:00:20.687 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.687 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2021-03-24T22:00:20.688 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "172.21.15.142:6800", 2021-03-24T22:00:20.688 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 1816503330 2021-03-24T22:00:20.689 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.689 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:20.690 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.690 INFO:teuthology.orchestra.run.smithi125.stdout: "cluster_addrs": { 2021-03-24T22:00:20.691 INFO:teuthology.orchestra.run.smithi125.stdout: "addrvec": [ 2021-03-24T22:00:20.691 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.691 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2021-03-24T22:00:20.692 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "172.21.15.142:6801", 2021-03-24T22:00:20.692 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 1816503330 2021-03-24T22:00:20.693 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.693 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:20.693 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.693 INFO:teuthology.orchestra.run.smithi125.stdout: "heartbeat_back_addrs": { 2021-03-24T22:00:20.694 INFO:teuthology.orchestra.run.smithi125.stdout: "addrvec": [ 2021-03-24T22:00:20.694 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.694 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2021-03-24T22:00:20.694 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "172.21.15.142:6803", 2021-03-24T22:00:20.695 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 1816503330 2021-03-24T22:00:20.695 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.695 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:20.695 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.696 INFO:teuthology.orchestra.run.smithi125.stdout: "heartbeat_front_addrs": { 2021-03-24T22:00:20.696 INFO:teuthology.orchestra.run.smithi125.stdout: "addrvec": [ 2021-03-24T22:00:20.696 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.697 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2021-03-24T22:00:20.697 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "172.21.15.142:6802", 2021-03-24T22:00:20.697 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 1816503330 2021-03-24T22:00:20.697 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.697 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:20.698 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.698 INFO:teuthology.orchestra.run.smithi125.stdout: "public_addr": "172.21.15.142:6800/1816503330", 2021-03-24T22:00:20.698 INFO:teuthology.orchestra.run.smithi125.stdout: "cluster_addr": "172.21.15.142:6801/1816503330", 2021-03-24T22:00:20.698 INFO:teuthology.orchestra.run.smithi125.stdout: "heartbeat_back_addr": "172.21.15.142:6803/1816503330", 2021-03-24T22:00:20.699 INFO:teuthology.orchestra.run.smithi125.stdout: "heartbeat_front_addr": "172.21.15.142:6802/1816503330", 2021-03-24T22:00:20.699 INFO:teuthology.orchestra.run.smithi125.stdout: "state": [ 2021-03-24T22:00:20.699 INFO:teuthology.orchestra.run.smithi125.stdout: "exists", 2021-03-24T22:00:20.699 INFO:teuthology.orchestra.run.smithi125.stdout: "up" 2021-03-24T22:00:20.700 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:20.700 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.700 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.700 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": 5, 2021-03-24T22:00:20.700 INFO:teuthology.orchestra.run.smithi125.stdout: "uuid": "ec0d43aa-6905-43f0-9e32-e19d4cf5129d", 2021-03-24T22:00:20.701 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2021-03-24T22:00:20.701 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 1, 2021-03-24T22:00:20.701 INFO:teuthology.orchestra.run.smithi125.stdout: "weight": 1, 2021-03-24T22:00:20.702 INFO:teuthology.orchestra.run.smithi125.stdout: "primary_affinity": 0.664520263671875, 2021-03-24T22:00:20.702 INFO:teuthology.orchestra.run.smithi125.stdout: "last_clean_begin": 39, 2021-03-24T22:00:20.703 INFO:teuthology.orchestra.run.smithi125.stdout: "last_clean_end": 339, 2021-03-24T22:00:20.703 INFO:teuthology.orchestra.run.smithi125.stdout: "up_from": 432, 2021-03-24T22:00:20.703 INFO:teuthology.orchestra.run.smithi125.stdout: "up_thru": 433, 2021-03-24T22:00:20.704 INFO:teuthology.orchestra.run.smithi125.stdout: "down_at": 340, 2021-03-24T22:00:20.704 INFO:teuthology.orchestra.run.smithi125.stdout: "lost_at": 0, 2021-03-24T22:00:20.704 INFO:teuthology.orchestra.run.smithi125.stdout: "public_addrs": { 2021-03-24T22:00:20.705 INFO:teuthology.orchestra.run.smithi125.stdout: "addrvec": [ 2021-03-24T22:00:20.705 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.705 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2021-03-24T22:00:20.705 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "172.21.15.142:6804", 2021-03-24T22:00:20.706 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 3992668882 2021-03-24T22:00:20.706 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.706 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:20.706 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.707 INFO:teuthology.orchestra.run.smithi125.stdout: "cluster_addrs": { 2021-03-24T22:00:20.707 INFO:teuthology.orchestra.run.smithi125.stdout: "addrvec": [ 2021-03-24T22:00:20.707 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.707 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2021-03-24T22:00:20.708 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "172.21.15.142:6805", 2021-03-24T22:00:20.708 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 3992668882 2021-03-24T22:00:20.708 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.708 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:20.709 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.709 INFO:teuthology.orchestra.run.smithi125.stdout: "heartbeat_back_addrs": { 2021-03-24T22:00:20.709 INFO:teuthology.orchestra.run.smithi125.stdout: "addrvec": [ 2021-03-24T22:00:20.710 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.710 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2021-03-24T22:00:20.711 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "172.21.15.142:6807", 2021-03-24T22:00:20.711 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 3992668882 2021-03-24T22:00:20.711 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.711 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:20.712 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.712 INFO:teuthology.orchestra.run.smithi125.stdout: "heartbeat_front_addrs": { 2021-03-24T22:00:20.713 INFO:teuthology.orchestra.run.smithi125.stdout: "addrvec": [ 2021-03-24T22:00:20.713 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.713 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2021-03-24T22:00:20.714 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "172.21.15.142:6806", 2021-03-24T22:00:20.714 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 3992668882 2021-03-24T22:00:20.714 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.714 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:20.715 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.715 INFO:teuthology.orchestra.run.smithi125.stdout: "public_addr": "172.21.15.142:6804/3992668882", 2021-03-24T22:00:20.715 INFO:teuthology.orchestra.run.smithi125.stdout: "cluster_addr": "172.21.15.142:6805/3992668882", 2021-03-24T22:00:20.715 INFO:teuthology.orchestra.run.smithi125.stdout: "heartbeat_back_addr": "172.21.15.142:6807/3992668882", 2021-03-24T22:00:20.715 INFO:teuthology.orchestra.run.smithi125.stdout: "heartbeat_front_addr": "172.21.15.142:6806/3992668882", 2021-03-24T22:00:20.716 INFO:teuthology.orchestra.run.smithi125.stdout: "state": [ 2021-03-24T22:00:20.716 INFO:teuthology.orchestra.run.smithi125.stdout: "exists", 2021-03-24T22:00:20.716 INFO:teuthology.orchestra.run.smithi125.stdout: "up" 2021-03-24T22:00:20.716 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:20.717 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.717 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.717 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": 6, 2021-03-24T22:00:20.717 INFO:teuthology.orchestra.run.smithi125.stdout: "uuid": "a26a487e-ed26-4d92-98ea-896739302921", 2021-03-24T22:00:20.718 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2021-03-24T22:00:20.718 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 0, 2021-03-24T22:00:20.718 INFO:teuthology.orchestra.run.smithi125.stdout: "weight": 0, 2021-03-24T22:00:20.718 INFO:teuthology.orchestra.run.smithi125.stdout: "primary_affinity": 1, 2021-03-24T22:00:20.719 INFO:teuthology.orchestra.run.smithi125.stdout: "last_clean_begin": 0, 2021-03-24T22:00:20.719 INFO:teuthology.orchestra.run.smithi125.stdout: "last_clean_end": 0, 2021-03-24T22:00:20.719 INFO:teuthology.orchestra.run.smithi125.stdout: "up_from": 47, 2021-03-24T22:00:20.719 INFO:teuthology.orchestra.run.smithi125.stdout: "up_thru": 99, 2021-03-24T22:00:20.720 INFO:teuthology.orchestra.run.smithi125.stdout: "down_at": 0, 2021-03-24T22:00:20.720 INFO:teuthology.orchestra.run.smithi125.stdout: "lost_at": 0, 2021-03-24T22:00:20.720 INFO:teuthology.orchestra.run.smithi125.stdout: "public_addrs": { 2021-03-24T22:00:20.720 INFO:teuthology.orchestra.run.smithi125.stdout: "addrvec": [ 2021-03-24T22:00:20.721 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.721 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2021-03-24T22:00:20.721 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "172.21.15.142:6808", 2021-03-24T22:00:20.721 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 4048483913 2021-03-24T22:00:20.722 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.722 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:20.722 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.722 INFO:teuthology.orchestra.run.smithi125.stdout: "cluster_addrs": { 2021-03-24T22:00:20.723 INFO:teuthology.orchestra.run.smithi125.stdout: "addrvec": [ 2021-03-24T22:00:20.723 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.723 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2021-03-24T22:00:20.723 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "172.21.15.142:6809", 2021-03-24T22:00:20.723 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 4048483913 2021-03-24T22:00:20.724 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.724 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:20.724 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.724 INFO:teuthology.orchestra.run.smithi125.stdout: "heartbeat_back_addrs": { 2021-03-24T22:00:20.724 INFO:teuthology.orchestra.run.smithi125.stdout: "addrvec": [ 2021-03-24T22:00:20.725 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.725 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2021-03-24T22:00:20.725 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "172.21.15.142:6811", 2021-03-24T22:00:20.725 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 4048483913 2021-03-24T22:00:20.726 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.726 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:20.726 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.726 INFO:teuthology.orchestra.run.smithi125.stdout: "heartbeat_front_addrs": { 2021-03-24T22:00:20.726 INFO:teuthology.orchestra.run.smithi125.stdout: "addrvec": [ 2021-03-24T22:00:20.727 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.727 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2021-03-24T22:00:20.727 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "172.21.15.142:6810", 2021-03-24T22:00:20.727 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 4048483913 2021-03-24T22:00:20.728 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.728 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:20.728 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.728 INFO:teuthology.orchestra.run.smithi125.stdout: "public_addr": "172.21.15.142:6808/4048483913", 2021-03-24T22:00:20.728 INFO:teuthology.orchestra.run.smithi125.stdout: "cluster_addr": "172.21.15.142:6809/4048483913", 2021-03-24T22:00:20.729 INFO:teuthology.orchestra.run.smithi125.stdout: "heartbeat_back_addr": "172.21.15.142:6811/4048483913", 2021-03-24T22:00:20.729 INFO:teuthology.orchestra.run.smithi125.stdout: "heartbeat_front_addr": "172.21.15.142:6810/4048483913", 2021-03-24T22:00:20.729 INFO:teuthology.orchestra.run.smithi125.stdout: "state": [ 2021-03-24T22:00:20.729 INFO:teuthology.orchestra.run.smithi125.stdout: "exists", 2021-03-24T22:00:20.729 INFO:teuthology.orchestra.run.smithi125.stdout: "up" 2021-03-24T22:00:20.730 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:20.730 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.730 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.730 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": 7, 2021-03-24T22:00:20.731 INFO:teuthology.orchestra.run.smithi125.stdout: "uuid": "682ba5ce-0172-48e3-82f8-57a3f98c911b", 2021-03-24T22:00:20.731 INFO:teuthology.orchestra.run.smithi125.stdout: "up": 1, 2021-03-24T22:00:20.731 INFO:teuthology.orchestra.run.smithi125.stdout: "in": 0, 2021-03-24T22:00:20.731 INFO:teuthology.orchestra.run.smithi125.stdout: "weight": 0, 2021-03-24T22:00:20.731 INFO:teuthology.orchestra.run.smithi125.stdout: "primary_affinity": 0.794097900390625, 2021-03-24T22:00:20.732 INFO:teuthology.orchestra.run.smithi125.stdout: "last_clean_begin": 0, 2021-03-24T22:00:20.732 INFO:teuthology.orchestra.run.smithi125.stdout: "last_clean_end": 0, 2021-03-24T22:00:20.732 INFO:teuthology.orchestra.run.smithi125.stdout: "up_from": 55, 2021-03-24T22:00:20.732 INFO:teuthology.orchestra.run.smithi125.stdout: "up_thru": 416, 2021-03-24T22:00:20.733 INFO:teuthology.orchestra.run.smithi125.stdout: "down_at": 0, 2021-03-24T22:00:20.733 INFO:teuthology.orchestra.run.smithi125.stdout: "lost_at": 0, 2021-03-24T22:00:20.733 INFO:teuthology.orchestra.run.smithi125.stdout: "public_addrs": { 2021-03-24T22:00:20.733 INFO:teuthology.orchestra.run.smithi125.stdout: "addrvec": [ 2021-03-24T22:00:20.734 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.734 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2021-03-24T22:00:20.734 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "172.21.15.142:6812", 2021-03-24T22:00:20.734 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 3678129570 2021-03-24T22:00:20.734 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.735 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:20.735 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.735 INFO:teuthology.orchestra.run.smithi125.stdout: "cluster_addrs": { 2021-03-24T22:00:20.735 INFO:teuthology.orchestra.run.smithi125.stdout: "addrvec": [ 2021-03-24T22:00:20.735 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.736 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2021-03-24T22:00:20.736 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "172.21.15.142:6813", 2021-03-24T22:00:20.736 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 3678129570 2021-03-24T22:00:20.736 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.737 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:20.737 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.737 INFO:teuthology.orchestra.run.smithi125.stdout: "heartbeat_back_addrs": { 2021-03-24T22:00:20.737 INFO:teuthology.orchestra.run.smithi125.stdout: "addrvec": [ 2021-03-24T22:00:20.737 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.738 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2021-03-24T22:00:20.738 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "172.21.15.142:6815", 2021-03-24T22:00:20.738 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 3678129570 2021-03-24T22:00:20.738 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.739 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:20.739 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.739 INFO:teuthology.orchestra.run.smithi125.stdout: "heartbeat_front_addrs": { 2021-03-24T22:00:20.739 INFO:teuthology.orchestra.run.smithi125.stdout: "addrvec": [ 2021-03-24T22:00:20.740 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.740 INFO:teuthology.orchestra.run.smithi125.stdout: "type": "v1", 2021-03-24T22:00:20.740 INFO:teuthology.orchestra.run.smithi125.stdout: "addr": "172.21.15.142:6814", 2021-03-24T22:00:20.740 INFO:teuthology.orchestra.run.smithi125.stdout: "nonce": 3678129570 2021-03-24T22:00:20.740 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.741 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:20.741 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.741 INFO:teuthology.orchestra.run.smithi125.stdout: "public_addr": "172.21.15.142:6812/3678129570", 2021-03-24T22:00:20.741 INFO:teuthology.orchestra.run.smithi125.stdout: "cluster_addr": "172.21.15.142:6813/3678129570", 2021-03-24T22:00:20.742 INFO:teuthology.orchestra.run.smithi125.stdout: "heartbeat_back_addr": "172.21.15.142:6815/3678129570", 2021-03-24T22:00:20.742 INFO:teuthology.orchestra.run.smithi125.stdout: "heartbeat_front_addr": "172.21.15.142:6814/3678129570", 2021-03-24T22:00:20.742 INFO:teuthology.orchestra.run.smithi125.stdout: "state": [ 2021-03-24T22:00:20.742 INFO:teuthology.orchestra.run.smithi125.stdout: "exists", 2021-03-24T22:00:20.742 INFO:teuthology.orchestra.run.smithi125.stdout: "up" 2021-03-24T22:00:20.743 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:20.743 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.743 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2021-03-24T22:00:20.743 INFO:teuthology.orchestra.run.smithi125.stdout: "osd_xinfo": [ 2021-03-24T22:00:20.744 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.744 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": 0, 2021-03-24T22:00:20.744 INFO:teuthology.orchestra.run.smithi125.stdout: "down_stamp": "0.000000", 2021-03-24T22:00:20.744 INFO:teuthology.orchestra.run.smithi125.stdout: "laggy_probability": 0, 2021-03-24T22:00:20.744 INFO:teuthology.orchestra.run.smithi125.stdout: "laggy_interval": 0, 2021-03-24T22:00:20.745 INFO:teuthology.orchestra.run.smithi125.stdout: "features": 4540138297136906239, 2021-03-24T22:00:20.745 INFO:teuthology.orchestra.run.smithi125.stdout: "old_weight": 0, 2021-03-24T22:00:20.745 INFO:teuthology.orchestra.run.smithi125.stdout: "last_purged_snaps_scrub": "2021-03-24T22:00:11.903425+0000", 2021-03-24T22:00:20.745 INFO:teuthology.orchestra.run.smithi125.stdout: "dead_epoch": 0 2021-03-24T22:00:20.745 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.746 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.746 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": 1, 2021-03-24T22:00:20.746 INFO:teuthology.orchestra.run.smithi125.stdout: "down_stamp": "0.000000", 2021-03-24T22:00:20.746 INFO:teuthology.orchestra.run.smithi125.stdout: "laggy_probability": 0, 2021-03-24T22:00:20.747 INFO:teuthology.orchestra.run.smithi125.stdout: "laggy_interval": 0, 2021-03-24T22:00:20.747 INFO:teuthology.orchestra.run.smithi125.stdout: "features": 4540138297136906239, 2021-03-24T22:00:20.747 INFO:teuthology.orchestra.run.smithi125.stdout: "old_weight": 65536, 2021-03-24T22:00:20.747 INFO:teuthology.orchestra.run.smithi125.stdout: "last_purged_snaps_scrub": "2021-03-24T21:59:31.832113+0000", 2021-03-24T22:00:20.747 INFO:teuthology.orchestra.run.smithi125.stdout: "dead_epoch": 0 2021-03-24T22:00:20.748 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.748 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.748 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": 2, 2021-03-24T22:00:20.748 INFO:teuthology.orchestra.run.smithi125.stdout: "down_stamp": "0.000000", 2021-03-24T22:00:20.748 INFO:teuthology.orchestra.run.smithi125.stdout: "laggy_probability": 0, 2021-03-24T22:00:20.749 INFO:teuthology.orchestra.run.smithi125.stdout: "laggy_interval": 0, 2021-03-24T22:00:20.749 INFO:teuthology.orchestra.run.smithi125.stdout: "features": 4540138297136906239, 2021-03-24T22:00:20.749 INFO:teuthology.orchestra.run.smithi125.stdout: "old_weight": 0, 2021-03-24T22:00:20.749 INFO:teuthology.orchestra.run.smithi125.stdout: "last_purged_snaps_scrub": "2021-03-24T21:59:55.500612+0000", 2021-03-24T22:00:20.750 INFO:teuthology.orchestra.run.smithi125.stdout: "dead_epoch": 0 2021-03-24T22:00:20.750 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.750 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.750 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": 3, 2021-03-24T22:00:20.750 INFO:teuthology.orchestra.run.smithi125.stdout: "down_stamp": "0.000000", 2021-03-24T22:00:20.751 INFO:teuthology.orchestra.run.smithi125.stdout: "laggy_probability": 0, 2021-03-24T22:00:20.751 INFO:teuthology.orchestra.run.smithi125.stdout: "laggy_interval": 0, 2021-03-24T22:00:20.751 INFO:teuthology.orchestra.run.smithi125.stdout: "features": 4540138297136906239, 2021-03-24T22:00:20.751 INFO:teuthology.orchestra.run.smithi125.stdout: "old_weight": 0, 2021-03-24T22:00:20.752 INFO:teuthology.orchestra.run.smithi125.stdout: "last_purged_snaps_scrub": "2021-03-24T22:00:14.787434+0000", 2021-03-24T22:00:20.752 INFO:teuthology.orchestra.run.smithi125.stdout: "dead_epoch": 0 2021-03-24T22:00:20.752 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.752 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.752 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": 4, 2021-03-24T22:00:20.753 INFO:teuthology.orchestra.run.smithi125.stdout: "down_stamp": "0.000000", 2021-03-24T22:00:20.753 INFO:teuthology.orchestra.run.smithi125.stdout: "laggy_probability": 0, 2021-03-24T22:00:20.753 INFO:teuthology.orchestra.run.smithi125.stdout: "laggy_interval": 0, 2021-03-24T22:00:20.753 INFO:teuthology.orchestra.run.smithi125.stdout: "features": 4540138297136906239, 2021-03-24T22:00:20.754 INFO:teuthology.orchestra.run.smithi125.stdout: "old_weight": 0, 2021-03-24T22:00:20.754 INFO:teuthology.orchestra.run.smithi125.stdout: "last_purged_snaps_scrub": "2021-03-24T21:59:34.555580+0000", 2021-03-24T22:00:20.754 INFO:teuthology.orchestra.run.smithi125.stdout: "dead_epoch": 0 2021-03-24T22:00:20.754 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.754 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.755 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": 5, 2021-03-24T22:00:20.755 INFO:teuthology.orchestra.run.smithi125.stdout: "down_stamp": "0.000000", 2021-03-24T22:00:20.755 INFO:teuthology.orchestra.run.smithi125.stdout: "laggy_probability": 0, 2021-03-24T22:00:20.755 INFO:teuthology.orchestra.run.smithi125.stdout: "laggy_interval": 0, 2021-03-24T22:00:20.755 INFO:teuthology.orchestra.run.smithi125.stdout: "features": 4540138297136906239, 2021-03-24T22:00:20.756 INFO:teuthology.orchestra.run.smithi125.stdout: "old_weight": 0, 2021-03-24T22:00:20.756 INFO:teuthology.orchestra.run.smithi125.stdout: "last_purged_snaps_scrub": "2021-03-24T22:00:01.559928+0000", 2021-03-24T22:00:20.756 INFO:teuthology.orchestra.run.smithi125.stdout: "dead_epoch": 340 2021-03-24T22:00:20.756 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.757 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.757 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": 6, 2021-03-24T22:00:20.757 INFO:teuthology.orchestra.run.smithi125.stdout: "down_stamp": "0.000000", 2021-03-24T22:00:20.757 INFO:teuthology.orchestra.run.smithi125.stdout: "laggy_probability": 0, 2021-03-24T22:00:20.758 INFO:teuthology.orchestra.run.smithi125.stdout: "laggy_interval": 0, 2021-03-24T22:00:20.758 INFO:teuthology.orchestra.run.smithi125.stdout: "features": 4540138297136906239, 2021-03-24T22:00:20.758 INFO:teuthology.orchestra.run.smithi125.stdout: "old_weight": 65536, 2021-03-24T22:00:20.758 INFO:teuthology.orchestra.run.smithi125.stdout: "last_purged_snaps_scrub": "2021-03-24T22:00:20.230832+0000", 2021-03-24T22:00:20.759 INFO:teuthology.orchestra.run.smithi125.stdout: "dead_epoch": 0 2021-03-24T22:00:20.759 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.759 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.759 INFO:teuthology.orchestra.run.smithi125.stdout: "osd": 7, 2021-03-24T22:00:20.760 INFO:teuthology.orchestra.run.smithi125.stdout: "down_stamp": "0.000000", 2021-03-24T22:00:20.760 INFO:teuthology.orchestra.run.smithi125.stdout: "laggy_probability": 0, 2021-03-24T22:00:20.760 INFO:teuthology.orchestra.run.smithi125.stdout: "laggy_interval": 0, 2021-03-24T22:00:20.760 INFO:teuthology.orchestra.run.smithi125.stdout: "features": 4540138297136906239, 2021-03-24T22:00:20.761 INFO:teuthology.orchestra.run.smithi125.stdout: "old_weight": 58984, 2021-03-24T22:00:20.761 INFO:teuthology.orchestra.run.smithi125.stdout: "last_purged_snaps_scrub": "2021-03-24T21:59:39.345289+0000", 2021-03-24T22:00:20.761 INFO:teuthology.orchestra.run.smithi125.stdout: "dead_epoch": 0 2021-03-24T22:00:20.761 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.761 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2021-03-24T22:00:20.762 INFO:teuthology.orchestra.run.smithi125.stdout: "pg_upmap": [], 2021-03-24T22:00:20.762 INFO:teuthology.orchestra.run.smithi125.stdout: "pg_upmap_items": [ 2021-03-24T22:00:20.762 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.762 INFO:teuthology.orchestra.run.smithi125.stdout: "pgid": "3.1d", 2021-03-24T22:00:20.762 INFO:teuthology.orchestra.run.smithi125.stdout: "mappings": [ 2021-03-24T22:00:20.763 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.763 INFO:teuthology.orchestra.run.smithi125.stdout: "from": 4, 2021-03-24T22:00:20.763 INFO:teuthology.orchestra.run.smithi125.stdout: "to": 2 2021-03-24T22:00:20.763 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.764 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:20.764 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.764 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.764 INFO:teuthology.orchestra.run.smithi125.stdout: "pgid": "6.4", 2021-03-24T22:00:20.764 INFO:teuthology.orchestra.run.smithi125.stdout: "mappings": [ 2021-03-24T22:00:20.765 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.765 INFO:teuthology.orchestra.run.smithi125.stdout: "from": 5, 2021-03-24T22:00:20.765 INFO:teuthology.orchestra.run.smithi125.stdout: "to": 0 2021-03-24T22:00:20.766 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.766 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:20.766 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.766 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.767 INFO:teuthology.orchestra.run.smithi125.stdout: "pgid": "7.10", 2021-03-24T22:00:20.767 INFO:teuthology.orchestra.run.smithi125.stdout: "mappings": [ 2021-03-24T22:00:20.767 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.768 INFO:teuthology.orchestra.run.smithi125.stdout: "from": 2, 2021-03-24T22:00:20.768 INFO:teuthology.orchestra.run.smithi125.stdout: "to": 5 2021-03-24T22:00:20.768 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.769 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:20.769 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.770 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2021-03-24T22:00:20.770 INFO:teuthology.orchestra.run.smithi125.stdout: "pg_temp": [], 2021-03-24T22:00:20.770 INFO:teuthology.orchestra.run.smithi125.stdout: "primary_temp": [], 2021-03-24T22:00:20.771 INFO:teuthology.orchestra.run.smithi125.stdout: "blocklist": { 2021-03-24T22:00:20.771 INFO:teuthology.orchestra.run.smithi125.stdout: "172.21.15.125:0/3077905055": "2021-03-25T21:47:04.502055+0000", 2021-03-24T22:00:20.771 INFO:teuthology.orchestra.run.smithi125.stdout: "172.21.15.125:0/2210544773": "2021-03-25T21:47:04.502055+0000", 2021-03-24T22:00:20.772 INFO:teuthology.orchestra.run.smithi125.stdout: "172.21.15.125:6800/3176807740": "2021-03-25T21:47:04.502055+0000", 2021-03-24T22:00:20.772 INFO:teuthology.orchestra.run.smithi125.stdout: "172.21.15.125:6800/3531208850": "2021-03-25T21:46:43.197724+0000", 2021-03-24T22:00:20.772 INFO:teuthology.orchestra.run.smithi125.stdout: "172.21.15.125:0/3337261959": "2021-03-25T21:46:43.197724+0000", 2021-03-24T22:00:20.773 INFO:teuthology.orchestra.run.smithi125.stdout: "172.21.15.125:0/1919150488": "2021-03-25T21:46:43.197724+0000" 2021-03-24T22:00:20.773 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.773 INFO:teuthology.orchestra.run.smithi125.stdout: "erasure_code_profiles": { 2021-03-24T22:00:20.774 INFO:teuthology.orchestra.run.smithi125.stdout: "default": { 2021-03-24T22:00:20.774 INFO:teuthology.orchestra.run.smithi125.stdout: "crush-failure-domain": "osd", 2021-03-24T22:00:20.774 INFO:teuthology.orchestra.run.smithi125.stdout: "k": "2", 2021-03-24T22:00:20.775 INFO:teuthology.orchestra.run.smithi125.stdout: "m": "1", 2021-03-24T22:00:20.775 INFO:teuthology.orchestra.run.smithi125.stdout: "plugin": "jerasure", 2021-03-24T22:00:20.776 INFO:teuthology.orchestra.run.smithi125.stdout: "ruleset-failure-domain": "osd", 2021-03-24T22:00:20.776 INFO:teuthology.orchestra.run.smithi125.stdout: "technique": "reed_sol_van" 2021-03-24T22:00:20.776 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.777 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.777 INFO:teuthology.orchestra.run.smithi125.stdout: "removed_snaps_queue": [ 2021-03-24T22:00:20.777 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.778 INFO:teuthology.orchestra.run.smithi125.stdout: "pool": 7, 2021-03-24T22:00:20.778 INFO:teuthology.orchestra.run.smithi125.stdout: "snaps": [ 2021-03-24T22:00:20.778 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.779 INFO:teuthology.orchestra.run.smithi125.stdout: "begin": 231, 2021-03-24T22:00:20.779 INFO:teuthology.orchestra.run.smithi125.stdout: "length": 1 2021-03-24T22:00:20.779 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.779 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.780 INFO:teuthology.orchestra.run.smithi125.stdout: "begin": 248, 2021-03-24T22:00:20.780 INFO:teuthology.orchestra.run.smithi125.stdout: "length": 1 2021-03-24T22:00:20.780 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.780 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.781 INFO:teuthology.orchestra.run.smithi125.stdout: "begin": 268, 2021-03-24T22:00:20.781 INFO:teuthology.orchestra.run.smithi125.stdout: "length": 1 2021-03-24T22:00:20.781 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.781 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.781 INFO:teuthology.orchestra.run.smithi125.stdout: "begin": 294, 2021-03-24T22:00:20.782 INFO:teuthology.orchestra.run.smithi125.stdout: "length": 1 2021-03-24T22:00:20.782 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.782 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.782 INFO:teuthology.orchestra.run.smithi125.stdout: "begin": 296, 2021-03-24T22:00:20.783 INFO:teuthology.orchestra.run.smithi125.stdout: "length": 1 2021-03-24T22:00:20.783 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.783 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:20.783 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.783 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2021-03-24T22:00:20.784 INFO:teuthology.orchestra.run.smithi125.stdout: "new_removed_snaps": [ 2021-03-24T22:00:20.784 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.784 INFO:teuthology.orchestra.run.smithi125.stdout: "pool": 7, 2021-03-24T22:00:20.784 INFO:teuthology.orchestra.run.smithi125.stdout: "snaps": [ 2021-03-24T22:00:20.784 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.785 INFO:teuthology.orchestra.run.smithi125.stdout: "begin": 296, 2021-03-24T22:00:20.785 INFO:teuthology.orchestra.run.smithi125.stdout: "length": 1 2021-03-24T22:00:20.785 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.785 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:20.785 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.786 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2021-03-24T22:00:20.786 INFO:teuthology.orchestra.run.smithi125.stdout: "new_purged_snaps": [ 2021-03-24T22:00:20.786 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.787 INFO:teuthology.orchestra.run.smithi125.stdout: "pool": 7, 2021-03-24T22:00:20.787 INFO:teuthology.orchestra.run.smithi125.stdout: "snaps": [ 2021-03-24T22:00:20.787 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.787 INFO:teuthology.orchestra.run.smithi125.stdout: "begin": 261, 2021-03-24T22:00:20.787 INFO:teuthology.orchestra.run.smithi125.stdout: "length": 1 2021-03-24T22:00:20.788 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.788 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.788 INFO:teuthology.orchestra.run.smithi125.stdout: "begin": 272, 2021-03-24T22:00:20.788 INFO:teuthology.orchestra.run.smithi125.stdout: "length": 1 2021-03-24T22:00:20.788 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.789 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.789 INFO:teuthology.orchestra.run.smithi125.stdout: "begin": 281, 2021-03-24T22:00:20.789 INFO:teuthology.orchestra.run.smithi125.stdout: "length": 2 2021-03-24T22:00:20.789 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.789 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.790 INFO:teuthology.orchestra.run.smithi125.stdout: "begin": 301, 2021-03-24T22:00:20.790 INFO:teuthology.orchestra.run.smithi125.stdout: "length": 1 2021-03-24T22:00:20.790 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.791 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.791 INFO:teuthology.orchestra.run.smithi125.stdout: "begin": 317, 2021-03-24T22:00:20.791 INFO:teuthology.orchestra.run.smithi125.stdout: "length": 1 2021-03-24T22:00:20.791 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:20.791 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:20.792 INFO:teuthology.orchestra.run.smithi125.stdout: "begin": 323, 2021-03-24T22:00:20.792 INFO:teuthology.orchestra.run.smithi125.stdout: "length": 1 2021-03-24T22:00:20.792 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.792 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:20.793 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.793 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2021-03-24T22:00:20.793 INFO:teuthology.orchestra.run.smithi125.stdout: "crush_node_flags": {}, 2021-03-24T22:00:20.793 INFO:teuthology.orchestra.run.smithi125.stdout: "device_class_flags": {}, 2021-03-24T22:00:20.793 INFO:teuthology.orchestra.run.smithi125.stdout: "stretch_mode": { 2021-03-24T22:00:20.794 INFO:teuthology.orchestra.run.smithi125.stdout: "stretch_mode_enabled": false, 2021-03-24T22:00:20.794 INFO:teuthology.orchestra.run.smithi125.stdout: "stretch_bucket_count": 0, 2021-03-24T22:00:20.794 INFO:teuthology.orchestra.run.smithi125.stdout: "degraded_stretch_mode": 0, 2021-03-24T22:00:20.794 INFO:teuthology.orchestra.run.smithi125.stdout: "recovering_stretch_mode": 0, 2021-03-24T22:00:20.794 INFO:teuthology.orchestra.run.smithi125.stdout: "stretch_mode_bucket": 0 2021-03-24T22:00:20.795 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:20.795 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-24T22:00:20.795 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:20.797 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:20 smithi142 conmon[20125]: cluster 2021-03-24T22:00:18.322171+0000 osd.3 (osd. 2021-03-24T22:00:20.797 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:20 smithi142 conmon[20125]: 3) 98 : cluster [DBG] 6.b scrub ok 2021-03-24T22:00:20.797 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:20 smithi142 conmon[20125]: cluster 2021-03-24T22:00:18.855770+0000 mgr.y (mgr.14144) 495 : cluster [DBG] pgmap v835: 175 pgs: 2 peering, 1 active+clean+wait, 172 active+clean; 443 MiB data, 1.1 GiB used, 446 GiB / 447 GiB avail; 255 B/s rd, 415 KiB/s wr, 1 op/s 2021-03-24T22:00:20.797 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:20 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:00:20.798 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:20 smithi142 conmon[20125]: -24T22:00:19.248831+0000 mon.a (mon.0) 1356 : cluster [DBG] osdmap e449: 8 total, 8 up, 5 in 2021-03-24T22:00:20.798 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:20 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:00:20.798 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:20 smithi142 conmon[20125]: T22:00:20.238274+0000 mon.a (mon 2021-03-24T22:00:20.798 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:20 smithi142 conmon[20125]: .0) 1357 : cluster [DBG] osdmap e450: 8 total, 8 up, 5 in 2021-03-24T22:00:20.799 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2021-03-24T22:00:20.799 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2021-03-24T22:00:20.799 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2021-03-24T22:00:20.799 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2021-03-24T22:00:20.800 INFO:teuthology.orchestra.run.smithi142.stdout:} 2021-03-24T22:00:20.860 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:20.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:20.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:20.966 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:21.026 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:21.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:21.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:00:21.132 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:21.188 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:21.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:21.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:00:21.274 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:21 smithi142 conmon[20125]: cluster 2021-03-24T22:00:21.275 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:21 smithi142 conmon[20125]: 2021-03-24T22 2021-03-24T22:00:21.275 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:21 smithi142 conmon[20125]: :00:19.934765+0000 osd.4 (osd.4) 105 : cluster [DBG] 5.c scrub ok 2021-03-24T22:00:21.276 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:21 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:00:21.276 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:21 smithi142 conmon[20125]: T22:00:20.230637+0000 osd.6 (osd 2021-03-24T22:00:21.277 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:21 smithi142 conmon[20125]: .6) 29 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:00:21.277 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:21 smithi142 conmon[20125]: cluster 2021-03-24T22:00:20.230826+0000 2021-03-24T22:00:21.277 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:21 smithi142 conmon[20125]: osd.6 (osd.6) 30 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:00:21.277 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:21 smithi142 conmon[20125]: audit 2021-03-24T22:00:20.478541+0000 2021-03-24T22:00:21.278 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:21 smithi142 conmon[20125]: mon.b (mon.1) 37 : audit [DBG] from='client.? v1:172.21.15.125:0/3873398293' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-03-24T22:00:21.278 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:21 smithi142 conmon[20125]: audit 2021-03-24T22:00:20.857612 2021-03-24T22:00:21.278 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:21 smithi142 conmon[20125]: +0000 mon.a (mon.0) 1358 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "27"}]: dispatch 2021-03-24T22:00:21.282 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[19936]: cluster 2021-03-24T22:00:19 2021-03-24T22:00:21.282 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[19936]: .934765+0000 osd.4 (osd.4) 105 : cluster [DBG] 5.c scrub ok 2021-03-24T22:00:21.283 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[19936]: cluster 2021-03-24T22 2021-03-24T22:00:21.283 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[19936]: :00:20.230637+0000 osd.6 (osd.6) 2021-03-24T22:00:21.283 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[19936]: 29 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:00:21.283 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:00:21.284 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[19936]: 03-24T22:00:20.230826+0000 osd.6 (osd. 2021-03-24T22:00:21.284 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[19936]: 6) 30 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:00:21.284 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[19936]: audit 2021- 2021-03-24T22:00:21.285 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[19936]: 03-24T22:00:20.478541+0000 2021-03-24T22:00:21.285 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[19936]: mon.b (mon.1) 37 : audit [DBG] from='client.? v1:172.21.15.125:0/3873398293' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-03-24T22:00:21.285 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[19936]: 2021-03-24T22:00:21.285 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[19936]: audit 2021-03-24T22:00:21.286 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[19936]: 2021-03-24T22:00:20 2021-03-24T22:00:21.286 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[19936]: .857612+0000 mon.a (mon. 2021-03-24T22:00:21.287 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[19936]: 0) 1358 : audit [INF] 2021-03-24T22:00:21.287 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "27"}]: dispatch 2021-03-24T22:00:21.288 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: cluster 2021-03-24T22:00:21.288 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: 2021-03-24T22:00:21.289 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: 2021 2021-03-24T22:00:21.290 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: - 2021-03-24T22:00:21.291 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: 03 2021-03-24T22:00:21.291 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: - 2021-03-24T22:00:21.291 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: 24 2021-03-24T22:00:21.291 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: T 2021-03-24T22:00:21.292 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: 22 2021-03-24T22:00:21.292 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: : 2021-03-24T22:00:21.292 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: 00 2021-03-24T22:00:21.292 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: : 2021-03-24T22:00:21.293 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: 19 2021-03-24T22:00:21.293 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: . 2021-03-24T22:00:21.293 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: 934765 2021-03-24T22:00:21.294 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: +0000 2021-03-24T22:00:21.294 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: 2021-03-24T22:00:21.295 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: osd.4 2021-03-24T22:00:21.295 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: ( 2021-03-24T22:00:21.295 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: osd 2021-03-24T22:00:21.296 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: . 2021-03-24T22:00:21.296 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: 4 2021-03-24T22:00:21.297 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: ) 2021-03-24T22:00:21.297 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: 105 2021-03-24T22:00:21.297 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: : 2021-03-24T22:00:21.298 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: cluster 2021-03-24T22:00:21.298 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: 2021-03-24T22:00:21.299 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: [DBG] 2021-03-24T22:00:21.299 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: 2021-03-24T22:00:21.300 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: 5.c scrub ok 2021-03-24T22:00:21.300 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: 2021-03-24T22:00:21.301 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: cluster 2021-03-24T22:00:21.302 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: 2021-03-24T22:00:21.302 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: 2021 2021-03-24T22:00:21.303 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: - 2021-03-24T22:00:21.303 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: 03 2021-03-24T22:00:21.303 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: - 2021-03-24T22:00:21.304 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: 24 2021-03-24T22:00:21.304 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: T 2021-03-24T22:00:21.304 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: 22 2021-03-24T22:00:21.304 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: : 2021-03-24T22:00:21.304 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: 00 2021-03-24T22:00:21.305 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: : 2021-03-24T22:00:21.305 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: 20 2021-03-24T22:00:21.306 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: . 2021-03-24T22:00:21.306 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: 230637 2021-03-24T22:00:21.306 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: +0000 2021-03-24T22:00:21.307 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: 2021-03-24T22:00:21.307 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: osd.6 2021-03-24T22:00:21.307 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: ( 2021-03-24T22:00:21.308 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: osd 2021-03-24T22:00:21.308 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: . 2021-03-24T22:00:21.308 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: 6 2021-03-24T22:00:21.309 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: ) 2021-03-24T22:00:21.309 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: 29 2021-03-24T22:00:21.310 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: : 2021-03-24T22:00:21.310 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: cluster 2021-03-24T22:00:21.311 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: 2021-03-24T22:00:21.311 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: [DBG] 2021-03-24T22:00:21.311 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: 2021-03-24T22:00:21.312 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: purged_snaps scrub starts 2021-03-24T22:00:21.312 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: 2021-03-24T22:00:21.313 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: cluster 2021-03-24T22:00:21.313 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: 2021-03-24T22:00:21.314 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: 2021 2021-03-24T22:00:21.314 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: - 2021-03-24T22:00:21.315 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: 03 2021-03-24T22:00:21.315 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: - 2021-03-24T22:00:21.315 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: 24 2021-03-24T22:00:21.316 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: T 2021-03-24T22:00:21.317 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: 22 2021-03-24T22:00:21.317 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: : 2021-03-24T22:00:21.317 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: 00 2021-03-24T22:00:21.318 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: : 2021-03-24T22:00:21.318 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: 20 2021-03-24T22:00:21.319 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: . 2021-03-24T22:00:21.320 INFO:tasks.rados.rados.0.smithi125.stdout:1683: done (1 left) 2021-03-24T22:00:21.321 INFO:tasks.rados.rados.0.smithi125.stdout:1684: read oid 17 snap -1 2021-03-24T22:00:21.321 INFO:tasks.rados.rados.0.smithi125.stdout:1684: expect (ObjNum 550 snap 171 seq_num 550) 2021-03-24T22:00:21.322 INFO:tasks.rados.rados.0.smithi125.stdout:1685: write oid 50 current snap is 179 2021-03-24T22:00:21.322 INFO:tasks.rados.rados.0.smithi125.stdout:1685: seq_num 599 ranges {467393=549685,1802361=265032} 2021-03-24T22:00:21.322 INFO:tasks.rados.rados.0.smithi125.stdout:1685: writing smithi12556470-50 from 467393 to 1017078 tid 1 2021-03-24T22:00:21.323 INFO:tasks.rados.rados.0.smithi125.stdout:1685: writing smithi12556470-50 from 1802361 to 2067393 tid 2 2021-03-24T22:00:21.323 INFO:tasks.rados.rados.0.smithi125.stdout:1686: snap_remove snap 174 2021-03-24T22:00:21.324 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:21.376 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:21.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:21.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:00:21.483 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:21.537 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:21.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:21.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:00:21.550 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: 230826 2021-03-24T22:00:21.550 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: +0000 2021-03-24T22:00:21.551 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: 2021-03-24T22:00:21.551 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: osd.6 2021-03-24T22:00:21.551 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: ( 2021-03-24T22:00:21.552 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: osd 2021-03-24T22:00:21.552 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: . 2021-03-24T22:00:21.553 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: 6 2021-03-24T22:00:21.553 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: ) 2021-03-24T22:00:21.553 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: 30 2021-03-24T22:00:21.554 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: : 2021-03-24T22:00:21.554 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: cluster 2021-03-24T22:00:21.554 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: 2021-03-24T22:00:21.555 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: [DBG] 2021-03-24T22:00:21.555 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: 2021-03-24T22:00:21.555 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: purged_snaps scrub ok 2021-03-24T22:00:21.555 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: 2021-03-24T22:00:21.556 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: audit 2021-03-24T22:00:21.556 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: 2021-03-24T22:00:21.556 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: 2021 2021-03-24T22:00:21.557 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: - 2021-03-24T22:00:21.557 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: 03 2021-03-24T22:00:21.557 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: - 2021-03-24T22:00:21.557 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: 24 2021-03-24T22:00:21.558 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: T22:00:20.478541+0000 mon.b (mon.1) 37 : audit 2021-03-24T22:00:21.558 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: [DBG] from='client.? v1:172.21.15.125:0/3873398293' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-03-24T22:00:21.558 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: audit 2021-03-24T22:00:20.857612+0000 mon.a (mon.0) 1358 : audit 2021-03-24T22:00:21.559 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:21 smithi125 conmon[24927]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "27"}]: dispatch 2021-03-24T22:00:21.644 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:21.706 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:21.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:21.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:00:21.786 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.7 dump_historic_ops 2021-03-24T22:00:21.814 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:21.866 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:21.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:21.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:00:21.886 INFO:tasks.thrashosds.thrasher:j is {'epoch': 450, 'fsid': '3cfa339a-8cea-11eb-90b2-001a4aab830c', 'created': '2021-03-24T21:46:19.163312+0000', 'modified': '2021-03-24T22:00:20.230419+0000', 'last_up_change': '2021-03-24T22:00:02.225213+0000', 'last_in_change': '2021-03-24T21:59:46.230295+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-03-24T21:49:51.001357+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'iscsi', 'create_time': '2021-03-24T21:51:39.190981+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '178', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2021-03-24T21:51:44.159145+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 42, 'pg_placement_num': 42, 'pg_placement_num_target': 42, 'pg_num_target': 42, 'pg_num_pending': 42, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '383', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '381', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2021-03-24T21:51:46.090924+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 28, 'pg_placement_num': 22, 'pg_placement_num_target': 22, 'pg_num_target': 22, 'pg_num_pending': 28, 'last_pg_merge_meta': {'source_pgid': '4.1c', 'ready_epoch': 449, 'last_epoch_started': 448, 'last_epoch_clean': 448, 'source_version': "69'15", 'target_version': "69'13"}, 'last_change': '449', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '449', 'last_force_op_resend_preluminous': '447', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2021-03-24T21:51:47.842521+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2021-03-24T21:51:50.730906+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 8}, 'application_metadata': {'rgw': {}}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2021-03-24T21:54:21.087744+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 36, 'pg_placement_num': 36, 'pg_placement_num_target': 36, 'pg_num_target': 36, 'pg_num_pending': 36, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '450', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '275', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 340, 'snap_epoch': 450, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': '1f8f9a69-4b85-4054-8b11-36d35574494c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 439, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.125:6801', 'nonce': 1197472172}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.125:6802', 'nonce': 1197472172}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.125:6804', 'nonce': 1197472172}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.125:6803', 'nonce': 1197472172}]}, 'public_addr': '172.21.15.125:6801/1197472172', 'cluster_addr': '172.21.15.125:6802/1197472172', 'heartbeat_back_addr': '172.21.15.125:6804/1197472172', 'heartbeat_front_addr': '172.21.15.125:6803/1197472172', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '48db0723-5806-4e5c-9919-f3d7f52ad169', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 101, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.125:6805', 'nonce': 113845288}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.125:6806', 'nonce': 113845288}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.125:6808', 'nonce': 113845288}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.125:6807', 'nonce': 113845288}]}, 'public_addr': '172.21.15.125:6805/113845288', 'cluster_addr': '172.21.15.125:6806/113845288', 'heartbeat_back_addr': '172.21.15.125:6808/113845288', 'heartbeat_front_addr': '172.21.15.125:6807/113845288', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'd6a361f8-4e4d-4f46-8ab9-01bf71dbb2c8', 'up': 1, 'in': 1, 'weight': 0.7242584228515625, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 433, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.125:6809', 'nonce': 246939163}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.125:6810', 'nonce': 246939163}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.125:6812', 'nonce': 246939163}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.125:6811', 'nonce': 246939163}]}, 'public_addr': '172.21.15.125:6809/246939163', 'cluster_addr': '172.21.15.125:6810/246939163', 'heartbeat_back_addr': '172.21.15.125:6812/246939163', 'heartbeat_front_addr': '172.21.15.125:6811/246939163', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '716a41ce-7a28-400f-bf75-4c1ad99c7f94', 'up': 1, 'in': 1, 'weight': 0.9000244140625, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 434, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.125:6813', 'nonce': 2209636173}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.125:6814', 'nonce': 2209636173}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.125:6816', 'nonce': 2209636173}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.125:6815', 'nonce': 2209636173}]}, 'public_addr': '172.21.15.125:6813/2209636173', 'cluster_addr': '172.21.15.125:6814/2209636173', 'heartbeat_back_addr': '172.21.15.125:6816/2209636173', 'heartbeat_front_addr': '172.21.15.125:6815/2209636173', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'f9bdd8d2-b292-49da-ad4f-99682122f543', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 449, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.142:6800', 'nonce': 1816503330}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.142:6801', 'nonce': 1816503330}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.142:6803', 'nonce': 1816503330}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.142:6802', 'nonce': 1816503330}]}, 'public_addr': '172.21.15.142:6800/1816503330', 'cluster_addr': '172.21.15.142:6801/1816503330', 'heartbeat_back_addr': '172.21.15.142:6803/1816503330', 'heartbeat_front_addr': '172.21.15.142:6802/1816503330', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'ec0d43aa-6905-43f0-9e32-e19d4cf5129d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.664520263671875, 'last_clean_begin': 39, 'last_clean_end': 339, 'up_from': 432, 'up_thru': 433, 'down_at': 340, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.142:6804', 'nonce': 3992668882}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.142:6805', 'nonce': 3992668882}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.142:6807', 'nonce': 3992668882}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.142:6806', 'nonce': 3992668882}]}, 'public_addr': '172.21.15.142:6804/3992668882', 'cluster_addr': '172.21.15.142:6805/3992668882', 'heartbeat_back_addr': '172.21.15.142:6807/3992668882', 'heartbeat_front_addr': '172.21.15.142:6806/3992668882', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'a26a487e-ed26-4d92-98ea-896739302921', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 99, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.142:6808', 'nonce': 4048483913}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.142:6809', 'nonce': 4048483913}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.142:6811', 'nonce': 4048483913}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.142:6810', 'nonce': 4048483913}]}, 'public_addr': '172.21.15.142:6808/4048483913', 'cluster_addr': '172.21.15.142:6809/4048483913', 'heartbeat_back_addr': '172.21.15.142:6811/4048483913', 'heartbeat_front_addr': '172.21.15.142:6810/4048483913', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '682ba5ce-0172-48e3-82f8-57a3f98c911b', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 0.794097900390625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 55, 'up_thru': 416, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.142:6812', 'nonce': 3678129570}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.142:6813', 'nonce': 3678129570}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.142:6815', 'nonce': 3678129570}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.142:6814', 'nonce': 3678129570}]}, 'public_addr': '172.21.15.142:6812/3678129570', 'cluster_addr': '172.21.15.142:6813/3678129570', 'heartbeat_back_addr': '172.21.15.142:6815/3678129570', 'heartbeat_front_addr': '172.21.15.142:6814/3678129570', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-24T22:00:11.903425+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 65536, 'last_purged_snaps_scrub': '2021-03-24T21:59:31.832113+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-24T21:59:55.500612+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-24T22:00:14.787434+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-24T21:59:34.555580+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-24T22:00:01.559928+0000', 'dead_epoch': 340}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 65536, 'last_purged_snaps_scrub': '2021-03-24T22:00:20.230832+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138297136906239, 'old_weight': 58984, 'last_purged_snaps_scrub': '2021-03-24T21:59:39.345289+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '3.1d', 'mappings': [{'from': 4, 'to': 2}]}, {'pgid': '6.4', 'mappings': [{'from': 5, 'to': 0}]}, {'pgid': '7.10', 'mappings': [{'from': 2, 'to': 5}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.125:0/3077905055': '2021-03-25T21:47:04.502055+0000', '172.21.15.125:0/2210544773': '2021-03-25T21:47:04.502055+0000', '172.21.15.125:6800/3176807740': '2021-03-25T21:47:04.502055+0000', '172.21.15.125:6800/3531208850': '2021-03-25T21:46:43.197724+0000', '172.21.15.125:0/3337261959': '2021-03-25T21:46:43.197724+0000', '172.21.15.125:0/1919150488': '2021-03-25T21:46:43.197724+0000'}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 7, 'snaps': [{'begin': 231, 'length': 1}, {'begin': 248, 'length': 1}, {'begin': 268, 'length': 1}, {'begin': 294, 'length': 1}, {'begin': 296, 'length': 1}]}], 'new_removed_snaps': [{'pool': 7, 'snaps': [{'begin': 296, 'length': 1}]}], 'new_purged_snaps': [{'pool': 7, 'snaps': [{'begin': 261, 'length': 1}, {'begin': 272, 'length': 1}, {'begin': 281, 'length': 2}, {'begin': 301, 'length': 1}, {'begin': 317, 'length': 1}, {'begin': 323, 'length': 1}]}], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2021-03-24T22:00:21.886 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph pg dump --format=json 2021-03-24T22:00:21.971 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:22.026 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:22.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:22.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:00:22.098 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T22:00:22.130 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:22.178 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:22.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:22.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:22.283 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:22.285 INFO:tasks.rados.rados.0.smithi125.stdout:1686: done (3 left) 2021-03-24T22:00:22.286 INFO:tasks.rados.rados.0.smithi125.stdout:1687: snap_create 2021-03-24T22:00:22.286 INFO:tasks.rados.rados.0.smithi125.stdout:1681: finishing write tid 3 to smithi12556470-13 2021-03-24T22:00:22.287 INFO:tasks.rados.rados.0.smithi125.stdout:1681: finishing write tid 4 to smithi12556470-13 2021-03-24T22:00:22.287 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 13 v 307 (ObjNum 597 snap 179 seq_num 597) dirty exists 2021-03-24T22:00:22.287 INFO:tasks.rados.rados.0.smithi125.stdout:1681: left oid 13 (ObjNum 597 snap 179 seq_num 597) 2021-03-24T22:00:22.288 INFO:tasks.rados.rados.0.smithi125.stdout:1684: expect (ObjNum 550 snap 171 seq_num 550) 2021-03-24T22:00:22.296 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T22:00:22.297 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[24927]: T22:00:20.543440+0000 osd.0 (osd.0) 99 : cluster [DBG] 4.16 scrub ok 2021-03-24T22:00:22.298 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:00:22.298 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[24927]: 00:20.856669+0000 mgr.y (mgr.14144) 496 : 2021-03-24T22:00:22.298 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[24927]: cluster [DBG] pgmap v838: 174 pgs: 1 peering, 1 active+clean+wait, 172 active+clean; 412 MiB data, 1.0 GiB used, 446 GiB / 447 GiB avail; 14 MiB/s rd, 2.1 MiB/s wr, 23 op/s; 155 B/s, 0 objects/s recovering 2021-03-24T22:00:22.299 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[24927]: cluster 2021-03-24T22:00:22.299 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[24927]: 2021-03- 2021-03-24T22:00:22.300 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[24927]: 24T22:00:20.952122+0000 2021-03-24T22:00:22.300 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[24927]: osd.4 (osd.4) 106 : cluster 2021-03-24T22:00:22.300 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[24927]: [DBG] 4.e scrub ok 2021-03-24T22:00:22.301 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[24927]: audit 2021 2021-03-24T22:00:22.301 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[24927]: -03-24T22: 2021-03-24T22:00:22.301 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[24927]: 00:21.273266+0000 2021-03-24T22:00:22.302 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[24927]: mon.a (mon.0) 1359 : audit [INF] 2021-03-24T22:00:22.302 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[24927]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "27"}]': finished 2021-03-24T22:00:22.303 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:00:22.303 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[24927]: 00:21.273401+0000 mon.a (mon. 2021-03-24T22:00:22.304 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[24927]: 0) 1360 : cluster [DBG] osdmap e451: 8 total, 8 up, 5 in 2021-03-24T22:00:22.304 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[24927]: 2021-03-24T22:00:22.304 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[24927]: cluster 2021-03-24T22:00:22.305 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[24927]: 2021-03- 2021-03-24T22:00:22.305 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[24927]: 24T22:00:21.298323 2021-03-24T22:00:22.306 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[24927]: +0000 osd.3 (osd.3) 99 2021-03-24T22:00:22.306 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[24927]: : cluster [DBG] 6.11 scrub ok 2021-03-24T22:00:22.307 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[24927]: audit 2021-03-24T22:00:22.245444+0000 mon.a (mon.0) 1361 : audit [INF] from='client.? v1:172.21.15.125:0/174882477' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:00:22.307 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:00:22.308 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[19936]: T22:00:20.543440+0000 osd.0 (osd.0) 99 : cluster [DBG] 4.16 scrub ok 2021-03-24T22:00:22.308 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[19936]: cluster 2021-03-24T22:00:22.309 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[19936]: 2021-03-24T22:00 2021-03-24T22:00:22.309 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[19936]: :20.856669+0000 mgr.y 2021-03-24T22:00:22.309 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[19936]: (mgr.14144) 496 : 2021-03-24T22:00:22.310 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[19936]: cluster [DBG] pgmap v838: 174 pgs: 1 peering, 1 active+clean+wait, 172 active+clean; 412 MiB data, 1.0 GiB used, 446 GiB / 447 GiB avail; 14 MiB/s rd, 2.1 MiB/s wr, 23 op/s; 155 B/s, 0 objects/s recovering 2021-03-24T22:00:22.310 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[19936]: cluster 2021-03-24T22:00:22.311 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[19936]: 2021-03-24T22:00 2021-03-24T22:00:22.311 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[19936]: :20.952122+0000 osd.4 2021-03-24T22:00:22.312 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[19936]: (osd.4) 106 : 2021-03-24T22:00:22.312 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[19936]: cluster [DBG] 4.e scrub ok 2021-03-24T22:00:22.312 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[19936]: audit 2021-03 2021-03-24T22:00:22.313 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[19936]: -24T22:00:21.273266 2021-03-24T22:00:22.313 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[19936]: +0000 mon.a (mon.0 2021-03-24T22:00:22.313 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[19936]: ) 1359 : audit 2021-03-24T22:00:22.314 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[19936]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "27"}]': finished 2021-03-24T22:00:22.314 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T22:00:22.315 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[19936]: 22:00:21 2021-03-24T22:00:22.315 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[19936]: .273401+0000 mon.a 2021-03-24T22:00:22.315 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[19936]: (mon.0 2021-03-24T22:00:22.316 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[19936]: ) 1360 : cluster 2021-03-24T22:00:22.316 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[19936]: [DBG] osdmap e451: 8 total, 8 up, 5 in 2021-03-24T22:00:22.317 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:00:22.317 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[19936]: -03-24T22:00:21.298323+0000 osd.3 (osd. 2021-03-24T22:00:22.317 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[19936]: 3) 99 : cluster [DBG] 6.11 scrub ok 2021-03-24T22:00:22.318 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[19936]: audit 2021-03-24T 2021-03-24T22:00:22.318 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[19936]: 22:00:22.245444+0000 mon.a (mon 2021-03-24T22:00:22.318 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:22 smithi125 conmon[19936]: .0) 1361 : audit [INF] from='client.? v1:172.21.15.125:0/174882477' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:00:22.319 INFO:teuthology.orchestra.run.smithi125.stderr:nodeep-scrub is unset 2021-03-24T22:00:22.337 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:22.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:22.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:00:22.395 INFO:tasks.rados.rados.0.smithi125.stdout:1685: finishing write tid 1 to smithi12556470-50 2021-03-24T22:00:22.395 INFO:tasks.rados.rados.0.smithi125.stdout:1685: finishing write tid 2 to smithi12556470-50 2021-03-24T22:00:22.396 INFO:tasks.rados.rados.0.smithi125.stdout:1685: finishing write tid 3 to smithi12556470-50 2021-03-24T22:00:22.396 INFO:tasks.rados.rados.0.smithi125.stdout:1685: finishing write tid 4 to smithi12556470-50 2021-03-24T22:00:22.397 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 50 v 452 (ObjNum 599 snap 179 seq_num 599) dirty exists 2021-03-24T22:00:22.397 INFO:tasks.rados.rados.0.smithi125.stdout:1685: left oid 50 (ObjNum 599 snap 179 seq_num 599) 2021-03-24T22:00:22.447 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:22.502 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:22.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:22.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:00:22.547 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:22 smithi142 conmon[20125]: cluster 2021-03-24T22:00:20.543440+0000 osd.0 (osd.0) 99 : cluster [DBG] 2021-03-24T22:00:22.547 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:22 smithi142 conmon[20125]: 4.16 scrub ok 2021-03-24T22:00:22.547 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:22 smithi142 conmon[20125]: cluster 2021-03-24T22:00:20.856669+0000 mgr.y (mgr.14144) 496 : cluster [DBG] 2021-03-24T22:00:22.547 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:22 smithi142 conmon[20125]: pgmap v838: 174 pgs: 1 peering, 1 active+clean+wait, 172 active+clean; 412 MiB data, 1.0 GiB used, 446 GiB / 447 GiB avail; 14 MiB/s rd, 2.1 MiB/s wr, 23 op/s; 155 B/s, 0 objects/s recovering 2021-03-24T22:00:22.548 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:22 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:00:22.548 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:22 smithi142 conmon[20125]: T22:00:20.952122+0000 osd.4 (osd.4) 106 : 2021-03-24T22:00:22.548 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:22 smithi142 conmon[20125]: cluster [DBG] 4.e scrub ok 2021-03-24T22:00:22.548 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:22 smithi142 conmon[20125]: audit 2021-03-24T22:00:22.549 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:22 smithi142 conmon[20125]: 2021-03-24T22: 2021-03-24T22:00:22.549 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:22 smithi142 conmon[20125]: 00:21.273266+0000 mon.a (mon.0 2021-03-24T22:00:22.549 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:22 smithi142 conmon[20125]: ) 1359 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "27"}]': finished 2021-03-24T22:00:22.549 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:22 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:00:22.549 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:22 smithi142 conmon[20125]: 24T22:00:21.273401+0000 mon.a 2021-03-24T22:00:22.550 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:22 smithi142 conmon[20125]: (mon.0) 1360 : cluster [DBG] osdmap e451: 8 total, 8 up, 5 in 2021-03-24T22:00:22.550 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:22 smithi142 conmon[20125]: 2021-03-24T22:00:22.550 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:22 smithi142 conmon[20125]: cluster 2021-03-24T22:00:22.550 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:22 smithi142 conmon[20125]: 2021-03-24T22:00:21.298323+0000 2021-03-24T22:00:22.551 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:22 smithi142 conmon[20125]: osd.3 (osd.3) 99 : cluster [DBG] 2021-03-24T22:00:22.551 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:22 smithi142 conmon[20125]: 6.11 scrub ok 2021-03-24T22:00:22.551 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:22 smithi142 conmon[20125]: audit 2021-03-24T22:00:22.551 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:22 smithi142 conmon[20125]: 2021-03-24T22:00:22.245444+0000 2021-03-24T22:00:22.552 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:22 smithi142 conmon[20125]: mon.a (mon.0) 1361 : audit [INF] 2021-03-24T22:00:22.552 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:22 smithi142 conmon[20125]: from='client.? v1:172.21.15.125:0/174882477' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:00:22.610 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:22.671 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:22.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:22.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:00:22.777 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:22.834 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:22.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:22.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:00:22.940 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:22.998 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:23.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:23.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:00:23.105 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:23.165 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:23.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:23.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:00:23.272 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:23.290 INFO:tasks.rados.rados.0.smithi125.stdout:1681: done (3 left) 2021-03-24T22:00:23.290 INFO:tasks.rados.rados.0.smithi125.stdout:1684: done (2 left) 2021-03-24T22:00:23.291 INFO:tasks.rados.rados.0.smithi125.stdout:1685: done (1 left) 2021-03-24T22:00:23.291 INFO:tasks.rados.rados.0.smithi125.stdout:1687: done (0 left) 2021-03-24T22:00:23.291 INFO:tasks.rados.rados.0.smithi125.stdout:1688: delete oid 49 current snap is 180 2021-03-24T22:00:23.300 INFO:tasks.rados.rados.0.smithi125.stdout:1688: done (0 left) 2021-03-24T22:00:23.301 INFO:tasks.rados.rados.0.smithi125.stdout:1689: write (excl) oid 30 current snap is 180 2021-03-24T22:00:23.302 INFO:tasks.rados.rados.0.smithi125.stdout:1689: seq_num 601 ranges {501007=416536,1635678=465329} 2021-03-24T22:00:23.305 INFO:tasks.rados.rados.0.smithi125.stdout:1689: writing smithi12556470-30 from 501007 to 917543 tid 1 2021-03-24T22:00:23.313 INFO:tasks.rados.rados.0.smithi125.stdout:1689: writing smithi12556470-30 from 1635678 to 2101007 tid 2 2021-03-24T22:00:23.313 INFO:tasks.rados.rados.0.smithi125.stdout:1690: rollback oid 23 current snap is 180 2021-03-24T22:00:23.313 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 23 to 151 2021-03-24T22:00:23.313 INFO:tasks.rados.rados.0.smithi125.stdout:1689: finishing write tid 1 to smithi12556470-30 2021-03-24T22:00:23.314 INFO:tasks.rados.rados.0.smithi125.stdout:1691: snap_create 2021-03-24T22:00:23.317 INFO:tasks.rados.rados.0.smithi125.stdout:1689: finishing write tid 2 to smithi12556470-30 2021-03-24T22:00:23.317 INFO:tasks.rados.rados.0.smithi125.stdout:1689: finishing write tid 3 to smithi12556470-30 2021-03-24T22:00:23.318 INFO:tasks.rados.rados.0.smithi125.stdout:1689: finishing write tid 4 to smithi12556470-30 2021-03-24T22:00:23.318 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 30 v 292 (ObjNum 601 snap 180 seq_num 601) dirty exists 2021-03-24T22:00:23.319 INFO:tasks.rados.rados.0.smithi125.stdout:1689: left oid 30 (ObjNum 601 snap 180 seq_num 601) 2021-03-24T22:00:23.319 INFO:tasks.rados.rados.0.smithi125.stdout:1690: finishing rollback tid 0 to smithi12556470-23 2021-03-24T22:00:23.319 INFO:tasks.rados.rados.0.smithi125.stdout:1690: finishing rollback tid 1 to smithi12556470-23 2021-03-24T22:00:23.320 INFO:tasks.rados.rados.0.smithi125.stdout:1690: finishing rollback tid 2 to smithi12556470-23 2021-03-24T22:00:23.320 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 23 v 287 (ObjNum 427 snap 113 seq_num 427) dirty exists 2021-03-24T22:00:23.327 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:23.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:23.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:00:23.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:23 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:00:23.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:23 smithi125 conmon[19936]: 00:22.266164+0000 mon.a (mon.0) 1362 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T22:00:23.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:23 smithi125 conmon[19936]: cluster 2021-03-24T22:00 2021-03-24T22:00:23.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:23 smithi125 conmon[19936]: :22.266208+0000 mon.a (mon.0) 1363 : cluster [INF] Cluster is now healthy 2021-03-24T22:00:23.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:23 smithi125 conmon[19936]: audit 2021-03- 2021-03-24T22:00:23.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:23 smithi125 conmon[19936]: 24T22:00:22.278643+0000 mon.a (mon.0) 1364 : audit [INF] from='client.? v1:172.21.15.125:0/174882477' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T22:00:23.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:23 smithi125 conmon[19936]: cluster 2021-03-24T22:00: 2021-03-24T22:00:23.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:23 smithi125 conmon[19936]: 22.278746+0000 mon.a (mon.0) 1365 : 2021-03-24T22:00:23.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:23 smithi125 conmon[19936]: cluster [DBG] osdmap e452: 8 total, 8 up, 5 in 2021-03-24T22:00:23.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:23 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:00:23.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:23 smithi125 conmon[19936]: T22:00:22.297219+0000 osd.5 ( 2021-03-24T22:00:23.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:23 smithi125 conmon[19936]: osd.5) 3 : cluster [DBG] 5.1b scrub ok 2021-03-24T22:00:23.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:23 smithi125 conmon[24927]: cluster 2021-03-24T22 2021-03-24T22:00:23.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:23 smithi125 conmon[24927]: :00:22.266164+0000 mon.a (mon.0) 1362 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T22:00:23.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:23 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:00:23.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:23 smithi125 conmon[24927]: 03-24T22:00:22.266208+0000 mon.a (mon.0 2021-03-24T22:00:23.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:23 smithi125 conmon[24927]: ) 1363 : cluster [INF] Cluster is now healthy 2021-03-24T22:00:23.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:23 smithi125 conmon[24927]: audit 2021-03-24T22:00:22.278643+0000 mon.a ( 2021-03-24T22:00:23.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:23 smithi125 conmon[24927]: mon.0) 1364 : audit [INF] from='client.? v1:172.21.15.125:0/174882477' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T22:00:23.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:23 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T22:00:23.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:23 smithi125 conmon[24927]: -24T22:00:22.278746+0000 mon.a 2021-03-24T22:00:23.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:23 smithi125 conmon[24927]: (mon.0) 1365 : cluster [DBG] osdmap e452: 8 total, 8 up, 5 in 2021-03-24T22:00:23.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:23 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:00:23.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:23 smithi125 conmon[24927]: 03-24T22:00:22.297219+0000 osd.5 2021-03-24T22:00:23.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:23 smithi125 conmon[24927]: (osd.5) 3 : cluster [DBG] 5.1b scrub ok 2021-03-24T22:00:23.433 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:23.492 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:23.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:23.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:23.580 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:23 smithi142 conmon[20125]: cluster 2021-03-24T22:00:22.266164+0000 mon.a (mon.0) 1362 : cluster 2021-03-24T22:00:23.580 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:23 smithi142 conmon[20125]: [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T22:00:23.580 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:23 smithi142 conmon[20125]: cluster 2021-03-24T22:00:22.266208+0000 mon.a (mon. 2021-03-24T22:00:23.580 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:23 smithi142 conmon[20125]: 0) 1363 : cluster [INF] Cluster is now healthy 2021-03-24T22:00:23.581 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:23 smithi142 conmon[20125]: audit 2021-03-24T22:00 2021-03-24T22:00:23.581 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:23 smithi142 conmon[20125]: :22.278643+0000 mon.a (mon.0) 2021-03-24T22:00:23.581 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:23 smithi142 conmon[20125]: 1364 : audit [INF] from='client.? v1:172.21.15.125:0/174882477' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T22:00:23.582 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:23 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:00:23.582 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:23 smithi142 conmon[20125]: 22:00:22.278746+0000 mon.a (mon 2021-03-24T22:00:23.582 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:23 smithi142 conmon[20125]: .0) 1365 : cluster [DBG] osdmap e452: 8 total, 8 up, 5 in 2021-03-24T22:00:23.583 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:23 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:00:23.583 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:23 smithi142 conmon[20125]: 03-24T22:00:22.297219+0000 2021-03-24T22:00:23.583 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:23 smithi142 conmon[20125]: osd.5 (osd.5) 3 : cluster [DBG] 2021-03-24T22:00:23.583 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:23 smithi142 conmon[20125]: 5.1b scrub ok 2021-03-24T22:00:23.597 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:23.657 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:23.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:23.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:00:23.763 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:23.815 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:23.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:23.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:23.922 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:23.974 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:23.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:23.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:00:24.067 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:00:24.081 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:24.136 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:24.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:24.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:00:24.154 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2021-03-24T22:00:24.155 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2021-03-24T22:00:24.155 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2021-03-24T22:00:24.155 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2021-03-24T22:00:24.155 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.156 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.34352.0:3 3.13 3:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected e61)", 2021-03-24T22:00:24.156 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:51:45.387966+0000", 2021-03-24T22:00:24.156 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 518.76123260199995, 2021-03-24T22:00:24.156 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.19192994199999999, 2021-03-24T22:00:24.157 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:24.157 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "reached pg", 2021-03-24T22:00:24.157 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:00:24.158 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.34352", 2021-03-24T22:00:24.158 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/3436348424", 2021-03-24T22:00:24.158 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 3 2021-03-24T22:00:24.158 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.159 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:24.159 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.159 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:24.159 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:45.387966+0000", 2021-03-24T22:00:24.160 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:24.160 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.160 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.160 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:24.161 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:45.387966+0000", 2021-03-24T22:00:24.161 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2021-03-24T22:00:24.162 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.162 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.163 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:24.163 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:45.387964+0000", 2021-03-24T22:00:24.163 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.7370000000000006e-06 2021-03-24T22:00:24.164 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.164 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.165 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:24.165 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:45.387972+0000", 2021-03-24T22:00:24.166 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5412000000000001e-05 2021-03-24T22:00:24.166 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.167 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.167 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:24.167 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:45.387987+0000", 2021-03-24T22:00:24.168 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.216e-06 2021-03-24T22:00:24.169 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.169 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.169 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:24.170 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:45.387994+0000", 2021-03-24T22:00:24.170 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0209000000000004e-05 2021-03-24T22:00:24.171 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.171 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.172 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:24.172 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:45.388074+0000", 2021-03-24T22:00:24.173 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2846e-05 2021-03-24T22:00:24.173 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.174 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.174 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for peered", 2021-03-24T22:00:24.174 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:45.388087+0000", 2021-03-24T22:00:24.175 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.19160065900000001 2021-03-24T22:00:24.175 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.176 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.176 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:24.177 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:45.579688+0000", 2021-03-24T22:00:24.177 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00020767600000000001 2021-03-24T22:00:24.181 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.181 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.181 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:24.181 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:51:45.579896+0000", 2021-03-24T22:00:24.182 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.19192994199999999 2021-03-24T22:00:24.182 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:24.182 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:24.182 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:24.182 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.183 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.183 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.24388.0:166 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected e107)", 2021-03-24T22:00:24.183 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:54:37.427176+0000", 2021-03-24T22:00:24.183 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 346.72202231099999, 2021-03-24T22:00:24.184 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.699168805, 2021-03-24T22:00:24.184 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:24.184 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:00:24.184 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:00:24.184 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.24388", 2021-03-24T22:00:24.185 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.142:0/626803041", 2021-03-24T22:00:24.185 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 166 2021-03-24T22:00:24.185 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.185 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:24.186 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.186 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:24.186 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:37.427176+0000", 2021-03-24T22:00:24.186 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:24.187 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.187 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.187 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:24.187 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:37.427176+0000", 2021-03-24T22:00:24.187 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999957 2021-03-24T22:00:24.188 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.188 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.188 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:24.188 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:37.427172+0000", 2021-03-24T22:00:24.189 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8757e-05 2021-03-24T22:00:24.189 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.189 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.189 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:24.189 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:37.427190+0000", 2021-03-24T22:00:24.190 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3581999999999998e-05 2021-03-24T22:00:24.190 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.190 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.190 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:24.191 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:37.427224+0000", 2021-03-24T22:00:24.191 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5787e-05 2021-03-24T22:00:24.191 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.191 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.191 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:24.192 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:37.427240+0000", 2021-03-24T22:00:24.192 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00020074000000000001 2021-03-24T22:00:24.192 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.192 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.192 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:24.193 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:37.427440+0000", 2021-03-24T22:00:24.193 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8619000000000002e-05 2021-03-24T22:00:24.193 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.193 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.194 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for peered", 2021-03-24T22:00:24.194 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:37.427469+0000", 2021-03-24T22:00:24.194 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.69859708399999998 2021-03-24T22:00:24.194 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.194 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.195 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:24.195 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:38.126066+0000", 2021-03-24T22:00:24.195 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133013 2021-03-24T22:00:24.195 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.195 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.196 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:00:24.196 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:38.126199+0000", 2021-03-24T22:00:24.196 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014562000000000001 2021-03-24T22:00:24.197 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.197 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.197 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:24.197 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:38.126345+0000", 2021-03-24T22:00:24.197 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.699168805 2021-03-24T22:00:24.198 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:24.198 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:24.199 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:24.199 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.199 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.199 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:1777 7.17 7:ef6d6bee:::smithi12556470-24:head [write 420241~546489,stat] snapc 7e=[7e,79,78] ondisk+write+known_if_redirected e232)", 2021-03-24T22:00:24.200 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:56:41.195831+0000", 2021-03-24T22:00:24.200 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 222.95336684700001, 2021-03-24T22:00:24.200 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.046456559000000001, 2021-03-24T22:00:24.200 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:24.200 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:00:24.201 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:00:24.201 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:00:24.201 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:00:24.201 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 1777 2021-03-24T22:00:24.201 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.202 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:24.202 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.202 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:24.202 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.195831+0000", 2021-03-24T22:00:24.203 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:24.203 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.203 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.203 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:24.203 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.195831+0000", 2021-03-24T22:00:24.204 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999866 2021-03-24T22:00:24.204 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.204 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.204 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:24.205 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.195818+0000", 2021-03-24T22:00:24.205 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00086681300000000002 2021-03-24T22:00:24.205 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.205 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.205 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:24.206 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.196685+0000", 2021-03-24T22:00:24.206 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5661000000000003e-05 2021-03-24T22:00:24.206 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.206 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.206 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:24.207 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.196721+0000", 2021-03-24T22:00:24.207 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5569999999999998e-05 2021-03-24T22:00:24.207 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.207 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.208 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:24.208 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.196736+0000", 2021-03-24T22:00:24.208 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000108026 2021-03-24T22:00:24.208 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.208 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.209 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:24.209 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.196844+0000", 2021-03-24T22:00:24.209 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1788e-05 2021-03-24T22:00:24.209 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.209 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.210 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op must wait for map", 2021-03-24T22:00:24.211 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.196856+0000", 2021-03-24T22:00:24.211 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0047483960000000002 2021-03-24T22:00:24.211 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.211 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.212 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:24.212 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.201604+0000", 2021-03-24T22:00:24.212 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00021254800000000001 2021-03-24T22:00:24.213 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.213 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.214 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:00:24.214 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.201817+0000", 2021-03-24T22:00:24.214 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00043252299999999998 2021-03-24T22:00:24.215 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.215 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.216 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 2,5", 2021-03-24T22:00:24.216 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.202249+0000", 2021-03-24T22:00:24.216 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0031197579999999998 2021-03-24T22:00:24.217 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.217 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.218 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:24.218 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.205369+0000", 2021-03-24T22:00:24.219 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.019907871000000001 2021-03-24T22:00:24.219 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.220 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.220 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T22:00:24.220 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.225277+0000", 2021-03-24T22:00:24.221 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.016760265 2021-03-24T22:00:24.221 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.222 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.222 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:24.223 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.242037+0000", 2021-03-24T22:00:24.223 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8051000000000001e-05 2021-03-24T22:00:24.223 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.224 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.224 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:00:24.225 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.242125+0000", 2021-03-24T22:00:24.225 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000162585 2021-03-24T22:00:24.226 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.226 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.226 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:24.227 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.242288+0000", 2021-03-24T22:00:24.227 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.046456559000000001 2021-03-24T22:00:24.228 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:24.228 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:24.229 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:24.229 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.230 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.230 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:1780 7.17 7:ef6d6bee:::smithi12556470-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e232)", 2021-03-24T22:00:24.231 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:56:41.202362+0000", 2021-03-24T22:00:24.231 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 222.94683577699999, 2021-03-24T22:00:24.231 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.044960122999999998, 2021-03-24T22:00:24.232 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:24.232 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:00:24.232 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:00:24.233 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:00:24.233 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:00:24.233 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 1780 2021-03-24T22:00:24.234 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.234 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:24.235 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.235 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:24.235 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.202362+0000", 2021-03-24T22:00:24.236 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:24.236 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.236 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.237 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:24.237 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.202362+0000", 2021-03-24T22:00:24.238 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2021-03-24T22:00:24.238 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.238 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.239 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:24.239 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.202362+0000", 2021-03-24T22:00:24.239 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1719999999999999e-06 2021-03-24T22:00:24.240 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.240 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.241 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:24.241 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.202365+0000", 2021-03-24T22:00:24.241 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1340000000000001e-06 2021-03-24T22:00:24.242 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.242 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.242 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:24.243 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.202371+0000", 2021-03-24T22:00:24.243 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0439999999999997e-06 2021-03-24T22:00:24.244 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.244 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.244 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:24.245 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.202375+0000", 2021-03-24T22:00:24.245 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0027201600000000001 2021-03-24T22:00:24.245 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.246 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.246 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:24.246 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.205095+0000", 2021-03-24T22:00:24.246 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018977600000000001 2021-03-24T22:00:24.247 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.247 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.247 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for rw locks", 2021-03-24T22:00:24.247 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.205285+0000", 2021-03-24T22:00:24.247 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.039686143 2021-03-24T22:00:24.248 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.248 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.248 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:24.248 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.244971+0000", 2021-03-24T22:00:24.249 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015303199999999999 2021-03-24T22:00:24.249 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.249 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.249 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for rw locks", 2021-03-24T22:00:24.250 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.245124+0000", 2021-03-24T22:00:24.250 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0018001289999999999 2021-03-24T22:00:24.250 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.250 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.250 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:24.251 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.246924+0000", 2021-03-24T22:00:24.251 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000103454 2021-03-24T22:00:24.251 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.251 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.252 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:00:24.252 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.247028+0000", 2021-03-24T22:00:24.252 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00029469099999999997 2021-03-24T22:00:24.252 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.252 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.253 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:24.253 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:41.247323+0000", 2021-03-24T22:00:24.253 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.044960122999999998 2021-03-24T22:00:24.253 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:24.254 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:24.254 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:24.254 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.254 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.255 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:2099 7.17 7:ec6a359b:::smithi12556470-33:head [copy-from ver 0] snapc 95=[95,93,92,8d,8b,82] ondisk+write+known_if_redirected e255)", 2021-03-24T22:00:24.255 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:57:04.197620+0000", 2021-03-24T22:00:24.255 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 199.951578149, 2021-03-24T22:00:24.255 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.95919054100000001, 2021-03-24T22:00:24.256 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:24.256 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:00:24.256 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:00:24.256 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:00:24.257 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:00:24.257 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 2099 2021-03-24T22:00:24.257 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.257 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:24.258 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.258 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:24.258 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.197620+0000", 2021-03-24T22:00:24.258 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:24.259 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.259 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.259 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:24.259 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.197620+0000", 2021-03-24T22:00:24.260 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999962 2021-03-24T22:00:24.260 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.260 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.260 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:24.260 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.197616+0000", 2021-03-24T22:00:24.261 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7652999999999999e-05 2021-03-24T22:00:24.261 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.261 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.261 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:24.262 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.197634+0000", 2021-03-24T22:00:24.262 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6417000000000001e-05 2021-03-24T22:00:24.262 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.263 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.263 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:24.263 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.197660+0000", 2021-03-24T22:00:24.263 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9066e-05 2021-03-24T22:00:24.264 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.264 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.264 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:24.264 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.197680+0000", 2021-03-24T22:00:24.264 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00017438800000000001 2021-03-24T22:00:24.265 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.265 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.265 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:24.265 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.197854+0000", 2021-03-24T22:00:24.266 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3726e-05 2021-03-24T22:00:24.266 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.266 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.266 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op must wait for map", 2021-03-24T22:00:24.267 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.197878+0000", 2021-03-24T22:00:24.267 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0099756870000000004 2021-03-24T22:00:24.267 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.267 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.268 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:24.268 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.207853+0000", 2021-03-24T22:00:24.268 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3523000000000007e-05 2021-03-24T22:00:24.268 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.268 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.269 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:00:24.269 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.207937+0000", 2021-03-24T22:00:24.269 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.948873784 2021-03-24T22:00:24.269 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.270 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.270 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:24.270 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:05.156811+0000", 2021-03-24T22:00:24.270 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.95919054100000001 2021-03-24T22:00:24.270 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:24.271 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:24.271 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:24.271 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.271 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.272 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:2100 7.17 7:ec6a359b:::smithi12556470-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e255)", 2021-03-24T22:00:24.272 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:57:04.198076+0000", 2021-03-24T22:00:24.272 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 199.95112177499999, 2021-03-24T22:00:24.272 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.95908168400000005, 2021-03-24T22:00:24.272 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:24.273 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:00:24.273 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:00:24.273 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:00:24.273 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:00:24.274 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 2100 2021-03-24T22:00:24.274 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.274 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:24.274 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.274 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:24.275 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.198076+0000", 2021-03-24T22:00:24.275 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:24.275 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.275 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.276 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:24.276 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.198076+0000", 2021-03-24T22:00:24.276 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999971 2021-03-24T22:00:24.276 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.277 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.277 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:24.277 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.198074+0000", 2021-03-24T22:00:24.277 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3433e-05 2021-03-24T22:00:24.278 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.278 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.278 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:24.278 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.198087+0000", 2021-03-24T22:00:24.279 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.262e-05 2021-03-24T22:00:24.279 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.279 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.279 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:24.280 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.198110+0000", 2021-03-24T22:00:24.280 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6405999999999998e-05 2021-03-24T22:00:24.280 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.280 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.280 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:24.281 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.198126+0000", 2021-03-24T22:00:24.281 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000109786 2021-03-24T22:00:24.281 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.282 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.282 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:24.282 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.198236+0000", 2021-03-24T22:00:24.282 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0522e-05 2021-03-24T22:00:24.282 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.283 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.283 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting_for_map not empty", 2021-03-24T22:00:24.283 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.198246+0000", 2021-03-24T22:00:24.283 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0098265899999999996 2021-03-24T22:00:24.284 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.284 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.284 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:24.284 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.208073+0000", 2021-03-24T22:00:24.284 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7707999999999999e-05 2021-03-24T22:00:24.285 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.285 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.285 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for blocked object", 2021-03-24T22:00:24.285 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.208111+0000", 2021-03-24T22:00:24.286 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.94884332199999999 2021-03-24T22:00:24.286 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.286 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.287 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:24.287 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:05.156954+0000", 2021-03-24T22:00:24.287 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.8762000000000001e-05 2021-03-24T22:00:24.287 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.288 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.288 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:00:24.288 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:05.157053+0000", 2021-03-24T22:00:24.288 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000105321 2021-03-24T22:00:24.289 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.289 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.289 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:24.289 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:05.157158+0000", 2021-03-24T22:00:24.289 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.95908168400000005 2021-03-24T22:00:24.290 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:24.290 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:24.290 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:24.290 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.291 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.291 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:2557 7.11 7:8dd64f9e:::smithi12556470-49:head [write 484134~708729] snapc b7=[b7,b4] ondisk+write+known_if_redirected e290)", 2021-03-24T22:00:24.291 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:57:40.204419+0000", 2021-03-24T22:00:24.291 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 163.944779184, 2021-03-24T22:00:24.292 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.054577542, 2021-03-24T22:00:24.292 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:24.292 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:00:24.292 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:00:24.293 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:00:24.293 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:00:24.293 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 2557 2021-03-24T22:00:24.293 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.294 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:24.294 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.294 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:24.295 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.204419+0000", 2021-03-24T22:00:24.295 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:24.295 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.295 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.296 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:24.296 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.204419+0000", 2021-03-24T22:00:24.296 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2021-03-24T22:00:24.296 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.296 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.297 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:24.297 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.204418+0000", 2021-03-24T22:00:24.297 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00064114499999999997 2021-03-24T22:00:24.297 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.298 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.298 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:24.298 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.205059+0000", 2021-03-24T22:00:24.298 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3905e-05 2021-03-24T22:00:24.298 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.299 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.299 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:24.299 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.205073+0000", 2021-03-24T22:00:24.299 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4770000000000002e-06 2021-03-24T22:00:24.300 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.300 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.301 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:24.301 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.205078+0000", 2021-03-24T22:00:24.301 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.0301000000000002e-05 2021-03-24T22:00:24.302 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.302 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.303 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:24.303 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.205168+0000", 2021-03-24T22:00:24.303 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010134200000000001 2021-03-24T22:00:24.304 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.304 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.304 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:00:24.304 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.205270+0000", 2021-03-24T22:00:24.305 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000210287 2021-03-24T22:00:24.305 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.305 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.305 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 2,3", 2021-03-24T22:00:24.306 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.205480+0000", 2021-03-24T22:00:24.306 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.017867886999999999 2021-03-24T22:00:24.306 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.306 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.307 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T22:00:24.307 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.223348+0000", 2021-03-24T22:00:24.307 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029882784999999999 2021-03-24T22:00:24.307 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.307 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.308 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:24.308 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.253231+0000", 2021-03-24T22:00:24.308 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0055580200000000003 2021-03-24T22:00:24.308 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.308 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.309 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:24.309 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.258789+0000", 2021-03-24T22:00:24.309 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010166999999999999 2021-03-24T22:00:24.309 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.310 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.310 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:00:24.310 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.258890+0000", 2021-03-24T22:00:24.310 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010621 2021-03-24T22:00:24.311 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.311 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.311 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:24.311 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.258997+0000", 2021-03-24T22:00:24.312 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.054577542 2021-03-24T22:00:24.312 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:24.312 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:24.312 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:24.312 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.313 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.313 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:2558 7.11 7:8dd64f9e:::smithi12556470-49:head [write 1883658~200476] snapc b7=[b7,b4] ondisk+write+known_if_redirected e290)", 2021-03-24T22:00:24.313 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:57:40.209863+0000", 2021-03-24T22:00:24.313 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 163.93933572399999, 2021-03-24T22:00:24.314 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.049555218999999998, 2021-03-24T22:00:24.314 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:24.314 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:00:24.314 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:00:24.315 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:00:24.315 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:00:24.315 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 2558 2021-03-24T22:00:24.315 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.315 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:24.316 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.316 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:24.316 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.209863+0000", 2021-03-24T22:00:24.316 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:24.316 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.317 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.317 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:24.317 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.209863+0000", 2021-03-24T22:00:24.317 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2021-03-24T22:00:24.318 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.318 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.318 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:24.318 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.209861+0000", 2021-03-24T22:00:24.319 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00020491199999999999 2021-03-24T22:00:24.319 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.319 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.320 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:24.320 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.210066+0000", 2021-03-24T22:00:24.320 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2717e-05 2021-03-24T22:00:24.321 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.321 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.321 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:24.321 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.210079+0000", 2021-03-24T22:00:24.322 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.7589999999999997e-06 2021-03-24T22:00:24.322 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.322 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.322 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:24.322 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.210083+0000", 2021-03-24T22:00:24.323 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000103578 2021-03-24T22:00:24.323 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.323 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.323 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:24.324 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.210187+0000", 2021-03-24T22:00:24.324 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4052e-05 2021-03-24T22:00:24.324 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.324 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.324 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:00:24.325 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.210281+0000", 2021-03-24T22:00:24.325 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000160385 2021-03-24T22:00:24.325 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.325 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.325 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 2,3", 2021-03-24T22:00:24.326 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.210441+0000", 2021-03-24T22:00:24.326 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.012921141000000001 2021-03-24T22:00:24.326 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.326 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.327 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T22:00:24.327 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.223362+0000", 2021-03-24T22:00:24.327 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.035776595000000001 2021-03-24T22:00:24.327 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.328 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.328 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:24.328 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.259139+0000", 2021-03-24T22:00:24.329 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.5758e-05 2021-03-24T22:00:24.329 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.330 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.330 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:24.330 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.259235+0000", 2021-03-24T22:00:24.331 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000102464 2021-03-24T22:00:24.331 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.332 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.332 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:00:24.333 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.259337+0000", 2021-03-24T22:00:24.333 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0425000000000003e-05 2021-03-24T22:00:24.333 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.333 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.334 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:24.334 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.259418+0000", 2021-03-24T22:00:24.334 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.049555218999999998 2021-03-24T22:00:24.334 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:24.335 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:24.335 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:24.335 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.335 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.336 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:2559 7.11 7:8dd64f9e:::smithi12556470-49:head [setxattr _header (57) in=7b,truncate 2084134] snapc b7=[b7,b4] ondisk+write+known_if_redirected e290)", 2021-03-24T22:00:24.336 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:57:40.210092+0000", 2021-03-24T22:00:24.336 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 163.93910609, 2021-03-24T22:00:24.336 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.049710130999999998, 2021-03-24T22:00:24.336 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:24.337 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:00:24.337 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:00:24.337 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:00:24.337 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:00:24.337 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 2559 2021-03-24T22:00:24.338 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.338 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:24.338 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.338 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:24.339 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.210092+0000", 2021-03-24T22:00:24.339 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:24.339 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.339 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.339 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:24.340 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.210092+0000", 2021-03-24T22:00:24.340 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2021-03-24T22:00:24.340 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.340 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.341 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:24.341 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.210091+0000", 2021-03-24T22:00:24.341 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8589999999999998e-06 2021-03-24T22:00:24.341 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.341 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.342 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:24.342 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.210095+0000", 2021-03-24T22:00:24.342 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4030000000000007e-06 2021-03-24T22:00:24.342 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.343 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.343 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:24.343 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.210104+0000", 2021-03-24T22:00:24.343 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6840000000000001e-06 2021-03-24T22:00:24.343 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.344 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.344 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:24.344 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.210106+0000", 2021-03-24T22:00:24.344 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0010918010000000001 2021-03-24T22:00:24.344 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.345 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.345 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:24.345 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.211198+0000", 2021-03-24T22:00:24.345 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015686199999999999 2021-03-24T22:00:24.346 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.346 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.346 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:00:24.346 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.211355+0000", 2021-03-24T22:00:24.346 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00025926900000000001 2021-03-24T22:00:24.347 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.347 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.347 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 2,3", 2021-03-24T22:00:24.347 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.211614+0000", 2021-03-24T22:00:24.347 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011760836 2021-03-24T22:00:24.348 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.348 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.348 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T22:00:24.348 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.223375+0000", 2021-03-24T22:00:24.349 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.036157878999999997 2021-03-24T22:00:24.349 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.349 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.349 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:24.349 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.259533+0000", 2021-03-24T22:00:24.350 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2786000000000005e-05 2021-03-24T22:00:24.350 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.350 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.350 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:24.351 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.259596+0000", 2021-03-24T22:00:24.351 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000106619 2021-03-24T22:00:24.351 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.351 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.351 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:00:24.352 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.259702+0000", 2021-03-24T22:00:24.352 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.9850000000000001e-05 2021-03-24T22:00:24.352 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.352 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.352 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:24.353 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.259802+0000", 2021-03-24T22:00:24.353 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.049710130999999998 2021-03-24T22:00:24.353 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:24.353 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:24.354 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:24.354 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.354 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.354 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:2560 7.11 7:8dd64f9e:::smithi12556470-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e290)", 2021-03-24T22:00:24.354 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:57:40.210109+0000", 2021-03-24T22:00:24.361 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 163.939089032, 2021-03-24T22:00:24.361 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.050155837000000002, 2021-03-24T22:00:24.361 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:24.362 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:00:24.362 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:00:24.362 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:00:24.362 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:00:24.363 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 2560 2021-03-24T22:00:24.363 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.363 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:24.363 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.363 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:24.364 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.210109+0000", 2021-03-24T22:00:24.364 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:24.364 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.364 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.365 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:24.365 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.210109+0000", 2021-03-24T22:00:24.365 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2021-03-24T22:00:24.365 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.365 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.366 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:24.366 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.210109+0000", 2021-03-24T22:00:24.366 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0779999999999998e-06 2021-03-24T22:00:24.366 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.367 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.367 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:24.367 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.210111+0000", 2021-03-24T22:00:24.367 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3440000000000001e-06 2021-03-24T22:00:24.367 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.368 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.368 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:24.368 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.210114+0000", 2021-03-24T22:00:24.368 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6359999999999998e-06 2021-03-24T22:00:24.368 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.369 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.369 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:24.369 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.210117+0000", 2021-03-24T22:00:24.369 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0018809759999999999 2021-03-24T22:00:24.370 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.370 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.370 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:24.370 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.211998+0000", 2021-03-24T22:00:24.371 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.8189999999999996e-05 2021-03-24T22:00:24.371 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.371 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.371 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for rw locks", 2021-03-24T22:00:24.371 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.212066+0000", 2021-03-24T22:00:24.372 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.047829064999999997 2021-03-24T22:00:24.372 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.372 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.372 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:24.372 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.259895+0000", 2021-03-24T22:00:24.373 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016446700000000001 2021-03-24T22:00:24.373 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.373 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.373 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:00:24.374 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.260060+0000", 2021-03-24T22:00:24.374 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000205427 2021-03-24T22:00:24.374 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.374 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.374 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:24.375 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:40.260265+0000", 2021-03-24T22:00:24.375 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.050155837000000002 2021-03-24T22:00:24.375 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:24.375 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:24.376 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:24.376 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.376 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.376 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:3112 7.17 7:ec6a359b:::smithi12556470-33:df [sparse-read 0~3008486,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e340)", 2021-03-24T22:00:24.376 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:58:30.211016+0000", 2021-03-24T22:00:24.377 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 113.938182348, 2021-03-24T22:00:24.377 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.99935758200000002, 2021-03-24T22:00:24.377 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:24.377 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:00:24.377 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:00:24.378 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:00:24.378 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:00:24.378 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 3112 2021-03-24T22:00:24.378 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.379 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:24.379 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.379 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:24.379 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211016+0000", 2021-03-24T22:00:24.379 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:24.380 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.380 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.380 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:24.380 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211016+0000", 2021-03-24T22:00:24.381 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2021-03-24T22:00:24.381 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.381 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.381 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:24.381 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211013+0000", 2021-03-24T22:00:24.382 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0642000000000001e-05 2021-03-24T22:00:24.382 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.382 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.382 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:24.382 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211024+0000", 2021-03-24T22:00:24.383 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2675e-05 2021-03-24T22:00:24.383 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.383 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.383 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:24.384 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211037+0000", 2021-03-24T22:00:24.384 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6210000000000004e-06 2021-03-24T22:00:24.384 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.384 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.384 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:24.385 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211044+0000", 2021-03-24T22:00:24.385 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.4183999999999994e-05 2021-03-24T22:00:24.385 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.385 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.386 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:24.386 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211109+0000", 2021-03-24T22:00:24.386 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1910000000000006e-06 2021-03-24T22:00:24.386 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.386 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.387 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for peered", 2021-03-24T22:00:24.387 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211118+0000", 2021-03-24T22:00:24.387 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.99877172599999997 2021-03-24T22:00:24.387 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.387 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.388 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:24.388 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:31.209889+0000", 2021-03-24T22:00:24.388 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000169588 2021-03-24T22:00:24.388 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.389 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.389 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:00:24.389 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:31.210059+0000", 2021-03-24T22:00:24.389 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00031441100000000001 2021-03-24T22:00:24.389 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.390 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.390 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:24.390 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:31.210373+0000", 2021-03-24T22:00:24.390 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.99935758200000002 2021-03-24T22:00:24.391 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:24.391 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:24.391 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:24.391 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.391 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.392 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:3113 7.17 7:ec6a359b:::smithi12556470-33:df [sparse-read 0~3008486] snapc 0=[] ondisk+read+known_if_redirected e340)", 2021-03-24T22:00:24.392 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:58:30.211055+0000", 2021-03-24T22:00:24.392 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 113.938143365, 2021-03-24T22:00:24.392 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.99976562300000005, 2021-03-24T22:00:24.393 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:24.393 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:00:24.393 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:00:24.393 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:00:24.393 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:00:24.394 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 3113 2021-03-24T22:00:24.394 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.394 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:24.394 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.394 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:24.395 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211055+0000", 2021-03-24T22:00:24.395 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:24.395 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.395 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.396 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:24.396 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211055+0000", 2021-03-24T22:00:24.396 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2021-03-24T22:00:24.396 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.396 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.397 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:24.397 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211054+0000", 2021-03-24T22:00:24.397 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8150000000000002e-06 2021-03-24T22:00:24.397 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.398 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.398 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:24.398 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211057+0000", 2021-03-24T22:00:24.398 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.7319999999999993e-06 2021-03-24T22:00:24.398 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.399 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.399 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:24.399 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211067+0000", 2021-03-24T22:00:24.399 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1489999999999998e-06 2021-03-24T22:00:24.399 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.400 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.400 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:24.400 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211070+0000", 2021-03-24T22:00:24.400 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.1780000000000006e-05 2021-03-24T22:00:24.401 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.401 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.401 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:24.401 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211152+0000", 2021-03-24T22:00:24.401 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.8709999999999999e-06 2021-03-24T22:00:24.402 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.402 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.402 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for peered", 2021-03-24T22:00:24.402 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211159+0000", 2021-03-24T22:00:24.403 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.999350081 2021-03-24T22:00:24.403 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.403 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.403 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:24.403 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:31.210509+0000", 2021-03-24T22:00:24.404 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.0376000000000003e-05 2021-03-24T22:00:24.404 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.404 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.404 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:00:24.404 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:31.210599+0000", 2021-03-24T22:00:24.405 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00022137900000000001 2021-03-24T22:00:24.405 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.405 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.405 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:24.406 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:31.210820+0000", 2021-03-24T22:00:24.406 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.99976562300000005 2021-03-24T22:00:24.406 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:24.406 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:24.406 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:24.407 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.407 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.407 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:3114 7.17 7:ec6a359b:::smithi12556470-33:df [sparse-read 0~3008486] snapc 0=[] ondisk+read+known_if_redirected e340)", 2021-03-24T22:00:24.407 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:58:30.211092+0000", 2021-03-24T22:00:24.408 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 113.93810645799999, 2021-03-24T22:00:24.408 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.000019572, 2021-03-24T22:00:24.408 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:24.408 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:00:24.408 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:00:24.409 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:00:24.409 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:00:24.409 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 3114 2021-03-24T22:00:24.409 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.409 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:24.410 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.410 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:24.410 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211092+0000", 2021-03-24T22:00:24.410 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:24.411 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.411 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.411 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:24.411 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211092+0000", 2021-03-24T22:00:24.411 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2021-03-24T22:00:24.412 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.412 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.412 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:24.412 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211091+0000", 2021-03-24T22:00:24.413 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2789999999999999e-06 2021-03-24T22:00:24.413 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.413 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.413 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:24.413 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211095+0000", 2021-03-24T22:00:24.414 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3040000000000001e-06 2021-03-24T22:00:24.414 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.414 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.414 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:24.415 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211101+0000", 2021-03-24T22:00:24.415 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4950000000000002e-06 2021-03-24T22:00:24.415 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.415 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.415 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:24.416 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211105+0000", 2021-03-24T22:00:24.416 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.8471000000000002e-05 2021-03-24T22:00:24.416 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.416 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.417 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:24.417 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211174+0000", 2021-03-24T22:00:24.417 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.9419999999999997e-06 2021-03-24T22:00:24.417 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.417 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.418 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for peered", 2021-03-24T22:00:24.418 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211178+0000", 2021-03-24T22:00:24.418 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.99967128199999999 2021-03-24T22:00:24.418 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.418 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.419 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:24.419 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:31.210849+0000", 2021-03-24T22:00:24.419 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011547 2021-03-24T22:00:24.419 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.420 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.420 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:00:24.420 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:31.210965+0000", 2021-03-24T22:00:24.420 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014683099999999999 2021-03-24T22:00:24.420 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.421 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.421 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:24.421 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:31.211111+0000", 2021-03-24T22:00:24.421 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.000019572 2021-03-24T22:00:24.422 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:24.422 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:24.422 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:24.422 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.422 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.423 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:3180 7.17 7:ec6a359b:::smithi12556470-33:head [copy-from ver 49] snapc ec=[ec,eb,ea,e8,e7,df,d6] ondisk+write+known_if_redirected e343)", 2021-03-24T22:00:24.423 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:58:33.396765+0000", 2021-03-24T22:00:24.423 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 110.752432879, 2021-03-24T22:00:24.423 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.066264190000000001, 2021-03-24T22:00:24.423 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:24.424 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:00:24.424 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:00:24.424 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:00:24.424 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:00:24.425 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 3180 2021-03-24T22:00:24.425 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.425 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:24.425 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.425 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:24.426 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:33.396765+0000", 2021-03-24T22:00:24.426 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:24.426 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.426 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.427 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:24.427 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:33.396765+0000", 2021-03-24T22:00:24.427 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2021-03-24T22:00:24.427 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.427 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.428 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:24.428 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:33.396764+0000", 2021-03-24T22:00:24.428 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.1859999999999992e-06 2021-03-24T22:00:24.428 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.428 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.429 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:24.429 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:33.396772+0000", 2021-03-24T22:00:24.429 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2490000000000007e-06 2021-03-24T22:00:24.429 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.430 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.430 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:24.430 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:33.396781+0000", 2021-03-24T22:00:24.430 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1340000000000001e-06 2021-03-24T22:00:24.430 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.431 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.431 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:24.431 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:33.396787+0000", 2021-03-24T22:00:24.431 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0026378320000000001 2021-03-24T22:00:24.432 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.432 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.432 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:24.432 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:33.399425+0000", 2021-03-24T22:00:24.432 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.6257000000000001e-05 2021-03-24T22:00:24.433 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.433 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.433 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:00:24.433 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:33.399472+0000", 2021-03-24T22:00:24.433 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.018698152999999999 2021-03-24T22:00:24.434 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.434 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.434 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 2", 2021-03-24T22:00:24.434 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:33.418170+0000", 2021-03-24T22:00:24.435 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0061273129999999997 2021-03-24T22:00:24.435 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.435 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.435 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T22:00:24.435 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:33.424297+0000", 2021-03-24T22:00:24.436 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038475431999999997 2021-03-24T22:00:24.436 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.436 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.436 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:24.437 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:33.462772+0000", 2021-03-24T22:00:24.437 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6662000000000006e-05 2021-03-24T22:00:24.437 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.437 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.437 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:00:24.438 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:33.462849+0000", 2021-03-24T22:00:24.438 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018039400000000001 2021-03-24T22:00:24.438 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.438 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.439 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:24.439 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:33.463030+0000", 2021-03-24T22:00:24.439 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.066264190000000001 2021-03-24T22:00:24.439 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:24.439 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:24.440 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:24.440 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.440 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.440 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:3181 7.17 7:ec6a359b:::smithi12556470-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e343)", 2021-03-24T22:00:24.440 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:58:33.396837+0000", 2021-03-24T22:00:24.441 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 110.75236131299999, 2021-03-24T22:00:24.441 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.066584470000000007, 2021-03-24T22:00:24.441 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:24.441 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:00:24.442 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:00:24.442 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:00:24.442 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:00:24.442 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 3181 2021-03-24T22:00:24.442 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.443 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:24.443 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.443 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:24.443 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:33.396837+0000", 2021-03-24T22:00:24.444 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:24.444 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.444 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.444 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:24.444 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:33.396837+0000", 2021-03-24T22:00:24.445 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2021-03-24T22:00:24.445 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.445 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.445 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:24.445 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:33.396836+0000", 2021-03-24T22:00:24.446 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2669999999999995e-06 2021-03-24T22:00:24.446 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.446 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.446 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:24.447 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:33.396845+0000", 2021-03-24T22:00:24.447 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0530000000000007e-06 2021-03-24T22:00:24.447 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.447 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.447 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:24.448 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:33.396853+0000", 2021-03-24T22:00:24.448 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.3079999999999998e-06 2021-03-24T22:00:24.448 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.448 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.449 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:24.449 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:33.396858+0000", 2021-03-24T22:00:24.449 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.002709478 2021-03-24T22:00:24.449 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.449 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.450 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:24.450 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:33.399568+0000", 2021-03-24T22:00:24.450 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8473e-05 2021-03-24T22:00:24.450 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.451 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.451 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for blocked object", 2021-03-24T22:00:24.451 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:33.399596+0000", 2021-03-24T22:00:24.451 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.024044605 2021-03-24T22:00:24.451 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.452 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.452 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:24.452 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:33.423641+0000", 2021-03-24T22:00:24.452 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000114824 2021-03-24T22:00:24.452 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.453 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.453 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for rw locks", 2021-03-24T22:00:24.453 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:33.423756+0000", 2021-03-24T22:00:24.453 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.039325355999999999 2021-03-24T22:00:24.454 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.454 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.454 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:24.454 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:33.463081+0000", 2021-03-24T22:00:24.454 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000109455 2021-03-24T22:00:24.455 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.455 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.455 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:00:24.455 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:33.463190+0000", 2021-03-24T22:00:24.456 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000230955 2021-03-24T22:00:24.456 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.456 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.456 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:24.456 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:33.463421+0000", 2021-03-24T22:00:24.457 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.066584470000000007 2021-03-24T22:00:24.457 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:24.457 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:24.457 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:24.458 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.458 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.458 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:3411 7.11 7:8c1d349c:::smithi12556470-48:head [copy-from ver 432] snapc fd=[fd,fc,fb,fa,f8,f5,f3,f2,f1,ef,ed,e8,e7,d6] ondisk+write+known_if_redirected e360)", 2021-03-24T22:00:24.458 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:58:50.255554+0000", 2021-03-24T22:00:24.458 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 93.893643776999994, 2021-03-24T22:00:24.459 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.053129592000000003, 2021-03-24T22:00:24.459 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:24.459 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:00:24.459 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:00:24.460 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:00:24.460 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:00:24.460 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 3411 2021-03-24T22:00:24.460 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.460 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:24.461 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.461 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:24.461 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:50.255554+0000", 2021-03-24T22:00:24.461 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:24.461 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.462 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.462 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:24.462 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:50.255554+0000", 2021-03-24T22:00:24.462 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999948 2021-03-24T22:00:24.463 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.463 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.463 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:24.463 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:50.255549+0000", 2021-03-24T22:00:24.463 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2198999999999999e-05 2021-03-24T22:00:24.464 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.464 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.464 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:24.464 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:50.255572+0000", 2021-03-24T22:00:24.465 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1501999999999998e-05 2021-03-24T22:00:24.465 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.465 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.465 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:24.465 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:50.255603+0000", 2021-03-24T22:00:24.466 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2313999999999999e-05 2021-03-24T22:00:24.466 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.466 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.466 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:24.467 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:50.255615+0000", 2021-03-24T22:00:24.467 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0358000000000005e-05 2021-03-24T22:00:24.467 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.467 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.467 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:24.468 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:50.255696+0000", 2021-03-24T22:00:24.468 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0014557870000000001 2021-03-24T22:00:24.468 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.468 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.468 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:00:24.469 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:50.257152+0000", 2021-03-24T22:00:24.469 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.035955173999999999 2021-03-24T22:00:24.469 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.469 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.470 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 2,3", 2021-03-24T22:00:24.470 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:50.293107+0000", 2021-03-24T22:00:24.470 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0086550150000000003 2021-03-24T22:00:24.470 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.470 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.471 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T22:00:24.471 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:50.301762+0000", 2021-03-24T22:00:24.471 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0065109670000000003 2021-03-24T22:00:24.471 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.471 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.472 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:24.472 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:50.308273+0000", 2021-03-24T22:00:24.472 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00026044700000000002 2021-03-24T22:00:24.472 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.473 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.473 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:24.473 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:50.308533+0000", 2021-03-24T22:00:24.473 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2230000000000005e-05 2021-03-24T22:00:24.473 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.474 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.474 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:00:24.474 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:50.308605+0000", 2021-03-24T22:00:24.474 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.8609000000000006e-05 2021-03-24T22:00:24.475 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.475 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.475 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:24.475 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:50.308684+0000", 2021-03-24T22:00:24.475 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.053129592000000003 2021-03-24T22:00:24.476 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:24.476 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:24.476 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:24.476 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.477 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.477 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:3412 7.11 7:8c1d349c:::smithi12556470-48:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e360)", 2021-03-24T22:00:24.477 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:58:50.255640+0000", 2021-03-24T22:00:24.477 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 93.893558014999996, 2021-03-24T22:00:24.477 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.053318229000000002, 2021-03-24T22:00:24.478 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:24.478 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:00:24.478 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:00:24.478 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:00:24.478 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:00:24.479 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 3412 2021-03-24T22:00:24.479 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.479 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:24.479 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.480 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:24.480 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:50.255640+0000", 2021-03-24T22:00:24.480 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:24.480 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.480 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.481 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:24.481 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:50.255640+0000", 2021-03-24T22:00:24.481 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2021-03-24T22:00:24.481 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.482 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.482 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:24.482 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:50.255640+0000", 2021-03-24T22:00:24.482 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8569999999999997e-06 2021-03-24T22:00:24.482 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.483 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.483 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:24.483 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:50.255643+0000", 2021-03-24T22:00:24.483 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6959999999999999e-06 2021-03-24T22:00:24.483 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.484 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.484 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:24.484 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:50.255650+0000", 2021-03-24T22:00:24.484 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9529999999999998e-06 2021-03-24T22:00:24.485 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.485 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.485 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:24.485 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:50.255655+0000", 2021-03-24T22:00:24.485 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0016955760000000001 2021-03-24T22:00:24.486 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.486 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.486 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:24.486 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:50.257351+0000", 2021-03-24T22:00:24.487 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0674000000000002e-05 2021-03-24T22:00:24.487 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.487 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.487 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for blocked object", 2021-03-24T22:00:24.487 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:50.257411+0000", 2021-03-24T22:00:24.488 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.041399131999999998 2021-03-24T22:00:24.488 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.488 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.488 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:24.488 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:50.298810+0000", 2021-03-24T22:00:24.489 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000136167 2021-03-24T22:00:24.489 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.489 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.489 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for rw locks", 2021-03-24T22:00:24.490 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:50.298947+0000", 2021-03-24T22:00:24.490 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0097910480000000001 2021-03-24T22:00:24.490 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.490 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.490 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:24.491 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:50.308738+0000", 2021-03-24T22:00:24.491 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000110586 2021-03-24T22:00:24.491 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.491 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.491 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:00:24.492 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:50.308848+0000", 2021-03-24T22:00:24.492 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000110229 2021-03-24T22:00:24.492 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.492 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.493 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:24.493 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:50.308958+0000", 2021-03-24T22:00:24.493 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.053318229000000002 2021-03-24T22:00:24.493 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:24.493 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:24.494 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:24.494 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.494 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.494 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:3847 7.19 7:9b48f357:::smithi12556470-17:head [write 2800046~74050] snapc 125=[125,123,120,11e,11b,11a,119,110,10f,10e,10d,10c,106,105,104,fa,f8,e7] ondisk+write+known_if_redirected e400)", 2021-03-24T22:00:24.495 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:59:30.262363+0000", 2021-03-24T22:00:24.495 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 53.886834776000001, 2021-03-24T22:00:24.495 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.061461771999999998, 2021-03-24T22:00:24.495 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:24.495 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:00:24.496 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:00:24.496 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:00:24.496 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:00:24.496 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 3847 2021-03-24T22:00:24.497 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.497 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:24.497 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.497 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:24.497 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:30.262363+0000", 2021-03-24T22:00:24.498 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:24.498 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.498 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.498 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:24.498 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:30.262363+0000", 2021-03-24T22:00:24.499 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2021-03-24T22:00:24.499 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.499 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.499 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:24.500 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:30.262362+0000", 2021-03-24T22:00:24.500 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8896000000000002e-05 2021-03-24T22:00:24.500 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.500 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.500 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:24.501 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:30.262401+0000", 2021-03-24T22:00:24.501 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7399000000000001e-05 2021-03-24T22:00:24.501 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.501 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.502 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:24.502 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:30.262418+0000", 2021-03-24T22:00:24.502 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.947e-06 2021-03-24T22:00:24.502 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.502 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.503 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:24.503 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:30.262424+0000", 2021-03-24T22:00:24.503 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00061357600000000001 2021-03-24T22:00:24.503 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.504 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.504 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:24.504 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:30.263038+0000", 2021-03-24T22:00:24.504 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000139381 2021-03-24T22:00:24.504 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.505 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.505 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:00:24.505 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:30.263177+0000", 2021-03-24T22:00:24.505 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00028644899999999998 2021-03-24T22:00:24.505 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.506 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.506 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 2", 2021-03-24T22:00:24.506 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:30.263464+0000", 2021-03-24T22:00:24.506 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.02236523 2021-03-24T22:00:24.507 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.507 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.507 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T22:00:24.507 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:30.285829+0000", 2021-03-24T22:00:24.507 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.037735386000000003 2021-03-24T22:00:24.508 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.508 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.508 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:24.508 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:30.323564+0000", 2021-03-24T22:00:24.509 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.7677999999999996e-05 2021-03-24T22:00:24.509 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.509 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.509 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:00:24.509 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:30.323652+0000", 2021-03-24T22:00:24.510 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000173396 2021-03-24T22:00:24.510 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.510 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.510 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:24.510 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:30.323825+0000", 2021-03-24T22:00:24.511 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.061461771999999998 2021-03-24T22:00:24.511 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:24.511 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:24.511 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:24.512 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.512 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.512 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:3848 7.19 7:9b48f357:::smithi12556470-17:head [setxattr _header (58) in=7b,truncate 2874096] snapc 125=[125,123,120,11e,11b,11a,119,110,10f,10e,10d,10c,106,105,104,fa,f8,e7] ondisk+write+known_if_redirected e400)", 2021-03-24T22:00:24.512 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:59:30.262433+0000", 2021-03-24T22:00:24.512 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 53.886765607999997, 2021-03-24T22:00:24.513 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.061735475999999997, 2021-03-24T22:00:24.513 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:24.513 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:00:24.513 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:00:24.513 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:00:24.514 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:00:24.514 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 3848 2021-03-24T22:00:24.514 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.514 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:24.515 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.515 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:24.515 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:30.262433+0000", 2021-03-24T22:00:24.515 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:24.515 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.516 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.516 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:24.516 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:30.262433+0000", 2021-03-24T22:00:24.516 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2021-03-24T22:00:24.517 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.517 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.517 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:24.517 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:30.262432+0000", 2021-03-24T22:00:24.517 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4859999999999999e-06 2021-03-24T22:00:24.518 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.518 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.518 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:24.518 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:30.262436+0000", 2021-03-24T22:00:24.519 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9090000000000006e-06 2021-03-24T22:00:24.519 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.519 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.519 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:24.519 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:30.262445+0000", 2021-03-24T22:00:24.520 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.9750000000000001e-06 2021-03-24T22:00:24.520 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.520 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.520 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:24.520 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:30.262449+0000", 2021-03-24T22:00:24.521 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0015533249999999999 2021-03-24T22:00:24.521 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.521 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.521 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:24.522 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:30.264003+0000", 2021-03-24T22:00:24.522 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013802899999999999 2021-03-24T22:00:24.522 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.522 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.522 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:00:24.523 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:30.264141+0000", 2021-03-24T22:00:24.523 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00025094000000000001 2021-03-24T22:00:24.523 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.523 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.523 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 2", 2021-03-24T22:00:24.524 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:30.264391+0000", 2021-03-24T22:00:24.524 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021446910999999999 2021-03-24T22:00:24.524 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.524 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.525 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T22:00:24.525 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:30.285838+0000", 2021-03-24T22:00:24.525 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038056797000000003 2021-03-24T22:00:24.525 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.525 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.526 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:24.526 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:30.323895+0000", 2021-03-24T22:00:24.526 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3928e-05 2021-03-24T22:00:24.526 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.527 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.527 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:00:24.527 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:30.323979+0000", 2021-03-24T22:00:24.527 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018895899999999999 2021-03-24T22:00:24.527 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.528 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.528 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:24.528 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:30.324168+0000", 2021-03-24T22:00:24.528 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.061735475999999997 2021-03-24T22:00:24.528 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:24.529 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:24.529 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:24.529 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.529 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.530 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:3849 7.19 7:9b48f357:::smithi12556470-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e400)", 2021-03-24T22:00:24.530 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:59:30.262454+0000", 2021-03-24T22:00:24.530 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 53.886744491999998, 2021-03-24T22:00:24.530 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.062253870000000003, 2021-03-24T22:00:24.530 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:24.531 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:00:24.531 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:00:24.531 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:00:24.531 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:00:24.531 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 3849 2021-03-24T22:00:24.532 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.532 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:24.532 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.532 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:24.533 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:30.262454+0000", 2021-03-24T22:00:24.533 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:24.533 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.533 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.533 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:24.534 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:30.262454+0000", 2021-03-24T22:00:24.534 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2021-03-24T22:00:24.534 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.534 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.535 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:24.535 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:30.262453+0000", 2021-03-24T22:00:24.535 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7779999999999999e-06 2021-03-24T22:00:24.535 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.535 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.536 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:24.536 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:30.262456+0000", 2021-03-24T22:00:24.536 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6359999999999999e-06 2021-03-24T22:00:24.536 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.537 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.537 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:24.537 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:30.262462+0000", 2021-03-24T22:00:24.537 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.5459999999999997e-06 2021-03-24T22:00:24.537 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.538 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.538 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:24.538 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:30.262468+0000", 2021-03-24T22:00:24.538 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0023242369999999998 2021-03-24T22:00:24.539 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.539 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.539 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:24.539 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:30.264792+0000", 2021-03-24T22:00:24.539 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000121439 2021-03-24T22:00:24.540 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.540 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.540 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for rw locks", 2021-03-24T22:00:24.540 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:30.264914+0000", 2021-03-24T22:00:24.540 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.059294360999999997 2021-03-24T22:00:24.541 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.541 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.541 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:24.541 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:30.324208+0000", 2021-03-24T22:00:24.542 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012582399999999999 2021-03-24T22:00:24.542 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.542 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.542 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:00:24.542 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:30.324334+0000", 2021-03-24T22:00:24.543 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00037348999999999998 2021-03-24T22:00:24.543 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:24.543 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:24.543 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:24.544 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:30.324708+0000", 2021-03-24T22:00:24.544 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.062253870000000003 2021-03-24T22:00:24.544 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:24.544 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:24.544 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:24.545 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:24.545 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:24.545 INFO:teuthology.orchestra.run.smithi142.stdout:} 2021-03-24T22:00:24.545 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:24.547 INFO:tasks.rados.rados.0.smithi125.stdout:1689: done (2 left) 2021-03-24T22:00:24.547 INFO:tasks.rados.rados.0.smithi125.stdout:1690: done (1 left) 2021-03-24T22:00:24.547 INFO:tasks.rados.rados.0.smithi125.stdout:1691: done (0 left) 2021-03-24T22:00:24.548 INFO:tasks.rados.rados.0.smithi125.stdout:1692: rollback oid 44 current snap is 181 2021-03-24T22:00:24.548 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 44 to 175 2021-03-24T22:00:24.548 INFO:tasks.rados.rados.0.smithi125.stdout:1693: snap_create 2021-03-24T22:00:24.549 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:24 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:00:24.549 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:24 smithi125 conmon[19936]: 00:22.857609+0000 mgr.y (mgr.14144) 497 : cluster [DBG] pgmap v841: 174 pgs: 1 peering, 173 active+clean; 412 MiB data, 1.0 GiB used, 446 GiB / 447 GiB avail; 16 MiB/s rd, 1.7 MiB/s wr, 11 op/s 2021-03-24T22:00:24.549 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:24 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T22:00:24.549 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:24 smithi125 conmon[19936]: -24T22:00:23.287689+0000 mon.a (mon.0) 1366 : cluster [DBG] osdmap e453: 8 total, 8 up, 5 in 2021-03-24T22:00:24.549 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:24 smithi125 conmon[19936]: cluster 2021-03-24T22 2021-03-24T22:00:24.550 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:24 smithi125 conmon[19936]: :00:23.343863+0000 osd.5 (osd. 2021-03-24T22:00:24.550 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:24 smithi125 conmon[19936]: 5) 4 : cluster [DBG] 4.a scrub ok 2021-03-24T22:00:24.550 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:24 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:00:24.551 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:24 smithi125 conmon[24927]: 00:22.857609+0000 mgr.y (mgr.14144) 497 : cluster [DBG] pgmap v841: 174 pgs: 1 peering, 173 active+clean; 412 MiB data, 1.0 GiB used, 446 GiB / 447 GiB avail; 16 MiB/s rd, 1.7 MiB/s wr, 11 op/s 2021-03-24T22:00:24.551 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:24 smithi125 conmon[24927]: cluster 2021-03-24T22:00:24.551 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:24 smithi125 conmon[24927]: 2021-03-24T22:00:23.287689+0000 mon.a (mon.0) 1366 : cluster [DBG] osdmap e453: 8 total, 8 up, 5 in 2021-03-24T22:00:24.551 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:24 smithi125 conmon[24927]: cluster 2021-03-24T22:00:23.343863+0000 osd.5 (osd.5) 4 : cluster [DBG] 4.a scrub ok 2021-03-24T22:00:24.552 INFO:tasks.rados.rados.0.smithi125.stdout:1692: finishing rollback tid 1 to smithi12556470-44 2021-03-24T22:00:24.552 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 44 v 388 (ObjNum 1116860432 snap 21916 seq_num 1114771472) dirty dne 2021-03-24T22:00:24.552 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T22:00:24.566 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:24 smithi142 conmon[20125]: cluster 2021-03-24T22:00: 2021-03-24T22:00:24.566 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:24 smithi142 conmon[20125]: 22.857609+0000 mgr.y (mgr.14144) 497 : cluster [DBG] pgmap v841: 174 pgs: 1 peering, 173 active+clean; 412 MiB data, 1.0 GiB used, 446 GiB / 447 GiB avail; 16 MiB/s rd, 1.7 MiB/s wr, 11 op/s 2021-03-24T22:00:24.566 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:24 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:00:24.567 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:24 smithi142 conmon[20125]: -24T22:00:23.287689+0000 mon.a (mon.0) 1366 : cluster 2021-03-24T22:00:24.567 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:24 smithi142 conmon[20125]: [DBG] osdmap e453: 8 total, 8 up, 5 in 2021-03-24T22:00:24.567 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:24 smithi142 conmon[20125]: cluster 2021-03-24T22:00 2021-03-24T22:00:24.567 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:24 smithi142 conmon[20125]: :23.343863+0000 osd.5 (osd.5) 2021-03-24T22:00:24.568 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:24 smithi142 conmon[20125]: 4 : cluster [DBG] 4.a scrub ok 2021-03-24T22:00:24.573 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T22:00:24.594 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T22:00:24.608 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:24.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:24.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:00:24.616 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T22:00:24.645 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T22:00:24.669 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T22:00:24.695 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T22:00:24.716 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:24.720 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T22:00:24.723 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-24T22:00:24.723 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-24T22:00:24.723 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-24T22:00:24.724 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-24T22:00:24.724 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-24T22:00:24.724 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-24T22:00:24.724 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-24T22:00:24.724 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-24T22:00:24.768 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:24.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:24.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:24.874 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:24.928 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:24.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:24.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:00:25.035 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:25.101 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:25.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:25.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:00:25.184 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.5 dump_ops_in_flight 2021-03-24T22:00:25.208 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:25.245 INFO:tasks.rados.rados.0.smithi125.stdout:1692: done (1 left) 2021-03-24T22:00:25.246 INFO:tasks.rados.rados.0.smithi125.stdout:1693: done (0 left) 2021-03-24T22:00:25.246 INFO:tasks.rados.rados.0.smithi125.stdout:1694: rollback oid 49 current snap is 182 2021-03-24T22:00:25.246 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 49 to 176 2021-03-24T22:00:25.247 INFO:tasks.rados.rados.0.smithi125.stdout:1695: read oid 21 snap 180 2021-03-24T22:00:25.247 INFO:tasks.rados.rados.0.smithi125.stdout:1695: expect (ObjNum 590 snap 177 seq_num 590) 2021-03-24T22:00:25.247 INFO:tasks.rados.rados.0.smithi125.stdout:1696: delete oid 6 current snap is 182 2021-03-24T22:00:25.262 INFO:tasks.rados.rados.0.smithi125.stdout:1694: finishing rollback tid 1 to smithi12556470-49 2021-03-24T22:00:25.263 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 49 v 485 (ObjNum 1116719296 snap 21916 seq_num 1116850352) dirty dne 2021-03-24T22:00:25.264 INFO:tasks.rados.rados.0.smithi125.stdout:1695: expect (ObjNum 590 snap 177 seq_num 590) 2021-03-24T22:00:25.272 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:25.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:25.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:00:25.377 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:25.434 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:25.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:25.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:00:25.485 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T22:00:25.522 INFO:tasks.rados.rados.0.smithi125.stdout:1694: done (2 left) 2021-03-24T22:00:25.522 INFO:tasks.rados.rados.0.smithi125.stdout:1695: done (1 left) 2021-03-24T22:00:25.523 INFO:tasks.rados.rados.0.smithi125.stdout:1696: done (0 left) 2021-03-24T22:00:25.523 INFO:tasks.rados.rados.0.smithi125.stdout:1697: copy_from oid 49 from oid 27 current snap is 182 2021-03-24T22:00:25.523 INFO:tasks.rados.rados.0.smithi125.stdout:1698: read oid 32 snap -1 2021-03-24T22:00:25.523 INFO:tasks.rados.rados.0.smithi125.stdout:1698: expect (ObjNum 427 snap 113 seq_num 427) 2021-03-24T22:00:25.524 INFO:tasks.rados.rados.0.smithi125.stdout:1699: copy_from oid 5 from oid 34 current snap is 182 2021-03-24T22:00:25.524 INFO:tasks.rados.rados.0.smithi125.stdout:1700: copy_from oid 39 from oid 35 current snap is 182 2021-03-24T22:00:25.524 INFO:tasks.rados.rados.0.smithi125.stdout:1701: snap_create 2021-03-24T22:00:25.525 INFO:tasks.rados.rados.0.smithi125.stdout:1700: finishing copy_from to smithi12556470-39 2021-03-24T22:00:25.526 INFO:tasks.rados.rados.0.smithi125.stdout:1700: got expected ENOENT (src dne) 2021-03-24T22:00:25.526 INFO:tasks.rados.rados.0.smithi125.stdout:1700: finishing copy_from racing read to smithi12556470-39 2021-03-24T22:00:25.529 INFO:tasks.rados.rados.0.smithi125.stdout:1698: expect (ObjNum 427 snap 113 seq_num 427) 2021-03-24T22:00:25.538 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:25.545 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:25 smithi125 conmon[19936]: cluster 2021-03-24T22:00:24. 2021-03-24T22:00:25.546 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:25 smithi125 conmon[19936]: 297920+0000 mon.a (mon.0) 1367 : cluster [DBG] osdmap e454: 8 total, 8 up, 5 in 2021-03-24T22:00:25.546 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:25 smithi125 conmon[19936]: cluster 2021-03-24T22:00:24.314886+0000 osd.5 (osd.5) 5 : cluster [DBG] 2021-03-24T22:00:25.546 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:25 smithi125 conmon[19936]: 5.15 scrub ok 2021-03-24T22:00:25.547 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:25 smithi125 conmon[19936]: audit 2021-03-24T22:00:24.529069+0000 mon.c ( 2021-03-24T22:00:25.547 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:25 smithi125 conmon[19936]: mon.2) 71 : audit [DBG] from='client.? v1:172.21.15.125:0/3335430797' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:00:25.547 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:25 smithi125 conmon[19936]: audit 2021-03- 2021-03-24T22:00:25.547 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:25 smithi125 conmon[19936]: 24T22:00:24.859782+0000 mon.a (mon.0) 1368 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "26"}]: dispatch 2021-03-24T22:00:25.548 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:25 smithi125 conmon[19936]: audit 2021-03-24 2021-03-24T22:00:25.548 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:25 smithi125 conmon[19936]: T22:00:25.240637+0000 2021-03-24T22:00:25.548 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:25 smithi125 conmon[19936]: mon.a (mon.0) 1369 : audit 2021-03-24T22:00:25.548 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:25 smithi125 conmon[19936]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "26"}]': finished 2021-03-24T22:00:25.549 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:25 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:00:25.549 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:25 smithi125 conmon[19936]: 03-24T22:00:25 2021-03-24T22:00:25.549 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:25 smithi125 conmon[19936]: .240760+0000 mon.a (mon.0) 1370 : 2021-03-24T22:00:25.549 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:25 smithi125 conmon[19936]: cluster [DBG] osdmap e455: 8 total, 8 up, 5 in 2021-03-24T22:00:25.550 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:25 smithi125 conmon[24927]: cluster 2021-03-24T22 2021-03-24T22:00:25.550 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:25 smithi125 conmon[24927]: :00:24.297920+0000 mon.a (mon.0) 1367 : cluster [DBG] osdmap e454: 8 total, 8 up, 5 in 2021-03-24T22:00:25.551 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:25 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T22:00:25.551 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:25 smithi125 conmon[24927]: 22:00:24.314886+0000 osd.5 (osd.5 2021-03-24T22:00:25.551 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:25 smithi125 conmon[24927]: ) 5 : cluster [DBG] 5.15 scrub ok 2021-03-24T22:00:25.551 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:25 smithi125 conmon[24927]: audit 2021-03-24T 2021-03-24T22:00:25.551 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:25 smithi125 conmon[24927]: 22:00:24.529069+0000 mon.c (mon. 2021-03-24T22:00:25.552 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:25 smithi125 conmon[24927]: 2) 71 : audit [DBG] from='client.? v1:172.21.15.125:0/3335430797' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:00:25.552 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:25 smithi125 conmon[24927]: audit 2021-03-24 2021-03-24T22:00:25.552 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:25 smithi125 conmon[24927]: T22:00:24.859782+0000 mon.a ( 2021-03-24T22:00:25.552 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:25 smithi125 conmon[24927]: mon.0) 1368 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "26"}]: dispatch 2021-03-24T22:00:25.553 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:25 smithi125 conmon[24927]: audit 2021 2021-03-24T22:00:25.553 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:25 smithi125 conmon[24927]: -03-24T22:00:25.240637+0000 2021-03-24T22:00:25.553 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:25 smithi125 conmon[24927]: mon.a (mon.0) 1369 : audit [INF] 2021-03-24T22:00:25.553 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:25 smithi125 conmon[24927]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "26"}]': finished 2021-03-24T22:00:25.553 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:25 smithi125 conmon[24927]: cluster 2021-03-24T22:00:25.554 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:25 smithi125 conmon[24927]: 2021-03-24T22:00:25.240760 2021-03-24T22:00:25.554 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:25 smithi125 conmon[24927]: +0000 mon.a (mon.0) 1370 : cluster [DBG] 2021-03-24T22:00:25.555 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:25 smithi125 conmon[24927]: osdmap e455: 8 total, 8 up, 5 in 2021-03-24T22:00:25.582 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:25.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:25.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:00:25.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:25 smithi142 conmon[20125]: cluster 2021-03-24T22:00:24.297920+0000 mon.a (mon.0) 1367 : cluster [DBG] osdmap e454: 8 total, 8 up, 5 in 2021-03-24T22:00:25.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:25 smithi142 conmon[20125]: cluster 2021-03-24T22:00:24.314886+0000 osd.5 (osd.5) 5 : cluster [DBG] 5.15 scrub ok 2021-03-24T22:00:25.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:25 smithi142 conmon[20125]: audit 2021-03-24T22:00:25.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:25 smithi142 conmon[20125]: 2021-03-24T22:00:24.529069 2021-03-24T22:00:25.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:25 smithi142 conmon[20125]: +0000 mon.c (mon.2) 71 : audit [DBG] 2021-03-24T22:00:25.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:25 smithi142 conmon[20125]: from='client.? v1:172.21.15.125:0/3335430797' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:00:25.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:25 smithi142 conmon[20125]: audit 2021-03-24T22:00:25.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:25 smithi142 conmon[20125]: 2021-03-24T22:00:24 2021-03-24T22:00:25.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:25 smithi142 conmon[20125]: .859782+0000 mon.a (mon.0) 1368 : audit 2021-03-24T22:00:25.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:25 smithi142 conmon[20125]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "26"}]: dispatch 2021-03-24T22:00:25.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:25 smithi142 conmon[20125]: audit 2021-03-24T22:00:25.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:25 smithi142 conmon[20125]: 2021-03-24T22:00:25.240637+0000 mon.a 2021-03-24T22:00:25.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:25 smithi142 conmon[20125]: (mon.0) 1369 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "26"}]': finished 2021-03-24T22:00:25.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:25 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:00:25.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:25 smithi142 conmon[20125]: -24T22:00:25.240760+0000 mon.a 2021-03-24T22:00:25.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:25 smithi142 conmon[20125]: (mon.0) 1370 : cluster [DBG] osdmap e455: 8 total, 8 up, 5 in 2021-03-24T22:00:25.687 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:25.731 INFO:tasks.rados.rados.0.smithi125.stdout:1699: finishing copy_from to smithi12556470-5 2021-03-24T22:00:25.732 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 5 v 543 (ObjNum 553 snap 172 seq_num 553) dirty exists 2021-03-24T22:00:25.732 INFO:tasks.rados.rados.0.smithi125.stdout:1697: finishing copy_from to smithi12556470-49 2021-03-24T22:00:25.732 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 49 v 547 (ObjNum 562 snap 173 seq_num 562) dirty exists 2021-03-24T22:00:25.732 INFO:tasks.rados.rados.0.smithi125.stdout:1697: finishing copy_from racing read to smithi12556470-49 2021-03-24T22:00:25.733 INFO:tasks.rados.rados.0.smithi125.stdout:1699: finishing copy_from racing read to smithi12556470-5 2021-03-24T22:00:25.746 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:25.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:25.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:00:25.853 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:25.910 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:25.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:25.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:00:26.016 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:26.070 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd set noscrub 2021-03-24T22:00:26.071 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:26.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:26.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:00:26.176 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:26.233 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:26.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:26.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:00:26.256 INFO:tasks.rados.rados.0.smithi125.stdout:1697: done (4 left) 2021-03-24T22:00:26.256 INFO:tasks.rados.rados.0.smithi125.stdout:1698: done (3 left) 2021-03-24T22:00:26.257 INFO:tasks.rados.rados.0.smithi125.stdout:1699: done (2 left) 2021-03-24T22:00:26.257 INFO:tasks.rados.rados.0.smithi125.stdout:1700: done (1 left) 2021-03-24T22:00:26.257 INFO:tasks.rados.rados.0.smithi125.stdout:1701: done (0 left) 2021-03-24T22:00:26.257 INFO:tasks.rados.rados.0.smithi125.stdout:1702: rollback oid 26 current snap is 183 2021-03-24T22:00:26.257 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 26 to 176 2021-03-24T22:00:26.258 INFO:tasks.rados.rados.0.smithi125.stdout:1703: delete oid 2 current snap is 183 2021-03-24T22:00:26.262 INFO:tasks.rados.rados.0.smithi125.stdout:1703: done (1 left) 2021-03-24T22:00:26.262 INFO:tasks.rados.rados.0.smithi125.stdout:1704: snap_create 2021-03-24T22:00:26.339 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:26.384 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:26 smithi125 conmon[19936]: cluster 2021-03-24T22:00:24.858266 2021-03-24T22:00:26.389 INFO:tasks.rados.rados.0.smithi125.stdout:1702: finishing rollback tid 1 to smithi12556470-26 2021-03-24T22:00:26.390 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 26 v 436 (ObjNum 1117784448 snap 21916 seq_num 1117784448) dirty dne 2021-03-24T22:00:26.394 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:26.397 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:26 smithi142 conmon[20125]: cluster 2021-03-24T22:00:24. 2021-03-24T22:00:26.398 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:26 smithi142 conmon[20125]: 858266+0000 mgr.y (mgr.14144) 498 : cluster [DBG] pgmap v844: 173 pgs: 1 peering, 172 active+clean; 413 MiB data, 1.0 GiB used, 446 GiB / 447 GiB avail; 3.5 MiB/s rd, 2.2 MiB/s wr, 24 op/s; 54 B/s, 3 objects/s recovering 2021-03-24T22:00:26.398 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:26 smithi142 conmon[20125]: cluster 2021-03-24T22:00:25. 2021-03-24T22:00:26.399 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:26 smithi142 conmon[20125]: 263836+0000 osd.3 (osd.3) 100 : cluster [DBG] 6.c scrub ok 2021-03-24T22:00:26.399 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:26 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:00:26.399 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:26 smithi142 conmon[20125]: T22:00:26.252611+0000 mon.a (mon 2021-03-24T22:00:26.400 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:26 smithi142 conmon[20125]: .0) 1371 : cluster [DBG] osdmap e456: 8 total, 8 up, 5 in 2021-03-24T22:00:26.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:26.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:00:26.504 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:26.558 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:26.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:26.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:00:26.655 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:00:26.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:26 smithi125 conmon[19936]: +0000 mgr.y (mgr.14144) 498 : cluster [DBG] pgmap v844: 173 pgs: 1 peering, 172 active+clean; 413 MiB data, 1.0 GiB used, 446 GiB / 447 GiB avail; 3.5 MiB/s rd, 2.2 MiB/s wr, 24 op/s; 54 B/s, 3 objects/s recovering 2021-03-24T22:00:26.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:26 smithi125 conmon[19936]: cluster 2021-03-24T22:00:25.263836+0000 osd.3 (osd. 2021-03-24T22:00:26.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:26 smithi125 conmon[19936]: 3) 100 : cluster [DBG] 6.c scrub ok 2021-03-24T22:00:26.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:26 smithi125 conmon[19936]: cluster 2021-03-24T22:00 2021-03-24T22:00:26.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:26 smithi125 conmon[19936]: :26.252611+0000 mon.a (mon.0) 1371 : cluster [DBG] osdmap e456: 8 total, 8 up, 5 in 2021-03-24T22:00:26.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:26 smithi125 conmon[24927]: cluster 2021-03-24T22:00:24.858266 2021-03-24T22:00:26.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:26 smithi125 conmon[24927]: +0000 mgr.y (mgr.14144) 498 : cluster [DBG] pgmap v844: 173 pgs: 1 peering, 172 active+clean; 413 MiB data, 1.0 GiB used, 446 GiB / 447 GiB avail; 3.5 MiB/s rd, 2.2 MiB/s wr, 24 op/s; 54 B/s, 3 objects/s recovering 2021-03-24T22:00:26.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:26 smithi125 conmon[24927]: cluster 2021-03-24T22:00 2021-03-24T22:00:26.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:26 smithi125 conmon[24927]: :25.263836+0000 osd.3 (osd.3) 100 : cluster [DBG] 6.c scrub ok 2021-03-24T22:00:26.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:26 smithi125 conmon[24927]: cluster 2021-03-24T22:00:26. 2021-03-24T22:00:26.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:26 smithi125 conmon[24927]: 252611+0000 mon.a (mon.0) 1371 : cluster [DBG] 2021-03-24T22:00:26.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:26 smithi125 conmon[24927]: osdmap e456: 8 total, 8 up, 5 in 2021-03-24T22:00:26.665 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:26.712 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:26.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:26.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:00:26.816 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:26.876 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:26.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:26.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:00:26.980 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:27.034 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:27.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:27.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:00:27.140 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:27.188 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:27.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:27.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:00:27.257 INFO:tasks.rados.rados.0.smithi125.stdout:1702: done (1 left) 2021-03-24T22:00:27.258 INFO:tasks.rados.rados.0.smithi125.stdout:1704: done (0 left) 2021-03-24T22:00:27.258 INFO:tasks.rados.rados.0.smithi125.stdout:1705: write (excl) oid 36 current snap is 184 2021-03-24T22:00:27.259 INFO:tasks.rados.rados.0.smithi125.stdout:1705: seq_num 604 ranges {551428=642283,1694771=456657} 2021-03-24T22:00:27.265 INFO:tasks.rados.rados.0.smithi125.stdout:1705: writing smithi12556470-36 from 551428 to 1193711 tid 1 2021-03-24T22:00:27.272 INFO:tasks.rados.rados.0.smithi125.stdout:1705: writing smithi12556470-36 from 1694771 to 2151428 tid 2 2021-03-24T22:00:27.273 INFO:tasks.rados.rados.0.smithi125.stdout:1706: rollback oid 5 current snap is 184 2021-03-24T22:00:27.273 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 5 to 158 2021-03-24T22:00:27.273 INFO:tasks.rados.rados.0.smithi125.stdout:1707: write oid 16 current snap is 184 2021-03-24T22:00:27.274 INFO:tasks.rados.rados.0.smithi125.stdout:1707: seq_num 605 ranges {568235=717532,1753253=414982} 2021-03-24T22:00:27.279 INFO:tasks.rados.rados.0.smithi125.stdout:1707: writing smithi12556470-16 from 568235 to 1285767 tid 1 2021-03-24T22:00:27.286 INFO:tasks.rados.rados.0.smithi125.stdout:1707: writing smithi12556470-16 from 1753253 to 2168235 tid 2 2021-03-24T22:00:27.286 INFO:tasks.rados.rados.0.smithi125.stdout:1708: snap_remove snap 172 2021-03-24T22:00:27.293 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:27.346 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:27.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:27.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:00:27.388 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-24T22:00:27.395 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2021-03-24T22:00:27.453 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2021-03-24T22:00:27.453 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T22:00:27.453 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:27.464 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:27 smithi142 conmon[20125]: cluster 2021-03-24T22:00:26.193725+0000 osd.2 (osd.2) 77 : cluster [DBG] 7.19 deep-scrub ok 2021-03-24T22:00:27.464 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:27 smithi142 conmon[20125]: 2021-03-24T22:00:27.464 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:27 smithi142 conmon[20125]: cluster 2021-03-24T22:00:26.392216+0000 osd.5 (osd.5) 6 : cluster [DBG] 5.16 deep-scrub ok 2021-03-24T22:00:27.465 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:27 smithi142 conmon[20125]: cluster 2021-03-24T22:00:27.253555+0000 mon.a (mon.0) 1372 : cluster [DBG] osdmap e457: 8 total, 8 up, 5 in 2021-03-24T22:00:27.465 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:27 smithi142 conmon[20125]: 2021-03-24T22:00:27.497 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:27 smithi125 conmon[19936]: cluster 2021-03-24T22:00:27.497 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:27 smithi125 conmon[19936]: 2021-03- 2021-03-24T22:00:27.497 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:27 smithi125 conmon[19936]: 24T22:00:26.193725+0000 osd.2 (osd.2) 77 : cluster [DBG] 7.19 deep-scrub ok 2021-03-24T22:00:27.498 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:27 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T22:00:27.498 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:27 smithi125 conmon[19936]: -24T22:00:26.392216+0000 2021-03-24T22:00:27.498 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:27 smithi125 conmon[19936]: osd.5 (osd.5) 6 : cluster 2021-03-24T22:00:27.498 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:27 smithi125 conmon[19936]: [DBG] 5.16 deep-scrub ok 2021-03-24T22:00:27.499 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:27 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:00:27.499 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:27 smithi125 conmon[19936]: 00:27.253555+0000 mon.a (mon. 2021-03-24T22:00:27.499 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:27 smithi125 conmon[19936]: 0) 1372 : cluster [DBG] osdmap e457: 8 total, 8 up, 5 in 2021-03-24T22:00:27.500 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:27 smithi125 conmon[24927]: cluster 2021-03-24T22:00:27.500 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:27 smithi125 conmon[24927]: 2021-03-24T22 2021-03-24T22:00:27.500 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:27 smithi125 conmon[24927]: :00:26.193725+0000 osd.2 (osd.2) 77 : cluster [DBG] 7.19 deep-scrub ok 2021-03-24T22:00:27.500 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:27 smithi125 conmon[24927]: cluster 2021-03-24T22:00:27.501 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:27 smithi125 conmon[24927]: 2021-03-24T22:00:26. 2021-03-24T22:00:27.501 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:27 smithi125 conmon[24927]: 392216+0000 osd.5 (osd.5) 6 : cluster 2021-03-24T22:00:27.501 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:27 smithi125 conmon[24927]: [DBG] 5.16 deep-scrub ok 2021-03-24T22:00:27.501 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:27 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:00:27.502 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:27 smithi125 conmon[24927]: 03-24T22:00:27.253555 2021-03-24T22:00:27.502 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:27 smithi125 conmon[24927]: +0000 mon.a (mon.0) 1372 : 2021-03-24T22:00:27.502 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:27 smithi125 conmon[24927]: cluster [DBG] osdmap e457: 8 total, 8 up, 5 in 2021-03-24T22:00:27.508 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:27.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:27.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:00:27.613 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:27.661 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:27.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:27.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:00:27.768 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:27.825 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:27.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:27.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:27.930 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:27.988 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:27.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:27.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:28.094 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:28.152 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:28.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:28.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:00:28.257 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:28.275 INFO:tasks.rados.rados.0.smithi125.stdout:1708: done (3 left) 2021-03-24T22:00:28.276 INFO:tasks.rados.rados.0.smithi125.stdout:1709: snap_remove snap 159 2021-03-24T22:00:28.312 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:28.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:28.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:00:28.357 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.5 dump_blocked_ops 2021-03-24T22:00:28.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:28 smithi125 conmon[19936]: cluster 2021-03-24T22:00:26.859188 2021-03-24T22:00:28.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:28 smithi125 conmon[19936]: +0000 mgr.y (mgr.14144) 499 : cluster [DBG] pgmap v847: 173 pgs: 2 activating, 1 peering, 170 active+clean; 416 MiB data, 1.0 GiB used, 446 GiB / 447 GiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 22 op/s; 54 B/s, 3 objects/s recovering 2021-03-24T22:00:28.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:28 smithi125 conmon[19936]: cluster 2021-03-24T22:00:28.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:28 smithi125 conmon[19936]: 2021-03-24T22:00:26.883390+0000 osd.4 (osd.4 2021-03-24T22:00:28.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:28 smithi125 conmon[19936]: ) 107 : cluster [DBG] 6.1e scrub ok 2021-03-24T22:00:28.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:28 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:00:28.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:28 smithi125 conmon[19936]: -03-24T22:00:27.169705+0000 osd.2 (osd. 2021-03-24T22:00:28.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:28 smithi125 conmon[19936]: 2) 78 : cluster [DBG] 7.17 scrub ok 2021-03-24T22:00:28.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:28 smithi125 conmon[19936]: audit 2021-03-24T22:00 2021-03-24T22:00:28.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:28 smithi125 conmon[19936]: :27.361835+0000 mgr.y (mgr.14144) 2021-03-24T22:00:28.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:28 smithi125 conmon[19936]: 500 : audit [DBG] from='client.15366 v1:172.21.15.125:0/181435277' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T22:00:28.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:28 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:00:28.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:28 smithi125 conmon[19936]: T22:00:28.261182+0000 mon.a ( 2021-03-24T22:00:28.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:28 smithi125 conmon[19936]: mon.0) 1373 : cluster [DBG] osdmap e458: 8 total, 8 up, 5 in 2021-03-24T22:00:28.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:28 smithi125 conmon[24927]: cluster 2021-03-24T22:00:26.859188+0000 2021-03-24T22:00:28.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:28 smithi125 conmon[24927]: mgr.y (mgr.14144) 499 : cluster [DBG] pgmap v847: 173 pgs: 2 activating, 1 peering, 170 active+clean; 416 MiB data, 1.0 GiB used, 446 GiB / 447 GiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 22 op/s; 54 B/s, 3 objects/s recovering 2021-03-24T22:00:28.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:28 smithi125 conmon[24927]: cluster 2021-03-24T22:00:28.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:28 smithi125 conmon[24927]: 2021-03-24T22:00:26.883390+0000 osd.4 (osd.4) 107 : cluster 2021-03-24T22:00:28.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:28 smithi125 conmon[24927]: [DBG] 6.1e scrub ok 2021-03-24T22:00:28.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:28 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:00:28.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:28 smithi125 conmon[24927]: 24T22:00:27.169705+0000 osd.2 (osd. 2021-03-24T22:00:28.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:28 smithi125 conmon[24927]: 2) 78 : cluster [DBG] 7.17 scrub ok 2021-03-24T22:00:28.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:28 smithi125 conmon[24927]: audit 2021- 2021-03-24T22:00:28.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:28 smithi125 conmon[24927]: 03-24T22:00:27.361835+0000 mgr.y ( 2021-03-24T22:00:28.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:28 smithi125 conmon[24927]: mgr.14144) 500 : audit [DBG] from='client.15366 v1:172.21.15.125:0/181435277' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T22:00:28.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:28 smithi125 conmon[24927]: cluster 2021-03-24T22:00:28.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:28 smithi125 conmon[24927]: 2021-03-24 2021-03-24T22:00:28.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:28 smithi125 conmon[24927]: T22:00:28.261182+0000 mon.a (mon. 2021-03-24T22:00:28.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:28 smithi125 conmon[24927]: 0) 1373 : cluster [DBG] osdmap e458: 8 total, 8 up, 5 in 2021-03-24T22:00:28.419 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:28.472 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:28.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:28.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:00:28.578 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:28.595 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-24T22:00:28.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:28 smithi142 conmon[20125]: cluster 2021-03-24T22:00:26.859188+0000 mgr.y (mgr.14144) 499 : 2021-03-24T22:00:28.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:28 smithi142 conmon[20125]: cluster [DBG] pgmap v847: 173 pgs: 2 activating, 1 peering, 170 active+clean; 416 MiB data, 1.0 GiB used, 446 GiB / 447 GiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 22 op/s; 54 B/s, 3 objects/s recovering 2021-03-24T22:00:28.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:28 smithi142 conmon[20125]: cluster 2021-03-24T22:00:26.883390+0000 osd.4 (osd. 2021-03-24T22:00:28.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:28 smithi142 conmon[20125]: 4) 107 : cluster [DBG] 6.1e scrub ok 2021-03-24T22:00:28.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:28 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:00:28.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:28 smithi142 conmon[20125]: -24T22:00:27.169705+0000 osd.2 (osd 2021-03-24T22:00:28.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:28 smithi142 conmon[20125]: .2) 78 : cluster [DBG] 7.17 scrub ok 2021-03-24T22:00:28.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:28 smithi142 conmon[20125]: 2021-03-24T22:00:28.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:28 smithi142 conmon[20125]: audit 2021-03-24T22 2021-03-24T22:00:28.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:28 smithi142 conmon[20125]: :00:27.361835+0000 mgr.y 2021-03-24T22:00:28.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:28 smithi142 conmon[20125]: (mgr.14144) 500 : audit 2021-03-24T22:00:28.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:28 smithi142 conmon[20125]: [DBG] from='client.15366 v1:172.21.15.125:0/181435277' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T22:00:28.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:28 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:00:28.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:28 smithi142 conmon[20125]: 24T22:00:28.261182 2021-03-24T22:00:28.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:28 smithi142 conmon[20125]: +0000 mon.a (mon.0) 1373 2021-03-24T22:00:28.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:28 smithi142 conmon[20125]: : cluster [DBG] osdmap e458: 8 total, 8 up, 5 in 2021-03-24T22:00:28.638 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:28.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:28.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:28.657 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T22:00:28.745 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:28.798 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:28.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:28.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:00:28.903 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:28.965 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:28.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:28.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:00:29.072 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:29.126 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:29.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:29.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:00:29.234 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:29.269 INFO:tasks.rados.rados.0.smithi125.stdout:1709: done (3 left) 2021-03-24T22:00:29.270 INFO:tasks.rados.rados.0.smithi125.stdout:1710: write (excl) oid 2 current snap is 184 2021-03-24T22:00:29.270 INFO:tasks.rados.rados.0.smithi125.stdout:1710: seq_num 606 ranges {585042=792781,2095382=89660} 2021-03-24T22:00:29.272 INFO:tasks.rados.rados.0.smithi125.stdout:1710: writing smithi12556470-2 from 585042 to 1377823 tid 1 2021-03-24T22:00:29.278 INFO:tasks.rados.rados.0.smithi125.stdout:1710: writing smithi12556470-2 from 2095382 to 2185042 tid 2 2021-03-24T22:00:29.278 INFO:tasks.rados.rados.0.smithi125.stdout:1711: read oid 4 snap -1 2021-03-24T22:00:29.278 INFO:tasks.rados.rados.0.smithi125.stdout:1711: expect (ObjNum 544 snap 168 seq_num 544) 2021-03-24T22:00:29.279 INFO:tasks.rados.rados.0.smithi125.stdout:1712: snap_create 2021-03-24T22:00:29.279 INFO:tasks.rados.rados.0.smithi125.stdout:1705: finishing write tid 1 to smithi12556470-36 2021-03-24T22:00:29.279 INFO:tasks.rados.rados.0.smithi125.stdout:1706: finishing rollback tid 0 to smithi12556470-5 2021-03-24T22:00:29.279 INFO:tasks.rados.rados.0.smithi125.stdout:1706: finishing rollback tid 1 to smithi12556470-5 2021-03-24T22:00:29.280 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 5 v 545 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-03-24T22:00:29.280 INFO:tasks.rados.rados.0.smithi125.stdout:1705: finishing write tid 2 to smithi12556470-36 2021-03-24T22:00:29.280 INFO:tasks.rados.rados.0.smithi125.stdout:1705: finishing write tid 3 to smithi12556470-36 2021-03-24T22:00:29.280 INFO:tasks.rados.rados.0.smithi125.stdout:1705: finishing write tid 4 to smithi12556470-36 2021-03-24T22:00:29.280 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 36 v 571 (ObjNum 604 snap 184 seq_num 604) dirty exists 2021-03-24T22:00:29.281 INFO:tasks.rados.rados.0.smithi125.stdout:1705: left oid 36 (ObjNum 604 snap 184 seq_num 604) 2021-03-24T22:00:29.281 INFO:tasks.rados.rados.0.smithi125.stdout:1707: finishing write tid 1 to smithi12556470-16 2021-03-24T22:00:29.281 INFO:tasks.rados.rados.0.smithi125.stdout:1707: finishing write tid 2 to smithi12556470-16 2021-03-24T22:00:29.281 INFO:tasks.rados.rados.0.smithi125.stdout:1707: finishing write tid 3 to smithi12556470-16 2021-03-24T22:00:29.281 INFO:tasks.rados.rados.0.smithi125.stdout:1707: finishing write tid 4 to smithi12556470-16 2021-03-24T22:00:29.282 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 16 v 392 (ObjNum 605 snap 184 seq_num 605) dirty exists 2021-03-24T22:00:29.283 INFO:tasks.rados.rados.0.smithi125.stdout:1707: left oid 16 (ObjNum 605 snap 184 seq_num 605) 2021-03-24T22:00:29.292 INFO:tasks.rados.rados.0.smithi125.stdout:1711: expect (ObjNum 544 snap 168 seq_num 544) 2021-03-24T22:00:29.303 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:29.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:29.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:00:29.361 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:29 smithi125 conmon[24927]: cluster 2021-03-24T22:00:28.289238 2021-03-24T22:00:29.362 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:29 smithi125 conmon[24927]: +0000 osd.3 (osd.3) 101 : cluster [DBG] 5.13 scrub ok 2021-03-24T22:00:29.362 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:29 smithi125 conmon[24927]: audit 2021- 2021-03-24T22:00:29.362 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:29 smithi125 conmon[24927]: 03-24T22:00:28.860767+0000 mon.a (mon.0) 1374 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "25"}]: dispatch 2021-03-24T22:00:29.363 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:29 smithi125 conmon[24927]: audit 2021- 2021-03-24T22:00:29.363 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:29 smithi125 conmon[24927]: 03-24T22:00:29.259833+0000 mon.a ( 2021-03-24T22:00:29.363 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:29 smithi125 conmon[24927]: mon.0) 1375 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "25"}]': finished 2021-03-24T22:00:29.364 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:29 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:00:29.364 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:29 smithi125 conmon[24927]: 24T22:00:29.259910+0000 mon.a (mon. 2021-03-24T22:00:29.364 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:29 smithi125 conmon[24927]: 0) 1376 : cluster [DBG] osdmap e459: 8 total, 8 up, 5 in 2021-03-24T22:00:29.365 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:29 smithi125 conmon[19936]: cluster 2021-03-24T22:00:28.289238 2021-03-24T22:00:29.365 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:29 smithi125 conmon[19936]: +0000 osd.3 (osd.3) 101 : cluster [DBG] 5.13 scrub ok 2021-03-24T22:00:29.365 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:29 smithi125 conmon[19936]: audit 2021-03-24T22:00:28.860767+0000 mon.a (mon.0 2021-03-24T22:00:29.365 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:29 smithi125 conmon[19936]: ) 1374 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "25"}]: dispatch 2021-03-24T22:00:29.366 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:29 smithi125 conmon[19936]: audit 2021-03-24T22:00:29.259833+0000 mon.a 2021-03-24T22:00:29.366 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:29 smithi125 conmon[19936]: (mon.0) 1375 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "25"}]': finished 2021-03-24T22:00:29.366 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:29 smithi125 conmon[19936]: cluster 2021-03-24T22:00:29.367 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:29 smithi125 conmon[19936]: 2021-03-24T 2021-03-24T22:00:29.367 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:29 smithi125 conmon[19936]: 22:00:29.259910+0000 mon.a (mon.0) 1376 2021-03-24T22:00:29.368 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:29 smithi125 conmon[19936]: : cluster [DBG] osdmap e459: 8 total, 8 up, 5 in 2021-03-24T22:00:29.387 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":848,"stamp":"2021-03-24T22:00:27.250237+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":436624118,"num_objects":502,"num_object_clones":138,"num_object_copies":1506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":502,"num_whiteouts":15,"num_read":3006,"num_read_kb":2050647,"num_write":1948,"num_write_kb":558583,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1107,"num_bytes_recovered":465670454,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9844,"ondisk_log_size":9844,"up":516,"acting":516,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":521,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":1068908,"kb_used_data":912708,"kb_used_omap":33,"kb_used_meta":156126,"kb_avail":467656852,"statfs":{"total":479975178240,"available":478880616448,"internally_reserved":0,"allocated":934612992,"data_stored":1248391336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33936,"internal_metadata":159873904},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4229394,"num_objects":-6,"num_object_clones":-1,"num_object_copies":-18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6,"num_whiteouts":-2,"num_read":7,"num_read_kb":9114,"num_write":22,"num_write_kb":12350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-13,"num_bytes_recovered":-1204,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.965205"},"pg_stats":[{"pgid":"3.29","version":"0'0","reported_seq":"79","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.296180+0000","last_change":"2021-03-24T21:59:14.374618+0000","last_active":"2021-03-24T22:00:22.296180+0000","last_peered":"2021-03-24T22:00:22.296180+0000","last_clean":"2021-03-24T22:00:22.296180+0000","last_became_active":"2021-03-24T21:59:14.374095+0000","last_became_peered":"2021-03-24T21:59:14.374095+0000","last_unstale":"2021-03-24T22:00:22.296180+0000","last_undegraded":"2021-03-24T22:00:22.296180+0000","last_fullsized":"2021-03-24T22:00:22.296180+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:55.592795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:55.592795+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":"32","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-24T22:00:25.253633+0000","last_change":"2021-03-24T22:00:04.289735+0000","last_active":"2021-03-24T22:00:25.253633+0000","last_peered":"2021-03-24T22:00:25.253633+0000","last_clean":"2021-03-24T22:00:25.253633+0000","last_became_active":"2021-03-24T22:00:04.286965+0000","last_became_peered":"2021-03-24T22:00:04.286965+0000","last_unstale":"2021-03-24T22:00:25.253633+0000","last_undegraded":"2021-03-24T22:00:25.253633+0000","last_fullsized":"2021-03-24T22:00:25.253633+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:39.439170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:39.439170+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"89'2","reported_seq":"120","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.291017+0000","last_change":"2021-03-24T21:59:14.374798+0000","last_active":"2021-03-24T22:00:22.291017+0000","last_peered":"2021-03-24T22:00:22.291017+0000","last_clean":"2021-03-24T22:00:22.291017+0000","last_became_active":"2021-03-24T21:59:14.374418+0000","last_became_peered":"2021-03-24T21:59:14.374418+0000","last_unstale":"2021-03-24T22:00:22.291017+0000","last_undegraded":"2021-03-24T22:00:22.291017+0000","last_fullsized":"2021-03-24T22:00:22.291017+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":6,"last_scrub":"89'2","last_scrub_stamp":"2021-03-24T21:57:58.491830+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2021-03-24T21:57:58.491830+0000","last_clean_scrub_stamp":"2021-03-24T21:57:58.491830+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":179,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":"448","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.283664+0000","last_change":"2021-03-24T22:00:04.297849+0000","last_active":"2021-03-24T22:00:22.283664+0000","last_peered":"2021-03-24T22:00:22.283664+0000","last_clean":"2021-03-24T22:00:22.283664+0000","last_became_active":"2021-03-24T22:00:04.297076+0000","last_became_peered":"2021-03-24T22:00:04.297076+0000","last_unstale":"2021-03-24T22:00:22.283664+0000","last_undegraded":"2021-03-24T22:00:22.283664+0000","last_fullsized":"2021-03-24T22:00:22.283664+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:33.533840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:33.533840+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":"398","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.296100+0000","last_change":"2021-03-24T22:00:04.295111+0000","last_active":"2021-03-24T22:00:22.296100+0000","last_peered":"2021-03-24T22:00:22.296100+0000","last_clean":"2021-03-24T22:00:22.296100+0000","last_became_active":"2021-03-24T22:00:04.294460+0000","last_became_peered":"2021-03-24T22:00:04.294460+0000","last_unstale":"2021-03-24T22:00:22.296100+0000","last_undegraded":"2021-03-24T22:00:22.296100+0000","last_fullsized":"2021-03-24T22:00:22.296100+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:38.304792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:38.304792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"61'1","reported_seq":"165","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.277332+0000","last_change":"2021-03-24T22:00:04.300541+0000","last_active":"2021-03-24T22:00:22.277332+0000","last_peered":"2021-03-24T22:00:22.277332+0000","last_clean":"2021-03-24T22:00:22.277332+0000","last_became_active":"2021-03-24T22:00:04.298351+0000","last_became_peered":"2021-03-24T22:00:04.298351+0000","last_unstale":"2021-03-24T22:00:22.277332+0000","last_undegraded":"2021-03-24T22:00:22.277332+0000","last_fullsized":"2021-03-24T22:00:22.277332+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":6,"last_scrub":"61'1","last_scrub_stamp":"2021-03-24T21:57:12.046206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:12.046206+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":437,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":"430","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.287016+0000","last_change":"2021-03-24T21:59:14.370910+0000","last_active":"2021-03-24T22:00:22.287016+0000","last_peered":"2021-03-24T22:00:22.287016+0000","last_clean":"2021-03-24T22:00:22.287016+0000","last_became_active":"2021-03-24T21:59:14.370506+0000","last_became_peered":"2021-03-24T21:59:14.370506+0000","last_unstale":"2021-03-24T22:00:22.287016+0000","last_undegraded":"2021-03-24T22:00:22.287016+0000","last_fullsized":"2021-03-24T22:00:22.287016+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:54.381668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:54.381668+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.23","version":"450'307","reported_seq":"1507","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.288584+0000","last_change":"2021-03-24T22:00:22.288584+0000","last_active":"2021-03-24T22:00:22.288584+0000","last_peered":"2021-03-24T22:00:22.288584+0000","last_clean":"2021-03-24T22:00:22.288584+0000","last_became_active":"2021-03-24T22:00:03.287690+0000","last_became_peered":"2021-03-24T22:00:03.287690+0000","last_unstale":"2021-03-24T22:00:22.288584+0000","last_undegraded":"2021-03-24T22:00:22.288584+0000","last_fullsized":"2021-03-24T22:00:22.288584+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":307,"ondisk_log_size":307,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10808648,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":111,"num_read_kb":127739,"num_write":77,"num_write_kb":27732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":5377656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"e7","length":"1"},{"start":"f8","length":"1"},{"start":"10c","length":"1"},{"start":"10f","length":"1"},{"start":"126","length":"1"},{"start":"128","length":"1"},{"start":"145","length":"1"}]},{"pgid":"3.26","version":"0'0","reported_seq":"418","reported_epoch":"451","state":"active+clean","last_fresh":"2021-03-24T22:00:21.279437+0000","last_change":"2021-03-24T22:00:04.268826+0000","last_active":"2021-03-24T22:00:21.279437+0000","last_peered":"2021-03-24T22:00:21.279437+0000","last_clean":"2021-03-24T22:00:21.279437+0000","last_became_active":"2021-03-24T22:00:04.268509+0000","last_became_peered":"2021-03-24T22:00:04.268509+0000","last_unstale":"2021-03-24T22:00:21.279437+0000","last_undegraded":"2021-03-24T22:00:21.279437+0000","last_fullsized":"2021-03-24T22:00:21.279437+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:42.879245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:59:42.879245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.22","version":"451'101","reported_seq":"605","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.289109+0000","last_change":"2021-03-24T22:00:22.292775+0000","last_active":"2021-03-24T22:00:22.289109+0000","last_peered":"2021-03-24T22:00:22.289109+0000","last_clean":"2021-03-24T22:00:22.289109+0000","last_became_active":"2021-03-24T22:00:03.287057+0000","last_became_peered":"2021-03-24T22:00:03.287057+0000","last_unstale":"2021-03-24T22:00:22.289109+0000","last_undegraded":"2021-03-24T22:00:22.289109+0000","last_fullsized":"2021-03-24T22:00:22.289109+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":101,"ondisk_log_size":101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6706158,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":8521,"num_write":25,"num_write_kb":6891,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":7075919,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e7","length":"1"},{"start":"f8","length":"1"},{"start":"10c","length":"1"},{"start":"10f","length":"1"},{"start":"126","length":"1"},{"start":"128","length":"1"},{"start":"145","length":"1"}]},{"pgid":"4.c","version":"69'15","reported_seq":"531","reported_epoch":"451","state":"active+clean","last_fresh":"2021-03-24T22:00:21.275780+0000","last_change":"2021-03-24T22:00:20.270042+0000","last_active":"2021-03-24T22:00:21.275780+0000","last_peered":"2021-03-24T22:00:21.275780+0000","last_clean":"2021-03-24T22:00:21.275780+0000","last_became_active":"2021-03-24T22:00:20.269552+0000","last_became_peered":"2021-03-24T22:00:20.269552+0000","last_unstale":"2021-03-24T22:00:21.275780+0000","last_undegraded":"2021-03-24T22:00:21.275780+0000","last_fullsized":"2021-03-24T22:00:21.275780+0000","mapping_epoch":416,"log_start":"69'15","ondisk_log_start":"69'15","created":62,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2021-03-24T21:58:27.722546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:58:27.722546+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":15,"num_object_clones":0,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":49,"num_read_kb":29,"num_write":29,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":1624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"418","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.287675+0000","last_change":"2021-03-24T21:59:17.264251+0000","last_active":"2021-03-24T22:00:22.287675+0000","last_peered":"2021-03-24T22:00:22.287675+0000","last_clean":"2021-03-24T22:00:22.287675+0000","last_became_active":"2021-03-24T21:51:45.375830+0000","last_became_peered":"2021-03-24T21:51:45.375830+0000","last_unstale":"2021-03-24T22:00:22.287675+0000","last_undegraded":"2021-03-24T22:00:22.287675+0000","last_fullsized":"2021-03-24T22:00:22.287675+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:17.264089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:59:17.264089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"434'9","reported_seq":"560","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-24T22:00:25.256710+0000","last_change":"2021-03-24T22:00:04.307080+0000","last_active":"2021-03-24T22:00:25.256710+0000","last_peered":"2021-03-24T22:00:25.256710+0000","last_clean":"2021-03-24T22:00:25.256710+0000","last_became_active":"2021-03-24T22:00:04.261622+0000","last_became_peered":"2021-03-24T22:00:04.261622+0000","last_unstale":"2021-03-24T22:00:25.256710+0000","last_undegraded":"2021-03-24T22:00:25.256710+0000","last_fullsized":"2021-03-24T22:00:25.256710+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2021-03-24T21:57:54.241842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:57:54.241842+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"451","reported_epoch":"451","state":"active+clean","last_fresh":"2021-03-24T22:00:21.276843+0000","last_change":"2021-03-24T22:00:13.834380+0000","last_active":"2021-03-24T22:00:21.276843+0000","last_peered":"2021-03-24T22:00:21.276843+0000","last_clean":"2021-03-24T22:00:21.276843+0000","last_became_active":"2021-03-24T22:00:04.260375+0000","last_became_peered":"2021-03-24T22:00:04.260375+0000","last_unstale":"2021-03-24T22:00:21.276843+0000","last_undegraded":"2021-03-24T22:00:21.276843+0000","last_fullsized":"2021-03-24T22:00:21.276843+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:13.834281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:00:13.834281+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"452'249","reported_seq":"1396","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.298919+0000","last_change":"2021-03-24T22:00:22.298919+0000","last_active":"2021-03-24T22:00:22.298919+0000","last_peered":"2021-03-24T22:00:22.298919+0000","last_clean":"2021-03-24T22:00:22.298919+0000","last_became_active":"2021-03-24T21:57:26.178185+0000","last_became_peered":"2021-03-24T21:57:26.178185+0000","last_unstale":"2021-03-24T22:00:22.298919+0000","last_undegraded":"2021-03-24T22:00:22.298919+0000","last_fullsized":"2021-03-24T22:00:22.298919+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'171","last_scrub_stamp":"2021-03-24T21:59:08.017874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:59:08.017874+0000","log_size":249,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14068419,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":55,"num_read_kb":68772,"num_write":13,"num_write_kb":4675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e7","length":"1"},{"start":"f8","length":"1"},{"start":"10c","length":"1"},{"start":"10f","length":"1"},{"start":"126","length":"1"},{"start":"128","length":"1"},{"start":"145","length":"1"}]},{"pgid":"4.d","version":"69'21","reported_seq":"505","reported_epoch":"451","state":"active+clean","last_fresh":"2021-03-24T22:00:21.276645+0000","last_change":"2021-03-24T22:00:16.255853+0000","last_active":"2021-03-24T22:00:21.276645+0000","last_peered":"2021-03-24T22:00:21.276645+0000","last_clean":"2021-03-24T22:00:21.276645+0000","last_became_active":"2021-03-24T22:00:16.255377+0000","last_became_peered":"2021-03-24T22:00:16.255377+0000","last_unstale":"2021-03-24T22:00:21.276645+0000","last_undegraded":"2021-03-24T22:00:21.276645+0000","last_fullsized":"2021-03-24T22:00:21.276645+0000","mapping_epoch":108,"log_start":"69'21","ondisk_log_start":"69'21","created":62,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'21","last_scrub_stamp":"2021-03-24T21:59:05.674550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:59:05.674550+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1578,"num_objects":21,"num_object_clones":0,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":77,"num_read_kb":42,"num_write":40,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"395","reported_epoch":"451","state":"active+clean","last_fresh":"2021-03-24T22:00:21.275545+0000","last_change":"2021-03-24T22:00:03.289409+0000","last_active":"2021-03-24T22:00:21.275545+0000","last_peered":"2021-03-24T22:00:21.275545+0000","last_clean":"2021-03-24T22:00:21.275545+0000","last_became_active":"2021-03-24T22:00:03.289062+0000","last_became_peered":"2021-03-24T22:00:03.289062+0000","last_unstale":"2021-03-24T22:00:21.275545+0000","last_undegraded":"2021-03-24T22:00:21.275545+0000","last_fullsized":"2021-03-24T22:00:21.275545+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:50.522344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:55:01.169461+0000","last_clean_scrub_stamp":"2021-03-24T21:57:50.522344+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"389","reported_epoch":"451","state":"active+clean","last_fresh":"2021-03-24T22:00:21.275662+0000","last_change":"2021-03-24T22:00:19.934863+0000","last_active":"2021-03-24T22:00:21.275662+0000","last_peered":"2021-03-24T22:00:21.275662+0000","last_clean":"2021-03-24T22:00:21.275662+0000","last_became_active":"2021-03-24T22:00:03.288581+0000","last_became_peered":"2021-03-24T22:00:03.288581+0000","last_unstale":"2021-03-24T22:00:21.275662+0000","last_undegraded":"2021-03-24T22:00:21.275662+0000","last_fullsized":"2021-03-24T22:00:21.275662+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:19.934770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:00:19.934770+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"227","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.286363+0000","last_change":"2021-03-24T22:00:03.282816+0000","last_active":"2021-03-24T22:00:22.286363+0000","last_peered":"2021-03-24T22:00:22.286363+0000","last_clean":"2021-03-24T22:00:22.286363+0000","last_became_active":"2021-03-24T22:00:03.282458+0000","last_became_peered":"2021-03-24T22:00:03.282458+0000","last_unstale":"2021-03-24T22:00:22.286363+0000","last_undegraded":"2021-03-24T22:00:22.286363+0000","last_fullsized":"2021-03-24T22:00:22.286363+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:49.970073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:49.970073+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"452'70","reported_seq":"1185","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-24T22:00:25.256493+0000","last_change":"2021-03-24T22:00:22.298637+0000","last_active":"2021-03-24T22:00:25.256493+0000","last_peered":"2021-03-24T22:00:25.256493+0000","last_clean":"2021-03-24T22:00:25.256493+0000","last_became_active":"2021-03-24T22:00:03.281149+0000","last_became_peered":"2021-03-24T22:00:03.281149+0000","last_unstale":"2021-03-24T22:00:25.256493+0000","last_undegraded":"2021-03-24T22:00:25.256493+0000","last_fullsized":"2021-03-24T22:00:25.256493+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'45","last_scrub_stamp":"2021-03-24T21:57:27.105464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:57:27.105464+0000","log_size":70,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5075879,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":22,"num_read_kb":19652,"num_write":12,"num_write_kb":2368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2421497,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"10f","length":"1"},{"start":"128","length":"1"},{"start":"145","length":"1"}]},{"pgid":"4.e","version":"89'27","reported_seq":"534","reported_epoch":"451","state":"active+clean","last_fresh":"2021-03-24T22:00:21.277700+0000","last_change":"2021-03-24T22:00:20.952318+0000","last_active":"2021-03-24T22:00:21.277700+0000","last_peered":"2021-03-24T22:00:21.277700+0000","last_clean":"2021-03-24T22:00:21.277700+0000","last_became_active":"2021-03-24T22:00:12.273191+0000","last_became_peered":"2021-03-24T22:00:12.273191+0000","last_unstale":"2021-03-24T22:00:21.277700+0000","last_undegraded":"2021-03-24T22:00:21.277700+0000","last_fullsized":"2021-03-24T22:00:21.277700+0000","mapping_epoch":432,"log_start":"89'27","ondisk_log_start":"89'27","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'27","last_scrub_stamp":"2021-03-24T22:00:20.952131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:00:20.952131+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1672,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":65,"num_read_kb":33,"num_write":32,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":2410,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"442","reported_epoch":"451","state":"active+clean","last_fresh":"2021-03-24T22:00:21.274617+0000","last_change":"2021-03-24T22:00:04.285118+0000","last_active":"2021-03-24T22:00:21.274617+0000","last_peered":"2021-03-24T22:00:21.274617+0000","last_clean":"2021-03-24T22:00:21.274617+0000","last_became_active":"2021-03-24T22:00:04.284806+0000","last_became_peered":"2021-03-24T22:00:04.284806+0000","last_unstale":"2021-03-24T22:00:21.274617+0000","last_undegraded":"2021-03-24T22:00:21.274617+0000","last_fullsized":"2021-03-24T22:00:21.274617+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:55.592795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:55.592795+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"401","reported_epoch":"451","state":"active+clean","last_fresh":"2021-03-24T22:00:21.277833+0000","last_change":"2021-03-24T22:00:03.283605+0000","last_active":"2021-03-24T22:00:21.277833+0000","last_peered":"2021-03-24T22:00:21.277833+0000","last_clean":"2021-03-24T22:00:21.277833+0000","last_became_active":"2021-03-24T22:00:03.283136+0000","last_became_peered":"2021-03-24T22:00:03.283136+0000","last_unstale":"2021-03-24T22:00:21.277833+0000","last_undegraded":"2021-03-24T22:00:21.277833+0000","last_fullsized":"2021-03-24T22:00:21.277833+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:43.970879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:43.970879+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"437","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.287543+0000","last_change":"2021-03-24T22:00:04.297784+0000","last_active":"2021-03-24T22:00:22.287543+0000","last_peered":"2021-03-24T22:00:22.287543+0000","last_clean":"2021-03-24T22:00:22.287543+0000","last_became_active":"2021-03-24T22:00:04.296290+0000","last_became_peered":"2021-03-24T22:00:04.296290+0000","last_unstale":"2021-03-24T22:00:22.287543+0000","last_undegraded":"2021-03-24T22:00:22.287543+0000","last_fullsized":"2021-03-24T22:00:22.287543+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:59.472393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:55:59.472393+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"452'432","reported_seq":"1624","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.299047+0000","last_change":"2021-03-24T22:00:22.299047+0000","last_active":"2021-03-24T22:00:22.299047+0000","last_peered":"2021-03-24T22:00:22.299047+0000","last_clean":"2021-03-24T22:00:22.299047+0000","last_became_active":"2021-03-24T22:00:10.249305+0000","last_became_peered":"2021-03-24T22:00:10.249305+0000","last_unstale":"2021-03-24T22:00:22.299047+0000","last_undegraded":"2021-03-24T22:00:22.299047+0000","last_fullsized":"2021-03-24T22:00:22.299047+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"384'362","last_scrub_stamp":"2021-03-24T21:59:15.076161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:59:15.076161+0000","log_size":432,"ondisk_log_size":432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14443119,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":11,"num_read_kb":7747,"num_write":8,"num_write_kb":5472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":11891547,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e7","length":"1"},{"start":"f8","length":"1"},{"start":"10c","length":"1"},{"start":"10f","length":"1"},{"start":"126","length":"1"},{"start":"128","length":"1"},{"start":"145","length":"1"}]},{"pgid":"4.f","version":"274'26","reported_seq":"498","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.286765+0000","last_change":"2021-03-24T22:00:05.249668+0000","last_active":"2021-03-24T22:00:22.286765+0000","last_peered":"2021-03-24T22:00:22.286765+0000","last_clean":"2021-03-24T22:00:22.286765+0000","last_became_active":"2021-03-24T22:00:05.249246+0000","last_became_peered":"2021-03-24T22:00:05.249246+0000","last_unstale":"2021-03-24T22:00:22.286765+0000","last_undegraded":"2021-03-24T22:00:22.286765+0000","last_fullsized":"2021-03-24T22:00:22.286765+0000","mapping_epoch":434,"log_start":"274'26","ondisk_log_start":"274'26","created":62,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'26","last_scrub_stamp":"2021-03-24T21:57:24.499090+0000","last_deep_scrub":"274'26","last_deep_scrub_stamp":"2021-03-24T21:57:24.499090+0000","last_clean_scrub_stamp":"2021-03-24T21:57:24.499090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1472,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":68,"num_read_kb":36,"num_write":36,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":3994,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":"446","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.283857+0000","last_change":"2021-03-24T22:00:04.267153+0000","last_active":"2021-03-24T22:00:22.283857+0000","last_peered":"2021-03-24T22:00:22.283857+0000","last_clean":"2021-03-24T22:00:22.283857+0000","last_became_active":"2021-03-24T22:00:04.266894+0000","last_became_peered":"2021-03-24T22:00:04.266894+0000","last_unstale":"2021-03-24T22:00:22.283857+0000","last_undegraded":"2021-03-24T22:00:22.283857+0000","last_fullsized":"2021-03-24T22:00:22.283857+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:39.439170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:39.439170+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"433'4","reported_seq":"529","reported_epoch":"451","state":"active+clean","last_fresh":"2021-03-24T22:00:21.277427+0000","last_change":"2021-03-24T22:00:14.879494+0000","last_active":"2021-03-24T22:00:21.277427+0000","last_peered":"2021-03-24T22:00:21.277427+0000","last_clean":"2021-03-24T22:00:21.277427+0000","last_became_active":"2021-03-24T22:00:03.287761+0000","last_became_peered":"2021-03-24T22:00:03.287761+0000","last_unstale":"2021-03-24T22:00:21.277427+0000","last_undegraded":"2021-03-24T22:00:21.277427+0000","last_fullsized":"2021-03-24T22:00:21.277427+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'4","last_scrub_stamp":"2021-03-24T22:00:14.879311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:00:14.879311+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"318","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-24T22:00:25.254299+0000","last_change":"2021-03-24T22:00:15.205264+0000","last_active":"2021-03-24T22:00:25.254299+0000","last_peered":"2021-03-24T22:00:25.254299+0000","last_clean":"2021-03-24T22:00:25.254299+0000","last_became_active":"2021-03-24T22:00:03.297628+0000","last_became_peered":"2021-03-24T22:00:03.297628+0000","last_unstale":"2021-03-24T22:00:25.254299+0000","last_undegraded":"2021-03-24T22:00:25.254299+0000","last_fullsized":"2021-03-24T22:00:25.254299+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:23.176750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:57:23.176750+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"451'566","reported_seq":"1781","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.289244+0000","last_change":"2021-03-24T22:00:22.289244+0000","last_active":"2021-03-24T22:00:22.289244+0000","last_peered":"2021-03-24T22:00:22.289244+0000","last_clean":"2021-03-24T22:00:22.289244+0000","last_became_active":"2021-03-24T22:00:02.289366+0000","last_became_peered":"2021-03-24T22:00:02.289366+0000","last_unstale":"2021-03-24T22:00:22.289244+0000","last_undegraded":"2021-03-24T22:00:22.289244+0000","last_fullsized":"2021-03-24T22:00:22.289244+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'393","last_scrub_stamp":"2021-03-24T21:59:12.370455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:59:12.370455+0000","log_size":566,"ondisk_log_size":566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37660228,"num_objects":15,"num_object_clones":11,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":2,"num_read":55,"num_read_kb":70628,"num_write":33,"num_write_kb":15558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":32790575,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"e7","length":"1"},{"start":"f8","length":"1"},{"start":"10c","length":"1"},{"start":"10f","length":"1"},{"start":"126","length":"1"},{"start":"128","length":"1"},{"start":"145","length":"1"}]},{"pgid":"5.1","version":"0'0","reported_seq":"418","reported_epoch":"451","state":"active+clean","last_fresh":"2021-03-24T22:00:21.274826+0000","last_change":"2021-03-24T21:59:47.306423+0000","last_active":"2021-03-24T22:00:21.274826+0000","last_peered":"2021-03-24T22:00:21.274826+0000","last_clean":"2021-03-24T22:00:21.274826+0000","last_became_active":"2021-03-24T21:59:47.305635+0000","last_became_peered":"2021-03-24T21:59:47.305635+0000","last_unstale":"2021-03-24T22:00:21.274826+0000","last_undegraded":"2021-03-24T22:00:21.274826+0000","last_fullsized":"2021-03-24T22:00:21.274826+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:59.461865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:25.220436+0000","last_clean_scrub_stamp":"2021-03-24T21:57:59.461865+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"69'22","reported_seq":"497","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.283745+0000","last_change":"2021-03-24T22:00:06.440877+0000","last_active":"2021-03-24T22:00:22.283745+0000","last_peered":"2021-03-24T22:00:22.283745+0000","last_clean":"2021-03-24T22:00:22.283745+0000","last_became_active":"2021-03-24T22:00:04.294791+0000","last_became_peered":"2021-03-24T22:00:04.294791+0000","last_unstale":"2021-03-24T22:00:22.283745+0000","last_undegraded":"2021-03-24T22:00:22.283745+0000","last_fullsized":"2021-03-24T22:00:22.283745+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'22","last_scrub_stamp":"2021-03-24T21:58:56.473510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:58:56.473510+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":46,"num_read_kb":26,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":846,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"442","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.286677+0000","last_change":"2021-03-24T22:00:04.299283+0000","last_active":"2021-03-24T22:00:22.286677+0000","last_peered":"2021-03-24T22:00:22.286677+0000","last_clean":"2021-03-24T22:00:22.286677+0000","last_became_active":"2021-03-24T22:00:04.298778+0000","last_became_peered":"2021-03-24T22:00:04.298778+0000","last_unstale":"2021-03-24T22:00:22.286677+0000","last_undegraded":"2021-03-24T22:00:22.286677+0000","last_fullsized":"2021-03-24T22:00:22.286677+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:54.381668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:54.381668+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"407","reported_epoch":"451","state":"active+clean","last_fresh":"2021-03-24T22:00:21.275367+0000","last_change":"2021-03-24T21:59:03.626326+0000","last_active":"2021-03-24T22:00:21.275367+0000","last_peered":"2021-03-24T22:00:21.275367+0000","last_clean":"2021-03-24T22:00:21.275367+0000","last_became_active":"2021-03-24T21:51:51.756114+0000","last_became_peered":"2021-03-24T21:51:51.756114+0000","last_unstale":"2021-03-24T22:00:21.275367+0000","last_undegraded":"2021-03-24T22:00:21.275367+0000","last_fullsized":"2021-03-24T22:00:21.275367+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:03.626250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:59:03.626250+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"451'458","reported_seq":"1601","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.289629+0000","last_change":"2021-03-24T22:00:22.289629+0000","last_active":"2021-03-24T22:00:22.289629+0000","last_peered":"2021-03-24T22:00:22.289629+0000","last_clean":"2021-03-24T22:00:22.289629+0000","last_became_active":"2021-03-24T21:57:26.169302+0000","last_became_peered":"2021-03-24T21:57:26.169302+0000","last_unstale":"2021-03-24T22:00:22.289629+0000","last_undegraded":"2021-03-24T22:00:22.289629+0000","last_fullsized":"2021-03-24T22:00:22.289629+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'330","last_scrub_stamp":"2021-03-24T21:59:07.373692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:59:07.373692+0000","log_size":458,"ondisk_log_size":458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19866685,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":24,"num_read_kb":21584,"num_write":22,"num_write_kb":9961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"e7","length":"1"},{"start":"f8","length":"1"},{"start":"10c","length":"1"},{"start":"10f","length":"1"},{"start":"126","length":"1"},{"start":"128","length":"1"},{"start":"145","length":"1"}]},{"pgid":"5.0","version":"109'3","reported_seq":"531","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.283530+0000","last_change":"2021-03-24T21:59:46.375249+0000","last_active":"2021-03-24T22:00:22.283530+0000","last_peered":"2021-03-24T22:00:22.283530+0000","last_clean":"2021-03-24T22:00:22.283530+0000","last_became_active":"2021-03-24T21:54:38.135726+0000","last_became_peered":"2021-03-24T21:54:38.135726+0000","last_unstale":"2021-03-24T22:00:22.283530+0000","last_undegraded":"2021-03-24T22:00:22.283530+0000","last_fullsized":"2021-03-24T22:00:22.283530+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'3","last_scrub_stamp":"2021-03-24T21:59:46.375145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:59:46.375145+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"69'20","reported_seq":"496","reported_epoch":"451","state":"active+clean","last_fresh":"2021-03-24T22:00:21.276214+0000","last_change":"2021-03-24T22:00:03.287155+0000","last_active":"2021-03-24T22:00:21.276214+0000","last_peered":"2021-03-24T22:00:21.276214+0000","last_clean":"2021-03-24T22:00:21.276214+0000","last_became_active":"2021-03-24T22:00:03.286760+0000","last_became_peered":"2021-03-24T22:00:03.286760+0000","last_unstale":"2021-03-24T22:00:21.276214+0000","last_undegraded":"2021-03-24T22:00:21.276214+0000","last_fullsized":"2021-03-24T22:00:21.276214+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'20","last_scrub_stamp":"2021-03-24T21:57:21.706127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:57:21.706127+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":39,"num_read_kb":20,"num_write":20,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":940,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"447","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.290612+0000","last_change":"2021-03-24T22:00:03.303270+0000","last_active":"2021-03-24T22:00:22.290612+0000","last_peered":"2021-03-24T22:00:22.290612+0000","last_clean":"2021-03-24T22:00:22.290612+0000","last_became_active":"2021-03-24T22:00:03.302949+0000","last_became_peered":"2021-03-24T22:00:03.302949+0000","last_unstale":"2021-03-24T22:00:22.290612+0000","last_undegraded":"2021-03-24T22:00:22.290612+0000","last_fullsized":"2021-03-24T22:00:22.290612+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:28.165408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:28.165408+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"114","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-24T22:00:25.255823+0000","last_change":"2021-03-24T21:59:47.280442+0000","last_active":"2021-03-24T22:00:25.255823+0000","last_peered":"2021-03-24T22:00:25.255823+0000","last_clean":"2021-03-24T22:00:25.255823+0000","last_became_active":"2021-03-24T21:59:47.280054+0000","last_became_peered":"2021-03-24T21:59:47.280054+0000","last_unstale":"2021-03-24T22:00:25.255823+0000","last_undegraded":"2021-03-24T22:00:25.255823+0000","last_fullsized":"2021-03-24T22:00:25.255823+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:54.826534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:58:54.826534+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2","version":"272'64","reported_seq":"1169","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.288945+0000","last_change":"2021-03-24T22:00:22.288945+0000","last_active":"2021-03-24T22:00:22.288945+0000","last_peered":"2021-03-24T22:00:22.288945+0000","last_clean":"2021-03-24T22:00:22.288945+0000","last_became_active":"2021-03-24T21:59:47.301298+0000","last_became_peered":"2021-03-24T21:59:47.301298+0000","last_unstale":"2021-03-24T22:00:22.288945+0000","last_undegraded":"2021-03-24T22:00:22.288945+0000","last_fullsized":"2021-03-24T22:00:22.288945+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'64","last_scrub_stamp":"2021-03-24T21:59:08.406188+0000","last_deep_scrub":"272'64","last_deep_scrub_stamp":"2021-03-24T21:59:08.406188+0000","last_clean_scrub_stamp":"2021-03-24T21:59:08.406188+0000","log_size":64,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"e7","length":"1"},{"start":"f8","length":"1"},{"start":"10c","length":"1"},{"start":"10f","length":"1"},{"start":"126","length":"1"},{"start":"128","length":"1"},{"start":"145","length":"1"}]},{"pgid":"5.3","version":"433'6","reported_seq":"543","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.281254+0000","last_change":"2021-03-24T22:00:16.563550+0000","last_active":"2021-03-24T22:00:22.281254+0000","last_peered":"2021-03-24T22:00:22.281254+0000","last_clean":"2021-03-24T22:00:22.281254+0000","last_became_active":"2021-03-24T22:00:03.302287+0000","last_became_peered":"2021-03-24T22:00:03.302287+0000","last_unstale":"2021-03-24T22:00:22.281254+0000","last_undegraded":"2021-03-24T22:00:22.281254+0000","last_fullsized":"2021-03-24T22:00:22.281254+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'6","last_scrub_stamp":"2021-03-24T22:00:16.563338+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-24T21:53:26.584678+0000","last_clean_scrub_stamp":"2021-03-24T22:00:16.563338+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"69'16","reported_seq":"329","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-24T22:00:25.258108+0000","last_change":"2021-03-24T22:00:15.219283+0000","last_active":"2021-03-24T22:00:25.258108+0000","last_peered":"2021-03-24T22:00:25.258108+0000","last_clean":"2021-03-24T22:00:25.258108+0000","last_became_active":"2021-03-24T22:00:03.297862+0000","last_became_peered":"2021-03-24T22:00:03.297862+0000","last_unstale":"2021-03-24T22:00:25.258108+0000","last_undegraded":"2021-03-24T22:00:25.258108+0000","last_fullsized":"2021-03-24T22:00:25.258108+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T21:56:41.235143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:41.235143+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":32,"num_read_kb":15,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"306","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-24T22:00:25.253495+0000","last_change":"2021-03-24T22:00:16.298744+0000","last_active":"2021-03-24T22:00:25.253495+0000","last_peered":"2021-03-24T22:00:25.253495+0000","last_clean":"2021-03-24T22:00:25.253495+0000","last_became_active":"2021-03-24T22:00:03.284574+0000","last_became_peered":"2021-03-24T22:00:03.284574+0000","last_unstale":"2021-03-24T22:00:25.253495+0000","last_undegraded":"2021-03-24T22:00:25.253495+0000","last_fullsized":"2021-03-24T22:00:25.253495+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:53.013129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:53.013129+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"407","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.277263+0000","last_change":"2021-03-24T21:59:04.077747+0000","last_active":"2021-03-24T22:00:22.277263+0000","last_peered":"2021-03-24T22:00:22.277263+0000","last_clean":"2021-03-24T22:00:22.277263+0000","last_became_active":"2021-03-24T21:51:51.759627+0000","last_became_peered":"2021-03-24T21:51:51.759627+0000","last_unstale":"2021-03-24T22:00:22.277263+0000","last_undegraded":"2021-03-24T22:00:22.277263+0000","last_fullsized":"2021-03-24T22:00:22.277263+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:04.077569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:59:04.077569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"446'321","reported_seq":"1580","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.292848+0000","last_change":"2021-03-24T22:00:22.293076+0000","last_active":"2021-03-24T22:00:22.292848+0000","last_peered":"2021-03-24T22:00:22.292848+0000","last_clean":"2021-03-24T22:00:22.292848+0000","last_became_active":"2021-03-24T22:00:06.241467+0000","last_became_peered":"2021-03-24T22:00:06.241467+0000","last_unstale":"2021-03-24T22:00:22.292848+0000","last_undegraded":"2021-03-24T22:00:22.292848+0000","last_fullsized":"2021-03-24T22:00:22.292848+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"365'287","last_scrub_stamp":"2021-03-24T21:58:56.648040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:58:56.648040+0000","log_size":321,"ondisk_log_size":321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3854257,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":16157,"num_write":4,"num_write_kb":1073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":7787128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e7","length":"1"},{"start":"f8","length":"1"},{"start":"10c","length":"1"},{"start":"10f","length":"1"},{"start":"126","length":"1"},{"start":"128","length":"1"},{"start":"145","length":"1"}]},{"pgid":"5.2","version":"0'0","reported_seq":"387","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.278548+0000","last_change":"2021-03-24T22:00:14.616941+0000","last_active":"2021-03-24T22:00:22.278548+0000","last_peered":"2021-03-24T22:00:22.278548+0000","last_clean":"2021-03-24T22:00:22.278548+0000","last_became_active":"2021-03-24T22:00:03.301785+0000","last_became_peered":"2021-03-24T22:00:03.301785+0000","last_unstale":"2021-03-24T22:00:22.278548+0000","last_undegraded":"2021-03-24T22:00:22.278548+0000","last_fullsized":"2021-03-24T22:00:22.278548+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:14.616769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:00:14.616769+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"69'21","reported_seq":"499","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.281403+0000","last_change":"2021-03-24T22:00:04.274866+0000","last_active":"2021-03-24T22:00:22.281403+0000","last_peered":"2021-03-24T22:00:22.281403+0000","last_clean":"2021-03-24T22:00:22.281403+0000","last_became_active":"2021-03-24T22:00:04.274418+0000","last_became_peered":"2021-03-24T22:00:04.274418+0000","last_unstale":"2021-03-24T22:00:22.281403+0000","last_undegraded":"2021-03-24T22:00:22.281403+0000","last_fullsized":"2021-03-24T22:00:22.281403+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'21","last_scrub_stamp":"2021-03-24T21:56:39.335613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:39.335613+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":45,"num_read_kb":27,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":642,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"135","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-24T22:00:25.251054+0000","last_change":"2021-03-24T22:00:16.307271+0000","last_active":"2021-03-24T22:00:25.251054+0000","last_peered":"2021-03-24T22:00:25.251054+0000","last_clean":"2021-03-24T22:00:25.251054+0000","last_became_active":"2021-03-24T22:00:03.293605+0000","last_became_peered":"2021-03-24T22:00:03.293605+0000","last_unstale":"2021-03-24T22:00:25.251054+0000","last_undegraded":"2021-03-24T22:00:25.251054+0000","last_fullsized":"2021-03-24T22:00:25.251054+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:38.144956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:38.144956+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"216","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.279383+0000","last_change":"2021-03-24T22:00:04.300961+0000","last_active":"2021-03-24T22:00:22.279383+0000","last_peered":"2021-03-24T22:00:22.279383+0000","last_clean":"2021-03-24T22:00:22.279383+0000","last_became_active":"2021-03-24T22:00:04.296317+0000","last_became_peered":"2021-03-24T22:00:04.296317+0000","last_unstale":"2021-03-24T22:00:22.279383+0000","last_undegraded":"2021-03-24T22:00:22.279383+0000","last_fullsized":"2021-03-24T22:00:22.279383+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:24.967391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:58:24.967391+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"175'22","reported_seq":"1043","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-24T22:00:25.250445+0000","last_change":"2021-03-24T22:00:22.284107+0000","last_active":"2021-03-24T22:00:25.250445+0000","last_peered":"2021-03-24T22:00:25.250445+0000","last_clean":"2021-03-24T22:00:25.250445+0000","last_became_active":"2021-03-24T22:00:03.301382+0000","last_became_peered":"2021-03-24T22:00:03.301382+0000","last_unstale":"2021-03-24T22:00:25.250445+0000","last_undegraded":"2021-03-24T22:00:25.250445+0000","last_fullsized":"2021-03-24T22:00:25.250445+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'22","last_scrub_stamp":"2021-03-24T21:58:22.940875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:58:22.940875+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3283,"num_write":3,"num_write_kb":845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":313386,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"10f","length":"1"},{"start":"128","length":"1"},{"start":"145","length":"1"}]},{"pgid":"5.7","version":"0'0","reported_seq":"321","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-24T22:00:25.252239+0000","last_change":"2021-03-24T22:00:15.229915+0000","last_active":"2021-03-24T22:00:25.252239+0000","last_peered":"2021-03-24T22:00:25.252239+0000","last_clean":"2021-03-24T22:00:25.252239+0000","last_became_active":"2021-03-24T22:00:03.293177+0000","last_became_peered":"2021-03-24T22:00:03.293177+0000","last_unstale":"2021-03-24T22:00:25.252239+0000","last_undegraded":"2021-03-24T22:00:25.252239+0000","last_fullsized":"2021-03-24T22:00:25.252239+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:43.194160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:43.194160+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"69'16","reported_seq":"547","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.290524+0000","last_change":"2021-03-24T22:00:04.275721+0000","last_active":"2021-03-24T22:00:22.290524+0000","last_peered":"2021-03-24T22:00:22.290524+0000","last_clean":"2021-03-24T22:00:22.290524+0000","last_became_active":"2021-03-24T22:00:04.275105+0000","last_became_peered":"2021-03-24T22:00:04.275105+0000","last_unstale":"2021-03-24T22:00:22.290524+0000","last_undegraded":"2021-03-24T22:00:22.290524+0000","last_fullsized":"2021-03-24T22:00:22.290524+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T21:57:26.257686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:57:26.257686+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":16,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1890,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"66'5","reported_seq":"122","reported_epoch":"453","state":"active+clean","last_fresh":"2021-03-24T22:00:23.287243+0000","last_change":"2021-03-24T22:00:05.171810+0000","last_active":"2021-03-24T22:00:23.287243+0000","last_peered":"2021-03-24T22:00:23.287243+0000","last_clean":"2021-03-24T22:00:23.287243+0000","last_became_active":"2021-03-24T22:00:04.299863+0000","last_became_peered":"2021-03-24T22:00:04.299863+0000","last_unstale":"2021-03-24T22:00:23.287243+0000","last_undegraded":"2021-03-24T22:00:23.287243+0000","last_fullsized":"2021-03-24T22:00:23.287243+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2021-03-24T21:58:58.818001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:39.286101+0000","last_clean_scrub_stamp":"2021-03-24T21:58:58.818001+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":39,"num_read_kb":34,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"427","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.279042+0000","last_change":"2021-03-24T21:59:46.941755+0000","last_active":"2021-03-24T22:00:22.279042+0000","last_peered":"2021-03-24T22:00:22.279042+0000","last_clean":"2021-03-24T22:00:22.279042+0000","last_became_active":"2021-03-24T21:59:12.281282+0000","last_became_peered":"2021-03-24T21:59:12.281282+0000","last_unstale":"2021-03-24T22:00:22.279042+0000","last_undegraded":"2021-03-24T22:00:22.279042+0000","last_fullsized":"2021-03-24T22:00:22.279042+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:46.941623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:59:46.941623+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"211","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.280139+0000","last_change":"2021-03-24T21:58:58.034141+0000","last_active":"2021-03-24T22:00:22.280139+0000","last_peered":"2021-03-24T22:00:22.280139+0000","last_clean":"2021-03-24T22:00:22.280139+0000","last_became_active":"2021-03-24T21:57:06.152772+0000","last_became_peered":"2021-03-24T21:57:06.152772+0000","last_unstale":"2021-03-24T22:00:22.280139+0000","last_undegraded":"2021-03-24T22:00:22.280139+0000","last_fullsized":"2021-03-24T22:00:22.280139+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:58.033876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:58:58.033876+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.5","version":"195'69","reported_seq":"508","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-24T22:00:25.256460+0000","last_change":"2021-03-24T22:00:22.286833+0000","last_active":"2021-03-24T22:00:25.256460+0000","last_peered":"2021-03-24T22:00:25.256460+0000","last_clean":"2021-03-24T22:00:25.256460+0000","last_became_active":"2021-03-24T21:59:47.281169+0000","last_became_peered":"2021-03-24T21:59:47.281169+0000","last_unstale":"2021-03-24T22:00:25.256460+0000","last_undegraded":"2021-03-24T22:00:25.256460+0000","last_fullsized":"2021-03-24T22:00:25.256460+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'69","last_scrub_stamp":"2021-03-24T21:59:09.930390+0000","last_deep_scrub":"195'69","last_deep_scrub_stamp":"2021-03-24T21:56:49.886058+0000","last_clean_scrub_stamp":"2021-03-24T21:59:09.930390+0000","log_size":69,"ondisk_log_size":69,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1672296,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":8383,"num_write":20,"num_write_kb":7737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"10f","length":"1"},{"start":"128","length":"1"},{"start":"145","length":"1"}]},{"pgid":"5.4","version":"0'0","reported_seq":"360","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-24T22:00:25.256712+0000","last_change":"2021-03-24T22:00:04.303336+0000","last_active":"2021-03-24T22:00:25.256712+0000","last_peered":"2021-03-24T22:00:25.256712+0000","last_clean":"2021-03-24T22:00:25.256712+0000","last_became_active":"2021-03-24T22:00:04.302985+0000","last_became_peered":"2021-03-24T22:00:04.302985+0000","last_unstale":"2021-03-24T22:00:25.256712+0000","last_undegraded":"2021-03-24T22:00:25.256712+0000","last_fullsized":"2021-03-24T22:00:25.256712+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:18.199919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:57:18.199919+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"69'15","reported_seq":"468","reported_epoch":"451","state":"active+clean","last_fresh":"2021-03-24T22:00:21.277083+0000","last_change":"2021-03-24T22:00:05.058429+0000","last_active":"2021-03-24T22:00:21.277083+0000","last_peered":"2021-03-24T22:00:21.277083+0000","last_clean":"2021-03-24T22:00:21.277083+0000","last_became_active":"2021-03-24T22:00:04.276581+0000","last_became_peered":"2021-03-24T22:00:04.276581+0000","last_unstale":"2021-03-24T22:00:21.277083+0000","last_undegraded":"2021-03-24T22:00:21.277083+0000","last_fullsized":"2021-03-24T22:00:21.277083+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:58:20.702393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:58:20.702393+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":26,"num_read_kb":14,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":932,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"438","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.295074+0000","last_change":"2021-03-24T22:00:03.281934+0000","last_active":"2021-03-24T22:00:22.295074+0000","last_peered":"2021-03-24T22:00:22.295074+0000","last_clean":"2021-03-24T22:00:22.295074+0000","last_became_active":"2021-03-24T22:00:03.281583+0000","last_became_peered":"2021-03-24T22:00:03.281583+0000","last_unstale":"2021-03-24T22:00:22.295074+0000","last_undegraded":"2021-03-24T22:00:22.295074+0000","last_fullsized":"2021-03-24T22:00:22.295074+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:33.533840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:33.533840+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"127","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.277383+0000","last_change":"2021-03-24T21:59:47.275322+0000","last_active":"2021-03-24T22:00:22.277383+0000","last_peered":"2021-03-24T22:00:22.277383+0000","last_clean":"2021-03-24T22:00:22.277383+0000","last_became_active":"2021-03-24T21:59:47.274973+0000","last_became_peered":"2021-03-24T21:59:47.274973+0000","last_unstale":"2021-03-24T22:00:22.277383+0000","last_undegraded":"2021-03-24T22:00:22.277383+0000","last_fullsized":"2021-03-24T22:00:22.277383+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:21.039180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:49:51.529051+0000","last_clean_scrub_stamp":"2021-03-24T21:58:21.039180+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"302","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-24T22:00:25.251870+0000","last_change":"2021-03-24T22:00:15.245041+0000","last_active":"2021-03-24T22:00:25.251870+0000","last_peered":"2021-03-24T22:00:25.251870+0000","last_clean":"2021-03-24T22:00:25.251870+0000","last_became_active":"2021-03-24T22:00:03.278687+0000","last_became_peered":"2021-03-24T22:00:03.278687+0000","last_unstale":"2021-03-24T22:00:25.251870+0000","last_undegraded":"2021-03-24T22:00:25.251870+0000","last_fullsized":"2021-03-24T22:00:25.251870+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:55.072694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:57:55.072694+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"451'202","reported_seq":"1395","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.299285+0000","last_change":"2021-03-24T22:00:22.299285+0000","last_active":"2021-03-24T22:00:22.299285+0000","last_peered":"2021-03-24T22:00:22.299285+0000","last_clean":"2021-03-24T22:00:22.299285+0000","last_became_active":"2021-03-24T22:00:03.306606+0000","last_became_peered":"2021-03-24T22:00:03.306606+0000","last_unstale":"2021-03-24T22:00:22.299285+0000","last_undegraded":"2021-03-24T22:00:22.299285+0000","last_fullsized":"2021-03-24T22:00:22.299285+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":202,"ondisk_log_size":202,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5210219,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":91,"num_read_kb":116480,"num_write":55,"num_write_kb":27263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":21191071,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e7","length":"1"},{"start":"f8","length":"1"},{"start":"10c","length":"1"},{"start":"10f","length":"1"},{"start":"126","length":"1"},{"start":"128","length":"1"},{"start":"145","length":"1"}]},{"pgid":"5.6","version":"0'0","reported_seq":"110","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-24T22:00:25.253729+0000","last_change":"2021-03-24T22:00:04.293927+0000","last_active":"2021-03-24T22:00:25.253729+0000","last_peered":"2021-03-24T22:00:25.253729+0000","last_clean":"2021-03-24T22:00:25.253729+0000","last_became_active":"2021-03-24T22:00:04.291202+0000","last_became_peered":"2021-03-24T22:00:04.291202+0000","last_unstale":"2021-03-24T22:00:25.253729+0000","last_undegraded":"2021-03-24T22:00:25.253729+0000","last_fullsized":"2021-03-24T22:00:25.253729+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:20.215065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:57:20.215065+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"89'22","reported_seq":"357","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-24T22:00:25.253035+0000","last_change":"2021-03-24T22:00:04.297652+0000","last_active":"2021-03-24T22:00:25.253035+0000","last_peered":"2021-03-24T22:00:25.253035+0000","last_clean":"2021-03-24T22:00:25.253035+0000","last_became_active":"2021-03-24T22:00:04.295008+0000","last_became_peered":"2021-03-24T22:00:04.295008+0000","last_unstale":"2021-03-24T22:00:25.253035+0000","last_undegraded":"2021-03-24T22:00:25.253035+0000","last_fullsized":"2021-03-24T22:00:25.253035+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'22","last_scrub_stamp":"2021-03-24T21:56:54.046194+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2021-03-24T21:53:19.717244+0000","last_clean_scrub_stamp":"2021-03-24T21:56:54.046194+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":36,"num_read_kb":21,"num_write":18,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":644,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"429","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-24T22:00:25.255210+0000","last_change":"2021-03-24T21:59:01.806818+0000","last_active":"2021-03-24T22:00:25.255210+0000","last_peered":"2021-03-24T22:00:25.255210+0000","last_clean":"2021-03-24T22:00:25.255210+0000","last_became_active":"2021-03-24T21:51:40.351000+0000","last_became_peered":"2021-03-24T21:51:40.351000+0000","last_unstale":"2021-03-24T22:00:25.255210+0000","last_undegraded":"2021-03-24T22:00:25.255210+0000","last_fullsized":"2021-03-24T22:00:25.255210+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:01.806758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:39.286101+0000","last_clean_scrub_stamp":"2021-03-24T21:59:01.806758+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"61'1","reported_seq":"398","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-24T22:00:25.254573+0000","last_change":"2021-03-24T22:00:03.296740+0000","last_active":"2021-03-24T22:00:25.254573+0000","last_peered":"2021-03-24T22:00:25.254573+0000","last_clean":"2021-03-24T22:00:25.254573+0000","last_became_active":"2021-03-24T22:00:03.296327+0000","last_became_peered":"2021-03-24T22:00:03.296327+0000","last_unstale":"2021-03-24T22:00:25.254573+0000","last_undegraded":"2021-03-24T22:00:25.254573+0000","last_fullsized":"2021-03-24T22:00:25.254573+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2021-03-24T21:57:12.046206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:12.046206+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":436,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"106","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.286353+0000","last_change":"2021-03-24T22:00:04.296431+0000","last_active":"2021-03-24T22:00:22.286353+0000","last_peered":"2021-03-24T22:00:22.286353+0000","last_clean":"2021-03-24T22:00:22.286353+0000","last_became_active":"2021-03-24T22:00:04.295912+0000","last_became_peered":"2021-03-24T22:00:04.295912+0000","last_unstale":"2021-03-24T22:00:22.286353+0000","last_undegraded":"2021-03-24T22:00:22.286353+0000","last_fullsized":"2021-03-24T22:00:22.286353+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:56.065113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:57:56.065113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"451'257","reported_seq":"1399","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.292428+0000","last_change":"2021-03-24T22:00:22.292428+0000","last_active":"2021-03-24T22:00:22.292428+0000","last_peered":"2021-03-24T22:00:22.292428+0000","last_clean":"2021-03-24T22:00:22.292428+0000","last_became_active":"2021-03-24T22:00:03.303059+0000","last_became_peered":"2021-03-24T22:00:03.303059+0000","last_unstale":"2021-03-24T22:00:22.292428+0000","last_undegraded":"2021-03-24T22:00:22.292428+0000","last_fullsized":"2021-03-24T22:00:22.292428+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'177","last_scrub_stamp":"2021-03-24T21:57:21.260725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:57:21.260725+0000","log_size":257,"ondisk_log_size":257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11255664,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":48,"num_read_kb":58453,"num_write":24,"num_write_kb":11665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":9212107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e7","length":"1"},{"start":"f8","length":"1"},{"start":"10c","length":"1"},{"start":"10f","length":"1"},{"start":"126","length":"1"},{"start":"128","length":"1"},{"start":"145","length":"1"}]},{"pgid":"5.5","version":"0'0","reported_seq":"427","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.280811+0000","last_change":"2021-03-24T21:59:47.325187+0000","last_active":"2021-03-24T22:00:22.280811+0000","last_peered":"2021-03-24T22:00:22.280811+0000","last_clean":"2021-03-24T22:00:22.280811+0000","last_became_active":"2021-03-24T21:59:47.324848+0000","last_became_peered":"2021-03-24T21:59:47.324848+0000","last_unstale":"2021-03-24T22:00:22.280811+0000","last_undegraded":"2021-03-24T22:00:22.280811+0000","last_fullsized":"2021-03-24T22:00:22.280811+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:23.865909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:58:23.865909+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"69'25","reported_seq":"506","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-24T22:00:25.256919+0000","last_change":"2021-03-24T22:00:05.851971+0000","last_active":"2021-03-24T22:00:25.256919+0000","last_peered":"2021-03-24T22:00:25.256919+0000","last_clean":"2021-03-24T22:00:25.256919+0000","last_became_active":"2021-03-24T22:00:04.304215+0000","last_became_peered":"2021-03-24T22:00:04.304215+0000","last_unstale":"2021-03-24T22:00:25.256919+0000","last_undegraded":"2021-03-24T22:00:25.256919+0000","last_fullsized":"2021-03-24T22:00:25.256919+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'25","last_scrub_stamp":"2021-03-24T21:58:54.693601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:58:54.693601+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":53,"num_read_kb":30,"num_write":28,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1908,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"89'2","reported_seq":"465","reported_epoch":"451","state":"active+clean","last_fresh":"2021-03-24T22:00:21.280681+0000","last_change":"2021-03-24T22:00:03.287361+0000","last_active":"2021-03-24T22:00:21.280681+0000","last_peered":"2021-03-24T22:00:21.280681+0000","last_clean":"2021-03-24T22:00:21.280681+0000","last_became_active":"2021-03-24T22:00:03.286357+0000","last_became_peered":"2021-03-24T22:00:03.286357+0000","last_unstale":"2021-03-24T22:00:21.280681+0000","last_undegraded":"2021-03-24T22:00:21.280681+0000","last_fullsized":"2021-03-24T22:00:21.280681+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2021-03-24T21:57:58.491830+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2021-03-24T21:57:58.491830+0000","last_clean_scrub_stamp":"2021-03-24T21:57:58.491830+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":179,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"418","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.278697+0000","last_change":"2021-03-24T22:00:03.300262+0000","last_active":"2021-03-24T22:00:22.278697+0000","last_peered":"2021-03-24T22:00:22.278697+0000","last_clean":"2021-03-24T22:00:22.278697+0000","last_became_active":"2021-03-24T22:00:03.299923+0000","last_became_peered":"2021-03-24T22:00:03.299923+0000","last_unstale":"2021-03-24T22:00:22.278697+0000","last_undegraded":"2021-03-24T22:00:22.278697+0000","last_fullsized":"2021-03-24T22:00:22.278697+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:01.817480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:06.800417+0000","last_clean_scrub_stamp":"2021-03-24T21:58:01.817480+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"427","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.285919+0000","last_change":"2021-03-24T22:00:04.275061+0000","last_active":"2021-03-24T22:00:22.285919+0000","last_peered":"2021-03-24T22:00:22.285919+0000","last_clean":"2021-03-24T22:00:22.285919+0000","last_became_active":"2021-03-24T22:00:04.274660+0000","last_became_peered":"2021-03-24T22:00:04.274660+0000","last_unstale":"2021-03-24T22:00:22.285919+0000","last_undegraded":"2021-03-24T22:00:22.285919+0000","last_fullsized":"2021-03-24T22:00:22.285919+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:22.422787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:52:59.706511+0000","last_clean_scrub_stamp":"2021-03-24T21:57:22.422787+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"452'323","reported_seq":"1490","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.302929+0000","last_change":"2021-03-24T22:00:22.302929+0000","last_active":"2021-03-24T22:00:22.302929+0000","last_peered":"2021-03-24T22:00:22.302929+0000","last_clean":"2021-03-24T22:00:22.302929+0000","last_became_active":"2021-03-24T21:59:59.328965+0000","last_became_peered":"2021-03-24T21:59:59.328965+0000","last_unstale":"2021-03-24T22:00:22.302929+0000","last_undegraded":"2021-03-24T22:00:22.302929+0000","last_fullsized":"2021-03-24T22:00:22.302929+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'218","last_scrub_stamp":"2021-03-24T21:57:22.239425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:57:22.239425+0000","log_size":323,"ondisk_log_size":323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13143338,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":7093,"num_write":25,"num_write_kb":13312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":13490025,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e7","length":"1"},{"start":"f8","length":"1"},{"start":"10c","length":"1"},{"start":"10f","length":"1"},{"start":"126","length":"1"},{"start":"128","length":"1"},{"start":"145","length":"1"}]},{"pgid":"4.b","version":"91'9","reported_seq":"476","reported_epoch":"453","state":"peering","last_fresh":"2021-03-24T22:00:23.291934+0000","last_change":"2021-03-24T22:00:23.292268+0000","last_active":"2021-03-24T22:00:21.279760+0000","last_peered":"2021-03-24T22:00:22.298908+0000","last_clean":"2021-03-24T22:00:22.298908+0000","last_became_active":"2021-03-24T21:54:38.138034+0000","last_became_peered":"2021-03-24T22:00:22.297518+0000","last_unstale":"2021-03-24T22:00:23.291934+0000","last_undegraded":"2021-03-24T22:00:23.291934+0000","last_fullsized":"2021-03-24T22:00:23.291934+0000","mapping_epoch":108,"log_start":"91'9","ondisk_log_start":"91'9","created":62,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'14","last_scrub_stamp":"2021-03-24T21:59:05.058513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:59:05.058513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1158,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":46,"num_read_kb":24,"num_write":22,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1158,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,4],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"418","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.287629+0000","last_change":"2021-03-24T22:00:03.287377+0000","last_active":"2021-03-24T22:00:22.287629+0000","last_peered":"2021-03-24T22:00:22.287629+0000","last_clean":"2021-03-24T22:00:22.287629+0000","last_became_active":"2021-03-24T22:00:03.286890+0000","last_became_peered":"2021-03-24T22:00:03.286890+0000","last_unstale":"2021-03-24T22:00:22.287629+0000","last_undegraded":"2021-03-24T22:00:22.287629+0000","last_fullsized":"2021-03-24T22:00:22.287629+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:12.514094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:12.514094+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"416","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-24T22:00:25.255258+0000","last_change":"2021-03-24T21:58:26.464636+0000","last_active":"2021-03-24T22:00:25.255258+0000","last_peered":"2021-03-24T22:00:25.255258+0000","last_clean":"2021-03-24T22:00:25.255258+0000","last_became_active":"2021-03-24T21:51:49.411921+0000","last_became_peered":"2021-03-24T21:51:49.411921+0000","last_unstale":"2021-03-24T22:00:25.255258+0000","last_undegraded":"2021-03-24T22:00:25.255258+0000","last_fullsized":"2021-03-24T22:00:25.255258+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:26.464456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:58:26.464456+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"425","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.278492+0000","last_change":"2021-03-24T21:59:47.304971+0000","last_active":"2021-03-24T22:00:22.278492+0000","last_peered":"2021-03-24T22:00:22.278492+0000","last_clean":"2021-03-24T22:00:22.278492+0000","last_became_active":"2021-03-24T21:59:47.304623+0000","last_became_peered":"2021-03-24T21:59:47.304623+0000","last_unstale":"2021-03-24T22:00:22.278492+0000","last_undegraded":"2021-03-24T22:00:22.278492+0000","last_fullsized":"2021-03-24T22:00:22.278492+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:00.033461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:35.636299+0000","last_clean_scrub_stamp":"2021-03-24T21:59:00.033461+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"451'200","reported_seq":"1325","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-24T22:00:25.254819+0000","last_change":"2021-03-24T22:00:22.286804+0000","last_active":"2021-03-24T22:00:25.254819+0000","last_peered":"2021-03-24T22:00:25.254819+0000","last_clean":"2021-03-24T22:00:25.254819+0000","last_became_active":"2021-03-24T21:59:51.238701+0000","last_became_peered":"2021-03-24T21:59:51.238701+0000","last_unstale":"2021-03-24T22:00:25.254819+0000","last_undegraded":"2021-03-24T22:00:25.254819+0000","last_fullsized":"2021-03-24T22:00:25.254819+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'168","last_scrub_stamp":"2021-03-24T21:59:44.828980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:59:44.828980+0000","log_size":200,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9983589,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":29,"num_read_kb":29855,"num_write":32,"num_write_kb":9100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":2860247,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"10f","length":"1"},{"start":"128","length":"1"},{"start":"145","length":"1"}]},{"pgid":"4.a","version":"69'23","reported_seq":"109","reported_epoch":"456","state":"activating","last_fresh":"2021-03-24T22:00:26.260532+0000","last_change":"2021-03-24T22:00:26.260532+0000","last_active":"2021-03-24T22:00:25.249060+0000","last_peered":"2021-03-24T22:00:24.312444+0000","last_clean":"2021-03-24T22:00:24.312444+0000","last_became_active":"2021-03-24T22:00:04.311478+0000","last_became_peered":"2021-03-24T22:00:04.311478+0000","last_unstale":"2021-03-24T22:00:26.260532+0000","last_undegraded":"2021-03-24T22:00:26.260532+0000","last_fullsized":"2021-03-24T22:00:26.260532+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'23","last_scrub_stamp":"2021-03-24T22:00:23.343879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:00:23.343879+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":43,"num_read_kb":24,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":732,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":["5","0","3"],"object_location_counts":[{"shards":"0,3,5","objects":13}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":"103","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-24T22:00:25.251242+0000","last_change":"2021-03-24T22:00:04.278932+0000","last_active":"2021-03-24T22:00:25.251242+0000","last_peered":"2021-03-24T22:00:25.251242+0000","last_clean":"2021-03-24T22:00:25.251242+0000","last_became_active":"2021-03-24T22:00:04.275887+0000","last_became_peered":"2021-03-24T22:00:04.275887+0000","last_unstale":"2021-03-24T22:00:25.251242+0000","last_undegraded":"2021-03-24T22:00:25.251242+0000","last_fullsized":"2021-03-24T22:00:25.251242+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:58.133341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:58.133341+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"238","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.291159+0000","last_change":"2021-03-24T22:00:04.301544+0000","last_active":"2021-03-24T22:00:22.291159+0000","last_peered":"2021-03-24T22:00:22.291159+0000","last_clean":"2021-03-24T22:00:22.291159+0000","last_became_active":"2021-03-24T22:00:04.301043+0000","last_became_peered":"2021-03-24T22:00:04.301043+0000","last_unstale":"2021-03-24T22:00:22.291159+0000","last_undegraded":"2021-03-24T22:00:22.291159+0000","last_fullsized":"2021-03-24T22:00:22.291159+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:44.018043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:44.018043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"178","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-24T22:00:25.254693+0000","last_change":"2021-03-24T21:59:47.301476+0000","last_active":"2021-03-24T22:00:25.254693+0000","last_peered":"2021-03-24T22:00:25.254693+0000","last_clean":"2021-03-24T22:00:25.254693+0000","last_became_active":"2021-03-24T21:59:47.300846+0000","last_became_peered":"2021-03-24T21:59:47.300846+0000","last_unstale":"2021-03-24T22:00:25.254693+0000","last_undegraded":"2021-03-24T22:00:25.254693+0000","last_fullsized":"2021-03-24T22:00:25.254693+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:42.813883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:59:42.813883+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"453'272","reported_seq":"1431","reported_epoch":"453","state":"active+clean","last_fresh":"2021-03-24T22:00:23.315614+0000","last_change":"2021-03-24T22:00:22.293109+0000","last_active":"2021-03-24T22:00:23.315614+0000","last_peered":"2021-03-24T22:00:23.315614+0000","last_clean":"2021-03-24T22:00:23.315614+0000","last_became_active":"2021-03-24T21:59:51.249587+0000","last_became_peered":"2021-03-24T21:59:51.249587+0000","last_unstale":"2021-03-24T22:00:23.315614+0000","last_undegraded":"2021-03-24T22:00:23.315614+0000","last_fullsized":"2021-03-24T22:00:23.315614+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'211","last_scrub_stamp":"2021-03-24T21:57:25.266209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:57:25.266209+0000","log_size":272,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6552237,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":51,"num_read_kb":72713,"num_write":18,"num_write_kb":6835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":2872830,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e7","length":"1"},{"start":"f8","length":"1"},{"start":"10c","length":"1"},{"start":"10f","length":"1"},{"start":"126","length":"1"},{"start":"128","length":"1"},{"start":"145","length":"1"}]},{"pgid":"5.8","version":"0'0","reported_seq":"440","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-24T22:00:25.254378+0000","last_change":"2021-03-24T22:00:03.281666+0000","last_active":"2021-03-24T22:00:25.254378+0000","last_peered":"2021-03-24T22:00:25.254378+0000","last_clean":"2021-03-24T22:00:25.254378+0000","last_became_active":"2021-03-24T22:00:03.280916+0000","last_became_peered":"2021-03-24T22:00:03.280916+0000","last_unstale":"2021-03-24T22:00:25.254378+0000","last_undegraded":"2021-03-24T22:00:25.254378+0000","last_fullsized":"2021-03-24T22:00:25.254378+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:28.480625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:58:28.480625+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"69'15","reported_seq":"467","reported_epoch":"451","state":"active+clean","last_fresh":"2021-03-24T22:00:21.273928+0000","last_change":"2021-03-24T21:58:30.678342+0000","last_active":"2021-03-24T22:00:21.273928+0000","last_peered":"2021-03-24T22:00:21.273928+0000","last_clean":"2021-03-24T22:00:21.273928+0000","last_became_active":"2021-03-24T21:54:38.139863+0000","last_became_peered":"2021-03-24T21:54:38.139863+0000","last_unstale":"2021-03-24T22:00:21.273928+0000","last_undegraded":"2021-03-24T22:00:21.273928+0000","last_fullsized":"2021-03-24T22:00:21.273928+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:58:30.678185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:58:30.678185+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":19,"num_write":17,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":692,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":"123","reported_epoch":"451","state":"active+clean","last_fresh":"2021-03-24T22:00:21.277884+0000","last_change":"2021-03-24T22:00:04.268832+0000","last_active":"2021-03-24T22:00:21.277884+0000","last_peered":"2021-03-24T22:00:21.277884+0000","last_clean":"2021-03-24T22:00:21.277884+0000","last_became_active":"2021-03-24T22:00:04.268485+0000","last_became_peered":"2021-03-24T22:00:04.268485+0000","last_unstale":"2021-03-24T22:00:21.277884+0000","last_undegraded":"2021-03-24T22:00:21.277884+0000","last_fullsized":"2021-03-24T22:00:21.277884+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:12.939226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:59:12.939226+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"440","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.287996+0000","last_change":"2021-03-24T22:00:18.322351+0000","last_active":"2021-03-24T22:00:22.287996+0000","last_peered":"2021-03-24T22:00:22.287996+0000","last_clean":"2021-03-24T22:00:22.287996+0000","last_became_active":"2021-03-24T22:00:03.287512+0000","last_became_peered":"2021-03-24T22:00:03.287512+0000","last_unstale":"2021-03-24T22:00:22.287996+0000","last_undegraded":"2021-03-24T22:00:22.287996+0000","last_fullsized":"2021-03-24T22:00:22.287996+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:18.322187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:22.724840+0000","last_clean_scrub_stamp":"2021-03-24T22:00:18.322187+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"453'156","reported_seq":"1247","reported_epoch":"453","state":"active+clean","last_fresh":"2021-03-24T22:00:23.314851+0000","last_change":"2021-03-24T22:00:22.300191+0000","last_active":"2021-03-24T22:00:23.314851+0000","last_peered":"2021-03-24T22:00:23.314851+0000","last_clean":"2021-03-24T22:00:23.314851+0000","last_became_active":"2021-03-24T22:00:03.286696+0000","last_became_peered":"2021-03-24T22:00:03.286696+0000","last_unstale":"2021-03-24T22:00:23.314851+0000","last_undegraded":"2021-03-24T22:00:23.314851+0000","last_fullsized":"2021-03-24T22:00:23.314851+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":156,"ondisk_log_size":156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11109691,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":38,"num_read_kb":42959,"num_write":44,"num_write_kb":17729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":11591117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"e7","length":"1"},{"start":"f8","length":"1"},{"start":"10c","length":"1"},{"start":"10f","length":"1"},{"start":"126","length":"1"},{"start":"128","length":"1"},{"start":"145","length":"1"}]},{"pgid":"5.9","version":"418'5","reported_seq":"67","reported_epoch":"451","state":"active+clean","last_fresh":"2021-03-24T22:00:21.275054+0000","last_change":"2021-03-24T21:59:47.440837+0000","last_active":"2021-03-24T22:00:21.275054+0000","last_peered":"2021-03-24T22:00:21.275054+0000","last_clean":"2021-03-24T22:00:21.275054+0000","last_became_active":"2021-03-24T21:59:47.306040+0000","last_became_peered":"2021-03-24T21:59:47.306040+0000","last_unstale":"2021-03-24T22:00:21.275054+0000","last_undegraded":"2021-03-24T22:00:21.275054+0000","last_fullsized":"2021-03-24T22:00:21.275054+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'4","last_scrub_stamp":"2021-03-24T21:59:06.906705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:59:06.906705+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"97'28","reported_seq":"401","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-24T22:00:25.249755+0000","last_change":"2021-03-24T22:00:04.287488+0000","last_active":"2021-03-24T22:00:25.249755+0000","last_peered":"2021-03-24T22:00:25.249755+0000","last_clean":"2021-03-24T22:00:25.249755+0000","last_became_active":"2021-03-24T22:00:04.282758+0000","last_became_peered":"2021-03-24T22:00:04.282758+0000","last_unstale":"2021-03-24T22:00:25.249755+0000","last_undegraded":"2021-03-24T22:00:25.249755+0000","last_fullsized":"2021-03-24T22:00:25.249755+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'28","last_scrub_stamp":"2021-03-24T21:58:20.925603+0000","last_deep_scrub":"97'28","last_deep_scrub_stamp":"2021-03-24T21:54:19.157265+0000","last_clean_scrub_stamp":"2021-03-24T21:58:20.925603+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":20,"num_write":19,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"89'4","reported_seq":"103","reported_epoch":"451","state":"active+clean","last_fresh":"2021-03-24T22:00:21.276630+0000","last_change":"2021-03-24T22:00:08.337219+0000","last_active":"2021-03-24T22:00:21.276630+0000","last_peered":"2021-03-24T22:00:21.276630+0000","last_clean":"2021-03-24T22:00:21.276630+0000","last_became_active":"2021-03-24T22:00:04.296046+0000","last_became_peered":"2021-03-24T22:00:04.296046+0000","last_unstale":"2021-03-24T22:00:21.276630+0000","last_undegraded":"2021-03-24T22:00:21.276630+0000","last_fullsized":"2021-03-24T22:00:21.276630+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'4","last_scrub_stamp":"2021-03-24T21:58:25.969865+0000","last_deep_scrub":"89'4","last_deep_scrub_stamp":"2021-03-24T21:53:19.926637+0000","last_clean_scrub_stamp":"2021-03-24T21:58:25.969865+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"401","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.278821+0000","last_change":"2021-03-24T22:00:03.300770+0000","last_active":"2021-03-24T22:00:22.278821+0000","last_peered":"2021-03-24T22:00:22.278821+0000","last_clean":"2021-03-24T22:00:22.278821+0000","last_became_active":"2021-03-24T22:00:03.300457+0000","last_became_peered":"2021-03-24T22:00:03.300457+0000","last_unstale":"2021-03-24T22:00:22.278821+0000","last_undegraded":"2021-03-24T22:00:22.278821+0000","last_fullsized":"2021-03-24T22:00:22.278821+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:54.262681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:54.262681+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"272'275","reported_seq":"1425","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-24T22:00:25.250964+0000","last_change":"2021-03-24T22:00:22.285730+0000","last_active":"2021-03-24T22:00:25.250964+0000","last_peered":"2021-03-24T22:00:25.250964+0000","last_clean":"2021-03-24T22:00:25.250964+0000","last_became_active":"2021-03-24T22:00:03.297353+0000","last_became_peered":"2021-03-24T22:00:03.297353+0000","last_unstale":"2021-03-24T22:00:25.250964+0000","last_undegraded":"2021-03-24T22:00:25.250964+0000","last_fullsized":"2021-03-24T22:00:25.250964+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'275","last_scrub_stamp":"2021-03-24T21:58:23.942036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:58:23.942036+0000","log_size":275,"ondisk_log_size":275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8857338,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":114,"num_read_kb":132714,"num_write":52,"num_write_kb":22259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":12587455,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"10f","length":"1"},{"start":"128","length":"1"},{"start":"145","length":"1"}]},{"pgid":"4.17","version":"69'7","reported_seq":"42","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-24T22:00:25.252115+0000","last_change":"2021-03-24T22:00:05.378274+0000","last_active":"2021-03-24T22:00:25.252115+0000","last_peered":"2021-03-24T22:00:25.252115+0000","last_clean":"2021-03-24T22:00:25.252115+0000","last_became_active":"2021-03-24T22:00:04.295420+0000","last_became_peered":"2021-03-24T22:00:04.295420+0000","last_unstale":"2021-03-24T22:00:25.252115+0000","last_undegraded":"2021-03-24T22:00:25.252115+0000","last_fullsized":"2021-03-24T22:00:25.252115+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'7","last_scrub_stamp":"2021-03-24T21:59:06.367172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:59:06.367172+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":7,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":474,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":"387","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.277503+0000","last_change":"2021-03-24T22:00:03.301982+0000","last_active":"2021-03-24T22:00:22.277503+0000","last_peered":"2021-03-24T22:00:22.277503+0000","last_clean":"2021-03-24T22:00:22.277503+0000","last_became_active":"2021-03-24T22:00:03.301593+0000","last_became_peered":"2021-03-24T22:00:03.301593+0000","last_unstale":"2021-03-24T22:00:22.277503+0000","last_undegraded":"2021-03-24T22:00:22.277503+0000","last_fullsized":"2021-03-24T22:00:22.277503+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:16.270874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:16.270874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"323","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-24T22:00:25.251539+0000","last_change":"2021-03-24T22:00:04.294431+0000","last_active":"2021-03-24T22:00:25.251539+0000","last_peered":"2021-03-24T22:00:25.251539+0000","last_clean":"2021-03-24T22:00:25.251539+0000","last_became_active":"2021-03-24T22:00:04.292101+0000","last_became_peered":"2021-03-24T22:00:04.292101+0000","last_unstale":"2021-03-24T22:00:25.251539+0000","last_undegraded":"2021-03-24T22:00:25.251539+0000","last_fullsized":"2021-03-24T22:00:25.251539+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:48.144391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:48.144391+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"363","reported_epoch":"451","state":"active+clean","last_fresh":"2021-03-24T22:00:21.274794+0000","last_change":"2021-03-24T22:00:04.267956+0000","last_active":"2021-03-24T22:00:21.274794+0000","last_peered":"2021-03-24T22:00:21.274794+0000","last_clean":"2021-03-24T22:00:21.274794+0000","last_became_active":"2021-03-24T22:00:04.267669+0000","last_became_peered":"2021-03-24T22:00:04.267669+0000","last_unstale":"2021-03-24T22:00:21.274794+0000","last_undegraded":"2021-03-24T22:00:21.274794+0000","last_fullsized":"2021-03-24T22:00:21.274794+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:04.012279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:58:04.012279+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"451'393","reported_seq":"1568","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.289331+0000","last_change":"2021-03-24T22:00:22.289331+0000","last_active":"2021-03-24T22:00:22.289331+0000","last_peered":"2021-03-24T22:00:22.289331+0000","last_clean":"2021-03-24T22:00:22.289331+0000","last_became_active":"2021-03-24T22:00:03.281974+0000","last_became_peered":"2021-03-24T22:00:03.281974+0000","last_unstale":"2021-03-24T22:00:22.289331+0000","last_undegraded":"2021-03-24T22:00:22.289331+0000","last_fullsized":"2021-03-24T22:00:22.289331+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":393,"ondisk_log_size":393,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28568505,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":181,"num_read_kb":176464,"num_write":141,"num_write_kb":48644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":33311730,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"e7","length":"1"},{"start":"f8","length":"1"},{"start":"10c","length":"1"},{"start":"10f","length":"1"},{"start":"126","length":"1"},{"start":"128","length":"1"},{"start":"145","length":"1"}]},{"pgid":"4.16","version":"69'16","reported_seq":"524","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.291003+0000","last_change":"2021-03-24T22:00:20.543635+0000","last_active":"2021-03-24T22:00:22.291003+0000","last_peered":"2021-03-24T22:00:22.291003+0000","last_clean":"2021-03-24T22:00:22.291003+0000","last_became_active":"2021-03-24T22:00:04.274564+0000","last_became_peered":"2021-03-24T22:00:04.274564+0000","last_unstale":"2021-03-24T22:00:22.291003+0000","last_undegraded":"2021-03-24T22:00:22.291003+0000","last_fullsized":"2021-03-24T22:00:22.291003+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T22:00:20.543450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:00:20.543450+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":38,"num_read_kb":18,"num_write":17,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":2412,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":"219","reported_epoch":"451","state":"active+clean","last_fresh":"2021-03-24T22:00:21.274166+0000","last_change":"2021-03-24T21:59:47.305303+0000","last_active":"2021-03-24T22:00:21.274166+0000","last_peered":"2021-03-24T22:00:21.274166+0000","last_clean":"2021-03-24T22:00:21.274166+0000","last_became_active":"2021-03-24T21:59:47.304918+0000","last_became_peered":"2021-03-24T21:59:47.304918+0000","last_unstale":"2021-03-24T22:00:21.274166+0000","last_undegraded":"2021-03-24T22:00:21.274166+0000","last_fullsized":"2021-03-24T22:00:21.274166+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:51.037780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:51.037780+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"427","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.287186+0000","last_change":"2021-03-24T21:59:47.329444+0000","last_active":"2021-03-24T22:00:22.287186+0000","last_peered":"2021-03-24T22:00:22.287186+0000","last_clean":"2021-03-24T22:00:22.287186+0000","last_became_active":"2021-03-24T21:59:47.329108+0000","last_became_peered":"2021-03-24T21:59:47.329108+0000","last_unstale":"2021-03-24T22:00:22.287186+0000","last_undegraded":"2021-03-24T22:00:22.287186+0000","last_fullsized":"2021-03-24T22:00:22.287186+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:52.376475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:55.493713+0000","last_clean_scrub_stamp":"2021-03-24T21:57:52.376475+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"423","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-24T22:00:25.256610+0000","last_change":"2021-03-24T21:59:47.289874+0000","last_active":"2021-03-24T22:00:25.256610+0000","last_peered":"2021-03-24T22:00:25.256610+0000","last_clean":"2021-03-24T22:00:25.256610+0000","last_became_active":"2021-03-24T21:59:47.289490+0000","last_became_peered":"2021-03-24T21:59:47.289490+0000","last_unstale":"2021-03-24T22:00:25.256610+0000","last_undegraded":"2021-03-24T22:00:25.256610+0000","last_fullsized":"2021-03-24T22:00:25.256610+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:59.862021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:58:59.862021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"451'144","reported_seq":"1305","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-24T22:00:25.256208+0000","last_change":"2021-03-24T22:00:22.285615+0000","last_active":"2021-03-24T22:00:25.256208+0000","last_peered":"2021-03-24T22:00:25.256208+0000","last_clean":"2021-03-24T22:00:25.256208+0000","last_became_active":"2021-03-24T22:00:04.295027+0000","last_became_peered":"2021-03-24T22:00:04.295027+0000","last_unstale":"2021-03-24T22:00:25.256208+0000","last_undegraded":"2021-03-24T22:00:25.256208+0000","last_fullsized":"2021-03-24T22:00:25.256208+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":5,"last_scrub":"447'140","last_scrub_stamp":"2021-03-24T22:00:17.991497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:00:17.991497+0000","log_size":144,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11381959,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":24135,"num_write":24,"num_write_kb":12965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":20850253,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"10f","length":"1"},{"start":"128","length":"1"},{"start":"145","length":"1"}]},{"pgid":"4.15","version":"69'13","reported_seq":"340","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-24T22:00:25.251457+0000","last_change":"2021-03-24T22:00:04.298050+0000","last_active":"2021-03-24T22:00:25.251457+0000","last_peered":"2021-03-24T22:00:25.251457+0000","last_clean":"2021-03-24T22:00:25.251457+0000","last_became_active":"2021-03-24T22:00:04.297636+0000","last_became_peered":"2021-03-24T22:00:04.297636+0000","last_unstale":"2021-03-24T22:00:25.251457+0000","last_undegraded":"2021-03-24T22:00:25.251457+0000","last_fullsized":"2021-03-24T22:00:25.251457+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2021-03-24T21:56:47.168558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:47.168558+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":14,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":"422","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.278618+0000","last_change":"2021-03-24T21:59:47.326068+0000","last_active":"2021-03-24T22:00:22.278618+0000","last_peered":"2021-03-24T22:00:22.278618+0000","last_clean":"2021-03-24T22:00:22.278618+0000","last_became_active":"2021-03-24T21:59:47.325743+0000","last_became_peered":"2021-03-24T21:59:47.325743+0000","last_unstale":"2021-03-24T22:00:22.278618+0000","last_undegraded":"2021-03-24T22:00:22.278618+0000","last_fullsized":"2021-03-24T22:00:22.278618+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:32.279119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:05.473348+0000","last_clean_scrub_stamp":"2021-03-24T21:56:32.279119+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"418'5","reported_seq":"540","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.287905+0000","last_change":"2021-03-24T21:59:47.521075+0000","last_active":"2021-03-24T22:00:22.287905+0000","last_peered":"2021-03-24T22:00:22.287905+0000","last_clean":"2021-03-24T22:00:22.287905+0000","last_became_active":"2021-03-24T21:59:47.327424+0000","last_became_peered":"2021-03-24T21:59:47.327424+0000","last_unstale":"2021-03-24T22:00:22.287905+0000","last_undegraded":"2021-03-24T22:00:22.287905+0000","last_fullsized":"2021-03-24T22:00:22.287905+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"235'4","last_scrub_stamp":"2021-03-24T21:58:20.324557+0000","last_deep_scrub":"86'3","last_deep_scrub_stamp":"2021-03-24T21:52:56.678111+0000","last_clean_scrub_stamp":"2021-03-24T21:58:20.324557+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"424","reported_epoch":"451","state":"active+clean","last_fresh":"2021-03-24T22:00:21.279091+0000","last_change":"2021-03-24T22:00:03.283133+0000","last_active":"2021-03-24T22:00:21.279091+0000","last_peered":"2021-03-24T22:00:21.279091+0000","last_clean":"2021-03-24T22:00:21.279091+0000","last_became_active":"2021-03-24T22:00:03.282474+0000","last_became_peered":"2021-03-24T22:00:03.282474+0000","last_unstale":"2021-03-24T22:00:21.279091+0000","last_undegraded":"2021-03-24T22:00:21.279091+0000","last_fullsized":"2021-03-24T22:00:21.279091+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:51.924963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:51.924963+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.16","version":"452'238","reported_seq":"1388","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.302249+0000","last_change":"2021-03-24T22:00:22.302678+0000","last_active":"2021-03-24T22:00:22.302249+0000","last_peered":"2021-03-24T22:00:22.302249+0000","last_clean":"2021-03-24T22:00:22.302249+0000","last_became_active":"2021-03-24T21:59:55.246836+0000","last_became_peered":"2021-03-24T21:59:55.246836+0000","last_unstale":"2021-03-24T22:00:22.302249+0000","last_undegraded":"2021-03-24T22:00:22.302249+0000","last_fullsized":"2021-03-24T22:00:22.302249+0000","mapping_epoch":424,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":425,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'146","last_scrub_stamp":"2021-03-24T21:57:22.669305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:57:22.669305+0000","log_size":238,"ondisk_log_size":238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7574837,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":26578,"num_write":23,"num_write_kb":5165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":4218563,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e7","length":"1"},{"start":"f8","length":"1"},{"start":"10c","length":"1"},{"start":"10f","length":"1"},{"start":"126","length":"1"},{"start":"128","length":"1"},{"start":"145","length":"1"}]},{"pgid":"4.14","version":"69'14","reported_seq":"506","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.288004+0000","last_change":"2021-03-24T22:00:05.830844+0000","last_active":"2021-03-24T22:00:22.288004+0000","last_peered":"2021-03-24T22:00:22.288004+0000","last_clean":"2021-03-24T22:00:22.288004+0000","last_became_active":"2021-03-24T22:00:04.297192+0000","last_became_peered":"2021-03-24T22:00:04.297192+0000","last_unstale":"2021-03-24T22:00:22.288004+0000","last_undegraded":"2021-03-24T22:00:22.288004+0000","last_fullsized":"2021-03-24T22:00:22.288004+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'14","last_scrub_stamp":"2021-03-24T21:58:57.435358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:58:57.435358+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":27,"num_read_kb":14,"num_write":14,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":"47","reported_epoch":"451","state":"active+clean","last_fresh":"2021-03-24T22:00:21.279721+0000","last_change":"2021-03-24T21:59:47.273141+0000","last_active":"2021-03-24T22:00:21.279721+0000","last_peered":"2021-03-24T22:00:21.279721+0000","last_clean":"2021-03-24T22:00:21.279721+0000","last_became_active":"2021-03-24T21:59:47.272761+0000","last_became_peered":"2021-03-24T21:59:47.272761+0000","last_unstale":"2021-03-24T22:00:21.279721+0000","last_undegraded":"2021-03-24T22:00:21.279721+0000","last_fullsized":"2021-03-24T22:00:21.279721+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:15.949538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:59:15.949538+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"445'6","reported_seq":"413","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-24T22:00:25.253719+0000","last_change":"2021-03-24T22:00:24.314998+0000","last_active":"2021-03-24T22:00:25.253719+0000","last_peered":"2021-03-24T22:00:25.253719+0000","last_clean":"2021-03-24T22:00:25.253719+0000","last_became_active":"2021-03-24T22:00:03.301161+0000","last_became_peered":"2021-03-24T22:00:03.301161+0000","last_unstale":"2021-03-24T22:00:25.253719+0000","last_undegraded":"2021-03-24T22:00:25.253719+0000","last_fullsized":"2021-03-24T22:00:25.253719+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"445'6","last_scrub_stamp":"2021-03-24T22:00:24.314895+0000","last_deep_scrub":"86'3","last_deep_scrub_stamp":"2021-03-24T21:53:08.815163+0000","last_clean_scrub_stamp":"2021-03-24T22:00:24.314895+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"416","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.290359+0000","last_change":"2021-03-24T21:59:47.313184+0000","last_active":"2021-03-24T22:00:22.290359+0000","last_peered":"2021-03-24T22:00:22.290359+0000","last_clean":"2021-03-24T22:00:22.290359+0000","last_became_active":"2021-03-24T21:59:47.312870+0000","last_became_peered":"2021-03-24T21:59:47.312870+0000","last_unstale":"2021-03-24T22:00:22.290359+0000","last_undegraded":"2021-03-24T22:00:22.290359+0000","last_fullsized":"2021-03-24T22:00:22.290359+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:58.768675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:57:58.768675+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.17","version":"452'567","reported_seq":"1931","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-24T22:00:25.256320+0000","last_change":"2021-03-24T22:00:22.299327+0000","last_active":"2021-03-24T22:00:25.256320+0000","last_peered":"2021-03-24T22:00:25.256320+0000","last_clean":"2021-03-24T22:00:25.256320+0000","last_became_active":"2021-03-24T22:00:04.304027+0000","last_became_peered":"2021-03-24T22:00:04.304027+0000","last_unstale":"2021-03-24T22:00:25.256320+0000","last_undegraded":"2021-03-24T22:00:25.256320+0000","last_fullsized":"2021-03-24T22:00:25.256320+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"265'274","last_scrub_stamp":"2021-03-24T21:57:14.281739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:57:14.281739+0000","log_size":567,"ondisk_log_size":567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22860124,"num_objects":16,"num_object_clones":12,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":3,"num_read":142,"num_read_kb":159660,"num_write":84,"num_write_kb":34433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":59539496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"10f","length":"1"},{"start":"128","length":"1"},{"start":"145","length":"1"}]},{"pgid":"4.13","version":"69'15","reported_seq":"511","reported_epoch":"451","state":"active+clean","last_fresh":"2021-03-24T22:00:21.279578+0000","last_change":"2021-03-24T22:00:03.276111+0000","last_active":"2021-03-24T22:00:21.279578+0000","last_peered":"2021-03-24T22:00:21.279578+0000","last_clean":"2021-03-24T22:00:21.279578+0000","last_became_active":"2021-03-24T22:00:03.275721+0000","last_became_peered":"2021-03-24T22:00:03.275721+0000","last_unstale":"2021-03-24T22:00:21.279578+0000","last_undegraded":"2021-03-24T22:00:21.279578+0000","last_fullsized":"2021-03-24T22:00:21.279578+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:58:02.548166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:58:02.548166+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":"431","reported_epoch":"451","state":"active+clean","last_fresh":"2021-03-24T22:00:21.275531+0000","last_change":"2021-03-24T21:59:47.286782+0000","last_active":"2021-03-24T22:00:21.275531+0000","last_peered":"2021-03-24T22:00:21.275531+0000","last_clean":"2021-03-24T22:00:21.275531+0000","last_became_active":"2021-03-24T21:59:47.286468+0000","last_became_peered":"2021-03-24T21:59:47.286468+0000","last_unstale":"2021-03-24T22:00:21.275531+0000","last_undegraded":"2021-03-24T22:00:21.275531+0000","last_fullsized":"2021-03-24T22:00:21.275531+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:29.716186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:58:29.716186+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"278","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-24T22:00:25.253694+0000","last_change":"2021-03-24T22:00:04.291933+0000","last_active":"2021-03-24T22:00:25.253694+0000","last_peered":"2021-03-24T22:00:25.253694+0000","last_clean":"2021-03-24T22:00:25.253694+0000","last_became_active":"2021-03-24T22:00:04.291385+0000","last_became_peered":"2021-03-24T22:00:04.291385+0000","last_unstale":"2021-03-24T22:00:25.253694+0000","last_undegraded":"2021-03-24T22:00:25.253694+0000","last_fullsized":"2021-03-24T22:00:25.253694+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:31.118050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:57:31.118050+0000","last_clean_scrub_stamp":"2021-03-24T21:57:31.118050+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"422","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.295691+0000","last_change":"2021-03-24T22:00:21.298498+0000","last_active":"2021-03-24T22:00:22.295691+0000","last_peered":"2021-03-24T22:00:22.295691+0000","last_clean":"2021-03-24T22:00:22.295691+0000","last_became_active":"2021-03-24T22:00:03.282227+0000","last_became_peered":"2021-03-24T22:00:03.282227+0000","last_unstale":"2021-03-24T22:00:22.295691+0000","last_undegraded":"2021-03-24T22:00:22.295691+0000","last_fullsized":"2021-03-24T22:00:22.295691+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:21.298330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:23.707914+0000","last_clean_scrub_stamp":"2021-03-24T22:00:21.298330+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.10","version":"450'51","reported_seq":"278","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-24T22:00:25.254544+0000","last_change":"2021-03-24T22:00:22.286352+0000","last_active":"2021-03-24T22:00:25.254544+0000","last_peered":"2021-03-24T22:00:25.254544+0000","last_clean":"2021-03-24T22:00:25.254544+0000","last_became_active":"2021-03-24T22:00:04.261537+0000","last_became_peered":"2021-03-24T22:00:04.261537+0000","last_unstale":"2021-03-24T22:00:25.254544+0000","last_undegraded":"2021-03-24T22:00:25.254544+0000","last_fullsized":"2021-03-24T22:00:25.254544+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":5,"last_scrub":"277'24","last_scrub_stamp":"2021-03-24T21:57:28.119420+0000","last_deep_scrub":"277'24","last_deep_scrub_stamp":"2021-03-24T21:57:28.119420+0000","last_clean_scrub_stamp":"2021-03-24T21:57:28.119420+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3664025,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":25,"num_read_kb":18239,"num_write":17,"num_write_kb":6809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"10f","length":"1"},{"start":"128","length":"1"},{"start":"145","length":"1"}]},{"pgid":"4.12","version":"69'15","reported_seq":"443","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-24T22:00:25.255170+0000","last_change":"2021-03-24T22:00:03.281188+0000","last_active":"2021-03-24T22:00:25.255170+0000","last_peered":"2021-03-24T22:00:25.255170+0000","last_clean":"2021-03-24T22:00:25.255170+0000","last_became_active":"2021-03-24T22:00:03.280309+0000","last_became_peered":"2021-03-24T22:00:03.280309+0000","last_unstale":"2021-03-24T22:00:25.255170+0000","last_undegraded":"2021-03-24T22:00:25.255170+0000","last_fullsized":"2021-03-24T22:00:25.255170+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:57:52.248247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:57:52.248247+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":34,"num_read_kb":16,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":"48","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.286124+0000","last_change":"2021-03-24T21:59:47.329681+0000","last_active":"2021-03-24T22:00:22.286124+0000","last_peered":"2021-03-24T22:00:22.286124+0000","last_clean":"2021-03-24T22:00:22.286124+0000","last_became_active":"2021-03-24T21:59:47.329293+0000","last_became_peered":"2021-03-24T21:59:47.329293+0000","last_unstale":"2021-03-24T22:00:22.286124+0000","last_undegraded":"2021-03-24T22:00:22.286124+0000","last_fullsized":"2021-03-24T22:00:22.286124+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:04.965369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:58:04.965369+0000","last_clean_scrub_stamp":"2021-03-24T21:58:04.965369+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"428","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.287760+0000","last_change":"2021-03-24T22:00:03.287828+0000","last_active":"2021-03-24T22:00:22.287760+0000","last_peered":"2021-03-24T22:00:22.287760+0000","last_clean":"2021-03-24T22:00:22.287760+0000","last_became_active":"2021-03-24T22:00:03.287197+0000","last_became_peered":"2021-03-24T22:00:03.287197+0000","last_unstale":"2021-03-24T22:00:22.287760+0000","last_undegraded":"2021-03-24T22:00:22.287760+0000","last_fullsized":"2021-03-24T22:00:22.287760+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:32.480115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:56:32.480115+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"431","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.277197+0000","last_change":"2021-03-24T22:00:03.301429+0000","last_active":"2021-03-24T22:00:22.277197+0000","last_peered":"2021-03-24T22:00:22.277197+0000","last_clean":"2021-03-24T22:00:22.277197+0000","last_became_active":"2021-03-24T22:00:03.301079+0000","last_became_peered":"2021-03-24T22:00:03.301079+0000","last_unstale":"2021-03-24T22:00:22.277197+0000","last_undegraded":"2021-03-24T22:00:22.277197+0000","last_fullsized":"2021-03-24T22:00:22.277197+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:52.226117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:52.226117+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.11","version":"453'571","reported_seq":"1816","reported_epoch":"453","state":"active+clean","last_fresh":"2021-03-24T22:00:23.293600+0000","last_change":"2021-03-24T22:00:22.302924+0000","last_active":"2021-03-24T22:00:23.293600+0000","last_peered":"2021-03-24T22:00:23.293600+0000","last_clean":"2021-03-24T22:00:23.293600+0000","last_became_active":"2021-03-24T21:59:59.325034+0000","last_became_peered":"2021-03-24T21:59:59.325034+0000","last_unstale":"2021-03-24T22:00:23.293600+0000","last_undegraded":"2021-03-24T22:00:23.293600+0000","last_fullsized":"2021-03-24T22:00:23.293600+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":5,"last_scrub":"381'425","last_scrub_stamp":"2021-03-24T21:59:11.936662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:59:11.936662+0000","log_size":571,"ondisk_log_size":571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29503590,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":83,"num_read_kb":98151,"num_write":77,"num_write_kb":26557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":24220548,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"e7","length":"1"},{"start":"f8","length":"1"},{"start":"10c","length":"1"},{"start":"10f","length":"1"},{"start":"126","length":"1"},{"start":"128","length":"1"},{"start":"145","length":"1"}]},{"pgid":"4.11","version":"86'22","reported_seq":"515","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.295309+0000","last_change":"2021-03-24T22:00:13.347253+0000","last_active":"2021-03-24T22:00:22.295309+0000","last_peered":"2021-03-24T22:00:22.295309+0000","last_clean":"2021-03-24T22:00:22.295309+0000","last_became_active":"2021-03-24T22:00:04.294762+0000","last_became_peered":"2021-03-24T22:00:04.294762+0000","last_unstale":"2021-03-24T22:00:22.295309+0000","last_undegraded":"2021-03-24T22:00:22.295309+0000","last_fullsized":"2021-03-24T22:00:22.295309+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'22","last_scrub_stamp":"2021-03-24T22:00:13.347155+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2021-03-24T21:52:54.597505+0000","last_clean_scrub_stamp":"2021-03-24T22:00:13.347155+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":16,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":"311","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-24T22:00:25.253631+0000","last_change":"2021-03-24T22:00:04.282872+0000","last_active":"2021-03-24T22:00:25.253631+0000","last_peered":"2021-03-24T22:00:25.253631+0000","last_clean":"2021-03-24T22:00:25.253631+0000","last_became_active":"2021-03-24T22:00:04.282282+0000","last_became_peered":"2021-03-24T22:00:04.282282+0000","last_unstale":"2021-03-24T22:00:25.253631+0000","last_undegraded":"2021-03-24T22:00:25.253631+0000","last_fullsized":"2021-03-24T22:00:25.253631+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:13.016117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:13.016117+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"107","reported_epoch":"451","state":"active+clean","last_fresh":"2021-03-24T22:00:21.273685+0000","last_change":"2021-03-24T22:00:04.268074+0000","last_active":"2021-03-24T22:00:21.273685+0000","last_peered":"2021-03-24T22:00:21.273685+0000","last_clean":"2021-03-24T22:00:21.273685+0000","last_became_active":"2021-03-24T22:00:04.267817+0000","last_became_peered":"2021-03-24T22:00:04.267817+0000","last_unstale":"2021-03-24T22:00:21.273685+0000","last_undegraded":"2021-03-24T22:00:21.273685+0000","last_fullsized":"2021-03-24T22:00:21.273685+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:41.839852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:59:41.839852+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"416","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.288309+0000","last_change":"2021-03-24T21:59:15.289579+0000","last_active":"2021-03-24T22:00:22.288309+0000","last_peered":"2021-03-24T22:00:22.288309+0000","last_clean":"2021-03-24T22:00:22.288309+0000","last_became_active":"2021-03-24T21:54:24.202686+0000","last_became_peered":"2021-03-24T21:54:24.202686+0000","last_unstale":"2021-03-24T22:00:22.288309+0000","last_undegraded":"2021-03-24T22:00:22.288309+0000","last_fullsized":"2021-03-24T22:00:22.288309+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:15.289444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:47.540793+0000","last_clean_scrub_stamp":"2021-03-24T21:59:15.289444+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.12","version":"183'26","reported_seq":"777","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.279282+0000","last_change":"2021-03-24T22:00:22.279282+0000","last_active":"2021-03-24T22:00:22.279282+0000","last_peered":"2021-03-24T22:00:22.279282+0000","last_clean":"2021-03-24T22:00:22.279282+0000","last_became_active":"2021-03-24T22:00:04.295945+0000","last_became_peered":"2021-03-24T22:00:04.295945+0000","last_unstale":"2021-03-24T22:00:22.279282+0000","last_undegraded":"2021-03-24T22:00:22.279282+0000","last_fullsized":"2021-03-24T22:00:22.279282+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":5,"last_scrub":"183'26","last_scrub_stamp":"2021-03-24T21:57:53.859843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:57:53.859843+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e7","length":"1"},{"start":"f8","length":"1"},{"start":"10c","length":"1"},{"start":"10f","length":"1"},{"start":"126","length":"1"},{"start":"128","length":"1"},{"start":"145","length":"1"}]},{"pgid":"4.10","version":"69'12","reported_seq":"543","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.286634+0000","last_change":"2021-03-24T22:00:03.289423+0000","last_active":"2021-03-24T22:00:22.286634+0000","last_peered":"2021-03-24T22:00:22.286634+0000","last_clean":"2021-03-24T22:00:22.286634+0000","last_became_active":"2021-03-24T22:00:03.288854+0000","last_became_peered":"2021-03-24T22:00:03.288854+0000","last_unstale":"2021-03-24T22:00:22.286634+0000","last_undegraded":"2021-03-24T22:00:22.286634+0000","last_fullsized":"2021-03-24T22:00:22.286634+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2021-03-24T21:56:53.443554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:56:53.443554+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":12,"num_write":12,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":3756,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":"428","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.281284+0000","last_change":"2021-03-24T22:00:03.302355+0000","last_active":"2021-03-24T22:00:22.281284+0000","last_peered":"2021-03-24T22:00:22.281284+0000","last_clean":"2021-03-24T22:00:22.281284+0000","last_became_active":"2021-03-24T22:00:03.301753+0000","last_became_peered":"2021-03-24T22:00:03.301753+0000","last_unstale":"2021-03-24T22:00:22.281284+0000","last_undegraded":"2021-03-24T22:00:22.281284+0000","last_fullsized":"2021-03-24T22:00:22.281284+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:11.229895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:11.229895+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"380","reported_epoch":"451","state":"active+clean","last_fresh":"2021-03-24T22:00:21.275508+0000","last_change":"2021-03-24T21:59:47.306611+0000","last_active":"2021-03-24T22:00:21.275508+0000","last_peered":"2021-03-24T22:00:21.275508+0000","last_clean":"2021-03-24T22:00:21.275508+0000","last_became_active":"2021-03-24T21:59:47.306180+0000","last_became_peered":"2021-03-24T21:59:47.306180+0000","last_unstale":"2021-03-24T22:00:21.275508+0000","last_undegraded":"2021-03-24T22:00:21.275508+0000","last_fullsized":"2021-03-24T22:00:21.275508+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:56.548933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:57:56.548933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"51","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-24T22:00:25.255518+0000","last_change":"2021-03-24T21:59:47.300473+0000","last_active":"2021-03-24T22:00:25.255518+0000","last_peered":"2021-03-24T22:00:25.255518+0000","last_clean":"2021-03-24T22:00:25.255518+0000","last_became_active":"2021-03-24T21:59:47.300126+0000","last_became_peered":"2021-03-24T21:59:47.300126+0000","last_unstale":"2021-03-24T22:00:25.255518+0000","last_undegraded":"2021-03-24T22:00:25.255518+0000","last_fullsized":"2021-03-24T22:00:25.255518+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:59.979772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:57:59.979772+0000","last_clean_scrub_stamp":"2021-03-24T21:57:59.979772+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.13","version":"452'396","reported_seq":"1653","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.303045+0000","last_change":"2021-03-24T22:00:22.303045+0000","last_active":"2021-03-24T22:00:22.303045+0000","last_peered":"2021-03-24T22:00:22.303045+0000","last_clean":"2021-03-24T22:00:22.303045+0000","last_became_active":"2021-03-24T22:00:04.301660+0000","last_became_peered":"2021-03-24T22:00:04.301660+0000","last_unstale":"2021-03-24T22:00:22.303045+0000","last_undegraded":"2021-03-24T22:00:22.303045+0000","last_fullsized":"2021-03-24T22:00:22.303045+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":396,"ondisk_log_size":396,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25983233,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":156,"num_read_kb":142965,"num_write":143,"num_write_kb":60186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":55510645,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e7","length":"1"},{"start":"f8","length":"1"},{"start":"10c","length":"1"},{"start":"10f","length":"1"},{"start":"126","length":"1"},{"start":"128","length":"1"},{"start":"145","length":"1"}]},{"pgid":"3.18","version":"0'0","reported_seq":"422","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.285246+0000","last_change":"2021-03-24T21:59:00.393853+0000","last_active":"2021-03-24T22:00:22.285246+0000","last_peered":"2021-03-24T22:00:22.285246+0000","last_clean":"2021-03-24T22:00:22.285246+0000","last_became_active":"2021-03-24T21:54:38.145870+0000","last_became_peered":"2021-03-24T21:54:38.145870+0000","last_unstale":"2021-03-24T22:00:22.285246+0000","last_undegraded":"2021-03-24T22:00:22.285246+0000","last_fullsized":"2021-03-24T22:00:22.285246+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:00.393742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:59:00.393742+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"427","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.291162+0000","last_change":"2021-03-24T21:59:47.325236+0000","last_active":"2021-03-24T22:00:22.291162+0000","last_peered":"2021-03-24T22:00:22.291162+0000","last_clean":"2021-03-24T22:00:22.291162+0000","last_became_active":"2021-03-24T21:59:47.324868+0000","last_became_peered":"2021-03-24T21:59:47.324868+0000","last_unstale":"2021-03-24T22:00:22.291162+0000","last_undegraded":"2021-03-24T22:00:22.291162+0000","last_fullsized":"2021-03-24T22:00:22.291162+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:17.040623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:59:17.040623+0000","last_clean_scrub_stamp":"2021-03-24T21:59:17.040623+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"290","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-24T22:00:25.253778+0000","last_change":"2021-03-24T22:00:15.203127+0000","last_active":"2021-03-24T22:00:25.253778+0000","last_peered":"2021-03-24T22:00:25.253778+0000","last_clean":"2021-03-24T22:00:25.253778+0000","last_became_active":"2021-03-24T22:00:03.285276+0000","last_became_peered":"2021-03-24T22:00:03.285276+0000","last_unstale":"2021-03-24T22:00:25.253778+0000","last_undegraded":"2021-03-24T22:00:25.253778+0000","last_fullsized":"2021-03-24T22:00:25.253778+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:26.887939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:58:26.887939+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1c","version":"273'245","reported_seq":"1382","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.289519+0000","last_change":"2021-03-24T22:00:22.289519+0000","last_active":"2021-03-24T22:00:22.289519+0000","last_peered":"2021-03-24T22:00:22.289519+0000","last_clean":"2021-03-24T22:00:22.289519+0000","last_became_active":"2021-03-24T22:00:04.295452+0000","last_became_peered":"2021-03-24T22:00:04.295452+0000","last_unstale":"2021-03-24T22:00:22.289519+0000","last_undegraded":"2021-03-24T22:00:22.289519+0000","last_fullsized":"2021-03-24T22:00:22.289519+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":245,"ondisk_log_size":245,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8581786,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":70,"num_read_kb":67597,"num_write":52,"num_write_kb":18055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"e7","length":"1"},{"start":"f8","length":"1"},{"start":"10c","length":"1"},{"start":"10f","length":"1"},{"start":"126","length":"1"},{"start":"128","length":"1"},{"start":"145","length":"1"}]},{"pgid":"3.19","version":"0'0","reported_seq":"397","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.295645+0000","last_change":"2021-03-24T22:00:03.282661+0000","last_active":"2021-03-24T22:00:22.295645+0000","last_peered":"2021-03-24T22:00:22.295645+0000","last_clean":"2021-03-24T22:00:22.295645+0000","last_became_active":"2021-03-24T22:00:03.281791+0000","last_became_peered":"2021-03-24T22:00:03.281791+0000","last_unstale":"2021-03-24T22:00:22.295645+0000","last_undegraded":"2021-03-24T22:00:22.295645+0000","last_fullsized":"2021-03-24T22:00:22.295645+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:26.417613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:57:26.417613+0000","last_clean_scrub_stamp":"2021-03-24T21:57:26.417613+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"235'4","reported_seq":"469","reported_epoch":"451","state":"active+clean","last_fresh":"2021-03-24T22:00:21.277101+0000","last_change":"2021-03-24T21:59:04.626032+0000","last_active":"2021-03-24T22:00:21.277101+0000","last_peered":"2021-03-24T22:00:21.277101+0000","last_clean":"2021-03-24T22:00:21.277101+0000","last_became_active":"2021-03-24T21:56:44.228196+0000","last_became_peered":"2021-03-24T21:56:44.228196+0000","last_unstale":"2021-03-24T22:00:21.277101+0000","last_undegraded":"2021-03-24T22:00:21.277101+0000","last_fullsized":"2021-03-24T22:00:21.277101+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"235'4","last_scrub_stamp":"2021-03-24T21:59:04.625954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:59:04.625954+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"185","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-24T22:00:25.256343+0000","last_change":"2021-03-24T22:00:04.300000+0000","last_active":"2021-03-24T22:00:25.256343+0000","last_peered":"2021-03-24T22:00:25.256343+0000","last_clean":"2021-03-24T22:00:25.256343+0000","last_became_active":"2021-03-24T22:00:04.299392+0000","last_became_peered":"2021-03-24T22:00:04.299392+0000","last_unstale":"2021-03-24T22:00:25.256343+0000","last_undegraded":"2021-03-24T22:00:25.256343+0000","last_fullsized":"2021-03-24T22:00:25.256343+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:26.129924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:57:26.129924+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1d","version":"451'366","reported_seq":"1625","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.292902+0000","last_change":"2021-03-24T22:00:22.292902+0000","last_active":"2021-03-24T22:00:22.292902+0000","last_peered":"2021-03-24T22:00:22.292902+0000","last_clean":"2021-03-24T22:00:22.292902+0000","last_became_active":"2021-03-24T22:00:04.305837+0000","last_became_peered":"2021-03-24T22:00:04.305837+0000","last_unstale":"2021-03-24T22:00:22.292902+0000","last_undegraded":"2021-03-24T22:00:22.292902+0000","last_fullsized":"2021-03-24T22:00:22.292902+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":5,"last_scrub":"233'169","last_scrub_stamp":"2021-03-24T21:56:47.285648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:56:47.285648+0000","log_size":366,"ondisk_log_size":366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18884844,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":120,"num_read_kb":125379,"num_write":99,"num_write_kb":45593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":35584547,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e7","length":"1"},{"start":"f8","length":"1"},{"start":"10c","length":"1"},{"start":"10f","length":"1"},{"start":"126","length":"1"},{"start":"128","length":"1"},{"start":"145","length":"1"}]},{"pgid":"3.1a","version":"0'0","reported_seq":"427","reported_epoch":"451","state":"active+clean","last_fresh":"2021-03-24T22:00:21.275730+0000","last_change":"2021-03-24T21:59:12.679897+0000","last_active":"2021-03-24T22:00:21.275730+0000","last_peered":"2021-03-24T22:00:21.275730+0000","last_clean":"2021-03-24T22:00:21.275730+0000","last_became_active":"2021-03-24T21:54:38.143876+0000","last_became_peered":"2021-03-24T21:54:38.143876+0000","last_unstale":"2021-03-24T22:00:21.275730+0000","last_undegraded":"2021-03-24T22:00:21.275730+0000","last_fullsized":"2021-03-24T22:00:21.275730+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:12.679659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:52:54.212459+0000","last_clean_scrub_stamp":"2021-03-24T21:59:12.679659+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"409","reported_epoch":"451","state":"active+clean","last_fresh":"2021-03-24T22:00:21.275873+0000","last_change":"2021-03-24T21:57:19.767717+0000","last_active":"2021-03-24T22:00:21.275873+0000","last_peered":"2021-03-24T22:00:21.275873+0000","last_clean":"2021-03-24T22:00:21.275873+0000","last_became_active":"2021-03-24T21:51:49.410920+0000","last_became_peered":"2021-03-24T21:51:49.410920+0000","last_unstale":"2021-03-24T22:00:21.275873+0000","last_undegraded":"2021-03-24T22:00:21.275873+0000","last_fullsized":"2021-03-24T22:00:21.275873+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:19.767465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:57:19.767465+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"435","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.286466+0000","last_change":"2021-03-24T22:00:03.282644+0000","last_active":"2021-03-24T22:00:22.286466+0000","last_peered":"2021-03-24T22:00:22.286466+0000","last_clean":"2021-03-24T22:00:22.286466+0000","last_became_active":"2021-03-24T22:00:03.282313+0000","last_became_peered":"2021-03-24T22:00:03.282313+0000","last_unstale":"2021-03-24T22:00:22.286466+0000","last_undegraded":"2021-03-24T22:00:22.286466+0000","last_fullsized":"2021-03-24T22:00:22.286466+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:33.440618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:33.440618+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1e","version":"265'45","reported_seq":"528","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.288924+0000","last_change":"2021-03-24T22:00:22.288924+0000","last_active":"2021-03-24T22:00:22.288924+0000","last_peered":"2021-03-24T22:00:22.288924+0000","last_clean":"2021-03-24T22:00:22.288924+0000","last_became_active":"2021-03-24T22:00:03.290576+0000","last_became_peered":"2021-03-24T22:00:03.290576+0000","last_unstale":"2021-03-24T22:00:22.288924+0000","last_undegraded":"2021-03-24T22:00:22.288924+0000","last_fullsized":"2021-03-24T22:00:22.288924+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":5,"last_scrub":"199'30","last_scrub_stamp":"2021-03-24T21:56:10.105084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:56:10.105084+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":18305,"num_write":13,"num_write_kb":3060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"e7","length":"1"},{"start":"f8","length":"1"},{"start":"10c","length":"1"},{"start":"10f","length":"1"},{"start":"126","length":"1"},{"start":"128","length":"1"},{"start":"145","length":"1"}]},{"pgid":"3.1b","version":"0'0","reported_seq":"422","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.290335+0000","last_change":"2021-03-24T21:59:47.326265+0000","last_active":"2021-03-24T22:00:22.290335+0000","last_peered":"2021-03-24T22:00:22.290335+0000","last_clean":"2021-03-24T22:00:22.290335+0000","last_became_active":"2021-03-24T21:59:47.325941+0000","last_became_peered":"2021-03-24T21:59:47.325941+0000","last_unstale":"2021-03-24T22:00:22.290335+0000","last_undegraded":"2021-03-24T22:00:22.290335+0000","last_fullsized":"2021-03-24T22:00:22.290335+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:42.254536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:42.254536+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"380","reported_epoch":"451","state":"active+clean","last_fresh":"2021-03-24T22:00:21.275176+0000","last_change":"2021-03-24T22:00:03.287252+0000","last_active":"2021-03-24T22:00:21.275176+0000","last_peered":"2021-03-24T22:00:21.275176+0000","last_clean":"2021-03-24T22:00:21.275176+0000","last_became_active":"2021-03-24T22:00:03.286931+0000","last_became_peered":"2021-03-24T22:00:03.286931+0000","last_unstale":"2021-03-24T22:00:21.275176+0000","last_undegraded":"2021-03-24T22:00:21.275176+0000","last_fullsized":"2021-03-24T22:00:21.275176+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:26.642883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:57:26.642883+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"426","reported_epoch":"451","state":"active+clean","last_fresh":"2021-03-24T22:00:21.277498+0000","last_change":"2021-03-24T22:00:03.289998+0000","last_active":"2021-03-24T22:00:21.277498+0000","last_peered":"2021-03-24T22:00:21.277498+0000","last_clean":"2021-03-24T22:00:21.277498+0000","last_became_active":"2021-03-24T22:00:03.289090+0000","last_became_peered":"2021-03-24T22:00:03.289090+0000","last_unstale":"2021-03-24T22:00:21.277498+0000","last_undegraded":"2021-03-24T22:00:21.277498+0000","last_fullsized":"2021-03-24T22:00:21.277498+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:38.075714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:56:38.075714+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1f","version":"272'99","reported_seq":"527","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.293149+0000","last_change":"2021-03-24T22:00:22.293564+0000","last_active":"2021-03-24T22:00:22.293149+0000","last_peered":"2021-03-24T22:00:22.293149+0000","last_clean":"2021-03-24T22:00:22.293149+0000","last_became_active":"2021-03-24T22:00:03.289079+0000","last_became_peered":"2021-03-24T22:00:03.289079+0000","last_unstale":"2021-03-24T22:00:22.293149+0000","last_undegraded":"2021-03-24T22:00:22.293149+0000","last_fullsized":"2021-03-24T22:00:22.293149+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":5,"last_scrub":"218'73","last_scrub_stamp":"2021-03-24T21:56:48.267711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:56:48.267711+0000","log_size":99,"ondisk_log_size":99,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3571504,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":43,"num_read_kb":44484,"num_write":22,"num_write_kb":9163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":942483,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e7","length":"1"},{"start":"f8","length":"1"},{"start":"10c","length":"1"},{"start":"10f","length":"1"},{"start":"126","length":"1"},{"start":"128","length":"1"},{"start":"145","length":"1"}]},{"pgid":"3.1c","version":"0'0","reported_seq":"423","reported_epoch":"451","state":"active+clean","last_fresh":"2021-03-24T22:00:21.275304+0000","last_change":"2021-03-24T22:00:04.293108+0000","last_active":"2021-03-24T22:00:21.275304+0000","last_peered":"2021-03-24T22:00:21.275304+0000","last_clean":"2021-03-24T22:00:21.275304+0000","last_became_active":"2021-03-24T22:00:04.292844+0000","last_became_peered":"2021-03-24T22:00:04.292844+0000","last_unstale":"2021-03-24T22:00:21.275304+0000","last_undegraded":"2021-03-24T22:00:21.275304+0000","last_fullsized":"2021-03-24T22:00:21.275304+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:47.920949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:47.920949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"103","reported_epoch":"451","state":"active+clean","last_fresh":"2021-03-24T22:00:21.276137+0000","last_change":"2021-03-24T21:59:47.306088+0000","last_active":"2021-03-24T22:00:21.276137+0000","last_peered":"2021-03-24T22:00:21.276137+0000","last_clean":"2021-03-24T22:00:21.276137+0000","last_became_active":"2021-03-24T21:59:47.305661+0000","last_became_peered":"2021-03-24T21:59:47.305661+0000","last_unstale":"2021-03-24T22:00:21.276137+0000","last_undegraded":"2021-03-24T22:00:21.276137+0000","last_fullsized":"2021-03-24T22:00:21.276137+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:08.942247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:56:51.977306+0000","last_clean_scrub_stamp":"2021-03-24T21:59:08.942247+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"311","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-24T22:00:25.252301+0000","last_change":"2021-03-24T22:00:15.201993+0000","last_active":"2021-03-24T22:00:25.252301+0000","last_peered":"2021-03-24T22:00:25.252301+0000","last_clean":"2021-03-24T22:00:25.252301+0000","last_became_active":"2021-03-24T22:00:03.281434+0000","last_became_peered":"2021-03-24T22:00:03.281434+0000","last_unstale":"2021-03-24T22:00:25.252301+0000","last_undegraded":"2021-03-24T22:00:25.252301+0000","last_fullsized":"2021-03-24T22:00:25.252301+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:16.076982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:57:16.076982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.18","version":"452'114","reported_seq":"863","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.291971+0000","last_change":"2021-03-24T22:00:22.291971+0000","last_active":"2021-03-24T22:00:22.291971+0000","last_peered":"2021-03-24T22:00:22.291971+0000","last_clean":"2021-03-24T22:00:22.291971+0000","last_became_active":"2021-03-24T22:00:04.300565+0000","last_became_peered":"2021-03-24T22:00:04.300565+0000","last_unstale":"2021-03-24T22:00:22.291971+0000","last_undegraded":"2021-03-24T22:00:22.291971+0000","last_fullsized":"2021-03-24T22:00:22.291971+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":5,"last_scrub":"249'90","last_scrub_stamp":"2021-03-24T21:57:56.848784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:57:56.848784+0000","log_size":114,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3462308,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":27,"num_read_kb":30649,"num_write":6,"num_write_kb":1725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":3530210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e7","length":"1"},{"start":"f8","length":"1"},{"start":"10c","length":"1"},{"start":"10f","length":"1"},{"start":"126","length":"1"},{"start":"128","length":"1"},{"start":"145","length":"1"}]},{"pgid":"3.1d","version":"0'0","reported_seq":"462","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-24T22:00:25.256074+0000","last_change":"2021-03-24T22:00:03.297074+0000","last_active":"2021-03-24T22:00:25.256074+0000","last_peered":"2021-03-24T22:00:25.256074+0000","last_clean":"2021-03-24T22:00:25.256074+0000","last_became_active":"2021-03-24T22:00:03.296769+0000","last_became_peered":"2021-03-24T22:00:03.296769+0000","last_unstale":"2021-03-24T22:00:25.256074+0000","last_undegraded":"2021-03-24T22:00:25.256074+0000","last_fullsized":"2021-03-24T22:00:25.256074+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:24.418893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:58:24.418893+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"312","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-24T22:00:25.251893+0000","last_change":"2021-03-24T22:00:22.297302+0000","last_active":"2021-03-24T22:00:25.251893+0000","last_peered":"2021-03-24T22:00:25.251893+0000","last_clean":"2021-03-24T22:00:25.251893+0000","last_became_active":"2021-03-24T22:00:04.290377+0000","last_became_peered":"2021-03-24T22:00:04.290377+0000","last_unstale":"2021-03-24T22:00:25.251893+0000","last_undegraded":"2021-03-24T22:00:25.251893+0000","last_fullsized":"2021-03-24T22:00:25.251893+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:22.297227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:00:22.297227+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"425","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.277653+0000","last_change":"2021-03-24T21:59:47.280828+0000","last_active":"2021-03-24T22:00:22.277653+0000","last_peered":"2021-03-24T22:00:22.277653+0000","last_clean":"2021-03-24T22:00:22.277653+0000","last_became_active":"2021-03-24T21:59:47.280438+0000","last_became_peered":"2021-03-24T21:59:47.280438+0000","last_unstale":"2021-03-24T22:00:22.277653+0000","last_undegraded":"2021-03-24T22:00:22.277653+0000","last_fullsized":"2021-03-24T22:00:22.277653+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:19.271351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:57:19.271351+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.19","version":"455'480","reported_seq":"1782","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-24T22:00:26.193840+0000","last_change":"2021-03-24T22:00:26.193810+0000","last_active":"2021-03-24T22:00:26.193840+0000","last_peered":"2021-03-24T22:00:26.193840+0000","last_clean":"2021-03-24T22:00:26.193840+0000","last_became_active":"2021-03-24T22:00:04.296935+0000","last_became_peered":"2021-03-24T22:00:04.296935+0000","last_unstale":"2021-03-24T22:00:26.193840+0000","last_undegraded":"2021-03-24T22:00:26.193840+0000","last_fullsized":"2021-03-24T22:00:26.193840+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"455'480","last_scrub_stamp":"2021-03-24T22:00:26.193730+0000","last_deep_scrub":"455'480","last_deep_scrub_stamp":"2021-03-24T22:00:26.193730+0000","last_clean_scrub_stamp":"2021-03-24T22:00:26.193730+0000","log_size":480,"ondisk_log_size":480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23754306,"num_objects":10,"num_object_clones":7,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":68,"num_read_kb":54122,"num_write":77,"num_write_kb":34019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":43822128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"10f","length":"1"},{"start":"128","length":"1"},{"start":"145","length":"1"}]},{"pgid":"4.19","version":"69'20","reported_seq":"84","reported_epoch":"451","state":"active+clean","last_fresh":"2021-03-24T22:00:21.275293+0000","last_change":"2021-03-24T21:59:36.691332+0000","last_active":"2021-03-24T22:00:21.275293+0000","last_peered":"2021-03-24T22:00:21.275293+0000","last_clean":"2021-03-24T22:00:21.275293+0000","last_became_active":"2021-03-24T21:59:34.394509+0000","last_became_peered":"2021-03-24T21:59:34.394509+0000","last_unstale":"2021-03-24T22:00:21.275293+0000","last_undegraded":"2021-03-24T22:00:21.275293+0000","last_fullsized":"2021-03-24T22:00:21.275293+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'20","last_scrub_stamp":"2021-03-24T21:58:31.309079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:58:31.309079+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":40,"num_read_kb":21,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":890,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":"440","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.285774+0000","last_change":"2021-03-24T21:59:47.327717+0000","last_active":"2021-03-24T22:00:22.285774+0000","last_peered":"2021-03-24T22:00:22.285774+0000","last_clean":"2021-03-24T22:00:22.285774+0000","last_became_active":"2021-03-24T21:59:47.327409+0000","last_became_peered":"2021-03-24T21:59:47.327409+0000","last_unstale":"2021-03-24T22:00:22.285774+0000","last_undegraded":"2021-03-24T22:00:22.285774+0000","last_fullsized":"2021-03-24T22:00:22.285774+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:05.405785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:59:05.405785+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"447","reported_epoch":"451","state":"active+clean","last_fresh":"2021-03-24T22:00:21.274344+0000","last_change":"2021-03-24T22:00:04.258440+0000","last_active":"2021-03-24T22:00:21.274344+0000","last_peered":"2021-03-24T22:00:21.274344+0000","last_clean":"2021-03-24T22:00:21.274344+0000","last_became_active":"2021-03-24T22:00:04.258074+0000","last_became_peered":"2021-03-24T22:00:04.258074+0000","last_unstale":"2021-03-24T22:00:21.274344+0000","last_undegraded":"2021-03-24T22:00:21.274344+0000","last_fullsized":"2021-03-24T22:00:21.274344+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:02.668717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:59:02.668717+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"434","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.287902+0000","last_change":"2021-03-24T21:59:47.327548+0000","last_active":"2021-03-24T22:00:22.287902+0000","last_peered":"2021-03-24T22:00:22.287902+0000","last_clean":"2021-03-24T22:00:22.287902+0000","last_became_active":"2021-03-24T21:59:47.327214+0000","last_became_peered":"2021-03-24T21:59:47.327214+0000","last_unstale":"2021-03-24T22:00:22.287902+0000","last_undegraded":"2021-03-24T22:00:22.287902+0000","last_fullsized":"2021-03-24T22:00:22.287902+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:29.298229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:58:29.298229+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"452'217","reported_seq":"1354","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.302192+0000","last_change":"2021-03-24T22:00:22.302619+0000","last_active":"2021-03-24T22:00:22.302192+0000","last_peered":"2021-03-24T22:00:22.302192+0000","last_clean":"2021-03-24T22:00:22.302192+0000","last_became_active":"2021-03-24T22:00:04.284963+0000","last_became_peered":"2021-03-24T22:00:04.284963+0000","last_unstale":"2021-03-24T22:00:22.302192+0000","last_undegraded":"2021-03-24T22:00:22.302192+0000","last_fullsized":"2021-03-24T22:00:22.302192+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":217,"ondisk_log_size":217,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6283787,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":59,"num_read_kb":55940,"num_write":70,"num_write_kb":22615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":19033114,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e7","length":"1"},{"start":"f8","length":"1"},{"start":"10c","length":"1"},{"start":"10f","length":"1"},{"start":"126","length":"1"},{"start":"128","length":"1"},{"start":"145","length":"1"}]},{"pgid":"4.18","version":"69'15","reported_seq":"54","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-24T22:00:25.249486+0000","last_change":"2021-03-24T22:00:06.039570+0000","last_active":"2021-03-24T22:00:25.249486+0000","last_peered":"2021-03-24T22:00:25.249486+0000","last_clean":"2021-03-24T22:00:25.249486+0000","last_became_active":"2021-03-24T22:00:04.286405+0000","last_became_peered":"2021-03-24T22:00:04.286405+0000","last_unstale":"2021-03-24T22:00:25.249486+0000","last_undegraded":"2021-03-24T22:00:25.249486+0000","last_fullsized":"2021-03-24T22:00:25.249486+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'15","last_scrub_stamp":"2021-03-24T21:58:58.719296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:58:58.719296+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":3114,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":"438","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.281067+0000","last_change":"2021-03-24T22:00:13.625119+0000","last_active":"2021-03-24T22:00:22.281067+0000","last_peered":"2021-03-24T22:00:22.281067+0000","last_clean":"2021-03-24T22:00:22.281067+0000","last_became_active":"2021-03-24T22:00:03.301019+0000","last_became_peered":"2021-03-24T22:00:03.301019+0000","last_unstale":"2021-03-24T22:00:22.281067+0000","last_undegraded":"2021-03-24T22:00:22.281067+0000","last_fullsized":"2021-03-24T22:00:22.281067+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:13.624943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:00:13.624943+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"287","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-24T22:00:25.255109+0000","last_change":"2021-03-24T22:00:04.296170+0000","last_active":"2021-03-24T22:00:25.255109+0000","last_peered":"2021-03-24T22:00:25.255109+0000","last_clean":"2021-03-24T22:00:25.255109+0000","last_became_active":"2021-03-24T22:00:04.295560+0000","last_became_peered":"2021-03-24T22:00:04.295560+0000","last_unstale":"2021-03-24T22:00:25.255109+0000","last_undegraded":"2021-03-24T22:00:25.255109+0000","last_fullsized":"2021-03-24T22:00:25.255109+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:27.920682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:14.710709+0000","last_clean_scrub_stamp":"2021-03-24T21:58:27.920682+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"442","reported_epoch":"451","state":"active+clean","last_fresh":"2021-03-24T22:00:21.275187+0000","last_change":"2021-03-24T22:00:04.283287+0000","last_active":"2021-03-24T22:00:21.275187+0000","last_peered":"2021-03-24T22:00:21.275187+0000","last_clean":"2021-03-24T22:00:21.275187+0000","last_became_active":"2021-03-24T22:00:04.282850+0000","last_became_peered":"2021-03-24T22:00:04.282850+0000","last_unstale":"2021-03-24T22:00:21.275187+0000","last_undegraded":"2021-03-24T22:00:21.275187+0000","last_fullsized":"2021-03-24T22:00:21.275187+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:15.687165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:57:15.687165+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"452'730","reported_seq":"2062","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.305681+0000","last_change":"2021-03-24T22:00:22.305681+0000","last_active":"2021-03-24T22:00:22.305681+0000","last_peered":"2021-03-24T22:00:22.305681+0000","last_clean":"2021-03-24T22:00:22.305681+0000","last_became_active":"2021-03-24T21:57:33.176110+0000","last_became_peered":"2021-03-24T21:57:33.176110+0000","last_unstale":"2021-03-24T22:00:22.305681+0000","last_undegraded":"2021-03-24T22:00:22.305681+0000","last_fullsized":"2021-03-24T22:00:22.305681+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":5,"last_scrub":"410'569","last_scrub_stamp":"2021-03-24T21:59:43.929595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:59:43.929595+0000","log_size":730,"ondisk_log_size":730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":48624779,"num_objects":18,"num_object_clones":12,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":1,"num_read":43,"num_read_kb":57569,"num_write":20,"num_write_kb":13075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e7","length":"1"},{"start":"f8","length":"1"},{"start":"10c","length":"1"},{"start":"10f","length":"1"},{"start":"126","length":"1"},{"start":"128","length":"1"},{"start":"145","length":"1"}]},{"pgid":"3.24","version":"0'0","reported_seq":"79","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.283856+0000","last_change":"2021-03-24T21:59:14.369914+0000","last_active":"2021-03-24T22:00:22.283856+0000","last_peered":"2021-03-24T22:00:22.283856+0000","last_clean":"2021-03-24T22:00:22.283856+0000","last_became_active":"2021-03-24T21:59:14.369606+0000","last_became_peered":"2021-03-24T21:59:14.369606+0000","last_unstale":"2021-03-24T22:00:22.283856+0000","last_undegraded":"2021-03-24T22:00:22.283856+0000","last_fullsized":"2021-03-24T22:00:22.283856+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:38.144956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:38.144956+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"175'22","reported_seq":"795","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-24T22:00:25.253869+0000","last_change":"2021-03-24T22:00:22.286079+0000","last_active":"2021-03-24T22:00:25.253869+0000","last_peered":"2021-03-24T22:00:25.253869+0000","last_clean":"2021-03-24T22:00:25.253869+0000","last_became_active":"2021-03-24T21:59:47.300296+0000","last_became_peered":"2021-03-24T21:59:47.300296+0000","last_unstale":"2021-03-24T22:00:25.253869+0000","last_undegraded":"2021-03-24T22:00:25.253869+0000","last_fullsized":"2021-03-24T22:00:25.253869+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":6,"last_scrub":"175'22","last_scrub_stamp":"2021-03-24T21:59:03.758935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:59:03.758935+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"10f","length":"1"},{"start":"128","length":"1"},{"start":"145","length":"1"}]},{"pgid":"3.25","version":"0'0","reported_seq":"155","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.283707+0000","last_change":"2021-03-24T22:00:04.296629+0000","last_active":"2021-03-24T22:00:22.283707+0000","last_peered":"2021-03-24T22:00:22.283707+0000","last_clean":"2021-03-24T22:00:22.283707+0000","last_became_active":"2021-03-24T22:00:04.293857+0000","last_became_peered":"2021-03-24T22:00:04.293857+0000","last_unstale":"2021-03-24T22:00:22.283707+0000","last_undegraded":"2021-03-24T22:00:22.283707+0000","last_fullsized":"2021-03-24T22:00:22.283707+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:53.013129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:53.013129+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.21","version":"452'367","reported_seq":"1652","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-24T22:00:22.300841+0000","last_change":"2021-03-24T22:00:22.300841+0000","last_active":"2021-03-24T22:00:22.300841+0000","last_peered":"2021-03-24T22:00:22.300841+0000","last_clean":"2021-03-24T22:00:22.300841+0000","last_became_active":"2021-03-24T22:00:04.298907+0000","last_became_peered":"2021-03-24T22:00:04.298907+0000","last_unstale":"2021-03-24T22:00:22.300841+0000","last_undegraded":"2021-03-24T22:00:22.300841+0000","last_fullsized":"2021-03-24T22:00:22.300841+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":367,"ondisk_log_size":367,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13632264,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":132,"num_read_kb":137092,"num_write":77,"num_write_kb":25884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":23606463,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"e7","length":"1"},{"start":"f8","length":"1"},{"start":"10c","length":"1"},{"start":"10f","length":"1"},{"start":"126","length":"1"},{"start":"128","length":"1"},{"start":"145","length":"1"}]}],"pool_stats":[{"poolid":7,"num_pg":36,"stat_sum":{"num_bytes":436599616,"num_objects":194,"num_object_clones":138,"num_object_copies":582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":194,"num_whiteouts":15,"num_read":1935,"num_read_kb":2050062,"num_write":1412,"num_write_kb":558423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":637,"num_bytes_recovered":465632740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":944738304,"data_stored":1266744748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9415,"ondisk_log_size":9415,"up":108,"acting":108,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":39,"num_read_kb":34,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":3,"num_pg":42,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1759,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":5650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":984,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":126,"acting":126,"num_store_stats":8},{"poolid":4,"num_pg":26,"stat_sum":{"num_bytes":22790,"num_objects":295,"num_object_clones":0,"num_object_copies":885,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":295,"num_whiteouts":0,"num_read":1029,"num_read_kb":548,"num_write":528,"num_write_kb":155,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":456,"num_bytes_recovered":35566,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2236416,"data_stored":80232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":372,"ondisk_log_size":372,"up":78,"acting":78,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":236223201388,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":6,"up_from":0,"seq":201863463025,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":0,"seq":55834574982,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738505,"num_pgs":115,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":218840,"kb_used_data":184528,"kb_used_omap":0,"kb_used_meta":34304,"kb_avail":93526312,"statfs":{"total":95995035648,"available":95770943488,"internally_reserved":0,"allocated":188956672,"data_stored":241405739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35127296},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Mar 24 21:59:51 2021","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.63100000000000001,"15min":0.63},"min":{"1min":0.26900000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.86299999999999999,"5min":1.2070000000000001,"15min":4.8040000000000003},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.747,"15min":0.70599999999999996},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":1.2450000000000001,"5min":2.734,"15min":8.8989999999999991},"last":1.0029999999999999}]},{"osd":2,"last update":"Wed Mar 24 22:00:11 2021","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":0.89300000000000002,"15min":0.76800000000000002},"min":{"1min":0.34999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":15.949999999999999,"5min":15.949999999999999,"15min":15.949999999999999},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.77700000000000002,"15min":0.75600000000000001},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.92600000000000005,"5min":2.5499999999999998,"15min":4.766},"last":0.94799999999999995}]},{"osd":3,"last update":"Wed Mar 24 21:59:28 2021","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.83799999999999997,"15min":0.83899999999999997},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.7569999999999999,"5min":2.79,"15min":5.9790000000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.90400000000000003,"15min":0.93700000000000006},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.3580000000000001,"5min":2.3180000000000001,"15min":6.4580000000000002},"last":1.0309999999999999}]},{"osd":4,"last update":"Wed Mar 24 21:59:51 2021","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":1.022,"15min":1.081},"min":{"1min":0.72299999999999998,"5min":0.436,"15min":0.436},"max":{"1min":1.2869999999999999,"5min":2.496,"15min":5.0330000000000004},"last":1.109},{"interface":"front","average":{"1min":1.016,"5min":1.038,"15min":1.0509999999999999},"min":{"1min":0.65800000000000003,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.8460000000000001,"5min":2.8180000000000001,"15min":6.1829999999999998},"last":1.0660000000000001}]},{"osd":5,"last update":"Wed Mar 24 21:58:05 2021","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.1040000000000001,"15min":1.0740000000000001},"min":{"1min":0.64400000000000002,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.603,"5min":5.9390000000000001,"15min":5.9390000000000001},"last":1.1899999999999999},{"interface":"front","average":{"1min":1.103,"5min":1.167,"15min":1.1739999999999999},"min":{"1min":0.75900000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.645,"5min":5.0590000000000002,"15min":5.0590000000000002},"last":1.329}]},{"osd":6,"last update":"Wed Mar 24 21:59:28 2021","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.1140000000000001,"15min":1.1040000000000001},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":4.476,"5min":4.476,"15min":5.9930000000000003},"last":1.3},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.1040000000000001,"15min":1.196},"min":{"1min":0.64500000000000002,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.1479999999999999,"5min":2.21,"15min":4.9960000000000004},"last":1.236}]},{"osd":7,"last update":"Wed Mar 24 22:00:00 2021","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":1.1180000000000001,"15min":1.1739999999999999},"min":{"1min":0.60699999999999998,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.528,"5min":2.8399999999999999,"15min":6.1319999999999997},"last":1.4510000000000001},{"interface":"front","average":{"1min":1.716,"5min":1.2909999999999999,"15min":1.2010000000000001},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":16.007000000000001,"5min":16.007000000000001,"15min":16.007000000000001},"last":1.518}]}]},{"osd":2,"up_from":18,"seq":77309411459,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":240832,"kb_used_data":206712,"kb_used_omap":0,"kb_used_meta":34112,"kb_avail":93504320,"statfs":{"total":95995035648,"available":95748423680,"internally_reserved":0,"allocated":211673088,"data_stored":294987056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":34930688},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 22:00:05 2021","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":0.78000000000000003,"15min":0.68799999999999994},"min":{"1min":0.435,"5min":0.307,"15min":0.307},"max":{"1min":10.054,"5min":10.054,"15min":10.054},"last":0.53900000000000003},{"interface":"front","average":{"1min":1.228,"5min":0.84099999999999997,"15min":0.749},"min":{"1min":0.375,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":12.099,"5min":12.099,"15min":12.099},"last":0.373}]},{"osd":1,"last update":"Wed Mar 24 22:00:05 2021","interfaces":[{"interface":"back","average":{"1min":1.5189999999999999,"5min":0.878,"15min":0.77600000000000002},"min":{"1min":0.309,"5min":0.28199999999999997,"15min":0.28100000000000003},"max":{"1min":17.954999999999998,"5min":17.954999999999998,"15min":17.954999999999998},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.748,"15min":0.75},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.95299999999999996,"5min":1.827,"15min":1.827},"last":0.44700000000000001}]},{"osd":3,"last update":"Wed Mar 24 21:59:27 2021","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.83699999999999997,"15min":0.83799999999999997},"min":{"1min":0.40000000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.472,"5min":1.9610000000000001,"15min":1.9890000000000001},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.89700000000000002,"5min":0.89100000000000001,"15min":0.90000000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.5169999999999999,"5min":1.8859999999999999,"15min":2.0950000000000002},"last":0.48199999999999998}]},{"osd":4,"last update":"Wed Mar 24 21:59:50 2021","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.999,"15min":1.03},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.2829999999999999,"5min":2.5390000000000001,"15min":3.3380000000000001},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.82599999999999996,"5min":1.0149999999999999,"15min":1.04},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.3200000000000001,"5min":2.3119999999999998,"15min":2.3119999999999998},"last":0.66600000000000004}]},{"osd":5,"last update":"Wed Mar 24 21:58:04 2021","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":1.014,"15min":1.0269999999999999},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.163,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":1.0089999999999999},{"interface":"front","average":{"1min":1.048,"5min":1.125,"15min":1.147},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.4390000000000001,"5min":2.5859999999999999,"15min":3.1509999999999998},"last":0.82099999999999995}]},{"osd":6,"last update":"Wed Mar 24 21:59:27 2021","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":1.0760000000000001,"15min":1.1930000000000001},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.425,"5min":2.4529999999999998,"15min":2.4529999999999998},"last":0.98199999999999998},{"interface":"front","average":{"1min":1.323,"5min":1.1459999999999999,"15min":1.1579999999999999},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":6.6520000000000001,"5min":6.6520000000000001,"15min":6.6520000000000001},"last":0.94299999999999995}]},{"osd":7,"last update":"Wed Mar 24 22:00:00 2021","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":1.1890000000000001,"15min":1.212},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":6.7720000000000002,"5min":6.7720000000000002,"15min":6.7720000000000002},"last":0.83499999999999996},{"interface":"front","average":{"1min":2.1190000000000002,"5min":1.385,"15min":1.292},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":24.969999999999999,"5min":24.969999999999999,"15min":24.969999999999999},"last":0.95999999999999996}]}]},{"osd":3,"up_from":26,"seq":111669149822,"num_pgs":109,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":212060,"kb_used_data":177172,"kb_used_omap":0,"kb_used_meta":34880,"kb_avail":93533092,"statfs":{"total":95995035648,"available":95777886208,"internally_reserved":0,"allocated":181424128,"data_stored":241366821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35717120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:59:29 2021","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.008,"15min":0.76000000000000001},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":10.881,"5min":23.846,"15min":23.846},"last":0.85699999999999998},{"interface":"front","average":{"1min":1.036,"5min":0.97799999999999998,"15min":0.81200000000000006},"min":{"1min":0.35899999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":6.8570000000000002,"5min":23.734999999999999,"15min":23.734999999999999},"last":0.82599999999999996}]},{"osd":1,"last update":"Wed Mar 24 21:59:29 2021","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.82999999999999996,"15min":0.77300000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":2.1520000000000001,"5min":14.324,"15min":14.324},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.85399999999999998,"15min":0.79100000000000004},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.28000000000000003},"max":{"1min":2.423,"5min":12.631,"15min":12.631},"last":0.78500000000000003}]},{"osd":2,"last update":"Wed Mar 24 21:59:29 2021","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.89600000000000002,"15min":0.82699999999999996},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":2.613,"5min":12.510999999999999,"15min":12.510999999999999},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.92700000000000005,"15min":0.82499999999999996},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.704,"5min":11.317,"15min":11.317},"last":0.97199999999999998}]},{"osd":4,"last update":"Wed Mar 24 21:59:50 2021","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":1,"15min":0.97699999999999998},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.413,"5min":2.601,"15min":2.601},"last":1.0569999999999999},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.95699999999999996,"15min":1.006},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.1060000000000001,"5min":1.923,"15min":14.291},"last":1.1870000000000001}]},{"osd":5,"last update":"Wed Mar 24 21:58:02 2021","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.1859999999999999,"15min":1.1040000000000001},"min":{"1min":0.73799999999999999,"5min":0.62,"15min":0.54100000000000004},"max":{"1min":1.5840000000000001,"5min":12.723000000000001,"15min":12.723000000000001},"last":1.3029999999999999},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.099,"15min":1.089},"min":{"1min":0.63900000000000001,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.452,"5min":2.375,"15min":2.375},"last":1.212}]},{"osd":6,"last update":"Wed Mar 24 21:59:29 2021","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":1.167,"15min":1.1359999999999999},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.248,"5min":14.255000000000001,"15min":14.255000000000001},"last":1.0089999999999999},{"interface":"front","average":{"1min":0.86199999999999999,"5min":1.1890000000000001,"15min":1.165},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.5620000000000001,"5min":12.666,"15min":12.666},"last":1.0820000000000001}]},{"osd":7,"last update":"Wed Mar 24 21:59:57 2021","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":1.1279999999999999,"15min":1.111},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.4650000000000001,"5min":2.633,"15min":2.633},"last":1.4870000000000001},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.1619999999999999,"15min":1.212},"min":{"1min":0.52000000000000002,"5min":0.49099999999999999,"15min":0.44700000000000001},"max":{"1min":9.609,"5min":9.609,"15min":14.727},"last":1.258}]}]},{"osd":4,"up_from":32,"seq":137438953592,"num_pgs":111,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":200396,"kb_used_data":168004,"kb_used_omap":0,"kb_used_meta":32384,"kb_avail":93544756,"statfs":{"total":95995035648,"available":95789830144,"internally_reserved":0,"allocated":172036096,"data_stored":244977156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":33161216},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 21:59:47 2021","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":1.272,"15min":0.98899999999999999},"min":{"1min":0.58499999999999996,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.601,"5min":41.472000000000001,"15min":41.472000000000001},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.89800000000000002,"5min":1.379,"15min":1.0660000000000001},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":3.5489999999999999,"5min":44.185000000000002,"15min":44.185000000000002},"last":0.58299999999999996}]},{"osd":1,"last update":"Wed Mar 24 21:59:47 2021","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.88200000000000001,"15min":0.90500000000000003},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.38100000000000001},"max":{"1min":1.494,"5min":1.494,"15min":1.494},"last":0.78800000000000003},{"interface":"front","average":{"1min":1.2310000000000001,"5min":0.93799999999999994,"15min":0.89100000000000001},"min":{"1min":0.47499999999999998,"5min":0.46899999999999997,"15min":0.436},"max":{"1min":4.7430000000000003,"5min":4.7430000000000003,"15min":4.7430000000000003},"last":1.113}]},{"osd":2,"last update":"Wed Mar 24 21:59:47 2021","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.90600000000000003,"15min":0.91500000000000004},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":2.1779999999999999,"5min":2.1779999999999999,"15min":4.4379999999999997},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.98499999999999999,"15min":0.94199999999999995},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.569,"5min":2.738,"15min":2.738},"last":0.89800000000000002}]},{"osd":3,"last update":"Wed Mar 24 21:59:47 2021","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.071,"15min":0.98199999999999998},"min":{"1min":0.61299999999999999,"5min":0.45200000000000001,"15min":0.439},"max":{"1min":4.8049999999999997,"5min":4.8049999999999997,"15min":4.8049999999999997},"last":0.98999999999999999},{"interface":"front","average":{"1min":1.123,"5min":1.0409999999999999,"15min":0.98599999999999999},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":3.4239999999999999,"5min":4.1189999999999998,"15min":4.1189999999999998},"last":1.0229999999999999}]},{"osd":5,"last update":"Wed Mar 24 21:58:08 2021","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":0.94099999999999995,"15min":0.95199999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.419,"5min":1.6579999999999999,"15min":1.9199999999999999},"last":0.73399999999999999},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.0760000000000001,"15min":1.03},"min":{"1min":0.56000000000000005,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.3759999999999999,"5min":1.607,"15min":2.0760000000000001},"last":1.133}]},{"osd":6,"last update":"Wed Mar 24 21:59:28 2021","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":1.034,"15min":1.0489999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.3660000000000001,"5min":1.6859999999999999,"15min":1.6859999999999999},"last":1.0720000000000001},{"interface":"front","average":{"1min":0.99099999999999999,"5min":1.0860000000000001,"15min":1.069},"min":{"1min":0.41899999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":3.2509999999999999,"5min":3.2509999999999999,"15min":3.2509999999999999},"last":1.202}]},{"osd":7,"last update":"Wed Mar 24 21:59:52 2021","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.1259999999999999,"15min":1.159},"min":{"1min":0.56200000000000006,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":3.3479999999999999,"5min":3.3479999999999999,"15min":3.3479999999999999},"last":1.163},{"interface":"front","average":{"1min":0.93700000000000006,"5min":1.119,"15min":1.1679999999999999},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":2.0350000000000001,"5min":2.0350000000000001,"15min":2.0350000000000001},"last":1.0960000000000001}]}]},{"osd":5,"up_from":432,"seq":1855425871878,"num_pgs":115,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":196780,"kb_used_data":176292,"kb_used_omap":33,"kb_used_meta":20446,"kb_avail":93548372,"statfs":{"total":95995035648,"available":95793532928,"internally_reserved":0,"allocated":180523008,"data_stored":225654564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33936,"internal_metadata":20937584},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73699999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80500000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":984,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":14620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":5520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":15336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":454656,"data_stored":16468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":516096,"data_stored":18628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":7498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":186327040,"data_stored":240095973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7032832,"data_stored":6977532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":209289216,"data_stored":293672747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":179691520,"data_stored":240946897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":169381888,"data_stored":243674174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":181059584,"data_stored":227620579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":11956224,"data_stored":13756846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-03-24T22:00:29.400 INFO:tasks.thrashosds.thrasher:Setting 6.16 to [7, 1, 4, 5, 3, 6] 2021-03-24T22:00:29.400 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '6.16', '7', '1', '4', '5', '3', '6'] 2021-03-24T22:00:29.401 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd pg-upmap-items 6.16 7 1 4 5 3 6 2021-03-24T22:00:29.409 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:29.412 INFO:tasks.rados.rados.0.smithi125.stdout:1710: finishing write tid 1 to smithi12556470-2 2021-03-24T22:00:29.412 INFO:tasks.rados.rados.0.smithi125.stdout:1710: finishing write tid 2 to smithi12556470-2 2021-03-24T22:00:29.412 INFO:tasks.rados.rados.0.smithi125.stdout:1710: finishing write tid 3 to smithi12556470-2 2021-03-24T22:00:29.413 INFO:tasks.rados.rados.0.smithi125.stdout:1710: finishing write tid 4 to smithi12556470-2 2021-03-24T22:00:29.413 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 2 v 580 (ObjNum 606 snap 184 seq_num 606) dirty exists 2021-03-24T22:00:29.413 INFO:tasks.rados.rados.0.smithi125.stdout:1710: left oid 2 (ObjNum 606 snap 184 seq_num 606) 2021-03-24T22:00:29.451 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:00:29.465 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:29.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:29.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:29.571 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:29.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:29 smithi142 conmon[20125]: cluster 2021-03-24T22:00:28.289238+0000 2021-03-24T22:00:29.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:29 smithi142 conmon[20125]: osd.3 (osd.3) 101 : cluster [DBG] 5.13 scrub ok 2021-03-24T22:00:29.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:29 smithi142 conmon[20125]: audit 2021-03-24T22:00: 2021-03-24T22:00:29.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:29 smithi142 conmon[20125]: 28.860767+0000 mon.a (mon.0) 1374 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "25"}]: dispatch 2021-03-24T22:00:29.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:29 smithi142 conmon[20125]: audit 2021-03 2021-03-24T22:00:29.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:29 smithi142 conmon[20125]: -24T22:00:29.259833+0000 mon.a (mon 2021-03-24T22:00:29.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:29 smithi142 conmon[20125]: .0) 1375 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "25"}]': finished 2021-03-24T22:00:29.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:29 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:00:29.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:29 smithi142 conmon[20125]: 03-24T22:00:29.259910+0000 mon.a ( 2021-03-24T22:00:29.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:29 smithi142 conmon[20125]: mon.0) 1376 : cluster [DBG] osdmap e459: 8 total, 8 up, 5 in 2021-03-24T22:00:29.622 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:29.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:29.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:00:29.728 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:29.778 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:29.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:29.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:00:29.883 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:29.914 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:00:29.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:29.990 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:29.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:00:30.091 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:30.143 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:30.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:30.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:00:30.249 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:30.250 INFO:tasks.rados.rados.0.smithi125.stdout:1705: done (5 left) 2021-03-24T22:00:30.251 INFO:tasks.rados.rados.0.smithi125.stdout:1706: done (4 left) 2021-03-24T22:00:30.251 INFO:tasks.rados.rados.0.smithi125.stdout:1707: done (3 left) 2021-03-24T22:00:30.251 INFO:tasks.rados.rados.0.smithi125.stdout:1710: done (2 left) 2021-03-24T22:00:30.251 INFO:tasks.rados.rados.0.smithi125.stdout:1711: done (1 left) 2021-03-24T22:00:30.252 INFO:tasks.rados.rados.0.smithi125.stdout:1712: done (0 left) 2021-03-24T22:00:30.252 INFO:tasks.rados.rados.0.smithi125.stdout:1713: rollback oid 32 current snap is 185 2021-03-24T22:00:30.252 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 32 to 135 2021-03-24T22:00:30.252 INFO:tasks.rados.rados.0.smithi125.stdout:1714: delete oid 36 current snap is 185 2021-03-24T22:00:30.268 INFO:teuthology.orchestra.run.smithi125.stderr:noscrub is set 2021-03-24T22:00:30.275 INFO:tasks.rados.rados.0.smithi125.stdout:1714: done (1 left) 2021-03-24T22:00:30.275 INFO:tasks.rados.rados.0.smithi125.stdout:1715: read oid 50 snap 177 2021-03-24T22:00:30.275 INFO:tasks.rados.rados.0.smithi125.stdout:1715: expect deleted 2021-03-24T22:00:30.275 INFO:tasks.rados.rados.0.smithi125.stdout:1716: write (excl) oid 45 current snap is 185 2021-03-24T22:00:30.276 INFO:tasks.rados.rados.0.smithi125.stdout:1716: seq_num 608 ranges {618656=543279,1812346=406310} 2021-03-24T22:00:30.282 INFO:tasks.rados.rados.0.smithi125.stdout:1716: writing smithi12556470-45 from 618656 to 1161935 tid 1 2021-03-24T22:00:30.289 INFO:tasks.rados.rados.0.smithi125.stdout:1716: writing smithi12556470-45 from 1812346 to 2218656 tid 2 2021-03-24T22:00:30.290 INFO:tasks.rados.rados.0.smithi125.stdout:1717: delete oid 6 current snap is 185 2021-03-24T22:00:30.290 INFO:tasks.rados.rados.0.smithi125.stdout:1713: finishing rollback tid 0 to smithi12556470-32 2021-03-24T22:00:30.291 INFO:tasks.rados.rados.0.smithi125.stdout:1713: finishing rollback tid 1 to smithi12556470-32 2021-03-24T22:00:30.291 INFO:tasks.rados.rados.0.smithi125.stdout:1713: finishing rollback tid 2 to smithi12556470-32 2021-03-24T22:00:30.292 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 32 v 746 (ObjNum 461 snap 129 seq_num 461) dirty exists 2021-03-24T22:00:30.292 INFO:tasks.rados.rados.0.smithi125.stdout:1716: finishing write tid 1 to smithi12556470-45 2021-03-24T22:00:30.294 INFO:tasks.rados.rados.0.smithi125.stdout:1713: done (3 left) 2021-03-24T22:00:30.295 INFO:tasks.rados.rados.0.smithi125.stdout:1715: done (2 left) 2021-03-24T22:00:30.295 INFO:tasks.rados.rados.0.smithi125.stdout:1717: done (1 left) 2021-03-24T22:00:30.295 INFO:tasks.rados.rados.0.smithi125.stdout:1718: write (excl) oid 34 current snap is 185 2021-03-24T22:00:30.296 INFO:tasks.rados.rados.0.smithi125.stdout:1718: seq_num 610 ranges {652270=410130,1529310=493867} 2021-03-24T22:00:30.302 INFO:tasks.rados.rados.0.smithi125.stdout:1718: writing smithi12556470-34 from 652270 to 1062400 tid 1 2021-03-24T22:00:30.305 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:30.308 INFO:tasks.rados.rados.0.smithi125.stdout:1718: writing smithi12556470-34 from 1529310 to 2023177 tid 2 2021-03-24T22:00:30.308 INFO:tasks.rados.rados.0.smithi125.stdout:1719: copy_from oid 42 from oid 22 current snap is 185 2021-03-24T22:00:30.308 INFO:tasks.rados.rados.0.smithi125.stdout:1716: finishing write tid 2 to smithi12556470-45 2021-03-24T22:00:30.309 INFO:tasks.rados.rados.0.smithi125.stdout:1720: write oid 29 current snap is 185 2021-03-24T22:00:30.309 INFO:tasks.rados.rados.0.smithi125.stdout:1720: seq_num 611 ranges {669077=485379,1587792=681285} 2021-03-24T22:00:30.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:30.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:30.314 INFO:tasks.rados.rados.0.smithi125.stdout:1720: writing smithi12556470-29 from 669077 to 1154456 tid 1 2021-03-24T22:00:30.321 INFO:tasks.rados.rados.0.smithi125.stdout:1720: writing smithi12556470-29 from 1587792 to 2269077 tid 2 2021-03-24T22:00:30.321 INFO:tasks.rados.rados.0.smithi125.stdout:1721: delete oid 49 current snap is 185 2021-03-24T22:00:30.340 INFO:tasks.rados.rados.0.smithi125.stdout:1716: finishing write tid 3 to smithi12556470-45 2021-03-24T22:00:30.340 INFO:tasks.rados.rados.0.smithi125.stdout:1716: finishing write tid 4 to smithi12556470-45 2021-03-24T22:00:30.340 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 45 v 492 (ObjNum 608 snap 185 seq_num 608) dirty exists 2021-03-24T22:00:30.340 INFO:tasks.rados.rados.0.smithi125.stdout:1716: left oid 45 (ObjNum 608 snap 185 seq_num 608) 2021-03-24T22:00:30.346 INFO:tasks.rados.rados.0.smithi125.stdout:1718: finishing write tid 1 to smithi12556470-34 2021-03-24T22:00:30.346 INFO:tasks.rados.rados.0.smithi125.stdout:1720: finishing write tid 1 to smithi12556470-29 2021-03-24T22:00:30.352 INFO:tasks.rados.rados.0.smithi125.stdout:1716: done (4 left) 2021-03-24T22:00:30.353 INFO:tasks.rados.rados.0.smithi125.stdout:1721: done (3 left) 2021-03-24T22:00:30.353 INFO:tasks.rados.rados.0.smithi125.stdout:1722: write oid 37 current snap is 185 2021-03-24T22:00:30.353 INFO:tasks.rados.rados.0.smithi125.stdout:1722: seq_num 613 ranges {702691=635877,2104756=197935} 2021-03-24T22:00:30.364 INFO:tasks.rados.rados.0.smithi125.stdout:1722: writing smithi12556470-37 from 702691 to 1338568 tid 1 2021-03-24T22:00:30.370 INFO:tasks.rados.rados.0.smithi125.stdout:1722: writing smithi12556470-37 from 2104756 to 2302691 tid 2 2021-03-24T22:00:30.371 INFO:tasks.rados.rados.0.smithi125.stdout:1723: read oid 9 snap 183 2021-03-24T22:00:30.371 INFO:tasks.rados.rados.0.smithi125.stdout:1723: expect (ObjNum 520 snap 154 seq_num 520) 2021-03-24T22:00:30.371 INFO:tasks.rados.rados.0.smithi125.stdout:1724: snap_remove snap 176 2021-03-24T22:00:30.412 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:30.458 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:30.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:30.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:00:30.564 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:30.578 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:30 smithi125 conmon[19936]: cluster 2021-03-24T22:00:28.482491+0000 2021-03-24T22:00:30.579 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:30 smithi125 conmon[19936]: osd.0 (osd.0) 100 : cluster [DBG] 3.12 scrub ok 2021-03-24T22:00:30.579 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:30 smithi125 conmon[19936]: cluster 2021-03-24T22:00:28.859765+0000 mgr.y (mgr 2021-03-24T22:00:30.580 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:30 smithi125 conmon[19936]: .14144) 501 : cluster [DBG] pgmap v850: 172 pgs: 1 activating, 1 peering, 170 active+clean; 416 MiB data, 1.0 GiB used, 446 GiB / 447 GiB avail 2021-03-24T22:00:30.580 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:30 smithi125 conmon[19936]: cluster 2021-03-24T22:00:30.580 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:30 smithi125 conmon[19936]: 2021-03-24T22:00:29. 2021-03-24T22:00:30.581 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:30 smithi125 conmon[19936]: 255342+0000 osd.3 (osd.3) 2021-03-24T22:00:30.581 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:30 smithi125 conmon[19936]: 102 : cluster [DBG] 4.10 scrub ok 2021-03-24T22:00:30.582 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:30 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:00:30.582 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:30 smithi125 conmon[19936]: 03-24T22:00:29 2021-03-24T22:00:30.582 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:30 smithi125 conmon[19936]: .462465+0000 osd.5 (osd.5) 7 : cluster 2021-03-24T22:00:30.583 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:30 smithi125 conmon[19936]: [DBG] 4.15 scrub ok 2021-03-24T22:00:30.583 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:30 smithi125 conmon[19936]: audit 2021-03-24 2021-03-24T22:00:30.584 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:30 smithi125 conmon[19936]: T22:00:29.969313+0000 mon.a 2021-03-24T22:00:30.584 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:30 smithi125 conmon[19936]: (mon.0) 1377 : audit [INF] from='client.? v1:172.21.15.125:0/4093820217' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T22:00:30.585 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:30 smithi125 conmon[19936]: 2021-03-24T22:00:30.585 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:30 smithi125 conmon[19936]: cluster 2021-03-24T22:00:30.585 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:30 smithi125 conmon[19936]: 2021-03-24T22:00:30 2021-03-24T22:00:30.586 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:30 smithi125 conmon[19936]: .235280+0000 mon.a (mon.0) 1378 : 2021-03-24T22:00:30.586 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:30 smithi125 conmon[19936]: cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:00:30.587 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:30 smithi125 conmon[19936]: audit 2021-03- 2021-03-24T22:00:30.587 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:30 smithi125 conmon[19936]: 24T22:00:30.246632+0000 2021-03-24T22:00:30.587 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:30 smithi125 conmon[19936]: mon.a (mon.0) 1379 : audit [INF] from='client.? v1:172.21.15.125:0/4093820217' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T22:00:30.588 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:30 smithi125 conmon[19936]: 2021-03-24T22:00:30.588 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:30 smithi125 conmon[19936]: cluster 2021-03-24T22:00:30.589 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:30 smithi125 conmon[19936]: 2021-03-24T22:00:30. 2021-03-24T22:00:30.589 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:30 smithi125 conmon[19936]: 246726+0000 mon.a (mon.0) 1380 : cluster 2021-03-24T22:00:30.589 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:30 smithi125 conmon[19936]: [DBG] osdmap e460: 8 total, 8 up, 5 in 2021-03-24T22:00:30.590 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:30 smithi125 conmon[24927]: cluster 2021-03-24T22:00:28.482491+0000 osd.0 (osd 2021-03-24T22:00:30.590 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:30 smithi125 conmon[24927]: .0) 100 : cluster [DBG] 3.12 scrub ok 2021-03-24T22:00:30.591 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:30 smithi125 conmon[24927]: cluster 2021-03-24T22:00:28.859765+0000 mgr.y (mgr.14144) 501 : cluster [DBG] pgmap v850: 172 pgs: 1 activating, 1 peering, 170 active+clean; 416 MiB data, 1.0 GiB used, 446 GiB / 447 GiB avail 2021-03-24T22:00:30.591 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:30 smithi125 conmon[24927]: cluster 2021-03-24T22:00:29.255342+0000 osd.3 2021-03-24T22:00:30.592 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:30 smithi125 conmon[24927]: (osd.3) 102 : cluster [DBG] 4.10 scrub ok 2021-03-24T22:00:30.592 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:30 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:00:30.592 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:30 smithi125 conmon[24927]: 00:29.462465+0000 osd.5 (osd.5) 7 : cluster [DBG] 2021-03-24T22:00:30.593 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:30 smithi125 conmon[24927]: 4.15 scrub ok 2021-03-24T22:00:30.593 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:30 smithi125 conmon[24927]: audit 2021 2021-03-24T22:00:30.594 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:30 smithi125 conmon[24927]: -03-24T22:00:29.969313+0000 mon.a (mon.0) 1377 : audit 2021-03-24T22:00:30.594 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:30 smithi125 conmon[24927]: [INF] from='client.? v1:172.21.15.125:0/4093820217' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T22:00:30.595 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:30 smithi125 conmon[24927]: cluster 2021-03-24T22:00:30.595 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:30 smithi125 conmon[24927]: 2021-03-24T22:00:30.235280+0000 mon.a 2021-03-24T22:00:30.595 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:30 smithi125 conmon[24927]: (mon.0) 1378 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:00:30.596 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:30 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T22:00:30.596 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:30 smithi125 conmon[24927]: 24T22:00:30.246632+0000 mon.a (mon.0 2021-03-24T22:00:30.597 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:30 smithi125 conmon[24927]: ) 1379 : audit [INF] from='client.? v1:172.21.15.125:0/4093820217' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T22:00:30.597 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:30 smithi125 conmon[24927]: cluster 2021-03-24T22:00: 2021-03-24T22:00:30.597 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:30 smithi125 conmon[24927]: 30.246726+0000 mon.a (mon.0) 1380 : cluster [DBG] osdmap e460: 8 total, 8 up, 5 in 2021-03-24T22:00:30.598 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:30 smithi125 conmon[24927]: 2021-03-24T22:00:30.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:30 smithi142 conmon[20125]: cluster 2021-03-24T22:00:28.482491+0000 2021-03-24T22:00:30.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:30 smithi142 conmon[20125]: osd.0 (osd.0) 100 : cluster [DBG] 3.12 scrub ok 2021-03-24T22:00:30.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:30 smithi142 conmon[20125]: cluster 2021-03-24T22:00:28.859765+0000 2021-03-24T22:00:30.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:30 smithi142 conmon[20125]: mgr.y (mgr.14144) 501 : cluster [DBG] pgmap v850: 172 pgs: 1 activating, 1 peering, 170 active+clean; 416 MiB data, 1.0 GiB used, 446 GiB / 447 GiB avail 2021-03-24T22:00:30.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:30 smithi142 conmon[20125]: cluster 2021-03-24T22:00:29.255342 2021-03-24T22:00:30.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:30 smithi142 conmon[20125]: +0000 osd.3 (osd.3) 102 : cluster 2021-03-24T22:00:30.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:30 smithi142 conmon[20125]: [DBG] 4.10 scrub ok 2021-03-24T22:00:30.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:30 smithi142 conmon[20125]: cluster 2021-03-24T22: 2021-03-24T22:00:30.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:30 smithi142 conmon[20125]: 00:29.462465+0000 osd.5 (osd.5) 7 : 2021-03-24T22:00:30.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:30 smithi142 conmon[20125]: cluster [DBG] 4.15 scrub ok 2021-03-24T22:00:30.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:30 smithi142 conmon[20125]: audit 2021-03-24T22:00:30.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:30 smithi142 conmon[20125]: 2021-03-24T22:00:29.969313+0000 mon.a (mon.0) 2021-03-24T22:00:30.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:30 smithi142 conmon[20125]: 1377 : audit [INF] from='client.? v1:172.21.15.125:0/4093820217' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T22:00:30.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:30 smithi142 conmon[20125]: cluster 2021-03-24T22:00:30.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:30 smithi142 conmon[20125]: 2021-03-24T22:00:30.235280+0000 mon.a ( 2021-03-24T22:00:30.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:30 smithi142 conmon[20125]: mon.0) 1378 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:00:30.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:30 smithi142 conmon[20125]: audit 2021 2021-03-24T22:00:30.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:30 smithi142 conmon[20125]: -03-24T22:00:30 2021-03-24T22:00:30.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:30 smithi142 conmon[20125]: .246632+0000 mon.a (mon.0) 1379 : audit 2021-03-24T22:00:30.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:30 smithi142 conmon[20125]: [INF] from='client.? v1:172.21.15.125:0/4093820217' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T22:00:30.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:30 smithi142 conmon[20125]: cluster 2021-03-24T22 2021-03-24T22:00:30.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:30 smithi142 conmon[20125]: :00:30.246726+0000 mon.a (mon. 2021-03-24T22:00:30.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:30 smithi142 conmon[20125]: 0) 1380 : cluster [DBG] osdmap e460: 8 total, 8 up, 5 in 2021-03-24T22:00:30.625 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:30.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:30.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:30.720 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2021-03-24T22:00:30.720 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T22:00:30.729 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:30.774 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:30.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:30.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:00:30.880 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:30.935 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:30.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:30.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:00:31.042 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:31.098 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:31.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:31.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:31.205 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:31.259 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:31.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:31.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:00:31.272 INFO:tasks.rados.rados.0.smithi125.stdout:1724: done (5 left) 2021-03-24T22:00:31.272 INFO:tasks.rados.rados.0.smithi125.stdout:1725: snap_create 2021-03-24T22:00:31.272 INFO:tasks.rados.rados.0.smithi125.stdout:1720: finishing write tid 2 to smithi12556470-29 2021-03-24T22:00:31.272 INFO:tasks.rados.rados.0.smithi125.stdout:1720: finishing write tid 3 to smithi12556470-29 2021-03-24T22:00:31.273 INFO:tasks.rados.rados.0.smithi125.stdout:1720: finishing write tid 4 to smithi12556470-29 2021-03-24T22:00:31.273 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 29 v 246 (ObjNum 611 snap 185 seq_num 611) dirty exists 2021-03-24T22:00:31.273 INFO:tasks.rados.rados.0.smithi125.stdout:1720: left oid 29 (ObjNum 611 snap 185 seq_num 611) 2021-03-24T22:00:31.273 INFO:tasks.rados.rados.0.smithi125.stdout:1718: finishing write tid 2 to smithi12556470-34 2021-03-24T22:00:31.274 INFO:tasks.rados.rados.0.smithi125.stdout:1718: finishing write tid 3 to smithi12556470-34 2021-03-24T22:00:31.274 INFO:tasks.rados.rados.0.smithi125.stdout:1718: finishing write tid 4 to smithi12556470-34 2021-03-24T22:00:31.274 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 34 v 550 (ObjNum 610 snap 185 seq_num 610) dirty exists 2021-03-24T22:00:31.274 INFO:tasks.rados.rados.0.smithi125.stdout:1718: left oid 34 (ObjNum 610 snap 185 seq_num 610) 2021-03-24T22:00:31.274 INFO:tasks.rados.rados.0.smithi125.stdout:1719: finishing copy_from to smithi12556470-42 2021-03-24T22:00:31.275 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 42 v 488 (ObjNum 556 snap 172 seq_num 556) dirty exists 2021-03-24T22:00:31.275 INFO:tasks.rados.rados.0.smithi125.stdout:1719: finishing copy_from racing read to smithi12556470-42 2021-03-24T22:00:31.275 INFO:tasks.rados.rados.0.smithi125.stdout:1722: finishing write tid 1 to smithi12556470-37 2021-03-24T22:00:31.275 INFO:tasks.rados.rados.0.smithi125.stdout:1723: expect (ObjNum 520 snap 154 seq_num 520) 2021-03-24T22:00:31.365 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:31.368 INFO:tasks.rados.rados.0.smithi125.stdout:1722: finishing write tid 2 to smithi12556470-37 2021-03-24T22:00:31.368 INFO:tasks.rados.rados.0.smithi125.stdout:1722: finishing write tid 3 to smithi12556470-37 2021-03-24T22:00:31.368 INFO:tasks.rados.rados.0.smithi125.stdout:1722: finishing write tid 4 to smithi12556470-37 2021-03-24T22:00:31.369 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 37 v 482 (ObjNum 613 snap 185 seq_num 613) dirty exists 2021-03-24T22:00:31.369 INFO:tasks.rados.rados.0.smithi125.stdout:1722: left oid 37 (ObjNum 613 snap 185 seq_num 613) 2021-03-24T22:00:31.408 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:31.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:31.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:00:31.513 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:31.526 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:31 smithi125 conmon[19936]: cluster 2021-03-24T22 2021-03-24T22:00:31.527 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:31 smithi125 conmon[19936]: :00:29.855888+0000 osd.4 (osd.4) 108 : cluster [DBG] 3.1c scrub ok 2021-03-24T22:00:31.527 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:31 smithi125 conmon[19936]: cluster 2021-03-24T22:00:31.527 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:31 smithi125 conmon[19936]: 2021-03-24T22:00: 2021-03-24T22:00:31.528 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:31 smithi125 conmon[19936]: 31.268086+0000 mon.a (mon.0) 2021-03-24T22:00:31.528 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:31 smithi125 conmon[19936]: 1381 : cluster [DBG] osdmap e461: 8 total, 8 up, 5 in 2021-03-24T22:00:31.528 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:31 smithi125 conmon[24927]: cluster 2021-03-24T22:00:29.855888+0000 2021-03-24T22:00:31.528 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:31 smithi125 conmon[24927]: osd.4 (osd.4) 108 : cluster [DBG] 3.1c scrub ok 2021-03-24T22:00:31.529 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:31 smithi125 conmon[24927]: cluster 2021-03-24T22:00:31.268086+0000 2021-03-24T22:00:31.529 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:31 smithi125 conmon[24927]: mon.a (mon.0) 1381 : cluster [DBG] osdmap e461: 8 total, 8 up, 5 in 2021-03-24T22:00:31.565 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:31.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:31.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:00:31.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:31 smithi142 conmon[20125]: cluster 2021-03-24T22:00:29.855888+0000 osd.4 (osd.4) 108 : cluster [DBG] 2021-03-24T22:00:31.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:31 smithi142 conmon[20125]: 3.1c scrub ok 2021-03-24T22:00:31.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:31 smithi142 conmon[20125]: cluster 2021-03-24T22:00:31.268086+0000 mon.a (mon.0) 1381 : cluster [DBG] osdmap e461: 8 total, 8 up, 5 in 2021-03-24T22:00:31.670 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:31.721 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:31.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:31.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:00:31.741 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.5 dump_historic_ops 2021-03-24T22:00:31.829 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:31.884 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:31.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:31.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:31.992 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:32.049 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T22:00:32.055 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:32.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:32.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:00:32.161 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:32.218 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:32.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:32.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:00:32.274 INFO:tasks.rados.rados.0.smithi125.stdout:1718: done (5 left) 2021-03-24T22:00:32.275 INFO:tasks.rados.rados.0.smithi125.stdout:1719: done (4 left) 2021-03-24T22:00:32.275 INFO:tasks.rados.rados.0.smithi125.stdout:1720: done (3 left) 2021-03-24T22:00:32.276 INFO:tasks.rados.rados.0.smithi125.stdout:1722: done (2 left) 2021-03-24T22:00:32.276 INFO:tasks.rados.rados.0.smithi125.stdout:1723: done (1 left) 2021-03-24T22:00:32.277 INFO:tasks.rados.rados.0.smithi125.stdout:1725: done (0 left) 2021-03-24T22:00:32.277 INFO:tasks.rados.rados.0.smithi125.stdout:1726: snap_remove snap 151 2021-03-24T22:00:32.324 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:32.390 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:32.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:32.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:00:32.496 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:32.553 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:32.557 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:32 smithi142 conmon[20125]: cluster 2021-03-24T22:00:30.860485+0000 mgr.y (mgr.14144) 502 : cluster [DBG] 2021-03-24T22:00:32.557 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:32 smithi142 conmon[20125]: pgmap v853: 172 pgs: 1 activating, 171 active+clean; 411 MiB data, 1.0 GiB used, 446 GiB / 447 GiB avail 2021-03-24T22:00:32.558 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:32 smithi142 conmon[20125]: cluster 2021-03-24T22:00:32.271190+0000 mon.a (mon.0) 1382 : cluster [DBG] osdmap e462: 8 total, 8 up, 5 in 2021-03-24T22:00:32.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:32.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:00:32.621 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:32 smithi125 conmon[24927]: cluster 2021-03-24T22:00:30.860485+0000 mgr.y (mgr.14144 2021-03-24T22:00:32.622 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:32 smithi125 conmon[24927]: ) 502 : cluster [DBG] pgmap v853: 172 pgs: 1 activating, 171 active+clean; 411 MiB data, 1.0 GiB used, 446 GiB / 447 GiB avail 2021-03-24T22:00:32.622 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:32 smithi125 conmon[24927]: cluster 2021-03-24T22:00:32.271190+0000 mon.a (mon.0) 1382 : cluster [DBG] osdmap e462: 8 total, 8 up, 5 in 2021-03-24T22:00:32.622 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:32 smithi125 conmon[19936]: cluster 2021-03-24T22:00:30.860485 2021-03-24T22:00:32.623 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:32 smithi125 conmon[19936]: +0000 mgr.y (mgr.14144) 502 : cluster [DBG] pgmap v853: 172 pgs: 1 activating, 171 active+clean; 411 MiB data, 1.0 GiB used, 446 GiB / 447 GiB avail 2021-03-24T22:00:32.623 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:32 smithi125 conmon[19936]: cluster 2021-03-24T22:00:32.271190+0000 mon.a (mon.0) 2021-03-24T22:00:32.623 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:32 smithi125 conmon[19936]: 1382 : cluster [DBG] osdmap e462: 8 total, 8 up, 5 in 2021-03-24T22:00:32.661 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:32.718 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:32.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:32.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:00:32.825 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:32.881 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:32.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:32.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:00:32.988 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:33.049 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:33.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:33.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:00:33.154 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:33.215 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:33.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:33.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:00:33.279 INFO:tasks.rados.rados.0.smithi125.stdout:1726: done (0 left) 2021-03-24T22:00:33.280 INFO:tasks.rados.rados.0.smithi125.stdout:1727: read oid 1 snap -1 2021-03-24T22:00:33.280 INFO:tasks.rados.rados.0.smithi125.stdout:1727: expect (ObjNum 595 snap 179 seq_num 595) 2021-03-24T22:00:33.280 INFO:tasks.rados.rados.0.smithi125.stdout:1728: rollback oid 14 current snap is 186 2021-03-24T22:00:33.281 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 14 to 179 2021-03-24T22:00:33.281 INFO:tasks.rados.rados.0.smithi125.stdout:1729: rollback oid 40 current snap is 186 2021-03-24T22:00:33.281 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 40 to 177 2021-03-24T22:00:33.282 INFO:tasks.rados.rados.0.smithi125.stdout:1730: read oid 23 snap -1 2021-03-24T22:00:33.282 INFO:tasks.rados.rados.0.smithi125.stdout:1730: expect (ObjNum 427 snap 113 seq_num 427) 2021-03-24T22:00:33.282 INFO:tasks.rados.rados.0.smithi125.stdout:1731: rollback oid 46 current snap is 186 2021-03-24T22:00:33.282 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 46 to 185 2021-03-24T22:00:33.283 INFO:tasks.rados.rados.0.smithi125.stdout:1732: copy_from oid 45 from oid 34 current snap is 186 2021-03-24T22:00:33.283 INFO:tasks.rados.rados.0.smithi125.stdout:1733: write (excl) oid 43 current snap is 186 2021-03-24T22:00:33.283 INFO:tasks.rados.rados.0.smithi125.stdout:1733: seq_num 614 ranges {719498=711126,2046885=272613} 2021-03-24T22:00:33.288 INFO:tasks.rados.rados.0.smithi125.stdout:1733: writing smithi12556470-43 from 719498 to 1430624 tid 1 2021-03-24T22:00:33.295 INFO:tasks.rados.rados.0.smithi125.stdout:1733: writing smithi12556470-43 from 2046885 to 2319498 tid 2 2021-03-24T22:00:33.295 INFO:tasks.rados.rados.0.smithi125.stdout:1734: read oid 24 snap 167 2021-03-24T22:00:33.296 INFO:tasks.rados.rados.0.smithi125.stdout:1734: expect (ObjNum 515 snap 151 seq_num 515) 2021-03-24T22:00:33.296 INFO:tasks.rados.rados.0.smithi125.stdout:1735: snap_remove snap 175 2021-03-24T22:00:33.320 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:33.363 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:33.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:33.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:00:33.468 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:33.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:33 smithi125 conmon[19936]: cluster 2021-03-24T22:00:32.546990 2021-03-24T22:00:33.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:33 smithi125 conmon[19936]: +0000 osd.1 (osd.1) 43 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:00:33.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:33 smithi125 conmon[19936]: cluster 2021-03-24T22:00:32.547156+0000 2021-03-24T22:00:33.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:33 smithi125 conmon[19936]: osd.1 (osd.1) 44 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:00:33.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:33 smithi125 conmon[19936]: audit 2021-03-24T22:00:33.487 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:33 smithi125 conmon[19936]: 2021-03-24T22:00:32 2021-03-24T22:00:33.488 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:33 smithi125 conmon[19936]: .863196+0000 mon.a (mon.0) 1383 2021-03-24T22:00:33.488 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:33 smithi125 conmon[19936]: : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "24"}]: dispatch 2021-03-24T22:00:33.488 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:33 smithi125 conmon[19936]: audit 2021-03-24T22:00:33.489 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:33 smithi125 conmon[19936]: 2021-03- 2021-03-24T22:00:33.489 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:33 smithi125 conmon[19936]: 24T22:00:33.274133 2021-03-24T22:00:33.489 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:33 smithi125 conmon[19936]: +0000 mon.a (mon.0) 1384 : 2021-03-24T22:00:33.489 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:33 smithi125 conmon[19936]: audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "24"}]': finished 2021-03-24T22:00:33.489 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:33 smithi125 conmon[19936]: cluster 2021-03-24T22:00:33.490 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:33 smithi125 conmon[19936]: 2021-03-24T 2021-03-24T22:00:33.490 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:33 smithi125 conmon[19936]: 22:00:33.274297+0000 mon.a ( 2021-03-24T22:00:33.490 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:33 smithi125 conmon[19936]: mon.0) 1385 : cluster [DBG] 2021-03-24T22:00:33.490 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:33 smithi125 conmon[19936]: osdmap e463: 8 total, 8 up, 5 in 2021-03-24T22:00:33.491 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:33 smithi125 conmon[24927]: cluster 2021-03-24T22:00:32.546990+0000 osd.1 (osd.1 2021-03-24T22:00:33.491 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:33 smithi125 conmon[24927]: ) 43 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:00:33.491 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:33 smithi125 conmon[24927]: cluster 2021-03-24T22:00:32.547156+0000 osd.1 (osd.1) 44 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:00:33.491 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:33 smithi125 conmon[24927]: audit 2021-03-24T22:00:32.863196+0000 mon.a (mon.0) 1383 : audit 2021-03-24T22:00:33.492 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:33 smithi125 conmon[24927]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "24"}]: dispatch 2021-03-24T22:00:33.492 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:33 smithi125 conmon[24927]: audit 2021- 2021-03-24T22:00:33.492 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:33 smithi125 conmon[24927]: 03-24T22:00:33.274133+0000 mon.a (mon.0) 1384 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "24"}]': finished 2021-03-24T22:00:33.492 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:33 smithi125 conmon[24927]: cluster 2021-03-24T22:00:33. 2021-03-24T22:00:33.492 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:33 smithi125 conmon[24927]: 274297+0000 mon.a (mon.0) 1385 : cluster [DBG] osdmap e463: 8 total, 8 up, 5 in 2021-03-24T22:00:33.520 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:33.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:33.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:00:33.626 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:33.676 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:33.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:33.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:00:33.686 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:33 smithi142 conmon[20125]: cluster 2021-03-24T22:00:32.546990+0000 osd.1 (osd.1) 43 : cluster 2021-03-24T22:00:33.686 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:33 smithi142 conmon[20125]: [DBG] purged_snaps scrub starts 2021-03-24T22:00:33.686 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:33 smithi142 conmon[20125]: cluster 2021-03-24T22:00:32.547156+0000 osd.1 (osd.1) 44 : 2021-03-24T22:00:33.687 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:33 smithi142 conmon[20125]: cluster [DBG] purged_snaps scrub ok 2021-03-24T22:00:33.687 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:33 smithi142 conmon[20125]: audit 2021 2021-03-24T22:00:33.687 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:33 smithi142 conmon[20125]: -03-24T22:00:32.863196+0000 mon.a (mon 2021-03-24T22:00:33.687 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:33 smithi142 conmon[20125]: .0) 1383 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "24"}]: dispatch 2021-03-24T22:00:33.687 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:33 smithi142 conmon[20125]: audit 2021-03- 2021-03-24T22:00:33.688 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:33 smithi142 conmon[20125]: 24T22:00:33.274133+0000 mon.a 2021-03-24T22:00:33.688 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:33 smithi142 conmon[20125]: (mon.0) 1384 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "24"}]': finished 2021-03-24T22:00:33.688 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:33 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:00:33.688 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:33 smithi142 conmon[20125]: 03-24T22:00:33.274297+0000 2021-03-24T22:00:33.689 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:33 smithi142 conmon[20125]: mon.a (mon.0) 1385 : cluster [DBG] 2021-03-24T22:00:33.689 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:33 smithi142 conmon[20125]: osdmap e463: 8 total, 8 up, 5 in 2021-03-24T22:00:33.718 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T22:00:33.745 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T22:00:33.766 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T22:00:33.781 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:33.790 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T22:00:33.819 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T22:00:33.839 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:33.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:33.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:00:33.848 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T22:00:33.870 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T22:00:33.894 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T22:00:33.897 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-24T22:00:33.897 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-24T22:00:33.898 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-24T22:00:33.898 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-24T22:00:33.898 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-24T22:00:33.898 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-24T22:00:33.898 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-24T22:00:33.899 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-24T22:00:33.944 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:33.999 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2021-03-24T22:00:33.999 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2021-03-24T22:00:34.000 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2021-03-24T22:00:34.000 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2021-03-24T22:00:34.000 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.000 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(7.14 433/432 [PushOp(7:2db4b4db:::smithi12556470-25:fd, version: 362'286, data_included: [2337754~226140,3243966~453640], data_size: 679780, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:2db4b4db:::smithi12556470-25:fd@362'286, size: 3697606, copy_subset: [497606~645222,1934400~629494,3243966~453640], clone_subset: {7:2db4b4db:::smithi12556470-25:f8=[0~497606,1142828~791572,2563894~680072]}, snapset: 104=[]:{f3=[f3,f2,f1,ef,ed,ea],f8=[f8,f5,f4],fd=[fd,fc,fb,fa],104=[104,103,100]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3697606, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2337754, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-03-24T22:00:34.001 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:00:03.658712+0000", 2021-03-24T22:00:34.001 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 30.333665633999999, 2021-03-24T22:00:34.001 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0054351690000000001, 2021-03-24T22:00:34.001 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:34.002 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:00:34.002 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:34.002 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.003 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:34.003 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.658712+0000", 2021-03-24T22:00:34.003 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:34.003 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.004 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.004 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:34.004 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.658712+0000", 2021-03-24T22:00:34.005 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999971 2021-03-24T22:00:34.005 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.005 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.005 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:34.005 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.658709+0000", 2021-03-24T22:00:34.006 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00066146900000000003 2021-03-24T22:00:34.006 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.006 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.006 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:34.007 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.659371+0000", 2021-03-24T22:00:34.007 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8121999999999999e-05 2021-03-24T22:00:34.007 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.007 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.008 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:34.008 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.659409+0000", 2021-03-24T22:00:34.008 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3133999999999999e-05 2021-03-24T22:00:34.008 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.009 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.009 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:34.009 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.659432+0000", 2021-03-24T22:00:34.010 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00023118800000000001 2021-03-24T22:00:34.010 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.010 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.010 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:34.011 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.659663+0000", 2021-03-24T22:00:34.011 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2948000000000001e-05 2021-03-24T22:00:34.011 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.011 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.012 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:00:34.012 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.659686+0000", 2021-03-24T22:00:34.012 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0044609560000000003 2021-03-24T22:00:34.013 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.013 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.013 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:34.013 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.664147+0000", 2021-03-24T22:00:34.013 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0054351690000000001 2021-03-24T22:00:34.014 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:34.014 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:34.014 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:34.015 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.015 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.015 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(7.4 433/432 [PushOp(7:260d8f69:::smithi12556470-28:113, version: 382'214, data_included: [2421213~353381,3432022~349619], data_size: 703000, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:260d8f69:::smithi12556470-28:113@382'214, size: 3781641, copy_subset: [581641~621467,1994104~780490,3126168~655473], clone_subset: {7:260d8f69:::smithi12556470-28:fa=[0~581641,1203108~790996,2774594~351574]}, snapset: 13d=[]:{d9=[d6],ed=[ed,eb,ea,e8,e7,df],fa=[fa,f8,f5,f3,f2,f1,ef],113=[113,110,10f,10e,10d,10c,109,106,105,104,103,100],123=[123,121,120,11e,11b,11a,119],13d=[13d,13b,139,137,136,133,132,12d,12b,129,128,127,126]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3781641, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2421213, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-03-24T22:00:34.016 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:00:03.854649+0000", 2021-03-24T22:00:34.016 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 30.137728759000002, 2021-03-24T22:00:34.016 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0054639409999999999, 2021-03-24T22:00:34.016 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:34.016 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:00:34.017 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:34.017 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.017 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:34.017 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.854649+0000", 2021-03-24T22:00:34.018 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:34.018 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.018 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.018 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:34.019 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.854649+0000", 2021-03-24T22:00:34.019 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2021-03-24T22:00:34.019 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.020 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.020 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:34.020 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.854647+0000", 2021-03-24T22:00:34.021 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0012032430000000001 2021-03-24T22:00:34.021 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.021 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.021 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:34.022 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.855851+0000", 2021-03-24T22:00:34.022 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0680999999999999e-05 2021-03-24T22:00:34.022 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.022 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.023 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:34.023 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.855891+0000", 2021-03-24T22:00:34.023 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3592e-05 2021-03-24T22:00:34.023 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.024 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.024 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:34.024 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.855915+0000", 2021-03-24T22:00:34.024 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000159591 2021-03-24T22:00:34.025 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.025 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.025 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:34.025 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.856074+0000", 2021-03-24T22:00:34.026 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4749000000000001e-05 2021-03-24T22:00:34.026 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.026 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.026 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:00:34.027 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.856089+0000", 2021-03-24T22:00:34.027 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0040237600000000004 2021-03-24T22:00:34.027 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.027 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.028 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:34.028 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.860113+0000", 2021-03-24T22:00:34.028 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0054639409999999999 2021-03-24T22:00:34.028 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:34.028 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:34.029 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:34.029 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.029 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.029 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(7.6 433/432 [PushOp(7:65c04537:::smithi12556470-26:106, version: 371'176, data_included: [2338506~335411,3095506~265960], data_size: 601371, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:65c04537:::smithi12556470-26:106@371'176, size: 3361466, copy_subset: [561466~791183,2081113~592804,3095506~265960], clone_subset: {7:65c04537:::smithi12556470-26:e8=[0~561466,1352649~728464,2673917~421589]}, snapset: 113=[]:{e8=[e8,e7,df,d6],106=[106,105,104,103,100,fd,fb,fa,f8,f1],10e=[10e,10d,10c,109],113=[113,110,10f]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3361466, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2338506, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-03-24T22:00:34.030 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:00:03.898657+0000", 2021-03-24T22:00:34.030 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 30.093720873999999, 2021-03-24T22:00:34.030 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0051833060000000004, 2021-03-24T22:00:34.030 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:34.030 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:00:34.031 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:34.031 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.031 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:34.031 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.898657+0000", 2021-03-24T22:00:34.031 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:34.032 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.032 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.032 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:34.032 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.898657+0000", 2021-03-24T22:00:34.033 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2021-03-24T22:00:34.033 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.033 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.033 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:34.033 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.898656+0000", 2021-03-24T22:00:34.034 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00079342499999999995 2021-03-24T22:00:34.034 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.034 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.034 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:34.035 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.899449+0000", 2021-03-24T22:00:34.035 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4879999999999999e-05 2021-03-24T22:00:34.035 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.035 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.035 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:34.036 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.899474+0000", 2021-03-24T22:00:34.036 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2201999999999999e-05 2021-03-24T22:00:34.036 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.036 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.036 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:34.037 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.899486+0000", 2021-03-24T22:00:34.037 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000179443 2021-03-24T22:00:34.038 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.038 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.038 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:34.039 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.899665+0000", 2021-03-24T22:00:34.039 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8017e-05 2021-03-24T22:00:34.039 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.040 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.040 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:00:34.040 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.899684+0000", 2021-03-24T22:00:34.040 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0041566299999999997 2021-03-24T22:00:34.040 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.041 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.041 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:34.041 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.903840+0000", 2021-03-24T22:00:34.041 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0051833060000000004 2021-03-24T22:00:34.042 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:34.042 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:34.042 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:34.042 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.043 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.043 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(osd.0.0:1423 7.6 e433/432 7:65c04537:::smithi12556470-26:e8 v 433'185, mlcod=0'0)", 2021-03-24T22:00:34.043 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:00:04.245653+0000", 2021-03-24T22:00:34.043 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 29.746724917000002, 2021-03-24T22:00:34.043 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.010907610999999999, 2021-03-24T22:00:34.044 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:34.044 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:00:34.044 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:34.044 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.045 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:34.045 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.245653+0000", 2021-03-24T22:00:34.045 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:34.045 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.045 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.046 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:34.046 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.245653+0000", 2021-03-24T22:00:34.046 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2021-03-24T22:00:34.046 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.047 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.047 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:34.047 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.245651+0000", 2021-03-24T22:00:34.047 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0915e-05 2021-03-24T22:00:34.047 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.048 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.048 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:34.048 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.245662+0000", 2021-03-24T22:00:34.048 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2146e-05 2021-03-24T22:00:34.049 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.049 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.049 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:34.050 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.245674+0000", 2021-03-24T22:00:34.050 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.0189999999999995e-06 2021-03-24T22:00:34.050 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.050 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.050 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:34.051 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.245683+0000", 2021-03-24T22:00:34.051 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.5519999999999996e-05 2021-03-24T22:00:34.051 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.051 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.052 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:34.052 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.245748+0000", 2021-03-24T22:00:34.052 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3873999999999999e-05 2021-03-24T22:00:34.052 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.052 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.053 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:00:34.053 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.245772+0000", 2021-03-24T22:00:34.053 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.010744946 2021-03-24T22:00:34.053 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.054 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.054 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:00:34.054 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.256517+0000", 2021-03-24T22:00:34.054 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.3090999999999998e-05 2021-03-24T22:00:34.054 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.055 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.055 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:34.055 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.256560+0000", 2021-03-24T22:00:34.055 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.010907610999999999 2021-03-24T22:00:34.056 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:34.056 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:34.056 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:34.056 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.056 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.057 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(7.14 434/432 [PushOp(7:2fcfb5bf:::smithi12556470-50:11e, version: 394'309, data_included: [470728~404794,1526454~481175], data_size: 885969, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:2fcfb5bf:::smithi12556470-50:11e@394'309, size: 2470728, copy_subset: [470728~404794,1526454~481175], clone_subset: {7:2fcfb5bf:::smithi12556470-50:113=[0~470728,875522~650932,2007629~463099]}, snapset: 13b=[]:{e0=[df,d6],113=[113,110,10f,10e,10d,10c,109,106,105,104,103,100,fa,f8,f1,e8,e7],11e=[11e,11b,11a,119],13b=[13b,13a,139,137,136,133,132,12d,12b,129,128,127,126,120]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2007629, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2021-03-24T22:00:34.057 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:00:04.270577+0000", 2021-03-24T22:00:34.057 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 29.721800988999998, 2021-03-24T22:00:34.057 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.020657128, 2021-03-24T22:00:34.058 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:34.058 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:00:34.058 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:34.058 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.058 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:34.059 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.270577+0000", 2021-03-24T22:00:34.059 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:34.059 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.059 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.060 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:34.060 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.270577+0000", 2021-03-24T22:00:34.060 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2021-03-24T22:00:34.061 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.061 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.061 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:34.061 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.270576+0000", 2021-03-24T22:00:34.062 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00050608000000000001 2021-03-24T22:00:34.062 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.062 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.062 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:34.063 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.271082+0000", 2021-03-24T22:00:34.063 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2589999999999999e-05 2021-03-24T22:00:34.063 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.063 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.063 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:34.064 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.271104+0000", 2021-03-24T22:00:34.064 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6656000000000001e-05 2021-03-24T22:00:34.064 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.064 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.065 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:34.065 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.271121+0000", 2021-03-24T22:00:34.065 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.016885359999999999 2021-03-24T22:00:34.065 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.066 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.066 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:34.066 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.288006+0000", 2021-03-24T22:00:34.066 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.1189999999999997e-06 2021-03-24T22:00:34.067 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.067 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.067 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:00:34.067 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.288014+0000", 2021-03-24T22:00:34.067 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0032194290000000002 2021-03-24T22:00:34.068 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.068 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.068 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:34.068 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.291234+0000", 2021-03-24T22:00:34.069 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.020657128 2021-03-24T22:00:34.069 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:34.069 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:34.069 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:34.070 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.070 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.070 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(5.d 434/433 [PushOp(5:b4812045:::notify.4:head, version: 343'5, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:b4812045:::notify.4:head@343'5, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-03-24T22:00:34.070 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:00:04.271509+0000", 2021-03-24T22:00:34.071 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 29.720868353, 2021-03-24T22:00:34.071 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0076492579999999999, 2021-03-24T22:00:34.072 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:34.072 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:00:34.072 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:34.073 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.073 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:34.073 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.271509+0000", 2021-03-24T22:00:34.073 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:34.073 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.074 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.074 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:34.074 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.271509+0000", 2021-03-24T22:00:34.074 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2021-03-24T22:00:34.075 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.075 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.075 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:34.075 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.271509+0000", 2021-03-24T22:00:34.075 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1328000000000001e-05 2021-03-24T22:00:34.076 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.076 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.076 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:34.076 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.271520+0000", 2021-03-24T22:00:34.077 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2507000000000001e-05 2021-03-24T22:00:34.077 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.077 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.077 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:34.077 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.271533+0000", 2021-03-24T22:00:34.078 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9239999999999996e-06 2021-03-24T22:00:34.078 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.078 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.078 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:34.078 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.271542+0000", 2021-03-24T22:00:34.079 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0075032079999999999 2021-03-24T22:00:34.079 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.079 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.079 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:34.080 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.279045+0000", 2021-03-24T22:00:34.080 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.5210000000000008e-06 2021-03-24T22:00:34.080 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.080 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.080 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:00:34.081 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.279054+0000", 2021-03-24T22:00:34.081 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000104326 2021-03-24T22:00:34.081 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.081 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.081 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:34.082 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.279159+0000", 2021-03-24T22:00:34.082 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0076492579999999999 2021-03-24T22:00:34.082 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:34.082 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:34.083 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:34.083 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.083 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.083 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(osd.0.0:1424 7.6 e434/432 7:65c04537:::smithi12556470-26:106 v 434'187, mlcod=382'183)", 2021-03-24T22:00:34.083 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:00:04.292178+0000", 2021-03-24T22:00:34.084 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 29.700200064000001, 2021-03-24T22:00:34.084 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0099396859999999997, 2021-03-24T22:00:34.084 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:34.084 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:00:34.084 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:34.085 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.085 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:34.085 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.292178+0000", 2021-03-24T22:00:34.085 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:34.085 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.086 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.086 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:34.086 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.292178+0000", 2021-03-24T22:00:34.086 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2021-03-24T22:00:34.086 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.087 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.087 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:34.087 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.292177+0000", 2021-03-24T22:00:34.087 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1260000000000001e-05 2021-03-24T22:00:34.088 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.088 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.088 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:34.088 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.292188+0000", 2021-03-24T22:00:34.088 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8340000000000004e-06 2021-03-24T22:00:34.089 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.089 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.089 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:34.089 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.292194+0000", 2021-03-24T22:00:34.089 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8969999999999996e-06 2021-03-24T22:00:34.090 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.090 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.090 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:34.090 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.292200+0000", 2021-03-24T22:00:34.091 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0023655519999999999 2021-03-24T22:00:34.091 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.091 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.091 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:34.091 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.294566+0000", 2021-03-24T22:00:34.092 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9349999999999999e-05 2021-03-24T22:00:34.092 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.092 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.092 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:00:34.092 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.294585+0000", 2021-03-24T22:00:34.093 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0075000520000000001 2021-03-24T22:00:34.093 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.093 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.093 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:00:34.094 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.302085+0000", 2021-03-24T22:00:34.094 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2225999999999999e-05 2021-03-24T22:00:34.094 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.094 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.094 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:34.095 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.302117+0000", 2021-03-24T22:00:34.095 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0099396859999999997 2021-03-24T22:00:34.095 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:34.095 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:34.095 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:34.096 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.096 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.096 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(7.1d 434/433 [PushOp(7:ba8515f1:::smithi12556470-45:head, version: 425'318, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7:ba8515f1:::smithi12556470-45:head@425'318, size: 2705993, copy_subset: [0~2705993], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2021-03-24T22:00:34.096 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:00:04.313732+0000", 2021-03-24T22:00:34.096 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 29.678645637999999, 2021-03-24T22:00:34.097 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0077728839999999999, 2021-03-24T22:00:34.097 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:34.097 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:00:34.097 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:34.098 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.098 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:34.098 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.313732+0000", 2021-03-24T22:00:34.098 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:34.098 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.099 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.099 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:34.099 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.313732+0000", 2021-03-24T22:00:34.099 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2021-03-24T22:00:34.100 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.100 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.100 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:34.100 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.313731+0000", 2021-03-24T22:00:34.100 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0071907359999999997 2021-03-24T22:00:34.101 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.101 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.101 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:34.101 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.320922+0000", 2021-03-24T22:00:34.101 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6305999999999999e-05 2021-03-24T22:00:34.102 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.102 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.102 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:34.102 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.320938+0000", 2021-03-24T22:00:34.102 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0054999999999999e-05 2021-03-24T22:00:34.103 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.103 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.103 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:34.103 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.320949+0000", 2021-03-24T22:00:34.103 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2479999999999997e-05 2021-03-24T22:00:34.104 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.104 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.104 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:34.104 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.321021+0000", 2021-03-24T22:00:34.104 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.7170000000000008e-06 2021-03-24T22:00:34.105 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.105 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.105 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:00:34.105 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.321029+0000", 2021-03-24T22:00:34.105 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00047625199999999998 2021-03-24T22:00:34.106 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.106 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.106 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:34.106 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.321505+0000", 2021-03-24T22:00:34.107 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0077728839999999999 2021-03-24T22:00:34.107 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:34.107 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:34.107 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:34.107 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.108 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.108 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(7.1 434/433 [PushOp(7:83e5d499:::smithi12556470-11:head, version: 428'313, data_included: [2270198~49267], data_size: 49267, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:83e5d499:::smithi12556470-11:head@428'313, size: 2319465, copy_subset: [719465~411200,1632822~686643], clone_subset: {7:83e5d499:::smithi12556470-11:10e=[0~719465,1130665~502157]}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2319465, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2270198, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-03-24T22:00:34.108 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:00:04.923806+0000", 2021-03-24T22:00:34.108 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 29.068571376000001, 2021-03-24T22:00:34.108 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0088824760000000003, 2021-03-24T22:00:34.109 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:34.109 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:00:34.109 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:34.109 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.110 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:34.110 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.923806+0000", 2021-03-24T22:00:34.110 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:34.110 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.110 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.111 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:34.111 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.923806+0000", 2021-03-24T22:00:34.111 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2021-03-24T22:00:34.111 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.111 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.112 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:34.112 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.923805+0000", 2021-03-24T22:00:34.112 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8309000000000002e-05 2021-03-24T22:00:34.112 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.113 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.113 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:34.113 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.923844+0000", 2021-03-24T22:00:34.113 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2789000000000002e-05 2021-03-24T22:00:34.113 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.114 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.114 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:34.114 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.923866+0000", 2021-03-24T22:00:34.114 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2225999999999999e-05 2021-03-24T22:00:34.114 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.115 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.115 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:34.115 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.923879+0000", 2021-03-24T22:00:34.115 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9759999999999994e-05 2021-03-24T22:00:34.116 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.116 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.116 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:34.116 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.923968+0000", 2021-03-24T22:00:34.116 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6799999999999993e-06 2021-03-24T22:00:34.117 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.117 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.117 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:00:34.117 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.923976+0000", 2021-03-24T22:00:34.118 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0087127160000000006 2021-03-24T22:00:34.118 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.118 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.118 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:34.118 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.932689+0000", 2021-03-24T22:00:34.119 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0088824760000000003 2021-03-24T22:00:34.119 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:34.119 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:34.119 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:34.119 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.120 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.120 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.0 434/433 [PushOp(2:353ec8f9:::gateway.conf:head, version: 66'5, data_included: [0~389], data_size: 389, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(2:353ec8f9:::gateway.conf:head@66'5, size: 389, copy_subset: [0~389], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:389, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2021-03-24T22:00:34.120 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:00:05.116382+0000", 2021-03-24T22:00:34.120 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 28.875995770999999, 2021-03-24T22:00:34.121 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.055895578000000001, 2021-03-24T22:00:34.121 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:34.121 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:00:34.121 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:34.121 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.122 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:34.122 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:05.116382+0000", 2021-03-24T22:00:34.122 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:34.122 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.123 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.123 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:34.123 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:05.116382+0000", 2021-03-24T22:00:34.123 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2021-03-24T22:00:34.123 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.124 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.124 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:34.124 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:05.116380+0000", 2021-03-24T22:00:34.124 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3156999999999999e-05 2021-03-24T22:00:34.124 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.125 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.125 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:34.125 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:05.116403+0000", 2021-03-24T22:00:34.125 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5398999999999999e-05 2021-03-24T22:00:34.126 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.126 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.126 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:34.126 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:05.116429+0000", 2021-03-24T22:00:34.126 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1512999999999999e-05 2021-03-24T22:00:34.127 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.127 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.127 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:34.127 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:05.116450+0000", 2021-03-24T22:00:34.127 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.055696180999999997 2021-03-24T22:00:34.128 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.128 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.128 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:34.128 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:05.172147+0000", 2021-03-24T22:00:34.129 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4039999999999997e-06 2021-03-24T22:00:34.129 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.129 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.129 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:00:34.130 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:05.172151+0000", 2021-03-24T22:00:34.130 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012657099999999999 2021-03-24T22:00:34.130 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.130 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.130 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:34.131 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:05.172278+0000", 2021-03-24T22:00:34.131 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.055895578000000001 2021-03-24T22:00:34.131 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:34.131 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:34.132 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:34.132 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.132 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.132 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(7.18 434/433 [PushOp(7:19ff009c:::smithi12556470-42:head, version: 342'94, data_included: [662308~559030,1632005~489546], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7:19ff009c:::smithi12556470-42:head@342'94, size: 3462308, copy_subset: [0~3462308], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2121551, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2021-03-24T22:00:34.132 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:00:05.174576+0000", 2021-03-24T22:00:34.133 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 28.817801960000001, 2021-03-24T22:00:34.133 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.088754672000000007, 2021-03-24T22:00:34.133 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:34.133 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:00:34.134 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:34.134 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.134 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:34.134 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:05.174576+0000", 2021-03-24T22:00:34.134 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:34.135 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.135 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.135 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:34.135 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:05.174576+0000", 2021-03-24T22:00:34.135 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2021-03-24T22:00:34.136 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.136 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.136 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:34.136 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:05.174575+0000", 2021-03-24T22:00:34.137 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001295725 2021-03-24T22:00:34.137 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.137 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.137 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:34.137 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:05.175870+0000", 2021-03-24T22:00:34.138 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0645999999999999e-05 2021-03-24T22:00:34.138 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.138 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.138 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:34.138 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:05.175891+0000", 2021-03-24T22:00:34.139 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1664999999999999e-05 2021-03-24T22:00:34.139 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.139 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.139 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:34.140 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:05.175903+0000", 2021-03-24T22:00:34.140 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.086726023999999999 2021-03-24T22:00:34.140 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.140 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.140 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:34.141 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:05.262629+0000", 2021-03-24T22:00:34.141 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2415000000000001e-05 2021-03-24T22:00:34.141 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.141 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.142 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:00:34.142 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:05.262641+0000", 2021-03-24T22:00:34.142 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00068924500000000001 2021-03-24T22:00:34.142 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.142 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.143 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:34.143 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:05.263330+0000", 2021-03-24T22:00:34.143 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.088754672000000007 2021-03-24T22:00:34.143 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:34.144 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:34.144 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:34.144 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.144 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.144 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(7.18 435/433 [PushOp(7:19ff009c:::smithi12556470-42:head, version: 342'94, data_included: [2121551~114265,2860044~602264], data_size: 716529, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:19ff009c:::smithi12556470-42:head@342'94, size: 3462308, copy_subset: [0~3462308], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3462308, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2121551, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-03-24T22:00:34.145 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:00:05.289062+0000", 2021-03-24T22:00:34.145 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 28.703315554, 2021-03-24T22:00:34.145 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.063256141000000002, 2021-03-24T22:00:34.145 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:34.145 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:00:34.146 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:34.146 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.146 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:34.146 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:05.289062+0000", 2021-03-24T22:00:34.147 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:34.147 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.147 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.147 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:34.147 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:05.289062+0000", 2021-03-24T22:00:34.148 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999933 2021-03-24T22:00:34.148 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.148 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.148 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:34.148 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:05.289056+0000", 2021-03-24T22:00:34.149 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001029657 2021-03-24T22:00:34.149 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.149 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.149 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:34.149 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:05.290085+0000", 2021-03-24T22:00:34.150 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9239000000000001e-05 2021-03-24T22:00:34.150 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.150 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.150 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:34.151 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:05.290135+0000", 2021-03-24T22:00:34.151 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2237000000000003e-05 2021-03-24T22:00:34.151 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.151 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.151 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:34.152 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:05.290167+0000", 2021-03-24T22:00:34.152 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.061331175000000002 2021-03-24T22:00:34.152 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.152 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.152 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:34.153 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:05.351498+0000", 2021-03-24T22:00:34.153 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8980000000000002e-06 2021-03-24T22:00:34.153 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.153 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.153 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:00:34.154 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:05.351507+0000", 2021-03-24T22:00:34.154 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081146899999999999 2021-03-24T22:00:34.154 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.154 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.155 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:34.155 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:05.352318+0000", 2021-03-24T22:00:34.155 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.063256141000000002 2021-03-24T22:00:34.155 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:34.155 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:34.156 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:34.156 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.156 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.156 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(7.13 437/433 [PushOp(7:cb55abc9:::smithi12556470-6:head, version: 431'342, data_included: [2202154~179428,2949285~480099], data_size: 659527, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:cb55abc9:::smithi12556470-6:head@431'342, size: 3495955, copy_subset: [695955~725807,1879385~502197,2949285~546670], clone_subset: {7:cb55abc9:::smithi12556470-6:140=[0~695955,1421762~457623,2381582~567703]}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3495955, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2202154, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-03-24T22:00:34.157 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:00:08.083310+0000", 2021-03-24T22:00:34.157 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 25.909067713999999, 2021-03-24T22:00:34.157 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0055004549999999996, 2021-03-24T22:00:34.157 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:34.157 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:00:34.158 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:34.158 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.158 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:34.158 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:08.083310+0000", 2021-03-24T22:00:34.158 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:34.159 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.159 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.159 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:34.159 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:08.083310+0000", 2021-03-24T22:00:34.160 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999971 2021-03-24T22:00:34.160 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.160 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.160 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:34.160 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:08.083307+0000", 2021-03-24T22:00:34.161 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00058589600000000003 2021-03-24T22:00:34.161 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.161 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.161 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:34.162 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:08.083893+0000", 2021-03-24T22:00:34.162 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2396e-05 2021-03-24T22:00:34.162 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.162 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.162 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:34.163 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:08.083925+0000", 2021-03-24T22:00:34.163 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4949999999999999e-05 2021-03-24T22:00:34.163 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.163 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.163 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:34.164 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:08.083940+0000", 2021-03-24T22:00:34.164 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00022642300000000001 2021-03-24T22:00:34.164 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.164 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.165 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:34.165 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:08.084167+0000", 2021-03-24T22:00:34.165 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9479999999999999e-05 2021-03-24T22:00:34.165 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.165 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.166 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:00:34.166 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:08.084196+0000", 2021-03-24T22:00:34.166 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0046141330000000003 2021-03-24T22:00:34.166 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.167 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.167 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:34.167 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:08.088810+0000", 2021-03-24T22:00:34.167 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0055004549999999996 2021-03-24T22:00:34.167 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:34.168 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:34.168 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:34.168 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.168 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.168 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(osd.0.0:1448 7.13 e437/433 7:cb55abc9:::smithi12556470-6:13d v 437'350, mlcod=437'344)", 2021-03-24T22:00:34.169 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:00:08.168588+0000", 2021-03-24T22:00:34.169 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 25.823789531999999, 2021-03-24T22:00:34.169 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0054671390000000002, 2021-03-24T22:00:34.169 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:34.169 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:00:34.170 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:34.170 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.170 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:34.170 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:08.168588+0000", 2021-03-24T22:00:34.170 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:34.171 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.171 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.171 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:34.171 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:08.168588+0000", 2021-03-24T22:00:34.172 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2021-03-24T22:00:34.172 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.172 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.172 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:34.172 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:08.168586+0000", 2021-03-24T22:00:34.173 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5866999999999999e-05 2021-03-24T22:00:34.173 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.173 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.173 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:34.173 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:08.168602+0000", 2021-03-24T22:00:34.174 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1931999999999999e-05 2021-03-24T22:00:34.174 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.174 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.174 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:34.175 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:08.168624+0000", 2021-03-24T22:00:34.175 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3983e-05 2021-03-24T22:00:34.175 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.175 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.175 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:34.176 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:08.168638+0000", 2021-03-24T22:00:34.176 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00047166200000000003 2021-03-24T22:00:34.176 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.176 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.176 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:34.177 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:08.169109+0000", 2021-03-24T22:00:34.177 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5259999999999998e-05 2021-03-24T22:00:34.177 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.177 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.177 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:00:34.178 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:08.169144+0000", 2021-03-24T22:00:34.178 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0048426720000000001 2021-03-24T22:00:34.178 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.178 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.178 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:00:34.179 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:08.173987+0000", 2021-03-24T22:00:34.179 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.8176000000000006e-05 2021-03-24T22:00:34.179 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.179 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.179 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:34.180 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:08.174055+0000", 2021-03-24T22:00:34.180 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0054671390000000002 2021-03-24T22:00:34.180 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:34.180 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:34.181 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:34.181 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.181 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.181 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.14760.0:4336 7.23 e446/432 7:c49de0b4:::smithi12556470-13:head v 446'298, mlcod=443'295)", 2021-03-24T22:00:34.181 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:00:16.265873+0000", 2021-03-24T22:00:34.182 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 17.726504740999999, 2021-03-24T22:00:34.182 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0066939679999999998, 2021-03-24T22:00:34.182 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:34.182 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:00:34.182 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:34.183 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.183 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:34.183 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:16.265873+0000", 2021-03-24T22:00:34.183 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:34.184 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.184 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.184 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:34.184 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:16.265873+0000", 2021-03-24T22:00:34.184 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2021-03-24T22:00:34.185 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.185 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.185 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:34.185 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:16.265872+0000", 2021-03-24T22:00:34.185 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001395152 2021-03-24T22:00:34.186 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.186 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.186 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:34.186 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:16.267267+0000", 2021-03-24T22:00:34.186 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3644e-05 2021-03-24T22:00:34.187 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.187 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.187 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:34.187 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:16.267281+0000", 2021-03-24T22:00:34.187 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.807e-06 2021-03-24T22:00:34.188 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.188 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.188 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:34.188 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:16.267288+0000", 2021-03-24T22:00:34.189 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7630000000000002e-05 2021-03-24T22:00:34.189 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.189 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.189 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:34.189 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:16.267346+0000", 2021-03-24T22:00:34.190 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4502e-05 2021-03-24T22:00:34.190 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.190 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.190 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:00:34.190 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:16.267371+0000", 2021-03-24T22:00:34.191 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0051498719999999998 2021-03-24T22:00:34.191 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.191 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.191 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:00:34.192 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:16.272520+0000", 2021-03-24T22:00:34.192 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.6551000000000002e-05 2021-03-24T22:00:34.192 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.192 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.192 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:34.193 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:16.272567+0000", 2021-03-24T22:00:34.193 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0066939679999999998 2021-03-24T22:00:34.193 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:34.193 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:34.193 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:34.194 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.194 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.194 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(osd.4.0:508 7.1a e450/433 7:5f18d753:::smithi12556470-18:12d v 450'213, mlcod=446'209)", 2021-03-24T22:00:34.194 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:00:20.247594+0000", 2021-03-24T22:00:34.195 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 13.744783311999999, 2021-03-24T22:00:34.195 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0061614549999999997, 2021-03-24T22:00:34.195 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:34.195 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:00:34.195 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:34.196 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.196 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:34.196 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:20.247594+0000", 2021-03-24T22:00:34.196 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:34.196 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.197 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.197 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:34.197 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:20.247594+0000", 2021-03-24T22:00:34.197 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2021-03-24T22:00:34.197 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.198 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.198 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:34.198 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:20.247594+0000", 2021-03-24T22:00:34.198 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3387000000000001e-05 2021-03-24T22:00:34.199 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.199 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.199 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:34.199 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:20.247607+0000", 2021-03-24T22:00:34.199 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9279999999999999e-06 2021-03-24T22:00:34.200 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.200 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.200 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:34.200 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:20.247616+0000", 2021-03-24T22:00:34.201 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2939999999999994e-06 2021-03-24T22:00:34.201 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.201 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.201 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:34.201 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:20.247626+0000", 2021-03-24T22:00:34.202 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.9957999999999994e-05 2021-03-24T22:00:34.202 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.202 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.202 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:34.202 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:20.247726+0000", 2021-03-24T22:00:34.203 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.7541999999999998e-05 2021-03-24T22:00:34.203 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.203 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.203 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:00:34.204 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:20.247793+0000", 2021-03-24T22:00:34.204 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.005910317 2021-03-24T22:00:34.204 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.204 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.204 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:00:34.205 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:20.253704+0000", 2021-03-24T22:00:34.205 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.2345000000000001e-05 2021-03-24T22:00:34.205 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.205 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.206 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:34.206 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:20.253756+0000", 2021-03-24T22:00:34.206 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0061614549999999997 2021-03-24T22:00:34.206 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:34.206 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:34.207 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:34.207 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.207 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.207 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(osd.2.0:333 7.19 e450/433 7:9c4f4ea5:::smithi12556470-41:149 v 450'464, mlcod=447'460)", 2021-03-24T22:00:34.207 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:00:20.250619+0000", 2021-03-24T22:00:34.208 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 13.741758554, 2021-03-24T22:00:34.208 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0068222969999999997, 2021-03-24T22:00:34.208 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:34.208 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:00:34.209 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:34.209 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.209 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:34.209 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:20.250619+0000", 2021-03-24T22:00:34.209 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:34.210 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.210 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.210 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:34.210 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:20.250619+0000", 2021-03-24T22:00:34.210 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2021-03-24T22:00:34.211 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.211 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.211 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:34.211 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:20.250619+0000", 2021-03-24T22:00:34.211 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1849e-05 2021-03-24T22:00:34.212 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.212 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.212 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:34.212 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:20.250631+0000", 2021-03-24T22:00:34.212 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3739e-05 2021-03-24T22:00:34.213 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.213 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.213 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:34.213 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:20.250644+0000", 2021-03-24T22:00:34.213 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6260000000000003e-06 2021-03-24T22:00:34.214 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.214 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.214 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:34.214 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:20.250652+0000", 2021-03-24T22:00:34.214 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0017848580000000001 2021-03-24T22:00:34.215 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.215 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.215 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:34.215 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:20.252437+0000", 2021-03-24T22:00:34.215 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8498999999999997e-05 2021-03-24T22:00:34.216 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.216 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.216 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:00:34.216 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:20.252495+0000", 2021-03-24T22:00:34.217 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0049071469999999997 2021-03-24T22:00:34.217 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.217 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.217 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:00:34.217 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:20.257402+0000", 2021-03-24T22:00:34.218 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.9004000000000001e-05 2021-03-24T22:00:34.218 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.218 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.218 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:34.218 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:20.257441+0000", 2021-03-24T22:00:34.219 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0068222969999999997 2021-03-24T22:00:34.219 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:34.219 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:34.219 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:34.219 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.220 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.220 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(osd.2.0:334 7.15 e450/433 7:ad88ec1b:::smithi12556470-47:137 v 450'142, mlcod=446'138)", 2021-03-24T22:00:34.220 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:00:20.250659+0000", 2021-03-24T22:00:34.220 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 13.741718839000001, 2021-03-24T22:00:34.220 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0064537580000000004, 2021-03-24T22:00:34.221 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:34.221 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:00:34.221 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:34.221 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.221 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:34.222 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:20.250659+0000", 2021-03-24T22:00:34.222 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:34.222 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.222 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.223 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:34.223 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:20.250659+0000", 2021-03-24T22:00:34.223 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2021-03-24T22:00:34.223 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.223 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.224 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:34.224 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:20.250658+0000", 2021-03-24T22:00:34.224 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0229999999999998e-06 2021-03-24T22:00:34.224 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.224 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.225 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:34.225 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:20.250664+0000", 2021-03-24T22:00:34.225 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.2630000000000003e-06 2021-03-24T22:00:34.225 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.226 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.226 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:34.226 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:20.250670+0000", 2021-03-24T22:00:34.226 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.4980000000000003e-06 2021-03-24T22:00:34.226 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.227 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.227 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:34.227 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:20.250676+0000", 2021-03-24T22:00:34.227 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000795219 2021-03-24T22:00:34.227 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.228 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.228 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:34.228 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:20.251471+0000", 2021-03-24T22:00:34.228 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7734999999999999e-05 2021-03-24T22:00:34.229 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.229 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.229 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:00:34.229 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:20.251489+0000", 2021-03-24T22:00:34.229 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0055935949999999998 2021-03-24T22:00:34.230 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.230 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.230 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:00:34.230 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:20.257083+0000", 2021-03-24T22:00:34.230 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0066e-05 2021-03-24T22:00:34.231 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.231 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.231 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:34.231 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:20.257113+0000", 2021-03-24T22:00:34.231 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0064537580000000004 2021-03-24T22:00:34.232 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:34.232 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:34.232 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:34.232 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.232 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.233 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(osd.2.0:350 7.e e452/432 7:777a79d9:::smithi12556470-38:152 v 452'70, mlcod=450'66)", 2021-03-24T22:00:34.233 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:00:22.289949+0000", 2021-03-24T22:00:34.233 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 11.702428845, 2021-03-24T22:00:34.233 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0050953149999999996, 2021-03-24T22:00:34.233 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:34.234 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:00:34.234 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:34.234 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.234 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:34.235 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:22.289949+0000", 2021-03-24T22:00:34.235 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:34.235 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.235 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.235 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:34.236 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:22.289949+0000", 2021-03-24T22:00:34.236 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2021-03-24T22:00:34.236 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.236 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.236 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:34.237 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:22.289948+0000", 2021-03-24T22:00:34.237 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.5580000000000003e-06 2021-03-24T22:00:34.237 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.237 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.238 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:34.238 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:22.289956+0000", 2021-03-24T22:00:34.238 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.167e-05 2021-03-24T22:00:34.238 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.238 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.239 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:34.239 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:22.289968+0000", 2021-03-24T22:00:34.239 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0879999999999995e-06 2021-03-24T22:00:34.239 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.239 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.240 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:34.240 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:22.289976+0000", 2021-03-24T22:00:34.240 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.1390999999999994e-05 2021-03-24T22:00:34.240 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.240 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.241 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:34.241 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:22.290057+0000", 2021-03-24T22:00:34.241 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2622e-05 2021-03-24T22:00:34.241 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.241 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.242 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:00:34.242 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:22.290090+0000", 2021-03-24T22:00:34.242 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00490874 2021-03-24T22:00:34.242 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.243 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.243 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:00:34.243 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:22.294998+0000", 2021-03-24T22:00:34.243 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.5821000000000001e-05 2021-03-24T22:00:34.243 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.244 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.244 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:34.244 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:22.295044+0000", 2021-03-24T22:00:34.244 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0050953149999999996 2021-03-24T22:00:34.244 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:34.245 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:34.245 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:34.245 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.245 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.245 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(osd.2.0:351 7.17 e452/433 7:e8068f91:::smithi12556470-10:145 v 452'565, mlcod=451'561)", 2021-03-24T22:00:34.246 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:00:22.290009+0000", 2021-03-24T22:00:34.246 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 11.702368593999999, 2021-03-24T22:00:34.246 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0050370969999999999, 2021-03-24T22:00:34.246 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:00:34.246 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:00:34.247 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:00:34.247 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.247 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:00:34.247 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:22.290009+0000", 2021-03-24T22:00:34.247 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:00:34.248 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.248 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.248 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:00:34.248 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:22.290009+0000", 2021-03-24T22:00:34.249 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2021-03-24T22:00:34.249 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.249 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.249 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:00:34.249 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:22.290008+0000", 2021-03-24T22:00:34.250 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.1720000000000007e-06 2021-03-24T22:00:34.250 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.250 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.250 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:00:34.250 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:22.290016+0000", 2021-03-24T22:00:34.251 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.0329999999999997e-06 2021-03-24T22:00:34.251 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.251 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.251 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:00:34.251 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:22.290025+0000", 2021-03-24T22:00:34.252 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6800000000000004e-06 2021-03-24T22:00:34.252 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.252 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.252 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:00:34.252 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:22.290032+0000", 2021-03-24T22:00:34.253 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4851000000000001e-05 2021-03-24T22:00:34.253 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.253 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.253 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:00:34.254 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:22.290087+0000", 2021-03-24T22:00:34.254 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7399e-05 2021-03-24T22:00:34.254 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.254 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.254 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:00:34.255 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:22.290114+0000", 2021-03-24T22:00:34.255 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.004876923 2021-03-24T22:00:34.255 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.255 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.255 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:00:34.256 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:22.294991+0000", 2021-03-24T22:00:34.256 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5223000000000003e-05 2021-03-24T22:00:34.256 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:00:34.256 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:00:34.257 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:00:34.257 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:22.295046+0000", 2021-03-24T22:00:34.257 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0050370969999999999 2021-03-24T22:00:34.257 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:34.257 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:34.258 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:34.258 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:00:34.258 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:00:34.258 INFO:teuthology.orchestra.run.smithi142.stdout:} 2021-03-24T22:00:34.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:34.260 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:34.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:00:34.285 INFO:tasks.rados.rados.0.smithi125.stdout:1735: done (8 left) 2021-03-24T22:00:34.286 INFO:tasks.rados.rados.0.smithi125.stdout:1736: snap_remove snap 170 2021-03-24T22:00:34.294 INFO:teuthology.orchestra.run.smithi125.stderr:set 6.16 pg_upmap_items mapping to [7->1,4->5,3->6] 2021-03-24T22:00:34.361 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:34.387 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-24T22:00:34.417 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:34.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:34.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:00:34.523 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:34.640 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:34 smithi142 conmon[20125]: cluster 2021-03-24T22:00 2021-03-24T22:00:34.640 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:34 smithi142 conmon[20125]: :32.861282+0000 mgr.y (mgr.14144) 503 : cluster [DBG] pgmap v856: 171 pgs: 171 active+clean; 410 MiB data, 1.0 GiB used, 446 GiB / 447 GiB avail; 7.5 MiB/s rd, 1.4 MiB/s wr, 4 op/s; 144 B/s, 0 objects/s recovering 2021-03-24T22:00:34.641 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:34 smithi142 conmon[20125]: audit 2021-03-24T22: 2021-03-24T22:00:34.641 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:34 smithi142 conmon[20125]: 00:33.692457+0000 mon.c (mon.2) 72 2021-03-24T22:00:34.641 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:34 smithi142 conmon[20125]: : audit [DBG] from='client.? v1:172.21.15.125:0/92274696' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:00:34.641 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:34 smithi142 conmon[20125]: audit 2021-03-24T22:00:34.642 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:34 smithi142 conmon[20125]: 2021-03-24T22: 2021-03-24T22:00:34.642 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:34 smithi142 conmon[20125]: 00:34.223984+0000 mon.a (mon.0) 2021-03-24T22:00:34.642 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:34 smithi142 conmon[20125]: 1386 : audit [INF] from='client.25279 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.16", "id": [7, 1, 4, 5, 3, 6]}]: dispatch 2021-03-24T22:00:34.642 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:34 smithi142 conmon[20125]: audit 2021- 2021-03-24T22:00:34.643 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:34 smithi142 conmon[20125]: 03-24T22:00:34.225247+0000 mon.b 2021-03-24T22:00:34.643 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:34 smithi142 conmon[20125]: (mon.1) 38 : audit [INF] from='client.? v1:172.21.15.125:0/1702076749' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.16", "id": [7, 1, 4, 5, 3, 6]}]: dispatch 2021-03-24T22:00:34.643 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:34 smithi142 conmon[20125]: audit 2021 2021-03-24T22:00:34.644 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:34 smithi142 conmon[20125]: -03-24T22:00:34.280633+0000 mon.a 2021-03-24T22:00:34.644 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:34 smithi142 conmon[20125]: (mon.0) 1387 : audit [INF] from='client.25279 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.16", "id": [7, 1, 4, 5, 3, 6]}]': finished 2021-03-24T22:00:34.644 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:34 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:00:34.645 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:34 smithi142 conmon[20125]: -03-24T22:00:34.280846+0000 mon.a (mon. 2021-03-24T22:00:34.645 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:34 smithi142 conmon[20125]: 0) 1388 : cluster [DBG] osdmap e464: 8 total, 8 up, 5 in 2021-03-24T22:00:34.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:34 smithi125 conmon[24927]: cluster 2021-03-24T22:00:34.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:34 smithi125 conmon[24927]: 2021-03 2021-03-24T22:00:34.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:34 smithi125 conmon[24927]: -24T22:00:32.861282+0000 mgr.y (mgr.14144) 503 : cluster [DBG] pgmap v856: 171 pgs: 171 active+clean; 410 MiB data, 1.0 GiB used, 446 GiB / 447 GiB avail; 7.5 MiB/s rd, 1.4 MiB/s wr, 4 op/s; 144 B/s, 0 objects/s recovering 2021-03-24T22:00:34.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:34 smithi125 conmon[24927]: 2021-03-24T22:00:34.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:34 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T22:00:34.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:34 smithi125 conmon[24927]: 24T22:00:33.692457+0000 mon.c (mon 2021-03-24T22:00:34.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:34 smithi125 conmon[24927]: .2) 72 : audit [DBG] from='client.? v1:172.21.15.125:0/92274696' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:00:34.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:34 smithi125 conmon[24927]: audit 2021 2021-03-24T22:00:34.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:34 smithi125 conmon[24927]: -03-24T22:00:34.223984+0000 mon.a ( 2021-03-24T22:00:34.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:34 smithi125 conmon[24927]: mon.0) 1386 : audit [INF] from='client.25279 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.16", "id": [7, 1, 4, 5, 3, 6]}]: dispatch 2021-03-24T22:00:34.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:34 smithi125 conmon[24927]: audit 2021 2021-03-24T22:00:34.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:34 smithi125 conmon[24927]: -03-24T22:00:34.225247+0000 2021-03-24T22:00:34.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:34 smithi125 conmon[24927]: mon.b (mon.1) 38 : audit [INF] from='client.? v1:172.21.15.125:0/1702076749' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.16", "id": [7, 1, 4, 5, 3, 6]}]: dispatch 2021-03-24T22:00:34.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:34 smithi125 conmon[24927]: 2021-03-24T22:00:34.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:34 smithi125 conmon[24927]: audit 2021-03-24T22:00:34 2021-03-24T22:00:34.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:34 smithi125 conmon[24927]: .280633+0000 mon.a (mon.0) 1387 : audit 2021-03-24T22:00:34.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:34 smithi125 conmon[24927]: [INF] from='client.25279 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.16", "id": [7, 1, 4, 5, 3, 6]}]': finished 2021-03-24T22:00:34.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:34 smithi125 conmon[24927]: cluster 2021-03-24T22 2021-03-24T22:00:34.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:34 smithi125 conmon[24927]: :00:34.280846+0000 mon.a (mon.0) 2021-03-24T22:00:34.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:34 smithi125 conmon[24927]: 1388 : cluster [DBG] osdmap e464: 8 total, 8 up, 5 in 2021-03-24T22:00:34.670 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:34 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:00:34.671 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:34 smithi125 conmon[19936]: 00:32.861282+0000 mgr.y (mgr.14144) 503 : cluster [DBG] pgmap v856: 171 pgs: 171 active+clean; 410 MiB data, 1.0 GiB used, 446 GiB / 447 GiB avail; 7.5 MiB/s rd, 1.4 MiB/s wr, 4 op/s; 144 B/s, 0 objects/s recovering 2021-03-24T22:00:34.671 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:34 smithi125 conmon[19936]: audit 2021 2021-03-24T22:00:34.672 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:34 smithi125 conmon[19936]: -03-24T22:00:33. 2021-03-24T22:00:34.672 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:34 smithi125 conmon[19936]: 692457+0000 mon.c (mon.2) 72 2021-03-24T22:00:34.672 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:34 smithi125 conmon[19936]: : audit [DBG] from='client.? v1:172.21.15.125:0/92274696' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:00:34.673 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:34 smithi125 conmon[19936]: audit 2021-03 2021-03-24T22:00:34.673 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:34 smithi125 conmon[19936]: -24T22:00:34.223984+0000 mon.a 2021-03-24T22:00:34.674 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:34 smithi125 conmon[19936]: (mon.0) 1386 : audit [INF] 2021-03-24T22:00:34.674 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:34 smithi125 conmon[19936]: from='client.25279 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.16", "id": [7, 1, 4, 5, 3, 6]}]: dispatch 2021-03-24T22:00:34.675 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:34 smithi125 conmon[19936]: audit 2021-03-24T22:00:34 2021-03-24T22:00:34.675 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:34 smithi125 conmon[19936]: .225247+0000 mon.b (mon.1) 38 : audit 2021-03-24T22:00:34.675 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:34 smithi125 conmon[19936]: [INF] from='client.? v1:172.21.15.125:0/1702076749' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.16", "id": [7, 1, 4, 5, 3, 6]}]: dispatch 2021-03-24T22:00:34.676 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:34 smithi125 conmon[19936]: audit 2021-03-24T22:00:34.676 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:34 smithi125 conmon[19936]: 2021-03-24T22:00:34.280633+0000 mon.a 2021-03-24T22:00:34.677 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:34 smithi125 conmon[19936]: (mon.0) 1387 : audit [INF] from='client.25279 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.16", "id": [7, 1, 4, 5, 3, 6]}]': finished 2021-03-24T22:00:34.677 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:34 smithi125 conmon[19936]: 2021-03-24T22:00:34.677 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:34 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:00:34.678 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:34 smithi125 conmon[19936]: 24T22:00:34.280846+0000 mon.a (mon.0 2021-03-24T22:00:34.678 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:34 smithi125 conmon[19936]: ) 1388 : cluster [DBG] osdmap e464: 8 total, 8 up, 5 in 2021-03-24T22:00:34.702 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:34.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:34.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:00:34.806 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:34.858 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:34.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:34.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:34.964 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:35.009 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.0 dump_ops_in_flight 2021-03-24T22:00:35.022 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:35.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:35.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:00:35.127 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:35.184 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:35.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:35.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:00:35.283 INFO:tasks.rados.rados.0.smithi125.stdout:1736: done (8 left) 2021-03-24T22:00:35.284 INFO:tasks.rados.rados.0.smithi125.stdout:1737: rollback oid 26 current snap is 186 2021-03-24T22:00:35.284 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 26 to 181 2021-03-24T22:00:35.285 INFO:tasks.rados.rados.0.smithi125.stdout:1738: snap_remove snap 150 2021-03-24T22:00:35.290 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:35.336 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:35.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:35.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:00:35.344 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:00:35.442 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:35.494 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:35.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:35.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:00:35.599 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:35.660 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:35.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:35.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:00:35.767 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:35.830 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:35.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:35.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:00:35.937 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:35.999 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:36.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:36.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:00:36.105 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:36.160 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:36.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:36.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:00:36.237 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:00:36.266 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:36.288 INFO:tasks.rados.rados.0.smithi125.stdout:1738: done (9 left) 2021-03-24T22:00:36.288 INFO:tasks.rados.rados.0.smithi125.stdout:1739: write (excl) oid 12 current snap is 186 2021-03-24T22:00:36.288 INFO:tasks.rados.rados.0.smithi125.stdout:1739: seq_num 615 ranges {736305=786375,2105367=230938} 2021-03-24T22:00:36.300 INFO:tasks.rados.rados.0.smithi125.stdout:1739: writing smithi12556470-12 from 736305 to 1522680 tid 1 2021-03-24T22:00:36.309 INFO:tasks.rados.rados.0.smithi125.stdout:1739: writing smithi12556470-12 from 2105367 to 2336305 tid 2 2021-03-24T22:00:36.310 INFO:tasks.rados.rados.0.smithi125.stdout:1740: snap_create 2021-03-24T22:00:36.310 INFO:tasks.rados.rados.0.smithi125.stdout:1730: expect (ObjNum 427 snap 113 seq_num 427) 2021-03-24T22:00:36.328 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:36.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:36.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:00:36.434 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:36.479 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:36.483 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:36 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:00:36.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:36 smithi125 conmon[19936]: 24T22:00:34.757640+0000 osd.4 (osd.4) 109 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:00:36.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:36 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:00:36.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:36 smithi125 conmon[19936]: 03-24T22:00:34.759887+0000 osd.4 (osd.4) 110 2021-03-24T22:00:36.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:36 smithi125 conmon[19936]: : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:00:36.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:36 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:00:36.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:36 smithi125 conmon[19936]: 03-24T22:00:34.861978+0000 mgr.y ( 2021-03-24T22:00:36.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:36 smithi125 conmon[19936]: mgr.14144) 504 : cluster [DBG] pgmap v859: 171 pgs: 171 active+clean; 427 MiB data, 1.1 GiB used, 446 GiB / 447 GiB avail; 8.2 MiB/s rd, 2.7 MiB/s wr, 31 op/s; 366 B/s, 3 objects/s recovering 2021-03-24T22:00:36.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:36 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:00:36.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:36 smithi125 conmon[19936]: 00:35.276289+0000 mon.a (mon.0) 1389 : cluster [DBG] osdmap e465: 8 total, 8 up, 5 in 2021-03-24T22:00:36.486 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:36 smithi125 conmon[24927]: cluster 2021-03-24T22:00:34 2021-03-24T22:00:36.486 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:36 smithi125 conmon[24927]: .757640+0000 osd.4 (osd.4) 109 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:00:36.487 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:36 smithi125 conmon[24927]: cluster 2021-03-24T22:00 2021-03-24T22:00:36.487 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:36 smithi125 conmon[24927]: :34.759887+0000 osd.4 (osd.4) 110 : cluster 2021-03-24T22:00:36.487 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:36 smithi125 conmon[24927]: [DBG] purged_snaps scrub ok 2021-03-24T22:00:36.487 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:36 smithi125 conmon[24927]: cluster 2021-03-24T22:00 2021-03-24T22:00:36.487 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:36 smithi125 conmon[24927]: :34.861978+0000 mgr.y (mgr.14144) 504 : cluster 2021-03-24T22:00:36.488 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:36 smithi125 conmon[24927]: [DBG] pgmap v859: 171 pgs: 171 active+clean; 427 MiB data, 1.1 GiB used, 446 GiB / 447 GiB avail; 8.2 MiB/s rd, 2.7 MiB/s wr, 31 op/s; 366 B/s, 3 objects/s recovering 2021-03-24T22:00:36.488 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:36 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:00:36.488 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:36 smithi125 conmon[24927]: 24T22:00:35.276289+0000 mon.a (mon.0) 1389 : cluster [DBG] osdmap e465: 8 total, 8 up, 5 in 2021-03-24T22:00:36.488 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:36 smithi125 conmon[24927]: 2021-03-24T22:00:36.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:36.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:36.525 INFO:tasks.rados.rados.0.smithi125.stdout:1727: expect (ObjNum 595 snap 179 seq_num 595) 2021-03-24T22:00:36.591 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:36.610 INFO:tasks.rados.rados.0.smithi125.stdout:1731: finishing rollback tid 0 to smithi12556470-46 2021-03-24T22:00:36.610 INFO:tasks.rados.rados.0.smithi125.stdout:1731: finishing rollback tid 1 to smithi12556470-46 2021-03-24T22:00:36.610 INFO:tasks.rados.rados.0.smithi125.stdout:1731: finishing rollback tid 2 to smithi12556470-46 2021-03-24T22:00:36.610 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 46 v 485 (ObjNum 497 snap 145 seq_num 497) dirty exists 2021-03-24T22:00:36.611 INFO:tasks.rados.rados.0.smithi125.stdout:1734: expect (ObjNum 515 snap 151 seq_num 515) 2021-03-24T22:00:36.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:36 smithi142 conmon[20125]: cluster 2021-03-24T22:00:34.757640+0000 2021-03-24T22:00:36.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:36 smithi142 conmon[20125]: osd.4 (osd.4) 109 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:00:36.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:36 smithi142 conmon[20125]: cluster 2021-03-24T22:00:36.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:36 smithi142 conmon[20125]: 2021-03-24T22:00:34.759887 2021-03-24T22:00:36.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:36 smithi142 conmon[20125]: +0000 osd.4 (osd 2021-03-24T22:00:36.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:36 smithi142 conmon[20125]: .4) 110 2021-03-24T22:00:36.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:36 smithi142 conmon[20125]: : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:00:36.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:36 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:00:36.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:36 smithi142 conmon[20125]: -03 2021-03-24T22:00:36.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:36 smithi142 conmon[20125]: -24T22:00:34. 2021-03-24T22:00:36.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:36 smithi142 conmon[20125]: 861978+0000 mgr.y (mgr.14144 2021-03-24T22:00:36.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:36 smithi142 conmon[20125]: ) 504 : cluster [DBG] pgmap v859: 171 pgs: 171 active+clean; 427 MiB data, 1.1 GiB used, 446 GiB / 447 GiB avail; 8.2 MiB/s rd, 2.7 MiB/s wr, 31 op/s; 366 B/s, 3 objects/s recovering 2021-03-24T22:00:36.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:36 smithi142 conmon[20125]: 2021-03-24T22:00:36.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:36 smithi142 conmon[20125]: cluster 2021-03-24T22:00:36.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:36 smithi142 conmon[20125]: 2021-03-24T22:00:35 2021-03-24T22:00:36.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:36 smithi142 conmon[20125]: .276289+0000 mon.a (mon.0 2021-03-24T22:00:36.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:36 smithi142 conmon[20125]: ) 1389 : cluster [DBG] osdmap e465: 8 total, 8 up, 5 in 2021-03-24T22:00:36.639 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:36.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:36.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:00:36.727 INFO:tasks.rados.rados.0.smithi125.stdout:1728: finishing rollback tid 0 to smithi12556470-14 2021-03-24T22:00:36.727 INFO:tasks.rados.rados.0.smithi125.stdout:1728: finishing rollback tid 1 to smithi12556470-14 2021-03-24T22:00:36.728 INFO:tasks.rados.rados.0.smithi125.stdout:1728: finishing rollback tid 2 to smithi12556470-14 2021-03-24T22:00:36.728 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 14 v 754 (ObjNum 594 snap 179 seq_num 594) dirty exists 2021-03-24T22:00:36.728 INFO:tasks.rados.rados.0.smithi125.stdout:1729: finishing rollback tid 1 to smithi12556470-40 2021-03-24T22:00:36.728 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 40 v 522 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-03-24T22:00:36.729 INFO:tasks.rados.rados.0.smithi125.stdout:1733: finishing write tid 1 to smithi12556470-43 2021-03-24T22:00:36.729 INFO:tasks.rados.rados.0.smithi125.stdout:1732: finishing copy_from to smithi12556470-45 2021-03-24T22:00:36.729 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 45 v 550 (ObjNum 610 snap 185 seq_num 610) dirty exists 2021-03-24T22:00:36.730 INFO:tasks.rados.rados.0.smithi125.stdout:1732: finishing copy_from racing read to smithi12556470-45 2021-03-24T22:00:36.730 INFO:tasks.rados.rados.0.smithi125.stdout:1733: finishing write tid 2 to smithi12556470-43 2021-03-24T22:00:36.730 INFO:tasks.rados.rados.0.smithi125.stdout:1733: finishing write tid 3 to smithi12556470-43 2021-03-24T22:00:36.730 INFO:tasks.rados.rados.0.smithi125.stdout:1733: finishing write tid 4 to smithi12556470-43 2021-03-24T22:00:36.731 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 43 v 759 (ObjNum 614 snap 186 seq_num 614) dirty exists 2021-03-24T22:00:36.731 INFO:tasks.rados.rados.0.smithi125.stdout:1733: left oid 43 (ObjNum 614 snap 186 seq_num 614) 2021-03-24T22:00:36.731 INFO:tasks.rados.rados.0.smithi125.stdout:1737: finishing rollback tid 1 to smithi12556470-26 2021-03-24T22:00:36.731 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 26 v 436 (ObjNum 1117563808 snap 21916 seq_num 1117588080) dirty dne 2021-03-24T22:00:36.731 INFO:tasks.rados.rados.0.smithi125.stdout:1739: finishing write tid 1 to smithi12556470-12 2021-03-24T22:00:36.732 INFO:tasks.rados.rados.0.smithi125.stdout:1739: finishing write tid 2 to smithi12556470-12 2021-03-24T22:00:36.732 INFO:tasks.rados.rados.0.smithi125.stdout:1739: finishing write tid 3 to smithi12556470-12 2021-03-24T22:00:36.732 INFO:tasks.rados.rados.0.smithi125.stdout:1739: finishing write tid 4 to smithi12556470-12 2021-03-24T22:00:36.732 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 12 v 424 (ObjNum 615 snap 186 seq_num 615) dirty exists 2021-03-24T22:00:36.732 INFO:tasks.rados.rados.0.smithi125.stdout:1739: left oid 12 (ObjNum 615 snap 186 seq_num 615) 2021-03-24T22:00:36.745 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:36.799 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:36.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:36.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:00:36.906 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:36.969 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:36.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:36.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:37.075 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:37.128 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:37.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:37.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:00:37.235 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:37.290 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:37.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:37.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:00:37.300 INFO:tasks.rados.rados.0.smithi125.stdout:1727: done (10 left) 2021-03-24T22:00:37.300 INFO:tasks.rados.rados.0.smithi125.stdout:1728: done (9 left) 2021-03-24T22:00:37.301 INFO:tasks.rados.rados.0.smithi125.stdout:1729: done (8 left) 2021-03-24T22:00:37.301 INFO:tasks.rados.rados.0.smithi125.stdout:1730: done (7 left) 2021-03-24T22:00:37.301 INFO:tasks.rados.rados.0.smithi125.stdout:1731: done (6 left) 2021-03-24T22:00:37.301 INFO:tasks.rados.rados.0.smithi125.stdout:1732: done (5 left) 2021-03-24T22:00:37.301 INFO:tasks.rados.rados.0.smithi125.stdout:1733: done (4 left) 2021-03-24T22:00:37.302 INFO:tasks.rados.rados.0.smithi125.stdout:1734: done (3 left) 2021-03-24T22:00:37.302 INFO:tasks.rados.rados.0.smithi125.stdout:1737: done (2 left) 2021-03-24T22:00:37.302 INFO:tasks.rados.rados.0.smithi125.stdout:1739: done (1 left) 2021-03-24T22:00:37.302 INFO:tasks.rados.rados.0.smithi125.stdout:1740: done (0 left) 2021-03-24T22:00:37.303 INFO:tasks.rados.rados.0.smithi125.stdout:1741: write oid 19 current snap is 187 2021-03-24T22:00:37.303 INFO:tasks.rados.rados.0.smithi125.stdout:1741: seq_num 616 ranges {753112=577977,1880202=472910} 2021-03-24T22:00:37.309 INFO:tasks.rados.rados.0.smithi125.stdout:1741: writing smithi12556470-19 from 753112 to 1331089 tid 1 2021-03-24T22:00:37.316 INFO:tasks.rados.rados.0.smithi125.stdout:1741: writing smithi12556470-19 from 1880202 to 2353112 tid 2 2021-03-24T22:00:37.316 INFO:tasks.rados.rados.0.smithi125.stdout:1742: write oid 14 current snap is 187 2021-03-24T22:00:37.316 INFO:tasks.rados.rados.0.smithi125.stdout:1741: finishing write tid 1 to smithi12556470-19 2021-03-24T22:00:37.316 INFO:tasks.rados.rados.0.smithi125.stdout:1742: seq_num 617 ranges {769919=653226,1938684=431235} 2021-03-24T22:00:37.323 INFO:tasks.rados.rados.0.smithi125.stdout:1742: writing smithi12556470-14 from 769919 to 1423145 tid 1 2021-03-24T22:00:37.329 INFO:tasks.rados.rados.0.smithi125.stdout:1742: writing smithi12556470-14 from 1938684 to 2369919 tid 2 2021-03-24T22:00:37.329 INFO:tasks.rados.rados.0.smithi125.stdout:1743: write oid 26 current snap is 187 2021-03-24T22:00:37.330 INFO:tasks.rados.rados.0.smithi125.stdout:1741: finishing write tid 2 to smithi12556470-19 2021-03-24T22:00:37.330 INFO:tasks.rados.rados.0.smithi125.stdout:1743: seq_num 618 ranges {786726=728475,2280813=105913} 2021-03-24T22:00:37.338 INFO:tasks.rados.rados.0.smithi125.stdout:1743: writing smithi12556470-26 from 786726 to 1515201 tid 1 2021-03-24T22:00:37.344 INFO:tasks.rados.rados.0.smithi125.stdout:1743: writing smithi12556470-26 from 2280813 to 2386726 tid 2 2021-03-24T22:00:37.344 INFO:tasks.rados.rados.0.smithi125.stdout:1744: copy_from oid 17 from oid 46 current snap is 187 2021-03-24T22:00:37.344 INFO:tasks.rados.rados.0.smithi125.stdout:1741: finishing write tid 3 to smithi12556470-19 2021-03-24T22:00:37.344 INFO:tasks.rados.rados.0.smithi125.stdout:1741: finishing write tid 4 to smithi12556470-19 2021-03-24T22:00:37.345 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 19 v 390 (ObjNum 616 snap 187 seq_num 616) dirty exists 2021-03-24T22:00:37.345 INFO:tasks.rados.rados.0.smithi125.stdout:1741: left oid 19 (ObjNum 616 snap 187 seq_num 616) 2021-03-24T22:00:37.345 INFO:tasks.rados.rados.0.smithi125.stdout:1742: finishing write tid 1 to smithi12556470-14 2021-03-24T22:00:37.345 INFO:tasks.rados.rados.0.smithi125.stdout:1742: finishing write tid 2 to smithi12556470-14 2021-03-24T22:00:37.345 INFO:tasks.rados.rados.0.smithi125.stdout:1742: finishing write tid 3 to smithi12556470-14 2021-03-24T22:00:37.346 INFO:tasks.rados.rados.0.smithi125.stdout:1742: finishing write tid 4 to smithi12556470-14 2021-03-24T22:00:37.346 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 14 v 793 (ObjNum 617 snap 187 seq_num 617) dirty exists 2021-03-24T22:00:37.346 INFO:tasks.rados.rados.0.smithi125.stdout:1742: left oid 14 (ObjNum 617 snap 187 seq_num 617) 2021-03-24T22:00:37.346 INFO:tasks.rados.rados.0.smithi125.stdout:1741: done (3 left) 2021-03-24T22:00:37.347 INFO:tasks.rados.rados.0.smithi125.stdout:1742: done (2 left) 2021-03-24T22:00:37.347 INFO:tasks.rados.rados.0.smithi125.stdout:1745: read oid 19 snap -1 2021-03-24T22:00:37.347 INFO:tasks.rados.rados.0.smithi125.stdout:1745: expect (ObjNum 616 snap 187 seq_num 616) 2021-03-24T22:00:37.347 INFO:tasks.rados.rados.0.smithi125.stdout:1746: read oid 15 snap -1 2021-03-24T22:00:37.347 INFO:tasks.rados.rados.0.smithi125.stdout:1746: expect deleted 2021-03-24T22:00:37.348 INFO:tasks.rados.rados.0.smithi125.stdout:1747: snap_create 2021-03-24T22:00:37.349 INFO:tasks.rados.rados.0.smithi125.stdout:1743: finishing write tid 1 to smithi12556470-26 2021-03-24T22:00:37.350 INFO:tasks.rados.rados.0.smithi125.stdout:1745: expect (ObjNum 616 snap 187 seq_num 616) 2021-03-24T22:00:37.366 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-24T22:00:37.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:37 smithi142 conmon[20125]: cluster 2021-03-24T22:00:37.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:37 smithi142 conmon[20125]: 2021-03-24T22:00:37.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:37 smithi142 conmon[20125]: 2021-03-24T22 2021-03-24T22:00:37.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:37 smithi142 conmon[20125]: :00:36.285501+0000 mon.a (mon.0) 1390 : cluster [DBG] osdmap e466: 8 total, 8 up, 5 in 2021-03-24T22:00:37.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:37 smithi142 conmon[20125]: audit 2021 2021-03-24T22:00:37.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:37 smithi142 conmon[20125]: -03-24T22:00:36.864763+0000 mon.a (mon 2021-03-24T22:00:37.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:37 smithi142 conmon[20125]: .0) 1391 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "23"}]: dispatch 2021-03-24T22:00:37.397 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:37.434 INFO:tasks.rados.rados.0.smithi125.stdout:1743: finishing write tid 2 to smithi12556470-26 2021-03-24T22:00:37.435 INFO:tasks.rados.rados.0.smithi125.stdout:1743: finishing write tid 3 to smithi12556470-26 2021-03-24T22:00:37.435 INFO:tasks.rados.rados.0.smithi125.stdout:1743: finishing write tid 4 to smithi12556470-26 2021-03-24T22:00:37.436 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 26 v 439 (ObjNum 618 snap 187 seq_num 618) dirty exists 2021-03-24T22:00:37.436 INFO:tasks.rados.rados.0.smithi125.stdout:1743: left oid 26 (ObjNum 618 snap 187 seq_num 618) 2021-03-24T22:00:37.436 INFO:tasks.rados.rados.0.smithi125.stdout:1744: finishing copy_from to smithi12556470-17 2021-03-24T22:00:37.437 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 17 v 485 (ObjNum 497 snap 145 seq_num 497) dirty exists 2021-03-24T22:00:37.437 INFO:tasks.rados.rados.0.smithi125.stdout:1744: finishing copy_from racing read to smithi12556470-17 2021-03-24T22:00:37.451 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:37.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:37.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:00:37.557 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:37.588 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:37 smithi125 conmon[24927]: cluster 2021-03-24T22:00:36.285501+0000 2021-03-24T22:00:37.589 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:37 smithi125 conmon[24927]: mon.a (mon.0) 1390 : cluster [DBG] osdmap e466: 8 total, 8 up, 5 in 2021-03-24T22:00:37.589 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:37 smithi125 conmon[24927]: audit 2021-03-24T22:00 2021-03-24T22:00:37.589 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:37 smithi125 conmon[24927]: :36.864763+0000 mon.a (mon.0) 1391 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "23"}]: dispatch 2021-03-24T22:00:37.590 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:37 smithi125 conmon[19936]: cluster 2021-03-24T22:00:37.590 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:37 smithi125 conmon[19936]: 2021-03-24 2021-03-24T22:00:37.590 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:37 smithi125 conmon[19936]: T22:00:36.285501+0000 mon.a (mon.0) 1390 : cluster [DBG] osdmap e466: 8 total, 8 up, 5 in 2021-03-24T22:00:37.590 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:37 smithi125 conmon[19936]: audit 2021- 2021-03-24T22:00:37.591 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:37 smithi125 conmon[19936]: 03-24T22:00:36.864763+0000 mon.a (mon. 2021-03-24T22:00:37.591 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:37 smithi125 conmon[19936]: 0) 1391 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "23"}]: dispatch 2021-03-24T22:00:37.622 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:37.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:37.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:00:37.728 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:37.789 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:37.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:37.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:00:37.895 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:37.953 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:37.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:37.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:00:38.050 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:00:38.059 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:38.117 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:38.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:38.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:00:38.223 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:38.298 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:38.302 INFO:tasks.rados.rados.0.smithi125.stdout:1743: done (4 left) 2021-03-24T22:00:38.303 INFO:tasks.rados.rados.0.smithi125.stdout:1744: done (3 left) 2021-03-24T22:00:38.303 INFO:tasks.rados.rados.0.smithi125.stdout:1745: done (2 left) 2021-03-24T22:00:38.303 INFO:tasks.rados.rados.0.smithi125.stdout:1746: done (1 left) 2021-03-24T22:00:38.304 INFO:tasks.rados.rados.0.smithi125.stdout:1747: done (0 left) 2021-03-24T22:00:38.304 INFO:tasks.rados.rados.0.smithi125.stdout:1748: snap_remove snap 180 2021-03-24T22:00:38.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:38.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:00:38.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:38 smithi142 conmon[20125]: cluster 2021-03-24T22:00:36.862913+0000 mgr.y 2021-03-24T22:00:38.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:38 smithi142 conmon[20125]: (mgr.14144) 505 : cluster [DBG] pgmap v862: 170 pgs: 170 active+clean; 423 MiB data, 1.0 GiB used, 446 GiB / 447 GiB avail; 11 MiB/s rd, 2.7 MiB/s wr, 37 op/s; 222 B/s, 2 objects/s recovering 2021-03-24T22:00:38.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:38 smithi142 conmon[20125]: audit 2021-03-24T22:00:37.296656+0000 mon.a (mon.0) 1392 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "23"}]': finished 2021-03-24T22:00:38.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:38 smithi142 conmon[20125]: cluster 2021-03-24T22:00:37.296764+0000 mon.a (mon.0) 2021-03-24T22:00:38.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:38 smithi142 conmon[20125]: 1393 : cluster [DBG] osdmap e467: 8 total, 8 up, 5 in 2021-03-24T22:00:38.409 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:38.477 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:38.479 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:38 smithi125 conmon[24927]: cluster 2021-03-24T22:00:36.862913+0000 mgr.y (mgr. 2021-03-24T22:00:38.479 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:38 smithi125 conmon[24927]: 14144) 505 : cluster [DBG] pgmap v862: 170 pgs: 170 active+clean; 423 MiB data, 1.0 GiB used, 446 GiB / 447 GiB avail; 11 MiB/s rd, 2.7 MiB/s wr, 37 op/s; 222 B/s, 2 objects/s recovering 2021-03-24T22:00:38.479 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:38 smithi125 conmon[24927]: audit 2021-03-24T22:00:37.296656+0000 mon.a (mon.0) 1392 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "23"}]': finished 2021-03-24T22:00:38.480 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:38 smithi125 conmon[24927]: cluster 2021-03-24T22:00:37.296764+0000 mon.a (mon.0) 1393 : cluster [DBG] osdmap e467: 8 total, 8 up, 5 in 2021-03-24T22:00:38.481 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:38 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:00:38.481 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:38 smithi125 conmon[19936]: 00:36.862913+0000 mgr.y (mgr.14144) 505 : cluster [DBG] pgmap v862: 170 pgs: 170 active+clean; 423 MiB data, 1.0 GiB used, 446 GiB / 447 GiB avail; 11 MiB/s rd, 2.7 MiB/s wr, 37 op/s; 222 B/s, 2 objects/s recovering 2021-03-24T22:00:38.481 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:38 smithi125 conmon[19936]: audit 2021-03-24T22:00:38.481 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:38 smithi125 conmon[19936]: 2021-03-24T22:00:37. 2021-03-24T22:00:38.482 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:38 smithi125 conmon[19936]: 296656+0000 mon.a (mon.0) 1392 2021-03-24T22:00:38.482 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:38 smithi125 conmon[19936]: : audit [INF] 2021-03-24T22:00:38.482 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:38 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "23"}]': finished 2021-03-24T22:00:38.482 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:38 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:00:38.483 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:38 smithi125 conmon[19936]: -03-24T22: 2021-03-24T22:00:38.483 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:38 smithi125 conmon[19936]: 00:37.296764+0000 2021-03-24T22:00:38.483 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:38 smithi125 conmon[19936]: mon.a (mon.0) 1393 2021-03-24T22:00:38.483 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:38 smithi125 conmon[19936]: : cluster [DBG] osdmap e467: 8 total, 8 up, 5 in 2021-03-24T22:00:38.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:38 smithi125 conmon[19936]: 2021-03-24T22:00:38.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:38.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:00:38.587 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:38.645 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:38.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:38.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:38.752 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:38.812 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:38.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:38.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:00:38.919 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:38.971 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:38.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:38.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:00:39.077 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:39.135 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:39.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:39.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:00:39.242 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:39.308 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:39.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:39.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:00:39.320 INFO:tasks.rados.rados.0.smithi125.stdout:1748: done (0 left) 2021-03-24T22:00:39.320 INFO:tasks.rados.rados.0.smithi125.stdout:1749: read oid 20 snap -1 2021-03-24T22:00:39.321 INFO:tasks.rados.rados.0.smithi125.stdout:1749: expect (ObjNum 462 snap 129 seq_num 462) 2021-03-24T22:00:39.321 INFO:tasks.rados.rados.0.smithi125.stdout:1750: write oid 17 current snap is 188 2021-03-24T22:00:39.321 INFO:tasks.rados.rados.0.smithi125.stdout:1750: seq_num 619 ranges {403533=403724,1539295=652201} 2021-03-24T22:00:39.321 INFO:tasks.rados.rados.0.smithi125.stdout:1750: writing smithi12556470-17 from 403533 to 807257 tid 1 2021-03-24T22:00:39.329 INFO:tasks.rados.rados.0.smithi125.stdout:1750: writing smithi12556470-17 from 1539295 to 2191496 tid 2 2021-03-24T22:00:39.330 INFO:tasks.rados.rados.0.smithi125.stdout:1751: read oid 44 snap -1 2021-03-24T22:00:39.330 INFO:tasks.rados.rados.0.smithi125.stdout:1751: expect deleted 2021-03-24T22:00:39.330 INFO:tasks.rados.rados.0.smithi125.stdout:1752: snap_create 2021-03-24T22:00:39.344 INFO:tasks.rados.rados.0.smithi125.stdout:1750: finishing write tid 1 to smithi12556470-17 2021-03-24T22:00:39.344 INFO:tasks.rados.rados.0.smithi125.stdout:1750: finishing write tid 2 to smithi12556470-17 2021-03-24T22:00:39.344 INFO:tasks.rados.rados.0.smithi125.stdout:1750: finishing write tid 3 to smithi12556470-17 2021-03-24T22:00:39.344 INFO:tasks.rados.rados.0.smithi125.stdout:1750: finishing write tid 4 to smithi12556470-17 2021-03-24T22:00:39.345 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 17 v 548 (ObjNum 619 snap 188 seq_num 619) dirty exists 2021-03-24T22:00:39.345 INFO:tasks.rados.rados.0.smithi125.stdout:1750: left oid 17 (ObjNum 619 snap 188 seq_num 619) 2021-03-24T22:00:39.345 INFO:tasks.rados.rados.0.smithi125.stdout:1749: expect (ObjNum 462 snap 129 seq_num 462) 2021-03-24T22:00:39.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:39 smithi142 conmon[20125]: cluster 2021-03-24T22:00:38.298236+0000 mon.a ( 2021-03-24T22:00:39.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:39 smithi142 conmon[20125]: mon.0) 1394 : cluster [DBG] osdmap e468: 8 total, 8 up, 5 in 2021-03-24T22:00:39.412 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:39.468 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:39.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:39.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:00:39.489 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:39 smithi125 conmon[24927]: cluster 2021-03-24T22:00:38.298236+0000 mon.a (mon.0) 1394 2021-03-24T22:00:39.490 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:39 smithi125 conmon[24927]: : cluster [DBG] osdmap e468: 8 total, 8 up, 5 in 2021-03-24T22:00:39.490 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:39 smithi125 conmon[19936]: cluster 2021-03-24T22:00:38.298236+0000 mon.a (mon.0) 1394 : cluster [DBG] 2021-03-24T22:00:39.490 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:39 smithi125 conmon[19936]: osdmap e468: 8 total, 8 up, 5 in 2021-03-24T22:00:39.572 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:39.625 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:39.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:39.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:00:39.732 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:39.783 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:39.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:39.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:39.888 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:39.954 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:39.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:39.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:40.059 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:40.117 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:40.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:40.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:40.224 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:40.256 INFO:tasks.rados.rados.0.smithi125.stdout:1749: done (3 left) 2021-03-24T22:00:40.256 INFO:tasks.rados.rados.0.smithi125.stdout:1750: done (2 left) 2021-03-24T22:00:40.257 INFO:tasks.rados.rados.0.smithi125.stdout:1751: done (1 left) 2021-03-24T22:00:40.257 INFO:tasks.rados.rados.0.smithi125.stdout:1752: done (0 left) 2021-03-24T22:00:40.257 INFO:tasks.rados.rados.0.smithi125.stdout:1753: read oid 30 snap -1 2021-03-24T22:00:40.257 INFO:tasks.rados.rados.0.smithi125.stdout:1753: expect (ObjNum 601 snap 180 seq_num 601) 2021-03-24T22:00:40.258 INFO:tasks.rados.rados.0.smithi125.stdout:1754: rollback oid 12 current snap is 189 2021-03-24T22:00:40.258 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 12 to 184 2021-03-24T22:00:40.258 INFO:tasks.rados.rados.0.smithi125.stdout:1755: read oid 45 snap -1 2021-03-24T22:00:40.259 INFO:tasks.rados.rados.0.smithi125.stdout:1755: expect (ObjNum 610 snap 185 seq_num 610) 2021-03-24T22:00:40.259 INFO:tasks.rados.rados.0.smithi125.stdout:1756: delete oid 41 current snap is 189 2021-03-24T22:00:40.267 INFO:teuthology.orchestra.run.smithi125.stderr:nodeep-scrub is set 2021-03-24T22:00:40.269 INFO:tasks.rados.rados.0.smithi125.stdout:1753: expect (ObjNum 601 snap 180 seq_num 601) 2021-03-24T22:00:40.281 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:40.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:40.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:00:40.343 INFO:tasks.rados.rados.0.smithi125.stdout:1753: done (3 left) 2021-03-24T22:00:40.344 INFO:tasks.rados.rados.0.smithi125.stdout:1756: done (2 left) 2021-03-24T22:00:40.344 INFO:tasks.rados.rados.0.smithi125.stdout:1757: write (excl) oid 28 current snap is 189 2021-03-24T22:00:40.344 INFO:tasks.rados.rados.0.smithi125.stdout:1757: seq_num 621 ranges {437147=554222,1656259=655870} 2021-03-24T22:00:40.349 INFO:tasks.rados.rados.0.smithi125.stdout:1757: writing smithi12556470-28 from 437147 to 991369 tid 1 2021-03-24T22:00:40.356 INFO:tasks.rados.rados.0.smithi125.stdout:1757: writing smithi12556470-28 from 1656259 to 2312129 tid 2 2021-03-24T22:00:40.356 INFO:tasks.rados.rados.0.smithi125.stdout:1758: snap_remove snap 177 2021-03-24T22:00:40.386 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:40.444 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:40.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:40.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:00:40.551 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:40.566 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:40 smithi125 conmon[19936]: cluster 2021-03-24T22:00: 2021-03-24T22:00:40.566 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:40 smithi125 conmon[19936]: 38.863741+0000 mgr.y (mgr.14144) 506 : cluster [DBG] pgmap v865: 170 pgs: 170 active+clean; 421 MiB data, 1.0 GiB used, 446 GiB / 447 GiB avail; 2.5 MiB/s rd, 1 op/s 2021-03-24T22:00:40.567 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:40 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:00:40.567 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:40 smithi125 conmon[19936]: 24T22:00:39.310790+0000 mon.a (mon.0) 1395 : cluster [DBG] osdmap e469: 8 total, 8 up, 5 in 2021-03-24T22:00:40.567 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:40 smithi125 conmon[19936]: audit 2021-03-24T22:00:40.567 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:40 smithi125 conmon[19936]: 2021-03- 2021-03-24T22:00:40.568 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:40 smithi125 conmon[19936]: 24T22:00:39.660153+0000 mon.a ( 2021-03-24T22:00:40.568 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:40 smithi125 conmon[19936]: mon.0) 1396 : audit [INF] from='client.? v1:172.21.15.125:0/2471315608' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:00:40.568 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:40 smithi125 conmon[19936]: audit 2021-03-24T22:00:40.568 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:40 smithi125 conmon[19936]: 2021-03-24T22:00:39.792902+0000 2021-03-24T22:00:40.568 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:40 smithi125 conmon[19936]: mon.a (mon.0) 1397 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:00:40.569 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:40 smithi125 conmon[19936]: 2021-03-24T22:00:40.569 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:40 smithi125 conmon[19936]: cluster 2021-03-24T22:00:40. 2021-03-24T22:00:40.570 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:40 smithi125 conmon[19936]: 240139+0000 mon.a (mon.0) 1398 : cluster 2021-03-24T22:00:40.570 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:40 smithi125 conmon[19936]: [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:00:40.571 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:40 smithi125 conmon[19936]: audit 2021-03-24T22: 2021-03-24T22:00:40.571 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:40 smithi125 conmon[19936]: 00:40.252425+0000 mon.a (mon.0) 2021-03-24T22:00:40.571 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:40 smithi125 conmon[19936]: 1399 : audit [INF] from='client.? v1:172.21.15.125:0/2471315608' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T22:00:40.572 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:40 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T22:00:40.572 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:40 smithi125 conmon[19936]: -24T22:00:40.252614+0000 mon.a ( 2021-03-24T22:00:40.573 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:40 smithi125 conmon[19936]: mon.0) 1400 : cluster [DBG] osdmap e470: 8 total, 8 up, 5 in 2021-03-24T22:00:40.573 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:40 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:00:40.574 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:40 smithi125 conmon[24927]: 00:38.863741+0000 mgr.y (mgr.14144) 506 : cluster [DBG] pgmap v865: 170 pgs: 170 active+clean; 421 MiB data, 1.0 GiB used, 446 GiB / 447 GiB avail; 2.5 MiB/s rd, 1 op/s 2021-03-24T22:00:40.574 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:40 smithi125 conmon[24927]: cluster 2021-03-24T22:00: 2021-03-24T22:00:40.575 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:40 smithi125 conmon[24927]: 39.310790+0000 mon.a (mon.0) 1395 : cluster 2021-03-24T22:00:40.575 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:40 smithi125 conmon[24927]: [DBG] osdmap e469: 8 total, 8 up, 5 in 2021-03-24T22:00:40.576 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:40 smithi125 conmon[24927]: audit 2021-03-24 2021-03-24T22:00:40.576 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:40 smithi125 conmon[24927]: T22:00:39.660153+0000 mon.a (mon 2021-03-24T22:00:40.576 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:40 smithi125 conmon[24927]: .0) 1396 : audit [INF] from='client.? v1:172.21.15.125:0/2471315608' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:00:40.577 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:40 smithi125 conmon[24927]: audit 2021 2021-03-24T22:00:40.577 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:40 smithi125 conmon[24927]: -03-24T22:00:39.792902+0000 2021-03-24T22:00:40.578 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:40 smithi125 conmon[24927]: mon.a (mon.0) 1397 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:00:40.578 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:40 smithi125 conmon[24927]: 2021-03-24T22:00:40.579 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:40 smithi125 conmon[24927]: cluster 2021-03-24T22:00:40. 2021-03-24T22:00:40.579 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:40 smithi125 conmon[24927]: 240139+0000 mon.a (mon.0) 1398 : cluster 2021-03-24T22:00:40.579 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:40 smithi125 conmon[24927]: [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:00:40.580 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:40 smithi125 conmon[24927]: audit 2021-03-24T22:00 2021-03-24T22:00:40.580 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:40 smithi125 conmon[24927]: :40.252425+0000 mon.a (mon.0) 1399 2021-03-24T22:00:40.581 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:40 smithi125 conmon[24927]: : audit [INF] from='client.? v1:172.21.15.125:0/2471315608' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T22:00:40.581 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:40 smithi125 conmon[24927]: cluster 2021-03-24T22:00:40.582 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:40 smithi125 conmon[24927]: 2021-03- 2021-03-24T22:00:40.582 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:40 smithi125 conmon[24927]: 24T22:00:40.252614+0000 mon.a 2021-03-24T22:00:40.583 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:40 smithi125 conmon[24927]: (mon.0) 1400 : cluster [DBG] osdmap e470: 8 total, 8 up, 5 in 2021-03-24T22:00:40.609 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:40.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:40.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:00:40.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:40 smithi142 conmon[20125]: cluster 2021-03-24T22:00:38.863741+0000 mgr.y (mgr.14144) 506 : 2021-03-24T22:00:40.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:40 smithi142 conmon[20125]: cluster [DBG] pgmap v865: 170 pgs: 170 active+clean; 421 MiB data, 1.0 GiB used, 446 GiB / 447 GiB avail; 2.5 MiB/s rd, 1 op/s 2021-03-24T22:00:40.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:40 smithi142 conmon[20125]: cluster 2021-03-24T22:00:39.310790+0000 mon.a (mon. 2021-03-24T22:00:40.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:40 smithi142 conmon[20125]: 0) 1395 : cluster [DBG] osdmap e469: 8 total, 8 up, 5 in 2021-03-24T22:00:40.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:40 smithi142 conmon[20125]: audit 2021-03-24T22:00:40.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:40 smithi142 conmon[20125]: 2021-03-24T22:00:39.660153+0000 2021-03-24T22:00:40.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:40 smithi142 conmon[20125]: mon.a (mon.0) 1396 : audit [INF] 2021-03-24T22:00:40.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:40 smithi142 conmon[20125]: from='client.? v1:172.21.15.125:0/2471315608' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:00:40.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:40 smithi142 conmon[20125]: audit 2021-03-24T22:00: 2021-03-24T22:00:40.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:40 smithi142 conmon[20125]: 39.792902+0000 mon.a (mon.0) 1397 2021-03-24T22:00:40.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:40 smithi142 conmon[20125]: : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:00:40.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:40 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:00:40.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:40 smithi142 conmon[20125]: 24T22:00:40.240139+0000 2021-03-24T22:00:40.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:40 smithi142 conmon[20125]: mon.a (mon.0) 1398 : cluster [WRN] 2021-03-24T22:00:40.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:40 smithi142 conmon[20125]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:00:40.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:40 smithi142 conmon[20125]: audit 2021-03-24T22:00:40.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:40 smithi142 conmon[20125]: 2021-03-24T22:00:40.252425+0000 2021-03-24T22:00:40.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:40 smithi142 conmon[20125]: mon.a (mon.0) 1399 : audit [INF] 2021-03-24T22:00:40.625 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:40 smithi142 conmon[20125]: from='client.? v1:172.21.15.125:0/2471315608' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T22:00:40.625 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:40 smithi142 conmon[20125]: cluster 2021-03-24T22:00:40.626 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:40 smithi142 conmon[20125]: 2021-03-24T22:00:40.252614+0000 2021-03-24T22:00:40.626 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:40 smithi142 conmon[20125]: mon.a (mon.0) 1400 : cluster [DBG] 2021-03-24T22:00:40.626 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:40 smithi142 conmon[20125]: osdmap e470: 8 total, 8 up, 5 in 2021-03-24T22:00:40.715 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:40.772 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:40.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:40.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:00:40.880 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:40.934 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:40.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:40.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:00:41.042 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:41.100 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:41.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:41.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:41.206 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:41.262 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5] out_osds: [6, 1, 7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2021-03-24T22:00:41.263 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-24T22:00:41.263 INFO:tasks.thrashosds.thrasher:Setting osd 4 primary_affinity to 0.000000 2021-03-24T22:00:41.263 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd primary-affinity 4 0 2021-03-24T22:00:41.265 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:41.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:41.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:00:41.338 INFO:tasks.rados.rados.0.smithi125.stdout:1758: done (3 left) 2021-03-24T22:00:41.338 INFO:tasks.rados.rados.0.smithi125.stdout:1759: read oid 33 snap 158 2021-03-24T22:00:41.339 INFO:tasks.rados.rados.0.smithi125.stdout:1759: expect (ObjNum 298 snap 74 seq_num 298) 2021-03-24T22:00:41.339 INFO:tasks.rados.rados.0.smithi125.stdout:1760: read oid 24 snap -1 2021-03-24T22:00:41.339 INFO:tasks.rados.rados.0.smithi125.stdout:1760: expect deleted 2021-03-24T22:00:41.339 INFO:tasks.rados.rados.0.smithi125.stdout:1761: snap_remove snap 183 2021-03-24T22:00:41.369 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:41.377 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:41 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:00:41.377 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:41 smithi125 conmon[19936]: 00:39.628923+0000 osd.7 (osd.7) 93 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:00:41.377 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:41 smithi125 conmon[19936]: cluster 2021-03-24T22:00: 2021-03-24T22:00:41.378 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:41 smithi125 conmon[19936]: 39.630262+0000 osd.7 (osd.7 2021-03-24T22:00:41.378 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:41 smithi125 conmon[19936]: ) 94 : cluster [DBG] 2021-03-24T22:00:41.378 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:41 smithi125 conmon[19936]: purged_snaps scrub ok 2021-03-24T22:00:41.378 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:41 smithi125 conmon[19936]: audit 2021-03- 2021-03-24T22:00:41.379 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:41 smithi125 conmon[19936]: 24T22:00:40.866239+0000 2021-03-24T22:00:41.379 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:41 smithi125 conmon[19936]: mon.a (mon.0) 1401 : 2021-03-24T22:00:41.379 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:41 smithi125 conmon[19936]: audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "22"}]: dispatch 2021-03-24T22:00:41.379 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:41 smithi125 conmon[24927]: cluster 2021-03-24T22:00:41.380 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:41 smithi125 conmon[24927]: 2021-03-24T22:00:41.380 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:41 smithi125 conmon[24927]: 2021 2021-03-24T22:00:41.380 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:41 smithi125 conmon[24927]: - 2021-03-24T22:00:41.381 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:41 smithi125 conmon[24927]: 03 2021-03-24T22:00:41.381 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:41 smithi125 conmon[24927]: - 2021-03-24T22:00:41.381 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:41 smithi125 conmon[24927]: 24 2021-03-24T22:00:41.381 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:41 smithi125 conmon[24927]: T 2021-03-24T22:00:41.382 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:41 smithi125 conmon[24927]: 22 2021-03-24T22:00:41.382 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:41 smithi125 conmon[24927]: : 2021-03-24T22:00:41.382 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:41 smithi125 conmon[24927]: 00 2021-03-24T22:00:41.382 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:41 smithi125 conmon[24927]: : 2021-03-24T22:00:41.383 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:41 smithi125 conmon[24927]: 39.628923+0000 osd.7 (osd.7) 93 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:00:41.383 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:41 smithi125 conmon[24927]: cluster 2021-03-24T22:00:39.630262+0000 osd.7 (osd.7) 94 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:00:41.383 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:41 smithi125 conmon[24927]: audit 2021-03-24T22:00:40.866239+0000 mon.a (mon.0) 1401 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "22"}]: dispatch 2021-03-24T22:00:41.420 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:41.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:41.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:00:41.527 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:41.576 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:41.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:41.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:00:41.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:41 smithi142 conmon[20125]: cluster 2021-03-24T22:00:41.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:41 smithi142 conmon[20125]: 2021-03-24T22:00:41.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:41 smithi142 conmon[20125]: 2021-03-24T22:00: 2021-03-24T22:00:41.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:41 smithi142 conmon[20125]: 39.628923+0000 osd.7 (osd.7) 93 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:00:41.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:41 smithi142 conmon[20125]: cluster 2021-03-24T22:00:41.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:41 smithi142 conmon[20125]: 2021-03-24T22:00:39. 2021-03-24T22:00:41.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:41 smithi142 conmon[20125]: 630262+0000 osd.7 (osd.7) 94 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:00:41.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:41 smithi142 conmon[20125]: 2021-03-24T22:00:41.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:41 smithi142 conmon[20125]: audit 2021- 2021-03-24T22:00:41.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:41 smithi142 conmon[20125]: 03-24T22:00:40.866239+0000 mon.a (mon.0) 1401 : audit 2021-03-24T22:00:41.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:41 smithi142 conmon[20125]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "22"}]: dispatch 2021-03-24T22:00:41.681 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:41.739 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:41.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:41.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:00:41.743 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-24T22:00:41.743 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-24T22:00:41.744 INFO:teuthology.orchestra.run.smithi125.stdout: "num_ops": 0 2021-03-24T22:00:41.744 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-24T22:00:41.844 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:41.902 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:41.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:41.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:00:42.008 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:42.062 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:42.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:42.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:00:42.167 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:42.221 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:42.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:42.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:42.328 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:42.342 INFO:tasks.rados.rados.0.smithi125.stdout:1761: done (5 left) 2021-03-24T22:00:42.342 INFO:tasks.rados.rados.0.smithi125.stdout:1762: rollback oid 47 current snap is 189 2021-03-24T22:00:42.343 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 47 to 158 2021-03-24T22:00:42.343 INFO:tasks.rados.rados.0.smithi125.stdout:1763: read oid 7 snap -1 2021-03-24T22:00:42.343 INFO:tasks.rados.rados.0.smithi125.stdout:1763: expect (ObjNum 588 snap 177 seq_num 588) 2021-03-24T22:00:42.344 INFO:tasks.rados.rados.0.smithi125.stdout:1764: rollback oid 49 current snap is 189 2021-03-24T22:00:42.344 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 49 to 135 2021-03-24T22:00:42.345 INFO:tasks.rados.rados.0.smithi125.stdout:1765: read oid 8 snap -1 2021-03-24T22:00:42.345 INFO:tasks.rados.rados.0.smithi125.stdout:1765: expect (ObjNum 580 snap 176 seq_num 580) 2021-03-24T22:00:42.345 INFO:tasks.rados.rados.0.smithi125.stdout:1766: read oid 20 snap 181 2021-03-24T22:00:42.346 INFO:tasks.rados.rados.0.smithi125.stdout:1766: expect (ObjNum 462 snap 129 seq_num 462) 2021-03-24T22:00:42.346 INFO:tasks.rados.rados.0.smithi125.stdout:1767: read oid 38 snap 167 2021-03-24T22:00:42.346 INFO:tasks.rados.rados.0.smithi125.stdout:1767: expect (ObjNum 489 snap 140 seq_num 489) 2021-03-24T22:00:42.346 INFO:tasks.rados.rados.0.smithi125.stdout:1768: copy_from oid 14 from oid 25 current snap is 189 2021-03-24T22:00:42.347 INFO:tasks.rados.rados.0.smithi125.stdout:1769: write oid 5 current snap is 189 2021-03-24T22:00:42.347 INFO:tasks.rados.rados.0.smithi125.stdout:1769: seq_num 622 ranges {453954=629471,1598388=515881} 2021-03-24T22:00:42.354 INFO:tasks.rados.rados.0.smithi125.stdout:1769: writing smithi12556470-5 from 453954 to 1083425 tid 1 2021-03-24T22:00:42.360 INFO:tasks.rados.rados.0.smithi125.stdout:1769: writing smithi12556470-5 from 1598388 to 2114269 tid 2 2021-03-24T22:00:42.361 INFO:tasks.rados.rados.0.smithi125.stdout:1755: expect (ObjNum 610 snap 185 seq_num 610) 2021-03-24T22:00:42.394 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:42.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:42.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:00:42.503 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:42.505 INFO:tasks.rados.rados.0.smithi125.stdout:1754: finishing rollback tid 0 to smithi12556470-12 2021-03-24T22:00:42.506 INFO:tasks.rados.rados.0.smithi125.stdout:1754: finishing rollback tid 1 to smithi12556470-12 2021-03-24T22:00:42.506 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 12 v 431 (ObjNum 1116667584 snap 21916 seq_num 1116857136) dirty dne 2021-03-24T22:00:42.506 INFO:tasks.rados.rados.0.smithi125.stdout:1757: finishing write tid 1 to smithi12556470-28 2021-03-24T22:00:42.507 INFO:tasks.rados.rados.0.smithi125.stdout:1754: done (12 left) 2021-03-24T22:00:42.507 INFO:tasks.rados.rados.0.smithi125.stdout:1755: done (11 left) 2021-03-24T22:00:42.507 INFO:tasks.rados.rados.0.smithi125.stdout:1770: delete oid 4 current snap is 189 2021-03-24T22:00:42.507 INFO:tasks.rados.rados.0.smithi125.stdout:1757: finishing write tid 2 to smithi12556470-28 2021-03-24T22:00:42.507 INFO:tasks.rados.rados.0.smithi125.stdout:1757: finishing write tid 3 to smithi12556470-28 2021-03-24T22:00:42.508 INFO:tasks.rados.rados.0.smithi125.stdout:1757: finishing write tid 4 to smithi12556470-28 2021-03-24T22:00:42.508 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 28 v 389 (ObjNum 621 snap 189 seq_num 621) dirty exists 2021-03-24T22:00:42.508 INFO:tasks.rados.rados.0.smithi125.stdout:1757: left oid 28 (ObjNum 621 snap 189 seq_num 621) 2021-03-24T22:00:42.508 INFO:tasks.rados.rados.0.smithi125.stdout:1759: expect (ObjNum 298 snap 74 seq_num 298) 2021-03-24T22:00:42.562 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:42.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:42.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:00:42.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:42 smithi142 conmon[20125]: cluster 2021-03-24T22:00:40.864500+0000 mgr.y (mgr.14144 2021-03-24T22:00:42.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:42 smithi142 conmon[20125]: ) 507 : cluster [DBG] pgmap v868: 169 pgs: 169 active+clean; 416 MiB data, 1.0 GiB used, 446 GiB / 447 GiB avail; 3.4 MiB/s rd, 974 KiB/s wr, 4 op/s 2021-03-24T22:00:42.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:42 smithi142 conmon[20125]: audit 2021-03-24T22:00:41.333598+0000 mon.a (mon.0) 1402 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "22"}]': finished 2021-03-24T22:00:42.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:42 smithi142 conmon[20125]: cluster 2021-03-24T22:00:41.333719+0000 mon.a ( 2021-03-24T22:00:42.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:42 smithi142 conmon[20125]: mon.0) 1403 : cluster [DBG] osdmap e471: 8 total, 8 up, 5 in 2021-03-24T22:00:42.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[24927]: cluster 2021-03-24T22:00:42.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[24927]: 2021-03-24T22:00:42.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[24927]: 2021 2021-03-24T22:00:42.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[24927]: - 2021-03-24T22:00:42.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[24927]: 03 2021-03-24T22:00:42.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[24927]: - 2021-03-24T22:00:42.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[24927]: 24 2021-03-24T22:00:42.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[24927]: T 2021-03-24T22:00:42.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[24927]: 22 2021-03-24T22:00:42.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[24927]: : 2021-03-24T22:00:42.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[24927]: 00 2021-03-24T22:00:42.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[24927]: : 2021-03-24T22:00:42.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[24927]: 40 2021-03-24T22:00:42.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[24927]: . 2021-03-24T22:00:42.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[24927]: 864500 2021-03-24T22:00:42.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[24927]: +0000 2021-03-24T22:00:42.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[24927]: 2021-03-24T22:00:42.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[24927]: mgr.y 2021-03-24T22:00:42.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[24927]: ( 2021-03-24T22:00:42.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[24927]: mgr 2021-03-24T22:00:42.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[24927]: . 2021-03-24T22:00:42.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[24927]: 14144 2021-03-24T22:00:42.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[24927]: ) 2021-03-24T22:00:42.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[24927]: 507 2021-03-24T22:00:42.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[24927]: : 2021-03-24T22:00:42.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[24927]: cluster 2021-03-24T22:00:42.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[24927]: 2021-03-24T22:00:42.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[24927]: [DBG] 2021-03-24T22:00:42.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[24927]: 2021-03-24T22:00:42.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[24927]: pgmap v868: 169 pgs: 169 active+clean; 416 MiB data, 1.0 GiB used, 446 GiB / 447 GiB avail; 3.4 MiB/s rd, 974 KiB/s wr, 4 op/s 2021-03-24T22:00:42.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[24927]: 2021-03-24T22:00:42.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[24927]: audit 2021-03-24T22:00:42.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[24927]: 2021-03-24T22:00:41. 2021-03-24T22:00:42.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[24927]: 333598+0000 mon.a (mon.0) 1402 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "22"}]': finished 2021-03-24T22:00:42.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[24927]: cluster 2021-03-24T22:00:41.333719+0000 2021-03-24T22:00:42.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[24927]: mon.a (mon.0) 1403 : cluster [DBG] osdmap e471: 8 total, 8 up, 5 in 2021-03-24T22:00:42.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[24927]: 2021-03-24T22:00:42.671 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: cluster 2021-03-24T22:00:42.672 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: 2021-03-24T22:00:42.672 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: 2021 2021-03-24T22:00:42.672 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: - 2021-03-24T22:00:42.672 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: 03 2021-03-24T22:00:42.673 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: - 2021-03-24T22:00:42.673 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: 24 2021-03-24T22:00:42.673 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: T 2021-03-24T22:00:42.673 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: 22 2021-03-24T22:00:42.673 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: : 2021-03-24T22:00:42.674 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: 00 2021-03-24T22:00:42.674 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: : 2021-03-24T22:00:42.674 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: 40 2021-03-24T22:00:42.674 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: . 2021-03-24T22:00:42.675 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: 864500 2021-03-24T22:00:42.675 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: +0000 2021-03-24T22:00:42.675 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: 2021-03-24T22:00:42.675 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: mgr.y 2021-03-24T22:00:42.676 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: ( 2021-03-24T22:00:42.676 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: mgr 2021-03-24T22:00:42.676 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: . 2021-03-24T22:00:42.676 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: 14144 2021-03-24T22:00:42.677 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: ) 2021-03-24T22:00:42.677 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: 507 2021-03-24T22:00:42.677 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: : 2021-03-24T22:00:42.677 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: cluster 2021-03-24T22:00:42.677 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: 2021-03-24T22:00:42.678 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: [DBG] 2021-03-24T22:00:42.678 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: 2021-03-24T22:00:42.678 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: pgmap v868: 169 pgs: 169 active+clean; 416 MiB data, 1.0 GiB used, 446 GiB / 447 GiB avail; 3.4 MiB/s rd, 974 KiB/s wr, 4 op/s 2021-03-24T22:00:42.678 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: 2021-03-24T22:00:42.679 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: audit 2021-03-24T22:00:42.679 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: 2021-03-24T22:00:42.679 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: 2021 2021-03-24T22:00:42.679 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: - 2021-03-24T22:00:42.680 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: 03 2021-03-24T22:00:42.680 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: - 2021-03-24T22:00:42.680 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: 24 2021-03-24T22:00:42.680 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: T 2021-03-24T22:00:42.680 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: 22 2021-03-24T22:00:42.681 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: : 2021-03-24T22:00:42.681 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: 00 2021-03-24T22:00:42.681 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: : 2021-03-24T22:00:42.681 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: 41 2021-03-24T22:00:42.682 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: . 2021-03-24T22:00:42.682 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: 333598 2021-03-24T22:00:42.682 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: +0000 2021-03-24T22:00:42.682 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: 2021-03-24T22:00:42.682 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: mon.a 2021-03-24T22:00:42.683 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: ( 2021-03-24T22:00:42.683 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: mon 2021-03-24T22:00:42.683 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: . 2021-03-24T22:00:42.683 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: 0 2021-03-24T22:00:42.684 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: ) 2021-03-24T22:00:42.684 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: 1402 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "22"}]': finished 2021-03-24T22:00:42.684 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:42 smithi125 conmon[19936]: cluster 2021-03-24T22:00:41.333719+0000 mon.a (mon.0) 1403 : cluster [DBG] osdmap e471: 8 total, 8 up, 5 in 2021-03-24T22:00:42.685 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:42.693 INFO:tasks.rados.rados.0.smithi125.stdout:1767: expect (ObjNum 489 snap 140 seq_num 489) 2021-03-24T22:00:42.734 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:42.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:42.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:00:42.764 INFO:tasks.rados.rados.0.smithi125.stdout:1765: expect (ObjNum 580 snap 176 seq_num 580) 2021-03-24T22:00:42.839 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:42.877 INFO:tasks.rados.rados.0.smithi125.stdout:1762: finishing rollback tid 0 to smithi12556470-47 2021-03-24T22:00:42.878 INFO:tasks.rados.rados.0.smithi125.stdout:1762: finishing rollback tid 1 to smithi12556470-47 2021-03-24T22:00:42.878 INFO:tasks.rados.rados.0.smithi125.stdout:1762: finishing rollback tid 2 to smithi12556470-47 2021-03-24T22:00:42.878 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 47 v 337 (ObjNum 466 snap 131 seq_num 466) dirty exists 2021-03-24T22:00:42.879 INFO:tasks.rados.rados.0.smithi125.stdout:1764: finishing rollback tid 1 to smithi12556470-49 2021-03-24T22:00:42.879 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 49 v 581 (ObjNum 1 snap 32741 seq_num 1116868064) dirty dne 2021-03-24T22:00:42.879 INFO:tasks.rados.rados.0.smithi125.stdout:1763: expect (ObjNum 588 snap 177 seq_num 588) 2021-03-24T22:00:42.901 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:42.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:42.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:00:43.010 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:43.058 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:43.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:43.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:00:43.142 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:00:43.164 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.0 dump_blocked_ops 2021-03-24T22:00:43.165 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:43.180 INFO:tasks.rados.rados.0.smithi125.stdout:1757: done (11 left) 2021-03-24T22:00:43.180 INFO:tasks.rados.rados.0.smithi125.stdout:1759: done (10 left) 2021-03-24T22:00:43.180 INFO:tasks.rados.rados.0.smithi125.stdout:1760: done (9 left) 2021-03-24T22:00:43.181 INFO:tasks.rados.rados.0.smithi125.stdout:1762: done (8 left) 2021-03-24T22:00:43.181 INFO:tasks.rados.rados.0.smithi125.stdout:1763: done (7 left) 2021-03-24T22:00:43.181 INFO:tasks.rados.rados.0.smithi125.stdout:1764: done (6 left) 2021-03-24T22:00:43.181 INFO:tasks.rados.rados.0.smithi125.stdout:1765: done (5 left) 2021-03-24T22:00:43.182 INFO:tasks.rados.rados.0.smithi125.stdout:1767: done (4 left) 2021-03-24T22:00:43.182 INFO:tasks.rados.rados.0.smithi125.stdout:1770: done (3 left) 2021-03-24T22:00:43.182 INFO:tasks.rados.rados.0.smithi125.stdout:1771: write oid 42 current snap is 189 2021-03-24T22:00:43.182 INFO:tasks.rados.rados.0.smithi125.stdout:1771: seq_num 624 ranges {487568=496322,1431705=519550} 2021-03-24T22:00:43.185 INFO:tasks.rados.rados.0.smithi125.stdout:1771: writing smithi12556470-42 from 487568 to 983890 tid 1 2021-03-24T22:00:43.192 INFO:tasks.rados.rados.0.smithi125.stdout:1771: writing smithi12556470-42 from 1431705 to 1951255 tid 2 2021-03-24T22:00:43.192 INFO:tasks.rados.rados.0.smithi125.stdout:1772: snap_create 2021-03-24T22:00:43.192 INFO:tasks.rados.rados.0.smithi125.stdout:1769: finishing write tid 1 to smithi12556470-5 2021-03-24T22:00:43.192 INFO:tasks.rados.rados.0.smithi125.stdout:1769: finishing write tid 2 to smithi12556470-5 2021-03-24T22:00:43.193 INFO:tasks.rados.rados.0.smithi125.stdout:1769: finishing write tid 3 to smithi12556470-5 2021-03-24T22:00:43.193 INFO:tasks.rados.rados.0.smithi125.stdout:1769: finishing write tid 4 to smithi12556470-5 2021-03-24T22:00:43.193 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 5 v 551 (ObjNum 622 snap 189 seq_num 622) dirty exists 2021-03-24T22:00:43.193 INFO:tasks.rados.rados.0.smithi125.stdout:1769: left oid 5 (ObjNum 622 snap 189 seq_num 622) 2021-03-24T22:00:43.194 INFO:tasks.rados.rados.0.smithi125.stdout:1766: expect (ObjNum 462 snap 129 seq_num 462) 2021-03-24T22:00:43.228 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:43.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:43.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:00:43.337 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:43.360 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T22:00:43.388 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:43.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:43.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:00:43.392 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T22:00:43.427 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T22:00:43.455 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T22:00:43.478 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T22:00:43.484 INFO:tasks.rados.rados.0.smithi125.stdout:1768: finishing copy_from to smithi12556470-14 2021-03-24T22:00:43.484 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 14 v 296 (ObjNum 477 snap 136 seq_num 477) dirty exists 2021-03-24T22:00:43.485 INFO:tasks.rados.rados.0.smithi125.stdout:1768: finishing copy_from racing read to smithi12556470-14 2021-03-24T22:00:43.485 INFO:tasks.rados.rados.0.smithi125.stdout:1771: finishing write tid 1 to smithi12556470-42 2021-03-24T22:00:43.486 INFO:tasks.rados.rados.0.smithi125.stdout:1771: finishing write tid 2 to smithi12556470-42 2021-03-24T22:00:43.486 INFO:tasks.rados.rados.0.smithi125.stdout:1771: finishing write tid 3 to smithi12556470-42 2021-03-24T22:00:43.487 INFO:tasks.rados.rados.0.smithi125.stdout:1771: finishing write tid 4 to smithi12556470-42 2021-03-24T22:00:43.487 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 42 v 491 (ObjNum 624 snap 189 seq_num 624) dirty exists 2021-03-24T22:00:43.488 INFO:tasks.rados.rados.0.smithi125.stdout:1771: left oid 42 (ObjNum 624 snap 189 seq_num 624) 2021-03-24T22:00:43.488 INFO:tasks.rados.rados.0.smithi125.stdout:1766: done (4 left) 2021-03-24T22:00:43.488 INFO:tasks.rados.rados.0.smithi125.stdout:1768: done (3 left) 2021-03-24T22:00:43.489 INFO:tasks.rados.rados.0.smithi125.stdout:1769: done (2 left) 2021-03-24T22:00:43.489 INFO:tasks.rados.rados.0.smithi125.stdout:1771: done (1 left) 2021-03-24T22:00:43.490 INFO:tasks.rados.rados.0.smithi125.stdout:1772: done (0 left) 2021-03-24T22:00:43.490 INFO:tasks.rados.rados.0.smithi125.stdout:1773: snap_create 2021-03-24T22:00:43.493 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:43.504 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T22:00:43.538 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T22:00:43.545 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:43.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:43.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:43.563 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T22:00:43.564 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-24T22:00:43.565 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-24T22:00:43.565 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-24T22:00:43.566 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-24T22:00:43.566 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-24T22:00:43.566 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-24T22:00:43.566 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-24T22:00:43.567 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-24T22:00:43.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:43 smithi142 conmon[20125]: cluster 2021-03-24T22:00:42.336055+0000 mon.a 2021-03-24T22:00:43.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:43 smithi142 conmon[20125]: (mon.0) 1404 : cluster [DBG] osdmap e472: 8 total, 8 up, 5 in 2021-03-24T22:00:43.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:43 smithi142 conmon[20125]: audit 2021-03-24T22:00:43.327472+0000 mon.c (mon. 2021-03-24T22:00:43.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:43 smithi142 conmon[20125]: 2) 73 : audit [DBG] from='client.? v1:172.21.15.125:0/2261894576' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:00:43.632 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:43 smithi125 conmon[19936]: cluster 2021-03-24T22:00: 2021-03-24T22:00:43.633 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:43 smithi125 conmon[19936]: 42.336055+0000 mon.a (mon.0) 1404 : cluster [DBG] osdmap e472: 8 total, 8 up, 5 in 2021-03-24T22:00:43.633 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:43 smithi125 conmon[19936]: audit 2021-03-24T22:00:43.327472+0000 mon.c ( 2021-03-24T22:00:43.633 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:43 smithi125 conmon[19936]: mon.2) 73 : audit [DBG] from='client.? v1:172.21.15.125:0/2261894576' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:00:43.634 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:43 smithi125 conmon[24927]: cluster 2021-03-24T22:00:42.336055+0000 2021-03-24T22:00:43.634 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:43 smithi125 conmon[24927]: mon.a (mon.0) 1404 : cluster [DBG] osdmap e472: 8 total, 8 up, 5 in 2021-03-24T22:00:43.634 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:43 smithi125 conmon[24927]: audit 2021-03-24T22:00:43.327472+0000 mon.c (mon.2) 73 : audit [DBG] from='client.? v1:172.21.15.125:0/2261894576' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:00:43.651 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:43.699 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:43.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:43.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:43.714 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:00:43.804 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd unset noscrub 2021-03-24T22:00:43.805 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:43.867 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:43.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:43.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:00:43.972 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:44.028 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:44.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:44.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:00:44.133 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:44.189 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:44.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:44.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:00:44.297 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:44.352 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:44.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:44.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:44.369 INFO:tasks.rados.rados.0.smithi125.stdout:1773: done (0 left) 2021-03-24T22:00:44.369 INFO:tasks.rados.rados.0.smithi125.stdout:1774: snap_create 2021-03-24T22:00:44.459 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:44.473 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:44 smithi142 conmon[20125]: cluster 2021-03-24T22:00: 2021-03-24T22:00:44.473 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:44 smithi142 conmon[20125]: 42.865315+0000 mgr.y (mgr.14144) 508 : cluster [DBG] pgmap v871: 169 pgs: 169 active+clean; 419 MiB data, 1.0 GiB used, 446 GiB / 447 GiB avail; 6.2 MiB/s rd, 1.8 MiB/s wr, 12 op/s 2021-03-24T22:00:44.474 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:44 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:00:44.474 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:44 smithi142 conmon[20125]: 24T22:00:43.356440+0000 mon.a (mon.0) 1405 : cluster [DBG] osdmap e473: 8 total, 8 up, 5 in 2021-03-24T22:00:44.513 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:44.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:44.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:00:44.623 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:44.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:44 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:00:44.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:44 smithi125 conmon[24927]: -03-24T22:00:42 2021-03-24T22:00:44.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:44 smithi125 conmon[24927]: .865315+0000 2021-03-24T22:00:44.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:44 smithi125 conmon[24927]: mgr.y ( 2021-03-24T22:00:44.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:44 smithi125 conmon[24927]: mgr.14144 2021-03-24T22:00:44.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:44 smithi125 conmon[24927]: ) 508 : 2021-03-24T22:00:44.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:44 smithi125 conmon[24927]: cluster [DBG] 2021-03-24T22:00:44.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:44 smithi125 conmon[24927]: pgmap v871: 169 pgs: 169 active+clean; 419 MiB data, 1.0 GiB used, 446 GiB / 447 GiB avail; 6.2 MiB/s rd, 1.8 MiB/s wr, 12 op/s 2021-03-24T22:00:44.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:44 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T22:00:44.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:44 smithi125 conmon[24927]: 22:00:43.356440+0000 mon.a (mon.0) 1405 : cluster [DBG] osdmap e473: 8 total, 8 up, 5 in 2021-03-24T22:00:44.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:44 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:00:44.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:44 smithi125 conmon[19936]: 00:42.865315+0000 mgr.y (mgr.14144) 508 : cluster [DBG] pgmap v871: 169 pgs: 169 active+clean; 419 MiB data, 1.0 GiB used, 446 GiB / 447 GiB avail; 6.2 MiB/s rd, 1.8 MiB/s wr, 12 op/s 2021-03-24T22:00:44.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:44 smithi125 conmon[19936]: cluster 2021-03-24T22:00:44.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:44 smithi125 conmon[19936]: 2021-03-24T22:00:43. 2021-03-24T22:00:44.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:44 smithi125 conmon[19936]: 356440+0000 mon.a (mon.0) 1405 2021-03-24T22:00:44.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:44 smithi125 conmon[19936]: : cluster [DBG] osdmap e473: 8 total, 8 up, 5 in 2021-03-24T22:00:44.680 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:44.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:44.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:00:44.786 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:44.844 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:44.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:44.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:00:44.951 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:45.014 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:45.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:45.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:00:45.121 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:45.175 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:45.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:45.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:00:45.262 INFO:tasks.rados.rados.0.smithi125.stdout:1774: done (0 left) 2021-03-24T22:00:45.262 INFO:tasks.rados.rados.0.smithi125.stdout:1775: write (excl) oid 4 current snap is 192 2021-03-24T22:00:45.262 INFO:tasks.rados.rados.0.smithi125.stdout:1775: seq_num 625 ranges {504375=571571,1490187=663208} 2021-03-24T22:00:45.263 INFO:tasks.rados.rados.0.smithi125.stdout:1775: writing smithi12556470-4 from 504375 to 1075946 tid 1 2021-03-24T22:00:45.270 INFO:tasks.rados.rados.0.smithi125.stdout:1775: writing smithi12556470-4 from 1490187 to 2153395 tid 2 2021-03-24T22:00:45.270 INFO:tasks.rados.rados.0.smithi125.stdout:1776: delete oid 25 current snap is 192 2021-03-24T22:00:45.281 INFO:tasks.rados.rados.0.smithi125.stdout:1776: done (1 left) 2021-03-24T22:00:45.281 INFO:tasks.rados.rados.0.smithi125.stdout:1777: read oid 3 snap -1 2021-03-24T22:00:45.281 INFO:tasks.rados.rados.0.smithi125.stdout:1777: expect (ObjNum 442 snap 119 seq_num 442) 2021-03-24T22:00:45.281 INFO:tasks.rados.rados.0.smithi125.stdout:1778: snap_create 2021-03-24T22:00:45.282 INFO:tasks.rados.rados.0.smithi125.stdout:1775: finishing write tid 1 to smithi12556470-4 2021-03-24T22:00:45.282 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:45.284 INFO:tasks.rados.rados.0.smithi125.stdout:1775: finishing write tid 2 to smithi12556470-4 2021-03-24T22:00:45.284 INFO:tasks.rados.rados.0.smithi125.stdout:1775: finishing write tid 3 to smithi12556470-4 2021-03-24T22:00:45.284 INFO:tasks.rados.rados.0.smithi125.stdout:1775: finishing write tid 4 to smithi12556470-4 2021-03-24T22:00:45.285 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 4 v 397 (ObjNum 625 snap 192 seq_num 625) dirty exists 2021-03-24T22:00:45.285 INFO:tasks.rados.rados.0.smithi125.stdout:1775: left oid 4 (ObjNum 625 snap 192 seq_num 625) 2021-03-24T22:00:45.290 INFO:tasks.rados.rados.0.smithi125.stdout:1777: expect (ObjNum 442 snap 119 seq_num 442) 2021-03-24T22:00:45.332 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:45.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:45.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:00:45.437 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:45.448 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:45 smithi142 conmon[20125]: cluster 2021-03-24T22:00:44.365036+0000 mon.a (mon.0) 1406 : cluster 2021-03-24T22:00:45.449 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:45 smithi142 conmon[20125]: [DBG] osdmap e474: 8 total, 8 up, 5 in 2021-03-24T22:00:45.449 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:45 smithi142 conmon[20125]: cluster 2021-03-24T22:00:45.249339+0000 mon.a (mon.0 2021-03-24T22:00:45.449 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:45 smithi142 conmon[20125]: ) 1407 : cluster [DBG] osdmap e475: 8 total, 8 up, 5 in 2021-03-24T22:00:45.490 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:45 smithi125 conmon[19936]: cluster 2021-03-24T22:00:45.491 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:45 smithi125 conmon[19936]: 2021-03-24T22:00:44.365036+0000 mon.a (mon.0) 1406 : cluster [DBG] osdmap e474: 8 total, 8 up, 5 in 2021-03-24T22:00:45.491 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:45 smithi125 conmon[19936]: cluster 2021-03-24T22:00: 2021-03-24T22:00:45.491 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:45 smithi125 conmon[19936]: 45.249339+0000 mon.a (mon.0) 1407 : cluster [DBG] osdmap e475: 8 total, 8 up, 5 in 2021-03-24T22:00:45.492 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:45 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T22:00:45.492 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:45 smithi125 conmon[24927]: 22:00:44.365036+0000 mon.a (mon.0) 1406 : cluster [DBG] 2021-03-24T22:00:45.492 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:45 smithi125 conmon[24927]: osdmap e474: 8 total, 8 up, 5 in 2021-03-24T22:00:45.493 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:45 smithi125 conmon[24927]: cluster 2021-03-24T22:00:45.493 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:45 smithi125 conmon[24927]: 2021- 2021-03-24T22:00:45.493 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:45 smithi125 conmon[24927]: 03-24T22:00: 2021-03-24T22:00:45.493 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:45 smithi125 conmon[24927]: 45.249339+0000 mon.a ( 2021-03-24T22:00:45.493 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:45 smithi125 conmon[24927]: mon.0) 1407 : cluster 2021-03-24T22:00:45.494 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:45 smithi125 conmon[24927]: [DBG] osdmap e475: 8 total, 8 up, 5 in 2021-03-24T22:00:45.498 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:45.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:45.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:00:45.576 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:00:45.606 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:45.660 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:45.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:45.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:00:45.768 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:45.824 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:45.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:45.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:00:45.931 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:45.989 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:45.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:45.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:00:46.096 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:46.154 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:46.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:46.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:00:46.260 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:46.265 INFO:tasks.rados.rados.0.smithi125.stdout:1775: done (2 left) 2021-03-24T22:00:46.265 INFO:tasks.rados.rados.0.smithi125.stdout:1777: done (1 left) 2021-03-24T22:00:46.265 INFO:tasks.rados.rados.0.smithi125.stdout:1778: done (0 left) 2021-03-24T22:00:46.266 INFO:tasks.rados.rados.0.smithi125.stdout:1779: rollback oid 38 current snap is 193 2021-03-24T22:00:46.266 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 38 to 188 2021-03-24T22:00:46.266 INFO:tasks.rados.rados.0.smithi125.stdout:1780: delete oid 36 current snap is 193 2021-03-24T22:00:46.270 INFO:tasks.rados.rados.0.smithi125.stdout:1780: done (1 left) 2021-03-24T22:00:46.270 INFO:tasks.rados.rados.0.smithi125.stdout:1781: delete oid 27 current snap is 193 2021-03-24T22:00:46.274 INFO:tasks.rados.rados.0.smithi125.stdout:1781: done (1 left) 2021-03-24T22:00:46.274 INFO:tasks.rados.rados.0.smithi125.stdout:1782: rollback oid 43 current snap is 193 2021-03-24T22:00:46.275 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 43 to 156 2021-03-24T22:00:46.275 INFO:tasks.rados.rados.0.smithi125.stdout:1783: read oid 27 snap -1 2021-03-24T22:00:46.275 INFO:tasks.rados.rados.0.smithi125.stdout:1783: expect deleted 2021-03-24T22:00:46.275 INFO:tasks.rados.rados.0.smithi125.stdout:1784: snap_create 2021-03-24T22:00:46.276 INFO:tasks.rados.rados.0.smithi125.stdout:1779: finishing rollback tid 0 to smithi12556470-38 2021-03-24T22:00:46.276 INFO:tasks.rados.rados.0.smithi125.stdout:1779: finishing rollback tid 1 to smithi12556470-38 2021-03-24T22:00:46.277 INFO:tasks.rados.rados.0.smithi125.stdout:1779: finishing rollback tid 2 to smithi12556470-38 2021-03-24T22:00:46.277 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 38 v 90 (ObjNum 489 snap 140 seq_num 489) dirty exists 2021-03-24T22:00:46.279 INFO:tasks.rados.rados.0.smithi125.stdout:1782: finishing rollback tid 0 to smithi12556470-43 2021-03-24T22:00:46.280 INFO:tasks.rados.rados.0.smithi125.stdout:1782: finishing rollback tid 1 to smithi12556470-43 2021-03-24T22:00:46.280 INFO:tasks.rados.rados.0.smithi125.stdout:1782: finishing rollback tid 2 to smithi12556470-43 2021-03-24T22:00:46.280 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 43 v 817 (ObjNum 399 snap 102 seq_num 399) dirty exists 2021-03-24T22:00:46.313 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:46.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:46.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:00:46.418 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:46.473 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:46.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:46.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:00:46.580 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:46.593 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:46 smithi142 conmon[20125]: cluster 2021-03-24T22:00:44.866060+0000 mgr.y (mgr.14144) 509 : cluster 2021-03-24T22:00:46.594 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:46 smithi142 conmon[20125]: [DBG] pgmap v874: 168 pgs: 1 peering, 167 active+clean; 424 MiB data, 1.0 GiB used, 446 GiB / 447 GiB avail; 23 MiB/s rd, 1.8 MiB/s wr, 26 op/s; 118 B/s, 2 objects/s recovering 2021-03-24T22:00:46.594 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:46 smithi142 conmon[20125]: cluster 2021-03-24T22:00:46.261620+0000 mon.a (mon.0) 1408 : cluster [DBG] osdmap e476: 8 total, 8 up, 5 in 2021-03-24T22:00:46.594 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:46 smithi142 conmon[20125]: 2021-03-24T22:00:46.634 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:46.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:46.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:00:46.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:46 smithi125 conmon[19936]: cluster 2021-03-24T22 2021-03-24T22:00:46.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:46 smithi125 conmon[19936]: :00:44.866060+0000 mgr.y (mgr.14144) 509 : cluster [DBG] pgmap v874: 168 pgs: 1 peering, 167 active+clean; 424 MiB data, 1.0 GiB used, 446 GiB / 447 GiB avail; 23 MiB/s rd, 1.8 MiB/s wr, 26 op/s; 118 B/s, 2 objects/s recovering 2021-03-24T22:00:46.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:46 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:00:46.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:46 smithi125 conmon[19936]: T22:00:46.261620+0000 mon.a (mon.0 2021-03-24T22:00:46.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:46 smithi125 conmon[19936]: ) 1408 : cluster [DBG] osdmap e476: 8 total, 8 up, 5 in 2021-03-24T22:00:46.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:46 smithi125 conmon[24927]: cluster 2021-03-24T22:00: 2021-03-24T22:00:46.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:46 smithi125 conmon[24927]: 44.866060+0000 mgr.y (mgr.14144) 509 : cluster [DBG] pgmap v874: 168 pgs: 1 peering, 167 active+clean; 424 MiB data, 1.0 GiB used, 446 GiB / 447 GiB avail; 23 MiB/s rd, 1.8 MiB/s wr, 26 op/s; 118 B/s, 2 objects/s recovering 2021-03-24T22:00:46.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:46 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T22:00:46.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:46 smithi125 conmon[24927]: 22:00:46.261620+0000 mon.a (mon.0) 1408 : cluster [DBG] osdmap e476: 8 total, 8 up, 5 in 2021-03-24T22:00:46.738 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:46.803 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:46.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:46.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:00:46.910 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:46.970 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:46.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:46.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:00:47.076 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:47.118 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:47.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:47.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:00:47.224 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:47.268 INFO:tasks.rados.rados.0.smithi125.stdout:1779: done (3 left) 2021-03-24T22:00:47.269 INFO:tasks.rados.rados.0.smithi125.stdout:1782: done (2 left) 2021-03-24T22:00:47.269 INFO:tasks.rados.rados.0.smithi125.stdout:1783: done (1 left) 2021-03-24T22:00:47.269 INFO:tasks.rados.rados.0.smithi125.stdout:1784: done (0 left) 2021-03-24T22:00:47.269 INFO:tasks.rados.rados.0.smithi125.stdout:1785: delete oid 43 current snap is 194 2021-03-24T22:00:47.277 INFO:tasks.rados.rados.0.smithi125.stdout:1785: done (0 left) 2021-03-24T22:00:47.277 INFO:tasks.rados.rados.0.smithi125.stdout:1786: read oid 5 snap 166 2021-03-24T22:00:47.278 INFO:tasks.rados.rados.0.smithi125.stdout:1786: expect deleted 2021-03-24T22:00:47.278 INFO:tasks.rados.rados.0.smithi125.stdout:1787: copy_from oid 33 from oid 31 current snap is 194 2021-03-24T22:00:47.278 INFO:tasks.rados.rados.0.smithi125.stdout:1788: rollback oid 36 current snap is 194 2021-03-24T22:00:47.278 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 36 to 182 2021-03-24T22:00:47.279 INFO:tasks.rados.rados.0.smithi125.stdout:1789: write oid 35 current snap is 194 2021-03-24T22:00:47.279 INFO:tasks.rados.rados.0.smithi125.stdout:1789: seq_num 630 ranges {588410=547816,1666244=414204,2484603=103807} 2021-03-24T22:00:47.283 INFO:tasks.rados.rados.0.smithi125.stdout:1789: writing smithi12556470-35 from 588410 to 1136226 tid 1 2021-03-24T22:00:47.285 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:47.289 INFO:tasks.rados.rados.0.smithi125.stdout:1789: writing smithi12556470-35 from 1666244 to 2080448 tid 2 2021-03-24T22:00:47.293 INFO:tasks.rados.rados.0.smithi125.stdout:1789: writing smithi12556470-35 from 2484603 to 2588410 tid 3 2021-03-24T22:00:47.293 INFO:tasks.rados.rados.0.smithi125.stdout:1790: snap_create 2021-03-24T22:00:47.293 INFO:tasks.rados.rados.0.smithi125.stdout:1788: finishing rollback tid 1 to smithi12556470-36 2021-03-24T22:00:47.293 INFO:tasks.rados.rados.0.smithi125.stdout:1788: finishing rollback tid 2 to smithi12556470-36 2021-03-24T22:00:47.294 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 36 v 613 (ObjNum 520 snap 154 seq_num 520) dirty exists 2021-03-24T22:00:47.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:47.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:00:47.297 INFO:tasks.rados.rados.0.smithi125.stdout:1789: finishing write tid 1 to smithi12556470-35 2021-03-24T22:00:47.299 INFO:tasks.rados.rados.0.smithi125.stdout:1789: finishing write tid 2 to smithi12556470-35 2021-03-24T22:00:47.303 INFO:tasks.rados.rados.0.smithi125.stdout:1789: finishing write tid 3 to smithi12556470-35 2021-03-24T22:00:47.303 INFO:tasks.rados.rados.0.smithi125.stdout:1787: finishing copy_from to smithi12556470-33 2021-03-24T22:00:47.303 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 33 v 645 (ObjNum 577 snap 176 seq_num 577) dirty exists 2021-03-24T22:00:47.303 INFO:tasks.rados.rados.0.smithi125.stdout:1789: finishing write tid 4 to smithi12556470-35 2021-03-24T22:00:47.304 INFO:tasks.rados.rados.0.smithi125.stdout:1787: finishing copy_from racing read to smithi12556470-33 2021-03-24T22:00:47.304 INFO:tasks.rados.rados.0.smithi125.stdout:1789: finishing write tid 5 to smithi12556470-35 2021-03-24T22:00:47.305 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 35 v 210 (ObjNum 630 snap 194 seq_num 630) dirty exists 2021-03-24T22:00:47.305 INFO:tasks.rados.rados.0.smithi125.stdout:1789: left oid 35 (ObjNum 630 snap 194 seq_num 630) 2021-03-24T22:00:47.397 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:47.458 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:47.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:47.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:00:47.564 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:47.600 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-24T22:00:47.622 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:47.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:47.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:00:47.729 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:47.778 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:47.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:47.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:00:47.885 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:47.943 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:47.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:47.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:00:48.050 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:48.103 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:48.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:48.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:00:48.210 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:48.244 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:00:48.266 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:48.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:48.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:00:48.286 INFO:tasks.rados.rados.0.smithi125.stdout:1786: done (4 left) 2021-03-24T22:00:48.287 INFO:tasks.rados.rados.0.smithi125.stdout:1787: done (3 left) 2021-03-24T22:00:48.287 INFO:tasks.rados.rados.0.smithi125.stdout:1788: done (2 left) 2021-03-24T22:00:48.287 INFO:tasks.rados.rados.0.smithi125.stdout:1789: done (1 left) 2021-03-24T22:00:48.287 INFO:tasks.rados.rados.0.smithi125.stdout:1790: done (0 left) 2021-03-24T22:00:48.288 INFO:tasks.rados.rados.0.smithi125.stdout:1791: read oid 17 snap 178 2021-03-24T22:00:48.288 INFO:tasks.rados.rados.0.smithi125.stdout:1791: expect (ObjNum 550 snap 171 seq_num 550) 2021-03-24T22:00:48.288 INFO:tasks.rados.rados.0.smithi125.stdout:1792: rollback oid 39 current snap is 195 2021-03-24T22:00:48.288 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 39 to 158 2021-03-24T22:00:48.289 INFO:tasks.rados.rados.0.smithi125.stdout:1793: delete oid 20 current snap is 195 2021-03-24T22:00:48.298 INFO:tasks.rados.rados.0.smithi125.stdout:1793: done (2 left) 2021-03-24T22:00:48.298 INFO:tasks.rados.rados.0.smithi125.stdout:1794: delete oid 38 current snap is 195 2021-03-24T22:00:48.301 INFO:tasks.rados.rados.0.smithi125.stdout:1791: expect (ObjNum 550 snap 171 seq_num 550) 2021-03-24T22:00:48.302 INFO:teuthology.orchestra.run.smithi125.stderr:set osd.4 primary-affinity to 0 (802) 2021-03-24T22:00:48.315 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:48 smithi125 conmon[24927]: cluster 2021-03-24T22:00:46.866948+0000 mgr.y (mgr.14144) 510 : cluster [DBG] pgmap v877: 168 pgs: 1 peering, 167 active+clean; 430 MiB data, 1.1 GiB used, 446 GiB / 447 GiB avail; 27 MiB/s rd, 1.3 MiB/s wr, 24 op/s 2021-03-24T22:00:48.315 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:48 smithi125 conmon[24927]: cluster 2021-03-24T22:00:47.265227+0000 mon.a (mon.0) 1409 : cluster [DBG] osdmap e477: 8 total, 8 up, 5 in 2021-03-24T22:00:48.316 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:48 smithi125 conmon[24927]: audit 2021-03-24T22:00:48.172102+0000 mon.c (mon.2) 74 : audit [INF] from='client.? v1:172.21.15.125:0/3884172356' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]: dispatch 2021-03-24T22:00:48.316 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:48 smithi125 conmon[24927]: audit 2021-03-24T22:00:48.172538+0000 mon.a (mon.0) 1410 : audit [INF] from='client.35309 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]: dispatch 2021-03-24T22:00:48.316 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:48 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:00:48.316 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:48 smithi125 conmon[19936]: T22:00:46.866948+0000 mgr.y (mgr.14144) 510 : cluster [DBG] pgmap v877: 168 pgs: 1 peering, 167 active+clean; 430 MiB data, 1.1 GiB used, 446 GiB / 447 GiB avail; 27 MiB/s rd, 1.3 MiB/s wr, 24 op/s 2021-03-24T22:00:48.317 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:48 smithi125 conmon[19936]: cluster 2021-03-24T22:00:47.265227+0000 mon.a (mon.0) 1409 : cluster [DBG] osdmap e477: 8 total, 8 up, 5 in 2021-03-24T22:00:48.317 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:48 smithi125 conmon[19936]: audit 2021-03-24T22:00:48.172102+0000 mon.c (mon.2) 74 : audit [INF] from='client.? v1:172.21.15.125:0/3884172356' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]: dispatch 2021-03-24T22:00:48.317 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:48 smithi125 conmon[19936]: audit 2021-03-24T22:00:48.172538+0000 mon.a (mon.0) 1410 : audit [INF] from='client.35309 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]: dispatch 2021-03-24T22:00:48.371 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:48.406 INFO:tasks.rados.rados.0.smithi125.stdout:1792: finishing rollback tid 1 to smithi12556470-39 2021-03-24T22:00:48.407 INFO:tasks.rados.rados.0.smithi125.stdout:1792: finishing rollback tid 2 to smithi12556470-39 2021-03-24T22:00:48.407 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 39 v 647 (ObjNum 524 snap 154 seq_num 524) dirty exists 2021-03-24T22:00:48.407 INFO:tasks.rados.rados.0.smithi125.stdout:1791: done (2 left) 2021-03-24T22:00:48.408 INFO:tasks.rados.rados.0.smithi125.stdout:1792: done (1 left) 2021-03-24T22:00:48.408 INFO:tasks.rados.rados.0.smithi125.stdout:1794: done (0 left) 2021-03-24T22:00:48.408 INFO:tasks.rados.rados.0.smithi125.stdout:1795: rollback oid 8 current snap is 195 2021-03-24T22:00:48.408 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 8 to 191 2021-03-24T22:00:48.408 INFO:tasks.rados.rados.0.smithi125.stdout:1796: snap_remove snap 190 2021-03-24T22:00:48.424 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:48.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:48.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:00:48.532 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:48.545 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:48 smithi142 conmon[20125]: cluster 2021-03-24T22:00:46.866948+0000 2021-03-24T22:00:48.546 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:48 smithi142 conmon[20125]: mgr.y (mgr.14144) 510 : cluster [DBG] pgmap v877: 168 pgs: 1 peering, 167 active+clean; 430 MiB data, 1.1 GiB used, 446 GiB / 447 GiB avail; 27 MiB/s rd, 1.3 MiB/s wr, 24 op/s 2021-03-24T22:00:48.546 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:48 smithi142 conmon[20125]: cluster 2021-03-24T22:00:47. 2021-03-24T22:00:48.551 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:48 smithi142 conmon[20125]: 265227+0000 mon.a (mon.0) 1409 : cluster [DBG] osdmap e477: 8 total, 8 up, 5 in 2021-03-24T22:00:48.552 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:48 smithi142 conmon[20125]: audit 2021-03- 2021-03-24T22:00:48.552 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:48 smithi142 conmon[20125]: 24T22:00:48.172102+0000 mon.c ( 2021-03-24T22:00:48.552 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:48 smithi142 conmon[20125]: mon.2) 74 : audit [INF] from='client.? v1:172.21.15.125:0/3884172356' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]: dispatch 2021-03-24T22:00:48.553 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:48 smithi142 conmon[20125]: audit 2021-03 2021-03-24T22:00:48.553 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:48 smithi142 conmon[20125]: -24T22:00:48.172538+0000 2021-03-24T22:00:48.553 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:48 smithi142 conmon[20125]: mon.a (mon.0) 1410 : audit [INF] 2021-03-24T22:00:48.553 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:48 smithi142 conmon[20125]: from='client.35309 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]: dispatch 2021-03-24T22:00:48.588 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:48.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:48.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:00:48.694 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:48.756 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:48.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:48.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:48.864 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:48.919 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:48.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:48.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:00:49.026 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:49.082 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:49.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:49.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:00:49.189 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:49.252 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:49.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:49.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:00:49.289 INFO:tasks.rados.rados.0.smithi125.stdout:1796: done (1 left) 2021-03-24T22:00:49.289 INFO:tasks.rados.rados.0.smithi125.stdout:1797: snap_create 2021-03-24T22:00:49.314 INFO:tasks.rados.rados.0.smithi125.stdout:1795: finishing rollback tid 0 to smithi12556470-8 2021-03-24T22:00:49.314 INFO:tasks.rados.rados.0.smithi125.stdout:1795: finishing rollback tid 1 to smithi12556470-8 2021-03-24T22:00:49.315 INFO:tasks.rados.rados.0.smithi125.stdout:1795: finishing rollback tid 2 to smithi12556470-8 2021-03-24T22:00:49.315 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 8 v 105 (ObjNum 580 snap 176 seq_num 580) dirty exists 2021-03-24T22:00:49.358 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:49.413 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:49.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:49.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:49.519 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:49.540 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:49 smithi125 conmon[19936]: audit 2021-03-24T22:00:48.274717+0000 2021-03-24T22:00:49.541 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:49 smithi125 conmon[19936]: mon.a (mon.0) 1411 : audit [INF] from='client.35309 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]': finished 2021-03-24T22:00:49.541 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:49 smithi125 conmon[19936]: cluster 2021-03-24T22:00:49.541 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:49 smithi125 conmon[19936]: 2021-03-24T22: 2021-03-24T22:00:49.542 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:49 smithi125 conmon[19936]: 00:48.275177+0000 mon.a (mon.0) 1412 : cluster [DBG] osdmap e478: 8 total, 8 up, 5 in 2021-03-24T22:00:49.542 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:49 smithi125 conmon[24927]: audit 2021-03-24T22:00:48.274717+0000 mon.a 2021-03-24T22:00:49.542 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:49 smithi125 conmon[24927]: (mon.0) 1411 : audit [INF] from='client.35309 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]': finished 2021-03-24T22:00:49.543 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:49 smithi125 conmon[24927]: cluster 2021-03-24T22:00:48.275177+0000 mon.a (mon.0) 2021-03-24T22:00:49.543 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:49 smithi125 conmon[24927]: 1412 : cluster [DBG] osdmap e478: 8 total, 8 up, 5 in 2021-03-24T22:00:49.576 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:49.580 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:49 smithi142 conmon[20125]: audit 2021-03-24T22:00:48.274717 2021-03-24T22:00:49.581 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:49 smithi142 conmon[20125]: +0000 mon.a (mon.0) 1411 : audit [INF] from='client.35309 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]': finished 2021-03-24T22:00:49.581 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:49 smithi142 conmon[20125]: cluster 2021-03-24T22:00:48.275177+0000 mon.a (mon. 2021-03-24T22:00:49.581 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:49 smithi142 conmon[20125]: 0) 1412 : cluster [DBG] osdmap e478: 8 total, 8 up, 5 in 2021-03-24T22:00:49.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:49.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:00:49.629 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-24T22:00:49.629 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T22:00:49.684 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:49.742 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:49.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:49.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:00:49.850 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:49.907 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:49.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:49.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:00:50.014 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:50.066 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:50.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:50.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:50.172 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:50.228 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:50.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:50.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:00:50.262 INFO:tasks.rados.rados.0.smithi125.stdout:1795: done (1 left) 2021-03-24T22:00:50.263 INFO:tasks.rados.rados.0.smithi125.stdout:1797: done (0 left) 2021-03-24T22:00:50.263 INFO:tasks.rados.rados.0.smithi125.stdout:1798: rollback oid 23 current snap is 196 2021-03-24T22:00:50.263 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 23 to 178 2021-03-24T22:00:50.264 INFO:tasks.rados.rados.0.smithi125.stdout:1799: write (excl) oid 3 current snap is 196 2021-03-24T22:00:50.264 INFO:tasks.rados.rados.0.smithi125.stdout:1799: seq_num 633 ranges {638831=489916,1841690=677884} 2021-03-24T22:00:50.269 INFO:tasks.rados.rados.0.smithi125.stdout:1799: writing smithi12556470-3 from 638831 to 1128747 tid 1 2021-03-24T22:00:50.281 INFO:tasks.rados.rados.0.smithi125.stdout:1799: writing smithi12556470-3 from 1841690 to 2519574 tid 2 2021-03-24T22:00:50.281 INFO:tasks.rados.rados.0.smithi125.stdout:1800: rollback oid 45 current snap is 196 2021-03-24T22:00:50.281 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 45 to 186 2021-03-24T22:00:50.282 INFO:tasks.rados.rados.0.smithi125.stdout:1798: finishing rollback tid 0 to smithi12556470-23 2021-03-24T22:00:50.283 INFO:tasks.rados.rados.0.smithi125.stdout:1798: finishing rollback tid 1 to smithi12556470-23 2021-03-24T22:00:50.283 INFO:tasks.rados.rados.0.smithi125.stdout:1798: finishing rollback tid 2 to smithi12556470-23 2021-03-24T22:00:50.283 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 23 v 292 (ObjNum 579 snap 176 seq_num 579) dirty exists 2021-03-24T22:00:50.283 INFO:tasks.rados.rados.0.smithi125.stdout:1798: done (2 left) 2021-03-24T22:00:50.284 INFO:tasks.rados.rados.0.smithi125.stdout:1801: read oid 35 snap -1 2021-03-24T22:00:50.284 INFO:tasks.rados.rados.0.smithi125.stdout:1801: expect (ObjNum 630 snap 194 seq_num 630) 2021-03-24T22:00:50.284 INFO:tasks.rados.rados.0.smithi125.stdout:1802: rollback oid 22 current snap is 196 2021-03-24T22:00:50.285 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 22 to 187 2021-03-24T22:00:50.285 INFO:tasks.rados.rados.0.smithi125.stdout:1803: copy_from oid 19 from oid 42 current snap is 196 2021-03-24T22:00:50.286 INFO:tasks.rados.rados.0.smithi125.stdout:1804: delete oid 33 current snap is 196 2021-03-24T22:00:50.287 INFO:tasks.rados.rados.0.smithi125.stdout:1800: finishing rollback tid 0 to smithi12556470-45 2021-03-24T22:00:50.289 INFO:tasks.rados.rados.0.smithi125.stdout:1800: finishing rollback tid 1 to smithi12556470-45 2021-03-24T22:00:50.302 INFO:tasks.rados.rados.0.smithi125.stdout:1801: expect (ObjNum 630 snap 194 seq_num 630) 2021-03-24T22:00:50.333 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:50.388 INFO:tasks.rados.rados.0.smithi125.stdout:1799: finishing write tid 1 to smithi12556470-3 2021-03-24T22:00:50.388 INFO:tasks.rados.rados.0.smithi125.stdout:1800: finishing rollback tid 2 to smithi12556470-45 2021-03-24T22:00:50.388 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 45 v 553 (ObjNum 610 snap 185 seq_num 610) dirty exists 2021-03-24T22:00:50.389 INFO:tasks.rados.rados.0.smithi125.stdout:1802: finishing rollback tid 0 to smithi12556470-22 2021-03-24T22:00:50.389 INFO:tasks.rados.rados.0.smithi125.stdout:1802: finishing rollback tid 1 to smithi12556470-22 2021-03-24T22:00:50.389 INFO:tasks.rados.rados.0.smithi125.stdout:1802: finishing rollback tid 2 to smithi12556470-22 2021-03-24T22:00:50.389 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 22 v 556 (ObjNum 556 snap 172 seq_num 556) dirty exists 2021-03-24T22:00:50.390 INFO:tasks.rados.rados.0.smithi125.stdout:1799: finishing write tid 2 to smithi12556470-3 2021-03-24T22:00:50.390 INFO:tasks.rados.rados.0.smithi125.stdout:1799: finishing write tid 3 to smithi12556470-3 2021-03-24T22:00:50.390 INFO:tasks.rados.rados.0.smithi125.stdout:1803: finishing copy_from to smithi12556470-19 2021-03-24T22:00:50.391 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 19 v 491 (ObjNum 624 snap 189 seq_num 624) dirty exists 2021-03-24T22:00:50.391 INFO:tasks.rados.rados.0.smithi125.stdout:1799: finishing write tid 4 to smithi12556470-3 2021-03-24T22:00:50.392 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 3 v 302 (ObjNum 633 snap 196 seq_num 633) dirty exists 2021-03-24T22:00:50.392 INFO:tasks.rados.rados.0.smithi125.stdout:1799: left oid 3 (ObjNum 633 snap 196 seq_num 633) 2021-03-24T22:00:50.392 INFO:tasks.rados.rados.0.smithi125.stdout:1803: finishing copy_from racing read to smithi12556470-19 2021-03-24T22:00:50.393 INFO:tasks.rados.rados.0.smithi125.stdout:1799: done (5 left) 2021-03-24T22:00:50.393 INFO:tasks.rados.rados.0.smithi125.stdout:1800: done (4 left) 2021-03-24T22:00:50.394 INFO:tasks.rados.rados.0.smithi125.stdout:1801: done (3 left) 2021-03-24T22:00:50.394 INFO:tasks.rados.rados.0.smithi125.stdout:1802: done (2 left) 2021-03-24T22:00:50.395 INFO:tasks.rados.rados.0.smithi125.stdout:1803: done (1 left) 2021-03-24T22:00:50.395 INFO:tasks.rados.rados.0.smithi125.stdout:1804: done (0 left) 2021-03-24T22:00:50.396 INFO:tasks.rados.rados.0.smithi125.stdout:1805: write (excl) oid 37 current snap is 196 2021-03-24T22:00:50.396 INFO:tasks.rados.rados.0.smithi125.stdout:1805: seq_num 635 ranges {672445=640414,1842301=681553} 2021-03-24T22:00:50.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:50.399 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:50.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:00:50.400 INFO:tasks.rados.rados.0.smithi125.stdout:1805: writing smithi12556470-37 from 672445 to 1312859 tid 1 2021-03-24T22:00:50.409 INFO:tasks.rados.rados.0.smithi125.stdout:1805: writing smithi12556470-37 from 1842301 to 2523854 tid 2 2021-03-24T22:00:50.410 INFO:tasks.rados.rados.0.smithi125.stdout:1806: read oid 50 snap -1 2021-03-24T22:00:50.410 INFO:tasks.rados.rados.0.smithi125.stdout:1806: expect (ObjNum 599 snap 179 seq_num 599) 2021-03-24T22:00:50.411 INFO:tasks.rados.rados.0.smithi125.stdout:1807: read oid 30 snap -1 2021-03-24T22:00:50.411 INFO:tasks.rados.rados.0.smithi125.stdout:1807: expect (ObjNum 601 snap 180 seq_num 601) 2021-03-24T22:00:50.412 INFO:tasks.rados.rados.0.smithi125.stdout:1808: delete oid 42 current snap is 196 2021-03-24T22:00:50.412 INFO:tasks.rados.rados.0.smithi125.stdout:1805: finishing write tid 1 to smithi12556470-37 2021-03-24T22:00:50.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:50 smithi125 conmon[19936]: cluster 2021-03-24T22:00:48.867645+0000 mgr.y (mgr.14144) 511 : cluster [DBG] pgmap v880: 168 pgs: 1 peering, 167 active+clean; 430 MiB data, 1.1 GiB used, 446 GiB / 447 GiB avail; 7.3 MiB/s rd, 280 KiB/s wr, 6 op/s 2021-03-24T22:00:50.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:50 smithi125 conmon[19936]: cluster 2021-03-24T22:00:49.280792+0000 mon.a (mon.0) 1413 : cluster [DBG] osdmap e479: 8 total, 8 up, 5 in 2021-03-24T22:00:50.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:50 smithi125 conmon[19936]: cluster 2021-03-24T22:00:50.254070+0000 mon.a (mon.0) 1414 : cluster [DBG] osdmap e480: 8 total, 8 up, 5 in 2021-03-24T22:00:50.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:50 smithi125 conmon[24927]: cluster 2021-03-24T22:00: 2021-03-24T22:00:50.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:50 smithi125 conmon[24927]: 48.867645+0000 mgr.y (mgr.14144) 511 : cluster [DBG] pgmap v880: 168 pgs: 1 peering, 167 active+clean; 430 MiB data, 1.1 GiB used, 446 GiB / 447 GiB avail; 7.3 MiB/s rd, 280 KiB/s wr, 6 op/s 2021-03-24T22:00:50.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:50 smithi125 conmon[24927]: cluster 2021-03-24T22:00 2021-03-24T22:00:50.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:50 smithi125 conmon[24927]: :49.280792+0000 mon.a (mon.0) 1413 : cluster 2021-03-24T22:00:50.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:50 smithi125 conmon[24927]: [DBG] osdmap e479: 8 total, 8 up, 5 in 2021-03-24T22:00:50.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:50 smithi125 conmon[24927]: cluster 2021-03-24T22:00:50.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:50 smithi125 conmon[24927]: 2021-03-24T22:00 2021-03-24T22:00:50.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:50 smithi125 conmon[24927]: :50.254070+0000 mon.a ( 2021-03-24T22:00:50.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:50 smithi125 conmon[24927]: mon.0) 1414 : cluster [DBG] 2021-03-24T22:00:50.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:50 smithi125 conmon[24927]: osdmap e480: 8 total, 8 up, 5 in 2021-03-24T22:00:50.420 INFO:tasks.rados.rados.0.smithi125.stdout:1805: finishing write tid 2 to smithi12556470-37 2021-03-24T22:00:50.420 INFO:tasks.rados.rados.0.smithi125.stdout:1808: done (3 left) 2021-03-24T22:00:50.421 INFO:tasks.rados.rados.0.smithi125.stdout:1809: snap_remove snap 182 2021-03-24T22:00:50.501 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:50.515 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:50 smithi142 conmon[20125]: cluster 2021-03-24T22:00:48.867645+0000 2021-03-24T22:00:50.515 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:50 smithi142 conmon[20125]: mgr.y (mgr.14144) 511 : cluster [DBG] pgmap v880: 168 pgs: 1 peering, 167 active+clean; 430 MiB data, 1.1 GiB used, 446 GiB / 447 GiB avail; 7.3 MiB/s rd, 280 KiB/s wr, 6 op/s 2021-03-24T22:00:50.516 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:50 smithi142 conmon[20125]: cluster 2021-03-24T22:00:49.280792+0000 mon.a (mon.0) 1413 : cluster [DBG] osdmap e479: 8 total, 8 up, 5 in 2021-03-24T22:00:50.516 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:50 smithi142 conmon[20125]: cluster 2021-03-24T22:00:50.254070+0000 mon.a (mon.0) 1414 : cluster [DBG] osdmap e480: 8 total, 8 up, 5 in 2021-03-24T22:00:50.557 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:50.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:50.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:00:50.668 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:50.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:50.731 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:50.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:50.833 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:50.891 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:50.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:50.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:00:50.998 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:51.053 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:51.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:51.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:00:51.159 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:51.213 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:51.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:51.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:51.301 INFO:tasks.rados.rados.0.smithi125.stdout:1809: done (3 left) 2021-03-24T22:00:51.301 INFO:tasks.rados.rados.0.smithi125.stdout:1810: read oid 32 snap 192 2021-03-24T22:00:51.302 INFO:tasks.rados.rados.0.smithi125.stdout:1810: expect (ObjNum 461 snap 129 seq_num 461) 2021-03-24T22:00:51.302 INFO:tasks.rados.rados.0.smithi125.stdout:1811: snap_remove snap 194 2021-03-24T22:00:51.320 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:51.349 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:51 smithi125 conmon[24927]: cluster 2021-03-24T22:00:51.293639+0000 2021-03-24T22:00:51.349 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:51 smithi125 conmon[24927]: mon.a (mon.0) 1415 : cluster [WRN] Health check failed: Reduced data availability: 9 pgs inactive, 15 pgs peering (PG_AVAILABILITY) 2021-03-24T22:00:51.350 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:51 smithi125 conmon[24927]: cluster 2021-03-24T22 2021-03-24T22:00:51.350 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:51 smithi125 conmon[24927]: :00:51.298401+0000 mon.a (mon.0) 1416 : cluster [DBG] osdmap e481: 8 total, 8 up, 5 in 2021-03-24T22:00:51.350 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:51 smithi125 conmon[19936]: cluster 2021-03-24T22:00:51.293639+0000 mon.a 2021-03-24T22:00:51.350 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:51 smithi125 conmon[19936]: (mon.0) 1415 : cluster [WRN] Health check failed: Reduced data availability: 9 pgs inactive, 15 pgs peering (PG_AVAILABILITY) 2021-03-24T22:00:51.351 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:51 smithi125 conmon[19936]: cluster 2021-03-24T22:00:51.298401+0000 mon.a (mon.0) 1416 2021-03-24T22:00:51.351 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:51 smithi125 conmon[19936]: : cluster [DBG] osdmap e481: 8 total, 8 up, 5 in 2021-03-24T22:00:51.367 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:51.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:51.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:00:51.472 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:51.530 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:51.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:51.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:00:51.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:51 smithi142 conmon[20125]: cluster 2021-03-24T22:00:51.293639 2021-03-24T22:00:51.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:51 smithi142 conmon[20125]: +0000 mon.a (mon.0) 1415 : cluster [WRN] Health check failed: Reduced data availability: 9 pgs inactive, 15 pgs peering (PG_AVAILABILITY) 2021-03-24T22:00:51.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:51 smithi142 conmon[20125]: cluster 2021-03-24T22:00:51.298401 2021-03-24T22:00:51.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:51 smithi142 conmon[20125]: +0000 mon.a (mon.0) 1416 : cluster [DBG] osdmap e481: 8 total, 8 up, 5 in 2021-03-24T22:00:51.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:51 smithi142 conmon[20125]: 2021-03-24T22:00:51.638 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:51.694 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:51.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:51.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:00:51.846 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:51.905 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:51.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:51.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:00:52.011 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:52.064 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.0 dump_historic_ops 2021-03-24T22:00:52.066 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:52.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:52.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:00:52.173 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:52.232 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:52.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:52.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:00:52.307 INFO:tasks.rados.rados.0.smithi125.stdout:1811: done (4 left) 2021-03-24T22:00:52.307 INFO:tasks.rados.rados.0.smithi125.stdout:1812: rollback oid 29 current snap is 196 2021-03-24T22:00:52.307 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 29 to 191 2021-03-24T22:00:52.308 INFO:tasks.rados.rados.0.smithi125.stdout:1813: write oid 20 current snap is 196 2021-03-24T22:00:52.308 INFO:tasks.rados.rados.0.smithi125.stdout:1813: seq_num 637 ranges {706059=790912,1959265=685222} 2021-03-24T22:00:52.316 INFO:tasks.rados.rados.0.smithi125.stdout:1813: writing smithi12556470-20 from 706059 to 1496971 tid 1 2021-03-24T22:00:52.319 INFO:teuthology.orchestra.run.smithi125.stderr:noscrub is unset 2021-03-24T22:00:52.325 INFO:tasks.rados.rados.0.smithi125.stdout:1813: writing smithi12556470-20 from 1959265 to 2644487 tid 2 2021-03-24T22:00:52.325 INFO:tasks.rados.rados.0.smithi125.stdout:1814: delete oid 22 current snap is 196 2021-03-24T22:00:52.326 INFO:tasks.rados.rados.0.smithi125.stdout:1805: finishing write tid 3 to smithi12556470-37 2021-03-24T22:00:52.327 INFO:tasks.rados.rados.0.smithi125.stdout:1805: finishing write tid 4 to smithi12556470-37 2021-03-24T22:00:52.327 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 37 v 488 (ObjNum 635 snap 196 seq_num 635) dirty exists 2021-03-24T22:00:52.327 INFO:tasks.rados.rados.0.smithi125.stdout:1805: left oid 37 (ObjNum 635 snap 196 seq_num 635) 2021-03-24T22:00:52.328 INFO:tasks.rados.rados.0.smithi125.stdout:1806: expect (ObjNum 599 snap 179 seq_num 599) 2021-03-24T22:00:52.338 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:52.389 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:52.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:52.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:00:52.413 INFO:tasks.rados.rados.0.smithi125.stdout:1807: expect (ObjNum 601 snap 180 seq_num 601) 2021-03-24T22:00:52.417 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:00:52.488 INFO:tasks.rados.rados.0.smithi125.stdout:1810: expect (ObjNum 461 snap 129 seq_num 461) 2021-03-24T22:00:52.495 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:52.544 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:52.549 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:52 smithi125 conmon[19936]: cluster 2021-03-24T22:00:50.868835 2021-03-24T22:00:52.550 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:52 smithi125 conmon[19936]: +0000 mgr.y (mgr.14144) 512 : cluster [DBG] pgmap v883: 168 pgs: 26 peering, 142 active+clean; 436 MiB data, 1.1 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 302 KiB/s wr, 4 op/s 2021-03-24T22:00:52.550 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:52 smithi125 conmon[19936]: audit 2021-03-24T22:00:51.501752+0000 mon.a 2021-03-24T22:00:52.551 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:52 smithi125 conmon[19936]: (mon.0) 1417 : audit [INF] from='client.? v1:172.21.15.125:0/4144890086' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T22:00:52.551 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:52 smithi125 conmon[19936]: cluster 2021-03-24T22:00:52.295485 2021-03-24T22:00:52.552 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:52 smithi125 conmon[19936]: +0000 mon.a (mon.0) 1418 : cluster [WRN] 2021-03-24T22:00:52.552 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:52 smithi125 conmon[19936]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:00:52.553 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:52 smithi125 conmon[19936]: audit 2021-03-24T 2021-03-24T22:00:52.553 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:52 smithi125 conmon[19936]: 22:00:52.301338+0000 mon.a (mon. 2021-03-24T22:00:52.554 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:52 smithi125 conmon[19936]: 0) 1419 : audit [INF] from='client.? v1:172.21.15.125:0/4144890086' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T22:00:52.554 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:52 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:00:52.554 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:52 smithi125 conmon[19936]: 03-24T22:00:52.301424+0000 mon.a (mon 2021-03-24T22:00:52.555 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:52 smithi125 conmon[19936]: .0) 1420 : cluster [DBG] osdmap e482: 8 total, 8 up, 5 in 2021-03-24T22:00:52.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:52.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:00:52.557 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:52 smithi125 conmon[24927]: cluster 2021-03-24T22:00:50. 2021-03-24T22:00:52.557 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:52 smithi125 conmon[24927]: 868835+0000 mgr.y (mgr.14144) 512 : cluster [DBG] pgmap v883: 168 pgs: 26 peering, 142 active+clean; 436 MiB data, 1.1 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 302 KiB/s wr, 4 op/s 2021-03-24T22:00:52.558 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:52 smithi125 conmon[24927]: audit 2021-03-24T22:00:51.501752 2021-03-24T22:00:52.558 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:52 smithi125 conmon[24927]: +0000 mon.a (mon.0) 1417 : audit [INF] 2021-03-24T22:00:52.558 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:52 smithi125 conmon[24927]: from='client.? v1:172.21.15.125:0/4144890086' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T22:00:52.559 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:52 smithi125 conmon[24927]: cluster 2021-03-24T22:00: 2021-03-24T22:00:52.559 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:52 smithi125 conmon[24927]: 52.295485+0000 mon.a (mon.0) 1418 : 2021-03-24T22:00:52.560 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:52 smithi125 conmon[24927]: cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:00:52.560 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:52 smithi125 conmon[24927]: audit 2021-03-24 2021-03-24T22:00:52.560 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:52 smithi125 conmon[24927]: T22:00:52.301338+0000 mon.a ( 2021-03-24T22:00:52.561 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:52 smithi125 conmon[24927]: mon.0) 1419 : audit [INF] from='client.? v1:172.21.15.125:0/4144890086' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T22:00:52.561 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:52 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:00:52.562 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:52 smithi125 conmon[24927]: -03-24T22:00:52.301424+0000 2021-03-24T22:00:52.562 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:52 smithi125 conmon[24927]: mon.a (mon.0) 1420 : cluster [DBG] 2021-03-24T22:00:52.563 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:52 smithi125 conmon[24927]: osdmap e482: 8 total, 8 up, 5 in 2021-03-24T22:00:52.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:52 smithi142 conmon[20125]: cluster 2021-03-24T22:00:50.868835+0000 2021-03-24T22:00:52.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:52 smithi142 conmon[20125]: mgr.y (mgr.14144) 512 : cluster [DBG] pgmap v883: 168 pgs: 26 peering, 142 active+clean; 436 MiB data, 1.1 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 302 KiB/s wr, 4 op/s 2021-03-24T22:00:52.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:52 smithi142 conmon[20125]: audit 2021-03-24T22:00:51.501752+0000 2021-03-24T22:00:52.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:52 smithi142 conmon[20125]: mon.a (mon.0) 1417 : audit [INF] from='client.? v1:172.21.15.125:0/4144890086' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T22:00:52.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:52 smithi142 conmon[20125]: cluster 2021-03-24T22:00:52 2021-03-24T22:00:52.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:52 smithi142 conmon[20125]: .295485+0000 mon.a (mon.0) 1418 : cluster 2021-03-24T22:00:52.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:52 smithi142 conmon[20125]: [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:00:52.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:52 smithi142 conmon[20125]: audit 2021-03-24 2021-03-24T22:00:52.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:52 smithi142 conmon[20125]: T22:00:52.301338+0000 mon.a (mon. 2021-03-24T22:00:52.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:52 smithi142 conmon[20125]: 0) 1419 : audit [INF] from='client.? v1:172.21.15.125:0/4144890086' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T22:00:52.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:52 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:00:52.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:52 smithi142 conmon[20125]: -24T22:00:52.301424+0000 mon.a ( 2021-03-24T22:00:52.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:52 smithi142 conmon[20125]: mon.0) 1420 : cluster [DBG] osdmap e482: 8 total, 8 up, 5 in 2021-03-24T22:00:52.657 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:52.701 INFO:tasks.rados.rados.0.smithi125.stdout:1812: finishing rollback tid 0 to smithi12556470-29 2021-03-24T22:00:52.702 INFO:tasks.rados.rados.0.smithi125.stdout:1812: finishing rollback tid 1 to smithi12556470-29 2021-03-24T22:00:52.703 INFO:tasks.rados.rados.0.smithi125.stdout:1812: finishing rollback tid 2 to smithi12556470-29 2021-03-24T22:00:52.703 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 29 v 268 (ObjNum 611 snap 185 seq_num 611) dirty exists 2021-03-24T22:00:52.703 INFO:tasks.rados.rados.0.smithi125.stdout:1813: finishing write tid 1 to smithi12556470-20 2021-03-24T22:00:52.704 INFO:tasks.rados.rados.0.smithi125.stdout:1813: finishing write tid 2 to smithi12556470-20 2021-03-24T22:00:52.704 INFO:tasks.rados.rados.0.smithi125.stdout:1813: finishing write tid 3 to smithi12556470-20 2021-03-24T22:00:52.705 INFO:tasks.rados.rados.0.smithi125.stdout:1813: finishing write tid 4 to smithi12556470-20 2021-03-24T22:00:52.705 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 20 v 457 (ObjNum 637 snap 196 seq_num 637) dirty exists 2021-03-24T22:00:52.705 INFO:tasks.rados.rados.0.smithi125.stdout:1813: left oid 20 (ObjNum 637 snap 196 seq_num 637) 2021-03-24T22:00:52.706 INFO:tasks.rados.rados.0.smithi125.stdout:1805: done (6 left) 2021-03-24T22:00:52.706 INFO:tasks.rados.rados.0.smithi125.stdout:1806: done (5 left) 2021-03-24T22:00:52.706 INFO:tasks.rados.rados.0.smithi125.stdout:1807: done (4 left) 2021-03-24T22:00:52.706 INFO:tasks.rados.rados.0.smithi125.stdout:1810: done (3 left) 2021-03-24T22:00:52.707 INFO:tasks.rados.rados.0.smithi125.stdout:1812: done (2 left) 2021-03-24T22:00:52.707 INFO:tasks.rados.rados.0.smithi125.stdout:1813: done (1 left) 2021-03-24T22:00:52.707 INFO:tasks.rados.rados.0.smithi125.stdout:1814: done (0 left) 2021-03-24T22:00:52.707 INFO:tasks.rados.rados.0.smithi125.stdout:1815: snap_remove snap 167 2021-03-24T22:00:52.716 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:52.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:52.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:00:52.824 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:52.881 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:52.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:52.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:00:52.988 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:53.051 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:53.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:53.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:00:53.158 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:53.223 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:53.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:53.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:00:53.328 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:53.369 INFO:tasks.rados.rados.0.smithi125.stdout:1815: done (0 left) 2021-03-24T22:00:53.369 INFO:tasks.rados.rados.0.smithi125.stdout:1816: write (excl) oid 35 current snap is 196 2021-03-24T22:00:53.370 INFO:tasks.rados.rados.0.smithi125.stdout:1816: seq_num 639 ranges {739673=657763,2076229=663444} 2021-03-24T22:00:53.379 INFO:tasks.rados.rados.0.smithi125.stdout:1816: writing smithi12556470-35 from 739673 to 1397436 tid 1 2021-03-24T22:00:53.389 INFO:tasks.rados.rados.0.smithi125.stdout:1816: writing smithi12556470-35 from 2076229 to 2739673 tid 2 2021-03-24T22:00:53.390 INFO:tasks.rados.rados.0.smithi125.stdout:1816: finishing write tid 1 to smithi12556470-35 2021-03-24T22:00:53.390 INFO:tasks.rados.rados.0.smithi125.stdout:1817: copy_from oid 28 from oid 43 current snap is 196 2021-03-24T22:00:53.391 INFO:tasks.rados.rados.0.smithi125.stdout:1818: write oid 29 current snap is 196 2021-03-24T22:00:53.391 INFO:tasks.rados.rados.0.smithi125.stdout:1818: seq_num 640 ranges {756480=733012,2134711=432549} 2021-03-24T22:00:53.400 INFO:tasks.rados.rados.0.smithi125.stdout:1818: writing smithi12556470-29 from 756480 to 1489492 tid 1 2021-03-24T22:00:53.401 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:53.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:53.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:00:53.407 INFO:tasks.rados.rados.0.smithi125.stdout:1818: writing smithi12556470-29 from 2134711 to 2567260 tid 2 2021-03-24T22:00:53.407 INFO:tasks.rados.rados.0.smithi125.stdout:1819: write oid 2 current snap is 196 2021-03-24T22:00:53.407 INFO:tasks.rados.rados.0.smithi125.stdout:1817: finishing copy_from to smithi12556470-28 2021-03-24T22:00:53.408 INFO:tasks.rados.rados.0.smithi125.stdout:1817: got expected ENOENT (src dne) 2021-03-24T22:00:53.408 INFO:tasks.rados.rados.0.smithi125.stdout:1819: seq_num 641 ranges {773287=408261,1793193=576207} 2021-03-24T22:00:53.413 INFO:tasks.rados.rados.0.smithi125.stdout:1819: writing smithi12556470-2 from 773287 to 1181548 tid 1 2021-03-24T22:00:53.421 INFO:tasks.rados.rados.0.smithi125.stdout:1819: writing smithi12556470-2 from 1793193 to 2369400 tid 2 2021-03-24T22:00:53.422 INFO:tasks.rados.rados.0.smithi125.stdout:1820: copy_from oid 45 from oid 31 current snap is 196 2021-03-24T22:00:53.422 INFO:tasks.rados.rados.0.smithi125.stdout:1817: finishing copy_from racing read to smithi12556470-28 2021-03-24T22:00:53.422 INFO:tasks.rados.rados.0.smithi125.stdout:1816: finishing write tid 2 to smithi12556470-35 2021-03-24T22:00:53.422 INFO:tasks.rados.rados.0.smithi125.stdout:1816: finishing write tid 3 to smithi12556470-35 2021-03-24T22:00:53.423 INFO:tasks.rados.rados.0.smithi125.stdout:1816: finishing write tid 4 to smithi12556470-35 2021-03-24T22:00:53.423 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 35 v 213 (ObjNum 639 snap 196 seq_num 639) dirty exists 2021-03-24T22:00:53.423 INFO:tasks.rados.rados.0.smithi125.stdout:1816: left oid 35 (ObjNum 639 snap 196 seq_num 639) 2021-03-24T22:00:53.423 INFO:tasks.rados.rados.0.smithi125.stdout:1818: finishing write tid 1 to smithi12556470-29 2021-03-24T22:00:53.423 INFO:tasks.rados.rados.0.smithi125.stdout:1818: finishing write tid 2 to smithi12556470-29 2021-03-24T22:00:53.424 INFO:tasks.rados.rados.0.smithi125.stdout:1818: finishing write tid 3 to smithi12556470-29 2021-03-24T22:00:53.424 INFO:tasks.rados.rados.0.smithi125.stdout:1818: finishing write tid 4 to smithi12556470-29 2021-03-24T22:00:53.424 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 29 v 273 (ObjNum 640 snap 196 seq_num 640) dirty exists 2021-03-24T22:00:53.424 INFO:tasks.rados.rados.0.smithi125.stdout:1818: left oid 29 (ObjNum 640 snap 196 seq_num 640) 2021-03-24T22:00:53.424 INFO:tasks.rados.rados.0.smithi125.stdout:1819: finishing write tid 1 to smithi12556470-2 2021-03-24T22:00:53.425 INFO:tasks.rados.rados.0.smithi125.stdout:1816: done (4 left) 2021-03-24T22:00:53.425 INFO:tasks.rados.rados.0.smithi125.stdout:1817: done (3 left) 2021-03-24T22:00:53.425 INFO:tasks.rados.rados.0.smithi125.stdout:1818: done (2 left) 2021-03-24T22:00:53.425 INFO:tasks.rados.rados.0.smithi125.stdout:1821: write oid 10 current snap is 196 2021-03-24T22:00:53.426 INFO:tasks.rados.rados.0.smithi125.stdout:1821: seq_num 642 ranges {790094=483510,1851675=436218} 2021-03-24T22:00:53.428 INFO:tasks.rados.rados.0.smithi125.stdout:1821: writing smithi12556470-10 from 790094 to 1273604 tid 1 2021-03-24T22:00:53.434 INFO:tasks.rados.rados.0.smithi125.stdout:1821: writing smithi12556470-10 from 1851675 to 2287893 tid 2 2021-03-24T22:00:53.435 INFO:tasks.rados.rados.0.smithi125.stdout:1822: delete oid 50 current snap is 196 2021-03-24T22:00:53.437 INFO:tasks.rados.rados.0.smithi125.stdout:1819: finishing write tid 2 to smithi12556470-2 2021-03-24T22:00:53.437 INFO:tasks.rados.rados.0.smithi125.stdout:1819: finishing write tid 3 to smithi12556470-2 2021-03-24T22:00:53.438 INFO:tasks.rados.rados.0.smithi125.stdout:1819: finishing write tid 4 to smithi12556470-2 2021-03-24T22:00:53.438 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 2 v 625 (ObjNum 641 snap 196 seq_num 641) dirty exists 2021-03-24T22:00:53.438 INFO:tasks.rados.rados.0.smithi125.stdout:1819: left oid 2 (ObjNum 641 snap 196 seq_num 641) 2021-03-24T22:00:53.438 INFO:tasks.rados.rados.0.smithi125.stdout:1821: finishing write tid 1 to smithi12556470-10 2021-03-24T22:00:53.440 INFO:tasks.rados.rados.0.smithi125.stdout:1819: done (3 left) 2021-03-24T22:00:53.440 INFO:tasks.rados.rados.0.smithi125.stdout:1822: done (2 left) 2021-03-24T22:00:53.440 INFO:tasks.rados.rados.0.smithi125.stdout:1823: delete oid 32 current snap is 196 2021-03-24T22:00:53.442 INFO:tasks.rados.rados.0.smithi125.stdout:1821: finishing write tid 2 to smithi12556470-10 2021-03-24T22:00:53.442 INFO:tasks.rados.rados.0.smithi125.stdout:1821: finishing write tid 3 to smithi12556470-10 2021-03-24T22:00:53.442 INFO:tasks.rados.rados.0.smithi125.stdout:1821: finishing write tid 4 to smithi12556470-10 2021-03-24T22:00:53.442 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 10 v 651 (ObjNum 642 snap 196 seq_num 642) dirty exists 2021-03-24T22:00:53.443 INFO:tasks.rados.rados.0.smithi125.stdout:1821: left oid 10 (ObjNum 642 snap 196 seq_num 642) 2021-03-24T22:00:53.444 INFO:tasks.rados.rados.0.smithi125.stdout:1821: done (2 left) 2021-03-24T22:00:53.444 INFO:tasks.rados.rados.0.smithi125.stdout:1823: done (1 left) 2021-03-24T22:00:53.444 INFO:tasks.rados.rados.0.smithi125.stdout:1824: rollback oid 2 current snap is 196 2021-03-24T22:00:53.445 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 2 to 189 2021-03-24T22:00:53.445 INFO:tasks.rados.rados.0.smithi125.stdout:1825: read oid 44 snap -1 2021-03-24T22:00:53.445 INFO:tasks.rados.rados.0.smithi125.stdout:1825: expect deleted 2021-03-24T22:00:53.446 INFO:tasks.rados.rados.0.smithi125.stdout:1826: snap_create 2021-03-24T22:00:53.447 INFO:tasks.rados.rados.0.smithi125.stdout:1820: finishing copy_from to smithi12556470-45 2021-03-24T22:00:53.448 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 45 v 645 (ObjNum 577 snap 176 seq_num 577) dirty exists 2021-03-24T22:00:53.448 INFO:tasks.rados.rados.0.smithi125.stdout:1820: finishing copy_from racing read to smithi12556470-45 2021-03-24T22:00:53.450 INFO:tasks.rados.rados.0.smithi125.stdout:1824: finishing rollback tid 0 to smithi12556470-2 2021-03-24T22:00:53.451 INFO:tasks.rados.rados.0.smithi125.stdout:1824: finishing rollback tid 1 to smithi12556470-2 2021-03-24T22:00:53.451 INFO:tasks.rados.rados.0.smithi125.stdout:1824: finishing rollback tid 2 to smithi12556470-2 2021-03-24T22:00:53.452 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 2 v 628 (ObjNum 606 snap 184 seq_num 606) dirty exists 2021-03-24T22:00:53.507 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:53.564 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:53.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:53.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:00:53.658 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T22:00:53.670 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:53.677 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T22:00:53.695 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T22:00:53.716 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T22:00:53.723 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:53.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:53.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:00:53.738 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T22:00:53.761 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T22:00:53.783 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T22:00:53.812 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T22:00:53.814 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-24T22:00:53.814 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-24T22:00:53.815 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-24T22:00:53.815 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-24T22:00:53.815 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-24T22:00:53.815 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-24T22:00:53.816 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-24T22:00:53.816 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-24T22:00:53.831 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:53.885 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:53.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:53.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:00:53.991 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:54.046 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:54.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:54.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:00:54.153 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:54.209 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:54.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:54.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:54.315 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:54.374 INFO:tasks.rados.rados.0.smithi125.stdout:1820: done (3 left) 2021-03-24T22:00:54.374 INFO:tasks.rados.rados.0.smithi125.stdout:1824: done (2 left) 2021-03-24T22:00:54.375 INFO:tasks.rados.rados.0.smithi125.stdout:1825: done (1 left) 2021-03-24T22:00:54.375 INFO:tasks.rados.rados.0.smithi125.stdout:1826: done (0 left) 2021-03-24T22:00:54.375 INFO:tasks.rados.rados.0.smithi125.stdout:1827: read oid 49 snap 189 2021-03-24T22:00:54.376 INFO:tasks.rados.rados.0.smithi125.stdout:1827: expect deleted 2021-03-24T22:00:54.376 INFO:tasks.rados.rados.0.smithi125.stdout:1828: delete oid 6 current snap is 197 2021-03-24T22:00:54.382 INFO:tasks.rados.rados.0.smithi125.stdout:1827: done (1 left) 2021-03-24T22:00:54.382 INFO:tasks.rados.rados.0.smithi125.stdout:1828: done (0 left) 2021-03-24T22:00:54.382 INFO:tasks.rados.rados.0.smithi125.stdout:1829: delete oid 24 current snap is 197 2021-03-24T22:00:54.386 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:54.387 INFO:tasks.rados.rados.0.smithi125.stdout:1829: done (0 left) 2021-03-24T22:00:54.387 INFO:tasks.rados.rados.0.smithi125.stdout:1830: rollback oid 5 current snap is 197 2021-03-24T22:00:54.387 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 5 to 192 2021-03-24T22:00:54.388 INFO:tasks.rados.rados.0.smithi125.stdout:1831: write oid 23 current snap is 197 2021-03-24T22:00:54.388 INFO:tasks.rados.rados.0.smithi125.stdout:1831: seq_num 647 ranges {474129=576108,1627732=587214,2718088=156041} 2021-03-24T22:00:54.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:54.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:00:54.393 INFO:tasks.rados.rados.0.smithi125.stdout:1831: writing smithi12556470-23 from 474129 to 1050237 tid 1 2021-03-24T22:00:54.400 INFO:tasks.rados.rados.0.smithi125.stdout:1831: writing smithi12556470-23 from 1627732 to 2214946 tid 2 2021-03-24T22:00:54.404 INFO:tasks.rados.rados.0.smithi125.stdout:1831: writing smithi12556470-23 from 2718088 to 2874129 tid 3 2021-03-24T22:00:54.405 INFO:tasks.rados.rados.0.smithi125.stdout:1832: read oid 45 snap -1 2021-03-24T22:00:54.405 INFO:tasks.rados.rados.0.smithi125.stdout:1832: expect (ObjNum 577 snap 176 seq_num 577) 2021-03-24T22:00:54.405 INFO:tasks.rados.rados.0.smithi125.stdout:1833: write oid 3 current snap is 197 2021-03-24T22:00:54.405 INFO:tasks.rados.rados.0.smithi125.stdout:1830: finishing rollback tid 0 to smithi12556470-5 2021-03-24T22:00:54.406 INFO:tasks.rados.rados.0.smithi125.stdout:1833: seq_num 648 ranges {490936=651357,1686214=447225,2861864=29072} 2021-03-24T22:00:54.411 INFO:tasks.rados.rados.0.smithi125.stdout:1833: writing smithi12556470-3 from 490936 to 1142293 tid 1 2021-03-24T22:00:54.417 INFO:tasks.rados.rados.0.smithi125.stdout:1833: writing smithi12556470-3 from 1686214 to 2133439 tid 2 2021-03-24T22:00:54.422 INFO:tasks.rados.rados.0.smithi125.stdout:1833: writing smithi12556470-3 from 2861864 to 2890936 tid 3 2021-03-24T22:00:54.422 INFO:tasks.rados.rados.0.smithi125.stdout:1834: delete oid 24 current snap is 197 2021-03-24T22:00:54.422 INFO:tasks.rados.rados.0.smithi125.stdout:1830: finishing rollback tid 1 to smithi12556470-5 2021-03-24T22:00:54.423 INFO:tasks.rados.rados.0.smithi125.stdout:1830: finishing rollback tid 2 to smithi12556470-5 2021-03-24T22:00:54.423 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 5 v 554 (ObjNum 622 snap 189 seq_num 622) dirty exists 2021-03-24T22:00:54.423 INFO:tasks.rados.rados.0.smithi125.stdout:1831: finishing write tid 1 to smithi12556470-23 2021-03-24T22:00:54.423 INFO:tasks.rados.rados.0.smithi125.stdout:1831: finishing write tid 2 to smithi12556470-23 2021-03-24T22:00:54.423 INFO:tasks.rados.rados.0.smithi125.stdout:1831: finishing write tid 3 to smithi12556470-23 2021-03-24T22:00:54.424 INFO:tasks.rados.rados.0.smithi125.stdout:1831: finishing write tid 4 to smithi12556470-23 2021-03-24T22:00:54.425 INFO:tasks.rados.rados.0.smithi125.stdout:1831: finishing write tid 5 to smithi12556470-23 2021-03-24T22:00:54.425 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 23 v 303 (ObjNum 647 snap 197 seq_num 647) dirty exists 2021-03-24T22:00:54.425 INFO:tasks.rados.rados.0.smithi125.stdout:1831: left oid 23 (ObjNum 647 snap 197 seq_num 647) 2021-03-24T22:00:54.425 INFO:tasks.rados.rados.0.smithi125.stdout:1832: expect (ObjNum 577 snap 176 seq_num 577) 2021-03-24T22:00:54.491 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:54.549 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:54.550 INFO:tasks.rados.rados.0.smithi125.stdout:1833: finishing write tid 1 to smithi12556470-3 2021-03-24T22:00:54.550 INFO:tasks.rados.rados.0.smithi125.stdout:1833: finishing write tid 2 to smithi12556470-3 2021-03-24T22:00:54.550 INFO:tasks.rados.rados.0.smithi125.stdout:1833: finishing write tid 3 to smithi12556470-3 2021-03-24T22:00:54.550 INFO:tasks.rados.rados.0.smithi125.stdout:1833: finishing write tid 4 to smithi12556470-3 2021-03-24T22:00:54.551 INFO:tasks.rados.rados.0.smithi125.stdout:1833: finishing write tid 5 to smithi12556470-3 2021-03-24T22:00:54.551 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 3 v 306 (ObjNum 648 snap 197 seq_num 648) dirty exists 2021-03-24T22:00:54.551 INFO:tasks.rados.rados.0.smithi125.stdout:1833: left oid 3 (ObjNum 648 snap 197 seq_num 648) 2021-03-24T22:00:54.551 INFO:tasks.rados.rados.0.smithi125.stdout:1830: done (4 left) 2021-03-24T22:00:54.552 INFO:tasks.rados.rados.0.smithi125.stdout:1831: done (3 left) 2021-03-24T22:00:54.552 INFO:tasks.rados.rados.0.smithi125.stdout:1832: done (2 left) 2021-03-24T22:00:54.552 INFO:tasks.rados.rados.0.smithi125.stdout:1833: done (1 left) 2021-03-24T22:00:54.552 INFO:tasks.rados.rados.0.smithi125.stdout:1834: done (0 left) 2021-03-24T22:00:54.552 INFO:tasks.rados.rados.0.smithi125.stdout:1835: write oid 49 current snap is 197 2021-03-24T22:00:54.553 INFO:tasks.rados.rados.0.smithi125.stdout:1835: seq_num 650 ranges {524550=401855,1403178=450894,2516710=407840} 2021-03-24T22:00:54.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:54.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:00:54.561 INFO:tasks.rados.rados.0.smithi125.stdout:1835: writing smithi12556470-49 from 524550 to 926405 tid 1 2021-03-24T22:00:54.568 INFO:tasks.rados.rados.0.smithi125.stdout:1835: writing smithi12556470-49 from 1403178 to 1854072 tid 2 2021-03-24T22:00:54.575 INFO:tasks.rados.rados.0.smithi125.stdout:1835: writing smithi12556470-49 from 2516710 to 2924550 tid 3 2021-03-24T22:00:54.575 INFO:tasks.rados.rados.0.smithi125.stdout:1836: delete oid 1 current snap is 197 2021-03-24T22:00:54.575 INFO:tasks.rados.rados.0.smithi125.stdout:1835: finishing write tid 1 to smithi12556470-49 2021-03-24T22:00:54.575 INFO:tasks.rados.rados.0.smithi125.stdout:1835: finishing write tid 2 to smithi12556470-49 2021-03-24T22:00:54.578 INFO:tasks.rados.rados.0.smithi125.stdout:1835: finishing write tid 3 to smithi12556470-49 2021-03-24T22:00:54.579 INFO:tasks.rados.rados.0.smithi125.stdout:1835: finishing write tid 4 to smithi12556470-49 2021-03-24T22:00:54.580 INFO:tasks.rados.rados.0.smithi125.stdout:1835: finishing write tid 5 to smithi12556470-49 2021-03-24T22:00:54.580 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 49 v 612 (ObjNum 650 snap 197 seq_num 650) dirty exists 2021-03-24T22:00:54.580 INFO:tasks.rados.rados.0.smithi125.stdout:1835: left oid 49 (ObjNum 650 snap 197 seq_num 650) 2021-03-24T22:00:54.580 INFO:tasks.rados.rados.0.smithi125.stdout:1835: done (1 left) 2021-03-24T22:00:54.581 INFO:tasks.rados.rados.0.smithi125.stdout:1836: done (0 left) 2021-03-24T22:00:54.581 INFO:tasks.rados.rados.0.smithi125.stdout:1837: snap_create 2021-03-24T22:00:54.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:54 smithi142 conmon[20125]: cluster 2021-03-24T22:00:52.869803+0000 mgr.y (mgr.14144) 513 : cluster 2021-03-24T22:00:54.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:54 smithi142 conmon[20125]: [DBG] pgmap v886: 168 pgs: 1 active+clean+snaptrim, 26 peering, 141 active+clean; 443 MiB data, 1.1 GiB used, 446 GiB / 447 GiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 11 op/s 2021-03-24T22:00:54.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:54 smithi142 conmon[20125]: cluster 2021-03-24T22:00:53.364183+0000 mon.a (mon.0 2021-03-24T22:00:54.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:54 smithi142 conmon[20125]: ) 1421 : cluster [DBG] osdmap e483: 8 total, 8 up, 5 in 2021-03-24T22:00:54.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:54 smithi142 conmon[20125]: audit 2021-03-24T22:00:54.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:54 smithi142 conmon[20125]: 2021-03-24T22:00:53.634428+0000 2021-03-24T22:00:54.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:54 smithi142 conmon[20125]: mon.a (mon.0) 1422 : audit [DBG] from='client.? v1:172.21.15.125:0/693741521' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:00:54.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:54 smithi142 conmon[20125]: 2021-03-24T22:00:54.656 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:54.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:54 smithi125 conmon[24927]: cluster 2021-03-24T22:00:52.869803+0000 mgr.y (mgr. 2021-03-24T22:00:54.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:54 smithi125 conmon[24927]: 14144) 513 : cluster [DBG] pgmap v886: 168 pgs: 1 active+clean+snaptrim, 26 peering, 141 active+clean; 443 MiB data, 1.1 GiB used, 446 GiB / 447 GiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 11 op/s 2021-03-24T22:00:54.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:54 smithi125 conmon[24927]: cluster 2021-03-24T22:00:53.364183+0000 mon.a (mon.0) 1421 : cluster [DBG] osdmap e483: 8 total, 8 up, 5 in 2021-03-24T22:00:54.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:54 smithi125 conmon[24927]: audit 2021-03-24T22:00:53.634428+0000 mon.a (mon.0) 1422 : audit [DBG] from='client.? v1:172.21.15.125:0/693741521' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:00:54.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:54 smithi125 conmon[19936]: cluster 2021-03-24T22:00: 2021-03-24T22:00:54.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:54 smithi125 conmon[19936]: 52.869803+0000 mgr.y (mgr.14144) 513 : cluster [DBG] pgmap v886: 168 pgs: 1 active+clean+snaptrim, 26 peering, 141 active+clean; 443 MiB data, 1.1 GiB used, 446 GiB / 447 GiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 11 op/s 2021-03-24T22:00:54.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:54 smithi125 conmon[19936]: cluster 2021-03-24T22:00:53.364183 2021-03-24T22:00:54.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:54 smithi125 conmon[19936]: +0000 mon.a (mon.0) 1421 : cluster [DBG] osdmap e483: 8 total, 8 up, 5 in 2021-03-24T22:00:54.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:54 smithi125 conmon[19936]: audit 2021-03-24T22:00:54.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:54 smithi125 conmon[19936]: 2021-03-24T22:00:53.634428+0000 2021-03-24T22:00:54.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:54 smithi125 conmon[19936]: mon.a (mon.0) 1422 : audit [DBG] 2021-03-24T22:00:54.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:54 smithi125 conmon[19936]: from='client.? v1:172.21.15.125:0/693741521' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:00:54.712 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:54.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:54.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:54.740 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5] out_osds: [6, 1, 7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2021-03-24T22:00:54.740 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-24T22:00:54.740 INFO:tasks.thrashosds.thrasher:Adding osd 7 2021-03-24T22:00:54.741 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd in 7 2021-03-24T22:00:54.819 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:54.877 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:54.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:54.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:54.984 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:55.039 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:55.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:55.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:55.144 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:55.206 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:55.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:55.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:55.312 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:55.372 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:55.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:55.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:00:55.385 INFO:tasks.rados.rados.0.smithi125.stdout:1837: done (0 left) 2021-03-24T22:00:55.386 INFO:tasks.rados.rados.0.smithi125.stdout:1838: write (excl) oid 5 current snap is 198 2021-03-24T22:00:55.386 INFO:tasks.rados.rados.0.smithi125.stdout:1838: seq_num 652 ranges {558164=552353,1803789=454563,2855203=102961} 2021-03-24T22:00:55.392 INFO:tasks.rados.rados.0.smithi125.stdout:1838: writing smithi12556470-5 from 558164 to 1110517 tid 1 2021-03-24T22:00:55.401 INFO:tasks.rados.rados.0.smithi125.stdout:1838: writing smithi12556470-5 from 1803789 to 2258352 tid 2 2021-03-24T22:00:55.405 INFO:tasks.rados.rados.0.smithi125.stdout:1838: writing smithi12556470-5 from 2855203 to 2958164 tid 3 2021-03-24T22:00:55.405 INFO:tasks.rados.rados.0.smithi125.stdout:1839: read oid 43 snap -1 2021-03-24T22:00:55.405 INFO:tasks.rados.rados.0.smithi125.stdout:1839: expect deleted 2021-03-24T22:00:55.406 INFO:tasks.rados.rados.0.smithi125.stdout:1840: copy_from oid 21 from oid 24 current snap is 198 2021-03-24T22:00:55.406 INFO:tasks.rados.rados.0.smithi125.stdout:1838: finishing write tid 1 to smithi12556470-5 2021-03-24T22:00:55.406 INFO:tasks.rados.rados.0.smithi125.stdout:1838: finishing write tid 2 to smithi12556470-5 2021-03-24T22:00:55.406 INFO:tasks.rados.rados.0.smithi125.stdout:1841: snap_create 2021-03-24T22:00:55.408 INFO:tasks.rados.rados.0.smithi125.stdout:1840: finishing copy_from to smithi12556470-21 2021-03-24T22:00:55.408 INFO:tasks.rados.rados.0.smithi125.stdout:1840: got expected ENOENT (src dne) 2021-03-24T22:00:55.408 INFO:tasks.rados.rados.0.smithi125.stdout:1840: finishing copy_from racing read to smithi12556470-21 2021-03-24T22:00:55.408 INFO:tasks.rados.rados.0.smithi125.stdout:1838: finishing write tid 3 to smithi12556470-5 2021-03-24T22:00:55.409 INFO:tasks.rados.rados.0.smithi125.stdout:1838: finishing write tid 4 to smithi12556470-5 2021-03-24T22:00:55.409 INFO:tasks.rados.rados.0.smithi125.stdout:1838: finishing write tid 5 to smithi12556470-5 2021-03-24T22:00:55.409 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 5 v 558 (ObjNum 652 snap 198 seq_num 652) dirty exists 2021-03-24T22:00:55.409 INFO:tasks.rados.rados.0.smithi125.stdout:1838: left oid 5 (ObjNum 652 snap 198 seq_num 652) 2021-03-24T22:00:55.476 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-24T22:00:55.481 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:55.537 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:55.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:55.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:00:55.593 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:55 smithi125 conmon[19936]: cluster 2021-03-24T22:00:54.370950+0000 mon.a (mon.0) 1423 : cluster [DBG] osdmap e484: 8 total, 8 up, 5 in 2021-03-24T22:00:55.593 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:55 smithi125 conmon[19936]: cluster 2021-03-24T22:00:55.309322+0000 osd.3 (osd.3) 103 : cluster [DBG] 6.1f scrub ok 2021-03-24T22:00:55.594 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:55 smithi125 conmon[24927]: cluster 2021-03-24T22:00:54.370950+0000 2021-03-24T22:00:55.594 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:55 smithi125 conmon[24927]: mon.a (mon.0) 1423 : cluster [DBG] osdmap e484: 8 total, 8 up, 5 in 2021-03-24T22:00:55.594 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:55 smithi125 conmon[24927]: cluster 2021-03-24T22:00: 2021-03-24T22:00:55.594 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:55 smithi125 conmon[24927]: 55.309322+0000 osd.3 (osd.3) 103 : cluster [DBG] 6.1f scrub ok 2021-03-24T22:00:55.644 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:55.692 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:55.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:55.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:00:55.798 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:55.812 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:00:55.813 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:55 smithi142 conmon[20125]: cluster 2021-03-24T22:00:55.814 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:55 smithi142 conmon[20125]: 2021-03-24T22:00:55.814 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:55 smithi142 conmon[20125]: 2021-03-24T22:00 2021-03-24T22:00:55.814 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:55 smithi142 conmon[20125]: :54.370950+0000 mon.a (mon.0) 1423 : cluster [DBG] osdmap e484: 8 total, 8 up, 5 in 2021-03-24T22:00:55.815 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:55 smithi142 conmon[20125]: cluster 2021-03-24T22: 2021-03-24T22:00:55.815 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:55 smithi142 conmon[20125]: 00:55.309322+0000 osd.3 (osd.3) 103 : cluster 2021-03-24T22:00:55.815 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:55 smithi142 conmon[20125]: [DBG] 6.1f scrub ok 2021-03-24T22:00:55.853 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:55.855 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:00:55.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:55.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:00:55.960 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:56.013 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:56.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:56.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:56.118 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:56.190 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:56.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:56.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:00:56.297 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:56.364 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:56.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:56.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:00:56.389 INFO:tasks.rados.rados.0.smithi125.stdout:1838: done (3 left) 2021-03-24T22:00:56.390 INFO:tasks.rados.rados.0.smithi125.stdout:1839: done (2 left) 2021-03-24T22:00:56.390 INFO:tasks.rados.rados.0.smithi125.stdout:1840: done (1 left) 2021-03-24T22:00:56.391 INFO:tasks.rados.rados.0.smithi125.stdout:1841: done (0 left) 2021-03-24T22:00:56.391 INFO:tasks.rados.rados.0.smithi125.stdout:1842: snap_create 2021-03-24T22:00:56.471 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:56.512 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:56 smithi125 conmon[24927]: cluster 2021-03-24T22:00:54.682643+0000 2021-03-24T22:00:56.512 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:56 smithi125 conmon[24927]: osd.0 (osd.0) 101 : cluster [DBG] 3.1b scrub ok 2021-03-24T22:00:56.513 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:56 smithi125 conmon[24927]: cluster 2021-03-24T22:00:54.870710+0000 mgr.y (mgr.14144) 514 : cluster [DBG] 2021-03-24T22:00:56.513 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:56 smithi125 conmon[24927]: pgmap v889: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 457 MiB data, 1.1 GiB used, 446 GiB / 447 GiB avail; 8.3 MiB/s rd, 4.1 MiB/s wr, 24 op/s 2021-03-24T22:00:56.514 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:56 smithi125 conmon[24927]: cluster 2021-03-24T22:00:55.366468+0000 mon.a (mon.0) 1424 : cluster 2021-03-24T22:00:56.515 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:56 smithi125 conmon[24927]: [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs inactive, 15 pgs peering) 2021-03-24T22:00:56.516 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:56 smithi125 conmon[24927]: cluster 2021-03-24T22 2021-03-24T22:00:56.516 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:56 smithi125 conmon[24927]: :00:55.379975+0000 mon.a (mon.0) 1425 : cluster [DBG] osdmap e485: 8 total, 8 up, 5 in 2021-03-24T22:00:56.517 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:56 smithi125 conmon[19936]: cluster 2021-03-24T22:00:54. 2021-03-24T22:00:56.517 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:56 smithi125 conmon[19936]: 682643+0000 osd.0 (osd.0) 101 : cluster [DBG] 3.1b scrub ok 2021-03-24T22:00:56.518 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:56 smithi125 conmon[19936]: cluster 2021-03-24T22:00:56.518 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:56 smithi125 conmon[19936]: 2021-03-24T22:00:54.870710+0000 mgr.y (mgr 2021-03-24T22:00:56.518 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:56 smithi125 conmon[19936]: .14144) 514 : cluster [DBG] pgmap v889: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 457 MiB data, 1.1 GiB used, 446 GiB / 447 GiB avail; 8.3 MiB/s rd, 4.1 MiB/s wr, 24 op/s 2021-03-24T22:00:56.519 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:56 smithi125 conmon[19936]: cluster 2021-03-24T22:00 2021-03-24T22:00:56.519 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:56 smithi125 conmon[19936]: :55.366468+0000 mon.a (mon.0) 1424 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs inactive, 15 pgs peering) 2021-03-24T22:00:56.520 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:56 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:00:56.520 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:56 smithi125 conmon[19936]: 00:55.379975+0000 mon.a (mon.0) 1425 : cluster [DBG] 2021-03-24T22:00:56.520 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:56 smithi125 conmon[19936]: osdmap e485: 8 total, 8 up, 5 in 2021-03-24T22:00:56.528 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-24T22:00:56.528 INFO:teuthology.orchestra.run.smithi125.stdout: "size": 20, 2021-03-24T22:00:56.528 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 600, 2021-03-24T22:00:56.529 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [ 2021-03-24T22:00:56.529 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.529 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:1235 7.9 7:9b48f357:::smithi12556470-17:head [write 426944~556633,stat] snapc 53=[] ondisk+write+known_if_redirected e191)", 2021-03-24T22:00:56.530 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:56:00.136481+0000", 2021-03-24T22:00:56.530 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 296.38367120599997, 2021-03-24T22:00:56.531 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.081696782999999995, 2021-03-24T22:00:56.531 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:00:56.531 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:00:56.532 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:00:56.532 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:00:56.533 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:00:56.534 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1235 2021-03-24T22:00:56.534 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.535 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:00:56.535 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.539 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:00:56.540 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.136481+0000", 2021-03-24T22:00:56.540 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:00:56.540 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.541 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.541 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:00:56.541 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.136481+0000", 2021-03-24T22:00:56.542 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999967 2021-03-24T22:00:56.542 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.542 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.543 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:00:56.543 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.136477+0000", 2021-03-24T22:00:56.543 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00036797499999999999 2021-03-24T22:00:56.543 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.544 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.544 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:00:56.544 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.136845+0000", 2021-03-24T22:00:56.544 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.069078045000000005 2021-03-24T22:00:56.545 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.545 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.546 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:00:56.546 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.205923+0000", 2021-03-24T22:00:56.546 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.6849999999999996e-06 2021-03-24T22:00:56.547 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.547 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.548 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:00:56.550 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.205933+0000", 2021-03-24T22:00:56.550 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00082229500000000001 2021-03-24T22:00:56.550 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.550 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.551 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.551 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.206755+0000", 2021-03-24T22:00:56.551 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000119758 2021-03-24T22:00:56.551 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.552 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.552 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:00:56.552 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.206875+0000", 2021-03-24T22:00:56.552 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000201941 2021-03-24T22:00:56.552 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.553 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.553 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 4,7", 2021-03-24T22:00:56.553 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.207077+0000", 2021-03-24T22:00:56.553 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0041193269999999999 2021-03-24T22:00:56.554 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.554 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.554 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:00:56.554 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.211196+0000", 2021-03-24T22:00:56.554 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0067061409999999997 2021-03-24T22:00:56.555 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.555 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.555 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:56.555 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.217903+0000", 2021-03-24T22:00:56.555 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00017204899999999999 2021-03-24T22:00:56.556 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.556 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.556 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:56.556 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.218075+0000", 2021-03-24T22:00:56.557 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.8982e-05 2021-03-24T22:00:56.557 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.557 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.557 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:00:56.557 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.218114+0000", 2021-03-24T22:00:56.558 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.3751000000000007e-05 2021-03-24T22:00:56.558 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.558 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.558 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:00:56.558 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:00.218177+0000", 2021-03-24T22:00:56.559 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.081696782999999995 2021-03-24T22:00:56.559 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:56.559 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:56.559 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:56.560 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.560 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.560 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:2585 7.4 7:260d8f69:::smithi12556470-28:head [copy-from ver 286] snapc ba=[ba] ondisk+write+known_if_redirected e293)", 2021-03-24T22:00:56.560 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:57:43.312471+0000", 2021-03-24T22:00:56.560 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 193.2076811, 2021-03-24T22:00:56.561 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.070470675999999996, 2021-03-24T22:00:56.561 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:00:56.561 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:00:56.561 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:00:56.561 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:00:56.562 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:00:56.562 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 2585 2021-03-24T22:00:56.562 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.562 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:00:56.563 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.563 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:00:56.563 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:43.312471+0000", 2021-03-24T22:00:56.563 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:00:56.563 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.564 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.564 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:00:56.564 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:43.312471+0000", 2021-03-24T22:00:56.564 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999967 2021-03-24T22:00:56.565 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.565 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.565 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:00:56.565 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:43.312467+0000", 2021-03-24T22:00:56.565 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.4620000000000001e-05 2021-03-24T22:00:56.566 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.566 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.566 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:00:56.567 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:43.312482+0000", 2021-03-24T22:00:56.567 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9545000000000002e-05 2021-03-24T22:00:56.567 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.567 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.567 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:00:56.568 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:43.312522+0000", 2021-03-24T22:00:56.568 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.5920000000000001e-06 2021-03-24T22:00:56.568 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.568 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.569 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:00:56.569 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:43.312530+0000", 2021-03-24T22:00:56.569 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.4134999999999994e-05 2021-03-24T22:00:56.569 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.569 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.570 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.570 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:43.312604+0000", 2021-03-24T22:00:56.570 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.1519999999999992e-06 2021-03-24T22:00:56.570 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.570 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.571 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op must wait for map", 2021-03-24T22:00:56.571 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:43.312612+0000", 2021-03-24T22:00:56.571 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.041857265999999997 2021-03-24T22:00:56.571 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.572 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.572 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.572 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:43.354470+0000", 2021-03-24T22:00:56.572 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000250654 2021-03-24T22:00:56.572 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.573 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.573 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:00:56.573 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:43.354720+0000", 2021-03-24T22:00:56.573 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.013261103999999999 2021-03-24T22:00:56.573 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.574 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.574 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 4,5", 2021-03-24T22:00:56.574 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:43.367981+0000", 2021-03-24T22:00:56.574 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012214762000000001 2021-03-24T22:00:56.575 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.575 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.575 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:00:56.575 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:43.380196+0000", 2021-03-24T22:00:56.575 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001655414 2021-03-24T22:00:56.576 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.576 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.576 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:56.576 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:43.381852+0000", 2021-03-24T22:00:56.577 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00080595299999999999 2021-03-24T22:00:56.577 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.577 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.577 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:56.577 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:43.382658+0000", 2021-03-24T22:00:56.578 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012293100000000001 2021-03-24T22:00:56.578 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.578 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.578 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:00:56.578 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:43.382781+0000", 2021-03-24T22:00:56.579 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00016078199999999999 2021-03-24T22:00:56.579 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.579 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.579 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:00:56.579 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:43.382941+0000", 2021-03-24T22:00:56.580 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.070470675999999996 2021-03-24T22:00:56.580 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:56.580 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:56.580 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:56.581 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.581 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.581 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:2586 7.4 7:260d8f69:::smithi12556470-28:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e293)", 2021-03-24T22:00:56.581 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:57:43.312548+0000", 2021-03-24T22:00:56.581 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 193.207603315, 2021-03-24T22:00:56.582 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.070590435000000007, 2021-03-24T22:00:56.582 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:00:56.582 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T22:00:56.582 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:00:56.583 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:00:56.583 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:00:56.583 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 2586 2021-03-24T22:00:56.583 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.583 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:00:56.584 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.584 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:00:56.584 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:43.312548+0000", 2021-03-24T22:00:56.584 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:00:56.584 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.585 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.585 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:00:56.585 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:43.312548+0000", 2021-03-24T22:00:56.585 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T22:00:56.585 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.586 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.586 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:00:56.586 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:43.312547+0000", 2021-03-24T22:00:56.586 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.8629999999999999e-06 2021-03-24T22:00:56.587 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.587 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.587 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:00:56.587 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:43.312552+0000", 2021-03-24T22:00:56.587 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4330000000000001e-06 2021-03-24T22:00:56.588 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.588 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.588 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:00:56.588 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:43.312559+0000", 2021-03-24T22:00:56.588 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.4399999999999998e-06 2021-03-24T22:00:56.589 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.589 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.589 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:00:56.589 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:43.312563+0000", 2021-03-24T22:00:56.589 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000115123 2021-03-24T22:00:56.590 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.590 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.590 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.590 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:43.312678+0000", 2021-03-24T22:00:56.591 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.4380000000000003e-06 2021-03-24T22:00:56.591 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.591 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.591 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting_for_map not empty", 2021-03-24T22:00:56.591 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:43.312684+0000", 2021-03-24T22:00:56.592 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.042846381000000003 2021-03-24T22:00:56.592 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.592 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.592 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.592 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:43.355530+0000", 2021-03-24T22:00:56.593 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00015239100000000001 2021-03-24T22:00:56.593 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.593 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.593 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-24T22:00:56.593 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:43.355682+0000", 2021-03-24T22:00:56.594 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.021898781999999999 2021-03-24T22:00:56.594 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.594 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.594 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.595 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:43.377581+0000", 2021-03-24T22:00:56.595 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.4071000000000001e-05 2021-03-24T22:00:56.595 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.595 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.595 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:00:56.596 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:43.377675+0000", 2021-03-24T22:00:56.596 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0053175449999999999 2021-03-24T22:00:56.596 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.596 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.596 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.597 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:43.382993+0000", 2021-03-24T22:00:56.597 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.7497999999999997e-05 2021-03-24T22:00:56.597 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.597 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.597 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:00:56.598 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:43.383050+0000", 2021-03-24T22:00:56.598 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.8628000000000006e-05 2021-03-24T22:00:56.598 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.598 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.599 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:00:56.599 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:43.383139+0000", 2021-03-24T22:00:56.599 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.070590435000000007 2021-03-24T22:00:56.599 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:56.599 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:56.600 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:56.600 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.600 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.600 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:3401 7.d 7:b4364827:::smithi12556470-44:head [write 1603478~535155] snapc fd=[fd,fc,fb,fa,f8,f5,f3,f2,f1,ef,ed,e8,e7,d6] ondisk+write+known_if_redirected e360)", 2021-03-24T22:00:56.600 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:58:50.243761+0000", 2021-03-24T22:00:56.601 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 126.276390992, 2021-03-24T22:00:56.601 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.068660958999999994, 2021-03-24T22:00:56.601 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:00:56.601 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:00:56.601 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:00:56.602 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:00:56.602 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:00:56.602 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3401 2021-03-24T22:00:56.602 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.602 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:00:56.603 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.603 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:00:56.603 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:50.243761+0000", 2021-03-24T22:00:56.603 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:00:56.604 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.604 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.604 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:00:56.604 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:50.243761+0000", 2021-03-24T22:00:56.604 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T22:00:56.605 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.605 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.605 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:00:56.605 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:50.243759+0000", 2021-03-24T22:00:56.605 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00026144500000000002 2021-03-24T22:00:56.606 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.606 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.606 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:00:56.606 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:50.244021+0000", 2021-03-24T22:00:56.606 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.624e-05 2021-03-24T22:00:56.607 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.607 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.607 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:00:56.607 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:50.244037+0000", 2021-03-24T22:00:56.608 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.0430000000000002e-06 2021-03-24T22:00:56.608 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.608 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.608 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:00:56.608 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:50.244044+0000", 2021-03-24T22:00:56.609 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9898999999999999e-05 2021-03-24T22:00:56.609 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.609 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.609 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.609 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:50.244084+0000", 2021-03-24T22:00:56.610 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6100000000000002e-06 2021-03-24T22:00:56.610 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.610 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.610 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting_for_map not empty", 2021-03-24T22:00:56.610 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:50.244090+0000", 2021-03-24T22:00:56.611 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.033898776999999998 2021-03-24T22:00:56.611 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.611 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.611 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.616 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:50.277989+0000", 2021-03-24T22:00:56.617 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000101745 2021-03-24T22:00:56.617 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.617 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.617 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:00:56.617 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:50.278091+0000", 2021-03-24T22:00:56.618 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.015180176 2021-03-24T22:00:56.618 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.618 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.618 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.619 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:50.293271+0000", 2021-03-24T22:00:56.619 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000117201 2021-03-24T22:00:56.619 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.619 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.619 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:00:56.620 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:50.293388+0000", 2021-03-24T22:00:56.620 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00020226 2021-03-24T22:00:56.620 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.620 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.620 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 3,7", 2021-03-24T22:00:56.621 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:50.293590+0000", 2021-03-24T22:00:56.621 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0072132810000000002 2021-03-24T22:00:56.621 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.621 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.622 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:00:56.622 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:50.300804+0000", 2021-03-24T22:00:56.622 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0040655300000000004 2021-03-24T22:00:56.622 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.622 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.623 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:56.623 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:50.304869+0000", 2021-03-24T22:00:56.624 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0072659539999999998 2021-03-24T22:00:56.624 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.624 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.624 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:56.625 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:50.312135+0000", 2021-03-24T22:00:56.625 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012496299999999999 2021-03-24T22:00:56.625 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.625 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.625 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:00:56.626 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:50.312260+0000", 2021-03-24T22:00:56.626 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000161485 2021-03-24T22:00:56.626 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.626 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.627 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:00:56.627 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:50.312422+0000", 2021-03-24T22:00:56.627 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.068660958999999994 2021-03-24T22:00:56.627 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:56.627 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:56.629 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:56 smithi142 conmon[20125]: cluster 2021-03-24T22:00:54.682643+0000 osd.0 (osd.0) 101 : 2021-03-24T22:00:56.629 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:56 smithi142 conmon[20125]: cluster [DBG] 3.1b scrub ok 2021-03-24T22:00:56.629 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:56 smithi142 conmon[20125]: cluster 2021-03-24T22:00:54.870710+0000 mgr.y (mgr.14144) 514 : cluster [DBG] pgmap v889: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 457 MiB data, 1.1 GiB used, 446 GiB / 447 GiB avail; 8.3 MiB/s rd, 4.1 MiB/s wr, 24 op/s 2021-03-24T22:00:56.629 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:56 smithi142 conmon[20125]: cluster 2021-03-24T22:00:55.366468+0000 mon.a (mon.0) 2021-03-24T22:00:56.630 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:56 smithi142 conmon[20125]: 1424 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs inactive, 15 pgs peering) 2021-03-24T22:00:56.630 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:56 smithi142 conmon[20125]: cluster 2021-03-24T22:00:55. 2021-03-24T22:00:56.630 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:56 smithi142 conmon[20125]: 379975+0000 mon.a (mon.0) 1425 : cluster [DBG] 2021-03-24T22:00:56.630 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:56 smithi142 conmon[20125]: osdmap e485: 8 total, 8 up, 5 in 2021-03-24T22:00:56.631 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:56.632 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.632 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.632 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:3402 7.d 7:b4364827:::smithi12556470-44:head [write 2823983~765846] snapc fd=[fd,fc,fb,fa,f8,f5,f3,f2,f1,ef,ed,e8,e7,d6] ondisk+write+known_if_redirected e360)", 2021-03-24T22:00:56.632 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:58:50.253543+0000", 2021-03-24T22:00:56.633 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 126.266608373, 2021-03-24T22:00:56.633 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.059085843999999998, 2021-03-24T22:00:56.633 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:00:56.633 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:00:56.633 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:00:56.634 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:00:56.634 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:00:56.634 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3402 2021-03-24T22:00:56.634 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.635 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:00:56.635 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.635 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:00:56.635 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:50.253543+0000", 2021-03-24T22:00:56.635 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:00:56.636 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.636 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.636 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:00:56.636 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:50.253543+0000", 2021-03-24T22:00:56.636 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T22:00:56.637 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.637 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.637 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:00:56.637 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:50.253542+0000", 2021-03-24T22:00:56.638 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00040178000000000002 2021-03-24T22:00:56.638 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.638 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.638 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:00:56.638 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:50.253944+0000", 2021-03-24T22:00:56.639 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.6481999999999999e-05 2021-03-24T22:00:56.639 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.639 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.639 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:00:56.639 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:50.253960+0000", 2021-03-24T22:00:56.640 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6290000000000004e-06 2021-03-24T22:00:56.640 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.640 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.640 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:00:56.641 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:50.253967+0000", 2021-03-24T22:00:56.641 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.1727999999999998e-05 2021-03-24T22:00:56.641 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.641 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.641 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.642 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:50.254008+0000", 2021-03-24T22:00:56.642 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.742e-06 2021-03-24T22:00:56.642 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.642 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.643 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting_for_map not empty", 2021-03-24T22:00:56.643 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:50.254015+0000", 2021-03-24T22:00:56.643 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.024102069 2021-03-24T22:00:56.643 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.643 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.644 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.644 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:50.278117+0000", 2021-03-24T22:00:56.644 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.2860000000000004e-05 2021-03-24T22:00:56.644 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.644 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.645 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:00:56.645 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:50.278190+0000", 2021-03-24T22:00:56.645 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.016228838999999998 2021-03-24T22:00:56.645 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.645 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.646 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.646 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:50.294419+0000", 2021-03-24T22:00:56.646 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.5475000000000003e-05 2021-03-24T22:00:56.646 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.647 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.647 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:00:56.647 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:50.294514+0000", 2021-03-24T22:00:56.647 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00016028700000000001 2021-03-24T22:00:56.647 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.648 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.648 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 3,7", 2021-03-24T22:00:56.648 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:50.294675+0000", 2021-03-24T22:00:56.648 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0061370699999999997 2021-03-24T22:00:56.648 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.649 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.649 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:00:56.649 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:50.300812+0000", 2021-03-24T22:00:56.649 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0041288540000000004 2021-03-24T22:00:56.650 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.650 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.650 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:56.650 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:50.304941+0000", 2021-03-24T22:00:56.650 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.007537434 2021-03-24T22:00:56.651 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.651 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.651 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:56.651 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:50.312478+0000", 2021-03-24T22:00:56.651 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000105918 2021-03-24T22:00:56.652 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.652 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.652 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:00:56.652 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:50.312584+0000", 2021-03-24T22:00:56.652 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5209e-05 2021-03-24T22:00:56.653 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.653 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.653 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:00:56.653 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:58:50.312629+0000", 2021-03-24T22:00:56.654 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.059085843999999998 2021-03-24T22:00:56.654 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:56.654 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:56.654 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:56.654 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.655 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.655 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:3778 7.4 7:260d8f69:::smithi12556470-28:head [copy-from ver 383] snapc 123=[123,121,120,11e,11b,11a,119,110,10f,10e,10d,10c,106,105,104,fa,f8,e7] ondisk+write+known_if_redirected e398)", 2021-03-24T22:00:56.655 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:28.285643+0000", 2021-03-24T22:00:56.655 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 88.234509208999995, 2021-03-24T22:00:56.655 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.077376048000000003, 2021-03-24T22:00:56.656 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:00:56.656 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:00:56.656 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:00:56.656 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:00:56.657 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:00:56.657 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3778 2021-03-24T22:00:56.657 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.657 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:00:56.657 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.658 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:00:56.658 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:28.285643+0000", 2021-03-24T22:00:56.658 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:00:56.658 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.658 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.659 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:00:56.659 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:28.285643+0000", 2021-03-24T22:00:56.659 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999967 2021-03-24T22:00:56.659 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.660 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.660 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:00:56.660 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:28.285639+0000", 2021-03-24T22:00:56.660 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.6449000000000001e-05 2021-03-24T22:00:56.660 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.661 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.661 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:00:56.661 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:28.285656+0000", 2021-03-24T22:00:56.661 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0163999999999999e-05 2021-03-24T22:00:56.661 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.662 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.662 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:00:56.662 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:28.285676+0000", 2021-03-24T22:00:56.662 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2092999999999999e-05 2021-03-24T22:00:56.662 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.663 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.663 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:00:56.663 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:28.285688+0000", 2021-03-24T22:00:56.663 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000115079 2021-03-24T22:00:56.664 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.664 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.664 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.664 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:28.285803+0000", 2021-03-24T22:00:56.664 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.4099e-05 2021-03-24T22:00:56.665 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.665 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.665 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op must wait for map", 2021-03-24T22:00:56.665 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:28.285817+0000", 2021-03-24T22:00:56.665 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0057123210000000002 2021-03-24T22:00:56.666 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.666 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.666 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.666 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:28.291529+0000", 2021-03-24T22:00:56.666 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000110865 2021-03-24T22:00:56.667 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.667 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.667 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:00:56.667 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:28.291640+0000", 2021-03-24T22:00:56.667 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.032122167 2021-03-24T22:00:56.668 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.668 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.668 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 4", 2021-03-24T22:00:56.668 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:28.323762+0000", 2021-03-24T22:00:56.669 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011717241 2021-03-24T22:00:56.669 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.669 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.669 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:00:56.669 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:28.335480+0000", 2021-03-24T22:00:56.670 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.027233928000000001 2021-03-24T22:00:56.670 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.670 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.670 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:56.670 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:28.362714+0000", 2021-03-24T22:00:56.671 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000141471 2021-03-24T22:00:56.671 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.671 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.671 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:00:56.672 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:28.362855+0000", 2021-03-24T22:00:56.672 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00016355100000000001 2021-03-24T22:00:56.672 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.672 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.672 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:00:56.673 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:28.363019+0000", 2021-03-24T22:00:56.673 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.077376048000000003 2021-03-24T22:00:56.673 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:56.673 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:56.673 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:56.674 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.674 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.674 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:3779 7.4 7:260d8f69:::smithi12556470-28:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e398)", 2021-03-24T22:00:56.674 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:28.285713+0000", 2021-03-24T22:00:56.675 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 88.234438920000002, 2021-03-24T22:00:56.675 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.077652874999999996, 2021-03-24T22:00:56.675 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:00:56.675 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T22:00:56.675 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:00:56.676 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:00:56.676 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:00:56.676 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3779 2021-03-24T22:00:56.676 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.676 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:00:56.677 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.677 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:00:56.677 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:28.285713+0000", 2021-03-24T22:00:56.677 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:00:56.677 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.678 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.678 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:00:56.678 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:28.285713+0000", 2021-03-24T22:00:56.678 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T22:00:56.678 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.679 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.679 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:00:56.679 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:28.285712+0000", 2021-03-24T22:00:56.679 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.3059999999999997e-06 2021-03-24T22:00:56.680 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.680 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.680 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:00:56.680 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:28.285717+0000", 2021-03-24T22:00:56.680 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.2310000000000002e-06 2021-03-24T22:00:56.681 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.681 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.681 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:00:56.681 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:28.285726+0000", 2021-03-24T22:00:56.681 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.2899999999999997e-06 2021-03-24T22:00:56.682 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.682 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.682 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:00:56.682 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:28.285734+0000", 2021-03-24T22:00:56.682 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000153034 2021-03-24T22:00:56.683 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.683 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.683 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.683 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:28.285887+0000", 2021-03-24T22:00:56.683 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0627e-05 2021-03-24T22:00:56.684 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.684 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.684 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting_for_map not empty", 2021-03-24T22:00:56.684 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:28.285897+0000", 2021-03-24T22:00:56.685 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0059166490000000004 2021-03-24T22:00:56.685 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.685 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.685 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.685 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:28.291814+0000", 2021-03-24T22:00:56.686 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.8195999999999996e-05 2021-03-24T22:00:56.686 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.686 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.686 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-24T22:00:56.686 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:28.291882+0000", 2021-03-24T22:00:56.687 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.040795590999999999 2021-03-24T22:00:56.687 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.687 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.687 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.687 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:28.332678+0000", 2021-03-24T22:00:56.688 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012351699999999999 2021-03-24T22:00:56.688 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.688 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.688 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:00:56.689 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:28.332801+0000", 2021-03-24T22:00:56.689 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.030258533000000001 2021-03-24T22:00:56.689 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.689 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.689 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.690 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:28.363060+0000", 2021-03-24T22:00:56.690 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000113842 2021-03-24T22:00:56.690 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.690 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.690 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:00:56.691 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:28.363174+0000", 2021-03-24T22:00:56.691 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000192089 2021-03-24T22:00:56.691 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.691 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.691 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:00:56.692 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:28.363366+0000", 2021-03-24T22:00:56.692 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.077652874999999996 2021-03-24T22:00:56.692 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:56.692 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:56.692 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:56.693 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.693 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.693 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:3844 7.1b 7:da3cc97e:::smithi12556470-14:head [append 0~0] snapc 125=[125,123,120,11e,11b,11a,119,110,10f,10e,10d,10c,106,105,104,fa,f8,e7] ondisk+write+known_if_redirected e400)", 2021-03-24T22:00:56.693 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:30.243488+0000", 2021-03-24T22:00:56.694 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 86.276663635000006, 2021-03-24T22:00:56.694 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.089353200999999993, 2021-03-24T22:00:56.694 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:00:56.694 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:00:56.694 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:00:56.695 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:00:56.695 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:00:56.695 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3844 2021-03-24T22:00:56.695 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.695 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:00:56.696 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.696 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:00:56.696 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.243488+0000", 2021-03-24T22:00:56.696 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:00:56.697 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.697 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.697 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:00:56.697 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.243488+0000", 2021-03-24T22:00:56.697 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999976 2021-03-24T22:00:56.698 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.698 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.698 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:00:56.698 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.243486+0000", 2021-03-24T22:00:56.698 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2992e-05 2021-03-24T22:00:56.699 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.699 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.699 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:00:56.699 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.243499+0000", 2021-03-24T22:00:56.699 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.1698000000000001e-05 2021-03-24T22:00:56.700 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.700 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.700 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:00:56.700 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.243520+0000", 2021-03-24T22:00:56.700 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.9710000000000003e-06 2021-03-24T22:00:56.701 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.701 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.701 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:00:56.701 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.243525+0000", 2021-03-24T22:00:56.702 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0095115719999999994 2021-03-24T22:00:56.702 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.702 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.702 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.702 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.253037+0000", 2021-03-24T22:00:56.703 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00049090899999999999 2021-03-24T22:00:56.703 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.703 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.703 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:00:56.703 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.253528+0000", 2021-03-24T22:00:56.704 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0071879860000000004 2021-03-24T22:00:56.704 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.704 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.704 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,3", 2021-03-24T22:00:56.704 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.260716+0000", 2021-03-24T22:00:56.705 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0015887189999999999 2021-03-24T22:00:56.705 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.705 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.705 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:56.706 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.262304+0000", 2021-03-24T22:00:56.706 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00039370300000000002 2021-03-24T22:00:56.706 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.706 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.706 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:00:56.707 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.262698+0000", 2021-03-24T22:00:56.707 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.069976014000000003 2021-03-24T22:00:56.707 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.707 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.707 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:56.708 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.332674+0000", 2021-03-24T22:00:56.708 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.5260999999999996e-05 2021-03-24T22:00:56.708 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.708 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.708 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:00:56.709 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.332759+0000", 2021-03-24T22:00:56.709 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.1948999999999998e-05 2021-03-24T22:00:56.709 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.709 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.710 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:00:56.710 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.332841+0000", 2021-03-24T22:00:56.710 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.089353200999999993 2021-03-24T22:00:56.710 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:56.710 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:56.711 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:56.711 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.711 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.711 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:3850 7.1b 7:de4c3db5:::smithi12556470-32:head [copy-from ver 279] snapc 125=[125,123,120,11e,11b,11a,119,110,10f,10e,10d,10c,106,105,104,fa,f8,e7] ondisk+write+known_if_redirected e400)", 2021-03-24T22:00:56.711 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:30.260574+0000", 2021-03-24T22:00:56.712 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 86.259577858, 2021-03-24T22:00:56.712 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.072702953000000001, 2021-03-24T22:00:56.712 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:00:56.712 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:00:56.713 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:00:56.713 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:00:56.713 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:00:56.713 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3850 2021-03-24T22:00:56.713 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.714 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:00:56.714 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.714 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:00:56.714 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.260574+0000", 2021-03-24T22:00:56.714 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:00:56.715 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.715 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.715 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:00:56.715 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.260574+0000", 2021-03-24T22:00:56.715 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999962 2021-03-24T22:00:56.716 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.716 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.716 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:00:56.716 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.260570+0000", 2021-03-24T22:00:56.717 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.6079999999999999e-05 2021-03-24T22:00:56.717 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.717 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.717 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:00:56.717 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.260586+0000", 2021-03-24T22:00:56.718 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.4446e-05 2021-03-24T22:00:56.718 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.718 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.718 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:00:56.718 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.260601+0000", 2021-03-24T22:00:56.719 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.3380000000000005e-06 2021-03-24T22:00:56.719 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.719 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.719 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:00:56.719 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.260609+0000", 2021-03-24T22:00:56.720 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0011181629999999999 2021-03-24T22:00:56.720 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.720 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.720 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.721 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.261727+0000", 2021-03-24T22:00:56.721 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000121348 2021-03-24T22:00:56.721 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.721 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.721 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:00:56.722 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.261849+0000", 2021-03-24T22:00:56.722 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.014128422 2021-03-24T22:00:56.722 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.722 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.722 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,3", 2021-03-24T22:00:56.723 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.275977+0000", 2021-03-24T22:00:56.723 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.045918194000000002 2021-03-24T22:00:56.723 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.723 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.723 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:56.724 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.321895+0000", 2021-03-24T22:00:56.724 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00068287199999999995 2021-03-24T22:00:56.724 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.724 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.725 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:00:56.725 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.322578+0000", 2021-03-24T22:00:56.725 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.010556596 2021-03-24T22:00:56.725 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.725 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.726 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:56.726 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.333135+0000", 2021-03-24T22:00:56.726 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.5276000000000003e-05 2021-03-24T22:00:56.726 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.726 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.727 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:00:56.727 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.333200+0000", 2021-03-24T22:00:56.727 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.6905999999999999e-05 2021-03-24T22:00:56.727 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.728 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.728 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:00:56.728 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.333277+0000", 2021-03-24T22:00:56.728 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.072702953000000001 2021-03-24T22:00:56.728 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:56.729 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:56.729 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:56.729 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.729 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.729 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:3851 7.1b 7:de4c3db5:::smithi12556470-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e400)", 2021-03-24T22:00:56.730 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:30.260621+0000", 2021-03-24T22:00:56.730 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 86.259531183999997, 2021-03-24T22:00:56.730 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.072989943000000002, 2021-03-24T22:00:56.730 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:00:56.730 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T22:00:56.731 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:00:56.731 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:00:56.731 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:00:56.731 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3851 2021-03-24T22:00:56.732 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.732 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:00:56.732 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.732 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:00:56.732 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.260621+0000", 2021-03-24T22:00:56.733 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:00:56.733 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.733 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.733 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:00:56.733 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.260621+0000", 2021-03-24T22:00:56.734 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T22:00:56.734 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.734 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.734 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:00:56.735 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.260620+0000", 2021-03-24T22:00:56.735 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9269999999999998e-06 2021-03-24T22:00:56.735 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.735 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.735 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:00:56.736 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.260623+0000", 2021-03-24T22:00:56.736 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2099999999999998e-06 2021-03-24T22:00:56.736 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.736 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.736 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:00:56.737 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.260630+0000", 2021-03-24T22:00:56.737 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.7609999999999998e-06 2021-03-24T22:00:56.737 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.737 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.738 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:00:56.738 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.260634+0000", 2021-03-24T22:00:56.738 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0013817339999999999 2021-03-24T22:00:56.738 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.738 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.739 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.739 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.262016+0000", 2021-03-24T22:00:56.739 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.7416000000000001e-05 2021-03-24T22:00:56.739 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.739 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.740 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-24T22:00:56.740 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.262074+0000", 2021-03-24T22:00:56.740 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.022024018999999999 2021-03-24T22:00:56.740 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.741 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.741 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.741 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.284098+0000", 2021-03-24T22:00:56.741 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012906699999999999 2021-03-24T22:00:56.741 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.742 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.742 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:00:56.742 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.284227+0000", 2021-03-24T22:00:56.742 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.049096738000000001 2021-03-24T22:00:56.742 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.743 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.743 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.743 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.333323+0000", 2021-03-24T22:00:56.743 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000140708 2021-03-24T22:00:56.743 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.744 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.744 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:00:56.744 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.333464+0000", 2021-03-24T22:00:56.744 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00014638699999999999 2021-03-24T22:00:56.745 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.745 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.745 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:00:56.745 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.333610+0000", 2021-03-24T22:00:56.745 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.072989943000000002 2021-03-24T22:00:56.746 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:56.746 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:56.746 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:56.746 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.746 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.747 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:3945 7.13 7:cb55abc9:::smithi12556470-6:head [write 658973~703921,stat] snapc 136=[136,133,132,130,12d,12c,12b,129,128,127,126,120,11e,11b,11a,119,110,10f,10c,106,105,104,fa,f8,e7] ondisk+write+known_if_redirected e417)", 2021-03-24T22:00:56.747 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:47.262908+0000", 2021-03-24T22:00:56.747 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 69.257243512000002, 2021-03-24T22:00:56.747 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0944426490000001, 2021-03-24T22:00:56.747 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:00:56.748 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:00:56.748 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:00:56.748 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:00:56.748 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:00:56.749 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3945 2021-03-24T22:00:56.749 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.749 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:00:56.749 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.749 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:00:56.750 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.262908+0000", 2021-03-24T22:00:56.750 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:00:56.750 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.750 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.750 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:00:56.751 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.262908+0000", 2021-03-24T22:00:56.751 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999967 2021-03-24T22:00:56.751 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.751 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.752 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:00:56.752 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.262905+0000", 2021-03-24T22:00:56.752 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00039945099999999999 2021-03-24T22:00:56.752 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.752 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.753 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:00:56.753 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.263304+0000", 2021-03-24T22:00:56.753 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2901000000000001e-05 2021-03-24T22:00:56.753 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.753 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.754 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:00:56.754 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.263317+0000", 2021-03-24T22:00:56.754 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6599999999999998e-06 2021-03-24T22:00:56.754 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.755 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.755 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:00:56.755 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.263324+0000", 2021-03-24T22:00:56.755 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.006548707 2021-03-24T22:00:56.755 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.756 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.756 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.756 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.269873+0000", 2021-03-24T22:00:56.756 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0464e-05 2021-03-24T22:00:56.756 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.757 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.757 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:00:56.757 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.269883+0000", 2021-03-24T22:00:56.757 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0772210010000001 2021-03-24T22:00:56.758 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.758 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.758 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.758 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.347104+0000", 2021-03-24T22:00:56.758 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000175124 2021-03-24T22:00:56.759 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.759 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.759 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:00:56.759 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.347279+0000", 2021-03-24T22:00:56.759 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00029503299999999999 2021-03-24T22:00:56.760 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.760 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.760 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 4,7", 2021-03-24T22:00:56.760 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.347574+0000", 2021-03-24T22:00:56.760 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0078104589999999996 2021-03-24T22:00:56.761 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.761 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.761 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:56.761 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.355385+0000", 2021-03-24T22:00:56.762 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.5687999999999996e-05 2021-03-24T22:00:56.762 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.762 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.762 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:56.762 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.355480+0000", 2021-03-24T22:00:56.763 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0011531040000000001 2021-03-24T22:00:56.763 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.763 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.763 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:00:56.763 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.356633+0000", 2021-03-24T22:00:56.764 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00046423500000000002 2021-03-24T22:00:56.764 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.764 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.764 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:00:56.764 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.357098+0000", 2021-03-24T22:00:56.765 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00025322799999999998 2021-03-24T22:00:56.765 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.765 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.765 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:00:56.765 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.357351+0000", 2021-03-24T22:00:56.766 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0944426490000001 2021-03-24T22:00:56.766 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:56.766 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:56.766 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:56.767 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.767 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.767 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:3946 7.13 7:cb55abc9:::smithi12556470-6:head [write 1907912~570853] snapc 136=[136,133,132,130,12d,12c,12b,129,128,127,126,120,11e,11b,11a,119,110,10f,10c,106,105,104,fa,f8,e7] ondisk+write+known_if_redirected e417)", 2021-03-24T22:00:56.767 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:47.278816+0000", 2021-03-24T22:00:56.767 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 69.241335809000006, 2021-03-24T22:00:56.768 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0903059930000001, 2021-03-24T22:00:56.768 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:00:56.768 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:00:56.768 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:00:56.768 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:00:56.769 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:00:56.769 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3946 2021-03-24T22:00:56.769 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.769 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:00:56.769 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.770 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:00:56.770 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.278816+0000", 2021-03-24T22:00:56.770 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:00:56.770 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.771 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.771 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:00:56.771 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.278816+0000", 2021-03-24T22:00:56.771 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999976 2021-03-24T22:00:56.771 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.772 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.772 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:00:56.772 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.278814+0000", 2021-03-24T22:00:56.772 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000351928 2021-03-24T22:00:56.772 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.773 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.773 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:00:56.773 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.279166+0000", 2021-03-24T22:00:56.773 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.26e-05 2021-03-24T22:00:56.774 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.774 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.774 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:00:56.774 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.279178+0000", 2021-03-24T22:00:56.774 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2020000000000004e-06 2021-03-24T22:00:56.775 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.775 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.775 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:00:56.775 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.279186+0000", 2021-03-24T22:00:56.775 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.9264000000000001e-05 2021-03-24T22:00:56.776 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.776 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.776 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.776 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.279236+0000", 2021-03-24T22:00:56.776 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1406000000000001e-05 2021-03-24T22:00:56.777 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.777 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.777 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:00:56.777 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.279247+0000", 2021-03-24T22:00:56.777 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.074288479 2021-03-24T22:00:56.778 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.778 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.778 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.778 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.353536+0000", 2021-03-24T22:00:56.779 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000171313 2021-03-24T22:00:56.779 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.779 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.779 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:00:56.779 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.353707+0000", 2021-03-24T22:00:56.780 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0036947859999999998 2021-03-24T22:00:56.780 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.780 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.780 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.780 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.357402+0000", 2021-03-24T22:00:56.781 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000152483 2021-03-24T22:00:56.781 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.781 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.781 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:00:56.781 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.357554+0000", 2021-03-24T22:00:56.782 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00030391000000000001 2021-03-24T22:00:56.782 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.782 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.782 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 4,7", 2021-03-24T22:00:56.783 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.357858+0000", 2021-03-24T22:00:56.783 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0042159850000000002 2021-03-24T22:00:56.783 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.783 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.783 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:00:56.784 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.362074+0000", 2021-03-24T22:00:56.784 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0025619319999999998 2021-03-24T22:00:56.784 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.784 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.784 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:56.785 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.364636+0000", 2021-03-24T22:00:56.785 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0041920079999999997 2021-03-24T22:00:56.785 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.785 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.785 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:56.786 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.368828+0000", 2021-03-24T22:00:56.786 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.5685000000000004e-05 2021-03-24T22:00:56.786 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.786 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.787 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:00:56.787 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.368914+0000", 2021-03-24T22:00:56.787 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00020822000000000001 2021-03-24T22:00:56.787 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.787 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.788 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:00:56.788 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.369122+0000", 2021-03-24T22:00:56.788 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0903059930000001 2021-03-24T22:00:56.788 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:56.788 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:56.789 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:56.789 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.789 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.789 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:3947 7.13 7:cb55abc9:::smithi12556470-6:head [write 2985346~73627] snapc 136=[136,133,132,130,12d,12c,12b,129,128,127,126,120,11e,11b,11a,119,110,10f,10c,106,105,104,fa,f8,e7] ondisk+write+known_if_redirected e417)", 2021-03-24T22:00:56.789 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:47.279203+0000", 2021-03-24T22:00:56.790 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 69.240948911999993, 2021-03-24T22:00:56.790 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.090169935, 2021-03-24T22:00:56.790 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:00:56.790 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:00:56.791 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:00:56.791 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:00:56.791 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:00:56.791 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3947 2021-03-24T22:00:56.791 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.792 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:00:56.792 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.792 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:00:56.792 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.279203+0000", 2021-03-24T22:00:56.792 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:00:56.793 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.793 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.793 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:00:56.793 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.279203+0000", 2021-03-24T22:00:56.793 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T22:00:56.794 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.794 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.794 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:00:56.794 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.279202+0000", 2021-03-24T22:00:56.795 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.3179999999999997e-05 2021-03-24T22:00:56.795 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.795 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.795 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:00:56.795 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.279235+0000", 2021-03-24T22:00:56.796 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.5270000000000001e-06 2021-03-24T22:00:56.796 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.796 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.796 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:00:56.796 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.279243+0000", 2021-03-24T22:00:56.797 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1930000000000001e-06 2021-03-24T22:00:56.797 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.797 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.797 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:00:56.797 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.279248+0000", 2021-03-24T22:00:56.798 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00013925299999999999 2021-03-24T22:00:56.798 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.798 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.798 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.798 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.279387+0000", 2021-03-24T22:00:56.799 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.636e-06 2021-03-24T22:00:56.799 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.799 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.799 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:00:56.800 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.279396+0000", 2021-03-24T22:00:56.800 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.074351117 2021-03-24T22:00:56.800 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.800 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.800 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.801 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.353747+0000", 2021-03-24T22:00:56.801 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00011241199999999999 2021-03-24T22:00:56.801 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.801 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.801 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:00:56.802 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.353859+0000", 2021-03-24T22:00:56.802 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0049894070000000004 2021-03-24T22:00:56.802 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.802 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.802 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.803 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.358849+0000", 2021-03-24T22:00:56.803 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00037728300000000001 2021-03-24T22:00:56.803 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.803 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.803 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:00:56.804 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.359226+0000", 2021-03-24T22:00:56.804 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000242387 2021-03-24T22:00:56.804 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.804 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.805 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 4,7", 2021-03-24T22:00:56.805 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.359468+0000", 2021-03-24T22:00:56.805 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0026794480000000001 2021-03-24T22:00:56.805 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.805 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.806 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:00:56.806 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.362148+0000", 2021-03-24T22:00:56.806 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0066132209999999999 2021-03-24T22:00:56.806 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.806 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.807 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:56.807 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.368761+0000", 2021-03-24T22:00:56.807 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00040853900000000003 2021-03-24T22:00:56.807 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.807 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.808 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:56.808 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.369170+0000", 2021-03-24T22:00:56.808 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.7600000000000002e-05 2021-03-24T22:00:56.808 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.809 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.809 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:00:56.809 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.369247+0000", 2021-03-24T22:00:56.809 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012559999999999999 2021-03-24T22:00:56.809 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.810 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.810 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:00:56.810 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.369373+0000", 2021-03-24T22:00:56.810 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.090169935 2021-03-24T22:00:56.810 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:56.811 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:56.811 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:56.811 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.811 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.811 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:3948 7.13 7:cb55abc9:::smithi12556470-6:head [setxattr _header (57) in=7b,truncate 3058973] snapc 136=[136,133,132,130,12d,12c,12b,129,128,127,126,120,11e,11b,11a,119,110,10f,10c,106,105,104,fa,f8,e7] ondisk+write+known_if_redirected e417)", 2021-03-24T22:00:56.812 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:47.279271+0000", 2021-03-24T22:00:56.812 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 69.240880308000001, 2021-03-24T22:00:56.812 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.091553282, 2021-03-24T22:00:56.812 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:00:56.813 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:00:56.813 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:00:56.813 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:00:56.813 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:00:56.813 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3948 2021-03-24T22:00:56.814 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.814 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:00:56.814 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.814 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:00:56.814 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.279271+0000", 2021-03-24T22:00:56.815 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:00:56.815 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.815 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.815 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:00:56.815 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.279271+0000", 2021-03-24T22:00:56.816 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T22:00:56.816 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.816 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.816 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:00:56.816 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.279270+0000", 2021-03-24T22:00:56.817 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.5469999999999997e-06 2021-03-24T22:00:56.817 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.817 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.817 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:00:56.818 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.279278+0000", 2021-03-24T22:00:56.818 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.6549999999999999e-06 2021-03-24T22:00:56.818 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.818 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.818 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:00:56.819 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.279288+0000", 2021-03-24T22:00:56.819 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.7039999999999997e-06 2021-03-24T22:00:56.819 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.819 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.819 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:00:56.820 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.279295+0000", 2021-03-24T22:00:56.820 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000122798 2021-03-24T22:00:56.820 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.820 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.820 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.821 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.279418+0000", 2021-03-24T22:00:56.821 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.601e-06 2021-03-24T22:00:56.821 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.821 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.822 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:00:56.822 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.279424+0000", 2021-03-24T22:00:56.822 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0745048589999999 2021-03-24T22:00:56.822 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.822 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.823 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.823 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.353929+0000", 2021-03-24T22:00:56.823 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00013278000000000001 2021-03-24T22:00:56.823 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.823 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.824 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:00:56.824 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.354062+0000", 2021-03-24T22:00:56.824 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0070735479999999998 2021-03-24T22:00:56.824 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.824 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.825 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.825 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.361135+0000", 2021-03-24T22:00:56.825 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00017249999999999999 2021-03-24T22:00:56.825 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.825 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.826 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:00:56.826 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.361308+0000", 2021-03-24T22:00:56.826 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00031053800000000001 2021-03-24T22:00:56.826 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.827 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.827 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 4,7", 2021-03-24T22:00:56.827 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.361618+0000", 2021-03-24T22:00:56.827 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00098851199999999998 2021-03-24T22:00:56.827 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.828 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.828 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:00:56.828 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.362607+0000", 2021-03-24T22:00:56.828 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0078176459999999993 2021-03-24T22:00:56.828 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.829 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.829 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:56.829 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.370425+0000", 2021-03-24T22:00:56.829 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.0278000000000003e-05 2021-03-24T22:00:56.830 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.830 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.830 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:56.830 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.370465+0000", 2021-03-24T22:00:56.830 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00023850499999999999 2021-03-24T22:00:56.831 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.831 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.831 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:00:56.831 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.370703+0000", 2021-03-24T22:00:56.831 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000121403 2021-03-24T22:00:56.832 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.832 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.832 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:00:56.832 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.370825+0000", 2021-03-24T22:00:56.833 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.091553282 2021-03-24T22:00:56.833 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:56.833 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:56.833 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:56.833 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.834 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.834 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:3949 7.13 7:cb55abc9:::smithi12556470-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e417)", 2021-03-24T22:00:56.834 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:47.279301+0000", 2021-03-24T22:00:56.834 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 69.240850358000003, 2021-03-24T22:00:56.834 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.091821189, 2021-03-24T22:00:56.835 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:00:56.835 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T22:00:56.835 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:00:56.835 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:00:56.836 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:00:56.836 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3949 2021-03-24T22:00:56.836 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.836 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:00:56.836 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.837 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:00:56.837 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.279301+0000", 2021-03-24T22:00:56.837 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:00:56.837 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.837 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.838 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:00:56.838 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.279301+0000", 2021-03-24T22:00:56.838 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999995 2021-03-24T22:00:56.838 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.838 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.839 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:00:56.839 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.279301+0000", 2021-03-24T22:00:56.839 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.2090000000000001e-06 2021-03-24T22:00:56.839 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.839 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.840 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:00:56.840 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.279304+0000", 2021-03-24T22:00:56.840 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4300000000000003e-06 2021-03-24T22:00:56.840 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.841 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.841 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:00:56.841 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.279310+0000", 2021-03-24T22:00:56.841 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.4800000000000003e-06 2021-03-24T22:00:56.841 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.842 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.842 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:00:56.842 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.279315+0000", 2021-03-24T22:00:56.842 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000130607 2021-03-24T22:00:56.842 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.843 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.843 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.843 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.279445+0000", 2021-03-24T22:00:56.843 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.7619999999999998e-06 2021-03-24T22:00:56.843 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.844 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.844 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:00:56.844 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.279452+0000", 2021-03-24T22:00:56.844 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.074645898 2021-03-24T22:00:56.844 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.845 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.845 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.845 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.354098+0000", 2021-03-24T22:00:56.845 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010655200000000001 2021-03-24T22:00:56.846 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.846 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.846 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:00:56.846 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.354205+0000", 2021-03-24T22:00:56.846 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0077295649999999999 2021-03-24T22:00:56.847 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.847 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.847 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.847 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.361934+0000", 2021-03-24T22:00:56.847 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000114502 2021-03-24T22:00:56.848 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.848 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.848 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:00:56.848 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.362049+0000", 2021-03-24T22:00:56.848 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0088094460000000003 2021-03-24T22:00:56.849 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.849 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.849 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.849 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.370858+0000", 2021-03-24T22:00:56.850 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.4110999999999994e-05 2021-03-24T22:00:56.850 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.850 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.850 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:00:56.850 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.370952+0000", 2021-03-24T22:00:56.851 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00017025100000000001 2021-03-24T22:00:56.851 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.851 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.851 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:00:56.851 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.371123+0000", 2021-03-24T22:00:56.852 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.091821189 2021-03-24T22:00:56.852 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:56.852 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:56.852 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:56.852 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.853 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.853 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:3955 7.1d 7:bdda3e67:::smithi12556470-22:head [stat,delete] snapc 136=[136,133,132,130,12d,12c,12b,129,128,127,126,120,11e,11b,11a,119,110,10f,10c,106,105,104,fa,f8,e7] ondisk+write+known_if_redirected e417)", 2021-03-24T22:00:56.853 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:47.293804+0000", 2021-03-24T22:00:56.853 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 69.226347974999996, 2021-03-24T22:00:56.854 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0689564869999999, 2021-03-24T22:00:56.854 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:00:56.854 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:00:56.854 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:00:56.854 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:00:56.855 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:00:56.855 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3955 2021-03-24T22:00:56.855 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.855 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:00:56.855 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.856 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:00:56.856 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.293804+0000", 2021-03-24T22:00:56.856 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:00:56.856 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.857 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.857 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:00:56.857 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.293804+0000", 2021-03-24T22:00:56.857 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999976 2021-03-24T22:00:56.857 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.858 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.858 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:00:56.858 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.293802+0000", 2021-03-24T22:00:56.858 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.3969999999999994e-06 2021-03-24T22:00:56.858 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.859 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.859 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:00:56.859 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.293810+0000", 2021-03-24T22:00:56.859 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.8681999999999998e-05 2021-03-24T22:00:56.859 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.860 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.860 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:00:56.860 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.293829+0000", 2021-03-24T22:00:56.860 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.2639999999999997e-06 2021-03-24T22:00:56.861 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.861 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.861 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:00:56.861 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.293838+0000", 2021-03-24T22:00:56.861 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0042688300000000004 2021-03-24T22:00:56.862 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.862 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.862 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.862 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.298107+0000", 2021-03-24T22:00:56.862 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1899000000000001e-05 2021-03-24T22:00:56.863 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.863 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.863 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:00:56.863 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.298119+0000", 2021-03-24T22:00:56.863 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0540712640000001 2021-03-24T22:00:56.864 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.864 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.864 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.864 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.352190+0000", 2021-03-24T22:00:56.865 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000138947 2021-03-24T22:00:56.865 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.865 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.865 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:00:56.865 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.352329+0000", 2021-03-24T22:00:56.866 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002299892 2021-03-24T22:00:56.866 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.866 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.866 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 4,7", 2021-03-24T22:00:56.866 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.354629+0000", 2021-03-24T22:00:56.867 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0064260860000000001 2021-03-24T22:00:56.867 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.867 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.867 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:56.868 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.361055+0000", 2021-03-24T22:00:56.868 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001158987 2021-03-24T22:00:56.868 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.868 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.868 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:56.869 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.362214+0000", 2021-03-24T22:00:56.869 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.7753999999999997e-05 2021-03-24T22:00:56.869 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.869 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.869 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:00:56.870 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.362262+0000", 2021-03-24T22:00:56.870 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00017364400000000001 2021-03-24T22:00:56.870 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.870 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.871 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:00:56.871 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.362435+0000", 2021-03-24T22:00:56.871 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00032506800000000003 2021-03-24T22:00:56.871 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.871 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.872 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:00:56.872 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.362760+0000", 2021-03-24T22:00:56.872 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0689564869999999 2021-03-24T22:00:56.872 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:56.872 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:56.873 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:56.873 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.873 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.873 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:3983 7.1b 7:d841eed8:::smithi12556470-31:head [copy-from ver 539] snapc 137=[137,136,133,132,130,12d,12c,12b,129,128,127,126,120,11e,11b,11a,119,110,10f,10c,106,105,104,fa,f8,e7] ondisk+write+known_if_redirected e419)", 2021-03-24T22:00:56.873 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:49.256602+0000", 2021-03-24T22:00:56.874 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 67.263549581000007, 2021-03-24T22:00:56.874 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.100122246, 2021-03-24T22:00:56.874 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:00:56.874 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:00:56.875 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:00:56.875 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:00:56.875 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:00:56.875 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3983 2021-03-24T22:00:56.875 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.876 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:00:56.876 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.876 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:00:56.876 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:49.256602+0000", 2021-03-24T22:00:56.876 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:00:56.877 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.877 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.877 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:00:56.877 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:49.256602+0000", 2021-03-24T22:00:56.877 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999976 2021-03-24T22:00:56.878 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.878 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.878 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:00:56.878 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:49.256600+0000", 2021-03-24T22:00:56.879 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.4447000000000001e-05 2021-03-24T22:00:56.879 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.879 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.879 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:00:56.879 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:49.256614+0000", 2021-03-24T22:00:56.880 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.8822999999999999e-05 2021-03-24T22:00:56.880 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.880 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.880 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:00:56.880 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:49.256633+0000", 2021-03-24T22:00:56.881 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0617e-05 2021-03-24T22:00:56.881 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.881 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.881 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:00:56.881 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:49.256643+0000", 2021-03-24T22:00:56.882 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.1607999999999995e-05 2021-03-24T22:00:56.882 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.882 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.882 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.882 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:49.256725+0000", 2021-03-24T22:00:56.883 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1218e-05 2021-03-24T22:00:56.883 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.883 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.883 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op must wait for map", 2021-03-24T22:00:56.884 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:49.256736+0000", 2021-03-24T22:00:56.884 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.032142691000000001 2021-03-24T22:00:56.884 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.884 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.884 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.885 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:49.288879+0000", 2021-03-24T22:00:56.885 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000113266 2021-03-24T22:00:56.885 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.885 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.885 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:00:56.886 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:49.288992+0000", 2021-03-24T22:00:56.886 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.031536324999999997 2021-03-24T22:00:56.886 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.886 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.886 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,3", 2021-03-24T22:00:56.887 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:49.320529+0000", 2021-03-24T22:00:56.887 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.024239630000000002 2021-03-24T22:00:56.887 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.887 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.887 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:00:56.888 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:49.344768+0000", 2021-03-24T22:00:56.888 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000186973 2021-03-24T22:00:56.888 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.888 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.888 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:56.889 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:49.344955+0000", 2021-03-24T22:00:56.889 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011632982 2021-03-24T22:00:56.889 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.889 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.890 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:56.890 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:49.356588+0000", 2021-03-24T22:00:56.890 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6771999999999998e-05 2021-03-24T22:00:56.890 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.890 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.891 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:00:56.891 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:49.356645+0000", 2021-03-24T22:00:56.891 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.9468999999999994e-05 2021-03-24T22:00:56.891 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.891 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.892 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:00:56.892 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:49.356724+0000", 2021-03-24T22:00:56.892 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.100122246 2021-03-24T22:00:56.892 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:56.892 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:56.893 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:56.893 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.893 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.893 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:3984 7.1b 7:d841eed8:::smithi12556470-31:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e419)", 2021-03-24T22:00:56.893 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:49.256669+0000", 2021-03-24T22:00:56.894 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 67.263482322000002, 2021-03-24T22:00:56.894 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.100343191, 2021-03-24T22:00:56.894 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:00:56.894 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T22:00:56.895 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:00:56.895 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:00:56.895 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:00:56.895 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3984 2021-03-24T22:00:56.895 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.896 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:00:56.896 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.896 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:00:56.896 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:49.256669+0000", 2021-03-24T22:00:56.896 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:00:56.897 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.897 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.897 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:00:56.897 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:49.256669+0000", 2021-03-24T22:00:56.897 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T22:00:56.898 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.898 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.898 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:00:56.898 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:49.256669+0000", 2021-03-24T22:00:56.898 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.0910000000000003e-06 2021-03-24T22:00:56.899 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.899 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.899 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:00:56.899 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:49.256673+0000", 2021-03-24T22:00:56.900 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.6039e-05 2021-03-24T22:00:56.900 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.900 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.900 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:00:56.900 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:49.256689+0000", 2021-03-24T22:00:56.901 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6370000000000004e-06 2021-03-24T22:00:56.901 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.901 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.901 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:00:56.901 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:49.256694+0000", 2021-03-24T22:00:56.902 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000107452 2021-03-24T22:00:56.902 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.902 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.902 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.902 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:49.256802+0000", 2021-03-24T22:00:56.903 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.1700000000000002e-06 2021-03-24T22:00:56.903 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.903 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.903 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting_for_map not empty", 2021-03-24T22:00:56.903 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:49.256808+0000", 2021-03-24T22:00:56.904 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.032340269999999997 2021-03-24T22:00:56.904 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.904 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.904 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.905 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:49.289148+0000", 2021-03-24T22:00:56.905 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9869000000000001e-05 2021-03-24T22:00:56.905 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.905 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.905 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-24T22:00:56.906 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:49.289188+0000", 2021-03-24T22:00:56.906 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.055619156000000003 2021-03-24T22:00:56.906 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.906 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.906 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.907 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:49.344807+0000", 2021-03-24T22:00:56.907 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000105342 2021-03-24T22:00:56.907 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.907 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.907 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:00:56.908 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:49.344913+0000", 2021-03-24T22:00:56.908 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011928909999999999 2021-03-24T22:00:56.908 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.908 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.908 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.909 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:49.356842+0000", 2021-03-24T22:00:56.909 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.8724e-05 2021-03-24T22:00:56.909 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.909 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.910 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:00:56.910 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:49.356920+0000", 2021-03-24T22:00:56.910 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.2346e-05 2021-03-24T22:00:56.910 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.910 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.911 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:00:56.911 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:49.357013+0000", 2021-03-24T22:00:56.911 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.100343191 2021-03-24T22:00:56.911 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:56.911 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:56.912 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:56.912 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.912 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.912 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24388.0:488 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected e431)", 2021-03-24T22:00:56.913 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:00:02.363827+0000", 2021-03-24T22:00:56.913 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 54.156324261999998, 2021-03-24T22:00:56.913 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.937808413, 2021-03-24T22:00:56.913 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:00:56.913 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T22:00:56.914 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:00:56.914 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24388", 2021-03-24T22:00:56.914 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.142:0/626803041", 2021-03-24T22:00:56.914 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 488 2021-03-24T22:00:56.914 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.915 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:00:56.915 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.915 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:00:56.915 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.363827+0000", 2021-03-24T22:00:56.916 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:00:56.916 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.916 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.916 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:00:56.916 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.363827+0000", 2021-03-24T22:00:56.917 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999962 2021-03-24T22:00:56.917 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.917 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.917 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:00:56.917 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.363824+0000", 2021-03-24T22:00:56.918 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.6974e-05 2021-03-24T22:00:56.918 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.918 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.918 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:00:56.918 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.363841+0000", 2021-03-24T22:00:56.919 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.6693e-05 2021-03-24T22:00:56.919 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.919 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.919 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:00:56.919 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.363867+0000", 2021-03-24T22:00:56.920 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.9620000000000005e-06 2021-03-24T22:00:56.920 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.920 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.920 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:00:56.921 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.363877+0000", 2021-03-24T22:00:56.921 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.6908000000000002e-05 2021-03-24T22:00:56.921 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.921 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.921 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.922 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.363964+0000", 2021-03-24T22:00:56.922 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.8352e-05 2021-03-24T22:00:56.922 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.922 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.922 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:00:56.923 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.363982+0000", 2021-03-24T22:00:56.923 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.90250374600000005 2021-03-24T22:00:56.923 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.923 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.923 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.924 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:03.266486+0000", 2021-03-24T22:00:56.924 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.3829999999999992e-06 2021-03-24T22:00:56.924 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.924 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.924 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:00:56.925 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:03.266495+0000", 2021-03-24T22:00:56.925 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.034854283 2021-03-24T22:00:56.925 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.925 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.926 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.926 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.301349+0000", 2021-03-24T22:00:56.926 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00013150099999999999 2021-03-24T22:00:56.926 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.926 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.927 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:00:56.927 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.301480+0000", 2021-03-24T22:00:56.927 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00015549499999999999 2021-03-24T22:00:56.927 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.927 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.928 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:00:56.928 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.301636+0000", 2021-03-24T22:00:56.928 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.937808413 2021-03-24T22:00:56.928 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:56.928 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:56.929 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:56.929 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.929 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.929 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:4159 7.1d 7:ba8515f1:::smithi12556470-45:head [delete] snapc 143=[143,140,13d,13b,139,137,136,12d,12b,129,128,127,126,120,11e,11b,11a,119,10f,10c,105,104,fa,f8,e7] ondisk+write+peerstat_old+known_if_redirected e433)", 2021-03-24T22:00:56.929 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:00:03.271739+0000", 2021-03-24T22:00:56.930 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 53.248412963, 2021-03-24T22:00:56.930 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0649968089999999, 2021-03-24T22:00:56.930 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:00:56.930 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:00:56.931 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:00:56.931 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:00:56.931 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:00:56.931 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4159 2021-03-24T22:00:56.931 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.932 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:00:56.932 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.932 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:00:56.932 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:03.271739+0000", 2021-03-24T22:00:56.932 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:00:56.933 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.933 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.933 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:00:56.933 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:03.271739+0000", 2021-03-24T22:00:56.933 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T22:00:56.934 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.934 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.934 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:00:56.934 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:03.271737+0000", 2021-03-24T22:00:56.935 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.4970000000000004e-06 2021-03-24T22:00:56.935 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.935 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.935 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:00:56.935 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:03.271744+0000", 2021-03-24T22:00:56.936 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0215e-05 2021-03-24T22:00:56.936 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.936 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.936 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:00:56.936 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:03.271754+0000", 2021-03-24T22:00:56.937 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.2169999999999997e-06 2021-03-24T22:00:56.937 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.937 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.937 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:00:56.937 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:03.271762+0000", 2021-03-24T22:00:56.938 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.2197999999999998e-05 2021-03-24T22:00:56.938 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.938 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.938 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.939 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:03.271804+0000", 2021-03-24T22:00:56.939 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6640000000000003e-06 2021-03-24T22:00:56.939 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.939 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.939 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:00:56.940 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:03.271810+0000", 2021-03-24T22:00:56.940 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0345716599999999 2021-03-24T22:00:56.940 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.940 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.940 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.941 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.306381+0000", 2021-03-24T22:00:56.941 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00087488900000000001 2021-03-24T22:00:56.941 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.941 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.941 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for degraded object", 2021-03-24T22:00:56.942 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.307256+0000", 2021-03-24T22:00:56.942 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.023185320999999998 2021-03-24T22:00:56.942 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.942 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.943 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:00:56.943 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.330441+0000", 2021-03-24T22:00:56.943 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000119104 2021-03-24T22:00:56.943 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.943 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.944 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:00:56.944 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.330561+0000", 2021-03-24T22:00:56.944 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00028368700000000002 2021-03-24T22:00:56.944 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.944 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.945 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 4,5", 2021-03-24T22:00:56.945 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.330844+0000", 2021-03-24T22:00:56.945 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00086687799999999996 2021-03-24T22:00:56.945 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.945 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.946 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:00:56.946 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.331711+0000", 2021-03-24T22:00:56.946 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00024616999999999998 2021-03-24T22:00:56.946 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.947 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.947 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:56.947 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.331957+0000", 2021-03-24T22:00:56.947 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0045320960000000002 2021-03-24T22:00:56.947 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.948 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.948 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:00:56.948 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.336489+0000", 2021-03-24T22:00:56.948 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.1700999999999997e-05 2021-03-24T22:00:56.948 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.949 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.949 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:00:56.949 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.336561+0000", 2021-03-24T22:00:56.949 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00017450500000000001 2021-03-24T22:00:56.949 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:00:56.950 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:00:56.950 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:00:56.950 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.336736+0000", 2021-03-24T22:00:56.950 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0649968089999999 2021-03-24T22:00:56.951 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:56.951 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:56.951 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:56.951 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:00:56.951 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:00:56.952 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-24T22:00:56.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:56.953 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:00:56.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:00:57.054 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:57.111 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:57.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:57.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:57.218 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:57.277 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:57.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:57.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:00:57.384 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:57.395 INFO:tasks.rados.rados.0.smithi125.stdout:1842: done (0 left) 2021-03-24T22:00:57.395 INFO:tasks.rados.rados.0.smithi125.stdout:1843: snap_remove snap 193 2021-03-24T22:00:57.399 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:57 smithi125 conmon[19936]: cluster 2021-03-24T22:00:56.217009+0000 osd.2 (osd.2 2021-03-24T22:00:57.399 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:57 smithi125 conmon[19936]: ) 79 : cluster [DBG] 5.f scrub ok 2021-03-24T22:00:57.399 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:57 smithi125 conmon[19936]: cluster 2021-03-24T22:00:56.386902+0000 mon.a (mon.0) 1426 : cluster [DBG] osdmap e486: 8 total, 8 up, 5 in 2021-03-24T22:00:57.400 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:57 smithi125 conmon[19936]: cluster 2021-03-24T22 2021-03-24T22:00:57.400 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:57 smithi125 conmon[19936]: :00:56.483489+0000 osd.2 (osd.2) 80 : 2021-03-24T22:00:57.400 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:57 smithi125 conmon[19936]: cluster [DBG] purged_snaps scrub starts 2021-03-24T22:00:57.400 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:57 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:00:57.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:57 smithi125 conmon[19936]: -03-24T22:00:56.486040+0000 2021-03-24T22:00:57.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:57 smithi125 conmon[19936]: osd.2 (osd.2) 81 : cluster [DBG] 2021-03-24T22:00:57.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:57 smithi125 conmon[19936]: purged_snaps scrub ok 2021-03-24T22:00:57.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:57 smithi125 conmon[19936]: cluster 2021-03-24T22:00: 2021-03-24T22:00:57.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:57 smithi125 conmon[19936]: 57.316670+0000 osd.3 (osd.3) 104 2021-03-24T22:00:57.435 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:57.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:57.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:00:57.541 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:57.556 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:57 smithi142 conmon[20125]: cluster 2021-03-24T22:00:56.217009+0000 osd.2 (osd.2) 79 : 2021-03-24T22:00:57.556 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:57 smithi142 conmon[20125]: cluster [DBG] 5.f scrub ok 2021-03-24T22:00:57.556 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:57 smithi142 conmon[20125]: cluster 2021-03-24T22:00:56.386902+0000 mon.a (mon.0) 1426 : cluster [DBG] osdmap e486: 8 total, 8 up, 5 in 2021-03-24T22:00:57.557 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:57 smithi142 conmon[20125]: cluster 2021-03-24T22:00:56.483489+0000 osd.2 (osd.2) 80 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:00:57.557 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:57 smithi142 conmon[20125]: cluster 2021-03-24T22:00:56.486040+0000 osd.2 (osd.2) 81 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:00:57.557 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:57 smithi142 conmon[20125]: cluster 2021-03-24T22:00:57.316670+0000 osd.3 (osd.3) 104 2021-03-24T22:00:57.557 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:57 smithi142 conmon[20125]: : cluster [DBG] 6.f scrub ok 2021-03-24T22:00:57.596 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-24T22:00:57.604 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:57.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:57.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:00:57.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:57 smithi125 conmon[19936]: : cluster [DBG] 6.f scrub ok 2021-03-24T22:00:57.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:57 smithi125 conmon[24927]: cluster 2021-03-24T22:00:56.217009+0000 osd.2 (osd 2021-03-24T22:00:57.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:57 smithi125 conmon[24927]: .2) 79 : cluster [DBG] 5.f scrub ok 2021-03-24T22:00:57.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:57 smithi125 conmon[24927]: cluster 2021-03-24T22:00: 2021-03-24T22:00:57.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:57 smithi125 conmon[24927]: 56.386902+0000 mon.a (mon.0) 1426 : cluster [DBG] osdmap e486: 8 total, 8 up, 5 in 2021-03-24T22:00:57.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:57 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:00:57.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:57 smithi125 conmon[24927]: -03-24T22:00:56.483489+0000 osd.2 (osd.2) 2021-03-24T22:00:57.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:57 smithi125 conmon[24927]: 80 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:00:57.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:57 smithi125 conmon[24927]: cluster 2021-03-24T22:00: 2021-03-24T22:00:57.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:57 smithi125 conmon[24927]: 56.486040+0000 osd.2 (osd.2) 81 : cluster 2021-03-24T22:00:57.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:57 smithi125 conmon[24927]: [DBG] purged_snaps scrub ok 2021-03-24T22:00:57.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:57 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:00:57.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:57 smithi125 conmon[24927]: 00:57.316670+0000 osd.3 (osd.3) 104 : cluster [DBG] 6.f scrub ok 2021-03-24T22:00:57.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:57 smithi125 conmon[24927]: 2021-03-24T22:00:57.712 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:57.770 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:57.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:57.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:00:57.874 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:57.928 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:57.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:57.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:00:58.035 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:58.092 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:58.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:58.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:00:58.199 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:58.260 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:58.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:58.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:58.367 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:58.398 INFO:tasks.rados.rados.0.smithi125.stdout:1843: done (0 left) 2021-03-24T22:00:58.399 INFO:tasks.rados.rados.0.smithi125.stdout:1844: snap_remove snap 189 2021-03-24T22:00:58.430 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:00:58.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:58.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:00:58.533 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:58.591 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:00:58.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:58.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:00:58.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:58 smithi125 conmon[24927]: cluster 2021-03-24T22:00:56.871610+0000 2021-03-24T22:00:58.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:58 smithi125 conmon[24927]: mgr.y (mgr.14144) 515 : cluster [DBG] pgmap v892: 168 pgs: 168 active+clean; 463 MiB data, 1.1 GiB used, 446 GiB / 447 GiB avail; 7.0 MiB/s rd, 3.7 MiB/s wr, 24 op/s 2021-03-24T22:00:58.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:58 smithi125 conmon[24927]: cluster 2021-03-24T22:00:57.391250+0000 2021-03-24T22:00:58.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:58 smithi125 conmon[24927]: mon.a (mon.0) 1427 : cluster [DBG] osdmap e487: 8 total, 8 up, 5 in 2021-03-24T22:00:58.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:58 smithi125 conmon[19936]: cluster 2021-03-24T22:00:58.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:58 smithi125 conmon[19936]: 2021-03-24T22:00:58.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:58 smithi125 conmon[19936]: 2021 2021-03-24T22:00:58.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:58 smithi125 conmon[19936]: - 2021-03-24T22:00:58.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:58 smithi125 conmon[19936]: 03-24T 2021-03-24T22:00:58.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:58 smithi125 conmon[19936]: 22:00:56.871610+0000 mgr.y (mgr.14144) 515 : cluster [DBG] 2021-03-24T22:00:58.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:58 smithi125 conmon[19936]: pgmap v892: 168 pgs: 168 active+clean; 463 MiB data, 1.1 GiB used, 446 GiB / 447 GiB avail; 7.0 MiB/s rd, 3.7 MiB/s wr, 24 op/s 2021-03-24T22:00:58.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:58 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T22:00:58.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:58 smithi125 conmon[19936]: -24T22:00:57.391250+0000 mon.a (mon.0) 1427 : cluster [DBG] 2021-03-24T22:00:58.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:58 smithi125 conmon[19936]: osdmap e487: 8 total, 8 up, 5 in 2021-03-24T22:00:58.699 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:58.714 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:58 smithi142 conmon[20125]: cluster 2021-03-24T22:00:56.871610+0000 mgr.y (mgr 2021-03-24T22:00:58.715 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:58 smithi142 conmon[20125]: .14144) 515 : cluster [DBG] pgmap v892: 168 pgs: 168 active+clean; 463 MiB data, 1.1 GiB used, 446 GiB / 447 GiB avail; 7.0 MiB/s rd, 3.7 MiB/s wr, 24 op/s 2021-03-24T22:00:58.715 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:58 smithi142 conmon[20125]: cluster 2021-03-24T22:00:57.391250+0000 mon.a (mon.0) 1427 : cluster [DBG] osdmap e487: 8 total, 8 up, 5 in 2021-03-24T22:00:58.757 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:00:58.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:58.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:00:58.863 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:58.923 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:00:58.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:58.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:00:59.030 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:59.088 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:59.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:59.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:00:59.195 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:59.257 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:59.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:59.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:00:59.328 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.1 dump_ops_in_flight 2021-03-24T22:00:59.354 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:00:59.365 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:59.407 INFO:tasks.rados.rados.0.smithi125.stdout:1844: done (0 left) 2021-03-24T22:00:59.408 INFO:tasks.rados.rados.0.smithi125.stdout:1845: read oid 18 snap -1 2021-03-24T22:00:59.408 INFO:tasks.rados.rados.0.smithi125.stdout:1845: expect (ObjNum 588 snap 177 seq_num 588) 2021-03-24T22:00:59.408 INFO:tasks.rados.rados.0.smithi125.stdout:1846: write oid 40 current snap is 200 2021-03-24T22:00:59.408 INFO:tasks.rados.rados.0.smithi125.stdout:1846: seq_num 653 ranges {574971=627602,1862271=598221} 2021-03-24T22:00:59.417 INFO:tasks.rados.rados.0.smithi125.stdout:1846: writing smithi12556470-40 from 574971 to 1202573 tid 1 2021-03-24T22:00:59.426 INFO:tasks.rados.rados.0.smithi125.stdout:1846: writing smithi12556470-40 from 1862271 to 2460492 tid 2 2021-03-24T22:00:59.428 INFO:tasks.rados.rados.0.smithi125.stdout:1847: write oid 38 current snap is 200 2021-03-24T22:00:59.429 INFO:tasks.rados.rados.0.smithi125.stdout:1847: seq_num 654 ranges {591778=419204,1637106=741879,2910049=81729} 2021-03-24T22:00:59.434 INFO:tasks.rados.rados.0.smithi125.stdout:1847: writing smithi12556470-38 from 591778 to 1010982 tid 1 2021-03-24T22:00:59.435 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:59.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:59.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:00:59.443 INFO:tasks.rados.rados.0.smithi125.stdout:1847: writing smithi12556470-38 from 1637106 to 2378985 tid 2 2021-03-24T22:00:59.447 INFO:tasks.rados.rados.0.smithi125.stdout:1847: writing smithi12556470-38 from 2910049 to 2991778 tid 3 2021-03-24T22:00:59.447 INFO:tasks.rados.rados.0.smithi125.stdout:1848: snap_create 2021-03-24T22:00:59.448 INFO:tasks.rados.rados.0.smithi125.stdout:1846: finishing write tid 1 to smithi12556470-40 2021-03-24T22:00:59.448 INFO:tasks.rados.rados.0.smithi125.stdout:1847: finishing write tid 1 to smithi12556470-38 2021-03-24T22:00:59.449 INFO:tasks.rados.rados.0.smithi125.stdout:1846: finishing write tid 2 to smithi12556470-40 2021-03-24T22:00:59.449 INFO:tasks.rados.rados.0.smithi125.stdout:1846: finishing write tid 3 to smithi12556470-40 2021-03-24T22:00:59.450 INFO:tasks.rados.rados.0.smithi125.stdout:1846: finishing write tid 4 to smithi12556470-40 2021-03-24T22:00:59.450 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 40 v 844 (ObjNum 653 snap 200 seq_num 653) dirty exists 2021-03-24T22:00:59.450 INFO:tasks.rados.rados.0.smithi125.stdout:1846: left oid 40 (ObjNum 653 snap 200 seq_num 653) 2021-03-24T22:00:59.451 INFO:tasks.rados.rados.0.smithi125.stdout:1845: expect (ObjNum 588 snap 177 seq_num 588) 2021-03-24T22:00:59.521 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:59 smithi125 conmon[19936]: cluster 2021-03-24T22:00:57.659198+0000 osd.0 (osd. 2021-03-24T22:00:59.521 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:59 smithi125 conmon[19936]: 0) 102 : cluster [DBG] 4.3 scrub ok 2021-03-24T22:00:59.522 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:59 smithi125 conmon[19936]: cluster 2021-03-24T22:00:58.395154+0000 mon.a (mon.0) 1428 : cluster [DBG] osdmap e488: 8 total, 8 up, 5 in 2021-03-24T22:00:59.522 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:59 smithi125 conmon[19936]: cluster 2021-03-24T22:00:59.396499+0000 mon.a (mon.0) 1429 : 2021-03-24T22:00:59.522 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:00:59 smithi125 conmon[19936]: cluster [DBG] osdmap e489: 8 total, 8 up, 5 in 2021-03-24T22:00:59.523 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:59 smithi125 conmon[24927]: cluster 2021-03-24T22:00:59.523 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:59 smithi125 conmon[24927]: 2021-03-24T22:00:59.523 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:59 smithi125 conmon[24927]: 2021-03-24T22:00 2021-03-24T22:00:59.524 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:59 smithi125 conmon[24927]: :57.659198+0000 osd.0 (osd.0) 102 : cluster [DBG] 4.3 scrub ok 2021-03-24T22:00:59.524 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:59 smithi125 conmon[24927]: cluster 2021-03-24T22:00:59.524 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:59 smithi125 conmon[24927]: 2021-03-24T22:00:58.395154+0000 mon.a (mon.0 2021-03-24T22:00:59.525 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:59 smithi125 conmon[24927]: ) 1428 : cluster [DBG] osdmap e488: 8 total, 8 up, 5 in 2021-03-24T22:00:59.525 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:59 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:00:59.525 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:59 smithi125 conmon[24927]: -03-24T22:00:59.396499+0000 mon.a ( 2021-03-24T22:00:59.525 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:00:59 smithi125 conmon[24927]: mon.0) 1429 : cluster [DBG] osdmap e489: 8 total, 8 up, 5 in 2021-03-24T22:00:59.542 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:59.604 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:00:59.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:59.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:00:59.711 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:59.775 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:00:59.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:59.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:00:59.792 INFO:tasks.rados.rados.0.smithi125.stdout:1847: finishing write tid 2 to smithi12556470-38 2021-03-24T22:00:59.793 INFO:tasks.rados.rados.0.smithi125.stdout:1847: finishing write tid 3 to smithi12556470-38 2021-03-24T22:00:59.793 INFO:tasks.rados.rados.0.smithi125.stdout:1847: finishing write tid 4 to smithi12556470-38 2021-03-24T22:00:59.794 INFO:tasks.rados.rados.0.smithi125.stdout:1847: finishing write tid 5 to smithi12556470-38 2021-03-24T22:00:59.794 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 38 v 108 (ObjNum 654 snap 200 seq_num 654) dirty exists 2021-03-24T22:00:59.794 INFO:tasks.rados.rados.0.smithi125.stdout:1847: left oid 38 (ObjNum 654 snap 200 seq_num 654) 2021-03-24T22:00:59.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:59 smithi142 conmon[20125]: cluster 2021-03-24T22:00:59.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:59 smithi142 conmon[20125]: 2021-03-24T22:00:59.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:59 smithi142 conmon[20125]: 2021 2021-03-24T22:00:59.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:59 smithi142 conmon[20125]: - 2021-03-24T22:00:59.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:59 smithi142 conmon[20125]: 03 2021-03-24T22:00:59.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:59 smithi142 conmon[20125]: - 2021-03-24T22:00:59.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:59 smithi142 conmon[20125]: 24 2021-03-24T22:00:59.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:59 smithi142 conmon[20125]: T 2021-03-24T22:00:59.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:59 smithi142 conmon[20125]: 22 2021-03-24T22:00:59.872 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:59 smithi142 conmon[20125]: : 2021-03-24T22:00:59.872 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:59 smithi142 conmon[20125]: 00 2021-03-24T22:00:59.872 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:59 smithi142 conmon[20125]: :57.659198+0000 osd.0 (osd. 2021-03-24T22:00:59.872 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:59 smithi142 conmon[20125]: 0) 102 : cluster [DBG] 4.3 scrub ok 2021-03-24T22:00:59.873 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:59 smithi142 conmon[20125]: cluster 2021-03-24T22:00:59.873 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:59 smithi142 conmon[20125]: 2021-03-24T22:00:58.395154+0000 mon.a 2021-03-24T22:00:59.873 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:59 smithi142 conmon[20125]: (mon.0) 1428 : cluster [DBG] osdmap e488: 8 total, 8 up, 5 in 2021-03-24T22:00:59.873 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:59 smithi142 conmon[20125]: cluster 2021-03-24T22:00:59.874 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:59 smithi142 conmon[20125]: 2021-03-24T22:00:59.396499+0000 mon.a (mon.0 2021-03-24T22:00:59.874 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:00:59 smithi142 conmon[20125]: ) 1429 : cluster [DBG] osdmap e489: 8 total, 8 up, 5 in 2021-03-24T22:00:59.882 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:59.927 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:00:59.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:00:59.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:00:59.959 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:01:00.033 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:00.087 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:00.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:00.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:01:00.192 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:00.247 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:00.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:00.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:01:00.264 INFO:tasks.rados.rados.0.smithi125.stdout:1845: done (3 left) 2021-03-24T22:01:00.264 INFO:tasks.rados.rados.0.smithi125.stdout:1846: done (2 left) 2021-03-24T22:01:00.265 INFO:tasks.rados.rados.0.smithi125.stdout:1847: done (1 left) 2021-03-24T22:01:00.265 INFO:tasks.rados.rados.0.smithi125.stdout:1848: done (0 left) 2021-03-24T22:01:00.265 INFO:tasks.rados.rados.0.smithi125.stdout:1849: copy_from oid 46 from oid 43 current snap is 201 2021-03-24T22:01:00.270 INFO:tasks.rados.rados.0.smithi125.stdout:1850: write (excl) oid 28 current snap is 201 2021-03-24T22:01:00.270 INFO:tasks.rados.rados.0.smithi125.stdout:1850: seq_num 655 ranges {608585=494453,1579235=601890,2821119=187466} 2021-03-24T22:01:00.271 INFO:tasks.rados.rados.0.smithi125.stdout:1850: writing smithi12556470-28 from 608585 to 1103038 tid 1 2021-03-24T22:01:00.279 INFO:tasks.rados.rados.0.smithi125.stdout:1850: writing smithi12556470-28 from 1579235 to 2181125 tid 2 2021-03-24T22:01:00.290 INFO:tasks.rados.rados.0.smithi125.stdout:1850: writing smithi12556470-28 from 2821119 to 3008585 tid 3 2021-03-24T22:01:00.291 INFO:teuthology.orchestra.run.smithi125.stderr:marked in osd.7. 2021-03-24T22:01:00.292 INFO:tasks.rados.rados.0.smithi125.stdout:1851: snap_create 2021-03-24T22:01:00.293 INFO:teuthology.orchestra.run.smithi125.stderr:nodeep-scrub is unset 2021-03-24T22:01:00.301 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[19936]: debug 2021-03-24T22:01:00.159+0000 7f2118128700 -1 mon.a@0(leader).osd e489 definitely_dead 0 2021-03-24T22:01:00.337 INFO:tasks.rados.rados.0.smithi125.stdout:1850: finishing write tid 1 to smithi12556470-28 2021-03-24T22:01:00.352 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:00.372 INFO:tasks.rados.rados.0.smithi125.stdout:1850: finishing write tid 2 to smithi12556470-28 2021-03-24T22:01:00.373 INFO:tasks.rados.rados.0.smithi125.stdout:1850: finishing write tid 3 to smithi12556470-28 2021-03-24T22:01:00.373 INFO:tasks.rados.rados.0.smithi125.stdout:1850: finishing write tid 4 to smithi12556470-28 2021-03-24T22:01:00.373 INFO:tasks.rados.rados.0.smithi125.stdout:1850: finishing write tid 5 to smithi12556470-28 2021-03-24T22:01:00.374 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 28 v 393 (ObjNum 655 snap 201 seq_num 655) dirty exists 2021-03-24T22:01:00.374 INFO:tasks.rados.rados.0.smithi125.stdout:1850: left oid 28 (ObjNum 655 snap 201 seq_num 655) 2021-03-24T22:01:00.404 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:00.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:00.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:01:00.511 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:00.527 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:00 smithi142 conmon[20125]: cluster 2021-03-24T22:00:58. 2021-03-24T22:01:00.527 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:00 smithi142 conmon[20125]: 712059+0000 osd.5 (osd.5) 8 : cluster [DBG] 4.2 scrub ok 2021-03-24T22:01:00.528 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:00 smithi142 conmon[20125]: cluster 2021-03-24T22:00:58.872363 2021-03-24T22:01:00.528 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:00 smithi142 conmon[20125]: +0000 mgr.y (mgr.14144) 516 : cluster [DBG] pgmap v895: 168 pgs: 168 active+clean; 463 MiB data, 1.1 GiB used, 446 GiB / 447 GiB avail; 1023 B/s rd, 1.1 MiB/s wr, 6 op/s 2021-03-24T22:01:00.529 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:00 smithi142 conmon[20125]: cluster 2021-03-24T22 2021-03-24T22:01:00.529 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:00 smithi142 conmon[20125]: :00:59.303893+0000 osd.2 (osd.2) 2021-03-24T22:01:00.529 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:00 smithi142 conmon[20125]: 82 : cluster [DBG] 6.17 scrub ok 2021-03-24T22:01:00.530 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:00 smithi142 conmon[20125]: audit 2021-03- 2021-03-24T22:01:00.530 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:00 smithi142 conmon[20125]: 24T22:00:59.550750+0000 mon.a (mon 2021-03-24T22:01:00.531 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:00 smithi142 conmon[20125]: .0) 1430 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T22:01:00.531 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:00 smithi142 conmon[20125]: audit 2021 2021-03-24T22:01:00.531 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:00 smithi142 conmon[20125]: -03-24T22:00:59.882693+0000 2021-03-24T22:01:00.532 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:00 smithi142 conmon[20125]: mon.a (mon.0) 1431 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:01:00.532 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:00 smithi142 conmon[20125]: audit 2021 2021-03-24T22:01:00.532 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:00 smithi142 conmon[20125]: -03-24T22:01:00.132277+0000 2021-03-24T22:01:00.533 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:00 smithi142 conmon[20125]: mon.c (mon.2) 75 : audit [INF] from='client.? v1:172.21.15.125:0/2703265940' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:01:00.533 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:00 smithi142 conmon[20125]: audit 2021-03-24T22:01:00.533 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:00 smithi142 conmon[20125]: 2021-03-24T22:01:00.132794+0000 2021-03-24T22:01:00.534 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:00 smithi142 conmon[20125]: mon.a (mon.0) 1432 : audit [INF] from='client.35327 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:01:00.534 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:00 smithi142 conmon[20125]: 2021-03-24T22:01:00.535 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:00 smithi142 conmon[20125]: audit 2021-03-24T22:01:00 2021-03-24T22:01:00.535 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:00 smithi142 conmon[20125]: .162763+0000 mon.c (mon.2) 76 : audit 2021-03-24T22:01:00.535 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:00 smithi142 conmon[20125]: [DBG] from='client.? v1:172.21.15.125:0/3097361609' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2021-03-24T22:01:00.536 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:00 smithi142 conmon[20125]: audit 2021-03-24 2021-03-24T22:01:00.536 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:00 smithi142 conmon[20125]: T22:01:00.163166+0000 mon.a (mon. 2021-03-24T22:01:00.536 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:00 smithi142 conmon[20125]: 0) 1433 : audit [DBG] from='client.35333 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2021-03-24T22:01:00.537 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:00 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:01:00.537 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:00 smithi142 conmon[20125]: 03-24T22:01:00.250987+0000 2021-03-24T22:01:00.537 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:00 smithi142 conmon[20125]: mon.a (mon.0) 1434 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T22:01:00.537 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:00 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:01:00.537 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:00 smithi142 conmon[20125]: -03-24T22:01:00.251048+0000 2021-03-24T22:01:00.538 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:00 smithi142 conmon[20125]: mon.a (mon.0) 1435 : cluster [INF] Cluster is now healthy 2021-03-24T22:01:00.538 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:00 smithi142 conmon[20125]: 2021-03-24T22:01:00.538 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:00 smithi142 conmon[20125]: audit 2021-03-24T22:01:00. 2021-03-24T22:01:00.538 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:00 smithi142 conmon[20125]: 259074+0000 mon.a (mon.0) 1436 : audit [INF] 2021-03-24T22:01:00.539 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:00 smithi142 conmon[20125]: from='client.35327 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T22:01:00.539 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:00 smithi142 conmon[20125]: audit 2021-03-24T22:01: 2021-03-24T22:01:00.539 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:00 smithi142 conmon[20125]: 00.259260+0000 mon.a (mon.0) 1437 : audit 2021-03-24T22:01:00.539 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:00 smithi142 conmon[20125]: [INF] from='client.35333 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2021-03-24T22:01:00.539 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:00 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:01:00.540 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:00 smithi142 conmon[20125]: 24T22:01:00.259397+0000 mon.a (mon.0) 1438 : cluster 2021-03-24T22:01:00.540 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:00 smithi142 conmon[20125]: [DBG] osdmap e490: 8 total, 8 up, 6 in 2021-03-24T22:01:00.565 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:00.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:00.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:01:00.594 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[19936]: cluster 2021-03-24T22:01:00.594 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[19936]: 2021-03-24T22: 2021-03-24T22:01:00.595 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[19936]: 00:58.712059+0000 osd.5 (osd.5) 8 : cluster [DBG] 4.2 scrub ok 2021-03-24T22:01:00.595 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:01:00.596 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[19936]: -03-24T22:00:58.872363+0000 mgr.y 2021-03-24T22:01:00.596 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[19936]: (mgr.14144) 516 : cluster [DBG] pgmap v895: 168 pgs: 168 active+clean; 463 MiB data, 1.1 GiB used, 446 GiB / 447 GiB avail; 1023 B/s rd, 1.1 MiB/s wr, 6 op/s 2021-03-24T22:01:00.596 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:01:00.597 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[19936]: 03-24T22:00:59.303893+0000 2021-03-24T22:01:00.597 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[19936]: osd.2 (osd.2) 82 : cluster [DBG] 6.17 scrub ok 2021-03-24T22:01:00.598 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[19936]: audit 2021-03- 2021-03-24T22:01:00.598 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[19936]: 24T22:00:59.550750+0000 mon.a (mon.0) 1430 : 2021-03-24T22:01:00.598 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[19936]: audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T22:01:00.599 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[19936]: audit 2021- 2021-03-24T22:01:00.599 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[19936]: 03-24T22:00:59.882693+0000 mon.a (mon.0) 1431 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:01:00.599 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[19936]: 2021-03-24T22:01:00.599 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[19936]: audit 2021-03-24T22:01: 2021-03-24T22:01:00.600 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[19936]: 00.132277+0000 mon.c (mon.2) 75 : audit [INF] from='client.? v1:172.21.15.125:0/2703265940' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:01:00.600 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[19936]: audit 2021-03-24T22:01:00.132794+0000 mon.a (mon 2021-03-24T22:01:00.600 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[19936]: .0) 1432 : audit [INF] from='client.35327 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:01:00.600 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[19936]: audit 2021 2021-03-24T22:01:00.601 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[19936]: -03-24T22:01:00.162763+0000 mon.c (mon.2) 76 : audit [DBG] from='client.? v1:172.21.15.125:0/3097361609' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2021-03-24T22:01:00.601 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[19936]: audit 2021-03-24T22:01:00.163166+0000 2021-03-24T22:01:00.601 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[19936]: mon.a (mon.0) 1433 : audit [DBG] from='client.35333 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2021-03-24T22:01:00.601 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[19936]: cluster 2021-03-24T22:01:00.250987+0000 mon.a ( 2021-03-24T22:01:00.602 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[19936]: mon.0) 1434 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T22:01:00.602 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[19936]: cluster 2021-03-24T22:01:00.602 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[19936]: 2021-03-24T22:01:00.251048+0000 mon.a (mon 2021-03-24T22:01:00.603 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[19936]: .0) 1435 : cluster [INF] Cluster is now healthy 2021-03-24T22:01:00.603 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[19936]: audit 2021-03-24T 2021-03-24T22:01:00.603 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[19936]: 22:01:00.259074+0000 mon.a (mon.0) 2021-03-24T22:01:00.603 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[19936]: 1436 : audit [INF] from='client.35327 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T22:01:00.604 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[19936]: audit 2021-03-24T22 2021-03-24T22:01:00.604 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[19936]: :01:00.259260+0000 mon.a (mon.0) 2021-03-24T22:01:00.604 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[19936]: 1437 : audit [INF] from='client.35333 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2021-03-24T22:01:00.604 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:01:00.605 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[19936]: 01:00.259397+0000 mon.a (mon.0) 1438 2021-03-24T22:01:00.605 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[19936]: : cluster [DBG] osdmap e490: 8 total, 8 up, 6 in 2021-03-24T22:01:00.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[24927]: cluster 2021-03-24T22:00:58.712059+0000 2021-03-24T22:01:00.606 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[24927]: osd.5 (osd.5) 8 : cluster [DBG] 4.2 scrub ok 2021-03-24T22:01:00.606 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T22:01:00.606 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[24927]: 22:00:58.872363+0000 mgr.y (mgr.14144) 516 : cluster [DBG] 2021-03-24T22:01:00.606 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[24927]: pgmap v895: 168 pgs: 168 active+clean; 463 MiB data, 1.1 GiB used, 446 GiB / 447 GiB avail; 1023 B/s rd, 1.1 MiB/s wr, 6 op/s 2021-03-24T22:01:00.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[24927]: cluster 2021-03-24T22:01:00.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[24927]: 2021-03-24 2021-03-24T22:01:00.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[24927]: T22:00:59.303893+0000 2021-03-24T22:01:00.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[24927]: osd.2 (osd.2) 82 : cluster [DBG] 6.17 scrub ok 2021-03-24T22:01:00.608 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[24927]: audit 2021-03-24 2021-03-24T22:01:00.608 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[24927]: T22:00:59.550750+0000 mon.a (mon.0) 2021-03-24T22:01:00.608 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[24927]: 1430 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T22:01:00.608 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[24927]: audit 2021 2021-03-24T22:01:00.608 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[24927]: -03-24T22:00:59.882693+0000 mon.a 2021-03-24T22:01:00.609 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[24927]: (mon.0) 1431 : audit [INF] 2021-03-24T22:01:00.609 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[24927]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:01:00.609 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[24927]: audit 2021-03-24T22:01:00. 2021-03-24T22:01:00.609 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[24927]: 132277+0000 mon.c (mon.2) 75 : audit [INF] from='client.? v1:172.21.15.125:0/2703265940' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:01:00.610 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[24927]: audit 2021-03-24T22 2021-03-24T22:01:00.610 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[24927]: :01:00.132794+0000 mon.a (mon. 2021-03-24T22:01:00.610 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[24927]: 0) 1432 : audit [INF] from='client.35327 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:01:00.610 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[24927]: audit 2021-03-24T22:01 2021-03-24T22:01:00.610 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[24927]: :00.162763+0000 mon.c (mon.2) 76 : audit [DBG] 2021-03-24T22:01:00.611 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[24927]: from='client.? v1:172.21.15.125:0/3097361609' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2021-03-24T22:01:00.611 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[24927]: audit 2021-03-24T22:01:00.611 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[24927]: 2021-03-24T22:01:00.163166+0000 2021-03-24T22:01:00.612 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[24927]: mon.a (mon.0) 1433 : audit [DBG] from='client.35333 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2021-03-24T22:01:00.612 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[24927]: cluster 2021-03-24T22:01 2021-03-24T22:01:00.612 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[24927]: :00.250987+0000 mon.a (mon.0) 1434 : cluster [INF] 2021-03-24T22:01:00.612 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[24927]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T22:01:00.613 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:01:00.613 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[24927]: -03-24T22:01:00.251048+0000 mon.a (mon.0) 1435 : cluster [INF] Cluster is now healthy 2021-03-24T22:01:00.613 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[24927]: audit 2021- 2021-03-24T22:01:00.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[24927]: 03-24T22:01:00.259074+0000 mon.a (mon.0) 1436 : audit 2021-03-24T22:01:00.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[24927]: [INF] from='client.35327 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T22:01:00.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[24927]: audit 2021-03 2021-03-24T22:01:00.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[24927]: -24T22:01:00.259260+0000 mon.a ( 2021-03-24T22:01:00.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[24927]: mon.0) 1437 : audit [INF] from='client.35333 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2021-03-24T22:01:00.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:01:00.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[24927]: -03-24T22:01:00.259397 2021-03-24T22:01:00.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[24927]: +0000 mon.a (mon.0) 1438 : cluster [DBG] osdmap e490: 8 total, 8 up, 6 in 2021-03-24T22:01:00.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:00 smithi125 conmon[24927]: 2021-03-24T22:01:00.672 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:00.726 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:00.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:00.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:01:00.833 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:00.900 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:00.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:00.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:01:01.006 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:01.055 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:01.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:01.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:01:01.162 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:01.217 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:01.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:01.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:01:01.266 INFO:tasks.rados.rados.0.smithi125.stdout:1850: done (2 left) 2021-03-24T22:01:01.267 INFO:tasks.rados.rados.0.smithi125.stdout:1851: done (1 left) 2021-03-24T22:01:01.267 INFO:tasks.rados.rados.0.smithi125.stdout:1852: write oid 7 current snap is 202 2021-03-24T22:01:01.268 INFO:tasks.rados.rados.0.smithi125.stdout:1852: seq_num 656 ranges {625392=569702,1637717=745548,2848542=176850} 2021-03-24T22:01:01.274 INFO:tasks.rados.rados.0.smithi125.stdout:1852: writing smithi12556470-7 from 625392 to 1195094 tid 1 2021-03-24T22:01:01.282 INFO:tasks.rados.rados.0.smithi125.stdout:1852: writing smithi12556470-7 from 1637717 to 2383265 tid 2 2021-03-24T22:01:01.286 INFO:tasks.rados.rados.0.smithi125.stdout:1852: writing smithi12556470-7 from 2848542 to 3025392 tid 3 2021-03-24T22:01:01.286 INFO:tasks.rados.rados.0.smithi125.stdout:1853: delete oid 19 current snap is 202 2021-03-24T22:01:01.322 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:01.375 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:01.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:01.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:01:01.481 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:01.530 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:01.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:01.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:01:01.636 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:01.692 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:01.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:01.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:01:01.797 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:01.851 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:01.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:01.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:01:01.957 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:02.005 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:02.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:02.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:01:02.110 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:02.159 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:02.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:02.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:01:02.264 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:02.324 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:02.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:02.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:01:02.334 INFO:tasks.rados.rados.0.smithi125.stdout:1849: finishing copy_from to smithi12556470-46 2021-03-24T22:01:02.335 INFO:tasks.rados.rados.0.smithi125.stdout:1849: got expected ENOENT (src dne) 2021-03-24T22:01:02.335 INFO:tasks.rados.rados.0.smithi125.stdout:1849: finishing copy_from racing read to smithi12556470-46 2021-03-24T22:01:02.338 INFO:tasks.rados.rados.0.smithi125.stdout:1852: finishing write tid 1 to smithi12556470-7 2021-03-24T22:01:02.338 INFO:tasks.rados.rados.0.smithi125.stdout:1852: finishing write tid 2 to smithi12556470-7 2021-03-24T22:01:02.339 INFO:tasks.rados.rados.0.smithi125.stdout:1852: finishing write tid 3 to smithi12556470-7 2021-03-24T22:01:02.339 INFO:tasks.rados.rados.0.smithi125.stdout:1852: finishing write tid 4 to smithi12556470-7 2021-03-24T22:01:02.340 INFO:tasks.rados.rados.0.smithi125.stdout:1852: finishing write tid 5 to smithi12556470-7 2021-03-24T22:01:02.340 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 7 v 637 (ObjNum 656 snap 202 seq_num 656) dirty exists 2021-03-24T22:01:02.340 INFO:tasks.rados.rados.0.smithi125.stdout:1852: left oid 7 (ObjNum 656 snap 202 seq_num 656) 2021-03-24T22:01:02.341 INFO:tasks.rados.rados.0.smithi125.stdout:1849: done (2 left) 2021-03-24T22:01:02.341 INFO:tasks.rados.rados.0.smithi125.stdout:1852: done (1 left) 2021-03-24T22:01:02.341 INFO:tasks.rados.rados.0.smithi125.stdout:1853: done (0 left) 2021-03-24T22:01:02.341 INFO:tasks.rados.rados.0.smithi125.stdout:1854: delete oid 11 current snap is 202 2021-03-24T22:01:02.343 INFO:tasks.rados.rados.0.smithi125.stdout:1854: done (0 left) 2021-03-24T22:01:02.343 INFO:tasks.rados.rados.0.smithi125.stdout:1855: read oid 33 snap 168 2021-03-24T22:01:02.343 INFO:tasks.rados.rados.0.smithi125.stdout:1855: expect (ObjNum 298 snap 74 seq_num 298) 2021-03-24T22:01:02.344 INFO:tasks.rados.rados.0.smithi125.stdout:1856: write (excl) oid 50 current snap is 202 2021-03-24T22:01:02.344 INFO:tasks.rados.rados.0.smithi125.stdout:1856: seq_num 659 ranges {675813=795449,2096810=609228} 2021-03-24T22:01:02.353 INFO:tasks.rados.rados.0.smithi125.stdout:1856: writing smithi12556470-50 from 675813 to 1471262 tid 1 2021-03-24T22:01:02.361 INFO:tasks.rados.rados.0.smithi125.stdout:1856: writing smithi12556470-50 from 2096810 to 2706038 tid 2 2021-03-24T22:01:02.361 INFO:tasks.rados.rados.0.smithi125.stdout:1857: delete oid 20 current snap is 202 2021-03-24T22:01:02.361 INFO:tasks.rados.rados.0.smithi125.stdout:1855: expect (ObjNum 298 snap 74 seq_num 298) 2021-03-24T22:01:02.429 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:02.486 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:02.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:02.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:01:02.558 INFO:tasks.rados.rados.0.smithi125.stdout:1856: finishing write tid 1 to smithi12556470-50 2021-03-24T22:01:02.558 INFO:tasks.rados.rados.0.smithi125.stdout:1855: done (2 left) 2021-03-24T22:01:02.559 INFO:tasks.rados.rados.0.smithi125.stdout:1857: done (1 left) 2021-03-24T22:01:02.559 INFO:tasks.rados.rados.0.smithi125.stdout:1858: copy_from oid 40 from oid 11 current snap is 202 2021-03-24T22:01:02.559 INFO:tasks.rados.rados.0.smithi125.stdout:1856: finishing write tid 2 to smithi12556470-50 2021-03-24T22:01:02.560 INFO:tasks.rados.rados.0.smithi125.stdout:1856: finishing write tid 3 to smithi12556470-50 2021-03-24T22:01:02.560 INFO:tasks.rados.rados.0.smithi125.stdout:1856: finishing write tid 4 to smithi12556470-50 2021-03-24T22:01:02.560 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 50 v 461 (ObjNum 659 snap 202 seq_num 659) dirty exists 2021-03-24T22:01:02.560 INFO:tasks.rados.rados.0.smithi125.stdout:1856: left oid 50 (ObjNum 659 snap 202 seq_num 659) 2021-03-24T22:01:02.561 INFO:tasks.rados.rados.0.smithi125.stdout:1856: done (1 left) 2021-03-24T22:01:02.561 INFO:tasks.rados.rados.0.smithi125.stdout:1859: read oid 9 snap -1 2021-03-24T22:01:02.561 INFO:tasks.rados.rados.0.smithi125.stdout:1859: expect (ObjNum 520 snap 154 seq_num 520) 2021-03-24T22:01:02.561 INFO:tasks.rados.rados.0.smithi125.stdout:1860: copy_from oid 44 from oid 15 current snap is 202 2021-03-24T22:01:02.562 INFO:tasks.rados.rados.0.smithi125.stdout:1861: copy_from oid 12 from oid 26 current snap is 202 2021-03-24T22:01:02.562 INFO:tasks.rados.rados.0.smithi125.stdout:1862: snap_remove snap 166 2021-03-24T22:01:02.591 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:02.601 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:02 smithi125 conmon[24927]: cluster 2021-03-24T22:01:00 2021-03-24T22:01:02.602 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:02 smithi125 conmon[24927]: .693667+0000 osd.5 (osd.5) 9 : cluster [DBG] 5.7 scrub ok 2021-03-24T22:01:02.602 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:02 smithi125 conmon[24927]: cluster 2021-03-24T22:01: 2021-03-24T22:01:02.603 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:02 smithi125 conmon[24927]: 00.873020+0000 mgr.y (mgr.14144) 517 : 2021-03-24T22:01:02.603 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:02 smithi125 conmon[24927]: cluster [DBG] pgmap v898: 168 pgs: 168 active+clean; 466 MiB data, 1.1 GiB used, 446 GiB / 447 GiB avail; 4.4 MiB/s rd, 631 KiB/s wr, 6 op/s 2021-03-24T22:01:02.603 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:02 smithi125 conmon[24927]: cluster 2021-03-24T22 2021-03-24T22:01:02.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:02 smithi125 conmon[24927]: :01:01.262042+0000 mon.a (mon. 2021-03-24T22:01:02.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:02 smithi125 conmon[24927]: 0) 1439 : cluster [DBG] osdmap e491: 8 total, 8 up, 6 in 2021-03-24T22:01:02.604 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:02 smithi125 conmon[19936]: cluster 2021-03-24T22:01:02.604 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:02 smithi125 conmon[19936]: 2021-03-24T22:01 2021-03-24T22:01:02.605 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:02 smithi125 conmon[19936]: :00.693667+0000 osd.5 (osd.5) 9 : cluster [DBG] 5.7 scrub ok 2021-03-24T22:01:02.605 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:02 smithi125 conmon[19936]: cluster 2021-03-24T22:01:02.605 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:02 smithi125 conmon[19936]: 2021-03-24T 2021-03-24T22:01:02.605 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:02 smithi125 conmon[19936]: 22:01:00.873020+0000 mgr.y (mgr 2021-03-24T22:01:02.605 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:02 smithi125 conmon[19936]: .14144) 517 : cluster [DBG] pgmap v898: 168 pgs: 168 active+clean; 466 MiB data, 1.1 GiB used, 446 GiB / 447 GiB avail; 4.4 MiB/s rd, 631 KiB/s wr, 6 op/s 2021-03-24T22:01:02.606 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:02 smithi125 conmon[19936]: cluster 2021-03-24T22:01:02.606 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:02 smithi125 conmon[19936]: 2021-03-24T22:01:02.606 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:02 smithi125 conmon[19936]: 2021 2021-03-24T22:01:02.606 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:02 smithi125 conmon[19936]: -03-24T22:01: 2021-03-24T22:01:02.607 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:02 smithi125 conmon[19936]: 01.262042+0000 mon.a (mon.0) 1439 : cluster [DBG] osdmap e491: 8 total, 8 up, 6 in 2021-03-24T22:01:02.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:02 smithi142 conmon[20125]: cluster 2021-03-24T22:01:02.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:02 smithi142 conmon[20125]: 2021-03-24T22:01:02.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:02 smithi142 conmon[20125]: 2021 2021-03-24T22:01:02.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:02 smithi142 conmon[20125]: - 2021-03-24T22:01:02.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:02 smithi142 conmon[20125]: 03 2021-03-24T22:01:02.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:02 smithi142 conmon[20125]: - 2021-03-24T22:01:02.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:02 smithi142 conmon[20125]: 24 2021-03-24T22:01:02.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:02 smithi142 conmon[20125]: T 2021-03-24T22:01:02.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:02 smithi142 conmon[20125]: 22 2021-03-24T22:01:02.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:02 smithi142 conmon[20125]: : 2021-03-24T22:01:02.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:02 smithi142 conmon[20125]: 01 2021-03-24T22:01:02.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:02 smithi142 conmon[20125]: : 2021-03-24T22:01:02.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:02 smithi142 conmon[20125]: 00 2021-03-24T22:01:02.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:02 smithi142 conmon[20125]: . 2021-03-24T22:01:02.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:02 smithi142 conmon[20125]: 693667 2021-03-24T22:01:02.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:02 smithi142 conmon[20125]: +0000 2021-03-24T22:01:02.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:02 smithi142 conmon[20125]: 2021-03-24T22:01:02.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:02 smithi142 conmon[20125]: osd.5 2021-03-24T22:01:02.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:02 smithi142 conmon[20125]: ( 2021-03-24T22:01:02.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:02 smithi142 conmon[20125]: osd 2021-03-24T22:01:02.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:02 smithi142 conmon[20125]: . 2021-03-24T22:01:02.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:02 smithi142 conmon[20125]: 5 2021-03-24T22:01:02.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:02 smithi142 conmon[20125]: ) 2021-03-24T22:01:02.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:02 smithi142 conmon[20125]: 9 2021-03-24T22:01:02.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:02 smithi142 conmon[20125]: : 2021-03-24T22:01:02.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:02 smithi142 conmon[20125]: cluster 2021-03-24T22:01:02.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:02 smithi142 conmon[20125]: 2021-03-24T22:01:02.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:02 smithi142 conmon[20125]: [DBG] 2021-03-24T22:01:02.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:02 smithi142 conmon[20125]: 2021-03-24T22:01:02.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:02 smithi142 conmon[20125]: 5.7 scrub ok 2021-03-24T22:01:02.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:02 smithi142 conmon[20125]: 2021-03-24T22:01:02.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:02 smithi142 conmon[20125]: cluster 2021-03-24T22:01:02.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:02 smithi142 conmon[20125]: 2021-03-24T22:01:02.625 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:02 smithi142 conmon[20125]: 2021 2021-03-24T22:01:02.625 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:02 smithi142 conmon[20125]: - 2021-03-24T22:01:02.625 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:02 smithi142 conmon[20125]: 03 2021-03-24T22:01:02.625 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:02 smithi142 conmon[20125]: - 2021-03-24T22:01:02.625 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:02 smithi142 conmon[20125]: 24 2021-03-24T22:01:02.626 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:02 smithi142 conmon[20125]: T 2021-03-24T22:01:02.626 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:02 smithi142 conmon[20125]: 22 2021-03-24T22:01:02.626 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:02 smithi142 conmon[20125]: : 2021-03-24T22:01:02.626 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:02 smithi142 conmon[20125]: 01 2021-03-24T22:01:02.626 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:02 smithi142 conmon[20125]: :00.873020+0000 mgr.y 2021-03-24T22:01:02.627 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:02 smithi142 conmon[20125]: (mgr.14144) 517 : cluster [DBG] pgmap v898: 168 pgs: 168 active+clean; 466 MiB data, 1.1 GiB used, 446 GiB / 447 GiB avail; 4.4 MiB/s rd, 631 KiB/s wr, 6 op/s 2021-03-24T22:01:02.627 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:02 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:01:02.627 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:02 smithi142 conmon[20125]: T22:01:01.262042+0000 mon.a (mon.0) 2021-03-24T22:01:02.627 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:02 smithi142 conmon[20125]: 1439 : cluster [DBG] osdmap e491: 8 total, 8 up, 6 in 2021-03-24T22:01:02.638 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:02.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:02.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:01:02.706 INFO:tasks.thrashosds.thrasher:Added osd 7 2021-03-24T22:01:02.744 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:02.791 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:02.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:02.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:01:02.897 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:02.941 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:02.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:02.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:01:03.049 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:03.111 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:03.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:03.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:01:03.217 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:03.272 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:03.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:03.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:01:03.290 INFO:tasks.rados.rados.0.smithi125.stdout:1862: done (4 left) 2021-03-24T22:01:03.290 INFO:tasks.rados.rados.0.smithi125.stdout:1863: snap_create 2021-03-24T22:01:03.291 INFO:tasks.rados.rados.0.smithi125.stdout:1858: finishing copy_from to smithi12556470-40 2021-03-24T22:01:03.291 INFO:tasks.rados.rados.0.smithi125.stdout:1858: got expected ENOENT (src dne) 2021-03-24T22:01:03.291 INFO:tasks.rados.rados.0.smithi125.stdout:1860: finishing copy_from to smithi12556470-44 2021-03-24T22:01:03.291 INFO:tasks.rados.rados.0.smithi125.stdout:1860: got expected ENOENT (src dne) 2021-03-24T22:01:03.292 INFO:tasks.rados.rados.0.smithi125.stdout:1860: finishing copy_from racing read to smithi12556470-44 2021-03-24T22:01:03.292 INFO:tasks.rados.rados.0.smithi125.stdout:1858: finishing copy_from racing read to smithi12556470-40 2021-03-24T22:01:03.292 INFO:tasks.rados.rados.0.smithi125.stdout:1859: expect (ObjNum 520 snap 154 seq_num 520) 2021-03-24T22:01:03.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:03 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:01:03.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:03 smithi142 conmon[20125]: 24T22:01:02.262977+0000 mon.a (mon.0) 1440 : cluster [DBG] osdmap e492: 8 total, 8 up, 6 in 2021-03-24T22:01:03.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:03 smithi142 conmon[20125]: cluster 2021-03-24T22:01:03.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:03 smithi142 conmon[20125]: 2021-03-24T22: 2021-03-24T22:01:03.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:03 smithi142 conmon[20125]: 01:02.268735+0000 2021-03-24T22:01:03.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:03 smithi142 conmon[20125]: osd.3 (osd. 2021-03-24T22:01:03.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:03 smithi142 conmon[20125]: 3) 105 : cluster 2021-03-24T22:01:03.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:03 smithi142 conmon[20125]: [DBG] 4.c starting backfill to osd.7 from (0'0,0'0] MAX to 69'15 2021-03-24T22:01:03.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:03 smithi142 conmon[20125]: cluster 2021-03-24T22:01:03.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:03 smithi142 conmon[20125]: 2021-03-24T22 2021-03-24T22:01:03.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:03 smithi142 conmon[20125]: :01:02.268828 2021-03-24T22:01:03.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:03 smithi142 conmon[20125]: +0000 osd.3 (osd. 2021-03-24T22:01:03.373 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:03 smithi142 conmon[20125]: 3) 106 : cluster 2021-03-24T22:01:03.373 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:03 smithi142 conmon[20125]: [DBG] 4.f starting backfill to osd.7 from (0'0,0'0] MAX to 274'26 2021-03-24T22:01:03.374 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:03 smithi142 conmon[20125]: cluster 2021-03-24T22:01:03.374 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:03 smithi142 conmon[20125]: 2021-03-24 2021-03-24T22:01:03.374 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:03 smithi142 conmon[20125]: T22:01:02 2021-03-24T22:01:03.375 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:03 smithi142 conmon[20125]: .271572+0000 osd.5 ( 2021-03-24T22:01:03.375 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:03 smithi142 conmon[20125]: osd.5) 10 : 2021-03-24T22:01:03.376 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:03 smithi142 conmon[20125]: cluster [DBG] 2021-03-24T22:01:03.376 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:03 smithi142 conmon[20125]: 4.8 starting backfill to osd.7 from (0'0,0'0] MAX to 97'28 2021-03-24T22:01:03.377 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:03 smithi142 conmon[20125]: cluster 2021-03-24T22:01:03.377 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:03 smithi142 conmon[20125]: 2021-03-24T22 2021-03-24T22:01:03.377 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:03 smithi142 conmon[20125]: :01:02.275269 2021-03-24T22:01:03.378 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:03 smithi142 conmon[20125]: +0000 osd.5 (osd. 2021-03-24T22:01:03.378 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:03 smithi142 conmon[20125]: 5) 11 : cluster 2021-03-24T22:01:03.379 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:03 smithi142 conmon[20125]: [DBG] 4.a starting backfill to osd.7 from (0'0,0'0] MAX to 69'23 2021-03-24T22:01:03.379 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:03 smithi142 conmon[20125]: cluster 2021-03-24T22:01:03.380 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:03 smithi142 conmon[20125]: 2021-03-24T22 2021-03-24T22:01:03.380 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:03 smithi142 conmon[20125]: :01:02.277126 2021-03-24T22:01:03.380 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:03 smithi142 conmon[20125]: +0000 osd.5 (osd 2021-03-24T22:01:03.381 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:03 smithi142 conmon[20125]: .5) 12 2021-03-24T22:01:03.381 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:03 smithi142 conmon[20125]: : cluster [DBG] 2021-03-24T22:01:03.382 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:03 smithi142 conmon[20125]: 4.7 starting backfill to osd.7 from (0'0,0'0] MAX to 89'22 2021-03-24T22:01:03.382 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:03 smithi142 conmon[20125]: cluster 2021-03-24T22:01:03.383 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:03 smithi142 conmon[20125]: 2021-03-24T22 2021-03-24T22:01:03.383 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:03 smithi142 conmon[20125]: :01:02.277345 2021-03-24T22:01:03.383 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:03 smithi142 conmon[20125]: +0000 osd.0 (osd. 2021-03-24T22:01:03.384 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:03 smithi142 conmon[20125]: 0) 103 : cluster 2021-03-24T22:01:03.384 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:03 smithi142 conmon[20125]: [DBG] 4.6 starting backfill to osd.7 from (0'0,0'0] MAX to 69'16 2021-03-24T22:01:03.385 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:03 smithi142 conmon[20125]: cluster 2021-03-24T22:01:03.385 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:03 smithi142 conmon[20125]: 2021-03-24T22 2021-03-24T22:01:03.386 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:03 smithi142 conmon[20125]: :01:02.303316+0000 osd.5 2021-03-24T22:01:03.386 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:03 smithi142 conmon[20125]: (osd.5) 13 : cluster 2021-03-24T22:01:03.386 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:03 smithi142 conmon[20125]: [DBG] purged_snaps scrub starts 2021-03-24T22:01:03.387 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:03 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:01:03.387 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:03 smithi142 conmon[20125]: -03-24T22 2021-03-24T22:01:03.388 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:03 smithi142 conmon[20125]: :01:02. 2021-03-24T22:01:03.388 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:03 smithi142 conmon[20125]: 305318+0000 osd.5 2021-03-24T22:01:03.388 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:03 smithi142 conmon[20125]: (osd.5) 2021-03-24T22:01:03.389 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:03 smithi142 conmon[20125]: 14 : cluster [DBG] 2021-03-24T22:01:03.389 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:03 smithi142 conmon[20125]: purged_snaps scrub ok 2021-03-24T22:01:03.390 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:03.409 INFO:tasks.rados.rados.0.smithi125.stdout:1861: finishing copy_from to smithi12556470-12 2021-03-24T22:01:03.410 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 12 v 439 (ObjNum 618 snap 187 seq_num 618) dirty exists 2021-03-24T22:01:03.410 INFO:tasks.rados.rados.0.smithi125.stdout:1861: finishing copy_from racing read to smithi12556470-12 2021-03-24T22:01:03.436 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:03.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:03.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:01:03.542 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:03.599 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:03.600 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:03 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:01:03.600 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:03 smithi125 conmon[24927]: 01:02.262977+0000 mon.a (mon.0) 1440 : cluster [DBG] osdmap e492: 8 total, 8 up, 6 in 2021-03-24T22:01:03.601 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:03 smithi125 conmon[24927]: cluster 2021-03-24T22:01:03.601 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:03 smithi125 conmon[24927]: 2021-03-24T22:01:02. 2021-03-24T22:01:03.602 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:03 smithi125 conmon[24927]: 268735+0000 osd.3 (osd.3) 105 2021-03-24T22:01:03.602 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:03 smithi125 conmon[24927]: : cluster [DBG] 4.c starting backfill to osd.7 from (0'0,0'0] MAX to 69'15 2021-03-24T22:01:03.603 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:03 smithi125 conmon[24927]: cluster 2021-03-24T22:01:02 2021-03-24T22:01:03.603 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:03 smithi125 conmon[24927]: .268828+0000 osd.3 (osd.3) 106 : cluster [DBG] 4.f starting backfill to osd.7 from (0'0,0'0] MAX to 274'26 2021-03-24T22:01:03.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:03 smithi125 conmon[24927]: cluster 2021-03-24T22:01: 2021-03-24T22:01:03.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:03 smithi125 conmon[24927]: 02.271572+0000 osd.5 (osd.5) 10 : cluster [DBG] 4.8 starting backfill to osd.7 from (0'0,0'0] MAX to 97'28 2021-03-24T22:01:03.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:03 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:01:03.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:03 smithi125 conmon[24927]: -03-24T22:01:02.275269+0000 osd.5 (osd.5) 11 : cluster [DBG] 4.a starting backfill to osd.7 from (0'0,0'0] MAX to 69'23 2021-03-24T22:01:03.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:03 smithi125 conmon[24927]: cluster 2021-03-24T22:01 2021-03-24T22:01:03.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:03 smithi125 conmon[24927]: :02.277126+0000 osd.5 (osd.5) 12 : cluster [DBG] 4.7 starting backfill to osd.7 from (0'0,0'0] MAX to 89'22 2021-03-24T22:01:03.606 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:03 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T22:01:03.606 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:03 smithi125 conmon[24927]: 22:01:02.277345+0000 osd.0 (osd.0) 103 : cluster [DBG] 2021-03-24T22:01:03.606 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:03 smithi125 conmon[24927]: 4.6 starting backfill to osd.7 from (0'0,0'0] MAX to 69'16 2021-03-24T22:01:03.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:03 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:01:03.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:03 smithi125 conmon[24927]: 24T22:01:02.303316+0000 osd.5 (osd.5) 13 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:01:03.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:03 smithi125 conmon[24927]: cluster 2021-03-24T22:01:02 2021-03-24T22:01:03.608 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:03 smithi125 conmon[24927]: .305318+0000 osd.5 (osd.5) 14 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:01:03.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:03.610 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:03 smithi125 conmon[19936]: cluster 2021-03-24T22:01 2021-03-24T22:01:03.610 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:03 smithi125 conmon[19936]: :02.262977+0000 mon.a (mon.0) 1440 : cluster [DBG] osdmap e492: 8 total, 8 up, 6 in 2021-03-24T22:01:03.610 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:03 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:01:03.611 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:03 smithi125 conmon[19936]: 24T22:01:02.268735+0000 osd.3 (osd.3) 105 : cluster [DBG] 2021-03-24T22:01:03.611 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:03 smithi125 conmon[19936]: 4.c starting backfill to osd.7 from (0'0,0'0] MAX to 69'15 2021-03-24T22:01:03.611 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:03 smithi125 conmon[19936]: cluster 2021-03-24T22:01:02. 2021-03-24T22:01:03.611 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:03 smithi125 conmon[19936]: 268828+0000 osd.3 (osd.3) 106 : cluster 2021-03-24T22:01:03.613 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:03 smithi125 conmon[19936]: [DBG] 4.f starting backfill to osd.7 from (0'0,0'0] MAX to 274'26 2021-03-24T22:01:03.613 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:03 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:01:03.614 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:03 smithi125 conmon[19936]: 01:02.271572+0000 osd.5 (osd.5) 2021-03-24T22:01:03.614 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:03 smithi125 conmon[19936]: 10 : cluster [DBG] 4.8 starting backfill to osd.7 from (0'0,0'0] MAX to 97'28 2021-03-24T22:01:03.615 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:03 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:01:03.615 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:03 smithi125 conmon[19936]: 24T22:01:02.275269+0000 osd.5 ( 2021-03-24T22:01:03.615 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:03 smithi125 conmon[19936]: osd.5) 11 : cluster [DBG] 4.a starting backfill to osd.7 from (0'0,0'0] MAX to 69'23 2021-03-24T22:01:03.616 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:03 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:01:03.616 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:03 smithi125 conmon[19936]: 03-24T22:01:02.277126+0000 2021-03-24T22:01:03.617 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:03 smithi125 conmon[19936]: osd.5 (osd.5) 12 : cluster [DBG] 4.7 starting backfill to osd.7 from (0'0,0'0] MAX to 89'22 2021-03-24T22:01:03.617 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:03 smithi125 conmon[19936]: cluster 2021-03-24T22:01:03.618 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:03 smithi125 conmon[19936]: 2021- 2021-03-24T22:01:03.618 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:03 smithi125 conmon[19936]: 03-24T22:01:02.277345+0000 osd.0 (osd. 2021-03-24T22:01:03.618 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:03 smithi125 conmon[19936]: 0) 103 : cluster [DBG] 4.6 starting backfill to osd.7 from (0'0,0'0] MAX to 69'16 2021-03-24T22:01:03.619 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:03 smithi125 conmon[19936]: cluster 2021-03-24T22:01:03.619 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:03 smithi125 conmon[19936]: 2021-03-24T22:01:03.620 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:03 smithi125 conmon[19936]: 2021-03-24 2021-03-24T22:01:03.621 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:03 smithi125 conmon[19936]: T22:01:02.303316+0000 osd.5 (osd.5) 13 2021-03-24T22:01:03.621 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:03 smithi125 conmon[19936]: : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:01:03.621 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:03 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:01:03.622 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:03 smithi125 conmon[19936]: T22:01:02.305318+0000 osd.5 (osd. 2021-03-24T22:01:03.622 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:03 smithi125 conmon[19936]: 5) 14 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:01:03.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:01:03.726 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:03.780 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:03.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:03.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:01:03.887 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:03.938 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:03.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:03.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:01:04.043 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:04.096 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:04.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:04.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:01:04.200 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:04.256 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:04.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:04.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:01:04.292 INFO:tasks.rados.rados.0.smithi125.stdout:1858: done (4 left) 2021-03-24T22:01:04.293 INFO:tasks.rados.rados.0.smithi125.stdout:1859: done (3 left) 2021-03-24T22:01:04.293 INFO:tasks.rados.rados.0.smithi125.stdout:1860: done (2 left) 2021-03-24T22:01:04.294 INFO:tasks.rados.rados.0.smithi125.stdout:1861: done (1 left) 2021-03-24T22:01:04.294 INFO:tasks.rados.rados.0.smithi125.stdout:1863: done (0 left) 2021-03-24T22:01:04.294 INFO:tasks.rados.rados.0.smithi125.stdout:1864: read oid 38 snap 179 2021-03-24T22:01:04.294 INFO:tasks.rados.rados.0.smithi125.stdout:1864: expect (ObjNum 489 snap 140 seq_num 489) 2021-03-24T22:01:04.295 INFO:tasks.rados.rados.0.smithi125.stdout:1865: snap_remove snap 184 2021-03-24T22:01:04.352 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:04 smithi125 conmon[19936]: cluster 2021-03-24T22:01:04.353 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:04 smithi125 conmon[19936]: 2021-03-24 2021-03-24T22:01:04.353 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:04 smithi125 conmon[19936]: T22:01:02.873617+0000 mgr.y (mgr.14144) 518 : cluster [DBG] pgmap v901: 168 pgs: 1 active+recovering+degraded, 1 active+recovery_wait+degraded, 8 remapped+peering, 158 active+clean; 469 MiB data, 1.1 GiB used, 446 GiB / 447 GiB avail; 4.4 MiB/s rd, 934 KiB/s wr, 8 op/s; 8/1623 objects degraded (0.493%); 3/1623 objects misplaced (0.185%); 238 B/s, 3 objects/s recovering 2021-03-24T22:01:04.353 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:04 smithi125 conmon[19936]: cluster 2021-03-24T22:01 2021-03-24T22:01:04.353 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:04 smithi125 conmon[19936]: :03.272649+0000 mon.a (mon.0) 1441 : cluster [WRN] 2021-03-24T22:01:04.354 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:04 smithi125 conmon[19936]: Health check failed: Degraded data redundancy: 8/1623 objects degraded (0.493%), 2 pgs degraded (PG_DEGRADED) 2021-03-24T22:01:04.354 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:04 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:01:04.355 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:04 smithi125 conmon[19936]: 01:03.286734+0000 mon.a (mon. 2021-03-24T22:01:04.355 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:04 smithi125 conmon[19936]: 0) 1442 : cluster [DBG] osdmap e493: 8 total, 8 up, 6 in 2021-03-24T22:01:04.355 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:04 smithi125 conmon[24927]: cluster 2021-03-24T22:01:04.356 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:04 smithi125 conmon[24927]: 2021-03-24T22: 2021-03-24T22:01:04.356 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:04 smithi125 conmon[24927]: 01:02.873617+0000 mgr.y (mgr.14144) 518 : cluster [DBG] pgmap v901: 168 pgs: 1 active+recovering+degraded, 1 active+recovery_wait+degraded, 8 remapped+peering, 158 active+clean; 469 MiB data, 1.1 GiB used, 446 GiB / 447 GiB avail; 4.4 MiB/s rd, 934 KiB/s wr, 8 op/s; 8/1623 objects degraded (0.493%); 3/1623 objects misplaced (0.185%); 238 B/s, 3 objects/s recovering 2021-03-24T22:01:04.356 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:04 smithi125 conmon[24927]: cluster 2021-03-24T22 2021-03-24T22:01:04.357 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:04 smithi125 conmon[24927]: :01:03.272649+0000 mon.a (mon.0) 2021-03-24T22:01:04.357 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:04 smithi125 conmon[24927]: 1441 : cluster [WRN] Health check failed: Degraded data redundancy: 8/1623 objects degraded (0.493%), 2 pgs degraded (PG_DEGRADED) 2021-03-24T22:01:04.357 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:04 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:01:04.357 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:04 smithi125 conmon[24927]: 01:03.286734+0000 mon.a (mon.0) 2021-03-24T22:01:04.358 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:04 smithi125 conmon[24927]: 1442 : cluster [DBG] osdmap e493: 8 total, 8 up, 6 in 2021-03-24T22:01:04.361 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:04.417 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:04.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:04.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:01:04.523 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:04.579 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:04.583 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:04 smithi142 conmon[20125]: cluster 2021-03-24T22:01:02.873617+0000 mgr.y 2021-03-24T22:01:04.584 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:04 smithi142 conmon[20125]: (mgr.14144) 518 : cluster [DBG] pgmap v901: 168 pgs: 1 active+recovering+degraded, 1 active+recovery_wait+degraded, 8 remapped+peering, 158 active+clean; 469 MiB data, 1.1 GiB used, 446 GiB / 447 GiB avail; 4.4 MiB/s rd, 934 KiB/s wr, 8 op/s; 8/1623 objects degraded (0.493%); 3/1623 objects misplaced (0.185%); 238 B/s, 3 objects/s recovering 2021-03-24T22:01:04.584 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:04 smithi142 conmon[20125]: cluster 2021-03-24T22:01:03.272649+0000 mon.a (mon.0) 1441 : cluster [WRN] Health check failed: Degraded data redundancy: 8/1623 objects degraded (0.493%), 2 pgs degraded (PG_DEGRADED) 2021-03-24T22:01:04.584 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:04 smithi142 conmon[20125]: cluster 2021-03-24T22:01:03.286734+0000 mon.a (mon.0) 2021-03-24T22:01:04.584 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:04 smithi142 conmon[20125]: 1442 : cluster [DBG] osdmap e493: 8 total, 8 up, 6 in 2021-03-24T22:01:04.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:04.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:01:04.688 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:04.734 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:04.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:04.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:01:04.839 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:04.882 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:04.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:04.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:01:04.890 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T22:01:04.910 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T22:01:04.929 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T22:01:04.948 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T22:01:04.972 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T22:01:04.989 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:04.995 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T22:01:05.021 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T22:01:05.036 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:05.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:05.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:01:05.049 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T22:01:05.051 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-24T22:01:05.051 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-24T22:01:05.052 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-24T22:01:05.052 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-24T22:01:05.052 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-24T22:01:05.052 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-24T22:01:05.052 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-24T22:01:05.053 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-24T22:01:05.095 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-24T22:01:05.095 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T22:01:05.142 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:05.192 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:05.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:05.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:01:05.293 INFO:tasks.rados.rados.0.smithi125.stdout:1865: done (1 left) 2021-03-24T22:01:05.293 INFO:tasks.rados.rados.0.smithi125.stdout:1866: delete oid 14 current snap is 203 2021-03-24T22:01:05.298 INFO:tasks.rados.rados.0.smithi125.stdout:1864: expect (ObjNum 489 snap 140 seq_num 489) 2021-03-24T22:01:05.300 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:05.386 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:05.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:05.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:01:05.405 INFO:tasks.rados.rados.0.smithi125.stdout:1864: done (1 left) 2021-03-24T22:01:05.405 INFO:tasks.rados.rados.0.smithi125.stdout:1866: done (0 left) 2021-03-24T22:01:05.406 INFO:tasks.rados.rados.0.smithi125.stdout:1867: rollback oid 46 current snap is 203 2021-03-24T22:01:05.406 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 46 to 135 2021-03-24T22:01:05.407 INFO:tasks.rados.rados.0.smithi125.stdout:1868: snap_remove snap 188 2021-03-24T22:01:05.495 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:05.512 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[19936]: cluster 2021-03-24T22:01:05.513 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[19936]: 2021-03-24T22:01:04.288442+0000 mon.a (mon.0) 1443 : cluster [DBG] osdmap e494: 8 total, 8 up, 6 in 2021-03-24T22:01:05.513 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[19936]: audit 2021-03-24T22 2021-03-24T22:01:05.513 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[19936]: :01:04.797370+0000 mon.a (mon 2021-03-24T22:01:05.514 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[19936]: .0) 1444 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T22:01:05.514 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[19936]: audit 2021- 2021-03-24T22:01:05.514 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[19936]: 03-24T22:01:04.797635 2021-03-24T22:01:05.515 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[19936]: +0000 mon.a (mon.0) 1445 : audit 2021-03-24T22:01:05.515 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[19936]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T22:01:05.515 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[19936]: audit 2021-03-24T22:01 2021-03-24T22:01:05.515 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[19936]: :04.861637+0000 mon.c (mon. 2021-03-24T22:01:05.515 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[19936]: 2) 77 : audit [DBG] from='client.? v1:172.21.15.125:0/1395646518' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:01:05.516 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T22:01:05.516 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[19936]: -24T22:01:05.290598+0000 2021-03-24T22:01:05.516 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[19936]: mon.a (mon.0) 1446 : cluster [DBG] 2021-03-24T22:01:05.517 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[19936]: osdmap e495: 8 total, 8 up, 6 in 2021-03-24T22:01:05.517 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: cluster 2021-03-24T22:01:05.518 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 2021-03-24T22:01:05.518 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 2021 2021-03-24T22:01:05.518 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: - 2021-03-24T22:01:05.519 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 03 2021-03-24T22:01:05.519 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: - 2021-03-24T22:01:05.519 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 24 2021-03-24T22:01:05.519 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: T 2021-03-24T22:01:05.519 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 22 2021-03-24T22:01:05.520 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: : 2021-03-24T22:01:05.520 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 01 2021-03-24T22:01:05.520 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: : 2021-03-24T22:01:05.520 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 04 2021-03-24T22:01:05.521 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: . 2021-03-24T22:01:05.521 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 288442 2021-03-24T22:01:05.521 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: +0000 2021-03-24T22:01:05.521 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 2021-03-24T22:01:05.521 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: mon.a 2021-03-24T22:01:05.522 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: ( 2021-03-24T22:01:05.522 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: mon 2021-03-24T22:01:05.522 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: . 2021-03-24T22:01:05.522 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 0 2021-03-24T22:01:05.523 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: ) 2021-03-24T22:01:05.523 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 1443 2021-03-24T22:01:05.523 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: : 2021-03-24T22:01:05.523 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: cluster 2021-03-24T22:01:05.524 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 2021-03-24T22:01:05.524 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: [DBG] 2021-03-24T22:01:05.524 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 2021-03-24T22:01:05.524 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: osdmap e494: 8 total, 8 up, 6 in 2021-03-24T22:01:05.525 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 2021-03-24T22:01:05.525 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: audit 2021-03-24T22:01:05.525 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 2021-03-24T22:01:05.525 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 2021 2021-03-24T22:01:05.526 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: - 2021-03-24T22:01:05.526 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 03 2021-03-24T22:01:05.526 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: - 2021-03-24T22:01:05.526 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 24 2021-03-24T22:01:05.527 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: T 2021-03-24T22:01:05.527 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 22 2021-03-24T22:01:05.527 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: : 2021-03-24T22:01:05.527 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 01 2021-03-24T22:01:05.527 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: : 2021-03-24T22:01:05.528 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 04 2021-03-24T22:01:05.528 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: . 2021-03-24T22:01:05.528 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 797370 2021-03-24T22:01:05.529 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: +0000 2021-03-24T22:01:05.529 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 2021-03-24T22:01:05.529 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: mon.a 2021-03-24T22:01:05.529 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: ( 2021-03-24T22:01:05.529 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: mon 2021-03-24T22:01:05.530 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: . 2021-03-24T22:01:05.530 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 0 2021-03-24T22:01:05.530 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: ) 2021-03-24T22:01:05.531 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 1444 2021-03-24T22:01:05.531 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: : 2021-03-24T22:01:05.531 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: audit 2021-03-24T22:01:05.532 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 2021-03-24T22:01:05.532 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: [INF] 2021-03-24T22:01:05.532 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 2021-03-24T22:01:05.533 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T22:01:05.533 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 2021-03-24T22:01:05.533 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: audit 2021-03-24T22:01:05.533 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 2021-03-24T22:01:05.534 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 2021 2021-03-24T22:01:05.534 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: - 2021-03-24T22:01:05.534 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 03 2021-03-24T22:01:05.534 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: - 2021-03-24T22:01:05.535 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 24 2021-03-24T22:01:05.535 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: T 2021-03-24T22:01:05.535 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 22 2021-03-24T22:01:05.535 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: : 2021-03-24T22:01:05.535 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 01 2021-03-24T22:01:05.536 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: : 2021-03-24T22:01:05.536 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 04 2021-03-24T22:01:05.536 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: . 2021-03-24T22:01:05.536 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 797635 2021-03-24T22:01:05.537 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: +0000 2021-03-24T22:01:05.537 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 2021-03-24T22:01:05.537 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: mon.a 2021-03-24T22:01:05.537 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: ( 2021-03-24T22:01:05.537 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: mon 2021-03-24T22:01:05.538 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: . 2021-03-24T22:01:05.538 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 0 2021-03-24T22:01:05.538 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: ) 2021-03-24T22:01:05.538 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 1445 2021-03-24T22:01:05.539 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: : 2021-03-24T22:01:05.539 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: audit 2021-03-24T22:01:05.539 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 2021-03-24T22:01:05.539 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: [INF] 2021-03-24T22:01:05.540 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 2021-03-24T22:01:05.540 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T22:01:05.540 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 2021-03-24T22:01:05.540 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: audit 2021-03-24T22:01:05.540 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 2021-03-24T22:01:05.541 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 2021 2021-03-24T22:01:05.541 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: - 2021-03-24T22:01:05.541 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 03 2021-03-24T22:01:05.541 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: - 2021-03-24T22:01:05.542 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 24 2021-03-24T22:01:05.542 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: T 2021-03-24T22:01:05.542 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 22 2021-03-24T22:01:05.542 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: : 2021-03-24T22:01:05.543 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 01 2021-03-24T22:01:05.543 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: : 2021-03-24T22:01:05.543 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 04 2021-03-24T22:01:05.543 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: . 2021-03-24T22:01:05.543 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 861637 2021-03-24T22:01:05.544 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: +0000 2021-03-24T22:01:05.544 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 2021-03-24T22:01:05.544 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: mon.c 2021-03-24T22:01:05.544 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: ( 2021-03-24T22:01:05.545 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: mon 2021-03-24T22:01:05.545 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: . 2021-03-24T22:01:05.545 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 2 2021-03-24T22:01:05.545 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: ) 2021-03-24T22:01:05.546 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 77 2021-03-24T22:01:05.546 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: : 2021-03-24T22:01:05.546 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: audit 2021-03-24T22:01:05.546 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 2021-03-24T22:01:05.546 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: [DBG] 2021-03-24T22:01:05.547 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 2021-03-24T22:01:05.547 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: from='client.? v1:172.21.15.125:0/1395646518' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:01:05.547 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 2021-03-24T22:01:05.547 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: cluster 2021-03-24T22:01:05.548 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 2021-03-24T22:01:05.548 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 2021 2021-03-24T22:01:05.548 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: - 2021-03-24T22:01:05.548 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 03 2021-03-24T22:01:05.549 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: - 2021-03-24T22:01:05.549 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 24 2021-03-24T22:01:05.549 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: T 2021-03-24T22:01:05.549 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 22 2021-03-24T22:01:05.549 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: : 2021-03-24T22:01:05.550 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 01 2021-03-24T22:01:05.550 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: : 2021-03-24T22:01:05.550 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 05 2021-03-24T22:01:05.550 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: . 2021-03-24T22:01:05.551 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 290598 2021-03-24T22:01:05.551 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: +0000 2021-03-24T22:01:05.551 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 2021-03-24T22:01:05.551 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: mon.a 2021-03-24T22:01:05.552 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: ( 2021-03-24T22:01:05.552 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: mon 2021-03-24T22:01:05.553 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: . 2021-03-24T22:01:05.553 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 0 2021-03-24T22:01:05.553 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: ) 2021-03-24T22:01:05.554 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 1446 2021-03-24T22:01:05.554 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: : 2021-03-24T22:01:05.555 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: cluster 2021-03-24T22:01:05.555 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 2021-03-24T22:01:05.556 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: [DBG] 2021-03-24T22:01:05.556 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 2021-03-24T22:01:05.556 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: osdmap e495: 8 total, 8 up, 6 in 2021-03-24T22:01:05.557 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:05 smithi125 conmon[24927]: 2021-03-24T22:01:05.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:05.560 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:05.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:01:05.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:05 smithi142 conmon[20125]: cluster 2021-03-24T22:01:04.288442+0000 mon.a (mon. 2021-03-24T22:01:05.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:05 smithi142 conmon[20125]: 0) 1443 : cluster [DBG] osdmap e494: 8 total, 8 up, 6 in 2021-03-24T22:01:05.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:05 smithi142 conmon[20125]: audit 2021-03-24T22:01:04.797370+0000 mon.a (mon.0) 1444 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T22:01:05.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:05 smithi142 conmon[20125]: audit 2021-03-24T22:01:04.797635 2021-03-24T22:01:05.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:05 smithi142 conmon[20125]: +0000 mon.a (mon.0) 1445 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T22:01:05.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:05 smithi142 conmon[20125]: audit 2021 2021-03-24T22:01:05.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:05 smithi142 conmon[20125]: -03-24T22:01:04.861637+0000 mon.c (mon.2) 77 : audit [DBG] from='client.? v1:172.21.15.125:0/1395646518' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:01:05.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:05 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:01:05.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:05 smithi142 conmon[20125]: 03-24T22:01:05.290598+0000 mon.a (mon.0) 1446 : cluster 2021-03-24T22:01:05.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:05 smithi142 conmon[20125]: [DBG] osdmap e495: 8 total, 8 up, 6 in 2021-03-24T22:01:05.662 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:05.710 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:05.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:05.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:01:05.815 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:05.874 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:05.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:05.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:01:05.982 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:06.031 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:06.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:06.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:01:06.136 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:06.198 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:06.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:06.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:01:06.305 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:06.316 INFO:tasks.rados.rados.0.smithi125.stdout:1868: done (1 left) 2021-03-24T22:01:06.317 INFO:tasks.rados.rados.0.smithi125.stdout:1869: read oid 6 snap -1 2021-03-24T22:01:06.317 INFO:tasks.rados.rados.0.smithi125.stdout:1869: expect deleted 2021-03-24T22:01:06.317 INFO:tasks.rados.rados.0.smithi125.stdout:1870: read oid 5 snap -1 2021-03-24T22:01:06.321 INFO:tasks.rados.rados.0.smithi125.stdout:1870: expect (ObjNum 652 snap 198 seq_num 652) 2021-03-24T22:01:06.321 INFO:tasks.rados.rados.0.smithi125.stdout:1871: snap_create 2021-03-24T22:01:06.321 INFO:tasks.rados.rados.0.smithi125.stdout:1867: finishing rollback tid 0 to smithi12556470-46 2021-03-24T22:01:06.321 INFO:tasks.rados.rados.0.smithi125.stdout:1867: finishing rollback tid 1 to smithi12556470-46 2021-03-24T22:01:06.322 INFO:tasks.rados.rados.0.smithi125.stdout:1867: finishing rollback tid 2 to smithi12556470-46 2021-03-24T22:01:06.322 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 46 v 491 (ObjNum 471 snap 133 seq_num 471) dirty exists 2021-03-24T22:01:06.338 INFO:tasks.rados.rados.0.smithi125.stdout:1870: expect (ObjNum 652 snap 198 seq_num 652) 2021-03-24T22:01:06.353 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd set noscrub 2021-03-24T22:01:06.360 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:06.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:06.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:01:06.467 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:06.520 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:06.523 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:06 smithi125 conmon[19936]: cluster 2021-03-24T22:01:04.874278+0000 mgr.y 2021-03-24T22:01:06.524 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:06 smithi125 conmon[19936]: (mgr.14144) 519 : cluster [DBG] pgmap v904: 168 pgs: 1 peering, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 1 active+recovery_wait+remapped, 1 active+recovering+degraded, 2 active+recovery_wait+degraded, 8 remapped+peering, 140 active+clean; 480 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 3.0 MiB/s rd, 2.2 MiB/s wr, 12 op/s; 233/1638 objects degraded (14.225%); 126/1638 objects misplaced (7.692%); 1.2 KiB/s, 22 objects/s recovering 2021-03-24T22:01:06.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:06.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:01:06.526 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:06 smithi125 conmon[24927]: cluster 2021-03-24T22:01:04 2021-03-24T22:01:06.526 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:06 smithi125 conmon[24927]: .874278+0000 mgr.y (mgr.14144) 519 : cluster [DBG] pgmap v904: 168 pgs: 1 peering, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 1 active+recovery_wait+remapped, 1 active+recovering+degraded, 2 active+recovery_wait+degraded, 8 remapped+peering, 140 active+clean; 480 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 3.0 MiB/s rd, 2.2 MiB/s wr, 12 op/s; 233/1638 objects degraded (14.225%); 126/1638 objects misplaced (7.692%); 1.2 KiB/s, 22 objects/s recovering 2021-03-24T22:01:06.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:06 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:01:06.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:06 smithi142 conmon[20125]: T22:01:04.874278+0000 mgr.y (mgr.14144) 519 : cluster [DBG] pgmap v904: 168 pgs: 1 peering, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 1 active+recovery_wait+remapped, 1 active+recovering+degraded, 2 active+recovery_wait+degraded, 8 remapped+peering, 140 active+clean; 480 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 3.0 MiB/s rd, 2.2 MiB/s wr, 12 op/s; 233/1638 objects degraded (14.225%); 126/1638 objects misplaced (7.692%); 1.2 KiB/s, 22 objects/s recovering 2021-03-24T22:01:06.626 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:06.667 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:06.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:06.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:01:06.772 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:06.821 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:06.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:06.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:01:06.926 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:06.984 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:06.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:06.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:01:07.093 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:07.144 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:07.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:07.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:01:07.252 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:07.314 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:07.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:07.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:01:07.322 INFO:tasks.rados.rados.0.smithi125.stdout:1867: done (3 left) 2021-03-24T22:01:07.322 INFO:tasks.rados.rados.0.smithi125.stdout:1870: done (2 left) 2021-03-24T22:01:07.323 INFO:tasks.rados.rados.0.smithi125.stdout:1871: done (1 left) 2021-03-24T22:01:07.323 INFO:tasks.rados.rados.0.smithi125.stdout:1872: delete oid 36 current snap is 204 2021-03-24T22:01:07.337 INFO:tasks.rados.rados.0.smithi125.stdout:1869: done (1 left) 2021-03-24T22:01:07.338 INFO:tasks.rados.rados.0.smithi125.stdout:1872: done (0 left) 2021-03-24T22:01:07.338 INFO:tasks.rados.rados.0.smithi125.stdout:1873: write oid 31 current snap is 204 2021-03-24T22:01:07.339 INFO:tasks.rados.rados.0.smithi125.stdout:1873: seq_num 663 ranges {743041=412798,1930738=616566,3040503=102538} 2021-03-24T22:01:07.344 INFO:tasks.rados.rados.0.smithi125.stdout:1873: writing smithi12556470-31 from 743041 to 1155839 tid 1 2021-03-24T22:01:07.355 INFO:tasks.rados.rados.0.smithi125.stdout:1873: writing smithi12556470-31 from 1930738 to 2547304 tid 2 2021-03-24T22:01:07.358 INFO:tasks.rados.rados.0.smithi125.stdout:1873: writing smithi12556470-31 from 3040503 to 3143041 tid 3 2021-03-24T22:01:07.359 INFO:tasks.rados.rados.0.smithi125.stdout:1874: write oid 7 current snap is 204 2021-03-24T22:01:07.359 INFO:tasks.rados.rados.0.smithi125.stdout:1874: seq_num 664 ranges {759848=488047,1989220=760224} 2021-03-24T22:01:07.367 INFO:tasks.rados.rados.0.smithi125.stdout:1874: writing smithi12556470-7 from 759848 to 1247895 tid 1 2021-03-24T22:01:07.370 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[19936]: cluster 2021-03-24T22:01:07.370 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[19936]: 2021-03-24T22:01:07.370 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[19936]: 2021 2021-03-24T22:01:07.370 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[19936]: - 2021-03-24T22:01:07.371 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[19936]: 03 2021-03-24T22:01:07.371 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[19936]: - 2021-03-24T22:01:07.371 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[19936]: 24 2021-03-24T22:01:07.371 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[19936]: T 2021-03-24T22:01:07.372 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[19936]: 22 2021-03-24T22:01:07.372 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[19936]: : 2021-03-24T22:01:07.372 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[19936]: 01 2021-03-24T22:01:07.372 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[19936]: : 2021-03-24T22:01:07.372 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[19936]: 06 2021-03-24T22:01:07.373 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[19936]: . 2021-03-24T22:01:07.373 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[19936]: 312600 2021-03-24T22:01:07.373 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[19936]: +0000 2021-03-24T22:01:07.373 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[19936]: 2021-03-24T22:01:07.374 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[19936]: mon.a 2021-03-24T22:01:07.374 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[19936]: ( 2021-03-24T22:01:07.374 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[19936]: mon 2021-03-24T22:01:07.375 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[19936]: . 2021-03-24T22:01:07.375 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[19936]: 0 2021-03-24T22:01:07.375 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[19936]: ) 2021-03-24T22:01:07.376 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[19936]: 1447 2021-03-24T22:01:07.376 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[19936]: : 2021-03-24T22:01:07.376 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[19936]: cluster 2021-03-24T22:01:07.376 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[19936]: 2021-03-24T22:01:07.376 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[19936]: [DBG] 2021-03-24T22:01:07.377 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[19936]: 2021-03-24T22:01:07.377 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[19936]: osdmap e496: 8 total, 8 up, 6 in 2021-03-24T22:01:07.377 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[19936]: 2021-03-24T22:01:07.378 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[24927]: cluster 2021-03-24T22:01:07.378 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[24927]: 2021-03-24T22:01:07.378 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[24927]: 2021 2021-03-24T22:01:07.378 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[24927]: - 2021-03-24T22:01:07.379 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[24927]: 03 2021-03-24T22:01:07.379 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[24927]: - 2021-03-24T22:01:07.379 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[24927]: 24 2021-03-24T22:01:07.379 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[24927]: T 2021-03-24T22:01:07.380 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[24927]: 22 2021-03-24T22:01:07.380 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[24927]: : 2021-03-24T22:01:07.380 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[24927]: 01 2021-03-24T22:01:07.380 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[24927]: : 2021-03-24T22:01:07.380 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[24927]: 06 2021-03-24T22:01:07.381 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[24927]: . 2021-03-24T22:01:07.381 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[24927]: 312600 2021-03-24T22:01:07.381 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[24927]: +0000 2021-03-24T22:01:07.381 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[24927]: 2021-03-24T22:01:07.381 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[24927]: mon.a 2021-03-24T22:01:07.382 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[24927]: ( 2021-03-24T22:01:07.382 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[24927]: mon 2021-03-24T22:01:07.382 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[24927]: . 2021-03-24T22:01:07.382 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[24927]: 0 2021-03-24T22:01:07.383 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[24927]: ) 2021-03-24T22:01:07.383 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[24927]: 1447 2021-03-24T22:01:07.383 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[24927]: : 2021-03-24T22:01:07.383 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[24927]: cluster 2021-03-24T22:01:07.383 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[24927]: 2021-03-24T22:01:07.384 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[24927]: [DBG] 2021-03-24T22:01:07.384 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:07 smithi125 conmon[24927]: osdmap e496: 8 total, 8 up, 6 in 2021-03-24T22:01:07.384 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:01:07.385 INFO:tasks.rados.rados.0.smithi125.stdout:1874: writing smithi12556470-7 from 1989220 to 2749444 tid 2 2021-03-24T22:01:07.386 INFO:tasks.rados.rados.0.smithi125.stdout:1875: snap_create 2021-03-24T22:01:07.386 INFO:tasks.rados.rados.0.smithi125.stdout:1873: finishing write tid 1 to smithi12556470-31 2021-03-24T22:01:07.386 INFO:tasks.rados.rados.0.smithi125.stdout:1873: finishing write tid 2 to smithi12556470-31 2021-03-24T22:01:07.386 INFO:tasks.rados.rados.0.smithi125.stdout:1873: finishing write tid 3 to smithi12556470-31 2021-03-24T22:01:07.387 INFO:tasks.rados.rados.0.smithi125.stdout:1873: finishing write tid 4 to smithi12556470-31 2021-03-24T22:01:07.387 INFO:tasks.rados.rados.0.smithi125.stdout:1873: finishing write tid 5 to smithi12556470-31 2021-03-24T22:01:07.387 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 31 v 875 (ObjNum 663 snap 204 seq_num 663) dirty exists 2021-03-24T22:01:07.388 INFO:tasks.rados.rados.0.smithi125.stdout:1873: left oid 31 (ObjNum 663 snap 204 seq_num 663) 2021-03-24T22:01:07.388 INFO:tasks.rados.rados.0.smithi125.stdout:1874: finishing write tid 1 to smithi12556470-7 2021-03-24T22:01:07.394 INFO:tasks.rados.rados.0.smithi125.stdout:1874: finishing write tid 2 to smithi12556470-7 2021-03-24T22:01:07.398 INFO:tasks.rados.rados.0.smithi125.stdout:1874: finishing write tid 3 to smithi12556470-7 2021-03-24T22:01:07.398 INFO:tasks.rados.rados.0.smithi125.stdout:1874: finishing write tid 4 to smithi12556470-7 2021-03-24T22:01:07.398 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 7 v 643 (ObjNum 664 snap 204 seq_num 664) dirty exists 2021-03-24T22:01:07.399 INFO:tasks.rados.rados.0.smithi125.stdout:1874: left oid 7 (ObjNum 664 snap 204 seq_num 664) 2021-03-24T22:01:07.412 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.1 dump_blocked_ops 2021-03-24T22:01:07.422 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:07.473 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:07.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:07.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:01:07.579 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:07.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:07 smithi142 conmon[20125]: cluster 2021-03-24T22:01:06.312600+0000 mon.a (mon. 2021-03-24T22:01:07.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:07 smithi142 conmon[20125]: 0) 1447 : cluster [DBG] osdmap e496: 8 total, 8 up, 6 in 2021-03-24T22:01:07.634 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:07.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:07.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:01:07.708 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 7] out_osds: [6, 1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2021-03-24T22:01:07.708 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-24T22:01:07.709 INFO:tasks.thrashosds.thrasher:Adding osd 1 2021-03-24T22:01:07.709 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd in 1 2021-03-24T22:01:07.740 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:07.798 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:07.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:07.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:01:07.903 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:07.959 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:07.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:07.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:01:08.064 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:08.111 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:08.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:08.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:01:08.216 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:08.266 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:08.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:08.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:01:08.315 INFO:tasks.rados.rados.0.smithi125.stdout:1873: done (2 left) 2021-03-24T22:01:08.316 INFO:tasks.rados.rados.0.smithi125.stdout:1874: done (1 left) 2021-03-24T22:01:08.316 INFO:tasks.rados.rados.0.smithi125.stdout:1875: done (0 left) 2021-03-24T22:01:08.316 INFO:tasks.rados.rados.0.smithi125.stdout:1876: read oid 10 snap 196 2021-03-24T22:01:08.316 INFO:tasks.rados.rados.0.smithi125.stdout:1876: expect (ObjNum 642 snap 196 seq_num 642) 2021-03-24T22:01:08.317 INFO:tasks.rados.rados.0.smithi125.stdout:1877: copy_from oid 11 from oid 1 current snap is 205 2021-03-24T22:01:08.317 INFO:tasks.rados.rados.0.smithi125.stdout:1878: read oid 48 snap -1 2021-03-24T22:01:08.317 INFO:tasks.rados.rados.0.smithi125.stdout:1878: expect (ObjNum 462 snap 129 seq_num 462) 2021-03-24T22:01:08.317 INFO:tasks.rados.rados.0.smithi125.stdout:1879: snap_remove snap 202 2021-03-24T22:01:08.370 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:08.423 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:08.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:08.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:01:08.528 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:08.575 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:08.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:08.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:01:08.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:08 smithi142 conmon[20125]: cluster 2021-03-24T22:01:06. 2021-03-24T22:01:08.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:08 smithi142 conmon[20125]: 874999+0000 mgr.y (mgr.14144) 520 : cluster [DBG] pgmap v907: 168 pgs: 1 peering, 4 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait, 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 1 remapped+peering, 143 active+clean; 480 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 8.2 MiB/s rd, 1.9 MiB/s wr, 15 op/s; 280/1638 objects degraded (17.094%); 194/1638 objects misplaced (11.844%); 2.4 MiB/s, 31 objects/s recovering 2021-03-24T22:01:08.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:08 smithi142 conmon[20125]: cluster 2021-03-24T22:01:07.312858+0000 mon.a (mon. 2021-03-24T22:01:08.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:08 smithi142 conmon[20125]: 0) 1448 : cluster [DBG] osdmap e497: 8 total, 8 up, 6 in 2021-03-24T22:01:08.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:08 smithi142 conmon[20125]: cluster 2021-03-24T22:01:08.313209+0000 mon.a (mon.0) 1449 : cluster [DBG] 2021-03-24T22:01:08.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:08 smithi142 conmon[20125]: osdmap e498: 8 total, 8 up, 6 in 2021-03-24T22:01:08.680 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:08.730 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:08.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:08.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:01:08.837 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:08.870 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:08 smithi125 conmon[19936]: cluster 2021-03-24T22:01:06.874999+0000 mgr.y 2021-03-24T22:01:08.870 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:08 smithi125 conmon[19936]: (mgr.14144) 520 : cluster [DBG] pgmap v907: 168 pgs: 1 peering, 4 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait, 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 1 remapped+peering, 143 active+clean; 480 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 8.2 MiB/s rd, 1.9 MiB/s wr, 15 op/s; 280/1638 objects degraded (17.094%); 194/1638 objects misplaced (11.844%); 2.4 MiB/s, 31 objects/s recovering 2021-03-24T22:01:08.871 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:08 smithi125 conmon[19936]: cluster 2021-03-24T22:01:07.312858+0000 mon.a (mon.0) 1448 : cluster [DBG] osdmap e497: 8 total, 8 up, 6 in 2021-03-24T22:01:08.871 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:08 smithi125 conmon[19936]: cluster 2021-03-24T22:01:08.313209+0000 mon.a (mon.0) 2021-03-24T22:01:08.871 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:08 smithi125 conmon[19936]: 1449 : cluster [DBG] osdmap e498: 8 total, 8 up, 6 in 2021-03-24T22:01:08.872 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:08 smithi125 conmon[24927]: cluster 2021-03-24T22:01:08.872 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:08 smithi125 conmon[24927]: 2021-03-24T22:01:08.873 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:08 smithi125 conmon[24927]: 2021 2021-03-24T22:01:08.873 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:08 smithi125 conmon[24927]: -03-24T22:01: 2021-03-24T22:01:08.873 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:08 smithi125 conmon[24927]: 06.874999+0000 mgr.y (mgr.14144) 520 : cluster [DBG] pgmap v907: 168 pgs: 1 peering, 4 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait, 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 1 remapped+peering, 143 active+clean; 480 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 8.2 MiB/s rd, 1.9 MiB/s wr, 15 op/s; 280/1638 objects degraded (17.094%); 194/1638 objects misplaced (11.844%); 2.4 MiB/s, 31 objects/s recovering 2021-03-24T22:01:08.874 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:08 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:01:08.874 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:08 smithi125 conmon[24927]: 24T22:01:07.312858+0000 mon.a (mon. 2021-03-24T22:01:08.874 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:08 smithi125 conmon[24927]: 0) 1448 : cluster [DBG] osdmap e497: 8 total, 8 up, 6 in 2021-03-24T22:01:08.874 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:08 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T22:01:08.875 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:08 smithi125 conmon[24927]: -24T22:01:08.313209+0000 mon.a ( 2021-03-24T22:01:08.875 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:08 smithi125 conmon[24927]: mon.0) 1449 : cluster [DBG] 2021-03-24T22:01:08.875 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:08 smithi125 conmon[24927]: osdmap e498: 8 total, 8 up, 6 in 2021-03-24T22:01:08.893 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:08.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:08.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:01:08.928 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:01:09.000 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:09.032 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:01:09.055 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:09.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:09.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:01:09.161 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:09.162 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-24T22:01:09.216 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:09.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:09.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:01:09.322 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:09.330 INFO:tasks.rados.rados.0.smithi125.stdout:1879: done (3 left) 2021-03-24T22:01:09.330 INFO:tasks.rados.rados.0.smithi125.stdout:1880: read oid 26 snap -1 2021-03-24T22:01:09.331 INFO:tasks.rados.rados.0.smithi125.stdout:1880: expect (ObjNum 618 snap 187 seq_num 618) 2021-03-24T22:01:09.331 INFO:tasks.rados.rados.0.smithi125.stdout:1881: read oid 29 snap -1 2021-03-24T22:01:09.332 INFO:tasks.rados.rados.0.smithi125.stdout:1881: expect (ObjNum 640 snap 196 seq_num 640) 2021-03-24T22:01:09.332 INFO:tasks.rados.rados.0.smithi125.stdout:1882: write oid 39 current snap is 205 2021-03-24T22:01:09.333 INFO:tasks.rados.rados.0.smithi125.stdout:1882: seq_num 665 ranges {776655=563296,2047702=620235,3095349=81306} 2021-03-24T22:01:09.333 INFO:tasks.rados.rados.0.smithi125.stdout:1882: writing smithi12556470-39 from 776655 to 1339951 tid 1 2021-03-24T22:01:09.334 INFO:tasks.rados.rados.0.smithi125.stdout:1882: writing smithi12556470-39 from 2047702 to 2667937 tid 2 2021-03-24T22:01:09.338 INFO:tasks.rados.rados.0.smithi125.stdout:1882: writing smithi12556470-39 from 3095349 to 3176655 tid 3 2021-03-24T22:01:09.339 INFO:tasks.rados.rados.0.smithi125.stdout:1883: snap_remove snap 204 2021-03-24T22:01:09.374 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:09.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:09.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:01:09.480 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:09.527 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:09.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:09.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:01:09.634 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:09.685 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:09.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:09.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:01:09.970 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:10.018 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:10.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:10.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:01:10.123 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:10.180 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:10.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:10.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:01:10.264 INFO:tasks.rados.rados.0.smithi125.stdout:1883: done (6 left) 2021-03-24T22:01:10.265 INFO:tasks.rados.rados.0.smithi125.stdout:1884: snap_remove snap 178 2021-03-24T22:01:10.269 INFO:teuthology.orchestra.run.smithi125.stderr:noscrub is set 2021-03-24T22:01:10.285 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:10.331 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:10.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:10.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:01:10.435 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:10.488 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:10.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:10.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:01:10.593 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:10.655 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:10 smithi142 conmon[20125]: cluster 2021-03-24T22: 2021-03-24T22:01:10.655 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:10 smithi142 conmon[20125]: 01:08.875537+0000 mgr.y (mgr.14144) 521 : cluster [DBG] pgmap v910: 168 pgs: 1 peering, 4 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait, 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 1 remapped+peering, 143 active+clean; 480 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 5.2 MiB/s rd, 4 op/s; 280/1638 objects degraded (17.094%); 194/1638 objects misplaced (11.844%); 2.4 MiB/s, 12 objects/s recovering 2021-03-24T22:01:10.656 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:10 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:01:10.656 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:10 smithi142 conmon[20125]: 24T22:01:09.313701+0000 mon.a (mon.0) 1450 2021-03-24T22:01:10.656 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:10 smithi142 conmon[20125]: : cluster [DBG] osdmap e499: 8 total, 8 up, 6 in 2021-03-24T22:01:10.656 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:10 smithi142 conmon[20125]: audit 2021- 2021-03-24T22:01:10.657 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:10 smithi142 conmon[20125]: 03-24T22:01:09.609507+0000 mon.a 2021-03-24T22:01:10.657 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:10 smithi142 conmon[20125]: (mon.0) 1451 : audit [INF] 2021-03-24T22:01:10.657 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:10 smithi142 conmon[20125]: from='client.? v1:172.21.15.125:0/1156778380' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T22:01:10.657 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:10 smithi142 conmon[20125]: cluster 2021-03-24T22:01 2021-03-24T22:01:10.658 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:10 smithi142 conmon[20125]: :10.254024+0000 mon.a (mon.0 2021-03-24T22:01:10.658 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:10 smithi142 conmon[20125]: ) 1452 : cluster [WRN] Health check update: Degraded data redundancy: 280/1638 objects degraded (17.094%), 12 pgs degraded (PG_DEGRADED) 2021-03-24T22:01:10.658 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:10 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:01:10.658 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:10 smithi142 conmon[20125]: 24T22:01:10.254820+0000 2021-03-24T22:01:10.658 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:10 smithi142 conmon[20125]: mon.a (mon.0) 1453 : cluster [WRN] 2021-03-24T22:01:10.659 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:10 smithi142 conmon[20125]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:01:10.659 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:10 smithi142 conmon[20125]: audit 2021-03-24T22:01:10.659 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:10 smithi142 conmon[20125]: 2021-03-24T22:01:10.259054+0000 2021-03-24T22:01:10.660 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:10 smithi142 conmon[20125]: mon.a (mon.0) 1454 : audit 2021-03-24T22:01:10.660 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:10 smithi142 conmon[20125]: [INF] from='client.? v1:172.21.15.125:0/1156778380' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T22:01:10.660 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:10 smithi142 conmon[20125]: cluster 2021-03-24T22: 2021-03-24T22:01:10.661 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:10 smithi142 conmon[20125]: 01:10.259107+0000 mon.a (mon.0 2021-03-24T22:01:10.661 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:10 smithi142 conmon[20125]: ) 1455 : cluster [DBG] osdmap e500: 8 total, 8 up, 6 in 2021-03-24T22:01:10.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:10 smithi125 conmon[19936]: cluster 2021-03-24T22:01:08.875537+0000 mgr.y (mgr.14144) 521 : cluster [DBG] pgmap v910: 168 pgs: 1 peering, 4 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait, 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 1 remapped+peering, 143 active+clean; 480 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 5.2 MiB/s rd, 4 op/s; 280/1638 objects degraded (17.094%); 194/1638 objects misplaced (11.844%); 2.4 MiB/s, 12 objects/s recovering 2021-03-24T22:01:10.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:10 smithi125 conmon[19936]: cluster 2021-03-24T22:01:09.313701+0000 mon.a (mon.0) 1450 : cluster [DBG] osdmap e499: 8 total, 8 up, 6 in 2021-03-24T22:01:10.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:10 smithi125 conmon[19936]: audit 2021-03-24T22:01:09.609507+0000 mon.a 2021-03-24T22:01:10.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:10 smithi125 conmon[19936]: (mon.0) 1451 : audit [INF] from='client.? v1:172.21.15.125:0/1156778380' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T22:01:10.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:10 smithi125 conmon[19936]: cluster 2021-03-24T22:01:10.254024+0000 mon.a (mon.0) 1452 : cluster [WRN] Health check update: Degraded data redundancy: 280/1638 objects degraded (17.094%), 12 pgs degraded (PG_DEGRADED) 2021-03-24T22:01:10.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:10 smithi125 conmon[19936]: cluster 2021-03-24T22:01:10.254820+0000 mon.a (mon.0) 2021-03-24T22:01:10.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:10 smithi125 conmon[19936]: 1453 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:01:10.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:10 smithi125 conmon[19936]: audit 2021-03-24T22:01:10 2021-03-24T22:01:10.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:10 smithi125 conmon[19936]: .259054+0000 mon.a (mon.0) 1454 : audit [INF] 2021-03-24T22:01:10.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:10 smithi125 conmon[19936]: from='client.? v1:172.21.15.125:0/1156778380' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T22:01:10.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:10 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:01:10.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:10 smithi125 conmon[19936]: -03-24T22:01:10.259107+0000 mon.a (mon.0 2021-03-24T22:01:10.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:10 smithi125 conmon[19936]: ) 1455 : cluster [DBG] osdmap e500: 8 total, 8 up, 6 in 2021-03-24T22:01:10.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:10 smithi125 conmon[24927]: cluster 2021-03-24T22:01:10.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:10 smithi125 conmon[24927]: 2021-03-24T22:01:10.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:10 smithi125 conmon[24927]: 2021-03-24T22: 2021-03-24T22:01:10.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:10 smithi125 conmon[24927]: 01:08.875537+0000 mgr.y (mgr.14144) 521 : cluster [DBG] pgmap v910: 168 pgs: 1 peering, 4 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait, 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 1 remapped+peering, 143 active+clean; 480 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 5.2 MiB/s rd, 4 op/s; 280/1638 objects degraded (17.094%); 194/1638 objects misplaced (11.844%); 2.4 MiB/s, 12 objects/s recovering 2021-03-24T22:01:10.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:10 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:01:10.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:10 smithi125 conmon[24927]: -03-24T22:01:09.313701 2021-03-24T22:01:10.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:10 smithi125 conmon[24927]: +0000 mon.a (mon.0) 1450 2021-03-24T22:01:10.672 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:10 smithi125 conmon[24927]: : cluster [DBG] osdmap e499: 8 total, 8 up, 6 in 2021-03-24T22:01:10.672 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:10 smithi125 conmon[24927]: audit 2021-03 2021-03-24T22:01:10.673 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:10 smithi125 conmon[24927]: -24T22:01:09.609507+0000 mon.a (mon. 2021-03-24T22:01:10.673 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:10 smithi125 conmon[24927]: 0) 1451 : audit [INF] from='client.? v1:172.21.15.125:0/1156778380' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T22:01:10.673 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:10 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:01:10.674 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:10 smithi125 conmon[24927]: 03-24T22:01:10.254024+0000 mon.a 2021-03-24T22:01:10.674 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:10 smithi125 conmon[24927]: (mon.0) 1452 : cluster [WRN] Health check update: Degraded data redundancy: 280/1638 objects degraded (17.094%), 12 pgs degraded (PG_DEGRADED) 2021-03-24T22:01:10.675 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:10 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T22:01:10.675 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:10 smithi125 conmon[24927]: -24T22:01:10.254820+0000 mon.a ( 2021-03-24T22:01:10.676 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:10 smithi125 conmon[24927]: mon.0) 1453 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:01:10.676 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:10 smithi125 conmon[24927]: audit 2021-03-24T22:01:10.677 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:10 smithi125 conmon[24927]: 2021-03- 2021-03-24T22:01:10.677 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:10 smithi125 conmon[24927]: 24T22:01:10.259054+0000 mon.a (mon 2021-03-24T22:01:10.677 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:10 smithi125 conmon[24927]: .0) 1454 : audit [INF] from='client.? v1:172.21.15.125:0/1156778380' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T22:01:10.678 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:10 smithi125 conmon[24927]: cluster 2021-03-24T22:01:10.679 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:10 smithi125 conmon[24927]: 2021-03-24T 2021-03-24T22:01:10.679 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:10 smithi125 conmon[24927]: 22:01:10.259107+0000 mon.a (mon. 2021-03-24T22:01:10.680 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:10 smithi125 conmon[24927]: 0) 1455 : cluster [DBG] osdmap e500: 8 total, 8 up, 6 in 2021-03-24T22:01:10.706 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:10.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:10.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:01:10.812 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:10.859 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:10.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:10.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:01:10.864 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:01:10.965 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:11.016 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:11.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:11.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:01:11.122 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:11.177 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:11.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:11.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:01:11.267 INFO:tasks.rados.rados.0.smithi125.stdout:1884: done (6 left) 2021-03-24T22:01:11.267 INFO:tasks.rados.rados.0.smithi125.stdout:1885: snap_create 2021-03-24T22:01:11.268 INFO:tasks.rados.rados.0.smithi125.stdout:1876: expect (ObjNum 642 snap 196 seq_num 642) 2021-03-24T22:01:11.286 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:11.337 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:11.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:11.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:01:11.392 INFO:tasks.rados.rados.0.smithi125.stdout:1878: expect (ObjNum 462 snap 129 seq_num 462) 2021-03-24T22:01:11.442 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:11.502 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:11.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:11.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:01:11.609 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:11.652 INFO:tasks.rados.rados.0.smithi125.stdout:1877: finishing copy_from to smithi12556470-11 2021-03-24T22:01:11.652 INFO:tasks.rados.rados.0.smithi125.stdout:1877: got expected ENOENT (src dne) 2021-03-24T22:01:11.653 INFO:tasks.rados.rados.0.smithi125.stdout:1877: finishing copy_from racing read to smithi12556470-11 2021-03-24T22:01:11.653 INFO:tasks.rados.rados.0.smithi125.stdout:1880: expect (ObjNum 618 snap 187 seq_num 618) 2021-03-24T22:01:11.667 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:11.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:11.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:01:11.736 INFO:tasks.rados.rados.0.smithi125.stdout:1881: expect (ObjNum 640 snap 196 seq_num 640) 2021-03-24T22:01:11.773 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:11.836 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:11.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:11.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:01:11.876 INFO:tasks.rados.rados.0.smithi125.stdout:1882: finishing write tid 1 to smithi12556470-39 2021-03-24T22:01:11.876 INFO:tasks.rados.rados.0.smithi125.stdout:1882: finishing write tid 2 to smithi12556470-39 2021-03-24T22:01:11.877 INFO:tasks.rados.rados.0.smithi125.stdout:1882: finishing write tid 3 to smithi12556470-39 2021-03-24T22:01:11.877 INFO:tasks.rados.rados.0.smithi125.stdout:1882: finishing write tid 4 to smithi12556470-39 2021-03-24T22:01:11.877 INFO:tasks.rados.rados.0.smithi125.stdout:1882: finishing write tid 5 to smithi12556470-39 2021-03-24T22:01:11.878 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 39 v 655 (ObjNum 665 snap 205 seq_num 665) dirty exists 2021-03-24T22:01:11.878 INFO:tasks.rados.rados.0.smithi125.stdout:1882: left oid 39 (ObjNum 665 snap 205 seq_num 665) 2021-03-24T22:01:11.947 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:12.003 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:12.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:12.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:01:12.110 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:12.175 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:12.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:12.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:01:12.277 INFO:tasks.rados.rados.0.smithi125.stdout:1876: done (6 left) 2021-03-24T22:01:12.278 INFO:tasks.rados.rados.0.smithi125.stdout:1877: done (5 left) 2021-03-24T22:01:12.278 INFO:tasks.rados.rados.0.smithi125.stdout:1878: done (4 left) 2021-03-24T22:01:12.279 INFO:tasks.rados.rados.0.smithi125.stdout:1880: done (3 left) 2021-03-24T22:01:12.279 INFO:tasks.rados.rados.0.smithi125.stdout:1881: done (2 left) 2021-03-24T22:01:12.280 INFO:tasks.rados.rados.0.smithi125.stdout:1882: done (1 left) 2021-03-24T22:01:12.280 INFO:tasks.rados.rados.0.smithi125.stdout:1885: done (0 left) 2021-03-24T22:01:12.281 INFO:tasks.rados.rados.0.smithi125.stdout:1886: snap_create 2021-03-24T22:01:12.281 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:12.296 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-24T22:01:12.297 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T22:01:12.329 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:12.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:12.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:01:12.438 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:12.481 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:12.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:12.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:01:12.585 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:12.597 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[19936]: cluster 2021-03-24T22:01:12.598 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[19936]: 2021-03-24T22: 2021-03-24T22:01:12.598 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[19936]: 01:10.876194+0000 mgr.y (mgr.14144) 522 : cluster [DBG] pgmap v913: 168 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 149 active+clean; 486 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 MiB/s rd, 582 KiB/s wr, 7 op/s; 196/1653 objects degraded (11.857%); 198/1653 objects misplaced (11.978%); 12 MiB/s, 27 objects/s recovering 2021-03-24T22:01:12.598 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[19936]: cluster 2021-03-24T22:01:12.599 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[19936]: 2021-03-24T22:01:12.599 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[19936]: 2021-03-24T22:01:11 2021-03-24T22:01:12.600 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[19936]: .262126+0000 mon.a (mon.0) 1456 : cluster [DBG] osdmap e501: 8 total, 8 up, 6 in 2021-03-24T22:01:12.600 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:01:12.600 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[19936]: 24T22:01:11.354304+0000 mon.a ( 2021-03-24T22:01:12.601 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[19936]: mon.0) 1457 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:01:12.602 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: cluster 2021-03-24T22:01:12.602 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: 2021-03-24T22:01:12.603 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: 2021 2021-03-24T22:01:12.603 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: - 2021-03-24T22:01:12.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: 03 2021-03-24T22:01:12.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: - 2021-03-24T22:01:12.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: 24 2021-03-24T22:01:12.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: T 2021-03-24T22:01:12.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: 22 2021-03-24T22:01:12.606 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: : 2021-03-24T22:01:12.606 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: 01 2021-03-24T22:01:12.606 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: : 2021-03-24T22:01:12.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: 10 2021-03-24T22:01:12.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: . 2021-03-24T22:01:12.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: 876194 2021-03-24T22:01:12.608 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: +0000 2021-03-24T22:01:12.608 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: 2021-03-24T22:01:12.608 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: mgr.y 2021-03-24T22:01:12.608 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: ( 2021-03-24T22:01:12.609 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: mgr 2021-03-24T22:01:12.609 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: . 2021-03-24T22:01:12.609 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: 14144 2021-03-24T22:01:12.610 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: ) 2021-03-24T22:01:12.610 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: 522 2021-03-24T22:01:12.610 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: : 2021-03-24T22:01:12.610 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: cluster 2021-03-24T22:01:12.611 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: 2021-03-24T22:01:12.611 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: [DBG] 2021-03-24T22:01:12.611 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: 2021-03-24T22:01:12.612 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: pgmap v913: 168 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 149 active+clean; 486 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 MiB/s rd, 582 KiB/s wr, 7 op/s; 196/1653 objects degraded (11.857%); 198/1653 objects misplaced (11.978%); 12 MiB/s, 27 objects/s recovering 2021-03-24T22:01:12.612 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: 2021-03-24T22:01:12.612 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: cluster 2021-03-24T22:01:12.613 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: 2021-03-24T22:01:12.613 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: 2021 2021-03-24T22:01:12.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: - 2021-03-24T22:01:12.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: 03 2021-03-24T22:01:12.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: - 2021-03-24T22:01:12.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: 24 2021-03-24T22:01:12.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: T 2021-03-24T22:01:12.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: 22 2021-03-24T22:01:12.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: : 2021-03-24T22:01:12.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: 01 2021-03-24T22:01:12.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: : 2021-03-24T22:01:12.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: 11 2021-03-24T22:01:12.618 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: . 2021-03-24T22:01:12.618 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: 262126 2021-03-24T22:01:12.619 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: +0000 2021-03-24T22:01:12.619 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: 2021-03-24T22:01:12.620 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: mon.a 2021-03-24T22:01:12.620 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: ( 2021-03-24T22:01:12.620 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: mon 2021-03-24T22:01:12.621 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: . 2021-03-24T22:01:12.621 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: 0 2021-03-24T22:01:12.622 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: ) 2021-03-24T22:01:12.622 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: 1456 2021-03-24T22:01:12.623 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: : 2021-03-24T22:01:12.623 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: cluster 2021-03-24T22:01:12.623 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: 2021-03-24T22:01:12.624 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: [DBG] 2021-03-24T22:01:12.624 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: 2021-03-24T22:01:12.624 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: osdmap e501: 8 total, 8 up, 6 in 2021-03-24T22:01:12.625 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: 2021-03-24T22:01:12.625 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: cluster 2021-03-24T22:01:12.625 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: 2021-03-24T22:01:12.625 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: 2021 2021-03-24T22:01:12.626 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: -03-24T22:01:11. 2021-03-24T22:01:12.626 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:12 smithi125 conmon[24927]: 354304+0000 mon.a (mon.0) 1457 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:01:12.627 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:12 smithi142 conmon[20125]: cluster 2021-03-24T22:01:10.876194+0000 mgr.y (mgr.14144) 2021-03-24T22:01:12.627 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:12 smithi142 conmon[20125]: 522 : cluster [DBG] pgmap v913: 168 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfill_toofull, 2 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 149 active+clean; 486 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 MiB/s rd, 582 KiB/s wr, 7 op/s; 196/1653 objects degraded (11.857%); 198/1653 objects misplaced (11.978%); 12 MiB/s, 27 objects/s recovering 2021-03-24T22:01:12.627 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:12 smithi142 conmon[20125]: cluster 2021-03-24T22:01:11.262126+0000 mon.a (mon.0) 1456 : cluster [DBG] osdmap e501: 8 total, 8 up, 6 in 2021-03-24T22:01:12.628 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:12 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:01:12.628 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:12 smithi142 conmon[20125]: T22:01:11.354304+0000 mon.a (mon.0) 1457 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:01:12.635 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:12.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:12.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:01:12.743 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:12.794 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:12.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:12.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:01:12.901 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:12.951 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:12.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:12.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:01:13.057 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:13.102 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:13.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:13.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:01:13.207 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:13.258 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:13.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:13.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:01:13.268 INFO:tasks.rados.rados.0.smithi125.stdout:1886: done (0 left) 2021-03-24T22:01:13.269 INFO:tasks.rados.rados.0.smithi125.stdout:1887: write (excl) oid 42 current snap is 207 2021-03-24T22:01:13.269 INFO:tasks.rados.rados.0.smithi125.stdout:1887: seq_num 666 ranges {793462=638545,2106184=763893} 2021-03-24T22:01:13.279 INFO:tasks.rados.rados.0.smithi125.stdout:1887: writing smithi12556470-42 from 793462 to 1432007 tid 1 2021-03-24T22:01:13.289 INFO:tasks.rados.rados.0.smithi125.stdout:1887: writing smithi12556470-42 from 2106184 to 2870077 tid 2 2021-03-24T22:01:13.289 INFO:tasks.rados.rados.0.smithi125.stdout:1888: read oid 50 snap 200 2021-03-24T22:01:13.290 INFO:tasks.rados.rados.0.smithi125.stdout:1888: expect deleted 2021-03-24T22:01:13.290 INFO:tasks.rados.rados.0.smithi125.stdout:1889: snap_remove snap 196 2021-03-24T22:01:13.363 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:13.422 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:13.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:13.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:01:13.529 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:13.542 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[24927]: cluster 2021-03-24T22:01:12.088066 2021-03-24T22:01:13.543 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[24927]: +0000 osd.0 (osd.0) 104 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:01:13.543 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[24927]: cluster 2021-03-24T22:01:12.089342+0000 2021-03-24T22:01:13.544 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[24927]: osd.0 (osd.0) 105 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:01:13.544 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[24927]: 2021-03-24T22:01:13.545 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[24927]: cluster 2021-03-24T22:01:13.545 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[24927]: 2021-03-24T22:01:12.267805+0000 mon.a (mon 2021-03-24T22:01:13.546 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[24927]: .0) 1458 : cluster [DBG] osdmap e502: 8 total, 8 up, 6 in 2021-03-24T22:01:13.546 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:01:13.546 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[24927]: 03-24T22:01:13.265919+0000 mon.a ( 2021-03-24T22:01:13.547 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[24927]: mon.0) 1459 : cluster [DBG] osdmap e503: 8 total, 8 up, 6 in 2021-03-24T22:01:13.548 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: cluster 2021-03-24T22:01:13.548 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 2021-03-24T22:01:13.549 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 2021 2021-03-24T22:01:13.549 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: - 2021-03-24T22:01:13.549 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 03 2021-03-24T22:01:13.549 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: - 2021-03-24T22:01:13.549 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 24 2021-03-24T22:01:13.550 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: T 2021-03-24T22:01:13.550 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 22 2021-03-24T22:01:13.550 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: : 2021-03-24T22:01:13.551 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 01 2021-03-24T22:01:13.551 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: : 2021-03-24T22:01:13.551 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 12 2021-03-24T22:01:13.551 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: . 2021-03-24T22:01:13.552 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 088066 2021-03-24T22:01:13.552 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: +0000 2021-03-24T22:01:13.552 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 2021-03-24T22:01:13.552 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: osd.0 2021-03-24T22:01:13.552 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: ( 2021-03-24T22:01:13.553 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: osd 2021-03-24T22:01:13.553 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: . 2021-03-24T22:01:13.553 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 0 2021-03-24T22:01:13.553 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: ) 2021-03-24T22:01:13.554 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 104 2021-03-24T22:01:13.554 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: : 2021-03-24T22:01:13.554 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: cluster 2021-03-24T22:01:13.554 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 2021-03-24T22:01:13.554 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: [DBG] 2021-03-24T22:01:13.555 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 2021-03-24T22:01:13.555 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: purged_snaps scrub starts 2021-03-24T22:01:13.555 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 2021-03-24T22:01:13.555 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: cluster 2021-03-24T22:01:13.556 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 2021-03-24T22:01:13.556 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 2021 2021-03-24T22:01:13.556 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: - 2021-03-24T22:01:13.557 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 03 2021-03-24T22:01:13.557 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: - 2021-03-24T22:01:13.558 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 24 2021-03-24T22:01:13.558 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: T 2021-03-24T22:01:13.559 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 22 2021-03-24T22:01:13.559 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: : 2021-03-24T22:01:13.559 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 01 2021-03-24T22:01:13.559 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: : 2021-03-24T22:01:13.559 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 12 2021-03-24T22:01:13.560 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: . 2021-03-24T22:01:13.560 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 089342 2021-03-24T22:01:13.560 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: +0000 2021-03-24T22:01:13.560 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 2021-03-24T22:01:13.561 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: osd.0 2021-03-24T22:01:13.561 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: ( 2021-03-24T22:01:13.562 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: osd 2021-03-24T22:01:13.562 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: . 2021-03-24T22:01:13.563 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 0 2021-03-24T22:01:13.563 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: ) 2021-03-24T22:01:13.563 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 105 2021-03-24T22:01:13.563 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: : 2021-03-24T22:01:13.564 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: cluster 2021-03-24T22:01:13.564 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 2021-03-24T22:01:13.564 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: [DBG] 2021-03-24T22:01:13.564 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 2021-03-24T22:01:13.565 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: purged_snaps scrub ok 2021-03-24T22:01:13.565 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 2021-03-24T22:01:13.565 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: cluster 2021-03-24T22:01:13.565 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 2021-03-24T22:01:13.566 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 2021 2021-03-24T22:01:13.566 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: - 2021-03-24T22:01:13.566 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 03 2021-03-24T22:01:13.566 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: - 2021-03-24T22:01:13.567 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 24 2021-03-24T22:01:13.567 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: T 2021-03-24T22:01:13.567 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 22 2021-03-24T22:01:13.567 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: : 2021-03-24T22:01:13.567 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 01 2021-03-24T22:01:13.568 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: : 2021-03-24T22:01:13.568 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 12 2021-03-24T22:01:13.568 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: . 2021-03-24T22:01:13.569 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 267805 2021-03-24T22:01:13.569 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: +0000 2021-03-24T22:01:13.569 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 2021-03-24T22:01:13.569 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: mon.a 2021-03-24T22:01:13.569 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: ( 2021-03-24T22:01:13.570 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: mon 2021-03-24T22:01:13.570 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: . 2021-03-24T22:01:13.570 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 0 2021-03-24T22:01:13.570 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: ) 2021-03-24T22:01:13.571 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 1458 2021-03-24T22:01:13.571 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: : 2021-03-24T22:01:13.571 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: cluster 2021-03-24T22:01:13.571 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 2021-03-24T22:01:13.572 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: [DBG] 2021-03-24T22:01:13.572 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 2021-03-24T22:01:13.572 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: osdmap e502: 8 total, 8 up, 6 in 2021-03-24T22:01:13.572 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 2021-03-24T22:01:13.572 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: cluster 2021-03-24T22:01:13.573 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 2021-03-24T22:01:13.573 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 2021 2021-03-24T22:01:13.573 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: - 2021-03-24T22:01:13.573 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 03 2021-03-24T22:01:13.574 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: - 2021-03-24T22:01:13.574 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 24 2021-03-24T22:01:13.574 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: T 2021-03-24T22:01:13.574 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 22 2021-03-24T22:01:13.575 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: : 2021-03-24T22:01:13.575 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 01 2021-03-24T22:01:13.575 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: : 2021-03-24T22:01:13.576 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 13 2021-03-24T22:01:13.576 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: . 2021-03-24T22:01:13.576 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 265919 2021-03-24T22:01:13.576 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: +0000 2021-03-24T22:01:13.577 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 2021-03-24T22:01:13.577 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: mon.a 2021-03-24T22:01:13.577 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: ( 2021-03-24T22:01:13.577 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: mon 2021-03-24T22:01:13.577 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: . 2021-03-24T22:01:13.578 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 0 2021-03-24T22:01:13.578 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: ) 2021-03-24T22:01:13.578 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 1459 2021-03-24T22:01:13.578 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: : 2021-03-24T22:01:13.579 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: cluster 2021-03-24T22:01:13.579 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 2021-03-24T22:01:13.579 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: [DBG] 2021-03-24T22:01:13.579 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 2021-03-24T22:01:13.580 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: osdmap e503: 8 total, 8 up, 6 in 2021-03-24T22:01:13.580 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:13 smithi125 conmon[19936]: 2021-03-24T22:01:13.580 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:13.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:13.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:01:13.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:13 smithi142 conmon[20125]: cluster 2021-03-24T22:01:12.088066+0000 osd.0 (osd.0) 104 : cluster 2021-03-24T22:01:13.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:13 smithi142 conmon[20125]: [DBG] purged_snaps scrub starts 2021-03-24T22:01:13.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:13 smithi142 conmon[20125]: cluster 2021-03-24T22:01:12.089342+0000 osd.0 (osd.0) 105 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:01:13.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:13 smithi142 conmon[20125]: cluster 2021-03-24T22:01:12.267805 2021-03-24T22:01:13.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:13 smithi142 conmon[20125]: +0000 mon.a (mon.0) 1458 : cluster [DBG] osdmap e502: 8 total, 8 up, 6 in 2021-03-24T22:01:13.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:13 smithi142 conmon[20125]: cluster 2021-03-24T22 2021-03-24T22:01:13.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:13 smithi142 conmon[20125]: :01:13.265919+0000 mon.a (mon.0) 2021-03-24T22:01:13.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:13 smithi142 conmon[20125]: 1459 : cluster [DBG] osdmap e503: 8 total, 8 up, 6 in 2021-03-24T22:01:13.686 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:13.742 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:13.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:13.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:01:13.849 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:13.901 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:13.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:13.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:01:14.008 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:14.061 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:14.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:14.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:01:14.166 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:14.226 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:14.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:14.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:01:14.275 INFO:tasks.rados.rados.0.smithi125.stdout:1889: done (2 left) 2021-03-24T22:01:14.276 INFO:tasks.rados.rados.0.smithi125.stdout:1890: snap_remove snap 186 2021-03-24T22:01:14.333 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:14.390 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:14.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:14.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:01:14.496 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:14.543 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:14.549 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:14 smithi125 conmon[24927]: cluster 2021-03-24T22:01 2021-03-24T22:01:14.549 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:14 smithi125 conmon[24927]: :12.877032+0000 mgr.y (mgr.14144) 523 : cluster [DBG] pgmap v916: 168 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 150 active+clean; 482 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 MiB/s rd, 582 KiB/s wr, 7 op/s; 196/1650 objects degraded (11.879%); 192/1650 objects misplaced (11.636%); 13 MiB/s, 29 objects/s recovering 2021-03-24T22:01:14.550 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:14 smithi125 conmon[24927]: cluster 2021-03-24T22:01:14.550 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:14 smithi125 conmon[24927]: 2021-03-24T22:01:14.269579+0000 mon.a (mon.0) 2021-03-24T22:01:14.551 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:14 smithi125 conmon[24927]: 1460 : cluster [DBG] osdmap e504: 8 total, 8 up, 6 in 2021-03-24T22:01:14.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:14.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:01:14.553 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:14 smithi125 conmon[19936]: cluster 2021-03-24T22:01:14.554 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:14 smithi125 conmon[19936]: 2021-03-24T22:01:14.554 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:14 smithi125 conmon[19936]: 2021 2021-03-24T22:01:14.555 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:14 smithi125 conmon[19936]: - 2021-03-24T22:01:14.555 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:14 smithi125 conmon[19936]: 03 2021-03-24T22:01:14.555 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:14 smithi125 conmon[19936]: - 2021-03-24T22:01:14.556 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:14 smithi125 conmon[19936]: 24 2021-03-24T22:01:14.556 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:14 smithi125 conmon[19936]: T 2021-03-24T22:01:14.556 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:14 smithi125 conmon[19936]: 22 2021-03-24T22:01:14.557 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:14 smithi125 conmon[19936]: : 2021-03-24T22:01:14.557 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:14 smithi125 conmon[19936]: 01:12.877032+0000 2021-03-24T22:01:14.558 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:14 smithi125 conmon[19936]: mgr.y (mgr.14144) 523 : cluster [DBG] pgmap v916: 168 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 150 active+clean; 482 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 MiB/s rd, 582 KiB/s wr, 7 op/s; 196/1650 objects degraded (11.879%); 192/1650 objects misplaced (11.636%); 13 MiB/s, 29 objects/s recovering 2021-03-24T22:01:14.558 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:14 smithi125 conmon[19936]: cluster 2021-03-24T22:01:14.269579+0000 mon.a (mon 2021-03-24T22:01:14.558 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:14 smithi125 conmon[19936]: .0) 1460 : cluster [DBG] osdmap e504: 8 total, 8 up, 6 in 2021-03-24T22:01:14.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:14 smithi142 conmon[20125]: cluster 2021-03-24T22:01:12.877032+0000 2021-03-24T22:01:14.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:14 smithi142 conmon[20125]: mgr.y (mgr.14144) 523 : cluster [DBG] pgmap v916: 168 pgs: 1 active+recovery_wait+remapped, 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 150 active+clean; 482 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 MiB/s rd, 582 KiB/s wr, 7 op/s; 196/1650 objects degraded (11.879%); 192/1650 objects misplaced (11.636%); 13 MiB/s, 29 objects/s recovering 2021-03-24T22:01:14.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:14 smithi142 conmon[20125]: cluster 2021-03-24T22:01:14.269579+0000 mon.a (mon.0) 1460 : cluster [DBG] osdmap e504: 8 total, 8 up, 6 in 2021-03-24T22:01:14.620 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.1 dump_historic_ops 2021-03-24T22:01:14.655 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:14.704 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:14.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:14.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:01:14.809 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:14.870 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:14.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:14.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:01:14.978 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:15.038 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:15.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:15.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:01:15.146 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:15.170 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:14 smithi125 conmon[19936]: debug 2021-03-24T22:01:14.951+0000 7f2118128700 -1 mon.a@0(leader).osd e504 definitely_dead 0 2021-03-24T22:01:15.196 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:15.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:15.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:01:15.286 INFO:tasks.rados.rados.0.smithi125.stdout:1890: done (2 left) 2021-03-24T22:01:15.286 INFO:tasks.rados.rados.0.smithi125.stdout:1891: write oid 40 current snap is 207 2021-03-24T22:01:15.287 INFO:tasks.rados.rados.0.smithi125.stdout:1891: seq_num 667 ranges {410269=713794,1648313=507551,2917489=292780} 2021-03-24T22:01:15.288 INFO:tasks.rados.rados.0.smithi125.stdout:1891: writing smithi12556470-40 from 410269 to 1124063 tid 1 2021-03-24T22:01:15.292 INFO:teuthology.orchestra.run.smithi125.stderr:marked in osd.1. 2021-03-24T22:01:15.295 INFO:tasks.rados.rados.0.smithi125.stdout:1891: writing smithi12556470-40 from 1648313 to 2155864 tid 2 2021-03-24T22:01:15.301 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:15.303 INFO:tasks.rados.rados.0.smithi125.stdout:1891: writing smithi12556470-40 from 2917489 to 3210269 tid 3 2021-03-24T22:01:15.303 INFO:tasks.rados.rados.0.smithi125.stdout:1887: finishing write tid 1 to smithi12556470-42 2021-03-24T22:01:15.303 INFO:tasks.rados.rados.0.smithi125.stdout:1887: finishing write tid 2 to smithi12556470-42 2021-03-24T22:01:15.304 INFO:tasks.rados.rados.0.smithi125.stdout:1887: finishing write tid 3 to smithi12556470-42 2021-03-24T22:01:15.304 INFO:tasks.rados.rados.0.smithi125.stdout:1887: finishing write tid 4 to smithi12556470-42 2021-03-24T22:01:15.304 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 42 v 495 (ObjNum 666 snap 207 seq_num 666) dirty exists 2021-03-24T22:01:15.304 INFO:tasks.rados.rados.0.smithi125.stdout:1887: left oid 42 (ObjNum 666 snap 207 seq_num 666) 2021-03-24T22:01:15.304 INFO:tasks.rados.rados.0.smithi125.stdout:1887: done (2 left) 2021-03-24T22:01:15.305 INFO:tasks.rados.rados.0.smithi125.stdout:1888: done (1 left) 2021-03-24T22:01:15.306 INFO:tasks.rados.rados.0.smithi125.stdout:1892: write oid 32 current snap is 207 2021-03-24T22:01:15.306 INFO:tasks.rados.rados.0.smithi125.stdout:1892: seq_num 668 ranges {427076=789043,1706795=767562,2944912=282164} 2021-03-24T22:01:15.311 INFO:tasks.rados.rados.0.smithi125.stdout:1892: writing smithi12556470-32 from 427076 to 1216119 tid 1 2021-03-24T22:01:15.320 INFO:tasks.rados.rados.0.smithi125.stdout:1892: writing smithi12556470-32 from 1706795 to 2474357 tid 2 2021-03-24T22:01:15.325 INFO:tasks.rados.rados.0.smithi125.stdout:1892: writing smithi12556470-32 from 2944912 to 3227076 tid 3 2021-03-24T22:01:15.325 INFO:tasks.rados.rados.0.smithi125.stdout:1893: copy_from oid 48 from oid 5 current snap is 207 2021-03-24T22:01:15.326 INFO:tasks.rados.rados.0.smithi125.stdout:1894: snap_create 2021-03-24T22:01:15.357 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:15.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:15.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:01:15.393 INFO:tasks.rados.rados.0.smithi125.stdout:1891: finishing write tid 1 to smithi12556470-40 2021-03-24T22:01:15.393 INFO:tasks.rados.rados.0.smithi125.stdout:1891: finishing write tid 2 to smithi12556470-40 2021-03-24T22:01:15.393 INFO:tasks.rados.rados.0.smithi125.stdout:1891: finishing write tid 3 to smithi12556470-40 2021-03-24T22:01:15.394 INFO:tasks.rados.rados.0.smithi125.stdout:1891: finishing write tid 4 to smithi12556470-40 2021-03-24T22:01:15.395 INFO:tasks.rados.rados.0.smithi125.stdout:1891: finishing write tid 5 to smithi12556470-40 2021-03-24T22:01:15.395 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 40 v 896 (ObjNum 667 snap 207 seq_num 667) dirty exists 2021-03-24T22:01:15.395 INFO:tasks.rados.rados.0.smithi125.stdout:1891: left oid 40 (ObjNum 667 snap 207 seq_num 667) 2021-03-24T22:01:15.395 INFO:tasks.rados.rados.0.smithi125.stdout:1892: finishing write tid 1 to smithi12556470-32 2021-03-24T22:01:15.396 INFO:tasks.rados.rados.0.smithi125.stdout:1892: finishing write tid 2 to smithi12556470-32 2021-03-24T22:01:15.396 INFO:tasks.rados.rados.0.smithi125.stdout:1892: finishing write tid 3 to smithi12556470-32 2021-03-24T22:01:15.396 INFO:tasks.rados.rados.0.smithi125.stdout:1892: finishing write tid 4 to smithi12556470-32 2021-03-24T22:01:15.397 INFO:tasks.rados.rados.0.smithi125.stdout:1892: finishing write tid 5 to smithi12556470-32 2021-03-24T22:01:15.398 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 32 v 900 (ObjNum 668 snap 207 seq_num 668) dirty exists 2021-03-24T22:01:15.398 INFO:tasks.rados.rados.0.smithi125.stdout:1892: left oid 32 (ObjNum 668 snap 207 seq_num 668) 2021-03-24T22:01:15.464 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:15.477 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: audit 2021-03-24T22:01:15.477 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 2021-03-24T22:01:15.477 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 2021 2021-03-24T22:01:15.477 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: - 2021-03-24T22:01:15.478 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 03 2021-03-24T22:01:15.478 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: - 2021-03-24T22:01:15.478 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 24 2021-03-24T22:01:15.478 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: T 2021-03-24T22:01:15.479 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 22 2021-03-24T22:01:15.479 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: : 2021-03-24T22:01:15.479 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 01 2021-03-24T22:01:15.479 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: : 2021-03-24T22:01:15.480 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 14 2021-03-24T22:01:15.480 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: . 2021-03-24T22:01:15.480 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 954681 2021-03-24T22:01:15.480 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: +0000 2021-03-24T22:01:15.480 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 2021-03-24T22:01:15.481 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: mon.c 2021-03-24T22:01:15.481 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: ( 2021-03-24T22:01:15.481 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: mon 2021-03-24T22:01:15.481 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: . 2021-03-24T22:01:15.481 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 2 2021-03-24T22:01:15.482 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: ) 2021-03-24T22:01:15.482 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 78 2021-03-24T22:01:15.482 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: : 2021-03-24T22:01:15.482 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: audit 2021-03-24T22:01:15.483 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 2021-03-24T22:01:15.483 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: [DBG] 2021-03-24T22:01:15.483 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 2021-03-24T22:01:15.483 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: from='client.? v1:172.21.15.125:0/4091977824' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2021-03-24T22:01:15.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 2021-03-24T22:01:15.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: audit 2021-03-24T22:01:15.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 2021-03-24T22:01:15.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 2021 2021-03-24T22:01:15.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: - 2021-03-24T22:01:15.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 03 2021-03-24T22:01:15.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: - 2021-03-24T22:01:15.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 24 2021-03-24T22:01:15.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: T 2021-03-24T22:01:15.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 22 2021-03-24T22:01:15.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: : 2021-03-24T22:01:15.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 01 2021-03-24T22:01:15.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: : 2021-03-24T22:01:15.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 14 2021-03-24T22:01:15.487 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: . 2021-03-24T22:01:15.487 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 955120 2021-03-24T22:01:15.487 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: +0000 2021-03-24T22:01:15.487 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 2021-03-24T22:01:15.488 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: mon.a 2021-03-24T22:01:15.488 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: ( 2021-03-24T22:01:15.488 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: mon 2021-03-24T22:01:15.488 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: . 2021-03-24T22:01:15.488 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 0 2021-03-24T22:01:15.489 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: ) 2021-03-24T22:01:15.489 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 1461 2021-03-24T22:01:15.489 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: : 2021-03-24T22:01:15.489 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: audit 2021-03-24T22:01:15.490 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 2021-03-24T22:01:15.490 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: [DBG] 2021-03-24T22:01:15.490 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 2021-03-24T22:01:15.490 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: from='client.35351 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2021-03-24T22:01:15.491 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 2021-03-24T22:01:15.491 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: cluster 2021-03-24T22:01:15.491 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 2021-03-24T22:01:15.491 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 2021 2021-03-24T22:01:15.492 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: - 2021-03-24T22:01:15.492 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 03 2021-03-24T22:01:15.492 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: - 2021-03-24T22:01:15.492 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 24 2021-03-24T22:01:15.492 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: T 2021-03-24T22:01:15.493 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 22 2021-03-24T22:01:15.493 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: : 2021-03-24T22:01:15.493 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 01 2021-03-24T22:01:15.493 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: : 2021-03-24T22:01:15.494 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 15 2021-03-24T22:01:15.494 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: . 2021-03-24T22:01:15.494 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 255761 2021-03-24T22:01:15.494 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: +0000 2021-03-24T22:01:15.495 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 2021-03-24T22:01:15.495 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: mon.a 2021-03-24T22:01:15.495 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: ( 2021-03-24T22:01:15.495 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: mon 2021-03-24T22:01:15.495 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: . 2021-03-24T22:01:15.496 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 0 2021-03-24T22:01:15.496 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: ) 2021-03-24T22:01:15.496 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 1462 2021-03-24T22:01:15.496 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: : 2021-03-24T22:01:15.497 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: audit 2021-03-24T22:01:15.497 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 2021-03-24T22:01:15.497 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 2021 2021-03-24T22:01:15.498 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: - 2021-03-24T22:01:15.498 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 03 2021-03-24T22:01:15.498 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: - 2021-03-24T22:01:15.498 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 24 2021-03-24T22:01:15.499 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: T 2021-03-24T22:01:15.499 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 22 2021-03-24T22:01:15.499 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: : 2021-03-24T22:01:15.499 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 01 2021-03-24T22:01:15.499 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: : 2021-03-24T22:01:15.500 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 14 2021-03-24T22:01:15.500 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: . 2021-03-24T22:01:15.500 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 954681 2021-03-24T22:01:15.500 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: +0000 2021-03-24T22:01:15.501 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 2021-03-24T22:01:15.501 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: mon.c 2021-03-24T22:01:15.501 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: ( 2021-03-24T22:01:15.501 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: mon 2021-03-24T22:01:15.501 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: . 2021-03-24T22:01:15.502 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 2 2021-03-24T22:01:15.502 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: ) 2021-03-24T22:01:15.502 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 78 2021-03-24T22:01:15.502 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: : 2021-03-24T22:01:15.503 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: audit 2021-03-24T22:01:15.503 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 2021-03-24T22:01:15.503 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: [DBG] 2021-03-24T22:01:15.503 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 2021-03-24T22:01:15.504 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: from='client.? v1:172.21.15.125:0/4091977824' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2021-03-24T22:01:15.504 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 2021-03-24T22:01:15.504 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: audit 2021-03-24T22:01:15.504 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 2021-03-24T22:01:15.504 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 2021 2021-03-24T22:01:15.505 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: - 2021-03-24T22:01:15.505 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 03 2021-03-24T22:01:15.505 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: - 2021-03-24T22:01:15.505 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 24 2021-03-24T22:01:15.505 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: T 2021-03-24T22:01:15.506 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 22 2021-03-24T22:01:15.506 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: : 2021-03-24T22:01:15.506 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 01 2021-03-24T22:01:15.506 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: : 2021-03-24T22:01:15.507 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 14 2021-03-24T22:01:15.507 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: . 2021-03-24T22:01:15.507 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 955120 2021-03-24T22:01:15.507 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: +0000 2021-03-24T22:01:15.507 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 2021-03-24T22:01:15.508 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: mon.a 2021-03-24T22:01:15.508 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: ( 2021-03-24T22:01:15.508 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: mon 2021-03-24T22:01:15.508 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: . 2021-03-24T22:01:15.508 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 0 2021-03-24T22:01:15.509 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: ) 2021-03-24T22:01:15.509 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 1461 2021-03-24T22:01:15.509 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: : 2021-03-24T22:01:15.509 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: audit 2021-03-24T22:01:15.510 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 2021-03-24T22:01:15.510 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: [DBG] 2021-03-24T22:01:15.510 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 2021-03-24T22:01:15.510 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: from='client.35351 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2021-03-24T22:01:15.510 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 2021-03-24T22:01:15.511 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: cluster 2021-03-24T22:01:15.511 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 2021-03-24T22:01:15.511 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 2021 2021-03-24T22:01:15.511 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: - 2021-03-24T22:01:15.511 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 03 2021-03-24T22:01:15.512 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: - 2021-03-24T22:01:15.512 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 24 2021-03-24T22:01:15.512 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: T 2021-03-24T22:01:15.512 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 22 2021-03-24T22:01:15.513 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: : 2021-03-24T22:01:15.513 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 01 2021-03-24T22:01:15.513 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: : 2021-03-24T22:01:15.513 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 15 2021-03-24T22:01:15.513 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: . 2021-03-24T22:01:15.514 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 255761 2021-03-24T22:01:15.514 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: +0000 2021-03-24T22:01:15.514 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 2021-03-24T22:01:15.514 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: mon.a 2021-03-24T22:01:15.515 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: ( 2021-03-24T22:01:15.515 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: mon 2021-03-24T22:01:15.515 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: . 2021-03-24T22:01:15.515 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 0 2021-03-24T22:01:15.515 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: ) 2021-03-24T22:01:15.516 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 1462 2021-03-24T22:01:15.516 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: : 2021-03-24T22:01:15.516 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: cluster 2021-03-24T22:01:15.517 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 2021-03-24T22:01:15.517 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: [WRN] 2021-03-24T22:01:15.517 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 2021-03-24T22:01:15.517 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: Health check update: Degraded data redundancy: 196/1650 objects degraded (11.879%), 10 pgs degraded (PG_DEGRADED) 2021-03-24T22:01:15.518 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 2021-03-24T22:01:15.518 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: audit 2021-03-24T22:01:15.518 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 2021-03-24T22:01:15.518 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 2021 2021-03-24T22:01:15.519 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: - 2021-03-24T22:01:15.519 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 03 2021-03-24T22:01:15.519 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: - 2021-03-24T22:01:15.519 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 24 2021-03-24T22:01:15.519 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: T 2021-03-24T22:01:15.520 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 22 2021-03-24T22:01:15.520 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: : 2021-03-24T22:01:15.520 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 01 2021-03-24T22:01:15.520 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: : 2021-03-24T22:01:15.521 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 15 2021-03-24T22:01:15.521 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: . 2021-03-24T22:01:15.521 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 263256 2021-03-24T22:01:15.521 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: +0000 2021-03-24T22:01:15.521 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 2021-03-24T22:01:15.522 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: mon.a 2021-03-24T22:01:15.522 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: ( 2021-03-24T22:01:15.522 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: mon 2021-03-24T22:01:15.522 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: . 2021-03-24T22:01:15.523 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 0 2021-03-24T22:01:15.523 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: ) 2021-03-24T22:01:15.523 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 1463 2021-03-24T22:01:15.523 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: : 2021-03-24T22:01:15.523 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: audit 2021-03-24T22:01:15.524 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 2021-03-24T22:01:15.524 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: [INF] 2021-03-24T22:01:15.524 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 2021-03-24T22:01:15.524 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: from='client.35351 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2021-03-24T22:01:15.525 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 2021-03-24T22:01:15.525 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: cluster 2021-03-24T22:01:15.525 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 2021-03-24T22:01:15.525 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 2021 2021-03-24T22:01:15.525 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: - 2021-03-24T22:01:15.526 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 03 2021-03-24T22:01:15.526 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: - 2021-03-24T22:01:15.526 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 24 2021-03-24T22:01:15.526 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: T 2021-03-24T22:01:15.527 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 22 2021-03-24T22:01:15.527 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: : 2021-03-24T22:01:15.527 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 01 2021-03-24T22:01:15.527 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: : 2021-03-24T22:01:15.527 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 15 2021-03-24T22:01:15.528 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: . 2021-03-24T22:01:15.528 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 263340 2021-03-24T22:01:15.528 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: +0000 2021-03-24T22:01:15.528 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 2021-03-24T22:01:15.529 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: mon.a 2021-03-24T22:01:15.529 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: ( 2021-03-24T22:01:15.529 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: mon 2021-03-24T22:01:15.529 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: . 2021-03-24T22:01:15.529 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 0 2021-03-24T22:01:15.530 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: ) 2021-03-24T22:01:15.530 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 1464 2021-03-24T22:01:15.530 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: : 2021-03-24T22:01:15.530 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: cluster 2021-03-24T22:01:15.530 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 2021-03-24T22:01:15.531 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: [DBG] 2021-03-24T22:01:15.531 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 2021-03-24T22:01:15.531 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: osdmap e505: 8 total, 8 up, 7 in 2021-03-24T22:01:15.531 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[19936]: 2021-03-24T22:01:15.532 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: cluster 2021-03-24T22:01:15.532 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 2021-03-24T22:01:15.532 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: [WRN] 2021-03-24T22:01:15.533 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 2021-03-24T22:01:15.533 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: Health check update: Degraded data redundancy: 196/1650 objects degraded (11.879%), 10 pgs degraded (PG_DEGRADED) 2021-03-24T22:01:15.533 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 2021-03-24T22:01:15.533 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: audit 2021-03-24T22:01:15.534 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 2021-03-24T22:01:15.534 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 2021 2021-03-24T22:01:15.534 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: - 2021-03-24T22:01:15.534 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 03 2021-03-24T22:01:15.534 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: - 2021-03-24T22:01:15.535 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 24 2021-03-24T22:01:15.535 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: T 2021-03-24T22:01:15.535 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 22 2021-03-24T22:01:15.535 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: : 2021-03-24T22:01:15.536 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 01 2021-03-24T22:01:15.536 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: : 2021-03-24T22:01:15.536 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 15 2021-03-24T22:01:15.536 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: . 2021-03-24T22:01:15.536 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 263256 2021-03-24T22:01:15.537 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: +0000 2021-03-24T22:01:15.537 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 2021-03-24T22:01:15.537 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: mon.a 2021-03-24T22:01:15.537 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: ( 2021-03-24T22:01:15.537 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: mon 2021-03-24T22:01:15.538 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: . 2021-03-24T22:01:15.538 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 0 2021-03-24T22:01:15.538 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: ) 2021-03-24T22:01:15.538 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 1463 2021-03-24T22:01:15.539 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: : 2021-03-24T22:01:15.539 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: audit 2021-03-24T22:01:15.539 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 2021-03-24T22:01:15.539 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: [INF] 2021-03-24T22:01:15.539 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 2021-03-24T22:01:15.540 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: from='client.35351 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2021-03-24T22:01:15.540 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 2021-03-24T22:01:15.540 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: cluster 2021-03-24T22:01:15.540 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 2021-03-24T22:01:15.541 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 2021 2021-03-24T22:01:15.541 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: - 2021-03-24T22:01:15.541 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 03 2021-03-24T22:01:15.541 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: - 2021-03-24T22:01:15.541 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 24 2021-03-24T22:01:15.542 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: T 2021-03-24T22:01:15.542 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 22 2021-03-24T22:01:15.542 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: : 2021-03-24T22:01:15.542 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 01 2021-03-24T22:01:15.542 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: : 2021-03-24T22:01:15.543 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 15 2021-03-24T22:01:15.543 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: . 2021-03-24T22:01:15.543 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 263340 2021-03-24T22:01:15.543 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: +0000 2021-03-24T22:01:15.543 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 2021-03-24T22:01:15.544 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: mon.a 2021-03-24T22:01:15.544 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: ( 2021-03-24T22:01:15.544 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: mon 2021-03-24T22:01:15.544 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: . 2021-03-24T22:01:15.544 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 0 2021-03-24T22:01:15.545 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: ) 2021-03-24T22:01:15.545 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 1464 2021-03-24T22:01:15.545 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: : 2021-03-24T22:01:15.545 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: cluster 2021-03-24T22:01:15.545 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 2021-03-24T22:01:15.546 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: [DBG] 2021-03-24T22:01:15.546 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 2021-03-24T22:01:15.546 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: osdmap e505: 8 total, 8 up, 7 in 2021-03-24T22:01:15.546 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:15 smithi125 conmon[24927]: 2021-03-24T22:01:15.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:15.548 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:15.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:01:15.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:15 smithi142 conmon[20125]: audit 2021-03-24T22:01:14.954681+0000 2021-03-24T22:01:15.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:15 smithi142 conmon[20125]: mon.c (mon.2) 78 : audit [DBG] from='client.? v1:172.21.15.125:0/4091977824' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2021-03-24T22:01:15.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:15 smithi142 conmon[20125]: audit 2021-03-24T22:01:14. 2021-03-24T22:01:15.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:15 smithi142 conmon[20125]: 955120+0000 mon.a (mon.0) 1461 : audit [DBG] from='client.35351 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2021-03-24T22:01:15.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:15 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:01:15.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:15 smithi142 conmon[20125]: -03-24T22:01:15.255761+0000 mon.a 2021-03-24T22:01:15.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:15 smithi142 conmon[20125]: (mon.0) 1462 : cluster [WRN] Health check update: Degraded data redundancy: 196/1650 objects degraded (11.879%), 10 pgs degraded (PG_DEGRADED) 2021-03-24T22:01:15.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:15 smithi142 conmon[20125]: audit 2021- 2021-03-24T22:01:15.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:15 smithi142 conmon[20125]: 03-24T22:01:15.263256+0000 mon.a (mon.0) 2021-03-24T22:01:15.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:15 smithi142 conmon[20125]: 1463 : audit [INF] from='client.35351 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2021-03-24T22:01:15.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:15 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:01:15.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:15 smithi142 conmon[20125]: -24T22:01:15.263340+0000 mon.a (mon 2021-03-24T22:01:15.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:15 smithi142 conmon[20125]: .0) 1464 : cluster [DBG] osdmap e505: 8 total, 8 up, 7 in 2021-03-24T22:01:15.649 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:15.663 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-24T22:01:15.703 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:15.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:15.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:01:15.807 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:15.857 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:15.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:15.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:01:15.963 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:16.016 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:16.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:16.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:01:16.121 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:16.169 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:16.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:16.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:01:16.275 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:16.276 INFO:tasks.rados.rados.0.smithi125.stdout:1891: done (3 left) 2021-03-24T22:01:16.277 INFO:tasks.rados.rados.0.smithi125.stdout:1892: done (2 left) 2021-03-24T22:01:16.277 INFO:tasks.rados.rados.0.smithi125.stdout:1894: done (1 left) 2021-03-24T22:01:16.277 INFO:tasks.rados.rados.0.smithi125.stdout:1895: write (excl) oid 8 current snap is 208 2021-03-24T22:01:16.277 INFO:tasks.rados.rados.0.smithi125.stdout:1895: seq_num 669 ranges {443883=464292,1365277=511220,2572335=671548} 2021-03-24T22:01:16.281 INFO:tasks.rados.rados.0.smithi125.stdout:1895: writing smithi12556470-8 from 443883 to 908175 tid 1 2021-03-24T22:01:16.287 INFO:tasks.rados.rados.0.smithi125.stdout:1895: writing smithi12556470-8 from 1365277 to 1876497 tid 2 2021-03-24T22:01:16.298 INFO:tasks.rados.rados.0.smithi125.stdout:1895: writing smithi12556470-8 from 2572335 to 3243883 tid 3 2021-03-24T22:01:16.299 INFO:tasks.rados.rados.0.smithi125.stdout:1896: rollback oid 18 current snap is 208 2021-03-24T22:01:16.299 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 18 to 185 2021-03-24T22:01:16.299 INFO:tasks.rados.rados.0.smithi125.stdout:1897: write (excl) oid 2 current snap is 208 2021-03-24T22:01:16.299 INFO:tasks.rados.rados.0.smithi125.stdout:1897: seq_num 670 ranges {460690=655894,1540112=771231,2716111=539778} 2021-03-24T22:01:16.305 INFO:tasks.rados.rados.0.smithi125.stdout:1897: writing smithi12556470-2 from 460690 to 1116584 tid 1 2021-03-24T22:01:16.313 INFO:tasks.rados.rados.0.smithi125.stdout:1897: writing smithi12556470-2 from 1540112 to 2311343 tid 2 2021-03-24T22:01:16.322 INFO:tasks.rados.rados.0.smithi125.stdout:1897: writing smithi12556470-2 from 2716111 to 3255889 tid 3 2021-03-24T22:01:16.322 INFO:tasks.rados.rados.0.smithi125.stdout:1898: write oid 49 current snap is 208 2021-03-24T22:01:16.322 INFO:tasks.rados.rados.0.smithi125.stdout:1896: finishing rollback tid 0 to smithi12556470-18 2021-03-24T22:01:16.322 INFO:tasks.rados.rados.0.smithi125.stdout:1898: seq_num 671 ranges {477497=731143,1998594=514889,3143534=133963} 2021-03-24T22:01:16.323 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:16.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:16.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:01:16.330 INFO:tasks.rados.rados.0.smithi125.stdout:1898: writing smithi12556470-49 from 477497 to 1208640 tid 1 2021-03-24T22:01:16.339 INFO:tasks.rados.rados.0.smithi125.stdout:1898: writing smithi12556470-49 from 1998594 to 2513483 tid 2 2021-03-24T22:01:16.344 INFO:tasks.rados.rados.0.smithi125.stdout:1898: writing smithi12556470-49 from 3143534 to 3277497 tid 3 2021-03-24T22:01:16.345 INFO:tasks.rados.rados.0.smithi125.stdout:1899: read oid 12 snap -1 2021-03-24T22:01:16.345 INFO:tasks.rados.rados.0.smithi125.stdout:1899: expect (ObjNum 618 snap 187 seq_num 618) 2021-03-24T22:01:16.345 INFO:tasks.rados.rados.0.smithi125.stdout:1900: read oid 22 snap -1 2021-03-24T22:01:16.345 INFO:tasks.rados.rados.0.smithi125.stdout:1900: expect deleted 2021-03-24T22:01:16.346 INFO:tasks.rados.rados.0.smithi125.stdout:1901: snap_create 2021-03-24T22:01:16.346 INFO:tasks.rados.rados.0.smithi125.stdout:1896: finishing rollback tid 1 to smithi12556470-18 2021-03-24T22:01:16.346 INFO:tasks.rados.rados.0.smithi125.stdout:1896: finishing rollback tid 2 to smithi12556470-18 2021-03-24T22:01:16.346 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 18 v 239 (ObjNum 588 snap 177 seq_num 588) dirty exists 2021-03-24T22:01:16.346 INFO:tasks.rados.rados.0.smithi125.stdout:1897: finishing write tid 1 to smithi12556470-2 2021-03-24T22:01:16.347 INFO:tasks.rados.rados.0.smithi125.stdout:1897: finishing write tid 2 to smithi12556470-2 2021-03-24T22:01:16.347 INFO:tasks.rados.rados.0.smithi125.stdout:1897: finishing write tid 3 to smithi12556470-2 2021-03-24T22:01:16.347 INFO:tasks.rados.rados.0.smithi125.stdout:1897: finishing write tid 4 to smithi12556470-2 2021-03-24T22:01:16.347 INFO:tasks.rados.rados.0.smithi125.stdout:1897: finishing write tid 5 to smithi12556470-2 2021-03-24T22:01:16.348 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 2 v 674 (ObjNum 670 snap 208 seq_num 670) dirty exists 2021-03-24T22:01:16.348 INFO:tasks.rados.rados.0.smithi125.stdout:1897: left oid 2 (ObjNum 670 snap 208 seq_num 670) 2021-03-24T22:01:16.348 INFO:tasks.rados.rados.0.smithi125.stdout:1899: expect (ObjNum 618 snap 187 seq_num 618) 2021-03-24T22:01:16.428 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:16.476 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:16.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:16.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:01:16.581 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:16.589 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[19936]: cluster 2021-03-24T22:01:16.589 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[19936]: 2021-03-24T22:01:16.590 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[19936]: 2021 2021-03-24T22:01:16.590 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[19936]: - 2021-03-24T22:01:16.590 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[19936]: 03-24T22:01:14.877739+0000 2021-03-24T22:01:16.591 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[19936]: mgr.y (mgr.14144) 524 : cluster [DBG] pgmap v919: 168 pgs: 1 active+recovery_wait+remapped, 1 peering, 1 active+remapped+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 155 active+clean; 466 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 12 MiB/s rd, 652 KiB/s wr, 12 op/s; 24/1629 objects degraded (1.473%); 177/1629 objects misplaced (10.866%); 31 MiB/s, 55 objects/s recovering 2021-03-24T22:01:16.591 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:01:16.591 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[19936]: T22:01:16.273664+0000 mon.a (mon 2021-03-24T22:01:16.591 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[19936]: .0) 1465 : cluster [DBG] osdmap e506: 8 total, 8 up, 7 in 2021-03-24T22:01:16.592 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: cluster 2021-03-24T22:01:16.592 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: 2021-03-24T22:01:16.592 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: 2021 2021-03-24T22:01:16.593 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: - 2021-03-24T22:01:16.593 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: 03 2021-03-24T22:01:16.593 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: - 2021-03-24T22:01:16.593 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: 24 2021-03-24T22:01:16.593 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: T 2021-03-24T22:01:16.594 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: 22 2021-03-24T22:01:16.594 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: : 2021-03-24T22:01:16.594 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: 01 2021-03-24T22:01:16.594 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: : 2021-03-24T22:01:16.594 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: 14 2021-03-24T22:01:16.595 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: . 2021-03-24T22:01:16.595 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: 877739 2021-03-24T22:01:16.595 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: +0000 2021-03-24T22:01:16.595 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: 2021-03-24T22:01:16.595 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: mgr.y 2021-03-24T22:01:16.596 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: ( 2021-03-24T22:01:16.596 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: mgr 2021-03-24T22:01:16.596 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: . 2021-03-24T22:01:16.596 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: 14144 2021-03-24T22:01:16.596 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: ) 2021-03-24T22:01:16.597 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: 524 2021-03-24T22:01:16.597 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: : 2021-03-24T22:01:16.597 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: cluster 2021-03-24T22:01:16.598 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: 2021-03-24T22:01:16.598 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: [DBG] 2021-03-24T22:01:16.599 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: 2021-03-24T22:01:16.599 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: pgmap v919: 168 pgs: 1 active+recovery_wait+remapped, 1 peering, 1 active+remapped+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 155 active+clean; 466 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 12 MiB/s rd, 652 KiB/s wr, 12 op/s; 24/1629 objects degraded (1.473%); 177/1629 objects misplaced (10.866%); 31 MiB/s, 55 objects/s recovering 2021-03-24T22:01:16.599 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: 2021-03-24T22:01:16.600 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: cluster 2021-03-24T22:01:16.600 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: 2021-03-24T22:01:16.601 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: 2021 2021-03-24T22:01:16.601 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: - 2021-03-24T22:01:16.601 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: 03 2021-03-24T22:01:16.602 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: - 2021-03-24T22:01:16.603 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: 24 2021-03-24T22:01:16.603 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: T 2021-03-24T22:01:16.603 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: 22 2021-03-24T22:01:16.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: : 2021-03-24T22:01:16.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: 01 2021-03-24T22:01:16.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: : 2021-03-24T22:01:16.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: 16 2021-03-24T22:01:16.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: . 2021-03-24T22:01:16.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: 273664 2021-03-24T22:01:16.606 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: +0000 2021-03-24T22:01:16.606 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: 2021-03-24T22:01:16.606 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: mon.a 2021-03-24T22:01:16.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: ( 2021-03-24T22:01:16.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: mon 2021-03-24T22:01:16.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: . 2021-03-24T22:01:16.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: 0 2021-03-24T22:01:16.608 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: ) 2021-03-24T22:01:16.608 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: 1465 2021-03-24T22:01:16.608 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: : 2021-03-24T22:01:16.608 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: cluster 2021-03-24T22:01:16.609 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: 2021-03-24T22:01:16.609 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: [DBG] 2021-03-24T22:01:16.609 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: 2021-03-24T22:01:16.609 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: osdmap e506: 8 total, 8 up, 7 in 2021-03-24T22:01:16.610 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:16 smithi125 conmon[24927]: 2021-03-24T22:01:16.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:16 smithi142 conmon[20125]: cluster 2021-03-24T22:01:14.877739+0000 mgr.y (mgr.14144) 524 : 2021-03-24T22:01:16.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:16 smithi142 conmon[20125]: cluster [DBG] pgmap v919: 168 pgs: 1 active+recovery_wait+remapped, 1 peering, 1 active+remapped+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 155 active+clean; 466 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 12 MiB/s rd, 652 KiB/s wr, 12 op/s; 24/1629 objects degraded (1.473%); 177/1629 objects misplaced (10.866%); 31 MiB/s, 55 objects/s recovering 2021-03-24T22:01:16.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:16 smithi142 conmon[20125]: cluster 2021-03-24T22:01:16.273664+0000 mon.a (mon.0) 1465 : cluster [DBG] osdmap e506: 8 total, 8 up, 7 in 2021-03-24T22:01:16.629 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:16.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:16.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:01:16.735 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:16.758 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:01:16.781 INFO:tasks.thrashosds.thrasher:Added osd 1 2021-03-24T22:01:16.786 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:16.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:16.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:01:16.827 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:01:16.894 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:16.953 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:16.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:16.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:01:17.059 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:17.106 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:17.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:17.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:01:17.213 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:17.268 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:17.272 INFO:tasks.rados.rados.0.smithi125.stdout:1896: done (7 left) 2021-03-24T22:01:17.272 INFO:tasks.rados.rados.0.smithi125.stdout:1897: done (6 left) 2021-03-24T22:01:17.273 INFO:tasks.rados.rados.0.smithi125.stdout:1899: done (5 left) 2021-03-24T22:01:17.273 INFO:tasks.rados.rados.0.smithi125.stdout:1900: done (4 left) 2021-03-24T22:01:17.274 INFO:tasks.rados.rados.0.smithi125.stdout:1901: done (3 left) 2021-03-24T22:01:17.274 INFO:tasks.rados.rados.0.smithi125.stdout:1902: write (excl) oid 32 current snap is 209 2021-03-24T22:01:17.275 INFO:tasks.rados.rados.0.smithi125.stdout:1902: seq_num 672 ranges {494304=406392,1540723=774900,2770957=523347} 2021-03-24T22:01:17.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:17.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:01:17.279 INFO:tasks.rados.rados.0.smithi125.stdout:1902: writing smithi12556470-32 from 494304 to 900696 tid 1 2021-03-24T22:01:17.287 INFO:tasks.rados.rados.0.smithi125.stdout:1902: writing smithi12556470-32 from 1540723 to 2315623 tid 2 2021-03-24T22:01:17.293 INFO:tasks.rados.rados.0.smithi125.stdout:1902: writing smithi12556470-32 from 2770957 to 3294304 tid 3 2021-03-24T22:01:17.293 INFO:tasks.rados.rados.0.smithi125.stdout:1903: delete oid 20 current snap is 209 2021-03-24T22:01:17.301 INFO:tasks.rados.rados.0.smithi125.stdout:1903: done (4 left) 2021-03-24T22:01:17.301 INFO:tasks.rados.rados.0.smithi125.stdout:1904: delete oid 24 current snap is 209 2021-03-24T22:01:17.304 INFO:tasks.rados.rados.0.smithi125.stdout:1904: done (4 left) 2021-03-24T22:01:17.305 INFO:tasks.rados.rados.0.smithi125.stdout:1905: read oid 35 snap -1 2021-03-24T22:01:17.305 INFO:tasks.rados.rados.0.smithi125.stdout:1905: expect (ObjNum 639 snap 196 seq_num 639) 2021-03-24T22:01:17.305 INFO:tasks.rados.rados.0.smithi125.stdout:1906: snap_create 2021-03-24T22:01:17.322 INFO:tasks.rados.rados.0.smithi125.stdout:1905: expect (ObjNum 639 snap 196 seq_num 639) 2021-03-24T22:01:17.378 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:17.434 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:17.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:17.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:01:17.447 INFO:tasks.rados.rados.0.smithi125.stdout:1902: finishing write tid 1 to smithi12556470-32 2021-03-24T22:01:17.448 INFO:tasks.rados.rados.0.smithi125.stdout:1895: finishing write tid 1 to smithi12556470-8 2021-03-24T22:01:17.448 INFO:tasks.rados.rados.0.smithi125.stdout:1902: finishing write tid 2 to smithi12556470-32 2021-03-24T22:01:17.448 INFO:tasks.rados.rados.0.smithi125.stdout:1902: finishing write tid 3 to smithi12556470-32 2021-03-24T22:01:17.449 INFO:tasks.rados.rados.0.smithi125.stdout:1902: finishing write tid 4 to smithi12556470-32 2021-03-24T22:01:17.449 INFO:tasks.rados.rados.0.smithi125.stdout:1902: finishing write tid 5 to smithi12556470-32 2021-03-24T22:01:17.449 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 32 v 913 (ObjNum 672 snap 209 seq_num 672) dirty exists 2021-03-24T22:01:17.449 INFO:tasks.rados.rados.0.smithi125.stdout:1902: left oid 32 (ObjNum 672 snap 209 seq_num 672) 2021-03-24T22:01:17.450 INFO:tasks.rados.rados.0.smithi125.stdout:1895: finishing write tid 2 to smithi12556470-8 2021-03-24T22:01:17.450 INFO:tasks.rados.rados.0.smithi125.stdout:1895: finishing write tid 3 to smithi12556470-8 2021-03-24T22:01:17.450 INFO:tasks.rados.rados.0.smithi125.stdout:1895: finishing write tid 4 to smithi12556470-8 2021-03-24T22:01:17.451 INFO:tasks.rados.rados.0.smithi125.stdout:1895: finishing write tid 5 to smithi12556470-8 2021-03-24T22:01:17.451 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 8 v 124 (ObjNum 669 snap 208 seq_num 669) dirty exists 2021-03-24T22:01:17.451 INFO:tasks.rados.rados.0.smithi125.stdout:1895: left oid 8 (ObjNum 669 snap 208 seq_num 669) 2021-03-24T22:01:17.452 INFO:tasks.rados.rados.0.smithi125.stdout:1898: finishing write tid 1 to smithi12556470-49 2021-03-24T22:01:17.452 INFO:tasks.rados.rados.0.smithi125.stdout:1898: finishing write tid 2 to smithi12556470-49 2021-03-24T22:01:17.452 INFO:tasks.rados.rados.0.smithi125.stdout:1898: finishing write tid 3 to smithi12556470-49 2021-03-24T22:01:17.452 INFO:tasks.rados.rados.0.smithi125.stdout:1898: finishing write tid 4 to smithi12556470-49 2021-03-24T22:01:17.453 INFO:tasks.rados.rados.0.smithi125.stdout:1898: finishing write tid 5 to smithi12556470-49 2021-03-24T22:01:17.453 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 49 v 635 (ObjNum 671 snap 208 seq_num 671) dirty exists 2021-03-24T22:01:17.453 INFO:tasks.rados.rados.0.smithi125.stdout:1898: left oid 49 (ObjNum 671 snap 208 seq_num 671) 2021-03-24T22:01:17.454 INFO:tasks.rados.rados.0.smithi125.stdout:1893: finishing copy_from to smithi12556470-48 2021-03-24T22:01:17.454 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 48 v 558 (ObjNum 652 snap 198 seq_num 652) dirty exists 2021-03-24T22:01:17.454 INFO:tasks.rados.rados.0.smithi125.stdout:1893: finishing copy_from racing read to smithi12556470-48 2021-03-24T22:01:17.476 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[24927]: cluster 2021-03-24T22:01:15.740472 2021-03-24T22:01:17.476 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[24927]: +0000 osd.3 (osd.3) 107 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:01:17.477 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[24927]: cluster 2021-03-24T22:01:15. 2021-03-24T22:01:17.477 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[24927]: 742856+0000 osd.3 (osd.3) 108 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:01:17.477 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:01:17.478 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[24927]: 01:16.270726+0000 osd.0 (osd.0) 106 2021-03-24T22:01:17.478 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[24927]: : cluster [DBG] 4.6 starting backfill to osd.1 from (0'0,0'0] MAX to 69'16 2021-03-24T22:01:17.478 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[24927]: cluster 2021-03-24T22 2021-03-24T22:01:17.478 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[24927]: :01:16.270886+0000 osd.0 (osd.0) 2021-03-24T22:01:17.479 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[24927]: 107 : cluster [INF] 4.6 continuing backfill to osd.7 from (69'16,69'16] MIN to 69'16 2021-03-24T22:01:17.479 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:01:17.479 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[24927]: 01:16.276337+0000 osd.3 (osd.3) 2021-03-24T22:01:17.479 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[24927]: 109 : cluster [DBG] 4.f starting backfill to osd.1 from (0'0,0'0] MAX to 274'26 2021-03-24T22:01:17.479 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:01:17.480 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[24927]: 01:16.279051+0000 osd.3 (osd.3) 110 2021-03-24T22:01:17.480 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[24927]: : cluster [DBG] 4.c starting backfill to osd.1 from (0'0,0'0] MAX to 69'15 2021-03-24T22:01:17.480 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:01:17.480 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[24927]: 01:16.279901+0000 osd.5 (osd.5) 2021-03-24T22:01:17.481 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[24927]: 15 : cluster [DBG] 4.a starting backfill to osd.1 from (0'0,0'0] MAX to 69'23 2021-03-24T22:01:17.481 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:01:17.481 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[24927]: 01:16.280027+0000 osd.5 (osd.5) 16 2021-03-24T22:01:17.481 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[24927]: : cluster [INF] 4.a continuing backfill to osd.7 from (69'23,69'23] MIN to 69'23 2021-03-24T22:01:17.482 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[24927]: cluster 2021-03-24T22:01 2021-03-24T22:01:17.482 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[24927]: :16.281158+0000 osd.5 (osd.5) 17 2021-03-24T22:01:17.482 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[24927]: : cluster [DBG] 4.7 starting backfill to osd.1 from (0'0,0'0] MAX to 89'22 2021-03-24T22:01:17.482 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:01:17.483 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[24927]: 01:17.266714+0000 mon.a (mon.0) 1466 2021-03-24T22:01:17.483 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[24927]: : cluster [DBG] osdmap e507: 8 total, 8 up, 7 in 2021-03-24T22:01:17.483 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[19936]: cluster 2021-03-24T22:01:15. 2021-03-24T22:01:17.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[19936]: 740472+0000 osd.3 (osd.3) 107 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:01:17.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[19936]: cluster 2021-03-24T22:01:15.742856+0000 osd.3 (osd.3) 108 : cluster [DBG] 2021-03-24T22:01:17.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[19936]: purged_snaps scrub ok 2021-03-24T22:01:17.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[19936]: cluster 2021-03-24T22:01:16. 2021-03-24T22:01:17.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[19936]: 270726+0000 osd.0 (osd.0) 106 : cluster [DBG] 4.6 starting backfill to osd.1 from (0'0,0'0] MAX to 69'16 2021-03-24T22:01:17.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:01:17.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[19936]: 24T22:01:16.270886+0000 osd.0 ( 2021-03-24T22:01:17.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[19936]: osd.0) 107 : cluster [INF] 4.6 continuing backfill to osd.7 from (69'16,69'16] MIN to 69'16 2021-03-24T22:01:17.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:01:17.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[19936]: 24T22:01:16.276337+0000 osd.3 2021-03-24T22:01:17.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[19936]: (osd.3) 109 : cluster [DBG] 4.f starting backfill to osd.1 from (0'0,0'0] MAX to 274'26 2021-03-24T22:01:17.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:01:17.487 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[19936]: 24T22:01:16.279051+0000 osd.3 2021-03-24T22:01:17.487 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[19936]: (osd.3) 110 : cluster [DBG] 4.c starting backfill to osd.1 from (0'0,0'0] MAX to 69'15 2021-03-24T22:01:17.487 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:01:17.487 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[19936]: 24T22:01:16.279901+0000 osd.5 ( 2021-03-24T22:01:17.488 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[19936]: osd.5) 15 : cluster [DBG] 4.a starting backfill to osd.1 from (0'0,0'0] MAX to 69'23 2021-03-24T22:01:17.488 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:01:17.488 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[19936]: T22:01:16.280027+0000 osd.5 (osd 2021-03-24T22:01:17.489 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[19936]: .5) 16 : cluster [INF] 4.a continuing backfill to osd.7 from (69'23,69'23] MIN to 69'23 2021-03-24T22:01:17.489 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[19936]: cluster 2021-03-24T22 2021-03-24T22:01:17.489 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[19936]: :01:16.281158+0000 osd.5 (osd.5 2021-03-24T22:01:17.489 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[19936]: ) 17 : cluster [DBG] 4.7 starting backfill to osd.1 from (0'0,0'0] MAX to 89'22 2021-03-24T22:01:17.490 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:01:17.490 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[19936]: 01:17.266714+0000 mon.a (mon.0) 1466 : cluster [DBG] osdmap e507: 8 total, 8 up, 7 in 2021-03-24T22:01:17.490 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:17 smithi125 conmon[19936]: 2021-03-24T22:01:17.542 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:17.556 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:17 smithi142 conmon[20125]: cluster 2021-03-24T22:01:15.740472+0000 2021-03-24T22:01:17.556 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:17 smithi142 conmon[20125]: osd.3 (osd.3) 107 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:01:17.556 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:17 smithi142 conmon[20125]: cluster 2021-03-24T22:01:15.742856+0000 osd.3 (osd.3) 108 : cluster [DBG] 2021-03-24T22:01:17.556 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:17 smithi142 conmon[20125]: purged_snaps scrub ok 2021-03-24T22:01:17.557 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:17 smithi142 conmon[20125]: cluster 2021-03-24T22: 2021-03-24T22:01:17.557 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:17 smithi142 conmon[20125]: 01:16.270726+0000 osd.0 (osd.0) 106 : cluster [DBG] 2021-03-24T22:01:17.557 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:17 smithi142 conmon[20125]: 4.6 starting backfill to osd.1 from (0'0,0'0] MAX to 69'16 2021-03-24T22:01:17.558 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:17 smithi142 conmon[20125]: cluster 2021-03-24T22:01:17.558 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:17 smithi142 conmon[20125]: 2021-03-24T22:01:16.270886+0000 osd.0 2021-03-24T22:01:17.558 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:17 smithi142 conmon[20125]: (osd.0) 107 : cluster [INF] 4.6 continuing backfill to osd.7 from (69'16,69'16] MIN to 69'16 2021-03-24T22:01:17.559 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:17 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:01:17.559 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:17 smithi142 conmon[20125]: 03-24T22:01:16.276337+0000 2021-03-24T22:01:17.559 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:17 smithi142 conmon[20125]: osd.3 (osd.3) 109 : cluster [DBG] 4.f starting backfill to osd.1 from (0'0,0'0] MAX to 274'26 2021-03-24T22:01:17.560 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:17 smithi142 conmon[20125]: 2021-03-24T22:01:17.560 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:17 smithi142 conmon[20125]: cluster 2021-03-24T22:01:17.560 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:17 smithi142 conmon[20125]: 2021-03-24T22:01:16.279051+0000 osd.3 2021-03-24T22:01:17.560 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:17 smithi142 conmon[20125]: (osd.3) 110 : cluster [DBG] 4.c starting backfill to osd.1 from (0'0,0'0] MAX to 69'15 2021-03-24T22:01:17.560 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:17 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:01:17.561 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:17 smithi142 conmon[20125]: 24T22:01:16.279901+0000 osd.5 ( 2021-03-24T22:01:17.561 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:17 smithi142 conmon[20125]: osd.5) 15 : cluster [DBG] 4.a starting backfill to osd.1 from (0'0,0'0] MAX to 69'23 2021-03-24T22:01:17.561 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:17 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:01:17.561 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:17 smithi142 conmon[20125]: 03-24T22:01:16.280027+0000 2021-03-24T22:01:17.562 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:17 smithi142 conmon[20125]: osd.5 (osd.5) 16 : cluster [INF] 4.a continuing backfill to osd.7 from (69'23,69'23] MIN to 69'23 2021-03-24T22:01:17.562 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:17 smithi142 conmon[20125]: 2021-03-24T22:01:17.562 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:17 smithi142 conmon[20125]: cluster 2021-03-24T22:01:17.563 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:17 smithi142 conmon[20125]: 2021-03-24T22:01:16.281158+0000 osd.5 2021-03-24T22:01:17.563 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:17 smithi142 conmon[20125]: (osd.5) 17 : cluster [DBG] 4.7 starting backfill to osd.1 from (0'0,0'0] MAX to 89'22 2021-03-24T22:01:17.563 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:17 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:01:17.563 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:17 smithi142 conmon[20125]: -24T22:01:17.266714+0000 mon.a 2021-03-24T22:01:17.563 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:17 smithi142 conmon[20125]: (mon.0) 1466 : cluster [DBG] osdmap e507: 8 total, 8 up, 7 in 2021-03-24T22:01:17.600 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:17.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:17.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:01:17.707 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:17.759 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:17.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:17.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:01:17.864 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:17.909 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:17.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:17.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:01:18.015 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:18.059 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:18.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:18.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:01:18.163 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:18.217 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:18.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:18.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:01:18.274 INFO:tasks.rados.rados.0.smithi125.stdout:1893: done (5 left) 2021-03-24T22:01:18.275 INFO:tasks.rados.rados.0.smithi125.stdout:1895: done (4 left) 2021-03-24T22:01:18.275 INFO:tasks.rados.rados.0.smithi125.stdout:1898: done (3 left) 2021-03-24T22:01:18.275 INFO:tasks.rados.rados.0.smithi125.stdout:1902: done (2 left) 2021-03-24T22:01:18.276 INFO:tasks.rados.rados.0.smithi125.stdout:1905: done (1 left) 2021-03-24T22:01:18.276 INFO:tasks.rados.rados.0.smithi125.stdout:1906: done (0 left) 2021-03-24T22:01:18.276 INFO:tasks.rados.rados.0.smithi125.stdout:1907: rollback oid 7 current snap is 210 2021-03-24T22:01:18.276 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 7 to 187 2021-03-24T22:01:18.277 INFO:tasks.rados.rados.0.smithi125.stdout:1908: read oid 15 snap -1 2021-03-24T22:01:18.277 INFO:tasks.rados.rados.0.smithi125.stdout:1908: expect deleted 2021-03-24T22:01:18.277 INFO:tasks.rados.rados.0.smithi125.stdout:1909: snap_remove snap 185 2021-03-24T22:01:18.323 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:18.332 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: cluster 2021-03-24T22:01:18.333 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:01:18.333 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[19936]: T22:01:16.878315+0000 mgr.y (mgr.14144) 525 : cluster [DBG] pgmap v922: 168 pgs: 1 activating, 1 remapped+peering, 1 active+remapped+backfilling, 2 active+recovery_wait+degraded, 1 active+recovery_wait+remapped, 1 peering, 1 active+remapped+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 151 active+clean; 466 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 12 MiB/s rd, 652 KiB/s wr, 13 op/s; 32/1629 objects degraded (1.964%); 188/1629 objects misplaced (11.541%); 30 MiB/s, 56 objects/s recovering 2021-03-24T22:01:18.334 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T22:01:18.334 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[19936]: -24T22:01:17.269073+0000 osd.0 2021-03-24T22:01:18.334 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[19936]: (osd.0) 108 2021-03-24T22:01:18.335 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[19936]: : cluster [DBG] 4.9 starting backfill to osd.1 from (0'0,0'0] MAX to 69'20 2021-03-24T22:01:18.335 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[19936]: 2021-03-24T22:01:18.335 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[19936]: cluster 2021-03-24T22:01:18.335 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[19936]: 2021- 2021-03-24T22:01:18.335 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[19936]: 03-24T22:01:17.272200 2021-03-24T22:01:18.336 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[19936]: +0000 osd.2 (osd. 2021-03-24T22:01:18.336 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[19936]: 2) 83 : cluster 2021-03-24T22:01:18.336 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[19936]: [DBG] 4.d starting backfill to osd.1 from (0'0,0'0] MAX to 69'21 2021-03-24T22:01:18.336 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[19936]: cluster 2021-03-24T22:01:18.337 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[19936]: 2021- 2021-03-24T22:01:18.337 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[19936]: 03-24T22:01 2021-03-24T22:01:18.337 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[19936]: :17.278957+0000 2021-03-24T22:01:18.338 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[19936]: osd.3 (osd. 2021-03-24T22:01:18.338 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[19936]: 3) 111 : cluster 2021-03-24T22:01:18.338 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[19936]: [DBG] 4.e starting backfill to osd.1 from (0'0,0'0] MAX to 89'27 2021-03-24T22:01:18.339 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[19936]: cluster 2021-03-24T22:01:18.339 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[19936]: 2021 2021-03-24T22:01:18.369 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:18.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:18.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:01:18.474 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:18.487 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:18 smithi142 conmon[20125]: cluster 2021-03-24T22:01:16.878315+0000 mgr.y ( 2021-03-24T22:01:18.487 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:18 smithi142 conmon[20125]: mgr.14144) 525 : cluster [DBG] pgmap v922: 168 pgs: 1 activating, 1 remapped+peering, 1 active+remapped+backfilling, 2 active+recovery_wait+degraded, 1 active+recovery_wait+remapped, 1 peering, 1 active+remapped+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 151 active+clean; 466 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 12 MiB/s rd, 652 KiB/s wr, 13 op/s; 32/1629 objects degraded (1.964%); 188/1629 objects misplaced (11.541%); 30 MiB/s, 56 objects/s recovering 2021-03-24T22:01:18.488 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:18 smithi142 conmon[20125]: cluster 2021-03-24T22:01:17.269073+0000 osd.0 (osd.0) 108 : cluster [DBG] 4.9 starting backfill to osd.1 from (0'0,0'0] MAX to 69'20 2021-03-24T22:01:18.488 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:18 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:01:18.488 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:18 smithi142 conmon[20125]: -03-24T22:01:17.272200+0000 osd.2 (osd.2) 83 : cluster [DBG] 4.d starting backfill to osd.1 from (0'0,0'0] MAX to 69'21 2021-03-24T22:01:18.488 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:18 smithi142 conmon[20125]: cluster 2021-03-24T22:01:17.278957+0000 2021-03-24T22:01:18.489 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:18 smithi142 conmon[20125]: osd.3 (osd.3) 111 : cluster [DBG] 4.e starting backfill to osd.1 from (0'0,0'0] MAX to 89'27 2021-03-24T22:01:18.489 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:18 smithi142 conmon[20125]: cluster 2021-03-24T22:01:17 2021-03-24T22:01:18.489 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:18 smithi142 conmon[20125]: .286733+0000 osd.0 (osd.0) 109 : cluster [DBG] 4.b starting backfill to osd.1 from (0'0,0'0] MAX to 91'9 2021-03-24T22:01:18.489 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:18 smithi142 conmon[20125]: 2021-03-24T22:01:18.490 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:18 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:01:18.490 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:18 smithi142 conmon[20125]: 22:01:18.267822+0000 mon.a (mon.0) 1467 : cluster [DBG] osdmap e508: 8 total, 8 up, 7 in 2021-03-24T22:01:18.525 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:18.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:18.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:01:18.594 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 2021-03-24T22:01:18.594 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 2021 2021-03-24T22:01:18.595 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: - 2021-03-24T22:01:18.595 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 03 2021-03-24T22:01:18.595 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: - 2021-03-24T22:01:18.595 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 24 2021-03-24T22:01:18.596 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: T 2021-03-24T22:01:18.596 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 22 2021-03-24T22:01:18.596 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: : 2021-03-24T22:01:18.596 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 01 2021-03-24T22:01:18.597 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: : 2021-03-24T22:01:18.597 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 16 2021-03-24T22:01:18.597 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: . 2021-03-24T22:01:18.597 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 878315 2021-03-24T22:01:18.598 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: +0000 2021-03-24T22:01:18.598 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 2021-03-24T22:01:18.598 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: mgr.y 2021-03-24T22:01:18.598 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: ( 2021-03-24T22:01:18.599 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: mgr 2021-03-24T22:01:18.599 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: . 2021-03-24T22:01:18.599 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 14144 2021-03-24T22:01:18.599 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: ) 2021-03-24T22:01:18.599 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 525 2021-03-24T22:01:18.600 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: : 2021-03-24T22:01:18.600 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: cluster 2021-03-24T22:01:18.600 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 2021-03-24T22:01:18.600 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: [DBG] 2021-03-24T22:01:18.601 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 2021-03-24T22:01:18.601 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: pgmap v922: 168 pgs: 1 activating, 1 remapped+peering, 1 active+remapped+backfilling, 2 active+recovery_wait+degraded, 1 active+recovery_wait+remapped, 1 peering, 1 active+remapped+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 151 active+clean; 466 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 12 MiB/s rd, 652 KiB/s wr, 13 op/s; 32/1629 objects degraded (1.964%); 188/1629 objects misplaced (11.541%); 30 MiB/s, 56 objects/s recovering 2021-03-24T22:01:18.601 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 2021-03-24T22:01:18.601 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: cluster 2021-03-24T22:01:18.601 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 2021-03-24T22:01:18.602 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 2021 2021-03-24T22:01:18.602 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: - 2021-03-24T22:01:18.602 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 03 2021-03-24T22:01:18.603 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: - 2021-03-24T22:01:18.603 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 24 2021-03-24T22:01:18.603 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: T 2021-03-24T22:01:18.603 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 22 2021-03-24T22:01:18.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: : 2021-03-24T22:01:18.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 01 2021-03-24T22:01:18.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: : 2021-03-24T22:01:18.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 17 2021-03-24T22:01:18.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: . 2021-03-24T22:01:18.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 269073 2021-03-24T22:01:18.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: +0000 2021-03-24T22:01:18.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 2021-03-24T22:01:18.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: osd.0 2021-03-24T22:01:18.606 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: ( 2021-03-24T22:01:18.606 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: osd 2021-03-24T22:01:18.606 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: . 2021-03-24T22:01:18.606 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 0 2021-03-24T22:01:18.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: ) 2021-03-24T22:01:18.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 108 2021-03-24T22:01:18.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: : 2021-03-24T22:01:18.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: cluster 2021-03-24T22:01:18.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 2021-03-24T22:01:18.608 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: [DBG] 2021-03-24T22:01:18.608 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 2021-03-24T22:01:18.608 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 4.9 starting backfill to osd.1 from (0'0,0'0] MAX to 69'20 2021-03-24T22:01:18.608 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 2021-03-24T22:01:18.609 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: cluster 2021-03-24T22:01:18.609 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 2021-03-24T22:01:18.609 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 2021 2021-03-24T22:01:18.609 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: - 2021-03-24T22:01:18.609 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 03 2021-03-24T22:01:18.610 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: - 2021-03-24T22:01:18.610 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 24 2021-03-24T22:01:18.610 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: T 2021-03-24T22:01:18.610 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 22 2021-03-24T22:01:18.611 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: : 2021-03-24T22:01:18.611 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 01 2021-03-24T22:01:18.611 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: : 2021-03-24T22:01:18.611 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 17 2021-03-24T22:01:18.612 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: . 2021-03-24T22:01:18.612 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 272200 2021-03-24T22:01:18.612 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: +0000 2021-03-24T22:01:18.613 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 2021-03-24T22:01:18.613 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: osd.2 2021-03-24T22:01:18.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: ( 2021-03-24T22:01:18.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: osd 2021-03-24T22:01:18.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: . 2021-03-24T22:01:18.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 2 2021-03-24T22:01:18.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: ) 2021-03-24T22:01:18.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 83 2021-03-24T22:01:18.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: : 2021-03-24T22:01:18.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: cluster 2021-03-24T22:01:18.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 2021-03-24T22:01:18.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: [DBG] 2021-03-24T22:01:18.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 2021-03-24T22:01:18.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 4.d starting backfill to osd.1 from (0'0,0'0] MAX to 69'21 2021-03-24T22:01:18.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 2021-03-24T22:01:18.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: cluster 2021-03-24T22:01:18.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 2021-03-24T22:01:18.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 2021 2021-03-24T22:01:18.618 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: - 2021-03-24T22:01:18.618 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 03 2021-03-24T22:01:18.618 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: - 2021-03-24T22:01:18.618 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 24 2021-03-24T22:01:18.618 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: T 2021-03-24T22:01:18.619 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 22 2021-03-24T22:01:18.619 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: : 2021-03-24T22:01:18.619 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 01 2021-03-24T22:01:18.619 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: : 2021-03-24T22:01:18.620 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 17 2021-03-24T22:01:18.620 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: . 2021-03-24T22:01:18.620 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 278957 2021-03-24T22:01:18.620 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: +0000 2021-03-24T22:01:18.620 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 2021-03-24T22:01:18.621 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: osd.3 2021-03-24T22:01:18.621 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: ( 2021-03-24T22:01:18.621 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: osd 2021-03-24T22:01:18.621 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: . 2021-03-24T22:01:18.622 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 3 2021-03-24T22:01:18.622 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: ) 2021-03-24T22:01:18.622 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 111 2021-03-24T22:01:18.622 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: : 2021-03-24T22:01:18.623 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: cluster 2021-03-24T22:01:18.623 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 2021-03-24T22:01:18.623 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: [DBG] 2021-03-24T22:01:18.623 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 2021-03-24T22:01:18.624 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 4.e starting backfill to osd.1 from (0'0,0'0] MAX to 89'27 2021-03-24T22:01:18.624 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 2021-03-24T22:01:18.624 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: cluster 2021-03-24T22:01:18.624 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 2021-03-24T22:01:18.625 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 2021 2021-03-24T22:01:18.625 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: - 2021-03-24T22:01:18.625 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 03 2021-03-24T22:01:18.625 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: - 2021-03-24T22:01:18.626 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 24 2021-03-24T22:01:18.626 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: T 2021-03-24T22:01:18.626 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 22 2021-03-24T22:01:18.626 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: : 2021-03-24T22:01:18.627 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 01 2021-03-24T22:01:18.627 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: : 2021-03-24T22:01:18.627 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 17 2021-03-24T22:01:18.627 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: . 2021-03-24T22:01:18.628 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 286733 2021-03-24T22:01:18.628 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: +0000 2021-03-24T22:01:18.628 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 2021-03-24T22:01:18.628 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: osd.0 2021-03-24T22:01:18.628 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: ( 2021-03-24T22:01:18.629 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: osd 2021-03-24T22:01:18.629 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: . 2021-03-24T22:01:18.629 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 0 2021-03-24T22:01:18.629 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: ) 2021-03-24T22:01:18.630 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 109 2021-03-24T22:01:18.630 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: : 2021-03-24T22:01:18.630 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: cluster 2021-03-24T22:01:18.630 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 2021-03-24T22:01:18.631 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: [DBG] 2021-03-24T22:01:18.631 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 2021-03-24T22:01:18.631 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 4.b starting backfill to osd.1 from (0'0,0'0] MAX to 91'9 2021-03-24T22:01:18.632 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 2021-03-24T22:01:18.632 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: cluster 2021-03-24T22:01:18.632 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 2021-03-24T22:01:18.632 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 2021 2021-03-24T22:01:18.632 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: - 2021-03-24T22:01:18.633 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 03 2021-03-24T22:01:18.633 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: - 2021-03-24T22:01:18.633 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 24 2021-03-24T22:01:18.634 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: T 2021-03-24T22:01:18.634 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 22 2021-03-24T22:01:18.634 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: : 2021-03-24T22:01:18.634 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 01 2021-03-24T22:01:18.635 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: : 2021-03-24T22:01:18.635 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 18 2021-03-24T22:01:18.635 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: . 2021-03-24T22:01:18.635 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 267822 2021-03-24T22:01:18.635 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: +0000 2021-03-24T22:01:18.636 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 2021-03-24T22:01:18.636 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: mon.a 2021-03-24T22:01:18.636 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: ( 2021-03-24T22:01:18.636 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: mon 2021-03-24T22:01:18.637 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: . 2021-03-24T22:01:18.637 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 0 2021-03-24T22:01:18.638 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: ) 2021-03-24T22:01:18.638 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 1467 2021-03-24T22:01:18.639 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: : 2021-03-24T22:01:18.639 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: cluster 2021-03-24T22:01:18.639 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 2021-03-24T22:01:18.640 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: [DBG] 2021-03-24T22:01:18.640 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 2021-03-24T22:01:18.641 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: osdmap e508: 8 total, 8 up, 7 in 2021-03-24T22:01:18.641 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[24927]: 2021-03-24T22:01:18.642 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[19936]: -03-24T22: 2021-03-24T22:01:18.642 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[19936]: 01:17.286733+0000 2021-03-24T22:01:18.643 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[19936]: osd.0 (osd. 2021-03-24T22:01:18.643 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[19936]: 0) 109 : cluster 2021-03-24T22:01:18.644 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[19936]: [DBG] 4.b starting backfill to osd.1 from (0'0,0'0] MAX to 91'9 2021-03-24T22:01:18.644 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[19936]: cluster 2021-03-24T22:01:18.645 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[19936]: 2021- 2021-03-24T22:01:18.645 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[19936]: 03-24T22:01: 2021-03-24T22:01:18.645 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[19936]: 18.267822+0000 2021-03-24T22:01:18.646 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[19936]: mon.a (mon.0) 2021-03-24T22:01:18.646 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[19936]: 1467 : cluster [DBG] 2021-03-24T22:01:18.647 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:18 smithi125 conmon[19936]: osdmap e508: 8 total, 8 up, 7 in 2021-03-24T22:01:18.647 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:18.692 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T22:01:18.699 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:18.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:18.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:01:18.723 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T22:01:18.755 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T22:01:18.783 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T22:01:18.804 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:18.807 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T22:01:18.832 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T22:01:18.853 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:18.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:18.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:01:18.861 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T22:01:18.883 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T22:01:18.884 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-24T22:01:18.884 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-24T22:01:18.884 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-24T22:01:18.885 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-24T22:01:18.885 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-24T22:01:18.885 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-24T22:01:18.885 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-24T22:01:18.885 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-24T22:01:18.959 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:19.012 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:19.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:19.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:01:19.117 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:19.165 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:19.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:19.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:01:19.270 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:19.276 INFO:tasks.rados.rados.0.smithi125.stdout:1909: done (2 left) 2021-03-24T22:01:19.277 INFO:tasks.rados.rados.0.smithi125.stdout:1910: rollback oid 42 current snap is 210 2021-03-24T22:01:19.277 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 42 to 200 2021-03-24T22:01:19.277 INFO:tasks.rados.rados.0.smithi125.stdout:1911: snap_remove snap 129 2021-03-24T22:01:19.317 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:19.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:19.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:01:19.422 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:19.435 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:19 smithi125 conmon[24927]: audit 2021-03-24T22:01:18.666845 2021-03-24T22:01:19.435 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:19 smithi125 conmon[24927]: +0000 mon.c (mon.2) 79 : audit [DBG] from='client.? v1:172.21.15.125:0/3887134740' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:01:19.436 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:19 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T22:01:19.436 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:19 smithi125 conmon[24927]: T22:01:19.273540+0000 mon.a (mon.0) 1468 : cluster 2021-03-24T22:01:19.436 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:19 smithi125 conmon[24927]: [DBG] osdmap e509: 8 total, 8 up, 7 in 2021-03-24T22:01:19.437 INFO:journalctl@ceph.rgw.foo.a.smithi125.stdout:Mar 24 22:01:19 smithi125 conmon[39595]: debug 2021-03-24T22:01:19.158+0000 7fc212827700 -1 RGWWatcher::handle_error cookie 94195313958912 err (110) Connection timed out 2021-03-24T22:01:19.438 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:19 smithi125 conmon[19936]: audit 2021-03-24T22:01:18.666845+0000 2021-03-24T22:01:19.439 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:19 smithi125 conmon[19936]: mon.c (mon.2) 79 : audit [DBG] from='client.? v1:172.21.15.125:0/3887134740' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:01:19.439 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:19 smithi125 conmon[19936]: cluster 2021-03-24T22:01:19.273540+0000 mon.a (mon.0) 1468 2021-03-24T22:01:19.439 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:19 smithi125 conmon[19936]: : cluster [DBG] osdmap e509: 8 total, 8 up, 7 in 2021-03-24T22:01:19.470 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:19.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:19.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:01:19.576 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:19.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:19 smithi142 conmon[20125]: audit 2021-03-24T22:01:18.666845+0000 mon.c (mon.2) 79 : audit [DBG] from='client.? v1:172.21.15.125:0/3887134740' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:01:19.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:19 smithi142 conmon[20125]: cluster 2021-03-24T22:01:19.273540+0000 mon.a (mon.0) 1468 : cluster [DBG] osdmap e509: 8 total, 8 up, 7 in 2021-03-24T22:01:19.621 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:19.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:19.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:01:19.729 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:19.777 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:19.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:19.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:01:19.882 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:19.939 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:19.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:19.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:01:20.045 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:20.096 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:20.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:20.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:01:20.204 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:20.259 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:20.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:20.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:01:20.271 INFO:tasks.rados.rados.0.smithi125.stdout:1911: done (3 left) 2021-03-24T22:01:20.271 INFO:tasks.rados.rados.0.smithi125.stdout:1912: read oid 8 snap 168 2021-03-24T22:01:20.272 INFO:tasks.rados.rados.0.smithi125.stdout:1912: expect deleted 2021-03-24T22:01:20.272 INFO:tasks.rados.rados.0.smithi125.stdout:1913: write (excl) oid 20 current snap is 210 2021-03-24T22:01:20.272 INFO:tasks.rados.rados.0.smithi125.stdout:1913: seq_num 675 ranges {544725=632139,1716169=522227,2736873=512491} 2021-03-24T22:01:20.280 INFO:tasks.rados.rados.0.smithi125.stdout:1913: writing smithi12556470-20 from 544725 to 1176864 tid 1 2021-03-24T22:01:20.288 INFO:tasks.rados.rados.0.smithi125.stdout:1913: writing smithi12556470-20 from 1716169 to 2238396 tid 2 2021-03-24T22:01:20.294 INFO:tasks.rados.rados.0.smithi125.stdout:1913: writing smithi12556470-20 from 2736873 to 3249364 tid 3 2021-03-24T22:01:20.295 INFO:tasks.rados.rados.0.smithi125.stdout:1914: read oid 3 snap -1 2021-03-24T22:01:20.295 INFO:tasks.rados.rados.0.smithi125.stdout:1914: expect (ObjNum 648 snap 197 seq_num 648) 2021-03-24T22:01:20.295 INFO:tasks.rados.rados.0.smithi125.stdout:1915: write oid 2 current snap is 210 2021-03-24T22:01:20.296 INFO:tasks.rados.rados.0.smithi125.stdout:1915: seq_num 676 ranges {561532=707388,2058298=782238} 2021-03-24T22:01:20.303 INFO:tasks.rados.rados.0.smithi125.stdout:1915: writing smithi12556470-2 from 561532 to 1268920 tid 1 2021-03-24T22:01:20.313 INFO:tasks.rados.rados.0.smithi125.stdout:1915: writing smithi12556470-2 from 2058298 to 2840536 tid 2 2021-03-24T22:01:20.313 INFO:tasks.rados.rados.0.smithi125.stdout:1916: read oid 44 snap 168 2021-03-24T22:01:20.313 INFO:tasks.rados.rados.0.smithi125.stdout:1916: expect (ObjNum 470 snap 133 seq_num 470) 2021-03-24T22:01:20.314 INFO:tasks.rados.rados.0.smithi125.stdout:1917: snap_remove snap 197 2021-03-24T22:01:20.336 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:20 smithi125 conmon[19936]: cluster 2021-03-24T22:01:18.878713+0000 2021-03-24T22:01:20.336 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:20 smithi125 conmon[19936]: mgr.y (mgr.14144) 526 : cluster [DBG] pgmap v925: 168 pgs: 1 activating, 1 remapped+peering, 1 active+remapped+backfilling, 2 active+recovery_wait+degraded, 1 active+recovery_wait+remapped, 1 peering, 1 active+remapped+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 151 active+clean; 466 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 0 op/s; 32/1629 objects degraded (1.964%); 188/1629 objects misplaced (11.541%); 144 B/s, 2 objects/s recovering 2021-03-24T22:01:20.336 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:20 smithi125 conmon[19936]: cluster 2021-03-24T22:01:20.258013+0000 mon.a (mon.0) 1469 : cluster [WRN] Health check update: Degraded data redundancy: 32/1629 objects degraded (1.964%), 5 pgs degraded (PG_DEGRADED) 2021-03-24T22:01:20.337 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:20 smithi125 conmon[19936]: cluster 2021-03-24T22:01:20.266071+0000 mon.a (mon.0) 1470 : cluster 2021-03-24T22:01:20.337 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:20 smithi125 conmon[19936]: [DBG] osdmap e510: 8 total, 8 up, 7 in 2021-03-24T22:01:20.337 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:20 smithi125 conmon[24927]: cluster 2021-03-24T22:01:18.878713 2021-03-24T22:01:20.337 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:20 smithi125 conmon[24927]: +0000 mgr.y (mgr.14144) 526 : cluster [DBG] pgmap v925: 168 pgs: 1 activating, 1 remapped+peering, 1 active+remapped+backfilling, 2 active+recovery_wait+degraded, 1 active+recovery_wait+remapped, 1 peering, 1 active+remapped+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 151 active+clean; 466 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 0 op/s; 32/1629 objects degraded (1.964%); 188/1629 objects misplaced (11.541%); 144 B/s, 2 objects/s recovering 2021-03-24T22:01:20.338 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:20 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T22:01:20.338 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:20 smithi125 conmon[24927]: 22:01:20.258013+0000 mon.a (mon.0) 1469 : cluster [WRN] Health check update: Degraded data redundancy: 32/1629 objects degraded (1.964%), 5 pgs degraded (PG_DEGRADED) 2021-03-24T22:01:20.338 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:20 smithi125 conmon[24927]: cluster 2021-03-24T22:01:20.266071+0000 mon.a 2021-03-24T22:01:20.338 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:20 smithi125 conmon[24927]: (mon.0) 1470 : cluster [DBG] osdmap e510: 8 total, 8 up, 7 in 2021-03-24T22:01:20.367 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:20.422 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:20.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:20.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:01:20.526 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:20.580 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:20.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:20.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:01:20.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:20 smithi142 conmon[20125]: cluster 2021-03-24T22:01:18.878713+0000 mgr.y (mgr.14144) 526 : cluster [DBG] 2021-03-24T22:01:20.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:20 smithi142 conmon[20125]: pgmap v925: 168 pgs: 1 activating, 1 remapped+peering, 1 active+remapped+backfilling, 2 active+recovery_wait+degraded, 1 active+recovery_wait+remapped, 1 peering, 1 active+remapped+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 151 active+clean; 466 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 0 op/s; 32/1629 objects degraded (1.964%); 188/1629 objects misplaced (11.541%); 144 B/s, 2 objects/s recovering 2021-03-24T22:01:20.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:20 smithi142 conmon[20125]: cluster 2021-03-24T22:01:20.258013+0000 mon.a (mon.0) 1469 : cluster [WRN] Health check update: Degraded data redundancy: 32/1629 objects degraded (1.964%), 5 pgs degraded (PG_DEGRADED) 2021-03-24T22:01:20.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:20 smithi142 conmon[20125]: 2021-03-24T22:01:20.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:20 smithi142 conmon[20125]: cluster 2021-03-24T22:01 2021-03-24T22:01:20.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:20 smithi142 conmon[20125]: :20.266071+0000 mon.a (mon.0) 1470 : cluster 2021-03-24T22:01:20.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:20 smithi142 conmon[20125]: [DBG] osdmap e510: 8 total, 8 up, 7 in 2021-03-24T22:01:20.686 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:20.739 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:20.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:20.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:01:20.845 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:20.901 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:20.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:20.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:01:21.008 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:21.056 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:21.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:21.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:01:21.163 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:21.218 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:21.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:21.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:01:21.284 INFO:tasks.rados.rados.0.smithi125.stdout:1917: done (8 left) 2021-03-24T22:01:21.284 INFO:tasks.rados.rados.0.smithi125.stdout:1918: write oid 11 current snap is 210 2021-03-24T22:01:21.285 INFO:tasks.rados.rados.0.smithi125.stdout:1907: finishing rollback tid 0 to smithi12556470-7 2021-03-24T22:01:21.285 INFO:tasks.rados.rados.0.smithi125.stdout:1907: finishing rollback tid 1 to smithi12556470-7 2021-03-24T22:01:21.286 INFO:tasks.rados.rados.0.smithi125.stdout:1918: seq_num 677 ranges {578339=498990,1833133=525896,2791719=586620} 2021-03-24T22:01:21.291 INFO:tasks.rados.rados.0.smithi125.stdout:1918: writing smithi12556470-11 from 578339 to 1077329 tid 1 2021-03-24T22:01:21.300 INFO:tasks.rados.rados.0.smithi125.stdout:1918: writing smithi12556470-11 from 1833133 to 2359029 tid 2 2021-03-24T22:01:21.309 INFO:tasks.rados.rados.0.smithi125.stdout:1918: writing smithi12556470-11 from 2791719 to 3378339 tid 3 2021-03-24T22:01:21.309 INFO:tasks.rados.rados.0.smithi125.stdout:1908: done (8 left) 2021-03-24T22:01:21.310 INFO:tasks.rados.rados.0.smithi125.stdout:1919: read oid 41 snap -1 2021-03-24T22:01:21.310 INFO:tasks.rados.rados.0.smithi125.stdout:1919: expect deleted 2021-03-24T22:01:21.311 INFO:tasks.rados.rados.0.smithi125.stdout:1920: read oid 43 snap -1 2021-03-24T22:01:21.311 INFO:tasks.rados.rados.0.smithi125.stdout:1920: expect deleted 2021-03-24T22:01:21.315 INFO:tasks.rados.rados.0.smithi125.stdout:1907: finishing rollback tid 2 to smithi12556470-7 2021-03-24T22:01:21.316 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 7 v 678 (ObjNum 588 snap 177 seq_num 588) dirty exists 2021-03-24T22:01:21.316 INFO:tasks.rados.rados.0.smithi125.stdout:1910: finishing rollback tid 0 to smithi12556470-42 2021-03-24T22:01:21.317 INFO:tasks.rados.rados.0.smithi125.stdout:1910: finishing rollback tid 1 to smithi12556470-42 2021-03-24T22:01:21.318 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 42 v 497 (ObjNum 1117500400 snap 21916 seq_num 1117570800) dirty dne 2021-03-24T22:01:21.318 INFO:tasks.rados.rados.0.smithi125.stdout:1913: finishing write tid 1 to smithi12556470-20 2021-03-24T22:01:21.318 INFO:tasks.rados.rados.0.smithi125.stdout:1914: expect (ObjNum 648 snap 197 seq_num 648) 2021-03-24T22:01:21.322 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-24T22:01:21.323 INFO:teuthology.orchestra.run.smithi125.stdout: "size": 20, 2021-03-24T22:01:21.323 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 600, 2021-03-24T22:01:21.323 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [ 2021-03-24T22:01:21.323 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.324 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.34352.0:535 4.b e65/62 4:d1061236:::data_log.86:head v 65'3, mlcod=65'1)", 2021-03-24T22:01:21.324 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:51:50.034175+0000", 2021-03-24T22:01:21.324 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 571.28243637599996, 2021-03-24T22:01:21.325 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.017611492999999999, 2021-03-24T22:01:21.325 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:01:21.326 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:01:21.326 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:01:21.326 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.327 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:01:21.327 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.034175+0000", 2021-03-24T22:01:21.327 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:01:21.327 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.327 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.328 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:01:21.328 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.034175+0000", 2021-03-24T22:01:21.328 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T22:01:21.328 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.330 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.331 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:01:21.331 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.034173+0000", 2021-03-24T22:01:21.331 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3985000000000001e-05 2021-03-24T22:01:21.332 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.332 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.332 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:01:21.333 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.034187+0000", 2021-03-24T22:01:21.333 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7768e-05 2021-03-24T22:01:21.333 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.333 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.334 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:01:21.334 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.034205+0000", 2021-03-24T22:01:21.334 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4919999999999998e-06 2021-03-24T22:01:21.335 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.335 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.335 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:01:21.336 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.034211+0000", 2021-03-24T22:01:21.336 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5782000000000002e-05 2021-03-24T22:01:21.336 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.337 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.337 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:21.337 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.034267+0000", 2021-03-24T22:01:21.337 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.4258000000000003e-05 2021-03-24T22:01:21.338 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.338 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.338 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:01:21.339 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.034301+0000", 2021-03-24T22:01:21.339 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.017442103 2021-03-24T22:01:21.339 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.340 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.340 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:01:21.340 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.051743+0000", 2021-03-24T22:01:21.340 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3061e-05 2021-03-24T22:01:21.341 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.341 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.341 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:01:21.342 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:50.051786+0000", 2021-03-24T22:01:21.342 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.017611492999999999 2021-03-24T22:01:21.342 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:21.343 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:01:21.343 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:21.343 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.343 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.344 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24376.0:221 4.2 4:479ba34f:::obj_delete_at_hint.0000000021:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e69)", 2021-03-24T22:01:21.344 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:51:53.942625+0000", 2021-03-24T22:01:21.344 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 567.37398673500002, 2021-03-24T22:01:21.345 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0044149480000000001, 2021-03-24T22:01:21.345 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:01:21.345 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:01:21.345 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:01:21.346 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24376", 2021-03-24T22:01:21.346 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/3051439499", 2021-03-24T22:01:21.346 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 221 2021-03-24T22:01:21.347 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.347 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:01:21.347 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.348 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:01:21.348 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.942625+0000", 2021-03-24T22:01:21.348 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:01:21.348 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.349 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.349 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:01:21.349 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.942625+0000", 2021-03-24T22:01:21.350 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T22:01:21.350 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.350 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.350 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:01:21.351 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.942623+0000", 2021-03-24T22:01:21.351 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.4860000000000003e-06 2021-03-24T22:01:21.351 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.352 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.352 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:01:21.352 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.942632+0000", 2021-03-24T22:01:21.353 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7484000000000001e-05 2021-03-24T22:01:21.353 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.353 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.353 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:01:21.354 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.942650+0000", 2021-03-24T22:01:21.354 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.131e-06 2021-03-24T22:01:21.354 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.355 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.355 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:01:21.355 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.942657+0000", 2021-03-24T22:01:21.355 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.9171999999999993e-05 2021-03-24T22:01:21.356 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.356 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.357 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:21.357 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.942736+0000", 2021-03-24T22:01:21.357 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000181931 2021-03-24T22:01:21.358 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.358 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.358 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:01:21.358 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.942918+0000", 2021-03-24T22:01:21.359 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00019801000000000001 2021-03-24T22:01:21.359 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.359 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.359 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 4,5", 2021-03-24T22:01:21.360 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.943116+0000", 2021-03-24T22:01:21.360 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00048897899999999995 2021-03-24T22:01:21.360 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.361 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.361 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:01:21.361 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.943605+0000", 2021-03-24T22:01:21.362 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000288492 2021-03-24T22:01:21.362 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.362 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.363 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:01:21.363 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.943893+0000", 2021-03-24T22:01:21.363 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0030618160000000002 2021-03-24T22:01:21.364 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.364 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.364 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:01:21.365 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.946955+0000", 2021-03-24T22:01:21.365 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.7068999999999999e-05 2021-03-24T22:01:21.365 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.365 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.366 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:01:21.366 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.947002+0000", 2021-03-24T22:01:21.366 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.7311999999999998e-05 2021-03-24T22:01:21.366 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.367 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.367 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:01:21.367 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:51:53.947040+0000", 2021-03-24T22:01:21.368 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0044149480000000001 2021-03-24T22:01:21.368 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:21.368 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:01:21.369 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:21.369 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.369 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.369 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:33 7.b 7:d821c111:::smithi12556470-9:head [write 534456~718345,stat] snapc 0=[] ondisk+write+known_if_redirected e107)", 2021-03-24T22:01:21.370 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.193321+0000", 2021-03-24T22:01:21.370 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 405.12329011600002, 2021-03-24T22:01:21.371 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0060392889999999998, 2021-03-24T22:01:21.371 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:01:21.371 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:01:21.371 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:01:21.372 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:01:21.372 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:01:21.372 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 33 2021-03-24T22:01:21.373 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.373 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:01:21.373 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.373 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:01:21.374 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.193321+0000", 2021-03-24T22:01:21.374 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:01:21.374 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.375 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.375 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:01:21.375 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.193321+0000", 2021-03-24T22:01:21.376 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999976 2021-03-24T22:01:21.376 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.376 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.376 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:01:21.377 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.193319+0000", 2021-03-24T22:01:21.377 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00038596399999999998 2021-03-24T22:01:21.377 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.378 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.378 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:01:21.378 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.193705+0000", 2021-03-24T22:01:21.378 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.8202e-05 2021-03-24T22:01:21.379 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.379 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.379 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:01:21.380 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.193723+0000", 2021-03-24T22:01:21.380 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.6779999999999995e-06 2021-03-24T22:01:21.380 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.380 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.381 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:01:21.381 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.193733+0000", 2021-03-24T22:01:21.381 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.7935000000000005e-05 2021-03-24T22:01:21.382 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.382 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.382 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:21.382 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.193810+0000", 2021-03-24T22:01:21.383 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00016634899999999999 2021-03-24T22:01:21.383 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.383 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.384 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:01:21.384 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.193977+0000", 2021-03-24T22:01:21.384 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00021176300000000001 2021-03-24T22:01:21.385 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.385 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.385 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,5", 2021-03-24T22:01:21.386 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.194189+0000", 2021-03-24T22:01:21.386 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0020706510000000002 2021-03-24T22:01:21.386 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.386 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.387 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:01:21.387 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.196259+0000", 2021-03-24T22:01:21.387 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.4345000000000002e-05 2021-03-24T22:01:21.387 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.387 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.388 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:01:21.388 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.196284+0000", 2021-03-24T22:01:21.388 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0028126039999999998 2021-03-24T22:01:21.388 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.389 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.389 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:01:21.389 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.199096+0000", 2021-03-24T22:01:21.389 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.7195999999999997e-05 2021-03-24T22:01:21.390 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.390 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.390 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:01:21.390 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.199173+0000", 2021-03-24T22:01:21.391 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00018708400000000001 2021-03-24T22:01:21.391 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.391 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.391 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:01:21.392 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.199360+0000", 2021-03-24T22:01:21.392 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0060392889999999998 2021-03-24T22:01:21.392 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:21.392 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:01:21.392 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:21.393 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.393 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.393 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:34 7.b 7:d821c111:::smithi12556470-9:head [write 1951503~182953] snapc 0=[] ondisk+write+known_if_redirected e107)", 2021-03-24T22:01:21.394 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.197792+0000", 2021-03-24T22:01:21.394 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 405.11881904299997, 2021-03-24T22:01:21.394 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0040527949999999997, 2021-03-24T22:01:21.394 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:01:21.394 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:01:21.395 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:01:21.395 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:01:21.395 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:01:21.395 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 34 2021-03-24T22:01:21.396 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.396 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:01:21.396 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.396 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:01:21.397 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.197792+0000", 2021-03-24T22:01:21.397 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:01:21.397 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.398 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.398 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:01:21.398 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.197792+0000", 2021-03-24T22:01:21.398 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999976 2021-03-24T22:01:21.399 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.399 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.399 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:01:21.399 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.197790+0000", 2021-03-24T22:01:21.400 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.4402999999999997e-05 2021-03-24T22:01:21.400 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.400 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.400 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:01:21.401 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.197884+0000", 2021-03-24T22:01:21.401 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9775999999999998e-05 2021-03-24T22:01:21.401 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.401 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.401 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:01:21.402 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.197904+0000", 2021-03-24T22:01:21.402 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.4100000000000008e-06 2021-03-24T22:01:21.402 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.402 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.403 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:01:21.403 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.197912+0000", 2021-03-24T22:01:21.404 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.2970000000000001e-05 2021-03-24T22:01:21.404 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.404 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.404 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:21.405 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.197985+0000", 2021-03-24T22:01:21.405 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000144194 2021-03-24T22:01:21.405 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.406 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.406 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:01:21.406 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.198130+0000", 2021-03-24T22:01:21.406 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0013126240000000001 2021-03-24T22:01:21.407 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.407 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.407 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:21.407 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.199442+0000", 2021-03-24T22:01:21.408 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.276e-05 2021-03-24T22:01:21.408 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.408 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.408 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:01:21.409 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.199535+0000", 2021-03-24T22:01:21.409 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00021443300000000001 2021-03-24T22:01:21.409 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.409 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.410 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,5", 2021-03-24T22:01:21.410 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.199749+0000", 2021-03-24T22:01:21.410 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001662048 2021-03-24T22:01:21.411 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.411 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.412 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:01:21.412 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.201411+0000", 2021-03-24T22:01:21.412 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.5590000000000006e-06 2021-03-24T22:01:21.413 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.413 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.414 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:01:21.414 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.201420+0000", 2021-03-24T22:01:21.415 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000318401 2021-03-24T22:01:21.415 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.416 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.416 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:01:21.417 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.201738+0000", 2021-03-24T22:01:21.418 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.8393999999999997e-05 2021-03-24T22:01:21.418 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.418 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.419 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:01:21.419 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.201797+0000", 2021-03-24T22:01:21.420 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.8189999999999998e-05 2021-03-24T22:01:21.420 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.421 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.421 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:01:21.422 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.201845+0000", 2021-03-24T22:01:21.422 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0040527949999999997 2021-03-24T22:01:21.423 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:21.423 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:01:21.423 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:21.423 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.424 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.424 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:35 7.b 7:d821c111:::smithi12556470-9:head [setxattr _header (55) in=7b,truncate 2134456] snapc 0=[] ondisk+write+known_if_redirected e107)", 2021-03-24T22:01:21.424 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.197928+0000", 2021-03-24T22:01:21.424 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 405.118683664, 2021-03-24T22:01:21.425 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0041834120000000001, 2021-03-24T22:01:21.425 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:01:21.425 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:01:21.425 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:01:21.426 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:01:21.426 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:01:21.426 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 35 2021-03-24T22:01:21.426 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.427 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:01:21.427 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.427 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:01:21.427 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.197928+0000", 2021-03-24T22:01:21.428 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:01:21.428 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.428 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.428 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:01:21.429 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.197928+0000", 2021-03-24T22:01:21.429 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T22:01:21.429 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.429 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.430 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:01:21.430 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.197926+0000", 2021-03-24T22:01:21.430 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.653e-06 2021-03-24T22:01:21.430 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.431 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.431 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:01:21.431 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.197935+0000", 2021-03-24T22:01:21.431 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1636e-05 2021-03-24T22:01:21.431 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.432 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.432 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:01:21.432 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.197946+0000", 2021-03-24T22:01:21.432 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.9549999999999998e-06 2021-03-24T22:01:21.433 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.433 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.433 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:01:21.433 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.197955+0000", 2021-03-24T22:01:21.434 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00023729900000000001 2021-03-24T22:01:21.434 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.434 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.434 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:21.435 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.198193+0000", 2021-03-24T22:01:21.435 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000111729 2021-03-24T22:01:21.435 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.435 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.435 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:01:21.436 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.198304+0000", 2021-03-24T22:01:21.436 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0021900470000000001 2021-03-24T22:01:21.436 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.436 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.437 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:21.437 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.200494+0000", 2021-03-24T22:01:21.437 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00020013600000000001 2021-03-24T22:01:21.437 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.438 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.438 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:01:21.438 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.200694+0000", 2021-03-24T22:01:21.438 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00025853600000000002 2021-03-24T22:01:21.439 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.439 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.439 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,5", 2021-03-24T22:01:21.439 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.200953+0000", 2021-03-24T22:01:21.440 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00048642299999999999 2021-03-24T22:01:21.440 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.440 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.440 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:01:21.440 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.201439+0000", 2021-03-24T22:01:21.441 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00047360399999999997 2021-03-24T22:01:21.441 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.441 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.441 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:01:21.442 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.201913+0000", 2021-03-24T22:01:21.442 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.1973999999999998e-05 2021-03-24T22:01:21.442 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.442 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.443 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:01:21.443 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.201995+0000", 2021-03-24T22:01:21.443 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.7000999999999999e-05 2021-03-24T22:01:21.443 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.444 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.444 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:01:21.444 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.202052+0000", 2021-03-24T22:01:21.444 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9033999999999998e-05 2021-03-24T22:01:21.444 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.445 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.445 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:01:21.445 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.202111+0000", 2021-03-24T22:01:21.445 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0041834120000000001 2021-03-24T22:01:21.446 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:21.446 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:01:21.446 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:21.446 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.447 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.447 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:36 7.b 7:d821c111:::smithi12556470-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e107)", 2021-03-24T22:01:21.447 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.197999+0000", 2021-03-24T22:01:21.447 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 405.11861260900002, 2021-03-24T22:01:21.448 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0044019999999999997, 2021-03-24T22:01:21.448 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:01:21.448 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T22:01:21.448 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:01:21.448 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:01:21.449 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:01:21.449 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 36 2021-03-24T22:01:21.449 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.449 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:01:21.450 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.450 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:01:21.450 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.197999+0000", 2021-03-24T22:01:21.450 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:01:21.450 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.451 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.451 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:01:21.451 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.197999+0000", 2021-03-24T22:01:21.451 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T22:01:21.452 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.452 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.452 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:01:21.452 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.197997+0000", 2021-03-24T22:01:21.453 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.576e-06 2021-03-24T22:01:21.453 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.453 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.453 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:01:21.453 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.198005+0000", 2021-03-24T22:01:21.454 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2581e-05 2021-03-24T22:01:21.454 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.454 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.454 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:01:21.455 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.198017+0000", 2021-03-24T22:01:21.455 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.6400000000000003e-06 2021-03-24T22:01:21.455 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.455 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.456 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:01:21.456 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.198026+0000", 2021-03-24T22:01:21.456 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00031787599999999998 2021-03-24T22:01:21.456 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.456 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.457 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:21.457 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.198344+0000", 2021-03-24T22:01:21.457 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.3399999999999993e-05 2021-03-24T22:01:21.458 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.458 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.458 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:01:21.458 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.198437+0000", 2021-03-24T22:01:21.459 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0028437660000000002 2021-03-24T22:01:21.459 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.459 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.459 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:21.459 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.201281+0000", 2021-03-24T22:01:21.460 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2627999999999996e-05 2021-03-24T22:01:21.460 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.460 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.460 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:01:21.461 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.201364+0000", 2021-03-24T22:01:21.461 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000821461 2021-03-24T22:01:21.461 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.461 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.462 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:21.462 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.202185+0000", 2021-03-24T22:01:21.462 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010005699999999999 2021-03-24T22:01:21.462 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.463 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.463 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:01:21.463 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.202285+0000", 2021-03-24T22:01:21.463 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000115427 2021-03-24T22:01:21.463 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.464 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.464 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:01:21.464 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.202401+0000", 2021-03-24T22:01:21.464 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0044019999999999997 2021-03-24T22:01:21.465 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:21.465 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:01:21.465 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:21.465 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.465 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.466 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:53 7.b 7:da3cc97e:::smithi12556470-14:head [write 618491~694590,stat] snapc 0=[] ondisk+write+known_if_redirected e107)", 2021-03-24T22:01:21.466 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.255785+0000", 2021-03-24T22:01:21.466 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 405.06082626800003, 2021-03-24T22:01:21.466 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0052466329999999997, 2021-03-24T22:01:21.467 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:01:21.467 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:01:21.467 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:01:21.467 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:01:21.468 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:01:21.468 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 53 2021-03-24T22:01:21.468 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.468 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:01:21.469 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.469 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:01:21.469 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.255785+0000", 2021-03-24T22:01:21.469 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:01:21.469 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.470 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.470 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:01:21.470 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.255785+0000", 2021-03-24T22:01:21.470 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999967 2021-03-24T22:01:21.471 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.471 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.471 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:01:21.471 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.255781+0000", 2021-03-24T22:01:21.472 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00027362600000000001 2021-03-24T22:01:21.472 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.472 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.472 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:01:21.472 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.256055+0000", 2021-03-24T22:01:21.473 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.9417e-05 2021-03-24T22:01:21.473 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.473 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.473 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:01:21.474 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.256084+0000", 2021-03-24T22:01:21.474 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1114000000000001e-05 2021-03-24T22:01:21.474 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.475 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.475 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:01:21.475 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.256096+0000", 2021-03-24T22:01:21.475 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00014862799999999999 2021-03-24T22:01:21.475 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.476 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.476 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:21.476 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.256244+0000", 2021-03-24T22:01:21.476 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000355247 2021-03-24T22:01:21.477 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.477 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.477 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:01:21.477 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.256599+0000", 2021-03-24T22:01:21.478 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00049522699999999999 2021-03-24T22:01:21.478 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.478 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.478 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,5", 2021-03-24T22:01:21.479 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.257095+0000", 2021-03-24T22:01:21.479 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0015160180000000001 2021-03-24T22:01:21.479 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.479 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.479 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:01:21.480 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.258611+0000", 2021-03-24T22:01:21.480 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0010272020000000001 2021-03-24T22:01:21.480 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.480 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.481 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:01:21.481 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.259638+0000", 2021-03-24T22:01:21.481 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001131791 2021-03-24T22:01:21.481 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.482 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.482 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:01:21.482 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.260770+0000", 2021-03-24T22:01:21.482 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012263000000000001 2021-03-24T22:01:21.483 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.483 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.483 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:01:21.483 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.260892+0000", 2021-03-24T22:01:21.483 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000139309 2021-03-24T22:01:21.484 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.484 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.484 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:01:21.484 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.261032+0000", 2021-03-24T22:01:21.485 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0052466329999999997 2021-03-24T22:01:21.485 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:21.485 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:01:21.485 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:21.486 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.486 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.486 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:62 7.7 e107/101 7:e7492b26:::smithi12556470-16:head v 107'5, mlcod=107'3)", 2021-03-24T22:01:21.486 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.287402+0000", 2021-03-24T22:01:21.486 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 405.02920884999997, 2021-03-24T22:01:21.487 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0054094420000000004, 2021-03-24T22:01:21.487 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:01:21.487 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:01:21.487 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:01:21.488 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.488 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:01:21.488 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.287402+0000", 2021-03-24T22:01:21.488 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:01:21.489 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.489 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.489 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:01:21.489 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.287402+0000", 2021-03-24T22:01:21.489 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999976 2021-03-24T22:01:21.490 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.490 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.490 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:01:21.490 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.287400+0000", 2021-03-24T22:01:21.491 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000213122 2021-03-24T22:01:21.491 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.491 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.491 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:01:21.492 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.287613+0000", 2021-03-24T22:01:21.492 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.8102e-05 2021-03-24T22:01:21.492 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.492 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.492 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:01:21.493 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.287641+0000", 2021-03-24T22:01:21.493 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.8050000000000001e-06 2021-03-24T22:01:21.493 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.493 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.494 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:01:21.494 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.287651+0000", 2021-03-24T22:01:21.494 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9985000000000003e-05 2021-03-24T22:01:21.494 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.494 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.495 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:21.495 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.287711+0000", 2021-03-24T22:01:21.495 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.1588999999999999e-05 2021-03-24T22:01:21.495 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.496 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.496 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:01:21.496 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.287743+0000", 2021-03-24T22:01:21.496 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0050277150000000003 2021-03-24T22:01:21.497 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.497 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.497 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:01:21.497 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.292770+0000", 2021-03-24T22:01:21.497 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.1461000000000002e-05 2021-03-24T22:01:21.498 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.498 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.498 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:01:21.498 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.292812+0000", 2021-03-24T22:01:21.499 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0054094420000000004 2021-03-24T22:01:21.499 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:21.499 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:01:21.499 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:21.500 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.500 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.500 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:63 7.7 e107/101 7:e7492b26:::smithi12556470-16:head v 107'6, mlcod=107'3)", 2021-03-24T22:01:21.500 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.288459+0000", 2021-03-24T22:01:21.500 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 405.02815207899999, 2021-03-24T22:01:21.501 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0043744769999999999, 2021-03-24T22:01:21.501 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:01:21.501 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:01:21.501 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:01:21.502 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.502 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:01:21.502 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.288459+0000", 2021-03-24T22:01:21.502 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:01:21.502 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.503 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.503 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:01:21.503 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.288459+0000", 2021-03-24T22:01:21.503 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T22:01:21.504 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.504 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.504 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:01:21.504 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.288458+0000", 2021-03-24T22:01:21.505 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.2450000000000004e-06 2021-03-24T22:01:21.505 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.505 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.506 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:01:21.506 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.288467+0000", 2021-03-24T22:01:21.506 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5017e-05 2021-03-24T22:01:21.506 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.506 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.507 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:01:21.507 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.288482+0000", 2021-03-24T22:01:21.507 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.5370000000000003e-06 2021-03-24T22:01:21.507 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.508 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.508 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:01:21.508 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.288488+0000", 2021-03-24T22:01:21.508 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6026999999999998e-05 2021-03-24T22:01:21.509 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.509 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.509 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:21.509 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.288534+0000", 2021-03-24T22:01:21.509 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9232000000000002e-05 2021-03-24T22:01:21.510 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.510 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.510 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:01:21.510 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.288554+0000", 2021-03-24T22:01:21.511 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0042627170000000001 2021-03-24T22:01:21.511 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.511 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.511 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:01:21.512 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.292816+0000", 2021-03-24T22:01:21.512 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7295e-05 2021-03-24T22:01:21.512 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.512 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.512 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:01:21.513 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.292834+0000", 2021-03-24T22:01:21.513 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0043744769999999999 2021-03-24T22:01:21.513 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:21.513 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:01:21.514 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:21.514 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.514 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.514 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:82 7.d 7:b2218182:::smithi12556470-21:head [write 2020581~315559] snapc 0=[] ondisk+write+known_if_redirected e107)", 2021-03-24T22:01:21.515 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.359676+0000", 2021-03-24T22:01:21.515 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 404.95693480699998, 2021-03-24T22:01:21.515 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0053478029999999999, 2021-03-24T22:01:21.515 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:01:21.515 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:01:21.516 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:01:21.516 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:01:21.516 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:01:21.516 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 82 2021-03-24T22:01:21.517 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.517 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:01:21.517 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.517 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:01:21.518 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359676+0000", 2021-03-24T22:01:21.518 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:01:21.518 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.519 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.519 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:01:21.519 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359676+0000", 2021-03-24T22:01:21.519 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T22:01:21.519 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.520 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.520 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:01:21.520 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359674+0000", 2021-03-24T22:01:21.520 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000130075 2021-03-24T22:01:21.521 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.521 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.521 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:01:21.521 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359804+0000", 2021-03-24T22:01:21.522 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.2365999999999999e-05 2021-03-24T22:01:21.522 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.522 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.522 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:01:21.522 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359827+0000", 2021-03-24T22:01:21.523 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.6319999999999997e-06 2021-03-24T22:01:21.523 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.523 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.523 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:01:21.524 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359835+0000", 2021-03-24T22:01:21.524 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.2173000000000002e-05 2021-03-24T22:01:21.524 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.524 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.525 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:21.525 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359908+0000", 2021-03-24T22:01:21.525 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00011309199999999999 2021-03-24T22:01:21.525 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.525 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.526 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:01:21.526 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.360021+0000", 2021-03-24T22:01:21.526 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00020120199999999999 2021-03-24T22:01:21.526 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.527 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.527 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,3", 2021-03-24T22:01:21.527 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.360222+0000", 2021-03-24T22:01:21.527 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0013165919999999999 2021-03-24T22:01:21.528 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.528 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.528 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:01:21.528 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.361538+0000", 2021-03-24T22:01:21.529 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00059871600000000001 2021-03-24T22:01:21.529 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.529 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.529 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:01:21.530 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.362137+0000", 2021-03-24T22:01:21.530 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0027486580000000002 2021-03-24T22:01:21.530 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.530 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.530 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:01:21.531 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.364886+0000", 2021-03-24T22:01:21.531 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.852e-05 2021-03-24T22:01:21.531 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.531 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.532 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:01:21.532 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.364964+0000", 2021-03-24T22:01:21.532 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9901000000000001e-05 2021-03-24T22:01:21.532 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.533 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.533 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:01:21.533 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.365024+0000", 2021-03-24T22:01:21.533 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0053478029999999999 2021-03-24T22:01:21.534 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:21.534 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:01:21.534 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:21.534 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.534 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.535 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:83 7.d 7:b2218182:::smithi12556470-21:head [setxattr _header (56) in=7b,truncate 2336140] snapc 0=[] ondisk+write+known_if_redirected e107)", 2021-03-24T22:01:21.535 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.359856+0000", 2021-03-24T22:01:21.535 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 404.95675509099999, 2021-03-24T22:01:21.535 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0053737860000000002, 2021-03-24T22:01:21.536 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:01:21.536 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:01:21.536 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:01:21.536 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:01:21.537 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:01:21.537 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 83 2021-03-24T22:01:21.537 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.537 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:01:21.538 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.538 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:01:21.538 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359856+0000", 2021-03-24T22:01:21.538 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:01:21.539 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.539 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.539 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:01:21.539 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359856+0000", 2021-03-24T22:01:21.539 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T22:01:21.540 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.540 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.540 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:01:21.540 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359854+0000", 2021-03-24T22:01:21.541 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.8180000000000004e-06 2021-03-24T22:01:21.541 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.541 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.541 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:01:21.542 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359862+0000", 2021-03-24T22:01:21.542 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.0979999999999996e-06 2021-03-24T22:01:21.542 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.542 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.543 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:01:21.543 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359869+0000", 2021-03-24T22:01:21.543 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2680000000000002e-06 2021-03-24T22:01:21.543 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.543 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.544 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:01:21.544 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359875+0000", 2021-03-24T22:01:21.544 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00094160000000000001 2021-03-24T22:01:21.544 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.545 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.545 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:21.545 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.360816+0000", 2021-03-24T22:01:21.545 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.3867000000000002e-05 2021-03-24T22:01:21.545 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.546 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.546 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:01:21.546 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.360900+0000", 2021-03-24T22:01:21.546 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00015279599999999999 2021-03-24T22:01:21.547 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.547 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.547 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,3", 2021-03-24T22:01:21.547 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.361053+0000", 2021-03-24T22:01:21.548 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00049272900000000002 2021-03-24T22:01:21.548 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.548 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.548 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:01:21.549 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.361546+0000", 2021-03-24T22:01:21.549 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00077420799999999997 2021-03-24T22:01:21.549 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.549 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.549 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:01:21.550 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.362320+0000", 2021-03-24T22:01:21.550 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0027734880000000002 2021-03-24T22:01:21.550 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.550 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.551 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:01:21.551 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.365093+0000", 2021-03-24T22:01:21.551 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.1638999999999997e-05 2021-03-24T22:01:21.551 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.552 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.552 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:01:21.552 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.365155+0000", 2021-03-24T22:01:21.552 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.4994999999999996e-05 2021-03-24T22:01:21.552 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.553 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.553 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:01:21.553 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.365230+0000", 2021-03-24T22:01:21.553 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0053737860000000002 2021-03-24T22:01:21.554 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:21.554 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:01:21.554 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:21.554 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.555 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.555 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:84 7.d 7:b2218182:::smithi12556470-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e107)", 2021-03-24T22:01:21.555 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.359915+0000", 2021-03-24T22:01:21.555 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 404.956696534, 2021-03-24T22:01:21.556 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0056067570000000004, 2021-03-24T22:01:21.556 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:01:21.556 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T22:01:21.556 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:01:21.556 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:01:21.557 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:01:21.557 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 84 2021-03-24T22:01:21.557 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.557 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:01:21.558 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.558 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:01:21.558 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359915+0000", 2021-03-24T22:01:21.558 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:01:21.559 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.559 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.559 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:01:21.559 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359915+0000", 2021-03-24T22:01:21.559 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T22:01:21.560 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.560 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.560 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:01:21.560 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359914+0000", 2021-03-24T22:01:21.561 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.8439999999999997e-06 2021-03-24T22:01:21.561 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.561 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.561 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:01:21.562 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359918+0000", 2021-03-24T22:01:21.562 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0258000000000001e-05 2021-03-24T22:01:21.562 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.562 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.563 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:01:21.563 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359929+0000", 2021-03-24T22:01:21.563 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.8179999999999998e-06 2021-03-24T22:01:21.563 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.563 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.564 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:01:21.564 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.359935+0000", 2021-03-24T22:01:21.564 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0013444279999999999 2021-03-24T22:01:21.564 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.565 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.565 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:21.565 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.361280+0000", 2021-03-24T22:01:21.565 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.7156999999999993e-05 2021-03-24T22:01:21.565 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.566 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.566 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:01:21.566 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.361357+0000", 2021-03-24T22:01:21.566 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0039437489999999999 2021-03-24T22:01:21.567 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.567 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.567 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:21.567 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.365301+0000", 2021-03-24T22:01:21.568 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000100756 2021-03-24T22:01:21.568 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.568 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.568 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:01:21.568 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.365402+0000", 2021-03-24T22:01:21.569 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00011997099999999999 2021-03-24T22:01:21.569 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.569 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.569 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:01:21.570 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.365522+0000", 2021-03-24T22:01:21.570 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0056067570000000004 2021-03-24T22:01:21.570 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:21.570 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:01:21.571 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:21.571 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.571 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.571 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:121 7.b 7:d841eed8:::smithi12556470-31:head [write 504210~606529,stat] snapc 0=[] ondisk+write+known_if_redirected e107)", 2021-03-24T22:01:21.572 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.487723+0000", 2021-03-24T22:01:21.572 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 404.82888869599998, 2021-03-24T22:01:21.572 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0043501010000000003, 2021-03-24T22:01:21.572 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:01:21.572 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:01:21.573 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:01:21.573 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:01:21.573 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:01:21.573 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 121 2021-03-24T22:01:21.574 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.574 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:01:21.574 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.574 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:01:21.575 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.487723+0000", 2021-03-24T22:01:21.575 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:01:21.575 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.575 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.575 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:01:21.576 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.487723+0000", 2021-03-24T22:01:21.576 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999962 2021-03-24T22:01:21.576 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.576 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.577 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:01:21.577 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.487719+0000", 2021-03-24T22:01:21.577 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00036455999999999999 2021-03-24T22:01:21.577 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.577 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.578 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:01:21.578 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.488084+0000", 2021-03-24T22:01:21.578 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.7018000000000001e-05 2021-03-24T22:01:21.578 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.579 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.579 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:01:21.579 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.488111+0000", 2021-03-24T22:01:21.579 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1592e-05 2021-03-24T22:01:21.580 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.580 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.580 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:01:21.580 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.488122+0000", 2021-03-24T22:01:21.580 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.7318999999999996e-05 2021-03-24T22:01:21.581 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.581 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.581 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:21.581 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.488200+0000", 2021-03-24T22:01:21.582 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00019132599999999999 2021-03-24T22:01:21.582 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.582 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.582 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:01:21.582 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.488391+0000", 2021-03-24T22:01:21.583 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00022982699999999999 2021-03-24T22:01:21.583 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.583 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.583 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,5", 2021-03-24T22:01:21.584 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.488621+0000", 2021-03-24T22:01:21.584 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001590948 2021-03-24T22:01:21.584 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.584 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.585 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:01:21.585 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.490212+0000", 2021-03-24T22:01:21.585 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00096796600000000001 2021-03-24T22:01:21.585 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.585 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.586 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:01:21.586 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.491180+0000", 2021-03-24T22:01:21.586 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00076374199999999998 2021-03-24T22:01:21.586 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.587 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.587 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:01:21.587 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.491943+0000", 2021-03-24T22:01:21.587 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.4624999999999998e-05 2021-03-24T22:01:21.588 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.588 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.588 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:01:21.588 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.492018+0000", 2021-03-24T22:01:21.588 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.4759000000000002e-05 2021-03-24T22:01:21.589 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.589 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.589 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:01:21.589 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.492073+0000", 2021-03-24T22:01:21.590 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0043501010000000003 2021-03-24T22:01:21.590 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:21.590 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:01:21.590 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:21.591 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.591 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.591 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:124 7.b 7:d841eed8:::smithi12556470-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e107)", 2021-03-24T22:01:21.591 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.494851+0000", 2021-03-24T22:01:21.591 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 404.82176000499999, 2021-03-24T22:01:21.592 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0039240020000000002, 2021-03-24T22:01:21.592 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:01:21.592 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T22:01:21.592 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:01:21.593 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:01:21.593 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:01:21.593 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 124 2021-03-24T22:01:21.593 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.593 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:01:21.594 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.594 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:01:21.594 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.494851+0000", 2021-03-24T22:01:21.594 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:01:21.595 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.595 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.595 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:01:21.595 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.494851+0000", 2021-03-24T22:01:21.596 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T22:01:21.596 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.596 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.596 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:01:21.597 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.494850+0000", 2021-03-24T22:01:21.597 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.8899999999999998e-06 2021-03-24T22:01:21.597 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.597 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.598 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:01:21.598 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.494855+0000", 2021-03-24T22:01:21.598 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.5820000000000004e-06 2021-03-24T22:01:21.598 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.598 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.599 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:01:21.599 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.494863+0000", 2021-03-24T22:01:21.599 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.9119999999999997e-06 2021-03-24T22:01:21.599 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.600 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.600 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:01:21.600 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.494868+0000", 2021-03-24T22:01:21.600 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0023491929999999999 2021-03-24T22:01:21.601 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.601 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.601 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:21.601 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.497217+0000", 2021-03-24T22:01:21.601 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.4723999999999998e-05 2021-03-24T22:01:21.602 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.602 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.602 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:01:21.602 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.497292+0000", 2021-03-24T22:01:21.603 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001295278 2021-03-24T22:01:21.603 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.603 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.603 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:21.604 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.498587+0000", 2021-03-24T22:01:21.604 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.5016000000000001e-05 2021-03-24T22:01:21.604 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.604 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.604 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:01:21.605 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.498672+0000", 2021-03-24T22:01:21.605 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010340099999999999 2021-03-24T22:01:21.605 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.605 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.606 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:01:21.606 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.498775+0000", 2021-03-24T22:01:21.606 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0039240020000000002 2021-03-24T22:01:21.606 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:21.607 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:01:21.607 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:21.607 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.607 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.608 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:125 7.b 7:de4c3db5:::smithi12556470-32:head [write 521017~798131,stat] snapc 0=[] ondisk+write+known_if_redirected e107)", 2021-03-24T22:01:21.608 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.502054+0000", 2021-03-24T22:01:21.608 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 404.814557337, 2021-03-24T22:01:21.608 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0041060309999999996, 2021-03-24T22:01:21.608 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:01:21.609 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:01:21.609 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:01:21.609 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:01:21.609 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:01:21.610 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 125 2021-03-24T22:01:21.610 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.610 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:01:21.610 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.611 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:01:21.611 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.502054+0000", 2021-03-24T22:01:21.611 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:01:21.611 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.611 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.612 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:01:21.612 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.502054+0000", 2021-03-24T22:01:21.612 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999976 2021-03-24T22:01:21.612 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.613 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.613 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:01:21.613 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.502052+0000", 2021-03-24T22:01:21.613 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00030999899999999998 2021-03-24T22:01:21.613 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.614 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.614 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:01:21.614 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.502362+0000", 2021-03-24T22:01:21.614 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.1806000000000001e-05 2021-03-24T22:01:21.615 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.615 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.615 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:01:21.615 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.502383+0000", 2021-03-24T22:01:21.616 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.6349999999999995e-06 2021-03-24T22:01:21.616 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.616 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.616 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:01:21.617 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.502392+0000", 2021-03-24T22:01:21.617 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.0788000000000002e-05 2021-03-24T22:01:21.617 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.617 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.617 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:21.618 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.502443+0000", 2021-03-24T22:01:21.618 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00013060400000000001 2021-03-24T22:01:21.618 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.618 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.619 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:01:21.619 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.502573+0000", 2021-03-24T22:01:21.619 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00017257399999999999 2021-03-24T22:01:21.619 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.620 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.620 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,5", 2021-03-24T22:01:21.620 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.502746+0000", 2021-03-24T22:01:21.620 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00132138 2021-03-24T22:01:21.620 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.621 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.621 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:01:21.621 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.504067+0000", 2021-03-24T22:01:21.621 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0011008719999999999 2021-03-24T22:01:21.622 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.622 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.622 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:01:21.622 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.505168+0000", 2021-03-24T22:01:21.623 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00081040500000000004 2021-03-24T22:01:21.623 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.623 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.623 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:01:21.624 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.505979+0000", 2021-03-24T22:01:21.624 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.2396000000000001e-05 2021-03-24T22:01:21.624 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.624 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.624 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:01:21.625 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.506071+0000", 2021-03-24T22:01:21.625 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.8867000000000001e-05 2021-03-24T22:01:21.625 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.625 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.626 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:01:21.626 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.506160+0000", 2021-03-24T22:01:21.626 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0041060309999999996 2021-03-24T22:01:21.626 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:21.627 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:01:21.627 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:21.627 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.627 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.627 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:137 7.0 e107/99 7:0ece3580:::smithi12556470-35:head v 107'1, mlcod=0'0)", 2021-03-24T22:01:21.628 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.545040+0000", 2021-03-24T22:01:21.628 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 404.77157166299997, 2021-03-24T22:01:21.628 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0054328359999999999, 2021-03-24T22:01:21.628 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:01:21.629 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:01:21.629 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:01:21.629 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.629 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:01:21.630 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.545040+0000", 2021-03-24T22:01:21.630 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:01:21.630 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.630 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.630 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:01:21.631 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.545040+0000", 2021-03-24T22:01:21.631 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999957 2021-03-24T22:01:21.631 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.631 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.632 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:01:21.632 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.545035+0000", 2021-03-24T22:01:21.632 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.004385702 2021-03-24T22:01:21.632 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.633 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.633 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:01:21.633 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.549421+0000", 2021-03-24T22:01:21.633 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.6047999999999999e-05 2021-03-24T22:01:21.633 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.634 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.634 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:01:21.634 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.549447+0000", 2021-03-24T22:01:21.634 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0923e-05 2021-03-24T22:01:21.635 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.635 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.635 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:01:21.635 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.549458+0000", 2021-03-24T22:01:21.635 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.0572000000000002e-05 2021-03-24T22:01:21.636 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.636 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.636 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:21.636 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.549519+0000", 2021-03-24T22:01:21.637 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.2843999999999999e-05 2021-03-24T22:01:21.637 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.637 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.637 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:01:21.637 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.549552+0000", 2021-03-24T22:01:21.638 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00088233300000000001 2021-03-24T22:01:21.638 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.638 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.638 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:01:21.639 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.550434+0000", 2021-03-24T22:01:21.639 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.8571000000000001e-05 2021-03-24T22:01:21.639 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.639 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.640 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:01:21.640 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.550472+0000", 2021-03-24T22:01:21.640 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0054328359999999999 2021-03-24T22:01:21.640 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:21.641 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:01:21.641 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:21.641 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.641 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.641 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:178 7.d 7:ba8515f1:::smithi12556470-45:head [write 1875090~648012] snapc 0=[] ondisk+write+known_if_redirected e107)", 2021-03-24T22:01:21.642 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.706281+0000", 2021-03-24T22:01:21.642 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 404.610330379, 2021-03-24T22:01:21.642 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0046999620000000002, 2021-03-24T22:01:21.642 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:01:21.643 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:01:21.643 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:01:21.643 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:01:21.643 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:01:21.644 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 178 2021-03-24T22:01:21.644 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.644 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:01:21.644 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.644 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:01:21.645 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706281+0000", 2021-03-24T22:01:21.645 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:01:21.645 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.645 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.646 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:01:21.646 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706281+0000", 2021-03-24T22:01:21.646 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999971 2021-03-24T22:01:21.646 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.647 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.647 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:01:21.647 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706278+0000", 2021-03-24T22:01:21.647 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00022174299999999999 2021-03-24T22:01:21.647 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.648 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.648 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:01:21.648 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706500+0000", 2021-03-24T22:01:21.648 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.3224e-05 2021-03-24T22:01:21.649 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.649 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.649 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:01:21.649 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706523+0000", 2021-03-24T22:01:21.650 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7773000000000001e-05 2021-03-24T22:01:21.650 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.650 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.650 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:01:21.650 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706541+0000", 2021-03-24T22:01:21.651 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000101025 2021-03-24T22:01:21.651 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.651 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.651 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:21.652 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706642+0000", 2021-03-24T22:01:21.652 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00019631299999999999 2021-03-24T22:01:21.652 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.652 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.652 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:01:21.653 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706838+0000", 2021-03-24T22:01:21.653 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000349081 2021-03-24T22:01:21.653 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.653 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.654 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,3", 2021-03-24T22:01:21.654 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.707187+0000", 2021-03-24T22:01:21.654 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002206515 2021-03-24T22:01:21.654 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.654 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.655 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:01:21.655 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.709394+0000", 2021-03-24T22:01:21.655 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00031391600000000002 2021-03-24T22:01:21.655 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.656 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.656 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:01:21.656 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.709708+0000", 2021-03-24T22:01:21.656 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0011238960000000001 2021-03-24T22:01:21.656 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.657 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.657 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:01:21.657 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.710832+0000", 2021-03-24T22:01:21.657 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.9524000000000006e-05 2021-03-24T22:01:21.658 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.658 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.658 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:01:21.658 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.710911+0000", 2021-03-24T22:01:21.658 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.9733999999999993e-05 2021-03-24T22:01:21.659 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.659 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.659 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:01:21.659 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.710981+0000", 2021-03-24T22:01:21.660 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0046999620000000002 2021-03-24T22:01:21.660 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:21.660 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:01:21.660 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:21.660 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.661 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.661 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:179 7.d 7:ba8515f1:::smithi12556470-45:head [setxattr _header (56) in=7b,truncate 2739508] snapc 0=[] ondisk+write+known_if_redirected e107)", 2021-03-24T22:01:21.661 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.706566+0000", 2021-03-24T22:01:21.661 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 404.610045614, 2021-03-24T22:01:21.661 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.004679417, 2021-03-24T22:01:21.662 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:01:21.662 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:01:21.662 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:01:21.662 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:01:21.663 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:01:21.663 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 179 2021-03-24T22:01:21.663 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.663 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:01:21.663 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.664 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:01:21.664 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706566+0000", 2021-03-24T22:01:21.664 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:01:21.664 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.665 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.665 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:01:21.665 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706566+0000", 2021-03-24T22:01:21.665 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T22:01:21.665 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.666 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.666 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:01:21.666 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706564+0000", 2021-03-24T22:01:21.666 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.3519999999999996e-06 2021-03-24T22:01:21.666 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.667 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.667 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:01:21.667 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706573+0000", 2021-03-24T22:01:21.667 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.4070000000000001e-05 2021-03-24T22:01:21.668 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.668 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.668 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:01:21.668 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706587+0000", 2021-03-24T22:01:21.668 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.7489999999999993e-06 2021-03-24T22:01:21.669 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.669 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.669 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:01:21.669 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706597+0000", 2021-03-24T22:01:21.670 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001623022 2021-03-24T22:01:21.670 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.670 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.670 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:21.671 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.708220+0000", 2021-03-24T22:01:21.671 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00013183199999999999 2021-03-24T22:01:21.671 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.671 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.671 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:01:21.672 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.708352+0000", 2021-03-24T22:01:21.672 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00025474500000000001 2021-03-24T22:01:21.672 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.672 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.673 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,3", 2021-03-24T22:01:21.673 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.708606+0000", 2021-03-24T22:01:21.673 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00080621600000000001 2021-03-24T22:01:21.673 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.674 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.674 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:01:21.674 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.709413+0000", 2021-03-24T22:01:21.674 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00047969100000000003 2021-03-24T22:01:21.674 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.675 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.675 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:01:21.675 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.709892+0000", 2021-03-24T22:01:21.675 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001199615 2021-03-24T22:01:21.676 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.676 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.676 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:01:21.676 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.711092+0000", 2021-03-24T22:01:21.676 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0001032 2021-03-24T22:01:21.677 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.677 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.677 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:01:21.677 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.711195+0000", 2021-03-24T22:01:21.678 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.9874999999999999e-05 2021-03-24T22:01:21.678 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.678 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.678 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:01:21.679 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.711245+0000", 2021-03-24T22:01:21.679 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.004679417 2021-03-24T22:01:21.679 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:21.679 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:01:21.679 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:21.680 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.680 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.680 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:180 7.d 7:ba8515f1:::smithi12556470-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e107)", 2021-03-24T22:01:21.680 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:54:36.706637+0000", 2021-03-24T22:01:21.681 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 404.60997431599998, 2021-03-24T22:01:21.681 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0049820649999999999, 2021-03-24T22:01:21.681 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:01:21.681 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T22:01:21.681 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:01:21.682 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:01:21.682 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:01:21.682 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 180 2021-03-24T22:01:21.682 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.683 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:01:21.683 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.683 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:01:21.683 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706637+0000", 2021-03-24T22:01:21.684 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:01:21.684 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.684 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.684 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:01:21.684 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706637+0000", 2021-03-24T22:01:21.685 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T22:01:21.685 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.685 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.685 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:01:21.686 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706636+0000", 2021-03-24T22:01:21.686 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5099999999999998e-06 2021-03-24T22:01:21.686 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.686 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.686 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:01:21.687 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706641+0000", 2021-03-24T22:01:21.687 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.6740000000000003e-06 2021-03-24T22:01:21.687 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.687 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.688 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:01:21.688 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706650+0000", 2021-03-24T22:01:21.688 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.7699999999999998e-06 2021-03-24T22:01:21.688 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.689 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.689 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:01:21.689 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.706656+0000", 2021-03-24T22:01:21.689 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0023962749999999998 2021-03-24T22:01:21.689 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.690 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.690 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:21.690 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.709052+0000", 2021-03-24T22:01:21.690 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000193926 2021-03-24T22:01:21.691 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.691 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.691 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:01:21.691 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.709246+0000", 2021-03-24T22:01:21.691 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002080279 2021-03-24T22:01:21.692 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.692 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.692 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:21.692 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.711326+0000", 2021-03-24T22:01:21.693 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00013017399999999999 2021-03-24T22:01:21.693 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.693 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.693 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:01:21.693 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.711456+0000", 2021-03-24T22:01:21.694 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000162767 2021-03-24T22:01:21.694 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.694 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.694 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:01:21.695 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:54:36.711619+0000", 2021-03-24T22:01:21.695 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0049820649999999999 2021-03-24T22:01:21.695 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:21.695 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:01:21.696 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:21.696 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.696 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.696 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "MOSDPGPush(7.9 509/505 [PushOp(7:9511f6a7:::smithi12556470-23:149, version: 483'297, data_included: [2525551~197249], data_size: 197249, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:9511f6a7:::smithi12556470-23:149@483'297, size: 3176589, copy_subset: [531253~411999,1381780~411527,2300501~586966], clone_subset: {7:9511f6a7:::smithi12556470-23:157=[0~531253,943252~438528,1793307~507194,2887467~289122]}, snapset: 176=[]:{149=[139,136,129,127,11b,104,fa],157=[157,153],172=[172,16e,16d,168,166,165,160,15e,159],176=[176]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2887467, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2525551, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-03-24T22:01:21.697 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:01:20.159160+0000", 2021-03-24T22:01:21.697 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 1.1574511270000001, 2021-03-24T22:01:21.697 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0041966850000000003, 2021-03-24T22:01:21.697 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:01:21.697 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T22:01:21.698 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:01:21.698 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.698 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:01:21.698 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:20.159160+0000", 2021-03-24T22:01:21.699 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:01:21.699 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.699 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.699 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:01:21.700 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:20.159160+0000", 2021-03-24T22:01:21.700 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999976 2021-03-24T22:01:21.700 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.700 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.700 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:01:21.701 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:20.159158+0000", 2021-03-24T22:01:21.701 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00027509000000000002 2021-03-24T22:01:21.701 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.701 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.702 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:01:21.702 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:20.159433+0000", 2021-03-24T22:01:21.702 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.4812999999999999e-05 2021-03-24T22:01:21.702 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.703 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.703 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:01:21.703 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:20.159478+0000", 2021-03-24T22:01:21.703 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.8510999999999999e-05 2021-03-24T22:01:21.703 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.704 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.704 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:01:21.704 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:20.159506+0000", 2021-03-24T22:01:21.704 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00017667099999999999 2021-03-24T22:01:21.705 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.705 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.705 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:21.705 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:20.159683+0000", 2021-03-24T22:01:21.706 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.6796999999999999e-05 2021-03-24T22:01:21.706 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.706 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.706 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:01:21.706 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:20.159700+0000", 2021-03-24T22:01:21.707 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.003657263 2021-03-24T22:01:21.707 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:21.707 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:21.707 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:01:21.708 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:20.163357+0000", 2021-03-24T22:01:21.708 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0041966850000000003 2021-03-24T22:01:21.708 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:21.708 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:01:21.708 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:21.709 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:21.709 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:01:21.709 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-24T22:01:21.710 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:21.712 INFO:tasks.rados.rados.0.smithi125.stdout:1913: finishing write tid 2 to smithi12556470-20 2021-03-24T22:01:21.713 INFO:tasks.rados.rados.0.smithi125.stdout:1913: finishing write tid 3 to smithi12556470-20 2021-03-24T22:01:21.713 INFO:tasks.rados.rados.0.smithi125.stdout:1913: finishing write tid 4 to smithi12556470-20 2021-03-24T22:01:21.713 INFO:tasks.rados.rados.0.smithi125.stdout:1915: finishing write tid 1 to smithi12556470-2 2021-03-24T22:01:21.713 INFO:tasks.rados.rados.0.smithi125.stdout:1913: finishing write tid 5 to smithi12556470-20 2021-03-24T22:01:21.714 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 20 v 488 (ObjNum 675 snap 210 seq_num 675) dirty exists 2021-03-24T22:01:21.714 INFO:tasks.rados.rados.0.smithi125.stdout:1913: left oid 20 (ObjNum 675 snap 210 seq_num 675) 2021-03-24T22:01:21.714 INFO:tasks.rados.rados.0.smithi125.stdout:1915: finishing write tid 2 to smithi12556470-2 2021-03-24T22:01:21.714 INFO:tasks.rados.rados.0.smithi125.stdout:1915: finishing write tid 3 to smithi12556470-2 2021-03-24T22:01:21.714 INFO:tasks.rados.rados.0.smithi125.stdout:1915: finishing write tid 4 to smithi12556470-2 2021-03-24T22:01:21.715 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 2 v 692 (ObjNum 676 snap 210 seq_num 676) dirty exists 2021-03-24T22:01:21.715 INFO:tasks.rados.rados.0.smithi125.stdout:1915: left oid 2 (ObjNum 676 snap 210 seq_num 676) 2021-03-24T22:01:21.715 INFO:tasks.rados.rados.0.smithi125.stdout:1916: expect (ObjNum 470 snap 133 seq_num 470) 2021-03-24T22:01:21.715 INFO:tasks.rados.rados.0.smithi125.stdout:1918: finishing write tid 1 to smithi12556470-11 2021-03-24T22:01:21.716 INFO:tasks.rados.rados.0.smithi125.stdout:1907: done (9 left) 2021-03-24T22:01:21.716 INFO:tasks.rados.rados.0.smithi125.stdout:1910: done (8 left) 2021-03-24T22:01:21.716 INFO:tasks.rados.rados.0.smithi125.stdout:1912: done (7 left) 2021-03-24T22:01:21.716 INFO:tasks.rados.rados.0.smithi125.stdout:1913: done (6 left) 2021-03-24T22:01:21.716 INFO:tasks.rados.rados.0.smithi125.stdout:1914: done (5 left) 2021-03-24T22:01:21.717 INFO:tasks.rados.rados.0.smithi125.stdout:1915: done (4 left) 2021-03-24T22:01:21.717 INFO:tasks.rados.rados.0.smithi125.stdout:1916: done (3 left) 2021-03-24T22:01:21.717 INFO:tasks.rados.rados.0.smithi125.stdout:1921: read oid 7 snap 156 2021-03-24T22:01:21.717 INFO:tasks.rados.rados.0.smithi125.stdout:1921: expect (ObjNum 527 snap 155 seq_num 527) 2021-03-24T22:01:21.718 INFO:tasks.rados.rados.0.smithi125.stdout:1922: delete oid 28 current snap is 210 2021-03-24T22:01:21.718 INFO:tasks.rados.rados.0.smithi125.stdout:1918: finishing write tid 2 to smithi12556470-11 2021-03-24T22:01:21.718 INFO:tasks.rados.rados.0.smithi125.stdout:1918: finishing write tid 3 to smithi12556470-11 2021-03-24T22:01:21.718 INFO:tasks.rados.rados.0.smithi125.stdout:1918: finishing write tid 4 to smithi12556470-11 2021-03-24T22:01:21.718 INFO:tasks.rados.rados.0.smithi125.stdout:1918: finishing write tid 5 to smithi12556470-11 2021-03-24T22:01:21.719 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 11 v 327 (ObjNum 677 snap 210 seq_num 677) dirty exists 2021-03-24T22:01:21.719 INFO:tasks.rados.rados.0.smithi125.stdout:1918: left oid 11 (ObjNum 677 snap 210 seq_num 677) 2021-03-24T22:01:21.719 INFO:tasks.rados.rados.0.smithi125.stdout:1918: done (4 left) 2021-03-24T22:01:21.719 INFO:tasks.rados.rados.0.smithi125.stdout:1919: done (3 left) 2021-03-24T22:01:21.720 INFO:tasks.rados.rados.0.smithi125.stdout:1920: done (2 left) 2021-03-24T22:01:21.720 INFO:tasks.rados.rados.0.smithi125.stdout:1922: done (1 left) 2021-03-24T22:01:21.720 INFO:tasks.rados.rados.0.smithi125.stdout:1923: rollback oid 47 current snap is 210 2021-03-24T22:01:21.720 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 47 to 192 2021-03-24T22:01:21.720 INFO:tasks.rados.rados.0.smithi125.stdout:1924: copy_from oid 26 from oid 28 current snap is 210 2021-03-24T22:01:21.721 INFO:tasks.rados.rados.0.smithi125.stdout:1925: snap_create 2021-03-24T22:01:21.721 INFO:tasks.rados.rados.0.smithi125.stdout:1924: finishing copy_from to smithi12556470-26 2021-03-24T22:01:21.721 INFO:tasks.rados.rados.0.smithi125.stdout:1924: got expected ENOENT (src dne) 2021-03-24T22:01:21.721 INFO:tasks.rados.rados.0.smithi125.stdout:1924: finishing copy_from racing read to smithi12556470-26 2021-03-24T22:01:21.722 INFO:tasks.rados.rados.0.smithi125.stdout:1921: expect (ObjNum 527 snap 155 seq_num 527) 2021-03-24T22:01:21.767 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:21.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:21.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:01:21.782 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 7, 1] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2021-03-24T22:01:21.783 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-24T22:01:21.784 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [0, 2, 3, 4, 5, 7, 1] 2021-03-24T22:01:21.784 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd out 7 2021-03-24T22:01:21.873 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:21.925 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:21.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:21.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:01:21.933 INFO:tasks.rados.rados.0.smithi125.stdout:1923: finishing rollback tid 0 to smithi12556470-47 2021-03-24T22:01:21.934 INFO:tasks.rados.rados.0.smithi125.stdout:1923: finishing rollback tid 1 to smithi12556470-47 2021-03-24T22:01:21.934 INFO:tasks.rados.rados.0.smithi125.stdout:1923: finishing rollback tid 2 to smithi12556470-47 2021-03-24T22:01:21.935 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 47 v 340 (ObjNum 466 snap 131 seq_num 466) dirty exists 2021-03-24T22:01:22.030 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:22.090 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:22.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:22.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:01:22.195 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:22.244 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:22.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:22.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:01:22.296 INFO:tasks.rados.rados.0.smithi125.stdout:1921: done (3 left) 2021-03-24T22:01:22.296 INFO:tasks.rados.rados.0.smithi125.stdout:1923: done (2 left) 2021-03-24T22:01:22.296 INFO:tasks.rados.rados.0.smithi125.stdout:1924: done (1 left) 2021-03-24T22:01:22.297 INFO:tasks.rados.rados.0.smithi125.stdout:1925: done (0 left) 2021-03-24T22:01:22.297 INFO:tasks.rados.rados.0.smithi125.stdout:1926: snap_create 2021-03-24T22:01:22.302 INFO:teuthology.orchestra.run.smithi125.stderr:nodeep-scrub is set 2021-03-24T22:01:22.350 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:22.404 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:22.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:22.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:01:22.499 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:22 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T22:01:22.500 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:22 smithi125 conmon[19936]: -24T22:01:20.879198+0000 mgr.y (mgr.14144) 527 : cluster [DBG] pgmap v928: 168 pgs: 2 peering, 1 activating, 1 remapped+peering, 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+recovering+remapped, 2 active+recovery_wait+degraded, 3 active+remapped+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 144 active+clean; 483 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 13 op/s; 86/1647 objects degraded (5.222%); 238/1647 objects misplaced (14.451%); 11 MiB/s, 46 objects/s recovering 2021-03-24T22:01:22.501 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:22 smithi125 conmon[19936]: cluster 2021-03-24T22:01:22.501 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:22 smithi125 conmon[19936]: 2021-03-24T22:01:20 2021-03-24T22:01:22.502 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:22 smithi125 conmon[19936]: .945137+0000 osd.6 (osd. 2021-03-24T22:01:22.502 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:22 smithi125 conmon[19936]: 6) 31 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:01:22.502 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:22 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:01:22.503 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:22 smithi125 conmon[19936]: -03-24T22:01:20.945274+0000 2021-03-24T22:01:22.503 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:22 smithi125 conmon[19936]: osd.6 (osd.6) 32 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:01:22.503 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:22 smithi125 conmon[19936]: cluster 2021-03-24T22:01:21. 2021-03-24T22:01:22.504 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:22 smithi125 conmon[19936]: 276182+0000 mon.a (mon.0) 1471 : cluster [DBG] osdmap e511: 8 total, 8 up, 7 in 2021-03-24T22:01:22.504 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:22 smithi125 conmon[19936]: cluster 2021-03-24T22:01:21.332359+0000 2021-03-24T22:01:22.504 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:22 smithi125 conmon[19936]: mon.a (mon.0) 1472 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:01:22.504 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:22 smithi125 conmon[19936]: audit 2021- 2021-03-24T22:01:22.504 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:22 smithi125 conmon[19936]: 03-24T22:01:21.802298+0000 2021-03-24T22:01:22.505 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:22 smithi125 conmon[19936]: mon.c (mon.2) 80 : audit [INF] 2021-03-24T22:01:22.505 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:22 smithi125 conmon[19936]: from='client.? v1:172.21.15.125:0/2906985014' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:01:22.505 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:22 smithi125 conmon[19936]: audit 2021-03-24T22:01:22.505 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:22 smithi125 conmon[19936]: 2021-03-24T22:01:21.802637+0000 mon.a 2021-03-24T22:01:22.506 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:22 smithi125 conmon[19936]: (mon.0) 1473 : audit [INF] 2021-03-24T22:01:22.506 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:22 smithi125 conmon[19936]: from='client.35363 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:01:22.506 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:22 smithi125 conmon[24927]: cluster 2021-03-24T22:01:20.879198+0000 mgr.y (mgr.14144) 527 : cluster [DBG] pgmap v928: 168 pgs: 2 peering, 1 activating, 1 remapped+peering, 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+recovering+remapped, 2 active+recovery_wait+degraded, 3 active+remapped+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 144 active+clean; 483 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 13 op/s; 86/1647 objects degraded (5.222%); 238/1647 objects misplaced (14.451%); 11 MiB/s, 46 objects/s recovering 2021-03-24T22:01:22.507 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:22 smithi125 conmon[24927]: cluster 2021-03-24T22:01:20.945137+0000 osd.6 (osd.6) 31 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:01:22.507 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:22 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T22:01:22.507 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:22 smithi125 conmon[24927]: T22:01:20.945274+0000 osd.6 (osd.6) 2021-03-24T22:01:22.507 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:22 smithi125 conmon[24927]: 32 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:01:22.507 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:22 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:01:22.508 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:22 smithi125 conmon[24927]: 03-24T22:01:21.276182+0000 mon.a 2021-03-24T22:01:22.508 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:22 smithi125 conmon[24927]: (mon.0) 1471 : cluster [DBG] osdmap e511: 8 total, 8 up, 7 in 2021-03-24T22:01:22.508 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:22 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:01:22.509 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:22 smithi125 conmon[24927]: -03-24T22:01:21.332359+0000 2021-03-24T22:01:22.509 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:22 smithi125 conmon[24927]: mon.a (mon.0) 1472 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:01:22.509 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:22 smithi125 conmon[24927]: audit 2021-03-24T22 2021-03-24T22:01:22.509 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:22 smithi125 conmon[24927]: :01:21.802298+0000 mon.c (mon.2 2021-03-24T22:01:22.509 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:22 smithi125 conmon[24927]: ) 80 : audit [INF] from='client.? v1:172.21.15.125:0/2906985014' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:01:22.510 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:22 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T22:01:22.510 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:22 smithi125 conmon[24927]: 24T22:01:21.802637+0000 mon.a (mon 2021-03-24T22:01:22.510 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:22 smithi125 conmon[24927]: .0) 1473 : audit [INF] from='client.35363 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:01:22.511 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:22.563 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:22.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:22.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:01:22.569 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:22 smithi142 conmon[20125]: cluster 2021-03-24T22:01:20.879198+0000 mgr.y (mgr. 2021-03-24T22:01:22.569 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:22 smithi142 conmon[20125]: 14144) 527 : cluster [DBG] pgmap v928: 168 pgs: 2 peering, 1 activating, 1 remapped+peering, 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+recovering+remapped, 2 active+recovery_wait+degraded, 3 active+remapped+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 144 active+clean; 483 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 13 op/s; 86/1647 objects degraded (5.222%); 238/1647 objects misplaced (14.451%); 11 MiB/s, 46 objects/s recovering 2021-03-24T22:01:22.569 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:22 smithi142 conmon[20125]: cluster 2021-03-24T22:01:20.945137+0000 2021-03-24T22:01:22.570 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:22 smithi142 conmon[20125]: osd.6 (osd.6) 31 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:01:22.570 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:22 smithi142 conmon[20125]: cluster 2021-03-24T22: 2021-03-24T22:01:22.570 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:22 smithi142 conmon[20125]: 01:20.945274+0000 osd.6 (osd.6) 2021-03-24T22:01:22.570 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:22 smithi142 conmon[20125]: 32 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:01:22.571 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:22 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:01:22.571 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:22 smithi142 conmon[20125]: 22:01:21.276182+0000 mon.a (mon. 2021-03-24T22:01:22.571 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:22 smithi142 conmon[20125]: 0) 1471 : cluster [DBG] osdmap e511: 8 total, 8 up, 7 in 2021-03-24T22:01:22.572 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:22 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:01:22.572 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:22 smithi142 conmon[20125]: 03-24T22:01:21.332359+0000 2021-03-24T22:01:22.572 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:22 smithi142 conmon[20125]: mon.a (mon.0) 1472 : cluster [WRN] 2021-03-24T22:01:22.572 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:22 smithi142 conmon[20125]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:01:22.573 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:22 smithi142 conmon[20125]: audit 2021-03-24T 2021-03-24T22:01:22.573 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:22 smithi142 conmon[20125]: 22:01:21.802298+0000 mon.c (mon 2021-03-24T22:01:22.573 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:22 smithi142 conmon[20125]: .2) 80 : audit [INF] from='client.? v1:172.21.15.125:0/2906985014' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:01:22.573 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:22 smithi142 conmon[20125]: audit 2021-03- 2021-03-24T22:01:22.574 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:22 smithi142 conmon[20125]: 24T22:01:21.802637+0000 mon.a 2021-03-24T22:01:22.574 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:22 smithi142 conmon[20125]: (mon.0) 1473 : audit [INF] from='client.35363 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:01:22.575 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:22 smithi142 conmon[20125]: 2021-03-24T22:01:22.660 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.2 dump_ops_in_flight 2021-03-24T22:01:22.670 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:22.724 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:22.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:22.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:01:22.831 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:22.879 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:22.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:22.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:01:22.985 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:23.032 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-24T22:01:23.038 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:23.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:23.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:01:23.146 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:23.207 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:23.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:23.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:01:23.305 INFO:tasks.rados.rados.0.smithi125.stdout:1926: done (0 left) 2021-03-24T22:01:23.305 INFO:tasks.rados.rados.0.smithi125.stdout:1927: delete oid 42 current snap is 212 2021-03-24T22:01:23.314 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:23.322 INFO:tasks.rados.rados.0.smithi125.stdout:1927: done (0 left) 2021-03-24T22:01:23.322 INFO:tasks.rados.rados.0.smithi125.stdout:1928: write (excl) oid 46 current snap is 212 2021-03-24T22:01:23.323 INFO:tasks.rados.rados.0.smithi125.stdout:1928: seq_num 680 ranges {628760=724737,1892226=673223,3157635=271125} 2021-03-24T22:01:23.327 INFO:tasks.rados.rados.0.smithi125.stdout:1928: writing smithi12556470-46 from 628760 to 1353497 tid 1 2021-03-24T22:01:23.336 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:01:23.339 INFO:tasks.rados.rados.0.smithi125.stdout:1928: writing smithi12556470-46 from 1892226 to 2565449 tid 2 2021-03-24T22:01:23.345 INFO:tasks.rados.rados.0.smithi125.stdout:1928: writing smithi12556470-46 from 3157635 to 3428760 tid 3 2021-03-24T22:01:23.345 INFO:tasks.rados.rados.0.smithi125.stdout:1929: read oid 21 snap 210 2021-03-24T22:01:23.346 INFO:tasks.rados.rados.0.smithi125.stdout:1929: expect (ObjNum 590 snap 177 seq_num 590) 2021-03-24T22:01:23.346 INFO:tasks.rados.rados.0.smithi125.stdout:1930: rollback oid 30 current snap is 212 2021-03-24T22:01:23.346 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 30 to 201 2021-03-24T22:01:23.347 INFO:tasks.rados.rados.0.smithi125.stdout:1928: finishing write tid 1 to smithi12556470-46 2021-03-24T22:01:23.347 INFO:tasks.rados.rados.0.smithi125.stdout:1931: read oid 15 snap -1 2021-03-24T22:01:23.347 INFO:tasks.rados.rados.0.smithi125.stdout:1931: expect deleted 2021-03-24T22:01:23.347 INFO:tasks.rados.rados.0.smithi125.stdout:1932: snap_create 2021-03-24T22:01:23.351 INFO:tasks.rados.rados.0.smithi125.stdout:1928: finishing write tid 2 to smithi12556470-46 2021-03-24T22:01:23.353 INFO:tasks.rados.rados.0.smithi125.stdout:1930: finishing rollback tid 0 to smithi12556470-30 2021-03-24T22:01:23.353 INFO:tasks.rados.rados.0.smithi125.stdout:1930: finishing rollback tid 1 to smithi12556470-30 2021-03-24T22:01:23.355 INFO:tasks.rados.rados.0.smithi125.stdout:1930: finishing rollback tid 2 to smithi12556470-30 2021-03-24T22:01:23.355 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 30 v 295 (ObjNum 601 snap 180 seq_num 601) dirty exists 2021-03-24T22:01:23.355 INFO:tasks.rados.rados.0.smithi125.stdout:1929: expect (ObjNum 590 snap 177 seq_num 590) 2021-03-24T22:01:23.360 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:23.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:23.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:01:23.466 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:23.527 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:23.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:23.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:01:23.532 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:23 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:01:23.532 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:23 smithi125 conmon[19936]: T22:01:22.274443+0000 mon.a (mon.0) 1474 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:01:23.532 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:23 smithi125 conmon[19936]: 2021-03-24T22:01:23.532 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:23 smithi125 conmon[19936]: audit 2021-03-24T22:01:23.533 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:23 smithi125 conmon[19936]: 2021-03-24T22:01:22 2021-03-24T22:01:23.533 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:23 smithi125 conmon[19936]: .291692+0000 mon.a ( 2021-03-24T22:01:23.533 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:23 smithi125 conmon[19936]: mon.0) 1475 : audit [INF] 2021-03-24T22:01:23.534 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:23 smithi125 conmon[19936]: from='client.35363 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T22:01:23.534 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:23 smithi125 conmon[19936]: cluster 2021-03-24T22:01:23.534 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:23 smithi125 conmon[19936]: 2021-03-24T22:01: 2021-03-24T22:01:23.534 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:23 smithi125 conmon[19936]: 22.291843+0000 mon.a ( 2021-03-24T22:01:23.535 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:23 smithi125 conmon[19936]: mon.0) 1476 : cluster 2021-03-24T22:01:23.535 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:23 smithi125 conmon[19936]: [DBG] osdmap e512: 8 total, 8 up, 7 in 2021-03-24T22:01:23.535 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:23 smithi125 conmon[24927]: cluster 2021-03-24T22:01:22. 2021-03-24T22:01:23.536 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:23 smithi125 conmon[24927]: 274443+0000 mon.a (mon.0) 1474 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:01:23.536 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:23 smithi125 conmon[24927]: audit 2021 2021-03-24T22:01:23.536 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:23 smithi125 conmon[24927]: -03-24T22:01:22.291692+0000 mon.a (mon.0) 1475 : audit [INF] 2021-03-24T22:01:23.537 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:23 smithi125 conmon[24927]: from='client.35363 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T22:01:23.537 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:23 smithi125 conmon[24927]: cluster 2021-03-24T22:01:22.291843 2021-03-24T22:01:23.537 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:23 smithi125 conmon[24927]: +0000 mon.a (mon.0) 1476 : cluster [DBG] osdmap e512: 8 total, 8 up, 7 in 2021-03-24T22:01:23.537 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:23 smithi125 conmon[24927]: 2021-03-24T22:01:23.611 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:01:23.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:23 smithi142 conmon[20125]: cluster 2021-03-24T22:01: 2021-03-24T22:01:23.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:23 smithi142 conmon[20125]: 22.274443+0000 mon.a (mon.0) 1474 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:01:23.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:23 smithi142 conmon[20125]: audit 2021-03 2021-03-24T22:01:23.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:23 smithi142 conmon[20125]: -24T22:01:22.291692+0000 mon.a (mon.0) 1475 2021-03-24T22:01:23.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:23 smithi142 conmon[20125]: : audit [INF] from='client.35363 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T22:01:23.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:23 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:01:23.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:23 smithi142 conmon[20125]: 24T22:01:22.291843+0000 mon.a ( 2021-03-24T22:01:23.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:23 smithi142 conmon[20125]: mon.0) 1476 : cluster [DBG] osdmap e512: 8 total, 8 up, 7 in 2021-03-24T22:01:23.629 INFO:tasks.rados.rados.0.smithi125.stdout:1928: finishing write tid 3 to smithi12556470-46 2021-03-24T22:01:23.629 INFO:tasks.rados.rados.0.smithi125.stdout:1928: finishing write tid 4 to smithi12556470-46 2021-03-24T22:01:23.629 INFO:tasks.rados.rados.0.smithi125.stdout:1928: finishing write tid 5 to smithi12556470-46 2021-03-24T22:01:23.630 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 46 v 495 (ObjNum 680 snap 212 seq_num 680) dirty exists 2021-03-24T22:01:23.630 INFO:tasks.rados.rados.0.smithi125.stdout:1928: left oid 46 (ObjNum 680 snap 212 seq_num 680) 2021-03-24T22:01:23.632 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:23.680 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:01:23.695 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:23.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:23.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:01:23.800 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:23.843 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:23.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:23.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:01:23.948 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:24.008 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:24.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:24.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:01:24.114 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:24.170 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:24.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:24.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:01:24.274 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:24.309 INFO:tasks.rados.rados.0.smithi125.stdout:1928: done (4 left) 2021-03-24T22:01:24.310 INFO:tasks.rados.rados.0.smithi125.stdout:1929: done (3 left) 2021-03-24T22:01:24.310 INFO:tasks.rados.rados.0.smithi125.stdout:1930: done (2 left) 2021-03-24T22:01:24.310 INFO:tasks.rados.rados.0.smithi125.stdout:1931: done (1 left) 2021-03-24T22:01:24.310 INFO:tasks.rados.rados.0.smithi125.stdout:1932: done (0 left) 2021-03-24T22:01:24.311 INFO:tasks.rados.rados.0.smithi125.stdout:1933: snap_create 2021-03-24T22:01:24.320 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:24 smithi125 conmon[19936]: cluster 2021-03-24T22:01:22.879762+0000 mgr.y 2021-03-24T22:01:24.320 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:24 smithi125 conmon[19936]: (mgr.14144) 528 : cluster [DBG] pgmap v931: 168 pgs: 2 peering, 1 active+recovery_wait+remapped, 1 active+recovering+remapped, 5 active+remapped+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 1 active+remapped, 147 active+clean; 483 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 9.5 MiB/s rd, 3.1 MiB/s wr, 18 op/s; 78/1644 objects degraded (4.745%); 257/1644 objects misplaced (15.633%); 12 MiB/s, 49 objects/s recovering 2021-03-24T22:01:24.320 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:24 smithi125 conmon[19936]: cluster 2021-03-24T22:01:23.298105+0000 mon.a (mon.0) 1477 : cluster [DBG] osdmap e513: 8 total, 8 up, 7 in 2021-03-24T22:01:24.347 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:24.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:24.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:01:24.454 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:24.512 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:24.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:24.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:01:24.561 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:24 smithi125 conmon[24927]: cluster 2021-03-24T22:01:24.562 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:24 smithi125 conmon[24927]: 2021-03-24T22:01:24.562 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:24 smithi125 conmon[24927]: 2021 2021-03-24T22:01:24.562 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:24 smithi125 conmon[24927]: - 2021-03-24T22:01:24.563 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:24 smithi125 conmon[24927]: 03 2021-03-24T22:01:24.563 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:24 smithi125 conmon[24927]: - 2021-03-24T22:01:24.564 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:24 smithi125 conmon[24927]: 24 2021-03-24T22:01:24.564 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:24 smithi125 conmon[24927]: T 2021-03-24T22:01:24.565 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:24 smithi125 conmon[24927]: 22 2021-03-24T22:01:24.565 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:24 smithi125 conmon[24927]: : 2021-03-24T22:01:24.566 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:24 smithi125 conmon[24927]: 01 2021-03-24T22:01:24.566 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:24 smithi125 conmon[24927]: : 2021-03-24T22:01:24.567 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:24 smithi125 conmon[24927]: 22 2021-03-24T22:01:24.567 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:24 smithi125 conmon[24927]: . 2021-03-24T22:01:24.567 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:24 smithi125 conmon[24927]: 879762 2021-03-24T22:01:24.568 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:24 smithi125 conmon[24927]: +0000 2021-03-24T22:01:24.568 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:24 smithi125 conmon[24927]: 2021-03-24T22:01:24.569 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:24 smithi125 conmon[24927]: mgr.y 2021-03-24T22:01:24.569 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:24 smithi125 conmon[24927]: ( 2021-03-24T22:01:24.570 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:24 smithi125 conmon[24927]: mgr 2021-03-24T22:01:24.570 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:24 smithi125 conmon[24927]: . 2021-03-24T22:01:24.570 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:24 smithi125 conmon[24927]: 14144 2021-03-24T22:01:24.571 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:24 smithi125 conmon[24927]: ) 2021-03-24T22:01:24.571 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:24 smithi125 conmon[24927]: 528 2021-03-24T22:01:24.572 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:24 smithi125 conmon[24927]: : 2021-03-24T22:01:24.572 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:24 smithi125 conmon[24927]: cluster 2021-03-24T22:01:24.573 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:24 smithi125 conmon[24927]: 2021-03-24T22:01:24.573 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:24 smithi125 conmon[24927]: [DBG] 2021-03-24T22:01:24.573 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:24 smithi125 conmon[24927]: 2021-03-24T22:01:24.574 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:24 smithi125 conmon[24927]: pgmap v931: 168 pgs: 2 peering, 1 active+recovery_wait+remapped, 1 active+recovering+remapped, 5 active+remapped+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 1 active+remapped, 147 active+clean; 483 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 9.5 MiB/s rd, 3.1 MiB/s wr, 18 op/s; 78/1644 objects degraded (4.745%); 257/1644 objects misplaced (15.633%); 12 MiB/s, 49 objects/s recovering 2021-03-24T22:01:24.574 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:24 smithi125 conmon[24927]: 2021-03-24T22:01:24.575 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:24 smithi125 conmon[24927]: cluster 2021-03-24T22:01:24.575 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:24 smithi125 conmon[24927]: 2021-03-24T22:01:24.576 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:24 smithi125 conmon[24927]: 2021 2021-03-24T22:01:24.576 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:24 smithi125 conmon[24927]: - 2021-03-24T22:01:24.577 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:24 smithi125 conmon[24927]: 03 2021-03-24T22:01:24.577 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:24 smithi125 conmon[24927]: - 2021-03-24T22:01:24.577 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:24 smithi125 conmon[24927]: 24 2021-03-24T22:01:24.578 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:24 smithi125 conmon[24927]: T 2021-03-24T22:01:24.578 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:24 smithi125 conmon[24927]: 22 2021-03-24T22:01:24.579 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:24 smithi125 conmon[24927]: : 2021-03-24T22:01:24.579 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:24 smithi125 conmon[24927]: 01 2021-03-24T22:01:24.580 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:24 smithi125 conmon[24927]: : 2021-03-24T22:01:24.580 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:24 smithi125 conmon[24927]: 23. 2021-03-24T22:01:24.581 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:24 smithi125 conmon[24927]: 298105+0000 mon.a (mon.0) 1477 : cluster [DBG] osdmap e513: 8 total, 8 up, 7 in 2021-03-24T22:01:24.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:24 smithi142 conmon[20125]: cluster 2021-03-24T22:01:22.879762+0000 mgr.y (mgr. 2021-03-24T22:01:24.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:24 smithi142 conmon[20125]: 14144) 528 : cluster [DBG] pgmap v931: 168 pgs: 2 peering, 1 active+recovery_wait+remapped, 1 active+recovering+remapped, 5 active+remapped+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 1 active+remapped, 147 active+clean; 483 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 9.5 MiB/s rd, 3.1 MiB/s wr, 18 op/s; 78/1644 objects degraded (4.745%); 257/1644 objects misplaced (15.633%); 12 MiB/s, 49 objects/s recovering 2021-03-24T22:01:24.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:24 smithi142 conmon[20125]: cluster 2021-03-24T22:01:23.298105+0000 mon.a (mon.0) 1477 : cluster [DBG] osdmap e513: 8 total, 8 up, 7 in 2021-03-24T22:01:24.618 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:24.666 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:24.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:24.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:01:24.771 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:24.829 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:24.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:24.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:01:24.934 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:24.990 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:24.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:24.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:01:25.097 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:25.142 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:25.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:25.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:01:25.247 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:25.273 INFO:tasks.rados.rados.0.smithi125.stdout:1933: done (0 left) 2021-03-24T22:01:25.273 INFO:tasks.rados.rados.0.smithi125.stdout:1934: snap_remove snap 201 2021-03-24T22:01:25.308 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:25.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:25.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:01:25.397 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:25 smithi125 conmon[24927]: cluster 2021-03-24T22:01:24.301701+0000 2021-03-24T22:01:25.398 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:25 smithi125 conmon[24927]: mon.a (mon.0) 1478 : cluster [DBG] osdmap e514: 8 total, 8 up, 7 in 2021-03-24T22:01:25.398 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:25 smithi125 conmon[24927]: cluster 2021-03-24T22:01:25.259676+0000 mon.a (mon.0) 1479 2021-03-24T22:01:25.398 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:25 smithi125 conmon[24927]: : cluster [WRN] Health check update: Degraded data redundancy: 78/1644 objects degraded (4.745%), 5 pgs degraded (PG_DEGRADED) 2021-03-24T22:01:25.398 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:25 smithi125 conmon[24927]: cluster 2021-03-24T22:01:25.269571+0000 2021-03-24T22:01:25.399 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:25 smithi125 conmon[24927]: mon.a (mon.0) 1480 : cluster [DBG] osdmap e515: 8 total, 8 up, 7 in 2021-03-24T22:01:25.399 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:25 smithi125 conmon[19936]: cluster 2021-03-24T22:01 2021-03-24T22:01:25.400 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:25 smithi125 conmon[19936]: :24.301701+0000 mon.a (mon.0) 1478 : cluster [DBG] osdmap e514: 8 total, 8 up, 7 in 2021-03-24T22:01:25.400 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:25 smithi125 conmon[19936]: cluster 2021-03-24T22:01:25.400 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:25 smithi125 conmon[19936]: 2021-03-24T22: 2021-03-24T22:01:25.400 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:25 smithi125 conmon[19936]: 01:25.259676+0000 2021-03-24T22:01:25.400 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:25 smithi125 conmon[19936]: mon.a (mon.0) 2021-03-24T22:01:25.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:25 smithi125 conmon[19936]: 1479 : cluster [WRN] 2021-03-24T22:01:25.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:25 smithi125 conmon[19936]: Health check update: Degraded data redundancy: 78/1644 objects degraded (4.745%), 5 pgs degraded (PG_DEGRADED) 2021-03-24T22:01:25.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:25 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:01:25.401 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:25 smithi125 conmon[19936]: -03-24T22:01:25.269571+0000 mon.a 2021-03-24T22:01:25.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:25 smithi125 conmon[19936]: (mon.0) 1480 : cluster [DBG] 2021-03-24T22:01:25.402 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:25 smithi125 conmon[19936]: osdmap e515: 8 total, 8 up, 7 in 2021-03-24T22:01:25.413 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:25.455 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:25.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:25.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:01:25.560 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:25.574 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:25 smithi142 conmon[20125]: cluster 2021-03-24T22:01:24 2021-03-24T22:01:25.574 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:25 smithi142 conmon[20125]: .301701+0000 mon.a (mon.0) 1478 : cluster [DBG] osdmap e514: 8 total, 8 up, 7 in 2021-03-24T22:01:25.575 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:25 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:01:25.575 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:25 smithi142 conmon[20125]: -03-24T22:01:25.259676+0000 2021-03-24T22:01:25.575 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:25 smithi142 conmon[20125]: mon.a (mon.0) 1479 : cluster 2021-03-24T22:01:25.576 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:25 smithi142 conmon[20125]: [WRN] Health check update: Degraded data redundancy: 78/1644 objects degraded (4.745%), 5 pgs degraded (PG_DEGRADED) 2021-03-24T22:01:25.576 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:25 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:01:25.576 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:25 smithi142 conmon[20125]: -03-24T22:01 2021-03-24T22:01:25.577 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:25 smithi142 conmon[20125]: :25.269571+0000 mon.a (mon 2021-03-24T22:01:25.577 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:25 smithi142 conmon[20125]: .0) 1480 : cluster [DBG] 2021-03-24T22:01:25.577 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:25 smithi142 conmon[20125]: osdmap e515: 8 total, 8 up, 7 in 2021-03-24T22:01:25.622 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:25.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:25.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:01:25.632 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd unset noscrub 2021-03-24T22:01:25.728 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:25.782 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:25.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:25.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:01:26.009 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:26.064 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:26.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:26.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:01:26.171 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:26.218 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:26.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:26.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:01:26.287 INFO:tasks.rados.rados.0.smithi125.stdout:1934: done (0 left) 2021-03-24T22:01:26.287 INFO:tasks.rados.rados.0.smithi125.stdout:1935: read oid 19 snap -1 2021-03-24T22:01:26.288 INFO:tasks.rados.rados.0.smithi125.stdout:1935: expect deleted 2021-03-24T22:01:26.288 INFO:tasks.rados.rados.0.smithi125.stdout:1936: copy_from oid 42 from oid 28 current snap is 214 2021-03-24T22:01:26.289 INFO:tasks.rados.rados.0.smithi125.stdout:1937: write oid 30 current snap is 214 2021-03-24T22:01:26.289 INFO:tasks.rados.rados.0.smithi125.stdout:1937: seq_num 681 ranges {645567=799986,1950708=533234,3185058=260509} 2021-03-24T22:01:26.291 INFO:tasks.rados.rados.0.smithi125.stdout:1937: writing smithi12556470-30 from 645567 to 1445553 tid 1 2021-03-24T22:01:26.299 INFO:tasks.rados.rados.0.smithi125.stdout:1937: writing smithi12556470-30 from 1950708 to 2483942 tid 2 2021-03-24T22:01:26.306 INFO:tasks.rados.rados.0.smithi125.stdout:1937: writing smithi12556470-30 from 3185058 to 3445567 tid 3 2021-03-24T22:01:26.306 INFO:tasks.rados.rados.0.smithi125.stdout:1938: write (excl) oid 40 current snap is 214 2021-03-24T22:01:26.307 INFO:tasks.rados.rados.0.smithi125.stdout:1938: seq_num 682 ranges {662374=475235,1609190=676892,2812481=624101} 2021-03-24T22:01:26.312 INFO:tasks.rados.rados.0.smithi125.stdout:1938: writing smithi12556470-40 from 662374 to 1137609 tid 1 2021-03-24T22:01:26.348 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:26.350 INFO:tasks.rados.rados.0.smithi125.stdout:1938: writing smithi12556470-40 from 1609190 to 2286082 tid 2 2021-03-24T22:01:26.350 INFO:tasks.rados.rados.0.smithi125.stdout:1938: writing smithi12556470-40 from 2812481 to 3436582 tid 3 2021-03-24T22:01:26.351 INFO:tasks.rados.rados.0.smithi125.stdout:1939: copy_from oid 7 from oid 18 current snap is 214 2021-03-24T22:01:26.351 INFO:tasks.rados.rados.0.smithi125.stdout:1936: finishing copy_from to smithi12556470-42 2021-03-24T22:01:26.351 INFO:tasks.rados.rados.0.smithi125.stdout:1936: got expected ENOENT (src dne) 2021-03-24T22:01:26.351 INFO:tasks.rados.rados.0.smithi125.stdout:1936: finishing copy_from racing read to smithi12556470-42 2021-03-24T22:01:26.351 INFO:tasks.rados.rados.0.smithi125.stdout:1937: finishing write tid 1 to smithi12556470-30 2021-03-24T22:01:26.352 INFO:tasks.rados.rados.0.smithi125.stdout:1937: finishing write tid 2 to smithi12556470-30 2021-03-24T22:01:26.352 INFO:tasks.rados.rados.0.smithi125.stdout:1935: done (4 left) 2021-03-24T22:01:26.352 INFO:tasks.rados.rados.0.smithi125.stdout:1936: done (3 left) 2021-03-24T22:01:26.352 INFO:tasks.rados.rados.0.smithi125.stdout:1940: write (excl) oid 45 current snap is 214 2021-03-24T22:01:26.353 INFO:tasks.rados.rados.0.smithi125.stdout:1937: finishing write tid 3 to smithi12556470-30 2021-03-24T22:01:26.353 INFO:tasks.rados.rados.0.smithi125.stdout:1940: seq_num 683 ranges {679181=550484,1667672=536903,2839904=435373} 2021-03-24T22:01:26.353 INFO:tasks.rados.rados.0.smithi125.stdout:1940: writing smithi12556470-45 from 679181 to 1229665 tid 1 2021-03-24T22:01:26.353 INFO:tasks.rados.rados.0.smithi125.stdout:1940: writing smithi12556470-45 from 1667672 to 2204575 tid 2 2021-03-24T22:01:26.353 INFO:tasks.rados.rados.0.smithi125.stdout:1940: writing smithi12556470-45 from 2839904 to 3275277 tid 3 2021-03-24T22:01:26.354 INFO:tasks.rados.rados.0.smithi125.stdout:1941: delete oid 31 current snap is 214 2021-03-24T22:01:26.354 INFO:tasks.rados.rados.0.smithi125.stdout:1937: finishing write tid 4 to smithi12556470-30 2021-03-24T22:01:26.354 INFO:tasks.rados.rados.0.smithi125.stdout:1937: finishing write tid 5 to smithi12556470-30 2021-03-24T22:01:26.354 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 30 v 299 (ObjNum 681 snap 214 seq_num 681) dirty exists 2021-03-24T22:01:26.355 INFO:tasks.rados.rados.0.smithi125.stdout:1937: left oid 30 (ObjNum 681 snap 214 seq_num 681) 2021-03-24T22:01:26.355 INFO:tasks.rados.rados.0.smithi125.stdout:1938: finishing write tid 1 to smithi12556470-40 2021-03-24T22:01:26.355 INFO:tasks.rados.rados.0.smithi125.stdout:1938: finishing write tid 2 to smithi12556470-40 2021-03-24T22:01:26.355 INFO:tasks.rados.rados.0.smithi125.stdout:1938: finishing write tid 3 to smithi12556470-40 2021-03-24T22:01:26.355 INFO:tasks.rados.rados.0.smithi125.stdout:1938: finishing write tid 4 to smithi12556470-40 2021-03-24T22:01:26.356 INFO:tasks.rados.rados.0.smithi125.stdout:1938: finishing write tid 5 to smithi12556470-40 2021-03-24T22:01:26.356 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 40 v 946 (ObjNum 682 snap 214 seq_num 682) dirty exists 2021-03-24T22:01:26.356 INFO:tasks.rados.rados.0.smithi125.stdout:1938: left oid 40 (ObjNum 682 snap 214 seq_num 682) 2021-03-24T22:01:26.356 INFO:tasks.rados.rados.0.smithi125.stdout:1940: finishing write tid 1 to smithi12556470-45 2021-03-24T22:01:26.356 INFO:tasks.rados.rados.0.smithi125.stdout:1940: finishing write tid 2 to smithi12556470-45 2021-03-24T22:01:26.358 INFO:tasks.rados.rados.0.smithi125.stdout:1940: finishing write tid 3 to smithi12556470-45 2021-03-24T22:01:26.358 INFO:tasks.rados.rados.0.smithi125.stdout:1940: finishing write tid 4 to smithi12556470-45 2021-03-24T22:01:26.358 INFO:tasks.rados.rados.0.smithi125.stdout:1940: finishing write tid 5 to smithi12556470-45 2021-03-24T22:01:26.358 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 45 v 649 (ObjNum 683 snap 214 seq_num 683) dirty exists 2021-03-24T22:01:26.359 INFO:tasks.rados.rados.0.smithi125.stdout:1940: left oid 45 (ObjNum 683 snap 214 seq_num 683) 2021-03-24T22:01:26.359 INFO:tasks.rados.rados.0.smithi125.stdout:1937: done (4 left) 2021-03-24T22:01:26.359 INFO:tasks.rados.rados.0.smithi125.stdout:1938: done (3 left) 2021-03-24T22:01:26.359 INFO:tasks.rados.rados.0.smithi125.stdout:1940: done (2 left) 2021-03-24T22:01:26.360 INFO:tasks.rados.rados.0.smithi125.stdout:1941: done (1 left) 2021-03-24T22:01:26.360 INFO:tasks.rados.rados.0.smithi125.stdout:1942: snap_remove snap 191 2021-03-24T22:01:26.401 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:26.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:26.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:01:26.450 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:26 smithi125 conmon[19936]: cluster 2021-03-24T22:01:24.880642+0000 2021-03-24T22:01:26.450 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:26 smithi125 conmon[19936]: mgr.y (mgr.14144) 529 : cluster [DBG] pgmap v934: 168 pgs: 2 active+remapped+backfilling, 1 peering, 1 active+recovery_wait+remapped, 6 active+remapped+backfill_toofull, 2 active+recovering+undersized+remapped, 2 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+remapped, 151 active+clean; 468 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 19 MiB/s rd, 1.2 MiB/s wr, 19 op/s; 9/1629 objects degraded (0.552%); 232/1629 objects misplaced (14.242%); 20 MiB/s, 33 objects/s recovering 2021-03-24T22:01:26.451 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:26 smithi125 conmon[19936]: cluster 2021-03-24T22:01:26.274205+0000 mon.a (mon.0) 1481 : cluster [DBG] osdmap e516: 8 total, 8 up, 7 in 2021-03-24T22:01:26.451 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:26 smithi125 conmon[19936]: 2021-03-24T22:01:26.451 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:26 smithi125 conmon[24927]: cluster 2021-03-24T22:01:24.880642+0000 mgr.y (mgr.14144) 529 2021-03-24T22:01:26.452 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:26 smithi125 conmon[24927]: : cluster [DBG] pgmap v934: 168 pgs: 2 active+remapped+backfilling, 1 peering, 1 active+recovery_wait+remapped, 6 active+remapped+backfill_toofull, 2 active+recovering+undersized+remapped, 2 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+remapped, 151 active+clean; 468 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 19 MiB/s rd, 1.2 MiB/s wr, 19 op/s; 9/1629 objects degraded (0.552%); 232/1629 objects misplaced (14.242%); 20 MiB/s, 33 objects/s recovering 2021-03-24T22:01:26.452 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:26 smithi125 conmon[24927]: cluster 2021-03-24T22:01:26.274205+0000 mon.a (mon.0) 1481 : cluster [DBG] osdmap e516: 8 total, 8 up, 7 in 2021-03-24T22:01:26.507 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:26.565 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:26.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:26.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:01:26.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:26 smithi142 conmon[20125]: cluster 2021-03-24T22:01:24 2021-03-24T22:01:26.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:26 smithi142 conmon[20125]: .880642+0000 mgr.y (mgr.14144) 529 : cluster [DBG] pgmap v934: 168 pgs: 2 active+remapped+backfilling, 1 peering, 1 active+recovery_wait+remapped, 6 active+remapped+backfill_toofull, 2 active+recovering+undersized+remapped, 2 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+remapped, 151 active+clean; 468 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 19 MiB/s rd, 1.2 MiB/s wr, 19 op/s; 9/1629 objects degraded (0.552%); 232/1629 objects misplaced (14.242%); 20 MiB/s, 33 objects/s recovering 2021-03-24T22:01:26.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:26 smithi142 conmon[20125]: cluster 2021-03-24T22:01:26.274205+0000 2021-03-24T22:01:26.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:26 smithi142 conmon[20125]: mon.a (mon.0) 1481 : cluster [DBG] osdmap e516: 8 total, 8 up, 7 in 2021-03-24T22:01:26.676 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:26.733 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:26.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:26.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:01:26.923 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:26.987 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:27.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:27.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:01:27.209 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:27.268 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:27.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:27.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:01:27.283 INFO:tasks.rados.rados.0.smithi125.stdout:1942: done (1 left) 2021-03-24T22:01:27.284 INFO:tasks.rados.rados.0.smithi125.stdout:1943: snap_create 2021-03-24T22:01:27.285 INFO:tasks.rados.rados.0.smithi125.stdout:1939: finishing copy_from to smithi12556470-7 2021-03-24T22:01:27.286 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 7 v 239 (ObjNum 588 snap 177 seq_num 588) dirty exists 2021-03-24T22:01:27.286 INFO:tasks.rados.rados.0.smithi125.stdout:1939: finishing copy_from racing read to smithi12556470-7 2021-03-24T22:01:27.377 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:27.441 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:27.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:27.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:01:27.552 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:27.582 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:01:27.613 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:27.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:27.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:01:27.722 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:27.782 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:27.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:27.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:01:27.893 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:27.961 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:27.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:27.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:01:27.992 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:27 smithi125 conmon[19936]: debug 2021-03-24T22:01:27.886+0000 7f2118128700 -1 mon.a@0(leader).osd e517 definitely_dead 0 2021-03-24T22:01:28.067 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:28.132 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:28.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:28.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:01:28.142 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-24T22:01:28.142 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-24T22:01:28.143 INFO:teuthology.orchestra.run.smithi125.stdout: "num_ops": 0 2021-03-24T22:01:28.143 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-24T22:01:28.237 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:28.284 INFO:tasks.rados.rados.0.smithi125.stdout:1939: done (1 left) 2021-03-24T22:01:28.284 INFO:tasks.rados.rados.0.smithi125.stdout:1943: done (0 left) 2021-03-24T22:01:28.285 INFO:tasks.rados.rados.0.smithi125.stdout:1944: delete oid 12 current snap is 215 2021-03-24T22:01:28.302 INFO:teuthology.orchestra.run.smithi125.stderr:marked out osd.7. 2021-03-24T22:01:28.310 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:28.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:28.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:01:28.337 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T22:01:28.357 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T22:01:28.377 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T22:01:28.398 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T22:01:28.419 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:28.426 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T22:01:28.459 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T22:01:28.474 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:28.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:28.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:01:28.488 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T22:01:28.511 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:28 smithi125 conmon[19936]: cluster 2021-03-24T22:01:26.881702+0000 mgr.y ( 2021-03-24T22:01:28.512 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:28 smithi125 conmon[19936]: mgr.14144) 530 : cluster [DBG] pgmap v937: 168 pgs: 2 active+clean+wait, 2 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 2 active+undersized+remapped, 1 active+remapped, 154 active+clean; 468 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 1.5 MiB/s wr, 16 op/s; 191/1620 objects misplaced (11.790%); 25 MiB/s, 46 objects/s recovering 2021-03-24T22:01:28.512 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:28 smithi125 conmon[19936]: cluster 2021-03-24T22:01:28.513 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:28 smithi125 conmon[19936]: 2021-03-24T22 2021-03-24T22:01:28.513 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:28 smithi125 conmon[19936]: :01:27 2021-03-24T22:01:28.514 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:28 smithi125 conmon[19936]: .275890+0000 mon.a (mon.0) 1482 2021-03-24T22:01:28.514 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:28 smithi125 conmon[19936]: : cluster 2021-03-24T22:01:28.514 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:28 smithi125 conmon[19936]: [DBG] osdmap e517: 8 total, 8 up, 7 in 2021-03-24T22:01:28.515 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:28 smithi125 conmon[19936]: cluster 2021-03-24T22:01:27.330878+0000 mon.a (mon.0) 1483 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 9/1629 objects degraded (0.552%), 1 pg degraded) 2021-03-24T22:01:28.515 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:28 smithi125 conmon[19936]: audit 2021-03-24T22:01:27.891272+0000 mon.a (mon.0) 1484 : audit [INF] from='client.? v1:172.21.15.125:0/1923751282' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2021-03-24T22:01:28.516 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:28 smithi125 conmon[19936]: cluster 2021-03-24T22:01:27.891404+0000 mon.a (mon.0) 1485 : cluster [INF] Client client.admin marked osd.7 out, while it was still marked up 2021-03-24T22:01:28.516 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:28 smithi125 conmon[19936]: audit 2021-03-24T22:01:28.266894+0000 mon.a (mon.0) 1486 : audit [DBG] from='client.? v1:172.21.15.125:0/1226451946' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:01:28.517 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:28 smithi125 conmon[24927]: cluster 2021-03-24T22:01:26.881702+0000 2021-03-24T22:01:28.517 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:28 smithi125 conmon[24927]: mgr.y (mgr.14144) 530 : cluster [DBG] pgmap v937: 168 pgs: 2 active+clean+wait, 2 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 2 active+undersized+remapped, 1 active+remapped, 154 active+clean; 468 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 1.5 MiB/s wr, 16 op/s; 191/1620 objects misplaced (11.790%); 25 MiB/s, 46 objects/s recovering 2021-03-24T22:01:28.518 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:28 smithi125 conmon[24927]: cluster 2021-03-24T22:01:27.275890+0000 mon.a (mon.0 2021-03-24T22:01:28.518 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:28 smithi125 conmon[24927]: ) 1482 : cluster [DBG] osdmap e517: 8 total, 8 up, 7 in 2021-03-24T22:01:28.519 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:28 smithi125 conmon[24927]: cluster 2021-03-24T22:01:28.519 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:28 smithi125 conmon[24927]: 2021-03-24T22:01:27.330878+0000 2021-03-24T22:01:28.520 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:28 smithi125 conmon[24927]: mon.a (mon.0) 1483 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 9/1629 objects degraded (0.552%), 1 pg degraded) 2021-03-24T22:01:28.520 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:28 smithi125 conmon[24927]: audit 2021-03-24T22:01:27.891272+0000 mon.a (mon.0) 1484 : audit [INF] from='client.? v1:172.21.15.125:0/1923751282' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2021-03-24T22:01:28.520 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:28 smithi125 conmon[24927]: cluster 2021-03-24T22:01:27.891404+0000 mon.a (mon.0) 1485 : cluster [INF] Client client.admin marked osd.7 out, while it was still marked up 2021-03-24T22:01:28.520 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:28 smithi125 conmon[24927]: audit 2021-03-24T22:01:28.266894+0000 mon.a (mon.0) 1486 : audit [DBG] from='client.? v1:172.21.15.125:0/1226451946' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:01:28.521 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T22:01:28.521 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-24T22:01:28.522 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-24T22:01:28.522 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-24T22:01:28.522 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-24T22:01:28.522 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-24T22:01:28.523 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-24T22:01:28.523 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-24T22:01:28.523 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-24T22:01:28.580 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:28.595 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: cluster 2021-03-24T22:01:28.596 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: 2021-03-24T22:01:28.596 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: 2021 2021-03-24T22:01:28.597 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: - 2021-03-24T22:01:28.597 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: 03 2021-03-24T22:01:28.598 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: - 2021-03-24T22:01:28.598 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: 24 2021-03-24T22:01:28.599 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: T 2021-03-24T22:01:28.599 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: 22 2021-03-24T22:01:28.599 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: : 2021-03-24T22:01:28.600 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: 01 2021-03-24T22:01:28.600 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: : 2021-03-24T22:01:28.601 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: 26 2021-03-24T22:01:28.601 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: . 2021-03-24T22:01:28.602 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: 881702 2021-03-24T22:01:28.602 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: +0000 2021-03-24T22:01:28.603 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: 2021-03-24T22:01:28.603 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: mgr.y 2021-03-24T22:01:28.603 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: ( 2021-03-24T22:01:28.603 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: mgr 2021-03-24T22:01:28.604 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: . 2021-03-24T22:01:28.604 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: 14144 2021-03-24T22:01:28.604 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: ) 2021-03-24T22:01:28.605 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: 530 2021-03-24T22:01:28.605 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: : 2021-03-24T22:01:28.606 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: cluster 2021-03-24T22:01:28.606 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: 2021-03-24T22:01:28.606 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: [DBG] 2021-03-24T22:01:28.607 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: 2021-03-24T22:01:28.607 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: pgmap v937: 168 pgs: 2 active+clean+wait, 2 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 2 active+undersized+remapped, 1 active+remapped, 154 active+clean; 468 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 1.5 MiB/s wr, 16 op/s; 191/1620 objects misplaced (11.790%); 25 MiB/s, 46 objects/s recovering 2021-03-24T22:01:28.607 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: 2021-03-24T22:01:28.608 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: cluster 2021-03-24T22:01:28.608 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: 2021-03-24T22:01:28.608 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: 2021 2021-03-24T22:01:28.609 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: - 2021-03-24T22:01:28.609 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: 03 2021-03-24T22:01:28.609 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: - 2021-03-24T22:01:28.609 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: 24 2021-03-24T22:01:28.610 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: T 2021-03-24T22:01:28.610 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: 22 2021-03-24T22:01:28.610 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: : 2021-03-24T22:01:28.611 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: 01 2021-03-24T22:01:28.611 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: : 2021-03-24T22:01:28.611 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: 27 2021-03-24T22:01:28.611 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: . 2021-03-24T22:01:28.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: 275890 2021-03-24T22:01:28.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: +0000 2021-03-24T22:01:28.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: 2021-03-24T22:01:28.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: mon.a 2021-03-24T22:01:28.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: ( 2021-03-24T22:01:28.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: mon.0) 1482 : cluster [DBG] osdmap e517: 8 total, 8 up, 7 in 2021-03-24T22:01:28.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: cluster 2021-03-24T22:01:27.330878+0000 mon.a (mon.0) 1483 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 9/1629 objects degraded (0.552%), 1 pg degraded) 2021-03-24T22:01:28.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: audit 2021-03-24T22:01:27.891272+0000 mon.a (mon.0) 1484 : audit [INF] from='client.? v1:172.21.15.125:0/1923751282' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2021-03-24T22:01:28.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: cluster 2021-03-24T22:01:27.891404+0000 mon.a (mon.0) 1485 : cluster [INF] Client client.admin marked osd.7 out, while it was still marked up 2021-03-24T22:01:28.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: audit 2021-03-24T22:01:28.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:28 smithi142 conmon[20125]: 2021-03-24T22:01:28.266894+0000 mon.a (mon.0) 1486 : audit [DBG] from='client.? v1:172.21.15.125:0/1226451946' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:01:28.629 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:28.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:28.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:01:28.736 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:28.793 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:28.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:28.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:01:28.900 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:28.957 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:28.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:28.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:01:29.062 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:29.119 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:29.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:29.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:01:29.229 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:29.298 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:29.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:29.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:01:29.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:29 smithi142 conmon[20125]: audit 2021-03-24T22:01:28.276314+0000 mon.a (mon.0) 2021-03-24T22:01:29.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:29 smithi142 conmon[20125]: 1487 : audit [INF] from='client.? v1:172.21.15.125:0/1923751282' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2021-03-24T22:01:29.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:29 smithi142 conmon[20125]: cluster 2021-03-24T22:01:28.276439+0000 mon.a (mon.0) 1488 : cluster [DBG] osdmap e518: 8 total, 8 up, 6 in 2021-03-24T22:01:29.403 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:29.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:29 smithi125 conmon[19936]: audit 2021-03-24T22:01:28.276314+0000 mon.a (mon. 2021-03-24T22:01:29.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:29 smithi125 conmon[19936]: 0) 1487 : audit [INF] from='client.? v1:172.21.15.125:0/1923751282' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2021-03-24T22:01:29.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:29 smithi125 conmon[19936]: cluster 2021-03-24T22:01:28.276439+0000 mon.a (mon.0) 1488 : cluster [DBG] osdmap e518: 8 total, 8 up, 6 in 2021-03-24T22:01:29.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:29 smithi125 conmon[24927]: audit 2021-03-24T22:01:29.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:29 smithi125 conmon[24927]: 2021-03-24T22:01:29.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:29 smithi125 conmon[24927]: 2021 2021-03-24T22:01:29.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:29 smithi125 conmon[24927]: - 2021-03-24T22:01:29.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:29 smithi125 conmon[24927]: 03 2021-03-24T22:01:29.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:29 smithi125 conmon[24927]: - 2021-03-24T22:01:29.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:29 smithi125 conmon[24927]: 24 2021-03-24T22:01:29.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:29 smithi125 conmon[24927]: T 2021-03-24T22:01:29.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:29 smithi125 conmon[24927]: 22 2021-03-24T22:01:29.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:29 smithi125 conmon[24927]: : 2021-03-24T22:01:29.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:29 smithi125 conmon[24927]: 01 2021-03-24T22:01:29.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:29 smithi125 conmon[24927]: : 2021-03-24T22:01:29.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:29 smithi125 conmon[24927]: 28 2021-03-24T22:01:29.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:29 smithi125 conmon[24927]: . 2021-03-24T22:01:29.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:29 smithi125 conmon[24927]: 276314 2021-03-24T22:01:29.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:29 smithi125 conmon[24927]: +0000 2021-03-24T22:01:29.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:29 smithi125 conmon[24927]: 2021-03-24T22:01:29.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:29 smithi125 conmon[24927]: mon.a 2021-03-24T22:01:29.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:29 smithi125 conmon[24927]: ( 2021-03-24T22:01:29.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:29 smithi125 conmon[24927]: mon 2021-03-24T22:01:29.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:29 smithi125 conmon[24927]: . 2021-03-24T22:01:29.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:29 smithi125 conmon[24927]: 0 2021-03-24T22:01:29.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:29 smithi125 conmon[24927]: ) 2021-03-24T22:01:29.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:29 smithi125 conmon[24927]: 1487 2021-03-24T22:01:29.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:29 smithi125 conmon[24927]: : 2021-03-24T22:01:29.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:29 smithi125 conmon[24927]: audit [INF] from='client.? v1:172.21.15.125:0/1923751282' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2021-03-24T22:01:29.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:29 smithi125 conmon[24927]: cluster 2021-03-24T22:01:28.276439 2021-03-24T22:01:29.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:29 smithi125 conmon[24927]: +0000 mon.a (mon.0) 1488 : cluster [DBG] osdmap e518: 8 total, 8 up, 6 in 2021-03-24T22:01:29.457 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:29.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:29.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:01:29.563 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:29.621 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:29.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:29.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:01:29.729 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:29.776 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:29.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:29.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:01:29.881 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:29.927 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:29.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:29.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:01:30.036 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:30.080 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:30.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:30.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:01:30.184 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:30.240 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:30.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:30.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:01:30.312 INFO:tasks.rados.rados.0.smithi125.stdout:1944: done (0 left) 2021-03-24T22:01:30.312 INFO:tasks.rados.rados.0.smithi125.stdout:1945: read oid 5 snap -1 2021-03-24T22:01:30.313 INFO:tasks.rados.rados.0.smithi125.stdout:1945: expect (ObjNum 652 snap 198 seq_num 652) 2021-03-24T22:01:30.313 INFO:tasks.rados.rados.0.smithi125.stdout:1946: delete oid 27 current snap is 215 2021-03-24T22:01:30.348 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:30.375 INFO:tasks.rados.rados.0.smithi125.stdout:1946: done (1 left) 2021-03-24T22:01:30.375 INFO:tasks.rados.rados.0.smithi125.stdout:1947: copy_from oid 31 from oid 14 current snap is 215 2021-03-24T22:01:30.376 INFO:tasks.rados.rados.0.smithi125.stdout:1948: write oid 25 current snap is 215 2021-03-24T22:01:30.376 INFO:tasks.rados.rados.0.smithi125.stdout:1948: seq_num 687 ranges {746409=567833,1901600=544241,3065949=480460} 2021-03-24T22:01:30.382 INFO:tasks.rados.rados.0.smithi125.stdout:1948: writing smithi12556470-25 from 746409 to 1314242 tid 1 2021-03-24T22:01:30.391 INFO:tasks.rados.rados.0.smithi125.stdout:1948: writing smithi12556470-25 from 1901600 to 2445841 tid 2 2021-03-24T22:01:30.403 INFO:tasks.rados.rados.0.smithi125.stdout:1948: writing smithi12556470-25 from 3065949 to 3546409 tid 3 2021-03-24T22:01:30.410 INFO:tasks.rados.rados.0.smithi125.stdout:1947: finishing copy_from to smithi12556470-31 2021-03-24T22:01:30.411 INFO:tasks.rados.rados.0.smithi125.stdout:1947: got expected ENOENT (src dne) 2021-03-24T22:01:30.411 INFO:tasks.rados.rados.0.smithi125.stdout:1947: finishing copy_from racing read to smithi12556470-31 2021-03-24T22:01:30.412 INFO:tasks.rados.rados.0.smithi125.stdout:1945: expect (ObjNum 652 snap 198 seq_num 652) 2021-03-24T22:01:30.416 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:30.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:30.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:01:30.524 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:30.538 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:30 smithi142 conmon[20125]: cluster 2021-03-24T22:01 2021-03-24T22:01:30.539 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:30 smithi142 conmon[20125]: :28.882509+0000 mgr.y (mgr.14144) 531 : cluster [DBG] pgmap v940: 168 pgs: 2 active+clean+wait, 2 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 2 active+undersized+remapped, 1 active+remapped, 154 active+clean; 468 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 255 B/s rd, 394 KiB/s wr, 1 op/s; 191/1620 objects misplaced (11.790%); 6.0 MiB/s, 15 objects/s recovering 2021-03-24T22:01:30.539 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:30 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:01:30.540 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:30 smithi142 conmon[20125]: -24T22:01:29.284262+0000 osd.5 (osd.5) 18 : cluster 2021-03-24T22:01:30.540 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:30 smithi142 conmon[20125]: [INF] 4.a continuing backfill to osd.1 from (69'23,69'23] MIN to 69'23 2021-03-24T22:01:30.540 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:30 smithi142 conmon[20125]: cluster 2021-03-24T22 2021-03-24T22:01:30.541 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:30 smithi142 conmon[20125]: :01:29.289602+0000 mon.a (mon 2021-03-24T22:01:30.541 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:30 smithi142 conmon[20125]: .0) 1489 : cluster [DBG] osdmap e519: 8 total, 8 up, 6 in 2021-03-24T22:01:30.542 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:30 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:01:30.542 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:30 smithi142 conmon[20125]: -03-24T22:01:29.290345 2021-03-24T22:01:30.542 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:30 smithi142 conmon[20125]: +0000 osd.0 (osd.0) 110 : cluster 2021-03-24T22:01:30.543 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:30 smithi142 conmon[20125]: [INF] 4.6 continuing backfill to osd.1 from (69'16,69'16] MIN to 69'16 2021-03-24T22:01:30.543 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:30 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:01:30.544 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:30 smithi142 conmon[20125]: T22:01:30.261634+0000 mon.a 2021-03-24T22:01:30.544 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:30 smithi142 conmon[20125]: (mon.0) 1490 : cluster [WRN] 2021-03-24T22:01:30.544 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:30 smithi142 conmon[20125]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:01:30.545 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:30 smithi142 conmon[20125]: cluster 2021-03-24T22:01:30.545 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:30 smithi142 conmon[20125]: 2021-03-24T22:01 2021-03-24T22:01:30.546 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:30 smithi142 conmon[20125]: :30.267579+0000 mon.a (mon.0 2021-03-24T22:01:30.546 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:30 smithi142 conmon[20125]: ) 1491 : cluster [DBG] osdmap e520: 8 total, 8 up, 6 in 2021-03-24T22:01:30.580 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:30.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:30.587 INFO:tasks.rados.rados.0.smithi125.stdout:1948: finishing write tid 1 to smithi12556470-25 2021-03-24T22:01:30.588 INFO:tasks.rados.rados.0.smithi125.stdout:1948: finishing write tid 2 to smithi12556470-25 2021-03-24T22:01:30.588 INFO:tasks.rados.rados.0.smithi125.stdout:1948: finishing write tid 3 to smithi12556470-25 2021-03-24T22:01:30.588 INFO:tasks.rados.rados.0.smithi125.stdout:1948: finishing write tid 4 to smithi12556470-25 2021-03-24T22:01:30.588 INFO:tasks.rados.rados.0.smithi125.stdout:1948: finishing write tid 5 to smithi12556470-25 2021-03-24T22:01:30.588 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 25 v 508 (ObjNum 687 snap 215 seq_num 687) dirty exists 2021-03-24T22:01:30.589 INFO:tasks.rados.rados.0.smithi125.stdout:1948: left oid 25 (ObjNum 687 snap 215 seq_num 687) 2021-03-24T22:01:30.589 INFO:tasks.rados.rados.0.smithi125.stdout:1945: done (2 left) 2021-03-24T22:01:30.589 INFO:tasks.rados.rados.0.smithi125.stdout:1947: done (1 left) 2021-03-24T22:01:30.589 INFO:tasks.rados.rados.0.smithi125.stdout:1948: done (0 left) 2021-03-24T22:01:30.590 INFO:tasks.rados.rados.0.smithi125.stdout:1949: write (excl) oid 39 current snap is 215 2021-03-24T22:01:30.590 INFO:tasks.rados.rados.0.smithi125.stdout:1949: seq_num 688 ranges {763216=643082,1843729=687899,3260666=302550} 2021-03-24T22:01:30.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:01:30.593 INFO:tasks.rados.rados.0.smithi125.stdout:1949: writing smithi12556470-39 from 763216 to 1406298 tid 1 2021-03-24T22:01:30.601 INFO:tasks.rados.rados.0.smithi125.stdout:1949: writing smithi12556470-39 from 1843729 to 2531628 tid 2 2021-03-24T22:01:30.608 INFO:tasks.rados.rados.0.smithi125.stdout:1949: writing smithi12556470-39 from 3260666 to 3563216 tid 3 2021-03-24T22:01:30.608 INFO:tasks.rados.rados.0.smithi125.stdout:1949: finishing write tid 1 to smithi12556470-39 2021-03-24T22:01:30.608 INFO:tasks.rados.rados.0.smithi125.stdout:1949: finishing write tid 2 to smithi12556470-39 2021-03-24T22:01:30.608 INFO:tasks.rados.rados.0.smithi125.stdout:1950: write (excl) oid 40 current snap is 215 2021-03-24T22:01:30.609 INFO:tasks.rados.rados.0.smithi125.stdout:1950: seq_num 689 ranges {780023=718331,1902211=547910,3004442=575581} 2021-03-24T22:01:30.617 INFO:tasks.rados.rados.0.smithi125.stdout:1950: writing smithi12556470-40 from 780023 to 1498354 tid 1 2021-03-24T22:01:30.624 INFO:tasks.rados.rados.0.smithi125.stdout:1950: writing smithi12556470-40 from 1902211 to 2450121 tid 2 2021-03-24T22:01:30.633 INFO:tasks.rados.rados.0.smithi125.stdout:1950: writing smithi12556470-40 from 3004442 to 3580023 tid 3 2021-03-24T22:01:30.633 INFO:tasks.rados.rados.0.smithi125.stdout:1951: snap_create 2021-03-24T22:01:30.634 INFO:tasks.rados.rados.0.smithi125.stdout:1949: finishing write tid 3 to smithi12556470-39 2021-03-24T22:01:30.634 INFO:tasks.rados.rados.0.smithi125.stdout:1949: finishing write tid 4 to smithi12556470-39 2021-03-24T22:01:30.634 INFO:tasks.rados.rados.0.smithi125.stdout:1949: finishing write tid 5 to smithi12556470-39 2021-03-24T22:01:30.635 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 39 v 677 (ObjNum 688 snap 215 seq_num 688) dirty exists 2021-03-24T22:01:30.635 INFO:tasks.rados.rados.0.smithi125.stdout:1949: left oid 39 (ObjNum 688 snap 215 seq_num 688) 2021-03-24T22:01:30.635 INFO:tasks.rados.rados.0.smithi125.stdout:1950: finishing write tid 1 to smithi12556470-40 2021-03-24T22:01:30.636 INFO:tasks.rados.rados.0.smithi125.stdout:1950: finishing write tid 2 to smithi12556470-40 2021-03-24T22:01:30.640 INFO:tasks.rados.rados.0.smithi125.stdout:1950: finishing write tid 3 to smithi12556470-40 2021-03-24T22:01:30.641 INFO:tasks.rados.rados.0.smithi125.stdout:1950: finishing write tid 4 to smithi12556470-40 2021-03-24T22:01:30.641 INFO:tasks.rados.rados.0.smithi125.stdout:1950: finishing write tid 5 to smithi12556470-40 2021-03-24T22:01:30.642 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 40 v 961 (ObjNum 689 snap 215 seq_num 689) dirty exists 2021-03-24T22:01:30.642 INFO:tasks.rados.rados.0.smithi125.stdout:1950: left oid 40 (ObjNum 689 snap 215 seq_num 689) 2021-03-24T22:01:30.643 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[19936]: cluster 2021-03-24T22:01:30.644 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[19936]: 2021-03-24T22:01:30.644 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[19936]: 2021-03-24T22 2021-03-24T22:01:30.644 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[19936]: :01:28.882509+0000 mgr.y (mgr.14144) 531 : cluster [DBG] pgmap v940: 168 pgs: 2 active+clean+wait, 2 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 2 active+undersized+remapped, 1 active+remapped, 154 active+clean; 468 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 255 B/s rd, 394 KiB/s wr, 1 op/s; 191/1620 objects misplaced (11.790%); 6.0 MiB/s, 15 objects/s recovering 2021-03-24T22:01:30.644 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[19936]: cluster 2021-03-24T22:01:30.645 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[19936]: 2021-03 2021-03-24T22:01:30.645 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[19936]: -24T22:01:29.284262+0000 osd.5 (osd.5) 18 2021-03-24T22:01:30.645 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[19936]: : cluster [INF] 4.a continuing backfill to osd.1 from (69'23,69'23] MIN to 69'23 2021-03-24T22:01:30.645 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:01:30.645 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[19936]: -03-24T22:01:29.289602+0000 mon.a 2021-03-24T22:01:30.646 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[19936]: (mon.0) 1489 : cluster [DBG] osdmap e519: 8 total, 8 up, 6 in 2021-03-24T22:01:30.646 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:01:30.646 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[19936]: 03-24T22:01:29.290345+0000 osd.0 2021-03-24T22:01:30.646 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[19936]: (osd.0) 110 : cluster [INF] 4.6 continuing backfill to osd.1 from (69'16,69'16] MIN to 69'16 2021-03-24T22:01:30.647 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:01:30.647 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[19936]: -03-24T22:01:30.261634+0000 2021-03-24T22:01:30.647 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[19936]: mon.a (mon.0) 1490 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:01:30.647 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[19936]: 2021-03-24T22:01:30.648 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[19936]: cluster 2021-03-24T22:01:30.648 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[19936]: 2021-03-24T22:01:30.267579+0000 mon.a (mon.0 2021-03-24T22:01:30.648 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[19936]: ) 1491 : cluster [DBG] osdmap e520: 8 total, 8 up, 6 in 2021-03-24T22:01:30.649 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T22:01:30.650 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: T22:01:28.882509+0000 mgr.y (mgr.14144) 531 2021-03-24T22:01:30.650 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: : cluster [DBG] pgmap v940: 168 pgs: 2 active+clean+wait, 2 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 2 active+undersized+remapped, 1 active+remapped, 154 active+clean; 468 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 255 B/s rd, 394 KiB/s wr, 1 op/s; 191/1620 objects misplaced (11.790%); 6.0 MiB/s, 15 objects/s recovering 2021-03-24T22:01:30.650 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: cluster 2021-03-24T22:01:30.650 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: 2021-03-24T22: 2021-03-24T22:01:30.651 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: 01:29.284262+0000 2021-03-24T22:01:30.651 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: osd.5 (osd.5) 18 2021-03-24T22:01:30.651 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: : cluster [INF] 4.a continuing backfill to osd.1 from (69'23,69'23] MIN to 69'23 2021-03-24T22:01:30.651 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:01:30.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: -03-24T22:01:29 2021-03-24T22:01:30.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: .289602+0000 mon.a (mon 2021-03-24T22:01:30.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: .0) 1489 : cluster 2021-03-24T22:01:30.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: [DBG] osdmap e519: 8 total, 8 up, 6 in 2021-03-24T22:01:30.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:01:30.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: -03-24T22:01 2021-03-24T22:01:30.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: :29.290345+0000 osd.0 2021-03-24T22:01:30.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: (osd.0) 110 : 2021-03-24T22:01:30.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: cluster [INF] 4.6 continuing backfill to osd.1 from (69'16,69'16] MIN to 69'16 2021-03-24T22:01:30.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: cluster 2021-03-24T22:01:30.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: 2021-03-24T22: 2021-03-24T22:01:30.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: 01:30.261634+0000 2021-03-24T22:01:30.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: mon.a (mon.0) 1490 2021-03-24T22:01:30.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:01:30.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: cluster 2021-03-24T22:01:30.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: 2021 2021-03-24T22:01:30.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: - 2021-03-24T22:01:30.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: 03 2021-03-24T22:01:30.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: - 2021-03-24T22:01:30.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: 24 2021-03-24T22:01:30.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: T 2021-03-24T22:01:30.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: 22 2021-03-24T22:01:30.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: : 2021-03-24T22:01:30.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: 01 2021-03-24T22:01:30.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: : 2021-03-24T22:01:30.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: 30 2021-03-24T22:01:30.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: . 2021-03-24T22:01:30.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: 267579 2021-03-24T22:01:30.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: +0000 2021-03-24T22:01:30.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: 2021-03-24T22:01:30.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: mon.a 2021-03-24T22:01:30.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: ( 2021-03-24T22:01:30.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: mon 2021-03-24T22:01:30.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: . 2021-03-24T22:01:30.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: 0 2021-03-24T22:01:30.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: ) 2021-03-24T22:01:30.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: 1491 2021-03-24T22:01:30.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: : 2021-03-24T22:01:30.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: cluster 2021-03-24T22:01:30.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: 2021-03-24T22:01:30.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: [DBG] 2021-03-24T22:01:30.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: 2021-03-24T22:01:30.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: osdmap e520: 8 total, 8 up, 6 in 2021-03-24T22:01:30.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:30 smithi125 conmon[24927]: 2021-03-24T22:01:30.692 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:30.747 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:30.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:30.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:01:30.854 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:30.908 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:30.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:30.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:01:31.017 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:31.067 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:31.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:31.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:01:31.176 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:31.234 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:31.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:31.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:01:31.345 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:31.395 INFO:tasks.rados.rados.0.smithi125.stdout:1949: done (2 left) 2021-03-24T22:01:31.396 INFO:tasks.rados.rados.0.smithi125.stdout:1950: done (1 left) 2021-03-24T22:01:31.396 INFO:tasks.rados.rados.0.smithi125.stdout:1951: done (0 left) 2021-03-24T22:01:31.396 INFO:tasks.rados.rados.0.smithi125.stdout:1952: read oid 21 snap 192 2021-03-24T22:01:31.397 INFO:tasks.rados.rados.0.smithi125.stdout:1952: expect (ObjNum 590 snap 177 seq_num 590) 2021-03-24T22:01:31.397 INFO:tasks.rados.rados.0.smithi125.stdout:1953: read oid 45 snap -1 2021-03-24T22:01:31.397 INFO:tasks.rados.rados.0.smithi125.stdout:1953: expect (ObjNum 683 snap 214 seq_num 683) 2021-03-24T22:01:31.397 INFO:tasks.rados.rados.0.smithi125.stdout:1954: rollback oid 16 current snap is 216 2021-03-24T22:01:31.397 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 16 to 195 2021-03-24T22:01:31.398 INFO:tasks.rados.rados.0.smithi125.stdout:1955: read oid 5 snap -1 2021-03-24T22:01:31.398 INFO:tasks.rados.rados.0.smithi125.stdout:1955: expect (ObjNum 652 snap 198 seq_num 652) 2021-03-24T22:01:31.398 INFO:tasks.rados.rados.0.smithi125.stdout:1956: write oid 41 current snap is 216 2021-03-24T22:01:31.398 INFO:tasks.rados.rados.0.smithi125.stdout:1956: seq_num 690 ranges {796830=793580,2360693=691568} 2021-03-24T22:01:31.404 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:31.404 INFO:tasks.rados.rados.0.smithi125.stdout:1956: writing smithi12556470-41 from 796830 to 1590410 tid 1 2021-03-24T22:01:31.415 INFO:tasks.rados.rados.0.smithi125.stdout:1956: writing smithi12556470-41 from 2360693 to 3052261 tid 2 2021-03-24T22:01:31.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:31.427 INFO:tasks.rados.rados.0.smithi125.stdout:1955: expect (ObjNum 652 snap 198 seq_num 652) 2021-03-24T22:01:31.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:01:31.529 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:31.557 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[19936]: cluster 2021-03-24T22:01:31.361794+0000 mon.a 2021-03-24T22:01:31.558 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[19936]: (mon.0) 1492 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2021-03-24T22:01:31.558 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[19936]: cluster 2021-03-24T22:01:31.389886+0000 mon.a (mon.0) 1493 : cluster [DBG] osdmap e521: 8 total, 8 up, 6 in 2021-03-24T22:01:31.559 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: cluster 2021-03-24T22:01:31.559 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: 2021-03-24T22:01:31.559 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: 2021 2021-03-24T22:01:31.560 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: - 2021-03-24T22:01:31.560 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: 03 2021-03-24T22:01:31.560 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: - 2021-03-24T22:01:31.560 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: 24 2021-03-24T22:01:31.561 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: T 2021-03-24T22:01:31.561 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: 22 2021-03-24T22:01:31.561 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: : 2021-03-24T22:01:31.561 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: 01 2021-03-24T22:01:31.562 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: : 2021-03-24T22:01:31.562 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: 31 2021-03-24T22:01:31.562 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: . 2021-03-24T22:01:31.562 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: 361794 2021-03-24T22:01:31.563 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: +0000 2021-03-24T22:01:31.563 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: 2021-03-24T22:01:31.563 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: mon.a 2021-03-24T22:01:31.564 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: ( 2021-03-24T22:01:31.564 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: mon 2021-03-24T22:01:31.564 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: . 2021-03-24T22:01:31.565 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: 0 2021-03-24T22:01:31.565 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: ) 2021-03-24T22:01:31.565 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: 1492 2021-03-24T22:01:31.565 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: : 2021-03-24T22:01:31.566 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: cluster 2021-03-24T22:01:31.566 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: 2021-03-24T22:01:31.567 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: [WRN] 2021-03-24T22:01:31.567 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: 2021-03-24T22:01:31.568 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2021-03-24T22:01:31.568 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: 2021-03-24T22:01:31.568 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: cluster 2021-03-24T22:01:31.569 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: 2021-03-24T22:01:31.569 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: 2021 2021-03-24T22:01:31.569 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: - 2021-03-24T22:01:31.570 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: 03 2021-03-24T22:01:31.570 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: - 2021-03-24T22:01:31.570 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: 24 2021-03-24T22:01:31.571 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: T 2021-03-24T22:01:31.571 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: 22 2021-03-24T22:01:31.572 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: : 2021-03-24T22:01:31.572 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: 01 2021-03-24T22:01:31.572 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: : 2021-03-24T22:01:31.573 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: 31 2021-03-24T22:01:31.573 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: . 2021-03-24T22:01:31.573 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: 389886 2021-03-24T22:01:31.574 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: +0000 2021-03-24T22:01:31.574 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: 2021-03-24T22:01:31.575 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: mon.a 2021-03-24T22:01:31.575 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: ( 2021-03-24T22:01:31.575 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: mon 2021-03-24T22:01:31.575 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: . 2021-03-24T22:01:31.576 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: 0 2021-03-24T22:01:31.576 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:31 smithi125 conmon[24927]: ) 1493 : cluster [DBG] osdmap e521: 8 total, 8 up, 6 in 2021-03-24T22:01:31.586 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:31.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:31.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:01:31.596 INFO:tasks.rados.rados.0.smithi125.stdout:1955: done (4 left) 2021-03-24T22:01:31.596 INFO:tasks.rados.rados.0.smithi125.stdout:1957: read oid 28 snap -1 2021-03-24T22:01:31.596 INFO:tasks.rados.rados.0.smithi125.stdout:1957: expect deleted 2021-03-24T22:01:31.597 INFO:tasks.rados.rados.0.smithi125.stdout:1958: snap_create 2021-03-24T22:01:31.597 INFO:tasks.rados.rados.0.smithi125.stdout:1954: finishing rollback tid 0 to smithi12556470-16 2021-03-24T22:01:31.597 INFO:tasks.rados.rados.0.smithi125.stdout:1954: finishing rollback tid 1 to smithi12556470-16 2021-03-24T22:01:31.597 INFO:tasks.rados.rados.0.smithi125.stdout:1954: finishing rollback tid 2 to smithi12556470-16 2021-03-24T22:01:31.597 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 16 v 395 (ObjNum 605 snap 184 seq_num 605) dirty exists 2021-03-24T22:01:31.598 INFO:tasks.rados.rados.0.smithi125.stdout:1956: finishing write tid 1 to smithi12556470-41 2021-03-24T22:01:31.598 INFO:tasks.rados.rados.0.smithi125.stdout:1956: finishing write tid 2 to smithi12556470-41 2021-03-24T22:01:31.598 INFO:tasks.rados.rados.0.smithi125.stdout:1956: finishing write tid 3 to smithi12556470-41 2021-03-24T22:01:31.598 INFO:tasks.rados.rados.0.smithi125.stdout:1956: finishing write tid 4 to smithi12556470-41 2021-03-24T22:01:31.599 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 41 v 561 (ObjNum 690 snap 216 seq_num 690) dirty exists 2021-03-24T22:01:31.599 INFO:tasks.rados.rados.0.smithi125.stdout:1956: left oid 41 (ObjNum 690 snap 216 seq_num 690) 2021-03-24T22:01:31.599 INFO:tasks.rados.rados.0.smithi125.stdout:1953: expect (ObjNum 683 snap 214 seq_num 683) 2021-03-24T22:01:31.692 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:31.706 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:31 smithi142 conmon[20125]: cluster 2021-03-24T22:01:31 2021-03-24T22:01:31.706 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:31 smithi142 conmon[20125]: .361794+0000 mon.a (mon.0) 1492 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2021-03-24T22:01:31.707 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:31 smithi142 conmon[20125]: cluster 2021-03-24T22:01:31.389886+0000 2021-03-24T22:01:31.707 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:31 smithi142 conmon[20125]: mon.a (mon.0) 1493 : cluster [DBG] osdmap e521: 8 total, 8 up, 6 in 2021-03-24T22:01:31.719 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.2 dump_blocked_ops 2021-03-24T22:01:31.752 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:31.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:31.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:01:31.796 INFO:tasks.rados.rados.0.smithi125.stdout:1952: expect (ObjNum 590 snap 177 seq_num 590) 2021-03-24T22:01:31.858 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:31.923 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:31.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:31.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:01:32.030 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:32.098 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:32.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:32.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:01:32.205 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:32.263 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:32.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:32.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:01:32.370 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:32.403 INFO:tasks.rados.rados.0.smithi125.stdout:1952: done (5 left) 2021-03-24T22:01:32.404 INFO:tasks.rados.rados.0.smithi125.stdout:1953: done (4 left) 2021-03-24T22:01:32.404 INFO:tasks.rados.rados.0.smithi125.stdout:1954: done (3 left) 2021-03-24T22:01:32.404 INFO:tasks.rados.rados.0.smithi125.stdout:1956: done (2 left) 2021-03-24T22:01:32.404 INFO:tasks.rados.rados.0.smithi125.stdout:1957: done (1 left) 2021-03-24T22:01:32.405 INFO:tasks.rados.rados.0.smithi125.stdout:1958: done (0 left) 2021-03-24T22:01:32.405 INFO:tasks.rados.rados.0.smithi125.stdout:1959: rollback oid 31 current snap is 217 2021-03-24T22:01:32.405 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 31 to 198 2021-03-24T22:01:32.405 INFO:tasks.rados.rados.0.smithi125.stdout:1960: snap_create 2021-03-24T22:01:32.425 INFO:tasks.rados.rados.0.smithi125.stdout:1959: finishing rollback tid 1 to smithi12556470-31 2021-03-24T22:01:32.426 INFO:tasks.rados.rados.0.smithi125.stdout:1959: finishing rollback tid 2 to smithi12556470-31 2021-03-24T22:01:32.427 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 31 v 963 (ObjNum 577 snap 176 seq_num 577) dirty exists 2021-03-24T22:01:32.431 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:32.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:32.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:01:32.536 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:32.549 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:32 smithi142 conmon[20125]: cluster 2021-03-24T22:01:30.883137+0000 mgr.y (mgr.14144) 532 2021-03-24T22:01:32.549 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:32 smithi142 conmon[20125]: : cluster [DBG] pgmap v943: 168 pgs: 2 remapped+peering, 40 peering, 2 active+clean+wait, 5 active+remapped+backfill_toofull, 1 active+remapped, 118 active+clean; 476 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 MiB/s wr, 4 op/s; 113/1632 objects misplaced (6.924%); 2.7 MiB/s, 6 objects/s recovering 2021-03-24T22:01:32.549 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:32 smithi142 conmon[20125]: cluster 2021-03-24T22:01:32.400718+0000 mon.a (mon.0) 1494 : cluster [DBG] osdmap e522: 8 total, 8 up, 6 in 2021-03-24T22:01:32.591 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:32.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:32.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:01:32.611 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:32 smithi125 conmon[19936]: cluster 2021-03-24T22:01:30.883137+0000 mgr.y 2021-03-24T22:01:32.612 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:32 smithi125 conmon[19936]: (mgr.14144) 532 : cluster [DBG] pgmap v943: 168 pgs: 2 remapped+peering, 40 peering, 2 active+clean+wait, 5 active+remapped+backfill_toofull, 1 active+remapped, 118 active+clean; 476 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 MiB/s wr, 4 op/s; 113/1632 objects misplaced (6.924%); 2.7 MiB/s, 6 objects/s recovering 2021-03-24T22:01:32.612 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:32 smithi125 conmon[19936]: cluster 2021-03-24T22:01:32.400718+0000 mon.a (mon.0) 1494 : cluster [DBG] osdmap e522: 8 total, 8 up, 6 in 2021-03-24T22:01:32.612 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:32 smithi125 conmon[24927]: cluster 2021-03-24T22:01: 2021-03-24T22:01:32.612 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:32 smithi125 conmon[24927]: 30.883137+0000 mgr.y (mgr.14144) 532 : cluster [DBG] pgmap v943: 168 pgs: 2 remapped+peering, 40 peering, 2 active+clean+wait, 5 active+remapped+backfill_toofull, 1 active+remapped, 118 active+clean; 476 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 MiB/s wr, 4 op/s; 113/1632 objects misplaced (6.924%); 2.7 MiB/s, 6 objects/s recovering 2021-03-24T22:01:32.613 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:32 smithi125 conmon[24927]: cluster 2021-03-24T22:01:32.400718+0000 mon.a (mon.0) 1494 : cluster [DBG] osdmap e522: 8 total, 8 up, 6 in 2021-03-24T22:01:32.698 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:32.758 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:32.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:32.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:01:32.867 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:01:32.868 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:32.939 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:32.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:32.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:01:33.049 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:33.123 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:33.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:33.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:01:33.231 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:33.289 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:33.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:33.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:01:33.397 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:33.408 INFO:tasks.rados.rados.0.smithi125.stdout:1959: done (1 left) 2021-03-24T22:01:33.409 INFO:tasks.rados.rados.0.smithi125.stdout:1960: done (0 left) 2021-03-24T22:01:33.409 INFO:tasks.rados.rados.0.smithi125.stdout:1961: write oid 29 current snap is 218 2021-03-24T22:01:33.409 INFO:tasks.rados.rados.0.smithi125.stdout:1961: seq_num 691 ranges {413637=468829,1619175=435226,2542935=758255} 2021-03-24T22:01:33.414 INFO:tasks.rados.rados.0.smithi125.stdout:1961: writing smithi12556470-29 from 413637 to 882466 tid 1 2021-03-24T22:01:33.421 INFO:tasks.rados.rados.0.smithi125.stdout:1961: writing smithi12556470-29 from 1619175 to 2054401 tid 2 2021-03-24T22:01:33.430 INFO:tasks.rados.rados.0.smithi125.stdout:1961: writing smithi12556470-29 from 2542935 to 3301190 tid 3 2021-03-24T22:01:33.431 INFO:tasks.rados.rados.0.smithi125.stdout:1962: write oid 32 current snap is 218 2021-03-24T22:01:33.431 INFO:tasks.rados.rados.0.smithi125.stdout:1962: seq_num 692 ranges {430444=660431,1677657=695237,2970358=569527} 2021-03-24T22:01:33.438 INFO:tasks.rados.rados.0.smithi125.stdout:1962: writing smithi12556470-32 from 430444 to 1090875 tid 1 2021-03-24T22:01:33.446 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:33.447 INFO:tasks.rados.rados.0.smithi125.stdout:1962: writing smithi12556470-32 from 1677657 to 2372894 tid 2 2021-03-24T22:01:33.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:33.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:01:33.455 INFO:tasks.rados.rados.0.smithi125.stdout:1962: writing smithi12556470-32 from 2970358 to 3539885 tid 3 2021-03-24T22:01:33.455 INFO:tasks.rados.rados.0.smithi125.stdout:1963: write (excl) oid 8 current snap is 218 2021-03-24T22:01:33.455 INFO:tasks.rados.rados.0.smithi125.stdout:1963: seq_num 693 ranges {447251=735680,1736139=438895,2597781=780799} 2021-03-24T22:01:33.463 INFO:tasks.rados.rados.0.smithi125.stdout:1963: writing smithi12556470-8 from 447251 to 1182931 tid 1 2021-03-24T22:01:33.469 INFO:tasks.rados.rados.0.smithi125.stdout:1963: writing smithi12556470-8 from 1736139 to 2175034 tid 2 2021-03-24T22:01:33.478 INFO:tasks.rados.rados.0.smithi125.stdout:1963: writing smithi12556470-8 from 2597781 to 3378580 tid 3 2021-03-24T22:01:33.480 INFO:tasks.rados.rados.0.smithi125.stdout:1962: finishing write tid 1 to smithi12556470-32 2021-03-24T22:01:33.480 INFO:tasks.rados.rados.0.smithi125.stdout:1962: finishing write tid 2 to smithi12556470-32 2021-03-24T22:01:33.480 INFO:tasks.rados.rados.0.smithi125.stdout:1962: finishing write tid 3 to smithi12556470-32 2021-03-24T22:01:33.480 INFO:tasks.rados.rados.0.smithi125.stdout:1962: finishing write tid 4 to smithi12556470-32 2021-03-24T22:01:33.481 INFO:tasks.rados.rados.0.smithi125.stdout:1963: finishing write tid 1 to smithi12556470-8 2021-03-24T22:01:33.481 INFO:tasks.rados.rados.0.smithi125.stdout:1962: finishing write tid 5 to smithi12556470-32 2021-03-24T22:01:33.481 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 32 v 968 (ObjNum 692 snap 218 seq_num 692) dirty exists 2021-03-24T22:01:33.481 INFO:tasks.rados.rados.0.smithi125.stdout:1962: left oid 32 (ObjNum 692 snap 218 seq_num 692) 2021-03-24T22:01:33.481 INFO:tasks.rados.rados.0.smithi125.stdout:1963: finishing write tid 2 to smithi12556470-8 2021-03-24T22:01:33.482 INFO:tasks.rados.rados.0.smithi125.stdout:1962: done (2 left) 2021-03-24T22:01:33.482 INFO:tasks.rados.rados.0.smithi125.stdout:1964: copy_from oid 17 from oid 41 current snap is 218 2021-03-24T22:01:33.482 INFO:tasks.rados.rados.0.smithi125.stdout:1965: rollback oid 14 current snap is 218 2021-03-24T22:01:33.483 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 14 to 210 2021-03-24T22:01:33.483 INFO:tasks.rados.rados.0.smithi125.stdout:1966: copy_from oid 9 from oid 7 current snap is 218 2021-03-24T22:01:33.483 INFO:tasks.rados.rados.0.smithi125.stdout:1967: rollback oid 39 current snap is 218 2021-03-24T22:01:33.483 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 39 to 217 2021-03-24T22:01:33.483 INFO:tasks.rados.rados.0.smithi125.stdout:1968: read oid 37 snap -1 2021-03-24T22:01:33.484 INFO:tasks.rados.rados.0.smithi125.stdout:1968: expect (ObjNum 635 snap 196 seq_num 635) 2021-03-24T22:01:33.485 INFO:tasks.rados.rados.0.smithi125.stdout:1969: write oid 45 current snap is 218 2021-03-24T22:01:33.485 INFO:tasks.rados.rados.0.smithi125.stdout:1969: seq_num 694 ranges {464058=410929,1394621=698906,2625204=592071,3656107=7951} 2021-03-24T22:01:33.487 INFO:tasks.rados.rados.0.smithi125.stdout:1969: writing smithi12556470-45 from 464058 to 874987 tid 1 2021-03-24T22:01:33.496 INFO:tasks.rados.rados.0.smithi125.stdout:1969: writing smithi12556470-45 from 1394621 to 2093527 tid 2 2021-03-24T22:01:33.505 INFO:tasks.rados.rados.0.smithi125.stdout:1969: writing smithi12556470-45 from 2625204 to 3217275 tid 3 2021-03-24T22:01:33.510 INFO:tasks.rados.rados.0.smithi125.stdout:1969: writing smithi12556470-45 from 3656107 to 3664058 tid 4 2021-03-24T22:01:33.511 INFO:tasks.rados.rados.0.smithi125.stdout:1963: finishing write tid 3 to smithi12556470-8 2021-03-24T22:01:33.511 INFO:tasks.rados.rados.0.smithi125.stdout:1963: finishing write tid 4 to smithi12556470-8 2021-03-24T22:01:33.511 INFO:tasks.rados.rados.0.smithi125.stdout:1963: finishing write tid 5 to smithi12556470-8 2021-03-24T22:01:33.511 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 8 v 139 (ObjNum 693 snap 218 seq_num 693) dirty exists 2021-03-24T22:01:33.511 INFO:tasks.rados.rados.0.smithi125.stdout:1963: left oid 8 (ObjNum 693 snap 218 seq_num 693) 2021-03-24T22:01:33.512 INFO:tasks.rados.rados.0.smithi125.stdout:1968: expect (ObjNum 635 snap 196 seq_num 635) 2021-03-24T22:01:33.552 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:33.702 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:33.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:33.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:01:33.725 INFO:tasks.rados.rados.0.smithi125.stdout:1967: finishing rollback tid 0 to smithi12556470-39 2021-03-24T22:01:33.726 INFO:tasks.rados.rados.0.smithi125.stdout:1967: finishing rollback tid 1 to smithi12556470-39 2021-03-24T22:01:33.726 INFO:tasks.rados.rados.0.smithi125.stdout:1967: finishing rollback tid 2 to smithi12556470-39 2021-03-24T22:01:33.726 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 39 v 681 (ObjNum 688 snap 215 seq_num 688) dirty exists 2021-03-24T22:01:33.726 INFO:tasks.rados.rados.0.smithi125.stdout:1969: finishing write tid 1 to smithi12556470-45 2021-03-24T22:01:33.727 INFO:tasks.rados.rados.0.smithi125.stdout:1969: finishing write tid 2 to smithi12556470-45 2021-03-24T22:01:33.727 INFO:tasks.rados.rados.0.smithi125.stdout:1969: finishing write tid 3 to smithi12556470-45 2021-03-24T22:01:33.727 INFO:tasks.rados.rados.0.smithi125.stdout:1969: finishing write tid 4 to smithi12556470-45 2021-03-24T22:01:33.727 INFO:tasks.rados.rados.0.smithi125.stdout:1969: finishing write tid 5 to smithi12556470-45 2021-03-24T22:01:33.727 INFO:tasks.rados.rados.0.smithi125.stdout:1969: finishing write tid 6 to smithi12556470-45 2021-03-24T22:01:33.728 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 45 v 654 (ObjNum 694 snap 218 seq_num 694) dirty exists 2021-03-24T22:01:33.728 INFO:tasks.rados.rados.0.smithi125.stdout:1969: left oid 45 (ObjNum 694 snap 218 seq_num 694) 2021-03-24T22:01:33.728 INFO:tasks.rados.rados.0.smithi125.stdout:1965: finishing rollback tid 1 to smithi12556470-14 2021-03-24T22:01:33.728 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 14 v 858 (ObjNum 1 snap 32741 seq_num 1116516752) dirty dne 2021-03-24T22:01:33.729 INFO:tasks.rados.rados.0.smithi125.stdout:1964: finishing copy_from to smithi12556470-17 2021-03-24T22:01:33.729 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 17 v 561 (ObjNum 690 snap 216 seq_num 690) dirty exists 2021-03-24T22:01:33.729 INFO:tasks.rados.rados.0.smithi125.stdout:1964: finishing copy_from racing read to smithi12556470-17 2021-03-24T22:01:33.729 INFO:tasks.rados.rados.0.smithi125.stdout:1966: finishing copy_from to smithi12556470-9 2021-03-24T22:01:33.729 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 9 v 239 (ObjNum 588 snap 177 seq_num 588) dirty exists 2021-03-24T22:01:33.730 INFO:tasks.rados.rados.0.smithi125.stdout:1966: finishing copy_from racing read to smithi12556470-9 2021-03-24T22:01:33.730 INFO:tasks.rados.rados.0.smithi125.stdout:1963: done (7 left) 2021-03-24T22:01:33.730 INFO:tasks.rados.rados.0.smithi125.stdout:1964: done (6 left) 2021-03-24T22:01:33.730 INFO:tasks.rados.rados.0.smithi125.stdout:1965: done (5 left) 2021-03-24T22:01:33.730 INFO:tasks.rados.rados.0.smithi125.stdout:1966: done (4 left) 2021-03-24T22:01:33.731 INFO:tasks.rados.rados.0.smithi125.stdout:1967: done (3 left) 2021-03-24T22:01:33.731 INFO:tasks.rados.rados.0.smithi125.stdout:1968: done (2 left) 2021-03-24T22:01:33.731 INFO:tasks.rados.rados.0.smithi125.stdout:1969: done (1 left) 2021-03-24T22:01:33.731 INFO:tasks.rados.rados.0.smithi125.stdout:1970: write (excl) oid 23 current snap is 218 2021-03-24T22:01:33.732 INFO:tasks.rados.rados.0.smithi125.stdout:1970: seq_num 695 ranges {480865=486178,1453103=442564,2652627=519696} 2021-03-24T22:01:33.732 INFO:tasks.rados.rados.0.smithi125.stdout:1970: writing smithi12556470-23 from 480865 to 967043 tid 1 2021-03-24T22:01:33.736 INFO:tasks.rados.rados.0.smithi125.stdout:1970: writing smithi12556470-23 from 1453103 to 1895667 tid 2 2021-03-24T22:01:33.744 INFO:tasks.rados.rados.0.smithi125.stdout:1970: writing smithi12556470-23 from 2652627 to 3172323 tid 3 2021-03-24T22:01:33.744 INFO:tasks.rados.rados.0.smithi125.stdout:1971: delete oid 3 current snap is 218 2021-03-24T22:01:33.745 INFO:tasks.rados.rados.0.smithi125.stdout:1970: finishing write tid 1 to smithi12556470-23 2021-03-24T22:01:33.748 INFO:tasks.rados.rados.0.smithi125.stdout:1971: done (2 left) 2021-03-24T22:01:33.748 INFO:tasks.rados.rados.0.smithi125.stdout:1972: rollback oid 47 current snap is 218 2021-03-24T22:01:33.748 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 47 to 210 2021-03-24T22:01:33.748 INFO:tasks.rados.rados.0.smithi125.stdout:1973: write oid 27 current snap is 218 2021-03-24T22:01:33.749 INFO:tasks.rados.rados.0.smithi125.stdout:1973: seq_num 697 ranges {514479=636676,1853714=446233,2991120=542240} 2021-03-24T22:01:33.755 INFO:tasks.rados.rados.0.smithi125.stdout:1973: writing smithi12556470-27 from 514479 to 1151155 tid 1 2021-03-24T22:01:33.767 INFO:tasks.rados.rados.0.smithi125.stdout:1973: writing smithi12556470-27 from 1853714 to 2299947 tid 2 2021-03-24T22:01:33.775 INFO:tasks.rados.rados.0.smithi125.stdout:1973: writing smithi12556470-27 from 2991120 to 3533360 tid 3 2021-03-24T22:01:33.776 INFO:tasks.rados.rados.0.smithi125.stdout:1974: rollback oid 39 current snap is 218 2021-03-24T22:01:33.776 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 39 to 216 2021-03-24T22:01:33.776 INFO:tasks.rados.rados.0.smithi125.stdout:1975: snap_create 2021-03-24T22:01:33.777 INFO:tasks.rados.rados.0.smithi125.stdout:1970: finishing write tid 2 to smithi12556470-23 2021-03-24T22:01:33.777 INFO:tasks.rados.rados.0.smithi125.stdout:1970: finishing write tid 3 to smithi12556470-23 2021-03-24T22:01:33.777 INFO:tasks.rados.rados.0.smithi125.stdout:1970: finishing write tid 4 to smithi12556470-23 2021-03-24T22:01:33.777 INFO:tasks.rados.rados.0.smithi125.stdout:1970: finishing write tid 5 to smithi12556470-23 2021-03-24T22:01:33.777 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 23 v 322 (ObjNum 695 snap 218 seq_num 695) dirty exists 2021-03-24T22:01:33.778 INFO:tasks.rados.rados.0.smithi125.stdout:1970: left oid 23 (ObjNum 695 snap 218 seq_num 695) 2021-03-24T22:01:33.778 INFO:tasks.rados.rados.0.smithi125.stdout:1972: finishing rollback tid 0 to smithi12556470-47 2021-03-24T22:01:33.778 INFO:tasks.rados.rados.0.smithi125.stdout:1972: finishing rollback tid 1 to smithi12556470-47 2021-03-24T22:01:33.778 INFO:tasks.rados.rados.0.smithi125.stdout:1972: finishing rollback tid 2 to smithi12556470-47 2021-03-24T22:01:33.779 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 47 v 343 (ObjNum 466 snap 131 seq_num 466) dirty exists 2021-03-24T22:01:33.779 INFO:tasks.rados.rados.0.smithi125.stdout:1973: finishing write tid 1 to smithi12556470-27 2021-03-24T22:01:33.782 INFO:tasks.rados.rados.0.smithi125.stdout:1974: finishing rollback tid 0 to smithi12556470-39 2021-03-24T22:01:33.782 INFO:tasks.rados.rados.0.smithi125.stdout:1974: finishing rollback tid 1 to smithi12556470-39 2021-03-24T22:01:33.783 INFO:tasks.rados.rados.0.smithi125.stdout:1973: finishing write tid 2 to smithi12556470-27 2021-03-24T22:01:33.783 INFO:tasks.rados.rados.0.smithi125.stdout:1973: finishing write tid 3 to smithi12556470-27 2021-03-24T22:01:33.783 INFO:tasks.rados.rados.0.smithi125.stdout:1973: finishing write tid 4 to smithi12556470-27 2021-03-24T22:01:33.785 INFO:tasks.rados.rados.0.smithi125.stdout:1974: finishing rollback tid 2 to smithi12556470-39 2021-03-24T22:01:33.786 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 39 v 684 (ObjNum 688 snap 215 seq_num 688) dirty exists 2021-03-24T22:01:33.786 INFO:tasks.rados.rados.0.smithi125.stdout:1973: finishing write tid 5 to smithi12556470-27 2021-03-24T22:01:33.786 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 27 v 555 (ObjNum 697 snap 218 seq_num 697) dirty exists 2021-03-24T22:01:33.786 INFO:tasks.rados.rados.0.smithi125.stdout:1973: left oid 27 (ObjNum 697 snap 218 seq_num 697) 2021-03-24T22:01:33.826 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:33.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:33.895 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:33.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:01:33.997 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:34.055 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:34.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:34.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:01:34.161 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:34.206 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:34.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:34.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:01:34.311 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:34.372 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:34.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:34.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:01:34.401 INFO:tasks.rados.rados.0.smithi125.stdout:1970: done (5 left) 2021-03-24T22:01:34.402 INFO:tasks.rados.rados.0.smithi125.stdout:1972: done (4 left) 2021-03-24T22:01:34.402 INFO:tasks.rados.rados.0.smithi125.stdout:1973: done (3 left) 2021-03-24T22:01:34.402 INFO:tasks.rados.rados.0.smithi125.stdout:1974: done (2 left) 2021-03-24T22:01:34.402 INFO:tasks.rados.rados.0.smithi125.stdout:1975: done (1 left) 2021-03-24T22:01:34.403 INFO:tasks.rados.rados.0.smithi125.stdout:1976: read oid 46 snap -1 2021-03-24T22:01:34.403 INFO:tasks.rados.rados.0.smithi125.stdout:1976: expect (ObjNum 680 snap 212 seq_num 680) 2021-03-24T22:01:34.403 INFO:tasks.rados.rados.0.smithi125.stdout:1977: read oid 17 snap 203 2021-03-24T22:01:34.403 INFO:tasks.rados.rados.0.smithi125.stdout:1977: expect (ObjNum 619 snap 188 seq_num 619) 2021-03-24T22:01:34.404 INFO:tasks.rados.rados.0.smithi125.stdout:1978: snap_create 2021-03-24T22:01:34.419 INFO:teuthology.orchestra.run.smithi125.stderr:noscrub is unset 2021-03-24T22:01:34.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: cluster 2021-03-24T22:01:34.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 2021-03-24T22:01:34.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 2021 2021-03-24T22:01:34.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: - 2021-03-24T22:01:34.421 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 03 2021-03-24T22:01:34.421 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: - 2021-03-24T22:01:34.421 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 24 2021-03-24T22:01:34.421 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: T 2021-03-24T22:01:34.422 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 22 2021-03-24T22:01:34.422 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: : 2021-03-24T22:01:34.422 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 01 2021-03-24T22:01:34.422 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: : 2021-03-24T22:01:34.423 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 32 2021-03-24T22:01:34.423 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: . 2021-03-24T22:01:34.423 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 883727 2021-03-24T22:01:34.423 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: +0000 2021-03-24T22:01:34.423 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 2021-03-24T22:01:34.424 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: mgr.y 2021-03-24T22:01:34.424 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: ( 2021-03-24T22:01:34.424 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: mgr 2021-03-24T22:01:34.424 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: . 2021-03-24T22:01:34.424 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 14144 2021-03-24T22:01:34.425 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: ) 2021-03-24T22:01:34.425 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 533 2021-03-24T22:01:34.425 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: : 2021-03-24T22:01:34.425 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: cluster 2021-03-24T22:01:34.426 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 2021-03-24T22:01:34.426 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: [DBG] 2021-03-24T22:01:34.426 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 2021-03-24T22:01:34.426 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: pgmap v946: 168 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 2 remapped+peering, 41 peering, 2 active+clean+wait, 4 active+remapped+backfill_toofull, 115 active+clean; 481 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 4.2 MiB/s rd, 2.8 MiB/s wr, 10 op/s; 19/1635 objects degraded (1.162%); 93/1635 objects misplaced (5.688%); 11 MiB/s, 33 objects/s recovering 2021-03-24T22:01:34.427 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 2021-03-24T22:01:34.427 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: cluster 2021-03-24T22:01:34.427 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 2021-03-24T22:01:34.427 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 2021 2021-03-24T22:01:34.427 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: - 2021-03-24T22:01:34.428 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 03 2021-03-24T22:01:34.428 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: -24T22:01:33.377067 2021-03-24T22:01:34.428 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: +0000 osd.1 (osd.1) 45 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:01:34.428 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: cluster 2021-03-24T22:01:34.430 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 2021-03-24T22:01:34.430 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 2021 2021-03-24T22:01:34.430 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: -03-24T22: 2021-03-24T22:01:34.430 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 01:33.377895+0000 osd.1 (osd.1) 46 2021-03-24T22:01:34.430 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:01:34.431 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: cluster 2021-03-24T22:01:34.431 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 2021-03-24T22:01:34.431 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 2021 2021-03-24T22:01:34.431 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: - 2021-03-24T22:01:34.432 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 03 2021-03-24T22:01:34.432 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: - 2021-03-24T22:01:34.432 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 24 2021-03-24T22:01:34.432 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: T 2021-03-24T22:01:34.432 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 22 2021-03-24T22:01:34.433 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: : 2021-03-24T22:01:34.433 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 01 2021-03-24T22:01:34.433 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: : 2021-03-24T22:01:34.433 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 33 2021-03-24T22:01:34.434 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: . 2021-03-24T22:01:34.434 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 405262 2021-03-24T22:01:34.434 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: +0000 2021-03-24T22:01:34.434 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 2021-03-24T22:01:34.434 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: mon.a 2021-03-24T22:01:34.435 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: ( 2021-03-24T22:01:34.435 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: mon 2021-03-24T22:01:34.435 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: . 2021-03-24T22:01:34.435 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 0 2021-03-24T22:01:34.435 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: ) 2021-03-24T22:01:34.436 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 1495 2021-03-24T22:01:34.436 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: : 2021-03-24T22:01:34.436 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: cluster 2021-03-24T22:01:34.436 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 2021-03-24T22:01:34.436 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: [DBG] 2021-03-24T22:01:34.437 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 2021-03-24T22:01:34.437 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: osdmap e523: 8 total, 8 up, 6 in 2021-03-24T22:01:34.437 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 2021-03-24T22:01:34.437 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: cluster 2021-03-24T22:01:34.438 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 2021-03-24T22:01:34.438 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 2021 2021-03-24T22:01:34.439 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: - 2021-03-24T22:01:34.439 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 03 2021-03-24T22:01:34.439 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: - 2021-03-24T22:01:34.439 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 24 2021-03-24T22:01:34.439 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: T 2021-03-24T22:01:34.440 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 22 2021-03-24T22:01:34.440 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: : 2021-03-24T22:01:34.440 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 01 2021-03-24T22:01:34.440 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: : 2021-03-24T22:01:34.441 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 33 2021-03-24T22:01:34.441 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: . 2021-03-24T22:01:34.441 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 419516 2021-03-24T22:01:34.441 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: +0000 2021-03-24T22:01:34.442 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 2021-03-24T22:01:34.442 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: mon.a 2021-03-24T22:01:34.442 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: ( 2021-03-24T22:01:34.442 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: mon 2021-03-24T22:01:34.442 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: . 2021-03-24T22:01:34.443 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 0 2021-03-24T22:01:34.443 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: ) 2021-03-24T22:01:34.443 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 1496 2021-03-24T22:01:34.443 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: : 2021-03-24T22:01:34.444 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: cluster 2021-03-24T22:01:34.444 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 2021-03-24T22:01:34.444 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: [WRN] 2021-03-24T22:01:34.444 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 2021-03-24T22:01:34.445 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: Health check failed: Degraded data redundancy: 19/1635 objects degraded (1.162%), 2 pgs degraded (PG_DEGRADED) 2021-03-24T22:01:34.445 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 2021-03-24T22:01:34.445 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: audit 2021-03-24T22:01:34.445 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 2021-03-24T22:01:34.445 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 2021 2021-03-24T22:01:34.446 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: - 2021-03-24T22:01:34.446 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 03 2021-03-24T22:01:34.446 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: - 2021-03-24T22:01:34.446 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 24 2021-03-24T22:01:34.447 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: T 2021-03-24T22:01:34.447 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 22 2021-03-24T22:01:34.447 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: : 2021-03-24T22:01:34.447 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 01 2021-03-24T22:01:34.447 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: : 2021-03-24T22:01:34.448 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 33 2021-03-24T22:01:34.448 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: . 2021-03-24T22:01:34.448 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 818443 2021-03-24T22:01:34.448 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: +0000 2021-03-24T22:01:34.448 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: mon.c (mon.2) 81 2021-03-24T22:01:34.449 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: : audit [INF] from='client.? v1:172.21.15.125:0/3903323567' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T22:01:34.449 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: audit 2021-03-24T22:01:34.449 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 2021-03-24T22:01:34.449 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 2021 2021-03-24T22:01:34.450 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: - 2021-03-24T22:01:34.450 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 03 2021-03-24T22:01:34.450 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: - 2021-03-24T22:01:34.450 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 24 2021-03-24T22:01:34.451 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: T 2021-03-24T22:01:34.451 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 22:01:33. 2021-03-24T22:01:34.451 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 818832 2021-03-24T22:01:34.451 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: +0000 2021-03-24T22:01:34.452 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 2021-03-24T22:01:34.452 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: mon.a 2021-03-24T22:01:34.452 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: ( 2021-03-24T22:01:34.453 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[24927]: cluster 2021-03-24T22:01:32. 2021-03-24T22:01:34.453 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[24927]: 883727+0000 mgr.y (mgr.14144) 533 : cluster [DBG] pgmap v946: 168 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 2 remapped+peering, 41 peering, 2 active+clean+wait, 4 active+remapped+backfill_toofull, 115 active+clean; 481 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 4.2 MiB/s rd, 2.8 MiB/s wr, 10 op/s; 19/1635 objects degraded (1.162%); 93/1635 objects misplaced (5.688%); 11 MiB/s, 33 objects/s recovering 2021-03-24T22:01:34.454 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[24927]: cluster 2021-03-24T22:01:33.377067+0000 osd.1 (osd 2021-03-24T22:01:34.454 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[24927]: .1) 45 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:01:34.454 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[24927]: cluster 2021-03-24T22:01: 2021-03-24T22:01:34.454 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[24927]: 33.377895+0000 osd.1 (osd.1) 46 : cluster 2021-03-24T22:01:34.454 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[24927]: [DBG] purged_snaps scrub ok 2021-03-24T22:01:34.455 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[24927]: cluster 2021-03-24T22:01:34.455 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[24927]: 2021-03-24T22:01:33.405262+0000 mon.a (mon.0 2021-03-24T22:01:34.455 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[24927]: ) 1495 : cluster [DBG] osdmap e523: 8 total, 8 up, 6 in 2021-03-24T22:01:34.455 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[24927]: cluster 2021-03-24T22 2021-03-24T22:01:34.456 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[24927]: :01:33.419516+0000 mon.a (mon.0) 1496 2021-03-24T22:01:34.456 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[24927]: : cluster [WRN] Health check failed: Degraded data redundancy: 19/1635 objects degraded (1.162%), 2 pgs degraded (PG_DEGRADED) 2021-03-24T22:01:34.456 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[24927]: audit 2021-03-24T22:01:33. 2021-03-24T22:01:34.456 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[24927]: 818443+0000 mon.c (mon.2) 81 : audit [INF] 2021-03-24T22:01:34.456 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[24927]: from='client.? v1:172.21.15.125:0/3903323567' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T22:01:34.457 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[24927]: audit 2021-03-24T22:01:34.457 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[24927]: 2021-03-24T22:01:33.818832+0000 mon.a (mon.0 2021-03-24T22:01:34.457 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[24927]: ) 1497 : audit [INF] from='client.35381 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T22:01:34.457 INFO:tasks.rados.rados.0.smithi125.stdout:1977: expect (ObjNum 619 snap 188 seq_num 619) 2021-03-24T22:01:34.477 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:34.536 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:34.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:34.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:01:34.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:34 smithi142 conmon[20125]: cluster 2021-03-24T22:01:32 2021-03-24T22:01:34.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:34 smithi142 conmon[20125]: .883727+0000 mgr.y (mgr.14144) 533 : cluster [DBG] pgmap v946: 168 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 2 remapped+peering, 41 peering, 2 active+clean+wait, 4 active+remapped+backfill_toofull, 115 active+clean; 481 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 4.2 MiB/s rd, 2.8 MiB/s wr, 10 op/s; 19/1635 objects degraded (1.162%); 93/1635 objects misplaced (5.688%); 11 MiB/s, 33 objects/s recovering 2021-03-24T22:01:34.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:34 smithi142 conmon[20125]: cluster 2021-03-24T22:01:33.377067+0000 osd.1 ( 2021-03-24T22:01:34.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:34 smithi142 conmon[20125]: osd.1) 45 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:01:34.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:34 smithi142 conmon[20125]: cluster 2021-03-24T22:01:34.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:34 smithi142 conmon[20125]: 2021-03-24T22:01:33.377895+0000 osd.1 (osd.1) 46 : cluster [DBG] 2021-03-24T22:01:34.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:34 smithi142 conmon[20125]: purged_snaps scrub ok 2021-03-24T22:01:34.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:34 smithi142 conmon[20125]: cluster 2021-03-24T22:01 2021-03-24T22:01:34.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:34 smithi142 conmon[20125]: :33.405262+0000 mon.a (mon.0) 1495 : cluster 2021-03-24T22:01:34.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:34 smithi142 conmon[20125]: [DBG] osdmap e523: 8 total, 8 up, 6 in 2021-03-24T22:01:34.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:34 smithi142 conmon[20125]: cluster 2021-03-24T22: 2021-03-24T22:01:34.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:34 smithi142 conmon[20125]: 01:33.419516+0000 mon.a (mon.0) 1496 : 2021-03-24T22:01:34.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:34 smithi142 conmon[20125]: cluster [WRN] Health check failed: Degraded data redundancy: 19/1635 objects degraded (1.162%), 2 pgs degraded (PG_DEGRADED) 2021-03-24T22:01:34.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:34 smithi142 conmon[20125]: audit 2021-03 2021-03-24T22:01:34.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:34 smithi142 conmon[20125]: -24T22:01:33.818443+0000 mon.c (mon 2021-03-24T22:01:34.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:34 smithi142 conmon[20125]: .2) 81 : audit [INF] from='client.? v1:172.21.15.125:0/3903323567' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T22:01:34.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:34 smithi142 conmon[20125]: audit 2021- 2021-03-24T22:01:34.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:34 smithi142 conmon[20125]: 03-24T22:01:33.818832+0000 mon.a (mon 2021-03-24T22:01:34.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:34 smithi142 conmon[20125]: .0) 1497 : audit [INF] from='client.35381 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T22:01:34.622 INFO:tasks.rados.rados.0.smithi125.stdout:1961: finishing write tid 1 to smithi12556470-29 2021-03-24T22:01:34.622 INFO:tasks.rados.rados.0.smithi125.stdout:1961: finishing write tid 2 to smithi12556470-29 2021-03-24T22:01:34.622 INFO:tasks.rados.rados.0.smithi125.stdout:1961: finishing write tid 3 to smithi12556470-29 2021-03-24T22:01:34.622 INFO:tasks.rados.rados.0.smithi125.stdout:1961: finishing write tid 4 to smithi12556470-29 2021-03-24T22:01:34.623 INFO:tasks.rados.rados.0.smithi125.stdout:1961: finishing write tid 5 to smithi12556470-29 2021-03-24T22:01:34.623 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 29 v 298 (ObjNum 691 snap 218 seq_num 691) dirty exists 2021-03-24T22:01:34.623 INFO:tasks.rados.rados.0.smithi125.stdout:1961: left oid 29 (ObjNum 691 snap 218 seq_num 691) 2021-03-24T22:01:34.623 INFO:tasks.rados.rados.0.smithi125.stdout:1976: expect (ObjNum 680 snap 212 seq_num 680) 2021-03-24T22:01:34.631 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-24T22:01:34.641 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:34.702 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:34.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:34.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:01:34.809 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:34.837 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: mon 2021-03-24T22:01:34.838 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: . 2021-03-24T22:01:34.838 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 0 2021-03-24T22:01:34.838 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: ) 2021-03-24T22:01:34.839 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 1497 2021-03-24T22:01:34.839 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: : 2021-03-24T22:01:34.839 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: audit 2021-03-24T22:01:34.839 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 2021-03-24T22:01:34.839 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: [INF] 2021-03-24T22:01:34.840 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 2021-03-24T22:01:34.840 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: from='client.35381 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T22:01:34.840 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:34 smithi125 conmon[19936]: 2021-03-24T22:01:34.865 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:34.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:34.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:01:34.972 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:35.018 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:35.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:35.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:01:35.127 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:35.191 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:35.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:35.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:01:35.281 INFO:tasks.rados.rados.0.smithi125.stdout:1961: done (3 left) 2021-03-24T22:01:35.282 INFO:tasks.rados.rados.0.smithi125.stdout:1976: done (2 left) 2021-03-24T22:01:35.282 INFO:tasks.rados.rados.0.smithi125.stdout:1977: done (1 left) 2021-03-24T22:01:35.282 INFO:tasks.rados.rados.0.smithi125.stdout:1978: done (0 left) 2021-03-24T22:01:35.283 INFO:tasks.rados.rados.0.smithi125.stdout:1979: snap_remove snap 211 2021-03-24T22:01:35.296 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:35.341 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:35.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:35.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:01:35.446 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:35.485 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:35.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:35.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:01:35.589 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:35.647 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:35.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:35.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:01:35.738 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:35 smithi125 conmon[24927]: cluster 2021-03-24T22:01:35.739 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:35 smithi125 conmon[24927]: 2021-03 2021-03-24T22:01:35.739 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:35 smithi125 conmon[24927]: -24T22:01:34.390000+0000 mon.a (mon.0) 1498 : cluster [WRN] 2021-03-24T22:01:35.739 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:35 smithi125 conmon[24927]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:01:35.740 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:35 smithi125 conmon[24927]: audit 2021-03-24T 2021-03-24T22:01:35.740 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:35 smithi125 conmon[24927]: 22:01:34.397115+0000 mon.a (mon.0 2021-03-24T22:01:35.740 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:35 smithi125 conmon[24927]: ) 1499 : audit [INF] from='client.35381 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T22:01:35.740 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:35 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T22:01:35.741 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:35 smithi125 conmon[24927]: T22:01:34.397456+0000 mon.a (mon. 2021-03-24T22:01:35.741 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:35 smithi125 conmon[24927]: 0) 1500 : cluster [DBG] osdmap e524: 8 total, 8 up, 6 in 2021-03-24T22:01:35.741 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:35 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T22:01:35.741 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:35 smithi125 conmon[24927]: -24T22:01:34.765281+0000 osd.4 ( 2021-03-24T22:01:35.742 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:35 smithi125 conmon[24927]: osd.4) 111 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:01:35.742 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:35 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T22:01:35.742 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:35 smithi125 conmon[24927]: -24T22:01:34.767050+0000 osd.4 ( 2021-03-24T22:01:35.742 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:35 smithi125 conmon[24927]: osd.4) 112 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:01:35.743 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:35 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:01:35.743 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:35 smithi125 conmon[24927]: 03-24T22:01:35.264454+0000 mon.a 2021-03-24T22:01:35.743 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:35 smithi125 conmon[24927]: (mon.0) 1501 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:01:35.743 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:35 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:01:35.744 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:35 smithi125 conmon[24927]: 03-24T22:01:35.278898+0000 mon.a 2021-03-24T22:01:35.744 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:35 smithi125 conmon[24927]: (mon.0) 1502 : cluster [DBG] osdmap e525: 8 total, 8 up, 6 in 2021-03-24T22:01:35.744 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:35 smithi125 conmon[19936]: cluster 2021-03-24T22:01:34.390000 2021-03-24T22:01:35.744 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:35 smithi125 conmon[19936]: +0000 mon.a (mon.0) 1498 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:01:35.745 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:35 smithi125 conmon[19936]: audit 2021-03-24T22:01:34.397115+0000 mon.a ( 2021-03-24T22:01:35.745 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:35 smithi125 conmon[19936]: mon.0) 1499 : audit [INF] from='client.35381 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T22:01:35.745 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:35 smithi125 conmon[19936]: cluster 2021-03-24T22:01:35.745 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:35 smithi125 conmon[19936]: 2021-03-24T22:01:34.397456+0000 mon.a 2021-03-24T22:01:35.746 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:35 smithi125 conmon[19936]: (mon.0) 1500 : cluster [DBG] osdmap e524: 8 total, 8 up, 6 in 2021-03-24T22:01:35.746 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:35 smithi125 conmon[19936]: 2021-03-24T22:01:35.746 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:35 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:01:35.747 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:35 smithi125 conmon[19936]: 03-24T22:01:34.765281+0000 osd.4 (osd 2021-03-24T22:01:35.747 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:35 smithi125 conmon[19936]: .4) 111 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:01:35.747 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:35 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:01:35.747 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:35 smithi125 conmon[19936]: 03-24T22:01:34. 2021-03-24T22:01:35.748 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:35 smithi125 conmon[19936]: 767050+0000 osd.4 (osd.4) 112 : cluster 2021-03-24T22:01:35.748 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:35 smithi125 conmon[19936]: [DBG] purged_snaps scrub ok 2021-03-24T22:01:35.748 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:35 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:01:35.748 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:35 smithi125 conmon[19936]: 01:35.264454+0000 mon.a (mon.0 2021-03-24T22:01:35.749 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:35 smithi125 conmon[19936]: ) 1501 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:01:35.749 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:35 smithi125 conmon[19936]: cluster 2021-03-24T22 2021-03-24T22:01:35.749 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:35 smithi125 conmon[19936]: :01:35.278898+0000 mon.a (mon. 2021-03-24T22:01:35.749 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:35 smithi125 conmon[19936]: 0) 1502 : cluster [DBG] osdmap e525: 8 total, 8 up, 6 in 2021-03-24T22:01:35.754 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:35.767 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:35 smithi142 conmon[20125]: cluster 2021-03-24T22:01:34.390000+0000 mon.a (mon.0) 1498 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:01:35.767 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:35 smithi142 conmon[20125]: audit 2021-03-24T22:01:35.767 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:35 smithi142 conmon[20125]: 2021-03-24T22:01:34.397115+0000 mon.a (mon.0) 1499 : audit [INF] from='client.35381 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T22:01:35.768 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:35 smithi142 conmon[20125]: cluster 2021-03-24T22:01:34.397456+0000 mon.a (mon.0) 1500 : cluster [DBG] osdmap e524: 8 total, 8 up, 6 in 2021-03-24T22:01:35.768 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:35 smithi142 conmon[20125]: cluster 2021-03-24T22:01:34. 2021-03-24T22:01:35.768 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:35 smithi142 conmon[20125]: 765281+0000 osd.4 (osd.4) 111 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:01:35.768 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:35 smithi142 conmon[20125]: cluster 2021-03-24T22:01:35.768 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:35 smithi142 conmon[20125]: 2021-03-24T22:01:34.767050+0000 2021-03-24T22:01:35.769 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:35 smithi142 conmon[20125]: osd.4 (osd.4) 112 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:01:35.769 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:35 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:01:35.769 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:35 smithi142 conmon[20125]: -24T22:01:35.264454+0000 mon.a 2021-03-24T22:01:35.769 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:35 smithi142 conmon[20125]: (mon.0) 1501 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:01:35.770 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:35 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:01:35.770 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:35 smithi142 conmon[20125]: T22:01:35.278898+0000 mon.a (mon.0) 1502 : 2021-03-24T22:01:35.770 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:35 smithi142 conmon[20125]: cluster [DBG] osdmap e525: 8 total, 8 up, 6 in 2021-03-24T22:01:35.811 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:35.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:35.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:01:35.918 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:35.980 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:35.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:35.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:01:36.087 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:36.104 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:01:36.143 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:36.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:36.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:01:36.248 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:36.303 INFO:tasks.rados.rados.0.smithi125.stdout:1979: done (0 left) 2021-03-24T22:01:36.303 INFO:tasks.rados.rados.0.smithi125.stdout:1980: rollback oid 47 current snap is 220 2021-03-24T22:01:36.303 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 47 to 218 2021-03-24T22:01:36.304 INFO:tasks.rados.rados.0.smithi125.stdout:1981: write (excl) oid 30 current snap is 220 2021-03-24T22:01:36.304 INFO:tasks.rados.rados.0.smithi125.stdout:1981: seq_num 698 ranges {531286=711925,1912196=706244,3018543=712743} 2021-03-24T22:01:36.304 INFO:tasks.rados.rados.0.smithi125.stdout:1981: writing smithi12556470-30 from 531286 to 1243211 tid 1 2021-03-24T22:01:36.309 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:36.309 INFO:tasks.rados.rados.0.smithi125.stdout:1981: writing smithi12556470-30 from 1912196 to 2618440 tid 2 2021-03-24T22:01:36.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:36.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:01:36.319 INFO:tasks.rados.rados.0.smithi125.stdout:1981: writing smithi12556470-30 from 3018543 to 3731286 tid 3 2021-03-24T22:01:36.320 INFO:tasks.rados.rados.0.smithi125.stdout:1980: finishing rollback tid 0 to smithi12556470-47 2021-03-24T22:01:36.320 INFO:tasks.rados.rados.0.smithi125.stdout:1980: finishing rollback tid 1 to smithi12556470-47 2021-03-24T22:01:36.320 INFO:tasks.rados.rados.0.smithi125.stdout:1980: finishing rollback tid 2 to smithi12556470-47 2021-03-24T22:01:36.320 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 47 v 346 (ObjNum 466 snap 131 seq_num 466) dirty exists 2021-03-24T22:01:36.321 INFO:tasks.rados.rados.0.smithi125.stdout:1980: done (1 left) 2021-03-24T22:01:36.321 INFO:tasks.rados.rados.0.smithi125.stdout:1982: delete oid 25 current snap is 220 2021-03-24T22:01:36.322 INFO:tasks.rados.rados.0.smithi125.stdout:1981: finishing write tid 1 to smithi12556470-30 2021-03-24T22:01:36.327 INFO:tasks.rados.rados.0.smithi125.stdout:1982: done (1 left) 2021-03-24T22:01:36.327 INFO:tasks.rados.rados.0.smithi125.stdout:1983: rollback oid 44 current snap is 220 2021-03-24T22:01:36.327 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 44 to 187 2021-03-24T22:01:36.327 INFO:tasks.rados.rados.0.smithi125.stdout:1984: snap_create 2021-03-24T22:01:36.328 INFO:tasks.rados.rados.0.smithi125.stdout:1981: finishing write tid 2 to smithi12556470-30 2021-03-24T22:01:36.329 INFO:tasks.rados.rados.0.smithi125.stdout:1981: finishing write tid 3 to smithi12556470-30 2021-03-24T22:01:36.329 INFO:tasks.rados.rados.0.smithi125.stdout:1981: finishing write tid 4 to smithi12556470-30 2021-03-24T22:01:36.330 INFO:tasks.rados.rados.0.smithi125.stdout:1981: finishing write tid 5 to smithi12556470-30 2021-03-24T22:01:36.330 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 30 v 303 (ObjNum 698 snap 220 seq_num 698) dirty exists 2021-03-24T22:01:36.330 INFO:tasks.rados.rados.0.smithi125.stdout:1981: left oid 30 (ObjNum 698 snap 220 seq_num 698) 2021-03-24T22:01:36.330 INFO:tasks.rados.rados.0.smithi125.stdout:1983: finishing rollback tid 1 to smithi12556470-44 2021-03-24T22:01:36.331 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 44 v 388 (ObjNum -986354688 snap 32741 seq_num 1117542880) dirty dne 2021-03-24T22:01:36.416 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:36.426 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-24T22:01:36.427 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T22:01:36.462 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:36.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:36.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:01:36.550 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:36 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:01:36.550 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:36 smithi125 conmon[19936]: 01:34.884307+0000 mgr.y (mgr.14144) 534 : cluster [DBG] pgmap v949: 168 pgs: 1 active+remapped+backfilling, 5 active+recovering+undersized+remapped, 1 active+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 remapped+peering, 2 peering, 2 active+clean+wait, 4 active+remapped+backfill_toofull, 147 active+clean; 501 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 6.1 MiB/s rd, 3.7 MiB/s wr, 18 op/s; 30/1653 objects degraded (1.815%); 142/1653 objects misplaced (8.590%); 17 MiB/s, 82 objects/s recovering 2021-03-24T22:01:36.550 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:36 smithi125 conmon[19936]: cluster 2021-03-24T22:01:36.551 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:36 smithi125 conmon[19936]: 2021-03-24T22:01:35.405239+0000 mon.a (mon.0) 1503 : cluster 2021-03-24T22:01:36.551 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:36 smithi125 conmon[19936]: [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2021-03-24T22:01:36.551 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:36 smithi125 conmon[19936]: cluster 2021-03-24T22 2021-03-24T22:01:36.551 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:36 smithi125 conmon[19936]: :01:36.282209+0000 mon.a (mon.0) 2021-03-24T22:01:36.552 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:36 smithi125 conmon[19936]: 1504 : cluster [DBG] osdmap e526: 8 total, 8 up, 6 in 2021-03-24T22:01:36.552 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:36 smithi125 conmon[24927]: cluster 2021-03-24T22:01:34. 2021-03-24T22:01:36.552 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:36 smithi125 conmon[24927]: 884307+0000 mgr.y (mgr.14144) 534 : cluster [DBG] pgmap v949: 168 pgs: 1 active+remapped+backfilling, 5 active+recovering+undersized+remapped, 1 active+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 remapped+peering, 2 peering, 2 active+clean+wait, 4 active+remapped+backfill_toofull, 147 active+clean; 501 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 6.1 MiB/s rd, 3.7 MiB/s wr, 18 op/s; 30/1653 objects degraded (1.815%); 142/1653 objects misplaced (8.590%); 17 MiB/s, 82 objects/s recovering 2021-03-24T22:01:36.553 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:36 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T22:01:36.553 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:36 smithi125 conmon[24927]: T22:01:35.405239+0000 mon.a (mon.0) 1503 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2021-03-24T22:01:36.553 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:36 smithi125 conmon[24927]: cluster 2021-03-24T22:01:36. 2021-03-24T22:01:36.553 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:36 smithi125 conmon[24927]: 282209+0000 mon.a (mon.0) 1504 : cluster [DBG] osdmap e526: 8 total, 8 up, 6 in 2021-03-24T22:01:36.568 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:36.610 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:36.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:36 smithi142 conmon[20125]: cluster 2021-03-24T22:01:34.884307+0000 mgr.y (mgr.14144) 534 : 2021-03-24T22:01:36.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:36 smithi142 conmon[20125]: cluster [DBG] pgmap v949: 168 pgs: 1 active+remapped+backfilling, 5 active+recovering+undersized+remapped, 1 active+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 remapped+peering, 2 peering, 2 active+clean+wait, 4 active+remapped+backfill_toofull, 147 active+clean; 501 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 6.1 MiB/s rd, 3.7 MiB/s wr, 18 op/s; 30/1653 objects degraded (1.815%); 142/1653 objects misplaced (8.590%); 17 MiB/s, 82 objects/s recovering 2021-03-24T22:01:36.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:36 smithi142 conmon[20125]: cluster 2021-03-24T22:01:35.405239+0000 mon.a (mon.0) 1503 : cluster [INF] 2021-03-24T22:01:36.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:36 smithi142 conmon[20125]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2021-03-24T22:01:36.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:36 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:01:36.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:36 smithi142 conmon[20125]: 22:01:36.282209+0000 mon.a (mon.0) 1504 : 2021-03-24T22:01:36.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:36 smithi142 conmon[20125]: cluster [DBG] osdmap e526: 8 total, 8 up, 6 in 2021-03-24T22:01:36.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:36.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:01:36.711 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 1] out_osds: [6, 7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2021-03-24T22:01:36.712 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-24T22:01:36.712 INFO:tasks.thrashosds.thrasher:Adding osd 6 2021-03-24T22:01:36.712 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd in 6 2021-03-24T22:01:36.720 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:36.805 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:36.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:36.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:01:36.910 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:36.964 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:36.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:36.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:01:37.071 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:37.136 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:37.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:37.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:01:37.242 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:37.290 INFO:tasks.rados.rados.0.smithi125.stdout:1981: done (2 left) 2021-03-24T22:01:37.290 INFO:tasks.rados.rados.0.smithi125.stdout:1983: done (1 left) 2021-03-24T22:01:37.291 INFO:tasks.rados.rados.0.smithi125.stdout:1984: done (0 left) 2021-03-24T22:01:37.291 INFO:tasks.rados.rados.0.smithi125.stdout:1985: copy_from oid 27 from oid 4 current snap is 221 2021-03-24T22:01:37.291 INFO:tasks.rados.rados.0.smithi125.stdout:1986: write oid 16 current snap is 221 2021-03-24T22:01:37.291 INFO:tasks.rados.rados.0.smithi125.stdout:1986: seq_num 700 ranges {564900=578776,1629160=709913,3073389=492409} 2021-03-24T22:01:37.292 INFO:tasks.rados.rados.0.smithi125.stdout:1986: writing smithi12556470-16 from 564900 to 1143676 tid 1 2021-03-24T22:01:37.299 INFO:tasks.rados.rados.0.smithi125.stdout:1986: writing smithi12556470-16 from 1629160 to 2339073 tid 2 2021-03-24T22:01:37.305 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:37.308 INFO:tasks.rados.rados.0.smithi125.stdout:1986: writing smithi12556470-16 from 3073389 to 3565798 tid 3 2021-03-24T22:01:37.308 INFO:tasks.rados.rados.0.smithi125.stdout:1987: read oid 18 snap -1 2021-03-24T22:01:37.308 INFO:tasks.rados.rados.0.smithi125.stdout:1987: expect (ObjNum 588 snap 177 seq_num 588) 2021-03-24T22:01:37.308 INFO:tasks.rados.rados.0.smithi125.stdout:1988: read oid 43 snap -1 2021-03-24T22:01:37.309 INFO:tasks.rados.rados.0.smithi125.stdout:1988: expect deleted 2021-03-24T22:01:37.309 INFO:tasks.rados.rados.0.smithi125.stdout:1989: read oid 29 snap 181 2021-03-24T22:01:37.309 INFO:tasks.rados.rados.0.smithi125.stdout:1989: expect (ObjNum 582 snap 176 seq_num 582) 2021-03-24T22:01:37.310 INFO:tasks.rados.rados.0.smithi125.stdout:1986: finishing write tid 1 to smithi12556470-16 2021-03-24T22:01:37.310 INFO:tasks.rados.rados.0.smithi125.stdout:1990: read oid 15 snap -1 2021-03-24T22:01:37.310 INFO:tasks.rados.rados.0.smithi125.stdout:1990: expect deleted 2021-03-24T22:01:37.310 INFO:tasks.rados.rados.0.smithi125.stdout:1991: snap_create 2021-03-24T22:01:37.311 INFO:tasks.rados.rados.0.smithi125.stdout:1986: finishing write tid 2 to smithi12556470-16 2021-03-24T22:01:37.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:37.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:01:37.314 INFO:tasks.rados.rados.0.smithi125.stdout:1989: expect (ObjNum 582 snap 176 seq_num 582) 2021-03-24T22:01:37.413 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:37.468 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:37.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:37.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:01:37.576 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:37.577 INFO:tasks.rados.rados.0.smithi125.stdout:1986: finishing write tid 3 to smithi12556470-16 2021-03-24T22:01:37.577 INFO:tasks.rados.rados.0.smithi125.stdout:1986: finishing write tid 4 to smithi12556470-16 2021-03-24T22:01:37.578 INFO:tasks.rados.rados.0.smithi125.stdout:1986: finishing write tid 5 to smithi12556470-16 2021-03-24T22:01:37.578 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 16 v 399 (ObjNum 700 snap 221 seq_num 700) dirty exists 2021-03-24T22:01:37.578 INFO:tasks.rados.rados.0.smithi125.stdout:1986: left oid 16 (ObjNum 700 snap 221 seq_num 700) 2021-03-24T22:01:37.578 INFO:tasks.rados.rados.0.smithi125.stdout:1987: expect (ObjNum 588 snap 177 seq_num 588) 2021-03-24T22:01:37.630 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:37.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:37.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:01:37.635 INFO:journalctl@ceph.osd.0.smithi125.stdout:Mar 24 22:01:37 smithi125 conmon[28026]: debug 2021-03-24T22:01:37.410+0000 7f61b847f700 -1 osd.0 pg_epoch: 527 pg[5.b( empty local-lis/les=518/519 n=0 ec=64/64 lis/c=518/518 les/c/f=519/519/0 sis=518) [0,5,1] r=0 lpr=518 crt=0'0 mlcod 0'0 active+clean] scrubber pg(5.b) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 5 (0x563be03a7e40) 2021-03-24T22:01:37.635 INFO:journalctl@ceph.osd.0.smithi125.stdout:Mar 24 22:01:37 smithi125 conmon[28026]: 2021-03-24T22:01:37.642 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.2 dump_historic_ops 2021-03-24T22:01:37.708 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:01:37.735 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:37.797 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:37.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:37.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:01:37.903 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:37.959 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:37.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:37.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:01:38.033 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-24T22:01:38.066 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:38.113 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:38.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:38.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:01:38.220 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:38.281 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:38.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:38.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:01:38.291 INFO:tasks.rados.rados.0.smithi125.stdout:1986: done (6 left) 2021-03-24T22:01:38.291 INFO:tasks.rados.rados.0.smithi125.stdout:1987: done (5 left) 2021-03-24T22:01:38.291 INFO:tasks.rados.rados.0.smithi125.stdout:1988: done (4 left) 2021-03-24T22:01:38.292 INFO:tasks.rados.rados.0.smithi125.stdout:1989: done (3 left) 2021-03-24T22:01:38.292 INFO:tasks.rados.rados.0.smithi125.stdout:1991: done (2 left) 2021-03-24T22:01:38.292 INFO:tasks.rados.rados.0.smithi125.stdout:1992: delete oid 50 current snap is 222 2021-03-24T22:01:38.297 INFO:tasks.rados.rados.0.smithi125.stdout:1990: done (2 left) 2021-03-24T22:01:38.297 INFO:tasks.rados.rados.0.smithi125.stdout:1992: done (1 left) 2021-03-24T22:01:38.297 INFO:tasks.rados.rados.0.smithi125.stdout:1993: rollback oid 49 current snap is 222 2021-03-24T22:01:38.298 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 49 to 219 2021-03-24T22:01:38.298 INFO:tasks.rados.rados.0.smithi125.stdout:1994: delete oid 38 current snap is 222 2021-03-24T22:01:38.303 INFO:tasks.rados.rados.0.smithi125.stdout:1994: done (2 left) 2021-03-24T22:01:38.304 INFO:tasks.rados.rados.0.smithi125.stdout:1995: read oid 45 snap -1 2021-03-24T22:01:38.304 INFO:tasks.rados.rados.0.smithi125.stdout:1995: expect (ObjNum 694 snap 218 seq_num 694) 2021-03-24T22:01:38.304 INFO:tasks.rados.rados.0.smithi125.stdout:1996: write oid 30 current snap is 222 2021-03-24T22:01:38.304 INFO:tasks.rados.rados.0.smithi125.stdout:1996: seq_num 703 ranges {615321=404523,1804606=457240,2755658=726225} 2021-03-24T22:01:38.313 INFO:tasks.rados.rados.0.smithi125.stdout:1996: writing smithi12556470-30 from 615321 to 1019844 tid 1 2021-03-24T22:01:38.317 INFO:tasks.rados.rados.0.smithi125.stdout:1996: writing smithi12556470-30 from 1804606 to 2261846 tid 2 2021-03-24T22:01:38.326 INFO:tasks.rados.rados.0.smithi125.stdout:1996: writing smithi12556470-30 from 2755658 to 3481883 tid 3 2021-03-24T22:01:38.327 INFO:tasks.rados.rados.0.smithi125.stdout:1997: snap_remove snap 156 2021-03-24T22:01:38.327 INFO:tasks.rados.rados.0.smithi125.stdout:1985: finishing copy_from to smithi12556470-27 2021-03-24T22:01:38.328 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 27 v 397 (ObjNum 625 snap 192 seq_num 625) dirty exists 2021-03-24T22:01:38.391 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:38.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:38 smithi125 conmon[24927]: cluster 2021-03-24T22:01:36.885010+0000 2021-03-24T22:01:38.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:38 smithi125 conmon[24927]: mgr.y (mgr.14144) 535 : cluster [DBG] pgmap v952: 168 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+remapped, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 peering, 4 active+remapped+backfill_toofull, 155 active+clean; 508 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 14 MiB/s rd, 4.7 MiB/s wr, 25 op/s; 11/1671 objects degraded (0.658%); 112/1671 objects misplaced (6.703%); 19 MiB/s, 71 objects/s recovering 2021-03-24T22:01:38.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:38 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T22:01:38.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:38 smithi125 conmon[24927]: -24T22:01:37.276216+0000 mon.a (mon.0) 1505 : cluster [DBG] osdmap e527: 8 total, 8 up, 6 in 2021-03-24T22:01:38.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:38 smithi125 conmon[19936]: cluster 2021-03-24T22:01:38.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:38 smithi125 conmon[19936]: 2021-03-24T22 2021-03-24T22:01:38.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:38 smithi125 conmon[19936]: :01:36.885010+0000 mgr.y (mgr.14144) 535 : cluster [DBG] pgmap v952: 168 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+remapped, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 peering, 4 active+remapped+backfill_toofull, 155 active+clean; 508 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 14 MiB/s rd, 4.7 MiB/s wr, 25 op/s; 11/1671 objects degraded (0.658%); 112/1671 objects misplaced (6.703%); 19 MiB/s, 71 objects/s recovering 2021-03-24T22:01:38.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:38 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:01:38.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:38 smithi125 conmon[19936]: 01:37.276216+0000 mon.a (mon.0) 1505 2021-03-24T22:01:38.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:38 smithi125 conmon[19936]: : cluster [DBG] osdmap e527: 8 total, 8 up, 6 in 2021-03-24T22:01:38.464 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:38.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:38.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:01:38.569 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:38.575 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:01:38.584 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:38 smithi142 conmon[20125]: cluster 2021-03-24T22:01:36.885010+0000 2021-03-24T22:01:38.585 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:38 smithi142 conmon[20125]: mgr.y (mgr.14144) 535 : cluster [DBG] pgmap v952: 168 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+remapped, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 peering, 4 active+remapped+backfill_toofull, 155 active+clean; 508 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 14 MiB/s rd, 4.7 MiB/s wr, 25 op/s; 11/1671 objects degraded (0.658%); 112/1671 objects misplaced (6.703%); 19 MiB/s, 71 objects/s recovering 2021-03-24T22:01:38.585 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:38 smithi142 conmon[20125]: cluster 2021-03-24T22:01:37.276216+0000 mon.a ( 2021-03-24T22:01:38.586 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:38 smithi142 conmon[20125]: mon.0) 1505 : cluster [DBG] osdmap e527: 8 total, 8 up, 6 in 2021-03-24T22:01:38.612 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:01:38.636 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:38.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:38.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:01:38.742 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:38.791 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:38.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:38.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:01:38.895 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:38.950 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:38.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:38.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:01:39.058 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:39.112 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:39.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:39.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:01:39.219 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:39.278 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:39.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:39.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:01:39.298 INFO:tasks.rados.rados.0.smithi125.stdout:1997: done (4 left) 2021-03-24T22:01:39.298 INFO:tasks.rados.rados.0.smithi125.stdout:1998: snap_create 2021-03-24T22:01:39.298 INFO:tasks.rados.rados.0.smithi125.stdout:1985: finishing copy_from racing read to smithi12556470-27 2021-03-24T22:01:39.299 INFO:tasks.rados.rados.0.smithi125.stdout:1995: expect (ObjNum 694 snap 218 seq_num 694) 2021-03-24T22:01:39.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:39 smithi142 conmon[20125]: cluster 2021-03-24T22:01:38.278108+0000 mon.a ( 2021-03-24T22:01:39.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:39 smithi142 conmon[20125]: mon.0) 1506 : cluster [DBG] osdmap e528: 8 total, 8 up, 6 in 2021-03-24T22:01:39.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:39 smithi142 conmon[20125]: audit 2021-03-24T22:01:39.098706+0000 mon.c (mon.2) 82 : audit [DBG] from='client.? v1:172.21.15.125:0/39621527' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:01:39.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:39 smithi142 conmon[20125]: cluster 2021-03-24T22:01:39.286209+0000 mon.a (mon.0) 1507 : cluster [DBG] osdmap e529: 8 total, 8 up, 6 in 2021-03-24T22:01:39.368 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T22:01:39.368 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T22:01:39.369 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T22:01:39.369 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T22:01:39.369 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T22:01:39.369 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T22:01:39.370 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T22:01:39.370 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T22:01:39.370 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-24T22:01:39.371 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-24T22:01:39.371 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-24T22:01:39.371 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-24T22:01:39.371 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-24T22:01:39.372 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-24T22:01:39.372 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-24T22:01:39.372 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-24T22:01:39.373 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:39 smithi125 conmon[19936]: cluster 2021-03-24T22:01:38.278108+0000 mon.a ( 2021-03-24T22:01:39.373 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:39 smithi125 conmon[19936]: mon.0) 1506 : cluster [DBG] osdmap e528: 8 total, 8 up, 6 in 2021-03-24T22:01:39.373 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:39 smithi125 conmon[19936]: audit 2021-03-24T22:01:39.098706+0000 mon.c (mon.2) 82 : audit [DBG] from='client.? v1:172.21.15.125:0/39621527' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:01:39.374 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:39 smithi125 conmon[19936]: cluster 2021-03-24T22:01:39.286209+0000 mon.a (mon.0) 1507 : cluster [DBG] 2021-03-24T22:01:39.374 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:39 smithi125 conmon[19936]: osdmap e529: 8 total, 8 up, 6 in 2021-03-24T22:01:39.374 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:39 smithi125 conmon[24927]: cluster 2021-03-24T22:01:39.375 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:39 smithi125 conmon[24927]: 2021-03-24T22:01:39.375 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:39 smithi125 conmon[24927]: 2021 2021-03-24T22:01:39.375 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:39 smithi125 conmon[24927]: - 2021-03-24T22:01:39.375 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:39 smithi125 conmon[24927]: 03 2021-03-24T22:01:39.376 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:39 smithi125 conmon[24927]: - 2021-03-24T22:01:39.376 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:39 smithi125 conmon[24927]: 24 2021-03-24T22:01:39.376 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:39 smithi125 conmon[24927]: T 2021-03-24T22:01:39.376 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:39 smithi125 conmon[24927]: 22 2021-03-24T22:01:39.376 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:39 smithi125 conmon[24927]: : 2021-03-24T22:01:39.377 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:39 smithi125 conmon[24927]: 01 2021-03-24T22:01:39.377 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:39 smithi125 conmon[24927]: : 2021-03-24T22:01:39.377 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:39 smithi125 conmon[24927]: 38 2021-03-24T22:01:39.377 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:39 smithi125 conmon[24927]: . 2021-03-24T22:01:39.378 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:39 smithi125 conmon[24927]: 278108+0000 mon.a ( 2021-03-24T22:01:39.378 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:39 smithi125 conmon[24927]: mon.0) 1506 : cluster [DBG] osdmap e528: 8 total, 8 up, 6 in 2021-03-24T22:01:39.378 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:39 smithi125 conmon[24927]: audit 2021- 2021-03-24T22:01:39.378 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:39 smithi125 conmon[24927]: 03-24T22:01:39.098706+0000 mon.c (mon.2) 82 : audit [DBG] 2021-03-24T22:01:39.379 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:39 smithi125 conmon[24927]: from='client.? v1:172.21.15.125:0/39621527' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:01:39.379 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:39 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:01:39.379 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:39 smithi125 conmon[24927]: 03-24T22:01:39.286209+0000 mon.a (mon.0) 1507 : 2021-03-24T22:01:39.379 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:39 smithi125 conmon[24927]: cluster [DBG] osdmap e529: 8 total, 8 up, 6 in 2021-03-24T22:01:39.387 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:39.438 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:39.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:39.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:01:39.516 INFO:tasks.rados.rados.0.smithi125.stdout:1996: finishing write tid 1 to smithi12556470-30 2021-03-24T22:01:39.517 INFO:tasks.rados.rados.0.smithi125.stdout:1996: finishing write tid 2 to smithi12556470-30 2021-03-24T22:01:39.517 INFO:tasks.rados.rados.0.smithi125.stdout:1996: finishing write tid 3 to smithi12556470-30 2021-03-24T22:01:39.517 INFO:tasks.rados.rados.0.smithi125.stdout:1996: finishing write tid 4 to smithi12556470-30 2021-03-24T22:01:39.518 INFO:tasks.rados.rados.0.smithi125.stdout:1996: finishing write tid 5 to smithi12556470-30 2021-03-24T22:01:39.518 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 30 v 307 (ObjNum 703 snap 222 seq_num 703) dirty exists 2021-03-24T22:01:39.518 INFO:tasks.rados.rados.0.smithi125.stdout:1996: left oid 30 (ObjNum 703 snap 222 seq_num 703) 2021-03-24T22:01:39.518 INFO:tasks.rados.rados.0.smithi125.stdout:1993: finishing rollback tid 0 to smithi12556470-49 2021-03-24T22:01:39.519 INFO:tasks.rados.rados.0.smithi125.stdout:1993: finishing rollback tid 1 to smithi12556470-49 2021-03-24T22:01:39.519 INFO:tasks.rados.rados.0.smithi125.stdout:1993: finishing rollback tid 2 to smithi12556470-49 2021-03-24T22:01:39.519 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 49 v 657 (ObjNum 671 snap 208 seq_num 671) dirty exists 2021-03-24T22:01:39.545 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:39.608 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:39.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:39.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:01:39.716 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:39.777 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:39.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:39.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:01:39.885 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:39.942 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:39.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:39.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:01:40.049 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:40.107 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:40.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:40.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:01:40.212 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:40.269 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:40.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:40.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:01:40.288 INFO:tasks.rados.rados.0.smithi125.stdout:1985: done (4 left) 2021-03-24T22:01:40.288 INFO:tasks.rados.rados.0.smithi125.stdout:1993: done (3 left) 2021-03-24T22:01:40.289 INFO:tasks.rados.rados.0.smithi125.stdout:1995: done (2 left) 2021-03-24T22:01:40.289 INFO:tasks.rados.rados.0.smithi125.stdout:1996: done (1 left) 2021-03-24T22:01:40.289 INFO:tasks.rados.rados.0.smithi125.stdout:1998: done (0 left) 2021-03-24T22:01:40.289 INFO:tasks.rados.rados.0.smithi125.stdout:1999: read oid 39 snap 215 2021-03-24T22:01:40.290 INFO:tasks.rados.rados.0.smithi125.stdout:1999: expect (ObjNum 688 snap 215 seq_num 688) 2021-03-24T22:01:40.290 INFO:tasks.rados.rados.0.smithi125.stdout:2000: read oid 44 snap 215 2021-03-24T22:01:40.290 INFO:tasks.rados.rados.0.smithi125.stdout:2000: expect deleted 2021-03-24T22:01:40.290 INFO:tasks.rados.rados.0.smithi125.stdout:2001: rollback oid 6 current snap is 223 2021-03-24T22:01:40.291 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 6 to 207 2021-03-24T22:01:40.291 INFO:tasks.rados.rados.0.smithi125.stdout:2002: delete oid 28 current snap is 223 2021-03-24T22:01:40.292 INFO:tasks.rados.rados.0.smithi125.stdout:2000: done (3 left) 2021-03-24T22:01:40.292 INFO:tasks.rados.rados.0.smithi125.stdout:2002: done (2 left) 2021-03-24T22:01:40.293 INFO:tasks.rados.rados.0.smithi125.stdout:2003: rollback oid 10 current snap is 223 2021-03-24T22:01:40.293 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 10 to 219 2021-03-24T22:01:40.293 INFO:tasks.rados.rados.0.smithi125.stdout:2001: finishing rollback tid 1 to smithi12556470-6 2021-03-24T22:01:40.293 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 6 v 398 (ObjNum 1117583952 snap 21916 seq_num 0) dirty dne 2021-03-24T22:01:40.294 INFO:tasks.rados.rados.0.smithi125.stdout:2001: done (2 left) 2021-03-24T22:01:40.294 INFO:tasks.rados.rados.0.smithi125.stdout:2004: read oid 35 snap -1 2021-03-24T22:01:40.294 INFO:tasks.rados.rados.0.smithi125.stdout:2004: expect (ObjNum 639 snap 196 seq_num 639) 2021-03-24T22:01:40.294 INFO:tasks.rados.rados.0.smithi125.stdout:2005: copy_from oid 40 from oid 42 current snap is 223 2021-03-24T22:01:40.295 INFO:tasks.rados.rados.0.smithi125.stdout:2006: rollback oid 4 current snap is 223 2021-03-24T22:01:40.295 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 4 to 213 2021-03-24T22:01:40.295 INFO:tasks.rados.rados.0.smithi125.stdout:2007: copy_from oid 17 from oid 30 current snap is 223 2021-03-24T22:01:40.295 INFO:tasks.rados.rados.0.smithi125.stdout:2008: copy_from oid 5 from oid 47 current snap is 223 2021-03-24T22:01:40.296 INFO:tasks.rados.rados.0.smithi125.stdout:2009: snap_remove snap 205 2021-03-24T22:01:40.375 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:40.432 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:40.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:40.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:01:40.539 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:40.593 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:40.596 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:40 smithi125 conmon[19936]: cluster 2021-03-24T22:01:38.885605 2021-03-24T22:01:40.596 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:40 smithi125 conmon[19936]: +0000 mgr.y (mgr.14144) 536 : cluster [DBG] pgmap v955: 168 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+remapped, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 peering, 4 active+remapped+backfill_toofull, 155 active+clean; 508 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 12 MiB/s rd, 1.7 MiB/s wr, 13 op/s; 11/1671 objects degraded (0.658%); 112/1671 objects misplaced (6.703%); 11 MiB/s, 15 objects/s recovering 2021-03-24T22:01:40.596 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:40 smithi125 conmon[19936]: cluster 2021-03-24T22:01:40.267558+0000 mon.a (mon.0) 1508 : cluster [WRN] Health check update: Degraded data redundancy: 11/1671 objects degraded (0.658%), 2 pgs degraded (PG_DEGRADED) 2021-03-24T22:01:40.597 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:40 smithi125 conmon[19936]: cluster 2021-03-24T22:01:40.597 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:40 smithi125 conmon[19936]: 2021-03-24T22:01:40.277169+0000 mon.a (mon.0) 1509 : cluster [DBG] osdmap e530: 8 total, 8 up, 6 in 2021-03-24T22:01:40.597 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:40 smithi125 conmon[24927]: cluster 2021-03-24T22:01:38.885605+0000 2021-03-24T22:01:40.598 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:40 smithi125 conmon[24927]: mgr.y (mgr.14144) 536 : cluster [DBG] pgmap v955: 168 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+remapped, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 peering, 4 active+remapped+backfill_toofull, 155 active+clean; 508 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 12 MiB/s rd, 1.7 MiB/s wr, 13 op/s; 11/1671 objects degraded (0.658%); 112/1671 objects misplaced (6.703%); 11 MiB/s, 15 objects/s recovering 2021-03-24T22:01:40.598 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:40 smithi125 conmon[24927]: cluster 2021-03-24T22:01 2021-03-24T22:01:40.598 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:40 smithi125 conmon[24927]: :40.267558+0000 mon.a (mon.0) 1508 : cluster [WRN] Health check update: Degraded data redundancy: 11/1671 objects degraded (0.658%), 2 pgs degraded (PG_DEGRADED) 2021-03-24T22:01:40.599 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:40 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T22:01:40.599 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:40 smithi125 conmon[24927]: -24T22:01:40.277169+0000 mon.a (mon 2021-03-24T22:01:40.599 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:40 smithi125 conmon[24927]: .0) 1509 : cluster [DBG] osdmap e530: 8 total, 8 up, 6 in 2021-03-24T22:01:40.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:40.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:01:40.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:40 smithi142 conmon[20125]: cluster 2021-03-24T22:01:38.885605+0000 2021-03-24T22:01:40.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:40 smithi142 conmon[20125]: mgr.y (mgr.14144) 536 : cluster [DBG] pgmap v955: 168 pgs: 1 active+remapped+backfilling, 4 active+recovering+undersized+remapped, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 peering, 4 active+remapped+backfill_toofull, 155 active+clean; 508 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 12 MiB/s rd, 1.7 MiB/s wr, 13 op/s; 11/1671 objects degraded (0.658%); 112/1671 objects misplaced (6.703%); 11 MiB/s, 15 objects/s recovering 2021-03-24T22:01:40.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:40 smithi142 conmon[20125]: cluster 2021-03-24T22:01:40.267558+0000 mon.a (mon.0) 1508 : cluster 2021-03-24T22:01:40.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:40 smithi142 conmon[20125]: [WRN] Health check update: Degraded data redundancy: 11/1671 objects degraded (0.658%), 2 pgs degraded (PG_DEGRADED) 2021-03-24T22:01:40.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:40 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:01:40.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:40 smithi142 conmon[20125]: T22:01:40.277169+0000 mon.a (mon.0) 1509 : cluster [DBG] 2021-03-24T22:01:40.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:40 smithi142 conmon[20125]: osdmap e530: 8 total, 8 up, 6 in 2021-03-24T22:01:40.702 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:40.762 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:40.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:40.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:01:40.869 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:40.929 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:40.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:40.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:01:41.035 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:41.094 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:41.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:41.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:01:41.202 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:41.265 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:41.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:41.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:01:41.301 INFO:tasks.rados.rados.0.smithi125.stdout:2009: done (7 left) 2021-03-24T22:01:41.302 INFO:tasks.rados.rados.0.smithi125.stdout:2010: delete oid 31 current snap is 223 2021-03-24T22:01:41.302 INFO:tasks.rados.rados.0.smithi125.stdout:2005: finishing copy_from to smithi12556470-40 2021-03-24T22:01:41.302 INFO:tasks.rados.rados.0.smithi125.stdout:2005: got expected ENOENT (src dne) 2021-03-24T22:01:41.303 INFO:tasks.rados.rados.0.smithi125.stdout:2005: finishing copy_from racing read to smithi12556470-40 2021-03-24T22:01:41.303 INFO:tasks.rados.rados.0.smithi125.stdout:2004: expect (ObjNum 639 snap 196 seq_num 639) 2021-03-24T22:01:41.372 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:41.427 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:41.428 INFO:tasks.rados.rados.0.smithi125.stdout:2006: finishing rollback tid 0 to smithi12556470-4 2021-03-24T22:01:41.428 INFO:tasks.rados.rados.0.smithi125.stdout:2006: finishing rollback tid 1 to smithi12556470-4 2021-03-24T22:01:41.428 INFO:tasks.rados.rados.0.smithi125.stdout:2006: finishing rollback tid 2 to smithi12556470-4 2021-03-24T22:01:41.428 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 4 v 495 (ObjNum 625 snap 192 seq_num 625) dirty exists 2021-03-24T22:01:41.429 INFO:tasks.rados.rados.0.smithi125.stdout:2004: done (7 left) 2021-03-24T22:01:41.429 INFO:tasks.rados.rados.0.smithi125.stdout:2005: done (6 left) 2021-03-24T22:01:41.429 INFO:tasks.rados.rados.0.smithi125.stdout:2006: done (5 left) 2021-03-24T22:01:41.429 INFO:tasks.rados.rados.0.smithi125.stdout:2010: done (4 left) 2021-03-24T22:01:41.429 INFO:tasks.rados.rados.0.smithi125.stdout:2011: snap_remove snap 210 2021-03-24T22:01:41.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:41.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:01:41.533 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:41.547 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:41 smithi142 conmon[20125]: cluster 2021-03-24T22:01:39. 2021-03-24T22:01:41.548 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:41 smithi142 conmon[20125]: 658422+0000 osd.7 (osd.7) 95 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:01:41.548 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:41 smithi142 conmon[20125]: cluster 2021-03-24T22:01:39 2021-03-24T22:01:41.548 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:41 smithi142 conmon[20125]: .659921+0000 osd.7 (osd.7) 96 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:01:41.549 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:41 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:01:41.549 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:41 smithi142 conmon[20125]: -24T22:01:40.445388+0000 osd.0 (osd. 2021-03-24T22:01:41.549 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:41 smithi142 conmon[20125]: 0) 111 : cluster [DBG] 5.b scrub ok 2021-03-24T22:01:41.549 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:41 smithi142 conmon[20125]: cluster 2021-03-24T22:01:41.549 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:41 smithi142 conmon[20125]: 2021-03-24 2021-03-24T22:01:41.550 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:41 smithi142 conmon[20125]: T22:01:40.788768+0000 2021-03-24T22:01:41.550 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:41 smithi142 conmon[20125]: osd.1 (osd.1) 2021-03-24T22:01:41.550 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:41 smithi142 conmon[20125]: 47 : cluster [DBG] 3.19 scrub ok 2021-03-24T22:01:41.550 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:41 smithi142 conmon[20125]: 2021-03-24T22:01:41.551 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:41 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:01:41.551 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:41 smithi142 conmon[20125]: -03-24T22:01:41 2021-03-24T22:01:41.551 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:41 smithi142 conmon[20125]: .295452+0000 mon.a (mon 2021-03-24T22:01:41.551 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:41 smithi142 conmon[20125]: .0) 1510 : cluster 2021-03-24T22:01:41.551 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:41 smithi142 conmon[20125]: [DBG] osdmap e531: 8 total, 8 up, 6 in 2021-03-24T22:01:41.588 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:41.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:41.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:01:41.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:41 smithi125 conmon[24927]: cluster 2021-03-24T22:01:39.658422+0000 2021-03-24T22:01:41.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:41 smithi125 conmon[24927]: osd.7 (osd.7) 95 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:01:41.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:41 smithi125 conmon[24927]: cluster 2021-03-24T22:01:39.659921+0000 osd.7 (osd.7 2021-03-24T22:01:41.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:41 smithi125 conmon[24927]: ) 96 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:01:41.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:41 smithi125 conmon[24927]: cluster 2021-03-24T22:01:40.445388+0000 osd.0 2021-03-24T22:01:41.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:41 smithi125 conmon[24927]: (osd.0) 111 : cluster [DBG] 5.b scrub ok 2021-03-24T22:01:41.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:41 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T22:01:41.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:41 smithi125 conmon[24927]: 22:01:40.788768+0000 osd.1 (osd.1) 47 : cluster [DBG] 3.19 scrub ok 2021-03-24T22:01:41.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:41 smithi125 conmon[24927]: 2021-03-24T22:01:41.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:41 smithi125 conmon[24927]: cluster 2021-03-24T22:01:41.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:41 smithi125 conmon[24927]: 2021-03-24T22:01:41.295452+0000 mon.a (mon 2021-03-24T22:01:41.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:41 smithi125 conmon[24927]: .0) 1510 : cluster [DBG] osdmap e531: 8 total, 8 up, 6 in 2021-03-24T22:01:41.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:41 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:01:41.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:41 smithi125 conmon[19936]: 01:39.658422+0000 osd.7 (osd.7) 95 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:01:41.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:41 smithi125 conmon[19936]: cluster 2021-03-24T22:01:39.659921+0000 osd.7 (osd.7) 96 : 2021-03-24T22:01:41.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:41 smithi125 conmon[19936]: cluster [DBG] purged_snaps scrub ok 2021-03-24T22:01:41.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:41 smithi125 conmon[19936]: cluster 2021-03-24T22:01: 2021-03-24T22:01:41.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:41 smithi125 conmon[19936]: 40.445388+0000 osd.0 (osd.0) 111 : cluster [DBG] 5.b scrub ok 2021-03-24T22:01:41.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:41 smithi125 conmon[19936]: 2021-03-24T22:01:41.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:41 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:01:41.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:41 smithi125 conmon[19936]: -03-24T22:01:40.788768+0000 osd.1 (osd.1) 47 : cluster 2021-03-24T22:01:41.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:41 smithi125 conmon[19936]: [DBG] 3.19 scrub ok 2021-03-24T22:01:41.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:41 smithi125 conmon[19936]: cluster 2021-03-24T22:01:41.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:41 smithi125 conmon[19936]: 2021-03-24T22:01:41.295452+0000 mon.a (mon.0) 1510 : cluster 2021-03-24T22:01:41.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:41 smithi125 conmon[19936]: [DBG] osdmap e531: 8 total, 8 up, 6 in 2021-03-24T22:01:41.695 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:41.754 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:41.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:41.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:01:41.863 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:41.919 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:41.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:41.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:01:42.027 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:42.082 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:42.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:42.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:01:42.190 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:42.250 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:42.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:42.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:01:42.351 INFO:tasks.rados.rados.0.smithi125.stdout:2011: done (4 left) 2021-03-24T22:01:42.352 INFO:tasks.rados.rados.0.smithi125.stdout:2012: delete oid 36 current snap is 223 2021-03-24T22:01:42.354 INFO:tasks.rados.rados.0.smithi125.stdout:1999: expect (ObjNum 688 snap 215 seq_num 688) 2021-03-24T22:01:42.356 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:42.388 INFO:teuthology.orchestra.run.smithi125.stderr:marked in osd.6. 2021-03-24T22:01:42.406 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:42.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:42.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:01:42.510 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:42.524 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:42 smithi142 conmon[20125]: cluster 2021-03-24T22:01:40.886380+0000 mgr.y (mgr.14144) 537 : cluster 2021-03-24T22:01:42.524 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:42 smithi142 conmon[20125]: [DBG] pgmap v958: 168 pgs: 1 peering, 4 active+remapped+backfill_toofull, 163 active+clean; 519 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 11 MiB/s rd, 2.3 MiB/s wr, 15 op/s; 63/1695 objects misplaced (3.717%); 21 MiB/s, 25 objects/s recovering 2021-03-24T22:01:42.525 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:42 smithi142 conmon[20125]: cluster 2021-03-24T22:01:41.320964+0000 mon.a (mon. 2021-03-24T22:01:42.525 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:42 smithi142 conmon[20125]: 0) 1511 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 11/1671 objects degraded (0.658%), 2 pgs degraded) 2021-03-24T22:01:42.525 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:42 smithi142 conmon[20125]: audit 2021-03-24T22:01:42.525 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:42 smithi142 conmon[20125]: 2021-03-24T22:01:42.278903+0000 mon.c 2021-03-24T22:01:42.526 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:42 smithi142 conmon[20125]: (mon.2) 83 : audit [DBG] from='client.? v1:172.21.15.125:0/2400120195' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2021-03-24T22:01:42.526 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:42 smithi142 conmon[20125]: 2021-03-24T22:01:42.526 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:42 smithi142 conmon[20125]: audit 2021-03-24T22:01:42.526 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:42 smithi142 conmon[20125]: 2021-03-24T22:01:42.279549+0000 mon.a ( 2021-03-24T22:01:42.527 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:42 smithi142 conmon[20125]: mon.0) 1512 : audit [DBG] from='client.35393 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2021-03-24T22:01:42.568 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:42.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:42.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:01:42.576 INFO:tasks.rados.rados.0.smithi125.stdout:2003: finishing rollback tid 0 to smithi12556470-10 2021-03-24T22:01:42.576 INFO:tasks.rados.rados.0.smithi125.stdout:2003: finishing rollback tid 1 to smithi12556470-10 2021-03-24T22:01:42.577 INFO:tasks.rados.rados.0.smithi125.stdout:2003: finishing rollback tid 2 to smithi12556470-10 2021-03-24T22:01:42.577 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 10 v 698 (ObjNum 642 snap 196 seq_num 642) dirty exists 2021-03-24T22:01:42.578 INFO:tasks.rados.rados.0.smithi125.stdout:2008: finishing copy_from to smithi12556470-5 2021-03-24T22:01:42.578 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 5 v 346 (ObjNum 466 snap 131 seq_num 466) dirty exists 2021-03-24T22:01:42.579 INFO:tasks.rados.rados.0.smithi125.stdout:2008: finishing copy_from racing read to smithi12556470-5 2021-03-24T22:01:42.579 INFO:tasks.rados.rados.0.smithi125.stdout:2007: finishing copy_from to smithi12556470-17 2021-03-24T22:01:42.579 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 17 v 307 (ObjNum 703 snap 222 seq_num 703) dirty exists 2021-03-24T22:01:42.580 INFO:tasks.rados.rados.0.smithi125.stdout:2007: finishing copy_from racing read to smithi12556470-17 2021-03-24T22:01:42.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:42 smithi125 conmon[24927]: cluster 2021-03-24T22 2021-03-24T22:01:42.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:42 smithi125 conmon[24927]: :01:40.886380+0000 mgr.y (mgr.14144) 537 : cluster [DBG] pgmap v958: 168 pgs: 1 peering, 4 active+remapped+backfill_toofull, 163 active+clean; 519 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 11 MiB/s rd, 2.3 MiB/s wr, 15 op/s; 63/1695 objects misplaced (3.717%); 21 MiB/s, 25 objects/s recovering 2021-03-24T22:01:42.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:42 smithi125 conmon[24927]: cluster 2021-03-24T22:01:42.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:42 smithi125 conmon[24927]: 2021-03-24T22:01:41.320964+0000 2021-03-24T22:01:42.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:42 smithi125 conmon[24927]: mon.a (mon.0) 1511 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 11/1671 objects degraded (0.658%), 2 pgs degraded) 2021-03-24T22:01:42.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:42 smithi125 conmon[24927]: 2021-03-24T22:01:42.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:42 smithi125 conmon[24927]: audit 2021-03-24T22:01:42.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:42 smithi125 conmon[24927]: 2021-03-24T22:01:42. 2021-03-24T22:01:42.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:42 smithi125 conmon[24927]: 278903+0000 mon.c (mon.2) 83 : audit 2021-03-24T22:01:42.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:42 smithi125 conmon[24927]: [DBG] from='client.? v1:172.21.15.125:0/2400120195' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2021-03-24T22:01:42.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:42 smithi125 conmon[24927]: audit 2021-03-24T22:01:42.279549 2021-03-24T22:01:42.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:42 smithi125 conmon[24927]: +0000 mon.a (mon.0) 1512 : audit [DBG] from='client.35393 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2021-03-24T22:01:42.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:42 smithi125 conmon[19936]: debug 2021-03-24T22:01:42.274+0000 7f2118128700 -1 mon.a@0(leader).osd e531 definitely_dead 0 2021-03-24T22:01:42.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:42 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:01:42.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:42 smithi125 conmon[19936]: 01:40.886380+0000 mgr.y (mgr.14144) 537 : cluster [DBG] pgmap v958: 168 pgs: 1 peering, 4 active+remapped+backfill_toofull, 163 active+clean; 519 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 11 MiB/s rd, 2.3 MiB/s wr, 15 op/s; 63/1695 objects misplaced (3.717%); 21 MiB/s, 25 objects/s recovering 2021-03-24T22:01:42.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:42 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:01:42.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:42 smithi125 conmon[19936]: 24T22:01:41.320964+0000 mon.a (mon 2021-03-24T22:01:42.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:42 smithi125 conmon[19936]: .0) 1511 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 11/1671 objects degraded (0.658%), 2 pgs degraded) 2021-03-24T22:01:42.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:42 smithi125 conmon[19936]: audit 2021-03-24T22:01:42.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:42 smithi125 conmon[19936]: 2021-03-24T22:01:42.278903+0000 2021-03-24T22:01:42.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:42 smithi125 conmon[19936]: mon.c (mon.2) 83 : audit [DBG] from='client.? v1:172.21.15.125:0/2400120195' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2021-03-24T22:01:42.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:42 smithi125 conmon[19936]: 2021-03-24T22:01:42.669 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:42 smithi125 conmon[19936]: audit 2021-03-24T22:01:42. 2021-03-24T22:01:42.669 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:42 smithi125 conmon[19936]: 279549+0000 mon.a (mon.0) 1512 : audit 2021-03-24T22:01:42.669 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:42 smithi125 conmon[19936]: [DBG] from='client.35393 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2021-03-24T22:01:42.676 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:42.734 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:42.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:42.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:01:42.840 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:42.897 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:42.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:42.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:01:43.005 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:43.058 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:43.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:43.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:01:43.165 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:43.221 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:43.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:43.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:01:43.327 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:43.400 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:43.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:43.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:01:43.472 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-24T22:01:43.506 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:43.560 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:43.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:43.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:01:43.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:43 smithi142 conmon[20125]: cluster 2021-03-24T22:01:41.968047+0000 osd.5 (osd.5) 19 : cluster [DBG] 6.19 scrub ok 2021-03-24T22:01:43.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:43 smithi142 conmon[20125]: audit 2021-03-24T22:01:42.348249+0000 mon.a (mon.0) 1513 : audit [INF] from='client.35393 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2021-03-24T22:01:43.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:43 smithi142 conmon[20125]: cluster 2021-03-24T22:01:42.348397+0000 mon.a (mon.0) 1514 : cluster [DBG] osdmap e532: 8 total, 8 up, 7 in 2021-03-24T22:01:43.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:01:43.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[19936]: 01:41.968047+0000 osd.5 (osd.5) 19 : cluster [DBG] 6.19 scrub ok 2021-03-24T22:01:43.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[19936]: audit 2021-03-24T22:01:43.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[19936]: 2021-03-24T22:01:42.348249+0000 mon.a (mon.0) 1513 : 2021-03-24T22:01:43.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[19936]: audit [INF] from='client.35393 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2021-03-24T22:01:43.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[19936]: cluster 2021-03-24T22:01:43.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[19936]: 2021-03-24T22:01 2021-03-24T22:01:43.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[19936]: :42.348397+0000 mon.a (mon.0) 2021-03-24T22:01:43.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[19936]: 1514 : cluster [DBG] osdmap e532: 8 total, 8 up, 7 in 2021-03-24T22:01:43.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: cluster 2021-03-24T22:01:43.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: 2021-03-24T22:01:43.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: 2021 2021-03-24T22:01:43.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: - 2021-03-24T22:01:43.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: 03 2021-03-24T22:01:43.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: - 2021-03-24T22:01:43.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: 24 2021-03-24T22:01:43.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: T 2021-03-24T22:01:43.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: 22 2021-03-24T22:01:43.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: : 2021-03-24T22:01:43.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: 01 2021-03-24T22:01:43.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: : 2021-03-24T22:01:43.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: 41 2021-03-24T22:01:43.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: . 2021-03-24T22:01:43.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: 968047 2021-03-24T22:01:43.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: +0000 2021-03-24T22:01:43.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: 2021-03-24T22:01:43.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: osd.5 2021-03-24T22:01:43.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: ( 2021-03-24T22:01:43.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: osd 2021-03-24T22:01:43.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: . 2021-03-24T22:01:43.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: 5 2021-03-24T22:01:43.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: ) 2021-03-24T22:01:43.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: 19 2021-03-24T22:01:43.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: : 2021-03-24T22:01:43.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: cluster 2021-03-24T22:01:43.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: 2021-03-24T22:01:43.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: [DBG] 2021-03-24T22:01:43.672 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: 2021-03-24T22:01:43.672 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: 6.19 scrub ok 2021-03-24T22:01:43.672 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: 2021-03-24T22:01:43.673 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: audit 2021-03-24T22:01:43.673 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: 2021-03-24T22:01:43.673 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: 2021 2021-03-24T22:01:43.673 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: - 2021-03-24T22:01:43.674 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: 03 2021-03-24T22:01:43.674 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: - 2021-03-24T22:01:43.674 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: 24 2021-03-24T22:01:43.675 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: T 2021-03-24T22:01:43.675 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: 22 2021-03-24T22:01:43.675 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: : 2021-03-24T22:01:43.675 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: 01 2021-03-24T22:01:43.676 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: : 2021-03-24T22:01:43.676 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: 42 2021-03-24T22:01:43.676 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: . 2021-03-24T22:01:43.676 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: 348249 2021-03-24T22:01:43.677 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: +0000 2021-03-24T22:01:43.677 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: 2021-03-24T22:01:43.677 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: mon.a 2021-03-24T22:01:43.677 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: ( 2021-03-24T22:01:43.678 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: mon 2021-03-24T22:01:43.678 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: . 2021-03-24T22:01:43.678 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: 0 2021-03-24T22:01:43.679 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: ) 2021-03-24T22:01:43.679 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: 1513 2021-03-24T22:01:43.679 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: : 2021-03-24T22:01:43.679 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: audit 2021-03-24T22:01:43.680 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: 2021-03-24T22:01:43.680 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: [INF] 2021-03-24T22:01:43.680 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: 2021-03-24T22:01:43.680 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: from='client.35393 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2021-03-24T22:01:43.681 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: 2021-03-24T22:01:43.681 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: cluster 2021-03-24T22:01:43.681 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: 2021-03-24T22:01:43.681 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: 2021 2021-03-24T22:01:43.682 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: - 2021-03-24T22:01:43.682 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: 03 2021-03-24T22:01:43.682 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: - 2021-03-24T22:01:43.682 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: 24 2021-03-24T22:01:43.683 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: T 2021-03-24T22:01:43.683 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: 22 2021-03-24T22:01:43.683 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: : 2021-03-24T22:01:43.684 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: 01 2021-03-24T22:01:43.684 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: : 2021-03-24T22:01:43.684 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: 42 2021-03-24T22:01:43.685 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: . 2021-03-24T22:01:43.685 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: 348397 2021-03-24T22:01:43.685 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: +0000 2021-03-24T22:01:43.686 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: 2021-03-24T22:01:43.686 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: mon.a 2021-03-24T22:01:43.686 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: ( 2021-03-24T22:01:43.687 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: mon 2021-03-24T22:01:43.687 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: . 2021-03-24T22:01:43.687 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: 0 2021-03-24T22:01:43.688 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: ) 2021-03-24T22:01:43.688 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: 1514 2021-03-24T22:01:43.688 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: : 2021-03-24T22:01:43.689 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: cluster 2021-03-24T22:01:43.689 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: 2021-03-24T22:01:43.689 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: [DBG] 2021-03-24T22:01:43.689 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: 2021-03-24T22:01:43.690 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: osdmap e532: 8 total, 8 up, 7 in 2021-03-24T22:01:43.690 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:43 smithi125 conmon[24927]: 2021-03-24T22:01:43.690 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:43.738 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:43.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:43.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:01:43.843 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:43.897 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:43.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:43.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:01:44.003 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:44.055 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:44.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:44.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:01:44.160 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:44.221 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:44.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:44.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:01:44.326 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:44.390 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:44.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:44.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:01:44.491 INFO:tasks.rados.rados.0.smithi125.stdout:1999: done (4 left) 2021-03-24T22:01:44.492 INFO:tasks.rados.rados.0.smithi125.stdout:2003: done (3 left) 2021-03-24T22:01:44.492 INFO:tasks.rados.rados.0.smithi125.stdout:2007: done (2 left) 2021-03-24T22:01:44.492 INFO:tasks.rados.rados.0.smithi125.stdout:2008: done (1 left) 2021-03-24T22:01:44.493 INFO:tasks.rados.rados.0.smithi125.stdout:2012: done (0 left) 2021-03-24T22:01:44.493 INFO:tasks.rados.rados.0.smithi125.stdout:2013: write oid 20 current snap is 223 2021-03-24T22:01:44.493 INFO:tasks.rados.rados.0.smithi125.stdout:2013: seq_num 707 ranges {682549=705519,1922181=464578,3148997=487666} 2021-03-24T22:01:44.493 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:44.502 INFO:tasks.rados.rados.0.smithi125.stdout:2013: writing smithi12556470-20 from 682549 to 1388068 tid 1 2021-03-24T22:01:44.510 INFO:tasks.rados.rados.0.smithi125.stdout:2013: writing smithi12556470-20 from 1922181 to 2386759 tid 2 2021-03-24T22:01:44.512 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:44 smithi125 conmon[19936]: cluster 2021-03-24T22:01 2021-03-24T22:01:44.512 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:44 smithi125 conmon[19936]: :42.886998+0000 mgr.y (mgr.14144) 538 : cluster [DBG] pgmap v961: 168 pgs: 1 peering, 4 active+remapped+backfill_toofull, 163 active+clean; 523 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 19 MiB/s rd, 4.1 MiB/s wr, 23 op/s; 63/1704 objects misplaced (3.697%); 21 MiB/s, 25 objects/s recovering 2021-03-24T22:01:44.513 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:44 smithi125 conmon[19936]: cluster 2021-03-24T22:01:44.513 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:44 smithi125 conmon[19936]: 2021-03-24T22:01:43.359302+0000 mon.a (mon.0) 1515 : cluster 2021-03-24T22:01:44.513 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:44 smithi125 conmon[19936]: [DBG] osdmap e533: 8 total, 8 up, 7 in 2021-03-24T22:01:44.513 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:44 smithi125 conmon[24927]: cluster 2021-03-24T22:01:44.514 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:44 smithi125 conmon[24927]: 2021-03-24T22:01:44.514 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:44 smithi125 conmon[24927]: 2021 2021-03-24T22:01:44.514 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:44 smithi125 conmon[24927]: - 2021-03-24T22:01:44.515 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:44 smithi125 conmon[24927]: 03 2021-03-24T22:01:44.515 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:44 smithi125 conmon[24927]: - 2021-03-24T22:01:44.515 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:44 smithi125 conmon[24927]: 24 2021-03-24T22:01:44.515 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:44 smithi125 conmon[24927]: T 2021-03-24T22:01:44.516 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:44 smithi125 conmon[24927]: 22 2021-03-24T22:01:44.516 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:44 smithi125 conmon[24927]: : 2021-03-24T22:01:44.516 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:44 smithi125 conmon[24927]: 01:42.886998+0000 mgr.y (mgr 2021-03-24T22:01:44.516 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:44 smithi125 conmon[24927]: .14144) 538 : cluster [DBG] pgmap v961: 168 pgs: 1 peering, 4 active+remapped+backfill_toofull, 163 active+clean; 523 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 19 MiB/s rd, 4.1 MiB/s wr, 23 op/s; 63/1704 objects misplaced (3.697%); 21 MiB/s, 25 objects/s recovering 2021-03-24T22:01:44.516 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:44 smithi125 conmon[24927]: cluster 2021-03-24T22:01 2021-03-24T22:01:44.517 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:44 smithi125 conmon[24927]: :43.359302+0000 mon.a (mon.0) 1515 2021-03-24T22:01:44.517 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:44 smithi125 conmon[24927]: : cluster [DBG] osdmap e533: 8 total, 8 up, 7 in 2021-03-24T22:01:44.518 INFO:tasks.rados.rados.0.smithi125.stdout:2013: writing smithi12556470-20 from 3148997 to 3636663 tid 3 2021-03-24T22:01:44.518 INFO:tasks.rados.rados.0.smithi125.stdout:2013: finishing write tid 1 to smithi12556470-20 2021-03-24T22:01:44.519 INFO:tasks.rados.rados.0.smithi125.stdout:2013: finishing write tid 2 to smithi12556470-20 2021-03-24T22:01:44.519 INFO:tasks.rados.rados.0.smithi125.stdout:2014: read oid 42 snap -1 2021-03-24T22:01:44.519 INFO:tasks.rados.rados.0.smithi125.stdout:2014: expect deleted 2021-03-24T22:01:44.519 INFO:tasks.rados.rados.0.smithi125.stdout:2015: rollback oid 4 current snap is 223 2021-03-24T22:01:44.520 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 4 to 220 2021-03-24T22:01:44.520 INFO:tasks.rados.rados.0.smithi125.stdout:2016: rollback oid 10 current snap is 223 2021-03-24T22:01:44.520 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 10 to 213 2021-03-24T22:01:44.520 INFO:tasks.rados.rados.0.smithi125.stdout:2017: read oid 49 snap -1 2021-03-24T22:01:44.520 INFO:tasks.rados.rados.0.smithi125.stdout:2017: expect (ObjNum 671 snap 208 seq_num 671) 2021-03-24T22:01:44.523 INFO:tasks.rados.rados.0.smithi125.stdout:2014: done (4 left) 2021-03-24T22:01:44.523 INFO:tasks.rados.rados.0.smithi125.stdout:2018: copy_from oid 14 from oid 27 current snap is 223 2021-03-24T22:01:44.523 INFO:tasks.rados.rados.0.smithi125.stdout:2019: delete oid 5 current snap is 223 2021-03-24T22:01:44.526 INFO:tasks.rados.rados.0.smithi125.stdout:2015: finishing rollback tid 0 to smithi12556470-4 2021-03-24T22:01:44.526 INFO:tasks.rados.rados.0.smithi125.stdout:2015: finishing rollback tid 1 to smithi12556470-4 2021-03-24T22:01:44.527 INFO:tasks.rados.rados.0.smithi125.stdout:2013: finishing write tid 3 to smithi12556470-20 2021-03-24T22:01:44.527 INFO:tasks.rados.rados.0.smithi125.stdout:2013: finishing write tid 4 to smithi12556470-20 2021-03-24T22:01:44.527 INFO:tasks.rados.rados.0.smithi125.stdout:2013: finishing write tid 5 to smithi12556470-20 2021-03-24T22:01:44.527 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 20 v 527 (ObjNum 707 snap 223 seq_num 707) dirty exists 2021-03-24T22:01:44.527 INFO:tasks.rados.rados.0.smithi125.stdout:2013: left oid 20 (ObjNum 707 snap 223 seq_num 707) 2021-03-24T22:01:44.528 INFO:tasks.rados.rados.0.smithi125.stdout:2016: finishing rollback tid 0 to smithi12556470-10 2021-03-24T22:01:44.529 INFO:tasks.rados.rados.0.smithi125.stdout:2016: finishing rollback tid 1 to smithi12556470-10 2021-03-24T22:01:44.529 INFO:tasks.rados.rados.0.smithi125.stdout:2015: finishing rollback tid 2 to smithi12556470-4 2021-03-24T22:01:44.529 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 4 v 498 (ObjNum 625 snap 192 seq_num 625) dirty exists 2021-03-24T22:01:44.529 INFO:tasks.rados.rados.0.smithi125.stdout:2016: finishing rollback tid 2 to smithi12556470-10 2021-03-24T22:01:44.530 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 10 v 709 (ObjNum 642 snap 196 seq_num 642) dirty exists 2021-03-24T22:01:44.530 INFO:tasks.rados.rados.0.smithi125.stdout:2013: done (5 left) 2021-03-24T22:01:44.530 INFO:tasks.rados.rados.0.smithi125.stdout:2015: done (4 left) 2021-03-24T22:01:44.530 INFO:tasks.rados.rados.0.smithi125.stdout:2016: done (3 left) 2021-03-24T22:01:44.531 INFO:tasks.rados.rados.0.smithi125.stdout:2019: done (2 left) 2021-03-24T22:01:44.531 INFO:tasks.rados.rados.0.smithi125.stdout:2020: write oid 39 current snap is 223 2021-03-24T22:01:44.531 INFO:tasks.rados.rados.0.smithi125.stdout:2020: seq_num 709 ranges {716163=572370,2039145=468247,3203843=626563} 2021-03-24T22:01:44.536 INFO:tasks.rados.rados.0.smithi125.stdout:2020: writing smithi12556470-39 from 716163 to 1288533 tid 1 2021-03-24T22:01:44.543 INFO:tasks.rados.rados.0.smithi125.stdout:2020: writing smithi12556470-39 from 2039145 to 2507392 tid 2 2021-03-24T22:01:44.551 INFO:tasks.rados.rados.0.smithi125.stdout:2020: writing smithi12556470-39 from 3203843 to 3830406 tid 3 2021-03-24T22:01:44.551 INFO:tasks.rados.rados.0.smithi125.stdout:2017: expect (ObjNum 671 snap 208 seq_num 671) 2021-03-24T22:01:44.569 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:44.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:44.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:01:44.673 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:44.725 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:44.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:44.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:01:44.730 INFO:tasks.rados.rados.0.smithi125.stdout:2020: finishing write tid 1 to smithi12556470-39 2021-03-24T22:01:44.731 INFO:tasks.rados.rados.0.smithi125.stdout:2018: finishing copy_from to smithi12556470-14 2021-03-24T22:01:44.731 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 14 v 397 (ObjNum 625 snap 192 seq_num 625) dirty exists 2021-03-24T22:01:44.731 INFO:tasks.rados.rados.0.smithi125.stdout:2018: finishing copy_from racing read to smithi12556470-14 2021-03-24T22:01:44.731 INFO:tasks.rados.rados.0.smithi125.stdout:2020: finishing write tid 2 to smithi12556470-39 2021-03-24T22:01:44.731 INFO:tasks.rados.rados.0.smithi125.stdout:2017: done (2 left) 2021-03-24T22:01:44.732 INFO:tasks.rados.rados.0.smithi125.stdout:2018: done (1 left) 2021-03-24T22:01:44.732 INFO:tasks.rados.rados.0.smithi125.stdout:2021: copy_from oid 20 from oid 10 current snap is 223 2021-03-24T22:01:44.732 INFO:tasks.rados.rados.0.smithi125.stdout:2020: finishing write tid 3 to smithi12556470-39 2021-03-24T22:01:44.732 INFO:tasks.rados.rados.0.smithi125.stdout:2020: finishing write tid 4 to smithi12556470-39 2021-03-24T22:01:44.733 INFO:tasks.rados.rados.0.smithi125.stdout:2020: finishing write tid 5 to smithi12556470-39 2021-03-24T22:01:44.733 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 39 v 714 (ObjNum 709 snap 223 seq_num 709) dirty exists 2021-03-24T22:01:44.733 INFO:tasks.rados.rados.0.smithi125.stdout:2020: left oid 39 (ObjNum 709 snap 223 seq_num 709) 2021-03-24T22:01:44.733 INFO:tasks.rados.rados.0.smithi125.stdout:2020: done (1 left) 2021-03-24T22:01:44.734 INFO:tasks.rados.rados.0.smithi125.stdout:2022: write oid 50 current snap is 223 2021-03-24T22:01:44.734 INFO:tasks.rados.rados.0.smithi125.stdout:2022: seq_num 710 ranges {732970=647619,2097627=611905,3231266=437835} 2021-03-24T22:01:44.739 INFO:tasks.rados.rados.0.smithi125.stdout:2022: writing smithi12556470-50 from 732970 to 1380589 tid 1 2021-03-24T22:01:44.748 INFO:tasks.rados.rados.0.smithi125.stdout:2022: writing smithi12556470-50 from 2097627 to 2709532 tid 2 2021-03-24T22:01:44.754 INFO:tasks.rados.rados.0.smithi125.stdout:2022: writing smithi12556470-50 from 3231266 to 3669101 tid 3 2021-03-24T22:01:44.754 INFO:tasks.rados.rados.0.smithi125.stdout:2023: snap_create 2021-03-24T22:01:44.754 INFO:tasks.rados.rados.0.smithi125.stdout:2021: finishing copy_from to smithi12556470-20 2021-03-24T22:01:44.755 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 20 v 709 (ObjNum 642 snap 196 seq_num 642) dirty exists 2021-03-24T22:01:44.755 INFO:tasks.rados.rados.0.smithi125.stdout:2021: finishing copy_from racing read to smithi12556470-20 2021-03-24T22:01:44.755 INFO:tasks.rados.rados.0.smithi125.stdout:2022: finishing write tid 1 to smithi12556470-50 2021-03-24T22:01:44.755 INFO:tasks.rados.rados.0.smithi125.stdout:2022: finishing write tid 2 to smithi12556470-50 2021-03-24T22:01:44.757 INFO:tasks.rados.rados.0.smithi125.stdout:2022: finishing write tid 3 to smithi12556470-50 2021-03-24T22:01:44.757 INFO:tasks.rados.rados.0.smithi125.stdout:2022: finishing write tid 4 to smithi12556470-50 2021-03-24T22:01:44.757 INFO:tasks.rados.rados.0.smithi125.stdout:2022: finishing write tid 5 to smithi12556470-50 2021-03-24T22:01:44.757 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 50 v 713 (ObjNum 710 snap 223 seq_num 710) dirty exists 2021-03-24T22:01:44.758 INFO:tasks.rados.rados.0.smithi125.stdout:2022: left oid 50 (ObjNum 710 snap 223 seq_num 710) 2021-03-24T22:01:44.829 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:44.840 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:44 smithi142 conmon[20125]: cluster 2021-03-24T22:01:42.886998+0000 mgr.y (mgr.14144 2021-03-24T22:01:44.841 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:44 smithi142 conmon[20125]: ) 538 : cluster [DBG] pgmap v961: 168 pgs: 1 peering, 4 active+remapped+backfill_toofull, 163 active+clean; 523 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 19 MiB/s rd, 4.1 MiB/s wr, 23 op/s; 63/1704 objects misplaced (3.697%); 21 MiB/s, 25 objects/s recovering 2021-03-24T22:01:44.841 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:44 smithi142 conmon[20125]: cluster 2021-03-24T22:01:43.359302+0000 mon.a (mon.0) 1515 : cluster [DBG] osdmap e533: 8 total, 8 up, 7 in 2021-03-24T22:01:44.871 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:44.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:44.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:01:44.975 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:45.025 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:45.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:45.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:01:45.132 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:45.186 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:45.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:45.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:01:45.286 INFO:tasks.rados.rados.0.smithi125.stdout:2021: done (2 left) 2021-03-24T22:01:45.286 INFO:tasks.rados.rados.0.smithi125.stdout:2022: done (1 left) 2021-03-24T22:01:45.287 INFO:tasks.rados.rados.0.smithi125.stdout:2023: done (0 left) 2021-03-24T22:01:45.287 INFO:tasks.rados.rados.0.smithi125.stdout:2024: rollback oid 17 current snap is 224 2021-03-24T22:01:45.287 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 17 to 208 2021-03-24T22:01:45.288 INFO:tasks.rados.rados.0.smithi125.stdout:2025: rollback oid 32 current snap is 224 2021-03-24T22:01:45.288 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 32 to 215 2021-03-24T22:01:45.288 INFO:tasks.rados.rados.0.smithi125.stdout:2026: snap_remove snap 200 2021-03-24T22:01:45.290 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:45.371 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:45.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:45.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:01:45.434 INFO:tasks.thrashosds.thrasher:Added osd 6 2021-03-24T22:01:45.471 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:01:45.480 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:45.531 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:45.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:45.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:01:45.536 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:45 smithi125 conmon[19936]: cluster 2021-03-24T22:01:45.536 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:45 smithi125 conmon[19936]: 2021-03-24T22:01:45.537 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:45 smithi125 conmon[19936]: 2021 2021-03-24T22:01:45.537 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:45 smithi125 conmon[19936]: - 2021-03-24T22:01:45.537 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:45 smithi125 conmon[19936]: 03 2021-03-24T22:01:45.537 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:45 smithi125 conmon[19936]: - 2021-03-24T22:01:45.538 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:45 smithi125 conmon[19936]: 24 2021-03-24T22:01:45.538 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:45 smithi125 conmon[19936]: T 2021-03-24T22:01:45.538 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:45 smithi125 conmon[19936]: 22 2021-03-24T22:01:45.538 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:45 smithi125 conmon[19936]: : 2021-03-24T22:01:45.539 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:45 smithi125 conmon[19936]: 01 2021-03-24T22:01:45.539 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:45 smithi125 conmon[19936]: : 2021-03-24T22:01:45.539 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:45 smithi125 conmon[19936]: 43 2021-03-24T22:01:45.539 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:45 smithi125 conmon[19936]: . 2021-03-24T22:01:45.540 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:45 smithi125 conmon[19936]: 363900 2021-03-24T22:01:45.540 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:45 smithi125 conmon[19936]: +0000 2021-03-24T22:01:45.540 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:45 smithi125 conmon[19936]: 2021-03-24T22:01:45.540 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:45 smithi125 conmon[19936]: osd.3 2021-03-24T22:01:45.540 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:45 smithi125 conmon[19936]: (osd.3) 112 : cluster [INF] 4.e continuing backfill to osd.1 from (89'27,89'27] MIN to 89'27 2021-03-24T22:01:45.541 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:45 smithi125 conmon[19936]: cluster 2021-03-24T22:01:43.363991 2021-03-24T22:01:45.541 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:45 smithi125 conmon[19936]: +0000 osd.3 (osd.3) 113 : cluster [DBG] 4.e starting backfill to osd.6 from (0'0,0'0] MAX to 89'27 2021-03-24T22:01:45.541 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:45 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T22:01:45.541 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:45 smithi125 conmon[19936]: -24T22:01:44.355870+0000 mon.a ( 2021-03-24T22:01:45.542 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:45 smithi125 conmon[19936]: mon.0) 1516 : cluster [DBG] osdmap e534: 8 total, 8 up, 7 in 2021-03-24T22:01:45.542 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:45 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T22:01:45.542 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:45 smithi125 conmon[19936]: -24T22:01:44.360096+0000 osd.0 ( 2021-03-24T22:01:45.542 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:45 smithi125 conmon[19936]: osd.0) 112 : cluster [DBG] 4.a starting backfill to osd.6 from (0'0,0'0] MAX to 69'23 2021-03-24T22:01:45.543 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:45 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:01:45.543 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:45 smithi125 conmon[19936]: T22:01:44.362479+0000 osd.5 (osd.5 2021-03-24T22:01:45.544 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:45 smithi125 conmon[19936]: ) 20 : cluster [DBG] 4.8 starting backfill to osd.6 from (0'0,0'0] MAX to 97'28 2021-03-24T22:01:45.544 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:45 smithi125 conmon[19936]: cluster 2021-03-24T22 2021-03-24T22:01:45.544 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:45 smithi125 conmon[19936]: :01:44.369943+0000 osd.3 (osd.3) 2021-03-24T22:01:45.544 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:45 smithi125 conmon[19936]: 114 : cluster [DBG] 4.c starting backfill to osd.6 from (0'0,0'0] MAX to 69'15 2021-03-24T22:01:45.544 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:45 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:01:45.545 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:45 smithi125 conmon[19936]: 24T22:01:45.277668+0000 mon.a (mon. 2021-03-24T22:01:45.545 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:45 smithi125 conmon[19936]: 0) 1517 : cluster [DBG] osdmap e535: 8 total, 8 up, 7 in 2021-03-24T22:01:45.546 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:45 smithi125 conmon[24927]: cluster 2021-03-24T22:01:43.363900+0000 osd.3 (osd.3) 112 : cluster [INF] 4.e continuing backfill to osd.1 from (89'27,89'27] MIN to 89'27 2021-03-24T22:01:45.546 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:45 smithi125 conmon[24927]: cluster 2021-03-24T22:01:43.363991+0000 osd.3 (osd.3) 113 : cluster [DBG] 4.e starting backfill to osd.6 from (0'0,0'0] MAX to 89'27 2021-03-24T22:01:45.546 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:45 smithi125 conmon[24927]: cluster 2021-03-24T22:01:44.355870+0000 mon.a (mon.0) 1516 : cluster [DBG] osdmap e534: 8 total, 8 up, 7 in 2021-03-24T22:01:45.546 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:45 smithi125 conmon[24927]: cluster 2021-03-24T22:01:44.360096+0000 osd.0 (osd.0) 112 : cluster [DBG] 4.a starting backfill to osd.6 from (0'0,0'0] MAX to 69'23 2021-03-24T22:01:45.547 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:45 smithi125 conmon[24927]: cluster 2021-03-24T22:01:44.362479+0000 osd.5 (osd.5) 20 : cluster [DBG] 4.8 starting backfill to osd.6 from (0'0,0'0] MAX to 97'28 2021-03-24T22:01:45.547 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:45 smithi125 conmon[24927]: cluster 2021-03-24T22:01:44.369943+0000 osd.3 (osd.3) 114 : cluster [DBG] 4.c starting backfill to osd.6 from (0'0,0'0] MAX to 69'15 2021-03-24T22:01:45.547 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:45 smithi125 conmon[24927]: cluster 2021-03-24T22:01:45.277668+0000 mon.a (mon.0) 1517 : cluster [DBG] osdmap e535: 8 total, 8 up, 7 in 2021-03-24T22:01:45.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:45 smithi142 conmon[20125]: cluster 2021-03-24T22:01:43.363900+0000 2021-03-24T22:01:45.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:45 smithi142 conmon[20125]: osd.3 (osd.3) 112 : cluster [INF] 4.e continuing backfill to osd.1 from (89'27,89'27] MIN to 89'27 2021-03-24T22:01:45.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:45 smithi142 conmon[20125]: cluster 2021-03-24T22:01:43.363991+0000 osd.3 (osd. 2021-03-24T22:01:45.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:45 smithi142 conmon[20125]: 3) 113 : cluster [DBG] 4.e starting backfill to osd.6 from (0'0,0'0] MAX to 89'27 2021-03-24T22:01:45.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:45 smithi142 conmon[20125]: cluster 2021-03-24T22:01:44.355870+0000 2021-03-24T22:01:45.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:45 smithi142 conmon[20125]: mon.a (mon.0) 1516 : cluster [DBG] osdmap e534: 8 total, 8 up, 7 in 2021-03-24T22:01:45.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:45 smithi142 conmon[20125]: cluster 2021-03-24T22:01:45.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:45 smithi142 conmon[20125]: 2021-03-24T22:01:44.360096+0000 osd.0 (osd.0) 112 : cluster 2021-03-24T22:01:45.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:45 smithi142 conmon[20125]: [DBG] 4.a starting backfill to osd.6 from (0'0,0'0] MAX to 69'23 2021-03-24T22:01:45.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:45 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:01:45.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:45 smithi142 conmon[20125]: -24T22:01:44.362479+0000 osd.5 (osd.5) 20 : cluster [DBG] 2021-03-24T22:01:45.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:45 smithi142 conmon[20125]: 4.8 starting backfill to osd.6 from (0'0,0'0] MAX to 97'28 2021-03-24T22:01:45.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:45 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:01:45.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:45 smithi142 conmon[20125]: 24T22:01:44.369943+0000 osd.3 (osd.3) 114 : cluster [DBG] 4.c starting backfill to osd.6 from (0'0,0'0] MAX to 69'15 2021-03-24T22:01:45.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:45 smithi142 conmon[20125]: 2021-03-24T22:01:45.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:45 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:01:45.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:45 smithi142 conmon[20125]: 22:01:45.277668+0000 mon.a (mon.0) 1517 : cluster [DBG] osdmap e535: 8 total, 8 up, 7 in 2021-03-24T22:01:45.637 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:45.693 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:45.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:45.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:01:45.801 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:45.860 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:45.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:45.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:01:45.965 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:46.026 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:46.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:46.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:01:46.132 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:46.189 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:46.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:46.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:01:46.212 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-24T22:01:46.212 INFO:teuthology.orchestra.run.smithi125.stdout: "size": 20, 2021-03-24T22:01:46.213 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 600, 2021-03-24T22:01:46.213 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [ 2021-03-24T22:01:46.213 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.213 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:1777 7.17 e232/203 7:ef6d6bee:::smithi12556470-24:head v 232'218, mlcod=230'214)", 2021-03-24T22:01:46.214 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:56:41.201926+0000", 2021-03-24T22:01:46.214 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 305.00489026100001, 2021-03-24T22:01:46.214 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.039154920000000003, 2021-03-24T22:01:46.214 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:01:46.214 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:01:46.215 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:01:46.215 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.215 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:01:46.215 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:41.201926+0000", 2021-03-24T22:01:46.216 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:01:46.216 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.216 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.216 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:01:46.216 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:41.201926+0000", 2021-03-24T22:01:46.217 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999995 2021-03-24T22:01:46.217 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.218 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.219 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:01:46.219 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:41.201926+0000", 2021-03-24T22:01:46.219 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00074986499999999995 2021-03-24T22:01:46.220 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.220 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.221 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:01:46.221 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:41.202676+0000", 2021-03-24T22:01:46.222 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5279000000000001e-05 2021-03-24T22:01:46.224 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.224 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.224 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:01:46.226 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:41.202691+0000", 2021-03-24T22:01:46.226 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.8719999999999996e-06 2021-03-24T22:01:46.226 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.226 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.227 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:01:46.227 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:41.202698+0000", 2021-03-24T22:01:46.228 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.9821999999999994e-05 2021-03-24T22:01:46.228 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.228 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.229 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:46.229 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:41.202798+0000", 2021-03-24T22:01:46.229 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.3599000000000001e-05 2021-03-24T22:01:46.229 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.230 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.230 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:01:46.230 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:41.202831+0000", 2021-03-24T22:01:46.231 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.038160301000000001 2021-03-24T22:01:46.231 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.231 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.231 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:01:46.232 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:41.240992+0000", 2021-03-24T22:01:46.232 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.9573000000000005e-05 2021-03-24T22:01:46.232 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.233 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.233 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:01:46.233 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:41.241081+0000", 2021-03-24T22:01:46.233 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.039154920000000003 2021-03-24T22:01:46.234 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:46.235 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:01:46.235 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:46.235 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.236 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.236 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:2557 7.11 e290/204 7:8dd64f9e:::smithi12556470-49:head v 290'308, mlcod=278'304)", 2021-03-24T22:01:46.236 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:57:40.204468+0000", 2021-03-24T22:01:46.236 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 246.00234876900001, 2021-03-24T22:01:46.237 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.052820996000000002, 2021-03-24T22:01:46.237 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:01:46.237 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:01:46.237 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:01:46.238 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.238 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:01:46.239 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.204468+0000", 2021-03-24T22:01:46.239 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:01:46.239 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.240 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.240 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:01:46.241 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.204468+0000", 2021-03-24T22:01:46.241 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999952 2021-03-24T22:01:46.241 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.242 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.242 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:01:46.242 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.204463+0000", 2021-03-24T22:01:46.243 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00085829000000000005 2021-03-24T22:01:46.243 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.243 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.243 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:01:46.243 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.205321+0000", 2021-03-24T22:01:46.244 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.4329999999999998e-05 2021-03-24T22:01:46.244 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.244 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.244 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:01:46.245 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.205356+0000", 2021-03-24T22:01:46.245 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2873e-05 2021-03-24T22:01:46.245 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.245 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.246 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:01:46.246 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.205369+0000", 2021-03-24T22:01:46.246 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.8431000000000007e-05 2021-03-24T22:01:46.246 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.247 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.247 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:46.247 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.205467+0000", 2021-03-24T22:01:46.248 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.0049000000000002e-05 2021-03-24T22:01:46.248 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.248 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.248 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:01:46.249 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.205507+0000", 2021-03-24T22:01:46.249 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.051707234999999997 2021-03-24T22:01:46.249 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.249 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.249 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:01:46.250 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.257214+0000", 2021-03-24T22:01:46.250 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.4583000000000001e-05 2021-03-24T22:01:46.250 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.251 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.251 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:01:46.251 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.257289+0000", 2021-03-24T22:01:46.251 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.052820996000000002 2021-03-24T22:01:46.252 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:46.252 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:01:46.252 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:46.252 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.252 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.253 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:2558 7.11 e290/204 7:8dd64f9e:::smithi12556470-49:head v 290'309, mlcod=278'304)", 2021-03-24T22:01:46.253 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:57:40.209426+0000", 2021-03-24T22:01:46.253 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 245.99739105699999, 2021-03-24T22:01:46.253 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.047890466999999999, 2021-03-24T22:01:46.254 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:01:46.254 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:01:46.254 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:01:46.254 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.255 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:01:46.255 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.209426+0000", 2021-03-24T22:01:46.255 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:01:46.256 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.256 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.256 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:01:46.256 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.209426+0000", 2021-03-24T22:01:46.257 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T22:01:46.258 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.258 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.259 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:01:46.259 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.209424+0000", 2021-03-24T22:01:46.260 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00021029499999999999 2021-03-24T22:01:46.260 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.261 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.261 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:01:46.261 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.209634+0000", 2021-03-24T22:01:46.262 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9154999999999999e-05 2021-03-24T22:01:46.263 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.263 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.263 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:01:46.263 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.209653+0000", 2021-03-24T22:01:46.264 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.9319999999999999e-06 2021-03-24T22:01:46.264 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.264 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.264 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:01:46.264 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.209660+0000", 2021-03-24T22:01:46.265 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6637e-05 2021-03-24T22:01:46.265 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.265 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.265 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:46.266 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.209717+0000", 2021-03-24T22:01:46.266 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0486e-05 2021-03-24T22:01:46.266 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.267 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.267 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:01:46.267 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.209737+0000", 2021-03-24T22:01:46.268 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.047558077999999997 2021-03-24T22:01:46.268 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.268 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.268 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:01:46.269 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.257295+0000", 2021-03-24T22:01:46.269 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0899000000000001e-05 2021-03-24T22:01:46.269 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.269 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.269 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:01:46.270 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.257316+0000", 2021-03-24T22:01:46.270 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.047890466999999999 2021-03-24T22:01:46.270 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:46.270 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:01:46.271 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:46.271 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.271 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.271 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:2559 7.11 e290/204 7:8dd64f9e:::smithi12556470-49:head v 290'310, mlcod=278'304)", 2021-03-24T22:01:46.271 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:57:40.210587+0000", 2021-03-24T22:01:46.272 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 245.99622938100001, 2021-03-24T22:01:46.272 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.046757605000000001, 2021-03-24T22:01:46.272 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:01:46.272 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:01:46.273 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:01:46.273 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.273 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:01:46.273 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.210587+0000", 2021-03-24T22:01:46.274 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:01:46.274 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.274 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.274 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:01:46.275 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.210587+0000", 2021-03-24T22:01:46.275 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T22:01:46.275 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.275 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.275 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:01:46.276 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.210585+0000", 2021-03-24T22:01:46.276 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.3989999999999993e-06 2021-03-24T22:01:46.276 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.276 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.277 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:01:46.277 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.210595+0000", 2021-03-24T22:01:46.277 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2702e-05 2021-03-24T22:01:46.277 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.277 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.278 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:01:46.278 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.210607+0000", 2021-03-24T22:01:46.278 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.3879999999999999e-06 2021-03-24T22:01:46.278 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.279 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.279 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:01:46.279 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.210613+0000", 2021-03-24T22:01:46.279 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.1072000000000002e-05 2021-03-24T22:01:46.279 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.280 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.280 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:46.280 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.210654+0000", 2021-03-24T22:01:46.280 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.5398999999999999e-05 2021-03-24T22:01:46.281 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.281 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.281 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:01:46.281 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.210679+0000", 2021-03-24T22:01:46.282 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.046641477000000001 2021-03-24T22:01:46.282 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.282 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.282 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:01:46.282 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.257321+0000", 2021-03-24T22:01:46.283 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.4128999999999999e-05 2021-03-24T22:01:46.283 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.283 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.283 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:01:46.284 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.257345+0000", 2021-03-24T22:01:46.284 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.046757605000000001 2021-03-24T22:01:46.284 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:46.284 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:01:46.284 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:46.285 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.285 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.285 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:3847 7.19 e400/340 7:9b48f357:::smithi12556470-17:head v 400'390, mlcod=391'386)", 2021-03-24T22:01:46.285 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:30.268181+0000", 2021-03-24T22:01:46.286 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 135.938636073, 2021-03-24T22:01:46.286 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.053258855000000001, 2021-03-24T22:01:46.286 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:01:46.286 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:01:46.287 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:01:46.287 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.287 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:01:46.287 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.268181+0000", 2021-03-24T22:01:46.287 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:01:46.288 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.288 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.288 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:01:46.288 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.268181+0000", 2021-03-24T22:01:46.289 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999933 2021-03-24T22:01:46.289 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.289 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.289 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:01:46.289 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.268174+0000", 2021-03-24T22:01:46.290 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.1306999999999998e-05 2021-03-24T22:01:46.290 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.290 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.290 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:01:46.291 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.268265+0000", 2021-03-24T22:01:46.291 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.0286000000000001e-05 2021-03-24T22:01:46.291 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.291 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.291 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:01:46.292 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.268296+0000", 2021-03-24T22:01:46.292 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.434e-05 2021-03-24T22:01:46.292 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.292 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.292 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:01:46.293 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.268310+0000", 2021-03-24T22:01:46.293 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4766999999999998e-05 2021-03-24T22:01:46.293 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.293 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.294 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:46.294 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.268375+0000", 2021-03-24T22:01:46.294 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.6304999999999999e-05 2021-03-24T22:01:46.294 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.295 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.295 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:01:46.295 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.268411+0000", 2021-03-24T22:01:46.295 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.052984651000000001 2021-03-24T22:01:46.295 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.296 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.296 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:01:46.296 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.321396+0000", 2021-03-24T22:01:46.296 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3760000000000001e-05 2021-03-24T22:01:46.297 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.297 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.297 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:01:46.297 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.321439+0000", 2021-03-24T22:01:46.297 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.053258855000000001 2021-03-24T22:01:46.298 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:46.298 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:01:46.298 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:46.298 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.299 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.299 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:3848 7.19 e400/340 7:9b48f357:::smithi12556470-17:head v 400'391, mlcod=391'386)", 2021-03-24T22:01:46.299 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:30.268965+0000", 2021-03-24T22:01:46.299 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 135.937851378, 2021-03-24T22:01:46.299 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.052493694, 2021-03-24T22:01:46.300 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:01:46.300 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:01:46.300 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:01:46.300 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.301 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:01:46.301 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.268965+0000", 2021-03-24T22:01:46.301 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:01:46.301 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.302 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.302 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:01:46.302 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.268965+0000", 2021-03-24T22:01:46.302 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T22:01:46.302 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.303 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.303 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:01:46.303 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.268964+0000", 2021-03-24T22:01:46.303 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.9339999999999995e-06 2021-03-24T22:01:46.304 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.304 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.304 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:01:46.304 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.268973+0000", 2021-03-24T22:01:46.304 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3319000000000001e-05 2021-03-24T22:01:46.305 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.305 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.305 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:01:46.305 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.268986+0000", 2021-03-24T22:01:46.306 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.0950000000000001e-06 2021-03-24T22:01:46.306 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.306 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.306 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:01:46.306 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.268992+0000", 2021-03-24T22:01:46.307 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9656000000000001e-05 2021-03-24T22:01:46.307 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.307 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.307 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:46.308 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.269032+0000", 2021-03-24T22:01:46.308 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0984000000000001e-05 2021-03-24T22:01:46.308 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.308 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.309 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:01:46.309 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.269053+0000", 2021-03-24T22:01:46.309 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.052390012999999999 2021-03-24T22:01:46.309 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.309 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.310 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:01:46.310 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.321443+0000", 2021-03-24T22:01:46.310 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5875e-05 2021-03-24T22:01:46.310 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.311 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.311 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:01:46.311 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.321459+0000", 2021-03-24T22:01:46.311 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.052493694 2021-03-24T22:01:46.312 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:46.312 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:01:46.312 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:46.312 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.312 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.313 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:3850 7.1b e400/282 7:de4c3db5:::smithi12556470-32:head v 400'549, mlcod=400'545)", 2021-03-24T22:01:46.313 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:30.276118+0000", 2021-03-24T22:01:46.313 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 135.930698326, 2021-03-24T22:01:46.313 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.045491524999999998, 2021-03-24T22:01:46.314 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:01:46.314 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:01:46.314 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:01:46.314 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.314 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:01:46.315 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.276118+0000", 2021-03-24T22:01:46.315 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:01:46.315 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.315 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.316 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:01:46.316 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.276118+0000", 2021-03-24T22:01:46.316 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999967 2021-03-24T22:01:46.316 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.316 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.317 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:01:46.317 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.276115+0000", 2021-03-24T22:01:46.317 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0020321900000000001 2021-03-24T22:01:46.317 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.318 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.318 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:01:46.318 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.278147+0000", 2021-03-24T22:01:46.318 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3791999999999999e-05 2021-03-24T22:01:46.319 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.319 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.319 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:01:46.319 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.278161+0000", 2021-03-24T22:01:46.319 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.5249999999999994e-06 2021-03-24T22:01:46.320 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.320 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.320 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:01:46.320 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.278170+0000", 2021-03-24T22:01:46.321 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.8162000000000001e-05 2021-03-24T22:01:46.321 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.321 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.321 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:46.321 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.278258+0000", 2021-03-24T22:01:46.322 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.6956000000000001e-05 2021-03-24T22:01:46.322 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.322 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.322 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:01:46.323 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.278285+0000", 2021-03-24T22:01:46.323 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.043282701999999999 2021-03-24T22:01:46.323 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.323 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.323 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:01:46.324 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.321568+0000", 2021-03-24T22:01:46.324 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.1721999999999998e-05 2021-03-24T22:01:46.324 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.324 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.325 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:01:46.325 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.321610+0000", 2021-03-24T22:01:46.325 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.045491524999999998 2021-03-24T22:01:46.325 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:46.325 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:01:46.326 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:46.326 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.326 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.326 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:3829 7.c e400/275 7:31cb714a:::smithi12556470-7:head v 400'435, mlcod=397'431)", 2021-03-24T22:01:46.327 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:30.277370+0000", 2021-03-24T22:01:46.327 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 135.92944711999999, 2021-03-24T22:01:46.327 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.044151612999999999, 2021-03-24T22:01:46.327 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:01:46.328 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:01:46.328 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:01:46.328 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.328 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:01:46.328 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.277370+0000", 2021-03-24T22:01:46.329 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:01:46.329 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.329 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.329 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:01:46.330 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.277370+0000", 2021-03-24T22:01:46.330 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999995 2021-03-24T22:01:46.330 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.330 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.330 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:01:46.331 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.277369+0000", 2021-03-24T22:01:46.331 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00013732499999999999 2021-03-24T22:01:46.331 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.331 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.332 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:01:46.332 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.277506+0000", 2021-03-24T22:01:46.332 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.5281000000000001e-05 2021-03-24T22:01:46.332 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.332 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.333 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:01:46.333 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.277532+0000", 2021-03-24T22:01:46.333 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1952e-05 2021-03-24T22:01:46.333 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.334 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.334 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:01:46.334 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.277544+0000", 2021-03-24T22:01:46.334 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000473554 2021-03-24T22:01:46.335 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.335 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.335 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:46.335 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.278017+0000", 2021-03-24T22:01:46.335 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.5218000000000001e-05 2021-03-24T22:01:46.336 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.336 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.336 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:01:46.336 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.278052+0000", 2021-03-24T22:01:46.337 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.043425692000000002 2021-03-24T22:01:46.337 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.337 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.337 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:01:46.337 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.321478+0000", 2021-03-24T22:01:46.338 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3025000000000002e-05 2021-03-24T22:01:46.338 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.338 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.338 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:01:46.339 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.321521+0000", 2021-03-24T22:01:46.339 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.044151612999999999 2021-03-24T22:01:46.339 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:46.339 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:01:46.339 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:46.340 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.340 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.340 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:3830 7.c e400/275 7:31cb714a:::smithi12556470-7:head v 400'436, mlcod=397'431)", 2021-03-24T22:01:46.340 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:30.277557+0000", 2021-03-24T22:01:46.341 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 135.929259204, 2021-03-24T22:01:46.341 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.043987660999999997, 2021-03-24T22:01:46.341 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:01:46.341 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:01:46.341 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:01:46.342 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.342 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:01:46.342 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.277557+0000", 2021-03-24T22:01:46.342 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:01:46.343 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.343 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.343 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:01:46.343 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.277557+0000", 2021-03-24T22:01:46.343 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999995 2021-03-24T22:01:46.344 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.344 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.344 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:01:46.344 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.277557+0000", 2021-03-24T22:01:46.345 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00034763900000000001 2021-03-24T22:01:46.345 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.345 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.345 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:01:46.346 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.277904+0000", 2021-03-24T22:01:46.346 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.4780000000000008e-06 2021-03-24T22:01:46.346 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.346 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.346 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:01:46.347 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.277913+0000", 2021-03-24T22:01:46.347 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.002e-06 2021-03-24T22:01:46.347 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.347 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.348 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:01:46.348 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.277920+0000", 2021-03-24T22:01:46.348 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001241455 2021-03-24T22:01:46.348 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.348 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.349 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:46.349 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.279161+0000", 2021-03-24T22:01:46.349 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.5119000000000001e-05 2021-03-24T22:01:46.349 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.350 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.350 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:01:46.350 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.279186+0000", 2021-03-24T22:01:46.350 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.042339505 2021-03-24T22:01:46.351 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.351 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.351 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:01:46.351 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.321526+0000", 2021-03-24T22:01:46.351 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9083999999999999e-05 2021-03-24T22:01:46.352 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.352 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.352 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:01:46.352 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.321545+0000", 2021-03-24T22:01:46.353 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.043987660999999997 2021-03-24T22:01:46.353 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:46.353 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:01:46.353 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:46.353 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.354 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.354 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:3831 7.c e400/275 7:31cb714a:::smithi12556470-7:head v 400'437, mlcod=397'431)", 2021-03-24T22:01:46.354 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:30.286442+0000", 2021-03-24T22:01:46.354 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 135.92037432399999, 2021-03-24T22:01:46.355 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.035122622999999999, 2021-03-24T22:01:46.355 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:01:46.355 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:01:46.355 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:01:46.355 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.356 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:01:46.356 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.286442+0000", 2021-03-24T22:01:46.356 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:01:46.356 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.357 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.357 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:01:46.357 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.286442+0000", 2021-03-24T22:01:46.357 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999971 2021-03-24T22:01:46.358 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.358 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.358 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:01:46.358 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.286439+0000", 2021-03-24T22:01:46.358 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.6485999999999999e-05 2021-03-24T22:01:46.359 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.359 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.359 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:01:46.359 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.286516+0000", 2021-03-24T22:01:46.360 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.2932e-05 2021-03-24T22:01:46.360 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.360 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.360 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:01:46.361 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.286539+0000", 2021-03-24T22:01:46.361 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.0440000000000001e-06 2021-03-24T22:01:46.361 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.361 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.362 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:01:46.362 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.286548+0000", 2021-03-24T22:01:46.362 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2671999999999997e-05 2021-03-24T22:01:46.362 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.362 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.363 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:46.363 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.286630+0000", 2021-03-24T22:01:46.363 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9359e-05 2021-03-24T22:01:46.363 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.364 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.364 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:01:46.364 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.286670+0000", 2021-03-24T22:01:46.364 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.034877342999999998 2021-03-24T22:01:46.364 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.365 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.365 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:01:46.365 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.321547+0000", 2021-03-24T22:01:46.365 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7813e-05 2021-03-24T22:01:46.366 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.366 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.366 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:01:46.366 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.321565+0000", 2021-03-24T22:01:46.367 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.035122622999999999 2021-03-24T22:01:46.367 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:46.367 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:01:46.367 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:46.368 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.368 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.368 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:3832 7.c e400/275 7:31cb714a:::smithi12556470-7:head v 400'438, mlcod=397'431)", 2021-03-24T22:01:46.368 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:30.287369+0000", 2021-03-24T22:01:46.368 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 135.91944787, 2021-03-24T22:01:46.369 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.034213645000000001, 2021-03-24T22:01:46.369 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:01:46.369 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:01:46.369 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:01:46.370 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.370 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:01:46.370 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.287369+0000", 2021-03-24T22:01:46.370 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:01:46.371 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.371 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.371 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:01:46.371 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.287369+0000", 2021-03-24T22:01:46.372 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T22:01:46.372 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.372 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.372 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:01:46.372 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.287368+0000", 2021-03-24T22:01:46.373 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.9680000000000006e-06 2021-03-24T22:01:46.373 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.373 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.373 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:01:46.374 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.287376+0000", 2021-03-24T22:01:46.374 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3944999999999999e-05 2021-03-24T22:01:46.374 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.374 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.375 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:01:46.375 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.287390+0000", 2021-03-24T22:01:46.375 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.1970000000000001e-06 2021-03-24T22:01:46.375 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.376 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.376 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:01:46.376 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.287396+0000", 2021-03-24T22:01:46.376 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.3905000000000006e-05 2021-03-24T22:01:46.376 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.377 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.377 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:46.377 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.287460+0000", 2021-03-24T22:01:46.377 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.7464999999999999e-05 2021-03-24T22:01:46.378 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.378 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.378 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:01:46.378 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.287487+0000", 2021-03-24T22:01:46.379 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.034079365 2021-03-24T22:01:46.379 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.379 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.379 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:01:46.380 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.321566+0000", 2021-03-24T22:01:46.380 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5937999999999999e-05 2021-03-24T22:01:46.380 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.380 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.380 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:01:46.381 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.321582+0000", 2021-03-24T22:01:46.381 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.034213645000000001 2021-03-24T22:01:46.381 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:46.381 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:01:46.382 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:46.382 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.382 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.382 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:3993 7.15 7:ad88ec1b:::smithi12556470-47:head [write 759815~471768,stat] snapc 137=[137,136,133,132,12d,12c,12b,129,128,127,126,120,11e,11b,11a,119,110,10f,10c,106,105,104,fa,f8,e7] ondisk+write+known_if_redirected e420)", 2021-03-24T22:01:46.383 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:50.232662+0000", 2021-03-24T22:01:46.383 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 115.974154183, 2021-03-24T22:01:46.383 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.032319643000000002, 2021-03-24T22:01:46.383 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:01:46.384 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:01:46.384 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:01:46.384 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:01:46.384 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:01:46.385 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3993 2021-03-24T22:01:46.385 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.385 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:01:46.385 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.385 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:01:46.386 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.232662+0000", 2021-03-24T22:01:46.386 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:01:46.386 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.386 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.387 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:01:46.387 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.232662+0000", 2021-03-24T22:01:46.387 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T22:01:46.387 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.387 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.388 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:01:46.388 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.232661+0000", 2021-03-24T22:01:46.388 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000106016 2021-03-24T22:01:46.388 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.389 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.389 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:01:46.389 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.232767+0000", 2021-03-24T22:01:46.389 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7954000000000001e-05 2021-03-24T22:01:46.390 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.390 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.390 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:01:46.390 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.232785+0000", 2021-03-24T22:01:46.390 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.0350000000000004e-06 2021-03-24T22:01:46.391 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.391 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.391 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:01:46.391 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.232792+0000", 2021-03-24T22:01:46.392 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9459000000000003e-05 2021-03-24T22:01:46.392 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.392 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.392 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:46.392 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.232831+0000", 2021-03-24T22:01:46.393 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.4829999999999999e-06 2021-03-24T22:01:46.393 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.393 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.393 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:01:46.394 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.232840+0000", 2021-03-24T22:01:46.394 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0027213670000000001 2021-03-24T22:01:46.394 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.394 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.395 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:46.395 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.235561+0000", 2021-03-24T22:01:46.395 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000121373 2021-03-24T22:01:46.395 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.395 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.396 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:01:46.396 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.235682+0000", 2021-03-24T22:01:46.396 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00023657800000000001 2021-03-24T22:01:46.396 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.397 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.397 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 4,7", 2021-03-24T22:01:46.397 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.235919+0000", 2021-03-24T22:01:46.397 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0018850539999999999 2021-03-24T22:01:46.397 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.398 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.398 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:01:46.398 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.237804+0000", 2021-03-24T22:01:46.398 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0032228640000000002 2021-03-24T22:01:46.399 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.399 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.399 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:01:46.399 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.241027+0000", 2021-03-24T22:01:46.400 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.023599372 2021-03-24T22:01:46.400 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.400 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.400 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:01:46.400 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.264626+0000", 2021-03-24T22:01:46.401 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.2217000000000003e-05 2021-03-24T22:01:46.401 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.401 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.401 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:01:46.402 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.264699+0000", 2021-03-24T22:01:46.402 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00028356900000000001 2021-03-24T22:01:46.402 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.402 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.403 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:01:46.403 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.264982+0000", 2021-03-24T22:01:46.403 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.032319643000000002 2021-03-24T22:01:46.403 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:46.403 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:01:46.404 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:46.404 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.404 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.404 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:3994 7.15 7:ad88ec1b:::smithi12556470-47:head [write 1742451~465507] snapc 137=[137,136,133,132,12d,12c,12b,129,128,127,126,120,11e,11b,11a,119,110,10f,10c,106,105,104,fa,f8,e7] ondisk+write+known_if_redirected e420)", 2021-03-24T22:01:46.405 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:50.238478+0000", 2021-03-24T22:01:46.405 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 115.968338833, 2021-03-24T22:01:46.405 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.042060243999999997, 2021-03-24T22:01:46.405 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:01:46.405 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:01:46.406 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:01:46.406 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:01:46.406 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:01:46.406 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3994 2021-03-24T22:01:46.407 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.407 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:01:46.407 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.407 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:01:46.408 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.238478+0000", 2021-03-24T22:01:46.408 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:01:46.408 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.408 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.408 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:01:46.409 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.238478+0000", 2021-03-24T22:01:46.409 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999967 2021-03-24T22:01:46.409 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.409 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.410 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:01:46.410 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.238474+0000", 2021-03-24T22:01:46.410 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00013515599999999999 2021-03-24T22:01:46.410 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.410 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.411 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:01:46.411 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.238610+0000", 2021-03-24T22:01:46.411 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2685e-05 2021-03-24T22:01:46.411 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.412 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.412 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:01:46.412 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.238622+0000", 2021-03-24T22:01:46.412 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.4679999999999998e-06 2021-03-24T22:01:46.412 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.413 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.413 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:01:46.413 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.238632+0000", 2021-03-24T22:01:46.413 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.6968000000000002e-05 2021-03-24T22:01:46.414 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.414 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.414 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:46.414 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.238669+0000", 2021-03-24T22:01:46.415 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.8763000000000003e-05 2021-03-24T22:01:46.415 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.415 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.415 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:01:46.415 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.238767+0000", 2021-03-24T22:01:46.416 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.026297406999999998 2021-03-24T22:01:46.416 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.416 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.416 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:46.417 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.265065+0000", 2021-03-24T22:01:46.417 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000118282 2021-03-24T22:01:46.417 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.417 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.417 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:01:46.418 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.265183+0000", 2021-03-24T22:01:46.418 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00025056299999999998 2021-03-24T22:01:46.418 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.418 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.419 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 4,7", 2021-03-24T22:01:46.419 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.265434+0000", 2021-03-24T22:01:46.419 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0024804789999999998 2021-03-24T22:01:46.419 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.419 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.420 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:01:46.420 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.267914+0000", 2021-03-24T22:01:46.420 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012115023000000001 2021-03-24T22:01:46.420 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.421 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.421 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:01:46.421 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.280029+0000", 2021-03-24T22:01:46.421 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00028268400000000001 2021-03-24T22:01:46.421 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.422 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.422 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:01:46.422 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.280312+0000", 2021-03-24T22:01:46.422 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.8313000000000002e-05 2021-03-24T22:01:46.423 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.423 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.423 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:01:46.423 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.280370+0000", 2021-03-24T22:01:46.424 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00016783 2021-03-24T22:01:46.424 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.424 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.424 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:01:46.424 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.280538+0000", 2021-03-24T22:01:46.425 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.042060243999999997 2021-03-24T22:01:46.425 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:46.425 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:01:46.425 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:46.425 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.426 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.426 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:3995 7.15 7:ad88ec1b:::smithi12556470-47:head [write 2917178~242637] snapc 137=[137,136,133,132,12d,12c,12b,129,128,127,126,120,11e,11b,11a,119,110,10f,10c,106,105,104,fa,f8,e7] ondisk+write+known_if_redirected e420)", 2021-03-24T22:01:46.426 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:50.244845+0000", 2021-03-24T22:01:46.426 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 115.96197189900001, 2021-03-24T22:01:46.427 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.035994086000000002, 2021-03-24T22:01:46.427 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:01:46.427 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:01:46.427 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:01:46.428 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:01:46.428 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:01:46.428 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3995 2021-03-24T22:01:46.428 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.428 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:01:46.429 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.429 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:01:46.429 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.244845+0000", 2021-03-24T22:01:46.429 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:01:46.430 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.430 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.430 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:01:46.430 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.244845+0000", 2021-03-24T22:01:46.430 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999971 2021-03-24T22:01:46.431 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.431 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.431 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:01:46.431 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.244842+0000", 2021-03-24T22:01:46.432 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4729999999999999e-05 2021-03-24T22:01:46.432 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.432 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.432 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:01:46.432 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.244906+0000", 2021-03-24T22:01:46.433 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.2957999999999999e-05 2021-03-24T22:01:46.433 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.433 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.433 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:01:46.434 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.244929+0000", 2021-03-24T22:01:46.434 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.3969999999999994e-06 2021-03-24T22:01:46.434 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.434 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.435 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:01:46.435 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.244938+0000", 2021-03-24T22:01:46.435 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.6421000000000006e-05 2021-03-24T22:01:46.435 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.435 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.436 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:46.436 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.245014+0000", 2021-03-24T22:01:46.436 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000115259 2021-03-24T22:01:46.436 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.437 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.437 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:01:46.437 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.245130+0000", 2021-03-24T22:01:46.437 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.021186351999999999 2021-03-24T22:01:46.437 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.438 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.438 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:46.438 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.266316+0000", 2021-03-24T22:01:46.438 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.9944000000000004e-05 2021-03-24T22:01:46.439 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.439 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.439 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:01:46.439 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.266416+0000", 2021-03-24T22:01:46.440 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00023467900000000001 2021-03-24T22:01:46.440 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.440 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.440 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 4,7", 2021-03-24T22:01:46.440 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.266650+0000", 2021-03-24T22:01:46.441 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001697325 2021-03-24T22:01:46.441 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.441 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.441 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:01:46.442 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.268348+0000", 2021-03-24T22:01:46.442 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011757292000000001 2021-03-24T22:01:46.442 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.442 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.442 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:01:46.443 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.280105+0000", 2021-03-24T22:01:46.443 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00051535899999999996 2021-03-24T22:01:46.443 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.443 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.444 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:01:46.444 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.280620+0000", 2021-03-24T22:01:46.444 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.8341999999999998e-05 2021-03-24T22:01:46.444 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.444 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.445 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:01:46.445 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.280679+0000", 2021-03-24T22:01:46.445 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000160008 2021-03-24T22:01:46.445 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.446 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.446 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:01:46.446 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.280839+0000", 2021-03-24T22:01:46.446 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.035994086000000002 2021-03-24T22:01:46.447 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:46.447 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:01:46.447 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:46.447 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.447 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.448 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:3996 7.15 7:ad88ec1b:::smithi12556470-47:head [setxattr _header (58) in=7b,truncate 3159815] snapc 137=[137,136,133,132,12d,12c,12b,129,128,127,126,120,11e,11b,11a,119,110,10f,10c,106,105,104,fa,f8,e7] ondisk+write+known_if_redirected e420)", 2021-03-24T22:01:46.448 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:50.244952+0000", 2021-03-24T22:01:46.448 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 115.961864394, 2021-03-24T22:01:46.448 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.036356240999999997, 2021-03-24T22:01:46.449 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:01:46.449 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:01:46.449 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:01:46.449 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:01:46.450 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:01:46.450 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3996 2021-03-24T22:01:46.450 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.450 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:01:46.450 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.451 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:01:46.451 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.244952+0000", 2021-03-24T22:01:46.451 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:01:46.451 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.452 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.452 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:01:46.452 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.244952+0000", 2021-03-24T22:01:46.452 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999995 2021-03-24T22:01:46.452 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.453 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.453 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:01:46.453 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.244952+0000", 2021-03-24T22:01:46.453 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3220000000000003e-06 2021-03-24T22:01:46.454 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.454 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.454 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:01:46.454 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.244956+0000", 2021-03-24T22:01:46.455 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5040000000000002e-06 2021-03-24T22:01:46.455 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.455 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.455 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:01:46.455 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.244961+0000", 2021-03-24T22:01:46.456 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.8210000000000001e-06 2021-03-24T22:01:46.456 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.456 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.456 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:01:46.457 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.244966+0000", 2021-03-24T22:01:46.457 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00018897200000000001 2021-03-24T22:01:46.457 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.457 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.457 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:46.458 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.245155+0000", 2021-03-24T22:01:46.458 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.0983000000000006e-05 2021-03-24T22:01:46.458 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.458 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.459 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:01:46.459 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.245246+0000", 2021-03-24T22:01:46.459 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.022078372999999998 2021-03-24T22:01:46.459 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.460 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.460 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:46.460 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.267324+0000", 2021-03-24T22:01:46.460 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.1389000000000002e-05 2021-03-24T22:01:46.460 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.461 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.461 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:01:46.461 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.267416+0000", 2021-03-24T22:01:46.461 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000216388 2021-03-24T22:01:46.462 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.462 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.462 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 4,7", 2021-03-24T22:01:46.462 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.267632+0000", 2021-03-24T22:01:46.462 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0030801959999999999 2021-03-24T22:01:46.463 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.463 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.463 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:01:46.463 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.270712+0000", 2021-03-24T22:01:46.464 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0094740310000000008 2021-03-24T22:01:46.464 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.464 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.464 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:01:46.464 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.280186+0000", 2021-03-24T22:01:46.465 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00078359100000000004 2021-03-24T22:01:46.465 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.465 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.465 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:01:46.466 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.280970+0000", 2021-03-24T22:01:46.466 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5875000000000003e-05 2021-03-24T22:01:46.466 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.466 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.467 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:01:46.467 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.281026+0000", 2021-03-24T22:01:46.467 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00028250399999999999 2021-03-24T22:01:46.467 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.467 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.468 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:01:46.468 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.281308+0000", 2021-03-24T22:01:46.468 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.036356240999999997 2021-03-24T22:01:46.468 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:46.469 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:01:46.469 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:46.469 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.469 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.469 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:3997 7.15 7:ad88ec1b:::smithi12556470-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e420)", 2021-03-24T22:01:46.470 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:50.245003+0000", 2021-03-24T22:01:46.470 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 115.961813357, 2021-03-24T22:01:46.470 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.036618082000000003, 2021-03-24T22:01:46.470 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:01:46.471 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T22:01:46.471 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:01:46.471 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:01:46.471 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:01:46.472 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3997 2021-03-24T22:01:46.472 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.472 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:01:46.472 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.472 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:01:46.473 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.245003+0000", 2021-03-24T22:01:46.473 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:01:46.473 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.473 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.474 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:01:46.474 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.245003+0000", 2021-03-24T22:01:46.474 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999995 2021-03-24T22:01:46.474 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.474 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.475 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:01:46.475 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.245003+0000", 2021-03-24T22:01:46.475 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.9249999999999999e-06 2021-03-24T22:01:46.475 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.476 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.476 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:01:46.476 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.245006+0000", 2021-03-24T22:01:46.476 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2450000000000003e-06 2021-03-24T22:01:46.477 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.477 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.477 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:01:46.477 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.245012+0000", 2021-03-24T22:01:46.477 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.2849999999999996e-06 2021-03-24T22:01:46.478 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.478 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.478 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:01:46.478 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.245016+0000", 2021-03-24T22:01:46.479 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000254861 2021-03-24T22:01:46.479 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.479 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.479 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:46.479 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.245271+0000", 2021-03-24T22:01:46.480 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.4078000000000005e-05 2021-03-24T22:01:46.480 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.480 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.480 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:01:46.481 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.245345+0000", 2021-03-24T22:01:46.481 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.022599312999999999 2021-03-24T22:01:46.481 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.481 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.481 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:46.482 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.267944+0000", 2021-03-24T22:01:46.482 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.3528999999999993e-05 2021-03-24T22:01:46.482 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.482 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.483 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:01:46.483 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.268018+0000", 2021-03-24T22:01:46.483 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.013332943 2021-03-24T22:01:46.483 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.483 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.484 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:46.484 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.281351+0000", 2021-03-24T22:01:46.484 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.2990000000000002e-05 2021-03-24T22:01:46.484 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.485 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.485 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:01:46.485 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.281444+0000", 2021-03-24T22:01:46.485 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000177443 2021-03-24T22:01:46.485 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.486 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.486 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:01:46.486 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.281621+0000", 2021-03-24T22:01:46.486 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.036618082000000003 2021-03-24T22:01:46.487 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:46.487 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:01:46.487 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:46.487 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.488 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.488 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.34352.0:1584 5.d 5:b4812045:::notify.4:head [watch ping cookie 94195313969280 gen 2] snapc 0=[] ondisk+write+known_if_redirected e428)", 2021-03-24T22:01:46.488 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:00:04.104713+0000", 2021-03-24T22:01:46.488 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 102.102103309, 2021-03-24T22:01:46.488 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.195035761, 2021-03-24T22:01:46.489 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:01:46.489 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T22:01:46.489 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:01:46.489 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.34352", 2021-03-24T22:01:46.490 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/3436348424", 2021-03-24T22:01:46.490 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1584 2021-03-24T22:01:46.490 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.490 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:01:46.491 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.491 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:01:46.491 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.104713+0000", 2021-03-24T22:01:46.491 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:01:46.491 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.492 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.492 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:01:46.492 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.104713+0000", 2021-03-24T22:01:46.492 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999952 2021-03-24T22:01:46.493 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.493 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.493 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:01:46.493 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.104708+0000", 2021-03-24T22:01:46.493 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.8836000000000001e-05 2021-03-24T22:01:46.494 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.494 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.494 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:01:46.494 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.104727+0000", 2021-03-24T22:01:46.495 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.7954000000000002e-05 2021-03-24T22:01:46.495 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.495 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.495 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:01:46.495 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.104775+0000", 2021-03-24T22:01:46.496 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.8598999999999999e-05 2021-03-24T22:01:46.496 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.496 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.496 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:01:46.497 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.104794+0000", 2021-03-24T22:01:46.497 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012397299999999999 2021-03-24T22:01:46.497 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.497 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.498 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:46.498 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.104918+0000", 2021-03-24T22:01:46.498 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1185e-05 2021-03-24T22:01:46.498 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.498 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.499 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:01:46.499 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.104929+0000", 2021-03-24T22:01:46.499 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.15697466500000001 2021-03-24T22:01:46.499 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.500 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.500 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:46.500 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.261904+0000", 2021-03-24T22:01:46.500 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00040677900000000002 2021-03-24T22:01:46.500 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.501 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.501 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for degraded object", 2021-03-24T22:01:46.501 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.262310+0000", 2021-03-24T22:01:46.501 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.037134201999999998 2021-03-24T22:01:46.502 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.502 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.502 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:46.502 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.299445+0000", 2021-03-24T22:01:46.502 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00014002 2021-03-24T22:01:46.503 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.503 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.503 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:01:46.503 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.299585+0000", 2021-03-24T22:01:46.504 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00016441900000000001 2021-03-24T22:01:46.504 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.504 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.504 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:01:46.505 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.299749+0000", 2021-03-24T22:01:46.505 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.195035761 2021-03-24T22:01:46.505 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:46.505 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:01:46.505 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:46.506 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.506 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.506 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.34352.0:1592 5.d 5:b4812045:::notify.4:head [watch reconnect cookie 94195313969280 gen 3] snapc 0=[] ondisk+write+known_if_redirected e433)", 2021-03-24T22:01:46.506 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:00:04.106360+0000", 2021-03-24T22:01:46.507 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 102.10045711799999, 2021-03-24T22:01:46.507 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.21183971600000001, 2021-03-24T22:01:46.507 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:01:46.507 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T22:01:46.507 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:01:46.508 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.34352", 2021-03-24T22:01:46.508 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/3436348424", 2021-03-24T22:01:46.508 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1592 2021-03-24T22:01:46.508 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.509 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:01:46.509 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.509 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:01:46.509 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.106360+0000", 2021-03-24T22:01:46.509 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:01:46.510 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.510 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.510 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:01:46.510 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.106360+0000", 2021-03-24T22:01:46.511 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T22:01:46.511 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.511 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.511 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:01:46.511 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.106358+0000", 2021-03-24T22:01:46.512 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6409999999999996e-06 2021-03-24T22:01:46.512 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.512 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.512 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:01:46.513 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.106364+0000", 2021-03-24T22:01:46.513 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3934000000000001e-05 2021-03-24T22:01:46.513 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.513 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.513 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:01:46.514 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.106378+0000", 2021-03-24T22:01:46.514 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.6040000000000004e-06 2021-03-24T22:01:46.514 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.514 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.515 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:01:46.515 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.106386+0000", 2021-03-24T22:01:46.515 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.2407e-05 2021-03-24T22:01:46.515 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.516 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.516 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:46.516 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.106458+0000", 2021-03-24T22:01:46.516 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.2400000000000001e-06 2021-03-24T22:01:46.516 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.517 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.517 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:01:46.517 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.106466+0000", 2021-03-24T22:01:46.517 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.156026741 2021-03-24T22:01:46.518 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.518 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.518 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:46.518 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.262492+0000", 2021-03-24T22:01:46.519 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6474000000000002e-05 2021-03-24T22:01:46.519 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.519 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.519 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for degraded object", 2021-03-24T22:01:46.520 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.262549+0000", 2021-03-24T22:01:46.520 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.037316715 2021-03-24T22:01:46.520 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.520 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.521 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:46.521 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.299866+0000", 2021-03-24T22:01:46.521 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.1278000000000003e-05 2021-03-24T22:01:46.521 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.521 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.522 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:01:46.522 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.299957+0000", 2021-03-24T22:01:46.522 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018242359 2021-03-24T22:01:46.522 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.523 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.523 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:01:46.523 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.318199+0000", 2021-03-24T22:01:46.523 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.21183971600000001 2021-03-24T22:01:46.524 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:46.524 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:01:46.524 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:46.524 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.524 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.525 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.34352.0:1594 5.d 5:b4812045:::notify.4:head [watch reconnect cookie 94195313969280 gen 4] snapc 0=[] ondisk+write+known_if_redirected e433)", 2021-03-24T22:01:46.525 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:00:04.106433+0000", 2021-03-24T22:01:46.525 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 102.10038348499999, 2021-03-24T22:01:46.525 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.21242717699999999, 2021-03-24T22:01:46.526 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:01:46.526 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T22:01:46.526 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:01:46.526 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.34352", 2021-03-24T22:01:46.526 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/3436348424", 2021-03-24T22:01:46.527 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1594 2021-03-24T22:01:46.527 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.527 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:01:46.527 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.528 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:01:46.528 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.106433+0000", 2021-03-24T22:01:46.528 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:01:46.528 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.528 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.529 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:01:46.529 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.106433+0000", 2021-03-24T22:01:46.529 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T22:01:46.529 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.530 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.530 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:01:46.530 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.106432+0000", 2021-03-24T22:01:46.530 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.7809999999999996e-06 2021-03-24T22:01:46.531 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.531 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.531 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:01:46.531 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.106437+0000", 2021-03-24T22:01:46.531 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.2429999999999999e-06 2021-03-24T22:01:46.532 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.532 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.532 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:01:46.532 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.106444+0000", 2021-03-24T22:01:46.533 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.254e-06 2021-03-24T22:01:46.533 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.533 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.533 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:01:46.534 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.106449+0000", 2021-03-24T22:01:46.534 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.9451999999999999e-05 2021-03-24T22:01:46.534 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.534 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.535 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:46.535 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.106479+0000", 2021-03-24T22:01:46.535 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.8959999999999999e-06 2021-03-24T22:01:46.535 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.536 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.536 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:01:46.536 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.106482+0000", 2021-03-24T22:01:46.536 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.15609192699999999 2021-03-24T22:01:46.537 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.537 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.537 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:46.537 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.262574+0000", 2021-03-24T22:01:46.538 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5994e-05 2021-03-24T22:01:46.538 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.538 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.538 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for degraded object", 2021-03-24T22:01:46.539 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.262630+0000", 2021-03-24T22:01:46.539 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.037710501 2021-03-24T22:01:46.539 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.539 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.539 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:46.540 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.300340+0000", 2021-03-24T22:01:46.540 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.5327000000000005e-05 2021-03-24T22:01:46.540 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.540 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.541 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:01:46.541 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.300426+0000", 2021-03-24T22:01:46.541 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018434756 2021-03-24T22:01:46.541 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.542 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.542 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:01:46.542 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.318860+0000", 2021-03-24T22:01:46.542 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.21242717699999999 2021-03-24T22:01:46.542 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:46.543 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:01:46.543 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:46.543 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.543 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.544 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:5162 7.19 7:9b48f357:::smithi12556470-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e530)", 2021-03-24T22:01:46.544 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:01:40.289707+0000", 2021-03-24T22:01:46.544 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 5.9171097599999998, 2021-03-24T22:01:46.544 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.031417562000000003, 2021-03-24T22:01:46.544 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:01:46.545 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T22:01:46.545 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:01:46.545 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:01:46.546 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:01:46.546 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 5162 2021-03-24T22:01:46.546 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.546 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:01:46.547 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.547 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:01:46.547 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:40.289707+0000", 2021-03-24T22:01:46.547 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:01:46.548 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.548 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.548 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:01:46.548 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:40.289707+0000", 2021-03-24T22:01:46.549 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T22:01:46.549 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.549 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.549 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:01:46.550 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:40.289706+0000", 2021-03-24T22:01:46.550 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.0130000000000001e-06 2021-03-24T22:01:46.550 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.550 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.551 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:01:46.551 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:40.289711+0000", 2021-03-24T22:01:46.551 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.52e-06 2021-03-24T22:01:46.551 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.551 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.552 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:01:46.552 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:40.289718+0000", 2021-03-24T22:01:46.552 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6110000000000001e-06 2021-03-24T22:01:46.552 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.553 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.553 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:01:46.553 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:40.289724+0000", 2021-03-24T22:01:46.553 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00042721700000000003 2021-03-24T22:01:46.554 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.554 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.554 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:46.554 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:40.290151+0000", 2021-03-24T22:01:46.555 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2470000000000003e-05 2021-03-24T22:01:46.555 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.555 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.555 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-24T22:01:46.556 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:40.290214+0000", 2021-03-24T22:01:46.556 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.027005734999999999 2021-03-24T22:01:46.556 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.556 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.556 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:46.557 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:40.317219+0000", 2021-03-24T22:01:46.557 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00016049799999999999 2021-03-24T22:01:46.557 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.557 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.558 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:01:46.558 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:40.317380+0000", 2021-03-24T22:01:46.558 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0035435789999999998 2021-03-24T22:01:46.558 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.559 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.559 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:01:46.559 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:40.320923+0000", 2021-03-24T22:01:46.559 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000107429 2021-03-24T22:01:46.559 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.560 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.560 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:01:46.560 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:40.321031+0000", 2021-03-24T22:01:46.560 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.3664000000000003e-05 2021-03-24T22:01:46.561 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:01:46.561 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:01:46.561 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:01:46.561 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:40.321124+0000", 2021-03-24T22:01:46.562 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.031417562000000003 2021-03-24T22:01:46.562 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:46.562 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:01:46.562 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:46.562 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:01:46.563 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:01:46.563 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-24T22:01:46.563 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:46.565 INFO:tasks.rados.rados.0.smithi125.stdout:2026: done (2 left) 2021-03-24T22:01:46.565 INFO:tasks.rados.rados.0.smithi125.stdout:2027: delete oid 26 current snap is 224 2021-03-24T22:01:46.565 INFO:tasks.rados.rados.0.smithi125.stdout:2024: finishing rollback tid 0 to smithi12556470-17 2021-03-24T22:01:46.566 INFO:tasks.rados.rados.0.smithi125.stdout:2025: finishing rollback tid 0 to smithi12556470-32 2021-03-24T22:01:46.566 INFO:tasks.rados.rados.0.smithi125.stdout:2025: finishing rollback tid 1 to smithi12556470-32 2021-03-24T22:01:46.566 INFO:tasks.rados.rados.0.smithi125.stdout:2025: finishing rollback tid 2 to smithi12556470-32 2021-03-24T22:01:46.566 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 32 v 1010 (ObjNum 672 snap 209 seq_num 672) dirty exists 2021-03-24T22:01:46.567 INFO:tasks.rados.rados.0.smithi125.stdout:2024: finishing rollback tid 1 to smithi12556470-17 2021-03-24T22:01:46.567 INFO:tasks.rados.rados.0.smithi125.stdout:2024: finishing rollback tid 2 to smithi12556470-17 2021-03-24T22:01:46.567 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 17 v 574 (ObjNum 619 snap 188 seq_num 619) dirty exists 2021-03-24T22:01:46.567 INFO:teuthology.orchestra.run.smithi125.stderr:nodeep-scrub is unset 2021-03-24T22:01:46.568 INFO:tasks.rados.rados.0.smithi125.stdout:2024: done (2 left) 2021-03-24T22:01:46.568 INFO:tasks.rados.rados.0.smithi125.stdout:2025: done (1 left) 2021-03-24T22:01:46.569 INFO:tasks.rados.rados.0.smithi125.stdout:2027: done (0 left) 2021-03-24T22:01:46.569 INFO:tasks.rados.rados.0.smithi125.stdout:2028: snap_create 2021-03-24T22:01:46.569 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:46 smithi125 conmon[19936]: cluster 2021-03-24T22:01:44.887978+0000 mgr.y (mgr.14144) 539 2021-03-24T22:01:46.570 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:46 smithi125 conmon[19936]: : cluster [DBG] pgmap v964: 168 pgs: 1 active+recovering, 8 remapped+peering, 1 peering, 4 active+remapped+backfill_toofull, 154 active+clean; 523 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 7.9 MiB/s rd, 1.8 MiB/s wr, 8 op/s; 87/1710 objects misplaced (5.088%); 468 B/s, 3 objects/s recovering 2021-03-24T22:01:46.570 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:46 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T22:01:46.570 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:46 smithi125 conmon[19936]: -24T22:01:45.366588+0000 mon.a (mon.0) 2021-03-24T22:01:46.570 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:46 smithi125 conmon[19936]: 1518 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2021-03-24T22:01:46.571 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:46 smithi125 conmon[19936]: audit 2021-03-24T22:01 2021-03-24T22:01:46.571 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:46 smithi125 conmon[19936]: :46.031689+0000 mon.c (mon.2) 84 : audit 2021-03-24T22:01:46.571 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:46 smithi125 conmon[19936]: [INF] from='client.? v1:172.21.15.125:0/276351450' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:01:46.571 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:46 smithi125 conmon[19936]: audit 2021-03-24T22:01:46.032156 2021-03-24T22:01:46.572 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:46 smithi125 conmon[19936]: +0000 mon.a (mon.0) 1519 : audit [INF] from='client.35399 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:01:46.572 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:46 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:01:46.572 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:46 smithi125 conmon[19936]: T22:01:46.284647+0000 mon.a (mon.0) 2021-03-24T22:01:46.572 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:46 smithi125 conmon[19936]: 1520 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T22:01:46.573 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:46 smithi125 conmon[19936]: audit 2021-03-24T22:01:46. 2021-03-24T22:01:46.573 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:46 smithi125 conmon[19936]: 289917+0000 mon.a (mon.0) 1521 : audit [INF] 2021-03-24T22:01:46.573 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:46 smithi125 conmon[19936]: from='client.35399 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T22:01:46.573 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:46 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:01:46.574 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:46 smithi125 conmon[19936]: 03-24T22:01:46.290011+0000 mon.a (mon.0) 1522 : cluster 2021-03-24T22:01:46.574 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:46 smithi125 conmon[19936]: [DBG] osdmap e536: 8 total, 8 up, 7 in 2021-03-24T22:01:46.574 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:46 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:01:46.575 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:46 smithi125 conmon[24927]: 01:44.887978+0000 mgr.y (mgr.14144) 539 : cluster [DBG] pgmap v964: 168 pgs: 1 active+recovering, 8 remapped+peering, 1 peering, 4 active+remapped+backfill_toofull, 154 active+clean; 523 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 7.9 MiB/s rd, 1.8 MiB/s wr, 8 op/s; 87/1710 objects misplaced (5.088%); 468 B/s, 3 objects/s recovering 2021-03-24T22:01:46.575 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:46 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:01:46.575 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:46 smithi125 conmon[24927]: -03-24T22:01:45 2021-03-24T22:01:46.576 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:46 smithi125 conmon[24927]: .366588+0000 mon.a (mon. 2021-03-24T22:01:46.576 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:46 smithi125 conmon[24927]: 0) 1518 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2021-03-24T22:01:46.576 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:46 smithi125 conmon[24927]: 2021-03-24T22:01:46.576 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:46 smithi125 conmon[24927]: audit 2021-03-24T22:01:46.576 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:46 smithi125 conmon[24927]: 2021-03-24T22: 2021-03-24T22:01:46.577 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:46 smithi125 conmon[24927]: 01:46.031689+0000 mon.c 2021-03-24T22:01:46.577 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:46 smithi125 conmon[24927]: (mon.2) 84 : audit 2021-03-24T22:01:46.577 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:46 smithi125 conmon[24927]: [INF] from='client.? v1:172.21.15.125:0/276351450' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:01:46.577 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:46 smithi125 conmon[24927]: audit 2021-03-24T22:01:46.578 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:46 smithi125 conmon[24927]: 2021-03-24T22 2021-03-24T22:01:46.578 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:46 smithi125 conmon[24927]: :01:46.032156+0000 mon.a 2021-03-24T22:01:46.578 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:46 smithi125 conmon[24927]: (mon.0) 1519 : audit 2021-03-24T22:01:46.578 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:46 smithi125 conmon[24927]: [INF] from='client.35399 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:01:46.579 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:46 smithi125 conmon[24927]: cluster 2021-03-24T22:01:46.579 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:46 smithi125 conmon[24927]: 2021-03-24T22:01 2021-03-24T22:01:46.579 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:46 smithi125 conmon[24927]: :46.284647+0000 mon.a ( 2021-03-24T22:01:46.579 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:46 smithi125 conmon[24927]: mon.0) 1520 : cluster 2021-03-24T22:01:46.579 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:46 smithi125 conmon[24927]: [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T22:01:46.580 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:46 smithi125 conmon[24927]: audit 2021 2021-03-24T22:01:46.580 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:46 smithi125 conmon[24927]: -03-24T22:01: 2021-03-24T22:01:46.580 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:46 smithi125 conmon[24927]: 46.289917+0000 mon.a (mon. 2021-03-24T22:01:46.580 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:46 smithi125 conmon[24927]: 0) 1521 : audit [INF] 2021-03-24T22:01:46.581 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:46 smithi125 conmon[24927]: from='client.35399 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T22:01:46.581 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:46 smithi125 conmon[24927]: cluster 2021-03-24T22:01:46.581 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:46 smithi125 conmon[24927]: 2021-03-24T22: 2021-03-24T22:01:46.581 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:46 smithi125 conmon[24927]: 01:46.290011+0000 2021-03-24T22:01:46.582 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:46 smithi125 conmon[24927]: mon.a (mon.0) 1522 : 2021-03-24T22:01:46.582 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:46 smithi125 conmon[24927]: cluster [DBG] osdmap e536: 8 total, 8 up, 7 in 2021-03-24T22:01:46.627 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:46.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:46.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:01:46.731 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:46.776 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:46.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:46.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:01:46.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:46 smithi142 conmon[20125]: cluster 2021-03-24T22:01:44.887978+0000 mgr.y (mgr.14144) 539 : cluster [DBG] 2021-03-24T22:01:46.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:46 smithi142 conmon[20125]: pgmap v964: 168 pgs: 1 active+recovering, 8 remapped+peering, 1 peering, 4 active+remapped+backfill_toofull, 154 active+clean; 523 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 7.9 MiB/s rd, 1.8 MiB/s wr, 8 op/s; 87/1710 objects misplaced (5.088%); 468 B/s, 3 objects/s recovering 2021-03-24T22:01:46.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:46 smithi142 conmon[20125]: cluster 2021-03-24T22:01:45.366588+0000 mon.a (mon.0 2021-03-24T22:01:46.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:46 smithi142 conmon[20125]: ) 1518 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2021-03-24T22:01:46.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:46 smithi142 conmon[20125]: audit 2021 2021-03-24T22:01:46.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:46 smithi142 conmon[20125]: -03-24T22:01:46. 2021-03-24T22:01:46.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:46 smithi142 conmon[20125]: 031689+0000 mon.c (mon.2) 84 2021-03-24T22:01:46.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:46 smithi142 conmon[20125]: : audit [INF] from='client.? v1:172.21.15.125:0/276351450' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:01:46.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:46 smithi142 conmon[20125]: audit 2021 2021-03-24T22:01:46.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:46 smithi142 conmon[20125]: -03-24T22:01:46 2021-03-24T22:01:46.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:46 smithi142 conmon[20125]: .032156+0000 mon.a (mon.0 2021-03-24T22:01:46.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:46 smithi142 conmon[20125]: ) 1519 : audit [INF] from='client.35399 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:01:46.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:46 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:01:46.872 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:46 smithi142 conmon[20125]: -03-24T22:01:46 2021-03-24T22:01:46.872 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:46 smithi142 conmon[20125]: .284647+0000 mon.a (mon.0 2021-03-24T22:01:46.873 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:46 smithi142 conmon[20125]: ) 1520 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T22:01:46.873 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:46 smithi142 conmon[20125]: audit 2021 2021-03-24T22:01:46.873 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:46 smithi142 conmon[20125]: -03-24T22:01:46 2021-03-24T22:01:46.874 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:46 smithi142 conmon[20125]: .289917+0000 mon.a (mon.0 2021-03-24T22:01:46.874 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:46 smithi142 conmon[20125]: ) 1521 : audit [INF] from='client.35399 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T22:01:46.875 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:46 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:01:46.875 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:46 smithi142 conmon[20125]: -03-24T22:01:46. 2021-03-24T22:01:46.876 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:46 smithi142 conmon[20125]: 290011+0000 mon.a (mon.0) 2021-03-24T22:01:46.876 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:46 smithi142 conmon[20125]: 1522 : cluster [DBG] osdmap e536: 8 total, 8 up, 7 in 2021-03-24T22:01:46.881 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:46.935 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:46.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:46.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:01:47.040 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:47.102 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:47.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:47.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:01:47.208 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:47.253 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:47.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:47.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:01:47.300 INFO:tasks.rados.rados.0.smithi125.stdout:2028: done (0 left) 2021-03-24T22:01:47.300 INFO:tasks.rados.rados.0.smithi125.stdout:2029: read oid 41 snap 216 2021-03-24T22:01:47.300 INFO:tasks.rados.rados.0.smithi125.stdout:2029: expect (ObjNum 690 snap 216 seq_num 690) 2021-03-24T22:01:47.300 INFO:tasks.rados.rados.0.smithi125.stdout:2030: snap_remove snap 206 2021-03-24T22:01:47.356 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:47.408 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:47.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:47.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:01:47.514 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:47.575 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:47.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:47.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:01:47.681 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:47.725 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:47.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:47.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:01:47.830 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:47.873 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:47.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:47.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:01:47.977 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:48.025 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:48.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:48.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:01:48.130 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:48.187 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:48.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:48.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:01:48.294 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:48.305 INFO:tasks.rados.rados.0.smithi125.stdout:2030: done (1 left) 2021-03-24T22:01:48.306 INFO:tasks.rados.rados.0.smithi125.stdout:2031: read oid 47 snap -1 2021-03-24T22:01:48.306 INFO:tasks.rados.rados.0.smithi125.stdout:2031: expect (ObjNum 466 snap 131 seq_num 466) 2021-03-24T22:01:48.306 INFO:tasks.rados.rados.0.smithi125.stdout:2032: write oid 38 current snap is 225 2021-03-24T22:01:48.306 INFO:tasks.rados.rados.0.smithi125.stdout:2032: seq_num 712 ranges {766584=798117,2098238=615574,3169759=460379} 2021-03-24T22:01:48.316 INFO:tasks.rados.rados.0.smithi125.stdout:2032: writing smithi12556470-38 from 766584 to 1564701 tid 1 2021-03-24T22:01:48.318 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:48 smithi125 conmon[19936]: cluster 2021-03-24T22:01:46.888647+0000 mgr.y ( 2021-03-24T22:01:48.318 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:48 smithi125 conmon[19936]: mgr.14144) 540 : cluster [DBG] pgmap v967: 168 pgs: 3 peering, 2 active+undersized+remapped, 1 active+recovering, 8 remapped+peering, 5 active+remapped+backfill_toofull, 149 active+clean; 525 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 407 KiB/s wr, 4 op/s; 107/1716 objects misplaced (6.235%); 3.8 MiB/s, 24 objects/s recovering 2021-03-24T22:01:48.318 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:48 smithi125 conmon[19936]: cluster 2021-03-24T22:01:47.292060+0000 mon.a (mon.0) 1523 : cluster [DBG] osdmap e537: 8 total, 8 up, 7 in 2021-03-24T22:01:48.318 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:48 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:01:48.319 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:48 smithi125 conmon[19936]: 03-24T22:01:47.381234+0000 mon.a (mon.0) 1524 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:01:48.319 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:48 smithi125 conmon[24927]: cluster 2021-03-24T22:01:48.319 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:48 smithi125 conmon[24927]: 2021-03-24T22:01:48.319 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:48 smithi125 conmon[24927]: 2021-03-24T 2021-03-24T22:01:48.320 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:48 smithi125 conmon[24927]: 22:01:46.888647+0000 mgr.y (mgr.14144) 540 : cluster [DBG] pgmap v967: 168 pgs: 3 peering, 2 active+undersized+remapped, 1 active+recovering, 8 remapped+peering, 5 active+remapped+backfill_toofull, 149 active+clean; 525 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 407 KiB/s wr, 4 op/s; 107/1716 objects misplaced (6.235%); 3.8 MiB/s, 24 objects/s recovering 2021-03-24T22:01:48.320 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:48 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T22:01:48.320 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:48 smithi125 conmon[24927]: -24T22:01:47.292060+0000 mon.a (mon.0) 1523 2021-03-24T22:01:48.320 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:48 smithi125 conmon[24927]: : cluster [DBG] osdmap e537: 8 total, 8 up, 7 in 2021-03-24T22:01:48.321 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:48 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T22:01:48.321 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:48 smithi125 conmon[24927]: 22:01:47.381234+0000 mon.a (mon.0 2021-03-24T22:01:48.321 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:48 smithi125 conmon[24927]: ) 1524 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:01:48.324 INFO:tasks.rados.rados.0.smithi125.stdout:2032: writing smithi12556470-38 from 2098238 to 2713812 tid 2 2021-03-24T22:01:48.330 INFO:tasks.rados.rados.0.smithi125.stdout:2032: writing smithi12556470-38 from 3169759 to 3630138 tid 3 2021-03-24T22:01:48.331 INFO:tasks.rados.rados.0.smithi125.stdout:2033: snap_create 2021-03-24T22:01:48.334 INFO:tasks.rados.rados.0.smithi125.stdout:2029: expect (ObjNum 690 snap 216 seq_num 690) 2021-03-24T22:01:48.348 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:48.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:48.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:01:48.453 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:48.495 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:48.630 INFO:tasks.rados.rados.0.smithi125.stdout:2031: expect (ObjNum 466 snap 131 seq_num 466) 2021-03-24T22:01:48.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:48.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:01:48.632 INFO:tasks.rados.rados.0.smithi125.stdout:2032: finishing write tid 1 to smithi12556470-38 2021-03-24T22:01:48.632 INFO:tasks.rados.rados.0.smithi125.stdout:2032: finishing write tid 2 to smithi12556470-38 2021-03-24T22:01:48.633 INFO:tasks.rados.rados.0.smithi125.stdout:2032: finishing write tid 3 to smithi12556470-38 2021-03-24T22:01:48.633 INFO:tasks.rados.rados.0.smithi125.stdout:2032: finishing write tid 4 to smithi12556470-38 2021-03-24T22:01:48.633 INFO:tasks.rados.rados.0.smithi125.stdout:2032: finishing write tid 5 to smithi12556470-38 2021-03-24T22:01:48.633 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 38 v 140 (ObjNum 712 snap 225 seq_num 712) dirty exists 2021-03-24T22:01:48.633 INFO:tasks.rados.rados.0.smithi125.stdout:2032: left oid 38 (ObjNum 712 snap 225 seq_num 712) 2021-03-24T22:01:48.634 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:48 smithi142 conmon[20125]: cluster 2021-03-24T22:01:48.635 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:48 smithi142 conmon[20125]: 2021-03-24T22:01:46.888647 2021-03-24T22:01:48.636 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:48 smithi142 conmon[20125]: +0000 mgr.y (mgr.14144) 540 : cluster [DBG] pgmap v967: 168 pgs: 3 peering, 2 active+undersized+remapped, 1 active+recovering, 8 remapped+peering, 5 active+remapped+backfill_toofull, 149 active+clean; 525 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 407 KiB/s wr, 4 op/s; 107/1716 objects misplaced (6.235%); 3.8 MiB/s, 24 objects/s recovering 2021-03-24T22:01:48.636 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:48 smithi142 conmon[20125]: cluster 2021-03-24T22:01:47.292060+0000 mon.a 2021-03-24T22:01:48.636 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:48 smithi142 conmon[20125]: (mon.0) 1523 : cluster [DBG] osdmap e537: 8 total, 8 up, 7 in 2021-03-24T22:01:48.637 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:48 smithi142 conmon[20125]: cluster 2021-03-24T22: 2021-03-24T22:01:48.637 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:48 smithi142 conmon[20125]: 01:47.381234+0000 mon.a (mon.0) 1524 : cluster [WRN] 2021-03-24T22:01:48.637 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:48 smithi142 conmon[20125]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:01:48.733 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:48.793 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:48.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:48.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:01:48.832 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.3 dump_ops_in_flight 2021-03-24T22:01:48.898 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:48.946 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:48.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:48.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:01:49.051 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:49.109 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:49.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:49.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:01:49.199 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:01:49.215 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:49.273 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T22:01:49.279 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:49.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:49.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:01:49.299 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T22:01:49.327 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[19936]: cluster 2021-03-24T22:01:49.327 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[19936]: 2021-03-24T22:01:49.328 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[19936]: 2021 2021-03-24T22:01:49.328 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[19936]: - 2021-03-24T22:01:49.328 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[19936]: 03 2021-03-24T22:01:49.329 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[19936]: - 2021-03-24T22:01:49.329 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[19936]: 24 2021-03-24T22:01:49.329 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[19936]: T 2021-03-24T22:01:49.330 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[19936]: 22 2021-03-24T22:01:49.330 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[19936]: : 2021-03-24T22:01:49.330 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[19936]: 01:48.301707+0000 mon.a (mon 2021-03-24T22:01:49.331 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[19936]: .0) 1525 : cluster [DBG] osdmap e538: 8 total, 8 up, 7 in 2021-03-24T22:01:49.331 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[19936]: audit 2021-03-24T22:01:49.331 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[19936]: 2021-03-24T22:01:49.247424+0000 mon.a ( 2021-03-24T22:01:49.331 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[19936]: mon.0) 1526 : audit [DBG] from='client.? v1:172.21.15.125:0/2702050667' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:01:49.334 INFO:tasks.rados.rados.0.smithi125.stdout:2029: done (3 left) 2021-03-24T22:01:49.334 INFO:tasks.rados.rados.0.smithi125.stdout:2031: done (2 left) 2021-03-24T22:01:49.334 INFO:tasks.rados.rados.0.smithi125.stdout:2032: done (1 left) 2021-03-24T22:01:49.334 INFO:tasks.rados.rados.0.smithi125.stdout:2033: done (0 left) 2021-03-24T22:01:49.335 INFO:tasks.rados.rados.0.smithi125.stdout:2034: rollback oid 8 current snap is 226 2021-03-24T22:01:49.335 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 8 to 209 2021-03-24T22:01:49.335 INFO:tasks.rados.rados.0.smithi125.stdout:2035: snap_create 2021-03-24T22:01:49.338 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T22:01:49.342 INFO:tasks.rados.rados.0.smithi125.stdout:2034: finishing rollback tid 0 to smithi12556470-8 2021-03-24T22:01:49.342 INFO:tasks.rados.rados.0.smithi125.stdout:2034: finishing rollback tid 1 to smithi12556470-8 2021-03-24T22:01:49.342 INFO:tasks.rados.rados.0.smithi125.stdout:2034: finishing rollback tid 2 to smithi12556470-8 2021-03-24T22:01:49.343 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 8 v 153 (ObjNum 669 snap 208 seq_num 669) dirty exists 2021-03-24T22:01:49.359 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T22:01:49.384 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:49.393 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T22:01:49.434 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T22:01:49.437 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:49.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:49.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:01:49.465 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T22:01:49.489 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T22:01:49.490 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-24T22:01:49.491 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-24T22:01:49.491 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-24T22:01:49.491 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-24T22:01:49.491 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-24T22:01:49.491 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-24T22:01:49.492 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-24T22:01:49.492 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-24T22:01:49.542 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:49.599 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:49.602 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:49 smithi142 conmon[20125]: cluster 2021-03-24T22:01:48.301707+0000 mon.a (mon.0) 2021-03-24T22:01:49.603 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:49 smithi142 conmon[20125]: 1525 : cluster [DBG] osdmap e538: 8 total, 8 up, 7 in 2021-03-24T22:01:49.603 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:49 smithi142 conmon[20125]: audit 2021-03-24T22:01:49.247424+0000 mon.a 2021-03-24T22:01:49.603 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:49 smithi142 conmon[20125]: (mon.0) 1526 : audit [DBG] from='client.? v1:172.21.15.125:0/2702050667' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:01:49.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:49.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:01:49.608 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: cluster 2021-03-24T22:01:49.608 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: 2021-03-24T22:01:49.608 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: 2021 2021-03-24T22:01:49.609 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: - 2021-03-24T22:01:49.609 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: 03 2021-03-24T22:01:49.609 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: - 2021-03-24T22:01:49.609 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: 24 2021-03-24T22:01:49.610 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: T 2021-03-24T22:01:49.610 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: 22 2021-03-24T22:01:49.610 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: : 2021-03-24T22:01:49.611 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: 01 2021-03-24T22:01:49.611 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: : 2021-03-24T22:01:49.611 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: 48 2021-03-24T22:01:49.612 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: . 2021-03-24T22:01:49.612 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: 301707 2021-03-24T22:01:49.612 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: +0000 2021-03-24T22:01:49.612 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: 2021-03-24T22:01:49.613 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: mon.a 2021-03-24T22:01:49.613 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: ( 2021-03-24T22:01:49.613 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: mon 2021-03-24T22:01:49.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: . 2021-03-24T22:01:49.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: 0 2021-03-24T22:01:49.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: ) 2021-03-24T22:01:49.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: 1525 2021-03-24T22:01:49.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: : 2021-03-24T22:01:49.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: cluster 2021-03-24T22:01:49.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: 2021-03-24T22:01:49.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: [DBG] 2021-03-24T22:01:49.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: 2021-03-24T22:01:49.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: osdmap e538: 8 total, 8 up, 7 in 2021-03-24T22:01:49.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: 2021-03-24T22:01:49.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: audit 2021-03-24T22:01:49.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: 2021-03-24T22:01:49.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: 2021 2021-03-24T22:01:49.618 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: - 2021-03-24T22:01:49.618 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: 03 2021-03-24T22:01:49.618 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: - 2021-03-24T22:01:49.619 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: 24 2021-03-24T22:01:49.619 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: T 2021-03-24T22:01:49.619 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: 22 2021-03-24T22:01:49.620 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: : 2021-03-24T22:01:49.620 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: 01 2021-03-24T22:01:49.620 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: : 2021-03-24T22:01:49.620 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: 49 2021-03-24T22:01:49.621 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: . 2021-03-24T22:01:49.621 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: 247424 2021-03-24T22:01:49.621 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: +0000 2021-03-24T22:01:49.622 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: 2021-03-24T22:01:49.622 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: mon.a 2021-03-24T22:01:49.622 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: ( 2021-03-24T22:01:49.623 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: mon 2021-03-24T22:01:49.623 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: . 2021-03-24T22:01:49.623 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: 0 2021-03-24T22:01:49.623 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: ) 2021-03-24T22:01:49.624 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: 1526 2021-03-24T22:01:49.624 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: : 2021-03-24T22:01:49.624 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: audit 2021-03-24T22:01:49.625 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: 2021-03-24T22:01:49.625 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: [DBG] 2021-03-24T22:01:49.625 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: 2021-03-24T22:01:49.626 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: from='client.? v1:172.21.15.125:0/2702050667' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:01:49.626 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[24927]: 2021-03-24T22:01:49.706 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:49.752 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:49.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:49.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:01:49.857 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:49.870 INFO:journalctl@ceph.osd.1.smithi125.stdout:Mar 24 22:01:49 smithi125 conmon[31297]: debug 2021-03-24T22:01:49.738+0000 7f8778d65700 -1 osd.1 pg_epoch: 539 pg[5.1d( empty local-lis/les=505/506 n=0 ec=64/64 lis/c=505/505 les/c/f=506/506/0 sis=505) [1,4,0] r=0 lpr=505 crt=0'0 mlcod 0'0 active+clean] scrubber pg(5.1d) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 4 (0x56142b219e40) 2021-03-24T22:01:49.906 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:49.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:49.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:01:50.010 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:50.067 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:50.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:50.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:01:50.283 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:50.289 INFO:tasks.rados.rados.0.smithi125.stdout:2034: done (1 left) 2021-03-24T22:01:50.290 INFO:tasks.rados.rados.0.smithi125.stdout:2035: done (0 left) 2021-03-24T22:01:50.290 INFO:tasks.rados.rados.0.smithi125.stdout:2036: write oid 27 current snap is 227 2021-03-24T22:01:50.290 INFO:tasks.rados.rados.0.smithi125.stdout:2036: seq_num 713 ranges {783391=473366,1756720=475585,2797182=788004} 2021-03-24T22:01:50.292 INFO:tasks.rados.rados.0.smithi125.stdout:2036: writing smithi12556470-27 from 783391 to 1256757 tid 1 2021-03-24T22:01:50.298 INFO:tasks.rados.rados.0.smithi125.stdout:2036: writing smithi12556470-27 from 1756720 to 2232305 tid 2 2021-03-24T22:01:50.307 INFO:tasks.rados.rados.0.smithi125.stdout:2036: writing smithi12556470-27 from 2797182 to 3585186 tid 3 2021-03-24T22:01:50.307 INFO:tasks.rados.rados.0.smithi125.stdout:2037: read oid 15 snap -1 2021-03-24T22:01:50.307 INFO:tasks.rados.rados.0.smithi125.stdout:2037: expect deleted 2021-03-24T22:01:50.308 INFO:tasks.rados.rados.0.smithi125.stdout:2038: write oid 44 current snap is 227 2021-03-24T22:01:50.308 INFO:tasks.rados.rados.0.smithi125.stdout:2036: finishing write tid 1 to smithi12556470-27 2021-03-24T22:01:50.308 INFO:tasks.rados.rados.0.smithi125.stdout:2038: seq_num 714 ranges {400198=548615,1415202=584996} 2021-03-24T22:01:50.312 INFO:tasks.rados.rados.0.smithi125.stdout:2038: writing smithi12556470-44 from 400198 to 948813 tid 1 2021-03-24T22:01:50.319 INFO:tasks.rados.rados.0.smithi125.stdout:2038: writing smithi12556470-44 from 1415202 to 2000198 tid 2 2021-03-24T22:01:50.319 INFO:tasks.rados.rados.0.smithi125.stdout:2039: rollback oid 6 current snap is 227 2021-03-24T22:01:50.320 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 6 to 225 2021-03-24T22:01:50.320 INFO:tasks.rados.rados.0.smithi125.stdout:2040: read oid 31 snap -1 2021-03-24T22:01:50.320 INFO:tasks.rados.rados.0.smithi125.stdout:2040: expect deleted 2021-03-24T22:01:50.320 INFO:tasks.rados.rados.0.smithi125.stdout:2041: read oid 43 snap 223 2021-03-24T22:01:50.321 INFO:tasks.rados.rados.0.smithi125.stdout:2041: expect deleted 2021-03-24T22:01:50.321 INFO:tasks.rados.rados.0.smithi125.stdout:2042: snap_remove snap 221 2021-03-24T22:01:50.337 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:50.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:50.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:01:50.435 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 1, 6] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2021-03-24T22:01:50.435 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-24T22:01:50.435 INFO:tasks.thrashosds.thrasher:Adding osd 7 2021-03-24T22:01:50.436 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd in 7 2021-03-24T22:01:50.443 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:50.496 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:50.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:50.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:01:50.602 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:50.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:50 smithi142 conmon[20125]: cluster 2021-03-24T22:01:48.889201+0000 mgr.y (mgr. 2021-03-24T22:01:50.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:50 smithi142 conmon[20125]: 14144) 541 : cluster [DBG] pgmap v970: 168 pgs: 3 peering, 2 active+undersized+remapped, 1 active+recovering, 8 remapped+peering, 5 active+remapped+backfill_toofull, 149 active+clean; 525 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 407 KiB/s wr, 3 op/s; 107/1716 objects misplaced (6.235%); 3.8 MiB/s, 20 objects/s recovering 2021-03-24T22:01:50.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:50 smithi142 conmon[20125]: cluster 2021-03-24T22:01:49.325393+0000 mon.a (mon 2021-03-24T22:01:50.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:50 smithi142 conmon[20125]: .0) 1527 : cluster [DBG] osdmap e539: 8 total, 8 up, 7 in 2021-03-24T22:01:50.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:50 smithi142 conmon[20125]: audit 2021-03 2021-03-24T22:01:50.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:50 smithi142 conmon[20125]: -24T22:01:49.805105+0000 mon.a (mon.0) 1528 : audit 2021-03-24T22:01:50.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:50 smithi142 conmon[20125]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:01:50.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:50 smithi142 conmon[20125]: cluster 2021-03-24T22:01:50.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:50 smithi142 conmon[20125]: 2021-03-24T22:01:50.279285+0000 mon.a (mon. 2021-03-24T22:01:50.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:50 smithi142 conmon[20125]: 0) 1529 : cluster [DBG] osdmap e540: 8 total, 8 up, 7 in 2021-03-24T22:01:50.659 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:50.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[19936]: cluster 2021-03-24T22 2021-03-24T22:01:50.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[19936]: :01:48.889201+0000 mgr.y (mgr.14144) 541 : cluster [DBG] pgmap v970: 168 pgs: 3 peering, 2 active+undersized+remapped, 1 active+recovering, 8 remapped+peering, 5 active+remapped+backfill_toofull, 149 active+clean; 525 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 407 KiB/s wr, 3 op/s; 107/1716 objects misplaced (6.235%); 3.8 MiB/s, 20 objects/s recovering 2021-03-24T22:01:50.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[19936]: cluster 2021-03-24T22:01:50.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[19936]: 2021-03-24T22:01:49.325393+0000 mon.a (mon.0 2021-03-24T22:01:50.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[19936]: ) 1527 : cluster [DBG] osdmap e539: 8 total, 8 up, 7 in 2021-03-24T22:01:50.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[19936]: audit 2021- 2021-03-24T22:01:50.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[19936]: 03-24T22:01:49.805105+0000 mon.a 2021-03-24T22:01:50.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[19936]: (mon.0) 1528 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:01:50.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[19936]: 2021-03-24T22:01:50.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T22:01:50.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[19936]: 22:01:50.279285+0000 mon.a ( 2021-03-24T22:01:50.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[19936]: mon.0) 1529 : cluster [DBG] osdmap e540: 8 total, 8 up, 7 in 2021-03-24T22:01:50.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:50.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: cluster 2021-03-24T22:01:50.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: 2021-03-24T22:01:50.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: 2021 2021-03-24T22:01:50.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: - 2021-03-24T22:01:50.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: 03 2021-03-24T22:01:50.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: - 2021-03-24T22:01:50.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: 24 2021-03-24T22:01:50.672 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: T 2021-03-24T22:01:50.672 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: 22 2021-03-24T22:01:50.672 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: : 2021-03-24T22:01:50.673 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: 01 2021-03-24T22:01:50.673 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: : 2021-03-24T22:01:50.674 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: 48 2021-03-24T22:01:50.674 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: . 2021-03-24T22:01:50.675 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: 889201 2021-03-24T22:01:50.675 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: +0000 2021-03-24T22:01:50.676 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: 2021-03-24T22:01:50.676 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: mgr.y 2021-03-24T22:01:50.676 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: ( 2021-03-24T22:01:50.677 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: mgr 2021-03-24T22:01:50.677 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: . 2021-03-24T22:01:50.678 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: 14144 2021-03-24T22:01:50.678 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: ) 2021-03-24T22:01:50.679 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: 541 2021-03-24T22:01:50.679 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: : 2021-03-24T22:01:50.679 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: cluster 2021-03-24T22:01:50.680 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: 2021-03-24T22:01:50.680 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: [DBG] 2021-03-24T22:01:50.681 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: 2021-03-24T22:01:50.681 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: pgmap v970: 168 pgs: 3 peering, 2 active+undersized+remapped, 1 active+recovering, 8 remapped+peering, 5 active+remapped+backfill_toofull, 149 active+clean; 525 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 407 KiB/s wr, 3 op/s; 107/1716 objects misplaced (6.235%); 3.8 MiB/s, 20 objects/s recovering 2021-03-24T22:01:50.682 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: 2021-03-24T22:01:50.682 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: cluster 2021-03-24T22:01:50.683 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: 2021-03-24T22:01:50.683 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: 2021 2021-03-24T22:01:50.683 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: - 2021-03-24T22:01:50.684 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: 03 2021-03-24T22:01:50.684 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: - 2021-03-24T22:01:50.685 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: 24 2021-03-24T22:01:50.685 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: T 2021-03-24T22:01:50.686 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: 22 2021-03-24T22:01:50.686 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: : 2021-03-24T22:01:50.687 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: 01 2021-03-24T22:01:50.687 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: : 2021-03-24T22:01:50.688 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: 49 2021-03-24T22:01:50.688 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: . 2021-03-24T22:01:50.688 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: 325393 2021-03-24T22:01:50.689 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: +0000 2021-03-24T22:01:50.689 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: 2021-03-24T22:01:50.690 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: mon.a 2021-03-24T22:01:50.690 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: ( 2021-03-24T22:01:50.691 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: mon 2021-03-24T22:01:50.691 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: . 2021-03-24T22:01:50.692 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: 0 2021-03-24T22:01:50.692 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: ) 2021-03-24T22:01:50.693 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: 1527 2021-03-24T22:01:50.693 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: : 2021-03-24T22:01:50.694 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: cluster 2021-03-24T22:01:50.694 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: 2021-03-24T22:01:50.694 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: [DBG] 2021-03-24T22:01:50.695 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: 2021-03-24T22:01:50.695 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: osdmap e539: 8 total, 8 up, 7 in 2021-03-24T22:01:50.696 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: 2021-03-24T22:01:50.696 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: audit 2021-03-24T22:01:49.805105+0000 mon.a (mon. 2021-03-24T22:01:50.697 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: 0) 1528 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:01:50.697 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:01:50.698 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: -03-24T22:01:50 2021-03-24T22:01:50.698 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: .279285+0000 mon.a (mon.0) 1529 : cluster 2021-03-24T22:01:50.698 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:50 smithi125 conmon[24927]: [DBG] osdmap e540: 8 total, 8 up, 7 in 2021-03-24T22:01:50.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:01:50.801 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:50.850 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:50.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:50.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:01:50.861 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd set noscrub 2021-03-24T22:01:50.956 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:51.055 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:51.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:51.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:01:51.161 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:51.215 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:51.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:51.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:01:51.308 INFO:tasks.rados.rados.0.smithi125.stdout:2042: done (6 left) 2021-03-24T22:01:51.308 INFO:tasks.rados.rados.0.smithi125.stdout:2043: write (excl) oid 20 current snap is 227 2021-03-24T22:01:51.309 INFO:tasks.rados.rados.0.smithi125.stdout:2043: seq_num 715 ranges {417005=740217,1590037=426968} 2021-03-24T22:01:51.316 INFO:tasks.rados.rados.0.smithi125.stdout:2043: writing smithi12556470-20 from 417005 to 1157222 tid 1 2021-03-24T22:01:51.321 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:51.323 INFO:tasks.rados.rados.0.smithi125.stdout:2043: writing smithi12556470-20 from 1590037 to 2017005 tid 2 2021-03-24T22:01:51.323 INFO:tasks.rados.rados.0.smithi125.stdout:2044: write (excl) oid 39 current snap is 227 2021-03-24T22:01:51.323 INFO:tasks.rados.rados.0.smithi125.stdout:2036: finishing write tid 2 to smithi12556470-27 2021-03-24T22:01:51.324 INFO:tasks.rados.rados.0.smithi125.stdout:2036: finishing write tid 3 to smithi12556470-27 2021-03-24T22:01:51.324 INFO:tasks.rados.rados.0.smithi125.stdout:2036: finishing write tid 4 to smithi12556470-27 2021-03-24T22:01:51.324 INFO:tasks.rados.rados.0.smithi125.stdout:2036: finishing write tid 5 to smithi12556470-27 2021-03-24T22:01:51.324 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 27 v 559 (ObjNum 713 snap 227 seq_num 713) dirty exists 2021-03-24T22:01:51.324 INFO:tasks.rados.rados.0.smithi125.stdout:2036: left oid 27 (ObjNum 713 snap 227 seq_num 713) 2021-03-24T22:01:51.325 INFO:tasks.rados.rados.0.smithi125.stdout:2038: finishing write tid 1 to smithi12556470-44 2021-03-24T22:01:51.325 INFO:tasks.rados.rados.0.smithi125.stdout:2039: finishing rollback tid 1 to smithi12556470-6 2021-03-24T22:01:51.325 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 6 v 398 (ObjNum 1117583856 snap 21916 seq_num 1116535712) dirty dne 2021-03-24T22:01:51.325 INFO:tasks.rados.rados.0.smithi125.stdout:2038: finishing write tid 2 to smithi12556470-44 2021-03-24T22:01:51.326 INFO:tasks.rados.rados.0.smithi125.stdout:2038: finishing write tid 3 to smithi12556470-44 2021-03-24T22:01:51.326 INFO:tasks.rados.rados.0.smithi125.stdout:2038: finishing write tid 4 to smithi12556470-44 2021-03-24T22:01:51.326 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 44 v 473 (ObjNum 714 snap 227 seq_num 714) dirty exists 2021-03-24T22:01:51.326 INFO:tasks.rados.rados.0.smithi125.stdout:2038: left oid 44 (ObjNum 714 snap 227 seq_num 714) 2021-03-24T22:01:51.326 INFO:tasks.rados.rados.0.smithi125.stdout:2044: seq_num 716 ranges {433812=415466,1648519=385293} 2021-03-24T22:01:51.327 INFO:tasks.rados.rados.0.smithi125.stdout:2044: writing smithi12556470-39 from 433812 to 849278 tid 1 2021-03-24T22:01:51.336 INFO:tasks.rados.rados.0.smithi125.stdout:2044: writing smithi12556470-39 from 1648519 to 2033812 tid 2 2021-03-24T22:01:51.337 INFO:tasks.rados.rados.0.smithi125.stdout:2036: done (7 left) 2021-03-24T22:01:51.337 INFO:tasks.rados.rados.0.smithi125.stdout:2038: done (6 left) 2021-03-24T22:01:51.337 INFO:tasks.rados.rados.0.smithi125.stdout:2039: done (5 left) 2021-03-24T22:01:51.338 INFO:tasks.rados.rados.0.smithi125.stdout:2040: done (4 left) 2021-03-24T22:01:51.338 INFO:tasks.rados.rados.0.smithi125.stdout:2041: done (3 left) 2021-03-24T22:01:51.338 INFO:tasks.rados.rados.0.smithi125.stdout:2045: read oid 34 snap -1 2021-03-24T22:01:51.338 INFO:tasks.rados.rados.0.smithi125.stdout:2045: expect (ObjNum 610 snap 185 seq_num 610) 2021-03-24T22:01:51.339 INFO:tasks.rados.rados.0.smithi125.stdout:2046: copy_from oid 48 from oid 7 current snap is 227 2021-03-24T22:01:51.339 INFO:tasks.rados.rados.0.smithi125.stdout:2043: finishing write tid 1 to smithi12556470-20 2021-03-24T22:01:51.339 INFO:tasks.rados.rados.0.smithi125.stdout:2044: finishing write tid 1 to smithi12556470-39 2021-03-24T22:01:51.339 INFO:tasks.rados.rados.0.smithi125.stdout:2043: finishing write tid 2 to smithi12556470-20 2021-03-24T22:01:51.340 INFO:tasks.rados.rados.0.smithi125.stdout:2043: finishing write tid 3 to smithi12556470-20 2021-03-24T22:01:51.340 INFO:tasks.rados.rados.0.smithi125.stdout:2043: finishing write tid 4 to smithi12556470-20 2021-03-24T22:01:51.340 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 20 v 716 (ObjNum 715 snap 227 seq_num 715) dirty exists 2021-03-24T22:01:51.340 INFO:tasks.rados.rados.0.smithi125.stdout:2043: left oid 20 (ObjNum 715 snap 227 seq_num 715) 2021-03-24T22:01:51.341 INFO:tasks.rados.rados.0.smithi125.stdout:2037: done (4 left) 2021-03-24T22:01:51.341 INFO:tasks.rados.rados.0.smithi125.stdout:2043: done (3 left) 2021-03-24T22:01:51.341 INFO:tasks.rados.rados.0.smithi125.stdout:2047: copy_from oid 11 from oid 19 current snap is 227 2021-03-24T22:01:51.341 INFO:tasks.rados.rados.0.smithi125.stdout:2048: read oid 44 snap -1 2021-03-24T22:01:51.341 INFO:tasks.rados.rados.0.smithi125.stdout:2048: expect (ObjNum 714 snap 227 seq_num 714) 2021-03-24T22:01:51.342 INFO:tasks.rados.rados.0.smithi125.stdout:2049: read oid 50 snap -1 2021-03-24T22:01:51.342 INFO:tasks.rados.rados.0.smithi125.stdout:2049: expect (ObjNum 710 snap 223 seq_num 710) 2021-03-24T22:01:51.342 INFO:tasks.rados.rados.0.smithi125.stdout:2050: write (excl) oid 30 current snap is 227 2021-03-24T22:01:51.342 INFO:tasks.rados.rados.0.smithi125.stdout:2050: seq_num 717 ranges {450619=490715,1590648=459971} 2021-03-24T22:01:51.343 INFO:tasks.rados.rados.0.smithi125.stdout:2050: writing smithi12556470-30 from 450619 to 941334 tid 1 2021-03-24T22:01:51.349 INFO:tasks.rados.rados.0.smithi125.stdout:2050: writing smithi12556470-30 from 1590648 to 2050619 tid 2 2021-03-24T22:01:51.350 INFO:tasks.rados.rados.0.smithi125.stdout:2051: snap_create 2021-03-24T22:01:51.350 INFO:tasks.rados.rados.0.smithi125.stdout:2047: finishing copy_from to smithi12556470-11 2021-03-24T22:01:51.350 INFO:tasks.rados.rados.0.smithi125.stdout:2047: got expected ENOENT (src dne) 2021-03-24T22:01:51.351 INFO:tasks.rados.rados.0.smithi125.stdout:2047: finishing copy_from racing read to smithi12556470-11 2021-03-24T22:01:51.351 INFO:tasks.rados.rados.0.smithi125.stdout:2045: expect (ObjNum 610 snap 185 seq_num 610) 2021-03-24T22:01:51.377 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:51.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:51.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:01:51.484 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:51.513 INFO:tasks.rados.rados.0.smithi125.stdout:2044: finishing write tid 2 to smithi12556470-39 2021-03-24T22:01:51.514 INFO:tasks.rados.rados.0.smithi125.stdout:2044: finishing write tid 3 to smithi12556470-39 2021-03-24T22:01:51.514 INFO:tasks.rados.rados.0.smithi125.stdout:2044: finishing write tid 4 to smithi12556470-39 2021-03-24T22:01:51.514 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 39 v 730 (ObjNum 716 snap 227 seq_num 716) dirty exists 2021-03-24T22:01:51.516 INFO:tasks.rados.rados.0.smithi125.stdout:2044: left oid 39 (ObjNum 716 snap 227 seq_num 716) 2021-03-24T22:01:51.516 INFO:tasks.rados.rados.0.smithi125.stdout:2050: finishing write tid 1 to smithi12556470-30 2021-03-24T22:01:51.516 INFO:tasks.rados.rados.0.smithi125.stdout:2048: expect (ObjNum 714 snap 227 seq_num 714) 2021-03-24T22:01:51.563 INFO:tasks.rados.rados.0.smithi125.stdout:2049: expect (ObjNum 710 snap 223 seq_num 710) 2021-03-24T22:01:51.570 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:51.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:51.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:01:51.627 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:01:51.676 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:51.680 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:01:51.720 INFO:tasks.rados.rados.0.smithi125.stdout:2050: finishing write tid 2 to smithi12556470-30 2021-03-24T22:01:51.721 INFO:tasks.rados.rados.0.smithi125.stdout:2050: finishing write tid 3 to smithi12556470-30 2021-03-24T22:01:51.721 INFO:tasks.rados.rados.0.smithi125.stdout:2050: finishing write tid 4 to smithi12556470-30 2021-03-24T22:01:51.722 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 30 v 310 (ObjNum 717 snap 227 seq_num 717) dirty exists 2021-03-24T22:01:51.722 INFO:tasks.rados.rados.0.smithi125.stdout:2050: left oid 30 (ObjNum 717 snap 227 seq_num 717) 2021-03-24T22:01:51.722 INFO:tasks.rados.rados.0.smithi125.stdout:2046: finishing copy_from to smithi12556470-48 2021-03-24T22:01:51.723 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 48 v 239 (ObjNum 588 snap 177 seq_num 588) dirty exists 2021-03-24T22:01:51.723 INFO:tasks.rados.rados.0.smithi125.stdout:2046: finishing copy_from racing read to smithi12556470-48 2021-03-24T22:01:51.733 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:51.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:51.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:01:51.837 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:51.898 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:51.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:51.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:01:52.004 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:52.081 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:52.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:52.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:01:52.157 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-24T22:01:52.158 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T22:01:52.186 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:52.243 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:52.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:52.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:01:52.319 INFO:tasks.rados.rados.0.smithi125.stdout:2044: done (7 left) 2021-03-24T22:01:52.354 INFO:tasks.rados.rados.0.smithi125.stdout:2045: done (6 left) 2021-03-24T22:01:52.355 INFO:tasks.rados.rados.0.smithi125.stdout:2046: done (5 left) 2021-03-24T22:01:52.355 INFO:tasks.rados.rados.0.smithi125.stdout:2047: done (4 left) 2021-03-24T22:01:52.355 INFO:tasks.rados.rados.0.smithi125.stdout:2048: done (3 left) 2021-03-24T22:01:52.355 INFO:tasks.rados.rados.0.smithi125.stdout:2049: done (2 left) 2021-03-24T22:01:52.356 INFO:tasks.rados.rados.0.smithi125.stdout:2050: done (1 left) 2021-03-24T22:01:52.356 INFO:tasks.rados.rados.0.smithi125.stdout:2051: done (0 left) 2021-03-24T22:01:52.356 INFO:tasks.rados.rados.0.smithi125.stdout:2052: copy_from oid 24 from oid 5 current snap is 228 2021-03-24T22:01:52.356 INFO:tasks.rados.rados.0.smithi125.stdout:2053: snap_create 2021-03-24T22:01:52.357 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:52.410 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:52.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:52 smithi125 conmon[19936]: cluster 2021-03-24T22:01:50.889725+0000 mgr.y (mgr 2021-03-24T22:01:52.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:52 smithi125 conmon[19936]: .14144) 542 : cluster [DBG] pgmap v973: 168 pgs: 2 active+recovering+remapped, 1 active+clean+remapped, 1 active+remapped+backfilling, 3 peering, 3 remapped+peering, 4 active+remapped+backfill_toofull, 154 active+clean; 536 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 8.4 MiB/s rd, 2.1 MiB/s wr, 12 op/s; 123/1731 objects misplaced (7.106%); 6.3 MiB/s, 33 objects/s recovering 2021-03-24T22:01:52.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:52 smithi125 conmon[19936]: cluster 2021-03-24T22:01:51.304829+0000 mon.a (mon.0) 1530 : cluster [DBG] osdmap e541: 8 total, 8 up, 7 in 2021-03-24T22:01:52.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:52 smithi125 conmon[19936]: cluster 2021-03-24T22:01:52.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:52 smithi125 conmon[19936]: 2021-03-24T22:01:51.370012+0000 mon.a (mon.0) 1531 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2021-03-24T22:01:52.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:52 smithi125 conmon[19936]: cluster 2021-03-24T22 2021-03-24T22:01:52.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:52 smithi125 conmon[19936]: :01:51.757800+0000 osd.1 (osd.1) 48 : cluster [DBG] 5.1d scrub ok 2021-03-24T22:01:52.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:52 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:01:52.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:52 smithi125 conmon[24927]: 01:50.889725+0000 mgr.y (mgr.14144) 542 : cluster [DBG] pgmap v973: 168 pgs: 2 active+recovering+remapped, 1 active+clean+remapped, 1 active+remapped+backfilling, 3 peering, 3 remapped+peering, 4 active+remapped+backfill_toofull, 154 active+clean; 536 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 8.4 MiB/s rd, 2.1 MiB/s wr, 12 op/s; 123/1731 objects misplaced (7.106%); 6.3 MiB/s, 33 objects/s recovering 2021-03-24T22:01:52.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:52 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:01:52.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:52 smithi125 conmon[24927]: 24T22:01:51.304829+0000 2021-03-24T22:01:52.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:52 smithi125 conmon[24927]: mon.a (mon.0) 1530 : cluster [DBG] osdmap e541: 8 total, 8 up, 7 in 2021-03-24T22:01:52.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:52 smithi125 conmon[24927]: 2021-03-24T22:01:52.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:52 smithi125 conmon[24927]: cluster 2021-03-24T22:01:52.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:52 smithi125 conmon[24927]: 2021-03-24T22 2021-03-24T22:01:52.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:52 smithi125 conmon[24927]: :01:51.370012+0000 mon.a (mon. 2021-03-24T22:01:52.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:52 smithi125 conmon[24927]: 0) 1531 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2021-03-24T22:01:52.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:52 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:01:52.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:52 smithi125 conmon[24927]: 03-24T22:01:51. 2021-03-24T22:01:52.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:52 smithi125 conmon[24927]: 757800+0000 osd.1 (osd.1) 48 : 2021-03-24T22:01:52.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:52 smithi125 conmon[24927]: cluster [DBG] 5.1d scrub ok 2021-03-24T22:01:52.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:52.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:01:52.519 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:52.575 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:52.579 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:52 smithi142 conmon[20125]: cluster 2021-03-24T22:01:50.889725+0000 mgr.y (mgr.14144) 542 : cluster 2021-03-24T22:01:52.579 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:52 smithi142 conmon[20125]: [DBG] pgmap v973: 168 pgs: 2 active+recovering+remapped, 1 active+clean+remapped, 1 active+remapped+backfilling, 3 peering, 3 remapped+peering, 4 active+remapped+backfill_toofull, 154 active+clean; 536 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 8.4 MiB/s rd, 2.1 MiB/s wr, 12 op/s; 123/1731 objects misplaced (7.106%); 6.3 MiB/s, 33 objects/s recovering 2021-03-24T22:01:52.580 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:52 smithi142 conmon[20125]: cluster 2021-03-24T22:01:51.304829+0000 mon.a (mon.0) 1530 : cluster [DBG] osdmap e541: 8 total, 8 up, 7 in 2021-03-24T22:01:52.580 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:52 smithi142 conmon[20125]: cluster 2021-03-24T22:01:51.370012+0000 mon.a (mon.0) 1531 : cluster [INF] 2021-03-24T22:01:52.580 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:52 smithi142 conmon[20125]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2021-03-24T22:01:52.580 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:52 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:01:52.580 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:52 smithi142 conmon[20125]: T22:01:51.757800+0000 osd.1 (osd.1) 48 : cluster [DBG] 5.1d scrub ok 2021-03-24T22:01:52.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:52.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:01:52.683 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:52.822 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:52.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:52.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:01:52.929 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:52.990 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:52.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:52.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:01:53.095 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:53.178 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:53.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:53.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:01:53.283 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:53.336 INFO:tasks.rados.rados.0.smithi125.stdout:2053: done (1 left) 2021-03-24T22:01:53.336 INFO:tasks.rados.rados.0.smithi125.stdout:2054: write oid 31 current snap is 229 2021-03-24T22:01:53.336 INFO:tasks.rados.rados.0.smithi125.stdout:2054: seq_num 718 ranges {467426=565964,1649130=418296} 2021-03-24T22:01:53.338 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.3 dump_blocked_ops 2021-03-24T22:01:53.342 INFO:tasks.rados.rados.0.smithi125.stdout:2054: writing smithi12556470-31 from 467426 to 1033390 tid 1 2021-03-24T22:01:53.350 INFO:tasks.rados.rados.0.smithi125.stdout:2054: writing smithi12556470-31 from 1649130 to 2067426 tid 2 2021-03-24T22:01:53.350 INFO:tasks.rados.rados.0.smithi125.stdout:2055: snap_remove snap 215 2021-03-24T22:01:53.359 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:53.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:53.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:01:53.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:53 smithi125 conmon[24927]: cluster 2021-03-24T22:01:51.981413 2021-03-24T22:01:53.454 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:53 smithi125 conmon[24927]: +0000 osd.6 (osd.6) 33 : cluster [DBG] 6.a scrub ok 2021-03-24T22:01:53.455 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:53 smithi125 conmon[24927]: cluster 2021-03-24T22:01:52.312003 2021-03-24T22:01:53.455 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:53 smithi125 conmon[24927]: +0000 mon.a (mon.0) 1532 : cluster [DBG] osdmap e542: 8 total, 8 up, 7 in 2021-03-24T22:01:53.456 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:53 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:01:53.456 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:53 smithi125 conmon[24927]: 24T22:01:52.461585+0000 osd.0 (osd. 2021-03-24T22:01:53.456 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:53 smithi125 conmon[24927]: 0) 113 : cluster [DBG] 6.10 scrub ok 2021-03-24T22:01:53.457 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:53 smithi125 conmon[19936]: cluster 2021-03-24T22:01:53.457 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:53 smithi125 conmon[19936]: 2021-03-24T22:01 2021-03-24T22:01:53.458 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:53 smithi125 conmon[19936]: :51.981413+0000 osd.6 (osd.6) 33 : cluster [DBG] 6.a scrub ok 2021-03-24T22:01:53.458 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:53 smithi125 conmon[19936]: cluster 2021-03-24T22:01:52 2021-03-24T22:01:53.459 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:53 smithi125 conmon[19936]: .312003+0000 mon.a (mon.0) 1532 : cluster [DBG] osdmap e542: 8 total, 8 up, 7 in 2021-03-24T22:01:53.459 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:53 smithi125 conmon[19936]: cluster 2021-03-24T22:01: 2021-03-24T22:01:53.460 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:53 smithi125 conmon[19936]: 52.461585+0000 osd.0 (osd.0) 113 : 2021-03-24T22:01:53.460 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:53 smithi125 conmon[19936]: cluster [DBG] 6.10 scrub ok 2021-03-24T22:01:53.465 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:53.524 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:53.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:53.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:01:53.600 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-24T22:01:53.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:53 smithi142 conmon[20125]: cluster 2021-03-24T22:01: 2021-03-24T22:01:53.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:53 smithi142 conmon[20125]: 51.981413+0000 osd.6 (osd.6) 33 : cluster [DBG] 6.a scrub ok 2021-03-24T22:01:53.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:53 smithi142 conmon[20125]: cluster 2021-03-24T22:01:52.312003+0000 mon.a (mon. 2021-03-24T22:01:53.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:53 smithi142 conmon[20125]: 0) 1532 : cluster [DBG] osdmap e542: 8 total, 8 up, 7 in 2021-03-24T22:01:53.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:53 smithi142 conmon[20125]: cluster 2021-03-24T22:01:52.461585+0000 osd.0 2021-03-24T22:01:53.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:53 smithi142 conmon[20125]: (osd.0) 113 : cluster [DBG] 6.10 scrub ok 2021-03-24T22:01:53.631 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:53.690 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:53.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:53.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:01:53.799 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:53.862 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:53.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:53.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:01:53.969 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:54.028 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:54.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:54.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:01:54.134 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:54.198 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:54.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:54.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:01:54.303 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:54.334 INFO:tasks.rados.rados.0.smithi125.stdout:2055: done (2 left) 2021-03-24T22:01:54.335 INFO:tasks.rados.rados.0.smithi125.stdout:2056: delete oid 8 current snap is 229 2021-03-24T22:01:54.335 INFO:tasks.rados.rados.0.smithi125.stdout:2052: finishing copy_from to smithi12556470-24 2021-03-24T22:01:54.336 INFO:tasks.rados.rados.0.smithi125.stdout:2052: got expected ENOENT (src dne) 2021-03-24T22:01:54.336 INFO:tasks.rados.rados.0.smithi125.stdout:2052: finishing copy_from racing read to smithi12556470-24 2021-03-24T22:01:54.337 INFO:tasks.rados.rados.0.smithi125.stdout:2054: finishing write tid 1 to smithi12556470-31 2021-03-24T22:01:54.337 INFO:tasks.rados.rados.0.smithi125.stdout:2054: finishing write tid 2 to smithi12556470-31 2021-03-24T22:01:54.338 INFO:tasks.rados.rados.0.smithi125.stdout:2054: finishing write tid 3 to smithi12556470-31 2021-03-24T22:01:54.338 INFO:tasks.rados.rados.0.smithi125.stdout:2054: finishing write tid 4 to smithi12556470-31 2021-03-24T22:01:54.338 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 31 v 1031 (ObjNum 718 snap 229 seq_num 718) dirty exists 2021-03-24T22:01:54.339 INFO:tasks.rados.rados.0.smithi125.stdout:2054: left oid 31 (ObjNum 718 snap 229 seq_num 718) 2021-03-24T22:01:54.346 INFO:tasks.rados.rados.0.smithi125.stdout:2052: done (2 left) 2021-03-24T22:01:54.346 INFO:tasks.rados.rados.0.smithi125.stdout:2054: done (1 left) 2021-03-24T22:01:54.347 INFO:tasks.rados.rados.0.smithi125.stdout:2056: done (0 left) 2021-03-24T22:01:54.347 INFO:tasks.rados.rados.0.smithi125.stdout:2057: delete oid 12 current snap is 229 2021-03-24T22:01:54.350 INFO:tasks.rados.rados.0.smithi125.stdout:2057: done (0 left) 2021-03-24T22:01:54.351 INFO:tasks.rados.rados.0.smithi125.stdout:2058: delete oid 38 current snap is 229 2021-03-24T22:01:54.355 INFO:tasks.rados.rados.0.smithi125.stdout:2058: done (0 left) 2021-03-24T22:01:54.355 INFO:tasks.rados.rados.0.smithi125.stdout:2059: snap_create 2021-03-24T22:01:54.366 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:54.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:54.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:01:54.470 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:54.474 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:54 smithi125 conmon[19936]: cluster 2021-03-24T22:01:52.890493+0000 mgr.y (mgr.14144) 2021-03-24T22:01:54.474 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:54 smithi125 conmon[19936]: 543 : cluster [DBG] pgmap v976: 168 pgs: 1 active+recovering+remapped, 1 active+clean+remapped, 2 peering, 4 active+remapped+backfill_toofull, 160 active+clean; 542 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 12 MiB/s rd, 3.0 MiB/s wr, 18 op/s; 97/1734 objects misplaced (5.594%); 18 MiB/s, 59 objects/s recovering 2021-03-24T22:01:54.475 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:54 smithi125 conmon[19936]: cluster 2021-03-24T22:01:53.322968+0000 mon.a (mon.0) 1533 : cluster [DBG] osdmap e543: 8 total, 8 up, 7 in 2021-03-24T22:01:54.475 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:54 smithi125 conmon[19936]: cluster 2021-03-24T22:01:53.591142+0000 osd.2 ( 2021-03-24T22:01:54.475 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:54 smithi125 conmon[19936]: osd.2) 84 : cluster [DBG] 6.1c scrub ok 2021-03-24T22:01:54.476 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:54 smithi125 conmon[19936]: cluster 2021-03-24T22:01:53 2021-03-24T22:01:54.476 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:54 smithi125 conmon[19936]: .692290+0000 osd.1 (osd.1) 49 : cluster [DBG] 4.f scrub ok 2021-03-24T22:01:54.476 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:54 smithi125 conmon[24927]: cluster 2021-03-24T22:01:52.890493 2021-03-24T22:01:54.477 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:54 smithi125 conmon[24927]: +0000 mgr.y (mgr.14144) 543 : cluster [DBG] pgmap v976: 168 pgs: 1 active+recovering+remapped, 1 active+clean+remapped, 2 peering, 4 active+remapped+backfill_toofull, 160 active+clean; 542 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 12 MiB/s rd, 3.0 MiB/s wr, 18 op/s; 97/1734 objects misplaced (5.594%); 18 MiB/s, 59 objects/s recovering 2021-03-24T22:01:54.477 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:54 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:01:54.477 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:54 smithi125 conmon[24927]: -03-24T22:01:53.322968+0000 mon.a (mon.0) 1533 : cluster 2021-03-24T22:01:54.477 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:54 smithi125 conmon[24927]: [DBG] osdmap e543: 8 total, 8 up, 7 in 2021-03-24T22:01:54.477 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:54 smithi125 conmon[24927]: cluster 2021-03-24T22:01:54.478 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:54 smithi125 conmon[24927]: 2021 2021-03-24T22:01:54.478 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:54 smithi125 conmon[24927]: - 2021-03-24T22:01:54.478 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:54 smithi125 conmon[24927]: 03 2021-03-24T22:01:54.478 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:54 smithi125 conmon[24927]: - 2021-03-24T22:01:54.479 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:54 smithi125 conmon[24927]: 24 2021-03-24T22:01:54.479 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:54 smithi125 conmon[24927]: T 2021-03-24T22:01:54.479 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:54 smithi125 conmon[24927]: 22 2021-03-24T22:01:54.479 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:54 smithi125 conmon[24927]: : 2021-03-24T22:01:54.479 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:54 smithi125 conmon[24927]: 01 2021-03-24T22:01:54.480 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:54 smithi125 conmon[24927]: : 2021-03-24T22:01:54.480 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:54 smithi125 conmon[24927]: 53 2021-03-24T22:01:54.480 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:54 smithi125 conmon[24927]: . 2021-03-24T22:01:54.481 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:54 smithi125 conmon[24927]: 591142 2021-03-24T22:01:54.481 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:54 smithi125 conmon[24927]: +0000 2021-03-24T22:01:54.481 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:54 smithi125 conmon[24927]: 2021-03-24T22:01:54.481 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:54 smithi125 conmon[24927]: osd.2 2021-03-24T22:01:54.482 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:54 smithi125 conmon[24927]: ( 2021-03-24T22:01:54.482 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:54 smithi125 conmon[24927]: osd 2021-03-24T22:01:54.482 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:54 smithi125 conmon[24927]: . 2021-03-24T22:01:54.482 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:54 smithi125 conmon[24927]: 2 2021-03-24T22:01:54.482 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:54 smithi125 conmon[24927]: ) 2021-03-24T22:01:54.483 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:54 smithi125 conmon[24927]: 84 2021-03-24T22:01:54.483 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:54 smithi125 conmon[24927]: : 2021-03-24T22:01:54.483 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:54 smithi125 conmon[24927]: cluster 2021-03-24T22:01:54.483 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:54 smithi125 conmon[24927]: 2021-03-24T22:01:54.484 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:54 smithi125 conmon[24927]: [DBG] 2021-03-24T22:01:54.484 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:54 smithi125 conmon[24927]: 2021-03-24T22:01:54.484 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:54 smithi125 conmon[24927]: 6.1c scrub ok 2021-03-24T22:01:54.485 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:54 smithi125 conmon[24927]: 2021-03-24T22:01:54.485 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:54 smithi125 conmon[24927]: cluster 2021-03-24T22:01:54.485 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:54 smithi125 conmon[24927]: 2021-03-24T22 2021-03-24T22:01:54.485 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:54 smithi125 conmon[24927]: :01:53.692290+0000 osd.1 (osd.1) 49 : cluster [DBG] 4.f scrub ok 2021-03-24T22:01:54.520 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:54.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:54.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:01:54.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:54 smithi142 conmon[20125]: cluster 2021-03-24T22:01:54.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:54 smithi142 conmon[20125]: 2021-03-24T22:01:54.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:54 smithi142 conmon[20125]: 2021 2021-03-24T22:01:54.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:54 smithi142 conmon[20125]: - 2021-03-24T22:01:54.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:54 smithi142 conmon[20125]: 03 2021-03-24T22:01:54.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:54 smithi142 conmon[20125]: -24T22:01:52. 2021-03-24T22:01:54.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:54 smithi142 conmon[20125]: 890493+0000 mgr.y (mgr.14144) 543 : cluster [DBG] pgmap v976: 168 pgs: 1 active+recovering+remapped, 1 active+clean+remapped, 2 peering, 4 active+remapped+backfill_toofull, 160 active+clean; 542 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 12 MiB/s rd, 3.0 MiB/s wr, 18 op/s; 97/1734 objects misplaced (5.594%); 18 MiB/s, 59 objects/s recovering 2021-03-24T22:01:54.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:54 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:01:54.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:54 smithi142 conmon[20125]: -03-24T22:01:53.322968+0000 2021-03-24T22:01:54.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:54 smithi142 conmon[20125]: mon.a (mon.0) 1533 : cluster [DBG] osdmap e543: 8 total, 8 up, 7 in 2021-03-24T22:01:54.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:54 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:01:54.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:54 smithi142 conmon[20125]: -24T22:01:53.591142+0000 osd.2 (osd. 2021-03-24T22:01:54.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:54 smithi142 conmon[20125]: 2) 84 : cluster [DBG] 6.1c scrub ok 2021-03-24T22:01:54.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:54 smithi142 conmon[20125]: cluster 2021-03-24T22:01:54.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:54 smithi142 conmon[20125]: 2021- 2021-03-24T22:01:54.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:54 smithi142 conmon[20125]: 03-24T22 2021-03-24T22:01:54.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:54 smithi142 conmon[20125]: :01:53.692290+0000 osd.1 (osd.1) 2021-03-24T22:01:54.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:54 smithi142 conmon[20125]: 49 : cluster [DBG] 4.f scrub ok 2021-03-24T22:01:54.624 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:54.671 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:54.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:54.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:01:54.779 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:54.830 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:54.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:54.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:01:54.937 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:54.990 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:54.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:54.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:01:55.097 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:55.156 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:55.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:55.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:01:55.262 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:55.292 INFO:tasks.rados.rados.0.smithi125.stdout:2059: done (0 left) 2021-03-24T22:01:55.292 INFO:tasks.rados.rados.0.smithi125.stdout:2060: rollback oid 46 current snap is 230 2021-03-24T22:01:55.292 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 46 to 199 2021-03-24T22:01:55.293 INFO:tasks.rados.rados.0.smithi125.stdout:2061: read oid 32 snap 199 2021-03-24T22:01:55.293 INFO:tasks.rados.rados.0.smithi125.stdout:2061: expect deleted 2021-03-24T22:01:55.293 INFO:tasks.rados.rados.0.smithi125.stdout:2062: rollback oid 5 current snap is 230 2021-03-24T22:01:55.293 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 5 to 208 2021-03-24T22:01:55.293 INFO:tasks.rados.rados.0.smithi125.stdout:2063: rollback oid 23 current snap is 230 2021-03-24T22:01:55.294 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 23 to 222 2021-03-24T22:01:55.294 INFO:tasks.rados.rados.0.smithi125.stdout:2064: snap_create 2021-03-24T22:01:55.298 INFO:tasks.rados.rados.0.smithi125.stdout:2062: finishing rollback tid 1 to smithi12556470-5 2021-03-24T22:01:55.301 INFO:tasks.rados.rados.0.smithi125.stdout:2062: finishing rollback tid 2 to smithi12556470-5 2021-03-24T22:01:55.302 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 5 v 586 (ObjNum 652 snap 198 seq_num 652) dirty exists 2021-03-24T22:01:55.302 INFO:tasks.rados.rados.0.smithi125.stdout:2063: finishing rollback tid 0 to smithi12556470-23 2021-03-24T22:01:55.302 INFO:tasks.rados.rados.0.smithi125.stdout:2060: finishing rollback tid 0 to smithi12556470-46 2021-03-24T22:01:55.303 INFO:tasks.rados.rados.0.smithi125.stdout:2060: finishing rollback tid 1 to smithi12556470-46 2021-03-24T22:01:55.303 INFO:tasks.rados.rados.0.smithi125.stdout:2063: finishing rollback tid 1 to smithi12556470-23 2021-03-24T22:01:55.304 INFO:tasks.rados.rados.0.smithi125.stdout:2063: finishing rollback tid 2 to smithi12556470-23 2021-03-24T22:01:55.304 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 23 v 340 (ObjNum 695 snap 218 seq_num 695) dirty exists 2021-03-24T22:01:55.304 INFO:tasks.rados.rados.0.smithi125.stdout:2060: finishing rollback tid 2 to smithi12556470-46 2021-03-24T22:01:55.304 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 46 v 498 (ObjNum 497 snap 145 seq_num 497) dirty exists 2021-03-24T22:01:55.320 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:55.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:55.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:01:55.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:55 smithi142 conmon[20125]: cluster 2021-03-24T22: 2021-03-24T22:01:55.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:55 smithi142 conmon[20125]: 01:54.330859+0000 mon.a (mon.0) 1534 : cluster [DBG] osdmap e544: 8 total, 8 up, 7 in 2021-03-24T22:01:55.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:55 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:01:55.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:55 smithi142 conmon[20125]: -03-24T22:01:55.273544+0000 2021-03-24T22:01:55.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:55 smithi142 conmon[20125]: mon.a (mon.0) 1535 : 2021-03-24T22:01:55.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:55 smithi142 conmon[20125]: cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:01:55.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:55 smithi142 conmon[20125]: cluster 2021-03-24T22:01:55.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:55 smithi142 conmon[20125]: 2021-03-24T22:01: 2021-03-24T22:01:55.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:55 smithi142 conmon[20125]: 55.286476+0000 mon.a (mon 2021-03-24T22:01:55.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:55 smithi142 conmon[20125]: .0) 1536 : cluster [DBG] 2021-03-24T22:01:55.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:55 smithi142 conmon[20125]: osdmap e545: 8 total, 8 up, 7 in 2021-03-24T22:01:55.410 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:01:55.426 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:55.435 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:01:55.481 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:55.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:55.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:01:55.557 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:55 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:01:55.558 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:55 smithi125 conmon[24927]: 24T22:01:54.330859+0000 mon.a (mon.0) 1534 : cluster [DBG] osdmap e544: 8 total, 8 up, 7 in 2021-03-24T22:01:55.558 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:55 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:01:55.558 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:55 smithi125 conmon[24927]: -03-24T22:01:55.273544+0000 mon.a (mon. 2021-03-24T22:01:55.559 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:55 smithi125 conmon[24927]: 0) 1535 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:01:55.559 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:55 smithi125 conmon[24927]: cluster 2021-03-24T22:01:55.559 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:55 smithi125 conmon[24927]: 2021-03-24T22:01 2021-03-24T22:01:55.559 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:55 smithi125 conmon[24927]: :55.286476+0000 mon.a (mon.0 2021-03-24T22:01:55.560 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:55 smithi125 conmon[24927]: ) 1536 : cluster [DBG] osdmap e545: 8 total, 8 up, 7 in 2021-03-24T22:01:55.560 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:55 smithi125 conmon[19936]: cluster 2021-03-24T22:01:54.330859+0000 2021-03-24T22:01:55.560 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:55 smithi125 conmon[19936]: mon.a (mon.0) 1534 : cluster [DBG] osdmap e544: 8 total, 8 up, 7 in 2021-03-24T22:01:55.561 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:55 smithi125 conmon[19936]: cluster 2021-03-24T22:01:55.273544+0000 mon.a (mon.0) 1535 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:01:55.561 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:55 smithi125 conmon[19936]: cluster 2021-03-24T22:01:55.286476+0000 mon.a (mon.0) 1536 : cluster [DBG] osdmap e545: 8 total, 8 up, 7 in 2021-03-24T22:01:55.587 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:55.641 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:55.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:55.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:01:55.747 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:55.805 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:55.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:55.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:01:55.911 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:55.964 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:55.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:55.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:01:56.068 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:56.122 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:56.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:56.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:01:56.212 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:55 smithi125 conmon[19936]: debug 2021-03-24T22:01:55.914+0000 7f2118128700 -1 mon.a@0(leader).osd e545 definitely_dead 0 2021-03-24T22:01:56.228 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:56.304 INFO:tasks.rados.rados.0.smithi125.stdout:2060: done (4 left) 2021-03-24T22:01:56.304 INFO:tasks.rados.rados.0.smithi125.stdout:2061: done (3 left) 2021-03-24T22:01:56.304 INFO:tasks.rados.rados.0.smithi125.stdout:2062: done (2 left) 2021-03-24T22:01:56.305 INFO:tasks.rados.rados.0.smithi125.stdout:2063: done (1 left) 2021-03-24T22:01:56.305 INFO:tasks.rados.rados.0.smithi125.stdout:2064: done (0 left) 2021-03-24T22:01:56.305 INFO:tasks.rados.rados.0.smithi125.stdout:2065: read oid 48 snap -1 2021-03-24T22:01:56.305 INFO:tasks.rados.rados.0.smithi125.stdout:2065: expect (ObjNum 588 snap 177 seq_num 588) 2021-03-24T22:01:56.306 INFO:tasks.rados.rados.0.smithi125.stdout:2066: write (excl) oid 5 current snap is 231 2021-03-24T22:01:56.306 INFO:tasks.rados.rados.0.smithi125.stdout:2066: seq_num 722 ranges {534654=466960,1766705=367949} 2021-03-24T22:01:56.306 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:56.307 INFO:tasks.rados.rados.0.smithi125.stdout:2066: writing smithi12556470-5 from 534654 to 1001614 tid 1 2021-03-24T22:01:56.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:56.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:01:56.314 INFO:tasks.rados.rados.0.smithi125.stdout:2066: writing smithi12556470-5 from 1766705 to 2134654 tid 2 2021-03-24T22:01:56.314 INFO:teuthology.orchestra.run.smithi125.stderr:marked in osd.7. 2021-03-24T22:01:56.315 INFO:tasks.rados.rados.0.smithi125.stdout:2067: delete oid 41 current snap is 231 2021-03-24T22:01:56.410 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:56.468 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:56 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:01:56.469 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:56 smithi125 conmon[19936]: 24T22:01:54.891286+0000 mgr.y (mgr.14144) 544 : cluster [DBG] pgmap v979: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 2 peering, 3 active+remapped+backfill_toofull, 161 active+clean; 548 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 11 MiB/s rd, 2.1 MiB/s wr, 24 op/s; 96/1743 objects misplaced (5.508%); 11 MiB/s, 26 objects/s recovering 2021-03-24T22:01:56.469 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:56 smithi125 conmon[19936]: cluster 2021-03-24T22:01: 2021-03-24T22:01:56.469 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:56 smithi125 conmon[19936]: 55.618226+0000 osd.1 (osd.1) 50 : cluster [DBG] 5.12 scrub ok 2021-03-24T22:01:56.469 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:56 smithi125 conmon[19936]: audit 2021-03-24T22:01:56.470 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:56 smithi125 conmon[19936]: 2021-03-24T22:01:55.917376+0000 mon.c (mon.2) 85 : audit [DBG] from='client.? v1:172.21.15.125:0/174479546' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2021-03-24T22:01:56.470 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:56 smithi125 conmon[19936]: audit 2021 2021-03-24T22:01:56.470 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:56 smithi125 conmon[19936]: -03-24T22:01:55.918007+0000 mon.a ( 2021-03-24T22:01:56.470 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:56 smithi125 conmon[19936]: mon.0) 1537 : audit [DBG] from='client.35411 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2021-03-24T22:01:56.471 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:56 smithi125 conmon[19936]: audit 2021-03-24T22:01:56.471 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:56 smithi125 conmon[19936]: 2021-03-24T22:01:56. 2021-03-24T22:01:56.471 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:56 smithi125 conmon[19936]: 282601+0000 mon.a (mon.0) 1538 : audit 2021-03-24T22:01:56.471 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:56 smithi125 conmon[19936]: [INF] from='client.35411 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2021-03-24T22:01:56.471 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:56 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:01:56.472 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:56 smithi125 conmon[19936]: 01:56.282706+0000 mon.a (mon.0) 2021-03-24T22:01:56.472 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:56 smithi125 conmon[19936]: 1539 : cluster [DBG] osdmap e546: 8 total, 8 up, 8 in 2021-03-24T22:01:56.472 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:56 smithi125 conmon[24927]: cluster 2021-03-24T22:01:56.473 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:56 smithi125 conmon[24927]: 2021-03-24T22:01:56.473 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:56 smithi125 conmon[24927]: 2021 2021-03-24T22:01:56.473 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:56 smithi125 conmon[24927]: - 2021-03-24T22:01:56.473 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:56 smithi125 conmon[24927]: 03-24T22:01:54 2021-03-24T22:01:56.474 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:56 smithi125 conmon[24927]: .891286+0000 mgr.y (mgr.14144) 544 : cluster [DBG] pgmap v979: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 2 peering, 3 active+remapped+backfill_toofull, 161 active+clean; 548 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 11 MiB/s rd, 2.1 MiB/s wr, 24 op/s; 96/1743 objects misplaced (5.508%); 11 MiB/s, 26 objects/s recovering 2021-03-24T22:01:56.474 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:56 smithi125 conmon[24927]: cluster 2021-03-24T22:01:55 2021-03-24T22:01:56.474 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:56 smithi125 conmon[24927]: .618226+0000 osd.1 (osd.1) 50 : cluster [DBG] 5.12 scrub ok 2021-03-24T22:01:56.474 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:56 smithi125 conmon[24927]: audit 2021 2021-03-24T22:01:56.475 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:56 smithi125 conmon[24927]: -03-24T22:01:55.917376+0000 mon.c (mon.2) 85 : audit 2021-03-24T22:01:56.475 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:56 smithi125 conmon[24927]: [DBG] from='client.? v1:172.21.15.125:0/174479546' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2021-03-24T22:01:56.475 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:56 smithi125 conmon[24927]: audit 2021-03-24T22: 2021-03-24T22:01:56.475 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:56 smithi125 conmon[24927]: 01:55.918007+0000 mon.a (mon.0) 1537 : audit 2021-03-24T22:01:56.475 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:56 smithi125 conmon[24927]: [DBG] from='client.35411 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2021-03-24T22:01:56.476 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:56 smithi125 conmon[24927]: audit 2021-03-24T22:01:56.476 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:56 smithi125 conmon[24927]: 2021-03-24T22:01:56.282601+0000 mon.a (mon 2021-03-24T22:01:56.476 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:56 smithi125 conmon[24927]: .0) 1538 : audit [INF] from='client.35411 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2021-03-24T22:01:56.476 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:56 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:01:56.477 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:56 smithi125 conmon[24927]: 03-24T22:01:56.282706+0000 mon.a 2021-03-24T22:01:56.477 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:56 smithi125 conmon[24927]: (mon.0) 1539 : cluster [DBG] osdmap e546: 8 total, 8 up, 8 in 2021-03-24T22:01:56.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:56.478 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:01:56.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:01:56.580 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:56.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:56 smithi142 conmon[20125]: cluster 2021-03-24T22:01:54 2021-03-24T22:01:56.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:56 smithi142 conmon[20125]: .891286+0000 mgr.y (mgr.14144) 544 : cluster [DBG] pgmap v979: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 2 peering, 3 active+remapped+backfill_toofull, 161 active+clean; 548 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 11 MiB/s rd, 2.1 MiB/s wr, 24 op/s; 96/1743 objects misplaced (5.508%); 11 MiB/s, 26 objects/s recovering 2021-03-24T22:01:56.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:56 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:01:56.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:56 smithi142 conmon[20125]: T22:01:55.618226+0000 osd.1 (osd.1) 50 : 2021-03-24T22:01:56.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:56 smithi142 conmon[20125]: cluster [DBG] 5.12 scrub ok 2021-03-24T22:01:56.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:56 smithi142 conmon[20125]: audit 2021 2021-03-24T22:01:56.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:56 smithi142 conmon[20125]: -03-24T22:01 2021-03-24T22:01:56.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:56 smithi142 conmon[20125]: :55.917376+0000 mon.c 2021-03-24T22:01:56.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:56 smithi142 conmon[20125]: (mon.2) 85 : audit [DBG] 2021-03-24T22:01:56.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:56 smithi142 conmon[20125]: from='client.? v1:172.21.15.125:0/174479546' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2021-03-24T22:01:56.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:56 smithi142 conmon[20125]: audit 2021-03 2021-03-24T22:01:56.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:56 smithi142 conmon[20125]: -24T22:01:55. 2021-03-24T22:01:56.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:56 smithi142 conmon[20125]: 918007+0000 mon.a (mon.0) 1537 2021-03-24T22:01:56.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:56 smithi142 conmon[20125]: : audit [DBG] from='client.35411 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2021-03-24T22:01:56.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:56 smithi142 conmon[20125]: audit 2021 2021-03-24T22:01:56.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:56 smithi142 conmon[20125]: -03-24T22: 2021-03-24T22:01:56.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:56 smithi142 conmon[20125]: 01:56.282601+0000 mon.a (mon 2021-03-24T22:01:56.625 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:56 smithi142 conmon[20125]: .0) 1538 : audit [INF] from='client.35411 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2021-03-24T22:01:56.625 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:56 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:01:56.625 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:56 smithi142 conmon[20125]: -03-24T22:01:56.282706 2021-03-24T22:01:56.625 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:56 smithi142 conmon[20125]: +0000 mon.a (mon.0) 1539 : 2021-03-24T22:01:56.626 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:56 smithi142 conmon[20125]: cluster [DBG] osdmap e546: 8 total, 8 up, 8 in 2021-03-24T22:01:56.636 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:56.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:56.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:01:56.741 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:56.799 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:56.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:56.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:01:56.906 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:56.961 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:56.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:56.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:01:57.069 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:57.122 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:57.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:57.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:01:57.229 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:57.283 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:01:57.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:57.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:01:57.358 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:57 smithi125 conmon[19936]: cluster 2021-03-24T22:01:55.431114+0000 2021-03-24T22:01:57.358 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:57 smithi125 conmon[19936]: osd.3 (osd.3) 115 : cluster [DBG] 3.c deep-scrub ok 2021-03-24T22:01:57.359 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:57 smithi125 conmon[19936]: cluster 2021-03-24T22:01:56.417658+0000 osd.0 (osd.0) 114 2021-03-24T22:01:57.359 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:57 smithi125 conmon[19936]: : cluster [DBG] 3.17 scrub ok 2021-03-24T22:01:57.359 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:57 smithi125 conmon[19936]: cluster 2021-03-24T22:01:57.359 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:57 smithi125 conmon[19936]: 2021-03-24T22:01:56 2021-03-24T22:01:57.359 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:57 smithi125 conmon[19936]: .662235+0000 osd.2 (osd.2 2021-03-24T22:01:57.360 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:57 smithi125 conmon[19936]: ) 85 : cluster [DBG] 7.e deep-scrub ok 2021-03-24T22:01:57.360 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:57 smithi125 conmon[19936]: 2021-03-24T22:01:57.360 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:57 smithi125 conmon[19936]: cluster 2021-03-24T22:01:57.360 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:57 smithi125 conmon[19936]: 2021-03-24T22:01 2021-03-24T22:01:57.361 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:57 smithi125 conmon[19936]: :57.292393+0000 mon.a (mon 2021-03-24T22:01:57.361 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:57 smithi125 conmon[19936]: .0) 1540 : cluster 2021-03-24T22:01:57.361 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:57 smithi125 conmon[19936]: [DBG] osdmap e547: 8 total, 8 up, 8 in 2021-03-24T22:01:57.392 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:57.443 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:57.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:57.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:01:57.547 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:57.559 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:57 smithi142 conmon[20125]: cluster 2021-03-24T22:01:55.431114 2021-03-24T22:01:57.560 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:57 smithi142 conmon[20125]: +0000 osd.3 (osd.3) 115 : cluster [DBG] 3.c deep-scrub ok 2021-03-24T22:01:57.560 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:57 smithi142 conmon[20125]: cluster 2021-03-24T22:01:56.417658+0000 osd.0 (osd.0) 114 : cluster [DBG] 3.17 scrub ok 2021-03-24T22:01:57.560 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:57 smithi142 conmon[20125]: cluster 2021-03-24T22:01:56.662235+0000 osd.2 (osd.2) 85 : cluster [DBG] 2021-03-24T22:01:57.561 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:57 smithi142 conmon[20125]: 7.e deep-scrub ok 2021-03-24T22:01:57.561 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:57 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:01:57.561 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:57 smithi142 conmon[20125]: 03-24T22:01:57.292393+0000 mon.a (mon.0) 1540 : cluster 2021-03-24T22:01:57.561 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:57 smithi142 conmon[20125]: [DBG] osdmap e547: 8 total, 8 up, 8 in 2021-03-24T22:01:57.598 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:57.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:57.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:01:57.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:57 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T22:01:57.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:57 smithi125 conmon[24927]: 22:01:55.431114+0000 osd.3 (osd.3) 115 : cluster [DBG] 2021-03-24T22:01:57.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:57 smithi125 conmon[24927]: 3.c deep-scrub ok 2021-03-24T22:01:57.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:57 smithi125 conmon[24927]: cluster 2021-03-24T22:01:57.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:57 smithi125 conmon[24927]: 2021-03- 2021-03-24T22:01:57.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:57 smithi125 conmon[24927]: 24T22:01:56.417658+0000 2021-03-24T22:01:57.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:57 smithi125 conmon[24927]: osd.0 (osd.0) 114 2021-03-24T22:01:57.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:57 smithi125 conmon[24927]: : cluster [DBG] 3.17 scrub ok 2021-03-24T22:01:57.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:57 smithi125 conmon[24927]: cluster 2021-03-24T22:01:57.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:57 smithi125 conmon[24927]: 2021-03-24T22: 2021-03-24T22:01:57.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:57 smithi125 conmon[24927]: 01:56.662235+0000 osd.2 2021-03-24T22:01:57.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:57 smithi125 conmon[24927]: (osd.2) 85 : cluster 2021-03-24T22:01:57.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:57 smithi125 conmon[24927]: [DBG] 7.e deep-scrub ok 2021-03-24T22:01:57.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:57 smithi125 conmon[24927]: cluster 2021-03-24T22:01:57.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:57 smithi125 conmon[24927]: 2021-03-24T22: 2021-03-24T22:01:57.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:57 smithi125 conmon[24927]: 01:57.292393+0000 mon.a (mon. 2021-03-24T22:01:57.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:57 smithi125 conmon[24927]: 0) 1540 : cluster [DBG] osdmap e547: 8 total, 8 up, 8 in 2021-03-24T22:01:57.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:57 smithi125 conmon[24927]: 2021-03-24T22:01:57.704 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:57.745 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:57.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:57.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:01:57.850 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:57.898 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:57.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:57.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:01:58.002 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:58.045 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:58.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:58.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:01:58.151 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:58.204 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:58.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:58.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:01:58.312 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:58.313 INFO:teuthology.orchestra.run.smithi125.stderr:noscrub is set 2021-03-24T22:01:58.353 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:58.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:58.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:01:58.406 INFO:tasks.rados.rados.0.smithi125.stdout:2066: finishing write tid 1 to smithi12556470-5 2021-03-24T22:01:58.408 INFO:tasks.rados.rados.0.smithi125.stdout:2067: done (2 left) 2021-03-24T22:01:58.408 INFO:tasks.rados.rados.0.smithi125.stdout:2068: read oid 39 snap 187 2021-03-24T22:01:58.409 INFO:tasks.rados.rados.0.smithi125.stdout:2068: expect deleted 2021-03-24T22:01:58.409 INFO:tasks.rados.rados.0.smithi125.stdout:2069: copy_from oid 26 from oid 12 current snap is 231 2021-03-24T22:01:58.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[19936]: cluster 2021-03-24T22:01: 2021-03-24T22:01:58.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[19936]: 56.876102+0000 osd.6 (osd.6) 34 : cluster [DBG] 3.10 scrub ok 2021-03-24T22:01:58.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[19936]: cluster 2021-03-24T22:01:58.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[19936]: 2021-03 2021-03-24T22:01:58.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[19936]: -24T22:01:56.892345 2021-03-24T22:01:58.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[19936]: +0000 mgr.y (mgr.14144) 545 : 2021-03-24T22:01:58.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[19936]: cluster [DBG] pgmap v982: 168 pgs: 1 remapped+peering, 12 peering, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 152 active+clean; 555 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 8.4 MiB/s rd, 2.1 MiB/s wr, 132 op/s; 62/1752 objects misplaced (3.539%); 16 MiB/s, 22 objects/s recovering 2021-03-24T22:01:58.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:01:58.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[19936]: -03-24T22:01:57. 2021-03-24T22:01:58.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[19936]: 271810+0000 osd.2 (osd.2) 86 2021-03-24T22:01:58.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[19936]: : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:01:58.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:01:58.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[19936]: -03-24T22:01:57.274985+0000 2021-03-24T22:01:58.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[19936]: osd.2 (osd.2) 87 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:01:58.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:01:58.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[19936]: T22:01:57.307185+0000 osd.5 ( 2021-03-24T22:01:58.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[19936]: osd.5) 21 : cluster [INF] 4.8 continuing backfill to osd.6 from (97'28,544'46] MIN to 544'46 2021-03-24T22:01:58.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[19936]: 2021-03-24T22:01:58.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[19936]: cluster 2021-03-24T22:01:58.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[19936]: 2021-03-24T22:01:57. 2021-03-24T22:01:58.421 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[19936]: 307217+0000 osd.5 (osd.5) 2021-03-24T22:01:58.421 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[19936]: 22 : cluster [DBG] 4.8 starting backfill to osd.7 from (0'0,0'0] MAX to 544'46 2021-03-24T22:01:58.421 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[19936]: audit 2021-03 2021-03-24T22:01:58.421 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[19936]: -24T22:01:57.884000 2021-03-24T22:01:58.422 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[19936]: +0000 mon.a (mon.0) 1541 2021-03-24T22:01:58.422 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[19936]: : audit [INF] from='client.? v1:172.21.15.125:0/3049805213' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T22:01:58.422 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:01:58.423 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[19936]: 03-24T22:01:58 2021-03-24T22:01:58.423 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[19936]: .281462+0000 mon.a (mon.0) 2021-03-24T22:01:58.423 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[19936]: 1542 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:01:58.423 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[19936]: audit 2021 2021-03-24T22:01:58.424 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[19936]: -03-24T22:01:58. 2021-03-24T22:01:58.424 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[19936]: 292574+0000 mon.a (mon.0) 2021-03-24T22:01:58.424 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[19936]: 1543 : audit [INF] from='client.? v1:172.21.15.125:0/3049805213' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T22:01:58.424 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:01:58.425 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[19936]: -03-24T22:01:58 2021-03-24T22:01:58.425 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[19936]: .292660+0000 mon.a (mon.0) 1544 2021-03-24T22:01:58.425 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[19936]: : cluster [DBG] osdmap e548: 8 total, 8 up, 8 in 2021-03-24T22:01:58.426 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: cluster 2021-03-24T22:01:58.426 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: 2021-03-24T22:01:58.426 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: 2021 2021-03-24T22:01:58.427 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: - 2021-03-24T22:01:58.427 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: 03 2021-03-24T22:01:58.427 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: - 2021-03-24T22:01:58.427 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: 24 2021-03-24T22:01:58.428 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: T 2021-03-24T22:01:58.428 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: 22 2021-03-24T22:01:58.428 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: : 2021-03-24T22:01:58.428 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: 01 2021-03-24T22:01:58.429 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: : 2021-03-24T22:01:58.429 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: 56 2021-03-24T22:01:58.429 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: . 2021-03-24T22:01:58.429 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: 876102 2021-03-24T22:01:58.430 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: +0000 2021-03-24T22:01:58.430 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: 2021-03-24T22:01:58.430 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: osd.6 2021-03-24T22:01:58.430 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: ( 2021-03-24T22:01:58.431 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: osd 2021-03-24T22:01:58.431 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: . 2021-03-24T22:01:58.431 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: 6 2021-03-24T22:01:58.431 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: ) 2021-03-24T22:01:58.432 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: 34 2021-03-24T22:01:58.432 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: : 2021-03-24T22:01:58.432 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: cluster 2021-03-24T22:01:58.432 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: 2021-03-24T22:01:58.432 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: [DBG] 2021-03-24T22:01:58.433 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: 2021-03-24T22:01:58.433 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: 3.10 scrub ok 2021-03-24T22:01:58.433 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: 2021-03-24T22:01:58.433 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: cluster 2021-03-24T22:01:58.434 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: 2021-03-24T22:01:58.434 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: 2021 2021-03-24T22:01:58.434 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: - 2021-03-24T22:01:58.435 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: 03 2021-03-24T22:01:58.435 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: - 2021-03-24T22:01:58.435 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: 24 2021-03-24T22:01:58.435 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: T 2021-03-24T22:01:58.436 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: 22 2021-03-24T22:01:58.436 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: : 2021-03-24T22:01:58.436 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: 01 2021-03-24T22:01:58.436 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: : 2021-03-24T22:01:58.437 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: 56 2021-03-24T22:01:58.437 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: . 2021-03-24T22:01:58.437 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: 892345 2021-03-24T22:01:58.438 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: +0000 2021-03-24T22:01:58.438 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: 2021-03-24T22:01:58.438 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: mgr.y 2021-03-24T22:01:58.438 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: ( 2021-03-24T22:01:58.439 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: mgr 2021-03-24T22:01:58.439 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: . 2021-03-24T22:01:58.439 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: 14144 2021-03-24T22:01:58.439 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: ) 2021-03-24T22:01:58.440 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: 545 2021-03-24T22:01:58.440 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: : 2021-03-24T22:01:58.440 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: cluster 2021-03-24T22:01:58.440 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: 2021-03-24T22:01:58.440 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: [DBG] 2021-03-24T22:01:58.441 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: 2021-03-24T22:01:58.441 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: pgmap v982: 168 pgs: 1 remapped+peering, 12 peering, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 152 active+clean; 555 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 8.4 MiB/s rd, 2.1 MiB/s wr, 132 op/s; 62/1752 objects misplaced (3.539%); 16 MiB/s, 22 objects/s recovering 2021-03-24T22:01:58.441 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: 2021-03-24T22:01:58.441 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: cluster 2021-03-24T22:01:58.442 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: 2021-03-24T22:01:58.442 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: 2021 2021-03-24T22:01:58.442 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: - 2021-03-24T22:01:58.442 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: 03 2021-03-24T22:01:58.443 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: - 2021-03-24T22:01:58.443 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: 24 2021-03-24T22:01:58.443 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: T 2021-03-24T22:01:58.444 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: 22 2021-03-24T22:01:58.444 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: : 2021-03-24T22:01:58.444 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: 01 2021-03-24T22:01:58.444 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: : 2021-03-24T22:01:58.444 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: 57 2021-03-24T22:01:58.445 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: . 2021-03-24T22:01:58.445 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: 271810 2021-03-24T22:01:58.445 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: +0000 2021-03-24T22:01:58.445 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: 2021-03-24T22:01:58.446 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: osd.2 2021-03-24T22:01:58.446 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: ( 2021-03-24T22:01:58.446 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: osd 2021-03-24T22:01:58.446 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: . 2021-03-24T22:01:58.447 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: 2) 86 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:01:58.447 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: cluster 2021-03-24T22:01: 2021-03-24T22:01:58.447 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: 57.274985+0000 osd.2 (osd.2) 87 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:01:58.447 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: 2021-03-24T22:01:58.448 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:01:58.448 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: -03-24T22:01:57.307185+0000 osd.5 (osd.5 2021-03-24T22:01:58.448 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: ) 21 : cluster [INF] 4.8 continuing backfill to osd.6 from (97'28,544'46] MIN to 544'46 2021-03-24T22:01:58.449 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T22:01:58.449 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: 22:01:57.307217+0000 osd.5 (osd.5 2021-03-24T22:01:58.450 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: ) 22 : cluster [DBG] 4.8 starting backfill to osd.7 from (0'0,0'0] MAX to 544'46 2021-03-24T22:01:58.450 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: audit 2021-03-24T 2021-03-24T22:01:58.450 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: 22:01:57.884000+0000 mon.a (mon. 2021-03-24T22:01:58.451 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: 0) 1541 : audit [INF] from='client.? v1:172.21.15.125:0/3049805213' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T22:01:58.451 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T22:01:58.451 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: T22:01:58.281462+0000 mon.a (mon. 2021-03-24T22:01:58.452 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: 0) 1542 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:01:58.452 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: audit 2021-03 2021-03-24T22:01:58.452 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: -24T22:01:58.292574+0000 mon.a 2021-03-24T22:01:58.453 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: (mon.0) 1543 : audit [INF] from='client.? v1:172.21.15.125:0/3049805213' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T22:01:58.453 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:01:58.453 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: -03-24T22:01:58.292660+0000 2021-03-24T22:01:58.453 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: mon.a (mon.0) 1544 : cluster [DBG] 2021-03-24T22:01:58.454 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:58 smithi125 conmon[24927]: osdmap e548: 8 total, 8 up, 8 in 2021-03-24T22:01:58.455 INFO:tasks.rados.rados.0.smithi125.stdout:2070: write (excl) oid 17 current snap is 231 2021-03-24T22:01:58.455 INFO:tasks.rados.rados.0.smithi125.stdout:2070: seq_num 724 ranges {568268=733811,2000022=168246} 2021-03-24T22:01:58.455 INFO:tasks.rados.rados.0.smithi125.stdout:2070: writing smithi12556470-17 from 568268 to 1302079 tid 1 2021-03-24T22:01:58.455 INFO:tasks.rados.rados.0.smithi125.stdout:2070: writing smithi12556470-17 from 2000022 to 2168268 tid 2 2021-03-24T22:01:58.456 INFO:tasks.rados.rados.0.smithi125.stdout:2071: snap_remove snap 223 2021-03-24T22:01:58.457 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:58.504 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:58.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:58.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:01:58.609 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:58.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:58 smithi142 conmon[20125]: cluster 2021-03-24T22:01:56. 2021-03-24T22:01:58.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:58 smithi142 conmon[20125]: 876102+0000 osd.6 (osd.6) 34 : cluster [DBG] 3.10 scrub ok 2021-03-24T22:01:58.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:58 smithi142 conmon[20125]: cluster 2021-03-24T22:01 2021-03-24T22:01:58.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:58 smithi142 conmon[20125]: :56.892345+0000 mgr.y (mgr.14144) 545 : cluster [DBG] 2021-03-24T22:01:58.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:58 smithi142 conmon[20125]: pgmap v982: 168 pgs: 1 remapped+peering, 12 peering, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 152 active+clean; 555 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 8.4 MiB/s rd, 2.1 MiB/s wr, 132 op/s; 62/1752 objects misplaced (3.539%); 16 MiB/s, 22 objects/s recovering 2021-03-24T22:01:58.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:58 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:01:58.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:58 smithi142 conmon[20125]: -24T22:01:57.271810+0000 2021-03-24T22:01:58.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:58 smithi142 conmon[20125]: osd.2 (osd.2) 86 : cluster 2021-03-24T22:01:58.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:58 smithi142 conmon[20125]: [DBG] purged_snaps scrub starts 2021-03-24T22:01:58.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:58 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:01:58.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:58 smithi142 conmon[20125]: 03-24T22:01:57.274985+0000 osd.2 2021-03-24T22:01:58.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:58 smithi142 conmon[20125]: (osd.2) 87 : cluster 2021-03-24T22:01:58.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:58 smithi142 conmon[20125]: [DBG] purged_snaps scrub ok 2021-03-24T22:01:58.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:58 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:01:58.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:58 smithi142 conmon[20125]: 03-24T22:01:57.307185+0000 2021-03-24T22:01:58.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:58 smithi142 conmon[20125]: osd.5 (osd.5) 21 : 2021-03-24T22:01:58.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:58 smithi142 conmon[20125]: cluster [INF] 4.8 continuing backfill to osd.6 from (97'28,544'46] MIN to 544'46 2021-03-24T22:01:58.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:58 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:01:58.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:58 smithi142 conmon[20125]: -03-24T22:01:57.307217 2021-03-24T22:01:58.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:58 smithi142 conmon[20125]: +0000 osd.5 (osd.5) 22 2021-03-24T22:01:58.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:58 smithi142 conmon[20125]: : cluster [DBG] 4.8 starting backfill to osd.7 from (0'0,0'0] MAX to 544'46 2021-03-24T22:01:58.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:58 smithi142 conmon[20125]: audit 2021- 2021-03-24T22:01:58.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:58 smithi142 conmon[20125]: 03-24T22:01:57.884000 2021-03-24T22:01:58.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:58 smithi142 conmon[20125]: +0000 mon.a (mon.0) 1541 2021-03-24T22:01:58.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:58 smithi142 conmon[20125]: : audit [INF] from='client.? v1:172.21.15.125:0/3049805213' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T22:01:58.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:58 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:01:58.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:58 smithi142 conmon[20125]: -03-24T22:01:58. 2021-03-24T22:01:58.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:58 smithi142 conmon[20125]: 281462+0000 mon.a (mon.0) 2021-03-24T22:01:58.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:58 smithi142 conmon[20125]: 1542 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:01:58.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:58 smithi142 conmon[20125]: audit 2021 2021-03-24T22:01:58.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:58 smithi142 conmon[20125]: -03-24T22:01:58 2021-03-24T22:01:58.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:58 smithi142 conmon[20125]: .292574+0000 mon.a (mon. 2021-03-24T22:01:58.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:58 smithi142 conmon[20125]: 0) 1543 : audit [INF] 2021-03-24T22:01:58.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:58 smithi142 conmon[20125]: from='client.? v1:172.21.15.125:0/3049805213' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T22:01:58.625 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:58 smithi142 conmon[20125]: cluster 2021-03-24T22:01:58 2021-03-24T22:01:58.625 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:58 smithi142 conmon[20125]: .292660+0000 mon.a (mon.0) 2021-03-24T22:01:58.625 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:58 smithi142 conmon[20125]: 1544 : cluster [DBG] osdmap e548: 8 total, 8 up, 8 in 2021-03-24T22:01:58.662 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:58.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:58.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:01:58.767 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:58.820 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:58.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:58.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:01:58.926 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:58.986 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:01:58.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:58.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:01:59.092 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:59.147 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:01:59.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:59.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:01:59.197 INFO:journalctl@ceph.rgw.foo.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[39595]: debug 2021-03-24T22:01:59.197 INFO:journalctl@ceph.rgw.foo.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[39595]: 2021-03-24T22:01:59.190+0000 7fc212026700 -1 RGWWatcher::handle_error cookie 94195313963520 err (110) Connection timed out 2021-03-24T22:01:59.197 INFO:journalctl@ceph.rgw.foo.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[39595]: 2021-03-24T22:01:59.254 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:59.315 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:59.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:59.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:01:59.372 INFO:tasks.rados.rados.0.smithi125.stdout:2071: done (5 left) 2021-03-24T22:01:59.372 INFO:tasks.rados.rados.0.smithi125.stdout:2072: rollback oid 11 current snap is 231 2021-03-24T22:01:59.372 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 11 to 158 2021-03-24T22:01:59.373 INFO:tasks.rados.rados.0.smithi125.stdout:2073: snap_remove snap 222 2021-03-24T22:01:59.421 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:59.466 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:01:59.468 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[24927]: cluster 2021-03-24T22:01:59.468 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[24927]: 2021-03-24T22:01:59.469 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[24927]: 2021 2021-03-24T22:01:59.469 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[24927]: - 2021-03-24T22:01:59.469 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[24927]: 03 2021-03-24T22:01:59.470 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[24927]: - 2021-03-24T22:01:59.470 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[24927]: 24 2021-03-24T22:01:59.470 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[24927]: T 2021-03-24T22:01:59.470 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[24927]: 22:01:58.294745+0000 osd.6 2021-03-24T22:01:59.471 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[24927]: (osd.6) 35 : cluster [DBG] 4.a starting backfill to osd.7 from (0'0,0'0] MAX to 544'45 2021-03-24T22:01:59.471 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[24927]: cluster 2021-03-24T22 2021-03-24T22:01:59.471 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[24927]: :01:58.297441+0000 osd.0 (osd.0) 2021-03-24T22:01:59.471 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[24927]: 115 : cluster [DBG] 4.6 starting backfill to osd.7 from (0'0,0'0] MAX to 544'34 2021-03-24T22:01:59.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:59.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:01:59.473 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: cluster 2021-03-24T22:01:59.474 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: 2021-03-24T22:01:59.474 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: 2021 2021-03-24T22:01:59.474 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: - 2021-03-24T22:01:59.474 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: 03 2021-03-24T22:01:59.474 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: - 2021-03-24T22:01:59.475 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: 24 2021-03-24T22:01:59.475 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: T 2021-03-24T22:01:59.475 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: 22 2021-03-24T22:01:59.475 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: : 2021-03-24T22:01:59.476 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: 01 2021-03-24T22:01:59.476 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: : 2021-03-24T22:01:59.476 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: 58 2021-03-24T22:01:59.476 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: . 2021-03-24T22:01:59.476 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: 294745 2021-03-24T22:01:59.477 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: +0000 2021-03-24T22:01:59.477 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: 2021-03-24T22:01:59.477 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: osd.6 2021-03-24T22:01:59.477 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: ( 2021-03-24T22:01:59.478 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: osd 2021-03-24T22:01:59.478 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: . 2021-03-24T22:01:59.478 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: 6 2021-03-24T22:01:59.478 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: ) 2021-03-24T22:01:59.479 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: 35 2021-03-24T22:01:59.479 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: : cluster [DBG] 4.a starting backfill to osd.7 from (0'0,0'0] MAX to 544'45 2021-03-24T22:01:59.479 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: 2021-03-24T22:01:59.479 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: cluster 2021-03-24T22:01:59.480 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: 2021-03-24T22:01:59.480 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: 2021 2021-03-24T22:01:59.480 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: - 2021-03-24T22:01:59.480 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: 03 2021-03-24T22:01:59.480 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: - 2021-03-24T22:01:59.481 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: 24 2021-03-24T22:01:59.481 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: T 2021-03-24T22:01:59.481 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: 22 2021-03-24T22:01:59.481 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: : 2021-03-24T22:01:59.482 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: 01 2021-03-24T22:01:59.482 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: : 2021-03-24T22:01:59.482 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: 58 2021-03-24T22:01:59.482 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: . 2021-03-24T22:01:59.482 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: 297441 2021-03-24T22:01:59.483 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: +0000 2021-03-24T22:01:59.483 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: 2021-03-24T22:01:59.483 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: osd.0 2021-03-24T22:01:59.483 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: ( 2021-03-24T22:01:59.483 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: osd 2021-03-24T22:01:59.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: . 2021-03-24T22:01:59.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: 0 2021-03-24T22:01:59.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: ) 2021-03-24T22:01:59.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: 115 2021-03-24T22:01:59.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: : 2021-03-24T22:01:59.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: cluster 2021-03-24T22:01:59.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: 2021-03-24T22:01:59.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: [DBG] 2021-03-24T22:01:59.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: 2021-03-24T22:01:59.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: 4.6 starting backfill to osd.7 from (0'0,0'0] MAX to 544'34 2021-03-24T22:01:59.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:01:59 smithi125 conmon[19936]: 2021-03-24T22:01:59.574 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:59.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:59 smithi142 conmon[20125]: cluster 2021-03-24T22 2021-03-24T22:01:59.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:59 smithi142 conmon[20125]: :01:58.294745+0000 osd.6 (osd.6) 35 : cluster [DBG] 4.a starting backfill to osd.7 from (0'0,0'0] MAX to 544'45 2021-03-24T22:01:59.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:59 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:01:59.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:01:59 smithi142 conmon[20125]: -03-24T22:01:58.297441+0000 osd.0 (osd.0) 115 : cluster [DBG] 4.6 starting backfill to osd.7 from (0'0,0'0] MAX to 544'34 2021-03-24T22:01:59.627 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:01:59.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:59.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:01:59.733 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:59.787 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:01:59.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:59.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:01:59.891 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:59.954 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:01:59.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:01:59.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:02:00.059 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:00.105 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:00.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:00.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:02:00.210 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:00.260 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:00.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:00.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:02:00.295 INFO:tasks.rados.rados.0.smithi125.stdout:2073: done (6 left) 2021-03-24T22:02:00.296 INFO:tasks.rados.rados.0.smithi125.stdout:2074: snap_remove snap 226 2021-03-24T22:02:00.366 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:00.416 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:00.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:00.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:02:00.523 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:00.574 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:00.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:00.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:02:00.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:00 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T22:02:00.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:00 smithi125 conmon[24927]: T22:01:58.892883+0000 mgr.y (mgr.14144) 546 : cluster [DBG] pgmap v985: 168 pgs: 1 remapped+peering, 12 peering, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 152 active+clean; 555 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1.6 MiB/s rd, 948 KiB/s wr, 114 op/s; 62/1752 objects misplaced (3.539%); 16 MiB/s, 22 objects/s recovering 2021-03-24T22:02:00.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:00 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:02:00.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:00 smithi125 conmon[24927]: 03-24T22:01:59.369066+0000 mon.a (mon.0) 1545 : cluster [DBG] osdmap e549: 8 total, 8 up, 8 in 2021-03-24T22:02:00.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:00 smithi125 conmon[24927]: audit 2021- 2021-03-24T22:02:00.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:00 smithi125 conmon[24927]: 03-24T22:01:59.811025+0000 mon.a (mon.0) 1546 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:02:00.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:00 smithi125 conmon[24927]: audit 2021-03-24T22:01:59.890070+0000 mon.a (mon.0) 1547 : 2021-03-24T22:02:00.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:00 smithi125 conmon[24927]: audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T22:02:00.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:00 smithi125 conmon[24927]: cluster 2021-03-24T22:02:00 2021-03-24T22:02:00.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:00 smithi125 conmon[24927]: .275873+0000 mon.a (mon.0) 1548 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:02:00.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:00 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:02:00.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:00 smithi125 conmon[24927]: 02:00.285942+0000 mon.a (mon.0) 1549 : cluster [DBG] osdmap e550: 8 total, 8 up, 8 in 2021-03-24T22:02:00.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:00 smithi125 conmon[24927]: audit 2021-03-24T 2021-03-24T22:02:00.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:00 smithi125 conmon[24927]: 22:02:00.316962+0000 mon.a (mon.0) 1550 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:02:00.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:00 smithi125 conmon[19936]: cluster 2021-03-24T22:01:58.892883+0000 mgr.y (mgr 2021-03-24T22:02:00.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:00 smithi125 conmon[19936]: .14144) 546 : cluster [DBG] pgmap v985: 168 pgs: 1 remapped+peering, 12 peering, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 152 active+clean; 555 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1.6 MiB/s rd, 948 KiB/s wr, 114 op/s; 62/1752 objects misplaced (3.539%); 16 MiB/s, 22 objects/s recovering 2021-03-24T22:02:00.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:00 smithi125 conmon[19936]: cluster 2021-03-24T22:01:59 2021-03-24T22:02:00.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:00 smithi125 conmon[19936]: .369066+0000 mon.a (mon. 2021-03-24T22:02:00.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:00 smithi125 conmon[19936]: 0) 1545 : cluster 2021-03-24T22:02:00.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:00 smithi125 conmon[19936]: [DBG] osdmap e549: 8 total, 8 up, 8 in 2021-03-24T22:02:00.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:00 smithi125 conmon[19936]: 2021-03-24T22:02:00.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:00 smithi125 conmon[19936]: audit 2021-03-24T22:01:59.811025 2021-03-24T22:02:00.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:00 smithi125 conmon[19936]: +0000 mon.a (mon.0) 1546 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:02:00.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:00 smithi125 conmon[19936]: audit 2021-03-24T22:02:00.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:00 smithi125 conmon[19936]: 2021-03-24T22:01:59.890070+0000 2021-03-24T22:02:00.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:00 smithi125 conmon[19936]: mon.a (mon.0) 1547 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T22:02:00.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:00 smithi125 conmon[19936]: 2021-03-24T22:02:00.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:00 smithi125 conmon[19936]: cluster 2021-03-24T22:02:00.669 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:00 smithi125 conmon[19936]: 2021-03-24T22:02:00.275873+0000 mon.a (mon. 2021-03-24T22:02:00.669 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:00 smithi125 conmon[19936]: 0) 1548 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:02:00.669 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:00 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T22:02:00.669 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:00 smithi125 conmon[19936]: 22:02:00.285942+0000 mon.a (mon. 2021-03-24T22:02:00.669 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:00 smithi125 conmon[19936]: 0) 1549 : cluster [DBG] osdmap e550: 8 total, 8 up, 8 in 2021-03-24T22:02:00.670 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:00 smithi125 conmon[19936]: audit 2021-03-24 2021-03-24T22:02:00.670 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:00 smithi125 conmon[19936]: T22:02:00.316962+0000 mon.a (mon.0 2021-03-24T22:02:00.670 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:00 smithi125 conmon[19936]: ) 1550 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:02:00.680 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:00.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:00 smithi142 conmon[20125]: cluster 2021-03-24T22: 2021-03-24T22:02:00.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:00 smithi142 conmon[20125]: 01:58.892883+0000 mgr.y (mgr.14144) 546 : cluster [DBG] pgmap v985: 168 pgs: 1 remapped+peering, 12 peering, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 152 active+clean; 555 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1.6 MiB/s rd, 948 KiB/s wr, 114 op/s; 62/1752 objects misplaced (3.539%); 16 MiB/s, 22 objects/s recovering 2021-03-24T22:02:00.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:00 smithi142 conmon[20125]: cluster 2021-03-24T22:02:00.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:00 smithi142 conmon[20125]: 2021-03-24T22:01:59. 2021-03-24T22:02:00.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:00 smithi142 conmon[20125]: 369066+0000 mon.a (mon.0) 2021-03-24T22:02:00.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:00 smithi142 conmon[20125]: 1545 : cluster [DBG] osdmap e549: 8 total, 8 up, 8 in 2021-03-24T22:02:00.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:00 smithi142 conmon[20125]: audit 2021 2021-03-24T22:02:00.695 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:00 smithi142 conmon[20125]: -03-24T22:01:59 2021-03-24T22:02:00.695 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:00 smithi142 conmon[20125]: .811025+0000 mon.a (mon.0) 1546 : 2021-03-24T22:02:00.695 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:00 smithi142 conmon[20125]: audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:02:00.695 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:00 smithi142 conmon[20125]: audit 2021-03- 2021-03-24T22:02:00.696 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:00 smithi142 conmon[20125]: 24T22:01:59.890070+0000 2021-03-24T22:02:00.696 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:00 smithi142 conmon[20125]: mon.a (mon.0) 1547 : audit [DBG] 2021-03-24T22:02:00.696 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:00 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T22:02:00.696 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:00 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:02:00.696 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:00 smithi142 conmon[20125]: 22:02:00.275873+0000 2021-03-24T22:02:00.697 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:00 smithi142 conmon[20125]: mon.a (mon.0) 1548 : cluster [WRN] 2021-03-24T22:02:00.697 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:00 smithi142 conmon[20125]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:02:00.697 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:00 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:02:00.697 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:00 smithi142 conmon[20125]: T22:02:00.285942+0000 2021-03-24T22:02:00.698 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:00 smithi142 conmon[20125]: mon.a (mon.0) 1549 : cluster [DBG] 2021-03-24T22:02:00.698 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:00 smithi142 conmon[20125]: osdmap e550: 8 total, 8 up, 8 in 2021-03-24T22:02:00.698 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:00 smithi142 conmon[20125]: audit 2021-03-24T22:02:00.698 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:00 smithi142 conmon[20125]: 2021-03-24T22:02: 2021-03-24T22:02:00.699 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:00 smithi142 conmon[20125]: 00.316962+0000 mon.a (mon.0) 1550 2021-03-24T22:02:00.699 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:00 smithi142 conmon[20125]: : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:02:00.729 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:00.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:00.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:02:00.834 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:00.891 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:00.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:00.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:02:00.996 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:01.041 INFO:tasks.thrashosds.thrasher:Added osd 7 2021-03-24T22:02:01.048 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:01.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:01.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:02:01.154 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:01.207 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:01.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:01.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:02:01.306 INFO:tasks.rados.rados.0.smithi125.stdout:2074: done (6 left) 2021-03-24T22:02:01.306 INFO:tasks.rados.rados.0.smithi125.stdout:2075: copy_from oid 18 from oid 2 current snap is 231 2021-03-24T22:02:01.307 INFO:tasks.rados.rados.0.smithi125.stdout:2076: write oid 42 current snap is 231 2021-03-24T22:02:01.307 INFO:tasks.rados.rados.0.smithi125.stdout:2076: seq_num 725 ranges {585075=409060,1542151=642924} 2021-03-24T22:02:01.307 INFO:tasks.rados.rados.0.smithi125.stdout:2076: writing smithi12556470-42 from 585075 to 994135 tid 1 2021-03-24T22:02:01.311 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:01.314 INFO:tasks.rados.rados.0.smithi125.stdout:2076: writing smithi12556470-42 from 1542151 to 2185075 tid 2 2021-03-24T22:02:01.315 INFO:tasks.rados.rados.0.smithi125.stdout:2077: read oid 31 snap -1 2021-03-24T22:02:01.315 INFO:tasks.rados.rados.0.smithi125.stdout:2077: expect (ObjNum 718 snap 229 seq_num 718) 2021-03-24T22:02:01.315 INFO:tasks.rados.rados.0.smithi125.stdout:2078: delete oid 23 current snap is 231 2021-03-24T22:02:01.315 INFO:tasks.rados.rados.0.smithi125.stdout:2066: finishing write tid 2 to smithi12556470-5 2021-03-24T22:02:01.316 INFO:tasks.rados.rados.0.smithi125.stdout:2066: finishing write tid 3 to smithi12556470-5 2021-03-24T22:02:01.316 INFO:tasks.rados.rados.0.smithi125.stdout:2066: finishing write tid 4 to smithi12556470-5 2021-03-24T22:02:01.316 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 5 v 592 (ObjNum 722 snap 231 seq_num 722) dirty exists 2021-03-24T22:02:01.316 INFO:tasks.rados.rados.0.smithi125.stdout:2066: left oid 5 (ObjNum 722 snap 231 seq_num 722) 2021-03-24T22:02:01.317 INFO:tasks.rados.rados.0.smithi125.stdout:2069: finishing copy_from to smithi12556470-26 2021-03-24T22:02:01.317 INFO:tasks.rados.rados.0.smithi125.stdout:2069: got expected ENOENT (src dne) 2021-03-24T22:02:01.317 INFO:tasks.rados.rados.0.smithi125.stdout:2065: expect (ObjNum 588 snap 177 seq_num 588) 2021-03-24T22:02:01.355 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:01.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:01.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:02:01.459 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:01.511 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:01.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:01.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:02:01.563 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-24T22:02:01.563 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T22:02:01.617 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:01.654 INFO:tasks.rados.rados.0.smithi125.stdout:2069: finishing copy_from racing read to smithi12556470-26 2021-03-24T22:02:01.655 INFO:tasks.rados.rados.0.smithi125.stdout:2065: done (9 left) 2021-03-24T22:02:01.655 INFO:tasks.rados.rados.0.smithi125.stdout:2066: done (8 left) 2021-03-24T22:02:01.656 INFO:tasks.rados.rados.0.smithi125.stdout:2068: done (7 left) 2021-03-24T22:02:01.656 INFO:tasks.rados.rados.0.smithi125.stdout:2069: done (6 left) 2021-03-24T22:02:01.657 INFO:tasks.rados.rados.0.smithi125.stdout:2078: done (5 left) 2021-03-24T22:02:01.657 INFO:tasks.rados.rados.0.smithi125.stdout:2079: snap_create 2021-03-24T22:02:01.657 INFO:tasks.rados.rados.0.smithi125.stdout:2070: finishing write tid 1 to smithi12556470-17 2021-03-24T22:02:01.657 INFO:tasks.rados.rados.0.smithi125.stdout:2070: finishing write tid 2 to smithi12556470-17 2021-03-24T22:02:01.657 INFO:tasks.rados.rados.0.smithi125.stdout:2070: finishing write tid 3 to smithi12556470-17 2021-03-24T22:02:01.658 INFO:tasks.rados.rados.0.smithi125.stdout:2070: finishing write tid 4 to smithi12556470-17 2021-03-24T22:02:01.658 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 17 v 596 (ObjNum 724 snap 231 seq_num 724) dirty exists 2021-03-24T22:02:01.658 INFO:tasks.rados.rados.0.smithi125.stdout:2070: left oid 17 (ObjNum 724 snap 231 seq_num 724) 2021-03-24T22:02:01.658 INFO:tasks.rados.rados.0.smithi125.stdout:2072: finishing rollback tid 0 to smithi12556470-11 2021-03-24T22:02:01.659 INFO:tasks.rados.rados.0.smithi125.stdout:2072: finishing rollback tid 1 to smithi12556470-11 2021-03-24T22:02:01.659 INFO:tasks.rados.rados.0.smithi125.stdout:2072: finishing rollback tid 2 to smithi12556470-11 2021-03-24T22:02:01.659 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 11 v 343 (ObjNum 495 snap 142 seq_num 495) dirty exists 2021-03-24T22:02:01.659 INFO:tasks.rados.rados.0.smithi125.stdout:2075: finishing copy_from to smithi12556470-18 2021-03-24T22:02:01.660 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 18 v 692 (ObjNum 676 snap 210 seq_num 676) dirty exists 2021-03-24T22:02:01.660 INFO:tasks.rados.rados.0.smithi125.stdout:2075: finishing copy_from racing read to smithi12556470-18 2021-03-24T22:02:01.660 INFO:tasks.rados.rados.0.smithi125.stdout:2076: finishing write tid 1 to smithi12556470-42 2021-03-24T22:02:01.660 INFO:tasks.rados.rados.0.smithi125.stdout:2076: finishing write tid 2 to smithi12556470-42 2021-03-24T22:02:01.660 INFO:tasks.rados.rados.0.smithi125.stdout:2076: finishing write tid 3 to smithi12556470-42 2021-03-24T22:02:01.661 INFO:tasks.rados.rados.0.smithi125.stdout:2076: finishing write tid 4 to smithi12556470-42 2021-03-24T22:02:01.661 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 42 v 500 (ObjNum 725 snap 231 seq_num 725) dirty exists 2021-03-24T22:02:01.661 INFO:tasks.rados.rados.0.smithi125.stdout:2076: left oid 42 (ObjNum 725 snap 231 seq_num 725) 2021-03-24T22:02:01.661 INFO:tasks.rados.rados.0.smithi125.stdout:2077: expect (ObjNum 718 snap 229 seq_num 718) 2021-03-24T22:02:01.669 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:01.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:01.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:02:01.773 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:01.788 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T22:02:01.809 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T22:02:01.820 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:01.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:01.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:02:01.835 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T22:02:01.855 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T22:02:01.875 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T22:02:01.903 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T22:02:01.925 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T22:02:01.926 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:01.950 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T22:02:01.952 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-24T22:02:01.952 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-24T22:02:01.953 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-24T22:02:01.953 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-24T22:02:01.953 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-24T22:02:01.954 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-24T22:02:01.954 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-24T22:02:01.955 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-24T22:02:01.971 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:01.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:01.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:02:02.078 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:02.121 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:02.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:02.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:02:02.227 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:02.281 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:02.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:02.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:02:02.299 INFO:tasks.rados.rados.0.smithi125.stdout:2070: done (5 left) 2021-03-24T22:02:02.300 INFO:tasks.rados.rados.0.smithi125.stdout:2072: done (4 left) 2021-03-24T22:02:02.300 INFO:tasks.rados.rados.0.smithi125.stdout:2075: done (3 left) 2021-03-24T22:02:02.300 INFO:tasks.rados.rados.0.smithi125.stdout:2076: done (2 left) 2021-03-24T22:02:02.301 INFO:tasks.rados.rados.0.smithi125.stdout:2077: done (1 left) 2021-03-24T22:02:02.301 INFO:tasks.rados.rados.0.smithi125.stdout:2079: done (0 left) 2021-03-24T22:02:02.301 INFO:tasks.rados.rados.0.smithi125.stdout:2080: snap_remove snap 218 2021-03-24T22:02:02.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:02 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:02:02.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:02 smithi142 conmon[20125]: 22:02:00.893414+0000 mgr.y (mgr.14144) 547 : cluster [DBG] pgmap v988: 168 pgs: 2 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 1 active+remapped+backfill_wait, 1 remapped+peering, 12 peering, 2 active+remapped+backfill_toofull, 143 active+clean; 557 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 0 B/s wr, 55 op/s; 131/1758 objects degraded (7.452%); 198/1758 objects misplaced (11.263%); 848 B/s, 13 objects/s recovering 2021-03-24T22:02:02.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:02 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:02:02.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:02 smithi142 conmon[20125]: -03-24 2021-03-24T22:02:02.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:02 smithi142 conmon[20125]: T22:02 2021-03-24T22:02:02.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:02 smithi142 conmon[20125]: :01.297081+0000 2021-03-24T22:02:02.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:02 smithi142 conmon[20125]: mon.a (mon. 2021-03-24T22:02:02.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:02 smithi142 conmon[20125]: 0) 1551 : cluster 2021-03-24T22:02:02.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:02 smithi142 conmon[20125]: [DBG] osdmap e551: 8 total, 8 up, 8 in 2021-03-24T22:02:02.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:02 smithi142 conmon[20125]: cluster 2021-03-24T22:02:02.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:02 smithi142 conmon[20125]: 2021-03 2021-03-24T22:02:02.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:02 smithi142 conmon[20125]: -24T22: 2021-03-24T22:02:02.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:02 smithi142 conmon[20125]: 02:01.373316 2021-03-24T22:02:02.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:02 smithi142 conmon[20125]: +0000 mon.a (mon 2021-03-24T22:02:02.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:02 smithi142 conmon[20125]: .0) 1552 : 2021-03-24T22:02:02.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:02 smithi142 conmon[20125]: cluster [WRN] Health check failed: Degraded data redundancy: 131/1758 objects degraded (7.452%), 5 pgs degraded (PG_DEGRADED) 2021-03-24T22:02:02.374 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:02 smithi142 conmon[20125]: 2021-03-24T22:02:02.374 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:02 smithi142 conmon[20125]: audit 2021-03-24T22:02:02.375 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:02 smithi142 conmon[20125]: 2021-03 2021-03-24T22:02:02.375 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:02 smithi142 conmon[20125]: -24T22 2021-03-24T22:02:02.375 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:02 smithi142 conmon[20125]: :02:01. 2021-03-24T22:02:02.376 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:02 smithi142 conmon[20125]: 761418+0000 mon.a ( 2021-03-24T22:02:02.376 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:02 smithi142 conmon[20125]: mon.0) 1553 : audit 2021-03-24T22:02:02.377 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:02 smithi142 conmon[20125]: [DBG] from='client.? v1:172.21.15.125:0/407879035' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:02:02.377 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:02 smithi142 conmon[20125]: 2021-03-24T22:02:02.387 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:02.437 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:02.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:02.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:02:02.542 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:02.555 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:02 smithi125 conmon[19936]: cluster 2021-03-24T22:02:00.893414+0000 mgr.y (mgr.14144) 547 : cluster 2021-03-24T22:02:02.555 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:02 smithi125 conmon[19936]: [DBG] pgmap v988: 168 pgs: 2 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 1 active+remapped+backfill_wait, 1 remapped+peering, 12 peering, 2 active+remapped+backfill_toofull, 143 active+clean; 557 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 0 B/s wr, 55 op/s; 131/1758 objects degraded (7.452%); 198/1758 objects misplaced (11.263%); 848 B/s, 13 objects/s recovering 2021-03-24T22:02:02.555 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:02 smithi125 conmon[19936]: cluster 2021-03-24T22:02:01.297081+0000 mon.a (mon.0) 1551 : cluster [DBG] osdmap e551: 8 total, 8 up, 8 in 2021-03-24T22:02:02.555 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:02 smithi125 conmon[19936]: cluster 2021-03-24T22:02:01.373316+0000 mon.a (mon.0) 1552 : cluster [WRN] Health check failed: Degraded data redundancy: 131/1758 objects degraded (7.452%), 5 pgs degraded (PG_DEGRADED) 2021-03-24T22:02:02.556 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:02 smithi125 conmon[19936]: audit 2021-03-24T22:02:01.761418+0000 mon.a (mon.0) 1553 : audit [DBG] from='client.? v1:172.21.15.125:0/407879035' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:02:02.556 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:02 smithi125 conmon[24927]: cluster 2021-03-24T22:02:02.557 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:02 smithi125 conmon[24927]: 2021-03-24T22:02:02.557 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:02 smithi125 conmon[24927]: 2021 2021-03-24T22:02:02.557 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:02 smithi125 conmon[24927]: - 2021-03-24T22:02:02.557 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:02 smithi125 conmon[24927]: 03 2021-03-24T22:02:02.557 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:02 smithi125 conmon[24927]: - 2021-03-24T22:02:02.558 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:02 smithi125 conmon[24927]: 24 2021-03-24T22:02:02.558 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:02 smithi125 conmon[24927]: T 2021-03-24T22:02:02.558 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:02 smithi125 conmon[24927]: 22 2021-03-24T22:02:02.558 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:02 smithi125 conmon[24927]: : 2021-03-24T22:02:02.559 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:02 smithi125 conmon[24927]: 02 2021-03-24T22:02:02.559 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:02 smithi125 conmon[24927]: : 2021-03-24T22:02:02.559 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:02 smithi125 conmon[24927]: 00 2021-03-24T22:02:02.559 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:02 smithi125 conmon[24927]: . 2021-03-24T22:02:02.560 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:02 smithi125 conmon[24927]: 893414 2021-03-24T22:02:02.560 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:02 smithi125 conmon[24927]: +0000 2021-03-24T22:02:02.560 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:02 smithi125 conmon[24927]: 2021-03-24T22:02:02.560 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:02 smithi125 conmon[24927]: mgr.y (mgr.14144) 547 : cluster 2021-03-24T22:02:02.560 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:02 smithi125 conmon[24927]: [DBG] pgmap v988: 168 pgs: 2 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 1 active+remapped+backfill_wait, 1 remapped+peering, 12 peering, 2 active+remapped+backfill_toofull, 143 active+clean; 557 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 0 B/s wr, 55 op/s; 131/1758 objects degraded (7.452%); 198/1758 objects misplaced (11.263%); 848 B/s, 13 objects/s recovering 2021-03-24T22:02:02.561 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:02 smithi125 conmon[24927]: cluster 2021-03-24T22:02:01.297081+0000 mon.a (mon 2021-03-24T22:02:02.561 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:02 smithi125 conmon[24927]: .0) 1551 : cluster [DBG] osdmap e551: 8 total, 8 up, 8 in 2021-03-24T22:02:02.561 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:02 smithi125 conmon[24927]: cluster 2021-03-24T22:02:02.561 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:02 smithi125 conmon[24927]: 2021-03-24T22:02:01.373316+0000 2021-03-24T22:02:02.562 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:02 smithi125 conmon[24927]: mon.a (mon.0) 1552 : cluster [WRN] 2021-03-24T22:02:02.562 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:02 smithi125 conmon[24927]: Health check failed: Degraded data redundancy: 131/1758 objects degraded (7.452%), 5 pgs degraded (PG_DEGRADED) 2021-03-24T22:02:02.562 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:02 smithi125 conmon[24927]: audit 2021-03-24T 2021-03-24T22:02:02.562 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:02 smithi125 conmon[24927]: 22:02:01.761418+0000 mon.a (mon. 2021-03-24T22:02:02.563 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:02 smithi125 conmon[24927]: 0) 1553 : audit [DBG] from='client.? v1:172.21.15.125:0/407879035' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:02:02.593 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:02.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:02.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:02.699 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:02.749 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:02.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:02.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:02:02.856 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:02.904 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:02.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:02.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:03.011 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:03.068 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:03.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:03.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:02:03.174 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:03.225 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:03.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:03.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:03.305 INFO:tasks.rados.rados.0.smithi125.stdout:2080: done (0 left) 2021-03-24T22:02:03.306 INFO:tasks.rados.rados.0.smithi125.stdout:2081: snap_create 2021-03-24T22:02:03.331 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:03.364 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:03 smithi125 conmon[19936]: cluster 2021-03-24T22:02:02.296971+0000 mon.a ( 2021-03-24T22:02:03.378 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:03.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:03.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:02:03.411 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-24T22:02:03.482 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:03.530 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:03.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:03.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:02:03.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:03 smithi142 conmon[20125]: cluster 2021-03-24T22:02:02.296971+0000 2021-03-24T22:02:03.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:03 smithi142 conmon[20125]: mon.a (mon.0) 1554 : cluster [DBG] osdmap e552: 8 total, 8 up, 8 in 2021-03-24T22:02:03.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:03 smithi142 conmon[20125]: cluster 2021-03-24T22:02:02.539807+0000 osd.5 2021-03-24T22:02:03.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:03 smithi142 conmon[20125]: (osd.5) 23 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:02:03.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:03 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:02:03.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:03 smithi142 conmon[20125]: -24T22:02:02.544166 2021-03-24T22:02:03.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:03 smithi142 conmon[20125]: +0000 osd.5 (osd.5) 24 2021-03-24T22:02:03.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:03 smithi142 conmon[20125]: : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:02:03.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:03 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:02:03.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:03 smithi142 conmon[20125]: 03-24T22:02:03 2021-03-24T22:02:03.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:03 smithi142 conmon[20125]: .301613+0000 mon.a (mon.0 2021-03-24T22:02:03.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:03 smithi142 conmon[20125]: ) 1555 : cluster [DBG] osdmap e553: 8 total, 8 up, 8 in 2021-03-24T22:02:03.637 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:03.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:03 smithi125 conmon[19936]: mon.0) 1554 : cluster [DBG] osdmap e552: 8 total, 8 up, 8 in 2021-03-24T22:02:03.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:03 smithi125 conmon[19936]: cluster 2021-03-24T22:02:02.539807+0000 osd.5 (osd.5) 23 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:02:03.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:03 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:02:03.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:03 smithi125 conmon[19936]: 03-24T22:02:02.544166+0000 osd.5 (osd.5) 24 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:02:03.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:03 smithi125 conmon[19936]: cluster 2021-03-24T22:02:03. 2021-03-24T22:02:03.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:03 smithi125 conmon[19936]: 301613+0000 mon.a (mon.0) 1555 : cluster 2021-03-24T22:02:03.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:03 smithi125 conmon[19936]: [DBG] osdmap e553: 8 total, 8 up, 8 in 2021-03-24T22:02:03.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:03 smithi125 conmon[24927]: cluster 2021-03-24T22:02:03.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:03 smithi125 conmon[24927]: 2021-03-24T22:02:03.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:03 smithi125 conmon[24927]: 2021 2021-03-24T22:02:03.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:03 smithi125 conmon[24927]: - 2021-03-24T22:02:03.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:03 smithi125 conmon[24927]: 03 2021-03-24T22:02:03.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:03 smithi125 conmon[24927]: - 2021-03-24T22:02:03.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:03 smithi125 conmon[24927]: 24 2021-03-24T22:02:03.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:03 smithi125 conmon[24927]: T 2021-03-24T22:02:03.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:03 smithi125 conmon[24927]: 22 2021-03-24T22:02:03.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:03 smithi125 conmon[24927]: : 2021-03-24T22:02:03.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:03 smithi125 conmon[24927]: 02 2021-03-24T22:02:03.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:03 smithi125 conmon[24927]: : 2021-03-24T22:02:03.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:03 smithi125 conmon[24927]: 02 2021-03-24T22:02:03.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:03 smithi125 conmon[24927]: . 2021-03-24T22:02:03.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:03 smithi125 conmon[24927]: 296971 2021-03-24T22:02:03.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:03 smithi125 conmon[24927]: +0000 2021-03-24T22:02:03.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:03 smithi125 conmon[24927]: 2021-03-24T22:02:03.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:03 smithi125 conmon[24927]: mon.a 2021-03-24T22:02:03.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:03 smithi125 conmon[24927]: ( 2021-03-24T22:02:03.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:03 smithi125 conmon[24927]: mon 2021-03-24T22:02:03.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:03 smithi125 conmon[24927]: . 2021-03-24T22:02:03.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:03 smithi125 conmon[24927]: 0) 1554 : cluster 2021-03-24T22:02:03.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:03 smithi125 conmon[24927]: [DBG] osdmap e552: 8 total, 8 up, 8 in 2021-03-24T22:02:03.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:03 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:02:03.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:03 smithi125 conmon[24927]: 24T 2021-03-24T22:02:03.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:03 smithi125 conmon[24927]: 22:02 2021-03-24T22:02:03.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:03 smithi125 conmon[24927]: :02.539807+0000 osd.5 (osd.5) 2021-03-24T22:02:03.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:03 smithi125 conmon[24927]: 23 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:02:03.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:03 smithi125 conmon[24927]: cluster 2021-03-24T22:02:03.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:03 smithi125 conmon[24927]: 2021-03-24T22:02: 2021-03-24T22:02:03.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:03 smithi125 conmon[24927]: 02.544166+0000 osd.5 (osd.5) 24 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:02:03.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:03 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:02:03.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:03 smithi125 conmon[24927]: -03-24T22:02:03.301613+0000 mon.a 2021-03-24T22:02:03.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:03 smithi125 conmon[24927]: (mon.0) 1555 : cluster [DBG] osdmap e553: 8 total, 8 up, 8 in 2021-03-24T22:02:03.691 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:03.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:03.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:02:03.795 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:03.819 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.3 dump_historic_ops 2021-03-24T22:02:03.837 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:03.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:03.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:02:03.900 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:02:03.943 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:03.997 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:04.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:04.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:02:04.102 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:04.156 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:04.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:04.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:02:04.261 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:04.314 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:04.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:04.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:02:04.321 INFO:tasks.rados.rados.0.smithi125.stdout:2081: done (0 left) 2021-03-24T22:02:04.321 INFO:tasks.rados.rados.0.smithi125.stdout:2082: write oid 7 current snap is 233 2021-03-24T22:02:04.322 INFO:tasks.rados.rados.0.smithi125.stdout:2082: seq_num 727 ranges {618689=559558,1659115=559574} 2021-03-24T22:02:04.331 INFO:tasks.rados.rados.0.smithi125.stdout:2082: writing smithi12556470-7 from 618689 to 1178247 tid 1 2021-03-24T22:02:04.336 INFO:tasks.rados.rados.0.smithi125.stdout:2082: writing smithi12556470-7 from 1659115 to 2218689 tid 2 2021-03-24T22:02:04.337 INFO:tasks.rados.rados.0.smithi125.stdout:2083: read oid 26 snap -1 2021-03-24T22:02:04.337 INFO:tasks.rados.rados.0.smithi125.stdout:2083: expect deleted 2021-03-24T22:02:04.337 INFO:tasks.rados.rados.0.smithi125.stdout:2084: snap_create 2021-03-24T22:02:04.337 INFO:tasks.rados.rados.0.smithi125.stdout:2082: finishing write tid 1 to smithi12556470-7 2021-03-24T22:02:04.339 INFO:tasks.rados.rados.0.smithi125.stdout:2082: finishing write tid 2 to smithi12556470-7 2021-03-24T22:02:04.340 INFO:tasks.rados.rados.0.smithi125.stdout:2082: finishing write tid 3 to smithi12556470-7 2021-03-24T22:02:04.340 INFO:tasks.rados.rados.0.smithi125.stdout:2082: finishing write tid 4 to smithi12556470-7 2021-03-24T22:02:04.341 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 7 v 735 (ObjNum 727 snap 233 seq_num 727) dirty exists 2021-03-24T22:02:04.341 INFO:tasks.rados.rados.0.smithi125.stdout:2082: left oid 7 (ObjNum 727 snap 233 seq_num 727) 2021-03-24T22:02:04.343 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:02:04.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:04 smithi142 conmon[20125]: cluster 2021-03-24T22:02:04.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:04 smithi142 conmon[20125]: 2021-03-24T22:02:04.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:04 smithi142 conmon[20125]: 2021 2021-03-24T22:02:04.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:04 smithi142 conmon[20125]: - 2021-03-24T22:02:04.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:04 smithi142 conmon[20125]: 03 2021-03-24T22:02:04.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:04 smithi142 conmon[20125]: - 2021-03-24T22:02:04.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:04 smithi142 conmon[20125]: 24 2021-03-24T22:02:04.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:04 smithi142 conmon[20125]: T 2021-03-24T22:02:04.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:04 smithi142 conmon[20125]: 22 2021-03-24T22:02:04.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:04 smithi142 conmon[20125]: : 2021-03-24T22:02:04.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:04 smithi142 conmon[20125]: 02 2021-03-24T22:02:04.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:04 smithi142 conmon[20125]: : 2021-03-24T22:02:04.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:04 smithi142 conmon[20125]: 02 2021-03-24T22:02:04.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:04 smithi142 conmon[20125]: . 2021-03-24T22:02:04.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:04 smithi142 conmon[20125]: 894101 2021-03-24T22:02:04.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:04 smithi142 conmon[20125]: +0000 2021-03-24T22:02:04.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:04 smithi142 conmon[20125]: 2021-03-24T22:02:04.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:04 smithi142 conmon[20125]: mgr.y 2021-03-24T22:02:04.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:04 smithi142 conmon[20125]: ( 2021-03-24T22:02:04.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:04 smithi142 conmon[20125]: mgr 2021-03-24T22:02:04.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:04 smithi142 conmon[20125]: . 2021-03-24T22:02:04.373 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:04 smithi142 conmon[20125]: 14144) 548 : cluster [DBG] pgmap v991: 168 pgs: 2 active+recovery_wait+remapped, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 152 active+clean; 561 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 425 KiB/s wr, 59 op/s; 187/1770 objects degraded (10.565%); 280/1770 objects misplaced (15.819%); 2.0 MiB/s, 28 objects/s recovering 2021-03-24T22:02:04.373 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:04 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:02:04.374 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:04 smithi142 conmon[20125]: 22:02:04.310544+0000 mon.a (mon.0) 1556 2021-03-24T22:02:04.374 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:04 smithi142 conmon[20125]: : cluster [DBG] osdmap e554: 8 total, 8 up, 8 in 2021-03-24T22:02:04.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:04 smithi125 conmon[19936]: cluster 2021-03-24T22:02:02.894101+0000 mgr.y ( 2021-03-24T22:02:04.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:04 smithi125 conmon[19936]: mgr.14144) 548 : cluster [DBG] pgmap v991: 168 pgs: 2 active+recovery_wait+remapped, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 152 active+clean; 561 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 425 KiB/s wr, 59 op/s; 187/1770 objects degraded (10.565%); 280/1770 objects misplaced (15.819%); 2.0 MiB/s, 28 objects/s recovering 2021-03-24T22:02:04.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:04 smithi125 conmon[19936]: cluster 2021-03-24T22:02:04.310544+0000 mon.a (mon.0) 2021-03-24T22:02:04.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:04 smithi125 conmon[19936]: 1556 : cluster [DBG] osdmap e554: 8 total, 8 up, 8 in 2021-03-24T22:02:04.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:04 smithi125 conmon[24927]: cluster 2021-03-24T22:02:02 2021-03-24T22:02:04.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:04 smithi125 conmon[24927]: .894101+0000 mgr.y (mgr.14144) 548 : cluster [DBG] pgmap v991: 168 pgs: 2 active+recovery_wait+remapped, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 152 active+clean; 561 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 425 KiB/s wr, 59 op/s; 187/1770 objects degraded (10.565%); 280/1770 objects misplaced (15.819%); 2.0 MiB/s, 28 objects/s recovering 2021-03-24T22:02:04.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:04 smithi125 conmon[24927]: cluster 2021-03-24T22:02:04.310544+0000 mon.a (mon 2021-03-24T22:02:04.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:04 smithi125 conmon[24927]: .0) 1556 : cluster [DBG] osdmap e554: 8 total, 8 up, 8 in 2021-03-24T22:02:04.422 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:04.476 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:04.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:04.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:02:04.580 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:04.633 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:04.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:04.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:04.738 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:04.790 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:04.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:04.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:02:04.896 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:04.953 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:04.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:04.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:02:05.062 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:05.115 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:05.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:05.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:02:05.220 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:05.278 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:05.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:05.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:02:05.292 INFO:tasks.rados.rados.0.smithi125.stdout:2082: done (2 left) 2021-03-24T22:02:05.292 INFO:tasks.rados.rados.0.smithi125.stdout:2083: done (1 left) 2021-03-24T22:02:05.293 INFO:tasks.rados.rados.0.smithi125.stdout:2084: done (0 left) 2021-03-24T22:02:05.293 INFO:tasks.rados.rados.0.smithi125.stdout:2085: write oid 4 current snap is 234 2021-03-24T22:02:05.293 INFO:tasks.rados.rados.0.smithi125.stdout:2085: seq_num 728 ranges {635496=634807,1717597=517899} 2021-03-24T22:02:05.299 INFO:tasks.rados.rados.0.smithi125.stdout:2085: writing smithi12556470-4 from 635496 to 1270303 tid 1 2021-03-24T22:02:05.305 INFO:tasks.rados.rados.0.smithi125.stdout:2085: writing smithi12556470-4 from 1717597 to 2235496 tid 2 2021-03-24T22:02:05.305 INFO:tasks.rados.rados.0.smithi125.stdout:2086: delete oid 5 current snap is 234 2021-03-24T22:02:05.306 INFO:tasks.rados.rados.0.smithi125.stdout:2085: finishing write tid 1 to smithi12556470-4 2021-03-24T22:02:05.308 INFO:tasks.rados.rados.0.smithi125.stdout:2086: done (1 left) 2021-03-24T22:02:05.308 INFO:tasks.rados.rados.0.smithi125.stdout:2087: snap_remove snap 135 2021-03-24T22:02:05.327 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:05 smithi125 conmon[19936]: audit 2021-03-24T22: 2021-03-24T22:02:05.328 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:05 smithi125 conmon[19936]: 02:04.808799+0000 mon.a (mon.0) 1557 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T22:02:05.328 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:05 smithi125 conmon[19936]: audit 2021-03-24T 2021-03-24T22:02:05.328 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:05 smithi125 conmon[19936]: 22:02:04.809265+0000 mon.a (mon.0) 1558 : 2021-03-24T22:02:05.328 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:05 smithi125 conmon[19936]: audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T22:02:05.329 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:05 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:02:05.329 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:05 smithi125 conmon[19936]: 03-24T22:02:05. 2021-03-24T22:02:05.329 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:05 smithi125 conmon[19936]: 278546+0000 mon.a (mon.0) 1559 2021-03-24T22:02:05.329 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:05 smithi125 conmon[19936]: : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:02:05.329 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:05 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:02:05.330 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:05 smithi125 conmon[19936]: 24T22:02:05.287638+0000 mon.a (mon 2021-03-24T22:02:05.330 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:05 smithi125 conmon[19936]: .0) 1560 : cluster [DBG] osdmap e555: 8 total, 8 up, 8 in 2021-03-24T22:02:05.384 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:05.430 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:05.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:05.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:02:05.537 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:05.578 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:05 smithi125 conmon[24927]: audit 2021-03-24T22:02:05.578 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:05 smithi125 conmon[24927]: 2021-03-24T22:02:05.578 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:05 smithi125 conmon[24927]: 2021 2021-03-24T22:02:05.579 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:05 smithi125 conmon[24927]: - 2021-03-24T22:02:05.579 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:05 smithi125 conmon[24927]: 03 2021-03-24T22:02:05.579 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:05 smithi125 conmon[24927]: - 2021-03-24T22:02:05.579 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:05 smithi125 conmon[24927]: 24T22:02: 2021-03-24T22:02:05.580 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:05 smithi125 conmon[24927]: 04.808799+0000 mon.a (mon.0) 1557 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T22:02:05.580 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:05 smithi125 conmon[24927]: audit 2021-03-24T22: 2021-03-24T22:02:05.580 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:05 smithi125 conmon[24927]: 02:04.809265+0000 mon.a (mon.0) 1558 2021-03-24T22:02:05.580 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:05 smithi125 conmon[24927]: : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T22:02:05.581 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:05 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:02:05.581 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:05 smithi125 conmon[24927]: -03-24T22:02:05.278546+0000 mon.a (mon 2021-03-24T22:02:05.581 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:05 smithi125 conmon[24927]: .0) 1559 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:02:05.581 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:05 smithi125 conmon[24927]: cluster 2021-03-24T22:02:05.582 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:05 smithi125 conmon[24927]: 2021 2021-03-24T22:02:05.582 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:05 smithi125 conmon[24927]: - 2021-03-24T22:02:05.582 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:05 smithi125 conmon[24927]: 03 2021-03-24T22:02:05.582 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:05 smithi125 conmon[24927]: - 2021-03-24T22:02:05.583 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:05 smithi125 conmon[24927]: 24 2021-03-24T22:02:05.583 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:05 smithi125 conmon[24927]: T 2021-03-24T22:02:05.583 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:05 smithi125 conmon[24927]: 22 2021-03-24T22:02:05.583 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:05 smithi125 conmon[24927]: : 2021-03-24T22:02:05.583 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:05 smithi125 conmon[24927]: 02 2021-03-24T22:02:05.584 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:05 smithi125 conmon[24927]: :05.287638+0000 mon.a 2021-03-24T22:02:05.584 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:05 smithi125 conmon[24927]: (mon.0) 1560 : cluster [DBG] 2021-03-24T22:02:05.584 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:05 smithi125 conmon[24927]: osdmap e555: 8 total, 8 up, 8 in 2021-03-24T22:02:05.586 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:05.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:05.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:02:05.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:05 smithi142 conmon[20125]: audit 2021-03-24T22:02:04.808799+0000 2021-03-24T22:02:05.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:05 smithi142 conmon[20125]: mon.a (mon.0) 1557 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T22:02:05.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:05 smithi142 conmon[20125]: audit 2021-03-24T22:02:04.809265+0000 mon.a (mon.0 2021-03-24T22:02:05.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:05 smithi142 conmon[20125]: ) 1558 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T22:02:05.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:05 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:02:05.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:05 smithi142 conmon[20125]: -03-24T22:02: 2021-03-24T22:02:05.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:05 smithi142 conmon[20125]: 05.278546+0000 mon.a ( 2021-03-24T22:02:05.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:05 smithi142 conmon[20125]: mon.0) 1559 : cluster 2021-03-24T22:02:05.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:05 smithi142 conmon[20125]: [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:02:05.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:05 smithi142 conmon[20125]: cluster 2021-03-24T22:02:05.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:05 smithi142 conmon[20125]: 2021-03-24T22:02 2021-03-24T22:02:05.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:05 smithi142 conmon[20125]: :05.287638+0000 mon.a 2021-03-24T22:02:05.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:05 smithi142 conmon[20125]: (mon.0) 1560 : 2021-03-24T22:02:05.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:05 smithi142 conmon[20125]: cluster [DBG] osdmap e555: 8 total, 8 up, 8 in 2021-03-24T22:02:05.693 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:05.743 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:05.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:05.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:05.848 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:05.854 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-24T22:02:05.902 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:05.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:05.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:02:06.008 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:06.042 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 1, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2021-03-24T22:02:06.042 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd reweight 0 1 2021-03-24T22:02:06.060 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:06.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:06.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:02:06.172 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:06.234 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:06.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:06.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:02:06.302 INFO:tasks.rados.rados.0.smithi125.stdout:2087: done (1 left) 2021-03-24T22:02:06.302 INFO:tasks.rados.rados.0.smithi125.stdout:2088: write (excl) oid 8 current snap is 234 2021-03-24T22:02:06.303 INFO:tasks.rados.rados.0.smithi125.stdout:2088: seq_num 730 ranges {669110=501658,1834561=434549} 2021-03-24T22:02:06.304 INFO:tasks.rados.rados.0.smithi125.stdout:2088: writing smithi12556470-8 from 669110 to 1170768 tid 1 2021-03-24T22:02:06.313 INFO:tasks.rados.rados.0.smithi125.stdout:2088: writing smithi12556470-8 from 1834561 to 2269110 tid 2 2021-03-24T22:02:06.314 INFO:tasks.rados.rados.0.smithi125.stdout:2085: finishing write tid 2 to smithi12556470-4 2021-03-24T22:02:06.315 INFO:tasks.rados.rados.0.smithi125.stdout:2085: finishing write tid 3 to smithi12556470-4 2021-03-24T22:02:06.315 INFO:tasks.rados.rados.0.smithi125.stdout:2085: finishing write tid 4 to smithi12556470-4 2021-03-24T22:02:06.315 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 4 v 501 (ObjNum 728 snap 234 seq_num 728) dirty exists 2021-03-24T22:02:06.315 INFO:tasks.rados.rados.0.smithi125.stdout:2085: left oid 4 (ObjNum 728 snap 234 seq_num 728) 2021-03-24T22:02:06.316 INFO:tasks.rados.rados.0.smithi125.stdout:2085: done (1 left) 2021-03-24T22:02:06.316 INFO:tasks.rados.rados.0.smithi125.stdout:2089: delete oid 11 current snap is 234 2021-03-24T22:02:06.318 INFO:tasks.rados.rados.0.smithi125.stdout:2088: finishing write tid 1 to smithi12556470-8 2021-03-24T22:02:06.318 INFO:tasks.rados.rados.0.smithi125.stdout:2089: done (1 left) 2021-03-24T22:02:06.319 INFO:tasks.rados.rados.0.smithi125.stdout:2090: snap_create 2021-03-24T22:02:06.319 INFO:tasks.rados.rados.0.smithi125.stdout:2088: finishing write tid 2 to smithi12556470-8 2021-03-24T22:02:06.320 INFO:tasks.rados.rados.0.smithi125.stdout:2088: finishing write tid 3 to smithi12556470-8 2021-03-24T22:02:06.320 INFO:tasks.rados.rados.0.smithi125.stdout:2088: finishing write tid 4 to smithi12556470-8 2021-03-24T22:02:06.320 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 8 v 172 (ObjNum 730 snap 234 seq_num 730) dirty exists 2021-03-24T22:02:06.321 INFO:tasks.rados.rados.0.smithi125.stdout:2088: left oid 8 (ObjNum 730 snap 234 seq_num 730) 2021-03-24T22:02:06.329 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:06 smithi125 conmon[19936]: cluster 2021-03-24T22:02:06.329 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:06 smithi125 conmon[19936]: 2021-03-24T22 2021-03-24T22:02:06.330 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:06 smithi125 conmon[19936]: :02:04.894879+0000 mgr.y (mgr.14144) 549 : cluster [DBG] pgmap v994: 168 pgs: 1 peering, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 156 active+clean; 563 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 682 KiB/s wr, 9 op/s; 141/1773 objects degraded (7.953%); 215/1773 objects misplaced (12.126%); 16 MiB/s, 47 objects/s recovering 2021-03-24T22:02:06.330 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:06 smithi125 conmon[19936]: cluster 2021-03-24T22:02:06.330 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:06 smithi125 conmon[19936]: 2021- 2021-03-24T22:02:06.330 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:06 smithi125 conmon[19936]: 03-24T22:02:06.286683+0000 mon.a 2021-03-24T22:02:06.331 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:06 smithi125 conmon[19936]: (mon.0) 1561 : cluster [DBG] osdmap e556: 8 total, 8 up, 8 in 2021-03-24T22:02:06.331 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:06 smithi125 conmon[24927]: cluster 2021-03-24T22:02:06.331 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:06 smithi125 conmon[24927]: 2021-03-24T22:02:06.332 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:06 smithi125 conmon[24927]: 2021-03-24T22:02: 2021-03-24T22:02:06.332 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:06 smithi125 conmon[24927]: 04.894879+0000 mgr.y (mgr.14144) 549 : cluster [DBG] pgmap v994: 168 pgs: 1 peering, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 156 active+clean; 563 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 682 KiB/s wr, 9 op/s; 141/1773 objects degraded (7.953%); 215/1773 objects misplaced (12.126%); 16 MiB/s, 47 objects/s recovering 2021-03-24T22:02:06.332 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:06 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T22:02:06.332 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:06 smithi125 conmon[24927]: -24T22:02:06.286683+0000 mon.a 2021-03-24T22:02:06.332 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:06 smithi125 conmon[24927]: (mon.0) 1561 : cluster [DBG] osdmap e556: 8 total, 8 up, 8 in 2021-03-24T22:02:06.333 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:06 smithi125 conmon[24927]: 2021-03-24T22:02:06.341 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:06.391 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:06.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:06.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:02:06.497 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:06.546 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:06.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:06.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:02:06.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:06 smithi142 conmon[20125]: cluster 2021-03-24T22:02:04.894879+0000 mgr.y (mgr.14144) 549 : cluster [DBG] pgmap v994: 168 pgs: 1 peering, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 156 active+clean; 563 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 682 KiB/s wr, 9 op/s; 141/1773 objects degraded (7.953%); 215/1773 objects misplaced (12.126%); 16 MiB/s, 47 objects/s recovering 2021-03-24T22:02:06.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:06 smithi142 conmon[20125]: cluster 2021-03-24T22:02:06.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:06 smithi142 conmon[20125]: 2021-03-24T22:02:06.286683+0000 mon.a (mon.0) 1561 : cluster [DBG] osdmap e556: 8 total, 8 up, 8 in 2021-03-24T22:02:06.651 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:06.707 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:06.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:06.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:02:06.814 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:06.900 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:06.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:06.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:02:06.923 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:02:07.006 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:07.062 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:07.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:07.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:02:07.070 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:02:07.178 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:07.239 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:07.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:07.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:02:07.294 INFO:tasks.rados.rados.0.smithi125.stdout:2088: done (1 left) 2021-03-24T22:02:07.294 INFO:tasks.rados.rados.0.smithi125.stdout:2090: done (0 left) 2021-03-24T22:02:07.295 INFO:tasks.rados.rados.0.smithi125.stdout:2091: delete oid 33 current snap is 235 2021-03-24T22:02:07.321 INFO:tasks.rados.rados.0.smithi125.stdout:2091: done (0 left) 2021-03-24T22:02:07.321 INFO:tasks.rados.rados.0.smithi125.stdout:2092: write (excl) oid 38 current snap is 235 2021-03-24T22:02:07.322 INFO:tasks.rados.rados.0.smithi125.stdout:2092: seq_num 733 ranges {719531=727405,1893654=425877} 2021-03-24T22:02:07.327 INFO:tasks.rados.rados.0.smithi125.stdout:2092: writing smithi12556470-38 from 719531 to 1446936 tid 1 2021-03-24T22:02:07.332 INFO:tasks.rados.rados.0.smithi125.stdout:2092: writing smithi12556470-38 from 1893654 to 2319531 tid 2 2021-03-24T22:02:07.333 INFO:tasks.rados.rados.0.smithi125.stdout:2093: snap_remove snap 228 2021-03-24T22:02:07.344 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:07.397 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:07.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:07.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:02:07.503 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:07.559 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:07.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:07.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:07.664 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:07.726 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:07.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:07.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:02:07.831 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:07.894 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:07.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:07.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:02:07.999 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:08.047 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:08.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:08.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:02:08.149 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-24T22:02:08.150 INFO:teuthology.orchestra.run.smithi125.stdout: "size": 20, 2021-03-24T22:02:08.150 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 600, 2021-03-24T22:02:08.150 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [ 2021-03-24T22:02:08.151 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.151 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:2390 7.23 7:c49de0b4:::smithi12556470-13:head [write 430345~727947] snapc a9=[a9,a8,a7,a0] ondisk+write+known_if_redirected e275)", 2021-03-24T22:02:08.151 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:57:25.173714+0000", 2021-03-24T22:02:08.151 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 282.96846124299998, 2021-03-24T22:02:08.152 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.018775558, 2021-03-24T22:02:08.152 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:02:08.153 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:02:08.153 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:02:08.153 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:02:08.153 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:02:08.154 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 2390 2021-03-24T22:02:08.154 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.154 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:02:08.154 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.155 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:02:08.155 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.173714+0000", 2021-03-24T22:02:08.155 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:02:08.155 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.156 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.156 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:02:08.156 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.173714+0000", 2021-03-24T22:02:08.156 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T22:02:08.157 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.157 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.157 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:02:08.158 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.173712+0000", 2021-03-24T22:02:08.158 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00044400899999999998 2021-03-24T22:02:08.158 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.158 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.158 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:02:08.159 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.174156+0000", 2021-03-24T22:02:08.159 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0098000000000001e-05 2021-03-24T22:02:08.159 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.159 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.160 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:02:08.160 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.174167+0000", 2021-03-24T22:02:08.160 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.7410000000000003e-06 2021-03-24T22:02:08.161 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.161 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.161 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:02:08.161 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.174173+0000", 2021-03-24T22:02:08.162 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.9721999999999994e-05 2021-03-24T22:02:08.162 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.163 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.163 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.163 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.174243+0000", 2021-03-24T22:02:08.163 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3264e-05 2021-03-24T22:02:08.163 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.164 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.164 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:02:08.164 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.174256+0000", 2021-03-24T22:02:08.164 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.003266601 2021-03-24T22:02:08.165 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.165 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.165 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.165 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.177523+0000", 2021-03-24T22:02:08.165 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00014048700000000001 2021-03-24T22:02:08.166 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.166 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.166 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:02:08.166 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.177663+0000", 2021-03-24T22:02:08.167 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00069449699999999997 2021-03-24T22:02:08.167 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.167 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.167 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,4", 2021-03-24T22:02:08.167 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.178358+0000", 2021-03-24T22:02:08.168 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.005790636 2021-03-24T22:02:08.168 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.168 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.168 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:02:08.169 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.184148+0000", 2021-03-24T22:02:08.169 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00056901200000000003 2021-03-24T22:02:08.169 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.169 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.169 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:02:08.170 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.184717+0000", 2021-03-24T22:02:08.170 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0076483189999999998 2021-03-24T22:02:08.170 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.170 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.170 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:02:08.171 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.192366+0000", 2021-03-24T22:02:08.171 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.1732999999999999e-05 2021-03-24T22:02:08.171 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.171 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.172 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:02:08.172 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.192438+0000", 2021-03-24T22:02:08.172 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2144000000000001e-05 2021-03-24T22:02:08.172 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.172 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.173 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:02:08.173 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.192490+0000", 2021-03-24T22:02:08.173 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.018775558 2021-03-24T22:02:08.174 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:02:08.174 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:02:08.174 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:02:08.174 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.175 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.175 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:2391 7.23 7:c49de0b4:::smithi12556470-13:head [write 1570056~727439] snapc a9=[a9,a8,a7,a0] ondisk+write+known_if_redirected e275)", 2021-03-24T22:02:08.175 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:57:25.174190+0000", 2021-03-24T22:02:08.175 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 282.96798516400003, 2021-03-24T22:02:08.175 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.018522293, 2021-03-24T22:02:08.176 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:02:08.176 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:02:08.176 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:02:08.177 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:02:08.177 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:02:08.177 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 2391 2021-03-24T22:02:08.177 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.177 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:02:08.178 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.178 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:02:08.178 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.174190+0000", 2021-03-24T22:02:08.178 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:02:08.179 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.179 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.179 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:02:08.179 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.174190+0000", 2021-03-24T22:02:08.179 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T22:02:08.180 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.180 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.180 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:02:08.180 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.174189+0000", 2021-03-24T22:02:08.181 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000389556 2021-03-24T22:02:08.181 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.181 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.181 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:02:08.181 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.174578+0000", 2021-03-24T22:02:08.182 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4030000000000004e-06 2021-03-24T22:02:08.182 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.182 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.182 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:02:08.183 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.174585+0000", 2021-03-24T22:02:08.183 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1669999999999998e-06 2021-03-24T22:02:08.184 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.184 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.184 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:02:08.184 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.174590+0000", 2021-03-24T22:02:08.184 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1109000000000001e-05 2021-03-24T22:02:08.185 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.185 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.185 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.185 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.174641+0000", 2021-03-24T22:02:08.185 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0509000000000001e-05 2021-03-24T22:02:08.186 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.186 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.186 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:02:08.186 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.174651+0000", 2021-03-24T22:02:08.187 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0054966999999999 2021-03-24T22:02:08.187 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.187 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.187 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.187 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.180148+0000", 2021-03-24T22:02:08.188 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00011527 2021-03-24T22:02:08.188 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.188 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.188 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:02:08.189 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.180263+0000", 2021-03-24T22:02:08.189 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000228352 2021-03-24T22:02:08.189 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.189 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.190 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,4", 2021-03-24T22:02:08.190 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.180492+0000", 2021-03-24T22:02:08.190 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.004233115 2021-03-24T22:02:08.190 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.191 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.191 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:02:08.191 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.184725+0000", 2021-03-24T22:02:08.191 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.006255404 2021-03-24T22:02:08.192 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.192 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.192 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:02:08.192 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.190980+0000", 2021-03-24T22:02:08.193 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0016284769999999999 2021-03-24T22:02:08.193 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.193 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.193 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:02:08.194 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.192609+0000", 2021-03-24T22:02:08.194 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5276999999999999e-05 2021-03-24T22:02:08.194 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.194 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.195 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:02:08.195 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.192664+0000", 2021-03-24T22:02:08.195 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.8547000000000001e-05 2021-03-24T22:02:08.195 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.196 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.196 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:02:08.196 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.192712+0000", 2021-03-24T22:02:08.196 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.018522293 2021-03-24T22:02:08.197 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:02:08.197 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:02:08.197 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:02:08.197 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.197 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.198 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:2393 7.23 7:c49de0b4:::smithi12556470-13:head [setxattr _header (57) in=7b,truncate 3630345] snapc a9=[a9,a8,a7,a0] ondisk+write+known_if_redirected e275)", 2021-03-24T22:02:08.198 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:57:25.176325+0000", 2021-03-24T22:02:08.198 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 282.96585052, 2021-03-24T22:02:08.198 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.024884068, 2021-03-24T22:02:08.199 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:02:08.199 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:02:08.199 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:02:08.199 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:02:08.199 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:02:08.200 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 2393 2021-03-24T22:02:08.200 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.200 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:02:08.200 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.201 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:02:08.201 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176325+0000", 2021-03-24T22:02:08.201 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:02:08.201 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.202 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.202 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:02:08.202 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176325+0000", 2021-03-24T22:02:08.202 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T22:02:08.203 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.203 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.203 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:02:08.203 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176324+0000", 2021-03-24T22:02:08.204 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1580000000000004e-06 2021-03-24T22:02:08.204 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.204 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.204 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:02:08.205 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176329+0000", 2021-03-24T22:02:08.205 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6590000000000001e-06 2021-03-24T22:02:08.205 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.205 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.206 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:02:08.206 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176336+0000", 2021-03-24T22:02:08.206 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1610000000000002e-06 2021-03-24T22:02:08.206 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.206 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.207 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:02:08.207 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176341+0000", 2021-03-24T22:02:08.207 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.3918000000000005e-05 2021-03-24T22:02:08.207 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.208 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.208 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.208 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176425+0000", 2021-03-24T22:02:08.208 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0132000000000001e-05 2021-03-24T22:02:08.209 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.209 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.209 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:02:08.209 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176435+0000", 2021-03-24T22:02:08.209 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0063475879999999 2021-03-24T22:02:08.210 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.210 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.210 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.210 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.182782+0000", 2021-03-24T22:02:08.211 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00028664400000000002 2021-03-24T22:02:08.211 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.211 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.211 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:02:08.212 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.183069+0000", 2021-03-24T22:02:08.212 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00021797799999999999 2021-03-24T22:02:08.212 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.212 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.212 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,4", 2021-03-24T22:02:08.213 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.183287+0000", 2021-03-24T22:02:08.213 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0014522280000000001 2021-03-24T22:02:08.213 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.213 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.214 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:02:08.214 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.184739+0000", 2021-03-24T22:02:08.214 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0095337649999999996 2021-03-24T22:02:08.215 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.215 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.215 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:02:08.215 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.194273+0000", 2021-03-24T22:02:08.216 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0067855449999999996 2021-03-24T22:02:08.216 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.216 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.216 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:02:08.217 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.201059+0000", 2021-03-24T22:02:08.217 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.5068999999999996e-05 2021-03-24T22:02:08.217 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.217 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.217 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:02:08.218 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.201134+0000", 2021-03-24T22:02:08.218 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.5298999999999999e-05 2021-03-24T22:02:08.218 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.218 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.219 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:02:08.219 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.201209+0000", 2021-03-24T22:02:08.219 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.024884068 2021-03-24T22:02:08.219 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:02:08.219 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:02:08.220 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:02:08.220 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.221 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.222 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:2394 7.23 7:c49de0b4:::smithi12556470-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e275)", 2021-03-24T22:02:08.222 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:57:25.176373+0000", 2021-03-24T22:02:08.222 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 282.96580276200001, 2021-03-24T22:02:08.222 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.025101692, 2021-03-24T22:02:08.223 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:02:08.223 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T22:02:08.223 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:02:08.223 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:02:08.224 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:02:08.224 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 2394 2021-03-24T22:02:08.224 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.225 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:02:08.225 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.225 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:02:08.225 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176373+0000", 2021-03-24T22:02:08.226 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:02:08.226 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.226 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.226 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:02:08.226 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176373+0000", 2021-03-24T22:02:08.227 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T22:02:08.227 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.227 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.227 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:02:08.228 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176371+0000", 2021-03-24T22:02:08.228 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.08e-06 2021-03-24T22:02:08.228 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.228 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.228 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:02:08.229 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176378+0000", 2021-03-24T22:02:08.229 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1309e-05 2021-03-24T22:02:08.229 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.229 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.230 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:02:08.230 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176389+0000", 2021-03-24T22:02:08.230 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.8714e-05 2021-03-24T22:02:08.230 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.231 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.231 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:02:08.231 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176418+0000", 2021-03-24T22:02:08.231 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6766000000000004e-05 2021-03-24T22:02:08.231 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.232 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.232 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.232 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176485+0000", 2021-03-24T22:02:08.233 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.1589999999999999e-06 2021-03-24T22:02:08.233 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.234 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.234 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:02:08.234 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176494+0000", 2021-03-24T22:02:08.235 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0071017390000001 2021-03-24T22:02:08.235 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.235 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.236 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.236 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.183596+0000", 2021-03-24T22:02:08.236 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6001999999999996e-05 2021-03-24T22:02:08.236 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.237 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.237 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:02:08.237 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.183662+0000", 2021-03-24T22:02:08.237 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.017585719 2021-03-24T22:02:08.237 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.238 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.238 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.238 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.201247+0000", 2021-03-24T22:02:08.239 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000106111 2021-03-24T22:02:08.239 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.239 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.239 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:02:08.239 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.201354+0000", 2021-03-24T22:02:08.240 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000120763 2021-03-24T22:02:08.240 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.240 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.240 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:02:08.241 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.201474+0000", 2021-03-24T22:02:08.241 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.025101692 2021-03-24T22:02:08.241 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:02:08.241 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:02:08.241 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:02:08.242 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.242 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.242 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:3950 7.c 7:32974111:::smithi12556470-2:head [write 675780~495523,stat] snapc 136=[136,133,132,130,12d,12c,12b,129,128,127,126,120,11e,11b,11a,119,110,10f,10c,106,105,104,fa,f8,e7] ondisk+write+known_if_redirected e417)", 2021-03-24T22:02:08.242 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:47.297626+0000", 2021-03-24T22:02:08.243 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 140.84454927100001, 2021-03-24T22:02:08.243 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0613052380000001, 2021-03-24T22:02:08.243 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:02:08.243 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:02:08.244 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:02:08.244 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:02:08.244 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:02:08.244 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3950 2021-03-24T22:02:08.244 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.245 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:02:08.245 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.245 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:02:08.246 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.297626+0000", 2021-03-24T22:02:08.246 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:02:08.246 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.246 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.247 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:02:08.247 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.297626+0000", 2021-03-24T22:02:08.247 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999976 2021-03-24T22:02:08.248 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.248 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.248 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:02:08.248 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.297624+0000", 2021-03-24T22:02:08.249 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00032374599999999997 2021-03-24T22:02:08.249 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.249 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.249 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:02:08.249 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.297948+0000", 2021-03-24T22:02:08.250 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1829e-05 2021-03-24T22:02:08.250 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.250 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.250 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:02:08.251 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.297959+0000", 2021-03-24T22:02:08.251 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.9770000000000003e-06 2021-03-24T22:02:08.251 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.251 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.252 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:02:08.252 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.297966+0000", 2021-03-24T22:02:08.252 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001568081 2021-03-24T22:02:08.252 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.252 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.253 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.253 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299534+0000", 2021-03-24T22:02:08.253 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.7069999999999998e-06 2021-03-24T22:02:08.253 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.254 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.254 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:02:08.254 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299543+0000", 2021-03-24T22:02:08.254 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.02708646 2021-03-24T22:02:08.254 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.255 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.255 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.255 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.326630+0000", 2021-03-24T22:02:08.255 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000180722 2021-03-24T22:02:08.256 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.256 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.256 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:02:08.256 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.326810+0000", 2021-03-24T22:02:08.257 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00025251599999999999 2021-03-24T22:02:08.257 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.257 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.258 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,2", 2021-03-24T22:02:08.258 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.327063+0000", 2021-03-24T22:02:08.258 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0041170629999999998 2021-03-24T22:02:08.259 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.259 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.260 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:02:08.260 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.331180+0000", 2021-03-24T22:02:08.260 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.024822364999999999 2021-03-24T22:02:08.261 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.261 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.261 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:02:08.262 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.356002+0000", 2021-03-24T22:02:08.262 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000346561 2021-03-24T22:02:08.262 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.262 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.262 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:02:08.263 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.356349+0000", 2021-03-24T22:02:08.263 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00048905499999999996 2021-03-24T22:02:08.263 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.263 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.264 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:02:08.264 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.356838+0000", 2021-03-24T22:02:08.264 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0020934259999999998 2021-03-24T22:02:08.264 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.265 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.265 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:02:08.266 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.358931+0000", 2021-03-24T22:02:08.266 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0613052380000001 2021-03-24T22:02:08.267 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:02:08.267 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:02:08.267 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:02:08.268 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.268 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.268 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:3951 7.c 7:32974111:::smithi12556470-2:head [write 1682747~714511] snapc 136=[136,133,132,130,12d,12c,12b,129,128,127,126,120,11e,11b,11a,119,110,10f,10c,106,105,104,fa,f8,e7] ondisk+write+known_if_redirected e417)", 2021-03-24T22:02:08.268 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:47.298949+0000", 2021-03-24T22:02:08.268 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 140.84322633100001, 2021-03-24T22:02:08.269 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0693593290000001, 2021-03-24T22:02:08.269 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:02:08.269 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:02:08.269 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:02:08.270 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:02:08.270 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:02:08.270 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3951 2021-03-24T22:02:08.270 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.271 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:02:08.271 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.271 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:02:08.272 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.298949+0000", 2021-03-24T22:02:08.272 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:02:08.272 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.272 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.273 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:02:08.273 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.298949+0000", 2021-03-24T22:02:08.273 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T22:02:08.273 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.273 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.274 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:02:08.274 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.298948+0000", 2021-03-24T22:02:08.274 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000412256 2021-03-24T22:02:08.274 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.275 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.275 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:02:08.275 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299360+0000", 2021-03-24T22:02:08.275 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1399999999999999e-05 2021-03-24T22:02:08.276 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.276 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.276 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:02:08.276 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299371+0000", 2021-03-24T22:02:08.277 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.3679999999999996e-06 2021-03-24T22:02:08.277 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.277 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.278 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:02:08.278 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299379+0000", 2021-03-24T22:02:08.279 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00048704199999999998 2021-03-24T22:02:08.279 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.279 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.280 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.280 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299866+0000", 2021-03-24T22:02:08.280 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.277e-06 2021-03-24T22:02:08.280 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.281 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.281 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:02:08.281 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299873+0000", 2021-03-24T22:02:08.281 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0281777940000001 2021-03-24T22:02:08.281 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.282 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.282 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.283 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.328051+0000", 2021-03-24T22:02:08.283 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000151185 2021-03-24T22:02:08.283 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.284 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.284 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:02:08.285 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.328202+0000", 2021-03-24T22:02:08.285 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.030788554999999999 2021-03-24T22:02:08.286 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.286 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.287 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.287 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.358991+0000", 2021-03-24T22:02:08.287 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012337099999999999 2021-03-24T22:02:08.288 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.288 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.289 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:02:08.289 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.359114+0000", 2021-03-24T22:02:08.290 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00022504400000000001 2021-03-24T22:02:08.290 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.290 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.291 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,2", 2021-03-24T22:02:08.291 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.359339+0000", 2021-03-24T22:02:08.292 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0044580849999999997 2021-03-24T22:02:08.292 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.293 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.293 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:02:08.293 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.363797+0000", 2021-03-24T22:02:08.294 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0031527479999999999 2021-03-24T22:02:08.294 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.295 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.295 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:02:08.295 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.366950+0000", 2021-03-24T22:02:08.295 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001068685 2021-03-24T22:02:08.296 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.296 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.296 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:02:08.296 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.368018+0000", 2021-03-24T22:02:08.297 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.4264999999999995e-05 2021-03-24T22:02:08.297 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.297 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.297 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:02:08.298 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.368093+0000", 2021-03-24T22:02:08.298 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00021562 2021-03-24T22:02:08.298 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.298 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.299 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:02:08.299 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.368308+0000", 2021-03-24T22:02:08.299 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0693593290000001 2021-03-24T22:02:08.299 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:02:08.300 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:02:08.300 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:02:08.300 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.301 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.301 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:3952 7.c 7:32974111:::smithi12556470-2:head [write 3012769~63011] snapc 136=[136,133,132,130,12d,12c,12b,129,128,127,126,120,11e,11b,11a,119,110,10f,10c,106,105,104,fa,f8,e7] ondisk+write+known_if_redirected e417)", 2021-03-24T22:02:08.301 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:47.299385+0000", 2021-03-24T22:02:08.302 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 140.842789985, 2021-03-24T22:02:08.302 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.070508668, 2021-03-24T22:02:08.303 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:02:08.303 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:02:08.303 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:02:08.303 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:02:08.304 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:02:08.304 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3952 2021-03-24T22:02:08.304 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.304 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:02:08.305 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.305 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:02:08.305 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299385+0000", 2021-03-24T22:02:08.305 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:02:08.305 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.306 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.306 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:02:08.306 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299385+0000", 2021-03-24T22:02:08.306 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T22:02:08.306 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.307 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.307 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:02:08.307 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299384+0000", 2021-03-24T22:02:08.307 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.6712000000000001e-05 2021-03-24T22:02:08.308 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.308 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.308 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:02:08.309 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299411+0000", 2021-03-24T22:02:08.309 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6420000000000001e-06 2021-03-24T22:02:08.309 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.309 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.310 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:02:08.310 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299418+0000", 2021-03-24T22:02:08.310 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6809999999999997e-06 2021-03-24T22:02:08.310 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.311 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.311 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:02:08.311 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299422+0000", 2021-03-24T22:02:08.311 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000472101 2021-03-24T22:02:08.311 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.312 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.312 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.313 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299895+0000", 2021-03-24T22:02:08.313 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.0889999999999996e-06 2021-03-24T22:02:08.313 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.314 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.314 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:02:08.314 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299901+0000", 2021-03-24T22:02:08.315 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0283333750000001 2021-03-24T22:02:08.315 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.315 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.316 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.316 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.328234+0000", 2021-03-24T22:02:08.316 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000104591 2021-03-24T22:02:08.316 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.317 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.317 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:02:08.317 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.328339+0000", 2021-03-24T22:02:08.317 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.033718655 2021-03-24T22:02:08.318 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.318 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.318 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.318 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.362057+0000", 2021-03-24T22:02:08.318 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00014753499999999999 2021-03-24T22:02:08.319 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.319 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.320 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:02:08.320 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.362205+0000", 2021-03-24T22:02:08.320 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000219259 2021-03-24T22:02:08.320 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.320 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.321 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,2", 2021-03-24T22:02:08.321 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.362424+0000", 2021-03-24T22:02:08.321 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001410356 2021-03-24T22:02:08.321 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.322 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.322 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:02:08.322 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.363834+0000", 2021-03-24T22:02:08.322 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.003124342 2021-03-24T22:02:08.323 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.323 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.323 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:02:08.323 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.366959+0000", 2021-03-24T22:02:08.324 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0014798490000000001 2021-03-24T22:02:08.324 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.324 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.324 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:02:08.324 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.368439+0000", 2021-03-24T22:02:08.325 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001231179 2021-03-24T22:02:08.325 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.325 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.325 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:02:08.326 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.369670+0000", 2021-03-24T22:02:08.326 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00022420900000000001 2021-03-24T22:02:08.327 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.327 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.328 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:02:08.328 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.369894+0000", 2021-03-24T22:02:08.328 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.070508668 2021-03-24T22:02:08.329 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:02:08.329 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:02:08.329 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:02:08.329 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.330 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.330 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:3953 7.c 7:32974111:::smithi12556470-2:head [setxattr _header (57) in=7b,truncate 3075780] snapc 136=[136,133,132,130,12d,12c,12b,129,128,127,126,120,11e,11b,11a,119,110,10f,10c,106,105,104,fa,f8,e7] ondisk+write+known_if_redirected e417)", 2021-03-24T22:02:08.331 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:47.299427+0000", 2021-03-24T22:02:08.331 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 140.84274821700001, 2021-03-24T22:02:08.331 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0708905799999999, 2021-03-24T22:02:08.331 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:02:08.331 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:02:08.332 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:02:08.332 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:02:08.332 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:02:08.333 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3953 2021-03-24T22:02:08.333 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.333 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:02:08.333 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.333 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:02:08.334 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299427+0000", 2021-03-24T22:02:08.334 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:02:08.334 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.335 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.335 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:02:08.335 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299427+0000", 2021-03-24T22:02:08.335 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999995 2021-03-24T22:02:08.335 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.336 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.336 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:02:08.336 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299427+0000", 2021-03-24T22:02:08.336 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.4259999999999999e-06 2021-03-24T22:02:08.337 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.337 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.337 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:02:08.337 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299430+0000", 2021-03-24T22:02:08.337 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1189999999999996e-06 2021-03-24T22:02:08.338 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.338 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.338 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:02:08.339 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299435+0000", 2021-03-24T22:02:08.339 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9140000000000001e-06 2021-03-24T22:02:08.340 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.341 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.341 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:02:08.342 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299439+0000", 2021-03-24T22:02:08.342 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00048008799999999999 2021-03-24T22:02:08.343 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.343 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.343 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.344 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299919+0000", 2021-03-24T22:02:08.344 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.835e-06 2021-03-24T22:02:08.345 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.345 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.346 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:02:08.346 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299925+0000", 2021-03-24T22:02:08.347 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.028441138 2021-03-24T22:02:08.347 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.347 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.348 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.348 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.328366+0000", 2021-03-24T22:02:08.349 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010274500000000001 2021-03-24T22:02:08.349 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.350 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.350 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:02:08.350 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.328469+0000", 2021-03-24T22:02:08.351 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.034522276999999997 2021-03-24T22:02:08.352 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.352 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.353 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.353 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.362991+0000", 2021-03-24T22:02:08.353 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000114469 2021-03-24T22:02:08.354 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.354 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.354 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:02:08.355 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.363106+0000", 2021-03-24T22:02:08.355 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00021205599999999999 2021-03-24T22:02:08.356 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.356 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.356 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,2", 2021-03-24T22:02:08.357 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.363318+0000", 2021-03-24T22:02:08.357 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00096840900000000004 2021-03-24T22:02:08.357 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.358 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.358 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:02:08.359 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.364286+0000", 2021-03-24T22:02:08.359 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002681143 2021-03-24T22:02:08.359 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.360 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.360 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:02:08.361 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.366967+0000", 2021-03-24T22:02:08.362 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002998559 2021-03-24T22:02:08.362 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.363 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.363 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:02:08.364 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.369966+0000", 2021-03-24T22:02:08.364 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012207299999999999 2021-03-24T22:02:08.364 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.365 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.365 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:02:08.365 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.370088+0000", 2021-03-24T22:02:08.365 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00022991299999999999 2021-03-24T22:02:08.365 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.366 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.366 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:02:08.366 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.370318+0000", 2021-03-24T22:02:08.366 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0708905799999999 2021-03-24T22:02:08.367 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:02:08.367 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:02:08.368 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:02:08.368 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.368 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.368 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:3954 7.c 7:32974111:::smithi12556470-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e417)", 2021-03-24T22:02:08.369 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:47.299443+0000", 2021-03-24T22:02:08.369 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 140.84273253399999, 2021-03-24T22:02:08.369 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.071832683, 2021-03-24T22:02:08.369 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:02:08.369 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T22:02:08.370 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:02:08.370 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:02:08.370 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:02:08.371 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3954 2021-03-24T22:02:08.371 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.371 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:02:08.372 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.372 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:02:08.372 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299443+0000", 2021-03-24T22:02:08.373 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:02:08.373 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.373 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.374 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:02:08.374 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299443+0000", 2021-03-24T22:02:08.374 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999995 2021-03-24T22:02:08.375 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.375 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.375 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:02:08.375 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299442+0000", 2021-03-24T22:02:08.375 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.2239999999999998e-06 2021-03-24T22:02:08.376 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.376 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.376 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:02:08.376 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299445+0000", 2021-03-24T22:02:08.377 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.3809999999999998e-06 2021-03-24T22:02:08.377 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.377 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.377 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:02:08.377 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299450+0000", 2021-03-24T22:02:08.378 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9160000000000003e-06 2021-03-24T22:02:08.378 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.378 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.378 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:02:08.379 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299454+0000", 2021-03-24T22:02:08.379 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00049169700000000003 2021-03-24T22:02:08.379 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.379 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.379 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.380 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299946+0000", 2021-03-24T22:02:08.380 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.003e-06 2021-03-24T22:02:08.380 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.380 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.380 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:02:08.381 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299952+0000", 2021-03-24T22:02:08.381 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0285410699999999 2021-03-24T22:02:08.381 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.381 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.382 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.382 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.328493+0000", 2021-03-24T22:02:08.382 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0015757410000000001 2021-03-24T22:02:08.382 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.383 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.383 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:02:08.383 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.330068+0000", 2021-03-24T22:02:08.383 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.033581221000000001 2021-03-24T22:02:08.384 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.384 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.384 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.384 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.363650+0000", 2021-03-24T22:02:08.385 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010419700000000001 2021-03-24T22:02:08.385 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.385 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.385 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:02:08.386 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.363754+0000", 2021-03-24T22:02:08.386 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0070060599999999997 2021-03-24T22:02:08.386 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.387 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.387 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.387 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.370760+0000", 2021-03-24T22:02:08.388 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00020776099999999999 2021-03-24T22:02:08.388 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.389 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.389 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:02:08.389 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.370968+0000", 2021-03-24T22:02:08.390 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00030779100000000002 2021-03-24T22:02:08.390 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.391 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.391 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:02:08.391 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.371275+0000", 2021-03-24T22:02:08.392 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.071832683 2021-03-24T22:02:08.392 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:02:08.393 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:02:08.393 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:02:08.394 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.394 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.394 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:4147 7.21 7:86662589:::smithi12556470-19:head [write 712762~401056,stat] snapc 143=[143,140,13d,13b,139,137,136,12d,12b,129,128,127,126,120,11e,11b,11a,119,110,10f,10c,105,104,fa,f8,e7] ondisk+write+known_if_redirected e432)", 2021-03-24T22:02:08.395 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:00:02.248651+0000", 2021-03-24T22:02:08.395 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 125.893524222, 2021-03-24T22:02:08.396 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0717329640000002, 2021-03-24T22:02:08.396 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:02:08.396 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:02:08.397 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:02:08.397 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:02:08.397 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:02:08.398 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4147 2021-03-24T22:02:08.398 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.399 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:02:08.399 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.400 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:02:08.400 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.248651+0000", 2021-03-24T22:02:08.401 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:02:08.401 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.402 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.402 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:02:08.403 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.248651+0000", 2021-03-24T22:02:08.403 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999967 2021-03-24T22:02:08.403 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.404 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.404 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:02:08.404 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.248648+0000", 2021-03-24T22:02:08.405 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000104109 2021-03-24T22:02:08.405 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.405 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.405 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:02:08.405 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.248752+0000", 2021-03-24T22:02:08.406 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002002804 2021-03-24T22:02:08.406 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.406 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.406 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:02:08.407 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.250755+0000", 2021-03-24T22:02:08.407 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5437000000000001e-05 2021-03-24T22:02:08.407 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.407 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.407 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:02:08.408 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.250770+0000", 2021-03-24T22:02:08.408 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0014452029999999999 2021-03-24T22:02:08.408 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.408 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.409 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.409 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.252215+0000", 2021-03-24T22:02:08.409 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.8291000000000002e-05 2021-03-24T22:02:08.409 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.409 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.410 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:02:08.410 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.252234+0000", 2021-03-24T22:02:08.410 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99172324700000003 2021-03-24T22:02:08.410 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.411 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.411 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.411 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:03.243957+0000", 2021-03-24T22:02:08.412 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.9359999999999994e-06 2021-03-24T22:02:08.412 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.412 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.412 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:02:08.413 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:03.243967+0000", 2021-03-24T22:02:08.413 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0558876720000001 2021-03-24T22:02:08.413 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.414 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.414 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.414 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.299854+0000", 2021-03-24T22:02:08.415 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00045708100000000002 2021-03-24T22:02:08.415 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.415 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.415 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for degraded object", 2021-03-24T22:02:08.416 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.300311+0000", 2021-03-24T22:02:08.416 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0088695200000000005 2021-03-24T22:02:08.416 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.416 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.416 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.417 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.309181+0000", 2021-03-24T22:02:08.417 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000101866 2021-03-24T22:02:08.417 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.417 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.418 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:02:08.418 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.309283+0000", 2021-03-24T22:02:08.418 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00025108500000000001 2021-03-24T22:02:08.418 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.418 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.419 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,5", 2021-03-24T22:02:08.419 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.309534+0000", 2021-03-24T22:02:08.419 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001499727 2021-03-24T22:02:08.419 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.420 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.420 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:02:08.420 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.311034+0000", 2021-03-24T22:02:08.420 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001462262 2021-03-24T22:02:08.420 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.421 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.421 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:02:08.421 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.312496+0000", 2021-03-24T22:02:08.421 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0076004449999999999 2021-03-24T22:02:08.422 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.422 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.422 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:02:08.422 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.320096+0000", 2021-03-24T22:02:08.422 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.8819999999999993e-05 2021-03-24T22:02:08.423 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.423 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.423 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:02:08.423 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.320185+0000", 2021-03-24T22:02:08.424 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00019890500000000001 2021-03-24T22:02:08.424 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.424 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.424 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:02:08.425 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.320384+0000", 2021-03-24T22:02:08.425 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0717329640000002 2021-03-24T22:02:08.425 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:02:08.425 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:02:08.425 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:02:08.426 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.426 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.426 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:4148 7.21 7:86662589:::smithi12556470-19:head [write 1537867~645855] snapc 143=[143,140,13d,13b,139,137,136,12d,12b,129,128,127,126,120,11e,11b,11a,119,110,10f,10c,105,104,fa,f8,e7] ondisk+write+known_if_redirected e432)", 2021-03-24T22:02:08.426 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:00:02.257275+0000", 2021-03-24T22:02:08.427 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 125.884899897, 2021-03-24T22:02:08.427 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.067389844, 2021-03-24T22:02:08.427 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:02:08.427 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:02:08.427 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:02:08.428 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:02:08.428 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:02:08.428 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4148 2021-03-24T22:02:08.428 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.428 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:02:08.429 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.429 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:02:08.429 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.257275+0000", 2021-03-24T22:02:08.429 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:02:08.430 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.430 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.430 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:02:08.430 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.257275+0000", 2021-03-24T22:02:08.431 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999962 2021-03-24T22:02:08.431 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.431 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.431 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:02:08.431 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.257271+0000", 2021-03-24T22:02:08.432 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00017391499999999999 2021-03-24T22:02:08.432 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.432 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.433 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:02:08.433 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.257445+0000", 2021-03-24T22:02:08.433 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0049961399999999996 2021-03-24T22:02:08.433 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.434 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.434 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:02:08.434 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.262441+0000", 2021-03-24T22:02:08.434 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2265e-05 2021-03-24T22:02:08.435 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.435 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.435 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:02:08.435 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.262454+0000", 2021-03-24T22:02:08.435 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.013052934 2021-03-24T22:02:08.436 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.436 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.436 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.436 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.275507+0000", 2021-03-24T22:02:08.437 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.5969999999999998e-06 2021-03-24T22:02:08.437 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.437 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.437 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:02:08.438 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.275516+0000", 2021-03-24T22:02:08.438 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.96847310900000005 2021-03-24T22:02:08.438 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.438 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.438 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.439 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:03.243989+0000", 2021-03-24T22:02:08.439 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.626e-06 2021-03-24T22:02:08.439 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.439 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.440 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:02:08.440 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:03.243995+0000", 2021-03-24T22:02:08.440 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0563536250000001 2021-03-24T22:02:08.440 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.440 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.441 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.441 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.300349+0000", 2021-03-24T22:02:08.441 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.6805000000000007e-05 2021-03-24T22:02:08.441 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.442 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.442 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for degraded object", 2021-03-24T22:02:08.442 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.300435+0000", 2021-03-24T22:02:08.442 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.009989965 2021-03-24T22:02:08.442 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.443 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.443 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.443 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.310425+0000", 2021-03-24T22:02:08.443 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000105184 2021-03-24T22:02:08.444 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.444 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.444 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:02:08.444 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.310531+0000", 2021-03-24T22:02:08.444 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0099567490000000008 2021-03-24T22:02:08.445 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.445 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.445 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.446 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.320487+0000", 2021-03-24T22:02:08.446 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00014939599999999999 2021-03-24T22:02:08.447 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.447 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.447 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:02:08.448 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.320637+0000", 2021-03-24T22:02:08.448 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00031880400000000001 2021-03-24T22:02:08.448 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.448 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.449 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,5", 2021-03-24T22:02:08.449 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.320956+0000", 2021-03-24T22:02:08.449 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.003448596 2021-03-24T22:02:08.450 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.450 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.450 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:02:08.450 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.324404+0000", 2021-03-24T22:02:08.451 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.4194e-05 2021-03-24T22:02:08.451 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.451 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.451 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:02:08.451 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.324418+0000", 2021-03-24T22:02:08.452 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.0278000000000002e-05 2021-03-24T22:02:08.452 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.452 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.452 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:02:08.453 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.324469+0000", 2021-03-24T22:02:08.453 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.4826999999999994e-05 2021-03-24T22:02:08.453 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.453 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.453 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:02:08.454 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.324543+0000", 2021-03-24T22:02:08.454 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000121864 2021-03-24T22:02:08.454 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.454 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.455 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:02:08.455 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.324665+0000", 2021-03-24T22:02:08.455 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.067389844 2021-03-24T22:02:08.455 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:02:08.456 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:02:08.456 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:02:08.457 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.457 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.458 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:4149 7.21 7:86662589:::smithi12556470-19:head [write 2860355~652407] snapc 143=[143,140,13d,13b,139,137,136,12d,12b,129,128,127,126,120,11e,11b,11a,119,110,10f,10c,105,104,fa,f8,e7] ondisk+write+known_if_redirected e432)", 2021-03-24T22:02:08.458 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:00:02.264982+0000", 2021-03-24T22:02:08.459 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 125.877193217, 2021-03-24T22:02:08.459 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.060287465, 2021-03-24T22:02:08.459 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:02:08.460 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:02:08.460 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:02:08.461 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:02:08.461 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:02:08.462 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4149 2021-03-24T22:02:08.462 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.463 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:02:08.463 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.464 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:02:08.464 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.264982+0000", 2021-03-24T22:02:08.464 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:02:08.465 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.465 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.466 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:02:08.466 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.264982+0000", 2021-03-24T22:02:08.466 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999881 2021-03-24T22:02:08.467 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.467 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.468 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:02:08.468 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.264970+0000", 2021-03-24T22:02:08.469 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0010432900000000001 2021-03-24T22:02:08.469 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.469 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.470 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:02:08.470 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.266013+0000", 2021-03-24T22:02:08.471 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0014047409999999999 2021-03-24T22:02:08.471 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.471 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.472 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:02:08.472 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.267418+0000", 2021-03-24T22:02:08.472 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2789000000000001e-05 2021-03-24T22:02:08.473 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.473 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.474 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:02:08.474 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.267431+0000", 2021-03-24T22:02:08.474 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0081054939999999995 2021-03-24T22:02:08.475 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.475 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.475 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.476 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.275536+0000", 2021-03-24T22:02:08.476 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.7870000000000004e-06 2021-03-24T22:02:08.477 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.477 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.477 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:02:08.478 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.275543+0000", 2021-03-24T22:02:08.478 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.96846978399999994 2021-03-24T22:02:08.479 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.479 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.480 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.480 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:03.244013+0000", 2021-03-24T22:02:08.480 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6439999999999997e-06 2021-03-24T22:02:08.481 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.481 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.482 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:02:08.482 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:03.244019+0000", 2021-03-24T22:02:08.483 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.056445708 2021-03-24T22:02:08.483 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.483 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.484 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.484 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.300464+0000", 2021-03-24T22:02:08.484 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001583262 2021-03-24T22:02:08.484 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.485 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.485 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for degraded object", 2021-03-24T22:02:08.486 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.302048+0000", 2021-03-24T22:02:08.486 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0085309080000000002 2021-03-24T22:02:08.487 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.487 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.487 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.488 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.310579+0000", 2021-03-24T22:02:08.488 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.8817e-05 2021-03-24T22:02:08.489 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.489 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.490 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:02:08.490 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.310667+0000", 2021-03-24T22:02:08.490 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011484482000000001 2021-03-24T22:02:08.491 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.491 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.492 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.492 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.322152+0000", 2021-03-24T22:02:08.492 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.6207999999999999e-05 2021-03-24T22:02:08.492 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.493 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.493 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:02:08.493 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.322228+0000", 2021-03-24T22:02:08.493 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00014250999999999999 2021-03-24T22:02:08.494 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.494 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.494 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,5", 2021-03-24T22:02:08.494 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.322371+0000", 2021-03-24T22:02:08.494 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0020567459999999999 2021-03-24T22:02:08.495 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.495 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.495 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:02:08.495 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.324427+0000", 2021-03-24T22:02:08.496 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00031726900000000001 2021-03-24T22:02:08.496 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.496 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.496 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:02:08.497 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.324745+0000", 2021-03-24T22:02:08.497 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00031702800000000001 2021-03-24T22:02:08.497 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.498 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.498 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:02:08.498 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.325062+0000", 2021-03-24T22:02:08.499 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.4227999999999997e-05 2021-03-24T22:02:08.499 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.499 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.499 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:02:08.500 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.325106+0000", 2021-03-24T22:02:08.500 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00016376499999999999 2021-03-24T22:02:08.500 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.500 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.501 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:02:08.501 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.325270+0000", 2021-03-24T22:02:08.501 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.060287465 2021-03-24T22:02:08.501 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:02:08.501 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:02:08.502 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:02:08.502 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.502 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.503 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:4150 7.21 7:86662589:::smithi12556470-19:head [setxattr _header (58) in=7b,truncate 3512762] snapc 143=[143,140,13d,13b,139,137,136,12d,12b,129,128,127,126,120,11e,11b,11a,119,110,10f,10c,105,104,fa,f8,e7] ondisk+write+known_if_redirected e432)", 2021-03-24T22:02:08.503 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:00:02.267443+0000", 2021-03-24T22:02:08.504 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 125.874732843, 2021-03-24T22:02:08.504 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.058203818, 2021-03-24T22:02:08.505 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:02:08.505 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:02:08.506 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:02:08.506 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:02:08.507 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:02:08.507 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4150 2021-03-24T22:02:08.507 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.508 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:02:08.508 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.509 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:02:08.509 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.267443+0000", 2021-03-24T22:02:08.510 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:02:08.510 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.511 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.511 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:02:08.511 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.267443+0000", 2021-03-24T22:02:08.512 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T22:02:08.512 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.513 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.513 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:02:08.513 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.267441+0000", 2021-03-24T22:02:08.514 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.1050000000000001e-06 2021-03-24T22:02:08.514 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.515 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.515 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:02:08.516 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.267450+0000", 2021-03-24T22:02:08.516 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.1170000000000004e-06 2021-03-24T22:02:08.516 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.517 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.517 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:02:08.518 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.267458+0000", 2021-03-24T22:02:08.518 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6759999999999998e-06 2021-03-24T22:02:08.518 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.519 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.519 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:02:08.519 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.267463+0000", 2021-03-24T22:02:08.520 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0081025090000000008 2021-03-24T22:02:08.520 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.520 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.521 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.521 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.275565+0000", 2021-03-24T22:02:08.522 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4189999999999999e-06 2021-03-24T22:02:08.522 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.522 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.523 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:02:08.523 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.275572+0000", 2021-03-24T22:02:08.524 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.96846758399999999 2021-03-24T22:02:08.524 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.524 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.525 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.525 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:03.244039+0000", 2021-03-24T22:02:08.526 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5249999999999996e-06 2021-03-24T22:02:08.526 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.527 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.527 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:02:08.528 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:03.244045+0000", 2021-03-24T22:02:08.528 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.058038864 2021-03-24T22:02:08.528 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.529 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.529 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.529 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.302084+0000", 2021-03-24T22:02:08.529 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.0760000000000005e-05 2021-03-24T22:02:08.530 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.530 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.530 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for degraded object", 2021-03-24T22:02:08.531 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.302174+0000", 2021-03-24T22:02:08.531 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0085442020000000007 2021-03-24T22:02:08.532 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.532 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.533 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.533 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.310718+0000", 2021-03-24T22:02:08.533 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.1786999999999994e-05 2021-03-24T22:02:08.534 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.534 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.535 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:02:08.535 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.310810+0000", 2021-03-24T22:02:08.535 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012559539 2021-03-24T22:02:08.536 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.536 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.536 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.536 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.323370+0000", 2021-03-24T22:02:08.537 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000127611 2021-03-24T22:02:08.537 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.537 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.537 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:02:08.538 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.323497+0000", 2021-03-24T22:02:08.538 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000301575 2021-03-24T22:02:08.538 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.538 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.539 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,5", 2021-03-24T22:02:08.539 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.323799+0000", 2021-03-24T22:02:08.539 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00088734600000000001 2021-03-24T22:02:08.539 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.540 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.540 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:02:08.540 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.324686+0000", 2021-03-24T22:02:08.540 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00066636400000000002 2021-03-24T22:02:08.540 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.541 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.541 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:02:08.541 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.325353+0000", 2021-03-24T22:02:08.541 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.1169999999999997e-05 2021-03-24T22:02:08.542 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.542 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.542 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:02:08.542 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.325434+0000", 2021-03-24T22:02:08.543 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.8564000000000002e-05 2021-03-24T22:02:08.543 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.543 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.543 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:02:08.543 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.325512+0000", 2021-03-24T22:02:08.544 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00013387499999999999 2021-03-24T22:02:08.544 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.544 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.545 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:02:08.545 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.325646+0000", 2021-03-24T22:02:08.545 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.058203818 2021-03-24T22:02:08.545 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:02:08.546 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:02:08.546 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:02:08.547 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.547 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.547 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:4151 7.21 7:86662589:::smithi12556470-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e432)", 2021-03-24T22:02:08.547 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:00:02.267468+0000", 2021-03-24T22:02:08.548 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 125.874707759, 2021-03-24T22:02:08.548 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0584516220000002, 2021-03-24T22:02:08.548 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:02:08.548 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T22:02:08.548 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:02:08.549 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:02:08.549 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:02:08.549 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4151 2021-03-24T22:02:08.550 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.550 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:02:08.550 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.550 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:02:08.550 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.267468+0000", 2021-03-24T22:02:08.551 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:02:08.551 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.551 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.551 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:02:08.552 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.267468+0000", 2021-03-24T22:02:08.552 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999995 2021-03-24T22:02:08.552 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.552 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.552 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:02:08.553 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.267467+0000", 2021-03-24T22:02:08.553 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.3579999999999999e-06 2021-03-24T22:02:08.553 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.553 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.554 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:02:08.554 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.267470+0000", 2021-03-24T22:02:08.554 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.3289999999999999e-06 2021-03-24T22:02:08.555 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.555 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.555 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:02:08.556 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.267477+0000", 2021-03-24T22:02:08.556 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3869999999999997e-06 2021-03-24T22:02:08.556 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.556 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.557 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:02:08.557 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.267481+0000", 2021-03-24T22:02:08.558 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0081161340000000005 2021-03-24T22:02:08.558 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.558 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.559 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.559 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.275597+0000", 2021-03-24T22:02:08.559 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.925e-06 2021-03-24T22:02:08.560 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.560 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.560 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:02:08.560 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.275603+0000", 2021-03-24T22:02:08.561 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.96846318600000003 2021-03-24T22:02:08.561 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.561 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.561 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.562 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:03.244066+0000", 2021-03-24T22:02:08.562 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.4770000000000002e-06 2021-03-24T22:02:08.562 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.562 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.562 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:02:08.563 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:03.244072+0000", 2021-03-24T22:02:08.563 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.058135101 2021-03-24T22:02:08.563 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.563 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.564 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.564 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.302207+0000", 2021-03-24T22:02:08.564 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.2978000000000005e-05 2021-03-24T22:02:08.564 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.564 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.565 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for degraded object", 2021-03-24T22:02:08.565 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.302280+0000", 2021-03-24T22:02:08.565 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0085603620000000002 2021-03-24T22:02:08.565 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.566 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.566 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.566 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.310840+0000", 2021-03-24T22:02:08.566 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.5315999999999994e-05 2021-03-24T22:02:08.567 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.567 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.567 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:02:08.567 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.310926+0000", 2021-03-24T22:02:08.568 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.013304694000000001 2021-03-24T22:02:08.568 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.568 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.569 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.569 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.324230+0000", 2021-03-24T22:02:08.569 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00011130599999999999 2021-03-24T22:02:08.569 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.570 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.570 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:02:08.571 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.324342+0000", 2021-03-24T22:02:08.571 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0013441519999999999 2021-03-24T22:02:08.571 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.572 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.572 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.572 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.325686+0000", 2021-03-24T22:02:08.572 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5726999999999997e-05 2021-03-24T22:02:08.573 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.573 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.573 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:02:08.573 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.325741+0000", 2021-03-24T22:02:08.574 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00017782400000000001 2021-03-24T22:02:08.574 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.574 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.574 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:02:08.574 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.325919+0000", 2021-03-24T22:02:08.575 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0584516220000002 2021-03-24T22:02:08.575 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:02:08.575 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:02:08.575 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:02:08.576 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.576 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.576 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:4763 7.c 7:31cb714a:::smithi12556470-7:head [write 625392~569702] snapc 17d=[17d,17c,179,178,177,176,172,16e,16d,168,166,165,160,15e,159,157,153,139,136,129,127,11b,104,fa] ondisk+write+known_if_redirected e491)", 2021-03-24T22:02:08.576 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:01:01.275026+0000", 2021-03-24T22:02:08.576 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 66.867149592000004, 2021-03-24T22:02:08.577 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.06165336, 2021-03-24T22:02:08.577 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:02:08.577 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:02:08.577 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:02:08.578 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:02:08.578 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:02:08.578 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4763 2021-03-24T22:02:08.578 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.579 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:02:08.579 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.579 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:02:08.579 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.275026+0000", 2021-03-24T22:02:08.579 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:02:08.580 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.580 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.580 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:02:08.580 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.275026+0000", 2021-03-24T22:02:08.581 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999976 2021-03-24T22:02:08.581 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.581 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.581 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:02:08.582 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.275023+0000", 2021-03-24T22:02:08.582 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00032953799999999999 2021-03-24T22:02:08.582 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.582 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.582 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:02:08.583 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.275353+0000", 2021-03-24T22:02:08.583 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.3589999999999999e-05 2021-03-24T22:02:08.583 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.583 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.584 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:02:08.584 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.275377+0000", 2021-03-24T22:02:08.584 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.4150000000000005e-06 2021-03-24T22:02:08.584 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.584 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.585 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:02:08.585 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.275386+0000", 2021-03-24T22:02:08.585 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012973314999999999 2021-03-24T22:02:08.585 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.586 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.586 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.586 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.288359+0000", 2021-03-24T22:02:08.586 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.8028000000000006e-05 2021-03-24T22:02:08.587 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.587 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.587 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:02:08.587 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.288437+0000", 2021-03-24T22:02:08.587 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.039910114 2021-03-24T22:02:08.588 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.588 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.588 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.588 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.328347+0000", 2021-03-24T22:02:08.588 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00022543800000000001 2021-03-24T22:02:08.589 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.589 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.589 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:02:08.590 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.328573+0000", 2021-03-24T22:02:08.590 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00039166100000000002 2021-03-24T22:02:08.590 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.590 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.591 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,7", 2021-03-24T22:02:08.591 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.328965+0000", 2021-03-24T22:02:08.591 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0071784850000000001 2021-03-24T22:02:08.591 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.592 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.592 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:02:08.592 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.336143+0000", 2021-03-24T22:02:08.592 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00024411499999999999 2021-03-24T22:02:08.593 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.593 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.593 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:02:08.593 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.336387+0000", 2021-03-24T22:02:08.593 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.4493999999999995e-05 2021-03-24T22:02:08.594 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.594 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.594 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:02:08.594 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.336462+0000", 2021-03-24T22:02:08.595 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.0748999999999999e-05 2021-03-24T22:02:08.595 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.595 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.595 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:02:08.595 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.336502+0000", 2021-03-24T22:02:08.596 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00017668299999999999 2021-03-24T22:02:08.596 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.596 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.596 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:02:08.597 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.336679+0000", 2021-03-24T22:02:08.597 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.06165336 2021-03-24T22:02:08.597 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:02:08.597 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:02:08.598 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:02:08.598 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.598 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.599 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:4764 7.c 7:31cb714a:::smithi12556470-7:head [write 1637717~745548] snapc 17d=[17d,17c,179,178,177,176,172,16e,16d,168,166,165,160,15e,159,157,153,139,136,129,127,11b,104,fa] ondisk+write+known_if_redirected e491)", 2021-03-24T22:02:08.599 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:01:01.285992+0000", 2021-03-24T22:02:08.599 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 66.856183200999993, 2021-03-24T22:02:08.599 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0510351490000001, 2021-03-24T22:02:08.600 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:02:08.600 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:02:08.601 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:02:08.601 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:02:08.602 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:02:08.602 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4764 2021-03-24T22:02:08.602 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.603 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:02:08.603 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.604 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:02:08.604 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.285992+0000", 2021-03-24T22:02:08.605 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:02:08.605 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.605 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.606 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:02:08.606 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.285992+0000", 2021-03-24T22:02:08.607 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T22:02:08.607 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.608 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.608 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:02:08.609 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.285990+0000", 2021-03-24T22:02:08.609 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00046463299999999999 2021-03-24T22:02:08.610 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.610 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.610 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:02:08.611 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286455+0000", 2021-03-24T22:02:08.611 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.464e-05 2021-03-24T22:02:08.612 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.612 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.613 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:02:08.613 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286469+0000", 2021-03-24T22:02:08.613 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.6230000000000003e-06 2021-03-24T22:02:08.614 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.614 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.614 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:02:08.614 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286478+0000", 2021-03-24T22:02:08.614 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002016393 2021-03-24T22:02:08.615 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.615 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.615 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.615 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.288494+0000", 2021-03-24T22:02:08.616 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.5879000000000001e-05 2021-03-24T22:02:08.616 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.616 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.616 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:02:08.616 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.288520+0000", 2021-03-24T22:02:08.617 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.04205784 2021-03-24T22:02:08.617 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.617 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.617 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.618 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.330578+0000", 2021-03-24T22:02:08.618 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000176718 2021-03-24T22:02:08.618 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.618 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.618 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:02:08.619 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.330755+0000", 2021-03-24T22:02:08.619 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00047453099999999999 2021-03-24T22:02:08.619 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.619 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.619 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,7", 2021-03-24T22:02:08.620 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.331229+0000", 2021-03-24T22:02:08.620 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.005170661 2021-03-24T22:02:08.620 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.620 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.621 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:02:08.621 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.336400+0000", 2021-03-24T22:02:08.621 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00033368500000000001 2021-03-24T22:02:08.621 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.621 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.622 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:02:08.622 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.336734+0000", 2021-03-24T22:02:08.622 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.5241999999999996e-05 2021-03-24T22:02:08.622 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.623 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.623 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:02:08.623 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.336809+0000", 2021-03-24T22:02:08.623 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.8261999999999999e-05 2021-03-24T22:02:08.623 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.624 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.624 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:02:08.624 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.336867+0000", 2021-03-24T22:02:08.624 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000160037 2021-03-24T22:02:08.624 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.625 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.625 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:02:08.625 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.337027+0000", 2021-03-24T22:02:08.625 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0510351490000001 2021-03-24T22:02:08.626 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:02:08.626 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:02:08.626 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:02:08.626 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.626 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.627 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:4765 7.c 7:31cb714a:::smithi12556470-7:head [write 2848542~176850] snapc 17d=[17d,17c,179,178,177,176,172,16e,16d,168,166,165,160,15e,159,157,153,139,136,129,127,11b,104,fa] ondisk+write+known_if_redirected e491)", 2021-03-24T22:02:08.627 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:01:01.286489+0000", 2021-03-24T22:02:08.627 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 66.855686329999997, 2021-03-24T22:02:08.627 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0507179900000001, 2021-03-24T22:02:08.628 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:02:08.628 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:02:08.628 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:02:08.628 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:02:08.628 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:02:08.629 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4765 2021-03-24T22:02:08.629 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.629 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:02:08.629 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.629 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:02:08.630 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286489+0000", 2021-03-24T22:02:08.630 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:02:08.630 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.630 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.631 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:02:08.631 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286489+0000", 2021-03-24T22:02:08.631 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T22:02:08.631 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.631 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.632 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:02:08.632 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286488+0000", 2021-03-24T22:02:08.632 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000106881 2021-03-24T22:02:08.632 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.633 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.633 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:02:08.633 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286595+0000", 2021-03-24T22:02:08.633 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.9570000000000002e-06 2021-03-24T22:02:08.633 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.634 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.634 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:02:08.634 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286603+0000", 2021-03-24T22:02:08.634 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.7579999999999998e-06 2021-03-24T22:02:08.634 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.635 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.635 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:02:08.635 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286608+0000", 2021-03-24T22:02:08.635 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00353412 2021-03-24T22:02:08.636 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.636 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.636 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.636 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.290143+0000", 2021-03-24T22:02:08.636 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.6600000000000001e-06 2021-03-24T22:02:08.637 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.637 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.637 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:02:08.637 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.290151+0000", 2021-03-24T22:02:08.637 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.04322256 2021-03-24T22:02:08.638 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.638 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.638 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.638 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.333374+0000", 2021-03-24T22:02:08.639 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000160939 2021-03-24T22:02:08.639 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.639 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.639 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:02:08.639 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.333535+0000", 2021-03-24T22:02:08.640 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00051926299999999995 2021-03-24T22:02:08.640 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.640 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.640 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,7", 2021-03-24T22:02:08.641 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.334054+0000", 2021-03-24T22:02:08.641 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002355233 2021-03-24T22:02:08.641 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.641 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.641 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:02:08.642 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.336409+0000", 2021-03-24T22:02:08.642 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00036032499999999999 2021-03-24T22:02:08.642 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.642 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.643 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:02:08.643 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.336769+0000", 2021-03-24T22:02:08.643 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00030155200000000002 2021-03-24T22:02:08.643 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.643 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.644 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:02:08.644 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.337071+0000", 2021-03-24T22:02:08.644 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.5614000000000002e-05 2021-03-24T22:02:08.644 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.645 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.645 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:02:08.645 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.337107+0000", 2021-03-24T22:02:08.645 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000100364 2021-03-24T22:02:08.645 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.646 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.646 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:02:08.646 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.337207+0000", 2021-03-24T22:02:08.646 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0507179900000001 2021-03-24T22:02:08.646 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:02:08.647 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:02:08.647 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:02:08.647 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.647 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.648 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:4766 7.c 7:31cb714a:::smithi12556470-7:head [setxattr _header (57) in=7b,truncate 3025392] snapc 17d=[17d,17c,179,178,177,176,172,16e,16d,168,166,165,160,15e,159,157,153,139,136,129,127,11b,104,fa] ondisk+write+known_if_redirected e491)", 2021-03-24T22:02:08.648 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:01:01.286615+0000", 2021-03-24T22:02:08.648 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 66.855560189000002, 2021-03-24T22:02:08.648 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0508728430000001, 2021-03-24T22:02:08.648 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:02:08.649 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:02:08.649 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:02:08.649 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:02:08.649 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:02:08.649 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4766 2021-03-24T22:02:08.650 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.650 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:02:08.650 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.650 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:02:08.651 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286615+0000", 2021-03-24T22:02:08.651 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:02:08.651 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.651 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.651 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:02:08.652 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286615+0000", 2021-03-24T22:02:08.652 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T22:02:08.652 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.652 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.652 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:02:08.653 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286614+0000", 2021-03-24T22:02:08.653 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6700000000000002e-06 2021-03-24T22:02:08.653 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.653 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.654 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:02:08.654 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286619+0000", 2021-03-24T22:02:08.654 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.3249999999999998e-06 2021-03-24T22:02:08.654 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.654 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.655 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:02:08.655 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286624+0000", 2021-03-24T22:02:08.655 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6290000000000001e-06 2021-03-24T22:02:08.655 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.656 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.656 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:02:08.656 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286629+0000", 2021-03-24T22:02:08.656 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0035533230000000002 2021-03-24T22:02:08.656 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.657 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.657 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.657 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.290182+0000", 2021-03-24T22:02:08.657 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.5209999999999999e-06 2021-03-24T22:02:08.657 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.658 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.658 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:02:08.658 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.290189+0000", 2021-03-24T22:02:08.658 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.044822132 2021-03-24T22:02:08.659 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.659 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.659 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.659 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.335011+0000", 2021-03-24T22:02:08.659 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012871200000000001 2021-03-24T22:02:08.660 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.660 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.660 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:02:08.660 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.335140+0000", 2021-03-24T22:02:08.661 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000225832 2021-03-24T22:02:08.661 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.661 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.661 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,7", 2021-03-24T22:02:08.661 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.335365+0000", 2021-03-24T22:02:08.662 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0010523500000000001 2021-03-24T22:02:08.662 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.662 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.662 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:02:08.662 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.336418+0000", 2021-03-24T22:02:08.663 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00084202099999999998 2021-03-24T22:02:08.663 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.663 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.663 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:02:08.664 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.337260+0000", 2021-03-24T22:02:08.664 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.0683999999999999e-05 2021-03-24T22:02:08.664 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.664 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.664 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:02:08.665 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.337300+0000", 2021-03-24T22:02:08.665 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.4521999999999999e-05 2021-03-24T22:02:08.665 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.665 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.665 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:02:08.666 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.337335+0000", 2021-03-24T22:02:08.666 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00015305900000000001 2021-03-24T22:02:08.666 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.666 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.667 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:02:08.667 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.337488+0000", 2021-03-24T22:02:08.667 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0508728430000001 2021-03-24T22:02:08.667 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:02:08.668 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:02:08.668 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:02:08.668 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.668 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.668 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:4767 7.c 7:31cb714a:::smithi12556470-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e491)", 2021-03-24T22:02:08.669 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:01:01.286635+0000", 2021-03-24T22:02:08.669 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 66.855540695000002, 2021-03-24T22:02:08.669 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.051186642, 2021-03-24T22:02:08.669 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:02:08.669 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T22:02:08.670 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:02:08.670 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:02:08.670 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:02:08.670 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4767 2021-03-24T22:02:08.671 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.671 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:02:08.671 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.671 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:02:08.672 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286635+0000", 2021-03-24T22:02:08.672 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:02:08.672 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.672 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.672 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:02:08.673 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286635+0000", 2021-03-24T22:02:08.673 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999995 2021-03-24T22:02:08.673 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.673 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.673 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:02:08.674 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286634+0000", 2021-03-24T22:02:08.674 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.633e-06 2021-03-24T22:02:08.674 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.674 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.675 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:02:08.675 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286637+0000", 2021-03-24T22:02:08.675 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.7699999999999998e-06 2021-03-24T22:02:08.675 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.675 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.676 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:02:08.676 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286643+0000", 2021-03-24T22:02:08.676 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6349999999999997e-06 2021-03-24T22:02:08.676 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.676 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.677 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:02:08.677 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286647+0000", 2021-03-24T22:02:08.677 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0037662949999999998 2021-03-24T22:02:08.677 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.678 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.678 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.678 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.290414+0000", 2021-03-24T22:02:08.678 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.4619999999999996e-06 2021-03-24T22:02:08.678 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.679 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.679 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:02:08.679 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.290422+0000", 2021-03-24T22:02:08.679 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0454484660000001 2021-03-24T22:02:08.679 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.680 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.680 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.680 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.335870+0000", 2021-03-24T22:02:08.680 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010779400000000001 2021-03-24T22:02:08.681 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.681 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.681 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:02:08.681 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.335978+0000", 2021-03-24T22:02:08.681 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001556396 2021-03-24T22:02:08.682 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.682 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.682 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.682 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.337535+0000", 2021-03-24T22:02:08.683 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010119999999999999 2021-03-24T22:02:08.683 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.683 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.683 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:02:08.683 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.337636+0000", 2021-03-24T22:02:08.684 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00018547400000000001 2021-03-24T22:02:08.684 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.684 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.684 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:02:08.684 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.337821+0000", 2021-03-24T22:02:08.685 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.051186642 2021-03-24T22:02:08.685 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:02:08.685 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:02:08.685 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:02:08.686 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.686 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.686 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:4768 7.21 7:86662589:::smithi12556470-19:head [stat,delete] snapc 17d=[17d,17c,179,178,177,176,172,16e,16d,168,166,165,160,15e,159,157,153,139,136,129,127,11b,104,fa] ondisk+write+known_if_redirected e491)", 2021-03-24T22:02:08.686 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:01:01.286651+0000", 2021-03-24T22:02:08.686 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 66.855524260999999, 2021-03-24T22:02:08.687 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0516308409999999, 2021-03-24T22:02:08.687 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:02:08.687 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:02:08.687 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:02:08.687 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:02:08.688 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:02:08.688 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4768 2021-03-24T22:02:08.688 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.688 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:02:08.689 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.689 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:02:08.689 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286651+0000", 2021-03-24T22:02:08.689 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:02:08.689 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.690 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.690 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:02:08.690 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286651+0000", 2021-03-24T22:02:08.690 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999995 2021-03-24T22:02:08.691 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.691 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.691 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:02:08.691 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286651+0000", 2021-03-24T22:02:08.691 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.4959999999999999e-06 2021-03-24T22:02:08.692 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.692 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.692 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:02:08.692 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286653+0000", 2021-03-24T22:02:08.692 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.7709999999999997e-06 2021-03-24T22:02:08.693 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.693 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.693 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:02:08.693 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286658+0000", 2021-03-24T22:02:08.694 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.66e-06 2021-03-24T22:02:08.694 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.694 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.694 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:02:08.694 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286664+0000", 2021-03-24T22:02:08.695 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0010190959999999999 2021-03-24T22:02:08.695 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.695 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.695 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.696 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.287683+0000", 2021-03-24T22:02:08.696 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.145e-05 2021-03-24T22:02:08.696 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.696 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.696 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:02:08.697 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.287694+0000", 2021-03-24T22:02:08.697 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.048107345 2021-03-24T22:02:08.697 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.697 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.698 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:02:08.698 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.335801+0000", 2021-03-24T22:02:08.698 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00025443900000000002 2021-03-24T22:02:08.698 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.698 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.699 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:02:08.699 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.336056+0000", 2021-03-24T22:02:08.699 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00044708600000000002 2021-03-24T22:02:08.699 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.700 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.700 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 5,7", 2021-03-24T22:02:08.700 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.336503+0000", 2021-03-24T22:02:08.700 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0013468359999999999 2021-03-24T22:02:08.700 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.701 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.701 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:02:08.701 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.337850+0000", 2021-03-24T22:02:08.701 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.1340999999999993e-05 2021-03-24T22:02:08.701 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.702 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.702 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:02:08.702 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.337921+0000", 2021-03-24T22:02:08.702 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.0593999999999999e-05 2021-03-24T22:02:08.703 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.703 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.703 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:02:08.703 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.337952+0000", 2021-03-24T22:02:08.703 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00011551 2021-03-24T22:02:08.704 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.704 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.704 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:02:08.704 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.338067+0000", 2021-03-24T22:02:08.705 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00021471399999999999 2021-03-24T22:02:08.705 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:02:08.705 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:02:08.705 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:02:08.705 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.338282+0000", 2021-03-24T22:02:08.706 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0516308409999999 2021-03-24T22:02:08.706 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:02:08.706 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:02:08.706 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:02:08.707 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:02:08.707 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:02:08.707 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-24T22:02:08.707 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:08.709 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:08 smithi142 conmon[20125]: cluster 2021-03-24T22: 2021-03-24T22:02:08.709 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:08 smithi142 conmon[20125]: 02:06.895561+0000 mgr.y (mgr.14144) 550 : cluster [DBG] pgmap v997: 168 pgs: 1 active+recovering+remapped, 2 peering, 1 active+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 156 active+clean; 565 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.3 MiB/s wr, 7 op/s; 85/1779 objects degraded (4.778%); 155/1779 objects misplaced (8.713%); 32 MiB/s, 65 objects/s recovering 2021-03-24T22:02:08.710 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:08 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:02:08.710 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:08 smithi142 conmon[20125]: T22:02:07.288837+0000 mon.a (mon.0) 1562 : cluster [DBG] 2021-03-24T22:02:08.710 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:08 smithi142 conmon[20125]: osdmap e557: 8 total, 8 up, 8 in 2021-03-24T22:02:08.710 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:08 smithi142 conmon[20125]: cluster 2021-03-24T22:02:08.711 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:08 smithi142 conmon[20125]: 2021-03-24T22:02:07.324945+0000 mon.a 2021-03-24T22:02:08.711 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:08 smithi142 conmon[20125]: (mon.0) 1563 : cluster [WRN] Health check update: Degraded data redundancy: 85/1779 objects degraded (4.778%), 3 pgs degraded (PG_DEGRADED) 2021-03-24T22:02:08.711 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:08 smithi142 conmon[20125]: 2021-03-24T22:02:08.711 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:08 smithi142 conmon[20125]: audit 2021 2021-03-24T22:02:08.711 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:08 smithi142 conmon[20125]: -03-24T22:02:07.449934+0000 mon.a ( 2021-03-24T22:02:08.712 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:08 smithi142 conmon[20125]: mon.0) 1564 : audit [INF] from='client.? v1:172.21.15.125:0/3907186001' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:02:08.713 INFO:tasks.rados.rados.0.smithi125.stdout:2093: done (1 left) 2021-03-24T22:02:08.713 INFO:tasks.rados.rados.0.smithi125.stdout:2094: write oid 49 current snap is 235 2021-03-24T22:02:08.713 INFO:tasks.rados.rados.0.smithi125.stdout:2094: seq_num 734 ranges {736338=402654,1552136=539580} 2021-03-24T22:02:08.714 INFO:tasks.rados.rados.0.smithi125.stdout:2094: writing smithi12556470-49 from 736338 to 1138992 tid 1 2021-03-24T22:02:08.717 INFO:tasks.rados.rados.0.smithi125.stdout:2094: writing smithi12556470-49 from 1552136 to 2091716 tid 2 2021-03-24T22:02:08.718 INFO:tasks.rados.rados.0.smithi125.stdout:2092: finishing write tid 1 to smithi12556470-38 2021-03-24T22:02:08.718 INFO:tasks.rados.rados.0.smithi125.stdout:2092: finishing write tid 2 to smithi12556470-38 2021-03-24T22:02:08.718 INFO:tasks.rados.rados.0.smithi125.stdout:2092: finishing write tid 3 to smithi12556470-38 2021-03-24T22:02:08.718 INFO:tasks.rados.rados.0.smithi125.stdout:2092: finishing write tid 4 to smithi12556470-38 2021-03-24T22:02:08.719 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 38 v 159 (ObjNum 733 snap 235 seq_num 733) dirty exists 2021-03-24T22:02:08.719 INFO:tasks.rados.rados.0.smithi125.stdout:2092: left oid 38 (ObjNum 733 snap 235 seq_num 733) 2021-03-24T22:02:08.719 INFO:tasks.rados.rados.0.smithi125.stdout:2092: done (1 left) 2021-03-24T22:02:08.719 INFO:tasks.rados.rados.0.smithi125.stdout:2095: delete oid 11 current snap is 235 2021-03-24T22:02:08.720 INFO:tasks.rados.rados.0.smithi125.stdout:2094: finishing write tid 1 to smithi12556470-49 2021-03-24T22:02:08.720 INFO:tasks.rados.rados.0.smithi125.stdout:2094: finishing write tid 2 to smithi12556470-49 2021-03-24T22:02:08.720 INFO:tasks.rados.rados.0.smithi125.stdout:2094: finishing write tid 3 to smithi12556470-49 2021-03-24T22:02:08.720 INFO:tasks.rados.rados.0.smithi125.stdout:2094: finishing write tid 4 to smithi12556470-49 2021-03-24T22:02:08.720 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 49 v 697 (ObjNum 734 snap 235 seq_num 734) dirty exists 2021-03-24T22:02:08.721 INFO:tasks.rados.rados.0.smithi125.stdout:2094: left oid 49 (ObjNum 734 snap 235 seq_num 734) 2021-03-24T22:02:08.721 INFO:teuthology.orchestra.run.smithi125.stderr:nodeep-scrub is set 2021-03-24T22:02:08.722 INFO:tasks.rados.rados.0.smithi125.stdout:2094: done (1 left) 2021-03-24T22:02:08.723 INFO:tasks.rados.rados.0.smithi125.stdout:2095: done (0 left) 2021-03-24T22:02:08.723 INFO:tasks.rados.rados.0.smithi125.stdout:2096: read oid 9 snap -1 2021-03-24T22:02:08.723 INFO:tasks.rados.rados.0.smithi125.stdout:2096: expect (ObjNum 588 snap 177 seq_num 588) 2021-03-24T22:02:08.723 INFO:tasks.rados.rados.0.smithi125.stdout:2097: read oid 47 snap -1 2021-03-24T22:02:08.724 INFO:tasks.rados.rados.0.smithi125.stdout:2097: expect (ObjNum 466 snap 131 seq_num 466) 2021-03-24T22:02:08.724 INFO:tasks.rados.rados.0.smithi125.stdout:2098: read oid 17 snap -1 2021-03-24T22:02:08.724 INFO:tasks.rados.rados.0.smithi125.stdout:2098: expect (ObjNum 724 snap 231 seq_num 724) 2021-03-24T22:02:08.724 INFO:tasks.rados.rados.0.smithi125.stdout:2099: write (excl) oid 42 current snap is 235 2021-03-24T22:02:08.724 INFO:tasks.rados.rados.0.smithi125.stdout:2099: seq_num 736 ranges {769952=553152,2069100=300852} 2021-03-24T22:02:08.725 INFO:tasks.rados.rados.0.smithi125.stdout:2099: writing smithi12556470-42 from 769952 to 1323104 tid 1 2021-03-24T22:02:08.725 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:08 smithi125 conmon[24927]: cluster 2021-03-24T22:02:06.895561+0000 mgr.y (mgr.14144) 550 : cluster 2021-03-24T22:02:08.725 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:08 smithi125 conmon[24927]: [DBG] pgmap v997: 168 pgs: 1 active+recovering+remapped, 2 peering, 1 active+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 156 active+clean; 565 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.3 MiB/s wr, 7 op/s; 85/1779 objects degraded (4.778%); 155/1779 objects misplaced (8.713%); 32 MiB/s, 65 objects/s recovering 2021-03-24T22:02:08.726 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:08 smithi125 conmon[24927]: cluster 2021-03-24T22:02:07.288837+0000 mon.a (mon.0) 1562 : cluster [DBG] osdmap e557: 8 total, 8 up, 8 in 2021-03-24T22:02:08.726 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:08 smithi125 conmon[24927]: cluster 2021-03-24T22:02: 2021-03-24T22:02:08.726 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:08 smithi125 conmon[24927]: 07.324945+0000 mon.a (mon.0) 1563 : cluster [WRN] Health check update: Degraded data redundancy: 85/1779 objects degraded (4.778%), 3 pgs degraded (PG_DEGRADED) 2021-03-24T22:02:08.726 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:08 smithi125 conmon[24927]: audit 2021-03-24T22:02:07.449934+0000 mon.a (mon.0) 1564 : audit [INF] from='client.? v1:172.21.15.125:0/3907186001' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:02:08.727 INFO:tasks.rados.rados.0.smithi125.stdout:2099: writing smithi12556470-42 from 2069100 to 2369952 tid 2 2021-03-24T22:02:08.728 INFO:tasks.rados.rados.0.smithi125.stdout:2100: snap_remove snap 199 2021-03-24T22:02:08.728 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:08 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:02:08.729 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:08 smithi125 conmon[19936]: 24T22:02:06.895561+0000 mgr.y (mgr.14144) 550 : 2021-03-24T22:02:08.729 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:08 smithi125 conmon[19936]: cluster [DBG] 2021-03-24T22:02:08.729 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:08 smithi125 conmon[19936]: pgmap v997: 168 pgs: 1 active+recovering+remapped, 2 peering, 1 active+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 156 active+clean; 565 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.3 MiB/s wr, 7 op/s; 85/1779 objects degraded (4.778%); 155/1779 objects misplaced (8.713%); 32 MiB/s, 65 objects/s recovering 2021-03-24T22:02:08.729 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:08 smithi125 conmon[19936]: cluster 2021-03-24T22:02:08.729 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:08 smithi125 conmon[19936]: 2021-03-24 2021-03-24T22:02:08.730 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:08 smithi125 conmon[19936]: T22: 2021-03-24T22:02:08.730 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:08 smithi125 conmon[19936]: 02:07 2021-03-24T22:02:08.730 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:08 smithi125 conmon[19936]: .288837+0000 2021-03-24T22:02:08.730 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:08 smithi125 conmon[19936]: mon.a ( 2021-03-24T22:02:08.731 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:08 smithi125 conmon[19936]: mon.0 2021-03-24T22:02:08.731 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:08 smithi125 conmon[19936]: ) 1562 : 2021-03-24T22:02:08.731 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:08 smithi125 conmon[19936]: cluster [DBG] 2021-03-24T22:02:08.731 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:08 smithi125 conmon[19936]: osdmap e557: 8 total, 8 up, 8 in 2021-03-24T22:02:08.732 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:08 smithi125 conmon[19936]: cluster 2021-03-24T22:02:08.732 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:08 smithi125 conmon[19936]: 2021-03- 2021-03-24T22:02:08.732 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:08 smithi125 conmon[19936]: 24T22 2021-03-24T22:02:08.732 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:08 smithi125 conmon[19936]: :02:07 2021-03-24T22:02:08.732 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:08 smithi125 conmon[19936]: .324945+0000 2021-03-24T22:02:08.733 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:08 smithi125 conmon[19936]: mon.a ( 2021-03-24T22:02:08.733 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:08 smithi125 conmon[19936]: mon.0) 2021-03-24T22:02:08.733 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:08 smithi125 conmon[19936]: 1563 2021-03-24T22:02:08.733 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:08 smithi125 conmon[19936]: : cluster 2021-03-24T22:02:08.733 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:08 smithi125 conmon[19936]: [WRN] Health check update: Degraded data redundancy: 85/1779 objects degraded (4.778%), 3 pgs degraded (PG_DEGRADED) 2021-03-24T22:02:08.734 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:08 smithi125 conmon[19936]: audit 2021-03-24T22:02:08.734 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:08 smithi125 conmon[19936]: 2021- 2021-03-24T22:02:08.734 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:08 smithi125 conmon[19936]: 03-24T22:02:07.449934 2021-03-24T22:02:08.734 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:08 smithi125 conmon[19936]: +0000 mon.a (mon. 2021-03-24T22:02:08.735 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:08 smithi125 conmon[19936]: 0) 1564 : audit 2021-03-24T22:02:08.735 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:08 smithi125 conmon[19936]: [INF] from='client.? v1:172.21.15.125:0/3907186001' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:02:08.764 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:08.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:08.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:02:08.869 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:08.970 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:08.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:08.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:02:09.074 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:09.127 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:09.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:09.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:02:09.233 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:09.283 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:09.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:09.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:02:09.310 INFO:tasks.rados.rados.0.smithi125.stdout:2100: done (4 left) 2021-03-24T22:02:09.311 INFO:tasks.rados.rados.0.smithi125.stdout:2101: read oid 6 snap 233 2021-03-24T22:02:09.311 INFO:tasks.rados.rados.0.smithi125.stdout:2101: expect deleted 2021-03-24T22:02:09.311 INFO:tasks.rados.rados.0.smithi125.stdout:2102: snap_create 2021-03-24T22:02:09.312 INFO:tasks.rados.rados.0.smithi125.stdout:2096: expect (ObjNum 588 snap 177 seq_num 588) 2021-03-24T22:02:09.366 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:09 smithi142 conmon[20125]: cluster 2021-03-24T22:02:08.285555+0000 mon.a (mon 2021-03-24T22:02:09.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:09 smithi142 conmon[20125]: .0) 1565 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:02:09.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:09 smithi142 conmon[20125]: audit 2021-03-24T22:02:09.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:09 smithi142 conmon[20125]: 2021-03-24T22 2021-03-24T22:02:09.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:09 smithi142 conmon[20125]: :02:08.301017 2021-03-24T22:02:09.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:09 smithi142 conmon[20125]: +0000 mon.a (mon 2021-03-24T22:02:09.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:09 smithi142 conmon[20125]: .0) 1566 : audit 2021-03-24T22:02:09.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:09 smithi142 conmon[20125]: [INF] from='client.? v1:172.21.15.125:0/3907186001' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T22:02:09.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:09 smithi142 conmon[20125]: cluster 2021-03-24T22:02:09.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:09 smithi142 conmon[20125]: 2021-03-24T 2021-03-24T22:02:09.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:09 smithi142 conmon[20125]: 22:02:08. 2021-03-24T22:02:09.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:09 smithi142 conmon[20125]: 301083+0000 mon.a ( 2021-03-24T22:02:09.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:09 smithi142 conmon[20125]: mon.0) 1567 2021-03-24T22:02:09.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:09 smithi142 conmon[20125]: : cluster [DBG] osdmap e558: 8 total, 8 up, 8 in 2021-03-24T22:02:09.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:09 smithi142 conmon[20125]: 2021-03-24T22:02:09.388 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:09.436 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:09.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:09.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:02:09.538 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:09 smithi125 conmon[19936]: cluster 2021-03-24T22:02: 2021-03-24T22:02:09.538 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:09 smithi125 conmon[19936]: 08.285555+0000 mon.a (mon.0) 1565 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:02:09.539 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:09 smithi125 conmon[19936]: audit 2021-03-24T22:02 2021-03-24T22:02:09.539 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:09 smithi125 conmon[19936]: :08.301017+0000 mon.a (mon.0) 1566 : audit [INF] from='client.? v1:172.21.15.125:0/3907186001' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T22:02:09.539 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:09 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:02:09.539 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:09 smithi125 conmon[19936]: -03-24T22:02:08.301083+0000 mon.a (mon.0) 1567 : cluster [DBG] 2021-03-24T22:02:09.540 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:09 smithi125 conmon[19936]: osdmap e558: 8 total, 8 up, 8 in 2021-03-24T22:02:09.540 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:09 smithi125 conmon[24927]: cluster 2021-03-24T22:02:08. 2021-03-24T22:02:09.540 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:09 smithi125 conmon[24927]: 285555+0000 mon.a (mon.0) 1565 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:02:09.540 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:09 smithi125 conmon[24927]: audit 2021-03-24 2021-03-24T22:02:09.541 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:09 smithi125 conmon[24927]: T22:02:08.301017+0000 mon.a ( 2021-03-24T22:02:09.541 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:09 smithi125 conmon[24927]: mon.0) 1566 : audit [INF] from='client.? v1:172.21.15.125:0/3907186001' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T22:02:09.541 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:09 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:02:09.542 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:09 smithi125 conmon[24927]: -03-24T22:02:08.301083 2021-03-24T22:02:09.542 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:09 smithi125 conmon[24927]: +0000 mon.a (mon.0) 1567 : cluster [DBG] 2021-03-24T22:02:09.542 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:09 smithi125 conmon[24927]: osdmap e558: 8 total, 8 up, 8 in 2021-03-24T22:02:09.542 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:09.595 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:09.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:09.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:02:09.639 INFO:tasks.rados.rados.0.smithi125.stdout:2099: finishing write tid 1 to smithi12556470-42 2021-03-24T22:02:09.639 INFO:tasks.rados.rados.0.smithi125.stdout:2098: expect (ObjNum 724 snap 231 seq_num 724) 2021-03-24T22:02:09.701 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:09.749 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:09.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:09.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:02:09.800 INFO:tasks.rados.rados.0.smithi125.stdout:2099: finishing write tid 2 to smithi12556470-42 2021-03-24T22:02:09.801 INFO:tasks.rados.rados.0.smithi125.stdout:2097: expect (ObjNum 466 snap 131 seq_num 466) 2021-03-24T22:02:09.854 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:09.910 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:09.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:09.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:02:09.915 INFO:tasks.rados.rados.0.smithi125.stdout:2099: finishing write tid 3 to smithi12556470-42 2021-03-24T22:02:09.915 INFO:tasks.rados.rados.0.smithi125.stdout:2099: finishing write tid 4 to smithi12556470-42 2021-03-24T22:02:09.915 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 42 v 503 (ObjNum 736 snap 235 seq_num 736) dirty exists 2021-03-24T22:02:09.915 INFO:tasks.rados.rados.0.smithi125.stdout:2099: left oid 42 (ObjNum 736 snap 235 seq_num 736) 2021-03-24T22:02:10.014 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:10.074 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:10.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:10.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:02:10.178 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:10.226 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:10.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:10.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:02:10.296 INFO:tasks.rados.rados.0.smithi125.stdout:2096: done (5 left) 2021-03-24T22:02:10.296 INFO:tasks.rados.rados.0.smithi125.stdout:2097: done (4 left) 2021-03-24T22:02:10.297 INFO:tasks.rados.rados.0.smithi125.stdout:2098: done (3 left) 2021-03-24T22:02:10.297 INFO:tasks.rados.rados.0.smithi125.stdout:2099: done (2 left) 2021-03-24T22:02:10.297 INFO:tasks.rados.rados.0.smithi125.stdout:2101: done (1 left) 2021-03-24T22:02:10.297 INFO:tasks.rados.rados.0.smithi125.stdout:2102: done (0 left) 2021-03-24T22:02:10.297 INFO:tasks.rados.rados.0.smithi125.stdout:2103: snap_create 2021-03-24T22:02:10.333 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:10.373 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:10.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:10.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:02:10.480 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:10.550 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:10.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:10.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:02:10.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:10 smithi142 conmon[20125]: cluster 2021-03-24T22:02:08.896257+0000 mgr.y (mgr 2021-03-24T22:02:10.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:10 smithi142 conmon[20125]: .14144) 551 : cluster [DBG] pgmap v1000: 168 pgs: 1 active+recovering+remapped, 2 peering, 1 active+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 156 active+clean; 565 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 1.1 MiB/s wr, 1 op/s; 85/1779 objects degraded (4.778%); 155/1779 objects misplaced (8.713%); 18 MiB/s, 32 objects/s recovering 2021-03-24T22:02:10.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:10 smithi142 conmon[20125]: cluster 2021-03-24T22:02:09.303488+0000 mon.a (mon.0) 1568 : cluster [DBG] osdmap e559: 8 total, 8 up, 8 in 2021-03-24T22:02:10.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:10 smithi142 conmon[20125]: cluster 2021-03-24T22:02:10.292042+0000 mon.a 2021-03-24T22:02:10.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:10 smithi142 conmon[20125]: (mon.0) 1569 : cluster 2021-03-24T22:02:10.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:10 smithi142 conmon[20125]: [DBG] osdmap e560: 8 total, 8 up, 8 in 2021-03-24T22:02:10.657 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:10.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:10 smithi125 conmon[24927]: cluster 2021-03-24T22:02:08.896257+0000 mgr.y ( 2021-03-24T22:02:10.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:10 smithi125 conmon[24927]: mgr.14144) 551 : cluster [DBG] pgmap v1000: 168 pgs: 1 active+recovering+remapped, 2 peering, 1 active+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 156 active+clean; 565 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 1.1 MiB/s wr, 1 op/s; 85/1779 objects degraded (4.778%); 155/1779 objects misplaced (8.713%); 18 MiB/s, 32 objects/s recovering 2021-03-24T22:02:10.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:10 smithi125 conmon[24927]: cluster 2021-03-24T22:02:09.303488+0000 mon.a (mon.0) 1568 : cluster [DBG] osdmap e559: 8 total, 8 up, 8 in 2021-03-24T22:02:10.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:10 smithi125 conmon[24927]: cluster 2021-03-24T22:02:10.292042+0000 mon.a (mon.0) 1569 : cluster 2021-03-24T22:02:10.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:10 smithi125 conmon[24927]: [DBG] osdmap e560: 8 total, 8 up, 8 in 2021-03-24T22:02:10.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:10 smithi125 conmon[19936]: cluster 2021-03-24T22:02:10.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:10 smithi125 conmon[19936]: 2021-03-24T22:02:10.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:10 smithi125 conmon[19936]: 2021-03-24T22:02 2021-03-24T22:02:10.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:10 smithi125 conmon[19936]: :08.896257+0000 mgr.y (mgr.14144) 551 : cluster [DBG] pgmap v1000: 168 pgs: 1 active+recovering+remapped, 2 peering, 1 active+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 156 active+clean; 565 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 1.1 MiB/s wr, 1 op/s; 85/1779 objects degraded (4.778%); 155/1779 objects misplaced (8.713%); 18 MiB/s, 32 objects/s recovering 2021-03-24T22:02:10.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:10 smithi125 conmon[19936]: cluster 2021-03-24T22:02:10.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:10 smithi125 conmon[19936]: 2021-03 2021-03-24T22:02:10.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:10 smithi125 conmon[19936]: -24T22:02:09.303488+0000 mon.a (mon 2021-03-24T22:02:10.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:10 smithi125 conmon[19936]: .0) 1568 : cluster [DBG] osdmap e559: 8 total, 8 up, 8 in 2021-03-24T22:02:10.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:10 smithi125 conmon[19936]: cluster 2021-03-24T22:02:10.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:10 smithi125 conmon[19936]: 2021-03-24T22:02:10.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:10 smithi125 conmon[19936]: 2021-03-24T22 2021-03-24T22:02:10.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:10 smithi125 conmon[19936]: :02:10.292042+0000 mon.a (mon.0) 1569 : cluster [DBG] 2021-03-24T22:02:10.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:10 smithi125 conmon[19936]: osdmap e560: 8 total, 8 up, 8 in 2021-03-24T22:02:10.710 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:10.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:10.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:02:10.816 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:10.882 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:10.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:10.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:02:10.988 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:11.040 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:11.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:11.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:02:11.145 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:11.201 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:11.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:11.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:02:11.299 INFO:tasks.rados.rados.0.smithi125.stdout:2103: done (0 left) 2021-03-24T22:02:11.299 INFO:tasks.rados.rados.0.smithi125.stdout:2104: read oid 14 snap -1 2021-03-24T22:02:11.299 INFO:tasks.rados.rados.0.smithi125.stdout:2104: expect (ObjNum 625 snap 192 seq_num 625) 2021-03-24T22:02:11.300 INFO:tasks.rados.rados.0.smithi125.stdout:2105: read oid 32 snap -1 2021-03-24T22:02:11.300 INFO:tasks.rados.rados.0.smithi125.stdout:2105: expect (ObjNum 672 snap 209 seq_num 672) 2021-03-24T22:02:11.300 INFO:tasks.rados.rados.0.smithi125.stdout:2106: read oid 24 snap 208 2021-03-24T22:02:11.300 INFO:tasks.rados.rados.0.smithi125.stdout:2106: expect deleted 2021-03-24T22:02:11.301 INFO:tasks.rados.rados.0.smithi125.stdout:2107: copy_from oid 29 from oid 36 current snap is 237 2021-03-24T22:02:11.301 INFO:tasks.rados.rados.0.smithi125.stdout:2108: read oid 15 snap -1 2021-03-24T22:02:11.301 INFO:tasks.rados.rados.0.smithi125.stdout:2108: expect deleted 2021-03-24T22:02:11.301 INFO:tasks.rados.rados.0.smithi125.stdout:2109: snap_create 2021-03-24T22:02:11.304 INFO:tasks.rados.rados.0.smithi125.stdout:2107: finishing copy_from to smithi12556470-29 2021-03-24T22:02:11.304 INFO:tasks.rados.rados.0.smithi125.stdout:2107: got expected ENOENT (src dne) 2021-03-24T22:02:11.305 INFO:tasks.rados.rados.0.smithi125.stdout:2107: finishing copy_from racing read to smithi12556470-29 2021-03-24T22:02:11.308 INFO:tasks.rados.rados.0.smithi125.stdout:2104: expect (ObjNum 625 snap 192 seq_num 625) 2021-03-24T22:02:11.312 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:11.355 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:11.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:11.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:02:11.398 INFO:tasks.rados.rados.0.smithi125.stdout:2105: expect (ObjNum 672 snap 209 seq_num 672) 2021-03-24T22:02:11.460 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:11.516 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:11.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:11.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:02:11.622 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:11.675 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:11.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:11.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:02:11.734 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.4 dump_ops_in_flight 2021-03-24T22:02:11.779 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:11.838 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:11.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:11.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:02:11.945 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:11.999 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:12.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:12.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:02:12.030 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T22:02:12.105 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:12.158 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:12.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:12.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:02:12.195 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T22:02:12.217 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T22:02:12.238 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T22:02:12.258 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T22:02:12.262 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:12.287 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T22:02:12.300 INFO:tasks.rados.rados.0.smithi125.stdout:2104: done (5 left) 2021-03-24T22:02:12.301 INFO:tasks.rados.rados.0.smithi125.stdout:2105: done (4 left) 2021-03-24T22:02:12.301 INFO:tasks.rados.rados.0.smithi125.stdout:2106: done (3 left) 2021-03-24T22:02:12.301 INFO:tasks.rados.rados.0.smithi125.stdout:2107: done (2 left) 2021-03-24T22:02:12.301 INFO:tasks.rados.rados.0.smithi125.stdout:2108: done (1 left) 2021-03-24T22:02:12.302 INFO:tasks.rados.rados.0.smithi125.stdout:2109: done (0 left) 2021-03-24T22:02:12.302 INFO:tasks.rados.rados.0.smithi125.stdout:2110: write oid 33 current snap is 238 2021-03-24T22:02:12.302 INFO:tasks.rados.rados.0.smithi125.stdout:2110: seq_num 737 ranges {786759=628401,2011229=375530} 2021-03-24T22:02:12.309 INFO:tasks.rados.rados.0.smithi125.stdout:2110: writing smithi12556470-33 from 786759 to 1415160 tid 1 2021-03-24T22:02:12.322 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:12.325 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T22:02:12.326 INFO:tasks.rados.rados.0.smithi125.stdout:2110: writing smithi12556470-33 from 2011229 to 2386759 tid 2 2021-03-24T22:02:12.327 INFO:tasks.rados.rados.0.smithi125.stdout:2111: rollback oid 4 current snap is 238 2021-03-24T22:02:12.327 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 4 to 214 2021-03-24T22:02:12.327 INFO:tasks.rados.rados.0.smithi125.stdout:2110: finishing write tid 1 to smithi12556470-33 2021-03-24T22:02:12.327 INFO:tasks.rados.rados.0.smithi125.stdout:2112: snap_remove snap 158 2021-03-24T22:02:12.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:12.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:02:12.350 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T22:02:12.369 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T22:02:12.371 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-24T22:02:12.371 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-24T22:02:12.372 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-24T22:02:12.372 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-24T22:02:12.372 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-24T22:02:12.372 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-24T22:02:12.373 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-24T22:02:12.373 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-24T22:02:12.433 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:12.444 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: cluster 2021-03-24T22:02:12.444 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 2021-03-24T22:02:12.444 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 2021 2021-03-24T22:02:12.444 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: - 2021-03-24T22:02:12.445 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 03 2021-03-24T22:02:12.445 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: - 2021-03-24T22:02:12.445 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 24 2021-03-24T22:02:12.445 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: T 2021-03-24T22:02:12.446 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 22 2021-03-24T22:02:12.446 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: : 2021-03-24T22:02:12.446 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 02 2021-03-24T22:02:12.446 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: : 2021-03-24T22:02:12.447 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 10 2021-03-24T22:02:12.447 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: . 2021-03-24T22:02:12.447 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 897054 2021-03-24T22:02:12.448 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: +0000 2021-03-24T22:02:12.448 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 2021-03-24T22:02:12.448 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: mgr.y 2021-03-24T22:02:12.448 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: ( 2021-03-24T22:02:12.449 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: mgr 2021-03-24T22:02:12.449 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: . 2021-03-24T22:02:12.450 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 14144 2021-03-24T22:02:12.450 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: ) 2021-03-24T22:02:12.450 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 552 2021-03-24T22:02:12.450 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: : 2021-03-24T22:02:12.451 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: cluster 2021-03-24T22:02:12.451 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 2021-03-24T22:02:12.451 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: [DBG] 2021-03-24T22:02:12.451 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 2021-03-24T22:02:12.452 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: pgmap v1003: 168 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 peering, 2 active+undersized+remapped, 1 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 159 active+clean; 537 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 6.4 MiB/s rd, 949 KiB/s wr, 10 op/s; 136/1740 objects misplaced (7.816%); 15 MiB/s, 28 objects/s recovering 2021-03-24T22:02:12.452 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 2021-03-24T22:02:12.453 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: cluster 2021-03-24T22:02:12.454 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 2021-03-24T22:02:12.454 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 2021 2021-03-24T22:02:12.454 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: - 2021-03-24T22:02:12.455 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 03 2021-03-24T22:02:12.455 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: - 2021-03-24T22:02:12.455 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 24 2021-03-24T22:02:12.455 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: T 2021-03-24T22:02:12.455 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 22 2021-03-24T22:02:12.456 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: : 2021-03-24T22:02:12.456 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 02 2021-03-24T22:02:12.456 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: : 2021-03-24T22:02:12.456 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 11 2021-03-24T22:02:12.457 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: . 2021-03-24T22:02:12.457 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 293211 2021-03-24T22:02:12.457 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: +0000 2021-03-24T22:02:12.457 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 2021-03-24T22:02:12.457 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: mon.a 2021-03-24T22:02:12.458 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: ( 2021-03-24T22:02:12.458 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: mon 2021-03-24T22:02:12.458 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: . 2021-03-24T22:02:12.458 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 0 2021-03-24T22:02:12.458 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: ) 2021-03-24T22:02:12.459 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 1570 2021-03-24T22:02:12.459 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: : 2021-03-24T22:02:12.459 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: cluster 2021-03-24T22:02:12.460 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 2021-03-24T22:02:12.460 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: [DBG] 2021-03-24T22:02:12.460 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 2021-03-24T22:02:12.461 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: osdmap e561: 8 total, 8 up, 8 in 2021-03-24T22:02:12.461 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 2021-03-24T22:02:12.461 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: cluster 2021-03-24T22:02:12.461 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 2021-03-24T22:02:12.462 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 2021 2021-03-24T22:02:12.462 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: - 2021-03-24T22:02:12.463 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 03 2021-03-24T22:02:12.464 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: - 2021-03-24T22:02:12.464 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 24 2021-03-24T22:02:12.464 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: T 2021-03-24T22:02:12.465 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 22 2021-03-24T22:02:12.465 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: : 2021-03-24T22:02:12.465 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 02 2021-03-24T22:02:12.465 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: : 2021-03-24T22:02:12.465 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 11 2021-03-24T22:02:12.466 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: . 2021-03-24T22:02:12.466 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 308903 2021-03-24T22:02:12.466 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: +0000 2021-03-24T22:02:12.466 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 2021-03-24T22:02:12.466 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: mon.a 2021-03-24T22:02:12.467 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: ( 2021-03-24T22:02:12.467 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: mon 2021-03-24T22:02:12.467 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: cluster 2021-03-24T22:02:12.468 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 2021-03-24T22:02:12.468 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 2021 2021-03-24T22:02:12.468 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: - 2021-03-24T22:02:12.468 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 03 2021-03-24T22:02:12.469 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: - 2021-03-24T22:02:12.469 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 24 2021-03-24T22:02:12.469 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: T 2021-03-24T22:02:12.469 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 22 2021-03-24T22:02:12.469 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: : 2021-03-24T22:02:12.470 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 02 2021-03-24T22:02:12.470 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: : 2021-03-24T22:02:12.470 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 10 2021-03-24T22:02:12.470 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: . 2021-03-24T22:02:12.470 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 897054 2021-03-24T22:02:12.471 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: +0000 2021-03-24T22:02:12.471 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 2021-03-24T22:02:12.471 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: mgr.y 2021-03-24T22:02:12.471 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: ( 2021-03-24T22:02:12.472 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: mgr 2021-03-24T22:02:12.472 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: . 2021-03-24T22:02:12.472 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 14144 2021-03-24T22:02:12.472 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: ) 2021-03-24T22:02:12.472 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 552 2021-03-24T22:02:12.473 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: : 2021-03-24T22:02:12.473 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: cluster 2021-03-24T22:02:12.473 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 2021-03-24T22:02:12.474 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: [DBG] 2021-03-24T22:02:12.474 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 2021-03-24T22:02:12.475 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: pgmap v1003: 168 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 peering, 2 active+undersized+remapped, 1 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 159 active+clean; 537 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 6.4 MiB/s rd, 949 KiB/s wr, 10 op/s; 136/1740 objects misplaced (7.816%); 15 MiB/s, 28 objects/s recovering 2021-03-24T22:02:12.475 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 2021-03-24T22:02:12.475 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: cluster 2021-03-24T22:02:12.475 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 2021-03-24T22:02:12.475 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 2021 2021-03-24T22:02:12.476 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: - 2021-03-24T22:02:12.476 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 03 2021-03-24T22:02:12.476 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: - 2021-03-24T22:02:12.476 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 24 2021-03-24T22:02:12.476 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: T 2021-03-24T22:02:12.477 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 22 2021-03-24T22:02:12.477 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: : 2021-03-24T22:02:12.477 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 02 2021-03-24T22:02:12.477 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: : 2021-03-24T22:02:12.477 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 11 2021-03-24T22:02:12.478 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: . 2021-03-24T22:02:12.478 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 293211 2021-03-24T22:02:12.478 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: +0000 2021-03-24T22:02:12.478 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 2021-03-24T22:02:12.479 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: mon.a 2021-03-24T22:02:12.479 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: ( 2021-03-24T22:02:12.479 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: mon 2021-03-24T22:02:12.479 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: . 2021-03-24T22:02:12.479 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 0 2021-03-24T22:02:12.480 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: ) 2021-03-24T22:02:12.480 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 1570 2021-03-24T22:02:12.480 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: : 2021-03-24T22:02:12.480 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: cluster 2021-03-24T22:02:12.480 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 2021-03-24T22:02:12.481 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: [DBG] 2021-03-24T22:02:12.481 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 2021-03-24T22:02:12.481 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: osdmap e561: 8 total, 8 up, 8 in 2021-03-24T22:02:12.481 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 2021-03-24T22:02:12.481 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: cluster 2021-03-24T22:02:12.482 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 2021-03-24T22:02:12.482 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 2021 2021-03-24T22:02:12.482 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: - 2021-03-24T22:02:12.482 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 03 2021-03-24T22:02:12.482 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: - 2021-03-24T22:02:12.483 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 24 2021-03-24T22:02:12.483 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: T 2021-03-24T22:02:12.483 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 22 2021-03-24T22:02:12.483 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: : 2021-03-24T22:02:12.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 02 2021-03-24T22:02:12.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: : 2021-03-24T22:02:12.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 11 2021-03-24T22:02:12.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: . 2021-03-24T22:02:12.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 308903 2021-03-24T22:02:12.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: +0000 2021-03-24T22:02:12.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 2021-03-24T22:02:12.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: mon.a 2021-03-24T22:02:12.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: ( 2021-03-24T22:02:12.487 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: mon 2021-03-24T22:02:12.487 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: . 2021-03-24T22:02:12.487 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 0 2021-03-24T22:02:12.488 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: ) 2021-03-24T22:02:12.488 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 1571 2021-03-24T22:02:12.488 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: : 2021-03-24T22:02:12.488 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: cluster 2021-03-24T22:02:12.489 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 2021-03-24T22:02:12.489 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: [INF] 2021-03-24T22:02:12.489 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 2021-03-24T22:02:12.489 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 85/1779 objects degraded (4.778%), 3 pgs degraded) 2021-03-24T22:02:12.489 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 2021-03-24T22:02:12.490 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: audit 2021-03-24T22:02:12.490 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 2021-03-24T22:02:12.490 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 2021 2021-03-24T22:02:12.490 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: - 2021-03-24T22:02:12.490 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 03 2021-03-24T22:02:12.491 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: - 2021-03-24T22:02:12.491 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 24 2021-03-24T22:02:12.491 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: T 2021-03-24T22:02:12.491 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 22 2021-03-24T22:02:12.492 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: : 2021-03-24T22:02:12.492 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 02 2021-03-24T22:02:12.492 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: : 2021-03-24T22:02:12.492 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 12 2021-03-24T22:02:12.492 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: . 2021-03-24T22:02:12.493 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 163197 2021-03-24T22:02:12.493 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: +0000 2021-03-24T22:02:12.493 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 2021-03-24T22:02:12.493 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: mon.a 2021-03-24T22:02:12.493 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: ( 2021-03-24T22:02:12.494 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: mon 2021-03-24T22:02:12.494 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: . 2021-03-24T22:02:12.494 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 0 2021-03-24T22:02:12.495 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: ) 2021-03-24T22:02:12.495 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: 1572 2021-03-24T22:02:12.496 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[24927]: : audit [DBG] from='client.? v1:172.21.15.125:0/3690395155' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:02:12.497 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: . 2021-03-24T22:02:12.497 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 0 2021-03-24T22:02:12.497 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: ) 2021-03-24T22:02:12.497 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 1571 2021-03-24T22:02:12.497 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: : 2021-03-24T22:02:12.498 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: cluster 2021-03-24T22:02:12.498 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 2021-03-24T22:02:12.498 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: [INF] 2021-03-24T22:02:12.498 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 2021-03-24T22:02:12.499 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 85/1779 objects degraded (4.778%), 3 pgs degraded) 2021-03-24T22:02:12.499 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 2021-03-24T22:02:12.499 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: audit 2021-03-24T22:02:12.499 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 2021-03-24T22:02:12.499 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 2021 2021-03-24T22:02:12.500 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: - 2021-03-24T22:02:12.500 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 03 2021-03-24T22:02:12.500 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: - 2021-03-24T22:02:12.500 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 24 2021-03-24T22:02:12.501 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: T 2021-03-24T22:02:12.501 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 22 2021-03-24T22:02:12.501 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: : 2021-03-24T22:02:12.501 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 02 2021-03-24T22:02:12.501 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: : 2021-03-24T22:02:12.502 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 12 2021-03-24T22:02:12.502 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: . 2021-03-24T22:02:12.502 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 163197 2021-03-24T22:02:12.502 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: +0000 2021-03-24T22:02:12.502 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 2021-03-24T22:02:12.503 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: mon.a 2021-03-24T22:02:12.503 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: ( 2021-03-24T22:02:12.503 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: mon 2021-03-24T22:02:12.503 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: . 2021-03-24T22:02:12.503 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 0 2021-03-24T22:02:12.504 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: ) 2021-03-24T22:02:12.504 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 1572 2021-03-24T22:02:12.504 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: : 2021-03-24T22:02:12.504 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: audit 2021-03-24T22:02:12.505 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 2021-03-24T22:02:12.505 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: [DBG] 2021-03-24T22:02:12.505 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 2021-03-24T22:02:12.506 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: from='client.? v1:172.21.15.125:0/3690395155' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:02:12.506 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:12 smithi125 conmon[19936]: 2021-03-24T22:02:12.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:12.509 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:12.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:02:12.578 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:12 smithi142 conmon[20125]: cluster 2021-03-24T22:02:10. 2021-03-24T22:02:12.579 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:12 smithi142 conmon[20125]: 897054+0000 mgr.y (mgr.14144) 552 : cluster [DBG] pgmap v1003: 168 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 peering, 2 active+undersized+remapped, 1 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 159 active+clean; 537 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 6.4 MiB/s rd, 949 KiB/s wr, 10 op/s; 136/1740 objects misplaced (7.816%); 15 MiB/s, 28 objects/s recovering 2021-03-24T22:02:12.579 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:12 smithi142 conmon[20125]: cluster 2021-03-24T22:02:12.580 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:12 smithi142 conmon[20125]: 2021-03-24T22:02:11.293211+0000 mon.a (mon 2021-03-24T22:02:12.580 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:12 smithi142 conmon[20125]: .0) 1570 : cluster [DBG] osdmap e561: 8 total, 8 up, 8 in 2021-03-24T22:02:12.580 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:12 smithi142 conmon[20125]: 2021-03-24T22:02:12.581 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:12 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:02:12.581 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:12 smithi142 conmon[20125]: 24T22:02:11.308903 2021-03-24T22:02:12.582 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:12 smithi142 conmon[20125]: +0000 mon.a (mon.0) 1571 : 2021-03-24T22:02:12.582 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:12 smithi142 conmon[20125]: cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 85/1779 objects degraded (4.778%), 3 pgs degraded) 2021-03-24T22:02:12.583 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:12 smithi142 conmon[20125]: audit 2021- 2021-03-24T22:02:12.583 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:12 smithi142 conmon[20125]: 03-24T22:02:12. 2021-03-24T22:02:12.583 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:12 smithi142 conmon[20125]: 163197+0000 mon.a (mon.0) 2021-03-24T22:02:12.584 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:12 smithi142 conmon[20125]: 1572 : audit [DBG] from='client.? v1:172.21.15.125:0/3690395155' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:02:12.611 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:12.666 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:12.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:12.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:02:12.772 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:12.836 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:12.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:12.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:02:12.942 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:12.996 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:13.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:13.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:02:13.104 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:13.166 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:13.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:13.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:02:13.271 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:13.318 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:13 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:02:13.318 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:13 smithi125 conmon[24927]: 02:12.295573+0000 mon.a (mon.0) 1573 : cluster [DBG] osdmap e562: 8 total, 8 up, 8 in 2021-03-24T22:02:13.319 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:13 smithi125 conmon[24927]: audit 2021-03-24T22: 2021-03-24T22:02:13.319 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:13 smithi125 conmon[24927]: 02:12.611800+0000 mon.a (mon.0 2021-03-24T22:02:13.319 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:13 smithi125 conmon[24927]: ) 1574 : audit [INF] from='client.? v1:172.21.15.125:0/475168767' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2021-03-24T22:02:13.329 INFO:tasks.rados.rados.0.smithi125.stdout:2112: done (2 left) 2021-03-24T22:02:13.329 INFO:tasks.rados.rados.0.smithi125.stdout:2113: read oid 5 snap -1 2021-03-24T22:02:13.329 INFO:tasks.rados.rados.0.smithi125.stdout:2113: expect deleted 2021-03-24T22:02:13.330 INFO:tasks.rados.rados.0.smithi125.stdout:2114: snap_create 2021-03-24T22:02:13.330 INFO:tasks.rados.rados.0.smithi125.stdout:2111: finishing rollback tid 0 to smithi12556470-4 2021-03-24T22:02:13.330 INFO:tasks.rados.rados.0.smithi125.stdout:2111: finishing rollback tid 1 to smithi12556470-4 2021-03-24T22:02:13.331 INFO:tasks.rados.rados.0.smithi125.stdout:2111: finishing rollback tid 2 to smithi12556470-4 2021-03-24T22:02:13.331 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 4 v 504 (ObjNum 625 snap 192 seq_num 625) dirty exists 2021-03-24T22:02:13.331 INFO:tasks.rados.rados.0.smithi125.stdout:2110: finishing write tid 2 to smithi12556470-33 2021-03-24T22:02:13.332 INFO:tasks.rados.rados.0.smithi125.stdout:2110: finishing write tid 3 to smithi12556470-33 2021-03-24T22:02:13.332 INFO:tasks.rados.rados.0.smithi125.stdout:2110: finishing write tid 4 to smithi12556470-33 2021-03-24T22:02:13.332 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 33 v 754 (ObjNum 737 snap 238 seq_num 737) dirty exists 2021-03-24T22:02:13.332 INFO:tasks.rados.rados.0.smithi125.stdout:2110: left oid 33 (ObjNum 737 snap 238 seq_num 737) 2021-03-24T22:02:13.342 INFO:teuthology.orchestra.run.smithi125.stderr:reweighted osd.0 to 1 (10000) 2021-03-24T22:02:13.346 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:13.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:13.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:02:13.451 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:13.504 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:13.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:13.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:02:13.609 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:13.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:13 smithi125 conmon[19936]: cluster 2021-03-24T22 2021-03-24T22:02:13.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:13 smithi125 conmon[19936]: :02:12.295573+0000 mon.a (mon.0) 1573 : cluster [DBG] osdmap e562: 8 total, 8 up, 8 in 2021-03-24T22:02:13.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:13 smithi125 conmon[19936]: audit 2021 2021-03-24T22:02:13.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:13 smithi125 conmon[19936]: -03-24T22:02:12 2021-03-24T22:02:13.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:13 smithi125 conmon[19936]: .611800+0000 mon.a (mon.0) 1574 : 2021-03-24T22:02:13.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:13 smithi125 conmon[19936]: audit [INF] from='client.? v1:172.21.15.125:0/475168767' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2021-03-24T22:02:13.670 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:13.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:13.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:02:13.770 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd unset noscrub 2021-03-24T22:02:13.776 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:13.789 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:13 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:02:13.789 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:13 smithi142 conmon[20125]: 24T22:02:12.295573+0000 mon.a (mon.0) 2021-03-24T22:02:13.790 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:13 smithi142 conmon[20125]: 1573 : cluster [DBG] 2021-03-24T22:02:13.790 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:13 smithi142 conmon[20125]: osdmap e562: 8 total, 8 up, 8 in 2021-03-24T22:02:13.790 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:13 smithi142 conmon[20125]: audit 2021-03-24T 2021-03-24T22:02:13.791 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:13 smithi142 conmon[20125]: 22:02:12.611800+0000 mon.a (mon.0) 1574 : audit [INF] from='client.? v1:172.21.15.125:0/475168767' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2021-03-24T22:02:13.791 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:13 smithi142 conmon[20125]: 2021-03-24T22:02:13.837 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:13.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:13.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:02:13.943 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:14.000 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:14.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:14.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:02:14.104 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:14.163 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:14.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:14.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:02:14.269 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:14.326 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:14.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:14.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:14.335 INFO:tasks.rados.rados.0.smithi125.stdout:2110: done (3 left) 2021-03-24T22:02:14.336 INFO:tasks.rados.rados.0.smithi125.stdout:2111: done (2 left) 2021-03-24T22:02:14.336 INFO:tasks.rados.rados.0.smithi125.stdout:2113: done (1 left) 2021-03-24T22:02:14.336 INFO:tasks.rados.rados.0.smithi125.stdout:2114: done (0 left) 2021-03-24T22:02:14.336 INFO:tasks.rados.rados.0.smithi125.stdout:2115: read oid 49 snap -1 2021-03-24T22:02:14.336 INFO:tasks.rados.rados.0.smithi125.stdout:2115: expect (ObjNum 734 snap 235 seq_num 734) 2021-03-24T22:02:14.337 INFO:tasks.rados.rados.0.smithi125.stdout:2116: write oid 40 current snap is 239 2021-03-24T22:02:14.337 INFO:tasks.rados.rados.0.smithi125.stdout:2116: seq_num 738 ranges {403566=420003,1386064=546918} 2021-03-24T22:02:14.341 INFO:tasks.rados.rados.0.smithi125.stdout:2116: writing smithi12556470-40 from 403566 to 823569 tid 1 2021-03-24T22:02:14.350 INFO:tasks.rados.rados.0.smithi125.stdout:2116: writing smithi12556470-40 from 1386064 to 1932982 tid 2 2021-03-24T22:02:14.350 INFO:tasks.rados.rados.0.smithi125.stdout:2117: write (excl) oid 45 current snap is 239 2021-03-24T22:02:14.350 INFO:tasks.rados.rados.0.smithi125.stdout:2116: finishing write tid 1 to smithi12556470-40 2021-03-24T22:02:14.351 INFO:tasks.rados.rados.0.smithi125.stdout:2117: seq_num 739 ranges {420373=495252,1444546=690576} 2021-03-24T22:02:14.354 INFO:tasks.rados.rados.0.smithi125.stdout:2117: writing smithi12556470-45 from 420373 to 915625 tid 1 2021-03-24T22:02:14.362 INFO:tasks.rados.rados.0.smithi125.stdout:2117: writing smithi12556470-45 from 1444546 to 2135122 tid 2 2021-03-24T22:02:14.362 INFO:tasks.rados.rados.0.smithi125.stdout:2118: snap_create 2021-03-24T22:02:14.363 INFO:tasks.rados.rados.0.smithi125.stdout:2115: expect (ObjNum 734 snap 235 seq_num 734) 2021-03-24T22:02:14.435 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:14.465 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2021-03-24T22:02:14.466 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T22:02:14.476 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:14 smithi142 conmon[20125]: cluster 2021-03-24T22:02:12.897831+0000 2021-03-24T22:02:14.477 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:14 smithi142 conmon[20125]: mgr.y (mgr.14144) 553 : cluster [DBG] pgmap v1006: 168 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 peering, 1 active+undersized+remapped, 3 active+remapped+backfill_toofull, 161 active+clean; 516 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.0 MiB/s rd, 1.2 MiB/s wr, 14 op/s; 115/1716 objects misplaced (6.702%); 15 MiB/s, 34 objects/s recovering 2021-03-24T22:02:14.477 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:14 smithi142 conmon[20125]: cluster 2021-03-24T22:02:12.899679+0000 osd.0 (osd.0) 116 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:02:14.477 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:14 smithi142 conmon[20125]: cluster 2021-03-24T22:02:14.478 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:14 smithi142 conmon[20125]: 2021-03-24T22:02:12.902838+0000 osd.0 ( 2021-03-24T22:02:14.478 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:14 smithi142 conmon[20125]: osd.0) 117 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:02:14.478 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:14 smithi142 conmon[20125]: audit 2021 2021-03-24T22:02:14.478 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:14 smithi142 conmon[20125]: -03-24T22:02:13.325355+0000 mon.a (mon.0 2021-03-24T22:02:14.479 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:14 smithi142 conmon[20125]: ) 1575 : audit [INF] from='client.? v1:172.21.15.125:0/475168767' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2021-03-24T22:02:14.479 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:14 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:02:14.479 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:14 smithi142 conmon[20125]: 22:02:13.325445+0000 mon.a (mon.0) 2021-03-24T22:02:14.479 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:14 smithi142 conmon[20125]: 1576 : cluster [DBG] osdmap e563: 8 total, 8 up, 8 in 2021-03-24T22:02:14.493 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:14.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:14.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:02:14.501 INFO:tasks.rados.rados.0.smithi125.stdout:2116: finishing write tid 2 to smithi12556470-40 2021-03-24T22:02:14.501 INFO:tasks.rados.rados.0.smithi125.stdout:2116: finishing write tid 3 to smithi12556470-40 2021-03-24T22:02:14.501 INFO:tasks.rados.rados.0.smithi125.stdout:2116: finishing write tid 4 to smithi12556470-40 2021-03-24T22:02:14.501 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 40 v 1075 (ObjNum 738 snap 239 seq_num 738) dirty exists 2021-03-24T22:02:14.502 INFO:tasks.rados.rados.0.smithi125.stdout:2116: left oid 40 (ObjNum 738 snap 239 seq_num 738) 2021-03-24T22:02:14.502 INFO:tasks.rados.rados.0.smithi125.stdout:2117: finishing write tid 1 to smithi12556470-45 2021-03-24T22:02:14.502 INFO:tasks.rados.rados.0.smithi125.stdout:2117: finishing write tid 2 to smithi12556470-45 2021-03-24T22:02:14.502 INFO:tasks.rados.rados.0.smithi125.stdout:2117: finishing write tid 3 to smithi12556470-45 2021-03-24T22:02:14.503 INFO:tasks.rados.rados.0.smithi125.stdout:2117: finishing write tid 4 to smithi12556470-45 2021-03-24T22:02:14.503 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 45 v 657 (ObjNum 739 snap 239 seq_num 739) dirty exists 2021-03-24T22:02:14.503 INFO:tasks.rados.rados.0.smithi125.stdout:2117: left oid 45 (ObjNum 739 snap 239 seq_num 739) 2021-03-24T22:02:14.592 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:14 smithi125 conmon[24927]: cluster 2021-03-24T22:02:14.593 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:14 smithi125 conmon[24927]: 2021-03-24T22:02:14.594 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:14 smithi125 conmon[24927]: 2021-03-24T22: 2021-03-24T22:02:14.594 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:14 smithi125 conmon[24927]: 02:12.897831+0000 mgr.y (mgr.14144) 553 : cluster [DBG] pgmap v1006: 168 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 peering, 1 active+undersized+remapped, 3 active+remapped+backfill_toofull, 161 active+clean; 516 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.0 MiB/s rd, 1.2 MiB/s wr, 14 op/s; 115/1716 objects misplaced (6.702%); 15 MiB/s, 34 objects/s recovering 2021-03-24T22:02:14.594 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:14 smithi125 conmon[24927]: cluster 2021-03-24T22:02:14.595 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:14 smithi125 conmon[24927]: 2021-03-24 2021-03-24T22:02:14.595 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:14 smithi125 conmon[24927]: T22:02:12.899679+0000 osd.0 (osd.0) 116 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:02:14.595 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:14 smithi125 conmon[24927]: 2021-03-24T22:02:14.595 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:14 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:02:14.595 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:14 smithi125 conmon[24927]: 24T22:02:12.902838+0000 osd.0 (osd. 2021-03-24T22:02:14.596 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:14 smithi125 conmon[24927]: 0) 117 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:02:14.596 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:14 smithi125 conmon[24927]: audit 2021-03 2021-03-24T22:02:14.596 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:14 smithi125 conmon[24927]: -24T22:02:13.325355+0000 mon.a (mon.0 2021-03-24T22:02:14.596 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:14 smithi125 conmon[24927]: ) 1575 : audit [INF] from='client.? v1:172.21.15.125:0/475168767' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2021-03-24T22:02:14.596 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:14 smithi125 conmon[24927]: cluster 2021-03-24T22 2021-03-24T22:02:14.597 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:14 smithi125 conmon[24927]: :02:13.325445+0000 mon.a (mon.0) 1576 : 2021-03-24T22:02:14.597 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:14 smithi125 conmon[24927]: cluster [DBG] osdmap e563: 8 total, 8 up, 8 in 2021-03-24T22:02:14.597 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:14 smithi125 conmon[19936]: cluster 2021-03-24T22:02:12.897831+0000 2021-03-24T22:02:14.598 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:14 smithi125 conmon[19936]: mgr.y (mgr.14144) 553 : cluster [DBG] pgmap v1006: 168 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 peering, 1 active+undersized+remapped, 3 active+remapped+backfill_toofull, 161 active+clean; 516 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.0 MiB/s rd, 1.2 MiB/s wr, 14 op/s; 115/1716 objects misplaced (6.702%); 15 MiB/s, 34 objects/s recovering 2021-03-24T22:02:14.598 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:14 smithi125 conmon[19936]: cluster 2021-03-24T22:02:12.899679+0000 osd.0 (osd.0) 116 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:02:14.598 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:14 smithi125 conmon[19936]: cluster 2021-03-24T22:02:12.902838+0000 osd.0 (osd.0) 117 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:02:14.598 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:14 smithi125 conmon[19936]: audit 2021-03- 2021-03-24T22:02:14.598 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:14 smithi125 conmon[19936]: 24T22:02:13.325355+0000 mon.a (mon.0) 1575 : audit 2021-03-24T22:02:14.599 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:14 smithi125 conmon[19936]: [INF] from='client.? v1:172.21.15.125:0/475168767' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2021-03-24T22:02:14.599 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:14 smithi125 conmon[19936]: cluster 2021-03-24T22:02:13.325445+0000 mon.a (mon.0 2021-03-24T22:02:14.599 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:14 smithi125 conmon[19936]: ) 1576 : cluster [DBG] osdmap e563: 8 total, 8 up, 8 in 2021-03-24T22:02:14.600 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:14.620 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:02:14.621 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd reweight 2 1 2021-03-24T22:02:14.659 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:14.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:14.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:02:14.764 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:14.824 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:14.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:14.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:02:14.933 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:14.986 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:14.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:14.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:02:15.057 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:02:15.092 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:15.144 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:15.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:15.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:15.249 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:15.305 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:15.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:15.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:02:15.380 INFO:tasks.rados.rados.0.smithi125.stdout:2115: done (3 left) 2021-03-24T22:02:15.381 INFO:tasks.rados.rados.0.smithi125.stdout:2116: done (2 left) 2021-03-24T22:02:15.381 INFO:tasks.rados.rados.0.smithi125.stdout:2117: done (1 left) 2021-03-24T22:02:15.381 INFO:tasks.rados.rados.0.smithi125.stdout:2118: done (0 left) 2021-03-24T22:02:15.382 INFO:tasks.rados.rados.0.smithi125.stdout:2119: snap_remove snap 234 2021-03-24T22:02:15.410 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:15.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:15 smithi125 conmon[19936]: cluster 2021-03-24T22:02:14.327214+0000 mon.a (mon 2021-03-24T22:02:15.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:15 smithi125 conmon[19936]: .0) 1577 : cluster [DBG] osdmap e564: 8 total, 8 up, 8 in 2021-03-24T22:02:15.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:15 smithi125 conmon[24927]: cluster 2021-03-24T22:02 2021-03-24T22:02:15.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:15 smithi125 conmon[24927]: :14.327214+0000 mon.a (mon.0) 1577 : cluster [DBG] osdmap e564: 8 total, 8 up, 8 in 2021-03-24T22:02:15.463 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:15.466 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-24T22:02:15.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:15.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:02:15.511 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:15 smithi142 conmon[20125]: cluster 2021-03-24T22:02:14.327214+0000 mon.a (mon 2021-03-24T22:02:15.511 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:15 smithi142 conmon[20125]: .0) 1577 : cluster [DBG] osdmap e564: 8 total, 8 up, 8 in 2021-03-24T22:02:15.541 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.4 dump_blocked_ops 2021-03-24T22:02:15.571 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:15.624 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:15.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:15.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:02:15.729 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:15.779 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:15.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:15.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:02:15.842 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T22:02:15.885 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:15.946 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:15.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:15.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:02:16.053 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:16.085 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:02:16.107 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:16.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:16.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:02:16.214 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:16.270 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:16.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:16.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:16.375 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:16.395 INFO:tasks.rados.rados.0.smithi125.stdout:2119: done (0 left) 2021-03-24T22:02:16.396 INFO:tasks.rados.rados.0.smithi125.stdout:2120: write oid 12 current snap is 240 2021-03-24T22:02:16.396 INFO:tasks.rados.rados.0.smithi125.stdout:2120: seq_num 740 ranges {437180=570501,1503028=550587} 2021-03-24T22:02:16.396 INFO:tasks.rados.rados.0.smithi125.stdout:2120: writing smithi12556470-12 from 437180 to 1007681 tid 1 2021-03-24T22:02:16.402 INFO:tasks.rados.rados.0.smithi125.stdout:2120: writing smithi12556470-12 from 1503028 to 2053615 tid 2 2021-03-24T22:02:16.402 INFO:tasks.rados.rados.0.smithi125.stdout:2121: write oid 40 current snap is 240 2021-03-24T22:02:16.402 INFO:tasks.rados.rados.0.smithi125.stdout:2121: seq_num 741 ranges {453987=645750,1845157=608830} 2021-03-24T22:02:16.408 INFO:tasks.rados.rados.0.smithi125.stdout:2121: writing smithi12556470-40 from 453987 to 1099737 tid 1 2021-03-24T22:02:16.416 INFO:tasks.rados.rados.0.smithi125.stdout:2121: writing smithi12556470-40 from 1845157 to 2453987 tid 2 2021-03-24T22:02:16.417 INFO:tasks.rados.rados.0.smithi125.stdout:2122: snap_remove snap 239 2021-03-24T22:02:16.428 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:16.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:16.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:02:16.533 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:16.587 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:16.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:16.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:02:16.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:16 smithi142 conmon[20125]: cluster 2021-03-24T22:02:14.898640+0000 mgr.y (mgr.14144) 554 : cluster [DBG] pgmap v1009: 168 pgs: 1 active+clean+remapped, 1 peering, 2 active+remapped+backfill_toofull, 164 active+clean; 508 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 527 KiB/s wr, 14 op/s; 51/1698 objects misplaced (3.004%); 3.2 MiB/s, 15 objects/s recovering 2021-03-24T22:02:16.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:16 smithi142 conmon[20125]: cluster 2021-03-24T22:02:15.367876+0000 mon.a (mon.0) 1578 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:02:16.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:16 smithi142 conmon[20125]: cluster 2021-03-24T22:02:15.377390+0000 mon.a (mon.0) 1579 : cluster [DBG] osdmap e565: 8 total, 8 up, 8 in 2021-03-24T22:02:16.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:16 smithi142 conmon[20125]: cluster 2021-03-24T22:02:16.380898+0000 mon.a (mon.0) 1580 : cluster [DBG] osdmap e566: 8 total, 8 up, 8 in 2021-03-24T22:02:16.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:16 smithi125 conmon[24927]: cluster 2021-03-24T22:02:14. 2021-03-24T22:02:16.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:16 smithi125 conmon[24927]: 898640+0000 mgr.y (mgr.14144) 554 : cluster [DBG] pgmap v1009: 168 pgs: 1 active+clean+remapped, 1 peering, 2 active+remapped+backfill_toofull, 164 active+clean; 508 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 527 KiB/s wr, 14 op/s; 51/1698 objects misplaced (3.004%); 3.2 MiB/s, 15 objects/s recovering 2021-03-24T22:02:16.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:16 smithi125 conmon[24927]: cluster 2021-03-24T22:02:15.367876+0000 mon.a 2021-03-24T22:02:16.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:16 smithi125 conmon[24927]: (mon.0) 1578 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:02:16.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:16 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T22:02:16.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:16 smithi125 conmon[24927]: -24T22:02:15.377390+0000 mon.a (mon 2021-03-24T22:02:16.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:16 smithi125 conmon[24927]: .0) 1579 : cluster [DBG] 2021-03-24T22:02:16.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:16 smithi125 conmon[24927]: osdmap e565: 8 total, 8 up, 8 in 2021-03-24T22:02:16.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:16 smithi125 conmon[24927]: cluster 2021-03-24T22:02:16.380898+0000 2021-03-24T22:02:16.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:16 smithi125 conmon[24927]: mon.a (mon.0) 1580 : cluster [DBG] osdmap e566: 8 total, 8 up, 8 in 2021-03-24T22:02:16.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:16 smithi125 conmon[24927]: 2021-03-24T22:02:16.657 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:16 smithi125 conmon[19936]: cluster 2021-03-24T22:02:16.657 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:16 smithi125 conmon[19936]: 2021-03-24T22:02:16.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:16 smithi125 conmon[19936]: 2021 2021-03-24T22:02:16.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:16 smithi125 conmon[19936]: - 2021-03-24T22:02:16.658 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:16 smithi125 conmon[19936]: 03 2021-03-24T22:02:16.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:16 smithi125 conmon[19936]: - 2021-03-24T22:02:16.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:16 smithi125 conmon[19936]: 24 2021-03-24T22:02:16.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:16 smithi125 conmon[19936]: T 2021-03-24T22:02:16.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:16 smithi125 conmon[19936]: 22 2021-03-24T22:02:16.659 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:16 smithi125 conmon[19936]: : 2021-03-24T22:02:16.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:16 smithi125 conmon[19936]: 02 2021-03-24T22:02:16.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:16 smithi125 conmon[19936]: : 2021-03-24T22:02:16.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:16 smithi125 conmon[19936]: 14 2021-03-24T22:02:16.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:16 smithi125 conmon[19936]: . 2021-03-24T22:02:16.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:16 smithi125 conmon[19936]: 898640 2021-03-24T22:02:16.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:16 smithi125 conmon[19936]: +0000 2021-03-24T22:02:16.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:16 smithi125 conmon[19936]: 2021-03-24T22:02:16.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:16 smithi125 conmon[19936]: mgr.y 2021-03-24T22:02:16.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:16 smithi125 conmon[19936]: ( 2021-03-24T22:02:16.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:16 smithi125 conmon[19936]: mgr 2021-03-24T22:02:16.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:16 smithi125 conmon[19936]: . 2021-03-24T22:02:16.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:16 smithi125 conmon[19936]: 14144 2021-03-24T22:02:16.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:16 smithi125 conmon[19936]: ) 2021-03-24T22:02:16.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:16 smithi125 conmon[19936]: 554 2021-03-24T22:02:16.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:16 smithi125 conmon[19936]: : 2021-03-24T22:02:16.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:16 smithi125 conmon[19936]: cluster 2021-03-24T22:02:16.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:16 smithi125 conmon[19936]: 2021-03-24T22:02:16.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:16 smithi125 conmon[19936]: [DBG] 2021-03-24T22:02:16.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:16 smithi125 conmon[19936]: 2021-03-24T22:02:16.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:16 smithi125 conmon[19936]: pgmap v1009: 168 pgs: 1 active+clean+remapped, 1 peering, 2 active+remapped+backfill_toofull, 164 active+clean; 508 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 527 KiB/s wr, 14 op/s; 51/1698 objects misplaced (3.004%); 3.2 MiB/s, 15 objects/s recovering 2021-03-24T22:02:16.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:16 smithi125 conmon[19936]: cluster 2021-03-24T22:02:15.367876+0000 mon.a (mon.0) 1578 : cluster 2021-03-24T22:02:16.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:16 smithi125 conmon[19936]: [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:02:16.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:16 smithi125 conmon[19936]: cluster 2021-03-24T22 2021-03-24T22:02:16.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:16 smithi125 conmon[19936]: :02:15.377390 2021-03-24T22:02:16.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:16 smithi125 conmon[19936]: +0000 2021-03-24T22:02:16.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:16 smithi125 conmon[19936]: mon.a (mon.0) 1579 : cluster [DBG] 2021-03-24T22:02:16.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:16 smithi125 conmon[19936]: osdmap e565: 8 total, 8 up, 8 in 2021-03-24T22:02:16.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:16 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T22:02:16.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:16 smithi125 conmon[19936]: -24T22:02:16.380898+0000 mon.a (mon.0) 1580 : cluster 2021-03-24T22:02:16.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:16 smithi125 conmon[19936]: [DBG] osdmap e566: 8 total, 8 up, 8 in 2021-03-24T22:02:16.692 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:16.747 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:16.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:16.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:16.856 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:16.918 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:16.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:16.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:02:17.024 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:17.076 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:17.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:17.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:17.182 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:17.236 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:17.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:17.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:02:17.342 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:17.393 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:17.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:17.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:02:17.404 INFO:tasks.rados.rados.0.smithi125.stdout:2122: done (2 left) 2021-03-24T22:02:17.405 INFO:tasks.rados.rados.0.smithi125.stdout:2123: snap_remove snap 225 2021-03-24T22:02:17.420 INFO:teuthology.orchestra.run.smithi125.stderr:noscrub is unset 2021-03-24T22:02:17.500 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:17.514 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:17 smithi125 conmon[24927]: cluster 2021-03-24T22:02:17.514 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:17 smithi125 conmon[24927]: 2021-03-24T22:02:17.514 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:17 smithi125 conmon[24927]: 2021 2021-03-24T22:02:17.514 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:17 smithi125 conmon[24927]: - 2021-03-24T22:02:17.515 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:17 smithi125 conmon[24927]: 03 2021-03-24T22:02:17.515 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:17 smithi125 conmon[24927]: - 2021-03-24T22:02:17.515 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:17 smithi125 conmon[24927]: 24 2021-03-24T22:02:17.515 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:17 smithi125 conmon[24927]: T 2021-03-24T22:02:17.516 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:17 smithi125 conmon[24927]: 22 2021-03-24T22:02:17.516 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:17 smithi125 conmon[24927]: : 2021-03-24T22:02:17.516 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:17 smithi125 conmon[24927]: 02:16.647405+0000 osd.3 (osd.3) 116 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:02:17.516 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:17 smithi125 conmon[24927]: 2021-03-24T22:02:17.516 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:17 smithi125 conmon[24927]: cluster 2021-03-24T22:02:16.650804+0000 osd.3 (osd.3) 117 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:02:17.517 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:17 smithi125 conmon[24927]: audit 2021-03-24T22:02:17.289779+0000 mon.c (mon.2) 86 : audit [INF] from='client.? v1:172.21.15.125:0/1667668199' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T22:02:17.517 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:17 smithi125 conmon[24927]: audit 2021-03-24T22:02:17.290402+0000 mon.a (mon.0) 1581 : audit [INF] from='client.35447 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T22:02:17.518 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:17 smithi125 conmon[19936]: cluster 2021-03-24T22:02:17.518 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:17 smithi125 conmon[19936]: 2021-03-24T22:02:17.518 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:17 smithi125 conmon[19936]: 2021 2021-03-24T22:02:17.518 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:17 smithi125 conmon[19936]: - 2021-03-24T22:02:17.518 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:17 smithi125 conmon[19936]: 03 2021-03-24T22:02:17.519 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:17 smithi125 conmon[19936]: - 2021-03-24T22:02:17.519 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:17 smithi125 conmon[19936]: 24 2021-03-24T22:02:17.519 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:17 smithi125 conmon[19936]: T 2021-03-24T22:02:17.519 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:17 smithi125 conmon[19936]: 22 2021-03-24T22:02:17.519 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:17 smithi125 conmon[19936]: : 2021-03-24T22:02:17.521 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:17 smithi125 conmon[19936]: 02 2021-03-24T22:02:17.521 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:17 smithi125 conmon[19936]: : 2021-03-24T22:02:17.521 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:17 smithi125 conmon[19936]: 16 2021-03-24T22:02:17.521 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:17 smithi125 conmon[19936]: .647405+0000 osd.3 (osd. 2021-03-24T22:02:17.522 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:17 smithi125 conmon[19936]: 3) 116 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:02:17.522 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:17 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:02:17.522 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:17 smithi125 conmon[19936]: 24T22:02:16.650804+0000 osd.3 (osd.3 2021-03-24T22:02:17.522 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:17 smithi125 conmon[19936]: ) 117 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:02:17.523 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:17 smithi125 conmon[19936]: audit 2021-03-24 2021-03-24T22:02:17.523 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:17 smithi125 conmon[19936]: T22:02:17.289779+0000 mon.c (mon.2 2021-03-24T22:02:17.523 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:17 smithi125 conmon[19936]: ) 86 : audit [INF] from='client.? v1:172.21.15.125:0/1667668199' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T22:02:17.523 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:17 smithi125 conmon[19936]: audit 2021-03-24T 2021-03-24T22:02:17.523 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:17 smithi125 conmon[19936]: 22:02:17.290402+0000 mon.a (mon. 2021-03-24T22:02:17.524 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:17 smithi125 conmon[19936]: 0) 1581 : audit [INF] from='client.35447 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T22:02:17.556 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:17.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:17.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:17.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:17 smithi142 conmon[20125]: cluster 2021-03-24T22:02:16.647405+0000 2021-03-24T22:02:17.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:17 smithi142 conmon[20125]: osd.3 (osd.3) 116 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:02:17.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:17 smithi142 conmon[20125]: cluster 2021-03-24T22:02:16.650804+0000 osd.3 2021-03-24T22:02:17.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:17 smithi142 conmon[20125]: (osd.3) 117 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:02:17.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:17 smithi142 conmon[20125]: audit 2021-03-24T22:02: 2021-03-24T22:02:17.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:17 smithi142 conmon[20125]: 17.289779+0000 mon.c (mon.2) 86 : audit 2021-03-24T22:02:17.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:17 smithi142 conmon[20125]: [INF] from='client.? v1:172.21.15.125:0/1667668199' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T22:02:17.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:17 smithi142 conmon[20125]: audit 2021-03-24T22 2021-03-24T22:02:17.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:17 smithi142 conmon[20125]: :02:17.290402+0000 mon.a (mon.0) 2021-03-24T22:02:17.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:17 smithi142 conmon[20125]: 1581 : audit [INF] from='client.35447 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T22:02:17.663 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:17.720 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:17.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:17.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:02:17.827 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:17.879 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:17.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:17.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:02:17.896 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2021-03-24T22:02:17.897 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T22:02:17.986 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:18.047 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:18.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:18.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:02:18.153 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:18.214 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:18.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:18.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:18.321 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:18.383 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:18.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:18.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:02:18.418 INFO:tasks.rados.rados.0.smithi125.stdout:2123: done (2 left) 2021-03-24T22:02:18.418 INFO:tasks.rados.rados.0.smithi125.stdout:2124: read oid 20 snap -1 2021-03-24T22:02:18.419 INFO:tasks.rados.rados.0.smithi125.stdout:2124: expect (ObjNum 715 snap 227 seq_num 715) 2021-03-24T22:02:18.419 INFO:tasks.rados.rados.0.smithi125.stdout:2125: write oid 24 current snap is 240 2021-03-24T22:02:18.419 INFO:tasks.rados.rados.0.smithi125.stdout:2125: seq_num 742 ranges {470794=720999,1903639=554256} 2021-03-24T22:02:18.421 INFO:tasks.rados.rados.0.smithi125.stdout:2125: writing smithi12556470-24 from 470794 to 1191793 tid 1 2021-03-24T22:02:18.431 INFO:tasks.rados.rados.0.smithi125.stdout:2125: writing smithi12556470-24 from 1903639 to 2457895 tid 2 2021-03-24T22:02:18.431 INFO:tasks.rados.rados.0.smithi125.stdout:2120: finishing write tid 1 to smithi12556470-12 2021-03-24T22:02:18.431 INFO:tasks.rados.rados.0.smithi125.stdout:2120: finishing write tid 2 to smithi12556470-12 2021-03-24T22:02:18.431 INFO:tasks.rados.rados.0.smithi125.stdout:2120: finishing write tid 3 to smithi12556470-12 2021-03-24T22:02:18.432 INFO:tasks.rados.rados.0.smithi125.stdout:2120: finishing write tid 4 to smithi12556470-12 2021-03-24T22:02:18.432 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 12 v 483 (ObjNum 740 snap 240 seq_num 740) dirty exists 2021-03-24T22:02:18.432 INFO:tasks.rados.rados.0.smithi125.stdout:2120: left oid 12 (ObjNum 740 snap 240 seq_num 740) 2021-03-24T22:02:18.432 INFO:tasks.rados.rados.0.smithi125.stdout:2121: finishing write tid 1 to smithi12556470-40 2021-03-24T22:02:18.433 INFO:tasks.rados.rados.0.smithi125.stdout:2121: finishing write tid 2 to smithi12556470-40 2021-03-24T22:02:18.433 INFO:tasks.rados.rados.0.smithi125.stdout:2121: finishing write tid 3 to smithi12556470-40 2021-03-24T22:02:18.433 INFO:tasks.rados.rados.0.smithi125.stdout:2121: finishing write tid 4 to smithi12556470-40 2021-03-24T22:02:18.433 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 40 v 1079 (ObjNum 741 snap 240 seq_num 741) dirty exists 2021-03-24T22:02:18.434 INFO:tasks.rados.rados.0.smithi125.stdout:2121: left oid 40 (ObjNum 741 snap 240 seq_num 741) 2021-03-24T22:02:18.434 INFO:tasks.rados.rados.0.smithi125.stdout:2120: done (3 left) 2021-03-24T22:02:18.434 INFO:tasks.rados.rados.0.smithi125.stdout:2121: done (2 left) 2021-03-24T22:02:18.435 INFO:tasks.rados.rados.0.smithi125.stdout:2126: copy_from oid 36 from oid 19 current snap is 240 2021-03-24T22:02:18.435 INFO:tasks.rados.rados.0.smithi125.stdout:2127: write oid 46 current snap is 240 2021-03-24T22:02:18.435 INFO:tasks.rados.rados.0.smithi125.stdout:2127: seq_num 743 ranges {487601=796248,1962121=525480} 2021-03-24T22:02:18.439 INFO:tasks.rados.rados.0.smithi125.stdout:2127: writing smithi12556470-46 from 487601 to 1283849 tid 1 2021-03-24T22:02:18.446 INFO:tasks.rados.rados.0.smithi125.stdout:2127: writing smithi12556470-46 from 1962121 to 2487601 tid 2 2021-03-24T22:02:18.446 INFO:tasks.rados.rados.0.smithi125.stdout:2128: rollback oid 50 current snap is 240 2021-03-24T22:02:18.446 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 50 to 168 2021-03-24T22:02:18.447 INFO:tasks.rados.rados.0.smithi125.stdout:2124: expect (ObjNum 715 snap 227 seq_num 715) 2021-03-24T22:02:18.488 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:18.533 INFO:tasks.rados.rados.0.smithi125.stdout:2126: finishing copy_from to smithi12556470-36 2021-03-24T22:02:18.533 INFO:tasks.rados.rados.0.smithi125.stdout:2126: got expected ENOENT (src dne) 2021-03-24T22:02:18.533 INFO:tasks.rados.rados.0.smithi125.stdout:2126: finishing copy_from racing read to smithi12556470-36 2021-03-24T22:02:18.534 INFO:tasks.rados.rados.0.smithi125.stdout:2125: finishing write tid 1 to smithi12556470-24 2021-03-24T22:02:18.534 INFO:tasks.rados.rados.0.smithi125.stdout:2125: finishing write tid 2 to smithi12556470-24 2021-03-24T22:02:18.535 INFO:tasks.rados.rados.0.smithi125.stdout:2125: finishing write tid 3 to smithi12556470-24 2021-03-24T22:02:18.535 INFO:tasks.rados.rados.0.smithi125.stdout:2125: finishing write tid 4 to smithi12556470-24 2021-03-24T22:02:18.535 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 24 v 767 (ObjNum 742 snap 240 seq_num 742) dirty exists 2021-03-24T22:02:18.536 INFO:tasks.rados.rados.0.smithi125.stdout:2125: left oid 24 (ObjNum 742 snap 240 seq_num 742) 2021-03-24T22:02:18.536 INFO:tasks.rados.rados.0.smithi125.stdout:2127: finishing write tid 1 to smithi12556470-46 2021-03-24T22:02:18.536 INFO:tasks.rados.rados.0.smithi125.stdout:2128: finishing rollback tid 0 to smithi12556470-50 2021-03-24T22:02:18.537 INFO:tasks.rados.rados.0.smithi125.stdout:2128: finishing rollback tid 1 to smithi12556470-50 2021-03-24T22:02:18.538 INFO:tasks.rados.rados.0.smithi125.stdout:2128: finishing rollback tid 2 to smithi12556470-50 2021-03-24T22:02:18.538 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 50 v 719 (ObjNum 514 snap 151 seq_num 514) dirty exists 2021-03-24T22:02:18.539 INFO:tasks.rados.rados.0.smithi125.stdout:2127: finishing write tid 2 to smithi12556470-46 2021-03-24T22:02:18.539 INFO:tasks.rados.rados.0.smithi125.stdout:2127: finishing write tid 3 to smithi12556470-46 2021-03-24T22:02:18.540 INFO:tasks.rados.rados.0.smithi125.stdout:2127: finishing write tid 4 to smithi12556470-46 2021-03-24T22:02:18.540 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 46 v 501 (ObjNum 743 snap 240 seq_num 743) dirty exists 2021-03-24T22:02:18.540 INFO:tasks.rados.rados.0.smithi125.stdout:2127: left oid 46 (ObjNum 743 snap 240 seq_num 743) 2021-03-24T22:02:18.541 INFO:tasks.rados.rados.0.smithi125.stdout:2124: done (4 left) 2021-03-24T22:02:18.541 INFO:tasks.rados.rados.0.smithi125.stdout:2125: done (3 left) 2021-03-24T22:02:18.541 INFO:tasks.rados.rados.0.smithi125.stdout:2126: done (2 left) 2021-03-24T22:02:18.542 INFO:tasks.rados.rados.0.smithi125.stdout:2127: done (1 left) 2021-03-24T22:02:18.542 INFO:tasks.rados.rados.0.smithi125.stdout:2128: done (0 left) 2021-03-24T22:02:18.542 INFO:tasks.rados.rados.0.smithi125.stdout:2129: write oid 17 current snap is 240 2021-03-24T22:02:18.542 INFO:tasks.rados.rados.0.smithi125.stdout:2129: seq_num 744 ranges {504408=471497,1620603=557925} 2021-03-24T22:02:18.544 INFO:tasks.rados.rados.0.smithi125.stdout:2129: writing smithi12556470-17 from 504408 to 975905 tid 1 2021-03-24T22:02:18.552 INFO:tasks.rados.rados.0.smithi125.stdout:2129: writing smithi12556470-17 from 1620603 to 2178528 tid 2 2021-03-24T22:02:18.552 INFO:tasks.rados.rados.0.smithi125.stdout:2130: copy_from oid 19 from oid 9 current snap is 240 2021-03-24T22:02:18.552 INFO:tasks.rados.rados.0.smithi125.stdout:2129: finishing write tid 1 to smithi12556470-17 2021-03-24T22:02:18.553 INFO:tasks.rados.rados.0.smithi125.stdout:2131: snap_create 2021-03-24T22:02:18.553 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:18.557 INFO:tasks.rados.rados.0.smithi125.stdout:2129: finishing write tid 2 to smithi12556470-17 2021-03-24T22:02:18.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:18.560 INFO:tasks.rados.rados.0.smithi125.stdout:2129: finishing write tid 3 to smithi12556470-17 2021-03-24T22:02:18.560 INFO:tasks.rados.rados.0.smithi125.stdout:2129: finishing write tid 4 to smithi12556470-17 2021-03-24T22:02:18.561 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 17 v 632 (ObjNum 744 snap 240 seq_num 744) dirty exists 2021-03-24T22:02:18.561 INFO:tasks.rados.rados.0.smithi125.stdout:2129: left oid 17 (ObjNum 744 snap 240 seq_num 744) 2021-03-24T22:02:18.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:02:18.575 INFO:tasks.rados.rados.0.smithi125.stdout:2130: finishing copy_from to smithi12556470-19 2021-03-24T22:02:18.576 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 19 v 239 (ObjNum 588 snap 177 seq_num 588) dirty exists 2021-03-24T22:02:18.576 INFO:tasks.rados.rados.0.smithi125.stdout:2130: finishing copy_from racing read to smithi12556470-19 2021-03-24T22:02:18.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:18 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:02:18.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:18 smithi125 conmon[19936]: 02:16.899635+0000 mgr.y (mgr.14144) 555 : cluster [DBG] pgmap v1012: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 509 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.9 MiB/s rd, 535 KiB/s wr, 11 op/s; 51/1698 objects misplaced (3.004%); 3.2 MiB/s, 10 objects/s recovering 2021-03-24T22:02:18.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:18 smithi125 conmon[19936]: cluster 2021-03-24T22:02 2021-03-24T22:02:18.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:18 smithi125 conmon[19936]: :17.387959+0000 mon.a (mon.0) 1582 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:02:18.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:18 smithi125 conmon[19936]: audit 2021-03-24T22:02 2021-03-24T22:02:18.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:18 smithi125 conmon[19936]: :17.402111+0000 mon.a (mon.0) 1583 2021-03-24T22:02:18.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:18 smithi125 conmon[19936]: : audit [INF] from='client.35447 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T22:02:18.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:18 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:02:18.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:18 smithi125 conmon[19936]: T22:02:17.402234+0000 mon.a ( 2021-03-24T22:02:18.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:18 smithi125 conmon[19936]: mon.0) 1584 : cluster [DBG] osdmap e567: 8 total, 8 up, 8 in 2021-03-24T22:02:18.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:18 smithi125 conmon[19936]: 2021-03-24T22:02:18.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:18 smithi125 conmon[19936]: cluster 2021-03-24T22:02:18.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:18 smithi125 conmon[19936]: 2021-03-24T22:02:17.672316+0000 osd.3 (osd.3) 118 : 2021-03-24T22:02:18.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:18 smithi125 conmon[19936]: cluster [DBG] 5.1c scrub ok 2021-03-24T22:02:18.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:18 smithi125 conmon[24927]: cluster 2021-03-24T22:02:16. 2021-03-24T22:02:18.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:18 smithi125 conmon[24927]: 899635+0000 mgr.y (mgr.14144) 555 : cluster [DBG] pgmap v1012: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 509 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.9 MiB/s rd, 535 KiB/s wr, 11 op/s; 51/1698 objects misplaced (3.004%); 3.2 MiB/s, 10 objects/s recovering 2021-03-24T22:02:18.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:18 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:02:18.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:18 smithi125 conmon[24927]: -03-24T22:02:17.387959+0000 mon.a (mon.0) 1582 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:02:18.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:18 smithi125 conmon[24927]: 2021-03-24T22:02:18.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:18 smithi125 conmon[24927]: audit 2021-03 2021-03-24T22:02:18.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:18 smithi125 conmon[24927]: -24T22:02:17.402111+0000 mon.a (mon.0) 1583 : audit 2021-03-24T22:02:18.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:18 smithi125 conmon[24927]: [INF] from='client.35447 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T22:02:18.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:18 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:02:18.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:18 smithi125 conmon[24927]: -03-24T22:02:17.402234+0000 mon.a (mon.0 2021-03-24T22:02:18.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:18 smithi125 conmon[24927]: ) 1584 : cluster [DBG] osdmap e567: 8 total, 8 up, 8 in 2021-03-24T22:02:18.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:18 smithi125 conmon[24927]: cluster 2021-03-24T22:02: 2021-03-24T22:02:18.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:18 smithi125 conmon[24927]: 17.672316+0000 osd.3 (osd.3) 118 : cluster [DBG] 5.1c scrub ok 2021-03-24T22:02:18.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:18 smithi125 conmon[24927]: 2021-03-24T22:02:18.669 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:18.728 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:18.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:18.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:18.835 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:18.841 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:18 smithi142 conmon[20125]: cluster 2021-03-24T22 2021-03-24T22:02:18.841 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:18 smithi142 conmon[20125]: :02:16.899635+0000 mgr.y (mgr.14144) 555 : cluster [DBG] pgmap v1012: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 509 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.9 MiB/s rd, 535 KiB/s wr, 11 op/s; 51/1698 objects misplaced (3.004%); 3.2 MiB/s, 10 objects/s recovering 2021-03-24T22:02:18.842 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:18 smithi142 conmon[20125]: cluster 2021-03-24T22:02:18.842 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:18 smithi142 conmon[20125]: 2021-03-24T 2021-03-24T22:02:18.842 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:18 smithi142 conmon[20125]: 22:02:17.387959+0000 2021-03-24T22:02:18.842 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:18 smithi142 conmon[20125]: mon.a (mon.0) 2021-03-24T22:02:18.843 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:18 smithi142 conmon[20125]: 1582 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:02:18.843 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:18 smithi142 conmon[20125]: audit 2021-03-24T22:02:17.402111+0000 mon.a 2021-03-24T22:02:18.843 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:18 smithi142 conmon[20125]: (mon.0) 1583 : audit [INF] from='client.35447 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T22:02:18.843 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:18 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:02:18.844 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:18 smithi142 conmon[20125]: 03-24T22:02: 2021-03-24T22:02:18.844 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:18 smithi142 conmon[20125]: 17.402234+0000 mon.a (mon.0) 1584 2021-03-24T22:02:18.844 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:18 smithi142 conmon[20125]: : cluster [DBG] osdmap e567: 8 total, 8 up, 8 in 2021-03-24T22:02:18.844 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:18 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:02:18.844 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:18 smithi142 conmon[20125]: -03-24T22:02:17.672316+0000 osd.3 2021-03-24T22:02:18.845 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:18 smithi142 conmon[20125]: (osd.3) 118 : cluster [DBG] 2021-03-24T22:02:18.845 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:18 smithi142 conmon[20125]: 5.1c scrub ok 2021-03-24T22:02:18.892 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:18.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:18.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:18.934 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.4 dump_historic_ops 2021-03-24T22:02:18.997 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:19.049 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:19.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:19.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:19.155 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:19.205 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:19.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:19.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:02:19.223 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T22:02:19.310 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:19.360 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:19.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:19.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:02:19.424 INFO:tasks.rados.rados.0.smithi125.stdout:2129: done (2 left) 2021-03-24T22:02:19.424 INFO:tasks.rados.rados.0.smithi125.stdout:2130: done (1 left) 2021-03-24T22:02:19.424 INFO:tasks.rados.rados.0.smithi125.stdout:2131: done (0 left) 2021-03-24T22:02:19.424 INFO:tasks.rados.rados.0.smithi125.stdout:2132: delete oid 15 current snap is 241 2021-03-24T22:02:19.429 INFO:tasks.rados.rados.0.smithi125.stdout:2132: done (0 left) 2021-03-24T22:02:19.430 INFO:tasks.rados.rados.0.smithi125.stdout:2133: snap_remove snap 187 2021-03-24T22:02:19.465 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:19.520 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:19.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:19.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:02:19.627 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:19.642 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:19 smithi125 conmon[24927]: cluster 2021-03-24T22:02:18.082504+0000 osd.5 (osd.5) 25 : cluster [DBG] 3.16 scrub ok 2021-03-24T22:02:19.642 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:19 smithi125 conmon[24927]: cluster 2021-03-24T22:02:18.410527+0000 mon.a (mon.0) 1585 : cluster [DBG] osdmap e568: 8 total, 8 up, 8 in 2021-03-24T22:02:19.642 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:19 smithi125 conmon[24927]: cluster 2021-03-24T22:02:18.706837+0000 osd.1 (osd.1) 51 : cluster [DBG] 4.12 scrub ok 2021-03-24T22:02:19.643 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:19 smithi125 conmon[24927]: cluster 2021-03-24T22:02:18.728262+0000 osd.2 (osd.2) 88 : cluster [DBG] 5.d scrub ok 2021-03-24T22:02:19.643 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:19 smithi125 conmon[19936]: cluster 2021-03-24T22:02 2021-03-24T22:02:19.643 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:19 smithi125 conmon[19936]: :18.082504+0000 osd.5 (osd.5) 25 : cluster [DBG] 3.16 scrub ok 2021-03-24T22:02:19.643 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:19 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T22:02:19.644 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:19 smithi125 conmon[19936]: -24T22:02:18.410527+0000 mon.a (mon.0) 1585 : cluster [DBG] 2021-03-24T22:02:19.644 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:19 smithi125 conmon[19936]: osdmap e568: 8 total, 8 up, 8 in 2021-03-24T22:02:19.644 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:19 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T22:02:19.644 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:19 smithi125 conmon[19936]: -24T22:02:18.706837+0000 osd.1 (osd.1) 2021-03-24T22:02:19.645 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:19 smithi125 conmon[19936]: 51 : cluster [DBG] 4.12 scrub ok 2021-03-24T22:02:19.645 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:19 smithi125 conmon[19936]: cluster 2021-03-24T22:02:18 2021-03-24T22:02:19.645 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:19 smithi125 conmon[19936]: .728262+0000 osd.2 (osd.2) 88 : cluster [DBG] 5.d scrub ok 2021-03-24T22:02:19.645 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:19 smithi125 conmon[19936]: 2021-03-24T22:02:19.672 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:19 smithi142 conmon[20125]: cluster 2021-03-24T22 2021-03-24T22:02:19.672 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:19 smithi142 conmon[20125]: :02:18.082504+0000 osd.5 (osd.5) 25 : cluster [DBG] 3.16 scrub ok 2021-03-24T22:02:19.673 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:19 smithi142 conmon[20125]: cluster 2021-03-24T22:02:18 2021-03-24T22:02:19.673 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:19 smithi142 conmon[20125]: .410527+0000 mon.a (mon.0 2021-03-24T22:02:19.673 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:19 smithi142 conmon[20125]: ) 1585 : cluster [DBG] osdmap e568: 8 total, 8 up, 8 in 2021-03-24T22:02:19.673 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:19 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:02:19.674 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:19 smithi142 conmon[20125]: -03-24T22:02:18.706837+0000 osd.1 ( 2021-03-24T22:02:19.674 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:19 smithi142 conmon[20125]: osd.1) 51 : cluster [DBG] 4.12 scrub ok 2021-03-24T22:02:19.674 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:19 smithi142 conmon[20125]: cluster 2021-03-24T22: 2021-03-24T22:02:19.674 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:19 smithi142 conmon[20125]: 02:18.728262+0000 osd.2 (osd.2) 88 : cluster [DBG] 5.d scrub ok 2021-03-24T22:02:19.685 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:19.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:19.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:19.790 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:19.850 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:19.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:19.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:19.955 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:20.009 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:20.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:20.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:02:20.116 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:20.176 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:20.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:20.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:02:20.281 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:20.301 INFO:tasks.rados.rados.0.smithi125.stdout:2133: done (0 left) 2021-03-24T22:02:20.301 INFO:tasks.rados.rados.0.smithi125.stdout:2134: snap_remove snap 217 2021-03-24T22:02:20.346 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:20.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:20.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:02:20.380 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T22:02:20.402 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T22:02:20.422 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:20 smithi125 conmon[19936]: cluster 2021-03-24T22:02:18.900467 2021-03-24T22:02:20.422 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:20 smithi125 conmon[19936]: +0000 mgr.y (mgr.14144) 556 : cluster [DBG] pgmap v1015: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 509 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 290 KiB/s wr, 1 op/s; 51/1698 objects misplaced (3.004%) 2021-03-24T22:02:20.423 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:20 smithi125 conmon[19936]: cluster 2021-03-24T22:02:19.420176+0000 mon.a (mon.0) 1586 2021-03-24T22:02:20.423 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:20 smithi125 conmon[19936]: : cluster [DBG] osdmap e569: 8 total, 8 up, 8 in 2021-03-24T22:02:20.423 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:20 smithi125 conmon[19936]: cluster 2021-03-24T22:02:19.457967 2021-03-24T22:02:20.424 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:20 smithi125 conmon[19936]: +0000 osd.0 (osd.0) 118 : cluster [DBG] 6.18 scrub ok 2021-03-24T22:02:20.424 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:20 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:02:20.424 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:20 smithi125 conmon[19936]: 03-24T22:02:19.684722+0000 osd.2 (osd.2) 89 : cluster [DBG] 3.1d scrub ok 2021-03-24T22:02:20.425 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:20 smithi125 conmon[19936]: cluster 2021-03-24T22:02:19.755642+0000 osd.6 (osd.6) 36 : cluster 2021-03-24T22:02:20.425 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:20 smithi125 conmon[19936]: [DBG] 7.7 scrub ok 2021-03-24T22:02:20.425 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:20 smithi125 conmon[19936]: audit 2021-03-24 2021-03-24T22:02:20.425 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:20 smithi125 conmon[19936]: T22:02:19.819199+0000 mon.a (mon.0) 1587 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:02:20.426 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:20 smithi125 conmon[19936]: 2021-03-24T22:02:20.426 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:20 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:02:20.426 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:20 smithi125 conmon[19936]: 24T22:02:20.296486+0000 mon.a (mon.0) 1588 : cluster [DBG] osdmap e570: 8 total, 8 up, 8 in 2021-03-24T22:02:20.426 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:20 smithi125 conmon[19936]: audit 2021-03-24T22 2021-03-24T22:02:20.427 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:20 smithi125 conmon[19936]: :02:20.355143+0000 mon.c (mon.2) 87 : audit [DBG] 2021-03-24T22:02:20.427 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:20 smithi125 conmon[19936]: from='client.? v1:172.21.15.125:0/3649417752' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:02:20.428 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:20 smithi125 conmon[24927]: cluster 2021-03-24T22:02:20.428 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:20 smithi125 conmon[24927]: 2021-03-24T22 2021-03-24T22:02:20.429 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:20 smithi125 conmon[24927]: :02:18.900467+0000 mgr.y (mgr.14144) 556 : cluster [DBG] pgmap v1015: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 509 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 290 KiB/s wr, 1 op/s; 51/1698 objects misplaced (3.004%) 2021-03-24T22:02:20.429 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:20 smithi125 conmon[24927]: cluster 2021-03-24T22:02:19.420176+0000 mon.a (mon.0) 2021-03-24T22:02:20.429 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:20 smithi125 conmon[24927]: 1586 : cluster [DBG] osdmap e569: 8 total, 8 up, 8 in 2021-03-24T22:02:20.429 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:20 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T22:02:20.430 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:20 smithi125 conmon[24927]: -24T22:02:19.457967+0000 osd.0 (osd.0) 118 : cluster [DBG] 6.18 scrub ok 2021-03-24T22:02:20.430 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:20 smithi125 conmon[24927]: cluster 2021-03-24T22:02:19.684722+0000 osd.2 (osd. 2021-03-24T22:02:20.430 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:20 smithi125 conmon[24927]: 2) 89 : cluster [DBG] 3.1d scrub ok 2021-03-24T22:02:20.430 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:20 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T22:02:20.431 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:20 smithi125 conmon[24927]: -24T22:02:19.755642+0000 osd.6 (osd.6) 36 : cluster [DBG] 7.7 scrub ok 2021-03-24T22:02:20.431 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:20 smithi125 conmon[24927]: audit 2021-03-24T22:02:19.819199+0000 mon.a (mon. 2021-03-24T22:02:20.431 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:20 smithi125 conmon[24927]: 0) 1587 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:02:20.431 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:20 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:02:20.432 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:20 smithi125 conmon[24927]: 03-24T22:02:20.296486+0000 mon.a (mon.0) 1588 : cluster [DBG] osdmap e570: 8 total, 8 up, 8 in 2021-03-24T22:02:20.432 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:20 smithi125 conmon[24927]: audit 2021-03-24T22 2021-03-24T22:02:20.432 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:20 smithi125 conmon[24927]: :02:20.355143+0000 mon.c (mon.2) 87 2021-03-24T22:02:20.432 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:20 smithi125 conmon[24927]: : audit [DBG] from='client.? v1:172.21.15.125:0/3649417752' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:02:20.433 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T22:02:20.452 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:20.453 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T22:02:20.481 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T22:02:20.496 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:20.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:20.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:02:20.508 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T22:02:20.536 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T22:02:20.560 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T22:02:20.562 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-24T22:02:20.562 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-24T22:02:20.562 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-24T22:02:20.563 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-24T22:02:20.563 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-24T22:02:20.563 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-24T22:02:20.563 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-24T22:02:20.564 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-24T22:02:20.602 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:20.658 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:20.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:20.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:02:20.766 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:20.780 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:20 smithi142 conmon[20125]: cluster 2021-03-24T22:02:18.900467+0000 mgr.y (mgr.14144) 556 : cluster 2021-03-24T22:02:20.780 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:20 smithi142 conmon[20125]: [DBG] pgmap v1015: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 509 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 290 KiB/s wr, 1 op/s; 51/1698 objects misplaced (3.004%) 2021-03-24T22:02:20.781 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:20 smithi142 conmon[20125]: cluster 2021-03-24T22:02:19.420176+0000 2021-03-24T22:02:20.781 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:20 smithi142 conmon[20125]: mon.a (mon.0) 1586 : cluster [DBG] osdmap e569: 8 total, 8 up, 8 in 2021-03-24T22:02:20.782 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:20 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:02:20.782 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:20 smithi142 conmon[20125]: 22:02:19.457967+0000 osd.0 (osd 2021-03-24T22:02:20.782 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:20 smithi142 conmon[20125]: .0) 118 : cluster [DBG] 6.18 scrub ok 2021-03-24T22:02:20.783 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:20 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:02:20.784 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:20 smithi142 conmon[20125]: 03-24T22:02:19.684722+0000 2021-03-24T22:02:20.784 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:20 smithi142 conmon[20125]: osd.2 (osd.2) 89 : cluster [DBG] 2021-03-24T22:02:20.785 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:20 smithi142 conmon[20125]: 3.1d scrub ok 2021-03-24T22:02:20.785 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:20 smithi142 conmon[20125]: cluster 2021-03-24T22:02:20.785 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:20 smithi142 conmon[20125]: 2021-03-24T22:02:19.755642+0000 2021-03-24T22:02:20.786 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:20 smithi142 conmon[20125]: osd.6 (osd.6) 36 : cluster [DBG] 2021-03-24T22:02:20.786 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:20 smithi142 conmon[20125]: 7.7 scrub ok 2021-03-24T22:02:20.787 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:20 smithi142 conmon[20125]: audit 2021-03-24T22:02:20.787 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:20 smithi142 conmon[20125]: 2021-03-24T22:02:19.819199+0000 2021-03-24T22:02:20.788 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:20 smithi142 conmon[20125]: mon.a (mon.0) 1587 : audit [INF] 2021-03-24T22:02:20.790 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:20 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:02:20.790 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:20 smithi142 conmon[20125]: cluster 2021-03-24T22:02:20.790 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:20 smithi142 conmon[20125]: 2021-03-24T22:02:20.296486+0000 2021-03-24T22:02:20.790 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:20 smithi142 conmon[20125]: mon.a (mon.0) 1588 : cluster [DBG] 2021-03-24T22:02:20.791 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:20 smithi142 conmon[20125]: osdmap e570: 8 total, 8 up, 8 in 2021-03-24T22:02:20.791 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:20 smithi142 conmon[20125]: audit 2021-03-24T22:02:20.791 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:20 smithi142 conmon[20125]: 2021-03-24T22:02:20.355143+0000 2021-03-24T22:02:20.792 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:20 smithi142 conmon[20125]: mon.c (mon.2) 87 : audit [DBG] 2021-03-24T22:02:20.792 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:20 smithi142 conmon[20125]: from='client.? v1:172.21.15.125:0/3649417752' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:02:20.826 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:20.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:20.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:02:20.933 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:20.995 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:20.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:21.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:21.101 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:21.161 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:21.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:21.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:02:21.266 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:21.322 INFO:tasks.rados.rados.0.smithi125.stdout:2134: done (0 left) 2021-03-24T22:02:21.323 INFO:tasks.rados.rados.0.smithi125.stdout:2135: copy_from oid 20 from oid 46 current snap is 241 2021-03-24T22:02:21.323 INFO:tasks.rados.rados.0.smithi125.stdout:2136: write (excl) oid 35 current snap is 241 2021-03-24T22:02:21.323 INFO:tasks.rados.rados.0.smithi125.stdout:2136: seq_num 746 ranges {538022=738348,1737567=561594} 2021-03-24T22:02:21.324 INFO:teuthology.orchestra.run.smithi125.stderr:reweighted osd.2 to 1 (10000) 2021-03-24T22:02:21.330 INFO:tasks.rados.rados.0.smithi125.stdout:2136: writing smithi12556470-35 from 538022 to 1276370 tid 1 2021-03-24T22:02:21.338 INFO:tasks.rados.rados.0.smithi125.stdout:2136: writing smithi12556470-35 from 1737567 to 2299161 tid 2 2021-03-24T22:02:21.338 INFO:tasks.rados.rados.0.smithi125.stdout:2136: finishing write tid 1 to smithi12556470-35 2021-03-24T22:02:21.339 INFO:tasks.rados.rados.0.smithi125.stdout:2137: copy_from oid 22 from oid 45 current snap is 241 2021-03-24T22:02:21.339 INFO:tasks.rados.rados.0.smithi125.stdout:2138: write oid 1 current snap is 241 2021-03-24T22:02:21.339 INFO:tasks.rados.rados.0.smithi125.stdout:2138: seq_num 747 ranges {554829=413597,1396049=705252} 2021-03-24T22:02:21.344 INFO:tasks.rados.rados.0.smithi125.stdout:2138: writing smithi12556470-1 from 554829 to 968426 tid 1 2021-03-24T22:02:21.345 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:21.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:21.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:21.353 INFO:tasks.rados.rados.0.smithi125.stdout:2138: writing smithi12556470-1 from 1396049 to 2101301 tid 2 2021-03-24T22:02:21.353 INFO:tasks.rados.rados.0.smithi125.stdout:2139: write oid 32 current snap is 241 2021-03-24T22:02:21.353 INFO:tasks.rados.rados.0.smithi125.stdout:2136: finishing write tid 2 to smithi12556470-35 2021-03-24T22:02:21.353 INFO:tasks.rados.rados.0.smithi125.stdout:2139: seq_num 748 ranges {571636=488846,1854531=565263} 2021-03-24T22:02:21.358 INFO:tasks.rados.rados.0.smithi125.stdout:2139: writing smithi12556470-32 from 571636 to 1060482 tid 1 2021-03-24T22:02:21.367 INFO:tasks.rados.rados.0.smithi125.stdout:2139: writing smithi12556470-32 from 1854531 to 2419794 tid 2 2021-03-24T22:02:21.367 INFO:tasks.rados.rados.0.smithi125.stdout:2140: snap_remove snap 236 2021-03-24T22:02:21.430 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2021-03-24T22:02:21.431 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2021-03-24T22:02:21.431 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2021-03-24T22:02:21.431 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2021-03-24T22:02:21.431 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.432 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPushReply(7.1 110/108 [PushReplyOp(7:83e5d499:::smithi12556470-11:head)])", 2021-03-24T22:02:21.432 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:54:40.024517+0000", 2021-03-24T22:02:21.432 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 461.40116513700002, 2021-03-24T22:02:21.432 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.086802544999999995, 2021-03-24T22:02:21.432 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:02:21.433 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "reached pg", 2021-03-24T22:02:21.433 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:02:21.433 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.433 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:02:21.434 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.024517+0000", 2021-03-24T22:02:21.434 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:02:21.434 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.434 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.435 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:02:21.435 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.024517+0000", 2021-03-24T22:02:21.435 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2021-03-24T22:02:21.435 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.436 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.436 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:02:21.436 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.024515+0000", 2021-03-24T22:02:21.436 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3049999999999998e-06 2021-03-24T22:02:21.437 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.437 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.437 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:02:21.437 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.024522+0000", 2021-03-24T22:02:21.437 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0208999999999999e-05 2021-03-24T22:02:21.438 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.438 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.438 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:02:21.439 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.024542+0000", 2021-03-24T22:02:21.439 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9319999999999999e-06 2021-03-24T22:02:21.440 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.440 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.440 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:02:21.440 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.024549+0000", 2021-03-24T22:02:21.441 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.086712301000000006 2021-03-24T22:02:21.441 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.441 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.441 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:21.442 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.111261+0000", 2021-03-24T22:02:21.442 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7893000000000003e-05 2021-03-24T22:02:21.442 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.442 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.442 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:02:21.443 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.111319+0000", 2021-03-24T22:02:21.443 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.086802544999999995 2021-03-24T22:02:21.443 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:21.443 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:02:21.444 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:21.444 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.444 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.444 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPushReply(4.9 111/108 [PushReplyOp(4:94d01cf7:::data_log.108:head)])", 2021-03-24T22:02:21.444 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:54:40.207652+0000", 2021-03-24T22:02:21.445 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 461.21802945000002, 2021-03-24T22:02:21.445 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.089012543, 2021-03-24T22:02:21.445 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:02:21.445 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "reached pg", 2021-03-24T22:02:21.445 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:02:21.446 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.446 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:02:21.446 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.207652+0000", 2021-03-24T22:02:21.446 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:02:21.447 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.447 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.447 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:02:21.447 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.207652+0000", 2021-03-24T22:02:21.447 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2021-03-24T22:02:21.448 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.448 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.448 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:02:21.448 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.207650+0000", 2021-03-24T22:02:21.449 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.7899999999999994e-06 2021-03-24T22:02:21.449 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.449 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.450 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:02:21.450 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.207660+0000", 2021-03-24T22:02:21.450 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8369e-05 2021-03-24T22:02:21.450 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.451 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.451 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:02:21.451 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.207678+0000", 2021-03-24T22:02:21.451 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2442e-05 2021-03-24T22:02:21.452 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.452 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.452 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:02:21.452 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.207691+0000", 2021-03-24T22:02:21.452 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.088841322 2021-03-24T22:02:21.453 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.453 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.453 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:21.453 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.296532+0000", 2021-03-24T22:02:21.454 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013249 2021-03-24T22:02:21.454 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.454 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.454 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:02:21.454 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.296665+0000", 2021-03-24T22:02:21.455 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.089012543 2021-03-24T22:02:21.455 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:21.455 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:02:21.455 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:21.455 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.456 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.456 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPushReply(7.1 111/108 [PushReplyOp(7:86662589:::smithi12556470-19:head)])", 2021-03-24T22:02:21.456 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:54:40.297084+0000", 2021-03-24T22:02:21.456 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 461.12859723700001, 2021-03-24T22:02:21.457 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.090340742000000002, 2021-03-24T22:02:21.457 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:02:21.457 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "reached pg", 2021-03-24T22:02:21.457 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:02:21.457 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.458 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:02:21.458 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.297084+0000", 2021-03-24T22:02:21.458 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:02:21.458 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.459 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.459 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:02:21.459 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.297084+0000", 2021-03-24T22:02:21.459 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999971 2021-03-24T22:02:21.459 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.460 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.460 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:02:21.460 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.297082+0000", 2021-03-24T22:02:21.460 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3006999999999999e-05 2021-03-24T22:02:21.461 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.461 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.461 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:02:21.461 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.297095+0000", 2021-03-24T22:02:21.461 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8322000000000001e-05 2021-03-24T22:02:21.462 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.462 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.462 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:02:21.462 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.297123+0000", 2021-03-24T22:02:21.463 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9049999999999999e-05 2021-03-24T22:02:21.463 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.463 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.463 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:02:21.463 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.297142+0000", 2021-03-24T22:02:21.464 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.089791975999999996 2021-03-24T22:02:21.464 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.464 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.464 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:21.465 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.386934+0000", 2021-03-24T22:02:21.465 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00049109999999999996 2021-03-24T22:02:21.465 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.465 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.465 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:02:21.466 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.387425+0000", 2021-03-24T22:02:21.466 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.090340742000000002 2021-03-24T22:02:21.466 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:21.466 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:02:21.466 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:21.467 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.467 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.467 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPushReply(4.9 111/108 [PushReplyOp(4:921ecdb1:gc::gc.25:head)])", 2021-03-24T22:02:21.467 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:54:40.483015+0000", 2021-03-24T22:02:21.468 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 460.94266617900001, 2021-03-24T22:02:21.468 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.089336145000000006, 2021-03-24T22:02:21.468 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:02:21.468 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "reached pg", 2021-03-24T22:02:21.468 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:02:21.469 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.469 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:02:21.469 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.483015+0000", 2021-03-24T22:02:21.469 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:02:21.469 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.470 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.470 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:02:21.470 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.483015+0000", 2021-03-24T22:02:21.470 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999971 2021-03-24T22:02:21.471 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.471 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.471 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:02:21.472 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.483012+0000", 2021-03-24T22:02:21.472 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4007e-05 2021-03-24T22:02:21.472 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.472 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.472 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:02:21.473 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.483026+0000", 2021-03-24T22:02:21.473 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9105999999999999e-05 2021-03-24T22:02:21.473 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.473 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.473 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:02:21.474 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.483056+0000", 2021-03-24T22:02:21.474 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1919999999999999e-05 2021-03-24T22:02:21.474 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.474 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.475 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:02:21.475 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.483067+0000", 2021-03-24T22:02:21.475 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.089135437999999997 2021-03-24T22:02:21.475 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.475 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.476 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:21.476 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.572203+0000", 2021-03-24T22:02:21.476 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014872799999999999 2021-03-24T22:02:21.476 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.477 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.477 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:02:21.477 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.572352+0000", 2021-03-24T22:02:21.477 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.089336145000000006 2021-03-24T22:02:21.477 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:21.478 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:02:21.478 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:21.478 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.478 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.479 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPushReply(7.1 111/108 [PushReplyOp(7:8c1d349c:::smithi12556470-48:head)])", 2021-03-24T22:02:21.479 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:54:40.572536+0000", 2021-03-24T22:02:21.479 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 460.85314519600001, 2021-03-24T22:02:21.479 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.090618961999999997, 2021-03-24T22:02:21.479 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:02:21.480 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "reached pg", 2021-03-24T22:02:21.480 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:02:21.480 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.480 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:02:21.481 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.572536+0000", 2021-03-24T22:02:21.481 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:02:21.481 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.481 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.481 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:02:21.482 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.572536+0000", 2021-03-24T22:02:21.482 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999967 2021-03-24T22:02:21.482 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.482 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.482 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:02:21.483 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.572533+0000", 2021-03-24T22:02:21.483 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3864e-05 2021-03-24T22:02:21.483 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.483 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.484 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:02:21.484 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.572547+0000", 2021-03-24T22:02:21.484 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4148999999999999e-05 2021-03-24T22:02:21.484 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.484 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.485 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:02:21.485 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.572571+0000", 2021-03-24T22:02:21.485 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3868e-05 2021-03-24T22:02:21.485 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.485 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.486 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:02:21.486 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.572585+0000", 2021-03-24T22:02:21.486 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.090012964000000001 2021-03-24T22:02:21.486 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.487 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.487 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:21.487 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.662598+0000", 2021-03-24T22:02:21.487 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00055724599999999996 2021-03-24T22:02:21.487 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.488 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.488 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:02:21.488 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.663155+0000", 2021-03-24T22:02:21.488 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.090618961999999997 2021-03-24T22:02:21.488 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:21.489 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:02:21.489 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:21.489 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.489 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.490 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPushReply(4.9 111/108 [PushReplyOp(4:9268b73e:::obj_delete_at_hint.0000000019:head)])", 2021-03-24T22:02:21.490 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:54:40.758844+0000", 2021-03-24T22:02:21.490 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 460.666837456, 2021-03-24T22:02:21.490 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.089171401999999997, 2021-03-24T22:02:21.490 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:02:21.491 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "reached pg", 2021-03-24T22:02:21.491 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:02:21.491 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.491 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:02:21.492 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.758844+0000", 2021-03-24T22:02:21.492 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:02:21.492 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.492 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.492 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:02:21.493 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.758844+0000", 2021-03-24T22:02:21.493 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999967 2021-03-24T22:02:21.493 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.493 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.493 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:02:21.494 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.758841+0000", 2021-03-24T22:02:21.494 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7056000000000001e-05 2021-03-24T22:02:21.494 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.494 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.495 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:02:21.495 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.758858+0000", 2021-03-24T22:02:21.495 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8711e-05 2021-03-24T22:02:21.495 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.495 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.496 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:02:21.496 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.758887+0000", 2021-03-24T22:02:21.496 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5746000000000001e-05 2021-03-24T22:02:21.496 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.496 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.497 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:02:21.497 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.758902+0000", 2021-03-24T22:02:21.497 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.088984598999999998 2021-03-24T22:02:21.497 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.498 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.498 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:21.498 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.847887+0000", 2021-03-24T22:02:21.498 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000128644 2021-03-24T22:02:21.498 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.499 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.499 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:02:21.499 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.848016+0000", 2021-03-24T22:02:21.499 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.089171401999999997 2021-03-24T22:02:21.499 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:21.500 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:02:21.500 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:21.500 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.500 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.501 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPushReply(7.1 111/108 [PushReplyOp(7:8dd64f9e:::smithi12556470-49:head)])", 2021-03-24T22:02:21.501 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:54:40.847956+0000", 2021-03-24T22:02:21.501 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 460.577725735, 2021-03-24T22:02:21.501 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.090871619000000001, 2021-03-24T22:02:21.501 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:02:21.502 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "reached pg", 2021-03-24T22:02:21.502 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:02:21.502 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.502 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:02:21.502 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.847956+0000", 2021-03-24T22:02:21.503 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:02:21.503 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.503 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.503 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:02:21.504 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.847956+0000", 2021-03-24T22:02:21.504 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999967 2021-03-24T22:02:21.504 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.504 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.504 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:02:21.505 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.847953+0000", 2021-03-24T22:02:21.505 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5254e-05 2021-03-24T22:02:21.505 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.505 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.505 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:02:21.506 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.847968+0000", 2021-03-24T22:02:21.506 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9885e-05 2021-03-24T22:02:21.506 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.506 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.507 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:02:21.507 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.847988+0000", 2021-03-24T22:02:21.507 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1423999999999999e-05 2021-03-24T22:02:21.507 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.507 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.508 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:02:21.508 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.847999+0000", 2021-03-24T22:02:21.508 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.090269889000000006 2021-03-24T22:02:21.508 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.508 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.509 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:21.509 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.938269+0000", 2021-03-24T22:02:21.509 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00055842100000000003 2021-03-24T22:02:21.509 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.510 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.510 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:02:21.510 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:40.938828+0000", 2021-03-24T22:02:21.510 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.090871619000000001 2021-03-24T22:02:21.510 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:21.511 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:02:21.511 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:21.511 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.511 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.512 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPushReply(4.9 111/108 [PushReplyOp(4:94441385:::obj_delete_at_hint.0000000039:head)])", 2021-03-24T22:02:21.512 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:54:41.034565+0000", 2021-03-24T22:02:21.512 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 460.39111714000001, 2021-03-24T22:02:21.512 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.089107565999999999, 2021-03-24T22:02:21.512 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:02:21.513 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "reached pg", 2021-03-24T22:02:21.513 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:02:21.513 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.513 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:02:21.513 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:41.034565+0000", 2021-03-24T22:02:21.514 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:02:21.514 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.514 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.514 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:02:21.515 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:41.034565+0000", 2021-03-24T22:02:21.515 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2021-03-24T22:02:21.515 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.515 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.515 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:02:21.516 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:41.034564+0000", 2021-03-24T22:02:21.516 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.4819999999999999e-06 2021-03-24T22:02:21.516 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.516 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.516 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:02:21.517 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:41.034570+0000", 2021-03-24T22:02:21.517 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6356999999999999e-05 2021-03-24T22:02:21.517 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.517 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.518 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:02:21.518 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:41.034587+0000", 2021-03-24T22:02:21.518 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0502999999999999e-05 2021-03-24T22:02:21.518 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.518 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.519 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:02:21.519 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:41.034597+0000", 2021-03-24T22:02:21.519 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.088942807999999998 2021-03-24T22:02:21.519 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.520 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.520 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:21.520 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:41.123540+0000", 2021-03-24T22:02:21.520 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000132261 2021-03-24T22:02:21.520 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.521 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.521 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:02:21.521 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:41.123672+0000", 2021-03-24T22:02:21.521 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.089107565999999999 2021-03-24T22:02:21.522 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:21.522 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:02:21.522 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:21.522 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.522 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.523 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:1246 7.1 7:86662589:::smithi12556470-19:head [write 443751~631882] snapc 53=[] ondisk+write+known_if_redirected e191)", 2021-03-24T22:02:21.523 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:56:00.156445+0000", 2021-03-24T22:02:21.523 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 381.269237089, 2021-03-24T22:02:21.523 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.060689377000000003, 2021-03-24T22:02:21.523 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:02:21.524 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:02:21.524 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:02:21.524 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:02:21.524 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:02:21.525 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 1246 2021-03-24T22:02:21.525 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.525 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:02:21.525 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.525 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:02:21.526 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.156445+0000", 2021-03-24T22:02:21.526 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:02:21.526 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.526 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.527 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:02:21.527 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.156445+0000", 2021-03-24T22:02:21.527 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999952 2021-03-24T22:02:21.527 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.527 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.528 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:02:21.528 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.156440+0000", 2021-03-24T22:02:21.528 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0021389159999999998 2021-03-24T22:02:21.528 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.529 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.529 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:02:21.529 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.158579+0000", 2021-03-24T22:02:21.529 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6568999999999999e-05 2021-03-24T22:02:21.529 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.530 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.530 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:02:21.530 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.158605+0000", 2021-03-24T22:02:21.530 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4353e-05 2021-03-24T22:02:21.530 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.531 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.531 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:02:21.531 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.158620+0000", 2021-03-24T22:02:21.531 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010933 2021-03-24T22:02:21.532 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.532 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.532 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:21.532 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.158729+0000", 2021-03-24T22:02:21.532 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016208300000000001 2021-03-24T22:02:21.533 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.533 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.533 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:02:21.533 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.158891+0000", 2021-03-24T22:02:21.534 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00030850199999999999 2021-03-24T22:02:21.534 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.534 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.534 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 0,3", 2021-03-24T22:02:21.534 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.159200+0000", 2021-03-24T22:02:21.535 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.003189473 2021-03-24T22:02:21.535 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.535 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.535 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T22:02:21.536 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.162389+0000", 2021-03-24T22:02:21.536 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.049777185000000002 2021-03-24T22:02:21.536 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.536 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.536 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:02:21.537 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.212166+0000", 2021-03-24T22:02:21.537 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0048629060000000002 2021-03-24T22:02:21.537 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.537 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.537 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:02:21.538 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.217029+0000", 2021-03-24T22:02:21.538 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8270000000000003e-05 2021-03-24T22:02:21.538 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.538 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.539 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:02:21.539 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.217088+0000", 2021-03-24T22:02:21.539 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.6387000000000002e-05 2021-03-24T22:02:21.539 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.539 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.540 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:02:21.540 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.217134+0000", 2021-03-24T22:02:21.540 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.060689377000000003 2021-03-24T22:02:21.540 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:21.540 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:02:21.541 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:21.541 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.541 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.541 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:1247 7.1 7:86662589:::smithi12556470-19:head [write 1810907~765058] snapc 53=[] ondisk+write+known_if_redirected e191)", 2021-03-24T22:02:21.542 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:56:00.164197+0000", 2021-03-24T22:02:21.542 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 381.261484978, 2021-03-24T22:02:21.542 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.054173828, 2021-03-24T22:02:21.542 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:02:21.542 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:02:21.543 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:02:21.543 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:02:21.543 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:02:21.543 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 1247 2021-03-24T22:02:21.543 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.544 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:02:21.544 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.544 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:02:21.544 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.164197+0000", 2021-03-24T22:02:21.545 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:02:21.545 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.545 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.545 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:02:21.545 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.164197+0000", 2021-03-24T22:02:21.546 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999962 2021-03-24T22:02:21.546 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.546 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.546 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:02:21.547 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.164193+0000", 2021-03-24T22:02:21.547 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011806989999999999 2021-03-24T22:02:21.547 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.547 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.547 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:02:21.548 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.165374+0000", 2021-03-24T22:02:21.548 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0941000000000001e-05 2021-03-24T22:02:21.548 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.548 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.549 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:02:21.549 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.165405+0000", 2021-03-24T22:02:21.549 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3784999999999999e-05 2021-03-24T22:02:21.549 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.549 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.550 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:02:21.550 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.165418+0000", 2021-03-24T22:02:21.550 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000101259 2021-03-24T22:02:21.550 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.551 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.551 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:21.551 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.165520+0000", 2021-03-24T22:02:21.551 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015882 2021-03-24T22:02:21.551 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.552 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.552 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:02:21.552 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.165678+0000", 2021-03-24T22:02:21.552 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000318105 2021-03-24T22:02:21.553 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.553 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.553 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 0,3", 2021-03-24T22:02:21.553 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.165996+0000", 2021-03-24T22:02:21.553 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0042140550000000004 2021-03-24T22:02:21.554 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.554 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.554 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T22:02:21.554 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.170211+0000", 2021-03-24T22:02:21.555 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.042059725999999999 2021-03-24T22:02:21.555 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.555 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.555 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:02:21.555 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.212270+0000", 2021-03-24T22:02:21.556 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0060031010000000003 2021-03-24T22:02:21.556 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.556 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.556 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:02:21.556 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.218273+0000", 2021-03-24T22:02:21.557 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5608e-05 2021-03-24T22:02:21.557 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.557 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.557 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:02:21.558 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.218329+0000", 2021-03-24T22:02:21.558 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1533999999999999e-05 2021-03-24T22:02:21.558 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.558 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.558 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:02:21.559 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.218371+0000", 2021-03-24T22:02:21.559 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.054173828 2021-03-24T22:02:21.559 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:21.559 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:02:21.559 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:21.560 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.560 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.560 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:1248 7.1 7:86662589:::smithi12556470-19:head [write 3067461~176290] snapc 53=[] ondisk+write+known_if_redirected e191)", 2021-03-24T22:02:21.560 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:56:00.167908+0000", 2021-03-24T22:02:21.561 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 381.257773418, 2021-03-24T22:02:21.561 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.050562023999999997, 2021-03-24T22:02:21.561 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:02:21.561 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:02:21.561 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:02:21.562 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:02:21.562 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:02:21.562 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 1248 2021-03-24T22:02:21.562 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.563 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:02:21.563 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.563 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:02:21.563 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.167908+0000", 2021-03-24T22:02:21.563 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:02:21.564 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.564 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.564 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:02:21.564 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.167908+0000", 2021-03-24T22:02:21.564 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999971 2021-03-24T22:02:21.565 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.565 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.565 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:02:21.565 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.167905+0000", 2021-03-24T22:02:21.566 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015547299999999999 2021-03-24T22:02:21.566 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.566 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.566 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:02:21.566 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168061+0000", 2021-03-24T22:02:21.567 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9348000000000001e-05 2021-03-24T22:02:21.567 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.567 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.567 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:02:21.568 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168090+0000", 2021-03-24T22:02:21.568 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1722000000000001e-05 2021-03-24T22:02:21.568 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.568 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.568 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:02:21.569 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168102+0000", 2021-03-24T22:02:21.569 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00022216099999999999 2021-03-24T22:02:21.569 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.569 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.569 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:21.570 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168324+0000", 2021-03-24T22:02:21.570 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013274000000000001 2021-03-24T22:02:21.570 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.570 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.571 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:02:21.571 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168457+0000", 2021-03-24T22:02:21.571 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000219247 2021-03-24T22:02:21.571 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.572 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.572 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 0,3", 2021-03-24T22:02:21.572 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168676+0000", 2021-03-24T22:02:21.572 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0015439570000000001 2021-03-24T22:02:21.572 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.573 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.573 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T22:02:21.573 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.170220+0000", 2021-03-24T22:02:21.573 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.042126005000000001 2021-03-24T22:02:21.574 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.574 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.574 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:02:21.574 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.212346+0000", 2021-03-24T22:02:21.574 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0060575079999999996 2021-03-24T22:02:21.575 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.575 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.575 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:02:21.575 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.218403+0000", 2021-03-24T22:02:21.576 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7947000000000001e-05 2021-03-24T22:02:21.576 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.576 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.576 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:02:21.576 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.218441+0000", 2021-03-24T22:02:21.577 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8948000000000001e-05 2021-03-24T22:02:21.577 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.577 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.577 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:02:21.577 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.218470+0000", 2021-03-24T22:02:21.578 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.050562023999999997 2021-03-24T22:02:21.578 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:21.578 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:02:21.578 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:21.579 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.579 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.579 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:1249 7.1 7:86662589:::smithi12556470-19:head [setxattr _header (57) in=7b,truncate 3243751] snapc 53=[] ondisk+write+known_if_redirected e191)", 2021-03-24T22:02:21.579 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:56:00.168119+0000", 2021-03-24T22:02:21.579 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 381.25756261999999, 2021-03-24T22:02:21.580 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.052051300000000002, 2021-03-24T22:02:21.580 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:02:21.580 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:02:21.580 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:02:21.581 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:02:21.581 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:02:21.581 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 1249 2021-03-24T22:02:21.581 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.581 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:02:21.582 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.582 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:02:21.582 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168119+0000", 2021-03-24T22:02:21.582 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:02:21.582 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.583 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.583 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:02:21.583 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168119+0000", 2021-03-24T22:02:21.583 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2021-03-24T22:02:21.584 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.584 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.584 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:02:21.584 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168118+0000", 2021-03-24T22:02:21.584 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.0740000000000004e-06 2021-03-24T22:02:21.585 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.585 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.585 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:02:21.585 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168125+0000", 2021-03-24T22:02:21.586 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2374e-05 2021-03-24T22:02:21.586 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.586 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.586 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:02:21.586 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168137+0000", 2021-03-24T22:02:21.587 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.5490000000000003e-06 2021-03-24T22:02:21.587 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.587 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.587 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:02:21.587 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168144+0000", 2021-03-24T22:02:21.588 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011986869999999999 2021-03-24T22:02:21.588 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.588 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.588 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:21.589 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.169342+0000", 2021-03-24T22:02:21.589 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000160089 2021-03-24T22:02:21.589 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.589 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.589 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:02:21.590 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.169502+0000", 2021-03-24T22:02:21.590 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00031500400000000002 2021-03-24T22:02:21.590 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.590 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.590 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 0,3", 2021-03-24T22:02:21.591 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.169817+0000", 2021-03-24T22:02:21.591 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00070204499999999999 2021-03-24T22:02:21.591 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.591 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.592 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T22:02:21.592 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.170519+0000", 2021-03-24T22:02:21.592 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.041887362999999997 2021-03-24T22:02:21.592 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.593 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.593 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:02:21.593 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.212407+0000", 2021-03-24T22:02:21.593 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0076951889999999998 2021-03-24T22:02:21.593 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.594 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.594 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:02:21.594 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.220102+0000", 2021-03-24T22:02:21.594 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3169e-05 2021-03-24T22:02:21.594 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.595 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.595 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:02:21.595 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.220135+0000", 2021-03-24T22:02:21.595 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5286000000000001e-05 2021-03-24T22:02:21.596 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.596 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.596 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:02:21.596 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.220170+0000", 2021-03-24T22:02:21.596 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.052051300000000002 2021-03-24T22:02:21.597 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:21.597 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:02:21.597 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:21.597 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.597 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.598 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:1250 7.1 7:86662589:::smithi12556470-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e191)", 2021-03-24T22:02:21.598 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:56:00.168170+0000", 2021-03-24T22:02:21.598 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 381.25751131800001, 2021-03-24T22:02:21.598 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.052145710999999997, 2021-03-24T22:02:21.599 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:02:21.599 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:02:21.599 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:02:21.599 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:02:21.599 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:02:21.600 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 1250 2021-03-24T22:02:21.600 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.600 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:02:21.600 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.601 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:02:21.601 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168170+0000", 2021-03-24T22:02:21.601 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:02:21.601 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.601 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.602 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:02:21.602 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168170+0000", 2021-03-24T22:02:21.602 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2021-03-24T22:02:21.602 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.603 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.603 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:02:21.603 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168169+0000", 2021-03-24T22:02:21.603 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.5979999999999999e-06 2021-03-24T22:02:21.603 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.604 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.604 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:02:21.604 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168174+0000", 2021-03-24T22:02:21.604 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3064000000000001e-05 2021-03-24T22:02:21.605 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.605 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.605 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:02:21.605 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168187+0000", 2021-03-24T22:02:21.605 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9599999999999997e-06 2021-03-24T22:02:21.606 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.606 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.606 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:02:21.606 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.168193+0000", 2021-03-24T22:02:21.607 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0021001650000000002 2021-03-24T22:02:21.607 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.607 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.607 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:21.607 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.170293+0000", 2021-03-24T22:02:21.608 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00019255100000000001 2021-03-24T22:02:21.608 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.608 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.608 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for rw locks", 2021-03-24T22:02:21.608 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.170485+0000", 2021-03-24T22:02:21.609 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.049706363000000003 2021-03-24T22:02:21.609 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.609 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.609 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:21.610 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.220192+0000", 2021-03-24T22:02:21.610 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9202e-05 2021-03-24T22:02:21.610 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.610 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.610 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:02:21.611 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.220251+0000", 2021-03-24T22:02:21.611 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.5007999999999998e-05 2021-03-24T22:02:21.611 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.611 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.611 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:02:21.612 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:56:00.220316+0000", 2021-03-24T22:02:21.612 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.052145710999999997 2021-03-24T22:02:21.612 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:21.612 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:02:21.613 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:21.613 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.613 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.613 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(4.3 422/417 [PushOp(4:c5eee8e2:::data_log.59:head, version: 65'2, data_included: [0~156], data_size: 156, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:c5eee8e2:::data_log.59:head@65'2, size: 156, copy_subset: [0~156], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:156, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2021-03-24T22:02:21.613 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:59:52.286731+0000", 2021-03-24T22:02:21.614 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 149.13895101099999, 2021-03-24T22:02:21.614 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.16378574300000001, 2021-03-24T22:02:21.614 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:02:21.614 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:02:21.615 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:02:21.615 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.615 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:02:21.615 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:52.286731+0000", 2021-03-24T22:02:21.615 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:02:21.616 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.616 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.616 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:02:21.616 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:52.286731+0000", 2021-03-24T22:02:21.616 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999971 2021-03-24T22:02:21.617 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.617 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.617 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:02:21.617 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:52.286728+0000", 2021-03-24T22:02:21.618 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3193e-05 2021-03-24T22:02:21.618 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.618 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.618 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:02:21.618 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:52.286751+0000", 2021-03-24T22:02:21.619 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1464e-05 2021-03-24T22:02:21.619 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.619 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.619 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:02:21.620 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:52.286773+0000", 2021-03-24T22:02:21.620 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3377000000000001e-05 2021-03-24T22:02:21.620 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.620 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.620 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:02:21.621 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:52.286786+0000", 2021-03-24T22:02:21.621 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.16356628100000001 2021-03-24T22:02:21.621 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.621 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.622 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:21.622 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:52.450352+0000", 2021-03-24T22:02:21.622 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4530000000000001e-06 2021-03-24T22:02:21.622 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.622 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.623 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:02:21.623 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:52.450358+0000", 2021-03-24T22:02:21.623 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015859899999999999 2021-03-24T22:02:21.623 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.624 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.624 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:02:21.624 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:52.450516+0000", 2021-03-24T22:02:21.624 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.16378574300000001 2021-03-24T22:02:21.624 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:21.625 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:02:21.625 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:21.625 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.625 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.625 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(4.6 424/417 [PushOp(4:620d8b71:::data_log.99:head, version: 65'1, data_included: [0~156], data_size: 156, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:620d8b71:::data_log.99:head@65'1, size: 156, copy_subset: [0~156], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:156, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2021-03-24T22:02:21.626 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:59:54.302766+0000", 2021-03-24T22:02:21.626 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 147.12291574899999, 2021-03-24T22:02:21.626 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.13019102099999999, 2021-03-24T22:02:21.626 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:02:21.627 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:02:21.627 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:02:21.627 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.627 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:02:21.628 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:54.302766+0000", 2021-03-24T22:02:21.628 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:02:21.628 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.628 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.628 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:02:21.629 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:54.302766+0000", 2021-03-24T22:02:21.629 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2021-03-24T22:02:21.629 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.629 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.630 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:02:21.630 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:54.302764+0000", 2021-03-24T22:02:21.630 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7988000000000001e-05 2021-03-24T22:02:21.630 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.630 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.631 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:02:21.631 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:54.302782+0000", 2021-03-24T22:02:21.631 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3334000000000001e-05 2021-03-24T22:02:21.631 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.632 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.632 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:02:21.632 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:54.302806+0000", 2021-03-24T22:02:21.632 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7897000000000002e-05 2021-03-24T22:02:21.632 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.633 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.633 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:02:21.633 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:54.302824+0000", 2021-03-24T22:02:21.633 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.129927036 2021-03-24T22:02:21.633 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.634 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.634 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:21.634 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:54.432751+0000", 2021-03-24T22:02:21.634 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.7220000000000002e-06 2021-03-24T22:02:21.635 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.635 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.635 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:02:21.635 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:54.432757+0000", 2021-03-24T22:02:21.635 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00019955999999999999 2021-03-24T22:02:21.636 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.636 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.636 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:02:21.636 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:59:54.432957+0000", 2021-03-24T22:02:21.637 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.13019102099999999 2021-03-24T22:02:21.637 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:21.637 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:02:21.637 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:21.637 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.638 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.638 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:4154 7.1a 7:5f18d753:::smithi12556470-18:head [write 729569~476305,stat] snapc 143=[143,140,13d,13b,139,137,136,12d,12b,129,128,127,126,120,11e,11b,11a,119,10f,10c,105,104,fa,f8,e7] ondisk+write+known_if_redirected e433)", 2021-03-24T22:02:21.638 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:00:03.255082+0000", 2021-03-24T22:02:21.638 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 138.170599756, 2021-03-24T22:02:21.638 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.059519543, 2021-03-24T22:02:21.639 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:02:21.639 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:02:21.639 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:02:21.639 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:02:21.640 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:02:21.640 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 4154 2021-03-24T22:02:21.640 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.640 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:02:21.640 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.641 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:02:21.641 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.255082+0000", 2021-03-24T22:02:21.641 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:02:21.641 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.642 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.642 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:02:21.642 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.255082+0000", 2021-03-24T22:02:21.642 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2021-03-24T22:02:21.642 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.643 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.643 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:02:21.643 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.255080+0000", 2021-03-24T22:02:21.643 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038918194000000003 2021-03-24T22:02:21.644 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.644 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.644 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:02:21.644 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.293998+0000", 2021-03-24T22:02:21.644 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9609999999999994e-06 2021-03-24T22:02:21.645 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.645 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.645 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:02:21.645 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.294007+0000", 2021-03-24T22:02:21.645 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4920000000000001e-06 2021-03-24T22:02:21.646 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.646 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.646 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:02:21.646 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.294013+0000", 2021-03-24T22:02:21.647 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.9619000000000001e-05 2021-03-24T22:02:21.647 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.647 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.647 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:21.647 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.294053+0000", 2021-03-24T22:02:21.648 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1392999999999999e-05 2021-03-24T22:02:21.648 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.648 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.648 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for peered", 2021-03-24T22:02:21.648 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.294064+0000", 2021-03-24T22:02:21.649 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.99142193300000003 2021-03-24T22:02:21.649 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.649 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.649 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:21.650 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.285486+0000", 2021-03-24T22:02:21.650 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0029515280000000001 2021-03-24T22:02:21.650 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.650 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.650 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for degraded object", 2021-03-24T22:02:21.651 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.288437+0000", 2021-03-24T22:02:21.651 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021212278000000001 2021-03-24T22:02:21.651 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.651 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.651 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:21.652 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.309650+0000", 2021-03-24T22:02:21.652 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011838899999999999 2021-03-24T22:02:21.652 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.652 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.653 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:02:21.653 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.309768+0000", 2021-03-24T22:02:21.653 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00029724599999999998 2021-03-24T22:02:21.653 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.653 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.654 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 2,5", 2021-03-24T22:02:21.654 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.310065+0000", 2021-03-24T22:02:21.654 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0042070629999999996 2021-03-24T22:02:21.654 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.654 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.655 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:02:21.655 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.314272+0000", 2021-03-24T22:02:21.655 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4518000000000003e-05 2021-03-24T22:02:21.655 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.656 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.656 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:02:21.656 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.314317+0000", 2021-03-24T22:02:21.656 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5512999999999999e-05 2021-03-24T22:02:21.656 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.657 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.657 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T22:02:21.657 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.314332+0000", 2021-03-24T22:02:21.657 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.0575999999999996e-05 2021-03-24T22:02:21.657 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.658 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.658 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:02:21.658 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.314403+0000", 2021-03-24T22:02:21.658 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00019848500000000001 2021-03-24T22:02:21.659 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.659 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.659 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:02:21.659 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.314601+0000", 2021-03-24T22:02:21.659 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.059519543 2021-03-24T22:02:21.660 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:21.660 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:02:21.660 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:21.660 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.660 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.661 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:4155 7.1a 7:5f18d753:::smithi12556470-18:head [write 1996349~505866] snapc 143=[143,140,13d,13b,139,137,136,12d,12b,129,128,127,126,120,11e,11b,11a,119,10f,10c,105,104,fa,f8,e7] ondisk+write+known_if_redirected e433)", 2021-03-24T22:02:21.661 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:00:03.294026+0000", 2021-03-24T22:02:21.661 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 138.13165606000001, 2021-03-24T22:02:21.661 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0259160039999999, 2021-03-24T22:02:21.662 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:02:21.662 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:02:21.662 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:02:21.662 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:02:21.662 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:02:21.663 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 4155 2021-03-24T22:02:21.663 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.663 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:02:21.663 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.663 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:02:21.664 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.294026+0000", 2021-03-24T22:02:21.664 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:02:21.664 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.664 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.665 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:02:21.665 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.294026+0000", 2021-03-24T22:02:21.665 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2021-03-24T22:02:21.665 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.665 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.666 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:02:21.666 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.294024+0000", 2021-03-24T22:02:21.666 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00052380500000000002 2021-03-24T22:02:21.666 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.666 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.667 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:02:21.667 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.294548+0000", 2021-03-24T22:02:21.667 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2310000000000002e-06 2021-03-24T22:02:21.667 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.668 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.668 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:02:21.668 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.294557+0000", 2021-03-24T22:02:21.668 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.186e-06 2021-03-24T22:02:21.668 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.669 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.669 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:02:21.669 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.294563+0000", 2021-03-24T22:02:21.669 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8330000000000001e-05 2021-03-24T22:02:21.669 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.670 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.670 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:21.670 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.294601+0000", 2021-03-24T22:02:21.670 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6229999999999997e-06 2021-03-24T22:02:21.671 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.671 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.671 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for peered", 2021-03-24T22:02:21.671 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.294608+0000", 2021-03-24T22:02:21.671 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.99387276199999997 2021-03-24T22:02:21.672 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.672 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.672 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:21.672 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.288481+0000", 2021-03-24T22:02:21.672 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3121999999999995e-05 2021-03-24T22:02:21.673 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.673 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.673 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for degraded object", 2021-03-24T22:02:21.673 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.288564+0000", 2021-03-24T22:02:21.674 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022893111000000001 2021-03-24T22:02:21.674 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.674 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.674 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:21.674 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.311457+0000", 2021-03-24T22:02:21.675 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011776499999999999 2021-03-24T22:02:21.675 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.675 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.675 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for rw locks", 2021-03-24T22:02:21.675 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.311575+0000", 2021-03-24T22:02:21.676 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0030701790000000001 2021-03-24T22:02:21.676 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.676 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.676 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:21.677 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.314645+0000", 2021-03-24T22:02:21.677 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000109585 2021-03-24T22:02:21.677 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.677 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.677 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:02:21.678 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.314755+0000", 2021-03-24T22:02:21.678 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00022446599999999999 2021-03-24T22:02:21.678 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.678 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.678 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 2,5", 2021-03-24T22:02:21.679 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.314979+0000", 2021-03-24T22:02:21.679 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0027908550000000001 2021-03-24T22:02:21.679 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.679 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.679 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T22:02:21.680 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.317770+0000", 2021-03-24T22:02:21.680 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00025293499999999998 2021-03-24T22:02:21.680 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.680 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.681 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:02:21.681 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.318023+0000", 2021-03-24T22:02:21.681 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001577099 2021-03-24T22:02:21.681 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.681 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.682 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:02:21.682 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.319600+0000", 2021-03-24T22:02:21.682 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0034000000000002e-05 2021-03-24T22:02:21.682 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.683 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.683 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:02:21.683 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.319660+0000", 2021-03-24T22:02:21.683 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00028120800000000002 2021-03-24T22:02:21.683 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.684 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.684 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:02:21.684 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.319942+0000", 2021-03-24T22:02:21.684 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0259160039999999 2021-03-24T22:02:21.684 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:21.685 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:02:21.685 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:21.685 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.685 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.686 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:4156 7.1a 7:5f18d753:::smithi12556470-18:head [write 3004131~525438] snapc 143=[143,140,13d,13b,139,137,136,12d,12b,129,128,127,126,120,11e,11b,11a,119,10f,10c,105,104,fa,f8,e7] ondisk+write+known_if_redirected e433)", 2021-03-24T22:02:21.686 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:00:03.294575+0000", 2021-03-24T22:02:21.686 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 138.131106332, 2021-03-24T22:02:21.686 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.02580494, 2021-03-24T22:02:21.686 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:02:21.687 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:02:21.687 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:02:21.687 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:02:21.687 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:02:21.688 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 4156 2021-03-24T22:02:21.688 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.688 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:02:21.688 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.688 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:02:21.689 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.294575+0000", 2021-03-24T22:02:21.689 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:02:21.689 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.689 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.689 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:02:21.690 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.294575+0000", 2021-03-24T22:02:21.690 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2021-03-24T22:02:21.690 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.690 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.691 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:02:21.691 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.294574+0000", 2021-03-24T22:02:21.691 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00039593300000000001 2021-03-24T22:02:21.691 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.691 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.692 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:02:21.692 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.294970+0000", 2021-03-24T22:02:21.692 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.3130000000000001e-06 2021-03-24T22:02:21.692 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.693 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.693 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:02:21.693 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.294978+0000", 2021-03-24T22:02:21.693 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.963e-06 2021-03-24T22:02:21.693 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.694 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.694 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:02:21.694 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.294982+0000", 2021-03-24T22:02:21.694 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5567e-05 2021-03-24T22:02:21.694 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.695 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.695 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:21.695 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.295017+0000", 2021-03-24T22:02:21.695 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.5760000000000002e-06 2021-03-24T22:02:21.696 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.696 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.696 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for peered", 2021-03-24T22:02:21.696 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.295024+0000", 2021-03-24T22:02:21.696 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.99356920500000001 2021-03-24T22:02:21.697 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.697 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.697 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:21.697 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.288593+0000", 2021-03-24T22:02:21.697 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0455000000000003e-05 2021-03-24T22:02:21.698 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.698 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.698 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for degraded object", 2021-03-24T22:02:21.698 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.288653+0000", 2021-03-24T22:02:21.699 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022985047000000002 2021-03-24T22:02:21.699 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.699 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.699 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:21.699 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.311638+0000", 2021-03-24T22:02:21.700 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100049 2021-03-24T22:02:21.700 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.700 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.700 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for rw locks", 2021-03-24T22:02:21.701 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.311738+0000", 2021-03-24T22:02:21.701 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0043855500000000002 2021-03-24T22:02:21.701 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.701 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.701 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:21.702 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.316124+0000", 2021-03-24T22:02:21.702 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000159736 2021-03-24T22:02:21.702 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.702 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.702 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:02:21.703 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.316284+0000", 2021-03-24T22:02:21.703 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00024911200000000001 2021-03-24T22:02:21.703 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.703 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.704 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 2,5", 2021-03-24T22:02:21.704 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.316533+0000", 2021-03-24T22:02:21.704 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0024677029999999999 2021-03-24T22:02:21.704 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.704 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.705 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T22:02:21.705 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.319001+0000", 2021-03-24T22:02:21.705 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8992000000000001e-05 2021-03-24T22:02:21.705 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.705 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.706 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:02:21.706 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.319050+0000", 2021-03-24T22:02:21.706 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001096269 2021-03-24T22:02:21.706 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.707 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.707 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:02:21.707 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.320146+0000", 2021-03-24T22:02:21.707 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.8133000000000006e-05 2021-03-24T22:02:21.707 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.708 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.708 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:02:21.708 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.320224+0000", 2021-03-24T22:02:21.708 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015631799999999999 2021-03-24T22:02:21.709 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.709 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.709 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:02:21.709 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.320380+0000", 2021-03-24T22:02:21.709 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.02580494 2021-03-24T22:02:21.710 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:21.710 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:02:21.710 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:21.710 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.710 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.711 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:4157 7.1a 7:5f18d753:::smithi12556470-18:head [setxattr _header (58) in=7b,truncate 3529569] snapc 143=[143,140,13d,13b,139,137,136,12d,12b,129,128,127,126,120,11e,11b,11a,119,10f,10c,105,104,fa,f8,e7] ondisk+write+known_if_redirected e433)", 2021-03-24T22:02:21.711 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:00:03.294991+0000", 2021-03-24T22:02:21.711 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 138.13069041700001, 2021-03-24T22:02:21.711 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.025603236, 2021-03-24T22:02:21.712 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:02:21.712 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:02:21.712 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:02:21.712 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:02:21.712 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:02:21.713 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 4157 2021-03-24T22:02:21.713 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.713 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:02:21.713 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.713 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:02:21.714 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.294991+0000", 2021-03-24T22:02:21.714 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:02:21.714 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.714 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.715 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:02:21.715 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.294991+0000", 2021-03-24T22:02:21.715 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2021-03-24T22:02:21.715 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.715 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.716 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:02:21.716 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.294990+0000", 2021-03-24T22:02:21.716 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7519999999999998e-06 2021-03-24T22:02:21.716 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.716 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.717 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:02:21.717 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.294994+0000", 2021-03-24T22:02:21.717 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.456e-06 2021-03-24T22:02:21.717 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.718 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.718 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:02:21.718 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.294998+0000", 2021-03-24T22:02:21.718 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.728e-06 2021-03-24T22:02:21.718 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.719 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.719 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:02:21.719 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.295000+0000", 2021-03-24T22:02:21.719 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.2621999999999999e-05 2021-03-24T22:02:21.719 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.720 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.720 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:21.720 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.295053+0000", 2021-03-24T22:02:21.720 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.743e-06 2021-03-24T22:02:21.721 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.721 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.721 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for peered", 2021-03-24T22:02:21.721 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.295057+0000", 2021-03-24T22:02:21.721 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.99362635899999996 2021-03-24T22:02:21.722 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.722 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.722 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:21.722 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.288683+0000", 2021-03-24T22:02:21.723 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6723999999999993e-05 2021-03-24T22:02:21.723 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.723 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.723 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for degraded object", 2021-03-24T22:02:21.723 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.288750+0000", 2021-03-24T22:02:21.724 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023038461999999999 2021-03-24T22:02:21.724 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.724 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.724 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:21.724 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.311788+0000", 2021-03-24T22:02:21.725 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6539000000000006e-05 2021-03-24T22:02:21.725 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.725 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.725 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for rw locks", 2021-03-24T22:02:21.726 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.311875+0000", 2021-03-24T22:02:21.726 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0054267600000000001 2021-03-24T22:02:21.726 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.726 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.726 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:21.727 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.317302+0000", 2021-03-24T22:02:21.727 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011916700000000001 2021-03-24T22:02:21.727 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.727 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.728 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:02:21.728 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.317421+0000", 2021-03-24T22:02:21.728 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000123566 2021-03-24T22:02:21.728 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.728 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.729 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 2,5", 2021-03-24T22:02:21.729 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.317544+0000", 2021-03-24T22:02:21.729 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0015721470000000001 2021-03-24T22:02:21.729 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.729 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.730 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:02:21.730 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.319116+0000", 2021-03-24T22:02:21.730 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000844446 2021-03-24T22:02:21.730 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.731 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.731 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T22:02:21.731 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.319961+0000", 2021-03-24T22:02:21.731 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00047249500000000002 2021-03-24T22:02:21.731 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.732 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.732 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:02:21.732 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.320433+0000", 2021-03-24T22:02:21.732 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.5798999999999998e-05 2021-03-24T22:02:21.732 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.733 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.733 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:02:21.733 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.320499+0000", 2021-03-24T22:02:21.733 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.5266000000000006e-05 2021-03-24T22:02:21.734 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.734 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.734 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:02:21.734 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.320594+0000", 2021-03-24T22:02:21.734 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.025603236 2021-03-24T22:02:21.735 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:21.735 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:02:21.735 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:21.735 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.735 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.736 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:4158 7.1a 7:5f18d753:::smithi12556470-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e433)", 2021-03-24T22:02:21.736 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:00:03.295019+0000", 2021-03-24T22:02:21.736 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 138.13066261, 2021-03-24T22:02:21.736 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.025976462, 2021-03-24T22:02:21.737 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:02:21.737 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:02:21.737 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:02:21.737 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:02:21.737 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:02:21.738 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 4158 2021-03-24T22:02:21.738 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.738 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:02:21.738 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.739 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:02:21.739 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.295019+0000", 2021-03-24T22:02:21.739 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:02:21.739 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.739 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.740 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:02:21.740 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.295019+0000", 2021-03-24T22:02:21.740 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2021-03-24T22:02:21.740 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.740 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.741 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:02:21.741 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.295019+0000", 2021-03-24T22:02:21.741 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2960000000000001e-06 2021-03-24T22:02:21.741 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.741 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.742 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:02:21.742 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.295021+0000", 2021-03-24T22:02:21.742 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.9999999999999998e-06 2021-03-24T22:02:21.742 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.743 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.743 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:02:21.743 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.295025+0000", 2021-03-24T22:02:21.743 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7049999999999999e-06 2021-03-24T22:02:21.743 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.744 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.744 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:02:21.744 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.295028+0000", 2021-03-24T22:02:21.744 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.5915999999999999e-05 2021-03-24T22:02:21.745 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.745 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.745 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:21.745 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.295073+0000", 2021-03-24T22:02:21.745 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.044e-06 2021-03-24T22:02:21.746 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.746 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.746 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for peered", 2021-03-24T22:02:21.746 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:03.295077+0000", 2021-03-24T22:02:21.747 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.99370038000000005 2021-03-24T22:02:21.747 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.747 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.747 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:21.747 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.288777+0000", 2021-03-24T22:02:21.748 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000105721 2021-03-24T22:02:21.748 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.748 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.748 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for degraded object", 2021-03-24T22:02:21.748 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.288883+0000", 2021-03-24T22:02:21.749 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023027081000000001 2021-03-24T22:02:21.749 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.749 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.749 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:21.749 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.311910+0000", 2021-03-24T22:02:21.750 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.5148999999999995e-05 2021-03-24T22:02:21.750 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.750 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.750 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for rw locks", 2021-03-24T22:02:21.751 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.311985+0000", 2021-03-24T22:02:21.751 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0058631760000000003 2021-03-24T22:02:21.751 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.751 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.751 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:21.752 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.317848+0000", 2021-03-24T22:02:21.752 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6810000000000004e-05 2021-03-24T22:02:21.752 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.752 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.753 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for rw locks", 2021-03-24T22:02:21.753 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.317935+0000", 2021-03-24T22:02:21.753 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0027239529999999999 2021-03-24T22:02:21.753 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.753 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.754 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:21.754 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.320659+0000", 2021-03-24T22:02:21.754 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010632399999999999 2021-03-24T22:02:21.754 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.754 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.755 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:02:21.755 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.320765+0000", 2021-03-24T22:02:21.755 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000230399 2021-03-24T22:02:21.755 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:21.756 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:21.756 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:02:21.756 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:00:04.320996+0000", 2021-03-24T22:02:21.756 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.025976462 2021-03-24T22:02:21.756 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:21.757 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:02:21.757 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:21.757 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:21.757 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:02:21.757 INFO:teuthology.orchestra.run.smithi142.stdout:} 2021-03-24T22:02:21.758 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:21.759 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:21 smithi125 conmon[24927]: audit 2021-03-24T22:02: 2021-03-24T22:02:21.760 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:21 smithi125 conmon[24927]: 20.648999+0000 mon.c (mon.2) 88 : audit [INF] from='client.? v1:172.21.15.125:0/940854627' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2021-03-24T22:02:21.760 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:21 smithi125 conmon[24927]: audit 2021-03-24 2021-03-24T22:02:21.760 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:21 smithi125 conmon[24927]: T22:02:20.649466+0000 mon.a (mon. 2021-03-24T22:02:21.760 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:21 smithi125 conmon[24927]: 0) 1589 : audit [INF] from='client.35459 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2021-03-24T22:02:21.760 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:21 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:02:21.761 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:21 smithi125 conmon[24927]: 03-24T22:02:20.729412+0000 osd.1 ( 2021-03-24T22:02:21.761 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:21 smithi125 conmon[24927]: osd.1) 52 : cluster [DBG] 2.2 scrub ok 2021-03-24T22:02:21.761 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:21 smithi125 conmon[24927]: audit 2021 2021-03-24T22:02:21.761 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:21 smithi125 conmon[24927]: -03-24T22:02:21.302659+0000 mon.a (mon 2021-03-24T22:02:21.762 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:21 smithi125 conmon[24927]: .0) 1590 : audit [INF] from='client.35459 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2021-03-24T22:02:21.762 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:21 smithi125 conmon[24927]: cluster 2021-03-24T22:02:21.762 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:21 smithi125 conmon[24927]: 2021-03-24T22:02:21.302886+0000 2021-03-24T22:02:21.762 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:21 smithi125 conmon[24927]: mon.a (mon.0) 1591 : cluster [DBG] osdmap e571: 8 total, 8 up, 8 in 2021-03-24T22:02:21.762 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:21 smithi125 conmon[24927]: 2021-03-24T22:02:21.763 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:21 smithi125 conmon[19936]: audit 2021-03-24T 2021-03-24T22:02:21.763 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:21 smithi125 conmon[19936]: 22:02:20.648999+0000 mon.c (mon.2) 88 : audit [INF] from='client.? v1:172.21.15.125:0/940854627' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2021-03-24T22:02:21.763 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:21 smithi125 conmon[19936]: audit 2021-03-24T22:02:21.764 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:21 smithi125 conmon[19936]: 2021-03- 2021-03-24T22:02:21.764 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:21 smithi125 conmon[19936]: 24T22:02:20. 2021-03-24T22:02:21.764 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:21 smithi125 conmon[19936]: 649466+0000 mon.a (mon.0 2021-03-24T22:02:21.764 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:21 smithi125 conmon[19936]: ) 1589 : audit [INF] 2021-03-24T22:02:21.764 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:21 smithi125 conmon[19936]: from='client.35459 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2021-03-24T22:02:21.765 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:21 smithi125 conmon[19936]: cluster 2021-03-24T22:02:21.765 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:21 smithi125 conmon[19936]: 2021-03-24T22: 2021-03-24T22:02:21.765 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:21 smithi125 conmon[19936]: 02:20.729412+0000 osd.1 2021-03-24T22:02:21.765 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:21 smithi125 conmon[19936]: (osd.1) 52 : cluster 2021-03-24T22:02:21.765 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:21 smithi125 conmon[19936]: [DBG] 2.2 scrub ok 2021-03-24T22:02:21.766 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:21 smithi125 conmon[19936]: audit 2021 2021-03-24T22:02:21.766 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:21 smithi125 conmon[19936]: -03-24T22: 2021-03-24T22:02:21.766 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:21 smithi125 conmon[19936]: 02:21.302659+0000 mon.a ( 2021-03-24T22:02:21.766 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:21 smithi125 conmon[19936]: mon.0) 1590 : audit 2021-03-24T22:02:21.767 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:21 smithi125 conmon[19936]: [INF] from='client.35459 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2021-03-24T22:02:21.767 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:21 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:02:21.767 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:21 smithi125 conmon[19936]: -03-24T22: 2021-03-24T22:02:21.767 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:21 smithi125 conmon[19936]: 02:21.302886+0000 mon.a ( 2021-03-24T22:02:21.767 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:21 smithi125 conmon[19936]: mon.0) 1591 : cluster [DBG] 2021-03-24T22:02:21.768 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:21 smithi125 conmon[19936]: osdmap e571: 8 total, 8 up, 8 in 2021-03-24T22:02:21.771 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:21 smithi142 conmon[20125]: audit 2021-03-24T22: 2021-03-24T22:02:21.771 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:21 smithi142 conmon[20125]: 02:20.648999+0000 mon.c (mon.2) 88 : audit [INF] from='client.? v1:172.21.15.125:0/940854627' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2021-03-24T22:02:21.771 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:21 smithi142 conmon[20125]: audit 2021-03-24T22:02:20.649466+0000 mon.a (mon.0) 1589 2021-03-24T22:02:21.771 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:21 smithi142 conmon[20125]: : audit [INF] from='client.35459 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2021-03-24T22:02:21.772 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:21 smithi142 conmon[20125]: cluster 2021-03-24T22:02:20.729412 2021-03-24T22:02:21.772 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:21 smithi142 conmon[20125]: +0000 osd.1 (osd.1) 52 : cluster [DBG] 2021-03-24T22:02:21.772 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:21 smithi142 conmon[20125]: 2.2 scrub ok 2021-03-24T22:02:21.772 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:21 smithi142 conmon[20125]: audit 2021-03-24T22 2021-03-24T22:02:21.773 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:21 smithi142 conmon[20125]: :02:21.302659+0000 mon.a (mon.0 2021-03-24T22:02:21.773 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:21 smithi142 conmon[20125]: ) 1590 : audit [INF] from='client.35459 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2021-03-24T22:02:21.773 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:21 smithi142 conmon[20125]: cluster 2021-03-24T22:02 2021-03-24T22:02:21.773 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:21 smithi142 conmon[20125]: :21.302886+0000 mon.a (mon.0) 1591 : cluster 2021-03-24T22:02:21.773 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:21 smithi142 conmon[20125]: [DBG] osdmap e571: 8 total, 8 up, 8 in 2021-03-24T22:02:21.818 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:21.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:21.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:02:21.924 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:21.939 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-24T22:02:21.980 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:21.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:21.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:02:22.087 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:22.139 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:22.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:22.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:22.244 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:22.312 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:22.313 INFO:tasks.rados.rados.0.smithi125.stdout:2140: done (5 left) 2021-03-24T22:02:22.313 INFO:tasks.rados.rados.0.smithi125.stdout:2141: write oid 33 current snap is 241 2021-03-24T22:02:22.313 INFO:tasks.rados.rados.0.smithi125.stdout:2141: seq_num 749 ranges {588443=564095,1796660=708921} 2021-03-24T22:02:22.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:22.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:02:22.323 INFO:tasks.rados.rados.0.smithi125.stdout:2141: writing smithi12556470-33 from 588443 to 1152538 tid 1 2021-03-24T22:02:22.333 INFO:tasks.rados.rados.0.smithi125.stdout:2141: writing smithi12556470-33 from 1796660 to 2505581 tid 2 2021-03-24T22:02:22.333 INFO:tasks.rados.rados.0.smithi125.stdout:2142: write oid 15 current snap is 241 2021-03-24T22:02:22.333 INFO:tasks.rados.rados.0.smithi125.stdout:2136: finishing write tid 3 to smithi12556470-35 2021-03-24T22:02:22.334 INFO:tasks.rados.rados.0.smithi125.stdout:2142: seq_num 750 ranges {605250=639344,1855142=452579} 2021-03-24T22:02:22.340 INFO:tasks.rados.rados.0.smithi125.stdout:2142: writing smithi12556470-15 from 605250 to 1244594 tid 1 2021-03-24T22:02:22.346 INFO:tasks.rados.rados.0.smithi125.stdout:2142: writing smithi12556470-15 from 1855142 to 2307721 tid 2 2021-03-24T22:02:22.347 INFO:tasks.rados.rados.0.smithi125.stdout:2143: read oid 3 snap -1 2021-03-24T22:02:22.347 INFO:tasks.rados.rados.0.smithi125.stdout:2143: expect deleted 2021-03-24T22:02:22.347 INFO:tasks.rados.rados.0.smithi125.stdout:2144: read oid 6 snap -1 2021-03-24T22:02:22.348 INFO:tasks.rados.rados.0.smithi125.stdout:2144: expect deleted 2021-03-24T22:02:22.348 INFO:tasks.rados.rados.0.smithi125.stdout:2145: write (excl) oid 10 current snap is 241 2021-03-24T22:02:22.348 INFO:tasks.rados.rados.0.smithi125.stdout:2136: finishing write tid 4 to smithi12556470-35 2021-03-24T22:02:22.348 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 35 v 216 (ObjNum 746 snap 241 seq_num 746) dirty exists 2021-03-24T22:02:22.349 INFO:tasks.rados.rados.0.smithi125.stdout:2136: left oid 35 (ObjNum 746 snap 241 seq_num 746) 2021-03-24T22:02:22.349 INFO:tasks.rados.rados.0.smithi125.stdout:2145: seq_num 751 ranges {622057=714593,1913624=708433} 2021-03-24T22:02:22.354 INFO:tasks.rados.rados.0.smithi125.stdout:2145: writing smithi12556470-10 from 622057 to 1336650 tid 1 2021-03-24T22:02:22.361 INFO:tasks.rados.rados.0.smithi125.stdout:2145: writing smithi12556470-10 from 1913624 to 2622057 tid 2 2021-03-24T22:02:22.362 INFO:tasks.rados.rados.0.smithi125.stdout:2136: done (9 left) 2021-03-24T22:02:22.362 INFO:tasks.rados.rados.0.smithi125.stdout:2146: snap_remove snap 203 2021-03-24T22:02:22.424 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:22.464 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:22 smithi125 conmon[19936]: cluster 2021-03-24T22:02:22.464 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:22 smithi125 conmon[19936]: 2021-03-24T22:02:22.465 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:22 smithi125 conmon[19936]: 2021-03-24T22:02:20.901265+0000 mgr.y (mgr.14144) 557 : 2021-03-24T22:02:22.465 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:22 smithi125 conmon[19936]: cluster [DBG] pgmap v1018: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 523 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.3 MiB/s wr, 10 op/s; 51/1707 objects misplaced (2.988%) 2021-03-24T22:02:22.465 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:22 smithi125 conmon[19936]: cluster 2021-03-24T22:02:21.678248+0000 osd.6 (osd.6) 2021-03-24T22:02:22.465 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:22 smithi125 conmon[19936]: 37 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:02:22.466 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:22 smithi125 conmon[19936]: cluster 2021-03-24T22:02:21 2021-03-24T22:02:22.466 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:22 smithi125 conmon[19936]: .680162+0000 osd.6 (osd.6) 38 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:02:22.466 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:22 smithi125 conmon[19936]: cluster 2021-03-24T22:02:21.716439+0000 osd.2 (osd. 2021-03-24T22:02:22.466 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:22 smithi125 conmon[19936]: 2) 90 : cluster [DBG] 5.a scrub ok 2021-03-24T22:02:22.467 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:22 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:02:22.467 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:22 smithi125 conmon[19936]: 03-24T22:02:21.766073+0000 osd.6 (osd.6) 39 : cluster [DBG] 7.12 scrub ok 2021-03-24T22:02:22.467 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:22 smithi125 conmon[19936]: cluster 2021-03-24T22:02:22.300950+0000 mon.a (mon 2021-03-24T22:02:22.467 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:22 smithi125 conmon[19936]: .0) 1592 : cluster [DBG] osdmap e572: 8 total, 8 up, 8 in 2021-03-24T22:02:22.468 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:22 smithi125 conmon[24927]: cluster 2021-03-24T22:02: 2021-03-24T22:02:22.468 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:22 smithi125 conmon[24927]: 20.901265+0000 mgr.y (mgr.14144) 557 : cluster [DBG] pgmap v1018: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 523 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.3 MiB/s wr, 10 op/s; 51/1707 objects misplaced (2.988%) 2021-03-24T22:02:22.468 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:22 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T22:02:22.469 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:22 smithi125 conmon[24927]: T22:02:21.678248+0000 osd.6 ( 2021-03-24T22:02:22.469 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:22 smithi125 conmon[24927]: osd.6) 37 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:02:22.469 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:22 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:02:22.469 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:22 smithi125 conmon[24927]: 03-24T22:02:21.680162+0000 2021-03-24T22:02:22.470 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:22 smithi125 conmon[24927]: osd.6 (osd.6) 38 : cluster [DBG] 2021-03-24T22:02:22.470 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:22 smithi125 conmon[24927]: purged_snaps scrub ok 2021-03-24T22:02:22.470 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:22 smithi125 conmon[24927]: cluster 2021-03-24T22:02: 2021-03-24T22:02:22.470 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:22 smithi125 conmon[24927]: 21.716439+0000 osd.2 (osd.2) 2021-03-24T22:02:22.471 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:22 smithi125 conmon[24927]: 90 : cluster [DBG] 5.a scrub ok 2021-03-24T22:02:22.471 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:22 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:02:22.471 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:22 smithi125 conmon[24927]: 24T22:02:21.766073+0000 2021-03-24T22:02:22.471 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:22 smithi125 conmon[24927]: osd.6 (osd.6) 39 : cluster [DBG] 2021-03-24T22:02:22.471 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:22 smithi125 conmon[24927]: 7.12 scrub ok 2021-03-24T22:02:22.472 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:22 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:02:22.472 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:22 smithi125 conmon[24927]: 02:22.300950+0000 mon.a (mon. 2021-03-24T22:02:22.472 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:22 smithi125 conmon[24927]: 0) 1592 : cluster [DBG] osdmap e572: 8 total, 8 up, 8 in 2021-03-24T22:02:22.474 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:22.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:22.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:02:22.508 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd reweight 3 1 2021-03-24T22:02:22.577 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:22.584 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:02:22.592 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.6 dump_ops_in_flight 2021-03-24T22:02:22.632 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:22.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:22.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:02:22.737 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:22.750 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:22 smithi142 conmon[20125]: cluster 2021-03-24T22:02:20.901265+0000 mgr.y (mgr.14144) 557 : cluster [DBG] 2021-03-24T22:02:22.750 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:22 smithi142 conmon[20125]: pgmap v1018: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 523 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.3 MiB/s wr, 10 op/s; 51/1707 objects misplaced (2.988%) 2021-03-24T22:02:22.751 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:22 smithi142 conmon[20125]: cluster 2021-03-24T22:02:21.678248+0000 osd.6 (osd.6) 37 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:02:22.751 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:22 smithi142 conmon[20125]: cluster 2021-03-24T22 2021-03-24T22:02:22.751 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:22 smithi142 conmon[20125]: :02:21.680162+0000 osd.6 (osd.6) 38 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:02:22.751 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:22 smithi142 conmon[20125]: cluster 2021-03-24T22:02:21. 2021-03-24T22:02:22.752 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:22 smithi142 conmon[20125]: 716439+0000 osd.2 (osd.2) 90 : cluster [DBG] 5.a scrub ok 2021-03-24T22:02:22.752 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:22 smithi142 conmon[20125]: cluster 2021-03-24T22:02:21. 2021-03-24T22:02:22.752 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:22 smithi142 conmon[20125]: 766073+0000 osd.6 (osd.6) 39 : cluster [DBG] 7.12 scrub ok 2021-03-24T22:02:22.752 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:22 smithi142 conmon[20125]: cluster 2021-03-24T22:02:22.753 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:22 smithi142 conmon[20125]: 2021-03-24T22:02:22 2021-03-24T22:02:22.753 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:22 smithi142 conmon[20125]: .300950+0000 mon.a (mon.0) 1592 : cluster [DBG] 2021-03-24T22:02:22.753 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:22 smithi142 conmon[20125]: osdmap e572: 8 total, 8 up, 8 in 2021-03-24T22:02:22.787 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:22.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:22.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:02:22.892 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:22.926 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T22:02:22.949 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:22.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:22.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:02:23.028 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:02:23.057 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:23.106 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:23.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:23.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:02:23.211 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:23.266 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:23.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:23.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:23.315 INFO:tasks.rados.rados.0.smithi125.stdout:2146: done (9 left) 2021-03-24T22:02:23.315 INFO:tasks.rados.rados.0.smithi125.stdout:2147: write oid 12 current snap is 241 2021-03-24T22:02:23.316 INFO:tasks.rados.rados.0.smithi125.stdout:2138: finishing write tid 1 to smithi12556470-1 2021-03-24T22:02:23.316 INFO:tasks.rados.rados.0.smithi125.stdout:2147: seq_num 752 ranges {638864=789842,1972106=456248} 2021-03-24T22:02:23.324 INFO:tasks.rados.rados.0.smithi125.stdout:2147: writing smithi12556470-12 from 638864 to 1428706 tid 1 2021-03-24T22:02:23.331 INFO:tasks.rados.rados.0.smithi125.stdout:2147: writing smithi12556470-12 from 1972106 to 2428354 tid 2 2021-03-24T22:02:23.332 INFO:tasks.rados.rados.0.smithi125.stdout:2148: rollback oid 39 current snap is 241 2021-03-24T22:02:23.333 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 39 to 237 2021-03-24T22:02:23.333 INFO:tasks.rados.rados.0.smithi125.stdout:2137: finishing copy_from to smithi12556470-22 2021-03-24T22:02:23.333 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 22 v 657 (ObjNum 739 snap 239 seq_num 739) dirty exists 2021-03-24T22:02:23.334 INFO:tasks.rados.rados.0.smithi125.stdout:2135: finishing copy_from to smithi12556470-20 2021-03-24T22:02:23.334 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 20 v 501 (ObjNum 743 snap 240 seq_num 743) dirty exists 2021-03-24T22:02:23.334 INFO:tasks.rados.rados.0.smithi125.stdout:2137: finishing copy_from racing read to smithi12556470-22 2021-03-24T22:02:23.334 INFO:tasks.rados.rados.0.smithi125.stdout:2135: finishing copy_from racing read to smithi12556470-20 2021-03-24T22:02:23.335 INFO:tasks.rados.rados.0.smithi125.stdout:2139: finishing write tid 1 to smithi12556470-32 2021-03-24T22:02:23.335 INFO:tasks.rados.rados.0.smithi125.stdout:2138: finishing write tid 2 to smithi12556470-1 2021-03-24T22:02:23.335 INFO:tasks.rados.rados.0.smithi125.stdout:2138: finishing write tid 3 to smithi12556470-1 2021-03-24T22:02:23.335 INFO:tasks.rados.rados.0.smithi125.stdout:2138: finishing write tid 4 to smithi12556470-1 2021-03-24T22:02:23.336 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 1 v 724 (ObjNum 747 snap 241 seq_num 747) dirty exists 2021-03-24T22:02:23.336 INFO:tasks.rados.rados.0.smithi125.stdout:2138: left oid 1 (ObjNum 747 snap 241 seq_num 747) 2021-03-24T22:02:23.336 INFO:tasks.rados.rados.0.smithi125.stdout:2135: done (10 left) 2021-03-24T22:02:23.336 INFO:tasks.rados.rados.0.smithi125.stdout:2137: done (9 left) 2021-03-24T22:02:23.336 INFO:tasks.rados.rados.0.smithi125.stdout:2138: done (8 left) 2021-03-24T22:02:23.337 INFO:tasks.rados.rados.0.smithi125.stdout:2149: write oid 44 current snap is 241 2021-03-24T22:02:23.337 INFO:tasks.rados.rados.0.smithi125.stdout:2139: finishing write tid 2 to smithi12556470-32 2021-03-24T22:02:23.337 INFO:tasks.rados.rados.0.smithi125.stdout:2139: finishing write tid 3 to smithi12556470-32 2021-03-24T22:02:23.337 INFO:tasks.rados.rados.0.smithi125.stdout:2139: finishing write tid 4 to smithi12556470-32 2021-03-24T22:02:23.338 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 32 v 1107 (ObjNum 748 snap 241 seq_num 748) dirty exists 2021-03-24T22:02:23.338 INFO:tasks.rados.rados.0.smithi125.stdout:2139: left oid 32 (ObjNum 748 snap 241 seq_num 748) 2021-03-24T22:02:23.338 INFO:tasks.rados.rados.0.smithi125.stdout:2141: finishing write tid 1 to smithi12556470-33 2021-03-24T22:02:23.338 INFO:tasks.rados.rados.0.smithi125.stdout:2141: finishing write tid 2 to smithi12556470-33 2021-03-24T22:02:23.338 INFO:tasks.rados.rados.0.smithi125.stdout:2141: finishing write tid 3 to smithi12556470-33 2021-03-24T22:02:23.339 INFO:tasks.rados.rados.0.smithi125.stdout:2149: seq_num 753 ranges {655671=581444,2030588=625083} 2021-03-24T22:02:23.339 INFO:tasks.rados.rados.0.smithi125.stdout:2149: writing smithi12556470-44 from 655671 to 1237115 tid 1 2021-03-24T22:02:23.349 INFO:tasks.rados.rados.0.smithi125.stdout:2149: writing smithi12556470-44 from 2030588 to 2655671 tid 2 2021-03-24T22:02:23.350 INFO:tasks.rados.rados.0.smithi125.stdout:2139: done (8 left) 2021-03-24T22:02:23.350 INFO:tasks.rados.rados.0.smithi125.stdout:2150: copy_from oid 50 from oid 47 current snap is 241 2021-03-24T22:02:23.350 INFO:tasks.rados.rados.0.smithi125.stdout:2141: finishing write tid 4 to smithi12556470-33 2021-03-24T22:02:23.350 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 33 v 777 (ObjNum 749 snap 241 seq_num 749) dirty exists 2021-03-24T22:02:23.351 INFO:tasks.rados.rados.0.smithi125.stdout:2141: left oid 33 (ObjNum 749 snap 241 seq_num 749) 2021-03-24T22:02:23.351 INFO:tasks.rados.rados.0.smithi125.stdout:2142: finishing write tid 1 to smithi12556470-15 2021-03-24T22:02:23.351 INFO:tasks.rados.rados.0.smithi125.stdout:2142: finishing write tid 2 to smithi12556470-15 2021-03-24T22:02:23.351 INFO:tasks.rados.rados.0.smithi125.stdout:2142: finishing write tid 3 to smithi12556470-15 2021-03-24T22:02:23.351 INFO:tasks.rados.rados.0.smithi125.stdout:2142: finishing write tid 4 to smithi12556470-15 2021-03-24T22:02:23.352 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 15 v 763 (ObjNum 750 snap 241 seq_num 750) dirty exists 2021-03-24T22:02:23.352 INFO:tasks.rados.rados.0.smithi125.stdout:2142: left oid 15 (ObjNum 750 snap 241 seq_num 750) 2021-03-24T22:02:23.352 INFO:tasks.rados.rados.0.smithi125.stdout:2145: finishing write tid 1 to smithi12556470-10 2021-03-24T22:02:23.352 INFO:tasks.rados.rados.0.smithi125.stdout:2145: finishing write tid 2 to smithi12556470-10 2021-03-24T22:02:23.353 INFO:tasks.rados.rados.0.smithi125.stdout:2145: finishing write tid 3 to smithi12556470-10 2021-03-24T22:02:23.353 INFO:tasks.rados.rados.0.smithi125.stdout:2141: done (8 left) 2021-03-24T22:02:23.353 INFO:tasks.rados.rados.0.smithi125.stdout:2142: done (7 left) 2021-03-24T22:02:23.353 INFO:tasks.rados.rados.0.smithi125.stdout:2143: done (6 left) 2021-03-24T22:02:23.353 INFO:tasks.rados.rados.0.smithi125.stdout:2151: snap_create 2021-03-24T22:02:23.354 INFO:tasks.rados.rados.0.smithi125.stdout:2145: finishing write tid 4 to smithi12556470-10 2021-03-24T22:02:23.354 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 10 v 781 (ObjNum 751 snap 241 seq_num 751) dirty exists 2021-03-24T22:02:23.354 INFO:tasks.rados.rados.0.smithi125.stdout:2145: left oid 10 (ObjNum 751 snap 241 seq_num 751) 2021-03-24T22:02:23.354 INFO:tasks.rados.rados.0.smithi125.stdout:2148: finishing rollback tid 0 to smithi12556470-39 2021-03-24T22:02:23.355 INFO:tasks.rados.rados.0.smithi125.stdout:2148: finishing rollback tid 1 to smithi12556470-39 2021-03-24T22:02:23.355 INFO:tasks.rados.rados.0.smithi125.stdout:2148: finishing rollback tid 2 to smithi12556470-39 2021-03-24T22:02:23.355 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 39 v 789 (ObjNum 716 snap 227 seq_num 716) dirty exists 2021-03-24T22:02:23.355 INFO:tasks.rados.rados.0.smithi125.stdout:2147: finishing write tid 1 to smithi12556470-12 2021-03-24T22:02:23.356 INFO:tasks.rados.rados.0.smithi125.stdout:2147: finishing write tid 2 to smithi12556470-12 2021-03-24T22:02:23.356 INFO:tasks.rados.rados.0.smithi125.stdout:2147: finishing write tid 3 to smithi12556470-12 2021-03-24T22:02:23.356 INFO:tasks.rados.rados.0.smithi125.stdout:2147: finishing write tid 4 to smithi12556470-12 2021-03-24T22:02:23.356 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 12 v 489 (ObjNum 752 snap 241 seq_num 752) dirty exists 2021-03-24T22:02:23.356 INFO:tasks.rados.rados.0.smithi125.stdout:2147: left oid 12 (ObjNum 752 snap 241 seq_num 752) 2021-03-24T22:02:23.357 INFO:tasks.rados.rados.0.smithi125.stdout:2149: finishing write tid 1 to smithi12556470-44 2021-03-24T22:02:23.357 INFO:tasks.rados.rados.0.smithi125.stdout:2149: finishing write tid 2 to smithi12556470-44 2021-03-24T22:02:23.358 INFO:tasks.rados.rados.0.smithi125.stdout:2149: finishing write tid 3 to smithi12556470-44 2021-03-24T22:02:23.358 INFO:tasks.rados.rados.0.smithi125.stdout:2149: finishing write tid 4 to smithi12556470-44 2021-03-24T22:02:23.358 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 44 v 485 (ObjNum 753 snap 241 seq_num 753) dirty exists 2021-03-24T22:02:23.358 INFO:tasks.rados.rados.0.smithi125.stdout:2149: left oid 44 (ObjNum 753 snap 241 seq_num 753) 2021-03-24T22:02:23.365 INFO:tasks.rados.rados.0.smithi125.stdout:2150: finishing copy_from to smithi12556470-50 2021-03-24T22:02:23.366 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 50 v 346 (ObjNum 466 snap 131 seq_num 466) dirty exists 2021-03-24T22:02:23.366 INFO:tasks.rados.rados.0.smithi125.stdout:2150: finishing copy_from racing read to smithi12556470-50 2021-03-24T22:02:23.373 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:23.427 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:23.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:23.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:02:23.534 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:23.593 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:23.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:23.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:02:23.697 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:23.752 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:23.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:23.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:02:23.858 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:23.913 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:23.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:23.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:02:24.018 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:24.080 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:24.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:24.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:02:24.185 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:24.238 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:24.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:24.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:02:24.317 INFO:tasks.rados.rados.0.smithi125.stdout:2144: done (6 left) 2021-03-24T22:02:24.318 INFO:tasks.rados.rados.0.smithi125.stdout:2145: done (5 left) 2021-03-24T22:02:24.318 INFO:tasks.rados.rados.0.smithi125.stdout:2147: done (4 left) 2021-03-24T22:02:24.318 INFO:tasks.rados.rados.0.smithi125.stdout:2148: done (3 left) 2021-03-24T22:02:24.318 INFO:tasks.rados.rados.0.smithi125.stdout:2149: done (2 left) 2021-03-24T22:02:24.319 INFO:tasks.rados.rados.0.smithi125.stdout:2150: done (1 left) 2021-03-24T22:02:24.319 INFO:tasks.rados.rados.0.smithi125.stdout:2151: done (0 left) 2021-03-24T22:02:24.319 INFO:tasks.rados.rados.0.smithi125.stdout:2152: rollback oid 5 current snap is 242 2021-03-24T22:02:24.319 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 5 to 216 2021-03-24T22:02:24.320 INFO:tasks.rados.rados.0.smithi125.stdout:2153: snap_remove snap 230 2021-03-24T22:02:24.345 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:24.401 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:24.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:24.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:02:24.474 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-24T22:02:24.483 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:24 smithi125 conmon[19936]: cluster 2021-03-24T22:02:22.796154+0000 osd.6 (osd.6) 40 : cluster [DBG] 3.11 scrub ok 2021-03-24T22:02:24.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:24 smithi125 conmon[19936]: cluster 2021-03-24T22:02:22.902820+0000 mgr.y (mgr.14144) 558 : cluster [DBG] pgmap v1021: 168 pgs: 4 peering, 2 active+remapped+backfill_toofull, 162 active+clean; 523 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.6 MiB/s wr, 11 op/s; 51/1707 objects misplaced (2.988%) 2021-03-24T22:02:24.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:24 smithi125 conmon[19936]: cluster 2021-03-24T22:02:23.307107+0000 mon.a (mon.0) 1593 : cluster [DBG] osdmap e573: 8 total, 8 up, 8 in 2021-03-24T22:02:24.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:24 smithi125 conmon[19936]: cluster 2021-03-24T22:02:23.312493+0000 osd.0 (osd.0) 119 : cluster [DBG] 4.6 starting backfill to osd.2 from (0'0,0'0] MAX to 544'34 2021-03-24T22:02:24.485 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:24 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:02:24.485 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:24 smithi125 conmon[24927]: 02:22.796154+0000 osd.6 (osd.6) 40 : cluster [DBG] 3.11 scrub ok 2021-03-24T22:02:24.485 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:24 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:02:24.486 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:24 smithi125 conmon[24927]: -03-24T22:02 2021-03-24T22:02:24.486 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:24 smithi125 conmon[24927]: :22.902820+0000 mgr.y ( 2021-03-24T22:02:24.486 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:24 smithi125 conmon[24927]: mgr.14144) 558 : cluster 2021-03-24T22:02:24.486 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:24 smithi125 conmon[24927]: [DBG] pgmap v1021: 168 pgs: 4 peering, 2 active+remapped+backfill_toofull, 162 active+clean; 523 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.6 MiB/s wr, 11 op/s; 51/1707 objects misplaced (2.988%) 2021-03-24T22:02:24.487 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:24 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:02:24.487 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:24 smithi125 conmon[24927]: -03-24T22:02 2021-03-24T22:02:24.487 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:24 smithi125 conmon[24927]: :23.307107+0000 mon.a ( 2021-03-24T22:02:24.487 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:24 smithi125 conmon[24927]: mon.0) 1593 : cluster 2021-03-24T22:02:24.488 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:24 smithi125 conmon[24927]: [DBG] osdmap e573: 8 total, 8 up, 8 in 2021-03-24T22:02:24.488 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:24 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:02:24.488 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:24 smithi125 conmon[24927]: -03-24T22: 2021-03-24T22:02:24.488 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:24 smithi125 conmon[24927]: 02:23.312493+0000 osd.0 2021-03-24T22:02:24.489 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:24 smithi125 conmon[24927]: (osd.0) 119 : cluster 2021-03-24T22:02:24.489 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:24 smithi125 conmon[24927]: [DBG] 4.6 starting backfill to osd.2 from (0'0,0'0] MAX to 544'34 2021-03-24T22:02:24.508 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:24.565 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:24.567 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:24 smithi142 conmon[20125]: cluster 2021-03-24T22:02:22.796154+0000 osd.6 (osd.6) 40 : cluster 2021-03-24T22:02:24.568 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:24 smithi142 conmon[20125]: [DBG] 3.11 scrub ok 2021-03-24T22:02:24.568 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:24 smithi142 conmon[20125]: cluster 2021-03-24T22:02:22.902820+0000 mgr.y (mgr.14144) 2021-03-24T22:02:24.568 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:24 smithi142 conmon[20125]: 558 : cluster 2021-03-24T22:02:24.568 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:24 smithi142 conmon[20125]: [DBG] pgmap v1021: 168 pgs: 4 peering, 2 active+remapped+backfill_toofull, 162 active+clean; 523 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.6 MiB/s wr, 11 op/s; 51/1707 objects misplaced (2.988%) 2021-03-24T22:02:24.569 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:24 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:02:24.569 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:24 smithi142 conmon[20125]: T22:02:23.307107+0000 mon.a (mon.0) 1593 : cluster 2021-03-24T22:02:24.569 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:24 smithi142 conmon[20125]: [DBG] osdmap e573: 8 total, 8 up, 8 in 2021-03-24T22:02:24.569 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:24 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:02:24.570 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:24 smithi142 conmon[20125]: 24T22:02:23.312493+0000 osd.0 2021-03-24T22:02:24.570 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:24 smithi142 conmon[20125]: (osd.0) 119 : cluster [DBG] 4.6 starting backfill to osd.2 from (0'0,0'0] MAX to 544'34 2021-03-24T22:02:24.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:24.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:02:24.672 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:24.723 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:24.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:24.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:02:24.827 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:24.885 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:24.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:24.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:02:24.991 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:25.029 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2021-03-24T22:02:25.029 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2021-03-24T22:02:25.030 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2021-03-24T22:02:25.030 INFO:teuthology.orchestra.run.smithi142.stdout:} 2021-03-24T22:02:25.045 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:25.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:25.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:02:25.151 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:25.196 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:25.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:25.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:25.301 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:25.306 INFO:tasks.rados.rados.0.smithi125.stdout:2153: done (1 left) 2021-03-24T22:02:25.306 INFO:tasks.rados.rados.0.smithi125.stdout:2154: snap_remove snap 209 2021-03-24T22:02:25.352 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:25.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:25.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:02:25.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:25 smithi125 conmon[24927]: cluster 2021-03-24T22:02:24.313400+0000 mon.a ( 2021-03-24T22:02:25.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:25 smithi125 conmon[24927]: mon.0) 1594 : cluster [DBG] osdmap e574: 8 total, 8 up, 8 in 2021-03-24T22:02:25.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:25 smithi125 conmon[24927]: cluster 2021-03-24T22:02:25.298863+0000 mon.a (mon.0) 1595 : cluster 2021-03-24T22:02:25.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:25 smithi125 conmon[24927]: [DBG] osdmap e575: 8 total, 8 up, 8 in 2021-03-24T22:02:25.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:25 smithi125 conmon[19936]: cluster 2021-03-24T22:02:24.313400+0000 mon.a (mon.0) 1594 2021-03-24T22:02:25.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:25 smithi125 conmon[19936]: : cluster [DBG] osdmap e574: 8 total, 8 up, 8 in 2021-03-24T22:02:25.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:25 smithi125 conmon[19936]: cluster 2021-03-24T22:02:25.298863+0000 mon.a (mon.0) 1595 : cluster [DBG] osdmap e575: 8 total, 8 up, 8 in 2021-03-24T22:02:25.457 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:25.472 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:02:25.506 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:25.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:25.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:02:25.612 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:25.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:25 smithi142 conmon[20125]: cluster 2021-03-24T22:02:25.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:25 smithi142 conmon[20125]: 2021-03-24T22:02:25.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:25 smithi142 conmon[20125]: 2021 2021-03-24T22:02:25.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:25 smithi142 conmon[20125]: - 2021-03-24T22:02:25.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:25 smithi142 conmon[20125]: 03 2021-03-24T22:02:25.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:25 smithi142 conmon[20125]: - 2021-03-24T22:02:25.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:25 smithi142 conmon[20125]: 24 2021-03-24T22:02:25.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:25 smithi142 conmon[20125]: T 2021-03-24T22:02:25.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:25 smithi142 conmon[20125]: 22 2021-03-24T22:02:25.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:25 smithi142 conmon[20125]: : 2021-03-24T22:02:25.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:25 smithi142 conmon[20125]: 02 2021-03-24T22:02:25.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:25 smithi142 conmon[20125]: : 2021-03-24T22:02:25.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:25 smithi142 conmon[20125]: 24 2021-03-24T22:02:25.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:25 smithi142 conmon[20125]: . 2021-03-24T22:02:25.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:25 smithi142 conmon[20125]: 313400 2021-03-24T22:02:25.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:25 smithi142 conmon[20125]: +0000 2021-03-24T22:02:25.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:25 smithi142 conmon[20125]: 2021-03-24T22:02:25.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:25 smithi142 conmon[20125]: mon.a 2021-03-24T22:02:25.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:25 smithi142 conmon[20125]: ( 2021-03-24T22:02:25.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:25 smithi142 conmon[20125]: mon 2021-03-24T22:02:25.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:25 smithi142 conmon[20125]: . 2021-03-24T22:02:25.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:25 smithi142 conmon[20125]: 0 2021-03-24T22:02:25.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:25 smithi142 conmon[20125]: ) 2021-03-24T22:02:25.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:25 smithi142 conmon[20125]: 1594 2021-03-24T22:02:25.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:25 smithi142 conmon[20125]: : 2021-03-24T22:02:25.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:25 smithi142 conmon[20125]: cluster [DBG] osdmap e574: 8 total, 8 up, 8 in 2021-03-24T22:02:25.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:25 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:02:25.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:25 smithi142 conmon[20125]: 22:02:25.298863+0000 mon.a (mon.0) 1595 : cluster [DBG] 2021-03-24T22:02:25.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:25 smithi142 conmon[20125]: osdmap e575: 8 total, 8 up, 8 in 2021-03-24T22:02:25.672 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:25.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:25.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:02:25.780 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:25.836 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:25.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:25.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:02:25.944 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:25.987 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:25.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:25.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:02:26.092 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:26.141 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.6 dump_blocked_ops 2021-03-24T22:02:26.154 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:26.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:26.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:02:26.260 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:26.306 INFO:tasks.rados.rados.0.smithi125.stdout:2154: done (1 left) 2021-03-24T22:02:26.307 INFO:tasks.rados.rados.0.smithi125.stdout:2155: snap_create 2021-03-24T22:02:26.307 INFO:tasks.rados.rados.0.smithi125.stdout:2152: finishing rollback tid 1 to smithi12556470-5 2021-03-24T22:02:26.307 INFO:tasks.rados.rados.0.smithi125.stdout:2152: finishing rollback tid 2 to smithi12556470-5 2021-03-24T22:02:26.307 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 5 v 648 (ObjNum 652 snap 198 seq_num 652) dirty exists 2021-03-24T22:02:26.313 INFO:teuthology.orchestra.run.smithi125.stderr:reweighted osd.3 to 1 (10000) 2021-03-24T22:02:26.324 INFO:teuthology.orchestra.run.smithi125.stderr:nodeep-scrub is unset 2021-03-24T22:02:26.329 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:26.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:26.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:02:26.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:26 smithi142 conmon[20125]: cluster 2021-03-24T22:02:24.903511+0000 mgr.y (mgr 2021-03-24T22:02:26.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:26 smithi142 conmon[20125]: .14144) 559 : cluster [DBG] pgmap v1024: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 4 peering, 3 active+remapped+backfill_toofull, 160 active+clean; 531 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 3.2 MiB/s wr, 10 op/s; 2/1716 objects degraded (0.117%); 78/1716 objects misplaced (4.545%); 0 B/s, 2 objects/s recovering 2021-03-24T22:02:26.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:26 smithi142 conmon[20125]: cluster 2021-03-24T22:02:25.310363+0000 mon.a (mon.0) 1596 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:02:26.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:26 smithi142 conmon[20125]: cluster 2021-03-24T22:02:25 2021-03-24T22:02:26.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:26 smithi142 conmon[20125]: .310396+0000 mon.a (mon.0) 1597 : cluster [WRN] Health check failed: Degraded data redundancy: 2/1716 objects degraded (0.117%), 1 pg degraded (PG_DEGRADED) 2021-03-24T22:02:26.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:26 smithi142 conmon[20125]: audit 2021-03-24 2021-03-24T22:02:26.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:26 smithi142 conmon[20125]: T22:02:26.002637+0000 mon.a (mon. 2021-03-24T22:02:26.368 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:26 smithi142 conmon[20125]: 0) 1598 : audit [INF] from='client.? v1:172.21.15.125:0/4123199439' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:02:26.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:26 smithi142 conmon[20125]: audit 2021-03-24T22:02: 2021-03-24T22:02:26.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:26 smithi142 conmon[20125]: 26.254746+0000 mon.a (mon.0) 1599 : audit 2021-03-24T22:02:26.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:26 smithi142 conmon[20125]: [INF] from='client.? v1:172.21.15.125:0/4200642048' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2021-03-24T22:02:26.369 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:26 smithi142 conmon[20125]: cluster 2021-03-24T22:02 2021-03-24T22:02:26.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:26 smithi142 conmon[20125]: :26.294172+0000 mon.a (mon.0) 1600 2021-03-24T22:02:26.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:26 smithi142 conmon[20125]: : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T22:02:26.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:26 smithi142 conmon[20125]: audit 2021-03-24T22:02:26.370 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:26 smithi142 conmon[20125]: 2021-03-24T22:02:26.298602+0000 mon.a (mon. 2021-03-24T22:02:26.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:26 smithi142 conmon[20125]: 0) 1601 : audit [INF] from='client.? v1:172.21.15.125:0/4123199439' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T22:02:26.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:26 smithi142 conmon[20125]: audit 2021-03-24T 2021-03-24T22:02:26.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:26 smithi142 conmon[20125]: 22:02:26.298674+0000 mon.a (mon. 2021-03-24T22:02:26.371 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:26 smithi142 conmon[20125]: 0) 1602 : audit [INF] from='client.? v1:172.21.15.125:0/4200642048' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2021-03-24T22:02:26.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:26 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:02:26.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:26 smithi142 conmon[20125]: 22:02:26.298706+0000 mon.a (mon.0 2021-03-24T22:02:26.372 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:26 smithi142 conmon[20125]: ) 1603 : cluster [DBG] osdmap e576: 8 total, 8 up, 8 in 2021-03-24T22:02:26.433 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:26.442 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T22:02:26.478 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:26.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:26.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:02:26.483 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[19936]: cluster 2021-03-24T22:02:24.903511+0000 2021-03-24T22:02:26.483 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[19936]: mgr.y (mgr.14144) 559 : cluster [DBG] pgmap v1024: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 4 peering, 3 active+remapped+backfill_toofull, 160 active+clean; 531 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 3.2 MiB/s wr, 10 op/s; 2/1716 objects degraded (0.117%); 78/1716 objects misplaced (4.545%); 0 B/s, 2 objects/s recovering 2021-03-24T22:02:26.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:02:26.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[19936]: 24 2021-03-24T22:02:26.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[19936]: T22:02:25.310363 2021-03-24T22:02:26.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[19936]: +0000 mon.a (mon.0) 1596 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:02:26.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:02:26.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[19936]: 03-24T22:02:25.310396+0000 mon.a 2021-03-24T22:02:26.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[19936]: (mon.0) 1597 : cluster [WRN] Health check failed: Degraded data redundancy: 2/1716 objects degraded (0.117%), 1 pg degraded (PG_DEGRADED) 2021-03-24T22:02:26.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[19936]: 2021-03-24T22:02:26.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[19936]: audit 2021-03-24T22:02:26.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[19936]: 2021-03-24T22:02:26.002637+0000 mon.a 2021-03-24T22:02:26.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[19936]: (mon.0) 1598 : audit [INF] from='client.? v1:172.21.15.125:0/4123199439' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:02:26.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[19936]: 2021-03-24T22:02:26.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[19936]: audit 2021-03-24T22: 2021-03-24T22:02:26.487 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[19936]: 02:26.254746+0000 mon.a (mon. 2021-03-24T22:02:26.487 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[19936]: 0) 1599 : audit [INF] from='client.? v1:172.21.15.125:0/4200642048' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2021-03-24T22:02:26.487 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:02:26.487 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[19936]: -03-24T22:02:26.294172 2021-03-24T22:02:26.488 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[19936]: +0000 mon.a (mon.0) 1600 : cluster 2021-03-24T22:02:26.488 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[19936]: [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T22:02:26.488 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[19936]: audit 2021-03 2021-03-24T22:02:26.488 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[19936]: -24T22:02:26.298602+0000 mon.a ( 2021-03-24T22:02:26.488 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[19936]: mon.0) 1601 : audit [INF] from='client.? v1:172.21.15.125:0/4123199439' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T22:02:26.489 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[19936]: 2021-03-24T22:02:26.489 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[19936]: audit 2021-03-24T22 2021-03-24T22:02:26.489 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[19936]: :02:26.298674+0000 mon.a (mon. 2021-03-24T22:02:26.489 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[19936]: 0) 1602 : audit [INF] from='client.? v1:172.21.15.125:0/4200642048' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2021-03-24T22:02:26.490 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[19936]: cluster 2021-03-24T22:02:26.490 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[19936]: 2021-03-24T22:02:26. 2021-03-24T22:02:26.490 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[19936]: 298706+0000 mon.a (mon.0) 1603 : 2021-03-24T22:02:26.491 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[19936]: cluster [DBG] osdmap e576: 8 total, 8 up, 8 in 2021-03-24T22:02:26.491 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: cluster 2021-03-24T22:02:26.491 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: 2021-03-24T22:02:26.492 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: 2021 2021-03-24T22:02:26.492 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: - 2021-03-24T22:02:26.492 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: 03 2021-03-24T22:02:26.493 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: - 2021-03-24T22:02:26.493 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: 24 2021-03-24T22:02:26.493 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: T 2021-03-24T22:02:26.493 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: 22 2021-03-24T22:02:26.493 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: :02:24.903511+0000 mgr.y (mgr.14144) 559 2021-03-24T22:02:26.494 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: : cluster [DBG] pgmap v1024: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 4 peering, 3 active+remapped+backfill_toofull, 160 active+clean; 531 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 3.2 MiB/s wr, 10 op/s; 2/1716 objects degraded (0.117%); 78/1716 objects misplaced (4.545%); 0 B/s, 2 objects/s recovering 2021-03-24T22:02:26.494 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: cluster 2021-03-24T22:02:26.494 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: 2021-03-24T22 2021-03-24T22:02:26.494 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: :02:25.310363+0000 mon.a (mon.0) 1596 : cluster [WRN] 2021-03-24T22:02:26.495 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:02:26.495 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:02:26.495 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: 03-24T22:02:25.310396+0000 mon.a (mon 2021-03-24T22:02:26.495 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: .0) 1597 : cluster [WRN] Health check failed: Degraded data redundancy: 2/1716 objects degraded (0.117%), 1 pg degraded (PG_DEGRADED) 2021-03-24T22:02:26.496 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: audit 2021- 2021-03-24T22:02:26.496 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: 03-24T22:02:26.002637+0000 mon.a ( 2021-03-24T22:02:26.496 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: mon.0) 1598 : audit [INF] from='client.? v1:172.21.15.125:0/4123199439' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:02:26.496 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: audit 2021-03-24T22:02:26.497 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: 2021-03-24T22:02:26.497 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: 2021 2021-03-24T22:02:26.497 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: - 2021-03-24T22:02:26.497 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: 03 2021-03-24T22:02:26.497 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: - 2021-03-24T22:02:26.498 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: 24 2021-03-24T22:02:26.498 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: T 2021-03-24T22:02:26.498 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: 22 2021-03-24T22:02:26.498 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: : 2021-03-24T22:02:26.498 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: 02 2021-03-24T22:02:26.499 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: : 2021-03-24T22:02:26.499 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: 26 2021-03-24T22:02:26.499 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: . 2021-03-24T22:02:26.499 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: 254746 2021-03-24T22:02:26.500 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: +0000 2021-03-24T22:02:26.500 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: 2021-03-24T22:02:26.500 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: mon.a 2021-03-24T22:02:26.500 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: ( 2021-03-24T22:02:26.500 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: mon 2021-03-24T22:02:26.501 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: . 2021-03-24T22:02:26.501 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: 0 2021-03-24T22:02:26.501 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: ) 2021-03-24T22:02:26.501 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: 1599 2021-03-24T22:02:26.502 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: : 2021-03-24T22:02:26.502 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: audit 2021-03-24T22:02:26.502 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: 2021-03-24T22:02:26.502 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: [INF] 2021-03-24T22:02:26.502 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: 2021-03-24T22:02:26.503 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: from='client.? v1:172.21.15.125:0/4200642048' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2021-03-24T22:02:26.503 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: 2021-03-24T22:02:26.503 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: cluster 2021-03-24T22:02:26.503 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: 2021-03-24T22:02:26.504 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: 2021 2021-03-24T22:02:26.504 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: - 2021-03-24T22:02:26.504 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: 03 2021-03-24T22:02:26.504 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: - 2021-03-24T22:02:26.504 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: 24 2021-03-24T22:02:26.505 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: T 2021-03-24T22:02:26.505 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: 22 2021-03-24T22:02:26.505 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: : 2021-03-24T22:02:26.505 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: 02 2021-03-24T22:02:26.505 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: : 2021-03-24T22:02:26.506 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: 26 2021-03-24T22:02:26.506 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: . 2021-03-24T22:02:26.506 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: 294172 2021-03-24T22:02:26.506 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: +0000 2021-03-24T22:02:26.507 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: 2021-03-24T22:02:26.507 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: mon.a 2021-03-24T22:02:26.507 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: ( 2021-03-24T22:02:26.507 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: mon 2021-03-24T22:02:26.508 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: . 2021-03-24T22:02:26.508 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: 0 2021-03-24T22:02:26.508 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: ) 2021-03-24T22:02:26.508 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: 1600 2021-03-24T22:02:26.509 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: : 2021-03-24T22:02:26.509 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: cluster 2021-03-24T22:02:26.509 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: 2021-03-24T22:02:26.509 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: [INF] 2021-03-24T22:02:26.509 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: 2021-03-24T22:02:26.510 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T22:02:26.510 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: 2021-03-24T22:02:26.510 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: audit 2021-03-24T22:02:26.510 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: 2021-03-24T22:02:26.511 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: 2021 2021-03-24T22:02:26.511 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: - 2021-03-24T22:02:26.511 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: 03 2021-03-24T22:02:26.511 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: - 2021-03-24T22:02:26.511 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: 24 2021-03-24T22:02:26.512 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: T 2021-03-24T22:02:26.512 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: 22 2021-03-24T22:02:26.512 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: : 2021-03-24T22:02:26.512 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: 02 2021-03-24T22:02:26.513 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: : 2021-03-24T22:02:26.513 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: 26 2021-03-24T22:02:26.513 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: . 2021-03-24T22:02:26.513 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: 298602 2021-03-24T22:02:26.513 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: +0000 2021-03-24T22:02:26.514 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: 2021-03-24T22:02:26.514 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: mon.a 2021-03-24T22:02:26.514 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: ( 2021-03-24T22:02:26.514 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: mon 2021-03-24T22:02:26.514 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: . 2021-03-24T22:02:26.515 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: 0) 1601 : audit [INF] from='client.? v1:172.21.15.125:0/4123199439' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T22:02:26.515 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: audit 2021- 2021-03-24T22:02:26.515 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: 03-24T22:02:26.298674+0000 mon.a (mon.0) 2021-03-24T22:02:26.515 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: 1602 : audit [INF] from='client.? v1:172.21.15.125:0/4200642048' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2021-03-24T22:02:26.516 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: cluster 2021-03-24T22:02:26.516 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: 2021-03-24T22:02:26.516 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: 2021-03-24T22:02 2021-03-24T22:02:26.516 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:26 smithi125 conmon[24927]: :26.298706+0000 mon.a (mon.0) 1603 : cluster [DBG] osdmap e576: 8 total, 8 up, 8 in 2021-03-24T22:02:26.584 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:26.641 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:26.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:26.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:02:26.748 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:26.802 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:26.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:26.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:26.908 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:26.968 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:26.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:26.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:02:27.074 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:27.132 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:27.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:27.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:02:27.238 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:27.302 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:27.307 INFO:tasks.rados.rados.0.smithi125.stdout:2152: done (1 left) 2021-03-24T22:02:27.307 INFO:tasks.rados.rados.0.smithi125.stdout:2155: done (0 left) 2021-03-24T22:02:27.308 INFO:tasks.rados.rados.0.smithi125.stdout:2156: write oid 22 current snap is 243 2021-03-24T22:02:27.308 INFO:tasks.rados.rados.0.smithi125.stdout:2156: seq_num 754 ranges {672478=656693,2089070=459917} 2021-03-24T22:02:27.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:27.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:02:27.317 INFO:tasks.rados.rados.0.smithi125.stdout:2156: writing smithi12556470-22 from 672478 to 1329171 tid 1 2021-03-24T22:02:27.325 INFO:tasks.rados.rados.0.smithi125.stdout:2156: writing smithi12556470-22 from 2089070 to 2548987 tid 2 2021-03-24T22:02:27.325 INFO:tasks.rados.rados.0.smithi125.stdout:2157: copy_from oid 28 from oid 34 current snap is 243 2021-03-24T22:02:27.325 INFO:tasks.rados.rados.0.smithi125.stdout:2156: finishing write tid 1 to smithi12556470-22 2021-03-24T22:02:27.326 INFO:tasks.rados.rados.0.smithi125.stdout:2158: read oid 20 snap 241 2021-03-24T22:02:27.326 INFO:tasks.rados.rados.0.smithi125.stdout:2158: expect (ObjNum 743 snap 240 seq_num 743) 2021-03-24T22:02:27.326 INFO:tasks.rados.rados.0.smithi125.stdout:2159: write oid 43 current snap is 243 2021-03-24T22:02:27.326 INFO:tasks.rados.rados.0.smithi125.stdout:2159: seq_num 755 ranges {689285=731942,2147552=541733} 2021-03-24T22:02:27.333 INFO:tasks.rados.rados.0.smithi125.stdout:2159: writing smithi12556470-43 from 689285 to 1421227 tid 1 2021-03-24T22:02:27.341 INFO:tasks.rados.rados.0.smithi125.stdout:2159: writing smithi12556470-43 from 2147552 to 2689285 tid 2 2021-03-24T22:02:27.342 INFO:tasks.rados.rados.0.smithi125.stdout:2160: snap_create 2021-03-24T22:02:27.342 INFO:tasks.rados.rados.0.smithi125.stdout:2156: finishing write tid 2 to smithi12556470-22 2021-03-24T22:02:27.342 INFO:tasks.rados.rados.0.smithi125.stdout:2156: finishing write tid 3 to smithi12556470-22 2021-03-24T22:02:27.342 INFO:tasks.rados.rados.0.smithi125.stdout:2157: finishing copy_from to smithi12556470-28 2021-03-24T22:02:27.343 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 28 v 550 (ObjNum 610 snap 185 seq_num 610) dirty exists 2021-03-24T22:02:27.343 INFO:tasks.rados.rados.0.smithi125.stdout:2157: finishing copy_from racing read to smithi12556470-28 2021-03-24T22:02:27.343 INFO:tasks.rados.rados.0.smithi125.stdout:2156: finishing write tid 4 to smithi12556470-22 2021-03-24T22:02:27.343 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 22 v 660 (ObjNum 754 snap 243 seq_num 754) dirty exists 2021-03-24T22:02:27.344 INFO:tasks.rados.rados.0.smithi125.stdout:2156: left oid 22 (ObjNum 754 snap 243 seq_num 754) 2021-03-24T22:02:27.344 INFO:tasks.rados.rados.0.smithi125.stdout:2159: finishing write tid 1 to smithi12556470-43 2021-03-24T22:02:27.344 INFO:tasks.rados.rados.0.smithi125.stdout:2158: expect (ObjNum 743 snap 240 seq_num 743) 2021-03-24T22:02:27.415 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:27.472 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:27.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:27.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:02:27.502 INFO:tasks.rados.rados.0.smithi125.stdout:2159: finishing write tid 2 to smithi12556470-43 2021-03-24T22:02:27.502 INFO:tasks.rados.rados.0.smithi125.stdout:2159: finishing write tid 3 to smithi12556470-43 2021-03-24T22:02:27.502 INFO:tasks.rados.rados.0.smithi125.stdout:2159: finishing write tid 4 to smithi12556470-43 2021-03-24T22:02:27.503 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 43 v 1132 (ObjNum 755 snap 243 seq_num 755) dirty exists 2021-03-24T22:02:27.503 INFO:tasks.rados.rados.0.smithi125.stdout:2159: left oid 43 (ObjNum 755 snap 243 seq_num 755) 2021-03-24T22:02:27.578 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:27.633 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:27.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:27.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:02:27.739 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:27.795 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:27.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:27.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:02:27.902 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:27.959 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:27.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:27.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:02:28.065 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:28.121 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:28.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:28.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:02:28.227 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:28.284 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:28.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:28.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:02:28.319 INFO:tasks.rados.rados.0.smithi125.stdout:2156: done (4 left) 2021-03-24T22:02:28.319 INFO:tasks.rados.rados.0.smithi125.stdout:2157: done (3 left) 2021-03-24T22:02:28.320 INFO:tasks.rados.rados.0.smithi125.stdout:2158: done (2 left) 2021-03-24T22:02:28.320 INFO:tasks.rados.rados.0.smithi125.stdout:2159: done (1 left) 2021-03-24T22:02:28.320 INFO:tasks.rados.rados.0.smithi125.stdout:2160: done (0 left) 2021-03-24T22:02:28.320 INFO:tasks.rados.rados.0.smithi125.stdout:2161: rollback oid 16 current snap is 244 2021-03-24T22:02:28.321 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 16 to 233 2021-03-24T22:02:28.321 INFO:tasks.rados.rados.0.smithi125.stdout:2162: read oid 29 snap -1 2021-03-24T22:02:28.321 INFO:tasks.rados.rados.0.smithi125.stdout:2162: expect (ObjNum 691 snap 218 seq_num 691) 2021-03-24T22:02:28.321 INFO:tasks.rados.rados.0.smithi125.stdout:2163: write oid 26 current snap is 244 2021-03-24T22:02:28.322 INFO:tasks.rados.rados.0.smithi125.stdout:2163: seq_num 756 ranges {706092=407191,1806034=463586} 2021-03-24T22:02:28.322 INFO:tasks.rados.rados.0.smithi125.stdout:2163: writing smithi12556470-26 from 706092 to 1113283 tid 1 2021-03-24T22:02:28.329 INFO:tasks.rados.rados.0.smithi125.stdout:2163: writing smithi12556470-26 from 1806034 to 2269620 tid 2 2021-03-24T22:02:28.330 INFO:tasks.rados.rados.0.smithi125.stdout:2162: expect (ObjNum 691 snap 218 seq_num 691) 2021-03-24T22:02:28.333 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:28 smithi142 conmon[20125]: cluster 2021-03-24T22:02:26.904669+0000 mgr.y (mgr 2021-03-24T22:02:28.333 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:28 smithi142 conmon[20125]: .14144) 560 : cluster [DBG] pgmap v1027: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 164 active+clean; 537 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.0 KiB/s rd, 4.1 MiB/s wr, 12 op/s; 2/1713 objects degraded (0.117%); 78/1713 objects misplaced (4.553%); 0 B/s, 2 objects/s recovering 2021-03-24T22:02:28.333 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:28 smithi142 conmon[20125]: cluster 2021-03-24T22:02:27.302422+0000 mon.a (mon.0) 1604 : cluster [DBG] osdmap e577: 8 total, 8 up, 8 in 2021-03-24T22:02:28.389 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:28.432 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:28.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:28.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:02:28.442 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2021-03-24T22:02:28.443 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2021-03-24T22:02:28.443 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2021-03-24T22:02:28.443 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2021-03-24T22:02:28.443 INFO:teuthology.orchestra.run.smithi142.stdout:} 2021-03-24T22:02:28.533 INFO:tasks.rados.rados.0.smithi125.stdout:2161: finishing rollback tid 0 to smithi12556470-16 2021-03-24T22:02:28.533 INFO:tasks.rados.rados.0.smithi125.stdout:2161: finishing rollback tid 1 to smithi12556470-16 2021-03-24T22:02:28.533 INFO:tasks.rados.rados.0.smithi125.stdout:2161: finishing rollback tid 2 to smithi12556470-16 2021-03-24T22:02:28.533 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 16 v 402 (ObjNum 700 snap 221 seq_num 700) dirty exists 2021-03-24T22:02:28.534 INFO:tasks.rados.rados.0.smithi125.stdout:2163: finishing write tid 1 to smithi12556470-26 2021-03-24T22:02:28.534 INFO:tasks.rados.rados.0.smithi125.stdout:2163: finishing write tid 2 to smithi12556470-26 2021-03-24T22:02:28.534 INFO:tasks.rados.rados.0.smithi125.stdout:2163: finishing write tid 3 to smithi12556470-26 2021-03-24T22:02:28.535 INFO:tasks.rados.rados.0.smithi125.stdout:2163: finishing write tid 4 to smithi12556470-26 2021-03-24T22:02:28.535 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 26 v 443 (ObjNum 756 snap 244 seq_num 756) dirty exists 2021-03-24T22:02:28.535 INFO:tasks.rados.rados.0.smithi125.stdout:2163: left oid 26 (ObjNum 756 snap 244 seq_num 756) 2021-03-24T22:02:28.535 INFO:tasks.rados.rados.0.smithi125.stdout:2161: done (2 left) 2021-03-24T22:02:28.536 INFO:tasks.rados.rados.0.smithi125.stdout:2162: done (1 left) 2021-03-24T22:02:28.536 INFO:tasks.rados.rados.0.smithi125.stdout:2163: done (0 left) 2021-03-24T22:02:28.537 INFO:tasks.rados.rados.0.smithi125.stdout:2164: read oid 15 snap -1 2021-03-24T22:02:28.537 INFO:tasks.rados.rados.0.smithi125.stdout:2164: expect (ObjNum 750 snap 241 seq_num 750) 2021-03-24T22:02:28.537 INFO:tasks.rados.rados.0.smithi125.stdout:2165: snap_remove snap 232 2021-03-24T22:02:28.538 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:28.588 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:28.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:28.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:02:28.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:28 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:02:28.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:28 smithi125 conmon[19936]: 02:26.904669+0000 mgr.y (mgr.14144) 560 : cluster [DBG] pgmap v1027: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 164 active+clean; 537 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.0 KiB/s rd, 4.1 MiB/s wr, 12 op/s; 2/1713 objects degraded (0.117%); 78/1713 objects misplaced (4.553%); 0 B/s, 2 objects/s recovering 2021-03-24T22:02:28.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:28 smithi125 conmon[19936]: cluster 2021-03-24T22:02:27 2021-03-24T22:02:28.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:28 smithi125 conmon[19936]: .302422+0000 mon.a (mon.0) 1604 : cluster [DBG] osdmap e577: 8 total, 8 up, 8 in 2021-03-24T22:02:28.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:28 smithi125 conmon[24927]: cluster 2021-03-24T22:02:26.904669 2021-03-24T22:02:28.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:28 smithi125 conmon[24927]: +0000 mgr.y (mgr.14144) 560 : cluster [DBG] pgmap v1027: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 164 active+clean; 537 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.0 KiB/s rd, 4.1 MiB/s wr, 12 op/s; 2/1713 objects degraded (0.117%); 78/1713 objects misplaced (4.553%); 0 B/s, 2 objects/s recovering 2021-03-24T22:02:28.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:28 smithi125 conmon[24927]: cluster 2021-03-24T22:02:27.302422+0000 mon.a (mon 2021-03-24T22:02:28.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:28 smithi125 conmon[24927]: .0) 1604 : cluster [DBG] osdmap e577: 8 total, 8 up, 8 in 2021-03-24T22:02:28.692 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:28.747 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:28.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:28.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:02:28.852 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:28.912 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:28.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:28.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:02:29.018 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:29.067 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:29.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:29.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:02:29.173 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:29.228 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd reweight 4 1 2021-03-24T22:02:29.230 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:29.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:29.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:02:29.335 INFO:tasks.rados.rados.0.smithi125.stdout:2165: done (1 left) 2021-03-24T22:02:29.335 INFO:tasks.rados.rados.0.smithi125.stdout:2166: read oid 33 snap -1 2021-03-24T22:02:29.336 INFO:tasks.rados.rados.0.smithi125.stdout:2166: expect (ObjNum 749 snap 241 seq_num 749) 2021-03-24T22:02:29.336 INFO:tasks.rados.rados.0.smithi125.stdout:2167: write (excl) oid 50 current snap is 244 2021-03-24T22:02:29.336 INFO:tasks.rados.rados.0.smithi125.stdout:2167: seq_num 757 ranges {722899=482440,1748163=723597} 2021-03-24T22:02:29.336 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:29.342 INFO:tasks.rados.rados.0.smithi125.stdout:2167: writing smithi12556470-50 from 722899 to 1205339 tid 1 2021-03-24T22:02:29.353 INFO:tasks.rados.rados.0.smithi125.stdout:2167: writing smithi12556470-50 from 1748163 to 2471760 tid 2 2021-03-24T22:02:29.354 INFO:tasks.rados.rados.0.smithi125.stdout:2168: snap_create 2021-03-24T22:02:29.354 INFO:tasks.rados.rados.0.smithi125.stdout:2164: expect (ObjNum 750 snap 241 seq_num 750) 2021-03-24T22:02:29.385 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.6 dump_historic_ops 2021-03-24T22:02:29.388 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:29.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:29.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:02:29.447 INFO:tasks.rados.rados.0.smithi125.stdout:2167: finishing write tid 1 to smithi12556470-50 2021-03-24T22:02:29.448 INFO:tasks.rados.rados.0.smithi125.stdout:2166: expect (ObjNum 749 snap 241 seq_num 749) 2021-03-24T22:02:29.493 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:29.541 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:29.544 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:02:29.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:29.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:29.557 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T22:02:29.578 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T22:02:29.579 INFO:tasks.rados.rados.0.smithi125.stdout:2167: finishing write tid 2 to smithi12556470-50 2021-03-24T22:02:29.580 INFO:tasks.rados.rados.0.smithi125.stdout:2167: finishing write tid 3 to smithi12556470-50 2021-03-24T22:02:29.580 INFO:tasks.rados.rados.0.smithi125.stdout:2167: finishing write tid 4 to smithi12556470-50 2021-03-24T22:02:29.580 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 50 v 722 (ObjNum 757 snap 244 seq_num 757) dirty exists 2021-03-24T22:02:29.580 INFO:tasks.rados.rados.0.smithi125.stdout:2167: left oid 50 (ObjNum 757 snap 244 seq_num 757) 2021-03-24T22:02:29.602 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T22:02:29.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:29 smithi142 conmon[20125]: cluster 2021-03-24T22:02:27 2021-03-24T22:02:29.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:29 smithi142 conmon[20125]: .831378+0000 osd.6 (osd.6) 41 : cluster [DBG] 5.11 scrub ok 2021-03-24T22:02:29.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:29 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:02:29.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:29 smithi142 conmon[20125]: 24T22:02:28.307377+0000 mon.a 2021-03-24T22:02:29.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:29 smithi142 conmon[20125]: (mon.0) 1605 : cluster [DBG] osdmap e578: 8 total, 8 up, 8 in 2021-03-24T22:02:29.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:29 smithi142 conmon[20125]: 2021-03-24T22:02:29.624 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T22:02:29.646 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T22:02:29.652 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:29.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:29 smithi125 conmon[19936]: cluster 2021-03-24T22:02:27.831378 2021-03-24T22:02:29.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:29 smithi125 conmon[19936]: +0000 osd.6 (osd.6) 41 : cluster [DBG] 5.11 scrub ok 2021-03-24T22:02:29.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:29 smithi125 conmon[19936]: cluster 2021-03-24T22:02:28.307377+0000 mon.a (mon.0) 1605 : cluster [DBG] osdmap e578: 8 total, 8 up, 8 in 2021-03-24T22:02:29.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:29 smithi125 conmon[24927]: cluster 2021-03-24T22:02 2021-03-24T22:02:29.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:29 smithi125 conmon[24927]: :27.831378+0000 osd.6 (osd.6) 41 : cluster [DBG] 5.11 scrub ok 2021-03-24T22:02:29.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:29 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T22:02:29.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:29 smithi125 conmon[24927]: T22:02:28.307377+0000 mon.a 2021-03-24T22:02:29.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:29 smithi125 conmon[24927]: (mon.0) 1605 : cluster [DBG] osdmap e578: 8 total, 8 up, 8 in 2021-03-24T22:02:29.670 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T22:02:29.679 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T22:02:29.694 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T22:02:29.698 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:29.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:29.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:29.717 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T22:02:29.718 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-24T22:02:29.718 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-24T22:02:29.719 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-24T22:02:29.719 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-24T22:02:29.719 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-24T22:02:29.720 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-24T22:02:29.720 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-24T22:02:29.720 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-24T22:02:29.806 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:29.862 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:29.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:29.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:02:29.970 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:30.114 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:30.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:30.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:02:30.218 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:30.279 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:30.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:30.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:02:30.308 INFO:tasks.rados.rados.0.smithi125.stdout:2164: done (3 left) 2021-03-24T22:02:30.309 INFO:tasks.rados.rados.0.smithi125.stdout:2166: done (2 left) 2021-03-24T22:02:30.309 INFO:tasks.rados.rados.0.smithi125.stdout:2167: done (1 left) 2021-03-24T22:02:30.309 INFO:tasks.rados.rados.0.smithi125.stdout:2168: done (0 left) 2021-03-24T22:02:30.309 INFO:tasks.rados.rados.0.smithi125.stdout:2169: snap_remove snap 149 2021-03-24T22:02:30.385 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:30.442 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:30.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:30.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:02:30.547 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:30.561 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:30 smithi125 conmon[19936]: cluster 2021-03-24T22:02:30.562 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:30 smithi125 conmon[19936]: 2021-03-24T22:02:28.905241+0000 2021-03-24T22:02:30.562 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:30 smithi125 conmon[19936]: mgr.y (mgr.14144) 561 : cluster [DBG] pgmap v1030: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 164 active+clean; 537 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 1.1 MiB/s wr, 3 op/s; 2/1713 objects degraded (0.117%); 78/1713 objects misplaced (4.553%) 2021-03-24T22:02:30.562 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:30 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:02:30.562 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:30 smithi125 conmon[19936]: 03-24T22:02:29.330081+0000 mon.a (mon.0) 1606 : cluster [DBG] osdmap e579: 8 total, 8 up, 8 in 2021-03-24T22:02:30.563 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:30 smithi125 conmon[19936]: audit 2021-03-24T22:02:29.525446+0000 2021-03-24T22:02:30.563 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:30 smithi125 conmon[19936]: mon.a (mon.0) 1607 : audit [DBG] from='client.? v1:172.21.15.125:0/3207132499' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:02:30.563 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:30 smithi125 conmon[19936]: cluster 2021-03-24T22:02:30.305770 2021-03-24T22:02:30.563 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:30 smithi125 conmon[19936]: +0000 mon.a (mon.0) 1608 : cluster [DBG] osdmap e580: 8 total, 8 up, 8 in 2021-03-24T22:02:30.564 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:30 smithi125 conmon[24927]: cluster 2021-03-24T22:02:28. 2021-03-24T22:02:30.564 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:30 smithi125 conmon[24927]: 905241+0000 mgr.y (mgr.14144) 561 : cluster [DBG] pgmap v1030: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 164 active+clean; 537 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 1.1 MiB/s wr, 3 op/s; 2/1713 objects degraded (0.117%); 78/1713 objects misplaced (4.553%) 2021-03-24T22:02:30.564 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:30 smithi125 conmon[24927]: cluster 2021-03-24T22:02:29 2021-03-24T22:02:30.565 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:30 smithi125 conmon[24927]: .330081+0000 mon.a (mon.0) 1606 : cluster [DBG] osdmap e579: 8 total, 8 up, 8 in 2021-03-24T22:02:30.565 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:30 smithi125 conmon[24927]: audit 2021-03-24T22:02:30.565 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:30 smithi125 conmon[24927]: 2021-03-24T22:02:29.525446+0000 mon.a (mon.0) 2021-03-24T22:02:30.565 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:30 smithi125 conmon[24927]: 1607 : audit [DBG] from='client.? v1:172.21.15.125:0/3207132499' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:02:30.565 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:30 smithi125 conmon[24927]: cluster 2021-03-24T22:02: 2021-03-24T22:02:30.566 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:30 smithi125 conmon[24927]: 30.305770+0000 mon.a (mon.0) 1608 : cluster 2021-03-24T22:02:30.566 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:30 smithi125 conmon[24927]: [DBG] osdmap e580: 8 total, 8 up, 8 in 2021-03-24T22:02:30.610 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:30.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:30.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:02:30.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:30 smithi142 conmon[20125]: cluster 2021-03-24T22:02:28.905241+0000 mgr.y (mgr. 2021-03-24T22:02:30.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:30 smithi142 conmon[20125]: 14144) 561 : cluster [DBG] pgmap v1030: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 164 active+clean; 537 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 1.1 MiB/s wr, 3 op/s; 2/1713 objects degraded (0.117%); 78/1713 objects misplaced (4.553%) 2021-03-24T22:02:30.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:30 smithi142 conmon[20125]: cluster 2021-03-24T22:02:29.330081+0000 mon.a (mon.0) 1606 : cluster [DBG] osdmap e579: 8 total, 8 up, 8 in 2021-03-24T22:02:30.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:30 smithi142 conmon[20125]: audit 2021-03-24T22:02:29.525446+0000 mon.a (mon.0 2021-03-24T22:02:30.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:30 smithi142 conmon[20125]: ) 1607 : audit [DBG] from='client.? v1:172.21.15.125:0/3207132499' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:02:30.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:30 smithi142 conmon[20125]: cluster 2021-03-24T22:02:30.305770+0000 2021-03-24T22:02:30.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:30 smithi142 conmon[20125]: mon.a (mon.0) 1608 : cluster [DBG] osdmap e580: 8 total, 8 up, 8 in 2021-03-24T22:02:30.715 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:30.770 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:30.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:30.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:02:30.875 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:30.925 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:30.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:30.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:02:31.032 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:31.092 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:31.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:31.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:02:31.104 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd set noscrub 2021-03-24T22:02:31.198 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:31.244 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:31.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:31.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:02:31.308 INFO:tasks.rados.rados.0.smithi125.stdout:2169: done (0 left) 2021-03-24T22:02:31.308 INFO:tasks.rados.rados.0.smithi125.stdout:2170: read oid 12 snap -1 2021-03-24T22:02:31.308 INFO:tasks.rados.rados.0.smithi125.stdout:2170: expect (ObjNum 752 snap 241 seq_num 752) 2021-03-24T22:02:31.308 INFO:tasks.rados.rados.0.smithi125.stdout:2171: snap_create 2021-03-24T22:02:31.334 INFO:tasks.rados.rados.0.smithi125.stdout:2170: expect (ObjNum 752 snap 241 seq_num 752) 2021-03-24T22:02:31.350 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:31.398 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:31.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:31.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:02:31.504 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:31.560 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:31.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:31.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:31.667 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:31.704 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2021-03-24T22:02:31.705 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T22:02:31.728 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:31.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:31.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:02:31.834 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:31.893 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:31.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:31.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:02:31.966 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:02:31.996 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:32.054 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:32.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:32.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:32.160 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:32.219 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:32.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:32.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:02:32.313 INFO:tasks.rados.rados.0.smithi125.stdout:2170: done (1 left) 2021-03-24T22:02:32.314 INFO:tasks.rados.rados.0.smithi125.stdout:2171: done (0 left) 2021-03-24T22:02:32.314 INFO:tasks.rados.rados.0.smithi125.stdout:2172: write (excl) oid 48 current snap is 246 2021-03-24T22:02:32.314 INFO:tasks.rados.rados.0.smithi125.stdout:2172: seq_num 758 ranges {739706=557689,1806645=467255} 2021-03-24T22:02:32.321 INFO:tasks.rados.rados.0.smithi125.stdout:2172: writing smithi12556470-48 from 739706 to 1297395 tid 1 2021-03-24T22:02:32.324 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:32.328 INFO:tasks.rados.rados.0.smithi125.stdout:2172: writing smithi12556470-48 from 1806645 to 2273900 tid 2 2021-03-24T22:02:32.329 INFO:tasks.rados.rados.0.smithi125.stdout:2173: read oid 32 snap -1 2021-03-24T22:02:32.329 INFO:tasks.rados.rados.0.smithi125.stdout:2173: expect (ObjNum 748 snap 241 seq_num 748) 2021-03-24T22:02:32.329 INFO:tasks.rados.rados.0.smithi125.stdout:2174: copy_from oid 39 from oid 24 current snap is 246 2021-03-24T22:02:32.329 INFO:tasks.rados.rados.0.smithi125.stdout:2175: snap_remove snap 244 2021-03-24T22:02:32.374 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:32.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:32.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:02:32.479 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:32.492 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:32 smithi125 conmon[24927]: cluster 2021-03-24T22:02:30.763297+0000 osd.3 (osd.3) 119 : 2021-03-24T22:02:32.492 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:32 smithi125 conmon[24927]: cluster [DBG] 7.10 scrub ok 2021-03-24T22:02:32.492 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:32 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:02:32.492 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:32 smithi125 conmon[24927]: 03-24T22:02:30.890585+0000 osd.6 (osd.6) 42 : cluster [DBG] 3.a scrub ok 2021-03-24T22:02:32.493 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:32 smithi125 conmon[24927]: cluster 2021-03-24T22:02:30 2021-03-24T22:02:32.493 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:32 smithi125 conmon[24927]: .906048+0000 mgr.y (mgr.14144) 562 : cluster [DBG] pgmap v1033: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 164 active+clean; 547 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 6.1 MiB/s rd, 1.6 MiB/s wr, 12 op/s; 1/1716 objects degraded (0.058%); 73/1716 objects misplaced (4.254%); 3.8 MiB/s, 10 objects/s recovering 2021-03-24T22:02:32.493 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:32 smithi125 conmon[24927]: cluster 2021-03-24T22:02:31.302538 2021-03-24T22:02:32.493 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:32 smithi125 conmon[24927]: +0000 mon.a (mon.0) 1609 : cluster [DBG] osdmap e581: 8 total, 8 up, 8 in 2021-03-24T22:02:32.494 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:32 smithi125 conmon[24927]: cluster 2021-03-24T22:02:31.328137 2021-03-24T22:02:32.494 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:32 smithi125 conmon[24927]: +0000 mon.a (mon.0) 1610 : cluster [WRN] Health check update: Degraded data redundancy: 1/1716 objects degraded (0.058%), 1 pg degraded (PG_DEGRADED) 2021-03-24T22:02:32.494 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:32 smithi125 conmon[19936]: cluster 2021-03-24T22:02:30.763297+0000 2021-03-24T22:02:32.494 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:32 smithi125 conmon[19936]: osd.3 (osd.3) 119 : cluster [DBG] 7.10 scrub ok 2021-03-24T22:02:32.495 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:32 smithi125 conmon[19936]: cluster 2021-03-24T22:02:30.890585+0000 osd.6 (osd.6) 42 : cluster [DBG] 3.a scrub ok 2021-03-24T22:02:32.495 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:32 smithi125 conmon[19936]: cluster 2021-03-24T22:02:30 2021-03-24T22:02:32.495 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:32 smithi125 conmon[19936]: .906048+0000 mgr.y (mgr.14144) 562 : cluster [DBG] 2021-03-24T22:02:32.495 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:32 smithi125 conmon[19936]: pgmap v1033: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 164 active+clean; 547 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 6.1 MiB/s rd, 1.6 MiB/s wr, 12 op/s; 1/1716 objects degraded (0.058%); 73/1716 objects misplaced (4.254%); 3.8 MiB/s, 10 objects/s recovering 2021-03-24T22:02:32.496 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:32 smithi125 conmon[19936]: cluster 2021-03-24T22 2021-03-24T22:02:32.496 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:32 smithi125 conmon[19936]: :02:31.302538+0000 mon.a (mon.0) 2021-03-24T22:02:32.496 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:32 smithi125 conmon[19936]: 1609 : cluster [DBG] osdmap e581: 8 total, 8 up, 8 in 2021-03-24T22:02:32.496 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:32 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:02:32.497 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:32 smithi125 conmon[19936]: 03-24T22:02:31.328137+0000 mon.a (mon 2021-03-24T22:02:32.497 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:32 smithi125 conmon[19936]: .0) 1610 : cluster [WRN] Health check update: Degraded data redundancy: 1/1716 objects degraded (0.058%), 1 pg degraded (PG_DEGRADED) 2021-03-24T22:02:32.497 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:32 smithi142 conmon[20125]: cluster 2021-03-24T22:02 2021-03-24T22:02:32.498 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:32 smithi142 conmon[20125]: :30.763297+0000 osd.3 (osd.3) 119 : cluster [DBG] 7.10 scrub ok 2021-03-24T22:02:32.498 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:32 smithi142 conmon[20125]: cluster 2021-03-24T22:02: 2021-03-24T22:02:32.498 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:32 smithi142 conmon[20125]: 30.890585+0000 osd.6 (osd.6) 42 : cluster [DBG] 3.a scrub ok 2021-03-24T22:02:32.498 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:32 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:02:32.499 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:32 smithi142 conmon[20125]: -24T22:02:30.906048+0000 mgr.y 2021-03-24T22:02:32.499 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:32 smithi142 conmon[20125]: (mgr.14144) 562 : cluster [DBG] pgmap v1033: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 164 active+clean; 547 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 6.1 MiB/s rd, 1.6 MiB/s wr, 12 op/s; 1/1716 objects degraded (0.058%); 73/1716 objects misplaced (4.254%); 3.8 MiB/s, 10 objects/s recovering 2021-03-24T22:02:32.499 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:32 smithi142 conmon[20125]: 2021-03-24T22:02:32.499 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:32 smithi142 conmon[20125]: cluster 2021-03-24T22:02:32.499 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:32 smithi142 conmon[20125]: 2021-03-24T 2021-03-24T22:02:32.500 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:32 smithi142 conmon[20125]: 22:02:31.302538+0000 mon.a (mon 2021-03-24T22:02:32.500 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:32 smithi142 conmon[20125]: .0) 1609 : cluster [DBG] osdmap e581: 8 total, 8 up, 8 in 2021-03-24T22:02:32.500 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:32 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:02:32.500 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:32 smithi142 conmon[20125]: -03-24T22:02:31.328137+0000 2021-03-24T22:02:32.501 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:32 smithi142 conmon[20125]: mon.a (mon.0) 1610 : cluster [WRN] 2021-03-24T22:02:32.501 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:32 smithi142 conmon[20125]: Health check update: Degraded data redundancy: 1/1716 objects degraded (0.058%), 1 pg degraded (PG_DEGRADED) 2021-03-24T22:02:32.532 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:32.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:32.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:02:32.631 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.7 dump_ops_in_flight 2021-03-24T22:02:32.638 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:32.694 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:32.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:32.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:02:32.798 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:32.858 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:32.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:32.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:02:32.931 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T22:02:32.964 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:32.995 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-24T22:02:33.026 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:33.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:33.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:33.133 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:33.189 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:33.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:33.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:33.294 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:33.346 INFO:tasks.rados.rados.0.smithi125.stdout:2175: done (3 left) 2021-03-24T22:02:33.347 INFO:tasks.rados.rados.0.smithi125.stdout:2176: rollback oid 37 current snap is 246 2021-03-24T22:02:33.347 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 37 to 243 2021-03-24T22:02:33.348 INFO:tasks.rados.rados.0.smithi125.stdout:2177: write oid 43 current snap is 246 2021-03-24T22:02:33.348 INFO:tasks.rados.rados.0.smithi125.stdout:2177: seq_num 759 ranges {756513=632938,1865127=727266} 2021-03-24T22:02:33.349 INFO:tasks.rados.rados.0.smithi125.stdout:2177: writing smithi12556470-43 from 756513 to 1389451 tid 1 2021-03-24T22:02:33.349 INFO:teuthology.orchestra.run.smithi125.stderr:reweighted osd.4 to 1 (10000) 2021-03-24T22:02:33.355 INFO:tasks.rados.rados.0.smithi125.stdout:2177: writing smithi12556470-43 from 1865127 to 2592393 tid 2 2021-03-24T22:02:33.355 INFO:tasks.rados.rados.0.smithi125.stdout:2178: write oid 46 current snap is 246 2021-03-24T22:02:33.356 INFO:tasks.rados.rados.0.smithi125.stdout:2172: finishing write tid 1 to smithi12556470-48 2021-03-24T22:02:33.356 INFO:tasks.rados.rados.0.smithi125.stdout:2172: finishing write tid 2 to smithi12556470-48 2021-03-24T22:02:33.356 INFO:tasks.rados.rados.0.smithi125.stdout:2173: expect (ObjNum 748 snap 241 seq_num 748) 2021-03-24T22:02:33.358 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:33.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:33.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:02:33.429 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:33 smithi142 conmon[20125]: cluster 2021-03-24T22:02:32.116065+0000 osd.5 (osd. 2021-03-24T22:02:33.430 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:33 smithi142 conmon[20125]: 5) 26 : cluster [DBG] 4.1 scrub ok 2021-03-24T22:02:33.430 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:33 smithi142 conmon[20125]: cluster 2021-03-24T22:02:32.306397+0000 mon.a (mon.0) 1611 : cluster [DBG] osdmap e582: 8 total, 8 up, 8 in 2021-03-24T22:02:33.430 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:33 smithi142 conmon[20125]: audit 2021 2021-03-24T22:02:33.431 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:33 smithi142 conmon[20125]: -03-24T22:02:32.902673+0000 mon.c (mon.2) 89 : audit [INF] from='client.? v1:172.21.15.125:0/3969215953' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2021-03-24T22:02:33.431 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:33 smithi142 conmon[20125]: audit 2021-03-24T 2021-03-24T22:02:33.431 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:33 smithi142 conmon[20125]: 22:02:32.903328+0000 mon.a (mon.0) 1612 : audit [INF] from='client.35483 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2021-03-24T22:02:33.464 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:33.477 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:33 smithi125 conmon[19936]: cluster 2021-03-24T22:02:32.116065+0000 2021-03-24T22:02:33.477 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:33 smithi125 conmon[19936]: osd.5 (osd.5) 26 : cluster [DBG] 4.1 scrub ok 2021-03-24T22:02:33.477 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:33 smithi125 conmon[19936]: cluster 2021-03-24T22:02:32.306397+0000 mon.a (mon.0) 1611 : cluster [DBG] osdmap e582: 8 total, 8 up, 8 in 2021-03-24T22:02:33.477 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:33 smithi125 conmon[19936]: audit 2021-03-24T22:02:32.902673+0000 mon.c (mon.2) 89 : audit [INF] from='client.? v1:172.21.15.125:0/3969215953' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2021-03-24T22:02:33.478 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:33 smithi125 conmon[19936]: audit 2021-03- 2021-03-24T22:02:33.478 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:33 smithi125 conmon[19936]: 24T22:02:32.903328+0000 mon.a ( 2021-03-24T22:02:33.478 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:33 smithi125 conmon[19936]: mon.0) 1612 : audit [INF] from='client.35483 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2021-03-24T22:02:33.479 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:33 smithi125 conmon[24927]: cluster 2021-03-24T22:02:33.479 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:33 smithi125 conmon[24927]: 2021-03-24T22:02:33.479 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:33 smithi125 conmon[24927]: 2021 2021-03-24T22:02:33.479 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:33 smithi125 conmon[24927]: - 2021-03-24T22:02:33.480 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:33 smithi125 conmon[24927]: 03-24T22:02: 2021-03-24T22:02:33.480 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:33 smithi125 conmon[24927]: 32.116065+0000 osd.5 (osd.5) 26 : cluster [DBG] 4.1 scrub ok 2021-03-24T22:02:33.480 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:33 smithi125 conmon[24927]: 2021-03-24T22:02:33.480 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:33 smithi125 conmon[24927]: cluster 2021-03-24T22:02:33.480 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:33 smithi125 conmon[24927]: 2021-03-24T22:02 2021-03-24T22:02:33.481 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:33 smithi125 conmon[24927]: :32.306397+0000 mon.a 2021-03-24T22:02:33.481 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:33 smithi125 conmon[24927]: (mon.0) 1611 : 2021-03-24T22:02:33.481 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:33 smithi125 conmon[24927]: cluster [DBG] osdmap e582: 8 total, 8 up, 8 in 2021-03-24T22:02:33.481 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:33 smithi125 conmon[24927]: audit 2021 2021-03-24T22:02:33.482 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:33 smithi125 conmon[24927]: -03-24T22:02:32. 2021-03-24T22:02:33.482 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:33 smithi125 conmon[24927]: 902673+0000 mon.c (mon. 2021-03-24T22:02:33.482 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:33 smithi125 conmon[24927]: 2) 89 : audit [INF] 2021-03-24T22:02:33.482 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:33 smithi125 conmon[24927]: from='client.? v1:172.21.15.125:0/3969215953' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2021-03-24T22:02:33.482 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:33 smithi125 conmon[24927]: audit 2021-03-24T22:02:33.483 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:33 smithi125 conmon[24927]: 2021-03-24T22: 2021-03-24T22:02:33.483 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:33 smithi125 conmon[24927]: 02:32.903328+0000 mon.a 2021-03-24T22:02:33.483 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:33 smithi125 conmon[24927]: (mon.0 2021-03-24T22:02:33.483 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:33 smithi125 conmon[24927]: ) 1612 : audit [INF] from='client.35483 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2021-03-24T22:02:33.508 INFO:tasks.rados.rados.0.smithi125.stdout:2172: finishing write tid 3 to smithi12556470-48 2021-03-24T22:02:33.509 INFO:tasks.rados.rados.0.smithi125.stdout:2172: finishing write tid 4 to smithi12556470-48 2021-03-24T22:02:33.509 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 48 v 746 (ObjNum 758 snap 246 seq_num 758) dirty exists 2021-03-24T22:02:33.509 INFO:tasks.rados.rados.0.smithi125.stdout:2172: left oid 48 (ObjNum 758 snap 246 seq_num 758) 2021-03-24T22:02:33.509 INFO:tasks.rados.rados.0.smithi125.stdout:2174: finishing copy_from to smithi12556470-39 2021-03-24T22:02:33.509 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 39 v 767 (ObjNum 742 snap 240 seq_num 742) dirty exists 2021-03-24T22:02:33.510 INFO:tasks.rados.rados.0.smithi125.stdout:2174: finishing copy_from racing read to smithi12556470-39 2021-03-24T22:02:33.510 INFO:tasks.rados.rados.0.smithi125.stdout:2178: seq_num 760 ranges {773320=708187,1923609=470924} 2021-03-24T22:02:33.519 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:33.519 INFO:tasks.rados.rados.0.smithi125.stdout:2178: writing smithi12556470-46 from 773320 to 1481507 tid 1 2021-03-24T22:02:33.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:33.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:02:33.525 INFO:tasks.rados.rados.0.smithi125.stdout:2178: writing smithi12556470-46 from 1923609 to 2394533 tid 2 2021-03-24T22:02:33.525 INFO:tasks.rados.rados.0.smithi125.stdout:2172: done (5 left) 2021-03-24T22:02:33.525 INFO:tasks.rados.rados.0.smithi125.stdout:2173: done (4 left) 2021-03-24T22:02:33.526 INFO:tasks.rados.rados.0.smithi125.stdout:2174: done (3 left) 2021-03-24T22:02:33.526 INFO:tasks.rados.rados.0.smithi125.stdout:2179: read oid 29 snap -1 2021-03-24T22:02:33.526 INFO:tasks.rados.rados.0.smithi125.stdout:2179: expect (ObjNum 691 snap 218 seq_num 691) 2021-03-24T22:02:33.526 INFO:tasks.rados.rados.0.smithi125.stdout:2180: rollback oid 6 current snap is 246 2021-03-24T22:02:33.527 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 6 to 212 2021-03-24T22:02:33.527 INFO:tasks.rados.rados.0.smithi125.stdout:2176: finishing rollback tid 0 to smithi12556470-37 2021-03-24T22:02:33.527 INFO:tasks.rados.rados.0.smithi125.stdout:2181: copy_from oid 20 from oid 33 current snap is 246 2021-03-24T22:02:33.527 INFO:tasks.rados.rados.0.smithi125.stdout:2176: finishing rollback tid 1 to smithi12556470-37 2021-03-24T22:02:33.527 INFO:tasks.rados.rados.0.smithi125.stdout:2176: finishing rollback tid 2 to smithi12556470-37 2021-03-24T22:02:33.528 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 37 v 504 (ObjNum 635 snap 196 seq_num 635) dirty exists 2021-03-24T22:02:33.528 INFO:tasks.rados.rados.0.smithi125.stdout:2177: finishing write tid 1 to smithi12556470-43 2021-03-24T22:02:33.528 INFO:tasks.rados.rados.0.smithi125.stdout:2177: finishing write tid 2 to smithi12556470-43 2021-03-24T22:02:33.528 INFO:tasks.rados.rados.0.smithi125.stdout:2176: done (5 left) 2021-03-24T22:02:33.528 INFO:tasks.rados.rados.0.smithi125.stdout:2182: snap_remove snap 241 2021-03-24T22:02:33.625 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:33.675 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:33.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:33.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:02:33.784 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:33.834 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:33.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:33.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:02:33.941 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:33.995 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:34.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:34.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:02:34.102 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:34.158 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:34.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:34.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:02:34.264 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:34.321 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:34.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:34.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:02:34.337 INFO:tasks.rados.rados.0.smithi125.stdout:2182: done (5 left) 2021-03-24T22:02:34.338 INFO:tasks.rados.rados.0.smithi125.stdout:2183: write (excl) oid 36 current snap is 246 2021-03-24T22:02:34.338 INFO:tasks.rados.rados.0.smithi125.stdout:2177: finishing write tid 3 to smithi12556470-43 2021-03-24T22:02:34.338 INFO:tasks.rados.rados.0.smithi125.stdout:2183: seq_num 761 ranges {790127=499789,1698444=730935} 2021-03-24T22:02:34.346 INFO:tasks.rados.rados.0.smithi125.stdout:2183: writing smithi12556470-36 from 790127 to 1289916 tid 1 2021-03-24T22:02:34.354 INFO:tasks.rados.rados.0.smithi125.stdout:2183: writing smithi12556470-36 from 1698444 to 2429379 tid 2 2021-03-24T22:02:34.354 INFO:tasks.rados.rados.0.smithi125.stdout:2184: snap_create 2021-03-24T22:02:34.358 INFO:tasks.rados.rados.0.smithi125.stdout:2177: finishing write tid 4 to smithi12556470-43 2021-03-24T22:02:34.358 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 43 v 1150 (ObjNum 759 snap 246 seq_num 759) dirty exists 2021-03-24T22:02:34.358 INFO:tasks.rados.rados.0.smithi125.stdout:2177: left oid 43 (ObjNum 759 snap 246 seq_num 759) 2021-03-24T22:02:34.359 INFO:tasks.rados.rados.0.smithi125.stdout:2179: expect (ObjNum 691 snap 218 seq_num 691) 2021-03-24T22:02:34.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:34 smithi142 conmon[20125]: cluster 2021-03-24T22:02:32.906942+0000 mgr.y (mgr.14144) 563 : cluster [DBG] pgmap v1036: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 164 active+clean; 543 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 6.8 MiB/s rd, 1.6 MiB/s wr, 14 op/s; 1/1716 objects degraded (0.058%); 73/1716 objects misplaced (4.254%); 3.8 MiB/s, 10 objects/s recovering 2021-03-24T22:02:34.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:34 smithi142 conmon[20125]: audit 2021-03-24T22:02:33.330110+0000 mon.a (mon.0) 1613 : audit [INF] from='client.35483 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2021-03-24T22:02:34.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:34 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:02:34.367 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:34 smithi142 conmon[20125]: T22:02:33.330279+0000 mon.a (mon.0) 1614 : cluster [DBG] osdmap e583: 8 total, 8 up, 8 in 2021-03-24T22:02:34.429 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:34.478 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:34.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:34.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:34.552 INFO:tasks.rados.rados.0.smithi125.stdout:2178: finishing write tid 1 to smithi12556470-46 2021-03-24T22:02:34.552 INFO:tasks.rados.rados.0.smithi125.stdout:2180: finishing rollback tid 1 to smithi12556470-6 2021-03-24T22:02:34.553 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 6 v 398 (ObjNum 1116398816 snap 21916 seq_num 1117592608) dirty dne 2021-03-24T22:02:34.553 INFO:tasks.rados.rados.0.smithi125.stdout:2178: finishing write tid 2 to smithi12556470-46 2021-03-24T22:02:34.553 INFO:tasks.rados.rados.0.smithi125.stdout:2178: finishing write tid 3 to smithi12556470-46 2021-03-24T22:02:34.553 INFO:tasks.rados.rados.0.smithi125.stdout:2178: finishing write tid 4 to smithi12556470-46 2021-03-24T22:02:34.554 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 46 v 507 (ObjNum 760 snap 246 seq_num 760) dirty exists 2021-03-24T22:02:34.554 INFO:tasks.rados.rados.0.smithi125.stdout:2178: left oid 46 (ObjNum 760 snap 246 seq_num 760) 2021-03-24T22:02:34.554 INFO:tasks.rados.rados.0.smithi125.stdout:2181: finishing copy_from to smithi12556470-20 2021-03-24T22:02:34.554 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 20 v 777 (ObjNum 749 snap 241 seq_num 749) dirty exists 2021-03-24T22:02:34.555 INFO:tasks.rados.rados.0.smithi125.stdout:2181: finishing copy_from racing read to smithi12556470-20 2021-03-24T22:02:34.555 INFO:tasks.rados.rados.0.smithi125.stdout:2183: finishing write tid 1 to smithi12556470-36 2021-03-24T22:02:34.555 INFO:tasks.rados.rados.0.smithi125.stdout:2183: finishing write tid 2 to smithi12556470-36 2021-03-24T22:02:34.556 INFO:tasks.rados.rados.0.smithi125.stdout:2183: finishing write tid 3 to smithi12556470-36 2021-03-24T22:02:34.556 INFO:tasks.rados.rados.0.smithi125.stdout:2183: finishing write tid 4 to smithi12556470-36 2021-03-24T22:02:34.556 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 36 v 788 (ObjNum 761 snap 246 seq_num 761) dirty exists 2021-03-24T22:02:34.556 INFO:tasks.rados.rados.0.smithi125.stdout:2183: left oid 36 (ObjNum 761 snap 246 seq_num 761) 2021-03-24T22:02:34.584 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:34.597 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:34 smithi125 conmon[24927]: cluster 2021-03-24T22:02:32.906942+0000 mgr.y (mgr.14144) 563 : cluster [DBG] pgmap v1036: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 164 active+clean; 543 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 6.8 MiB/s rd, 1.6 MiB/s wr, 14 op/s; 1/1716 objects degraded (0.058%); 73/1716 objects misplaced (4.254%); 3.8 MiB/s, 10 objects/s recovering 2021-03-24T22:02:34.598 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:34 smithi125 conmon[24927]: audit 2021-03-24T22:02:33.330110+0000 mon.a (mon.0) 1613 : audit [INF] from='client.35483 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2021-03-24T22:02:34.598 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:34 smithi125 conmon[24927]: cluster 2021-03-24T22:02:33.330279+0000 mon.a (mon.0) 1614 : cluster [DBG] osdmap e583: 8 total, 8 up, 8 in 2021-03-24T22:02:34.599 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:34 smithi125 conmon[19936]: cluster 2021-03-24T22:02:32.906942+0000 mgr.y (mgr.14144) 563 : cluster [DBG] pgmap v1036: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 164 active+clean; 543 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 6.8 MiB/s rd, 1.6 MiB/s wr, 14 op/s; 1/1716 objects degraded (0.058%); 73/1716 objects misplaced (4.254%); 3.8 MiB/s, 10 objects/s recovering 2021-03-24T22:02:34.599 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:34 smithi125 conmon[19936]: audit 2021-03-24T22:02:33.330110+0000 mon.a (mon.0) 1613 : audit [INF] from='client.35483 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2021-03-24T22:02:34.599 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:34 smithi125 conmon[19936]: cluster 2021-03-24T22:02:34.599 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:34 smithi125 conmon[19936]: 2021-03-24T22 2021-03-24T22:02:34.600 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:34 smithi125 conmon[19936]: :02:33.330279+0000 mon.a (mon.0) 1614 : cluster [DBG] osdmap e583: 8 total, 8 up, 8 in 2021-03-24T22:02:34.639 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:34.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:34.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:02:34.704 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:02:34.731 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd reweight 5 1 2021-03-24T22:02:34.744 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:34.794 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:34.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:34.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:02:34.902 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:34.942 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2021-03-24T22:02:34.942 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T22:02:34.990 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:34.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:34.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:02:35.045 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:02:35.097 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:35.153 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:35.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:35.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:35.258 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:35.311 INFO:tasks.rados.rados.0.smithi125.stdout:2177: done (6 left) 2021-03-24T22:02:35.312 INFO:tasks.rados.rados.0.smithi125.stdout:2178: done (5 left) 2021-03-24T22:02:35.312 INFO:tasks.rados.rados.0.smithi125.stdout:2179: done (4 left) 2021-03-24T22:02:35.312 INFO:tasks.rados.rados.0.smithi125.stdout:2180: done (3 left) 2021-03-24T22:02:35.312 INFO:tasks.rados.rados.0.smithi125.stdout:2181: done (2 left) 2021-03-24T22:02:35.313 INFO:tasks.rados.rados.0.smithi125.stdout:2183: done (1 left) 2021-03-24T22:02:35.313 INFO:tasks.rados.rados.0.smithi125.stdout:2184: done (0 left) 2021-03-24T22:02:35.313 INFO:tasks.rados.rados.0.smithi125.stdout:2185: delete oid 20 current snap is 247 2021-03-24T22:02:35.316 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:35.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:35.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:02:35.323 INFO:teuthology.orchestra.run.smithi125.stderr:noscrub is set 2021-03-24T22:02:35.324 INFO:tasks.rados.rados.0.smithi125.stdout:2185: done (0 left) 2021-03-24T22:02:35.324 INFO:tasks.rados.rados.0.smithi125.stdout:2186: snap_create 2021-03-24T22:02:35.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:35 smithi125 conmon[19936]: cluster 2021-03-24T22:02:33 2021-03-24T22:02:35.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:35 smithi125 conmon[19936]: .959909+0000 osd.1 (osd.1) 53 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:02:35.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:35 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:02:35.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:35 smithi125 conmon[19936]: -03-24T22:02:33.962411+0000 osd.1 (osd.1) 54 : cluster [DBG] 2021-03-24T22:02:35.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:35 smithi125 conmon[19936]: purged_snaps scrub ok 2021-03-24T22:02:35.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:35 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T22:02:35.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:35 smithi125 conmon[19936]: 22:02:34.319416+0000 osd.7 (osd.7 2021-03-24T22:02:35.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:35 smithi125 conmon[19936]: ) 97 : cluster [DBG] 5.4 scrub ok 2021-03-24T22:02:35.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:35 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:02:35.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:35 smithi125 conmon[19936]: -03-24T22:02:34.333284+0000 2021-03-24T22:02:35.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:35 smithi125 conmon[19936]: mon.a (mon.0) 1615 : cluster [DBG] osdmap e584: 8 total, 8 up, 8 in 2021-03-24T22:02:35.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:35 smithi125 conmon[19936]: audit 2021 2021-03-24T22:02:35.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:35 smithi125 conmon[19936]: -03-24T22:02:35.206956+0000 mon.a ( 2021-03-24T22:02:35.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:35 smithi125 conmon[19936]: mon.0) 1616 : audit [INF] from='client.? v1:172.21.15.125:0/683402277' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T22:02:35.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:35 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:02:35.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:35 smithi125 conmon[19936]: -03-24T22:02:35.298187+0000 mon.a 2021-03-24T22:02:35.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:35 smithi125 conmon[19936]: (mon.0) 1617 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:02:35.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:35 smithi125 conmon[19936]: audit 2021 2021-03-24T22:02:35.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:35 smithi125 conmon[19936]: -03-24T22:02:35.307523+0000 mon.a 2021-03-24T22:02:35.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:35 smithi125 conmon[19936]: (mon.0) 1618 : audit [INF] from='client.? v1:172.21.15.125:0/683402277' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T22:02:35.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:35 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:02:35.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:35 smithi125 conmon[19936]: 24T22:02:35.307735+0000 mon.a (mon. 2021-03-24T22:02:35.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:35 smithi125 conmon[19936]: 0) 1619 : cluster [DBG] osdmap e585: 8 total, 8 up, 8 in 2021-03-24T22:02:35.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:35 smithi125 conmon[24927]: cluster 2021-03-24T22:02:33.959909+0000 2021-03-24T22:02:35.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:35 smithi125 conmon[24927]: osd.1 (osd.1) 53 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:02:35.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:35 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T22:02:35.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:35 smithi125 conmon[24927]: T22:02:33.962411+0000 osd.1 (osd.1) 54 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:02:35.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:35 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:02:35.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:35 smithi125 conmon[24927]: 24T22:02:34.319416+0000 osd.7 (osd.7 2021-03-24T22:02:35.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:35 smithi125 conmon[24927]: ) 97 : cluster [DBG] 5.4 scrub ok 2021-03-24T22:02:35.420 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:35 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T22:02:35.420 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:35 smithi125 conmon[24927]: 22:02:34.333284+0000 mon.a (mon.0) 1615 : 2021-03-24T22:02:35.420 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:35 smithi125 conmon[24927]: cluster [DBG] osdmap e584: 8 total, 8 up, 8 in 2021-03-24T22:02:35.420 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:35 smithi125 conmon[24927]: audit 2021-03-24T22:02:35.206956 2021-03-24T22:02:35.420 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:35 smithi125 conmon[24927]: +0000 mon.a (mon.0) 1616 : audit [INF] from='client.? v1:172.21.15.125:0/683402277' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T22:02:35.421 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:35 smithi125 conmon[24927]: 2021-03-24T22:02:35.421 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:35 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:02:35.421 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:35 smithi125 conmon[24927]: -03-24T22:02:35.298187+0000 mon.a (mon.0) 1617 2021-03-24T22:02:35.421 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:35 smithi125 conmon[24927]: : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:02:35.421 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:35 smithi125 conmon[24927]: audit 2021-03-24T22:02: 2021-03-24T22:02:35.422 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:35 smithi125 conmon[24927]: 35.307523+0000 mon.a (mon.0) 1618 : audit 2021-03-24T22:02:35.422 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:35 smithi125 conmon[24927]: [INF] from='client.? v1:172.21.15.125:0/683402277' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T22:02:35.422 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:35 smithi125 conmon[24927]: cluster 2021-03-24T22:02:35.307735+0000 2021-03-24T22:02:35.422 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:35 smithi125 conmon[24927]: mon.a (mon.0) 1619 : cluster [DBG] osdmap e585: 8 total, 8 up, 8 in 2021-03-24T22:02:35.423 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:35 smithi125 conmon[24927]: 2021-03-24T22:02:35.423 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:35.435 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:35 smithi142 conmon[20125]: cluster 2021-03-24T22:02:35.436 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:35 smithi142 conmon[20125]: 2021-03-24T22:02:35.436 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:35 smithi142 conmon[20125]: 2021 2021-03-24T22:02:35.436 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:35 smithi142 conmon[20125]: - 2021-03-24T22:02:35.436 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:35 smithi142 conmon[20125]: 03-24T22:02:33. 2021-03-24T22:02:35.436 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:35 smithi142 conmon[20125]: 959909+0000 osd.1 (osd.1) 53 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:02:35.437 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:35 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:02:35.437 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:35 smithi142 conmon[20125]: 03-24T22:02:33.962411+0000 osd.1 (osd.1 2021-03-24T22:02:35.437 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:35 smithi142 conmon[20125]: ) 54 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:02:35.437 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:35 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:02:35.438 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:35 smithi142 conmon[20125]: T22:02:34.319416+0000 2021-03-24T22:02:35.438 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:35 smithi142 conmon[20125]: osd.7 (osd.7) 97 : 2021-03-24T22:02:35.438 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:35 smithi142 conmon[20125]: cluster [DBG] 5.4 scrub ok 2021-03-24T22:02:35.438 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:35 smithi142 conmon[20125]: cluster 2021-03-24T22:02:35.438 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:35 smithi142 conmon[20125]: 2021-03-24T22:02:34 2021-03-24T22:02:35.439 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:35 smithi142 conmon[20125]: .333284+0000 mon.a (mon. 2021-03-24T22:02:35.439 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:35 smithi142 conmon[20125]: 0) 1615 : cluster [DBG] osdmap e584: 8 total, 8 up, 8 in 2021-03-24T22:02:35.439 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:35 smithi142 conmon[20125]: 2021-03-24T22:02:35.439 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:35 smithi142 conmon[20125]: audit 2021- 2021-03-24T22:02:35.439 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:35 smithi142 conmon[20125]: 03-24T22:02:35.206956+0000 2021-03-24T22:02:35.440 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:35 smithi142 conmon[20125]: mon.a (mon.0) 1616 : audit [INF] 2021-03-24T22:02:35.440 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:35 smithi142 conmon[20125]: from='client.? v1:172.21.15.125:0/683402277' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T22:02:35.440 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:35 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:02:35.440 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:35 smithi142 conmon[20125]: -03-24T22:02:35.298187+0000 2021-03-24T22:02:35.441 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:35 smithi142 conmon[20125]: mon.a (mon.0) 1617 : cluster 2021-03-24T22:02:35.441 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:35 smithi142 conmon[20125]: [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:02:35.441 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:35 smithi142 conmon[20125]: audit 2021- 2021-03-24T22:02:35.441 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:35 smithi142 conmon[20125]: 03-24T22:02:35.307523+0000 2021-03-24T22:02:35.442 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:35 smithi142 conmon[20125]: mon.a (mon.0) 1618 : audit 2021-03-24T22:02:35.442 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:35 smithi142 conmon[20125]: [INF] from='client.? v1:172.21.15.125:0/683402277' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T22:02:35.442 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:35 smithi142 conmon[20125]: cluster 2021-03-24T22:02:35.307735+0000 mon.a (mon 2021-03-24T22:02:35.442 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:35 smithi142 conmon[20125]: .0) 1619 : cluster [DBG] osdmap e585: 8 total, 8 up, 8 in 2021-03-24T22:02:35.478 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:35.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:35.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:02:35.583 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:35.640 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:35.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:35.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:35.747 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:35.803 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:35.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:35.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:35.910 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:35.931 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.7 dump_blocked_ops 2021-03-24T22:02:35.981 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:35.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:35.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:02:36.086 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:36.147 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:36.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:36.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:02:36.241 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T22:02:36.257 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:36.325 INFO:tasks.rados.rados.0.smithi125.stdout:2186: done (0 left) 2021-03-24T22:02:36.326 INFO:tasks.rados.rados.0.smithi125.stdout:2187: snap_remove snap 208 2021-03-24T22:02:36.329 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:36.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:36.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:02:36.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:36 smithi125 conmon[19936]: cluster 2021-03-24T22:02:34.813622+0000 osd.6 ( 2021-03-24T22:02:36.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:36 smithi125 conmon[19936]: osd.6) 43 : cluster [DBG] 4.13 deep-scrub ok 2021-03-24T22:02:36.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:36 smithi125 conmon[19936]: cluster 2021-03-24T22:02:34.907785+0000 mgr.y (mgr.14144) 564 : cluster [DBG] 2021-03-24T22:02:36.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:36 smithi125 conmon[19936]: pgmap v1039: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 517 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.7 MiB/s rd, 2.3 MiB/s wr, 16 op/s; 72/1680 objects misplaced (4.286%); 1.9 MiB/s, 5 objects/s recovering 2021-03-24T22:02:36.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:36 smithi125 conmon[19936]: cluster 2021-03-24T22:02: 2021-03-24T22:02:36.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:36 smithi125 conmon[19936]: 35.095483+0000 osd.5 (osd.5) 27 : cluster 2021-03-24T22:02:36.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:36 smithi125 conmon[19936]: [DBG] 6.1a scrub ok 2021-03-24T22:02:36.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:36 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:02:36.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:36 smithi125 conmon[19936]: 24T22:02:35.330179+0000 2021-03-24T22:02:36.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:36 smithi125 conmon[19936]: mon.a (mon.0) 1620 : cluster 2021-03-24T22:02:36.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:36 smithi125 conmon[19936]: [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/1716 objects degraded (0.058%), 1 pg degraded) 2021-03-24T22:02:36.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:36 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:02:36.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:36 smithi125 conmon[19936]: 24T22:02:36.321900+0000 2021-03-24T22:02:36.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:36 smithi125 conmon[19936]: mon.a (mon.0) 1621 : 2021-03-24T22:02:36.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:36 smithi125 conmon[19936]: cluster [DBG] osdmap e586: 8 total, 8 up, 8 in 2021-03-24T22:02:36.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:36 smithi125 conmon[24927]: cluster 2021-03-24T22:02: 2021-03-24T22:02:36.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:36 smithi125 conmon[24927]: 34.813622+0000 osd.6 (osd.6) 43 : cluster [DBG] 4.13 deep-scrub ok 2021-03-24T22:02:36.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:36 smithi125 conmon[24927]: cluster 2021-03-24T22:02:36.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:36 smithi125 conmon[24927]: 2021-03-24T22:02:34.907785 2021-03-24T22:02:36.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:36 smithi125 conmon[24927]: +0000 mgr.y (mgr.14144) 564 : cluster 2021-03-24T22:02:36.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:36 smithi125 conmon[24927]: [DBG] pgmap v1039: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 517 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.7 MiB/s rd, 2.3 MiB/s wr, 16 op/s; 72/1680 objects misplaced (4.286%); 1.9 MiB/s, 5 objects/s recovering 2021-03-24T22:02:36.417 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:36 smithi125 conmon[24927]: cluster 2021-03-24T22:02:36.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:36 smithi125 conmon[24927]: 2021-03- 2021-03-24T22:02:36.418 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:36 smithi125 conmon[24927]: 24T22:02:35.095483+0000 2021-03-24T22:02:36.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:36 smithi125 conmon[24927]: osd.5 (osd.5) 27 : cluster 2021-03-24T22:02:36.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:36 smithi125 conmon[24927]: [DBG] 6.1a scrub ok 2021-03-24T22:02:36.419 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:36 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:02:36.420 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:36 smithi125 conmon[24927]: -03-24T22:02:35.330179+0000 mon.a 2021-03-24T22:02:36.420 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:36 smithi125 conmon[24927]: (mon.0) 1620 : cluster [INF] 2021-03-24T22:02:36.420 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:36 smithi125 conmon[24927]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/1716 objects degraded (0.058%), 1 pg degraded) 2021-03-24T22:02:36.420 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:36 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:02:36.420 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:36 smithi125 conmon[24927]: -03-24T22:02:36. 2021-03-24T22:02:36.421 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:36 smithi125 conmon[24927]: 321900+0000 mon.a (mon.0) 1621 2021-03-24T22:02:36.421 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:36 smithi125 conmon[24927]: : cluster [DBG] osdmap e586: 8 total, 8 up, 8 in 2021-03-24T22:02:36.435 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:36.492 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:36.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:36.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:02:36.599 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:36.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:36 smithi142 conmon[20125]: cluster 2021-03-24T22:02:34.813622+0000 osd.6 (osd 2021-03-24T22:02:36.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:36 smithi142 conmon[20125]: .6) 43 : cluster [DBG] 4.13 deep-scrub ok 2021-03-24T22:02:36.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:36 smithi142 conmon[20125]: cluster 2021-03-24T22:02:34.907785+0000 mgr.y (mgr.14144) 564 : cluster [DBG] pgmap v1039: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 517 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.7 MiB/s rd, 2.3 MiB/s wr, 16 op/s; 72/1680 objects misplaced (4.286%); 1.9 MiB/s, 5 objects/s recovering 2021-03-24T22:02:36.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:36 smithi142 conmon[20125]: cluster 2021-03-24T22 2021-03-24T22:02:36.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:36 smithi142 conmon[20125]: :02:35.095483+0000 osd.5 (osd.5) 27 : cluster [DBG] 6.1a scrub ok 2021-03-24T22:02:36.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:36 smithi142 conmon[20125]: cluster 2021-03-24T22:02:35. 2021-03-24T22:02:36.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:36 smithi142 conmon[20125]: 330179+0000 mon.a (mon.0) 1620 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/1716 objects degraded (0.058%), 1 pg degraded) 2021-03-24T22:02:36.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:36 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:02:36.613 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:36 smithi142 conmon[20125]: 24T22:02:36.321900+0000 mon.a (mon.0) 1621 2021-03-24T22:02:36.614 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:36 smithi142 conmon[20125]: : cluster [DBG] osdmap e586: 8 total, 8 up, 8 in 2021-03-24T22:02:36.655 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:36.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:36.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:02:36.762 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:36.819 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:36.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:36.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:02:36.923 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:36.985 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:36.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:36.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:02:37.089 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:37.152 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:37.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:37.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:37.258 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:37.313 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:37.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:37.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:02:37.330 INFO:tasks.rados.rados.0.smithi125.stdout:2187: done (0 left) 2021-03-24T22:02:37.330 INFO:tasks.rados.rados.0.smithi125.stdout:2188: rollback oid 11 current snap is 248 2021-03-24T22:02:37.331 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 11 to 179 2021-03-24T22:02:37.331 INFO:tasks.rados.rados.0.smithi125.stdout:2189: read oid 8 snap -1 2021-03-24T22:02:37.331 INFO:tasks.rados.rados.0.smithi125.stdout:2189: expect (ObjNum 730 snap 234 seq_num 730) 2021-03-24T22:02:37.331 INFO:tasks.rados.rados.0.smithi125.stdout:2190: snap_create 2021-03-24T22:02:37.346 INFO:tasks.rados.rados.0.smithi125.stdout:2189: expect (ObjNum 730 snap 234 seq_num 730) 2021-03-24T22:02:37.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:37 smithi125 conmon[24927]: cluster 2021-03-24T22:02:35.629006+0000 2021-03-24T22:02:37.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:37 smithi125 conmon[24927]: osd.4 (osd.4) 113 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:02:37.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:37 smithi125 conmon[24927]: cluster 2021-03-24T22:02:35.630862+0000 osd.4 (osd.4) 114 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:02:37.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:37 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:02:37.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:37 smithi125 conmon[24927]: 02:37.325428+0000 mon.a (mon.0) 1622 : cluster [DBG] osdmap e587: 8 total, 8 up, 8 in 2021-03-24T22:02:37.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:37 smithi125 conmon[19936]: cluster 2021-03-24T22:02:35.629006+0000 osd.4 2021-03-24T22:02:37.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:37 smithi125 conmon[19936]: (osd.4) 113 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:02:37.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:37 smithi125 conmon[19936]: cluster 2021-03-24T22:02:35.630862+0000 osd.4 (osd.4) 114 : cluster 2021-03-24T22:02:37.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:37 smithi125 conmon[19936]: [DBG] purged_snaps scrub ok 2021-03-24T22:02:37.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:37 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:02:37.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:37 smithi125 conmon[19936]: T22:02:37.325428+0000 mon.a (mon.0) 1622 : cluster [DBG] 2021-03-24T22:02:37.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:37 smithi125 conmon[19936]: osdmap e587: 8 total, 8 up, 8 in 2021-03-24T22:02:37.419 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:37.430 INFO:tasks.rados.rados.0.smithi125.stdout:2188: finishing rollback tid 1 to smithi12556470-11 2021-03-24T22:02:37.431 INFO:tasks.rados.rados.0.smithi125.stdout:2188: finishing rollback tid 2 to smithi12556470-11 2021-03-24T22:02:37.431 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 11 v 374 (ObjNum 495 snap 142 seq_num 495) dirty exists 2021-03-24T22:02:37.462 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:37.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:37.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:02:37.568 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:37.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:37 smithi142 conmon[20125]: cluster 2021-03-24T22:02:35.629006+0000 2021-03-24T22:02:37.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:37 smithi142 conmon[20125]: osd.4 (osd.4) 113 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:02:37.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:37 smithi142 conmon[20125]: cluster 2021-03-24T22:02:35.630862+0000 osd.4 (osd.4) 114 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:02:37.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:37 smithi142 conmon[20125]: cluster 2021-03-24T22:02:37.325428+0000 mon.a (mon.0) 2021-03-24T22:02:37.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:37 smithi142 conmon[20125]: 1622 : cluster [DBG] osdmap e587: 8 total, 8 up, 8 in 2021-03-24T22:02:37.632 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:37.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:37.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:02:37.737 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:37.793 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:37.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:37.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:02:37.898 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:37.958 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:37.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:37.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:02:38.065 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:38.120 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:38.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:38.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:02:38.225 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:38.272 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:38.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:38.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:02:38.331 INFO:tasks.rados.rados.0.smithi125.stdout:2188: done (2 left) 2021-03-24T22:02:38.331 INFO:tasks.rados.rados.0.smithi125.stdout:2189: done (1 left) 2021-03-24T22:02:38.331 INFO:tasks.rados.rados.0.smithi125.stdout:2190: done (0 left) 2021-03-24T22:02:38.332 INFO:tasks.rados.rados.0.smithi125.stdout:2191: read oid 39 snap -1 2021-03-24T22:02:38.332 INFO:tasks.rados.rados.0.smithi125.stdout:2191: expect (ObjNum 742 snap 240 seq_num 742) 2021-03-24T22:02:38.332 INFO:tasks.rados.rados.0.smithi125.stdout:2192: write (excl) oid 13 current snap is 249 2021-03-24T22:02:38.332 INFO:tasks.rados.rados.0.smithi125.stdout:2192: seq_num 763 ranges {423741=650287,1699055=618251} 2021-03-24T22:02:38.336 INFO:tasks.rados.rados.0.smithi125.stdout:2192: writing smithi12556470-13 from 423741 to 1074028 tid 1 2021-03-24T22:02:38.344 INFO:tasks.rados.rados.0.smithi125.stdout:2192: writing smithi12556470-13 from 1699055 to 2317306 tid 2 2021-03-24T22:02:38.344 INFO:tasks.rados.rados.0.smithi125.stdout:2193: rollback oid 28 current snap is 249 2021-03-24T22:02:38.344 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 28 to 246 2021-03-24T22:02:38.345 INFO:tasks.rados.rados.0.smithi125.stdout:2194: write oid 33 current snap is 249 2021-03-24T22:02:38.345 INFO:tasks.rados.rados.0.smithi125.stdout:2194: seq_num 764 ranges {440548=725536,1757537=478262,2813990=26558} 2021-03-24T22:02:38.353 INFO:tasks.rados.rados.0.smithi125.stdout:2194: writing smithi12556470-33 from 440548 to 1166084 tid 1 2021-03-24T22:02:38.360 INFO:tasks.rados.rados.0.smithi125.stdout:2194: writing smithi12556470-33 from 1757537 to 2235799 tid 2 2021-03-24T22:02:38.364 INFO:tasks.rados.rados.0.smithi125.stdout:2194: writing smithi12556470-33 from 2813990 to 2840548 tid 3 2021-03-24T22:02:38.364 INFO:tasks.rados.rados.0.smithi125.stdout:2195: read oid 26 snap -1 2021-03-24T22:02:38.365 INFO:tasks.rados.rados.0.smithi125.stdout:2195: expect (ObjNum 756 snap 244 seq_num 756) 2021-03-24T22:02:38.365 INFO:tasks.rados.rados.0.smithi125.stdout:2196: copy_from oid 9 from oid 35 current snap is 249 2021-03-24T22:02:38.365 INFO:tasks.rados.rados.0.smithi125.stdout:2192: finishing write tid 1 to smithi12556470-13 2021-03-24T22:02:38.365 INFO:tasks.rados.rados.0.smithi125.stdout:2193: finishing rollback tid 0 to smithi12556470-28 2021-03-24T22:02:38.366 INFO:tasks.rados.rados.0.smithi125.stdout:2193: finishing rollback tid 1 to smithi12556470-28 2021-03-24T22:02:38.366 INFO:tasks.rados.rados.0.smithi125.stdout:2193: finishing rollback tid 2 to smithi12556470-28 2021-03-24T22:02:38.366 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 28 v 553 (ObjNum 610 snap 185 seq_num 610) dirty exists 2021-03-24T22:02:38.366 INFO:tasks.rados.rados.0.smithi125.stdout:2191: expect (ObjNum 742 snap 240 seq_num 742) 2021-03-24T22:02:38.378 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:38.433 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:38.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:38.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:02:38.451 INFO:tasks.rados.rados.0.smithi125.stdout:2194: finishing write tid 1 to smithi12556470-33 2021-03-24T22:02:38.451 INFO:tasks.rados.rados.0.smithi125.stdout:2192: finishing write tid 2 to smithi12556470-13 2021-03-24T22:02:38.452 INFO:tasks.rados.rados.0.smithi125.stdout:2192: finishing write tid 3 to smithi12556470-13 2021-03-24T22:02:38.452 INFO:tasks.rados.rados.0.smithi125.stdout:2194: finishing write tid 2 to smithi12556470-33 2021-03-24T22:02:38.452 INFO:tasks.rados.rados.0.smithi125.stdout:2192: finishing write tid 4 to smithi12556470-13 2021-03-24T22:02:38.452 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 13 v 323 (ObjNum 763 snap 249 seq_num 763) dirty exists 2021-03-24T22:02:38.453 INFO:tasks.rados.rados.0.smithi125.stdout:2192: left oid 13 (ObjNum 763 snap 249 seq_num 763) 2021-03-24T22:02:38.453 INFO:tasks.rados.rados.0.smithi125.stdout:2195: expect (ObjNum 756 snap 244 seq_num 756) 2021-03-24T22:02:38.473 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2021-03-24T22:02:38.473 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T22:02:38.538 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:38.553 INFO:tasks.rados.rados.0.smithi125.stdout:2194: finishing write tid 3 to smithi12556470-33 2021-03-24T22:02:38.553 INFO:tasks.rados.rados.0.smithi125.stdout:2194: finishing write tid 4 to smithi12556470-33 2021-03-24T22:02:38.553 INFO:tasks.rados.rados.0.smithi125.stdout:2191: done (5 left) 2021-03-24T22:02:38.554 INFO:tasks.rados.rados.0.smithi125.stdout:2192: done (4 left) 2021-03-24T22:02:38.554 INFO:tasks.rados.rados.0.smithi125.stdout:2193: done (3 left) 2021-03-24T22:02:38.554 INFO:tasks.rados.rados.0.smithi125.stdout:2195: done (2 left) 2021-03-24T22:02:38.554 INFO:tasks.rados.rados.0.smithi125.stdout:2197: snap_create 2021-03-24T22:02:38.555 INFO:tasks.rados.rados.0.smithi125.stdout:2194: finishing write tid 5 to smithi12556470-33 2021-03-24T22:02:38.555 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 33 v 824 (ObjNum 764 snap 249 seq_num 764) dirty exists 2021-03-24T22:02:38.555 INFO:tasks.rados.rados.0.smithi125.stdout:2194: left oid 33 (ObjNum 764 snap 249 seq_num 764) 2021-03-24T22:02:38.555 INFO:tasks.rados.rados.0.smithi125.stdout:2196: finishing copy_from to smithi12556470-9 2021-03-24T22:02:38.556 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 9 v 216 (ObjNum 746 snap 241 seq_num 746) dirty exists 2021-03-24T22:02:38.556 INFO:tasks.rados.rados.0.smithi125.stdout:2196: finishing copy_from racing read to smithi12556470-9 2021-03-24T22:02:38.590 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:38 smithi125 conmon[19936]: cluster 2021-03-24T22:02:36.908804+0000 mgr.y ( 2021-03-24T22:02:38.590 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:38 smithi125 conmon[19936]: mgr.14144) 565 : cluster [DBG] pgmap v1042: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 499 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.9 MiB/s rd, 2.6 MiB/s wr, 16 op/s; 72/1656 objects misplaced (4.348%); 1.9 MiB/s, 5 objects/s recovering 2021-03-24T22:02:38.591 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:38 smithi125 conmon[19936]: cluster 2021-03-24T22:02:38.326095+0000 mon.a (mon.0) 1623 : cluster [DBG] osdmap e588: 8 total, 8 up, 8 in 2021-03-24T22:02:38.591 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:38 smithi125 conmon[24927]: cluster 2021-03-24T22:02:36.908804+0000 mgr.y 2021-03-24T22:02:38.591 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:38 smithi125 conmon[24927]: (mgr.14144) 565 : cluster [DBG] pgmap v1042: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 499 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.9 MiB/s rd, 2.6 MiB/s wr, 16 op/s; 72/1656 objects misplaced (4.348%); 1.9 MiB/s, 5 objects/s recovering 2021-03-24T22:02:38.592 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:38 smithi125 conmon[24927]: cluster 2021-03-24T22:02:38.326095+0000 mon.a (mon.0) 1623 : cluster [DBG] 2021-03-24T22:02:38.592 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:38 smithi125 conmon[24927]: osdmap e588: 8 total, 8 up, 8 in 2021-03-24T22:02:38.592 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:38.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:38.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:02:38.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:38 smithi142 conmon[20125]: cluster 2021-03-24T22:02:36.908804+0000 mgr.y (mgr.14144) 565 : cluster 2021-03-24T22:02:38.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:38 smithi142 conmon[20125]: [DBG] pgmap v1042: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 499 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.9 MiB/s rd, 2.6 MiB/s wr, 16 op/s; 72/1656 objects misplaced (4.348%); 1.9 MiB/s, 5 objects/s recovering 2021-03-24T22:02:38.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:38 smithi142 conmon[20125]: cluster 2021-03-24T22:02:38.326095+0000 mon.a (mon.0) 1623 : cluster [DBG] osdmap e588: 8 total, 8 up, 8 in 2021-03-24T22:02:38.696 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:38.755 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:38.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:38.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:02:38.790 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T22:02:38.815 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T22:02:38.836 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T22:02:38.856 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T22:02:38.862 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:38.879 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T22:02:38.902 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T22:02:38.915 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:38.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:38.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:02:38.927 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T22:02:38.951 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T22:02:38.953 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-24T22:02:38.953 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-24T22:02:38.953 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-24T22:02:38.954 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-24T22:02:38.954 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-24T22:02:38.954 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-24T22:02:38.954 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-24T22:02:38.955 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-24T22:02:39.023 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:39.083 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:39.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:39.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:02:39.188 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:39.241 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:39.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:39.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:39.346 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:39.360 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-24T22:02:39.377 INFO:tasks.rados.rados.0.smithi125.stdout:2194: done (2 left) 2021-03-24T22:02:39.377 INFO:tasks.rados.rados.0.smithi125.stdout:2196: done (1 left) 2021-03-24T22:02:39.377 INFO:tasks.rados.rados.0.smithi125.stdout:2197: done (0 left) 2021-03-24T22:02:39.378 INFO:tasks.rados.rados.0.smithi125.stdout:2198: copy_from oid 5 from oid 11 current snap is 250 2021-03-24T22:02:39.378 INFO:tasks.rados.rados.0.smithi125.stdout:2199: write (excl) oid 34 current snap is 250 2021-03-24T22:02:39.378 INFO:tasks.rados.rados.0.smithi125.stdout:2199: seq_num 765 ranges {457355=400785,1416019=621920,2725060=132295} 2021-03-24T22:02:39.381 INFO:teuthology.orchestra.run.smithi125.stderr:reweighted osd.5 to 1 (10000) 2021-03-24T22:02:39.382 INFO:tasks.rados.rados.0.smithi125.stdout:2199: writing smithi12556470-34 from 457355 to 858140 tid 1 2021-03-24T22:02:39.389 INFO:tasks.rados.rados.0.smithi125.stdout:2199: writing smithi12556470-34 from 1416019 to 2037939 tid 2 2021-03-24T22:02:39.395 INFO:tasks.rados.rados.0.smithi125.stdout:2199: writing smithi12556470-34 from 2725060 to 2857355 tid 3 2021-03-24T22:02:39.395 INFO:tasks.rados.rados.0.smithi125.stdout:2200: read oid 10 snap -1 2021-03-24T22:02:39.395 INFO:tasks.rados.rados.0.smithi125.stdout:2200: expect (ObjNum 751 snap 241 seq_num 751) 2021-03-24T22:02:39.395 INFO:tasks.rados.rados.0.smithi125.stdout:2201: snap_create 2021-03-24T22:02:39.396 INFO:tasks.rados.rados.0.smithi125.stdout:2199: finishing write tid 1 to smithi12556470-34 2021-03-24T22:02:39.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:39 smithi125 conmon[19936]: audit 2021-03-24T22:02:38.768969+0000 mon.b (mon. 2021-03-24T22:02:39.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:39 smithi125 conmon[19936]: 1) 39 : audit [DBG] from='client.? v1:172.21.15.125:0/585044795' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:02:39.404 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:39 smithi125 conmon[19936]: audit 2021-03-24T22:02:39.205957+0000 mon.a (mon.0) 1624 : audit [INF] from='client.? v1:172.21.15.125:0/1045809968' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2021-03-24T22:02:39.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:39 smithi125 conmon[24927]: audit 2021-03-24T22:02:38.768969+0000 2021-03-24T22:02:39.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:39 smithi125 conmon[24927]: mon.b (mon.1) 39 : audit [DBG] from='client.? v1:172.21.15.125:0/585044795' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:02:39.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:39 smithi125 conmon[24927]: audit 2021-03-24T22:02:39.205957+0000 2021-03-24T22:02:39.405 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:39 smithi125 conmon[24927]: mon.a (mon.0) 1624 : audit [INF] from='client.? v1:172.21.15.125:0/1045809968' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2021-03-24T22:02:39.407 INFO:tasks.rados.rados.0.smithi125.stdout:2198: finishing copy_from to smithi12556470-5 2021-03-24T22:02:39.407 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 5 v 374 (ObjNum 495 snap 142 seq_num 495) dirty exists 2021-03-24T22:02:39.407 INFO:tasks.rados.rados.0.smithi125.stdout:2199: finishing write tid 2 to smithi12556470-34 2021-03-24T22:02:39.407 INFO:tasks.rados.rados.0.smithi125.stdout:2199: finishing write tid 3 to smithi12556470-34 2021-03-24T22:02:39.407 INFO:tasks.rados.rados.0.smithi125.stdout:2198: finishing copy_from racing read to smithi12556470-5 2021-03-24T22:02:39.408 INFO:tasks.rados.rados.0.smithi125.stdout:2199: finishing write tid 4 to smithi12556470-34 2021-03-24T22:02:39.408 INFO:tasks.rados.rados.0.smithi125.stdout:2199: finishing write tid 5 to smithi12556470-34 2021-03-24T22:02:39.408 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 34 v 575 (ObjNum 765 snap 250 seq_num 765) dirty exists 2021-03-24T22:02:39.408 INFO:tasks.rados.rados.0.smithi125.stdout:2199: left oid 34 (ObjNum 765 snap 250 seq_num 765) 2021-03-24T22:02:39.410 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:39.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:39.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:02:39.414 INFO:tasks.rados.rados.0.smithi125.stdout:2200: expect (ObjNum 751 snap 241 seq_num 751) 2021-03-24T22:02:39.515 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:39.570 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.7 dump_historic_ops 2021-03-24T22:02:39.574 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:39.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:39.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:02:39.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:39 smithi142 conmon[20125]: audit 2021-03- 2021-03-24T22:02:39.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:39 smithi142 conmon[20125]: 24T22:02:38.768969+0000 mon.b (mon.1) 39 : audit [DBG] 2021-03-24T22:02:39.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:39 smithi142 conmon[20125]: from='client.? v1:172.21.15.125:0/585044795' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:02:39.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:39 smithi142 conmon[20125]: audit 2021-03-24T22:02:39.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:39 smithi142 conmon[20125]: 2021-03-24T22:02 2021-03-24T22:02:39.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:39 smithi142 conmon[20125]: :39.205957+0000 2021-03-24T22:02:39.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:39 smithi142 conmon[20125]: mon.a (mon.0 2021-03-24T22:02:39.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:39 smithi142 conmon[20125]: ) 1624 : audit 2021-03-24T22:02:39.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:39 smithi142 conmon[20125]: [INF] from='client.? v1:172.21.15.125:0/1045809968' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2021-03-24T22:02:39.670 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:02:39.679 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:39.731 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:39.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:39.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:02:39.837 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:39.877 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T22:02:39.894 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:39.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:39.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:02:40.000 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:40.051 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:40.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:40.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:02:40.157 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:40.215 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:40.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:40.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:02:40.317 INFO:tasks.rados.rados.0.smithi125.stdout:2198: done (3 left) 2021-03-24T22:02:40.317 INFO:tasks.rados.rados.0.smithi125.stdout:2199: done (2 left) 2021-03-24T22:02:40.318 INFO:tasks.rados.rados.0.smithi125.stdout:2200: done (1 left) 2021-03-24T22:02:40.318 INFO:tasks.rados.rados.0.smithi125.stdout:2201: done (0 left) 2021-03-24T22:02:40.318 INFO:tasks.rados.rados.0.smithi125.stdout:2202: snap_remove snap 233 2021-03-24T22:02:40.328 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:40.371 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:40 smithi125 conmon[19936]: cluster 2021-03-24T22:02:38 2021-03-24T22:02:40.371 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:40 smithi125 conmon[19936]: .909698+0000 mgr.y (mgr.14144) 566 : cluster [DBG] pgmap v1045: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 499 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 301 KiB/s wr, 1 op/s; 72/1656 objects misplaced (4.348%) 2021-03-24T22:02:40.371 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:40 smithi125 conmon[19936]: audit 2021-03-24T22:02 2021-03-24T22:02:40.371 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:40 smithi125 conmon[19936]: :39.368404+0000 mon.a (mon.0) 1625 : audit [INF] from='client.? v1:172.21.15.125:0/1045809968' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2021-03-24T22:02:40.372 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:40 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:02:40.372 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:40 smithi125 conmon[19936]: 03-24T22:02:39.368503+0000 mon.a (mon.0 2021-03-24T22:02:40.372 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:40 smithi125 conmon[19936]: ) 1626 : cluster [DBG] osdmap e589: 8 total, 8 up, 8 in 2021-03-24T22:02:40.372 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:40 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T22:02:40.373 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:40 smithi125 conmon[19936]: -24T22:02:40.309869+0000 mon.a (mon.0) 2021-03-24T22:02:40.373 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:40 smithi125 conmon[19936]: 1627 : cluster [DBG] osdmap e590: 8 total, 8 up, 8 in 2021-03-24T22:02:40.376 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:40.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:40.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:02:40.458 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:40 smithi142 conmon[20125]: cluster 2021-03-24T22:02:38. 2021-03-24T22:02:40.458 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:40 smithi142 conmon[20125]: 909698+0000 mgr.y (mgr.14144) 566 : cluster [DBG] pgmap v1045: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 499 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 301 KiB/s wr, 1 op/s; 72/1656 objects misplaced (4.348%) 2021-03-24T22:02:40.458 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:40 smithi142 conmon[20125]: audit 2021-03-24T22:02:39.368404+0000 mon.a (mon. 2021-03-24T22:02:40.458 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:40 smithi142 conmon[20125]: 0) 1625 : audit [INF] from='client.? v1:172.21.15.125:0/1045809968' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2021-03-24T22:02:40.459 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:40 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:02:40.459 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:40 smithi142 conmon[20125]: 24T22:02:39.368503+0000 mon.a (mon.0) 1626 : 2021-03-24T22:02:40.459 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:40 smithi142 conmon[20125]: cluster [DBG] osdmap e589: 8 total, 8 up, 8 in 2021-03-24T22:02:40.459 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:40 smithi142 conmon[20125]: cluster 2021-03-24T22:02:40.460 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:40 smithi142 conmon[20125]: 2021-03-24T22:02:40.309869+0000 mon.a ( 2021-03-24T22:02:40.460 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:40 smithi142 conmon[20125]: mon.0) 1627 : cluster [DBG] osdmap e590: 8 total, 8 up, 8 in 2021-03-24T22:02:40.483 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:40.540 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:40.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:40.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:02:40.622 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:40 smithi125 conmon[24927]: cluster 2021-03-24T22:02:40.622 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:40 smithi125 conmon[24927]: 2021-03-24T22:02 2021-03-24T22:02:40.623 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:40 smithi125 conmon[24927]: :38.909698+0000 mgr.y (mgr.14144) 566 : cluster [DBG] pgmap v1045: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 499 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 301 KiB/s wr, 1 op/s; 72/1656 objects misplaced (4.348%) 2021-03-24T22:02:40.623 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:40 smithi125 conmon[24927]: audit 2021-03-24T22:02:39. 2021-03-24T22:02:40.624 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:40 smithi125 conmon[24927]: 368404+0000 mon.a (mon.0) 1625 : audit [INF] 2021-03-24T22:02:40.624 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:40 smithi125 conmon[24927]: from='client.? v1:172.21.15.125:0/1045809968' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2021-03-24T22:02:40.624 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:40 smithi125 conmon[24927]: cluster 2021-03-24T22:02:39. 2021-03-24T22:02:40.625 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:40 smithi125 conmon[24927]: 368503+0000 mon.a (mon.0) 1626 : cluster [DBG] 2021-03-24T22:02:40.625 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:40 smithi125 conmon[24927]: osdmap e589: 8 total, 8 up, 8 in 2021-03-24T22:02:40.626 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:40 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:02:40.626 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:40 smithi125 conmon[24927]: 24T22:02:40.309869+0000 mon.a (mon.0) 1627 : cluster [DBG] osdmap e590: 8 total, 8 up, 8 in 2021-03-24T22:02:40.646 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:40.705 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:40.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:40.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:02:40.811 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:40.852 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd reweight 1 1 2021-03-24T22:02:40.870 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:40.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:40.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:02:40.977 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:41.029 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:41.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:41.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:02:41.135 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:41.181 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:41.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:41.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:02:41.288 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:41.323 INFO:tasks.rados.rados.0.smithi125.stdout:2202: done (0 left) 2021-03-24T22:02:41.323 INFO:tasks.rados.rados.0.smithi125.stdout:2203: read oid 10 snap 240 2021-03-24T22:02:41.323 INFO:tasks.rados.rados.0.smithi125.stdout:2203: expect (ObjNum 642 snap 196 seq_num 642) 2021-03-24T22:02:41.323 INFO:tasks.rados.rados.0.smithi125.stdout:2204: copy_from oid 23 from oid 38 current snap is 251 2021-03-24T22:02:41.324 INFO:tasks.rados.rados.0.smithi125.stdout:2205: read oid 50 snap -1 2021-03-24T22:02:41.324 INFO:tasks.rados.rados.0.smithi125.stdout:2205: expect (ObjNum 757 snap 244 seq_num 757) 2021-03-24T22:02:41.324 INFO:tasks.rados.rados.0.smithi125.stdout:2206: read oid 28 snap -1 2021-03-24T22:02:41.324 INFO:tasks.rados.rados.0.smithi125.stdout:2206: expect (ObjNum 610 snap 185 seq_num 610) 2021-03-24T22:02:41.325 INFO:tasks.rados.rados.0.smithi125.stdout:2207: copy_from oid 13 from oid 36 current snap is 251 2021-03-24T22:02:41.325 INFO:tasks.rados.rados.0.smithi125.stdout:2208: write (excl) oid 29 current snap is 251 2021-03-24T22:02:41.325 INFO:tasks.rados.rados.0.smithi125.stdout:2208: seq_num 766 ranges {474162=476034,1358148=481931,2352483=465303} 2021-03-24T22:02:41.328 INFO:tasks.rados.rados.0.smithi125.stdout:2208: writing smithi12556470-29 from 474162 to 950196 tid 1 2021-03-24T22:02:41.334 INFO:tasks.rados.rados.0.smithi125.stdout:2208: writing smithi12556470-29 from 1358148 to 1840079 tid 2 2021-03-24T22:02:41.347 INFO:tasks.rados.rados.0.smithi125.stdout:2208: writing smithi12556470-29 from 2352483 to 2817786 tid 3 2021-03-24T22:02:41.348 INFO:tasks.rados.rados.0.smithi125.stdout:2209: snap_remove snap 214 2021-03-24T22:02:41.356 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:41.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:41.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:02:41.461 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:41.515 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:41.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:41.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:02:41.622 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:41.682 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:41.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:41.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:02:41.792 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:41.846 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:41.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:41.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:02:41.907 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2021-03-24T22:02:41.908 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2021-03-24T22:02:41.908 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2021-03-24T22:02:41.908 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2021-03-24T22:02:41.908 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.909 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.24388.0:166 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected e107)", 2021-03-24T22:02:41.909 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:54:37.427176+0000", 2021-03-24T22:02:41.909 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 484.47584557800002, 2021-03-24T22:02:41.909 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.699168805, 2021-03-24T22:02:41.909 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:02:41.910 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:02:41.910 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:02:41.910 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.24388", 2021-03-24T22:02:41.910 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.142:0/626803041", 2021-03-24T22:02:41.911 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 166 2021-03-24T22:02:41.911 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.911 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:02:41.911 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.911 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:02:41.912 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:37.427176+0000", 2021-03-24T22:02:41.912 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:02:41.912 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.912 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.913 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:02:41.913 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:37.427176+0000", 2021-03-24T22:02:41.913 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999957 2021-03-24T22:02:41.913 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.913 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.914 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:02:41.914 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:37.427172+0000", 2021-03-24T22:02:41.914 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8757e-05 2021-03-24T22:02:41.914 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.915 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.915 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:02:41.915 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:37.427190+0000", 2021-03-24T22:02:41.915 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3581999999999998e-05 2021-03-24T22:02:41.916 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.916 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.916 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:02:41.916 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:37.427224+0000", 2021-03-24T22:02:41.917 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5787e-05 2021-03-24T22:02:41.917 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.917 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.917 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:02:41.917 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:37.427240+0000", 2021-03-24T22:02:41.918 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00020074000000000001 2021-03-24T22:02:41.918 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.918 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.918 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:41.919 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:37.427440+0000", 2021-03-24T22:02:41.919 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8619000000000002e-05 2021-03-24T22:02:41.919 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.919 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.919 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for peered", 2021-03-24T22:02:41.920 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:37.427469+0000", 2021-03-24T22:02:41.920 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.69859708399999998 2021-03-24T22:02:41.920 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.920 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.921 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:41.921 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:38.126066+0000", 2021-03-24T22:02:41.921 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133013 2021-03-24T22:02:41.921 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.921 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.922 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:02:41.922 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:38.126199+0000", 2021-03-24T22:02:41.922 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014562000000000001 2021-03-24T22:02:41.922 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.922 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.923 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:02:41.923 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:54:38.126345+0000", 2021-03-24T22:02:41.923 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.699168805 2021-03-24T22:02:41.923 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:41.924 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:02:41.924 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:41.924 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.924 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.924 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:2099 7.17 7:ec6a359b:::smithi12556470-33:head [copy-from ver 0] snapc 95=[95,93,92,8d,8b,82] ondisk+write+known_if_redirected e255)", 2021-03-24T22:02:41.925 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:57:04.197620+0000", 2021-03-24T22:02:41.925 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 337.70540141599997, 2021-03-24T22:02:41.925 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.95919054100000001, 2021-03-24T22:02:41.925 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:02:41.926 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:02:41.926 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:02:41.926 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:02:41.926 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:02:41.926 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 2099 2021-03-24T22:02:41.927 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.927 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:02:41.927 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.927 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:02:41.927 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.197620+0000", 2021-03-24T22:02:41.928 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:02:41.928 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.928 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.928 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:02:41.929 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.197620+0000", 2021-03-24T22:02:41.929 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999962 2021-03-24T22:02:41.929 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.929 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.930 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:02:41.930 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.197616+0000", 2021-03-24T22:02:41.930 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7652999999999999e-05 2021-03-24T22:02:41.930 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.930 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.931 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:02:41.931 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.197634+0000", 2021-03-24T22:02:41.931 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6417000000000001e-05 2021-03-24T22:02:41.931 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.932 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.932 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:02:41.932 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.197660+0000", 2021-03-24T22:02:41.932 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9066e-05 2021-03-24T22:02:41.932 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.933 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.933 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:02:41.933 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.197680+0000", 2021-03-24T22:02:41.933 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00017438800000000001 2021-03-24T22:02:41.934 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.934 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.934 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:41.934 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.197854+0000", 2021-03-24T22:02:41.934 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3726e-05 2021-03-24T22:02:41.935 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.935 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.935 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op must wait for map", 2021-03-24T22:02:41.935 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.197878+0000", 2021-03-24T22:02:41.935 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0099756870000000004 2021-03-24T22:02:41.936 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.936 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.936 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:41.936 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.207853+0000", 2021-03-24T22:02:41.937 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3523000000000007e-05 2021-03-24T22:02:41.937 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.937 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.937 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:02:41.937 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.207937+0000", 2021-03-24T22:02:41.938 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.948873784 2021-03-24T22:02:41.938 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.938 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.939 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:02:41.939 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:05.156811+0000", 2021-03-24T22:02:41.939 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.95919054100000001 2021-03-24T22:02:41.939 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:41.939 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:02:41.940 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:41.940 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.940 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.940 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:2100 7.17 7:ec6a359b:::smithi12556470-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e255)", 2021-03-24T22:02:41.941 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:57:04.198076+0000", 2021-03-24T22:02:41.941 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 337.70494504200002, 2021-03-24T22:02:41.941 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.95908168400000005, 2021-03-24T22:02:41.941 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:02:41.941 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:02:41.942 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:02:41.942 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:02:41.942 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:02:41.942 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 2100 2021-03-24T22:02:41.942 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.943 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:02:41.943 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.943 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:02:41.943 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.198076+0000", 2021-03-24T22:02:41.943 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:02:41.944 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.944 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.944 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:02:41.944 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.198076+0000", 2021-03-24T22:02:41.945 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999971 2021-03-24T22:02:41.945 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.945 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.945 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:02:41.945 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.198074+0000", 2021-03-24T22:02:41.946 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3433e-05 2021-03-24T22:02:41.946 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.946 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.946 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:02:41.947 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.198087+0000", 2021-03-24T22:02:41.947 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.262e-05 2021-03-24T22:02:41.947 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.947 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.947 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:02:41.948 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.198110+0000", 2021-03-24T22:02:41.948 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6405999999999998e-05 2021-03-24T22:02:41.948 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.948 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.948 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:02:41.949 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.198126+0000", 2021-03-24T22:02:41.949 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000109786 2021-03-24T22:02:41.949 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.950 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.950 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:41.950 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.198236+0000", 2021-03-24T22:02:41.950 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0522e-05 2021-03-24T22:02:41.950 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.951 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.951 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting_for_map not empty", 2021-03-24T22:02:41.951 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.198246+0000", 2021-03-24T22:02:41.951 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0098265899999999996 2021-03-24T22:02:41.952 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.952 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.952 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:41.952 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.208073+0000", 2021-03-24T22:02:41.952 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7707999999999999e-05 2021-03-24T22:02:41.953 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.953 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.953 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for blocked object", 2021-03-24T22:02:41.953 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.208111+0000", 2021-03-24T22:02:41.953 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.94884332199999999 2021-03-24T22:02:41.954 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.954 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.954 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:41.954 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:05.156954+0000", 2021-03-24T22:02:41.955 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.8762000000000001e-05 2021-03-24T22:02:41.955 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.955 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.955 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:02:41.955 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:05.157053+0000", 2021-03-24T22:02:41.956 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000105321 2021-03-24T22:02:41.956 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.956 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.956 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:02:41.956 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:05.157158+0000", 2021-03-24T22:02:41.957 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.95908168400000005 2021-03-24T22:02:41.957 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:41.957 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:02:41.957 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:41.958 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.958 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.958 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:3112 7.17 7:ec6a359b:::smithi12556470-33:df [sparse-read 0~3008486,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e340)", 2021-03-24T22:02:41.958 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:58:30.211016+0000", 2021-03-24T22:02:41.958 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 251.692005615, 2021-03-24T22:02:41.959 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.99935758200000002, 2021-03-24T22:02:41.959 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:02:41.959 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:02:41.959 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:02:41.960 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:02:41.960 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:02:41.960 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 3112 2021-03-24T22:02:41.960 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.960 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:02:41.961 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.961 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:02:41.961 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211016+0000", 2021-03-24T22:02:41.961 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:02:41.961 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.962 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.962 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:02:41.962 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211016+0000", 2021-03-24T22:02:41.962 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2021-03-24T22:02:41.963 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.963 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.963 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:02:41.963 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211013+0000", 2021-03-24T22:02:41.963 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0642000000000001e-05 2021-03-24T22:02:41.964 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.964 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.964 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:02:41.964 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211024+0000", 2021-03-24T22:02:41.964 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2675e-05 2021-03-24T22:02:41.965 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.965 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.965 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:02:41.965 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211037+0000", 2021-03-24T22:02:41.965 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6210000000000004e-06 2021-03-24T22:02:41.966 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.966 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.966 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:02:41.966 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211044+0000", 2021-03-24T22:02:41.967 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.4183999999999994e-05 2021-03-24T22:02:41.967 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.967 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.967 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:41.967 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211109+0000", 2021-03-24T22:02:41.968 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1910000000000006e-06 2021-03-24T22:02:41.968 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.968 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.968 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for peered", 2021-03-24T22:02:41.969 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211118+0000", 2021-03-24T22:02:41.969 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.99877172599999997 2021-03-24T22:02:41.969 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.969 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.969 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:41.970 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:31.209889+0000", 2021-03-24T22:02:41.970 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000169588 2021-03-24T22:02:41.970 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.970 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.970 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:02:41.971 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:31.210059+0000", 2021-03-24T22:02:41.971 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00031441100000000001 2021-03-24T22:02:41.971 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.971 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.972 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:02:41.972 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:31.210373+0000", 2021-03-24T22:02:41.972 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.99935758200000002 2021-03-24T22:02:41.972 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:41.972 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:02:41.973 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:41.973 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.973 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.973 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:3113 7.17 7:ec6a359b:::smithi12556470-33:df [sparse-read 0~3008486] snapc 0=[] ondisk+read+known_if_redirected e340)", 2021-03-24T22:02:41.974 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:58:30.211055+0000", 2021-03-24T22:02:41.974 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 251.691966632, 2021-03-24T22:02:41.974 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.99976562300000005, 2021-03-24T22:02:41.974 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:02:41.974 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:02:41.975 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:02:41.975 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:02:41.975 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:02:41.975 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 3113 2021-03-24T22:02:41.976 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.976 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:02:41.976 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.976 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:02:41.976 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211055+0000", 2021-03-24T22:02:41.977 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:02:41.977 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.977 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.977 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:02:41.978 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211055+0000", 2021-03-24T22:02:41.978 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2021-03-24T22:02:41.978 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.978 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.978 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:02:41.979 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211054+0000", 2021-03-24T22:02:41.979 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8150000000000002e-06 2021-03-24T22:02:41.979 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.979 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.979 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:02:41.980 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211057+0000", 2021-03-24T22:02:41.980 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.7319999999999993e-06 2021-03-24T22:02:41.980 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.980 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.980 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:02:41.981 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211067+0000", 2021-03-24T22:02:41.981 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1489999999999998e-06 2021-03-24T22:02:41.981 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.981 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.982 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:02:41.982 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211070+0000", 2021-03-24T22:02:41.982 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.1780000000000006e-05 2021-03-24T22:02:41.982 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.982 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.983 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:41.983 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211152+0000", 2021-03-24T22:02:41.983 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.8709999999999999e-06 2021-03-24T22:02:41.983 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.983 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.984 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for peered", 2021-03-24T22:02:41.984 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211159+0000", 2021-03-24T22:02:41.984 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.999350081 2021-03-24T22:02:41.984 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.985 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.985 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:41.985 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:31.210509+0000", 2021-03-24T22:02:41.985 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.0376000000000003e-05 2021-03-24T22:02:41.985 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.986 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.986 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:02:41.986 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:31.210599+0000", 2021-03-24T22:02:41.986 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00022137900000000001 2021-03-24T22:02:41.986 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.987 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.987 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:02:41.987 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:31.210820+0000", 2021-03-24T22:02:41.987 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.99976562300000005 2021-03-24T22:02:41.988 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:41.988 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:02:41.988 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:41.988 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.988 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.989 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:3114 7.17 7:ec6a359b:::smithi12556470-33:df [sparse-read 0~3008486] snapc 0=[] ondisk+read+known_if_redirected e340)", 2021-03-24T22:02:41.989 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:58:30.211092+0000", 2021-03-24T22:02:41.989 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 251.69192972499999, 2021-03-24T22:02:41.989 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.000019572, 2021-03-24T22:02:41.990 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:02:41.990 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:02:41.990 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:02:41.990 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:02:41.990 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:02:41.991 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 3114 2021-03-24T22:02:41.991 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.991 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:02:41.991 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.992 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:02:41.992 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211092+0000", 2021-03-24T22:02:41.992 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:02:41.992 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.992 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.993 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:02:41.993 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211092+0000", 2021-03-24T22:02:41.993 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2021-03-24T22:02:41.993 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.994 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.994 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:02:41.994 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211091+0000", 2021-03-24T22:02:41.994 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2789999999999999e-06 2021-03-24T22:02:41.994 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.995 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.995 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:02:41.995 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211095+0000", 2021-03-24T22:02:41.995 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3040000000000001e-06 2021-03-24T22:02:41.996 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.996 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.996 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:02:41.996 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211101+0000", 2021-03-24T22:02:41.997 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4950000000000002e-06 2021-03-24T22:02:41.997 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.997 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.997 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:02:41.997 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211105+0000", 2021-03-24T22:02:41.998 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.8471000000000002e-05 2021-03-24T22:02:41.998 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.998 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.998 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:41.998 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211174+0000", 2021-03-24T22:02:41.999 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.9419999999999997e-06 2021-03-24T22:02:41.999 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:41.999 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:41.999 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for peered", 2021-03-24T22:02:42.000 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211178+0000", 2021-03-24T22:02:42.000 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.99967128199999999 2021-03-24T22:02:42.000 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.000 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.000 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:42.001 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:31.210849+0000", 2021-03-24T22:02:42.001 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011547 2021-03-24T22:02:42.001 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.001 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.002 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:02:42.002 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:31.210965+0000", 2021-03-24T22:02:42.002 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014683099999999999 2021-03-24T22:02:42.002 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.002 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.003 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:02:42.003 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:31.211111+0000", 2021-03-24T22:02:42.003 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.000019572 2021-03-24T22:02:42.003 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:42.003 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:02:42.004 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:42.004 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.004 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.004 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.24388.0:544 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e490)", 2021-03-24T22:02:42.005 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:01:00.454826+0000", 2021-03-24T22:02:42.005 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 101.448195235, 2021-03-24T22:02:42.005 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.86119727800000001, 2021-03-24T22:02:42.005 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:02:42.005 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:02:42.006 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:02:42.006 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.24388", 2021-03-24T22:02:42.006 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.142:0/626803041", 2021-03-24T22:02:42.006 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 544 2021-03-24T22:02:42.007 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.007 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:02:42.007 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.007 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:02:42.007 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:00.454826+0000", 2021-03-24T22:02:42.008 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:02:42.008 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.008 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.008 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:02:42.009 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:00.454826+0000", 2021-03-24T22:02:42.009 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999957 2021-03-24T22:02:42.009 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.009 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.009 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:02:42.010 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:00.454822+0000", 2021-03-24T22:02:42.010 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8848999999999999e-05 2021-03-24T22:02:42.010 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.010 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.010 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:02:42.011 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:00.454841+0000", 2021-03-24T22:02:42.011 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8226000000000003e-05 2021-03-24T22:02:42.011 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.011 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.012 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:02:42.012 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:00.454879+0000", 2021-03-24T22:02:42.012 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5817000000000001e-05 2021-03-24T22:02:42.012 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.012 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.013 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:02:42.013 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:00.454895+0000", 2021-03-24T22:02:42.013 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015252899999999999 2021-03-24T22:02:42.013 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.013 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.014 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:42.014 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:00.455047+0000", 2021-03-24T22:02:42.014 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1634000000000002e-05 2021-03-24T22:02:42.014 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.015 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.015 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for peered", 2021-03-24T22:02:42.015 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:00.455079+0000", 2021-03-24T22:02:42.015 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.85495980199999999 2021-03-24T22:02:42.015 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.016 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.016 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:42.016 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:01.310039+0000", 2021-03-24T22:02:42.016 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013964399999999999 2021-03-24T22:02:42.016 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.017 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.017 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for missing object", 2021-03-24T22:02:42.017 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:01.310178+0000", 2021-03-24T22:02:42.017 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.005680488 2021-03-24T22:02:42.018 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.018 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.018 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:42.018 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:01.315859+0000", 2021-03-24T22:02:42.018 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3120000000000006e-05 2021-03-24T22:02:42.019 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.019 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.019 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:02:42.019 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:01.315922+0000", 2021-03-24T22:02:42.019 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000101475 2021-03-24T22:02:42.020 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.020 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.020 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:02:42.020 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:01.316024+0000", 2021-03-24T22:02:42.021 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.86119727800000001 2021-03-24T22:02:42.021 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:42.021 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:02:42.021 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:42.021 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.022 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.022 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(7.19 508/491 [PushOp(7:9b48f357:::smithi12556470-17:114, version: 455'474, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:9b48f357:::smithi12556470-17:114@455'474, size: 3142975, copy_subset: [0~3142975], clone_subset: {}, snapset: 166=[]:{114=[104,fa],11b=[11b],127=[127],165=[165,160,15e,159,157,153],166=[166]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-03-24T22:02:42.022 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:01:19.123177+0000", 2021-03-24T22:02:42.022 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 82.779844053000005, 2021-03-24T22:02:42.022 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.088544329000000005, 2021-03-24T22:02:42.023 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:02:42.023 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:02:42.023 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:02:42.023 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.024 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:02:42.024 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.123177+0000", 2021-03-24T22:02:42.024 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:02:42.024 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.024 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.025 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:02:42.025 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.123177+0000", 2021-03-24T22:02:42.025 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999733 2021-03-24T22:02:42.025 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.025 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.026 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:02:42.026 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.123150+0000", 2021-03-24T22:02:42.026 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00178196 2021-03-24T22:02:42.026 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.027 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.027 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:02:42.027 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.124932+0000", 2021-03-24T22:02:42.027 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4739e-05 2021-03-24T22:02:42.027 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.028 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.028 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:02:42.028 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.124977+0000", 2021-03-24T22:02:42.028 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3717e-05 2021-03-24T22:02:42.028 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.029 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.029 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:02:42.029 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.125001+0000", 2021-03-24T22:02:42.029 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.086317912999999996 2021-03-24T22:02:42.030 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.030 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.030 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:42.030 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.211319+0000", 2021-03-24T22:02:42.030 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8689999999999993e-06 2021-03-24T22:02:42.031 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.031 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.031 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:02:42.031 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.211328+0000", 2021-03-24T22:02:42.032 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00039407100000000002 2021-03-24T22:02:42.032 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.032 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.032 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:02:42.032 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.211722+0000", 2021-03-24T22:02:42.033 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.088544329000000005 2021-03-24T22:02:42.033 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:42.033 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:02:42.033 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:42.034 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.034 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.034 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(7.19 508/491 [PushOp(7:9b48f357:::smithi12556470-17:114, version: 455'474, data_included: [2097152~283533,2971713~171262], data_size: 454795, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:9b48f357:::smithi12556470-17:114@455'474, size: 3142975, copy_subset: [0~3142975], clone_subset: {}, snapset: 166=[]:{114=[104,fa],11b=[11b],127=[127],165=[165,160,15e,159,157,153],166=[166]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3142975, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-03-24T22:02:42.034 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:01:19.213332+0000", 2021-03-24T22:02:42.034 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 82.689689591000004, 2021-03-24T22:02:42.035 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.086796499999999999, 2021-03-24T22:02:42.035 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:02:42.035 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:02:42.035 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:02:42.035 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.036 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:02:42.036 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.213332+0000", 2021-03-24T22:02:42.036 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:02:42.036 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.037 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.037 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:02:42.037 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.213332+0000", 2021-03-24T22:02:42.037 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2021-03-24T22:02:42.037 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.038 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.038 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:02:42.038 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.213330+0000", 2021-03-24T22:02:42.038 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00062080700000000002 2021-03-24T22:02:42.039 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.039 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.039 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:02:42.039 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.213950+0000", 2021-03-24T22:02:42.039 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4645999999999999e-05 2021-03-24T22:02:42.040 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.040 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.040 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:02:42.040 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.213995+0000", 2021-03-24T22:02:42.041 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7702e-05 2021-03-24T22:02:42.041 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.041 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.041 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:02:42.041 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.214023+0000", 2021-03-24T22:02:42.042 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.085201523000000001 2021-03-24T22:02:42.042 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.042 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.042 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:42.042 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.299224+0000", 2021-03-24T22:02:42.043 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4891e-05 2021-03-24T22:02:42.043 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.043 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.043 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:02:42.044 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.299239+0000", 2021-03-24T22:02:42.044 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00088927799999999997 2021-03-24T22:02:42.044 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.044 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.044 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:02:42.045 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.300128+0000", 2021-03-24T22:02:42.045 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.086796499999999999 2021-03-24T22:02:42.045 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:42.045 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:02:42.045 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:42.046 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.046 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.046 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(7.19 509/491 [PushOp(7:9b48f357:::smithi12556470-17:11b, version: 455'475, data_included: [521149~630541,1701900~418035], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:9b48f357:::smithi12556470-17:11b@455'475, size: 2521149, copy_subset: [521149~630541,1701900~628502], clone_subset: {7:9b48f357:::smithi12556470-17:114=[0~521149,1151690~550210,2330402~190747]}, snapset: 166=[]:{114=[104,fa],11b=[11b],127=[127],165=[165,160,15e,159,157,153],166=[166]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2119935, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2021-03-24T22:02:42.046 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:01:19.472229+0000", 2021-03-24T22:02:42.047 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 82.430792287000003, 2021-03-24T22:02:42.047 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.089665769000000006, 2021-03-24T22:02:42.047 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:02:42.047 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:02:42.047 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:02:42.048 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.048 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:02:42.048 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.472229+0000", 2021-03-24T22:02:42.048 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:02:42.048 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.049 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.049 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:02:42.049 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.472229+0000", 2021-03-24T22:02:42.049 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2021-03-24T22:02:42.050 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.050 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.050 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:02:42.050 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.472228+0000", 2021-03-24T22:02:42.050 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00095283100000000001 2021-03-24T22:02:42.051 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.051 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.051 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:02:42.051 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.473181+0000", 2021-03-24T22:02:42.052 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4737999999999999e-05 2021-03-24T22:02:42.052 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.052 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.052 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:02:42.052 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.473205+0000", 2021-03-24T22:02:42.053 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3817e-05 2021-03-24T22:02:42.053 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.053 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.053 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:02:42.053 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.473219+0000", 2021-03-24T22:02:42.054 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.088154109999999994 2021-03-24T22:02:42.054 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.054 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.054 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:42.055 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.561373+0000", 2021-03-24T22:02:42.055 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.578e-06 2021-03-24T22:02:42.055 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.055 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.055 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:02:42.056 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.561378+0000", 2021-03-24T22:02:42.056 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00051709599999999998 2021-03-24T22:02:42.056 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.056 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.057 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:02:42.057 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.561895+0000", 2021-03-24T22:02:42.057 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.089665769000000006 2021-03-24T22:02:42.057 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:42.057 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:02:42.058 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:42.058 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.058 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.058 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(7.19 509/491 [PushOp(7:9b48f357:::smithi12556470-17:11b, version: 455'475, data_included: [2119935~210467], data_size: 210467, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:9b48f357:::smithi12556470-17:11b@455'475, size: 2521149, copy_subset: [521149~630541,1701900~628502], clone_subset: {7:9b48f357:::smithi12556470-17:114=[0~521149,1151690~550210,2330402~190747]}, snapset: 166=[]:{114=[104,fa],11b=[11b],127=[127],165=[165,160,15e,159,157,153],166=[166]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2330402, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2119935, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-03-24T22:02:42.059 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:01:19.563380+0000", 2021-03-24T22:02:42.059 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 82.339641766, 2021-03-24T22:02:42.059 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.088066119999999998, 2021-03-24T22:02:42.059 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:02:42.059 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:02:42.060 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:02:42.060 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.060 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:02:42.060 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.563380+0000", 2021-03-24T22:02:42.060 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:02:42.061 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.061 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.061 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:02:42.061 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.563380+0000", 2021-03-24T22:02:42.062 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2021-03-24T22:02:42.062 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.062 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.062 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:02:42.062 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.563379+0000", 2021-03-24T22:02:42.063 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00020590499999999999 2021-03-24T22:02:42.063 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.063 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.063 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:02:42.063 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.563585+0000", 2021-03-24T22:02:42.064 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1863999999999999e-05 2021-03-24T22:02:42.064 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.064 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.064 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:02:42.065 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.563607+0000", 2021-03-24T22:02:42.065 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8245000000000001e-05 2021-03-24T22:02:42.065 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.065 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.065 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:02:42.066 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.563635+0000", 2021-03-24T22:02:42.066 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.084418923000000007 2021-03-24T22:02:42.066 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.066 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.067 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:42.067 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.648054+0000", 2021-03-24T22:02:42.067 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1840000000000001e-06 2021-03-24T22:02:42.067 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.067 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.068 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:02:42.068 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.648058+0000", 2021-03-24T22:02:42.068 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0033879330000000001 2021-03-24T22:02:42.068 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.068 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.069 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:02:42.069 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.651446+0000", 2021-03-24T22:02:42.069 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.088066119999999998 2021-03-24T22:02:42.069 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:42.070 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:02:42.070 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:42.070 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.070 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.070 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(7.19 509/491 [PushOp(7:9b48f357:::smithi12556470-17:127, version: 455'477, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:9b48f357:::smithi12556470-17:127@455'477, size: 2874096, copy_subset: [0~2874096], clone_subset: {7:9b48f357:::smithi12556470-17:11b=[]}, snapset: 166=[]:{114=[104,fa],11b=[11b],127=[127],165=[165,160,15e,159,157,153],166=[166]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-03-24T22:02:42.071 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:01:19.826600+0000", 2021-03-24T22:02:42.071 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 82.076421741000004, 2021-03-24T22:02:42.071 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.088799362000000007, 2021-03-24T22:02:42.071 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:02:42.072 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:02:42.072 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:02:42.072 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.072 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:02:42.072 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.826600+0000", 2021-03-24T22:02:42.073 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:02:42.073 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.073 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.073 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:02:42.074 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.826600+0000", 2021-03-24T22:02:42.074 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2021-03-24T22:02:42.074 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.074 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.074 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:02:42.075 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.826599+0000", 2021-03-24T22:02:42.075 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00097982800000000008 2021-03-24T22:02:42.075 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.075 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.075 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:02:42.076 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.827578+0000", 2021-03-24T22:02:42.076 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.084e-05 2021-03-24T22:02:42.076 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.076 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.077 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:02:42.077 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.827599+0000", 2021-03-24T22:02:42.077 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2388000000000001e-05 2021-03-24T22:02:42.077 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.077 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.078 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:02:42.078 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.827612+0000", 2021-03-24T22:02:42.078 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.087205894000000006 2021-03-24T22:02:42.078 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.079 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.079 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:42.079 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.914818+0000", 2021-03-24T22:02:42.079 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0579999999999997e-06 2021-03-24T22:02:42.079 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.080 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.080 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:02:42.080 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.914823+0000", 2021-03-24T22:02:42.080 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00057645600000000004 2021-03-24T22:02:42.080 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.081 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.081 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:02:42.081 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.915399+0000", 2021-03-24T22:02:42.081 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.088799362000000007 2021-03-24T22:02:42.082 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:42.082 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:02:42.082 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:42.082 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.082 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.083 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(7.19 509/491 [PushOp(7:9b48f357:::smithi12556470-17:127, version: 455'477, data_included: [2097152~776944], data_size: 776944, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:9b48f357:::smithi12556470-17:127@455'477, size: 2874096, copy_subset: [0~2874096], clone_subset: {7:9b48f357:::smithi12556470-17:11b=[]}, snapset: 166=[]:{114=[104,fa],11b=[11b],127=[127],165=[165,160,15e,159,157,153],166=[166]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2874096, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-03-24T22:02:42.083 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:01:19.917170+0000", 2021-03-24T22:02:42.083 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 81.985851702000005, 2021-03-24T22:02:42.083 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.087589146000000007, 2021-03-24T22:02:42.083 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:02:42.084 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:02:42.084 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:02:42.084 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.084 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:02:42.085 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.917170+0000", 2021-03-24T22:02:42.085 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:02:42.085 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.085 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.085 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:02:42.086 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.917170+0000", 2021-03-24T22:02:42.086 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2021-03-24T22:02:42.086 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.086 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.086 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:02:42.087 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.917168+0000", 2021-03-24T22:02:42.087 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00075607099999999998 2021-03-24T22:02:42.087 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.087 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.088 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:02:42.088 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.917924+0000", 2021-03-24T22:02:42.088 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7515999999999999e-05 2021-03-24T22:02:42.088 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.088 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.089 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:02:42.089 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.917951+0000", 2021-03-24T22:02:42.089 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6047000000000001e-05 2021-03-24T22:02:42.089 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.089 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.090 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:02:42.090 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.917967+0000", 2021-03-24T22:02:42.090 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.086132426999999998 2021-03-24T22:02:42.090 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.090 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.091 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:42.091 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:20.004100+0000", 2021-03-24T22:02:42.091 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0030000000000002e-06 2021-03-24T22:02:42.091 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.092 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.092 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:02:42.092 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:20.004105+0000", 2021-03-24T22:02:42.092 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000654223 2021-03-24T22:02:42.092 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.093 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.093 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:02:42.093 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:20.004759+0000", 2021-03-24T22:02:42.093 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.087589146000000007 2021-03-24T22:02:42.093 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:42.094 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:02:42.094 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:42.094 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.094 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.095 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(7.19 510/491 [PushOp(7:9c4f4ea5:::smithi12556470-41:119, version: 455'479, data_included: [2097152~722659,3066106~325917], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:9c4f4ea5:::smithi12556470-41:119@455'479, size: 3731220, copy_subset: [0~3731220], clone_subset: {}, snapset: 149=[]:{119=[104,fa],149=[139,136,129,127,11b]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3392023, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-03-24T22:02:42.095 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:01:20.359492+0000", 2021-03-24T22:02:42.095 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 81.543529152999994, 2021-03-24T22:02:42.095 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.087993191999999998, 2021-03-24T22:02:42.095 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:02:42.096 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:02:42.096 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:02:42.096 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.096 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:02:42.097 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:20.359492+0000", 2021-03-24T22:02:42.097 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:02:42.097 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.097 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.097 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:02:42.098 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:20.359492+0000", 2021-03-24T22:02:42.098 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999971 2021-03-24T22:02:42.098 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.098 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.098 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:02:42.099 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:20.359489+0000", 2021-03-24T22:02:42.099 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001299632 2021-03-24T22:02:42.099 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.099 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.100 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:02:42.100 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:20.360789+0000", 2021-03-24T22:02:42.100 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.3445000000000002e-05 2021-03-24T22:02:42.100 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.100 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.101 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:02:42.101 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:20.360832+0000", 2021-03-24T22:02:42.101 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2379999999999999e-05 2021-03-24T22:02:42.101 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.102 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.102 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:02:42.102 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:20.360855+0000", 2021-03-24T22:02:42.102 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.086302106000000003 2021-03-24T22:02:42.102 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.103 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.103 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:42.103 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:20.447157+0000", 2021-03-24T22:02:42.103 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1219999999999997e-06 2021-03-24T22:02:42.103 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.104 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.104 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:02:42.104 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:20.447161+0000", 2021-03-24T22:02:42.104 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00032459299999999997 2021-03-24T22:02:42.104 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.105 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.105 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:02:42.105 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:20.447486+0000", 2021-03-24T22:02:42.105 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.087993191999999998 2021-03-24T22:02:42.105 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:42.106 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:02:42.106 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:42.106 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.106 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.107 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(7.19 510/491 [PushOp(7:9c4f4ea5:::smithi12556470-41:119, version: 455'479, data_included: [3392023~94503], data_size: 94503, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:9c4f4ea5:::smithi12556470-41:119@455'479, size: 3731220, copy_subset: [0~3731220], clone_subset: {}, snapset: 149=[]:{119=[104,fa],149=[139,136,129,127,11b]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3731220, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3392023, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-03-24T22:02:42.107 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:01:20.448959+0000", 2021-03-24T22:02:42.107 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 81.454062285999996, 2021-03-24T22:02:42.107 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.085756993000000004, 2021-03-24T22:02:42.107 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:02:42.108 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:02:42.108 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:02:42.108 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.108 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:02:42.108 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:20.448959+0000", 2021-03-24T22:02:42.109 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:02:42.109 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.109 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.109 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:02:42.110 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:20.448959+0000", 2021-03-24T22:02:42.110 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999971 2021-03-24T22:02:42.110 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.110 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.110 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:02:42.111 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:20.448956+0000", 2021-03-24T22:02:42.111 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000156064 2021-03-24T22:02:42.111 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.111 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.111 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:02:42.112 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:20.449112+0000", 2021-03-24T22:02:42.112 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.3847000000000002e-05 2021-03-24T22:02:42.112 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.112 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.112 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:02:42.113 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:20.449156+0000", 2021-03-24T22:02:42.113 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3349000000000002e-05 2021-03-24T22:02:42.113 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.113 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.113 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:02:42.114 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:20.449190+0000", 2021-03-24T22:02:42.114 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.084277240000000003 2021-03-24T22:02:42.114 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.114 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.115 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:42.115 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:20.533467+0000", 2021-03-24T22:02:42.115 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1710000000000004e-06 2021-03-24T22:02:42.115 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.115 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.116 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:02:42.116 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:20.533471+0000", 2021-03-24T22:02:42.116 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001245269 2021-03-24T22:02:42.116 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.117 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.117 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:02:42.117 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:20.534716+0000", 2021-03-24T22:02:42.117 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.085756993000000004 2021-03-24T22:02:42.117 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:42.118 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:02:42.118 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:42.118 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.118 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.118 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(7.19 510/491 [PushOp(7:9b48f357:::smithi12556470-17:166, version: 469'504, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:9b48f357:::smithi12556470-17:166@469'504, size: 2353079, copy_subset: [0~2353079], clone_subset: {7:9b48f357:::smithi12556470-17:127=[]}, snapset: 166=[]:{114=[104,fa],11b=[11b],127=[127],165=[165,160,15e,159,157,153],166=[166]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-03-24T22:02:42.119 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:01:20.884182+0000", 2021-03-24T22:02:42.119 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 81.018839331999999, 2021-03-24T22:02:42.119 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.088397383999999996, 2021-03-24T22:02:42.119 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:02:42.120 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:02:42.120 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:02:42.120 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.120 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:02:42.120 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:20.884182+0000", 2021-03-24T22:02:42.121 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:02:42.121 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.121 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.121 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:02:42.121 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:20.884182+0000", 2021-03-24T22:02:42.122 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999971 2021-03-24T22:02:42.122 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.122 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.122 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:02:42.122 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:20.884179+0000", 2021-03-24T22:02:42.123 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001025679 2021-03-24T22:02:42.123 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.123 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.123 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:02:42.124 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:20.885205+0000", 2021-03-24T22:02:42.124 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.7153000000000001e-05 2021-03-24T22:02:42.124 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.124 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.124 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:02:42.125 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:20.885252+0000", 2021-03-24T22:02:42.125 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4223000000000002e-05 2021-03-24T22:02:42.125 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.125 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.125 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:02:42.126 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:20.885276+0000", 2021-03-24T22:02:42.126 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.086821926999999993 2021-03-24T22:02:42.126 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.126 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.127 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:42.127 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:20.972098+0000", 2021-03-24T22:02:42.127 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0560000000000004e-06 2021-03-24T22:02:42.127 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.127 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.128 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:02:42.128 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:20.972103+0000", 2021-03-24T22:02:42.128 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00047635599999999999 2021-03-24T22:02:42.128 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.128 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.129 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:02:42.129 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:20.972580+0000", 2021-03-24T22:02:42.129 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.088397383999999996 2021-03-24T22:02:42.129 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:42.130 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:02:42.130 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:42.130 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.130 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.130 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(7.19 511/491 [PushOp(7:9b48f357:::smithi12556470-17:165, version: 481'520, data_included: [2119071~248646,2770646~473204], data_size: 721850, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:9b48f357:::smithi12556470-17:165@481'520, size: 3243850, copy_subset: [0~3243850], clone_subset: {7:9b48f357:::smithi12556470-17:127=[],7:9b48f357:::smithi12556470-17:166=[]}, snapset: 166=[]:{114=[104,fa],11b=[11b],127=[127],165=[165,160,15e,159,157,153],166=[166]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3243850, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2119071, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-03-24T22:02:42.131 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:01:21.495555+0000", 2021-03-24T22:02:42.131 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 80.407466306000003, 2021-03-24T22:02:42.131 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.089930140000000006, 2021-03-24T22:02:42.131 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:02:42.132 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:02:42.132 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:02:42.132 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.132 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:02:42.132 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:21.495555+0000", 2021-03-24T22:02:42.133 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:02:42.133 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.133 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.133 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:02:42.133 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:21.495555+0000", 2021-03-24T22:02:42.134 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2021-03-24T22:02:42.134 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.134 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.134 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:02:42.135 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:21.495554+0000", 2021-03-24T22:02:42.135 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00060347000000000005 2021-03-24T22:02:42.135 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.135 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.135 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:02:42.136 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:21.496158+0000", 2021-03-24T22:02:42.136 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1359e-05 2021-03-24T22:02:42.136 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.136 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.136 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:02:42.137 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:21.496179+0000", 2021-03-24T22:02:42.137 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9062999999999999e-05 2021-03-24T22:02:42.137 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.137 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.138 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:02:42.138 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:21.496208+0000", 2021-03-24T22:02:42.138 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.085999172999999998 2021-03-24T22:02:42.138 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.138 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.139 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:42.139 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:21.582207+0000", 2021-03-24T22:02:42.139 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.7330000000000003e-06 2021-03-24T22:02:42.139 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.139 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.140 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:02:42.140 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:21.582212+0000", 2021-03-24T22:02:42.140 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0032732009999999999 2021-03-24T22:02:42.140 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.140 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.141 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:02:42.141 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:21.585485+0000", 2021-03-24T22:02:42.141 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.089930140000000006 2021-03-24T22:02:42.141 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:42.142 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:02:42.142 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:42.142 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.142 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.142 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(7.19 511/491 [PushOp(7:9c4f4ea5:::smithi12556470-41:149, version: 483'522, data_included: [2353281~218289], data_size: 218289, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:9c4f4ea5:::smithi12556470-41:149@483'522, size: 2571570, copy_subset: [571570~572641,1877346~694224], clone_subset: {7:9c4f4ea5:::smithi12556470-41:119=[0~571570,1144211~733135]}, snapset: 149=[]:{119=[104,fa],149=[139,136,129,127,11b]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2571570, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2353281, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-03-24T22:02:42.143 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:01:21.938977+0000", 2021-03-24T22:02:42.143 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 79.964044272999999, 2021-03-24T22:02:42.143 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.085955047000000007, 2021-03-24T22:02:42.143 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:02:42.144 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:02:42.144 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:02:42.144 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.144 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:02:42.144 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:21.938977+0000", 2021-03-24T22:02:42.145 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:02:42.145 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.145 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.145 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:02:42.145 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:21.938977+0000", 2021-03-24T22:02:42.146 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2021-03-24T22:02:42.146 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.146 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.146 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:02:42.147 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:21.938975+0000", 2021-03-24T22:02:42.147 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00022327999999999999 2021-03-24T22:02:42.147 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.147 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.147 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:02:42.148 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:21.939199+0000", 2021-03-24T22:02:42.148 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8547e-05 2021-03-24T22:02:42.148 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.148 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.148 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:02:42.149 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:21.939227+0000", 2021-03-24T22:02:42.149 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6735000000000001e-05 2021-03-24T22:02:42.149 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.149 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.150 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:02:42.150 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:21.939254+0000", 2021-03-24T22:02:42.150 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.084343505999999999 2021-03-24T22:02:42.150 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.150 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.151 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:42.151 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:22.023598+0000", 2021-03-24T22:02:42.151 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.053e-06 2021-03-24T22:02:42.151 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.152 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.152 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:02:42.152 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:22.023602+0000", 2021-03-24T22:02:42.152 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001330651 2021-03-24T22:02:42.152 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.153 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.153 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:02:42.153 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:22.024932+0000", 2021-03-24T22:02:42.153 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.085955047000000007 2021-03-24T22:02:42.153 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:42.154 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:02:42.154 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:42.154 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.154 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.155 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(7.19 512/491 [PushOp(7:992cb487:::smithi12556470-5:head, version: 489'536, data_included: [2896863~61301], data_size: 61301, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:992cb487:::smithi12556470-5:head@489'536, size: 2958164, copy_subset: [558164~552353,1803789~454563,2453954~504210], clone_subset: {7:992cb487:::smithi12556470-5:177=[0~558164,1110517~693272,2258352~195602]}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2958164, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2896863, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-03-24T22:02:42.155 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:01:22.483192+0000", 2021-03-24T22:02:42.155 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 79.419829476999993, 2021-03-24T22:02:42.155 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.088451157000000002, 2021-03-24T22:02:42.156 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:02:42.156 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:02:42.156 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:02:42.156 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.156 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:02:42.157 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:22.483192+0000", 2021-03-24T22:02:42.157 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:02:42.157 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.157 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.157 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:02:42.158 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:22.483192+0000", 2021-03-24T22:02:42.158 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2021-03-24T22:02:42.158 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.158 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.159 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:02:42.159 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:22.483190+0000", 2021-03-24T22:02:42.159 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9178000000000007e-05 2021-03-24T22:02:42.159 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.159 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.160 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:02:42.160 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:22.483269+0000", 2021-03-24T22:02:42.160 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4663000000000003e-05 2021-03-24T22:02:42.160 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.160 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.161 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:02:42.161 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:22.483314+0000", 2021-03-24T22:02:42.161 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2889000000000001e-05 2021-03-24T22:02:42.161 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.162 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.162 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:02:42.162 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:22.483337+0000", 2021-03-24T22:02:42.162 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.084227195000000005 2021-03-24T22:02:42.162 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.163 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.163 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:42.163 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:22.567564+0000", 2021-03-24T22:02:42.163 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.9276999999999997e-05 2021-03-24T22:02:42.163 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.164 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.164 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:02:42.164 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:22.567603+0000", 2021-03-24T22:02:42.164 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0040401880000000001 2021-03-24T22:02:42.165 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.165 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.165 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:02:42.165 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:22.571643+0000", 2021-03-24T22:02:42.165 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.088451157000000002 2021-03-24T22:02:42.166 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:42.166 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:02:42.166 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:42.166 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.166 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.167 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(osd.0.0:2222 7.f e560/559 7:f3f60ced:::smithi12556470-46:13b v 560'360, mlcod=0'0)", 2021-03-24T22:02:42.167 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:02:10.333458+0000", 2021-03-24T22:02:42.167 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 31.569563651999999, 2021-03-24T22:02:42.167 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021569179000000001, 2021-03-24T22:02:42.168 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:02:42.168 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:02:42.168 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:02:42.168 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.168 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:02:42.169 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:10.333458+0000", 2021-03-24T22:02:42.169 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:02:42.169 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.169 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.169 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:02:42.170 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:10.333458+0000", 2021-03-24T22:02:42.170 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999967 2021-03-24T22:02:42.170 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.170 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.171 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:02:42.171 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:10.333455+0000", 2021-03-24T22:02:42.171 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7776999999999999e-05 2021-03-24T22:02:42.171 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.171 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.172 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:02:42.172 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:10.333472+0000", 2021-03-24T22:02:42.172 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6299e-05 2021-03-24T22:02:42.172 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.172 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.173 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:02:42.173 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:10.333499+0000", 2021-03-24T22:02:42.173 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.083e-05 2021-03-24T22:02:42.173 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.174 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.174 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:02:42.174 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:10.333509+0000", 2021-03-24T22:02:42.174 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.5450000000000003e-05 2021-03-24T22:02:42.174 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.175 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.175 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:02:42.175 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:10.333605+0000", 2021-03-24T22:02:42.175 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8571000000000001e-05 2021-03-24T22:02:42.175 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.176 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.176 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:02:42.176 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:10.333633+0000", 2021-03-24T22:02:42.176 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021306110999999999 2021-03-24T22:02:42.177 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.177 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.177 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:02:42.177 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:10.354940+0000", 2021-03-24T22:02:42.177 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.7441000000000005e-05 2021-03-24T22:02:42.178 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:02:42.178 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:02:42.178 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:02:42.178 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:10.355027+0000", 2021-03-24T22:02:42.179 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021569179000000001 2021-03-24T22:02:42.179 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:42.179 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:02:42.179 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:42.179 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:02:42.180 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:02:42.180 INFO:teuthology.orchestra.run.smithi142.stdout:} 2021-03-24T22:02:42.180 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:42.182 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:02:42.234 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:42.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:42.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:02:42.340 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:42.343 INFO:tasks.rados.rados.0.smithi125.stdout:2209: done (6 left) 2021-03-24T22:02:42.343 INFO:tasks.rados.rados.0.smithi125.stdout:2210: write oid 40 current snap is 251 2021-03-24T22:02:42.343 INFO:tasks.rados.rados.0.smithi125.stdout:2210: seq_num 767 ranges {490969=551283,1816630=625589} 2021-03-24T22:02:42.349 INFO:tasks.rados.rados.0.smithi125.stdout:2210: writing smithi12556470-40 from 490969 to 1042252 tid 1 2021-03-24T22:02:42.358 INFO:tasks.rados.rados.0.smithi125.stdout:2210: writing smithi12556470-40 from 1816630 to 2442219 tid 2 2021-03-24T22:02:42.362 INFO:tasks.rados.rados.0.smithi125.stdout:2206: expect (ObjNum 610 snap 185 seq_num 610) 2021-03-24T22:02:42.370 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:42 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:02:42.370 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:42 smithi125 conmon[24927]: 02:40.612697+0000 osd.7 (osd.7) 98 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:02:42.371 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:42 smithi125 conmon[24927]: cluster 2021-03-24T22:02:40.615384 2021-03-24T22:02:42.371 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:42 smithi125 conmon[24927]: +0000 osd.7 (osd.7) 99 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:02:42.371 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:42 smithi125 conmon[24927]: 2021-03-24T22:02:42.371 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:42 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T22:02:42.372 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:42 smithi125 conmon[24927]: -24T22:02:40.910267+0000 mgr.y (mgr.14144) 567 : 2021-03-24T22:02:42.372 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:42 smithi125 conmon[24927]: cluster [DBG] pgmap v1048: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 505 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.3 MiB/s rd, 920 KiB/s wr, 10 op/s; 72/1668 objects misplaced (4.317%) 2021-03-24T22:02:42.372 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:42 smithi125 conmon[24927]: cluster 2021-03-24T22:02 2021-03-24T22:02:42.372 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:42 smithi125 conmon[24927]: :41.311273+0000 mon.a (mon.0) 2021-03-24T22:02:42.373 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:42 smithi125 conmon[24927]: 1628 : cluster [DBG] osdmap e591: 8 total, 8 up, 8 in 2021-03-24T22:02:42.373 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:42 smithi125 conmon[19936]: cluster 2021-03-24T22:02: 2021-03-24T22:02:42.373 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:42 smithi125 conmon[19936]: 40.612697+0000 osd.7 (osd.7) 98 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:02:42.373 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:42 smithi125 conmon[19936]: cluster 2021-03-24T22:02:42.374 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:42 smithi125 conmon[19936]: 2021-03-24T22:02:40.615384+0000 2021-03-24T22:02:42.374 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:42 smithi125 conmon[19936]: osd.7 (osd.7) 99 : 2021-03-24T22:02:42.374 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:42 smithi125 conmon[19936]: cluster [DBG] purged_snaps scrub ok 2021-03-24T22:02:42.374 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:42 smithi125 conmon[19936]: cluster 2021-03-24T22 2021-03-24T22:02:42.375 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:42 smithi125 conmon[19936]: :02:40.910267+0000 mgr.y (mgr.14144) 567 : cluster [DBG] pgmap v1048: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 505 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.3 MiB/s rd, 920 KiB/s wr, 10 op/s; 72/1668 objects misplaced (4.317%) 2021-03-24T22:02:42.375 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:42 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:02:42.375 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:42 smithi125 conmon[19936]: -03-24T22:02:41.311273+0000 mon.a 2021-03-24T22:02:42.375 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:42 smithi125 conmon[19936]: (mon.0) 1628 : cluster [DBG] 2021-03-24T22:02:42.375 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:42 smithi125 conmon[19936]: osdmap e591: 8 total, 8 up, 8 in 2021-03-24T22:02:42.385 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:42.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:42.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:02:42.390 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:42 smithi142 conmon[20125]: cluster 2021-03-24T22: 2021-03-24T22:02:42.390 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:42 smithi142 conmon[20125]: 02:40.612697+0000 osd.7 (osd.7) 98 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:02:42.390 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:42 smithi142 conmon[20125]: cluster 2021-03-24T22:02:42.391 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:42 smithi142 conmon[20125]: 2021-03-24T22:02:40.615384+0000 osd.7 (osd.7) 99 : cluster 2021-03-24T22:02:42.391 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:42 smithi142 conmon[20125]: [DBG] purged_snaps scrub ok 2021-03-24T22:02:42.391 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:42 smithi142 conmon[20125]: cluster 2021-03-24T22 2021-03-24T22:02:42.392 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:42 smithi142 conmon[20125]: :02:40.910267+0000 mgr.y (mgr.14144) 2021-03-24T22:02:42.392 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:42 smithi142 conmon[20125]: 567 : cluster [DBG] pgmap v1048: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 505 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.3 MiB/s rd, 920 KiB/s wr, 10 op/s; 72/1668 objects misplaced (4.317%) 2021-03-24T22:02:42.392 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:42 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:02:42.392 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:42 smithi142 conmon[20125]: T22:02:41.311273+0000 mon.a (mon. 2021-03-24T22:02:42.393 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:42 smithi142 conmon[20125]: 0) 1628 : cluster [DBG] osdmap e591: 8 total, 8 up, 8 in 2021-03-24T22:02:42.491 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:42.502 INFO:tasks.rados.rados.0.smithi125.stdout:2205: expect (ObjNum 757 snap 244 seq_num 757) 2021-03-24T22:02:42.555 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:42.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:42.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:02:42.608 INFO:tasks.rados.rados.0.smithi125.stdout:2203: expect (ObjNum 642 snap 196 seq_num 642) 2021-03-24T22:02:42.661 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:42.707 INFO:tasks.rados.rados.0.smithi125.stdout:2208: finishing write tid 1 to smithi12556470-29 2021-03-24T22:02:42.707 INFO:tasks.rados.rados.0.smithi125.stdout:2204: finishing copy_from to smithi12556470-23 2021-03-24T22:02:42.708 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 23 v 159 (ObjNum 733 snap 235 seq_num 733) dirty exists 2021-03-24T22:02:42.708 INFO:tasks.rados.rados.0.smithi125.stdout:2204: finishing copy_from racing read to smithi12556470-23 2021-03-24T22:02:42.708 INFO:tasks.rados.rados.0.smithi125.stdout:2208: finishing write tid 2 to smithi12556470-29 2021-03-24T22:02:42.708 INFO:tasks.rados.rados.0.smithi125.stdout:2208: finishing write tid 3 to smithi12556470-29 2021-03-24T22:02:42.708 INFO:tasks.rados.rados.0.smithi125.stdout:2203: done (6 left) 2021-03-24T22:02:42.709 INFO:tasks.rados.rados.0.smithi125.stdout:2204: done (5 left) 2021-03-24T22:02:42.709 INFO:tasks.rados.rados.0.smithi125.stdout:2205: done (4 left) 2021-03-24T22:02:42.709 INFO:tasks.rados.rados.0.smithi125.stdout:2206: done (3 left) 2021-03-24T22:02:42.709 INFO:tasks.rados.rados.0.smithi125.stdout:2211: read oid 14 snap -1 2021-03-24T22:02:42.710 INFO:tasks.rados.rados.0.smithi125.stdout:2211: expect (ObjNum 625 snap 192 seq_num 625) 2021-03-24T22:02:42.710 INFO:tasks.rados.rados.0.smithi125.stdout:2212: write oid 5 current snap is 251 2021-03-24T22:02:42.710 INFO:tasks.rados.rados.0.smithi125.stdout:2208: finishing write tid 4 to smithi12556470-29 2021-03-24T22:02:42.711 INFO:tasks.rados.rados.0.smithi125.stdout:2208: finishing write tid 5 to smithi12556470-29 2021-03-24T22:02:42.711 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 29 v 335 (ObjNum 766 snap 251 seq_num 766) dirty exists 2021-03-24T22:02:42.714 INFO:tasks.rados.rados.0.smithi125.stdout:2208: left oid 29 (ObjNum 766 snap 251 seq_num 766) 2021-03-24T22:02:42.714 INFO:tasks.rados.rados.0.smithi125.stdout:2207: finishing copy_from to smithi12556470-13 2021-03-24T22:02:42.714 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 13 v 788 (ObjNum 761 snap 246 seq_num 761) dirty exists 2021-03-24T22:02:42.715 INFO:tasks.rados.rados.0.smithi125.stdout:2212: seq_num 768 ranges {507776=742885,1991465=485600} 2021-03-24T22:02:42.721 INFO:tasks.rados.rados.0.smithi125.stdout:2212: writing smithi12556470-5 from 507776 to 1250661 tid 1 2021-03-24T22:02:42.722 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:42.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:42.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:02:42.729 INFO:tasks.rados.rados.0.smithi125.stdout:2212: writing smithi12556470-5 from 1991465 to 2477065 tid 2 2021-03-24T22:02:42.730 INFO:tasks.rados.rados.0.smithi125.stdout:2208: done (4 left) 2021-03-24T22:02:42.730 INFO:tasks.rados.rados.0.smithi125.stdout:2213: rollback oid 19 current snap is 251 2021-03-24T22:02:42.730 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 19 to 248 2021-03-24T22:02:42.730 INFO:tasks.rados.rados.0.smithi125.stdout:2207: finishing copy_from racing read to smithi12556470-13 2021-03-24T22:02:42.731 INFO:tasks.rados.rados.0.smithi125.stdout:2210: finishing write tid 1 to smithi12556470-40 2021-03-24T22:02:42.731 INFO:tasks.rados.rados.0.smithi125.stdout:2210: finishing write tid 2 to smithi12556470-40 2021-03-24T22:02:42.731 INFO:tasks.rados.rados.0.smithi125.stdout:2210: finishing write tid 3 to smithi12556470-40 2021-03-24T22:02:42.731 INFO:tasks.rados.rados.0.smithi125.stdout:2210: finishing write tid 4 to smithi12556470-40 2021-03-24T22:02:42.731 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 40 v 1174 (ObjNum 767 snap 251 seq_num 767) dirty exists 2021-03-24T22:02:42.732 INFO:tasks.rados.rados.0.smithi125.stdout:2210: left oid 40 (ObjNum 767 snap 251 seq_num 767) 2021-03-24T22:02:42.732 INFO:tasks.rados.rados.0.smithi125.stdout:2207: done (4 left) 2021-03-24T22:02:42.732 INFO:tasks.rados.rados.0.smithi125.stdout:2210: done (3 left) 2021-03-24T22:02:42.732 INFO:tasks.rados.rados.0.smithi125.stdout:2214: delete oid 8 current snap is 251 2021-03-24T22:02:42.732 INFO:tasks.rados.rados.0.smithi125.stdout:2211: expect (ObjNum 625 snap 192 seq_num 625) 2021-03-24T22:02:42.799 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-24T22:02:42.813 INFO:tasks.rados.rados.0.smithi125.stdout:2211: done (3 left) 2021-03-24T22:02:42.813 INFO:tasks.rados.rados.0.smithi125.stdout:2214: done (2 left) 2021-03-24T22:02:42.813 INFO:tasks.rados.rados.0.smithi125.stdout:2215: read oid 25 snap 168 2021-03-24T22:02:42.813 INFO:tasks.rados.rados.0.smithi125.stdout:2215: expect (ObjNum 477 snap 136 seq_num 477) 2021-03-24T22:02:42.814 INFO:tasks.rados.rados.0.smithi125.stdout:2216: read oid 42 snap 212 2021-03-24T22:02:42.814 INFO:tasks.rados.rados.0.smithi125.stdout:2216: expect deleted 2021-03-24T22:02:42.814 INFO:tasks.rados.rados.0.smithi125.stdout:2217: write (excl) oid 16 current snap is 251 2021-03-24T22:02:42.814 INFO:tasks.rados.rados.0.smithi125.stdout:2212: finishing write tid 1 to smithi12556470-5 2021-03-24T22:02:42.814 INFO:tasks.rados.rados.0.smithi125.stdout:2217: seq_num 770 ranges {541390=493383,1592076=489269,2862175=79215} 2021-03-24T22:02:42.818 INFO:tasks.rados.rados.0.smithi125.stdout:2217: writing smithi12556470-16 from 541390 to 1034773 tid 1 2021-03-24T22:02:42.825 INFO:tasks.rados.rados.0.smithi125.stdout:2217: writing smithi12556470-16 from 1592076 to 2081345 tid 2 2021-03-24T22:02:42.826 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:42.831 INFO:tasks.rados.rados.0.smithi125.stdout:2217: writing smithi12556470-16 from 2862175 to 2941390 tid 3 2021-03-24T22:02:42.831 INFO:tasks.rados.rados.0.smithi125.stdout:2218: write oid 32 current snap is 251 2021-03-24T22:02:42.831 INFO:tasks.rados.rados.0.smithi125.stdout:2213: finishing rollback tid 0 to smithi12556470-19 2021-03-24T22:02:42.831 INFO:tasks.rados.rados.0.smithi125.stdout:2213: finishing rollback tid 1 to smithi12556470-19 2021-03-24T22:02:42.832 INFO:tasks.rados.rados.0.smithi125.stdout:2218: seq_num 771 ranges {558197=568632,1650558=632927,2773245=184952} 2021-03-24T22:02:42.837 INFO:tasks.rados.rados.0.smithi125.stdout:2218: writing smithi12556470-32 from 558197 to 1126829 tid 1 2021-03-24T22:02:42.845 INFO:tasks.rados.rados.0.smithi125.stdout:2218: writing smithi12556470-32 from 1650558 to 2283485 tid 2 2021-03-24T22:02:42.849 INFO:tasks.rados.rados.0.smithi125.stdout:2218: writing smithi12556470-32 from 2773245 to 2958197 tid 3 2021-03-24T22:02:42.850 INFO:tasks.rados.rados.0.smithi125.stdout:2219: copy_from oid 37 from oid 50 current snap is 251 2021-03-24T22:02:42.850 INFO:tasks.rados.rados.0.smithi125.stdout:2213: finishing rollback tid 2 to smithi12556470-19 2021-03-24T22:02:42.850 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 19 v 507 (ObjNum 588 snap 177 seq_num 588) dirty exists 2021-03-24T22:02:42.850 INFO:tasks.rados.rados.0.smithi125.stdout:2212: finishing write tid 2 to smithi12556470-5 2021-03-24T22:02:42.850 INFO:tasks.rados.rados.0.smithi125.stdout:2212: finishing write tid 3 to smithi12556470-5 2021-03-24T22:02:42.851 INFO:tasks.rados.rados.0.smithi125.stdout:2212: finishing write tid 4 to smithi12556470-5 2021-03-24T22:02:42.851 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 5 v 684 (ObjNum 768 snap 251 seq_num 768) dirty exists 2021-03-24T22:02:42.851 INFO:tasks.rados.rados.0.smithi125.stdout:2212: left oid 5 (ObjNum 768 snap 251 seq_num 768) 2021-03-24T22:02:42.851 INFO:tasks.rados.rados.0.smithi125.stdout:2212: done (6 left) 2021-03-24T22:02:42.852 INFO:tasks.rados.rados.0.smithi125.stdout:2213: done (5 left) 2021-03-24T22:02:42.852 INFO:tasks.rados.rados.0.smithi125.stdout:2220: delete oid 48 current snap is 251 2021-03-24T22:02:42.852 INFO:tasks.rados.rados.0.smithi125.stdout:2215: expect (ObjNum 477 snap 136 seq_num 477) 2021-03-24T22:02:42.881 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:42.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:42.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:02:42.891 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.5 dump_ops_in_flight 2021-03-24T22:02:42.969 INFO:tasks.rados.rados.0.smithi125.stdout:2215: done (5 left) 2021-03-24T22:02:42.970 INFO:tasks.rados.rados.0.smithi125.stdout:2216: done (4 left) 2021-03-24T22:02:42.970 INFO:tasks.rados.rados.0.smithi125.stdout:2220: done (3 left) 2021-03-24T22:02:42.970 INFO:tasks.rados.rados.0.smithi125.stdout:2221: read oid 18 snap 168 2021-03-24T22:02:42.971 INFO:tasks.rados.rados.0.smithi125.stdout:2221: expect (ObjNum 534 snap 162 seq_num 534) 2021-03-24T22:02:42.971 INFO:tasks.rados.rados.0.smithi125.stdout:2222: rollback oid 12 current snap is 251 2021-03-24T22:02:42.971 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 12 to 242 2021-03-24T22:02:42.971 INFO:tasks.rados.rados.0.smithi125.stdout:2217: finishing write tid 1 to smithi12556470-16 2021-03-24T22:02:42.971 INFO:tasks.rados.rados.0.smithi125.stdout:2217: finishing write tid 2 to smithi12556470-16 2021-03-24T22:02:42.972 INFO:tasks.rados.rados.0.smithi125.stdout:2217: finishing write tid 3 to smithi12556470-16 2021-03-24T22:02:42.972 INFO:tasks.rados.rados.0.smithi125.stdout:2223: rollback oid 25 current snap is 251 2021-03-24T22:02:42.972 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 25 to 238 2021-03-24T22:02:42.972 INFO:tasks.rados.rados.0.smithi125.stdout:2224: rollback oid 23 current snap is 251 2021-03-24T22:02:42.973 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 23 to 224 2021-03-24T22:02:42.973 INFO:tasks.rados.rados.0.smithi125.stdout:2217: finishing write tid 4 to smithi12556470-16 2021-03-24T22:02:42.973 INFO:tasks.rados.rados.0.smithi125.stdout:2217: finishing write tid 5 to smithi12556470-16 2021-03-24T22:02:42.973 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 16 v 415 (ObjNum 770 snap 251 seq_num 770) dirty exists 2021-03-24T22:02:42.973 INFO:tasks.rados.rados.0.smithi125.stdout:2217: left oid 16 (ObjNum 770 snap 251 seq_num 770) 2021-03-24T22:02:42.974 INFO:tasks.rados.rados.0.smithi125.stdout:2218: finishing write tid 1 to smithi12556470-32 2021-03-24T22:02:42.974 INFO:tasks.rados.rados.0.smithi125.stdout:2217: done (6 left) 2021-03-24T22:02:42.975 INFO:tasks.rados.rados.0.smithi125.stdout:2225: write (excl) oid 19 current snap is 251 2021-03-24T22:02:42.975 INFO:tasks.rados.rados.0.smithi125.stdout:2218: finishing write tid 2 to smithi12556470-32 2021-03-24T22:02:42.975 INFO:tasks.rados.rados.0.smithi125.stdout:2218: finishing write tid 3 to smithi12556470-32 2021-03-24T22:02:42.975 INFO:tasks.rados.rados.0.smithi125.stdout:2218: finishing write tid 4 to smithi12556470-32 2021-03-24T22:02:42.975 INFO:tasks.rados.rados.0.smithi125.stdout:2218: finishing write tid 5 to smithi12556470-32 2021-03-24T22:02:42.976 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 32 v 1181 (ObjNum 771 snap 251 seq_num 771) dirty exists 2021-03-24T22:02:42.976 INFO:tasks.rados.rados.0.smithi125.stdout:2218: left oid 32 (ObjNum 771 snap 251 seq_num 771) 2021-03-24T22:02:42.976 INFO:tasks.rados.rados.0.smithi125.stdout:2219: finishing copy_from to smithi12556470-37 2021-03-24T22:02:42.976 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 37 v 722 (ObjNum 757 snap 244 seq_num 757) dirty exists 2021-03-24T22:02:42.977 INFO:tasks.rados.rados.0.smithi125.stdout:2219: finishing copy_from racing read to smithi12556470-37 2021-03-24T22:02:42.977 INFO:tasks.rados.rados.0.smithi125.stdout:2225: seq_num 773 ranges {591811=719130,1767522=636596,2828091=163720} 2021-03-24T22:02:42.978 INFO:tasks.rados.rados.0.smithi125.stdout:2225: writing smithi12556470-19 from 591811 to 1310941 tid 1 2021-03-24T22:02:42.985 INFO:tasks.rados.rados.0.smithi125.stdout:2225: writing smithi12556470-19 from 1767522 to 2404118 tid 2 2021-03-24T22:02:42.986 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:42.989 INFO:tasks.rados.rados.0.smithi125.stdout:2225: writing smithi12556470-19 from 2828091 to 2991811 tid 3 2021-03-24T22:02:42.990 INFO:tasks.rados.rados.0.smithi125.stdout:2223: finishing rollback tid 1 to smithi12556470-25 2021-03-24T22:02:42.990 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 25 v 510 (ObjNum 0 snap 3 seq_num 7) dirty dne 2021-03-24T22:02:42.990 INFO:tasks.rados.rados.0.smithi125.stdout:2218: done (6 left) 2021-03-24T22:02:42.990 INFO:tasks.rados.rados.0.smithi125.stdout:2219: done (5 left) 2021-03-24T22:02:42.991 INFO:tasks.rados.rados.0.smithi125.stdout:2223: done (4 left) 2021-03-24T22:02:42.991 INFO:tasks.rados.rados.0.smithi125.stdout:2226: rollback oid 31 current snap is 251 2021-03-24T22:02:42.991 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 31 to 247 2021-03-24T22:02:42.991 INFO:tasks.rados.rados.0.smithi125.stdout:2224: finishing rollback tid 0 to smithi12556470-23 2021-03-24T22:02:42.992 INFO:tasks.rados.rados.0.smithi125.stdout:2224: finishing rollback tid 1 to smithi12556470-23 2021-03-24T22:02:42.992 INFO:tasks.rados.rados.0.smithi125.stdout:2222: finishing rollback tid 0 to smithi12556470-12 2021-03-24T22:02:42.992 INFO:tasks.rados.rados.0.smithi125.stdout:2224: finishing rollback tid 2 to smithi12556470-23 2021-03-24T22:02:42.992 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 23 v 380 (ObjNum 695 snap 218 seq_num 695) dirty exists 2021-03-24T22:02:42.992 INFO:tasks.rados.rados.0.smithi125.stdout:2222: finishing rollback tid 1 to smithi12556470-12 2021-03-24T22:02:42.993 INFO:tasks.rados.rados.0.smithi125.stdout:2222: finishing rollback tid 2 to smithi12556470-12 2021-03-24T22:02:42.993 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 12 v 502 (ObjNum 752 snap 241 seq_num 752) dirty exists 2021-03-24T22:02:42.993 INFO:tasks.rados.rados.0.smithi125.stdout:2221: expect (ObjNum 534 snap 162 seq_num 534) 2021-03-24T22:02:43.044 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:43.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:43.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:02:43.150 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:43.185 INFO:tasks.rados.rados.0.smithi125.stdout:2225: finishing write tid 1 to smithi12556470-19 2021-03-24T22:02:43.185 INFO:tasks.rados.rados.0.smithi125.stdout:2225: finishing write tid 2 to smithi12556470-19 2021-03-24T22:02:43.185 INFO:tasks.rados.rados.0.smithi125.stdout:2225: finishing write tid 3 to smithi12556470-19 2021-03-24T22:02:43.186 INFO:tasks.rados.rados.0.smithi125.stdout:2225: finishing write tid 4 to smithi12556470-19 2021-03-24T22:02:43.186 INFO:tasks.rados.rados.0.smithi125.stdout:2225: finishing write tid 5 to smithi12556470-19 2021-03-24T22:02:43.186 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 19 v 511 (ObjNum 773 snap 251 seq_num 773) dirty exists 2021-03-24T22:02:43.187 INFO:tasks.rados.rados.0.smithi125.stdout:2225: left oid 19 (ObjNum 773 snap 251 seq_num 773) 2021-03-24T22:02:43.187 INFO:tasks.rados.rados.0.smithi125.stdout:2226: finishing rollback tid 0 to smithi12556470-31 2021-03-24T22:02:43.187 INFO:tasks.rados.rados.0.smithi125.stdout:2226: finishing rollback tid 1 to smithi12556470-31 2021-03-24T22:02:43.187 INFO:tasks.rados.rados.0.smithi125.stdout:2221: done (4 left) 2021-03-24T22:02:43.188 INFO:tasks.rados.rados.0.smithi125.stdout:2222: done (3 left) 2021-03-24T22:02:43.188 INFO:tasks.rados.rados.0.smithi125.stdout:2224: done (2 left) 2021-03-24T22:02:43.188 INFO:tasks.rados.rados.0.smithi125.stdout:2225: done (1 left) 2021-03-24T22:02:43.188 INFO:tasks.rados.rados.0.smithi125.stdout:2227: write oid 48 current snap is 251 2021-03-24T22:02:43.189 INFO:tasks.rados.rados.0.smithi125.stdout:2226: finishing rollback tid 2 to smithi12556470-31 2021-03-24T22:02:43.189 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 31 v 1185 (ObjNum 718 snap 229 seq_num 718) dirty exists 2021-03-24T22:02:43.190 INFO:tasks.rados.rados.0.smithi125.stdout:2227: seq_num 774 ranges {608618=794379,2109651=780254} 2021-03-24T22:02:43.191 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T22:02:43.198 INFO:tasks.rados.rados.0.smithi125.stdout:2227: writing smithi12556470-48 from 608618 to 1402997 tid 1 2021-03-24T22:02:43.206 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:43.208 INFO:tasks.rados.rados.0.smithi125.stdout:2227: writing smithi12556470-48 from 2109651 to 2889905 tid 2 2021-03-24T22:02:43.208 INFO:tasks.rados.rados.0.smithi125.stdout:2226: done (1 left) 2021-03-24T22:02:43.208 INFO:tasks.rados.rados.0.smithi125.stdout:2228: read oid 7 snap -1 2021-03-24T22:02:43.208 INFO:tasks.rados.rados.0.smithi125.stdout:2228: expect (ObjNum 727 snap 233 seq_num 727) 2021-03-24T22:02:43.209 INFO:tasks.rados.rados.0.smithi125.stdout:2229: copy_from oid 20 from oid 38 current snap is 251 2021-03-24T22:02:43.209 INFO:tasks.rados.rados.0.smithi125.stdout:2227: finishing write tid 1 to smithi12556470-48 2021-03-24T22:02:43.209 INFO:tasks.rados.rados.0.smithi125.stdout:2230: read oid 49 snap -1 2021-03-24T22:02:43.209 INFO:tasks.rados.rados.0.smithi125.stdout:2230: expect (ObjNum 734 snap 235 seq_num 734) 2021-03-24T22:02:43.210 INFO:tasks.rados.rados.0.smithi125.stdout:2231: snap_create 2021-03-24T22:02:43.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:43.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:02:43.220 INFO:tasks.rados.rados.0.smithi125.stdout:2230: expect (ObjNum 734 snap 235 seq_num 734) 2021-03-24T22:02:43.312 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:43.362 INFO:tasks.rados.rados.0.smithi125.stdout:2227: finishing write tid 2 to smithi12556470-48 2021-03-24T22:02:43.363 INFO:tasks.rados.rados.0.smithi125.stdout:2227: finishing write tid 3 to smithi12556470-48 2021-03-24T22:02:43.363 INFO:tasks.rados.rados.0.smithi125.stdout:2227: finishing write tid 4 to smithi12556470-48 2021-03-24T22:02:43.363 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 48 v 767 (ObjNum 774 snap 251 seq_num 774) dirty exists 2021-03-24T22:02:43.364 INFO:tasks.rados.rados.0.smithi125.stdout:2227: left oid 48 (ObjNum 774 snap 251 seq_num 774) 2021-03-24T22:02:43.364 INFO:tasks.rados.rados.0.smithi125.stdout:2229: finishing copy_from to smithi12556470-20 2021-03-24T22:02:43.364 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 20 v 159 (ObjNum 733 snap 235 seq_num 733) dirty exists 2021-03-24T22:02:43.364 INFO:tasks.rados.rados.0.smithi125.stdout:2229: finishing copy_from racing read to smithi12556470-20 2021-03-24T22:02:43.364 INFO:tasks.rados.rados.0.smithi125.stdout:2228: expect (ObjNum 727 snap 233 seq_num 727) 2021-03-24T22:02:43.366 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:43.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:43.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:02:43.446 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:43 smithi142 conmon[20125]: cluster 2021-03-24T22:02:42 2021-03-24T22:02:43.446 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:43 smithi142 conmon[20125]: .328545+0000 mon.a (mon.0) 1629 : cluster [DBG] osdmap e592: 8 total, 8 up, 8 in 2021-03-24T22:02:43.471 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:43.530 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:43.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:43.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:02:43.543 INFO:tasks.rados.rados.0.smithi125.stdout:2227: done (4 left) 2021-03-24T22:02:43.544 INFO:tasks.rados.rados.0.smithi125.stdout:2228: done (3 left) 2021-03-24T22:02:43.544 INFO:tasks.rados.rados.0.smithi125.stdout:2229: done (2 left) 2021-03-24T22:02:43.544 INFO:tasks.rados.rados.0.smithi125.stdout:2230: done (1 left) 2021-03-24T22:02:43.545 INFO:tasks.rados.rados.0.smithi125.stdout:2231: done (0 left) 2021-03-24T22:02:43.545 INFO:tasks.rados.rados.0.smithi125.stdout:2232: snap_remove snap 247 2021-03-24T22:02:43.637 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:43.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:43 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:02:43.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:43 smithi125 conmon[19936]: T22:02:42.328545+0000 mon.a (mon.0) 1629 : cluster [DBG] osdmap e592: 8 total, 8 up, 8 in 2021-03-24T22:02:43.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:43 smithi125 conmon[24927]: cluster 2021-03-24T22:02:42.328545+0000 2021-03-24T22:02:43.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:43 smithi125 conmon[24927]: mon.a (mon.0) 1629 : cluster [DBG] osdmap e592: 8 total, 8 up, 8 in 2021-03-24T22:02:43.698 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:43.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:43.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:02:43.803 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:43.855 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:43.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:43.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:43.961 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:44.014 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:44.016 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:02:44.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:44.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:02:44.119 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:44.169 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:44.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:44.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:44.275 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:44.325 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:44.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:44.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:02:44.354 INFO:tasks.rados.rados.0.smithi125.stdout:2232: done (0 left) 2021-03-24T22:02:44.354 INFO:tasks.rados.rados.0.smithi125.stdout:2233: write (excl) oid 30 current snap is 252 2021-03-24T22:02:44.355 INFO:tasks.rados.rados.0.smithi125.stdout:2233: seq_num 775 ranges {625425=469628,1768133=640265} 2021-03-24T22:02:44.355 INFO:tasks.rados.rados.0.smithi125.stdout:2233: writing smithi12556470-30 from 625425 to 1095053 tid 1 2021-03-24T22:02:44.359 INFO:tasks.rados.rados.0.smithi125.stdout:2233: writing smithi12556470-30 from 1768133 to 2408398 tid 2 2021-03-24T22:02:44.359 INFO:tasks.rados.rados.0.smithi125.stdout:2234: rollback oid 50 current snap is 252 2021-03-24T22:02:44.359 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 50 to 219 2021-03-24T22:02:44.360 INFO:tasks.rados.rados.0.smithi125.stdout:2235: write oid 10 current snap is 252 2021-03-24T22:02:44.360 INFO:tasks.rados.rados.0.smithi125.stdout:2235: seq_num 776 ranges {642232=661230,1942968=783923} 2021-03-24T22:02:44.367 INFO:teuthology.orchestra.run.smithi125.stderr:nodeep-scrub is set 2021-03-24T22:02:44.368 INFO:tasks.rados.rados.0.smithi125.stdout:2235: writing smithi12556470-10 from 642232 to 1303462 tid 1 2021-03-24T22:02:44.380 INFO:tasks.rados.rados.0.smithi125.stdout:2235: writing smithi12556470-10 from 1942968 to 2726891 tid 2 2021-03-24T22:02:44.380 INFO:tasks.rados.rados.0.smithi125.stdout:2236: copy_from oid 22 from oid 42 current snap is 252 2021-03-24T22:02:44.380 INFO:tasks.rados.rados.0.smithi125.stdout:2233: finishing write tid 1 to smithi12556470-30 2021-03-24T22:02:44.381 INFO:tasks.rados.rados.0.smithi125.stdout:2233: finishing write tid 2 to smithi12556470-30 2021-03-24T22:02:44.381 INFO:tasks.rados.rados.0.smithi125.stdout:2234: finishing rollback tid 0 to smithi12556470-50 2021-03-24T22:02:44.382 INFO:tasks.rados.rados.0.smithi125.stdout:2234: finishing rollback tid 1 to smithi12556470-50 2021-03-24T22:02:44.382 INFO:tasks.rados.rados.0.smithi125.stdout:2233: finishing write tid 3 to smithi12556470-30 2021-03-24T22:02:44.383 INFO:tasks.rados.rados.0.smithi125.stdout:2234: finishing rollback tid 2 to smithi12556470-50 2021-03-24T22:02:44.383 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 50 v 781 (ObjNum 659 snap 202 seq_num 659) dirty exists 2021-03-24T22:02:44.384 INFO:tasks.rados.rados.0.smithi125.stdout:2233: finishing write tid 4 to smithi12556470-30 2021-03-24T22:02:44.384 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 30 v 313 (ObjNum 775 snap 252 seq_num 775) dirty exists 2021-03-24T22:02:44.385 INFO:tasks.rados.rados.0.smithi125.stdout:2233: left oid 30 (ObjNum 775 snap 252 seq_num 775) 2021-03-24T22:02:44.385 INFO:tasks.rados.rados.0.smithi125.stdout:2235: finishing write tid 1 to smithi12556470-10 2021-03-24T22:02:44.385 INFO:tasks.rados.rados.0.smithi125.stdout:2233: done (3 left) 2021-03-24T22:02:44.386 INFO:tasks.rados.rados.0.smithi125.stdout:2234: done (2 left) 2021-03-24T22:02:44.386 INFO:tasks.rados.rados.0.smithi125.stdout:2237: copy_from oid 5 from oid 39 current snap is 252 2021-03-24T22:02:44.387 INFO:tasks.rados.rados.0.smithi125.stdout:2238: read oid 4 snap -1 2021-03-24T22:02:44.387 INFO:tasks.rados.rados.0.smithi125.stdout:2238: expect (ObjNum 625 snap 192 seq_num 625) 2021-03-24T22:02:44.388 INFO:tasks.rados.rados.0.smithi125.stdout:2239: rollback oid 44 current snap is 252 2021-03-24T22:02:44.388 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 44 to 251 2021-03-24T22:02:44.388 INFO:tasks.rados.rados.0.smithi125.stdout:2240: rollback oid 35 current snap is 252 2021-03-24T22:02:44.389 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 35 to 207 2021-03-24T22:02:44.389 INFO:tasks.rados.rados.0.smithi125.stdout:2241: snap_create 2021-03-24T22:02:44.391 INFO:tasks.rados.rados.0.smithi125.stdout:2235: finishing write tid 2 to smithi12556470-10 2021-03-24T22:02:44.391 INFO:tasks.rados.rados.0.smithi125.stdout:2235: finishing write tid 3 to smithi12556470-10 2021-03-24T22:02:44.392 INFO:tasks.rados.rados.0.smithi125.stdout:2235: finishing write tid 4 to smithi12556470-10 2021-03-24T22:02:44.392 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 10 v 838 (ObjNum 776 snap 252 seq_num 776) dirty exists 2021-03-24T22:02:44.392 INFO:tasks.rados.rados.0.smithi125.stdout:2235: left oid 10 (ObjNum 776 snap 252 seq_num 776) 2021-03-24T22:02:44.393 INFO:tasks.rados.rados.0.smithi125.stdout:2238: expect (ObjNum 625 snap 192 seq_num 625) 2021-03-24T22:02:44.430 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:44.479 INFO:tasks.rados.rados.0.smithi125.stdout:2239: finishing rollback tid 0 to smithi12556470-44 2021-03-24T22:02:44.479 INFO:tasks.rados.rados.0.smithi125.stdout:2240: finishing rollback tid 0 to smithi12556470-35 2021-03-24T22:02:44.479 INFO:tasks.rados.rados.0.smithi125.stdout:2239: finishing rollback tid 1 to smithi12556470-44 2021-03-24T22:02:44.480 INFO:tasks.rados.rados.0.smithi125.stdout:2239: finishing rollback tid 2 to smithi12556470-44 2021-03-24T22:02:44.480 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 44 v 499 (ObjNum 753 snap 241 seq_num 753) dirty exists 2021-03-24T22:02:44.480 INFO:tasks.rados.rados.0.smithi125.stdout:2240: finishing rollback tid 1 to smithi12556470-35 2021-03-24T22:02:44.480 INFO:tasks.rados.rados.0.smithi125.stdout:2240: finishing rollback tid 2 to smithi12556470-35 2021-03-24T22:02:44.481 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 35 v 219 (ObjNum 639 snap 196 seq_num 639) dirty exists 2021-03-24T22:02:44.481 INFO:tasks.rados.rados.0.smithi125.stdout:2237: finishing copy_from to smithi12556470-5 2021-03-24T22:02:44.481 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 5 v 767 (ObjNum 742 snap 240 seq_num 742) dirty exists 2021-03-24T22:02:44.481 INFO:tasks.rados.rados.0.smithi125.stdout:2237: finishing copy_from racing read to smithi12556470-5 2021-03-24T22:02:44.482 INFO:tasks.rados.rados.0.smithi125.stdout:2236: finishing copy_from to smithi12556470-22 2021-03-24T22:02:44.482 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 22 v 503 (ObjNum 736 snap 235 seq_num 736) dirty exists 2021-03-24T22:02:44.482 INFO:tasks.rados.rados.0.smithi125.stdout:2236: finishing copy_from racing read to smithi12556470-22 2021-03-24T22:02:44.492 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:44.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:44.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:02:44.521 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:44 smithi125 conmon[19936]: cluster 2021-03-24T22:02:42.910855+0000 2021-03-24T22:02:44.521 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:44 smithi125 conmon[19936]: mgr.y (mgr.14144) 568 : cluster [DBG] pgmap v1051: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 508 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.3 MiB/s rd, 1.8 MiB/s wr, 12 op/s; 72/1671 objects misplaced (4.309%) 2021-03-24T22:02:44.522 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:44 smithi125 conmon[19936]: cluster 2021-03-24T22:02:43.333508+0000 2021-03-24T22:02:44.522 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:44 smithi125 conmon[19936]: mon.a (mon.0) 1630 : cluster [DBG] osdmap e593: 8 total, 8 up, 8 in 2021-03-24T22:02:44.522 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:44 smithi125 conmon[19936]: audit 2021-03-24T22:02:44.522 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:44 smithi125 conmon[19936]: 2021-03-24T22:02:44.026552+0000 mon.c (mon. 2021-03-24T22:02:44.523 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:44 smithi125 conmon[19936]: 2) 90 : audit [INF] from='client.? v1:172.21.15.125:0/3450467452' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:02:44.523 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:44 smithi125 conmon[19936]: audit 2021 2021-03-24T22:02:44.523 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:44 smithi125 conmon[19936]: -03-24T22:02:44.026958+0000 mon.a ( 2021-03-24T22:02:44.523 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:44 smithi125 conmon[19936]: mon.0) 1631 : audit [INF] from='client.35504 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:02:44.524 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:44 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T22:02:44.524 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:44 smithi125 conmon[24927]: T22:02:42.910855+0000 mgr.y (mgr.14144) 568 : cluster [DBG] 2021-03-24T22:02:44.524 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:44 smithi125 conmon[24927]: pgmap v1051: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 508 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.3 MiB/s rd, 1.8 MiB/s wr, 12 op/s; 72/1671 objects misplaced (4.309%) 2021-03-24T22:02:44.524 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:44 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:02:44.525 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:44 smithi125 conmon[24927]: 24T22:02:43.333508 2021-03-24T22:02:44.525 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:44 smithi125 conmon[24927]: +0000 mon.a (mon.0) 1630 2021-03-24T22:02:44.525 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:44 smithi125 conmon[24927]: : cluster [DBG] osdmap e593: 8 total, 8 up, 8 in 2021-03-24T22:02:44.525 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:44 smithi125 conmon[24927]: audit 2021- 2021-03-24T22:02:44.525 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:44 smithi125 conmon[24927]: 03-24T22:02:44. 2021-03-24T22:02:44.526 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:44 smithi125 conmon[24927]: 026552+0000 mon.c (mon.2) 2021-03-24T22:02:44.526 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:44 smithi125 conmon[24927]: 90 : audit [INF] from='client.? v1:172.21.15.125:0/3450467452' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:02:44.526 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:44 smithi125 conmon[24927]: audit 2021-03 2021-03-24T22:02:44.526 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:44 smithi125 conmon[24927]: -24T22:02:44. 2021-03-24T22:02:44.527 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:44 smithi125 conmon[24927]: 026958+0000 mon.a (mon.0) 2021-03-24T22:02:44.527 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:44 smithi125 conmon[24927]: 1631 : audit [INF] from='client.35504 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:02:44.598 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:44.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:44 smithi142 conmon[20125]: cluster 2021-03-24T22:02:42.910855+0000 mgr.y (mgr.14144) 568 : cluster 2021-03-24T22:02:44.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:44 smithi142 conmon[20125]: [DBG] pgmap v1051: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 508 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 7.3 MiB/s rd, 1.8 MiB/s wr, 12 op/s; 72/1671 objects misplaced (4.309%) 2021-03-24T22:02:44.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:44 smithi142 conmon[20125]: cluster 2021-03-24T22:02:43.333508+0000 mon.a (mon.0) 1630 : cluster 2021-03-24T22:02:44.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:44 smithi142 conmon[20125]: [DBG] osdmap e593: 8 total, 8 up, 8 in 2021-03-24T22:02:44.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:44 smithi142 conmon[20125]: audit 2021 2021-03-24T22:02:44.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:44 smithi142 conmon[20125]: -03-24T22:02:44.026552+0000 mon.c ( 2021-03-24T22:02:44.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:44 smithi142 conmon[20125]: mon.2) 90 : audit [INF] from='client.? v1:172.21.15.125:0/3450467452' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:02:44.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:44 smithi142 conmon[20125]: audit 2021- 2021-03-24T22:02:44.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:44 smithi142 conmon[20125]: 03-24T22:02:44.026958+0000 2021-03-24T22:02:44.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:44 smithi142 conmon[20125]: mon.a (mon.0) 1631 : audit [INF] 2021-03-24T22:02:44.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:44 smithi142 conmon[20125]: from='client.35504 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:02:44.652 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:44.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:44.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:44.757 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:44.818 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:44.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:44.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:02:44.923 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:44.982 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:44.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:44.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:02:45.087 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:45.141 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:45.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:45.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:02:45.165 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2021-03-24T22:02:45.166 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2021-03-24T22:02:45.166 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2021-03-24T22:02:45.166 INFO:teuthology.orchestra.run.smithi142.stdout:} 2021-03-24T22:02:45.246 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:45.300 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:45.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:45.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:02:45.315 INFO:tasks.rados.rados.0.smithi125.stdout:2235: done (6 left) 2021-03-24T22:02:45.315 INFO:tasks.rados.rados.0.smithi125.stdout:2236: done (5 left) 2021-03-24T22:02:45.316 INFO:tasks.rados.rados.0.smithi125.stdout:2237: done (4 left) 2021-03-24T22:02:45.316 INFO:tasks.rados.rados.0.smithi125.stdout:2238: done (3 left) 2021-03-24T22:02:45.316 INFO:tasks.rados.rados.0.smithi125.stdout:2239: done (2 left) 2021-03-24T22:02:45.316 INFO:tasks.rados.rados.0.smithi125.stdout:2240: done (1 left) 2021-03-24T22:02:45.317 INFO:tasks.rados.rados.0.smithi125.stdout:2241: done (0 left) 2021-03-24T22:02:45.317 INFO:tasks.rados.rados.0.smithi125.stdout:2242: snap_create 2021-03-24T22:02:45.405 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:45.452 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:45.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:45.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:02:45.559 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:45.620 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:45.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:45.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:02:45.637 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:45 smithi125 conmon[19936]: cluster 2021-03-24T22:02:44.330071+0000 mon.a (mon.0 2021-03-24T22:02:45.637 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:45 smithi125 conmon[19936]: ) 1632 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:02:45.637 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:45 smithi125 conmon[19936]: audit 2021-03-24T22:02:44.340952+0000 mon.a (mon.0) 1633 : audit [INF] from='client.35504 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T22:02:45.638 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:45 smithi125 conmon[19936]: cluster 2021-03-24T22:02:44.341037 2021-03-24T22:02:45.638 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:45 smithi125 conmon[19936]: +0000 mon.a (mon.0) 1634 : cluster [DBG] osdmap e594: 8 total, 8 up, 8 in 2021-03-24T22:02:45.638 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:45 smithi125 conmon[19936]: cluster 2021-03-24T22:02:45.638 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:45 smithi125 conmon[19936]: 2021-03-24T22:02:45.309124+0000 mon.a (mon 2021-03-24T22:02:45.639 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:45 smithi125 conmon[19936]: .0) 1635 : cluster [DBG] osdmap e595: 8 total, 8 up, 8 in 2021-03-24T22:02:45.639 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:45 smithi125 conmon[24927]: cluster 2021-03-24T22:02:45.639 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:45 smithi125 conmon[24927]: 2021-03-24T22:02: 2021-03-24T22:02:45.640 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:45 smithi125 conmon[24927]: 44.330071+0000 mon.a (mon.0) 1632 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:02:45.640 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:45 smithi125 conmon[24927]: audit 2021-03-24T22:02:45.640 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:45 smithi125 conmon[24927]: 2021-03- 2021-03-24T22:02:45.640 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:45 smithi125 conmon[24927]: 24T22:02:44.340952+0000 2021-03-24T22:02:45.640 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:45 smithi125 conmon[24927]: mon.a (mon.0) 1633 : audit [INF] 2021-03-24T22:02:45.641 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:45 smithi125 conmon[24927]: from='client.35504 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T22:02:45.641 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:45 smithi125 conmon[24927]: cluster 2021-03-24T22:02:45.641 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:45 smithi125 conmon[24927]: 2021-03-24T22:02: 2021-03-24T22:02:45.641 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:45 smithi125 conmon[24927]: 44.341037+0000 mon.a (mon.0) 2021-03-24T22:02:45.642 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:45 smithi125 conmon[24927]: 1634 : cluster [DBG] osdmap e594: 8 total, 8 up, 8 in 2021-03-24T22:02:45.642 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:45 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:02:45.642 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:45 smithi125 conmon[24927]: -03-24T22:02:45.309124 2021-03-24T22:02:45.642 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:45 smithi125 conmon[24927]: +0000 mon.a (mon.0) 1635 : cluster 2021-03-24T22:02:45.642 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:45 smithi125 conmon[24927]: [DBG] osdmap e595: 8 total, 8 up, 8 in 2021-03-24T22:02:45.727 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:45.776 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:45 smithi142 conmon[20125]: cluster 2021-03-24T22:02:44.330071 2021-03-24T22:02:45.776 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:45 smithi142 conmon[20125]: +0000 mon.a (mon.0) 1632 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:02:45.776 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:45 smithi142 conmon[20125]: audit 2021-03-24T22:02:44.340952+0000 2021-03-24T22:02:45.777 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:45 smithi142 conmon[20125]: mon.a (mon.0) 1633 : audit [INF] from='client.35504 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T22:02:45.777 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:45 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:02:45.777 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:45 smithi142 conmon[20125]: T22:02:44.341037+0000 mon.a (mon.0) 1634 : cluster 2021-03-24T22:02:45.777 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:45 smithi142 conmon[20125]: [DBG] osdmap e594: 8 total, 8 up, 8 in 2021-03-24T22:02:45.778 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:45 smithi142 conmon[20125]: cluster 2021-03-24T22:02:45.778 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:45 smithi142 conmon[20125]: 2021-03-24T22:02:45. 2021-03-24T22:02:45.778 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:45 smithi142 conmon[20125]: 309124+0000 mon.a (mon.0) 1635 : cluster 2021-03-24T22:02:45.778 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:45 smithi142 conmon[20125]: [DBG] osdmap e595: 8 total, 8 up, 8 in 2021-03-24T22:02:45.780 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:45.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:45.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:02:45.886 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:45.939 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:45.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:45.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:02:46.045 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:46.106 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:46.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:46.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:02:46.211 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:46.274 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:46.279 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.5 dump_blocked_ops 2021-03-24T22:02:46.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:46.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:02:46.320 INFO:tasks.rados.rados.0.smithi125.stdout:2242: done (0 left) 2021-03-24T22:02:46.320 INFO:tasks.rados.rados.0.smithi125.stdout:2243: read oid 39 snap 212 2021-03-24T22:02:46.320 INFO:tasks.rados.rados.0.smithi125.stdout:2243: expect (ObjNum 665 snap 205 seq_num 665) 2021-03-24T22:02:46.321 INFO:tasks.rados.rados.0.smithi125.stdout:2244: copy_from oid 40 from oid 13 current snap is 254 2021-03-24T22:02:46.321 INFO:tasks.rados.rados.0.smithi125.stdout:2245: snap_remove snap 227 2021-03-24T22:02:46.323 INFO:teuthology.orchestra.run.smithi125.stderr:reweighted osd.1 to 1 (10000) 2021-03-24T22:02:46.382 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:46.439 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:46.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:46.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:46.544 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:46.554 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:46 smithi125 conmon[19936]: cluster 2021-03-24T22:02:44.911750+0000 mgr.y (mgr 2021-03-24T22:02:46.555 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:46 smithi125 conmon[19936]: .14144) 569 : cluster [DBG] pgmap v1054: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 525 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 4.6 MiB/s wr, 30 op/s; 72/1695 objects misplaced (4.248%) 2021-03-24T22:02:46.555 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:46 smithi125 conmon[19936]: audit 2021-03-24T22:02:45.738155+0000 mon.c (mon.2) 91 : audit [INF] from='client.? v1:172.21.15.125:0/3167039915' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2021-03-24T22:02:46.555 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:46 smithi125 conmon[19936]: audit 2021-03-24T22:02:46.556 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:46 smithi125 conmon[19936]: 2021-03-24T22:02:45.738570+0000 mon.a (mon.0) 1636 : audit [INF] from='client.35510 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2021-03-24T22:02:46.556 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:46 smithi125 conmon[19936]: audit 2021-03-24T22:02:46.315194 2021-03-24T22:02:46.556 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:46 smithi125 conmon[19936]: +0000 mon.a (mon.0) 1637 : audit [INF] from='client.35510 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2021-03-24T22:02:46.556 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:46 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:02:46.557 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:46 smithi125 conmon[19936]: 02:46.315680+0000 mon.a (mon.0) 1638 : cluster [DBG] 2021-03-24T22:02:46.557 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:46 smithi125 conmon[19936]: osdmap e596: 8 total, 8 up, 8 in 2021-03-24T22:02:46.558 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:46 smithi125 conmon[24927]: cluster 2021-03-24T22:02: 2021-03-24T22:02:46.558 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:46 smithi125 conmon[24927]: 44.911750+0000 mgr.y (mgr.14144) 569 : cluster [DBG] pgmap v1054: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 525 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 4.6 MiB/s wr, 30 op/s; 72/1695 objects misplaced (4.248%) 2021-03-24T22:02:46.558 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:46 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T22:02:46.558 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:46 smithi125 conmon[24927]: 24T22:02:45.738155+0000 mon.c (mon.2) 2021-03-24T22:02:46.558 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:46 smithi125 conmon[24927]: 91 : audit [INF] from='client.? v1:172.21.15.125:0/3167039915' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2021-03-24T22:02:46.559 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:46 smithi125 conmon[24927]: audit 2021- 2021-03-24T22:02:46.559 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:46 smithi125 conmon[24927]: 03-24T22:02:45.738570+0000 mon.a (mon.0) 2021-03-24T22:02:46.559 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:46 smithi125 conmon[24927]: 1636 : audit [INF] from='client.35510 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2021-03-24T22:02:46.559 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:46 smithi125 conmon[24927]: audit 2021 2021-03-24T22:02:46.560 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:46 smithi125 conmon[24927]: -03-24T22:02:46.315194+0000 mon.a 2021-03-24T22:02:46.560 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:46 smithi125 conmon[24927]: (mon.0) 1637 : audit [INF] 2021-03-24T22:02:46.560 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:46 smithi125 conmon[24927]: from='client.35510 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2021-03-24T22:02:46.560 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:46 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:02:46.560 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:46 smithi125 conmon[24927]: 03-24T22:02:46.315680+0000 mon.a ( 2021-03-24T22:02:46.561 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:46 smithi125 conmon[24927]: mon.0) 1638 : cluster [DBG] 2021-03-24T22:02:46.561 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:46 smithi125 conmon[24927]: osdmap e596: 8 total, 8 up, 8 in 2021-03-24T22:02:46.574 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T22:02:46.594 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:46.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:46.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:46.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:46 smithi142 conmon[20125]: cluster 2021-03-24T22:02:44.911750+0000 mgr.y ( 2021-03-24T22:02:46.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:46 smithi142 conmon[20125]: mgr.14144) 569 : cluster [DBG] pgmap v1054: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 525 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 4.6 MiB/s wr, 30 op/s; 72/1695 objects misplaced (4.248%) 2021-03-24T22:02:46.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:46 smithi142 conmon[20125]: audit 2021-03-24T22:02:45.738155+0000 mon.c (mon.2) 91 : audit [INF] from='client.? v1:172.21.15.125:0/3167039915' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2021-03-24T22:02:46.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:46 smithi142 conmon[20125]: audit 2021-03-24T22:02:45.738570+0000 mon.a (mon.0) 1636 2021-03-24T22:02:46.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:46 smithi142 conmon[20125]: : audit [INF] from='client.35510 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2021-03-24T22:02:46.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:46 smithi142 conmon[20125]: audit 2021-03-24 2021-03-24T22:02:46.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:46 smithi142 conmon[20125]: T22:02:46.315194+0000 mon.a (mon.0) 1637 : 2021-03-24T22:02:46.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:46 smithi142 conmon[20125]: audit [INF] from='client.35510 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2021-03-24T22:02:46.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:46 smithi142 conmon[20125]: cluster 2021-03-24T22:02:46.315680+0000 2021-03-24T22:02:46.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:46 smithi142 conmon[20125]: mon.a (mon.0) 1638 : cluster [DBG] osdmap e596: 8 total, 8 up, 8 in 2021-03-24T22:02:46.701 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:46.761 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:46.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:46.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:02:46.866 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:46.922 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:46.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:46.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:02:47.026 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:47.085 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:47.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:47.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:02:47.191 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:47.252 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:47.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:47.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:02:47.329 INFO:tasks.rados.rados.0.smithi125.stdout:2245: done (2 left) 2021-03-24T22:02:47.330 INFO:tasks.rados.rados.0.smithi125.stdout:2246: read oid 6 snap -1 2021-03-24T22:02:47.330 INFO:tasks.rados.rados.0.smithi125.stdout:2246: expect deleted 2021-03-24T22:02:47.330 INFO:tasks.rados.rados.0.smithi125.stdout:2247: write oid 29 current snap is 254 2021-03-24T22:02:47.331 INFO:tasks.rados.rados.0.smithi125.stdout:2247: seq_num 777 ranges {659039=736479,2001450=643934} 2021-03-24T22:02:47.335 INFO:tasks.rados.rados.0.smithi125.stdout:2247: writing smithi12556470-29 from 659039 to 1395518 tid 1 2021-03-24T22:02:47.345 INFO:tasks.rados.rados.0.smithi125.stdout:2247: writing smithi12556470-29 from 2001450 to 2645384 tid 2 2021-03-24T22:02:47.345 INFO:tasks.rados.rados.0.smithi125.stdout:2248: read oid 17 snap -1 2021-03-24T22:02:47.346 INFO:tasks.rados.rados.0.smithi125.stdout:2248: expect (ObjNum 744 snap 240 seq_num 744) 2021-03-24T22:02:47.346 INFO:tasks.rados.rados.0.smithi125.stdout:2249: read oid 11 snap -1 2021-03-24T22:02:47.346 INFO:tasks.rados.rados.0.smithi125.stdout:2249: expect (ObjNum 495 snap 142 seq_num 495) 2021-03-24T22:02:47.346 INFO:tasks.rados.rados.0.smithi125.stdout:2250: read oid 22 snap -1 2021-03-24T22:02:47.347 INFO:tasks.rados.rados.0.smithi125.stdout:2250: expect (ObjNum 736 snap 235 seq_num 736) 2021-03-24T22:02:47.347 INFO:tasks.rados.rados.0.smithi125.stdout:2251: rollback oid 3 current snap is 254 2021-03-24T22:02:47.347 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 3 to 231 2021-03-24T22:02:47.347 INFO:tasks.rados.rados.0.smithi125.stdout:2243: expect (ObjNum 665 snap 205 seq_num 665) 2021-03-24T22:02:47.358 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:47.400 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:47.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:47.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:47.506 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:47.534 INFO:tasks.rados.rados.0.smithi125.stdout:2244: finishing copy_from to smithi12556470-40 2021-03-24T22:02:47.534 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 40 v 788 (ObjNum 761 snap 246 seq_num 761) dirty exists 2021-03-24T22:02:47.535 INFO:tasks.rados.rados.0.smithi125.stdout:2244: finishing copy_from racing read to smithi12556470-40 2021-03-24T22:02:47.535 INFO:tasks.rados.rados.0.smithi125.stdout:2251: finishing rollback tid 1 to smithi12556470-3 2021-03-24T22:02:47.535 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 3 v 307 (ObjNum -986354656 snap 32741 seq_num 1116325488) dirty dne 2021-03-24T22:02:47.535 INFO:tasks.rados.rados.0.smithi125.stdout:2249: expect (ObjNum 495 snap 142 seq_num 495) 2021-03-24T22:02:47.556 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:47.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:47.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:47.661 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:47.684 INFO:tasks.rados.rados.0.smithi125.stdout:2250: expect (ObjNum 736 snap 235 seq_num 736) 2021-03-24T22:02:47.719 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:47.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:47.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:02:47.755 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd reweight 6 1 2021-03-24T22:02:47.784 INFO:tasks.rados.rados.0.smithi125.stdout:2247: finishing write tid 1 to smithi12556470-29 2021-03-24T22:02:47.785 INFO:tasks.rados.rados.0.smithi125.stdout:2248: expect (ObjNum 744 snap 240 seq_num 744) 2021-03-24T22:02:47.826 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:47.887 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:47.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:47.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:02:47.994 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:47.996 INFO:tasks.rados.rados.0.smithi125.stdout:2247: finishing write tid 2 to smithi12556470-29 2021-03-24T22:02:47.996 INFO:tasks.rados.rados.0.smithi125.stdout:2247: finishing write tid 3 to smithi12556470-29 2021-03-24T22:02:47.997 INFO:tasks.rados.rados.0.smithi125.stdout:2243: done (7 left) 2021-03-24T22:02:47.997 INFO:tasks.rados.rados.0.smithi125.stdout:2244: done (6 left) 2021-03-24T22:02:47.998 INFO:tasks.rados.rados.0.smithi125.stdout:2246: done (5 left) 2021-03-24T22:02:47.998 INFO:tasks.rados.rados.0.smithi125.stdout:2248: done (4 left) 2021-03-24T22:02:47.999 INFO:tasks.rados.rados.0.smithi125.stdout:2249: done (3 left) 2021-03-24T22:02:47.999 INFO:tasks.rados.rados.0.smithi125.stdout:2250: done (2 left) 2021-03-24T22:02:48.000 INFO:tasks.rados.rados.0.smithi125.stdout:2251: done (1 left) 2021-03-24T22:02:48.000 INFO:tasks.rados.rados.0.smithi125.stdout:2252: snap_create 2021-03-24T22:02:48.000 INFO:tasks.rados.rados.0.smithi125.stdout:2247: finishing write tid 4 to smithi12556470-29 2021-03-24T22:02:48.001 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 29 v 345 (ObjNum 777 snap 254 seq_num 777) dirty exists 2021-03-24T22:02:48.001 INFO:tasks.rados.rados.0.smithi125.stdout:2247: left oid 29 (ObjNum 777 snap 254 seq_num 777) 2021-03-24T22:02:48.034 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:02:48.040 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T22:02:48.059 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T22:02:48.060 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:48.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:48.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:02:48.091 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T22:02:48.115 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T22:02:48.143 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T22:02:48.165 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:48.166 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T22:02:48.187 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T22:02:48.214 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:48.217 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T22:02:48.219 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-24T22:02:48.219 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-24T22:02:48.219 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-24T22:02:48.219 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-24T22:02:48.220 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-24T22:02:48.220 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-24T22:02:48.220 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-24T22:02:48.221 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-24T22:02:48.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:48.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:02:48.324 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:48.363 INFO:tasks.rados.rados.0.smithi125.stdout:2247: done (1 left) 2021-03-24T22:02:48.363 INFO:tasks.rados.rados.0.smithi125.stdout:2252: done (0 left) 2021-03-24T22:02:48.364 INFO:tasks.rados.rados.0.smithi125.stdout:2253: snap_remove snap 245 2021-03-24T22:02:48.385 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:48.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:48.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:02:48.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:48 smithi125 conmon[19936]: cluster 2021-03-24T22:02:46.912833+0000 mgr.y ( 2021-03-24T22:02:48.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:48 smithi125 conmon[19936]: mgr.14144) 570 : cluster [DBG] pgmap v1057: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 535 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 20 MiB/s rd, 6.1 MiB/s wr, 37 op/s; 72/1716 objects misplaced (4.196%) 2021-03-24T22:02:48.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:48 smithi125 conmon[19936]: cluster 2021-03-24T22:02:47.322842+0000 mon.a (mon.0) 1639 : cluster [DBG] osdmap e597: 8 total, 8 up, 8 in 2021-03-24T22:02:48.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:48 smithi125 conmon[19936]: audit 2021-03-24T22:02:48.014576+0000 mon.a (mon.0) 1640 : audit [DBG] 2021-03-24T22:02:48.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:48 smithi125 conmon[19936]: from='client.? v1:172.21.15.125:0/1126550245' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:02:48.485 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:48 smithi125 conmon[24927]: cluster 2021-03-24T22:02:46.912833+0000 mgr.y (mgr 2021-03-24T22:02:48.485 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:48 smithi125 conmon[24927]: .14144) 570 : cluster [DBG] pgmap v1057: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 535 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 20 MiB/s rd, 6.1 MiB/s wr, 37 op/s; 72/1716 objects misplaced (4.196%) 2021-03-24T22:02:48.486 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:48 smithi125 conmon[24927]: cluster 2021-03-24T22:02 2021-03-24T22:02:48.486 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:48 smithi125 conmon[24927]: :47.322842+0000 mon.a (mon.0) 1639 : cluster [DBG] osdmap e597: 8 total, 8 up, 8 in 2021-03-24T22:02:48.486 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:48 smithi125 conmon[24927]: audit 2021-03-24T22:02:48.486 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:48 smithi125 conmon[24927]: 2021-03-24T22:02:48.014576+0000 mon.a (mon.0) 1640 2021-03-24T22:02:48.487 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:48 smithi125 conmon[24927]: : audit [DBG] from='client.? v1:172.21.15.125:0/1126550245' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:02:48.490 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:48.544 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:48.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:48.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:48.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:48 smithi142 conmon[20125]: cluster 2021-03-24T22:02:46 2021-03-24T22:02:48.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:48 smithi142 conmon[20125]: .912833+0000 mgr.y (mgr.14144) 570 : cluster [DBG] pgmap v1057: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 535 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 20 MiB/s rd, 6.1 MiB/s wr, 37 op/s; 72/1716 objects misplaced (4.196%) 2021-03-24T22:02:48.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:48 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:02:48.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:48 smithi142 conmon[20125]: 22:02:47.322842+0000 mon.a (mon.0) 1639 : cluster [DBG] 2021-03-24T22:02:48.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:48 smithi142 conmon[20125]: osdmap e597: 8 total, 8 up, 8 in 2021-03-24T22:02:48.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:48 smithi142 conmon[20125]: audit 2021-03-24T22:02:48.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:48 smithi142 conmon[20125]: 2021-03-24T22:02:48.014576+0000 2021-03-24T22:02:48.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:48 smithi142 conmon[20125]: mon.a (mon.0) 1640 : audit [DBG] from='client.? v1:172.21.15.125:0/1126550245' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:02:48.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:48 smithi142 conmon[20125]: 2021-03-24T22:02:48.625 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2021-03-24T22:02:48.625 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T22:02:48.650 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:48.703 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:48.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:48.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:02:48.810 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:48.869 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:48.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:48.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:02:48.974 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:49.035 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:49.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:49.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:02:49.139 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:49.193 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:49.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:49.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:02:49.298 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:49.350 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:49.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:49.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:02:49.374 INFO:tasks.rados.rados.0.smithi125.stdout:2253: done (0 left) 2021-03-24T22:02:49.374 INFO:tasks.rados.rados.0.smithi125.stdout:2254: snap_remove snap 231 2021-03-24T22:02:49.393 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:49 smithi125 conmon[19936]: cluster 2021-03-24T22:02:48.360135+0000 2021-03-24T22:02:49.393 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:49 smithi125 conmon[19936]: mon.a (mon.0) 1641 : cluster [DBG] osdmap e598: 8 total, 8 up, 8 in 2021-03-24T22:02:49.394 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:49 smithi125 conmon[24927]: cluster 2021-03-24T22 2021-03-24T22:02:49.394 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:49 smithi125 conmon[24927]: :02:48.360135+0000 mon.a (mon.0) 1641 : cluster [DBG] osdmap e598: 8 total, 8 up, 8 in 2021-03-24T22:02:49.456 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:49.548 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd unset noscrub 2021-03-24T22:02:49.559 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:49.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:49.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:02:49.655 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:49 smithi142 conmon[20125]: cluster 2021-03-24T22:02:48.360135+0000 mon.a (mon.0) 1641 : cluster [DBG] osdmap e598: 8 total, 8 up, 8 in 2021-03-24T22:02:49.666 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:49.690 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.5 dump_historic_ops 2021-03-24T22:02:49.721 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:49.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:49.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:02:49.826 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:49.874 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:49.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:49.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:02:49.979 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:49.989 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T22:02:50.043 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:50.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:50.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:02:50.148 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:50.207 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:50.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:50.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:02:50.312 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:50.322 INFO:tasks.rados.rados.0.smithi125.stdout:2254: done (0 left) 2021-03-24T22:02:50.322 INFO:tasks.rados.rados.0.smithi125.stdout:2255: write (excl) oid 41 current snap is 255 2021-03-24T22:02:50.322 INFO:tasks.rados.rados.0.smithi125.stdout:2255: seq_num 778 ranges {675846=411728,1543579=787592,2848853=226993} 2021-03-24T22:02:50.326 INFO:tasks.rados.rados.0.smithi125.stdout:2255: writing smithi12556470-41 from 675846 to 1087574 tid 1 2021-03-24T22:02:50.336 INFO:tasks.rados.rados.0.smithi125.stdout:2255: writing smithi12556470-41 from 1543579 to 2331171 tid 2 2021-03-24T22:02:50.342 INFO:tasks.rados.rados.0.smithi125.stdout:2255: writing smithi12556470-41 from 2848853 to 3075846 tid 3 2021-03-24T22:02:50.342 INFO:tasks.rados.rados.0.smithi125.stdout:2256: read oid 31 snap -1 2021-03-24T22:02:50.342 INFO:tasks.rados.rados.0.smithi125.stdout:2256: expect (ObjNum 718 snap 229 seq_num 718) 2021-03-24T22:02:50.342 INFO:tasks.rados.rados.0.smithi125.stdout:2257: write oid 24 current snap is 255 2021-03-24T22:02:50.343 INFO:tasks.rados.rados.0.smithi125.stdout:2255: finishing write tid 1 to smithi12556470-41 2021-03-24T22:02:50.343 INFO:tasks.rados.rados.0.smithi125.stdout:2257: seq_num 779 ranges {692653=486977,1602061=647603,2992629=100024} 2021-03-24T22:02:50.349 INFO:tasks.rados.rados.0.smithi125.stdout:2257: writing smithi12556470-24 from 692653 to 1179630 tid 1 2021-03-24T22:02:50.356 INFO:tasks.rados.rados.0.smithi125.stdout:2257: writing smithi12556470-24 from 1602061 to 2249664 tid 2 2021-03-24T22:02:50.362 INFO:tasks.rados.rados.0.smithi125.stdout:2257: writing smithi12556470-24 from 2992629 to 3092653 tid 3 2021-03-24T22:02:50.362 INFO:tasks.rados.rados.0.smithi125.stdout:2258: read oid 13 snap -1 2021-03-24T22:02:50.362 INFO:tasks.rados.rados.0.smithi125.stdout:2258: expect (ObjNum 761 snap 246 seq_num 761) 2021-03-24T22:02:50.362 INFO:tasks.rados.rados.0.smithi125.stdout:2259: read oid 37 snap 198 2021-03-24T22:02:50.363 INFO:tasks.rados.rados.0.smithi125.stdout:2259: expect (ObjNum 635 snap 196 seq_num 635) 2021-03-24T22:02:50.363 INFO:tasks.rados.rados.0.smithi125.stdout:2260: snap_remove snap 212 2021-03-24T22:02:50.370 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:50.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:50.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:02:50.464 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:02:50.474 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:50.527 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:50.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:50.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:02:50.631 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:50.688 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:50.691 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:50 smithi142 conmon[20125]: cluster 2021-03-24T22:02:48.913539+0000 mgr.y (mgr.14144) 571 : cluster [DBG] pgmap v1060: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 535 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.2 MiB/s rd, 2.4 MiB/s wr, 8 op/s; 72/1716 objects misplaced (4.196%) 2021-03-24T22:02:50.691 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:50 smithi142 conmon[20125]: cluster 2021-03-24T22:02:49.367150+0000 mon.a (mon.0) 1642 : cluster [DBG] osdmap e599: 8 total, 8 up, 8 in 2021-03-24T22:02:50.692 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:50 smithi142 conmon[20125]: cluster 2021-03-24T22:02:50.316400+0000 mon.a (mon.0) 1643 : cluster [DBG] osdmap e600: 8 total, 8 up, 8 in 2021-03-24T22:02:50.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:50.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:02:50.728 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:50 smithi125 conmon[24927]: cluster 2021-03-24T22:02:50.728 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:50 smithi125 conmon[24927]: 2021-03-24T22:02:50.728 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:50 smithi125 conmon[24927]: 2021 2021-03-24T22:02:50.729 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:50 smithi125 conmon[24927]: - 2021-03-24T22:02:50.729 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:50 smithi125 conmon[24927]: 03 2021-03-24T22:02:50.729 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:50 smithi125 conmon[24927]: - 2021-03-24T22:02:50.729 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:50 smithi125 conmon[24927]: 24 2021-03-24T22:02:50.730 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:50 smithi125 conmon[24927]: T 2021-03-24T22:02:50.730 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:50 smithi125 conmon[24927]: 22 2021-03-24T22:02:50.730 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:50 smithi125 conmon[24927]: :02:48.913539+0000 2021-03-24T22:02:50.730 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:50 smithi125 conmon[24927]: mgr.y (mgr.14144) 571 : cluster [DBG] pgmap v1060: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 535 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.2 MiB/s rd, 2.4 MiB/s wr, 8 op/s; 72/1716 objects misplaced (4.196%) 2021-03-24T22:02:50.730 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:50 smithi125 conmon[24927]: cluster 2021-03-24T22:02:50.731 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:50 smithi125 conmon[24927]: 2021-03-24T22:02:50.731 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:50 smithi125 conmon[24927]: 2021-03-24T22:02 2021-03-24T22:02:50.731 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:50 smithi125 conmon[24927]: :49.367150+0000 mon.a (mon.0) 1642 : cluster [DBG] osdmap e599: 8 total, 8 up, 8 in 2021-03-24T22:02:50.731 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:50 smithi125 conmon[24927]: cluster 2021-03-24T22:02:50.732 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:50 smithi125 conmon[24927]: 2021-03- 2021-03-24T22:02:50.732 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:50 smithi125 conmon[24927]: 24T22:02:50.316400+0000 2021-03-24T22:02:50.732 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:50 smithi125 conmon[24927]: mon.a (mon.0) 1643 : cluster [DBG] osdmap e600: 8 total, 8 up, 8 in 2021-03-24T22:02:50.732 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:50 smithi125 conmon[19936]: cluster 2021-03-24T22:02: 2021-03-24T22:02:50.733 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:50 smithi125 conmon[19936]: 48.913539+0000 mgr.y (mgr.14144) 571 : cluster [DBG] pgmap v1060: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 535 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.2 MiB/s rd, 2.4 MiB/s wr, 8 op/s; 72/1716 objects misplaced (4.196%) 2021-03-24T22:02:50.733 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:50 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:02:50.733 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:50 smithi125 conmon[19936]: T22:02:49.367150+0000 mon.a (mon.0) 1642 : cluster [DBG] osdmap e599: 8 total, 8 up, 8 in 2021-03-24T22:02:50.733 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:50 smithi125 conmon[19936]: cluster 2021-03-24T22:02 2021-03-24T22:02:50.734 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:50 smithi125 conmon[19936]: :50. 2021-03-24T22:02:50.734 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:50 smithi125 conmon[19936]: 316400+0000 mon.a (mon.0) 1643 : cluster [DBG] 2021-03-24T22:02:50.734 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:50 smithi125 conmon[19936]: osdmap e600: 8 total, 8 up, 8 in 2021-03-24T22:02:50.794 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:50.850 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:50.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:50.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:02:50.957 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:51.010 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:51.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:51.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:51.116 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:51.173 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:51.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:51.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:51.279 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:51.333 INFO:tasks.rados.rados.0.smithi125.stdout:2260: done (5 left) 2021-03-24T22:02:51.334 INFO:tasks.rados.rados.0.smithi125.stdout:2261: snap_remove snap 250 2021-03-24T22:02:51.357 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:51.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:51.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:51.459 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-24T22:02:51.462 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:51.514 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:51.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:51.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:02:51.621 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:51.682 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:51.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:51.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:02:51.788 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:51.844 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:51.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:51.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:02:51.951 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:52.006 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:52.007 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2021-03-24T22:02:52.007 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T22:02:52.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:52.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:02:52.113 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:52.169 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:02:52.173 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:52.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:52.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:02:52.277 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:52.340 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:52.343 INFO:tasks.rados.rados.0.smithi125.stdout:2261: done (5 left) 2021-03-24T22:02:52.344 INFO:tasks.rados.rados.0.smithi125.stdout:2262: read oid 47 snap 207 2021-03-24T22:02:52.344 INFO:tasks.rados.rados.0.smithi125.stdout:2262: expect (ObjNum 466 snap 131 seq_num 466) 2021-03-24T22:02:52.344 INFO:tasks.rados.rados.0.smithi125.stdout:2263: snap_create 2021-03-24T22:02:52.344 INFO:tasks.rados.rados.0.smithi125.stdout:2256: expect (ObjNum 718 snap 229 seq_num 718) 2021-03-24T22:02:52.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:52.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:02:52.355 INFO:teuthology.orchestra.run.smithi125.stderr:reweighted osd.6 to 1 (10000) 2021-03-24T22:02:52.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:52 smithi125 conmon[24927]: cluster 2021-03-24T22:02:50.914272+0000 mgr.y (mgr 2021-03-24T22:02:52.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:52 smithi125 conmon[24927]: .14144) 572 : cluster [DBG] pgmap v1063: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 545 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 1.6 MiB/s wr, 15 op/s; 72/1731 objects misplaced (4.159%) 2021-03-24T22:02:52.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:52 smithi125 conmon[24927]: cluster 2021-03-24T22:02:51.327462+0000 2021-03-24T22:02:52.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:52 smithi125 conmon[24927]: mon.a (mon.0) 1644 : cluster [DBG] osdmap e601: 8 total, 8 up, 8 in 2021-03-24T22:02:52.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:52 smithi125 conmon[24927]: audit 2021-03-24T22:02:52 2021-03-24T22:02:52.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:52 smithi125 conmon[24927]: .258735+0000 mon.c (mon.2) 92 : audit [INF] from='client.? v1:172.21.15.125:0/2997205844' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-03-24T22:02:52.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:52 smithi125 conmon[24927]: audit 2021-03 2021-03-24T22:02:52.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:52 smithi125 conmon[24927]: -24T22:02:52.259188+0000 mon.a (mon.0) 2021-03-24T22:02:52.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:52 smithi125 conmon[24927]: 1645 : audit [INF] from='client.35522 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-03-24T22:02:52.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:52 smithi125 conmon[19936]: cluster 2021-03-24T22:02:50 2021-03-24T22:02:52.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:52 smithi125 conmon[19936]: .914272+0000 mgr.y (mgr.14144) 572 : cluster [DBG] pgmap v1063: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 545 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 1.6 MiB/s wr, 15 op/s; 72/1731 objects misplaced (4.159%) 2021-03-24T22:02:52.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:52 smithi125 conmon[19936]: cluster 2021-03-24T22:02:51.327462 2021-03-24T22:02:52.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:52 smithi125 conmon[19936]: +0000 mon.a (mon.0) 1644 : cluster [DBG] osdmap e601: 8 total, 8 up, 8 in 2021-03-24T22:02:52.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:52 smithi125 conmon[19936]: audit 2021- 2021-03-24T22:02:52.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:52 smithi125 conmon[19936]: 03-24T22:02:52.258735 2021-03-24T22:02:52.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:52 smithi125 conmon[19936]: +0000 mon.c (mon.2) 92 : 2021-03-24T22:02:52.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:52 smithi125 conmon[19936]: audit [INF] from='client.? v1:172.21.15.125:0/2997205844' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-03-24T22:02:52.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:52 smithi125 conmon[19936]: audit 2021- 2021-03-24T22:02:52.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:52 smithi125 conmon[19936]: 03-24T22:02: 2021-03-24T22:02:52.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:52 smithi125 conmon[19936]: 52.259188+0000 mon.a (mon 2021-03-24T22:02:52.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:52 smithi125 conmon[19936]: .0) 1645 : audit [INF] 2021-03-24T22:02:52.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:52 smithi125 conmon[19936]: from='client.35522 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-03-24T22:02:52.426 INFO:tasks.rados.rados.0.smithi125.stdout:2255: finishing write tid 2 to smithi12556470-41 2021-03-24T22:02:52.426 INFO:tasks.rados.rados.0.smithi125.stdout:2255: finishing write tid 3 to smithi12556470-41 2021-03-24T22:02:52.426 INFO:tasks.rados.rados.0.smithi125.stdout:2255: finishing write tid 4 to smithi12556470-41 2021-03-24T22:02:52.426 INFO:tasks.rados.rados.0.smithi125.stdout:2257: finishing write tid 1 to smithi12556470-24 2021-03-24T22:02:52.427 INFO:tasks.rados.rados.0.smithi125.stdout:2255: finishing write tid 5 to smithi12556470-41 2021-03-24T22:02:52.427 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 41 v 771 (ObjNum 778 snap 255 seq_num 778) dirty exists 2021-03-24T22:02:52.427 INFO:tasks.rados.rados.0.smithi125.stdout:2255: left oid 41 (ObjNum 778 snap 255 seq_num 778) 2021-03-24T22:02:52.427 INFO:tasks.rados.rados.0.smithi125.stdout:2257: finishing write tid 2 to smithi12556470-24 2021-03-24T22:02:52.428 INFO:tasks.rados.rados.0.smithi125.stdout:2259: expect (ObjNum 635 snap 196 seq_num 635) 2021-03-24T22:02:52.447 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:52.491 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:52.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:52.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:02:52.596 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:52.601 INFO:tasks.rados.rados.0.smithi125.stdout:2258: expect (ObjNum 761 snap 246 seq_num 761) 2021-03-24T22:02:52.610 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:52 smithi142 conmon[20125]: cluster 2021-03-24T22:02:50.914272+0000 mgr.y (mgr.14144) 572 : cluster [DBG] pgmap v1063: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 545 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 1.6 MiB/s wr, 15 op/s; 72/1731 objects misplaced (4.159%) 2021-03-24T22:02:52.611 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:52 smithi142 conmon[20125]: cluster 2021-03-24T22:02:51.327462+0000 mon.a (mon.0) 1644 : cluster [DBG] osdmap e601: 8 total, 8 up, 8 in 2021-03-24T22:02:52.611 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:52 smithi142 conmon[20125]: audit 2021-03-24T22:02:52.258735+0000 mon.c (mon.2) 92 : audit [INF] from='client.? v1:172.21.15.125:0/2997205844' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-03-24T22:02:52.611 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:52 smithi142 conmon[20125]: audit 2021-03-24T22: 2021-03-24T22:02:52.611 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:52 smithi142 conmon[20125]: 02:52.259188+0000 mon.a (mon.0) 1645 : audit [INF] from='client.35522 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-03-24T22:02:52.650 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:52.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:52.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:02:52.692 INFO:tasks.rados.rados.0.smithi125.stdout:2257: finishing write tid 3 to smithi12556470-24 2021-03-24T22:02:52.692 INFO:tasks.rados.rados.0.smithi125.stdout:2257: finishing write tid 4 to smithi12556470-24 2021-03-24T22:02:52.693 INFO:tasks.rados.rados.0.smithi125.stdout:2257: finishing write tid 5 to smithi12556470-24 2021-03-24T22:02:52.693 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 24 v 857 (ObjNum 779 snap 255 seq_num 779) dirty exists 2021-03-24T22:02:52.693 INFO:tasks.rados.rados.0.smithi125.stdout:2257: left oid 24 (ObjNum 779 snap 255 seq_num 779) 2021-03-24T22:02:52.693 INFO:tasks.rados.rados.0.smithi125.stdout:2262: expect (ObjNum 466 snap 131 seq_num 466) 2021-03-24T22:02:52.757 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:52.814 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:52.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:52.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:02:52.920 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:52.972 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.0 dump_ops_in_flight 2021-03-24T22:02:52.980 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:52.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:52.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:02:53.086 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:53.146 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:53.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:53.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:02:53.253 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:53.308 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:53.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:53.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:02:53.344 INFO:tasks.rados.rados.0.smithi125.stdout:2255: done (6 left) 2021-03-24T22:02:53.344 INFO:tasks.rados.rados.0.smithi125.stdout:2256: done (5 left) 2021-03-24T22:02:53.344 INFO:tasks.rados.rados.0.smithi125.stdout:2257: done (4 left) 2021-03-24T22:02:53.345 INFO:tasks.rados.rados.0.smithi125.stdout:2258: done (3 left) 2021-03-24T22:02:53.345 INFO:tasks.rados.rados.0.smithi125.stdout:2259: done (2 left) 2021-03-24T22:02:53.345 INFO:tasks.rados.rados.0.smithi125.stdout:2262: done (1 left) 2021-03-24T22:02:53.345 INFO:tasks.rados.rados.0.smithi125.stdout:2263: done (0 left) 2021-03-24T22:02:53.345 INFO:tasks.rados.rados.0.smithi125.stdout:2264: read oid 40 snap 192 2021-03-24T22:02:53.346 INFO:tasks.rados.rados.0.smithi125.stdout:2264: expect deleted 2021-03-24T22:02:53.346 INFO:tasks.rados.rados.0.smithi125.stdout:2265: read oid 6 snap -1 2021-03-24T22:02:53.346 INFO:tasks.rados.rados.0.smithi125.stdout:2265: expect deleted 2021-03-24T22:02:53.346 INFO:tasks.rados.rados.0.smithi125.stdout:2266: delete oid 46 current snap is 256 2021-03-24T22:02:53.353 INFO:tasks.rados.rados.0.smithi125.stdout:2264: done (2 left) 2021-03-24T22:02:53.353 INFO:tasks.rados.rados.0.smithi125.stdout:2265: done (1 left) 2021-03-24T22:02:53.354 INFO:tasks.rados.rados.0.smithi125.stdout:2266: done (0 left) 2021-03-24T22:02:53.354 INFO:tasks.rados.rados.0.smithi125.stdout:2267: delete oid 16 current snap is 256 2021-03-24T22:02:53.359 INFO:tasks.rados.rados.0.smithi125.stdout:2267: done (0 left) 2021-03-24T22:02:53.359 INFO:tasks.rados.rados.0.smithi125.stdout:2268: read oid 11 snap -1 2021-03-24T22:02:53.359 INFO:tasks.rados.rados.0.smithi125.stdout:2268: expect (ObjNum 495 snap 142 seq_num 495) 2021-03-24T22:02:53.360 INFO:tasks.rados.rados.0.smithi125.stdout:2269: read oid 38 snap -1 2021-03-24T22:02:53.360 INFO:tasks.rados.rados.0.smithi125.stdout:2269: expect (ObjNum 733 snap 235 seq_num 733) 2021-03-24T22:02:53.360 INFO:tasks.rados.rados.0.smithi125.stdout:2270: delete oid 6 current snap is 256 2021-03-24T22:02:53.361 INFO:tasks.rados.rados.0.smithi125.stdout:2270: done (2 left) 2021-03-24T22:02:53.361 INFO:tasks.rados.rados.0.smithi125.stdout:2271: rollback oid 28 current snap is 256 2021-03-24T22:02:53.361 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 28 to 235 2021-03-24T22:02:53.362 INFO:tasks.rados.rados.0.smithi125.stdout:2272: write (excl) oid 8 current snap is 256 2021-03-24T22:02:53.362 INFO:tasks.rados.rados.0.smithi125.stdout:2272: seq_num 783 ranges {759881=787973,2119636=654941} 2021-03-24T22:02:53.371 INFO:tasks.rados.rados.0.smithi125.stdout:2272: writing smithi12556470-8 from 759881 to 1547854 tid 1 2021-03-24T22:02:53.380 INFO:tasks.rados.rados.0.smithi125.stdout:2272: writing smithi12556470-8 from 2119636 to 2774577 tid 2 2021-03-24T22:02:53.380 INFO:tasks.rados.rados.0.smithi125.stdout:2273: snap_remove snap 181 2021-03-24T22:02:53.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:53 smithi125 conmon[19936]: audit 2021-03-24T22:02:52.338855+0000 2021-03-24T22:02:53.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:53 smithi125 conmon[19936]: mon.a (mon.0) 1646 : audit [INF] from='client.35522 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2021-03-24T22:02:53.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:53 smithi125 conmon[19936]: cluster 2021-03-24T22:02:52.338964+0000 mon.a (mon.0) 1647 : cluster [DBG] osdmap e602: 8 total, 8 up, 8 in 2021-03-24T22:02:53.413 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:53 smithi125 conmon[19936]: 2021-03-24T22:02:53.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:53 smithi125 conmon[24927]: audit 2021-03-24T22:02:53.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:53 smithi125 conmon[24927]: 2021-03-24T 2021-03-24T22:02:53.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:53 smithi125 conmon[24927]: 22:02:52.338855+0000 mon.a (mon.0) 1646 : audit [INF] from='client.35522 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2021-03-24T22:02:53.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:53 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:02:53.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:53 smithi125 conmon[24927]: 03-24T22:02:52.338964+0000 mon.a (mon. 2021-03-24T22:02:53.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:53 smithi125 conmon[24927]: 0) 1647 : cluster [DBG] osdmap e602: 8 total, 8 up, 8 in 2021-03-24T22:02:53.415 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:53.470 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:53.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:53.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:02:53.577 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:53.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:53 smithi142 conmon[20125]: audit 2021-03-24T22:02:53.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:53 smithi142 conmon[20125]: 2021-03-24 2021-03-24T22:02:53.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:53 smithi142 conmon[20125]: T22:02:52.338855+0000 mon.a (mon.0) 1646 : audit [INF] from='client.35522 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2021-03-24T22:02:53.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:53 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:02:53.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:53 smithi142 conmon[20125]: 24T22:02:52.338964+0000 mon.a (mon.0) 1647 2021-03-24T22:02:53.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:53 smithi142 conmon[20125]: : cluster [DBG] osdmap e602: 8 total, 8 up, 8 in 2021-03-24T22:02:53.635 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:53.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:53.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:02:53.742 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:53.805 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:53.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:53.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:02:53.910 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:53.963 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:53.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:53.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:54.067 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:54.124 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:54.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:54.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:02:54.231 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:54.277 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:02:54.285 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:54.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:54.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:02:54.296 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd reweight 7 1 2021-03-24T22:02:54.362 INFO:tasks.rados.rados.0.smithi125.stdout:2273: done (4 left) 2021-03-24T22:02:54.363 INFO:tasks.rados.rados.0.smithi125.stdout:2274: read oid 22 snap -1 2021-03-24T22:02:54.363 INFO:tasks.rados.rados.0.smithi125.stdout:2274: expect (ObjNum 736 snap 235 seq_num 736) 2021-03-24T22:02:54.363 INFO:tasks.rados.rados.0.smithi125.stdout:2275: write oid 44 current snap is 256 2021-03-24T22:02:54.364 INFO:tasks.rados.rados.0.smithi125.stdout:2275: seq_num 784 ranges {776688=579575,1894471=798599} 2021-03-24T22:02:54.371 INFO:tasks.rados.rados.0.smithi125.stdout:2275: writing smithi12556470-44 from 776688 to 1356263 tid 1 2021-03-24T22:02:54.381 INFO:tasks.rados.rados.0.smithi125.stdout:2275: writing smithi12556470-44 from 1894471 to 2693070 tid 2 2021-03-24T22:02:54.381 INFO:tasks.rados.rados.0.smithi125.stdout:2269: expect (ObjNum 733 snap 235 seq_num 733) 2021-03-24T22:02:54.392 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:54.439 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:54.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:54.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:02:54.463 INFO:tasks.rados.rados.0.smithi125.stdout:2268: expect (ObjNum 495 snap 142 seq_num 495) 2021-03-24T22:02:54.518 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:54 smithi125 conmon[19936]: cluster 2021-03-24T22:02:52.914977 2021-03-24T22:02:54.519 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:54 smithi125 conmon[19936]: +0000 mgr.y (mgr.14144) 573 : cluster [DBG] pgmap v1066: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 542 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 1.9 MiB/s wr, 21 op/s; 72/1725 objects misplaced (4.174%) 2021-03-24T22:02:54.519 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:54 smithi125 conmon[19936]: cluster 2021-03-24T22:02:53.340451 2021-03-24T22:02:54.519 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:54 smithi125 conmon[19936]: +0000 mon.a (mon.0) 1648 : cluster [DBG] osdmap e603: 8 total, 8 up, 8 in 2021-03-24T22:02:54.520 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:54 smithi125 conmon[24927]: cluster 2021-03-24T22:02:52.914977 2021-03-24T22:02:54.520 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:54 smithi125 conmon[24927]: +0000 mgr.y (mgr.14144) 573 : cluster [DBG] pgmap v1066: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 542 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 1.9 MiB/s wr, 21 op/s; 72/1725 objects misplaced (4.174%) 2021-03-24T22:02:54.520 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:54 smithi125 conmon[24927]: cluster 2021-03-24T22:02:54.520 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:54 smithi125 conmon[24927]: 2021-03-24T22:02:53.340451+0000 mon.a (mon.0) 2021-03-24T22:02:54.521 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:54 smithi125 conmon[24927]: 1648 : cluster [DBG] osdmap e603: 8 total, 8 up, 8 in 2021-03-24T22:02:54.543 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:54.598 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:54.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:54.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:02:54.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:54 smithi142 conmon[20125]: cluster 2021-03-24T22:02 2021-03-24T22:02:54.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:54 smithi142 conmon[20125]: :52.914977+0000 mgr.y (mgr.14144) 573 : cluster [DBG] pgmap v1066: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 542 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 1.9 MiB/s wr, 21 op/s; 72/1725 objects misplaced (4.174%) 2021-03-24T22:02:54.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:54 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:02:54.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:54 smithi142 conmon[20125]: 24T22:02:53.340451+0000 mon.a (mon.0) 1648 : cluster 2021-03-24T22:02:54.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:54 smithi142 conmon[20125]: [DBG] osdmap e603: 8 total, 8 up, 8 in 2021-03-24T22:02:54.639 INFO:tasks.rados.rados.0.smithi125.stdout:2271: finishing rollback tid 0 to smithi12556470-28 2021-03-24T22:02:54.639 INFO:tasks.rados.rados.0.smithi125.stdout:2271: finishing rollback tid 1 to smithi12556470-28 2021-03-24T22:02:54.639 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 28 v 555 (ObjNum 1953066355 snap 842099048 seq_num 875967797) dirty dne 2021-03-24T22:02:54.639 INFO:tasks.rados.rados.0.smithi125.stdout:2272: finishing write tid 1 to smithi12556470-8 2021-03-24T22:02:54.640 INFO:tasks.rados.rados.0.smithi125.stdout:2272: finishing write tid 2 to smithi12556470-8 2021-03-24T22:02:54.640 INFO:tasks.rados.rados.0.smithi125.stdout:2272: finishing write tid 3 to smithi12556470-8 2021-03-24T22:02:54.640 INFO:tasks.rados.rados.0.smithi125.stdout:2272: finishing write tid 4 to smithi12556470-8 2021-03-24T22:02:54.640 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 8 v 205 (ObjNum 783 snap 256 seq_num 783) dirty exists 2021-03-24T22:02:54.641 INFO:tasks.rados.rados.0.smithi125.stdout:2272: left oid 8 (ObjNum 783 snap 256 seq_num 783) 2021-03-24T22:02:54.641 INFO:tasks.rados.rados.0.smithi125.stdout:2268: done (5 left) 2021-03-24T22:02:54.641 INFO:tasks.rados.rados.0.smithi125.stdout:2269: done (4 left) 2021-03-24T22:02:54.641 INFO:tasks.rados.rados.0.smithi125.stdout:2271: done (3 left) 2021-03-24T22:02:54.641 INFO:tasks.rados.rados.0.smithi125.stdout:2272: done (2 left) 2021-03-24T22:02:54.642 INFO:tasks.rados.rados.0.smithi125.stdout:2276: read oid 37 snap -1 2021-03-24T22:02:54.642 INFO:tasks.rados.rados.0.smithi125.stdout:2276: expect (ObjNum 757 snap 244 seq_num 757) 2021-03-24T22:02:54.642 INFO:tasks.rados.rados.0.smithi125.stdout:2277: snap_create 2021-03-24T22:02:54.642 INFO:tasks.rados.rados.0.smithi125.stdout:2274: expect (ObjNum 736 snap 235 seq_num 736) 2021-03-24T22:02:54.669 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:02:54.702 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:54.755 INFO:tasks.rados.rados.0.smithi125.stdout:2275: finishing write tid 1 to smithi12556470-44 2021-03-24T22:02:54.755 INFO:tasks.rados.rados.0.smithi125.stdout:2275: finishing write tid 2 to smithi12556470-44 2021-03-24T22:02:54.755 INFO:tasks.rados.rados.0.smithi125.stdout:2275: finishing write tid 3 to smithi12556470-44 2021-03-24T22:02:54.756 INFO:tasks.rados.rados.0.smithi125.stdout:2275: finishing write tid 4 to smithi12556470-44 2021-03-24T22:02:54.756 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 44 v 511 (ObjNum 784 snap 256 seq_num 784) dirty exists 2021-03-24T22:02:54.756 INFO:tasks.rados.rados.0.smithi125.stdout:2275: left oid 44 (ObjNum 784 snap 256 seq_num 784) 2021-03-24T22:02:54.756 INFO:tasks.rados.rados.0.smithi125.stdout:2276: expect (ObjNum 757 snap 244 seq_num 757) 2021-03-24T22:02:54.765 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:54.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:54.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:54.871 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:54.928 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:54.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:54.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:02:55.035 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:55.096 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:55.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:55.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:02:55.201 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:55.262 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:55.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:55.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:02:55.326 INFO:teuthology.orchestra.run.smithi125.stderr:noscrub is unset 2021-03-24T22:02:55.327 INFO:tasks.rados.rados.0.smithi125.stdout:2274: done (3 left) 2021-03-24T22:02:55.327 INFO:tasks.rados.rados.0.smithi125.stdout:2275: done (2 left) 2021-03-24T22:02:55.328 INFO:tasks.rados.rados.0.smithi125.stdout:2276: done (1 left) 2021-03-24T22:02:55.328 INFO:tasks.rados.rados.0.smithi125.stdout:2277: done (0 left) 2021-03-24T22:02:55.328 INFO:tasks.rados.rados.0.smithi125.stdout:2278: write (excl) oid 6 current snap is 257 2021-03-24T22:02:55.328 INFO:tasks.rados.rados.0.smithi125.stdout:2278: seq_num 785 ranges {793495=654824,1952953=658610,3157167=36328} 2021-03-24T22:02:55.335 INFO:tasks.rados.rados.0.smithi125.stdout:2278: writing smithi12556470-6 from 793495 to 1448319 tid 1 2021-03-24T22:02:55.343 INFO:tasks.rados.rados.0.smithi125.stdout:2278: writing smithi12556470-6 from 1952953 to 2611563 tid 2 2021-03-24T22:02:55.348 INFO:tasks.rados.rados.0.smithi125.stdout:2278: writing smithi12556470-6 from 3157167 to 3193495 tid 3 2021-03-24T22:02:55.348 INFO:tasks.rados.rados.0.smithi125.stdout:2279: read oid 27 snap -1 2021-03-24T22:02:55.348 INFO:tasks.rados.rados.0.smithi125.stdout:2279: expect (ObjNum 713 snap 227 seq_num 713) 2021-03-24T22:02:55.349 INFO:tasks.rados.rados.0.smithi125.stdout:2280: write (excl) oid 35 current snap is 257 2021-03-24T22:02:55.349 INFO:tasks.rados.rados.0.smithi125.stdout:2278: finishing write tid 1 to smithi12556470-6 2021-03-24T22:02:55.349 INFO:tasks.rados.rados.0.smithi125.stdout:2280: seq_num 786 ranges {410302=730073,1895082=402268,2951884=258418} 2021-03-24T22:02:55.354 INFO:tasks.rados.rados.0.smithi125.stdout:2280: writing smithi12556470-35 from 410302 to 1140375 tid 1 2021-03-24T22:02:55.361 INFO:tasks.rados.rados.0.smithi125.stdout:2280: writing smithi12556470-35 from 1895082 to 2297350 tid 2 2021-03-24T22:02:55.368 INFO:tasks.rados.rados.0.smithi125.stdout:2280: writing smithi12556470-35 from 2951884 to 3210302 tid 3 2021-03-24T22:02:55.368 INFO:tasks.rados.rados.0.smithi125.stdout:2281: copy_from oid 31 from oid 25 current snap is 257 2021-03-24T22:02:55.368 INFO:tasks.rados.rados.0.smithi125.stdout:2278: finishing write tid 2 to smithi12556470-6 2021-03-24T22:02:55.369 INFO:tasks.rados.rados.0.smithi125.stdout:2278: finishing write tid 3 to smithi12556470-6 2021-03-24T22:02:55.369 INFO:tasks.rados.rados.0.smithi125.stdout:2278: finishing write tid 4 to smithi12556470-6 2021-03-24T22:02:55.369 INFO:tasks.rados.rados.0.smithi125.stdout:2278: finishing write tid 5 to smithi12556470-6 2021-03-24T22:02:55.369 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 6 v 517 (ObjNum 785 snap 257 seq_num 785) dirty exists 2021-03-24T22:02:55.369 INFO:tasks.rados.rados.0.smithi125.stdout:2278: left oid 6 (ObjNum 785 snap 257 seq_num 785) 2021-03-24T22:02:55.370 INFO:tasks.rados.rados.0.smithi125.stdout:2278: done (3 left) 2021-03-24T22:02:55.370 INFO:tasks.rados.rados.0.smithi125.stdout:2282: rollback oid 43 current snap is 257 2021-03-24T22:02:55.370 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 43 to 192 2021-03-24T22:02:55.370 INFO:tasks.rados.rados.0.smithi125.stdout:2283: read oid 49 snap -1 2021-03-24T22:02:55.371 INFO:tasks.rados.rados.0.smithi125.stdout:2283: expect (ObjNum 734 snap 235 seq_num 734) 2021-03-24T22:02:55.371 INFO:tasks.rados.rados.0.smithi125.stdout:2284: write (excl) oid 29 current snap is 257 2021-03-24T22:02:55.371 INFO:tasks.rados.rados.0.smithi125.stdout:2284: seq_num 787 ranges {427109=405322,1553564=545926,2579307=647802} 2021-03-24T22:02:55.372 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:55.373 INFO:tasks.rados.rados.0.smithi125.stdout:2284: writing smithi12556470-29 from 427109 to 832431 tid 1 2021-03-24T22:02:55.380 INFO:tasks.rados.rados.0.smithi125.stdout:2284: writing smithi12556470-29 from 1553564 to 2099490 tid 2 2021-03-24T22:02:55.386 INFO:tasks.rados.rados.0.smithi125.stdout:2284: writing smithi12556470-29 from 2579307 to 3227109 tid 3 2021-03-24T22:02:55.387 INFO:tasks.rados.rados.0.smithi125.stdout:2285: snap_remove snap 220 2021-03-24T22:02:55.421 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:55.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:55.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:02:55.528 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:55.541 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:55 smithi142 conmon[20125]: cluster 2021-03-24T22:02:54.348027+0000 mon.a (mon 2021-03-24T22:02:55.542 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:55 smithi142 conmon[20125]: .0) 1649 : cluster [DBG] osdmap e604: 8 total, 8 up, 8 in 2021-03-24T22:02:55.542 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:55 smithi142 conmon[20125]: audit 2021-03-24T22:02:54.895413+0000 mon.a (mon.0) 1650 : audit [INF] from='client.? v1:172.21.15.125:0/1265372226' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T22:02:55.542 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:55 smithi142 conmon[20125]: cluster 2021-03-24T22:02:55.308278 2021-03-24T22:02:55.543 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:55 smithi142 conmon[20125]: +0000 mon.a (mon.0) 1651 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:02:55.543 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:55 smithi142 conmon[20125]: audit 2021-03-24T22:02:55.318980+0000 mon.a 2021-03-24T22:02:55.543 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:55 smithi142 conmon[20125]: (mon.0) 1652 : audit [INF] from='client.? v1:172.21.15.125:0/1265372226' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T22:02:55.543 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:55 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:02:55.543 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:55 smithi142 conmon[20125]: -03-24T22:02:55.319093+0000 mon.a (mon.0) 1653 : cluster [DBG] osdmap e605: 8 total, 8 up, 8 in 2021-03-24T22:02:55.594 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:55.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:55.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:02:55.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:55 smithi125 conmon[19936]: cluster 2021-03-24T22:02:54.348027+0000 mon.a 2021-03-24T22:02:55.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:55 smithi125 conmon[19936]: (mon.0) 1649 : cluster [DBG] osdmap e604: 8 total, 8 up, 8 in 2021-03-24T22:02:55.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:55 smithi125 conmon[19936]: audit 2021-03-24T22:02:54.895413+0000 mon.a (mon.0) 1650 : audit [INF] from='client.? v1:172.21.15.125:0/1265372226' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T22:02:55.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:55 smithi125 conmon[19936]: cluster 2021-03-24T22:02:55.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:55 smithi125 conmon[19936]: 2021-03-24T22:02:55.308278+0000 mon.a (mon.0) 1651 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:02:55.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:55 smithi125 conmon[19936]: audit 2021-03-24T22:02 2021-03-24T22:02:55.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:55 smithi125 conmon[19936]: :55.318980+0000 mon.a (mon.0) 1652 : 2021-03-24T22:02:55.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:55 smithi125 conmon[19936]: audit [INF] from='client.? v1:172.21.15.125:0/1265372226' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T22:02:55.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:55 smithi125 conmon[19936]: cluster 2021-03-24T22:02: 2021-03-24T22:02:55.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:55 smithi125 conmon[19936]: 55.319093+0000 mon.a (mon.0) 1653 : cluster 2021-03-24T22:02:55.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:55 smithi125 conmon[19936]: [DBG] osdmap e605: 8 total, 8 up, 8 in 2021-03-24T22:02:55.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:55 smithi125 conmon[24927]: cluster 2021-03-24T22:02:54.348027+0000 mon.a ( 2021-03-24T22:02:55.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:55 smithi125 conmon[24927]: mon.0) 1649 : cluster [DBG] osdmap e604: 8 total, 8 up, 8 in 2021-03-24T22:02:55.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:55 smithi125 conmon[24927]: audit 2021-03-24T22:02:54.895413+0000 mon.a (mon.0) 1650 : audit [INF] from='client.? v1:172.21.15.125:0/1265372226' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T22:02:55.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:55 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T22:02:55.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:55 smithi125 conmon[24927]: 22:02:55.308278+0000 mon.a (mon.0) 2021-03-24T22:02:55.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:55 smithi125 conmon[24927]: 1651 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:02:55.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:55 smithi125 conmon[24927]: audit 2021-03-24T 2021-03-24T22:02:55.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:55 smithi125 conmon[24927]: 22:02:55.318980+0000 mon.a (mon.0 2021-03-24T22:02:55.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:55 smithi125 conmon[24927]: ) 1652 : audit [INF] from='client.? v1:172.21.15.125:0/1265372226' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T22:02:55.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:55 smithi125 conmon[24927]: cluster 2021-03-24T22 2021-03-24T22:02:55.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:55 smithi125 conmon[24927]: :02:55.319093+0000 mon.a (mon.0 2021-03-24T22:02:55.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:55 smithi125 conmon[24927]: ) 1653 : cluster [DBG] osdmap e605: 8 total, 8 up, 8 in 2021-03-24T22:02:55.699 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:55.760 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:55.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:55.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:02:55.867 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:55.921 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:55.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:55.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:02:56.028 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:56.088 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:56.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:56.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:02:56.195 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:56.254 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:56.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:56.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:02:56.327 INFO:tasks.rados.rados.0.smithi125.stdout:2279: expect (ObjNum 713 snap 227 seq_num 713) 2021-03-24T22:02:56.361 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:56.401 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:56.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:56.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:56.505 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:56.519 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:56 smithi142 conmon[20125]: cluster 2021-03-24T22:02:54.915830+0000 mgr.y (mgr.14144) 2021-03-24T22:02:56.519 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:56 smithi142 conmon[20125]: 574 : cluster [DBG] pgmap v1069: 168 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 165 active+clean; 541 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 650 KiB/s wr, 17 op/s; 65/1728 objects misplaced (3.762%); 157 B/s, 1 objects/s recovering 2021-03-24T22:02:56.520 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:56 smithi142 conmon[20125]: cluster 2021-03-24T22:02:55.344205+0000 mon.a (mon.0) 1654 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:02:56.520 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:56 smithi142 conmon[20125]: cluster 2021-03-24T22:02:55.848089+0000 2021-03-24T22:02:56.520 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:56 smithi142 conmon[20125]: osd.2 (osd.2) 91 : cluster [DBG] 5.6 scrub ok 2021-03-24T22:02:56.520 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:56 smithi142 conmon[20125]: 2021-03-24T22:02:56.521 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:56 smithi142 conmon[20125]: cluster 2021-03-24T22:02:56.521 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:56 smithi142 conmon[20125]: 2021-03-24T22:02: 2021-03-24T22:02:56.521 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:56 smithi142 conmon[20125]: 56.323891+0000 mon.a (mon.0) 1655 : 2021-03-24T22:02:56.521 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:56 smithi142 conmon[20125]: cluster [DBG] osdmap e606: 8 total, 8 up, 8 in 2021-03-24T22:02:56.522 INFO:tasks.rados.rados.0.smithi125.stdout:2280: finishing write tid 1 to smithi12556470-35 2021-03-24T22:02:56.522 INFO:tasks.rados.rados.0.smithi125.stdout:2281: finishing copy_from to smithi12556470-31 2021-03-24T22:02:56.523 INFO:tasks.rados.rados.0.smithi125.stdout:2281: got expected ENOENT (src dne) 2021-03-24T22:02:56.523 INFO:tasks.rados.rados.0.smithi125.stdout:2281: finishing copy_from racing read to smithi12556470-31 2021-03-24T22:02:56.523 INFO:tasks.rados.rados.0.smithi125.stdout:2283: expect (ObjNum 734 snap 235 seq_num 734) 2021-03-24T22:02:56.561 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:02:56.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:56.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:02:56.634 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T22:02:56.635 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T22:02:56.635 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T22:02:56.635 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T22:02:56.635 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T22:02:56.636 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T22:02:56.636 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T22:02:56.636 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:56 smithi125 conmon[19936]: cluster 2021-03-24T22:02:56.637 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:56 smithi125 conmon[19936]: 2021-03-24T22 2021-03-24T22:02:56.637 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:56 smithi125 conmon[19936]: :02:54.915830+0000 mgr.y (mgr.14144) 574 : cluster [DBG] pgmap v1069: 168 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 165 active+clean; 541 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 650 KiB/s wr, 17 op/s; 65/1728 objects misplaced (3.762%); 157 B/s, 1 objects/s recovering 2021-03-24T22:02:56.637 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:56 smithi125 conmon[19936]: cluster 2021-03-24T22:02:56.637 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:56 smithi125 conmon[19936]: 2021-03- 2021-03-24T22:02:56.638 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:56 smithi125 conmon[19936]: 24T22:02:55.344205+0000 mon.a (mon.0) 1654 : 2021-03-24T22:02:56.638 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:56 smithi125 conmon[19936]: cluster 2021-03-24T22:02:56.638 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:56 smithi125 conmon[19936]: [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:02:56.638 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:56 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:02:56.639 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:56 smithi125 conmon[19936]: 03-24T22:02:55.848089+0000 osd.2 (osd.2) 91 : 2021-03-24T22:02:56.639 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:56 smithi125 conmon[19936]: cluster 2021-03-24T22:02:56.639 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:56 smithi125 conmon[19936]: [DBG] 5.6 scrub ok 2021-03-24T22:02:56.639 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:56 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:02:56.639 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:56 smithi125 conmon[19936]: -03-24T22:02:56.323891+0000 mon.a (mon 2021-03-24T22:02:56.640 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:56 smithi125 conmon[19936]: .0) 1655 : 2021-03-24T22:02:56.640 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:56 smithi125 conmon[19936]: cluster [DBG] osdmap e606: 8 total, 8 up, 8 in 2021-03-24T22:02:56.640 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:56 smithi125 conmon[24927]: cluster 2021-03-24T22:02:54.915830+0000 2021-03-24T22:02:56.641 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:56 smithi125 conmon[24927]: mgr.y (mgr.14144) 574 : cluster [DBG] pgmap v1069: 168 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 165 active+clean; 541 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 650 KiB/s wr, 17 op/s; 65/1728 objects misplaced (3.762%); 157 B/s, 1 objects/s recovering 2021-03-24T22:02:56.641 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:56 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T22:02:56.641 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:56 smithi125 conmon[24927]: T22:02:55.344205+0000 mon.a (mon.0) 1654 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:02:56.641 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:56 smithi125 conmon[24927]: cluster 2021-03-24T22:02:55.848089 2021-03-24T22:02:56.641 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:56 smithi125 conmon[24927]: +0000 osd.2 (osd.2) 91 : cluster [DBG] 5.6 scrub ok 2021-03-24T22:02:56.642 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:56 smithi125 conmon[24927]: 2021-03-24T22:02:56.642 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:56 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:02:56.642 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:56 smithi125 conmon[24927]: 24T22:02:56.323891+0000 mon.a (mon.0) 2021-03-24T22:02:56.642 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:56 smithi125 conmon[24927]: 1655 : cluster [DBG] osdmap e606: 8 total, 8 up, 8 in 2021-03-24T22:02:56.659 INFO:tasks.rados.rados.0.smithi125.stdout:2280: finishing write tid 2 to smithi12556470-35 2021-03-24T22:02:56.660 INFO:tasks.rados.rados.0.smithi125.stdout:2280: finishing write tid 3 to smithi12556470-35 2021-03-24T22:02:56.660 INFO:tasks.rados.rados.0.smithi125.stdout:2279: done (6 left) 2021-03-24T22:02:56.660 INFO:tasks.rados.rados.0.smithi125.stdout:2281: done (5 left) 2021-03-24T22:02:56.660 INFO:tasks.rados.rados.0.smithi125.stdout:2283: done (4 left) 2021-03-24T22:02:56.660 INFO:tasks.rados.rados.0.smithi125.stdout:2285: done (3 left) 2021-03-24T22:02:56.661 INFO:tasks.rados.rados.0.smithi125.stdout:2286: delete oid 10 current snap is 257 2021-03-24T22:02:56.661 INFO:tasks.rados.rados.0.smithi125.stdout:2282: finishing rollback tid 0 to smithi12556470-43 2021-03-24T22:02:56.661 INFO:tasks.rados.rados.0.smithi125.stdout:2282: finishing rollback tid 1 to smithi12556470-43 2021-03-24T22:02:56.661 INFO:tasks.rados.rados.0.smithi125.stdout:2280: finishing write tid 4 to smithi12556470-35 2021-03-24T22:02:56.662 INFO:tasks.rados.rados.0.smithi125.stdout:2282: finishing rollback tid 2 to smithi12556470-43 2021-03-24T22:02:56.662 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 43 v 1247 (ObjNum 614 snap 186 seq_num 614) dirty exists 2021-03-24T22:02:56.662 INFO:tasks.rados.rados.0.smithi125.stdout:2280: finishing write tid 5 to smithi12556470-35 2021-03-24T22:02:56.662 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 35 v 223 (ObjNum 786 snap 257 seq_num 786) dirty exists 2021-03-24T22:02:56.663 INFO:tasks.rados.rados.0.smithi125.stdout:2280: left oid 35 (ObjNum 786 snap 257 seq_num 786) 2021-03-24T22:02:56.663 INFO:tasks.rados.rados.0.smithi125.stdout:2284: finishing write tid 1 to smithi12556470-29 2021-03-24T22:02:56.663 INFO:tasks.rados.rados.0.smithi125.stdout:2284: finishing write tid 2 to smithi12556470-29 2021-03-24T22:02:56.663 INFO:tasks.rados.rados.0.smithi125.stdout:2284: finishing write tid 3 to smithi12556470-29 2021-03-24T22:02:56.663 INFO:tasks.rados.rados.0.smithi125.stdout:2284: finishing write tid 4 to smithi12556470-29 2021-03-24T22:02:56.664 INFO:tasks.rados.rados.0.smithi125.stdout:2284: finishing write tid 5 to smithi12556470-29 2021-03-24T22:02:56.664 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 29 v 360 (ObjNum 787 snap 257 seq_num 787) dirty exists 2021-03-24T22:02:56.664 INFO:tasks.rados.rados.0.smithi125.stdout:2284: left oid 29 (ObjNum 787 snap 257 seq_num 787) 2021-03-24T22:02:56.665 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T22:02:56.665 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-24T22:02:56.665 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-24T22:02:56.666 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-24T22:02:56.666 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-24T22:02:56.666 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-24T22:02:56.666 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-24T22:02:56.667 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-24T22:02:56.667 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-24T22:02:56.668 INFO:tasks.rados.rados.0.smithi125.stdout:2280: done (3 left) 2021-03-24T22:02:56.668 INFO:tasks.rados.rados.0.smithi125.stdout:2282: done (2 left) 2021-03-24T22:02:56.668 INFO:tasks.rados.rados.0.smithi125.stdout:2284: done (1 left) 2021-03-24T22:02:56.668 INFO:tasks.rados.rados.0.smithi125.stdout:2286: done (0 left) 2021-03-24T22:02:56.669 INFO:tasks.rados.rados.0.smithi125.stdout:2287: read oid 15 snap -1 2021-03-24T22:02:56.669 INFO:tasks.rados.rados.0.smithi125.stdout:2287: expect (ObjNum 750 snap 241 seq_num 750) 2021-03-24T22:02:56.669 INFO:tasks.rados.rados.0.smithi125.stdout:2288: write (excl) oid 44 current snap is 257 2021-03-24T22:02:56.669 INFO:tasks.rados.rados.0.smithi125.stdout:2288: seq_num 789 ranges {460723=555820,1670528=549595,2634153=626570} 2021-03-24T22:02:56.670 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:56.674 INFO:tasks.rados.rados.0.smithi125.stdout:2288: writing smithi12556470-44 from 460723 to 1016543 tid 1 2021-03-24T22:02:56.683 INFO:tasks.rados.rados.0.smithi125.stdout:2288: writing smithi12556470-44 from 1670528 to 2220123 tid 2 2021-03-24T22:02:56.689 INFO:tasks.rados.rados.0.smithi125.stdout:2288: writing smithi12556470-44 from 2634153 to 3260723 tid 3 2021-03-24T22:02:56.689 INFO:tasks.rados.rados.0.smithi125.stdout:2289: copy_from oid 38 from oid 1 current snap is 257 2021-03-24T22:02:56.689 INFO:tasks.rados.rados.0.smithi125.stdout:2287: expect (ObjNum 750 snap 241 seq_num 750) 2021-03-24T22:02:56.721 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:56.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:56.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:56.798 INFO:tasks.rados.rados.0.smithi125.stdout:2288: finishing write tid 1 to smithi12556470-44 2021-03-24T22:02:56.798 INFO:tasks.rados.rados.0.smithi125.stdout:2288: finishing write tid 2 to smithi12556470-44 2021-03-24T22:02:56.798 INFO:tasks.rados.rados.0.smithi125.stdout:2288: finishing write tid 3 to smithi12556470-44 2021-03-24T22:02:56.799 INFO:tasks.rados.rados.0.smithi125.stdout:2288: finishing write tid 4 to smithi12556470-44 2021-03-24T22:02:56.799 INFO:tasks.rados.rados.0.smithi125.stdout:2288: finishing write tid 5 to smithi12556470-44 2021-03-24T22:02:56.799 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 44 v 516 (ObjNum 789 snap 257 seq_num 789) dirty exists 2021-03-24T22:02:56.799 INFO:tasks.rados.rados.0.smithi125.stdout:2288: left oid 44 (ObjNum 789 snap 257 seq_num 789) 2021-03-24T22:02:56.800 INFO:tasks.rados.rados.0.smithi125.stdout:2289: finishing copy_from to smithi12556470-38 2021-03-24T22:02:56.800 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 38 v 724 (ObjNum 747 snap 241 seq_num 747) dirty exists 2021-03-24T22:02:56.801 INFO:tasks.rados.rados.0.smithi125.stdout:2289: finishing copy_from racing read to smithi12556470-38 2021-03-24T22:02:56.801 INFO:tasks.rados.rados.0.smithi125.stdout:2287: done (2 left) 2021-03-24T22:02:56.802 INFO:tasks.rados.rados.0.smithi125.stdout:2288: done (1 left) 2021-03-24T22:02:56.802 INFO:tasks.rados.rados.0.smithi125.stdout:2289: done (0 left) 2021-03-24T22:02:56.802 INFO:tasks.rados.rados.0.smithi125.stdout:2290: read oid 2 snap 219 2021-03-24T22:02:56.803 INFO:tasks.rados.rados.0.smithi125.stdout:2290: expect (ObjNum 676 snap 210 seq_num 676) 2021-03-24T22:02:56.804 INFO:tasks.rados.rados.0.smithi125.stdout:2291: read oid 39 snap -1 2021-03-24T22:02:56.804 INFO:tasks.rados.rados.0.smithi125.stdout:2291: expect (ObjNum 742 snap 240 seq_num 742) 2021-03-24T22:02:56.804 INFO:tasks.rados.rados.0.smithi125.stdout:2292: write (excl) oid 11 current snap is 257 2021-03-24T22:02:56.804 INFO:tasks.rados.rados.0.smithi125.stdout:2292: seq_num 790 ranges {477530=631069,1612657=409606,2545223=517596} 2021-03-24T22:02:56.811 INFO:tasks.rados.rados.0.smithi125.stdout:2292: writing smithi12556470-11 from 477530 to 1108599 tid 1 2021-03-24T22:02:56.817 INFO:tasks.rados.rados.0.smithi125.stdout:2292: writing smithi12556470-11 from 1612657 to 2022263 tid 2 2021-03-24T22:02:56.823 INFO:tasks.rados.rados.0.smithi125.stdout:2292: writing smithi12556470-11 from 2545223 to 3062819 tid 3 2021-03-24T22:02:56.823 INFO:tasks.rados.rados.0.smithi125.stdout:2293: write oid 41 current snap is 257 2021-03-24T22:02:56.823 INFO:tasks.rados.rados.0.smithi125.stdout:2293: seq_num 791 ranges {494337=422671,1387492=553264,2688999=445221} 2021-03-24T22:02:56.828 INFO:tasks.rados.rados.0.smithi125.stdout:2293: writing smithi12556470-41 from 494337 to 917008 tid 1 2021-03-24T22:02:56.832 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:56.835 INFO:tasks.rados.rados.0.smithi125.stdout:2293: writing smithi12556470-41 from 1387492 to 1940756 tid 2 2021-03-24T22:02:56.841 INFO:tasks.rados.rados.0.smithi125.stdout:2293: writing smithi12556470-41 from 2688999 to 3134220 tid 3 2021-03-24T22:02:56.842 INFO:tasks.rados.rados.0.smithi125.stdout:2294: read oid 6 snap -1 2021-03-24T22:02:56.842 INFO:tasks.rados.rados.0.smithi125.stdout:2294: expect (ObjNum 785 snap 257 seq_num 785) 2021-03-24T22:02:56.842 INFO:tasks.rados.rados.0.smithi125.stdout:2295: write (excl) oid 19 current snap is 257 2021-03-24T22:02:56.842 INFO:tasks.rados.rados.0.smithi125.stdout:2295: seq_num 792 ranges {511144=497920,1445974=413275,2316422=656493} 2021-03-24T22:02:56.847 INFO:tasks.rados.rados.0.smithi125.stdout:2295: writing smithi12556470-19 from 511144 to 1009064 tid 1 2021-03-24T22:02:56.852 INFO:tasks.rados.rados.0.smithi125.stdout:2295: writing smithi12556470-19 from 1445974 to 1859249 tid 2 2021-03-24T22:02:56.859 INFO:tasks.rados.rados.0.smithi125.stdout:2295: writing smithi12556470-19 from 2316422 to 2972915 tid 3 2021-03-24T22:02:56.859 INFO:tasks.rados.rados.0.smithi125.stdout:2296: write (excl) oid 45 current snap is 257 2021-03-24T22:02:56.860 INFO:tasks.rados.rados.0.smithi125.stdout:2296: seq_num 793 ranges {527951=573169,1504456=556933,2743845=467765} 2021-03-24T22:02:56.865 INFO:tasks.rados.rados.0.smithi125.stdout:2296: writing smithi12556470-45 from 527951 to 1101120 tid 1 2021-03-24T22:02:56.870 INFO:tasks.rados.rados.0.smithi125.stdout:2296: writing smithi12556470-45 from 1504456 to 2061389 tid 2 2021-03-24T22:02:56.877 INFO:tasks.rados.rados.0.smithi125.stdout:2296: writing smithi12556470-45 from 2743845 to 3211610 tid 3 2021-03-24T22:02:56.877 INFO:tasks.rados.rados.0.smithi125.stdout:2297: rollback oid 14 current snap is 257 2021-03-24T22:02:56.877 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 14 to 219 2021-03-24T22:02:56.878 INFO:tasks.rados.rados.0.smithi125.stdout:2291: expect (ObjNum 742 snap 240 seq_num 742) 2021-03-24T22:02:56.889 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:02:56.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:56.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:02:56.963 INFO:tasks.rados.rados.0.smithi125.stdout:2290: expect (ObjNum 676 snap 210 seq_num 676) 2021-03-24T22:02:56.996 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:57.053 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:57.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:57.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:02:57.159 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:57.166 INFO:tasks.rados.rados.0.smithi125.stdout:2292: finishing write tid 1 to smithi12556470-11 2021-03-24T22:02:57.167 INFO:tasks.rados.rados.0.smithi125.stdout:2292: finishing write tid 2 to smithi12556470-11 2021-03-24T22:02:57.167 INFO:tasks.rados.rados.0.smithi125.stdout:2292: finishing write tid 3 to smithi12556470-11 2021-03-24T22:02:57.167 INFO:tasks.rados.rados.0.smithi125.stdout:2292: finishing write tid 4 to smithi12556470-11 2021-03-24T22:02:57.168 INFO:tasks.rados.rados.0.smithi125.stdout:2292: finishing write tid 5 to smithi12556470-11 2021-03-24T22:02:57.168 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 11 v 393 (ObjNum 790 snap 257 seq_num 790) dirty exists 2021-03-24T22:02:57.168 INFO:tasks.rados.rados.0.smithi125.stdout:2292: left oid 11 (ObjNum 790 snap 257 seq_num 790) 2021-03-24T22:02:57.168 INFO:tasks.rados.rados.0.smithi125.stdout:2293: finishing write tid 1 to smithi12556470-41 2021-03-24T22:02:57.169 INFO:tasks.rados.rados.0.smithi125.stdout:2293: finishing write tid 2 to smithi12556470-41 2021-03-24T22:02:57.169 INFO:tasks.rados.rados.0.smithi125.stdout:2293: finishing write tid 3 to smithi12556470-41 2021-03-24T22:02:57.169 INFO:tasks.rados.rados.0.smithi125.stdout:2293: finishing write tid 4 to smithi12556470-41 2021-03-24T22:02:57.169 INFO:tasks.rados.rados.0.smithi125.stdout:2293: finishing write tid 5 to smithi12556470-41 2021-03-24T22:02:57.169 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 41 v 775 (ObjNum 791 snap 257 seq_num 791) dirty exists 2021-03-24T22:02:57.170 INFO:tasks.rados.rados.0.smithi125.stdout:2293: left oid 41 (ObjNum 791 snap 257 seq_num 791) 2021-03-24T22:02:57.170 INFO:tasks.rados.rados.0.smithi125.stdout:2294: expect (ObjNum 785 snap 257 seq_num 785) 2021-03-24T22:02:57.212 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:57.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:57.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:02:57.276 INFO:tasks.rados.rados.0.smithi125.stdout:2295: finishing write tid 1 to smithi12556470-19 2021-03-24T22:02:57.277 INFO:tasks.rados.rados.0.smithi125.stdout:2295: finishing write tid 2 to smithi12556470-19 2021-03-24T22:02:57.277 INFO:tasks.rados.rados.0.smithi125.stdout:2295: finishing write tid 3 to smithi12556470-19 2021-03-24T22:02:57.277 INFO:tasks.rados.rados.0.smithi125.stdout:2295: finishing write tid 4 to smithi12556470-19 2021-03-24T22:02:57.277 INFO:tasks.rados.rados.0.smithi125.stdout:2295: finishing write tid 5 to smithi12556470-19 2021-03-24T22:02:57.278 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 19 v 532 (ObjNum 792 snap 257 seq_num 792) dirty exists 2021-03-24T22:02:57.278 INFO:tasks.rados.rados.0.smithi125.stdout:2295: left oid 19 (ObjNum 792 snap 257 seq_num 792) 2021-03-24T22:02:57.278 INFO:tasks.rados.rados.0.smithi125.stdout:2296: finishing write tid 1 to smithi12556470-45 2021-03-24T22:02:57.278 INFO:tasks.rados.rados.0.smithi125.stdout:2296: finishing write tid 2 to smithi12556470-45 2021-03-24T22:02:57.278 INFO:tasks.rados.rados.0.smithi125.stdout:2297: finishing rollback tid 0 to smithi12556470-14 2021-03-24T22:02:57.279 INFO:tasks.rados.rados.0.smithi125.stdout:2297: finishing rollback tid 1 to smithi12556470-14 2021-03-24T22:02:57.279 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 14 v 1258 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-03-24T22:02:57.279 INFO:tasks.rados.rados.0.smithi125.stdout:2296: finishing write tid 3 to smithi12556470-45 2021-03-24T22:02:57.279 INFO:tasks.rados.rados.0.smithi125.stdout:2296: finishing write tid 4 to smithi12556470-45 2021-03-24T22:02:57.280 INFO:tasks.rados.rados.0.smithi125.stdout:2296: finishing write tid 5 to smithi12556470-45 2021-03-24T22:02:57.280 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 45 v 664 (ObjNum 793 snap 257 seq_num 793) dirty exists 2021-03-24T22:02:57.280 INFO:tasks.rados.rados.0.smithi125.stdout:2296: left oid 45 (ObjNum 793 snap 257 seq_num 793) 2021-03-24T22:02:57.280 INFO:tasks.rados.rados.0.smithi125.stdout:2290: done (7 left) 2021-03-24T22:02:57.280 INFO:tasks.rados.rados.0.smithi125.stdout:2291: done (6 left) 2021-03-24T22:02:57.281 INFO:tasks.rados.rados.0.smithi125.stdout:2292: done (5 left) 2021-03-24T22:02:57.281 INFO:tasks.rados.rados.0.smithi125.stdout:2293: done (4 left) 2021-03-24T22:02:57.281 INFO:tasks.rados.rados.0.smithi125.stdout:2294: done (3 left) 2021-03-24T22:02:57.281 INFO:tasks.rados.rados.0.smithi125.stdout:2295: done (2 left) 2021-03-24T22:02:57.281 INFO:tasks.rados.rados.0.smithi125.stdout:2296: done (1 left) 2021-03-24T22:02:57.282 INFO:tasks.rados.rados.0.smithi125.stdout:2297: done (0 left) 2021-03-24T22:02:57.282 INFO:tasks.rados.rados.0.smithi125.stdout:2298: write (excl) oid 26 current snap is 257 2021-03-24T22:02:57.282 INFO:tasks.rados.rados.0.smithi125.stdout:2298: seq_num 794 ranges {544758=648418,1846585=416944,2771268=573490} 2021-03-24T22:02:57.288 INFO:tasks.rados.rados.0.smithi125.stdout:2298: writing smithi12556470-26 from 544758 to 1193176 tid 1 2021-03-24T22:02:57.296 INFO:tasks.rados.rados.0.smithi125.stdout:2298: writing smithi12556470-26 from 1846585 to 2263529 tid 2 2021-03-24T22:02:57.306 INFO:tasks.rados.rados.0.smithi125.stdout:2298: writing smithi12556470-26 from 2771268 to 3344758 tid 3 2021-03-24T22:02:57.307 INFO:tasks.rados.rados.0.smithi125.stdout:2299: write (excl) oid 49 current snap is 257 2021-03-24T22:02:57.308 INFO:tasks.rados.rados.0.smithi125.stdout:2299: seq_num 795 ranges {561565=723667,1905067=560602,3082338=279227} 2021-03-24T22:02:57.309 INFO:tasks.rados.rados.0.smithi125.stdout:2299: writing smithi12556470-49 from 561565 to 1285232 tid 1 2021-03-24T22:02:57.316 INFO:tasks.rados.rados.0.smithi125.stdout:2299: writing smithi12556470-49 from 1905067 to 2465669 tid 2 2021-03-24T22:02:57.317 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:57.321 INFO:tasks.rados.rados.0.smithi125.stdout:2299: writing smithi12556470-49 from 3082338 to 3361565 tid 3 2021-03-24T22:02:57.322 INFO:tasks.rados.rados.0.smithi125.stdout:2298: finishing write tid 1 to smithi12556470-26 2021-03-24T22:02:57.322 INFO:tasks.rados.rados.0.smithi125.stdout:2298: finishing write tid 2 to smithi12556470-26 2021-03-24T22:02:57.322 INFO:tasks.rados.rados.0.smithi125.stdout:2298: finishing write tid 3 to smithi12556470-26 2021-03-24T22:02:57.323 INFO:tasks.rados.rados.0.smithi125.stdout:2298: finishing write tid 4 to smithi12556470-26 2021-03-24T22:02:57.323 INFO:tasks.rados.rados.0.smithi125.stdout:2298: finishing write tid 5 to smithi12556470-26 2021-03-24T22:02:57.323 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 26 v 447 (ObjNum 794 snap 257 seq_num 794) dirty exists 2021-03-24T22:02:57.324 INFO:tasks.rados.rados.0.smithi125.stdout:2298: left oid 26 (ObjNum 794 snap 257 seq_num 794) 2021-03-24T22:02:57.324 INFO:tasks.rados.rados.0.smithi125.stdout:2299: finishing write tid 1 to smithi12556470-49 2021-03-24T22:02:57.325 INFO:tasks.rados.rados.0.smithi125.stdout:2299: finishing write tid 2 to smithi12556470-49 2021-03-24T22:02:57.325 INFO:tasks.rados.rados.0.smithi125.stdout:2298: done (1 left) 2021-03-24T22:02:57.325 INFO:tasks.rados.rados.0.smithi125.stdout:2300: delete oid 21 current snap is 257 2021-03-24T22:02:57.332 INFO:tasks.rados.rados.0.smithi125.stdout:2300: done (1 left) 2021-03-24T22:02:57.333 INFO:tasks.rados.rados.0.smithi125.stdout:2301: read oid 41 snap -1 2021-03-24T22:02:57.333 INFO:tasks.rados.rados.0.smithi125.stdout:2301: expect (ObjNum 791 snap 257 seq_num 791) 2021-03-24T22:02:57.333 INFO:tasks.rados.rados.0.smithi125.stdout:2302: delete oid 48 current snap is 257 2021-03-24T22:02:57.336 INFO:tasks.rados.rados.0.smithi125.stdout:2299: finishing write tid 3 to smithi12556470-49 2021-03-24T22:02:57.336 INFO:tasks.rados.rados.0.smithi125.stdout:2299: finishing write tid 4 to smithi12556470-49 2021-03-24T22:02:57.336 INFO:tasks.rados.rados.0.smithi125.stdout:2299: finishing write tid 5 to smithi12556470-49 2021-03-24T22:02:57.337 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 49 v 798 (ObjNum 795 snap 257 seq_num 795) dirty exists 2021-03-24T22:02:57.337 INFO:tasks.rados.rados.0.smithi125.stdout:2299: left oid 49 (ObjNum 795 snap 257 seq_num 795) 2021-03-24T22:02:57.342 INFO:tasks.rados.rados.0.smithi125.stdout:2299: done (2 left) 2021-03-24T22:02:57.342 INFO:tasks.rados.rados.0.smithi125.stdout:2302: done (1 left) 2021-03-24T22:02:57.343 INFO:tasks.rados.rados.0.smithi125.stdout:2303: rollback oid 16 current snap is 257 2021-03-24T22:02:57.343 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 16 to 192 2021-03-24T22:02:57.343 INFO:tasks.rados.rados.0.smithi125.stdout:2304: read oid 50 snap -1 2021-03-24T22:02:57.343 INFO:tasks.rados.rados.0.smithi125.stdout:2304: expect (ObjNum 659 snap 202 seq_num 659) 2021-03-24T22:02:57.344 INFO:tasks.rados.rados.0.smithi125.stdout:2305: snap_remove snap 179 2021-03-24T22:02:57.370 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:57.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:57.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:57.475 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:57.534 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:57.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:57.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:02:57.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:57 smithi142 conmon[20125]: cluster 2021-03-24T22:02:55.490736+0000 osd.3 2021-03-24T22:02:57.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:57 smithi142 conmon[20125]: (osd.3) 120 : cluster [DBG] 6.6 scrub ok 2021-03-24T22:02:57.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:57 smithi142 conmon[20125]: cluster 2021-03-24T22:02:55.950413+0000 osd.5 (osd.5) 28 : cluster [DBG] 6.d scrub ok 2021-03-24T22:02:57.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:57 smithi142 conmon[20125]: audit 2021-03-24T22: 2021-03-24T22:02:57.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:57 smithi142 conmon[20125]: 02:56.473128+0000 mon.a ( 2021-03-24T22:02:57.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:57 smithi142 conmon[20125]: mon.0) 1656 : audit [DBG] from='client.? v1:172.21.15.125:0/1262688186' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:02:57.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:57 smithi142 conmon[20125]: cluster 2021-03-24T22:02:57.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:57 smithi142 conmon[20125]: 2021-03-24T22:02 2021-03-24T22:02:57.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:57 smithi142 conmon[20125]: :57.324614+0000 mon.a (mon.0 2021-03-24T22:02:57.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:57 smithi142 conmon[20125]: ) 1657 : cluster [DBG] osdmap e607: 8 total, 8 up, 8 in 2021-03-24T22:02:57.640 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:57.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:57 smithi125 conmon[19936]: cluster 2021-03-24T22:02:55.490736+0000 osd.3 (osd.3 2021-03-24T22:02:57.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:57 smithi125 conmon[19936]: ) 120 : cluster [DBG] 6.6 scrub ok 2021-03-24T22:02:57.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:57 smithi125 conmon[19936]: cluster 2021-03-24T22:02:55.950413+0000 osd.5 (osd.5) 28 : cluster [DBG] 6.d scrub ok 2021-03-24T22:02:57.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:57 smithi125 conmon[19936]: audit 2021-03-24T22:02:57.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:57 smithi125 conmon[19936]: 2021-03-24T22:02:56.473128+0000 mon.a (mon.0) 1656 : audit [DBG] from='client.? v1:172.21.15.125:0/1262688186' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:02:57.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:57 smithi125 conmon[19936]: cluster 2021-03-24T22 2021-03-24T22:02:57.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:57 smithi125 conmon[19936]: :02:57.324614+0000 mon.a (mon.0) 1657 2021-03-24T22:02:57.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:57 smithi125 conmon[19936]: : cluster [DBG] osdmap e607: 8 total, 8 up, 8 in 2021-03-24T22:02:57.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:57 smithi125 conmon[24927]: cluster 2021-03-24T22:02:55.490736+0000 osd.3 (osd 2021-03-24T22:02:57.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:57 smithi125 conmon[24927]: .3) 120 : cluster [DBG] 6.6 scrub ok 2021-03-24T22:02:57.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:57 smithi125 conmon[24927]: cluster 2021-03-24T22:02:55.950413+0000 osd.5 (osd.5) 28 : cluster [DBG] 6.d scrub ok 2021-03-24T22:02:57.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:57 smithi125 conmon[24927]: audit 2021-03 2021-03-24T22:02:57.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:57 smithi125 conmon[24927]: -24T22:02:56.473128+0000 mon.a (mon.0) 1656 : audit [DBG] from='client.? v1:172.21.15.125:0/1262688186' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:02:57.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:57 smithi125 conmon[24927]: cluster 2021-03-24T22:02: 2021-03-24T22:02:57.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:57 smithi125 conmon[24927]: 57.324614+0000 mon.a (mon.0) 1657 : cluster [DBG] osdmap e607: 8 total, 8 up, 8 in 2021-03-24T22:02:57.702 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:02:57.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:57.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:02:57.807 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:57.866 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:57.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:57.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:02:57.972 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:58.034 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:58.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:58.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:02:58.141 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:58.199 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:58.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:58.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:58.305 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:58.334 INFO:tasks.rados.rados.0.smithi125.stdout:2305: done (3 left) 2021-03-24T22:02:58.334 INFO:tasks.rados.rados.0.smithi125.stdout:2306: snap_create 2021-03-24T22:02:58.339 INFO:tasks.rados.rados.0.smithi125.stdout:2304: expect (ObjNum 659 snap 202 seq_num 659) 2021-03-24T22:02:58.377 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:58.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:58.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:02:58.452 INFO:tasks.rados.rados.0.smithi125.stdout:2301: expect (ObjNum 791 snap 257 seq_num 791) 2021-03-24T22:02:58.482 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:58.545 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:58.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:58.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:02:58.608 INFO:tasks.rados.rados.0.smithi125.stdout:2303: finishing rollback tid 1 to smithi12556470-16 2021-03-24T22:02:58.609 INFO:tasks.rados.rados.0.smithi125.stdout:2303: finishing rollback tid 2 to smithi12556470-16 2021-03-24T22:02:58.609 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 16 v 435 (ObjNum 605 snap 184 seq_num 605) dirty exists 2021-03-24T22:02:58.653 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:58.705 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:58.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:58.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:58.711 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:58 smithi125 conmon[19936]: cluster 2021-03-24T22:02:56.523302+0000 osd.3 2021-03-24T22:02:58.711 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:58 smithi125 conmon[19936]: (osd.3) 121 : cluster [DBG] 5.17 scrub ok 2021-03-24T22:02:58.711 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:58 smithi125 conmon[19936]: cluster 2021-03-24T22:02:56.916809+0000 mgr.y (mgr.14144) 575 : cluster [DBG] pgmap v1072: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 166 active+clean; 545 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 990 KiB/s wr, 18 op/s; 54/1731 objects misplaced (3.120%); 357 B/s, 4 objects/s recovering 2021-03-24T22:02:58.712 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:58 smithi125 conmon[19936]: cluster 2021-03-24T22:02:57.768555+0000 osd.2 (osd.2) 2021-03-24T22:02:58.712 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:58 smithi125 conmon[19936]: 92 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:02:58.712 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:58 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T22:02:58.712 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:58 smithi125 conmon[19936]: -24T22:02:57.771900+0000 osd.2 (osd 2021-03-24T22:02:58.713 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:58 smithi125 conmon[19936]: .2) 93 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:02:58.713 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:58 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:02:58.713 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:58 smithi125 conmon[19936]: 03-24T22:02:58.331223+0000 2021-03-24T22:02:58.713 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:58 smithi125 conmon[19936]: mon.a (mon.0) 1658 : cluster [DBG] 2021-03-24T22:02:58.714 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:58 smithi125 conmon[19936]: osdmap e608: 8 total, 8 up, 8 in 2021-03-24T22:02:58.714 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:58 smithi125 conmon[24927]: cluster 2021-03-24T22:02:56.523302+0000 osd.3 (osd 2021-03-24T22:02:58.715 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:58 smithi125 conmon[24927]: .3) 121 : cluster [DBG] 5.17 scrub ok 2021-03-24T22:02:58.715 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:58 smithi125 conmon[24927]: cluster 2021-03-24T22:02:56.916809+0000 mgr.y (mgr.14144) 575 : cluster [DBG] 2021-03-24T22:02:58.715 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:58 smithi125 conmon[24927]: pgmap v1072: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 166 active+clean; 545 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 990 KiB/s wr, 18 op/s; 54/1731 objects misplaced (3.120%); 357 B/s, 4 objects/s recovering 2021-03-24T22:02:58.715 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:58 smithi125 conmon[24927]: cluster 2021-03-24T22:02:57 2021-03-24T22:02:58.715 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:58 smithi125 conmon[24927]: .768555+0000 osd.2 (osd.2) 92 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:02:58.716 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:58 smithi125 conmon[24927]: cluster 2021-03-24T22:02:57.771900 2021-03-24T22:02:58.716 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:58 smithi125 conmon[24927]: +0000 osd.2 (osd.2) 93 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:02:58.716 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:58 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T22:02:58.716 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:58 smithi125 conmon[24927]: 22:02:58.331223+0000 mon.a (mon.0) 1658 : 2021-03-24T22:02:58.717 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:58 smithi125 conmon[24927]: cluster [DBG] osdmap e608: 8 total, 8 up, 8 in 2021-03-24T22:02:58.811 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:58.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:58 smithi142 conmon[20125]: cluster 2021-03-24T22:02:56.523302+0000 osd.3 ( 2021-03-24T22:02:58.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:58 smithi142 conmon[20125]: osd.3) 121 : cluster [DBG] 5.17 scrub ok 2021-03-24T22:02:58.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:58 smithi142 conmon[20125]: cluster 2021-03-24T22:02:56.916809+0000 mgr.y (mgr.14144) 575 : cluster [DBG] pgmap v1072: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 166 active+clean; 545 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 990 KiB/s wr, 18 op/s; 54/1731 objects misplaced (3.120%); 357 B/s, 4 objects/s recovering 2021-03-24T22:02:58.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:58 smithi142 conmon[20125]: cluster 2021-03-24T22:02: 2021-03-24T22:02:58.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:58 smithi142 conmon[20125]: 57.768555+0000 osd.2 (osd.2) 92 2021-03-24T22:02:58.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:58 smithi142 conmon[20125]: : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:02:58.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:58 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:02:58.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:58 smithi142 conmon[20125]: T22:02:57.771900+0000 osd.2 ( 2021-03-24T22:02:58.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:58 smithi142 conmon[20125]: osd.2) 93 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:02:58.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:58 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:02:58.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:58 smithi142 conmon[20125]: -24T22:02:58.331223+0000 2021-03-24T22:02:58.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:58 smithi142 conmon[20125]: mon.a (mon.0) 1658 : cluster [DBG] 2021-03-24T22:02:58.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:58 smithi142 conmon[20125]: osdmap e608: 8 total, 8 up, 8 in 2021-03-24T22:02:58.870 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:02:58.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:58.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:02:58.977 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:59.036 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:02:59.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:59.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:02:59.144 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:59.205 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:59.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:59.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:02:59.312 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:59.338 INFO:tasks.rados.rados.0.smithi125.stdout:2301: done (3 left) 2021-03-24T22:02:59.339 INFO:tasks.rados.rados.0.smithi125.stdout:2303: done (2 left) 2021-03-24T22:02:59.339 INFO:tasks.rados.rados.0.smithi125.stdout:2304: done (1 left) 2021-03-24T22:02:59.339 INFO:tasks.rados.rados.0.smithi125.stdout:2306: done (0 left) 2021-03-24T22:02:59.339 INFO:tasks.rados.rados.0.smithi125.stdout:2307: read oid 46 snap -1 2021-03-24T22:02:59.340 INFO:tasks.rados.rados.0.smithi125.stdout:2307: expect deleted 2021-03-24T22:02:59.340 INFO:tasks.rados.rados.0.smithi125.stdout:2308: write (excl) oid 13 current snap is 258 2021-03-24T22:02:59.340 INFO:tasks.rados.rados.0.smithi125.stdout:2308: seq_num 798 ranges {611986=549414,1564160=424282,2764607=440478} 2021-03-24T22:02:59.342 INFO:tasks.rados.rados.0.smithi125.stdout:2308: writing smithi12556470-13 from 611986 to 1161400 tid 1 2021-03-24T22:02:59.347 INFO:tasks.rados.rados.0.smithi125.stdout:2308: writing smithi12556470-13 from 1564160 to 1988442 tid 2 2021-03-24T22:02:59.355 INFO:tasks.rados.rados.0.smithi125.stdout:2308: writing smithi12556470-13 from 2764607 to 3205085 tid 3 2021-03-24T22:02:59.356 INFO:tasks.rados.rados.0.smithi125.stdout:2309: write oid 23 current snap is 258 2021-03-24T22:02:59.356 INFO:tasks.rados.rados.0.smithi125.stdout:2309: seq_num 799 ranges {628793=741016,2138995=567940,3192030=236763} 2021-03-24T22:02:59.366 INFO:tasks.rados.rados.0.smithi125.stdout:2309: writing smithi12556470-23 from 628793 to 1369809 tid 1 2021-03-24T22:02:59.367 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:59.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:59.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:02:59.375 INFO:tasks.rados.rados.0.smithi125.stdout:2309: writing smithi12556470-23 from 2138995 to 2706935 tid 2 2021-03-24T22:02:59.380 INFO:tasks.rados.rados.0.smithi125.stdout:2309: writing smithi12556470-23 from 3192030 to 3428793 tid 3 2021-03-24T22:02:59.380 INFO:tasks.rados.rados.0.smithi125.stdout:2310: snap_create 2021-03-24T22:02:59.380 INFO:tasks.rados.rados.0.smithi125.stdout:2308: finishing write tid 1 to smithi12556470-13 2021-03-24T22:02:59.381 INFO:tasks.rados.rados.0.smithi125.stdout:2308: finishing write tid 2 to smithi12556470-13 2021-03-24T22:02:59.381 INFO:tasks.rados.rados.0.smithi125.stdout:2308: finishing write tid 3 to smithi12556470-13 2021-03-24T22:02:59.381 INFO:tasks.rados.rados.0.smithi125.stdout:2308: finishing write tid 4 to smithi12556470-13 2021-03-24T22:02:59.381 INFO:tasks.rados.rados.0.smithi125.stdout:2308: finishing write tid 5 to smithi12556470-13 2021-03-24T22:02:59.381 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 13 v 792 (ObjNum 798 snap 258 seq_num 798) dirty exists 2021-03-24T22:02:59.382 INFO:tasks.rados.rados.0.smithi125.stdout:2308: left oid 13 (ObjNum 798 snap 258 seq_num 798) 2021-03-24T22:02:59.382 INFO:tasks.rados.rados.0.smithi125.stdout:2309: finishing write tid 1 to smithi12556470-23 2021-03-24T22:02:59.383 INFO:tasks.rados.rados.0.smithi125.stdout:2309: finishing write tid 2 to smithi12556470-23 2021-03-24T22:02:59.383 INFO:tasks.rados.rados.0.smithi125.stdout:2309: finishing write tid 3 to smithi12556470-23 2021-03-24T22:02:59.384 INFO:tasks.rados.rados.0.smithi125.stdout:2309: finishing write tid 4 to smithi12556470-23 2021-03-24T22:02:59.384 INFO:tasks.rados.rados.0.smithi125.stdout:2309: finishing write tid 5 to smithi12556470-23 2021-03-24T22:02:59.384 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 23 v 395 (ObjNum 799 snap 258 seq_num 799) dirty exists 2021-03-24T22:02:59.384 INFO:tasks.rados.rados.0.smithi125.stdout:2309: left oid 23 (ObjNum 799 snap 258 seq_num 799) 2021-03-24T22:02:59.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:59 smithi125 conmon[24927]: cluster 2021-03-24T22:02:57.478360+0000 osd.0 (osd.0) 120 : 2021-03-24T22:02:59.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:59 smithi125 conmon[24927]: cluster [DBG] 7.16 scrub ok 2021-03-24T22:02:59.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:59 smithi125 conmon[24927]: cluster 2021-03-24T22:02:57.502794+0000 osd.3 (osd.3) 122 : cluster [DBG] 2021-03-24T22:02:59.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:59 smithi125 conmon[24927]: 3.27 scrub ok 2021-03-24T22:02:59.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:59 smithi125 conmon[24927]: cluster 2021-03-24T22 2021-03-24T22:02:59.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:59 smithi125 conmon[24927]: :02:58.801641+0000 osd.2 (osd.2) 94 : cluster [DBG] 2021-03-24T22:02:59.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:59 smithi125 conmon[24927]: 5.8 scrub ok 2021-03-24T22:02:59.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:59 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:02:59.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:02:59 smithi125 conmon[24927]: 24T22:02:59.328647+0000 mon.a (mon.0) 1659 : cluster [DBG] osdmap e609: 8 total, 8 up, 8 in 2021-03-24T22:02:59.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:59 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:02:59.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:59 smithi125 conmon[19936]: -03-24T22:02:57.478360+0000 osd.0 (osd.0) 120 : cluster [DBG] 7.16 scrub ok 2021-03-24T22:02:59.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:59 smithi125 conmon[19936]: cluster 2021-03-24T22:02:59.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:59 smithi125 conmon[19936]: 2021-03-24T22:02:57.502794+0000 osd.3 (osd.3) 122 : cluster 2021-03-24T22:02:59.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:59 smithi125 conmon[19936]: [DBG] 3.27 scrub ok 2021-03-24T22:02:59.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:59 smithi125 conmon[19936]: cluster 2021-03-24T22:02: 2021-03-24T22:02:59.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:59 smithi125 conmon[19936]: 58.801641+0000 osd.2 (osd.2) 94 : cluster 2021-03-24T22:02:59.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:59 smithi125 conmon[19936]: [DBG] 5.8 scrub ok 2021-03-24T22:02:59.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:59 smithi125 conmon[19936]: cluster 2021-03-24T22:02: 2021-03-24T22:02:59.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:59 smithi125 conmon[19936]: 59.328647+0000 mon.a (mon.0) 1659 : cluster 2021-03-24T22:02:59.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:02:59 smithi125 conmon[19936]: [DBG] osdmap e609: 8 total, 8 up, 8 in 2021-03-24T22:02:59.474 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:59.530 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:02:59.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:59.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:02:59.635 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:59.700 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:02:59.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:59.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:02:59.805 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:59.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:59 smithi142 conmon[20125]: cluster 2021-03-24T22:02:57.478360+0000 osd.0 (osd.0) 120 : cluster 2021-03-24T22:02:59.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:59 smithi142 conmon[20125]: [DBG] 7.16 scrub ok 2021-03-24T22:02:59.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:59 smithi142 conmon[20125]: cluster 2021-03-24T22:02:57.502794+0000 osd.3 (osd.3) 122 : cluster [DBG] 3.27 scrub ok 2021-03-24T22:02:59.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:59 smithi142 conmon[20125]: 2021-03-24T22:02:59.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:59 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:02:59.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:59 smithi142 conmon[20125]: 03-24T22:02:58.801641+0000 osd.2 2021-03-24T22:02:59.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:59 smithi142 conmon[20125]: (osd.2) 94 : cluster [DBG] 2021-03-24T22:02:59.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:59 smithi142 conmon[20125]: 5.8 scrub ok 2021-03-24T22:02:59.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:59 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:02:59.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:59 smithi142 conmon[20125]: 22:02:59.328647+0000 mon.a 2021-03-24T22:02:59.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:59 smithi142 conmon[20125]: (mon.0) 1659 : cluster [DBG] 2021-03-24T22:02:59.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:02:59 smithi142 conmon[20125]: osdmap e609: 8 total, 8 up, 8 in 2021-03-24T22:02:59.872 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:02:59.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:02:59.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:02:59.963 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-24T22:02:59.963 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-24T22:02:59.964 INFO:teuthology.orchestra.run.smithi125.stdout: "num_ops": 0 2021-03-24T22:02:59.964 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-24T22:02:59.976 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:00.027 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:00.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:00.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:00.138 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:00.184 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:00.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:00.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:00.290 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:00.326 INFO:tasks.rados.rados.0.smithi125.stdout:2307: done (3 left) 2021-03-24T22:03:00.326 INFO:tasks.rados.rados.0.smithi125.stdout:2308: done (2 left) 2021-03-24T22:03:00.327 INFO:tasks.rados.rados.0.smithi125.stdout:2309: done (1 left) 2021-03-24T22:03:00.327 INFO:tasks.rados.rados.0.smithi125.stdout:2310: done (0 left) 2021-03-24T22:03:00.328 INFO:tasks.rados.rados.0.smithi125.stdout:2311: snap_remove snap 240 2021-03-24T22:03:00.363 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:00.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:00.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:03:00.468 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:00.519 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:00.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:00.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:03:00.525 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:00 smithi125 conmon[24927]: cluster 2021-03-24T22:02:58.917561 2021-03-24T22:03:00.525 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:00 smithi125 conmon[24927]: +0000 mgr.y (mgr.14144) 576 : cluster [DBG] pgmap v1075: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 166 active+clean; 545 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 689 KiB/s wr, 6 op/s; 54/1731 objects misplaced (3.120%); 200 B/s, 2 objects/s recovering 2021-03-24T22:03:00.525 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:00 smithi125 conmon[24927]: cluster 2021-03-24T22:03:00.525 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:00 smithi125 conmon[24927]: 2021-03-24T22:02:59.400785+0000 2021-03-24T22:03:00.526 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:00 smithi125 conmon[24927]: osd.1 (osd.1) 55 : cluster 2021-03-24T22:03:00.526 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:00 smithi125 conmon[24927]: [DBG] 4.7 scrub ok 2021-03-24T22:03:00.526 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:00 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:03:00.526 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:00 smithi125 conmon[24927]: -03-24T22:03:00. 2021-03-24T22:03:00.527 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:00 smithi125 conmon[24927]: 319254+0000 mon.a (mon.0) 1660 2021-03-24T22:03:00.527 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:00 smithi125 conmon[24927]: : cluster [DBG] osdmap e610: 8 total, 8 up, 8 in 2021-03-24T22:03:00.527 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:00 smithi125 conmon[24927]: audit 2021 2021-03-24T22:03:00.527 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:00 smithi125 conmon[24927]: -03-24T22:03:00. 2021-03-24T22:03:00.527 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:00 smithi125 conmon[24927]: 328427+0000 mon.a (mon.0) 2021-03-24T22:03:00.528 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:00 smithi125 conmon[24927]: 1661 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T22:03:00.528 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:00 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:03:00.528 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:00 smithi125 conmon[19936]: T22:02:58.917561+0000 mgr.y (mgr.14144) 576 : cluster [DBG] pgmap v1075: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 166 active+clean; 545 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 689 KiB/s wr, 6 op/s; 54/1731 objects misplaced (3.120%); 200 B/s, 2 objects/s recovering 2021-03-24T22:03:00.528 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:00 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:03:00.529 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:00 smithi125 conmon[19936]: T22:02:59. 2021-03-24T22:03:00.529 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:00 smithi125 conmon[19936]: 400785+0000 osd.1 ( 2021-03-24T22:03:00.529 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:00 smithi125 conmon[19936]: osd.1) 55 : cluster 2021-03-24T22:03:00.529 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:00 smithi125 conmon[19936]: [DBG] 4.7 scrub ok 2021-03-24T22:03:00.530 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:00 smithi125 conmon[19936]: cluster 2021-03-24T22:03:00.530 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:00 smithi125 conmon[19936]: 2021-03-24T22: 2021-03-24T22:03:00.530 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:00 smithi125 conmon[19936]: 03:00.319254+0000 mon.a 2021-03-24T22:03:00.530 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:00 smithi125 conmon[19936]: (mon.0) 1660 : cluster 2021-03-24T22:03:00.530 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:00 smithi125 conmon[19936]: [DBG] osdmap e610: 8 total, 8 up, 8 in 2021-03-24T22:03:00.531 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:00 smithi125 conmon[19936]: audit 2021-03-24T22:03:00.531 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:00 smithi125 conmon[19936]: 2021-03-24 2021-03-24T22:03:00.531 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:00 smithi125 conmon[19936]: T22:03:00 2021-03-24T22:03:00.531 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:00 smithi125 conmon[19936]: .328427+0000 mon.a (mon. 2021-03-24T22:03:00.531 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:00 smithi125 conmon[19936]: 0) 1661 : audit [DBG] 2021-03-24T22:03:00.532 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:00 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T22:03:00.624 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-24T22:03:00.625 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:00.680 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:00.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:00.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:03:00.686 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:00 smithi142 conmon[20125]: cluster 2021-03-24T22:02:58. 2021-03-24T22:03:00.686 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:00 smithi142 conmon[20125]: 917561+0000 mgr.y (mgr.14144) 576 : cluster [DBG] pgmap v1075: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 166 active+clean; 545 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 689 KiB/s wr, 6 op/s; 54/1731 objects misplaced (3.120%); 200 B/s, 2 objects/s recovering 2021-03-24T22:03:00.686 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:00 smithi142 conmon[20125]: cluster 2021-03-24T22:02:59.400785+0000 osd.1 ( 2021-03-24T22:03:00.687 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:00 smithi142 conmon[20125]: osd.1) 55 : cluster [DBG] 4.7 scrub ok 2021-03-24T22:03:00.687 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:00 smithi142 conmon[20125]: cluster 2021-03-24T22:03:00.687 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:00 smithi142 conmon[20125]: 2021-03-24T22:03:00.319254+0000 mon.a (mon.0 2021-03-24T22:03:00.687 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:00 smithi142 conmon[20125]: ) 1660 : cluster [DBG] osdmap e610: 8 total, 8 up, 8 in 2021-03-24T22:03:00.688 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:00 smithi142 conmon[20125]: audit 2021-03-24T22: 2021-03-24T22:03:00.688 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:00 smithi142 conmon[20125]: 03:00.328427+0000 mon.a (mon.0) 1661 : audit [DBG] 2021-03-24T22:03:00.688 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:00 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T22:03:00.787 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:00.850 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:00.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:00.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:03:00.957 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:01.021 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:01.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:01.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:01.126 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:01.186 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:01.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:01.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:03:01.292 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:01.332 INFO:tasks.rados.rados.0.smithi125.stdout:2311: done (0 left) 2021-03-24T22:03:01.332 INFO:tasks.rados.rados.0.smithi125.stdout:2312: read oid 29 snap -1 2021-03-24T22:03:01.333 INFO:tasks.rados.rados.0.smithi125.stdout:2312: expect (ObjNum 787 snap 257 seq_num 787) 2021-03-24T22:03:01.333 INFO:tasks.rados.rados.0.smithi125.stdout:2313: write oid 8 current snap is 259 2021-03-24T22:03:01.333 INFO:tasks.rados.rados.0.smithi125.stdout:2313: seq_num 800 ranges {645600=416265,1797477=711598,3219453=226147} 2021-03-24T22:03:01.340 INFO:tasks.rados.rados.0.smithi125.stdout:2313: writing smithi12556470-8 from 645600 to 1061865 tid 1 2021-03-24T22:03:01.349 INFO:tasks.rados.rados.0.smithi125.stdout:2313: writing smithi12556470-8 from 1797477 to 2509075 tid 2 2021-03-24T22:03:01.356 INFO:tasks.rados.rados.0.smithi125.stdout:2313: writing smithi12556470-8 from 3219453 to 3445600 tid 3 2021-03-24T22:03:01.356 INFO:tasks.rados.rados.0.smithi125.stdout:2314: read oid 36 snap 219 2021-03-24T22:03:01.356 INFO:tasks.rados.rados.0.smithi125.stdout:2314: expect deleted 2021-03-24T22:03:01.357 INFO:tasks.rados.rados.0.smithi125.stdout:2315: write (excl) oid 6 current snap is 259 2021-03-24T22:03:01.357 INFO:tasks.rados.rados.0.smithi125.stdout:2315: seq_num 801 ranges {662407=491514,1855959=571609,2846876=615531} 2021-03-24T22:03:01.360 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:01.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:01.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:03:01.364 INFO:tasks.rados.rados.0.smithi125.stdout:2315: writing smithi12556470-6 from 662407 to 1153921 tid 1 2021-03-24T22:03:01.371 INFO:tasks.rados.rados.0.smithi125.stdout:2315: writing smithi12556470-6 from 1855959 to 2427568 tid 2 2021-03-24T22:03:01.377 INFO:tasks.rados.rados.0.smithi125.stdout:2315: writing smithi12556470-6 from 2846876 to 3462407 tid 3 2021-03-24T22:03:01.377 INFO:tasks.rados.rados.0.smithi125.stdout:2316: snap_create 2021-03-24T22:03:01.377 INFO:tasks.rados.rados.0.smithi125.stdout:2313: finishing write tid 1 to smithi12556470-8 2021-03-24T22:03:01.377 INFO:tasks.rados.rados.0.smithi125.stdout:2312: expect (ObjNum 787 snap 257 seq_num 787) 2021-03-24T22:03:01.429 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:03:01.445 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.0 dump_blocked_ops 2021-03-24T22:03:01.465 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:01.521 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:01.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:01.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:03:01.554 INFO:tasks.rados.rados.0.smithi125.stdout:2313: finishing write tid 2 to smithi12556470-8 2021-03-24T22:03:01.555 INFO:tasks.rados.rados.0.smithi125.stdout:2313: finishing write tid 3 to smithi12556470-8 2021-03-24T22:03:01.555 INFO:tasks.rados.rados.0.smithi125.stdout:2313: finishing write tid 4 to smithi12556470-8 2021-03-24T22:03:01.555 INFO:tasks.rados.rados.0.smithi125.stdout:2313: finishing write tid 5 to smithi12556470-8 2021-03-24T22:03:01.556 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 8 v 218 (ObjNum 800 snap 259 seq_num 800) dirty exists 2021-03-24T22:03:01.556 INFO:tasks.rados.rados.0.smithi125.stdout:2313: left oid 8 (ObjNum 800 snap 259 seq_num 800) 2021-03-24T22:03:01.556 INFO:tasks.rados.rados.0.smithi125.stdout:2315: finishing write tid 1 to smithi12556470-6 2021-03-24T22:03:01.556 INFO:tasks.rados.rados.0.smithi125.stdout:2315: finishing write tid 2 to smithi12556470-6 2021-03-24T22:03:01.556 INFO:tasks.rados.rados.0.smithi125.stdout:2315: finishing write tid 3 to smithi12556470-6 2021-03-24T22:03:01.557 INFO:tasks.rados.rados.0.smithi125.stdout:2315: finishing write tid 4 to smithi12556470-6 2021-03-24T22:03:01.557 INFO:tasks.rados.rados.0.smithi125.stdout:2315: finishing write tid 5 to smithi12556470-6 2021-03-24T22:03:01.557 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 6 v 526 (ObjNum 801 snap 259 seq_num 801) dirty exists 2021-03-24T22:03:01.557 INFO:tasks.rados.rados.0.smithi125.stdout:2315: left oid 6 (ObjNum 801 snap 259 seq_num 801) 2021-03-24T22:03:01.625 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-24T22:03:01.629 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:01.633 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:01 smithi125 conmon[24927]: cluster 2021-03-24T22:02:59.519378+0000 osd.0 (osd.0) 121 : cluster [DBG] 4.6 scrub ok 2021-03-24T22:03:01.634 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:01 smithi125 conmon[24927]: cluster 2021-03-24T22:03:00.406991+0000 osd.1 (osd.1) 56 : cluster [DBG] 5.19 scrub ok 2021-03-24T22:03:01.634 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:01 smithi125 conmon[24927]: cluster 2021-03-24T22:03:00.471941+0000 osd.3 (osd.3) 123 : cluster [DBG] 5.1 scrub ok 2021-03-24T22:03:01.634 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:01 smithi125 conmon[24927]: audit 2021-03-24T22:03:00.640264+0000 mon.a (mon.0) 1662 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:03:01.634 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:01 smithi125 conmon[24927]: cluster 2021-03-24T22:03:00.744086+0000 osd.6 (osd.6) 44 : cluster [DBG] 4.5 scrub ok 2021-03-24T22:03:01.635 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:01 smithi125 conmon[24927]: cluster 2021-03-24T22:03:01.329449+0000 mon.a (mon.0) 1663 : cluster [DBG] osdmap e611: 8 total, 8 up, 8 in 2021-03-24T22:03:01.635 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:01 smithi125 conmon[19936]: cluster 2021-03-24T22:02 2021-03-24T22:03:01.635 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:01 smithi125 conmon[19936]: :59.519378+0000 osd.0 (osd.0) 121 : cluster [DBG] 4.6 scrub ok 2021-03-24T22:03:01.636 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:01 smithi125 conmon[19936]: cluster 2021-03-24T22:03:01.636 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:01 smithi125 conmon[19936]: 2021-03-24T 2021-03-24T22:03:01.636 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:01 smithi125 conmon[19936]: 22:03:00.406991+0000 2021-03-24T22:03:01.636 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:01 smithi125 conmon[19936]: osd.1 (osd.1) 56 : 2021-03-24T22:03:01.637 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:01 smithi125 conmon[19936]: cluster [DBG] 5.19 scrub ok 2021-03-24T22:03:01.637 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:01 smithi125 conmon[19936]: cluster 2021-03-24T22:03:01.637 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:01 smithi125 conmon[19936]: 2021-03-24T22 2021-03-24T22:03:01.637 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:01 smithi125 conmon[19936]: :03:00.471941+0000 2021-03-24T22:03:01.637 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:01 smithi125 conmon[19936]: osd.3 (osd.3) 123 : 2021-03-24T22:03:01.638 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:01 smithi125 conmon[19936]: cluster [DBG] 5.1 scrub ok 2021-03-24T22:03:01.638 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:01 smithi125 conmon[19936]: audit 2021-03-24T22:03:01.638 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:01 smithi125 conmon[19936]: 2021-03-24T22 2021-03-24T22:03:01.638 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:01 smithi125 conmon[19936]: :03:00.640264+0000 2021-03-24T22:03:01.639 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:01 smithi125 conmon[19936]: mon.a (mon.0) 1662 : 2021-03-24T22:03:01.639 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:01 smithi125 conmon[19936]: audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:03:01.639 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:01 smithi125 conmon[19936]: cluster 2021-03-24T22:03:01.639 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:01 smithi125 conmon[19936]: 2021-03-24T22:03 2021-03-24T22:03:01.639 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:01 smithi125 conmon[19936]: :00.744086+0000 osd.6 ( 2021-03-24T22:03:01.640 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:01 smithi125 conmon[19936]: osd.6) 44 : cluster [DBG] 2021-03-24T22:03:01.640 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:01 smithi125 conmon[19936]: 4.5 scrub ok 2021-03-24T22:03:01.640 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:01 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:03:01.640 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:01 smithi125 conmon[19936]: T22:03:01.329449+0000 2021-03-24T22:03:01.640 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:01 smithi125 conmon[19936]: mon.a (mon.0) 1663 : 2021-03-24T22:03:01.641 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:01 smithi125 conmon[19936]: cluster [DBG] osdmap e611: 8 total, 8 up, 8 in 2021-03-24T22:03:01.686 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:01.690 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:01 smithi142 conmon[20125]: cluster 2021-03-24T22:02: 2021-03-24T22:03:01.690 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:01 smithi142 conmon[20125]: 59.519378+0000 osd.0 (osd.0) 121 : cluster [DBG] 4.6 scrub ok 2021-03-24T22:03:01.691 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:01 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:03:01.691 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:01 smithi142 conmon[20125]: T22:03:00.406991+0000 osd.1 (osd.1) 56 : cluster 2021-03-24T22:03:01.691 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:01 smithi142 conmon[20125]: [DBG] 5.19 scrub ok 2021-03-24T22:03:01.691 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:01 smithi142 conmon[20125]: cluster 2021-03-24T22:03:01.692 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:01 smithi142 conmon[20125]: 2021-03-24T22:03 2021-03-24T22:03:01.692 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:01 smithi142 conmon[20125]: :00.471941+0000 osd.3 (osd.3) 123 2021-03-24T22:03:01.692 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:01 smithi142 conmon[20125]: : cluster [DBG] 5.1 scrub ok 2021-03-24T22:03:01.692 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:01 smithi142 conmon[20125]: audit 2021-03 2021-03-24T22:03:01.692 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:01 smithi142 conmon[20125]: -24T22:03:00.640264+0000 mon.a (mon.0 2021-03-24T22:03:01.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:01 smithi142 conmon[20125]: ) 1662 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:03:01.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:01 smithi142 conmon[20125]: cluster 2021-03-24T22:03 2021-03-24T22:03:01.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:01 smithi142 conmon[20125]: :00.744086+0000 osd.6 (osd.6) 2021-03-24T22:03:01.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:01 smithi142 conmon[20125]: 44 : cluster [DBG] 4.5 scrub ok 2021-03-24T22:03:01.693 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:01 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:03:01.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:01 smithi142 conmon[20125]: 22:03:01.329449+0000 mon.a (mon 2021-03-24T22:03:01.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:01 smithi142 conmon[20125]: .0) 1663 : cluster [DBG] osdmap e611: 8 total, 8 up, 8 in 2021-03-24T22:03:01.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:01.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:03:01.796 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:01.846 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:03:01.848 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:01.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:01.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:01.953 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:02.007 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:02.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:02.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:03:02.114 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:02.173 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:02.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:02.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:03:02.278 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:02.344 INFO:tasks.rados.rados.0.smithi125.stdout:2312: done (4 left) 2021-03-24T22:03:02.344 INFO:tasks.rados.rados.0.smithi125.stdout:2313: done (3 left) 2021-03-24T22:03:02.344 INFO:tasks.rados.rados.0.smithi125.stdout:2314: done (2 left) 2021-03-24T22:03:02.345 INFO:tasks.rados.rados.0.smithi125.stdout:2315: done (1 left) 2021-03-24T22:03:02.345 INFO:tasks.rados.rados.0.smithi125.stdout:2316: done (0 left) 2021-03-24T22:03:02.345 INFO:tasks.rados.rados.0.smithi125.stdout:2317: snap_create 2021-03-24T22:03:02.346 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:02.346 INFO:teuthology.orchestra.run.smithi125.stderr:reweighted osd.7 to 1 (10000) 2021-03-24T22:03:02.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:02.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:03:02.450 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:02.505 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:02.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:02.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:03:02.612 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:02.626 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[19936]: cluster 2021-03-24T22:03:00. 2021-03-24T22:03:02.626 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[19936]: 918391+0000 mgr.y (mgr.14144) 577 : cluster [DBG] pgmap v1078: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 543 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 2.9 MiB/s wr, 29 op/s; 40/1731 objects misplaced (2.311%); 301 B/s, 3 objects/s recovering 2021-03-24T22:03:02.627 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:03:02.627 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[19936]: T22:03:00.965001+0000 osd.5 (osd.5) 29 : cluster [DBG] 3.d scrub ok 2021-03-24T22:03:02.627 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T22:03:02.627 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[19936]: -24T22:03:01.351465+0000 osd.7 (osd 2021-03-24T22:03:02.627 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[19936]: .7) 100 : cluster [DBG] 6.15 scrub ok 2021-03-24T22:03:02.628 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:03:02.628 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[19936]: 24T22:03:01.394459+0000 osd.1 ( 2021-03-24T22:03:02.628 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[19936]: osd.1) 57 : cluster [DBG] 6.1 scrub ok 2021-03-24T22:03:02.628 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[19936]: audit 2021- 2021-03-24T22:03:02.629 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[19936]: 03-24T22:03:01.436691+0000 mon.c 2021-03-24T22:03:02.629 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[19936]: (mon.2) 93 : audit [INF] from='client.? v1:172.21.15.125:0/397218346' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2021-03-24T22:03:02.629 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[19936]: audit 2021- 2021-03-24T22:03:02.629 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[19936]: 03-24T22:03:01.437169+0000 mon.a 2021-03-24T22:03:02.629 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[19936]: (mon.0) 1664 : audit [INF] from='client.35540 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2021-03-24T22:03:02.630 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T22:03:02.630 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[19936]: -24T22:03:01.782474+0000 osd.6 ( 2021-03-24T22:03:02.630 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[19936]: osd.6) 45 : cluster [DBG] 5.18 scrub ok 2021-03-24T22:03:02.630 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[19936]: audit 2021-03 2021-03-24T22:03:02.631 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[19936]: -24T22:03:02.335805+0000 mon.a ( 2021-03-24T22:03:02.631 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[19936]: mon.0) 1665 : audit [INF] from='client.35540 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2021-03-24T22:03:02.631 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:03:02.631 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[19936]: -03-24T22:03:02.336005+0000 2021-03-24T22:03:02.631 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[19936]: mon.a (mon.0) 1666 : cluster [DBG] osdmap e612: 8 total, 8 up, 8 in 2021-03-24T22:03:02.632 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[24927]: cluster 2021-03-24T22:03:02.632 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[24927]: 2021-03-24T22:03:02.633 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[24927]: 2021 2021-03-24T22:03:02.633 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[24927]: - 2021-03-24T22:03:02.633 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[24927]: 03 2021-03-24T22:03:02.633 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[24927]: - 2021-03-24T22:03:02.634 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[24927]: 24 2021-03-24T22:03:02.636 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[24927]: T 2021-03-24T22:03:02.637 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[24927]: 22 2021-03-24T22:03:02.639 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[24927]: : 2021-03-24T22:03:02.639 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[24927]: 03 2021-03-24T22:03:02.640 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[24927]: : 2021-03-24T22:03:02.641 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[24927]: 00 2021-03-24T22:03:02.643 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[24927]: . 2021-03-24T22:03:02.644 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[24927]: 918391 2021-03-24T22:03:02.645 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[24927]: +0000 mgr.y (mgr.14144) 577 : cluster 2021-03-24T22:03:02.647 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[24927]: [DBG] pgmap v1078: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 543 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 2.9 MiB/s wr, 29 op/s; 40/1731 objects misplaced (2.311%); 301 B/s, 3 objects/s recovering 2021-03-24T22:03:02.649 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[24927]: cluster 2021-03-24T22:03:00. 2021-03-24T22:03:02.649 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[24927]: 965001+0000 osd.5 (osd.5) 29 : cluster 2021-03-24T22:03:02.650 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[24927]: [DBG] 3.d scrub ok 2021-03-24T22:03:02.650 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[24927]: cluster 2021-03-24T22:03:02.650 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[24927]: 2021-03-24T22: 2021-03-24T22:03:02.650 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[24927]: 03:01.351465+0000 osd.7 (osd.7) 100 : cluster [DBG] 6.15 scrub ok 2021-03-24T22:03:02.651 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[24927]: cluster 2021-03-24T22:03:02.651 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[24927]: 2021-03-24T22:03:02.651 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[24927]: 2021 2021-03-24T22:03:02.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[24927]: - 2021-03-24T22:03:02.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[24927]: 03 2021-03-24T22:03:02.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[24927]: - 2021-03-24T22:03:02.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[24927]: 24 2021-03-24T22:03:02.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[24927]: T 2021-03-24T22:03:02.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[24927]: 22:03:01.394459+0000 2021-03-24T22:03:02.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[24927]: osd.1 (osd.1) 57 : cluster [DBG] 6.1 scrub ok 2021-03-24T22:03:02.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[24927]: audit 2021-03-24T22:03:02.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[24927]: 2021-03-24T22:03:01.436691+0000 mon.c (mon 2021-03-24T22:03:02.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[24927]: .2) 93 : audit [INF] from='client.? v1:172.21.15.125:0/397218346' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2021-03-24T22:03:02.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[24927]: audit 2021-03-24T22:03:01.437169+0000 mon.a ( 2021-03-24T22:03:02.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[24927]: mon.0) 1664 : audit [INF] from='client.35540 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2021-03-24T22:03:02.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T22:03:02.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[24927]: -24T22:03:01.782474+0000 osd.6 ( 2021-03-24T22:03:02.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[24927]: osd.6) 45 : cluster [DBG] 5.18 scrub ok 2021-03-24T22:03:02.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[24927]: audit 2021-03-24 2021-03-24T22:03:02.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[24927]: T22:03:02.335805+0000 mon.a (mon.0) 1665 : audit 2021-03-24T22:03:02.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[24927]: [INF] from='client.35540 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2021-03-24T22:03:02.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[24927]: cluster 2021-03-24T22:03: 2021-03-24T22:03:02.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[24927]: 02.336005+0000 mon.a (mon.0) 1666 : 2021-03-24T22:03:02.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:02 smithi125 conmon[24927]: cluster [DBG] osdmap e612: 8 total, 8 up, 8 in 2021-03-24T22:03:02.665 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:02.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:02.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:03:02.770 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:02.826 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:02.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:02.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:02.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:02 smithi142 conmon[20125]: cluster 2021-03-24T22:03:00.918391+0000 mgr.y (mgr.14144) 577 : cluster 2021-03-24T22:03:02.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:02 smithi142 conmon[20125]: [DBG] pgmap v1078: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 543 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 2.9 MiB/s wr, 29 op/s; 40/1731 objects misplaced (2.311%); 301 B/s, 3 objects/s recovering 2021-03-24T22:03:02.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:02 smithi142 conmon[20125]: cluster 2021-03-24T22:03:00.965001+0000 osd.5 ( 2021-03-24T22:03:02.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:02 smithi142 conmon[20125]: osd.5) 29 : cluster [DBG] 3.d scrub ok 2021-03-24T22:03:02.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:02 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:03:02.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:02 smithi142 conmon[20125]: -03-24T22:03:01. 2021-03-24T22:03:02.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:02 smithi142 conmon[20125]: 351465+0000 osd.7 (osd.7) 2021-03-24T22:03:02.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:02 smithi142 conmon[20125]: 100 : cluster [DBG] 6.15 scrub ok 2021-03-24T22:03:02.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:02 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:03:02.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:02 smithi142 conmon[20125]: 03-24T22:03:01.394459 2021-03-24T22:03:02.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:02 smithi142 conmon[20125]: +0000 osd.1 (osd.1) 2021-03-24T22:03:02.872 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:02 smithi142 conmon[20125]: 57 : cluster [DBG] 6.1 scrub ok 2021-03-24T22:03:02.872 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:02 smithi142 conmon[20125]: audit 2021 2021-03-24T22:03:02.872 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:02 smithi142 conmon[20125]: -03-24T22:03: 2021-03-24T22:03:02.872 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:02 smithi142 conmon[20125]: 01.436691+0000 mon.c (mon. 2021-03-24T22:03:02.873 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:02 smithi142 conmon[20125]: 2) 93 : audit [INF] 2021-03-24T22:03:02.873 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:02 smithi142 conmon[20125]: from='client.? v1:172.21.15.125:0/397218346' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2021-03-24T22:03:02.873 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:02 smithi142 conmon[20125]: audit 2021- 2021-03-24T22:03:02.873 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:02 smithi142 conmon[20125]: 03-24T22:03:01 2021-03-24T22:03:02.874 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:02 smithi142 conmon[20125]: .437169+0000 mon.a (mon. 2021-03-24T22:03:02.874 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:02 smithi142 conmon[20125]: 0) 1664 : audit [INF] 2021-03-24T22:03:02.874 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:02 smithi142 conmon[20125]: from='client.35540 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2021-03-24T22:03:02.874 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:02 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:03:02.874 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:02 smithi142 conmon[20125]: 03-24T22:03:01.782474+0000 2021-03-24T22:03:02.875 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:02 smithi142 conmon[20125]: osd.6 (osd.6) 45 : cluster [DBG] 5.18 scrub ok 2021-03-24T22:03:02.875 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:02 smithi142 conmon[20125]: 2021-03-24T22:03:02.875 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:02 smithi142 conmon[20125]: audit 2021-03-24 2021-03-24T22:03:02.875 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:02 smithi142 conmon[20125]: T22:03:02.335805+0000 2021-03-24T22:03:02.876 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:02 smithi142 conmon[20125]: mon.a (mon.0) 1665 2021-03-24T22:03:02.876 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:02 smithi142 conmon[20125]: : audit [INF] from='client.35540 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2021-03-24T22:03:02.876 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:02 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:03:02.876 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:02 smithi142 conmon[20125]: -03-24T22:03: 2021-03-24T22:03:02.876 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:02 smithi142 conmon[20125]: 02.336005+0000 mon.a (mon 2021-03-24T22:03:02.877 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:02 smithi142 conmon[20125]: .0) 1666 : cluster [DBG] 2021-03-24T22:03:02.877 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:02 smithi142 conmon[20125]: osdmap e612: 8 total, 8 up, 8 in 2021-03-24T22:03:02.931 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:02.985 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:02.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:02.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:03:03.092 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:03.095 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:03:03.149 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:03.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:03.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:03:03.256 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:03.313 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:03.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:03.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:03:03.346 INFO:tasks.rados.rados.0.smithi125.stdout:2317: done (0 left) 2021-03-24T22:03:03.346 INFO:tasks.rados.rados.0.smithi125.stdout:2318: snap_create 2021-03-24T22:03:03.422 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:03.482 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:03.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:03.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:03:03.487 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:03 smithi125 conmon[19936]: cluster 2021-03-24T22:03: 2021-03-24T22:03:03.488 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:03 smithi125 conmon[19936]: 01.931126+0000 osd.5 (osd.5) 30 : cluster [DBG] 6.7 scrub ok 2021-03-24T22:03:03.488 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:03 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:03:03.488 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:03 smithi125 conmon[19936]: 03-24T22:03:02.455937+0000 osd.3 (osd.3 2021-03-24T22:03:03.488 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:03 smithi125 conmon[19936]: ) 124 : cluster [DBG] 5.14 scrub ok 2021-03-24T22:03:03.488 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:03 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T22:03:03.489 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:03 smithi125 conmon[19936]: -24T22:03:02. 2021-03-24T22:03:03.489 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:03 smithi125 conmon[19936]: 553357+0000 osd.5 (osd. 2021-03-24T22:03:03.489 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:03 smithi125 conmon[19936]: 5) 31 : cluster [DBG] 2021-03-24T22:03:03.489 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:03 smithi125 conmon[19936]: purged_snaps scrub starts 2021-03-24T22:03:03.490 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:03 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:03:03.490 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:03 smithi125 conmon[19936]: 03-24T22:03:02.556422 2021-03-24T22:03:03.490 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:03 smithi125 conmon[19936]: +0000 osd.5 (osd.5) 32 : cluster 2021-03-24T22:03:03.490 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:03 smithi125 conmon[19936]: [DBG] purged_snaps scrub ok 2021-03-24T22:03:03.491 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:03 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:03:03.491 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:03 smithi125 conmon[19936]: -03-24T22:03: 2021-03-24T22:03:03.491 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:03 smithi125 conmon[19936]: 03.338729+0000 mon.a 2021-03-24T22:03:03.491 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:03 smithi125 conmon[19936]: (mon.0) 2021-03-24T22:03:03.491 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:03 smithi125 conmon[19936]: 1667 : cluster [DBG] 2021-03-24T22:03:03.492 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:03 smithi125 conmon[19936]: osdmap e613: 8 total, 8 up, 8 in 2021-03-24T22:03:03.493 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:03 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T22:03:03.493 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:03 smithi125 conmon[24927]: T22:03:01.931126+0000 osd.5 (osd.5) 30 : cluster 2021-03-24T22:03:03.493 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:03 smithi125 conmon[24927]: [DBG] 6.7 scrub ok 2021-03-24T22:03:03.494 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:03 smithi125 conmon[24927]: cluster 2021-03-24T22 2021-03-24T22:03:03.494 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:03 smithi125 conmon[24927]: :03:02.455937+0000 osd.3 (osd 2021-03-24T22:03:03.494 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:03 smithi125 conmon[24927]: .3) 124 : cluster [DBG] 2021-03-24T22:03:03.494 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:03 smithi125 conmon[24927]: 5.14 scrub ok 2021-03-24T22:03:03.495 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:03 smithi125 conmon[24927]: cluster 2021-03-24T22:03:03.495 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:03 smithi125 conmon[24927]: 2021-03-24T22 2021-03-24T22:03:03.495 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:03 smithi125 conmon[24927]: :03:02.553357+0000 2021-03-24T22:03:03.495 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:03 smithi125 conmon[24927]: osd.5 (osd.5) 2021-03-24T22:03:03.495 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:03 smithi125 conmon[24927]: 31 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:03:03.496 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:03 smithi125 conmon[24927]: 2021-03-24T22:03:03.496 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:03 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:03:03.496 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:03 smithi125 conmon[24927]: 03-24T22:03 2021-03-24T22:03:03.497 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:03 smithi125 conmon[24927]: :02.556422+0000 osd.5 2021-03-24T22:03:03.497 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:03 smithi125 conmon[24927]: (osd.5) 32 : 2021-03-24T22:03:03.497 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:03 smithi125 conmon[24927]: cluster [DBG] purged_snaps scrub ok 2021-03-24T22:03:03.497 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:03 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:03:03.498 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:03 smithi125 conmon[24927]: -03-24T22:03 2021-03-24T22:03:03.498 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:03 smithi125 conmon[24927]: :03.338729+0000 mon.a 2021-03-24T22:03:03.498 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:03 smithi125 conmon[24927]: (mon.0) 1667 : 2021-03-24T22:03:03.498 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:03 smithi125 conmon[24927]: cluster [DBG] osdmap e613: 8 total, 8 up, 8 in 2021-03-24T22:03:03.588 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:03.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:03 smithi142 conmon[20125]: cluster 2021-03-24T22 2021-03-24T22:03:03.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:03 smithi142 conmon[20125]: :03:01.931126+0000 osd.5 (osd.5) 30 : cluster [DBG] 6.7 scrub ok 2021-03-24T22:03:03.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:03 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:03:03.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:03 smithi142 conmon[20125]: T22:03:02.455937 2021-03-24T22:03:03.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:03 smithi142 conmon[20125]: +0000 osd.3 (osd.3) 2021-03-24T22:03:03.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:03 smithi142 conmon[20125]: 124 : cluster [DBG] 5.14 scrub ok 2021-03-24T22:03:03.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:03 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:03:03.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:03 smithi142 conmon[20125]: -03-24T22:03:02 2021-03-24T22:03:03.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:03 smithi142 conmon[20125]: .553357+0000 osd.5 (osd 2021-03-24T22:03:03.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:03 smithi142 conmon[20125]: .5) 31 : cluster [DBG] 2021-03-24T22:03:03.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:03 smithi142 conmon[20125]: purged_snaps scrub starts 2021-03-24T22:03:03.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:03 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:03:03.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:03 smithi142 conmon[20125]: -03-24T22:03 2021-03-24T22:03:03.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:03 smithi142 conmon[20125]: :02.556422+0000 osd.5 ( 2021-03-24T22:03:03.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:03 smithi142 conmon[20125]: osd.5) 32 : cluster 2021-03-24T22:03:03.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:03 smithi142 conmon[20125]: [DBG] purged_snaps scrub ok 2021-03-24T22:03:03.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:03 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:03:03.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:03 smithi142 conmon[20125]: -03-24T22:03 2021-03-24T22:03:03.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:03 smithi142 conmon[20125]: :03.338729+0000 mon.a ( 2021-03-24T22:03:03.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:03 smithi142 conmon[20125]: mon.0) 1667 : cluster 2021-03-24T22:03:03.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:03 smithi142 conmon[20125]: [DBG] osdmap e613: 8 total, 8 up, 8 in 2021-03-24T22:03:03.647 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:03.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:03.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:03.752 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:03.813 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:03.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:03.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:03:03.920 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:03.977 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:03.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:03.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:03:04.084 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:04.146 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:04.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:04.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:03:04.253 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:04.313 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:04.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:04.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:03:04.344 INFO:tasks.rados.rados.0.smithi125.stdout:2318: done (0 left) 2021-03-24T22:03:04.345 INFO:tasks.rados.rados.0.smithi125.stdout:2319: snap_remove snap 229 2021-03-24T22:03:04.420 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:04.448 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:04 smithi125 conmon[19936]: cluster 2021-03-24T22:03:02.919141+0000 mgr.y ( 2021-03-24T22:03:04.449 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:04 smithi125 conmon[19936]: mgr.14144) 578 : cluster [DBG] pgmap v1081: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 544 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 24 MiB/s rd, 4.6 MiB/s wr, 41 op/s; 40/1737 objects misplaced (2.303%); 301 B/s, 3 objects/s recovering 2021-03-24T22:03:04.449 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:04 smithi125 conmon[19936]: cluster 2021-03-24T22:03:03.442377+0000 osd.1 (osd.1) 58 : 2021-03-24T22:03:04.449 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:04 smithi125 conmon[19936]: cluster [DBG] 4.9 scrub ok 2021-03-24T22:03:04.449 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:04 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T22:03:04.450 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:04 smithi125 conmon[19936]: 22:03:03.453703+0000 osd.3 (osd.3) 125 : cluster 2021-03-24T22:03:04.450 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:04 smithi125 conmon[19936]: [DBG] 4.c scrub ok 2021-03-24T22:03:04.450 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:04 smithi125 conmon[19936]: cluster 2021-03-24T22:03:04.450 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:04 smithi125 conmon[19936]: 2021-03-24T22:03 2021-03-24T22:03:04.450 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:04 smithi125 conmon[19936]: :03.723410+0000 osd.2 (osd.2 2021-03-24T22:03:04.451 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:04 smithi125 conmon[19936]: ) 95 : cluster [DBG] 4.d scrub ok 2021-03-24T22:03:04.451 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:04 smithi125 conmon[19936]: cluster 2021-03-24T22:03:04.451 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:04 smithi125 conmon[19936]: 2021-03-24T22:03:04 2021-03-24T22:03:04.451 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:04 smithi125 conmon[19936]: .339015+0000 mon.a (mon.0) 1668 : cluster [DBG] osdmap e614: 8 total, 8 up, 8 in 2021-03-24T22:03:04.452 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:04 smithi125 conmon[24927]: cluster 2021-03-24T22:03:02.919141 2021-03-24T22:03:04.452 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:04 smithi125 conmon[24927]: +0000 mgr.y (mgr.14144) 578 : cluster [DBG] pgmap v1081: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 544 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 24 MiB/s rd, 4.6 MiB/s wr, 41 op/s; 40/1737 objects misplaced (2.303%); 301 B/s, 3 objects/s recovering 2021-03-24T22:03:04.452 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:04 smithi125 conmon[24927]: cluster 2021-03-24T22:03:03. 2021-03-24T22:03:04.452 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:04 smithi125 conmon[24927]: 442377+0000 osd.1 (osd.1) 58 : cluster [DBG] 4.9 scrub ok 2021-03-24T22:03:04.453 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:04 smithi125 conmon[24927]: cluster 2021-03-24T22:03:04.453 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:04 smithi125 conmon[24927]: 2021-03-24T22:03:03.453703+0000 osd.3 (osd.3) 125 : cluster [DBG] 4.c scrub ok 2021-03-24T22:03:04.453 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:04 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T22:03:04.453 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:04 smithi125 conmon[24927]: T22:03:03.723410+0000 osd.2 (osd.2) 95 : cluster [DBG] 4.d scrub ok 2021-03-24T22:03:04.454 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:04 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T22:03:04.454 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:04 smithi125 conmon[24927]: T22:03:04.339015+0000 mon.a (mon.0) 1668 : cluster [DBG] osdmap e614: 8 total, 8 up, 8 in 2021-03-24T22:03:04.454 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:04 smithi125 conmon[24927]: 2021-03-24T22:03:04.478 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:04.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:04.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:03:04.493 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2021-03-24T22:03:04.493 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph pg dump --format=json 2021-03-24T22:03:04.584 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:04.624 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:04.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:04.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:03:04.729 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:04.771 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:04.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:04.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:03:04.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:04 smithi142 conmon[20125]: cluster 2021-03-24T22:03 2021-03-24T22:03:04.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:04 smithi142 conmon[20125]: :02.919141+0000 mgr.y (mgr.14144) 578 : cluster [DBG] pgmap v1081: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 544 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 24 MiB/s rd, 4.6 MiB/s wr, 41 op/s; 40/1737 objects misplaced (2.303%); 301 B/s, 3 objects/s recovering 2021-03-24T22:03:04.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:04 smithi142 conmon[20125]: cluster 2021-03-24T22:03:04.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:04 smithi142 conmon[20125]: 2021-03-24T22:03:03.442377+0000 osd.1 (osd.1) 58 2021-03-24T22:03:04.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:04 smithi142 conmon[20125]: : cluster [DBG] 4.9 scrub ok 2021-03-24T22:03:04.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:04 smithi142 conmon[20125]: cluster 2021-03-24T22:03:04.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:04 smithi142 conmon[20125]: 2021-03-24T22:03:03.453703+0000 2021-03-24T22:03:04.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:04 smithi142 conmon[20125]: osd.3 (osd.3) 125 : cluster [DBG] 4.c scrub ok 2021-03-24T22:03:04.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:04 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:03:04.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:04 smithi142 conmon[20125]: -03-24T22:03:03 2021-03-24T22:03:04.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:04 smithi142 conmon[20125]: .723410+0000 osd.2 (osd.2) 95 2021-03-24T22:03:04.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:04 smithi142 conmon[20125]: : cluster [DBG] 4.d scrub ok 2021-03-24T22:03:04.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:04 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:03:04.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:04 smithi142 conmon[20125]: -03-24T22:03:04.339015 2021-03-24T22:03:04.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:04 smithi142 conmon[20125]: +0000 mon.a (mon.0) 1668 : cluster 2021-03-24T22:03:04.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:04 smithi142 conmon[20125]: [DBG] osdmap e614: 8 total, 8 up, 8 in 2021-03-24T22:03:04.876 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:04.935 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:04.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:04.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:03:05.042 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:05.105 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:05.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:05.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:05.210 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:05.264 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:05.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:05.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:03:05.332 INFO:tasks.rados.rados.0.smithi125.stdout:2319: done (0 left) 2021-03-24T22:03:05.332 INFO:tasks.rados.rados.0.smithi125.stdout:2320: snap_remove snap 198 2021-03-24T22:03:05.371 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:05.421 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:05.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:05.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:03:05.527 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:05.587 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:05.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:05.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:05.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:05 smithi142 conmon[20125]: audit 2021-03-24T22:03 2021-03-24T22:03:05.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:05 smithi142 conmon[20125]: :04.818281+0000 mon.a (mon.0) 1669 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T22:03:05.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:05 smithi142 conmon[20125]: audit 2021-03-24T22:03: 2021-03-24T22:03:05.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:05 smithi142 conmon[20125]: 05.167442+0000 mon.a (mon.0) 1670 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 1]}]: dispatch 2021-03-24T22:03:05.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:05 smithi142 conmon[20125]: audit 2021-03 2021-03-24T22:03:05.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:05 smithi142 conmon[20125]: -24T22:03:05.167949+0000 2021-03-24T22:03:05.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:05 smithi142 conmon[20125]: mon.a (mon.0) 1671 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2021-03-24T22:03:05.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:05 smithi142 conmon[20125]: 2021-03-24T22:03:05.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:05 smithi142 conmon[20125]: cluster 2021-03-24T22:03:05.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:05 smithi142 conmon[20125]: 2021-03-24T22:03:05.313471+0000 mon.a 2021-03-24T22:03:05.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:05 smithi142 conmon[20125]: (mon.0) 1672 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:03:05.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:05 smithi142 conmon[20125]: 2021-03-24T22:03:05.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:05 smithi142 conmon[20125]: audit 2021 2021-03-24T22:03:05.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:05 smithi142 conmon[20125]: -03-24T22:03:05.324163+0000 mon.a 2021-03-24T22:03:05.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:05 smithi142 conmon[20125]: (mon.0) 1673 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 1]}]': finished 2021-03-24T22:03:05.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:05 smithi142 conmon[20125]: 2021-03-24T22:03:05.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:05 smithi142 conmon[20125]: audit 2021 2021-03-24T22:03:05.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:05 smithi142 conmon[20125]: -03-24T22:03:05.324470+0000 mon.a ( 2021-03-24T22:03:05.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:05 smithi142 conmon[20125]: mon.0) 1674 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2021-03-24T22:03:05.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:05 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:03:05.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:05 smithi142 conmon[20125]: -03-24T22:03:05.324716 2021-03-24T22:03:05.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:05 smithi142 conmon[20125]: +0000 mon.a (mon.0) 1675 : cluster 2021-03-24T22:03:05.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:05 smithi142 conmon[20125]: [DBG] osdmap e615: 8 total, 8 up, 8 in 2021-03-24T22:03:05.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:05 smithi125 conmon[24927]: audit 2021-03-24T22:03:04.818281 2021-03-24T22:03:05.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:05 smithi125 conmon[24927]: +0000 mon.a (mon.0) 1669 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T22:03:05.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:05 smithi125 conmon[24927]: audit 2021-03-24T 2021-03-24T22:03:05.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:05 smithi125 conmon[24927]: 22:03:05.167442+0000 mon.a (mon.0) 1670 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 1]}]: dispatch 2021-03-24T22:03:05.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:05 smithi125 conmon[24927]: audit 2021-03-24T22:03:05.167949+0000 2021-03-24T22:03:05.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:05 smithi125 conmon[24927]: mon.a (mon.0) 1671 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2021-03-24T22:03:05.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:05 smithi125 conmon[24927]: cluster 2021-03-24T22 2021-03-24T22:03:05.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:05 smithi125 conmon[24927]: :03:05.313471+0000 mon.a (mon.0) 1672 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:03:05.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:05 smithi125 conmon[24927]: 2021-03-24T22:03:05.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:05 smithi125 conmon[24927]: audit 2021-03-24T22:03:05.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:05 smithi125 conmon[24927]: 2021-03-24T22:03:05.324163+0000 mon.a (mon.0) 1673 : audit [INF] 2021-03-24T22:03:05.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:05 smithi125 conmon[24927]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 1]}]': finished 2021-03-24T22:03:05.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:05 smithi125 conmon[24927]: audit 2021- 2021-03-24T22:03:05.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:05 smithi125 conmon[24927]: 03-24T22:03:05.324470+0000 mon.a (mon.0) 1674 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2021-03-24T22:03:05.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:05 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:03:05.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:05 smithi125 conmon[24927]: 24T22:03:05.324716+0000 mon.a (mon.0) 1675 : cluster 2021-03-24T22:03:05.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:05 smithi125 conmon[24927]: [DBG] osdmap e615: 8 total, 8 up, 8 in 2021-03-24T22:03:05.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:05 smithi125 conmon[19936]: audit 2021-03-24T22:03 2021-03-24T22:03:05.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:05 smithi125 conmon[19936]: :04.818281+0000 mon.a (mon.0) 1669 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T22:03:05.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:05 smithi125 conmon[19936]: audit 2021-03 2021-03-24T22:03:05.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:05 smithi125 conmon[19936]: -24T22:03:05 2021-03-24T22:03:05.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:05 smithi125 conmon[19936]: .167442+0000 mon.a (mon 2021-03-24T22:03:05.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:05 smithi125 conmon[19936]: .0) 1670 : audit 2021-03-24T22:03:05.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:05 smithi125 conmon[19936]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 1]}]: dispatch 2021-03-24T22:03:05.669 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:05 smithi125 conmon[19936]: audit 2021-03-24T22:03:05.669 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:05 smithi125 conmon[19936]: 2021-03-24T22 2021-03-24T22:03:05.669 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:05 smithi125 conmon[19936]: :03:05.167949+0000 2021-03-24T22:03:05.669 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:05 smithi125 conmon[19936]: mon.a (mon.0) 2021-03-24T22:03:05.670 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:05 smithi125 conmon[19936]: 1671 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2021-03-24T22:03:05.670 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:05 smithi125 conmon[19936]: 2021-03-24T22:03:05.670 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:05 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T22:03:05.670 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:05 smithi125 conmon[19936]: -24T22:03 2021-03-24T22:03:05.671 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:05 smithi125 conmon[19936]: :05.313471+0000 mon.a 2021-03-24T22:03:05.671 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:05 smithi125 conmon[19936]: (mon.0) 1672 : cluster 2021-03-24T22:03:05.671 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:05 smithi125 conmon[19936]: [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:03:05.671 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:05 smithi125 conmon[19936]: audit 2021-03-24T22:03:05.671 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:05 smithi125 conmon[19936]: 2021-03-24T22 2021-03-24T22:03:05.672 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:05 smithi125 conmon[19936]: :03:05.324163+0000 2021-03-24T22:03:05.672 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:05 smithi125 conmon[19936]: mon.a (mon.0) 2021-03-24T22:03:05.672 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:05 smithi125 conmon[19936]: 1673 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 1]}]': finished 2021-03-24T22:03:05.672 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:05 smithi125 conmon[19936]: 2021-03-24T22:03:05.673 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:05 smithi125 conmon[19936]: audit 2021-03 2021-03-24T22:03:05.673 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:05 smithi125 conmon[19936]: -24T22:03: 2021-03-24T22:03:05.673 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:05 smithi125 conmon[19936]: 05.324470+0000 mon.a ( 2021-03-24T22:03:05.674 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:05 smithi125 conmon[19936]: mon.0) 1674 : audit 2021-03-24T22:03:05.674 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:05 smithi125 conmon[19936]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2021-03-24T22:03:05.674 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:05 smithi125 conmon[19936]: cluster 2021-03-24T22:03:05.674 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:05 smithi125 conmon[19936]: 2021-03-24T 2021-03-24T22:03:05.675 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:05 smithi125 conmon[19936]: 22:03:05. 2021-03-24T22:03:05.675 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:05 smithi125 conmon[19936]: 324716+0000 mon.a ( 2021-03-24T22:03:05.675 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:05 smithi125 conmon[19936]: mon.0) 1675 : 2021-03-24T22:03:05.675 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:05 smithi125 conmon[19936]: cluster [DBG] osdmap e615: 8 total, 8 up, 8 in 2021-03-24T22:03:05.692 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:05.749 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:05.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:05.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:03:05.855 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:05.911 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:05.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:05.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:03:06.017 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:06.074 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:06.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:06.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:03:06.180 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:06.235 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:06.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:06.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:06.342 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:06.343 INFO:tasks.rados.rados.0.smithi125.stdout:2320: done (0 left) 2021-03-24T22:03:06.343 INFO:tasks.rados.rados.0.smithi125.stdout:2321: read oid 7 snap 248 2021-03-24T22:03:06.344 INFO:tasks.rados.rados.0.smithi125.stdout:2321: expect (ObjNum 727 snap 233 seq_num 727) 2021-03-24T22:03:06.344 INFO:tasks.rados.rados.0.smithi125.stdout:2322: copy_from oid 45 from oid 30 current snap is 262 2021-03-24T22:03:06.344 INFO:tasks.rados.rados.0.smithi125.stdout:2323: rollback oid 8 current snap is 262 2021-03-24T22:03:06.344 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 8 to 216 2021-03-24T22:03:06.344 INFO:tasks.rados.rados.0.smithi125.stdout:2324: read oid 10 snap -1 2021-03-24T22:03:06.345 INFO:tasks.rados.rados.0.smithi125.stdout:2324: expect deleted 2021-03-24T22:03:06.345 INFO:tasks.rados.rados.0.smithi125.stdout:2325: write (excl) oid 11 current snap is 262 2021-03-24T22:03:06.345 INFO:tasks.rados.rados.0.smithi125.stdout:2325: seq_num 802 ranges {679214=566763,1798088=715267,3157946=321268} 2021-03-24T22:03:06.346 INFO:tasks.rados.rados.0.smithi125.stdout:2325: writing smithi12556470-11 from 679214 to 1245977 tid 1 2021-03-24T22:03:06.355 INFO:tasks.rados.rados.0.smithi125.stdout:2325: writing smithi12556470-11 from 1798088 to 2513355 tid 2 2021-03-24T22:03:06.361 INFO:tasks.rados.rados.0.smithi125.stdout:2325: writing smithi12556470-11 from 3157946 to 3479214 tid 3 2021-03-24T22:03:06.361 INFO:tasks.rados.rados.0.smithi125.stdout:2326: copy_from oid 44 from oid 42 current snap is 262 2021-03-24T22:03:06.361 INFO:tasks.rados.rados.0.smithi125.stdout:2327: snap_create 2021-03-24T22:03:06.366 INFO:tasks.rados.rados.0.smithi125.stdout:2323: finishing rollback tid 0 to smithi12556470-8 2021-03-24T22:03:06.366 INFO:tasks.rados.rados.0.smithi125.stdout:2323: finishing rollback tid 1 to smithi12556470-8 2021-03-24T22:03:06.367 INFO:tasks.rados.rados.0.smithi125.stdout:2323: finishing rollback tid 2 to smithi12556470-8 2021-03-24T22:03:06.367 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 8 v 222 (ObjNum 669 snap 208 seq_num 669) dirty exists 2021-03-24T22:03:06.367 INFO:tasks.rados.rados.0.smithi125.stdout:2321: expect (ObjNum 727 snap 233 seq_num 727) 2021-03-24T22:03:06.368 INFO:teuthology.orchestra.run.smithi125.stderr:nodeep-scrub is unset 2021-03-24T22:03:06.391 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:06.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:06.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:03:06.453 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[19936]: cluster 2021-03-24T22:03:04.501644+0000 2021-03-24T22:03:06.453 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[19936]: osd.0 (osd.0) 122 : cluster [DBG] 7.9 scrub ok 2021-03-24T22:03:06.453 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:03:06.453 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[19936]: 03:04.919966+0000 mgr.y (mgr.14144) 579 : cluster [DBG] pgmap v1084: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 543 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 2.5 MiB/s wr, 19 op/s; 40/1737 objects misplaced (2.303%) 2021-03-24T22:03:06.454 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[19936]: audit 2021-03-24T22:03:05.888587+0000 mon.c (mon.2) 94 2021-03-24T22:03:06.454 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[19936]: : audit [INF] from='client.? v1:172.21.15.125:0/3868258002' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:03:06.454 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[19936]: audit 2021- 2021-03-24T22:03:06.454 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[19936]: 03-24T22:03:05.889231+0000 mon.a (mon.0) 1676 : audit [INF] from='client.35546 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:03:06.455 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[19936]: cluster 2021-03-24T22:03 2021-03-24T22:03:06.455 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[19936]: :06.317685+0000 mon.a (mon.0) 1677 : cluster [INF] 2021-03-24T22:03:06.455 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[19936]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T22:03:06.455 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[19936]: audit 2021-03-24T22:03:06.456 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[19936]: 2021-03-24T 2021-03-24T22:03:06.456 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[19936]: 22:03:06.330460+0000 mon.a (mon 2021-03-24T22:03:06.456 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[19936]: .0) 1678 : audit [INF] from='client.35546 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T22:03:06.456 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[19936]: cluster 2021-03-24T22:03:06.457 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[19936]: 2021-03-24T22:03:06.330807+0000 mon.a (mon.0) 1679 : cluster [DBG] osdmap e616: 8 total, 8 up, 8 in 2021-03-24T22:03:06.457 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[19936]: 2021-03-24T22:03:06.496 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:06.551 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:06.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:06.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:03:06.557 INFO:tasks.rados.rados.0.smithi125.stdout:2325: finishing write tid 1 to smithi12556470-11 2021-03-24T22:03:06.557 INFO:tasks.rados.rados.0.smithi125.stdout:2322: finishing copy_from to smithi12556470-45 2021-03-24T22:03:06.558 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 45 v 313 (ObjNum 775 snap 252 seq_num 775) dirty exists 2021-03-24T22:03:06.558 INFO:tasks.rados.rados.0.smithi125.stdout:2322: finishing copy_from racing read to smithi12556470-45 2021-03-24T22:03:06.559 INFO:tasks.rados.rados.0.smithi125.stdout:2325: finishing write tid 2 to smithi12556470-11 2021-03-24T22:03:06.559 INFO:tasks.rados.rados.0.smithi125.stdout:2325: finishing write tid 3 to smithi12556470-11 2021-03-24T22:03:06.559 INFO:tasks.rados.rados.0.smithi125.stdout:2325: finishing write tid 4 to smithi12556470-11 2021-03-24T22:03:06.560 INFO:tasks.rados.rados.0.smithi125.stdout:2325: finishing write tid 5 to smithi12556470-11 2021-03-24T22:03:06.560 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 11 v 402 (ObjNum 802 snap 262 seq_num 802) dirty exists 2021-03-24T22:03:06.561 INFO:tasks.rados.rados.0.smithi125.stdout:2325: left oid 11 (ObjNum 802 snap 262 seq_num 802) 2021-03-24T22:03:06.561 INFO:tasks.rados.rados.0.smithi125.stdout:2326: finishing copy_from to smithi12556470-44 2021-03-24T22:03:06.561 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 44 v 503 (ObjNum 736 snap 235 seq_num 736) dirty exists 2021-03-24T22:03:06.562 INFO:tasks.rados.rados.0.smithi125.stdout:2326: finishing copy_from racing read to smithi12556470-44 2021-03-24T22:03:06.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:06 smithi142 conmon[20125]: cluster 2021-03-24T22:03:04.501644+0000 osd.0 (osd.0) 122 : 2021-03-24T22:03:06.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:06 smithi142 conmon[20125]: cluster [DBG] 7.9 scrub ok 2021-03-24T22:03:06.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:06 smithi142 conmon[20125]: cluster 2021-03-24T22:03:04.919966+0000 mgr.y (mgr. 2021-03-24T22:03:06.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:06 smithi142 conmon[20125]: 14144) 579 : cluster [DBG] pgmap v1084: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 543 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 2.5 MiB/s wr, 19 op/s; 40/1737 objects misplaced (2.303%) 2021-03-24T22:03:06.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:06 smithi142 conmon[20125]: audit 2021-03-24T22:03: 2021-03-24T22:03:06.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:06 smithi142 conmon[20125]: 05.888587+0000 mon.c (mon. 2021-03-24T22:03:06.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:06 smithi142 conmon[20125]: 2) 94 : audit [INF] from='client.? v1:172.21.15.125:0/3868258002' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:03:06.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:06 smithi142 conmon[20125]: 2021-03-24T22:03:06.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:06 smithi142 conmon[20125]: audit 2021 2021-03-24T22:03:06.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:06 smithi142 conmon[20125]: -03-24T22:03: 2021-03-24T22:03:06.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:06 smithi142 conmon[20125]: 05.889231+0000 mon.a (mon 2021-03-24T22:03:06.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:06 smithi142 conmon[20125]: .0) 1676 : audit [INF] 2021-03-24T22:03:06.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:06 smithi142 conmon[20125]: from='client.35546 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:03:06.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:06 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:03:06.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:06 smithi142 conmon[20125]: -03-24T22:03 2021-03-24T22:03:06.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:06 smithi142 conmon[20125]: :06.317685+0000 mon.a (mon 2021-03-24T22:03:06.625 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:06 smithi142 conmon[20125]: .0) 1677 : cluster [INF] 2021-03-24T22:03:06.625 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:06 smithi142 conmon[20125]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T22:03:06.626 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:06 smithi142 conmon[20125]: audit 2021 2021-03-24T22:03:06.626 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:06 smithi142 conmon[20125]: -03-24T22:03 2021-03-24T22:03:06.627 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:06 smithi142 conmon[20125]: :06.330460+0000 mon.a ( 2021-03-24T22:03:06.627 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:06 smithi142 conmon[20125]: mon.0) 1678 : audit [INF] 2021-03-24T22:03:06.628 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:06 smithi142 conmon[20125]: from='client.35546 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T22:03:06.628 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:06 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:03:06.628 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:06 smithi142 conmon[20125]: 22:03:06.330807+0000 mon.a (mon 2021-03-24T22:03:06.629 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:06 smithi142 conmon[20125]: .0) 1679 : cluster [DBG] osdmap e616: 8 total, 8 up, 8 in 2021-03-24T22:03:06.658 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:06.712 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:06.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:06.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:03:06.715 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[24927]: cluster 2021-03-24T22:03:04.501644 2021-03-24T22:03:06.716 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[24927]: +0000 osd.0 (osd.0) 122 : cluster [DBG] 7.9 scrub ok 2021-03-24T22:03:06.716 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[24927]: cluster 2021-03-24T22:03:04. 2021-03-24T22:03:06.716 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[24927]: 919966+0000 mgr.y (mgr.14144) 579 : cluster [DBG] pgmap v1084: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 543 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 2.5 MiB/s wr, 19 op/s; 40/1737 objects misplaced (2.303%) 2021-03-24T22:03:06.716 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[24927]: audit 2021-03-24T 2021-03-24T22:03:06.716 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[24927]: 22:03:05.888587+0000 mon.c (mon.2 2021-03-24T22:03:06.717 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[24927]: ) 94 : audit [INF] from='client.? v1:172.21.15.125:0/3868258002' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:03:06.717 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[24927]: audit 2021 2021-03-24T22:03:06.717 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[24927]: -03-24T22:03:05.889231+0000 mon.a 2021-03-24T22:03:06.717 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[24927]: (mon.0) 1676 : audit [INF] from='client.35546 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:03:06.718 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:03:06.718 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[24927]: -03-24T22:03:06.317685+0000 mon.a 2021-03-24T22:03:06.718 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[24927]: (mon.0) 1677 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T22:03:06.718 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[24927]: audit 2021-03-24T22:03:06.719 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[24927]: 2021-03-24T22:03:06.330460+0000 2021-03-24T22:03:06.719 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[24927]: mon.a (mon.0) 1678 : audit [INF] from='client.35546 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T22:03:06.719 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[24927]: cluster 2021-03-24T22:03:06.719 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[24927]: 2021-03-24T22:03:06.720 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[24927]: 2021 2021-03-24T22:03:06.720 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[24927]: - 2021-03-24T22:03:06.720 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[24927]: 03 2021-03-24T22:03:06.720 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[24927]: - 2021-03-24T22:03:06.720 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[24927]: 24 2021-03-24T22:03:06.721 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[24927]: T 2021-03-24T22:03:06.721 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[24927]: 22 2021-03-24T22:03:06.721 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[24927]: : 2021-03-24T22:03:06.721 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[24927]: 03 2021-03-24T22:03:06.721 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[24927]: : 2021-03-24T22:03:06.722 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[24927]: 06 2021-03-24T22:03:06.722 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[24927]: . 2021-03-24T22:03:06.722 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[24927]: 330807 2021-03-24T22:03:06.722 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[24927]: +0000 2021-03-24T22:03:06.723 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[24927]: 2021-03-24T22:03:06.723 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[24927]: mon.a 2021-03-24T22:03:06.723 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[24927]: ( 2021-03-24T22:03:06.723 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[24927]: mon 2021-03-24T22:03:06.723 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[24927]: . 2021-03-24T22:03:06.724 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[24927]: 0 2021-03-24T22:03:06.724 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[24927]: ) 2021-03-24T22:03:06.724 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[24927]: 1679 2021-03-24T22:03:06.724 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[24927]: : 2021-03-24T22:03:06.724 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[24927]: cluster 2021-03-24T22:03:06.725 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[24927]: 2021-03-24T22:03:06.725 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[24927]: [DBG] 2021-03-24T22:03:06.725 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[24927]: 2021-03-24T22:03:06.725 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[24927]: osdmap e616: 8 total, 8 up, 8 in 2021-03-24T22:03:06.726 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:06 smithi125 conmon[24927]: 2021-03-24T22:03:06.815 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:06.867 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:06.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:06.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:03:06.973 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:07.035 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:07.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:07.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:03:07.140 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:07.191 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:07.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:07.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:03:07.297 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:07.334 INFO:tasks.rados.rados.0.smithi125.stdout:2321: done (6 left) 2021-03-24T22:03:07.335 INFO:tasks.rados.rados.0.smithi125.stdout:2322: done (5 left) 2021-03-24T22:03:07.335 INFO:tasks.rados.rados.0.smithi125.stdout:2323: done (4 left) 2021-03-24T22:03:07.335 INFO:tasks.rados.rados.0.smithi125.stdout:2324: done (3 left) 2021-03-24T22:03:07.335 INFO:tasks.rados.rados.0.smithi125.stdout:2325: done (2 left) 2021-03-24T22:03:07.336 INFO:tasks.rados.rados.0.smithi125.stdout:2326: done (1 left) 2021-03-24T22:03:07.336 INFO:tasks.rados.rados.0.smithi125.stdout:2327: done (0 left) 2021-03-24T22:03:07.336 INFO:tasks.rados.rados.0.smithi125.stdout:2328: snap_create 2021-03-24T22:03:07.354 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:07.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:07.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:07.459 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:07.513 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:07.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:07.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:07.620 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:07.682 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:07.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:07.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:03:07.698 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:03:07.788 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:07.800 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:07 smithi125 conmon[19936]: cluster 2021-03-24T22:03: 2021-03-24T22:03:07.800 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:07 smithi125 conmon[19936]: 05.892400+0000 osd.5 (osd.5) 33 : cluster [DBG] 3.5 scrub ok 2021-03-24T22:03:07.801 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:07 smithi125 conmon[19936]: cluster 2021-03-24T22:03:06.420784+0000 2021-03-24T22:03:07.801 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:07 smithi125 conmon[19936]: osd.1 (osd.1) 59 : cluster [DBG] 6.1d scrub ok 2021-03-24T22:03:07.801 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:07 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:03:07.801 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:07 smithi125 conmon[19936]: 03-24T22:03:06.473109+0000 osd.0 (osd.0) 123 : cluster [DBG] 2021-03-24T22:03:07.802 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:07 smithi125 conmon[19936]: 7.4 deep-scrub ok 2021-03-24T22:03:07.802 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:07 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:03:07.802 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:07 smithi125 conmon[19936]: T22:03:07.328131+0000 mon.a (mon.0) 1680 : cluster 2021-03-24T22:03:07.802 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:07 smithi125 conmon[19936]: [DBG] osdmap e617: 8 total, 8 up, 8 in 2021-03-24T22:03:07.803 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:07 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:03:07.803 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:07 smithi125 conmon[24927]: 24T22:03:05.892400+0000 osd.5 (osd.5) 33 : cluster 2021-03-24T22:03:07.804 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:07 smithi125 conmon[24927]: [DBG] 3.5 scrub ok 2021-03-24T22:03:07.804 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:07 smithi125 conmon[24927]: cluster 2021-03-24T22:03:07.804 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:07 smithi125 conmon[24927]: 2021-03-24T 2021-03-24T22:03:07.805 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:07 smithi125 conmon[24927]: 22:03:06.420784+0000 2021-03-24T22:03:07.805 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:07 smithi125 conmon[24927]: osd.1 (osd.1) 59 : 2021-03-24T22:03:07.805 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:07 smithi125 conmon[24927]: cluster [DBG] 6.1d scrub ok 2021-03-24T22:03:07.805 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:07 smithi125 conmon[24927]: cluster 2021-03-24T22:03:07.805 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:07 smithi125 conmon[24927]: 2021-03-24T22:03: 2021-03-24T22:03:07.806 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:07 smithi125 conmon[24927]: 06.473109+0000 osd.0 (osd 2021-03-24T22:03:07.806 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:07 smithi125 conmon[24927]: .0) 123 : cluster [DBG] 2021-03-24T22:03:07.806 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:07 smithi125 conmon[24927]: 7.4 deep-scrub ok 2021-03-24T22:03:07.806 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:07 smithi125 conmon[24927]: cluster 2021-03-24T22:03:07.807 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:07 smithi125 conmon[24927]: 2021-03-24T22:03 2021-03-24T22:03:07.807 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:07 smithi125 conmon[24927]: :07.328131+0000 mon.a ( 2021-03-24T22:03:07.807 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:07 smithi125 conmon[24927]: mon.0) 1680 : cluster [DBG] 2021-03-24T22:03:07.808 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:07 smithi125 conmon[24927]: osdmap e617: 8 total, 8 up, 8 in 2021-03-24T22:03:07.840 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:07.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:07.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:03:07.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:07 smithi142 conmon[20125]: cluster 2021-03-24T22:03 2021-03-24T22:03:07.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:07 smithi142 conmon[20125]: :05.892400+0000 osd.5 (osd.5) 33 : cluster [DBG] 3.5 scrub ok 2021-03-24T22:03:07.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:07 smithi142 conmon[20125]: cluster 2021-03-24T22:03:06 2021-03-24T22:03:07.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:07 smithi142 conmon[20125]: .420784+0000 osd.1 (osd.1) 59 : cluster [DBG] 6.1d scrub ok 2021-03-24T22:03:07.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:07 smithi142 conmon[20125]: cluster 2021-03-24T22:03:06.473109+0000 2021-03-24T22:03:07.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:07 smithi142 conmon[20125]: osd.0 (osd.0) 123 : cluster [DBG] 7.4 deep-scrub ok 2021-03-24T22:03:07.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:07 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:03:07.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:07 smithi142 conmon[20125]: 24T22:03:07.328131+0000 mon.a (mon.0) 2021-03-24T22:03:07.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:07 smithi142 conmon[20125]: 1680 : cluster [DBG] osdmap e617: 8 total, 8 up, 8 in 2021-03-24T22:03:07.945 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:07.997 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:08.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:08.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:03:08.102 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:08.118 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-24T22:03:08.118 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-24T22:03:08.119 INFO:teuthology.orchestra.run.smithi125.stdout: "complaint_time": 30, 2021-03-24T22:03:08.119 INFO:teuthology.orchestra.run.smithi125.stdout: "num_blocked_ops": 0 2021-03-24T22:03:08.119 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-24T22:03:08.159 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:08.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:08.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:03:08.265 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:08.323 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:08.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:08.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:03:08.334 INFO:tasks.rados.rados.0.smithi125.stdout:2328: done (0 left) 2021-03-24T22:03:08.334 INFO:tasks.rados.rados.0.smithi125.stdout:2329: read oid 42 snap -1 2021-03-24T22:03:08.334 INFO:tasks.rados.rados.0.smithi125.stdout:2329: expect (ObjNum 736 snap 235 seq_num 736) 2021-03-24T22:03:08.335 INFO:tasks.rados.rados.0.smithi125.stdout:2330: read oid 12 snap -1 2021-03-24T22:03:08.335 INFO:tasks.rados.rados.0.smithi125.stdout:2330: expect (ObjNum 752 snap 241 seq_num 752) 2021-03-24T22:03:08.335 INFO:tasks.rados.rados.0.smithi125.stdout:2331: read oid 14 snap 257 2021-03-24T22:03:08.335 INFO:tasks.rados.rados.0.smithi125.stdout:2331: expect deleted 2021-03-24T22:03:08.336 INFO:tasks.rados.rados.0.smithi125.stdout:2332: copy_from oid 30 from oid 35 current snap is 264 2021-03-24T22:03:08.336 INFO:tasks.rados.rados.0.smithi125.stdout:2333: write (excl) oid 2 current snap is 264 2021-03-24T22:03:08.336 INFO:tasks.rados.rados.0.smithi125.stdout:2333: seq_num 803 ranges {696021=642012,1856570=575278,3185369=310652} 2021-03-24T22:03:08.342 INFO:tasks.rados.rados.0.smithi125.stdout:2333: writing smithi12556470-2 from 696021 to 1338033 tid 1 2021-03-24T22:03:08.350 INFO:tasks.rados.rados.0.smithi125.stdout:2333: writing smithi12556470-2 from 1856570 to 2431848 tid 2 2021-03-24T22:03:08.357 INFO:tasks.rados.rados.0.smithi125.stdout:2333: writing smithi12556470-2 from 3185369 to 3496021 tid 3 2021-03-24T22:03:08.358 INFO:tasks.rados.rados.0.smithi125.stdout:2334: write (excl) oid 16 current snap is 264 2021-03-24T22:03:08.358 INFO:tasks.rados.rados.0.smithi125.stdout:2334: seq_num 804 ranges {712828=717261,1915052=718936,3212792=300036} 2021-03-24T22:03:08.365 INFO:tasks.rados.rados.0.smithi125.stdout:2334: writing smithi12556470-16 from 712828 to 1430089 tid 1 2021-03-24T22:03:08.373 INFO:tasks.rados.rados.0.smithi125.stdout:2334: writing smithi12556470-16 from 1915052 to 2633988 tid 2 2021-03-24T22:03:08.379 INFO:tasks.rados.rados.0.smithi125.stdout:2334: writing smithi12556470-16 from 3212792 to 3512828 tid 3 2021-03-24T22:03:08.379 INFO:tasks.rados.rados.0.smithi125.stdout:2335: snap_remove snap 168 2021-03-24T22:03:08.429 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:08.489 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:08.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:08.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:03:08.596 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:08.659 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:08.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:08 smithi125 conmon[24927]: cluster 2021-03-24T22:03: 2021-03-24T22:03:08.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:08 smithi125 conmon[24927]: 06.920998+0000 mgr.y (mgr.14144) 580 : cluster [DBG] pgmap v1087: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+remapped, 166 active+clean; 544 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.6 MiB/s rd, 1.1 MiB/s wr, 8 op/s; 40/1740 objects misplaced (2.299%); 2.2 MiB/s, 4 objects/s recovering 2021-03-24T22:03:08.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:08 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T22:03:08.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:08 smithi125 conmon[24927]: T22:03:07.709065+0000 osd.6 (osd.6) 46 : cluster [DBG] 5.1f scrub ok 2021-03-24T22:03:08.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:08 smithi125 conmon[24927]: cluster 2021-03-24T22:03:08.328985+0000 2021-03-24T22:03:08.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:08 smithi125 conmon[24927]: mon.a (mon.0) 1681 : cluster [DBG] osdmap e618: 8 total, 8 up, 8 in 2021-03-24T22:03:08.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:08 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:03:08.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:08 smithi125 conmon[19936]: 03:06.920998+0000 mgr.y (mgr.14144) 580 : cluster [DBG] pgmap v1087: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+remapped, 166 active+clean; 544 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.6 MiB/s rd, 1.1 MiB/s wr, 8 op/s; 40/1740 objects misplaced (2.299%); 2.2 MiB/s, 4 objects/s recovering 2021-03-24T22:03:08.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:08 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:03:08.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:08 smithi125 conmon[19936]: T22:03:07.709065+0000 osd.6 (osd.6 2021-03-24T22:03:08.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:08 smithi125 conmon[19936]: ) 46 : cluster [DBG] 5.1f scrub ok 2021-03-24T22:03:08.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:08 smithi125 conmon[19936]: cluster 2021-03-24T22:03:08.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:08 smithi125 conmon[19936]: 2021-03-24T22:03:08.328985+0000 2021-03-24T22:03:08.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:08 smithi125 conmon[19936]: mon.a (mon.0) 1681 : cluster [DBG] 2021-03-24T22:03:08.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:08 smithi125 conmon[19936]: osdmap e618: 8 total, 8 up, 8 in 2021-03-24T22:03:08.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:08.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:03:08.768 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:08.829 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:08.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:08.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:03:08.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:08 smithi142 conmon[20125]: cluster 2021-03-24T22:03:06.920998+0000 mgr.y (mgr.14144) 580 : cluster [DBG] pgmap v1087: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+remapped, 166 active+clean; 544 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.6 MiB/s rd, 1.1 MiB/s wr, 8 op/s; 40/1740 objects misplaced (2.299%); 2.2 MiB/s, 4 objects/s recovering 2021-03-24T22:03:08.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:08 smithi142 conmon[20125]: cluster 2021-03-24T22:03:07.709065+0000 osd.6 (osd.6) 46 : cluster [DBG] 5.1f scrub ok 2021-03-24T22:03:08.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:08 smithi142 conmon[20125]: cluster 2021-03-24T22:03:08.328985+0000 mon.a (mon.0) 2021-03-24T22:03:08.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:08 smithi142 conmon[20125]: 1681 : cluster [DBG] osdmap e618: 8 total, 8 up, 8 in 2021-03-24T22:03:08.936 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:08.996 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:08.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:08.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:03:09.100 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:09.155 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:09.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:09.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:03:09.262 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:09.325 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:09.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:09.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:03:09.346 INFO:tasks.rados.rados.0.smithi125.stdout:2335: done (6 left) 2021-03-24T22:03:09.346 INFO:tasks.rados.rados.0.smithi125.stdout:2336: delete oid 38 current snap is 264 2021-03-24T22:03:09.347 INFO:tasks.rados.rados.0.smithi125.stdout:2329: expect (ObjNum 736 snap 235 seq_num 736) 2021-03-24T22:03:09.432 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:09.434 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.0 dump_historic_ops 2021-03-24T22:03:09.462 INFO:tasks.rados.rados.0.smithi125.stdout:2329: done (6 left) 2021-03-24T22:03:09.462 INFO:tasks.rados.rados.0.smithi125.stdout:2331: done (5 left) 2021-03-24T22:03:09.462 INFO:tasks.rados.rados.0.smithi125.stdout:2336: done (4 left) 2021-03-24T22:03:09.463 INFO:tasks.rados.rados.0.smithi125.stdout:2337: snap_remove snap 257 2021-03-24T22:03:09.483 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:09.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:09.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:03:09.588 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:09.610 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T22:03:09.636 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T22:03:09.637 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:09.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:09.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:03:09.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:09 smithi125 conmon[19936]: cluster 2021-03-24T22:03:07.837663+0000 osd.5 (osd.5) 34 : cluster [DBG] 3.28 scrub ok 2021-03-24T22:03:09.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:09 smithi125 conmon[19936]: cluster 2021-03-24T22:03:08.379576+0000 osd.7 (osd.7) 101 : cluster [DBG] 6.12 scrub ok 2021-03-24T22:03:09.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:09 smithi125 conmon[19936]: cluster 2021-03-24T22:03:08.392052+0000 osd.3 (osd.3) 126 : cluster [DBG] 6.1b scrub ok 2021-03-24T22:03:09.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:09 smithi125 conmon[19936]: cluster 2021-03-24T22:03:08.699460+0000 osd.2 (osd.2) 96 : cluster [DBG] 2.1 scrub ok 2021-03-24T22:03:09.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:09 smithi125 conmon[19936]: cluster 2021-03-24T22:03:09.337126+0000 mon.a (mon.0) 1682 : cluster [DBG] osdmap e619: 8 total, 8 up, 8 in 2021-03-24T22:03:09.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:09 smithi125 conmon[24927]: cluster 2021-03-24T22:03:07.837663+0000 osd.5 (osd. 2021-03-24T22:03:09.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:09 smithi125 conmon[24927]: 5) 34 : cluster [DBG] 3.28 scrub ok 2021-03-24T22:03:09.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:09 smithi125 conmon[24927]: cluster 2021-03-24T22:03:08.379576+0000 osd.7 (osd.7) 101 : cluster [DBG] 6.12 scrub ok 2021-03-24T22:03:09.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:09 smithi125 conmon[24927]: cluster 2021-03-24T22:03:08.392052+0000 osd.3 (osd.3) 2021-03-24T22:03:09.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:09 smithi125 conmon[24927]: 126 : cluster [DBG] 6.1b scrub ok 2021-03-24T22:03:09.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:09 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:03:09.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:09 smithi125 conmon[24927]: 24T22:03:08.699460+0000 osd.2 (osd. 2021-03-24T22:03:09.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:09 smithi125 conmon[24927]: 2) 96 : cluster [DBG] 2.1 scrub ok 2021-03-24T22:03:09.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:09 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:03:09.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:09 smithi125 conmon[24927]: -03-24T22:03:09. 2021-03-24T22:03:09.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:09 smithi125 conmon[24927]: 337126+0000 mon.a (mon.0) 1682 : cluster [DBG] 2021-03-24T22:03:09.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:09 smithi125 conmon[24927]: osdmap e619: 8 total, 8 up, 8 in 2021-03-24T22:03:09.671 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T22:03:09.693 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T22:03:09.708 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd set noscrub 2021-03-24T22:03:09.717 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T22:03:09.740 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T22:03:09.744 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:09.758 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:09 smithi142 conmon[20125]: cluster 2021-03-24T22:03:07.837663+0000 osd.5 (osd.5) 34 : cluster [DBG] 2021-03-24T22:03:09.758 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:09 smithi142 conmon[20125]: 3.28 scrub ok 2021-03-24T22:03:09.758 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:09 smithi142 conmon[20125]: cluster 2021-03-24T22:03:08.379576+0000 osd.7 (osd.7) 2021-03-24T22:03:09.759 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:09 smithi142 conmon[20125]: 101 : cluster [DBG] 6.12 scrub ok 2021-03-24T22:03:09.759 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:09 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:03:09.759 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:09 smithi142 conmon[20125]: -03-24T22:03:08. 2021-03-24T22:03:09.759 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:09 smithi142 conmon[20125]: 392052+0000 osd.3 (osd.3 2021-03-24T22:03:09.760 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:09 smithi142 conmon[20125]: ) 126 : cluster [DBG] 6.1b scrub ok 2021-03-24T22:03:09.760 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:09 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:03:09.760 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:09 smithi142 conmon[20125]: -03-24T22:03: 2021-03-24T22:03:09.760 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:09 smithi142 conmon[20125]: 08.699460+0000 osd.2 (osd 2021-03-24T22:03:09.760 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:09 smithi142 conmon[20125]: .2) 96 : cluster [DBG] 2021-03-24T22:03:09.761 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:09 smithi142 conmon[20125]: 2.1 scrub ok 2021-03-24T22:03:09.761 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:09 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:03:09.761 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:09 smithi142 conmon[20125]: -03-24T22:03 2021-03-24T22:03:09.761 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:09 smithi142 conmon[20125]: :09.337126+0000 mon.a ( 2021-03-24T22:03:09.762 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:09 smithi142 conmon[20125]: mon.0) 1682 : cluster [DBG] 2021-03-24T22:03:09.762 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:09 smithi142 conmon[20125]: osdmap e619: 8 total, 8 up, 8 in 2021-03-24T22:03:09.764 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T22:03:09.787 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T22:03:09.789 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-24T22:03:09.789 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-24T22:03:09.789 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-24T22:03:09.789 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-24T22:03:09.790 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-24T22:03:09.790 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-24T22:03:09.790 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-24T22:03:09.790 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-24T22:03:09.798 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:09.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:09.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:03:09.905 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:09.962 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:09.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:09.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:03:10.069 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:10.122 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:10.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:10.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:10.228 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:10.283 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:10.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:10.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:03:10.334 INFO:tasks.rados.rados.0.smithi125.stdout:2337: done (4 left) 2021-03-24T22:03:10.334 INFO:tasks.rados.rados.0.smithi125.stdout:2338: read oid 21 snap -1 2021-03-24T22:03:10.335 INFO:tasks.rados.rados.0.smithi125.stdout:2338: expect deleted 2021-03-24T22:03:10.335 INFO:tasks.rados.rados.0.smithi125.stdout:2339: snap_remove snap 263 2021-03-24T22:03:10.390 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:10.444 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:10.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:10.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:03:10.550 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:10.610 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:10.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:10.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:03:10.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:10 smithi125 conmon[24927]: cluster 2021-03-24T22:03:08.921766+0000 mgr.y 2021-03-24T22:03:10.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:10 smithi125 conmon[24927]: (mgr.14144) 581 : cluster [DBG] pgmap v1090: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+remapped, 166 active+clean; 544 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 331 KiB/s wr, 1 op/s; 40/1740 objects misplaced (2.299%); 2.2 MiB/s, 4 objects/s recovering 2021-03-24T22:03:10.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:10 smithi125 conmon[24927]: cluster 2021-03-24T22:03:09.417455+0000 osd.7 (osd.7) 102 : cluster [DBG] 3.15 deep-scrub ok 2021-03-24T22:03:10.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:10 smithi125 conmon[24927]: 2021-03-24T22:03:10.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:10 smithi125 conmon[24927]: audit 2021-03-24T22:03:09.582532+0000 mon.c (mon.2) 95 : audit [DBG] from='client.? v1:172.21.15.125:0/1061864561' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:03:10.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:10 smithi125 conmon[24927]: cluster 2021-03-24T22:03:09.686042+0000 osd.2 (osd.2) 97 : cluster [DBG] 6.14 scrub ok 2021-03-24T22:03:10.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:10 smithi125 conmon[24927]: cluster 2021-03-24T22:03:10.329584+0000 mon.a (mon.0 2021-03-24T22:03:10.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:10 smithi125 conmon[24927]: ) 1683 : cluster [DBG] osdmap e620: 8 total, 8 up, 8 in 2021-03-24T22:03:10.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:10 smithi125 conmon[19936]: cluster 2021-03-24T22:03: 2021-03-24T22:03:10.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:10 smithi125 conmon[19936]: 08.921766+0000 mgr.y (mgr.14144) 581 : cluster [DBG] pgmap v1090: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+remapped, 166 active+clean; 544 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 331 KiB/s wr, 1 op/s; 40/1740 objects misplaced (2.299%); 2.2 MiB/s, 4 objects/s recovering 2021-03-24T22:03:10.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:10 smithi125 conmon[19936]: cluster 2021-03-24T22:03 2021-03-24T22:03:10.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:10 smithi125 conmon[19936]: :09.417455+0000 osd.7 (osd. 2021-03-24T22:03:10.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:10 smithi125 conmon[19936]: 7) 102 : cluster [DBG] 3.15 deep-scrub ok 2021-03-24T22:03:10.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:10 smithi125 conmon[19936]: 2021-03-24T22:03:10.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:10 smithi125 conmon[19936]: audit 2021-03-24T22:03:10.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:10 smithi125 conmon[19936]: 2021-03- 2021-03-24T22:03:10.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:10 smithi125 conmon[19936]: 24T22:03:09.582532 2021-03-24T22:03:10.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:10 smithi125 conmon[19936]: +0000 mon.c (mon.2) 2021-03-24T22:03:10.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:10 smithi125 conmon[19936]: 95 : audit [DBG] from='client.? v1:172.21.15.125:0/1061864561' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:03:10.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:10 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:03:10.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:10 smithi125 conmon[19936]: -03-24T22:03:09.686042 2021-03-24T22:03:10.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:10 smithi125 conmon[19936]: +0000 osd.2 (osd.2) 97 2021-03-24T22:03:10.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:10 smithi125 conmon[19936]: : cluster [DBG] 6.14 scrub ok 2021-03-24T22:03:10.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:10 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:03:10.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:10 smithi125 conmon[19936]: 03-24T22:03:10.329584+0000 2021-03-24T22:03:10.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:10 smithi125 conmon[19936]: mon.a (mon.0) 1683 : cluster [DBG] 2021-03-24T22:03:10.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:10 smithi125 conmon[19936]: osdmap e620: 8 total, 8 up, 8 in 2021-03-24T22:03:10.716 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:10.730 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:10 smithi142 conmon[20125]: cluster 2021-03-24T22:03:08.921766+0000 2021-03-24T22:03:10.730 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:10 smithi142 conmon[20125]: mgr.y (mgr.14144) 581 : cluster [DBG] pgmap v1090: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+remapped, 166 active+clean; 544 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 331 KiB/s wr, 1 op/s; 40/1740 objects misplaced (2.299%); 2.2 MiB/s, 4 objects/s recovering 2021-03-24T22:03:10.730 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:10 smithi142 conmon[20125]: cluster 2021-03-24T22:03:09.417455+0000 osd.7 (osd.7) 102 : cluster [DBG] 3.15 deep-scrub ok 2021-03-24T22:03:10.731 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:10 smithi142 conmon[20125]: 2021-03-24T22:03:10.731 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:10 smithi142 conmon[20125]: audit 2021-03-24T22:03:09.582532+0000 mon.c (mon.2) 95 2021-03-24T22:03:10.731 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:10 smithi142 conmon[20125]: : audit [DBG] from='client.? v1:172.21.15.125:0/1061864561' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:03:10.731 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:10 smithi142 conmon[20125]: cluster 2021-03-24T22:03:09 2021-03-24T22:03:10.731 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:10 smithi142 conmon[20125]: .686042+0000 osd.2 (osd.2) 97 : cluster [DBG] 6.14 scrub ok 2021-03-24T22:03:10.732 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:10 smithi142 conmon[20125]: cluster 2021-03-24T22:03:10.329584+0000 mon.a (mon. 2021-03-24T22:03:10.732 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:10 smithi142 conmon[20125]: 0) 1683 : cluster [DBG] osdmap e620: 8 total, 8 up, 8 in 2021-03-24T22:03:10.774 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:10.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:10.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:03:10.880 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:10.943 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:10.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:10.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:03:11.051 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:11.108 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:11.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:11.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:11.215 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:11.272 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:11.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:11.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:03:11.347 INFO:tasks.rados.rados.0.smithi125.stdout:2339: done (5 left) 2021-03-24T22:03:11.348 INFO:tasks.rados.rados.0.smithi125.stdout:2340: snap_create 2021-03-24T22:03:11.349 INFO:tasks.rados.rados.0.smithi125.stdout:2330: expect (ObjNum 752 snap 241 seq_num 752) 2021-03-24T22:03:11.380 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:11.431 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:11.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:11.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:03:11.473 INFO:tasks.rados.rados.0.smithi125.stdout:2333: finishing write tid 1 to smithi12556470-2 2021-03-24T22:03:11.473 INFO:tasks.rados.rados.0.smithi125.stdout:2333: finishing write tid 2 to smithi12556470-2 2021-03-24T22:03:11.474 INFO:tasks.rados.rados.0.smithi125.stdout:2333: finishing write tid 3 to smithi12556470-2 2021-03-24T22:03:11.474 INFO:tasks.rados.rados.0.smithi125.stdout:2333: finishing write tid 4 to smithi12556470-2 2021-03-24T22:03:11.475 INFO:tasks.rados.rados.0.smithi125.stdout:2333: finishing write tid 5 to smithi12556470-2 2021-03-24T22:03:11.475 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 2 v 823 (ObjNum 803 snap 264 seq_num 803) dirty exists 2021-03-24T22:03:11.476 INFO:tasks.rados.rados.0.smithi125.stdout:2333: left oid 2 (ObjNum 803 snap 264 seq_num 803) 2021-03-24T22:03:11.476 INFO:tasks.rados.rados.0.smithi125.stdout:2332: finishing copy_from to smithi12556470-30 2021-03-24T22:03:11.476 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 30 v 223 (ObjNum 786 snap 257 seq_num 786) dirty exists 2021-03-24T22:03:11.477 INFO:tasks.rados.rados.0.smithi125.stdout:2332: finishing copy_from racing read to smithi12556470-30 2021-03-24T22:03:11.477 INFO:tasks.rados.rados.0.smithi125.stdout:2334: finishing write tid 1 to smithi12556470-16 2021-03-24T22:03:11.478 INFO:tasks.rados.rados.0.smithi125.stdout:2334: finishing write tid 2 to smithi12556470-16 2021-03-24T22:03:11.478 INFO:tasks.rados.rados.0.smithi125.stdout:2334: finishing write tid 3 to smithi12556470-16 2021-03-24T22:03:11.478 INFO:tasks.rados.rados.0.smithi125.stdout:2334: finishing write tid 4 to smithi12556470-16 2021-03-24T22:03:11.479 INFO:tasks.rados.rados.0.smithi125.stdout:2334: finishing write tid 5 to smithi12556470-16 2021-03-24T22:03:11.479 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 16 v 448 (ObjNum 804 snap 264 seq_num 804) dirty exists 2021-03-24T22:03:11.480 INFO:tasks.rados.rados.0.smithi125.stdout:2334: left oid 16 (ObjNum 804 snap 264 seq_num 804) 2021-03-24T22:03:11.537 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:11.594 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:11.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:11.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:03:11.689 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:03:11.701 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:11.736 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:03:11.760 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:11.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:11.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:03:11.867 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:11.926 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:11.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:11.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:03:12.032 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:12.087 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:12.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:12.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:03:12.192 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:12.249 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:12.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:12.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:03:12.352 INFO:tasks.rados.rados.0.smithi125.stdout:2330: done (5 left) 2021-03-24T22:03:12.353 INFO:tasks.rados.rados.0.smithi125.stdout:2332: done (4 left) 2021-03-24T22:03:12.353 INFO:tasks.rados.rados.0.smithi125.stdout:2333: done (3 left) 2021-03-24T22:03:12.353 INFO:tasks.rados.rados.0.smithi125.stdout:2334: done (2 left) 2021-03-24T22:03:12.353 INFO:tasks.rados.rados.0.smithi125.stdout:2338: done (1 left) 2021-03-24T22:03:12.354 INFO:tasks.rados.rados.0.smithi125.stdout:2340: done (0 left) 2021-03-24T22:03:12.354 INFO:tasks.rados.rados.0.smithi125.stdout:2341: write (excl) oid 50 current snap is 265 2021-03-24T22:03:12.354 INFO:tasks.rados.rados.0.smithi125.stdout:2341: seq_num 806 ranges {746442=584112,1748369=722605,2983991=562451} 2021-03-24T22:03:12.356 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:12.361 INFO:tasks.rados.rados.0.smithi125.stdout:2341: writing smithi12556470-50 from 746442 to 1330554 tid 1 2021-03-24T22:03:12.369 INFO:tasks.rados.rados.0.smithi125.stdout:2341: writing smithi12556470-50 from 1748369 to 2470974 tid 2 2021-03-24T22:03:12.375 INFO:tasks.rados.rados.0.smithi125.stdout:2341: writing smithi12556470-50 from 2983991 to 3546442 tid 3 2021-03-24T22:03:12.376 INFO:tasks.rados.rados.0.smithi125.stdout:2342: delete oid 14 current snap is 265 2021-03-24T22:03:12.376 INFO:tasks.rados.rados.0.smithi125.stdout:2341: finishing write tid 1 to smithi12556470-50 2021-03-24T22:03:12.376 INFO:tasks.rados.rados.0.smithi125.stdout:2341: finishing write tid 2 to smithi12556470-50 2021-03-24T22:03:12.380 INFO:tasks.rados.rados.0.smithi125.stdout:2342: done (1 left) 2021-03-24T22:03:12.380 INFO:tasks.rados.rados.0.smithi125.stdout:2343: copy_from oid 43 from oid 37 current snap is 265 2021-03-24T22:03:12.380 INFO:tasks.rados.rados.0.smithi125.stdout:2341: finishing write tid 3 to smithi12556470-50 2021-03-24T22:03:12.380 INFO:tasks.rados.rados.0.smithi125.stdout:2344: rollback oid 47 current snap is 265 2021-03-24T22:03:12.381 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 47 to 255 2021-03-24T22:03:12.381 INFO:tasks.rados.rados.0.smithi125.stdout:2345: write oid 1 current snap is 265 2021-03-24T22:03:12.381 INFO:tasks.rados.rados.0.smithi125.stdout:2345: seq_num 808 ranges {780056=734610,2148980=726274,3322484=257572} 2021-03-24T22:03:12.390 INFO:tasks.rados.rados.0.smithi125.stdout:2345: writing smithi12556470-1 from 780056 to 1514666 tid 1 2021-03-24T22:03:12.401 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-24T22:03:12.405 INFO:tasks.rados.rados.0.smithi125.stdout:2345: writing smithi12556470-1 from 2148980 to 2875254 tid 2 2021-03-24T22:03:12.405 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2021-03-24T22:03:12.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:12 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:03:12.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:12 smithi125 conmon[24927]: 03:10.922501+0000 mgr.y (mgr.14144) 582 : cluster [DBG] pgmap v1093: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 532 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.9 MiB/s rd, 1.9 MiB/s wr, 8 op/s; 40/1725 objects misplaced (2.319%); 312 B/s, 2 objects/s recovering 2021-03-24T22:03:12.406 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:12 smithi125 conmon[24927]: cluster 2021-03-24T22:03:12.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:12 smithi125 conmon[24927]: 2021-03-24T22:03 2021-03-24T22:03:12.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:12 smithi125 conmon[24927]: :11.343349+0000 mon.a (mon. 2021-03-24T22:03:12.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:12 smithi125 conmon[24927]: 0) 1684 : cluster [DBG] osdmap e621: 8 total, 8 up, 8 in 2021-03-24T22:03:12.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:12 smithi125 conmon[24927]: cluster 2021-03-24T22:03:12.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:12 smithi125 conmon[24927]: 2021-03-24T22: 2021-03-24T22:03:12.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:12 smithi125 conmon[24927]: 03:11.533312+0000 2021-03-24T22:03:12.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:12 smithi125 conmon[24927]: osd.1 (osd.1) 60 2021-03-24T22:03:12.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:12 smithi125 conmon[24927]: : cluster [DBG] 6.4 scrub ok 2021-03-24T22:03:12.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:12 smithi125 conmon[24927]: cluster 2021-03-24T22:03:12.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:12 smithi125 conmon[24927]: 2021-03-24T22:03 2021-03-24T22:03:12.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:12 smithi125 conmon[24927]: :11.749434+0000 osd.2 2021-03-24T22:03:12.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:12 smithi125 conmon[24927]: (osd.2) 98 : cluster [DBG] 2021-03-24T22:03:12.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:12 smithi125 conmon[24927]: 7.20 scrub ok 2021-03-24T22:03:12.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:12 smithi125 conmon[19936]: cluster 2021-03-24T22:03:10.922501+0000 mgr.y 2021-03-24T22:03:12.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:12 smithi125 conmon[19936]: (mgr.14144) 582 : cluster [DBG] pgmap v1093: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 532 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.9 MiB/s rd, 1.9 MiB/s wr, 8 op/s; 40/1725 objects misplaced (2.319%); 312 B/s, 2 objects/s recovering 2021-03-24T22:03:12.410 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:12 smithi125 conmon[19936]: cluster 2021-03-24T22:03:11 2021-03-24T22:03:12.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:12 smithi125 conmon[19936]: .343349+0000 mon.a (mon.0) 1684 : cluster [DBG] osdmap e621: 8 total, 8 up, 8 in 2021-03-24T22:03:12.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:12 smithi125 conmon[19936]: cluster 2021-03-24T22:03:12.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:12 smithi125 conmon[19936]: 2021-03-24T22:03:11.533312+0000 osd.1 (osd.1) 60 : cluster 2021-03-24T22:03:12.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:12 smithi125 conmon[19936]: [DBG] 6.4 scrub ok 2021-03-24T22:03:12.411 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:12 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:03:12.412 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:12 smithi125 conmon[19936]: 03-24T22:03:11.749434+0000 osd.2 (osd.2) 98 : cluster [DBG] 7.20 scrub ok 2021-03-24T22:03:12.413 INFO:tasks.rados.rados.0.smithi125.stdout:2345: writing smithi12556470-1 from 3322484 to 3580056 tid 3 2021-03-24T22:03:12.413 INFO:tasks.rados.rados.0.smithi125.stdout:2346: write (excl) oid 4 current snap is 265 2021-03-24T22:03:12.413 INFO:tasks.rados.rados.0.smithi125.stdout:2341: finishing write tid 4 to smithi12556470-50 2021-03-24T22:03:12.414 INFO:tasks.rados.rados.0.smithi125.stdout:2346: seq_num 809 ranges {796863=409859,1807462=586285,3066260=530603} 2021-03-24T22:03:12.414 INFO:tasks.rados.rados.0.smithi125.stdout:2346: writing smithi12556470-4 from 796863 to 1206722 tid 1 2021-03-24T22:03:12.415 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:12.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:12.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:03:12.418 INFO:tasks.rados.rados.0.smithi125.stdout:2346: writing smithi12556470-4 from 1807462 to 2393747 tid 2 2021-03-24T22:03:12.424 INFO:tasks.rados.rados.0.smithi125.stdout:2346: writing smithi12556470-4 from 3066260 to 3596863 tid 3 2021-03-24T22:03:12.425 INFO:tasks.rados.rados.0.smithi125.stdout:2347: delete oid 21 current snap is 265 2021-03-24T22:03:12.425 INFO:tasks.rados.rados.0.smithi125.stdout:2341: finishing write tid 5 to smithi12556470-50 2021-03-24T22:03:12.425 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 50 v 785 (ObjNum 806 snap 265 seq_num 806) dirty exists 2021-03-24T22:03:12.425 INFO:tasks.rados.rados.0.smithi125.stdout:2341: left oid 50 (ObjNum 806 snap 265 seq_num 806) 2021-03-24T22:03:12.426 INFO:tasks.rados.rados.0.smithi125.stdout:2344: finishing rollback tid 0 to smithi12556470-47 2021-03-24T22:03:12.426 INFO:tasks.rados.rados.0.smithi125.stdout:2344: finishing rollback tid 1 to smithi12556470-47 2021-03-24T22:03:12.426 INFO:tasks.rados.rados.0.smithi125.stdout:2344: finishing rollback tid 2 to smithi12556470-47 2021-03-24T22:03:12.426 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 47 v 349 (ObjNum 466 snap 131 seq_num 466) dirty exists 2021-03-24T22:03:12.427 INFO:tasks.rados.rados.0.smithi125.stdout:2343: finishing copy_from to smithi12556470-43 2021-03-24T22:03:12.427 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 43 v 722 (ObjNum 757 snap 244 seq_num 757) dirty exists 2021-03-24T22:03:12.427 INFO:tasks.rados.rados.0.smithi125.stdout:2343: finishing copy_from racing read to smithi12556470-43 2021-03-24T22:03:12.427 INFO:tasks.rados.rados.0.smithi125.stdout:2345: finishing write tid 1 to smithi12556470-1 2021-03-24T22:03:12.428 INFO:tasks.rados.rados.0.smithi125.stdout:2345: finishing write tid 2 to smithi12556470-1 2021-03-24T22:03:12.428 INFO:tasks.rados.rados.0.smithi125.stdout:2345: finishing write tid 3 to smithi12556470-1 2021-03-24T22:03:12.428 INFO:tasks.rados.rados.0.smithi125.stdout:2345: finishing write tid 4 to smithi12556470-1 2021-03-24T22:03:12.428 INFO:tasks.rados.rados.0.smithi125.stdout:2345: finishing write tid 5 to smithi12556470-1 2021-03-24T22:03:12.429 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 1 v 827 (ObjNum 808 snap 265 seq_num 808) dirty exists 2021-03-24T22:03:12.429 INFO:tasks.rados.rados.0.smithi125.stdout:2345: left oid 1 (ObjNum 808 snap 265 seq_num 808) 2021-03-24T22:03:12.429 INFO:tasks.rados.rados.0.smithi125.stdout:2346: finishing write tid 1 to smithi12556470-4 2021-03-24T22:03:12.429 INFO:tasks.rados.rados.0.smithi125.stdout:2346: finishing write tid 2 to smithi12556470-4 2021-03-24T22:03:12.430 INFO:tasks.rados.rados.0.smithi125.stdout:2341: done (5 left) 2021-03-24T22:03:12.430 INFO:tasks.rados.rados.0.smithi125.stdout:2343: done (4 left) 2021-03-24T22:03:12.431 INFO:tasks.rados.rados.0.smithi125.stdout:2344: done (3 left) 2021-03-24T22:03:12.431 INFO:tasks.rados.rados.0.smithi125.stdout:2345: done (2 left) 2021-03-24T22:03:12.431 INFO:tasks.rados.rados.0.smithi125.stdout:2347: done (1 left) 2021-03-24T22:03:12.431 INFO:tasks.rados.rados.0.smithi125.stdout:2348: copy_from oid 28 from oid 31 current snap is 265 2021-03-24T22:03:12.432 INFO:tasks.rados.rados.0.smithi125.stdout:2349: snap_remove snap 248 2021-03-24T22:03:12.518 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:12.562 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:12.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:12.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:03:12.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:12 smithi142 conmon[20125]: cluster 2021-03-24T22:03:10.922501+0000 mgr.y (mgr.14144) 582 : cluster 2021-03-24T22:03:12.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:12 smithi142 conmon[20125]: [DBG] pgmap v1093: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 532 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.9 MiB/s rd, 1.9 MiB/s wr, 8 op/s; 40/1725 objects misplaced (2.319%); 312 B/s, 2 objects/s recovering 2021-03-24T22:03:12.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:12 smithi142 conmon[20125]: cluster 2021-03-24T22:03:11.343349 2021-03-24T22:03:12.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:12 smithi142 conmon[20125]: +0000 mon.a (mon.0) 1684 : cluster [DBG] osdmap e621: 8 total, 8 up, 8 in 2021-03-24T22:03:12.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:12 smithi142 conmon[20125]: 2021-03-24T22:03:12.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:12 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:03:12.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:12 smithi142 conmon[20125]: -03-24T22:03:11. 2021-03-24T22:03:12.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:12 smithi142 conmon[20125]: 533312+0000 osd.1 (osd.1) 2021-03-24T22:03:12.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:12 smithi142 conmon[20125]: 60 : cluster [DBG] 6.4 scrub ok 2021-03-24T22:03:12.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:12 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:03:12.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:12 smithi142 conmon[20125]: -03-24T22:03: 2021-03-24T22:03:12.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:12 smithi142 conmon[20125]: 11.749434+0000 osd.2 (osd 2021-03-24T22:03:12.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:12 smithi142 conmon[20125]: .2) 98 : cluster [DBG] 2021-03-24T22:03:12.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:12 smithi142 conmon[20125]: 7.20 scrub ok 2021-03-24T22:03:12.668 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:12.724 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:12.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:12.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:03:12.831 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:12.886 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:12.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:12.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:12.993 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:13.054 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:13.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:13.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:03:13.159 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:13.223 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:13.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:13.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:03:13.328 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:13.369 INFO:tasks.rados.rados.0.smithi125.stdout:2349: done (2 left) 2021-03-24T22:03:13.369 INFO:tasks.rados.rados.0.smithi125.stdout:2350: write oid 20 current snap is 265 2021-03-24T22:03:13.369 INFO:tasks.rados.rados.0.smithi125.stdout:2350: seq_num 811 ranges {430477=560357,1408073=473601,2488400=736073} 2021-03-24T22:03:13.376 INFO:tasks.rados.rados.0.smithi125.stdout:2350: writing smithi12556470-20 from 430477 to 990834 tid 1 2021-03-24T22:03:13.383 INFO:tasks.rados.rados.0.smithi125.stdout:2350: writing smithi12556470-20 from 1408073 to 1881674 tid 2 2021-03-24T22:03:13.392 INFO:tasks.rados.rados.0.smithi125.stdout:2350: writing smithi12556470-20 from 2488400 to 3224473 tid 3 2021-03-24T22:03:13.393 INFO:tasks.rados.rados.0.smithi125.stdout:2351: snap_remove snap 207 2021-03-24T22:03:13.405 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:13.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:13.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:03:13.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:13 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T22:03:13.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:13 smithi125 conmon[19936]: 22:03:12.348145+0000 mon.a (mon.0) 1685 : cluster [DBG] osdmap e622: 8 total, 8 up, 8 in 2021-03-24T22:03:13.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:13 smithi125 conmon[19936]: audit 2021-03 2021-03-24T22:03:13.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:13 smithi125 conmon[19936]: -24T22:03:12.371495+0000 mgr.y (mgr. 2021-03-24T22:03:13.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:13 smithi125 conmon[19936]: 14144) 583 : audit [DBG] 2021-03-24T22:03:13.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:13 smithi125 conmon[19936]: from='client.35558 v1:172.21.15.125:0/2121376998' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T22:03:13.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:13 smithi125 conmon[19936]: cluster 2021-03-24T22:03:13.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:13 smithi125 conmon[19936]: 2021-03-24T22:03:13.487 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:13 smithi125 conmon[19936]: 2021-03-24T22: 2021-03-24T22:03:13.487 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:13 smithi125 conmon[19936]: 03:12.502770+0000 osd.1 (osd.1) 61 : cluster [DBG] 4.4 scrub ok 2021-03-24T22:03:13.487 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:13 smithi125 conmon[19936]: cluster 2021-03-24T22:03:13.487 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:13 smithi125 conmon[19936]: 2021-03-24T22:03:12.788331+0000 osd.2 (osd 2021-03-24T22:03:13.488 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:13 smithi125 conmon[19936]: .2) 99 : cluster [DBG] 7.8 scrub ok 2021-03-24T22:03:13.488 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:13 smithi125 conmon[24927]: cluster 2021-03-24T22:03:13.488 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:13 smithi125 conmon[24927]: 2021-03-24T22:03:13.489 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:13 smithi125 conmon[24927]: 2021 2021-03-24T22:03:13.489 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:13 smithi125 conmon[24927]: - 2021-03-24T22:03:13.489 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:13 smithi125 conmon[24927]: 03 2021-03-24T22:03:13.489 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:13 smithi125 conmon[24927]: - 2021-03-24T22:03:13.489 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:13 smithi125 conmon[24927]: 24T22:03:12.348145+0000 2021-03-24T22:03:13.490 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:13 smithi125 conmon[24927]: mon.a (mon.0) 1685 : cluster [DBG] osdmap e622: 8 total, 8 up, 8 in 2021-03-24T22:03:13.490 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:13 smithi125 conmon[24927]: audit 2021-03 2021-03-24T22:03:13.490 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:13 smithi125 conmon[24927]: -24T22:03:12.371495+0000 mgr.y 2021-03-24T22:03:13.490 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:13 smithi125 conmon[24927]: (mgr.14144) 583 : audit [DBG] from='client.35558 v1:172.21.15.125:0/2121376998' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T22:03:13.491 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:13 smithi125 conmon[24927]: 2021-03-24T22:03:13.491 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:13 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:03:13.491 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:13 smithi125 conmon[24927]: 03:12.502770+0000 osd.1 (osd.1 2021-03-24T22:03:13.491 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:13 smithi125 conmon[24927]: ) 61 : cluster [DBG] 4.4 scrub ok 2021-03-24T22:03:13.491 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:13 smithi125 conmon[24927]: cluster 2021-03-24T22:03:13.492 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:13 smithi125 conmon[24927]: 2021-03-24T22:03:12. 2021-03-24T22:03:13.492 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:13 smithi125 conmon[24927]: 788331+0000 osd.2 (osd.2) 99 : 2021-03-24T22:03:13.492 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:13 smithi125 conmon[24927]: cluster [DBG] 7.8 scrub ok 2021-03-24T22:03:13.511 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:13.568 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:13.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:13.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:03:13.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:13 smithi142 conmon[20125]: cluster 2021-03-24T22:03:12.348145+0000 mon.a (mon.0) 1685 : cluster [DBG] 2021-03-24T22:03:13.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:13 smithi142 conmon[20125]: osdmap e622: 8 total, 8 up, 8 in 2021-03-24T22:03:13.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:13 smithi142 conmon[20125]: audit 2021-03-24T22:03:12.371495+0000 mgr.y (mgr.14144) 583 : audit [DBG] from='client.35558 v1:172.21.15.125:0/2121376998' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T22:03:13.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:13 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:03:13.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:13 smithi142 conmon[20125]: -24T22:03:12.502770+0000 osd.1 (osd.1) 61 : cluster [DBG] 4.4 scrub ok 2021-03-24T22:03:13.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:13 smithi142 conmon[20125]: cluster 2021-03-24T22 2021-03-24T22:03:13.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:13 smithi142 conmon[20125]: :03:12.788331+0000 osd.2 (osd.2) 99 : cluster 2021-03-24T22:03:13.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:13 smithi142 conmon[20125]: [DBG] 7.8 scrub ok 2021-03-24T22:03:13.676 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:13.698 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-24T22:03:13.732 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:13.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:13.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:03:13.838 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:13.895 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:13.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:13.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:03:14.000 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:14.057 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:14.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:14.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:03:14.164 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:14.219 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:14.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:14.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:14.325 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:14.373 INFO:tasks.rados.rados.0.smithi125.stdout:2351: done (3 left) 2021-03-24T22:03:14.373 INFO:tasks.rados.rados.0.smithi125.stdout:2352: delete oid 39 current snap is 265 2021-03-24T22:03:14.373 INFO:tasks.rados.rados.0.smithi125.stdout:2346: finishing write tid 3 to smithi12556470-4 2021-03-24T22:03:14.373 INFO:tasks.rados.rados.0.smithi125.stdout:2346: finishing write tid 4 to smithi12556470-4 2021-03-24T22:03:14.374 INFO:tasks.rados.rados.0.smithi125.stdout:2346: finishing write tid 5 to smithi12556470-4 2021-03-24T22:03:14.374 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 4 v 549 (ObjNum 809 snap 265 seq_num 809) dirty exists 2021-03-24T22:03:14.374 INFO:tasks.rados.rados.0.smithi125.stdout:2346: left oid 4 (ObjNum 809 snap 265 seq_num 809) 2021-03-24T22:03:14.374 INFO:tasks.rados.rados.0.smithi125.stdout:2348: finishing copy_from to smithi12556470-28 2021-03-24T22:03:14.375 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 28 v 1185 (ObjNum 718 snap 229 seq_num 718) dirty exists 2021-03-24T22:03:14.375 INFO:tasks.rados.rados.0.smithi125.stdout:2348: finishing copy_from racing read to smithi12556470-28 2021-03-24T22:03:14.375 INFO:tasks.rados.rados.0.smithi125.stdout:2350: finishing write tid 1 to smithi12556470-20 2021-03-24T22:03:14.375 INFO:tasks.rados.rados.0.smithi125.stdout:2350: finishing write tid 2 to smithi12556470-20 2021-03-24T22:03:14.375 INFO:tasks.rados.rados.0.smithi125.stdout:2350: finishing write tid 3 to smithi12556470-20 2021-03-24T22:03:14.376 INFO:tasks.rados.rados.0.smithi125.stdout:2350: finishing write tid 4 to smithi12556470-20 2021-03-24T22:03:14.376 INFO:tasks.rados.rados.0.smithi125.stdout:2350: finishing write tid 5 to smithi12556470-20 2021-03-24T22:03:14.376 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 20 v 789 (ObjNum 811 snap 265 seq_num 811) dirty exists 2021-03-24T22:03:14.376 INFO:tasks.rados.rados.0.smithi125.stdout:2350: left oid 20 (ObjNum 811 snap 265 seq_num 811) 2021-03-24T22:03:14.393 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:14.395 INFO:tasks.rados.rados.0.smithi125.stdout:2346: done (3 left) 2021-03-24T22:03:14.395 INFO:tasks.rados.rados.0.smithi125.stdout:2348: done (2 left) 2021-03-24T22:03:14.395 INFO:tasks.rados.rados.0.smithi125.stdout:2350: done (1 left) 2021-03-24T22:03:14.395 INFO:tasks.rados.rados.0.smithi125.stdout:2352: done (0 left) 2021-03-24T22:03:14.396 INFO:tasks.rados.rados.0.smithi125.stdout:2353: read oid 17 snap -1 2021-03-24T22:03:14.396 INFO:tasks.rados.rados.0.smithi125.stdout:2353: expect (ObjNum 744 snap 240 seq_num 744) 2021-03-24T22:03:14.396 INFO:tasks.rados.rados.0.smithi125.stdout:2354: snap_remove snap 213 2021-03-24T22:03:14.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:14.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:03:14.500 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:14.513 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:14 smithi125 conmon[19936]: cluster 2021-03-24T22:03:12.756595 2021-03-24T22:03:14.514 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:14 smithi125 conmon[19936]: +0000 osd.6 (osd.6) 47 : cluster [DBG] 7.d scrub ok 2021-03-24T22:03:14.514 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:14 smithi125 conmon[19936]: cluster 2021-03-24T22:03:12.923216+0000 mgr.y (mgr.14144) 584 : cluster [DBG] 2021-03-24T22:03:14.514 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:14 smithi125 conmon[19936]: pgmap v1096: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 514 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 7.0 MiB/s rd, 3.2 MiB/s wr, 14 op/s; 40/1707 objects misplaced (2.343%); 312 B/s, 2 objects/s recovering 2021-03-24T22:03:14.515 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:14 smithi125 conmon[19936]: cluster 2021-03-24T22 2021-03-24T22:03:14.515 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:14 smithi125 conmon[19936]: :03:13.361680+0000 mon.a (mon.0) 1686 : cluster [DBG] osdmap e623: 8 total, 8 up, 8 in 2021-03-24T22:03:14.515 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:14 smithi125 conmon[24927]: cluster 2021-03-24T22:03:12.756595 2021-03-24T22:03:14.516 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:14 smithi125 conmon[24927]: +0000 osd.6 (osd.6) 47 : cluster [DBG] 7.d scrub ok 2021-03-24T22:03:14.516 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:14 smithi125 conmon[24927]: cluster 2021-03-24T22:03:12 2021-03-24T22:03:14.516 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:14 smithi125 conmon[24927]: .923216+0000 mgr.y (mgr. 2021-03-24T22:03:14.516 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:14 smithi125 conmon[24927]: 14144) 584 : cluster [DBG] pgmap v1096: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 514 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 7.0 MiB/s rd, 3.2 MiB/s wr, 14 op/s; 40/1707 objects misplaced (2.343%); 312 B/s, 2 objects/s recovering 2021-03-24T22:03:14.516 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:14 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:03:14.517 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:14 smithi125 conmon[24927]: -03-24T22:03:13.361680 2021-03-24T22:03:14.517 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:14 smithi125 conmon[24927]: +0000 mon.a (mon.0) 1686 2021-03-24T22:03:14.517 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:14 smithi125 conmon[24927]: : cluster [DBG] osdmap e623: 8 total, 8 up, 8 in 2021-03-24T22:03:14.559 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:14.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:14.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:03:14.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:14 smithi142 conmon[20125]: cluster 2021-03-24T22:03:12.756595+0000 osd.6 (osd.6) 47 : 2021-03-24T22:03:14.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:14 smithi142 conmon[20125]: cluster [DBG] 7.d scrub ok 2021-03-24T22:03:14.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:14 smithi142 conmon[20125]: cluster 2021-03-24T22:03:12.923216+0000 2021-03-24T22:03:14.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:14 smithi142 conmon[20125]: mgr.y (mgr.14144) 584 : cluster [DBG] pgmap v1096: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 514 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 7.0 MiB/s rd, 3.2 MiB/s wr, 14 op/s; 40/1707 objects misplaced (2.343%); 312 B/s, 2 objects/s recovering 2021-03-24T22:03:14.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:14 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:03:14.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:14 smithi142 conmon[20125]: -03-24T22:03:13.361680+0000 2021-03-24T22:03:14.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:14 smithi142 conmon[20125]: mon.a (mon.0) 1686 : cluster [DBG] osdmap e623: 8 total, 8 up, 8 in 2021-03-24T22:03:14.668 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:14.727 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:14.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:14.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:03:14.833 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:14.888 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:14.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:14.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:03:14.994 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:15.054 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:15.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:15.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:03:15.148 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":1095,"stamp":"2021-03-24T22:03:12.345403+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":538829577,"num_objects":569,"num_object_clones":197,"num_object_copies":1707,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":40,"num_objects_unfound":0,"num_objects_dirty":569,"num_whiteouts":9,"num_read":4749,"num_read_kb":3288696,"num_write":3077,"num_write_kb":854868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3484,"num_bytes_recovered":1437165687,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14335,"ondisk_log_size":14335,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":507,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1438800,"kb_used_data":1151504,"kb_used_omap":33,"kb_used_meta":287198,"kb_avail":748522416,"statfs":{"total":767960285184,"available":766486953984,"internally_reserved":0,"allocated":1179140096,"data_stored":1593094005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33936,"internal_metadata":294091632},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-32042356,"num_objects":-11,"num_object_clones":-11,"num_object_copies":-33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-11,"num_whiteouts":1,"num_read":39,"num_read_kb":28562,"num_write":21,"num_write_kb":13164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.019957"},"pg_stats":[{"pgid":"3.29","version":"0'0","reported_seq":"253","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.332421+0000","last_change":"2021-03-24T22:01:16.299847+0000","last_active":"2021-03-24T22:03:07.332421+0000","last_peered":"2021-03-24T22:03:07.332421+0000","last_clean":"2021-03-24T22:03:07.332421+0000","last_became_active":"2021-03-24T22:01:16.299327+0000","last_became_peered":"2021-03-24T22:01:16.299327+0000","last_unstale":"2021-03-24T22:03:07.332421+0000","last_undegraded":"2021-03-24T22:03:07.332421+0000","last_fullsized":"2021-03-24T22:03:07.332421+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:55.592795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:55.592795+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":"201","reported_epoch":"619","state":"active+clean","last_fresh":"2021-03-24T22:03:09.343030+0000","last_change":"2021-03-24T22:03:07.837859+0000","last_active":"2021-03-24T22:03:09.343030+0000","last_peered":"2021-03-24T22:03:09.343030+0000","last_clean":"2021-03-24T22:03:09.343030+0000","last_became_active":"2021-03-24T22:00:04.286965+0000","last_became_peered":"2021-03-24T22:00:04.286965+0000","last_unstale":"2021-03-24T22:03:09.343030+0000","last_undegraded":"2021-03-24T22:03:09.343030+0000","last_fullsized":"2021-03-24T22:03:09.343030+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:07.837673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:03:07.837673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"89'2","reported_seq":"285","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.476734+0000","last_change":"2021-03-24T21:59:14.374798+0000","last_active":"2021-03-24T22:03:07.476734+0000","last_peered":"2021-03-24T22:03:07.476734+0000","last_clean":"2021-03-24T22:03:07.476734+0000","last_became_active":"2021-03-24T21:59:14.374418+0000","last_became_peered":"2021-03-24T21:59:14.374418+0000","last_unstale":"2021-03-24T22:03:07.476734+0000","last_undegraded":"2021-03-24T22:03:07.476734+0000","last_fullsized":"2021-03-24T22:03:07.476734+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":6,"last_scrub":"89'2","last_scrub_stamp":"2021-03-24T21:57:58.491830+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2021-03-24T21:57:58.491830+0000","last_clean_scrub_stamp":"2021-03-24T21:57:58.491830+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":179,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":"613","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.331654+0000","last_change":"2021-03-24T22:00:04.297849+0000","last_active":"2021-03-24T22:03:07.331654+0000","last_peered":"2021-03-24T22:03:07.331654+0000","last_clean":"2021-03-24T22:03:07.331654+0000","last_became_active":"2021-03-24T22:00:04.297076+0000","last_became_peered":"2021-03-24T22:00:04.297076+0000","last_unstale":"2021-03-24T22:03:07.331654+0000","last_undegraded":"2021-03-24T22:03:07.331654+0000","last_fullsized":"2021-03-24T22:03:07.331654+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:33.533840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:33.533840+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":"59","reported_epoch":"620","state":"active+clean","last_fresh":"2021-03-24T22:03:10.323904+0000","last_change":"2021-03-24T22:02:22.324306+0000","last_active":"2021-03-24T22:03:10.323904+0000","last_peered":"2021-03-24T22:03:10.323904+0000","last_clean":"2021-03-24T22:03:10.323904+0000","last_became_active":"2021-03-24T22:02:22.323235+0000","last_became_peered":"2021-03-24T22:02:22.323235+0000","last_unstale":"2021-03-24T22:03:10.323904+0000","last_undegraded":"2021-03-24T22:03:10.323904+0000","last_fullsized":"2021-03-24T22:03:10.323904+0000","mapping_epoch":571,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:38.304792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:38.304792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"61'1","reported_seq":"101","reported_epoch":"619","state":"active+clean","last_fresh":"2021-03-24T22:03:09.338744+0000","last_change":"2021-03-24T22:01:43.577425+0000","last_active":"2021-03-24T22:03:09.338744+0000","last_peered":"2021-03-24T22:03:09.338744+0000","last_clean":"2021-03-24T22:03:09.338744+0000","last_became_active":"2021-03-24T22:01:43.390951+0000","last_became_peered":"2021-03-24T22:01:43.390951+0000","last_unstale":"2021-03-24T22:03:09.338744+0000","last_undegraded":"2021-03-24T22:03:09.338744+0000","last_fullsized":"2021-03-24T22:03:09.338744+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":6,"last_scrub":"61'1","last_scrub_stamp":"2021-03-24T21:57:12.046206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:12.046206+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":437,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3492,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":"600","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.329551+0000","last_change":"2021-03-24T22:02:57.502941+0000","last_active":"2021-03-24T22:03:07.329551+0000","last_peered":"2021-03-24T22:03:07.329551+0000","last_clean":"2021-03-24T22:03:07.329551+0000","last_became_active":"2021-03-24T21:59:14.370506+0000","last_became_peered":"2021-03-24T21:59:14.370506+0000","last_unstale":"2021-03-24T22:03:07.329551+0000","last_undegraded":"2021-03-24T22:03:07.329551+0000","last_fullsized":"2021-03-24T22:03:07.329551+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:57.502803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:02:57.502803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.23","version":"620'360","reported_seq":"2072","reported_epoch":"620","state":"active+clean","last_fresh":"2021-03-24T22:03:10.341224+0000","last_change":"2021-03-24T22:03:10.341618+0000","last_active":"2021-03-24T22:03:10.341224+0000","last_peered":"2021-03-24T22:03:10.341224+0000","last_clean":"2021-03-24T22:03:10.341224+0000","last_became_active":"2021-03-24T22:01:46.311925+0000","last_became_peered":"2021-03-24T22:01:46.311925+0000","last_unstale":"2021-03-24T22:03:10.341224+0000","last_undegraded":"2021-03-24T22:03:10.341224+0000","last_fullsized":"2021-03-24T22:03:10.341224+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":360,"ondisk_log_size":360,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8766165,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":125,"num_read_kb":141370,"num_write":87,"num_write_kb":33080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":12165888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"139","length":"1"},{"start":"178","length":"1"},{"start":"1ae","length":"1"},{"start":"1eb","length":"1"}]},{"pgid":"3.26","version":"0'0","reported_seq":"106","reported_epoch":"619","state":"active+clean","last_fresh":"2021-03-24T22:03:09.339199+0000","last_change":"2021-03-24T22:01:57.311479+0000","last_active":"2021-03-24T22:03:09.339199+0000","last_peered":"2021-03-24T22:03:09.339199+0000","last_clean":"2021-03-24T22:03:09.339199+0000","last_became_active":"2021-03-24T22:01:57.311128+0000","last_became_peered":"2021-03-24T22:01:57.311128+0000","last_unstale":"2021-03-24T22:03:09.339199+0000","last_undegraded":"2021-03-24T22:03:09.339199+0000","last_fullsized":"2021-03-24T22:03:09.339199+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:42.879245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:59:42.879245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"616'224","reported_seq":"1154","reported_epoch":"619","state":"active+clean","last_fresh":"2021-03-24T22:03:09.345824+0000","last_change":"2021-03-24T22:03:09.346316+0000","last_active":"2021-03-24T22:03:09.345824+0000","last_peered":"2021-03-24T22:03:09.345824+0000","last_clean":"2021-03-24T22:03:09.345824+0000","last_became_active":"2021-03-24T22:03:06.343802+0000","last_became_peered":"2021-03-24T22:03:06.343802+0000","last_unstale":"2021-03-24T22:03:09.345824+0000","last_undegraded":"2021-03-24T22:03:09.345824+0000","last_fullsized":"2021-03-24T22:03:09.345824+0000","mapping_epoch":615,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":616,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":224,"ondisk_log_size":224,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17705621,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":14791,"num_write":56,"num_write_kb":14060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":21273869,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"139","length":"1"},{"start":"178","length":"1"},{"start":"1ae","length":"1"},{"start":"1c4","length":"1"}]},{"pgid":"4.c","version":"544'37","reported_seq":"349","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.331176+0000","last_change":"2021-03-24T22:03:03.453859+0000","last_active":"2021-03-24T22:03:07.331176+0000","last_peered":"2021-03-24T22:03:07.331176+0000","last_clean":"2021-03-24T22:03:07.331176+0000","last_became_active":"2021-03-24T22:01:50.301325+0000","last_became_peered":"2021-03-24T22:01:50.301325+0000","last_unstale":"2021-03-24T22:03:07.331176+0000","last_undegraded":"2021-03-24T22:03:07.331176+0000","last_fullsized":"2021-03-24T22:03:07.331176+0000","mapping_epoch":539,"log_start":"69'15","ondisk_log_start":"69'15","created":62,"last_epoch_clean":540,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'37","last_scrub_stamp":"2021-03-24T22:03:03.453713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:03:03.453713+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":15,"num_object_clones":0,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":81,"num_read_kb":61,"num_write":51,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":1266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"583","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.326732+0000","last_change":"2021-03-24T21:59:17.264251+0000","last_active":"2021-03-24T22:03:07.326732+0000","last_peered":"2021-03-24T22:03:07.326732+0000","last_clean":"2021-03-24T22:03:07.326732+0000","last_became_active":"2021-03-24T21:51:45.375830+0000","last_became_peered":"2021-03-24T21:51:45.375830+0000","last_unstale":"2021-03-24T22:03:07.326732+0000","last_undegraded":"2021-03-24T22:03:07.326732+0000","last_fullsized":"2021-03-24T22:03:07.326732+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:17.264089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:59:17.264089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"548'12","reported_seq":"802","reported_epoch":"620","state":"active+clean","last_fresh":"2021-03-24T22:03:10.326923+0000","last_change":"2021-03-24T22:02:18.728472+0000","last_active":"2021-03-24T22:03:10.326923+0000","last_peered":"2021-03-24T22:03:10.326923+0000","last_clean":"2021-03-24T22:03:10.326923+0000","last_became_active":"2021-03-24T22:01:57.320271+0000","last_became_peered":"2021-03-24T22:01:57.320271+0000","last_unstale":"2021-03-24T22:03:10.326923+0000","last_undegraded":"2021-03-24T22:03:10.326923+0000","last_fullsized":"2021-03-24T22:03:10.326923+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"548'12","last_scrub_stamp":"2021-03-24T22:02:18.728277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:02:18.728277+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"132","reported_epoch":"618","state":"active+clean","last_fresh":"2021-03-24T22:03:08.333111+0000","last_change":"2021-03-24T22:02:22.483455+0000","last_active":"2021-03-24T22:03:08.333111+0000","last_peered":"2021-03-24T22:03:08.333111+0000","last_clean":"2021-03-24T22:03:08.333111+0000","last_became_active":"2021-03-24T22:02:22.482833+0000","last_became_peered":"2021-03-24T22:02:22.482833+0000","last_unstale":"2021-03-24T22:03:08.333111+0000","last_undegraded":"2021-03-24T22:03:08.333111+0000","last_fullsized":"2021-03-24T22:03:08.333111+0000","mapping_epoch":571,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:13.834281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:00:13.834281+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.f","version":"616'464","reported_seq":"2299","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.476469+0000","last_change":"2021-03-24T22:03:06.370990+0000","last_active":"2021-03-24T22:03:07.476469+0000","last_peered":"2021-03-24T22:03:07.476469+0000","last_clean":"2021-03-24T22:03:07.476469+0000","last_became_active":"2021-03-24T22:02:10.313297+0000","last_became_peered":"2021-03-24T22:02:10.313297+0000","last_unstale":"2021-03-24T22:03:07.476469+0000","last_undegraded":"2021-03-24T22:03:07.476469+0000","last_fullsized":"2021-03-24T22:03:07.476469+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'171","last_scrub_stamp":"2021-03-24T21:59:08.017874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:59:08.017874+0000","log_size":464,"ondisk_log_size":464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25046623,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":100,"num_read_kb":118330,"num_write":44,"num_write_kb":13517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":46545574,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"178","length":"1"},{"start":"1ae","length":"1"},{"start":"1c4","length":"1"}]},{"pgid":"4.d","version":"544'41","reported_seq":"306","reported_epoch":"620","state":"active+clean","last_fresh":"2021-03-24T22:03:10.326460+0000","last_change":"2021-03-24T22:03:03.723634+0000","last_active":"2021-03-24T22:03:10.326460+0000","last_peered":"2021-03-24T22:03:10.326460+0000","last_clean":"2021-03-24T22:03:10.326460+0000","last_became_active":"2021-03-24T22:01:25.276760+0000","last_became_peered":"2021-03-24T22:01:25.276760+0000","last_unstale":"2021-03-24T22:03:10.326460+0000","last_undegraded":"2021-03-24T22:03:10.326460+0000","last_fullsized":"2021-03-24T22:03:10.326460+0000","mapping_epoch":514,"log_start":"69'21","ondisk_log_start":"69'21","created":62,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'41","last_scrub_stamp":"2021-03-24T22:03:03.723420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:03:03.723420+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1578,"num_objects":21,"num_object_clones":0,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":106,"num_read_kb":71,"num_write":60,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"102","reported_epoch":"619","state":"active+clean","last_fresh":"2021-03-24T22:03:09.339376+0000","last_change":"2021-03-24T22:02:30.890732+0000","last_active":"2021-03-24T22:03:09.339376+0000","last_peered":"2021-03-24T22:03:09.339376+0000","last_clean":"2021-03-24T22:03:09.339376+0000","last_became_active":"2021-03-24T22:01:43.396233+0000","last_became_peered":"2021-03-24T22:01:43.396233+0000","last_unstale":"2021-03-24T22:03:09.339376+0000","last_undegraded":"2021-03-24T22:03:09.339376+0000","last_fullsized":"2021-03-24T22:03:09.339376+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:30.890594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:55:01.169461+0000","last_clean_scrub_stamp":"2021-03-24T22:02:30.890594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"123","reported_epoch":"618","state":"active+clean","last_fresh":"2021-03-24T22:03:08.332080+0000","last_change":"2021-03-24T22:01:16.334950+0000","last_active":"2021-03-24T22:03:08.332080+0000","last_peered":"2021-03-24T22:03:08.332080+0000","last_clean":"2021-03-24T22:03:08.332080+0000","last_became_active":"2021-03-24T22:01:16.334606+0000","last_became_peered":"2021-03-24T22:01:16.334606+0000","last_unstale":"2021-03-24T22:03:08.332080+0000","last_undegraded":"2021-03-24T22:03:08.332080+0000","last_fullsized":"2021-03-24T22:03:08.332080+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:19.934770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:00:19.934770+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"59","reported_epoch":"620","state":"active+clean","last_fresh":"2021-03-24T22:03:10.326103+0000","last_change":"2021-03-24T22:02:22.483059+0000","last_active":"2021-03-24T22:03:10.326103+0000","last_peered":"2021-03-24T22:03:10.326103+0000","last_clean":"2021-03-24T22:03:10.326103+0000","last_became_active":"2021-03-24T22:02:22.482380+0000","last_became_peered":"2021-03-24T22:02:22.482380+0000","last_unstale":"2021-03-24T22:03:10.326103+0000","last_undegraded":"2021-03-24T22:03:10.326103+0000","last_fullsized":"2021-03-24T22:03:10.326103+0000","mapping_epoch":571,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:57.316683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:00:57.316683+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"621'201","reported_seq":"1820","reported_epoch":"621","state":"active+clean","last_fresh":"2021-03-24T22:03:11.350295+0000","last_change":"2021-03-24T22:03:11.350561+0000","last_active":"2021-03-24T22:03:11.350295+0000","last_peered":"2021-03-24T22:03:11.350295+0000","last_clean":"2021-03-24T22:03:11.350295+0000","last_became_active":"2021-03-24T22:00:03.281149+0000","last_became_peered":"2021-03-24T22:00:03.281149+0000","last_unstale":"2021-03-24T22:03:11.350295+0000","last_undegraded":"2021-03-24T22:03:11.350295+0000","last_fullsized":"2021-03-24T22:03:11.350295+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"546'150","last_scrub_stamp":"2021-03-24T22:01:56.662245+0000","last_deep_scrub":"546'150","last_deep_scrub_stamp":"2021-03-24T22:01:56.662245+0000","last_clean_scrub_stamp":"2021-03-24T22:01:56.662245+0000","log_size":201,"ondisk_log_size":201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10084761,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":52,"num_read_kb":39509,"num_write":33,"num_write_kb":9037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2421497,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"139","length":"1"},{"start":"178","length":"1"},{"start":"1ae","length":"1"},{"start":"1eb","length":"1"},{"start":"1f4","length":"1"}]},{"pgid":"4.e","version":"544'45","reported_seq":"358","reported_epoch":"619","state":"active+clean","last_fresh":"2021-03-24T22:03:09.338940+0000","last_change":"2021-03-24T22:02:15.401403+0000","last_active":"2021-03-24T22:03:09.338940+0000","last_peered":"2021-03-24T22:03:09.338940+0000","last_clean":"2021-03-24T22:03:09.338940+0000","last_became_active":"2021-03-24T22:02:15.400902+0000","last_became_peered":"2021-03-24T22:02:15.400902+0000","last_unstale":"2021-03-24T22:03:09.338940+0000","last_undegraded":"2021-03-24T22:03:09.338940+0000","last_fullsized":"2021-03-24T22:03:09.338940+0000","mapping_epoch":564,"log_start":"89'27","ondisk_log_start":"89'27","created":62,"last_epoch_clean":565,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'27","last_scrub_stamp":"2021-03-24T22:00:20.952131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:00:20.952131+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1672,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":92,"num_read_kb":60,"num_write":50,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"596","reported_epoch":"620","state":"active+clean","last_fresh":"2021-03-24T22:03:10.324392+0000","last_change":"2021-03-24T22:01:57.302726+0000","last_active":"2021-03-24T22:03:10.324392+0000","last_peered":"2021-03-24T22:03:10.324392+0000","last_clean":"2021-03-24T22:03:10.324392+0000","last_became_active":"2021-03-24T22:01:57.302297+0000","last_became_peered":"2021-03-24T22:01:57.302297+0000","last_unstale":"2021-03-24T22:03:10.324392+0000","last_undegraded":"2021-03-24T22:03:10.324392+0000","last_fullsized":"2021-03-24T22:03:10.324392+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:55.592795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:55.592795+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"97","reported_epoch":"619","state":"active+clean","last_fresh":"2021-03-24T22:03:09.339341+0000","last_change":"2021-03-24T22:01:43.384687+0000","last_active":"2021-03-24T22:03:09.339341+0000","last_peered":"2021-03-24T22:03:09.339341+0000","last_clean":"2021-03-24T22:03:09.339341+0000","last_became_active":"2021-03-24T22:01:43.384327+0000","last_became_peered":"2021-03-24T22:01:43.384327+0000","last_unstale":"2021-03-24T22:03:09.339341+0000","last_undegraded":"2021-03-24T22:03:09.339341+0000","last_fullsized":"2021-03-24T22:03:09.339341+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:56.217032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:00:56.217032+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"634","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.327932+0000","last_change":"2021-03-24T22:01:43.384970+0000","last_active":"2021-03-24T22:03:07.327932+0000","last_peered":"2021-03-24T22:03:07.327932+0000","last_clean":"2021-03-24T22:03:07.327932+0000","last_became_active":"2021-03-24T22:01:43.384573+0000","last_became_peered":"2021-03-24T22:01:43.384573+0000","last_unstale":"2021-03-24T22:03:07.327932+0000","last_undegraded":"2021-03-24T22:03:07.327932+0000","last_fullsized":"2021-03-24T22:03:07.327932+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:25.263840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:00:25.263840+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"620'538","reported_seq":"2281","reported_epoch":"620","state":"active+clean","last_fresh":"2021-03-24T22:03:10.348282+0000","last_change":"2021-03-24T22:03:10.348523+0000","last_active":"2021-03-24T22:03:10.348282+0000","last_peered":"2021-03-24T22:03:10.348282+0000","last_clean":"2021-03-24T22:03:10.348282+0000","last_became_active":"2021-03-24T22:01:48.324477+0000","last_became_peered":"2021-03-24T22:01:48.324477+0000","last_unstale":"2021-03-24T22:03:10.348282+0000","last_undegraded":"2021-03-24T22:03:10.348282+0000","last_fullsized":"2021-03-24T22:03:10.348282+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":0,"last_scrub":"384'362","last_scrub_stamp":"2021-03-24T21:59:15.076161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:59:15.076161+0000","log_size":538,"ondisk_log_size":538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13116974,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":61,"num_read_kb":86468,"num_write":29,"num_write_kb":13112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":45770019,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"139","length":"1"},{"start":"178","length":"1"},{"start":"1ae","length":"1"},{"start":"1eb","length":"1"}]},{"pgid":"4.f","version":"544'46","reported_seq":"772","reported_epoch":"618","state":"active+clean","last_fresh":"2021-03-24T22:03:08.331647+0000","last_change":"2021-03-24T22:01:53.692379+0000","last_active":"2021-03-24T22:03:08.331647+0000","last_peered":"2021-03-24T22:03:08.331647+0000","last_clean":"2021-03-24T22:03:08.331647+0000","last_became_active":"2021-03-24T22:01:48.328342+0000","last_became_peered":"2021-03-24T22:01:48.328342+0000","last_unstale":"2021-03-24T22:03:08.331647+0000","last_undegraded":"2021-03-24T22:03:08.331647+0000","last_fullsized":"2021-03-24T22:03:08.331647+0000","mapping_epoch":537,"log_start":"274'26","ondisk_log_start":"274'26","created":62,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'26","last_scrub_stamp":"2021-03-24T22:01:53.692294+0000","last_deep_scrub":"274'26","last_deep_scrub_stamp":"2021-03-24T21:57:24.499090+0000","last_clean_scrub_stamp":"2021-03-24T22:01:53.692294+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1472,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":98,"num_read_kb":66,"num_write":56,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":5466,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":"647","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.325820+0000","last_change":"2021-03-24T22:01:57.316662+0000","last_active":"2021-03-24T22:03:07.325820+0000","last_peered":"2021-03-24T22:03:07.325820+0000","last_clean":"2021-03-24T22:03:07.325820+0000","last_became_active":"2021-03-24T22:01:57.316262+0000","last_became_peered":"2021-03-24T22:01:57.316262+0000","last_unstale":"2021-03-24T22:03:07.325820+0000","last_undegraded":"2021-03-24T22:03:07.325820+0000","last_fullsized":"2021-03-24T22:03:07.325820+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:39.439170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:39.439170+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"479'5","reported_seq":"188","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.477268+0000","last_change":"2021-03-24T22:00:49.311757+0000","last_active":"2021-03-24T22:03:07.477268+0000","last_peered":"2021-03-24T22:03:07.477268+0000","last_clean":"2021-03-24T22:03:07.477268+0000","last_became_active":"2021-03-24T22:00:49.310037+0000","last_became_peered":"2021-03-24T22:00:49.310037+0000","last_unstale":"2021-03-24T22:03:07.477268+0000","last_undegraded":"2021-03-24T22:03:07.477268+0000","last_fullsized":"2021-03-24T22:03:07.477268+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'4","last_scrub_stamp":"2021-03-24T22:00:14.879311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:00:14.879311+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"496","reported_epoch":"619","state":"active+clean","last_fresh":"2021-03-24T22:03:09.343733+0000","last_change":"2021-03-24T22:02:55.950583+0000","last_active":"2021-03-24T22:03:09.343733+0000","last_peered":"2021-03-24T22:03:09.343733+0000","last_clean":"2021-03-24T22:03:09.343733+0000","last_became_active":"2021-03-24T22:01:16.314269+0000","last_became_peered":"2021-03-24T22:01:16.314269+0000","last_unstale":"2021-03-24T22:03:09.343733+0000","last_undegraded":"2021-03-24T22:03:09.343733+0000","last_fullsized":"2021-03-24T22:03:09.343733+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:55.950428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:02:55.950428+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"620'831","reported_seq":"2947","reported_epoch":"620","state":"active+clean","last_fresh":"2021-03-24T22:03:10.342369+0000","last_change":"2021-03-24T22:03:10.342664+0000","last_active":"2021-03-24T22:03:10.342369+0000","last_peered":"2021-03-24T22:03:10.342369+0000","last_clean":"2021-03-24T22:03:10.342369+0000","last_became_active":"2021-03-24T22:02:06.309737+0000","last_became_peered":"2021-03-24T22:02:06.309737+0000","last_unstale":"2021-03-24T22:03:10.342369+0000","last_undegraded":"2021-03-24T22:03:10.342369+0000","last_fullsized":"2021-03-24T22:03:10.342369+0000","mapping_epoch":555,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":556,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'393","last_scrub_stamp":"2021-03-24T21:59:12.370455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:59:12.370455+0000","log_size":831,"ondisk_log_size":831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28975665,"num_objects":12,"num_object_clones":8,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":127,"num_read_kb":138355,"num_write":88,"num_write_kb":33163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":296,"num_bytes_recovered":189413266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"139","length":"1"},{"start":"178","length":"1"},{"start":"1ae","length":"1"},{"start":"1eb","length":"1"}]},{"pgid":"5.1","version":"0'0","reported_seq":"557","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.330671+0000","last_change":"2021-03-24T22:03:00.472146+0000","last_active":"2021-03-24T22:03:07.330671+0000","last_peered":"2021-03-24T22:03:07.330671+0000","last_clean":"2021-03-24T22:03:07.330671+0000","last_became_active":"2021-03-24T22:01:57.305430+0000","last_became_peered":"2021-03-24T22:01:57.305430+0000","last_unstale":"2021-03-24T22:03:07.330671+0000","last_undegraded":"2021-03-24T22:03:07.330671+0000","last_fullsized":"2021-03-24T22:03:07.330671+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:00.471949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:25.220436+0000","last_clean_scrub_stamp":"2021-03-24T22:03:00.471949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"544'34","reported_seq":"785","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.325915+0000","last_change":"2021-03-24T22:01:58.356469+0000","last_active":"2021-03-24T22:03:07.325915+0000","last_peered":"2021-03-24T22:03:07.325915+0000","last_clean":"2021-03-24T22:03:07.325915+0000","last_became_active":"2021-03-24T22:01:57.309916+0000","last_became_peered":"2021-03-24T22:01:57.309916+0000","last_unstale":"2021-03-24T22:03:07.325915+0000","last_undegraded":"2021-03-24T22:03:07.325915+0000","last_fullsized":"2021-03-24T22:03:07.325915+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'22","last_scrub_stamp":"2021-03-24T21:58:56.473510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:58:56.473510+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":64,"num_read_kb":44,"num_write":36,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":3384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"634","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.329429+0000","last_change":"2021-03-24T22:01:57.314681+0000","last_active":"2021-03-24T22:03:07.329429+0000","last_peered":"2021-03-24T22:03:07.329429+0000","last_clean":"2021-03-24T22:03:07.329429+0000","last_became_active":"2021-03-24T22:01:57.314345+0000","last_became_peered":"2021-03-24T22:01:57.314345+0000","last_unstale":"2021-03-24T22:03:07.329429+0000","last_undegraded":"2021-03-24T22:03:07.329429+0000","last_fullsized":"2021-03-24T22:03:07.329429+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:54.381668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:54.381668+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"151","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.332149+0000","last_change":"2021-03-24T22:00:49.305933+0000","last_active":"2021-03-24T22:03:07.332149+0000","last_peered":"2021-03-24T22:03:07.332149+0000","last_clean":"2021-03-24T22:03:07.332149+0000","last_became_active":"2021-03-24T22:00:49.305612+0000","last_became_peered":"2021-03-24T22:00:49.305612+0000","last_unstale":"2021-03-24T22:03:07.332149+0000","last_undegraded":"2021-03-24T22:03:07.332149+0000","last_fullsized":"2021-03-24T22:03:07.332149+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:03.626250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:59:03.626250+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"616'607","reported_seq":"2222","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.327629+0000","last_change":"2021-03-24T22:03:06.380492+0000","last_active":"2021-03-24T22:03:07.327629+0000","last_peered":"2021-03-24T22:03:07.327629+0000","last_clean":"2021-03-24T22:03:07.327629+0000","last_became_active":"2021-03-24T21:57:26.169302+0000","last_became_peered":"2021-03-24T21:57:26.169302+0000","last_unstale":"2021-03-24T22:03:07.327629+0000","last_undegraded":"2021-03-24T22:03:07.327629+0000","last_fullsized":"2021-03-24T22:03:07.327629+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'330","last_scrub_stamp":"2021-03-24T21:59:07.373692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:59:07.373692+0000","log_size":607,"ondisk_log_size":607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21149136,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":42244,"num_write":43,"num_write_kb":17707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"178","length":"1"},{"start":"1ae","length":"1"},{"start":"1c4","length":"1"}]},{"pgid":"5.0","version":"508'4","reported_seq":"742","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.325541+0000","last_change":"2021-03-24T22:01:16.693197+0000","last_active":"2021-03-24T22:03:07.325541+0000","last_peered":"2021-03-24T22:03:07.325541+0000","last_clean":"2021-03-24T22:03:07.325541+0000","last_became_active":"2021-03-24T22:01:16.300485+0000","last_became_peered":"2021-03-24T22:01:16.300485+0000","last_unstale":"2021-03-24T22:03:07.325541+0000","last_undegraded":"2021-03-24T22:03:07.325541+0000","last_fullsized":"2021-03-24T22:03:07.325541+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'3","last_scrub_stamp":"2021-03-24T21:59:46.375145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:59:46.375145+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"544'34","reported_seq":"266","reported_epoch":"619","state":"active+clean","last_fresh":"2021-03-24T22:03:09.342366+0000","last_change":"2021-03-24T22:02:32.116263+0000","last_active":"2021-03-24T22:03:09.342366+0000","last_peered":"2021-03-24T22:03:09.342366+0000","last_clean":"2021-03-24T22:03:09.342366+0000","last_became_active":"2021-03-24T22:01:43.388853+0000","last_became_peered":"2021-03-24T22:01:43.388853+0000","last_unstale":"2021-03-24T22:03:09.342366+0000","last_undegraded":"2021-03-24T22:03:09.342366+0000","last_fullsized":"2021-03-24T22:03:09.342366+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'34","last_scrub_stamp":"2021-03-24T22:02:32.116075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:02:32.116075+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":59,"num_read_kb":40,"num_write":34,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":940,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"621","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.476449+0000","last_change":"2021-03-24T22:01:16.336120+0000","last_active":"2021-03-24T22:03:07.476449+0000","last_peered":"2021-03-24T22:03:07.476449+0000","last_clean":"2021-03-24T22:03:07.476449+0000","last_became_active":"2021-03-24T22:01:16.335790+0000","last_became_peered":"2021-03-24T22:01:16.335790+0000","last_unstale":"2021-03-24T22:03:07.476449+0000","last_undegraded":"2021-03-24T22:03:07.476449+0000","last_fullsized":"2021-03-24T22:03:07.476449+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:28.165408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:28.165408+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"82","reported_epoch":"618","state":"active+clean","last_fresh":"2021-03-24T22:03:08.353301+0000","last_change":"2021-03-24T22:01:57.318244+0000","last_active":"2021-03-24T22:03:08.353301+0000","last_peered":"2021-03-24T22:03:08.353301+0000","last_clean":"2021-03-24T22:03:08.353301+0000","last_became_active":"2021-03-24T22:01:57.316610+0000","last_became_peered":"2021-03-24T22:01:57.316610+0000","last_unstale":"2021-03-24T22:03:08.353301+0000","last_undegraded":"2021-03-24T22:03:08.353301+0000","last_fullsized":"2021-03-24T22:03:08.353301+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:54.826534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:58:54.826534+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"272'64","reported_seq":"1664","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.331757+0000","last_change":"2021-03-24T22:03:06.353033+0000","last_active":"2021-03-24T22:03:07.331757+0000","last_peered":"2021-03-24T22:03:07.331757+0000","last_clean":"2021-03-24T22:03:07.331757+0000","last_became_active":"2021-03-24T22:01:16.299738+0000","last_became_peered":"2021-03-24T22:01:16.299738+0000","last_unstale":"2021-03-24T22:03:07.331757+0000","last_undegraded":"2021-03-24T22:03:07.331757+0000","last_fullsized":"2021-03-24T22:03:07.331757+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'64","last_scrub_stamp":"2021-03-24T21:59:08.406188+0000","last_deep_scrub":"272'64","last_deep_scrub_stamp":"2021-03-24T21:59:08.406188+0000","last_clean_scrub_stamp":"2021-03-24T21:59:08.406188+0000","log_size":64,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"178","length":"1"},{"start":"1ae","length":"1"},{"start":"1c4","length":"1"}]},{"pgid":"5.3","version":"533'7","reported_seq":"754","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.475892+0000","last_change":"2021-03-24T22:01:43.634882+0000","last_active":"2021-03-24T22:03:07.475892+0000","last_peered":"2021-03-24T22:03:07.475892+0000","last_clean":"2021-03-24T22:03:07.475892+0000","last_became_active":"2021-03-24T22:01:43.393325+0000","last_became_peered":"2021-03-24T22:01:43.393325+0000","last_unstale":"2021-03-24T22:03:07.475892+0000","last_undegraded":"2021-03-24T22:03:07.475892+0000","last_fullsized":"2021-03-24T22:03:07.475892+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'6","last_scrub_stamp":"2021-03-24T22:00:16.563338+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-24T21:53:26.584678+0000","last_clean_scrub_stamp":"2021-03-24T22:00:16.563338+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"544'26","reported_seq":"162","reported_epoch":"618","state":"active+clean","last_fresh":"2021-03-24T22:03:08.330490+0000","last_change":"2021-03-24T22:01:16.373466+0000","last_active":"2021-03-24T22:03:08.330490+0000","last_peered":"2021-03-24T22:03:08.330490+0000","last_clean":"2021-03-24T22:03:08.330490+0000","last_became_active":"2021-03-24T22:01:16.334794+0000","last_became_peered":"2021-03-24T22:01:16.334794+0000","last_unstale":"2021-03-24T22:03:08.330490+0000","last_undegraded":"2021-03-24T22:03:08.330490+0000","last_fullsized":"2021-03-24T22:03:08.330490+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T22:00:58.712071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:00:58.712071+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":47,"num_read_kb":30,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":940,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"475","reported_epoch":"619","state":"active+clean","last_fresh":"2021-03-24T22:03:09.343422+0000","last_change":"2021-03-24T22:03:05.892591+0000","last_active":"2021-03-24T22:03:09.343422+0000","last_peered":"2021-03-24T22:03:09.343422+0000","last_clean":"2021-03-24T22:03:09.343422+0000","last_became_active":"2021-03-24T22:00:03.284574+0000","last_became_peered":"2021-03-24T22:00:03.284574+0000","last_unstale":"2021-03-24T22:03:09.343422+0000","last_undegraded":"2021-03-24T22:03:09.343422+0000","last_fullsized":"2021-03-24T22:03:09.343422+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:05.892415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:03:05.892415+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"572","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.475463+0000","last_change":"2021-03-24T21:59:04.077747+0000","last_active":"2021-03-24T22:03:07.475463+0000","last_peered":"2021-03-24T22:03:07.475463+0000","last_clean":"2021-03-24T22:03:07.475463+0000","last_became_active":"2021-03-24T21:51:51.759627+0000","last_became_peered":"2021-03-24T21:51:51.759627+0000","last_unstale":"2021-03-24T22:03:07.475463+0000","last_undegraded":"2021-03-24T22:03:07.475463+0000","last_fullsized":"2021-03-24T22:03:07.475463+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:04.077569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:59:04.077569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"616'404","reported_seq":"2105","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.326205+0000","last_change":"2021-03-24T22:03:06.373067+0000","last_active":"2021-03-24T22:03:07.326205+0000","last_peered":"2021-03-24T22:03:07.326205+0000","last_clean":"2021-03-24T22:03:07.326205+0000","last_became_active":"2021-03-24T22:02:30.322922+0000","last_became_peered":"2021-03-24T22:02:30.322922+0000","last_unstale":"2021-03-24T22:03:07.326205+0000","last_undegraded":"2021-03-24T22:03:07.326205+0000","last_fullsized":"2021-03-24T22:03:07.326205+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"365'287","last_scrub_stamp":"2021-03-24T21:58:56.648040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:58:56.648040+0000","log_size":404,"ondisk_log_size":404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11619307,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":36562,"num_write":24,"num_write_kb":5739,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":24969090,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"178","length":"1"},{"start":"1ae","length":"1"},{"start":"1c4","length":"1"}]},{"pgid":"5.2","version":"0'0","reported_seq":"97","reported_epoch":"619","state":"active+clean","last_fresh":"2021-03-24T22:03:09.338929+0000","last_change":"2021-03-24T22:01:43.391207+0000","last_active":"2021-03-24T22:03:09.338929+0000","last_peered":"2021-03-24T22:03:09.338929+0000","last_clean":"2021-03-24T22:03:09.338929+0000","last_became_active":"2021-03-24T22:01:43.390850+0000","last_became_peered":"2021-03-24T22:01:43.390850+0000","last_unstale":"2021-03-24T22:03:09.338929+0000","last_undegraded":"2021-03-24T22:03:09.338929+0000","last_fullsized":"2021-03-24T22:03:09.338929+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:14.616769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:00:14.616769+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"544'31","reported_seq":"777","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.476029+0000","last_change":"2021-03-24T22:01:58.947168+0000","last_active":"2021-03-24T22:03:07.476029+0000","last_peered":"2021-03-24T22:03:07.476029+0000","last_clean":"2021-03-24T22:03:07.476029+0000","last_became_active":"2021-03-24T22:01:57.324818+0000","last_became_peered":"2021-03-24T22:01:57.324818+0000","last_unstale":"2021-03-24T22:03:07.476029+0000","last_undegraded":"2021-03-24T22:03:07.476029+0000","last_fullsized":"2021-03-24T22:03:07.476029+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'21","last_scrub_stamp":"2021-03-24T22:00:57.659209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:00:57.659209+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":60,"num_read_kb":42,"num_write":34,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":2568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"132","reported_epoch":"618","state":"active+clean","last_fresh":"2021-03-24T22:03:08.332298+0000","last_change":"2021-03-24T22:02:22.326409+0000","last_active":"2021-03-24T22:03:08.332298+0000","last_peered":"2021-03-24T22:03:08.332298+0000","last_clean":"2021-03-24T22:03:08.332298+0000","last_became_active":"2021-03-24T22:02:22.325996+0000","last_became_peered":"2021-03-24T22:02:22.325996+0000","last_unstale":"2021-03-24T22:03:08.332298+0000","last_undegraded":"2021-03-24T22:03:08.332298+0000","last_fullsized":"2021-03-24T22:03:08.332298+0000","mapping_epoch":571,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:38.144956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:38.144956+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"164","reported_epoch":"618","state":"active+clean","last_fresh":"2021-03-24T22:03:08.328849+0000","last_change":"2021-03-24T22:03:01.394566+0000","last_active":"2021-03-24T22:03:08.328849+0000","last_peered":"2021-03-24T22:03:08.328849+0000","last_clean":"2021-03-24T22:03:08.328849+0000","last_became_active":"2021-03-24T22:02:22.325346+0000","last_became_peered":"2021-03-24T22:02:22.325346+0000","last_unstale":"2021-03-24T22:03:08.328849+0000","last_undegraded":"2021-03-24T22:03:08.328849+0000","last_fullsized":"2021-03-24T22:03:08.328849+0000","mapping_epoch":571,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:01.394465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:03:01.394465+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"175'22","reported_seq":"1537","reported_epoch":"620","state":"active+clean","last_fresh":"2021-03-24T22:03:10.340471+0000","last_change":"2021-03-24T22:03:10.341092+0000","last_active":"2021-03-24T22:03:10.340471+0000","last_peered":"2021-03-24T22:03:10.340471+0000","last_clean":"2021-03-24T22:03:10.340471+0000","last_became_active":"2021-03-24T22:01:16.311297+0000","last_became_peered":"2021-03-24T22:01:16.311297+0000","last_unstale":"2021-03-24T22:03:10.340471+0000","last_undegraded":"2021-03-24T22:03:10.340471+0000","last_fullsized":"2021-03-24T22:03:10.340471+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'22","last_scrub_stamp":"2021-03-24T21:58:22.940875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:58:22.940875+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3283,"num_write":3,"num_write_kb":845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":313386,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"139","length":"1"},{"start":"178","length":"1"},{"start":"1ae","length":"1"},{"start":"1eb","length":"1"}]},{"pgid":"5.7","version":"0'0","reported_seq":"499","reported_epoch":"619","state":"active+clean","last_fresh":"2021-03-24T22:03:09.343918+0000","last_change":"2021-03-24T22:01:16.328244+0000","last_active":"2021-03-24T22:03:09.343918+0000","last_peered":"2021-03-24T22:03:09.343918+0000","last_clean":"2021-03-24T22:03:09.343918+0000","last_became_active":"2021-03-24T22:01:16.327948+0000","last_became_peered":"2021-03-24T22:01:16.327948+0000","last_unstale":"2021-03-24T22:03:09.343918+0000","last_undegraded":"2021-03-24T22:03:09.343918+0000","last_fullsized":"2021-03-24T22:03:09.343918+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:01:00.693679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:01:00.693679+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"544'34","reported_seq":"984","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.477068+0000","last_change":"2021-03-24T22:02:59.519639+0000","last_active":"2021-03-24T22:03:07.477068+0000","last_peered":"2021-03-24T22:03:07.477068+0000","last_clean":"2021-03-24T22:03:07.477068+0000","last_became_active":"2021-03-24T22:02:57.332868+0000","last_became_peered":"2021-03-24T22:02:57.332868+0000","last_unstale":"2021-03-24T22:03:07.477068+0000","last_undegraded":"2021-03-24T22:03:07.477068+0000","last_fullsized":"2021-03-24T22:03:07.477068+0000","mapping_epoch":606,"log_start":"69'16","ondisk_log_start":"69'16","created":62,"last_epoch_clean":607,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'34","last_scrub_stamp":"2021-03-24T22:02:59.519401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:02:59.519401+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1836,"num_objects":21,"num_object_clones":0,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":95,"num_read_kb":61,"num_write":51,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":9810,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"66'5","reported_seq":"158","reported_epoch":"618","state":"active+clean","last_fresh":"2021-03-24T22:03:08.460519+0000","last_change":"2021-03-24T22:01:57.334340+0000","last_active":"2021-03-24T22:03:08.460519+0000","last_peered":"2021-03-24T22:03:08.460519+0000","last_clean":"2021-03-24T22:03:08.460519+0000","last_became_active":"2021-03-24T22:01:57.325746+0000","last_became_peered":"2021-03-24T22:01:57.325746+0000","last_unstale":"2021-03-24T22:03:08.460519+0000","last_undegraded":"2021-03-24T22:03:08.460519+0000","last_fullsized":"2021-03-24T22:03:08.460519+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2021-03-24T21:58:58.818001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:39.286101+0000","last_clean_scrub_stamp":"2021-03-24T21:58:58.818001+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":146,"num_read_kb":141,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"592","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.475904+0000","last_change":"2021-03-24T21:59:46.941755+0000","last_active":"2021-03-24T22:03:07.475904+0000","last_peered":"2021-03-24T22:03:07.475904+0000","last_clean":"2021-03-24T22:03:07.475904+0000","last_became_active":"2021-03-24T21:59:12.281282+0000","last_became_peered":"2021-03-24T21:59:12.281282+0000","last_unstale":"2021-03-24T22:03:07.475904+0000","last_undegraded":"2021-03-24T22:03:07.475904+0000","last_fullsized":"2021-03-24T22:03:07.475904+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:46.941623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:59:46.941623+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"123","reported_epoch":"618","state":"active+clean","last_fresh":"2021-03-24T22:03:08.331803+0000","last_change":"2021-03-24T22:01:16.329175+0000","last_active":"2021-03-24T22:03:08.331803+0000","last_peered":"2021-03-24T22:03:08.331803+0000","last_clean":"2021-03-24T22:03:08.331803+0000","last_became_active":"2021-03-24T22:01:16.328444+0000","last_became_peered":"2021-03-24T22:01:16.328444+0000","last_unstale":"2021-03-24T22:03:08.331803+0000","last_undegraded":"2021-03-24T22:03:08.331803+0000","last_fullsized":"2021-03-24T22:03:08.331803+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:58.033876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:58:58.033876+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"195'69","reported_seq":"242","reported_epoch":"619","state":"active+clean","last_fresh":"2021-03-24T22:03:09.346485+0000","last_change":"2021-03-24T22:03:09.346879+0000","last_active":"2021-03-24T22:03:09.346485+0000","last_peered":"2021-03-24T22:03:09.346485+0000","last_clean":"2021-03-24T22:03:09.346485+0000","last_became_active":"2021-03-24T22:01:57.317669+0000","last_became_peered":"2021-03-24T22:01:57.317669+0000","last_unstale":"2021-03-24T22:03:09.346485+0000","last_undegraded":"2021-03-24T22:03:09.346485+0000","last_fullsized":"2021-03-24T22:03:09.346485+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'69","last_scrub_stamp":"2021-03-24T21:59:09.930390+0000","last_deep_scrub":"195'69","last_deep_scrub_stamp":"2021-03-24T21:56:49.886058+0000","last_clean_scrub_stamp":"2021-03-24T21:59:09.930390+0000","log_size":69,"ondisk_log_size":69,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1672296,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":8383,"num_write":20,"num_write_kb":7737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"139","length":"1"},{"start":"178","length":"1"},{"start":"1ae","length":"1"},{"start":"1c4","length":"1"}]},{"pgid":"5.4","version":"0'0","reported_seq":"89","reported_epoch":"618","state":"active+clean","last_fresh":"2021-03-24T22:03:08.355570+0000","last_change":"2021-03-24T22:02:34.319531+0000","last_active":"2021-03-24T22:03:08.355570+0000","last_peered":"2021-03-24T22:03:08.355570+0000","last_clean":"2021-03-24T22:03:08.355570+0000","last_became_active":"2021-03-24T22:01:57.322178+0000","last_became_peered":"2021-03-24T22:01:57.322178+0000","last_unstale":"2021-03-24T22:03:08.355570+0000","last_undegraded":"2021-03-24T22:03:08.355570+0000","last_fullsized":"2021-03-24T22:03:08.355570+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:34.319421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:02:34.319421+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"544'27","reported_seq":"169","reported_epoch":"619","state":"active+clean","last_fresh":"2021-03-24T22:03:09.339454+0000","last_change":"2021-03-24T22:03:00.744262+0000","last_active":"2021-03-24T22:03:09.339454+0000","last_peered":"2021-03-24T22:03:09.339454+0000","last_clean":"2021-03-24T22:03:09.339454+0000","last_became_active":"2021-03-24T22:01:57.311205+0000","last_became_peered":"2021-03-24T22:01:57.311205+0000","last_unstale":"2021-03-24T22:03:09.339454+0000","last_undegraded":"2021-03-24T22:03:09.339454+0000","last_fullsized":"2021-03-24T22:03:09.339454+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'27","last_scrub_stamp":"2021-03-24T22:03:00.744096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:03:00.744096+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":27,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":1398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"612","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.331911+0000","last_change":"2021-03-24T22:01:43.380966+0000","last_active":"2021-03-24T22:03:07.331911+0000","last_peered":"2021-03-24T22:03:07.331911+0000","last_clean":"2021-03-24T22:03:07.331911+0000","last_became_active":"2021-03-24T22:01:43.380686+0000","last_became_peered":"2021-03-24T22:01:43.380686+0000","last_unstale":"2021-03-24T22:03:07.331911+0000","last_undegraded":"2021-03-24T22:03:07.331911+0000","last_fullsized":"2021-03-24T22:03:07.331911+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:33.533840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:33.533840+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"82","reported_epoch":"618","state":"active+clean","last_fresh":"2021-03-24T22:03:08.352271+0000","last_change":"2021-03-24T22:01:57.316110+0000","last_active":"2021-03-24T22:03:08.352271+0000","last_peered":"2021-03-24T22:03:08.352271+0000","last_clean":"2021-03-24T22:03:08.352271+0000","last_became_active":"2021-03-24T22:01:57.315729+0000","last_became_peered":"2021-03-24T22:01:57.315729+0000","last_unstale":"2021-03-24T22:03:08.352271+0000","last_undegraded":"2021-03-24T22:03:08.352271+0000","last_fullsized":"2021-03-24T22:03:08.352271+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:21.039180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:49:51.529051+0000","last_clean_scrub_stamp":"2021-03-24T21:58:21.039180+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"471","reported_epoch":"619","state":"active+clean","last_fresh":"2021-03-24T22:03:09.343696+0000","last_change":"2021-03-24T22:03:01.931322+0000","last_active":"2021-03-24T22:03:09.343696+0000","last_peered":"2021-03-24T22:03:09.343696+0000","last_clean":"2021-03-24T22:03:09.343696+0000","last_became_active":"2021-03-24T22:00:03.278687+0000","last_became_peered":"2021-03-24T22:00:03.278687+0000","last_unstale":"2021-03-24T22:03:09.343696+0000","last_undegraded":"2021-03-24T22:03:09.343696+0000","last_fullsized":"2021-03-24T22:03:09.343696+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:01.931136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:03:01.931136+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"616'251","reported_seq":"2153","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.477294+0000","last_change":"2021-03-24T22:03:06.360568+0000","last_active":"2021-03-24T22:03:07.477294+0000","last_peered":"2021-03-24T22:03:07.477294+0000","last_clean":"2021-03-24T22:03:07.477294+0000","last_became_active":"2021-03-24T22:02:31.331916+0000","last_became_peered":"2021-03-24T22:02:31.331916+0000","last_unstale":"2021-03-24T22:03:07.477294+0000","last_undegraded":"2021-03-24T22:03:07.477294+0000","last_fullsized":"2021-03-24T22:03:07.477294+0000","mapping_epoch":580,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":581,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":251,"ondisk_log_size":251,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8645599,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":114,"num_read_kb":138053,"num_write":69,"num_write_kb":30533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":35483798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"178","length":"1"},{"start":"1ae","length":"1"},{"start":"1c4","length":"1"}]},{"pgid":"5.6","version":"0'0","reported_seq":"64","reported_epoch":"620","state":"active+clean","last_fresh":"2021-03-24T22:03:10.327858+0000","last_change":"2021-03-24T22:02:55.848268+0000","last_active":"2021-03-24T22:03:10.327858+0000","last_peered":"2021-03-24T22:03:10.327858+0000","last_clean":"2021-03-24T22:03:10.327858+0000","last_became_active":"2021-03-24T22:02:22.324262+0000","last_became_peered":"2021-03-24T22:02:22.324262+0000","last_unstale":"2021-03-24T22:03:10.327858+0000","last_undegraded":"2021-03-24T22:03:10.327858+0000","last_fullsized":"2021-03-24T22:03:10.327858+0000","mapping_epoch":571,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:55.848098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:02:55.848098+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"544'32","reported_seq":"630","reported_epoch":"618","state":"active+clean","last_fresh":"2021-03-24T22:03:08.332286+0000","last_change":"2021-03-24T22:02:59.400913+0000","last_active":"2021-03-24T22:03:08.332286+0000","last_peered":"2021-03-24T22:03:08.332286+0000","last_clean":"2021-03-24T22:03:08.332286+0000","last_became_active":"2021-03-24T22:02:51.334303+0000","last_became_peered":"2021-03-24T22:02:51.334303+0000","last_unstale":"2021-03-24T22:03:08.332286+0000","last_undegraded":"2021-03-24T22:03:08.332286+0000","last_fullsized":"2021-03-24T22:03:08.332286+0000","mapping_epoch":600,"log_start":"89'22","ondisk_log_start":"89'22","created":62,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'32","last_scrub_stamp":"2021-03-24T22:02:59.400796+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2021-03-24T21:53:19.717244+0000","last_clean_scrub_stamp":"2021-03-24T22:02:59.400796+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":802,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":63,"num_read_kb":43,"num_write":35,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"599","reported_epoch":"620","state":"active+clean","last_fresh":"2021-03-24T22:03:10.324653+0000","last_change":"2021-03-24T22:03:08.699641+0000","last_active":"2021-03-24T22:03:10.324653+0000","last_peered":"2021-03-24T22:03:10.324653+0000","last_clean":"2021-03-24T22:03:10.324653+0000","last_became_active":"2021-03-24T21:51:40.351000+0000","last_became_peered":"2021-03-24T21:51:40.351000+0000","last_unstale":"2021-03-24T22:03:10.324653+0000","last_undegraded":"2021-03-24T22:03:10.324653+0000","last_fullsized":"2021-03-24T22:03:10.324653+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:08.699469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:39.286101+0000","last_clean_scrub_stamp":"2021-03-24T22:03:08.699469+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"61'1","reported_seq":"132","reported_epoch":"618","state":"active+clean","last_fresh":"2021-03-24T22:03:08.330044+0000","last_change":"2021-03-24T22:01:43.380354+0000","last_active":"2021-03-24T22:03:08.330044+0000","last_peered":"2021-03-24T22:03:08.330044+0000","last_clean":"2021-03-24T22:03:08.330044+0000","last_became_active":"2021-03-24T22:01:43.379922+0000","last_became_peered":"2021-03-24T22:01:43.379922+0000","last_unstale":"2021-03-24T22:03:08.330044+0000","last_undegraded":"2021-03-24T22:03:08.330044+0000","last_fullsized":"2021-03-24T22:03:08.330044+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2021-03-24T21:57:12.046206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:12.046206+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":436,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"82","reported_epoch":"618","state":"active+clean","last_fresh":"2021-03-24T22:03:08.353940+0000","last_change":"2021-03-24T22:01:57.318666+0000","last_active":"2021-03-24T22:03:08.353940+0000","last_peered":"2021-03-24T22:03:08.353940+0000","last_clean":"2021-03-24T22:03:08.353940+0000","last_became_active":"2021-03-24T22:01:57.318275+0000","last_became_peered":"2021-03-24T22:01:57.318275+0000","last_unstale":"2021-03-24T22:03:08.353940+0000","last_undegraded":"2021-03-24T22:03:08.353940+0000","last_fullsized":"2021-03-24T22:03:08.353940+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:56.065113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:57:56.065113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"616'351","reported_seq":"2039","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.476197+0000","last_change":"2021-03-24T22:03:06.473259+0000","last_active":"2021-03-24T22:03:07.476197+0000","last_peered":"2021-03-24T22:03:07.476197+0000","last_clean":"2021-03-24T22:03:07.476197+0000","last_became_active":"2021-03-24T22:01:25.281813+0000","last_became_peered":"2021-03-24T22:01:25.281813+0000","last_unstale":"2021-03-24T22:03:07.476197+0000","last_undegraded":"2021-03-24T22:03:07.476197+0000","last_fullsized":"2021-03-24T22:03:07.476197+0000","mapping_epoch":514,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"616'351","last_scrub_stamp":"2021-03-24T22:03:06.473123+0000","last_deep_scrub":"616'351","last_deep_scrub_stamp":"2021-03-24T22:03:06.473123+0000","last_clean_scrub_stamp":"2021-03-24T22:03:06.473123+0000","log_size":351,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8928768,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":63,"num_read_kb":67259,"num_write":39,"num_write_kb":16303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":22504555,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"178","length":"1"},{"start":"1ae","length":"1"},{"start":"1c4","length":"1"}]},{"pgid":"5.5","version":"0'0","reported_seq":"610","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.476619+0000","last_change":"2021-03-24T22:01:16.337675+0000","last_active":"2021-03-24T22:03:07.476619+0000","last_peered":"2021-03-24T22:03:07.476619+0000","last_clean":"2021-03-24T22:03:07.476619+0000","last_became_active":"2021-03-24T22:01:16.336798+0000","last_became_peered":"2021-03-24T22:01:16.336798+0000","last_unstale":"2021-03-24T22:03:07.476619+0000","last_undegraded":"2021-03-24T22:03:07.476619+0000","last_fullsized":"2021-03-24T22:03:07.476619+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:23.865909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:58:23.865909+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"544'37","reported_seq":"169","reported_epoch":"618","state":"active+clean","last_fresh":"2021-03-24T22:03:08.331685+0000","last_change":"2021-03-24T22:01:16.366977+0000","last_active":"2021-03-24T22:03:08.331685+0000","last_peered":"2021-03-24T22:03:08.331685+0000","last_clean":"2021-03-24T22:03:08.331685+0000","last_became_active":"2021-03-24T22:01:16.309129+0000","last_became_peered":"2021-03-24T22:01:16.309129+0000","last_unstale":"2021-03-24T22:03:08.331685+0000","last_undegraded":"2021-03-24T22:03:08.331685+0000","last_fullsized":"2021-03-24T22:03:08.331685+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'25","last_scrub_stamp":"2021-03-24T21:58:54.693601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:58:54.693601+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":71,"num_read_kb":48,"num_write":40,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":3816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"89'2","reported_seq":"216","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.477316+0000","last_change":"2021-03-24T22:01:43.721006+0000","last_active":"2021-03-24T22:03:07.477316+0000","last_peered":"2021-03-24T22:03:07.477316+0000","last_clean":"2021-03-24T22:03:07.477316+0000","last_became_active":"2021-03-24T22:01:43.393718+0000","last_became_peered":"2021-03-24T22:01:43.393718+0000","last_unstale":"2021-03-24T22:03:07.477316+0000","last_undegraded":"2021-03-24T22:03:07.477316+0000","last_fullsized":"2021-03-24T22:03:07.477316+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2021-03-24T21:57:58.491830+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2021-03-24T21:57:58.491830+0000","last_clean_scrub_stamp":"2021-03-24T21:57:58.491830+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":179,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"137","reported_epoch":"618","state":"active+clean","last_fresh":"2021-03-24T22:03:08.330654+0000","last_change":"2021-03-24T22:02:20.729667+0000","last_active":"2021-03-24T22:03:08.330654+0000","last_peered":"2021-03-24T22:03:08.330654+0000","last_clean":"2021-03-24T22:03:08.330654+0000","last_became_active":"2021-03-24T22:01:43.401295+0000","last_became_peered":"2021-03-24T22:01:43.401295+0000","last_unstale":"2021-03-24T22:03:08.330654+0000","last_undegraded":"2021-03-24T22:03:08.330654+0000","last_fullsized":"2021-03-24T22:03:08.330654+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:20.729422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:06.800417+0000","last_clean_scrub_stamp":"2021-03-24T22:02:20.729422+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"633","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.329527+0000","last_change":"2021-03-24T22:02:55.490903+0000","last_active":"2021-03-24T22:03:07.329527+0000","last_peered":"2021-03-24T22:03:07.329527+0000","last_clean":"2021-03-24T22:03:07.329527+0000","last_became_active":"2021-03-24T22:01:57.314372+0000","last_became_peered":"2021-03-24T22:01:57.314372+0000","last_unstale":"2021-03-24T22:03:07.329527+0000","last_undegraded":"2021-03-24T22:03:07.329527+0000","last_fullsized":"2021-03-24T22:03:07.329527+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:55.490744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:52:59.706511+0000","last_clean_scrub_stamp":"2021-03-24T22:02:55.490744+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"620'452","reported_seq":"2224","reported_epoch":"620","state":"active+clean","last_fresh":"2021-03-24T22:03:10.351795+0000","last_change":"2021-03-24T22:03:10.352067+0000","last_active":"2021-03-24T22:03:10.351795+0000","last_peered":"2021-03-24T22:03:10.351795+0000","last_clean":"2021-03-24T22:03:10.351795+0000","last_became_active":"2021-03-24T22:01:47.305050+0000","last_became_peered":"2021-03-24T22:01:47.305050+0000","last_unstale":"2021-03-24T22:03:10.351795+0000","last_undegraded":"2021-03-24T22:03:10.351795+0000","last_fullsized":"2021-03-24T22:03:10.351795+0000","mapping_epoch":536,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":537,"parent":"0.0","parent_split_bits":0,"last_scrub":"563'386","last_scrub_stamp":"2021-03-24T22:02:19.755652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:02:19.755652+0000","log_size":452,"ondisk_log_size":452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10427890,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":23,"num_read_kb":7097,"num_write":50,"num_write_kb":18896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":55849492,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"139","length":"1"},{"start":"178","length":"1"},{"start":"1ae","length":"1"},{"start":"1eb","length":"1"}]},{"pgid":"4.b","version":"544'17","reported_seq":"712","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.476585+0000","last_change":"2021-03-24T22:01:56.334554+0000","last_active":"2021-03-24T22:03:07.476585+0000","last_peered":"2021-03-24T22:03:07.476585+0000","last_clean":"2021-03-24T22:03:07.476585+0000","last_became_active":"2021-03-24T22:01:56.334081+0000","last_became_peered":"2021-03-24T22:01:56.334081+0000","last_unstale":"2021-03-24T22:03:07.476585+0000","last_undegraded":"2021-03-24T22:03:07.476585+0000","last_fullsized":"2021-03-24T22:03:07.476585+0000","mapping_epoch":545,"log_start":"91'9","ondisk_log_start":"91'9","created":62,"last_epoch_clean":546,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'14","last_scrub_stamp":"2021-03-24T21:59:05.058513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:59:05.058513+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1158,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":58,"num_read_kb":36,"num_write":30,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2316,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"608","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.331443+0000","last_change":"2021-03-24T22:01:55.431327+0000","last_active":"2021-03-24T22:03:07.331443+0000","last_peered":"2021-03-24T22:03:07.331443+0000","last_clean":"2021-03-24T22:03:07.331443+0000","last_became_active":"2021-03-24T22:01:43.390371+0000","last_became_peered":"2021-03-24T22:01:43.390371+0000","last_unstale":"2021-03-24T22:03:07.331443+0000","last_undegraded":"2021-03-24T22:03:07.331443+0000","last_fullsized":"2021-03-24T22:03:07.331443+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:01:55.431129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T22:01:55.431129+0000","last_clean_scrub_stamp":"2021-03-24T22:01:55.431129+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"586","reported_epoch":"620","state":"active+clean","last_fresh":"2021-03-24T22:03:10.325184+0000","last_change":"2021-03-24T22:02:21.716625+0000","last_active":"2021-03-24T22:03:10.325184+0000","last_peered":"2021-03-24T22:03:10.325184+0000","last_clean":"2021-03-24T22:03:10.325184+0000","last_became_active":"2021-03-24T21:51:49.411921+0000","last_became_peered":"2021-03-24T21:51:49.411921+0000","last_unstale":"2021-03-24T22:03:10.325184+0000","last_undegraded":"2021-03-24T22:03:10.325184+0000","last_fullsized":"2021-03-24T22:03:10.325184+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:21.716448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:02:21.716448+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"635","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.475542+0000","last_change":"2021-03-24T22:02:22.324168+0000","last_active":"2021-03-24T22:03:07.475542+0000","last_peered":"2021-03-24T22:03:07.475542+0000","last_clean":"2021-03-24T22:03:07.475542+0000","last_became_active":"2021-03-24T22:02:22.323876+0000","last_became_peered":"2021-03-24T22:02:22.323876+0000","last_unstale":"2021-03-24T22:03:07.475542+0000","last_undegraded":"2021-03-24T22:03:07.475542+0000","last_fullsized":"2021-03-24T22:03:07.475542+0000","mapping_epoch":571,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:00.033461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:35.636299+0000","last_clean_scrub_stamp":"2021-03-24T21:59:00.033461+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"619'282","reported_seq":"2055","reported_epoch":"621","state":"active+clean","last_fresh":"2021-03-24T22:03:11.342926+0000","last_change":"2021-03-24T22:03:11.343697+0000","last_active":"2021-03-24T22:03:11.342926+0000","last_peered":"2021-03-24T22:03:11.342926+0000","last_clean":"2021-03-24T22:03:11.342926+0000","last_became_active":"2021-03-24T22:02:01.325483+0000","last_became_peered":"2021-03-24T22:02:01.325483+0000","last_unstale":"2021-03-24T22:03:11.342926+0000","last_undegraded":"2021-03-24T22:03:11.342926+0000","last_fullsized":"2021-03-24T22:03:11.342926+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'168","last_scrub_stamp":"2021-03-24T21:59:44.828980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:59:44.828980+0000","log_size":282,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5165516,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":53,"num_read_kb":66903,"num_write":42,"num_write_kb":11344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":21446378,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"139","length":"1"},{"start":"178","length":"1"},{"start":"1ae","length":"1"},{"start":"1eb","length":"1"},{"start":"1f4","length":"1"}]},{"pgid":"4.a","version":"544'45","reported_seq":"447","reported_epoch":"619","state":"active+clean","last_fresh":"2021-03-24T22:03:09.338795+0000","last_change":"2021-03-24T22:02:11.306107+0000","last_active":"2021-03-24T22:03:09.338795+0000","last_peered":"2021-03-24T22:03:09.338795+0000","last_clean":"2021-03-24T22:03:09.338795+0000","last_became_active":"2021-03-24T22:02:11.305345+0000","last_became_peered":"2021-03-24T22:02:11.305345+0000","last_unstale":"2021-03-24T22:03:09.338795+0000","last_undegraded":"2021-03-24T22:03:09.338795+0000","last_fullsized":"2021-03-24T22:03:09.338795+0000","mapping_epoch":560,"log_start":"69'23","ondisk_log_start":"69'23","created":62,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'23","last_scrub_stamp":"2021-03-24T22:00:23.343879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:00:23.343879+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1466,"num_objects":21,"num_object_clones":0,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":104,"num_read_kb":71,"num_write":60,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":4398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":"308","reported_epoch":"619","state":"active+clean","last_fresh":"2021-03-24T22:03:09.343081+0000","last_change":"2021-03-24T22:03:00.965270+0000","last_active":"2021-03-24T22:03:09.343081+0000","last_peered":"2021-03-24T22:03:09.343081+0000","last_clean":"2021-03-24T22:03:09.343081+0000","last_became_active":"2021-03-24T22:01:57.323503+0000","last_became_peered":"2021-03-24T22:01:57.323503+0000","last_unstale":"2021-03-24T22:03:09.343081+0000","last_undegraded":"2021-03-24T22:03:09.343081+0000","last_fullsized":"2021-03-24T22:03:09.343081+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:00.965024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:03:00.965024+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"59","reported_epoch":"620","state":"active+clean","last_fresh":"2021-03-24T22:03:10.324588+0000","last_change":"2021-03-24T22:02:22.323418+0000","last_active":"2021-03-24T22:03:10.324588+0000","last_peered":"2021-03-24T22:03:10.324588+0000","last_clean":"2021-03-24T22:03:10.324588+0000","last_became_active":"2021-03-24T22:02:22.323020+0000","last_became_peered":"2021-03-24T22:02:22.323020+0000","last_unstale":"2021-03-24T22:03:10.324588+0000","last_undegraded":"2021-03-24T22:03:10.324588+0000","last_fullsized":"2021-03-24T22:03:10.324588+0000","mapping_epoch":571,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:01:40.445404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:01:40.445404+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"379","reported_epoch":"620","state":"active+clean","last_fresh":"2021-03-24T22:03:10.323923+0000","last_change":"2021-03-24T22:01:57.311688+0000","last_active":"2021-03-24T22:03:10.323923+0000","last_peered":"2021-03-24T22:03:10.323923+0000","last_clean":"2021-03-24T22:03:10.323923+0000","last_became_active":"2021-03-24T22:01:57.311325+0000","last_became_peered":"2021-03-24T22:01:57.311325+0000","last_unstale":"2021-03-24T22:03:10.323923+0000","last_undegraded":"2021-03-24T22:03:10.323923+0000","last_fullsized":"2021-03-24T22:03:10.323923+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:42.813883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:59:42.813883+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"616'399","reported_seq":"2093","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.475455+0000","last_change":"2021-03-24T22:03:06.357522+0000","last_active":"2021-03-24T22:03:07.475455+0000","last_peered":"2021-03-24T22:03:07.475455+0000","last_clean":"2021-03-24T22:03:07.475455+0000","last_became_active":"2021-03-24T22:01:22.309266+0000","last_became_peered":"2021-03-24T22:01:22.309266+0000","last_unstale":"2021-03-24T22:03:07.475455+0000","last_undegraded":"2021-03-24T22:03:07.475455+0000","last_fullsized":"2021-03-24T22:03:07.475455+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"609'395","last_scrub_stamp":"2021-03-24T22:03:04.501653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:03:04.501653+0000","log_size":399,"ondisk_log_size":399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13753547,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":64,"num_read_kb":81138,"num_write":41,"num_write_kb":13318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":11840390,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"178","length":"1"},{"start":"1ae","length":"1"},{"start":"1c4","length":"1"}]},{"pgid":"5.8","version":"0'0","reported_seq":"619","reported_epoch":"620","state":"active+clean","last_fresh":"2021-03-24T22:03:10.323291+0000","last_change":"2021-03-24T22:02:58.801826+0000","last_active":"2021-03-24T22:03:10.323291+0000","last_peered":"2021-03-24T22:03:10.323291+0000","last_clean":"2021-03-24T22:03:10.323291+0000","last_became_active":"2021-03-24T22:01:16.329336+0000","last_became_peered":"2021-03-24T22:01:16.329336+0000","last_unstale":"2021-03-24T22:03:10.323291+0000","last_undegraded":"2021-03-24T22:03:10.323291+0000","last_fullsized":"2021-03-24T22:03:10.323291+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:58.801650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:02:58.801650+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"544'38","reported_seq":"293","reported_epoch":"618","state":"active+clean","last_fresh":"2021-03-24T22:03:08.328389+0000","last_change":"2021-03-24T22:03:03.442490+0000","last_active":"2021-03-24T22:03:08.328389+0000","last_peered":"2021-03-24T22:03:08.328389+0000","last_clean":"2021-03-24T22:03:08.328389+0000","last_became_active":"2021-03-24T22:01:27.288204+0000","last_became_peered":"2021-03-24T22:01:27.288204+0000","last_unstale":"2021-03-24T22:03:08.328389+0000","last_undegraded":"2021-03-24T22:03:08.328389+0000","last_fullsized":"2021-03-24T22:03:08.328389+0000","mapping_epoch":516,"log_start":"69'20","ondisk_log_start":"69'20","created":62,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'38","last_scrub_stamp":"2021-03-24T22:03:03.442382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:03:03.442382+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1582,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":101,"num_read_kb":67,"num_write":56,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1582,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":"82","reported_epoch":"618","state":"active+clean","last_fresh":"2021-03-24T22:03:08.355160+0000","last_change":"2021-03-24T22:01:57.326649+0000","last_active":"2021-03-24T22:03:08.355160+0000","last_peered":"2021-03-24T22:03:08.355160+0000","last_clean":"2021-03-24T22:03:08.355160+0000","last_became_active":"2021-03-24T22:01:57.325643+0000","last_became_peered":"2021-03-24T22:01:57.325643+0000","last_unstale":"2021-03-24T22:03:08.355160+0000","last_undegraded":"2021-03-24T22:03:08.355160+0000","last_fullsized":"2021-03-24T22:03:08.355160+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:12.939226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:59:12.939226+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"632","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.327131+0000","last_change":"2021-03-24T22:03:03.347078+0000","last_active":"2021-03-24T22:03:07.327131+0000","last_peered":"2021-03-24T22:03:07.327131+0000","last_clean":"2021-03-24T22:03:07.327131+0000","last_became_active":"2021-03-24T22:03:03.346637+0000","last_became_peered":"2021-03-24T22:03:03.346637+0000","last_unstale":"2021-03-24T22:03:07.327131+0000","last_undegraded":"2021-03-24T22:03:07.327131+0000","last_fullsized":"2021-03-24T22:03:07.327131+0000","mapping_epoch":612,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":613,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:18.322187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:22.724840+0000","last_clean_scrub_stamp":"2021-03-24T22:00:18.322187+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"618'267","reported_seq":"1845","reported_epoch":"618","state":"active+clean","last_fresh":"2021-03-24T22:03:08.361380+0000","last_change":"2021-03-24T22:03:06.377983+0000","last_active":"2021-03-24T22:03:08.361380+0000","last_peered":"2021-03-24T22:03:08.361380+0000","last_clean":"2021-03-24T22:03:08.361380+0000","last_became_active":"2021-03-24T22:00:03.286696+0000","last_became_peered":"2021-03-24T22:00:03.286696+0000","last_unstale":"2021-03-24T22:03:08.361380+0000","last_undegraded":"2021-03-24T22:03:08.361380+0000","last_fullsized":"2021-03-24T22:03:08.361380+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":267,"ondisk_log_size":267,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18365137,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":65,"num_read_kb":61112,"num_write":70,"num_write_kb":28074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":11591117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"178","length":"1"},{"start":"1ae","length":"1"},{"start":"1c4","length":"1"}]},{"pgid":"5.9","version":"548'11","reported_seq":"101","reported_epoch":"618","state":"active+clean","last_fresh":"2021-03-24T22:03:09.252726+0000","last_change":"2021-03-24T22:01:57.504107+0000","last_active":"2021-03-24T22:03:09.252726+0000","last_peered":"2021-03-24T22:03:09.252726+0000","last_clean":"2021-03-24T22:03:09.252726+0000","last_became_active":"2021-03-24T22:01:57.313563+0000","last_became_peered":"2021-03-24T22:01:57.313563+0000","last_unstale":"2021-03-24T22:03:09.252726+0000","last_undegraded":"2021-03-24T22:03:09.252726+0000","last_fullsized":"2021-03-24T22:03:09.252726+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'4","last_scrub_stamp":"2021-03-24T21:59:06.906705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:59:06.906705+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"544'46","reported_seq":"692","reported_epoch":"619","state":"active+remapped+backfill_toofull","last_fresh":"2021-03-24T22:03:09.342161+0000","last_change":"2021-03-24T22:02:57.332413+0000","last_active":"2021-03-24T22:03:09.342161+0000","last_peered":"2021-03-24T22:03:09.342161+0000","last_clean":"2021-03-24T22:01:41.303005+0000","last_became_active":"2021-03-24T22:01:57.327903+0000","last_became_peered":"2021-03-24T22:01:57.327903+0000","last_unstale":"2021-03-24T22:03:09.342161+0000","last_undegraded":"2021-03-24T22:03:09.342161+0000","last_fullsized":"2021-03-24T22:03:09.342161+0000","mapping_epoch":546,"log_start":"97'28","ondisk_log_start":"97'28","created":62,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'28","last_scrub_stamp":"2021-03-24T21:58:20.925603+0000","last_deep_scrub":"97'28","last_deep_scrub_stamp":"2021-03-24T21:54:19.157265+0000","last_clean_scrub_stamp":"2021-03-24T21:58:20.925603+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1616,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":40,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":97,"num_read_kb":64,"num_write":53,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":3692,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,4,3],"avail_no_missing":["5","3","4"],"object_location_counts":[{"shards":"3,4,5","objects":20}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"89'4","reported_seq":"88","reported_epoch":"618","state":"active+clean","last_fresh":"2021-03-24T22:03:08.356114+0000","last_change":"2021-03-24T22:01:57.336372+0000","last_active":"2021-03-24T22:03:08.356114+0000","last_peered":"2021-03-24T22:03:08.356114+0000","last_clean":"2021-03-24T22:03:08.356114+0000","last_became_active":"2021-03-24T22:01:57.316325+0000","last_became_peered":"2021-03-24T22:01:57.316325+0000","last_unstale":"2021-03-24T22:03:08.356114+0000","last_undegraded":"2021-03-24T22:03:08.356114+0000","last_fullsized":"2021-03-24T22:03:08.356114+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'4","last_scrub_stamp":"2021-03-24T21:58:25.969865+0000","last_deep_scrub":"89'4","last_deep_scrub_stamp":"2021-03-24T21:53:19.926637+0000","last_clean_scrub_stamp":"2021-03-24T21:58:25.969865+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"102","reported_epoch":"619","state":"active+clean","last_fresh":"2021-03-24T22:03:09.339548+0000","last_change":"2021-03-24T22:01:51.981589+0000","last_active":"2021-03-24T22:03:09.339548+0000","last_peered":"2021-03-24T22:03:09.339548+0000","last_clean":"2021-03-24T22:03:09.339548+0000","last_became_active":"2021-03-24T22:01:43.394043+0000","last_became_peered":"2021-03-24T22:01:43.394043+0000","last_unstale":"2021-03-24T22:03:09.339548+0000","last_undegraded":"2021-03-24T22:03:09.339548+0000","last_fullsized":"2021-03-24T22:03:09.339548+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:01:51.981423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:01:51.981423+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"272'275","reported_seq":"1330","reported_epoch":"619","state":"active+clean","last_fresh":"2021-03-24T22:03:09.334364+0000","last_change":"2021-03-24T22:03:09.334681+0000","last_active":"2021-03-24T22:03:09.334364+0000","last_peered":"2021-03-24T22:03:09.334364+0000","last_clean":"2021-03-24T22:03:09.334364+0000","last_became_active":"2021-03-24T22:01:46.306345+0000","last_became_peered":"2021-03-24T22:01:46.306345+0000","last_unstale":"2021-03-24T22:03:09.334364+0000","last_undegraded":"2021-03-24T22:03:09.334364+0000","last_fullsized":"2021-03-24T22:03:09.334364+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'275","last_scrub_stamp":"2021-03-24T21:58:23.942036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:58:23.942036+0000","log_size":275,"ondisk_log_size":275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8857338,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":114,"num_read_kb":132714,"num_write":52,"num_write_kb":22259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":12587455,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"139","length":"1"},{"start":"178","length":"1"},{"start":"1ae","length":"1"},{"start":"1c4","length":"1"}]},{"pgid":"3.10","version":"0'0","reported_seq":"104","reported_epoch":"619","state":"active+clean","last_fresh":"2021-03-24T22:03:09.338727+0000","last_change":"2021-03-24T22:01:56.876290+0000","last_active":"2021-03-24T22:03:09.338727+0000","last_peered":"2021-03-24T22:03:09.338727+0000","last_clean":"2021-03-24T22:03:09.338727+0000","last_became_active":"2021-03-24T22:01:43.392580+0000","last_became_peered":"2021-03-24T22:01:43.392580+0000","last_unstale":"2021-03-24T22:03:09.338727+0000","last_undegraded":"2021-03-24T22:03:09.338727+0000","last_fullsized":"2021-03-24T22:03:09.338727+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:01:56.876119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:01:56.876119+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"510","reported_epoch":"619","state":"active+clean","last_fresh":"2021-03-24T22:03:09.343029+0000","last_change":"2021-03-24T22:01:16.312978+0000","last_active":"2021-03-24T22:03:09.343029+0000","last_peered":"2021-03-24T22:03:09.343029+0000","last_clean":"2021-03-24T22:03:09.343029+0000","last_became_active":"2021-03-24T22:01:16.311233+0000","last_became_peered":"2021-03-24T22:01:16.311233+0000","last_unstale":"2021-03-24T22:03:09.343029+0000","last_undegraded":"2021-03-24T22:03:09.343029+0000","last_fullsized":"2021-03-24T22:03:09.343029+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:26.392232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T22:00:26.392232+0000","last_clean_scrub_stamp":"2021-03-24T22:00:26.392232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"87","reported_epoch":"618","state":"active+clean","last_fresh":"2021-03-24T22:03:08.353839+0000","last_change":"2021-03-24T22:03:01.351580+0000","last_active":"2021-03-24T22:03:08.353839+0000","last_peered":"2021-03-24T22:03:08.353839+0000","last_clean":"2021-03-24T22:03:08.353839+0000","last_became_active":"2021-03-24T22:01:57.312376+0000","last_became_peered":"2021-03-24T22:01:57.312376+0000","last_unstale":"2021-03-24T22:03:08.353839+0000","last_undegraded":"2021-03-24T22:03:08.353839+0000","last_fullsized":"2021-03-24T22:03:08.353839+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:01.351474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:03:01.351474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.14","version":"616'692","reported_seq":"2336","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.327374+0000","last_change":"2021-03-24T22:03:06.365273+0000","last_active":"2021-03-24T22:03:07.327374+0000","last_peered":"2021-03-24T22:03:07.327374+0000","last_clean":"2021-03-24T22:03:07.327374+0000","last_became_active":"2021-03-24T22:00:03.281974+0000","last_became_peered":"2021-03-24T22:00:03.281974+0000","last_unstale":"2021-03-24T22:03:07.327374+0000","last_undegraded":"2021-03-24T22:03:07.327374+0000","last_fullsized":"2021-03-24T22:03:07.327374+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":692,"ondisk_log_size":692,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44871570,"num_objects":18,"num_object_clones":15,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":1,"num_read":282,"num_read_kb":285896,"num_write":193,"num_write_kb":73935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":33311730,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"178","length":"1"},{"start":"1ae","length":"1"},{"start":"1c4","length":"1"}]},{"pgid":"3.11","version":"0'0","reported_seq":"111","reported_epoch":"619","state":"active+clean","last_fresh":"2021-03-24T22:03:09.345919+0000","last_change":"2021-03-24T22:02:22.796277+0000","last_active":"2021-03-24T22:03:09.345919+0000","last_peered":"2021-03-24T22:03:09.345919+0000","last_clean":"2021-03-24T22:03:09.345919+0000","last_became_active":"2021-03-24T22:01:57.304423+0000","last_became_peered":"2021-03-24T22:01:57.304423+0000","last_unstale":"2021-03-24T22:03:09.345919+0000","last_undegraded":"2021-03-24T22:03:09.345919+0000","last_fullsized":"2021-03-24T22:03:09.345919+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:22.796161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:02:22.796161+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"633","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.329668+0000","last_change":"2021-03-24T22:02:56.523447+0000","last_active":"2021-03-24T22:03:07.329668+0000","last_peered":"2021-03-24T22:03:07.329668+0000","last_clean":"2021-03-24T22:03:07.329668+0000","last_became_active":"2021-03-24T22:01:57.315711+0000","last_became_peered":"2021-03-24T22:01:57.315711+0000","last_unstale":"2021-03-24T22:03:07.329668+0000","last_undegraded":"2021-03-24T22:03:07.329668+0000","last_fullsized":"2021-03-24T22:03:07.329668+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:56.523309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:55.493713+0000","last_clean_scrub_stamp":"2021-03-24T22:02:56.523309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"620","reported_epoch":"620","state":"active+clean","last_fresh":"2021-03-24T22:03:10.324967+0000","last_change":"2021-03-24T22:03:09.686125+0000","last_active":"2021-03-24T22:03:10.324967+0000","last_peered":"2021-03-24T22:03:10.324967+0000","last_clean":"2021-03-24T22:03:10.324967+0000","last_became_active":"2021-03-24T22:01:57.312185+0000","last_became_peered":"2021-03-24T22:01:57.312185+0000","last_unstale":"2021-03-24T22:03:10.324967+0000","last_undegraded":"2021-03-24T22:03:10.324967+0000","last_fullsized":"2021-03-24T22:03:10.324967+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:09.686047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:03:09.686047+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"619'230","reported_seq":"1997","reported_epoch":"619","state":"active+clean","last_fresh":"2021-03-24T22:03:09.359825+0000","last_change":"2021-03-24T22:03:09.360147+0000","last_active":"2021-03-24T22:03:09.359825+0000","last_peered":"2021-03-24T22:03:09.359825+0000","last_clean":"2021-03-24T22:03:09.359825+0000","last_became_active":"2021-03-24T22:02:00.309364+0000","last_became_peered":"2021-03-24T22:02:00.309364+0000","last_unstale":"2021-03-24T22:03:09.359825+0000","last_undegraded":"2021-03-24T22:03:09.359825+0000","last_fullsized":"2021-03-24T22:03:09.359825+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":5,"last_scrub":"447'140","last_scrub_stamp":"2021-03-24T22:00:17.991497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:00:17.991497+0000","log_size":230,"ondisk_log_size":230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3832062,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":48,"num_read_kb":65320,"num_write":32,"num_write_kb":12965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":76697653,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"139","length":"1"},{"start":"178","length":"1"},{"start":"1ae","length":"1"},{"start":"1c4","length":"1"}]},{"pgid":"4.15","version":"544'19","reported_seq":"626","reported_epoch":"619","state":"active+clean","last_fresh":"2021-03-24T22:03:09.343296+0000","last_change":"2021-03-24T22:01:58.033012+0000","last_active":"2021-03-24T22:03:09.343296+0000","last_peered":"2021-03-24T22:03:09.343296+0000","last_clean":"2021-03-24T22:03:09.343296+0000","last_became_active":"2021-03-24T22:01:57.323247+0000","last_became_peered":"2021-03-24T22:01:57.323247+0000","last_unstale":"2021-03-24T22:03:09.343296+0000","last_undegraded":"2021-03-24T22:03:09.343296+0000","last_fullsized":"2021-03-24T22:03:09.343296+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2021-03-24T22:00:29.462476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:00:29.462476+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":37,"num_read_kb":23,"num_write":20,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":2936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":"619","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.475771+0000","last_change":"2021-03-24T22:01:57.316563+0000","last_active":"2021-03-24T22:03:07.475771+0000","last_peered":"2021-03-24T22:03:07.475771+0000","last_clean":"2021-03-24T22:03:07.475771+0000","last_became_active":"2021-03-24T22:01:57.315967+0000","last_became_peered":"2021-03-24T22:01:57.315967+0000","last_unstale":"2021-03-24T22:03:07.475771+0000","last_undegraded":"2021-03-24T22:03:07.475771+0000","last_fullsized":"2021-03-24T22:03:07.475771+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:28.482500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:05.473348+0000","last_clean_scrub_stamp":"2021-03-24T22:00:28.482500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"573'12","reported_seq":"810","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.331544+0000","last_change":"2021-03-24T22:03:02.456113+0000","last_active":"2021-03-24T22:03:07.331544+0000","last_peered":"2021-03-24T22:03:07.331544+0000","last_clean":"2021-03-24T22:03:07.331544+0000","last_became_active":"2021-03-24T22:02:22.326377+0000","last_became_peered":"2021-03-24T22:02:22.326377+0000","last_unstale":"2021-03-24T22:03:07.331544+0000","last_undegraded":"2021-03-24T22:03:07.331544+0000","last_fullsized":"2021-03-24T22:03:07.331544+0000","mapping_epoch":571,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":0,"last_scrub":"573'12","last_scrub_stamp":"2021-03-24T22:03:02.455946+0000","last_deep_scrub":"86'3","last_deep_scrub_stamp":"2021-03-24T21:52:56.678111+0000","last_clean_scrub_stamp":"2021-03-24T22:03:02.455946+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"555","reported_epoch":"620","state":"active+clean","last_fresh":"2021-03-24T22:03:10.325938+0000","last_change":"2021-03-24T22:00:59.303976+0000","last_active":"2021-03-24T22:03:10.325938+0000","last_peered":"2021-03-24T22:03:10.325938+0000","last_clean":"2021-03-24T22:03:10.325938+0000","last_became_active":"2021-03-24T22:00:49.302172+0000","last_became_peered":"2021-03-24T22:00:49.302172+0000","last_unstale":"2021-03-24T22:03:10.325938+0000","last_undegraded":"2021-03-24T22:03:10.325938+0000","last_fullsized":"2021-03-24T22:03:10.325938+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:59.303897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:00:59.303897+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.16","version":"616'370","reported_seq":"2186","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.476344+0000","last_change":"2021-03-24T22:03:06.351608+0000","last_active":"2021-03-24T22:03:07.476344+0000","last_peered":"2021-03-24T22:03:07.476344+0000","last_clean":"2021-03-24T22:03:07.476344+0000","last_became_active":"2021-03-24T22:02:08.322570+0000","last_became_peered":"2021-03-24T22:02:08.322570+0000","last_unstale":"2021-03-24T22:03:07.476344+0000","last_undegraded":"2021-03-24T22:03:07.476344+0000","last_fullsized":"2021-03-24T22:03:07.476344+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"606'362","last_scrub_stamp":"2021-03-24T22:02:57.478372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:02:57.478372+0000","log_size":370,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11975784,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":80,"num_read_kb":90887,"num_write":52,"num_write_kb":13373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":30734442,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"178","length":"1"},{"start":"1ae","length":"1"},{"start":"1c4","length":"1"}]},{"pgid":"4.14","version":"544'24","reported_seq":"802","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.331538+0000","last_change":"2021-03-24T22:01:59.424897+0000","last_active":"2021-03-24T22:03:07.331538+0000","last_peered":"2021-03-24T22:03:07.331538+0000","last_clean":"2021-03-24T22:03:07.331538+0000","last_became_active":"2021-03-24T22:01:57.324895+0000","last_became_peered":"2021-03-24T22:01:57.324895+0000","last_unstale":"2021-03-24T22:03:07.331538+0000","last_undegraded":"2021-03-24T22:03:07.331538+0000","last_fullsized":"2021-03-24T22:03:07.331538+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'14","last_scrub_stamp":"2021-03-24T21:58:57.435358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:58:57.435358+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":42,"num_read_kb":29,"num_write":24,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":3744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":"82","reported_epoch":"618","state":"active+clean","last_fresh":"2021-03-24T22:03:08.353468+0000","last_change":"2021-03-24T22:01:57.320074+0000","last_active":"2021-03-24T22:03:08.353468+0000","last_peered":"2021-03-24T22:03:08.353468+0000","last_clean":"2021-03-24T22:03:08.353468+0000","last_became_active":"2021-03-24T22:01:57.318685+0000","last_became_peered":"2021-03-24T22:01:57.318685+0000","last_unstale":"2021-03-24T22:03:08.353468+0000","last_undegraded":"2021-03-24T22:03:08.353468+0000","last_fullsized":"2021-03-24T22:03:08.353468+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:15.949538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:59:15.949538+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"508'8","reported_seq":"624","reported_epoch":"619","state":"active+clean","last_fresh":"2021-03-24T22:03:09.343538+0000","last_change":"2021-03-24T22:01:16.331383+0000","last_active":"2021-03-24T22:03:09.343538+0000","last_peered":"2021-03-24T22:03:09.343538+0000","last_clean":"2021-03-24T22:03:09.343538+0000","last_became_active":"2021-03-24T22:01:16.313639+0000","last_became_peered":"2021-03-24T22:01:16.313639+0000","last_unstale":"2021-03-24T22:03:09.343538+0000","last_undegraded":"2021-03-24T22:03:09.343538+0000","last_fullsized":"2021-03-24T22:03:09.343538+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"445'6","last_scrub_stamp":"2021-03-24T22:00:24.314895+0000","last_deep_scrub":"86'3","last_deep_scrub_stamp":"2021-03-24T21:53:08.815163+0000","last_clean_scrub_stamp":"2021-03-24T22:00:24.314895+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"617","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.476170+0000","last_change":"2021-03-24T22:01:57.317084+0000","last_active":"2021-03-24T22:03:07.476170+0000","last_peered":"2021-03-24T22:03:07.476170+0000","last_clean":"2021-03-24T22:03:07.476170+0000","last_became_active":"2021-03-24T22:01:57.316729+0000","last_became_peered":"2021-03-24T22:01:57.316729+0000","last_unstale":"2021-03-24T22:03:07.476170+0000","last_undegraded":"2021-03-24T22:03:07.476170+0000","last_fullsized":"2021-03-24T22:03:07.476170+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:58.768675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:57:58.768675+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.17","version":"619'897","reported_seq":"2864","reported_epoch":"619","state":"active+clean","last_fresh":"2021-03-24T22:03:09.361897+0000","last_change":"2021-03-24T22:03:09.362450+0000","last_active":"2021-03-24T22:03:09.361897+0000","last_peered":"2021-03-24T22:03:09.361897+0000","last_clean":"2021-03-24T22:03:09.361897+0000","last_became_active":"2021-03-24T22:02:09.319642+0000","last_became_peered":"2021-03-24T22:02:09.319642+0000","last_unstale":"2021-03-24T22:03:09.361897+0000","last_undegraded":"2021-03-24T22:03:09.361897+0000","last_fullsized":"2021-03-24T22:03:09.361897+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"452'567","last_scrub_stamp":"2021-03-24T22:00:27.169710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:00:27.169710+0000","log_size":897,"ondisk_log_size":897,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33796041,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":253,"num_read_kb":279321,"num_write":152,"num_write_kb":55286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":242,"num_bytes_recovered":147823286,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"139","length":"1"},{"start":"178","length":"1"},{"start":"1ae","length":"1"},{"start":"1c4","length":"1"}]},{"pgid":"4.13","version":"584'32","reported_seq":"143","reported_epoch":"619","state":"active+clean","last_fresh":"2021-03-24T22:03:09.339777+0000","last_change":"2021-03-24T22:02:34.813717+0000","last_active":"2021-03-24T22:03:09.339777+0000","last_peered":"2021-03-24T22:03:09.339777+0000","last_clean":"2021-03-24T22:03:09.339777+0000","last_became_active":"2021-03-24T22:01:43.393580+0000","last_became_peered":"2021-03-24T22:01:43.393580+0000","last_unstale":"2021-03-24T22:03:09.339777+0000","last_undegraded":"2021-03-24T22:03:09.339777+0000","last_fullsized":"2021-03-24T22:03:09.339777+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"584'32","last_scrub_stamp":"2021-03-24T22:02:34.813628+0000","last_deep_scrub":"584'32","last_deep_scrub_stamp":"2021-03-24T22:02:34.813628+0000","last_clean_scrub_stamp":"2021-03-24T22:02:34.813628+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":29,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":2208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":"82","reported_epoch":"618","state":"active+clean","last_fresh":"2021-03-24T22:03:08.355670+0000","last_change":"2021-03-24T22:01:57.307496+0000","last_active":"2021-03-24T22:03:08.355670+0000","last_peered":"2021-03-24T22:03:08.355670+0000","last_clean":"2021-03-24T22:03:08.355670+0000","last_became_active":"2021-03-24T22:01:57.307018+0000","last_became_peered":"2021-03-24T22:01:57.307018+0000","last_unstale":"2021-03-24T22:03:08.355670+0000","last_undegraded":"2021-03-24T22:03:08.355670+0000","last_fullsized":"2021-03-24T22:03:08.355670+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:29.716186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:58:29.716186+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"164","reported_epoch":"618","state":"active+clean","last_fresh":"2021-03-24T22:03:08.330575+0000","last_change":"2021-03-24T22:02:27.315123+0000","last_active":"2021-03-24T22:03:08.330575+0000","last_peered":"2021-03-24T22:03:08.330575+0000","last_clean":"2021-03-24T22:03:08.330575+0000","last_became_active":"2021-03-24T22:02:27.314405+0000","last_became_peered":"2021-03-24T22:02:27.314405+0000","last_unstale":"2021-03-24T22:03:08.330575+0000","last_undegraded":"2021-03-24T22:03:08.330575+0000","last_fullsized":"2021-03-24T22:03:08.330575+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:01:55.618238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:57:31.118050+0000","last_clean_scrub_stamp":"2021-03-24T22:01:55.618238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"587","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.332276+0000","last_change":"2021-03-24T22:00:21.298498+0000","last_active":"2021-03-24T22:03:07.332276+0000","last_peered":"2021-03-24T22:03:07.332276+0000","last_clean":"2021-03-24T22:03:07.332276+0000","last_became_active":"2021-03-24T22:00:03.282227+0000","last_became_peered":"2021-03-24T22:00:03.282227+0000","last_unstale":"2021-03-24T22:03:07.332276+0000","last_undegraded":"2021-03-24T22:03:07.332276+0000","last_fullsized":"2021-03-24T22:03:07.332276+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:21.298330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:23.707914+0000","last_clean_scrub_stamp":"2021-03-24T22:00:21.298330+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.10","version":"616'114","reported_seq":"200","reported_epoch":"618","state":"active+clean","last_fresh":"2021-03-24T22:03:08.327559+0000","last_change":"2021-03-24T22:03:06.348784+0000","last_active":"2021-03-24T22:03:08.327559+0000","last_peered":"2021-03-24T22:03:08.327559+0000","last_clean":"2021-03-24T22:03:08.327559+0000","last_became_active":"2021-03-24T22:02:27.314608+0000","last_became_peered":"2021-03-24T22:02:27.314608+0000","last_unstale":"2021-03-24T22:03:08.327559+0000","last_undegraded":"2021-03-24T22:03:08.327559+0000","last_fullsized":"2021-03-24T22:03:08.327559+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":5,"last_scrub":"579'81","last_scrub_stamp":"2021-03-24T22:02:30.763305+0000","last_deep_scrub":"277'24","last_deep_scrub_stamp":"2021-03-24T21:57:28.119420+0000","last_clean_scrub_stamp":"2021-03-24T22:02:30.763305+0000","log_size":114,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8314122,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":57,"num_read_kb":50980,"num_write":37,"num_write_kb":11772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":15290726,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"178","length":"1"},{"start":"1ae","length":"1"},{"start":"1c4","length":"1"}]},{"pgid":"4.12","version":"544'21","reported_seq":"190","reported_epoch":"618","state":"active+clean","last_fresh":"2021-03-24T22:03:08.331322+0000","last_change":"2021-03-24T22:02:18.706930+0000","last_active":"2021-03-24T22:03:08.331322+0000","last_peered":"2021-03-24T22:03:08.331322+0000","last_clean":"2021-03-24T22:03:08.331322+0000","last_became_active":"2021-03-24T22:01:43.401402+0000","last_became_peered":"2021-03-24T22:01:43.401402+0000","last_unstale":"2021-03-24T22:03:08.331322+0000","last_undegraded":"2021-03-24T22:03:08.331322+0000","last_fullsized":"2021-03-24T22:03:08.331322+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'21","last_scrub_stamp":"2021-03-24T22:02:18.706842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:02:18.706842+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":26,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":"82","reported_epoch":"618","state":"active+clean","last_fresh":"2021-03-24T22:03:08.354365+0000","last_change":"2021-03-24T22:01:57.320413+0000","last_active":"2021-03-24T22:03:08.354365+0000","last_peered":"2021-03-24T22:03:08.354365+0000","last_clean":"2021-03-24T22:03:08.354365+0000","last_became_active":"2021-03-24T22:01:57.318513+0000","last_became_peered":"2021-03-24T22:01:57.318513+0000","last_unstale":"2021-03-24T22:03:08.354365+0000","last_undegraded":"2021-03-24T22:03:08.354365+0000","last_fullsized":"2021-03-24T22:03:08.354365+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:04.965369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:58:04.965369+0000","last_clean_scrub_stamp":"2021-03-24T21:58:04.965369+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"607","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.326918+0000","last_change":"2021-03-24T22:01:16.302201+0000","last_active":"2021-03-24T22:03:07.326918+0000","last_peered":"2021-03-24T22:03:07.326918+0000","last_clean":"2021-03-24T22:03:07.326918+0000","last_became_active":"2021-03-24T22:01:16.300935+0000","last_became_peered":"2021-03-24T22:01:16.300935+0000","last_unstale":"2021-03-24T22:03:07.326918+0000","last_undegraded":"2021-03-24T22:03:07.326918+0000","last_fullsized":"2021-03-24T22:03:07.326918+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:28.289244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:00:28.289244+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"610","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.475288+0000","last_change":"2021-03-24T22:01:52.461763+0000","last_active":"2021-03-24T22:03:07.475288+0000","last_peered":"2021-03-24T22:03:07.475288+0000","last_clean":"2021-03-24T22:03:07.475288+0000","last_became_active":"2021-03-24T22:01:16.336700+0000","last_became_peered":"2021-03-24T22:01:16.336700+0000","last_unstale":"2021-03-24T22:03:07.475288+0000","last_undegraded":"2021-03-24T22:03:07.475288+0000","last_fullsized":"2021-03-24T22:03:07.475288+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:01:52.461594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:01:52.461594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.11","version":"619'822","reported_seq":"2781","reported_epoch":"619","state":"active+clean","last_fresh":"2021-03-24T22:03:09.354282+0000","last_change":"2021-03-24T22:03:09.354605+0000","last_active":"2021-03-24T22:03:09.354282+0000","last_peered":"2021-03-24T22:03:09.354282+0000","last_clean":"2021-03-24T22:03:09.354282+0000","last_became_active":"2021-03-24T22:02:04.322694+0000","last_became_peered":"2021-03-24T22:02:04.322694+0000","last_unstale":"2021-03-24T22:03:09.354282+0000","last_undegraded":"2021-03-24T22:03:09.354282+0000","last_fullsized":"2021-03-24T22:03:09.354282+0000","mapping_epoch":553,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":554,"parent":"0.0","parent_split_bits":5,"last_scrub":"381'425","last_scrub_stamp":"2021-03-24T21:59:11.936662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:59:11.936662+0000","log_size":822,"ondisk_log_size":822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26839998,"num_objects":13,"num_object_clones":10,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":155,"num_read_kb":179195,"num_write":117,"num_write_kb":45267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":325,"num_bytes_recovered":148467814,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"139","length":"1"},{"start":"178","length":"1"},{"start":"1ae","length":"1"},{"start":"1c4","length":"1"}]},{"pgid":"4.11","version":"544'30","reported_seq":"800","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.330552+0000","last_change":"2021-03-24T22:01:58.012725+0000","last_active":"2021-03-24T22:03:07.330552+0000","last_peered":"2021-03-24T22:03:07.330552+0000","last_clean":"2021-03-24T22:03:07.330552+0000","last_became_active":"2021-03-24T22:01:57.310370+0000","last_became_peered":"2021-03-24T22:01:57.310370+0000","last_unstale":"2021-03-24T22:03:07.330552+0000","last_undegraded":"2021-03-24T22:03:07.330552+0000","last_fullsized":"2021-03-24T22:03:07.330552+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'22","last_scrub_stamp":"2021-03-24T22:00:13.347155+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2021-03-24T21:52:54.597505+0000","last_clean_scrub_stamp":"2021-03-24T22:00:13.347155+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":41,"num_read_kb":28,"num_write":23,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":3480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":"516","reported_epoch":"619","state":"active+clean","last_fresh":"2021-03-24T22:03:09.343362+0000","last_change":"2021-03-24T22:02:18.082676+0000","last_active":"2021-03-24T22:03:09.343362+0000","last_peered":"2021-03-24T22:03:09.343362+0000","last_clean":"2021-03-24T22:03:09.343362+0000","last_became_active":"2021-03-24T22:01:57.327451+0000","last_became_peered":"2021-03-24T22:01:57.327451+0000","last_unstale":"2021-03-24T22:03:09.343362+0000","last_undegraded":"2021-03-24T22:03:09.343362+0000","last_fullsized":"2021-03-24T22:03:09.343362+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:18.082513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:02:18.082513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"82","reported_epoch":"618","state":"active+clean","last_fresh":"2021-03-24T22:03:08.352026+0000","last_change":"2021-03-24T22:01:57.308893+0000","last_active":"2021-03-24T22:03:08.352026+0000","last_peered":"2021-03-24T22:03:08.352026+0000","last_clean":"2021-03-24T22:03:08.352026+0000","last_became_active":"2021-03-24T22:01:57.307995+0000","last_became_peered":"2021-03-24T22:01:57.307995+0000","last_unstale":"2021-03-24T22:03:08.352026+0000","last_undegraded":"2021-03-24T22:03:08.352026+0000","last_fullsized":"2021-03-24T22:03:08.352026+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:41.839852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:59:41.839852+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"590","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.327458+0000","last_change":"2021-03-24T22:01:43.389560+0000","last_active":"2021-03-24T22:03:07.327458+0000","last_peered":"2021-03-24T22:03:07.327458+0000","last_clean":"2021-03-24T22:03:07.327458+0000","last_became_active":"2021-03-24T22:01:43.389017+0000","last_became_peered":"2021-03-24T22:01:43.389017+0000","last_unstale":"2021-03-24T22:03:07.327458+0000","last_undegraded":"2021-03-24T22:03:07.327458+0000","last_fullsized":"2021-03-24T22:03:07.327458+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:15.289444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:47.540793+0000","last_clean_scrub_stamp":"2021-03-24T21:59:15.289444+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.12","version":"183'26","reported_seq":"288","reported_epoch":"620","state":"active+clean","last_fresh":"2021-03-24T22:03:10.328769+0000","last_change":"2021-03-24T22:03:10.329576+0000","last_active":"2021-03-24T22:03:10.328769+0000","last_peered":"2021-03-24T22:03:10.328769+0000","last_clean":"2021-03-24T22:03:10.328769+0000","last_became_active":"2021-03-24T22:01:43.392614+0000","last_became_peered":"2021-03-24T22:01:43.392614+0000","last_unstale":"2021-03-24T22:03:10.328769+0000","last_undegraded":"2021-03-24T22:03:10.328769+0000","last_fullsized":"2021-03-24T22:03:10.328769+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":5,"last_scrub":"183'26","last_scrub_stamp":"2021-03-24T22:02:21.766090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:02:21.766090+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"139","length":"1"},{"start":"178","length":"1"},{"start":"1ae","length":"1"},{"start":"1eb","length":"1"}]},{"pgid":"4.10","version":"544'16","reported_seq":"812","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.326469+0000","last_change":"2021-03-24T22:03:07.216177+0000","last_active":"2021-03-24T22:03:07.326469+0000","last_peered":"2021-03-24T22:03:07.326469+0000","last_clean":"2021-03-24T22:03:07.326469+0000","last_became_active":"2021-03-24T22:03:06.345705+0000","last_became_peered":"2021-03-24T22:03:06.345705+0000","last_unstale":"2021-03-24T22:03:07.326469+0000","last_undegraded":"2021-03-24T22:03:07.326469+0000","last_fullsized":"2021-03-24T22:03:07.326469+0000","mapping_epoch":615,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":616,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2021-03-24T22:00:29.255353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:00:29.255353+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":36,"num_read_kb":18,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":7512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":"598","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.475320+0000","last_change":"2021-03-24T22:01:56.417737+0000","last_active":"2021-03-24T22:03:07.475320+0000","last_peered":"2021-03-24T22:03:07.475320+0000","last_clean":"2021-03-24T22:03:07.475320+0000","last_became_active":"2021-03-24T22:00:03.301753+0000","last_became_peered":"2021-03-24T22:00:03.301753+0000","last_unstale":"2021-03-24T22:03:07.475320+0000","last_undegraded":"2021-03-24T22:03:07.475320+0000","last_fullsized":"2021-03-24T22:03:07.475320+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:01:56.417662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:01:56.417662+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"113","reported_epoch":"619","state":"active+clean","last_fresh":"2021-03-24T22:03:09.345538+0000","last_change":"2021-03-24T22:02:27.831482+0000","last_active":"2021-03-24T22:03:09.345538+0000","last_peered":"2021-03-24T22:03:09.345538+0000","last_clean":"2021-03-24T22:03:09.345538+0000","last_became_active":"2021-03-24T22:01:57.304385+0000","last_became_peered":"2021-03-24T22:01:57.304385+0000","last_unstale":"2021-03-24T22:03:09.345538+0000","last_undegraded":"2021-03-24T22:03:09.345538+0000","last_fullsized":"2021-03-24T22:03:09.345538+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:27.831383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:02:27.831383+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"89","reported_epoch":"618","state":"active+clean","last_fresh":"2021-03-24T22:03:08.379782+0000","last_change":"2021-03-24T22:03:08.379727+0000","last_active":"2021-03-24T22:03:08.379782+0000","last_peered":"2021-03-24T22:03:08.379782+0000","last_clean":"2021-03-24T22:03:08.379782+0000","last_became_active":"2021-03-24T22:01:57.315718+0000","last_became_peered":"2021-03-24T22:01:57.315718+0000","last_unstale":"2021-03-24T22:03:08.379782+0000","last_undegraded":"2021-03-24T22:03:08.379782+0000","last_fullsized":"2021-03-24T22:03:08.379782+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:08.379591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:57:59.979772+0000","last_clean_scrub_stamp":"2021-03-24T22:03:08.379591+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.13","version":"611'526","reported_seq":"2623","reported_epoch":"618","state":"active+clean","last_fresh":"2021-03-24T22:03:08.343159+0000","last_change":"2021-03-24T22:03:06.361393+0000","last_active":"2021-03-24T22:03:08.343159+0000","last_peered":"2021-03-24T22:03:08.343159+0000","last_clean":"2021-03-24T22:03:08.343159+0000","last_became_active":"2021-03-24T22:02:28.318669+0000","last_became_peered":"2021-03-24T22:02:28.318669+0000","last_unstale":"2021-03-24T22:03:08.343159+0000","last_undegraded":"2021-03-24T22:03:08.343159+0000","last_fullsized":"2021-03-24T22:03:08.343159+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":526,"ondisk_log_size":526,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19209015,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":201,"num_read_kb":190473,"num_write":172,"num_write_kb":68786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":204,"num_bytes_recovered":118383596,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"178","length":"1"},{"start":"1ae","length":"1"},{"start":"1c4","length":"1"}]},{"pgid":"3.18","version":"0'0","reported_seq":"596","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.326722+0000","last_change":"2021-03-24T22:01:16.301917+0000","last_active":"2021-03-24T22:03:07.326722+0000","last_peered":"2021-03-24T22:03:07.326722+0000","last_clean":"2021-03-24T22:03:07.326722+0000","last_became_active":"2021-03-24T22:01:16.301235+0000","last_became_peered":"2021-03-24T22:01:16.301235+0000","last_unstale":"2021-03-24T22:03:07.326722+0000","last_undegraded":"2021-03-24T22:03:07.326722+0000","last_fullsized":"2021-03-24T22:03:07.326722+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:00.393742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:59:00.393742+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"637","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.477408+0000","last_change":"2021-03-24T22:02:22.328403+0000","last_active":"2021-03-24T22:03:07.477408+0000","last_peered":"2021-03-24T22:03:07.477408+0000","last_clean":"2021-03-24T22:03:07.477408+0000","last_became_active":"2021-03-24T22:02:22.328021+0000","last_became_peered":"2021-03-24T22:02:22.328021+0000","last_unstale":"2021-03-24T22:03:07.477408+0000","last_undegraded":"2021-03-24T22:03:07.477408+0000","last_fullsized":"2021-03-24T22:03:07.477408+0000","mapping_epoch":571,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:17.040623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:59:17.040623+0000","last_clean_scrub_stamp":"2021-03-24T21:59:17.040623+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"128","reported_epoch":"618","state":"active+clean","last_fresh":"2021-03-24T22:03:08.333955+0000","last_change":"2021-03-24T22:03:06.420940+0000","last_active":"2021-03-24T22:03:08.333955+0000","last_peered":"2021-03-24T22:03:08.333955+0000","last_clean":"2021-03-24T22:03:08.333955+0000","last_became_active":"2021-03-24T22:01:16.335096+0000","last_became_peered":"2021-03-24T22:01:16.335096+0000","last_unstale":"2021-03-24T22:03:08.333955+0000","last_undegraded":"2021-03-24T22:03:08.333955+0000","last_fullsized":"2021-03-24T22:03:08.333955+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:06.420794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:03:06.420794+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1c","version":"273'245","reported_seq":"1893","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.327700+0000","last_change":"2021-03-24T22:03:06.378524+0000","last_active":"2021-03-24T22:03:07.327700+0000","last_peered":"2021-03-24T22:03:07.327700+0000","last_clean":"2021-03-24T22:03:07.327700+0000","last_became_active":"2021-03-24T22:01:19.284869+0000","last_became_peered":"2021-03-24T22:01:19.284869+0000","last_unstale":"2021-03-24T22:03:07.327700+0000","last_undegraded":"2021-03-24T22:03:07.327700+0000","last_fullsized":"2021-03-24T22:03:07.327700+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":245,"ondisk_log_size":245,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8581786,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":70,"num_read_kb":67597,"num_write":52,"num_write_kb":18055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"178","length":"1"},{"start":"1ae","length":"1"},{"start":"1c4","length":"1"}]},{"pgid":"3.19","version":"0'0","reported_seq":"128","reported_epoch":"618","state":"active+clean","last_fresh":"2021-03-24T22:03:08.328560+0000","last_change":"2021-03-24T22:01:40.788970+0000","last_active":"2021-03-24T22:03:08.328560+0000","last_peered":"2021-03-24T22:03:08.328560+0000","last_clean":"2021-03-24T22:03:08.328560+0000","last_became_active":"2021-03-24T22:01:16.298021+0000","last_became_peered":"2021-03-24T22:01:16.298021+0000","last_unstale":"2021-03-24T22:03:08.328560+0000","last_undegraded":"2021-03-24T22:03:08.328560+0000","last_fullsized":"2021-03-24T22:03:08.328560+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:01:40.788778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:57:26.417613+0000","last_clean_scrub_stamp":"2021-03-24T22:01:40.788778+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"613'8","reported_seq":"137","reported_epoch":"619","state":"active+clean","last_fresh":"2021-03-24T22:03:09.339897+0000","last_change":"2021-03-24T22:03:07.709341+0000","last_active":"2021-03-24T22:03:09.339897+0000","last_peered":"2021-03-24T22:03:09.339897+0000","last_clean":"2021-03-24T22:03:09.339897+0000","last_became_active":"2021-03-24T22:03:03.353426+0000","last_became_peered":"2021-03-24T22:03:03.353426+0000","last_unstale":"2021-03-24T22:03:09.339897+0000","last_undegraded":"2021-03-24T22:03:09.339897+0000","last_fullsized":"2021-03-24T22:03:09.339897+0000","mapping_epoch":612,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":613,"parent":"0.0","parent_split_bits":0,"last_scrub":"613'8","last_scrub_stamp":"2021-03-24T22:03:07.709086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:03:07.709086+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"382","reported_epoch":"620","state":"active+clean","last_fresh":"2021-03-24T22:03:10.324208+0000","last_change":"2021-03-24T22:01:57.307527+0000","last_active":"2021-03-24T22:03:10.324208+0000","last_peered":"2021-03-24T22:03:10.324208+0000","last_clean":"2021-03-24T22:03:10.324208+0000","last_became_active":"2021-03-24T22:01:57.307196+0000","last_became_peered":"2021-03-24T22:01:57.307196+0000","last_unstale":"2021-03-24T22:03:10.324208+0000","last_undegraded":"2021-03-24T22:03:10.324208+0000","last_fullsized":"2021-03-24T22:03:10.324208+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:01:53.591151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:01:53.591151+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1d","version":"619'538","reported_seq":"2438","reported_epoch":"619","state":"active+clean","last_fresh":"2021-03-24T22:03:09.339010+0000","last_change":"2021-03-24T22:03:09.339868+0000","last_active":"2021-03-24T22:03:09.339010+0000","last_peered":"2021-03-24T22:03:09.339010+0000","last_clean":"2021-03-24T22:03:09.339010+0000","last_became_active":"2021-03-24T22:01:46.322605+0000","last_became_peered":"2021-03-24T22:01:46.322605+0000","last_unstale":"2021-03-24T22:03:09.339010+0000","last_undegraded":"2021-03-24T22:03:09.339010+0000","last_fullsized":"2021-03-24T22:03:09.339010+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":5,"last_scrub":"233'169","last_scrub_stamp":"2021-03-24T21:56:47.285648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:56:47.285648+0000","log_size":538,"ondisk_log_size":538,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28874373,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":186,"num_read_kb":205496,"num_write":137,"num_write_kb":66939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":139,"num_bytes_recovered":72061573,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"139","length":"1"},{"start":"178","length":"1"},{"start":"1ae","length":"1"},{"start":"1c4","length":"1"}]},{"pgid":"3.1a","version":"0'0","reported_seq":"123","reported_epoch":"618","state":"active+clean","last_fresh":"2021-03-24T22:03:08.330911+0000","last_change":"2021-03-24T22:01:16.335123+0000","last_active":"2021-03-24T22:03:08.330911+0000","last_peered":"2021-03-24T22:03:08.330911+0000","last_clean":"2021-03-24T22:03:08.330911+0000","last_became_active":"2021-03-24T22:01:16.334794+0000","last_became_peered":"2021-03-24T22:01:16.334794+0000","last_unstale":"2021-03-24T22:03:08.330911+0000","last_undegraded":"2021-03-24T22:03:08.330911+0000","last_fullsized":"2021-03-24T22:03:08.330911+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:12.679659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:52:54.212459+0000","last_clean_scrub_stamp":"2021-03-24T21:59:12.679659+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"156","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.331081+0000","last_change":"2021-03-24T22:02:17.672555+0000","last_active":"2021-03-24T22:03:07.331081+0000","last_peered":"2021-03-24T22:03:07.331081+0000","last_clean":"2021-03-24T22:03:07.331081+0000","last_became_active":"2021-03-24T22:00:49.305569+0000","last_became_peered":"2021-03-24T22:00:49.305569+0000","last_unstale":"2021-03-24T22:03:07.331081+0000","last_undegraded":"2021-03-24T22:03:07.331081+0000","last_fullsized":"2021-03-24T22:03:07.331081+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:17.672325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:02:17.672325+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"623","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.329316+0000","last_change":"2021-03-24T22:01:43.383380+0000","last_active":"2021-03-24T22:03:07.329316+0000","last_peered":"2021-03-24T22:03:07.329316+0000","last_clean":"2021-03-24T22:03:07.329316+0000","last_became_active":"2021-03-24T22:01:43.383075+0000","last_became_peered":"2021-03-24T22:01:43.383075+0000","last_unstale":"2021-03-24T22:03:07.329316+0000","last_undegraded":"2021-03-24T22:03:07.329316+0000","last_fullsized":"2021-03-24T22:03:07.329316+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:55.309332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:00:55.309332+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1e","version":"265'45","reported_seq":"341","reported_epoch":"619","state":"active+clean","last_fresh":"2021-03-24T22:03:09.333002+0000","last_change":"2021-03-24T22:03:09.333743+0000","last_active":"2021-03-24T22:03:09.333002+0000","last_peered":"2021-03-24T22:03:09.333002+0000","last_clean":"2021-03-24T22:03:09.333002+0000","last_became_active":"2021-03-24T22:01:16.321204+0000","last_became_peered":"2021-03-24T22:01:16.321204+0000","last_unstale":"2021-03-24T22:03:09.333002+0000","last_undegraded":"2021-03-24T22:03:09.333002+0000","last_fullsized":"2021-03-24T22:03:09.333002+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":5,"last_scrub":"199'30","last_scrub_stamp":"2021-03-24T21:56:10.105084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:56:10.105084+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":18305,"num_write":13,"num_write_kb":3060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"139","length":"1"},{"start":"178","length":"1"},{"start":"1ae","length":"1"},{"start":"1c4","length":"1"}]},{"pgid":"3.1b","version":"0'0","reported_seq":"628","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.476158+0000","last_change":"2021-03-24T22:03:06.348579+0000","last_active":"2021-03-24T22:03:07.476158+0000","last_peered":"2021-03-24T22:03:07.476158+0000","last_clean":"2021-03-24T22:03:07.476158+0000","last_became_active":"2021-03-24T22:03:06.348172+0000","last_became_peered":"2021-03-24T22:03:06.348172+0000","last_unstale":"2021-03-24T22:03:07.476158+0000","last_undegraded":"2021-03-24T22:03:07.476158+0000","last_fullsized":"2021-03-24T22:03:07.476158+0000","mapping_epoch":615,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":616,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:54.682649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:00:54.682649+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"130","reported_epoch":"618","state":"active+clean","last_fresh":"2021-03-24T22:03:08.332479+0000","last_change":"2021-03-24T22:01:51.757918+0000","last_active":"2021-03-24T22:03:08.332479+0000","last_peered":"2021-03-24T22:03:08.332479+0000","last_clean":"2021-03-24T22:03:08.332479+0000","last_became_active":"2021-03-24T22:01:16.334651+0000","last_became_peered":"2021-03-24T22:01:16.334651+0000","last_unstale":"2021-03-24T22:03:08.332479+0000","last_undegraded":"2021-03-24T22:03:08.332479+0000","last_fullsized":"2021-03-24T22:03:08.332479+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:01:51.757809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:01:51.757809+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"97","reported_epoch":"619","state":"active+clean","last_fresh":"2021-03-24T22:03:09.339071+0000","last_change":"2021-03-24T22:01:43.384457+0000","last_active":"2021-03-24T22:03:09.339071+0000","last_peered":"2021-03-24T22:03:09.339071+0000","last_clean":"2021-03-24T22:03:09.339071+0000","last_became_active":"2021-03-24T22:01:43.384083+0000","last_became_peered":"2021-03-24T22:01:43.384083+0000","last_unstale":"2021-03-24T22:03:09.339071+0000","last_undegraded":"2021-03-24T22:03:09.339071+0000","last_fullsized":"2021-03-24T22:03:09.339071+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:26.883399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:00:26.883399+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1f","version":"272'99","reported_seq":"283","reported_epoch":"620","state":"active+clean","last_fresh":"2021-03-24T22:03:10.342668+0000","last_change":"2021-03-24T22:03:10.343088+0000","last_active":"2021-03-24T22:03:10.342668+0000","last_peered":"2021-03-24T22:03:10.342668+0000","last_clean":"2021-03-24T22:03:10.342668+0000","last_became_active":"2021-03-24T22:01:43.385324+0000","last_became_peered":"2021-03-24T22:01:43.385324+0000","last_unstale":"2021-03-24T22:03:10.342668+0000","last_undegraded":"2021-03-24T22:03:10.342668+0000","last_fullsized":"2021-03-24T22:03:10.342668+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":5,"last_scrub":"218'73","last_scrub_stamp":"2021-03-24T21:56:48.267711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:56:48.267711+0000","log_size":99,"ondisk_log_size":99,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3571504,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":43,"num_read_kb":44484,"num_write":22,"num_write_kb":9163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":942483,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"139","length":"1"},{"start":"178","length":"1"},{"start":"1ae","length":"1"},{"start":"1eb","length":"1"}]},{"pgid":"3.1c","version":"0'0","reported_seq":"123","reported_epoch":"618","state":"active+clean","last_fresh":"2021-03-24T22:03:08.332180+0000","last_change":"2021-03-24T22:01:16.334949+0000","last_active":"2021-03-24T22:03:08.332180+0000","last_peered":"2021-03-24T22:03:08.332180+0000","last_clean":"2021-03-24T22:03:08.332180+0000","last_became_active":"2021-03-24T22:01:16.334631+0000","last_became_peered":"2021-03-24T22:01:16.334631+0000","last_unstale":"2021-03-24T22:03:08.332180+0000","last_undegraded":"2021-03-24T22:03:08.332180+0000","last_fullsized":"2021-03-24T22:03:08.332180+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:29.855897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:00:29.855897+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"82","reported_epoch":"618","state":"active+clean","last_fresh":"2021-03-24T22:03:08.352802+0000","last_change":"2021-03-24T22:01:57.322203+0000","last_active":"2021-03-24T22:03:08.352802+0000","last_peered":"2021-03-24T22:03:08.352802+0000","last_clean":"2021-03-24T22:03:08.352802+0000","last_became_active":"2021-03-24T22:01:57.321935+0000","last_became_peered":"2021-03-24T22:01:57.321935+0000","last_unstale":"2021-03-24T22:03:08.352802+0000","last_undegraded":"2021-03-24T22:03:08.352802+0000","last_fullsized":"2021-03-24T22:03:08.352802+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:08.942247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:56:51.977306+0000","last_clean_scrub_stamp":"2021-03-24T21:59:08.942247+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"489","reported_epoch":"619","state":"active+clean","last_fresh":"2021-03-24T22:03:09.342471+0000","last_change":"2021-03-24T22:01:41.968251+0000","last_active":"2021-03-24T22:03:09.342471+0000","last_peered":"2021-03-24T22:03:09.342471+0000","last_clean":"2021-03-24T22:03:09.342471+0000","last_became_active":"2021-03-24T22:01:16.312575+0000","last_became_peered":"2021-03-24T22:01:16.312575+0000","last_unstale":"2021-03-24T22:03:09.342471+0000","last_undegraded":"2021-03-24T22:03:09.342471+0000","last_fullsized":"2021-03-24T22:03:09.342471+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:01:41.968063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:01:41.968063+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.18","version":"600'189","reported_seq":"1582","reported_epoch":"618","state":"active+clean","last_fresh":"2021-03-24T22:03:08.341629+0000","last_change":"2021-03-24T22:03:06.353727+0000","last_active":"2021-03-24T22:03:08.341629+0000","last_peered":"2021-03-24T22:03:08.341629+0000","last_clean":"2021-03-24T22:03:08.341629+0000","last_became_active":"2021-03-24T22:02:00.304128+0000","last_became_peered":"2021-03-24T22:02:00.304128+0000","last_unstale":"2021-03-24T22:03:08.341629+0000","last_undegraded":"2021-03-24T22:03:08.341629+0000","last_fullsized":"2021-03-24T22:03:08.341629+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":5,"last_scrub":"249'90","last_scrub_stamp":"2021-03-24T21:57:56.848784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:57:56.848784+0000","log_size":189,"ondisk_log_size":189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11513290,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":42,"num_read_kb":38347,"num_write":26,"num_write_kb":9218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":32240200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"178","length":"1"},{"start":"1ae","length":"1"},{"start":"1c4","length":"1"}]},{"pgid":"3.1d","version":"0'0","reported_seq":"650","reported_epoch":"620","state":"active+clean","last_fresh":"2021-03-24T22:03:10.326285+0000","last_change":"2021-03-24T22:02:19.684802+0000","last_active":"2021-03-24T22:03:10.326285+0000","last_peered":"2021-03-24T22:03:10.326285+0000","last_clean":"2021-03-24T22:03:10.326285+0000","last_became_active":"2021-03-24T22:01:43.375494+0000","last_became_peered":"2021-03-24T22:01:43.375494+0000","last_unstale":"2021-03-24T22:03:10.326285+0000","last_undegraded":"2021-03-24T22:03:10.326285+0000","last_fullsized":"2021-03-24T22:03:10.326285+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:19.684726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:02:19.684726+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"503","reported_epoch":"619","state":"active+clean","last_fresh":"2021-03-24T22:03:09.342672+0000","last_change":"2021-03-24T22:01:57.323397+0000","last_active":"2021-03-24T22:03:09.342672+0000","last_peered":"2021-03-24T22:03:09.342672+0000","last_clean":"2021-03-24T22:03:09.342672+0000","last_became_active":"2021-03-24T22:01:57.323085+0000","last_became_peered":"2021-03-24T22:01:57.323085+0000","last_unstale":"2021-03-24T22:03:09.342672+0000","last_undegraded":"2021-03-24T22:03:09.342672+0000","last_fullsized":"2021-03-24T22:03:09.342672+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:22.297227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:00:22.297227+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"631","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.476166+0000","last_change":"2021-03-24T22:02:19.458206+0000","last_active":"2021-03-24T22:03:07.476166+0000","last_peered":"2021-03-24T22:03:07.476166+0000","last_clean":"2021-03-24T22:03:07.476166+0000","last_became_active":"2021-03-24T22:01:57.315933+0000","last_became_peered":"2021-03-24T22:01:57.315933+0000","last_unstale":"2021-03-24T22:03:07.476166+0000","last_undegraded":"2021-03-24T22:03:07.476166+0000","last_fullsized":"2021-03-24T22:03:07.476166+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:19.457980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:02:19.457980+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.19","version":"619'733","reported_seq":"2751","reported_epoch":"620","state":"active+clean","last_fresh":"2021-03-24T22:03:10.329676+0000","last_change":"2021-03-24T22:03:10.330593+0000","last_active":"2021-03-24T22:03:10.329676+0000","last_peered":"2021-03-24T22:03:10.329676+0000","last_clean":"2021-03-24T22:03:10.329676+0000","last_became_active":"2021-03-24T22:02:07.304440+0000","last_became_peered":"2021-03-24T22:02:07.304440+0000","last_unstale":"2021-03-24T22:03:10.329676+0000","last_undegraded":"2021-03-24T22:03:10.329676+0000","last_fullsized":"2021-03-24T22:03:10.329676+0000","mapping_epoch":556,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":557,"parent":"0.0","parent_split_bits":5,"last_scrub":"455'480","last_scrub_stamp":"2021-03-24T22:00:26.193730+0000","last_deep_scrub":"455'480","last_deep_scrub_stamp":"2021-03-24T22:00:26.193730+0000","last_clean_scrub_stamp":"2021-03-24T22:00:26.193730+0000","log_size":733,"ondisk_log_size":733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31217018,"num_objects":13,"num_object_clones":10,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":164,"num_read_kb":145014,"num_write":137,"num_write_kb":63355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":265,"num_bytes_recovered":154749512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"139","length":"1"},{"start":"178","length":"1"},{"start":"1ae","length":"1"},{"start":"1eb","length":"1"}]},{"pgid":"3.1e","version":"0'0","reported_seq":"632","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.329009+0000","last_change":"2021-03-24T22:01:43.382796+0000","last_active":"2021-03-24T22:03:07.329009+0000","last_peered":"2021-03-24T22:03:07.329009+0000","last_clean":"2021-03-24T22:03:07.329009+0000","last_became_active":"2021-03-24T22:01:43.381824+0000","last_became_peered":"2021-03-24T22:01:43.381824+0000","last_unstale":"2021-03-24T22:03:07.329009+0000","last_undegraded":"2021-03-24T22:03:07.329009+0000","last_fullsized":"2021-03-24T22:03:07.329009+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:05.405785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:59:05.405785+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"102","reported_epoch":"619","state":"active+clean","last_fresh":"2021-03-24T22:03:09.339017+0000","last_change":"2021-03-24T22:03:01.782651+0000","last_active":"2021-03-24T22:03:09.339017+0000","last_peered":"2021-03-24T22:03:09.339017+0000","last_clean":"2021-03-24T22:03:09.339017+0000","last_became_active":"2021-03-24T22:01:43.393091+0000","last_became_peered":"2021-03-24T22:01:43.393091+0000","last_unstale":"2021-03-24T22:03:09.339017+0000","last_undegraded":"2021-03-24T22:03:09.339017+0000","last_fullsized":"2021-03-24T22:03:09.339017+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:01.782483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:03:01.782483+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"651","reported_epoch":"618","state":"active+clean","last_fresh":"2021-03-24T22:03:08.392175+0000","last_change":"2021-03-24T22:03:08.392175+0000","last_active":"2021-03-24T22:03:08.392175+0000","last_peered":"2021-03-24T22:03:08.392175+0000","last_clean":"2021-03-24T22:03:08.392175+0000","last_became_active":"2021-03-24T22:01:57.310369+0000","last_became_peered":"2021-03-24T22:01:57.310369+0000","last_unstale":"2021-03-24T22:03:08.392175+0000","last_undegraded":"2021-03-24T22:03:08.392175+0000","last_fullsized":"2021-03-24T22:03:08.392175+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:08.392060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:03:08.392060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"619'303","reported_seq":"2034","reported_epoch":"620","state":"active+clean","last_fresh":"2021-03-24T22:03:10.341959+0000","last_change":"2021-03-24T22:03:10.342557+0000","last_active":"2021-03-24T22:03:10.341959+0000","last_peered":"2021-03-24T22:03:10.341959+0000","last_clean":"2021-03-24T22:03:10.341959+0000","last_became_active":"2021-03-24T22:01:46.306324+0000","last_became_peered":"2021-03-24T22:01:46.306324+0000","last_unstale":"2021-03-24T22:03:10.341959+0000","last_undegraded":"2021-03-24T22:03:10.341959+0000","last_fullsized":"2021-03-24T22:03:10.341959+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":303,"ondisk_log_size":303,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5260822,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":76,"num_read_kb":80682,"num_write":73,"num_write_kb":25898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":37089076,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"139","length":"1"},{"start":"178","length":"1"},{"start":"1ae","length":"1"},{"start":"1eb","length":"1"}]},{"pgid":"3.1f","version":"0'0","reported_seq":"621","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.476768+0000","last_change":"2021-03-24T22:02:22.325068+0000","last_active":"2021-03-24T22:03:07.476768+0000","last_peered":"2021-03-24T22:03:07.476768+0000","last_clean":"2021-03-24T22:03:07.476768+0000","last_became_active":"2021-03-24T22:02:22.324672+0000","last_became_peered":"2021-03-24T22:02:22.324672+0000","last_unstale":"2021-03-24T22:03:07.476768+0000","last_undegraded":"2021-03-24T22:03:07.476768+0000","last_fullsized":"2021-03-24T22:03:07.476768+0000","mapping_epoch":571,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:13.624943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:00:13.624943+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"148","reported_epoch":"618","state":"active+clean","last_fresh":"2021-03-24T22:03:08.328696+0000","last_change":"2021-03-24T22:03:00.407206+0000","last_active":"2021-03-24T22:03:08.328696+0000","last_peered":"2021-03-24T22:03:08.328696+0000","last_clean":"2021-03-24T22:03:08.328696+0000","last_became_active":"2021-03-24T22:01:57.326514+0000","last_became_peered":"2021-03-24T22:01:57.326514+0000","last_unstale":"2021-03-24T22:03:08.328696+0000","last_undegraded":"2021-03-24T22:03:08.328696+0000","last_fullsized":"2021-03-24T22:03:08.328696+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:00.407009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:14.710709+0000","last_clean_scrub_stamp":"2021-03-24T22:03:00.407009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"591","reported_epoch":"619","state":"active+clean","last_fresh":"2021-03-24T22:03:09.343849+0000","last_change":"2021-03-24T22:02:35.095682+0000","last_active":"2021-03-24T22:03:09.343849+0000","last_peered":"2021-03-24T22:03:09.343849+0000","last_clean":"2021-03-24T22:03:09.343849+0000","last_became_active":"2021-03-24T22:01:16.311234+0000","last_became_peered":"2021-03-24T22:01:16.311234+0000","last_unstale":"2021-03-24T22:03:09.343849+0000","last_undegraded":"2021-03-24T22:03:09.343849+0000","last_fullsized":"2021-03-24T22:03:09.343849+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:35.095493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:02:35.095493+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1b","version":"616'1294","reported_seq":"3045","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.477418+0000","last_change":"2021-03-24T22:03:06.386324+0000","last_active":"2021-03-24T22:03:07.477418+0000","last_peered":"2021-03-24T22:03:07.477418+0000","last_clean":"2021-03-24T22:03:07.477418+0000","last_became_active":"2021-03-24T21:57:33.176110+0000","last_became_peered":"2021-03-24T21:57:33.176110+0000","last_unstale":"2021-03-24T22:03:07.477418+0000","last_undegraded":"2021-03-24T22:03:07.477418+0000","last_fullsized":"2021-03-24T22:03:07.477418+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":5,"last_scrub":"410'569","last_scrub_stamp":"2021-03-24T21:59:43.929595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:59:43.929595+0000","log_size":1294,"ondisk_log_size":1294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":71310679,"num_objects":31,"num_object_clones":25,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":1,"num_read":171,"num_read_kb":186344,"num_write":125,"num_write_kb":50423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"178","length":"1"},{"start":"1ae","length":"1"},{"start":"1c4","length":"1"}]},{"pgid":"3.24","version":"0'0","reported_seq":"244","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.327046+0000","last_change":"2021-03-24T21:59:14.369914+0000","last_active":"2021-03-24T22:03:07.327046+0000","last_peered":"2021-03-24T22:03:07.327046+0000","last_clean":"2021-03-24T22:03:07.327046+0000","last_became_active":"2021-03-24T21:59:14.369606+0000","last_became_peered":"2021-03-24T21:59:14.369606+0000","last_unstale":"2021-03-24T22:03:07.327046+0000","last_undegraded":"2021-03-24T22:03:07.327046+0000","last_fullsized":"2021-03-24T22:03:07.327046+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:38.144956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:38.144956+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"175'22","reported_seq":"1320","reported_epoch":"621","state":"active+clean","last_fresh":"2021-03-24T22:03:11.340841+0000","last_change":"2021-03-24T22:03:11.341711+0000","last_active":"2021-03-24T22:03:11.340841+0000","last_peered":"2021-03-24T22:03:11.340841+0000","last_clean":"2021-03-24T22:03:11.340841+0000","last_became_active":"2021-03-24T22:01:43.384871+0000","last_became_peered":"2021-03-24T22:01:43.384871+0000","last_unstale":"2021-03-24T22:03:11.340841+0000","last_undegraded":"2021-03-24T22:03:11.340841+0000","last_fullsized":"2021-03-24T22:03:11.340841+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":6,"last_scrub":"175'22","last_scrub_stamp":"2021-03-24T21:59:03.758935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:59:03.758935+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"139","length":"1"},{"start":"178","length":"1"},{"start":"1ae","length":"1"},{"start":"1eb","length":"1"},{"start":"1f4","length":"1"}]},{"pgid":"3.25","version":"0'0","reported_seq":"123","reported_epoch":"618","state":"active+clean","last_fresh":"2021-03-24T22:03:08.333911+0000","last_change":"2021-03-24T22:01:16.312159+0000","last_active":"2021-03-24T22:03:08.333911+0000","last_peered":"2021-03-24T22:03:08.333911+0000","last_clean":"2021-03-24T22:03:08.333911+0000","last_became_active":"2021-03-24T22:01:16.311807+0000","last_became_peered":"2021-03-24T22:01:16.311807+0000","last_unstale":"2021-03-24T22:03:08.333911+0000","last_undegraded":"2021-03-24T22:03:08.333911+0000","last_fullsized":"2021-03-24T22:03:08.333911+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:53.013129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:53.013129+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.21","version":"616'542","reported_seq":"2519","reported_epoch":"617","state":"active+clean","last_fresh":"2021-03-24T22:03:07.330845+0000","last_change":"2021-03-24T22:03:06.350519+0000","last_active":"2021-03-24T22:03:07.330845+0000","last_peered":"2021-03-24T22:03:07.330845+0000","last_clean":"2021-03-24T22:03:07.330845+0000","last_became_active":"2021-03-24T22:02:04.324578+0000","last_became_peered":"2021-03-24T22:02:04.324578+0000","last_unstale":"2021-03-24T22:03:07.330845+0000","last_undegraded":"2021-03-24T22:03:07.330845+0000","last_fullsized":"2021-03-24T22:03:07.330845+0000","mapping_epoch":553,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":554,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":542,"ondisk_log_size":542,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17355934,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":171,"num_read_kb":175650,"num_write":111,"num_write_kb":38484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":55123926,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"178","length":"1"},{"start":"1ae","length":"1"},{"start":"1c4","length":"1"}]}],"pool_stats":[{"poolid":7,"num_pg":36,"stat_sum":{"num_bytes":538804341,"num_objects":253,"num_object_clones":197,"num_object_copies":759,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":9,"num_read":3115,"num_read_kb":3287562,"num_write":2241,"num_write_kb":854703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2574,"num_bytes_recovered":1437091791,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1105235968,"data_stored":1533757696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":13748,"ondisk_log_size":13748,"up":108,"acting":108,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":146,"num_read_kb":141,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":3,"num_pg":42,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":4392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":984,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":126,"acting":126,"num_store_stats":8},{"poolid":4,"num_pg":22,"stat_sum":{"num_bytes":23524,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":40,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1484,"num_read_kb":989,"num_write":828,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":876,"num_bytes_recovered":68726,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966080,"data_stored":70572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":505,"ondisk_log_size":505,"up":66,"acting":66,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201421,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":241844,"kb_used_data":201964,"kb_used_omap":0,"kb_used_meta":39872,"kb_avail":93503308,"statfs":{"total":95995035648,"available":95747387392,"internally_reserved":0,"allocated":206811136,"data_stored":270361427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40828928},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 22:02:04 2021","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.77800000000000002,"15min":0.90800000000000003},"min":{"1min":0.56799999999999995,"5min":0.312,"15min":0.312},"max":{"1min":1.298,"5min":1.298,"15min":12.222},"last":0.75},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.98999999999999999,"15min":0.997},"min":{"1min":0.38100000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.2989999999999999,"5min":7.1609999999999996,"15min":11.506},"last":0.79400000000000004}]},{"osd":1,"last update":"Wed Mar 24 22:02:04 2021","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.82799999999999996,"15min":0.88100000000000001},"min":{"1min":0.47999999999999998,"5min":0.308,"15min":0.308},"max":{"1min":1.042,"5min":1.476,"15min":5.7750000000000004},"last":1.1379999999999999},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.94899999999999995,"15min":0.96899999999999997},"min":{"1min":0.49099999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.155,"5min":6.1849999999999996,"15min":6.1849999999999996},"last":1.0960000000000001}]},{"osd":2,"last update":"Wed Mar 24 22:02:04 2021","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.88100000000000001,"15min":1.0169999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.0700000000000001,"5min":5.7629999999999999,"15min":12.135999999999999},"last":0.97499999999999998},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.96199999999999997,"15min":1.038},"min":{"1min":0.60899999999999999,"5min":0.432,"15min":0.42999999999999999},"max":{"1min":1.4379999999999999,"5min":7.0460000000000003,"15min":12.178000000000001},"last":0.84099999999999997}]},{"osd":3,"last update":"Wed Mar 24 22:02:04 2021","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":0.998,"15min":1.077},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.39800000000000002},"max":{"1min":2.9470000000000001,"5min":7.2800000000000002,"15min":12.529},"last":0.877},{"interface":"front","average":{"1min":1.042,"5min":1.0249999999999999,"15min":1.0640000000000001},"min":{"1min":0.67500000000000004,"5min":0.443,"15min":0.443},"max":{"1min":1.496,"5min":3.101,"15min":12.666},"last":1.008}]},{"osd":4,"last update":"Wed Mar 24 22:02:04 2021","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.96199999999999997,"15min":0.97699999999999998},"min":{"1min":0.34799999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.3200000000000001,"5min":1.649,"15min":6.2089999999999996},"last":0.94399999999999995},{"interface":"front","average":{"1min":1.0089999999999999,"5min":1.0049999999999999,"15min":1.0880000000000001},"min":{"1min":0.56000000000000005,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.5229999999999999,"5min":1.921,"15min":12.244},"last":1.0649999999999999}]},{"osd":5,"last update":"Wed Mar 24 22:02:04 2021","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.151,"15min":1.1559999999999999},"min":{"1min":0.76500000000000001,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.5489999999999999,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":1.23},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.3069999999999999,"15min":1.3180000000000001},"min":{"1min":0.876,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.623,"5min":2.1190000000000002,"15min":2.1190000000000002},"last":1.298}]},{"osd":6,"last update":"Wed Mar 24 22:02:04 2021","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.083,"15min":1.1359999999999999},"min":{"1min":0.51800000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.429,"5min":1.7869999999999999,"15min":6.1159999999999997},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.139,"5min":1.0649999999999999,"15min":1.1830000000000001},"min":{"1min":0.502,"5min":0.434,"15min":0.434},"max":{"1min":1.607,"5min":1.867,"15min":12.287000000000001},"last":1.1719999999999999}]}]},{"osd":6,"up_from":47,"seq":201863463058,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":164928,"kb_used_data":135992,"kb_used_omap":0,"kb_used_meta":28928,"kb_avail":93580224,"statfs":{"total":95995035648,"available":95826149376,"internally_reserved":0,"allocated":139255808,"data_stored":185888292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29622272},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 22:02:37 2021","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.95599999999999996,"15min":0.93899999999999995},"min":{"1min":0.45500000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.194,"5min":9.0280000000000005,"15min":9.0280000000000005},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.95499999999999996,"15min":0.92600000000000005},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.21,"5min":5.3310000000000004,"15min":5.3310000000000004},"last":1.196}]},{"osd":1,"last update":"Wed Mar 24 22:02:37 2021","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.92400000000000004,"15min":0.92600000000000005},"min":{"1min":0.42899999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.1279999999999999,"5min":5.625,"15min":5.625},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.85899999999999999,"5min":1.002,"15min":0.96999999999999997},"min":{"1min":0.45400000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.196,"5min":5.4550000000000001,"15min":5.4550000000000001},"last":0.95399999999999996}]},{"osd":2,"last update":"Wed Mar 24 22:02:37 2021","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.92200000000000004,"15min":0.95199999999999996},"min":{"1min":0.42899999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.234,"5min":5.0860000000000003,"15min":5.0860000000000003},"last":1.246},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.93000000000000005,"15min":0.96699999999999997},"min":{"1min":0.41199999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.306,"5min":4.0700000000000003,"15min":4.0700000000000003},"last":1.0389999999999999}]},{"osd":3,"last update":"Wed Mar 24 22:02:37 2021","interfaces":[{"interface":"back","average":{"1min":1.0149999999999999,"5min":0.97399999999999998,"15min":0.97199999999999998},"min":{"1min":0.50600000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":4.7400000000000002,"5min":5.5039999999999996,"15min":5.5039999999999996},"last":1.119},{"interface":"front","average":{"1min":0.93200000000000005,"5min":1.0780000000000001,"15min":1.0640000000000001},"min":{"1min":0.57399999999999995,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.2689999999999999,"5min":4.3460000000000001,"15min":4.3460000000000001},"last":1.294}]},{"osd":4,"last update":"Wed Mar 24 22:02:37 2021","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.95899999999999996,"15min":0.97899999999999998},"min":{"1min":0.44700000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.25,"5min":1.7849999999999999,"15min":2.1640000000000001},"last":1.1319999999999999},{"interface":"front","average":{"1min":0.872,"5min":1.012,"15min":1.028},"min":{"1min":0.55000000000000004,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.405,"5min":2.3109999999999999,"15min":2.3109999999999999},"last":1.171}]},{"osd":5,"last update":"Wed Mar 24 22:03:08 2021","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.05,"15min":1.0760000000000001},"min":{"1min":0.55500000000000005,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.454,"5min":1.915,"15min":1.915},"last":0.998},{"interface":"front","average":{"1min":1.05,"5min":1.087,"15min":1.117},"min":{"1min":0.60199999999999998,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.4330000000000001,"5min":2.7069999999999999,"15min":2.7069999999999999},"last":1.2669999999999999}]},{"osd":7,"last update":"Wed Mar 24 22:03:03 2021","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.137,"15min":1.1699999999999999},"min":{"1min":0.63200000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.387,"5min":9.5909999999999993,"15min":9.5909999999999993},"last":1.331},{"interface":"front","average":{"1min":1.125,"5min":1.075,"15min":1.157},"min":{"1min":0.69099999999999995,"5min":0.442,"15min":0.442},"max":{"1min":1.4419999999999999,"5min":1.6080000000000001,"15min":2.8410000000000002},"last":1.3440000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575015,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":96220,"kb_used_data":76820,"kb_used_omap":0,"kb_used_meta":19392,"kb_avail":93648932,"statfs":{"total":95995035648,"available":95896506368,"internally_reserved":0,"allocated":78663680,"data_stored":122316435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":19857408},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 22:02:57 2021","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":1.0880000000000001,"15min":0.78900000000000003},"min":{"1min":0.39400000000000002,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.90800000000000003,"5min":21.085000000000001,"15min":21.085000000000001},"last":1.1559999999999999},{"interface":"front","average":{"1min":0.749,"5min":0.84099999999999997,"15min":0.751},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.4319999999999999,"5min":8.5180000000000007,"15min":8.5180000000000007},"last":1.0669999999999999}]},{"osd":2,"last update":"Wed Mar 24 22:03:08 2021","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.73399999999999999,"15min":0.68700000000000006},"min":{"1min":0.38900000000000001,"5min":0.307,"15min":0.307},"max":{"1min":1.5660000000000001,"5min":9.9870000000000001,"15min":9.9870000000000001},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.89900000000000002,"15min":0.81999999999999995},"min":{"1min":0.46700000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.4359999999999999,"5min":9.2059999999999995,"15min":9.2059999999999995},"last":0.98999999999999999}]},{"osd":3,"last update":"Wed Mar 24 22:02:37 2021","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.89000000000000001,"15min":1.0680000000000001},"min":{"1min":0.37,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.6390000000000002,"5min":8.5990000000000002,"15min":17.032},"last":0.72899999999999998},{"interface":"front","average":{"1min":1.5369999999999999,"5min":1.159,"15min":1.1240000000000001},"min":{"1min":0.45900000000000002,"5min":0.35499999999999998,"15min":0.316},"max":{"1min":15.775,"5min":15.775,"15min":16.908999999999999},"last":0.93899999999999995}]},{"osd":4,"last update":"Wed Mar 24 22:03:02 2021","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.98299999999999998,"15min":0.97399999999999998},"min":{"1min":0.66200000000000003,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.258,"5min":9.6150000000000002,"15min":9.6150000000000002},"last":1.034},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.0009999999999999,"15min":1.0589999999999999},"min":{"1min":0.58399999999999996,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.605,"5min":3.9359999999999999,"15min":3.9359999999999999},"last":1.2310000000000001}]},{"osd":5,"last update":"Wed Mar 24 22:03:02 2021","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.0449999999999999,"15min":1.054},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.6719999999999999,"5min":3.976,"15min":3.976},"last":1.4730000000000001},{"interface":"front","average":{"1min":1.036,"5min":1.127,"15min":1.095},"min":{"1min":0.48099999999999998,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.647,"5min":9.6809999999999992,"15min":9.6809999999999992},"last":1.1299999999999999}]},{"osd":6,"last update":"Wed Mar 24 22:02:37 2021","interfaces":[{"interface":"back","average":{"1min":1.585,"5min":1.274,"15min":1.177},"min":{"1min":0.63700000000000001,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":11.254,"5min":11.254,"15min":11.254},"last":1.2909999999999999},{"interface":"front","average":{"1min":1.532,"5min":1.091,"15min":1.1359999999999999},"min":{"1min":0.55600000000000005,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":9.7300000000000004,"5min":9.7300000000000004,"15min":9.7300000000000004},"last":1.331}]},{"osd":7,"last update":"Wed Mar 24 22:03:02 2021","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":1.0760000000000001,"15min":1.1850000000000001},"min":{"1min":0.68200000000000005,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.718,"5min":3.8439999999999999,"15min":4.4699999999999998},"last":1.407},{"interface":"front","average":{"1min":1.0880000000000001,"5min":1.296,"15min":1.2090000000000001},"min":{"1min":0.58399999999999996,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.6539999999999999,"5min":11.289,"15min":11.289},"last":1.379}]}]},{"osd":0,"up_from":8,"seq":34359738538,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":218204,"kb_used_data":172436,"kb_used_omap":0,"kb_used_meta":45760,"kb_avail":93526948,"statfs":{"total":95995035648,"available":95771594752,"internally_reserved":0,"allocated":176574464,"data_stored":245749468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":46858240},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Mar 24 22:02:58 2021","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.61799999999999999,"15min":0.64200000000000002},"min":{"1min":0.27700000000000002,"5min":0.249,"15min":0.249},"max":{"1min":1.248,"5min":1.248,"15min":4.8040000000000003},"last":1.131},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.69499999999999995,"15min":0.72099999999999997},"min":{"1min":0.33200000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.0429999999999999,"5min":2.5179999999999998,"15min":8.8989999999999991},"last":0.69499999999999995}]},{"osd":2,"last update":"Wed Mar 24 22:02:15 2021","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.83499999999999996,"15min":0.75800000000000001},"min":{"1min":0.36399999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.78000000000000003,"5min":15.949999999999999,"15min":15.949999999999999},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.73099999999999998,"15min":0.75900000000000001},"min":{"1min":0.371,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.1399999999999999,"5min":2.5499999999999998,"15min":4.766},"last":0.98799999999999999}]},{"osd":3,"last update":"Wed Mar 24 22:02:38 2021","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.86199999999999999,"15min":0.84399999999999997},"min":{"1min":0.5,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.1899999999999999,"5min":3.7869999999999999,"15min":5.9790000000000001},"last":1.145},{"interface":"front","average":{"1min":0.755,"5min":0.88,"15min":0.92300000000000004},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":1.2010000000000001,"5min":4.0229999999999997,"15min":6.4580000000000002},"last":0.72499999999999998}]},{"osd":4,"last update":"Wed Mar 24 22:02:58 2021","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":1.0149999999999999,"15min":1.0640000000000001},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.436},"max":{"1min":1.4790000000000001,"5min":3.8300000000000001,"15min":5.0330000000000004},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.99299999999999999,"5min":1.032,"15min":1.054},"min":{"1min":0.63100000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.3160000000000001,"5min":4.008,"15min":6.1829999999999998},"last":1.087}]},{"osd":5,"last update":"Wed Mar 24 22:03:05 2021","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.151,"15min":1.208},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.6379999999999999,"5min":3.9550000000000001,"15min":3.9550000000000001},"last":1.214},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.165,"15min":1.2250000000000001},"min":{"1min":0.73399999999999999,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.6240000000000001,"5min":4.2560000000000002,"15min":4.2560000000000002},"last":1.1819999999999999}]},{"osd":6,"last update":"Wed Mar 24 22:02:38 2021","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":1.204,"15min":1.1399999999999999},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.4199999999999999,"5min":15.983000000000001,"15min":15.983000000000001},"last":0.92700000000000005},{"interface":"front","average":{"1min":1.0289999999999999,"5min":1.0509999999999999,"15min":1.1599999999999999},"min":{"1min":0.60699999999999998,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.6910000000000001,"5min":2.21,"15min":4.9960000000000004},"last":1.292}]},{"osd":7,"last update":"Wed Mar 24 22:03:05 2021","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.1140000000000001,"15min":1.1639999999999999},"min":{"1min":0.60799999999999998,"5min":0.58599999999999997,"15min":0.55700000000000005},"max":{"1min":1.704,"5min":4.1669999999999998,"15min":6.1319999999999997},"last":1.2549999999999999},{"interface":"front","average":{"1min":1.052,"5min":1.214,"15min":1.181},"min":{"1min":0.65700000000000003,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.5069999999999999,"5min":16.007000000000001,"15min":16.007000000000001},"last":1.24}]}]},{"osd":2,"up_from":18,"seq":77309411492,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":205644,"kb_used_data":160708,"kb_used_omap":0,"kb_used_meta":44928,"kb_avail":93539508,"statfs":{"total":95995035648,"available":95784456192,"internally_reserved":0,"allocated":164564992,"data_stored":217874864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":46006272},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 22:02:11 2021","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.77300000000000002,"15min":0.69799999999999995},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":2.7080000000000002,"5min":10.054,"15min":10.054},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.78000000000000003,"15min":0.73999999999999999},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.91300000000000003,"5min":12.099,"15min":12.099},"last":0.66400000000000003}]},{"osd":1,"last update":"Wed Mar 24 22:02:11 2021","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.85799999999999998,"15min":0.77600000000000002},"min":{"1min":0.32600000000000001,"5min":0.28199999999999997,"15min":0.28100000000000003},"max":{"1min":2.3820000000000001,"5min":17.954999999999998,"15min":17.954999999999998},"last":0.621},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.71799999999999997,"15min":0.747},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.593,"5min":1.827,"15min":1.827},"last":0.51800000000000002}]},{"osd":3,"last update":"Wed Mar 24 22:02:40 2021","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":1.075,"15min":0.92600000000000005},"min":{"1min":0.54600000000000004,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.3089999999999999,"5min":25.009,"15min":25.009},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.87,"5min":0.86099999999999999,"15min":0.89100000000000001},"min":{"1min":0.56399999999999995,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.202,"5min":2.6440000000000001,"15min":2.6440000000000001},"last":0.877}]},{"osd":4,"last update":"Wed Mar 24 22:02:58 2021","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":1.1850000000000001,"15min":1.1000000000000001},"min":{"1min":0.47799999999999998,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.4079999999999999,"5min":24.914000000000001,"15min":24.914000000000001},"last":1.208},{"interface":"front","average":{"1min":1.0009999999999999,"5min":0.99399999999999999,"15min":1.0349999999999999},"min":{"1min":0.622,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.323,"5min":2.8290000000000002,"15min":2.8290000000000002},"last":0.98999999999999999}]},{"osd":5,"last update":"Wed Mar 24 22:03:09 2021","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.121,"15min":1.1200000000000001},"min":{"1min":0.78900000000000003,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":4.3019999999999996,"5min":4.3019999999999996,"15min":4.3019999999999996},"last":1.379},{"interface":"front","average":{"1min":1.149,"5min":1.101,"15min":1.0960000000000001},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.8360000000000001,"5min":2.3250000000000002,"15min":2.3250000000000002},"last":1.2609999999999999}]},{"osd":6,"last update":"Wed Mar 24 22:02:40 2021","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.0209999999999999,"15min":1.1419999999999999},"min":{"1min":0.83699999999999997,"5min":0.434,"15min":0.434},"max":{"1min":1.3180000000000001,"5min":2.7719999999999998,"15min":2.7719999999999998},"last":1.0720000000000001},{"interface":"front","average":{"1min":1.1439999999999999,"5min":1.113,"15min":1.1299999999999999},"min":{"1min":0.57999999999999996,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.3999999999999999,"5min":6.6520000000000001,"15min":6.6520000000000001},"last":1.4330000000000001}]},{"osd":7,"last update":"Wed Mar 24 22:03:01 2021","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.173,"15min":1.1990000000000001},"min":{"1min":0.80500000000000005,"5min":0.442,"15min":0.442},"max":{"1min":4.3630000000000004,"5min":6.7720000000000002,"15min":6.7720000000000002},"last":1.4650000000000001},{"interface":"front","average":{"1min":1.226,"5min":1.3620000000000001,"15min":1.2749999999999999},"min":{"1min":0.997,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.6399999999999999,"5min":24.969999999999999,"15min":24.969999999999999},"last":1.292}]}]},{"osd":3,"up_from":26,"seq":111669149855,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":277020,"kb_used_data":231252,"kb_used_omap":0,"kb_used_meta":45760,"kb_avail":93468132,"statfs":{"total":95995035648,"available":95711367168,"internally_reserved":0,"allocated":236802048,"data_stored":308343274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":46858240},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 22:02:31 2021","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":1.0489999999999999,"15min":0.85199999999999998},"min":{"1min":0.38,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":1.032,"5min":27.548999999999999,"15min":27.548999999999999},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.63300000000000001,"5min":1.0049999999999999,"15min":0.89000000000000001},"min":{"1min":0.35099999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.1719999999999999,"5min":26.593,"15min":26.593},"last":0.39400000000000002}]},{"osd":1,"last update":"Wed Mar 24 22:02:31 2021","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.69499999999999995,"15min":0.77000000000000002},"min":{"1min":0.314,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.83599999999999997,"5min":5.1280000000000001,"15min":14.324},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.69799999999999995,"15min":0.77000000000000002},"min":{"1min":0.38700000000000001,"5min":0.29599999999999999,"15min":0.28000000000000003},"max":{"1min":1.1479999999999999,"5min":2.423,"15min":12.631},"last":0.40999999999999998}]},{"osd":2,"last update":"Wed Mar 24 22:02:31 2021","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.80600000000000005,"15min":0.83599999999999997},"min":{"1min":0.42399999999999999,"5min":0.316,"15min":0.316},"max":{"1min":1.0169999999999999,"5min":7.9450000000000003,"15min":12.510999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.76400000000000001,"15min":0.82199999999999995},"min":{"1min":0.52800000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.004,"5min":1.704,"15min":11.317},"last":0.69999999999999996}]},{"osd":4,"last update":"Wed Mar 24 22:02:57 2021","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.95499999999999996,"15min":0.97399999999999998},"min":{"1min":0.622,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.4570000000000001,"5min":2.601,"15min":2.601},"last":1.0369999999999999},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.92800000000000005,"15min":1.0089999999999999},"min":{"1min":0.54900000000000004,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.258,"5min":1.6299999999999999,"15min":14.291},"last":1.155}]},{"osd":5,"last update":"Wed Mar 24 22:03:06 2021","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":1.0609999999999999,"15min":1.0940000000000001},"min":{"1min":0.68999999999999995,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.4099999999999999,"5min":1.5369999999999999,"15min":1.5369999999999999},"last":0.97899999999999998},{"interface":"front","average":{"1min":1.042,"5min":1.0680000000000001,"15min":1.0940000000000001},"min":{"1min":0.71499999999999997,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.5960000000000001,"5min":2.2759999999999998,"15min":2.2759999999999998},"last":0.82299999999999995}]},{"osd":6,"last update":"Wed Mar 24 22:02:31 2021","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.96999999999999997,"15min":1.1100000000000001},"min":{"1min":0.59799999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.3100000000000001,"5min":1.639,"15min":14.255000000000001},"last":0.95399999999999996},{"interface":"front","average":{"1min":0.94699999999999995,"5min":0.99099999999999999,"15min":1.131},"min":{"1min":0.69999999999999996,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.3069999999999999,"5min":1.7,"15min":12.666},"last":1.1879999999999999}]},{"osd":7,"last update":"Wed Mar 24 22:03:01 2021","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.069,"15min":1.109},"min":{"1min":0.69299999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.419,"5min":2.633,"15min":2.633},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.05,"5min":1.1200000000000001,"15min":1.1990000000000001},"min":{"1min":0.627,"5min":0.52000000000000002,"15min":0.44700000000000001},"max":{"1min":1.458,"5min":9.609,"15min":14.727},"last":1.0029999999999999}]}]},{"osd":4,"up_from":32,"seq":137438953625,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":112804,"kb_used_data":76188,"kb_used_omap":0,"kb_used_meta":36608,"kb_avail":93632348,"statfs":{"total":95995035648,"available":95879524352,"internally_reserved":0,"allocated":78016512,"data_stored":113357589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":37486592},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 22:02:56 2021","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.86299999999999999,"15min":1},"min":{"1min":0.47799999999999998,"5min":0.223,"15min":0.223},"max":{"1min":1.264,"5min":5.9050000000000002,"15min":41.472000000000001},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.80700000000000005,"5min":1.0700000000000001,"15min":1.137},"min":{"1min":0.47799999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.2769999999999999,"5min":17.513999999999999,"15min":44.185000000000002},"last":0.748}]},{"osd":1,"last update":"Wed Mar 24 22:02:56 2021","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.90900000000000003,"15min":0.92000000000000004},"min":{"1min":0.42699999999999999,"5min":0.39400000000000002,"15min":0.38100000000000001},"max":{"1min":3.371,"5min":5.75,"15min":5.75},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.95099999999999996,"5min":1.071,"15min":0.94299999999999995},"min":{"1min":0.54600000000000004,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.363,"5min":11.983000000000001,"15min":11.983000000000001},"last":0.71999999999999997}]},{"osd":2,"last update":"Wed Mar 24 22:02:56 2021","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.84599999999999997,"15min":0.90600000000000003},"min":{"1min":0.62,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.2090000000000001,"5min":2.1779999999999999,"15min":4.4379999999999997},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.96099999999999997,"15min":0.96699999999999997},"min":{"1min":0.46999999999999997,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.8580000000000001,"5min":5.7999999999999998,"15min":5.7999999999999998},"last":0.68000000000000005}]},{"osd":3,"last update":"Wed Mar 24 22:02:56 2021","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.056,"15min":1},"min":{"1min":0.501,"5min":0.45200000000000001,"15min":0.439},"max":{"1min":1.702,"5min":4.8049999999999997,"15min":4.8049999999999997},"last":0.624},{"interface":"front","average":{"1min":1.04,"5min":1.0289999999999999,"15min":0.99299999999999999},"min":{"1min":0.71899999999999997,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.3320000000000001,"5min":4.1189999999999998,"15min":4.1189999999999998},"last":0.78900000000000003}]},{"osd":5,"last update":"Wed Mar 24 22:03:05 2021","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.96099999999999997,"15min":0.92700000000000005},"min":{"1min":0.63,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.387,"5min":2.9569999999999999,"15min":2.9569999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":1.125,"5min":1.143,"15min":1.1299999999999999},"min":{"1min":0.76700000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.744,"5min":2.9089999999999998,"15min":2.9089999999999998},"last":0.76700000000000002}]},{"osd":6,"last update":"Wed Mar 24 22:02:28 2021","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.077,"15min":1.0740000000000001},"min":{"1min":0.746,"5min":0.373,"15min":0.373},"max":{"1min":7.7679999999999998,"5min":7.7679999999999998,"15min":7.7679999999999998},"last":0.56499999999999995},{"interface":"front","average":{"1min":1.8819999999999999,"5min":1.1819999999999999,"15min":1.1160000000000001},"min":{"1min":0.874,"5min":0.41199999999999998,"15min":0.41099999999999998},"max":{"1min":14.122999999999999,"5min":14.122999999999999,"15min":14.122999999999999},"last":0.82499999999999996}]},{"osd":7,"last update":"Wed Mar 24 22:03:00 2021","interfaces":[{"interface":"back","average":{"1min":1.2250000000000001,"5min":1.101,"15min":1.147},"min":{"1min":0.74099999999999999,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.8899999999999999,"5min":3.3479999999999999,"15min":3.3479999999999999},"last":0.85299999999999998},{"interface":"front","average":{"1min":1.103,"5min":1.0580000000000001,"15min":1.1499999999999999},"min":{"1min":0.84799999999999998,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.448,"5min":2.0350000000000001,"15min":2.0350000000000001},"last":0.78200000000000003}]}]},{"osd":5,"up_from":432,"seq":1855425871911,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":122136,"kb_used_data":96144,"kb_used_omap":33,"kb_used_meta":25950,"kb_avail":93623016,"statfs":{"total":95995035648,"available":95869968384,"internally_reserved":0,"allocated":98451456,"data_stored":129202656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33936,"internal_metadata":26573680},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 22:03:04 2021","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":3.4769999999999999,"15min":4.6840000000000002},"min":{"1min":0.51000000000000001,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.1579999999999999,"5min":76.159999999999997,"15min":76.159999999999997},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.82199999999999995,"5min":3.419,"15min":4.593},"min":{"1min":0.42099999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.804,"5min":75.311999999999998,"15min":75.311999999999998},"last":0.875}]},{"osd":1,"last update":"Wed Mar 24 22:03:04 2021","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.83099999999999996,"15min":0.81499999999999995},"min":{"1min":0.64600000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.694,"5min":1.694,"15min":1.694},"last":1.516},{"interface":"front","average":{"1min":1.046,"5min":0.89500000000000002,"15min":0.88400000000000001},"min":{"1min":0.69499999999999995,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":2.5089999999999999,"5min":2.5089999999999999,"15min":2.5089999999999999},"last":0.85099999999999998}]},{"osd":2,"last update":"Wed Mar 24 22:03:04 2021","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.84799999999999998,"15min":0.86099999999999999},"min":{"1min":0.59199999999999997,"5min":0.434,"15min":0.434},"max":{"1min":1.262,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":1.6160000000000001},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.95699999999999996,"15min":0.97199999999999998},"min":{"1min":0.65200000000000002,"5min":0.495,"15min":0.495},"max":{"1min":2.5910000000000002,"5min":2.6070000000000002,"15min":2.6070000000000002},"last":0.92100000000000004}]},{"osd":3,"last update":"Wed Mar 24 22:03:04 2021","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":1.0840000000000001,"15min":1.149},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.23,"5min":4.1340000000000003,"15min":4.1340000000000003},"last":1.3109999999999999},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.1819999999999999,"15min":1.242},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.3700000000000001,"5min":4.173,"15min":4.173},"last":1.357}]},{"osd":4,"last update":"Wed Mar 24 22:03:04 2021","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":0.94899999999999995,"15min":0.94499999999999995},"min":{"1min":0.61599999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.4690000000000001,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":1.05},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.0860000000000001,"15min":1.1080000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.444,"5min":1.9350000000000001,"15min":1.9350000000000001},"last":1.1040000000000001}]},{"osd":6,"last update":"Wed Mar 24 22:03:04 2021","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.1180000000000001,"15min":1.125},"min":{"1min":0.73199999999999998,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.554,"5min":2.0510000000000002,"15min":2.0510000000000002},"last":1.02},{"interface":"front","average":{"1min":1.056,"5min":1.081,"15min":1.083},"min":{"1min":0.70899999999999996,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.516,"5min":1.851,"15min":1.851},"last":1.8109999999999999}]},{"osd":7,"last update":"Wed Mar 24 22:03:04 2021","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.181,"15min":1.208},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.6359999999999999,"5min":2.161,"15min":2.161},"last":1.1950000000000001},{"interface":"front","average":{"1min":1.198,"5min":1.177,"15min":1.173},"min":{"1min":0.873,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.698,"5min":1.8919999999999999,"15min":1.8919999999999999},"last":1.8979999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":984,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":5284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":15870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":6670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":286720,"data_stored":10238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":352256,"data_stored":12738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":5674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":8740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":5358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":172175360,"data_stored":243238423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73977856,"data_stored":119789004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":160129024,"data_stored":215349843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":232288256,"data_stored":305828987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73449472,"data_stored":110854454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":94035968,"data_stored":126677758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":134758400,"data_stored":183359970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":164421632,"data_stored":228659257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-03-24T22:03:15.161 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph pg dump --format=json 2021-03-24T22:03:15.162 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:15.216 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:15.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:15.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:03:15.319 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:03:15.323 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:15.339 INFO:tasks.rados.rados.0.smithi125.stdout:2354: done (1 left) 2021-03-24T22:03:15.339 INFO:tasks.rados.rados.0.smithi125.stdout:2355: write (excl) oid 20 current snap is 265 2021-03-24T22:03:15.339 INFO:tasks.rados.rados.0.smithi125.stdout:2355: seq_num 813 ranges {464091=710855,1925037=477270,2943246=474970} 2021-03-24T22:03:15.352 INFO:tasks.rados.rados.0.smithi125.stdout:2355: writing smithi12556470-20 from 464091 to 1174946 tid 1 2021-03-24T22:03:15.361 INFO:tasks.rados.rados.0.smithi125.stdout:2355: writing smithi12556470-20 from 1925037 to 2402307 tid 2 2021-03-24T22:03:15.369 INFO:tasks.rados.rados.0.smithi125.stdout:2355: writing smithi12556470-20 from 2943246 to 3418216 tid 3 2021-03-24T22:03:15.370 INFO:tasks.rados.rados.0.smithi125.stdout:2353: expect (ObjNum 744 snap 240 seq_num 744) 2021-03-24T22:03:15.402 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:15.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:15.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:03:15.507 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:15.556 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:15.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:15.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:03:15.598 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:15 smithi125 conmon[19936]: cluster 2021-03-24T22:03:13.700634+0000 2021-03-24T22:03:15.598 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:15 smithi125 conmon[19936]: osd.0 (osd.0) 124 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:03:15.599 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:15 smithi125 conmon[19936]: cluster 2021-03-24T22:03:13.703424+0000 osd.0 (osd.0) 125 2021-03-24T22:03:15.599 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:15 smithi125 conmon[19936]: : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:03:15.599 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:15 smithi125 conmon[19936]: cluster 2021-03-24T22:03:15.599 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:15 smithi125 conmon[19936]: 2021-03 2021-03-24T22:03:15.599 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:15 smithi125 conmon[19936]: -24T22:03:14.369191+0000 mon.a (mon.0 2021-03-24T22:03:15.600 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:15 smithi125 conmon[19936]: ) 1687 : cluster [DBG] osdmap e624: 8 total, 8 up, 8 in 2021-03-24T22:03:15.600 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:15 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T22:03:15.600 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:15 smithi125 conmon[19936]: 22:03:15.329642+0000 mon.a (mon.0) 2021-03-24T22:03:15.600 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:15 smithi125 conmon[19936]: 1688 : cluster [DBG] osdmap e625: 8 total, 8 up, 8 in 2021-03-24T22:03:15.601 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:15 smithi125 conmon[24927]: cluster 2021-03-24T22:03:13 2021-03-24T22:03:15.601 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:15 smithi125 conmon[24927]: .700634+0000 osd.0 (osd.0) 124 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:03:15.601 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:15 smithi125 conmon[24927]: cluster 2021-03-24T22:03:15.601 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:15 smithi125 conmon[24927]: 2021-03- 2021-03-24T22:03:15.602 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:15 smithi125 conmon[24927]: 24T22:03:13.703424+0000 2021-03-24T22:03:15.602 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:15 smithi125 conmon[24927]: osd.0 (osd.0) 125 : cluster [DBG] 2021-03-24T22:03:15.602 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:15 smithi125 conmon[24927]: purged_snaps scrub ok 2021-03-24T22:03:15.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:15 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:03:15.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:15 smithi125 conmon[24927]: -03-24T22:03 2021-03-24T22:03:15.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:15 smithi125 conmon[24927]: :14.369191+0000 mon.a (mon.0 2021-03-24T22:03:15.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:15 smithi125 conmon[24927]: ) 1687 : cluster [DBG] osdmap e624: 8 total, 8 up, 8 in 2021-03-24T22:03:15.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:15 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:03:15.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:15 smithi125 conmon[24927]: 03-24T22:03:15.329642+0000 2021-03-24T22:03:15.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:15 smithi125 conmon[24927]: mon.a (mon.0) 1688 : cluster 2021-03-24T22:03:15.606 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:15 smithi125 conmon[24927]: [DBG] osdmap e625: 8 total, 8 up, 8 in 2021-03-24T22:03:15.612 INFO:tasks.rados.rados.0.smithi125.stdout:2355: finishing write tid 1 to smithi12556470-20 2021-03-24T22:03:15.613 INFO:tasks.rados.rados.0.smithi125.stdout:2355: finishing write tid 2 to smithi12556470-20 2021-03-24T22:03:15.613 INFO:tasks.rados.rados.0.smithi125.stdout:2355: finishing write tid 3 to smithi12556470-20 2021-03-24T22:03:15.613 INFO:tasks.rados.rados.0.smithi125.stdout:2355: finishing write tid 4 to smithi12556470-20 2021-03-24T22:03:15.614 INFO:tasks.rados.rados.0.smithi125.stdout:2353: done (1 left) 2021-03-24T22:03:15.614 INFO:tasks.rados.rados.0.smithi125.stdout:2356: write oid 19 current snap is 265 2021-03-24T22:03:15.614 INFO:tasks.rados.rados.0.smithi125.stdout:2355: finishing write tid 5 to smithi12556470-20 2021-03-24T22:03:15.614 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 20 v 793 (ObjNum 813 snap 265 seq_num 813) dirty exists 2021-03-24T22:03:15.615 INFO:tasks.rados.rados.0.smithi125.stdout:2355: left oid 20 (ObjNum 813 snap 265 seq_num 813) 2021-03-24T22:03:15.615 INFO:tasks.rados.rados.0.smithi125.stdout:2356: seq_num 814 ranges {480898=502457,1699872=737281,3087022=593876} 2021-03-24T22:03:15.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:15 smithi142 conmon[20125]: cluster 2021-03-24T22:03:13.700634+0000 osd.0 2021-03-24T22:03:15.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:15 smithi142 conmon[20125]: (osd.0) 124 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:03:15.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:15 smithi142 conmon[20125]: cluster 2021-03-24T22:03:13.703424+0000 osd.0 (osd.0) 125 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:03:15.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:15 smithi142 conmon[20125]: cluster 2021-03-24T22:03:15.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:15 smithi142 conmon[20125]: 2021-03-24T22:03:14.369191+0000 mon.a (mon.0) 1687 : cluster [DBG] osdmap e624: 8 total, 8 up, 8 in 2021-03-24T22:03:15.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:15 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:03:15.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:15 smithi142 conmon[20125]: 03-24T22:03:15.329642+0000 2021-03-24T22:03:15.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:15 smithi142 conmon[20125]: mon.a (mon.0) 1688 : cluster [DBG] osdmap e625: 8 total, 8 up, 8 in 2021-03-24T22:03:15.618 INFO:tasks.rados.rados.0.smithi125.stdout:2356: writing smithi12556470-19 from 480898 to 983355 tid 1 2021-03-24T22:03:15.630 INFO:tasks.rados.rados.0.smithi125.stdout:2356: writing smithi12556470-19 from 1699872 to 2437153 tid 2 2021-03-24T22:03:15.643 INFO:tasks.rados.rados.0.smithi125.stdout:2356: writing smithi12556470-19 from 3087022 to 3680898 tid 3 2021-03-24T22:03:15.643 INFO:tasks.rados.rados.0.smithi125.stdout:2355: done (1 left) 2021-03-24T22:03:15.643 INFO:tasks.rados.rados.0.smithi125.stdout:2357: copy_from oid 12 from oid 21 current snap is 265 2021-03-24T22:03:15.643 INFO:tasks.rados.rados.0.smithi125.stdout:2356: finishing write tid 1 to smithi12556470-19 2021-03-24T22:03:15.644 INFO:tasks.rados.rados.0.smithi125.stdout:2356: finishing write tid 2 to smithi12556470-19 2021-03-24T22:03:15.644 INFO:tasks.rados.rados.0.smithi125.stdout:2358: write (excl) oid 18 current snap is 265 2021-03-24T22:03:15.644 INFO:tasks.rados.rados.0.smithi125.stdout:2358: seq_num 815 ranges {497705=577706,1642001=480939,2598092=497514} 2021-03-24T22:03:15.648 INFO:tasks.rados.rados.0.smithi125.stdout:2358: writing smithi12556470-18 from 497705 to 1075411 tid 1 2021-03-24T22:03:15.658 INFO:tasks.rados.rados.0.smithi125.stdout:2358: writing smithi12556470-18 from 1642001 to 2122940 tid 2 2021-03-24T22:03:15.662 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:15.663 INFO:tasks.rados.rados.0.smithi125.stdout:2358: writing smithi12556470-18 from 2598092 to 3095606 tid 3 2021-03-24T22:03:15.664 INFO:tasks.rados.rados.0.smithi125.stdout:2359: snap_remove snap 242 2021-03-24T22:03:15.676 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-24T22:03:15.676 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T22:03:15.695 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:03:15.719 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:15.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:15.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:03:15.824 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:15.874 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:15.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:15.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:03:15.979 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:16.035 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:16.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:16.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:03:16.141 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:16.197 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:16.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:16.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:03:16.303 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:16.359 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:16.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:16.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:03:16.399 INFO:tasks.rados.rados.0.smithi125.stdout:2359: done (3 left) 2021-03-24T22:03:16.400 INFO:tasks.rados.rados.0.smithi125.stdout:2360: copy_from oid 46 from oid 23 current snap is 265 2021-03-24T22:03:16.400 INFO:tasks.rados.rados.0.smithi125.stdout:2361: rollback oid 29 current snap is 265 2021-03-24T22:03:16.400 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 29 to 256 2021-03-24T22:03:16.400 INFO:tasks.rados.rados.0.smithi125.stdout:2362: copy_from oid 30 from oid 33 current snap is 265 2021-03-24T22:03:16.401 INFO:tasks.rados.rados.0.smithi125.stdout:2363: snap_create 2021-03-24T22:03:16.401 INFO:tasks.rados.rados.0.smithi125.stdout:2357: finishing copy_from to smithi12556470-12 2021-03-24T22:03:16.401 INFO:tasks.rados.rados.0.smithi125.stdout:2357: got expected ENOENT (src dne) 2021-03-24T22:03:16.402 INFO:tasks.rados.rados.0.smithi125.stdout:2357: finishing copy_from racing read to smithi12556470-12 2021-03-24T22:03:16.402 INFO:tasks.rados.rados.0.smithi125.stdout:2356: finishing write tid 3 to smithi12556470-19 2021-03-24T22:03:16.402 INFO:tasks.rados.rados.0.smithi125.stdout:2356: finishing write tid 4 to smithi12556470-19 2021-03-24T22:03:16.403 INFO:tasks.rados.rados.0.smithi125.stdout:2356: finishing write tid 5 to smithi12556470-19 2021-03-24T22:03:16.403 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 19 v 562 (ObjNum 814 snap 265 seq_num 814) dirty exists 2021-03-24T22:03:16.403 INFO:tasks.rados.rados.0.smithi125.stdout:2356: left oid 19 (ObjNum 814 snap 265 seq_num 814) 2021-03-24T22:03:16.403 INFO:tasks.rados.rados.0.smithi125.stdout:2358: finishing write tid 1 to smithi12556470-18 2021-03-24T22:03:16.404 INFO:tasks.rados.rados.0.smithi125.stdout:2358: finishing write tid 2 to smithi12556470-18 2021-03-24T22:03:16.405 INFO:tasks.rados.rados.0.smithi125.stdout:2358: finishing write tid 3 to smithi12556470-18 2021-03-24T22:03:16.405 INFO:tasks.rados.rados.0.smithi125.stdout:2358: finishing write tid 4 to smithi12556470-18 2021-03-24T22:03:16.405 INFO:tasks.rados.rados.0.smithi125.stdout:2358: finishing write tid 5 to smithi12556470-18 2021-03-24T22:03:16.405 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 18 v 696 (ObjNum 815 snap 265 seq_num 815) dirty exists 2021-03-24T22:03:16.406 INFO:tasks.rados.rados.0.smithi125.stdout:2358: left oid 18 (ObjNum 815 snap 265 seq_num 815) 2021-03-24T22:03:16.423 INFO:teuthology.orchestra.run.smithi125.stderr:noscrub is set 2021-03-24T22:03:16.431 INFO:tasks.rados.rados.0.smithi125.stdout:2361: finishing rollback tid 0 to smithi12556470-29 2021-03-24T22:03:16.431 INFO:tasks.rados.rados.0.smithi125.stdout:2361: finishing rollback tid 1 to smithi12556470-29 2021-03-24T22:03:16.432 INFO:tasks.rados.rados.0.smithi125.stdout:2361: finishing rollback tid 2 to smithi12556470-29 2021-03-24T22:03:16.432 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 29 v 382 (ObjNum 777 snap 254 seq_num 777) dirty exists 2021-03-24T22:03:16.435 INFO:tasks.rados.rados.0.smithi125.stdout:2360: finishing copy_from to smithi12556470-46 2021-03-24T22:03:16.435 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 46 v 395 (ObjNum 799 snap 258 seq_num 799) dirty exists 2021-03-24T22:03:16.435 INFO:tasks.rados.rados.0.smithi125.stdout:2360: finishing copy_from racing read to smithi12556470-46 2021-03-24T22:03:16.441 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:16 smithi125 conmon[19936]: cluster 2021-03-24T22:03: 2021-03-24T22:03:16.441 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:16 smithi125 conmon[19936]: 14.428058+0000 osd.3 (osd.3) 127 : cluster [DBG] 7.a scrub ok 2021-03-24T22:03:16.442 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:16 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:03:16.442 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:16 smithi125 conmon[19936]: T22:03:14.924092+0000 mgr.y (mgr.14144) 585 : cluster [DBG] pgmap v1099: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 478 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 15 op/s; 40/1668 objects misplaced (2.398%) 2021-03-24T22:03:16.442 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:16 smithi125 conmon[19936]: audit 2021-03-24T22:03:16.442 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:16 smithi125 conmon[19936]: 2021-03-24T22: 2021-03-24T22:03:16.443 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:16 smithi125 conmon[19936]: 03:15.437231+0000 mon.a (mon.0) 1689 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T22:03:16.443 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:16 smithi125 conmon[19936]: 2021-03-24T22:03:16.443 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:16 smithi125 conmon[19936]: cluster 2021-03-24T22:03:16.444 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:16 smithi125 conmon[19936]: 2021-03-24T 2021-03-24T22:03:16.444 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:16 smithi125 conmon[19936]: 22:03:15.443161+0000 osd.7 (osd.7) 103 : cluster [DBG] 6.5 scrub ok 2021-03-24T22:03:16.444 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:16 smithi125 conmon[19936]: audit 2021- 2021-03-24T22:03:16.444 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:16 smithi125 conmon[19936]: 03-24T22:03:16.096906+0000 mon.a (mon 2021-03-24T22:03:16.445 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:16 smithi125 conmon[19936]: .0) 1690 : audit [INF] from='client.? v1:172.21.15.125:0/1197418689' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T22:03:16.445 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:16 smithi125 conmon[24927]: cluster 2021-03-24T22:03:16.445 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:16 smithi125 conmon[24927]: 2021-03-24T22:03:16.446 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:16 smithi125 conmon[24927]: 2021 2021-03-24T22:03:16.446 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:16 smithi125 conmon[24927]: -03-24T22: 2021-03-24T22:03:16.446 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:16 smithi125 conmon[24927]: 03:14.428058+0000 osd.3 (osd.3) 127 : cluster [DBG] 7.a scrub ok 2021-03-24T22:03:16.447 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:16 smithi125 conmon[24927]: cluster 2021-03-24T22:03:16.447 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:16 smithi125 conmon[24927]: 2021-03-24T22:03:16.447 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:16 smithi125 conmon[24927]: 2021 2021-03-24T22:03:16.447 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:16 smithi125 conmon[24927]: -03-24T22:03:14. 2021-03-24T22:03:16.448 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:16 smithi125 conmon[24927]: 924092+0000 mgr.y (mgr.14144) 585 : cluster [DBG] pgmap v1099: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 478 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 15 op/s; 40/1668 objects misplaced (2.398%) 2021-03-24T22:03:16.448 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:16 smithi125 conmon[24927]: audit 2021-03-24 2021-03-24T22:03:16.448 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:16 smithi125 conmon[24927]: T22:03:15.437231+0000 mon.a (mon. 2021-03-24T22:03:16.448 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:16 smithi125 conmon[24927]: 0) 1689 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T22:03:16.449 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:16 smithi125 conmon[24927]: cluster 2021-03-24T22:03: 2021-03-24T22:03:16.449 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:16 smithi125 conmon[24927]: 15.443161+0000 osd.7 (osd.7) 103 : cluster [DBG] 6.5 scrub ok 2021-03-24T22:03:16.449 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:16 smithi125 conmon[24927]: audit 2021-03-24T22: 2021-03-24T22:03:16.449 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:16 smithi125 conmon[24927]: 03:16.096906+0000 mon.a (mon.0) 1690 : audit [INF] 2021-03-24T22:03:16.449 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:16 smithi125 conmon[24927]: from='client.? v1:172.21.15.125:0/1197418689' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T22:03:16.450 INFO:tasks.rados.rados.0.smithi125.stdout:2362: finishing copy_from to smithi12556470-30 2021-03-24T22:03:16.450 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 30 v 824 (ObjNum 764 snap 249 seq_num 764) dirty exists 2021-03-24T22:03:16.451 INFO:tasks.rados.rados.0.smithi125.stdout:2362: finishing copy_from racing read to smithi12556470-30 2021-03-24T22:03:16.465 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:16.527 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:16.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:16.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:03:16.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:16 smithi142 conmon[20125]: cluster 2021-03-24T22:03:14.428058+0000 osd.3 (osd.3) 127 2021-03-24T22:03:16.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:16 smithi142 conmon[20125]: : cluster [DBG] 7.a scrub ok 2021-03-24T22:03:16.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:16 smithi142 conmon[20125]: cluster 2021-03-24T22:03:14.924092+0000 2021-03-24T22:03:16.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:16 smithi142 conmon[20125]: mgr.y (mgr.14144) 585 : cluster [DBG] pgmap v1099: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 478 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 15 op/s; 40/1668 objects misplaced (2.398%) 2021-03-24T22:03:16.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:16 smithi142 conmon[20125]: audit 2021-03-24 2021-03-24T22:03:16.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:16 smithi142 conmon[20125]: T22:03:15.437231+0000 mon.a ( 2021-03-24T22:03:16.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:16 smithi142 conmon[20125]: mon.0) 1689 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T22:03:16.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:16 smithi142 conmon[20125]: 2021-03-24T22:03:16.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:16 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:03:16.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:16 smithi142 conmon[20125]: -03-24T22:03:15.443161+0000 osd.7 2021-03-24T22:03:16.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:16 smithi142 conmon[20125]: (osd.7) 103 : cluster [DBG] 2021-03-24T22:03:16.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:16 smithi142 conmon[20125]: 6.5 scrub ok 2021-03-24T22:03:16.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:16 smithi142 conmon[20125]: audit 2021-03-24T22:03:16.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:16 smithi142 conmon[20125]: 2021-03-24T22:03:16.096906+0000 mon.a (mon 2021-03-24T22:03:16.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:16 smithi142 conmon[20125]: .0) 1690 : audit [INF] from='client.? v1:172.21.15.125:0/1197418689' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T22:03:16.634 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:16.691 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:16.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:16.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:03:16.797 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:16.858 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:16.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:16.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:16.965 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:17.025 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:17.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:17.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:03:17.131 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:17.189 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:17.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:17.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:03:17.295 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:17.356 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:17.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:17.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:17.406 INFO:tasks.rados.rados.0.smithi125.stdout:2356: done (6 left) 2021-03-24T22:03:17.406 INFO:tasks.rados.rados.0.smithi125.stdout:2357: done (5 left) 2021-03-24T22:03:17.406 INFO:tasks.rados.rados.0.smithi125.stdout:2358: done (4 left) 2021-03-24T22:03:17.407 INFO:tasks.rados.rados.0.smithi125.stdout:2360: done (3 left) 2021-03-24T22:03:17.407 INFO:tasks.rados.rados.0.smithi125.stdout:2361: done (2 left) 2021-03-24T22:03:17.407 INFO:tasks.rados.rados.0.smithi125.stdout:2362: done (1 left) 2021-03-24T22:03:17.407 INFO:tasks.rados.rados.0.smithi125.stdout:2363: done (0 left) 2021-03-24T22:03:17.407 INFO:tasks.rados.rados.0.smithi125.stdout:2364: copy_from oid 31 from oid 28 current snap is 266 2021-03-24T22:03:17.408 INFO:tasks.rados.rados.0.smithi125.stdout:2365: copy_from oid 43 from oid 8 current snap is 266 2021-03-24T22:03:17.408 INFO:tasks.rados.rados.0.smithi125.stdout:2366: snap_remove snap 256 2021-03-24T22:03:17.463 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:17.500 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:17 smithi125 conmon[19936]: cluster 2021-03-24T22:03: 2021-03-24T22:03:17.500 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:17 smithi125 conmon[19936]: 16.377874+0000 mon.a (mon.0) 1691 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:03:17.501 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:17 smithi125 conmon[19936]: audit 2021-03-24T22 2021-03-24T22:03:17.501 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:17 smithi125 conmon[19936]: :03:16.395577+0000 mon.a (mon.0) 1692 : audit [INF] 2021-03-24T22:03:17.501 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:17 smithi125 conmon[19936]: from='client.? v1:172.21.15.125:0/1197418689' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T22:03:17.525 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:17.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:17.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:17.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:17 smithi142 conmon[20125]: cluster 2021-03-24T22:03:16.377874+0000 mon.a (mon.0) 1691 : 2021-03-24T22:03:17.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:17 smithi142 conmon[20125]: cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:03:17.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:17 smithi142 conmon[20125]: audit 2021-03-24T22:03:16.395577+0000 mon.a (mon.0) 1692 : 2021-03-24T22:03:17.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:17 smithi142 conmon[20125]: audit [INF] from='client.? v1:172.21.15.125:0/1197418689' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T22:03:17.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:17 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:03:17.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:17 smithi142 conmon[20125]: 03-24T22:03:16.395670+0000 mon.a (mon.0 2021-03-24T22:03:17.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:17 smithi142 conmon[20125]: ) 1693 : cluster [DBG] osdmap e626: 8 total, 8 up, 8 in 2021-03-24T22:03:17.629 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:17.690 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:17.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:17.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:03:17.797 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:17.802 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:17 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:03:17.802 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:17 smithi125 conmon[19936]: 03-24T22:03:16.395670+0000 mon.a (mon. 2021-03-24T22:03:17.802 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:17 smithi125 conmon[19936]: 0) 1693 : cluster [DBG] osdmap e626: 8 total, 8 up, 8 in 2021-03-24T22:03:17.803 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:17 smithi125 conmon[24927]: cluster 2021-03-24T22:03:16.377874+0000 mon.a (mon.0) 1691 : 2021-03-24T22:03:17.803 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:17 smithi125 conmon[24927]: cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:03:17.803 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:17 smithi125 conmon[24927]: audit 2021-03-24T22:03:16.395577+0000 mon.a (mon.0) 1692 : audit [INF] from='client.? v1:172.21.15.125:0/1197418689' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T22:03:17.803 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:17 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T22:03:17.804 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:17 smithi125 conmon[24927]: 22:03:16.395670+0000 mon.a (mon.0) 1693 : cluster [DBG] osdmap e626: 8 total, 8 up, 8 in 2021-03-24T22:03:17.832 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.1 dump_ops_in_flight 2021-03-24T22:03:17.854 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:17.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:17.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:03:17.961 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:18.019 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:18.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:18.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:03:18.125 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:18.189 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:18.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:18.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:03:18.295 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:18.352 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:18.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:18.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:03:18.418 INFO:tasks.rados.rados.0.smithi125.stdout:2366: done (2 left) 2021-03-24T22:03:18.418 INFO:tasks.rados.rados.0.smithi125.stdout:2367: delete oid 33 current snap is 266 2021-03-24T22:03:18.418 INFO:tasks.rados.rados.0.smithi125.stdout:2364: finishing copy_from to smithi12556470-31 2021-03-24T22:03:18.419 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 31 v 1185 (ObjNum 718 snap 229 seq_num 718) dirty exists 2021-03-24T22:03:18.419 INFO:tasks.rados.rados.0.smithi125.stdout:2364: finishing copy_from racing read to smithi12556470-31 2021-03-24T22:03:18.419 INFO:tasks.rados.rados.0.smithi125.stdout:2365: finishing copy_from to smithi12556470-43 2021-03-24T22:03:18.419 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 43 v 222 (ObjNum 669 snap 208 seq_num 669) dirty exists 2021-03-24T22:03:18.420 INFO:tasks.rados.rados.0.smithi125.stdout:2365: finishing copy_from racing read to smithi12556470-43 2021-03-24T22:03:18.434 INFO:tasks.rados.rados.0.smithi125.stdout:2364: done (2 left) 2021-03-24T22:03:18.434 INFO:tasks.rados.rados.0.smithi125.stdout:2365: done (1 left) 2021-03-24T22:03:18.435 INFO:tasks.rados.rados.0.smithi125.stdout:2367: done (0 left) 2021-03-24T22:03:18.435 INFO:tasks.rados.rados.0.smithi125.stdout:2368: read oid 18 snap -1 2021-03-24T22:03:18.435 INFO:tasks.rados.rados.0.smithi125.stdout:2368: expect (ObjNum 815 snap 265 seq_num 815) 2021-03-24T22:03:18.435 INFO:tasks.rados.rados.0.smithi125.stdout:2369: delete oid 8 current snap is 266 2021-03-24T22:03:18.437 INFO:tasks.rados.rados.0.smithi125.stdout:2369: done (1 left) 2021-03-24T22:03:18.438 INFO:tasks.rados.rados.0.smithi125.stdout:2370: rollback oid 19 current snap is 266 2021-03-24T22:03:18.438 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 19 to 219 2021-03-24T22:03:18.438 INFO:tasks.rados.rados.0.smithi125.stdout:2371: rollback oid 33 current snap is 266 2021-03-24T22:03:18.438 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 33 to 258 2021-03-24T22:03:18.438 INFO:tasks.rados.rados.0.smithi125.stdout:2372: write oid 14 current snap is 266 2021-03-24T22:03:18.439 INFO:tasks.rados.rados.0.smithi125.stdout:2372: seq_num 818 ranges {548126=403453,1417447=744619,2680361=447683,3680450=67676} 2021-03-24T22:03:18.443 INFO:tasks.rados.rados.0.smithi125.stdout:2372: writing smithi12556470-14 from 548126 to 951579 tid 1 2021-03-24T22:03:18.451 INFO:tasks.rados.rados.0.smithi125.stdout:2372: writing smithi12556470-14 from 1417447 to 2162066 tid 2 2021-03-24T22:03:18.456 INFO:tasks.rados.rados.0.smithi125.stdout:2372: writing smithi12556470-14 from 2680361 to 3128044 tid 3 2021-03-24T22:03:18.459 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:18.460 INFO:tasks.rados.rados.0.smithi125.stdout:2372: writing smithi12556470-14 from 3680450 to 3748126 tid 4 2021-03-24T22:03:18.460 INFO:tasks.rados.rados.0.smithi125.stdout:2373: snap_create 2021-03-24T22:03:18.461 INFO:tasks.rados.rados.0.smithi125.stdout:2370: finishing rollback tid 0 to smithi12556470-19 2021-03-24T22:03:18.461 INFO:tasks.rados.rados.0.smithi125.stdout:2371: finishing rollback tid 1 to smithi12556470-33 2021-03-24T22:03:18.461 INFO:tasks.rados.rados.0.smithi125.stdout:2371: finishing rollback tid 2 to smithi12556470-33 2021-03-24T22:03:18.461 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 33 v 929 (ObjNum 764 snap 249 seq_num 764) dirty exists 2021-03-24T22:03:18.461 INFO:tasks.rados.rados.0.smithi125.stdout:2370: finishing rollback tid 1 to smithi12556470-19 2021-03-24T22:03:18.462 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 19 v 573 (ObjNum 1 snap 21916 seq_num 1116466112) dirty dne 2021-03-24T22:03:18.462 INFO:tasks.rados.rados.0.smithi125.stdout:2372: finishing write tid 1 to smithi12556470-14 2021-03-24T22:03:18.462 INFO:tasks.rados.rados.0.smithi125.stdout:2368: expect (ObjNum 815 snap 265 seq_num 815) 2021-03-24T22:03:18.510 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:18.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:18.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:03:18.558 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:18 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:03:18.558 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:18 smithi125 conmon[19936]: 03:16.924971+0000 mgr.y (mgr.14144) 586 : cluster [DBG] pgmap v1102: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 474 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.6 MiB/s rd, 3.2 MiB/s wr, 12 op/s; 40/1662 objects misplaced (2.407%) 2021-03-24T22:03:18.559 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:18 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:03:18.559 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:18 smithi125 conmon[19936]: T22:03:17.401418+0000 2021-03-24T22:03:18.559 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:18 smithi125 conmon[19936]: mon.a (mon.0) 1694 : 2021-03-24T22:03:18.559 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:18 smithi125 conmon[19936]: cluster [DBG] osdmap e627: 8 total, 8 up, 8 in 2021-03-24T22:03:18.559 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:18 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:03:18.560 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:18 smithi125 conmon[19936]: 03-24T22:03:17.401825+0000 2021-03-24T22:03:18.560 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:18 smithi125 conmon[19936]: osd.3 (osd.3) 128 : cluster [DBG] 2021-03-24T22:03:18.560 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:18 smithi125 conmon[19936]: purged_snaps scrub starts 2021-03-24T22:03:18.560 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:18 smithi125 conmon[19936]: cluster 2021-03-24T22 2021-03-24T22:03:18.561 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:18 smithi125 conmon[19936]: :03:17.492311+0000 osd.3 (osd. 2021-03-24T22:03:18.561 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:18 smithi125 conmon[19936]: 3) 129 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:03:18.561 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:18 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:03:18.561 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:18 smithi125 conmon[24927]: 03:16.924971+0000 mgr.y (mgr.14144) 586 : cluster [DBG] pgmap v1102: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 474 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.6 MiB/s rd, 3.2 MiB/s wr, 12 op/s; 40/1662 objects misplaced (2.407%) 2021-03-24T22:03:18.562 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:18 smithi125 conmon[24927]: cluster 2021-03-24T22:03:17.401418 2021-03-24T22:03:18.562 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:18 smithi125 conmon[24927]: +0000 mon.a (mon.0) 1694 : cluster [DBG] 2021-03-24T22:03:18.562 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:18 smithi125 conmon[24927]: osdmap e627: 8 total, 8 up, 8 in 2021-03-24T22:03:18.562 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:18 smithi125 conmon[24927]: cluster 2021-03-24T22:03: 2021-03-24T22:03:18.563 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:18 smithi125 conmon[24927]: 17.401825+0000 osd.3 (osd.3) 128 2021-03-24T22:03:18.563 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:18 smithi125 conmon[24927]: : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:03:18.563 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:18 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T22:03:18.563 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:18 smithi125 conmon[24927]: 22:03:17.492311+0000 osd.3 (osd. 2021-03-24T22:03:18.563 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:18 smithi125 conmon[24927]: 3) 129 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:03:18.617 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:18.672 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:18.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:18.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:03:18.749 INFO:tasks.rados.rados.0.smithi125.stdout:2372: finishing write tid 2 to smithi12556470-14 2021-03-24T22:03:18.749 INFO:tasks.rados.rados.0.smithi125.stdout:2372: finishing write tid 3 to smithi12556470-14 2021-03-24T22:03:18.749 INFO:tasks.rados.rados.0.smithi125.stdout:2372: finishing write tid 4 to smithi12556470-14 2021-03-24T22:03:18.749 INFO:tasks.rados.rados.0.smithi125.stdout:2372: finishing write tid 5 to smithi12556470-14 2021-03-24T22:03:18.750 INFO:tasks.rados.rados.0.smithi125.stdout:2372: finishing write tid 6 to smithi12556470-14 2021-03-24T22:03:18.750 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 14 v 1360 (ObjNum 818 snap 266 seq_num 818) dirty exists 2021-03-24T22:03:18.750 INFO:tasks.rados.rados.0.smithi125.stdout:2372: left oid 14 (ObjNum 818 snap 266 seq_num 818) 2021-03-24T22:03:18.779 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:18.793 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:18 smithi142 conmon[20125]: cluster 2021-03-24T22:03 2021-03-24T22:03:18.793 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:18 smithi142 conmon[20125]: :16.924971+0000 mgr.y (mgr.14144) 586 : cluster [DBG] pgmap v1102: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 474 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.6 MiB/s rd, 3.2 MiB/s wr, 12 op/s; 40/1662 objects misplaced (2.407%) 2021-03-24T22:03:18.793 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:18 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:03:18.794 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:18 smithi142 conmon[20125]: 24T22:03:17.401418+0000 mon.a (mon.0) 1694 : cluster 2021-03-24T22:03:18.794 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:18 smithi142 conmon[20125]: [DBG] osdmap e627: 8 total, 8 up, 8 in 2021-03-24T22:03:18.794 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:18 smithi142 conmon[20125]: cluster 2021-03-24T22:03: 2021-03-24T22:03:18.794 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:18 smithi142 conmon[20125]: 17.401825+0000 osd.3 (osd.3) 128 2021-03-24T22:03:18.794 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:18 smithi142 conmon[20125]: : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:03:18.795 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:18 smithi142 conmon[20125]: cluster 2021-03-24T22 2021-03-24T22:03:18.795 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:18 smithi142 conmon[20125]: :03:17.492311+0000 osd.3 (osd. 2021-03-24T22:03:18.795 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:18 smithi142 conmon[20125]: 3) 129 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:03:18.840 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:18.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:18.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:03:18.947 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:19.002 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:19.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:19.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:03:19.109 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:19.168 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:19.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:19.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:19.273 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:19.327 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:19.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:19.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:03:19.430 INFO:tasks.rados.rados.0.smithi125.stdout:2368: done (4 left) 2021-03-24T22:03:19.430 INFO:tasks.rados.rados.0.smithi125.stdout:2370: done (3 left) 2021-03-24T22:03:19.431 INFO:tasks.rados.rados.0.smithi125.stdout:2371: done (2 left) 2021-03-24T22:03:19.431 INFO:tasks.rados.rados.0.smithi125.stdout:2372: done (1 left) 2021-03-24T22:03:19.431 INFO:tasks.rados.rados.0.smithi125.stdout:2373: done (0 left) 2021-03-24T22:03:19.431 INFO:tasks.rados.rados.0.smithi125.stdout:2374: copy_from oid 33 from oid 42 current snap is 267 2021-03-24T22:03:19.432 INFO:tasks.rados.rados.0.smithi125.stdout:2375: snap_remove snap 224 2021-03-24T22:03:19.433 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:19.482 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:19.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:19.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:03:19.584 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:19.644 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:19.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:19.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:03:19.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:19 smithi125 conmon[24927]: cluster 2021-03-24T22:03:18. 2021-03-24T22:03:19.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:19 smithi125 conmon[24927]: 412883+0000 mon.a (mon.0) 1695 : cluster [DBG] osdmap e628: 8 total, 8 up, 8 in 2021-03-24T22:03:19.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:19 smithi125 conmon[19936]: cluster 2021-03-24T22:03:18.412883+0000 mon.a (mon.0) 2021-03-24T22:03:19.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:19 smithi125 conmon[19936]: 1695 : cluster [DBG] osdmap e628: 8 total, 8 up, 8 in 2021-03-24T22:03:19.749 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:19.762 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:19 smithi142 conmon[20125]: cluster 2021-03-24T22:03:18.412883+0000 2021-03-24T22:03:19.762 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:19 smithi142 conmon[20125]: mon.a (mon.0) 1695 : cluster [DBG] osdmap e628: 8 total, 8 up, 8 in 2021-03-24T22:03:19.803 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:19.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:19.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:03:19.910 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:19.971 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:19.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:19.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:03:20.078 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:20.083 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-24T22:03:20.137 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:20.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:20.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:03:20.215 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:03:20.244 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:20.297 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:20.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:20.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:03:20.332 INFO:tasks.rados.rados.0.smithi125.stdout:2375: done (1 left) 2021-03-24T22:03:20.333 INFO:tasks.rados.rados.0.smithi125.stdout:2376: delete oid 21 current snap is 267 2021-03-24T22:03:20.333 INFO:tasks.rados.rados.0.smithi125.stdout:2374: finishing copy_from to smithi12556470-33 2021-03-24T22:03:20.333 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 33 v 503 (ObjNum 736 snap 235 seq_num 736) dirty exists 2021-03-24T22:03:20.334 INFO:tasks.rados.rados.0.smithi125.stdout:2374: finishing copy_from racing read to smithi12556470-33 2021-03-24T22:03:20.347 INFO:tasks.rados.rados.0.smithi125.stdout:2374: done (1 left) 2021-03-24T22:03:20.348 INFO:tasks.rados.rados.0.smithi125.stdout:2376: done (0 left) 2021-03-24T22:03:20.348 INFO:tasks.rados.rados.0.smithi125.stdout:2377: read oid 30 snap -1 2021-03-24T22:03:20.348 INFO:tasks.rados.rados.0.smithi125.stdout:2377: expect (ObjNum 764 snap 249 seq_num 764) 2021-03-24T22:03:20.348 INFO:tasks.rados.rados.0.smithi125.stdout:2378: delete oid 15 current snap is 267 2021-03-24T22:03:20.357 INFO:tasks.rados.rados.0.smithi125.stdout:2378: done (1 left) 2021-03-24T22:03:20.357 INFO:tasks.rados.rados.0.smithi125.stdout:2379: read oid 49 snap -1 2021-03-24T22:03:20.357 INFO:tasks.rados.rados.0.smithi125.stdout:2379: expect (ObjNum 795 snap 257 seq_num 795) 2021-03-24T22:03:20.358 INFO:tasks.rados.rados.0.smithi125.stdout:2380: delete oid 43 current snap is 267 2021-03-24T22:03:20.363 INFO:tasks.rados.rados.0.smithi125.stdout:2377: expect (ObjNum 764 snap 249 seq_num 764) 2021-03-24T22:03:20.403 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:20.460 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:20.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:20.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:03:20.555 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:03:20.556 INFO:tasks.rados.rados.0.smithi125.stdout:2379: expect (ObjNum 795 snap 257 seq_num 795) 2021-03-24T22:03:20.565 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:20.623 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:20.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:20.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:03:20.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:20 smithi125 conmon[19936]: cluster 2021-03-24T22:03:18.925706+0000 mgr.y (mgr.14144) 587 : cluster [DBG] pgmap v1105: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 474 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.6 MiB/s rd, 380 KiB/s wr, 3 op/s; 40/1662 objects misplaced (2.407%) 2021-03-24T22:03:20.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:20 smithi125 conmon[19936]: cluster 2021-03-24T22:03:19.423871+0000 mon.a (mon.0) 1696 : cluster [DBG] osdmap e629: 8 total, 8 up, 8 in 2021-03-24T22:03:20.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:20 smithi125 conmon[19936]: cluster 2021-03-24T22:03:20.328225+0000 mon.a (mon.0) 1697 : cluster [DBG] osdmap e630: 8 total, 8 up, 8 in 2021-03-24T22:03:20.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:20 smithi125 conmon[24927]: cluster 2021-03-24T22:03:18.925706+0000 mgr.y (mgr.14144) 587 : 2021-03-24T22:03:20.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:20 smithi125 conmon[24927]: cluster [DBG] pgmap v1105: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 474 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.6 MiB/s rd, 380 KiB/s wr, 3 op/s; 40/1662 objects misplaced (2.407%) 2021-03-24T22:03:20.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:20 smithi125 conmon[24927]: cluster 2021-03-24T22:03:19.423871+0000 mon.a (mon.0) 1696 : cluster [DBG] osdmap e629: 8 total, 8 up, 8 in 2021-03-24T22:03:20.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:20 smithi125 conmon[24927]: cluster 2021-03-24T22:03:20.328225+0000 mon.a (mon.0) 1697 : cluster [DBG] osdmap e630: 8 total, 8 up, 8 in 2021-03-24T22:03:20.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:20 smithi125 conmon[24927]: 2021-03-24T22:03:20.730 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:20.743 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:20 smithi142 conmon[20125]: cluster 2021-03-24T22:03:18.925706+0000 mgr.y (mgr.14144) 587 : cluster [DBG] 2021-03-24T22:03:20.744 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:20 smithi142 conmon[20125]: pgmap v1105: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 474 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.6 MiB/s rd, 380 KiB/s wr, 3 op/s; 40/1662 objects misplaced (2.407%) 2021-03-24T22:03:20.744 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:20 smithi142 conmon[20125]: cluster 2021-03-24T22:03:19.423871+0000 mon.a (mon.0) 1696 : cluster [DBG] osdmap e629: 8 total, 8 up, 8 in 2021-03-24T22:03:20.744 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:20 smithi142 conmon[20125]: cluster 2021-03-24T22:03:20.745 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:20 smithi142 conmon[20125]: 2021-03-24T22:03:20.328225+0000 mon.a (mon.0) 1697 : cluster [DBG] osdmap e630: 8 total, 8 up, 8 in 2021-03-24T22:03:20.750 INFO:tasks.rados.rados.0.smithi125.stdout:2377: done (2 left) 2021-03-24T22:03:20.750 INFO:tasks.rados.rados.0.smithi125.stdout:2379: done (1 left) 2021-03-24T22:03:20.750 INFO:tasks.rados.rados.0.smithi125.stdout:2380: done (0 left) 2021-03-24T22:03:20.750 INFO:tasks.rados.rados.0.smithi125.stdout:2381: read oid 19 snap -1 2021-03-24T22:03:20.751 INFO:tasks.rados.rados.0.smithi125.stdout:2381: expect deleted 2021-03-24T22:03:20.751 INFO:tasks.rados.rados.0.smithi125.stdout:2382: snap_create 2021-03-24T22:03:20.787 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:20.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:20.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:03:20.872 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T22:03:20.890 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T22:03:20.893 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:20.913 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T22:03:20.940 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T22:03:20.951 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:20.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:20.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:03:20.970 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T22:03:20.996 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T22:03:21.019 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T22:03:21.043 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T22:03:21.044 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-24T22:03:21.045 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-24T22:03:21.045 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-24T22:03:21.045 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-24T22:03:21.045 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-24T22:03:21.046 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-24T22:03:21.046 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-24T22:03:21.046 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-24T22:03:21.056 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:21.112 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:21.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:21.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:03:21.218 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:21.274 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:21.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:21.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:03:21.381 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:21.440 INFO:tasks.rados.rados.0.smithi125.stdout:2381: done (1 left) 2021-03-24T22:03:21.441 INFO:tasks.rados.rados.0.smithi125.stdout:2382: done (0 left) 2021-03-24T22:03:21.441 INFO:tasks.rados.rados.0.smithi125.stdout:2383: write oid 33 current snap is 268 2021-03-24T22:03:21.442 INFO:tasks.rados.rados.0.smithi125.stdout:2383: seq_num 822 ranges {615354=420802,1651375=751957,3190053=492771} 2021-03-24T22:03:21.445 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:21.447 INFO:tasks.rados.rados.0.smithi125.stdout:2383: writing smithi12556470-33 from 615354 to 1036156 tid 1 2021-03-24T22:03:21.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:21.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:03:21.455 INFO:tasks.rados.rados.0.smithi125.stdout:2383: writing smithi12556470-33 from 1651375 to 2403332 tid 2 2021-03-24T22:03:21.462 INFO:tasks.rados.rados.0.smithi125.stdout:2383: writing smithi12556470-33 from 3190053 to 3682824 tid 3 2021-03-24T22:03:21.463 INFO:tasks.rados.rados.0.smithi125.stdout:2384: snap_create 2021-03-24T22:03:21.463 INFO:tasks.rados.rados.0.smithi125.stdout:2383: finishing write tid 1 to smithi12556470-33 2021-03-24T22:03:21.463 INFO:tasks.rados.rados.0.smithi125.stdout:2383: finishing write tid 2 to smithi12556470-33 2021-03-24T22:03:21.466 INFO:tasks.rados.rados.0.smithi125.stdout:2383: finishing write tid 3 to smithi12556470-33 2021-03-24T22:03:21.467 INFO:tasks.rados.rados.0.smithi125.stdout:2383: finishing write tid 4 to smithi12556470-33 2021-03-24T22:03:21.467 INFO:tasks.rados.rados.0.smithi125.stdout:2383: finishing write tid 5 to smithi12556470-33 2021-03-24T22:03:21.468 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 33 v 940 (ObjNum 822 snap 268 seq_num 822) dirty exists 2021-03-24T22:03:21.468 INFO:tasks.rados.rados.0.smithi125.stdout:2383: left oid 33 (ObjNum 822 snap 268 seq_num 822) 2021-03-24T22:03:21.551 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:21.606 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:21.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:21.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:03:21.713 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:21.725 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:21 smithi125 conmon[24927]: audit 2021-03-24T22: 2021-03-24T22:03:21.725 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:21 smithi125 conmon[24927]: 03:20.849231+0000 mon.c (mon.2) 96 : audit [DBG] from='client.? v1:172.21.15.125:0/4283852162' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:03:21.726 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:21 smithi125 conmon[19936]: audit 2021-03-24T22:03: 2021-03-24T22:03:21.726 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:21 smithi125 conmon[19936]: 20.849231+0000 mon.c (mon.2) 96 : audit [DBG] from='client.? v1:172.21.15.125:0/4283852162' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:03:21.769 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:21.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:21.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:03:21.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:21 smithi142 conmon[20125]: audit 2021-03-24T22:03 2021-03-24T22:03:21.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:21 smithi142 conmon[20125]: :20.849231+0000 mon.c (mon.2) 96 : audit [DBG] from='client.? v1:172.21.15.125:0/4283852162' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:03:21.873 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:21.931 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:21.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:21.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:03:22.037 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:22.093 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:22.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:22.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:03:22.201 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:22.258 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:22.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:22.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:03:22.364 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:22.418 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:22.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:22.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:03:22.442 INFO:tasks.rados.rados.0.smithi125.stdout:2383: done (1 left) 2021-03-24T22:03:22.442 INFO:tasks.rados.rados.0.smithi125.stdout:2384: done (0 left) 2021-03-24T22:03:22.443 INFO:tasks.rados.rados.0.smithi125.stdout:2385: snap_remove snap 235 2021-03-24T22:03:22.524 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:22.583 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:22.590 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:22 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:03:22.590 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:22 smithi125 conmon[24927]: 03:20.926275+0000 mgr.y (mgr.14144) 588 : cluster [DBG] pgmap v1108: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 475 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 40/1653 objects misplaced (2.420%) 2021-03-24T22:03:22.590 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:22 smithi125 conmon[24927]: cluster 2021-03-24T22:03:21 2021-03-24T22:03:22.590 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:22 smithi125 conmon[24927]: .434938+0000 mon.a (mon.0) 1698 : cluster 2021-03-24T22:03:22.591 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:22 smithi125 conmon[24927]: [DBG] osdmap e631: 8 total, 8 up, 8 in 2021-03-24T22:03:22.591 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:22 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:03:22.591 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:22 smithi125 conmon[24927]: -03-24T22:03:21.852588+0000 osd.6 (osd. 2021-03-24T22:03:22.591 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:22 smithi125 conmon[24927]: 6) 48 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:03:22.591 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:22 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:03:22.592 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:22 smithi125 conmon[24927]: 03:21.854954+0000 osd.6 (osd.6) 2021-03-24T22:03:22.592 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:22 smithi125 conmon[24927]: 49 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:03:22.593 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:22 smithi125 conmon[19936]: cluster 2021-03-24T22:03:20.926275+0000 2021-03-24T22:03:22.593 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:22 smithi125 conmon[19936]: mgr.y (mgr.14144) 588 : cluster [DBG] pgmap v1108: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 475 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 40/1653 objects misplaced (2.420%) 2021-03-24T22:03:22.593 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:22 smithi125 conmon[19936]: cluster 2021-03-24T22:03:21.434938+0000 mon.a 2021-03-24T22:03:22.593 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:22 smithi125 conmon[19936]: (mon.0) 1698 : cluster [DBG] osdmap e631: 8 total, 8 up, 8 in 2021-03-24T22:03:22.594 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:22 smithi125 conmon[19936]: cluster 2021-03-24T22:03:21.852588+0000 2021-03-24T22:03:22.594 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:22 smithi125 conmon[19936]: osd.6 (osd.6) 48 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:03:22.594 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:22 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T22:03:22.595 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:22 smithi125 conmon[19936]: -24T22:03:21.854954+0000 osd.6 (osd. 2021-03-24T22:03:22.595 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:22 smithi125 conmon[19936]: 6) 49 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:03:22.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:22.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:22.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:22 smithi142 conmon[20125]: cluster 2021-03-24T22:03: 2021-03-24T22:03:22.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:22 smithi142 conmon[20125]: 20.926275+0000 mgr.y (mgr.14144) 588 : cluster [DBG] pgmap v1108: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 475 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 40/1653 objects misplaced (2.420%) 2021-03-24T22:03:22.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:22 smithi142 conmon[20125]: cluster 2021-03-24T22:03:21.434938 2021-03-24T22:03:22.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:22 smithi142 conmon[20125]: +0000 mon.a (mon.0) 1698 : cluster [DBG] osdmap e631: 8 total, 8 up, 8 in 2021-03-24T22:03:22.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:22 smithi142 conmon[20125]: cluster 2021-03-24T22: 2021-03-24T22:03:22.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:22 smithi142 conmon[20125]: 03:21.852588+0000 osd.6 (osd.6) 48 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:03:22.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:22 smithi142 conmon[20125]: cluster 2021-03-24T22:03:21 2021-03-24T22:03:22.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:22 smithi142 conmon[20125]: .854954+0000 osd.6 (osd.6) 49 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:03:22.697 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:22.754 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:22.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:22.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:03:22.861 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:22.921 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:22.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:22.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:03:23.026 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:23.074 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-24T22:03:23.077 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2021-03-24T22:03:23.083 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:23.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:23.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:03:23.187 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:23.241 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:23.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:23.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:03:23.346 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:23.402 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:23.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:23.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:03:23.457 INFO:tasks.rados.rados.0.smithi125.stdout:2385: done (0 left) 2021-03-24T22:03:23.458 INFO:tasks.rados.rados.0.smithi125.stdout:2386: delete oid 22 current snap is 269 2021-03-24T22:03:23.466 INFO:tasks.rados.rados.0.smithi125.stdout:2386: done (0 left) 2021-03-24T22:03:23.466 INFO:tasks.rados.rados.0.smithi125.stdout:2387: read oid 18 snap 264 2021-03-24T22:03:23.466 INFO:tasks.rados.rados.0.smithi125.stdout:2387: expect (ObjNum 676 snap 210 seq_num 676) 2021-03-24T22:03:23.466 INFO:tasks.rados.rados.0.smithi125.stdout:2388: read oid 26 snap 246 2021-03-24T22:03:23.467 INFO:tasks.rados.rados.0.smithi125.stdout:2388: expect (ObjNum 756 snap 244 seq_num 756) 2021-03-24T22:03:23.467 INFO:tasks.rados.rados.0.smithi125.stdout:2389: read oid 31 snap -1 2021-03-24T22:03:23.467 INFO:tasks.rados.rados.0.smithi125.stdout:2389: expect (ObjNum 718 snap 229 seq_num 718) 2021-03-24T22:03:23.467 INFO:tasks.rados.rados.0.smithi125.stdout:2390: read oid 6 snap -1 2021-03-24T22:03:23.467 INFO:tasks.rados.rados.0.smithi125.stdout:2390: expect (ObjNum 801 snap 259 seq_num 801) 2021-03-24T22:03:23.468 INFO:tasks.rados.rados.0.smithi125.stdout:2391: copy_from oid 2 from oid 1 current snap is 269 2021-03-24T22:03:23.468 INFO:tasks.rados.rados.0.smithi125.stdout:2392: read oid 40 snap 237 2021-03-24T22:03:23.468 INFO:tasks.rados.rados.0.smithi125.stdout:2392: expect (ObjNum 689 snap 215 seq_num 689) 2021-03-24T22:03:23.468 INFO:tasks.rados.rados.0.smithi125.stdout:2393: copy_from oid 8 from oid 4 current snap is 269 2021-03-24T22:03:23.469 INFO:tasks.rados.rados.0.smithi125.stdout:2394: read oid 3 snap -1 2021-03-24T22:03:23.469 INFO:tasks.rados.rados.0.smithi125.stdout:2394: expect deleted 2021-03-24T22:03:23.469 INFO:tasks.rados.rados.0.smithi125.stdout:2395: snap_create 2021-03-24T22:03:23.473 INFO:tasks.rados.rados.0.smithi125.stdout:2388: expect (ObjNum 756 snap 244 seq_num 756) 2021-03-24T22:03:23.509 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:23.551 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:23.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:23.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:23.579 INFO:tasks.rados.rados.0.smithi125.stdout:2389: expect (ObjNum 718 snap 229 seq_num 718) 2021-03-24T22:03:23.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:23 smithi142 conmon[20125]: cluster 2021-03-24T22: 2021-03-24T22:03:23.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:23 smithi142 conmon[20125]: 03:22.439581+0000 mon.a (mon.0) 1699 : cluster [DBG] osdmap e632: 8 total, 8 up, 8 in 2021-03-24T22:03:23.645 INFO:tasks.rados.rados.0.smithi125.stdout:2390: expect (ObjNum 801 snap 259 seq_num 801) 2021-03-24T22:03:23.658 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:23.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:23 smithi125 conmon[24927]: cluster 2021-03-24T22:03: 2021-03-24T22:03:23.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:23 smithi125 conmon[24927]: 22.439581+0000 mon.a (mon.0) 1699 : cluster [DBG] osdmap e632: 8 total, 8 up, 8 in 2021-03-24T22:03:23.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:23 smithi125 conmon[19936]: cluster 2021-03-24T22:03:22.439581+0000 mon.a (mon.0) 1699 : cluster [DBG] osdmap e632: 8 total, 8 up, 8 in 2021-03-24T22:03:23.717 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:23.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:23.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:03:23.796 INFO:tasks.rados.rados.0.smithi125.stdout:2387: expect (ObjNum 676 snap 210 seq_num 676) 2021-03-24T22:03:23.821 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:23.871 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:23.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:23.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:23.977 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:23.995 INFO:tasks.rados.rados.0.smithi125.stdout:2392: expect (ObjNum 689 snap 215 seq_num 689) 2021-03-24T22:03:24.026 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:24.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:24.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:03:24.132 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:24.187 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:24.190 INFO:tasks.rados.rados.0.smithi125.stdout:2391: finishing copy_from to smithi12556470-2 2021-03-24T22:03:24.190 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 2 v 827 (ObjNum 808 snap 265 seq_num 808) dirty exists 2021-03-24T22:03:24.191 INFO:tasks.rados.rados.0.smithi125.stdout:2391: finishing copy_from racing read to smithi12556470-2 2021-03-24T22:03:24.191 INFO:tasks.rados.rados.0.smithi125.stdout:2393: finishing copy_from to smithi12556470-8 2021-03-24T22:03:24.191 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 8 v 549 (ObjNum 809 snap 265 seq_num 809) dirty exists 2021-03-24T22:03:24.191 INFO:tasks.rados.rados.0.smithi125.stdout:2393: finishing copy_from racing read to smithi12556470-8 2021-03-24T22:03:24.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:24.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:03:24.294 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:24.345 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:24.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:24.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:24.450 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:24.461 INFO:tasks.rados.rados.0.smithi125.stdout:2387: done (8 left) 2021-03-24T22:03:24.461 INFO:tasks.rados.rados.0.smithi125.stdout:2388: done (7 left) 2021-03-24T22:03:24.461 INFO:tasks.rados.rados.0.smithi125.stdout:2389: done (6 left) 2021-03-24T22:03:24.462 INFO:tasks.rados.rados.0.smithi125.stdout:2390: done (5 left) 2021-03-24T22:03:24.462 INFO:tasks.rados.rados.0.smithi125.stdout:2391: done (4 left) 2021-03-24T22:03:24.462 INFO:tasks.rados.rados.0.smithi125.stdout:2392: done (3 left) 2021-03-24T22:03:24.463 INFO:tasks.rados.rados.0.smithi125.stdout:2393: done (2 left) 2021-03-24T22:03:24.463 INFO:tasks.rados.rados.0.smithi125.stdout:2394: done (1 left) 2021-03-24T22:03:24.463 INFO:tasks.rados.rados.0.smithi125.stdout:2395: done (0 left) 2021-03-24T22:03:24.464 INFO:tasks.rados.rados.0.smithi125.stdout:2396: read oid 28 snap 260 2021-03-24T22:03:24.464 INFO:tasks.rados.rados.0.smithi125.stdout:2396: expect deleted 2021-03-24T22:03:24.464 INFO:tasks.rados.rados.0.smithi125.stdout:2397: rollback oid 6 current snap is 270 2021-03-24T22:03:24.465 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 6 to 269 2021-03-24T22:03:24.465 INFO:tasks.rados.rados.0.smithi125.stdout:2398: write (excl) oid 29 current snap is 270 2021-03-24T22:03:24.466 INFO:tasks.rados.rados.0.smithi125.stdout:2398: seq_num 824 ranges {648968=571300,1651986=639273,2728546=631668} 2021-03-24T22:03:24.468 INFO:tasks.rados.rados.0.smithi125.stdout:2398: writing smithi12556470-29 from 648968 to 1220268 tid 1 2021-03-24T22:03:24.474 INFO:tasks.rados.rados.0.smithi125.stdout:2398: writing smithi12556470-29 from 1651986 to 2291259 tid 2 2021-03-24T22:03:24.481 INFO:tasks.rados.rados.0.smithi125.stdout:2398: writing smithi12556470-29 from 2728546 to 3360214 tid 3 2021-03-24T22:03:24.481 INFO:tasks.rados.rados.0.smithi125.stdout:2399: write oid 36 current snap is 270 2021-03-24T22:03:24.482 INFO:tasks.rados.rados.0.smithi125.stdout:2399: seq_num 825 ranges {665775=646549,2110468=499284,3155969=442940} 2021-03-24T22:03:24.490 INFO:tasks.rados.rados.0.smithi125.stdout:2399: writing smithi12556470-36 from 665775 to 1312324 tid 1 2021-03-24T22:03:24.501 INFO:tasks.rados.rados.0.smithi125.stdout:2399: writing smithi12556470-36 from 2110468 to 2609752 tid 2 2021-03-24T22:03:24.511 INFO:tasks.rados.rados.0.smithi125.stdout:2399: writing smithi12556470-36 from 3155969 to 3598909 tid 3 2021-03-24T22:03:24.511 INFO:tasks.rados.rados.0.smithi125.stdout:2397: finishing rollback tid 0 to smithi12556470-6 2021-03-24T22:03:24.512 INFO:tasks.rados.rados.0.smithi125.stdout:2397: finishing rollback tid 1 to smithi12556470-6 2021-03-24T22:03:24.512 INFO:tasks.rados.rados.0.smithi125.stdout:2397: finishing rollback tid 2 to smithi12556470-6 2021-03-24T22:03:24.512 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 6 v 544 (ObjNum 801 snap 259 seq_num 801) dirty exists 2021-03-24T22:03:24.512 INFO:tasks.rados.rados.0.smithi125.stdout:2398: finishing write tid 1 to smithi12556470-29 2021-03-24T22:03:24.513 INFO:tasks.rados.rados.0.smithi125.stdout:2398: finishing write tid 2 to smithi12556470-29 2021-03-24T22:03:24.513 INFO:tasks.rados.rados.0.smithi125.stdout:2398: finishing write tid 3 to smithi12556470-29 2021-03-24T22:03:24.513 INFO:tasks.rados.rados.0.smithi125.stdout:2398: finishing write tid 4 to smithi12556470-29 2021-03-24T22:03:24.513 INFO:tasks.rados.rados.0.smithi125.stdout:2398: finishing write tid 5 to smithi12556470-29 2021-03-24T22:03:24.514 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 29 v 395 (ObjNum 824 snap 270 seq_num 824) dirty exists 2021-03-24T22:03:24.514 INFO:tasks.rados.rados.0.smithi125.stdout:2398: left oid 29 (ObjNum 824 snap 270 seq_num 824) 2021-03-24T22:03:24.514 INFO:tasks.rados.rados.0.smithi125.stdout:2399: finishing write tid 1 to smithi12556470-36 2021-03-24T22:03:24.514 INFO:tasks.rados.rados.0.smithi125.stdout:2399: finishing write tid 2 to smithi12556470-36 2021-03-24T22:03:24.515 INFO:tasks.rados.rados.0.smithi125.stdout:2396: done (3 left) 2021-03-24T22:03:24.515 INFO:tasks.rados.rados.0.smithi125.stdout:2397: done (2 left) 2021-03-24T22:03:24.516 INFO:tasks.rados.rados.0.smithi125.stdout:2398: done (1 left) 2021-03-24T22:03:24.516 INFO:tasks.rados.rados.0.smithi125.stdout:2400: delete oid 49 current snap is 270 2021-03-24T22:03:24.516 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:24.517 INFO:tasks.rados.rados.0.smithi125.stdout:2400: done (1 left) 2021-03-24T22:03:24.517 INFO:tasks.rados.rados.0.smithi125.stdout:2401: read oid 41 snap -1 2021-03-24T22:03:24.518 INFO:tasks.rados.rados.0.smithi125.stdout:2401: expect (ObjNum 791 snap 257 seq_num 791) 2021-03-24T22:03:24.518 INFO:tasks.rados.rados.0.smithi125.stdout:2402: snap_remove snap 255 2021-03-24T22:03:24.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:24.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:03:24.621 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:24.635 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:24 smithi125 conmon[19936]: audit 2021-03-24T22:03:22.766035+0000 mgr.y (mgr 2021-03-24T22:03:24.635 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:24 smithi125 conmon[19936]: .14144) 589 : audit [DBG] from='client.35576 v1:172.21.15.125:0/3835472975' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T22:03:24.636 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:24 smithi125 conmon[19936]: cluster 2021-03-24T22:03:22.927052+0000 mgr.y ( 2021-03-24T22:03:24.636 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:24 smithi125 conmon[19936]: mgr.14144) 590 : cluster [DBG] pgmap v1111: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 473 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 40/1650 objects misplaced (2.424%) 2021-03-24T22:03:24.636 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:24 smithi125 conmon[19936]: cluster 2021-03-24T22:03:24.636 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:24 smithi125 conmon[19936]: 2021-03-24T22:03:23.454454+0000 mon.a (mon.0) 1700 : cluster 2021-03-24T22:03:24.637 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:24 smithi125 conmon[19936]: [DBG] osdmap e633: 8 total, 8 up, 8 in 2021-03-24T22:03:24.637 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:24 smithi125 conmon[24927]: audit 2021-03-24T22:03:24.637 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:24 smithi125 conmon[24927]: 2021-03-24T22:03:24.638 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:24 smithi125 conmon[24927]: 2021-03-24T22:03:22.766035 2021-03-24T22:03:24.638 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:24 smithi125 conmon[24927]: +0000 mgr.y (mgr.14144) 589 : audit [DBG] from='client.35576 v1:172.21.15.125:0/3835472975' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T22:03:24.638 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:24 smithi125 conmon[24927]: cluster 2021-03-24T22:03 2021-03-24T22:03:24.638 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:24 smithi125 conmon[24927]: :22.927052+0000 mgr.y (mgr.14144) 590 : cluster 2021-03-24T22:03:24.639 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:24 smithi125 conmon[24927]: [DBG] pgmap v1111: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 473 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 40/1650 objects misplaced (2.424%) 2021-03-24T22:03:24.639 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:24 smithi125 conmon[24927]: cluster 2021-03-24T22:03:23.454454+0000 2021-03-24T22:03:24.639 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:24 smithi125 conmon[24927]: mon.a (mon.0) 1700 : cluster [DBG] osdmap e633: 8 total, 8 up, 8 in 2021-03-24T22:03:24.677 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:24.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:24.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:03:24.784 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:24.843 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:24.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:24.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:24.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:24 smithi142 conmon[20125]: audit 2021-03-24T22:03 2021-03-24T22:03:24.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:24 smithi142 conmon[20125]: :22.766035+0000 mgr.y (mgr.14144) 589 : audit [DBG] from='client.35576 v1:172.21.15.125:0/3835472975' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T22:03:24.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:24 smithi142 conmon[20125]: cluster 2021-03-24T22:03:22 2021-03-24T22:03:24.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:24 smithi142 conmon[20125]: .927052+0000 mgr.y (mgr.14144) 590 : cluster [DBG] pgmap v1111: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 473 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 40/1650 objects misplaced (2.424%) 2021-03-24T22:03:24.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:24 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:03:24.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:24 smithi142 conmon[20125]: 24T22:03:23.454454+0000 2021-03-24T22:03:24.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:24 smithi142 conmon[20125]: mon.a (mon.0) 1700 : cluster 2021-03-24T22:03:24.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:24 smithi142 conmon[20125]: [DBG] osdmap e633: 8 total, 8 up, 8 in 2021-03-24T22:03:24.951 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:25.008 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:25.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:25.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:03:25.066 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-24T22:03:25.085 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":1110,"stamp":"2021-03-24T22:03:22.436784+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":495808991,"num_objects":550,"num_object_clones":178,"num_object_copies":1650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":40,"num_objects_unfound":0,"num_objects_dirty":550,"num_whiteouts":10,"num_read":4730,"num_read_kb":3251725,"num_write":3065,"num_write_kb":856079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3480,"num_bytes_recovered":1425574570,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14818,"ondisk_log_size":14818,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":506,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1353716,"kb_used_data":1053108,"kb_used_omap":33,"kb_used_meta":300510,"kb_avail":748607500,"statfs":{"total":767960285184,"available":766574080000,"internally_reserved":0,"allocated":1078382592,"data_stored":1448130278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33936,"internal_metadata":307723120},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1453323,"num_objects":-4,"num_object_clones":-4,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4,"num_whiteouts":2,"num_read":-41,"num_read_kb":-43369,"num_write":-41,"num_write_kb":-11719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-4,"num_bytes_recovered":-11591117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.024124"},"pg_stats":[{"pgid":"3.29","version":"0'0","reported_seq":"263","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.501356+0000","last_change":"2021-03-24T22:01:16.299847+0000","last_active":"2021-03-24T22:03:17.501356+0000","last_peered":"2021-03-24T22:03:17.501356+0000","last_clean":"2021-03-24T22:03:17.501356+0000","last_became_active":"2021-03-24T22:01:16.299327+0000","last_became_peered":"2021-03-24T22:01:16.299327+0000","last_unstale":"2021-03-24T22:03:17.501356+0000","last_undegraded":"2021-03-24T22:03:17.501356+0000","last_fullsized":"2021-03-24T22:03:17.501356+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:55.592795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:55.592795+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":"211","reported_epoch":"629","state":"active+clean","last_fresh":"2021-03-24T22:03:19.447627+0000","last_change":"2021-03-24T22:03:07.837859+0000","last_active":"2021-03-24T22:03:19.447627+0000","last_peered":"2021-03-24T22:03:19.447627+0000","last_clean":"2021-03-24T22:03:19.447627+0000","last_became_active":"2021-03-24T22:00:04.286965+0000","last_became_peered":"2021-03-24T22:00:04.286965+0000","last_unstale":"2021-03-24T22:03:19.447627+0000","last_undegraded":"2021-03-24T22:03:19.447627+0000","last_fullsized":"2021-03-24T22:03:19.447627+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:07.837673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:03:07.837673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"89'2","reported_seq":"295","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.494499+0000","last_change":"2021-03-24T21:59:14.374798+0000","last_active":"2021-03-24T22:03:17.494499+0000","last_peered":"2021-03-24T22:03:17.494499+0000","last_clean":"2021-03-24T22:03:17.494499+0000","last_became_active":"2021-03-24T21:59:14.374418+0000","last_became_peered":"2021-03-24T21:59:14.374418+0000","last_unstale":"2021-03-24T22:03:17.494499+0000","last_undegraded":"2021-03-24T22:03:17.494499+0000","last_fullsized":"2021-03-24T22:03:17.494499+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":6,"last_scrub":"89'2","last_scrub_stamp":"2021-03-24T21:57:58.491830+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2021-03-24T21:57:58.491830+0000","last_clean_scrub_stamp":"2021-03-24T21:57:58.491830+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":179,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":"623","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.499702+0000","last_change":"2021-03-24T22:00:04.297849+0000","last_active":"2021-03-24T22:03:17.499702+0000","last_peered":"2021-03-24T22:03:17.499702+0000","last_clean":"2021-03-24T22:03:17.499702+0000","last_became_active":"2021-03-24T22:00:04.297076+0000","last_became_peered":"2021-03-24T22:00:04.297076+0000","last_unstale":"2021-03-24T22:03:17.499702+0000","last_undegraded":"2021-03-24T22:03:17.499702+0000","last_fullsized":"2021-03-24T22:03:17.499702+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:33.533840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:33.533840+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":"69","reported_epoch":"630","state":"active+clean","last_fresh":"2021-03-24T22:03:20.345699+0000","last_change":"2021-03-24T22:02:22.324306+0000","last_active":"2021-03-24T22:03:20.345699+0000","last_peered":"2021-03-24T22:03:20.345699+0000","last_clean":"2021-03-24T22:03:20.345699+0000","last_became_active":"2021-03-24T22:02:22.323235+0000","last_became_peered":"2021-03-24T22:02:22.323235+0000","last_unstale":"2021-03-24T22:03:20.345699+0000","last_undegraded":"2021-03-24T22:03:20.345699+0000","last_fullsized":"2021-03-24T22:03:20.345699+0000","mapping_epoch":571,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:38.304792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:38.304792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"61'1","reported_seq":"111","reported_epoch":"629","state":"active+clean","last_fresh":"2021-03-24T22:03:19.423258+0000","last_change":"2021-03-24T22:01:43.577425+0000","last_active":"2021-03-24T22:03:19.423258+0000","last_peered":"2021-03-24T22:03:19.423258+0000","last_clean":"2021-03-24T22:03:19.423258+0000","last_became_active":"2021-03-24T22:01:43.390951+0000","last_became_peered":"2021-03-24T22:01:43.390951+0000","last_unstale":"2021-03-24T22:03:19.423258+0000","last_undegraded":"2021-03-24T22:03:19.423258+0000","last_fullsized":"2021-03-24T22:03:19.423258+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":6,"last_scrub":"61'1","last_scrub_stamp":"2021-03-24T21:57:12.046206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:12.046206+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":437,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3492,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":"610","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.502466+0000","last_change":"2021-03-24T22:02:57.502941+0000","last_active":"2021-03-24T22:03:17.502466+0000","last_peered":"2021-03-24T22:03:17.502466+0000","last_clean":"2021-03-24T22:03:17.502466+0000","last_became_active":"2021-03-24T21:59:14.370506+0000","last_became_peered":"2021-03-24T21:59:14.370506+0000","last_unstale":"2021-03-24T22:03:17.502466+0000","last_undegraded":"2021-03-24T22:03:17.502466+0000","last_fullsized":"2021-03-24T22:03:17.502466+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:57.502803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:02:57.502803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.23","version":"630'372","reported_seq":"2124","reported_epoch":"630","state":"active+clean","last_fresh":"2021-03-24T22:03:20.348293+0000","last_change":"2021-03-24T22:03:20.348630+0000","last_active":"2021-03-24T22:03:20.348293+0000","last_peered":"2021-03-24T22:03:20.348293+0000","last_clean":"2021-03-24T22:03:20.348293+0000","last_became_active":"2021-03-24T22:01:46.311925+0000","last_became_peered":"2021-03-24T22:01:46.311925+0000","last_unstale":"2021-03-24T22:03:20.348293+0000","last_undegraded":"2021-03-24T22:03:20.348293+0000","last_fullsized":"2021-03-24T22:03:20.348293+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":372,"ondisk_log_size":372,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8766165,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":125,"num_read_kb":141370,"num_write":87,"num_write_kb":33080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":12165888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"18b","length":"1"},{"start":"194","length":"1"},{"start":"1a6","length":"1"},{"start":"1cc","length":"1"},{"start":"1e8","length":"1"}]},{"pgid":"3.26","version":"0'0","reported_seq":"116","reported_epoch":"629","state":"active+clean","last_fresh":"2021-03-24T22:03:19.424394+0000","last_change":"2021-03-24T22:01:57.311479+0000","last_active":"2021-03-24T22:03:19.424394+0000","last_peered":"2021-03-24T22:03:19.424394+0000","last_clean":"2021-03-24T22:03:19.424394+0000","last_became_active":"2021-03-24T22:01:57.311128+0000","last_became_peered":"2021-03-24T22:01:57.311128+0000","last_unstale":"2021-03-24T22:03:19.424394+0000","last_undegraded":"2021-03-24T22:03:19.424394+0000","last_fullsized":"2021-03-24T22:03:19.424394+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:42.879245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:59:42.879245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"628'238","reported_seq":"1207","reported_epoch":"628","state":"active+clean","last_fresh":"2021-03-24T22:03:18.434592+0000","last_change":"2021-03-24T22:03:18.434592+0000","last_active":"2021-03-24T22:03:18.434592+0000","last_peered":"2021-03-24T22:03:18.434592+0000","last_clean":"2021-03-24T22:03:18.434592+0000","last_became_active":"2021-03-24T22:03:06.343802+0000","last_became_peered":"2021-03-24T22:03:06.343802+0000","last_unstale":"2021-03-24T22:03:18.434592+0000","last_undegraded":"2021-03-24T22:03:18.434592+0000","last_fullsized":"2021-03-24T22:03:18.434592+0000","mapping_epoch":615,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":616,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":238,"ondisk_log_size":238,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17705621,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":33,"num_read_kb":14791,"num_write":57,"num_write_kb":14060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":21273869,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"139","length":"1"},{"start":"18b","length":"1"},{"start":"194","length":"1"},{"start":"1cc","length":"1"},{"start":"1d7","length":"1"},{"start":"1e8","length":"1"},{"start":"1eb","length":"1"},{"start":"1f4","length":"1"}]},{"pgid":"4.c","version":"544'37","reported_seq":"359","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.502381+0000","last_change":"2021-03-24T22:03:03.453859+0000","last_active":"2021-03-24T22:03:17.502381+0000","last_peered":"2021-03-24T22:03:17.502381+0000","last_clean":"2021-03-24T22:03:17.502381+0000","last_became_active":"2021-03-24T22:01:50.301325+0000","last_became_peered":"2021-03-24T22:01:50.301325+0000","last_unstale":"2021-03-24T22:03:17.502381+0000","last_undegraded":"2021-03-24T22:03:17.502381+0000","last_fullsized":"2021-03-24T22:03:17.502381+0000","mapping_epoch":539,"log_start":"69'15","ondisk_log_start":"69'15","created":62,"last_epoch_clean":540,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'37","last_scrub_stamp":"2021-03-24T22:03:03.453713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:03:03.453713+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":15,"num_object_clones":0,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":81,"num_read_kb":61,"num_write":51,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":1266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"593","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.500378+0000","last_change":"2021-03-24T21:59:17.264251+0000","last_active":"2021-03-24T22:03:17.500378+0000","last_peered":"2021-03-24T22:03:17.500378+0000","last_clean":"2021-03-24T22:03:17.500378+0000","last_became_active":"2021-03-24T21:51:45.375830+0000","last_became_peered":"2021-03-24T21:51:45.375830+0000","last_unstale":"2021-03-24T22:03:17.500378+0000","last_undegraded":"2021-03-24T22:03:17.500378+0000","last_fullsized":"2021-03-24T22:03:17.500378+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:17.264089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:59:17.264089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"548'12","reported_seq":"814","reported_epoch":"630","state":"active+clean","last_fresh":"2021-03-24T22:03:20.347722+0000","last_change":"2021-03-24T22:02:18.728472+0000","last_active":"2021-03-24T22:03:20.347722+0000","last_peered":"2021-03-24T22:03:20.347722+0000","last_clean":"2021-03-24T22:03:20.347722+0000","last_became_active":"2021-03-24T22:01:57.320271+0000","last_became_peered":"2021-03-24T22:01:57.320271+0000","last_unstale":"2021-03-24T22:03:20.347722+0000","last_undegraded":"2021-03-24T22:03:20.347722+0000","last_fullsized":"2021-03-24T22:03:20.347722+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"548'12","last_scrub_stamp":"2021-03-24T22:02:18.728277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:02:18.728277+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"142","reported_epoch":"628","state":"active+clean","last_fresh":"2021-03-24T22:03:18.413445+0000","last_change":"2021-03-24T22:02:22.483455+0000","last_active":"2021-03-24T22:03:18.413445+0000","last_peered":"2021-03-24T22:03:18.413445+0000","last_clean":"2021-03-24T22:03:18.413445+0000","last_became_active":"2021-03-24T22:02:22.482833+0000","last_became_peered":"2021-03-24T22:02:22.482833+0000","last_unstale":"2021-03-24T22:03:18.413445+0000","last_undegraded":"2021-03-24T22:03:18.413445+0000","last_fullsized":"2021-03-24T22:03:18.413445+0000","mapping_epoch":571,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:13.834281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:00:13.834281+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.f","version":"626'485","reported_seq":"2361","reported_epoch":"628","state":"active+clean","last_fresh":"2021-03-24T22:03:18.419478+0000","last_change":"2021-03-24T22:03:18.420182+0000","last_active":"2021-03-24T22:03:18.419478+0000","last_peered":"2021-03-24T22:03:18.419478+0000","last_clean":"2021-03-24T22:03:18.419478+0000","last_became_active":"2021-03-24T22:02:10.313297+0000","last_became_peered":"2021-03-24T22:02:10.313297+0000","last_unstale":"2021-03-24T22:03:18.419478+0000","last_undegraded":"2021-03-24T22:03:18.419478+0000","last_fullsized":"2021-03-24T22:03:18.419478+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'171","last_scrub_stamp":"2021-03-24T21:59:08.017874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:59:08.017874+0000","log_size":485,"ondisk_log_size":485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20846417,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":101,"num_read_kb":118330,"num_write":45,"num_write_kb":16866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":46545574,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"139","length":"1"},{"start":"18b","length":"1"},{"start":"194","length":"1"},{"start":"1cc","length":"1"},{"start":"1d7","length":"1"},{"start":"1e8","length":"1"},{"start":"1eb","length":"1"},{"start":"1f4","length":"1"}]},{"pgid":"4.d","version":"544'41","reported_seq":"316","reported_epoch":"630","state":"active+clean","last_fresh":"2021-03-24T22:03:20.347411+0000","last_change":"2021-03-24T22:03:03.723634+0000","last_active":"2021-03-24T22:03:20.347411+0000","last_peered":"2021-03-24T22:03:20.347411+0000","last_clean":"2021-03-24T22:03:20.347411+0000","last_became_active":"2021-03-24T22:01:25.276760+0000","last_became_peered":"2021-03-24T22:01:25.276760+0000","last_unstale":"2021-03-24T22:03:20.347411+0000","last_undegraded":"2021-03-24T22:03:20.347411+0000","last_fullsized":"2021-03-24T22:03:20.347411+0000","mapping_epoch":514,"log_start":"69'21","ondisk_log_start":"69'21","created":62,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'41","last_scrub_stamp":"2021-03-24T22:03:03.723420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:03:03.723420+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1578,"num_objects":21,"num_object_clones":0,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":106,"num_read_kb":71,"num_write":60,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"112","reported_epoch":"629","state":"active+clean","last_fresh":"2021-03-24T22:03:19.424592+0000","last_change":"2021-03-24T22:02:30.890732+0000","last_active":"2021-03-24T22:03:19.424592+0000","last_peered":"2021-03-24T22:03:19.424592+0000","last_clean":"2021-03-24T22:03:19.424592+0000","last_became_active":"2021-03-24T22:01:43.396233+0000","last_became_peered":"2021-03-24T22:01:43.396233+0000","last_unstale":"2021-03-24T22:03:19.424592+0000","last_undegraded":"2021-03-24T22:03:19.424592+0000","last_fullsized":"2021-03-24T22:03:19.424592+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:30.890594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:55:01.169461+0000","last_clean_scrub_stamp":"2021-03-24T22:02:30.890594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"133","reported_epoch":"628","state":"active+clean","last_fresh":"2021-03-24T22:03:18.412906+0000","last_change":"2021-03-24T22:01:16.334950+0000","last_active":"2021-03-24T22:03:18.412906+0000","last_peered":"2021-03-24T22:03:18.412906+0000","last_clean":"2021-03-24T22:03:18.412906+0000","last_became_active":"2021-03-24T22:01:16.334606+0000","last_became_peered":"2021-03-24T22:01:16.334606+0000","last_unstale":"2021-03-24T22:03:18.412906+0000","last_undegraded":"2021-03-24T22:03:18.412906+0000","last_fullsized":"2021-03-24T22:03:18.412906+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:19.934770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:00:19.934770+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"69","reported_epoch":"630","state":"active+clean","last_fresh":"2021-03-24T22:03:20.347932+0000","last_change":"2021-03-24T22:02:22.483059+0000","last_active":"2021-03-24T22:03:20.347932+0000","last_peered":"2021-03-24T22:03:20.347932+0000","last_clean":"2021-03-24T22:03:20.347932+0000","last_became_active":"2021-03-24T22:02:22.482380+0000","last_became_peered":"2021-03-24T22:02:22.482380+0000","last_unstale":"2021-03-24T22:03:20.347932+0000","last_undegraded":"2021-03-24T22:03:20.347932+0000","last_fullsized":"2021-03-24T22:03:20.347932+0000","mapping_epoch":571,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:57.316683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:00:57.316683+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"628'211","reported_seq":"1866","reported_epoch":"630","state":"active+clean","last_fresh":"2021-03-24T22:03:20.349635+0000","last_change":"2021-03-24T22:03:20.349635+0000","last_active":"2021-03-24T22:03:20.349635+0000","last_peered":"2021-03-24T22:03:20.349635+0000","last_clean":"2021-03-24T22:03:20.349635+0000","last_became_active":"2021-03-24T22:00:03.281149+0000","last_became_peered":"2021-03-24T22:00:03.281149+0000","last_unstale":"2021-03-24T22:03:20.349635+0000","last_undegraded":"2021-03-24T22:03:20.349635+0000","last_fullsized":"2021-03-24T22:03:20.349635+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"546'150","last_scrub_stamp":"2021-03-24T22:01:56.662245+0000","last_deep_scrub":"546'150","last_deep_scrub_stamp":"2021-03-24T22:01:56.662245+0000","last_clean_scrub_stamp":"2021-03-24T22:01:56.662245+0000","log_size":211,"ondisk_log_size":211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10084761,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":52,"num_read_kb":39509,"num_write":33,"num_write_kb":9037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2421497,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"18b","length":"1"},{"start":"194","length":"1"},{"start":"1a6","length":"1"},{"start":"1cc","length":"1"},{"start":"1e8","length":"1"}]},{"pgid":"4.e","version":"544'45","reported_seq":"368","reported_epoch":"629","state":"active+clean","last_fresh":"2021-03-24T22:03:19.424213+0000","last_change":"2021-03-24T22:02:15.401403+0000","last_active":"2021-03-24T22:03:19.424213+0000","last_peered":"2021-03-24T22:03:19.424213+0000","last_clean":"2021-03-24T22:03:19.424213+0000","last_became_active":"2021-03-24T22:02:15.400902+0000","last_became_peered":"2021-03-24T22:02:15.400902+0000","last_unstale":"2021-03-24T22:03:19.424213+0000","last_undegraded":"2021-03-24T22:03:19.424213+0000","last_fullsized":"2021-03-24T22:03:19.424213+0000","mapping_epoch":564,"log_start":"89'27","ondisk_log_start":"89'27","created":62,"last_epoch_clean":565,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'27","last_scrub_stamp":"2021-03-24T22:00:20.952131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:00:20.952131+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1672,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":92,"num_read_kb":60,"num_write":50,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"606","reported_epoch":"630","state":"active+clean","last_fresh":"2021-03-24T22:03:20.345806+0000","last_change":"2021-03-24T22:01:57.302726+0000","last_active":"2021-03-24T22:03:20.345806+0000","last_peered":"2021-03-24T22:03:20.345806+0000","last_clean":"2021-03-24T22:03:20.345806+0000","last_became_active":"2021-03-24T22:01:57.302297+0000","last_became_peered":"2021-03-24T22:01:57.302297+0000","last_unstale":"2021-03-24T22:03:20.345806+0000","last_undegraded":"2021-03-24T22:03:20.345806+0000","last_fullsized":"2021-03-24T22:03:20.345806+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:55.592795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:55.592795+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"107","reported_epoch":"629","state":"active+clean","last_fresh":"2021-03-24T22:03:19.424948+0000","last_change":"2021-03-24T22:01:43.384687+0000","last_active":"2021-03-24T22:03:19.424948+0000","last_peered":"2021-03-24T22:03:19.424948+0000","last_clean":"2021-03-24T22:03:19.424948+0000","last_became_active":"2021-03-24T22:01:43.384327+0000","last_became_peered":"2021-03-24T22:01:43.384327+0000","last_unstale":"2021-03-24T22:03:19.424948+0000","last_undegraded":"2021-03-24T22:03:19.424948+0000","last_fullsized":"2021-03-24T22:03:19.424948+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:56.217032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:00:56.217032+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"644","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.501970+0000","last_change":"2021-03-24T22:01:43.384970+0000","last_active":"2021-03-24T22:03:17.501970+0000","last_peered":"2021-03-24T22:03:17.501970+0000","last_clean":"2021-03-24T22:03:17.501970+0000","last_became_active":"2021-03-24T22:01:43.384573+0000","last_became_peered":"2021-03-24T22:01:43.384573+0000","last_unstale":"2021-03-24T22:03:17.501970+0000","last_undegraded":"2021-03-24T22:03:17.501970+0000","last_fullsized":"2021-03-24T22:03:17.501970+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:25.263840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:00:25.263840+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"630'558","reported_seq":"2343","reported_epoch":"630","state":"active+clean","last_fresh":"2021-03-24T22:03:20.361027+0000","last_change":"2021-03-24T22:03:20.361466+0000","last_active":"2021-03-24T22:03:20.361027+0000","last_peered":"2021-03-24T22:03:20.361027+0000","last_clean":"2021-03-24T22:03:20.361027+0000","last_became_active":"2021-03-24T22:01:48.324477+0000","last_became_peered":"2021-03-24T22:01:48.324477+0000","last_unstale":"2021-03-24T22:03:20.361027+0000","last_undegraded":"2021-03-24T22:03:20.361027+0000","last_fullsized":"2021-03-24T22:03:20.361027+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":0,"last_scrub":"622'539","last_scrub_stamp":"2021-03-24T22:03:12.756606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:03:12.756606+0000","log_size":558,"ondisk_log_size":558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12008932,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":61,"num_read_kb":86468,"num_write":29,"num_write_kb":13112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":45770019,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"18b","length":"1"},{"start":"194","length":"1"},{"start":"1a6","length":"1"},{"start":"1cc","length":"1"},{"start":"1e8","length":"1"}]},{"pgid":"4.f","version":"544'46","reported_seq":"782","reported_epoch":"628","state":"active+clean","last_fresh":"2021-03-24T22:03:18.413510+0000","last_change":"2021-03-24T22:01:53.692379+0000","last_active":"2021-03-24T22:03:18.413510+0000","last_peered":"2021-03-24T22:03:18.413510+0000","last_clean":"2021-03-24T22:03:18.413510+0000","last_became_active":"2021-03-24T22:01:48.328342+0000","last_became_peered":"2021-03-24T22:01:48.328342+0000","last_unstale":"2021-03-24T22:03:18.413510+0000","last_undegraded":"2021-03-24T22:03:18.413510+0000","last_fullsized":"2021-03-24T22:03:18.413510+0000","mapping_epoch":537,"log_start":"274'26","ondisk_log_start":"274'26","created":62,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'26","last_scrub_stamp":"2021-03-24T22:01:53.692294+0000","last_deep_scrub":"274'26","last_deep_scrub_stamp":"2021-03-24T21:57:24.499090+0000","last_clean_scrub_stamp":"2021-03-24T22:01:53.692294+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1472,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":98,"num_read_kb":66,"num_write":56,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":5466,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":"657","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.498806+0000","last_change":"2021-03-24T22:01:57.316662+0000","last_active":"2021-03-24T22:03:17.498806+0000","last_peered":"2021-03-24T22:03:17.498806+0000","last_clean":"2021-03-24T22:03:17.498806+0000","last_became_active":"2021-03-24T22:01:57.316262+0000","last_became_peered":"2021-03-24T22:01:57.316262+0000","last_unstale":"2021-03-24T22:03:17.498806+0000","last_undegraded":"2021-03-24T22:03:17.498806+0000","last_fullsized":"2021-03-24T22:03:17.498806+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:39.439170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:39.439170+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"479'5","reported_seq":"200","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.497179+0000","last_change":"2021-03-24T22:00:49.311757+0000","last_active":"2021-03-24T22:03:17.497179+0000","last_peered":"2021-03-24T22:03:17.497179+0000","last_clean":"2021-03-24T22:03:17.497179+0000","last_became_active":"2021-03-24T22:00:49.310037+0000","last_became_peered":"2021-03-24T22:00:49.310037+0000","last_unstale":"2021-03-24T22:03:17.497179+0000","last_undegraded":"2021-03-24T22:03:17.497179+0000","last_fullsized":"2021-03-24T22:03:17.497179+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'4","last_scrub_stamp":"2021-03-24T22:00:14.879311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:00:14.879311+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"506","reported_epoch":"629","state":"active+clean","last_fresh":"2021-03-24T22:03:19.448339+0000","last_change":"2021-03-24T22:02:55.950583+0000","last_active":"2021-03-24T22:03:19.448339+0000","last_peered":"2021-03-24T22:03:19.448339+0000","last_clean":"2021-03-24T22:03:19.448339+0000","last_became_active":"2021-03-24T22:01:16.314269+0000","last_became_peered":"2021-03-24T22:01:16.314269+0000","last_unstale":"2021-03-24T22:03:19.448339+0000","last_undegraded":"2021-03-24T22:03:19.448339+0000","last_fullsized":"2021-03-24T22:03:19.448339+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:55.950428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:02:55.950428+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"630'853","reported_seq":"3005","reported_epoch":"630","state":"active+clean","last_fresh":"2021-03-24T22:03:20.355342+0000","last_change":"2021-03-24T22:03:20.355342+0000","last_active":"2021-03-24T22:03:20.355342+0000","last_peered":"2021-03-24T22:03:20.355342+0000","last_clean":"2021-03-24T22:03:20.355342+0000","last_became_active":"2021-03-24T22:02:06.309737+0000","last_became_peered":"2021-03-24T22:02:06.309737+0000","last_unstale":"2021-03-24T22:03:20.355342+0000","last_undegraded":"2021-03-24T22:03:20.355342+0000","last_fullsized":"2021-03-24T22:03:20.355342+0000","mapping_epoch":555,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":556,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'393","last_scrub_stamp":"2021-03-24T21:59:12.370455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:59:12.370455+0000","log_size":853,"ondisk_log_size":853,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23361815,"num_objects":10,"num_object_clones":6,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":127,"num_read_kb":138355,"num_write":89,"num_write_kb":33163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":296,"num_bytes_recovered":189413266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"18b","length":"1"},{"start":"194","length":"1"},{"start":"1a6","length":"1"},{"start":"1cc","length":"1"},{"start":"1e8","length":"1"}]},{"pgid":"5.1","version":"0'0","reported_seq":"567","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.500502+0000","last_change":"2021-03-24T22:03:00.472146+0000","last_active":"2021-03-24T22:03:17.500502+0000","last_peered":"2021-03-24T22:03:17.500502+0000","last_clean":"2021-03-24T22:03:17.500502+0000","last_became_active":"2021-03-24T22:01:57.305430+0000","last_became_peered":"2021-03-24T22:01:57.305430+0000","last_unstale":"2021-03-24T22:03:17.500502+0000","last_undegraded":"2021-03-24T22:03:17.500502+0000","last_fullsized":"2021-03-24T22:03:17.500502+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:00.471949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:25.220436+0000","last_clean_scrub_stamp":"2021-03-24T22:03:00.471949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"544'34","reported_seq":"795","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.499326+0000","last_change":"2021-03-24T22:01:58.356469+0000","last_active":"2021-03-24T22:03:17.499326+0000","last_peered":"2021-03-24T22:03:17.499326+0000","last_clean":"2021-03-24T22:03:17.499326+0000","last_became_active":"2021-03-24T22:01:57.309916+0000","last_became_peered":"2021-03-24T22:01:57.309916+0000","last_unstale":"2021-03-24T22:03:17.499326+0000","last_undegraded":"2021-03-24T22:03:17.499326+0000","last_fullsized":"2021-03-24T22:03:17.499326+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'22","last_scrub_stamp":"2021-03-24T21:58:56.473510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:58:56.473510+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":64,"num_read_kb":44,"num_write":36,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":3384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"644","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.502294+0000","last_change":"2021-03-24T22:01:57.314681+0000","last_active":"2021-03-24T22:03:17.502294+0000","last_peered":"2021-03-24T22:03:17.502294+0000","last_clean":"2021-03-24T22:03:17.502294+0000","last_became_active":"2021-03-24T22:01:57.314345+0000","last_became_peered":"2021-03-24T22:01:57.314345+0000","last_unstale":"2021-03-24T22:03:17.502294+0000","last_undegraded":"2021-03-24T22:03:17.502294+0000","last_fullsized":"2021-03-24T22:03:17.502294+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:54.381668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:54.381668+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"161","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.500198+0000","last_change":"2021-03-24T22:00:49.305933+0000","last_active":"2021-03-24T22:03:17.500198+0000","last_peered":"2021-03-24T22:03:17.500198+0000","last_clean":"2021-03-24T22:03:17.500198+0000","last_became_active":"2021-03-24T22:00:49.305612+0000","last_became_peered":"2021-03-24T22:00:49.305612+0000","last_unstale":"2021-03-24T22:03:17.500198+0000","last_undegraded":"2021-03-24T22:03:17.500198+0000","last_fullsized":"2021-03-24T22:03:17.500198+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:03.626250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:59:03.626250+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"626'619","reported_seq":"2277","reported_epoch":"628","state":"active+clean","last_fresh":"2021-03-24T22:03:18.431717+0000","last_change":"2021-03-24T22:03:18.432015+0000","last_active":"2021-03-24T22:03:18.431717+0000","last_peered":"2021-03-24T22:03:18.431717+0000","last_clean":"2021-03-24T22:03:18.431717+0000","last_became_active":"2021-03-24T21:57:26.169302+0000","last_became_peered":"2021-03-24T21:57:26.169302+0000","last_unstale":"2021-03-24T22:03:18.431717+0000","last_undegraded":"2021-03-24T22:03:18.431717+0000","last_fullsized":"2021-03-24T22:03:18.431717+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'330","last_scrub_stamp":"2021-03-24T21:59:07.373692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:59:07.373692+0000","log_size":619,"ondisk_log_size":619,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15972415,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":48,"num_read_kb":42244,"num_write":43,"num_write_kb":17707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"139","length":"1"},{"start":"18b","length":"1"},{"start":"194","length":"1"},{"start":"1cc","length":"1"},{"start":"1d7","length":"1"},{"start":"1e8","length":"1"},{"start":"1eb","length":"1"},{"start":"1f4","length":"1"}]},{"pgid":"5.0","version":"508'4","reported_seq":"754","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.498621+0000","last_change":"2021-03-24T22:01:16.693197+0000","last_active":"2021-03-24T22:03:17.498621+0000","last_peered":"2021-03-24T22:03:17.498621+0000","last_clean":"2021-03-24T22:03:17.498621+0000","last_became_active":"2021-03-24T22:01:16.300485+0000","last_became_peered":"2021-03-24T22:01:16.300485+0000","last_unstale":"2021-03-24T22:03:17.498621+0000","last_undegraded":"2021-03-24T22:03:17.498621+0000","last_fullsized":"2021-03-24T22:03:17.498621+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'3","last_scrub_stamp":"2021-03-24T21:59:46.375145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:59:46.375145+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"544'34","reported_seq":"276","reported_epoch":"629","state":"active+clean","last_fresh":"2021-03-24T22:03:19.447078+0000","last_change":"2021-03-24T22:02:32.116263+0000","last_active":"2021-03-24T22:03:19.447078+0000","last_peered":"2021-03-24T22:03:19.447078+0000","last_clean":"2021-03-24T22:03:19.447078+0000","last_became_active":"2021-03-24T22:01:43.388853+0000","last_became_peered":"2021-03-24T22:01:43.388853+0000","last_unstale":"2021-03-24T22:03:19.447078+0000","last_undegraded":"2021-03-24T22:03:19.447078+0000","last_fullsized":"2021-03-24T22:03:19.447078+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'34","last_scrub_stamp":"2021-03-24T22:02:32.116075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:02:32.116075+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":59,"num_read_kb":40,"num_write":34,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":940,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"631","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.496379+0000","last_change":"2021-03-24T22:01:16.336120+0000","last_active":"2021-03-24T22:03:17.496379+0000","last_peered":"2021-03-24T22:03:17.496379+0000","last_clean":"2021-03-24T22:03:17.496379+0000","last_became_active":"2021-03-24T22:01:16.335790+0000","last_became_peered":"2021-03-24T22:01:16.335790+0000","last_unstale":"2021-03-24T22:03:17.496379+0000","last_undegraded":"2021-03-24T22:03:17.496379+0000","last_fullsized":"2021-03-24T22:03:17.496379+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:28.165408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:28.165408+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"91","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.460902+0000","last_change":"2021-03-24T22:01:57.318244+0000","last_active":"2021-03-24T22:03:17.460902+0000","last_peered":"2021-03-24T22:03:17.460902+0000","last_clean":"2021-03-24T22:03:17.460902+0000","last_became_active":"2021-03-24T22:01:57.316610+0000","last_became_peered":"2021-03-24T22:01:57.316610+0000","last_unstale":"2021-03-24T22:03:17.460902+0000","last_undegraded":"2021-03-24T22:03:17.460902+0000","last_fullsized":"2021-03-24T22:03:17.460902+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:54.826534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:58:54.826534+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"272'64","reported_seq":"1708","reported_epoch":"628","state":"active+clean","last_fresh":"2021-03-24T22:03:18.425957+0000","last_change":"2021-03-24T22:03:18.426147+0000","last_active":"2021-03-24T22:03:18.425957+0000","last_peered":"2021-03-24T22:03:18.425957+0000","last_clean":"2021-03-24T22:03:18.425957+0000","last_became_active":"2021-03-24T22:01:16.299738+0000","last_became_peered":"2021-03-24T22:01:16.299738+0000","last_unstale":"2021-03-24T22:03:18.425957+0000","last_undegraded":"2021-03-24T22:03:18.425957+0000","last_fullsized":"2021-03-24T22:03:18.425957+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'64","last_scrub_stamp":"2021-03-24T21:59:08.406188+0000","last_deep_scrub":"272'64","last_deep_scrub_stamp":"2021-03-24T21:59:08.406188+0000","last_clean_scrub_stamp":"2021-03-24T21:59:08.406188+0000","log_size":64,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"139","length":"1"},{"start":"18b","length":"1"},{"start":"194","length":"1"},{"start":"1cc","length":"1"},{"start":"1d7","length":"1"},{"start":"1e8","length":"1"},{"start":"1eb","length":"1"},{"start":"1f4","length":"1"}]},{"pgid":"5.3","version":"533'7","reported_seq":"766","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.406425+0000","last_change":"2021-03-24T22:01:43.634882+0000","last_active":"2021-03-24T22:03:17.406425+0000","last_peered":"2021-03-24T22:03:17.406425+0000","last_clean":"2021-03-24T22:03:17.406425+0000","last_became_active":"2021-03-24T22:01:43.393325+0000","last_became_peered":"2021-03-24T22:01:43.393325+0000","last_unstale":"2021-03-24T22:03:17.406425+0000","last_undegraded":"2021-03-24T22:03:17.406425+0000","last_fullsized":"2021-03-24T22:03:17.406425+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'6","last_scrub_stamp":"2021-03-24T22:00:16.563338+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-24T21:53:26.584678+0000","last_clean_scrub_stamp":"2021-03-24T22:00:16.563338+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"544'26","reported_seq":"172","reported_epoch":"628","state":"active+clean","last_fresh":"2021-03-24T22:03:18.411362+0000","last_change":"2021-03-24T22:01:16.373466+0000","last_active":"2021-03-24T22:03:18.411362+0000","last_peered":"2021-03-24T22:03:18.411362+0000","last_clean":"2021-03-24T22:03:18.411362+0000","last_became_active":"2021-03-24T22:01:16.334794+0000","last_became_peered":"2021-03-24T22:01:16.334794+0000","last_unstale":"2021-03-24T22:03:18.411362+0000","last_undegraded":"2021-03-24T22:03:18.411362+0000","last_fullsized":"2021-03-24T22:03:18.411362+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T22:00:58.712071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:00:58.712071+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":47,"num_read_kb":30,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":940,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"485","reported_epoch":"629","state":"active+clean","last_fresh":"2021-03-24T22:03:19.448151+0000","last_change":"2021-03-24T22:03:05.892591+0000","last_active":"2021-03-24T22:03:19.448151+0000","last_peered":"2021-03-24T22:03:19.448151+0000","last_clean":"2021-03-24T22:03:19.448151+0000","last_became_active":"2021-03-24T22:00:03.284574+0000","last_became_peered":"2021-03-24T22:00:03.284574+0000","last_unstale":"2021-03-24T22:03:19.448151+0000","last_undegraded":"2021-03-24T22:03:19.448151+0000","last_fullsized":"2021-03-24T22:03:19.448151+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:05.892415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:03:05.892415+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"582","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.406025+0000","last_change":"2021-03-24T21:59:04.077747+0000","last_active":"2021-03-24T22:03:17.406025+0000","last_peered":"2021-03-24T22:03:17.406025+0000","last_clean":"2021-03-24T22:03:17.406025+0000","last_became_active":"2021-03-24T21:51:51.759627+0000","last_became_peered":"2021-03-24T21:51:51.759627+0000","last_unstale":"2021-03-24T22:03:17.406025+0000","last_undegraded":"2021-03-24T22:03:17.406025+0000","last_fullsized":"2021-03-24T22:03:17.406025+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:04.077569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:59:04.077569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"628'414","reported_seq":"2159","reported_epoch":"628","state":"active+clean","last_fresh":"2021-03-24T22:03:18.426428+0000","last_change":"2021-03-24T22:03:18.428342+0000","last_active":"2021-03-24T22:03:18.426428+0000","last_peered":"2021-03-24T22:03:18.426428+0000","last_clean":"2021-03-24T22:03:18.426428+0000","last_became_active":"2021-03-24T22:02:30.322922+0000","last_became_peered":"2021-03-24T22:02:30.322922+0000","last_unstale":"2021-03-24T22:03:18.426428+0000","last_undegraded":"2021-03-24T22:03:18.426428+0000","last_fullsized":"2021-03-24T22:03:18.426428+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"365'287","last_scrub_stamp":"2021-03-24T21:58:56.648040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:58:56.648040+0000","log_size":414,"ondisk_log_size":414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11619307,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":36562,"num_write":24,"num_write_kb":5739,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":24969090,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"139","length":"1"},{"start":"18b","length":"1"},{"start":"194","length":"1"},{"start":"1cc","length":"1"},{"start":"1d7","length":"1"},{"start":"1e8","length":"1"},{"start":"1eb","length":"1"},{"start":"1f4","length":"1"}]},{"pgid":"5.2","version":"0'0","reported_seq":"107","reported_epoch":"629","state":"active+clean","last_fresh":"2021-03-24T22:03:19.424132+0000","last_change":"2021-03-24T22:01:43.391207+0000","last_active":"2021-03-24T22:03:19.424132+0000","last_peered":"2021-03-24T22:03:19.424132+0000","last_clean":"2021-03-24T22:03:19.424132+0000","last_became_active":"2021-03-24T22:01:43.390850+0000","last_became_peered":"2021-03-24T22:01:43.390850+0000","last_unstale":"2021-03-24T22:03:19.424132+0000","last_undegraded":"2021-03-24T22:03:19.424132+0000","last_fullsized":"2021-03-24T22:03:19.424132+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:14.616769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:00:14.616769+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"544'31","reported_seq":"787","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.494164+0000","last_change":"2021-03-24T22:01:58.947168+0000","last_active":"2021-03-24T22:03:17.494164+0000","last_peered":"2021-03-24T22:03:17.494164+0000","last_clean":"2021-03-24T22:03:17.494164+0000","last_became_active":"2021-03-24T22:01:57.324818+0000","last_became_peered":"2021-03-24T22:01:57.324818+0000","last_unstale":"2021-03-24T22:03:17.494164+0000","last_undegraded":"2021-03-24T22:03:17.494164+0000","last_fullsized":"2021-03-24T22:03:17.494164+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'21","last_scrub_stamp":"2021-03-24T22:00:57.659209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:00:57.659209+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":60,"num_read_kb":42,"num_write":34,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":2568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"142","reported_epoch":"628","state":"active+clean","last_fresh":"2021-03-24T22:03:18.413464+0000","last_change":"2021-03-24T22:02:22.326409+0000","last_active":"2021-03-24T22:03:18.413464+0000","last_peered":"2021-03-24T22:03:18.413464+0000","last_clean":"2021-03-24T22:03:18.413464+0000","last_became_active":"2021-03-24T22:02:22.325996+0000","last_became_peered":"2021-03-24T22:02:22.325996+0000","last_unstale":"2021-03-24T22:03:18.413464+0000","last_undegraded":"2021-03-24T22:03:18.413464+0000","last_fullsized":"2021-03-24T22:03:18.413464+0000","mapping_epoch":571,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:38.144956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:38.144956+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"174","reported_epoch":"628","state":"active+clean","last_fresh":"2021-03-24T22:03:18.411856+0000","last_change":"2021-03-24T22:03:01.394566+0000","last_active":"2021-03-24T22:03:18.411856+0000","last_peered":"2021-03-24T22:03:18.411856+0000","last_clean":"2021-03-24T22:03:18.411856+0000","last_became_active":"2021-03-24T22:02:22.325346+0000","last_became_peered":"2021-03-24T22:02:22.325346+0000","last_unstale":"2021-03-24T22:03:18.411856+0000","last_undegraded":"2021-03-24T22:03:18.411856+0000","last_fullsized":"2021-03-24T22:03:18.411856+0000","mapping_epoch":571,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:01.394465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:03:01.394465+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"175'22","reported_seq":"1577","reported_epoch":"630","state":"active+clean","last_fresh":"2021-03-24T22:03:20.340111+0000","last_change":"2021-03-24T22:03:20.341250+0000","last_active":"2021-03-24T22:03:20.340111+0000","last_peered":"2021-03-24T22:03:20.340111+0000","last_clean":"2021-03-24T22:03:20.340111+0000","last_became_active":"2021-03-24T22:01:16.311297+0000","last_became_peered":"2021-03-24T22:01:16.311297+0000","last_unstale":"2021-03-24T22:03:20.340111+0000","last_undegraded":"2021-03-24T22:03:20.340111+0000","last_fullsized":"2021-03-24T22:03:20.340111+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'22","last_scrub_stamp":"2021-03-24T21:58:22.940875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:58:22.940875+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3283,"num_write":3,"num_write_kb":845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":313386,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"18b","length":"1"},{"start":"194","length":"1"},{"start":"1a6","length":"1"},{"start":"1cc","length":"1"},{"start":"1e8","length":"1"}]},{"pgid":"5.7","version":"0'0","reported_seq":"509","reported_epoch":"629","state":"active+clean","last_fresh":"2021-03-24T22:03:19.449641+0000","last_change":"2021-03-24T22:01:16.328244+0000","last_active":"2021-03-24T22:03:19.449641+0000","last_peered":"2021-03-24T22:03:19.449641+0000","last_clean":"2021-03-24T22:03:19.449641+0000","last_became_active":"2021-03-24T22:01:16.327948+0000","last_became_peered":"2021-03-24T22:01:16.327948+0000","last_unstale":"2021-03-24T22:03:19.449641+0000","last_undegraded":"2021-03-24T22:03:19.449641+0000","last_fullsized":"2021-03-24T22:03:19.449641+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:01:00.693679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:01:00.693679+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"544'34","reported_seq":"994","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.496176+0000","last_change":"2021-03-24T22:02:59.519639+0000","last_active":"2021-03-24T22:03:17.496176+0000","last_peered":"2021-03-24T22:03:17.496176+0000","last_clean":"2021-03-24T22:03:17.496176+0000","last_became_active":"2021-03-24T22:02:57.332868+0000","last_became_peered":"2021-03-24T22:02:57.332868+0000","last_unstale":"2021-03-24T22:03:17.496176+0000","last_undegraded":"2021-03-24T22:03:17.496176+0000","last_fullsized":"2021-03-24T22:03:17.496176+0000","mapping_epoch":606,"log_start":"69'16","ondisk_log_start":"69'16","created":62,"last_epoch_clean":607,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'34","last_scrub_stamp":"2021-03-24T22:02:59.519401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:02:59.519401+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1836,"num_objects":21,"num_object_clones":0,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":95,"num_read_kb":61,"num_write":51,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":9810,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"66'5","reported_seq":"177","reported_epoch":"628","state":"active+clean","last_fresh":"2021-03-24T22:03:18.421872+0000","last_change":"2021-03-24T22:01:57.334340+0000","last_active":"2021-03-24T22:03:18.421872+0000","last_peered":"2021-03-24T22:03:18.421872+0000","last_clean":"2021-03-24T22:03:18.421872+0000","last_became_active":"2021-03-24T22:01:57.325746+0000","last_became_peered":"2021-03-24T22:01:57.325746+0000","last_unstale":"2021-03-24T22:03:18.421872+0000","last_undegraded":"2021-03-24T22:03:18.421872+0000","last_fullsized":"2021-03-24T22:03:18.421872+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2021-03-24T21:58:58.818001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:39.286101+0000","last_clean_scrub_stamp":"2021-03-24T21:58:58.818001+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":156,"num_read_kb":151,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"602","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.406302+0000","last_change":"2021-03-24T21:59:46.941755+0000","last_active":"2021-03-24T22:03:17.406302+0000","last_peered":"2021-03-24T22:03:17.406302+0000","last_clean":"2021-03-24T22:03:17.406302+0000","last_became_active":"2021-03-24T21:59:12.281282+0000","last_became_peered":"2021-03-24T21:59:12.281282+0000","last_unstale":"2021-03-24T22:03:17.406302+0000","last_undegraded":"2021-03-24T22:03:17.406302+0000","last_fullsized":"2021-03-24T22:03:17.406302+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:46.941623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:59:46.941623+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"138","reported_epoch":"628","state":"active+clean","last_fresh":"2021-03-24T22:03:18.413297+0000","last_change":"2021-03-24T22:03:11.533500+0000","last_active":"2021-03-24T22:03:18.413297+0000","last_peered":"2021-03-24T22:03:18.413297+0000","last_clean":"2021-03-24T22:03:18.413297+0000","last_became_active":"2021-03-24T22:01:16.328444+0000","last_became_peered":"2021-03-24T22:01:16.328444+0000","last_unstale":"2021-03-24T22:03:18.413297+0000","last_undegraded":"2021-03-24T22:03:18.413297+0000","last_fullsized":"2021-03-24T22:03:18.413297+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:11.533322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:03:11.533322+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"195'69","reported_seq":"281","reported_epoch":"628","state":"active+clean","last_fresh":"2021-03-24T22:03:18.434713+0000","last_change":"2021-03-24T22:03:18.435113+0000","last_active":"2021-03-24T22:03:18.434713+0000","last_peered":"2021-03-24T22:03:18.434713+0000","last_clean":"2021-03-24T22:03:18.434713+0000","last_became_active":"2021-03-24T22:01:57.317669+0000","last_became_peered":"2021-03-24T22:01:57.317669+0000","last_unstale":"2021-03-24T22:03:18.434713+0000","last_undegraded":"2021-03-24T22:03:18.434713+0000","last_fullsized":"2021-03-24T22:03:18.434713+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'69","last_scrub_stamp":"2021-03-24T21:59:09.930390+0000","last_deep_scrub":"195'69","last_deep_scrub_stamp":"2021-03-24T21:56:49.886058+0000","last_clean_scrub_stamp":"2021-03-24T21:59:09.930390+0000","log_size":69,"ondisk_log_size":69,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1672296,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":8383,"num_write":20,"num_write_kb":7737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"139","length":"1"},{"start":"18b","length":"1"},{"start":"194","length":"1"},{"start":"1cc","length":"1"},{"start":"1d7","length":"1"},{"start":"1e8","length":"1"},{"start":"1eb","length":"1"},{"start":"1f4","length":"1"}]},{"pgid":"5.4","version":"0'0","reported_seq":"98","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.460561+0000","last_change":"2021-03-24T22:02:34.319531+0000","last_active":"2021-03-24T22:03:17.460561+0000","last_peered":"2021-03-24T22:03:17.460561+0000","last_clean":"2021-03-24T22:03:17.460561+0000","last_became_active":"2021-03-24T22:01:57.322178+0000","last_became_peered":"2021-03-24T22:01:57.322178+0000","last_unstale":"2021-03-24T22:03:17.460561+0000","last_undegraded":"2021-03-24T22:03:17.460561+0000","last_fullsized":"2021-03-24T22:03:17.460561+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:34.319421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:02:34.319421+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"544'27","reported_seq":"179","reported_epoch":"629","state":"active+clean","last_fresh":"2021-03-24T22:03:19.425514+0000","last_change":"2021-03-24T22:03:00.744262+0000","last_active":"2021-03-24T22:03:19.425514+0000","last_peered":"2021-03-24T22:03:19.425514+0000","last_clean":"2021-03-24T22:03:19.425514+0000","last_became_active":"2021-03-24T22:01:57.311205+0000","last_became_peered":"2021-03-24T22:01:57.311205+0000","last_unstale":"2021-03-24T22:03:19.425514+0000","last_undegraded":"2021-03-24T22:03:19.425514+0000","last_fullsized":"2021-03-24T22:03:19.425514+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'27","last_scrub_stamp":"2021-03-24T22:03:00.744096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:03:00.744096+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":27,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":1398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"622","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.500005+0000","last_change":"2021-03-24T22:01:43.380966+0000","last_active":"2021-03-24T22:03:17.500005+0000","last_peered":"2021-03-24T22:03:17.500005+0000","last_clean":"2021-03-24T22:03:17.500005+0000","last_became_active":"2021-03-24T22:01:43.380686+0000","last_became_peered":"2021-03-24T22:01:43.380686+0000","last_unstale":"2021-03-24T22:03:17.500005+0000","last_undegraded":"2021-03-24T22:03:17.500005+0000","last_fullsized":"2021-03-24T22:03:17.500005+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:33.533840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:33.533840+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"91","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.458565+0000","last_change":"2021-03-24T22:01:57.316110+0000","last_active":"2021-03-24T22:03:17.458565+0000","last_peered":"2021-03-24T22:03:17.458565+0000","last_clean":"2021-03-24T22:03:17.458565+0000","last_became_active":"2021-03-24T22:01:57.315729+0000","last_became_peered":"2021-03-24T22:01:57.315729+0000","last_unstale":"2021-03-24T22:03:17.458565+0000","last_undegraded":"2021-03-24T22:03:17.458565+0000","last_fullsized":"2021-03-24T22:03:17.458565+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:21.039180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:49:51.529051+0000","last_clean_scrub_stamp":"2021-03-24T21:58:21.039180+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"481","reported_epoch":"629","state":"active+clean","last_fresh":"2021-03-24T22:03:19.449291+0000","last_change":"2021-03-24T22:03:01.931322+0000","last_active":"2021-03-24T22:03:19.449291+0000","last_peered":"2021-03-24T22:03:19.449291+0000","last_clean":"2021-03-24T22:03:19.449291+0000","last_became_active":"2021-03-24T22:00:03.278687+0000","last_became_peered":"2021-03-24T22:00:03.278687+0000","last_unstale":"2021-03-24T22:03:19.449291+0000","last_undegraded":"2021-03-24T22:03:19.449291+0000","last_fullsized":"2021-03-24T22:03:19.449291+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:01.931136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:03:01.931136+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"628'259","reported_seq":"2205","reported_epoch":"628","state":"active+clean","last_fresh":"2021-03-24T22:03:18.425960+0000","last_change":"2021-03-24T22:03:18.426563+0000","last_active":"2021-03-24T22:03:18.425960+0000","last_peered":"2021-03-24T22:03:18.425960+0000","last_clean":"2021-03-24T22:03:18.425960+0000","last_became_active":"2021-03-24T22:02:31.331916+0000","last_became_peered":"2021-03-24T22:02:31.331916+0000","last_unstale":"2021-03-24T22:03:18.425960+0000","last_undegraded":"2021-03-24T22:03:18.425960+0000","last_fullsized":"2021-03-24T22:03:18.425960+0000","mapping_epoch":580,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":581,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":259,"ondisk_log_size":259,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8645599,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":114,"num_read_kb":138053,"num_write":69,"num_write_kb":30533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":35483798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"139","length":"1"},{"start":"18b","length":"1"},{"start":"194","length":"1"},{"start":"1cc","length":"1"},{"start":"1d7","length":"1"},{"start":"1e8","length":"1"},{"start":"1eb","length":"1"},{"start":"1f4","length":"1"}]},{"pgid":"5.6","version":"0'0","reported_seq":"74","reported_epoch":"630","state":"active+clean","last_fresh":"2021-03-24T22:03:20.347601+0000","last_change":"2021-03-24T22:02:55.848268+0000","last_active":"2021-03-24T22:03:20.347601+0000","last_peered":"2021-03-24T22:03:20.347601+0000","last_clean":"2021-03-24T22:03:20.347601+0000","last_became_active":"2021-03-24T22:02:22.324262+0000","last_became_peered":"2021-03-24T22:02:22.324262+0000","last_unstale":"2021-03-24T22:03:20.347601+0000","last_undegraded":"2021-03-24T22:03:20.347601+0000","last_fullsized":"2021-03-24T22:03:20.347601+0000","mapping_epoch":571,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:55.848098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:02:55.848098+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"544'32","reported_seq":"640","reported_epoch":"628","state":"active+clean","last_fresh":"2021-03-24T22:03:18.412072+0000","last_change":"2021-03-24T22:02:59.400913+0000","last_active":"2021-03-24T22:03:18.412072+0000","last_peered":"2021-03-24T22:03:18.412072+0000","last_clean":"2021-03-24T22:03:18.412072+0000","last_became_active":"2021-03-24T22:02:51.334303+0000","last_became_peered":"2021-03-24T22:02:51.334303+0000","last_unstale":"2021-03-24T22:03:18.412072+0000","last_undegraded":"2021-03-24T22:03:18.412072+0000","last_fullsized":"2021-03-24T22:03:18.412072+0000","mapping_epoch":600,"log_start":"89'22","ondisk_log_start":"89'22","created":62,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'32","last_scrub_stamp":"2021-03-24T22:02:59.400796+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2021-03-24T21:53:19.717244+0000","last_clean_scrub_stamp":"2021-03-24T22:02:59.400796+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":802,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":63,"num_read_kb":43,"num_write":35,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"609","reported_epoch":"630","state":"active+clean","last_fresh":"2021-03-24T22:03:20.346384+0000","last_change":"2021-03-24T22:03:08.699641+0000","last_active":"2021-03-24T22:03:20.346384+0000","last_peered":"2021-03-24T22:03:20.346384+0000","last_clean":"2021-03-24T22:03:20.346384+0000","last_became_active":"2021-03-24T21:51:40.351000+0000","last_became_peered":"2021-03-24T21:51:40.351000+0000","last_unstale":"2021-03-24T22:03:20.346384+0000","last_undegraded":"2021-03-24T22:03:20.346384+0000","last_fullsized":"2021-03-24T22:03:20.346384+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:08.699469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:39.286101+0000","last_clean_scrub_stamp":"2021-03-24T22:03:08.699469+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"61'1","reported_seq":"142","reported_epoch":"628","state":"active+clean","last_fresh":"2021-03-24T22:03:18.411722+0000","last_change":"2021-03-24T22:01:43.380354+0000","last_active":"2021-03-24T22:03:18.411722+0000","last_peered":"2021-03-24T22:03:18.411722+0000","last_clean":"2021-03-24T22:03:18.411722+0000","last_became_active":"2021-03-24T22:01:43.379922+0000","last_became_peered":"2021-03-24T22:01:43.379922+0000","last_unstale":"2021-03-24T22:03:18.411722+0000","last_undegraded":"2021-03-24T22:03:18.411722+0000","last_fullsized":"2021-03-24T22:03:18.411722+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2021-03-24T21:57:12.046206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:12.046206+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":436,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"96","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.461560+0000","last_change":"2021-03-24T22:03:15.443335+0000","last_active":"2021-03-24T22:03:17.461560+0000","last_peered":"2021-03-24T22:03:17.461560+0000","last_clean":"2021-03-24T22:03:17.461560+0000","last_became_active":"2021-03-24T22:01:57.318275+0000","last_became_peered":"2021-03-24T22:01:57.318275+0000","last_unstale":"2021-03-24T22:03:17.461560+0000","last_undegraded":"2021-03-24T22:03:17.461560+0000","last_fullsized":"2021-03-24T22:03:17.461560+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:15.443169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:03:15.443169+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"624'358","reported_seq":"2092","reported_epoch":"628","state":"active+clean","last_fresh":"2021-03-24T22:03:18.418074+0000","last_change":"2021-03-24T22:03:18.419005+0000","last_active":"2021-03-24T22:03:18.418074+0000","last_peered":"2021-03-24T22:03:18.418074+0000","last_clean":"2021-03-24T22:03:18.418074+0000","last_became_active":"2021-03-24T22:01:25.281813+0000","last_became_peered":"2021-03-24T22:01:25.281813+0000","last_unstale":"2021-03-24T22:03:18.418074+0000","last_undegraded":"2021-03-24T22:03:18.418074+0000","last_fullsized":"2021-03-24T22:03:18.418074+0000","mapping_epoch":514,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"616'351","last_scrub_stamp":"2021-03-24T22:03:06.473123+0000","last_deep_scrub":"616'351","last_deep_scrub_stamp":"2021-03-24T22:03:06.473123+0000","last_clean_scrub_stamp":"2021-03-24T22:03:06.473123+0000","log_size":358,"ondisk_log_size":358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6756843,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":64,"num_read_kb":67259,"num_write":40,"num_write_kb":18322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":22504555,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"139","length":"1"},{"start":"18b","length":"1"},{"start":"194","length":"1"},{"start":"1cc","length":"1"},{"start":"1d7","length":"1"},{"start":"1e8","length":"1"},{"start":"1eb","length":"1"},{"start":"1f4","length":"1"}]},{"pgid":"5.5","version":"0'0","reported_seq":"620","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.495941+0000","last_change":"2021-03-24T22:01:16.337675+0000","last_active":"2021-03-24T22:03:17.495941+0000","last_peered":"2021-03-24T22:03:17.495941+0000","last_clean":"2021-03-24T22:03:17.495941+0000","last_became_active":"2021-03-24T22:01:16.336798+0000","last_became_peered":"2021-03-24T22:01:16.336798+0000","last_unstale":"2021-03-24T22:03:17.495941+0000","last_undegraded":"2021-03-24T22:03:17.495941+0000","last_fullsized":"2021-03-24T22:03:17.495941+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:23.865909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:58:23.865909+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"544'37","reported_seq":"184","reported_epoch":"628","state":"active+clean","last_fresh":"2021-03-24T22:03:18.412700+0000","last_change":"2021-03-24T22:03:12.502901+0000","last_active":"2021-03-24T22:03:18.412700+0000","last_peered":"2021-03-24T22:03:18.412700+0000","last_clean":"2021-03-24T22:03:18.412700+0000","last_became_active":"2021-03-24T22:01:16.309129+0000","last_became_peered":"2021-03-24T22:01:16.309129+0000","last_unstale":"2021-03-24T22:03:18.412700+0000","last_undegraded":"2021-03-24T22:03:18.412700+0000","last_fullsized":"2021-03-24T22:03:18.412700+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'37","last_scrub_stamp":"2021-03-24T22:03:12.502778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:03:12.502778+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":71,"num_read_kb":48,"num_write":40,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":3816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"89'2","reported_seq":"226","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.494484+0000","last_change":"2021-03-24T22:01:43.721006+0000","last_active":"2021-03-24T22:03:17.494484+0000","last_peered":"2021-03-24T22:03:17.494484+0000","last_clean":"2021-03-24T22:03:17.494484+0000","last_became_active":"2021-03-24T22:01:43.393718+0000","last_became_peered":"2021-03-24T22:01:43.393718+0000","last_unstale":"2021-03-24T22:03:17.494484+0000","last_undegraded":"2021-03-24T22:03:17.494484+0000","last_fullsized":"2021-03-24T22:03:17.494484+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2021-03-24T21:57:58.491830+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2021-03-24T21:57:58.491830+0000","last_clean_scrub_stamp":"2021-03-24T21:57:58.491830+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":179,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"147","reported_epoch":"628","state":"active+clean","last_fresh":"2021-03-24T22:03:18.411543+0000","last_change":"2021-03-24T22:02:20.729667+0000","last_active":"2021-03-24T22:03:18.411543+0000","last_peered":"2021-03-24T22:03:18.411543+0000","last_clean":"2021-03-24T22:03:18.411543+0000","last_became_active":"2021-03-24T22:01:43.401295+0000","last_became_peered":"2021-03-24T22:01:43.401295+0000","last_unstale":"2021-03-24T22:03:18.411543+0000","last_undegraded":"2021-03-24T22:03:18.411543+0000","last_fullsized":"2021-03-24T22:03:18.411543+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:20.729422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:06.800417+0000","last_clean_scrub_stamp":"2021-03-24T22:02:20.729422+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"643","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.501510+0000","last_change":"2021-03-24T22:02:55.490903+0000","last_active":"2021-03-24T22:03:17.501510+0000","last_peered":"2021-03-24T22:03:17.501510+0000","last_clean":"2021-03-24T22:03:17.501510+0000","last_became_active":"2021-03-24T22:01:57.314372+0000","last_became_peered":"2021-03-24T22:01:57.314372+0000","last_unstale":"2021-03-24T22:03:17.501510+0000","last_undegraded":"2021-03-24T22:03:17.501510+0000","last_fullsized":"2021-03-24T22:03:17.501510+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:55.490744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:52:59.706511+0000","last_clean_scrub_stamp":"2021-03-24T22:02:55.490744+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"630'464","reported_seq":"2276","reported_epoch":"630","state":"active+clean","last_fresh":"2021-03-24T22:03:20.360885+0000","last_change":"2021-03-24T22:03:20.361309+0000","last_active":"2021-03-24T22:03:20.360885+0000","last_peered":"2021-03-24T22:03:20.360885+0000","last_clean":"2021-03-24T22:03:20.360885+0000","last_became_active":"2021-03-24T22:01:47.305050+0000","last_became_peered":"2021-03-24T22:01:47.305050+0000","last_unstale":"2021-03-24T22:03:20.360885+0000","last_undegraded":"2021-03-24T22:03:20.360885+0000","last_fullsized":"2021-03-24T22:03:20.360885+0000","mapping_epoch":536,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":537,"parent":"0.0","parent_split_bits":0,"last_scrub":"563'386","last_scrub_stamp":"2021-03-24T22:02:19.755652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:02:19.755652+0000","log_size":464,"ondisk_log_size":464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10427890,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":23,"num_read_kb":7097,"num_write":50,"num_write_kb":18896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":55849492,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"18b","length":"1"},{"start":"194","length":"1"},{"start":"1a6","length":"1"},{"start":"1cc","length":"1"},{"start":"1e8","length":"1"}]},{"pgid":"4.b","version":"544'17","reported_seq":"722","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.494413+0000","last_change":"2021-03-24T22:01:56.334554+0000","last_active":"2021-03-24T22:03:17.494413+0000","last_peered":"2021-03-24T22:03:17.494413+0000","last_clean":"2021-03-24T22:03:17.494413+0000","last_became_active":"2021-03-24T22:01:56.334081+0000","last_became_peered":"2021-03-24T22:01:56.334081+0000","last_unstale":"2021-03-24T22:03:17.494413+0000","last_undegraded":"2021-03-24T22:03:17.494413+0000","last_fullsized":"2021-03-24T22:03:17.494413+0000","mapping_epoch":545,"log_start":"91'9","ondisk_log_start":"91'9","created":62,"last_epoch_clean":546,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'14","last_scrub_stamp":"2021-03-24T21:59:05.058513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:59:05.058513+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1158,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":58,"num_read_kb":36,"num_write":30,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2316,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"618","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.502049+0000","last_change":"2021-03-24T22:01:55.431327+0000","last_active":"2021-03-24T22:03:17.502049+0000","last_peered":"2021-03-24T22:03:17.502049+0000","last_clean":"2021-03-24T22:03:17.502049+0000","last_became_active":"2021-03-24T22:01:43.390371+0000","last_became_peered":"2021-03-24T22:01:43.390371+0000","last_unstale":"2021-03-24T22:03:17.502049+0000","last_undegraded":"2021-03-24T22:03:17.502049+0000","last_fullsized":"2021-03-24T22:03:17.502049+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:01:55.431129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T22:01:55.431129+0000","last_clean_scrub_stamp":"2021-03-24T22:01:55.431129+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"596","reported_epoch":"630","state":"active+clean","last_fresh":"2021-03-24T22:03:20.346176+0000","last_change":"2021-03-24T22:02:21.716625+0000","last_active":"2021-03-24T22:03:20.346176+0000","last_peered":"2021-03-24T22:03:20.346176+0000","last_clean":"2021-03-24T22:03:20.346176+0000","last_became_active":"2021-03-24T21:51:49.411921+0000","last_became_peered":"2021-03-24T21:51:49.411921+0000","last_unstale":"2021-03-24T22:03:20.346176+0000","last_undegraded":"2021-03-24T22:03:20.346176+0000","last_fullsized":"2021-03-24T22:03:20.346176+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:21.716448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:02:21.716448+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"645","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.406182+0000","last_change":"2021-03-24T22:02:22.324168+0000","last_active":"2021-03-24T22:03:17.406182+0000","last_peered":"2021-03-24T22:03:17.406182+0000","last_clean":"2021-03-24T22:03:17.406182+0000","last_became_active":"2021-03-24T22:02:22.323876+0000","last_became_peered":"2021-03-24T22:02:22.323876+0000","last_unstale":"2021-03-24T22:03:17.406182+0000","last_undegraded":"2021-03-24T22:03:17.406182+0000","last_fullsized":"2021-03-24T22:03:17.406182+0000","mapping_epoch":571,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:00.033461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:35.636299+0000","last_clean_scrub_stamp":"2021-03-24T21:59:00.033461+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"625'286","reported_seq":"2100","reported_epoch":"630","state":"active+clean","last_fresh":"2021-03-24T22:03:20.347003+0000","last_change":"2021-03-24T22:03:20.347003+0000","last_active":"2021-03-24T22:03:20.347003+0000","last_peered":"2021-03-24T22:03:20.347003+0000","last_clean":"2021-03-24T22:03:20.347003+0000","last_became_active":"2021-03-24T22:02:01.325483+0000","last_became_peered":"2021-03-24T22:02:01.325483+0000","last_unstale":"2021-03-24T22:03:20.347003+0000","last_undegraded":"2021-03-24T22:03:20.347003+0000","last_fullsized":"2021-03-24T22:03:20.347003+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"619'282","last_scrub_stamp":"2021-03-24T22:03:12.788348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:03:12.788348+0000","log_size":286,"ondisk_log_size":286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5165516,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":53,"num_read_kb":66903,"num_write":42,"num_write_kb":11344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":21446378,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"18b","length":"1"},{"start":"194","length":"1"},{"start":"1a6","length":"1"},{"start":"1cc","length":"1"},{"start":"1e8","length":"1"}]},{"pgid":"4.a","version":"544'45","reported_seq":"457","reported_epoch":"629","state":"active+clean","last_fresh":"2021-03-24T22:03:19.423922+0000","last_change":"2021-03-24T22:02:11.306107+0000","last_active":"2021-03-24T22:03:19.423922+0000","last_peered":"2021-03-24T22:03:19.423922+0000","last_clean":"2021-03-24T22:03:19.423922+0000","last_became_active":"2021-03-24T22:02:11.305345+0000","last_became_peered":"2021-03-24T22:02:11.305345+0000","last_unstale":"2021-03-24T22:03:19.423922+0000","last_undegraded":"2021-03-24T22:03:19.423922+0000","last_fullsized":"2021-03-24T22:03:19.423922+0000","mapping_epoch":560,"log_start":"69'23","ondisk_log_start":"69'23","created":62,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'23","last_scrub_stamp":"2021-03-24T22:00:23.343879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:00:23.343879+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1466,"num_objects":21,"num_object_clones":0,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":104,"num_read_kb":71,"num_write":60,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":4398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":"318","reported_epoch":"629","state":"active+clean","last_fresh":"2021-03-24T22:03:19.447690+0000","last_change":"2021-03-24T22:03:00.965270+0000","last_active":"2021-03-24T22:03:19.447690+0000","last_peered":"2021-03-24T22:03:19.447690+0000","last_clean":"2021-03-24T22:03:19.447690+0000","last_became_active":"2021-03-24T22:01:57.323503+0000","last_became_peered":"2021-03-24T22:01:57.323503+0000","last_unstale":"2021-03-24T22:03:19.447690+0000","last_undegraded":"2021-03-24T22:03:19.447690+0000","last_fullsized":"2021-03-24T22:03:19.447690+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:00.965024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:03:00.965024+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"69","reported_epoch":"630","state":"active+clean","last_fresh":"2021-03-24T22:03:20.347098+0000","last_change":"2021-03-24T22:02:22.323418+0000","last_active":"2021-03-24T22:03:20.347098+0000","last_peered":"2021-03-24T22:03:20.347098+0000","last_clean":"2021-03-24T22:03:20.347098+0000","last_became_active":"2021-03-24T22:02:22.323020+0000","last_became_peered":"2021-03-24T22:02:22.323020+0000","last_unstale":"2021-03-24T22:03:20.347098+0000","last_undegraded":"2021-03-24T22:03:20.347098+0000","last_fullsized":"2021-03-24T22:03:20.347098+0000","mapping_epoch":571,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:01:40.445404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:01:40.445404+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"389","reported_epoch":"630","state":"active+clean","last_fresh":"2021-03-24T22:03:20.346101+0000","last_change":"2021-03-24T22:01:57.311688+0000","last_active":"2021-03-24T22:03:20.346101+0000","last_peered":"2021-03-24T22:03:20.346101+0000","last_clean":"2021-03-24T22:03:20.346101+0000","last_became_active":"2021-03-24T22:01:57.311325+0000","last_became_peered":"2021-03-24T22:01:57.311325+0000","last_unstale":"2021-03-24T22:03:20.346101+0000","last_undegraded":"2021-03-24T22:03:20.346101+0000","last_fullsized":"2021-03-24T22:03:20.346101+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:42.813883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:59:42.813883+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"628'409","reported_seq":"2148","reported_epoch":"628","state":"active+clean","last_fresh":"2021-03-24T22:03:18.422475+0000","last_change":"2021-03-24T22:03:18.423169+0000","last_active":"2021-03-24T22:03:18.422475+0000","last_peered":"2021-03-24T22:03:18.422475+0000","last_clean":"2021-03-24T22:03:18.422475+0000","last_became_active":"2021-03-24T22:01:22.309266+0000","last_became_peered":"2021-03-24T22:01:22.309266+0000","last_unstale":"2021-03-24T22:03:18.422475+0000","last_undegraded":"2021-03-24T22:03:18.422475+0000","last_fullsized":"2021-03-24T22:03:18.422475+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"609'395","last_scrub_stamp":"2021-03-24T22:03:04.501653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:03:04.501653+0000","log_size":409,"ondisk_log_size":409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12343055,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":64,"num_read_kb":81138,"num_write":41,"num_write_kb":13318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":11840390,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"139","length":"1"},{"start":"18b","length":"1"},{"start":"194","length":"1"},{"start":"1cc","length":"1"},{"start":"1d7","length":"1"},{"start":"1e8","length":"1"},{"start":"1eb","length":"1"},{"start":"1f4","length":"1"}]},{"pgid":"5.8","version":"0'0","reported_seq":"629","reported_epoch":"630","state":"active+clean","last_fresh":"2021-03-24T22:03:20.345829+0000","last_change":"2021-03-24T22:02:58.801826+0000","last_active":"2021-03-24T22:03:20.345829+0000","last_peered":"2021-03-24T22:03:20.345829+0000","last_clean":"2021-03-24T22:03:20.345829+0000","last_became_active":"2021-03-24T22:01:16.329336+0000","last_became_peered":"2021-03-24T22:01:16.329336+0000","last_unstale":"2021-03-24T22:03:20.345829+0000","last_undegraded":"2021-03-24T22:03:20.345829+0000","last_fullsized":"2021-03-24T22:03:20.345829+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:58.801650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:02:58.801650+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"544'38","reported_seq":"303","reported_epoch":"628","state":"active+clean","last_fresh":"2021-03-24T22:03:18.411316+0000","last_change":"2021-03-24T22:03:03.442490+0000","last_active":"2021-03-24T22:03:18.411316+0000","last_peered":"2021-03-24T22:03:18.411316+0000","last_clean":"2021-03-24T22:03:18.411316+0000","last_became_active":"2021-03-24T22:01:27.288204+0000","last_became_peered":"2021-03-24T22:01:27.288204+0000","last_unstale":"2021-03-24T22:03:18.411316+0000","last_undegraded":"2021-03-24T22:03:18.411316+0000","last_fullsized":"2021-03-24T22:03:18.411316+0000","mapping_epoch":516,"log_start":"69'20","ondisk_log_start":"69'20","created":62,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'38","last_scrub_stamp":"2021-03-24T22:03:03.442382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:03:03.442382+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1582,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":101,"num_read_kb":67,"num_write":56,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1582,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":"91","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.462746+0000","last_change":"2021-03-24T22:01:57.326649+0000","last_active":"2021-03-24T22:03:17.462746+0000","last_peered":"2021-03-24T22:03:17.462746+0000","last_clean":"2021-03-24T22:03:17.462746+0000","last_became_active":"2021-03-24T22:01:57.325643+0000","last_became_peered":"2021-03-24T22:01:57.325643+0000","last_unstale":"2021-03-24T22:03:17.462746+0000","last_undegraded":"2021-03-24T22:03:17.462746+0000","last_fullsized":"2021-03-24T22:03:17.462746+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:12.939226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:59:12.939226+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"642","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.500658+0000","last_change":"2021-03-24T22:03:03.347078+0000","last_active":"2021-03-24T22:03:17.500658+0000","last_peered":"2021-03-24T22:03:17.500658+0000","last_clean":"2021-03-24T22:03:17.500658+0000","last_became_active":"2021-03-24T22:03:03.346637+0000","last_became_peered":"2021-03-24T22:03:03.346637+0000","last_unstale":"2021-03-24T22:03:17.500658+0000","last_undegraded":"2021-03-24T22:03:17.500658+0000","last_fullsized":"2021-03-24T22:03:17.500658+0000","mapping_epoch":612,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":613,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:18.322187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:22.724840+0000","last_clean_scrub_stamp":"2021-03-24T22:00:18.322187+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"628'285","reported_seq":"1912","reported_epoch":"628","state":"active+clean","last_fresh":"2021-03-24T22:03:18.435446+0000","last_change":"2021-03-24T22:03:18.435882+0000","last_active":"2021-03-24T22:03:18.435446+0000","last_peered":"2021-03-24T22:03:18.435446+0000","last_clean":"2021-03-24T22:03:18.435446+0000","last_became_active":"2021-03-24T22:00:03.286696+0000","last_became_peered":"2021-03-24T22:00:03.286696+0000","last_unstale":"2021-03-24T22:03:18.435446+0000","last_undegraded":"2021-03-24T22:03:18.435446+0000","last_fullsized":"2021-03-24T22:03:18.435446+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"624'277","last_scrub_stamp":"2021-03-24T22:03:14.428074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:03:14.428074+0000","log_size":285,"ondisk_log_size":285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15709730,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":2774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"139","length":"1"},{"start":"18b","length":"1"},{"start":"194","length":"1"},{"start":"1cc","length":"1"},{"start":"1d7","length":"1"},{"start":"1e8","length":"1"},{"start":"1eb","length":"1"},{"start":"1f4","length":"1"}]},{"pgid":"5.9","version":"548'11","reported_seq":"112","reported_epoch":"628","state":"active+clean","last_fresh":"2021-03-24T22:03:18.423759+0000","last_change":"2021-03-24T22:01:57.504107+0000","last_active":"2021-03-24T22:03:18.423759+0000","last_peered":"2021-03-24T22:03:18.423759+0000","last_clean":"2021-03-24T22:03:18.423759+0000","last_became_active":"2021-03-24T22:01:57.313563+0000","last_became_peered":"2021-03-24T22:01:57.313563+0000","last_unstale":"2021-03-24T22:03:18.423759+0000","last_undegraded":"2021-03-24T22:03:18.423759+0000","last_fullsized":"2021-03-24T22:03:18.423759+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'4","last_scrub_stamp":"2021-03-24T21:59:06.906705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:59:06.906705+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"544'46","reported_seq":"702","reported_epoch":"629","state":"active+remapped+backfill_toofull","last_fresh":"2021-03-24T22:03:19.446891+0000","last_change":"2021-03-24T22:02:57.332413+0000","last_active":"2021-03-24T22:03:19.446891+0000","last_peered":"2021-03-24T22:03:19.446891+0000","last_clean":"2021-03-24T22:01:41.303005+0000","last_became_active":"2021-03-24T22:01:57.327903+0000","last_became_peered":"2021-03-24T22:01:57.327903+0000","last_unstale":"2021-03-24T22:03:19.446891+0000","last_undegraded":"2021-03-24T22:03:19.446891+0000","last_fullsized":"2021-03-24T22:03:19.446891+0000","mapping_epoch":546,"log_start":"97'28","ondisk_log_start":"97'28","created":62,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'28","last_scrub_stamp":"2021-03-24T21:58:20.925603+0000","last_deep_scrub":"97'28","last_deep_scrub_stamp":"2021-03-24T21:54:19.157265+0000","last_clean_scrub_stamp":"2021-03-24T21:58:20.925603+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1616,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":40,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":97,"num_read_kb":64,"num_write":53,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":3692,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,4,3],"avail_no_missing":["5","3","4"],"object_location_counts":[{"shards":"3,4,5","objects":20}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"89'4","reported_seq":"97","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.462492+0000","last_change":"2021-03-24T22:01:57.336372+0000","last_active":"2021-03-24T22:03:17.462492+0000","last_peered":"2021-03-24T22:03:17.462492+0000","last_clean":"2021-03-24T22:03:17.462492+0000","last_became_active":"2021-03-24T22:01:57.316325+0000","last_became_peered":"2021-03-24T22:01:57.316325+0000","last_unstale":"2021-03-24T22:03:17.462492+0000","last_undegraded":"2021-03-24T22:03:17.462492+0000","last_fullsized":"2021-03-24T22:03:17.462492+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'4","last_scrub_stamp":"2021-03-24T21:58:25.969865+0000","last_deep_scrub":"89'4","last_deep_scrub_stamp":"2021-03-24T21:53:19.926637+0000","last_clean_scrub_stamp":"2021-03-24T21:58:25.969865+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"112","reported_epoch":"629","state":"active+clean","last_fresh":"2021-03-24T22:03:19.424756+0000","last_change":"2021-03-24T22:01:51.981589+0000","last_active":"2021-03-24T22:03:19.424756+0000","last_peered":"2021-03-24T22:03:19.424756+0000","last_clean":"2021-03-24T22:03:19.424756+0000","last_became_active":"2021-03-24T22:01:43.394043+0000","last_became_peered":"2021-03-24T22:01:43.394043+0000","last_unstale":"2021-03-24T22:03:19.424756+0000","last_undegraded":"2021-03-24T22:03:19.424756+0000","last_fullsized":"2021-03-24T22:03:19.424756+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:01:51.981423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:01:51.981423+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"272'275","reported_seq":"1370","reported_epoch":"628","state":"active+clean","last_fresh":"2021-03-24T22:03:18.413921+0000","last_change":"2021-03-24T22:03:18.413921+0000","last_active":"2021-03-24T22:03:18.413921+0000","last_peered":"2021-03-24T22:03:18.413921+0000","last_clean":"2021-03-24T22:03:18.413921+0000","last_became_active":"2021-03-24T22:01:46.306345+0000","last_became_peered":"2021-03-24T22:01:46.306345+0000","last_unstale":"2021-03-24T22:03:18.413921+0000","last_undegraded":"2021-03-24T22:03:18.413921+0000","last_fullsized":"2021-03-24T22:03:18.413921+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'275","last_scrub_stamp":"2021-03-24T21:58:23.942036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:58:23.942036+0000","log_size":275,"ondisk_log_size":275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8857338,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":114,"num_read_kb":132714,"num_write":52,"num_write_kb":22259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":12587455,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"139","length":"1"},{"start":"18b","length":"1"},{"start":"194","length":"1"},{"start":"1cc","length":"1"},{"start":"1d7","length":"1"},{"start":"1e8","length":"1"},{"start":"1eb","length":"1"},{"start":"1f4","length":"1"}]},{"pgid":"3.10","version":"0'0","reported_seq":"114","reported_epoch":"629","state":"active+clean","last_fresh":"2021-03-24T22:03:19.423349+0000","last_change":"2021-03-24T22:01:56.876290+0000","last_active":"2021-03-24T22:03:19.423349+0000","last_peered":"2021-03-24T22:03:19.423349+0000","last_clean":"2021-03-24T22:03:19.423349+0000","last_became_active":"2021-03-24T22:01:43.392580+0000","last_became_peered":"2021-03-24T22:01:43.392580+0000","last_unstale":"2021-03-24T22:03:19.423349+0000","last_undegraded":"2021-03-24T22:03:19.423349+0000","last_fullsized":"2021-03-24T22:03:19.423349+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:01:56.876119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:01:56.876119+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"520","reported_epoch":"629","state":"active+clean","last_fresh":"2021-03-24T22:03:19.448939+0000","last_change":"2021-03-24T22:01:16.312978+0000","last_active":"2021-03-24T22:03:19.448939+0000","last_peered":"2021-03-24T22:03:19.448939+0000","last_clean":"2021-03-24T22:03:19.448939+0000","last_became_active":"2021-03-24T22:01:16.311233+0000","last_became_peered":"2021-03-24T22:01:16.311233+0000","last_unstale":"2021-03-24T22:03:19.448939+0000","last_undegraded":"2021-03-24T22:03:19.448939+0000","last_fullsized":"2021-03-24T22:03:19.448939+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:26.392232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T22:00:26.392232+0000","last_clean_scrub_stamp":"2021-03-24T22:00:26.392232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"96","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.462174+0000","last_change":"2021-03-24T22:03:01.351580+0000","last_active":"2021-03-24T22:03:17.462174+0000","last_peered":"2021-03-24T22:03:17.462174+0000","last_clean":"2021-03-24T22:03:17.462174+0000","last_became_active":"2021-03-24T22:01:57.312376+0000","last_became_peered":"2021-03-24T22:01:57.312376+0000","last_unstale":"2021-03-24T22:03:17.462174+0000","last_undegraded":"2021-03-24T22:03:17.462174+0000","last_fullsized":"2021-03-24T22:03:17.462174+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:01.351474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:03:01.351474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.14","version":"628'728","reported_seq":"2414","reported_epoch":"628","state":"active+clean","last_fresh":"2021-03-24T22:03:18.430511+0000","last_change":"2021-03-24T22:03:18.431738+0000","last_active":"2021-03-24T22:03:18.430511+0000","last_peered":"2021-03-24T22:03:18.430511+0000","last_clean":"2021-03-24T22:03:18.430511+0000","last_became_active":"2021-03-24T22:00:03.281974+0000","last_became_peered":"2021-03-24T22:00:03.281974+0000","last_unstale":"2021-03-24T22:03:18.430511+0000","last_undegraded":"2021-03-24T22:03:18.430511+0000","last_fullsized":"2021-03-24T22:03:18.430511+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":728,"ondisk_log_size":728,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":42537783,"num_objects":17,"num_object_clones":14,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":287,"num_read_kb":285899,"num_write":208,"num_write_kb":79118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":33311730,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"139","length":"1"},{"start":"18b","length":"1"},{"start":"194","length":"1"},{"start":"1cc","length":"1"},{"start":"1d7","length":"1"},{"start":"1e8","length":"1"},{"start":"1eb","length":"1"},{"start":"1f4","length":"1"}]},{"pgid":"3.11","version":"0'0","reported_seq":"121","reported_epoch":"629","state":"active+clean","last_fresh":"2021-03-24T22:03:19.424568+0000","last_change":"2021-03-24T22:02:22.796277+0000","last_active":"2021-03-24T22:03:19.424568+0000","last_peered":"2021-03-24T22:03:19.424568+0000","last_clean":"2021-03-24T22:03:19.424568+0000","last_became_active":"2021-03-24T22:01:57.304423+0000","last_became_peered":"2021-03-24T22:01:57.304423+0000","last_unstale":"2021-03-24T22:03:19.424568+0000","last_undegraded":"2021-03-24T22:03:19.424568+0000","last_fullsized":"2021-03-24T22:03:19.424568+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:22.796161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:02:22.796161+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"643","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.502631+0000","last_change":"2021-03-24T22:02:56.523447+0000","last_active":"2021-03-24T22:03:17.502631+0000","last_peered":"2021-03-24T22:03:17.502631+0000","last_clean":"2021-03-24T22:03:17.502631+0000","last_became_active":"2021-03-24T22:01:57.315711+0000","last_became_peered":"2021-03-24T22:01:57.315711+0000","last_unstale":"2021-03-24T22:03:17.502631+0000","last_undegraded":"2021-03-24T22:03:17.502631+0000","last_fullsized":"2021-03-24T22:03:17.502631+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:56.523309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:55.493713+0000","last_clean_scrub_stamp":"2021-03-24T22:02:56.523309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"630","reported_epoch":"630","state":"active+clean","last_fresh":"2021-03-24T22:03:20.346972+0000","last_change":"2021-03-24T22:03:09.686125+0000","last_active":"2021-03-24T22:03:20.346972+0000","last_peered":"2021-03-24T22:03:20.346972+0000","last_clean":"2021-03-24T22:03:20.346972+0000","last_became_active":"2021-03-24T22:01:57.312185+0000","last_became_peered":"2021-03-24T22:01:57.312185+0000","last_unstale":"2021-03-24T22:03:20.346972+0000","last_undegraded":"2021-03-24T22:03:20.346972+0000","last_fullsized":"2021-03-24T22:03:20.346972+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:09.686047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:03:09.686047+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"628'244","reported_seq":"2049","reported_epoch":"628","state":"active+clean","last_fresh":"2021-03-24T22:03:18.438228+0000","last_change":"2021-03-24T22:03:18.438519+0000","last_active":"2021-03-24T22:03:18.438228+0000","last_peered":"2021-03-24T22:03:18.438228+0000","last_clean":"2021-03-24T22:03:18.438228+0000","last_became_active":"2021-03-24T22:02:00.309364+0000","last_became_peered":"2021-03-24T22:02:00.309364+0000","last_unstale":"2021-03-24T22:03:18.438228+0000","last_undegraded":"2021-03-24T22:03:18.438228+0000","last_fullsized":"2021-03-24T22:03:18.438228+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":5,"last_scrub":"447'140","last_scrub_stamp":"2021-03-24T22:00:17.991497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:00:17.991497+0000","log_size":244,"ondisk_log_size":244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3832062,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":48,"num_read_kb":65320,"num_write":34,"num_write_kb":12965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":76697653,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"139","length":"1"},{"start":"18b","length":"1"},{"start":"194","length":"1"},{"start":"1cc","length":"1"},{"start":"1d7","length":"1"},{"start":"1e8","length":"1"},{"start":"1eb","length":"1"},{"start":"1f4","length":"1"}]},{"pgid":"4.15","version":"544'19","reported_seq":"636","reported_epoch":"629","state":"active+clean","last_fresh":"2021-03-24T22:03:19.448013+0000","last_change":"2021-03-24T22:01:58.033012+0000","last_active":"2021-03-24T22:03:19.448013+0000","last_peered":"2021-03-24T22:03:19.448013+0000","last_clean":"2021-03-24T22:03:19.448013+0000","last_became_active":"2021-03-24T22:01:57.323247+0000","last_became_peered":"2021-03-24T22:01:57.323247+0000","last_unstale":"2021-03-24T22:03:19.448013+0000","last_undegraded":"2021-03-24T22:03:19.448013+0000","last_fullsized":"2021-03-24T22:03:19.448013+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2021-03-24T22:00:29.462476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:00:29.462476+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":37,"num_read_kb":23,"num_write":20,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":2936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":"629","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.406429+0000","last_change":"2021-03-24T22:01:57.316563+0000","last_active":"2021-03-24T22:03:17.406429+0000","last_peered":"2021-03-24T22:03:17.406429+0000","last_clean":"2021-03-24T22:03:17.406429+0000","last_became_active":"2021-03-24T22:01:57.315967+0000","last_became_peered":"2021-03-24T22:01:57.315967+0000","last_unstale":"2021-03-24T22:03:17.406429+0000","last_undegraded":"2021-03-24T22:03:17.406429+0000","last_fullsized":"2021-03-24T22:03:17.406429+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:28.482500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:05.473348+0000","last_clean_scrub_stamp":"2021-03-24T22:00:28.482500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"573'12","reported_seq":"822","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.502614+0000","last_change":"2021-03-24T22:03:02.456113+0000","last_active":"2021-03-24T22:03:17.502614+0000","last_peered":"2021-03-24T22:03:17.502614+0000","last_clean":"2021-03-24T22:03:17.502614+0000","last_became_active":"2021-03-24T22:02:22.326377+0000","last_became_peered":"2021-03-24T22:02:22.326377+0000","last_unstale":"2021-03-24T22:03:17.502614+0000","last_undegraded":"2021-03-24T22:03:17.502614+0000","last_fullsized":"2021-03-24T22:03:17.502614+0000","mapping_epoch":571,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":0,"last_scrub":"573'12","last_scrub_stamp":"2021-03-24T22:03:02.455946+0000","last_deep_scrub":"86'3","last_deep_scrub_stamp":"2021-03-24T21:52:56.678111+0000","last_clean_scrub_stamp":"2021-03-24T22:03:02.455946+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"565","reported_epoch":"630","state":"active+clean","last_fresh":"2021-03-24T22:03:20.347832+0000","last_change":"2021-03-24T22:00:59.303976+0000","last_active":"2021-03-24T22:03:20.347832+0000","last_peered":"2021-03-24T22:03:20.347832+0000","last_clean":"2021-03-24T22:03:20.347832+0000","last_became_active":"2021-03-24T22:00:49.302172+0000","last_became_peered":"2021-03-24T22:00:49.302172+0000","last_unstale":"2021-03-24T22:03:20.347832+0000","last_undegraded":"2021-03-24T22:03:20.347832+0000","last_fullsized":"2021-03-24T22:03:20.347832+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:59.303897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:00:59.303897+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.16","version":"628'386","reported_seq":"2245","reported_epoch":"628","state":"active+clean","last_fresh":"2021-03-24T22:03:18.426738+0000","last_change":"2021-03-24T22:03:18.427600+0000","last_active":"2021-03-24T22:03:18.426738+0000","last_peered":"2021-03-24T22:03:18.426738+0000","last_clean":"2021-03-24T22:03:18.426738+0000","last_became_active":"2021-03-24T22:02:08.322570+0000","last_became_peered":"2021-03-24T22:02:08.322570+0000","last_unstale":"2021-03-24T22:03:18.426738+0000","last_undegraded":"2021-03-24T22:03:18.426738+0000","last_fullsized":"2021-03-24T22:03:18.426738+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"606'362","last_scrub_stamp":"2021-03-24T22:02:57.478372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:02:57.478372+0000","log_size":386,"ondisk_log_size":386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11617203,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":80,"num_read_kb":90887,"num_write":54,"num_write_kb":13373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":30734442,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"139","length":"1"},{"start":"18b","length":"1"},{"start":"194","length":"1"},{"start":"1cc","length":"1"},{"start":"1d7","length":"1"},{"start":"1e8","length":"1"},{"start":"1eb","length":"1"},{"start":"1f4","length":"1"}]},{"pgid":"4.14","version":"544'24","reported_seq":"812","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.502856+0000","last_change":"2021-03-24T22:01:59.424897+0000","last_active":"2021-03-24T22:03:17.502856+0000","last_peered":"2021-03-24T22:03:17.502856+0000","last_clean":"2021-03-24T22:03:17.502856+0000","last_became_active":"2021-03-24T22:01:57.324895+0000","last_became_peered":"2021-03-24T22:01:57.324895+0000","last_unstale":"2021-03-24T22:03:17.502856+0000","last_undegraded":"2021-03-24T22:03:17.502856+0000","last_fullsized":"2021-03-24T22:03:17.502856+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'14","last_scrub_stamp":"2021-03-24T21:58:57.435358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:58:57.435358+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":42,"num_read_kb":29,"num_write":24,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":3744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":"91","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.462232+0000","last_change":"2021-03-24T22:01:57.320074+0000","last_active":"2021-03-24T22:03:17.462232+0000","last_peered":"2021-03-24T22:03:17.462232+0000","last_clean":"2021-03-24T22:03:17.462232+0000","last_became_active":"2021-03-24T22:01:57.318685+0000","last_became_peered":"2021-03-24T22:01:57.318685+0000","last_unstale":"2021-03-24T22:03:17.462232+0000","last_undegraded":"2021-03-24T22:03:17.462232+0000","last_fullsized":"2021-03-24T22:03:17.462232+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:15.949538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:59:15.949538+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"508'8","reported_seq":"636","reported_epoch":"629","state":"active+clean","last_fresh":"2021-03-24T22:03:19.448329+0000","last_change":"2021-03-24T22:01:16.331383+0000","last_active":"2021-03-24T22:03:19.448329+0000","last_peered":"2021-03-24T22:03:19.448329+0000","last_clean":"2021-03-24T22:03:19.448329+0000","last_became_active":"2021-03-24T22:01:16.313639+0000","last_became_peered":"2021-03-24T22:01:16.313639+0000","last_unstale":"2021-03-24T22:03:19.448329+0000","last_undegraded":"2021-03-24T22:03:19.448329+0000","last_fullsized":"2021-03-24T22:03:19.448329+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"445'6","last_scrub_stamp":"2021-03-24T22:00:24.314895+0000","last_deep_scrub":"86'3","last_deep_scrub_stamp":"2021-03-24T21:53:08.815163+0000","last_clean_scrub_stamp":"2021-03-24T22:00:24.314895+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"627","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.495957+0000","last_change":"2021-03-24T22:01:57.317084+0000","last_active":"2021-03-24T22:03:17.495957+0000","last_peered":"2021-03-24T22:03:17.495957+0000","last_clean":"2021-03-24T22:03:17.495957+0000","last_became_active":"2021-03-24T22:01:57.316729+0000","last_became_peered":"2021-03-24T22:01:57.316729+0000","last_unstale":"2021-03-24T22:03:17.495957+0000","last_undegraded":"2021-03-24T22:03:17.495957+0000","last_fullsized":"2021-03-24T22:03:17.495957+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:58.768675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:57:58.768675+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.17","version":"628'929","reported_seq":"2928","reported_epoch":"628","state":"active+clean","last_fresh":"2021-03-24T22:03:18.444225+0000","last_change":"2021-03-24T22:03:18.442965+0000","last_active":"2021-03-24T22:03:18.444225+0000","last_peered":"2021-03-24T22:03:18.444225+0000","last_clean":"2021-03-24T22:03:18.444225+0000","last_became_active":"2021-03-24T22:02:09.319642+0000","last_became_peered":"2021-03-24T22:02:09.319642+0000","last_unstale":"2021-03-24T22:03:18.444225+0000","last_undegraded":"2021-03-24T22:03:18.444225+0000","last_fullsized":"2021-03-24T22:03:18.444225+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"452'567","last_scrub_stamp":"2021-03-24T22:00:27.169710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:00:27.169710+0000","log_size":929,"ondisk_log_size":929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35589367,"num_objects":21,"num_object_clones":17,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":2,"num_read":254,"num_read_kb":279321,"num_write":155,"num_write_kb":55286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":242,"num_bytes_recovered":147823286,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"139","length":"1"},{"start":"18b","length":"1"},{"start":"194","length":"1"},{"start":"1cc","length":"1"},{"start":"1d7","length":"1"},{"start":"1e8","length":"1"},{"start":"1eb","length":"1"},{"start":"1f4","length":"1"}]},{"pgid":"4.13","version":"584'32","reported_seq":"153","reported_epoch":"629","state":"active+clean","last_fresh":"2021-03-24T22:03:19.424704+0000","last_change":"2021-03-24T22:02:34.813717+0000","last_active":"2021-03-24T22:03:19.424704+0000","last_peered":"2021-03-24T22:03:19.424704+0000","last_clean":"2021-03-24T22:03:19.424704+0000","last_became_active":"2021-03-24T22:01:43.393580+0000","last_became_peered":"2021-03-24T22:01:43.393580+0000","last_unstale":"2021-03-24T22:03:19.424704+0000","last_undegraded":"2021-03-24T22:03:19.424704+0000","last_fullsized":"2021-03-24T22:03:19.424704+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"584'32","last_scrub_stamp":"2021-03-24T22:02:34.813628+0000","last_deep_scrub":"584'32","last_deep_scrub_stamp":"2021-03-24T22:02:34.813628+0000","last_clean_scrub_stamp":"2021-03-24T22:02:34.813628+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":29,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":2208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":"91","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.460689+0000","last_change":"2021-03-24T22:01:57.307496+0000","last_active":"2021-03-24T22:03:17.460689+0000","last_peered":"2021-03-24T22:03:17.460689+0000","last_clean":"2021-03-24T22:03:17.460689+0000","last_became_active":"2021-03-24T22:01:57.307018+0000","last_became_peered":"2021-03-24T22:01:57.307018+0000","last_unstale":"2021-03-24T22:03:17.460689+0000","last_undegraded":"2021-03-24T22:03:17.460689+0000","last_fullsized":"2021-03-24T22:03:17.460689+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:29.716186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:58:29.716186+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"174","reported_epoch":"628","state":"active+clean","last_fresh":"2021-03-24T22:03:18.411393+0000","last_change":"2021-03-24T22:02:27.315123+0000","last_active":"2021-03-24T22:03:18.411393+0000","last_peered":"2021-03-24T22:03:18.411393+0000","last_clean":"2021-03-24T22:03:18.411393+0000","last_became_active":"2021-03-24T22:02:27.314405+0000","last_became_peered":"2021-03-24T22:02:27.314405+0000","last_unstale":"2021-03-24T22:03:18.411393+0000","last_undegraded":"2021-03-24T22:03:18.411393+0000","last_fullsized":"2021-03-24T22:03:18.411393+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:01:55.618238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:57:31.118050+0000","last_clean_scrub_stamp":"2021-03-24T22:01:55.618238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"597","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.501251+0000","last_change":"2021-03-24T22:00:21.298498+0000","last_active":"2021-03-24T22:03:17.501251+0000","last_peered":"2021-03-24T22:03:17.501251+0000","last_clean":"2021-03-24T22:03:17.501251+0000","last_became_active":"2021-03-24T22:00:03.282227+0000","last_became_peered":"2021-03-24T22:00:03.282227+0000","last_unstale":"2021-03-24T22:03:17.501251+0000","last_undegraded":"2021-03-24T22:03:17.501251+0000","last_fullsized":"2021-03-24T22:03:17.501251+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:21.298330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:23.707914+0000","last_clean_scrub_stamp":"2021-03-24T22:00:21.298330+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.10","version":"628'124","reported_seq":"254","reported_epoch":"628","state":"active+clean","last_fresh":"2021-03-24T22:03:18.425527+0000","last_change":"2021-03-24T22:03:18.425854+0000","last_active":"2021-03-24T22:03:18.425527+0000","last_peered":"2021-03-24T22:03:18.425527+0000","last_clean":"2021-03-24T22:03:18.425527+0000","last_became_active":"2021-03-24T22:02:27.314608+0000","last_became_peered":"2021-03-24T22:02:27.314608+0000","last_unstale":"2021-03-24T22:03:18.425527+0000","last_undegraded":"2021-03-24T22:03:18.425527+0000","last_fullsized":"2021-03-24T22:03:18.425527+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":5,"last_scrub":"579'81","last_scrub_stamp":"2021-03-24T22:02:30.763305+0000","last_deep_scrub":"277'24","last_deep_scrub_stamp":"2021-03-24T21:57:28.119420+0000","last_clean_scrub_stamp":"2021-03-24T22:02:30.763305+0000","log_size":124,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8314122,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":57,"num_read_kb":50980,"num_write":37,"num_write_kb":11772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":15290726,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"139","length":"1"},{"start":"18b","length":"1"},{"start":"194","length":"1"},{"start":"1cc","length":"1"},{"start":"1d7","length":"1"},{"start":"1e8","length":"1"},{"start":"1eb","length":"1"},{"start":"1f4","length":"1"}]},{"pgid":"4.12","version":"544'21","reported_seq":"200","reported_epoch":"628","state":"active+clean","last_fresh":"2021-03-24T22:03:18.412304+0000","last_change":"2021-03-24T22:02:18.706930+0000","last_active":"2021-03-24T22:03:18.412304+0000","last_peered":"2021-03-24T22:03:18.412304+0000","last_clean":"2021-03-24T22:03:18.412304+0000","last_became_active":"2021-03-24T22:01:43.401402+0000","last_became_peered":"2021-03-24T22:01:43.401402+0000","last_unstale":"2021-03-24T22:03:18.412304+0000","last_undegraded":"2021-03-24T22:03:18.412304+0000","last_fullsized":"2021-03-24T22:03:18.412304+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'21","last_scrub_stamp":"2021-03-24T22:02:18.706842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:02:18.706842+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":26,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":"96","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.462038+0000","last_change":"2021-03-24T22:03:09.417640+0000","last_active":"2021-03-24T22:03:17.462038+0000","last_peered":"2021-03-24T22:03:17.462038+0000","last_clean":"2021-03-24T22:03:17.462038+0000","last_became_active":"2021-03-24T22:01:57.318513+0000","last_became_peered":"2021-03-24T22:01:57.318513+0000","last_unstale":"2021-03-24T22:03:17.462038+0000","last_undegraded":"2021-03-24T22:03:17.462038+0000","last_fullsized":"2021-03-24T22:03:17.462038+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:09.417465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T22:03:09.417465+0000","last_clean_scrub_stamp":"2021-03-24T22:03:09.417465+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"617","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.500465+0000","last_change":"2021-03-24T22:01:16.302201+0000","last_active":"2021-03-24T22:03:17.500465+0000","last_peered":"2021-03-24T22:03:17.500465+0000","last_clean":"2021-03-24T22:03:17.500465+0000","last_became_active":"2021-03-24T22:01:16.300935+0000","last_became_peered":"2021-03-24T22:01:16.300935+0000","last_unstale":"2021-03-24T22:03:17.500465+0000","last_undegraded":"2021-03-24T22:03:17.500465+0000","last_fullsized":"2021-03-24T22:03:17.500465+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:28.289244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:00:28.289244+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"620","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.406004+0000","last_change":"2021-03-24T22:01:52.461763+0000","last_active":"2021-03-24T22:03:17.406004+0000","last_peered":"2021-03-24T22:03:17.406004+0000","last_clean":"2021-03-24T22:03:17.406004+0000","last_became_active":"2021-03-24T22:01:16.336700+0000","last_became_peered":"2021-03-24T22:01:16.336700+0000","last_unstale":"2021-03-24T22:03:17.406004+0000","last_undegraded":"2021-03-24T22:03:17.406004+0000","last_fullsized":"2021-03-24T22:03:17.406004+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:01:52.461594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:01:52.461594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.11","version":"628'853","reported_seq":"2846","reported_epoch":"628","state":"active+clean","last_fresh":"2021-03-24T22:03:18.443934+0000","last_change":"2021-03-24T22:03:18.444376+0000","last_active":"2021-03-24T22:03:18.443934+0000","last_peered":"2021-03-24T22:03:18.443934+0000","last_clean":"2021-03-24T22:03:18.443934+0000","last_became_active":"2021-03-24T22:02:04.322694+0000","last_became_peered":"2021-03-24T22:02:04.322694+0000","last_unstale":"2021-03-24T22:03:18.443934+0000","last_undegraded":"2021-03-24T22:03:18.443934+0000","last_fullsized":"2021-03-24T22:03:18.443934+0000","mapping_epoch":553,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":554,"parent":"0.0","parent_split_bits":5,"last_scrub":"381'425","last_scrub_stamp":"2021-03-24T21:59:11.936662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:59:11.936662+0000","log_size":853,"ondisk_log_size":853,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27759652,"num_objects":13,"num_object_clones":10,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":156,"num_read_kb":179196,"num_write":122,"num_write_kb":46947,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":325,"num_bytes_recovered":148467814,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"139","length":"1"},{"start":"18b","length":"1"},{"start":"194","length":"1"},{"start":"1cc","length":"1"},{"start":"1d7","length":"1"},{"start":"1e8","length":"1"},{"start":"1eb","length":"1"},{"start":"1f4","length":"1"}]},{"pgid":"4.11","version":"544'30","reported_seq":"810","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.500418+0000","last_change":"2021-03-24T22:01:58.012725+0000","last_active":"2021-03-24T22:03:17.500418+0000","last_peered":"2021-03-24T22:03:17.500418+0000","last_clean":"2021-03-24T22:03:17.500418+0000","last_became_active":"2021-03-24T22:01:57.310370+0000","last_became_peered":"2021-03-24T22:01:57.310370+0000","last_unstale":"2021-03-24T22:03:17.500418+0000","last_undegraded":"2021-03-24T22:03:17.500418+0000","last_fullsized":"2021-03-24T22:03:17.500418+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'22","last_scrub_stamp":"2021-03-24T22:00:13.347155+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2021-03-24T21:52:54.597505+0000","last_clean_scrub_stamp":"2021-03-24T22:00:13.347155+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":41,"num_read_kb":28,"num_write":23,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":3480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":"526","reported_epoch":"629","state":"active+clean","last_fresh":"2021-03-24T22:03:19.448844+0000","last_change":"2021-03-24T22:02:18.082676+0000","last_active":"2021-03-24T22:03:19.448844+0000","last_peered":"2021-03-24T22:03:19.448844+0000","last_clean":"2021-03-24T22:03:19.448844+0000","last_became_active":"2021-03-24T22:01:57.327451+0000","last_became_peered":"2021-03-24T22:01:57.327451+0000","last_unstale":"2021-03-24T22:03:19.448844+0000","last_undegraded":"2021-03-24T22:03:19.448844+0000","last_fullsized":"2021-03-24T22:03:19.448844+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:18.082513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:02:18.082513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"91","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.458329+0000","last_change":"2021-03-24T22:01:57.308893+0000","last_active":"2021-03-24T22:03:17.458329+0000","last_peered":"2021-03-24T22:03:17.458329+0000","last_clean":"2021-03-24T22:03:17.458329+0000","last_became_active":"2021-03-24T22:01:57.307995+0000","last_became_peered":"2021-03-24T22:01:57.307995+0000","last_unstale":"2021-03-24T22:03:17.458329+0000","last_undegraded":"2021-03-24T22:03:17.458329+0000","last_fullsized":"2021-03-24T22:03:17.458329+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:41.839852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:59:41.839852+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"600","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.501072+0000","last_change":"2021-03-24T22:01:43.389560+0000","last_active":"2021-03-24T22:03:17.501072+0000","last_peered":"2021-03-24T22:03:17.501072+0000","last_clean":"2021-03-24T22:03:17.501072+0000","last_became_active":"2021-03-24T22:01:43.389017+0000","last_became_peered":"2021-03-24T22:01:43.389017+0000","last_unstale":"2021-03-24T22:03:17.501072+0000","last_undegraded":"2021-03-24T22:03:17.501072+0000","last_fullsized":"2021-03-24T22:03:17.501072+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:15.289444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:47.540793+0000","last_clean_scrub_stamp":"2021-03-24T21:59:15.289444+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.12","version":"183'26","reported_seq":"328","reported_epoch":"630","state":"active+clean","last_fresh":"2021-03-24T22:03:20.339958+0000","last_change":"2021-03-24T22:03:20.341934+0000","last_active":"2021-03-24T22:03:20.339958+0000","last_peered":"2021-03-24T22:03:20.339958+0000","last_clean":"2021-03-24T22:03:20.339958+0000","last_became_active":"2021-03-24T22:01:43.392614+0000","last_became_peered":"2021-03-24T22:01:43.392614+0000","last_unstale":"2021-03-24T22:03:20.339958+0000","last_undegraded":"2021-03-24T22:03:20.339958+0000","last_fullsized":"2021-03-24T22:03:20.339958+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":5,"last_scrub":"183'26","last_scrub_stamp":"2021-03-24T22:02:21.766090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:02:21.766090+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"18b","length":"1"},{"start":"194","length":"1"},{"start":"1a6","length":"1"},{"start":"1cc","length":"1"},{"start":"1e8","length":"1"}]},{"pgid":"4.10","version":"544'16","reported_seq":"822","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.498991+0000","last_change":"2021-03-24T22:03:07.216177+0000","last_active":"2021-03-24T22:03:17.498991+0000","last_peered":"2021-03-24T22:03:17.498991+0000","last_clean":"2021-03-24T22:03:17.498991+0000","last_became_active":"2021-03-24T22:03:06.345705+0000","last_became_peered":"2021-03-24T22:03:06.345705+0000","last_unstale":"2021-03-24T22:03:17.498991+0000","last_undegraded":"2021-03-24T22:03:17.498991+0000","last_fullsized":"2021-03-24T22:03:17.498991+0000","mapping_epoch":615,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":616,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2021-03-24T22:00:29.255353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:00:29.255353+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":36,"num_read_kb":18,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":7512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":"608","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.496658+0000","last_change":"2021-03-24T22:01:56.417737+0000","last_active":"2021-03-24T22:03:17.496658+0000","last_peered":"2021-03-24T22:03:17.496658+0000","last_clean":"2021-03-24T22:03:17.496658+0000","last_became_active":"2021-03-24T22:00:03.301753+0000","last_became_peered":"2021-03-24T22:00:03.301753+0000","last_unstale":"2021-03-24T22:03:17.496658+0000","last_undegraded":"2021-03-24T22:03:17.496658+0000","last_fullsized":"2021-03-24T22:03:17.496658+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:01:56.417662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:01:56.417662+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"123","reported_epoch":"629","state":"active+clean","last_fresh":"2021-03-24T22:03:19.424861+0000","last_change":"2021-03-24T22:02:27.831482+0000","last_active":"2021-03-24T22:03:19.424861+0000","last_peered":"2021-03-24T22:03:19.424861+0000","last_clean":"2021-03-24T22:03:19.424861+0000","last_became_active":"2021-03-24T22:01:57.304385+0000","last_became_peered":"2021-03-24T22:01:57.304385+0000","last_unstale":"2021-03-24T22:03:19.424861+0000","last_undegraded":"2021-03-24T22:03:19.424861+0000","last_fullsized":"2021-03-24T22:03:19.424861+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:27.831383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:02:27.831383+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"98","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.459235+0000","last_change":"2021-03-24T22:03:08.379727+0000","last_active":"2021-03-24T22:03:17.459235+0000","last_peered":"2021-03-24T22:03:17.459235+0000","last_clean":"2021-03-24T22:03:17.459235+0000","last_became_active":"2021-03-24T22:01:57.315718+0000","last_became_peered":"2021-03-24T22:01:57.315718+0000","last_unstale":"2021-03-24T22:03:17.459235+0000","last_undegraded":"2021-03-24T22:03:17.459235+0000","last_fullsized":"2021-03-24T22:03:17.459235+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:08.379591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:57:59.979772+0000","last_clean_scrub_stamp":"2021-03-24T22:03:08.379591+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.13","version":"626'540","reported_seq":"2681","reported_epoch":"628","state":"active+clean","last_fresh":"2021-03-24T22:03:18.426538+0000","last_change":"2021-03-24T22:03:18.427206+0000","last_active":"2021-03-24T22:03:18.426538+0000","last_peered":"2021-03-24T22:03:18.426538+0000","last_clean":"2021-03-24T22:03:18.426538+0000","last_became_active":"2021-03-24T22:02:28.318669+0000","last_became_peered":"2021-03-24T22:02:28.318669+0000","last_unstale":"2021-03-24T22:03:18.426538+0000","last_undegraded":"2021-03-24T22:03:18.426538+0000","last_fullsized":"2021-03-24T22:03:18.426538+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":540,"ondisk_log_size":540,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10855606,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":202,"num_read_kb":190473,"num_write":172,"num_write_kb":68786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":204,"num_bytes_recovered":118383596,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"139","length":"1"},{"start":"18b","length":"1"},{"start":"194","length":"1"},{"start":"1cc","length":"1"},{"start":"1d7","length":"1"},{"start":"1e8","length":"1"},{"start":"1eb","length":"1"},{"start":"1f4","length":"1"}]},{"pgid":"3.18","version":"0'0","reported_seq":"606","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.499481+0000","last_change":"2021-03-24T22:01:16.301917+0000","last_active":"2021-03-24T22:03:17.499481+0000","last_peered":"2021-03-24T22:03:17.499481+0000","last_clean":"2021-03-24T22:03:17.499481+0000","last_became_active":"2021-03-24T22:01:16.301235+0000","last_became_peered":"2021-03-24T22:01:16.301235+0000","last_unstale":"2021-03-24T22:03:17.499481+0000","last_undegraded":"2021-03-24T22:03:17.499481+0000","last_fullsized":"2021-03-24T22:03:17.499481+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:00.393742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:59:00.393742+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"647","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.497384+0000","last_change":"2021-03-24T22:02:22.328403+0000","last_active":"2021-03-24T22:03:17.497384+0000","last_peered":"2021-03-24T22:03:17.497384+0000","last_clean":"2021-03-24T22:03:17.497384+0000","last_became_active":"2021-03-24T22:02:22.328021+0000","last_became_peered":"2021-03-24T22:02:22.328021+0000","last_unstale":"2021-03-24T22:03:17.497384+0000","last_undegraded":"2021-03-24T22:03:17.497384+0000","last_fullsized":"2021-03-24T22:03:17.497384+0000","mapping_epoch":571,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:17.040623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:59:17.040623+0000","last_clean_scrub_stamp":"2021-03-24T21:59:17.040623+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"138","reported_epoch":"628","state":"active+clean","last_fresh":"2021-03-24T22:03:18.413174+0000","last_change":"2021-03-24T22:03:06.420940+0000","last_active":"2021-03-24T22:03:18.413174+0000","last_peered":"2021-03-24T22:03:18.413174+0000","last_clean":"2021-03-24T22:03:18.413174+0000","last_became_active":"2021-03-24T22:01:16.335096+0000","last_became_peered":"2021-03-24T22:01:16.335096+0000","last_unstale":"2021-03-24T22:03:18.413174+0000","last_undegraded":"2021-03-24T22:03:18.413174+0000","last_fullsized":"2021-03-24T22:03:18.413174+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:06.420794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:03:06.420794+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1c","version":"273'245","reported_seq":"1937","reported_epoch":"628","state":"active+clean","last_fresh":"2021-03-24T22:03:18.428440+0000","last_change":"2021-03-24T22:03:18.429261+0000","last_active":"2021-03-24T22:03:18.428440+0000","last_peered":"2021-03-24T22:03:18.428440+0000","last_clean":"2021-03-24T22:03:18.428440+0000","last_became_active":"2021-03-24T22:01:19.284869+0000","last_became_peered":"2021-03-24T22:01:19.284869+0000","last_unstale":"2021-03-24T22:03:18.428440+0000","last_undegraded":"2021-03-24T22:03:18.428440+0000","last_fullsized":"2021-03-24T22:03:18.428440+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":245,"ondisk_log_size":245,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8581786,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":70,"num_read_kb":67597,"num_write":52,"num_write_kb":18055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"139","length":"1"},{"start":"18b","length":"1"},{"start":"194","length":"1"},{"start":"1cc","length":"1"},{"start":"1d7","length":"1"},{"start":"1e8","length":"1"},{"start":"1eb","length":"1"},{"start":"1f4","length":"1"}]},{"pgid":"3.19","version":"0'0","reported_seq":"138","reported_epoch":"628","state":"active+clean","last_fresh":"2021-03-24T22:03:18.411502+0000","last_change":"2021-03-24T22:01:40.788970+0000","last_active":"2021-03-24T22:03:18.411502+0000","last_peered":"2021-03-24T22:03:18.411502+0000","last_clean":"2021-03-24T22:03:18.411502+0000","last_became_active":"2021-03-24T22:01:16.298021+0000","last_became_peered":"2021-03-24T22:01:16.298021+0000","last_unstale":"2021-03-24T22:03:18.411502+0000","last_undegraded":"2021-03-24T22:03:18.411502+0000","last_fullsized":"2021-03-24T22:03:18.411502+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:01:40.788778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:57:26.417613+0000","last_clean_scrub_stamp":"2021-03-24T22:01:40.788778+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"613'8","reported_seq":"149","reported_epoch":"629","state":"active+clean","last_fresh":"2021-03-24T22:03:19.425128+0000","last_change":"2021-03-24T22:03:07.709341+0000","last_active":"2021-03-24T22:03:19.425128+0000","last_peered":"2021-03-24T22:03:19.425128+0000","last_clean":"2021-03-24T22:03:19.425128+0000","last_became_active":"2021-03-24T22:03:03.353426+0000","last_became_peered":"2021-03-24T22:03:03.353426+0000","last_unstale":"2021-03-24T22:03:19.425128+0000","last_undegraded":"2021-03-24T22:03:19.425128+0000","last_fullsized":"2021-03-24T22:03:19.425128+0000","mapping_epoch":612,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":613,"parent":"0.0","parent_split_bits":0,"last_scrub":"613'8","last_scrub_stamp":"2021-03-24T22:03:07.709086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:03:07.709086+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"392","reported_epoch":"630","state":"active+clean","last_fresh":"2021-03-24T22:03:20.346707+0000","last_change":"2021-03-24T22:01:57.307527+0000","last_active":"2021-03-24T22:03:20.346707+0000","last_peered":"2021-03-24T22:03:20.346707+0000","last_clean":"2021-03-24T22:03:20.346707+0000","last_became_active":"2021-03-24T22:01:57.307196+0000","last_became_peered":"2021-03-24T22:01:57.307196+0000","last_unstale":"2021-03-24T22:03:20.346707+0000","last_undegraded":"2021-03-24T22:03:20.346707+0000","last_fullsized":"2021-03-24T22:03:20.346707+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:01:53.591151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:01:53.591151+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1d","version":"628'554","reported_seq":"2492","reported_epoch":"628","state":"active+clean","last_fresh":"2021-03-24T22:03:18.418500+0000","last_change":"2021-03-24T22:03:18.418500+0000","last_active":"2021-03-24T22:03:18.418500+0000","last_peered":"2021-03-24T22:03:18.418500+0000","last_clean":"2021-03-24T22:03:18.418500+0000","last_became_active":"2021-03-24T22:01:46.322605+0000","last_became_peered":"2021-03-24T22:01:46.322605+0000","last_unstale":"2021-03-24T22:03:18.418500+0000","last_undegraded":"2021-03-24T22:03:18.418500+0000","last_fullsized":"2021-03-24T22:03:18.418500+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":5,"last_scrub":"233'169","last_scrub_stamp":"2021-03-24T21:56:47.285648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:56:47.285648+0000","log_size":554,"ondisk_log_size":554,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26145159,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":186,"num_read_kb":205496,"num_write":137,"num_write_kb":66939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":139,"num_bytes_recovered":72061573,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"139","length":"1"},{"start":"18b","length":"1"},{"start":"194","length":"1"},{"start":"1cc","length":"1"},{"start":"1d7","length":"1"},{"start":"1e8","length":"1"},{"start":"1eb","length":"1"},{"start":"1f4","length":"1"}]},{"pgid":"3.1a","version":"0'0","reported_seq":"133","reported_epoch":"628","state":"active+clean","last_fresh":"2021-03-24T22:03:18.412043+0000","last_change":"2021-03-24T22:01:16.335123+0000","last_active":"2021-03-24T22:03:18.412043+0000","last_peered":"2021-03-24T22:03:18.412043+0000","last_clean":"2021-03-24T22:03:18.412043+0000","last_became_active":"2021-03-24T22:01:16.334794+0000","last_became_peered":"2021-03-24T22:01:16.334794+0000","last_unstale":"2021-03-24T22:03:18.412043+0000","last_undegraded":"2021-03-24T22:03:18.412043+0000","last_fullsized":"2021-03-24T22:03:18.412043+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:12.679659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:52:54.212459+0000","last_clean_scrub_stamp":"2021-03-24T21:59:12.679659+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"166","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.502274+0000","last_change":"2021-03-24T22:02:17.672555+0000","last_active":"2021-03-24T22:03:17.502274+0000","last_peered":"2021-03-24T22:03:17.502274+0000","last_clean":"2021-03-24T22:03:17.502274+0000","last_became_active":"2021-03-24T22:00:49.305569+0000","last_became_peered":"2021-03-24T22:00:49.305569+0000","last_unstale":"2021-03-24T22:03:17.502274+0000","last_undegraded":"2021-03-24T22:03:17.502274+0000","last_fullsized":"2021-03-24T22:03:17.502274+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:17.672325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:02:17.672325+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"633","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.502132+0000","last_change":"2021-03-24T22:01:43.383380+0000","last_active":"2021-03-24T22:03:17.502132+0000","last_peered":"2021-03-24T22:03:17.502132+0000","last_clean":"2021-03-24T22:03:17.502132+0000","last_became_active":"2021-03-24T22:01:43.383075+0000","last_became_peered":"2021-03-24T22:01:43.383075+0000","last_unstale":"2021-03-24T22:03:17.502132+0000","last_undegraded":"2021-03-24T22:03:17.502132+0000","last_fullsized":"2021-03-24T22:03:17.502132+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:55.309332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:00:55.309332+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1e","version":"265'45","reported_seq":"381","reported_epoch":"628","state":"active+clean","last_fresh":"2021-03-24T22:03:18.414429+0000","last_change":"2021-03-24T22:03:18.414429+0000","last_active":"2021-03-24T22:03:18.414429+0000","last_peered":"2021-03-24T22:03:18.414429+0000","last_clean":"2021-03-24T22:03:18.414429+0000","last_became_active":"2021-03-24T22:01:16.321204+0000","last_became_peered":"2021-03-24T22:01:16.321204+0000","last_unstale":"2021-03-24T22:03:18.414429+0000","last_undegraded":"2021-03-24T22:03:18.414429+0000","last_fullsized":"2021-03-24T22:03:18.414429+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":5,"last_scrub":"199'30","last_scrub_stamp":"2021-03-24T21:56:10.105084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:56:10.105084+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":18305,"num_write":13,"num_write_kb":3060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"139","length":"1"},{"start":"18b","length":"1"},{"start":"194","length":"1"},{"start":"1cc","length":"1"},{"start":"1d7","length":"1"},{"start":"1e8","length":"1"},{"start":"1eb","length":"1"},{"start":"1f4","length":"1"}]},{"pgid":"3.1b","version":"0'0","reported_seq":"638","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.494300+0000","last_change":"2021-03-24T22:03:06.348579+0000","last_active":"2021-03-24T22:03:17.494300+0000","last_peered":"2021-03-24T22:03:17.494300+0000","last_clean":"2021-03-24T22:03:17.494300+0000","last_became_active":"2021-03-24T22:03:06.348172+0000","last_became_peered":"2021-03-24T22:03:06.348172+0000","last_unstale":"2021-03-24T22:03:17.494300+0000","last_undegraded":"2021-03-24T22:03:17.494300+0000","last_fullsized":"2021-03-24T22:03:17.494300+0000","mapping_epoch":615,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":616,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:54.682649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:00:54.682649+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"140","reported_epoch":"628","state":"active+clean","last_fresh":"2021-03-24T22:03:18.412974+0000","last_change":"2021-03-24T22:01:51.757918+0000","last_active":"2021-03-24T22:03:18.412974+0000","last_peered":"2021-03-24T22:03:18.412974+0000","last_clean":"2021-03-24T22:03:18.412974+0000","last_became_active":"2021-03-24T22:01:16.334651+0000","last_became_peered":"2021-03-24T22:01:16.334651+0000","last_unstale":"2021-03-24T22:03:18.412974+0000","last_undegraded":"2021-03-24T22:03:18.412974+0000","last_fullsized":"2021-03-24T22:03:18.412974+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:01:51.757809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:01:51.757809+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"107","reported_epoch":"629","state":"active+clean","last_fresh":"2021-03-24T22:03:19.424289+0000","last_change":"2021-03-24T22:01:43.384457+0000","last_active":"2021-03-24T22:03:19.424289+0000","last_peered":"2021-03-24T22:03:19.424289+0000","last_clean":"2021-03-24T22:03:19.424289+0000","last_became_active":"2021-03-24T22:01:43.384083+0000","last_became_peered":"2021-03-24T22:01:43.384083+0000","last_unstale":"2021-03-24T22:03:19.424289+0000","last_undegraded":"2021-03-24T22:03:19.424289+0000","last_fullsized":"2021-03-24T22:03:19.424289+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:26.883399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:00:26.883399+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1f","version":"272'99","reported_seq":"323","reported_epoch":"630","state":"active+clean","last_fresh":"2021-03-24T22:03:20.354961+0000","last_change":"2021-03-24T22:03:20.355375+0000","last_active":"2021-03-24T22:03:20.354961+0000","last_peered":"2021-03-24T22:03:20.354961+0000","last_clean":"2021-03-24T22:03:20.354961+0000","last_became_active":"2021-03-24T22:01:43.385324+0000","last_became_peered":"2021-03-24T22:01:43.385324+0000","last_unstale":"2021-03-24T22:03:20.354961+0000","last_undegraded":"2021-03-24T22:03:20.354961+0000","last_fullsized":"2021-03-24T22:03:20.354961+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":5,"last_scrub":"218'73","last_scrub_stamp":"2021-03-24T21:56:48.267711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:56:48.267711+0000","log_size":99,"ondisk_log_size":99,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3571504,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":43,"num_read_kb":44484,"num_write":22,"num_write_kb":9163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":942483,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"18b","length":"1"},{"start":"194","length":"1"},{"start":"1a6","length":"1"},{"start":"1cc","length":"1"},{"start":"1e8","length":"1"}]},{"pgid":"3.1c","version":"0'0","reported_seq":"133","reported_epoch":"628","state":"active+clean","last_fresh":"2021-03-24T22:03:18.413097+0000","last_change":"2021-03-24T22:01:16.334949+0000","last_active":"2021-03-24T22:03:18.413097+0000","last_peered":"2021-03-24T22:03:18.413097+0000","last_clean":"2021-03-24T22:03:18.413097+0000","last_became_active":"2021-03-24T22:01:16.334631+0000","last_became_peered":"2021-03-24T22:01:16.334631+0000","last_unstale":"2021-03-24T22:03:18.413097+0000","last_undegraded":"2021-03-24T22:03:18.413097+0000","last_fullsized":"2021-03-24T22:03:18.413097+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:29.855897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:00:29.855897+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"91","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.459626+0000","last_change":"2021-03-24T22:01:57.322203+0000","last_active":"2021-03-24T22:03:17.459626+0000","last_peered":"2021-03-24T22:03:17.459626+0000","last_clean":"2021-03-24T22:03:17.459626+0000","last_became_active":"2021-03-24T22:01:57.321935+0000","last_became_peered":"2021-03-24T22:01:57.321935+0000","last_unstale":"2021-03-24T22:03:17.459626+0000","last_undegraded":"2021-03-24T22:03:17.459626+0000","last_fullsized":"2021-03-24T22:03:17.459626+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:08.942247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:56:51.977306+0000","last_clean_scrub_stamp":"2021-03-24T21:59:08.942247+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"499","reported_epoch":"629","state":"active+clean","last_fresh":"2021-03-24T22:03:19.447181+0000","last_change":"2021-03-24T22:01:41.968251+0000","last_active":"2021-03-24T22:03:19.447181+0000","last_peered":"2021-03-24T22:03:19.447181+0000","last_clean":"2021-03-24T22:03:19.447181+0000","last_became_active":"2021-03-24T22:01:16.312575+0000","last_became_peered":"2021-03-24T22:01:16.312575+0000","last_unstale":"2021-03-24T22:03:19.447181+0000","last_undegraded":"2021-03-24T22:03:19.447181+0000","last_fullsized":"2021-03-24T22:03:19.447181+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:01:41.968063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:01:41.968063+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.18","version":"624'193","reported_seq":"1630","reported_epoch":"628","state":"active+clean","last_fresh":"2021-03-24T22:03:18.416683+0000","last_change":"2021-03-24T22:03:18.417361+0000","last_active":"2021-03-24T22:03:18.416683+0000","last_peered":"2021-03-24T22:03:18.416683+0000","last_clean":"2021-03-24T22:03:18.416683+0000","last_became_active":"2021-03-24T22:02:00.304128+0000","last_became_peered":"2021-03-24T22:02:00.304128+0000","last_unstale":"2021-03-24T22:03:18.416683+0000","last_undegraded":"2021-03-24T22:03:18.416683+0000","last_fullsized":"2021-03-24T22:03:18.416683+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":5,"last_scrub":"249'90","last_scrub_stamp":"2021-03-24T21:57:56.848784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:57:56.848784+0000","log_size":193,"ondisk_log_size":193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4857520,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":42,"num_read_kb":38347,"num_write":26,"num_write_kb":9218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":32240200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"139","length":"1"},{"start":"18b","length":"1"},{"start":"194","length":"1"},{"start":"1cc","length":"1"},{"start":"1d7","length":"1"},{"start":"1e8","length":"1"},{"start":"1eb","length":"1"},{"start":"1f4","length":"1"}]},{"pgid":"3.1d","version":"0'0","reported_seq":"660","reported_epoch":"630","state":"active+clean","last_fresh":"2021-03-24T22:03:20.347262+0000","last_change":"2021-03-24T22:02:19.684802+0000","last_active":"2021-03-24T22:03:20.347262+0000","last_peered":"2021-03-24T22:03:20.347262+0000","last_clean":"2021-03-24T22:03:20.347262+0000","last_became_active":"2021-03-24T22:01:43.375494+0000","last_became_peered":"2021-03-24T22:01:43.375494+0000","last_unstale":"2021-03-24T22:03:20.347262+0000","last_undegraded":"2021-03-24T22:03:20.347262+0000","last_fullsized":"2021-03-24T22:03:20.347262+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:19.684726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:02:19.684726+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"513","reported_epoch":"629","state":"active+clean","last_fresh":"2021-03-24T22:03:19.447345+0000","last_change":"2021-03-24T22:01:57.323397+0000","last_active":"2021-03-24T22:03:19.447345+0000","last_peered":"2021-03-24T22:03:19.447345+0000","last_clean":"2021-03-24T22:03:19.447345+0000","last_became_active":"2021-03-24T22:01:57.323085+0000","last_became_peered":"2021-03-24T22:01:57.323085+0000","last_unstale":"2021-03-24T22:03:19.447345+0000","last_undegraded":"2021-03-24T22:03:19.447345+0000","last_fullsized":"2021-03-24T22:03:19.447345+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:22.297227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:00:22.297227+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"641","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.493905+0000","last_change":"2021-03-24T22:02:19.458206+0000","last_active":"2021-03-24T22:03:17.493905+0000","last_peered":"2021-03-24T22:03:17.493905+0000","last_clean":"2021-03-24T22:03:17.493905+0000","last_became_active":"2021-03-24T22:01:57.315933+0000","last_became_peered":"2021-03-24T22:01:57.315933+0000","last_unstale":"2021-03-24T22:03:17.493905+0000","last_undegraded":"2021-03-24T22:03:17.493905+0000","last_fullsized":"2021-03-24T22:03:17.493905+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:19.457980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:02:19.457980+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.19","version":"630'751","reported_seq":"2809","reported_epoch":"630","state":"active+clean","last_fresh":"2021-03-24T22:03:20.351286+0000","last_change":"2021-03-24T22:03:20.352415+0000","last_active":"2021-03-24T22:03:20.351286+0000","last_peered":"2021-03-24T22:03:20.351286+0000","last_clean":"2021-03-24T22:03:20.351286+0000","last_became_active":"2021-03-24T22:02:07.304440+0000","last_became_peered":"2021-03-24T22:02:07.304440+0000","last_unstale":"2021-03-24T22:03:20.351286+0000","last_undegraded":"2021-03-24T22:03:20.351286+0000","last_fullsized":"2021-03-24T22:03:20.351286+0000","mapping_epoch":556,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":557,"parent":"0.0","parent_split_bits":5,"last_scrub":"455'480","last_scrub_stamp":"2021-03-24T22:00:26.193730+0000","last_deep_scrub":"455'480","last_deep_scrub_stamp":"2021-03-24T22:00:26.193730+0000","last_clean_scrub_stamp":"2021-03-24T22:00:26.193730+0000","log_size":751,"ondisk_log_size":751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30079696,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":171,"num_read_kb":151402,"num_write":137,"num_write_kb":63355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":265,"num_bytes_recovered":154749512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"18b","length":"1"},{"start":"194","length":"1"},{"start":"1a6","length":"1"},{"start":"1cc","length":"1"},{"start":"1e8","length":"1"}]},{"pgid":"3.1e","version":"0'0","reported_seq":"642","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.501336+0000","last_change":"2021-03-24T22:01:43.382796+0000","last_active":"2021-03-24T22:03:17.501336+0000","last_peered":"2021-03-24T22:03:17.501336+0000","last_clean":"2021-03-24T22:03:17.501336+0000","last_became_active":"2021-03-24T22:01:43.381824+0000","last_became_peered":"2021-03-24T22:01:43.381824+0000","last_unstale":"2021-03-24T22:03:17.501336+0000","last_undegraded":"2021-03-24T22:03:17.501336+0000","last_fullsized":"2021-03-24T22:03:17.501336+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:05.405785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:59:05.405785+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"112","reported_epoch":"629","state":"active+clean","last_fresh":"2021-03-24T22:03:19.423554+0000","last_change":"2021-03-24T22:03:01.782651+0000","last_active":"2021-03-24T22:03:19.423554+0000","last_peered":"2021-03-24T22:03:19.423554+0000","last_clean":"2021-03-24T22:03:19.423554+0000","last_became_active":"2021-03-24T22:01:43.393091+0000","last_became_peered":"2021-03-24T22:01:43.393091+0000","last_unstale":"2021-03-24T22:03:19.423554+0000","last_undegraded":"2021-03-24T22:03:19.423554+0000","last_fullsized":"2021-03-24T22:03:19.423554+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:01.782483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:03:01.782483+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"661","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.500571+0000","last_change":"2021-03-24T22:03:08.392175+0000","last_active":"2021-03-24T22:03:17.500571+0000","last_peered":"2021-03-24T22:03:17.500571+0000","last_clean":"2021-03-24T22:03:17.500571+0000","last_became_active":"2021-03-24T22:01:57.310369+0000","last_became_peered":"2021-03-24T22:01:57.310369+0000","last_unstale":"2021-03-24T22:03:17.500571+0000","last_undegraded":"2021-03-24T22:03:17.500571+0000","last_fullsized":"2021-03-24T22:03:17.500571+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:08.392060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:03:08.392060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"630'318","reported_seq":"2092","reported_epoch":"630","state":"active+clean","last_fresh":"2021-03-24T22:03:20.354607+0000","last_change":"2021-03-24T22:03:20.355308+0000","last_active":"2021-03-24T22:03:20.354607+0000","last_peered":"2021-03-24T22:03:20.354607+0000","last_clean":"2021-03-24T22:03:20.354607+0000","last_became_active":"2021-03-24T22:01:46.306324+0000","last_became_peered":"2021-03-24T22:01:46.306324+0000","last_unstale":"2021-03-24T22:03:20.354607+0000","last_undegraded":"2021-03-24T22:03:20.354607+0000","last_fullsized":"2021-03-24T22:03:20.354607+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":318,"ondisk_log_size":318,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7153154,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":87,"num_read_kb":98418,"num_write":78,"num_write_kb":27419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":37089076,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"18b","length":"1"},{"start":"194","length":"1"},{"start":"1a6","length":"1"},{"start":"1cc","length":"1"},{"start":"1e8","length":"1"}]},{"pgid":"3.1f","version":"0'0","reported_seq":"631","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.496153+0000","last_change":"2021-03-24T22:02:22.325068+0000","last_active":"2021-03-24T22:03:17.496153+0000","last_peered":"2021-03-24T22:03:17.496153+0000","last_clean":"2021-03-24T22:03:17.496153+0000","last_became_active":"2021-03-24T22:02:22.324672+0000","last_became_peered":"2021-03-24T22:02:22.324672+0000","last_unstale":"2021-03-24T22:03:17.496153+0000","last_undegraded":"2021-03-24T22:03:17.496153+0000","last_fullsized":"2021-03-24T22:03:17.496153+0000","mapping_epoch":571,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:13.624943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:00:13.624943+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"158","reported_epoch":"628","state":"active+clean","last_fresh":"2021-03-24T22:03:18.411666+0000","last_change":"2021-03-24T22:03:00.407206+0000","last_active":"2021-03-24T22:03:18.411666+0000","last_peered":"2021-03-24T22:03:18.411666+0000","last_clean":"2021-03-24T22:03:18.411666+0000","last_became_active":"2021-03-24T22:01:57.326514+0000","last_became_peered":"2021-03-24T22:01:57.326514+0000","last_unstale":"2021-03-24T22:03:18.411666+0000","last_undegraded":"2021-03-24T22:03:18.411666+0000","last_fullsized":"2021-03-24T22:03:18.411666+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:00.407009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:14.710709+0000","last_clean_scrub_stamp":"2021-03-24T22:03:00.407009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"601","reported_epoch":"629","state":"active+clean","last_fresh":"2021-03-24T22:03:19.447842+0000","last_change":"2021-03-24T22:02:35.095682+0000","last_active":"2021-03-24T22:03:19.447842+0000","last_peered":"2021-03-24T22:03:19.447842+0000","last_clean":"2021-03-24T22:03:19.447842+0000","last_became_active":"2021-03-24T22:01:16.311234+0000","last_became_peered":"2021-03-24T22:01:16.311234+0000","last_unstale":"2021-03-24T22:03:19.447842+0000","last_undegraded":"2021-03-24T22:03:19.447842+0000","last_fullsized":"2021-03-24T22:03:19.447842+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:35.095493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:02:35.095493+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1b","version":"628'1360","reported_seq":"3145","reported_epoch":"628","state":"active+clean","last_fresh":"2021-03-24T22:03:18.468277+0000","last_change":"2021-03-24T22:03:18.436730+0000","last_active":"2021-03-24T22:03:18.468277+0000","last_peered":"2021-03-24T22:03:18.468277+0000","last_clean":"2021-03-24T22:03:18.468277+0000","last_became_active":"2021-03-24T21:57:33.176110+0000","last_became_peered":"2021-03-24T21:57:33.176110+0000","last_unstale":"2021-03-24T22:03:18.468277+0000","last_undegraded":"2021-03-24T22:03:18.468277+0000","last_fullsized":"2021-03-24T22:03:18.468277+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":5,"last_scrub":"410'569","last_scrub_stamp":"2021-03-24T21:59:43.929595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:59:43.929595+0000","log_size":1360,"ondisk_log_size":1360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":66810269,"num_objects":27,"num_object_clones":21,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":175,"num_read_kb":186345,"num_write":134,"num_write_kb":59897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"139","length":"1"},{"start":"18b","length":"1"},{"start":"194","length":"1"},{"start":"1cc","length":"1"},{"start":"1d7","length":"1"},{"start":"1e8","length":"1"},{"start":"1eb","length":"1"},{"start":"1f4","length":"1"}]},{"pgid":"3.24","version":"0'0","reported_seq":"254","reported_epoch":"627","state":"active+clean","last_fresh":"2021-03-24T22:03:17.500726+0000","last_change":"2021-03-24T21:59:14.369914+0000","last_active":"2021-03-24T22:03:17.500726+0000","last_peered":"2021-03-24T22:03:17.500726+0000","last_clean":"2021-03-24T22:03:17.500726+0000","last_became_active":"2021-03-24T21:59:14.369606+0000","last_became_peered":"2021-03-24T21:59:14.369606+0000","last_unstale":"2021-03-24T22:03:17.500726+0000","last_undegraded":"2021-03-24T22:03:17.500726+0000","last_fullsized":"2021-03-24T22:03:17.500726+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:38.144956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:38.144956+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"175'22","reported_seq":"1361","reported_epoch":"630","state":"active+clean","last_fresh":"2021-03-24T22:03:20.347484+0000","last_change":"2021-03-24T22:03:20.347484+0000","last_active":"2021-03-24T22:03:20.347484+0000","last_peered":"2021-03-24T22:03:20.347484+0000","last_clean":"2021-03-24T22:03:20.347484+0000","last_became_active":"2021-03-24T22:01:43.384871+0000","last_became_peered":"2021-03-24T22:01:43.384871+0000","last_unstale":"2021-03-24T22:03:20.347484+0000","last_undegraded":"2021-03-24T22:03:20.347484+0000","last_fullsized":"2021-03-24T22:03:20.347484+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":6,"last_scrub":"175'22","last_scrub_stamp":"2021-03-24T22:03:11.749445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:03:11.749445+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"18b","length":"1"},{"start":"194","length":"1"},{"start":"1a6","length":"1"},{"start":"1cc","length":"1"},{"start":"1e8","length":"1"}]},{"pgid":"3.25","version":"0'0","reported_seq":"133","reported_epoch":"628","state":"active+clean","last_fresh":"2021-03-24T22:03:18.413248+0000","last_change":"2021-03-24T22:01:16.312159+0000","last_active":"2021-03-24T22:03:18.413248+0000","last_peered":"2021-03-24T22:03:18.413248+0000","last_clean":"2021-03-24T22:03:18.413248+0000","last_became_active":"2021-03-24T22:01:16.311807+0000","last_became_peered":"2021-03-24T22:01:16.311807+0000","last_unstale":"2021-03-24T22:03:18.413248+0000","last_undegraded":"2021-03-24T22:03:18.413248+0000","last_fullsized":"2021-03-24T22:03:18.413248+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:53.013129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:53.013129+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.21","version":"628'573","reported_seq":"2590","reported_epoch":"628","state":"active+clean","last_fresh":"2021-03-24T22:03:18.440754+0000","last_change":"2021-03-24T22:03:18.438299+0000","last_active":"2021-03-24T22:03:18.440754+0000","last_peered":"2021-03-24T22:03:18.440754+0000","last_clean":"2021-03-24T22:03:18.440754+0000","last_became_active":"2021-03-24T22:02:04.324578+0000","last_became_peered":"2021-03-24T22:02:04.324578+0000","last_unstale":"2021-03-24T22:03:18.440754+0000","last_undegraded":"2021-03-24T22:03:18.440754+0000","last_fullsized":"2021-03-24T22:03:18.440754+0000","mapping_epoch":553,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":554,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":573,"ondisk_log_size":573,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18135172,"num_objects":11,"num_object_clones":10,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":174,"num_read_kb":175652,"num_write":123,"num_write_kb":41769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":55123926,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"139","length":"1"},{"start":"18b","length":"1"},{"start":"194","length":"1"},{"start":"1cc","length":"1"},{"start":"1d7","length":"1"},{"start":"1e8","length":"1"},{"start":"1eb","length":"1"},{"start":"1f4","length":"1"}]}],"pool_stats":[{"poolid":7,"num_pg":36,"stat_sum":{"num_bytes":495783755,"num_objects":234,"num_object_clones":178,"num_object_copies":702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":10,"num_read":3086,"num_read_kb":3250581,"num_write":2229,"num_write_kb":855914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2570,"num_bytes_recovered":1425500674,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1043156992,"data_stored":1430301793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14231,"ondisk_log_size":14231,"up":108,"acting":108,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":156,"num_read_kb":151,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":3,"num_pg":42,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":4392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":984,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":126,"acting":126,"num_store_stats":8},{"poolid":4,"num_pg":22,"stat_sum":{"num_bytes":23524,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":40,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1484,"num_read_kb":989,"num_write":828,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":876,"num_bytes_recovered":68726,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966080,"data_stored":70572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":505,"ondisk_log_size":505,"up":66,"acting":66,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201423,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":202088,"kb_used_data":161440,"kb_used_omap":0,"kb_used_meta":40640,"kb_avail":93543064,"statfs":{"total":95995035648,"available":95788097536,"internally_reserved":0,"allocated":165314560,"data_stored":232113827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41615360},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 22:03:09 2021","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.79000000000000004,"15min":0.90200000000000002},"min":{"1min":0.39100000000000001,"5min":0.33300000000000002,"15min":0.312},"max":{"1min":1.234,"5min":1.298,"15min":12.222},"last":1.2070000000000001},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.97499999999999998,"15min":0.99099999999999999},"min":{"1min":0.36299999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.284,"5min":7.1609999999999996,"15min":11.506},"last":1.075}]},{"osd":1,"last update":"Wed Mar 24 22:03:09 2021","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.81999999999999995,"15min":0.88100000000000001},"min":{"1min":0.52100000000000002,"5min":0.308,"15min":0.308},"max":{"1min":1.1379999999999999,"5min":1.319,"15min":5.7750000000000004},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.91700000000000004,"15min":0.96499999999999997},"min":{"1min":0.503,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.266,"5min":6.1849999999999996,"15min":6.1849999999999996},"last":1.2569999999999999}]},{"osd":2,"last update":"Wed Mar 24 22:03:09 2021","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.85799999999999998,"15min":1.0069999999999999},"min":{"1min":0.54500000000000004,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.1020000000000001,"5min":5.7629999999999999,"15min":12.135999999999999},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.875,"5min":0.95399999999999996,"15min":1.0289999999999999},"min":{"1min":0.55700000000000005,"5min":0.432,"15min":0.42999999999999999},"max":{"1min":1.341,"5min":7.0460000000000003,"15min":12.178000000000001},"last":1.1299999999999999}]},{"osd":3,"last update":"Wed Mar 24 22:03:09 2021","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.98299999999999998,"15min":1.0669999999999999},"min":{"1min":0.47799999999999998,"5min":0.47399999999999998,"15min":0.39800000000000002},"max":{"1min":1.2130000000000001,"5min":7.2800000000000002,"15min":12.529},"last":1.395},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.0609999999999999,"15min":1.077},"min":{"1min":0.82799999999999996,"5min":0.443,"15min":0.443},"max":{"1min":3.0510000000000002,"5min":3.101,"15min":12.666},"last":1.2889999999999999}]},{"osd":4,"last update":"Wed Mar 24 22:03:09 2021","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":0.94899999999999995,"15min":0.97699999999999998},"min":{"1min":0.59299999999999997,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.3089999999999999,"5min":1.649,"15min":6.2089999999999996},"last":1.4279999999999999},{"interface":"front","average":{"1min":0.97899999999999998,"5min":0.997,"15min":1.081},"min":{"1min":0.60699999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.2689999999999999,"5min":1.921,"15min":12.244},"last":1.476}]},{"osd":5,"last update":"Wed Mar 24 22:03:09 2021","interfaces":[{"interface":"back","average":{"1min":1.0600000000000001,"5min":1.131,"15min":1.149},"min":{"1min":0.82499999999999996,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.304,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":1.369},{"interface":"front","average":{"1min":1.1990000000000001,"5min":1.282,"15min":1.3100000000000001},"min":{"1min":1.0109999999999999,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.498,"5min":2.1190000000000002,"15min":2.1190000000000002},"last":1.4470000000000001}]},{"osd":6,"last update":"Wed Mar 24 22:03:09 2021","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":1.0669999999999999,"15min":1.135},"min":{"1min":0.63500000000000001,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.506,"5min":1.7869999999999999,"15min":6.1159999999999997},"last":1.3440000000000001},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.0600000000000001,"15min":1.1779999999999999},"min":{"1min":0.71399999999999997,"5min":0.434,"15min":0.434},"max":{"1min":1.3540000000000001,"5min":1.867,"15min":12.287000000000001},"last":1.522}]}]},{"osd":6,"up_from":47,"seq":201863463060,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":157576,"kb_used_data":128128,"kb_used_omap":0,"kb_used_meta":29440,"kb_avail":93587576,"statfs":{"total":95995035648,"available":95833677824,"internally_reserved":0,"allocated":131203072,"data_stored":168454799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":30146560},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 22:02:37 2021","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.95599999999999996,"15min":0.93899999999999995},"min":{"1min":0.45500000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.194,"5min":9.0280000000000005,"15min":9.0280000000000005},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.95499999999999996,"15min":0.92600000000000005},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.21,"5min":5.3310000000000004,"15min":5.3310000000000004},"last":0.753}]},{"osd":1,"last update":"Wed Mar 24 22:02:37 2021","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.92400000000000004,"15min":0.92600000000000005},"min":{"1min":0.42899999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.1279999999999999,"5min":5.625,"15min":5.625},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.85899999999999999,"5min":1.002,"15min":0.96999999999999997},"min":{"1min":0.45400000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.196,"5min":5.4550000000000001,"15min":5.4550000000000001},"last":0.95999999999999996}]},{"osd":2,"last update":"Wed Mar 24 22:02:37 2021","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.92200000000000004,"15min":0.95199999999999996},"min":{"1min":0.42899999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.234,"5min":5.0860000000000003,"15min":5.0860000000000003},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.93000000000000005,"15min":0.96699999999999997},"min":{"1min":0.41199999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.306,"5min":4.0700000000000003,"15min":4.0700000000000003},"last":0.91800000000000004}]},{"osd":3,"last update":"Wed Mar 24 22:02:37 2021","interfaces":[{"interface":"back","average":{"1min":1.0149999999999999,"5min":0.97399999999999998,"15min":0.97199999999999998},"min":{"1min":0.50600000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":4.7400000000000002,"5min":5.5039999999999996,"15min":5.5039999999999996},"last":0.94299999999999995},{"interface":"front","average":{"1min":0.93200000000000005,"5min":1.0780000000000001,"15min":1.0640000000000001},"min":{"1min":0.57399999999999995,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.2689999999999999,"5min":4.3460000000000001,"15min":4.3460000000000001},"last":0.78200000000000003}]},{"osd":4,"last update":"Wed Mar 24 22:02:37 2021","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.95899999999999996,"15min":0.97899999999999998},"min":{"1min":0.44700000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.25,"5min":1.7849999999999999,"15min":2.1640000000000001},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.872,"5min":1.012,"15min":1.028},"min":{"1min":0.55000000000000004,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.405,"5min":2.3109999999999999,"15min":2.3109999999999999},"last":0.79900000000000004}]},{"osd":5,"last update":"Wed Mar 24 22:03:08 2021","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.05,"15min":1.0760000000000001},"min":{"1min":0.55500000000000005,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.454,"5min":1.915,"15min":1.915},"last":0.89600000000000002},{"interface":"front","average":{"1min":1.05,"5min":1.087,"15min":1.117},"min":{"1min":0.60199999999999998,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.4330000000000001,"5min":2.7069999999999999,"15min":2.7069999999999999},"last":0.86699999999999999}]},{"osd":7,"last update":"Wed Mar 24 22:03:03 2021","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.137,"15min":1.1699999999999999},"min":{"1min":0.63200000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.387,"5min":9.5909999999999993,"15min":9.5909999999999993},"last":0.98499999999999999},{"interface":"front","average":{"1min":1.125,"5min":1.075,"15min":1.157},"min":{"1min":0.69099999999999995,"5min":0.442,"15min":0.442},"max":{"1min":1.4419999999999999,"5min":1.6080000000000001,"15min":2.8410000000000002},"last":0.85499999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575017,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":98380,"kb_used_data":74436,"kb_used_omap":0,"kb_used_meta":23936,"kb_avail":93646772,"statfs":{"total":95995035648,"available":95894294528,"internally_reserved":0,"allocated":76222464,"data_stored":117327879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":24510464},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 22:02:57 2021","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":1.0880000000000001,"15min":0.78900000000000003},"min":{"1min":0.39400000000000002,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.90800000000000003,"5min":21.085000000000001,"15min":21.085000000000001},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.749,"5min":0.84099999999999997,"15min":0.751},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.4319999999999999,"5min":8.5180000000000007,"15min":8.5180000000000007},"last":0.45500000000000002}]},{"osd":2,"last update":"Wed Mar 24 22:03:08 2021","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.73399999999999999,"15min":0.68700000000000006},"min":{"1min":0.38900000000000001,"5min":0.307,"15min":0.307},"max":{"1min":1.5660000000000001,"5min":9.9870000000000001,"15min":9.9870000000000001},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.89900000000000002,"15min":0.81999999999999995},"min":{"1min":0.46700000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.4359999999999999,"5min":9.2059999999999995,"15min":9.2059999999999995},"last":0.58299999999999996}]},{"osd":3,"last update":"Wed Mar 24 22:02:37 2021","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.89000000000000001,"15min":1.0680000000000001},"min":{"1min":0.37,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.6390000000000002,"5min":8.5990000000000002,"15min":17.032},"last":0.72899999999999998},{"interface":"front","average":{"1min":1.5369999999999999,"5min":1.159,"15min":1.1240000000000001},"min":{"1min":0.45900000000000002,"5min":0.35499999999999998,"15min":0.316},"max":{"1min":15.775,"5min":15.775,"15min":16.908999999999999},"last":0.745}]},{"osd":4,"last update":"Wed Mar 24 22:03:02 2021","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.98299999999999998,"15min":0.97399999999999998},"min":{"1min":0.66200000000000003,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.258,"5min":9.6150000000000002,"15min":9.6150000000000002},"last":1.1970000000000001},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.0009999999999999,"15min":1.0589999999999999},"min":{"1min":0.58399999999999996,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.605,"5min":3.9359999999999999,"15min":3.9359999999999999},"last":0.78400000000000003}]},{"osd":5,"last update":"Wed Mar 24 22:03:02 2021","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.0449999999999999,"15min":1.054},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.6719999999999999,"5min":3.976,"15min":3.976},"last":0.76100000000000001},{"interface":"front","average":{"1min":1.036,"5min":1.127,"15min":1.095},"min":{"1min":0.48099999999999998,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.647,"5min":9.6809999999999992,"15min":9.6809999999999992},"last":1.1140000000000001}]},{"osd":6,"last update":"Wed Mar 24 22:02:37 2021","interfaces":[{"interface":"back","average":{"1min":1.585,"5min":1.274,"15min":1.177},"min":{"1min":0.63700000000000001,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":11.254,"5min":11.254,"15min":11.254},"last":0.93200000000000005},{"interface":"front","average":{"1min":1.532,"5min":1.091,"15min":1.1359999999999999},"min":{"1min":0.55600000000000005,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":9.7300000000000004,"5min":9.7300000000000004,"15min":9.7300000000000004},"last":1.2390000000000001}]},{"osd":7,"last update":"Wed Mar 24 22:03:02 2021","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":1.0760000000000001,"15min":1.1850000000000001},"min":{"1min":0.68200000000000005,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.718,"5min":3.8439999999999999,"15min":4.4699999999999998},"last":0.79900000000000004},{"interface":"front","average":{"1min":1.0880000000000001,"5min":1.296,"15min":1.2090000000000001},"min":{"1min":0.58399999999999996,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.6539999999999999,"5min":11.289,"15min":11.289},"last":0.96099999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738540,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":191836,"kb_used_data":145428,"kb_used_omap":0,"kb_used_meta":46400,"kb_avail":93553316,"statfs":{"total":95995035648,"available":95798595584,"internally_reserved":0,"allocated":148918272,"data_stored":200700657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":47513600},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Mar 24 22:02:58 2021","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.61799999999999999,"15min":0.64200000000000002},"min":{"1min":0.27700000000000002,"5min":0.249,"15min":0.249},"max":{"1min":1.248,"5min":1.248,"15min":4.8040000000000003},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.69499999999999995,"15min":0.72099999999999997},"min":{"1min":0.33200000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.0429999999999999,"5min":2.5179999999999998,"15min":8.8989999999999991},"last":0.85799999999999998}]},{"osd":2,"last update":"Wed Mar 24 22:03:16 2021","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.82399999999999995,"15min":0.76000000000000001},"min":{"1min":0.308,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.96599999999999997,"5min":15.949999999999999,"15min":15.949999999999999},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.71299999999999997,"15min":0.76300000000000001},"min":{"1min":0.36799999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.046,"5min":2.5499999999999998,"15min":4.766},"last":0.67700000000000005}]},{"osd":3,"last update":"Wed Mar 24 22:02:38 2021","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.86199999999999999,"15min":0.84399999999999997},"min":{"1min":0.5,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.1899999999999999,"5min":3.7869999999999999,"15min":5.9790000000000001},"last":1.2},{"interface":"front","average":{"1min":0.755,"5min":0.88,"15min":0.92300000000000004},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":1.2010000000000001,"5min":4.0229999999999997,"15min":6.4580000000000002},"last":0.83099999999999996}]},{"osd":4,"last update":"Wed Mar 24 22:02:58 2021","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":1.0149999999999999,"15min":1.0640000000000001},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.436},"max":{"1min":1.4790000000000001,"5min":3.8300000000000001,"15min":5.0330000000000004},"last":1.1100000000000001},{"interface":"front","average":{"1min":0.99299999999999999,"5min":1.032,"15min":1.054},"min":{"1min":0.63100000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.3160000000000001,"5min":4.008,"15min":6.1829999999999998},"last":1.325}]},{"osd":5,"last update":"Wed Mar 24 22:03:05 2021","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.151,"15min":1.208},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.6379999999999999,"5min":3.9550000000000001,"15min":3.9550000000000001},"last":1.0189999999999999},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.165,"15min":1.2250000000000001},"min":{"1min":0.73399999999999999,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.6240000000000001,"5min":4.2560000000000002,"15min":4.2560000000000002},"last":1.425}]},{"osd":6,"last update":"Wed Mar 24 22:02:38 2021","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":1.204,"15min":1.1399999999999999},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.4199999999999999,"5min":15.983000000000001,"15min":15.983000000000001},"last":0.96699999999999997},{"interface":"front","average":{"1min":1.0289999999999999,"5min":1.0509999999999999,"15min":1.1599999999999999},"min":{"1min":0.60699999999999998,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.6910000000000001,"5min":2.21,"15min":4.9960000000000004},"last":1.2390000000000001}]},{"osd":7,"last update":"Wed Mar 24 22:03:05 2021","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.1140000000000001,"15min":1.1639999999999999},"min":{"1min":0.60799999999999998,"5min":0.58599999999999997,"15min":0.55700000000000005},"max":{"1min":1.704,"5min":4.1669999999999998,"15min":6.1319999999999997},"last":1.3799999999999999},{"interface":"front","average":{"1min":1.052,"5min":1.214,"15min":1.181},"min":{"1min":0.65700000000000003,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.5069999999999999,"5min":16.007000000000001,"15min":16.007000000000001},"last":1.165}]}]},{"osd":2,"up_from":18,"seq":77309411494,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":211984,"kb_used_data":166280,"kb_used_omap":0,"kb_used_meta":45696,"kb_avail":93533168,"statfs":{"total":95995035648,"available":95777964032,"internally_reserved":0,"allocated":170270720,"data_stored":217872961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":46792704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 22:03:12 2021","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.78000000000000003,"15min":0.70299999999999996},"min":{"1min":0.42899999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":1.1200000000000001,"5min":10.054,"15min":10.054},"last":0.68100000000000005},{"interface":"front","average":{"1min":1.0580000000000001,"5min":0.871,"15min":0.76600000000000001},"min":{"1min":0.505,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":7.8979999999999997,"5min":12.099,"15min":12.099},"last":0.59099999999999997}]},{"osd":1,"last update":"Wed Mar 24 22:03:12 2021","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.88400000000000001,"15min":0.78500000000000003},"min":{"1min":0.54800000000000004,"5min":0.309,"15min":0.28100000000000003},"max":{"1min":1.5880000000000001,"5min":17.954999999999998,"15min":17.954999999999998},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.72399999999999998,"15min":0.755},"min":{"1min":0.28699999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.9490000000000001,"5min":1.9490000000000001,"15min":1.9490000000000001},"last":0.64100000000000001}]},{"osd":3,"last update":"Wed Mar 24 22:02:40 2021","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":1.075,"15min":0.92600000000000005},"min":{"1min":0.54600000000000004,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.3089999999999999,"5min":25.009,"15min":25.009},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.87,"5min":0.86099999999999999,"15min":0.89100000000000001},"min":{"1min":0.56399999999999995,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.202,"5min":2.6440000000000001,"15min":2.6440000000000001},"last":0.71199999999999997}]},{"osd":4,"last update":"Wed Mar 24 22:02:58 2021","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":1.1850000000000001,"15min":1.1000000000000001},"min":{"1min":0.47799999999999998,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.4079999999999999,"5min":24.914000000000001,"15min":24.914000000000001},"last":0.80700000000000005},{"interface":"front","average":{"1min":1.0009999999999999,"5min":0.99399999999999999,"15min":1.0349999999999999},"min":{"1min":0.622,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.323,"5min":2.8290000000000002,"15min":2.8290000000000002},"last":0.84399999999999997}]},{"osd":5,"last update":"Wed Mar 24 22:03:09 2021","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.121,"15min":1.1200000000000001},"min":{"1min":0.78900000000000003,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":4.3019999999999996,"5min":4.3019999999999996,"15min":4.3019999999999996},"last":0.77200000000000002},{"interface":"front","average":{"1min":1.149,"5min":1.101,"15min":1.0960000000000001},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.8360000000000001,"5min":2.3250000000000002,"15min":2.3250000000000002},"last":0.93400000000000005}]},{"osd":6,"last update":"Wed Mar 24 22:02:40 2021","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.0209999999999999,"15min":1.1419999999999999},"min":{"1min":0.83699999999999997,"5min":0.434,"15min":0.434},"max":{"1min":1.3180000000000001,"5min":2.7719999999999998,"15min":2.7719999999999998},"last":0.753},{"interface":"front","average":{"1min":1.1439999999999999,"5min":1.113,"15min":1.1299999999999999},"min":{"1min":0.57999999999999996,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.3999999999999999,"5min":6.6520000000000001,"15min":6.6520000000000001},"last":0.85799999999999998}]},{"osd":7,"last update":"Wed Mar 24 22:03:01 2021","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.173,"15min":1.1990000000000001},"min":{"1min":0.80500000000000005,"5min":0.442,"15min":0.442},"max":{"1min":4.3630000000000004,"5min":6.7720000000000002,"15min":6.7720000000000002},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.226,"5min":1.3620000000000001,"15min":1.2749999999999999},"min":{"1min":0.997,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.6399999999999999,"5min":24.969999999999999,"15min":24.969999999999999},"last":0.76300000000000001}]}]},{"osd":3,"up_from":26,"seq":111669149857,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":257004,"kb_used_data":206180,"kb_used_omap":0,"kb_used_meta":50816,"kb_avail":93488148,"statfs":{"total":95995035648,"available":95731863552,"internally_reserved":0,"allocated":211128320,"data_stored":272483297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":52035584},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 22:02:31 2021","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":1.0489999999999999,"15min":0.85199999999999998},"min":{"1min":0.38,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":1.032,"5min":27.548999999999999,"15min":27.548999999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.63300000000000001,"5min":1.0049999999999999,"15min":0.89000000000000001},"min":{"1min":0.35099999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.1719999999999999,"5min":26.593,"15min":26.593},"last":0.80600000000000005}]},{"osd":1,"last update":"Wed Mar 24 22:02:31 2021","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.69499999999999995,"15min":0.77000000000000002},"min":{"1min":0.314,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.83599999999999997,"5min":5.1280000000000001,"15min":14.324},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.69799999999999995,"15min":0.77000000000000002},"min":{"1min":0.38700000000000001,"5min":0.29599999999999999,"15min":0.28000000000000003},"max":{"1min":1.1479999999999999,"5min":2.423,"15min":12.631},"last":0.77300000000000002}]},{"osd":2,"last update":"Wed Mar 24 22:02:31 2021","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.80600000000000005,"15min":0.83599999999999997},"min":{"1min":0.42399999999999999,"5min":0.316,"15min":0.316},"max":{"1min":1.0169999999999999,"5min":7.9450000000000003,"15min":12.510999999999999},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.76400000000000001,"15min":0.82199999999999995},"min":{"1min":0.52800000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.004,"5min":1.704,"15min":11.317},"last":1.069}]},{"osd":4,"last update":"Wed Mar 24 22:02:57 2021","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.95499999999999996,"15min":0.97399999999999998},"min":{"1min":0.622,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.4570000000000001,"5min":2.601,"15min":2.601},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.92800000000000005,"15min":1.0089999999999999},"min":{"1min":0.54900000000000004,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.258,"5min":1.6299999999999999,"15min":14.291},"last":0.996}]},{"osd":5,"last update":"Wed Mar 24 22:03:06 2021","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":1.0609999999999999,"15min":1.0940000000000001},"min":{"1min":0.68999999999999995,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.4099999999999999,"5min":1.5369999999999999,"15min":1.5369999999999999},"last":1.286},{"interface":"front","average":{"1min":1.042,"5min":1.0680000000000001,"15min":1.0940000000000001},"min":{"1min":0.71499999999999997,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.5960000000000001,"5min":2.2759999999999998,"15min":2.2759999999999998},"last":1.2370000000000001}]},{"osd":6,"last update":"Wed Mar 24 22:02:31 2021","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.96999999999999997,"15min":1.1100000000000001},"min":{"1min":0.59799999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.3100000000000001,"5min":1.639,"15min":14.255000000000001},"last":1.105},{"interface":"front","average":{"1min":0.94699999999999995,"5min":0.99099999999999999,"15min":1.131},"min":{"1min":0.69999999999999996,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.3069999999999999,"5min":1.7,"15min":12.666},"last":1.135}]},{"osd":7,"last update":"Wed Mar 24 22:03:01 2021","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.069,"15min":1.109},"min":{"1min":0.69299999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.419,"5min":2.633,"15min":2.633},"last":1.254},{"interface":"front","average":{"1min":1.05,"5min":1.1200000000000001,"15min":1.1990000000000001},"min":{"1min":0.627,"5min":0.52000000000000002,"15min":0.44700000000000001},"max":{"1min":1.458,"5min":9.609,"15min":14.727},"last":1.173}]}]},{"osd":4,"up_from":32,"seq":137438953627,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":103908,"kb_used_data":66908,"kb_used_omap":0,"kb_used_meta":36992,"kb_avail":93641244,"statfs":{"total":95995035648,"available":95888633856,"internally_reserved":0,"allocated":68513792,"data_stored":102423866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":37879808},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 22:02:56 2021","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.86299999999999999,"15min":1},"min":{"1min":0.47799999999999998,"5min":0.223,"15min":0.223},"max":{"1min":1.264,"5min":5.9050000000000002,"15min":41.472000000000001},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.80700000000000005,"5min":1.0700000000000001,"15min":1.137},"min":{"1min":0.47799999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.2769999999999999,"5min":17.513999999999999,"15min":44.185000000000002},"last":1.2130000000000001}]},{"osd":1,"last update":"Wed Mar 24 22:02:56 2021","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.90900000000000003,"15min":0.92000000000000004},"min":{"1min":0.42699999999999999,"5min":0.39400000000000002,"15min":0.38100000000000001},"max":{"1min":3.371,"5min":5.75,"15min":5.75},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.95099999999999996,"5min":1.071,"15min":0.94299999999999995},"min":{"1min":0.54600000000000004,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.363,"5min":11.983000000000001,"15min":11.983000000000001},"last":1.087}]},{"osd":2,"last update":"Wed Mar 24 22:02:56 2021","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.84599999999999997,"15min":0.90600000000000003},"min":{"1min":0.62,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.2090000000000001,"5min":2.1779999999999999,"15min":4.4379999999999997},"last":0.93799999999999994},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.96099999999999997,"15min":0.96699999999999997},"min":{"1min":0.46999999999999997,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.8580000000000001,"5min":5.7999999999999998,"15min":5.7999999999999998},"last":1.24}]},{"osd":3,"last update":"Wed Mar 24 22:02:56 2021","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.056,"15min":1},"min":{"1min":0.501,"5min":0.45200000000000001,"15min":0.439},"max":{"1min":1.702,"5min":4.8049999999999997,"15min":4.8049999999999997},"last":0.998},{"interface":"front","average":{"1min":1.04,"5min":1.0289999999999999,"15min":0.99299999999999999},"min":{"1min":0.71899999999999997,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.3320000000000001,"5min":4.1189999999999998,"15min":4.1189999999999998},"last":1.272}]},{"osd":5,"last update":"Wed Mar 24 22:03:05 2021","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.96099999999999997,"15min":0.92700000000000005},"min":{"1min":0.63,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.387,"5min":2.9569999999999999,"15min":2.9569999999999999},"last":1.153},{"interface":"front","average":{"1min":1.125,"5min":1.143,"15min":1.1299999999999999},"min":{"1min":0.76700000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.744,"5min":2.9089999999999998,"15min":2.9089999999999998},"last":1.3300000000000001}]},{"osd":6,"last update":"Wed Mar 24 22:02:28 2021","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.077,"15min":1.0740000000000001},"min":{"1min":0.746,"5min":0.373,"15min":0.373},"max":{"1min":7.7679999999999998,"5min":7.7679999999999998,"15min":7.7679999999999998},"last":1.127},{"interface":"front","average":{"1min":1.8819999999999999,"5min":1.1819999999999999,"15min":1.1160000000000001},"min":{"1min":0.874,"5min":0.41199999999999998,"15min":0.41099999999999998},"max":{"1min":14.122999999999999,"5min":14.122999999999999,"15min":14.122999999999999},"last":1.181}]},{"osd":7,"last update":"Wed Mar 24 22:03:00 2021","interfaces":[{"interface":"back","average":{"1min":1.2250000000000001,"5min":1.101,"15min":1.147},"min":{"1min":0.74099999999999999,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.8899999999999999,"5min":3.3479999999999999,"15min":3.3479999999999999},"last":1.363},{"interface":"front","average":{"1min":1.103,"5min":1.0580000000000001,"15min":1.1499999999999999},"min":{"1min":0.84799999999999998,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.448,"5min":2.0350000000000001,"15min":2.0350000000000001},"last":1.3819999999999999}]}]},{"osd":5,"up_from":432,"seq":1855425871913,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":130940,"kb_used_data":104308,"kb_used_omap":33,"kb_used_meta":26590,"kb_avail":93614212,"statfs":{"total":95995035648,"available":95860953088,"internally_reserved":0,"allocated":106811392,"data_stored":136752992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33936,"internal_metadata":27229040},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 22:03:04 2021","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":3.4769999999999999,"15min":4.6840000000000002},"min":{"1min":0.51000000000000001,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.1579999999999999,"5min":76.159999999999997,"15min":76.159999999999997},"last":46.884999999999998},{"interface":"front","average":{"1min":0.82199999999999995,"5min":3.419,"15min":4.593},"min":{"1min":0.42099999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.804,"5min":75.311999999999998,"15min":75.311999999999998},"last":43.854999999999997}]},{"osd":1,"last update":"Wed Mar 24 22:03:04 2021","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.83099999999999996,"15min":0.81499999999999995},"min":{"1min":0.64600000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.694,"5min":1.694,"15min":1.694},"last":43.801000000000002},{"interface":"front","average":{"1min":1.046,"5min":0.89500000000000002,"15min":0.88400000000000001},"min":{"1min":0.69499999999999995,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":2.5089999999999999,"5min":2.5089999999999999,"15min":2.5089999999999999},"last":49.761000000000003}]},{"osd":2,"last update":"Wed Mar 24 22:03:04 2021","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.84799999999999998,"15min":0.86099999999999999},"min":{"1min":0.59199999999999997,"5min":0.434,"15min":0.434},"max":{"1min":1.262,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":44.591000000000001},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.95699999999999996,"15min":0.97199999999999998},"min":{"1min":0.65200000000000002,"5min":0.495,"15min":0.495},"max":{"1min":2.5910000000000002,"5min":2.6070000000000002,"15min":2.6070000000000002},"last":49.448999999999998}]},{"osd":3,"last update":"Wed Mar 24 22:03:04 2021","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":1.0840000000000001,"15min":1.149},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.23,"5min":4.1340000000000003,"15min":4.1340000000000003},"last":44.939},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.1819999999999999,"15min":1.242},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.3700000000000001,"5min":4.173,"15min":4.173},"last":45.020000000000003}]},{"osd":4,"last update":"Wed Mar 24 22:03:04 2021","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":0.94899999999999995,"15min":0.94499999999999995},"min":{"1min":0.61599999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.4690000000000001,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":0.85899999999999999},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.0860000000000001,"15min":1.1080000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.444,"5min":1.9350000000000001,"15min":1.9350000000000001},"last":1.0469999999999999}]},{"osd":6,"last update":"Wed Mar 24 22:03:04 2021","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.1180000000000001,"15min":1.125},"min":{"1min":0.73199999999999998,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.554,"5min":2.0510000000000002,"15min":2.0510000000000002},"last":1.256},{"interface":"front","average":{"1min":1.056,"5min":1.081,"15min":1.083},"min":{"1min":0.70899999999999996,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.516,"5min":1.851,"15min":1.851},"last":1.091}]},{"osd":7,"last update":"Wed Mar 24 22:03:04 2021","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.181,"15min":1.208},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.6359999999999999,"5min":2.161,"15min":2.161},"last":1.3540000000000001},{"interface":"front","average":{"1min":1.198,"5min":1.177,"15min":1.173},"min":{"1min":0.873,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.698,"5min":1.8919999999999999,"15min":1.8919999999999999},"last":0.96999999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":984,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":5284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":15870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":6670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":286720,"data_stored":10238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":352256,"data_stored":12738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":5674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":8740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":5358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":144396288,"data_stored":198117946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":71413760,"data_stored":114728774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":167387136,"data_stored":218172257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":206491648,"data_stored":269897344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":63811584,"data_stored":99842200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":102273024,"data_stored":134156308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":126582784,"data_stored":165854691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":160800768,"data_stored":229532273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-03-24T22:03:25.095 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2021-03-24T22:03:25.096 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-24T22:03:25.096 INFO:tasks.thrashosds.thrasher:Removing osd 1, in_osds are: [0, 2, 3, 4, 5, 1, 6, 7] 2021-03-24T22:03:25.096 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd out 1 2021-03-24T22:03:25.113 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:25.196 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:25.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:25.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:03:25.302 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:25.333 INFO:tasks.rados.rados.0.smithi125.stdout:2402: done (2 left) 2021-03-24T22:03:25.334 INFO:tasks.rados.rados.0.smithi125.stdout:2403: delete oid 15 current snap is 270 2021-03-24T22:03:25.334 INFO:tasks.rados.rados.0.smithi125.stdout:2399: finishing write tid 3 to smithi12556470-36 2021-03-24T22:03:25.334 INFO:tasks.rados.rados.0.smithi125.stdout:2399: finishing write tid 4 to smithi12556470-36 2021-03-24T22:03:25.335 INFO:tasks.rados.rados.0.smithi125.stdout:2399: finishing write tid 5 to smithi12556470-36 2021-03-24T22:03:25.335 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 36 v 862 (ObjNum 825 snap 270 seq_num 825) dirty exists 2021-03-24T22:03:25.335 INFO:tasks.rados.rados.0.smithi125.stdout:2399: left oid 36 (ObjNum 825 snap 270 seq_num 825) 2021-03-24T22:03:25.335 INFO:tasks.rados.rados.0.smithi125.stdout:2401: expect (ObjNum 791 snap 257 seq_num 791) 2021-03-24T22:03:25.365 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:25.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:25.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:03:25.438 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-24T22:03:25.438 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-24T22:03:25.438 INFO:teuthology.orchestra.run.smithi125.stdout: "num_ops": 0 2021-03-24T22:03:25.439 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-24T22:03:25.469 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:25.521 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:25 smithi125 conmon[19936]: cluster 2021-03-24T22:03:24.456277 2021-03-24T22:03:25.522 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:25 smithi125 conmon[19936]: +0000 mon.a (mon.0) 1701 : cluster [DBG] osdmap e634: 8 total, 8 up, 8 in 2021-03-24T22:03:25.522 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:25 smithi125 conmon[19936]: cluster 2021-03-24T22:03:25.329978+0000 mon.a (mon.0) 1702 : 2021-03-24T22:03:25.522 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:25 smithi125 conmon[19936]: cluster [DBG] osdmap e635: 8 total, 8 up, 8 in 2021-03-24T22:03:25.523 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:25 smithi125 conmon[24927]: cluster 2021-03-24T22:03:24.456277+0000 2021-03-24T22:03:25.523 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:25 smithi125 conmon[24927]: mon.a (mon.0) 1701 : cluster [DBG] osdmap e634: 8 total, 8 up, 8 in 2021-03-24T22:03:25.523 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:25 smithi125 conmon[24927]: cluster 2021-03-24T22 2021-03-24T22:03:25.523 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:25 smithi125 conmon[24927]: :03:25.329978+0000 mon.a (mon.0) 1702 : cluster 2021-03-24T22:03:25.524 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:25 smithi125 conmon[24927]: [DBG] osdmap e635: 8 total, 8 up, 8 in 2021-03-24T22:03:25.524 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:25.525 INFO:tasks.rados.rados.0.smithi125.stdout:2399: done (2 left) 2021-03-24T22:03:25.525 INFO:tasks.rados.rados.0.smithi125.stdout:2401: done (1 left) 2021-03-24T22:03:25.525 INFO:tasks.rados.rados.0.smithi125.stdout:2403: done (0 left) 2021-03-24T22:03:25.525 INFO:tasks.rados.rados.0.smithi125.stdout:2404: snap_create 2021-03-24T22:03:25.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:25.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:03:25.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:25 smithi142 conmon[20125]: cluster 2021-03-24T22:03: 2021-03-24T22:03:25.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:25 smithi142 conmon[20125]: 24.456277+0000 mon.a (mon.0) 1701 : cluster [DBG] osdmap e634: 8 total, 8 up, 8 in 2021-03-24T22:03:25.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:25 smithi142 conmon[20125]: cluster 2021-03-24T22 2021-03-24T22:03:25.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:25 smithi142 conmon[20125]: :03:25.329978+0000 mon.a (mon.0) 1702 : cluster [DBG] 2021-03-24T22:03:25.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:25 smithi142 conmon[20125]: osdmap e635: 8 total, 8 up, 8 in 2021-03-24T22:03:25.633 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:25.689 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:25.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:25.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:03:25.795 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:25.810 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:03:25.851 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:25.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:25.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:03:25.858 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:03:25.958 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:26.207 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:26.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:26.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:03:26.313 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:26.369 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:26.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:26.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:03:26.474 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:26.490 INFO:tasks.rados.rados.0.smithi125.stdout:2404: done (0 left) 2021-03-24T22:03:26.490 INFO:tasks.rados.rados.0.smithi125.stdout:2405: write oid 17 current snap is 271 2021-03-24T22:03:26.491 INFO:tasks.rados.rados.0.smithi125.stdout:2405: seq_num 828 ranges {716196=472296,1769561=646611,3121885=793109} 2021-03-24T22:03:26.495 INFO:teuthology.orchestra.run.smithi125.stderr:nodeep-scrub is set 2021-03-24T22:03:26.497 INFO:tasks.rados.rados.0.smithi125.stdout:2405: writing smithi12556470-17 from 716196 to 1188492 tid 1 2021-03-24T22:03:26.504 INFO:tasks.rados.rados.0.smithi125.stdout:2405: writing smithi12556470-17 from 1769561 to 2416172 tid 2 2021-03-24T22:03:26.514 INFO:tasks.rados.rados.0.smithi125.stdout:2405: writing smithi12556470-17 from 3121885 to 3914994 tid 3 2021-03-24T22:03:26.514 INFO:tasks.rados.rados.0.smithi125.stdout:2406: rollback oid 10 current snap is 271 2021-03-24T22:03:26.514 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 10 to 216 2021-03-24T22:03:26.515 INFO:tasks.rados.rados.0.smithi125.stdout:2405: finishing write tid 1 to smithi12556470-17 2021-03-24T22:03:26.515 INFO:tasks.rados.rados.0.smithi125.stdout:2405: finishing write tid 2 to smithi12556470-17 2021-03-24T22:03:26.515 INFO:tasks.rados.rados.0.smithi125.stdout:2407: read oid 35 snap -1 2021-03-24T22:03:26.515 INFO:tasks.rados.rados.0.smithi125.stdout:2407: expect (ObjNum 786 snap 257 seq_num 786) 2021-03-24T22:03:26.515 INFO:tasks.rados.rados.0.smithi125.stdout:2408: snap_remove snap 219 2021-03-24T22:03:26.532 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:26.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:26.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:03:26.638 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:26.699 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:26.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:26.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:03:26.805 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:26.864 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:26.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:26 smithi142 conmon[20125]: cluster 2021-03-24T22:03:24.927897+0000 mgr.y (mgr.14144) 591 : 2021-03-24T22:03:26.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:26 smithi142 conmon[20125]: cluster [DBG] pgmap v1114: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 475 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 21 MiB/s rd, 1.8 MiB/s wr, 18 op/s; 40/1641 objects misplaced (2.438%) 2021-03-24T22:03:26.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:26 smithi142 conmon[20125]: audit 2021-03-24T22:03:25.832227+0000 mon.c (mon.2) 97 : audit [INF] from='client.? v1:172.21.15.125:0/1685352764' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:03:26.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:26 smithi142 conmon[20125]: audit 2021 2021-03-24T22:03:26.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:26 smithi142 conmon[20125]: -03-24T22:03:25.832824+0000 mon.a (mon.0) 1703 : audit [INF] from='client.35579 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:03:26.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:26.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:03:26.971 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:27.033 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:27.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:27.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:03:27.139 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:27.162 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:26 smithi125 conmon[19936]: cluster 2021-03-24T22:03:24.927897+0000 2021-03-24T22:03:27.162 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:26 smithi125 conmon[19936]: mgr.y (mgr.14144) 591 : cluster [DBG] pgmap v1114: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 475 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 21 MiB/s rd, 1.8 MiB/s wr, 18 op/s; 40/1641 objects misplaced (2.438%) 2021-03-24T22:03:27.162 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:26 smithi125 conmon[19936]: audit 2021-03-24T22:03:25.832227 2021-03-24T22:03:27.163 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:26 smithi125 conmon[19936]: +0000 mon.c (mon.2) 97 : audit [INF] from='client.? v1:172.21.15.125:0/1685352764' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:03:27.163 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:26 smithi125 conmon[19936]: audit 2021-03 2021-03-24T22:03:27.163 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:26 smithi125 conmon[19936]: -24T22:03:25.832824+0000 mon.a 2021-03-24T22:03:27.163 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:26 smithi125 conmon[19936]: (mon.0) 1703 : audit [INF] 2021-03-24T22:03:27.163 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:26 smithi125 conmon[19936]: from='client.35579 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:03:27.164 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:26 smithi125 conmon[24927]: cluster 2021-03-24T22:03:24.927897+0000 2021-03-24T22:03:27.164 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:26 smithi125 conmon[24927]: mgr.y (mgr.14144) 591 : cluster [DBG] pgmap v1114: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 475 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 21 MiB/s rd, 1.8 MiB/s wr, 18 op/s; 40/1641 objects misplaced (2.438%) 2021-03-24T22:03:27.164 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:26 smithi125 conmon[24927]: audit 2021-03-24T22:03: 2021-03-24T22:03:27.165 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:26 smithi125 conmon[24927]: 25.832227+0000 mon.c (mon.2) 97 : audit [INF] from='client.? v1:172.21.15.125:0/1685352764' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:03:27.165 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:26 smithi125 conmon[24927]: 2021-03-24T22:03:27.165 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:26 smithi125 conmon[24927]: audit 2021-03 2021-03-24T22:03:27.165 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:26 smithi125 conmon[24927]: -24T22:03:25. 2021-03-24T22:03:27.165 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:26 smithi125 conmon[24927]: 832824+0000 mon.a (mon.0 2021-03-24T22:03:27.166 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:26 smithi125 conmon[24927]: ) 1703 : audit [INF] from='client.35579 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:03:27.199 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:27.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:27.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:03:27.306 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:27.359 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:27.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:27.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:03:27.464 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:27.518 INFO:tasks.rados.rados.0.smithi125.stdout:2408: done (3 left) 2021-03-24T22:03:27.519 INFO:tasks.rados.rados.0.smithi125.stdout:2409: write oid 6 current snap is 271 2021-03-24T22:03:27.519 INFO:tasks.rados.rados.0.smithi125.stdout:2409: seq_num 829 ranges {733003=663898,1944396=506622,2865661=604381} 2021-03-24T22:03:27.519 INFO:tasks.rados.rados.0.smithi125.stdout:2409: writing smithi12556470-6 from 733003 to 1396901 tid 1 2021-03-24T22:03:27.522 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:27.526 INFO:tasks.rados.rados.0.smithi125.stdout:2409: writing smithi12556470-6 from 1944396 to 2451018 tid 2 2021-03-24T22:03:27.533 INFO:tasks.rados.rados.0.smithi125.stdout:2409: writing smithi12556470-6 from 2865661 to 3470042 tid 3 2021-03-24T22:03:27.534 INFO:tasks.rados.rados.0.smithi125.stdout:2406: finishing rollback tid 1 to smithi12556470-10 2021-03-24T22:03:27.534 INFO:tasks.rados.rados.0.smithi125.stdout:2406: finishing rollback tid 2 to smithi12556470-10 2021-03-24T22:03:27.534 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 10 v 952 (ObjNum 642 snap 196 seq_num 642) dirty exists 2021-03-24T22:03:27.535 INFO:tasks.rados.rados.0.smithi125.stdout:2407: expect (ObjNum 786 snap 257 seq_num 786) 2021-03-24T22:03:27.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:27.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:03:27.648 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:27.706 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:27.711 INFO:tasks.rados.rados.0.smithi125.stdout:2405: finishing write tid 3 to smithi12556470-17 2021-03-24T22:03:27.711 INFO:tasks.rados.rados.0.smithi125.stdout:2405: finishing write tid 4 to smithi12556470-17 2021-03-24T22:03:27.711 INFO:tasks.rados.rados.0.smithi125.stdout:2405: finishing write tid 5 to smithi12556470-17 2021-03-24T22:03:27.712 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 17 v 779 (ObjNum 828 snap 271 seq_num 828) dirty exists 2021-03-24T22:03:27.712 INFO:tasks.rados.rados.0.smithi125.stdout:2405: left oid 17 (ObjNum 828 snap 271 seq_num 828) 2021-03-24T22:03:27.712 INFO:tasks.rados.rados.0.smithi125.stdout:2409: finishing write tid 1 to smithi12556470-6 2021-03-24T22:03:27.712 INFO:tasks.rados.rados.0.smithi125.stdout:2405: done (3 left) 2021-03-24T22:03:27.713 INFO:tasks.rados.rados.0.smithi125.stdout:2406: done (2 left) 2021-03-24T22:03:27.713 INFO:tasks.rados.rados.0.smithi125.stdout:2407: done (1 left) 2021-03-24T22:03:27.713 INFO:tasks.rados.rados.0.smithi125.stdout:2410: copy_from oid 45 from oid 26 current snap is 271 2021-03-24T22:03:27.713 INFO:tasks.rados.rados.0.smithi125.stdout:2409: finishing write tid 2 to smithi12556470-6 2021-03-24T22:03:27.713 INFO:tasks.rados.rados.0.smithi125.stdout:2409: finishing write tid 3 to smithi12556470-6 2021-03-24T22:03:27.714 INFO:tasks.rados.rados.0.smithi125.stdout:2409: finishing write tid 4 to smithi12556470-6 2021-03-24T22:03:27.714 INFO:tasks.rados.rados.0.smithi125.stdout:2409: finishing write tid 5 to smithi12556470-6 2021-03-24T22:03:27.714 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 6 v 549 (ObjNum 829 snap 271 seq_num 829) dirty exists 2021-03-24T22:03:27.714 INFO:tasks.rados.rados.0.smithi125.stdout:2409: left oid 6 (ObjNum 829 snap 271 seq_num 829) 2021-03-24T22:03:27.715 INFO:tasks.rados.rados.0.smithi125.stdout:2409: done (1 left) 2021-03-24T22:03:27.715 INFO:tasks.rados.rados.0.smithi125.stdout:2411: read oid 39 snap -1 2021-03-24T22:03:27.715 INFO:tasks.rados.rados.0.smithi125.stdout:2411: expect deleted 2021-03-24T22:03:27.715 INFO:tasks.rados.rados.0.smithi125.stdout:2412: rollback oid 10 current snap is 271 2021-03-24T22:03:27.715 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 10 to 266 2021-03-24T22:03:27.716 INFO:tasks.rados.rados.0.smithi125.stdout:2413: write (excl) oid 6 current snap is 271 2021-03-24T22:03:27.716 INFO:tasks.rados.rados.0.smithi125.stdout:2413: seq_num 830 ranges {749810=739147,2002878=650280,3293084=415653} 2021-03-24T22:03:27.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:27.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:03:27.723 INFO:tasks.rados.rados.0.smithi125.stdout:2413: writing smithi12556470-6 from 749810 to 1488957 tid 1 2021-03-24T22:03:27.731 INFO:tasks.rados.rados.0.smithi125.stdout:2413: writing smithi12556470-6 from 2002878 to 2653158 tid 2 2021-03-24T22:03:27.738 INFO:tasks.rados.rados.0.smithi125.stdout:2413: writing smithi12556470-6 from 3293084 to 3708737 tid 3 2021-03-24T22:03:27.738 INFO:tasks.rados.rados.0.smithi125.stdout:2414: read oid 3 snap -1 2021-03-24T22:03:27.738 INFO:tasks.rados.rados.0.smithi125.stdout:2414: expect deleted 2021-03-24T22:03:27.739 INFO:tasks.rados.rados.0.smithi125.stdout:2415: read oid 7 snap 246 2021-03-24T22:03:27.739 INFO:tasks.rados.rados.0.smithi125.stdout:2415: expect (ObjNum 727 snap 233 seq_num 727) 2021-03-24T22:03:27.739 INFO:tasks.rados.rados.0.smithi125.stdout:2416: write oid 17 current snap is 271 2021-03-24T22:03:27.739 INFO:tasks.rados.rados.0.smithi125.stdout:2416: seq_num 831 ranges {766617=414396,1945007=510291,3204154=626925} 2021-03-24T22:03:27.744 INFO:tasks.rados.rados.0.smithi125.stdout:2416: writing smithi12556470-17 from 766617 to 1181013 tid 1 2021-03-24T22:03:27.752 INFO:tasks.rados.rados.0.smithi125.stdout:2416: writing smithi12556470-17 from 1945007 to 2455298 tid 2 2021-03-24T22:03:27.760 INFO:tasks.rados.rados.0.smithi125.stdout:2416: writing smithi12556470-17 from 3204154 to 3831079 tid 3 2021-03-24T22:03:27.760 INFO:tasks.rados.rados.0.smithi125.stdout:2417: rollback oid 50 current snap is 271 2021-03-24T22:03:27.761 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 50 to 192 2021-03-24T22:03:27.761 INFO:tasks.rados.rados.0.smithi125.stdout:2412: finishing rollback tid 0 to smithi12556470-10 2021-03-24T22:03:27.761 INFO:tasks.rados.rados.0.smithi125.stdout:2412: finishing rollback tid 1 to smithi12556470-10 2021-03-24T22:03:27.761 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 10 v 958 (ObjNum 1953066355 snap 842099048 seq_num 875967797) dirty dne 2021-03-24T22:03:27.761 INFO:tasks.rados.rados.0.smithi125.stdout:2411: done (7 left) 2021-03-24T22:03:27.762 INFO:tasks.rados.rados.0.smithi125.stdout:2412: done (6 left) 2021-03-24T22:03:27.762 INFO:tasks.rados.rados.0.smithi125.stdout:2418: snap_remove snap 266 2021-03-24T22:03:27.819 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:27.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:27 smithi142 conmon[20125]: cluster 2021-03-24T22:03:26.470738+0000 mon.a (mon.0) 1704 : cluster 2021-03-24T22:03:27.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:27 smithi142 conmon[20125]: [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:03:27.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:27 smithi142 conmon[20125]: audit 2021-03-24T22:03:26.485698+0000 mon.a (mon.0) 1705 : audit [INF] from='client.35579 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T22:03:27.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:27 smithi142 conmon[20125]: 2021-03-24T22:03:27.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:27 smithi142 conmon[20125]: cluster 2021-03-24T22:03 2021-03-24T22:03:27.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:27 smithi142 conmon[20125]: :26.485885+0000 mon.a (mon.0) 1706 : cluster [DBG] osdmap e636: 8 total, 8 up, 8 in 2021-03-24T22:03:27.871 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:27.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:27.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:03:27.978 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:28.026 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:27 smithi125 conmon[19936]: cluster 2021-03-24T22:03 2021-03-24T22:03:28.026 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:27 smithi125 conmon[19936]: :26.470738+0000 mon.a (mon.0) 1704 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:03:28.027 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:27 smithi125 conmon[19936]: audit 2021-03-24T22:03:26.485698+0000 2021-03-24T22:03:28.027 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:27 smithi125 conmon[19936]: mon.a (mon.0) 1705 : audit [INF] from='client.35579 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T22:03:28.027 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:27 smithi125 conmon[19936]: 2021-03-24T22:03:28.027 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:27 smithi125 conmon[19936]: cluster 2021-03-24T22:03 2021-03-24T22:03:28.028 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:27 smithi125 conmon[19936]: :26.485885+0000 mon.a (mon.0) 1706 : 2021-03-24T22:03:28.028 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:27 smithi125 conmon[19936]: cluster [DBG] osdmap e636: 8 total, 8 up, 8 in 2021-03-24T22:03:28.028 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:27 smithi125 conmon[24927]: cluster 2021-03-24T22 2021-03-24T22:03:28.028 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:27 smithi125 conmon[24927]: :03:26.470738+0000 mon.a (mon.0) 1704 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:03:28.029 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:27 smithi125 conmon[24927]: audit 2021- 2021-03-24T22:03:28.029 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:27 smithi125 conmon[24927]: 03-24T22:03:26. 2021-03-24T22:03:28.029 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:27 smithi125 conmon[24927]: 485698+0000 mon.a (mon.0) 2021-03-24T22:03:28.029 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:27 smithi125 conmon[24927]: 1705 : audit [INF] from='client.35579 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T22:03:28.030 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:27 smithi125 conmon[24927]: cluster 2021-03-24T22:03:28.030 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:27 smithi125 conmon[24927]: 2021-03-24T22:03 2021-03-24T22:03:28.030 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:27 smithi125 conmon[24927]: :26.485885+0000 mon.a (mon 2021-03-24T22:03:28.030 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:27 smithi125 conmon[24927]: .0) 1706 : cluster [DBG] osdmap e636: 8 total, 8 up, 8 in 2021-03-24T22:03:28.031 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:27 smithi125 conmon[24927]: 2021-03-24T22:03:28.035 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:28.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:28.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:03:28.142 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:28.196 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:28.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:28.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:03:28.303 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:28.362 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:28.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:28.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:03:28.468 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.1 dump_blocked_ops 2021-03-24T22:03:28.469 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:28.524 INFO:tasks.rados.rados.0.smithi125.stdout:2418: done (6 left) 2021-03-24T22:03:28.525 INFO:tasks.rados.rados.0.smithi125.stdout:2419: snap_create 2021-03-24T22:03:28.525 INFO:tasks.rados.rados.0.smithi125.stdout:2410: finishing copy_from to smithi12556470-45 2021-03-24T22:03:28.525 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 45 v 447 (ObjNum 794 snap 257 seq_num 794) dirty exists 2021-03-24T22:03:28.525 INFO:tasks.rados.rados.0.smithi125.stdout:2410: finishing copy_from racing read to smithi12556470-45 2021-03-24T22:03:28.526 INFO:tasks.rados.rados.0.smithi125.stdout:2413: finishing write tid 1 to smithi12556470-6 2021-03-24T22:03:28.526 INFO:tasks.rados.rados.0.smithi125.stdout:2413: finishing write tid 2 to smithi12556470-6 2021-03-24T22:03:28.526 INFO:tasks.rados.rados.0.smithi125.stdout:2413: finishing write tid 3 to smithi12556470-6 2021-03-24T22:03:28.526 INFO:tasks.rados.rados.0.smithi125.stdout:2413: finishing write tid 4 to smithi12556470-6 2021-03-24T22:03:28.527 INFO:tasks.rados.rados.0.smithi125.stdout:2413: finishing write tid 5 to smithi12556470-6 2021-03-24T22:03:28.527 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 6 v 553 (ObjNum 830 snap 271 seq_num 830) dirty exists 2021-03-24T22:03:28.527 INFO:tasks.rados.rados.0.smithi125.stdout:2413: left oid 6 (ObjNum 830 snap 271 seq_num 830) 2021-03-24T22:03:28.527 INFO:tasks.rados.rados.0.smithi125.stdout:2415: expect (ObjNum 727 snap 233 seq_num 727) 2021-03-24T22:03:28.545 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:28.549 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:28 smithi125 conmon[19936]: cluster 2021-03-24T22:03:26.928825+0000 2021-03-24T22:03:28.550 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:28 smithi125 conmon[19936]: mgr.y (mgr.14144) 592 : cluster [DBG] pgmap v1117: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 479 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 23 MiB/s rd, 3.1 MiB/s wr, 23 op/s; 40/1647 objects misplaced (2.429%) 2021-03-24T22:03:28.550 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:28 smithi125 conmon[19936]: cluster 2021-03-24T22:03:27.506468+0000 mon.a 2021-03-24T22:03:28.550 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:28 smithi125 conmon[19936]: (mon.0) 1707 : cluster [DBG] osdmap e637: 8 total, 8 up, 8 in 2021-03-24T22:03:28.551 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:28 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:03:28.551 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:28 smithi125 conmon[24927]: 03:26.928825+0000 mgr.y (mgr.14144) 592 : cluster [DBG] pgmap v1117: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 479 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 23 MiB/s rd, 3.1 MiB/s wr, 23 op/s; 40/1647 objects misplaced (2.429%) 2021-03-24T22:03:28.552 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:28 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:03:28.552 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:28 smithi125 conmon[24927]: 03:27.506468+0000 mon.a (mon.0) 2021-03-24T22:03:28.553 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:28 smithi125 conmon[24927]: 1707 : cluster [DBG] osdmap e637: 8 total, 8 up, 8 in 2021-03-24T22:03:28.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:28.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:03:28.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:28 smithi142 conmon[20125]: cluster 2021-03-24T22:03:26.928825+0000 mgr.y (mgr.14144) 592 : cluster 2021-03-24T22:03:28.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:28 smithi142 conmon[20125]: [DBG] pgmap v1117: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 479 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 23 MiB/s rd, 3.1 MiB/s wr, 23 op/s; 40/1647 objects misplaced (2.429%) 2021-03-24T22:03:28.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:28 smithi142 conmon[20125]: cluster 2021-03-24T22:03:27.506468+0000 2021-03-24T22:03:28.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:28 smithi142 conmon[20125]: mon.a (mon.0) 1707 : cluster [DBG] osdmap e637: 8 total, 8 up, 8 in 2021-03-24T22:03:28.656 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:28.718 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:28.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:28.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:03:28.725 INFO:tasks.rados.rados.0.smithi125.stdout:2416: finishing write tid 1 to smithi12556470-17 2021-03-24T22:03:28.726 INFO:tasks.rados.rados.0.smithi125.stdout:2416: finishing write tid 2 to smithi12556470-17 2021-03-24T22:03:28.726 INFO:tasks.rados.rados.0.smithi125.stdout:2417: finishing rollback tid 0 to smithi12556470-50 2021-03-24T22:03:28.726 INFO:tasks.rados.rados.0.smithi125.stdout:2417: finishing rollback tid 1 to smithi12556470-50 2021-03-24T22:03:28.727 INFO:tasks.rados.rados.0.smithi125.stdout:2417: finishing rollback tid 2 to smithi12556470-50 2021-03-24T22:03:28.727 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 50 v 796 (ObjNum 599 snap 179 seq_num 599) dirty exists 2021-03-24T22:03:28.727 INFO:tasks.rados.rados.0.smithi125.stdout:2416: finishing write tid 3 to smithi12556470-17 2021-03-24T22:03:28.727 INFO:tasks.rados.rados.0.smithi125.stdout:2416: finishing write tid 4 to smithi12556470-17 2021-03-24T22:03:28.728 INFO:tasks.rados.rados.0.smithi125.stdout:2416: finishing write tid 5 to smithi12556470-17 2021-03-24T22:03:28.728 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 17 v 783 (ObjNum 831 snap 271 seq_num 831) dirty exists 2021-03-24T22:03:28.728 INFO:tasks.rados.rados.0.smithi125.stdout:2416: left oid 17 (ObjNum 831 snap 271 seq_num 831) 2021-03-24T22:03:28.825 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:28.879 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:28.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:28.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:28.986 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:29.046 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:29.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:29.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:03:29.153 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:29.217 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:29.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:29.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:29.323 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:29.381 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:29.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:29.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:29.486 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:29.530 INFO:tasks.rados.rados.0.smithi125.stdout:2410: done (6 left) 2021-03-24T22:03:29.531 INFO:tasks.rados.rados.0.smithi125.stdout:2413: done (5 left) 2021-03-24T22:03:29.531 INFO:tasks.rados.rados.0.smithi125.stdout:2414: done (4 left) 2021-03-24T22:03:29.531 INFO:tasks.rados.rados.0.smithi125.stdout:2415: done (3 left) 2021-03-24T22:03:29.532 INFO:tasks.rados.rados.0.smithi125.stdout:2416: done (2 left) 2021-03-24T22:03:29.532 INFO:tasks.rados.rados.0.smithi125.stdout:2417: done (1 left) 2021-03-24T22:03:29.532 INFO:tasks.rados.rados.0.smithi125.stdout:2419: done (0 left) 2021-03-24T22:03:29.532 INFO:tasks.rados.rados.0.smithi125.stdout:2420: snap_remove snap 267 2021-03-24T22:03:29.546 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:29.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:29.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:29.646 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:29 smithi125 conmon[19936]: cluster 2021-03-24T22:03:28.515730 2021-03-24T22:03:29.646 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:29 smithi125 conmon[19936]: +0000 mon.a (mon.0) 1708 : cluster [DBG] osdmap e638: 8 total, 8 up, 8 in 2021-03-24T22:03:29.647 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:29 smithi125 conmon[24927]: cluster 2021-03-24T22:03:29.647 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:29 smithi125 conmon[24927]: 2021-03-24T22:03:29.647 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:29 smithi125 conmon[24927]: 2021 2021-03-24T22:03:29.648 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:29 smithi125 conmon[24927]: - 2021-03-24T22:03:29.648 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:29 smithi125 conmon[24927]: 03 2021-03-24T22:03:29.648 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:29 smithi125 conmon[24927]: - 2021-03-24T22:03:29.648 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:29 smithi125 conmon[24927]: 24 2021-03-24T22:03:29.649 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:29 smithi125 conmon[24927]: T 2021-03-24T22:03:29.649 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:29 smithi125 conmon[24927]: 22 2021-03-24T22:03:29.649 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:29 smithi125 conmon[24927]: : 2021-03-24T22:03:29.649 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:29 smithi125 conmon[24927]: 03 2021-03-24T22:03:29.650 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:29 smithi125 conmon[24927]: :28.515730+0000 mon.a (mon.0) 1708 : cluster [DBG] osdmap e638: 8 total, 8 up, 8 in 2021-03-24T22:03:29.651 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:29.715 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:29.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:29.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:03:29.821 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:29.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:29 smithi142 conmon[20125]: cluster 2021-03-24T22: 2021-03-24T22:03:29.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:29 smithi142 conmon[20125]: 03:28.515730+0000 mon.a (mon.0) 1708 : cluster [DBG] osdmap e638: 8 total, 8 up, 8 in 2021-03-24T22:03:29.877 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:29.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:29.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:03:29.909 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:03:29.991 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:30.047 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:30.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:30.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:30.157 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:30.214 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:30.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:30.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:03:30.321 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:30.341 INFO:tasks.rados.rados.0.smithi125.stdout:2420: done (0 left) 2021-03-24T22:03:30.342 INFO:tasks.rados.rados.0.smithi125.stdout:2421: read oid 25 snap -1 2021-03-24T22:03:30.342 INFO:tasks.rados.rados.0.smithi125.stdout:2421: expect deleted 2021-03-24T22:03:30.342 INFO:tasks.rados.rados.0.smithi125.stdout:2422: write (excl) oid 40 current snap is 272 2021-03-24T22:03:30.342 INFO:tasks.rados.rados.0.smithi125.stdout:2422: seq_num 832 ranges {783424=489645,2003489=653949,3231577=554550} 2021-03-24T22:03:30.350 INFO:tasks.rados.rados.0.smithi125.stdout:2422: writing smithi12556470-40 from 783424 to 1273069 tid 1 2021-03-24T22:03:30.359 INFO:tasks.rados.rados.0.smithi125.stdout:2422: writing smithi12556470-40 from 2003489 to 2657438 tid 2 2021-03-24T22:03:30.362 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd unset noscrub 2021-03-24T22:03:30.366 INFO:tasks.rados.rados.0.smithi125.stdout:2422: writing smithi12556470-40 from 3231577 to 3786127 tid 3 2021-03-24T22:03:30.366 INFO:tasks.rados.rados.0.smithi125.stdout:2423: snap_remove snap 270 2021-03-24T22:03:30.379 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:30.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:30.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:03:30.484 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:30.541 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:30.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:30.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:03:30.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:30 smithi142 conmon[20125]: cluster 2021-03-24T22:03 2021-03-24T22:03:30.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:30 smithi142 conmon[20125]: :28.929556+0000 mgr.y (mgr.14144) 593 : cluster [DBG] pgmap v1120: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 479 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 6.8 MiB/s rd, 1.2 MiB/s wr, 6 op/s; 40/1647 objects misplaced (2.429%) 2021-03-24T22:03:30.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:30 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:03:30.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:30 smithi142 conmon[20125]: 24T22:03:29.524161+0000 mon.a (mon.0) 1709 : cluster [DBG] osdmap e639: 8 total, 8 up, 8 in 2021-03-24T22:03:30.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:30 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:03:30.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:30 smithi142 conmon[20125]: 03-24T22:03:30.336597+0000 mon.a 2021-03-24T22:03:30.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:30 smithi142 conmon[20125]: (mon.0) 1710 : cluster [DBG] osdmap e640: 8 total, 8 up, 8 in 2021-03-24T22:03:30.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:30 smithi142 conmon[20125]: 2021-03-24T22:03:30.648 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:30.700 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:30.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:30.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:03:30.806 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:30.866 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:30.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:30.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:03:30.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:30 smithi125 conmon[19936]: cluster 2021-03-24T22:03:28.929556+0000 mgr.y (mgr. 2021-03-24T22:03:30.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:30 smithi125 conmon[19936]: 14144) 593 : cluster [DBG] pgmap v1120: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 479 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 6.8 MiB/s rd, 1.2 MiB/s wr, 6 op/s; 40/1647 objects misplaced (2.429%) 2021-03-24T22:03:30.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:30 smithi125 conmon[19936]: cluster 2021-03-24T22:03:29.524161+0000 mon.a (mon.0) 1709 : cluster [DBG] osdmap e639: 8 total, 8 up, 8 in 2021-03-24T22:03:30.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:30 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:03:30.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:30 smithi125 conmon[19936]: 03-24T22:03:30.336597+0000 mon.a (mon.0) 1710 : cluster [DBG] osdmap e640: 8 total, 8 up, 8 in 2021-03-24T22:03:30.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:30 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:03:30.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:30 smithi125 conmon[24927]: 03:28.929556+0000 mgr.y (mgr.14144) 593 : cluster [DBG] pgmap v1120: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 479 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 6.8 MiB/s rd, 1.2 MiB/s wr, 6 op/s; 40/1647 objects misplaced (2.429%) 2021-03-24T22:03:30.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:30 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:03:30.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:30 smithi125 conmon[24927]: 24T22:03:29.524161+0000 mon.a (mon 2021-03-24T22:03:30.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:30 smithi125 conmon[24927]: .0) 1709 : cluster [DBG] osdmap e639: 8 total, 8 up, 8 in 2021-03-24T22:03:30.917 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:30 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:03:30.917 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:30 smithi125 conmon[24927]: -03-24T22:03:30.336597+0000 2021-03-24T22:03:30.917 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:30 smithi125 conmon[24927]: mon.a (mon.0) 1710 : cluster [DBG] osdmap e640: 8 total, 8 up, 8 in 2021-03-24T22:03:30.973 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:31.026 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:31.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:31.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:03:31.092 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:03:31.133 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:31.191 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:31.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:31.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:31.255 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:31 smithi125 conmon[19936]: debug 2021-03-24T22:03:31.028+0000 7f2118128700 -1 mon.a@0(leader).osd e640 definitely_dead 0 2021-03-24T22:03:31.297 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:31.361 INFO:tasks.rados.rados.0.smithi125.stdout:2421: done (2 left) 2021-03-24T22:03:31.361 INFO:tasks.rados.rados.0.smithi125.stdout:2423: done (1 left) 2021-03-24T22:03:31.362 INFO:tasks.rados.rados.0.smithi125.stdout:2424: snap_create 2021-03-24T22:03:31.363 INFO:tasks.rados.rados.0.smithi125.stdout:2422: finishing write tid 1 to smithi12556470-40 2021-03-24T22:03:31.364 INFO:tasks.rados.rados.0.smithi125.stdout:2422: finishing write tid 2 to smithi12556470-40 2021-03-24T22:03:31.364 INFO:tasks.rados.rados.0.smithi125.stdout:2422: finishing write tid 3 to smithi12556470-40 2021-03-24T22:03:31.365 INFO:tasks.rados.rados.0.smithi125.stdout:2422: finishing write tid 4 to smithi12556470-40 2021-03-24T22:03:31.365 INFO:tasks.rados.rados.0.smithi125.stdout:2422: finishing write tid 5 to smithi12556470-40 2021-03-24T22:03:31.367 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 40 v 1401 (ObjNum 832 snap 272 seq_num 832) dirty exists 2021-03-24T22:03:31.367 INFO:tasks.rados.rados.0.smithi125.stdout:2422: left oid 40 (ObjNum 832 snap 272 seq_num 832) 2021-03-24T22:03:31.388 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:31.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:31.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:03:31.394 INFO:teuthology.orchestra.run.smithi125.stderr:marked out osd.1. 2021-03-24T22:03:31.492 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:31.542 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:31.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:31.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:31.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:31 smithi142 conmon[20125]: audit 2021-03-24T22:03:31 2021-03-24T22:03:31.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:31 smithi142 conmon[20125]: .030125+0000 mon.a (mon.0) 1711 : audit [INF] from='client.? v1:172.21.15.125:0/431425811' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2021-03-24T22:03:31.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:31 smithi142 conmon[20125]: cluster 2021-03-24T22:03:31.030317+0000 mon.a (mon.0 2021-03-24T22:03:31.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:31 smithi142 conmon[20125]: ) 1712 : cluster [INF] Client client.admin marked osd.1 out, while it was still marked up 2021-03-24T22:03:31.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:31 smithi142 conmon[20125]: audit 2021-03-24T22:03:31.352429+0000 mon.a (mon 2021-03-24T22:03:31.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:31 smithi142 conmon[20125]: .0) 1713 : audit [INF] from='client.? v1:172.21.15.125:0/431425811' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2021-03-24T22:03:31.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:31 smithi142 conmon[20125]: cluster 2021-03-24T22: 2021-03-24T22:03:31.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:31 smithi142 conmon[20125]: 03:31.352571+0000 mon.a (mon.0) 1714 : cluster [DBG] osdmap e641: 8 total, 8 up, 7 in 2021-03-24T22:03:31.646 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:31.706 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:31.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:31.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:03:31.792 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:31 smithi125 conmon[24927]: audit 2021-03-24T22:03:31.792 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:31 smithi125 conmon[24927]: 2021-03-24T22 2021-03-24T22:03:31.793 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:31 smithi125 conmon[24927]: :03:31.030125+0000 mon.a (mon.0) 1711 : audit [INF] from='client.? v1:172.21.15.125:0/431425811' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2021-03-24T22:03:31.793 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:31 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:03:31.793 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:31 smithi125 conmon[24927]: 03-24T22:03:31.030317+0000 mon.a ( 2021-03-24T22:03:31.793 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:31 smithi125 conmon[24927]: mon.0) 1712 : cluster [INF] Client client.admin marked osd.1 out, while it was still marked up 2021-03-24T22:03:31.794 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:31 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T22:03:31.794 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:31 smithi125 conmon[24927]: 24T22:03:31.352429+0000 2021-03-24T22:03:31.794 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:31 smithi125 conmon[24927]: mon.a (mon.0) 1713 : audit [INF] from='client.? v1:172.21.15.125:0/431425811' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2021-03-24T22:03:31.794 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:31 smithi125 conmon[24927]: cluster 2021-03-24T22:03:31.794 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:31 smithi125 conmon[24927]: 2021-03-24T22:03:31. 2021-03-24T22:03:31.795 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:31 smithi125 conmon[24927]: 352571+0000 mon.a (mon.0) 1714 : cluster [DBG] osdmap e641: 8 total, 8 up, 7 in 2021-03-24T22:03:31.795 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:31 smithi125 conmon[24927]: 2021-03-24T22:03:31.795 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:31 smithi125 conmon[19936]: audit 2021-03-24T22:03:31.030125+0000 mon.a (mon.0) 1711 : audit [INF] from='client.? v1:172.21.15.125:0/431425811' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2021-03-24T22:03:31.796 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:31 smithi125 conmon[19936]: cluster 2021-03-24T22:03:31.030317+0000 mon.a (mon.0) 1712 : cluster [INF] Client client.admin marked osd.1 out, while it was still marked up 2021-03-24T22:03:31.796 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:31 smithi125 conmon[19936]: audit 2021-03-24T 2021-03-24T22:03:31.796 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:31 smithi125 conmon[19936]: 22:03:31.352429+0000 mon.a (mon.0) 1713 : audit [INF] from='client.? v1:172.21.15.125:0/431425811' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2021-03-24T22:03:31.796 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:31 smithi125 conmon[19936]: cluster 2021-03-24T22 2021-03-24T22:03:31.797 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:31 smithi125 conmon[19936]: :03:31.352571+0000 mon.a (mon.0) 1714 2021-03-24T22:03:31.797 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:31 smithi125 conmon[19936]: : cluster [DBG] osdmap e641: 8 total, 8 up, 7 in 2021-03-24T22:03:31.814 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:31.872 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:31.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:31.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:31.977 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:32.036 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:32.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:32.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:03:32.141 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:32.204 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:32.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:32.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:03:32.311 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:32.363 INFO:tasks.rados.rados.0.smithi125.stdout:2422: done (1 left) 2021-03-24T22:03:32.363 INFO:tasks.rados.rados.0.smithi125.stdout:2424: done (0 left) 2021-03-24T22:03:32.364 INFO:tasks.rados.rados.0.smithi125.stdout:2425: write (excl) oid 6 current snap is 273 2021-03-24T22:03:32.364 INFO:tasks.rados.rados.0.smithi125.stdout:2425: seq_num 833 ranges {400231=564894,1661971=338260} 2021-03-24T22:03:32.369 INFO:tasks.rados.rados.0.smithi125.stdout:2425: writing smithi12556470-6 from 400231 to 965125 tid 1 2021-03-24T22:03:32.376 INFO:tasks.rados.rados.0.smithi125.stdout:2425: writing smithi12556470-6 from 1661971 to 2000231 tid 2 2021-03-24T22:03:32.376 INFO:tasks.rados.rados.0.smithi125.stdout:2426: write oid 16 current snap is 273 2021-03-24T22:03:32.376 INFO:tasks.rados.rados.0.smithi125.stdout:2426: seq_num 834 ranges {417038=640143,1720453=296585} 2021-03-24T22:03:32.383 INFO:tasks.rados.rados.0.smithi125.stdout:2426: writing smithi12556470-16 from 417038 to 1057181 tid 1 2021-03-24T22:03:32.384 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:32.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:32.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:03:32.390 INFO:tasks.rados.rados.0.smithi125.stdout:2426: writing smithi12556470-16 from 1720453 to 2017038 tid 2 2021-03-24T22:03:32.390 INFO:tasks.rados.rados.0.smithi125.stdout:2427: delete oid 29 current snap is 273 2021-03-24T22:03:32.403 INFO:tasks.rados.rados.0.smithi125.stdout:2425: finishing write tid 1 to smithi12556470-6 2021-03-24T22:03:32.408 INFO:tasks.rados.rados.0.smithi125.stdout:2427: done (2 left) 2021-03-24T22:03:32.408 INFO:tasks.rados.rados.0.smithi125.stdout:2428: rollback oid 17 current snap is 273 2021-03-24T22:03:32.408 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 17 to 253 2021-03-24T22:03:32.408 INFO:tasks.rados.rados.0.smithi125.stdout:2429: read oid 21 snap -1 2021-03-24T22:03:32.409 INFO:tasks.rados.rados.0.smithi125.stdout:2429: expect deleted 2021-03-24T22:03:32.409 INFO:tasks.rados.rados.0.smithi125.stdout:2430: read oid 28 snap -1 2021-03-24T22:03:32.409 INFO:tasks.rados.rados.0.smithi125.stdout:2430: expect (ObjNum 718 snap 229 seq_num 718) 2021-03-24T22:03:32.409 INFO:tasks.rados.rados.0.smithi125.stdout:2431: rollback oid 7 current snap is 273 2021-03-24T22:03:32.409 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 7 to 264 2021-03-24T22:03:32.410 INFO:tasks.rados.rados.0.smithi125.stdout:2432: snap_create 2021-03-24T22:03:32.411 INFO:tasks.rados.rados.0.smithi125.stdout:2426: finishing write tid 1 to smithi12556470-16 2021-03-24T22:03:32.411 INFO:tasks.rados.rados.0.smithi125.stdout:2426: finishing write tid 2 to smithi12556470-16 2021-03-24T22:03:32.411 INFO:tasks.rados.rados.0.smithi125.stdout:2426: finishing write tid 3 to smithi12556470-16 2021-03-24T22:03:32.411 INFO:tasks.rados.rados.0.smithi125.stdout:2425: finishing write tid 2 to smithi12556470-6 2021-03-24T22:03:32.412 INFO:tasks.rados.rados.0.smithi125.stdout:2426: finishing write tid 4 to smithi12556470-16 2021-03-24T22:03:32.412 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 16 v 474 (ObjNum 834 snap 273 seq_num 834) dirty exists 2021-03-24T22:03:32.412 INFO:tasks.rados.rados.0.smithi125.stdout:2426: left oid 16 (ObjNum 834 snap 273 seq_num 834) 2021-03-24T22:03:32.412 INFO:tasks.rados.rados.0.smithi125.stdout:2425: finishing write tid 3 to smithi12556470-6 2021-03-24T22:03:32.413 INFO:tasks.rados.rados.0.smithi125.stdout:2425: finishing write tid 4 to smithi12556470-6 2021-03-24T22:03:32.413 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 6 v 563 (ObjNum 833 snap 273 seq_num 833) dirty exists 2021-03-24T22:03:32.413 INFO:tasks.rados.rados.0.smithi125.stdout:2425: left oid 6 (ObjNum 833 snap 273 seq_num 833) 2021-03-24T22:03:32.415 INFO:tasks.rados.rados.0.smithi125.stdout:2431: finishing rollback tid 0 to smithi12556470-7 2021-03-24T22:03:32.416 INFO:tasks.rados.rados.0.smithi125.stdout:2431: finishing rollback tid 1 to smithi12556470-7 2021-03-24T22:03:32.416 INFO:tasks.rados.rados.0.smithi125.stdout:2428: finishing rollback tid 0 to smithi12556470-17 2021-03-24T22:03:32.416 INFO:tasks.rados.rados.0.smithi125.stdout:2428: finishing rollback tid 1 to smithi12556470-17 2021-03-24T22:03:32.416 INFO:tasks.rados.rados.0.smithi125.stdout:2428: finishing rollback tid 2 to smithi12556470-17 2021-03-24T22:03:32.417 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 17 v 786 (ObjNum 744 snap 240 seq_num 744) dirty exists 2021-03-24T22:03:32.419 INFO:tasks.rados.rados.0.smithi125.stdout:2431: finishing rollback tid 2 to smithi12556470-7 2021-03-24T22:03:32.419 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 7 v 887 (ObjNum 727 snap 233 seq_num 727) dirty exists 2021-03-24T22:03:32.490 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:32.547 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:32.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:32.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:03:32.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:32 smithi142 conmon[20125]: cluster 2021-03-24T22:03:30 2021-03-24T22:03:32.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:32 smithi142 conmon[20125]: .930311+0000 mgr.y (mgr.14144) 594 : cluster [DBG] pgmap v1123: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 470 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.1 MiB/s wr, 10 op/s; 40/1638 objects misplaced (2.442%) 2021-03-24T22:03:32.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:32 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:03:32.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:32 smithi142 conmon[20125]: T22:03:32.357358+0000 mon.a (mon.0) 2021-03-24T22:03:32.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:32 smithi142 conmon[20125]: 1715 : cluster [DBG] osdmap e642: 8 total, 8 up, 7 in 2021-03-24T22:03:32.652 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:32.705 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:32.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:32.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:32.811 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:32.868 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:32.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:32.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:03:32.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:32 smithi125 conmon[19936]: cluster 2021-03-24T22:03:30. 2021-03-24T22:03:32.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:32 smithi125 conmon[19936]: 930311+0000 mgr.y (mgr.14144) 594 : cluster [DBG] pgmap v1123: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 470 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.1 MiB/s wr, 10 op/s; 40/1638 objects misplaced (2.442%) 2021-03-24T22:03:32.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:32 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:03:32.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:32 smithi125 conmon[19936]: 24T22:03:32.357358+0000 mon.a (mon.0) 1715 : cluster 2021-03-24T22:03:32.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:32 smithi125 conmon[19936]: [DBG] osdmap e642: 8 total, 8 up, 7 in 2021-03-24T22:03:32.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:32 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T22:03:32.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:32 smithi125 conmon[24927]: 22:03:30.930311+0000 mgr.y (mgr.14144) 594 : cluster [DBG] 2021-03-24T22:03:32.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:32 smithi125 conmon[24927]: pgmap v1123: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 470 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.1 MiB/s wr, 10 op/s; 40/1638 objects misplaced (2.442%) 2021-03-24T22:03:32.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:32 smithi125 conmon[24927]: cluster 2021-03-24T22:03:32.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:32 smithi125 conmon[24927]: 2021-03-24T22:03 2021-03-24T22:03:32.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:32 smithi125 conmon[24927]: :32.357358+0000 mon.a ( 2021-03-24T22:03:32.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:32 smithi125 conmon[24927]: mon.0) 1715 : cluster 2021-03-24T22:03:32.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:32 smithi125 conmon[24927]: [DBG] osdmap e642: 8 total, 8 up, 7 in 2021-03-24T22:03:32.975 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:33.022 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:33.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:33.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:33.127 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:33.175 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:33.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:33.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:03:33.280 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:33.332 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:33.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:33.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:03:33.363 INFO:tasks.rados.rados.0.smithi125.stdout:2425: done (6 left) 2021-03-24T22:03:33.363 INFO:tasks.rados.rados.0.smithi125.stdout:2426: done (5 left) 2021-03-24T22:03:33.363 INFO:tasks.rados.rados.0.smithi125.stdout:2428: done (4 left) 2021-03-24T22:03:33.364 INFO:tasks.rados.rados.0.smithi125.stdout:2431: done (3 left) 2021-03-24T22:03:33.364 INFO:tasks.rados.rados.0.smithi125.stdout:2432: done (2 left) 2021-03-24T22:03:33.364 INFO:tasks.rados.rados.0.smithi125.stdout:2433: delete oid 5 current snap is 274 2021-03-24T22:03:33.394 INFO:tasks.rados.rados.0.smithi125.stdout:2430: expect (ObjNum 718 snap 229 seq_num 718) 2021-03-24T22:03:33.437 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:33.458 INFO:tasks.rados.rados.0.smithi125.stdout:2429: done (2 left) 2021-03-24T22:03:33.458 INFO:tasks.rados.rados.0.smithi125.stdout:2430: done (1 left) 2021-03-24T22:03:33.458 INFO:tasks.rados.rados.0.smithi125.stdout:2433: done (0 left) 2021-03-24T22:03:33.459 INFO:tasks.rados.rados.0.smithi125.stdout:2434: copy_from oid 49 from oid 11 current snap is 274 2021-03-24T22:03:33.459 INFO:tasks.rados.rados.0.smithi125.stdout:2435: write (excl) oid 40 current snap is 274 2021-03-24T22:03:33.459 INFO:tasks.rados.rados.0.smithi125.stdout:2435: seq_num 837 ranges {467459=582243,1495899=404945} 2021-03-24T22:03:33.466 INFO:tasks.rados.rados.0.smithi125.stdout:2435: writing smithi12556470-40 from 467459 to 1049702 tid 1 2021-03-24T22:03:33.472 INFO:tasks.rados.rados.0.smithi125.stdout:2435: writing smithi12556470-40 from 1495899 to 1900844 tid 2 2021-03-24T22:03:33.472 INFO:tasks.rados.rados.0.smithi125.stdout:2436: write oid 27 current snap is 274 2021-03-24T22:03:33.472 INFO:tasks.rados.rados.0.smithi125.stdout:2436: seq_num 838 ranges {484266=657492,1554381=529885} 2021-03-24T22:03:33.479 INFO:tasks.rados.rados.0.smithi125.stdout:2436: writing smithi12556470-27 from 484266 to 1141758 tid 1 2021-03-24T22:03:33.481 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:33.484 INFO:tasks.rados.rados.0.smithi125.stdout:2436: writing smithi12556470-27 from 1554381 to 2084266 tid 2 2021-03-24T22:03:33.484 INFO:tasks.rados.rados.0.smithi125.stdout:2437: delete oid 46 current snap is 274 2021-03-24T22:03:33.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:33.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:33.495 INFO:tasks.rados.rados.0.smithi125.stdout:2437: done (3 left) 2021-03-24T22:03:33.495 INFO:tasks.rados.rados.0.smithi125.stdout:2438: rollback oid 33 current snap is 274 2021-03-24T22:03:33.495 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 33 to 253 2021-03-24T22:03:33.495 INFO:tasks.rados.rados.0.smithi125.stdout:2439: snap_create 2021-03-24T22:03:33.498 INFO:tasks.rados.rados.0.smithi125.stdout:2436: finishing write tid 1 to smithi12556470-27 2021-03-24T22:03:33.499 INFO:tasks.rados.rados.0.smithi125.stdout:2436: finishing write tid 2 to smithi12556470-27 2021-03-24T22:03:33.500 INFO:tasks.rados.rados.0.smithi125.stdout:2436: finishing write tid 3 to smithi12556470-27 2021-03-24T22:03:33.500 INFO:tasks.rados.rados.0.smithi125.stdout:2436: finishing write tid 4 to smithi12556470-27 2021-03-24T22:03:33.500 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 27 v 633 (ObjNum 838 snap 274 seq_num 838) dirty exists 2021-03-24T22:03:33.500 INFO:tasks.rados.rados.0.smithi125.stdout:2436: left oid 27 (ObjNum 838 snap 274 seq_num 838) 2021-03-24T22:03:33.501 INFO:tasks.rados.rados.0.smithi125.stdout:2438: finishing rollback tid 0 to smithi12556470-33 2021-03-24T22:03:33.502 INFO:tasks.rados.rados.0.smithi125.stdout:2438: finishing rollback tid 1 to smithi12556470-33 2021-03-24T22:03:33.502 INFO:tasks.rados.rados.0.smithi125.stdout:2438: finishing rollback tid 2 to smithi12556470-33 2021-03-24T22:03:33.502 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 33 v 966 (ObjNum 764 snap 249 seq_num 764) dirty exists 2021-03-24T22:03:33.507 INFO:tasks.rados.rados.0.smithi125.stdout:2435: finishing write tid 1 to smithi12556470-40 2021-03-24T22:03:33.510 INFO:tasks.rados.rados.0.smithi125.stdout:2434: finishing copy_from to smithi12556470-49 2021-03-24T22:03:33.511 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 49 v 402 (ObjNum 802 snap 262 seq_num 802) dirty exists 2021-03-24T22:03:33.511 INFO:tasks.rados.rados.0.smithi125.stdout:2434: finishing copy_from racing read to smithi12556470-49 2021-03-24T22:03:33.512 INFO:tasks.rados.rados.0.smithi125.stdout:2435: finishing write tid 2 to smithi12556470-40 2021-03-24T22:03:33.512 INFO:tasks.rados.rados.0.smithi125.stdout:2435: finishing write tid 3 to smithi12556470-40 2021-03-24T22:03:33.513 INFO:tasks.rados.rados.0.smithi125.stdout:2435: finishing write tid 4 to smithi12556470-40 2021-03-24T22:03:33.513 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 40 v 1407 (ObjNum 837 snap 274 seq_num 837) dirty exists 2021-03-24T22:03:33.513 INFO:tasks.rados.rados.0.smithi125.stdout:2435: left oid 40 (ObjNum 837 snap 274 seq_num 837) 2021-03-24T22:03:33.587 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:33.634 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:33.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:33.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:03:33.738 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:33.786 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:33.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:33.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:03:33.892 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:33.943 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:33.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:33.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:03:34.048 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:34.130 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:34.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:34.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:34.240 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:34.266 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T22:03:34.287 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T22:03:34.290 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:34.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:34.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:03:34.305 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T22:03:34.333 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T22:03:34.360 INFO:tasks.rados.rados.0.smithi125.stdout:2434: done (4 left) 2021-03-24T22:03:34.361 INFO:tasks.rados.rados.0.smithi125.stdout:2435: done (3 left) 2021-03-24T22:03:34.361 INFO:tasks.rados.rados.0.smithi125.stdout:2436: done (2 left) 2021-03-24T22:03:34.361 INFO:tasks.rados.rados.0.smithi125.stdout:2438: done (1 left) 2021-03-24T22:03:34.361 INFO:tasks.rados.rados.0.smithi125.stdout:2439: done (0 left) 2021-03-24T22:03:34.362 INFO:tasks.rados.rados.0.smithi125.stdout:2440: snap_remove snap 253 2021-03-24T22:03:34.375 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T22:03:34.377 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: cluster 2021-03-24T22:03:34.377 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: 2021-03-24T22:03:34.377 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: 2021 2021-03-24T22:03:34.378 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: - 2021-03-24T22:03:34.378 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: 03 2021-03-24T22:03:34.378 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: - 2021-03-24T22:03:34.378 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: 24 2021-03-24T22:03:34.379 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: cluster 2021-03-24T22:03:34.379 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 2021-03-24T22:03:34.379 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 2021 2021-03-24T22:03:34.380 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: - 2021-03-24T22:03:34.380 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 03 2021-03-24T22:03:34.380 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: - 2021-03-24T22:03:34.380 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 24 2021-03-24T22:03:34.380 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: T 2021-03-24T22:03:34.381 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 22 2021-03-24T22:03:34.381 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: : 2021-03-24T22:03:34.381 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 03 2021-03-24T22:03:34.381 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: : 2021-03-24T22:03:34.382 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 32 2021-03-24T22:03:34.382 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: . 2021-03-24T22:03:34.382 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 931086 2021-03-24T22:03:34.382 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: +0000 2021-03-24T22:03:34.382 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 2021-03-24T22:03:34.383 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: mgr.y 2021-03-24T22:03:34.395 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:34.403 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T22:03:34.437 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T22:03:34.444 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:34.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:34.452 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: T 2021-03-24T22:03:34.452 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: 22 2021-03-24T22:03:34.453 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: : 2021-03-24T22:03:34.453 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: 03 2021-03-24T22:03:34.453 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: : 2021-03-24T22:03:34.453 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: 32 2021-03-24T22:03:34.454 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: . 2021-03-24T22:03:34.454 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: 931086 2021-03-24T22:03:34.454 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: +0000 2021-03-24T22:03:34.454 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: 2021-03-24T22:03:34.455 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: mgr.y 2021-03-24T22:03:34.455 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: ( 2021-03-24T22:03:34.455 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: mgr 2021-03-24T22:03:34.455 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: . 2021-03-24T22:03:34.455 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: 14144 2021-03-24T22:03:34.456 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: ) 2021-03-24T22:03:34.456 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: 595 2021-03-24T22:03:34.456 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: : 2021-03-24T22:03:34.456 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: cluster 2021-03-24T22:03:34.457 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: 2021-03-24T22:03:34.457 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: [DBG] 2021-03-24T22:03:34.457 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: 2021-03-24T22:03:34.457 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: pgmap v1126: 168 pgs: 8 peering, 1 active+remapped+backfill_toofull, 159 active+clean; 469 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 4.8 MiB/s rd, 2.9 MiB/s wr, 15 op/s; 40/1638 objects misplaced (2.442%); 108 B/s, 0 objects/s recovering 2021-03-24T22:03:34.457 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: 2021-03-24T22:03:34.458 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: cluster 2021-03-24T22:03:34.458 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: 2021-03-24T22:03:34.458 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: 2021 2021-03-24T22:03:34.458 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: - 2021-03-24T22:03:34.459 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: 03 2021-03-24T22:03:34.459 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: - 2021-03-24T22:03:34.459 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: 24 2021-03-24T22:03:34.460 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: T 2021-03-24T22:03:34.460 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: 22 2021-03-24T22:03:34.460 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: : 2021-03-24T22:03:34.460 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: 03 2021-03-24T22:03:34.461 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: : 2021-03-24T22:03:34.461 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: 33 2021-03-24T22:03:34.461 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: . 2021-03-24T22:03:34.462 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: 356983 2021-03-24T22:03:34.462 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: +0000 2021-03-24T22:03:34.462 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: 2021-03-24T22:03:34.462 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: osd.2 2021-03-24T22:03:34.463 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: (osd.2) 100 : cluster [DBG] 2021-03-24T22:03:34.463 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: 4.d starting backfill to osd.6 from (0'0,0'0] MAX to 544'41 2021-03-24T22:03:34.463 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: cluster 2021-03-24T22:03:33.359023+0000 mon.a (mon.0) 1716 : 2021-03-24T22:03:34.463 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: cluster [DBG] osdmap e643: 8 total, 8 up, 7 in 2021-03-24T22:03:34.463 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:03:34.464 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: 03-24T22:03:33.361631+0000 osd.0 (osd. 2021-03-24T22:03:34.464 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: 0) 126 : cluster [DBG] 4.b starting backfill to osd.3 from (0'0,0'0] MAX to 544'17 2021-03-24T22:03:34.464 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:03:34.464 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: 03-24T22:03:33.361975+0000 2021-03-24T22:03:34.465 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: osd.0 (osd.0) 127 : cluster [DBG] 2021-03-24T22:03:34.465 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: 4.6 starting backfill to osd.7 from (0'0,0'0] MAX to 544'34 2021-03-24T22:03:34.465 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: cluster 2021-03-24T22:03 2021-03-24T22:03:34.465 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: :33.362503+0000 osd.3 (osd.3) 2021-03-24T22:03:34.465 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: 130 : cluster [DBG] 4.f starting backfill to osd.7 from (0'0,0'0] MAX to 544'46 2021-03-24T22:03:34.466 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:03:34.466 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: 24T22:03:33.362626+0000 osd.3 2021-03-24T22:03:34.466 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: (osd.3) 131 : cluster [DBG] 4.9 starting backfill to osd.6 from (0'0,0'0] MAX to 544'38 2021-03-24T22:03:34.466 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: 2021-03-24T22:03:34.466 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: cluster 2021-03-24T22:03:34.467 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: 2021-03-24T22:03:33.369967+0000 2021-03-24T22:03:34.467 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: osd.5 (osd.5) 35 : cluster [DBG] 2021-03-24T22:03:34.467 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: 4.7 starting backfill to osd.7 from (0'0,0'0] MAX to 544'32 2021-03-24T22:03:34.467 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: cluster 2021-03-24T22:03: 2021-03-24T22:03:34.468 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: 33.372063+0000 osd.6 (osd.6) 50 : 2021-03-24T22:03:34.468 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: cluster [DBG] 4.a starting backfill to osd.5 from (0'0,0'0] MAX to 544'45 2021-03-24T22:03:34.468 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T22:03:34.468 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: T22:03:33.373943+0000 osd.6 2021-03-24T22:03:34.468 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: (osd.6) 51 : cluster [DBG] 4.e starting backfill to osd.3 from (0'0,0'0] MAX to 544'45 2021-03-24T22:03:34.469 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: 2021-03-24T22:03:34.469 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: cluster 2021-03-24T22:03:34.469 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: 2021-03-24T22:03:33.530784+0000 2021-03-24T22:03:34.469 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: mon.a (mon.0) 1717 : cluster [WRN] 2021-03-24T22:03:34.470 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: Health check failed: Reduced data availability: 3 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2021-03-24T22:03:34.470 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: audit 2021-03-24T22:03:34.230869+0000 mon.a (mon 2021-03-24T22:03:34.470 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[24927]: .0) 1718 : audit [DBG] from='client.? v1:172.21.15.125:0/542844208' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:03:34.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:34.472 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: ( 2021-03-24T22:03:34.473 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: mgr 2021-03-24T22:03:34.473 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: . 2021-03-24T22:03:34.473 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 14144 2021-03-24T22:03:34.473 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: ) 2021-03-24T22:03:34.473 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 595 2021-03-24T22:03:34.474 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: : 2021-03-24T22:03:34.474 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: cluster 2021-03-24T22:03:34.474 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 2021-03-24T22:03:34.474 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: [DBG] 2021-03-24T22:03:34.475 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 2021-03-24T22:03:34.475 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: pgmap v1126: 168 pgs: 8 peering, 1 active+remapped+backfill_toofull, 159 active+clean; 469 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 4.8 MiB/s rd, 2.9 MiB/s wr, 15 op/s; 40/1638 objects misplaced (2.442%); 108 B/s, 0 objects/s recovering 2021-03-24T22:03:34.476 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 2021-03-24T22:03:34.476 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: cluster 2021-03-24T22:03:34.476 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 2021-03-24T22:03:34.476 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 2021 2021-03-24T22:03:34.476 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: - 2021-03-24T22:03:34.477 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 03 2021-03-24T22:03:34.477 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: - 2021-03-24T22:03:34.477 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 24 2021-03-24T22:03:34.477 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: T 2021-03-24T22:03:34.478 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 22 2021-03-24T22:03:34.478 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: : 2021-03-24T22:03:34.478 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 03 2021-03-24T22:03:34.478 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: : 2021-03-24T22:03:34.478 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 33 2021-03-24T22:03:34.479 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: . 2021-03-24T22:03:34.479 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 356983 2021-03-24T22:03:34.479 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: +0000 2021-03-24T22:03:34.479 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 2021-03-24T22:03:34.480 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: osd.2 2021-03-24T22:03:34.480 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: ( 2021-03-24T22:03:34.480 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: osd 2021-03-24T22:03:34.481 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: . 2021-03-24T22:03:34.481 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 2 2021-03-24T22:03:34.481 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: ) 2021-03-24T22:03:34.481 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 100 2021-03-24T22:03:34.482 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: : 2021-03-24T22:03:34.482 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: cluster 2021-03-24T22:03:34.482 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 2021-03-24T22:03:34.482 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: [DBG] 2021-03-24T22:03:34.483 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 2021-03-24T22:03:34.483 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 4.d starting backfill to osd.6 from (0'0,0'0] MAX to 544'41 2021-03-24T22:03:34.483 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 2021-03-24T22:03:34.483 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: cluster 2021-03-24T22:03:34.483 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 2021-03-24T22:03:34.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 2021 2021-03-24T22:03:34.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: - 2021-03-24T22:03:34.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 03 2021-03-24T22:03:34.484 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: - 2021-03-24T22:03:34.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 24 2021-03-24T22:03:34.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: T 2021-03-24T22:03:34.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 22 2021-03-24T22:03:34.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: : 2021-03-24T22:03:34.485 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 03 2021-03-24T22:03:34.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: : 2021-03-24T22:03:34.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 33 2021-03-24T22:03:34.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: . 2021-03-24T22:03:34.486 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 359023 2021-03-24T22:03:34.487 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: +0000 2021-03-24T22:03:34.487 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 2021-03-24T22:03:34.487 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: mon.a 2021-03-24T22:03:34.487 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: ( 2021-03-24T22:03:34.487 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: mon 2021-03-24T22:03:34.488 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: . 2021-03-24T22:03:34.488 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 0 2021-03-24T22:03:34.488 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: ) 2021-03-24T22:03:34.488 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 1716 2021-03-24T22:03:34.488 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: : 2021-03-24T22:03:34.489 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: cluster 2021-03-24T22:03:34.489 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 2021-03-24T22:03:34.489 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: [DBG] 2021-03-24T22:03:34.489 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 2021-03-24T22:03:34.490 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: osdmap e643: 8 total, 8 up, 7 in 2021-03-24T22:03:34.490 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 2021-03-24T22:03:34.490 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: cluster 2021-03-24T22:03:34.491 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 2021-03-24T22:03:34.491 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 2021 2021-03-24T22:03:34.491 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: - 2021-03-24T22:03:34.491 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 03 2021-03-24T22:03:34.492 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: - 2021-03-24T22:03:34.492 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 24 2021-03-24T22:03:34.492 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: T 2021-03-24T22:03:34.492 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 22 2021-03-24T22:03:34.492 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: : 2021-03-24T22:03:34.493 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 03 2021-03-24T22:03:34.493 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: : 2021-03-24T22:03:34.493 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 33 2021-03-24T22:03:34.493 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: . 2021-03-24T22:03:34.494 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 361631 2021-03-24T22:03:34.494 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: +0000 2021-03-24T22:03:34.494 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 2021-03-24T22:03:34.494 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: osd.0 2021-03-24T22:03:34.494 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: ( 2021-03-24T22:03:34.495 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: osd 2021-03-24T22:03:34.495 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: . 2021-03-24T22:03:34.495 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 0 2021-03-24T22:03:34.495 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: ) 2021-03-24T22:03:34.496 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 126 2021-03-24T22:03:34.496 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: : 2021-03-24T22:03:34.496 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: cluster 2021-03-24T22:03:34.496 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 2021-03-24T22:03:34.496 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: [DBG] 2021-03-24T22:03:34.497 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 2021-03-24T22:03:34.497 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 4.b starting backfill to osd.3 from (0'0,0'0] MAX to 544'17 2021-03-24T22:03:34.497 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 2021-03-24T22:03:34.497 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: cluster 2021-03-24T22:03:34.498 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 2021-03-24T22:03:34.498 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 2021 2021-03-24T22:03:34.498 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: - 2021-03-24T22:03:34.498 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 03 2021-03-24T22:03:34.498 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: - 2021-03-24T22:03:34.499 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 24 2021-03-24T22:03:34.499 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: T 2021-03-24T22:03:34.499 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 22 2021-03-24T22:03:34.499 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: : 2021-03-24T22:03:34.500 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 03 2021-03-24T22:03:34.500 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: : 2021-03-24T22:03:34.500 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 33 2021-03-24T22:03:34.500 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: . 2021-03-24T22:03:34.500 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 361975 2021-03-24T22:03:34.501 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: +0000 2021-03-24T22:03:34.501 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 2021-03-24T22:03:34.501 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: osd.0 2021-03-24T22:03:34.501 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: ( 2021-03-24T22:03:34.501 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: osd 2021-03-24T22:03:34.502 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: . 2021-03-24T22:03:34.502 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 0 2021-03-24T22:03:34.502 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: ) 2021-03-24T22:03:34.502 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 127 2021-03-24T22:03:34.503 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: : 2021-03-24T22:03:34.503 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: cluster 2021-03-24T22:03:34.503 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 2021-03-24T22:03:34.503 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: [DBG] 2021-03-24T22:03:34.503 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 2021-03-24T22:03:34.504 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 4.6 starting backfill to osd.7 from (0'0,0'0] MAX to 544'34 2021-03-24T22:03:34.504 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 2021-03-24T22:03:34.504 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: cluster 2021-03-24T22:03:34.504 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 2021-03-24T22:03:34.505 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 2021 2021-03-24T22:03:34.505 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: - 2021-03-24T22:03:34.505 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 03 2021-03-24T22:03:34.505 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: - 2021-03-24T22:03:34.505 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 24 2021-03-24T22:03:34.506 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: T 2021-03-24T22:03:34.506 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 22 2021-03-24T22:03:34.506 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: : 2021-03-24T22:03:34.506 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 03 2021-03-24T22:03:34.506 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: : 2021-03-24T22:03:34.507 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 33 2021-03-24T22:03:34.507 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: . 2021-03-24T22:03:34.507 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 362503 2021-03-24T22:03:34.507 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: +0000 2021-03-24T22:03:34.508 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 2021-03-24T22:03:34.508 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: osd.3 2021-03-24T22:03:34.508 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: ( 2021-03-24T22:03:34.508 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: osd 2021-03-24T22:03:34.508 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: . 2021-03-24T22:03:34.509 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 3 2021-03-24T22:03:34.509 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: ) 2021-03-24T22:03:34.509 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 130 2021-03-24T22:03:34.509 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: : 2021-03-24T22:03:34.509 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: cluster 2021-03-24T22:03:34.510 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 2021-03-24T22:03:34.510 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: [DBG] 2021-03-24T22:03:34.510 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 2021-03-24T22:03:34.510 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 4.f starting backfill to osd.7 from (0'0,0'0] MAX to 544'46 2021-03-24T22:03:34.511 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 2021-03-24T22:03:34.511 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: cluster 2021-03-24T22:03:34.511 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 2021-03-24T22:03:34.511 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 2021 2021-03-24T22:03:34.511 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: - 2021-03-24T22:03:34.512 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 03 2021-03-24T22:03:34.512 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: - 2021-03-24T22:03:34.512 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 24 2021-03-24T22:03:34.512 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: T 2021-03-24T22:03:34.512 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 22 2021-03-24T22:03:34.513 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: : 2021-03-24T22:03:34.513 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 03 2021-03-24T22:03:34.513 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: : 2021-03-24T22:03:34.513 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 33 2021-03-24T22:03:34.514 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: . 2021-03-24T22:03:34.514 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 362626+0000 osd.3 (osd 2021-03-24T22:03:34.514 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: .3) 131 : cluster [DBG] 4.9 starting backfill to osd.6 from (0'0,0'0] MAX to 544'38 2021-03-24T22:03:34.514 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:03:34.514 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: T22:03:33.369967+0000 osd.5 (osd. 2021-03-24T22:03:34.515 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 5) 35 : cluster [DBG] 4.7 starting backfill to osd.7 from (0'0,0'0] MAX to 544'32 2021-03-24T22:03:34.515 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T22:03:34.515 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: -24T22:03:33.372063+0000 osd.6 ( 2021-03-24T22:03:34.515 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: osd.6) 50 : cluster [DBG] 4.a starting backfill to osd.5 from (0'0,0'0] MAX to 544'45 2021-03-24T22:03:34.515 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T22:03:34.516 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: -24T22:03:33.373943+0000 osd.6 ( 2021-03-24T22:03:34.516 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: osd.6) 51 : cluster 2021-03-24T22:03:34.516 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: [DBG] 4.e starting backfill to osd.3 from (0'0,0'0] MAX to 544'45 2021-03-24T22:03:34.516 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:03:34.517 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: -03-24T22:03:33. 2021-03-24T22:03:34.517 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 530784+0000 mon.a (mon.0) 1717 : cluster 2021-03-24T22:03:34.517 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: [WRN] Health check failed: Reduced data availability: 3 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2021-03-24T22:03:34.517 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: audit 2021-03-24T22:03:34.517 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: 2021-03-24T22:03 2021-03-24T22:03:34.518 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: :34.230869+0000 mon.a (mon.0) 1718 2021-03-24T22:03:34.518 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:34 smithi125 conmon[19936]: : audit [DBG] from='client.? v1:172.21.15.125:0/542844208' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:03:34.518 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T22:03:34.519 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-24T22:03:34.519 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-24T22:03:34.519 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-24T22:03:34.520 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-24T22:03:34.520 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-24T22:03:34.520 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-24T22:03:34.520 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-24T22:03:34.520 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-24T22:03:34.573 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:34.581 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: cluster 2021-03-24T22:03:34.582 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: 2021-03-24T22:03:32.931086+0000 mgr.y (mgr.14144) 595 : cluster [DBG] pgmap v1126: 168 pgs: 8 peering, 1 active+remapped+backfill_toofull, 159 active+clean; 469 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 4.8 MiB/s rd, 2.9 MiB/s wr, 15 op/s; 40/1638 objects misplaced (2.442%); 108 B/s, 0 objects/s recovering 2021-03-24T22:03:34.582 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:03:34.582 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: -03-24T22:03: 2021-03-24T22:03:34.582 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: 33.356983+0000 osd.2 ( 2021-03-24T22:03:34.583 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: osd.2) 100 : cluster 2021-03-24T22:03:34.583 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: [DBG] 4.d starting backfill to osd.6 from (0'0,0'0] MAX to 544'41 2021-03-24T22:03:34.583 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: cluster 2021-03-24T22:03:34.583 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: 2021-03-24T22: 2021-03-24T22:03:34.583 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: 03:33.359023+0000 2021-03-24T22:03:34.584 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: mon.a (mon.0) 1716 : 2021-03-24T22:03:34.584 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: cluster [DBG] osdmap e643: 8 total, 8 up, 7 in 2021-03-24T22:03:34.584 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:03:34.584 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: -24T22:03:33.361631+0000 osd.0 2021-03-24T22:03:34.585 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: (osd.0) 126 : cluster 2021-03-24T22:03:34.585 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: [DBG] 4.b starting backfill to osd.3 from (0'0,0'0] MAX to 544'17 2021-03-24T22:03:34.585 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: cluster 2021-03-24T22:03:34.585 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: 2021-03-24T22: 2021-03-24T22:03:34.585 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: 03:33.361975+0000 osd.0 2021-03-24T22:03:34.586 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: (osd.0) 127 : cluster 2021-03-24T22:03:34.586 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: [DBG] 4.6 starting backfill to osd.7 from (0'0,0'0] MAX to 544'34 2021-03-24T22:03:34.586 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:03:34.586 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: -24T22:03:33.362503+0000 2021-03-24T22:03:34.587 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: osd.3 (osd.3) 130 : 2021-03-24T22:03:34.587 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: cluster [DBG] 4.f starting backfill to osd.7 from (0'0,0'0] MAX to 544'46 2021-03-24T22:03:34.587 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:03:34.587 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: -24T22:03:33.362626+0000 osd.3 2021-03-24T22:03:34.588 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: (osd.3) 131 : cluster 2021-03-24T22:03:34.588 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: [DBG] 4.9 starting backfill to osd.6 from (0'0,0'0] MAX to 544'38 2021-03-24T22:03:34.588 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:03:34.588 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: -03-24T22:03 2021-03-24T22:03:34.588 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: :33.369967+0000 osd.5 ( 2021-03-24T22:03:34.589 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: osd.5) 35 : cluster 2021-03-24T22:03:34.589 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: [DBG] 4.7 starting backfill to osd.7 from (0'0,0'0] MAX to 544'32 2021-03-24T22:03:34.589 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:03:34.589 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: -03-24T22:03:33. 2021-03-24T22:03:34.589 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: 372063+0000 osd.6 (osd.6 2021-03-24T22:03:34.590 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: ) 50 : cluster [DBG] 4.a starting backfill to osd.5 from (0'0,0'0] MAX to 544'45 2021-03-24T22:03:34.590 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: 2021-03-24T22:03:34.590 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:03:34.590 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: 24T22:03:33. 2021-03-24T22:03:34.590 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: 373943+0000 osd.6 (osd.6 2021-03-24T22:03:34.591 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: ) 51 : cluster 2021-03-24T22:03:34.591 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: [DBG] 4.e starting backfill to osd.3 from (0'0,0'0] MAX to 544'45 2021-03-24T22:03:34.591 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:03:34.591 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: -03-24T22:03 2021-03-24T22:03:34.592 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: :33.530784+0000 mon.a ( 2021-03-24T22:03:34.592 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: mon.0) 1717 : cluster 2021-03-24T22:03:34.592 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: [WRN] Health check failed: Reduced data availability: 3 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2021-03-24T22:03:34.592 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: audit 2021-03-24T22:03:34.592 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: 2021-03-24T22: 2021-03-24T22:03:34.593 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: 03:34.230869+0000 mon.a 2021-03-24T22:03:34.593 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: (mon.0) 1718 : audit 2021-03-24T22:03:34.593 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:34 smithi142 conmon[20125]: [DBG] from='client.? v1:172.21.15.125:0/542844208' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:03:34.621 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:34.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:34.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:03:34.726 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:34.781 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:34.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:34.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:03:34.885 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:34.922 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:34.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:34.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:03:35.027 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:35.077 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:35.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:35.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:03:35.182 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:35.229 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:35.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:35.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:03:35.334 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:35.349 INFO:tasks.rados.rados.0.smithi125.stdout:2440: done (0 left) 2021-03-24T22:03:35.349 INFO:tasks.rados.rados.0.smithi125.stdout:2441: rollback oid 40 current snap is 275 2021-03-24T22:03:35.349 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 40 to 272 2021-03-24T22:03:35.349 INFO:tasks.rados.rados.0.smithi125.stdout:2442: snap_create 2021-03-24T22:03:35.382 INFO:tasks.rados.rados.0.smithi125.stdout:2441: finishing rollback tid 0 to smithi12556470-40 2021-03-24T22:03:35.383 INFO:tasks.rados.rados.0.smithi125.stdout:2441: finishing rollback tid 1 to smithi12556470-40 2021-03-24T22:03:35.383 INFO:tasks.rados.rados.0.smithi125.stdout:2441: finishing rollback tid 2 to smithi12556470-40 2021-03-24T22:03:35.383 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 40 v 1411 (ObjNum 832 snap 272 seq_num 832) dirty exists 2021-03-24T22:03:35.395 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:35.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:35.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:03:35.455 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[19936]: cluster 2021-03-24T22:03:35.456 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[19936]: 2021-03-24T22:03:35.456 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[19936]: 2021 2021-03-24T22:03:35.457 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[19936]: - 2021-03-24T22:03:35.457 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[19936]: 03 2021-03-24T22:03:35.457 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[19936]: - 2021-03-24T22:03:35.458 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[19936]: 24 2021-03-24T22:03:35.458 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[19936]: T 2021-03-24T22:03:35.459 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[19936]: 22 2021-03-24T22:03:35.459 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[19936]: : 2021-03-24T22:03:35.460 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[19936]: 03 2021-03-24T22:03:35.460 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[19936]: : 2021-03-24T22:03:35.462 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[19936]: 34 2021-03-24T22:03:35.463 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[19936]: .347681+0000 osd.1 (osd.1) 62 : cluster 2021-03-24T22:03:35.464 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[19936]: [DBG] purged_snaps scrub starts 2021-03-24T22:03:35.464 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[19936]: cluster 2021-03-24T22:03:34. 2021-03-24T22:03:35.465 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[19936]: 348716+0000 osd.1 (osd.1) 63 : cluster [DBG] 2021-03-24T22:03:35.465 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[19936]: purged_snaps scrub ok 2021-03-24T22:03:35.465 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[19936]: cluster 2021-03-24T22:03:35.466 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[19936]: 2021-03-24T22:03:34.357456+0000 mon.a (mon 2021-03-24T22:03:35.466 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[19936]: .0) 1719 : cluster [DBG] osdmap e644: 8 total, 8 up, 7 in 2021-03-24T22:03:35.467 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:03:35.467 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[19936]: T22:03:35.341935+0000 mon.a (mon. 2021-03-24T22:03:35.468 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[19936]: 0) 1720 : cluster [DBG] osdmap e645: 8 total, 8 up, 7 in 2021-03-24T22:03:35.468 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: cluster 2021-03-24T22:03:35.469 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: 2021-03-24T22:03:35.469 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: 2021 2021-03-24T22:03:35.470 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: - 2021-03-24T22:03:35.470 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: 03 2021-03-24T22:03:35.470 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: - 2021-03-24T22:03:35.471 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: 24 2021-03-24T22:03:35.471 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: T 2021-03-24T22:03:35.472 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: 22 2021-03-24T22:03:35.472 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: : 2021-03-24T22:03:35.473 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: 03 2021-03-24T22:03:35.473 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: : 2021-03-24T22:03:35.474 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: 34 2021-03-24T22:03:35.474 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: . 2021-03-24T22:03:35.475 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: 347681 2021-03-24T22:03:35.475 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: +0000 2021-03-24T22:03:35.475 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: 2021-03-24T22:03:35.476 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: osd.1 2021-03-24T22:03:35.476 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: ( 2021-03-24T22:03:35.477 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: osd 2021-03-24T22:03:35.477 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: . 2021-03-24T22:03:35.478 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: 1 2021-03-24T22:03:35.478 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: ) 2021-03-24T22:03:35.479 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: 62 2021-03-24T22:03:35.479 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: : 2021-03-24T22:03:35.480 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: cluster 2021-03-24T22:03:35.480 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: 2021-03-24T22:03:35.480 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: [DBG] 2021-03-24T22:03:35.481 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: 2021-03-24T22:03:35.481 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: purged_snaps scrub starts 2021-03-24T22:03:35.482 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: 2021-03-24T22:03:35.482 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: cluster 2021-03-24T22:03:35.482 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: 2021-03-24T22:03:35.483 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: 2021 2021-03-24T22:03:35.483 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: - 2021-03-24T22:03:35.484 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: 03-24T22:03:34 2021-03-24T22:03:35.485 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: .348716+0000 osd.1 (osd.1) 63 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:03:35.485 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: cluster 2021-03-24T22:03:35.485 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: 2021-03-24T22:03:35.485 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: 2021 2021-03-24T22:03:35.486 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: - 2021-03-24T22:03:35.486 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: 03 2021-03-24T22:03:35.486 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: - 2021-03-24T22:03:35.486 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: 24 2021-03-24T22:03:35.486 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: T 2021-03-24T22:03:35.487 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: 22 2021-03-24T22:03:35.487 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: : 2021-03-24T22:03:35.487 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: 03 2021-03-24T22:03:35.487 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: : 2021-03-24T22:03:35.487 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: 34 2021-03-24T22:03:35.488 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: . 2021-03-24T22:03:35.488 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: 357456 2021-03-24T22:03:35.488 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: +0000 2021-03-24T22:03:35.488 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: 2021-03-24T22:03:35.489 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: mon.a 2021-03-24T22:03:35.489 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: ( 2021-03-24T22:03:35.489 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: mon 2021-03-24T22:03:35.489 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: . 2021-03-24T22:03:35.489 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: 0 2021-03-24T22:03:35.490 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: ) 2021-03-24T22:03:35.490 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: 1719 2021-03-24T22:03:35.490 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: : 2021-03-24T22:03:35.490 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: cluster 2021-03-24T22:03:35.490 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: 2021-03-24T22:03:35.491 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: [DBG] osdmap e644: 8 total, 8 up, 7 in 2021-03-24T22:03:35.491 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: cluster 2021-03-24T22:03:35 2021-03-24T22:03:35.491 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:35 smithi125 conmon[24927]: .341935+0000 mon.a (mon.0) 1720 : cluster [DBG] osdmap e645: 8 total, 8 up, 7 in 2021-03-24T22:03:35.504 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:35.555 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:35.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:35.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:03:35.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: cluster 2021-03-24T22:03:35.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 2021-03-24T22:03:35.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 2021 2021-03-24T22:03:35.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: - 2021-03-24T22:03:35.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 03 2021-03-24T22:03:35.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: - 2021-03-24T22:03:35.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 24 2021-03-24T22:03:35.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: T 2021-03-24T22:03:35.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 22 2021-03-24T22:03:35.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: : 2021-03-24T22:03:35.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 03 2021-03-24T22:03:35.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: : 2021-03-24T22:03:35.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 34 2021-03-24T22:03:35.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: . 2021-03-24T22:03:35.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 347681 2021-03-24T22:03:35.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: +0000 2021-03-24T22:03:35.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 2021-03-24T22:03:35.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: osd.1 2021-03-24T22:03:35.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: ( 2021-03-24T22:03:35.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: osd 2021-03-24T22:03:35.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: . 2021-03-24T22:03:35.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 1 2021-03-24T22:03:35.625 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: ) 2021-03-24T22:03:35.625 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 62 2021-03-24T22:03:35.625 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: : 2021-03-24T22:03:35.625 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: cluster 2021-03-24T22:03:35.626 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 2021-03-24T22:03:35.626 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: [DBG] 2021-03-24T22:03:35.626 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 2021-03-24T22:03:35.626 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: purged_snaps scrub starts 2021-03-24T22:03:35.626 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 2021-03-24T22:03:35.627 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: cluster 2021-03-24T22:03:35.627 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 2021-03-24T22:03:35.627 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 2021 2021-03-24T22:03:35.627 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: - 2021-03-24T22:03:35.628 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 03 2021-03-24T22:03:35.628 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: - 2021-03-24T22:03:35.628 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 24 2021-03-24T22:03:35.628 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: T 2021-03-24T22:03:35.630 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 22 2021-03-24T22:03:35.630 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: : 2021-03-24T22:03:35.631 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 03 2021-03-24T22:03:35.631 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: : 2021-03-24T22:03:35.631 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 34 2021-03-24T22:03:35.632 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: . 2021-03-24T22:03:35.632 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 348716 2021-03-24T22:03:35.632 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: +0000 2021-03-24T22:03:35.632 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 2021-03-24T22:03:35.633 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: osd.1 2021-03-24T22:03:35.633 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: ( 2021-03-24T22:03:35.633 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: osd 2021-03-24T22:03:35.633 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: . 2021-03-24T22:03:35.634 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 1 2021-03-24T22:03:35.634 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: ) 2021-03-24T22:03:35.634 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 63 2021-03-24T22:03:35.634 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: : 2021-03-24T22:03:35.634 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: cluster 2021-03-24T22:03:35.635 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 2021-03-24T22:03:35.635 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: [DBG] 2021-03-24T22:03:35.635 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 2021-03-24T22:03:35.635 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: purged_snaps scrub ok 2021-03-24T22:03:35.636 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 2021-03-24T22:03:35.636 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: cluster 2021-03-24T22:03:35.636 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 2021-03-24T22:03:35.636 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 2021 2021-03-24T22:03:35.637 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: - 2021-03-24T22:03:35.637 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 03 2021-03-24T22:03:35.637 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: - 2021-03-24T22:03:35.637 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 24 2021-03-24T22:03:35.638 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: T 2021-03-24T22:03:35.638 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 22 2021-03-24T22:03:35.638 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: : 2021-03-24T22:03:35.638 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 03 2021-03-24T22:03:35.639 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: : 2021-03-24T22:03:35.639 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 34 2021-03-24T22:03:35.639 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: . 2021-03-24T22:03:35.639 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 357456 2021-03-24T22:03:35.639 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: +0000 2021-03-24T22:03:35.640 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 2021-03-24T22:03:35.640 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: mon.a 2021-03-24T22:03:35.640 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: ( 2021-03-24T22:03:35.640 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: mon 2021-03-24T22:03:35.641 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: . 2021-03-24T22:03:35.641 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 0 2021-03-24T22:03:35.641 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: ) 2021-03-24T22:03:35.641 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 1719 2021-03-24T22:03:35.641 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: : 2021-03-24T22:03:35.642 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: cluster 2021-03-24T22:03:35.642 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 2021-03-24T22:03:35.642 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: [DBG] 2021-03-24T22:03:35.642 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 2021-03-24T22:03:35.643 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: osdmap e644: 8 total, 8 up, 7 in 2021-03-24T22:03:35.643 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 2021-03-24T22:03:35.643 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: cluster 2021-03-24T22:03:35.643 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 2021-03-24T22:03:35.643 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 2021 2021-03-24T22:03:35.644 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: - 2021-03-24T22:03:35.644 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 03 2021-03-24T22:03:35.644 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: - 2021-03-24T22:03:35.644 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 24 2021-03-24T22:03:35.645 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: T 2021-03-24T22:03:35.645 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 22 2021-03-24T22:03:35.645 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: : 2021-03-24T22:03:35.645 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 03 2021-03-24T22:03:35.645 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: : 2021-03-24T22:03:35.646 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 35 2021-03-24T22:03:35.646 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: . 2021-03-24T22:03:35.646 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 341935 2021-03-24T22:03:35.646 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: +0000 2021-03-24T22:03:35.646 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 2021-03-24T22:03:35.647 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: mon.a 2021-03-24T22:03:35.647 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: ( 2021-03-24T22:03:35.647 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: mon 2021-03-24T22:03:35.647 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: . 2021-03-24T22:03:35.648 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 0 2021-03-24T22:03:35.648 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: ) 2021-03-24T22:03:35.648 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 1720 2021-03-24T22:03:35.648 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: : 2021-03-24T22:03:35.648 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: cluster 2021-03-24T22:03:35.649 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 2021-03-24T22:03:35.649 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: [DBG] 2021-03-24T22:03:35.649 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 2021-03-24T22:03:35.649 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: osdmap e645: 8 total, 8 up, 7 in 2021-03-24T22:03:35.649 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:35 smithi142 conmon[20125]: 2021-03-24T22:03:35.660 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:35.705 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:35.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:35.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:03:35.809 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:35.862 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:35.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:35.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:03:35.969 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:36.011 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:36.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:36.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:03:36.117 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:36.167 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:36.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:36.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:03:36.272 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:36.323 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:36.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:36.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:03:36.350 INFO:tasks.rados.rados.0.smithi125.stdout:2441: done (1 left) 2021-03-24T22:03:36.351 INFO:tasks.rados.rados.0.smithi125.stdout:2442: done (0 left) 2021-03-24T22:03:36.351 INFO:tasks.rados.rados.0.smithi125.stdout:2443: write (excl) oid 5 current snap is 276 2021-03-24T22:03:36.351 INFO:tasks.rados.rados.0.smithi125.stdout:2443: seq_num 840 ranges {517880=407990,1554992=562888} 2021-03-24T22:03:36.352 INFO:tasks.rados.rados.0.smithi125.stdout:2443: writing smithi12556470-5 from 517880 to 925870 tid 1 2021-03-24T22:03:36.359 INFO:tasks.rados.rados.0.smithi125.stdout:2443: writing smithi12556470-5 from 1554992 to 2117880 tid 2 2021-03-24T22:03:36.359 INFO:tasks.rados.rados.0.smithi125.stdout:2443: finishing write tid 1 to smithi12556470-5 2021-03-24T22:03:36.359 INFO:tasks.rados.rados.0.smithi125.stdout:2444: snap_create 2021-03-24T22:03:36.362 INFO:tasks.rados.rados.0.smithi125.stdout:2443: finishing write tid 2 to smithi12556470-5 2021-03-24T22:03:36.362 INFO:tasks.rados.rados.0.smithi125.stdout:2443: finishing write tid 3 to smithi12556470-5 2021-03-24T22:03:36.362 INFO:tasks.rados.rados.0.smithi125.stdout:2443: finishing write tid 4 to smithi12556470-5 2021-03-24T22:03:36.363 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 5 v 790 (ObjNum 840 snap 276 seq_num 840) dirty exists 2021-03-24T22:03:36.364 INFO:tasks.rados.rados.0.smithi125.stdout:2443: left oid 5 (ObjNum 840 snap 276 seq_num 840) 2021-03-24T22:03:36.390 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:36 smithi125 conmon[19936]: cluster 2021-03-24T22:03:34.932067+0000 mgr.y (mgr. 2021-03-24T22:03:36.391 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:36 smithi125 conmon[19936]: 14144) 596 : cluster [DBG] pgmap v1129: 168 pgs: 1 active+undersized+remapped, 4 active+recovering+undersized+remapped, 1 active+clean+remapped, 2 active+remapped+backfilling, 9 peering, 2 active+remapped+backfill_toofull, 149 active+clean; 479 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 4.1 MiB/s rd, 2.8 MiB/s wr, 14 op/s; 98/1653 objects misplaced (5.929%); 2.5 MiB/s, 27 objects/s recovering 2021-03-24T22:03:36.391 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:36 smithi125 conmon[19936]: cluster 2021-03-24T22:03:35.360125+0000 mon.a (mon.0) 1721 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:03:36.391 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:36 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T22:03:36.391 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:36 smithi125 conmon[19936]: -24T22:03:35.878488+0000 osd.4 (osd.4) 115 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:03:36.392 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:36 smithi125 conmon[19936]: cluster 2021-03-24T22:03:36.392 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:36 smithi125 conmon[19936]: 2021-03-24T22:03:35.879834+0000 2021-03-24T22:03:36.392 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:36 smithi125 conmon[19936]: osd.4 (osd.4) 116 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:03:36.392 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:36 smithi125 conmon[19936]: cluster 2021-03-24T22:03:36.392 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:36 smithi125 conmon[19936]: 2021-03-24T22:03: 2021-03-24T22:03:36.393 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:36 smithi125 conmon[19936]: 36.338031+0000 mon.a (mon.0) 1722 : 2021-03-24T22:03:36.393 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:36 smithi125 conmon[19936]: cluster [DBG] osdmap e646: 8 total, 8 up, 7 in 2021-03-24T22:03:36.393 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:36 smithi125 conmon[24927]: cluster 2021-03-24T22:03:34 2021-03-24T22:03:36.394 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:36 smithi125 conmon[24927]: .932067+0000 mgr.y (mgr.14144) 596 : cluster [DBG] pgmap v1129: 168 pgs: 1 active+undersized+remapped, 4 active+recovering+undersized+remapped, 1 active+clean+remapped, 2 active+remapped+backfilling, 9 peering, 2 active+remapped+backfill_toofull, 149 active+clean; 479 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 4.1 MiB/s rd, 2.8 MiB/s wr, 14 op/s; 98/1653 objects misplaced (5.929%); 2.5 MiB/s, 27 objects/s recovering 2021-03-24T22:03:36.394 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:36 smithi125 conmon[24927]: cluster 2021-03-24T22 2021-03-24T22:03:36.394 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:36 smithi125 conmon[24927]: :03:35.360125+0000 mon.a (mon.0) 1721 : 2021-03-24T22:03:36.394 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:36 smithi125 conmon[24927]: cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:03:36.395 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:36 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:03:36.395 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:36 smithi125 conmon[24927]: 03:35.878488+0000 osd.4 (osd 2021-03-24T22:03:36.395 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:36 smithi125 conmon[24927]: .4) 115 : cluster 2021-03-24T22:03:36.395 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:36 smithi125 conmon[24927]: [DBG] purged_snaps scrub starts 2021-03-24T22:03:36.396 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:36 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:03:36.396 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:36 smithi125 conmon[24927]: 03-24T22:03:35.879834 2021-03-24T22:03:36.396 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:36 smithi125 conmon[24927]: +0000 osd.4 (osd.4) 116 : cluster 2021-03-24T22:03:36.396 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:36 smithi125 conmon[24927]: [DBG] purged_snaps scrub ok 2021-03-24T22:03:36.397 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:36 smithi125 conmon[24927]: cluster 2021-03-24T22:03:36 2021-03-24T22:03:36.397 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:36 smithi125 conmon[24927]: .338031+0000 mon.a (mon.0) 1722 : 2021-03-24T22:03:36.397 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:36 smithi125 conmon[24927]: cluster [DBG] osdmap e646: 8 total, 8 up, 7 in 2021-03-24T22:03:36.428 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:36.481 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:36.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:36.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:03:36.593 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:36.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:36 smithi142 conmon[20125]: cluster 2021-03-24T22:03:34.932067+0000 mgr.y (mgr.14144 2021-03-24T22:03:36.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:36 smithi142 conmon[20125]: ) 596 : cluster [DBG] pgmap v1129: 168 pgs: 1 active+undersized+remapped, 4 active+recovering+undersized+remapped, 1 active+clean+remapped, 2 active+remapped+backfilling, 9 peering, 2 active+remapped+backfill_toofull, 149 active+clean; 479 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 4.1 MiB/s rd, 2.8 MiB/s wr, 14 op/s; 98/1653 objects misplaced (5.929%); 2.5 MiB/s, 27 objects/s recovering 2021-03-24T22:03:36.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:36 smithi142 conmon[20125]: cluster 2021-03-24T22:03:35.360125+0000 mon.a (mon.0) 1721 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:03:36.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:36 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:03:36.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:36 smithi142 conmon[20125]: 03-24T22:03:35.878488+0000 osd.4 (osd.4) 115 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:03:36.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:36 smithi142 conmon[20125]: cluster 2021-03-24T22:03 2021-03-24T22:03:36.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:36 smithi142 conmon[20125]: :35.879834+0000 osd.4 (osd.4) 116 : cluster [DBG] 2021-03-24T22:03:36.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:36 smithi142 conmon[20125]: purged_snaps scrub ok 2021-03-24T22:03:36.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:36 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:03:36.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:36 smithi142 conmon[20125]: -03-24T22:03:36.338031+0000 mon.a (mon.0) 1722 2021-03-24T22:03:36.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:36 smithi142 conmon[20125]: : cluster [DBG] osdmap e646: 8 total, 8 up, 7 in 2021-03-24T22:03:36.646 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:36.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:36.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:03:36.655 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-24T22:03:36.656 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T22:03:36.752 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:36.797 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:36.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:36.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:03:36.903 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:36.953 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:36.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:36.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:03:37.059 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:37.113 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:37.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:37.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:03:37.219 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:37.275 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:37.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:37.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:37.350 INFO:tasks.rados.rados.0.smithi125.stdout:2443: done (1 left) 2021-03-24T22:03:37.351 INFO:tasks.rados.rados.0.smithi125.stdout:2444: done (0 left) 2021-03-24T22:03:37.351 INFO:tasks.rados.rados.0.smithi125.stdout:2445: delete oid 5 current snap is 277 2021-03-24T22:03:37.358 INFO:tasks.rados.rados.0.smithi125.stdout:2445: done (0 left) 2021-03-24T22:03:37.359 INFO:tasks.rados.rados.0.smithi125.stdout:2446: write oid 18 current snap is 277 2021-03-24T22:03:37.359 INFO:tasks.rados.rados.0.smithi125.stdout:2446: seq_num 842 ranges {551494=558488,1671956=479538} 2021-03-24T22:03:37.359 INFO:teuthology.orchestra.run.smithi125.stderr:noscrub is unset 2021-03-24T22:03:37.365 INFO:tasks.rados.rados.0.smithi125.stdout:2446: writing smithi12556470-18 from 551494 to 1109982 tid 1 2021-03-24T22:03:37.371 INFO:tasks.rados.rados.0.smithi125.stdout:2446: writing smithi12556470-18 from 1671956 to 2151494 tid 2 2021-03-24T22:03:37.371 INFO:tasks.rados.rados.0.smithi125.stdout:2447: write oid 14 current snap is 277 2021-03-24T22:03:37.371 INFO:tasks.rados.rados.0.smithi125.stdout:2447: seq_num 843 ranges {568301=633737,1614085=415952} 2021-03-24T22:03:37.378 INFO:tasks.rados.rados.0.smithi125.stdout:2447: writing smithi12556470-14 from 568301 to 1202038 tid 1 2021-03-24T22:03:37.382 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:37.383 INFO:tasks.rados.rados.0.smithi125.stdout:2447: writing smithi12556470-14 from 1614085 to 2030037 tid 2 2021-03-24T22:03:37.384 INFO:tasks.rados.rados.0.smithi125.stdout:2448: delete oid 23 current snap is 277 2021-03-24T22:03:37.384 INFO:tasks.rados.rados.0.smithi125.stdout:2446: finishing write tid 1 to smithi12556470-18 2021-03-24T22:03:37.384 INFO:tasks.rados.rados.0.smithi125.stdout:2446: finishing write tid 2 to smithi12556470-18 2021-03-24T22:03:37.384 INFO:tasks.rados.rados.0.smithi125.stdout:2446: finishing write tid 3 to smithi12556470-18 2021-03-24T22:03:37.385 INFO:tasks.rados.rados.0.smithi125.stdout:2446: finishing write tid 4 to smithi12556470-18 2021-03-24T22:03:37.385 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 18 v 699 (ObjNum 842 snap 277 seq_num 842) dirty exists 2021-03-24T22:03:37.385 INFO:tasks.rados.rados.0.smithi125.stdout:2446: left oid 18 (ObjNum 842 snap 277 seq_num 842) 2021-03-24T22:03:37.385 INFO:tasks.rados.rados.0.smithi125.stdout:2447: finishing write tid 1 to smithi12556470-14 2021-03-24T22:03:37.387 INFO:tasks.rados.rados.0.smithi125.stdout:2446: done (2 left) 2021-03-24T22:03:37.387 INFO:tasks.rados.rados.0.smithi125.stdout:2448: done (1 left) 2021-03-24T22:03:37.387 INFO:tasks.rados.rados.0.smithi125.stdout:2449: write oid 45 current snap is 277 2021-03-24T22:03:37.388 INFO:tasks.rados.rados.0.smithi125.stdout:2449: seq_num 845 ranges {601915=500588,1847402=354513} 2021-03-24T22:03:37.394 INFO:tasks.rados.rados.0.smithi125.stdout:2449: writing smithi12556470-45 from 601915 to 1102503 tid 1 2021-03-24T22:03:37.400 INFO:tasks.rados.rados.0.smithi125.stdout:2449: writing smithi12556470-45 from 1847402 to 2201915 tid 2 2021-03-24T22:03:37.401 INFO:tasks.rados.rados.0.smithi125.stdout:2450: snap_remove snap 259 2021-03-24T22:03:37.427 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:37.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:37.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:03:37.532 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:37.589 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:37.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:37.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:03:37.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:37 smithi125 conmon[24927]: audit 2021-03-24T22:03:37. 2021-03-24T22:03:37.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:37 smithi125 conmon[24927]: 060749+0000 mon.a (mon.0) 1723 : audit [INF] from='client.? v1:172.21.15.125:0/996550213' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T22:03:37.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:37 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:03:37.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:37 smithi125 conmon[24927]: 03-24T22:03:37.333921+0000 mon.a (mon.0) 1724 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:03:37.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:37 smithi125 conmon[24927]: audit 2021-03-24T22 2021-03-24T22:03:37.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:37 smithi125 conmon[24927]: :03:37.345060+0000 mon.a (mon.0) 1725 : audit 2021-03-24T22:03:37.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:37 smithi125 conmon[24927]: [INF] from='client.? v1:172.21.15.125:0/996550213' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T22:03:37.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:37 smithi125 conmon[24927]: cluster 2021-03-24T22:03:37.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:37 smithi125 conmon[24927]: 2021-03-24T22:03:37.345127+0000 mon.a (mon.0) 1726 : 2021-03-24T22:03:37.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:37 smithi125 conmon[24927]: cluster [DBG] osdmap e647: 8 total, 8 up, 7 in 2021-03-24T22:03:37.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:37 smithi125 conmon[19936]: audit 2021-03-24T22:03:37.060749+0000 2021-03-24T22:03:37.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:37 smithi125 conmon[19936]: mon.a (mon.0) 1723 : audit [INF] from='client.? v1:172.21.15.125:0/996550213' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T22:03:37.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:37 smithi125 conmon[19936]: cluster 2021-03-24T22:03:37.333921+0000 mon.a (mon.0) 1724 : 2021-03-24T22:03:37.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:37 smithi125 conmon[19936]: cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:03:37.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:37 smithi125 conmon[19936]: audit 2021-03-24T22:03:37.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:37 smithi125 conmon[19936]: 2021-03-24 2021-03-24T22:03:37.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:37 smithi125 conmon[19936]: T22:03:37.345060+0000 2021-03-24T22:03:37.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:37 smithi125 conmon[19936]: mon.a (mon.0) 1725 : audit 2021-03-24T22:03:37.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:37 smithi125 conmon[19936]: [INF] from='client.? v1:172.21.15.125:0/996550213' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T22:03:37.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:37 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T22:03:37.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:37 smithi125 conmon[19936]: -24T22:03:37.345127+0000 mon.a 2021-03-24T22:03:37.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:37 smithi125 conmon[19936]: (mon.0) 1726 : cluster [DBG] 2021-03-24T22:03:37.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:37 smithi125 conmon[19936]: osdmap e647: 8 total, 8 up, 7 in 2021-03-24T22:03:37.693 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:37.701 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:37 smithi142 conmon[20125]: audit 2021-03-24T22:03:37.060749+0000 2021-03-24T22:03:37.701 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:37 smithi142 conmon[20125]: mon.a (mon.0) 1723 : audit [INF] from='client.? v1:172.21.15.125:0/996550213' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T22:03:37.701 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:37 smithi142 conmon[20125]: cluster 2021-03-24T22:03:37.333921+0000 mon.a (mon.0) 1724 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:03:37.701 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:37 smithi142 conmon[20125]: audit 2021-03-24T22:03:37.345060+0000 mon.a ( 2021-03-24T22:03:37.702 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:37 smithi142 conmon[20125]: mon.0) 1725 : audit [INF] from='client.? v1:172.21.15.125:0/996550213' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T22:03:37.702 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:37 smithi142 conmon[20125]: cluster 2021-03-24T22: 2021-03-24T22:03:37.702 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:37 smithi142 conmon[20125]: 03:37.345127+0000 mon.a (mon.0) 1726 : cluster 2021-03-24T22:03:37.702 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:37 smithi142 conmon[20125]: [DBG] osdmap e647: 8 total, 8 up, 7 in 2021-03-24T22:03:37.738 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:37.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:37.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:03:37.845 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:37.899 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:37.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:37.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:03:38.005 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:38.060 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:38.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:38.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:03:38.164 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:38.218 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:38.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:38.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:03:38.324 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:38.359 INFO:tasks.rados.rados.0.smithi125.stdout:2450: done (2 left) 2021-03-24T22:03:38.359 INFO:tasks.rados.rados.0.smithi125.stdout:2451: delete oid 46 current snap is 277 2021-03-24T22:03:38.359 INFO:tasks.rados.rados.0.smithi125.stdout:2447: finishing write tid 2 to smithi12556470-14 2021-03-24T22:03:38.360 INFO:tasks.rados.rados.0.smithi125.stdout:2447: finishing write tid 3 to smithi12556470-14 2021-03-24T22:03:38.360 INFO:tasks.rados.rados.0.smithi125.stdout:2447: finishing write tid 4 to smithi12556470-14 2021-03-24T22:03:38.360 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 14 v 1423 (ObjNum 843 snap 277 seq_num 843) dirty exists 2021-03-24T22:03:38.360 INFO:tasks.rados.rados.0.smithi125.stdout:2447: left oid 14 (ObjNum 843 snap 277 seq_num 843) 2021-03-24T22:03:38.360 INFO:tasks.rados.rados.0.smithi125.stdout:2449: finishing write tid 1 to smithi12556470-45 2021-03-24T22:03:38.361 INFO:tasks.rados.rados.0.smithi125.stdout:2449: finishing write tid 2 to smithi12556470-45 2021-03-24T22:03:38.361 INFO:tasks.rados.rados.0.smithi125.stdout:2449: finishing write tid 3 to smithi12556470-45 2021-03-24T22:03:38.361 INFO:tasks.rados.rados.0.smithi125.stdout:2449: finishing write tid 4 to smithi12556470-45 2021-03-24T22:03:38.361 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 45 v 668 (ObjNum 845 snap 277 seq_num 845) dirty exists 2021-03-24T22:03:38.361 INFO:tasks.rados.rados.0.smithi125.stdout:2449: left oid 45 (ObjNum 845 snap 277 seq_num 845) 2021-03-24T22:03:38.374 INFO:tasks.rados.rados.0.smithi125.stdout:2447: done (2 left) 2021-03-24T22:03:38.375 INFO:tasks.rados.rados.0.smithi125.stdout:2449: done (1 left) 2021-03-24T22:03:38.375 INFO:tasks.rados.rados.0.smithi125.stdout:2451: done (0 left) 2021-03-24T22:03:38.375 INFO:tasks.rados.rados.0.smithi125.stdout:2452: write oid 21 current snap is 277 2021-03-24T22:03:38.375 INFO:tasks.rados.rados.0.smithi125.stdout:2452: seq_num 847 ranges {635529=651086,1848013=387516} 2021-03-24T22:03:38.376 INFO:tasks.rados.rados.0.smithi125.stdout:2452: writing smithi12556470-21 from 635529 to 1286615 tid 1 2021-03-24T22:03:38.382 INFO:tasks.rados.rados.0.smithi125.stdout:2452: writing smithi12556470-21 from 1848013 to 2235529 tid 2 2021-03-24T22:03:38.383 INFO:tasks.rados.rados.0.smithi125.stdout:2452: finishing write tid 1 to smithi12556470-21 2021-03-24T22:03:38.383 INFO:tasks.rados.rados.0.smithi125.stdout:2453: write (excl) oid 46 current snap is 277 2021-03-24T22:03:38.383 INFO:tasks.rados.rados.0.smithi125.stdout:2453: seq_num 848 ranges {652336=726335,1906495=345841} 2021-03-24T22:03:38.386 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:38.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:38.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:38.390 INFO:tasks.rados.rados.0.smithi125.stdout:2453: writing smithi12556470-46 from 652336 to 1378671 tid 1 2021-03-24T22:03:38.395 INFO:tasks.rados.rados.0.smithi125.stdout:2453: writing smithi12556470-46 from 1906495 to 2252336 tid 2 2021-03-24T22:03:38.395 INFO:tasks.rados.rados.0.smithi125.stdout:2454: snap_remove snap 272 2021-03-24T22:03:38.491 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:38.508 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-24T22:03:38.546 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:38.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:38.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:03:38.556 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:38 smithi125 conmon[19936]: cluster 2021-03-24T22:03:36.932935 2021-03-24T22:03:38.556 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:38 smithi125 conmon[19936]: +0000 mgr.y (mgr.14144) 597 : cluster [DBG] pgmap v1132: 168 pgs: 2 active+undersized+remapped, 6 active+recovering+undersized+remapped, 1 active+clean+remapped, 3 active+remapped+backfilling, 2 peering, 4 active+remapped+backfill_toofull, 150 active+clean; 481 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 12 op/s; 159/1665 objects misplaced (9.550%); 4.2 MiB/s, 57 objects/s recovering 2021-03-24T22:03:38.556 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:38 smithi125 conmon[19936]: cluster 2021-03-24T22:03:37.364107+0000 2021-03-24T22:03:38.557 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:38 smithi125 conmon[19936]: mon.a (mon.0) 1727 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 5 pgs peering) 2021-03-24T22:03:38.557 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:38 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:03:38.557 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:38 smithi125 conmon[19936]: 24T22:03:38.348471+0000 mon.a (mon.0) 1728 : cluster [DBG] osdmap e648: 8 total, 8 up, 7 in 2021-03-24T22:03:38.557 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:38 smithi125 conmon[19936]: 2021-03-24T22:03:38.651 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:38.665 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:38 smithi142 conmon[20125]: cluster 2021-03-24T22:03:36.932935+0000 2021-03-24T22:03:38.665 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:38 smithi142 conmon[20125]: mgr.y (mgr.14144) 597 : cluster [DBG] pgmap v1132: 168 pgs: 2 active+undersized+remapped, 6 active+recovering+undersized+remapped, 1 active+clean+remapped, 3 active+remapped+backfilling, 2 peering, 4 active+remapped+backfill_toofull, 150 active+clean; 481 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 12 op/s; 159/1665 objects misplaced (9.550%); 4.2 MiB/s, 57 objects/s recovering 2021-03-24T22:03:38.665 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:38 smithi142 conmon[20125]: cluster 2021-03-24T22:03:37.364107+0000 mon.a ( 2021-03-24T22:03:38.666 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:38 smithi142 conmon[20125]: mon.0) 1727 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 5 pgs peering) 2021-03-24T22:03:38.666 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:38 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:03:38.666 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:38 smithi142 conmon[20125]: -03-24T22:03:38. 2021-03-24T22:03:38.667 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:38 smithi142 conmon[20125]: 348471+0000 mon.a (mon.0) 2021-03-24T22:03:38.667 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:38 smithi142 conmon[20125]: 1728 : cluster [DBG] osdmap e648: 8 total, 8 up, 7 in 2021-03-24T22:03:38.704 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:38.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:38.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:03:38.718 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.1 dump_historic_ops 2021-03-24T22:03:38.810 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:38.822 INFO:journalctl@ceph.osd.2.smithi125.stdout:Mar 24 22:03:38 smithi125 conmon[34620]: debug 2021-03-24T22:03:38.628+0000 7f47049d4700 -1 osd.2 pg_epoch: 648 pg[3.1a( empty local-lis/les=641/642 n=0 ec=60/60 lis/c=641/641 les/c/f=642/642/0 sis=641) [2,4,0] r=0 lpr=641 crt=0'0 mlcod 0'0 active+clean] scrubber pg(3.1a) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 4 (0x5604bd1a51e0) 2021-03-24T22:03:38.822 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:38 smithi125 conmon[24927]: cluster 2021-03-24T22:03:36.932935+0000 2021-03-24T22:03:38.822 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:38 smithi125 conmon[24927]: mgr.y (mgr.14144) 597 : cluster [DBG] pgmap v1132: 168 pgs: 2 active+undersized+remapped, 6 active+recovering+undersized+remapped, 1 active+clean+remapped, 3 active+remapped+backfilling, 2 peering, 4 active+remapped+backfill_toofull, 150 active+clean; 481 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 12 op/s; 159/1665 objects misplaced (9.550%); 4.2 MiB/s, 57 objects/s recovering 2021-03-24T22:03:38.823 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:38 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:03:38.823 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:38 smithi125 conmon[24927]: 03-24T22:03:37.364107+0000 mon.a (mon.0) 1727 : cluster 2021-03-24T22:03:38.823 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:38 smithi125 conmon[24927]: [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 5 pgs peering) 2021-03-24T22:03:38.823 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:38 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T22:03:38.824 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:38 smithi125 conmon[24927]: T22:03:38.348471+0000 mon.a (mon 2021-03-24T22:03:38.824 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:38 smithi125 conmon[24927]: .0) 1728 : cluster [DBG] osdmap e648: 8 total, 8 up, 7 in 2021-03-24T22:03:38.828 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2021-03-24T22:03:38.829 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-24T22:03:38.829 INFO:tasks.thrashosds.thrasher:Adding osd 1 2021-03-24T22:03:38.830 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd in 1 2021-03-24T22:03:38.864 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:38.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:38.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:38.970 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:39.019 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:03:39.022 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:39.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:39.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:03:39.080 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:03:39.127 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:39.167 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:03:39.177 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:39.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:39.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:03:39.283 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:39.336 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:39.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:39.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:39.389 INFO:tasks.rados.rados.0.smithi125.stdout:2454: done (2 left) 2021-03-24T22:03:39.389 INFO:tasks.rados.rados.0.smithi125.stdout:2455: rollback oid 29 current snap is 277 2021-03-24T22:03:39.389 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 29 to 243 2021-03-24T22:03:39.390 INFO:tasks.rados.rados.0.smithi125.stdout:2452: finishing write tid 2 to smithi12556470-21 2021-03-24T22:03:39.390 INFO:tasks.rados.rados.0.smithi125.stdout:2452: finishing write tid 3 to smithi12556470-21 2021-03-24T22:03:39.390 INFO:tasks.rados.rados.0.smithi125.stdout:2452: finishing write tid 4 to smithi12556470-21 2021-03-24T22:03:39.390 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 21 v 571 (ObjNum 847 snap 277 seq_num 847) dirty exists 2021-03-24T22:03:39.391 INFO:tasks.rados.rados.0.smithi125.stdout:2452: left oid 21 (ObjNum 847 snap 277 seq_num 847) 2021-03-24T22:03:39.391 INFO:tasks.rados.rados.0.smithi125.stdout:2453: finishing write tid 1 to smithi12556470-46 2021-03-24T22:03:39.391 INFO:tasks.rados.rados.0.smithi125.stdout:2453: finishing write tid 2 to smithi12556470-46 2021-03-24T22:03:39.391 INFO:tasks.rados.rados.0.smithi125.stdout:2453: finishing write tid 3 to smithi12556470-46 2021-03-24T22:03:39.392 INFO:tasks.rados.rados.0.smithi125.stdout:2453: finishing write tid 4 to smithi12556470-46 2021-03-24T22:03:39.392 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 46 v 727 (ObjNum 848 snap 277 seq_num 848) dirty exists 2021-03-24T22:03:39.392 INFO:tasks.rados.rados.0.smithi125.stdout:2453: left oid 46 (ObjNum 848 snap 277 seq_num 848) 2021-03-24T22:03:39.392 INFO:tasks.rados.rados.0.smithi125.stdout:2452: done (2 left) 2021-03-24T22:03:39.392 INFO:tasks.rados.rados.0.smithi125.stdout:2453: done (1 left) 2021-03-24T22:03:39.393 INFO:tasks.rados.rados.0.smithi125.stdout:2456: snap_create 2021-03-24T22:03:39.404 INFO:tasks.rados.rados.0.smithi125.stdout:2455: finishing rollback tid 1 to smithi12556470-29 2021-03-24T22:03:39.405 INFO:tasks.rados.rados.0.smithi125.stdout:2455: finishing rollback tid 2 to smithi12556470-29 2021-03-24T22:03:39.405 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 29 v 411 (ObjNum 691 snap 218 seq_num 691) dirty exists 2021-03-24T22:03:39.442 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:39.494 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:39.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:39.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:03:39.600 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:39.653 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:39.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:39.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:03:39.759 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:39.820 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:39.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:39.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:03:39.926 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:39.977 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:39.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:39.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:03:40.084 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:40.135 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:40.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:40.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:03:40.245 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:40.306 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:40.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:40.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:40.348 INFO:tasks.rados.rados.0.smithi125.stdout:2455: done (1 left) 2021-03-24T22:03:40.349 INFO:tasks.rados.rados.0.smithi125.stdout:2456: done (0 left) 2021-03-24T22:03:40.349 INFO:tasks.rados.rados.0.smithi125.stdout:2457: read oid 41 snap -1 2021-03-24T22:03:40.350 INFO:tasks.rados.rados.0.smithi125.stdout:2457: expect (ObjNum 791 snap 257 seq_num 791) 2021-03-24T22:03:40.350 INFO:tasks.rados.rados.0.smithi125.stdout:2458: delete oid 2 current snap is 278 2021-03-24T22:03:40.364 INFO:tasks.rados.rados.0.smithi125.stdout:2457: expect (ObjNum 791 snap 257 seq_num 791) 2021-03-24T22:03:40.413 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:40.469 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:40.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:40.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:03:40.527 INFO:tasks.rados.rados.0.smithi125.stdout:2457: done (1 left) 2021-03-24T22:03:40.527 INFO:tasks.rados.rados.0.smithi125.stdout:2458: done (0 left) 2021-03-24T22:03:40.527 INFO:tasks.rados.rados.0.smithi125.stdout:2459: rollback oid 9 current snap is 278 2021-03-24T22:03:40.528 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 9 to 258 2021-03-24T22:03:40.528 INFO:tasks.rados.rados.0.smithi125.stdout:2460: read oid 45 snap -1 2021-03-24T22:03:40.528 INFO:tasks.rados.rados.0.smithi125.stdout:2460: expect (ObjNum 845 snap 277 seq_num 845) 2021-03-24T22:03:40.528 INFO:tasks.rados.rados.0.smithi125.stdout:2461: write (excl) oid 47 current snap is 278 2021-03-24T22:03:40.529 INFO:tasks.rados.rados.0.smithi125.stdout:2461: seq_num 850 ranges {685950=476833,1623459=570617} 2021-03-24T22:03:40.544 INFO:tasks.rados.rados.0.smithi125.stdout:2461: writing smithi12556470-47 from 685950 to 1162783 tid 1 2021-03-24T22:03:40.550 INFO:tasks.rados.rados.0.smithi125.stdout:2461: writing smithi12556470-47 from 1623459 to 2194076 tid 2 2021-03-24T22:03:40.550 INFO:tasks.rados.rados.0.smithi125.stdout:2462: rollback oid 41 current snap is 278 2021-03-24T22:03:40.551 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 41 to 192 2021-03-24T22:03:40.551 INFO:tasks.rados.rados.0.smithi125.stdout:2459: finishing rollback tid 0 to smithi12556470-9 2021-03-24T22:03:40.551 INFO:tasks.rados.rados.0.smithi125.stdout:2459: finishing rollback tid 1 to smithi12556470-9 2021-03-24T22:03:40.551 INFO:tasks.rados.rados.0.smithi125.stdout:2459: finishing rollback tid 2 to smithi12556470-9 2021-03-24T22:03:40.552 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 9 v 1437 (ObjNum 746 snap 241 seq_num 746) dirty exists 2021-03-24T22:03:40.552 INFO:tasks.rados.rados.0.smithi125.stdout:2459: done (3 left) 2021-03-24T22:03:40.552 INFO:tasks.rados.rados.0.smithi125.stdout:2463: write oid 24 current snap is 278 2021-03-24T22:03:40.552 INFO:tasks.rados.rados.0.smithi125.stdout:2463: seq_num 851 ranges {702757=552082,1681941=430628} 2021-03-24T22:03:40.557 INFO:tasks.rados.rados.0.smithi125.stdout:2463: writing smithi12556470-24 from 702757 to 1254839 tid 1 2021-03-24T22:03:40.562 INFO:tasks.rados.rados.0.smithi125.stdout:2463: writing smithi12556470-24 from 1681941 to 2112569 tid 2 2021-03-24T22:03:40.563 INFO:tasks.rados.rados.0.smithi125.stdout:2464: copy_from oid 11 from oid 4 current snap is 278 2021-03-24T22:03:40.563 INFO:tasks.rados.rados.0.smithi125.stdout:2461: finishing write tid 1 to smithi12556470-47 2021-03-24T22:03:40.563 INFO:tasks.rados.rados.0.smithi125.stdout:2461: finishing write tid 2 to smithi12556470-47 2021-03-24T22:03:40.563 INFO:tasks.rados.rados.0.smithi125.stdout:2462: finishing rollback tid 0 to smithi12556470-41 2021-03-24T22:03:40.564 INFO:tasks.rados.rados.0.smithi125.stdout:2462: finishing rollback tid 1 to smithi12556470-41 2021-03-24T22:03:40.565 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 41 v 802 (ObjNum 1 snap 32741 seq_num 1118594576) dirty dne 2021-03-24T22:03:40.565 INFO:tasks.rados.rados.0.smithi125.stdout:2462: done (4 left) 2021-03-24T22:03:40.565 INFO:tasks.rados.rados.0.smithi125.stdout:2465: write (excl) oid 50 current snap is 278 2021-03-24T22:03:40.565 INFO:tasks.rados.rados.0.smithi125.stdout:2461: finishing write tid 3 to smithi12556470-47 2021-03-24T22:03:40.566 INFO:tasks.rados.rados.0.smithi125.stdout:2461: finishing write tid 4 to smithi12556470-47 2021-03-24T22:03:40.566 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 47 v 352 (ObjNum 850 snap 278 seq_num 850) dirty exists 2021-03-24T22:03:40.567 INFO:tasks.rados.rados.0.smithi125.stdout:2461: left oid 47 (ObjNum 850 snap 278 seq_num 850) 2021-03-24T22:03:40.567 INFO:tasks.rados.rados.0.smithi125.stdout:2465: seq_num 852 ranges {719564=743684,2140423=179141} 2021-03-24T22:03:40.570 INFO:tasks.rados.rados.0.smithi125.stdout:2465: writing smithi12556470-50 from 719564 to 1463248 tid 1 2021-03-24T22:03:40.573 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:40.577 INFO:tasks.rados.rados.0.smithi125.stdout:2465: writing smithi12556470-50 from 2140423 to 2319564 tid 2 2021-03-24T22:03:40.577 INFO:tasks.rados.rados.0.smithi125.stdout:2461: done (4 left) 2021-03-24T22:03:40.578 INFO:tasks.rados.rados.0.smithi125.stdout:2466: snap_remove snap 258 2021-03-24T22:03:40.590 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:40 smithi125 conmon[19936]: cluster 2021-03-24T22:03:38.933676+0000 mgr.y (mgr.14144) 598 : cluster 2021-03-24T22:03:40.590 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:40 smithi125 conmon[19936]: [DBG] pgmap v1135: 168 pgs: 2 active+undersized+remapped, 6 active+recovering+undersized+remapped, 1 active+clean+remapped, 3 active+remapped+backfilling, 2 peering, 4 active+remapped+backfill_toofull, 150 active+clean; 481 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 229 KiB/s wr, 2 op/s; 159/1665 objects misplaced (9.550%); 1.6 MiB/s, 30 objects/s recovering 2021-03-24T22:03:40.591 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:40 smithi125 conmon[19936]: cluster 2021-03-24T22:03:39.385241+0000 2021-03-24T22:03:40.591 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:40 smithi125 conmon[19936]: mon.a (mon.0) 1729 : cluster [DBG] 2021-03-24T22:03:40.591 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:40 smithi125 conmon[19936]: osdmap e649: 8 total, 8 up, 7 in 2021-03-24T22:03:40.591 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:40 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:03:40.591 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:40 smithi125 conmon[19936]: 03-24T22:03: 2021-03-24T22:03:40.592 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:40 smithi125 conmon[19936]: 40.344080+0000 2021-03-24T22:03:40.593 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:40 smithi125 conmon[19936]: mon.a (mon.0) 2021-03-24T22:03:40.593 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:40 smithi125 conmon[19936]: 1730 : cluster [DBG] osdmap e650: 8 total, 8 up, 7 in 2021-03-24T22:03:40.594 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:40 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:03:40.594 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:40 smithi125 conmon[24927]: 03-24T22:03:38.933676 2021-03-24T22:03:40.595 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:40 smithi125 conmon[24927]: +0000 mgr.y ( 2021-03-24T22:03:40.595 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:40 smithi125 conmon[24927]: mgr.14144) 2021-03-24T22:03:40.595 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:40 smithi125 conmon[24927]: 598 : cluster [DBG] 2021-03-24T22:03:40.596 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:40 smithi125 conmon[24927]: pgmap v1135: 168 pgs: 2 active+undersized+remapped, 6 active+recovering+undersized+remapped, 1 active+clean+remapped, 3 active+remapped+backfilling, 2 peering, 4 active+remapped+backfill_toofull, 150 active+clean; 481 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 229 KiB/s wr, 2 op/s; 159/1665 objects misplaced (9.550%); 1.6 MiB/s, 30 objects/s recovering 2021-03-24T22:03:40.596 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:40 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T22:03:40.596 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:40 smithi125 conmon[24927]: 22:03:39. 2021-03-24T22:03:40.596 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:40 smithi125 conmon[24927]: 385241+0000 mon.a 2021-03-24T22:03:40.596 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:40 smithi125 conmon[24927]: (mon.0 2021-03-24T22:03:40.597 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:40 smithi125 conmon[24927]: ) 1729 : cluster 2021-03-24T22:03:40.597 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:40 smithi125 conmon[24927]: [DBG] osdmap e649: 8 total, 8 up, 7 in 2021-03-24T22:03:40.597 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:40 smithi125 conmon[24927]: cluster 2021-03-24T22:03:40.597 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:40 smithi125 conmon[24927]: 2021-03- 2021-03-24T22:03:40.598 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:40 smithi125 conmon[24927]: 24T22:03 2021-03-24T22:03:40.598 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:40 smithi125 conmon[24927]: :40.344080 2021-03-24T22:03:40.598 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:40 smithi125 conmon[24927]: +0000 mon.a ( 2021-03-24T22:03:40.598 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:40 smithi125 conmon[24927]: mon.0) 1730 2021-03-24T22:03:40.598 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:40 smithi125 conmon[24927]: : cluster [DBG] 2021-03-24T22:03:40.599 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:40 smithi125 conmon[24927]: osdmap e650: 8 total, 8 up, 7 in 2021-03-24T22:03:40.618 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:40.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:40.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:03:40.724 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:40.738 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:40 smithi142 conmon[20125]: cluster 2021-03-24T22:03:38.933676+0000 mgr.y (mgr.14144) 598 : cluster 2021-03-24T22:03:40.738 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:40 smithi142 conmon[20125]: [DBG] pgmap v1135: 168 pgs: 2 active+undersized+remapped, 6 active+recovering+undersized+remapped, 1 active+clean+remapped, 3 active+remapped+backfilling, 2 peering, 4 active+remapped+backfill_toofull, 150 active+clean; 481 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 229 KiB/s wr, 2 op/s; 159/1665 objects misplaced (9.550%); 1.6 MiB/s, 30 objects/s recovering 2021-03-24T22:03:40.739 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:40 smithi142 conmon[20125]: cluster 2021-03-24T22:03:39.385241+0000 mon.a (mon.0) 1729 : cluster [DBG] osdmap e649: 8 total, 8 up, 7 in 2021-03-24T22:03:40.739 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:40 smithi142 conmon[20125]: cluster 2021-03-24T22:03:40.344080+0000 mon.a ( 2021-03-24T22:03:40.739 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:40 smithi142 conmon[20125]: mon.0) 1730 : cluster [DBG] osdmap e650: 8 total, 8 up, 7 in 2021-03-24T22:03:40.779 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:40.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:40.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:03:40.885 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:40.942 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:40.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:40.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:03:40.977 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-24T22:03:41.049 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:41.107 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:41.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:41.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:03:41.214 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:41.262 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:41.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:41.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:03:41.360 INFO:tasks.rados.rados.0.smithi125.stdout:2466: done (4 left) 2021-03-24T22:03:41.360 INFO:tasks.rados.rados.0.smithi125.stdout:2467: snap_remove snap 268 2021-03-24T22:03:41.368 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:41.414 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:41.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:41.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:03:41.460 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:41 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:03:41.460 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:41 smithi125 conmon[24927]: 03:40.584688+0000 osd.2 (osd.2) 101 : cluster [DBG] 3.1a scrub ok 2021-03-24T22:03:41.461 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:41 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:03:41.461 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:41 smithi125 conmon[24927]: 24T22:03:41.356406+0000 2021-03-24T22:03:41.461 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:41 smithi125 conmon[19936]: cluster 2021-03-24T22:03:40.584688+0000 2021-03-24T22:03:41.462 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:41 smithi125 conmon[19936]: osd.2 (osd.2) 101 : cluster [DBG] 3.1a scrub ok 2021-03-24T22:03:41.462 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:41 smithi125 conmon[19936]: cluster 2021-03-24T22:03:41.356406+0000 mon.a (mon.0) 1731 : cluster [DBG] osdmap e651: 8 total, 8 up, 7 in 2021-03-24T22:03:41.521 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:41.532 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:41 smithi125 conmon[24927]: mon.a (mon.0) 1731 : cluster 2021-03-24T22:03:41.532 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:41 smithi125 conmon[24927]: [DBG] osdmap e651: 8 total, 8 up, 7 in 2021-03-24T22:03:41.578 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:41.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:41.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:03:41.684 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:41.696 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:41 smithi142 conmon[20125]: cluster 2021-03-24T22:03:40.584688+0000 osd.2 (osd.2) 2021-03-24T22:03:41.697 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:41 smithi142 conmon[20125]: 101 : cluster [DBG] 3.1a scrub ok 2021-03-24T22:03:41.697 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:41 smithi142 conmon[20125]: cluster 2021-03-24T22:03:41.356406+0000 mon.a 2021-03-24T22:03:41.697 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:41 smithi142 conmon[20125]: (mon.0) 1731 : cluster [DBG] osdmap e651: 8 total, 8 up, 7 in 2021-03-24T22:03:41.739 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:41.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:41.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:03:41.845 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:41.902 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:41.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:41.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:03:42.008 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:42.065 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:42.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:42.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:03:42.172 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:42.229 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:42.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:42.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:03:42.335 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:42.374 INFO:tasks.rados.rados.0.smithi125.stdout:2467: done (4 left) 2021-03-24T22:03:42.375 INFO:tasks.rados.rados.0.smithi125.stdout:2468: delete oid 32 current snap is 278 2021-03-24T22:03:42.375 INFO:tasks.rados.rados.0.smithi125.stdout:2463: finishing write tid 1 to smithi12556470-24 2021-03-24T22:03:42.375 INFO:tasks.rados.rados.0.smithi125.stdout:2463: finishing write tid 2 to smithi12556470-24 2021-03-24T22:03:42.376 INFO:tasks.rados.rados.0.smithi125.stdout:2463: finishing write tid 3 to smithi12556470-24 2021-03-24T22:03:42.376 INFO:tasks.rados.rados.0.smithi125.stdout:2463: finishing write tid 4 to smithi12556470-24 2021-03-24T22:03:42.376 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 24 v 984 (ObjNum 851 snap 278 seq_num 851) dirty exists 2021-03-24T22:03:42.376 INFO:tasks.rados.rados.0.smithi125.stdout:2463: left oid 24 (ObjNum 851 snap 278 seq_num 851) 2021-03-24T22:03:42.377 INFO:tasks.rados.rados.0.smithi125.stdout:2465: finishing write tid 1 to smithi12556470-50 2021-03-24T22:03:42.377 INFO:tasks.rados.rados.0.smithi125.stdout:2465: finishing write tid 2 to smithi12556470-50 2021-03-24T22:03:42.377 INFO:tasks.rados.rados.0.smithi125.stdout:2464: finishing copy_from to smithi12556470-11 2021-03-24T22:03:42.378 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 11 v 549 (ObjNum 809 snap 265 seq_num 809) dirty exists 2021-03-24T22:03:42.378 INFO:tasks.rados.rados.0.smithi125.stdout:2465: finishing write tid 3 to smithi12556470-50 2021-03-24T22:03:42.378 INFO:tasks.rados.rados.0.smithi125.stdout:2464: finishing copy_from racing read to smithi12556470-11 2021-03-24T22:03:42.378 INFO:tasks.rados.rados.0.smithi125.stdout:2465: finishing write tid 4 to smithi12556470-50 2021-03-24T22:03:42.379 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 50 v 799 (ObjNum 852 snap 278 seq_num 852) dirty exists 2021-03-24T22:03:42.379 INFO:tasks.rados.rados.0.smithi125.stdout:2465: left oid 50 (ObjNum 852 snap 278 seq_num 852) 2021-03-24T22:03:42.379 INFO:tasks.rados.rados.0.smithi125.stdout:2460: expect (ObjNum 845 snap 277 seq_num 845) 2021-03-24T22:03:42.398 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:42.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:42.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:03:42.500 INFO:tasks.rados.rados.0.smithi125.stdout:2460: done (4 left) 2021-03-24T22:03:42.500 INFO:tasks.rados.rados.0.smithi125.stdout:2463: done (3 left) 2021-03-24T22:03:42.501 INFO:tasks.rados.rados.0.smithi125.stdout:2464: done (2 left) 2021-03-24T22:03:42.501 INFO:tasks.rados.rados.0.smithi125.stdout:2465: done (1 left) 2021-03-24T22:03:42.501 INFO:tasks.rados.rados.0.smithi125.stdout:2468: done (0 left) 2021-03-24T22:03:42.501 INFO:tasks.rados.rados.0.smithi125.stdout:2469: read oid 3 snap -1 2021-03-24T22:03:42.502 INFO:tasks.rados.rados.0.smithi125.stdout:2469: expect deleted 2021-03-24T22:03:42.502 INFO:tasks.rados.rados.0.smithi125.stdout:2470: delete oid 14 current snap is 278 2021-03-24T22:03:42.502 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:42.504 INFO:tasks.rados.rados.0.smithi125.stdout:2469: done (1 left) 2021-03-24T22:03:42.504 INFO:tasks.rados.rados.0.smithi125.stdout:2470: done (0 left) 2021-03-24T22:03:42.504 INFO:tasks.rados.rados.0.smithi125.stdout:2471: write (excl) oid 27 current snap is 278 2021-03-24T22:03:42.505 INFO:tasks.rados.rados.0.smithi125.stdout:2471: seq_num 855 ranges {769985=569431,1915869=437966} 2021-03-24T22:03:42.518 INFO:tasks.rados.rados.0.smithi125.stdout:2471: writing smithi12556470-27 from 769985 to 1339416 tid 1 2021-03-24T22:03:42.524 INFO:tasks.rados.rados.0.smithi125.stdout:2471: writing smithi12556470-27 from 1915869 to 2353835 tid 2 2021-03-24T22:03:42.525 INFO:tasks.rados.rados.0.smithi125.stdout:2472: read oid 32 snap -1 2021-03-24T22:03:42.525 INFO:tasks.rados.rados.0.smithi125.stdout:2472: expect deleted 2021-03-24T22:03:42.525 INFO:tasks.rados.rados.0.smithi125.stdout:2473: write (excl) oid 23 current snap is 278 2021-03-24T22:03:42.526 INFO:tasks.rados.rados.0.smithi125.stdout:2473: seq_num 856 ranges {786792=644680,1857998=528794} 2021-03-24T22:03:42.533 INFO:tasks.rados.rados.0.smithi125.stdout:2473: writing smithi12556470-23 from 786792 to 1431472 tid 1 2021-03-24T22:03:42.538 INFO:tasks.rados.rados.0.smithi125.stdout:2473: writing smithi12556470-23 from 1857998 to 2386792 tid 2 2021-03-24T22:03:42.539 INFO:tasks.rados.rados.0.smithi125.stdout:2474: write oid 26 current snap is 278 2021-03-24T22:03:42.539 INFO:tasks.rados.rados.0.smithi125.stdout:2474: seq_num 857 ranges {403599=719929,1916480=441635} 2021-03-24T22:03:42.545 INFO:tasks.rados.rados.0.smithi125.stdout:2474: writing smithi12556470-26 from 403599 to 1123528 tid 1 2021-03-24T22:03:42.552 INFO:tasks.rados.rados.0.smithi125.stdout:2474: writing smithi12556470-26 from 1916480 to 2358115 tid 2 2021-03-24T22:03:42.552 INFO:tasks.rados.rados.0.smithi125.stdout:2475: write oid 41 current snap is 278 2021-03-24T22:03:42.552 INFO:tasks.rados.rados.0.smithi125.stdout:2475: seq_num 858 ranges {420406=795178,1974962=445444} 2021-03-24T22:03:42.559 INFO:tasks.rados.rados.0.smithi125.stdout:2475: writing smithi12556470-41 from 420406 to 1215584 tid 1 2021-03-24T22:03:42.563 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:42.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:42.568 INFO:tasks.rados.rados.0.smithi125.stdout:2475: writing smithi12556470-41 from 1974962 to 2420406 tid 2 2021-03-24T22:03:42.568 INFO:tasks.rados.rados.0.smithi125.stdout:2476: rollback oid 46 current snap is 278 2021-03-24T22:03:42.568 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 46 to 192 2021-03-24T22:03:42.569 INFO:tasks.rados.rados.0.smithi125.stdout:2471: finishing write tid 1 to smithi12556470-27 2021-03-24T22:03:42.569 INFO:tasks.rados.rados.0.smithi125.stdout:2471: finishing write tid 2 to smithi12556470-27 2021-03-24T22:03:42.569 INFO:tasks.rados.rados.0.smithi125.stdout:2471: finishing write tid 3 to smithi12556470-27 2021-03-24T22:03:42.569 INFO:tasks.rados.rados.0.smithi125.stdout:2471: finishing write tid 4 to smithi12556470-27 2021-03-24T22:03:42.570 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 27 v 647 (ObjNum 855 snap 278 seq_num 855) dirty exists 2021-03-24T22:03:42.570 INFO:tasks.rados.rados.0.smithi125.stdout:2471: left oid 27 (ObjNum 855 snap 278 seq_num 855) 2021-03-24T22:03:42.570 INFO:tasks.rados.rados.0.smithi125.stdout:2473: finishing write tid 1 to smithi12556470-23 2021-03-24T22:03:42.571 INFO:tasks.rados.rados.0.smithi125.stdout:2471: done (5 left) 2021-03-24T22:03:42.571 INFO:tasks.rados.rados.0.smithi125.stdout:2472: done (4 left) 2021-03-24T22:03:42.571 INFO:tasks.rados.rados.0.smithi125.stdout:2477: snap_create 2021-03-24T22:03:42.572 INFO:tasks.rados.rados.0.smithi125.stdout:2473: finishing write tid 2 to smithi12556470-23 2021-03-24T22:03:42.572 INFO:tasks.rados.rados.0.smithi125.stdout:2473: finishing write tid 3 to smithi12556470-23 2021-03-24T22:03:42.573 INFO:tasks.rados.rados.0.smithi125.stdout:2473: finishing write tid 4 to smithi12556470-23 2021-03-24T22:03:42.573 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 23 v 430 (ObjNum 856 snap 278 seq_num 856) dirty exists 2021-03-24T22:03:42.574 INFO:tasks.rados.rados.0.smithi125.stdout:2473: left oid 23 (ObjNum 856 snap 278 seq_num 856) 2021-03-24T22:03:42.574 INFO:tasks.rados.rados.0.smithi125.stdout:2474: finishing write tid 1 to smithi12556470-26 2021-03-24T22:03:42.575 INFO:tasks.rados.rados.0.smithi125.stdout:2474: finishing write tid 2 to smithi12556470-26 2021-03-24T22:03:42.575 INFO:tasks.rados.rados.0.smithi125.stdout:2474: finishing write tid 3 to smithi12556470-26 2021-03-24T22:03:42.576 INFO:tasks.rados.rados.0.smithi125.stdout:2474: finishing write tid 4 to smithi12556470-26 2021-03-24T22:03:42.576 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 26 v 450 (ObjNum 857 snap 278 seq_num 857) dirty exists 2021-03-24T22:03:42.576 INFO:tasks.rados.rados.0.smithi125.stdout:2474: left oid 26 (ObjNum 857 snap 278 seq_num 857) 2021-03-24T22:03:42.577 INFO:tasks.rados.rados.0.smithi125.stdout:2475: finishing write tid 1 to smithi12556470-41 2021-03-24T22:03:42.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:03:42.580 INFO:tasks.rados.rados.0.smithi125.stdout:2476: finishing rollback tid 0 to smithi12556470-46 2021-03-24T22:03:42.580 INFO:tasks.rados.rados.0.smithi125.stdout:2475: finishing write tid 2 to smithi12556470-41 2021-03-24T22:03:42.581 INFO:tasks.rados.rados.0.smithi125.stdout:2476: finishing rollback tid 1 to smithi12556470-46 2021-03-24T22:03:42.581 INFO:tasks.rados.rados.0.smithi125.stdout:2475: finishing write tid 3 to smithi12556470-41 2021-03-24T22:03:42.581 INFO:tasks.rados.rados.0.smithi125.stdout:2476: finishing rollback tid 2 to smithi12556470-46 2021-03-24T22:03:42.581 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 46 v 730 (ObjNum 497 snap 145 seq_num 497) dirty exists 2021-03-24T22:03:42.582 INFO:tasks.rados.rados.0.smithi125.stdout:2475: finishing write tid 4 to smithi12556470-41 2021-03-24T22:03:42.582 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 41 v 817 (ObjNum 858 snap 278 seq_num 858) dirty exists 2021-03-24T22:03:42.582 INFO:tasks.rados.rados.0.smithi125.stdout:2475: left oid 41 (ObjNum 858 snap 278 seq_num 858) 2021-03-24T22:03:42.610 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:42 smithi125 conmon[24927]: cluster 2021-03-24T22:03: 2021-03-24T22:03:42.611 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:42 smithi125 conmon[24927]: 40.670195+0000 osd.7 (osd.7) 104 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:03:42.612 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:42 smithi125 conmon[24927]: cluster 2021-03-24T22:03:40. 2021-03-24T22:03:42.612 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:42 smithi125 conmon[24927]: 674147+0000 osd.7 (osd.7) 105 : cluster [DBG] 2021-03-24T22:03:42.613 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:42 smithi125 conmon[24927]: purged_snaps scrub ok 2021-03-24T22:03:42.613 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:42 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:03:42.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:42 smithi125 conmon[24927]: 03:40.934454+0000 mgr.y (mgr.14144) 599 2021-03-24T22:03:42.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:42 smithi125 conmon[24927]: : cluster [DBG] pgmap v1138: 168 pgs: 1 active+undersized+remapped, 2 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 peering, 4 active+remapped+backfill_toofull, 159 active+clean; 487 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 518 KiB/s wr, 4 op/s; 113/1674 objects misplaced (6.750%); 6.9 MiB/s, 15 objects/s recovering 2021-03-24T22:03:42.614 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:42 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:03:42.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:42 smithi125 conmon[24927]: 24T22:03:41.269895+0000 osd.3 (osd 2021-03-24T22:03:42.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:42 smithi125 conmon[24927]: .3) 132 : cluster [DBG] 3.24 scrub ok 2021-03-24T22:03:42.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:42 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:03:42.615 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:42 smithi125 conmon[24927]: 03-24T22:03:42.361219+0000 mon.a ( 2021-03-24T22:03:42.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:42 smithi125 conmon[24927]: mon.0) 1732 : cluster [DBG] osdmap e652: 8 total, 8 up, 7 in 2021-03-24T22:03:42.616 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:42 smithi125 conmon[19936]: cluster 2021-03-24T22:03 2021-03-24T22:03:42.616 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:42 smithi125 conmon[19936]: :40.670195+0000 osd.7 (osd.7) 104 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:03:42.617 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:42 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T22:03:42.617 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:42 smithi125 conmon[19936]: 22:03:40.674147+0000 osd.7 (osd.7) 105 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:03:42.617 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:42 smithi125 conmon[19936]: cluster 2021-03-24T22:03:42.617 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:42 smithi125 conmon[19936]: 2021-03 2021-03-24T22:03:42.617 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:42 smithi125 conmon[19936]: -24T22:03:40.934454+0000 mgr.y (mgr.14144 2021-03-24T22:03:42.618 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:42 smithi125 conmon[19936]: ) 599 : cluster [DBG] pgmap v1138: 168 pgs: 1 active+undersized+remapped, 2 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 peering, 4 active+remapped+backfill_toofull, 159 active+clean; 487 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 518 KiB/s wr, 4 op/s; 113/1674 objects misplaced (6.750%); 6.9 MiB/s, 15 objects/s recovering 2021-03-24T22:03:42.618 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:42 smithi125 conmon[19936]: cluster 2021-03-24T22:03:42.618 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:42 smithi125 conmon[19936]: 2021-03-24T22:03:41.269895+0000 2021-03-24T22:03:42.619 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:42 smithi125 conmon[19936]: osd.3 (osd.3) 132 : cluster [DBG] 3.24 scrub ok 2021-03-24T22:03:42.619 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:42 smithi125 conmon[19936]: 2021-03-24T22:03:42.619 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:42 smithi125 conmon[19936]: cluster 2021-03-24T22:03: 2021-03-24T22:03:42.620 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:42 smithi125 conmon[19936]: 42.361219+0000 mon.a (mon.0) 1732 : 2021-03-24T22:03:42.620 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:42 smithi125 conmon[19936]: cluster [DBG] osdmap e652: 8 total, 8 up, 7 in 2021-03-24T22:03:42.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:42 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:03:42.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:42 smithi142 conmon[20125]: 22:03:40.670195+0000 osd.7 (osd.7) 104 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:03:42.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:42 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:03:42.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:42 smithi142 conmon[20125]: 24T22:03:40.674147+0000 osd.7 2021-03-24T22:03:42.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:42 smithi142 conmon[20125]: (osd.7) 105 : cluster [DBG] 2021-03-24T22:03:42.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:42 smithi142 conmon[20125]: purged_snaps scrub ok 2021-03-24T22:03:42.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:42 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:03:42.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:42 smithi142 conmon[20125]: 24T22:03:40.934454+0000 mgr.y 2021-03-24T22:03:42.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:42 smithi142 conmon[20125]: (mgr.14144) 599 : cluster 2021-03-24T22:03:42.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:42 smithi142 conmon[20125]: [DBG] pgmap v1138: 168 pgs: 1 active+undersized+remapped, 2 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 peering, 4 active+remapped+backfill_toofull, 159 active+clean; 487 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 518 KiB/s wr, 4 op/s; 113/1674 objects misplaced (6.750%); 6.9 MiB/s, 15 objects/s recovering 2021-03-24T22:03:42.625 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:42 smithi142 conmon[20125]: cluster 2021-03-24T22:03:42.625 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:42 smithi142 conmon[20125]: 2021-03-24T22 2021-03-24T22:03:42.626 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:42 smithi142 conmon[20125]: :03:41.269895+0000 osd.3 2021-03-24T22:03:42.626 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:42 smithi142 conmon[20125]: (osd.3) 132 : cluster 2021-03-24T22:03:42.626 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:42 smithi142 conmon[20125]: [DBG] 3.24 scrub ok 2021-03-24T22:03:42.627 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:42 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:03:42.627 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:42 smithi142 conmon[20125]: 03-24T22:03:42.361219+0000 2021-03-24T22:03:42.627 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:42 smithi142 conmon[20125]: mon.a (mon.0) 1732 2021-03-24T22:03:42.628 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:42 smithi142 conmon[20125]: : cluster [DBG] osdmap e652: 8 total, 8 up, 7 in 2021-03-24T22:03:42.680 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:42.741 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:42.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:42.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:03:42.752 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:03:42.845 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:42.901 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:42.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:42.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:03:43.007 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:43.063 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:43.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:43.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:03:43.170 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:43.227 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:43.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:43.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:03:43.332 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:43.353 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T22:03:43.376 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T22:03:43.388 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:43.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:43.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:03:43.402 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T22:03:43.425 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T22:03:43.449 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T22:03:43.459 INFO:tasks.rados.rados.0.smithi125.stdout:2473: done (4 left) 2021-03-24T22:03:43.459 INFO:tasks.rados.rados.0.smithi125.stdout:2474: done (3 left) 2021-03-24T22:03:43.459 INFO:tasks.rados.rados.0.smithi125.stdout:2475: done (2 left) 2021-03-24T22:03:43.460 INFO:tasks.rados.rados.0.smithi125.stdout:2476: done (1 left) 2021-03-24T22:03:43.460 INFO:tasks.rados.rados.0.smithi125.stdout:2477: done (0 left) 2021-03-24T22:03:43.460 INFO:tasks.rados.rados.0.smithi125.stdout:2478: read oid 29 snap -1 2021-03-24T22:03:43.460 INFO:tasks.rados.rados.0.smithi125.stdout:2478: expect (ObjNum 691 snap 218 seq_num 691) 2021-03-24T22:03:43.460 INFO:tasks.rados.rados.0.smithi125.stdout:2479: copy_from oid 30 from oid 44 current snap is 279 2021-03-24T22:03:43.461 INFO:tasks.rados.rados.0.smithi125.stdout:2480: write (excl) oid 10 current snap is 279 2021-03-24T22:03:43.461 INFO:tasks.rados.rados.0.smithi125.stdout:2480: seq_num 859 ranges {437213=470427,1633444=728951} 2021-03-24T22:03:43.462 INFO:tasks.rados.rados.0.smithi125.stdout:2480: writing smithi12556470-10 from 437213 to 907640 tid 1 2021-03-24T22:03:43.471 INFO:tasks.rados.rados.0.smithi125.stdout:2480: writing smithi12556470-10 from 1633444 to 2362395 tid 2 2021-03-24T22:03:43.471 INFO:tasks.rados.rados.0.smithi125.stdout:2481: write (excl) oid 5 current snap is 279 2021-03-24T22:03:43.472 INFO:tasks.rados.rados.0.smithi125.stdout:2481: seq_num 860 ranges {454020=662029,1691926=588962} 2021-03-24T22:03:43.478 INFO:tasks.rados.rados.0.smithi125.stdout:2481: writing smithi12556470-5 from 454020 to 1116049 tid 1 2021-03-24T22:03:43.486 INFO:tasks.rados.rados.0.smithi125.stdout:2481: writing smithi12556470-5 from 1691926 to 2280888 tid 2 2021-03-24T22:03:43.487 INFO:tasks.rados.rados.0.smithi125.stdout:2482: write (excl) oid 18 current snap is 279 2021-03-24T22:03:43.487 INFO:tasks.rados.rados.0.smithi125.stdout:2482: seq_num 861 ranges {470827=737278,1750408=720419} 2021-03-24T22:03:43.492 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T22:03:43.493 INFO:tasks.rados.rados.0.smithi125.stdout:2482: writing smithi12556470-18 from 470827 to 1208105 tid 1 2021-03-24T22:03:43.493 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:43.501 INFO:tasks.rados.rados.0.smithi125.stdout:2482: writing smithi12556470-18 from 1750408 to 2470827 tid 2 2021-03-24T22:03:43.501 INFO:tasks.rados.rados.0.smithi125.stdout:2483: read oid 27 snap 216 2021-03-24T22:03:43.502 INFO:tasks.rados.rados.0.smithi125.stdout:2483: expect deleted 2021-03-24T22:03:43.502 INFO:tasks.rados.rados.0.smithi125.stdout:2484: write oid 35 current snap is 279 2021-03-24T22:03:43.502 INFO:tasks.rados.rados.0.smithi125.stdout:2480: finishing write tid 1 to smithi12556470-10 2021-03-24T22:03:43.503 INFO:tasks.rados.rados.0.smithi125.stdout:2484: seq_num 862 ranges {487634=412527,1408890=592631} 2021-03-24T22:03:43.506 INFO:tasks.rados.rados.0.smithi125.stdout:2484: writing smithi12556470-35 from 487634 to 900161 tid 1 2021-03-24T22:03:43.509 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-24T22:03:43.509 INFO:teuthology.orchestra.run.smithi125.stdout: "size": 20, 2021-03-24T22:03:43.510 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 600, 2021-03-24T22:03:43.510 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [ 2021-03-24T22:03:43.510 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.510 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:5354 7.17 7:ec6a359b:::smithi12556470-33:head [stat,delete] snapc 1b8=[1b8,1b6,1b5,1b3,1af,1ae,1ac,1ab,1a8,1a6,19e,19c,19a,199,197,194,193,18d,18c,18b,182,179,178,172,16e,166,159,157,139,129,11b] ondisk+write+known_if_redirected e557)", 2021-03-24T22:03:43.511 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:02:07.292786+0000", 2021-03-24T22:03:43.511 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 96.211044442000002, 2021-03-24T22:03:43.511 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018594981999999999, 2021-03-24T22:03:43.511 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:03:43.512 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T22:03:43.512 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:03:43.512 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:03:43.513 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:03:43.513 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 5354 2021-03-24T22:03:43.514 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.514 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:03:43.514 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.515 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:03:43.515 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:07.292786+0000", 2021-03-24T22:03:43.516 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:03:43.516 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.516 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.517 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:03:43.517 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:07.292786+0000", 2021-03-24T22:03:43.517 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T22:03:43.518 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.518 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.518 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:03:43.518 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:07.292784+0000", 2021-03-24T22:03:43.519 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0672999999999999e-05 2021-03-24T22:03:43.519 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.519 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.519 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:03:43.520 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:07.292795+0000", 2021-03-24T22:03:43.520 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0008e-05 2021-03-24T22:03:43.520 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.521 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.521 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:03:43.521 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:07.292805+0000", 2021-03-24T22:03:43.521 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.8202e-05 2021-03-24T22:03:43.522 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.522 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.522 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:03:43.522 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:07.292823+0000", 2021-03-24T22:03:43.523 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.0215000000000005e-05 2021-03-24T22:03:43.523 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.523 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.523 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:03:43.523 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:07.292893+0000", 2021-03-24T22:03:43.524 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.273e-05 2021-03-24T22:03:43.524 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.524 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.525 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op must wait for map", 2021-03-24T22:03:43.525 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:07.292906+0000", 2021-03-24T22:03:43.525 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0045973999999999998 2021-03-24T22:03:43.526 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.526 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.527 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:03:43.527 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:07.297503+0000", 2021-03-24T22:03:43.527 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00013446500000000001 2021-03-24T22:03:43.528 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.528 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.528 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:03:43.528 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:07.297638+0000", 2021-03-24T22:03:43.528 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.01374296 2021-03-24T22:03:43.529 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.529 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.529 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:03:43.529 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:07.311381+0000", 2021-03-24T22:03:43.530 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018594981999999999 2021-03-24T22:03:43.530 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:03:43.530 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:03:43.530 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:03:43.531 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.531 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.531 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "MOSDPGPushReply(7.17 557/547 [PushReplyOp(7:ec7ca62b:::smithi12556470-39:183)])", 2021-03-24T22:03:43.532 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:02:07.313231+0000", 2021-03-24T22:03:43.532 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 96.190598983000001, 2021-03-24T22:03:43.532 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.01167613, 2021-03-24T22:03:43.532 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:03:43.532 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "reached pg", 2021-03-24T22:03:43.533 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:03:43.533 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.533 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:03:43.533 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:07.313231+0000", 2021-03-24T22:03:43.534 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:03:43.534 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.534 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.534 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:03:43.535 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:07.313231+0000", 2021-03-24T22:03:43.535 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T22:03:43.535 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.535 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.535 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:03:43.536 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:07.313230+0000", 2021-03-24T22:03:43.536 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.2779999999999999e-06 2021-03-24T22:03:43.536 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.536 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.537 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:03:43.537 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:07.313239+0000", 2021-03-24T22:03:43.537 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9352e-05 2021-03-24T22:03:43.538 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.538 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.539 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:03:43.539 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:07.313258+0000", 2021-03-24T22:03:43.539 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.4780000000000008e-06 2021-03-24T22:03:43.539 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.540 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.540 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:03:43.540 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:07.313267+0000", 2021-03-24T22:03:43.540 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2298000000000004e-05 2021-03-24T22:03:43.541 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.541 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.541 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:03:43.541 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:07.313349+0000", 2021-03-24T22:03:43.542 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011558389000000001 2021-03-24T22:03:43.542 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.542 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.543 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:03:43.543 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:07.324907+0000", 2021-03-24T22:03:43.543 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.01167613 2021-03-24T22:03:43.543 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:03:43.544 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:03:43.544 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:03:43.544 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.545 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.545 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:5466 7.1d 7:bdda3e67:::smithi12556470-22:head [copy-from ver 657] snapc 1c4=[1c4,1bf,1bd,1bc,1bb,1b6,1b5,1b3,1af,1ae,1ab,1a6,19e,19c,199,197,194,193,18d,18c,18b,182,178,172,16e,159,157,139,11b] ondisk+write+known_if_redirected e571)", 2021-03-24T22:03:43.545 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:02:21.337604+0000", 2021-03-24T22:03:43.545 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 82.166226002000002, 2021-03-24T22:03:43.546 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.022577342, 2021-03-24T22:03:43.546 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:03:43.546 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:03:43.546 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:03:43.547 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:03:43.547 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:03:43.547 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 5466 2021-03-24T22:03:43.547 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.548 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:03:43.548 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.548 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:03:43.549 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:21.337604+0000", 2021-03-24T22:03:43.549 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:03:43.549 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.549 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.549 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:03:43.550 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:21.337604+0000", 2021-03-24T22:03:43.550 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T22:03:43.550 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.550 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.551 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:03:43.551 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:21.337602+0000", 2021-03-24T22:03:43.551 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0352e-05 2021-03-24T22:03:43.551 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.552 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.552 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:03:43.552 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:21.337613+0000", 2021-03-24T22:03:43.553 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7992999999999999e-05 2021-03-24T22:03:43.553 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.553 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.554 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:03:43.554 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:21.337631+0000", 2021-03-24T22:03:43.555 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.3520000000000007e-06 2021-03-24T22:03:43.555 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.555 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.556 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:03:43.556 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:21.337639+0000", 2021-03-24T22:03:43.556 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.0822000000000001e-05 2021-03-24T22:03:43.557 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.557 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.557 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:03:43.558 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:21.337700+0000", 2021-03-24T22:03:43.558 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00013524899999999999 2021-03-24T22:03:43.559 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.559 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.559 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:03:43.560 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:21.337835+0000", 2021-03-24T22:03:43.560 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0099655500000000001 2021-03-24T22:03:43.560 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.561 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.561 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,6", 2021-03-24T22:03:43.561 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:21.347801+0000", 2021-03-24T22:03:43.562 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0069512180000000003 2021-03-24T22:03:43.562 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.563 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.563 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:03:43.563 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:21.354752+0000", 2021-03-24T22:03:43.564 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0044033420000000002 2021-03-24T22:03:43.564 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.564 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.566 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:03:43.566 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:21.359155+0000", 2021-03-24T22:03:43.567 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00088427700000000004 2021-03-24T22:03:43.567 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.567 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.567 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:03:43.567 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:21.360039+0000", 2021-03-24T22:03:43.568 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.7307999999999999e-05 2021-03-24T22:03:43.568 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.568 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.568 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:03:43.569 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:21.360117+0000", 2021-03-24T22:03:43.569 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4873e-05 2021-03-24T22:03:43.569 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.569 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.570 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:03:43.570 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:21.360182+0000", 2021-03-24T22:03:43.571 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.022577342 2021-03-24T22:03:43.571 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:03:43.571 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:03:43.571 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:03:43.571 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.572 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.572 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:5467 7.1d 7:bdda3e67:::smithi12556470-22:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e571)", 2021-03-24T22:03:43.572 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:02:21.337984+0000", 2021-03-24T22:03:43.572 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 82.165846430000002, 2021-03-24T22:03:43.573 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.022554026000000001, 2021-03-24T22:03:43.573 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:03:43.573 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T22:03:43.574 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:03:43.574 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:03:43.574 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:03:43.574 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 5467 2021-03-24T22:03:43.575 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.575 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:03:43.575 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.575 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:03:43.575 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:21.337984+0000", 2021-03-24T22:03:43.576 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:03:43.576 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.576 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.576 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:03:43.577 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:21.337984+0000", 2021-03-24T22:03:43.577 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T22:03:43.577 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.577 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.577 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:03:43.578 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:21.337983+0000", 2021-03-24T22:03:43.578 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.4369999999999998e-06 2021-03-24T22:03:43.578 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.578 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.579 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:03:43.579 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:21.337988+0000", 2021-03-24T22:03:43.579 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.0499999999999992e-06 2021-03-24T22:03:43.579 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.580 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.580 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:03:43.580 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:21.337996+0000", 2021-03-24T22:03:43.581 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.4859999999999997e-06 2021-03-24T22:03:43.581 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.581 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.581 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:03:43.581 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:21.338002+0000", 2021-03-24T22:03:43.582 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00047269199999999998 2021-03-24T22:03:43.582 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.582 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.583 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:03:43.583 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:21.338474+0000", 2021-03-24T22:03:43.583 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.1631999999999997e-05 2021-03-24T22:03:43.583 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.583 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.584 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-24T22:03:43.584 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:21.338516+0000", 2021-03-24T22:03:43.585 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.014071686999999999 2021-03-24T22:03:43.585 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.585 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.585 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:03:43.585 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:21.352588+0000", 2021-03-24T22:03:43.586 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012302999999999999 2021-03-24T22:03:43.586 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.586 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.586 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:03:43.587 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:21.352711+0000", 2021-03-24T22:03:43.587 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0075133120000000003 2021-03-24T22:03:43.587 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.587 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.587 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:03:43.588 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:21.360224+0000", 2021-03-24T22:03:43.588 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00014969199999999999 2021-03-24T22:03:43.588 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.588 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.589 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:03:43.589 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:21.360374+0000", 2021-03-24T22:03:43.589 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00016424699999999999 2021-03-24T22:03:43.589 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.590 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.590 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:03:43.591 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:21.360538+0000", 2021-03-24T22:03:43.591 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.022554026000000001 2021-03-24T22:03:43.591 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:03:43.592 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:03:43.592 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:03:43.592 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.592 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.593 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(osd.0.0:2300 7.9 e576/511 7:9511f6a7:::smithi12556470-23:19a v 576'370, mlcod=573'366)", 2021-03-24T22:03:43.593 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:02:26.311200+0000", 2021-03-24T22:03:43.593 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 77.192630218000005, 2021-03-24T22:03:43.593 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011541202, 2021-03-24T22:03:43.593 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:03:43.594 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:03:43.594 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:03:43.594 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.594 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:03:43.595 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:26.311200+0000", 2021-03-24T22:03:43.595 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:03:43.595 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.596 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.596 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:03:43.596 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:26.311200+0000", 2021-03-24T22:03:43.596 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999995 2021-03-24T22:03:43.597 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.597 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.598 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:03:43.598 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:26.311200+0000", 2021-03-24T22:03:43.598 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.3870000000000003e-06 2021-03-24T22:03:43.599 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.599 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.599 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:03:43.600 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:26.311205+0000", 2021-03-24T22:03:43.600 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.7559999999999994e-06 2021-03-24T22:03:43.600 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.601 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.601 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:03:43.602 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:26.311215+0000", 2021-03-24T22:03:43.602 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6499999999999999e-06 2021-03-24T22:03:43.602 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.603 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.603 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:03:43.603 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:26.311221+0000", 2021-03-24T22:03:43.603 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0040186520000000002 2021-03-24T22:03:43.604 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.604 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.604 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:03:43.605 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:26.315240+0000", 2021-03-24T22:03:43.605 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0852999999999998e-05 2021-03-24T22:03:43.605 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.606 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.606 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:03:43.606 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:26.315261+0000", 2021-03-24T22:03:43.606 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0074376429999999999 2021-03-24T22:03:43.607 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.607 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.607 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:03:43.607 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:26.322699+0000", 2021-03-24T22:03:43.608 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.2654999999999997e-05 2021-03-24T22:03:43.608 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.608 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.608 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:03:43.608 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:26.322741+0000", 2021-03-24T22:03:43.609 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011541202 2021-03-24T22:03:43.609 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:03:43.609 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:03:43.609 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:03:43.610 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.610 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.610 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:5518 7.1d 7:bdda3e67:::smithi12556470-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e577)", 2021-03-24T22:03:43.610 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:02:27.323252+0000", 2021-03-24T22:03:43.611 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 76.180577850000006, 2021-03-24T22:03:43.611 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012499077000000001, 2021-03-24T22:03:43.611 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:03:43.611 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T22:03:43.611 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:03:43.612 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:03:43.612 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:03:43.612 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 5518 2021-03-24T22:03:43.612 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.613 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:03:43.613 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.613 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:03:43.613 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:27.323252+0000", 2021-03-24T22:03:43.614 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:03:43.614 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.614 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.614 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:03:43.615 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:27.323252+0000", 2021-03-24T22:03:43.615 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999995 2021-03-24T22:03:43.615 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.615 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.616 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:03:43.616 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:27.323252+0000", 2021-03-24T22:03:43.616 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.0230000000000001e-06 2021-03-24T22:03:43.617 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.617 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.617 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:03:43.617 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:27.323255+0000", 2021-03-24T22:03:43.618 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.4840000000000003e-06 2021-03-24T22:03:43.618 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.618 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.618 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:03:43.618 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:27.323260+0000", 2021-03-24T22:03:43.619 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.1010000000000002e-06 2021-03-24T22:03:43.619 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.619 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.619 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:03:43.620 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:27.323265+0000", 2021-03-24T22:03:43.620 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0022869740000000002 2021-03-24T22:03:43.620 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.620 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.621 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:03:43.621 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:27.325552+0000", 2021-03-24T22:03:43.621 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2544999999999999e-05 2021-03-24T22:03:43.621 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.621 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.622 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:03:43.622 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:27.325604+0000", 2021-03-24T22:03:43.622 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0083708329999999994 2021-03-24T22:03:43.622 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.623 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.623 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:03:43.623 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:27.333975+0000", 2021-03-24T22:03:43.623 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.4221999999999994e-05 2021-03-24T22:03:43.624 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.624 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.624 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:03:43.624 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:27.334059+0000", 2021-03-24T22:03:43.624 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001692348 2021-03-24T22:03:43.625 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.625 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.625 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:03:43.625 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:27.335751+0000", 2021-03-24T22:03:43.625 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012499077000000001 2021-03-24T22:03:43.626 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:03:43.626 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:03:43.626 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:03:43.627 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.627 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.627 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:5642 7.23 e591/535 7:c49de0b4:::smithi12556470-13:head v 591'327, mlcod=588'323)", 2021-03-24T22:03:43.627 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:02:41.336810+0000", 2021-03-24T22:03:43.628 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 62.167019895000003, 2021-03-24T22:03:43.628 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.019465100999999999, 2021-03-24T22:03:43.628 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:03:43.628 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:03:43.629 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:03:43.629 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.629 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:03:43.629 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:41.336810+0000", 2021-03-24T22:03:43.630 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:03:43.630 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.630 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.630 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:03:43.630 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:41.336810+0000", 2021-03-24T22:03:43.631 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999948 2021-03-24T22:03:43.631 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.631 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.631 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:03:43.632 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:41.336805+0000", 2021-03-24T22:03:43.632 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.017312919 2021-03-24T22:03:43.632 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.632 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.632 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:03:43.633 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:41.354118+0000", 2021-03-24T22:03:43.633 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.4385999999999999e-05 2021-03-24T22:03:43.633 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.633 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.633 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:03:43.634 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:41.354152+0000", 2021-03-24T22:03:43.634 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.6129999999999993e-06 2021-03-24T22:03:43.634 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.634 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.635 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:03:43.635 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:41.354162+0000", 2021-03-24T22:03:43.635 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9636000000000003e-05 2021-03-24T22:03:43.635 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.635 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.636 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:03:43.636 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:41.354221+0000", 2021-03-24T22:03:43.636 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.7183e-05 2021-03-24T22:03:43.636 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.637 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.637 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:03:43.637 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:41.354249+0000", 2021-03-24T22:03:43.637 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001991877 2021-03-24T22:03:43.638 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.638 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.639 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:03:43.639 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:41.356241+0000", 2021-03-24T22:03:43.639 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.4928999999999997e-05 2021-03-24T22:03:43.639 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.639 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.640 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:03:43.640 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:41.356275+0000", 2021-03-24T22:03:43.640 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.019465100999999999 2021-03-24T22:03:43.641 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:03:43.641 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:03:43.641 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:03:43.641 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.641 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.642 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:5727 7.1d 7:bdda3e67:::smithi12556470-22:head [copy-from ver 503] snapc 1dc=[1dc,1d9,1d8,1d7,1d4,1d1,1cd,1cc,1c4,1bf,1bd,1bb,1b3,1ae,1ab,1a6,19e,19c,199,194,193,18b,178,172,16e,159,157,139] ondisk+write+known_if_redirected e594)", 2021-03-24T22:03:43.642 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:02:44.376548+0000", 2021-03-24T22:03:43.642 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 59.127282645999998, 2021-03-24T22:03:43.642 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.020174076999999999, 2021-03-24T22:03:43.643 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:03:43.643 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:03:43.643 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:03:43.643 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:03:43.643 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:03:43.644 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 5727 2021-03-24T22:03:43.644 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.644 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:03:43.644 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.645 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:03:43.645 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.376548+0000", 2021-03-24T22:03:43.645 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:03:43.645 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.645 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.646 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:03:43.646 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.376548+0000", 2021-03-24T22:03:43.646 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999962 2021-03-24T22:03:43.646 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.647 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.647 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:03:43.647 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.376544+0000", 2021-03-24T22:03:43.647 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.6580999999999999e-05 2021-03-24T22:03:43.648 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.648 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.648 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:03:43.648 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.376561+0000", 2021-03-24T22:03:43.649 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.7586e-05 2021-03-24T22:03:43.649 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.649 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.649 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:03:43.649 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.376588+0000", 2021-03-24T22:03:43.650 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1216000000000001e-05 2021-03-24T22:03:43.650 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.650 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.650 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:03:43.651 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.376599+0000", 2021-03-24T22:03:43.651 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.0196000000000002e-05 2021-03-24T22:03:43.651 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.651 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.652 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:03:43.652 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.376690+0000", 2021-03-24T22:03:43.652 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00014224100000000001 2021-03-24T22:03:43.652 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.653 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.653 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:03:43.653 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.376832+0000", 2021-03-24T22:03:43.653 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.010210917999999999 2021-03-24T22:03:43.654 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.654 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.654 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,6", 2021-03-24T22:03:43.654 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.387043+0000", 2021-03-24T22:03:43.654 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0078389830000000008 2021-03-24T22:03:43.655 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.655 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.655 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:03:43.655 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.394882+0000", 2021-03-24T22:03:43.656 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00057339700000000001 2021-03-24T22:03:43.656 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.656 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.656 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:03:43.657 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.395455+0000", 2021-03-24T22:03:43.657 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0011732559999999999 2021-03-24T22:03:43.657 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.657 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.658 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:03:43.658 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.396628+0000", 2021-03-24T22:03:43.658 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.8783999999999999e-05 2021-03-24T22:03:43.658 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.658 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.659 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:03:43.659 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.396677+0000", 2021-03-24T22:03:43.659 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.4496999999999998e-05 2021-03-24T22:03:43.659 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.660 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.660 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:03:43.660 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.396722+0000", 2021-03-24T22:03:43.660 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.020174076999999999 2021-03-24T22:03:43.661 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:03:43.661 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:03:43.661 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:03:43.661 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.662 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.662 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:5728 7.1d 7:bdda3e67:::smithi12556470-22:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e594)", 2021-03-24T22:03:43.662 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:02:44.376628+0000", 2021-03-24T22:03:43.663 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 59.127201929000002, 2021-03-24T22:03:43.663 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.020236212, 2021-03-24T22:03:43.663 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:03:43.663 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T22:03:43.664 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:03:43.664 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:03:43.664 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:03:43.664 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 5728 2021-03-24T22:03:43.665 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.665 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:03:43.665 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.665 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:03:43.665 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.376628+0000", 2021-03-24T22:03:43.666 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:03:43.666 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.666 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.667 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:03:43.667 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.376628+0000", 2021-03-24T22:03:43.667 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T22:03:43.667 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.668 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.668 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:03:43.668 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.376628+0000", 2021-03-24T22:03:43.668 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.4020000000000004e-06 2021-03-24T22:03:43.669 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.669 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.669 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:03:43.670 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.376632+0000", 2021-03-24T22:03:43.670 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2160000000000006e-06 2021-03-24T22:03:43.670 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.670 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.671 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:03:43.671 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.376640+0000", 2021-03-24T22:03:43.671 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.7250000000000002e-06 2021-03-24T22:03:43.671 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.672 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.672 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:03:43.672 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.376646+0000", 2021-03-24T22:03:43.672 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000412731 2021-03-24T22:03:43.673 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.673 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.673 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:03:43.673 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.377059+0000", 2021-03-24T22:03:43.674 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2562000000000003e-05 2021-03-24T22:03:43.674 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.674 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.674 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-24T22:03:43.675 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.377111+0000", 2021-03-24T22:03:43.675 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.01476828 2021-03-24T22:03:43.675 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.675 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.676 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:03:43.676 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.391879+0000", 2021-03-24T22:03:43.676 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000105158 2021-03-24T22:03:43.676 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.677 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.677 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:03:43.677 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.391985+0000", 2021-03-24T22:03:43.678 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0047677149999999996 2021-03-24T22:03:43.678 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.678 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.679 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:03:43.679 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.396752+0000", 2021-03-24T22:03:43.679 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.9604999999999997e-05 2021-03-24T22:03:43.679 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.679 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.680 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:03:43.680 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.396802+0000", 2021-03-24T22:03:43.680 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2619999999999993e-05 2021-03-24T22:03:43.680 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.681 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.681 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:03:43.681 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.396865+0000", 2021-03-24T22:03:43.681 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.020236212 2021-03-24T22:03:43.682 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:03:43.682 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:03:43.682 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:03:43.682 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.683 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.683 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:5734 7.d e594/537 7:b4364827:::smithi12556470-44:head v 594'497, mlcod=581'493)", 2021-03-24T22:03:43.683 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:02:44.377555+0000", 2021-03-24T22:03:43.683 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 59.126275745000001, 2021-03-24T22:03:43.684 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.014362239000000001, 2021-03-24T22:03:43.684 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:03:43.684 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:03:43.684 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:03:43.685 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.685 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:03:43.685 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.377555+0000", 2021-03-24T22:03:43.685 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:03:43.685 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.686 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.686 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:03:43.686 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.377555+0000", 2021-03-24T22:03:43.686 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T22:03:43.687 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.687 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.687 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:03:43.687 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.377553+0000", 2021-03-24T22:03:43.688 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0672e-05 2021-03-24T22:03:43.688 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.688 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.689 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:03:43.689 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.377563+0000", 2021-03-24T22:03:43.689 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5922000000000001e-05 2021-03-24T22:03:43.689 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.690 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.690 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:03:43.690 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.377579+0000", 2021-03-24T22:03:43.690 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.7580000000000003e-06 2021-03-24T22:03:43.691 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.691 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.691 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:03:43.691 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.377586+0000", 2021-03-24T22:03:43.692 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6195000000000001e-05 2021-03-24T22:03:43.692 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.692 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.692 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:03:43.692 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.377632+0000", 2021-03-24T22:03:43.693 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.3645000000000001e-05 2021-03-24T22:03:43.693 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.693 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.693 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:03:43.694 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.377656+0000", 2021-03-24T22:03:43.694 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.014210834 2021-03-24T22:03:43.694 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.694 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.695 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:03:43.695 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.391867+0000", 2021-03-24T22:03:43.695 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.0161000000000003e-05 2021-03-24T22:03:43.695 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.696 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.696 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:03:43.696 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.391917+0000", 2021-03-24T22:03:43.697 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.014362239000000001 2021-03-24T22:03:43.697 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:03:43.697 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:03:43.697 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:03:43.698 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.698 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.698 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:5735 7.d e594/537 7:b4364827:::smithi12556470-44:head v 594'498, mlcod=581'493)", 2021-03-24T22:03:43.698 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:02:44.378777+0000", 2021-03-24T22:03:43.699 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 59.125053055000002, 2021-03-24T22:03:43.699 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.013160862000000001, 2021-03-24T22:03:43.699 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:03:43.699 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:03:43.700 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:03:43.700 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.700 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:03:43.700 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.378777+0000", 2021-03-24T22:03:43.701 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:03:43.701 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.701 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.701 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:03:43.702 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.378777+0000", 2021-03-24T22:03:43.702 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T22:03:43.702 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.702 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.703 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:03:43.703 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.378776+0000", 2021-03-24T22:03:43.703 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.7049999999999994e-06 2021-03-24T22:03:43.703 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.703 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.704 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:03:43.704 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.378785+0000", 2021-03-24T22:03:43.704 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2018000000000001e-05 2021-03-24T22:03:43.704 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.705 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.705 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:03:43.705 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.378797+0000", 2021-03-24T22:03:43.705 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.8129999999999998e-06 2021-03-24T22:03:43.706 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.706 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.706 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:03:43.707 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.378804+0000", 2021-03-24T22:03:43.707 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0047160830000000003 2021-03-24T22:03:43.707 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.707 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.708 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:03:43.708 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.383520+0000", 2021-03-24T22:03:43.708 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.4428999999999999e-05 2021-03-24T22:03:43.708 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.708 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.709 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:03:43.709 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.383545+0000", 2021-03-24T22:03:43.709 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0083767140000000004 2021-03-24T22:03:43.709 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.710 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.710 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:03:43.710 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.391921+0000", 2021-03-24T22:03:43.710 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.6696000000000001e-05 2021-03-24T22:03:43.711 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.711 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.711 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:03:43.711 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.391938+0000", 2021-03-24T22:03:43.712 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.013160862000000001 2021-03-24T22:03:43.712 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:03:43.712 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:03:43.712 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:03:43.712 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.713 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.713 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:5736 7.d e594/537 7:b4364827:::smithi12556470-44:head v 594'499, mlcod=581'493)", 2021-03-24T22:03:43.713 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:02:44.380028+0000", 2021-03-24T22:03:43.713 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 59.123802646000001, 2021-03-24T22:03:43.714 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011933084, 2021-03-24T22:03:43.714 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:03:43.714 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:03:43.714 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:03:43.715 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.715 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:03:43.715 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.380028+0000", 2021-03-24T22:03:43.715 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:03:43.716 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.716 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.716 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:03:43.716 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.380028+0000", 2021-03-24T22:03:43.717 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T22:03:43.717 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.717 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.717 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:03:43.718 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.380026+0000", 2021-03-24T22:03:43.718 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0376e-05 2021-03-24T22:03:43.718 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.718 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.719 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:03:43.719 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.380036+0000", 2021-03-24T22:03:43.719 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3839000000000001e-05 2021-03-24T22:03:43.719 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.720 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.720 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:03:43.720 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.380050+0000", 2021-03-24T22:03:43.720 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.6270000000000006e-06 2021-03-24T22:03:43.721 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.721 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.721 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:03:43.721 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.380059+0000", 2021-03-24T22:03:43.722 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0043420869999999997 2021-03-24T22:03:43.722 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.722 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.722 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:03:43.723 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.384401+0000", 2021-03-24T22:03:43.723 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.9570999999999999e-05 2021-03-24T22:03:43.723 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.723 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.724 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:03:43.724 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.384431+0000", 2021-03-24T22:03:43.724 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0075091869999999996 2021-03-24T22:03:43.724 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.725 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.725 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:03:43.725 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.391940+0000", 2021-03-24T22:03:43.726 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.1009000000000002e-05 2021-03-24T22:03:43.726 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.726 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.726 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:03:43.727 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:02:44.391961+0000", 2021-03-24T22:03:43.727 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011933084 2021-03-24T22:03:43.727 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:03:43.727 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:03:43.728 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:03:43.728 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.728 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.728 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:5935 7.22 7:454f21f6:::smithi12556470-8:head [write 645600~416265] snapc 1ec=[1ec,1eb,1e8,1e6,1e1,1df,1de,1dc,1d8,1d7,1d4,1cd,1cc,1bf,1bd,1bb,1ae,1a6,19c,199,194,18b,178,172,16e,139] ondisk+write+known_if_redirected e611)", 2021-03-24T22:03:43.728 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:03:01.339063+0000", 2021-03-24T22:03:43.729 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 42.164767505999997, 2021-03-24T22:03:43.729 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012852247000000001, 2021-03-24T22:03:43.729 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:03:43.729 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:03:43.730 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:03:43.730 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:03:43.730 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:03:43.730 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 5935 2021-03-24T22:03:43.731 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.731 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:03:43.731 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.731 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:03:43.731 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:01.339063+0000", 2021-03-24T22:03:43.732 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:03:43.732 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.732 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.732 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:03:43.733 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:01.339063+0000", 2021-03-24T22:03:43.733 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T22:03:43.733 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.733 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.734 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:03:43.734 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:01.339061+0000", 2021-03-24T22:03:43.734 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000240437 2021-03-24T22:03:43.735 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.735 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.735 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:03:43.735 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:01.339301+0000", 2021-03-24T22:03:43.736 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2252999999999999e-05 2021-03-24T22:03:43.736 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.736 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.736 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:03:43.736 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:01.339314+0000", 2021-03-24T22:03:43.737 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.9129999999999997e-06 2021-03-24T22:03:43.737 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.737 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.737 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:03:43.738 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:01.339320+0000", 2021-03-24T22:03:43.738 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000989071 2021-03-24T22:03:43.738 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.738 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.738 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:03:43.739 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:01.340309+0000", 2021-03-24T22:03:43.739 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000116684 2021-03-24T22:03:43.739 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.739 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.740 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:03:43.740 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:01.340426+0000", 2021-03-24T22:03:43.740 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000445859 2021-03-24T22:03:43.740 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.741 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.741 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 4,5", 2021-03-24T22:03:43.741 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:01.340872+0000", 2021-03-24T22:03:43.741 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0089134239999999997 2021-03-24T22:03:43.741 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.742 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.742 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:03:43.742 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:01.349785+0000", 2021-03-24T22:03:43.743 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.7698000000000003e-05 2021-03-24T22:03:43.743 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.743 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.744 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:03:43.744 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:01.349833+0000", 2021-03-24T22:03:43.744 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001948658 2021-03-24T22:03:43.744 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.745 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.745 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:03:43.745 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:01.351782+0000", 2021-03-24T22:03:43.745 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.9661999999999998e-05 2021-03-24T22:03:43.746 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.746 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.746 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:03:43.746 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:01.351851+0000", 2021-03-24T22:03:43.747 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.3526000000000001e-05 2021-03-24T22:03:43.747 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.747 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.747 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:03:43.748 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:01.351915+0000", 2021-03-24T22:03:43.748 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012852247000000001 2021-03-24T22:03:43.748 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:03:43.748 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:03:43.749 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:03:43.749 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.749 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.749 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(osd.7.0:782 7.15 e616/549 7:ad88ec1b:::smithi12556470-47:18d v 616'228, mlcod=604'224)", 2021-03-24T22:03:43.750 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:03:06.332728+0000", 2021-03-24T22:03:43.750 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 37.171102664999999, 2021-03-24T22:03:43.750 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012114936, 2021-03-24T22:03:43.750 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:03:43.751 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:03:43.751 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:03:43.751 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.751 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:03:43.752 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:06.332728+0000", 2021-03-24T22:03:43.752 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:03:43.752 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.752 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.753 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:03:43.753 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:06.332728+0000", 2021-03-24T22:03:43.753 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T22:03:43.753 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.754 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.754 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:03:43.754 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:06.332726+0000", 2021-03-24T22:03:43.754 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7669999999999999e-05 2021-03-24T22:03:43.755 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.755 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.755 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:03:43.755 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:06.332744+0000", 2021-03-24T22:03:43.755 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1216000000000001e-05 2021-03-24T22:03:43.756 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.756 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.756 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:03:43.756 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:06.332755+0000", 2021-03-24T22:03:43.757 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.9129999999999998e-06 2021-03-24T22:03:43.757 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.757 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.757 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:03:43.758 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:06.332765+0000", 2021-03-24T22:03:43.758 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00059538799999999997 2021-03-24T22:03:43.758 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.758 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.759 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:03:43.759 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:06.333361+0000", 2021-03-24T22:03:43.759 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.5976e-05 2021-03-24T22:03:43.759 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.759 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.760 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:03:43.760 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:06.333387+0000", 2021-03-24T22:03:43.760 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011426419 2021-03-24T22:03:43.760 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.761 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.761 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:03:43.761 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:06.344813+0000", 2021-03-24T22:03:43.761 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.9502999999999999e-05 2021-03-24T22:03:43.762 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.762 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.762 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:03:43.763 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:06.344843+0000", 2021-03-24T22:03:43.763 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012114936 2021-03-24T22:03:43.763 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:03:43.763 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:03:43.764 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:03:43.764 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.764 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.764 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:5956 7.1d 7:ba8515f1:::smithi12556470-45:head [copy-from ver 313] snapc 1f0=[1f0,1ef,1ee,1ec,1eb,1e8,1e6,1e1,1df,1de,1dc,1d8,1d7,1d4,1cd,1cc,1bf,1bd,1bb,1a6,19c,199,194,18b,172,16e,139] ondisk+write+known_if_redirected e616)", 2021-03-24T22:03:43.765 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:03:06.341078+0000", 2021-03-24T22:03:43.765 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 37.162751876999998, 2021-03-24T22:03:43.765 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.030990668999999998, 2021-03-24T22:03:43.765 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:03:43.766 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:03:43.766 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:03:43.766 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:03:43.766 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:03:43.767 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 5956 2021-03-24T22:03:43.767 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.767 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:03:43.767 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.767 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:03:43.768 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:06.341078+0000", 2021-03-24T22:03:43.768 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:03:43.768 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.768 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.769 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:03:43.769 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:06.341078+0000", 2021-03-24T22:03:43.769 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999976 2021-03-24T22:03:43.769 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.770 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.770 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:03:43.770 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:06.341076+0000", 2021-03-24T22:03:43.770 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.9799999999999993e-06 2021-03-24T22:03:43.771 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.771 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.771 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:03:43.772 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:06.341086+0000", 2021-03-24T22:03:43.772 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2088e-05 2021-03-24T22:03:43.772 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.772 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.773 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:03:43.773 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:06.341098+0000", 2021-03-24T22:03:43.773 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.3968999999999997e-05 2021-03-24T22:03:43.773 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.774 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.774 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:03:43.774 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:06.341162+0000", 2021-03-24T22:03:43.775 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000110896 2021-03-24T22:03:43.775 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.775 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.775 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:03:43.776 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:06.341273+0000", 2021-03-24T22:03:43.776 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00014733699999999999 2021-03-24T22:03:43.776 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.776 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.777 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:03:43.777 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:06.341420+0000", 2021-03-24T22:03:43.777 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.017018533999999998 2021-03-24T22:03:43.777 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.778 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.778 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,6", 2021-03-24T22:03:43.779 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:06.358439+0000", 2021-03-24T22:03:43.779 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011883279 2021-03-24T22:03:43.780 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.780 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.780 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:03:43.780 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:06.370322+0000", 2021-03-24T22:03:43.780 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0010308990000000001 2021-03-24T22:03:43.781 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.781 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.781 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:03:43.781 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:06.371353+0000", 2021-03-24T22:03:43.782 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00050479699999999997 2021-03-24T22:03:43.782 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.782 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.782 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:03:43.783 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:06.371858+0000", 2021-03-24T22:03:43.783 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4503000000000002e-05 2021-03-24T22:03:43.783 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.783 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.784 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:03:43.784 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:06.371922+0000", 2021-03-24T22:03:43.784 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000146883 2021-03-24T22:03:43.784 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.785 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.785 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:03:43.785 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:06.372069+0000", 2021-03-24T22:03:43.785 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.030990668999999998 2021-03-24T22:03:43.785 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:03:43.786 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:03:43.786 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:03:43.786 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.786 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.787 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:5957 7.1d 7:ba8515f1:::smithi12556470-45:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e616)", 2021-03-24T22:03:43.787 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:03:06.341180+0000", 2021-03-24T22:03:43.787 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 37.162649989000002, 2021-03-24T22:03:43.787 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.031160876000000001, 2021-03-24T22:03:43.788 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:03:43.788 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T22:03:43.788 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:03:43.788 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:03:43.789 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:03:43.789 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 5957 2021-03-24T22:03:43.789 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.789 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:03:43.790 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.790 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:03:43.790 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:06.341180+0000", 2021-03-24T22:03:43.790 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:03:43.791 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.791 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.791 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:03:43.791 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:06.341180+0000", 2021-03-24T22:03:43.792 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T22:03:43.792 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.792 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.792 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:03:43.792 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:06.341179+0000", 2021-03-24T22:03:43.793 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.0769999999999996e-06 2021-03-24T22:03:43.793 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.793 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.793 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:03:43.794 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:06.341185+0000", 2021-03-24T22:03:43.794 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.5521000000000001e-05 2021-03-24T22:03:43.794 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.795 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.795 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:03:43.795 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:06.341221+0000", 2021-03-24T22:03:43.795 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.4190000000000008e-06 2021-03-24T22:03:43.795 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.796 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.796 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:03:43.796 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:06.341230+0000", 2021-03-24T22:03:43.796 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00041625800000000003 2021-03-24T22:03:43.797 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.797 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.797 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:03:43.797 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:06.341646+0000", 2021-03-24T22:03:43.798 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.3479000000000006e-05 2021-03-24T22:03:43.798 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.798 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.798 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-24T22:03:43.799 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:06.341720+0000", 2021-03-24T22:03:43.799 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.025484413000000001 2021-03-24T22:03:43.799 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.800 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.800 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:03:43.800 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:06.367204+0000", 2021-03-24T22:03:43.800 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.7209999999999999e-05 2021-03-24T22:03:43.801 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.801 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.801 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:03:43.801 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:06.367302+0000", 2021-03-24T22:03:43.802 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.004807792 2021-03-24T22:03:43.802 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.802 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.802 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:03:43.802 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:06.372109+0000", 2021-03-24T22:03:43.803 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000113078 2021-03-24T22:03:43.803 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.803 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.803 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:03:43.804 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:06.372222+0000", 2021-03-24T22:03:43.804 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000118724 2021-03-24T22:03:43.804 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.804 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.805 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:03:43.805 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:06.372341+0000", 2021-03-24T22:03:43.805 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.031160876000000001 2021-03-24T22:03:43.805 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:03:43.806 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:03:43.806 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:03:43.806 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.806 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.807 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:6126 7.22 e633/615 7:454f21f6:::smithi12556470-8:head v 633'243, mlcod=628'238)", 2021-03-24T22:03:43.807 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:03:23.489746+0000", 2021-03-24T22:03:43.807 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 20.014084238999999, 2021-03-24T22:03:43.807 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012074781, 2021-03-24T22:03:43.808 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:03:43.808 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:03:43.808 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:03:43.808 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.809 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:03:43.809 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:23.489746+0000", 2021-03-24T22:03:43.809 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:03:43.809 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.810 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.810 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:03:43.810 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:23.489746+0000", 2021-03-24T22:03:43.810 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T22:03:43.811 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.811 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.811 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:03:43.811 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:23.489745+0000", 2021-03-24T22:03:43.812 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0053858719999999999 2021-03-24T22:03:43.812 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.812 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.812 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:03:43.813 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:23.495131+0000", 2021-03-24T22:03:43.813 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9454e-05 2021-03-24T22:03:43.813 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.813 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.814 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:03:43.814 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:23.495150+0000", 2021-03-24T22:03:43.814 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.737e-06 2021-03-24T22:03:43.814 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.815 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.815 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:03:43.815 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:23.495157+0000", 2021-03-24T22:03:43.815 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6013999999999995e-05 2021-03-24T22:03:43.816 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.816 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.816 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:03:43.816 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:23.495223+0000", 2021-03-24T22:03:43.817 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.3664999999999997e-05 2021-03-24T22:03:43.817 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.818 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.818 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:03:43.818 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:23.495256+0000", 2021-03-24T22:03:43.819 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0065266450000000002 2021-03-24T22:03:43.819 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.819 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.819 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:03:43.820 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:23.501783+0000", 2021-03-24T22:03:43.820 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.7675000000000001e-05 2021-03-24T22:03:43.820 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.821 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.821 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:03:43.821 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:23.501821+0000", 2021-03-24T22:03:43.821 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012074781 2021-03-24T22:03:43.822 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:03:43.822 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:03:43.822 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:03:43.823 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.823 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.823 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(osd.6.0:506 7.d e637/537 7:b2218182:::smithi12556470-21:1e8 v 637'568, mlcod=635'564)", 2021-03-24T22:03:43.824 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:03:27.565355+0000", 2021-03-24T22:03:43.824 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 15.938475567999999, 2021-03-24T22:03:43.824 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.023227808999999999, 2021-03-24T22:03:43.824 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:03:43.825 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:03:43.825 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:03:43.825 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.826 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:03:43.826 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:27.565355+0000", 2021-03-24T22:03:43.826 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:03:43.826 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.827 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.827 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:03:43.827 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:27.565355+0000", 2021-03-24T22:03:43.828 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999971 2021-03-24T22:03:43.828 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.828 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.828 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:03:43.829 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:27.565352+0000", 2021-03-24T22:03:43.829 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5926999999999998e-05 2021-03-24T22:03:43.829 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.829 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.830 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:03:43.830 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:27.565368+0000", 2021-03-24T22:03:43.830 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.6967000000000001e-05 2021-03-24T22:03:43.831 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.831 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.832 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:03:43.832 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:27.565395+0000", 2021-03-24T22:03:43.832 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1131000000000001e-05 2021-03-24T22:03:43.833 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.833 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.833 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:03:43.834 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:27.565406+0000", 2021-03-24T22:03:43.834 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.1031000000000005e-05 2021-03-24T22:03:43.834 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.835 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.835 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:03:43.835 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:27.565477+0000", 2021-03-24T22:03:43.835 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.7407999999999999e-05 2021-03-24T22:03:43.836 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.836 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.836 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:03:43.836 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:27.565504+0000", 2021-03-24T22:03:43.837 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.022983337 2021-03-24T22:03:43.837 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.837 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.837 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:03:43.838 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:27.588488+0000", 2021-03-24T22:03:43.838 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.4949000000000002e-05 2021-03-24T22:03:43.838 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.838 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.839 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:03:43.839 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:27.588583+0000", 2021-03-24T22:03:43.839 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.023227808999999999 2021-03-24T22:03:43.839 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:03:43.840 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:03:43.840 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:03:43.840 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.840 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.841 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:6171 7.1d 7:ba8515f1:::smithi12556470-45:head [copy-from ver 447] snapc 206=[206,204,202,201,1ff,1fd,1f8,1f3,1f0,1ef,1ee,1ec,1e1,1df,1de,1dc,1d8,1d4,1cd,1bf,1bd,199,172,16e] ondisk+write+known_if_redirected e637)", 2021-03-24T22:03:43.841 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:03:27.708527+0000", 2021-03-24T22:03:43.841 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 15.795303321, 2021-03-24T22:03:43.841 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.020039169999999999, 2021-03-24T22:03:43.842 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:03:43.842 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:03:43.842 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:03:43.842 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:03:43.843 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:03:43.843 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 6171 2021-03-24T22:03:43.843 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.843 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:03:43.844 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.844 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:03:43.844 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:27.708527+0000", 2021-03-24T22:03:43.845 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:03:43.845 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.845 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.845 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:03:43.846 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:27.708527+0000", 2021-03-24T22:03:43.846 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999948 2021-03-24T22:03:43.846 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.846 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.847 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:03:43.847 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:27.708522+0000", 2021-03-24T22:03:43.847 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0809999999999999e-05 2021-03-24T22:03:43.847 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.848 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.848 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:03:43.848 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:27.708543+0000", 2021-03-24T22:03:43.848 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.5670999999999999e-05 2021-03-24T22:03:43.849 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.849 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.849 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:03:43.849 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:27.708568+0000", 2021-03-24T22:03:43.850 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1715999999999999e-05 2021-03-24T22:03:43.850 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.850 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.850 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:03:43.851 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:27.708580+0000", 2021-03-24T22:03:43.851 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.3381999999999996e-05 2021-03-24T22:03:43.851 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.851 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.852 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:03:43.852 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:27.708653+0000", 2021-03-24T22:03:43.852 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000111616 2021-03-24T22:03:43.852 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.853 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.853 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:03:43.853 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:27.708765+0000", 2021-03-24T22:03:43.853 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0071500579999999999 2021-03-24T22:03:43.854 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.854 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.854 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,6", 2021-03-24T22:03:43.854 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:27.715915+0000", 2021-03-24T22:03:43.855 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0077652850000000002 2021-03-24T22:03:43.855 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.855 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.855 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:03:43.856 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:27.723680+0000", 2021-03-24T22:03:43.856 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0038801249999999999 2021-03-24T22:03:43.856 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.856 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.857 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:03:43.857 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:27.727560+0000", 2021-03-24T22:03:43.857 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00084256299999999995 2021-03-24T22:03:43.857 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.858 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.858 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:03:43.858 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:27.728403+0000", 2021-03-24T22:03:43.858 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2683999999999999e-05 2021-03-24T22:03:43.859 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.859 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.859 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:03:43.859 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:27.728456+0000", 2021-03-24T22:03:43.860 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00011050800000000001 2021-03-24T22:03:43.860 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.860 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.861 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:03:43.861 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:27.728566+0000", 2021-03-24T22:03:43.861 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.020039169999999999 2021-03-24T22:03:43.861 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:03:43.862 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:03:43.862 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:03:43.862 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.863 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.863 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:6172 7.1d 7:ba8515f1:::smithi12556470-45:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e637)", 2021-03-24T22:03:43.863 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:03:27.708620+0000", 2021-03-24T22:03:43.863 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 15.795210515999999, 2021-03-24T22:03:43.864 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.020351724000000002, 2021-03-24T22:03:43.864 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:03:43.864 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T22:03:43.864 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:03:43.865 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:03:43.865 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:03:43.865 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 6172 2021-03-24T22:03:43.865 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.866 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:03:43.866 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.866 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:03:43.866 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:27.708620+0000", 2021-03-24T22:03:43.867 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:03:43.867 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.867 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.867 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:03:43.868 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:27.708620+0000", 2021-03-24T22:03:43.868 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T22:03:43.868 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.868 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.869 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:03:43.869 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:27.708619+0000", 2021-03-24T22:03:43.869 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.028e-05 2021-03-24T22:03:43.869 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.870 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.870 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:03:43.870 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:27.708629+0000", 2021-03-24T22:03:43.871 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.1269999999999992e-06 2021-03-24T22:03:43.871 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.871 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.871 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:03:43.872 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:27.708638+0000", 2021-03-24T22:03:43.872 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.8139999999999997e-06 2021-03-24T22:03:43.872 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.872 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.873 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:03:43.873 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:27.708644+0000", 2021-03-24T22:03:43.873 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00026915199999999999 2021-03-24T22:03:43.873 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.874 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.874 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:03:43.874 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:27.708913+0000", 2021-03-24T22:03:43.874 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000120556 2021-03-24T22:03:43.875 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.875 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.875 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-24T22:03:43.875 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:27.709034+0000", 2021-03-24T22:03:43.876 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.014245223 2021-03-24T22:03:43.876 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.876 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.876 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:03:43.877 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:27.723279+0000", 2021-03-24T22:03:43.877 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00011618 2021-03-24T22:03:43.877 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.877 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.877 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:03:43.878 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:27.723395+0000", 2021-03-24T22:03:43.878 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.005242114 2021-03-24T22:03:43.878 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.879 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.879 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:03:43.879 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:27.728637+0000", 2021-03-24T22:03:43.879 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00017118 2021-03-24T22:03:43.879 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.880 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.880 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:03:43.880 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:27.728808+0000", 2021-03-24T22:03:43.880 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000163272 2021-03-24T22:03:43.880 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:03:43.881 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:03:43.881 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:03:43.881 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:27.728971+0000", 2021-03-24T22:03:43.881 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.020351724000000002 2021-03-24T22:03:43.882 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:03:43.882 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:03:43.882 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:03:43.883 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:03:43.883 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:03:43.883 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-24T22:03:43.884 INFO:tasks.rados.rados.0.smithi125.stdout:2484: writing smithi12556470-35 from 1408890 to 2001521 tid 2 2021-03-24T22:03:43.884 INFO:tasks.rados.rados.0.smithi125.stdout:2485: snap_create 2021-03-24T22:03:43.884 INFO:tasks.rados.rados.0.smithi125.stdout:2478: expect (ObjNum 691 snap 218 seq_num 691) 2021-03-24T22:03:43.885 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T22:03:43.886 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T22:03:43.886 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-24T22:03:43.886 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-24T22:03:43.887 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-24T22:03:43.887 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-24T22:03:43.887 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-24T22:03:43.887 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-24T22:03:43.888 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-24T22:03:43.888 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-24T22:03:43.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:43.890 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:43.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:03:43.891 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:43 smithi125 conmon[19936]: cluster 2021-03-24T22:03:42.250608+0000 osd.3 (osd.3) 133 : cluster [DBG] 7.14 scrub ok 2021-03-24T22:03:43.891 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:43 smithi125 conmon[19936]: 2021-03-24T22:03:43.891 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:43 smithi125 conmon[19936]: audit 2021-03-24T22:03:43.326626+0000 mon.c (mon.2) 98 : audit [DBG] from='client.? v1:172.21.15.125:0/403993369' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:03:43.892 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:43 smithi125 conmon[24927]: cluster 2021-03-24T22:03:43.892 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:43 smithi125 conmon[24927]: 2021-03-24T22:03:43.893 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:43 smithi125 conmon[24927]: 2021 2021-03-24T22:03:43.893 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:43 smithi125 conmon[24927]: - 2021-03-24T22:03:43.893 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:43 smithi125 conmon[24927]: 03 2021-03-24T22:03:43.893 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:43 smithi125 conmon[24927]: - 2021-03-24T22:03:43.894 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:43 smithi125 conmon[24927]: 24 2021-03-24T22:03:43.894 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:43 smithi125 conmon[24927]: T 2021-03-24T22:03:43.894 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:43 smithi125 conmon[24927]: 22 2021-03-24T22:03:43.894 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:43 smithi125 conmon[24927]: : 2021-03-24T22:03:43.895 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:43 smithi125 conmon[24927]: 03 2021-03-24T22:03:43.895 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:43 smithi125 conmon[24927]: : 2021-03-24T22:03:43.895 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:43 smithi125 conmon[24927]: 42 2021-03-24T22:03:43.895 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:43 smithi125 conmon[24927]: . 2021-03-24T22:03:43.896 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:43 smithi125 conmon[24927]: 250608 2021-03-24T22:03:43.896 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:43 smithi125 conmon[24927]: +0000 2021-03-24T22:03:43.896 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:43 smithi125 conmon[24927]: 2021-03-24T22:03:43.896 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:43 smithi125 conmon[24927]: osd.3 2021-03-24T22:03:43.897 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:43 smithi125 conmon[24927]: ( 2021-03-24T22:03:43.897 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:43 smithi125 conmon[24927]: osd 2021-03-24T22:03:43.897 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:43 smithi125 conmon[24927]: . 2021-03-24T22:03:43.897 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:43 smithi125 conmon[24927]: 3 2021-03-24T22:03:43.898 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:43 smithi125 conmon[24927]: ) 2021-03-24T22:03:43.898 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:43 smithi125 conmon[24927]: 133 2021-03-24T22:03:43.898 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:43 smithi125 conmon[24927]: : 2021-03-24T22:03:43.898 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:43 smithi125 conmon[24927]: cluster 2021-03-24T22:03:43.899 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:43 smithi125 conmon[24927]: 2021-03-24T22:03:43.899 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:43 smithi125 conmon[24927]: [DBG] 2021-03-24T22:03:43.899 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:43 smithi125 conmon[24927]: 2021-03-24T22:03:43.900 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:43 smithi125 conmon[24927]: 7.14 scrub ok 2021-03-24T22:03:43.900 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:43 smithi125 conmon[24927]: 2021-03-24T22:03:43.900 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:43 smithi125 conmon[24927]: audit 2021-03-24T22:03:43.900 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:43 smithi125 conmon[24927]: 2021-03-24T22:03:43.900 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:43 smithi125 conmon[24927]: 2021-03-24T22:03: 2021-03-24T22:03:43.901 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:43 smithi125 conmon[24927]: 43.326626+0000 mon.c (mon.2) 98 : audit [DBG] from='client.? v1:172.21.15.125:0/403993369' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:03:43.901 INFO:tasks.rados.rados.0.smithi125.stdout:2480: finishing write tid 2 to smithi12556470-10 2021-03-24T22:03:43.902 INFO:tasks.rados.rados.0.smithi125.stdout:2480: finishing write tid 3 to smithi12556470-10 2021-03-24T22:03:43.902 INFO:tasks.rados.rados.0.smithi125.stdout:2480: finishing write tid 4 to smithi12556470-10 2021-03-24T22:03:43.902 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 10 v 997 (ObjNum 859 snap 279 seq_num 859) dirty exists 2021-03-24T22:03:43.903 INFO:tasks.rados.rados.0.smithi125.stdout:2480: left oid 10 (ObjNum 859 snap 279 seq_num 859) 2021-03-24T22:03:43.903 INFO:tasks.rados.rados.0.smithi125.stdout:2481: finishing write tid 1 to smithi12556470-5 2021-03-24T22:03:43.903 INFO:tasks.rados.rados.0.smithi125.stdout:2479: finishing copy_from to smithi12556470-30 2021-03-24T22:03:43.903 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 30 v 503 (ObjNum 736 snap 235 seq_num 736) dirty exists 2021-03-24T22:03:43.903 INFO:tasks.rados.rados.0.smithi125.stdout:2479: finishing copy_from racing read to smithi12556470-30 2021-03-24T22:03:43.904 INFO:tasks.rados.rados.0.smithi125.stdout:2481: finishing write tid 2 to smithi12556470-5 2021-03-24T22:03:43.904 INFO:tasks.rados.rados.0.smithi125.stdout:2481: finishing write tid 3 to smithi12556470-5 2021-03-24T22:03:43.904 INFO:tasks.rados.rados.0.smithi125.stdout:2481: finishing write tid 4 to smithi12556470-5 2021-03-24T22:03:43.904 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 5 v 820 (ObjNum 860 snap 279 seq_num 860) dirty exists 2021-03-24T22:03:43.905 INFO:tasks.rados.rados.0.smithi125.stdout:2481: left oid 5 (ObjNum 860 snap 279 seq_num 860) 2021-03-24T22:03:43.905 INFO:tasks.rados.rados.0.smithi125.stdout:2482: finishing write tid 1 to smithi12556470-18 2021-03-24T22:03:43.905 INFO:tasks.rados.rados.0.smithi125.stdout:2482: finishing write tid 2 to smithi12556470-18 2021-03-24T22:03:43.905 INFO:tasks.rados.rados.0.smithi125.stdout:2482: finishing write tid 3 to smithi12556470-18 2021-03-24T22:03:43.906 INFO:tasks.rados.rados.0.smithi125.stdout:2482: finishing write tid 4 to smithi12556470-18 2021-03-24T22:03:43.906 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 18 v 702 (ObjNum 861 snap 279 seq_num 861) dirty exists 2021-03-24T22:03:43.906 INFO:tasks.rados.rados.0.smithi125.stdout:2482: left oid 18 (ObjNum 861 snap 279 seq_num 861) 2021-03-24T22:03:43.906 INFO:tasks.rados.rados.0.smithi125.stdout:2484: finishing write tid 1 to smithi12556470-35 2021-03-24T22:03:43.907 INFO:tasks.rados.rados.0.smithi125.stdout:2484: finishing write tid 2 to smithi12556470-35 2021-03-24T22:03:43.907 INFO:tasks.rados.rados.0.smithi125.stdout:2484: finishing write tid 3 to smithi12556470-35 2021-03-24T22:03:43.907 INFO:tasks.rados.rados.0.smithi125.stdout:2484: finishing write tid 4 to smithi12556470-35 2021-03-24T22:03:43.907 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 35 v 226 (ObjNum 862 snap 279 seq_num 862) dirty exists 2021-03-24T22:03:43.908 INFO:tasks.rados.rados.0.smithi125.stdout:2484: left oid 35 (ObjNum 862 snap 279 seq_num 862) 2021-03-24T22:03:43.908 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:43 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:03:43.908 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:43 smithi142 conmon[20125]: 22:03:42.250608+0000 osd.3 (osd.3) 133 : cluster [DBG] 7.14 scrub ok 2021-03-24T22:03:43.909 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:43 smithi142 conmon[20125]: audit 2021-03-24T22 2021-03-24T22:03:43.909 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:43 smithi142 conmon[20125]: :03:43.326626+0000 mon.c (mon 2021-03-24T22:03:43.909 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:43 smithi142 conmon[20125]: .2) 98 : audit [DBG] from='client.? v1:172.21.15.125:0/403993369' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:03:43.992 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:44.050 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:44.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:44.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:03:44.155 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:44.208 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:44.212 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:43 smithi125 conmon[19936]: debug 2021-03-24T22:03:43.952+0000 7f2118128700 -1 mon.a@0(leader).osd e653 definitely_dead 0 2021-03-24T22:03:44.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:44.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:03:44.314 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:44.371 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:44.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:44.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:03:44.468 INFO:tasks.rados.rados.0.smithi125.stdout:2478: done (7 left) 2021-03-24T22:03:44.468 INFO:tasks.rados.rados.0.smithi125.stdout:2479: done (6 left) 2021-03-24T22:03:44.468 INFO:tasks.rados.rados.0.smithi125.stdout:2480: done (5 left) 2021-03-24T22:03:44.469 INFO:tasks.rados.rados.0.smithi125.stdout:2481: done (4 left) 2021-03-24T22:03:44.469 INFO:tasks.rados.rados.0.smithi125.stdout:2482: done (3 left) 2021-03-24T22:03:44.469 INFO:tasks.rados.rados.0.smithi125.stdout:2483: done (2 left) 2021-03-24T22:03:44.469 INFO:tasks.rados.rados.0.smithi125.stdout:2484: done (1 left) 2021-03-24T22:03:44.470 INFO:tasks.rados.rados.0.smithi125.stdout:2485: done (0 left) 2021-03-24T22:03:44.470 INFO:tasks.rados.rados.0.smithi125.stdout:2486: snap_create 2021-03-24T22:03:44.476 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:44.479 INFO:teuthology.orchestra.run.smithi125.stderr:marked in osd.1. 2021-03-24T22:03:44.491 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:44 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:03:44.492 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:44 smithi125 conmon[24927]: 03:42.740375+0000 osd.5 (osd.5) 36 : cluster [DBG] 5.16 scrub ok 2021-03-24T22:03:44.492 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:44 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:03:44.492 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:44 smithi125 conmon[24927]: 24T22:03:42.935303+0000 mgr.y 2021-03-24T22:03:44.492 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:44 smithi125 conmon[24927]: (mgr.14144) 600 : cluster [DBG] pgmap v1141: 168 pgs: 1 peering, 4 active+remapped+backfill_toofull, 163 active+clean; 493 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 2.5 MiB/s rd, 1.4 MiB/s wr, 13 op/s; 84/1689 objects misplaced (4.973%); 15 MiB/s, 28 objects/s recovering 2021-03-24T22:03:44.493 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:44 smithi125 conmon[24927]: 2021-03-24T22:03:44.493 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:44 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:03:44.493 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:44 smithi125 conmon[24927]: 24T22:03:43.233866+0000 osd.0 (osd.0) 128 : cluster [DBG] 7.18 scrub ok 2021-03-24T22:03:44.494 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:44 smithi125 conmon[24927]: 2021-03-24T22:03:44.494 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:44 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:03:44.494 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:44 smithi125 conmon[24927]: 24T22:03:43.239977+0000 osd.3 (osd.3) 134 : 2021-03-24T22:03:44.494 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:44 smithi125 conmon[24927]: cluster [DBG] 3.22 scrub ok 2021-03-24T22:03:44.495 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:44 smithi125 conmon[24927]: cluster 2021-03-24T22:03:43 2021-03-24T22:03:44.495 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:44 smithi125 conmon[24927]: .285365+0000 osd.7 (osd.7) 106 : 2021-03-24T22:03:44.495 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:44 smithi125 conmon[24927]: cluster [DBG] 1.0 scrub ok 2021-03-24T22:03:44.495 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:44 smithi125 conmon[24927]: cluster 2021-03-24T22 2021-03-24T22:03:44.495 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:44 smithi125 conmon[24927]: :03:43.449405+0000 mon.a (mon. 2021-03-24T22:03:44.496 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:44 smithi125 conmon[24927]: 0) 1733 : cluster [DBG] osdmap e653: 8 total, 8 up, 7 in 2021-03-24T22:03:44.496 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:44 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T22:03:44.496 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:44 smithi125 conmon[24927]: 24T22:03:43.952898+0000 mon.a 2021-03-24T22:03:44.496 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:44 smithi125 conmon[24927]: (mon.0) 1734 : audit [DBG] from='client.? v1:172.21.15.125:0/1429638974' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2021-03-24T22:03:44.497 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:44 smithi125 conmon[19936]: cluster 2021-03-24T22:03: 2021-03-24T22:03:44.497 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:44 smithi125 conmon[19936]: 42.740375+0000 osd.5 (osd.5) 36 : cluster [DBG] 5.16 scrub ok 2021-03-24T22:03:44.497 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:44 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:03:44.497 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:44 smithi125 conmon[19936]: 24T22:03:42.935303+0000 mgr.y (mgr.14144) 600 : cluster 2021-03-24T22:03:44.498 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:44 smithi125 conmon[19936]: [DBG] pgmap v1141: 168 pgs: 1 peering, 4 active+remapped+backfill_toofull, 163 active+clean; 493 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 2.5 MiB/s rd, 1.4 MiB/s wr, 13 op/s; 84/1689 objects misplaced (4.973%); 15 MiB/s, 28 objects/s recovering 2021-03-24T22:03:44.498 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:44 smithi125 conmon[19936]: cluster 2021-03-24T22:03: 2021-03-24T22:03:44.498 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:44 smithi125 conmon[19936]: 43.233866+0000 osd.0 (osd.0) 128 2021-03-24T22:03:44.498 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:44 smithi125 conmon[19936]: : cluster [DBG] 7.18 scrub ok 2021-03-24T22:03:44.499 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:44 smithi125 conmon[19936]: cluster 2021-03-24T22 2021-03-24T22:03:44.499 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:44 smithi125 conmon[19936]: :03:43.239977+0000 osd.3 (osd. 2021-03-24T22:03:44.499 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:44 smithi125 conmon[19936]: 3) 134 : cluster [DBG] 3.22 scrub ok 2021-03-24T22:03:44.500 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:44 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:03:44.500 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:44 smithi125 conmon[19936]: 03-24T22:03:43.285365+0000 2021-03-24T22:03:44.500 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:44 smithi125 conmon[19936]: osd.7 (osd.7) 106 : cluster [DBG] 2021-03-24T22:03:44.501 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:44 smithi125 conmon[19936]: 1.0 scrub ok 2021-03-24T22:03:44.501 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:44 smithi125 conmon[19936]: cluster 2021-03-24T22:03 2021-03-24T22:03:44.501 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:44 smithi125 conmon[19936]: :43.449405+0000 mon.a (mon.0) 2021-03-24T22:03:44.501 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:44 smithi125 conmon[19936]: 1733 : cluster [DBG] osdmap e653: 8 total, 8 up, 7 in 2021-03-24T22:03:44.501 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:44 smithi125 conmon[19936]: audit 2021- 2021-03-24T22:03:44.502 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:44 smithi125 conmon[19936]: 03-24T22:03:43 2021-03-24T22:03:44.502 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:44 smithi125 conmon[19936]: .952898+0000 mon.a (mon.0 2021-03-24T22:03:44.502 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:44 smithi125 conmon[19936]: ) 1734 : audit [DBG] from='client.? v1:172.21.15.125:0/1429638974' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2021-03-24T22:03:44.525 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:44.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:44.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:03:44.630 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:44.686 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:44.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:44.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:03:44.793 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:44.853 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:44.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:44.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:03:44.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:44 smithi142 conmon[20125]: cluster 2021-03-24T22:03:42.740375+0000 osd.5 (osd.5) 36 : 2021-03-24T22:03:44.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:44 smithi142 conmon[20125]: cluster [DBG] 5.16 scrub ok 2021-03-24T22:03:44.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:44 smithi142 conmon[20125]: cluster 2021-03-24T22:03:42.935303+0000 mgr.y (mgr.14144) 600 : cluster [DBG] pgmap v1141: 168 pgs: 1 peering, 4 active+remapped+backfill_toofull, 163 active+clean; 493 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 2.5 MiB/s rd, 1.4 MiB/s wr, 13 op/s; 84/1689 objects misplaced (4.973%); 15 MiB/s, 28 objects/s recovering 2021-03-24T22:03:44.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:44 smithi142 conmon[20125]: 2021-03-24T22:03:44.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:44 smithi142 conmon[20125]: cluster 2021-03-24T22:03:43.233866+0000 osd.0 (osd.0) 128 : cluster [DBG] 7.18 scrub ok 2021-03-24T22:03:44.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:44 smithi142 conmon[20125]: cluster 2021-03-24T22:03:43.239977+0000 osd.3 (osd. 2021-03-24T22:03:44.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:44 smithi142 conmon[20125]: 3) 134 : cluster [DBG] 3.22 scrub ok 2021-03-24T22:03:44.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:44 smithi142 conmon[20125]: cluster 2021-03-24T22:03:44.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:44 smithi142 conmon[20125]: 2021-03-24T22:03:43.285365+0000 osd.7 (osd.7) 106 : 2021-03-24T22:03:44.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:44 smithi142 conmon[20125]: cluster [DBG] 1.0 scrub ok 2021-03-24T22:03:44.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:44 smithi142 conmon[20125]: cluster 2021-03-24T22:03:44.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:44 smithi142 conmon[20125]: 2021-03-24T22:03:43.449405+0000 mon.a (mon.0) 1733 2021-03-24T22:03:44.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:44 smithi142 conmon[20125]: : cluster [DBG] osdmap e653: 8 total, 8 up, 7 in 2021-03-24T22:03:44.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:44 smithi142 conmon[20125]: audit 2021-03-24T22:03:44.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:44 smithi142 conmon[20125]: 2021-03-24T22:03:43.952898+0000 mon.a (mon.0) 2021-03-24T22:03:44.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:44 smithi142 conmon[20125]: 1734 : audit [DBG] from='client.? v1:172.21.15.125:0/1429638974' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2021-03-24T22:03:44.959 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:45.017 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:45.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:45.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:03:45.124 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:45.189 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:45.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:45.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:03:45.295 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:45.352 INFO:tasks.rados.rados.0.smithi125.stdout:2486: done (0 left) 2021-03-24T22:03:45.353 INFO:tasks.rados.rados.0.smithi125.stdout:2487: read oid 14 snap 246 2021-03-24T22:03:45.353 INFO:tasks.rados.rados.0.smithi125.stdout:2487: expect (ObjNum 625 snap 192 seq_num 625) 2021-03-24T22:03:45.353 INFO:tasks.rados.rados.0.smithi125.stdout:2488: read oid 35 snap -1 2021-03-24T22:03:45.353 INFO:tasks.rados.rados.0.smithi125.stdout:2488: expect (ObjNum 862 snap 279 seq_num 862) 2021-03-24T22:03:45.354 INFO:tasks.rados.rados.0.smithi125.stdout:2489: copy_from oid 2 from oid 23 current snap is 281 2021-03-24T22:03:45.354 INFO:tasks.rados.rados.0.smithi125.stdout:2490: write oid 6 current snap is 281 2021-03-24T22:03:45.354 INFO:tasks.rados.rados.0.smithi125.stdout:2490: seq_num 863 ranges {504441=487776,1467372=736289} 2021-03-24T22:03:45.358 INFO:tasks.rados.rados.0.smithi125.stdout:2490: writing smithi12556470-6 from 504441 to 992217 tid 1 2021-03-24T22:03:45.366 INFO:tasks.rados.rados.0.smithi125.stdout:2490: writing smithi12556470-6 from 1467372 to 2203661 tid 2 2021-03-24T22:03:45.366 INFO:tasks.rados.rados.0.smithi125.stdout:2491: rollback oid 3 current snap is 281 2021-03-24T22:03:45.366 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 3 to 275 2021-03-24T22:03:45.366 INFO:tasks.rados.rados.0.smithi125.stdout:2492: write oid 41 current snap is 281 2021-03-24T22:03:45.367 INFO:tasks.rados.rados.0.smithi125.stdout:2492: seq_num 864 ranges {521248=563025,1809501=596300} 2021-03-24T22:03:45.368 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:45.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:45.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:03:45.371 INFO:tasks.rados.rados.0.smithi125.stdout:2492: writing smithi12556470-41 from 521248 to 1084273 tid 1 2021-03-24T22:03:45.379 INFO:tasks.rados.rados.0.smithi125.stdout:2492: writing smithi12556470-41 from 1809501 to 2405801 tid 2 2021-03-24T22:03:45.379 INFO:tasks.rados.rados.0.smithi125.stdout:2493: snap_remove snap 265 2021-03-24T22:03:45.470 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:45.516 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:45.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:45.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:03:45.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:45 smithi142 conmon[20125]: audit 2021-03-24T22:03:44.462394+0000 2021-03-24T22:03:45.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:45 smithi142 conmon[20125]: mon.a (mon.0) 1735 : audit [INF] from='client.? v1:172.21.15.125:0/1429638974' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2021-03-24T22:03:45.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:45 smithi142 conmon[20125]: cluster 2021-03-24T22:03:44.462513+0000 mon.a (mon.0) 1736 : cluster [DBG] 2021-03-24T22:03:45.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:45 smithi142 conmon[20125]: osdmap e654: 8 total, 8 up, 8 in 2021-03-24T22:03:45.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:45 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:03:45.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:45 smithi142 conmon[20125]: -24T22:03:44.560027+0000 osd.2 ( 2021-03-24T22:03:45.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:45 smithi142 conmon[20125]: osd.2) 102 : cluster [DBG] 6.8 scrub ok 2021-03-24T22:03:45.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:45 smithi142 conmon[20125]: audit 2021-03-24T22:03:45.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:45 smithi142 conmon[20125]: 2021-03-24T22:03: 2021-03-24T22:03:45.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:45 smithi142 conmon[20125]: 44.827717+0000 mon.a (mon.0) 1737 2021-03-24T22:03:45.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:45 smithi142 conmon[20125]: : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:03:45.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:45 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:03:45.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:45 smithi142 conmon[20125]: -24T22:03:45. 2021-03-24T22:03:45.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:45 smithi142 conmon[20125]: 222729+0000 osd.3 (osd.3) 135 : cluster 2021-03-24T22:03:45.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:45 smithi142 conmon[20125]: [DBG] 6.2 scrub ok 2021-03-24T22:03:45.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:45 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:03:45.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:45 smithi142 conmon[20125]: 24T22:03:45.335068 2021-03-24T22:03:45.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:45 smithi142 conmon[20125]: +0000 mon.a (mon.0) 1738 : cluster 2021-03-24T22:03:45.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:45 smithi142 conmon[20125]: [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:03:45.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:45 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:03:45.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:45 smithi142 conmon[20125]: T22:03:45.346928+0000 mon.a 2021-03-24T22:03:45.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:45 smithi142 conmon[20125]: (mon.0) 1739 : cluster [DBG] osdmap e655: 8 total, 8 up, 8 in 2021-03-24T22:03:45.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:45 smithi142 conmon[20125]: 2021-03-24T22:03:45.623 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:45.671 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:45.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:45.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:03:45.777 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:45.828 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:45.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:45.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:03:45.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:45 smithi125 conmon[19936]: audit 2021-03-24T22:03:45.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:45 smithi125 conmon[19936]: 2021-03-24T 2021-03-24T22:03:45.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:45 smithi125 conmon[19936]: 22:03:44.462394+0000 mon.a (mon.0) 1735 : audit [INF] from='client.? v1:172.21.15.125:0/1429638974' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2021-03-24T22:03:45.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:45 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:03:45.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:45 smithi125 conmon[19936]: -03-24T22:03:44.462513+0000 mon.a (mon. 2021-03-24T22:03:45.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:45 smithi125 conmon[19936]: 0) 1736 : cluster [DBG] osdmap e654: 8 total, 8 up, 8 in 2021-03-24T22:03:45.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:45 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:03:45.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:45 smithi125 conmon[19936]: 03-24T22:03:44.560027+0000 osd.2 (osd 2021-03-24T22:03:45.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:45 smithi125 conmon[19936]: .2) 102 : cluster [DBG] 6.8 scrub ok 2021-03-24T22:03:45.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:45 smithi125 conmon[19936]: audit 2021- 2021-03-24T22:03:45.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:45 smithi125 conmon[19936]: 03-24T22:03:44.827717+0000 mon.a ( 2021-03-24T22:03:45.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:45 smithi125 conmon[19936]: mon.0) 1737 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:03:45.915 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:45 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:03:45.915 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:45 smithi125 conmon[19936]: 03-24T22:03:45.222729+0000 osd.3 ( 2021-03-24T22:03:45.915 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:45 smithi125 conmon[19936]: osd.3) 135 : cluster [DBG] 6.2 scrub ok 2021-03-24T22:03:45.915 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:45 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T22:03:45.916 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:45 smithi125 conmon[19936]: -24T22:03:45.335068+0000 mon.a ( 2021-03-24T22:03:45.916 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:45 smithi125 conmon[19936]: mon.0) 1738 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:03:45.916 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:45 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:03:45.916 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:45 smithi125 conmon[19936]: T22:03:45.346928+0000 mon.a (mon.0 2021-03-24T22:03:45.916 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:45 smithi125 conmon[19936]: ) 1739 : cluster [DBG] osdmap e655: 8 total, 8 up, 8 in 2021-03-24T22:03:45.917 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:45 smithi125 conmon[24927]: audit 2021-03-24T22:03:44 2021-03-24T22:03:45.917 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:45 smithi125 conmon[24927]: .462394+0000 mon.a (mon.0) 1735 : audit [INF] from='client.? v1:172.21.15.125:0/1429638974' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2021-03-24T22:03:45.917 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:45 smithi125 conmon[24927]: cluster 2021-03-24T22:03:45.917 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:45 smithi125 conmon[24927]: 2021-03-24T22:03:44.462513+0000 mon.a (mon.0) 1736 : cluster 2021-03-24T22:03:45.918 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:45 smithi125 conmon[24927]: [DBG] osdmap e654: 8 total, 8 up, 8 in 2021-03-24T22:03:45.918 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:45 smithi125 conmon[24927]: cluster 2021-03-24T22:03 2021-03-24T22:03:45.918 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:45 smithi125 conmon[24927]: :44.560027+0000 osd.2 (osd.2) 102 2021-03-24T22:03:45.918 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:45 smithi125 conmon[24927]: : cluster [DBG] 6.8 scrub ok 2021-03-24T22:03:45.919 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:45 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T22:03:45.919 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:45 smithi125 conmon[24927]: 24T22:03:44.827717+0000 mon.a ( 2021-03-24T22:03:45.919 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:45 smithi125 conmon[24927]: mon.0) 1737 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:03:45.919 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:45 smithi125 conmon[24927]: 2021-03-24T22:03:45.919 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:45 smithi125 conmon[24927]: cluster 2021-03-24T22:03:45. 2021-03-24T22:03:45.920 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:45 smithi125 conmon[24927]: 222729+0000 osd.3 (osd.3) 135 : cluster 2021-03-24T22:03:45.920 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:45 smithi125 conmon[24927]: [DBG] 6.2 scrub ok 2021-03-24T22:03:45.920 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:45 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T22:03:45.920 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:45 smithi125 conmon[24927]: T22:03:45.335068+0000 mon.a (mon 2021-03-24T22:03:45.920 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:45 smithi125 conmon[24927]: .0) 1738 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:03:45.921 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:45 smithi125 conmon[24927]: 2021-03-24T22:03:45.921 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:45 smithi125 conmon[24927]: cluster 2021-03-24T22:03:45.921 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:45 smithi125 conmon[24927]: 2021-03-24T22:03:45.346928+0000 mon.a ( 2021-03-24T22:03:45.921 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:45 smithi125 conmon[24927]: mon.0) 1739 : cluster [DBG] 2021-03-24T22:03:45.922 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:45 smithi125 conmon[24927]: osdmap e655: 8 total, 8 up, 8 in 2021-03-24T22:03:45.932 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:45.975 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:45.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:45.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:03:46.079 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:46.132 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:46.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:46.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:03:46.236 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:46.282 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:46.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:46.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:03:46.351 INFO:tasks.rados.rados.0.smithi125.stdout:2493: done (6 left) 2021-03-24T22:03:46.351 INFO:tasks.rados.rados.0.smithi125.stdout:2494: copy_from oid 34 from oid 40 current snap is 281 2021-03-24T22:03:46.351 INFO:tasks.rados.rados.0.smithi125.stdout:2495: write oid 8 current snap is 281 2021-03-24T22:03:46.352 INFO:tasks.rados.rados.0.smithi125.stdout:2495: seq_num 865 ranges {538055=638274,1867983=670072} 2021-03-24T22:03:46.361 INFO:tasks.rados.rados.0.smithi125.stdout:2495: writing smithi12556470-8 from 538055 to 1176329 tid 1 2021-03-24T22:03:46.369 INFO:tasks.rados.rados.0.smithi125.stdout:2495: writing smithi12556470-8 from 1867983 to 2538055 tid 2 2021-03-24T22:03:46.369 INFO:tasks.rados.rados.0.smithi125.stdout:2496: snap_create 2021-03-24T22:03:46.369 INFO:tasks.rados.rados.0.smithi125.stdout:2491: finishing rollback tid 1 to smithi12556470-3 2021-03-24T22:03:46.370 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 3 v 307 (ObjNum 1116422336 snap 21916 seq_num 3308612720) dirty dne 2021-03-24T22:03:46.370 INFO:tasks.rados.rados.0.smithi125.stdout:2492: finishing write tid 1 to smithi12556470-41 2021-03-24T22:03:46.370 INFO:tasks.rados.rados.0.smithi125.stdout:2492: finishing write tid 2 to smithi12556470-41 2021-03-24T22:03:46.370 INFO:tasks.rados.rados.0.smithi125.stdout:2492: finishing write tid 3 to smithi12556470-41 2021-03-24T22:03:46.371 INFO:tasks.rados.rados.0.smithi125.stdout:2492: finishing write tid 4 to smithi12556470-41 2021-03-24T22:03:46.371 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 41 v 824 (ObjNum 864 snap 281 seq_num 864) dirty exists 2021-03-24T22:03:46.371 INFO:tasks.rados.rados.0.smithi125.stdout:2492: left oid 41 (ObjNum 864 snap 281 seq_num 864) 2021-03-24T22:03:46.371 INFO:tasks.rados.rados.0.smithi125.stdout:2487: expect (ObjNum 625 snap 192 seq_num 625) 2021-03-24T22:03:46.386 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:46.448 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:46.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:46.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:46.476 INFO:tasks.rados.rados.0.smithi125.stdout:2490: finishing write tid 1 to smithi12556470-6 2021-03-24T22:03:46.477 INFO:tasks.rados.rados.0.smithi125.stdout:2490: finishing write tid 2 to smithi12556470-6 2021-03-24T22:03:46.477 INFO:tasks.rados.rados.0.smithi125.stdout:2490: finishing write tid 3 to smithi12556470-6 2021-03-24T22:03:46.477 INFO:tasks.rados.rados.0.smithi125.stdout:2490: finishing write tid 4 to smithi12556470-6 2021-03-24T22:03:46.477 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 6 v 575 (ObjNum 863 snap 281 seq_num 863) dirty exists 2021-03-24T22:03:46.478 INFO:tasks.rados.rados.0.smithi125.stdout:2490: left oid 6 (ObjNum 863 snap 281 seq_num 863) 2021-03-24T22:03:46.478 INFO:tasks.rados.rados.0.smithi125.stdout:2494: finishing copy_from to smithi12556470-34 2021-03-24T22:03:46.478 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 34 v 1411 (ObjNum 832 snap 272 seq_num 832) dirty exists 2021-03-24T22:03:46.478 INFO:tasks.rados.rados.0.smithi125.stdout:2494: finishing copy_from racing read to smithi12556470-34 2021-03-24T22:03:46.478 INFO:tasks.rados.rados.0.smithi125.stdout:2488: expect (ObjNum 862 snap 279 seq_num 862) 2021-03-24T22:03:46.555 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:46.607 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:46.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:46.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:03:46.625 INFO:tasks.rados.rados.0.smithi125.stdout:2489: finishing copy_from to smithi12556470-2 2021-03-24T22:03:46.625 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 2 v 430 (ObjNum 856 snap 278 seq_num 856) dirty exists 2021-03-24T22:03:46.625 INFO:tasks.rados.rados.0.smithi125.stdout:2495: finishing write tid 1 to smithi12556470-8 2021-03-24T22:03:46.625 INFO:tasks.rados.rados.0.smithi125.stdout:2489: finishing copy_from racing read to smithi12556470-2 2021-03-24T22:03:46.626 INFO:tasks.rados.rados.0.smithi125.stdout:2495: finishing write tid 2 to smithi12556470-8 2021-03-24T22:03:46.626 INFO:tasks.rados.rados.0.smithi125.stdout:2495: finishing write tid 3 to smithi12556470-8 2021-03-24T22:03:46.626 INFO:tasks.rados.rados.0.smithi125.stdout:2495: finishing write tid 4 to smithi12556470-8 2021-03-24T22:03:46.626 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 8 v 552 (ObjNum 865 snap 281 seq_num 865) dirty exists 2021-03-24T22:03:46.627 INFO:tasks.rados.rados.0.smithi125.stdout:2495: left oid 8 (ObjNum 865 snap 281 seq_num 865) 2021-03-24T22:03:46.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[19936]: cluster 2021-03-24T22:03:44.728750 2021-03-24T22:03:46.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[19936]: +0000 osd.5 (osd.5) 37 : cluster [DBG] 5.1b scrub ok 2021-03-24T22:03:46.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[19936]: cluster 2021-03-24T22:03:44.936128+0000 mgr.y (mgr.14144) 601 : 2021-03-24T22:03:46.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[19936]: cluster [DBG] pgmap v1144: 168 pgs: 1 peering, 4 active+remapped+backfill_toofull, 163 active+clean; 514 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 84/1719 objects misplaced (4.887%); 658 KiB/s, 5 objects/s recovering 2021-03-24T22:03:46.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:03:46.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[19936]: -03-24T22:03:46.347679+0000 mon.a (mon.0 2021-03-24T22:03:46.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[19936]: ) 1740 2021-03-24T22:03:46.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[19936]: : cluster [DBG] osdmap e656: 8 total, 8 up, 8 in 2021-03-24T22:03:46.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:03:46.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[19936]: 03-24T22:03:46.351055 2021-03-24T22:03:46.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[19936]: +0000 osd.3 (osd.3) 136 : cluster 2021-03-24T22:03:46.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[19936]: [DBG] 4.9 starting backfill to osd.1 from (0'0,0'0] MAX to 544'38 2021-03-24T22:03:46.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:03:46.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[19936]: 03-24T 2021-03-24T22:03:46.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[19936]: 22:03:46.351137+0000 osd.3 (osd 2021-03-24T22:03:46.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[19936]: .3) 137 : cluster [DBG] 4.f starting backfill to osd.1 from (0'0,0'0] MAX to 544'46 2021-03-24T22:03:46.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[19936]: cluster 2021-03-24T22:03:46.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[19936]: 2021-03-24T22:03:46. 2021-03-24T22:03:46.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[19936]: 351243+0000 osd.6 (osd.6) 52 : 2021-03-24T22:03:46.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[19936]: cluster [DBG] 4.e starting backfill to osd.1 from (0'0,0'0] MAX to 544'45 2021-03-24T22:03:46.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T22:03:46.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[19936]: 22:03:46.352571+0000 osd.6 ( 2021-03-24T22:03:46.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[19936]: osd.6) 53 : cluster [DBG] 4.a starting backfill to osd.1 from (0'0,0'0] MAX to 544'45 2021-03-24T22:03:46.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[19936]: 2021-03-24T22:03:46.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[24927]: cluster 2021-03-24T22:03:44.728750+0000 2021-03-24T22:03:46.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[24927]: osd.5 (osd.5) 37 : cluster [DBG] 5.1b scrub ok 2021-03-24T22:03:46.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:03:46.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[24927]: 03:44.936128+0000 mgr.y (mgr.14144) 601 : cluster [DBG] pgmap v1144: 168 pgs: 1 peering, 4 active+remapped+backfill_toofull, 163 active+clean; 514 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 84/1719 objects misplaced (4.887%); 658 KiB/s, 5 objects/s recovering 2021-03-24T22:03:46.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[24927]: cluster 2021-03-24T22 2021-03-24T22:03:46.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[24927]: :03:46.347679+0000 mon.a (mon.0) 2021-03-24T22:03:46.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[24927]: 1740 : cluster [DBG] osdmap e656: 8 total, 8 up, 8 in 2021-03-24T22:03:46.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[24927]: cluster 2021-03-24T22 2021-03-24T22:03:46.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[24927]: :03:46.351055+0000 osd.3 (osd.3) 136 2021-03-24T22:03:46.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[24927]: : cluster [DBG] 4.9 starting backfill to osd.1 from (0'0,0'0] MAX to 544'38 2021-03-24T22:03:46.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[24927]: cluster 2021-03-24T22:03 2021-03-24T22:03:46.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[24927]: :46.351137+0000 osd.3 (osd.3) 137 2021-03-24T22:03:46.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[24927]: : cluster [DBG] 4.f starting backfill to osd.1 from (0'0,0'0] MAX to 544'46 2021-03-24T22:03:46.672 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[24927]: cluster 2021-03-24T22:03:46 2021-03-24T22:03:46.672 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[24927]: .351243+0000 osd.6 (osd.6) 52 : cluster 2021-03-24T22:03:46.672 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[24927]: [DBG] 4.e starting backfill to osd.1 from (0'0,0'0] MAX to 544'45 2021-03-24T22:03:46.672 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[24927]: cluster 2021-03-24T22:03:46.672 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[24927]: 2021-03-24T22:03:46.673 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[24927]: 2021 2021-03-24T22:03:46.673 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[24927]: - 2021-03-24T22:03:46.673 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[24927]: 03 2021-03-24T22:03:46.673 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[24927]: - 2021-03-24T22:03:46.674 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[24927]: 24 2021-03-24T22:03:46.674 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[24927]: T 2021-03-24T22:03:46.674 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[24927]: 22 2021-03-24T22:03:46.674 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[24927]: : 2021-03-24T22:03:46.675 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[24927]: 03 2021-03-24T22:03:46.675 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[24927]: : 2021-03-24T22:03:46.675 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[24927]: 46 2021-03-24T22:03:46.675 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[24927]: . 2021-03-24T22:03:46.675 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[24927]: 352571 2021-03-24T22:03:46.676 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[24927]: +0000 2021-03-24T22:03:46.676 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[24927]: 2021-03-24T22:03:46.676 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[24927]: osd.6 2021-03-24T22:03:46.677 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[24927]: ( 2021-03-24T22:03:46.677 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[24927]: osd 2021-03-24T22:03:46.677 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[24927]: . 2021-03-24T22:03:46.677 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[24927]: 6 2021-03-24T22:03:46.678 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[24927]: ) 2021-03-24T22:03:46.678 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[24927]: 53 2021-03-24T22:03:46.678 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[24927]: : 2021-03-24T22:03:46.678 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[24927]: cluster 2021-03-24T22:03:46.678 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[24927]: 2021-03-24T22:03:46.679 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[24927]: [DBG] 2021-03-24T22:03:46.679 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[24927]: 2021-03-24T22:03:46.679 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[24927]: 4.a starting backfill to osd.1 from (0'0,0'0] MAX to 544'45 2021-03-24T22:03:46.679 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:46 smithi125 conmon[24927]: 2021-03-24T22:03:46.712 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:46.725 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:46 smithi142 conmon[20125]: cluster 2021-03-24T22:03:46.725 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:46 smithi142 conmon[20125]: 2021-03-24T22:03:44.728750 2021-03-24T22:03:46.726 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:46 smithi142 conmon[20125]: +0000 osd.5 (osd.5) 37 : cluster [DBG] 5.1b scrub ok 2021-03-24T22:03:46.726 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:46 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:03:46.726 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:46 smithi142 conmon[20125]: 22:03:44.936128+0000 mgr.y (mgr.14144) 601 : cluster [DBG] pgmap v1144: 168 pgs: 1 peering, 4 active+remapped+backfill_toofull, 163 active+clean; 514 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 84/1719 objects misplaced (4.887%); 658 KiB/s, 5 objects/s recovering 2021-03-24T22:03:46.726 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:46 smithi142 conmon[20125]: cluster 2021-03-24T22:03:46.347679+0000 mon.a 2021-03-24T22:03:46.727 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:46 smithi142 conmon[20125]: (mon.0) 1740 : cluster [DBG] osdmap e656: 8 total, 8 up, 8 in 2021-03-24T22:03:46.727 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:46 smithi142 conmon[20125]: cluster 2021-03-24T22:03:46.351055+0000 osd.3 2021-03-24T22:03:46.727 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:46 smithi142 conmon[20125]: (osd.3) 136 : cluster [DBG] 4.9 starting backfill to osd.1 from (0'0,0'0] MAX to 544'38 2021-03-24T22:03:46.727 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:46 smithi142 conmon[20125]: cluster 2021-03-24T22:03:46.727 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:46 smithi142 conmon[20125]: 2021-03-24T22:03:46.351137+0000 osd.3 (osd.3 2021-03-24T22:03:46.728 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:46 smithi142 conmon[20125]: ) 137 : cluster [DBG] 4.f starting backfill to osd.1 from (0'0,0'0] MAX to 544'46 2021-03-24T22:03:46.728 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:46 smithi142 conmon[20125]: cluster 2021-03-24T22:03 2021-03-24T22:03:46.728 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:46 smithi142 conmon[20125]: :46.351243+0000 osd.6 (osd.6) 52 : cluster [DBG] 2021-03-24T22:03:46.728 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:46 smithi142 conmon[20125]: 4.e starting backfill to osd.1 from (0'0,0'0] MAX to 544'45 2021-03-24T22:03:46.729 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:46 smithi142 conmon[20125]: cluster 2021-03-24T22:03:46.729 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:46 smithi142 conmon[20125]: 2021-03-24T22:03:46.352571+0000 osd.6 (osd.6) 53 : cluster 2021-03-24T22:03:46.729 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:46 smithi142 conmon[20125]: [DBG] 4.a starting backfill to osd.1 from (0'0,0'0] MAX to 544'45 2021-03-24T22:03:46.768 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:46.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:46.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:03:46.872 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:46.922 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:46.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:46.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:03:47.028 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:47.076 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:47.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:47.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:47.181 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:47.234 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:47.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:47.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:03:47.340 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:47.350 INFO:tasks.rados.rados.0.smithi125.stdout:2487: done (8 left) 2021-03-24T22:03:47.351 INFO:tasks.rados.rados.0.smithi125.stdout:2488: done (7 left) 2021-03-24T22:03:47.352 INFO:tasks.rados.rados.0.smithi125.stdout:2489: done (6 left) 2021-03-24T22:03:47.352 INFO:tasks.rados.rados.0.smithi125.stdout:2490: done (5 left) 2021-03-24T22:03:47.353 INFO:tasks.rados.rados.0.smithi125.stdout:2491: done (4 left) 2021-03-24T22:03:47.353 INFO:tasks.rados.rados.0.smithi125.stdout:2492: done (3 left) 2021-03-24T22:03:47.354 INFO:tasks.rados.rados.0.smithi125.stdout:2494: done (2 left) 2021-03-24T22:03:47.354 INFO:tasks.rados.rados.0.smithi125.stdout:2495: done (1 left) 2021-03-24T22:03:47.355 INFO:tasks.rados.rados.0.smithi125.stdout:2496: done (0 left) 2021-03-24T22:03:47.355 INFO:tasks.rados.rados.0.smithi125.stdout:2497: delete oid 30 current snap is 282 2021-03-24T22:03:47.363 INFO:tasks.rados.rados.0.smithi125.stdout:2497: done (0 left) 2021-03-24T22:03:47.363 INFO:tasks.rados.rados.0.smithi125.stdout:2498: read oid 11 snap 195 2021-03-24T22:03:47.363 INFO:tasks.rados.rados.0.smithi125.stdout:2498: expect (ObjNum 495 snap 142 seq_num 495) 2021-03-24T22:03:47.363 INFO:tasks.rados.rados.0.smithi125.stdout:2499: write (excl) oid 26 current snap is 282 2021-03-24T22:03:47.364 INFO:tasks.rados.rados.0.smithi125.stdout:2499: seq_num 867 ranges {571669=505125,1701300=743627} 2021-03-24T22:03:47.367 INFO:tasks.rados.rados.0.smithi125.stdout:2499: writing smithi12556470-26 from 571669 to 1076794 tid 1 2021-03-24T22:03:47.376 INFO:tasks.rados.rados.0.smithi125.stdout:2499: writing smithi12556470-26 from 1701300 to 2444927 tid 2 2021-03-24T22:03:47.376 INFO:tasks.rados.rados.0.smithi125.stdout:2500: read oid 10 snap 246 2021-03-24T22:03:47.376 INFO:tasks.rados.rados.0.smithi125.stdout:2500: expect (ObjNum 751 snap 241 seq_num 751) 2021-03-24T22:03:47.377 INFO:tasks.rados.rados.0.smithi125.stdout:2501: snap_remove snap 237 2021-03-24T22:03:47.390 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:47.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:47.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:03:47.479 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:47 smithi125 conmon[24927]: cluster 2021-03-24T22:03:47.479 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:47 smithi125 conmon[24927]: 2021-03-24T22:03:47.479 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:47 smithi125 conmon[24927]: 2021 2021-03-24T22:03:47.479 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:47 smithi125 conmon[24927]: - 2021-03-24T22:03:47.480 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:47 smithi125 conmon[24927]: 03 2021-03-24T22:03:47.480 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:47 smithi125 conmon[24927]: - 2021-03-24T22:03:47.480 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:47 smithi125 conmon[24927]: 24 2021-03-24T22:03:47.481 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:47 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:03:47.481 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:47 smithi125 conmon[19936]: 03:46.417624+0000 osd.0 (osd.0) 129 : cluster [DBG] 4.6 starting backfill to osd.1 from (0'0,0'0] MAX to 544'34 2021-03-24T22:03:47.481 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:47 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T22:03:47.481 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:47 smithi125 conmon[19936]: -24T22:03:47.346787+0000 mon.a (mon.0) 1741 : cluster [DBG] osdmap e657: 8 total, 8 up, 8 in 2021-03-24T22:03:47.482 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:47 smithi125 conmon[19936]: 2021-03-24T22:03:47.496 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:47.545 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:47.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:47.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:03:47.652 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:47.706 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:47.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:47.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:03:47.813 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:47.826 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:47 smithi142 conmon[20125]: cluster 2021-03-24T22:03:46.417624+0000 osd.0 (osd.0) 129 2021-03-24T22:03:47.826 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:47 smithi142 conmon[20125]: : cluster [DBG] 4.6 starting backfill to osd.1 from (0'0,0'0] MAX to 544'34 2021-03-24T22:03:47.826 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:47 smithi142 conmon[20125]: cluster 2021-03-24T22:03:47.346787 2021-03-24T22:03:47.827 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:47 smithi142 conmon[20125]: +0000 mon.a (mon.0) 1741 : cluster [DBG] 2021-03-24T22:03:47.827 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:47 smithi142 conmon[20125]: osdmap e657: 8 total, 8 up, 8 in 2021-03-24T22:03:47.865 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:47.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:47.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:03:47.912 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:47 smithi125 conmon[24927]: T 2021-03-24T22:03:47.912 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:47 smithi125 conmon[24927]: 22 2021-03-24T22:03:47.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:47 smithi125 conmon[24927]: : 2021-03-24T22:03:47.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:47 smithi125 conmon[24927]: 03 2021-03-24T22:03:47.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:47 smithi125 conmon[24927]: : 2021-03-24T22:03:47.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:47 smithi125 conmon[24927]: 46 2021-03-24T22:03:47.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:47 smithi125 conmon[24927]: . 2021-03-24T22:03:47.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:47 smithi125 conmon[24927]: 417624 2021-03-24T22:03:47.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:47 smithi125 conmon[24927]: +0000 2021-03-24T22:03:47.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:47 smithi125 conmon[24927]: 2021-03-24T22:03:47.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:47 smithi125 conmon[24927]: osd.0 2021-03-24T22:03:47.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:47 smithi125 conmon[24927]: ( 2021-03-24T22:03:47.917 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:47 smithi125 conmon[24927]: osd 2021-03-24T22:03:47.918 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:47 smithi125 conmon[24927]: . 2021-03-24T22:03:47.918 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:47 smithi125 conmon[24927]: 0 2021-03-24T22:03:47.919 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:47 smithi125 conmon[24927]: ) 2021-03-24T22:03:47.919 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:47 smithi125 conmon[24927]: 129 2021-03-24T22:03:47.920 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:47 smithi125 conmon[24927]: : 2021-03-24T22:03:47.920 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:47 smithi125 conmon[24927]: cluster 2021-03-24T22:03:47.920 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:47 smithi125 conmon[24927]: [DBG] 4.6 starting backfill to osd.1 from (0'0,0'0] MAX to 544'34 2021-03-24T22:03:47.921 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:47 smithi125 conmon[24927]: 2021-03-24T22:03:47.921 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:47 smithi125 conmon[24927]: cluster 2021-03-24T22:03:47.921 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:47 smithi125 conmon[24927]: 2021-03 2021-03-24T22:03:47.922 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:47 smithi125 conmon[24927]: -24T22 2021-03-24T22:03:47.922 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:47 smithi125 conmon[24927]: :03: 2021-03-24T22:03:47.922 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:47 smithi125 conmon[24927]: 47.346787 2021-03-24T22:03:47.923 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:47 smithi125 conmon[24927]: +0000 mon.a ( 2021-03-24T22:03:47.923 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:47 smithi125 conmon[24927]: mon.0 2021-03-24T22:03:47.923 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:47 smithi125 conmon[24927]: ) 1741 : cluster 2021-03-24T22:03:47.924 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:47 smithi125 conmon[24927]: [DBG] 2021-03-24T22:03:47.924 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:47 smithi125 conmon[24927]: osdmap e657: 8 total, 8 up, 8 in 2021-03-24T22:03:47.971 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:48.029 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:48.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:48.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:03:48.063 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.2 dump_ops_in_flight 2021-03-24T22:03:48.103 INFO:tasks.thrashosds.thrasher:Added osd 1 2021-03-24T22:03:48.134 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:48.184 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:48.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:48.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:03:48.290 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:48.343 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:48.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:48.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:03:48.360 INFO:tasks.rados.rados.0.smithi125.stdout:2501: done (3 left) 2021-03-24T22:03:48.360 INFO:tasks.rados.rados.0.smithi125.stdout:2502: copy_from oid 9 from oid 18 current snap is 282 2021-03-24T22:03:48.360 INFO:tasks.rados.rados.0.smithi125.stdout:2498: expect (ObjNum 495 snap 142 seq_num 495) 2021-03-24T22:03:48.448 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:48.504 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:48.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:48.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:03:48.521 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:03:48.527 INFO:tasks.rados.rados.0.smithi125.stdout:2500: expect (ObjNum 751 snap 241 seq_num 751) 2021-03-24T22:03:48.610 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:48.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:48 smithi142 conmon[20125]: cluster 2021-03-24T22:03: 2021-03-24T22:03:48.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:48 smithi142 conmon[20125]: 46.936786+0000 mgr.y (mgr.14144) 602 : cluster [DBG] pgmap v1147: 168 pgs: 1 active+recovering, 4 remapped+peering, 20 peering, 3 active+remapped+backfill_toofull, 1 active+remapped, 139 active+clean; 521 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 65/1623 objects misplaced (4.005%) 2021-03-24T22:03:48.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:48 smithi142 conmon[20125]: cluster 2021-03-24T22:03 2021-03-24T22:03:48.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:48 smithi142 conmon[20125]: :48.350878+0000 mon.a (mon.0) 1742 : cluster [DBG] osdmap e658: 8 total, 8 up, 8 in 2021-03-24T22:03:48.624 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:48 smithi125 conmon[24927]: cluster 2021-03-24T22:03 2021-03-24T22:03:48.624 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:48 smithi125 conmon[24927]: :46.936786 2021-03-24T22:03:48.625 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:48 smithi125 conmon[24927]: +0000 mgr.y (mgr.14144) 602 : cluster [DBG] pgmap v1147: 168 pgs: 1 active+recovering, 4 remapped+peering, 20 peering, 3 active+remapped+backfill_toofull, 1 active+remapped, 139 active+clean; 521 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 65/1623 objects misplaced (4.005%) 2021-03-24T22:03:48.625 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:48 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T22:03:48.625 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:48 smithi125 conmon[24927]: 22:03:48.350878+0000 mon.a (mon.0) 2021-03-24T22:03:48.625 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:48 smithi125 conmon[24927]: 1742 : cluster [DBG] osdmap e658: 8 total, 8 up, 8 in 2021-03-24T22:03:48.626 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:48 smithi125 conmon[19936]: cluster 2021-03-24T22:03 2021-03-24T22:03:48.626 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:48 smithi125 conmon[19936]: :46.936786+0000 mgr.y (mgr.14144) 602 : cluster [DBG] pgmap v1147: 168 pgs: 1 active+recovering, 4 remapped+peering, 20 peering, 3 active+remapped+backfill_toofull, 1 active+remapped, 139 active+clean; 521 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 65/1623 objects misplaced (4.005%) 2021-03-24T22:03:48.627 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:48 smithi125 conmon[19936]: cluster 2021-03-24T22:03:48.627 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:48 smithi125 conmon[19936]: 2021-03-24T22: 2021-03-24T22:03:48.627 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:48 smithi125 conmon[19936]: 03:48.350878+0000 mon.a 2021-03-24T22:03:48.627 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:48 smithi125 conmon[19936]: (mon.0) 1742 : cluster 2021-03-24T22:03:48.628 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:48 smithi125 conmon[19936]: [DBG] osdmap e658: 8 total, 8 up, 8 in 2021-03-24T22:03:48.658 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:48.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:48.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:03:48.664 INFO:tasks.rados.rados.0.smithi125.stdout:2499: finishing write tid 1 to smithi12556470-26 2021-03-24T22:03:48.664 INFO:tasks.rados.rados.0.smithi125.stdout:2499: finishing write tid 2 to smithi12556470-26 2021-03-24T22:03:48.664 INFO:tasks.rados.rados.0.smithi125.stdout:2499: finishing write tid 3 to smithi12556470-26 2021-03-24T22:03:48.665 INFO:tasks.rados.rados.0.smithi125.stdout:2499: finishing write tid 4 to smithi12556470-26 2021-03-24T22:03:48.665 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 26 v 453 (ObjNum 867 snap 282 seq_num 867) dirty exists 2021-03-24T22:03:48.665 INFO:tasks.rados.rados.0.smithi125.stdout:2499: left oid 26 (ObjNum 867 snap 282 seq_num 867) 2021-03-24T22:03:48.665 INFO:tasks.rados.rados.0.smithi125.stdout:2498: done (3 left) 2021-03-24T22:03:48.666 INFO:tasks.rados.rados.0.smithi125.stdout:2499: done (2 left) 2021-03-24T22:03:48.666 INFO:tasks.rados.rados.0.smithi125.stdout:2500: done (1 left) 2021-03-24T22:03:48.666 INFO:tasks.rados.rados.0.smithi125.stdout:2503: snap_create 2021-03-24T22:03:48.666 INFO:tasks.rados.rados.0.smithi125.stdout:2502: finishing copy_from to smithi12556470-9 2021-03-24T22:03:48.666 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 9 v 702 (ObjNum 861 snap 279 seq_num 861) dirty exists 2021-03-24T22:03:48.667 INFO:tasks.rados.rados.0.smithi125.stdout:2502: finishing copy_from racing read to smithi12556470-9 2021-03-24T22:03:48.762 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:48.809 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:48.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:48.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:03:48.917 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:48.969 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:48.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:48.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:03:49.076 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:49.131 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:49.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:49.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:03:49.236 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:49.286 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:49.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:49.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:03:49.359 INFO:tasks.rados.rados.0.smithi125.stdout:2502: done (1 left) 2021-03-24T22:03:49.360 INFO:tasks.rados.rados.0.smithi125.stdout:2503: done (0 left) 2021-03-24T22:03:49.360 INFO:tasks.rados.rados.0.smithi125.stdout:2504: delete oid 23 current snap is 283 2021-03-24T22:03:49.367 INFO:teuthology.orchestra.run.smithi125.stderr:nodeep-scrub is unset 2021-03-24T22:03:49.382 INFO:tasks.rados.rados.0.smithi125.stdout:2504: done (0 left) 2021-03-24T22:03:49.383 INFO:tasks.rados.rados.0.smithi125.stdout:2505: copy_from oid 4 from oid 8 current snap is 283 2021-03-24T22:03:49.383 INFO:tasks.rados.rados.0.smithi125.stdout:2506: delete oid 10 current snap is 283 2021-03-24T22:03:49.384 INFO:tasks.rados.rados.0.smithi125.stdout:2506: done (1 left) 2021-03-24T22:03:49.385 INFO:tasks.rados.rados.0.smithi125.stdout:2507: delete oid 36 current snap is 283 2021-03-24T22:03:49.387 INFO:tasks.rados.rados.0.smithi125.stdout:2507: done (1 left) 2021-03-24T22:03:49.388 INFO:tasks.rados.rados.0.smithi125.stdout:2508: write (excl) oid 5 current snap is 283 2021-03-24T22:03:49.388 INFO:tasks.rados.rados.0.smithi125.stdout:2508: seq_num 871 ranges {638897=406121,1818875=750965} 2021-03-24T22:03:49.392 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:49.395 INFO:tasks.rados.rados.0.smithi125.stdout:2508: writing smithi12556470-5 from 638897 to 1045018 tid 1 2021-03-24T22:03:49.404 INFO:tasks.rados.rados.0.smithi125.stdout:2508: writing smithi12556470-5 from 1818875 to 2569840 tid 2 2021-03-24T22:03:49.404 INFO:tasks.rados.rados.0.smithi125.stdout:2509: write (excl) oid 29 current snap is 283 2021-03-24T22:03:49.405 INFO:tasks.rados.rados.0.smithi125.stdout:2509: seq_num 872 ranges {655704=481370,1761004=494623} 2021-03-24T22:03:49.410 INFO:tasks.rados.rados.0.smithi125.stdout:2509: writing smithi12556470-29 from 655704 to 1137074 tid 1 2021-03-24T22:03:49.417 INFO:tasks.rados.rados.0.smithi125.stdout:2509: writing smithi12556470-29 from 1761004 to 2255627 tid 2 2021-03-24T22:03:49.417 INFO:tasks.rados.rados.0.smithi125.stdout:2510: write (excl) oid 20 current snap is 283 2021-03-24T22:03:49.418 INFO:tasks.rados.rados.0.smithi125.stdout:2510: seq_num 873 ranges {672511=556619,1819486=754634} 2021-03-24T22:03:49.423 INFO:tasks.rados.rados.0.smithi125.stdout:2510: writing smithi12556470-20 from 672511 to 1229130 tid 1 2021-03-24T22:03:49.432 INFO:tasks.rados.rados.0.smithi125.stdout:2510: writing smithi12556470-20 from 1819486 to 2574120 tid 2 2021-03-24T22:03:49.433 INFO:tasks.rados.rados.0.smithi125.stdout:2511: delete oid 24 current snap is 283 2021-03-24T22:03:49.433 INFO:tasks.rados.rados.0.smithi125.stdout:2505: finishing copy_from to smithi12556470-4 2021-03-24T22:03:49.433 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 4 v 552 (ObjNum 865 snap 281 seq_num 865) dirty exists 2021-03-24T22:03:49.433 INFO:tasks.rados.rados.0.smithi125.stdout:2505: finishing copy_from racing read to smithi12556470-4 2021-03-24T22:03:49.434 INFO:tasks.rados.rados.0.smithi125.stdout:2508: finishing write tid 1 to smithi12556470-5 2021-03-24T22:03:49.434 INFO:tasks.rados.rados.0.smithi125.stdout:2508: finishing write tid 2 to smithi12556470-5 2021-03-24T22:03:49.434 INFO:tasks.rados.rados.0.smithi125.stdout:2508: finishing write tid 3 to smithi12556470-5 2021-03-24T22:03:49.434 INFO:tasks.rados.rados.0.smithi125.stdout:2508: finishing write tid 4 to smithi12556470-5 2021-03-24T22:03:49.435 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 5 v 836 (ObjNum 871 snap 283 seq_num 871) dirty exists 2021-03-24T22:03:49.435 INFO:tasks.rados.rados.0.smithi125.stdout:2508: left oid 5 (ObjNum 871 snap 283 seq_num 871) 2021-03-24T22:03:49.435 INFO:tasks.rados.rados.0.smithi125.stdout:2509: finishing write tid 1 to smithi12556470-29 2021-03-24T22:03:49.435 INFO:tasks.rados.rados.0.smithi125.stdout:2509: finishing write tid 2 to smithi12556470-29 2021-03-24T22:03:49.436 INFO:tasks.rados.rados.0.smithi125.stdout:2509: finishing write tid 3 to smithi12556470-29 2021-03-24T22:03:49.436 INFO:tasks.rados.rados.0.smithi125.stdout:2509: finishing write tid 4 to smithi12556470-29 2021-03-24T22:03:49.436 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 29 v 425 (ObjNum 872 snap 283 seq_num 872) dirty exists 2021-03-24T22:03:49.436 INFO:tasks.rados.rados.0.smithi125.stdout:2509: left oid 29 (ObjNum 872 snap 283 seq_num 872) 2021-03-24T22:03:49.437 INFO:tasks.rados.rados.0.smithi125.stdout:2510: finishing write tid 1 to smithi12556470-20 2021-03-24T22:03:49.437 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:49.438 INFO:tasks.rados.rados.0.smithi125.stdout:2505: done (4 left) 2021-03-24T22:03:49.438 INFO:tasks.rados.rados.0.smithi125.stdout:2508: done (3 left) 2021-03-24T22:03:49.438 INFO:tasks.rados.rados.0.smithi125.stdout:2509: done (2 left) 2021-03-24T22:03:49.438 INFO:tasks.rados.rados.0.smithi125.stdout:2511: done (1 left) 2021-03-24T22:03:49.439 INFO:tasks.rados.rados.0.smithi125.stdout:2512: write oid 30 current snap is 283 2021-03-24T22:03:49.439 INFO:tasks.rados.rados.0.smithi125.stdout:2512: seq_num 875 ranges {706125=423470,1652803=758303} 2021-03-24T22:03:49.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:49.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:03:49.442 INFO:tasks.rados.rados.0.smithi125.stdout:2512: writing smithi12556470-30 from 706125 to 1129595 tid 1 2021-03-24T22:03:49.452 INFO:tasks.rados.rados.0.smithi125.stdout:2512: writing smithi12556470-30 from 1652803 to 2411106 tid 2 2021-03-24T22:03:49.452 INFO:tasks.rados.rados.0.smithi125.stdout:2513: snap_create 2021-03-24T22:03:49.452 INFO:tasks.rados.rados.0.smithi125.stdout:2510: finishing write tid 2 to smithi12556470-20 2021-03-24T22:03:49.452 INFO:tasks.rados.rados.0.smithi125.stdout:2510: finishing write tid 3 to smithi12556470-20 2021-03-24T22:03:49.453 INFO:tasks.rados.rados.0.smithi125.stdout:2510: finishing write tid 4 to smithi12556470-20 2021-03-24T22:03:49.453 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 20 v 802 (ObjNum 873 snap 283 seq_num 873) dirty exists 2021-03-24T22:03:49.453 INFO:tasks.rados.rados.0.smithi125.stdout:2510: left oid 20 (ObjNum 873 snap 283 seq_num 873) 2021-03-24T22:03:49.453 INFO:tasks.rados.rados.0.smithi125.stdout:2512: finishing write tid 1 to smithi12556470-30 2021-03-24T22:03:49.455 INFO:tasks.rados.rados.0.smithi125.stdout:2512: finishing write tid 2 to smithi12556470-30 2021-03-24T22:03:49.456 INFO:tasks.rados.rados.0.smithi125.stdout:2512: finishing write tid 3 to smithi12556470-30 2021-03-24T22:03:49.456 INFO:tasks.rados.rados.0.smithi125.stdout:2512: finishing write tid 4 to smithi12556470-30 2021-03-24T22:03:49.456 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 30 v 828 (ObjNum 875 snap 283 seq_num 875) dirty exists 2021-03-24T22:03:49.456 INFO:tasks.rados.rados.0.smithi125.stdout:2512: left oid 30 (ObjNum 875 snap 283 seq_num 875) 2021-03-24T22:03:49.541 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:49.595 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:49.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:49.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:03:49.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:49 smithi142 conmon[20125]: audit 2021-03-24T22:03: 2021-03-24T22:03:49.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:49 smithi142 conmon[20125]: 48.815582+0000 mon.c (mon.2) 99 : audit [INF] from='client.? v1:172.21.15.125:0/365242569' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:03:49.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:49 smithi142 conmon[20125]: audit 2021-03-24T22:03:49.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:49 smithi142 conmon[20125]: 2021-03-24T22:03:48.816006 2021-03-24T22:03:49.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:49 smithi142 conmon[20125]: +0000 mon.a (mon.0) 1743 : audit 2021-03-24T22:03:49.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:49 smithi142 conmon[20125]: [INF] from='client.35615 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:03:49.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:49 smithi142 conmon[20125]: cluster 2021-03-24T22:03:49.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:49 smithi142 conmon[20125]: 2021-03-24T22:03:49 2021-03-24T22:03:49.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:49 smithi142 conmon[20125]: .347706+0000 mon.a (mon.0) 1744 : cluster 2021-03-24T22:03:49.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:49 smithi142 conmon[20125]: [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T22:03:49.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:49 smithi142 conmon[20125]: audit 2021-03-24T22:03:49.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:49 smithi142 conmon[20125]: 2021-03-24T22:03:49. 2021-03-24T22:03:49.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:49 smithi142 conmon[20125]: 355091+0000 mon.a (mon.0) 1745 2021-03-24T22:03:49.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:49 smithi142 conmon[20125]: : audit [INF] from='client.35615 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T22:03:49.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:49 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:03:49.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:49 smithi142 conmon[20125]: -03-24T22:03:49.355275+0000 2021-03-24T22:03:49.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:49 smithi142 conmon[20125]: mon.a (mon.0) 1746 : cluster [DBG] osdmap e659: 8 total, 8 up, 8 in 2021-03-24T22:03:49.702 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:49.742 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:49 smithi125 conmon[24927]: audit 2021-03-24T22: 2021-03-24T22:03:49.743 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:49 smithi125 conmon[24927]: 03:48.815582+0000 mon.c (mon.2) 99 : audit [INF] from='client.? v1:172.21.15.125:0/365242569' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:03:49.743 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:49 smithi125 conmon[24927]: audit 2021 2021-03-24T22:03:49.743 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:49 smithi125 conmon[24927]: -03-24T22:03:48.816006+0000 2021-03-24T22:03:49.743 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:49 smithi125 conmon[24927]: mon.a (mon.0) 1743 : audit [INF] from='client.35615 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:03:49.743 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:49 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T22:03:49.744 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:49 smithi125 conmon[24927]: -24T22:03:49.347706+0000 mon.a (mon.0) 1744 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T22:03:49.744 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:49 smithi125 conmon[24927]: audit 2021-03-24T22:03 2021-03-24T22:03:49.744 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:49 smithi125 conmon[24927]: :49.355091+0000 mon.a (mon.0) 1745 : audit [INF] 2021-03-24T22:03:49.744 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:49 smithi125 conmon[24927]: from='client.35615 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T22:03:49.745 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:49 smithi125 conmon[24927]: cluster 2021-03-24T22:03:49. 2021-03-24T22:03:49.745 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:49 smithi125 conmon[24927]: 355275+0000 mon.a (mon.0) 1746 : cluster [DBG] 2021-03-24T22:03:49.745 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:49 smithi125 conmon[24927]: osdmap e659: 8 total, 8 up, 8 in 2021-03-24T22:03:49.746 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:49 smithi125 conmon[19936]: audit 2021-03-24 2021-03-24T22:03:49.746 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:49 smithi125 conmon[19936]: T22:03:48.815582+0000 mon.c (mon.2) 99 : audit [INF] from='client.? v1:172.21.15.125:0/365242569' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:03:49.746 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:49 smithi125 conmon[19936]: audit 2021-03-24T22:03:49.746 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:49 smithi125 conmon[19936]: 2021-03-24T22: 2021-03-24T22:03:49.747 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:49 smithi125 conmon[19936]: 03:48.816006+0000 mon.a ( 2021-03-24T22:03:49.747 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:49 smithi125 conmon[19936]: mon.0) 1743 : audit 2021-03-24T22:03:49.747 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:49 smithi125 conmon[19936]: [INF] from='client.35615 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:03:49.747 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:49 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:03:49.748 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:49 smithi125 conmon[19936]: -03-24T22:03:49.347706+0000 2021-03-24T22:03:49.748 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:49 smithi125 conmon[19936]: mon.a (mon.0) 1744 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T22:03:49.748 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:49 smithi125 conmon[19936]: audit 2021-03-24T22: 2021-03-24T22:03:49.748 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:49 smithi125 conmon[19936]: 03:49.355091+0000 mon.a (mon.0) 1745 : audit 2021-03-24T22:03:49.748 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:49 smithi125 conmon[19936]: [INF] from='client.35615 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T22:03:49.749 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:49 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:03:49.749 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:49 smithi125 conmon[19936]: 03-24T22:03:49.355275+0000 mon.a (mon.0) 1746 : cluster [DBG] 2021-03-24T22:03:49.749 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:49 smithi125 conmon[19936]: osdmap e659: 8 total, 8 up, 8 in 2021-03-24T22:03:49.756 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:49.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:49.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:03:49.860 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:49.915 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:49.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:49.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:03:50.006 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-24T22:03:50.021 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:50.075 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:50.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:50.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:03:50.180 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:50.227 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:50.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:50.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:03:50.332 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:50.351 INFO:tasks.rados.rados.0.smithi125.stdout:2510: done (2 left) 2021-03-24T22:03:50.351 INFO:tasks.rados.rados.0.smithi125.stdout:2512: done (1 left) 2021-03-24T22:03:50.352 INFO:tasks.rados.rados.0.smithi125.stdout:2513: done (0 left) 2021-03-24T22:03:50.352 INFO:tasks.rados.rados.0.smithi125.stdout:2514: copy_from oid 34 from oid 46 current snap is 284 2021-03-24T22:03:50.352 INFO:tasks.rados.rados.0.smithi125.stdout:2515: copy_from oid 5 from oid 37 current snap is 284 2021-03-24T22:03:50.352 INFO:tasks.rados.rados.0.smithi125.stdout:2516: snap_create 2021-03-24T22:03:50.380 INFO:tasks.rados.rados.0.smithi125.stdout:2515: finishing copy_from to smithi12556470-5 2021-03-24T22:03:50.380 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 5 v 722 (ObjNum 757 snap 244 seq_num 757) dirty exists 2021-03-24T22:03:50.381 INFO:tasks.rados.rados.0.smithi125.stdout:2514: finishing copy_from to smithi12556470-34 2021-03-24T22:03:50.381 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 34 v 730 (ObjNum 497 snap 145 seq_num 497) dirty exists 2021-03-24T22:03:50.381 INFO:tasks.rados.rados.0.smithi125.stdout:2515: finishing copy_from racing read to smithi12556470-5 2021-03-24T22:03:50.382 INFO:tasks.rados.rados.0.smithi125.stdout:2514: finishing copy_from racing read to smithi12556470-34 2021-03-24T22:03:50.392 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:50.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:50.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:03:50.488 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 24 22:03:50 smithi142 conmon[33921]: debug 2021-03-24T22:03:50.364+0000 7fc7b7c21700 -1 osd.7 pg_epoch: 660 pg[3.14( empty local-lis/les=546/547 n=0 ec=60/60 lis/c=546/546 les/c/f=547/547/0 sis=546) [7,4,6] r=0 lpr=546 crt=0'0 mlcod 0'0 active+clean] scrubber pg(3.14) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 4 (0x56270f8cf4a0) 2021-03-24T22:03:50.496 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:50.544 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:50.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:50.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:03:50.650 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:50.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:50 smithi125 conmon[24927]: cluster 2021-03-24T22:03: 2021-03-24T22:03:50.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:50 smithi125 conmon[24927]: 48.937528+0000 mgr.y (mgr.14144) 603 : cluster [DBG] pgmap v1150: 168 pgs: 1 active+recovering, 4 remapped+peering, 20 peering, 3 active+remapped+backfill_toofull, 1 active+remapped, 139 active+clean; 521 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 65/1623 objects misplaced (4.005%) 2021-03-24T22:03:50.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:50 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:03:50.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:50 smithi125 conmon[24927]: 03:50.337480+0000 mon.a (mon.0) 1747 2021-03-24T22:03:50.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:50 smithi125 conmon[24927]: : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:03:50.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:50 smithi125 conmon[24927]: cluster 2021-03-24T22:03:50.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:50 smithi125 conmon[24927]: 2021-03-24T22:03:50 2021-03-24T22:03:50.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:50 smithi125 conmon[24927]: .346377+0000 mon.a (mon.0) 1748 : cluster 2021-03-24T22:03:50.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:50 smithi125 conmon[24927]: [DBG] osdmap e660: 8 total, 8 up, 8 in 2021-03-24T22:03:50.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:50 smithi125 conmon[19936]: cluster 2021-03-24T22:03:48 2021-03-24T22:03:50.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:50 smithi125 conmon[19936]: .937528+0000 mgr.y (mgr.14144) 603 : cluster [DBG] pgmap v1150: 168 pgs: 1 active+recovering, 4 remapped+peering, 20 peering, 3 active+remapped+backfill_toofull, 1 active+remapped, 139 active+clean; 521 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 65/1623 objects misplaced (4.005%) 2021-03-24T22:03:50.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:50 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:03:50.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:50 smithi125 conmon[19936]: 24T22:03:50. 2021-03-24T22:03:50.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:50 smithi125 conmon[19936]: 337480+0000 mon.a (mon.0 2021-03-24T22:03:50.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:50 smithi125 conmon[19936]: ) 1747 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:03:50.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:50 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T22:03:50.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:50 smithi125 conmon[19936]: -24T22:03:50.346377+0000 mon.a (mon. 2021-03-24T22:03:50.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:50 smithi125 conmon[19936]: 0) 1748 : cluster [DBG] 2021-03-24T22:03:50.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:50 smithi125 conmon[19936]: osdmap e660: 8 total, 8 up, 8 in 2021-03-24T22:03:50.703 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:50.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:50.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:03:50.808 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:50.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:50 smithi142 conmon[20125]: cluster 2021-03-24T22 2021-03-24T22:03:50.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:50 smithi142 conmon[20125]: :03:48.937528+0000 mgr.y (mgr.14144) 603 : cluster [DBG] pgmap v1150: 168 pgs: 1 active+recovering, 4 remapped+peering, 20 peering, 3 active+remapped+backfill_toofull, 1 active+remapped, 139 active+clean; 521 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 65/1623 objects misplaced (4.005%) 2021-03-24T22:03:50.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:50 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:03:50.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:50 smithi142 conmon[20125]: 03-24T22:03:50.337480+0000 mon.a (mon.0) 1747 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:03:50.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:50 smithi142 conmon[20125]: 2021-03-24T22:03:50.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:50 smithi142 conmon[20125]: cluster 2021-03-24T22:03:50.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:50 smithi142 conmon[20125]: 2021-03-24T22:03:50.346377+0000 mon.a (mon. 2021-03-24T22:03:50.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:50 smithi142 conmon[20125]: 0) 1748 : cluster [DBG] osdmap e660: 8 total, 8 up, 8 in 2021-03-24T22:03:50.870 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:50.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:50.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:50.974 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:51.028 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:51.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:51.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:51.135 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:51.137 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:03:51.189 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:51.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:51.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:03:51.295 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:51.349 INFO:tasks.rados.rados.0.smithi125.stdout:2514: done (2 left) 2021-03-24T22:03:51.350 INFO:tasks.rados.rados.0.smithi125.stdout:2515: done (1 left) 2021-03-24T22:03:51.350 INFO:tasks.rados.rados.0.smithi125.stdout:2516: done (0 left) 2021-03-24T22:03:51.351 INFO:tasks.rados.rados.0.smithi125.stdout:2517: snap_create 2021-03-24T22:03:51.351 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:51.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:51.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:03:51.463 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:51.515 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:51.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:51.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:03:51.599 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-24T22:03:51.600 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-24T22:03:51.600 INFO:teuthology.orchestra.run.smithi125.stdout: "num_ops": 0 2021-03-24T22:03:51.600 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-24T22:03:51.626 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:51.683 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:51.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:51.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:03:51.746 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:51 smithi125 conmon[24927]: cluster 2021-03-24T22:03:50.590120+0000 osd.2 ( 2021-03-24T22:03:51.746 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:51 smithi125 conmon[24927]: osd.2) 103 : cluster [DBG] 3.9 scrub ok 2021-03-24T22:03:51.746 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:51 smithi125 conmon[24927]: cluster 2021-03-24T22:03:51.346685+0000 mon.a 2021-03-24T22:03:51.746 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:51 smithi125 conmon[24927]: (mon.0) 1749 : cluster [DBG] osdmap e661: 8 total, 8 up, 8 in 2021-03-24T22:03:51.747 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:51 smithi125 conmon[19936]: cluster 2021-03-24T22:03:51.747 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:51 smithi125 conmon[19936]: 2021-03-24T22:03:51.747 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:51 smithi125 conmon[19936]: 2021 2021-03-24T22:03:51.748 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:51 smithi125 conmon[19936]: - 2021-03-24T22:03:51.748 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:51 smithi125 conmon[19936]: 03 2021-03-24T22:03:51.748 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:51 smithi125 conmon[19936]: - 2021-03-24T22:03:51.748 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:51 smithi125 conmon[19936]: 24 2021-03-24T22:03:51.748 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:51 smithi125 conmon[19936]: T22:03:50. 2021-03-24T22:03:51.749 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:51 smithi125 conmon[19936]: 590120+0000 osd.2 (osd.2) 103 : cluster [DBG] 2021-03-24T22:03:51.749 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:51 smithi125 conmon[19936]: 3.9 scrub ok 2021-03-24T22:03:51.749 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:51 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:03:51.749 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:51 smithi125 conmon[19936]: -03-24T22:03:51. 2021-03-24T22:03:51.749 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:51 smithi125 conmon[19936]: 346685+0000 mon.a 2021-03-24T22:03:51.750 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:51 smithi125 conmon[19936]: (mon.0) 1749 : cluster 2021-03-24T22:03:51.750 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:51 smithi125 conmon[19936]: [DBG] osdmap e661: 8 total, 8 up, 8 in 2021-03-24T22:03:51.750 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:51 smithi125 conmon[19936]: 2021-03-24T22:03:51.791 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:51.802 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:51 smithi142 conmon[20125]: cluster 2021-03-24T22:03:50.590120+0000 2021-03-24T22:03:51.802 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:51 smithi142 conmon[20125]: osd.2 (osd.2) 103 : cluster [DBG] 3.9 scrub ok 2021-03-24T22:03:51.803 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:51 smithi142 conmon[20125]: cluster 2021-03-24T22:03: 2021-03-24T22:03:51.803 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:51 smithi142 conmon[20125]: 51.346685+0000 mon.a (mon.0) 1749 : cluster [DBG] osdmap e661: 8 total, 8 up, 8 in 2021-03-24T22:03:51.838 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:51.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:51.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:03:51.943 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:52.087 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:52.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:52.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:03:52.193 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:52.241 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:52.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:52.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:03:52.346 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:52.354 INFO:tasks.rados.rados.0.smithi125.stdout:2517: done (0 left) 2021-03-24T22:03:52.354 INFO:tasks.rados.rados.0.smithi125.stdout:2518: snap_create 2021-03-24T22:03:52.393 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:52.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:52.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:03:52.500 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:52.543 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:52.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:52.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:03:52.648 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:52.702 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:03:52.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:52.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:03:52.785 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:52 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:03:52.786 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:52 smithi125 conmon[19936]: 24T22:03:50.938056+0000 mgr.y (mgr.14144) 604 : cluster [DBG] pgmap v1153: 168 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+remapped, 1 active+remapped+backfill_wait, 1 active+recovering, 4 remapped+peering, 1 peering, 3 active+remapped+backfill_toofull, 153 active+clean; 526 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 3.2 MiB/s wr, 87 op/s; 23/1734 objects degraded (1.326%); 115/1734 objects misplaced (6.632%); 8.9 MiB/s, 57 objects/s recovering 2021-03-24T22:03:52.786 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:52 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:03:52.786 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:52 smithi125 conmon[19936]: 03-24T22:03:51.485380+0000 2021-03-24T22:03:52.787 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:52 smithi125 conmon[19936]: mon.a (mon.0) 1750 : cluster [WRN] Health check failed: Degraded data redundancy: 23/1734 objects degraded (1.326%), 2 pgs degraded (PG_DEGRADED) 2021-03-24T22:03:52.787 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:52 smithi125 conmon[19936]: 2021-03-24T22:03:52.787 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:52 smithi125 conmon[19936]: cluster 2021-03-24T22:03 2021-03-24T22:03:52.787 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:52 smithi125 conmon[19936]: :52.350643+0000 mon.a (mon.0) 1751 2021-03-24T22:03:52.788 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:52 smithi125 conmon[19936]: : cluster [DBG] osdmap e662: 8 total, 8 up, 8 in 2021-03-24T22:03:52.788 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:52 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T22:03:52.788 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:52 smithi125 conmon[24927]: 22:03:50.938056+0000 mgr.y (mgr.14144) 604 : cluster [DBG] pgmap v1153: 168 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+remapped, 1 active+remapped+backfill_wait, 1 active+recovering, 4 remapped+peering, 1 peering, 3 active+remapped+backfill_toofull, 153 active+clean; 526 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 3.2 MiB/s wr, 87 op/s; 23/1734 objects degraded (1.326%); 115/1734 objects misplaced (6.632%); 8.9 MiB/s, 57 objects/s recovering 2021-03-24T22:03:52.788 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:52 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:03:52.789 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:52 smithi125 conmon[24927]: 24T22:03:51.485380+0000 2021-03-24T22:03:52.789 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:52 smithi125 conmon[24927]: mon.a (mon.0) 1750 : cluster [WRN] 2021-03-24T22:03:52.789 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:52 smithi125 conmon[24927]: Health check failed: Degraded data redundancy: 23/1734 objects degraded (1.326%), 2 pgs degraded (PG_DEGRADED) 2021-03-24T22:03:52.789 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:52 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T22:03:52.790 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:52 smithi125 conmon[24927]: -24T22:03:52.350643+0000 mon.a (mon. 2021-03-24T22:03:52.790 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:52 smithi125 conmon[24927]: 0) 1751 : cluster [DBG] osdmap e662: 8 total, 8 up, 8 in 2021-03-24T22:03:52.808 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:52.859 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:52.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:52.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:03:52.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:52 smithi142 conmon[20125]: cluster 2021-03-24T22: 2021-03-24T22:03:52.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:52 smithi142 conmon[20125]: 03:50.938056+0000 mgr.y (mgr.14144) 604 : cluster [DBG] pgmap v1153: 168 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+remapped, 1 active+remapped+backfill_wait, 1 active+recovering, 4 remapped+peering, 1 peering, 3 active+remapped+backfill_toofull, 153 active+clean; 526 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 3.2 MiB/s wr, 87 op/s; 23/1734 objects degraded (1.326%); 115/1734 objects misplaced (6.632%); 8.9 MiB/s, 57 objects/s recovering 2021-03-24T22:03:52.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:52 smithi142 conmon[20125]: cluster 2021-03-24T22:03:52.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:52 smithi142 conmon[20125]: 2021-03-24T22:03:51.485380+0000 mon.a (mon.0) 1750 : cluster 2021-03-24T22:03:52.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:52 smithi142 conmon[20125]: [WRN] Health check failed: Degraded data redundancy: 23/1734 objects degraded (1.326%), 2 pgs degraded (PG_DEGRADED) 2021-03-24T22:03:52.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:52 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:03:52.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:52 smithi142 conmon[20125]: 03-24T22:03:52.350643+0000 2021-03-24T22:03:52.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:52 smithi142 conmon[20125]: mon.a (mon.0) 1751 : cluster [DBG] osdmap e662: 8 total, 8 up, 8 in 2021-03-24T22:03:52.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:52 smithi142 conmon[20125]: 2021-03-24T22:03:52.965 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:53.020 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:53.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:53.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:03:53.105 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 1] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2021-03-24T22:03:53.105 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-24T22:03:53.106 INFO:tasks.thrashosds.thrasher:Killing osd 5, live_osds are [0, 1, 2, 3, 4, 6, 7, 5] 2021-03-24T22:03:53.106 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2021-03-24T22:03:53.107 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl stop ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c@osd.5 2021-03-24T22:03:53.126 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:53.150 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd set noscrub 2021-03-24T22:03:53.197 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:53.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:53.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:03:53.290 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 22:03:53 smithi142 systemd[1]: Stopping Ceph osd.5 for 3cfa339a-8cea-11eb-90b2-001a4aab830c... 2021-03-24T22:03:53.290 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[66800]: debug 2021-03-24T22:03:53.220+0000 7f2250c6a700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-03-24T22:03:53.291 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[66800]: debug 2021-03-24T22:03:53.220+0000 7f2250c6a700 -1 osd.5 662 *** Got signal Terminated *** 2021-03-24T22:03:53.291 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[66800]: debug 2021-03-24T22:03:53.220+0000 7f2250c6a700 -1 osd.5 662 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-03-24T22:03:53.301 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:53.358 INFO:tasks.rados.rados.0.smithi125.stdout:2518: done (0 left) 2021-03-24T22:03:53.358 INFO:tasks.rados.rados.0.smithi125.stdout:2519: write oid 19 current snap is 287 2021-03-24T22:03:53.359 INFO:tasks.rados.rados.0.smithi125.stdout:2519: seq_num 876 ranges {722932=498719,1994932=501961} 2021-03-24T22:03:53.366 INFO:tasks.rados.rados.0.smithi125.stdout:2519: writing smithi12556470-19 from 722932 to 1221651 tid 1 2021-03-24T22:03:53.373 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:53.374 INFO:tasks.rados.rados.0.smithi125.stdout:2519: writing smithi12556470-19 from 1994932 to 2496893 tid 2 2021-03-24T22:03:53.374 INFO:tasks.rados.rados.0.smithi125.stdout:2520: snap_remove snap 261 2021-03-24T22:03:53.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:53.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:53.482 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:53.530 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:53.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:53.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:03:53.597 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 22:03:53 smithi142 podman[82739]: 2021-03-24 22:03:53.324578228 +0000 UTC m=+0.182550507 container stop a1b2ec26c85d1a9908b76a3702d8d989a024aa1578d51f03de80747c615d29d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:e9db06906b8d0a24f3ef6835ac5aa4b8eea2950a2560bad78f83d4f101eca0db, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20201204, maintainer=Dimitri Savineau , ceph=True, GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06, org.label-schema.name=CentOS Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.18.0, RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2021-03-24T22:03:53.598 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 22:03:53 smithi142 podman[82739]: 2021-03-24 22:03:53.437961518 +0000 UTC m=+0.295933884 container died a1b2ec26c85d1a9908b76a3702d8d989a024aa1578d51f03de80747c615d29d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:e9db06906b8d0a24f3ef6835ac5aa4b8eea2950a2560bad78f83d4f101eca0db, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5) 2021-03-24T22:03:53.598 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 22:03:53 smithi142 podman[82739]: a1b2ec26c85d1a9908b76a3702d8d989a024aa1578d51f03de80747c615d29d9 2021-03-24T22:03:53.599 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: cluster 2021-03-24T22:03:52.612131+0000 2021-03-24T22:03:53.599 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: osd.2 (osd.2) 104 : cluster [DBG] 6.f scrub ok 2021-03-24T22:03:53.599 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: cluster 2021-03-24T22:03:53.287987+0000 mon.a (mon.0) 2021-03-24T22:03:53.599 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: 1752 : cluster [DBG] osd.5 reported immediately failed by osd.3 2021-03-24T22:03:53.599 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:03:53.600 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: T22:03:53.288088+0000 mon.a (mon.0) 1753 : cluster [INF] 2021-03-24T22:03:53.600 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: osd.5 failed (root=default,host=smithi142) (connection refused reported by osd.3) 2021-03-24T22:03:53.600 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: cluster 2021-03-24T22:03:53.600 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: 2021-03-24T22:03:53.288602+0000 mon.a (mon. 2021-03-24T22:03:53.601 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: 0) 1754 : cluster [DBG] osd.5 reported immediately failed by osd.0 2021-03-24T22:03:53.601 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: cluster 2021-03-24T22:03:53.601 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: 2021-03-24T22:03:53.288894+0000 mon.a (mon.0) 1755 : cluster [DBG] osd.5 reported immediately failed by osd.2 2021-03-24T22:03:53.601 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: 2021-03-24T22:03:53.601 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:03:53.602 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: -24T22:03:53.289071+0000 mon.a (mon.0) 1756 : cluster [DBG] 2021-03-24T22:03:53.602 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: osd.5 reported immediately failed by osd.3 2021-03-24T22:03:53.602 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:03:53.602 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: -03-24T22:03:53.289320+0000 mon.a (mon.0) 1757 : cluster [DBG] osd.5 reported immediately failed by osd.7 2021-03-24T22:03:53.602 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: cluster 2021-03-24T22:03:53.289556+0000 2021-03-24T22:03:53.603 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: mon.a (mon.0) 1758 : cluster [DBG] osd.5 reported immediately failed by osd.1 2021-03-24T22:03:53.603 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:03:53.603 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: T22:03:53.289818+0000 mon.a (mon.0) 1759 2021-03-24T22:03:53.603 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: : cluster [DBG] osd.5 reported immediately failed by osd.2 2021-03-24T22:03:53.604 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: cluster 2021-03-24T22:03:53.290045 2021-03-24T22:03:53.604 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: +0000 mon.a (mon.0) 1760 : cluster [DBG] osd.5 reported immediately failed by osd.4 2021-03-24T22:03:53.604 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:03:53.604 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: 24T22:03:53.290273+0000 mon.a (mon. 2021-03-24T22:03:53.604 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: 0) 1761 : cluster [DBG] osd.5 reported immediately failed by osd.1 2021-03-24T22:03:53.605 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: cluster 2021-03-24T22:03:53.605 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: 2021-03-24T22:03:53.290545+0000 2021-03-24T22:03:53.605 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: mon.a (mon.0) 1762 : cluster [DBG] osd.5 reported immediately failed by osd.4 2021-03-24T22:03:53.606 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: cluster 2021-03-24T22:03:53.606 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: 2021-03-24T22:03: 2021-03-24T22:03:53.606 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: 53.290789+0000 mon.a (mon.0) 1763 : cluster [DBG] 2021-03-24T22:03:53.606 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: osd.5 reported immediately failed by osd.1 2021-03-24T22:03:53.606 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:03:53.607 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: 03-24T22:03:53.291081+0000 mon.a (mon.0) 1764 : cluster [DBG] osd.5 reported immediately failed by osd.6 2021-03-24T22:03:53.607 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: cluster 2021-03-24T22:03:53.291326+0000 2021-03-24T22:03:53.607 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: mon.a (mon.0) 1765 : cluster [DBG] osd.5 reported immediately failed by osd.6 2021-03-24T22:03:53.607 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: cluster 2021-03-24T22:03:53.291565 2021-03-24T22:03:53.608 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: +0000 mon.a (mon.0) 1766 : cluster [DBG] osd.5 reported immediately failed by osd.7 2021-03-24T22:03:53.608 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:03:53.608 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: 03-24T22:03:53.291812+0000 mon.a (mon 2021-03-24T22:03:53.608 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: .0) 1767 : cluster [DBG] osd.5 reported immediately failed by osd.7 2021-03-24T22:03:53.608 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: cluster 2021-03-24T22 2021-03-24T22:03:53.609 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: :03:53.329120+0000 mon.a (mon.0) 1768 2021-03-24T22:03:53.609 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: : cluster [DBG] osd.5 reported immediately failed by osd.7 2021-03-24T22:03:53.609 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: cluster 2021-03-24T22: 2021-03-24T22:03:53.609 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: 03:53.337182+0000 mon.a (mon.0) 1769 : cluster 2021-03-24T22:03:53.609 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: [DBG] osd.5 reported immediately failed by osd.2 2021-03-24T22:03:53.610 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: cluster 2021-03-24T22:03:53.610 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: 2021-03-24T22:03:53.345562+0000 mon.a (mon.0) 2021-03-24T22:03:53.610 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: 1770 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2021-03-24T22:03:53.610 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: cluster 2021-03-24T22:03:53.352316 2021-03-24T22:03:53.611 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:53 smithi142 conmon[20125]: +0000 mon.a (mon.0) 1771 : cluster [DBG] osdmap e663: 8 total, 7 up, 8 in 2021-03-24T22:03:53.633 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:53.692 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:53.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:53.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:03:53.722 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[24927]: cluster 2021-03-24T22:03 2021-03-24T22:03:53.723 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[24927]: :52.612131+0000 osd.2 (osd.2) 104 : cluster [DBG] 6.f scrub ok 2021-03-24T22:03:53.723 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[24927]: cluster 2021-03-24T22:03:53.723 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[24927]: 2021-03-24T 2021-03-24T22:03:53.723 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[24927]: 22:03:53.287987+0000 mon.a (mon.0) 1752 : cluster [DBG] osd.5 reported immediately failed by osd.3 2021-03-24T22:03:53.724 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:03:53.724 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[24927]: 03-24T22:03:53.288088+0000 mon.a (mon.0) 1753 : cluster 2021-03-24T22:03:53.724 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[24927]: [INF] osd.5 failed (root=default,host=smithi142) (connection refused reported by osd.3) 2021-03-24T22:03:53.724 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T22:03:53.725 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[24927]: -24T22:03:53.288602+0000 mon.a (mon.0) 1754 : cluster [DBG] osd.5 reported immediately failed by osd.0 2021-03-24T22:03:53.725 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[24927]: 2021-03-24T22:03:53.725 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T22:03:53.725 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[24927]: T22:03:53.288894+0000 mon.a (mon.0) 1755 : cluster [DBG] osd.5 reported immediately failed by osd.2 2021-03-24T22:03:53.725 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[24927]: 2021-03-24T22:03:53.726 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T22:03:53.726 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[24927]: T22:03:53.289071+0000 mon.a (mon.0) 1756 : cluster [DBG] osd.5 reported immediately failed by osd.3 2021-03-24T22:03:53.726 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[24927]: cluster 2021-03-24T22:03:53.289320+0000 2021-03-24T22:03:53.726 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[24927]: mon.a (mon.0) 1757 : cluster [DBG] osd.5 reported immediately failed by osd.7 2021-03-24T22:03:53.726 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[24927]: cluster 2021-03-24T22:03:53.289556+0000 mon.a ( 2021-03-24T22:03:53.727 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[24927]: mon.0) 1758 : cluster [DBG] osd.5 reported immediately failed by osd.1 2021-03-24T22:03:53.727 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[24927]: cluster 2021-03-24T22:03:53.289818+0000 mon.a (mon.0 2021-03-24T22:03:53.727 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[24927]: ) 1759 : cluster [DBG] osd.5 reported immediately failed by osd.2 2021-03-24T22:03:53.727 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:03:53.728 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[24927]: 03-24T22:03:53.290045+0000 mon.a (mon.0) 1760 : cluster [DBG] osd.5 reported immediately failed by osd.4 2021-03-24T22:03:53.728 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[24927]: cluster 2021-03-24T22:03:53.728 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[24927]: 2021-03-24T22:03:53.290273+0000 mon.a (mon.0) 1761 : cluster [DBG] osd.5 reported immediately failed by osd.1 2021-03-24T22:03:53.728 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[24927]: cluster 2021-03-24T22:03:53.290545+0000 mon.a ( 2021-03-24T22:03:53.728 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[24927]: mon.0) 1762 : cluster [DBG] osd.5 reported immediately failed by osd.4 2021-03-24T22:03:53.729 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:03:53.729 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[24927]: 03-24T22:03:53.290789+0000 mon.a (mon.0) 1763 : cluster [DBG] osd.5 reported immediately failed by osd.1 2021-03-24T22:03:53.729 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[24927]: cluster 2021-03-24T22:03:53.291081+0000 mon.a (mon.0) 2021-03-24T22:03:53.729 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[24927]: 1764 : cluster [DBG] osd.5 reported immediately failed by osd.6 2021-03-24T22:03:53.730 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[24927]: cluster 2021-03-24T22 2021-03-24T22:03:53.730 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[24927]: :03:53.291326+0000 mon.a (mon.0) 1765 : cluster [DBG] osd.5 reported immediately failed by osd.6 2021-03-24T22:03:53.730 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[24927]: cluster 2021-03-24T22:03:53.730 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[24927]: 2021-03-24T22:03:53.291565+0000 mon.a (mon.0) 1766 : cluster [DBG] osd.5 reported immediately failed by osd.7 2021-03-24T22:03:53.730 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[24927]: cluster 2021-03-24T22:03:53.291812+0000 mon.a (mon.0) 1767 2021-03-24T22:03:53.731 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[24927]: : cluster [DBG] osd.5 reported immediately failed by osd.7 2021-03-24T22:03:53.731 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T22:03:53.731 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[24927]: 22:03:53.329120+0000 mon.a (mon.0) 1768 : cluster [DBG] osd.5 reported immediately failed by osd.7 2021-03-24T22:03:53.731 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T22:03:53.732 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[24927]: -24T22:03:53.337182+0000 mon.a (mon.0) 1769 : cluster [DBG] osd.5 reported immediately failed by osd.2 2021-03-24T22:03:53.732 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[24927]: cluster 2021-03-24T22:03:53.345562+0000 mon.a (mon.0 2021-03-24T22:03:53.732 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[24927]: ) 1770 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2021-03-24T22:03:53.732 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T22:03:53.732 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[24927]: 22:03:53.352316+0000 mon.a (mon.0) 1771 : cluster [DBG] osdmap e663: 8 total, 7 up, 8 in 2021-03-24T22:03:53.734 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.2 dump_blocked_ops 2021-03-24T22:03:53.735 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[19936]: cluster 2021-03-24T22 2021-03-24T22:03:53.735 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[19936]: :03:52.612131+0000 osd.2 (osd.2) 104 : cluster [DBG] 6.f scrub ok 2021-03-24T22:03:53.735 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:03:53.735 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[19936]: 24T22:03:53.287987+0000 mon.a (mon.0) 1752 : cluster [DBG] osd.5 reported immediately failed by osd.3 2021-03-24T22:03:53.736 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[19936]: 2021-03-24T22:03:53.736 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[19936]: cluster 2021-03-24T22:03:53.736 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[19936]: 2021 2021-03-24T22:03:53.736 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[19936]: -03-24T22: 2021-03-24T22:03:53.736 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[19936]: 03:53.288088+0000 mon.a (mon.0) 1753 : 2021-03-24T22:03:53.737 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[19936]: cluster [INF] osd.5 failed (root=default,host=smithi142) (connection refused reported by osd.3) 2021-03-24T22:03:53.737 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[19936]: cluster 2021-03-24T22:03:53.737 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[19936]: 2021-03-24T22:03:53.737 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[19936]: 2021 2021-03-24T22:03:53.738 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[19936]: -03-24T22:03:53.288602+0000 mon.a (mon.0) 1754 : cluster [DBG] osd.5 reported immediately failed by osd.0 2021-03-24T22:03:53.738 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[19936]: cluster 2021-03-24T22:03:53.288894+0000 mon.a (mon.0) 1755 : cluster [DBG] osd.5 reported immediately failed by osd.2 2021-03-24T22:03:53.738 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[19936]: cluster 2021-03-24T22:03:53.289071+0000 mon.a (mon.0) 1756 : cluster [DBG] osd.5 reported immediately failed by osd.3 2021-03-24T22:03:53.738 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[19936]: cluster 2021-03-24T22:03:53.289320+0000 mon.a (mon.0) 1757 : cluster [DBG] osd.5 reported immediately failed by osd.7 2021-03-24T22:03:53.738 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[19936]: cluster 2021-03-24T22:03:53.289556+0000 mon.a (mon.0) 1758 : cluster [DBG] osd.5 reported immediately failed by osd.1 2021-03-24T22:03:53.739 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[19936]: cluster 2021-03-24T22:03:53.289818+0000 mon.a (mon.0) 1759 : cluster [DBG] osd.5 reported immediately failed by osd.2 2021-03-24T22:03:53.739 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[19936]: cluster 2021-03-24T22:03:53.290045+0000 mon.a (mon.0) 1760 : cluster [DBG] osd.5 reported immediately failed by osd.4 2021-03-24T22:03:53.739 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[19936]: cluster 2021-03-24T22:03:53.290273+0000 mon.a (mon.0) 1761 : cluster [DBG] osd.5 reported immediately failed by osd.1 2021-03-24T22:03:53.739 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[19936]: cluster 2021-03-24T22:03:53.290545+0000 mon.a (mon.0) 1762 : cluster [DBG] osd.5 reported immediately failed by osd.4 2021-03-24T22:03:53.740 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[19936]: cluster 2021-03-24T22:03:53.290789+0000 mon.a (mon.0) 1763 : cluster [DBG] osd.5 reported immediately failed by osd.1 2021-03-24T22:03:53.740 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[19936]: cluster 2021-03-24T22:03:53.291081+0000 mon.a (mon.0) 1764 : cluster [DBG] osd.5 reported immediately failed by osd.6 2021-03-24T22:03:53.740 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[19936]: cluster 2021-03-24T22:03:53.291326+0000 mon.a (mon.0) 1765 : cluster [DBG] osd.5 reported immediately failed by osd.6 2021-03-24T22:03:53.740 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[19936]: cluster 2021-03-24T22:03:53.291565+0000 mon.a (mon.0) 1766 : cluster [DBG] osd.5 reported immediately failed by osd.7 2021-03-24T22:03:53.740 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[19936]: cluster 2021-03-24T22:03:53.291812+0000 mon.a (mon.0) 1767 : cluster [DBG] osd.5 reported immediately failed by osd.7 2021-03-24T22:03:53.741 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[19936]: cluster 2021-03-24T22:03:53.329120+0000 mon.a (mon.0) 1768 : cluster [DBG] osd.5 reported immediately failed by osd.7 2021-03-24T22:03:53.741 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[19936]: cluster 2021-03-24T22:03:53.337182+0000 mon.a (mon.0) 1769 : cluster [DBG] osd.5 reported immediately failed by osd.2 2021-03-24T22:03:53.741 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[19936]: cluster 2021-03-24T22:03:53.345562+0000 mon.a (mon.0) 1770 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2021-03-24T22:03:53.741 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:53 smithi125 conmon[19936]: cluster 2021-03-24T22:03:53.352316+0000 mon.a (mon.0) 1771 : cluster [DBG] osdmap e663: 8 total, 7 up, 8 in 2021-03-24T22:03:53.760 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:03:53.797 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:53.860 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:53.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:53.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:03:53.967 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:54.028 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:54.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:54.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:54.132 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:54.194 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:54.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:54.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:03:54.299 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:54.347 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T22:03:54.367 INFO:tasks.rados.rados.0.smithi125.stdout:2520: done (1 left) 2021-03-24T22:03:54.368 INFO:tasks.rados.rados.0.smithi125.stdout:2521: read oid 26 snap -1 2021-03-24T22:03:54.368 INFO:tasks.rados.rados.0.smithi125.stdout:2521: expect (ObjNum 867 snap 282 seq_num 867) 2021-03-24T22:03:54.368 INFO:tasks.rados.rados.0.smithi125.stdout:2522: read oid 18 snap 243 2021-03-24T22:03:54.368 INFO:tasks.rados.rados.0.smithi125.stdout:2522: expect (ObjNum 676 snap 210 seq_num 676) 2021-03-24T22:03:54.369 INFO:tasks.rados.rados.0.smithi125.stdout:2523: delete oid 44 current snap is 287 2021-03-24T22:03:54.369 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:54.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:54.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:03:54.394 INFO:tasks.rados.rados.0.smithi125.stdout:2522: expect (ObjNum 676 snap 210 seq_num 676) 2021-03-24T22:03:54.406 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T22:03:54.437 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T22:03:54.461 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T22:03:54.480 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:54.483 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T22:03:54.512 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T22:03:54.530 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:54.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:54.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:03:54.538 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T22:03:54.539 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-24T22:03:54.539 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-24T22:03:54.539 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-24T22:03:54.540 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-24T22:03:54.540 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-24T22:03:54.540 INFO:teuthology.orchestra.run.smithi125.stderr:Error ENXIO: problem getting command descriptions from osd.5 2021-03-24T22:03:54.540 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-24T22:03:54.541 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-24T22:03:54.613 INFO:tasks.rados.rados.0.smithi125.stdout:2519: finishing write tid 1 to smithi12556470-19 2021-03-24T22:03:54.613 INFO:tasks.rados.rados.0.smithi125.stdout:2519: finishing write tid 2 to smithi12556470-19 2021-03-24T22:03:54.613 INFO:tasks.rados.rados.0.smithi125.stdout:2522: done (3 left) 2021-03-24T22:03:54.614 INFO:tasks.rados.rados.0.smithi125.stdout:2523: done (2 left) 2021-03-24T22:03:54.614 INFO:tasks.rados.rados.0.smithi125.stdout:2524: read oid 44 snap 281 2021-03-24T22:03:54.614 INFO:tasks.rados.rados.0.smithi125.stdout:2524: expect (ObjNum 736 snap 235 seq_num 736) 2021-03-24T22:03:54.615 INFO:tasks.rados.rados.0.smithi125.stdout:2525: snap_create 2021-03-24T22:03:54.615 INFO:tasks.rados.rados.0.smithi125.stdout:2519: finishing write tid 3 to smithi12556470-19 2021-03-24T22:03:54.616 INFO:tasks.rados.rados.0.smithi125.stdout:2519: finishing write tid 4 to smithi12556470-19 2021-03-24T22:03:54.616 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 19 v 604 (ObjNum 876 snap 287 seq_num 876) dirty exists 2021-03-24T22:03:54.617 INFO:tasks.rados.rados.0.smithi125.stdout:2519: left oid 19 (ObjNum 876 snap 287 seq_num 876) 2021-03-24T22:03:54.617 INFO:tasks.rados.rados.0.smithi125.stdout:2521: expect (ObjNum 867 snap 282 seq_num 867) 2021-03-24T22:03:54.634 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:54.641 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:03:54.691 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:54.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:54.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:03:54.719 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 22:03:54 smithi142 podman[82791]: 2021-03-24 22:03:54.505255225 +0000 UTC m=+1.198515357 container remove a1b2ec26c85d1a9908b76a3702d8d989a024aa1578d51f03de80747c615d29d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:e9db06906b8d0a24f3ef6835ac5aa4b8eea2950a2560bad78f83d4f101eca0db, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Dimitri Savineau , io.buildah.version=1.18.0, org.label-schema.build-date=20201204, RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06) 2021-03-24T22:03:54.720 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:54 smithi142 conmon[20125]: cluster 2021-03-24T22:03:52. 2021-03-24T22:03:54.720 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:54 smithi142 conmon[20125]: 938800+0000 mgr.y (mgr.14144) 605 : cluster [DBG] pgmap v1156: 168 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+remapped, 2 active+remapped+backfill_wait, 1 peering, 3 active+remapped+backfill_toofull, 155 active+clean; 532 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 4.7 MiB/s wr, 90 op/s; 23/1743 objects degraded (1.320%); 145/1743 objects misplaced (8.319%); 11 MiB/s, 71 objects/s recovering 2021-03-24T22:03:54.720 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:54 smithi142 conmon[20125]: cluster 2021-03-24T22:03:53 2021-03-24T22:03:54.721 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:54 smithi142 conmon[20125]: .662462+0000 osd.6 (osd.6) 54 : cluster [DBG] 7.c scrub ok 2021-03-24T22:03:54.721 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:54 smithi142 conmon[20125]: cluster 2021-03-24T22:03:54.320183+0000 osd.7 (osd. 2021-03-24T22:03:54.721 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:54 smithi142 conmon[20125]: 7) 107 : cluster [DBG] 3.14 scrub ok 2021-03-24T22:03:54.721 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:54 smithi142 conmon[20125]: audit 2021-03-24T22:03:54.323791+0000 2021-03-24T22:03:54.722 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:54 smithi142 conmon[20125]: mon.c (mon.2) 100 : audit [DBG] from='client.? v1:172.21.15.125:0/3726895907' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:03:54.722 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:54 smithi142 conmon[20125]: cluster 2021-03-24T22:03:54. 2021-03-24T22:03:54.722 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:54 smithi142 conmon[20125]: 354440+0000 mon.a (mon.0) 1772 : cluster [DBG] osdmap e664: 8 total, 7 up, 8 in 2021-03-24T22:03:54.722 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:54 smithi142 conmon[20125]: cluster 2021-03-24T22:03:54.354896+0000 osd.4 2021-03-24T22:03:54.723 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:54 smithi142 conmon[20125]: (osd.4) 117 : cluster [INF] 4.8 continuing backfill to osd.6 from (97'28,544'46] MIN to 544'46 2021-03-24T22:03:54.723 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:54 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:03:54.723 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:54 smithi142 conmon[20125]: T22:03:54.354936+0000 osd.4 (osd.4) 118 : cluster [INF] 4.8 continuing backfill to osd.7 from (97'28,544'46] MIN to 544'46 2021-03-24T22:03:54.723 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:54 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:03:54.723 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:54 smithi142 conmon[20125]: 03-24T22:03:54.358588+0000 osd.6 (osd.6) 55 : cluster [INF] 4.a continuing backfill to osd.1 from (69'23,544'45] 4:5909e671:gc::gc.27:head to 544'45 2021-03-24T22:03:54.770 INFO:tasks.rados.rados.0.smithi125.stdout:2524: expect (ObjNum 736 snap 235 seq_num 736) 2021-03-24T22:03:54.803 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:54.860 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:54.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:54.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:03:54.912 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[24927]: cluster 2021-03-24T22:03 2021-03-24T22:03:54.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[24927]: :52.938800+0000 mgr.y (mgr.14144) 605 : cluster [DBG] pgmap v1156: 168 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+remapped, 2 active+remapped+backfill_wait, 1 peering, 3 active+remapped+backfill_toofull, 155 active+clean; 532 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 4.7 MiB/s wr, 90 op/s; 23/1743 objects degraded (1.320%); 145/1743 objects misplaced (8.319%); 11 MiB/s, 71 objects/s recovering 2021-03-24T22:03:54.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[24927]: cluster 2021-03-24T22:03:53.662462+0000 osd.6 (osd.6) 54 : cluster [DBG] 7.c scrub ok 2021-03-24T22:03:54.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:03:54.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[24927]: 03-24T22:03:54.320183+0000 osd.7 2021-03-24T22:03:54.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[24927]: (osd.7) 107 : cluster [DBG] 3.14 scrub ok 2021-03-24T22:03:54.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[24927]: audit 2021-03-24T22:03:54.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[24927]: 2021-03-24T22:03:54.323791+0000 mon.c (mon.2) 2021-03-24T22:03:54.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[24927]: 100 : audit [DBG] from='client.? v1:172.21.15.125:0/3726895907' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:03:54.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:03:54.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[24927]: 24T22:03:54.354440+0000 mon.a (mon 2021-03-24T22:03:54.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[24927]: .0) 1772 : cluster [DBG] osdmap e664: 8 total, 7 up, 8 in 2021-03-24T22:03:54.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:03:54.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[24927]: 03-24T22:03:54.354896+0000 osd.4 ( 2021-03-24T22:03:54.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[24927]: osd.4) 117 : cluster [INF] 4.8 continuing backfill to osd.6 from (97'28,544'46] MIN to 544'46 2021-03-24T22:03:54.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:03:54.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[24927]: -03-24T22:03:54.354936+0000 2021-03-24T22:03:54.917 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[24927]: osd.4 (osd.4) 118 : cluster [INF] 4.8 continuing backfill to osd.7 from (97'28,544'46] MIN to 544'46 2021-03-24T22:03:54.917 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[24927]: cluster 2021-03-24T22:03:54.917 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[24927]: 2021-03-24T22:03:54.358588 2021-03-24T22:03:54.917 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[24927]: +0000 osd.6 (osd.6) 55 : cluster [INF] 2021-03-24T22:03:54.917 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[24927]: 4.a continuing backfill to osd.1 from (69'23,544'45] 4:5909e671:gc::gc.27:head to 544'45 2021-03-24T22:03:54.918 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:03:54.918 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[19936]: 24T22:03:52.938800+0000 mgr.y (mgr.14144) 605 : cluster [DBG] pgmap v1156: 168 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+remapped, 2 active+remapped+backfill_wait, 1 peering, 3 active+remapped+backfill_toofull, 155 active+clean; 532 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 4.7 MiB/s wr, 90 op/s; 23/1743 objects degraded (1.320%); 145/1743 objects misplaced (8.319%); 11 MiB/s, 71 objects/s recovering 2021-03-24T22:03:54.918 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:03:54.919 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[19936]: 03-24T22:03:53.662462+0000 osd.6 (osd. 2021-03-24T22:03:54.919 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[19936]: 6) 54 : cluster [DBG] 7.c scrub ok 2021-03-24T22:03:54.919 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[19936]: cluster 2021-03-24T22:03:54.920 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[19936]: 2021- 2021-03-24T22:03:54.920 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[19936]: 03-24T22:03:54 2021-03-24T22:03:54.920 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[19936]: .320183+0000 osd.7 2021-03-24T22:03:54.920 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[19936]: (osd.7) 2021-03-24T22:03:54.920 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[19936]: 107 : cluster [DBG] 2021-03-24T22:03:54.921 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[19936]: 3.14 scrub ok 2021-03-24T22:03:54.921 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[19936]: audit 2021-03-24T22:03:54.921 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[19936]: 2021-03-24T22 2021-03-24T22:03:54.921 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[19936]: :03:54.323791 2021-03-24T22:03:54.922 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[19936]: +0000 mon.c ( 2021-03-24T22:03:54.922 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[19936]: mon.2) 100 2021-03-24T22:03:54.922 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[19936]: : audit [DBG] from='client.? v1:172.21.15.125:0/3726895907' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:03:54.922 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[19936]: 2021-03-24T22:03:54.923 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[19936]: cluster 2021-03-24T22:03:54.923 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[19936]: 2021-03-24 2021-03-24T22:03:54.923 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[19936]: T22:03 2021-03-24T22:03:54.923 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[19936]: :54.354440+0000 2021-03-24T22:03:54.923 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[19936]: mon.a (mon. 2021-03-24T22:03:54.924 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[19936]: 0) 1772 : cluster [DBG] 2021-03-24T22:03:54.924 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[19936]: osdmap e664: 8 total, 7 up, 8 in 2021-03-24T22:03:54.924 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[19936]: cluster 2021-03-24T22:03:54.924 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[19936]: 2021-03-24 2021-03-24T22:03:54.925 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[19936]: T22:03: 2021-03-24T22:03:54.925 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[19936]: 54.354896+0000 2021-03-24T22:03:54.925 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[19936]: osd.4 (osd.4 2021-03-24T22:03:54.925 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[19936]: ) 117 : cluster [INF] 2021-03-24T22:03:54.925 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[19936]: 4.8 continuing backfill to osd.6 from (97'28,544'46] MIN to 544'46 2021-03-24T22:03:54.926 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[19936]: cluster 2021-03-24T22:03:54.927 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[19936]: 2021-03-24T 2021-03-24T22:03:54.928 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[19936]: 22:03: 2021-03-24T22:03:54.928 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[19936]: 54.354936+0000 2021-03-24T22:03:54.929 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[19936]: osd.4 (osd. 2021-03-24T22:03:54.929 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[19936]: 4) 118 : cluster 2021-03-24T22:03:54.929 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[19936]: [INF] 4.8 continuing backfill to osd.7 from (97'28,544'46] MIN to 544'46 2021-03-24T22:03:54.930 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[19936]: cluster 2021-03-24T22:03:54.930 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[19936]: 2021-03-24T 2021-03-24T22:03:54.930 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[19936]: 22:03:54 2021-03-24T22:03:54.930 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[19936]: .358588+0000 osd.6 2021-03-24T22:03:54.931 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[19936]: (osd.6) 55 2021-03-24T22:03:54.931 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:54 smithi125 conmon[19936]: : cluster [INF] 4.a continuing backfill to osd.1 from (69'23,544'45] 4:5909e671:gc::gc.27:head to 544'45 2021-03-24T22:03:54.965 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:54.978 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 22:03:54 smithi142 podman[82969]: 2021-03-24 22:03:54.719931977 +0000 UTC m=+0.314819447 container create 172e751ff74a8cb158cfae8136a6da9aa1112d8cc6626e9417972317407da73f (image=quay.ceph.io/ceph-ci/ceph@sha256:e9db06906b8d0a24f3ef6835ac5aa4b8eea2950a2560bad78f83d4f101eca0db, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5-deactivate, io.buildah.version=1.18.0, org.label-schema.name=CentOS Base Image, org.label-schema.build-date=20201204, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d, maintainer=Dimitri Savineau , org.label-schema.schema-version=1.0) 2021-03-24T22:03:54.979 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 22:03:54 smithi142 podman[82969]: 2021-03-24 22:03:54.878202025 +0000 UTC m=+0.473089572 container init 172e751ff74a8cb158cfae8136a6da9aa1112d8cc6626e9417972317407da73f (image=quay.ceph.io/ceph-ci/ceph@sha256:e9db06906b8d0a24f3ef6835ac5aa4b8eea2950a2560bad78f83d4f101eca0db, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Base Image, maintainer=Dimitri Savineau , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.18.0, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20201204, ceph=True, GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06, RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d) 2021-03-24T22:03:54.979 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 22:03:54 smithi142 podman[82969]: 2021-03-24 22:03:54.919861754 +0000 UTC m=+0.514749278 container start 172e751ff74a8cb158cfae8136a6da9aa1112d8cc6626e9417972317407da73f (image=quay.ceph.io/ceph-ci/ceph@sha256:e9db06906b8d0a24f3ef6835ac5aa4b8eea2950a2560bad78f83d4f101eca0db, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.18.0, maintainer=Dimitri Savineau , GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06, RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Base Image, ceph=True, org.label-schema.build-date=20201204, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2021-03-24T22:03:54.979 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 22:03:54 smithi142 podman[82969]: 2021-03-24 22:03:54.920094269 +0000 UTC m=+0.514981834 container attach 172e751ff74a8cb158cfae8136a6da9aa1112d8cc6626e9417972317407da73f (image=quay.ceph.io/ceph-ci/ceph@sha256:e9db06906b8d0a24f3ef6835ac5aa4b8eea2950a2560bad78f83d4f101eca0db, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.18.0, maintainer=Dimitri Savineau , GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20201204, org.label-schema.vendor=CentOS, RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Base Image, ceph=True) 2021-03-24T22:03:55.019 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:55.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:55.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:03:55.125 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:55.182 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:55.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:55.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:55.291 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:55.344 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:55.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:55.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:03:55.362 INFO:tasks.rados.rados.0.smithi125.stdout:2519: done (3 left) 2021-03-24T22:03:55.363 INFO:tasks.rados.rados.0.smithi125.stdout:2521: done (2 left) 2021-03-24T22:03:55.363 INFO:tasks.rados.rados.0.smithi125.stdout:2524: done (1 left) 2021-03-24T22:03:55.363 INFO:tasks.rados.rados.0.smithi125.stdout:2525: done (0 left) 2021-03-24T22:03:55.364 INFO:tasks.rados.rados.0.smithi125.stdout:2526: read oid 36 snap 277 2021-03-24T22:03:55.364 INFO:tasks.rados.rados.0.smithi125.stdout:2526: expect (ObjNum 825 snap 270 seq_num 825) 2021-03-24T22:03:55.364 INFO:tasks.rados.rados.0.smithi125.stdout:2527: read oid 25 snap 260 2021-03-24T22:03:55.364 INFO:tasks.rados.rados.0.smithi125.stdout:2527: expect deleted 2021-03-24T22:03:55.365 INFO:tasks.rados.rados.0.smithi125.stdout:2528: write (excl) oid 44 current snap is 288 2021-03-24T22:03:55.365 INFO:tasks.rados.rados.0.smithi125.stdout:2528: seq_num 878 ranges {756546=649217,2111896=505630} 2021-03-24T22:03:55.365 INFO:tasks.rados.rados.0.smithi125.stdout:2528: writing smithi12556470-44 from 756546 to 1405763 tid 1 2021-03-24T22:03:55.373 INFO:tasks.rados.rados.0.smithi125.stdout:2528: writing smithi12556470-44 from 2111896 to 2617526 tid 2 2021-03-24T22:03:55.374 INFO:tasks.rados.rados.0.smithi125.stdout:2529: snap_remove snap 280 2021-03-24T22:03:55.450 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:55.500 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:55.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:55.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:03:55.605 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:55.616 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 22:03:55 smithi142 podman[82969]: 2021-03-24 22:03:55.28202468 +0000 UTC m=+0.876912165 container died 172e751ff74a8cb158cfae8136a6da9aa1112d8cc6626e9417972317407da73f (image=quay.ceph.io/ceph-ci/ceph@sha256:e9db06906b8d0a24f3ef6835ac5aa4b8eea2950a2560bad78f83d4f101eca0db, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5-deactivate) 2021-03-24T22:03:55.660 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:55.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:55.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:03:55.766 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:55.826 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:55.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:55.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:03:55.932 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:55.984 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:55.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:55.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:03:56.092 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:56.155 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:56.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:56.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:03:56.223 DEBUG:teuthology.orchestra.run.smithi142:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c@osd.5.service' 2021-03-24T22:03:56.262 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:56.318 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:56.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:56.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:56.327 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-03-24T22:03:56.327 INFO:tasks.cephadm.osd.5:Stopped osd.5 2021-03-24T22:03:56.328 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.5 2021-03-24T22:03:56.328 INFO:tasks.thrashosds.thrasher:remote for osd 5 is ubuntu@smithi142.front.sepia.ceph.com 2021-03-24T22:03:56.328 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell -n osd.5 --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph-objectstore-tool --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op list-pgs 2021-03-24T22:03:56.356 INFO:tasks.rados.rados.0.smithi125.stdout:2529: done (3 left) 2021-03-24T22:03:56.357 INFO:tasks.rados.rados.0.smithi125.stdout:2530: copy_from oid 9 from oid 13 current snap is 288 2021-03-24T22:03:56.357 INFO:tasks.rados.rados.0.smithi125.stdout:2526: expect (ObjNum 825 snap 270 seq_num 825) 2021-03-24T22:03:56.425 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:56.434 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:56 smithi125 conmon[19936]: cluster 2021-03-24T22:03:56.434 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:56 smithi125 conmon[19936]: 2021-03-24T22:03:56.434 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:56 smithi125 conmon[19936]: 2021 2021-03-24T22:03:56.434 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:56 smithi125 conmon[19936]: - 2021-03-24T22:03:56.435 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:56 smithi125 conmon[19936]: 03 2021-03-24T22:03:56.435 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:56 smithi125 conmon[19936]: - 2021-03-24T22:03:56.435 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:56 smithi125 conmon[19936]: 24 2021-03-24T22:03:56.435 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:56 smithi125 conmon[19936]: T 2021-03-24T22:03:56.436 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:56 smithi125 conmon[19936]: 22 2021-03-24T22:03:56.436 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:56 smithi125 conmon[19936]: : 2021-03-24T22:03:56.436 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:56 smithi125 conmon[19936]: 03 2021-03-24T22:03:56.436 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:56 smithi125 conmon[19936]: : 2021-03-24T22:03:56.437 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:56 smithi125 conmon[19936]: 54 2021-03-24T22:03:56.437 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:56 smithi125 conmon[19936]: . 2021-03-24T22:03:56.437 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:56 smithi125 conmon[19936]: 939864+0000 mgr.y (mgr.14144) 606 : cluster 2021-03-24T22:03:56.437 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:56 smithi125 conmon[19936]: [DBG] pgmap v1159: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+snaptrim, 2 activating+undersized, 9 stale+active+clean, 1 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 1 active+undersized+degraded, 1 active+remapped+backfill_wait, 23 peering, 2 active+remapped+backfill_toofull, 1 stale+active+remapped+backfill_toofull, 123 active+clean; 541 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 3.5 MiB/s wr, 10 op/s; 31/1764 objects degraded (1.757%); 124/1764 objects misplaced (7.029%); 3.1 MiB/s, 23 objects/s recovering 2021-03-24T22:03:56.438 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:56 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:03:56.438 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:56 smithi125 conmon[19936]: 03-24T22:03:55.350149+0000 mon.a (mon.0) 1773 : cluster [DBG] osdmap e665: 8 total, 7 up, 8 in 2021-03-24T22:03:56.438 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:56 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:03:56.438 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:56 smithi125 conmon[19936]: 03:55.506730+0000 mon.a (mon.0) 1774 : cluster [WRN] 2021-03-24T22:03:56.438 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:56 smithi125 conmon[19936]: Health check failed: Reduced data availability: 4 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2021-03-24T22:03:56.439 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:56 smithi125 conmon[19936]: cluster 2021-03-24T22:03:56.439 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:56 smithi125 conmon[19936]: 2021-03-24T22:03:55.642566+0000 osd.2 (osd.2) 105 : cluster 2021-03-24T22:03:56.439 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:56 smithi125 conmon[19936]: [DBG] 5.a scrub ok 2021-03-24T22:03:56.440 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:56 smithi125 conmon[24927]: cluster 2021-03-24T22:03:56.440 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:56 smithi125 conmon[24927]: 2021-03-24T22:03:56.440 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:56 smithi125 conmon[24927]: 2021 2021-03-24T22:03:56.440 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:56 smithi125 conmon[24927]: - 2021-03-24T22:03:56.441 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:56 smithi125 conmon[24927]: 03 2021-03-24T22:03:56.441 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:56 smithi125 conmon[24927]: - 2021-03-24T22:03:56.441 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:56 smithi125 conmon[24927]: 24 2021-03-24T22:03:56.442 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:56 smithi125 conmon[24927]: T 2021-03-24T22:03:56.442 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:56 smithi125 conmon[24927]: 22 2021-03-24T22:03:56.442 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:56 smithi125 conmon[24927]: :03:54.939864+0000 mgr.y ( 2021-03-24T22:03:56.442 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:56 smithi125 conmon[24927]: mgr.14144) 606 : cluster [DBG] pgmap v1159: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+snaptrim, 2 activating+undersized, 9 stale+active+clean, 1 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 1 active+undersized+degraded, 1 active+remapped+backfill_wait, 23 peering, 2 active+remapped+backfill_toofull, 1 stale+active+remapped+backfill_toofull, 123 active+clean; 541 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 3.5 MiB/s wr, 10 op/s; 31/1764 objects degraded (1.757%); 124/1764 objects misplaced (7.029%); 3.1 MiB/s, 23 objects/s recovering 2021-03-24T22:03:56.443 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:56 smithi125 conmon[24927]: cluster 2021-03-24T22:03:55.350149+0000 mon.a (mon.0 2021-03-24T22:03:56.443 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:56 smithi125 conmon[24927]: ) 1773 : cluster [DBG] osdmap e665: 8 total, 7 up, 8 in 2021-03-24T22:03:56.443 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:56 smithi125 conmon[24927]: cluster 2021-03-24T22:03:55. 2021-03-24T22:03:56.443 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:56 smithi125 conmon[24927]: 506730+0000 mon.a (mon.0) 1774 : cluster [WRN] Health check failed: Reduced data availability: 4 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2021-03-24T22:03:56.443 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:56 smithi125 conmon[24927]: 2021-03-24T22:03:56.444 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:56 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:03:56.444 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:56 smithi125 conmon[24927]: 03-24T22:03:55.642566+0000 osd.2 (osd.2) 105 : 2021-03-24T22:03:56.444 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:56 smithi125 conmon[24927]: cluster [DBG] 5.a scrub ok 2021-03-24T22:03:56.493 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:56.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:56.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:03:56.544 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/osd.5/config 2021-03-24T22:03:56.574 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-03-24T22:03:56.576 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-03-24T22:03:56.595 INFO:tasks.rados.rados.0.smithi125.stdout:2528: finishing write tid 1 to smithi12556470-44 2021-03-24T22:03:56.596 INFO:tasks.rados.rados.0.smithi125.stdout:2528: finishing write tid 2 to smithi12556470-44 2021-03-24T22:03:56.596 INFO:tasks.rados.rados.0.smithi125.stdout:2528: finishing write tid 3 to smithi12556470-44 2021-03-24T22:03:56.596 INFO:tasks.rados.rados.0.smithi125.stdout:2528: finishing write tid 4 to smithi12556470-44 2021-03-24T22:03:56.596 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 44 v 590 (ObjNum 878 snap 288 seq_num 878) dirty exists 2021-03-24T22:03:56.597 INFO:tasks.rados.rados.0.smithi125.stdout:2528: left oid 44 (ObjNum 878 snap 288 seq_num 878) 2021-03-24T22:03:56.597 INFO:tasks.rados.rados.0.smithi125.stdout:2530: finishing copy_from to smithi12556470-9 2021-03-24T22:03:56.597 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 9 v 792 (ObjNum 798 snap 258 seq_num 798) dirty exists 2021-03-24T22:03:56.597 INFO:tasks.rados.rados.0.smithi125.stdout:2530: finishing copy_from racing read to smithi12556470-9 2021-03-24T22:03:56.597 INFO:tasks.rados.rados.0.smithi125.stdout:2526: done (3 left) 2021-03-24T22:03:56.598 INFO:tasks.rados.rados.0.smithi125.stdout:2527: done (2 left) 2021-03-24T22:03:56.598 INFO:tasks.rados.rados.0.smithi125.stdout:2528: done (1 left) 2021-03-24T22:03:56.598 INFO:tasks.rados.rados.0.smithi125.stdout:2530: done (0 left) 2021-03-24T22:03:56.598 INFO:tasks.rados.rados.0.smithi125.stdout:2531: rollback oid 17 current snap is 288 2021-03-24T22:03:56.598 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 17 to 286 2021-03-24T22:03:56.599 INFO:tasks.rados.rados.0.smithi125.stdout:2532: delete oid 19 current snap is 288 2021-03-24T22:03:56.599 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:56.600 INFO:tasks.rados.rados.0.smithi125.stdout:2532: done (1 left) 2021-03-24T22:03:56.600 INFO:tasks.rados.rados.0.smithi125.stdout:2533: write (excl) oid 38 current snap is 288 2021-03-24T22:03:56.601 INFO:tasks.rados.rados.0.smithi125.stdout:2533: seq_num 880 ranges {790160=799715,2112507=509299} 2021-03-24T22:03:56.611 INFO:tasks.rados.rados.0.smithi125.stdout:2533: writing smithi12556470-38 from 790160 to 1589875 tid 1 2021-03-24T22:03:56.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:56 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:03:56.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:56 smithi142 conmon[20125]: -03-24T22:03:54.939864+0000 mgr.y (mgr.14144) 606 : cluster [DBG] pgmap v1159: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+snaptrim, 2 activating+undersized, 9 stale+active+clean, 1 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 1 active+undersized+degraded, 1 active+remapped+backfill_wait, 23 peering, 2 active+remapped+backfill_toofull, 1 stale+active+remapped+backfill_toofull, 123 active+clean; 541 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 3.5 MiB/s wr, 10 op/s; 31/1764 objects degraded (1.757%); 124/1764 objects misplaced (7.029%); 3.1 MiB/s, 23 objects/s recovering 2021-03-24T22:03:56.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:56 smithi142 conmon[20125]: cluster 2021-03-24T22:03:55.350149+0000 mon.a (mon. 2021-03-24T22:03:56.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:56 smithi142 conmon[20125]: 0) 1773 : cluster [DBG] osdmap e665: 8 total, 7 up, 8 in 2021-03-24T22:03:56.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:56 smithi142 conmon[20125]: cluster 2021-03-24T22:03:55 2021-03-24T22:03:56.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:56 smithi142 conmon[20125]: .506730+0000 mon.a (mon.0) 1774 : cluster [WRN] 2021-03-24T22:03:56.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:56 smithi142 conmon[20125]: Health check failed: Reduced data availability: 4 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2021-03-24T22:03:56.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:56 smithi142 conmon[20125]: cluster 2021-03-24T22:03:56.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:56 smithi142 conmon[20125]: 2021-03-24T22:03:55.642566+0000 osd.2 (osd.2) 2021-03-24T22:03:56.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:56 smithi142 conmon[20125]: 105 : cluster [DBG] 5.a scrub ok 2021-03-24T22:03:56.619 INFO:tasks.rados.rados.0.smithi125.stdout:2533: writing smithi12556470-38 from 2112507 to 2621806 tid 2 2021-03-24T22:03:56.620 INFO:tasks.rados.rados.0.smithi125.stdout:2534: rollback oid 19 current snap is 288 2021-03-24T22:03:56.620 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 19 to 251 2021-03-24T22:03:56.620 INFO:tasks.rados.rados.0.smithi125.stdout:2531: finishing rollback tid 0 to smithi12556470-17 2021-03-24T22:03:56.620 INFO:tasks.rados.rados.0.smithi125.stdout:2531: finishing rollback tid 1 to smithi12556470-17 2021-03-24T22:03:56.621 INFO:tasks.rados.rados.0.smithi125.stdout:2531: finishing rollback tid 2 to smithi12556470-17 2021-03-24T22:03:56.621 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 17 v 852 (ObjNum 744 snap 240 seq_num 744) dirty exists 2021-03-24T22:03:56.621 INFO:tasks.rados.rados.0.smithi125.stdout:2533: finishing write tid 1 to smithi12556470-38 2021-03-24T22:03:56.621 INFO:tasks.rados.rados.0.smithi125.stdout:2531: done (2 left) 2021-03-24T22:03:56.622 INFO:tasks.rados.rados.0.smithi125.stdout:2535: write oid 23 current snap is 288 2021-03-24T22:03:56.622 INFO:tasks.rados.rados.0.smithi125.stdout:2535: seq_num 881 ranges {406967=474964,1370989=769310,2677186=129781} 2021-03-24T22:03:56.623 INFO:tasks.rados.rados.0.smithi125.stdout:2535: writing smithi12556470-23 from 406967 to 881931 tid 1 2021-03-24T22:03:56.631 INFO:tasks.rados.rados.0.smithi125.stdout:2535: writing smithi12556470-23 from 1370989 to 2140299 tid 2 2021-03-24T22:03:56.636 INFO:tasks.rados.rados.0.smithi125.stdout:2535: writing smithi12556470-23 from 2677186 to 2806967 tid 3 2021-03-24T22:03:56.636 INFO:tasks.rados.rados.0.smithi125.stdout:2536: rollback oid 13 current snap is 288 2021-03-24T22:03:56.636 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 13 to 282 2021-03-24T22:03:56.637 INFO:tasks.rados.rados.0.smithi125.stdout:2533: finishing write tid 2 to smithi12556470-38 2021-03-24T22:03:56.637 INFO:tasks.rados.rados.0.smithi125.stdout:2534: finishing rollback tid 1 to smithi12556470-19 2021-03-24T22:03:56.637 INFO:tasks.rados.rados.0.smithi125.stdout:2534: finishing rollback tid 2 to smithi12556470-19 2021-03-24T22:03:56.637 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 19 v 608 (ObjNum 773 snap 251 seq_num 773) dirty exists 2021-03-24T22:03:56.638 INFO:tasks.rados.rados.0.smithi125.stdout:2533: finishing write tid 3 to smithi12556470-38 2021-03-24T22:03:56.638 INFO:tasks.rados.rados.0.smithi125.stdout:2533: finishing write tid 4 to smithi12556470-38 2021-03-24T22:03:56.638 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 38 v 728 (ObjNum 880 snap 288 seq_num 880) dirty exists 2021-03-24T22:03:56.639 INFO:tasks.rados.rados.0.smithi125.stdout:2533: left oid 38 (ObjNum 880 snap 288 seq_num 880) 2021-03-24T22:03:56.639 INFO:tasks.rados.rados.0.smithi125.stdout:2535: finishing write tid 1 to smithi12556470-23 2021-03-24T22:03:56.639 INFO:tasks.rados.rados.0.smithi125.stdout:2535: finishing write tid 2 to smithi12556470-23 2021-03-24T22:03:56.639 INFO:tasks.rados.rados.0.smithi125.stdout:2533: done (3 left) 2021-03-24T22:03:56.640 INFO:tasks.rados.rados.0.smithi125.stdout:2534: done (2 left) 2021-03-24T22:03:56.640 INFO:tasks.rados.rados.0.smithi125.stdout:2537: write oid 19 current snap is 288 2021-03-24T22:03:56.640 INFO:tasks.rados.rados.0.smithi125.stdout:2537: seq_num 882 ranges {423774=666566,1545824=512968,2704609=119165} 2021-03-24T22:03:56.642 INFO:tasks.rados.rados.0.smithi125.stdout:2537: writing smithi12556470-19 from 423774 to 1090340 tid 1 2021-03-24T22:03:56.643 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:56.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:56.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:03:56.648 INFO:tasks.rados.rados.0.smithi125.stdout:2537: writing smithi12556470-19 from 1545824 to 2058792 tid 2 2021-03-24T22:03:56.653 INFO:tasks.rados.rados.0.smithi125.stdout:2537: writing smithi12556470-19 from 2704609 to 2823774 tid 3 2021-03-24T22:03:56.653 INFO:tasks.rados.rados.0.smithi125.stdout:2538: snap_create 2021-03-24T22:03:56.653 INFO:tasks.rados.rados.0.smithi125.stdout:2535: finishing write tid 3 to smithi12556470-23 2021-03-24T22:03:56.654 INFO:tasks.rados.rados.0.smithi125.stdout:2535: finishing write tid 4 to smithi12556470-23 2021-03-24T22:03:56.654 INFO:tasks.rados.rados.0.smithi125.stdout:2535: finishing write tid 5 to smithi12556470-23 2021-03-24T22:03:56.654 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 23 v 442 (ObjNum 881 snap 288 seq_num 881) dirty exists 2021-03-24T22:03:56.654 INFO:tasks.rados.rados.0.smithi125.stdout:2535: left oid 23 (ObjNum 881 snap 288 seq_num 881) 2021-03-24T22:03:56.655 INFO:tasks.rados.rados.0.smithi125.stdout:2536: finishing rollback tid 0 to smithi12556470-13 2021-03-24T22:03:56.655 INFO:tasks.rados.rados.0.smithi125.stdout:2536: finishing rollback tid 1 to smithi12556470-13 2021-03-24T22:03:56.655 INFO:tasks.rados.rados.0.smithi125.stdout:2536: finishing rollback tid 2 to smithi12556470-13 2021-03-24T22:03:56.655 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 13 v 795 (ObjNum 798 snap 258 seq_num 798) dirty exists 2021-03-24T22:03:56.655 INFO:tasks.rados.rados.0.smithi125.stdout:2537: finishing write tid 1 to smithi12556470-19 2021-03-24T22:03:56.656 INFO:tasks.rados.rados.0.smithi125.stdout:2537: finishing write tid 2 to smithi12556470-19 2021-03-24T22:03:56.657 INFO:tasks.rados.rados.0.smithi125.stdout:2537: finishing write tid 3 to smithi12556470-19 2021-03-24T22:03:56.657 INFO:tasks.rados.rados.0.smithi125.stdout:2537: finishing write tid 4 to smithi12556470-19 2021-03-24T22:03:56.657 INFO:tasks.rados.rados.0.smithi125.stdout:2537: finishing write tid 5 to smithi12556470-19 2021-03-24T22:03:56.657 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 19 v 612 (ObjNum 882 snap 288 seq_num 882) dirty exists 2021-03-24T22:03:56.657 INFO:tasks.rados.rados.0.smithi125.stdout:2537: left oid 19 (ObjNum 882 snap 288 seq_num 882) 2021-03-24T22:03:56.748 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:56.803 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:56.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:56.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:56.909 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:56.968 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:56.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:56.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:03:57.072 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:57.146 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:57.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:57.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:03:57.253 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:57.340 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:57.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:57.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:03:57.390 INFO:tasks.rados.rados.0.smithi125.stdout:2535: done (3 left) 2021-03-24T22:03:57.391 INFO:tasks.rados.rados.0.smithi125.stdout:2536: done (2 left) 2021-03-24T22:03:57.391 INFO:tasks.rados.rados.0.smithi125.stdout:2537: done (1 left) 2021-03-24T22:03:57.391 INFO:tasks.rados.rados.0.smithi125.stdout:2538: done (0 left) 2021-03-24T22:03:57.392 INFO:tasks.rados.rados.0.smithi125.stdout:2539: write (excl) oid 3 current snap is 289 2021-03-24T22:03:57.392 INFO:tasks.rados.rados.0.smithi125.stdout:2539: seq_num 883 ranges {440581=741815,1604306=656626,2732032=108549} 2021-03-24T22:03:57.393 INFO:tasks.rados.rados.0.smithi125.stdout:2539: writing smithi12556470-3 from 440581 to 1182396 tid 1 2021-03-24T22:03:57.398 INFO:tasks.rados.rados.0.smithi125.stdout:2539: writing smithi12556470-3 from 1604306 to 2260932 tid 2 2021-03-24T22:03:57.401 INFO:tasks.rados.rados.0.smithi125.stdout:2539: writing smithi12556470-3 from 2732032 to 2840581 tid 3 2021-03-24T22:03:57.402 INFO:tasks.rados.rados.0.smithi125.stdout:2540: snap_create 2021-03-24T22:03:57.402 INFO:tasks.rados.rados.0.smithi125.stdout:2539: finishing write tid 1 to smithi12556470-3 2021-03-24T22:03:57.404 INFO:tasks.rados.rados.0.smithi125.stdout:2539: finishing write tid 2 to smithi12556470-3 2021-03-24T22:03:57.406 INFO:tasks.rados.rados.0.smithi125.stdout:2539: finishing write tid 3 to smithi12556470-3 2021-03-24T22:03:57.406 INFO:tasks.rados.rados.0.smithi125.stdout:2539: finishing write tid 4 to smithi12556470-3 2021-03-24T22:03:57.406 INFO:tasks.rados.rados.0.smithi125.stdout:2539: finishing write tid 5 to smithi12556470-3 2021-03-24T22:03:57.407 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 3 v 311 (ObjNum 883 snap 289 seq_num 883) dirty exists 2021-03-24T22:03:57.407 INFO:tasks.rados.rados.0.smithi125.stdout:2539: left oid 3 (ObjNum 883 snap 289 seq_num 883) 2021-03-24T22:03:57.449 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:57.514 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:57.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:57.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:57.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:57 smithi142 conmon[20125]: cluster 2021-03-24T22: 2021-03-24T22:03:57.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:57 smithi142 conmon[20125]: 03:55.671088+0000 osd.6 (osd.6) 56 : cluster [DBG] 7.19 scrub ok 2021-03-24T22:03:57.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:57 smithi142 conmon[20125]: cluster 2021-03-24T22:03:57.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:57 smithi142 conmon[20125]: 2021-03-24T22:03:56.352430+0000 mon.a 2021-03-24T22:03:57.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:57 smithi142 conmon[20125]: (mon.0) 1775 : cluster 2021-03-24T22:03:57.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:57 smithi142 conmon[20125]: [DBG] osdmap e666: 8 total, 7 up, 8 in 2021-03-24T22:03:57.622 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:57.636 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:57 smithi125 conmon[19936]: cluster 2021-03-24T22:03:55.671088+0000 osd.6 (osd 2021-03-24T22:03:57.636 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:57 smithi125 conmon[19936]: .6) 56 : cluster [DBG] 7.19 scrub ok 2021-03-24T22:03:57.637 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:57 smithi125 conmon[19936]: cluster 2021-03-24T22:03:56.352430+0000 mon.a (mon.0) 1775 : cluster [DBG] osdmap e666: 8 total, 7 up, 8 in 2021-03-24T22:03:57.637 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:57 smithi125 conmon[24927]: cluster 2021-03-24T22:03:55.671088+0000 2021-03-24T22:03:57.637 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:57 smithi125 conmon[24927]: osd.6 (osd.6) 56 : cluster [DBG] 7.19 scrub ok 2021-03-24T22:03:57.638 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:57 smithi125 conmon[24927]: cluster 2021-03-24T22:03:56.352430+0000 mon.a ( 2021-03-24T22:03:57.638 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:57 smithi125 conmon[24927]: mon.0) 1775 : cluster [DBG] osdmap e666: 8 total, 7 up, 8 in 2021-03-24T22:03:57.679 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:57.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:57.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:03:57.785 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:57.838 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:03:57.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:57.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:03:57.944 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:57.994 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:57.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:57.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:03:58.098 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:58.152 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:58.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:58.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:03:58.258 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:58.308 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:58.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:58.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:03:58.391 INFO:tasks.rados.rados.0.smithi125.stdout:2539: done (1 left) 2021-03-24T22:03:58.391 INFO:tasks.rados.rados.0.smithi125.stdout:2540: done (0 left) 2021-03-24T22:03:58.391 INFO:tasks.rados.rados.0.smithi125.stdout:2541: delete oid 48 current snap is 290 2021-03-24T22:03:58.402 INFO:tasks.rados.rados.0.smithi125.stdout:2541: done (0 left) 2021-03-24T22:03:58.402 INFO:tasks.rados.rados.0.smithi125.stdout:2542: copy_from oid 21 from oid 40 current snap is 290 2021-03-24T22:03:58.402 INFO:tasks.rados.rados.0.smithi125.stdout:2543: rollback oid 24 current snap is 290 2021-03-24T22:03:58.402 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 24 to 289 2021-03-24T22:03:58.403 INFO:tasks.rados.rados.0.smithi125.stdout:2544: rollback oid 32 current snap is 290 2021-03-24T22:03:58.403 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 32 to 281 2021-03-24T22:03:58.403 INFO:tasks.rados.rados.0.smithi125.stdout:2545: write (excl) oid 25 current snap is 290 2021-03-24T22:03:58.410 INFO:tasks.rados.rados.0.smithi125.stdout:2545: seq_num 885 ranges {474195=492313,1604917=660295,2670525=203670} 2021-03-24T22:03:58.410 INFO:tasks.rados.rados.0.smithi125.stdout:2545: writing smithi12556470-25 from 474195 to 966508 tid 1 2021-03-24T22:03:58.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[24927]: cluster 2021-03-24T22:03:56. 2021-03-24T22:03:58.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[24927]: 693891+0000 osd.6 (osd.6) 57 : cluster [DBG] 4.e scrub ok 2021-03-24T22:03:58.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:03:58.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[24927]: 24T22:03:56.941102+0000 mgr.y (mgr.14144) 607 : cluster [DBG] pgmap v1162: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+clean+snaptrim_wait, 1 active+clean+snaptrim, 2 activating+undersized, 25 active+undersized, 10 active+undersized+degraded, 23 peering, 2 active+remapped+backfill_toofull, 1 stale+active+remapped+backfill_toofull, 101 active+clean; 547 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 9.6 MiB/s rd, 2.6 MiB/s wr, 16 op/s; 89/1767 objects degraded (5.037%); 80/1767 objects misplaced (4.527%); 10 MiB/s, 23 objects/s recovering 2021-03-24T22:03:58.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[24927]: cluster 2021-03-24T22:03:56.955159+0000 osd.1 (osd.1) 64 : cluster 2021-03-24T22:03:58.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[24927]: [DBG] 5.c scrub ok 2021-03-24T22:03:58.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[24927]: cluster 2021-03-24T22:03:58.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[24927]: 2021-03-24T22:03:57.076533+0000 osd.0 (osd.0) 130 : cluster [DBG] 6.16 scrub ok 2021-03-24T22:03:58.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T22:03:58.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[24927]: T22:03:57.292839+0000 osd.7 (osd.7) 108 : cluster [DBG] 4.8 scrub ok 2021-03-24T22:03:58.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[24927]: 2021-03-24T22:03:58.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[24927]: cluster 2021-03-24T22:03:57 2021-03-24T22:03:58.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[24927]: .355920+0000 mon.a (mon.0) 1776 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:03:58.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:03:58.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[24927]: -03-24T22:03:57.355989+0000 mon.a (mon.0) 1777 : cluster [WRN] Health check update: Degraded data redundancy: 89/1767 objects degraded (5.037%), 11 pgs degraded (PG_DEGRADED) 2021-03-24T22:03:58.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[24927]: cluster 2021-03-24T22:03:57.373814+0000 mon.a 2021-03-24T22:03:58.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[24927]: (mon.0) 1778 : cluster [DBG] osdmap e667: 8 total, 7 up, 8 in 2021-03-24T22:03:58.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[24927]: audit 2021-03-24T22:03:58.257400+0000 mon.a (mon.0 2021-03-24T22:03:58.416 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[24927]: ) 1779 : audit [INF] from='client.? v1:172.21.15.125:0/2132982102' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T22:03:58.417 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:58.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T22:03:58.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[19936]: -24T22:03:56.693891+0000 osd.6 (osd.6 2021-03-24T22:03:58.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[19936]: ) 57 : cluster [DBG] 4.e scrub ok 2021-03-24T22:03:58.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[19936]: 2021-03-24T22:03:58.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[19936]: cluster 2021-03-24T22:03:58.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[19936]: 2021-03 2021-03-24T22:03:58.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[19936]: -24T22:03:56. 2021-03-24T22:03:58.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[19936]: 941102+0000 mgr.y (mgr 2021-03-24T22:03:58.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[19936]: .14144) 607 : cluster 2021-03-24T22:03:58.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[19936]: [DBG] pgmap v1162: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+clean+snaptrim_wait, 1 active+clean+snaptrim, 2 activating+undersized, 25 active+undersized, 10 active+undersized+degraded, 23 peering, 2 active+remapped+backfill_toofull, 1 stale+active+remapped+backfill_toofull, 101 active+clean; 547 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 9.6 MiB/s rd, 2.6 MiB/s wr, 16 op/s; 89/1767 objects degraded (5.037%); 80/1767 objects misplaced (4.527%); 10 MiB/s, 23 objects/s recovering 2021-03-24T22:03:58.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[19936]: cluster 2021-03-24T22:03:58.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[19936]: 2021-03-24T22 2021-03-24T22:03:58.421 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[19936]: :03:56.955159 2021-03-24T22:03:58.421 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[19936]: +0000 osd.1 (osd.1 2021-03-24T22:03:58.421 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[19936]: ) 64 : cluster [DBG] 5.c scrub ok 2021-03-24T22:03:58.422 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[19936]: 2021-03-24T22:03:58.422 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[19936]: cluster 2021-03-24T22:03:58.422 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[19936]: 2021-03-24 2021-03-24T22:03:58.422 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[19936]: T22:03:57 2021-03-24T22:03:58.423 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[19936]: .076533+0000 osd.0 ( 2021-03-24T22:03:58.423 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[19936]: osd.0) 130 : cluster [DBG] 2021-03-24T22:03:58.423 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[19936]: 6.16 scrub ok 2021-03-24T22:03:58.423 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:03:58.423 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[19936]: -03-24T22:03 2021-03-24T22:03:58.424 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[19936]: :57.292839+0000 osd.7 ( 2021-03-24T22:03:58.424 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[19936]: osd.7) 108 : cluster [DBG] 2021-03-24T22:03:58.424 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[19936]: 4.8 scrub ok 2021-03-24T22:03:58.424 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[19936]: cluster 2021-03-24T22:03:58.425 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[19936]: 2021-03-24T22 2021-03-24T22:03:58.425 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[19936]: :03:57.355920+0000 2021-03-24T22:03:58.425 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[19936]: mon.a (mon.0) 2021-03-24T22:03:58.426 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[19936]: 1776 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:03:58.426 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[19936]: 2021-03-24T22:03:58.426 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[19936]: cluster 2021-03-24T22:03:58.426 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[19936]: 2021-03-24T22 2021-03-24T22:03:58.426 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[19936]: :03:57.355989+0000 mon.a 2021-03-24T22:03:58.427 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[19936]: (mon.0) 1777 : cluster [WRN] 2021-03-24T22:03:58.427 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[19936]: Health check update: Degraded data redundancy: 89/1767 objects degraded (5.037%), 11 pgs degraded (PG_DEGRADED) 2021-03-24T22:03:58.427 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T22:03:58.427 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[19936]: 22:03:57.373814+0000 2021-03-24T22:03:58.428 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[19936]: mon.a (mon.0) 1778 : cluster 2021-03-24T22:03:58.428 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[19936]: [DBG] osdmap e667: 8 total, 7 up, 8 in 2021-03-24T22:03:58.428 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[19936]: audit 2021 2021-03-24T22:03:58.428 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[19936]: -03-24T22:03: 2021-03-24T22:03:58.428 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[19936]: 58.257400+0000 mon.a (mon 2021-03-24T22:03:58.429 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[19936]: .0) 1779 : audit [INF] 2021-03-24T22:03:58.429 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:58 smithi125 conmon[19936]: from='client.? v1:172.21.15.125:0/2132982102' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T22:03:58.429 INFO:tasks.rados.rados.0.smithi125.stdout:2545: writing smithi12556470-25 from 1604917 to 2265212 tid 2 2021-03-24T22:03:58.430 INFO:teuthology.orchestra.run.smithi125.stderr:noscrub is set 2021-03-24T22:03:58.432 INFO:tasks.rados.rados.0.smithi125.stdout:2545: writing smithi12556470-25 from 2670525 to 2874195 tid 3 2021-03-24T22:03:58.432 INFO:tasks.rados.rados.0.smithi125.stdout:2546: rollback oid 23 current snap is 290 2021-03-24T22:03:58.432 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 23 to 288 2021-03-24T22:03:58.433 INFO:tasks.rados.rados.0.smithi125.stdout:2543: finishing rollback tid 1 to smithi12556470-24 2021-03-24T22:03:58.433 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 24 v 1009 (ObjNum 1117586368 snap 21916 seq_num 1118594496) dirty dne 2021-03-24T22:03:58.433 INFO:tasks.rados.rados.0.smithi125.stdout:2544: finishing rollback tid 1 to smithi12556470-32 2021-03-24T22:03:58.433 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 32 v 1447 (ObjNum 1117552112 snap 21916 seq_num 1116454656) dirty dne 2021-03-24T22:03:58.434 INFO:tasks.rados.rados.0.smithi125.stdout:2545: finishing write tid 1 to smithi12556470-25 2021-03-24T22:03:58.434 INFO:tasks.rados.rados.0.smithi125.stdout:2543: done (4 left) 2021-03-24T22:03:58.434 INFO:tasks.rados.rados.0.smithi125.stdout:2544: done (3 left) 2021-03-24T22:03:58.434 INFO:tasks.rados.rados.0.smithi125.stdout:2547: snap_remove snap 274 2021-03-24T22:03:58.503 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:03:58.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:58.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:03:58.533 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-24T22:03:58.534 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T22:03:58.576 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-24T22:03:58.613 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:58.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:58 smithi142 conmon[20125]: cluster 2021-03-24T22:03:56 2021-03-24T22:03:58.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:58 smithi142 conmon[20125]: .693891+0000 osd.6 (osd.6) 57 : cluster [DBG] 4.e scrub ok 2021-03-24T22:03:58.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:58 smithi142 conmon[20125]: cluster 2021-03-24T22:03:58.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:58 smithi142 conmon[20125]: 2021-03-24T22:03:58.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:58 smithi142 conmon[20125]: 2021-03-24T22:03 2021-03-24T22:03:58.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:58 smithi142 conmon[20125]: :56.941102+0000 mgr.y (mgr 2021-03-24T22:03:58.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:58 smithi142 conmon[20125]: .14144) 607 : cluster [DBG] pgmap v1162: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+clean+snaptrim_wait, 1 active+clean+snaptrim, 2 activating+undersized, 25 active+undersized, 10 active+undersized+degraded, 23 peering, 2 active+remapped+backfill_toofull, 1 stale+active+remapped+backfill_toofull, 101 active+clean; 547 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 9.6 MiB/s rd, 2.6 MiB/s wr, 16 op/s; 89/1767 objects degraded (5.037%); 80/1767 objects misplaced (4.527%); 10 MiB/s, 23 objects/s recovering 2021-03-24T22:03:58.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:58 smithi142 conmon[20125]: 2021-03-24T22:03:58.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:58 smithi142 conmon[20125]: cluster 2021-03-24T22:03:58.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:58 smithi142 conmon[20125]: 2021-03-24T22:03: 2021-03-24T22:03:58.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:58 smithi142 conmon[20125]: 56.955159+0000 osd.1 (osd.1 2021-03-24T22:03:58.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:58 smithi142 conmon[20125]: ) 64 : cluster [DBG] 5.c scrub ok 2021-03-24T22:03:58.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:58 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:03:58.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:58 smithi142 conmon[20125]: 03-24T22:03:57.076533 2021-03-24T22:03:58.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:58 smithi142 conmon[20125]: +0000 osd.0 (osd.0) 130 : 2021-03-24T22:03:58.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:58 smithi142 conmon[20125]: cluster [DBG] 6.16 scrub ok 2021-03-24T22:03:58.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:58 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:03:58.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:58 smithi142 conmon[20125]: 03-24T22:03:57 2021-03-24T22:03:58.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:58 smithi142 conmon[20125]: .292839+0000 osd.7 (osd.7) 2021-03-24T22:03:58.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:58 smithi142 conmon[20125]: 108 : cluster [DBG] 4.8 scrub ok 2021-03-24T22:03:58.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:58 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:03:58.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:58 smithi142 conmon[20125]: -03-24T22:03:57. 2021-03-24T22:03:58.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:58 smithi142 conmon[20125]: 355920+0000 mon.a (mon.0) 1776 2021-03-24T22:03:58.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:58 smithi142 conmon[20125]: : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:03:58.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:58 smithi142 conmon[20125]: cluster 2021-03-24T22:03:58.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:58 smithi142 conmon[20125]: 2021-03-24T22:03:57 2021-03-24T22:03:58.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:58 smithi142 conmon[20125]: .355989+0000 mon.a (mon.0) 2021-03-24T22:03:58.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:58 smithi142 conmon[20125]: 1777 : cluster [WRN] Health check update: Degraded data redundancy: 89/1767 objects degraded (5.037%), 11 pgs degraded (PG_DEGRADED) 2021-03-24T22:03:58.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:58 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:03:58.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:58 smithi142 conmon[20125]: -03-24T22:03:57 2021-03-24T22:03:58.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:58 smithi142 conmon[20125]: .373814+0000 mon.a (mon.0) 2021-03-24T22:03:58.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:58 smithi142 conmon[20125]: 1778 : cluster [DBG] osdmap e667: 8 total, 7 up, 8 in 2021-03-24T22:03:58.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:58 smithi142 conmon[20125]: audit 2021 2021-03-24T22:03:58.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:58 smithi142 conmon[20125]: -03-24T22:03:58.257400 2021-03-24T22:03:58.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:58 smithi142 conmon[20125]: +0000 mon.a (mon.0) 1779 : 2021-03-24T22:03:58.625 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:58 smithi142 conmon[20125]: audit [INF] from='client.? v1:172.21.15.125:0/2132982102' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T22:03:58.677 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:03:58.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:58.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:03:58.783 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:58.837 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:58.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:58.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:03:58.849 INFO:teuthology.orchestra.run.smithi142.stdout:7.e 2021-03-24T22:03:58.850 INFO:teuthology.orchestra.run.smithi142.stdout:7.b 2021-03-24T22:03:58.850 INFO:teuthology.orchestra.run.smithi142.stdout:7.1e 2021-03-24T22:03:58.850 INFO:teuthology.orchestra.run.smithi142.stdout:7.1c 2021-03-24T22:03:58.850 INFO:teuthology.orchestra.run.smithi142.stdout:7.1a 2021-03-24T22:03:58.851 INFO:teuthology.orchestra.run.smithi142.stdout:6.d 2021-03-24T22:03:58.851 INFO:teuthology.orchestra.run.smithi142.stdout:7.1d 2021-03-24T22:03:58.851 INFO:teuthology.orchestra.run.smithi142.stdout:6.c 2021-03-24T22:03:58.851 INFO:teuthology.orchestra.run.smithi142.stdout:6.1e 2021-03-24T22:03:58.851 INFO:teuthology.orchestra.run.smithi142.stdout:6.1c 2021-03-24T22:03:58.852 INFO:teuthology.orchestra.run.smithi142.stdout:6.7 2021-03-24T22:03:58.852 INFO:teuthology.orchestra.run.smithi142.stdout:6.11 2021-03-24T22:03:58.852 INFO:teuthology.orchestra.run.smithi142.stdout:6.17 2021-03-24T22:03:58.852 INFO:teuthology.orchestra.run.smithi142.stdout:6.10 2021-03-24T22:03:58.853 INFO:teuthology.orchestra.run.smithi142.stdout:5.f 2021-03-24T22:03:58.853 INFO:teuthology.orchestra.run.smithi142.stdout:3.d 2021-03-24T22:03:58.853 INFO:teuthology.orchestra.run.smithi142.stdout:6.1f 2021-03-24T22:03:58.853 INFO:teuthology.orchestra.run.smithi142.stdout:3.4 2021-03-24T22:03:58.853 INFO:teuthology.orchestra.run.smithi142.stdout:3.1c 2021-03-24T22:03:58.854 INFO:teuthology.orchestra.run.smithi142.stdout:3.c 2021-03-24T22:03:58.854 INFO:teuthology.orchestra.run.smithi142.stdout:4.8 2021-03-24T22:03:58.854 INFO:teuthology.orchestra.run.smithi142.stdout:3.2 2021-03-24T22:03:58.854 INFO:teuthology.orchestra.run.smithi142.stdout:5.16 2021-03-24T22:03:58.855 INFO:teuthology.orchestra.run.smithi142.stdout:3.22 2021-03-24T22:03:58.855 INFO:teuthology.orchestra.run.smithi142.stdout:7.17 2021-03-24T22:03:58.855 INFO:teuthology.orchestra.run.smithi142.stdout:6.1a 2021-03-24T22:03:58.855 INFO:teuthology.orchestra.run.smithi142.stdout:3.21 2021-03-24T22:03:58.855 INFO:teuthology.orchestra.run.smithi142.stdout:7.a 2021-03-24T22:03:58.856 INFO:teuthology.orchestra.run.smithi142.stdout:4.15 2021-03-24T22:03:58.856 INFO:teuthology.orchestra.run.smithi142.stdout:3.1f 2021-03-24T22:03:58.856 INFO:teuthology.orchestra.run.smithi142.stdout:5.7 2021-03-24T22:03:58.856 INFO:teuthology.orchestra.run.smithi142.stdout:3.5 2021-03-24T22:03:58.856 INFO:teuthology.orchestra.run.smithi142.stdout:3.20 2021-03-24T22:03:58.857 INFO:teuthology.orchestra.run.smithi142.stdout:3.10 2021-03-24T22:03:58.857 INFO:teuthology.orchestra.run.smithi142.stdout:7.14 2021-03-24T22:03:58.857 INFO:teuthology.orchestra.run.smithi142.stdout:2.2 2021-03-24T22:03:58.857 INFO:teuthology.orchestra.run.smithi142.stdout:3.16 2021-03-24T22:03:58.858 INFO:teuthology.orchestra.run.smithi142.stdout:7.10 2021-03-24T22:03:58.858 INFO:teuthology.orchestra.run.smithi142.stdout:5.1b 2021-03-24T22:03:58.858 INFO:teuthology.orchestra.run.smithi142.stdout:3.1d 2021-03-24T22:03:58.858 INFO:teuthology.orchestra.run.smithi142.stdout:4.1 2021-03-24T22:03:58.859 INFO:teuthology.orchestra.run.smithi142.stdout:6.a 2021-03-24T22:03:58.859 INFO:teuthology.orchestra.run.smithi142.stdout:5.2 2021-03-24T22:03:58.859 INFO:teuthology.orchestra.run.smithi142.stdout:4.3 2021-03-24T22:03:58.859 INFO:teuthology.orchestra.run.smithi142.stdout:7.0 2021-03-24T22:03:58.859 INFO:teuthology.orchestra.run.smithi142.stdout:4.7 2021-03-24T22:03:58.860 INFO:teuthology.orchestra.run.smithi142.stdout:3.17 2021-03-24T22:03:58.860 INFO:teuthology.orchestra.run.smithi142.stdout:4.a 2021-03-24T22:03:58.860 INFO:teuthology.orchestra.run.smithi142.stdout:5.e 2021-03-24T22:03:58.860 INFO:teuthology.orchestra.run.smithi142.stdout:3.28 2021-03-24T22:03:58.860 INFO:teuthology.orchestra.run.smithi142.stdout:5.12 2021-03-24T22:03:58.861 INFO:teuthology.orchestra.run.smithi142.stdout:5.15 2021-03-24T22:03:58.861 INFO:teuthology.orchestra.run.smithi142.stdout:7.21 2021-03-24T22:03:58.861 INFO:teuthology.orchestra.run.smithi142.stdout:5.3 2021-03-24T22:03:58.861 INFO:teuthology.orchestra.run.smithi142.stdout:5.19 2021-03-24T22:03:58.862 INFO:teuthology.orchestra.run.smithi142.stdout:6.19 2021-03-24T22:03:58.862 INFO:teuthology.orchestra.run.smithi142.stdout:5.4 2021-03-24T22:03:58.862 INFO:teuthology.orchestra.run.smithi142.stdout:4.2 2021-03-24T22:03:58.862 INFO:teuthology.orchestra.run.smithi142.stdout:5.6 2021-03-24T22:03:58.862 INFO:teuthology.orchestra.run.smithi142.stdout:5.b 2021-03-24T22:03:58.863 INFO:teuthology.orchestra.run.smithi142.stdout:6.1d 2021-03-24T22:03:58.863 INFO:teuthology.orchestra.run.smithi142.stdout:5.d 2021-03-24T22:03:58.942 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:58.997 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:03:59.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:59.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:03:59.104 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:59.161 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:59.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:59.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:03:59.267 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:59.325 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:59.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:59.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:03:59.401 INFO:tasks.rados.rados.0.smithi125.stdout:2547: done (3 left) 2021-03-24T22:03:59.401 INFO:tasks.rados.rados.0.smithi125.stdout:2548: snap_remove snap 285 2021-03-24T22:03:59.432 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:59.479 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:59.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:59.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:03:59.584 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:59.593 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:59 smithi142 conmon[20125]: cluster 2021-03-24T22: 2021-03-24T22:03:59.594 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:59 smithi142 conmon[20125]: 03:57.739276+0000 osd.6 (osd.6) 58 : cluster [DBG] 7.1f scrub ok 2021-03-24T22:03:59.594 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:59 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:03:59.595 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:59 smithi142 conmon[20125]: 03-24T22:03:58.240259+0000 osd.7 (osd.7) 109 2021-03-24T22:03:59.595 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:59 smithi142 conmon[20125]: : cluster [DBG] 3.f scrub ok 2021-03-24T22:03:59.596 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:59 smithi142 conmon[20125]: cluster 2021-03-24T22 2021-03-24T22:03:59.596 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:59 smithi142 conmon[20125]: :03:58.368808+0000 mon.a (mon. 2021-03-24T22:03:59.596 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:59 smithi142 conmon[20125]: 0) 1780 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:03:59.597 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:59 smithi142 conmon[20125]: audit 2021-03- 2021-03-24T22:03:59.597 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:59 smithi142 conmon[20125]: 24T22:03:58.385715+0000 mon.a 2021-03-24T22:03:59.597 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:59 smithi142 conmon[20125]: (mon.0) 1781 : audit [INF] from='client.? v1:172.21.15.125:0/2132982102' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T22:03:59.597 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:59 smithi142 conmon[20125]: 2021-03-24T22:03:59.597 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:59 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:03:59.598 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:59 smithi142 conmon[20125]: 03-24T22:03:58.385789+0000 mon.a (mon 2021-03-24T22:03:59.598 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:59 smithi142 conmon[20125]: .0) 1782 : cluster [DBG] osdmap e668: 8 total, 7 up, 8 in 2021-03-24T22:03:59.598 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:59 smithi142 conmon[20125]: cluster 2021-03-24T22:03:59.598 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:59 smithi142 conmon[20125]: 2021-03-24T22:03:58. 2021-03-24T22:03:59.599 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:59 smithi142 conmon[20125]: 697445+0000 osd.2 (osd.2) 106 : cluster 2021-03-24T22:03:59.599 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:59 smithi142 conmon[20125]: [DBG] purged_snaps scrub starts 2021-03-24T22:03:59.599 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:59 smithi142 conmon[20125]: cluster 2021-03-24T22:03:59.599 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:59 smithi142 conmon[20125]: 2021-03-24T22: 2021-03-24T22:03:59.599 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:59 smithi142 conmon[20125]: 03:58.699886+0000 osd.2 (osd.2 2021-03-24T22:03:59.600 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:03:59 smithi142 conmon[20125]: ) 107 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:03:59.641 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:03:59.643 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:59 smithi125 conmon[19936]: cluster 2021-03-24T22:03: 2021-03-24T22:03:59.644 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:59 smithi125 conmon[19936]: 57.739276+0000 osd.6 (osd.6) 58 : cluster [DBG] 7.1f scrub ok 2021-03-24T22:03:59.644 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:59 smithi125 conmon[19936]: cluster 2021-03-24T22:03:58.240259+0000 2021-03-24T22:03:59.644 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:59 smithi125 conmon[19936]: osd.7 (osd.7) 109 : cluster [DBG] 3.f scrub ok 2021-03-24T22:03:59.645 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:59 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:03:59.645 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:59 smithi125 conmon[19936]: 03:58.368808+0000 mon.a (mon.0) 1780 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:03:59.645 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:59 smithi125 conmon[19936]: audit 2021-03-24T22:03:58.385715+0000 2021-03-24T22:03:59.645 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:59 smithi125 conmon[19936]: mon.a (mon.0) 1781 : audit [INF] from='client.? v1:172.21.15.125:0/2132982102' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T22:03:59.646 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:59 smithi125 conmon[19936]: cluster 2021-03-24T22:03:58. 2021-03-24T22:03:59.646 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:59 smithi125 conmon[19936]: 385789+0000 mon.a (mon.0) 1782 : cluster [DBG] osdmap e668: 8 total, 7 up, 8 in 2021-03-24T22:03:59.646 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:59 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:03:59.647 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:59 smithi125 conmon[19936]: T22:03:58.697445+0000 osd.2 (osd.2) 106 2021-03-24T22:03:59.647 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:59 smithi125 conmon[19936]: : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:03:59.647 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:59 smithi125 conmon[19936]: cluster 2021-03-24T22:03:59.648 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:59 smithi125 conmon[19936]: 2021-03-24T22:03:58.699886+0000 osd.2 (osd.2) 107 : cluster 2021-03-24T22:03:59.648 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:03:59 smithi125 conmon[19936]: [DBG] purged_snaps scrub ok 2021-03-24T22:03:59.650 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:59 smithi125 conmon[24927]: cluster 2021-03-24T22:03:57 2021-03-24T22:03:59.650 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:59 smithi125 conmon[24927]: .739276+0000 osd.6 (osd.6) 58 : cluster [DBG] 7.1f scrub ok 2021-03-24T22:03:59.651 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:59 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T22:03:59.651 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:59 smithi125 conmon[24927]: 22:03:58.240259+0000 osd.7 (osd.7) 109 : cluster [DBG] 2021-03-24T22:03:59.651 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:59 smithi125 conmon[24927]: 3.f scrub ok 2021-03-24T22:03:59.651 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:59 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T22:03:59.651 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:59 smithi125 conmon[24927]: 22:03:58.368808+0000 2021-03-24T22:03:59.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:59 smithi125 conmon[24927]: mon.a (mon.0) 1780 : 2021-03-24T22:03:59.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:59 smithi125 conmon[24927]: cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:03:59.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:59 smithi125 conmon[24927]: audit 2021- 2021-03-24T22:03:59.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:59 smithi125 conmon[24927]: 03-24T22:03:58.385715+0000 mon.a (mon 2021-03-24T22:03:59.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:59 smithi125 conmon[24927]: .0) 1781 : audit [INF] 2021-03-24T22:03:59.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:59 smithi125 conmon[24927]: from='client.? v1:172.21.15.125:0/2132982102' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T22:03:59.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:59 smithi125 conmon[24927]: cluster 2021-03-24T22:03:59.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:59 smithi125 conmon[24927]: 2021-03-24T22:03:58 2021-03-24T22:03:59.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:59 smithi125 conmon[24927]: .385789+0000 mon.a (mon.0) 1782 : cluster [DBG] osdmap e668: 8 total, 7 up, 8 in 2021-03-24T22:03:59.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:59 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T22:03:59.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:59 smithi125 conmon[24927]: T22:03:58.697445+0000 osd.2 (osd 2021-03-24T22:03:59.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:59 smithi125 conmon[24927]: .2) 106 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:03:59.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:59 smithi125 conmon[24927]: cluster 2021-03-24T22:03:59.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:59 smithi125 conmon[24927]: 2021-03-24T22:03:58.699886+0000 2021-03-24T22:03:59.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:59 smithi125 conmon[24927]: osd.2 (osd.2) 107 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:03:59.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:03:59 smithi125 conmon[24927]: 2021-03-24T22:03:59.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:59.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:03:59.758 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:59.822 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:03:59.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:59.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:03:59.927 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:59.984 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:03:59.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:03:59.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:04:00.091 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:00.146 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:00.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:00.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:04:00.251 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:00.307 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:00.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:00.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:00.361 INFO:tasks.rados.rados.0.smithi125.stdout:2548: done (3 left) 2021-03-24T22:04:00.362 INFO:tasks.rados.rados.0.smithi125.stdout:2549: snap_remove snap 283 2021-03-24T22:04:00.416 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:00.450 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:00 smithi125 conmon[19936]: cluster 2021-03-24T22:03:58.942104+0000 mgr.y (mgr.14144) 2021-03-24T22:04:00.451 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:00 smithi125 conmon[19936]: 608 : cluster [DBG] pgmap v1165: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+clean+snaptrim_wait, 1 active+clean+snaptrim, 2 activating+undersized, 25 active+undersized, 10 active+undersized+degraded, 23 peering, 2 active+remapped+backfill_toofull, 1 stale+active+remapped+backfill_toofull, 101 active+clean; 547 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 9.6 MiB/s rd, 602 KiB/s wr, 9 op/s; 89/1767 objects degraded (5.037%); 80/1767 objects misplaced (4.527%); 8.6 MiB/s, 14 objects/s recovering 2021-03-24T22:04:00.451 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:00 smithi125 conmon[19936]: cluster 2021-03-24T22:03:59.395326+0000 mon.a (mon.0) 1783 : cluster [DBG] osdmap e669: 8 total, 7 up, 8 in 2021-03-24T22:04:00.451 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:00 smithi125 conmon[19936]: cluster 2021-03-24T22:04:00.354311+0000 mon.a (mon.0) 1784 : cluster [DBG] 2021-03-24T22:04:00.451 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:00 smithi125 conmon[19936]: osdmap e670: 8 total, 7 up, 8 in 2021-03-24T22:04:00.452 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:00 smithi125 conmon[24927]: cluster 2021-03-24T22:03:58.942104+0000 2021-03-24T22:04:00.452 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:00 smithi125 conmon[24927]: mgr.y (mgr.14144) 608 : cluster [DBG] pgmap v1165: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+clean+snaptrim_wait, 1 active+clean+snaptrim, 2 activating+undersized, 25 active+undersized, 10 active+undersized+degraded, 23 peering, 2 active+remapped+backfill_toofull, 1 stale+active+remapped+backfill_toofull, 101 active+clean; 547 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 9.6 MiB/s rd, 602 KiB/s wr, 9 op/s; 89/1767 objects degraded (5.037%); 80/1767 objects misplaced (4.527%); 8.6 MiB/s, 14 objects/s recovering 2021-03-24T22:04:00.452 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:00 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T22:04:00.452 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:00 smithi125 conmon[24927]: T22:03:59.395326+0000 mon.a (mon.0) 1783 : cluster [DBG] osdmap e669: 8 total, 7 up, 8 in 2021-03-24T22:04:00.453 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:00 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T22:04:00.453 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:00 smithi125 conmon[24927]: 22:04:00.354311+0000 mon.a (mon. 2021-03-24T22:04:00.454 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:00 smithi125 conmon[24927]: 0) 1784 : cluster [DBG] osdmap e670: 8 total, 7 up, 8 in 2021-03-24T22:04:00.475 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:04:00.482 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.2 dump_historic_ops 2021-03-24T22:04:00.484 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:00.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:00.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:00.517 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell -n osd.5 --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph-objectstore-tool --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op export --pgid 6.1c --file /var/log/ceph/exp.6.1c.5 2021-03-24T22:04:00.589 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:00.640 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:00.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:00.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:04:00.724 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/osd.5/config 2021-03-24T22:04:00.747 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:00.804 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:00.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:00.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:00.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:00 smithi142 conmon[20125]: cluster 2021-03-24T22:03:58.942104+0000 mgr.y (mgr.14144) 608 : 2021-03-24T22:04:00.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:00 smithi142 conmon[20125]: cluster [DBG] pgmap v1165: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+clean+snaptrim_wait, 1 active+clean+snaptrim, 2 activating+undersized, 25 active+undersized, 10 active+undersized+degraded, 23 peering, 2 active+remapped+backfill_toofull, 1 stale+active+remapped+backfill_toofull, 101 active+clean; 547 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 9.6 MiB/s rd, 602 KiB/s wr, 9 op/s; 89/1767 objects degraded (5.037%); 80/1767 objects misplaced (4.527%); 8.6 MiB/s, 14 objects/s recovering 2021-03-24T22:04:00.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:00 smithi142 conmon[20125]: cluster 2021-03-24T22:03:59.395326+0000 mon.a (mon.0) 1783 : 2021-03-24T22:04:00.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:00 smithi142 conmon[20125]: cluster [DBG] osdmap e669: 8 total, 7 up, 8 in 2021-03-24T22:04:00.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:00 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:04:00.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:00 smithi142 conmon[20125]: 22:04:00.354311+0000 mon.a (mon.0) 1784 : cluster 2021-03-24T22:04:00.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:00 smithi142 conmon[20125]: [DBG] osdmap e670: 8 total, 7 up, 8 in 2021-03-24T22:04:00.909 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:00.950 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:04:00.964 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:00.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:00.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:01.068 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:01.117 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:01.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:01.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:01.222 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:01.276 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:01.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:01.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:01.359 INFO:tasks.rados.rados.0.smithi125.stdout:2549: done (3 left) 2021-03-24T22:04:01.359 INFO:tasks.rados.rados.0.smithi125.stdout:2550: read oid 4 snap -1 2021-03-24T22:04:01.359 INFO:tasks.rados.rados.0.smithi125.stdout:2550: expect (ObjNum 865 snap 281 seq_num 865) 2021-03-24T22:04:01.359 INFO:tasks.rados.rados.0.smithi125.stdout:2551: snap_remove snap 243 2021-03-24T22:04:01.380 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:01.400 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:01 smithi125 conmon[19936]: audit 2021-03-24T22:04:00.644254+0000 mon.a (mon.0 2021-03-24T22:04:01.400 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:01 smithi125 conmon[19936]: ) 1785 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T22:04:01.400 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:01 smithi125 conmon[19936]: cluster 2021-03-24T22:04:01.354637+0000 mon.a (mon.0) 1786 : cluster [DBG] osdmap e671: 8 total, 7 up, 8 in 2021-03-24T22:04:01.427 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:01.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:01.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:01.533 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:01.574 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:01.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:01.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:01.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:01 smithi125 conmon[24927]: audit 2021-03-24T22:04:00. 2021-03-24T22:04:01.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:01 smithi125 conmon[24927]: 644254+0000 mon.a (mon.0) 1785 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T22:04:01.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:01 smithi125 conmon[24927]: cluster 2021-03-24T22:04:01 2021-03-24T22:04:01.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:01 smithi125 conmon[24927]: .354637+0000 mon.a (mon.0) 1786 : cluster [DBG] osdmap e671: 8 total, 7 up, 8 in 2021-03-24T22:04:01.680 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:01.718 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:01 smithi142 conmon[20125]: audit 2021-03-24T22:04:00.644254+0000 mon.a (mon.0) 2021-03-24T22:04:01.718 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:01 smithi142 conmon[20125]: 1785 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T22:04:01.719 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:01 smithi142 conmon[20125]: cluster 2021-03-24T22:04:01.354637+0000 mon.a (mon.0) 1786 : cluster [DBG] osdmap e671: 8 total, 7 up, 8 in 2021-03-24T22:04:01.732 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:01.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:01.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:01.839 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:01.896 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:01.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:01.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:04:02.002 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:02.061 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:02.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:02.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:02.167 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:02.228 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:02.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:02.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:04:02.335 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:02.362 INFO:tasks.rados.rados.0.smithi125.stdout:2551: done (4 left) 2021-03-24T22:04:02.363 INFO:tasks.rados.rados.0.smithi125.stdout:2552: rollback oid 28 current snap is 290 2021-03-24T22:04:02.363 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 28 to 284 2021-03-24T22:04:02.363 INFO:tasks.rados.rados.0.smithi125.stdout:2553: rollback oid 26 current snap is 290 2021-03-24T22:04:02.364 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 26 to 216 2021-03-24T22:04:02.364 INFO:tasks.rados.rados.0.smithi125.stdout:2554: snap_remove snap 276 2021-03-24T22:04:02.424 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:02.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:02.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:04:02.451 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-24T22:04:02.452 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:02 smithi142 conmon[20125]: cluster 2021-03-24T22:04:00.943002+0000 2021-03-24T22:04:02.453 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:02 smithi142 conmon[20125]: mgr.y (mgr.14144) 609 : cluster [DBG] pgmap v1168: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 2 active+remapped+backfill_toofull, 105 active+clean; 561 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 10 op/s; 149/1776 objects degraded (8.390%); 60/1776 objects misplaced (3.378%) 2021-03-24T22:04:02.453 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:02 smithi142 conmon[20125]: cluster 2021-03-24T22:04:01.390626+0000 2021-03-24T22:04:02.453 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:02 smithi142 conmon[20125]: mon.a (mon.0) 1787 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 4 pgs peering) 2021-03-24T22:04:02.453 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:02 smithi142 conmon[20125]: cluster 2021-03-24T22: 2021-03-24T22:04:02.454 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:02 smithi142 conmon[20125]: 04:02.355594+0000 mon.a (mon.0) 1788 2021-03-24T22:04:02.454 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:02 smithi142 conmon[20125]: : cluster [DBG] osdmap e672: 8 total, 7 up, 8 in 2021-03-24T22:04:02.533 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:02.543 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:02 smithi125 conmon[24927]: cluster 2021-03-24T22:04:00.943002+0000 mgr.y 2021-03-24T22:04:02.543 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:02 smithi125 conmon[24927]: (mgr.14144) 609 : cluster [DBG] pgmap v1168: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 2 active+remapped+backfill_toofull, 105 active+clean; 561 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 10 op/s; 149/1776 objects degraded (8.390%); 60/1776 objects misplaced (3.378%) 2021-03-24T22:04:02.543 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:02 smithi125 conmon[24927]: cluster 2021-03-24T22:04:01.390626+0000 2021-03-24T22:04:02.544 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:02 smithi125 conmon[24927]: mon.a (mon.0) 1787 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 4 pgs peering) 2021-03-24T22:04:02.544 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:02 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T22:04:02.544 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:02 smithi125 conmon[24927]: -24T22:04:02.355594+0000 mon.a (mon.0) 1788 : cluster [DBG] 2021-03-24T22:04:02.544 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:02 smithi125 conmon[24927]: osdmap e672: 8 total, 7 up, 8 in 2021-03-24T22:04:02.545 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:02 smithi125 conmon[19936]: cluster 2021-03-24T22:04:00.943002+0000 mgr.y (mgr. 2021-03-24T22:04:02.545 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:02 smithi125 conmon[19936]: 14144) 609 : cluster [DBG] pgmap v1168: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 2 active+remapped+backfill_toofull, 105 active+clean; 561 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 10 op/s; 149/1776 objects degraded (8.390%); 60/1776 objects misplaced (3.378%) 2021-03-24T22:04:02.545 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:02 smithi125 conmon[19936]: cluster 2021-03-24T22:04:01.390626+0000 mon.a (mon.0) 1787 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 4 pgs peering) 2021-03-24T22:04:02.546 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:02 smithi125 conmon[19936]: cluster 2021-03-24T22:04:02.546 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:02 smithi125 conmon[19936]: 2021-03-24T22:04:02.355594+0000 mon.a (mon.0) 1788 : cluster [DBG] osdmap e672: 8 total, 7 up, 8 in 2021-03-24T22:04:02.583 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:02.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:02.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:02.689 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:02.737 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:02.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:02.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:02.843 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:02.904 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:02.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:02.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:03.010 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:03.066 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:03.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:03.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:03.171 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:03.190 INFO:teuthology.orchestra.run.smithi142.stderr:Exporting 6.1c info 6.1c( empty local-lis/les=546/547 n=0 ec=66/66 lis/c=546/546 les/c/f=547/547/0 sis=546) 2021-03-24T22:04:03.191 INFO:teuthology.orchestra.run.smithi142.stderr:Export successful 2021-03-24T22:04:03.229 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:03.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:03.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:04:03.335 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:03.366 INFO:tasks.rados.rados.0.smithi125.stdout:2554: done (6 left) 2021-03-24T22:04:03.366 INFO:tasks.rados.rados.0.smithi125.stdout:2555: rollback oid 41 current snap is 290 2021-03-24T22:04:03.366 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 41 to 260 2021-03-24T22:04:03.366 INFO:tasks.rados.rados.0.smithi125.stdout:2556: delete oid 34 current snap is 290 2021-03-24T22:04:03.367 INFO:tasks.rados.rados.0.smithi125.stdout:2546: finishing rollback tid 0 to smithi12556470-23 2021-03-24T22:04:03.367 INFO:tasks.rados.rados.0.smithi125.stdout:2546: finishing rollback tid 1 to smithi12556470-23 2021-03-24T22:04:03.367 INFO:tasks.rados.rados.0.smithi125.stdout:2546: finishing rollback tid 2 to smithi12556470-23 2021-03-24T22:04:03.367 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 23 v 446 (ObjNum 881 snap 288 seq_num 881) dirty exists 2021-03-24T22:04:03.368 INFO:tasks.rados.rados.0.smithi125.stdout:2545: finishing write tid 2 to smithi12556470-25 2021-03-24T22:04:03.368 INFO:tasks.rados.rados.0.smithi125.stdout:2542: finishing copy_from to smithi12556470-21 2021-03-24T22:04:03.368 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 21 v 1411 (ObjNum 832 snap 272 seq_num 832) dirty exists 2021-03-24T22:04:03.368 INFO:tasks.rados.rados.0.smithi125.stdout:2542: finishing copy_from racing read to smithi12556470-21 2021-03-24T22:04:03.368 INFO:tasks.rados.rados.0.smithi125.stdout:2545: finishing write tid 3 to smithi12556470-25 2021-03-24T22:04:03.369 INFO:tasks.rados.rados.0.smithi125.stdout:2545: finishing write tid 4 to smithi12556470-25 2021-03-24T22:04:03.369 INFO:tasks.rados.rados.0.smithi125.stdout:2545: finishing write tid 5 to smithi12556470-25 2021-03-24T22:04:03.369 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 25 v 806 (ObjNum 885 snap 290 seq_num 885) dirty exists 2021-03-24T22:04:03.369 INFO:tasks.rados.rados.0.smithi125.stdout:2545: left oid 25 (ObjNum 885 snap 290 seq_num 885) 2021-03-24T22:04:03.369 INFO:tasks.rados.rados.0.smithi125.stdout:2550: expect (ObjNum 865 snap 281 seq_num 865) 2021-03-24T22:04:03.396 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:03.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:03.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:03.502 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:03.533 INFO:tasks.rados.rados.0.smithi125.stdout:2552: finishing rollback tid 0 to smithi12556470-28 2021-03-24T22:04:03.534 INFO:tasks.rados.rados.0.smithi125.stdout:2552: finishing rollback tid 1 to smithi12556470-28 2021-03-24T22:04:03.534 INFO:tasks.rados.rados.0.smithi125.stdout:2552: finishing rollback tid 2 to smithi12556470-28 2021-03-24T22:04:03.534 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 28 v 1188 (ObjNum 718 snap 229 seq_num 718) dirty exists 2021-03-24T22:04:03.535 INFO:tasks.rados.rados.0.smithi125.stdout:2553: finishing rollback tid 0 to smithi12556470-26 2021-03-24T22:04:03.535 INFO:tasks.rados.rados.0.smithi125.stdout:2553: finishing rollback tid 1 to smithi12556470-26 2021-03-24T22:04:03.535 INFO:tasks.rados.rados.0.smithi125.stdout:2553: finishing rollback tid 2 to smithi12556470-26 2021-03-24T22:04:03.535 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 26 v 456 (ObjNum 618 snap 187 seq_num 618) dirty exists 2021-03-24T22:04:03.535 INFO:tasks.rados.rados.0.smithi125.stdout:2542: done (7 left) 2021-03-24T22:04:03.536 INFO:tasks.rados.rados.0.smithi125.stdout:2545: done (6 left) 2021-03-24T22:04:03.536 INFO:tasks.rados.rados.0.smithi125.stdout:2546: done (5 left) 2021-03-24T22:04:03.536 INFO:tasks.rados.rados.0.smithi125.stdout:2550: done (4 left) 2021-03-24T22:04:03.536 INFO:tasks.rados.rados.0.smithi125.stdout:2552: done (3 left) 2021-03-24T22:04:03.536 INFO:tasks.rados.rados.0.smithi125.stdout:2553: done (2 left) 2021-03-24T22:04:03.537 INFO:tasks.rados.rados.0.smithi125.stdout:2556: done (1 left) 2021-03-24T22:04:03.537 INFO:tasks.rados.rados.0.smithi125.stdout:2557: write oid 47 current snap is 290 2021-03-24T22:04:03.537 INFO:tasks.rados.rados.0.smithi125.stdout:2555: finishing rollback tid 0 to smithi12556470-41 2021-03-24T22:04:03.537 INFO:tasks.rados.rados.0.smithi125.stdout:2555: finishing rollback tid 1 to smithi12556470-41 2021-03-24T22:04:03.537 INFO:tasks.rados.rados.0.smithi125.stdout:2555: finishing rollback tid 2 to smithi12556470-41 2021-03-24T22:04:03.538 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 41 v 870 (ObjNum 791 snap 257 seq_num 791) dirty exists 2021-03-24T22:04:03.538 INFO:tasks.rados.rados.0.smithi125.stdout:2557: seq_num 887 ranges {507809=642811,1721881=663964} 2021-03-24T22:04:03.540 INFO:tasks.rados.rados.0.smithi125.stdout:2557: writing smithi12556470-47 from 507809 to 1150620 tid 1 2021-03-24T22:04:03.548 INFO:tasks.rados.rados.0.smithi125.stdout:2557: writing smithi12556470-47 from 1721881 to 2385845 tid 2 2021-03-24T22:04:03.548 INFO:tasks.rados.rados.0.smithi125.stdout:2555: done (1 left) 2021-03-24T22:04:03.548 INFO:tasks.rados.rados.0.smithi125.stdout:2558: rollback oid 43 current snap is 290 2021-03-24T22:04:03.549 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 43 to 238 2021-03-24T22:04:03.549 INFO:tasks.rados.rados.0.smithi125.stdout:2557: finishing write tid 1 to smithi12556470-47 2021-03-24T22:04:03.549 INFO:tasks.rados.rados.0.smithi125.stdout:2559: snap_create 2021-03-24T22:04:03.550 INFO:tasks.rados.rados.0.smithi125.stdout:2558: finishing rollback tid 1 to smithi12556470-43 2021-03-24T22:04:03.551 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 43 v 1366 (ObjNum 1 snap 21916 seq_num 1117558496) dirty dne 2021-03-24T22:04:03.553 INFO:tasks.rados.rados.0.smithi125.stdout:2557: finishing write tid 2 to smithi12556470-47 2021-03-24T22:04:03.553 INFO:tasks.rados.rados.0.smithi125.stdout:2557: finishing write tid 3 to smithi12556470-47 2021-03-24T22:04:03.554 INFO:tasks.rados.rados.0.smithi125.stdout:2557: finishing write tid 4 to smithi12556470-47 2021-03-24T22:04:03.554 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 47 v 355 (ObjNum 887 snap 290 seq_num 887) dirty exists 2021-03-24T22:04:03.554 INFO:tasks.rados.rados.0.smithi125.stdout:2557: left oid 47 (ObjNum 887 snap 290 seq_num 887) 2021-03-24T22:04:03.561 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:03.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:03.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:04:03.665 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:03.717 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:03.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:03.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:03.823 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:03.870 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:03.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:03.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:03.898 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:04:03.964 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T22:04:03.978 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:03.991 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T22:04:04.018 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T22:04:04.024 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:04.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:04.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:04:04.044 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T22:04:04.065 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T22:04:04.091 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T22:04:04.121 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T22:04:04.121 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-24T22:04:04.122 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-24T22:04:04.122 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-24T22:04:04.122 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-24T22:04:04.122 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-24T22:04:04.123 INFO:teuthology.orchestra.run.smithi125.stderr:Error ENXIO: problem getting command descriptions from osd.5 2021-03-24T22:04:04.123 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-24T22:04:04.123 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-24T22:04:04.129 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:04.177 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:04.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:04.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:04.282 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:04.341 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:04.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:04.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:04.370 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-24T22:04:04.371 INFO:teuthology.orchestra.run.smithi125.stdout: "size": 20, 2021-03-24T22:04:04.371 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 600, 2021-03-24T22:04:04.371 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [ 2021-03-24T22:04:04.372 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.372 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:1777 7.17 e232/203 7:ef6d6bee:::smithi12556470-24:head v 232'218, mlcod=230'214)", 2021-03-24T22:04:04.372 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:56:41.201926+0000", 2021-03-24T22:04:04.372 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 443.16324309200002, 2021-03-24T22:04:04.373 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.039154920000000003, 2021-03-24T22:04:04.373 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:04:04.373 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:04:04.373 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:04:04.374 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.374 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:04:04.375 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:41.201926+0000", 2021-03-24T22:04:04.375 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:04:04.375 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.375 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.375 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:04:04.376 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:41.201926+0000", 2021-03-24T22:04:04.376 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999995 2021-03-24T22:04:04.376 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.377 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.377 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:04:04.377 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:41.201926+0000", 2021-03-24T22:04:04.377 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00074986499999999995 2021-03-24T22:04:04.377 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.378 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.378 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:04:04.378 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:41.202676+0000", 2021-03-24T22:04:04.378 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5279000000000001e-05 2021-03-24T22:04:04.379 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.379 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.379 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:04:04.380 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:41.202691+0000", 2021-03-24T22:04:04.380 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.8719999999999996e-06 2021-03-24T22:04:04.380 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.380 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.381 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:04:04.381 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:41.202698+0000", 2021-03-24T22:04:04.381 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.9821999999999994e-05 2021-03-24T22:04:04.381 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.381 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.382 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:04.382 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:41.202798+0000", 2021-03-24T22:04:04.382 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.3599000000000001e-05 2021-03-24T22:04:04.382 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.383 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.383 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:04:04.383 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:41.202831+0000", 2021-03-24T22:04:04.384 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.038160301000000001 2021-03-24T22:04:04.384 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.384 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.384 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:04:04.385 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:41.240992+0000", 2021-03-24T22:04:04.385 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.9573000000000005e-05 2021-03-24T22:04:04.385 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.385 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.386 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:04:04.386 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:56:41.241081+0000", 2021-03-24T22:04:04.386 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.039154920000000003 2021-03-24T22:04:04.386 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:04.386 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:04:04.387 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:04.387 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.387 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.387 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:2557 7.11 e290/204 7:8dd64f9e:::smithi12556470-49:head v 290'308, mlcod=278'304)", 2021-03-24T22:04:04.388 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:57:40.204468+0000", 2021-03-24T22:04:04.388 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 384.16070159999998, 2021-03-24T22:04:04.388 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.052820996000000002, 2021-03-24T22:04:04.388 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:04:04.388 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:04:04.389 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:04:04.389 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.389 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:04:04.389 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.204468+0000", 2021-03-24T22:04:04.389 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:04:04.390 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.390 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.390 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:04:04.391 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.204468+0000", 2021-03-24T22:04:04.391 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999952 2021-03-24T22:04:04.391 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.392 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.392 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:04:04.392 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.204463+0000", 2021-03-24T22:04:04.392 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00085829000000000005 2021-03-24T22:04:04.393 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.393 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.393 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:04:04.393 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.205321+0000", 2021-03-24T22:04:04.393 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.4329999999999998e-05 2021-03-24T22:04:04.394 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.394 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.394 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:04:04.394 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.205356+0000", 2021-03-24T22:04:04.395 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2873e-05 2021-03-24T22:04:04.395 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.395 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.395 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:04:04.396 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.205369+0000", 2021-03-24T22:04:04.396 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.8431000000000007e-05 2021-03-24T22:04:04.396 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.396 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.397 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:04.397 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.205467+0000", 2021-03-24T22:04:04.397 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.0049000000000002e-05 2021-03-24T22:04:04.397 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.398 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.398 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:04:04.398 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.205507+0000", 2021-03-24T22:04:04.398 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.051707234999999997 2021-03-24T22:04:04.399 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.399 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.399 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:04:04.399 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.257214+0000", 2021-03-24T22:04:04.400 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.4583000000000001e-05 2021-03-24T22:04:04.400 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.400 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.401 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:04:04.401 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.257289+0000", 2021-03-24T22:04:04.402 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.052820996000000002 2021-03-24T22:04:04.402 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:04.402 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:04:04.402 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:04.403 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.403 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.403 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:2558 7.11 e290/204 7:8dd64f9e:::smithi12556470-49:head v 290'309, mlcod=278'304)", 2021-03-24T22:04:04.403 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:57:40.209426+0000", 2021-03-24T22:04:04.404 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 384.15574388800002, 2021-03-24T22:04:04.404 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.047890466999999999, 2021-03-24T22:04:04.404 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:04:04.405 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:04:04.405 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:04:04.405 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.406 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:04:04.406 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.209426+0000", 2021-03-24T22:04:04.407 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:04:04.407 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.407 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.407 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:04:04.407 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.209426+0000", 2021-03-24T22:04:04.408 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T22:04:04.408 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.408 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.408 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:04:04.409 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.209424+0000", 2021-03-24T22:04:04.409 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00021029499999999999 2021-03-24T22:04:04.409 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.409 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.409 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:04:04.410 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.209634+0000", 2021-03-24T22:04:04.410 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9154999999999999e-05 2021-03-24T22:04:04.410 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.411 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.411 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:04:04.411 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.209653+0000", 2021-03-24T22:04:04.412 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.9319999999999999e-06 2021-03-24T22:04:04.412 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.413 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.414 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:04:04.414 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.209660+0000", 2021-03-24T22:04:04.414 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6637e-05 2021-03-24T22:04:04.415 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.415 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.416 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:04.416 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.209717+0000", 2021-03-24T22:04:04.416 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0486e-05 2021-03-24T22:04:04.417 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.417 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.417 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:04:04.418 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.209737+0000", 2021-03-24T22:04:04.418 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.047558077999999997 2021-03-24T22:04:04.418 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.418 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.419 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:04:04.419 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.257295+0000", 2021-03-24T22:04:04.420 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0899000000000001e-05 2021-03-24T22:04:04.420 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.420 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.421 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:04:04.421 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.257316+0000", 2021-03-24T22:04:04.422 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.047890466999999999 2021-03-24T22:04:04.422 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:04.422 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:04:04.423 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:04.423 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.423 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.424 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:2559 7.11 e290/204 7:8dd64f9e:::smithi12556470-49:head v 290'310, mlcod=278'304)", 2021-03-24T22:04:04.424 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:57:40.210587+0000", 2021-03-24T22:04:04.424 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 384.15458221199998, 2021-03-24T22:04:04.425 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.046757605000000001, 2021-03-24T22:04:04.425 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:04:04.425 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:04:04.426 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:04:04.426 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.426 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:04:04.427 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.210587+0000", 2021-03-24T22:04:04.427 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:04:04.428 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.428 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.428 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:04:04.429 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.210587+0000", 2021-03-24T22:04:04.429 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T22:04:04.430 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.430 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.430 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:04:04.431 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.210585+0000", 2021-03-24T22:04:04.431 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.3989999999999993e-06 2021-03-24T22:04:04.432 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.432 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.432 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:04:04.433 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.210595+0000", 2021-03-24T22:04:04.433 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2702e-05 2021-03-24T22:04:04.434 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.434 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.435 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:04:04.435 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.210607+0000", 2021-03-24T22:04:04.435 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.3879999999999999e-06 2021-03-24T22:04:04.436 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.436 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.437 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:04:04.437 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.210613+0000", 2021-03-24T22:04:04.437 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.1072000000000002e-05 2021-03-24T22:04:04.438 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.438 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.439 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:04.439 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.210654+0000", 2021-03-24T22:04:04.440 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.5398999999999999e-05 2021-03-24T22:04:04.440 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.440 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.441 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:04:04.441 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.210679+0000", 2021-03-24T22:04:04.442 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.046641477000000001 2021-03-24T22:04:04.442 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.442 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.443 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:04:04.443 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.257321+0000", 2021-03-24T22:04:04.444 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.4128999999999999e-05 2021-03-24T22:04:04.444 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.445 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.445 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:04:04.445 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:40.257345+0000", 2021-03-24T22:04:04.446 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.046757605000000001 2021-03-24T22:04:04.446 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:04.447 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:04:04.447 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:04.448 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.448 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.448 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:3847 7.19 e400/340 7:9b48f357:::smithi12556470-17:head v 400'390, mlcod=391'386)", 2021-03-24T22:04:04.449 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:30.268181+0000", 2021-03-24T22:04:04.449 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 274.096988904, 2021-03-24T22:04:04.450 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.053258855000000001, 2021-03-24T22:04:04.450 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:04:04.451 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:04:04.451 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:04:04.451 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.452 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:04:04.452 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.268181+0000", 2021-03-24T22:04:04.453 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:04:04.453 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.453 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.454 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:04:04.454 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.268181+0000", 2021-03-24T22:04:04.455 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999933 2021-03-24T22:04:04.455 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.456 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.456 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:04:04.456 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.268174+0000", 2021-03-24T22:04:04.457 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.1306999999999998e-05 2021-03-24T22:04:04.457 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.457 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.458 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:04:04.458 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.268265+0000", 2021-03-24T22:04:04.458 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.0286000000000001e-05 2021-03-24T22:04:04.458 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.458 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.459 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:04:04.459 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.268296+0000", 2021-03-24T22:04:04.459 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.434e-05 2021-03-24T22:04:04.459 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.459 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.460 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:04:04.460 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.268310+0000", 2021-03-24T22:04:04.460 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4766999999999998e-05 2021-03-24T22:04:04.460 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.460 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.461 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:04.461 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.268375+0000", 2021-03-24T22:04:04.461 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.6304999999999999e-05 2021-03-24T22:04:04.461 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.462 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.462 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:04:04.462 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.268411+0000", 2021-03-24T22:04:04.462 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.052984651000000001 2021-03-24T22:04:04.462 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.463 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.463 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:04:04.463 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.321396+0000", 2021-03-24T22:04:04.463 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3760000000000001e-05 2021-03-24T22:04:04.463 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.464 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.464 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:04:04.464 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.321439+0000", 2021-03-24T22:04:04.464 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.053258855000000001 2021-03-24T22:04:04.464 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:04.465 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:04:04.465 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:04.465 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.465 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.465 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:3848 7.19 e400/340 7:9b48f357:::smithi12556470-17:head v 400'391, mlcod=391'386)", 2021-03-24T22:04:04.466 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:30.268965+0000", 2021-03-24T22:04:04.466 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 274.09620420900001, 2021-03-24T22:04:04.466 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.052493694, 2021-03-24T22:04:04.466 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:04:04.467 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:04:04.467 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:04:04.467 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.467 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:04:04.467 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.268965+0000", 2021-03-24T22:04:04.468 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:04:04.468 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.468 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.468 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:04:04.468 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.268965+0000", 2021-03-24T22:04:04.469 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T22:04:04.469 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.469 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.469 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:04:04.470 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.268964+0000", 2021-03-24T22:04:04.470 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.9339999999999995e-06 2021-03-24T22:04:04.470 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.470 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.470 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:04:04.471 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.268973+0000", 2021-03-24T22:04:04.471 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3319000000000001e-05 2021-03-24T22:04:04.471 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.471 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.471 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:04:04.472 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.268986+0000", 2021-03-24T22:04:04.472 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.0950000000000001e-06 2021-03-24T22:04:04.472 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.472 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.472 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:04:04.473 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.268992+0000", 2021-03-24T22:04:04.473 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9656000000000001e-05 2021-03-24T22:04:04.473 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.473 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.473 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:04.474 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.269032+0000", 2021-03-24T22:04:04.474 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0984000000000001e-05 2021-03-24T22:04:04.474 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.474 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.475 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:04:04.475 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.269053+0000", 2021-03-24T22:04:04.475 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.052390012999999999 2021-03-24T22:04:04.475 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.475 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.476 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:04:04.476 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.321443+0000", 2021-03-24T22:04:04.476 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5875e-05 2021-03-24T22:04:04.476 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.476 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.477 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:04:04.477 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.321459+0000", 2021-03-24T22:04:04.477 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.052493694 2021-03-24T22:04:04.477 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:04.477 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:04:04.478 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:04.478 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.478 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.478 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:3850 7.1b e400/282 7:de4c3db5:::smithi12556470-32:head v 400'549, mlcod=400'545)", 2021-03-24T22:04:04.478 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:30.276118+0000", 2021-03-24T22:04:04.479 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 274.08905115699997, 2021-03-24T22:04:04.479 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.045491524999999998, 2021-03-24T22:04:04.479 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:04:04.479 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:04:04.480 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:04:04.480 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.480 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:04:04.480 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.276118+0000", 2021-03-24T22:04:04.480 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:04:04.481 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.481 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.481 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:04:04.481 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.276118+0000", 2021-03-24T22:04:04.481 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999967 2021-03-24T22:04:04.482 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.482 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.482 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:04:04.482 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.276115+0000", 2021-03-24T22:04:04.483 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0020321900000000001 2021-03-24T22:04:04.483 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.483 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.483 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:04:04.484 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.278147+0000", 2021-03-24T22:04:04.484 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3791999999999999e-05 2021-03-24T22:04:04.484 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.484 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.484 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:04:04.485 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.278161+0000", 2021-03-24T22:04:04.485 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.5249999999999994e-06 2021-03-24T22:04:04.485 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.485 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.485 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:04:04.486 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.278170+0000", 2021-03-24T22:04:04.486 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.8162000000000001e-05 2021-03-24T22:04:04.486 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.486 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.486 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:04.487 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.278258+0000", 2021-03-24T22:04:04.487 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.6956000000000001e-05 2021-03-24T22:04:04.487 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.487 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.487 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:04:04.488 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.278285+0000", 2021-03-24T22:04:04.488 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.043282701999999999 2021-03-24T22:04:04.488 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.488 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.489 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:04:04.489 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.321568+0000", 2021-03-24T22:04:04.489 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.1721999999999998e-05 2021-03-24T22:04:04.489 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.489 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.490 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:04:04.490 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.321610+0000", 2021-03-24T22:04:04.490 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.045491524999999998 2021-03-24T22:04:04.490 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:04.490 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:04:04.491 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:04.491 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.491 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.491 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:3829 7.c e400/275 7:31cb714a:::smithi12556470-7:head v 400'435, mlcod=397'431)", 2021-03-24T22:04:04.491 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:30.277370+0000", 2021-03-24T22:04:04.492 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 274.08779995100002, 2021-03-24T22:04:04.492 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.044151612999999999, 2021-03-24T22:04:04.492 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:04:04.492 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:04:04.492 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:04:04.493 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.493 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:04:04.493 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.277370+0000", 2021-03-24T22:04:04.493 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:04:04.493 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.494 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.494 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:04:04.494 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.277370+0000", 2021-03-24T22:04:04.494 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999995 2021-03-24T22:04:04.495 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.495 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.495 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:04:04.495 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.277369+0000", 2021-03-24T22:04:04.495 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00013732499999999999 2021-03-24T22:04:04.496 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.496 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.496 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:04:04.496 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.277506+0000", 2021-03-24T22:04:04.496 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.5281000000000001e-05 2021-03-24T22:04:04.497 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.497 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.497 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:04:04.497 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.277532+0000", 2021-03-24T22:04:04.498 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1952e-05 2021-03-24T22:04:04.498 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.498 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.498 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:04:04.498 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.277544+0000", 2021-03-24T22:04:04.499 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000473554 2021-03-24T22:04:04.499 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.499 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.499 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:04.499 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.278017+0000", 2021-03-24T22:04:04.500 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.5218000000000001e-05 2021-03-24T22:04:04.500 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.500 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.500 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:04:04.500 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.278052+0000", 2021-03-24T22:04:04.501 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.043425692000000002 2021-03-24T22:04:04.501 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.501 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.501 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:04:04.501 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.321478+0000", 2021-03-24T22:04:04.502 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3025000000000002e-05 2021-03-24T22:04:04.502 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.502 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.502 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:04:04.503 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.321521+0000", 2021-03-24T22:04:04.503 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.044151612999999999 2021-03-24T22:04:04.503 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:04.503 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:04:04.503 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:04.504 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.504 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.504 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:3830 7.c e400/275 7:31cb714a:::smithi12556470-7:head v 400'436, mlcod=397'431)", 2021-03-24T22:04:04.504 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:30.277557+0000", 2021-03-24T22:04:04.504 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 274.08761203500001, 2021-03-24T22:04:04.505 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.043987660999999997, 2021-03-24T22:04:04.505 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:04:04.505 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:04:04.505 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:04:04.505 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.506 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:04:04.506 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.277557+0000", 2021-03-24T22:04:04.506 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:04:04.506 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.506 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.507 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:04:04.507 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.277557+0000", 2021-03-24T22:04:04.507 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999995 2021-03-24T22:04:04.507 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.507 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.508 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:04:04.508 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.277557+0000", 2021-03-24T22:04:04.508 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00034763900000000001 2021-03-24T22:04:04.508 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.508 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.509 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:04:04.509 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.277904+0000", 2021-03-24T22:04:04.509 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.4780000000000008e-06 2021-03-24T22:04:04.509 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.510 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.510 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:04:04.510 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.277913+0000", 2021-03-24T22:04:04.510 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.002e-06 2021-03-24T22:04:04.510 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.511 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.511 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:04:04.511 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.277920+0000", 2021-03-24T22:04:04.511 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001241455 2021-03-24T22:04:04.511 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.512 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.512 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:04.512 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.279161+0000", 2021-03-24T22:04:04.512 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.5119000000000001e-05 2021-03-24T22:04:04.512 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.513 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.513 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:04:04.513 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.279186+0000", 2021-03-24T22:04:04.513 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.042339505 2021-03-24T22:04:04.513 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.514 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.514 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:04:04.514 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.321526+0000", 2021-03-24T22:04:04.514 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9083999999999999e-05 2021-03-24T22:04:04.515 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.515 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.515 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:04:04.515 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.321545+0000", 2021-03-24T22:04:04.515 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.043987660999999997 2021-03-24T22:04:04.516 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:04.516 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:04:04.516 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:04.516 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.516 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.517 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:3831 7.c e400/275 7:31cb714a:::smithi12556470-7:head v 400'437, mlcod=397'431)", 2021-03-24T22:04:04.517 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:30.286442+0000", 2021-03-24T22:04:04.517 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 274.07872715500002, 2021-03-24T22:04:04.517 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.035122622999999999, 2021-03-24T22:04:04.517 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:04:04.518 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:04:04.518 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:04:04.518 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.518 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:04:04.518 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.286442+0000", 2021-03-24T22:04:04.519 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:04:04.519 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.519 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.519 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:04:04.519 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.286442+0000", 2021-03-24T22:04:04.520 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999971 2021-03-24T22:04:04.520 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.520 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.520 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:04:04.520 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.286439+0000", 2021-03-24T22:04:04.521 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.6485999999999999e-05 2021-03-24T22:04:04.521 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.521 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.521 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:04:04.522 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.286516+0000", 2021-03-24T22:04:04.522 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.2932e-05 2021-03-24T22:04:04.522 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.522 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.522 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:04:04.523 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.286539+0000", 2021-03-24T22:04:04.523 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.0440000000000001e-06 2021-03-24T22:04:04.523 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.523 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.523 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:04:04.524 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.286548+0000", 2021-03-24T22:04:04.524 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2671999999999997e-05 2021-03-24T22:04:04.524 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.524 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.525 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:04.525 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.286630+0000", 2021-03-24T22:04:04.525 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9359e-05 2021-03-24T22:04:04.525 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.525 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.526 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:04:04.526 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.286670+0000", 2021-03-24T22:04:04.526 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.034877342999999998 2021-03-24T22:04:04.526 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.526 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.527 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:04:04.527 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.321547+0000", 2021-03-24T22:04:04.527 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7813e-05 2021-03-24T22:04:04.527 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.527 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.528 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:04:04.528 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.321565+0000", 2021-03-24T22:04:04.528 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.035122622999999999 2021-03-24T22:04:04.528 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:04.528 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:04:04.529 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:04.529 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.529 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.529 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.14760.0:3832 7.c e400/275 7:31cb714a:::smithi12556470-7:head v 400'438, mlcod=397'431)", 2021-03-24T22:04:04.530 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:30.287369+0000", 2021-03-24T22:04:04.530 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 274.077800701, 2021-03-24T22:04:04.530 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.034213645000000001, 2021-03-24T22:04:04.530 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:04:04.530 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:04:04.531 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:04:04.531 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.531 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:04:04.531 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.287369+0000", 2021-03-24T22:04:04.531 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:04:04.532 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.532 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.532 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:04:04.532 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.287369+0000", 2021-03-24T22:04:04.532 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T22:04:04.533 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.533 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.533 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:04:04.533 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.287368+0000", 2021-03-24T22:04:04.533 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.9680000000000006e-06 2021-03-24T22:04:04.534 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.534 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.534 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:04:04.534 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.287376+0000", 2021-03-24T22:04:04.534 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3944999999999999e-05 2021-03-24T22:04:04.535 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.535 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.535 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:04:04.535 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.287390+0000", 2021-03-24T22:04:04.536 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.1970000000000001e-06 2021-03-24T22:04:04.536 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.536 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.536 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:04:04.536 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.287396+0000", 2021-03-24T22:04:04.537 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.3905000000000006e-05 2021-03-24T22:04:04.537 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.537 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.537 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:04.537 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.287460+0000", 2021-03-24T22:04:04.538 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.7464999999999999e-05 2021-03-24T22:04:04.538 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.538 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.538 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:04:04.538 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.287487+0000", 2021-03-24T22:04:04.539 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.034079365 2021-03-24T22:04:04.539 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.539 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.539 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:04:04.539 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.321566+0000", 2021-03-24T22:04:04.540 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5937999999999999e-05 2021-03-24T22:04:04.540 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.540 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.540 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:04:04.540 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:30.321582+0000", 2021-03-24T22:04:04.541 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.034213645000000001 2021-03-24T22:04:04.541 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:04.541 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:04:04.541 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:04.541 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.542 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.542 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:3994 7.15 7:ad88ec1b:::smithi12556470-47:head [write 1742451~465507] snapc 137=[137,136,133,132,12d,12c,12b,129,128,127,126,120,11e,11b,11a,119,110,10f,10c,106,105,104,fa,f8,e7] ondisk+write+known_if_redirected e420)", 2021-03-24T22:04:04.542 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:50.238478+0000", 2021-03-24T22:04:04.542 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 254.12669166399999, 2021-03-24T22:04:04.543 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.042060243999999997, 2021-03-24T22:04:04.543 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:04:04.543 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:04:04.543 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:04:04.543 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:04:04.544 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:04:04.544 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3994 2021-03-24T22:04:04.544 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.544 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:04:04.544 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.545 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:04:04.545 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.238478+0000", 2021-03-24T22:04:04.545 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:04:04.545 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.545 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.546 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:04:04.546 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.238478+0000", 2021-03-24T22:04:04.546 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999967 2021-03-24T22:04:04.546 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.546 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.547 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:04:04.547 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.238474+0000", 2021-03-24T22:04:04.547 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00013515599999999999 2021-03-24T22:04:04.547 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.547 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.548 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:04:04.548 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.238610+0000", 2021-03-24T22:04:04.548 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2685e-05 2021-03-24T22:04:04.548 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.549 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.549 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:04:04.549 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.238622+0000", 2021-03-24T22:04:04.549 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.4679999999999998e-06 2021-03-24T22:04:04.549 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.550 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.550 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:04:04.550 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.238632+0000", 2021-03-24T22:04:04.550 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.6968000000000002e-05 2021-03-24T22:04:04.550 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.551 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.551 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:04.551 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.238669+0000", 2021-03-24T22:04:04.551 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.8763000000000003e-05 2021-03-24T22:04:04.551 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.552 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.552 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:04:04.552 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.238767+0000", 2021-03-24T22:04:04.552 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.026297406999999998 2021-03-24T22:04:04.552 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.553 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.553 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:04.553 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.265065+0000", 2021-03-24T22:04:04.553 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000118282 2021-03-24T22:04:04.554 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.554 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.554 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:04:04.554 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.265183+0000", 2021-03-24T22:04:04.554 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00025056299999999998 2021-03-24T22:04:04.555 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.555 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.555 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 4,7", 2021-03-24T22:04:04.555 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.265434+0000", 2021-03-24T22:04:04.555 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0024804789999999998 2021-03-24T22:04:04.556 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.556 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.556 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:04:04.556 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.267914+0000", 2021-03-24T22:04:04.556 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012115023000000001 2021-03-24T22:04:04.557 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.557 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.557 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:04:04.557 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.280029+0000", 2021-03-24T22:04:04.557 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00028268400000000001 2021-03-24T22:04:04.558 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.558 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.558 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:04:04.558 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.280312+0000", 2021-03-24T22:04:04.558 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.8313000000000002e-05 2021-03-24T22:04:04.559 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.559 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.559 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:04:04.559 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.280370+0000", 2021-03-24T22:04:04.560 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00016783 2021-03-24T22:04:04.560 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.560 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.560 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:04:04.560 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.280538+0000", 2021-03-24T22:04:04.561 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.042060243999999997 2021-03-24T22:04:04.561 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:04.561 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:04:04.561 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:04.561 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.562 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.562 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:3995 7.15 7:ad88ec1b:::smithi12556470-47:head [write 2917178~242637] snapc 137=[137,136,133,132,12d,12c,12b,129,128,127,126,120,11e,11b,11a,119,110,10f,10c,106,105,104,fa,f8,e7] ondisk+write+known_if_redirected e420)", 2021-03-24T22:04:04.562 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:50.244845+0000", 2021-03-24T22:04:04.562 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 254.12032472999999, 2021-03-24T22:04:04.563 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.035994086000000002, 2021-03-24T22:04:04.563 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:04:04.563 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:04:04.563 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:04:04.563 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:04:04.564 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:04:04.564 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3995 2021-03-24T22:04:04.564 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.564 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:04:04.564 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.565 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:04:04.565 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.244845+0000", 2021-03-24T22:04:04.565 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:04:04.565 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.565 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.566 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:04:04.566 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.244845+0000", 2021-03-24T22:04:04.566 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999971 2021-03-24T22:04:04.566 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.566 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.567 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:04:04.567 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.244842+0000", 2021-03-24T22:04:04.567 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4729999999999999e-05 2021-03-24T22:04:04.567 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.567 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.568 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:04:04.568 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.244906+0000", 2021-03-24T22:04:04.568 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.2957999999999999e-05 2021-03-24T22:04:04.568 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.569 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.569 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:04:04.569 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.244929+0000", 2021-03-24T22:04:04.569 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.3969999999999994e-06 2021-03-24T22:04:04.569 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.570 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.570 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:04:04.570 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.244938+0000", 2021-03-24T22:04:04.570 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.6421000000000006e-05 2021-03-24T22:04:04.570 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.571 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.571 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:04.571 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.245014+0000", 2021-03-24T22:04:04.571 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000115259 2021-03-24T22:04:04.572 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.572 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.572 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:04:04.572 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.245130+0000", 2021-03-24T22:04:04.572 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.021186351999999999 2021-03-24T22:04:04.573 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.573 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.573 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:04.573 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.266316+0000", 2021-03-24T22:04:04.573 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.9944000000000004e-05 2021-03-24T22:04:04.574 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.574 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.574 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:04:04.574 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.266416+0000", 2021-03-24T22:04:04.575 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00023467900000000001 2021-03-24T22:04:04.575 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.575 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.575 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 4,7", 2021-03-24T22:04:04.575 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.266650+0000", 2021-03-24T22:04:04.576 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001697325 2021-03-24T22:04:04.576 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.576 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.576 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:04:04.576 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.268348+0000", 2021-03-24T22:04:04.577 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011757292000000001 2021-03-24T22:04:04.577 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.577 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.577 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:04:04.577 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.280105+0000", 2021-03-24T22:04:04.578 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00051535899999999996 2021-03-24T22:04:04.578 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.578 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.578 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:04:04.579 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.280620+0000", 2021-03-24T22:04:04.579 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.8341999999999998e-05 2021-03-24T22:04:04.579 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.579 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.579 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:04:04.580 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.280679+0000", 2021-03-24T22:04:04.580 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000160008 2021-03-24T22:04:04.580 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.580 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.580 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:04:04.581 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.280839+0000", 2021-03-24T22:04:04.581 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.035994086000000002 2021-03-24T22:04:04.581 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:04.581 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:04:04.581 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:04.582 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.582 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.582 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:3996 7.15 7:ad88ec1b:::smithi12556470-47:head [setxattr _header (58) in=7b,truncate 3159815] snapc 137=[137,136,133,132,12d,12c,12b,129,128,127,126,120,11e,11b,11a,119,110,10f,10c,106,105,104,fa,f8,e7] ondisk+write+known_if_redirected e420)", 2021-03-24T22:04:04.582 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:50.244952+0000", 2021-03-24T22:04:04.583 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 254.120217225, 2021-03-24T22:04:04.583 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.036356240999999997, 2021-03-24T22:04:04.583 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:04:04.583 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:04:04.583 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:04:04.584 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:04:04.584 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:04:04.584 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3996 2021-03-24T22:04:04.584 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.585 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:04:04.585 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.585 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:04:04.585 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.244952+0000", 2021-03-24T22:04:04.585 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:04:04.586 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.586 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.586 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:04:04.586 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.244952+0000", 2021-03-24T22:04:04.586 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999995 2021-03-24T22:04:04.587 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.587 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.587 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:04:04.587 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.244952+0000", 2021-03-24T22:04:04.587 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3220000000000003e-06 2021-03-24T22:04:04.588 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.588 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.588 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:04:04.588 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.244956+0000", 2021-03-24T22:04:04.588 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5040000000000002e-06 2021-03-24T22:04:04.589 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.589 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.589 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:04:04.589 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.244961+0000", 2021-03-24T22:04:04.590 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.8210000000000001e-06 2021-03-24T22:04:04.590 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.590 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.590 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:04:04.590 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.244966+0000", 2021-03-24T22:04:04.591 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00018897200000000001 2021-03-24T22:04:04.591 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.591 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.591 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:04.591 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.245155+0000", 2021-03-24T22:04:04.592 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.0983000000000006e-05 2021-03-24T22:04:04.592 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.592 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.592 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:04:04.592 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.245246+0000", 2021-03-24T22:04:04.593 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.022078372999999998 2021-03-24T22:04:04.593 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.593 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.593 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:04.593 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.267324+0000", 2021-03-24T22:04:04.594 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.1389000000000002e-05 2021-03-24T22:04:04.594 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.594 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.594 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:04:04.595 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.267416+0000", 2021-03-24T22:04:04.595 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000216388 2021-03-24T22:04:04.595 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.595 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.595 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 4,7", 2021-03-24T22:04:04.596 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.267632+0000", 2021-03-24T22:04:04.596 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0030801959999999999 2021-03-24T22:04:04.596 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.596 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.596 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:04:04.597 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.270712+0000", 2021-03-24T22:04:04.597 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0094740310000000008 2021-03-24T22:04:04.597 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.597 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.597 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:04:04.598 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.280186+0000", 2021-03-24T22:04:04.598 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00078359100000000004 2021-03-24T22:04:04.598 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.598 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.598 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:04:04.599 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.280970+0000", 2021-03-24T22:04:04.599 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5875000000000003e-05 2021-03-24T22:04:04.599 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.599 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.599 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:04:04.600 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.281026+0000", 2021-03-24T22:04:04.600 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00028250399999999999 2021-03-24T22:04:04.600 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.600 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.601 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:04:04.601 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.281308+0000", 2021-03-24T22:04:04.601 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.036356240999999997 2021-03-24T22:04:04.601 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:04.601 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:04:04.602 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:04.602 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.602 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.602 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:3997 7.15 7:ad88ec1b:::smithi12556470-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e420)", 2021-03-24T22:04:04.602 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:50.245003+0000", 2021-03-24T22:04:04.603 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 254.12016618800001, 2021-03-24T22:04:04.603 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.036618082000000003, 2021-03-24T22:04:04.603 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:04:04.603 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T22:04:04.603 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:04:04.604 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:04:04.604 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:04:04.604 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3997 2021-03-24T22:04:04.604 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.604 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:04:04.605 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.605 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:04:04.605 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.245003+0000", 2021-03-24T22:04:04.605 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:04:04.605 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.606 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.606 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:04:04.606 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.245003+0000", 2021-03-24T22:04:04.606 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999995 2021-03-24T22:04:04.607 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.607 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.607 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:04:04.607 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.245003+0000", 2021-03-24T22:04:04.607 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.9249999999999999e-06 2021-03-24T22:04:04.608 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.608 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.608 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:04:04.608 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.245006+0000", 2021-03-24T22:04:04.608 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2450000000000003e-06 2021-03-24T22:04:04.609 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.609 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.609 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:04:04.609 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.245012+0000", 2021-03-24T22:04:04.609 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.2849999999999996e-06 2021-03-24T22:04:04.610 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.610 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.610 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:04:04.610 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.245016+0000", 2021-03-24T22:04:04.610 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000254861 2021-03-24T22:04:04.611 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.611 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.611 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:04.611 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.245271+0000", 2021-03-24T22:04:04.611 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.4078000000000005e-05 2021-03-24T22:04:04.612 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.612 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.612 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:04:04.612 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.245345+0000", 2021-03-24T22:04:04.612 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.022599312999999999 2021-03-24T22:04:04.613 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.613 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.613 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:04.613 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.267944+0000", 2021-03-24T22:04:04.614 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.3528999999999993e-05 2021-03-24T22:04:04.614 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.614 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.614 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:04:04.614 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.268018+0000", 2021-03-24T22:04:04.615 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.013332943 2021-03-24T22:04:04.615 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.615 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.615 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:04.615 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.281351+0000", 2021-03-24T22:04:04.616 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.2990000000000002e-05 2021-03-24T22:04:04.616 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.616 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.616 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:04:04.616 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.281444+0000", 2021-03-24T22:04:04.617 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000177443 2021-03-24T22:04:04.617 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.617 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.617 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:04:04.617 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:50.281621+0000", 2021-03-24T22:04:04.618 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.036618082000000003 2021-03-24T22:04:04.618 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:04.618 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:04:04.618 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:04.618 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.619 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.619 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.34352.0:1584 5.d 5:b4812045:::notify.4:head [watch ping cookie 94195313969280 gen 2] snapc 0=[] ondisk+write+known_if_redirected e428)", 2021-03-24T22:04:04.619 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:00:04.104713+0000", 2021-03-24T22:04:04.619 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 240.26045614, 2021-03-24T22:04:04.619 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.195035761, 2021-03-24T22:04:04.620 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:04:04.620 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T22:04:04.620 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:04:04.620 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.34352", 2021-03-24T22:04:04.620 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/3436348424", 2021-03-24T22:04:04.621 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1584 2021-03-24T22:04:04.621 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.621 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:04:04.621 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.622 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:04:04.622 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.104713+0000", 2021-03-24T22:04:04.622 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:04:04.622 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.622 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.623 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:04:04.623 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.104713+0000", 2021-03-24T22:04:04.623 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999952 2021-03-24T22:04:04.623 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.624 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.624 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:04:04.624 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.104708+0000", 2021-03-24T22:04:04.624 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.8836000000000001e-05 2021-03-24T22:04:04.624 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.625 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.625 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:04:04.625 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.104727+0000", 2021-03-24T22:04:04.625 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.7954000000000002e-05 2021-03-24T22:04:04.625 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.626 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.626 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:04:04.626 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.104775+0000", 2021-03-24T22:04:04.626 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.8598999999999999e-05 2021-03-24T22:04:04.626 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.627 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.627 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:04:04.627 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.104794+0000", 2021-03-24T22:04:04.627 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012397299999999999 2021-03-24T22:04:04.627 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.628 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.628 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:04.628 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.104918+0000", 2021-03-24T22:04:04.628 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1185e-05 2021-03-24T22:04:04.628 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.629 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.629 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:04:04.629 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.104929+0000", 2021-03-24T22:04:04.629 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.15697466500000001 2021-03-24T22:04:04.629 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.630 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.630 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:04.630 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.261904+0000", 2021-03-24T22:04:04.630 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00040677900000000002 2021-03-24T22:04:04.631 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.631 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.631 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for degraded object", 2021-03-24T22:04:04.631 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.262310+0000", 2021-03-24T22:04:04.631 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.037134201999999998 2021-03-24T22:04:04.632 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.632 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.632 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:04.632 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.299445+0000", 2021-03-24T22:04:04.632 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00014002 2021-03-24T22:04:04.633 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.633 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.633 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:04:04.633 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.299585+0000", 2021-03-24T22:04:04.633 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00016441900000000001 2021-03-24T22:04:04.634 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.634 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.634 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:04:04.634 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.299749+0000", 2021-03-24T22:04:04.634 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.195035761 2021-03-24T22:04:04.635 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:04.635 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:04:04.635 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:04.635 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.635 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.636 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.34352.0:1592 5.d 5:b4812045:::notify.4:head [watch reconnect cookie 94195313969280 gen 3] snapc 0=[] ondisk+write+known_if_redirected e433)", 2021-03-24T22:04:04.636 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:00:04.106360+0000", 2021-03-24T22:04:04.636 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 240.25880994900001, 2021-03-24T22:04:04.636 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.21183971600000001, 2021-03-24T22:04:04.636 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:04:04.637 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T22:04:04.637 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:04:04.637 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.34352", 2021-03-24T22:04:04.637 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/3436348424", 2021-03-24T22:04:04.638 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1592 2021-03-24T22:04:04.638 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.638 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:04:04.638 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.638 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:04:04.639 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.106360+0000", 2021-03-24T22:04:04.639 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:04:04.639 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.639 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.639 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:04:04.640 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.106360+0000", 2021-03-24T22:04:04.640 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T22:04:04.640 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.640 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.640 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:04:04.641 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.106358+0000", 2021-03-24T22:04:04.641 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6409999999999996e-06 2021-03-24T22:04:04.641 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.641 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.641 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:04:04.642 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.106364+0000", 2021-03-24T22:04:04.642 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3934000000000001e-05 2021-03-24T22:04:04.642 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.642 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.643 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:04:04.643 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.106378+0000", 2021-03-24T22:04:04.643 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.6040000000000004e-06 2021-03-24T22:04:04.643 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.643 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.644 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:04:04.644 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.106386+0000", 2021-03-24T22:04:04.644 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.2407e-05 2021-03-24T22:04:04.644 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.644 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.645 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:04.645 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.106458+0000", 2021-03-24T22:04:04.645 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.2400000000000001e-06 2021-03-24T22:04:04.645 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.645 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.646 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:04:04.646 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.106466+0000", 2021-03-24T22:04:04.646 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.156026741 2021-03-24T22:04:04.646 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.646 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.647 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:04.647 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.262492+0000", 2021-03-24T22:04:04.647 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6474000000000002e-05 2021-03-24T22:04:04.647 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.648 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.648 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for degraded object", 2021-03-24T22:04:04.648 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.262549+0000", 2021-03-24T22:04:04.648 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.037316715 2021-03-24T22:04:04.648 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.649 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.649 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:04.649 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.299866+0000", 2021-03-24T22:04:04.649 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.1278000000000003e-05 2021-03-24T22:04:04.649 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.650 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.650 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:04:04.650 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.299957+0000", 2021-03-24T22:04:04.650 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018242359 2021-03-24T22:04:04.650 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.651 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.651 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:04:04.651 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.318199+0000", 2021-03-24T22:04:04.651 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.21183971600000001 2021-03-24T22:04:04.651 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:04.652 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:04:04.652 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:04.652 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.652 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.652 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.34352.0:1594 5.d 5:b4812045:::notify.4:head [watch reconnect cookie 94195313969280 gen 4] snapc 0=[] ondisk+write+known_if_redirected e433)", 2021-03-24T22:04:04.653 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:00:04.106433+0000", 2021-03-24T22:04:04.653 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 240.25873631600001, 2021-03-24T22:04:04.653 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.21242717699999999, 2021-03-24T22:04:04.653 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:04:04.653 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T22:04:04.654 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:04:04.654 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.34352", 2021-03-24T22:04:04.654 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/3436348424", 2021-03-24T22:04:04.654 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1594 2021-03-24T22:04:04.654 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.655 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:04:04.655 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.655 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:04:04.655 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.106433+0000", 2021-03-24T22:04:04.656 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:04:04.656 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.656 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.656 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:04:04.656 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.106433+0000", 2021-03-24T22:04:04.657 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T22:04:04.657 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.657 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.657 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:04:04.657 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.106432+0000", 2021-03-24T22:04:04.658 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.7809999999999996e-06 2021-03-24T22:04:04.658 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.658 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.658 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:04:04.658 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.106437+0000", 2021-03-24T22:04:04.659 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.2429999999999999e-06 2021-03-24T22:04:04.659 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.659 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.659 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:04:04.659 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.106444+0000", 2021-03-24T22:04:04.660 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.254e-06 2021-03-24T22:04:04.660 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.660 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.660 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:04:04.660 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.106449+0000", 2021-03-24T22:04:04.661 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.9451999999999999e-05 2021-03-24T22:04:04.661 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.661 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.661 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:04.661 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.106479+0000", 2021-03-24T22:04:04.662 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.8959999999999999e-06 2021-03-24T22:04:04.662 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.662 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.662 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:04:04.663 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.106482+0000", 2021-03-24T22:04:04.663 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.15609192699999999 2021-03-24T22:04:04.663 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.663 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.663 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:04.664 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.262574+0000", 2021-03-24T22:04:04.664 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5994e-05 2021-03-24T22:04:04.664 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.664 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.664 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for degraded object", 2021-03-24T22:04:04.665 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.262630+0000", 2021-03-24T22:04:04.665 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.037710501 2021-03-24T22:04:04.665 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.665 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.665 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:04.666 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.300340+0000", 2021-03-24T22:04:04.666 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.5327000000000005e-05 2021-03-24T22:04:04.666 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.666 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.666 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:04:04.667 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.300426+0000", 2021-03-24T22:04:04.667 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018434756 2021-03-24T22:04:04.667 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.667 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.668 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:04:04.668 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.318860+0000", 2021-03-24T22:04:04.668 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.21242717699999999 2021-03-24T22:04:04.668 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:04.668 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:04:04.669 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:04.669 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.669 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.669 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.34352.0:2115 5.d 5:b4812045:::notify.4:head [watch reconnect cookie 94195313969280 gen 8] snapc 0=[] ondisk+write+known_if_redirected e663)", 2021-03-24T22:04:04.669 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:03:53.348621+0000", 2021-03-24T22:04:04.670 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 11.016548131, 2021-03-24T22:04:04.670 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0321329290000001, 2021-03-24T22:04:04.670 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:04:04.670 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T22:04:04.670 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:04:04.671 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.34352", 2021-03-24T22:04:04.671 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/3436348424", 2021-03-24T22:04:04.671 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 2115 2021-03-24T22:04:04.671 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.671 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:04:04.672 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.672 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:04:04.672 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:53.348621+0000", 2021-03-24T22:04:04.672 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:04:04.672 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.673 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.673 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:04:04.673 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:53.348621+0000", 2021-03-24T22:04:04.673 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999967 2021-03-24T22:04:04.673 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.674 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.674 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:04:04.674 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:53.348618+0000", 2021-03-24T22:04:04.674 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2857000000000001e-05 2021-03-24T22:04:04.675 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.675 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.675 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:04:04.675 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:53.348631+0000", 2021-03-24T22:04:04.675 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.0049e-05 2021-03-24T22:04:04.676 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.676 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.676 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:04:04.676 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:53.348661+0000", 2021-03-24T22:04:04.676 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3046e-05 2021-03-24T22:04:04.677 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.677 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.677 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:04:04.677 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:53.348674+0000", 2021-03-24T22:04:04.677 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.0600999999999996e-05 2021-03-24T22:04:04.678 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.678 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.678 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:04.678 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:53.348755+0000", 2021-03-24T22:04:04.678 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1681e-05 2021-03-24T22:04:04.679 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.679 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.679 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op must wait for map", 2021-03-24T22:04:04.679 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:53.348766+0000", 2021-03-24T22:04:04.679 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018762787 2021-03-24T22:04:04.680 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.680 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.680 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:04.680 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:53.367529+0000", 2021-03-24T22:04:04.681 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0835e-05 2021-03-24T22:04:04.681 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.681 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.681 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:04:04.681 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:53.367540+0000", 2021-03-24T22:04:04.682 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.00589598 2021-03-24T22:04:04.682 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.682 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.682 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:04.682 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:54.373436+0000", 2021-03-24T22:04:04.683 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0040566159999999999 2021-03-24T22:04:04.683 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.683 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.683 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:04:04.683 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:54.377492+0000", 2021-03-24T22:04:04.684 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0032618159999999998 2021-03-24T22:04:04.684 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.684 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.684 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:04:04.684 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:54.380754+0000", 2021-03-24T22:04:04.685 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0321329290000001 2021-03-24T22:04:04.685 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:04.685 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:04:04.685 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:04.685 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.686 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.686 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.34352.0:2120 5.d 5:b4812045:::notify.4:head [watch ping cookie 94195313969280 gen 8] snapc 0=[] ondisk+write+known_if_redirected e663)", 2021-03-24T22:04:04.686 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:03:54.280810+0000", 2021-03-24T22:04:04.686 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 10.084359412, 2021-03-24T22:04:04.687 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.097273784000000002, 2021-03-24T22:04:04.687 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:04:04.687 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T22:04:04.687 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:04:04.687 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.34352", 2021-03-24T22:04:04.688 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/3436348424", 2021-03-24T22:04:04.688 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 2120 2021-03-24T22:04:04.688 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.688 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:04:04.688 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.689 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:04:04.689 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:54.280810+0000", 2021-03-24T22:04:04.689 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:04:04.689 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.689 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.690 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:04:04.690 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:54.280810+0000", 2021-03-24T22:04:04.690 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999943 2021-03-24T22:04:04.690 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.690 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.691 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:04:04.691 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:54.280804+0000", 2021-03-24T22:04:04.691 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0305999999999998e-05 2021-03-24T22:04:04.691 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.692 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.692 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:04:04.692 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:54.280824+0000", 2021-03-24T22:04:04.692 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00047079700000000001 2021-03-24T22:04:04.692 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.693 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.693 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:04:04.693 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:54.281295+0000", 2021-03-24T22:04:04.693 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.2852000000000001e-05 2021-03-24T22:04:04.693 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.694 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.694 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:04:04.694 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:54.281318+0000", 2021-03-24T22:04:04.694 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012973800000000001 2021-03-24T22:04:04.694 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.695 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.695 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:04.695 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:54.281448+0000", 2021-03-24T22:04:04.695 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.2286999999999999e-05 2021-03-24T22:04:04.695 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.696 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.696 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:04:04.696 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:54.281470+0000", 2021-03-24T22:04:04.696 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.096323992999999997 2021-03-24T22:04:04.696 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.697 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.697 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:04.697 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:54.377794+0000", 2021-03-24T22:04:04.697 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.8368000000000005e-05 2021-03-24T22:04:04.697 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.698 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.698 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:04:04.698 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:54.377882+0000", 2021-03-24T22:04:04.698 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000201339 2021-03-24T22:04:04.698 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:04.699 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:04.699 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:04:04.699 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:54.378084+0000", 2021-03-24T22:04:04.699 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.097273784000000002 2021-03-24T22:04:04.699 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:04.700 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:04:04.700 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:04.700 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:04.700 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:04:04.701 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-24T22:04:04.701 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:04.702 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:04 smithi142 conmon[20125]: cluster 2021-03-24T22:04:02.943965+0000 mgr.y (mgr.14144) 610 : cluster [DBG] 2021-03-24T22:04:04.703 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:04 smithi142 conmon[20125]: pgmap v1171: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 2 active+remapped+backfill_toofull, 105 active+clean; 564 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s wr, 2 op/s; 149/1779 objects degraded (8.375%); 60/1779 objects misplaced (3.373%) 2021-03-24T22:04:04.703 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:04 smithi142 conmon[20125]: cluster 2021-03-24T22:04:03.360898+0000 mon.a (mon.0) 1789 : cluster [DBG] osdmap e673: 8 total, 7 up, 8 in 2021-03-24T22:04:04.703 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:04 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:04:04.703 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:04 smithi142 conmon[20125]: -03-24T22:04:03.401676+0000 mon.a (mon.0) 1790 : cluster [WRN] Health check update: Degraded data redundancy: 149/1779 objects degraded (8.375%), 20 pgs degraded (PG_DEGRADED) 2021-03-24T22:04:04.703 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:04 smithi142 conmon[20125]: audit 2021-03-24T22:04:04.704 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:04 smithi142 conmon[20125]: 2021-03-24T22:04:03.937007+0000 mon.a (mon.0) 1791 : audit 2021-03-24T22:04:04.704 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:04 smithi142 conmon[20125]: [DBG] from='client.? v1:172.21.15.125:0/2626592343' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:04:04.705 INFO:tasks.rados.rados.0.smithi125.stdout:2557: done (2 left) 2021-03-24T22:04:04.706 INFO:tasks.rados.rados.0.smithi125.stdout:2558: done (1 left) 2021-03-24T22:04:04.706 INFO:tasks.rados.rados.0.smithi125.stdout:2559: done (0 left) 2021-03-24T22:04:04.706 INFO:tasks.rados.rados.0.smithi125.stdout:2560: read oid 25 snap -1 2021-03-24T22:04:04.706 INFO:tasks.rados.rados.0.smithi125.stdout:2560: expect (ObjNum 885 snap 290 seq_num 885) 2021-03-24T22:04:04.707 INFO:tasks.rados.rados.0.smithi125.stdout:2561: copy_from oid 39 from oid 2 current snap is 291 2021-03-24T22:04:04.707 INFO:tasks.rados.rados.0.smithi125.stdout:2562: rollback oid 28 current snap is 291 2021-03-24T22:04:04.707 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 28 to 249 2021-03-24T22:04:04.707 INFO:tasks.rados.rados.0.smithi125.stdout:2563: read oid 15 snap -1 2021-03-24T22:04:04.707 INFO:tasks.rados.rados.0.smithi125.stdout:2563: expect deleted 2021-03-24T22:04:04.708 INFO:tasks.rados.rados.0.smithi125.stdout:2564: read oid 33 snap -1 2021-03-24T22:04:04.708 INFO:tasks.rados.rados.0.smithi125.stdout:2564: expect (ObjNum 764 snap 249 seq_num 764) 2021-03-24T22:04:04.708 INFO:tasks.rados.rados.0.smithi125.stdout:2565: rollback oid 24 current snap is 291 2021-03-24T22:04:04.708 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 24 to 282 2021-03-24T22:04:04.708 INFO:tasks.rados.rados.0.smithi125.stdout:2566: snap_create 2021-03-24T22:04:04.709 INFO:tasks.rados.rados.0.smithi125.stdout:2562: finishing rollback tid 0 to smithi12556470-28 2021-03-24T22:04:04.709 INFO:tasks.rados.rados.0.smithi125.stdout:2562: finishing rollback tid 1 to smithi12556470-28 2021-03-24T22:04:04.709 INFO:tasks.rados.rados.0.smithi125.stdout:2562: finishing rollback tid 2 to smithi12556470-28 2021-03-24T22:04:04.709 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 28 v 1191 (ObjNum 610 snap 185 seq_num 610) dirty exists 2021-03-24T22:04:04.710 INFO:tasks.rados.rados.0.smithi125.stdout:2560: expect (ObjNum 885 snap 290 seq_num 885) 2021-03-24T22:04:04.710 INFO:tasks.rados.rados.0.smithi125.stdout:2564: expect (ObjNum 764 snap 249 seq_num 764) 2021-03-24T22:04:04.710 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:04 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:04:04.711 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:04 smithi125 conmon[19936]: 04:02.943965+0000 mgr.y (mgr.14144) 610 : cluster [DBG] pgmap v1171: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 2 active+remapped+backfill_toofull, 105 active+clean; 564 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s wr, 2 op/s; 149/1779 objects degraded (8.375%); 60/1779 objects misplaced (3.373%) 2021-03-24T22:04:04.711 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:04 smithi125 conmon[19936]: cluster 2021-03-24T22:04:04.711 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:04 smithi125 conmon[19936]: 2021-03-24T22:04:03.360898+0000 mon.a (mon.0) 1789 : cluster [DBG] osdmap e673: 8 total, 7 up, 8 in 2021-03-24T22:04:04.711 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:04 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:04:04.711 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:04 smithi125 conmon[19936]: 03-24T22:04:03.401676+0000 mon.a (mon 2021-03-24T22:04:04.712 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:04 smithi125 conmon[19936]: .0) 1790 : cluster [WRN] Health check update: Degraded data redundancy: 149/1779 objects degraded (8.375%), 20 pgs degraded (PG_DEGRADED) 2021-03-24T22:04:04.712 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:04 smithi125 conmon[19936]: audit 2021- 2021-03-24T22:04:04.712 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:04 smithi125 conmon[19936]: 03-24T22:04:03.937007+0000 2021-03-24T22:04:04.712 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:04 smithi125 conmon[19936]: mon.a (mon.0) 1791 : audit [DBG] from='client.? v1:172.21.15.125:0/2626592343' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:04:04.713 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:04 smithi125 conmon[24927]: cluster 2021-03-24T22:04:04.713 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:04 smithi125 conmon[24927]: 2021-03 2021-03-24T22:04:04.713 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:04 smithi125 conmon[24927]: -24T22:04:02.943965+0000 mgr.y (mgr.14144) 610 : cluster [DBG] 2021-03-24T22:04:04.713 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:04 smithi125 conmon[24927]: pgmap v1171: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 2 active+remapped+backfill_toofull, 105 active+clean; 564 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s wr, 2 op/s; 149/1779 objects degraded (8.375%); 60/1779 objects misplaced (3.373%) 2021-03-24T22:04:04.714 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:04 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T22:04:04.714 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:04 smithi125 conmon[24927]: T22:04:03.360898+0000 mon.a (mon 2021-03-24T22:04:04.714 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:04 smithi125 conmon[24927]: .0) 1789 : cluster [DBG] osdmap e673: 8 total, 7 up, 8 in 2021-03-24T22:04:04.714 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:04 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:04:04.714 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:04 smithi125 conmon[24927]: -03-24T22:04:03.401676+0000 2021-03-24T22:04:04.715 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:04 smithi125 conmon[24927]: mon.a (mon.0) 1790 : cluster [WRN] Health check update: Degraded data redundancy: 149/1779 objects degraded (8.375%), 20 pgs degraded (PG_DEGRADED) 2021-03-24T22:04:04.715 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:04 smithi125 conmon[24927]: 2021-03-24T22:04:04.715 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:04 smithi125 conmon[24927]: audit 2021-03-24T22:04:04.715 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:04 smithi125 conmon[24927]: 2021-03-24T22:04:03.937007+0000 mon.a (mon 2021-03-24T22:04:04.715 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:04 smithi125 conmon[24927]: .0) 1791 : audit [DBG] from='client.? v1:172.21.15.125:0/2626592343' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:04:04.717 INFO:tasks.rados.rados.0.smithi125.stdout:2565: finishing rollback tid 1 to smithi12556470-24 2021-03-24T22:04:04.718 INFO:tasks.rados.rados.0.smithi125.stdout:2565: finishing rollback tid 2 to smithi12556470-24 2021-03-24T22:04:04.718 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 24 v 1012 (ObjNum 851 snap 278 seq_num 851) dirty exists 2021-03-24T22:04:04.718 INFO:tasks.rados.rados.0.smithi125.stdout:2561: finishing copy_from to smithi12556470-39 2021-03-24T22:04:04.718 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 39 v 430 (ObjNum 856 snap 278 seq_num 856) dirty exists 2021-03-24T22:04:04.719 INFO:tasks.rados.rados.0.smithi125.stdout:2561: finishing copy_from racing read to smithi12556470-39 2021-03-24T22:04:04.762 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:04.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:04.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:04:04.868 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:04.894 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell -n osd.5 --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph-objectstore-tool --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --force --op remove --pgid 6.1c 2021-03-24T22:04:04.928 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:04.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:04.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:04:05.033 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:05.098 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:05.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:05.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:04:05.106 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/osd.5/config 2021-03-24T22:04:05.206 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:05.258 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:05.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:05.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:05.366 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:05.432 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:05.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:05.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:04:05.448 INFO:tasks.rados.rados.0.smithi125.stdout:2560: done (6 left) 2021-03-24T22:04:05.448 INFO:tasks.rados.rados.0.smithi125.stdout:2561: done (5 left) 2021-03-24T22:04:05.448 INFO:tasks.rados.rados.0.smithi125.stdout:2562: done (4 left) 2021-03-24T22:04:05.449 INFO:tasks.rados.rados.0.smithi125.stdout:2563: done (3 left) 2021-03-24T22:04:05.449 INFO:tasks.rados.rados.0.smithi125.stdout:2564: done (2 left) 2021-03-24T22:04:05.449 INFO:tasks.rados.rados.0.smithi125.stdout:2565: done (1 left) 2021-03-24T22:04:05.449 INFO:tasks.rados.rados.0.smithi125.stdout:2566: done (0 left) 2021-03-24T22:04:05.449 INFO:tasks.rados.rados.0.smithi125.stdout:2567: delete oid 14 current snap is 292 2021-03-24T22:04:05.470 INFO:tasks.rados.rados.0.smithi125.stdout:2567: done (0 left) 2021-03-24T22:04:05.471 INFO:tasks.rados.rados.0.smithi125.stdout:2568: rollback oid 31 current snap is 292 2021-03-24T22:04:05.471 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 31 to 260 2021-03-24T22:04:05.471 INFO:tasks.rados.rados.0.smithi125.stdout:2569: write oid 15 current snap is 292 2021-03-24T22:04:05.471 INFO:tasks.rados.rados.0.smithi125.stdout:2569: seq_num 889 ranges {541423=793309,2122492=667633} 2021-03-24T22:04:05.476 INFO:tasks.rados.rados.0.smithi125.stdout:2569: writing smithi12556470-15 from 541423 to 1334732 tid 1 2021-03-24T22:04:05.486 INFO:tasks.rados.rados.0.smithi125.stdout:2569: writing smithi12556470-15 from 2122492 to 2790125 tid 2 2021-03-24T22:04:05.486 INFO:tasks.rados.rados.0.smithi125.stdout:2570: rollback oid 3 current snap is 292 2021-03-24T22:04:05.486 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 3 to 277 2021-03-24T22:04:05.487 INFO:tasks.rados.rados.0.smithi125.stdout:2571: delete oid 13 current snap is 292 2021-03-24T22:04:05.487 INFO:tasks.rados.rados.0.smithi125.stdout:2568: finishing rollback tid 0 to smithi12556470-31 2021-03-24T22:04:05.488 INFO:tasks.rados.rados.0.smithi125.stdout:2568: finishing rollback tid 1 to smithi12556470-31 2021-03-24T22:04:05.488 INFO:tasks.rados.rados.0.smithi125.stdout:2568: finishing rollback tid 2 to smithi12556470-31 2021-03-24T22:04:05.489 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 31 v 1528 (ObjNum 718 snap 229 seq_num 718) dirty exists 2021-03-24T22:04:05.491 INFO:tasks.rados.rados.0.smithi125.stdout:2570: finishing rollback tid 0 to smithi12556470-3 2021-03-24T22:04:05.493 INFO:tasks.rados.rados.0.smithi125.stdout:2570: finishing rollback tid 1 to smithi12556470-3 2021-03-24T22:04:05.493 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 3 v 313 (ObjNum 1116520976 snap 21916 seq_num 0) dirty dne 2021-03-24T22:04:05.493 INFO:tasks.rados.rados.0.smithi125.stdout:2568: done (3 left) 2021-03-24T22:04:05.493 INFO:tasks.rados.rados.0.smithi125.stdout:2570: done (2 left) 2021-03-24T22:04:05.494 INFO:tasks.rados.rados.0.smithi125.stdout:2571: done (1 left) 2021-03-24T22:04:05.494 INFO:tasks.rados.rados.0.smithi125.stdout:2572: read oid 46 snap -1 2021-03-24T22:04:05.494 INFO:tasks.rados.rados.0.smithi125.stdout:2572: expect (ObjNum 497 snap 145 seq_num 497) 2021-03-24T22:04:05.494 INFO:tasks.rados.rados.0.smithi125.stdout:2573: snap_remove snap 262 2021-03-24T22:04:05.537 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:05.561 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:05 smithi125 conmon[24927]: cluster 2021-03-24T22:04:04.410068+0000 2021-03-24T22:04:05.562 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:05 smithi125 conmon[24927]: mon.a (mon.0) 1792 : cluster [DBG] osdmap e674: 8 total, 7 up, 8 in 2021-03-24T22:04:05.562 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:05 smithi125 conmon[24927]: audit 2021-03-24 2021-03-24T22:04:05.563 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:05 smithi125 conmon[24927]: T22:04:04.828342+0000 mon.a (mon.0) 1793 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T22:04:05.563 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:05 smithi125 conmon[24927]: audit 2021- 2021-03-24T22:04:05.563 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:05 smithi125 conmon[24927]: 03-24T22:04:04.839580+0000 mon.a (mon. 2021-03-24T22:04:05.564 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:05 smithi125 conmon[24927]: 0) 1794 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:04:05.565 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:05 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:04:05.565 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:05 smithi125 conmon[24927]: 24T22:04:05.345715+0000 mon.a (mon. 2021-03-24T22:04:05.566 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:05 smithi125 conmon[24927]: 0) 1795 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:04:05.567 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:05 smithi125 conmon[19936]: cluster 2021-03-24T22:04:05.567 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:05 smithi125 conmon[19936]: 2021-03-24T22 2021-03-24T22:04:05.567 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:05 smithi125 conmon[19936]: :04:04.410068+0000 mon.a (mon.0) 1792 : cluster [DBG] osdmap e674: 8 total, 7 up, 8 in 2021-03-24T22:04:05.568 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:05 smithi125 conmon[19936]: audit 2021-03-24T22:04:04.828342+0000 2021-03-24T22:04:05.568 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:05 smithi125 conmon[19936]: mon.a (mon.0) 1793 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T22:04:05.569 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:05 smithi125 conmon[19936]: audit 2021- 2021-03-24T22:04:05.569 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:05 smithi125 conmon[19936]: 03-24T22:04:04 2021-03-24T22:04:05.570 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:05 smithi125 conmon[19936]: .839580+0000 mon.a (mon.0 2021-03-24T22:04:05.570 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:05 smithi125 conmon[19936]: ) 1794 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:04:05.571 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:05 smithi125 conmon[19936]: cluster 2021-03-24T22:04:05.571 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:05 smithi125 conmon[19936]: 2021-03-24T22:04:05.571 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:05 smithi125 conmon[19936]: 2021 2021-03-24T22:04:05.572 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:05 smithi125 conmon[19936]: - 2021-03-24T22:04:05.572 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:05 smithi125 conmon[19936]: 03 2021-03-24T22:04:05.573 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:05 smithi125 conmon[19936]: - 2021-03-24T22:04:05.574 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:05 smithi125 conmon[19936]: 24 2021-03-24T22:04:05.574 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:05 smithi125 conmon[19936]: T 2021-03-24T22:04:05.575 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:05 smithi125 conmon[19936]: 22 2021-03-24T22:04:05.575 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:05 smithi125 conmon[19936]: : 2021-03-24T22:04:05.576 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:05 smithi125 conmon[19936]: 04 2021-03-24T22:04:05.576 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:05 smithi125 conmon[19936]: : 2021-03-24T22:04:05.577 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:05 smithi125 conmon[19936]: 05 2021-03-24T22:04:05.577 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:05 smithi125 conmon[19936]: . 2021-03-24T22:04:05.578 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:05 smithi125 conmon[19936]: 345715 2021-03-24T22:04:05.578 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:05 smithi125 conmon[19936]: +0000 2021-03-24T22:04:05.579 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:05 smithi125 conmon[19936]: 2021-03-24T22:04:05.579 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:05 smithi125 conmon[19936]: mon.a 2021-03-24T22:04:05.580 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:05 smithi125 conmon[19936]: ( 2021-03-24T22:04:05.580 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:05 smithi125 conmon[19936]: mon 2021-03-24T22:04:05.581 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:05 smithi125 conmon[19936]: . 2021-03-24T22:04:05.581 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:05 smithi125 conmon[19936]: 0 2021-03-24T22:04:05.581 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:05 smithi125 conmon[19936]: ) 2021-03-24T22:04:05.582 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:05 smithi125 conmon[19936]: 1795 2021-03-24T22:04:05.582 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:05 smithi125 conmon[19936]: : 2021-03-24T22:04:05.583 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:05 smithi125 conmon[19936]: cluster 2021-03-24T22:04:05.583 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:05 smithi125 conmon[19936]: 2021-03-24T22:04:05.584 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:05 smithi125 conmon[19936]: [WRN] 2021-03-24T22:04:05.584 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:05 smithi125 conmon[19936]: 2021-03-24T22:04:05.584 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:05 smithi125 conmon[19936]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:04:05.585 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:05 smithi125 conmon[19936]: 2021-03-24T22:04:05.593 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:05.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:05.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:04:05.663 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:05 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:04:05.663 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:05 smithi142 conmon[20125]: 22:04:04.410068+0000 mon.a (mon.0) 1792 : cluster [DBG] osdmap e674: 8 total, 7 up, 8 in 2021-03-24T22:04:05.664 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:05 smithi142 conmon[20125]: audit 2021 2021-03-24T22:04:05.664 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:05 smithi142 conmon[20125]: -03-24T22: 2021-03-24T22:04:05.664 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:05 smithi142 conmon[20125]: 04:04.828342+0000 2021-03-24T22:04:05.664 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:05 smithi142 conmon[20125]: mon.a (mon. 2021-03-24T22:04:05.665 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:05 smithi142 conmon[20125]: 0) 1793 : audit 2021-03-24T22:04:05.665 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:05 smithi142 conmon[20125]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T22:04:05.665 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:05 smithi142 conmon[20125]: audit 2021-03-24T22:04:05.665 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:05 smithi142 conmon[20125]: 2021-03-24T22 2021-03-24T22:04:05.666 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:05 smithi142 conmon[20125]: :04:04. 2021-03-24T22:04:05.666 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:05 smithi142 conmon[20125]: 839580+0000 mon.a (mon 2021-03-24T22:04:05.666 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:05 smithi142 conmon[20125]: .0) 1794 : audit 2021-03-24T22:04:05.666 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:05 smithi142 conmon[20125]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:04:05.667 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:05 smithi142 conmon[20125]: cluster 2021-03-24T22:04:05.667 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:05 smithi142 conmon[20125]: 2021-03-24T22 2021-03-24T22:04:05.667 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:05 smithi142 conmon[20125]: :04:05. 2021-03-24T22:04:05.667 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:05 smithi142 conmon[20125]: 345715+0000 mon.a (mon 2021-03-24T22:04:05.668 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:05 smithi142 conmon[20125]: .0) 1795 : 2021-03-24T22:04:05.668 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:05 smithi142 conmon[20125]: cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:04:05.698 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:05.751 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:05.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:05.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:05.858 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:05.910 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:05.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:05.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:06.017 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:06.075 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:06.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:06.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:04:06.180 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:06.239 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:06.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:06.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:04:06.346 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:06.402 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:06.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:06.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:06.469 INFO:tasks.rados.rados.0.smithi125.stdout:2573: done (2 left) 2021-03-24T22:04:06.469 INFO:tasks.rados.rados.0.smithi125.stdout:2574: delete oid 31 current snap is 292 2021-03-24T22:04:06.473 INFO:tasks.rados.rados.0.smithi125.stdout:2569: finishing write tid 1 to smithi12556470-15 2021-03-24T22:04:06.473 INFO:tasks.rados.rados.0.smithi125.stdout:2569: finishing write tid 2 to smithi12556470-15 2021-03-24T22:04:06.473 INFO:tasks.rados.rados.0.smithi125.stdout:2569: finishing write tid 3 to smithi12556470-15 2021-03-24T22:04:06.473 INFO:tasks.rados.rados.0.smithi125.stdout:2569: finishing write tid 4 to smithi12556470-15 2021-03-24T22:04:06.474 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 15 v 963 (ObjNum 889 snap 292 seq_num 889) dirty exists 2021-03-24T22:04:06.474 INFO:tasks.rados.rados.0.smithi125.stdout:2569: left oid 15 (ObjNum 889 snap 292 seq_num 889) 2021-03-24T22:04:06.474 INFO:tasks.rados.rados.0.smithi125.stdout:2572: expect (ObjNum 497 snap 145 seq_num 497) 2021-03-24T22:04:06.507 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:06.554 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:06.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:06.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:06.626 INFO:tasks.rados.rados.0.smithi125.stdout:2569: done (2 left) 2021-03-24T22:04:06.627 INFO:tasks.rados.rados.0.smithi125.stdout:2572: done (1 left) 2021-03-24T22:04:06.627 INFO:tasks.rados.rados.0.smithi125.stdout:2574: done (0 left) 2021-03-24T22:04:06.627 INFO:tasks.rados.rados.0.smithi125.stdout:2575: snap_create 2021-03-24T22:04:06.660 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:06.711 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:06.716 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:06 smithi125 conmon[19936]: cluster 2021-03-24T22:04:04. 2021-03-24T22:04:06.717 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:06 smithi125 conmon[19936]: 944956+0000 mgr.y (mgr.14144) 611 : cluster [DBG] pgmap v1174: 168 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 40 active+undersized, 19 active+undersized+degraded, 2 active+remapped+backfill_toofull, 105 active+clean; 565 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 149/1782 objects degraded (8.361%); 60/1782 objects misplaced (3.367%) 2021-03-24T22:04:06.717 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:06 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:04:06.718 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:06 smithi125 conmon[19936]: 03-24T22:04:05.443762+0000 mon.a (mon.0) 1796 2021-03-24T22:04:06.718 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:06 smithi125 conmon[19936]: : cluster [DBG] osdmap e675: 8 total, 7 up, 8 in 2021-03-24T22:04:06.719 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:06 smithi125 conmon[24927]: cluster 2021-03-24T22:04 2021-03-24T22:04:06.719 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:06 smithi125 conmon[24927]: :04.944956+0000 mgr.y (mgr.14144) 611 : cluster [DBG] pgmap v1174: 168 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 40 active+undersized, 19 active+undersized+degraded, 2 active+remapped+backfill_toofull, 105 active+clean; 565 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 149/1782 objects degraded (8.361%); 60/1782 objects misplaced (3.367%) 2021-03-24T22:04:06.720 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:06 smithi125 conmon[24927]: cluster 2021-03-24T22:04:05.443762+0000 mon.a 2021-03-24T22:04:06.720 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:06 smithi125 conmon[24927]: (mon.0) 1796 : cluster [DBG] osdmap e675: 8 total, 7 up, 8 in 2021-03-24T22:04:06.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:06.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:06.795 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:06 smithi142 conmon[20125]: cluster 2021-03-24T22:04:04 2021-03-24T22:04:06.795 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:06 smithi142 conmon[20125]: .944956+0000 mgr.y (mgr.14144) 611 : cluster [DBG] pgmap v1174: 168 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 40 active+undersized, 19 active+undersized+degraded, 2 active+remapped+backfill_toofull, 105 active+clean; 565 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 149/1782 objects degraded (8.361%); 60/1782 objects misplaced (3.367%) 2021-03-24T22:04:06.796 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:06 smithi142 conmon[20125]: cluster 2021-03-24T22 2021-03-24T22:04:06.796 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:06 smithi142 conmon[20125]: :04:05.443762+0000 mon.a (mon.0) 1796 : cluster [DBG] osdmap e675: 8 total, 7 up, 8 in 2021-03-24T22:04:06.824 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:06.880 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:06.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:06.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:06.918 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.3 dump_ops_in_flight 2021-03-24T22:04:06.986 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:07.045 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:07.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:07.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:04:07.070 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-03-24T22:04:07.071 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-03-24T22:04:07.150 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:07.206 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:07.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:07.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:07.312 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:07.373 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:07.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:07.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:07.480 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:07.481 INFO:tasks.rados.rados.0.smithi125.stdout:2575: done (0 left) 2021-03-24T22:04:07.482 INFO:tasks.rados.rados.0.smithi125.stdout:2576: snap_remove snap 279 2021-03-24T22:04:07.542 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:07.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:07.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:04:07.649 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:07.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:07 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:04:07.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:07 smithi125 conmon[19936]: 24T22:04:06.465256+0000 mon.a (mon.0) 1797 : cluster [DBG] osdmap e676: 8 total, 7 up, 8 in 2021-03-24T22:04:07.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:07 smithi125 conmon[24927]: cluster 2021-03-24T22:04:07.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:07 smithi125 conmon[24927]: 2021-03-24T22:04:06. 2021-03-24T22:04:07.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:07 smithi125 conmon[24927]: 465256+0000 mon.a (mon.0) 1797 : cluster [DBG] osdmap e676: 8 total, 7 up, 8 in 2021-03-24T22:04:07.690 INFO:teuthology.orchestra.run.smithi142.stdout: marking collection for removal 2021-03-24T22:04:07.691 INFO:teuthology.orchestra.run.smithi142.stdout:setting '_remove' omap key 2021-03-24T22:04:07.691 INFO:teuthology.orchestra.run.smithi142.stdout:finish_remove_pgs 6.1c_head removing 6.1c 2021-03-24T22:04:07.692 INFO:teuthology.orchestra.run.smithi142.stdout:Remove successful 2021-03-24T22:04:07.702 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:07.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:07.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:07.807 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:07.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:07 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:04:07.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:07 smithi142 conmon[20125]: 22:04:06.465256+0000 mon.a (mon.0) 1797 : cluster [DBG] osdmap e676: 8 total, 7 up, 8 in 2021-03-24T22:04:07.870 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:07.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:07.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:07.891 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:04:07.976 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:08.029 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:08.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:08.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:08.133 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:08.192 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:08.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:08.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:04:08.300 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:08.353 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:08.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:08.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:04:08.458 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:08.495 INFO:tasks.rados.rados.0.smithi125.stdout:2576: done (0 left) 2021-03-24T22:04:08.495 INFO:tasks.rados.rados.0.smithi125.stdout:2577: read oid 44 snap 264 2021-03-24T22:04:08.495 INFO:tasks.rados.rados.0.smithi125.stdout:2577: expect (ObjNum 736 snap 235 seq_num 736) 2021-03-24T22:04:08.495 INFO:tasks.rados.rados.0.smithi125.stdout:2578: read oid 29 snap -1 2021-03-24T22:04:08.496 INFO:tasks.rados.rados.0.smithi125.stdout:2578: expect (ObjNum 872 snap 283 seq_num 872) 2021-03-24T22:04:08.496 INFO:tasks.rados.rados.0.smithi125.stdout:2579: rollback oid 9 current snap is 293 2021-03-24T22:04:08.496 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 9 to 286 2021-03-24T22:04:08.496 INFO:tasks.rados.rados.0.smithi125.stdout:2580: snap_create 2021-03-24T22:04:08.509 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:08 smithi142 conmon[20125]: cluster 2021-03-24T22:04:06 2021-03-24T22:04:08.526 INFO:tasks.rados.rados.0.smithi125.stdout:2577: expect (ObjNum 736 snap 235 seq_num 736) 2021-03-24T22:04:08.540 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:08.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:08.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:08.615 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:08 smithi125 conmon[19936]: cluster 2021-03-24T22:04:06.945790+0000 2021-03-24T22:04:08.616 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:08 smithi125 conmon[19936]: mgr.y (mgr.14144) 612 : cluster [DBG] pgmap v1177: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 2 active+remapped+backfill_toofull, 105 active+clean; 567 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.4 MiB/s rd, 676 KiB/s wr, 8 op/s; 149/1785 objects degraded (8.347%); 60/1785 objects misplaced (3.361%) 2021-03-24T22:04:08.616 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:08 smithi125 conmon[19936]: cluster 2021-03-24T22:04:07.477532+0000 mon.a (mon.0) 1798 : cluster [DBG] osdmap e677: 8 total, 7 up, 8 in 2021-03-24T22:04:08.617 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:08 smithi125 conmon[19936]: audit 2021-03-24T22:04:07.809919+0000 mon.a (mon.0) 1799 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:04:08.617 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:08 smithi125 conmon[24927]: cluster 2021-03-24T22:04:06.945790+0000 mgr.y (mgr.14144) 612 : cluster [DBG] pgmap v1177: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 2 active+remapped+backfill_toofull, 105 active+clean; 567 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.4 MiB/s rd, 676 KiB/s wr, 8 op/s; 149/1785 objects degraded (8.347%); 60/1785 objects misplaced (3.361%) 2021-03-24T22:04:08.618 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:08 smithi125 conmon[24927]: cluster 2021-03-24T22:04:07.477532+0000 mon.a (mon.0) 1798 : cluster [DBG] osdmap e677: 8 total, 7 up, 8 in 2021-03-24T22:04:08.618 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:08 smithi125 conmon[24927]: audit 2021-03-24T22:04:07.809919+0000 mon.a (mon.0) 1799 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:04:08.649 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:08.668 INFO:tasks.rados.rados.0.smithi125.stdout:2578: expect (ObjNum 872 snap 283 seq_num 872) 2021-03-24T22:04:08.702 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:08.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:08.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:04:08.807 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:08.820 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:08 smithi142 conmon[20125]: .945790+0000 mgr.y (mgr.14144) 612 : cluster [DBG] pgmap v1177: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 2 active+remapped+backfill_toofull, 105 active+clean; 567 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.4 MiB/s rd, 676 KiB/s wr, 8 op/s; 149/1785 objects degraded (8.347%); 60/1785 objects misplaced (3.361%) 2021-03-24T22:04:08.820 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:08 smithi142 conmon[20125]: cluster 2021-03-24T22:04:07 2021-03-24T22:04:08.820 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:08 smithi142 conmon[20125]: .477532+0000 mon.a (mon.0) 1798 : cluster [DBG] osdmap e677: 8 total, 7 up, 8 in 2021-03-24T22:04:08.821 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:08 smithi142 conmon[20125]: audit 2021-03-24T22:04:07. 2021-03-24T22:04:08.821 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:08 smithi142 conmon[20125]: 809919+0000 mon.a (mon.0) 1799 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:04:08.821 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:08 smithi142 conmon[20125]: 2021-03-24T22:04:08.856 INFO:tasks.rados.rados.0.smithi125.stdout:2579: finishing rollback tid 0 to smithi12556470-9 2021-03-24T22:04:08.856 INFO:tasks.rados.rados.0.smithi125.stdout:2579: finishing rollback tid 1 to smithi12556470-9 2021-03-24T22:04:08.857 INFO:tasks.rados.rados.0.smithi125.stdout:2579: finishing rollback tid 2 to smithi12556470-9 2021-03-24T22:04:08.857 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 9 v 1543 (ObjNum 861 snap 279 seq_num 861) dirty exists 2021-03-24T22:04:08.860 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:08.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:08.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:08.967 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:09.024 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:09.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:09.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:09.072 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-24T22:04:09.130 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:09.184 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:09.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:09.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:09.291 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:09.347 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:09.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:09.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:09.381 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell -n osd.5 --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.6.1c.5 2021-03-24T22:04:09.451 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:09.494 INFO:tasks.rados.rados.0.smithi125.stdout:2577: done (3 left) 2021-03-24T22:04:09.495 INFO:tasks.rados.rados.0.smithi125.stdout:2578: done (2 left) 2021-03-24T22:04:09.495 INFO:tasks.rados.rados.0.smithi125.stdout:2579: done (1 left) 2021-03-24T22:04:09.495 INFO:tasks.rados.rados.0.smithi125.stdout:2580: done (0 left) 2021-03-24T22:04:09.495 INFO:tasks.rados.rados.0.smithi125.stdout:2581: snap_create 2021-03-24T22:04:09.496 INFO:teuthology.orchestra.run.smithi125.stderr:nodeep-scrub is set 2021-03-24T22:04:09.520 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:09.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:09.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:09.582 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:04:09.592 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/osd.5/config 2021-03-24T22:04:09.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:09 smithi142 conmon[20125]: cluster 2021-03-24T22:04:08 2021-03-24T22:04:09.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:09 smithi142 conmon[20125]: .485822+0000 mon.a (mon.0) 1800 : cluster [DBG] osdmap e678: 8 total, 7 up, 8 in 2021-03-24T22:04:09.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:09 smithi142 conmon[20125]: audit 2021-03-24T 2021-03-24T22:04:09.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:09 smithi142 conmon[20125]: 22:04:08.597252+0000 mon.c (mon.2) 101 : audit 2021-03-24T22:04:09.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:09 smithi142 conmon[20125]: [INF] from='client.? v1:172.21.15.125:0/843102968' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:04:09.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:09 smithi142 conmon[20125]: audit 2021-03-24T22 2021-03-24T22:04:09.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:09 smithi142 conmon[20125]: :04:08.597612+0000 mon.a (mon.0 2021-03-24T22:04:09.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:09 smithi142 conmon[20125]: ) 1801 : audit [INF] from='client.35636 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:04:09.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:09 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:04:09.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:09 smithi142 conmon[20125]: -24T22:04:09.472789+0000 2021-03-24T22:04:09.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:09 smithi142 conmon[20125]: mon.a (mon.0) 1802 : cluster [WRN] 2021-03-24T22:04:09.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:09 smithi142 conmon[20125]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:04:09.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:09 smithi142 conmon[20125]: audit 2021-03-24T22:04:09.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:09 smithi142 conmon[20125]: 2021-03-24T22:04:09. 2021-03-24T22:04:09.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:09 smithi142 conmon[20125]: 479992+0000 mon.a (mon.0) 1803 : audit 2021-03-24T22:04:09.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:09 smithi142 conmon[20125]: [INF] from='client.35636 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T22:04:09.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:09 smithi142 conmon[20125]: cluster 2021-03-24T22 2021-03-24T22:04:09.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:09 smithi142 conmon[20125]: :04:09.480083+0000 mon.a (mon. 2021-03-24T22:04:09.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:09 smithi142 conmon[20125]: 0) 1804 : cluster [DBG] osdmap e679: 8 total, 7 up, 8 in 2021-03-24T22:04:09.625 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:09.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[24927]: cluster 2021-03-24T22:04:08.485822+0000 2021-03-24T22:04:09.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[24927]: mon.a (mon.0) 1800 : cluster [DBG] osdmap e678: 8 total, 7 up, 8 in 2021-03-24T22:04:09.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[24927]: audit 2021-03 2021-03-24T22:04:09.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[24927]: -24T22:04:08.597252+0000 mon.c (mon. 2021-03-24T22:04:09.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[24927]: 2) 101 : audit [INF] from='client.? v1:172.21.15.125:0/843102968' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:04:09.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[24927]: audit 2021-03 2021-03-24T22:04:09.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[24927]: -24T22:04:08.597612+0000 mon.a (mon.0) 2021-03-24T22:04:09.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[24927]: 1801 : audit [INF] from='client.35636 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:04:09.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T22:04:09.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[24927]: -24T22:04:09.472789+0000 mon.a (mon.0 2021-03-24T22:04:09.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[24927]: ) 1802 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:04:09.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[24927]: audit 2021-03 2021-03-24T22:04:09.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[24927]: -24T22:04:09.479992+0000 mon.a (mon.0 2021-03-24T22:04:09.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[24927]: ) 1803 : audit [INF] from='client.35636 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T22:04:09.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T22:04:09.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[24927]: -24T22:04:09.480083+0000 mon.a (mon 2021-03-24T22:04:09.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[24927]: .0) 1804 : cluster [DBG] osdmap e679: 8 total, 7 up, 8 in 2021-03-24T22:04:09.671 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: cluster 2021-03-24T22:04:09.671 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 2021-03-24T22:04:09.672 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 2021 2021-03-24T22:04:09.672 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: - 2021-03-24T22:04:09.672 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 03 2021-03-24T22:04:09.673 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: - 2021-03-24T22:04:09.673 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 24 2021-03-24T22:04:09.674 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: T 2021-03-24T22:04:09.674 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 22 2021-03-24T22:04:09.675 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: : 2021-03-24T22:04:09.675 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 04 2021-03-24T22:04:09.675 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: : 2021-03-24T22:04:09.676 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 08 2021-03-24T22:04:09.676 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: . 2021-03-24T22:04:09.677 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 485822 2021-03-24T22:04:09.677 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: +0000 2021-03-24T22:04:09.678 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 2021-03-24T22:04:09.678 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: mon.a 2021-03-24T22:04:09.679 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: ( 2021-03-24T22:04:09.679 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: mon 2021-03-24T22:04:09.682 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: . 2021-03-24T22:04:09.682 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 0 2021-03-24T22:04:09.683 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: ) 2021-03-24T22:04:09.683 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 1800 2021-03-24T22:04:09.683 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: : 2021-03-24T22:04:09.684 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: cluster 2021-03-24T22:04:09.684 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 2021-03-24T22:04:09.684 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: [DBG] 2021-03-24T22:04:09.685 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 2021-03-24T22:04:09.685 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: osdmap e678: 8 total, 7 up, 8 in 2021-03-24T22:04:09.685 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 2021-03-24T22:04:09.685 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: audit 2021-03-24T22:04:09.686 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 2021-03-24T22:04:09.686 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 2021 2021-03-24T22:04:09.686 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: - 2021-03-24T22:04:09.687 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 03 2021-03-24T22:04:09.687 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: - 2021-03-24T22:04:09.688 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 24 2021-03-24T22:04:09.688 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: T 2021-03-24T22:04:09.689 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 22 2021-03-24T22:04:09.689 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: : 2021-03-24T22:04:09.691 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 04 2021-03-24T22:04:09.691 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: : 2021-03-24T22:04:09.691 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 08 2021-03-24T22:04:09.691 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: . 2021-03-24T22:04:09.692 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 597252 2021-03-24T22:04:09.692 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: +0000 2021-03-24T22:04:09.692 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 2021-03-24T22:04:09.692 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: mon.c 2021-03-24T22:04:09.693 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: ( 2021-03-24T22:04:09.693 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: mon 2021-03-24T22:04:09.693 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: . 2021-03-24T22:04:09.693 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 2 2021-03-24T22:04:09.694 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: ) 2021-03-24T22:04:09.694 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 101 2021-03-24T22:04:09.694 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: : 2021-03-24T22:04:09.695 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: audit 2021-03-24T22:04:09.695 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 2021-03-24T22:04:09.695 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: [INF] 2021-03-24T22:04:09.696 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 2021-03-24T22:04:09.696 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: from='client.? v1:172.21.15.125:0/843102968' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:04:09.696 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 2021-03-24T22:04:09.696 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: audit 2021-03-24T22:04:09.697 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 2021-03-24T22:04:09.697 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 2021 2021-03-24T22:04:09.697 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: - 2021-03-24T22:04:09.697 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 03 2021-03-24T22:04:09.698 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: - 2021-03-24T22:04:09.698 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 24 2021-03-24T22:04:09.698 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: T 2021-03-24T22:04:09.698 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 22 2021-03-24T22:04:09.699 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: : 2021-03-24T22:04:09.699 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 04 2021-03-24T22:04:09.699 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: : 2021-03-24T22:04:09.699 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 08 2021-03-24T22:04:09.700 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: . 2021-03-24T22:04:09.700 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 597612 2021-03-24T22:04:09.700 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: +0000 2021-03-24T22:04:09.700 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 2021-03-24T22:04:09.700 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: mon.a 2021-03-24T22:04:09.701 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: ( 2021-03-24T22:04:09.701 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: mon 2021-03-24T22:04:09.701 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: . 2021-03-24T22:04:09.701 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 0 2021-03-24T22:04:09.702 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: ) 2021-03-24T22:04:09.702 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 1801 2021-03-24T22:04:09.702 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: : 2021-03-24T22:04:09.702 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: audit 2021-03-24T22:04:09.703 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 2021-03-24T22:04:09.703 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: [INF] 2021-03-24T22:04:09.703 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 2021-03-24T22:04:09.703 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: from='client.35636 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:04:09.703 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 2021-03-24T22:04:09.704 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: cluster 2021-03-24T22:04:09.704 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 2021-03-24T22:04:09.704 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 2021 2021-03-24T22:04:09.704 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: - 2021-03-24T22:04:09.705 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 03 2021-03-24T22:04:09.705 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: - 2021-03-24T22:04:09.705 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 24 2021-03-24T22:04:09.705 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: T 2021-03-24T22:04:09.706 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 22 2021-03-24T22:04:09.706 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: : 2021-03-24T22:04:09.706 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 04 2021-03-24T22:04:09.706 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: : 2021-03-24T22:04:09.707 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 09 2021-03-24T22:04:09.707 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: . 2021-03-24T22:04:09.707 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 472789 2021-03-24T22:04:09.707 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: +0000 2021-03-24T22:04:09.708 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 2021-03-24T22:04:09.708 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: mon.a 2021-03-24T22:04:09.708 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: ( 2021-03-24T22:04:09.708 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: mon 2021-03-24T22:04:09.708 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: . 2021-03-24T22:04:09.709 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 0 2021-03-24T22:04:09.709 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: ) 2021-03-24T22:04:09.709 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 1802 2021-03-24T22:04:09.709 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: : 2021-03-24T22:04:09.710 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: cluster 2021-03-24T22:04:09.710 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 2021-03-24T22:04:09.710 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: [WRN] 2021-03-24T22:04:09.710 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 2021-03-24T22:04:09.711 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:04:09.711 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 2021-03-24T22:04:09.711 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: audit 2021-03-24T22:04:09.711 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 2021-03-24T22:04:09.712 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 2021 2021-03-24T22:04:09.712 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: - 2021-03-24T22:04:09.712 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 03 2021-03-24T22:04:09.712 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: - 2021-03-24T22:04:09.713 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 24 2021-03-24T22:04:09.713 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: T 2021-03-24T22:04:09.713 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 22 2021-03-24T22:04:09.713 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: : 2021-03-24T22:04:09.714 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 04 2021-03-24T22:04:09.714 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: : 2021-03-24T22:04:09.714 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 09 2021-03-24T22:04:09.714 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: . 2021-03-24T22:04:09.715 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 479992 2021-03-24T22:04:09.715 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: +0000 2021-03-24T22:04:09.715 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 2021-03-24T22:04:09.715 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: mon.a 2021-03-24T22:04:09.716 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: ( 2021-03-24T22:04:09.716 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: mon 2021-03-24T22:04:09.716 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: . 2021-03-24T22:04:09.716 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 0 2021-03-24T22:04:09.716 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: ) 2021-03-24T22:04:09.717 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 1803 2021-03-24T22:04:09.717 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: : 2021-03-24T22:04:09.717 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: audit 2021-03-24T22:04:09.717 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 2021-03-24T22:04:09.718 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: [INF] 2021-03-24T22:04:09.718 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 2021-03-24T22:04:09.718 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: from='client.35636 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T22:04:09.718 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 2021-03-24T22:04:09.719 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: cluster 2021-03-24T22:04:09.719 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 2021-03-24T22:04:09.719 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 2021 2021-03-24T22:04:09.719 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: - 2021-03-24T22:04:09.720 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 03 2021-03-24T22:04:09.720 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: - 2021-03-24T22:04:09.720 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 24 2021-03-24T22:04:09.720 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: T 2021-03-24T22:04:09.721 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 22:04:09.480083+0000 mon.a (mon.0) 2021-03-24T22:04:09.721 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:09 smithi125 conmon[19936]: 1804 : cluster [DBG] osdmap e679: 8 total, 7 up, 8 in 2021-03-24T22:04:09.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:09.723 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:09.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:04:09.824 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:09.881 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:09.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:09.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:04:09.986 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:10.037 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:10.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:10.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:10.143 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:10.192 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:10.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:10.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:10.301 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:10.368 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:10.371 INFO:tasks.rados.rados.0.smithi125.stdout:2581: done (0 left) 2021-03-24T22:04:10.371 INFO:tasks.rados.rados.0.smithi125.stdout:2582: delete oid 46 current snap is 295 2021-03-24T22:04:10.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:10.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:10.408 INFO:tasks.rados.rados.0.smithi125.stdout:2582: done (0 left) 2021-03-24T22:04:10.409 INFO:tasks.rados.rados.0.smithi125.stdout:2583: delete oid 19 current snap is 295 2021-03-24T22:04:10.460 INFO:tasks.rados.rados.0.smithi125.stdout:2583: done (0 left) 2021-03-24T22:04:10.460 INFO:tasks.rados.rados.0.smithi125.stdout:2584: snap_remove snap 294 2021-03-24T22:04:10.478 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:10.540 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:10.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:10.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:10.646 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:10.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:10 smithi125 conmon[19936]: cluster 2021-03-24T22:04: 2021-03-24T22:04:10.660 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:10 smithi125 conmon[19936]: 08.946623+0000 mgr.y (mgr.14144) 613 : cluster [DBG] pgmap v1180: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 2 active+remapped+backfill_toofull, 105 active+clean; 567 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 357 KiB/s wr, 2 op/s; 149/1785 objects degraded (8.347%); 60/1785 objects misplaced (3.361%) 2021-03-24T22:04:10.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:10 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:04:10.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:10 smithi125 conmon[19936]: T22:04:10.348306+0000 mon.a ( 2021-03-24T22:04:10.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:10 smithi125 conmon[19936]: mon.0) 1805 : cluster 2021-03-24T22:04:10.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:10 smithi125 conmon[19936]: [WRN] Health check update: Degraded data redundancy: 149/1785 objects degraded (8.347%), 20 pgs degraded (PG_DEGRADED) 2021-03-24T22:04:10.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:10 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:04:10.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:10 smithi125 conmon[19936]: -03-24T22:04:10 2021-03-24T22:04:10.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:10 smithi125 conmon[19936]: .355320+0000 mon.a (mon. 2021-03-24T22:04:10.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:10 smithi125 conmon[19936]: 0) 1806 : cluster [DBG] 2021-03-24T22:04:10.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:10 smithi125 conmon[19936]: osdmap e680: 8 total, 7 up, 8 in 2021-03-24T22:04:10.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:10 smithi125 conmon[24927]: cluster 2021-03-24T22 2021-03-24T22:04:10.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:10 smithi125 conmon[24927]: :04:08.946623+0000 mgr.y (mgr.14144) 613 : cluster [DBG] 2021-03-24T22:04:10.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:10 smithi125 conmon[24927]: pgmap v1180: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 2 active+remapped+backfill_toofull, 105 active+clean; 567 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 357 KiB/s wr, 2 op/s; 149/1785 objects degraded (8.347%); 60/1785 objects misplaced (3.361%) 2021-03-24T22:04:10.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:10 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:04:10.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:10 smithi125 conmon[24927]: 03-24T22:04:10.348306 2021-03-24T22:04:10.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:10 smithi125 conmon[24927]: +0000 mon.a (mon 2021-03-24T22:04:10.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:10 smithi125 conmon[24927]: .0) 1805 : cluster [WRN] Health check update: Degraded data redundancy: 149/1785 objects degraded (8.347%), 20 pgs degraded (PG_DEGRADED) 2021-03-24T22:04:10.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:10 smithi125 conmon[24927]: cluster 2021-03-24T22:04:10.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:10 smithi125 conmon[24927]: 2021-03-24T22:04:10.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:10 smithi125 conmon[24927]: 2021 2021-03-24T22:04:10.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:10 smithi125 conmon[24927]: - 2021-03-24T22:04:10.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:10 smithi125 conmon[24927]: 03-24T22:04 2021-03-24T22:04:10.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:10 smithi125 conmon[24927]: :10.355320+0000 mon.a (mon.0) 1806 : cluster [DBG] osdmap e680: 8 total, 7 up, 8 in 2021-03-24T22:04:10.670 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-24T22:04:10.670 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-24T22:04:10.670 INFO:teuthology.orchestra.run.smithi125.stdout: "num_ops": 0 2021-03-24T22:04:10.670 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-24T22:04:10.706 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:10.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:10.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:04:10.811 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:10.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:10 smithi142 conmon[20125]: cluster 2021-03-24T22:04:08.946623+0000 mgr.y (mgr.14144) 613 : cluster [DBG] pgmap v1180: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 2 active+remapped+backfill_toofull, 105 active+clean; 567 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 357 KiB/s wr, 2 op/s; 149/1785 objects degraded (8.347%); 60/1785 objects misplaced (3.361%) 2021-03-24T22:04:10.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:10 smithi142 conmon[20125]: cluster 2021-03-24T22:04:10.348306+0000 mon.a (mon.0) 1805 : cluster [WRN] Health check update: Degraded data redundancy: 149/1785 objects degraded (8.347%), 20 pgs degraded (PG_DEGRADED) 2021-03-24T22:04:10.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:10 smithi142 conmon[20125]: cluster 2021-03-24T22:04:10.355320+0000 mon.a (mon.0) 1806 : cluster [DBG] osdmap e680: 8 total, 7 up, 8 in 2021-03-24T22:04:10.867 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:10.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:10.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:10.973 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:11.017 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:11.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:11.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:04:11.123 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:11.173 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:11.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:11.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:11.280 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:11.340 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:11.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:11.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:04:11.383 INFO:tasks.rados.rados.0.smithi125.stdout:2584: done (0 left) 2021-03-24T22:04:11.383 INFO:tasks.rados.rados.0.smithi125.stdout:2585: snap_create 2021-03-24T22:04:11.448 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:11.504 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:11.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:11.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:11.610 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:11.663 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:11.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:11.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:11.768 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:11.831 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:11.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:11.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:11.842 INFO:teuthology.orchestra.run.smithi142.stdout:get_pg_num_history pg_num_history pg_num_history(e662 pg_nums {1={19=1},2={56=3},3={60=32,381=42},4={62=32,413=31,441=30,445=29,449=28,453=27,457=26,461=25,465=24,469=23,473=22},5={64=32},6={66=32},7={275=36}} deleted_pools ) 2021-03-24T22:04:11.842 INFO:teuthology.orchestra.run.smithi142.stdout:Importing pgid 6.1c 2021-03-24T22:04:11.843 INFO:teuthology.orchestra.run.smithi142.stdout:write_pg epoch 629 info 6.1c( empty local-lis/les=546/547 n=0 ec=66/66 lis/c=546/546 les/c/f=547/547/0 sis=546) 2021-03-24T22:04:11.843 INFO:teuthology.orchestra.run.smithi142.stdout:Import successful 2021-03-24T22:04:11.936 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:11.993 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:11.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:11.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:12.099 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:12.156 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:12.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:12.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:12.261 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:12.323 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:12.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:12.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:04:12.388 INFO:tasks.rados.rados.0.smithi125.stdout:2585: done (0 left) 2021-03-24T22:04:12.388 INFO:tasks.rados.rados.0.smithi125.stdout:2586: write (excl) oid 10 current snap is 296 2021-03-24T22:04:12.388 INFO:tasks.rados.rados.0.smithi125.stdout:2586: seq_num 894 ranges {625458=485907,1614902=534982,2917332=108126} 2021-03-24T22:04:12.394 INFO:tasks.rados.rados.0.smithi125.stdout:2586: writing smithi12556470-10 from 625458 to 1111365 tid 1 2021-03-24T22:04:12.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: cluster 2021-03-24T22:04:12.407 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: 2021-03-24T22:04:12.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: 2021 2021-03-24T22:04:12.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: - 2021-03-24T22:04:12.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: 03 2021-03-24T22:04:12.408 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: - 2021-03-24T22:04:12.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: 24 2021-03-24T22:04:12.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: T 2021-03-24T22:04:12.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: 22 2021-03-24T22:04:12.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: : 2021-03-24T22:04:12.409 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: 04 2021-03-24T22:04:12.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: : 2021-03-24T22:04:12.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: 10 2021-03-24T22:04:12.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: . 2021-03-24T22:04:12.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: 947574 2021-03-24T22:04:12.410 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: +0000 2021-03-24T22:04:12.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: 2021-03-24T22:04:12.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: mgr.y 2021-03-24T22:04:12.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: ( 2021-03-24T22:04:12.411 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: mgr 2021-03-24T22:04:12.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: . 2021-03-24T22:04:12.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: 14144 2021-03-24T22:04:12.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: ) 2021-03-24T22:04:12.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: 614 2021-03-24T22:04:12.412 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: : 2021-03-24T22:04:12.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: cluster 2021-03-24T22:04:12.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: 2021-03-24T22:04:12.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: [DBG] 2021-03-24T22:04:12.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: 2021-03-24T22:04:12.413 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: pgmap v1183: 168 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 40 active+undersized, 19 active+undersized+degraded, 2 active+remapped+backfill_toofull, 105 active+clean; 576 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 583 KiB/s wr, 12 op/s; 149/1794 objects degraded (8.305%); 60/1794 objects misplaced (3.344%) 2021-03-24T22:04:12.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: 2021-03-24T22:04:12.414 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: cluster 2021-03-24T22:04:12.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: 2021-03-24T22:04:12.415 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: 2021 2021-03-24T22:04:12.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: cluster 2021-03-24T22:04:12.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: 2021-03-24T22:04:12.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: 2021 2021-03-24T22:04:12.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: - 2021-03-24T22:04:12.416 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: 03 2021-03-24T22:04:12.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: - 2021-03-24T22:04:12.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: 24 2021-03-24T22:04:12.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: T 2021-03-24T22:04:12.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: 22 2021-03-24T22:04:12.417 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: : 2021-03-24T22:04:12.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: 04 2021-03-24T22:04:12.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: : 2021-03-24T22:04:12.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: 10 2021-03-24T22:04:12.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: . 2021-03-24T22:04:12.418 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: 947574 2021-03-24T22:04:12.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: +0000 2021-03-24T22:04:12.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: 2021-03-24T22:04:12.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: mgr.y 2021-03-24T22:04:12.419 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: ( 2021-03-24T22:04:12.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: mgr 2021-03-24T22:04:12.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: . 2021-03-24T22:04:12.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: 14144 2021-03-24T22:04:12.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: ) 2021-03-24T22:04:12.420 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: 614 2021-03-24T22:04:12.421 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: : 2021-03-24T22:04:12.421 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: cluster 2021-03-24T22:04:12.421 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: 2021-03-24T22:04:12.421 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: [DBG] 2021-03-24T22:04:12.421 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: 2021-03-24T22:04:12.422 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: pgmap v1183: 168 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 40 active+undersized, 19 active+undersized+degraded, 2 active+remapped+backfill_toofull, 105 active+clean; 576 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 583 KiB/s wr, 12 op/s; 149/1794 objects degraded (8.305%); 60/1794 objects misplaced (3.344%) 2021-03-24T22:04:12.422 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: 2021-03-24T22:04:12.422 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: cluster 2021-03-24T22:04:12.422 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: 2021-03-24T22:04:12.422 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: 2021 2021-03-24T22:04:12.423 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: - 2021-03-24T22:04:12.423 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: 03 2021-03-24T22:04:12.423 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: - 2021-03-24T22:04:12.423 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: 24 2021-03-24T22:04:12.423 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: T 2021-03-24T22:04:12.424 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: 22 2021-03-24T22:04:12.424 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: : 2021-03-24T22:04:12.424 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: 04 2021-03-24T22:04:12.425 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: : 2021-03-24T22:04:12.425 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: 11 2021-03-24T22:04:12.425 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: . 2021-03-24T22:04:12.426 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: 376619 2021-03-24T22:04:12.426 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: +0000 2021-03-24T22:04:12.426 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: 2021-03-24T22:04:12.426 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: mon.a 2021-03-24T22:04:12.426 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: ( 2021-03-24T22:04:12.427 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: mon 2021-03-24T22:04:12.427 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: . 2021-03-24T22:04:12.427 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: 0 2021-03-24T22:04:12.427 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: ) 2021-03-24T22:04:12.427 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: 1807 2021-03-24T22:04:12.428 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: : 2021-03-24T22:04:12.428 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: cluster 2021-03-24T22:04:12.428 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: 2021-03-24T22:04:12.428 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: [DBG] 2021-03-24T22:04:12.428 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: 2021-03-24T22:04:12.429 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: osdmap e681: 8 total, 7 up, 8 in 2021-03-24T22:04:12.429 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[19936]: 2021-03-24T22:04:12.430 INFO:tasks.rados.rados.0.smithi125.stdout:2586: writing smithi12556470-10 from 1614902 to 2149884 tid 2 2021-03-24T22:04:12.430 INFO:tasks.rados.rados.0.smithi125.stdout:2586: writing smithi12556470-10 from 2917332 to 3025458 tid 3 2021-03-24T22:04:12.431 INFO:tasks.rados.rados.0.smithi125.stdout:2587: delete oid 13 current snap is 296 2021-03-24T22:04:12.431 INFO:tasks.rados.rados.0.smithi125.stdout:2586: finishing write tid 1 to smithi12556470-10 2021-03-24T22:04:12.431 INFO:tasks.rados.rados.0.smithi125.stdout:2586: finishing write tid 2 to smithi12556470-10 2021-03-24T22:04:12.431 INFO:tasks.rados.rados.0.smithi125.stdout:2587: done (1 left) 2021-03-24T22:04:12.431 INFO:tasks.rados.rados.0.smithi125.stdout:2588: delete oid 12 current snap is 296 2021-03-24T22:04:12.432 INFO:tasks.rados.rados.0.smithi125.stdout:2586: finishing write tid 3 to smithi12556470-10 2021-03-24T22:04:12.432 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:12.433 INFO:tasks.rados.rados.0.smithi125.stdout:2586: finishing write tid 4 to smithi12556470-10 2021-03-24T22:04:12.434 INFO:tasks.rados.rados.0.smithi125.stdout:2586: finishing write tid 5 to smithi12556470-10 2021-03-24T22:04:12.434 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 10 v 1017 (ObjNum 894 snap 296 seq_num 894) dirty exists 2021-03-24T22:04:12.434 INFO:tasks.rados.rados.0.smithi125.stdout:2586: left oid 10 (ObjNum 894 snap 296 seq_num 894) 2021-03-24T22:04:12.442 INFO:tasks.rados.rados.0.smithi125.stdout:2586: done (1 left) 2021-03-24T22:04:12.442 INFO:tasks.rados.rados.0.smithi125.stdout:2588: done (0 left) 2021-03-24T22:04:12.443 INFO:tasks.rados.rados.0.smithi125.stdout:2589: snap_create 2021-03-24T22:04:12.487 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:12.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:12.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:04:12.592 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:12.597 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: - 2021-03-24T22:04:12.597 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: 03 2021-03-24T22:04:12.598 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: - 2021-03-24T22:04:12.598 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: 24 2021-03-24T22:04:12.598 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: T 2021-03-24T22:04:12.598 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: 22 2021-03-24T22:04:12.598 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: : 2021-03-24T22:04:12.599 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: 04 2021-03-24T22:04:12.599 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: : 2021-03-24T22:04:12.599 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: 11 2021-03-24T22:04:12.599 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: . 2021-03-24T22:04:12.600 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: 376619 2021-03-24T22:04:12.600 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: +0000 2021-03-24T22:04:12.600 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: 2021-03-24T22:04:12.600 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: mon.a 2021-03-24T22:04:12.600 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: ( 2021-03-24T22:04:12.601 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: mon 2021-03-24T22:04:12.601 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: . 2021-03-24T22:04:12.601 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: 0 2021-03-24T22:04:12.601 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: ) 2021-03-24T22:04:12.602 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: 1807 2021-03-24T22:04:12.602 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: : 2021-03-24T22:04:12.602 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: cluster 2021-03-24T22:04:12.602 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: 2021-03-24T22:04:12.603 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: [DBG] 2021-03-24T22:04:12.603 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: 2021-03-24T22:04:12.603 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: osdmap e681: 8 total, 7 up, 8 in 2021-03-24T22:04:12.603 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:12 smithi125 conmon[24927]: 2021-03-24T22:04:12.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:12 smithi142 conmon[20125]: cluster 2021-03-24T22:04:10.947574 2021-03-24T22:04:12.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:12 smithi142 conmon[20125]: +0000 mgr.y (mgr.14144) 614 : cluster [DBG] pgmap v1183: 168 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 40 active+undersized, 19 active+undersized+degraded, 2 active+remapped+backfill_toofull, 105 active+clean; 576 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 583 KiB/s wr, 12 op/s; 149/1794 objects degraded (8.305%); 60/1794 objects misplaced (3.344%) 2021-03-24T22:04:12.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:12 smithi142 conmon[20125]: cluster 2021-03-24T22:04 2021-03-24T22:04:12.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:12 smithi142 conmon[20125]: :11.376619+0000 mon.a (mon.0) 1807 : cluster [DBG] osdmap e681: 8 total, 7 up, 8 in 2021-03-24T22:04:12.647 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:12.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:12.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:12.662 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.3 dump_blocked_ops 2021-03-24T22:04:12.753 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:12.801 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:12.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:12.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:12.906 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:12.965 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:12.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:12.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:13.071 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:13.129 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:13.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:13.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:04:13.236 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:13.292 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:13.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:13.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:13.397 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:13.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:13 smithi125 conmon[19936]: cluster 2021-03-24T22:04:13.414 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:13 smithi125 conmon[19936]: 2021-03- 2021-03-24T22:04:13.415 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:13 smithi125 conmon[19936]: 24T22:04:12.381392+0000 mon.a (mon.0) 1808 : cluster [DBG] osdmap e682: 8 total, 7 up, 8 in 2021-03-24T22:04:13.416 INFO:tasks.rados.rados.0.smithi125.stdout:2589: done (0 left) 2021-03-24T22:04:13.416 INFO:tasks.rados.rados.0.smithi125.stdout:2590: snap_create 2021-03-24T22:04:13.427 DEBUG:teuthology.orchestra.run.smithi142:> sudo rm -f /var/log/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/exp.6.1c.5 2021-03-24T22:04:13.451 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:13.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:13.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:13.557 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:13.611 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:13.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:13.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:04:13.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:13 smithi142 conmon[20125]: cluster 2021-03-24T22:04:12.381392+0000 mon.a 2021-03-24T22:04:13.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:13 smithi142 conmon[20125]: (mon.0) 1808 : cluster [DBG] osdmap e682: 8 total, 7 up, 8 in 2021-03-24T22:04:13.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:13 smithi125 conmon[24927]: cluster 2021-03-24T22 2021-03-24T22:04:13.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:13 smithi125 conmon[24927]: :04:12.381392+0000 mon.a (mon.0) 1808 : cluster [DBG] osdmap e682: 8 total, 7 up, 8 in 2021-03-24T22:04:13.718 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:13.781 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:13.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:13.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:04:13.889 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:13.946 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:13.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:13.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:04:14.000 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:04:14.054 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:14.116 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:14.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:14.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:04:14.223 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:14.289 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:14.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:14.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:14.394 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:14.424 INFO:tasks.rados.rados.0.smithi125.stdout:2590: done (0 left) 2021-03-24T22:04:14.425 INFO:tasks.rados.rados.0.smithi125.stdout:2591: write oid 27 current snap is 298 2021-03-24T22:04:14.425 INFO:tasks.rados.rados.0.smithi125.stdout:2591: seq_num 897 ranges {675879=711654,2073995=682309} 2021-03-24T22:04:14.434 INFO:tasks.rados.rados.0.smithi125.stdout:2591: writing smithi12556470-27 from 675879 to 1387533 tid 1 2021-03-24T22:04:14.443 INFO:tasks.rados.rados.0.smithi125.stdout:2591: writing smithi12556470-27 from 2073995 to 2756304 tid 2 2021-03-24T22:04:14.443 INFO:tasks.rados.rados.0.smithi125.stdout:2592: snap_remove snap 246 2021-03-24T22:04:14.451 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:14.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:14.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:04:14.508 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:14 smithi125 conmon[19936]: cluster 2021-03-24T22:04:12.948293+0000 2021-03-24T22:04:14.508 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:14 smithi125 conmon[19936]: mgr.y (mgr.14144) 615 : cluster [DBG] pgmap v1186: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 2 active+remapped+backfill_toofull, 105 active+clean; 576 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 582 KiB/s wr, 14 op/s; 149/1794 objects degraded (8.305%); 60/1794 objects misplaced (3.344%) 2021-03-24T22:04:14.509 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:14 smithi125 conmon[19936]: cluster 2021-03-24T22:04:13.412545+0000 mon.a (mon.0) 2021-03-24T22:04:14.509 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:14 smithi125 conmon[19936]: 1809 : cluster [DBG] osdmap e683: 8 total, 7 up, 8 in 2021-03-24T22:04:14.509 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:14 smithi125 conmon[19936]: audit 2021-03 2021-03-24T22:04:14.509 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:14 smithi125 conmon[19936]: -24T22:04:14.400126+0000 mon.a (mon.0) 2021-03-24T22:04:14.510 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:14 smithi125 conmon[19936]: 1810 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:04:14.510 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:14 smithi125 conmon[24927]: cluster 2021-03-24T22:04:12.948293+0000 mgr.y (mgr.14144) 615 : cluster [DBG] pgmap v1186: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 2 active+remapped+backfill_toofull, 105 active+clean; 576 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 582 KiB/s wr, 14 op/s; 149/1794 objects degraded (8.305%); 60/1794 objects misplaced (3.344%) 2021-03-24T22:04:14.510 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:14 smithi125 conmon[24927]: cluster 2021-03-24T22:04:13.412545+0000 mon.a (mon.0) 1809 : cluster [DBG] osdmap e683: 8 total, 7 up, 8 in 2021-03-24T22:04:14.510 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:14 smithi125 conmon[24927]: audit 2021-03-24T22:04:14.400126+0000 mon.a (mon.0) 1810 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:04:14.557 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:14.613 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:14.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:14.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:04:14.635 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd unset noscrub 2021-03-24T22:04:14.718 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:14.733 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:14 smithi142 conmon[20125]: cluster 2021-03-24T22:04:12 2021-03-24T22:04:14.733 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:14 smithi142 conmon[20125]: .948293+0000 mgr.y (mgr.14144) 615 : cluster [DBG] pgmap v1186: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 2 active+remapped+backfill_toofull, 105 active+clean; 576 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 582 KiB/s wr, 14 op/s; 149/1794 objects degraded (8.305%); 60/1794 objects misplaced (3.344%) 2021-03-24T22:04:14.733 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:14 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:04:14.734 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:14 smithi142 conmon[20125]: -24T22:04:13.412545+0000 mon.a (mon.0) 1809 : 2021-03-24T22:04:14.734 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:14 smithi142 conmon[20125]: cluster [DBG] osdmap e683: 8 total, 7 up, 8 in 2021-03-24T22:04:14.734 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:14 smithi142 conmon[20125]: audit 2021-03-24T22: 2021-03-24T22:04:14.734 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:14 smithi142 conmon[20125]: 04:14.400126+0000 mon.a (mon.0 2021-03-24T22:04:14.735 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:14 smithi142 conmon[20125]: ) 1810 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:04:14.776 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:14.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:14.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:04:14.883 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:14.936 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:14.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:14.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:04:15.043 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:15.099 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:15.101 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T22:04:15.101 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T22:04:15.102 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T22:04:15.102 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T22:04:15.102 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T22:04:15.102 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T22:04:15.103 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T22:04:15.103 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-24T22:04:15.103 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-24T22:04:15.103 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-24T22:04:15.104 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-24T22:04:15.104 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-24T22:04:15.104 INFO:teuthology.orchestra.run.smithi125.stderr:Error ENXIO: problem getting command descriptions from osd.5 2021-03-24T22:04:15.104 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-24T22:04:15.105 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-24T22:04:15.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:15.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:04:15.147 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:04:15.208 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:15.260 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:15.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:15.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:04:15.367 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:15.431 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:15.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:15.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:15.439 INFO:tasks.rados.rados.0.smithi125.stdout:2592: done (1 left) 2021-03-24T22:04:15.439 INFO:tasks.rados.rados.0.smithi125.stdout:2593: write (excl) oid 11 current snap is 298 2021-03-24T22:04:15.440 INFO:tasks.rados.rados.0.smithi125.stdout:2593: seq_num 898 ranges {692686=503256,1848830=425967,2910671=182015} 2021-03-24T22:04:15.451 INFO:tasks.rados.rados.0.smithi125.stdout:2593: writing smithi12556470-11 from 692686 to 1195942 tid 1 2021-03-24T22:04:15.458 INFO:tasks.rados.rados.0.smithi125.stdout:2593: writing smithi12556470-11 from 1848830 to 2274797 tid 2 2021-03-24T22:04:15.465 INFO:tasks.rados.rados.0.smithi125.stdout:2593: writing smithi12556470-11 from 2910671 to 3092686 tid 3 2021-03-24T22:04:15.465 INFO:tasks.rados.rados.0.smithi125.stdout:2594: write oid 26 current snap is 298 2021-03-24T22:04:15.466 INFO:tasks.rados.rados.0.smithi125.stdout:2594: seq_num 899 ranges {709493=578505,1907312=685978} 2021-03-24T22:04:15.473 INFO:tasks.rados.rados.0.smithi125.stdout:2594: writing smithi12556470-26 from 709493 to 1287998 tid 1 2021-03-24T22:04:15.482 INFO:tasks.rados.rados.0.smithi125.stdout:2594: writing smithi12556470-26 from 1907312 to 2593290 tid 2 2021-03-24T22:04:15.482 INFO:tasks.rados.rados.0.smithi125.stdout:2595: write oid 48 current snap is 298 2021-03-24T22:04:15.482 INFO:tasks.rados.rados.0.smithi125.stdout:2591: finishing write tid 1 to smithi12556470-27 2021-03-24T22:04:15.483 INFO:tasks.rados.rados.0.smithi125.stdout:2591: finishing write tid 2 to smithi12556470-27 2021-03-24T22:04:15.483 INFO:tasks.rados.rados.0.smithi125.stdout:2591: finishing write tid 3 to smithi12556470-27 2021-03-24T22:04:15.483 INFO:tasks.rados.rados.0.smithi125.stdout:2591: finishing write tid 4 to smithi12556470-27 2021-03-24T22:04:15.483 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 27 v 1415 (ObjNum 897 snap 298 seq_num 897) dirty exists 2021-03-24T22:04:15.484 INFO:tasks.rados.rados.0.smithi125.stdout:2591: left oid 27 (ObjNum 897 snap 298 seq_num 897) 2021-03-24T22:04:15.484 INFO:tasks.rados.rados.0.smithi125.stdout:2593: finishing write tid 1 to smithi12556470-11 2021-03-24T22:04:15.484 INFO:tasks.rados.rados.0.smithi125.stdout:2593: finishing write tid 2 to smithi12556470-11 2021-03-24T22:04:15.484 INFO:tasks.rados.rados.0.smithi125.stdout:2593: finishing write tid 3 to smithi12556470-11 2021-03-24T22:04:15.485 INFO:tasks.rados.rados.0.smithi125.stdout:2593: finishing write tid 4 to smithi12556470-11 2021-03-24T22:04:15.485 INFO:tasks.rados.rados.0.smithi125.stdout:2595: seq_num 900 ranges {726300=653754,1965794=429636,2965517=160783} 2021-03-24T22:04:15.491 INFO:tasks.rados.rados.0.smithi125.stdout:2595: writing smithi12556470-48 from 726300 to 1380054 tid 1 2021-03-24T22:04:15.497 INFO:tasks.rados.rados.0.smithi125.stdout:2595: writing smithi12556470-48 from 1965794 to 2395430 tid 2 2021-03-24T22:04:15.503 INFO:tasks.rados.rados.0.smithi125.stdout:2595: writing smithi12556470-48 from 2965517 to 3126300 tid 3 2021-03-24T22:04:15.503 INFO:tasks.rados.rados.0.smithi125.stdout:2591: done (3 left) 2021-03-24T22:04:15.504 INFO:tasks.rados.rados.0.smithi125.stdout:2596: copy_from oid 19 from oid 42 current snap is 298 2021-03-24T22:04:15.504 INFO:tasks.rados.rados.0.smithi125.stdout:2593: finishing write tid 5 to smithi12556470-11 2021-03-24T22:04:15.505 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 11 v 553 (ObjNum 898 snap 298 seq_num 898) dirty exists 2021-03-24T22:04:15.505 INFO:tasks.rados.rados.0.smithi125.stdout:2593: left oid 11 (ObjNum 898 snap 298 seq_num 898) 2021-03-24T22:04:15.506 INFO:tasks.rados.rados.0.smithi125.stdout:2593: done (3 left) 2021-03-24T22:04:15.506 INFO:tasks.rados.rados.0.smithi125.stdout:2597: snap_remove snap 292 2021-03-24T22:04:15.536 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:15.591 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:15.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:15.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:04:15.699 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:15.713 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:15 smithi125 conmon[19936]: cluster 2021-03-24T22:04:14 2021-03-24T22:04:15.713 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:15 smithi125 conmon[19936]: .416169+0000 mon.a (mon.0) 1811 : cluster [DBG] osdmap e684: 8 total, 7 up, 8 in 2021-03-24T22:04:15.713 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:15 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T22:04:15.714 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:15 smithi125 conmon[19936]: -24T22:04:14.479417+0000 osd.0 (osd.0) 131 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:04:15.714 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:15 smithi125 conmon[19936]: 2021-03-24T22:04:15.714 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:15 smithi125 conmon[19936]: cluster 2021-03-24T22:04: 2021-03-24T22:04:15.714 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:15 smithi125 conmon[19936]: 14.481545+0000 osd.0 (osd.0) 132 : cluster 2021-03-24T22:04:15.715 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:15 smithi125 conmon[19936]: [DBG] purged_snaps scrub ok 2021-03-24T22:04:15.715 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:15 smithi125 conmon[19936]: audit 2021-03-24T22 2021-03-24T22:04:15.715 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:15 smithi125 conmon[19936]: :04:14.759533+0000 mon.a (mon.0) 2021-03-24T22:04:15.715 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:15 smithi125 conmon[19936]: 1812 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:04:15.715 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:15 smithi125 conmon[19936]: audit 2021-03- 2021-03-24T22:04:15.716 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:15 smithi125 conmon[19936]: 24T22:04:14.818076+0000 mon.c (mon. 2021-03-24T22:04:15.716 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:15 smithi125 conmon[19936]: 2) 102 : audit [DBG] from='client.? v1:172.21.15.125:0/4015101019' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:04:15.716 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:15 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:04:15.716 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:15 smithi125 conmon[19936]: 03-24T22:04:15.350472+0000 2021-03-24T22:04:15.717 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:15 smithi125 conmon[19936]: mon.a (mon.0) 1813 : cluster [WRN] Health check update: Degraded data redundancy: 149/1794 objects degraded (8.305%), 20 pgs degraded (PG_DEGRADED) 2021-03-24T22:04:15.717 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:15 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T22:04:15.718 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:15 smithi125 conmon[24927]: T22:04:14.416169+0000 mon.a (mon.0) 1811 : cluster [DBG] osdmap e684: 8 total, 7 up, 8 in 2021-03-24T22:04:15.718 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:15 smithi125 conmon[24927]: 2021-03-24T22:04:15.718 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:15 smithi125 conmon[24927]: cluster 2021-03-24T22:04:15.718 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:15 smithi125 conmon[24927]: 2021-03-24T22: 2021-03-24T22:04:15.719 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:15 smithi125 conmon[24927]: 04:14.479417+0000 osd.0 ( 2021-03-24T22:04:15.719 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:15 smithi125 conmon[24927]: osd.0) 131 : cluster 2021-03-24T22:04:15.719 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:15 smithi125 conmon[24927]: [DBG] purged_snaps scrub starts 2021-03-24T22:04:15.719 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:15 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:04:15.720 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:15 smithi125 conmon[24927]: 24T22:04:14.481545+0000 osd.0 (osd.0 2021-03-24T22:04:15.720 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:15 smithi125 conmon[24927]: ) 132 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:04:15.720 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:15 smithi125 conmon[24927]: audit 2021-03-24T22:04:15.720 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:15 smithi125 conmon[24927]: 2021-03-24 2021-03-24T22:04:15.720 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:15 smithi125 conmon[24927]: T22:04:14.759533+0000 mon.a (mon 2021-03-24T22:04:15.721 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:15 smithi125 conmon[24927]: .0) 1812 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:04:15.721 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:15 smithi125 conmon[24927]: audit 2021- 2021-03-24T22:04:15.721 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:15 smithi125 conmon[24927]: 03-24T22:04:14.818076+0000 mon.c 2021-03-24T22:04:15.722 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:15 smithi125 conmon[24927]: (mon.2) 102 : audit [DBG] from='client.? v1:172.21.15.125:0/4015101019' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:04:15.722 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:15 smithi125 conmon[24927]: 2021-03-24T22:04:15.722 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:15 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T22:04:15.723 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:15 smithi125 conmon[24927]: T22:04:15.350472+0000 mon.a (mon. 2021-03-24T22:04:15.723 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:15 smithi125 conmon[24927]: 0) 1813 : cluster [WRN] Health check update: Degraded data redundancy: 149/1794 objects degraded (8.305%), 20 pgs degraded (PG_DEGRADED) 2021-03-24T22:04:15.751 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:15.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:15.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:15.858 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:15.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:15 smithi142 conmon[20125]: cluster 2021-03-24T22:04:14.416169+0000 mon.a (mon.0) 1811 : cluster [DBG] osdmap e684: 8 total, 7 up, 8 in 2021-03-24T22:04:15.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:15 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:04:15.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:15 smithi142 conmon[20125]: -24T22:04:14.479417+0000 osd.0 (osd.0) 131 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:04:15.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:15 smithi142 conmon[20125]: cluster 2021-03-24T22:04:14.481545+0000 osd.0 (osd.0) 132 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:04:15.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:15 smithi142 conmon[20125]: audit 2021-03-24T22:04: 2021-03-24T22:04:15.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:15 smithi142 conmon[20125]: 14.759533+0000 mon.a (mon.0) 1812 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:04:15.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:15 smithi142 conmon[20125]: audit 2021-03-24T22:04: 2021-03-24T22:04:15.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:15 smithi142 conmon[20125]: 14.818076+0000 mon.c (mon.2) 102 : audit [DBG] from='client.? v1:172.21.15.125:0/4015101019' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:04:15.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:15 smithi142 conmon[20125]: cluster 2021-03-24T22:04:15.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:15 smithi142 conmon[20125]: 2021-03-24T22:04:15.350472+0000 mon.a (mon.0) 1813 : cluster [WRN] Health check update: Degraded data redundancy: 149/1794 objects degraded (8.305%), 20 pgs degraded (PG_DEGRADED) 2021-03-24T22:04:15.922 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:15.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:15.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:16.029 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:16.085 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:16.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:16.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:16.190 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:16.247 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:16.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:16.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:16.355 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:16.411 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:16.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:16.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:16.443 INFO:tasks.rados.rados.0.smithi125.stdout:2597: done (3 left) 2021-03-24T22:04:16.443 INFO:tasks.rados.rados.0.smithi125.stdout:2598: snap_create 2021-03-24T22:04:16.443 INFO:tasks.rados.rados.0.smithi125.stdout:2594: finishing write tid 1 to smithi12556470-26 2021-03-24T22:04:16.443 INFO:tasks.rados.rados.0.smithi125.stdout:2594: finishing write tid 2 to smithi12556470-26 2021-03-24T22:04:16.444 INFO:tasks.rados.rados.0.smithi125.stdout:2594: finishing write tid 3 to smithi12556470-26 2021-03-24T22:04:16.444 INFO:tasks.rados.rados.0.smithi125.stdout:2594: finishing write tid 4 to smithi12556470-26 2021-03-24T22:04:16.444 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 26 v 459 (ObjNum 899 snap 298 seq_num 899) dirty exists 2021-03-24T22:04:16.444 INFO:tasks.rados.rados.0.smithi125.stdout:2594: left oid 26 (ObjNum 899 snap 298 seq_num 899) 2021-03-24T22:04:16.445 INFO:tasks.rados.rados.0.smithi125.stdout:2595: finishing write tid 1 to smithi12556470-48 2021-03-24T22:04:16.445 INFO:tasks.rados.rados.0.smithi125.stdout:2595: finishing write tid 2 to smithi12556470-48 2021-03-24T22:04:16.445 INFO:tasks.rados.rados.0.smithi125.stdout:2595: finishing write tid 3 to smithi12556470-48 2021-03-24T22:04:16.445 INFO:tasks.rados.rados.0.smithi125.stdout:2595: finishing write tid 4 to smithi12556470-48 2021-03-24T22:04:16.446 INFO:tasks.rados.rados.0.smithi125.stdout:2595: finishing write tid 5 to smithi12556470-48 2021-03-24T22:04:16.446 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 48 v 925 (ObjNum 900 snap 298 seq_num 900) dirty exists 2021-03-24T22:04:16.446 INFO:tasks.rados.rados.0.smithi125.stdout:2595: left oid 48 (ObjNum 900 snap 298 seq_num 900) 2021-03-24T22:04:16.446 INFO:tasks.rados.rados.0.smithi125.stdout:2596: finishing copy_from to smithi12556470-19 2021-03-24T22:04:16.447 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 19 v 503 (ObjNum 736 snap 235 seq_num 736) dirty exists 2021-03-24T22:04:16.447 INFO:tasks.rados.rados.0.smithi125.stdout:2596: finishing copy_from racing read to smithi12556470-19 2021-03-24T22:04:16.517 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:16.583 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:16.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:16.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:16.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:16 smithi125 conmon[24927]: cluster 2021-03-24T22:04 2021-03-24T22:04:16.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:16 smithi125 conmon[24927]: :14.949169+0000 mgr.y (mgr.14144) 616 : cluster [DBG] pgmap v1189: 168 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 40 active+undersized, 19 active+undersized+degraded, 2 active+remapped+backfill_toofull, 105 active+clean; 579 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.8 MiB/s rd, 276 KiB/s wr, 5 op/s; 150/1803 objects degraded (8.319%); 60/1803 objects misplaced (3.328%) 2021-03-24T22:04:16.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:16 smithi125 conmon[24927]: cluster 2021-03-24T22 2021-03-24T22:04:16.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:16 smithi125 conmon[24927]: :04:15.433570+0000 mon.a (mon.0) 1814 : cluster [DBG] osdmap e685: 8 total, 7 up, 8 in 2021-03-24T22:04:16.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:16 smithi125 conmon[24927]: audit 2021-03-24T22:04:16.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:16 smithi125 conmon[24927]: 2021-03-24T22:04:15.442660+0000 mon.a (mon.0) 1815 : audit 2021-03-24T22:04:16.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:16 smithi125 conmon[24927]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T22:04:16.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:16 smithi125 conmon[19936]: cluster 2021-03-24T22:04: 2021-03-24T22:04:16.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:16 smithi125 conmon[19936]: 14.949169+0000 mgr.y (mgr.14144) 616 : cluster [DBG] pgmap v1189: 168 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 40 active+undersized, 19 active+undersized+degraded, 2 active+remapped+backfill_toofull, 105 active+clean; 579 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.8 MiB/s rd, 276 KiB/s wr, 5 op/s; 150/1803 objects degraded (8.319%); 60/1803 objects misplaced (3.328%) 2021-03-24T22:04:16.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:16 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:04:16.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:16 smithi125 conmon[19936]: 04:15.433570+0000 mon.a (mon.0) 1814 : cluster 2021-03-24T22:04:16.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:16 smithi125 conmon[19936]: [DBG] osdmap e685: 8 total, 7 up, 8 in 2021-03-24T22:04:16.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:16 smithi125 conmon[19936]: audit 2021 2021-03-24T22:04:16.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:16 smithi125 conmon[19936]: -03-24T22:04:15 2021-03-24T22:04:16.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:16 smithi125 conmon[19936]: .442660+0000 mon.a (mon.0 2021-03-24T22:04:16.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:16 smithi125 conmon[19936]: ) 1815 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T22:04:16.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:16 smithi125 conmon[19936]: 2021-03-24T22:04:16.691 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:16.755 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:16.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:16.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:16.864 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:16.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:16 smithi142 conmon[20125]: cluster 2021-03-24T22:04:14.949169 2021-03-24T22:04:16.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:16 smithi142 conmon[20125]: +0000 mgr.y (mgr.14144) 616 : cluster [DBG] pgmap v1189: 168 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 40 active+undersized, 19 active+undersized+degraded, 2 active+remapped+backfill_toofull, 105 active+clean; 579 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.8 MiB/s rd, 276 KiB/s wr, 5 op/s; 150/1803 objects degraded (8.319%); 60/1803 objects misplaced (3.328%) 2021-03-24T22:04:16.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:16 smithi142 conmon[20125]: cluster 2021-03-24T22:04:15.433570+0000 mon.a ( 2021-03-24T22:04:16.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:16 smithi142 conmon[20125]: mon.0) 1814 : cluster [DBG] osdmap e685: 8 total, 7 up, 8 in 2021-03-24T22:04:16.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:16 smithi142 conmon[20125]: audit 2021-03-24 2021-03-24T22:04:16.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:16 smithi142 conmon[20125]: T22:04:15.442660+0000 mon.a (mon.0) 1815 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T22:04:16.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:16 smithi142 conmon[20125]: 2021-03-24T22:04:16.930 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:16.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:16.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:04:17.040 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:17.093 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:17.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:17.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:17.198 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:17.252 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:17.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:17.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:04:17.288 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-03-24T22:04:17.290 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-03-24T22:04:17.358 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:17.408 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:17.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:17.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:17.452 INFO:tasks.rados.rados.0.smithi125.stdout:2594: done (3 left) 2021-03-24T22:04:17.452 INFO:tasks.rados.rados.0.smithi125.stdout:2595: done (2 left) 2021-03-24T22:04:17.452 INFO:tasks.rados.rados.0.smithi125.stdout:2596: done (1 left) 2021-03-24T22:04:17.453 INFO:tasks.rados.rados.0.smithi125.stdout:2598: done (0 left) 2021-03-24T22:04:17.453 INFO:tasks.rados.rados.0.smithi125.stdout:2599: snap_remove snap 286 2021-03-24T22:04:17.514 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:17.559 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:17.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:17.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:04:17.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:17 smithi125 conmon[24927]: cluster 2021-03-24T22:04:16.439277+0000 2021-03-24T22:04:17.654 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:17 smithi125 conmon[24927]: mon.a (mon.0) 1816 : cluster [DBG] osdmap e686: 8 total, 7 up, 8 in 2021-03-24T22:04:17.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:17 smithi125 conmon[19936]: cluster 2021-03-24T22:04:17.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:17 smithi125 conmon[19936]: 2021-03-24T22:04:17.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:17 smithi125 conmon[19936]: 2021 2021-03-24T22:04:17.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:17 smithi125 conmon[19936]: - 2021-03-24T22:04:17.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:17 smithi125 conmon[19936]: 03 2021-03-24T22:04:17.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:17 smithi125 conmon[19936]: - 2021-03-24T22:04:17.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:17 smithi125 conmon[19936]: 24 2021-03-24T22:04:17.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:17 smithi125 conmon[19936]: T22:04:16.439277+0000 mon.a (mon. 2021-03-24T22:04:17.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:17 smithi125 conmon[19936]: 0) 1816 : cluster [DBG] osdmap e686: 8 total, 7 up, 8 in 2021-03-24T22:04:17.664 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:17.719 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:17.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:17.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:04:17.824 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:18.093 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-24T22:04:18.093 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T22:04:18.095 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:17 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:04:18.095 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:17 smithi142 conmon[20125]: T22:04:16.439277+0000 mon.a (mon.0) 1816 : cluster [DBG] osdmap e686: 8 total, 7 up, 8 in 2021-03-24T22:04:18.160 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:18.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:18.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:04:18.267 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:18.323 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:18.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:18.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:04:18.428 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:18.438 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 1] out_osds: [] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 6, 7] 2021-03-24T22:04:18.438 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-24T22:04:18.439 INFO:tasks.thrashosds.thrasher:fixing pg num pool default.rgw.control 2021-03-24T22:04:18.440 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph pg dump --format=json 2021-03-24T22:04:18.462 INFO:tasks.rados.rados.0.smithi125.stdout:2599: done (0 left) 2021-03-24T22:04:18.463 INFO:tasks.rados.rados.0.smithi125.stdout:2600: write (excl) oid 39 current snap is 299 2021-03-24T22:04:18.463 INFO:tasks.rados.rados.0.smithi125.stdout:2600: seq_num 901 ranges {743107=729003,1907923=689647} 2021-03-24T22:04:18.474 INFO:tasks.rados.rados.0.smithi125.stdout:2600: writing smithi12556470-39 from 743107 to 1472110 tid 1 2021-03-24T22:04:18.482 INFO:tasks.rados.rados.0.smithi125.stdout:2600: writing smithi12556470-39 from 1907923 to 2597570 tid 2 2021-03-24T22:04:18.483 INFO:tasks.rados.rados.0.smithi125.stdout:2600: finishing write tid 1 to smithi12556470-39 2021-03-24T22:04:18.483 INFO:tasks.rados.rados.0.smithi125.stdout:2601: copy_from oid 26 from oid 1 current snap is 299 2021-03-24T22:04:18.483 INFO:tasks.rados.rados.0.smithi125.stdout:2602: delete oid 35 current snap is 299 2021-03-24T22:04:18.492 INFO:tasks.rados.rados.0.smithi125.stdout:2600: finishing write tid 2 to smithi12556470-39 2021-03-24T22:04:18.493 INFO:tasks.rados.rados.0.smithi125.stdout:2600: finishing write tid 3 to smithi12556470-39 2021-03-24T22:04:18.498 INFO:tasks.rados.rados.0.smithi125.stdout:2600: finishing write tid 4 to smithi12556470-39 2021-03-24T22:04:18.499 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 39 v 1021 (ObjNum 901 snap 299 seq_num 901) dirty exists 2021-03-24T22:04:18.499 INFO:tasks.rados.rados.0.smithi125.stdout:2600: left oid 39 (ObjNum 901 snap 299 seq_num 901) 2021-03-24T22:04:18.506 INFO:tasks.rados.rados.0.smithi125.stdout:2600: done (2 left) 2021-03-24T22:04:18.507 INFO:tasks.rados.rados.0.smithi125.stdout:2602: done (1 left) 2021-03-24T22:04:18.507 INFO:tasks.rados.rados.0.smithi125.stdout:2603: copy_from oid 5 from oid 10 current snap is 299 2021-03-24T22:04:18.508 INFO:tasks.rados.rados.0.smithi125.stdout:2604: snap_create 2021-03-24T22:04:18.531 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:18.538 INFO:tasks.rados.rados.0.smithi125.stdout:2603: finishing copy_from to smithi12556470-5 2021-03-24T22:04:18.538 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 5 v 1017 (ObjNum 894 snap 296 seq_num 894) dirty exists 2021-03-24T22:04:18.538 INFO:tasks.rados.rados.0.smithi125.stdout:2603: finishing copy_from racing read to smithi12556470-5 2021-03-24T22:04:18.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:18.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:18.547 INFO:tasks.rados.rados.0.smithi125.stdout:2601: finishing copy_from to smithi12556470-26 2021-03-24T22:04:18.547 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 26 v 827 (ObjNum 808 snap 265 seq_num 808) dirty exists 2021-03-24T22:04:18.547 INFO:tasks.rados.rados.0.smithi125.stdout:2601: finishing copy_from racing read to smithi12556470-26 2021-03-24T22:04:18.641 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:18.687 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:18.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:18.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:04:18.692 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:18 smithi125 conmon[24927]: cluster 2021-03-24T22:04:16.949946+0000 mgr.y ( 2021-03-24T22:04:18.692 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:18 smithi125 conmon[24927]: mgr.14144) 617 : cluster [DBG] pgmap v1192: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 2 active+remapped+backfill_toofull, 105 active+clean; 579 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 276 KiB/s wr, 3 op/s; 150/1803 objects degraded (8.319%); 60/1803 objects misplaced (3.328%) 2021-03-24T22:04:18.692 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:18 smithi125 conmon[24927]: cluster 2021-03-24T22:04:17.447415+0000 mon.a (mon.0) 1817 : cluster [DBG] osdmap e687: 8 total, 7 up, 8 in 2021-03-24T22:04:18.693 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:18 smithi125 conmon[19936]: cluster 2021-03-24T22:04:16.949946+0000 2021-03-24T22:04:18.693 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:18 smithi125 conmon[19936]: mgr.y (mgr.14144) 617 : cluster [DBG] pgmap v1192: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 2 active+remapped+backfill_toofull, 105 active+clean; 579 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 276 KiB/s wr, 3 op/s; 150/1803 objects degraded (8.319%); 60/1803 objects misplaced (3.328%) 2021-03-24T22:04:18.693 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:18 smithi125 conmon[19936]: cluster 2021-03-24T22:04:17.447415+0000 mon.a (mon. 2021-03-24T22:04:18.693 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:18 smithi125 conmon[19936]: 0) 1817 : cluster [DBG] osdmap e687: 8 total, 7 up, 8 in 2021-03-24T22:04:18.793 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:18.850 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:18.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:18.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:18.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:18 smithi142 conmon[20125]: cluster 2021-03-24T22:04:18.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:18 smithi142 conmon[20125]: 2021-03 2021-03-24T22:04:18.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:18 smithi142 conmon[20125]: -24T22:04:16.949946+0000 mgr.y (mgr.14144) 617 : cluster [DBG] pgmap v1192: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 2 active+remapped+backfill_toofull, 105 active+clean; 579 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 276 KiB/s wr, 3 op/s; 150/1803 objects degraded (8.319%); 60/1803 objects misplaced (3.328%) 2021-03-24T22:04:18.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:18 smithi142 conmon[20125]: cluster 2021-03-24T22:04:18.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:18 smithi142 conmon[20125]: 2021-03-24T22:04 2021-03-24T22:04:18.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:18 smithi142 conmon[20125]: :17.447415+0000 mon.a (mon.0) 1817 : cluster [DBG] osdmap e687: 8 total, 7 up, 8 in 2021-03-24T22:04:18.956 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:19.011 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:19.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:19.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:04:19.118 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:19.177 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:19.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:19.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:04:19.285 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:19.290 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-24T22:04:19.342 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:19.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:19.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:19.448 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:19.486 INFO:tasks.rados.rados.0.smithi125.stdout:2601: done (2 left) 2021-03-24T22:04:19.487 INFO:tasks.rados.rados.0.smithi125.stdout:2603: done (1 left) 2021-03-24T22:04:19.487 INFO:tasks.rados.rados.0.smithi125.stdout:2604: done (0 left) 2021-03-24T22:04:19.487 INFO:tasks.rados.rados.0.smithi125.stdout:2605: delete oid 10 current snap is 300 2021-03-24T22:04:19.491 INFO:tasks.rados.rados.0.smithi125.stdout:2605: done (0 left) 2021-03-24T22:04:19.491 INFO:tasks.rados.rados.0.smithi125.stdout:2606: write oid 39 current snap is 300 2021-03-24T22:04:19.491 INFO:tasks.rados.rados.0.smithi125.stdout:2606: seq_num 904 ranges {793528=554750,2083369=436974,2958856=234672} 2021-03-24T22:04:19.501 INFO:tasks.rados.rados.0.smithi125.stdout:2606: writing smithi12556470-39 from 793528 to 1348278 tid 1 2021-03-24T22:04:19.504 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:19.507 INFO:tasks.rados.rados.0.smithi125.stdout:2606: writing smithi12556470-39 from 2083369 to 2520343 tid 2 2021-03-24T22:04:19.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:19.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:19.512 INFO:tasks.rados.rados.0.smithi125.stdout:2606: writing smithi12556470-39 from 2958856 to 3193528 tid 3 2021-03-24T22:04:19.512 INFO:tasks.rados.rados.0.smithi125.stdout:2607: rollback oid 28 current snap is 300 2021-03-24T22:04:19.513 INFO:tasks.rados.rados.0.smithi125.stdout:2606: finishing write tid 1 to smithi12556470-39 2021-03-24T22:04:19.513 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 28 to 264 2021-03-24T22:04:19.513 INFO:tasks.rados.rados.0.smithi125.stdout:2608: write oid 26 current snap is 300 2021-03-24T22:04:19.513 INFO:tasks.rados.rados.0.smithi125.stdout:2608: seq_num 905 ranges {410335=746352,1741851=696985,3102632=107703} 2021-03-24T22:04:19.524 INFO:tasks.rados.rados.0.smithi125.stdout:2608: writing smithi12556470-26 from 410335 to 1156687 tid 1 2021-03-24T22:04:19.532 INFO:tasks.rados.rados.0.smithi125.stdout:2608: writing smithi12556470-26 from 1741851 to 2438836 tid 2 2021-03-24T22:04:19.537 INFO:tasks.rados.rados.0.smithi125.stdout:2608: writing smithi12556470-26 from 3102632 to 3210335 tid 3 2021-03-24T22:04:19.539 INFO:tasks.rados.rados.0.smithi125.stdout:2606: finishing write tid 2 to smithi12556470-39 2021-03-24T22:04:19.539 INFO:tasks.rados.rados.0.smithi125.stdout:2606: finishing write tid 3 to smithi12556470-39 2021-03-24T22:04:19.539 INFO:tasks.rados.rados.0.smithi125.stdout:2606: finishing write tid 4 to smithi12556470-39 2021-03-24T22:04:19.539 INFO:tasks.rados.rados.0.smithi125.stdout:2606: finishing write tid 5 to smithi12556470-39 2021-03-24T22:04:19.540 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 39 v 1028 (ObjNum 904 snap 300 seq_num 904) dirty exists 2021-03-24T22:04:19.540 INFO:tasks.rados.rados.0.smithi125.stdout:2606: left oid 39 (ObjNum 904 snap 300 seq_num 904) 2021-03-24T22:04:19.540 INFO:tasks.rados.rados.0.smithi125.stdout:2607: finishing rollback tid 0 to smithi12556470-28 2021-03-24T22:04:19.540 INFO:tasks.rados.rados.0.smithi125.stdout:2607: finishing rollback tid 1 to smithi12556470-28 2021-03-24T22:04:19.540 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 28 v 1193 (ObjNum 0 snap 7 seq_num 3) dirty dne 2021-03-24T22:04:19.541 INFO:tasks.rados.rados.0.smithi125.stdout:2606: done (2 left) 2021-03-24T22:04:19.541 INFO:tasks.rados.rados.0.smithi125.stdout:2607: done (1 left) 2021-03-24T22:04:19.541 INFO:tasks.rados.rados.0.smithi125.stdout:2609: write oid 6 current snap is 300 2021-03-24T22:04:19.541 INFO:tasks.rados.rados.0.smithi125.stdout:2609: seq_num 906 ranges {427142=421601,1400333=440643,2613702=450326} 2021-03-24T22:04:19.545 INFO:tasks.rados.rados.0.smithi125.stdout:2609: writing smithi12556470-6 from 427142 to 848743 tid 1 2021-03-24T22:04:19.552 INFO:tasks.rados.rados.0.smithi125.stdout:2609: writing smithi12556470-6 from 1400333 to 1840976 tid 2 2021-03-24T22:04:19.561 INFO:tasks.rados.rados.0.smithi125.stdout:2609: writing smithi12556470-6 from 2613702 to 3064028 tid 3 2021-03-24T22:04:19.562 INFO:tasks.rados.rados.0.smithi125.stdout:2608: finishing write tid 1 to smithi12556470-26 2021-03-24T22:04:19.562 INFO:tasks.rados.rados.0.smithi125.stdout:2608: finishing write tid 2 to smithi12556470-26 2021-03-24T22:04:19.563 INFO:tasks.rados.rados.0.smithi125.stdout:2608: finishing write tid 3 to smithi12556470-26 2021-03-24T22:04:19.563 INFO:tasks.rados.rados.0.smithi125.stdout:2608: finishing write tid 4 to smithi12556470-26 2021-03-24T22:04:19.563 INFO:tasks.rados.rados.0.smithi125.stdout:2608: finishing write tid 5 to smithi12556470-26 2021-03-24T22:04:19.564 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 26 v 831 (ObjNum 905 snap 300 seq_num 905) dirty exists 2021-03-24T22:04:19.564 INFO:tasks.rados.rados.0.smithi125.stdout:2608: left oid 26 (ObjNum 905 snap 300 seq_num 905) 2021-03-24T22:04:19.564 INFO:tasks.rados.rados.0.smithi125.stdout:2609: finishing write tid 1 to smithi12556470-6 2021-03-24T22:04:19.565 INFO:tasks.rados.rados.0.smithi125.stdout:2608: done (1 left) 2021-03-24T22:04:19.565 INFO:tasks.rados.rados.0.smithi125.stdout:2610: write (excl) oid 10 current snap is 300 2021-03-24T22:04:19.565 INFO:tasks.rados.rados.0.smithi125.stdout:2610: seq_num 907 ranges {443949=496850,1458815=584301,2641125=602824} 2021-03-24T22:04:19.567 INFO:tasks.rados.rados.0.smithi125.stdout:2610: writing smithi12556470-10 from 443949 to 940799 tid 1 2021-03-24T22:04:19.576 INFO:tasks.rados.rados.0.smithi125.stdout:2610: writing smithi12556470-10 from 1458815 to 2043116 tid 2 2021-03-24T22:04:19.585 INFO:tasks.rados.rados.0.smithi125.stdout:2610: writing smithi12556470-10 from 2641125 to 3243949 tid 3 2021-03-24T22:04:19.586 INFO:tasks.rados.rados.0.smithi125.stdout:2609: finishing write tid 2 to smithi12556470-6 2021-03-24T22:04:19.587 INFO:tasks.rados.rados.0.smithi125.stdout:2609: finishing write tid 3 to smithi12556470-6 2021-03-24T22:04:19.587 INFO:tasks.rados.rados.0.smithi125.stdout:2609: finishing write tid 4 to smithi12556470-6 2021-03-24T22:04:19.587 INFO:tasks.rados.rados.0.smithi125.stdout:2609: finishing write tid 5 to smithi12556470-6 2021-03-24T22:04:19.587 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 6 v 604 (ObjNum 906 snap 300 seq_num 906) dirty exists 2021-03-24T22:04:19.588 INFO:tasks.rados.rados.0.smithi125.stdout:2609: left oid 6 (ObjNum 906 snap 300 seq_num 906) 2021-03-24T22:04:19.588 INFO:tasks.rados.rados.0.smithi125.stdout:2610: finishing write tid 1 to smithi12556470-10 2021-03-24T22:04:19.588 INFO:tasks.rados.rados.0.smithi125.stdout:2609: done (1 left) 2021-03-24T22:04:19.588 INFO:tasks.rados.rados.0.smithi125.stdout:2611: read oid 9 snap 238 2021-03-24T22:04:19.589 INFO:tasks.rados.rados.0.smithi125.stdout:2611: expect (ObjNum 588 snap 177 seq_num 588) 2021-03-24T22:04:19.589 INFO:tasks.rados.rados.0.smithi125.stdout:2612: rollback oid 40 current snap is 300 2021-03-24T22:04:19.589 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 40 to 192 2021-03-24T22:04:19.589 INFO:tasks.rados.rados.0.smithi125.stdout:2613: write (excl) oid 35 current snap is 300 2021-03-24T22:04:19.589 INFO:tasks.rados.rados.0.smithi125.stdout:2613: seq_num 908 ranges {460756=572099,1517297=444312,2668548=472870} 2021-03-24T22:04:19.593 INFO:tasks.rados.rados.0.smithi125.stdout:2613: writing smithi12556470-35 from 460756 to 1032855 tid 1 2021-03-24T22:04:19.601 INFO:tasks.rados.rados.0.smithi125.stdout:2613: writing smithi12556470-35 from 1517297 to 1961609 tid 2 2021-03-24T22:04:19.607 INFO:tasks.rados.rados.0.smithi125.stdout:2613: writing smithi12556470-35 from 2668548 to 3141418 tid 3 2021-03-24T22:04:19.609 INFO:tasks.rados.rados.0.smithi125.stdout:2611: expect (ObjNum 588 snap 177 seq_num 588) 2021-03-24T22:04:19.612 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:19 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:04:19.613 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:19 smithi125 conmon[19936]: 24T22:04:18.447716+0000 mon.a (mon.0) 1818 : 2021-03-24T22:04:19.613 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:19 smithi125 conmon[19936]: cluster [DBG] osdmap e688: 8 total, 7 up, 8 in 2021-03-24T22:04:19.614 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:19.616 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:19 smithi125 conmon[24927]: cluster 2021-03-24T22:04:18.447716+0000 mon.a (mon.0) 1818 : cluster [DBG] osdmap e688: 8 total, 7 up, 8 in 2021-03-24T22:04:19.672 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:19.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:19.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:19.680 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.3 dump_historic_ops 2021-03-24T22:04:19.726 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:04:19.777 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:19.780 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:04:19.829 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:19.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:19.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:04:19.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:19 smithi142 conmon[20125]: cluster 2021-03-24T22:04:18 2021-03-24T22:04:19.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:19 smithi142 conmon[20125]: .447716+0000 mon.a (mon.0) 1818 : cluster [DBG] osdmap e688: 8 total, 7 up, 8 in 2021-03-24T22:04:19.940 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:19.993 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:19.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:19.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:04:19.998 INFO:tasks.rados.rados.0.smithi125.stdout:2612: finishing rollback tid 0 to smithi12556470-40 2021-03-24T22:04:19.998 INFO:tasks.rados.rados.0.smithi125.stdout:2610: finishing write tid 2 to smithi12556470-10 2021-03-24T22:04:19.999 INFO:tasks.rados.rados.0.smithi125.stdout:2610: finishing write tid 3 to smithi12556470-10 2021-03-24T22:04:19.999 INFO:tasks.rados.rados.0.smithi125.stdout:2611: done (3 left) 2021-03-24T22:04:19.999 INFO:tasks.rados.rados.0.smithi125.stdout:2614: write oid 2 current snap is 300 2021-03-24T22:04:19.999 INFO:tasks.rados.rados.0.smithi125.stdout:2614: seq_num 909 ranges {477563=647348,1859426=587970,2979618=297945} 2021-03-24T22:04:20.005 INFO:tasks.rados.rados.0.smithi125.stdout:2614: writing smithi12556470-2 from 477563 to 1124911 tid 1 2021-03-24T22:04:20.013 INFO:tasks.rados.rados.0.smithi125.stdout:2614: writing smithi12556470-2 from 1859426 to 2447396 tid 2 2021-03-24T22:04:20.018 INFO:tasks.rados.rados.0.smithi125.stdout:2614: writing smithi12556470-2 from 2979618 to 3277563 tid 3 2021-03-24T22:04:20.018 INFO:tasks.rados.rados.0.smithi125.stdout:2615: read oid 34 snap -1 2021-03-24T22:04:20.018 INFO:tasks.rados.rados.0.smithi125.stdout:2615: expect deleted 2021-03-24T22:04:20.019 INFO:tasks.rados.rados.0.smithi125.stdout:2616: read oid 45 snap 290 2021-03-24T22:04:20.019 INFO:tasks.rados.rados.0.smithi125.stdout:2616: expect (ObjNum 845 snap 277 seq_num 845) 2021-03-24T22:04:20.019 INFO:tasks.rados.rados.0.smithi125.stdout:2617: snap_remove snap 277 2021-03-24T22:04:20.099 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:20.150 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:20.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:20.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:20.223 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:04:20.259 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:20.309 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:20.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:20.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:20.362 INFO:tasks.rados.rados.0.smithi125.stdout:2617: done (6 left) 2021-03-24T22:04:20.363 INFO:tasks.rados.rados.0.smithi125.stdout:2618: read oid 31 snap -1 2021-03-24T22:04:20.363 INFO:tasks.rados.rados.0.smithi125.stdout:2618: expect deleted 2021-03-24T22:04:20.363 INFO:tasks.rados.rados.0.smithi125.stdout:2619: snap_create 2021-03-24T22:04:20.364 INFO:tasks.rados.rados.0.smithi125.stdout:2610: finishing write tid 4 to smithi12556470-10 2021-03-24T22:04:20.364 INFO:tasks.rados.rados.0.smithi125.stdout:2610: finishing write tid 5 to smithi12556470-10 2021-03-24T22:04:20.364 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 10 v 1032 (ObjNum 907 snap 300 seq_num 907) dirty exists 2021-03-24T22:04:20.364 INFO:tasks.rados.rados.0.smithi125.stdout:2610: left oid 10 (ObjNum 907 snap 300 seq_num 907) 2021-03-24T22:04:20.364 INFO:tasks.rados.rados.0.smithi125.stdout:2613: finishing write tid 1 to smithi12556470-35 2021-03-24T22:04:20.365 INFO:tasks.rados.rados.0.smithi125.stdout:2612: finishing rollback tid 1 to smithi12556470-40 2021-03-24T22:04:20.365 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 40 v 1568 (ObjNum 288 snap 0 seq_num 129) dirty dne 2021-03-24T22:04:20.365 INFO:tasks.rados.rados.0.smithi125.stdout:2613: finishing write tid 2 to smithi12556470-35 2021-03-24T22:04:20.365 INFO:tasks.rados.rados.0.smithi125.stdout:2613: finishing write tid 3 to smithi12556470-35 2021-03-24T22:04:20.366 INFO:tasks.rados.rados.0.smithi125.stdout:2613: finishing write tid 4 to smithi12556470-35 2021-03-24T22:04:20.366 INFO:tasks.rados.rados.0.smithi125.stdout:2613: finishing write tid 5 to smithi12556470-35 2021-03-24T22:04:20.366 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 35 v 231 (ObjNum 908 snap 300 seq_num 908) dirty exists 2021-03-24T22:04:20.366 INFO:tasks.rados.rados.0.smithi125.stdout:2613: left oid 35 (ObjNum 908 snap 300 seq_num 908) 2021-03-24T22:04:20.367 INFO:tasks.rados.rados.0.smithi125.stdout:2614: finishing write tid 1 to smithi12556470-2 2021-03-24T22:04:20.367 INFO:tasks.rados.rados.0.smithi125.stdout:2614: finishing write tid 2 to smithi12556470-2 2021-03-24T22:04:20.367 INFO:tasks.rados.rados.0.smithi125.stdout:2614: finishing write tid 3 to smithi12556470-2 2021-03-24T22:04:20.367 INFO:tasks.rados.rados.0.smithi125.stdout:2614: finishing write tid 4 to smithi12556470-2 2021-03-24T22:04:20.367 INFO:tasks.rados.rados.0.smithi125.stdout:2616: expect (ObjNum 845 snap 277 seq_num 845) 2021-03-24T22:04:20.414 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:20.491 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:20.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:20.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:04:20.550 INFO:tasks.rados.rados.0.smithi125.stdout:2614: finishing write tid 5 to smithi12556470-2 2021-03-24T22:04:20.551 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 2 v 986 (ObjNum 909 snap 300 seq_num 909) dirty exists 2021-03-24T22:04:20.551 INFO:tasks.rados.rados.0.smithi125.stdout:2614: left oid 2 (ObjNum 909 snap 300 seq_num 909) 2021-03-24T22:04:20.603 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:20.661 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:20.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:20.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:20.766 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:20.781 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:20 smithi142 conmon[20125]: cluster 2021-03-24T22 2021-03-24T22:04:20.781 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:20 smithi142 conmon[20125]: :04:18.520466+0000 osd.3 (osd.3) 138 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:04:20.781 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:20 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:04:20.782 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:20 smithi142 conmon[20125]: T22:04:18. 2021-03-24T22:04:20.782 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:20 smithi142 conmon[20125]: 523018+0000 osd.3 (osd. 2021-03-24T22:04:20.782 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:20 smithi142 conmon[20125]: 3) 139 : cluster [DBG] 2021-03-24T22:04:20.782 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:20 smithi142 conmon[20125]: purged_snaps scrub ok 2021-03-24T22:04:20.783 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:20 smithi142 conmon[20125]: cluster 2021-03-24T22:04:20.783 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:20 smithi142 conmon[20125]: 2021-03-24T 2021-03-24T22:04:20.783 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:20 smithi142 conmon[20125]: 22:04:18.950826 2021-03-24T22:04:20.783 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:20 smithi142 conmon[20125]: +0000 mgr.y (mgr.14144 2021-03-24T22:04:20.784 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:20 smithi142 conmon[20125]: ) 618 : cluster [DBG] 2021-03-24T22:04:20.784 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:20 smithi142 conmon[20125]: pgmap v1195: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 2 active+remapped+backfill_toofull, 105 active+clean; 579 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 150/1803 objects degraded (8.319%); 60/1803 objects misplaced (3.328%) 2021-03-24T22:04:20.784 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:20 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:04:20.784 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:20 smithi142 conmon[20125]: -24T22:04:19 2021-03-24T22:04:20.784 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:20 smithi142 conmon[20125]: .473924+0000 mon.a (mon 2021-03-24T22:04:20.785 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:20 smithi142 conmon[20125]: .0) 1819 : cluster 2021-03-24T22:04:20.785 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:20 smithi142 conmon[20125]: [DBG] osdmap e689: 8 total, 7 up, 8 in 2021-03-24T22:04:20.785 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:20 smithi142 conmon[20125]: cluster 2021-03-24T22:04:20.785 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:20 smithi142 conmon[20125]: 2021-03 2021-03-24T22:04:20.786 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:20 smithi142 conmon[20125]: -24T22:04:20 2021-03-24T22:04:20.786 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:20 smithi142 conmon[20125]: .352181+0000 mon.a (mon 2021-03-24T22:04:20.786 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:20 smithi142 conmon[20125]: .0) 1820 : cluster 2021-03-24T22:04:20.786 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:20 smithi142 conmon[20125]: [WRN] Health check update: Degraded data redundancy: 150/1803 objects degraded (8.319%), 20 pgs degraded (PG_DEGRADED) 2021-03-24T22:04:20.786 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:20 smithi142 conmon[20125]: cluster 2021-03-24T22:04:20.787 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:20 smithi142 conmon[20125]: 2021-03-24T 2021-03-24T22:04:20.787 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:20 smithi142 conmon[20125]: 22:04:20.358745 2021-03-24T22:04:20.787 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:20 smithi142 conmon[20125]: +0000 mon.a (mon.0 2021-03-24T22:04:20.787 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:20 smithi142 conmon[20125]: ) 1821 : cluster [DBG] 2021-03-24T22:04:20.788 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:20 smithi142 conmon[20125]: osdmap e690: 8 total, 7 up, 8 in 2021-03-24T22:04:20.819 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:20.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:20.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:04:20.833 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:20 smithi125 conmon[19936]: cluster 2021-03-24T22:04:18 2021-03-24T22:04:20.834 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:20 smithi125 conmon[19936]: .520466+0000 osd.3 (osd.3) 138 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:04:20.834 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:20 smithi125 conmon[19936]: cluster 2021-03-24T22:04:18.523018 2021-03-24T22:04:20.834 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:20 smithi125 conmon[19936]: +0000 osd.3 (osd.3) 139 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:04:20.834 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:20 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T22:04:20.835 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:20 smithi125 conmon[19936]: -24T22:04:18.950826+0000 2021-03-24T22:04:20.835 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:20 smithi125 conmon[19936]: mgr.y (mgr.14144) 618 : cluster 2021-03-24T22:04:20.835 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:20 smithi125 conmon[19936]: [DBG] pgmap v1195: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 2 active+remapped+backfill_toofull, 105 active+clean; 579 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 150/1803 objects degraded (8.319%); 60/1803 objects misplaced (3.328%) 2021-03-24T22:04:20.836 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:20 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:04:20.836 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:20 smithi125 conmon[19936]: 04:19.473924+0000 mon.a (mon. 2021-03-24T22:04:20.836 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:20 smithi125 conmon[19936]: 0) 1819 : cluster [DBG] osdmap e689: 8 total, 7 up, 8 in 2021-03-24T22:04:20.836 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:20 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:04:20.836 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:20 smithi125 conmon[19936]: 24T22:04:20.352181+0000 2021-03-24T22:04:20.837 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:20 smithi125 conmon[19936]: mon.a (mon.0) 1820 : cluster 2021-03-24T22:04:20.837 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:20 smithi125 conmon[19936]: [WRN] Health check update: Degraded data redundancy: 150/1803 objects degraded (8.319%), 20 pgs degraded (PG_DEGRADED) 2021-03-24T22:04:20.837 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:20 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:04:20.838 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:20 smithi125 conmon[19936]: 04:20.358745+0000 mon.a (mon. 2021-03-24T22:04:20.838 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:20 smithi125 conmon[19936]: 0) 1821 : cluster [DBG] osdmap e690: 8 total, 7 up, 8 in 2021-03-24T22:04:20.838 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:20 smithi125 conmon[24927]: cluster 2021-03-24T22:04:20.839 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:20 smithi125 conmon[24927]: 2021-03-24T22:04:20.839 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:20 smithi125 conmon[24927]: 2021 2021-03-24T22:04:20.839 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:20 smithi125 conmon[24927]: - 2021-03-24T22:04:20.839 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:20 smithi125 conmon[24927]: 03 2021-03-24T22:04:20.839 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:20 smithi125 conmon[24927]: - 2021-03-24T22:04:20.840 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:20 smithi125 conmon[24927]: 24 2021-03-24T22:04:20.840 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:20 smithi125 conmon[24927]: T 2021-03-24T22:04:20.840 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:20 smithi125 conmon[24927]: 22 2021-03-24T22:04:20.840 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:20 smithi125 conmon[24927]: : 2021-03-24T22:04:20.840 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:20 smithi125 conmon[24927]: 04 2021-03-24T22:04:20.841 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:20 smithi125 conmon[24927]: : 2021-03-24T22:04:20.841 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:20 smithi125 conmon[24927]: 18 2021-03-24T22:04:20.841 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:20 smithi125 conmon[24927]: . 2021-03-24T22:04:20.841 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:20 smithi125 conmon[24927]: 520466+0000 osd.3 ( 2021-03-24T22:04:20.842 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:20 smithi125 conmon[24927]: osd.3) 138 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:04:20.842 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:20 smithi125 conmon[24927]: cluster 2021-03-24T22:04:20.842 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:20 smithi125 conmon[24927]: 2021-03-24T22:04:20.843 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:20 smithi125 conmon[24927]: 2021-03-24T 2021-03-24T22:04:20.843 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:20 smithi125 conmon[24927]: 22:04:18.523018+0000 osd.3 (osd.3) 139 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:04:20.843 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:20 smithi125 conmon[24927]: cluster 2021-03-24T22:04:18.950826+0000 2021-03-24T22:04:20.844 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:20 smithi125 conmon[24927]: mgr.y (mgr.14144) 618 : cluster [DBG] pgmap v1195: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 2 active+remapped+backfill_toofull, 105 active+clean; 579 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 150/1803 objects degraded (8.319%); 60/1803 objects misplaced (3.328%) 2021-03-24T22:04:20.844 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:20 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:04:20.844 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:20 smithi125 conmon[24927]: 24T22:04:19.473924+0000 mon.a (mon.0) 1819 : cluster 2021-03-24T22:04:20.844 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:20 smithi125 conmon[24927]: [DBG] osdmap e689: 8 total, 7 up, 8 in 2021-03-24T22:04:20.844 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:20 smithi125 conmon[24927]: cluster 2021-03-24T22:04:20 2021-03-24T22:04:20.845 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:20 smithi125 conmon[24927]: .352181+0000 mon.a (mon.0) 1820 : cluster 2021-03-24T22:04:20.845 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:20 smithi125 conmon[24927]: [WRN] Health check update: Degraded data redundancy: 150/1803 objects degraded (8.319%), 20 pgs degraded (PG_DEGRADED) 2021-03-24T22:04:20.845 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:20 smithi125 conmon[24927]: cluster 2021-03-24T22:04:20. 2021-03-24T22:04:20.845 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:20 smithi125 conmon[24927]: 358745+0000 mon.a (mon.0) 1821 : cluster [DBG] 2021-03-24T22:04:20.846 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:20 smithi125 conmon[24927]: osdmap e690: 8 total, 7 up, 8 in 2021-03-24T22:04:20.925 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:20.981 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:20.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:20.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:21.087 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:21.147 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:21.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:21.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:21.254 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:21.316 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:21.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:21.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:21.368 INFO:tasks.rados.rados.0.smithi125.stdout:2610: done (7 left) 2021-03-24T22:04:21.368 INFO:tasks.rados.rados.0.smithi125.stdout:2612: done (6 left) 2021-03-24T22:04:21.369 INFO:tasks.rados.rados.0.smithi125.stdout:2613: done (5 left) 2021-03-24T22:04:21.369 INFO:tasks.rados.rados.0.smithi125.stdout:2614: done (4 left) 2021-03-24T22:04:21.370 INFO:tasks.rados.rados.0.smithi125.stdout:2615: done (3 left) 2021-03-24T22:04:21.370 INFO:tasks.rados.rados.0.smithi125.stdout:2616: done (2 left) 2021-03-24T22:04:21.371 INFO:tasks.rados.rados.0.smithi125.stdout:2618: done (1 left) 2021-03-24T22:04:21.371 INFO:tasks.rados.rados.0.smithi125.stdout:2619: done (0 left) 2021-03-24T22:04:21.372 INFO:tasks.rados.rados.0.smithi125.stdout:2620: copy_from oid 40 from oid 5 current snap is 301 2021-03-24T22:04:21.372 INFO:tasks.rados.rados.0.smithi125.stdout:2621: delete oid 37 current snap is 301 2021-03-24T22:04:21.385 INFO:teuthology.orchestra.run.smithi125.stderr:noscrub is unset 2021-03-24T22:04:21.395 INFO:tasks.rados.rados.0.smithi125.stdout:2621: done (1 left) 2021-03-24T22:04:21.395 INFO:tasks.rados.rados.0.smithi125.stdout:2622: snap_remove snap 298 2021-03-24T22:04:21.425 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:21.476 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:21.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:21.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:21.580 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:21.632 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:21.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:21.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:21.738 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:21.749 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[19936]: audit 2021-03-24T22:04:21.750 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[19936]: 2021-03-24T22:04:21.750 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[19936]: 2021 2021-03-24T22:04:21.751 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[19936]: -03-24T22:04:20.552087+0000 mon.c (mon.2) 103 : audit [INF] from='client.? v1:172.21.15.125:0/2796367216' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T22:04:21.751 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[19936]: audit 2021-03-24T22:04:20.552395+0000 mon.a (mon.0) 1822 : audit [INF] from='client.35651 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T22:04:21.751 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[19936]: cluster 2021-03-24T22:04:21.354749+0000 mon.a (mon.0) 1823 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:04:21.752 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[19936]: audit 2021-03-24T22:04:21.363194+0000 mon.a (mon.0) 1824 : audit [INF] from='client.35651 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T22:04:21.752 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[19936]: cluster 2021-03-24T22:04:21.363289+0000 mon.a (mon.0) 1825 : cluster [DBG] osdmap e691: 8 total, 7 up, 8 in 2021-03-24T22:04:21.753 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: audit 2021-03-24T22:04:21.754 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 2021-03-24T22:04:21.754 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 2021 2021-03-24T22:04:21.755 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: - 2021-03-24T22:04:21.755 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 03 2021-03-24T22:04:21.755 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: - 2021-03-24T22:04:21.756 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 24 2021-03-24T22:04:21.756 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: T 2021-03-24T22:04:21.756 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 22 2021-03-24T22:04:21.756 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: : 2021-03-24T22:04:21.757 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 04 2021-03-24T22:04:21.757 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: : 2021-03-24T22:04:21.757 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 20 2021-03-24T22:04:21.757 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: . 2021-03-24T22:04:21.757 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 552087 2021-03-24T22:04:21.758 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: +0000 2021-03-24T22:04:21.758 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 2021-03-24T22:04:21.758 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: mon.c 2021-03-24T22:04:21.758 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: ( 2021-03-24T22:04:21.759 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: mon 2021-03-24T22:04:21.759 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: . 2021-03-24T22:04:21.759 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 2 2021-03-24T22:04:21.759 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: ) 2021-03-24T22:04:21.760 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 103 2021-03-24T22:04:21.760 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: : 2021-03-24T22:04:21.760 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: audit 2021-03-24T22:04:21.761 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 2021-03-24T22:04:21.761 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: [INF] 2021-03-24T22:04:21.762 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 2021-03-24T22:04:21.762 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: from='client.? v1:172.21.15.125:0/2796367216' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T22:04:21.762 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 2021-03-24T22:04:21.763 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: audit 2021-03-24T22:04:21.763 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 2021-03-24T22:04:21.764 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 2021 2021-03-24T22:04:21.764 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: - 2021-03-24T22:04:21.764 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 03 2021-03-24T22:04:21.765 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: - 2021-03-24T22:04:21.765 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 24 2021-03-24T22:04:21.766 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: T 2021-03-24T22:04:21.766 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 22 2021-03-24T22:04:21.766 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: : 2021-03-24T22:04:21.767 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 04 2021-03-24T22:04:21.767 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: : 2021-03-24T22:04:21.768 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 20 2021-03-24T22:04:21.768 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: . 2021-03-24T22:04:21.769 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 552395 2021-03-24T22:04:21.769 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: +0000 2021-03-24T22:04:21.769 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 2021-03-24T22:04:21.770 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: mon.a 2021-03-24T22:04:21.770 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: ( 2021-03-24T22:04:21.771 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: mon 2021-03-24T22:04:21.771 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: . 2021-03-24T22:04:21.772 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 0 2021-03-24T22:04:21.772 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: ) 2021-03-24T22:04:21.773 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 1822 2021-03-24T22:04:21.773 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: : 2021-03-24T22:04:21.773 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: audit 2021-03-24T22:04:21.774 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 2021-03-24T22:04:21.774 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: [INF] 2021-03-24T22:04:21.775 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 2021-03-24T22:04:21.775 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: from='client.35651 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T22:04:21.776 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 2021-03-24T22:04:21.776 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: cluster 2021-03-24T22:04:21.776 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 2021-03-24T22:04:21.777 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 2021 2021-03-24T22:04:21.777 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: - 2021-03-24T22:04:21.778 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 03 2021-03-24T22:04:21.778 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: - 2021-03-24T22:04:21.779 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 24 2021-03-24T22:04:21.779 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: T 2021-03-24T22:04:21.779 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 22 2021-03-24T22:04:21.780 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: : 2021-03-24T22:04:21.780 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 04 2021-03-24T22:04:21.781 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: : 2021-03-24T22:04:21.781 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 21 2021-03-24T22:04:21.782 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: . 2021-03-24T22:04:21.782 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 354749 2021-03-24T22:04:21.783 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: +0000 2021-03-24T22:04:21.783 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 2021-03-24T22:04:21.783 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: mon.a 2021-03-24T22:04:21.784 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: ( 2021-03-24T22:04:21.784 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: mon 2021-03-24T22:04:21.784 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: . 2021-03-24T22:04:21.785 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 0 2021-03-24T22:04:21.785 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: ) 2021-03-24T22:04:21.785 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 1823 2021-03-24T22:04:21.785 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: : 2021-03-24T22:04:21.785 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: cluster 2021-03-24T22:04:21.786 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 2021-03-24T22:04:21.786 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: [WRN] 2021-03-24T22:04:21.786 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 2021-03-24T22:04:21.787 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:04:21.787 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 2021-03-24T22:04:21.788 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: audit 2021-03-24T22:04:21.788 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 2021-03-24T22:04:21.788 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 2021 2021-03-24T22:04:21.789 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: - 2021-03-24T22:04:21.789 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 03 2021-03-24T22:04:21.789 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: - 2021-03-24T22:04:21.789 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 24 2021-03-24T22:04:21.789 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: T 2021-03-24T22:04:21.790 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 22 2021-03-24T22:04:21.790 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: : 2021-03-24T22:04:21.790 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 04 2021-03-24T22:04:21.791 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: : 2021-03-24T22:04:21.791 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 21 2021-03-24T22:04:21.791 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: . 2021-03-24T22:04:21.791 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 363194 2021-03-24T22:04:21.791 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: +0000 2021-03-24T22:04:21.792 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 2021-03-24T22:04:21.792 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: mon.a 2021-03-24T22:04:21.792 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: ( 2021-03-24T22:04:21.792 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: mon 2021-03-24T22:04:21.793 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: . 2021-03-24T22:04:21.793 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 0 2021-03-24T22:04:21.793 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: ) 2021-03-24T22:04:21.793 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 1824 2021-03-24T22:04:21.793 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: : 2021-03-24T22:04:21.794 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: audit 2021-03-24T22:04:21.794 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 2021-03-24T22:04:21.794 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: [INF] 2021-03-24T22:04:21.794 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 2021-03-24T22:04:21.795 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: from='client.35651 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T22:04:21.795 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 2021-03-24T22:04:21.795 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: cluster 2021-03-24T22:04:21.795 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 2021-03-24T22:04:21.796 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 2021 2021-03-24T22:04:21.796 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: - 2021-03-24T22:04:21.796 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 03 2021-03-24T22:04:21.796 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: - 2021-03-24T22:04:21.796 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 24 2021-03-24T22:04:21.797 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: T 2021-03-24T22:04:21.797 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 22 2021-03-24T22:04:21.797 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: : 2021-03-24T22:04:21.797 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 04 2021-03-24T22:04:21.797 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: : 2021-03-24T22:04:21.798 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 21 2021-03-24T22:04:21.798 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: . 2021-03-24T22:04:21.798 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 363289 2021-03-24T22:04:21.798 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: +0000 2021-03-24T22:04:21.799 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 2021-03-24T22:04:21.799 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: mon.a 2021-03-24T22:04:21.799 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: ( 2021-03-24T22:04:21.799 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: mon 2021-03-24T22:04:21.799 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: . 2021-03-24T22:04:21.800 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 0 2021-03-24T22:04:21.800 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: ) 2021-03-24T22:04:21.800 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 1825 2021-03-24T22:04:21.800 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: : 2021-03-24T22:04:21.800 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: cluster 2021-03-24T22:04:21.801 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 2021-03-24T22:04:21.801 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: [DBG] 2021-03-24T22:04:21.801 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 2021-03-24T22:04:21.801 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: osdmap e691: 8 total, 7 up, 8 in 2021-03-24T22:04:21.801 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:21 smithi125 conmon[24927]: 2021-03-24T22:04:21.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:21.803 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:21.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:21.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:21 smithi142 conmon[20125]: audit 2021-03-24T22:04:20.552087+0000 mon.c (mon.2) 103 2021-03-24T22:04:21.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:21 smithi142 conmon[20125]: : audit [INF] from='client.? v1:172.21.15.125:0/2796367216' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T22:04:21.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:21 smithi142 conmon[20125]: audit 2021-03-24T22:04:20.552395+0000 mon.a (mon.0) 1822 2021-03-24T22:04:21.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:21 smithi142 conmon[20125]: : audit [INF] from='client.35651 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T22:04:21.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:21 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:04:21.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:21 smithi142 conmon[20125]: -03-24T22:04:21.354749+0000 mon.a (mon 2021-03-24T22:04:21.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:21 smithi142 conmon[20125]: .0) 1823 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:04:21.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:21 smithi142 conmon[20125]: audit 2021-03-24T22 2021-03-24T22:04:21.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:21 smithi142 conmon[20125]: :04:21.363194+0000 mon.a (mon. 2021-03-24T22:04:21.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:21 smithi142 conmon[20125]: 0) 1824 : audit [INF] from='client.35651 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T22:04:21.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:21 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:04:21.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:21 smithi142 conmon[20125]: T22:04:21.363289+0000 mon.a 2021-03-24T22:04:21.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:21 smithi142 conmon[20125]: (mon.0) 1825 : cluster [DBG] osdmap e691: 8 total, 7 up, 8 in 2021-03-24T22:04:21.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:21 smithi142 conmon[20125]: 2021-03-24T22:04:21.904 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:21.964 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:21.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:21.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:04:22.070 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:22.126 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:22.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:22.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:04:22.232 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:22.291 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:22.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:22.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:04:22.379 INFO:tasks.rados.rados.0.smithi125.stdout:2622: done (1 left) 2021-03-24T22:04:22.379 INFO:tasks.rados.rados.0.smithi125.stdout:2623: snap_create 2021-03-24T22:04:22.379 INFO:tasks.rados.rados.0.smithi125.stdout:2620: finishing copy_from to smithi12556470-40 2021-03-24T22:04:22.379 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 40 v 1017 (ObjNum 894 snap 296 seq_num 894) dirty exists 2021-03-24T22:04:22.380 INFO:tasks.rados.rados.0.smithi125.stdout:2620: finishing copy_from racing read to smithi12556470-40 2021-03-24T22:04:22.397 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:22.450 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:22.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:22.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:04:22.559 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:22.602 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:22.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:22.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:22.707 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:22.767 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:22.771 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:22 smithi125 conmon[19936]: cluster 2021-03-24T22:04:20.951740+0000 mgr.y 2021-03-24T22:04:22.771 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:22 smithi125 conmon[19936]: (mgr.14144) 619 : cluster [DBG] pgmap v1198: 168 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 40 active+undersized, 19 active+undersized+degraded, 1 active+remapped+backfill_toofull, 106 active+clean; 594 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 11 op/s; 152/1818 objects degraded (8.361%); 40/1818 objects misplaced (2.200%); 367 B/s, 4 objects/s recovering 2021-03-24T22:04:22.771 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:22 smithi125 conmon[19936]: cluster 2021-03-24T22:04:21.493222+0000 mon.a (mon.0) 1826 : 2021-03-24T22:04:22.772 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:22 smithi125 conmon[19936]: cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:04:22.772 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:22 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:04:22.772 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:22 smithi125 conmon[19936]: 03-24T22:04:21.723755+0000 osd.6 (osd.6) 2021-03-24T22:04:22.772 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:22 smithi125 conmon[19936]: 59 2021-03-24T22:04:22.772 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:22 smithi125 conmon[19936]: : cluster [DBG] 3.26 scrub ok 2021-03-24T22:04:22.773 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:22 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:04:22.773 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:22 smithi125 conmon[19936]: 03-24T22:04:22.338894+0000 2021-03-24T22:04:22.773 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:22 smithi125 conmon[19936]: osd.7 (osd.7) 110 : cluster [DBG] 2021-03-24T22:04:22.773 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:22 smithi125 conmon[19936]: 6.3 scrub ok 2021-03-24T22:04:22.774 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:22 smithi125 conmon[19936]: cluster 2021-03-24T22 2021-03-24T22:04:22.774 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:22 smithi125 conmon[19936]: :04:22.366166+0000 mon.a (mon. 2021-03-24T22:04:22.774 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:22 smithi125 conmon[19936]: 0) 1827 : cluster [DBG] osdmap e692: 8 total, 7 up, 8 in 2021-03-24T22:04:22.775 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:22 smithi125 conmon[24927]: cluster 2021-03-24T22:04:20.951740 2021-03-24T22:04:22.775 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:22 smithi125 conmon[24927]: +0000 mgr.y (mgr.14144) 619 : cluster [DBG] pgmap v1198: 168 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 40 active+undersized, 19 active+undersized+degraded, 1 active+remapped+backfill_toofull, 106 active+clean; 594 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 11 op/s; 152/1818 objects degraded (8.361%); 40/1818 objects misplaced (2.200%); 367 B/s, 4 objects/s recovering 2021-03-24T22:04:22.775 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:22 smithi125 conmon[24927]: cluster 2021-03-24T22:04:22.775 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:22 smithi125 conmon[24927]: 2021-03-24T22:04:21.493222+0000 mon.a (mon.0) 1826 2021-03-24T22:04:22.776 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:22 smithi125 conmon[24927]: : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:04:22.776 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:22 smithi125 conmon[24927]: cluster 2021-03-24T22:04 2021-03-24T22:04:22.777 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:22 smithi125 conmon[24927]: :21.723755+0000 osd.6 (osd.6) 2021-03-24T22:04:22.777 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:22 smithi125 conmon[24927]: 59 : cluster [DBG] 3.26 scrub ok 2021-03-24T22:04:22.777 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:22 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:04:22.778 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:22 smithi125 conmon[24927]: 04:22.338894+0000 osd.7 (osd.7 2021-03-24T22:04:22.778 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:22 smithi125 conmon[24927]: ) 110 : cluster [DBG] 6.3 scrub ok 2021-03-24T22:04:22.778 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:22 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:04:22.779 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:22 smithi125 conmon[24927]: 04:22.366166+0000 mon.a (mon.0) 2021-03-24T22:04:22.779 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:22 smithi125 conmon[24927]: 1827 : cluster [DBG] osdmap e692: 8 total, 7 up, 8 in 2021-03-24T22:04:22.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:22.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:22.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:22 smithi142 conmon[20125]: cluster 2021-03-24T22:04 2021-03-24T22:04:22.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:22 smithi142 conmon[20125]: :20.951740+0000 mgr.y (mgr.14144) 619 : cluster [DBG] pgmap v1198: 168 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 40 active+undersized, 19 active+undersized+degraded, 1 active+remapped+backfill_toofull, 106 active+clean; 594 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 11 op/s; 152/1818 objects degraded (8.361%); 40/1818 objects misplaced (2.200%); 367 B/s, 4 objects/s recovering 2021-03-24T22:04:22.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:22 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:04:22.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:22 smithi142 conmon[20125]: 03-24T22:04:21.493222+0000 mon.a (mon.0) 1826 : cluster [WRN] 2021-03-24T22:04:22.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:22 smithi142 conmon[20125]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:04:22.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:22 smithi142 conmon[20125]: cluster 2021-03-24T22:04:22.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:22 smithi142 conmon[20125]: 2021-03-24T22 2021-03-24T22:04:22.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:22 smithi142 conmon[20125]: :04:21.723755+0000 osd.6 (osd 2021-03-24T22:04:22.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:22 smithi142 conmon[20125]: .6) 59 : cluster [DBG] 3.26 scrub ok 2021-03-24T22:04:22.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:22 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:04:22.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:22 smithi142 conmon[20125]: -03-24T22:04:22.338894 2021-03-24T22:04:22.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:22 smithi142 conmon[20125]: +0000 osd.7 (osd.7) 110 : cluster 2021-03-24T22:04:22.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:22 smithi142 conmon[20125]: [DBG] 6.3 scrub ok 2021-03-24T22:04:22.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:22 smithi142 conmon[20125]: cluster 2021-03-24T22:04:22.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:22 smithi142 conmon[20125]: 2021-03-24T22:04:22.366166+0000 mon.a 2021-03-24T22:04:22.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:22 smithi142 conmon[20125]: (mon.0) 1827 : cluster [DBG] osdmap e692: 8 total, 7 up, 8 in 2021-03-24T22:04:22.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:22 smithi142 conmon[20125]: 2021-03-24T22:04:22.884 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:22.948 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:22.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:22.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:04:23.056 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:23.135 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:23.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:23.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:23.239 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:23.308 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:23.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:23.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:23.374 INFO:tasks.rados.rados.0.smithi125.stdout:2620: done (1 left) 2021-03-24T22:04:23.375 INFO:tasks.rados.rados.0.smithi125.stdout:2623: done (0 left) 2021-03-24T22:04:23.375 INFO:tasks.rados.rados.0.smithi125.stdout:2624: copy_from oid 20 from oid 37 current snap is 302 2021-03-24T22:04:23.375 INFO:tasks.rados.rados.0.smithi125.stdout:2625: rollback oid 25 current snap is 302 2021-03-24T22:04:23.376 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 25 to 273 2021-03-24T22:04:23.376 INFO:tasks.rados.rados.0.smithi125.stdout:2626: rollback oid 28 current snap is 302 2021-03-24T22:04:23.376 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 28 to 273 2021-03-24T22:04:23.376 INFO:tasks.rados.rados.0.smithi125.stdout:2627: read oid 14 snap -1 2021-03-24T22:04:23.376 INFO:tasks.rados.rados.0.smithi125.stdout:2627: expect deleted 2021-03-24T22:04:23.377 INFO:tasks.rados.rados.0.smithi125.stdout:2628: read oid 41 snap -1 2021-03-24T22:04:23.377 INFO:tasks.rados.rados.0.smithi125.stdout:2628: expect (ObjNum 791 snap 257 seq_num 791) 2021-03-24T22:04:23.377 INFO:tasks.rados.rados.0.smithi125.stdout:2629: snap_create 2021-03-24T22:04:23.383 INFO:tasks.rados.rados.0.smithi125.stdout:2625: finishing rollback tid 0 to smithi12556470-25 2021-03-24T22:04:23.385 INFO:tasks.rados.rados.0.smithi125.stdout:2625: finishing rollback tid 1 to smithi12556470-25 2021-03-24T22:04:23.385 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 25 v 808 (ObjNum 1953066355 snap 842099048 seq_num 875967797) dirty dne 2021-03-24T22:04:23.388 INFO:tasks.rados.rados.0.smithi125.stdout:2624: finishing copy_from to smithi12556470-20 2021-03-24T22:04:23.388 INFO:tasks.rados.rados.0.smithi125.stdout:2624: got expected ENOENT (src dne) 2021-03-24T22:04:23.398 INFO:tasks.rados.rados.0.smithi125.stdout:2628: expect (ObjNum 791 snap 257 seq_num 791) 2021-03-24T22:04:23.414 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:23.465 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:23.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:23.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:23.570 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:23.579 INFO:tasks.rados.rados.0.smithi125.stdout:2624: finishing copy_from racing read to smithi12556470-20 2021-03-24T22:04:23.579 INFO:tasks.rados.rados.0.smithi125.stdout:2626: finishing rollback tid 1 to smithi12556470-28 2021-03-24T22:04:23.579 INFO:tasks.rados.rados.0.smithi125.stdout:2626: finishing rollback tid 2 to smithi12556470-28 2021-03-24T22:04:23.580 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 28 v 1195 (ObjNum 718 snap 229 seq_num 718) dirty exists 2021-03-24T22:04:23.620 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:23.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:23.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:04:23.727 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:23.785 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:23.789 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[24927]: cluster 2021-03-24T22:04:23.790 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[24927]: 2021-03-24T22:04:23.790 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[24927]: 2021 2021-03-24T22:04:23.790 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[24927]: - 2021-03-24T22:04:23.790 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[24927]: 03-24T22:04:22.588880 2021-03-24T22:04:23.791 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[24927]: +0000 osd.1 (osd.1) 65 : cluster [DBG] 6.e scrub ok 2021-03-24T22:04:23.791 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T22:04:23.791 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[24927]: T22:04:22.661278+0000 osd.6 (osd.6) 60 : cluster [DBG] 2021-03-24T22:04:23.791 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[24927]: purged_snaps scrub starts 2021-03-24T22:04:23.791 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T22:04:23.792 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[24927]: T22:04:22.662466+0000 osd.6 (osd.6) 61 : cluster 2021-03-24T22:04:23.792 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[24927]: [DBG] purged_snaps scrub ok 2021-03-24T22:04:23.792 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T22:04:23.792 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[24927]: T22:04:23.369617+0000 mon.a (mon.0) 1828 : cluster 2021-03-24T22:04:23.792 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[24927]: [DBG] osdmap e693: 8 total, 7 up, 8 in 2021-03-24T22:04:23.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:23.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:23.795 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: cluster 2021-03-24T22:04:23.795 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 2021-03-24T22:04:23.795 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 2021 2021-03-24T22:04:23.795 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: - 2021-03-24T22:04:23.795 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 03 2021-03-24T22:04:23.796 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: - 2021-03-24T22:04:23.796 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 24 2021-03-24T22:04:23.796 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: T 2021-03-24T22:04:23.796 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 22 2021-03-24T22:04:23.797 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: : 2021-03-24T22:04:23.797 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 04 2021-03-24T22:04:23.797 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: : 2021-03-24T22:04:23.797 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 22 2021-03-24T22:04:23.797 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: . 2021-03-24T22:04:23.798 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 588880 2021-03-24T22:04:23.798 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: +0000 2021-03-24T22:04:23.798 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 2021-03-24T22:04:23.798 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: osd.1 2021-03-24T22:04:23.799 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: ( 2021-03-24T22:04:23.799 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: osd 2021-03-24T22:04:23.799 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: . 2021-03-24T22:04:23.799 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 1 2021-03-24T22:04:23.799 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: ) 2021-03-24T22:04:23.800 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 65 2021-03-24T22:04:23.800 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: : 2021-03-24T22:04:23.800 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: cluster 2021-03-24T22:04:23.800 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 2021-03-24T22:04:23.800 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: [DBG] 2021-03-24T22:04:23.801 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 2021-03-24T22:04:23.801 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 6.e scrub ok 2021-03-24T22:04:23.801 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 2021-03-24T22:04:23.801 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: cluster 2021-03-24T22:04:23.801 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 2021-03-24T22:04:23.802 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 2021 2021-03-24T22:04:23.802 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: - 2021-03-24T22:04:23.802 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 03 2021-03-24T22:04:23.802 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: - 2021-03-24T22:04:23.803 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 24 2021-03-24T22:04:23.803 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: T 2021-03-24T22:04:23.803 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 22 2021-03-24T22:04:23.803 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: : 2021-03-24T22:04:23.803 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 04 2021-03-24T22:04:23.804 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: : 2021-03-24T22:04:23.804 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 22 2021-03-24T22:04:23.804 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: . 2021-03-24T22:04:23.804 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 661278 2021-03-24T22:04:23.804 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: +0000 2021-03-24T22:04:23.805 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 2021-03-24T22:04:23.805 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: osd.6 2021-03-24T22:04:23.805 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: ( 2021-03-24T22:04:23.805 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: osd 2021-03-24T22:04:23.806 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: . 2021-03-24T22:04:23.806 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 6 2021-03-24T22:04:23.806 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: ) 2021-03-24T22:04:23.806 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 60 2021-03-24T22:04:23.806 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: : 2021-03-24T22:04:23.807 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: cluster 2021-03-24T22:04:23.807 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 2021-03-24T22:04:23.807 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: [DBG] 2021-03-24T22:04:23.807 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 2021-03-24T22:04:23.807 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: purged_snaps scrub starts 2021-03-24T22:04:23.808 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 2021-03-24T22:04:23.808 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: cluster 2021-03-24T22:04:23.808 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 2021-03-24T22:04:23.808 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 2021 2021-03-24T22:04:23.808 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: - 2021-03-24T22:04:23.809 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 03 2021-03-24T22:04:23.809 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: - 2021-03-24T22:04:23.809 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 24 2021-03-24T22:04:23.809 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: T 2021-03-24T22:04:23.810 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 22 2021-03-24T22:04:23.810 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: : 2021-03-24T22:04:23.810 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 04 2021-03-24T22:04:23.810 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: : 2021-03-24T22:04:23.810 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 22 2021-03-24T22:04:23.811 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: . 2021-03-24T22:04:23.811 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 662466 2021-03-24T22:04:23.811 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: +0000 2021-03-24T22:04:23.811 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 2021-03-24T22:04:23.811 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: osd.6 2021-03-24T22:04:23.812 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: ( 2021-03-24T22:04:23.812 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: osd 2021-03-24T22:04:23.812 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: . 2021-03-24T22:04:23.812 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 6 2021-03-24T22:04:23.813 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: ) 2021-03-24T22:04:23.813 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 61 2021-03-24T22:04:23.813 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: : 2021-03-24T22:04:23.813 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: cluster 2021-03-24T22:04:23.813 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 2021-03-24T22:04:23.814 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: [DBG] 2021-03-24T22:04:23.814 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 2021-03-24T22:04:23.814 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: purged_snaps scrub ok 2021-03-24T22:04:23.814 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 2021-03-24T22:04:23.814 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: cluster 2021-03-24T22:04:23.815 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 2021-03-24T22:04:23.815 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 2021 2021-03-24T22:04:23.815 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: - 2021-03-24T22:04:23.815 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 03 2021-03-24T22:04:23.816 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: - 2021-03-24T22:04:23.816 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 24 2021-03-24T22:04:23.816 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: T 2021-03-24T22:04:23.816 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 22 2021-03-24T22:04:23.816 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: : 2021-03-24T22:04:23.817 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 04 2021-03-24T22:04:23.817 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: : 2021-03-24T22:04:23.817 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 23 2021-03-24T22:04:23.817 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: . 2021-03-24T22:04:23.817 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 369617 2021-03-24T22:04:23.818 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: +0000 2021-03-24T22:04:23.818 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 2021-03-24T22:04:23.818 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: mon.a 2021-03-24T22:04:23.818 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: ( 2021-03-24T22:04:23.819 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: mon 2021-03-24T22:04:23.819 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: . 2021-03-24T22:04:23.819 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 0 2021-03-24T22:04:23.819 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: ) 2021-03-24T22:04:23.819 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 1828 2021-03-24T22:04:23.820 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: : 2021-03-24T22:04:23.820 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: cluster 2021-03-24T22:04:23.820 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 2021-03-24T22:04:23.820 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: [DBG] 2021-03-24T22:04:23.820 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 2021-03-24T22:04:23.821 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: osdmap e693: 8 total, 7 up, 8 in 2021-03-24T22:04:23.821 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:23 smithi125 conmon[19936]: 2021-03-24T22:04:23.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:23 smithi142 conmon[20125]: cluster 2021-03-24T22:04:22.588880+0000 osd.1 (osd.1) 65 : cluster 2021-03-24T22:04:23.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:23 smithi142 conmon[20125]: [DBG] 6.e scrub ok 2021-03-24T22:04:23.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:23 smithi142 conmon[20125]: cluster 2021-03-24T22:04:22.661278+0000 osd.6 (osd.6) 60 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:04:23.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:23 smithi142 conmon[20125]: cluster 2021-03-24T22:04:22.662466+0000 osd.6 (osd.6) 61 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:04:23.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:23 smithi142 conmon[20125]: 2021-03-24T22:04:23.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:23 smithi142 conmon[20125]: cluster 2021-03-24T22:04 2021-03-24T22:04:23.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:23 smithi142 conmon[20125]: :23.369617+0000 mon.a (mon.0) 1828 : cluster [DBG] osdmap e693: 8 total, 7 up, 8 in 2021-03-24T22:04:23.896 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:23.949 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:23.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:23.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:04:24.056 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:24.113 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:24.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:24.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:04:24.220 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:24.275 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:24.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:24.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:04:24.381 INFO:tasks.rados.rados.0.smithi125.stdout:2624: done (5 left) 2021-03-24T22:04:24.381 INFO:tasks.rados.rados.0.smithi125.stdout:2625: done (4 left) 2021-03-24T22:04:24.381 INFO:tasks.rados.rados.0.smithi125.stdout:2626: done (3 left) 2021-03-24T22:04:24.382 INFO:tasks.rados.rados.0.smithi125.stdout:2627: done (2 left) 2021-03-24T22:04:24.382 INFO:tasks.rados.rados.0.smithi125.stdout:2628: done (1 left) 2021-03-24T22:04:24.382 INFO:tasks.rados.rados.0.smithi125.stdout:2629: done (0 left) 2021-03-24T22:04:24.382 INFO:tasks.rados.rados.0.smithi125.stdout:2630: read oid 20 snap -1 2021-03-24T22:04:24.383 INFO:tasks.rados.rados.0.smithi125.stdout:2630: expect (ObjNum 873 snap 283 seq_num 873) 2021-03-24T22:04:24.383 INFO:tasks.rados.rados.0.smithi125.stdout:2631: delete oid 9 current snap is 303 2021-03-24T22:04:24.383 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:24.414 INFO:tasks.rados.rados.0.smithi125.stdout:2630: expect (ObjNum 873 snap 283 seq_num 873) 2021-03-24T22:04:24.497 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:24.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:24.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:24.544 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-24T22:04:24.548 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2021-03-24T22:04:24.602 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:24 smithi125 conmon[19936]: cluster 2021-03-24T22:04:22.952883+0000 mgr.y (mgr.14144) 620 : cluster [DBG] pgmap v1201: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 1 active+remapped+backfill_toofull, 106 active+clean; 599 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.0 MiB/s wr, 13 op/s; 152/1824 objects degraded (8.333%); 40/1824 objects misplaced (2.193%); 367 B/s, 4 objects/s recovering 2021-03-24T22:04:24.603 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:24 smithi125 conmon[19936]: cluster 2021-03-24T22:04:24.371750+0000 mon.a (mon.0) 1829 : cluster [DBG] osdmap e694: 8 total, 7 up, 8 in 2021-03-24T22:04:24.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:24 smithi125 conmon[24927]: cluster 2021-03-24T22:04:24.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:24 smithi125 conmon[24927]: 2021-03-24 2021-03-24T22:04:24.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:24 smithi125 conmon[24927]: T22:04:22.952883+0000 mgr.y (mgr.14144) 620 : cluster 2021-03-24T22:04:24.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:24 smithi125 conmon[24927]: [DBG] pgmap v1201: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 1 active+remapped+backfill_toofull, 106 active+clean; 599 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.0 MiB/s wr, 13 op/s; 152/1824 objects degraded (8.333%); 40/1824 objects misplaced (2.193%); 367 B/s, 4 objects/s recovering 2021-03-24T22:04:24.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:24 smithi125 conmon[24927]: cluster 2021-03-24T22:04:24.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:24 smithi125 conmon[24927]: 2021-03-24T22:04:24.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:24 smithi125 conmon[24927]: 2021 2021-03-24T22:04:24.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:24 smithi125 conmon[24927]: - 2021-03-24T22:04:24.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:24 smithi125 conmon[24927]: 03 2021-03-24T22:04:24.606 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:24 smithi125 conmon[24927]: - 2021-03-24T22:04:24.606 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:24 smithi125 conmon[24927]: 24 2021-03-24T22:04:24.606 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:24 smithi125 conmon[24927]: T 2021-03-24T22:04:24.606 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:24 smithi125 conmon[24927]: 22 2021-03-24T22:04:24.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:24 smithi125 conmon[24927]: : 2021-03-24T22:04:24.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:24 smithi125 conmon[24927]: 04 2021-03-24T22:04:24.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:24 smithi125 conmon[24927]: : 2021-03-24T22:04:24.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:24 smithi125 conmon[24927]: 24 2021-03-24T22:04:24.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:24 smithi125 conmon[24927]: . 2021-03-24T22:04:24.608 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:24 smithi125 conmon[24927]: 371750 2021-03-24T22:04:24.608 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:24 smithi125 conmon[24927]: +0000 2021-03-24T22:04:24.608 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:24 smithi125 conmon[24927]: 2021-03-24T22:04:24.609 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:24 smithi125 conmon[24927]: mon.a 2021-03-24T22:04:24.610 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:24 smithi125 conmon[24927]: ( 2021-03-24T22:04:24.610 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:24 smithi125 conmon[24927]: mon 2021-03-24T22:04:24.610 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:24 smithi125 conmon[24927]: . 2021-03-24T22:04:24.611 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:24 smithi125 conmon[24927]: 0 2021-03-24T22:04:24.611 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:24 smithi125 conmon[24927]: ) 2021-03-24T22:04:24.612 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:24 smithi125 conmon[24927]: 1829 2021-03-24T22:04:24.612 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:24 smithi125 conmon[24927]: : 2021-03-24T22:04:24.612 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:24 smithi125 conmon[24927]: cluster 2021-03-24T22:04:24.613 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:24 smithi125 conmon[24927]: 2021-03-24T22:04:24.613 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:24 smithi125 conmon[24927]: [DBG] 2021-03-24T22:04:24.613 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:24 smithi125 conmon[24927]: 2021-03-24T22:04:24.613 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:24 smithi125 conmon[24927]: osdmap e694: 8 total, 7 up, 8 in 2021-03-24T22:04:24.613 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:24 smithi125 conmon[24927]: 2021-03-24T22:04:24.614 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:24.624 INFO:tasks.rados.rados.0.smithi125.stdout:2630: done (1 left) 2021-03-24T22:04:24.625 INFO:tasks.rados.rados.0.smithi125.stdout:2631: done (0 left) 2021-03-24T22:04:24.625 INFO:tasks.rados.rados.0.smithi125.stdout:2632: write (excl) oid 48 current snap is 303 2021-03-24T22:04:24.625 INFO:tasks.rados.rados.0.smithi125.stdout:2632: seq_num 912 ranges {527984=473095,1634872=451650,2661887=634311} 2021-03-24T22:04:24.639 INFO:tasks.rados.rados.0.smithi125.stdout:2632: writing smithi12556470-48 from 527984 to 1001079 tid 1 2021-03-24T22:04:24.645 INFO:tasks.rados.rados.0.smithi125.stdout:2632: writing smithi12556470-48 from 1634872 to 2086522 tid 2 2021-03-24T22:04:24.652 INFO:tasks.rados.rados.0.smithi125.stdout:2632: writing smithi12556470-48 from 2661887 to 3296198 tid 3 2021-03-24T22:04:24.653 INFO:tasks.rados.rados.0.smithi125.stdout:2633: rollback oid 38 current snap is 303 2021-03-24T22:04:24.653 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 38 to 249 2021-03-24T22:04:24.653 INFO:tasks.rados.rados.0.smithi125.stdout:2632: finishing write tid 1 to smithi12556470-48 2021-03-24T22:04:24.654 INFO:tasks.rados.rados.0.smithi125.stdout:2634: delete oid 39 current snap is 303 2021-03-24T22:04:24.657 INFO:tasks.rados.rados.0.smithi125.stdout:2633: finishing rollback tid 0 to smithi12556470-38 2021-03-24T22:04:24.658 INFO:tasks.rados.rados.0.smithi125.stdout:2633: finishing rollback tid 1 to smithi12556470-38 2021-03-24T22:04:24.658 INFO:tasks.rados.rados.0.smithi125.stdout:2633: finishing rollback tid 2 to smithi12556470-38 2021-03-24T22:04:24.658 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 38 v 731 (ObjNum 733 snap 235 seq_num 733) dirty exists 2021-03-24T22:04:24.658 INFO:tasks.rados.rados.0.smithi125.stdout:2633: done (2 left) 2021-03-24T22:04:24.659 INFO:tasks.rados.rados.0.smithi125.stdout:2634: done (1 left) 2021-03-24T22:04:24.659 INFO:tasks.rados.rados.0.smithi125.stdout:2635: snap_remove snap 273 2021-03-24T22:04:24.673 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:24.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:24.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:04:24.779 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:24.792 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:24 smithi142 conmon[20125]: cluster 2021-03-24T22:04 2021-03-24T22:04:24.792 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:24 smithi142 conmon[20125]: :22.952883+0000 mgr.y (mgr.14144) 620 : cluster [DBG] pgmap v1201: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 1 active+remapped+backfill_toofull, 106 active+clean; 599 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.0 MiB/s wr, 13 op/s; 152/1824 objects degraded (8.333%); 40/1824 objects misplaced (2.193%); 367 B/s, 4 objects/s recovering 2021-03-24T22:04:24.792 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:24 smithi142 conmon[20125]: cluster 2021-03-24T22:04:24.371750+0000 2021-03-24T22:04:24.793 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:24 smithi142 conmon[20125]: mon.a (mon.0) 1829 : cluster [DBG] osdmap e694: 8 total, 7 up, 8 in 2021-03-24T22:04:24.830 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:24.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:24.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:24.936 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:24.997 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:25.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:25.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:25.104 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:25.157 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:25.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:25.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:25.263 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:25.318 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:25.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:25.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:04:25.424 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:25.480 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:25.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:25.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:04:25.531 INFO:tasks.rados.rados.0.smithi125.stdout:2635: done (1 left) 2021-03-24T22:04:25.531 INFO:tasks.rados.rados.0.smithi125.stdout:2636: delete oid 11 current snap is 303 2021-03-24T22:04:25.532 INFO:tasks.rados.rados.0.smithi125.stdout:2632: finishing write tid 2 to smithi12556470-48 2021-03-24T22:04:25.532 INFO:tasks.rados.rados.0.smithi125.stdout:2632: finishing write tid 3 to smithi12556470-48 2021-03-24T22:04:25.532 INFO:tasks.rados.rados.0.smithi125.stdout:2632: finishing write tid 4 to smithi12556470-48 2021-03-24T22:04:25.533 INFO:tasks.rados.rados.0.smithi125.stdout:2632: finishing write tid 5 to smithi12556470-48 2021-03-24T22:04:25.533 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 48 v 930 (ObjNum 912 snap 303 seq_num 912) dirty exists 2021-03-24T22:04:25.533 INFO:tasks.rados.rados.0.smithi125.stdout:2632: left oid 48 (ObjNum 912 snap 303 seq_num 912) 2021-03-24T22:04:25.538 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:25 smithi142 conmon[20125]: audit 2021-03-24T22:04:24. 2021-03-24T22:04:25.538 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:25 smithi142 conmon[20125]: 521979+0000 mgr.y (mgr.14144) 621 : audit [DBG] from='client.35657 v1:172.21.15.125:0/2229896375' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T22:04:25.538 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:25 smithi142 conmon[20125]: cluster 2021-03-24T22:04:25.354042 2021-03-24T22:04:25.539 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:25 smithi142 conmon[20125]: +0000 mon.a (mon.0) 1830 : cluster [WRN] Health check update: Degraded data redundancy: 152/1824 objects degraded (8.333%), 20 pgs degraded (PG_DEGRADED) 2021-03-24T22:04:25.562 INFO:tasks.rados.rados.0.smithi125.stdout:2632: done (1 left) 2021-03-24T22:04:25.563 INFO:tasks.rados.rados.0.smithi125.stdout:2636: done (0 left) 2021-03-24T22:04:25.563 INFO:tasks.rados.rados.0.smithi125.stdout:2637: copy_from oid 16 from oid 30 current snap is 303 2021-03-24T22:04:25.563 INFO:tasks.rados.rados.0.smithi125.stdout:2638: delete oid 22 current snap is 303 2021-03-24T22:04:25.575 INFO:tasks.rados.rados.0.smithi125.stdout:2638: done (1 left) 2021-03-24T22:04:25.575 INFO:tasks.rados.rados.0.smithi125.stdout:2639: snap_create 2021-03-24T22:04:25.587 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:25.594 INFO:tasks.rados.rados.0.smithi125.stdout:2637: finishing copy_from to smithi12556470-16 2021-03-24T22:04:25.595 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 16 v 828 (ObjNum 875 snap 283 seq_num 875) dirty exists 2021-03-24T22:04:25.595 INFO:tasks.rados.rados.0.smithi125.stdout:2637: finishing copy_from racing read to smithi12556470-16 2021-03-24T22:04:25.631 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:25.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:25.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:25.738 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:25.749 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:25 smithi125 conmon[19936]: audit 2021-03-24T22:04:24.521979+0000 mgr.y (mgr.14144) 621 : audit [DBG] from='client.35657 v1:172.21.15.125:0/2229896375' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T22:04:25.750 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:25 smithi125 conmon[19936]: cluster 2021-03-24T22:04:25.354042+0000 mon.a (mon.0) 1830 : cluster [WRN] Health check update: Degraded data redundancy: 152/1824 objects degraded (8.333%), 20 pgs degraded (PG_DEGRADED) 2021-03-24T22:04:25.750 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:25 smithi125 conmon[24927]: audit 2021-03-24T22:04:24.521979+0000 mgr.y 2021-03-24T22:04:25.750 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:25 smithi125 conmon[24927]: (mgr.14144) 621 : audit [DBG] from='client.35657 v1:172.21.15.125:0/2229896375' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T22:04:25.751 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:25 smithi125 conmon[24927]: cluster 2021-03-24T22:04:25.354042+0000 mon.a (mon.0) 1830 : cluster [WRN] Health check update: Degraded data redundancy: 152/1824 objects degraded (8.333%), 20 pgs degraded (PG_DEGRADED) 2021-03-24T22:04:25.751 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:25 smithi125 conmon[24927]: 2021-03-24T22:04:25.789 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:25.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:25.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:25.894 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:25.945 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:25.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:25.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:04:26.051 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:26.108 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:26.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:26.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:04:26.215 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:26.266 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:26.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:26.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:26.327 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-24T22:04:26.374 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:26.430 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:26.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:26.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:26.536 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:26.558 INFO:tasks.rados.rados.0.smithi125.stdout:2637: done (1 left) 2021-03-24T22:04:26.558 INFO:tasks.rados.rados.0.smithi125.stdout:2639: done (0 left) 2021-03-24T22:04:26.558 INFO:tasks.rados.rados.0.smithi125.stdout:2640: write oid 41 current snap is 304 2021-03-24T22:04:26.559 INFO:tasks.rados.rados.0.smithi125.stdout:2640: seq_num 916 ranges {595212=490444,1868800=458988,2887932=507280} 2021-03-24T22:04:26.564 INFO:tasks.rados.rados.0.smithi125.stdout:2640: writing smithi12556470-41 from 595212 to 1085656 tid 1 2021-03-24T22:04:26.576 INFO:tasks.rados.rados.0.smithi125.stdout:2640: writing smithi12556470-41 from 1868800 to 2327788 tid 2 2021-03-24T22:04:26.585 INFO:tasks.rados.rados.0.smithi125.stdout:2640: writing smithi12556470-41 from 2887932 to 3395212 tid 3 2021-03-24T22:04:26.585 INFO:tasks.rados.rados.0.smithi125.stdout:2640: finishing write tid 1 to smithi12556470-41 2021-03-24T22:04:26.585 INFO:tasks.rados.rados.0.smithi125.stdout:2640: finishing write tid 2 to smithi12556470-41 2021-03-24T22:04:26.586 INFO:tasks.rados.rados.0.smithi125.stdout:2641: snap_create 2021-03-24T22:04:26.590 INFO:tasks.rados.rados.0.smithi125.stdout:2640: finishing write tid 3 to smithi12556470-41 2021-03-24T22:04:26.590 INFO:tasks.rados.rados.0.smithi125.stdout:2640: finishing write tid 4 to smithi12556470-41 2021-03-24T22:04:26.591 INFO:tasks.rados.rados.0.smithi125.stdout:2640: finishing write tid 5 to smithi12556470-41 2021-03-24T22:04:26.591 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 41 v 1021 (ObjNum 916 snap 304 seq_num 916) dirty exists 2021-03-24T22:04:26.591 INFO:tasks.rados.rados.0.smithi125.stdout:2640: left oid 41 (ObjNum 916 snap 304 seq_num 916) 2021-03-24T22:04:26.592 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:26.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:26.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:04:26.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:26 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T22:04:26.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:26 smithi125 conmon[24927]: 22:04:24.954130+0000 mgr.y (mgr.14144) 622 : cluster [DBG] pgmap v1204: 168 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 40 active+undersized, 19 active+undersized+degraded, 106 active+clean; 609 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 3.5 MiB/s wr, 16 op/s; 155/1839 objects degraded (8.428%); 40/1839 objects misplaced (2.175%) 2021-03-24T22:04:26.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:26 smithi125 conmon[24927]: cluster 2021-03-24T22:04:26.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:26 smithi125 conmon[24927]: 2021-03-24T22:04:25.527240+0000 2021-03-24T22:04:26.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:26 smithi125 conmon[24927]: mon.a (mon.0) 1831 : cluster [DBG] osdmap e695: 8 total, 7 up, 8 in 2021-03-24T22:04:26.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:26 smithi125 conmon[24927]: 2021-03-24T22:04:26.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:26 smithi125 conmon[24927]: cluster 2021-03-24T22:04: 2021-03-24T22:04:26.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:26 smithi125 conmon[24927]: 25.596432+0000 osd.6 (osd.6) 62 : cluster [DBG] 2021-03-24T22:04:26.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:26 smithi125 conmon[24927]: 7.12 scrub ok 2021-03-24T22:04:26.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:26 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:04:26.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:26 smithi125 conmon[24927]: 03-24T22:04:25.611733+0000 osd.1 ( 2021-03-24T22:04:26.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:26 smithi125 conmon[24927]: osd.1) 66 : cluster [DBG] 3.19 scrub ok 2021-03-24T22:04:26.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:26 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:04:26.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:26 smithi125 conmon[24927]: -03-24T22:04:25.830845+0000 osd.2 2021-03-24T22:04:26.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:26 smithi125 conmon[24927]: (osd.2) 108 : cluster [DBG] 5.8 scrub ok 2021-03-24T22:04:26.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:26 smithi125 conmon[19936]: cluster 2021-03-24T22:04:24.954130 2021-03-24T22:04:26.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:26 smithi125 conmon[19936]: +0000 mgr.y (mgr.14144) 622 : cluster [DBG] pgmap v1204: 168 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 40 active+undersized, 19 active+undersized+degraded, 106 active+clean; 609 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 3.5 MiB/s wr, 16 op/s; 155/1839 objects degraded (8.428%); 40/1839 objects misplaced (2.175%) 2021-03-24T22:04:26.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:26 smithi125 conmon[19936]: cluster 2021-03-24T22:04:25.527240+0000 mon.a ( 2021-03-24T22:04:26.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:26 smithi125 conmon[19936]: mon.0) 1831 : cluster [DBG] osdmap e695: 8 total, 7 up, 8 in 2021-03-24T22:04:26.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:26 smithi125 conmon[19936]: cluster 2021-03-24T22:04:26.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:26 smithi125 conmon[19936]: 2021-03-24T22:04:25.596432+0000 osd.6 (osd.6) 2021-03-24T22:04:26.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:26 smithi125 conmon[19936]: 62 : cluster [DBG] 7.12 scrub ok 2021-03-24T22:04:26.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:26 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T22:04:26.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:26 smithi125 conmon[19936]: -24T22:04:25.611733+0000 osd.1 ( 2021-03-24T22:04:26.669 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:26 smithi125 conmon[19936]: osd.1) 66 : cluster [DBG] 3.19 scrub ok 2021-03-24T22:04:26.669 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:26 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:04:26.669 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:26 smithi125 conmon[19936]: -03-24T22:04:25.830845+0000 osd.2 2021-03-24T22:04:26.669 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:26 smithi125 conmon[19936]: (osd.2) 108 : cluster [DBG] 5.8 scrub ok 2021-03-24T22:04:26.699 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:26.748 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:26.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:26.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:26.819 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T22:04:26.819 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T22:04:26.820 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T22:04:26.820 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T22:04:26.820 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T22:04:26.841 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T22:04:26.856 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:26.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:26 smithi142 conmon[20125]: cluster 2021-03-24T22:04:24 2021-03-24T22:04:26.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:26 smithi142 conmon[20125]: .954130+0000 mgr.y (mgr.14144) 622 : cluster [DBG] pgmap v1204: 168 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 40 active+undersized, 19 active+undersized+degraded, 106 active+clean; 609 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 3.5 MiB/s wr, 16 op/s; 155/1839 objects degraded (8.428%); 40/1839 objects misplaced (2.175%) 2021-03-24T22:04:26.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:26 smithi142 conmon[20125]: cluster 2021-03-24T22:04:25.527240+0000 2021-03-24T22:04:26.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:26 smithi142 conmon[20125]: mon.a (mon.0) 1831 : cluster [DBG] osdmap e695: 8 total, 7 up, 8 in 2021-03-24T22:04:26.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:26 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:04:26.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:26 smithi142 conmon[20125]: -03-24T22:04:25.596432+0000 osd.6 (osd.6) 62 : cluster 2021-03-24T22:04:26.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:26 smithi142 conmon[20125]: [DBG] 7.12 scrub ok 2021-03-24T22:04:26.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:26 smithi142 conmon[20125]: cluster 2021-03-24T22:04:25.611733+0000 osd.1 (osd 2021-03-24T22:04:26.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:26 smithi142 conmon[20125]: .1) 66 : cluster [DBG] 3.19 scrub ok 2021-03-24T22:04:26.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:26 smithi142 conmon[20125]: cluster 2021-03-24T22:04:25.830845+0000 osd.2 ( 2021-03-24T22:04:26.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:26 smithi142 conmon[20125]: osd.2) 108 : cluster [DBG] 5.8 scrub ok 2021-03-24T22:04:26.871 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T22:04:26.872 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-24T22:04:26.872 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-24T22:04:26.872 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-24T22:04:26.872 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-24T22:04:26.873 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-24T22:04:26.873 INFO:teuthology.orchestra.run.smithi125.stderr:Error ENXIO: problem getting command descriptions from osd.5 2021-03-24T22:04:26.873 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-24T22:04:26.873 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-24T22:04:26.902 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:26.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:26.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:26.956 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":1203,"stamp":"2021-03-24T22:04:24.366759+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":634604281,"num_objects":611,"num_object_clones":240,"num_object_copies":1833,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":153,"num_objects_misplaced":40,"num_objects_unfound":0,"num_objects_dirty":611,"num_whiteouts":13,"num_read":4812,"num_read_kb":3264595,"num_write":3169,"num_write_kb":895569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4086,"num_bytes_recovered":1560251070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16421,"ondisk_log_size":16421,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":512,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1676144,"kb_used_data":1356912,"kb_used_omap":33,"kb_used_meta":319134,"kb_avail":748285072,"statfs":{"total":767960285184,"available":766243913728,"internally_reserved":0,"allocated":1389477888,"data_stored":1882834849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33936,"internal_metadata":326794096},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11497857,"num_objects":5,"num_object_clones":5,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":16,"num_read_kb":15176,"num_write":30,"num_write_kb":11663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.010796"},"pg_stats":[{"pgid":"3.29","version":"0'0","reported_seq":"345","reported_epoch":"692","state":"active+clean","last_fresh":"2021-03-24T22:04:22.396487+0000","last_change":"2021-03-24T22:03:45.428201+0000","last_active":"2021-03-24T22:04:22.396487+0000","last_peered":"2021-03-24T22:04:22.396487+0000","last_clean":"2021-03-24T22:04:22.396487+0000","last_became_active":"2021-03-24T22:03:45.427894+0000","last_became_peered":"2021-03-24T22:03:45.427894+0000","last_unstale":"2021-03-24T22:04:22.396487+0000","last_undegraded":"2021-03-24T22:04:22.396487+0000","last_fullsized":"2021-03-24T22:04:22.396487+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:55.592795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:55.592795+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":"452","reported_epoch":"692","state":"active+undersized","last_fresh":"2021-03-24T22:04:22.391904+0000","last_change":"2021-03-24T22:03:54.377067+0000","last_active":"2021-03-24T22:04:22.391904+0000","last_peered":"2021-03-24T22:04:22.391904+0000","last_clean":"2021-03-24T21:59:12.287458+0000","last_became_active":"2021-03-24T22:03:54.377067+0000","last_became_peered":"2021-03-24T22:03:54.377067+0000","last_unstale":"2021-03-24T22:04:22.391904+0000","last_undegraded":"2021-03-24T22:04:22.391904+0000","last_fullsized":"2021-03-24T22:03:54.358699+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:07.837673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:03:07.837673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"89'2","reported_seq":"360","reported_epoch":"692","state":"active+clean","last_fresh":"2021-03-24T22:04:22.391881+0000","last_change":"2021-03-24T21:59:14.374798+0000","last_active":"2021-03-24T22:04:22.391881+0000","last_peered":"2021-03-24T22:04:22.391881+0000","last_clean":"2021-03-24T22:04:22.391881+0000","last_became_active":"2021-03-24T21:59:14.374418+0000","last_became_peered":"2021-03-24T21:59:14.374418+0000","last_unstale":"2021-03-24T22:04:22.391881+0000","last_undegraded":"2021-03-24T22:04:22.391881+0000","last_fullsized":"2021-03-24T22:04:22.391881+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":6,"last_scrub":"89'2","last_scrub_stamp":"2021-03-24T21:57:58.491830+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2021-03-24T21:57:58.491830+0000","last_clean_scrub_stamp":"2021-03-24T21:57:58.491830+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":179,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":"698","reported_epoch":"692","state":"active+undersized","last_fresh":"2021-03-24T22:04:22.392792+0000","last_change":"2021-03-24T22:03:54.375571+0000","last_active":"2021-03-24T22:04:22.392792+0000","last_peered":"2021-03-24T22:04:22.392792+0000","last_clean":"2021-03-24T22:03:52.352816+0000","last_became_active":"2021-03-24T22:03:54.375571+0000","last_became_peered":"2021-03-24T22:03:54.375571+0000","last_unstale":"2021-03-24T22:04:22.392792+0000","last_undegraded":"2021-03-24T22:04:22.392792+0000","last_fullsized":"2021-03-24T22:03:54.360656+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:43.239983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:03:43.239983+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":"134","reported_epoch":"690","state":"active+undersized","last_fresh":"2021-03-24T22:04:20.358637+0000","last_change":"2021-03-24T22:03:54.377735+0000","last_active":"2021-03-24T22:04:20.358637+0000","last_peered":"2021-03-24T22:04:20.358637+0000","last_clean":"2021-03-24T22:03:52.503322+0000","last_became_active":"2021-03-24T22:03:54.377735+0000","last_became_peered":"2021-03-24T22:03:54.377735+0000","last_unstale":"2021-03-24T22:04:20.358637+0000","last_undegraded":"2021-03-24T22:04:20.358637+0000","last_fullsized":"2021-03-24T22:03:54.352388+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:38.304792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:38.304792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"61'1","reported_seq":"174","reported_epoch":"689","state":"active+undersized+degraded","last_fresh":"2021-03-24T22:04:20.010899+0000","last_change":"2021-03-24T22:03:54.377676+0000","last_active":"2021-03-24T22:04:20.010899+0000","last_peered":"2021-03-24T22:04:20.010899+0000","last_clean":"2021-03-24T22:03:52.356582+0000","last_became_active":"2021-03-24T22:03:54.377676+0000","last_became_peered":"2021-03-24T22:03:54.377676+0000","last_unstale":"2021-03-24T22:04:20.010899+0000","last_undegraded":"2021-03-24T22:03:54.360690+0000","last_fullsized":"2021-03-24T22:03:54.360615+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":6,"last_scrub":"61'1","last_scrub_stamp":"2021-03-24T21:57:12.046206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:12.046206+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":437,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3492,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[{"shards":"0,6","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":"674","reported_epoch":"692","state":"active+clean","last_fresh":"2021-03-24T22:04:22.396046+0000","last_change":"2021-03-24T22:02:57.502941+0000","last_active":"2021-03-24T22:04:22.396046+0000","last_peered":"2021-03-24T22:04:22.396046+0000","last_clean":"2021-03-24T22:04:22.396046+0000","last_became_active":"2021-03-24T21:59:14.370506+0000","last_became_peered":"2021-03-24T21:59:14.370506+0000","last_unstale":"2021-03-24T22:04:22.396046+0000","last_undegraded":"2021-03-24T22:04:22.396046+0000","last_fullsized":"2021-03-24T22:04:22.396046+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:57.502803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:02:57.502803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.23","version":"690'409","reported_seq":"2429","reported_epoch":"690","state":"active+clean","last_fresh":"2021-03-24T22:04:20.380306+0000","last_change":"2021-03-24T22:04:20.380571+0000","last_active":"2021-03-24T22:04:20.380306+0000","last_peered":"2021-03-24T22:04:20.380306+0000","last_clean":"2021-03-24T22:04:20.380306+0000","last_became_active":"2021-03-24T22:03:49.378704+0000","last_became_peered":"2021-03-24T22:03:49.378704+0000","last_unstale":"2021-03-24T22:04:20.380306+0000","last_undegraded":"2021-03-24T22:04:20.380306+0000","last_fullsized":"2021-03-24T22:04:20.380306+0000","mapping_epoch":658,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":659,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":409,"ondisk_log_size":409,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8766165,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":126,"num_read_kb":141370,"num_write":90,"num_write_kb":33080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":25982392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1cd","length":"1"},{"start":"1d4","length":"1"},{"start":"1f0","length":"1"},{"start":"1f3","length":"1"},{"start":"20e","length":"1"},{"start":"211","length":"1"},{"start":"214","length":"1"},{"start":"218","length":"2"},{"start":"21e","length":"1"},{"start":"220","length":"2"},{"start":"22f","length":"1"},{"start":"232","length":"1"}]},{"pgid":"3.26","version":"0'0","reported_seq":"173","reported_epoch":"689","state":"active+clean","last_fresh":"2021-03-24T22:04:20.013369+0000","last_change":"2021-03-24T22:01:57.311479+0000","last_active":"2021-03-24T22:04:20.013369+0000","last_peered":"2021-03-24T22:04:20.013369+0000","last_clean":"2021-03-24T22:04:20.013369+0000","last_became_active":"2021-03-24T22:01:57.311128+0000","last_became_peered":"2021-03-24T22:01:57.311128+0000","last_unstale":"2021-03-24T22:04:20.013369+0000","last_undegraded":"2021-03-24T22:04:20.013369+0000","last_fullsized":"2021-03-24T22:04:20.013369+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:42.879245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:59:42.879245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"685'273","reported_seq":"1534","reported_epoch":"688","state":"active+clean","last_fresh":"2021-03-24T22:04:18.451034+0000","last_change":"2021-03-24T22:04:18.451710+0000","last_active":"2021-03-24T22:04:18.451034+0000","last_peered":"2021-03-24T22:04:18.451034+0000","last_clean":"2021-03-24T22:04:18.451034+0000","last_became_active":"2021-03-24T22:03:51.364119+0000","last_became_peered":"2021-03-24T22:03:51.364119+0000","last_unstale":"2021-03-24T22:04:18.451034+0000","last_undegraded":"2021-03-24T22:04:18.451034+0000","last_fullsized":"2021-03-24T22:04:18.451034+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":273,"ondisk_log_size":273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19258922,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":35,"num_read_kb":14792,"num_write":62,"num_write_kb":18852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":47777940,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1cd","length":"1"},{"start":"1d4","length":"1"},{"start":"1f0","length":"1"},{"start":"1f3","length":"1"},{"start":"20e","length":"1"},{"start":"211","length":"1"},{"start":"218","length":"2"},{"start":"21e","length":"1"},{"start":"220","length":"2"},{"start":"22f","length":"1"},{"start":"232","length":"1"}]},{"pgid":"4.c","version":"544'37","reported_seq":"423","reported_epoch":"692","state":"active+clean","last_fresh":"2021-03-24T22:04:22.395230+0000","last_change":"2021-03-24T22:03:03.453859+0000","last_active":"2021-03-24T22:04:22.395230+0000","last_peered":"2021-03-24T22:04:22.395230+0000","last_clean":"2021-03-24T22:04:22.395230+0000","last_became_active":"2021-03-24T22:01:50.301325+0000","last_became_peered":"2021-03-24T22:01:50.301325+0000","last_unstale":"2021-03-24T22:04:22.395230+0000","last_undegraded":"2021-03-24T22:04:22.395230+0000","last_fullsized":"2021-03-24T22:04:22.395230+0000","mapping_epoch":539,"log_start":"69'15","ondisk_log_start":"69'15","created":62,"last_epoch_clean":540,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'37","last_scrub_stamp":"2021-03-24T22:03:03.453713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:03:03.453713+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":15,"num_object_clones":0,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":81,"num_read_kb":61,"num_write":51,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":1266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"657","reported_epoch":"692","state":"active+clean","last_fresh":"2021-03-24T22:04:22.393217+0000","last_change":"2021-03-24T21:59:17.264251+0000","last_active":"2021-03-24T22:04:22.393217+0000","last_peered":"2021-03-24T22:04:22.393217+0000","last_clean":"2021-03-24T22:04:22.393217+0000","last_became_active":"2021-03-24T21:51:45.375830+0000","last_became_peered":"2021-03-24T21:51:45.375830+0000","last_unstale":"2021-03-24T22:04:22.393217+0000","last_undegraded":"2021-03-24T22:04:22.393217+0000","last_fullsized":"2021-03-24T22:04:22.393217+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:17.264089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:59:17.264089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"664'13","reported_seq":"892","reported_epoch":"690","state":"active+undersized+degraded","last_fresh":"2021-03-24T22:04:20.358524+0000","last_change":"2021-03-24T22:03:54.370161+0000","last_active":"2021-03-24T22:04:20.358524+0000","last_peered":"2021-03-24T22:04:20.358524+0000","last_clean":"2021-03-24T22:03:52.504290+0000","last_became_active":"2021-03-24T22:03:54.370161+0000","last_became_peered":"2021-03-24T22:03:54.370161+0000","last_unstale":"2021-03-24T22:04:20.358524+0000","last_undegraded":"2021-03-24T22:03:54.361937+0000","last_fullsized":"2021-03-24T22:03:54.361805+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"548'12","last_scrub_stamp":"2021-03-24T22:02:18.728277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:02:18.728277+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[{"shards":"2,7","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"44","reported_epoch":"688","state":"active+clean","last_fresh":"2021-03-24T22:04:18.457104+0000","last_change":"2021-03-24T22:03:45.430592+0000","last_active":"2021-03-24T22:04:18.457104+0000","last_peered":"2021-03-24T22:04:18.457104+0000","last_clean":"2021-03-24T22:04:18.457104+0000","last_became_active":"2021-03-24T22:03:45.430245+0000","last_became_peered":"2021-03-24T22:03:45.430245+0000","last_unstale":"2021-03-24T22:04:18.457104+0000","last_undegraded":"2021-03-24T22:04:18.457104+0000","last_fullsized":"2021-03-24T22:04:18.457104+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:13.834281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:00:13.834281+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.f","version":"692'543","reported_seq":"2603","reported_epoch":"692","state":"active+clean","last_fresh":"2021-03-24T22:04:22.418110+0000","last_change":"2021-03-24T22:04:22.418110+0000","last_active":"2021-03-24T22:04:22.418110+0000","last_peered":"2021-03-24T22:04:22.418110+0000","last_clean":"2021-03-24T22:04:22.418110+0000","last_became_active":"2021-03-24T22:02:10.313297+0000","last_became_peered":"2021-03-24T22:02:10.313297+0000","last_unstale":"2021-03-24T22:04:22.418110+0000","last_undegraded":"2021-03-24T22:04:22.418110+0000","last_fullsized":"2021-03-24T22:04:22.418110+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'171","last_scrub_stamp":"2021-03-24T21:59:08.017874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:59:08.017874+0000","log_size":543,"ondisk_log_size":543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22256555,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":2,"num_read":113,"num_read_kb":129825,"num_write":54,"num_write_kb":17914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":46545574,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1cd","length":"1"},{"start":"1d4","length":"1"},{"start":"1f0","length":"1"},{"start":"1f3","length":"1"},{"start":"20e","length":"1"},{"start":"211","length":"1"},{"start":"214","length":"1"},{"start":"218","length":"2"},{"start":"21e","length":"1"},{"start":"220","length":"2"},{"start":"22f","length":"1"},{"start":"232","length":"1"},{"start":"239","length":"1"}]},{"pgid":"4.d","version":"544'41","reported_seq":"396","reported_epoch":"690","state":"active+clean","last_fresh":"2021-03-24T22:04:20.358213+0000","last_change":"2021-03-24T22:03:45.430602+0000","last_active":"2021-03-24T22:04:20.358213+0000","last_peered":"2021-03-24T22:04:20.358213+0000","last_clean":"2021-03-24T22:04:20.358213+0000","last_became_active":"2021-03-24T22:03:45.430156+0000","last_became_peered":"2021-03-24T22:03:45.430156+0000","last_unstale":"2021-03-24T22:04:20.358213+0000","last_undegraded":"2021-03-24T22:04:20.358213+0000","last_fullsized":"2021-03-24T22:04:20.358213+0000","mapping_epoch":654,"log_start":"69'21","ondisk_log_start":"69'21","created":62,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'41","last_scrub_stamp":"2021-03-24T22:03:03.723420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:03:03.723420+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1578,"num_objects":21,"num_object_clones":0,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":106,"num_read_kb":71,"num_write":60,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"187","reported_epoch":"689","state":"active+clean","last_fresh":"2021-03-24T22:04:20.011478+0000","last_change":"2021-03-24T22:03:45.420025+0000","last_active":"2021-03-24T22:04:20.011478+0000","last_peered":"2021-03-24T22:04:20.011478+0000","last_clean":"2021-03-24T22:04:20.011478+0000","last_became_active":"2021-03-24T22:03:45.419665+0000","last_became_peered":"2021-03-24T22:03:45.419665+0000","last_unstale":"2021-03-24T22:04:20.011478+0000","last_undegraded":"2021-03-24T22:04:20.011478+0000","last_fullsized":"2021-03-24T22:04:20.011478+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:30.890594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:55:01.169461+0000","last_clean_scrub_stamp":"2021-03-24T22:02:30.890594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"49","reported_epoch":"688","state":"active+clean","last_fresh":"2021-03-24T22:04:18.452839+0000","last_change":"2021-03-24T22:03:56.955470+0000","last_active":"2021-03-24T22:04:18.452839+0000","last_peered":"2021-03-24T22:04:18.452839+0000","last_clean":"2021-03-24T22:04:18.452839+0000","last_became_active":"2021-03-24T22:03:45.430779+0000","last_became_peered":"2021-03-24T22:03:45.430779+0000","last_unstale":"2021-03-24T22:04:18.452839+0000","last_undegraded":"2021-03-24T22:04:18.452839+0000","last_fullsized":"2021-03-24T22:04:18.452839+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:56.955183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:03:56.955183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"133","reported_epoch":"690","state":"active+clean","last_fresh":"2021-03-24T22:04:20.360071+0000","last_change":"2021-03-24T22:03:52.612272+0000","last_active":"2021-03-24T22:04:20.360071+0000","last_peered":"2021-03-24T22:04:20.360071+0000","last_clean":"2021-03-24T22:04:20.360071+0000","last_became_active":"2021-03-24T22:02:22.482380+0000","last_became_peered":"2021-03-24T22:02:22.482380+0000","last_unstale":"2021-03-24T22:04:20.360071+0000","last_undegraded":"2021-03-24T22:04:20.360071+0000","last_fullsized":"2021-03-24T22:04:20.360071+0000","mapping_epoch":571,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:52.612138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:03:52.612138+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"666'228","reported_seq":"2007","reported_epoch":"690","state":"active+undersized+degraded","last_fresh":"2021-03-24T22:04:20.359568+0000","last_change":"2021-03-24T22:03:54.402429+0000","last_active":"2021-03-24T22:04:20.359568+0000","last_peered":"2021-03-24T22:04:20.359568+0000","last_clean":"2021-03-24T22:03:52.504379+0000","last_became_active":"2021-03-24T22:03:54.402429+0000","last_became_peered":"2021-03-24T22:03:54.402429+0000","last_unstale":"2021-03-24T22:04:20.359568+0000","last_undegraded":"2021-03-24T22:03:54.354586+0000","last_fullsized":"2021-03-24T22:03:54.354145+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"546'150","last_scrub_stamp":"2021-03-24T22:01:56.662245+0000","last_deep_scrub":"546'150","last_deep_scrub_stamp":"2021-03-24T22:01:56.662245+0000","last_clean_scrub_stamp":"2021-03-24T22:01:56.662245+0000","log_size":228,"ondisk_log_size":228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":14,"stat_sum":{"num_bytes":12874921,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":54,"num_read_kb":39510,"num_write":37,"num_write_kb":10316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2421497,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":5}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"544'45","reported_seq":"556","reported_epoch":"689","state":"active+clean","last_fresh":"2021-03-24T22:04:20.013031+0000","last_change":"2021-03-24T22:03:56.694071+0000","last_active":"2021-03-24T22:04:20.013031+0000","last_peered":"2021-03-24T22:04:20.013031+0000","last_clean":"2021-03-24T22:04:20.013031+0000","last_became_active":"2021-03-24T22:03:53.368885+0000","last_became_peered":"2021-03-24T22:03:53.368885+0000","last_unstale":"2021-03-24T22:04:20.013031+0000","last_undegraded":"2021-03-24T22:04:20.013031+0000","last_fullsized":"2021-03-24T22:04:20.013031+0000","mapping_epoch":662,"log_start":"89'27","ondisk_log_start":"89'27","created":62,"last_epoch_clean":663,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'45","last_scrub_stamp":"2021-03-24T22:03:56.693900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:03:56.693900+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1672,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":92,"num_read_kb":60,"num_write":50,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":5016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"674","reported_epoch":"690","state":"active+clean","last_fresh":"2021-03-24T22:04:20.358888+0000","last_change":"2021-03-24T22:03:50.590203+0000","last_active":"2021-03-24T22:04:20.358888+0000","last_peered":"2021-03-24T22:04:20.358888+0000","last_clean":"2021-03-24T22:04:20.358888+0000","last_became_active":"2021-03-24T22:01:57.302297+0000","last_became_peered":"2021-03-24T22:01:57.302297+0000","last_unstale":"2021-03-24T22:04:20.358888+0000","last_undegraded":"2021-03-24T22:04:20.358888+0000","last_fullsized":"2021-03-24T22:04:20.358888+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:50.590127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:03:50.590127+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"170","reported_epoch":"689","state":"active+undersized","last_fresh":"2021-03-24T22:04:20.013434+0000","last_change":"2021-03-24T22:03:54.368487+0000","last_active":"2021-03-24T22:04:20.013434+0000","last_peered":"2021-03-24T22:04:20.013434+0000","last_clean":"2021-03-24T22:03:52.359158+0000","last_became_active":"2021-03-24T22:03:54.368487+0000","last_became_peered":"2021-03-24T22:03:54.368487+0000","last_unstale":"2021-03-24T22:04:20.013434+0000","last_undegraded":"2021-03-24T22:04:20.013434+0000","last_fullsized":"2021-03-24T22:03:54.362316+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:56.217032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:00:56.217032+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"714","reported_epoch":"692","state":"active+undersized","last_fresh":"2021-03-24T22:04:22.394899+0000","last_change":"2021-03-24T22:03:54.374824+0000","last_active":"2021-03-24T22:04:22.394899+0000","last_peered":"2021-03-24T22:04:22.394899+0000","last_clean":"2021-03-24T22:03:52.354086+0000","last_became_active":"2021-03-24T22:03:54.374824+0000","last_became_peered":"2021-03-24T22:03:54.374824+0000","last_unstale":"2021-03-24T22:04:22.394899+0000","last_undegraded":"2021-03-24T22:04:22.394899+0000","last_fullsized":"2021-03-24T22:03:54.363392+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:25.263840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:00:25.263840+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"690'628","reported_seq":"2754","reported_epoch":"690","state":"active+clean","last_fresh":"2021-03-24T22:04:20.388048+0000","last_change":"2021-03-24T22:04:20.388457+0000","last_active":"2021-03-24T22:04:20.388048+0000","last_peered":"2021-03-24T22:04:20.388048+0000","last_clean":"2021-03-24T22:04:20.388048+0000","last_became_active":"2021-03-24T22:03:53.379184+0000","last_became_peered":"2021-03-24T22:03:53.379184+0000","last_unstale":"2021-03-24T22:04:20.388048+0000","last_undegraded":"2021-03-24T22:04:20.388048+0000","last_fullsized":"2021-03-24T22:04:20.388048+0000","mapping_epoch":662,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":663,"parent":"0.0","parent_split_bits":0,"last_scrub":"622'539","last_scrub_stamp":"2021-03-24T22:03:12.756606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:03:12.756606+0000","log_size":628,"ondisk_log_size":628,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20984431,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":82,"num_read_kb":107313,"num_write":39,"num_write_kb":19147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":179,"num_bytes_recovered":70071133,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1cd","length":"1"},{"start":"1d4","length":"1"},{"start":"1f0","length":"1"},{"start":"1f3","length":"1"},{"start":"20e","length":"1"},{"start":"211","length":"1"},{"start":"214","length":"1"},{"start":"218","length":"2"},{"start":"21e","length":"1"},{"start":"220","length":"2"},{"start":"22f","length":"1"},{"start":"232","length":"1"}]},{"pgid":"4.f","version":"544'46","reported_seq":"873","reported_epoch":"688","state":"active+clean","last_fresh":"2021-03-24T22:04:18.469295+0000","last_change":"2021-03-24T22:04:18.462896+0000","last_active":"2021-03-24T22:04:18.469295+0000","last_peered":"2021-03-24T22:04:18.469295+0000","last_clean":"2021-03-24T22:04:18.469295+0000","last_became_active":"2021-03-24T22:04:18.462489+0000","last_became_peered":"2021-03-24T22:04:18.462489+0000","last_unstale":"2021-03-24T22:04:18.469295+0000","last_undegraded":"2021-03-24T22:04:18.469295+0000","last_fullsized":"2021-03-24T22:04:18.469295+0000","mapping_epoch":687,"log_start":"274'26","ondisk_log_start":"274'26","created":62,"last_epoch_clean":688,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'26","last_scrub_stamp":"2021-03-24T22:01:53.692294+0000","last_deep_scrub":"274'26","last_deep_scrub_stamp":"2021-03-24T21:57:24.499090+0000","last_clean_scrub_stamp":"2021-03-24T22:01:53.692294+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1472,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":97,"num_read_kb":65,"num_write":56,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":8410,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":"739","reported_epoch":"692","state":"active+clean","last_fresh":"2021-03-24T22:04:22.391278+0000","last_change":"2021-03-24T22:03:45.428200+0000","last_active":"2021-03-24T22:04:22.391278+0000","last_peered":"2021-03-24T22:04:22.391278+0000","last_clean":"2021-03-24T22:04:22.391278+0000","last_became_active":"2021-03-24T22:03:45.427907+0000","last_became_peered":"2021-03-24T22:03:45.427907+0000","last_unstale":"2021-03-24T22:04:22.391278+0000","last_undegraded":"2021-03-24T22:04:22.391278+0000","last_fullsized":"2021-03-24T22:04:22.391278+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:39.439170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:39.439170+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"664'6","reported_seq":"285","reported_epoch":"692","state":"active+undersized+degraded","last_fresh":"2021-03-24T22:04:22.394802+0000","last_change":"2021-03-24T22:03:54.400861+0000","last_active":"2021-03-24T22:04:22.394802+0000","last_peered":"2021-03-24T22:04:22.394802+0000","last_clean":"2021-03-24T22:03:52.359763+0000","last_became_active":"2021-03-24T22:03:54.400861+0000","last_became_peered":"2021-03-24T22:03:54.400861+0000","last_unstale":"2021-03-24T22:04:22.394802+0000","last_undegraded":"2021-03-24T22:03:54.371776+0000","last_fullsized":"2021-03-24T22:03:54.371477+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'4","last_scrub_stamp":"2021-03-24T22:00:14.879311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:00:14.879311+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"574","reported_epoch":"688","state":"active+undersized","last_fresh":"2021-03-24T22:04:18.454922+0000","last_change":"2021-03-24T22:03:54.398143+0000","last_active":"2021-03-24T22:04:18.454922+0000","last_peered":"2021-03-24T22:04:18.454922+0000","last_clean":"2021-03-24T22:03:43.466357+0000","last_became_active":"2021-03-24T22:03:54.398143+0000","last_became_peered":"2021-03-24T22:03:54.398143+0000","last_unstale":"2021-03-24T22:04:18.454922+0000","last_undegraded":"2021-03-24T22:04:18.454922+0000","last_fullsized":"2021-03-24T22:03:54.357356+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:55.950428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:02:55.950428+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"690'990","reported_seq":"3300","reported_epoch":"690","state":"active+clean","last_fresh":"2021-03-24T22:04:20.396022+0000","last_change":"2021-03-24T22:04:20.396457+0000","last_active":"2021-03-24T22:04:20.396022+0000","last_peered":"2021-03-24T22:04:20.396022+0000","last_clean":"2021-03-24T22:04:20.396022+0000","last_became_active":"2021-03-24T22:02:06.309737+0000","last_became_peered":"2021-03-24T22:02:06.309737+0000","last_unstale":"2021-03-24T22:04:20.396022+0000","last_undegraded":"2021-03-24T22:04:20.396022+0000","last_fullsized":"2021-03-24T22:04:20.396022+0000","mapping_epoch":555,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":556,"parent":"0.0","parent_split_bits":0,"last_scrub":"659'936","last_scrub_stamp":"2021-03-24T22:03:53.662470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:03:53.662470+0000","log_size":990,"ondisk_log_size":990,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36557052,"num_objects":16,"num_object_clones":12,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":1,"num_read":151,"num_read_kb":164092,"num_write":109,"num_write_kb":43470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":296,"num_bytes_recovered":189413266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1cd","length":"1"},{"start":"1d4","length":"1"},{"start":"1f0","length":"1"},{"start":"1f3","length":"1"},{"start":"20e","length":"1"},{"start":"211","length":"1"},{"start":"214","length":"1"},{"start":"218","length":"2"},{"start":"21e","length":"1"},{"start":"220","length":"2"},{"start":"22f","length":"1"},{"start":"232","length":"1"}]},{"pgid":"5.1","version":"0'0","reported_seq":"631","reported_epoch":"692","state":"active+clean","last_fresh":"2021-03-24T22:04:22.392606+0000","last_change":"2021-03-24T22:03:00.472146+0000","last_active":"2021-03-24T22:04:22.392606+0000","last_peered":"2021-03-24T22:04:22.392606+0000","last_clean":"2021-03-24T22:04:22.392606+0000","last_became_active":"2021-03-24T22:01:57.305430+0000","last_became_peered":"2021-03-24T22:01:57.305430+0000","last_unstale":"2021-03-24T22:04:22.392606+0000","last_undegraded":"2021-03-24T22:04:22.392606+0000","last_fullsized":"2021-03-24T22:04:22.392606+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:00.471949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:25.220436+0000","last_clean_scrub_stamp":"2021-03-24T22:03:00.471949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"544'34","reported_seq":"859","reported_epoch":"692","state":"active+clean","last_fresh":"2021-03-24T22:04:22.391154+0000","last_change":"2021-03-24T22:01:58.356469+0000","last_active":"2021-03-24T22:04:22.391154+0000","last_peered":"2021-03-24T22:04:22.391154+0000","last_clean":"2021-03-24T22:04:22.391154+0000","last_became_active":"2021-03-24T22:01:57.309916+0000","last_became_peered":"2021-03-24T22:01:57.309916+0000","last_unstale":"2021-03-24T22:04:22.391154+0000","last_undegraded":"2021-03-24T22:04:22.391154+0000","last_fullsized":"2021-03-24T22:04:22.391154+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'22","last_scrub_stamp":"2021-03-24T21:58:56.473510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:58:56.473510+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":64,"num_read_kb":44,"num_write":36,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":3384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"708","reported_epoch":"692","state":"active+clean","last_fresh":"2021-03-24T22:04:22.395913+0000","last_change":"2021-03-24T22:01:57.314681+0000","last_active":"2021-03-24T22:04:22.395913+0000","last_peered":"2021-03-24T22:04:22.395913+0000","last_clean":"2021-03-24T22:04:22.395913+0000","last_became_active":"2021-03-24T22:01:57.314345+0000","last_became_peered":"2021-03-24T22:01:57.314345+0000","last_unstale":"2021-03-24T22:04:22.395913+0000","last_undegraded":"2021-03-24T22:04:22.395913+0000","last_fullsized":"2021-03-24T22:04:22.395913+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:54.381668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:54.381668+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"230","reported_epoch":"692","state":"active+clean","last_fresh":"2021-03-24T22:04:22.397241+0000","last_change":"2021-03-24T22:03:45.222962+0000","last_active":"2021-03-24T22:04:22.397241+0000","last_peered":"2021-03-24T22:04:22.397241+0000","last_clean":"2021-03-24T22:04:22.397241+0000","last_became_active":"2021-03-24T22:00:49.305612+0000","last_became_peered":"2021-03-24T22:00:49.305612+0000","last_unstale":"2021-03-24T22:04:22.397241+0000","last_undegraded":"2021-03-24T22:04:22.397241+0000","last_fullsized":"2021-03-24T22:04:22.397241+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:45.222743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:03:45.222743+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"690'703","reported_seq":"2536","reported_epoch":"692","state":"active+clean","last_fresh":"2021-03-24T22:04:22.397545+0000","last_change":"2021-03-24T22:04:22.397545+0000","last_active":"2021-03-24T22:04:22.397545+0000","last_peered":"2021-03-24T22:04:22.397545+0000","last_clean":"2021-03-24T22:04:22.397545+0000","last_became_active":"2021-03-24T21:57:26.169302+0000","last_became_peered":"2021-03-24T21:57:26.169302+0000","last_unstale":"2021-03-24T22:04:22.397545+0000","last_undegraded":"2021-03-24T22:04:22.397545+0000","last_fullsized":"2021-03-24T22:04:22.397545+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'330","last_scrub_stamp":"2021-03-24T21:59:07.373692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:59:07.373692+0000","log_size":703,"ondisk_log_size":703,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21569950,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":54,"num_read_kb":42247,"num_write":58,"num_write_kb":27404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1cd","length":"1"},{"start":"1d4","length":"1"},{"start":"1f0","length":"1"},{"start":"1f3","length":"1"},{"start":"20e","length":"1"},{"start":"211","length":"1"},{"start":"214","length":"1"},{"start":"218","length":"2"},{"start":"21e","length":"1"},{"start":"220","length":"2"},{"start":"22f","length":"1"},{"start":"232","length":"1"},{"start":"239","length":"1"}]},{"pgid":"5.0","version":"658'6","reported_seq":"857","reported_epoch":"692","state":"active+clean","last_fresh":"2021-03-24T22:04:22.390735+0000","last_change":"2021-03-24T22:03:45.699782+0000","last_active":"2021-03-24T22:04:22.390735+0000","last_peered":"2021-03-24T22:04:22.390735+0000","last_clean":"2021-03-24T22:04:22.390735+0000","last_became_active":"2021-03-24T22:03:45.432883+0000","last_became_peered":"2021-03-24T22:03:45.432883+0000","last_unstale":"2021-03-24T22:04:22.390735+0000","last_undegraded":"2021-03-24T22:04:22.390735+0000","last_fullsized":"2021-03-24T22:04:22.390735+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'3","last_scrub_stamp":"2021-03-24T21:59:46.375145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:59:46.375145+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"544'34","reported_seq":"216","reported_epoch":"689","state":"active+undersized+degraded","last_fresh":"2021-03-24T22:04:20.010919+0000","last_change":"2021-03-24T22:03:54.371960+0000","last_active":"2021-03-24T22:04:20.010919+0000","last_peered":"2021-03-24T22:04:20.010919+0000","last_clean":"2021-03-24T22:01:55.032684+0000","last_became_active":"2021-03-24T22:03:54.371960+0000","last_became_peered":"2021-03-24T22:03:54.371960+0000","last_unstale":"2021-03-24T22:04:20.010919+0000","last_undegraded":"2021-03-24T22:03:54.358132+0000","last_fullsized":"2021-03-24T22:03:54.357976+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'34","last_scrub_stamp":"2021-03-24T22:02:32.116075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:02:32.116075+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":58,"num_read_kb":39,"num_write":34,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":940,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[{"shards":"4,6","objects":13}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"714","reported_epoch":"692","state":"active+clean","last_fresh":"2021-03-24T22:04:22.394172+0000","last_change":"2021-03-24T22:03:45.431590+0000","last_active":"2021-03-24T22:04:22.394172+0000","last_peered":"2021-03-24T22:04:22.394172+0000","last_clean":"2021-03-24T22:04:22.394172+0000","last_became_active":"2021-03-24T22:03:45.431240+0000","last_became_peered":"2021-03-24T22:03:45.431240+0000","last_unstale":"2021-03-24T22:04:22.394172+0000","last_undegraded":"2021-03-24T22:04:22.394172+0000","last_fullsized":"2021-03-24T22:04:22.394172+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:28.165408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:28.165408+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"151","reported_epoch":"687","state":"active+clean","last_fresh":"2021-03-24T22:04:17.456717+0000","last_change":"2021-03-24T22:01:57.318244+0000","last_active":"2021-03-24T22:04:17.456717+0000","last_peered":"2021-03-24T22:04:17.456717+0000","last_clean":"2021-03-24T22:04:17.456717+0000","last_became_active":"2021-03-24T22:01:57.316610+0000","last_became_peered":"2021-03-24T22:01:57.316610+0000","last_unstale":"2021-03-24T22:04:17.456717+0000","last_undegraded":"2021-03-24T22:04:17.456717+0000","last_fullsized":"2021-03-24T22:04:17.456717+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:54.826534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:58:54.826534+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"272'64","reported_seq":"1913","reported_epoch":"692","state":"active+clean","last_fresh":"2021-03-24T22:04:22.397635+0000","last_change":"2021-03-24T22:04:22.397635+0000","last_active":"2021-03-24T22:04:22.397635+0000","last_peered":"2021-03-24T22:04:22.397635+0000","last_clean":"2021-03-24T22:04:22.397635+0000","last_became_active":"2021-03-24T22:03:45.428845+0000","last_became_peered":"2021-03-24T22:03:45.428845+0000","last_unstale":"2021-03-24T22:04:22.397635+0000","last_undegraded":"2021-03-24T22:04:22.397635+0000","last_fullsized":"2021-03-24T22:04:22.397635+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'64","last_scrub_stamp":"2021-03-24T21:59:08.406188+0000","last_deep_scrub":"272'64","last_deep_scrub_stamp":"2021-03-24T21:59:08.406188+0000","last_clean_scrub_stamp":"2021-03-24T21:59:08.406188+0000","log_size":64,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1cd","length":"1"},{"start":"1d4","length":"1"},{"start":"1f0","length":"1"},{"start":"1f3","length":"1"},{"start":"20e","length":"1"},{"start":"211","length":"1"},{"start":"214","length":"1"},{"start":"218","length":"2"},{"start":"21e","length":"1"},{"start":"220","length":"2"},{"start":"22f","length":"1"},{"start":"232","length":"1"},{"start":"239","length":"1"}]},{"pgid":"5.3","version":"664'8","reported_seq":"851","reported_epoch":"692","state":"active+undersized+degraded","last_fresh":"2021-03-24T22:04:22.391236+0000","last_change":"2021-03-24T22:03:54.387760+0000","last_active":"2021-03-24T22:04:22.391236+0000","last_peered":"2021-03-24T22:04:22.391236+0000","last_clean":"2021-03-24T22:03:52.358381+0000","last_became_active":"2021-03-24T22:03:54.387760+0000","last_became_peered":"2021-03-24T22:03:54.387760+0000","last_unstale":"2021-03-24T22:04:22.391236+0000","last_undegraded":"2021-03-24T22:03:54.373162+0000","last_fullsized":"2021-03-24T22:03:54.372415+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'6","last_scrub_stamp":"2021-03-24T22:00:16.563338+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-24T21:53:26.584678+0000","last_clean_scrub_stamp":"2021-03-24T22:00:16.563338+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[{"shards":"0,6","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"544'26","reported_seq":"74","reported_epoch":"688","state":"active+undersized+degraded","last_fresh":"2021-03-24T22:04:18.453920+0000","last_change":"2021-03-24T22:03:54.381576+0000","last_active":"2021-03-24T22:04:18.453920+0000","last_peered":"2021-03-24T22:04:18.453920+0000","last_clean":"2021-03-24T22:03:52.362115+0000","last_became_active":"2021-03-24T22:03:54.381576+0000","last_became_peered":"2021-03-24T22:03:54.381576+0000","last_unstale":"2021-03-24T22:04:18.453920+0000","last_undegraded":"2021-03-24T22:03:54.355692+0000","last_fullsized":"2021-03-24T22:03:54.355502+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'16","last_scrub_stamp":"2021-03-24T22:00:58.712071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:00:58.712071+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":46,"num_read_kb":29,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":11}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"311","reported_epoch":"692","state":"active+undersized","last_fresh":"2021-03-24T22:04:22.392680+0000","last_change":"2021-03-24T22:03:54.377379+0000","last_active":"2021-03-24T22:04:22.392680+0000","last_peered":"2021-03-24T22:04:22.392680+0000","last_clean":"2021-03-24T21:51:44.337765+0000","last_became_active":"2021-03-24T22:03:54.377379+0000","last_became_peered":"2021-03-24T22:03:54.377379+0000","last_unstale":"2021-03-24T22:04:22.392680+0000","last_undegraded":"2021-03-24T22:04:22.392680+0000","last_fullsized":"2021-03-24T22:03:54.362378+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:05.892415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:03:05.892415+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"647","reported_epoch":"692","state":"active+clean","last_fresh":"2021-03-24T22:04:22.392747+0000","last_change":"2021-03-24T21:59:04.077747+0000","last_active":"2021-03-24T22:04:22.392747+0000","last_peered":"2021-03-24T22:04:22.392747+0000","last_clean":"2021-03-24T22:04:22.392747+0000","last_became_active":"2021-03-24T21:51:51.759627+0000","last_became_peered":"2021-03-24T21:51:51.759627+0000","last_unstale":"2021-03-24T22:04:22.392747+0000","last_undegraded":"2021-03-24T22:04:22.392747+0000","last_fullsized":"2021-03-24T22:04:22.392747+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:04.077569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T21:59:04.077569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"690'451","reported_seq":"2387","reported_epoch":"692","state":"active+clean","last_fresh":"2021-03-24T22:04:22.392399+0000","last_change":"2021-03-24T22:04:22.392399+0000","last_active":"2021-03-24T22:04:22.392399+0000","last_peered":"2021-03-24T22:04:22.392399+0000","last_clean":"2021-03-24T22:04:22.392399+0000","last_became_active":"2021-03-24T22:02:30.322922+0000","last_became_peered":"2021-03-24T22:02:30.322922+0000","last_unstale":"2021-03-24T22:04:22.392399+0000","last_undegraded":"2021-03-24T22:04:22.392399+0000","last_fullsized":"2021-03-24T22:04:22.392399+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"365'287","last_scrub_stamp":"2021-03-24T21:58:56.648040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:58:56.648040+0000","log_size":451,"ondisk_log_size":451,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16327408,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":46,"num_read_kb":45631,"num_write":30,"num_write_kb":10338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":24969090,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1cd","length":"1"},{"start":"1d4","length":"1"},{"start":"1f0","length":"1"},{"start":"1f3","length":"1"},{"start":"20e","length":"1"},{"start":"211","length":"1"},{"start":"214","length":"1"},{"start":"218","length":"2"},{"start":"21e","length":"1"},{"start":"220","length":"2"},{"start":"22f","length":"1"},{"start":"232","length":"1"},{"start":"239","length":"1"}]},{"pgid":"5.2","version":"0'0","reported_seq":"170","reported_epoch":"689","state":"active+undersized","last_fresh":"2021-03-24T22:04:20.011188+0000","last_change":"2021-03-24T22:03:54.378864+0000","last_active":"2021-03-24T22:04:20.011188+0000","last_peered":"2021-03-24T22:04:20.011188+0000","last_clean":"2021-03-24T22:03:52.356719+0000","last_became_active":"2021-03-24T22:03:54.378864+0000","last_became_peered":"2021-03-24T22:03:54.378864+0000","last_unstale":"2021-03-24T22:04:20.011188+0000","last_undegraded":"2021-03-24T22:04:20.011188+0000","last_fullsized":"2021-03-24T22:03:54.358462+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:14.616769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:00:14.616769+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"544'31","reported_seq":"858","reported_epoch":"692","state":"active+undersized+degraded","last_fresh":"2021-03-24T22:04:22.391440+0000","last_change":"2021-03-24T22:03:54.397050+0000","last_active":"2021-03-24T22:04:22.391440+0000","last_peered":"2021-03-24T22:04:22.391440+0000","last_clean":"2021-03-24T22:03:52.358525+0000","last_became_active":"2021-03-24T22:03:54.397050+0000","last_became_peered":"2021-03-24T22:03:54.397050+0000","last_unstale":"2021-03-24T22:04:22.391440+0000","last_undegraded":"2021-03-24T22:03:54.386375+0000","last_fullsized":"2021-03-24T22:03:54.372051+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'21","last_scrub_stamp":"2021-03-24T22:00:57.659209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:00:57.659209+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":60,"num_read_kb":42,"num_write":34,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":2568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":10}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"50","reported_epoch":"688","state":"active+undersized","last_fresh":"2021-03-24T22:04:18.452677+0000","last_change":"2021-03-24T22:03:54.384167+0000","last_active":"2021-03-24T22:04:18.452677+0000","last_peered":"2021-03-24T22:04:18.452677+0000","last_clean":"2021-03-24T22:03:52.363095+0000","last_became_active":"2021-03-24T22:03:54.384167+0000","last_became_peered":"2021-03-24T22:03:54.384167+0000","last_unstale":"2021-03-24T22:04:18.452677+0000","last_undegraded":"2021-03-24T22:04:18.452677+0000","last_fullsized":"2021-03-24T22:03:54.357535+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:38.144956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:38.144956+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"44","reported_epoch":"688","state":"active+clean","last_fresh":"2021-03-24T22:04:18.449926+0000","last_change":"2021-03-24T22:03:45.427189+0000","last_active":"2021-03-24T22:04:18.449926+0000","last_peered":"2021-03-24T22:04:18.449926+0000","last_clean":"2021-03-24T22:04:18.449926+0000","last_became_active":"2021-03-24T22:03:45.426809+0000","last_became_peered":"2021-03-24T22:03:45.426809+0000","last_unstale":"2021-03-24T22:04:18.449926+0000","last_undegraded":"2021-03-24T22:04:18.449926+0000","last_fullsized":"2021-03-24T22:04:18.449926+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:01.394465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:03:01.394465+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"175'22","reported_seq":"1695","reported_epoch":"688","state":"active+undersized","last_fresh":"2021-03-24T22:04:18.451285+0000","last_change":"2021-03-24T22:03:54.370354+0000","last_active":"2021-03-24T22:04:18.451285+0000","last_peered":"2021-03-24T22:04:18.451285+0000","last_clean":"2021-03-24T22:03:43.464570+0000","last_became_active":"2021-03-24T22:03:54.370354+0000","last_became_peered":"2021-03-24T22:03:54.370354+0000","last_unstale":"2021-03-24T22:04:18.451285+0000","last_undegraded":"2021-03-24T22:04:18.451285+0000","last_fullsized":"2021-03-24T22:03:54.354859+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'22","last_scrub_stamp":"2021-03-24T21:58:22.940875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:58:22.940875+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":13,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3283,"num_write":3,"num_write_kb":845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":313386,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"577","reported_epoch":"688","state":"active+undersized","last_fresh":"2021-03-24T22:04:18.455525+0000","last_change":"2021-03-24T22:03:54.384689+0000","last_active":"2021-03-24T22:04:18.455525+0000","last_peered":"2021-03-24T22:04:18.455525+0000","last_clean":"2021-03-24T22:03:43.465999+0000","last_became_active":"2021-03-24T22:03:54.384689+0000","last_became_peered":"2021-03-24T22:03:54.384689+0000","last_unstale":"2021-03-24T22:04:18.455525+0000","last_undegraded":"2021-03-24T22:04:18.455525+0000","last_fullsized":"2021-03-24T22:03:54.357377+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:01:00.693679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:01:00.693679+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"544'34","reported_seq":"1192","reported_epoch":"692","state":"active+clean","last_fresh":"2021-03-24T22:04:22.394043+0000","last_change":"2021-03-24T22:03:54.403711+0000","last_active":"2021-03-24T22:04:22.394043+0000","last_peered":"2021-03-24T22:04:22.394043+0000","last_clean":"2021-03-24T22:04:22.394043+0000","last_became_active":"2021-03-24T22:03:54.401857+0000","last_became_peered":"2021-03-24T22:03:54.401857+0000","last_unstale":"2021-03-24T22:04:22.394043+0000","last_undegraded":"2021-03-24T22:04:22.394043+0000","last_fullsized":"2021-03-24T22:04:22.394043+0000","mapping_epoch":663,"log_start":"69'16","ondisk_log_start":"69'16","created":62,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'34","last_scrub_stamp":"2021-03-24T22:02:59.519401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:02:59.519401+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1836,"num_objects":21,"num_object_clones":0,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":95,"num_read_kb":61,"num_write":51,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":157,"num_bytes_recovered":13482,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"66'5","reported_seq":"322","reported_epoch":"688","state":"active+clean","last_fresh":"2021-03-24T22:04:18.450920+0000","last_change":"2021-03-24T22:03:46.204959+0000","last_active":"2021-03-24T22:04:18.450920+0000","last_peered":"2021-03-24T22:04:18.450920+0000","last_clean":"2021-03-24T22:04:18.450920+0000","last_became_active":"2021-03-24T22:03:45.433742+0000","last_became_peered":"2021-03-24T22:03:45.433742+0000","last_unstale":"2021-03-24T22:04:18.450920+0000","last_undegraded":"2021-03-24T22:04:18.450920+0000","last_fullsized":"2021-03-24T22:04:18.450920+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2021-03-24T21:58:58.818001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:39.286101+0000","last_clean_scrub_stamp":"2021-03-24T21:58:58.818001+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":213,"num_read_kb":208,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"667","reported_epoch":"692","state":"active+clean","last_fresh":"2021-03-24T22:04:22.397717+0000","last_change":"2021-03-24T21:59:46.941755+0000","last_active":"2021-03-24T22:04:22.397717+0000","last_peered":"2021-03-24T22:04:22.397717+0000","last_clean":"2021-03-24T22:04:22.397717+0000","last_became_active":"2021-03-24T21:59:12.281282+0000","last_became_peered":"2021-03-24T21:59:12.281282+0000","last_unstale":"2021-03-24T22:04:22.397717+0000","last_undegraded":"2021-03-24T22:04:22.397717+0000","last_fullsized":"2021-03-24T22:04:22.397717+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:46.941623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:59:46.941623+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"44","reported_epoch":"688","state":"active+clean","last_fresh":"2021-03-24T22:04:18.452955+0000","last_change":"2021-03-24T22:03:45.425244+0000","last_active":"2021-03-24T22:04:18.452955+0000","last_peered":"2021-03-24T22:04:18.452955+0000","last_clean":"2021-03-24T22:04:18.452955+0000","last_became_active":"2021-03-24T22:03:45.424562+0000","last_became_peered":"2021-03-24T22:03:45.424562+0000","last_unstale":"2021-03-24T22:04:18.452955+0000","last_undegraded":"2021-03-24T22:04:18.452955+0000","last_fullsized":"2021-03-24T22:04:18.452955+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:11.533322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:03:11.533322+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"195'69","reported_seq":"456","reported_epoch":"688","state":"active+clean","last_fresh":"2021-03-24T22:04:18.453412+0000","last_change":"2021-03-24T22:04:18.453864+0000","last_active":"2021-03-24T22:04:18.453412+0000","last_peered":"2021-03-24T22:04:18.453412+0000","last_clean":"2021-03-24T22:04:18.453412+0000","last_became_active":"2021-03-24T22:01:57.317669+0000","last_became_peered":"2021-03-24T22:01:57.317669+0000","last_unstale":"2021-03-24T22:04:18.453412+0000","last_undegraded":"2021-03-24T22:04:18.453412+0000","last_fullsized":"2021-03-24T22:04:18.453412+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'69","last_scrub_stamp":"2021-03-24T21:59:09.930390+0000","last_deep_scrub":"195'69","last_deep_scrub_stamp":"2021-03-24T21:56:49.886058+0000","last_clean_scrub_stamp":"2021-03-24T21:59:09.930390+0000","log_size":69,"ondisk_log_size":69,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1672296,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":8383,"num_write":20,"num_write_kb":7737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1cd","length":"1"},{"start":"1d4","length":"1"},{"start":"1f0","length":"1"},{"start":"1f3","length":"1"},{"start":"20e","length":"1"},{"start":"211","length":"1"},{"start":"218","length":"2"},{"start":"21e","length":"1"},{"start":"220","length":"2"},{"start":"22f","length":"1"},{"start":"232","length":"1"}]},{"pgid":"5.4","version":"0'0","reported_seq":"164","reported_epoch":"687","state":"active+undersized","last_fresh":"2021-03-24T22:04:17.457579+0000","last_change":"2021-03-24T22:03:54.381736+0000","last_active":"2021-03-24T22:04:17.457579+0000","last_peered":"2021-03-24T22:04:17.457579+0000","last_clean":"2021-03-24T22:03:52.506468+0000","last_became_active":"2021-03-24T22:03:54.381736+0000","last_became_peered":"2021-03-24T22:03:54.381736+0000","last_unstale":"2021-03-24T22:04:17.457579+0000","last_undegraded":"2021-03-24T22:04:17.457579+0000","last_fullsized":"2021-03-24T22:03:54.363560+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:34.319421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:02:34.319421+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"544'27","reported_seq":"236","reported_epoch":"689","state":"active+clean","last_fresh":"2021-03-24T22:04:20.012966+0000","last_change":"2021-03-24T22:03:00.744262+0000","last_active":"2021-03-24T22:04:20.012966+0000","last_peered":"2021-03-24T22:04:20.012966+0000","last_clean":"2021-03-24T22:04:20.012966+0000","last_became_active":"2021-03-24T22:01:57.311205+0000","last_became_peered":"2021-03-24T22:01:57.311205+0000","last_unstale":"2021-03-24T22:04:20.012966+0000","last_undegraded":"2021-03-24T22:04:20.012966+0000","last_fullsized":"2021-03-24T22:04:20.012966+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'27","last_scrub_stamp":"2021-03-24T22:03:00.744096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:03:00.744096+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":27,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":1398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"692","reported_epoch":"692","state":"active+undersized","last_fresh":"2021-03-24T22:04:22.396883+0000","last_change":"2021-03-24T22:03:54.376369+0000","last_active":"2021-03-24T22:04:22.396883+0000","last_peered":"2021-03-24T22:04:22.396883+0000","last_clean":"2021-03-24T22:03:52.353076+0000","last_became_active":"2021-03-24T22:03:54.376369+0000","last_became_peered":"2021-03-24T22:03:54.376369+0000","last_unstale":"2021-03-24T22:04:22.396883+0000","last_undegraded":"2021-03-24T22:04:22.396883+0000","last_fullsized":"2021-03-24T22:03:54.359110+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:33.533840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:33.533840+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"156","reported_epoch":"687","state":"active+clean","last_fresh":"2021-03-24T22:04:17.456193+0000","last_change":"2021-03-24T22:03:43.285542+0000","last_active":"2021-03-24T22:04:17.456193+0000","last_peered":"2021-03-24T22:04:17.456193+0000","last_clean":"2021-03-24T22:04:17.456193+0000","last_became_active":"2021-03-24T22:01:57.315729+0000","last_became_peered":"2021-03-24T22:01:57.315729+0000","last_unstale":"2021-03-24T22:04:17.456193+0000","last_undegraded":"2021-03-24T22:04:17.456193+0000","last_fullsized":"2021-03-24T22:04:17.456193+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:43.285374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:49:51.529051+0000","last_clean_scrub_stamp":"2021-03-24T22:03:43.285374+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"307","reported_epoch":"692","state":"active+undersized","last_fresh":"2021-03-24T22:04:22.396138+0000","last_change":"2021-03-24T22:03:54.369996+0000","last_active":"2021-03-24T22:04:22.396138+0000","last_peered":"2021-03-24T22:04:22.396138+0000","last_clean":"2021-03-24T21:51:50.731137+0000","last_became_active":"2021-03-24T22:03:54.369996+0000","last_became_peered":"2021-03-24T22:03:54.369996+0000","last_unstale":"2021-03-24T22:04:22.396138+0000","last_undegraded":"2021-03-24T22:04:22.396138+0000","last_fullsized":"2021-03-24T22:03:54.358993+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:01.931136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:03:01.931136+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.6","version":"692'312","reported_seq":"2452","reported_epoch":"692","state":"active+clean","last_fresh":"2021-03-24T22:04:22.417220+0000","last_change":"2021-03-24T22:04:22.417220+0000","last_active":"2021-03-24T22:04:22.417220+0000","last_peered":"2021-03-24T22:04:22.417220+0000","last_clean":"2021-03-24T22:04:22.417220+0000","last_became_active":"2021-03-24T22:02:31.331916+0000","last_became_peered":"2021-03-24T22:02:31.331916+0000","last_unstale":"2021-03-24T22:04:22.417220+0000","last_undegraded":"2021-03-24T22:04:22.417220+0000","last_fullsized":"2021-03-24T22:04:22.417220+0000","mapping_epoch":580,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":581,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":312,"ondisk_log_size":312,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18700479,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":136,"num_read_kb":153073,"num_write":89,"num_write_kb":39138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":35483798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1cd","length":"1"},{"start":"1d4","length":"1"},{"start":"1f0","length":"1"},{"start":"1f3","length":"1"},{"start":"20e","length":"1"},{"start":"211","length":"1"},{"start":"214","length":"1"},{"start":"218","length":"2"},{"start":"21e","length":"1"},{"start":"220","length":"2"},{"start":"22f","length":"1"},{"start":"232","length":"1"},{"start":"239","length":"1"}]},{"pgid":"5.6","version":"0'0","reported_seq":"139","reported_epoch":"690","state":"active+undersized","last_fresh":"2021-03-24T22:04:20.359177+0000","last_change":"2021-03-24T22:03:54.372249+0000","last_active":"2021-03-24T22:04:20.359177+0000","last_peered":"2021-03-24T22:04:20.359177+0000","last_clean":"2021-03-24T22:03:52.503908+0000","last_became_active":"2021-03-24T22:03:54.372249+0000","last_became_peered":"2021-03-24T22:03:54.372249+0000","last_unstale":"2021-03-24T22:04:20.359177+0000","last_undegraded":"2021-03-24T22:04:20.359177+0000","last_fullsized":"2021-03-24T22:03:54.353070+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:55.848098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:02:55.848098+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"544'32","reported_seq":"701","reported_epoch":"688","state":"active+undersized+degraded","last_fresh":"2021-03-24T22:04:18.455734+0000","last_change":"2021-03-24T22:03:54.385512+0000","last_active":"2021-03-24T22:04:18.455734+0000","last_peered":"2021-03-24T22:04:18.455734+0000","last_clean":"2021-03-24T22:03:52.362534+0000","last_became_active":"2021-03-24T22:03:54.385512+0000","last_became_peered":"2021-03-24T22:03:54.385512+0000","last_unstale":"2021-03-24T22:04:18.455734+0000","last_undegraded":"2021-03-24T22:03:54.365962+0000","last_fullsized":"2021-03-24T22:03:54.365788+0000","mapping_epoch":663,"log_start":"89'22","ondisk_log_start":"89'22","created":62,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'32","last_scrub_stamp":"2021-03-24T22:02:59.400796+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2021-03-24T21:53:19.717244+0000","last_clean_scrub_stamp":"2021-03-24T22:02:59.400796+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":802,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":63,"num_read_kb":43,"num_write":35,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":11}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"668","reported_epoch":"690","state":"active+clean","last_fresh":"2021-03-24T22:04:20.358980+0000","last_change":"2021-03-24T22:03:08.699641+0000","last_active":"2021-03-24T22:04:20.358980+0000","last_peered":"2021-03-24T22:04:20.358980+0000","last_clean":"2021-03-24T22:04:20.358980+0000","last_became_active":"2021-03-24T21:51:40.351000+0000","last_became_peered":"2021-03-24T21:51:40.351000+0000","last_unstale":"2021-03-24T22:04:20.358980+0000","last_undegraded":"2021-03-24T22:04:20.358980+0000","last_fullsized":"2021-03-24T22:04:20.358980+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:08.699469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:39.286101+0000","last_clean_scrub_stamp":"2021-03-24T22:03:08.699469+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"61'1","reported_seq":"44","reported_epoch":"688","state":"active+clean","last_fresh":"2021-03-24T22:04:18.451570+0000","last_change":"2021-03-24T22:03:45.427087+0000","last_active":"2021-03-24T22:04:18.451570+0000","last_peered":"2021-03-24T22:04:18.451570+0000","last_clean":"2021-03-24T22:04:18.451570+0000","last_became_active":"2021-03-24T22:03:45.426685+0000","last_became_peered":"2021-03-24T22:03:45.426685+0000","last_unstale":"2021-03-24T22:04:18.451570+0000","last_undegraded":"2021-03-24T22:04:18.451570+0000","last_fullsized":"2021-03-24T22:04:18.451570+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2021-03-24T21:57:12.046206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:12.046206+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":436,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"156","reported_epoch":"687","state":"active+clean","last_fresh":"2021-03-24T22:04:17.459367+0000","last_change":"2021-03-24T22:03:15.443335+0000","last_active":"2021-03-24T22:04:17.459367+0000","last_peered":"2021-03-24T22:04:17.459367+0000","last_clean":"2021-03-24T22:04:17.459367+0000","last_became_active":"2021-03-24T22:01:57.318275+0000","last_became_peered":"2021-03-24T22:01:57.318275+0000","last_unstale":"2021-03-24T22:04:17.459367+0000","last_undegraded":"2021-03-24T22:04:17.459367+0000","last_fullsized":"2021-03-24T22:04:17.459367+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:15.443169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:03:15.443169+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"693'389","reported_seq":"2441","reported_epoch":"693","state":"active+clean","last_fresh":"2021-03-24T22:04:23.400818+0000","last_change":"2021-03-24T22:04:22.401657+0000","last_active":"2021-03-24T22:04:23.400818+0000","last_peered":"2021-03-24T22:04:23.400818+0000","last_clean":"2021-03-24T22:04:23.400818+0000","last_became_active":"2021-03-24T22:03:50.369486+0000","last_became_peered":"2021-03-24T22:03:50.369486+0000","last_unstale":"2021-03-24T22:04:23.400818+0000","last_undegraded":"2021-03-24T22:04:23.400818+0000","last_fullsized":"2021-03-24T22:04:23.400818+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"616'351","last_scrub_stamp":"2021-03-24T22:03:06.473123+0000","last_deep_scrub":"616'351","last_deep_scrub_stamp":"2021-03-24T22:03:06.473123+0000","last_clean_scrub_stamp":"2021-03-24T22:03:06.473123+0000","log_size":389,"ondisk_log_size":389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11076539,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":72,"num_read_kb":75339,"num_write":47,"num_write_kb":18322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":35984561,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1cd","length":"1"},{"start":"1d4","length":"1"},{"start":"1f0","length":"1"},{"start":"1f3","length":"1"},{"start":"20e","length":"1"},{"start":"211","length":"1"},{"start":"214","length":"1"},{"start":"218","length":"2"},{"start":"21e","length":"1"},{"start":"220","length":"2"},{"start":"22f","length":"1"},{"start":"232","length":"1"},{"start":"239","length":"1"}]},{"pgid":"5.5","version":"0'0","reported_seq":"703","reported_epoch":"692","state":"active+clean","last_fresh":"2021-03-24T22:04:22.392154+0000","last_change":"2021-03-24T22:03:45.430539+0000","last_active":"2021-03-24T22:04:22.392154+0000","last_peered":"2021-03-24T22:04:22.392154+0000","last_clean":"2021-03-24T22:04:22.392154+0000","last_became_active":"2021-03-24T22:03:45.430121+0000","last_became_peered":"2021-03-24T22:03:45.430121+0000","last_unstale":"2021-03-24T22:04:22.392154+0000","last_undegraded":"2021-03-24T22:04:22.392154+0000","last_fullsized":"2021-03-24T22:04:22.392154+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:58:23.865909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:58:23.865909+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"544'37","reported_seq":"72","reported_epoch":"688","state":"active+clean","last_fresh":"2021-03-24T22:04:18.452552+0000","last_change":"2021-03-24T22:03:45.462291+0000","last_active":"2021-03-24T22:04:18.452552+0000","last_peered":"2021-03-24T22:04:18.452552+0000","last_clean":"2021-03-24T22:04:18.452552+0000","last_became_active":"2021-03-24T22:03:45.426862+0000","last_became_peered":"2021-03-24T22:03:45.426862+0000","last_unstale":"2021-03-24T22:04:18.452552+0000","last_undegraded":"2021-03-24T22:04:18.452552+0000","last_fullsized":"2021-03-24T22:04:18.452552+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'37","last_scrub_stamp":"2021-03-24T22:03:12.502778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:03:12.502778+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":70,"num_read_kb":47,"num_write":40,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":5724,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"89'2","reported_seq":"291","reported_epoch":"692","state":"active+clean","last_fresh":"2021-03-24T22:04:22.391695+0000","last_change":"2021-03-24T22:01:43.721006+0000","last_active":"2021-03-24T22:04:22.391695+0000","last_peered":"2021-03-24T22:04:22.391695+0000","last_clean":"2021-03-24T22:04:22.391695+0000","last_became_active":"2021-03-24T22:01:43.393718+0000","last_became_peered":"2021-03-24T22:01:43.393718+0000","last_unstale":"2021-03-24T22:04:22.391695+0000","last_undegraded":"2021-03-24T22:04:22.391695+0000","last_fullsized":"2021-03-24T22:04:22.391695+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2021-03-24T21:57:58.491830+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2021-03-24T21:57:58.491830+0000","last_clean_scrub_stamp":"2021-03-24T21:57:58.491830+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":179,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"50","reported_epoch":"688","state":"active+undersized","last_fresh":"2021-03-24T22:04:18.453729+0000","last_change":"2021-03-24T22:03:54.380635+0000","last_active":"2021-03-24T22:04:18.453729+0000","last_peered":"2021-03-24T22:04:18.453729+0000","last_clean":"2021-03-24T22:03:52.362012+0000","last_became_active":"2021-03-24T22:03:54.380635+0000","last_became_peered":"2021-03-24T22:03:54.380635+0000","last_unstale":"2021-03-24T22:04:18.453729+0000","last_undegraded":"2021-03-24T22:04:18.453729+0000","last_fullsized":"2021-03-24T22:03:54.364402+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:20.729422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:06.800417+0000","last_clean_scrub_stamp":"2021-03-24T22:02:20.729422+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"707","reported_epoch":"692","state":"active+clean","last_fresh":"2021-03-24T22:04:22.393849+0000","last_change":"2021-03-24T22:02:55.490903+0000","last_active":"2021-03-24T22:04:22.393849+0000","last_peered":"2021-03-24T22:04:22.393849+0000","last_clean":"2021-03-24T22:04:22.393849+0000","last_became_active":"2021-03-24T22:01:57.314372+0000","last_became_peered":"2021-03-24T22:01:57.314372+0000","last_unstale":"2021-03-24T22:04:22.393849+0000","last_undegraded":"2021-03-24T22:04:22.393849+0000","last_fullsized":"2021-03-24T22:04:22.393849+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:55.490744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:52:59.706511+0000","last_clean_scrub_stamp":"2021-03-24T22:02:55.490744+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"685'496","reported_seq":"2479","reported_epoch":"690","state":"active+clean","last_fresh":"2021-03-24T22:04:20.389119+0000","last_change":"2021-03-24T22:04:20.389464+0000","last_active":"2021-03-24T22:04:20.389119+0000","last_peered":"2021-03-24T22:04:20.389119+0000","last_clean":"2021-03-24T22:04:20.389119+0000","last_became_active":"2021-03-24T22:01:47.305050+0000","last_became_peered":"2021-03-24T22:01:47.305050+0000","last_unstale":"2021-03-24T22:04:20.389119+0000","last_undegraded":"2021-03-24T22:04:20.389119+0000","last_fullsized":"2021-03-24T22:04:20.389119+0000","mapping_epoch":536,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":537,"parent":"0.0","parent_split_bits":0,"last_scrub":"563'386","last_scrub_stamp":"2021-03-24T22:02:19.755652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:02:19.755652+0000","log_size":496,"ondisk_log_size":496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11364618,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":7098,"num_write":54,"num_write_kb":19812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":55849492,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1cd","length":"1"},{"start":"1d4","length":"1"},{"start":"1f0","length":"1"},{"start":"1f3","length":"1"},{"start":"20e","length":"1"},{"start":"211","length":"1"},{"start":"214","length":"1"},{"start":"218","length":"2"},{"start":"21e","length":"1"},{"start":"220","length":"2"},{"start":"22f","length":"1"},{"start":"232","length":"1"}]},{"pgid":"4.b","version":"544'17","reported_seq":"809","reported_epoch":"692","state":"active+clean","last_fresh":"2021-03-24T22:04:22.391787+0000","last_change":"2021-03-24T22:03:46.443743+0000","last_active":"2021-03-24T22:04:22.391787+0000","last_peered":"2021-03-24T22:04:22.391787+0000","last_clean":"2021-03-24T22:04:22.391787+0000","last_became_active":"2021-03-24T22:03:46.443398+0000","last_became_peered":"2021-03-24T22:03:46.443398+0000","last_unstale":"2021-03-24T22:04:22.391787+0000","last_undegraded":"2021-03-24T22:04:22.391787+0000","last_fullsized":"2021-03-24T22:04:22.391787+0000","mapping_epoch":655,"log_start":"91'9","ondisk_log_start":"91'9","created":62,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'14","last_scrub_stamp":"2021-03-24T21:59:05.058513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:59:05.058513+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1158,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":58,"num_read_kb":36,"num_write":30,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2316,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"688","reported_epoch":"692","state":"active+undersized","last_fresh":"2021-03-24T22:04:22.395030+0000","last_change":"2021-03-24T22:03:54.375391+0000","last_active":"2021-03-24T22:04:22.395030+0000","last_peered":"2021-03-24T22:04:22.395030+0000","last_clean":"2021-03-24T22:03:52.354140+0000","last_became_active":"2021-03-24T22:03:54.375391+0000","last_became_peered":"2021-03-24T22:03:54.375391+0000","last_unstale":"2021-03-24T22:04:22.395030+0000","last_undegraded":"2021-03-24T22:04:22.395030+0000","last_fullsized":"2021-03-24T22:03:54.363766+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:01:55.431129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T22:01:55.431129+0000","last_clean_scrub_stamp":"2021-03-24T22:01:55.431129+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"660","reported_epoch":"690","state":"active+clean","last_fresh":"2021-03-24T22:04:20.360083+0000","last_change":"2021-03-24T22:03:55.642769+0000","last_active":"2021-03-24T22:04:20.360083+0000","last_peered":"2021-03-24T22:04:20.360083+0000","last_clean":"2021-03-24T22:04:20.360083+0000","last_became_active":"2021-03-24T21:51:49.411921+0000","last_became_peered":"2021-03-24T21:51:49.411921+0000","last_unstale":"2021-03-24T22:04:20.360083+0000","last_undegraded":"2021-03-24T22:04:20.360083+0000","last_fullsized":"2021-03-24T22:04:20.360083+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:55.642573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:03:55.642573+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"710","reported_epoch":"692","state":"active+clean","last_fresh":"2021-03-24T22:04:22.398134+0000","last_change":"2021-03-24T22:02:22.324168+0000","last_active":"2021-03-24T22:04:22.398134+0000","last_peered":"2021-03-24T22:04:22.398134+0000","last_clean":"2021-03-24T22:04:22.398134+0000","last_became_active":"2021-03-24T22:02:22.323876+0000","last_became_peered":"2021-03-24T22:02:22.323876+0000","last_unstale":"2021-03-24T22:04:22.398134+0000","last_undegraded":"2021-03-24T22:04:22.398134+0000","last_fullsized":"2021-03-24T22:04:22.398134+0000","mapping_epoch":571,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:00.033461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:35.636299+0000","last_clean_scrub_stamp":"2021-03-24T21:59:00.033461+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"675'294","reported_seq":"2281","reported_epoch":"690","state":"active+clean","last_fresh":"2021-03-24T22:04:20.361157+0000","last_change":"2021-03-24T22:04:20.361157+0000","last_active":"2021-03-24T22:04:20.361157+0000","last_peered":"2021-03-24T22:04:20.361157+0000","last_clean":"2021-03-24T22:04:20.361157+0000","last_became_active":"2021-03-24T22:02:01.325483+0000","last_became_peered":"2021-03-24T22:02:01.325483+0000","last_unstale":"2021-03-24T22:04:20.361157+0000","last_undegraded":"2021-03-24T22:04:20.361157+0000","last_fullsized":"2021-03-24T22:04:20.361157+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"619'282","last_scrub_stamp":"2021-03-24T22:03:12.788348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:03:12.788348+0000","log_size":294,"ondisk_log_size":294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8006097,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":55,"num_read_kb":66904,"num_write":49,"num_write_kb":12818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":21446378,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1cd","length":"1"},{"start":"1d4","length":"1"},{"start":"1f0","length":"1"},{"start":"1f3","length":"1"},{"start":"20e","length":"1"},{"start":"211","length":"1"},{"start":"214","length":"1"},{"start":"218","length":"2"},{"start":"21e","length":"1"},{"start":"220","length":"2"},{"start":"22f","length":"1"},{"start":"232","length":"1"}]},{"pgid":"4.a","version":"544'45","reported_seq":"632","reported_epoch":"689","state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2021-03-24T22:04:20.011023+0000","last_change":"2021-03-24T22:03:54.383856+0000","last_active":"2021-03-24T22:04:20.011023+0000","last_peered":"2021-03-24T22:04:20.011023+0000","last_clean":"2021-03-24T22:03:43.471717+0000","last_became_active":"2021-03-24T22:03:54.377044+0000","last_became_peered":"2021-03-24T22:03:54.377044+0000","last_unstale":"2021-03-24T22:04:20.011023+0000","last_undegraded":"2021-03-24T22:03:54.358686+0000","last_fullsized":"2021-03-24T22:03:54.358470+0000","mapping_epoch":663,"log_start":"69'23","ondisk_log_start":"69'23","created":62,"last_epoch_clean":648,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'23","last_scrub_stamp":"2021-03-24T22:00:23.343879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:00:23.343879+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1466,"num_objects":21,"num_object_clones":0,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":104,"num_read_kb":71,"num_write":60,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":100,"num_bytes_recovered":7018,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":21}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":"254","reported_epoch":"689","state":"active+undersized","last_fresh":"2021-03-24T22:04:20.012645+0000","last_change":"2021-03-24T22:03:54.368452+0000","last_active":"2021-03-24T22:04:20.012645+0000","last_peered":"2021-03-24T22:04:20.012645+0000","last_clean":"2021-03-24T22:01:55.300506+0000","last_became_active":"2021-03-24T22:03:54.368452+0000","last_became_peered":"2021-03-24T22:03:54.368452+0000","last_unstale":"2021-03-24T22:04:20.012645+0000","last_undegraded":"2021-03-24T22:04:20.012645+0000","last_fullsized":"2021-03-24T22:03:54.360715+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:00.965024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:03:00.965024+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"134","reported_epoch":"690","state":"active+undersized","last_fresh":"2021-03-24T22:04:20.358873+0000","last_change":"2021-03-24T22:03:54.389287+0000","last_active":"2021-03-24T22:04:20.358873+0000","last_peered":"2021-03-24T22:04:20.358873+0000","last_clean":"2021-03-24T22:03:52.504705+0000","last_became_active":"2021-03-24T22:03:54.389287+0000","last_became_peered":"2021-03-24T22:03:54.389287+0000","last_unstale":"2021-03-24T22:04:20.358873+0000","last_undegraded":"2021-03-24T22:04:20.358873+0000","last_fullsized":"2021-03-24T22:03:54.366298+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:01:40.445404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:01:40.445404+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"453","reported_epoch":"690","state":"active+clean","last_fresh":"2021-03-24T22:04:20.357650+0000","last_change":"2021-03-24T22:03:44.560178+0000","last_active":"2021-03-24T22:04:20.357650+0000","last_peered":"2021-03-24T22:04:20.357650+0000","last_clean":"2021-03-24T22:04:20.357650+0000","last_became_active":"2021-03-24T22:01:57.311325+0000","last_became_peered":"2021-03-24T22:01:57.311325+0000","last_unstale":"2021-03-24T22:04:20.357650+0000","last_undegraded":"2021-03-24T22:04:20.357650+0000","last_fullsized":"2021-03-24T22:04:20.357650+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:44.560035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:03:44.560035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"678'454","reported_seq":"2488","reported_epoch":"692","state":"active+clean","last_fresh":"2021-03-24T22:04:22.409471+0000","last_change":"2021-03-24T22:04:22.409471+0000","last_active":"2021-03-24T22:04:22.409471+0000","last_peered":"2021-03-24T22:04:22.409471+0000","last_clean":"2021-03-24T22:04:22.409471+0000","last_became_active":"2021-03-24T22:03:49.375663+0000","last_became_peered":"2021-03-24T22:03:49.375663+0000","last_unstale":"2021-03-24T22:04:22.409471+0000","last_undegraded":"2021-03-24T22:04:22.409471+0000","last_fullsized":"2021-03-24T22:04:22.409471+0000","mapping_epoch":658,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":659,"parent":"0.0","parent_split_bits":0,"last_scrub":"609'395","last_scrub_stamp":"2021-03-24T22:03:04.501653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:03:04.501653+0000","log_size":454,"ondisk_log_size":454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15579834,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":67,"num_read_kb":81140,"num_write":54,"num_write_kb":15808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":32161412,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1cd","length":"1"},{"start":"1d4","length":"1"},{"start":"1f0","length":"1"},{"start":"1f3","length":"1"},{"start":"20e","length":"1"},{"start":"211","length":"1"},{"start":"214","length":"1"},{"start":"218","length":"2"},{"start":"21e","length":"1"},{"start":"220","length":"2"},{"start":"22f","length":"1"},{"start":"232","length":"1"},{"start":"239","length":"1"}]},{"pgid":"5.8","version":"0'0","reported_seq":"706","reported_epoch":"690","state":"active+clean","last_fresh":"2021-03-24T22:04:20.357401+0000","last_change":"2021-03-24T22:03:45.433545+0000","last_active":"2021-03-24T22:04:20.357401+0000","last_peered":"2021-03-24T22:04:20.357401+0000","last_clean":"2021-03-24T22:04:20.357401+0000","last_became_active":"2021-03-24T22:03:45.432819+0000","last_became_peered":"2021-03-24T22:03:45.432819+0000","last_unstale":"2021-03-24T22:04:20.357401+0000","last_undegraded":"2021-03-24T22:04:20.357401+0000","last_fullsized":"2021-03-24T22:04:20.357401+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:58.801650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:02:58.801650+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"544'38","reported_seq":"342","reported_epoch":"692","state":"active+remapped+backfill_toofull","last_fresh":"2021-03-24T22:04:22.391358+0000","last_change":"2021-03-24T22:04:16.909026+0000","last_active":"2021-03-24T22:04:22.391358+0000","last_peered":"2021-03-24T22:04:22.391358+0000","last_clean":"2021-03-24T22:03:43.462713+0000","last_became_active":"2021-03-24T22:03:46.436848+0000","last_became_peered":"2021-03-24T22:03:46.436848+0000","last_unstale":"2021-03-24T22:04:22.391358+0000","last_undegraded":"2021-03-24T22:04:22.391358+0000","last_fullsized":"2021-03-24T22:04:22.391358+0000","mapping_epoch":655,"log_start":"69'20","ondisk_log_start":"69'20","created":62,"last_epoch_clean":647,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'38","last_scrub_stamp":"2021-03-24T22:03:03.442382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:03:03.442382+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1582,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":20,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":100,"num_read_kb":66,"num_write":56,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":3164,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[3,4,6],"avail_no_missing":["3","4","6"],"object_location_counts":[{"shards":"3,4,6","objects":20}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":"169","reported_epoch":"687","state":"active+clean","last_fresh":"2021-03-24T22:04:17.457692+0000","last_change":"2021-03-24T22:03:45.437719+0000","last_active":"2021-03-24T22:04:17.457692+0000","last_peered":"2021-03-24T22:04:17.457692+0000","last_clean":"2021-03-24T22:04:17.457692+0000","last_became_active":"2021-03-24T22:03:45.437360+0000","last_became_peered":"2021-03-24T22:03:45.437360+0000","last_unstale":"2021-03-24T22:04:17.457692+0000","last_undegraded":"2021-03-24T22:04:17.457692+0000","last_fullsized":"2021-03-24T22:04:17.457692+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:12.939226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:59:12.939226+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"724","reported_epoch":"692","state":"active+clean","last_fresh":"2021-03-24T22:04:22.393842+0000","last_change":"2021-03-24T22:03:45.436416+0000","last_active":"2021-03-24T22:04:22.393842+0000","last_peered":"2021-03-24T22:04:22.393842+0000","last_clean":"2021-03-24T22:04:22.393842+0000","last_became_active":"2021-03-24T22:03:45.436096+0000","last_became_peered":"2021-03-24T22:03:45.436096+0000","last_unstale":"2021-03-24T22:04:22.393842+0000","last_undegraded":"2021-03-24T22:04:22.393842+0000","last_fullsized":"2021-03-24T22:04:22.393842+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:18.322187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:22.724840+0000","last_clean_scrub_stamp":"2021-03-24T22:00:18.322187+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"659'310","reported_seq":"2073","reported_epoch":"692","state":"active+undersized+degraded","last_fresh":"2021-03-24T22:04:22.398169+0000","last_change":"2021-03-24T22:03:54.377252+0000","last_active":"2021-03-24T22:04:22.398169+0000","last_peered":"2021-03-24T22:04:22.398169+0000","last_clean":"2021-03-24T22:03:52.353258+0000","last_became_active":"2021-03-24T22:03:54.377252+0000","last_became_peered":"2021-03-24T22:03:54.377252+0000","last_unstale":"2021-03-24T22:04:22.398169+0000","last_undegraded":"2021-03-24T22:03:54.361571+0000","last_fullsized":"2021-03-24T22:03:54.361406+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"624'277","last_scrub_stamp":"2021-03-24T22:03:14.428074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:03:14.428074+0000","log_size":310,"ondisk_log_size":310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":15,"stat_sum":{"num_bytes":18502262,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":13,"num_read_kb":13875,"num_write":7,"num_write_kb":6244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":8}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"548'11","reported_seq":"184","reported_epoch":"688","state":"active+clean","last_fresh":"2021-03-24T22:04:18.452149+0000","last_change":"2021-03-24T22:01:57.504107+0000","last_active":"2021-03-24T22:04:18.452149+0000","last_peered":"2021-03-24T22:04:18.452149+0000","last_clean":"2021-03-24T22:04:18.452149+0000","last_became_active":"2021-03-24T22:01:57.313563+0000","last_became_peered":"2021-03-24T22:01:57.313563+0000","last_unstale":"2021-03-24T22:04:18.452149+0000","last_undegraded":"2021-03-24T22:04:18.452149+0000","last_fullsized":"2021-03-24T22:04:18.452149+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'4","last_scrub_stamp":"2021-03-24T21:59:06.906705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:59:06.906705+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"544'46","reported_seq":"706","reported_epoch":"687","state":"active+clean+remapped","last_fresh":"2021-03-24T22:04:17.455591+0000","last_change":"2021-03-24T22:03:57.293147+0000","last_active":"2021-03-24T22:04:17.455591+0000","last_peered":"2021-03-24T22:04:17.455591+0000","last_clean":"2021-03-24T22:04:17.455591+0000","last_became_active":"2021-03-24T22:03:56.367578+0000","last_became_peered":"2021-03-24T22:03:56.367578+0000","last_unstale":"2021-03-24T22:04:17.455591+0000","last_undegraded":"2021-03-24T22:04:17.455591+0000","last_fullsized":"2021-03-24T22:04:17.455591+0000","mapping_epoch":665,"log_start":"97'28","ondisk_log_start":"97'28","created":62,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'46","last_scrub_stamp":"2021-03-24T22:03:57.292854+0000","last_deep_scrub":"97'28","last_deep_scrub_stamp":"2021-03-24T21:54:19.157265+0000","last_clean_scrub_stamp":"2021-03-24T22:03:57.292854+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1616,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":20,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":96,"num_read_kb":63,"num_write":53,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":3692,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6,4],"avail_no_missing":["7","4","6"],"object_location_counts":[{"shards":"4,6,7","objects":20}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"89'4","reported_seq":"162","reported_epoch":"687","state":"active+clean","last_fresh":"2021-03-24T22:04:17.457210+0000","last_change":"2021-03-24T22:03:58.240340+0000","last_active":"2021-03-24T22:04:17.457210+0000","last_peered":"2021-03-24T22:04:17.457210+0000","last_clean":"2021-03-24T22:04:17.457210+0000","last_became_active":"2021-03-24T22:01:57.316325+0000","last_became_peered":"2021-03-24T22:01:57.316325+0000","last_unstale":"2021-03-24T22:04:17.457210+0000","last_undegraded":"2021-03-24T22:04:17.457210+0000","last_fullsized":"2021-03-24T22:04:17.457210+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'4","last_scrub_stamp":"2021-03-24T22:03:58.240264+0000","last_deep_scrub":"89'4","last_deep_scrub_stamp":"2021-03-24T21:53:19.926637+0000","last_clean_scrub_stamp":"2021-03-24T22:03:58.240264+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"175","reported_epoch":"689","state":"active+undersized","last_fresh":"2021-03-24T22:04:20.012341+0000","last_change":"2021-03-24T22:03:54.378670+0000","last_active":"2021-03-24T22:04:20.012341+0000","last_peered":"2021-03-24T22:04:20.012341+0000","last_clean":"2021-03-24T22:03:52.357093+0000","last_became_active":"2021-03-24T22:03:54.378670+0000","last_became_peered":"2021-03-24T22:03:54.378670+0000","last_unstale":"2021-03-24T22:04:20.012341+0000","last_undegraded":"2021-03-24T22:04:20.012341+0000","last_fullsized":"2021-03-24T22:03:54.359212+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:01:51.981423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:01:51.981423+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"272'275","reported_seq":"1638","reported_epoch":"688","state":"active+undersized+degraded","last_fresh":"2021-03-24T22:04:18.453349+0000","last_change":"2021-03-24T22:03:54.370761+0000","last_active":"2021-03-24T22:04:18.453349+0000","last_peered":"2021-03-24T22:04:18.453349+0000","last_clean":"2021-03-24T22:03:52.362477+0000","last_became_active":"2021-03-24T22:03:54.370761+0000","last_became_peered":"2021-03-24T22:03:54.370761+0000","last_unstale":"2021-03-24T22:04:18.453349+0000","last_undegraded":"2021-03-24T22:03:54.357176+0000","last_fullsized":"2021-03-24T22:03:54.356899+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":658,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'275","last_scrub_stamp":"2021-03-24T21:58:23.942036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:58:23.942036+0000","log_size":275,"ondisk_log_size":275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":13,"stat_sum":{"num_bytes":8857338,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":114,"num_read_kb":132714,"num_write":52,"num_write_kb":22259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":12587455,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":3}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":"177","reported_epoch":"689","state":"active+undersized","last_fresh":"2021-03-24T22:04:20.010951+0000","last_change":"2021-03-24T22:03:54.376287+0000","last_active":"2021-03-24T22:04:20.010951+0000","last_peered":"2021-03-24T22:04:20.010951+0000","last_clean":"2021-03-24T22:03:52.356719+0000","last_became_active":"2021-03-24T22:03:54.376287+0000","last_became_peered":"2021-03-24T22:03:54.376287+0000","last_unstale":"2021-03-24T22:04:20.010951+0000","last_undegraded":"2021-03-24T22:04:20.010951+0000","last_fullsized":"2021-03-24T22:03:54.360828+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:01:56.876119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:01:56.876119+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"597","reported_epoch":"692","state":"active+undersized","last_fresh":"2021-03-24T22:04:22.393887+0000","last_change":"2021-03-24T22:03:54.393879+0000","last_active":"2021-03-24T22:04:22.393887+0000","last_peered":"2021-03-24T22:04:22.393887+0000","last_clean":"2021-03-24T22:03:43.465188+0000","last_became_active":"2021-03-24T22:03:54.393879+0000","last_became_peered":"2021-03-24T22:03:54.393879+0000","last_unstale":"2021-03-24T22:04:22.393887+0000","last_undegraded":"2021-03-24T22:04:22.393887+0000","last_fullsized":"2021-03-24T22:03:54.362587+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:42.740380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T22:00:26.392232+0000","last_clean_scrub_stamp":"2021-03-24T22:03:42.740380+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"156","reported_epoch":"687","state":"active+clean","last_fresh":"2021-03-24T22:04:17.459196+0000","last_change":"2021-03-24T22:03:01.351580+0000","last_active":"2021-03-24T22:04:17.459196+0000","last_peered":"2021-03-24T22:04:17.459196+0000","last_clean":"2021-03-24T22:04:17.459196+0000","last_became_active":"2021-03-24T22:01:57.312376+0000","last_became_peered":"2021-03-24T22:01:57.312376+0000","last_unstale":"2021-03-24T22:04:17.459196+0000","last_undegraded":"2021-03-24T22:04:17.459196+0000","last_fullsized":"2021-03-24T22:04:17.459196+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:01.351474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:03:01.351474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.14","version":"693'791","reported_seq":"2610","reported_epoch":"693","state":"active+undersized+degraded","last_fresh":"2021-03-24T22:04:23.382793+0000","last_change":"2021-03-24T22:03:54.378573+0000","last_active":"2021-03-24T22:04:23.382793+0000","last_peered":"2021-03-24T22:04:23.382793+0000","last_clean":"2021-03-24T22:03:52.353869+0000","last_became_active":"2021-03-24T22:03:54.378573+0000","last_became_peered":"2021-03-24T22:03:54.378573+0000","last_unstale":"2021-03-24T22:04:23.382793+0000","last_undegraded":"2021-03-24T22:03:54.362577+0000","last_fullsized":"2021-03-24T22:03:54.362412+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":5,"last_scrub":"651'772","last_scrub_stamp":"2021-03-24T22:03:42.250616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:03:42.250616+0000","log_size":791,"ondisk_log_size":791,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":15,"stat_sum":{"num_bytes":49333624,"num_objects":20,"num_object_clones":17,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":15,"num_read_kb":16848,"num_write":11,"num_write_kb":2606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":20}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":"178","reported_epoch":"689","state":"active+clean","last_fresh":"2021-03-24T22:04:20.011990+0000","last_change":"2021-03-24T22:02:22.796277+0000","last_active":"2021-03-24T22:04:20.011990+0000","last_peered":"2021-03-24T22:04:20.011990+0000","last_clean":"2021-03-24T22:04:20.011990+0000","last_became_active":"2021-03-24T22:01:57.304423+0000","last_became_peered":"2021-03-24T22:01:57.304423+0000","last_unstale":"2021-03-24T22:04:20.011990+0000","last_undegraded":"2021-03-24T22:04:20.011990+0000","last_fullsized":"2021-03-24T22:04:20.011990+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:22.796161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:02:22.796161+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"725","reported_epoch":"692","state":"active+clean","last_fresh":"2021-03-24T22:04:22.396259+0000","last_change":"2021-03-24T22:03:45.435984+0000","last_active":"2021-03-24T22:04:22.396259+0000","last_peered":"2021-03-24T22:04:22.396259+0000","last_clean":"2021-03-24T22:04:22.396259+0000","last_became_active":"2021-03-24T22:03:45.434516+0000","last_became_peered":"2021-03-24T22:03:45.434516+0000","last_unstale":"2021-03-24T22:04:22.396259+0000","last_undegraded":"2021-03-24T22:04:22.396259+0000","last_fullsized":"2021-03-24T22:04:22.396259+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:56.523309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:55.493713+0000","last_clean_scrub_stamp":"2021-03-24T22:02:56.523309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"689","reported_epoch":"690","state":"active+clean","last_fresh":"2021-03-24T22:04:20.358128+0000","last_change":"2021-03-24T22:03:09.686125+0000","last_active":"2021-03-24T22:04:20.358128+0000","last_peered":"2021-03-24T22:04:20.358128+0000","last_clean":"2021-03-24T22:04:20.358128+0000","last_became_active":"2021-03-24T22:01:57.312185+0000","last_became_peered":"2021-03-24T22:01:57.312185+0000","last_unstale":"2021-03-24T22:04:20.358128+0000","last_undegraded":"2021-03-24T22:04:20.358128+0000","last_fullsized":"2021-03-24T22:04:20.358128+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:09.686047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:03:09.686047+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"688'288","reported_seq":"2391","reported_epoch":"688","state":"active+clean","last_fresh":"2021-03-24T22:04:18.460859+0000","last_change":"2021-03-24T22:04:18.461336+0000","last_active":"2021-03-24T22:04:18.460859+0000","last_peered":"2021-03-24T22:04:18.460859+0000","last_clean":"2021-03-24T22:04:18.460859+0000","last_became_active":"2021-03-24T22:03:51.363965+0000","last_became_peered":"2021-03-24T22:03:51.363965+0000","last_unstale":"2021-03-24T22:04:18.460859+0000","last_undegraded":"2021-03-24T22:04:18.460859+0000","last_fullsized":"2021-03-24T22:04:18.460859+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":5,"last_scrub":"447'140","last_scrub_stamp":"2021-03-24T22:00:17.991497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:00:17.991497+0000","log_size":288,"ondisk_log_size":288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6708251,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":51,"num_read_kb":65322,"num_write":42,"num_write_kb":15266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":85409227,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1cd","length":"1"},{"start":"1d4","length":"1"},{"start":"1f0","length":"1"},{"start":"1f3","length":"1"},{"start":"20e","length":"1"},{"start":"211","length":"1"},{"start":"218","length":"2"},{"start":"21e","length":"1"},{"start":"220","length":"2"},{"start":"22f","length":"1"},{"start":"232","length":"1"}]},{"pgid":"4.15","version":"544'19","reported_seq":"559","reported_epoch":"687","state":"active+undersized+degraded","last_fresh":"2021-03-24T22:04:17.458408+0000","last_change":"2021-03-24T22:03:54.366573+0000","last_active":"2021-03-24T22:04:17.458408+0000","last_peered":"2021-03-24T22:04:17.458408+0000","last_clean":"2021-03-24T22:01:55.300768+0000","last_became_active":"2021-03-24T22:03:54.366573+0000","last_became_peered":"2021-03-24T22:03:54.366573+0000","last_unstale":"2021-03-24T22:04:17.458408+0000","last_undegraded":"2021-03-24T22:03:54.363208+0000","last_fullsized":"2021-03-24T22:03:54.363041+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'13","last_scrub_stamp":"2021-03-24T22:00:29.462476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:00:29.462476+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":37,"num_read_kb":23,"num_write":20,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2202,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[{"shards":"3,7","objects":8}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":"694","reported_epoch":"692","state":"active+clean","last_fresh":"2021-03-24T22:04:22.391590+0000","last_change":"2021-03-24T22:01:57.316563+0000","last_active":"2021-03-24T22:04:22.391590+0000","last_peered":"2021-03-24T22:04:22.391590+0000","last_clean":"2021-03-24T22:04:22.391590+0000","last_became_active":"2021-03-24T22:01:57.315967+0000","last_became_peered":"2021-03-24T22:01:57.315967+0000","last_unstale":"2021-03-24T22:04:22.391590+0000","last_undegraded":"2021-03-24T22:04:22.391590+0000","last_fullsized":"2021-03-24T22:04:22.391590+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:28.482500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:05.473348+0000","last_clean_scrub_stamp":"2021-03-24T22:00:28.482500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"573'12","reported_seq":"899","reported_epoch":"692","state":"active+clean","last_fresh":"2021-03-24T22:04:22.395373+0000","last_change":"2021-03-24T22:03:02.456113+0000","last_active":"2021-03-24T22:04:22.395373+0000","last_peered":"2021-03-24T22:04:22.395373+0000","last_clean":"2021-03-24T22:04:22.395373+0000","last_became_active":"2021-03-24T22:02:22.326377+0000","last_became_peered":"2021-03-24T22:02:22.326377+0000","last_unstale":"2021-03-24T22:04:22.395373+0000","last_undegraded":"2021-03-24T22:04:22.395373+0000","last_fullsized":"2021-03-24T22:04:22.395373+0000","mapping_epoch":571,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":0,"last_scrub":"573'12","last_scrub_stamp":"2021-03-24T22:03:02.455946+0000","last_deep_scrub":"86'3","last_deep_scrub_stamp":"2021-03-24T21:52:56.678111+0000","last_clean_scrub_stamp":"2021-03-24T22:03:02.455946+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"630","reported_epoch":"690","state":"active+undersized","last_fresh":"2021-03-24T22:04:20.358604+0000","last_change":"2021-03-24T22:03:54.372569+0000","last_active":"2021-03-24T22:04:20.358604+0000","last_peered":"2021-03-24T22:04:20.358604+0000","last_clean":"2021-03-24T22:03:52.503738+0000","last_became_active":"2021-03-24T22:03:54.372569+0000","last_became_peered":"2021-03-24T22:03:54.372569+0000","last_unstale":"2021-03-24T22:04:20.358604+0000","last_undegraded":"2021-03-24T22:04:20.358604+0000","last_fullsized":"2021-03-24T22:03:54.359580+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:59.303897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:00:59.303897+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.16","version":"690'439","reported_seq":"2491","reported_epoch":"692","state":"active+clean","last_fresh":"2021-03-24T22:04:22.408711+0000","last_change":"2021-03-24T22:04:22.408711+0000","last_active":"2021-03-24T22:04:22.408711+0000","last_peered":"2021-03-24T22:04:22.408711+0000","last_clean":"2021-03-24T22:04:22.408711+0000","last_became_active":"2021-03-24T22:02:08.322570+0000","last_became_peered":"2021-03-24T22:02:08.322570+0000","last_unstale":"2021-03-24T22:04:22.408711+0000","last_undegraded":"2021-03-24T22:04:22.408711+0000","last_fullsized":"2021-03-24T22:04:22.408711+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"606'362","last_scrub_stamp":"2021-03-24T22:02:57.478372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:02:57.478372+0000","log_size":439,"ondisk_log_size":439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18537828,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":101,"num_read_kb":117113,"num_write":65,"num_write_kb":16128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":30734442,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1cd","length":"1"},{"start":"1d4","length":"1"},{"start":"1f0","length":"1"},{"start":"1f3","length":"1"},{"start":"20e","length":"1"},{"start":"211","length":"1"},{"start":"214","length":"1"},{"start":"218","length":"2"},{"start":"21e","length":"1"},{"start":"220","length":"2"},{"start":"22f","length":"1"},{"start":"232","length":"1"},{"start":"239","length":"1"}]},{"pgid":"4.14","version":"544'24","reported_seq":"934","reported_epoch":"692","state":"active+clean","last_fresh":"2021-03-24T22:04:22.395457+0000","last_change":"2021-03-24T22:03:46.474982+0000","last_active":"2021-03-24T22:04:22.395457+0000","last_peered":"2021-03-24T22:04:22.395457+0000","last_clean":"2021-03-24T22:04:22.395457+0000","last_became_active":"2021-03-24T22:03:45.433725+0000","last_became_peered":"2021-03-24T22:03:45.433725+0000","last_unstale":"2021-03-24T22:04:22.395457+0000","last_undegraded":"2021-03-24T22:04:22.395457+0000","last_fullsized":"2021-03-24T22:04:22.395457+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'14","last_scrub_stamp":"2021-03-24T21:58:57.435358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T21:58:57.435358+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":42,"num_read_kb":29,"num_write":24,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":4992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":"151","reported_epoch":"687","state":"active+clean","last_fresh":"2021-03-24T22:04:17.456811+0000","last_change":"2021-03-24T22:01:57.320074+0000","last_active":"2021-03-24T22:04:17.456811+0000","last_peered":"2021-03-24T22:04:17.456811+0000","last_clean":"2021-03-24T22:04:17.456811+0000","last_became_active":"2021-03-24T22:01:57.318685+0000","last_became_peered":"2021-03-24T22:01:57.318685+0000","last_unstale":"2021-03-24T22:04:17.456811+0000","last_undegraded":"2021-03-24T22:04:17.456811+0000","last_fullsized":"2021-03-24T22:04:17.456811+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:15.949538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:59:15.949538+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"664'11","reported_seq":"720","reported_epoch":"688","state":"active+undersized+degraded","last_fresh":"2021-03-24T22:04:19.301718+0000","last_change":"2021-03-24T22:03:54.399120+0000","last_active":"2021-03-24T22:04:19.301718+0000","last_peered":"2021-03-24T22:04:19.301718+0000","last_clean":"2021-03-24T22:03:44.276751+0000","last_became_active":"2021-03-24T22:03:54.399120+0000","last_became_peered":"2021-03-24T22:03:54.399120+0000","last_unstale":"2021-03-24T22:04:19.301718+0000","last_undegraded":"2021-03-24T22:03:54.366983+0000","last_fullsized":"2021-03-24T22:03:54.357904+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"445'6","last_scrub_stamp":"2021-03-24T22:00:24.314895+0000","last_deep_scrub":"86'3","last_deep_scrub_stamp":"2021-03-24T21:53:08.815163+0000","last_clean_scrub_stamp":"2021-03-24T22:00:24.314895+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"697","reported_epoch":"692","state":"active+clean","last_fresh":"2021-03-24T22:04:22.393908+0000","last_change":"2021-03-24T22:03:57.076812+0000","last_active":"2021-03-24T22:04:22.393908+0000","last_peered":"2021-03-24T22:04:22.393908+0000","last_clean":"2021-03-24T22:04:22.393908+0000","last_became_active":"2021-03-24T22:01:57.316729+0000","last_became_peered":"2021-03-24T22:01:57.316729+0000","last_unstale":"2021-03-24T22:04:22.393908+0000","last_undegraded":"2021-03-24T22:04:22.393908+0000","last_fullsized":"2021-03-24T22:04:22.393908+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:57.076558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:03:57.076558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.17","version":"688'1021","reported_seq":"3145","reported_epoch":"688","state":"active+undersized+degraded","last_fresh":"2021-03-24T22:04:18.493494+0000","last_change":"2021-03-24T22:03:54.378784+0000","last_active":"2021-03-24T22:04:18.493494+0000","last_peered":"2021-03-24T22:04:18.493494+0000","last_clean":"2021-03-24T22:03:52.506482+0000","last_became_active":"2021-03-24T22:03:54.378784+0000","last_became_peered":"2021-03-24T22:03:54.378784+0000","last_unstale":"2021-03-24T22:04:18.493494+0000","last_undegraded":"2021-03-24T22:03:54.362415+0000","last_fullsized":"2021-03-24T22:03:54.361409+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":5,"last_scrub":"452'567","last_scrub_stamp":"2021-03-24T22:00:27.169710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:00:27.169710+0000","log_size":1021,"ondisk_log_size":1021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":13,"stat_sum":{"num_bytes":51190156,"num_objects":24,"num_object_clones":20,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":282,"num_read_kb":300163,"num_write":187,"num_write_kb":66183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":242,"num_bytes_recovered":147823286,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":24}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"584'32","reported_seq":"268","reported_epoch":"689","state":"active+clean","last_fresh":"2021-03-24T22:04:20.012100+0000","last_change":"2021-03-24T22:03:46.201676+0000","last_active":"2021-03-24T22:04:20.012100+0000","last_peered":"2021-03-24T22:04:20.012100+0000","last_clean":"2021-03-24T22:04:20.012100+0000","last_became_active":"2021-03-24T22:03:45.426065+0000","last_became_peered":"2021-03-24T22:03:45.426065+0000","last_unstale":"2021-03-24T22:04:20.012100+0000","last_undegraded":"2021-03-24T22:04:20.012100+0000","last_fullsized":"2021-03-24T22:04:20.012100+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"584'32","last_scrub_stamp":"2021-03-24T22:02:34.813628+0000","last_deep_scrub":"584'32","last_deep_scrub_stamp":"2021-03-24T22:02:34.813628+0000","last_clean_scrub_stamp":"2021-03-24T22:02:34.813628+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":29,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":3680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":"158","reported_epoch":"687","state":"active+clean","last_fresh":"2021-03-24T22:04:17.457894+0000","last_change":"2021-03-24T22:03:54.320264+0000","last_active":"2021-03-24T22:04:17.457894+0000","last_peered":"2021-03-24T22:04:17.457894+0000","last_clean":"2021-03-24T22:04:17.457894+0000","last_became_active":"2021-03-24T22:01:57.307018+0000","last_became_peered":"2021-03-24T22:01:57.307018+0000","last_unstale":"2021-03-24T22:04:17.457894+0000","last_undegraded":"2021-03-24T22:04:17.457894+0000","last_fullsized":"2021-03-24T22:04:17.457894+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:54.320187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:03:54.320187+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"50","reported_epoch":"688","state":"active+undersized","last_fresh":"2021-03-24T22:04:18.453802+0000","last_change":"2021-03-24T22:03:54.395916+0000","last_active":"2021-03-24T22:04:18.453802+0000","last_peered":"2021-03-24T22:04:18.453802+0000","last_clean":"2021-03-24T22:03:52.362042+0000","last_became_active":"2021-03-24T22:03:54.395916+0000","last_became_peered":"2021-03-24T22:03:54.395916+0000","last_unstale":"2021-03-24T22:04:18.453802+0000","last_undegraded":"2021-03-24T22:04:18.453802+0000","last_fullsized":"2021-03-24T22:03:54.364541+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:01:55.618238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:57:31.118050+0000","last_clean_scrub_stamp":"2021-03-24T22:01:55.618238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"667","reported_epoch":"692","state":"active+undersized","last_fresh":"2021-03-24T22:04:22.394057+0000","last_change":"2021-03-24T22:03:54.373686+0000","last_active":"2021-03-24T22:04:22.394057+0000","last_peered":"2021-03-24T22:04:22.394057+0000","last_clean":"2021-03-24T22:03:52.352390+0000","last_became_active":"2021-03-24T22:03:54.373686+0000","last_became_peered":"2021-03-24T22:03:54.373686+0000","last_unstale":"2021-03-24T22:04:22.394057+0000","last_undegraded":"2021-03-24T22:04:22.394057+0000","last_fullsized":"2021-03-24T22:03:54.360228+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:21.298330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:23.707914+0000","last_clean_scrub_stamp":"2021-03-24T22:00:21.298330+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.10","version":"689'148","reported_seq":"496","reported_epoch":"692","state":"active+undersized+degraded","last_fresh":"2021-03-24T22:04:22.390742+0000","last_change":"2021-03-24T22:03:54.394563+0000","last_active":"2021-03-24T22:04:22.390742+0000","last_peered":"2021-03-24T22:04:22.390742+0000","last_clean":"2021-03-24T22:03:52.351563+0000","last_became_active":"2021-03-24T22:03:54.394563+0000","last_became_peered":"2021-03-24T22:03:54.394563+0000","last_unstale":"2021-03-24T22:04:22.390742+0000","last_undegraded":"2021-03-24T22:03:54.359975+0000","last_fullsized":"2021-03-24T22:03:54.359799+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":5,"last_scrub":"579'81","last_scrub_stamp":"2021-03-24T22:02:30.763305+0000","last_deep_scrub":"277'24","last_deep_scrub_stamp":"2021-03-24T21:57:28.119420+0000","last_clean_scrub_stamp":"2021-03-24T22:02:30.763305+0000","log_size":148,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":15,"stat_sum":{"num_bytes":12580036,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":77,"num_read_kb":73530,"num_write":47,"num_write_kb":14209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":29353764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"544'21","reported_seq":"66","reported_epoch":"688","state":"active+clean","last_fresh":"2021-03-24T22:04:18.454108+0000","last_change":"2021-03-24T22:03:45.453450+0000","last_active":"2021-03-24T22:04:18.454108+0000","last_peered":"2021-03-24T22:04:18.454108+0000","last_clean":"2021-03-24T22:04:18.454108+0000","last_became_active":"2021-03-24T22:03:45.427113+0000","last_became_peered":"2021-03-24T22:03:45.427113+0000","last_unstale":"2021-03-24T22:04:18.454108+0000","last_undegraded":"2021-03-24T22:04:18.454108+0000","last_fullsized":"2021-03-24T22:04:18.454108+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'21","last_scrub_stamp":"2021-03-24T22:02:18.706842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:02:18.706842+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":43,"num_read_kb":25,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":3150,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":"156","reported_epoch":"687","state":"active+clean","last_fresh":"2021-03-24T22:04:17.459991+0000","last_change":"2021-03-24T22:03:09.417640+0000","last_active":"2021-03-24T22:04:17.459991+0000","last_peered":"2021-03-24T22:04:17.459991+0000","last_clean":"2021-03-24T22:04:17.459991+0000","last_became_active":"2021-03-24T22:01:57.318513+0000","last_became_peered":"2021-03-24T22:01:57.318513+0000","last_unstale":"2021-03-24T22:04:17.459991+0000","last_undegraded":"2021-03-24T22:04:17.459991+0000","last_fullsized":"2021-03-24T22:04:17.459991+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:09.417465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T22:03:09.417465+0000","last_clean_scrub_stamp":"2021-03-24T22:03:09.417465+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"699","reported_epoch":"692","state":"active+clean","last_fresh":"2021-03-24T22:04:22.393296+0000","last_change":"2021-03-24T22:03:45.435731+0000","last_active":"2021-03-24T22:04:22.393296+0000","last_peered":"2021-03-24T22:04:22.393296+0000","last_clean":"2021-03-24T22:04:22.393296+0000","last_became_active":"2021-03-24T22:03:45.435094+0000","last_became_peered":"2021-03-24T22:03:45.435094+0000","last_unstale":"2021-03-24T22:04:22.393296+0000","last_undegraded":"2021-03-24T22:04:22.393296+0000","last_fullsized":"2021-03-24T22:04:22.393296+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:28.289244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:00:28.289244+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"709","reported_epoch":"692","state":"active+undersized","last_fresh":"2021-03-24T22:04:22.392503+0000","last_change":"2021-03-24T22:03:54.392957+0000","last_active":"2021-03-24T22:04:22.392503+0000","last_peered":"2021-03-24T22:04:22.392503+0000","last_clean":"2021-03-24T22:03:52.357508+0000","last_became_active":"2021-03-24T22:03:54.392957+0000","last_became_peered":"2021-03-24T22:03:54.392957+0000","last_unstale":"2021-03-24T22:04:22.392503+0000","last_undegraded":"2021-03-24T22:04:22.392503+0000","last_fullsized":"2021-03-24T22:03:54.370297+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:01:52.461594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:01:52.461594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.11","version":"685'925","reported_seq":"3087","reported_epoch":"688","state":"active+clean","last_fresh":"2021-03-24T22:04:18.453975+0000","last_change":"2021-03-24T22:04:18.453975+0000","last_active":"2021-03-24T22:04:18.453975+0000","last_peered":"2021-03-24T22:04:18.453975+0000","last_clean":"2021-03-24T22:04:18.453975+0000","last_became_active":"2021-03-24T22:02:04.322694+0000","last_became_peered":"2021-03-24T22:02:04.322694+0000","last_unstale":"2021-03-24T22:04:18.453975+0000","last_undegraded":"2021-03-24T22:04:18.453975+0000","last_fullsized":"2021-03-24T22:04:18.453975+0000","mapping_epoch":553,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":554,"parent":"0.0","parent_split_bits":5,"last_scrub":"381'425","last_scrub_stamp":"2021-03-24T21:59:11.936662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:59:11.936662+0000","log_size":925,"ondisk_log_size":925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34365166,"num_objects":13,"num_object_clones":10,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":165,"num_read_kb":185501,"num_write":129,"num_write_kb":51562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":325,"num_bytes_recovered":148467814,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1cd","length":"1"},{"start":"1d4","length":"1"},{"start":"1f0","length":"1"},{"start":"1f3","length":"1"},{"start":"20e","length":"1"},{"start":"211","length":"1"},{"start":"218","length":"2"},{"start":"21e","length":"1"},{"start":"220","length":"2"},{"start":"22f","length":"1"},{"start":"232","length":"1"}]},{"pgid":"4.11","version":"544'30","reported_seq":"874","reported_epoch":"692","state":"active+clean","last_fresh":"2021-03-24T22:04:22.391620+0000","last_change":"2021-03-24T22:01:58.012725+0000","last_active":"2021-03-24T22:04:22.391620+0000","last_peered":"2021-03-24T22:04:22.391620+0000","last_clean":"2021-03-24T22:04:22.391620+0000","last_became_active":"2021-03-24T22:01:57.310370+0000","last_became_peered":"2021-03-24T22:01:57.310370+0000","last_unstale":"2021-03-24T22:04:22.391620+0000","last_undegraded":"2021-03-24T22:04:22.391620+0000","last_fullsized":"2021-03-24T22:04:22.391620+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'22","last_scrub_stamp":"2021-03-24T22:00:13.347155+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2021-03-24T21:52:54.597505+0000","last_clean_scrub_stamp":"2021-03-24T22:00:13.347155+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":41,"num_read_kb":28,"num_write":23,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":3480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":"594","reported_epoch":"688","state":"active+undersized","last_fresh":"2021-03-24T22:04:18.456818+0000","last_change":"2021-03-24T22:03:54.367081+0000","last_active":"2021-03-24T22:04:18.456818+0000","last_peered":"2021-03-24T22:04:18.456818+0000","last_clean":"2021-03-24T22:03:43.465856+0000","last_became_active":"2021-03-24T22:03:54.367081+0000","last_became_peered":"2021-03-24T22:03:54.367081+0000","last_unstale":"2021-03-24T22:04:18.456818+0000","last_undegraded":"2021-03-24T22:04:18.456818+0000","last_fullsized":"2021-03-24T22:03:54.357747+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:18.082513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:02:18.082513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"151","reported_epoch":"687","state":"active+clean","last_fresh":"2021-03-24T22:04:17.455981+0000","last_change":"2021-03-24T22:01:57.308893+0000","last_active":"2021-03-24T22:04:17.455981+0000","last_peered":"2021-03-24T22:04:17.455981+0000","last_clean":"2021-03-24T22:04:17.455981+0000","last_became_active":"2021-03-24T22:01:57.307995+0000","last_became_peered":"2021-03-24T22:01:57.307995+0000","last_unstale":"2021-03-24T22:04:17.455981+0000","last_undegraded":"2021-03-24T22:04:17.455981+0000","last_fullsized":"2021-03-24T22:04:17.455981+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:41.839852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T21:59:41.839852+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"664","reported_epoch":"692","state":"active+clean","last_fresh":"2021-03-24T22:04:22.395824+0000","last_change":"2021-03-24T22:01:43.389560+0000","last_active":"2021-03-24T22:04:22.395824+0000","last_peered":"2021-03-24T22:04:22.395824+0000","last_clean":"2021-03-24T22:04:22.395824+0000","last_became_active":"2021-03-24T22:01:43.389017+0000","last_became_peered":"2021-03-24T22:01:43.389017+0000","last_unstale":"2021-03-24T22:04:22.395824+0000","last_undegraded":"2021-03-24T22:04:22.395824+0000","last_fullsized":"2021-03-24T22:04:22.395824+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:15.289444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:47.540793+0000","last_clean_scrub_stamp":"2021-03-24T21:59:15.289444+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.12","version":"183'26","reported_seq":"499","reported_epoch":"690","state":"active+clean","last_fresh":"2021-03-24T22:04:20.364006+0000","last_change":"2021-03-24T22:04:20.365047+0000","last_active":"2021-03-24T22:04:20.364006+0000","last_peered":"2021-03-24T22:04:20.364006+0000","last_clean":"2021-03-24T22:04:20.364006+0000","last_became_active":"2021-03-24T22:01:43.392614+0000","last_became_peered":"2021-03-24T22:01:43.392614+0000","last_unstale":"2021-03-24T22:04:20.364006+0000","last_undegraded":"2021-03-24T22:04:20.364006+0000","last_fullsized":"2021-03-24T22:04:20.364006+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":5,"last_scrub":"183'26","last_scrub_stamp":"2021-03-24T22:02:21.766090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:02:21.766090+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1cd","length":"1"},{"start":"1d4","length":"1"},{"start":"1f0","length":"1"},{"start":"1f3","length":"1"},{"start":"20e","length":"1"},{"start":"211","length":"1"},{"start":"214","length":"1"},{"start":"218","length":"2"},{"start":"21e","length":"1"},{"start":"220","length":"2"},{"start":"22f","length":"1"},{"start":"232","length":"1"}]},{"pgid":"4.10","version":"544'16","reported_seq":"948","reported_epoch":"692","state":"active+clean","last_fresh":"2021-03-24T22:04:22.391407+0000","last_change":"2021-03-24T22:03:46.473385+0000","last_active":"2021-03-24T22:04:22.391407+0000","last_peered":"2021-03-24T22:04:22.391407+0000","last_clean":"2021-03-24T22:04:22.391407+0000","last_became_active":"2021-03-24T22:03:45.428746+0000","last_became_peered":"2021-03-24T22:03:45.428746+0000","last_unstale":"2021-03-24T22:04:22.391407+0000","last_undegraded":"2021-03-24T22:04:22.391407+0000","last_fullsized":"2021-03-24T22:04:22.391407+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'12","last_scrub_stamp":"2021-03-24T22:00:29.255353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:00:29.255353+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":36,"num_read_kb":18,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":10016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":"679","reported_epoch":"692","state":"active+undersized","last_fresh":"2021-03-24T22:04:22.399270+0000","last_change":"2021-03-24T22:03:54.387298+0000","last_active":"2021-03-24T22:04:22.399270+0000","last_peered":"2021-03-24T22:04:22.399270+0000","last_clean":"2021-03-24T22:03:52.359541+0000","last_became_active":"2021-03-24T22:03:54.387298+0000","last_became_peered":"2021-03-24T22:03:54.387298+0000","last_unstale":"2021-03-24T22:04:22.399270+0000","last_undegraded":"2021-03-24T22:04:22.399270+0000","last_fullsized":"2021-03-24T22:03:54.371107+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:01:56.417662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:01:56.417662+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"180","reported_epoch":"689","state":"active+clean","last_fresh":"2021-03-24T22:04:20.011393+0000","last_change":"2021-03-24T22:02:27.831482+0000","last_active":"2021-03-24T22:04:20.011393+0000","last_peered":"2021-03-24T22:04:20.011393+0000","last_clean":"2021-03-24T22:04:20.011393+0000","last_became_active":"2021-03-24T22:01:57.304385+0000","last_became_peered":"2021-03-24T22:01:57.304385+0000","last_unstale":"2021-03-24T22:04:20.011393+0000","last_undegraded":"2021-03-24T22:04:20.011393+0000","last_fullsized":"2021-03-24T22:04:20.011393+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:27.831383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:02:27.831383+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"158","reported_epoch":"687","state":"active+clean","last_fresh":"2021-03-24T22:04:17.456276+0000","last_change":"2021-03-24T22:03:08.379727+0000","last_active":"2021-03-24T22:04:17.456276+0000","last_peered":"2021-03-24T22:04:17.456276+0000","last_clean":"2021-03-24T22:04:17.456276+0000","last_became_active":"2021-03-24T22:01:57.315718+0000","last_became_peered":"2021-03-24T22:01:57.315718+0000","last_unstale":"2021-03-24T22:04:17.456276+0000","last_undegraded":"2021-03-24T22:04:17.456276+0000","last_fullsized":"2021-03-24T22:04:17.456276+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:08.379591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:57:59.979772+0000","last_clean_scrub_stamp":"2021-03-24T22:03:08.379591+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.13","version":"692'610","reported_seq":"2940","reported_epoch":"692","state":"active+clean","last_fresh":"2021-03-24T22:04:22.406696+0000","last_change":"2021-03-24T22:04:22.406696+0000","last_active":"2021-03-24T22:04:22.406696+0000","last_peered":"2021-03-24T22:04:22.406696+0000","last_clean":"2021-03-24T22:04:22.406696+0000","last_became_active":"2021-03-24T22:02:28.318669+0000","last_became_peered":"2021-03-24T22:02:28.318669+0000","last_unstale":"2021-03-24T22:04:22.406696+0000","last_undegraded":"2021-03-24T22:04:22.406696+0000","last_fullsized":"2021-03-24T22:04:22.406696+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":610,"ondisk_log_size":610,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16007673,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":217,"num_read_kb":201460,"num_write":198,"num_write_kb":75648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":204,"num_bytes_recovered":118383596,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1cd","length":"1"},{"start":"1d4","length":"1"},{"start":"1f0","length":"1"},{"start":"1f3","length":"1"},{"start":"20e","length":"1"},{"start":"211","length":"1"},{"start":"214","length":"1"},{"start":"218","length":"2"},{"start":"21e","length":"1"},{"start":"220","length":"2"},{"start":"22f","length":"1"},{"start":"232","length":"1"},{"start":"239","length":"1"}]},{"pgid":"3.18","version":"0'0","reported_seq":"688","reported_epoch":"692","state":"active+clean","last_fresh":"2021-03-24T22:04:22.391526+0000","last_change":"2021-03-24T22:03:45.429118+0000","last_active":"2021-03-24T22:04:22.391526+0000","last_peered":"2021-03-24T22:04:22.391526+0000","last_clean":"2021-03-24T22:04:22.391526+0000","last_became_active":"2021-03-24T22:03:45.428772+0000","last_became_peered":"2021-03-24T22:03:45.428772+0000","last_unstale":"2021-03-24T22:04:22.391526+0000","last_undegraded":"2021-03-24T22:04:22.391526+0000","last_fullsized":"2021-03-24T22:04:22.391526+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:00.393742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:59:00.393742+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"712","reported_epoch":"692","state":"active+clean","last_fresh":"2021-03-24T22:04:22.398363+0000","last_change":"2021-03-24T22:02:22.328403+0000","last_active":"2021-03-24T22:04:22.398363+0000","last_peered":"2021-03-24T22:04:22.398363+0000","last_clean":"2021-03-24T22:04:22.398363+0000","last_became_active":"2021-03-24T22:02:22.328021+0000","last_became_peered":"2021-03-24T22:02:22.328021+0000","last_unstale":"2021-03-24T22:04:22.398363+0000","last_undegraded":"2021-03-24T22:04:22.398363+0000","last_fullsized":"2021-03-24T22:04:22.398363+0000","mapping_epoch":571,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:17.040623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:59:17.040623+0000","last_clean_scrub_stamp":"2021-03-24T21:59:17.040623+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"50","reported_epoch":"688","state":"active+undersized","last_fresh":"2021-03-24T22:04:18.455545+0000","last_change":"2021-03-24T22:03:54.381826+0000","last_active":"2021-03-24T22:04:18.455545+0000","last_peered":"2021-03-24T22:04:18.455545+0000","last_clean":"2021-03-24T22:03:52.364461+0000","last_became_active":"2021-03-24T22:03:54.381826+0000","last_became_peered":"2021-03-24T22:03:54.381826+0000","last_unstale":"2021-03-24T22:04:18.455545+0000","last_undegraded":"2021-03-24T22:04:18.455545+0000","last_fullsized":"2021-03-24T22:03:54.355081+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:06.420794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:03:06.420794+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1c","version":"273'245","reported_seq":"2103","reported_epoch":"692","state":"active+undersized+degraded","last_fresh":"2021-03-24T22:04:22.394717+0000","last_change":"2021-03-24T22:03:54.394005+0000","last_active":"2021-03-24T22:04:22.394717+0000","last_peered":"2021-03-24T22:04:22.394717+0000","last_clean":"2021-03-24T22:03:52.354015+0000","last_became_active":"2021-03-24T22:03:54.394005+0000","last_became_peered":"2021-03-24T22:03:54.394005+0000","last_unstale":"2021-03-24T22:04:22.394717+0000","last_undegraded":"2021-03-24T22:03:54.363131+0000","last_fullsized":"2021-03-24T22:03:54.362958+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":658,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":245,"ondisk_log_size":245,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":15,"stat_sum":{"num_bytes":8581786,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":70,"num_read_kb":67597,"num_write":52,"num_write_kb":18055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":"44","reported_epoch":"688","state":"active+clean","last_fresh":"2021-03-24T22:04:18.449682+0000","last_change":"2021-03-24T22:03:45.430996+0000","last_active":"2021-03-24T22:04:18.449682+0000","last_peered":"2021-03-24T22:04:18.449682+0000","last_clean":"2021-03-24T22:04:18.449682+0000","last_became_active":"2021-03-24T22:03:45.430602+0000","last_became_peered":"2021-03-24T22:03:45.430602+0000","last_unstale":"2021-03-24T22:04:18.449682+0000","last_undegraded":"2021-03-24T22:04:18.449682+0000","last_fullsized":"2021-03-24T22:04:18.449682+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:01:40.788778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:57:26.417613+0000","last_clean_scrub_stamp":"2021-03-24T22:01:40.788778+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"613'8","reported_seq":"218","reported_epoch":"689","state":"active+clean","last_fresh":"2021-03-24T22:04:20.014172+0000","last_change":"2021-03-24T22:03:07.709341+0000","last_active":"2021-03-24T22:04:20.014172+0000","last_peered":"2021-03-24T22:04:20.014172+0000","last_clean":"2021-03-24T22:04:20.014172+0000","last_became_active":"2021-03-24T22:03:03.353426+0000","last_became_peered":"2021-03-24T22:03:03.353426+0000","last_unstale":"2021-03-24T22:04:20.014172+0000","last_undegraded":"2021-03-24T22:04:20.014172+0000","last_fullsized":"2021-03-24T22:04:20.014172+0000","mapping_epoch":612,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":613,"parent":"0.0","parent_split_bits":0,"last_scrub":"613'8","last_scrub_stamp":"2021-03-24T22:03:07.709086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:03:07.709086+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"457","reported_epoch":"690","state":"active+undersized","last_fresh":"2021-03-24T22:04:20.357480+0000","last_change":"2021-03-24T22:03:54.376062+0000","last_active":"2021-03-24T22:04:20.357480+0000","last_peered":"2021-03-24T22:04:20.357480+0000","last_clean":"2021-03-24T22:03:52.503978+0000","last_became_active":"2021-03-24T22:03:54.376062+0000","last_became_peered":"2021-03-24T22:03:54.376062+0000","last_unstale":"2021-03-24T22:04:20.357480+0000","last_undegraded":"2021-03-24T22:04:20.357480+0000","last_fullsized":"2021-03-24T22:03:54.362499+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:01:53.591151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:01:53.591151+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1d","version":"685'620","reported_seq":"2822","reported_epoch":"689","state":"active+clean","last_fresh":"2021-03-24T22:04:20.019797+0000","last_change":"2021-03-24T22:04:18.457182+0000","last_active":"2021-03-24T22:04:20.019797+0000","last_peered":"2021-03-24T22:04:20.019797+0000","last_clean":"2021-03-24T22:04:20.019797+0000","last_became_active":"2021-03-24T22:03:53.384369+0000","last_became_peered":"2021-03-24T22:03:53.384369+0000","last_unstale":"2021-03-24T22:04:20.019797+0000","last_undegraded":"2021-03-24T22:04:20.019797+0000","last_fullsized":"2021-03-24T22:04:20.019797+0000","mapping_epoch":662,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":663,"parent":"0.0","parent_split_bits":5,"last_scrub":"233'169","last_scrub_stamp":"2021-03-24T21:56:47.285648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:56:47.285648+0000","log_size":620,"ondisk_log_size":620,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30345018,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":204,"num_read_kb":220562,"num_write":143,"num_write_kb":71042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":219,"num_bytes_recovered":119771193,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1cd","length":"1"},{"start":"1d4","length":"1"},{"start":"1f0","length":"1"},{"start":"1f3","length":"1"},{"start":"20e","length":"1"},{"start":"211","length":"1"},{"start":"218","length":"2"},{"start":"21e","length":"1"},{"start":"220","length":"2"},{"start":"22f","length":"1"},{"start":"232","length":"1"}]},{"pgid":"3.1a","version":"0'0","reported_seq":"44","reported_epoch":"688","state":"active+clean","last_fresh":"2021-03-24T22:04:18.453929+0000","last_change":"2021-03-24T22:03:45.432700+0000","last_active":"2021-03-24T22:04:18.453929+0000","last_peered":"2021-03-24T22:04:18.453929+0000","last_clean":"2021-03-24T22:04:18.453929+0000","last_became_active":"2021-03-24T22:03:45.431828+0000","last_became_peered":"2021-03-24T22:03:45.431828+0000","last_unstale":"2021-03-24T22:04:18.453929+0000","last_undegraded":"2021-03-24T22:04:18.453929+0000","last_fullsized":"2021-03-24T22:04:18.453929+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:40.584694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:52:54.212459+0000","last_clean_scrub_stamp":"2021-03-24T22:03:40.584694+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"230","reported_epoch":"692","state":"active+clean","last_fresh":"2021-03-24T22:04:22.395148+0000","last_change":"2021-03-24T22:02:17.672555+0000","last_active":"2021-03-24T22:04:22.395148+0000","last_peered":"2021-03-24T22:04:22.395148+0000","last_clean":"2021-03-24T22:04:22.395148+0000","last_became_active":"2021-03-24T22:00:49.305569+0000","last_became_peered":"2021-03-24T22:00:49.305569+0000","last_unstale":"2021-03-24T22:04:22.395148+0000","last_undegraded":"2021-03-24T22:04:22.395148+0000","last_fullsized":"2021-03-24T22:04:22.395148+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:17.672325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:02:17.672325+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"703","reported_epoch":"692","state":"active+undersized","last_fresh":"2021-03-24T22:04:22.393934+0000","last_change":"2021-03-24T22:03:54.372359+0000","last_active":"2021-03-24T22:04:22.393934+0000","last_peered":"2021-03-24T22:04:22.393934+0000","last_clean":"2021-03-24T22:03:52.356808+0000","last_became_active":"2021-03-24T22:03:54.372359+0000","last_became_peered":"2021-03-24T22:03:54.372359+0000","last_unstale":"2021-03-24T22:04:22.393934+0000","last_undegraded":"2021-03-24T22:04:22.393934+0000","last_fullsized":"2021-03-24T22:03:54.366027+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:55.309332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:00:55.309332+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1e","version":"265'45","reported_seq":"60","reported_epoch":"688","state":"active+undersized","last_fresh":"2021-03-24T22:04:18.457400+0000","last_change":"2021-03-24T22:03:54.395190+0000","last_active":"2021-03-24T22:04:18.457400+0000","last_peered":"2021-03-24T22:04:18.457400+0000","last_clean":"2021-03-24T22:03:52.363791+0000","last_became_active":"2021-03-24T22:03:54.395190+0000","last_became_peered":"2021-03-24T22:03:54.395190+0000","last_unstale":"2021-03-24T22:04:18.457400+0000","last_undegraded":"2021-03-24T22:04:18.457400+0000","last_fullsized":"2021-03-24T22:03:54.363808+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":5,"last_scrub":"199'30","last_scrub_stamp":"2021-03-24T21:56:10.105084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:56:10.105084+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":13,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":18305,"num_write":13,"num_write_kb":3060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":"712","reported_epoch":"692","state":"active+clean","last_fresh":"2021-03-24T22:04:22.391591+0000","last_change":"2021-03-24T22:03:32.381214+0000","last_active":"2021-03-24T22:04:22.391591+0000","last_peered":"2021-03-24T22:04:22.391591+0000","last_clean":"2021-03-24T22:04:22.391591+0000","last_became_active":"2021-03-24T22:03:32.380554+0000","last_became_peered":"2021-03-24T22:03:32.380554+0000","last_unstale":"2021-03-24T22:04:22.391591+0000","last_undegraded":"2021-03-24T22:04:22.391591+0000","last_fullsized":"2021-03-24T22:04:22.391591+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:54.682649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:00:54.682649+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"44","reported_epoch":"688","state":"active+clean","last_fresh":"2021-03-24T22:04:18.455331+0000","last_change":"2021-03-24T22:03:45.433760+0000","last_active":"2021-03-24T22:04:18.455331+0000","last_peered":"2021-03-24T22:04:18.455331+0000","last_clean":"2021-03-24T22:04:18.455331+0000","last_became_active":"2021-03-24T22:03:45.433015+0000","last_became_peered":"2021-03-24T22:03:45.433015+0000","last_unstale":"2021-03-24T22:04:18.455331+0000","last_undegraded":"2021-03-24T22:04:18.455331+0000","last_fullsized":"2021-03-24T22:04:18.455331+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:01:51.757809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:01:51.757809+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"170","reported_epoch":"689","state":"active+undersized","last_fresh":"2021-03-24T22:04:20.013187+0000","last_change":"2021-03-24T22:03:54.367941+0000","last_active":"2021-03-24T22:04:20.013187+0000","last_peered":"2021-03-24T22:04:20.013187+0000","last_clean":"2021-03-24T22:03:52.359430+0000","last_became_active":"2021-03-24T22:03:54.367941+0000","last_became_peered":"2021-03-24T22:03:54.367941+0000","last_unstale":"2021-03-24T22:04:20.013187+0000","last_undegraded":"2021-03-24T22:04:20.013187+0000","last_fullsized":"2021-03-24T22:03:54.359587+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:26.883399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:00:26.883399+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1f","version":"272'99","reported_seq":"499","reported_epoch":"690","state":"active+clean","last_fresh":"2021-03-24T22:04:20.388560+0000","last_change":"2021-03-24T22:04:20.388980+0000","last_active":"2021-03-24T22:04:20.388560+0000","last_peered":"2021-03-24T22:04:20.388560+0000","last_clean":"2021-03-24T22:04:20.388560+0000","last_became_active":"2021-03-24T22:01:43.385324+0000","last_became_peered":"2021-03-24T22:01:43.385324+0000","last_unstale":"2021-03-24T22:04:20.388560+0000","last_undegraded":"2021-03-24T22:04:20.388560+0000","last_fullsized":"2021-03-24T22:04:20.388560+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'99","last_scrub_stamp":"2021-03-24T22:03:57.739288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:03:57.739288+0000","log_size":99,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1cd","length":"1"},{"start":"1d4","length":"1"},{"start":"1f0","length":"1"},{"start":"1f3","length":"1"},{"start":"20e","length":"1"},{"start":"211","length":"1"},{"start":"214","length":"1"},{"start":"218","length":"2"},{"start":"21e","length":"1"},{"start":"220","length":"2"},{"start":"22f","length":"1"},{"start":"232","length":"1"}]},{"pgid":"3.1c","version":"0'0","reported_seq":"50","reported_epoch":"688","state":"active+undersized","last_fresh":"2021-03-24T22:04:18.452376+0000","last_change":"2021-03-24T22:03:54.381276+0000","last_active":"2021-03-24T22:04:18.452376+0000","last_peered":"2021-03-24T22:04:18.452376+0000","last_clean":"2021-03-24T22:03:52.362950+0000","last_became_active":"2021-03-24T22:03:54.381276+0000","last_became_peered":"2021-03-24T22:03:54.381276+0000","last_unstale":"2021-03-24T22:04:18.452376+0000","last_undegraded":"2021-03-24T22:04:18.452376+0000","last_fullsized":"2021-03-24T22:03:54.355145+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:29.855897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:00:29.855897+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"169","reported_epoch":"687","state":"active+clean","last_fresh":"2021-03-24T22:04:17.456694+0000","last_change":"2021-03-24T22:03:45.434678+0000","last_active":"2021-03-24T22:04:17.456694+0000","last_peered":"2021-03-24T22:04:17.456694+0000","last_clean":"2021-03-24T22:04:17.456694+0000","last_became_active":"2021-03-24T22:03:45.434093+0000","last_became_peered":"2021-03-24T22:03:45.434093+0000","last_unstale":"2021-03-24T22:04:17.456694+0000","last_undegraded":"2021-03-24T22:04:17.456694+0000","last_fullsized":"2021-03-24T22:04:17.456694+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:08.942247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:56:51.977306+0000","last_clean_scrub_stamp":"2021-03-24T21:59:08.942247+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"567","reported_epoch":"688","state":"active+undersized","last_fresh":"2021-03-24T22:04:18.449482+0000","last_change":"2021-03-24T22:03:54.380961+0000","last_active":"2021-03-24T22:04:18.449482+0000","last_peered":"2021-03-24T22:04:18.449482+0000","last_clean":"2021-03-24T22:03:43.465255+0000","last_became_active":"2021-03-24T22:03:54.380961+0000","last_became_peered":"2021-03-24T22:03:54.380961+0000","last_unstale":"2021-03-24T22:04:18.449482+0000","last_undegraded":"2021-03-24T22:04:18.449482+0000","last_fullsized":"2021-03-24T22:03:54.356292+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:01:41.968063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:01:41.968063+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.18","version":"624'193","reported_seq":"1825","reported_epoch":"692","state":"active+clean","last_fresh":"2021-03-24T22:04:22.395887+0000","last_change":"2021-03-24T22:04:22.395887+0000","last_active":"2021-03-24T22:04:22.395887+0000","last_peered":"2021-03-24T22:04:22.395887+0000","last_clean":"2021-03-24T22:04:22.395887+0000","last_became_active":"2021-03-24T22:02:00.304128+0000","last_became_peered":"2021-03-24T22:02:00.304128+0000","last_unstale":"2021-03-24T22:04:22.395887+0000","last_undegraded":"2021-03-24T22:04:22.395887+0000","last_fullsized":"2021-03-24T22:04:22.395887+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":5,"last_scrub":"624'193","last_scrub_stamp":"2021-03-24T22:03:43.233875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:03:43.233875+0000","log_size":193,"ondisk_log_size":193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4857520,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":42,"num_read_kb":38347,"num_write":26,"num_write_kb":9218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":32240200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1cd","length":"1"},{"start":"1d4","length":"1"},{"start":"1f0","length":"1"},{"start":"1f3","length":"1"},{"start":"20e","length":"1"},{"start":"211","length":"1"},{"start":"214","length":"1"},{"start":"218","length":"2"},{"start":"21e","length":"1"},{"start":"220","length":"2"},{"start":"22f","length":"1"},{"start":"232","length":"1"},{"start":"239","length":"1"}]},{"pgid":"3.1d","version":"0'0","reported_seq":"725","reported_epoch":"690","state":"active+undersized","last_fresh":"2021-03-24T22:04:20.358066+0000","last_change":"2021-03-24T22:03:54.369825+0000","last_active":"2021-03-24T22:04:20.358066+0000","last_peered":"2021-03-24T22:04:20.358066+0000","last_clean":"2021-03-24T22:03:52.504099+0000","last_became_active":"2021-03-24T22:03:54.369825+0000","last_became_peered":"2021-03-24T22:03:54.369825+0000","last_unstale":"2021-03-24T22:04:20.358066+0000","last_undegraded":"2021-03-24T22:04:20.358066+0000","last_fullsized":"2021-03-24T22:03:54.355366+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:19.684726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:02:19.684726+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"459","reported_epoch":"692","state":"active+undersized","last_fresh":"2021-03-24T22:04:22.392144+0000","last_change":"2021-03-24T22:03:54.392675+0000","last_active":"2021-03-24T22:04:22.392144+0000","last_peered":"2021-03-24T22:04:22.392144+0000","last_clean":"2021-03-24T22:01:55.300870+0000","last_became_active":"2021-03-24T22:03:54.392675+0000","last_became_peered":"2021-03-24T22:03:54.392675+0000","last_unstale":"2021-03-24T22:04:22.392144+0000","last_undegraded":"2021-03-24T22:04:22.392144+0000","last_fullsized":"2021-03-24T22:03:54.371106+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:44.728766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:03:44.728766+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"724","reported_epoch":"692","state":"active+clean","last_fresh":"2021-03-24T22:04:22.393361+0000","last_change":"2021-03-24T22:03:45.431802+0000","last_active":"2021-03-24T22:04:22.393361+0000","last_peered":"2021-03-24T22:04:22.393361+0000","last_clean":"2021-03-24T22:04:22.393361+0000","last_became_active":"2021-03-24T22:03:45.431424+0000","last_became_peered":"2021-03-24T22:03:45.431424+0000","last_unstale":"2021-03-24T22:04:22.393361+0000","last_undegraded":"2021-03-24T22:04:22.393361+0000","last_fullsized":"2021-03-24T22:04:22.393361+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:19.457980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:02:19.457980+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.19","version":"690'902","reported_seq":"3122","reported_epoch":"690","state":"active+clean","last_fresh":"2021-03-24T22:04:20.371474+0000","last_change":"2021-03-24T22:04:20.371791+0000","last_active":"2021-03-24T22:04:20.371474+0000","last_peered":"2021-03-24T22:04:20.371474+0000","last_clean":"2021-03-24T22:04:20.371474+0000","last_became_active":"2021-03-24T22:02:07.304440+0000","last_became_peered":"2021-03-24T22:02:07.304440+0000","last_unstale":"2021-03-24T22:04:20.371474+0000","last_undegraded":"2021-03-24T22:04:20.371474+0000","last_fullsized":"2021-03-24T22:04:20.371474+0000","mapping_epoch":556,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":557,"parent":"0.0","parent_split_bits":5,"last_scrub":"664'844","last_scrub_stamp":"2021-03-24T22:03:55.671094+0000","last_deep_scrub":"455'480","last_deep_scrub_stamp":"2021-03-24T22:00:26.193730+0000","last_clean_scrub_stamp":"2021-03-24T22:03:55.671094+0000","log_size":902,"ondisk_log_size":902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44075646,"num_objects":19,"num_object_clones":16,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":202,"num_read_kb":176533,"num_write":179,"num_write_kb":78005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":265,"num_bytes_recovered":154749512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1cd","length":"1"},{"start":"1d4","length":"1"},{"start":"1f0","length":"1"},{"start":"1f3","length":"1"},{"start":"20e","length":"1"},{"start":"211","length":"1"},{"start":"214","length":"1"},{"start":"218","length":"2"},{"start":"21e","length":"1"},{"start":"220","length":"2"},{"start":"22f","length":"1"},{"start":"232","length":"1"}]},{"pgid":"3.1e","version":"0'0","reported_seq":"710","reported_epoch":"692","state":"active+clean","last_fresh":"2021-03-24T22:04:22.393624+0000","last_change":"2021-03-24T22:03:51.276882+0000","last_active":"2021-03-24T22:04:22.393624+0000","last_peered":"2021-03-24T22:04:22.393624+0000","last_clean":"2021-03-24T22:04:22.393624+0000","last_became_active":"2021-03-24T22:01:43.381824+0000","last_became_peered":"2021-03-24T22:01:43.381824+0000","last_unstale":"2021-03-24T22:04:22.393624+0000","last_undegraded":"2021-03-24T22:04:22.393624+0000","last_fullsized":"2021-03-24T22:04:22.393624+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:59:05.405785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:59:05.405785+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"187","reported_epoch":"689","state":"active+clean","last_fresh":"2021-03-24T22:04:20.011352+0000","last_change":"2021-03-24T22:03:45.417417+0000","last_active":"2021-03-24T22:04:20.011352+0000","last_peered":"2021-03-24T22:04:20.011352+0000","last_clean":"2021-03-24T22:04:20.011352+0000","last_became_active":"2021-03-24T22:03:45.416917+0000","last_became_peered":"2021-03-24T22:03:45.416917+0000","last_unstale":"2021-03-24T22:04:20.011352+0000","last_undegraded":"2021-03-24T22:04:20.011352+0000","last_fullsized":"2021-03-24T22:04:20.011352+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:01.782483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:03:01.782483+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"725","reported_epoch":"692","state":"active+clean","last_fresh":"2021-03-24T22:04:22.393751+0000","last_change":"2021-03-24T22:03:08.392175+0000","last_active":"2021-03-24T22:04:22.393751+0000","last_peered":"2021-03-24T22:04:22.393751+0000","last_clean":"2021-03-24T22:04:22.393751+0000","last_became_active":"2021-03-24T22:01:57.310369+0000","last_became_peered":"2021-03-24T22:01:57.310369+0000","last_unstale":"2021-03-24T22:04:22.393751+0000","last_undegraded":"2021-03-24T22:04:22.393751+0000","last_fullsized":"2021-03-24T22:04:22.393751+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:08.392060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:03:08.392060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"658'346","reported_seq":"2248","reported_epoch":"689","state":"active+undersized+degraded","last_fresh":"2021-03-24T22:04:20.012376+0000","last_change":"2021-03-24T22:03:54.369383+0000","last_active":"2021-03-24T22:04:20.012376+0000","last_peered":"2021-03-24T22:04:20.012376+0000","last_clean":"2021-03-24T22:03:52.357269+0000","last_became_active":"2021-03-24T22:03:54.369383+0000","last_became_peered":"2021-03-24T22:03:54.369383+0000","last_unstale":"2021-03-24T22:04:20.012376+0000","last_undegraded":"2021-03-24T22:03:54.358065+0000","last_fullsized":"2021-03-24T22:03:54.357692+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":346,"ondisk_log_size":346,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":14,"stat_sum":{"num_bytes":9648877,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":106,"num_read_kb":124692,"num_write":86,"num_write_kb":29858,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":37089076,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[{"shards":"4,6","objects":6}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":"702","reported_epoch":"692","state":"active+undersized","last_fresh":"2021-03-24T22:04:22.399014+0000","last_change":"2021-03-24T22:03:54.374973+0000","last_active":"2021-03-24T22:04:22.399014+0000","last_peered":"2021-03-24T22:04:22.399014+0000","last_clean":"2021-03-24T22:03:52.359351+0000","last_became_active":"2021-03-24T22:03:54.374973+0000","last_became_peered":"2021-03-24T22:03:54.374973+0000","last_unstale":"2021-03-24T22:04:22.399014+0000","last_undegraded":"2021-03-24T22:04:22.399014+0000","last_fullsized":"2021-03-24T22:03:54.370393+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:00:13.624943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:00:13.624943+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"50","reported_epoch":"688","state":"active+undersized","last_fresh":"2021-03-24T22:04:18.450062+0000","last_change":"2021-03-24T22:03:54.367036+0000","last_active":"2021-03-24T22:04:18.450062+0000","last_peered":"2021-03-24T22:04:18.450062+0000","last_clean":"2021-03-24T22:03:52.361672+0000","last_became_active":"2021-03-24T22:03:54.367036+0000","last_became_peered":"2021-03-24T22:03:54.367036+0000","last_unstale":"2021-03-24T22:04:18.450062+0000","last_undegraded":"2021-03-24T22:04:18.450062+0000","last_fullsized":"2021-03-24T22:03:54.356699+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:00.407009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:14.710709+0000","last_clean_scrub_stamp":"2021-03-24T22:03:00.407009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"669","reported_epoch":"688","state":"active+undersized","last_fresh":"2021-03-24T22:04:18.453553+0000","last_change":"2021-03-24T22:03:54.381864+0000","last_active":"2021-03-24T22:04:18.453553+0000","last_peered":"2021-03-24T22:04:18.453553+0000","last_clean":"2021-03-24T22:03:43.465271+0000","last_became_active":"2021-03-24T22:03:54.381864+0000","last_became_peered":"2021-03-24T22:03:54.381864+0000","last_unstale":"2021-03-24T22:04:18.453553+0000","last_undegraded":"2021-03-24T22:04:18.453553+0000","last_fullsized":"2021-03-24T22:03:54.356066+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:35.095493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:02:35.095493+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1b","version":"692'1581","reported_seq":"3527","reported_epoch":"692","state":"active+clean","last_fresh":"2021-03-24T22:04:22.408817+0000","last_change":"2021-03-24T22:04:22.408817+0000","last_active":"2021-03-24T22:04:22.408817+0000","last_peered":"2021-03-24T22:04:22.408817+0000","last_clean":"2021-03-24T22:04:22.408817+0000","last_became_active":"2021-03-24T21:57:33.176110+0000","last_became_peered":"2021-03-24T21:57:33.176110+0000","last_unstale":"2021-03-24T22:04:22.408817+0000","last_undegraded":"2021-03-24T22:04:22.408817+0000","last_fullsized":"2021-03-24T22:04:22.408817+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":5,"last_scrub":"410'569","last_scrub_stamp":"2021-03-24T21:59:43.929595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:59:43.929595+0000","log_size":1581,"ondisk_log_size":1581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":70389032,"num_objects":33,"num_object_clones":27,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":4,"num_read":220,"num_read_kb":245266,"num_write":164,"num_write_kb":72249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1cd","length":"1"},{"start":"1d4","length":"1"},{"start":"1f0","length":"1"},{"start":"1f3","length":"1"},{"start":"20e","length":"1"},{"start":"211","length":"1"},{"start":"214","length":"1"},{"start":"218","length":"2"},{"start":"21e","length":"1"},{"start":"220","length":"2"},{"start":"22f","length":"1"},{"start":"232","length":"1"},{"start":"239","length":"1"}]},{"pgid":"3.24","version":"0'0","reported_seq":"323","reported_epoch":"692","state":"active+clean","last_fresh":"2021-03-24T22:04:22.394056+0000","last_change":"2021-03-24T22:03:41.269971+0000","last_active":"2021-03-24T22:04:22.394056+0000","last_peered":"2021-03-24T22:04:22.394056+0000","last_clean":"2021-03-24T22:04:22.394056+0000","last_became_active":"2021-03-24T21:59:14.369606+0000","last_became_peered":"2021-03-24T21:59:14.369606+0000","last_unstale":"2021-03-24T22:04:22.394056+0000","last_undegraded":"2021-03-24T22:04:22.394056+0000","last_fullsized":"2021-03-24T22:04:22.394056+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:41.269900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:03:41.269900+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"175'22","reported_seq":"1552","reported_epoch":"690","state":"active+clean","last_fresh":"2021-03-24T22:04:20.360925+0000","last_change":"2021-03-24T22:04:20.360925+0000","last_active":"2021-03-24T22:04:20.360925+0000","last_peered":"2021-03-24T22:04:20.360925+0000","last_clean":"2021-03-24T22:04:20.360925+0000","last_became_active":"2021-03-24T22:03:45.432211+0000","last_became_peered":"2021-03-24T22:03:45.432211+0000","last_unstale":"2021-03-24T22:04:20.360925+0000","last_undegraded":"2021-03-24T22:04:20.360925+0000","last_fullsized":"2021-03-24T22:04:20.360925+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":6,"last_scrub":"175'22","last_scrub_stamp":"2021-03-24T22:03:11.749445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:03:11.749445+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1cd","length":"1"},{"start":"1d4","length":"1"},{"start":"1f0","length":"1"},{"start":"1f3","length":"1"},{"start":"20e","length":"1"},{"start":"211","length":"1"},{"start":"214","length":"1"},{"start":"218","length":"2"},{"start":"21e","length":"1"},{"start":"220","length":"2"},{"start":"22f","length":"1"},{"start":"232","length":"1"}]},{"pgid":"3.25","version":"0'0","reported_seq":"44","reported_epoch":"688","state":"active+clean","last_fresh":"2021-03-24T22:04:18.455249+0000","last_change":"2021-03-24T22:03:45.427436+0000","last_active":"2021-03-24T22:04:18.455249+0000","last_peered":"2021-03-24T22:04:18.455249+0000","last_clean":"2021-03-24T22:04:18.455249+0000","last_became_active":"2021-03-24T22:03:45.426841+0000","last_became_peered":"2021-03-24T22:03:45.426841+0000","last_unstale":"2021-03-24T22:04:18.455249+0000","last_undegraded":"2021-03-24T22:04:18.455249+0000","last_fullsized":"2021-03-24T22:04:18.455249+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:53.013129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:53.013129+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.21","version":"685'615","reported_seq":"2767","reported_epoch":"692","state":"active+undersized+degraded","last_fresh":"2021-03-24T22:04:22.393118+0000","last_change":"2021-03-24T22:03:54.363453+0000","last_active":"2021-03-24T22:04:22.393118+0000","last_peered":"2021-03-24T22:04:22.393118+0000","last_clean":"2021-03-24T22:03:52.352078+0000","last_became_active":"2021-03-24T22:03:54.363453+0000","last_became_peered":"2021-03-24T22:03:54.363453+0000","last_unstale":"2021-03-24T22:04:22.393118+0000","last_undegraded":"2021-03-24T22:03:54.360309+0000","last_fullsized":"2021-03-24T22:03:54.360141+0000","mapping_epoch":663,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":554,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":615,"ondisk_log_size":615,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":15,"stat_sum":{"num_bytes":25603565,"num_objects":11,"num_object_clones":10,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":186,"num_read_kb":185574,"num_write":137,"num_write_kb":48811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":55123926,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":11}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":36,"stat_sum":{"num_bytes":634579045,"num_objects":295,"num_object_clones":240,"num_object_copies":885,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":89,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":295,"num_whiteouts":13,"num_read":3119,"num_read_kb":3263402,"num_write":2333,"num_write_kb":895404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2897,"num_bytes_recovered":1560153410,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1358766080,"data_stored":1873924629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15826,"ondisk_log_size":15826,"up":97,"acting":97,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":213,"num_read_kb":208,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":8,"acting":8,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":3,"num_pg":42,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":4828,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":984,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":111,"acting":111,"num_store_stats":8},{"poolid":4,"num_pg":22,"stat_sum":{"num_bytes":23524,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":59,"num_objects_misplaced":40,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1477,"num_read_kb":982,"num_write":828,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1150,"num_bytes_recovered":91276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2048000,"data_stored":73498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":505,"ondisk_log_size":505,"up":60,"acting":60,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":82,"acting":82,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":83,"acting":83,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201435,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":262724,"kb_used_data":205948,"kb_used_omap":0,"kb_used_meta":56768,"kb_avail":93482428,"statfs":{"total":95995035648,"available":95726006272,"internally_reserved":0,"allocated":210890752,"data_stored":284858399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":58130432},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 22:04:11 2021","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.80200000000000005,"15min":0.89600000000000002},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.2070000000000001,"5min":1.298,"15min":12.222},"last":150.68000000000001},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.84299999999999997,"15min":0.98699999999999999},"min":{"1min":0.375,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.292,"5min":1.5009999999999999,"15min":11.506},"last":153.91}]},{"osd":1,"last update":"Wed Mar 24 22:04:11 2021","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.81399999999999995,"15min":0.879},"min":{"1min":0.318,"5min":0.308,"15min":0.308},"max":{"1min":1.224,"5min":1.319,"15min":5.7750000000000004},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.90000000000000002,"15min":0.95399999999999996},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":1.2569999999999999,"5min":6.1849999999999996,"15min":6.1849999999999996},"last":0.80500000000000005}]},{"osd":2,"last update":"Wed Mar 24 22:04:11 2021","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.84199999999999997,"15min":0.99299999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.1930000000000001,"5min":5.7629999999999999,"15min":12.135999999999999},"last":0.89600000000000002},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.88,"15min":1.0209999999999999},"min":{"1min":0.45900000000000002,"5min":0.432,"15min":0.42999999999999999},"max":{"1min":1.2569999999999999,"5min":1.524,"15min":12.178000000000001},"last":0.85299999999999998}]},{"osd":3,"last update":"Wed Mar 24 22:04:11 2021","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":0.95799999999999996,"15min":1.0760000000000001},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":6.1449999999999996,"5min":6.1449999999999996,"15min":12.529},"last":1.0920000000000001},{"interface":"front","average":{"1min":0.97099999999999997,"5min":1.0389999999999999,"15min":1.077},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.4390000000000001,"5min":3.101,"15min":12.666},"last":0.95799999999999996}]},{"osd":4,"last update":"Wed Mar 24 22:04:11 2021","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.94299999999999995,"15min":0.97799999999999998},"min":{"1min":0.40200000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.4279999999999999,"5min":1.649,"15min":6.2089999999999996},"last":1.016},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.98699999999999999,"15min":1.073},"min":{"1min":0.42399999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.476,"5min":1.921,"15min":12.244},"last":1.1870000000000001}]},{"osd":5,"last update":"Wed Mar 24 22:03:09 2021","interfaces":[{"interface":"back","average":{"1min":1.0600000000000001,"5min":1.131,"15min":1.149},"min":{"1min":0.82499999999999996,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.304,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":1.0089999999999999},{"interface":"front","average":{"1min":1.1990000000000001,"5min":1.282,"15min":1.3100000000000001},"min":{"1min":1.0109999999999999,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.498,"5min":2.1190000000000002,"15min":2.1190000000000002},"last":1.0880000000000001}]},{"osd":6,"last update":"Wed Mar 24 22:04:11 2021","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.0649999999999999,"15min":1.1319999999999999},"min":{"1min":0.54200000000000004,"5min":0.44500000000000001,"15min":0.40899999999999997},"max":{"1min":1.399,"5min":1.7869999999999999,"15min":6.1159999999999997},"last":1.2190000000000001},{"interface":"front","average":{"1min":1.002,"5min":1.052,"15min":1.1659999999999999},"min":{"1min":0.51400000000000001,"5min":0.441,"15min":0.434},"max":{"1min":1.522,"5min":1.867,"15min":12.287000000000001},"last":1.387}]}]},{"osd":6,"up_from":47,"seq":201863463072,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":193264,"kb_used_data":168808,"kb_used_omap":0,"kb_used_meta":24448,"kb_avail":93551888,"statfs":{"total":95995035648,"available":95797133312,"internally_reserved":0,"allocated":172859392,"data_stored":230482757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":25034752},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 22:03:41 2021","interfaces":[{"interface":"back","average":{"1min":2.0249999999999999,"5min":1.163,"15min":1.0129999999999999},"min":{"1min":0.373,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":19.257999999999999,"5min":19.257999999999999,"15min":19.257999999999999},"last":0.47999999999999998},{"interface":"front","average":{"1min":1.9410000000000001,"5min":1.165,"15min":0.99399999999999999},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":16.795000000000002,"5min":16.795000000000002,"15min":16.795000000000002},"last":0.55000000000000004}]},{"osd":1,"last update":"Wed Mar 24 22:03:41 2021","interfaces":[{"interface":"back","average":{"1min":2.2709999999999999,"5min":1.1850000000000001,"15min":1.014},"min":{"1min":0.38400000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":21.995999999999999,"5min":21.995999999999999,"15min":21.995999999999999},"last":0.77800000000000002},{"interface":"front","average":{"1min":2.6400000000000001,"5min":1.333,"15min":1.085},"min":{"1min":0.60499999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":26.762,"5min":26.762,"15min":26.762},"last":0.72199999999999998}]},{"osd":2,"last update":"Wed Mar 24 22:03:41 2021","interfaces":[{"interface":"back","average":{"1min":1.778,"5min":1.0840000000000001,"15min":1.0069999999999999},"min":{"1min":0.42399999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":13.044,"5min":13.044,"15min":13.044},"last":0.81399999999999995},{"interface":"front","average":{"1min":2.9670000000000001,"5min":1.319,"15min":1.099},"min":{"1min":0.52000000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":31.375,"5min":31.375,"15min":31.375},"last":0.76400000000000001}]},{"osd":3,"last update":"Wed Mar 24 22:03:41 2021","interfaces":[{"interface":"back","average":{"1min":2.5310000000000001,"5min":1.27,"15min":1.077},"min":{"1min":0.73499999999999999,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":22.07,"5min":22.07,"15min":22.07},"last":0.69299999999999995},{"interface":"front","average":{"1min":1.6559999999999999,"5min":1.1850000000000001,"15min":1.107},"min":{"1min":0.48599999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":11.583,"5min":11.583,"15min":11.583},"last":0.89700000000000002}]},{"osd":4,"last update":"Wed Mar 24 22:03:41 2021","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":1.0209999999999999,"15min":1.0129999999999999},"min":{"1min":0.61499999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":8.4350000000000005,"5min":8.4350000000000005,"15min":8.4350000000000005},"last":0.65200000000000002},{"interface":"front","average":{"1min":2.4369999999999998,"5min":1.26,"15min":1.127},"min":{"1min":0.57899999999999996,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":22.033000000000001,"5min":22.033000000000001,"15min":22.033000000000001},"last":0.86699999999999999}]},{"osd":5,"last update":"Wed Mar 24 22:03:08 2021","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.05,"15min":1.0760000000000001},"min":{"1min":0.55500000000000005,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.454,"5min":1.915,"15min":1.915},"last":1.2490000000000001},{"interface":"front","average":{"1min":1.05,"5min":1.087,"15min":1.117},"min":{"1min":0.60199999999999998,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.4330000000000001,"5min":2.7069999999999999,"15min":2.7069999999999999},"last":0.83899999999999997}]},{"osd":7,"last update":"Wed Mar 24 22:04:04 2021","interfaces":[{"interface":"back","average":{"1min":1.5620000000000001,"5min":1.2230000000000001,"15min":1.1990000000000001},"min":{"1min":0.66700000000000004,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":8.3599999999999994,"5min":9.5909999999999993,"15min":9.5909999999999993},"last":0.84499999999999997},{"interface":"front","average":{"1min":1.45,"5min":1.1259999999999999,"15min":1.1799999999999999},"min":{"1min":0.51300000000000001,"5min":0.442,"15min":0.442},"max":{"1min":8.7959999999999994,"5min":8.7959999999999994,"15min":8.7959999999999994},"last":0.95999999999999996}]}]},{"osd":1,"up_from":13,"seq":55834575029,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":130568,"kb_used_data":98304,"kb_used_omap":0,"kb_used_meta":32256,"kb_avail":93614584,"statfs":{"total":95995035648,"available":95861334016,"internally_reserved":0,"allocated":100663296,"data_stored":141328114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":33030144},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 22:03:57 2021","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":1.1359999999999999,"15min":0.80800000000000005},"min":{"1min":0.33800000000000002,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":3.4809999999999999,"5min":21.085000000000001,"15min":21.085000000000001},"last":0.66200000000000003},{"interface":"front","average":{"1min":1.028,"5min":0.89700000000000002,"15min":0.77900000000000003},"min":{"1min":0.39700000000000002,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":5.0129999999999999,"5min":8.5180000000000007,"15min":8.5180000000000007},"last":0.68300000000000005}]},{"osd":2,"last update":"Wed Mar 24 22:04:09 2021","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.754,"15min":0.69599999999999995},"min":{"1min":0.441,"5min":0.307,"15min":0.307},"max":{"1min":2.9580000000000002,"5min":9.9870000000000001,"15min":9.9870000000000001},"last":0.55200000000000005},{"interface":"front","average":{"1min":1.115,"5min":0.95399999999999996,"15min":0.84499999999999997},"min":{"1min":0.55900000000000005,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":5.056,"5min":9.2059999999999995,"15min":9.2059999999999995},"last":0.59799999999999998}]},{"osd":3,"last update":"Wed Mar 24 22:03:40 2021","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":0.96799999999999997,"15min":1.042},"min":{"1min":0.54600000000000004,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":5.0860000000000003,"5min":8.5990000000000002,"15min":17.032},"last":0.79800000000000004},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.218,"15min":1.097},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.316},"max":{"1min":5.4800000000000004,"5min":15.775,"15min":16.908999999999999},"last":0.77400000000000002}]},{"osd":4,"last update":"Wed Mar 24 22:04:05 2021","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":0.996,"15min":0.97899999999999998},"min":{"1min":0.57999999999999996,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":3.1389999999999998,"5min":9.6150000000000002,"15min":9.6150000000000002},"last":1.089},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.0169999999999999,"15min":1.0649999999999999},"min":{"1min":0.77400000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":2.2829999999999999,"5min":3.9359999999999999,"15min":3.9359999999999999},"last":0.94499999999999995}]},{"osd":5,"last update":"Wed Mar 24 22:03:02 2021","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.0449999999999999,"15min":1.054},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.6719999999999999,"5min":3.976,"15min":3.976},"last":0.84399999999999997},{"interface":"front","average":{"1min":1.036,"5min":1.127,"15min":1.095},"min":{"1min":0.48099999999999998,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.647,"5min":9.6809999999999992,"15min":9.6809999999999992},"last":1.0820000000000001}]},{"osd":6,"last update":"Wed Mar 24 22:03:40 2021","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.359,"15min":1.202},"min":{"1min":0.86899999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":7.7160000000000002,"5min":11.254,"15min":11.254},"last":1.0649999999999999},{"interface":"front","average":{"1min":1.145,"5min":1.103,"15min":1.135},"min":{"1min":0.73999999999999999,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.667,"5min":9.7300000000000004,"15min":9.7300000000000004},"last":1}]},{"osd":7,"last update":"Wed Mar 24 22:04:05 2021","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.105,"15min":1.1879999999999999},"min":{"1min":0.79200000000000004,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":2.25,"5min":3.8439999999999999,"15min":4.4699999999999998},"last":1.036},{"interface":"front","average":{"1min":1.452,"5min":1.3720000000000001,"15min":1.2270000000000001},"min":{"1min":0.80700000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":7.8799999999999999,"5min":11.289,"15min":11.289},"last":1.1100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738553,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":218344,"kb_used_data":176736,"kb_used_omap":0,"kb_used_meta":41600,"kb_avail":93526808,"statfs":{"total":95995035648,"available":95771451392,"internally_reserved":0,"allocated":180977664,"data_stored":253088773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":42598400},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Mar 24 22:04:01 2021","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":0.69599999999999995,"15min":0.67500000000000004},"min":{"1min":0.28599999999999998,"5min":0.26900000000000002,"15min":0.249},"max":{"1min":5.0419999999999998,"5min":5.0419999999999998,"15min":5.0419999999999998},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.99299999999999999,"5min":0.73199999999999998,"15min":0.749},"min":{"1min":0.33400000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":5.4320000000000004,"5min":5.4320000000000004,"15min":8.8989999999999991},"last":0.65700000000000003}]},{"osd":2,"last update":"Wed Mar 24 22:04:18 2021","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.78500000000000003,"15min":0.76200000000000001},"min":{"1min":0.41699999999999998,"5min":0.308,"15min":0.28999999999999998},"max":{"1min":1.1559999999999999,"5min":15.949999999999999,"15min":15.949999999999999},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69899999999999995,"15min":0.76700000000000002},"min":{"1min":0.41099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.3580000000000001,"5min":1.3580000000000001,"15min":4.766},"last":0.64100000000000001}]},{"osd":3,"last update":"Wed Mar 24 22:03:39 2021","interfaces":[{"interface":"back","average":{"1min":2.1190000000000002,"5min":1.0780000000000001,"15min":0.93200000000000005},"min":{"1min":0.375,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":24.75,"5min":24.75,"15min":24.75},"last":1.038},{"interface":"front","average":{"1min":2.1499999999999999,"5min":1.1020000000000001,"15min":1.004},"min":{"1min":0.438,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":24.832999999999998,"5min":24.832999999999998,"15min":24.832999999999998},"last":0.76900000000000002}]},{"osd":4,"last update":"Wed Mar 24 22:04:01 2021","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":0.98999999999999999,"15min":1.0549999999999999},"min":{"1min":0.52700000000000002,"5min":0.44600000000000001,"15min":0.436},"max":{"1min":1.3420000000000001,"5min":3.8300000000000001,"15min":5.0330000000000004},"last":0.80800000000000005},{"interface":"front","average":{"1min":1.5409999999999999,"5min":1.125,"15min":1.0900000000000001},"min":{"1min":0.61199999999999999,"5min":0.46400000000000002,"15min":0.44900000000000001},"max":{"1min":12.366,"5min":12.366,"15min":12.366},"last":1.1679999999999999}]},{"osd":5,"last update":"Wed Mar 24 22:03:05 2021","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.151,"15min":1.208},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.6379999999999999,"5min":3.9550000000000001,"15min":3.9550000000000001},"last":1.2250000000000001},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.165,"15min":1.2250000000000001},"min":{"1min":0.73399999999999999,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.6240000000000001,"5min":4.2560000000000002,"15min":4.2560000000000002},"last":1.3919999999999999}]},{"osd":6,"last update":"Wed Mar 24 22:03:39 2021","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":1.1599999999999999,"15min":1.133},"min":{"1min":0.63600000000000001,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.5489999999999999,"5min":15.983000000000001,"15min":15.983000000000001},"last":1.107},{"interface":"front","average":{"1min":1.131,"5min":1.028,"15min":1.1539999999999999},"min":{"1min":0.59999999999999998,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.9630000000000001,"5min":1.9630000000000001,"15min":4.9960000000000004},"last":0.89700000000000002}]},{"osd":7,"last update":"Wed Mar 24 22:04:08 2021","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.127,"15min":1.173},"min":{"1min":0.65800000000000003,"5min":0.58599999999999997,"15min":0.55700000000000005},"max":{"1min":4.9470000000000001,"5min":4.9470000000000001,"15min":6.1319999999999997},"last":0.95499999999999996},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.167,"15min":1.1759999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.528,"5min":16.007000000000001,"15min":16.007000000000001},"last":1.2}]}]},{"osd":2,"up_from":18,"seq":77309411506,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":248556,"kb_used_data":207780,"kb_used_omap":0,"kb_used_meta":40768,"kb_avail":93496596,"statfs":{"total":95995035648,"available":95740514304,"internally_reserved":0,"allocated":212766720,"data_stored":282835612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41746432},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 22:04:13 2021","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":0.85999999999999999,"15min":0.73599999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":7.468,"5min":10.054,"15min":10.054},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.84299999999999997,"15min":0.77100000000000002},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.8069999999999999,"5min":12.099,"15min":12.099},"last":0.69499999999999995}]},{"osd":1,"last update":"Wed Mar 24 22:04:13 2021","interfaces":[{"interface":"back","average":{"1min":1.425,"5min":1.022,"15min":0.83499999999999996},"min":{"1min":0.39500000000000002,"5min":0.309,"15min":0.28100000000000003},"max":{"1min":12.343999999999999,"5min":17.954999999999998,"15min":17.954999999999998},"last":0.83099999999999996},{"interface":"front","average":{"1min":1.143,"5min":0.80100000000000005,"15min":0.78400000000000003},"min":{"1min":0.30399999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":7.4989999999999997,"5min":7.4989999999999997,"15min":7.4989999999999997},"last":0.996}]},{"osd":3,"last update":"Wed Mar 24 22:03:43 2021","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":1.2190000000000001,"15min":0.97599999999999998},"min":{"1min":0.53800000000000003,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":12.308,"5min":25.009,"15min":25.009},"last":0.78700000000000003},{"interface":"front","average":{"1min":1.4059999999999999,"5min":0.95999999999999996,"15min":0.92600000000000005},"min":{"1min":0.61199999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":7.5199999999999996,"5min":7.5199999999999996,"15min":7.5199999999999996},"last":0.626}]},{"osd":4,"last update":"Wed Mar 24 22:03:58 2021","interfaces":[{"interface":"back","average":{"1min":1.617,"5min":1.3149999999999999,"15min":1.139},"min":{"1min":0.62,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":12.467000000000001,"5min":24.914000000000001,"15min":24.914000000000001},"last":0.879},{"interface":"front","average":{"1min":1.0349999999999999,"5min":0.97699999999999998,"15min":1.0349999999999999},"min":{"1min":0.47199999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":2.0310000000000001,"5min":2.8290000000000002,"15min":2.8290000000000002},"last":0.93400000000000005}]},{"osd":5,"last update":"Wed Mar 24 22:03:09 2021","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.121,"15min":1.1200000000000001},"min":{"1min":0.78900000000000003,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":4.3019999999999996,"5min":4.3019999999999996,"15min":4.3019999999999996},"last":0.60999999999999999},{"interface":"front","average":{"1min":1.149,"5min":1.101,"15min":1.0960000000000001},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.8360000000000001,"5min":2.3250000000000002,"15min":2.3250000000000002},"last":0.89300000000000002}]},{"osd":6,"last update":"Wed Mar 24 22:03:43 2021","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.02,"15min":1.135},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":2.702,"5min":2.7719999999999998,"15min":2.7719999999999998},"last":0.84299999999999997},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.1499999999999999,"15min":1.1419999999999999},"min":{"1min":0.79700000000000004,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":4.3369999999999997,"5min":6.6520000000000001,"15min":6.6520000000000001},"last":0.89900000000000002}]},{"osd":7,"last update":"Wed Mar 24 22:04:06 2021","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.1679999999999999,"15min":1.1950000000000001},"min":{"1min":0.51800000000000002,"5min":0.442,"15min":0.442},"max":{"1min":2.8690000000000002,"5min":6.7720000000000002,"15min":6.7720000000000002},"last":0.97599999999999998},{"interface":"front","average":{"1min":1.694,"5min":1.4610000000000001,"15min":1.3049999999999999},"min":{"1min":0.65100000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":12.492000000000001,"5min":24.969999999999999,"15min":24.969999999999999},"last":1.242}]}]},{"osd":3,"up_from":26,"seq":111669149870,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":299008,"kb_used_data":252216,"kb_used_omap":0,"kb_used_meta":46784,"kb_avail":93446144,"statfs":{"total":95995035648,"available":95688851456,"internally_reserved":0,"allocated":258269184,"data_stored":340792372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":47906816},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 22:03:32 2021","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":1.0589999999999999,"15min":0.85399999999999998},"min":{"1min":0.375,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":1.4279999999999999,"5min":27.548999999999999,"15min":27.548999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.71499999999999997,"5min":1.012,"15min":0.89000000000000001},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.7250000000000001,"5min":26.593,"15min":26.593},"last":0.749}]},{"osd":1,"last update":"Wed Mar 24 22:03:32 2021","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.70599999999999996,"15min":0.76700000000000002},"min":{"1min":0.34499999999999997,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.623,"5min":5.1280000000000001,"15min":14.324},"last":1.004},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.73199999999999998,"15min":0.78100000000000003},"min":{"1min":0.33300000000000002,"5min":0.29599999999999999,"15min":0.28000000000000003},"max":{"1min":4.5540000000000003,"5min":4.5540000000000003,"15min":12.631},"last":0.55900000000000005}]},{"osd":2,"last update":"Wed Mar 24 22:03:32 2021","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.79400000000000004,"15min":0.83099999999999996},"min":{"1min":0.36599999999999999,"5min":0.316,"15min":0.316},"max":{"1min":1.284,"5min":7.9450000000000003,"15min":12.510999999999999},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.77400000000000002,"15min":0.82599999999999996},"min":{"1min":0.53400000000000003,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.3129999999999999,"5min":1.704,"15min":11.317},"last":0.71399999999999997}]},{"osd":4,"last update":"Wed Mar 24 22:03:59 2021","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.91800000000000004,"15min":0.97399999999999998},"min":{"1min":0.64000000000000001,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.5069999999999999,"5min":1.5069999999999999,"15min":2.601},"last":0.876},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.91400000000000003,"15min":1.008},"min":{"1min":0.47899999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.522,"5min":1.522,"15min":14.291},"last":1.1120000000000001}]},{"osd":5,"last update":"Wed Mar 24 22:03:06 2021","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":1.0609999999999999,"15min":1.0940000000000001},"min":{"1min":0.68999999999999995,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.4099999999999999,"5min":1.5369999999999999,"15min":1.5369999999999999},"last":0.85599999999999998},{"interface":"front","average":{"1min":1.042,"5min":1.0680000000000001,"15min":1.0940000000000001},"min":{"1min":0.71499999999999997,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.5960000000000001,"5min":2.2759999999999998,"15min":2.2759999999999998},"last":0.874}]},{"osd":6,"last update":"Wed Mar 24 22:03:32 2021","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":0.98699999999999999,"15min":1.107},"min":{"1min":0.71599999999999997,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.7949999999999999,"5min":1.7949999999999999,"15min":14.255000000000001},"last":1.2729999999999999},{"interface":"front","average":{"1min":1.107,"5min":0.98399999999999999,"15min":1.1279999999999999},"min":{"1min":0.621,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.6639999999999999,"5min":1.6659999999999999,"15min":12.666},"last":0.94099999999999995}]},{"osd":7,"last update":"Wed Mar 24 22:04:03 2021","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":1.0189999999999999,"15min":1.1040000000000001},"min":{"1min":0.69099999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.581,"5min":1.581,"15min":2.633},"last":0.78900000000000003},{"interface":"front","average":{"1min":1.016,"5min":1.107,"15min":1.1899999999999999},"min":{"1min":0.60399999999999998,"5min":0.52000000000000002,"15min":0.44700000000000001},"max":{"1min":2.0299999999999998,"5min":9.609,"15min":14.727},"last":1.173}]}]},{"osd":4,"up_from":32,"seq":137438953640,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":134376,"kb_used_data":90912,"kb_used_omap":0,"kb_used_meta":43456,"kb_avail":93610776,"statfs":{"total":95995035648,"available":95857434624,"internally_reserved":0,"allocated":93093888,"data_stored":138154192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":44498944},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 22:03:59 2021","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":0.91000000000000003,"15min":1.0109999999999999},"min":{"1min":0.437,"5min":0.223,"15min":0.223},"max":{"1min":3.133,"5min":5.9050000000000002,"15min":41.472000000000001},"last":1.171},{"interface":"front","average":{"1min":0.97099999999999997,"5min":1.095,"15min":1.145},"min":{"1min":0.41999999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":3.0819999999999999,"5min":17.513999999999999,"15min":44.185000000000002},"last":0.83399999999999996}]},{"osd":1,"last update":"Wed Mar 24 22:03:59 2021","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":0.94099999999999995,"15min":0.92600000000000005},"min":{"1min":0.47899999999999998,"5min":0.39400000000000002,"15min":0.38100000000000001},"max":{"1min":3.786,"5min":5.75,"15min":5.75},"last":0.80500000000000005},{"interface":"front","average":{"1min":1.0009999999999999,"5min":1.1040000000000001,"15min":0.95399999999999996},"min":{"1min":0.66500000000000004,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":2.5059999999999998,"5min":11.983000000000001,"15min":11.983000000000001},"last":1.232}]},{"osd":2,"last update":"Wed Mar 24 22:03:59 2021","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.86899999999999999,"15min":0.90900000000000003},"min":{"1min":0.51000000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":3.21,"5min":3.21,"15min":4.4379999999999997},"last":0.59699999999999998},{"interface":"front","average":{"1min":1.095,"5min":1.006,"15min":0.97999999999999998},"min":{"1min":0.58199999999999996,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":4.3970000000000002,"5min":5.7999999999999998,"15min":5.7999999999999998},"last":0.65200000000000002}]},{"osd":3,"last update":"Wed Mar 24 22:03:59 2021","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":1.0680000000000001,"15min":1.008},"min":{"1min":0.624,"5min":0.501,"15min":0.439},"max":{"1min":3.9809999999999999,"5min":4.8049999999999997,"15min":4.8049999999999997},"last":0.86499999999999999},{"interface":"front","average":{"1min":1.095,"5min":1.0309999999999999,"15min":1.0029999999999999},"min":{"1min":0.64300000000000002,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":2.4700000000000002,"5min":3.4239999999999999,"15min":4.1189999999999998},"last":1.0860000000000001}]},{"osd":5,"last update":"Wed Mar 24 22:03:05 2021","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.96099999999999997,"15min":0.92700000000000005},"min":{"1min":0.63,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.387,"5min":2.9569999999999999,"15min":2.9569999999999999},"last":0.76200000000000001},{"interface":"front","average":{"1min":1.125,"5min":1.143,"15min":1.1299999999999999},"min":{"1min":0.76700000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.744,"5min":2.9089999999999998,"15min":2.9089999999999998},"last":1.252}]},{"osd":6,"last update":"Wed Mar 24 22:03:29 2021","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.1100000000000001,"15min":1.0800000000000001},"min":{"1min":0.56499999999999995,"5min":0.373,"15min":0.373},"max":{"1min":1.8500000000000001,"5min":7.7679999999999998,"15min":7.7679999999999998},"last":0.90300000000000002},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.216,"15min":1.1240000000000001},"min":{"1min":0.82499999999999996,"5min":0.41199999999999998,"15min":0.41099999999999998},"max":{"1min":1.5029999999999999,"5min":14.122999999999999,"15min":14.122999999999999},"last":1.2729999999999999}]},{"osd":7,"last update":"Wed Mar 24 22:04:01 2021","interfaces":[{"interface":"back","average":{"1min":1.2230000000000001,"5min":1.149,"15min":1.149},"min":{"1min":0.63100000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":2.399,"5min":3.3479999999999999,"15min":3.3479999999999999},"last":1.3109999999999999},{"interface":"front","average":{"1min":1.1439999999999999,"5min":1.087,"15min":1.1459999999999999},"min":{"1min":0.66400000000000003,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":3.1680000000000001,"5min":3.1680000000000001,"15min":3.1680000000000001},"last":1.0509999999999999}]}]},{"osd":5,"up_from":432,"seq":1855425871919,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":189304,"kb_used_data":156208,"kb_used_omap":33,"kb_used_meta":33054,"kb_avail":93555848,"statfs":{"total":95995035648,"available":95801188352,"internally_reserved":0,"allocated":159956992,"data_stored":211294630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33936,"internal_metadata":33848176},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 22:03:04 2021","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":3.4769999999999999,"15min":4.6840000000000002},"min":{"1min":0.51000000000000001,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.1579999999999999,"5min":76.159999999999997,"15min":76.159999999999997},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.82199999999999995,"5min":3.419,"15min":4.593},"min":{"1min":0.42099999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.804,"5min":75.311999999999998,"15min":75.311999999999998},"last":0.57699999999999996}]},{"osd":1,"last update":"Wed Mar 24 22:03:04 2021","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.83099999999999996,"15min":0.81499999999999995},"min":{"1min":0.64600000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.694,"5min":1.694,"15min":1.694},"last":0.71499999999999997},{"interface":"front","average":{"1min":1.046,"5min":0.89500000000000002,"15min":0.88400000000000001},"min":{"1min":0.69499999999999995,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":2.5089999999999999,"5min":2.5089999999999999,"15min":2.5089999999999999},"last":0.54600000000000004}]},{"osd":2,"last update":"Wed Mar 24 22:03:04 2021","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.84799999999999998,"15min":0.86099999999999999},"min":{"1min":0.59199999999999997,"5min":0.434,"15min":0.434},"max":{"1min":1.262,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.95699999999999996,"15min":0.97199999999999998},"min":{"1min":0.65200000000000002,"5min":0.495,"15min":0.495},"max":{"1min":2.5910000000000002,"5min":2.6070000000000002,"15min":2.6070000000000002},"last":0.67500000000000004}]},{"osd":3,"last update":"Wed Mar 24 22:03:04 2021","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":1.0840000000000001,"15min":1.149},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.23,"5min":4.1340000000000003,"15min":4.1340000000000003},"last":0.90000000000000002},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.1819999999999999,"15min":1.242},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.3700000000000001,"5min":4.173,"15min":4.173},"last":0.79100000000000004}]},{"osd":4,"last update":"Wed Mar 24 22:03:04 2021","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":0.94899999999999995,"15min":0.94499999999999995},"min":{"1min":0.61599999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.4690000000000001,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.0860000000000001,"15min":1.1080000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.444,"5min":1.9350000000000001,"15min":1.9350000000000001},"last":0.97099999999999997}]},{"osd":6,"last update":"Wed Mar 24 22:03:04 2021","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.1180000000000001,"15min":1.125},"min":{"1min":0.73199999999999998,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.554,"5min":2.0510000000000002,"15min":2.0510000000000002},"last":1.024},{"interface":"front","average":{"1min":1.056,"5min":1.081,"15min":1.083},"min":{"1min":0.70899999999999996,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.516,"5min":1.851,"15min":1.851},"last":0.95499999999999996}]},{"osd":7,"last update":"Wed Mar 24 22:03:04 2021","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.181,"15min":1.208},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.6359999999999999,"5min":2.161,"15min":2.161},"last":1.0569999999999999},{"interface":"front","average":{"1min":1.198,"5min":1.177,"15min":1.173},"min":{"1min":0.873,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.698,"5min":1.8919999999999999,"15min":1.8919999999999999},"last":0.875}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":984,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":5284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":425984,"data_stored":15384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":5418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":241664,"data_stored":8622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":352256,"data_stored":12738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":7140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":11938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":6974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":175669248,"data_stored":252103881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":96055296,"data_stored":139205938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":210530304,"data_stored":282824808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":252891136,"data_stored":337738428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":87592960,"data_stored":135095682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":155009024,"data_stored":208473689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":167411712,"data_stored":227438744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":213606400,"data_stored":291043459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-03-24T22:04:26.968 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd pool set default.rgw.control pgp_num 32 2021-03-24T22:04:27.007 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:27.070 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:27.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:27.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:04:27.177 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:27.240 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:27.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:27.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:27.304 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-24T22:04:27.304 INFO:teuthology.orchestra.run.smithi125.stdout: "size": 20, 2021-03-24T22:04:27.304 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 600, 2021-03-24T22:04:27.304 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [ 2021-03-24T22:04:27.305 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.305 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:2394 7.23 7:c49de0b4:::smithi12556470-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e275)", 2021-03-24T22:04:27.305 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:57:25.176373+0000", 2021-03-24T22:04:27.305 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 422.12149150400001, 2021-03-24T22:04:27.306 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.025101692, 2021-03-24T22:04:27.306 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:04:27.306 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T22:04:27.306 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:04:27.306 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:04:27.307 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:04:27.307 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 2394 2021-03-24T22:04:27.307 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.307 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:04:27.308 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.308 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:04:27.308 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176373+0000", 2021-03-24T22:04:27.308 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:04:27.308 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.309 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.309 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:04:27.309 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176373+0000", 2021-03-24T22:04:27.309 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T22:04:27.310 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.310 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.310 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:04:27.310 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176371+0000", 2021-03-24T22:04:27.311 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.08e-06 2021-03-24T22:04:27.311 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.311 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.311 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:04:27.311 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176378+0000", 2021-03-24T22:04:27.312 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1309e-05 2021-03-24T22:04:27.312 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.312 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.312 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:04:27.313 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176389+0000", 2021-03-24T22:04:27.313 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.8714e-05 2021-03-24T22:04:27.313 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.313 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.313 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:04:27.314 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176418+0000", 2021-03-24T22:04:27.314 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6766000000000004e-05 2021-03-24T22:04:27.314 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.314 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.315 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.315 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176485+0000", 2021-03-24T22:04:27.315 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.1589999999999999e-06 2021-03-24T22:04:27.315 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.315 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.316 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:04:27.316 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176494+0000", 2021-03-24T22:04:27.316 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0071017390000001 2021-03-24T22:04:27.316 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.316 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.317 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.317 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.183596+0000", 2021-03-24T22:04:27.317 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6001999999999996e-05 2021-03-24T22:04:27.317 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.318 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.318 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:04:27.318 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.183662+0000", 2021-03-24T22:04:27.318 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.017585719 2021-03-24T22:04:27.318 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.319 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.319 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.319 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.201247+0000", 2021-03-24T22:04:27.319 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000106111 2021-03-24T22:04:27.320 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.320 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.320 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:04:27.320 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.201354+0000", 2021-03-24T22:04:27.320 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000120763 2021-03-24T22:04:27.321 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.321 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.321 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:04:27.321 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.201474+0000", 2021-03-24T22:04:27.321 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.025101692 2021-03-24T22:04:27.322 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:27.322 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:04:27.322 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:27.322 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.323 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.323 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:3950 7.c 7:32974111:::smithi12556470-2:head [write 675780~495523,stat] snapc 136=[136,133,132,130,12d,12c,12b,129,128,127,126,120,11e,11b,11a,119,110,10f,10c,106,105,104,fa,f8,e7] ondisk+write+known_if_redirected e417)", 2021-03-24T22:04:27.323 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:47.297626+0000", 2021-03-24T22:04:27.323 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 280.000238013, 2021-03-24T22:04:27.323 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0613052380000001, 2021-03-24T22:04:27.324 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:04:27.324 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:04:27.324 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:04:27.324 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:04:27.325 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:04:27.325 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3950 2021-03-24T22:04:27.325 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.325 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:04:27.325 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.326 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:04:27.326 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.297626+0000", 2021-03-24T22:04:27.326 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:04:27.326 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.327 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.327 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:04:27.327 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.297626+0000", 2021-03-24T22:04:27.327 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999976 2021-03-24T22:04:27.327 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.328 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.328 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:04:27.328 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.297624+0000", 2021-03-24T22:04:27.328 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00032374599999999997 2021-03-24T22:04:27.328 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.329 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.329 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:04:27.329 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.297948+0000", 2021-03-24T22:04:27.329 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1829e-05 2021-03-24T22:04:27.330 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.330 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.330 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:04:27.330 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.297959+0000", 2021-03-24T22:04:27.330 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.9770000000000003e-06 2021-03-24T22:04:27.331 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.331 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.331 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:04:27.331 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.297966+0000", 2021-03-24T22:04:27.332 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001568081 2021-03-24T22:04:27.332 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.332 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.332 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.332 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299534+0000", 2021-03-24T22:04:27.333 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.7069999999999998e-06 2021-03-24T22:04:27.333 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.333 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.333 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:04:27.334 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299543+0000", 2021-03-24T22:04:27.334 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.02708646 2021-03-24T22:04:27.334 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.334 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.334 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.335 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.326630+0000", 2021-03-24T22:04:27.335 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000180722 2021-03-24T22:04:27.335 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.335 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.335 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:04:27.336 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.326810+0000", 2021-03-24T22:04:27.336 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00025251599999999999 2021-03-24T22:04:27.336 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.336 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.337 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,2", 2021-03-24T22:04:27.337 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.327063+0000", 2021-03-24T22:04:27.337 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0041170629999999998 2021-03-24T22:04:27.337 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.337 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.338 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:04:27.338 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.331180+0000", 2021-03-24T22:04:27.338 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.024822364999999999 2021-03-24T22:04:27.338 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.338 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.339 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:04:27.339 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.356002+0000", 2021-03-24T22:04:27.339 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000346561 2021-03-24T22:04:27.339 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.340 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.340 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:04:27.340 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.356349+0000", 2021-03-24T22:04:27.340 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00048905499999999996 2021-03-24T22:04:27.340 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.341 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.341 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:04:27.341 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.356838+0000", 2021-03-24T22:04:27.341 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0020934259999999998 2021-03-24T22:04:27.341 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.342 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.342 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:04:27.342 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.358931+0000", 2021-03-24T22:04:27.342 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0613052380000001 2021-03-24T22:04:27.343 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:27.343 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:04:27.343 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:27.343 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.343 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.344 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:3951 7.c 7:32974111:::smithi12556470-2:head [write 1682747~714511] snapc 136=[136,133,132,130,12d,12c,12b,129,128,127,126,120,11e,11b,11a,119,110,10f,10c,106,105,104,fa,f8,e7] ondisk+write+known_if_redirected e417)", 2021-03-24T22:04:27.344 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:47.298949+0000", 2021-03-24T22:04:27.344 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 279.99891507299998, 2021-03-24T22:04:27.344 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0693593290000001, 2021-03-24T22:04:27.345 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:04:27.345 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:04:27.345 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:04:27.345 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:04:27.345 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:04:27.346 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3951 2021-03-24T22:04:27.346 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.346 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:04:27.346 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.347 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:04:27.347 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.298949+0000", 2021-03-24T22:04:27.347 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:04:27.347 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.347 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.348 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:04:27.348 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.298949+0000", 2021-03-24T22:04:27.348 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T22:04:27.348 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.349 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.349 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:04:27.349 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.298948+0000", 2021-03-24T22:04:27.349 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000412256 2021-03-24T22:04:27.349 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.350 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.350 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:04:27.350 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299360+0000", 2021-03-24T22:04:27.350 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1399999999999999e-05 2021-03-24T22:04:27.351 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.351 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.351 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:04:27.351 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299371+0000", 2021-03-24T22:04:27.351 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.3679999999999996e-06 2021-03-24T22:04:27.352 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.352 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.352 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:04:27.352 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299379+0000", 2021-03-24T22:04:27.353 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00048704199999999998 2021-03-24T22:04:27.353 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.353 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.353 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.353 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299866+0000", 2021-03-24T22:04:27.354 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.277e-06 2021-03-24T22:04:27.354 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.354 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.354 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:04:27.354 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299873+0000", 2021-03-24T22:04:27.355 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0281777940000001 2021-03-24T22:04:27.355 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.355 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.355 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.356 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.328051+0000", 2021-03-24T22:04:27.356 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000151185 2021-03-24T22:04:27.356 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.356 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.356 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:04:27.357 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.328202+0000", 2021-03-24T22:04:27.357 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.030788554999999999 2021-03-24T22:04:27.357 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.357 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.357 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.358 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.358991+0000", 2021-03-24T22:04:27.358 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012337099999999999 2021-03-24T22:04:27.358 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.358 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.359 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:04:27.359 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.359114+0000", 2021-03-24T22:04:27.359 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00022504400000000001 2021-03-24T22:04:27.359 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.359 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.360 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,2", 2021-03-24T22:04:27.360 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.359339+0000", 2021-03-24T22:04:27.360 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0044580849999999997 2021-03-24T22:04:27.360 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.360 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.361 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:04:27.361 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.363797+0000", 2021-03-24T22:04:27.361 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0031527479999999999 2021-03-24T22:04:27.361 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.361 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.362 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:04:27.362 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.366950+0000", 2021-03-24T22:04:27.362 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001068685 2021-03-24T22:04:27.362 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.363 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.363 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:04:27.363 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.368018+0000", 2021-03-24T22:04:27.363 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.4264999999999995e-05 2021-03-24T22:04:27.364 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.364 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.364 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:04:27.364 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.368093+0000", 2021-03-24T22:04:27.365 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00021562 2021-03-24T22:04:27.365 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.365 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.365 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:04:27.366 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.368308+0000", 2021-03-24T22:04:27.366 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0693593290000001 2021-03-24T22:04:27.366 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:27.366 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:04:27.366 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:27.367 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.367 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.367 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:3952 7.c 7:32974111:::smithi12556470-2:head [write 3012769~63011] snapc 136=[136,133,132,130,12d,12c,12b,129,128,127,126,120,11e,11b,11a,119,110,10f,10c,106,105,104,fa,f8,e7] ondisk+write+known_if_redirected e417)", 2021-03-24T22:04:27.367 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:47.299385+0000", 2021-03-24T22:04:27.368 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 279.99847872700002, 2021-03-24T22:04:27.368 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.070508668, 2021-03-24T22:04:27.368 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:04:27.368 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:04:27.368 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:04:27.369 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:04:27.369 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:04:27.369 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3952 2021-03-24T22:04:27.369 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.369 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:04:27.370 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.370 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:04:27.370 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299385+0000", 2021-03-24T22:04:27.370 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:04:27.371 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.371 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.371 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:04:27.371 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299385+0000", 2021-03-24T22:04:27.371 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T22:04:27.372 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.372 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.372 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:04:27.372 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299384+0000", 2021-03-24T22:04:27.372 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.6712000000000001e-05 2021-03-24T22:04:27.373 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.373 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.373 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:04:27.373 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299411+0000", 2021-03-24T22:04:27.374 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6420000000000001e-06 2021-03-24T22:04:27.374 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.374 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.374 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:04:27.374 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299418+0000", 2021-03-24T22:04:27.375 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6809999999999997e-06 2021-03-24T22:04:27.375 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.375 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.375 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:04:27.376 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299422+0000", 2021-03-24T22:04:27.376 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000472101 2021-03-24T22:04:27.376 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.376 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.376 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.377 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299895+0000", 2021-03-24T22:04:27.377 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.0889999999999996e-06 2021-03-24T22:04:27.377 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.377 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.377 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:04:27.378 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299901+0000", 2021-03-24T22:04:27.378 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0283333750000001 2021-03-24T22:04:27.378 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.378 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.379 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.379 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.328234+0000", 2021-03-24T22:04:27.379 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000104591 2021-03-24T22:04:27.379 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.379 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.380 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:04:27.380 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.328339+0000", 2021-03-24T22:04:27.380 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.033718655 2021-03-24T22:04:27.380 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.380 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.381 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.381 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.362057+0000", 2021-03-24T22:04:27.381 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00014753499999999999 2021-03-24T22:04:27.381 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.384 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.384 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:04:27.384 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.362205+0000", 2021-03-24T22:04:27.384 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000219259 2021-03-24T22:04:27.385 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.385 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.385 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,2", 2021-03-24T22:04:27.386 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.362424+0000", 2021-03-24T22:04:27.386 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001410356 2021-03-24T22:04:27.386 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.386 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.386 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:04:27.387 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.363834+0000", 2021-03-24T22:04:27.387 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.003124342 2021-03-24T22:04:27.387 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.387 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.388 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:04:27.388 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.366959+0000", 2021-03-24T22:04:27.388 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0014798490000000001 2021-03-24T22:04:27.388 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.388 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.389 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:04:27.389 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.368439+0000", 2021-03-24T22:04:27.389 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001231179 2021-03-24T22:04:27.389 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.390 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.390 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:04:27.390 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.369670+0000", 2021-03-24T22:04:27.390 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00022420900000000001 2021-03-24T22:04:27.390 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.391 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.391 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:04:27.392 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.369894+0000", 2021-03-24T22:04:27.392 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.070508668 2021-03-24T22:04:27.392 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:27.392 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:04:27.393 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:27.393 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.393 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.393 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:3953 7.c 7:32974111:::smithi12556470-2:head [setxattr _header (57) in=7b,truncate 3075780] snapc 136=[136,133,132,130,12d,12c,12b,129,128,127,126,120,11e,11b,11a,119,110,10f,10c,106,105,104,fa,f8,e7] ondisk+write+known_if_redirected e417)", 2021-03-24T22:04:27.394 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:47.299427+0000", 2021-03-24T22:04:27.394 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 279.998436959, 2021-03-24T22:04:27.394 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0708905799999999, 2021-03-24T22:04:27.394 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:04:27.394 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:04:27.395 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:04:27.395 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:04:27.395 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:04:27.395 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3953 2021-03-24T22:04:27.396 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.396 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:04:27.396 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.396 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:04:27.396 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299427+0000", 2021-03-24T22:04:27.397 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:04:27.397 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.397 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.397 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:04:27.398 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299427+0000", 2021-03-24T22:04:27.398 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999995 2021-03-24T22:04:27.398 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.398 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.399 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:04:27.399 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299427+0000", 2021-03-24T22:04:27.399 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.4259999999999999e-06 2021-03-24T22:04:27.399 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.400 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.400 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:04:27.400 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299430+0000", 2021-03-24T22:04:27.400 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1189999999999996e-06 2021-03-24T22:04:27.400 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.401 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.401 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:04:27.401 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299435+0000", 2021-03-24T22:04:27.401 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9140000000000001e-06 2021-03-24T22:04:27.402 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.402 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.402 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:04:27.402 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299439+0000", 2021-03-24T22:04:27.402 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00048008799999999999 2021-03-24T22:04:27.403 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.403 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.403 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.403 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299919+0000", 2021-03-24T22:04:27.403 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.835e-06 2021-03-24T22:04:27.404 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.404 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.404 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:04:27.404 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299925+0000", 2021-03-24T22:04:27.405 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.028441138 2021-03-24T22:04:27.405 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.405 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.405 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.405 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.328366+0000", 2021-03-24T22:04:27.406 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010274500000000001 2021-03-24T22:04:27.406 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.406 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.406 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:04:27.407 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.328469+0000", 2021-03-24T22:04:27.407 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.034522276999999997 2021-03-24T22:04:27.407 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.407 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.407 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.408 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.362991+0000", 2021-03-24T22:04:27.408 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000114469 2021-03-24T22:04:27.408 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.408 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.408 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:04:27.409 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.363106+0000", 2021-03-24T22:04:27.409 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00021205599999999999 2021-03-24T22:04:27.409 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.409 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.410 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,2", 2021-03-24T22:04:27.410 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.363318+0000", 2021-03-24T22:04:27.410 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00096840900000000004 2021-03-24T22:04:27.410 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.410 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.411 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:04:27.411 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.364286+0000", 2021-03-24T22:04:27.411 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002681143 2021-03-24T22:04:27.411 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.412 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.412 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:04:27.412 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.366967+0000", 2021-03-24T22:04:27.412 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002998559 2021-03-24T22:04:27.412 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.413 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.413 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:04:27.413 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.369966+0000", 2021-03-24T22:04:27.413 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012207299999999999 2021-03-24T22:04:27.413 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.414 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.414 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:04:27.414 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.370088+0000", 2021-03-24T22:04:27.414 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00022991299999999999 2021-03-24T22:04:27.415 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.415 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.415 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:04:27.415 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.370318+0000", 2021-03-24T22:04:27.415 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0708905799999999 2021-03-24T22:04:27.416 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:27.416 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:04:27.416 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:27.416 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.417 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.417 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:3954 7.c 7:32974111:::smithi12556470-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e417)", 2021-03-24T22:04:27.417 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:47.299443+0000", 2021-03-24T22:04:27.417 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 279.99842127599999, 2021-03-24T22:04:27.417 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.071832683, 2021-03-24T22:04:27.418 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:04:27.418 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T22:04:27.418 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:04:27.418 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:04:27.419 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:04:27.419 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3954 2021-03-24T22:04:27.419 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.419 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:04:27.419 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.420 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:04:27.420 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299443+0000", 2021-03-24T22:04:27.420 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:04:27.420 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.420 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.421 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:04:27.421 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299443+0000", 2021-03-24T22:04:27.421 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999995 2021-03-24T22:04:27.421 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.422 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.422 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:04:27.422 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299442+0000", 2021-03-24T22:04:27.422 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.2239999999999998e-06 2021-03-24T22:04:27.422 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.423 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.423 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:04:27.423 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299445+0000", 2021-03-24T22:04:27.423 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.3809999999999998e-06 2021-03-24T22:04:27.424 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.424 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.424 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:04:27.424 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299450+0000", 2021-03-24T22:04:27.424 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9160000000000003e-06 2021-03-24T22:04:27.425 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.425 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.425 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:04:27.425 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299454+0000", 2021-03-24T22:04:27.425 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00049169700000000003 2021-03-24T22:04:27.426 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.426 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.426 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.426 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299946+0000", 2021-03-24T22:04:27.427 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.003e-06 2021-03-24T22:04:27.427 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.427 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.427 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:04:27.427 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299952+0000", 2021-03-24T22:04:27.428 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0285410699999999 2021-03-24T22:04:27.428 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.428 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.428 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.428 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.328493+0000", 2021-03-24T22:04:27.429 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0015757410000000001 2021-03-24T22:04:27.429 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.429 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.429 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:04:27.430 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.330068+0000", 2021-03-24T22:04:27.430 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.033581221000000001 2021-03-24T22:04:27.430 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.430 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.430 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.431 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.363650+0000", 2021-03-24T22:04:27.431 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010419700000000001 2021-03-24T22:04:27.431 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.431 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.432 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:04:27.432 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.363754+0000", 2021-03-24T22:04:27.432 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0070060599999999997 2021-03-24T22:04:27.432 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.432 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.433 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.433 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.370760+0000", 2021-03-24T22:04:27.433 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00020776099999999999 2021-03-24T22:04:27.433 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.433 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.434 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:04:27.434 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.370968+0000", 2021-03-24T22:04:27.434 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00030779100000000002 2021-03-24T22:04:27.434 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.435 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.435 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:04:27.435 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.371275+0000", 2021-03-24T22:04:27.435 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.071832683 2021-03-24T22:04:27.435 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:27.436 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:04:27.436 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:27.436 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.436 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.437 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:4147 7.21 7:86662589:::smithi12556470-19:head [write 712762~401056,stat] snapc 143=[143,140,13d,13b,139,137,136,12d,12b,129,128,127,126,120,11e,11b,11a,119,110,10f,10c,105,104,fa,f8,e7] ondisk+write+known_if_redirected e432)", 2021-03-24T22:04:27.437 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:00:02.248651+0000", 2021-03-24T22:04:27.437 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 265.04921296399999, 2021-03-24T22:04:27.437 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0717329640000002, 2021-03-24T22:04:27.437 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:04:27.438 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:04:27.438 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:04:27.438 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:04:27.438 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:04:27.438 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4147 2021-03-24T22:04:27.439 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.439 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:04:27.439 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.439 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:04:27.440 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.248651+0000", 2021-03-24T22:04:27.440 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:04:27.440 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.440 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.440 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:04:27.441 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.248651+0000", 2021-03-24T22:04:27.441 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999967 2021-03-24T22:04:27.441 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.441 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.441 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:04:27.442 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.248648+0000", 2021-03-24T22:04:27.442 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000104109 2021-03-24T22:04:27.442 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.442 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.443 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:04:27.443 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.248752+0000", 2021-03-24T22:04:27.443 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002002804 2021-03-24T22:04:27.443 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.443 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.444 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:04:27.444 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.250755+0000", 2021-03-24T22:04:27.444 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5437000000000001e-05 2021-03-24T22:04:27.444 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.444 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.445 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:04:27.445 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.250770+0000", 2021-03-24T22:04:27.445 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0014452029999999999 2021-03-24T22:04:27.445 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.446 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.446 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.446 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.252215+0000", 2021-03-24T22:04:27.446 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.8291000000000002e-05 2021-03-24T22:04:27.446 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.447 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.447 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:04:27.447 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.252234+0000", 2021-03-24T22:04:27.447 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99172324700000003 2021-03-24T22:04:27.447 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.448 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.448 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.448 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:03.243957+0000", 2021-03-24T22:04:27.448 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.9359999999999994e-06 2021-03-24T22:04:27.449 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.449 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.449 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:04:27.449 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:03.243967+0000", 2021-03-24T22:04:27.449 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0558876720000001 2021-03-24T22:04:27.450 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.450 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.451 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.451 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.299854+0000", 2021-03-24T22:04:27.451 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00045708100000000002 2021-03-24T22:04:27.452 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.452 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.452 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for degraded object", 2021-03-24T22:04:27.452 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.300311+0000", 2021-03-24T22:04:27.452 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0088695200000000005 2021-03-24T22:04:27.453 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.453 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.453 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.453 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.309181+0000", 2021-03-24T22:04:27.454 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000101866 2021-03-24T22:04:27.454 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.454 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.454 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:04:27.454 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.309283+0000", 2021-03-24T22:04:27.455 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00025108500000000001 2021-03-24T22:04:27.455 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.455 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.455 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,5", 2021-03-24T22:04:27.456 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.309534+0000", 2021-03-24T22:04:27.456 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001499727 2021-03-24T22:04:27.456 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.456 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.456 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:04:27.457 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.311034+0000", 2021-03-24T22:04:27.457 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001462262 2021-03-24T22:04:27.457 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.457 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.457 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:04:27.458 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.312496+0000", 2021-03-24T22:04:27.458 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0076004449999999999 2021-03-24T22:04:27.458 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.458 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.459 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:04:27.459 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.320096+0000", 2021-03-24T22:04:27.459 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.8819999999999993e-05 2021-03-24T22:04:27.459 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.459 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.460 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:04:27.460 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.320185+0000", 2021-03-24T22:04:27.460 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00019890500000000001 2021-03-24T22:04:27.460 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.460 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.461 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:04:27.461 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.320384+0000", 2021-03-24T22:04:27.461 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0717329640000002 2021-03-24T22:04:27.461 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:27.462 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:04:27.462 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:27.462 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.462 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.462 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:4148 7.21 7:86662589:::smithi12556470-19:head [write 1537867~645855] snapc 143=[143,140,13d,13b,139,137,136,12d,12b,129,128,127,126,120,11e,11b,11a,119,110,10f,10c,105,104,fa,f8,e7] ondisk+write+known_if_redirected e432)", 2021-03-24T22:04:27.463 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:00:02.257275+0000", 2021-03-24T22:04:27.463 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 265.04058863900002, 2021-03-24T22:04:27.463 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.067389844, 2021-03-24T22:04:27.463 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:04:27.464 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:04:27.464 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:04:27.464 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:04:27.464 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:04:27.464 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4148 2021-03-24T22:04:27.465 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.465 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:04:27.465 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.465 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:04:27.465 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.257275+0000", 2021-03-24T22:04:27.466 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:04:27.466 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.466 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.466 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:04:27.467 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.257275+0000", 2021-03-24T22:04:27.467 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999962 2021-03-24T22:04:27.467 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.467 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.467 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:04:27.468 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.257271+0000", 2021-03-24T22:04:27.468 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00017391499999999999 2021-03-24T22:04:27.468 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.468 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.468 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:04:27.469 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.257445+0000", 2021-03-24T22:04:27.469 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0049961399999999996 2021-03-24T22:04:27.469 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.469 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.469 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:04:27.470 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.262441+0000", 2021-03-24T22:04:27.470 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2265e-05 2021-03-24T22:04:27.470 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.470 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.471 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:04:27.471 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.262454+0000", 2021-03-24T22:04:27.471 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.013052934 2021-03-24T22:04:27.471 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.471 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.472 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.472 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.275507+0000", 2021-03-24T22:04:27.472 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.5969999999999998e-06 2021-03-24T22:04:27.472 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.473 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.473 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:04:27.473 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.275516+0000", 2021-03-24T22:04:27.473 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.96847310900000005 2021-03-24T22:04:27.473 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.474 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.474 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.474 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:03.243989+0000", 2021-03-24T22:04:27.474 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.626e-06 2021-03-24T22:04:27.474 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.475 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.475 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:04:27.475 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:03.243995+0000", 2021-03-24T22:04:27.475 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0563536250000001 2021-03-24T22:04:27.476 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.476 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.476 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.476 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.300349+0000", 2021-03-24T22:04:27.476 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.6805000000000007e-05 2021-03-24T22:04:27.477 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.477 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.477 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for degraded object", 2021-03-24T22:04:27.477 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.300435+0000", 2021-03-24T22:04:27.477 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.009989965 2021-03-24T22:04:27.478 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.478 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.478 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.478 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.310425+0000", 2021-03-24T22:04:27.479 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000105184 2021-03-24T22:04:27.479 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.479 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.479 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:04:27.479 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.310531+0000", 2021-03-24T22:04:27.480 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0099567490000000008 2021-03-24T22:04:27.480 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.480 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.480 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.480 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.320487+0000", 2021-03-24T22:04:27.481 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00014939599999999999 2021-03-24T22:04:27.481 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.481 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.481 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:04:27.482 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.320637+0000", 2021-03-24T22:04:27.482 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00031880400000000001 2021-03-24T22:04:27.482 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.482 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.482 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,5", 2021-03-24T22:04:27.483 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.320956+0000", 2021-03-24T22:04:27.483 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.003448596 2021-03-24T22:04:27.483 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.483 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.483 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:04:27.484 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.324404+0000", 2021-03-24T22:04:27.484 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.4194e-05 2021-03-24T22:04:27.484 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.484 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.485 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:04:27.485 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.324418+0000", 2021-03-24T22:04:27.485 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.0278000000000002e-05 2021-03-24T22:04:27.485 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.485 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.486 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:04:27.486 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.324469+0000", 2021-03-24T22:04:27.486 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.4826999999999994e-05 2021-03-24T22:04:27.486 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.486 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.487 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:04:27.487 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.324543+0000", 2021-03-24T22:04:27.487 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000121864 2021-03-24T22:04:27.487 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.487 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.488 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:04:27.488 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.324665+0000", 2021-03-24T22:04:27.488 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.067389844 2021-03-24T22:04:27.488 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:27.489 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:04:27.489 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:27.489 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.489 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.489 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:4149 7.21 7:86662589:::smithi12556470-19:head [write 2860355~652407] snapc 143=[143,140,13d,13b,139,137,136,12d,12b,129,128,127,126,120,11e,11b,11a,119,110,10f,10c,105,104,fa,f8,e7] ondisk+write+known_if_redirected e432)", 2021-03-24T22:04:27.490 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:00:02.264982+0000", 2021-03-24T22:04:27.490 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 265.03288195900001, 2021-03-24T22:04:27.490 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.060287465, 2021-03-24T22:04:27.490 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:04:27.490 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:04:27.491 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:04:27.491 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:04:27.491 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:04:27.491 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4149 2021-03-24T22:04:27.492 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.492 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:04:27.492 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.492 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:04:27.492 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.264982+0000", 2021-03-24T22:04:27.493 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:04:27.493 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.493 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.493 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:04:27.493 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.264982+0000", 2021-03-24T22:04:27.494 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999881 2021-03-24T22:04:27.494 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.494 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.494 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:04:27.495 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.264970+0000", 2021-03-24T22:04:27.495 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0010432900000000001 2021-03-24T22:04:27.495 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.495 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.495 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:04:27.496 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.266013+0000", 2021-03-24T22:04:27.496 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0014047409999999999 2021-03-24T22:04:27.496 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.496 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.497 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:04:27.497 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.267418+0000", 2021-03-24T22:04:27.497 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2789000000000001e-05 2021-03-24T22:04:27.497 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.497 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.498 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:04:27.498 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.267431+0000", 2021-03-24T22:04:27.498 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0081054939999999995 2021-03-24T22:04:27.498 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.498 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.499 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.499 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.275536+0000", 2021-03-24T22:04:27.499 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.7870000000000004e-06 2021-03-24T22:04:27.499 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.500 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.500 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:04:27.500 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.275543+0000", 2021-03-24T22:04:27.500 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.96846978399999994 2021-03-24T22:04:27.500 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.501 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.501 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.501 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:03.244013+0000", 2021-03-24T22:04:27.501 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6439999999999997e-06 2021-03-24T22:04:27.501 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.502 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.502 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:04:27.502 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:03.244019+0000", 2021-03-24T22:04:27.502 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.056445708 2021-03-24T22:04:27.503 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.503 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.503 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.503 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.300464+0000", 2021-03-24T22:04:27.503 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001583262 2021-03-24T22:04:27.504 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.504 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.504 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for degraded object", 2021-03-24T22:04:27.504 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.302048+0000", 2021-03-24T22:04:27.504 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0085309080000000002 2021-03-24T22:04:27.505 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.505 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.505 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.505 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.310579+0000", 2021-03-24T22:04:27.506 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.8817e-05 2021-03-24T22:04:27.506 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.506 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.506 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:04:27.506 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.310667+0000", 2021-03-24T22:04:27.507 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011484482000000001 2021-03-24T22:04:27.507 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.507 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.507 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.507 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.322152+0000", 2021-03-24T22:04:27.508 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.6207999999999999e-05 2021-03-24T22:04:27.508 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.508 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.508 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:04:27.509 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.322228+0000", 2021-03-24T22:04:27.509 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00014250999999999999 2021-03-24T22:04:27.509 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.509 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.509 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,5", 2021-03-24T22:04:27.510 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.322371+0000", 2021-03-24T22:04:27.510 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0020567459999999999 2021-03-24T22:04:27.510 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.510 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.510 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:04:27.511 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.324427+0000", 2021-03-24T22:04:27.511 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00031726900000000001 2021-03-24T22:04:27.511 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.511 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.511 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:04:27.512 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.324745+0000", 2021-03-24T22:04:27.512 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00031702800000000001 2021-03-24T22:04:27.512 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.512 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.513 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:04:27.513 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.325062+0000", 2021-03-24T22:04:27.513 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.4227999999999997e-05 2021-03-24T22:04:27.513 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.513 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.514 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:04:27.514 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.325106+0000", 2021-03-24T22:04:27.514 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00016376499999999999 2021-03-24T22:04:27.514 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.515 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.515 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:04:27.515 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.325270+0000", 2021-03-24T22:04:27.515 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.060287465 2021-03-24T22:04:27.515 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:27.516 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:04:27.516 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:27.516 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.516 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.517 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:4150 7.21 7:86662589:::smithi12556470-19:head [setxattr _header (58) in=7b,truncate 3512762] snapc 143=[143,140,13d,13b,139,137,136,12d,12b,129,128,127,126,120,11e,11b,11a,119,110,10f,10c,105,104,fa,f8,e7] ondisk+write+known_if_redirected e432)", 2021-03-24T22:04:27.517 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:00:02.267443+0000", 2021-03-24T22:04:27.517 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 265.030421585, 2021-03-24T22:04:27.517 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.058203818, 2021-03-24T22:04:27.517 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:04:27.518 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:04:27.518 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:04:27.518 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:04:27.518 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:04:27.519 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4150 2021-03-24T22:04:27.519 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.519 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:04:27.519 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.519 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:04:27.520 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.267443+0000", 2021-03-24T22:04:27.520 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:04:27.520 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.520 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.521 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:04:27.521 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.267443+0000", 2021-03-24T22:04:27.521 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T22:04:27.521 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.521 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.522 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:04:27.522 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.267441+0000", 2021-03-24T22:04:27.522 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.1050000000000001e-06 2021-03-24T22:04:27.522 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.522 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.523 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:04:27.523 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.267450+0000", 2021-03-24T22:04:27.523 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.1170000000000004e-06 2021-03-24T22:04:27.523 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.524 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.524 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:04:27.524 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.267458+0000", 2021-03-24T22:04:27.524 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6759999999999998e-06 2021-03-24T22:04:27.524 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.525 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.525 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:04:27.525 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.267463+0000", 2021-03-24T22:04:27.525 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0081025090000000008 2021-03-24T22:04:27.525 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.526 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.526 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.526 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.275565+0000", 2021-03-24T22:04:27.526 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4189999999999999e-06 2021-03-24T22:04:27.526 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.527 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.527 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:04:27.527 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.275572+0000", 2021-03-24T22:04:27.527 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.96846758399999999 2021-03-24T22:04:27.528 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.528 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.528 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.528 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:03.244039+0000", 2021-03-24T22:04:27.528 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5249999999999996e-06 2021-03-24T22:04:27.529 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.529 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.529 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:04:27.529 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:03.244045+0000", 2021-03-24T22:04:27.529 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.058038864 2021-03-24T22:04:27.530 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.530 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.530 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.530 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.302084+0000", 2021-03-24T22:04:27.531 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.0760000000000005e-05 2021-03-24T22:04:27.531 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.531 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.531 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for degraded object", 2021-03-24T22:04:27.531 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.302174+0000", 2021-03-24T22:04:27.532 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0085442020000000007 2021-03-24T22:04:27.532 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.532 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.532 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.532 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.310718+0000", 2021-03-24T22:04:27.533 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.1786999999999994e-05 2021-03-24T22:04:27.533 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.533 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.533 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:04:27.534 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.310810+0000", 2021-03-24T22:04:27.534 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012559539 2021-03-24T22:04:27.534 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.534 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.534 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.535 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.323370+0000", 2021-03-24T22:04:27.535 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000127611 2021-03-24T22:04:27.535 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.535 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.535 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:04:27.536 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.323497+0000", 2021-03-24T22:04:27.536 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000301575 2021-03-24T22:04:27.536 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.536 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.537 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,5", 2021-03-24T22:04:27.537 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.323799+0000", 2021-03-24T22:04:27.537 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00088734600000000001 2021-03-24T22:04:27.537 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.537 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.538 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:04:27.538 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.324686+0000", 2021-03-24T22:04:27.538 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00066636400000000002 2021-03-24T22:04:27.538 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.538 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.539 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:04:27.539 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.325353+0000", 2021-03-24T22:04:27.539 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.1169999999999997e-05 2021-03-24T22:04:27.539 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.540 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.540 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:04:27.540 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.325434+0000", 2021-03-24T22:04:27.540 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.8564000000000002e-05 2021-03-24T22:04:27.540 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.541 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.541 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:04:27.541 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.325512+0000", 2021-03-24T22:04:27.541 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00013387499999999999 2021-03-24T22:04:27.541 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.542 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.542 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:04:27.542 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.325646+0000", 2021-03-24T22:04:27.542 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.058203818 2021-03-24T22:04:27.543 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:27.543 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:04:27.543 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:27.543 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.543 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.544 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:4151 7.21 7:86662589:::smithi12556470-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e432)", 2021-03-24T22:04:27.544 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:00:02.267468+0000", 2021-03-24T22:04:27.544 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 265.03039650099998, 2021-03-24T22:04:27.544 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0584516220000002, 2021-03-24T22:04:27.544 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:04:27.545 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T22:04:27.545 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:04:27.545 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:04:27.545 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:04:27.546 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4151 2021-03-24T22:04:27.546 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.546 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:04:27.546 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.546 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:04:27.547 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.267468+0000", 2021-03-24T22:04:27.547 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:04:27.547 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.547 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.547 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:04:27.548 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.267468+0000", 2021-03-24T22:04:27.548 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999995 2021-03-24T22:04:27.548 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.548 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.549 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:04:27.549 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.267467+0000", 2021-03-24T22:04:27.549 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.3579999999999999e-06 2021-03-24T22:04:27.549 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.549 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.550 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:04:27.550 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.267470+0000", 2021-03-24T22:04:27.550 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.3289999999999999e-06 2021-03-24T22:04:27.550 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.550 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.551 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:04:27.551 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.267477+0000", 2021-03-24T22:04:27.551 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3869999999999997e-06 2021-03-24T22:04:27.551 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.552 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.552 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:04:27.552 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.267481+0000", 2021-03-24T22:04:27.552 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0081161340000000005 2021-03-24T22:04:27.552 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.553 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.553 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.553 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.275597+0000", 2021-03-24T22:04:27.553 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.925e-06 2021-03-24T22:04:27.553 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.554 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.554 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:04:27.554 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.275603+0000", 2021-03-24T22:04:27.554 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.96846318600000003 2021-03-24T22:04:27.555 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.555 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.555 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.555 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:03.244066+0000", 2021-03-24T22:04:27.555 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.4770000000000002e-06 2021-03-24T22:04:27.556 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.556 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.556 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:04:27.556 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:03.244072+0000", 2021-03-24T22:04:27.556 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.058135101 2021-03-24T22:04:27.557 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.557 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.557 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.557 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.302207+0000", 2021-03-24T22:04:27.558 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.2978000000000005e-05 2021-03-24T22:04:27.558 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.558 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.558 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for degraded object", 2021-03-24T22:04:27.558 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.302280+0000", 2021-03-24T22:04:27.559 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0085603620000000002 2021-03-24T22:04:27.559 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.559 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.559 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.559 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.310840+0000", 2021-03-24T22:04:27.560 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.5315999999999994e-05 2021-03-24T22:04:27.560 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.560 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.560 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:04:27.561 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.310926+0000", 2021-03-24T22:04:27.561 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.013304694000000001 2021-03-24T22:04:27.561 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.561 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.561 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.562 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.324230+0000", 2021-03-24T22:04:27.562 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00011130599999999999 2021-03-24T22:04:27.562 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.562 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.563 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:04:27.563 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.324342+0000", 2021-03-24T22:04:27.563 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0013441519999999999 2021-03-24T22:04:27.563 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.563 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.564 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.564 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.325686+0000", 2021-03-24T22:04:27.564 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5726999999999997e-05 2021-03-24T22:04:27.564 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.564 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.565 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:04:27.565 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.325741+0000", 2021-03-24T22:04:27.565 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00017782400000000001 2021-03-24T22:04:27.565 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.565 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.566 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:04:27.566 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.325919+0000", 2021-03-24T22:04:27.566 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0584516220000002 2021-03-24T22:04:27.566 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:27.567 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:04:27.567 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:27.567 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.567 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.567 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:4763 7.c 7:31cb714a:::smithi12556470-7:head [write 625392~569702] snapc 17d=[17d,17c,179,178,177,176,172,16e,16d,168,166,165,160,15e,159,157,153,139,136,129,127,11b,104,fa] ondisk+write+known_if_redirected e491)", 2021-03-24T22:04:27.568 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:01:01.275026+0000", 2021-03-24T22:04:27.568 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 206.022838334, 2021-03-24T22:04:27.568 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.06165336, 2021-03-24T22:04:27.568 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:04:27.568 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:04:27.569 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:04:27.569 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:04:27.569 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:04:27.569 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4763 2021-03-24T22:04:27.570 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.570 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:04:27.570 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.570 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:04:27.570 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.275026+0000", 2021-03-24T22:04:27.571 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:04:27.571 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.571 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.571 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:04:27.571 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.275026+0000", 2021-03-24T22:04:27.572 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999976 2021-03-24T22:04:27.572 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.572 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.572 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:04:27.573 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.275023+0000", 2021-03-24T22:04:27.573 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00032953799999999999 2021-03-24T22:04:27.573 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.573 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.573 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:04:27.574 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.275353+0000", 2021-03-24T22:04:27.574 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.3589999999999999e-05 2021-03-24T22:04:27.574 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.574 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.574 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:04:27.575 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.275377+0000", 2021-03-24T22:04:27.575 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.4150000000000005e-06 2021-03-24T22:04:27.575 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.575 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.576 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:04:27.576 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.275386+0000", 2021-03-24T22:04:27.576 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012973314999999999 2021-03-24T22:04:27.576 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.576 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.577 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.577 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.288359+0000", 2021-03-24T22:04:27.577 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.8028000000000006e-05 2021-03-24T22:04:27.577 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.577 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.578 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:04:27.578 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.288437+0000", 2021-03-24T22:04:27.578 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.039910114 2021-03-24T22:04:27.578 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.579 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.579 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.579 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.328347+0000", 2021-03-24T22:04:27.579 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00022543800000000001 2021-03-24T22:04:27.579 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.580 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.580 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:04:27.580 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.328573+0000", 2021-03-24T22:04:27.580 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00039166100000000002 2021-03-24T22:04:27.580 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.581 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.581 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,7", 2021-03-24T22:04:27.581 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.328965+0000", 2021-03-24T22:04:27.581 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0071784850000000001 2021-03-24T22:04:27.581 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.582 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.582 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:04:27.582 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.336143+0000", 2021-03-24T22:04:27.582 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00024411499999999999 2021-03-24T22:04:27.583 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.583 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.583 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:04:27.583 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.336387+0000", 2021-03-24T22:04:27.583 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.4493999999999995e-05 2021-03-24T22:04:27.584 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.584 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.584 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:04:27.584 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.336462+0000", 2021-03-24T22:04:27.585 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.0748999999999999e-05 2021-03-24T22:04:27.585 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.585 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.585 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:04:27.585 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.336502+0000", 2021-03-24T22:04:27.586 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00017668299999999999 2021-03-24T22:04:27.586 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.586 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.586 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:04:27.586 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.336679+0000", 2021-03-24T22:04:27.587 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.06165336 2021-03-24T22:04:27.587 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:27.587 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:04:27.587 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:27.587 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.588 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.588 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:4764 7.c 7:31cb714a:::smithi12556470-7:head [write 1637717~745548] snapc 17d=[17d,17c,179,178,177,176,172,16e,16d,168,166,165,160,15e,159,157,153,139,136,129,127,11b,104,fa] ondisk+write+known_if_redirected e491)", 2021-03-24T22:04:27.588 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:01:01.285992+0000", 2021-03-24T22:04:27.588 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 206.01187194299999, 2021-03-24T22:04:27.589 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0510351490000001, 2021-03-24T22:04:27.589 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:04:27.589 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:04:27.589 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:04:27.589 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:04:27.590 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:04:27.590 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4764 2021-03-24T22:04:27.590 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.590 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:04:27.590 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.591 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:04:27.591 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.285992+0000", 2021-03-24T22:04:27.591 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:04:27.591 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.592 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.592 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:04:27.592 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.285992+0000", 2021-03-24T22:04:27.592 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T22:04:27.592 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.593 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.593 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:04:27.593 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.285990+0000", 2021-03-24T22:04:27.593 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00046463299999999999 2021-03-24T22:04:27.594 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.594 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.594 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:04:27.594 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286455+0000", 2021-03-24T22:04:27.594 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.464e-05 2021-03-24T22:04:27.595 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.595 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.595 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:04:27.595 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286469+0000", 2021-03-24T22:04:27.595 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.6230000000000003e-06 2021-03-24T22:04:27.596 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.596 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.596 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:04:27.596 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286478+0000", 2021-03-24T22:04:27.596 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002016393 2021-03-24T22:04:27.597 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.597 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.597 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.597 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.288494+0000", 2021-03-24T22:04:27.598 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.5879000000000001e-05 2021-03-24T22:04:27.598 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.598 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.598 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:04:27.598 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.288520+0000", 2021-03-24T22:04:27.599 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.04205784 2021-03-24T22:04:27.599 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.599 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.599 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.599 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.330578+0000", 2021-03-24T22:04:27.600 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000176718 2021-03-24T22:04:27.600 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.600 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.600 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:04:27.601 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.330755+0000", 2021-03-24T22:04:27.601 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00047453099999999999 2021-03-24T22:04:27.601 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.601 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.601 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,7", 2021-03-24T22:04:27.602 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.331229+0000", 2021-03-24T22:04:27.602 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.005170661 2021-03-24T22:04:27.602 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.602 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.602 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:04:27.603 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.336400+0000", 2021-03-24T22:04:27.603 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00033368500000000001 2021-03-24T22:04:27.603 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.603 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.604 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:04:27.604 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.336734+0000", 2021-03-24T22:04:27.604 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.5241999999999996e-05 2021-03-24T22:04:27.604 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.604 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.605 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:04:27.605 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.336809+0000", 2021-03-24T22:04:27.605 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.8261999999999999e-05 2021-03-24T22:04:27.605 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.605 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.606 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:04:27.606 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.336867+0000", 2021-03-24T22:04:27.606 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000160037 2021-03-24T22:04:27.606 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.607 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.607 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:04:27.607 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.337027+0000", 2021-03-24T22:04:27.607 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0510351490000001 2021-03-24T22:04:27.607 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:27.608 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:04:27.608 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:27.608 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.608 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.608 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:4765 7.c 7:31cb714a:::smithi12556470-7:head [write 2848542~176850] snapc 17d=[17d,17c,179,178,177,176,172,16e,16d,168,166,165,160,15e,159,157,153,139,136,129,127,11b,104,fa] ondisk+write+known_if_redirected e491)", 2021-03-24T22:04:27.609 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:01:01.286489+0000", 2021-03-24T22:04:27.609 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 206.01137507199999, 2021-03-24T22:04:27.609 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0507179900000001, 2021-03-24T22:04:27.609 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:04:27.609 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:04:27.610 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:04:27.610 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:04:27.610 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:04:27.610 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4765 2021-03-24T22:04:27.611 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.611 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:04:27.611 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.611 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:04:27.611 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286489+0000", 2021-03-24T22:04:27.612 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:04:27.612 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.612 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.612 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:04:27.612 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286489+0000", 2021-03-24T22:04:27.613 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T22:04:27.613 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.613 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.613 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:04:27.614 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286488+0000", 2021-03-24T22:04:27.614 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000106881 2021-03-24T22:04:27.614 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.614 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.614 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:04:27.615 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286595+0000", 2021-03-24T22:04:27.615 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.9570000000000002e-06 2021-03-24T22:04:27.615 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.615 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.615 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:04:27.616 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286603+0000", 2021-03-24T22:04:27.616 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.7579999999999998e-06 2021-03-24T22:04:27.616 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.616 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.617 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:04:27.617 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286608+0000", 2021-03-24T22:04:27.617 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00353412 2021-03-24T22:04:27.617 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.617 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.618 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.618 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.290143+0000", 2021-03-24T22:04:27.618 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.6600000000000001e-06 2021-03-24T22:04:27.618 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.618 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.619 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:04:27.619 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.290151+0000", 2021-03-24T22:04:27.619 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.04322256 2021-03-24T22:04:27.619 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.620 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.620 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.620 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.333374+0000", 2021-03-24T22:04:27.620 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000160939 2021-03-24T22:04:27.620 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.621 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.621 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:04:27.621 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.333535+0000", 2021-03-24T22:04:27.621 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00051926299999999995 2021-03-24T22:04:27.621 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.622 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.622 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,7", 2021-03-24T22:04:27.622 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.334054+0000", 2021-03-24T22:04:27.622 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002355233 2021-03-24T22:04:27.623 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.623 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.623 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:04:27.623 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.336409+0000", 2021-03-24T22:04:27.623 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00036032499999999999 2021-03-24T22:04:27.624 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.624 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.624 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:04:27.624 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.336769+0000", 2021-03-24T22:04:27.624 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00030155200000000002 2021-03-24T22:04:27.625 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.625 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.625 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:04:27.625 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.337071+0000", 2021-03-24T22:04:27.625 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.5614000000000002e-05 2021-03-24T22:04:27.626 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.626 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.626 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:04:27.626 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.337107+0000", 2021-03-24T22:04:27.627 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000100364 2021-03-24T22:04:27.627 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.627 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.627 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:04:27.627 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.337207+0000", 2021-03-24T22:04:27.628 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0507179900000001 2021-03-24T22:04:27.628 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:27.628 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:04:27.628 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:27.628 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.629 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.629 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:4766 7.c 7:31cb714a:::smithi12556470-7:head [setxattr _header (57) in=7b,truncate 3025392] snapc 17d=[17d,17c,179,178,177,176,172,16e,16d,168,166,165,160,15e,159,157,153,139,136,129,127,11b,104,fa] ondisk+write+known_if_redirected e491)", 2021-03-24T22:04:27.629 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:01:01.286615+0000", 2021-03-24T22:04:27.629 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 206.01124893100001, 2021-03-24T22:04:27.630 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0508728430000001, 2021-03-24T22:04:27.630 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:04:27.630 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:04:27.630 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:04:27.630 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:04:27.631 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:04:27.631 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4766 2021-03-24T22:04:27.631 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.631 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:04:27.631 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.632 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:04:27.632 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286615+0000", 2021-03-24T22:04:27.632 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:04:27.632 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.633 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.633 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:04:27.633 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286615+0000", 2021-03-24T22:04:27.633 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T22:04:27.633 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.634 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.634 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:04:27.634 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286614+0000", 2021-03-24T22:04:27.634 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6700000000000002e-06 2021-03-24T22:04:27.634 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.635 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.635 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:04:27.635 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286619+0000", 2021-03-24T22:04:27.635 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.3249999999999998e-06 2021-03-24T22:04:27.636 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.636 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.636 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:04:27.636 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286624+0000", 2021-03-24T22:04:27.636 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6290000000000001e-06 2021-03-24T22:04:27.637 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.637 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.637 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:04:27.637 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286629+0000", 2021-03-24T22:04:27.637 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0035533230000000002 2021-03-24T22:04:27.638 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.638 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.638 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.638 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.290182+0000", 2021-03-24T22:04:27.639 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.5209999999999999e-06 2021-03-24T22:04:27.639 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.639 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.639 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:04:27.639 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.290189+0000", 2021-03-24T22:04:27.640 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.044822132 2021-03-24T22:04:27.640 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.640 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.640 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.640 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.335011+0000", 2021-03-24T22:04:27.641 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012871200000000001 2021-03-24T22:04:27.641 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.641 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.641 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:04:27.642 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.335140+0000", 2021-03-24T22:04:27.642 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000225832 2021-03-24T22:04:27.642 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.642 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.642 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,7", 2021-03-24T22:04:27.643 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.335365+0000", 2021-03-24T22:04:27.643 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0010523500000000001 2021-03-24T22:04:27.643 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.643 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.643 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:04:27.644 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.336418+0000", 2021-03-24T22:04:27.644 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00084202099999999998 2021-03-24T22:04:27.644 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.644 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.645 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:04:27.645 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.337260+0000", 2021-03-24T22:04:27.645 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.0683999999999999e-05 2021-03-24T22:04:27.645 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.645 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.646 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:04:27.646 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.337300+0000", 2021-03-24T22:04:27.646 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.4521999999999999e-05 2021-03-24T22:04:27.646 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.646 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.647 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:04:27.647 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.337335+0000", 2021-03-24T22:04:27.647 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00015305900000000001 2021-03-24T22:04:27.647 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.647 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.648 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:04:27.648 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.337488+0000", 2021-03-24T22:04:27.648 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0508728430000001 2021-03-24T22:04:27.648 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:27.649 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:04:27.649 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:27.649 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.649 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.649 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:4767 7.c 7:31cb714a:::smithi12556470-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e491)", 2021-03-24T22:04:27.650 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:01:01.286635+0000", 2021-03-24T22:04:27.650 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 206.011229437, 2021-03-24T22:04:27.650 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.051186642, 2021-03-24T22:04:27.650 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:04:27.650 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T22:04:27.651 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:04:27.651 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:04:27.651 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:04:27.651 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4767 2021-03-24T22:04:27.652 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.652 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:04:27.652 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.652 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:04:27.652 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286635+0000", 2021-03-24T22:04:27.653 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:04:27.653 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.653 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.653 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:04:27.653 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286635+0000", 2021-03-24T22:04:27.654 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999995 2021-03-24T22:04:27.654 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.654 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.654 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:04:27.655 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286634+0000", 2021-03-24T22:04:27.655 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.633e-06 2021-03-24T22:04:27.655 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.655 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.655 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:04:27.656 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286637+0000", 2021-03-24T22:04:27.656 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.7699999999999998e-06 2021-03-24T22:04:27.656 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.656 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.656 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:04:27.657 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286643+0000", 2021-03-24T22:04:27.657 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6349999999999997e-06 2021-03-24T22:04:27.657 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.657 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.657 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:04:27.658 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286647+0000", 2021-03-24T22:04:27.658 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0037662949999999998 2021-03-24T22:04:27.658 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.658 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.659 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.659 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.290414+0000", 2021-03-24T22:04:27.659 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.4619999999999996e-06 2021-03-24T22:04:27.659 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.659 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.660 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:04:27.660 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.290422+0000", 2021-03-24T22:04:27.660 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0454484660000001 2021-03-24T22:04:27.660 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.660 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.661 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.661 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.335870+0000", 2021-03-24T22:04:27.661 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010779400000000001 2021-03-24T22:04:27.661 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.662 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.662 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:04:27.662 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.335978+0000", 2021-03-24T22:04:27.662 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001556396 2021-03-24T22:04:27.662 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.663 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.663 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.663 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.337535+0000", 2021-03-24T22:04:27.663 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010119999999999999 2021-03-24T22:04:27.663 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.664 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.664 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:04:27.664 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.337636+0000", 2021-03-24T22:04:27.664 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00018547400000000001 2021-03-24T22:04:27.665 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.665 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.665 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:04:27.665 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.337821+0000", 2021-03-24T22:04:27.665 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.051186642 2021-03-24T22:04:27.666 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:27.666 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:04:27.666 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:27.666 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.666 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.667 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:4768 7.21 7:86662589:::smithi12556470-19:head [stat,delete] snapc 17d=[17d,17c,179,178,177,176,172,16e,16d,168,166,165,160,15e,159,157,153,139,136,129,127,11b,104,fa] ondisk+write+known_if_redirected e491)", 2021-03-24T22:04:27.667 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:01:01.286651+0000", 2021-03-24T22:04:27.667 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 206.01121300299999, 2021-03-24T22:04:27.667 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0516308409999999, 2021-03-24T22:04:27.668 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:04:27.668 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:04:27.668 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:04:27.668 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:04:27.668 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:04:27.669 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4768 2021-03-24T22:04:27.669 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.669 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:04:27.669 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.669 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:04:27.670 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286651+0000", 2021-03-24T22:04:27.670 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:04:27.670 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.670 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.670 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:04:27.671 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286651+0000", 2021-03-24T22:04:27.671 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999995 2021-03-24T22:04:27.671 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.671 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.672 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:04:27.672 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286651+0000", 2021-03-24T22:04:27.672 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.4959999999999999e-06 2021-03-24T22:04:27.672 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.672 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.673 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:04:27.673 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286653+0000", 2021-03-24T22:04:27.673 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.7709999999999997e-06 2021-03-24T22:04:27.673 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.673 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.674 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:04:27.674 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286658+0000", 2021-03-24T22:04:27.674 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.66e-06 2021-03-24T22:04:27.674 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.675 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.675 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:04:27.675 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286664+0000", 2021-03-24T22:04:27.675 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0010190959999999999 2021-03-24T22:04:27.675 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.676 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.676 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.676 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.287683+0000", 2021-03-24T22:04:27.676 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.145e-05 2021-03-24T22:04:27.676 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.677 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.677 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:04:27.677 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.287694+0000", 2021-03-24T22:04:27.677 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.048107345 2021-03-24T22:04:27.678 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.678 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.678 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.678 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.335801+0000", 2021-03-24T22:04:27.678 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00025443900000000002 2021-03-24T22:04:27.679 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.679 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.679 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:04:27.679 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.336056+0000", 2021-03-24T22:04:27.679 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00044708600000000002 2021-03-24T22:04:27.680 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.680 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.680 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 5,7", 2021-03-24T22:04:27.680 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.336503+0000", 2021-03-24T22:04:27.681 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0013468359999999999 2021-03-24T22:04:27.681 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.681 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.681 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:04:27.681 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.337850+0000", 2021-03-24T22:04:27.682 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.1340999999999993e-05 2021-03-24T22:04:27.682 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.682 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.682 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:04:27.682 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.337921+0000", 2021-03-24T22:04:27.683 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.0593999999999999e-05 2021-03-24T22:04:27.683 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.683 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.683 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:04:27.684 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.337952+0000", 2021-03-24T22:04:27.684 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00011551 2021-03-24T22:04:27.684 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.684 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.684 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:04:27.685 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.338067+0000", 2021-03-24T22:04:27.685 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00021471399999999999 2021-03-24T22:04:27.685 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.685 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.685 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:04:27.686 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.338282+0000", 2021-03-24T22:04:27.686 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0516308409999999 2021-03-24T22:04:27.686 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:27.686 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:04:27.687 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:27.687 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.687 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.687 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:6371 7.10 7:0ece3580:::smithi12556470-35:head [read 0~2487634,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e655)", 2021-03-24T22:04:27.687 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:03:45.401411+0000", 2021-03-24T22:04:27.688 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 41.896452947999997, 2021-03-24T22:04:27.688 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.056283334, 2021-03-24T22:04:27.688 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:04:27.688 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T22:04:27.688 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:04:27.689 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:04:27.689 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:04:27.689 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 6371 2021-03-24T22:04:27.689 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.690 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:04:27.690 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.690 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:04:27.690 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:45.401411+0000", 2021-03-24T22:04:27.690 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:04:27.691 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.691 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.691 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:04:27.691 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:45.401411+0000", 2021-03-24T22:04:27.691 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999971 2021-03-24T22:04:27.692 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.692 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.692 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:04:27.692 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:45.401409+0000", 2021-03-24T22:04:27.693 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.7960000000000007e-06 2021-03-24T22:04:27.693 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.693 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.693 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:04:27.693 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:45.401418+0000", 2021-03-24T22:04:27.694 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1674999999999999e-05 2021-03-24T22:04:27.694 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.694 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.694 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:04:27.694 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:45.401430+0000", 2021-03-24T22:04:27.695 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.7100000000000002e-06 2021-03-24T22:04:27.695 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.695 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.695 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:04:27.696 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:45.401440+0000", 2021-03-24T22:04:27.696 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0015687570000000001 2021-03-24T22:04:27.696 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.696 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.696 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.697 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:45.403008+0000", 2021-03-24T22:04:27.697 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0859000000000001e-05 2021-03-24T22:04:27.697 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.697 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.697 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:04:27.698 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:45.403019+0000", 2021-03-24T22:04:27.698 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0516452110000001 2021-03-24T22:04:27.698 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.698 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.699 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.699 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:46.454665+0000", 2021-03-24T22:04:27.699 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00014620900000000001 2021-03-24T22:04:27.699 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.699 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.700 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:04:27.700 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:46.454811+0000", 2021-03-24T22:04:27.700 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0028837509999999999 2021-03-24T22:04:27.700 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.700 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.701 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:04:27.701 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:46.457694+0000", 2021-03-24T22:04:27.701 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.056283334 2021-03-24T22:04:27.701 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:27.701 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:04:27.702 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:27.702 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.702 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.702 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:6372 7.10 7:0ece3580:::smithi12556470-35:head [sparse-read 0~2065878] snapc 0=[] ondisk+read+known_if_redirected e655)", 2021-03-24T22:04:27.703 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:03:45.401450+0000", 2021-03-24T22:04:27.703 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 41.896414163999999, 2021-03-24T22:04:27.703 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0567293149999999, 2021-03-24T22:04:27.703 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:04:27.703 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T22:04:27.704 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:04:27.704 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:04:27.704 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:04:27.704 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 6372 2021-03-24T22:04:27.704 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.705 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:04:27.705 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.705 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:04:27.705 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:45.401450+0000", 2021-03-24T22:04:27.706 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:04:27.706 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.706 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.706 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:04:27.706 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:45.401450+0000", 2021-03-24T22:04:27.707 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999995 2021-03-24T22:04:27.707 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.707 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.707 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:04:27.707 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:45.401449+0000", 2021-03-24T22:04:27.708 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.281e-06 2021-03-24T22:04:27.708 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.708 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.708 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:04:27.709 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:45.401453+0000", 2021-03-24T22:04:27.709 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.3589999999999999e-06 2021-03-24T22:04:27.709 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.709 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.709 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:04:27.710 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:45.401458+0000", 2021-03-24T22:04:27.710 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.4800000000000003e-06 2021-03-24T22:04:27.710 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.710 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.711 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:04:27.711 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:45.401462+0000", 2021-03-24T22:04:27.711 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0027488669999999999 2021-03-24T22:04:27.711 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.711 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.712 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.712 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:45.404211+0000", 2021-03-24T22:04:27.712 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0886e-05 2021-03-24T22:04:27.712 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.713 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.713 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:04:27.713 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:45.404222+0000", 2021-03-24T22:04:27.713 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.053514386 2021-03-24T22:04:27.713 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.714 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.714 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.714 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:46.457737+0000", 2021-03-24T22:04:27.714 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00014444600000000001 2021-03-24T22:04:27.715 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.715 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.715 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:04:27.715 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:46.457881+0000", 2021-03-24T22:04:27.715 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00029818999999999999 2021-03-24T22:04:27.716 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.716 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.716 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:04:27.716 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:46.458179+0000", 2021-03-24T22:04:27.716 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0567293149999999 2021-03-24T22:04:27.717 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:27.717 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:04:27.717 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:27.717 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.718 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.718 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:6373 7.10 7:0ece3580:::smithi12556470-35:head [read 0~2487634,checksum] snapc 0=[] ondisk+read+known_if_redirected e655)", 2021-03-24T22:04:27.718 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:03:45.401467+0000", 2021-03-24T22:04:27.718 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 41.896397325999999, 2021-03-24T22:04:27.718 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.057669988, 2021-03-24T22:04:27.719 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:04:27.719 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T22:04:27.719 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:04:27.719 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:04:27.719 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:04:27.720 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 6373 2021-03-24T22:04:27.720 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.720 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:04:27.720 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.721 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:04:27.721 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:45.401467+0000", 2021-03-24T22:04:27.721 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:04:27.721 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.721 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.722 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:04:27.722 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:45.401467+0000", 2021-03-24T22:04:27.722 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999995 2021-03-24T22:04:27.722 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.722 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.723 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:04:27.723 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:45.401466+0000", 2021-03-24T22:04:27.723 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.6430000000000001e-06 2021-03-24T22:04:27.723 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.723 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.724 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:04:27.724 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:45.401470+0000", 2021-03-24T22:04:27.724 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2900000000000002e-06 2021-03-24T22:04:27.724 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.725 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.725 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:04:27.725 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:45.401475+0000", 2021-03-24T22:04:27.725 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.0239999999999999e-06 2021-03-24T22:04:27.725 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.726 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.726 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:04:27.726 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:45.401479+0000", 2021-03-24T22:04:27.726 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0027671940000000002 2021-03-24T22:04:27.727 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.727 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.727 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.727 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:45.404246+0000", 2021-03-24T22:04:27.727 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.8539999999999999e-06 2021-03-24T22:04:27.728 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.728 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.728 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:04:27.728 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:45.404253+0000", 2021-03-24T22:04:27.728 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.053960521 2021-03-24T22:04:27.729 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.729 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.729 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:04:27.729 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:46.458214+0000", 2021-03-24T22:04:27.730 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00015035600000000001 2021-03-24T22:04:27.730 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.730 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.730 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:04:27.730 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:46.458364+0000", 2021-03-24T22:04:27.731 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000772571 2021-03-24T22:04:27.731 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:04:27.731 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:04:27.731 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:04:27.731 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:46.459137+0000", 2021-03-24T22:04:27.732 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.057669988 2021-03-24T22:04:27.732 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:27.732 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:04:27.732 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:27.732 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:04:27.733 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:04:27.733 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-24T22:04:27.733 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:27.736 INFO:tasks.rados.rados.0.smithi125.stdout:2640: done (1 left) 2021-03-24T22:04:27.737 INFO:tasks.rados.rados.0.smithi125.stdout:2641: done (0 left) 2021-03-24T22:04:27.737 INFO:tasks.rados.rados.0.smithi125.stdout:2642: write (excl) oid 2 current snap is 305 2021-03-24T22:04:27.737 INFO:tasks.rados.rados.0.smithi125.stdout:2642: seq_num 917 ranges {612019=565693,1810929=602646,3082649=329370} 2021-03-24T22:04:27.737 INFO:tasks.rados.rados.0.smithi125.stdout:2642: writing smithi12556470-2 from 612019 to 1177712 tid 1 2021-03-24T22:04:27.737 INFO:tasks.rados.rados.0.smithi125.stdout:2642: writing smithi12556470-2 from 1810929 to 2413575 tid 2 2021-03-24T22:04:27.738 INFO:tasks.rados.rados.0.smithi125.stdout:2642: writing smithi12556470-2 from 3082649 to 3412019 tid 3 2021-03-24T22:04:27.738 INFO:tasks.rados.rados.0.smithi125.stdout:2643: write oid 10 current snap is 305 2021-03-24T22:04:27.738 INFO:tasks.rados.rados.0.smithi125.stdout:2643: seq_num 918 ranges {628826=640942,1869411=462657,2826425=534649} 2021-03-24T22:04:27.738 INFO:tasks.rados.rados.0.smithi125.stdout:2643: writing smithi12556470-10 from 628826 to 1269768 tid 1 2021-03-24T22:04:27.739 INFO:tasks.rados.rados.0.smithi125.stdout:2643: writing smithi12556470-10 from 1869411 to 2332068 tid 2 2021-03-24T22:04:27.739 INFO:tasks.rados.rados.0.smithi125.stdout:2643: writing smithi12556470-10 from 2826425 to 3361074 tid 3 2021-03-24T22:04:27.739 INFO:tasks.rados.rados.0.smithi125.stdout:2644: snap_create 2021-03-24T22:04:27.739 INFO:tasks.rados.rados.0.smithi125.stdout:2642: finishing write tid 1 to smithi12556470-2 2021-03-24T22:04:27.739 INFO:tasks.rados.rados.0.smithi125.stdout:2642: finishing write tid 2 to smithi12556470-2 2021-03-24T22:04:27.740 INFO:tasks.rados.rados.0.smithi125.stdout:2642: finishing write tid 3 to smithi12556470-2 2021-03-24T22:04:27.740 INFO:tasks.rados.rados.0.smithi125.stdout:2642: finishing write tid 4 to smithi12556470-2 2021-03-24T22:04:27.740 INFO:tasks.rados.rados.0.smithi125.stdout:2642: finishing write tid 5 to smithi12556470-2 2021-03-24T22:04:27.740 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 2 v 1001 (ObjNum 917 snap 305 seq_num 917) dirty exists 2021-03-24T22:04:27.741 INFO:tasks.rados.rados.0.smithi125.stdout:2642: left oid 2 (ObjNum 917 snap 305 seq_num 917) 2021-03-24T22:04:27.741 INFO:tasks.rados.rados.0.smithi125.stdout:2643: finishing write tid 1 to smithi12556470-10 2021-03-24T22:04:27.741 INFO:tasks.rados.rados.0.smithi125.stdout:2643: finishing write tid 2 to smithi12556470-10 2021-03-24T22:04:27.741 INFO:tasks.rados.rados.0.smithi125.stdout:2643: finishing write tid 3 to smithi12556470-10 2021-03-24T22:04:27.741 INFO:tasks.rados.rados.0.smithi125.stdout:2643: finishing write tid 4 to smithi12556470-10 2021-03-24T22:04:27.742 INFO:tasks.rados.rados.0.smithi125.stdout:2643: finishing write tid 5 to smithi12556470-10 2021-03-24T22:04:27.742 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 10 v 1039 (ObjNum 918 snap 305 seq_num 918) dirty exists 2021-03-24T22:04:27.742 INFO:tasks.rados.rados.0.smithi125.stdout:2643: left oid 10 (ObjNum 918 snap 305 seq_num 918) 2021-03-24T22:04:27.777 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:27.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:27.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:27.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:27 smithi142 conmon[20125]: cluster 2021-03-24T22:04:27.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:27 smithi142 conmon[20125]: 2021-03-24T22:04:26 2021-03-24T22:04:27.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:27 smithi142 conmon[20125]: .546659+0000 mon.a (mon.0) 1832 : cluster [DBG] osdmap e696: 8 total, 7 up, 8 in 2021-03-24T22:04:27.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:27 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:04:27.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:27 smithi142 conmon[20125]: -24T22:04:26.640128+0000 osd.6 (osd.6) 63 : cluster [DBG] 4.a scrub ok 2021-03-24T22:04:27.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:27 smithi142 conmon[20125]: audit 2021-03-24T22: 2021-03-24T22:04:27.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:27 smithi142 conmon[20125]: 04:26.701031+0000 mon.a (mon.0) 1833 : audit [DBG] 2021-03-24T22:04:27.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:27 smithi142 conmon[20125]: from='client.? v1:172.21.15.125:0/937064691' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:04:27.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:27 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:04:27.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:27 smithi142 conmon[20125]: -24T22:04:27.142655+0000 osd.0 (osd.0) 133 : cluster 2021-03-24T22:04:27.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:27 smithi142 conmon[20125]: [DBG] 5.5 scrub ok 2021-03-24T22:04:27.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:27 smithi142 conmon[20125]: cluster 2021-03-24T22:04:27.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:27 smithi142 conmon[20125]: 2021-03-24T22:04:27.240492+0000 osd.3 (osd.3) 140 2021-03-24T22:04:27.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:27 smithi142 conmon[20125]: : cluster [DBG] 3.1e scrub ok 2021-03-24T22:04:27.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:27 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:04:27.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:27 smithi142 conmon[20125]: -03-24T22:04:27.335474+0000 osd.7 (osd.7 2021-03-24T22:04:27.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:27 smithi142 conmon[20125]: ) 111 : cluster [DBG] 5.1a scrub ok 2021-03-24T22:04:27.883 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:27.894 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:27 smithi125 conmon[19936]: cluster 2021-03-24T22:04:26.546659+0000 mon.a (mon.0) 2021-03-24T22:04:27.895 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:27 smithi125 conmon[19936]: 1832 : cluster [DBG] osdmap e696: 8 total, 7 up, 8 in 2021-03-24T22:04:27.895 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:27 smithi125 conmon[19936]: cluster 2021-03-24T22:04:26.640128+0000 osd.6 (osd.6) 63 : cluster [DBG] 4.a scrub ok 2021-03-24T22:04:27.895 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:27 smithi125 conmon[19936]: audit 2021-03-24T22:04:26.701031+0000 2021-03-24T22:04:27.895 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:27 smithi125 conmon[19936]: mon.a (mon.0) 1833 : audit [DBG] from='client.? v1:172.21.15.125:0/937064691' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:04:27.896 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:27 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T22:04:27.896 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:27 smithi125 conmon[19936]: -24T22:04:27.142655+0000 osd.0 (osd.0 2021-03-24T22:04:27.896 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:27 smithi125 conmon[19936]: ) 133 : cluster [DBG] 5.5 scrub ok 2021-03-24T22:04:27.896 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:27 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:04:27.897 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:27 smithi125 conmon[19936]: T22:04:27.240492+0000 osd.3 (osd.3 2021-03-24T22:04:27.897 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:27 smithi125 conmon[19936]: ) 140 : cluster [DBG] 3.1e scrub ok 2021-03-24T22:04:27.897 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:27 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T22:04:27.897 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:27 smithi125 conmon[19936]: 22:04:27.335474+0000 osd.7 (osd.7) 2021-03-24T22:04:27.897 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:27 smithi125 conmon[19936]: 111 : cluster [DBG] 5.1a scrub ok 2021-03-24T22:04:27.898 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:27 smithi125 conmon[24927]: cluster 2021-03-24T22:04:26.546659+0000 2021-03-24T22:04:27.898 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:27 smithi125 conmon[24927]: mon.a (mon.0) 1832 : cluster [DBG] osdmap e696: 8 total, 7 up, 8 in 2021-03-24T22:04:27.898 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:27 smithi125 conmon[24927]: cluster 2021-03-24T22:04:27.899 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:27 smithi125 conmon[24927]: 2021 2021-03-24T22:04:27.899 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:27 smithi125 conmon[24927]: -03-24T22:04:26.640128 2021-03-24T22:04:27.899 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:27 smithi125 conmon[24927]: +0000 osd.6 (osd.6) 63 : cluster [DBG] 4.a scrub ok 2021-03-24T22:04:27.899 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:27 smithi125 conmon[24927]: audit 2021-03-24T 2021-03-24T22:04:27.899 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:27 smithi125 conmon[24927]: 22:04:26.701031+0000 mon.a (mon.0) 1833 : audit [DBG] from='client.? v1:172.21.15.125:0/937064691' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:04:27.900 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:27 smithi125 conmon[24927]: 2021-03-24T22:04:27.900 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:27 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:04:27.900 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:27 smithi125 conmon[24927]: -03-24T22:04:27.142655+0000 osd.0 (osd.0 2021-03-24T22:04:27.900 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:27 smithi125 conmon[24927]: ) 133 : cluster [DBG] 5.5 scrub ok 2021-03-24T22:04:27.901 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:27 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:04:27.901 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:27 smithi125 conmon[24927]: 04:27.240492+0000 osd.3 (osd.3) 140 2021-03-24T22:04:27.901 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:27 smithi125 conmon[24927]: : cluster [DBG] 3.1e scrub ok 2021-03-24T22:04:27.901 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:27 smithi125 conmon[24927]: cluster 2021-03-24T22 2021-03-24T22:04:27.902 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:27 smithi125 conmon[24927]: :04:27.335474+0000 osd.7 (osd.7 2021-03-24T22:04:27.902 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:27 smithi125 conmon[24927]: ) 111 : cluster [DBG] 5.1a scrub ok 2021-03-24T22:04:27.933 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:27.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:27.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:28.038 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:28.098 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:28.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:28.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:28.203 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:28.259 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:28.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:28.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:04:28.366 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:28.432 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:28.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:28.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:28.538 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:28.575 INFO:tasks.rados.rados.0.smithi125.stdout:2642: done (2 left) 2021-03-24T22:04:28.575 INFO:tasks.rados.rados.0.smithi125.stdout:2643: done (1 left) 2021-03-24T22:04:28.576 INFO:tasks.rados.rados.0.smithi125.stdout:2644: done (0 left) 2021-03-24T22:04:28.576 INFO:tasks.rados.rados.0.smithi125.stdout:2645: snap_remove snap 287 2021-03-24T22:04:28.592 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:28.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:28.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:28.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:28 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:04:28.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:28 smithi125 conmon[19936]: 03-24T22:04:26.954974+0000 mgr.y (mgr.14144) 623 : cluster [DBG] pgmap v1207: 168 pgs: 1 peering, 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 40 active+undersized, 19 active+undersized+degraded, 106 active+clean; 614 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 6.2 MiB/s rd, 3.1 MiB/s wr, 17 op/s; 150/1845 objects degraded (8.130%); 40/1845 objects misplaced (2.168%); 38 B/s, 0 objects/s recovering 2021-03-24T22:04:28.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:28 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:04:28.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:28 smithi125 conmon[19936]: -03-24T22:04:27.542207+0000 mon.a ( 2021-03-24T22:04:28.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:28 smithi125 conmon[19936]: mon.0) 1834 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:04:28.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:28 smithi125 conmon[19936]: 2021-03-24T22:04:28.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:28 smithi125 conmon[19936]: cluster 2021-03-24T22:04:28.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:28 smithi125 conmon[19936]: 2021-03-24T22:04:27. 2021-03-24T22:04:28.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:28 smithi125 conmon[19936]: 560397+0000 mon.a (mon.0 2021-03-24T22:04:28.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:28 smithi125 conmon[19936]: ) 1835 : cluster [DBG] osdmap e697: 8 total, 7 up, 8 in 2021-03-24T22:04:28.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:28 smithi125 conmon[19936]: 2021-03-24T22:04:28.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:28 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T22:04:28.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:28 smithi125 conmon[19936]: -24T22:04:27. 2021-03-24T22:04:28.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:28 smithi125 conmon[19936]: 649853+0000 osd.1 (osd. 2021-03-24T22:04:28.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:28 smithi125 conmon[19936]: 1) 67 : cluster [DBG] 5.1d scrub ok 2021-03-24T22:04:28.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:28 smithi125 conmon[19936]: 2021-03-24T22:04:28.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:28 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:04:28.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:28 smithi125 conmon[19936]: 04:28.159268+0000 osd.0 (osd. 2021-03-24T22:04:28.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:28 smithi125 conmon[19936]: 0) 134 : cluster [DBG] 3.23 scrub ok 2021-03-24T22:04:28.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:28 smithi125 conmon[19936]: cluster 2021-03-24T22:04 2021-03-24T22:04:28.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:28 smithi125 conmon[19936]: :28.210926+0000 osd.3 (osd.3) 2021-03-24T22:04:28.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:28 smithi125 conmon[19936]: 141 : cluster [DBG] 7.1 scrub ok 2021-03-24T22:04:28.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:28 smithi125 conmon[19936]: cluster 2021-03-24T22:04:28 2021-03-24T22:04:28.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:28 smithi125 conmon[19936]: .355690+0000 osd.7 (osd.7) 112 2021-03-24T22:04:28.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:28 smithi125 conmon[19936]: : cluster [DBG] 5.9 scrub ok 2021-03-24T22:04:28.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:28 smithi125 conmon[24927]: cluster 2021-03-24T22:04:26.954974 2021-03-24T22:04:28.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:28 smithi125 conmon[24927]: +0000 mgr.y (mgr.14144) 623 : cluster [DBG] pgmap v1207: 168 pgs: 1 peering, 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 40 active+undersized, 19 active+undersized+degraded, 106 active+clean; 614 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 6.2 MiB/s rd, 3.1 MiB/s wr, 17 op/s; 150/1845 objects degraded (8.130%); 40/1845 objects misplaced (2.168%); 38 B/s, 0 objects/s recovering 2021-03-24T22:04:28.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:28 smithi125 conmon[24927]: cluster 2021-03-24T22:04:28.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:28 smithi125 conmon[24927]: 2021-03-24T22:04:27.542207 2021-03-24T22:04:28.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:28 smithi125 conmon[24927]: +0000 mon.a (mon.0) 1834 : cluster 2021-03-24T22:04:28.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:28 smithi125 conmon[24927]: [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:04:28.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:28 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:04:28.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:28 smithi125 conmon[24927]: -03-24T22:04:27.560397+0000 2021-03-24T22:04:28.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:28 smithi125 conmon[24927]: mon.a (mon.0) 1835 : cluster 2021-03-24T22:04:28.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:28 smithi125 conmon[24927]: [DBG] osdmap e697: 8 total, 7 up, 8 in 2021-03-24T22:04:28.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:28 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T22:04:28.672 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:28 smithi125 conmon[24927]: -24T22:04:27.649853+0000 osd.1 2021-03-24T22:04:28.672 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:28 smithi125 conmon[24927]: (osd.1) 67 : cluster [DBG] 5.1d scrub ok 2021-03-24T22:04:28.672 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:28 smithi125 conmon[24927]: 2021-03-24T22:04:28.672 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:28 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:04:28.672 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:28 smithi125 conmon[24927]: 04:28.159268+0000 osd.0 (osd.0) 2021-03-24T22:04:28.673 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:28 smithi125 conmon[24927]: 134 : cluster [DBG] 3.23 scrub ok 2021-03-24T22:04:28.673 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:28 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:04:28.673 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:28 smithi125 conmon[24927]: 03-24T22:04:28.210926+0000 2021-03-24T22:04:28.673 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:28 smithi125 conmon[24927]: osd.3 (osd.3) 141 : cluster [DBG] 2021-03-24T22:04:28.674 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:28 smithi125 conmon[24927]: 7.1 scrub ok 2021-03-24T22:04:28.674 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:28 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:04:28.674 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:28 smithi125 conmon[24927]: 04:28.355690+0000 osd.7 (osd.7 2021-03-24T22:04:28.674 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:28 smithi125 conmon[24927]: ) 112 : cluster [DBG] 5.9 scrub ok 2021-03-24T22:04:28.696 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:28.745 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:28.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:28.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:28.851 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:28.862 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:28 smithi142 conmon[20125]: cluster 2021-03-24T22:04:26.954974+0000 mgr.y (mgr.14144) 623 : cluster 2021-03-24T22:04:28.862 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:28 smithi142 conmon[20125]: [DBG] pgmap v1207: 168 pgs: 1 peering, 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 40 active+undersized, 19 active+undersized+degraded, 106 active+clean; 614 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 6.2 MiB/s rd, 3.1 MiB/s wr, 17 op/s; 150/1845 objects degraded (8.130%); 40/1845 objects misplaced (2.168%); 38 B/s, 0 objects/s recovering 2021-03-24T22:04:28.863 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:28 smithi142 conmon[20125]: cluster 2021-03-24T22:04:27.542207+0000 mon.a 2021-03-24T22:04:28.863 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:28 smithi142 conmon[20125]: (mon.0) 1834 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:04:28.863 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:28 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:04:28.863 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:28 smithi142 conmon[20125]: T22:04:27.560397+0000 mon.a 2021-03-24T22:04:28.864 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:28 smithi142 conmon[20125]: (mon.0) 1835 : cluster [DBG] 2021-03-24T22:04:28.864 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:28 smithi142 conmon[20125]: osdmap e697: 8 total, 7 up, 8 in 2021-03-24T22:04:28.864 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:28 smithi142 conmon[20125]: cluster 2021-03-24T22:04:28.864 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:28 smithi142 conmon[20125]: 2021-03-24T22 2021-03-24T22:04:28.865 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:28 smithi142 conmon[20125]: :04:27.649853+0000 osd.1 (osd.1 2021-03-24T22:04:28.865 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:28 smithi142 conmon[20125]: ) 67 : cluster [DBG] 5.1d scrub ok 2021-03-24T22:04:28.865 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:28 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:04:28.865 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:28 smithi142 conmon[20125]: -24T22:04:28. 2021-03-24T22:04:28.865 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:28 smithi142 conmon[20125]: 159268+0000 osd.0 (osd.0) 134 : 2021-03-24T22:04:28.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:28 smithi142 conmon[20125]: cluster [DBG] 3.23 scrub ok 2021-03-24T22:04:28.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:28 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:04:28.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:28 smithi142 conmon[20125]: -24T22:04:28.210926+0000 2021-03-24T22:04:28.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:28 smithi142 conmon[20125]: osd.3 (osd.3) 141 : cluster [DBG] 2021-03-24T22:04:28.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:28 smithi142 conmon[20125]: 7.1 scrub ok 2021-03-24T22:04:28.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:28 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:04:28.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:28 smithi142 conmon[20125]: 22:04:28.355690+0000 osd.7 (osd 2021-03-24T22:04:28.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:28 smithi142 conmon[20125]: .7) 112 : cluster [DBG] 5.9 scrub ok 2021-03-24T22:04:28.903 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:28.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:28.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:04:29.008 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:29.061 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:29.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:29.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:04:29.098 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:04:29.152 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:04:29.163 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.4 dump_ops_in_flight 2021-03-24T22:04:29.169 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:29.170 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-03-24T22:04:29.171 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-03-24T22:04:29.221 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:29.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:29.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:29.326 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:29.383 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:29.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:29.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:29.474 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T22:04:29.488 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:29.538 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:29.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:29.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:29.588 INFO:tasks.rados.rados.0.smithi125.stdout:2645: done (0 left) 2021-03-24T22:04:29.588 INFO:tasks.rados.rados.0.smithi125.stdout:2646: snap_create 2021-03-24T22:04:29.593 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:29 smithi125 conmon[24927]: cluster 2021-03-24T22:04:28.568153+0000 2021-03-24T22:04:29.593 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:29 smithi125 conmon[24927]: mon.a (mon.0) 1836 : cluster [DBG] osdmap e698: 8 total, 7 up, 8 in 2021-03-24T22:04:29.594 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:29 smithi125 conmon[19936]: cluster 2021-03-24T22:04:29.594 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:29 smithi125 conmon[19936]: 2021-03-24T 2021-03-24T22:04:29.594 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:29 smithi125 conmon[19936]: 22:04:28.568153+0000 mon.a (mon.0) 1836 : cluster [DBG] osdmap e698: 8 total, 7 up, 8 in 2021-03-24T22:04:29.644 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:29.689 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:29.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:29.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:04:29.770 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:29 smithi142 conmon[20125]: cluster 2021-03-24T22: 2021-03-24T22:04:29.770 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:29 smithi142 conmon[20125]: 04:28.568153+0000 mon.a (mon.0) 1836 : cluster [DBG] osdmap e698: 8 total, 7 up, 8 in 2021-03-24T22:04:29.796 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:29.844 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:29.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:29.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:04:29.950 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:30.013 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:30.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:30.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:30.118 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:30.175 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:30.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:30.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:30.282 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:30.343 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:30.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:30.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:30.382 INFO:tasks.rados.rados.0.smithi125.stdout:2646: done (0 left) 2021-03-24T22:04:30.383 INFO:tasks.rados.rados.0.smithi125.stdout:2647: write oid 36 current snap is 307 2021-03-24T22:04:30.383 INFO:tasks.rados.rados.0.smithi125.stdout:2647: seq_num 919 ranges {645633=716191,1927893=606315,3137495=308138} 2021-03-24T22:04:30.385 INFO:tasks.rados.rados.0.smithi125.stdout:2647: writing smithi12556470-36 from 645633 to 1361824 tid 1 2021-03-24T22:04:30.393 INFO:tasks.rados.rados.0.smithi125.stdout:2647: writing smithi12556470-36 from 1927893 to 2534208 tid 2 2021-03-24T22:04:30.399 INFO:tasks.rados.rados.0.smithi125.stdout:2647: writing smithi12556470-36 from 3137495 to 3445633 tid 3 2021-03-24T22:04:30.400 INFO:tasks.rados.rados.0.smithi125.stdout:2648: snap_create 2021-03-24T22:04:30.422 INFO:tasks.rados.rados.0.smithi125.stdout:2647: finishing write tid 1 to smithi12556470-36 2021-03-24T22:04:30.422 INFO:tasks.rados.rados.0.smithi125.stdout:2647: finishing write tid 2 to smithi12556470-36 2021-03-24T22:04:30.422 INFO:tasks.rados.rados.0.smithi125.stdout:2647: finishing write tid 3 to smithi12556470-36 2021-03-24T22:04:30.423 INFO:tasks.rados.rados.0.smithi125.stdout:2647: finishing write tid 4 to smithi12556470-36 2021-03-24T22:04:30.423 INFO:tasks.rados.rados.0.smithi125.stdout:2647: finishing write tid 5 to smithi12556470-36 2021-03-24T22:04:30.423 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 36 v 1007 (ObjNum 919 snap 307 seq_num 919) dirty exists 2021-03-24T22:04:30.424 INFO:tasks.rados.rados.0.smithi125.stdout:2647: left oid 36 (ObjNum 919 snap 307 seq_num 919) 2021-03-24T22:04:30.450 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:30.499 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:30.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:30.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:30.604 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:30.665 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:30.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:30.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:30.736 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:30 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T22:04:30.736 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:30 smithi125 conmon[24927]: -24T22:04:28.955573+0000 mgr.y 2021-03-24T22:04:30.737 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:30 smithi125 conmon[24927]: (mgr.14144) 624 2021-03-24T22:04:30.737 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:30 smithi125 conmon[24927]: : cluster [DBG] 2021-03-24T22:04:30.737 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:30 smithi125 conmon[24927]: pgmap v1210: 168 pgs: 1 peering, 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 40 active+undersized, 19 active+undersized+degraded, 106 active+clean; 614 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.5 MiB/s rd, 660 KiB/s wr, 2 op/s; 150/1845 objects degraded (8.130%); 40/1845 objects misplaced (2.168%); 38 B/s, 0 objects/s recovering 2021-03-24T22:04:30.737 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:30 smithi125 conmon[24927]: cluster 2021-03-24T22 2021-03-24T22:04:30.738 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:30 smithi125 conmon[24927]: :04:29.585113+0000 mon.a (mon.0) 1837 : cluster [DBG] osdmap e699: 8 total, 7 up, 8 in 2021-03-24T22:04:30.738 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:30 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T22:04:30.738 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:30 smithi125 conmon[24927]: T22:04:29.904527+0000 osd.2 (osd.2) 2021-03-24T22:04:30.738 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:30 smithi125 conmon[24927]: 109 : cluster [DBG] 4.d scrub ok 2021-03-24T22:04:30.738 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:30 smithi125 conmon[24927]: cluster 2021-03-24T22:04 2021-03-24T22:04:30.739 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:30 smithi125 conmon[24927]: :30.205232+0000 osd.3 (osd.3) 142 : cluster 2021-03-24T22:04:30.739 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:30 smithi125 conmon[24927]: [DBG] 4.0 scrub ok 2021-03-24T22:04:30.739 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:30 smithi125 conmon[24927]: cluster 2021-03-24T22:04:30.739 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:30 smithi125 conmon[24927]: 2021-03-24T22:04:30.328883+0000 osd.7 (osd.7) 2021-03-24T22:04:30.740 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:30 smithi125 conmon[24927]: 113 : cluster [DBG] 2.0 scrub ok 2021-03-24T22:04:30.740 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:30 smithi125 conmon[24927]: cluster 2021-03-24T22:04:30 2021-03-24T22:04:30.740 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:30 smithi125 conmon[24927]: .355911+0000 mon.a (mon.0) 1838 : cluster [WRN] 2021-03-24T22:04:30.740 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:30 smithi125 conmon[24927]: Health check update: Degraded data redundancy: 150/1845 objects degraded (8.130%), 19 pgs degraded (PG_DEGRADED) 2021-03-24T22:04:30.740 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:30 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:04:30.741 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:30 smithi125 conmon[24927]: 24T22:04:30.365867+0000 mon.a (mon.0) 1839 : cluster [DBG] 2021-03-24T22:04:30.741 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:30 smithi125 conmon[24927]: osdmap e700: 8 total, 7 up, 8 in 2021-03-24T22:04:30.741 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:30 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:04:30.741 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:30 smithi125 conmon[19936]: 04:28.955573+0000 mgr.y (mgr.14144) 624 : cluster [DBG] pgmap v1210: 168 pgs: 1 peering, 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 40 active+undersized, 19 active+undersized+degraded, 106 active+clean; 614 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.5 MiB/s rd, 660 KiB/s wr, 2 op/s; 150/1845 objects degraded (8.130%); 40/1845 objects misplaced (2.168%); 38 B/s, 0 objects/s recovering 2021-03-24T22:04:30.742 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:30 smithi125 conmon[19936]: cluster 2021-03-24T22:04 2021-03-24T22:04:30.742 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:30 smithi125 conmon[19936]: :29.585113+0000 mon.a (mon 2021-03-24T22:04:30.742 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:30 smithi125 conmon[19936]: .0) 1837 : cluster [DBG] 2021-03-24T22:04:30.743 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:30 smithi125 conmon[19936]: osdmap e699: 8 total, 7 up, 8 in 2021-03-24T22:04:30.743 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:30 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T22:04:30.743 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:30 smithi125 conmon[19936]: 22:04:29.904527+0000 2021-03-24T22:04:30.743 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:30 smithi125 conmon[19936]: osd.2 (osd.2) 2021-03-24T22:04:30.744 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:30 smithi125 conmon[19936]: 109 : cluster [DBG] 4.d scrub ok 2021-03-24T22:04:30.744 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:30 smithi125 conmon[19936]: 2021-03-24T22:04:30.744 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:30 smithi125 conmon[19936]: cluster 2021-03-24T22:04:30.744 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:30 smithi125 conmon[19936]: 2021-03-24T22 2021-03-24T22:04:30.744 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:30 smithi125 conmon[19936]: :04:30.205232 2021-03-24T22:04:30.745 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:30 smithi125 conmon[19936]: +0000 osd.3 (osd.3) 142 2021-03-24T22:04:30.745 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:30 smithi125 conmon[19936]: : cluster [DBG] 4.0 scrub ok 2021-03-24T22:04:30.745 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:30 smithi125 conmon[19936]: cluster 2021-03-24T22:04: 2021-03-24T22:04:30.745 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:30 smithi125 conmon[19936]: 30.328883+0000 osd.7 (osd.7) 113 : cluster [DBG] 2.0 scrub ok 2021-03-24T22:04:30.746 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:30 smithi125 conmon[19936]: cluster 2021-03-24T22:04:30.746 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:30 smithi125 conmon[19936]: 2021-03-24T 2021-03-24T22:04:30.746 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:30 smithi125 conmon[19936]: 22:04:30.355911+0000 mon.a (mon.0) 2021-03-24T22:04:30.746 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:30 smithi125 conmon[19936]: 1838 : cluster [WRN] Health check update: Degraded data redundancy: 150/1845 objects degraded (8.130%), 19 pgs degraded (PG_DEGRADED) 2021-03-24T22:04:30.746 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:30 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:04:30.747 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:30 smithi125 conmon[19936]: 24T22:04:30.365867+0000 mon.a (mon. 2021-03-24T22:04:30.747 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:30 smithi125 conmon[19936]: 0) 1839 : cluster [DBG] osdmap e700: 8 total, 7 up, 8 in 2021-03-24T22:04:30.775 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:30.832 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:30.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:30.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:04:30.838 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:30 smithi142 conmon[20125]: cluster 2021-03-24T22:04:28.955573+0000 mgr.y ( 2021-03-24T22:04:30.838 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:30 smithi142 conmon[20125]: mgr.14144) 624 : cluster [DBG] pgmap v1210: 168 pgs: 1 peering, 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 40 active+undersized, 19 active+undersized+degraded, 106 active+clean; 614 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.5 MiB/s rd, 660 KiB/s wr, 2 op/s; 150/1845 objects degraded (8.130%); 40/1845 objects misplaced (2.168%); 38 B/s, 0 objects/s recovering 2021-03-24T22:04:30.838 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:30 smithi142 conmon[20125]: cluster 2021-03-24T22 2021-03-24T22:04:30.839 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:30 smithi142 conmon[20125]: :04:29.585113+0000 mon.a (mon.0) 1837 : cluster [DBG] osdmap e699: 8 total, 7 up, 8 in 2021-03-24T22:04:30.839 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:30 smithi142 conmon[20125]: 2021-03-24T22:04:30.839 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:30 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:04:30.839 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:30 smithi142 conmon[20125]: 24T22:04:29 2021-03-24T22:04:30.839 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:30 smithi142 conmon[20125]: .904527+0000 osd.2 (osd 2021-03-24T22:04:30.840 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:30 smithi142 conmon[20125]: .2) 109 : cluster [DBG] 2021-03-24T22:04:30.840 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:30 smithi142 conmon[20125]: 4.d scrub ok 2021-03-24T22:04:30.840 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:30 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:04:30.840 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:30 smithi142 conmon[20125]: 22:04:30.205232+0000 osd.3 (osd.3) 142 : cluster [DBG] 4.0 scrub ok 2021-03-24T22:04:30.841 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:30 smithi142 conmon[20125]: cluster 2021-03-24T22:04:30.841 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:30 smithi142 conmon[20125]: 2021-03- 2021-03-24T22:04:30.841 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:30 smithi142 conmon[20125]: 24T22:04:30.328883+0000 osd.7 2021-03-24T22:04:30.842 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:30 smithi142 conmon[20125]: (osd.7) 113 : cluster [DBG] 2.0 scrub ok 2021-03-24T22:04:30.842 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:30 smithi142 conmon[20125]: 2021-03-24T22:04:30.842 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:30 smithi142 conmon[20125]: cluster 2021-03-24T22:04:30.842 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:30 smithi142 conmon[20125]: 2021-03-24T22: 2021-03-24T22:04:30.843 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:30 smithi142 conmon[20125]: 04:30.355911+0000 mon.a (mon.0) 2021-03-24T22:04:30.843 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:30 smithi142 conmon[20125]: 1838 : cluster [WRN] Health check update: Degraded data redundancy: 150/1845 objects degraded (8.130%), 19 pgs degraded (PG_DEGRADED) 2021-03-24T22:04:30.843 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:30 smithi142 conmon[20125]: cluster 2021-03-24T22:04:30.844 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:30 smithi142 conmon[20125]: 2021-03-24 2021-03-24T22:04:30.844 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:30 smithi142 conmon[20125]: T22:04:30.365867+0000 mon.a ( 2021-03-24T22:04:30.844 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:30 smithi142 conmon[20125]: mon.0) 1839 : cluster [DBG] osdmap e700: 8 total, 7 up, 8 in 2021-03-24T22:04:30.845 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:30 smithi142 conmon[20125]: 2021-03-24T22:04:30.938 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:30.988 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:30.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:30.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:31.096 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:31.144 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:31.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:31.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:04:31.173 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-24T22:04:31.250 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:31.307 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:31.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:31.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:31.394 INFO:tasks.rados.rados.0.smithi125.stdout:2647: done (1 left) 2021-03-24T22:04:31.395 INFO:tasks.rados.rados.0.smithi125.stdout:2648: done (0 left) 2021-03-24T22:04:31.395 INFO:tasks.rados.rados.0.smithi125.stdout:2649: delete oid 29 current snap is 308 2021-03-24T22:04:31.402 INFO:tasks.rados.rados.0.smithi125.stdout:2649: done (0 left) 2021-03-24T22:04:31.402 INFO:tasks.rados.rados.0.smithi125.stdout:2650: snap_remove snap 278 2021-03-24T22:04:31.414 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:31.514 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:31.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:31.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:31.618 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:31.670 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:31.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:31.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:31.777 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:31.786 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:31 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:04:31.787 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:31 smithi142 conmon[20125]: 22:04:30.667846+0000 osd.1 (osd.1) 68 : cluster [DBG] 4.f scrub ok 2021-03-24T22:04:31.787 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:31 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:04:31.787 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:31 smithi142 conmon[20125]: 24T22:04:31.107245+0000 osd.0 (osd 2021-03-24T22:04:31.788 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:31 smithi142 conmon[20125]: .0) 135 : cluster [DBG] 6.9 scrub ok 2021-03-24T22:04:31.788 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:31 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:04:31.788 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:31 smithi142 conmon[20125]: -03-24T22:04:31.389820+0000 2021-03-24T22:04:31.788 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:31 smithi142 conmon[20125]: mon.a (mon.0) 1840 : cluster [DBG] osdmap e701: 8 total, 7 up, 8 in 2021-03-24T22:04:31.799 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2021-03-24T22:04:31.800 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2021-03-24T22:04:31.800 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2021-03-24T22:04:31.800 INFO:teuthology.orchestra.run.smithi142.stdout:} 2021-03-24T22:04:31.825 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:31.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:31.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:04:31.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:31 smithi125 conmon[19936]: cluster 2021-03-24T22:04:30.667846+0000 2021-03-24T22:04:31.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:31 smithi125 conmon[19936]: osd.1 (osd.1) 68 : cluster [DBG] 4.f scrub ok 2021-03-24T22:04:31.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:31 smithi125 conmon[19936]: cluster 2021-03-24T22:04:31.107245+0000 2021-03-24T22:04:31.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:31 smithi125 conmon[19936]: osd.0 (osd.0) 135 : cluster [DBG] 6.9 scrub ok 2021-03-24T22:04:31.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:31 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:04:31.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:31 smithi125 conmon[19936]: -03-24T22:04:31.389820 2021-03-24T22:04:31.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:31 smithi125 conmon[19936]: +0000 mon.a (mon.0) 1840 2021-03-24T22:04:31.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:31 smithi125 conmon[19936]: : cluster [DBG] osdmap e701: 8 total, 7 up, 8 in 2021-03-24T22:04:31.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:31 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T22:04:31.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:31 smithi125 conmon[24927]: 22:04:30.667846+0000 osd.1 (osd.1) 68 : cluster [DBG] 2021-03-24T22:04:31.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:31 smithi125 conmon[24927]: 4.f scrub ok 2021-03-24T22:04:31.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:31 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T22:04:31.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:31 smithi125 conmon[24927]: T22:04:31.107245+0000 osd.0 ( 2021-03-24T22:04:31.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:31 smithi125 conmon[24927]: osd.0) 135 : cluster [DBG] 6.9 scrub ok 2021-03-24T22:04:31.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:31 smithi125 conmon[24927]: 2021-03-24T22:04:31.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:31 smithi125 conmon[24927]: cluster 2021-03-24T22:04:31.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:31 smithi125 conmon[24927]: 2021-03-24T22:04:31.389820 2021-03-24T22:04:31.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:31 smithi125 conmon[24927]: +0000 mon.a (mon.0) 1840 : cluster 2021-03-24T22:04:31.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:31 smithi125 conmon[24927]: [DBG] osdmap e701: 8 total, 7 up, 8 in 2021-03-24T22:04:31.934 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:31.996 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:32.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:32.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:04:32.105 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:32.167 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:32.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:32.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:04:32.189 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:04:32.274 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:32.321 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:32.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:32.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:32.404 INFO:tasks.rados.rados.0.smithi125.stdout:2650: done (0 left) 2021-03-24T22:04:32.404 INFO:tasks.rados.rados.0.smithi125.stdout:2651: rollback oid 49 current snap is 308 2021-03-24T22:04:32.405 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 49 to 303 2021-03-24T22:04:32.405 INFO:tasks.rados.rados.0.smithi125.stdout:2652: rollback oid 48 current snap is 308 2021-03-24T22:04:32.405 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 48 to 251 2021-03-24T22:04:32.406 INFO:tasks.rados.rados.0.smithi125.stdout:2653: write oid 5 current snap is 308 2021-03-24T22:04:32.406 INFO:tasks.rados.rados.0.smithi125.stdout:2653: seq_num 921 ranges {679247=583042,2044857=609984,3192341=286906} 2021-03-24T22:04:32.413 INFO:tasks.rados.rados.0.smithi125.stdout:2653: writing smithi12556470-5 from 679247 to 1262289 tid 1 2021-03-24T22:04:32.425 INFO:tasks.rados.rados.0.smithi125.stdout:2653: writing smithi12556470-5 from 2044857 to 2654841 tid 2 2021-03-24T22:04:32.431 INFO:tasks.rados.rados.0.smithi125.stdout:2653: writing smithi12556470-5 from 3192341 to 3479247 tid 3 2021-03-24T22:04:32.431 INFO:tasks.rados.rados.0.smithi125.stdout:2651: finishing rollback tid 0 to smithi12556470-49 2021-03-24T22:04:32.431 INFO:tasks.rados.rados.0.smithi125.stdout:2651: finishing rollback tid 1 to smithi12556470-49 2021-03-24T22:04:32.432 INFO:tasks.rados.rados.0.smithi125.stdout:2651: finishing rollback tid 2 to smithi12556470-49 2021-03-24T22:04:32.432 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 49 v 934 (ObjNum 802 snap 262 seq_num 802) dirty exists 2021-03-24T22:04:32.432 INFO:tasks.rados.rados.0.smithi125.stdout:2652: finishing rollback tid 0 to smithi12556470-48 2021-03-24T22:04:32.433 INFO:tasks.rados.rados.0.smithi125.stdout:2652: finishing rollback tid 1 to smithi12556470-48 2021-03-24T22:04:32.433 INFO:tasks.rados.rados.0.smithi125.stdout:2652: finishing rollback tid 2 to smithi12556470-48 2021-03-24T22:04:32.433 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 48 v 938 (ObjNum 774 snap 251 seq_num 774) dirty exists 2021-03-24T22:04:32.433 INFO:tasks.rados.rados.0.smithi125.stdout:2653: finishing write tid 1 to smithi12556470-5 2021-03-24T22:04:32.434 INFO:tasks.rados.rados.0.smithi125.stdout:2651: done (2 left) 2021-03-24T22:04:32.434 INFO:tasks.rados.rados.0.smithi125.stdout:2652: done (1 left) 2021-03-24T22:04:32.434 INFO:tasks.rados.rados.0.smithi125.stdout:2654: rollback oid 47 current snap is 308 2021-03-24T22:04:32.434 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 47 to 269 2021-03-24T22:04:32.434 INFO:tasks.rados.rados.0.smithi125.stdout:2655: write (excl) oid 8 current snap is 308 2021-03-24T22:04:32.435 INFO:tasks.rados.rados.0.smithi125.stdout:2655: seq_num 922 ranges {696054=658291,2103339=753642} 2021-03-24T22:04:32.435 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:32.441 INFO:tasks.rados.rados.0.smithi125.stdout:2655: writing smithi12556470-8 from 696054 to 1354345 tid 1 2021-03-24T22:04:32.467 INFO:tasks.rados.rados.0.smithi125.stdout:2655: writing smithi12556470-8 from 2103339 to 2856981 tid 2 2021-03-24T22:04:32.467 INFO:tasks.rados.rados.0.smithi125.stdout:2656: snap_remove snap 281 2021-03-24T22:04:32.499 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:32.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:32.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:32.610 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:32.669 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:32.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:32.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:04:32.774 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:32.805 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.4 dump_blocked_ops 2021-03-24T22:04:32.825 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:32.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:32.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:04:32.864 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:32 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:04:32.865 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:32 smithi125 conmon[19936]: T22:04:30.956502+0000 mgr.y (mgr.14144) 625 : cluster [DBG] pgmap v1213: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 1 peering, 40 active+undersized, 19 active+undersized+degraded, 106 active+clean; 615 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.8 MiB/s rd, 781 KiB/s wr, 7 op/s; 152/1854 objects degraded (8.198%); 40/1854 objects misplaced (2.157%) 2021-03-24T22:04:32.865 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:32 smithi125 conmon[19936]: cluster 2021-03-24T22:04:32.865 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:32 smithi125 conmon[19936]: 2021-03-24T22: 2021-03-24T22:04:32.865 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:32 smithi125 conmon[19936]: 04:31.592317+0000 2021-03-24T22:04:32.866 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:32 smithi125 conmon[19936]: osd.6 (osd.6 2021-03-24T22:04:32.866 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:32 smithi125 conmon[19936]: ) 64 : cluster [DBG] 2021-03-24T22:04:32.866 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:32 smithi125 conmon[19936]: 7.7 scrub ok 2021-03-24T22:04:32.866 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:32 smithi125 conmon[19936]: cluster 2021-03-24T22:04:32.867 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:32 smithi125 conmon[19936]: 2021-03-24T22:04:32.392461 2021-03-24T22:04:32.867 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:32 smithi125 conmon[19936]: +0000 mon.a (mon.0) 1841 2021-03-24T22:04:32.867 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:32 smithi125 conmon[19936]: : cluster [DBG] osdmap e702: 8 total, 7 up, 8 in 2021-03-24T22:04:32.867 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:32 smithi125 conmon[19936]: audit 2021 2021-03-24T22:04:32.867 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:32 smithi125 conmon[19936]: -03-24T22:04 2021-03-24T22:04:32.868 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:32 smithi125 conmon[19936]: :32.453777+0000 2021-03-24T22:04:32.868 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:32 smithi125 conmon[19936]: mon.a (mon.0) 1842 : audit 2021-03-24T22:04:32.868 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:32 smithi125 conmon[19936]: [INF] from='client.? v1:172.21.15.125:0/2477240861' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:04:32.868 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:32 smithi125 conmon[24927]: cluster 2021-03-24T22:04:32.869 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:32 smithi125 conmon[24927]: 2021-03-24T22:04:32.869 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:32 smithi125 conmon[24927]: 2021 2021-03-24T22:04:32.869 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:32 smithi125 conmon[24927]: - 2021-03-24T22:04:32.870 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:32 smithi125 conmon[24927]: 03 2021-03-24T22:04:32.870 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:32 smithi125 conmon[24927]: - 2021-03-24T22:04:32.871 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:32 smithi125 conmon[24927]: 24 2021-03-24T22:04:32.871 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:32 smithi125 conmon[24927]: T 2021-03-24T22:04:32.871 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:32 smithi125 conmon[24927]: 22 2021-03-24T22:04:32.872 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:32 smithi125 conmon[24927]: : 2021-03-24T22:04:32.872 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:32 smithi125 conmon[24927]: 04 2021-03-24T22:04:32.872 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:32 smithi125 conmon[24927]: : 2021-03-24T22:04:32.872 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:32 smithi125 conmon[24927]: 30 2021-03-24T22:04:32.872 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:32 smithi125 conmon[24927]: . 2021-03-24T22:04:32.873 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:32 smithi125 conmon[24927]: 956502 2021-03-24T22:04:32.873 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:32 smithi125 conmon[24927]: +0000 2021-03-24T22:04:32.873 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:32 smithi125 conmon[24927]: 2021-03-24T22:04:32.874 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:32 smithi125 conmon[24927]: mgr.y 2021-03-24T22:04:32.875 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:32 smithi125 conmon[24927]: ( 2021-03-24T22:04:32.875 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:32 smithi125 conmon[24927]: mgr 2021-03-24T22:04:32.875 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:32 smithi125 conmon[24927]: . 2021-03-24T22:04:32.876 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:32 smithi125 conmon[24927]: 14144 2021-03-24T22:04:32.876 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:32 smithi125 conmon[24927]: ) 2021-03-24T22:04:32.876 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:32 smithi125 conmon[24927]: 625 2021-03-24T22:04:32.877 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:32 smithi125 conmon[24927]: : 2021-03-24T22:04:32.877 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:32 smithi125 conmon[24927]: cluster [DBG] pgmap v1213: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 1 peering, 40 active+undersized, 19 active+undersized+degraded, 106 active+clean; 615 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.8 MiB/s rd, 781 KiB/s wr, 7 op/s; 152/1854 objects degraded (8.198%); 40/1854 objects misplaced (2.157%) 2021-03-24T22:04:32.877 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:32 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:04:32.878 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:32 smithi125 conmon[24927]: -03-24T22:04:31.592317+0000 osd.6 (osd.6 2021-03-24T22:04:32.878 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:32 smithi125 conmon[24927]: ) 64 : cluster [DBG] 7.7 scrub ok 2021-03-24T22:04:32.878 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:32 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:04:32.878 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:32 smithi125 conmon[24927]: -03-24T22:04:32.392461+0000 mon.a 2021-03-24T22:04:32.879 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:32 smithi125 conmon[24927]: (mon.0) 1841 : cluster [DBG] osdmap e702: 8 total, 7 up, 8 in 2021-03-24T22:04:32.879 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:32 smithi125 conmon[24927]: audit 2021 2021-03-24T22:04:32.879 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:32 smithi125 conmon[24927]: -03-24T22:04:32.453777+0000 mon.a ( 2021-03-24T22:04:32.879 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:32 smithi125 conmon[24927]: mon.0) 1842 : audit [INF] from='client.? v1:172.21.15.125:0/2477240861' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:04:32.880 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:32 smithi142 conmon[20125]: cluster 2021-03-24T22:04:30.956502+0000 mgr.y (mgr.14144 2021-03-24T22:04:32.880 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:32 smithi142 conmon[20125]: ) 625 : cluster [DBG] pgmap v1213: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 1 peering, 40 active+undersized, 19 active+undersized+degraded, 106 active+clean; 615 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.8 MiB/s rd, 781 KiB/s wr, 7 op/s; 152/1854 objects degraded (8.198%); 40/1854 objects misplaced (2.157%) 2021-03-24T22:04:32.881 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:32 smithi142 conmon[20125]: cluster 2021-03-24T22:04:31.592317+0000 2021-03-24T22:04:32.881 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:32 smithi142 conmon[20125]: osd.6 (osd.6) 64 : cluster [DBG] 7.7 scrub ok 2021-03-24T22:04:32.881 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:32 smithi142 conmon[20125]: cluster 2021-03-24T22:04:32.881 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:32 smithi142 conmon[20125]: 2021-03-24T22:04:32.392461+0000 mon.a (mon 2021-03-24T22:04:32.882 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:32 smithi142 conmon[20125]: .0) 1841 : cluster [DBG] osdmap e702: 8 total, 7 up, 8 in 2021-03-24T22:04:32.882 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:32 smithi142 conmon[20125]: audit 2021-03-24 2021-03-24T22:04:32.882 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:32 smithi142 conmon[20125]: T22:04:32.453777+0000 mon.a (mon 2021-03-24T22:04:32.882 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:32 smithi142 conmon[20125]: .0) 1842 : audit [INF] from='client.? v1:172.21.15.125:0/2477240861' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:04:32.931 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:32.987 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:32.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:32.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:04:33.093 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:33.115 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T22:04:33.145 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:33.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:33.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:04:33.250 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:33.309 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:33.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:33.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:04:33.406 INFO:tasks.rados.rados.0.smithi125.stdout:2656: done (3 left) 2021-03-24T22:04:33.406 INFO:tasks.rados.rados.0.smithi125.stdout:2657: write oid 29 current snap is 308 2021-03-24T22:04:33.407 INFO:tasks.rados.rados.0.smithi125.stdout:2653: finishing write tid 2 to smithi12556470-5 2021-03-24T22:04:33.407 INFO:tasks.rados.rados.0.smithi125.stdout:2653: finishing write tid 3 to smithi12556470-5 2021-03-24T22:04:33.407 INFO:tasks.rados.rados.0.smithi125.stdout:2653: finishing write tid 4 to smithi12556470-5 2021-03-24T22:04:33.407 INFO:tasks.rados.rados.0.smithi125.stdout:2653: finishing write tid 5 to smithi12556470-5 2021-03-24T22:04:33.407 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 5 v 1025 (ObjNum 921 snap 308 seq_num 921) dirty exists 2021-03-24T22:04:33.408 INFO:tasks.rados.rados.0.smithi125.stdout:2653: left oid 5 (ObjNum 921 snap 308 seq_num 921) 2021-03-24T22:04:33.408 INFO:tasks.rados.rados.0.smithi125.stdout:2654: finishing rollback tid 0 to smithi12556470-47 2021-03-24T22:04:33.408 INFO:tasks.rados.rados.0.smithi125.stdout:2654: finishing rollback tid 1 to smithi12556470-47 2021-03-24T22:04:33.409 INFO:tasks.rados.rados.0.smithi125.stdout:2654: finishing rollback tid 2 to smithi12556470-47 2021-03-24T22:04:33.409 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 47 v 358 (ObjNum 466 snap 131 seq_num 466) dirty exists 2021-03-24T22:04:33.409 INFO:tasks.rados.rados.0.smithi125.stdout:2655: finishing write tid 1 to smithi12556470-8 2021-03-24T22:04:33.410 INFO:tasks.rados.rados.0.smithi125.stdout:2655: finishing write tid 2 to smithi12556470-8 2021-03-24T22:04:33.410 INFO:tasks.rados.rados.0.smithi125.stdout:2655: finishing write tid 3 to smithi12556470-8 2021-03-24T22:04:33.410 INFO:tasks.rados.rados.0.smithi125.stdout:2655: finishing write tid 4 to smithi12556470-8 2021-03-24T22:04:33.411 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 8 v 555 (ObjNum 922 snap 308 seq_num 922) dirty exists 2021-03-24T22:04:33.411 INFO:tasks.rados.rados.0.smithi125.stdout:2655: left oid 8 (ObjNum 922 snap 308 seq_num 922) 2021-03-24T22:04:33.411 INFO:tasks.rados.rados.0.smithi125.stdout:2657: seq_num 923 ranges {712861=733540,2161821=613653,3247187=265674} 2021-03-24T22:04:33.414 INFO:tasks.rados.rados.0.smithi125.stdout:2657: writing smithi12556470-29 from 712861 to 1446401 tid 1 2021-03-24T22:04:33.416 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:33.417 INFO:teuthology.orchestra.run.smithi125.stderr:nodeep-scrub is unset 2021-03-24T22:04:33.423 INFO:tasks.rados.rados.0.smithi125.stdout:2657: writing smithi12556470-29 from 2161821 to 2775474 tid 2 2021-03-24T22:04:33.429 INFO:tasks.rados.rados.0.smithi125.stdout:2657: writing smithi12556470-29 from 3247187 to 3512861 tid 3 2021-03-24T22:04:33.430 INFO:tasks.rados.rados.0.smithi125.stdout:2653: done (3 left) 2021-03-24T22:04:33.430 INFO:tasks.rados.rados.0.smithi125.stdout:2654: done (2 left) 2021-03-24T22:04:33.431 INFO:tasks.rados.rados.0.smithi125.stdout:2655: done (1 left) 2021-03-24T22:04:33.431 INFO:tasks.rados.rados.0.smithi125.stdout:2658: delete oid 9 current snap is 308 2021-03-24T22:04:33.446 INFO:tasks.rados.rados.0.smithi125.stdout:2658: done (1 left) 2021-03-24T22:04:33.446 INFO:tasks.rados.rados.0.smithi125.stdout:2659: write (excl) oid 13 current snap is 308 2021-03-24T22:04:33.447 INFO:tasks.rados.rados.0.smithi125.stdout:2659: seq_num 925 ranges {746475=484038,1762432=617322,2785680=529906} 2021-03-24T22:04:33.460 INFO:tasks.rados.rados.0.smithi125.stdout:2659: writing smithi12556470-13 from 746475 to 1230513 tid 1 2021-03-24T22:04:33.471 INFO:tasks.rados.rados.0.smithi125.stdout:2659: writing smithi12556470-13 from 1762432 to 2379754 tid 2 2021-03-24T22:04:33.478 INFO:tasks.rados.rados.0.smithi125.stdout:2659: writing smithi12556470-13 from 2785680 to 3315586 tid 3 2021-03-24T22:04:33.479 INFO:tasks.rados.rados.0.smithi125.stdout:2657: finishing write tid 1 to smithi12556470-29 2021-03-24T22:04:33.480 INFO:tasks.rados.rados.0.smithi125.stdout:2657: finishing write tid 2 to smithi12556470-29 2021-03-24T22:04:33.480 INFO:tasks.rados.rados.0.smithi125.stdout:2657: finishing write tid 3 to smithi12556470-29 2021-03-24T22:04:33.480 INFO:tasks.rados.rados.0.smithi125.stdout:2657: finishing write tid 4 to smithi12556470-29 2021-03-24T22:04:33.480 INFO:tasks.rados.rados.0.smithi125.stdout:2657: finishing write tid 5 to smithi12556470-29 2021-03-24T22:04:33.481 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 29 v 447 (ObjNum 923 snap 308 seq_num 923) dirty exists 2021-03-24T22:04:33.481 INFO:tasks.rados.rados.0.smithi125.stdout:2657: left oid 29 (ObjNum 923 snap 308 seq_num 923) 2021-03-24T22:04:33.481 INFO:tasks.rados.rados.0.smithi125.stdout:2657: done (1 left) 2021-03-24T22:04:33.481 INFO:tasks.rados.rados.0.smithi125.stdout:2660: read oid 37 snap -1 2021-03-24T22:04:33.481 INFO:tasks.rados.rados.0.smithi125.stdout:2660: expect deleted 2021-03-24T22:04:33.482 INFO:tasks.rados.rados.0.smithi125.stdout:2661: copy_from oid 40 from oid 6 current snap is 308 2021-03-24T22:04:33.482 INFO:tasks.rados.rados.0.smithi125.stdout:2662: rollback oid 26 current snap is 308 2021-03-24T22:04:33.482 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 26 to 301 2021-03-24T22:04:33.482 INFO:tasks.rados.rados.0.smithi125.stdout:2663: snap_create 2021-03-24T22:04:33.487 INFO:tasks.rados.rados.0.smithi125.stdout:2659: finishing write tid 1 to smithi12556470-13 2021-03-24T22:04:33.493 INFO:tasks.rados.rados.0.smithi125.stdout:2659: finishing write tid 2 to smithi12556470-13 2021-03-24T22:04:33.493 INFO:tasks.rados.rados.0.smithi125.stdout:2662: finishing rollback tid 0 to smithi12556470-26 2021-03-24T22:04:33.496 INFO:tasks.rados.rados.0.smithi125.stdout:2659: finishing write tid 3 to smithi12556470-13 2021-03-24T22:04:33.496 INFO:tasks.rados.rados.0.smithi125.stdout:2659: finishing write tid 4 to smithi12556470-13 2021-03-24T22:04:33.496 INFO:tasks.rados.rados.0.smithi125.stdout:2659: finishing write tid 5 to smithi12556470-13 2021-03-24T22:04:33.496 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 13 v 800 (ObjNum 925 snap 308 seq_num 925) dirty exists 2021-03-24T22:04:33.496 INFO:tasks.rados.rados.0.smithi125.stdout:2659: left oid 13 (ObjNum 925 snap 308 seq_num 925) 2021-03-24T22:04:33.500 INFO:tasks.rados.rados.0.smithi125.stdout:2662: finishing rollback tid 1 to smithi12556470-26 2021-03-24T22:04:33.500 INFO:tasks.rados.rados.0.smithi125.stdout:2662: finishing rollback tid 2 to smithi12556470-26 2021-03-24T22:04:33.500 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 26 v 834 (ObjNum 905 snap 300 seq_num 905) dirty exists 2021-03-24T22:04:33.503 INFO:tasks.rados.rados.0.smithi125.stdout:2661: finishing copy_from to smithi12556470-40 2021-03-24T22:04:33.503 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 40 v 604 (ObjNum 906 snap 300 seq_num 906) dirty exists 2021-03-24T22:04:33.510 INFO:tasks.rados.rados.0.smithi125.stdout:2661: finishing copy_from racing read to smithi12556470-40 2021-03-24T22:04:33.512 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:33.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:33.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:33.621 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:33.674 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:33.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:33.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:04:33.703 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:33 smithi125 conmon[19936]: cluster 2021-03-24T22:04:32.551672+0000 2021-03-24T22:04:33.704 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:33 smithi125 conmon[19936]: osd.6 (osd.6) 65 : cluster [DBG] 3.11 scrub ok 2021-03-24T22:04:33.704 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:33 smithi125 conmon[19936]: cluster 2021-03-24T22:04:33.248532+0000 osd.7 (osd.7) 114 : cluster [DBG] 2021-03-24T22:04:33.705 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:33 smithi125 conmon[19936]: 3.e scrub ok 2021-03-24T22:04:33.705 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:33 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:04:33.706 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:33 smithi125 conmon[19936]: T22:04:33.385641+0000 mon.a (mon.0) 2021-03-24T22:04:33.706 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:33 smithi125 conmon[19936]: 1843 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T22:04:33.707 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:33 smithi125 conmon[19936]: audit 2021-03-24T22:04:33.707 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:33 smithi125 conmon[19936]: 2021-03-24T22:04:33.708 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:33 smithi125 conmon[19936]: 2021 2021-03-24T22:04:33.708 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:33 smithi125 conmon[19936]: -03-24T22:04:33 2021-03-24T22:04:33.709 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:33 smithi125 conmon[19936]: .399708+0000 mon.a (mon.0) 1844 : audit [INF] from='client.? v1:172.21.15.125:0/2477240861' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T22:04:33.709 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:33 smithi125 conmon[19936]: cluster 2021-03-24T22:04:33.399908 2021-03-24T22:04:33.709 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:33 smithi125 conmon[19936]: +0000 mon.a (mon.0) 1845 : cluster [DBG] osdmap e703: 8 total, 7 up, 8 in 2021-03-24T22:04:33.710 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:33 smithi125 conmon[19936]: 2021-03-24T22:04:33.711 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:33 smithi125 conmon[24927]: cluster 2021-03-24T22:04:33.712 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:33 smithi125 conmon[24927]: 2021-03-24T22:04:33.712 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:33 smithi125 conmon[24927]: 2021 2021-03-24T22:04:33.713 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:33 smithi125 conmon[24927]: -03-24T22: 2021-03-24T22:04:33.713 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:33 smithi125 conmon[24927]: 04:32.551672+0000 osd.6 (osd.6) 65 : cluster [DBG] 3.11 scrub ok 2021-03-24T22:04:33.713 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:33 smithi125 conmon[24927]: cluster 2021-03-24T22:04:33.714 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:33 smithi125 conmon[24927]: 2021- 2021-03-24T22:04:33.714 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:33 smithi125 conmon[24927]: 03-24T22:04:33.248532+0000 osd.7 (osd.7) 2021-03-24T22:04:33.715 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:33 smithi125 conmon[24927]: 114 : cluster [DBG] 3.e scrub ok 2021-03-24T22:04:33.715 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:33 smithi125 conmon[24927]: cluster 2021-03-24T22:04:33.716 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:33 smithi125 conmon[24927]: 2021-03-24T22:04:33.385641+0000 2021-03-24T22:04:33.716 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:33 smithi125 conmon[24927]: mon.a (mon.0) 1843 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T22:04:33.716 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:33 smithi125 conmon[24927]: audit 2021- 2021-03-24T22:04:33.717 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:33 smithi125 conmon[24927]: 03-24T22:04:33.399708+0000 mon.a ( 2021-03-24T22:04:33.717 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:33 smithi125 conmon[24927]: mon.0) 1844 : audit [INF] from='client.? v1:172.21.15.125:0/2477240861' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T22:04:33.718 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:33 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T22:04:33.718 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:33 smithi125 conmon[24927]: -24T22:04:33.399908+0000 mon.a (mon 2021-03-24T22:04:33.719 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:33 smithi125 conmon[24927]: .0) 1845 : cluster [DBG] osdmap e703: 8 total, 7 up, 8 in 2021-03-24T22:04:33.783 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:33.840 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:33.844 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:33 smithi142 conmon[20125]: cluster 2021-03-24T22:04 2021-03-24T22:04:33.844 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:33 smithi142 conmon[20125]: :32.551672+0000 osd.6 (osd.6) 65 : cluster [DBG] 3.11 scrub ok 2021-03-24T22:04:33.844 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:33 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:04:33.845 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:33 smithi142 conmon[20125]: 03-24T22:04:33.248532+0000 osd.7 (osd.7 2021-03-24T22:04:33.845 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:33 smithi142 conmon[20125]: ) 114 : cluster [DBG] 3.e scrub ok 2021-03-24T22:04:33.845 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:33 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:04:33.845 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:33 smithi142 conmon[20125]: -03-24T22:04:33. 2021-03-24T22:04:33.845 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:33 smithi142 conmon[20125]: 385641+0000 mon.a (mon.0) 1843 2021-03-24T22:04:33.846 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:33 smithi142 conmon[20125]: : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T22:04:33.846 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:33 smithi142 conmon[20125]: audit 2021-03- 2021-03-24T22:04:33.846 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:33 smithi142 conmon[20125]: 24T22:04:33.399708+0000 2021-03-24T22:04:33.846 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:33 smithi142 conmon[20125]: mon.a (mon.0) 1844 : audit [INF] 2021-03-24T22:04:33.847 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:33 smithi142 conmon[20125]: from='client.? v1:172.21.15.125:0/2477240861' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T22:04:33.847 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:33 smithi142 conmon[20125]: cluster 2021-03-24T22: 2021-03-24T22:04:33.847 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:33 smithi142 conmon[20125]: 04:33.399908+0000 mon.a (mon. 2021-03-24T22:04:33.847 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:33 smithi142 conmon[20125]: 0) 1845 : cluster [DBG] osdmap e703: 8 total, 7 up, 8 in 2021-03-24T22:04:33.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:33.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:04:33.951 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:34.002 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:34.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:34.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:04:34.108 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:34.162 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:34.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:34.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:34.270 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:34.325 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:34.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:34.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:04:34.410 INFO:tasks.rados.rados.0.smithi125.stdout:2659: done (4 left) 2021-03-24T22:04:34.411 INFO:tasks.rados.rados.0.smithi125.stdout:2660: done (3 left) 2021-03-24T22:04:34.411 INFO:tasks.rados.rados.0.smithi125.stdout:2661: done (2 left) 2021-03-24T22:04:34.411 INFO:tasks.rados.rados.0.smithi125.stdout:2662: done (1 left) 2021-03-24T22:04:34.412 INFO:tasks.rados.rados.0.smithi125.stdout:2663: done (0 left) 2021-03-24T22:04:34.412 INFO:tasks.rados.rados.0.smithi125.stdout:2664: snap_remove snap 289 2021-03-24T22:04:34.422 INFO:teuthology.orchestra.run.smithi125.stderr:set pool 5 pgp_num to 32 2021-03-24T22:04:34.430 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:34.483 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:34.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:34.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:34.590 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:34.648 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:34.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:34.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:34.754 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:34.758 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:04:34.780 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:34 smithi142 conmon[20125]: cluster 2021-03-24T22:04:32 2021-03-24T22:04:34.781 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:34 smithi142 conmon[20125]: .957449+0000 mgr.y (mgr.14144) 626 : cluster [DBG] pgmap v1216: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 107 active+clean; 622 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.8 MiB/s rd, 1.9 MiB/s wr, 12 op/s; 152/1860 objects degraded (8.172%); 40/1860 objects misplaced (2.151%) 2021-03-24T22:04:34.781 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:34 smithi142 conmon[20125]: cluster 2021-03-24T22:04:33.627562+0000 osd.1 (osd 2021-03-24T22:04:34.782 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:34 smithi142 conmon[20125]: .1) 69 : cluster [DBG] 4.12 scrub ok 2021-03-24T22:04:34.782 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:34 smithi142 conmon[20125]: cluster 2021-03-24T22:04:34.783 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:34 smithi142 conmon[20125]: 2021-03-24T22:04:33.895972+0000 osd.2 (osd.2) 110 : cluster 2021-03-24T22:04:34.783 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:34 smithi142 conmon[20125]: [DBG] 6.14 scrub ok 2021-03-24T22:04:34.783 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:34 smithi142 conmon[20125]: audit 2021- 2021-03-24T22:04:34.784 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:34 smithi142 conmon[20125]: 03-24T22:04:34.104834+0000 mon.a (mon.0) 1846 : audit [INF] 2021-03-24T22:04:34.784 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:34 smithi142 conmon[20125]: from='client.? v1:172.21.15.125:0/3163997709' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num", "val": "32"}]: dispatch 2021-03-24T22:04:34.785 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:34 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:04:34.785 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:34 smithi142 conmon[20125]: 24T22:04:34.106147+0000 osd.0 (osd.0) 136 : cluster [DBG] 4.b scrub ok 2021-03-24T22:04:34.786 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:34 smithi142 conmon[20125]: 2021-03-24T22:04:34.786 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:34 smithi142 conmon[20125]: audit 2021-03-24 2021-03-24T22:04:34.787 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:34 smithi142 conmon[20125]: T22:04:34.398292+0000 mon.a (mon.0) 1847 : audit [INF] from='client.? v1:172.21.15.125:0/3163997709' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num", "val": "32"}]': finished 2021-03-24T22:04:34.787 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:34 smithi142 conmon[20125]: 2021-03-24T22:04:34.788 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:34 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:04:34.788 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:34 smithi142 conmon[20125]: 22:04:34.398460+0000 mon.a (mon.0) 1848 : cluster [DBG] osdmap e704: 8 total, 7 up, 8 in 2021-03-24T22:04:34.813 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:34.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:34.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:34.920 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:34.969 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:34.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:34.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:35.080 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:35.095 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[24927]: cluster 2021-03-24T22:04:32.957449+0000 mgr.y (mgr.14144) 626 : cluster [DBG] pgmap v1216: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 107 active+clean; 622 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.8 MiB/s rd, 1.9 MiB/s wr, 12 op/s; 152/1860 objects degraded (8.172%); 40/1860 objects misplaced (2.151%) 2021-03-24T22:04:35.095 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[24927]: cluster 2021-03-24T22:04:33.627562+0000 osd.1 (osd.1) 69 : cluster [DBG] 4.12 scrub ok 2021-03-24T22:04:35.095 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:04:35.096 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[24927]: 24T22:04:33. 2021-03-24T22:04:35.096 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[24927]: 895972+0000 osd.2 (osd 2021-03-24T22:04:35.096 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[24927]: .2) 110 : cluster 2021-03-24T22:04:35.096 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[24927]: [DBG] 6.14 scrub ok 2021-03-24T22:04:35.096 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[24927]: audit 2021-03-24T22:04:35.097 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[24927]: 2021-03-24T22:04:34 2021-03-24T22:04:35.097 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[24927]: .104834+0000 mon.a ( 2021-03-24T22:04:35.097 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[24927]: mon.0) 1846 : 2021-03-24T22:04:35.097 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[24927]: audit [INF] from='client.? v1:172.21.15.125:0/3163997709' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num", "val": "32"}]: dispatch 2021-03-24T22:04:35.098 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[24927]: cluster 2021-03-24T22:04:35.098 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[24927]: 2021-03-24T22 2021-03-24T22:04:35.098 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[24927]: :04:34.106147+0000 osd.0 2021-03-24T22:04:35.098 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[24927]: (osd.0) 136 : cluster [DBG] 4.b scrub ok 2021-03-24T22:04:35.098 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[24927]: audit 2021 2021-03-24T22:04:35.099 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[24927]: -03-24T22:04:34.398292 2021-03-24T22:04:35.099 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[24927]: +0000 mon.a (mon.0) 1847 : audit [INF] 2021-03-24T22:04:35.099 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[24927]: from='client.? v1:172.21.15.125:0/3163997709' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num", "val": "32"}]': finished 2021-03-24T22:04:35.099 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:04:35.100 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[24927]: 03-24T22:04:34.398460+0000 mon.a (mon.0) 1848 2021-03-24T22:04:35.100 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[24927]: : cluster [DBG] osdmap e704: 8 total, 7 up, 8 in 2021-03-24T22:04:35.100 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[19936]: 24T22:04:32.957449+0000 mgr.y (mgr.14144) 626 2021-03-24T22:04:35.100 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[19936]: : cluster [DBG] 2021-03-24T22:04:35.101 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[19936]: pgmap v1216: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 107 active+clean; 622 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.8 MiB/s rd, 1.9 MiB/s wr, 12 op/s; 152/1860 objects degraded (8.172%); 40/1860 objects misplaced (2.151%) 2021-03-24T22:04:35.101 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[19936]: cluster 2021-03-24T22:04:33.627562+0000 osd.1 2021-03-24T22:04:35.101 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[19936]: (osd.1) 69 : cluster 2021-03-24T22:04:35.102 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[19936]: [DBG] 4.12 scrub ok 2021-03-24T22:04:35.102 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[19936]: cluster 2021-03-24T22:04:35.102 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[19936]: 2021-03-24T 2021-03-24T22:04:35.102 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[19936]: 22:04: 2021-03-24T22:04:35.102 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[19936]: 33.895972+0000 2021-03-24T22:04:35.103 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[19936]: osd.2 (osd. 2021-03-24T22:04:35.103 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[19936]: 2) 110 : cluster 2021-03-24T22:04:35.103 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[19936]: [DBG] 6.14 scrub ok 2021-03-24T22:04:35.103 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[19936]: audit 2021-03-24T22:04:35.103 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[19936]: 2021-03-24T22: 2021-03-24T22:04:35.104 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[19936]: 04:34.104834 2021-03-24T22:04:35.104 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[19936]: +0000 mon.a ( 2021-03-24T22:04:35.104 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[19936]: mon.0) 1846 2021-03-24T22:04:35.104 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[19936]: : audit [INF] 2021-03-24T22:04:35.104 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[19936]: from='client.? v1:172.21.15.125:0/3163997709' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num", "val": "32"}]: dispatch 2021-03-24T22:04:35.105 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[19936]: cluster 2021-03-24T22:04:35.105 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[19936]: 2021-03-24 2021-03-24T22:04:35.105 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[19936]: T22:04: 2021-03-24T22:04:35.105 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[19936]: 34.106147+0000 2021-03-24T22:04:35.106 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[19936]: osd.0 (osd. 2021-03-24T22:04:35.106 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[19936]: 0) 136 : 2021-03-24T22:04:35.106 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[19936]: cluster [DBG] 4.b scrub ok 2021-03-24T22:04:35.106 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[19936]: 2021-03-24T22:04:35.106 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[19936]: audit 2021-03- 2021-03-24T22:04:35.107 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[19936]: 24T22:04:34.398292+0000 mon.a (mon.0) 1847 2021-03-24T22:04:35.107 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[19936]: : audit [INF] from='client.? v1:172.21.15.125:0/3163997709' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num", "val": "32"}]': finished 2021-03-24T22:04:35.107 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:04:35.107 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[19936]: -03-24T22:04:34. 2021-03-24T22:04:35.107 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[19936]: 398460+0000 mon.a (mon.0) 1848 : 2021-03-24T22:04:35.108 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:34 smithi125 conmon[19936]: cluster [DBG] osdmap e704: 8 total, 7 up, 8 in 2021-03-24T22:04:35.139 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:35.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:35.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:35.243 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:35.299 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:35.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:35.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:04:35.375 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2021-03-24T22:04:35.376 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2021-03-24T22:04:35.376 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2021-03-24T22:04:35.376 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2021-03-24T22:04:35.377 INFO:teuthology.orchestra.run.smithi142.stdout:} 2021-03-24T22:04:35.404 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:35.414 INFO:tasks.rados.rados.0.smithi125.stdout:2664: done (0 left) 2021-03-24T22:04:35.414 INFO:tasks.rados.rados.0.smithi125.stdout:2665: read oid 25 snap 290 2021-03-24T22:04:35.415 INFO:tasks.rados.rados.0.smithi125.stdout:2665: expect (ObjNum 885 snap 290 seq_num 885) 2021-03-24T22:04:35.415 INFO:tasks.rados.rados.0.smithi125.stdout:2666: write oid 23 current snap is 309 2021-03-24T22:04:35.416 INFO:tasks.rados.rados.0.smithi125.stdout:2666: seq_num 926 ranges {763282=559287,1820914=760980,3213103=350179} 2021-03-24T22:04:35.428 INFO:tasks.rados.rados.0.smithi125.stdout:2666: writing smithi12556470-23 from 763282 to 1322569 tid 1 2021-03-24T22:04:35.438 INFO:tasks.rados.rados.0.smithi125.stdout:2666: writing smithi12556470-23 from 1820914 to 2581894 tid 2 2021-03-24T22:04:35.445 INFO:tasks.rados.rados.0.smithi125.stdout:2666: writing smithi12556470-23 from 3213103 to 3563282 tid 3 2021-03-24T22:04:35.445 INFO:tasks.rados.rados.0.smithi125.stdout:2667: read oid 45 snap -1 2021-03-24T22:04:35.445 INFO:tasks.rados.rados.0.smithi125.stdout:2667: expect (ObjNum 845 snap 277 seq_num 845) 2021-03-24T22:04:35.445 INFO:tasks.rados.rados.0.smithi125.stdout:2668: snap_create 2021-03-24T22:04:35.454 INFO:tasks.rados.rados.0.smithi125.stdout:2665: expect (ObjNum 885 snap 290 seq_num 885) 2021-03-24T22:04:35.520 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:35.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:35.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:35.574 INFO:tasks.rados.rados.0.smithi125.stdout:2666: finishing write tid 1 to smithi12556470-23 2021-03-24T22:04:35.575 INFO:tasks.rados.rados.0.smithi125.stdout:2666: finishing write tid 2 to smithi12556470-23 2021-03-24T22:04:35.575 INFO:tasks.rados.rados.0.smithi125.stdout:2667: expect (ObjNum 845 snap 277 seq_num 845) 2021-03-24T22:04:35.628 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:35.678 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:35.680 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[19936]: cluster 2021-03-24T22:04 2021-03-24T22:04:35.681 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[19936]: :34.593781+0000 osd.1 (osd.1) 70 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:04:35.681 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:04:35.681 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[19936]: 24T22:04:34 2021-03-24T22:04:35.681 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[19936]: .594811+0000 osd.1 ( 2021-03-24T22:04:35.681 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[19936]: osd.1) 71 : 2021-03-24T22:04:35.682 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[19936]: cluster [DBG] purged_snaps scrub ok 2021-03-24T22:04:35.682 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[19936]: cluster 2021-03-24T22:04:35.682 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[19936]: 2021-03-24T22 2021-03-24T22:04:35.682 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[19936]: :04:34.667172 2021-03-24T22:04:35.683 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[19936]: +0000 osd.1 (osd.1 2021-03-24T22:04:35.683 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[19936]: ) 72 : cluster [DBG] 2021-03-24T22:04:35.683 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[19936]: 3.0 scrub ok 2021-03-24T22:04:35.683 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[19936]: cluster 2021-03-24T22:04:35.684 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[19936]: 2021-03-24T22 2021-03-24T22:04:35.684 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[19936]: :04:35.126869 2021-03-24T22:04:35.684 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[19936]: +0000 osd.0 (osd. 2021-03-24T22:04:35.684 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[19936]: 0) 137 : cluster 2021-03-24T22:04:35.684 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[19936]: [DBG] 5.1e scrub ok 2021-03-24T22:04:35.685 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[19936]: cluster 2021-03-24T22:04:35.685 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[19936]: 2021-03-24T22 2021-03-24T22:04:35.685 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[19936]: :04:35.296159 2021-03-24T22:04:35.685 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[19936]: +0000 osd.7 (osd 2021-03-24T22:04:35.685 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[19936]: .7) 115 : cluster 2021-03-24T22:04:35.686 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[19936]: [DBG] 7.11 scrub ok 2021-03-24T22:04:35.686 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[19936]: cluster 2021-03-24T22:04:35.686 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[19936]: 2021-03-24T22 2021-03-24T22:04:35.686 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[19936]: :04:35.358524 2021-03-24T22:04:35.687 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[19936]: +0000 mon.a (mon 2021-03-24T22:04:35.687 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[19936]: .0) 1849 : cluster 2021-03-24T22:04:35.687 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[19936]: [WRN] Health check update: Degraded data redundancy: 152/1860 objects degraded (8.172%), 19 pgs degraded (PG_DEGRADED) 2021-03-24T22:04:35.687 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[19936]: cluster 2021-03-24T22:04:35.688 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[19936]: 2021-03-24T22 2021-03-24T22:04:35.688 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[19936]: :04:35.403514 2021-03-24T22:04:35.688 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[19936]: +0000 mon.a (mon.0 2021-03-24T22:04:35.688 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[19936]: ) 1850 : cluster [DBG] 2021-03-24T22:04:35.688 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[19936]: osdmap e705: 8 total, 7 up, 8 in 2021-03-24T22:04:35.689 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: cluster 2021-03-24T22:04:35.689 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 2021-03-24T22:04:35.690 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 2021 2021-03-24T22:04:35.690 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: - 2021-03-24T22:04:35.690 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 03 2021-03-24T22:04:35.690 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: - 2021-03-24T22:04:35.691 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 24 2021-03-24T22:04:35.691 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: T 2021-03-24T22:04:35.691 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 22 2021-03-24T22:04:35.691 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: : 2021-03-24T22:04:35.691 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 04 2021-03-24T22:04:35.692 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: : 2021-03-24T22:04:35.692 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 34 2021-03-24T22:04:35.692 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: . 2021-03-24T22:04:35.692 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 593781 2021-03-24T22:04:35.693 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: +0000 2021-03-24T22:04:35.693 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 2021-03-24T22:04:35.693 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: osd.1 2021-03-24T22:04:35.693 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: ( 2021-03-24T22:04:35.693 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: osd 2021-03-24T22:04:35.694 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: . 2021-03-24T22:04:35.694 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 1 2021-03-24T22:04:35.694 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: ) 2021-03-24T22:04:35.694 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 70 2021-03-24T22:04:35.694 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: : 2021-03-24T22:04:35.695 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: cluster 2021-03-24T22:04:35.695 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 2021-03-24T22:04:35.695 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: [DBG] 2021-03-24T22:04:35.695 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 2021-03-24T22:04:35.696 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: purged_snaps scrub starts 2021-03-24T22:04:35.696 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 2021-03-24T22:04:35.696 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: cluster 2021-03-24T22:04:35.696 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 2021-03-24T22:04:35.696 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 2021 2021-03-24T22:04:35.697 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: - 2021-03-24T22:04:35.697 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 03 2021-03-24T22:04:35.697 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: - 2021-03-24T22:04:35.697 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 24 2021-03-24T22:04:35.697 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: T 2021-03-24T22:04:35.698 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 22 2021-03-24T22:04:35.698 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: : 2021-03-24T22:04:35.698 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 04 2021-03-24T22:04:35.698 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: : 2021-03-24T22:04:35.699 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 34 2021-03-24T22:04:35.699 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: . 2021-03-24T22:04:35.699 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 594811 2021-03-24T22:04:35.699 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: +0000 2021-03-24T22:04:35.700 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 2021-03-24T22:04:35.700 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: osd.1 2021-03-24T22:04:35.700 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: ( 2021-03-24T22:04:35.700 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: osd 2021-03-24T22:04:35.701 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: . 2021-03-24T22:04:35.701 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 1 2021-03-24T22:04:35.701 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: ) 2021-03-24T22:04:35.701 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 71 2021-03-24T22:04:35.702 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: : 2021-03-24T22:04:35.702 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: cluster 2021-03-24T22:04:35.702 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 2021-03-24T22:04:35.702 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: [DBG] 2021-03-24T22:04:35.702 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 2021-03-24T22:04:35.703 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: purged_snaps scrub ok 2021-03-24T22:04:35.703 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 2021-03-24T22:04:35.703 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: cluster 2021-03-24T22:04:35.703 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 2021-03-24T22:04:35.703 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 2021 2021-03-24T22:04:35.704 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: - 2021-03-24T22:04:35.704 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 03 2021-03-24T22:04:35.704 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: - 2021-03-24T22:04:35.704 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 24 2021-03-24T22:04:35.705 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: T 2021-03-24T22:04:35.705 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 22 2021-03-24T22:04:35.705 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: : 2021-03-24T22:04:35.705 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 04 2021-03-24T22:04:35.705 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: : 2021-03-24T22:04:35.706 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 34 2021-03-24T22:04:35.706 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: . 2021-03-24T22:04:35.706 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 667172 2021-03-24T22:04:35.706 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: +0000 2021-03-24T22:04:35.706 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 2021-03-24T22:04:35.707 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: osd.1 2021-03-24T22:04:35.707 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: ( 2021-03-24T22:04:35.707 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: osd 2021-03-24T22:04:35.707 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: . 2021-03-24T22:04:35.708 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 1 2021-03-24T22:04:35.708 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: ) 2021-03-24T22:04:35.708 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 72 2021-03-24T22:04:35.708 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: : 2021-03-24T22:04:35.708 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: cluster 2021-03-24T22:04:35.709 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 2021-03-24T22:04:35.709 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: [DBG] 2021-03-24T22:04:35.709 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 2021-03-24T22:04:35.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:35.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:35.711 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 3.0 scrub ok 2021-03-24T22:04:35.712 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 2021-03-24T22:04:35.712 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: cluster 2021-03-24T22:04:35.712 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 2021-03-24T22:04:35.712 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 2021 2021-03-24T22:04:35.712 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: - 2021-03-24T22:04:35.713 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 03 2021-03-24T22:04:35.713 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: - 2021-03-24T22:04:35.713 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 24 2021-03-24T22:04:35.713 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: T 2021-03-24T22:04:35.714 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 22 2021-03-24T22:04:35.714 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: : 2021-03-24T22:04:35.714 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 04 2021-03-24T22:04:35.714 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: : 2021-03-24T22:04:35.714 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 35 2021-03-24T22:04:35.715 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: . 2021-03-24T22:04:35.715 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 126869 2021-03-24T22:04:35.715 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: +0000 2021-03-24T22:04:35.715 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 2021-03-24T22:04:35.716 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: osd.0 2021-03-24T22:04:35.716 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: ( 2021-03-24T22:04:35.716 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: osd 2021-03-24T22:04:35.716 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: . 2021-03-24T22:04:35.717 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 0 2021-03-24T22:04:35.717 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: ) 2021-03-24T22:04:35.717 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 137 2021-03-24T22:04:35.717 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: : 2021-03-24T22:04:35.718 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: cluster 2021-03-24T22:04:35.718 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 2021-03-24T22:04:35.718 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: [DBG] 2021-03-24T22:04:35.718 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 2021-03-24T22:04:35.719 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 5.1e scrub ok 2021-03-24T22:04:35.719 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 2021-03-24T22:04:35.719 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: cluster 2021-03-24T22:04:35.719 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 2021-03-24T22:04:35.720 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 2021 2021-03-24T22:04:35.720 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: - 2021-03-24T22:04:35.720 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 03 2021-03-24T22:04:35.720 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: - 2021-03-24T22:04:35.720 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 24 2021-03-24T22:04:35.721 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: T 2021-03-24T22:04:35.721 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 22 2021-03-24T22:04:35.721 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: : 2021-03-24T22:04:35.721 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 04 2021-03-24T22:04:35.721 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: : 2021-03-24T22:04:35.722 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 35 2021-03-24T22:04:35.722 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: . 2021-03-24T22:04:35.722 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 296159 2021-03-24T22:04:35.722 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: +0000 2021-03-24T22:04:35.723 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 2021-03-24T22:04:35.723 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: osd.7 2021-03-24T22:04:35.723 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: ( 2021-03-24T22:04:35.723 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: osd 2021-03-24T22:04:35.723 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: . 2021-03-24T22:04:35.724 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 7 2021-03-24T22:04:35.724 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: ) 2021-03-24T22:04:35.724 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 115 2021-03-24T22:04:35.724 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: : 2021-03-24T22:04:35.725 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: cluster 2021-03-24T22:04:35.725 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 2021-03-24T22:04:35.725 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: [DBG] 2021-03-24T22:04:35.725 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 2021-03-24T22:04:35.725 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 7.11 scrub ok 2021-03-24T22:04:35.726 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 2021-03-24T22:04:35.726 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: cluster 2021-03-24T22:04:35.726 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 2021-03-24T22:04:35.726 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 2021 2021-03-24T22:04:35.726 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: - 2021-03-24T22:04:35.727 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 03 2021-03-24T22:04:35.727 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: - 2021-03-24T22:04:35.727 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 24 2021-03-24T22:04:35.727 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: T 2021-03-24T22:04:35.727 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 22 2021-03-24T22:04:35.728 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: : 2021-03-24T22:04:35.728 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 04 2021-03-24T22:04:35.728 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: : 2021-03-24T22:04:35.728 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 35 2021-03-24T22:04:35.729 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: . 2021-03-24T22:04:35.729 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 358524 2021-03-24T22:04:35.729 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: +0000 2021-03-24T22:04:35.729 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 2021-03-24T22:04:35.729 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: mon.a 2021-03-24T22:04:35.730 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: ( 2021-03-24T22:04:35.730 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: mon 2021-03-24T22:04:35.730 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: . 2021-03-24T22:04:35.730 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 0 2021-03-24T22:04:35.730 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: ) 2021-03-24T22:04:35.731 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 1849 2021-03-24T22:04:35.731 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: : 2021-03-24T22:04:35.731 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: cluster 2021-03-24T22:04:35.731 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 2021-03-24T22:04:35.732 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: [WRN] 2021-03-24T22:04:35.732 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 2021-03-24T22:04:35.732 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: Health check update: Degraded data redundancy: 152/1860 objects degraded (8.172%), 19 pgs degraded (PG_DEGRADED) 2021-03-24T22:04:35.732 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 2021-03-24T22:04:35.733 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: cluster 2021-03-24T22:04:35.733 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 2021-03-24T22:04:35.733 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 2021 2021-03-24T22:04:35.733 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: - 2021-03-24T22:04:35.733 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 03 2021-03-24T22:04:35.734 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: - 2021-03-24T22:04:35.734 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 24 2021-03-24T22:04:35.734 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: T 2021-03-24T22:04:35.734 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 22 2021-03-24T22:04:35.735 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: : 2021-03-24T22:04:35.735 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 04 2021-03-24T22:04:35.735 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: : 2021-03-24T22:04:35.735 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 35 2021-03-24T22:04:35.735 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: . 2021-03-24T22:04:35.736 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: 403514 2021-03-24T22:04:35.736 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: +0000 mon.a (mon.0) 1850 : cluster [DBG] 2021-03-24T22:04:35.736 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:35 smithi125 conmon[24927]: osdmap e705: 8 total, 7 up, 8 in 2021-03-24T22:04:35.737 INFO:tasks.rados.rados.0.smithi125.stdout:2666: finishing write tid 3 to smithi12556470-23 2021-03-24T22:04:35.737 INFO:tasks.rados.rados.0.smithi125.stdout:2666: finishing write tid 4 to smithi12556470-23 2021-03-24T22:04:35.737 INFO:tasks.rados.rados.0.smithi125.stdout:2666: finishing write tid 5 to smithi12556470-23 2021-03-24T22:04:35.737 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 23 v 467 (ObjNum 926 snap 309 seq_num 926) dirty exists 2021-03-24T22:04:35.738 INFO:tasks.rados.rados.0.smithi125.stdout:2666: left oid 23 (ObjNum 926 snap 309 seq_num 926) 2021-03-24T22:04:35.812 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:35.826 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:35 smithi142 conmon[20125]: cluster 2021-03-24T22: 2021-03-24T22:04:35.826 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:35 smithi142 conmon[20125]: 04:34.593781+0000 osd.1 (osd.1) 70 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:04:35.827 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:35 smithi142 conmon[20125]: cluster 2021-03-24T22:04 2021-03-24T22:04:35.827 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:35 smithi142 conmon[20125]: :34.594811+0000 osd.1 (osd.1) 71 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:04:35.827 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:35 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:04:35.827 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:35 smithi142 conmon[20125]: -24T22:04:34.667172+0000 osd.1 (osd.1) 72 : cluster [DBG] 3.0 scrub ok 2021-03-24T22:04:35.827 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:35 smithi142 conmon[20125]: 2021-03-24T22:04:35.828 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:35 smithi142 conmon[20125]: cluster 2021-03-24T22 2021-03-24T22:04:35.828 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:35 smithi142 conmon[20125]: :04:35.126869+0000 osd.0 (osd.0) 137 : cluster [DBG] 5.1e scrub ok 2021-03-24T22:04:35.828 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:35 smithi142 conmon[20125]: cluster 2021-03-24T22:04: 2021-03-24T22:04:35.828 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:35 smithi142 conmon[20125]: 35.296159+0000 osd.7 (osd.7) 115 : cluster [DBG] 2021-03-24T22:04:35.829 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:35 smithi142 conmon[20125]: 7.11 scrub ok 2021-03-24T22:04:35.829 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:35 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:04:35.829 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:35 smithi142 conmon[20125]: T22:04:35.358524+0000 mon.a (mon.0) 1849 : cluster [WRN] Health check update: Degraded data redundancy: 152/1860 objects degraded (8.172%), 19 pgs degraded (PG_DEGRADED) 2021-03-24T22:04:35.829 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:35 smithi142 conmon[20125]: cluster 2021-03-24T22:04:35.403514+0000 2021-03-24T22:04:35.829 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:35 smithi142 conmon[20125]: mon.a (mon.0) 1850 : cluster [DBG] 2021-03-24T22:04:35.830 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:35 smithi142 conmon[20125]: osdmap e705: 8 total, 7 up, 8 in 2021-03-24T22:04:35.867 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:35.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:35.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:35.972 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:36.029 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:36.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:36.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:36.136 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:36.192 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:36.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:36.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:04:36.300 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:36.354 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:36.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:36.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:04:36.411 INFO:tasks.rados.rados.0.smithi125.stdout:2665: done (3 left) 2021-03-24T22:04:36.412 INFO:tasks.rados.rados.0.smithi125.stdout:2666: done (2 left) 2021-03-24T22:04:36.412 INFO:tasks.rados.rados.0.smithi125.stdout:2667: done (1 left) 2021-03-24T22:04:36.412 INFO:tasks.rados.rados.0.smithi125.stdout:2668: done (0 left) 2021-03-24T22:04:36.413 INFO:tasks.rados.rados.0.smithi125.stdout:2669: read oid 11 snap -1 2021-03-24T22:04:36.413 INFO:tasks.rados.rados.0.smithi125.stdout:2669: expect deleted 2021-03-24T22:04:36.414 INFO:tasks.rados.rados.0.smithi125.stdout:2670: rollback oid 47 current snap is 310 2021-03-24T22:04:36.414 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 47 to 192 2021-03-24T22:04:36.414 INFO:tasks.rados.rados.0.smithi125.stdout:2671: snap_create 2021-03-24T22:04:36.423 INFO:tasks.rados.rados.0.smithi125.stdout:2670: finishing rollback tid 0 to smithi12556470-47 2021-03-24T22:04:36.425 INFO:tasks.rados.rados.0.smithi125.stdout:2670: finishing rollback tid 1 to smithi12556470-47 2021-03-24T22:04:36.425 INFO:tasks.rados.rados.0.smithi125.stdout:2670: finishing rollback tid 2 to smithi12556470-47 2021-03-24T22:04:36.425 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 47 v 361 (ObjNum 466 snap 131 seq_num 466) dirty exists 2021-03-24T22:04:36.429 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.4 dump_historic_ops 2021-03-24T22:04:36.464 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:36.520 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:36.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:36.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:04:36.626 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:36.677 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:36.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:36.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:04:36.729 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T22:04:36.741 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T22:04:36.752 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T22:04:36.779 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T22:04:36.786 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:36.799 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:36 smithi125 conmon[19936]: cluster 2021-03-24T22:04:34 2021-03-24T22:04:36.800 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:36 smithi125 conmon[19936]: .958342+0000 mgr.y (mgr.14144) 627 : cluster [DBG] pgmap v1219: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 107 active+clean; 635 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.5 KiB/s rd, 2.6 MiB/s wr, 11 op/s; 152/1875 objects degraded (8.107%); 40/1875 objects misplaced (2.133%) 2021-03-24T22:04:36.800 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:36 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T22:04:36.800 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:36 smithi125 conmon[19936]: 22:04:35.824877+0000 osd.2 (osd.2) 111 : cluster [DBG] 2021-03-24T22:04:36.801 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:36 smithi125 conmon[19936]: 7.8 scrub ok 2021-03-24T22:04:36.801 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:36 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:04:36.801 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:36 smithi125 conmon[19936]: 03-24T22:04:36 2021-03-24T22:04:36.801 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:36 smithi125 conmon[19936]: .224063+0000 osd.0 (osd. 2021-03-24T22:04:36.801 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:36 smithi125 conmon[19936]: 0) 138 : cluster [DBG] 7.f deep-scrub ok 2021-03-24T22:04:36.802 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:36 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:04:36.802 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:36 smithi125 conmon[19936]: -03-24T22:04:36.405095+0000 2021-03-24T22:04:36.802 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:36 smithi125 conmon[19936]: mon.a (mon.0) 1851 : cluster [DBG] 2021-03-24T22:04:36.802 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:36 smithi125 conmon[19936]: osdmap e706: 8 total, 7 up, 8 in 2021-03-24T22:04:36.803 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:36 smithi125 conmon[24927]: cluster 2021-03-24T22:04: 2021-03-24T22:04:36.803 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:36 smithi125 conmon[24927]: 34.958342+0000 mgr.y (mgr.14144) 627 : cluster [DBG] pgmap v1219: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 107 active+clean; 635 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.5 KiB/s rd, 2.6 MiB/s wr, 11 op/s; 152/1875 objects degraded (8.107%); 40/1875 objects misplaced (2.133%) 2021-03-24T22:04:36.803 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:36 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:04:36.803 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:36 smithi125 conmon[24927]: 24T22:04:35.824877+0000 2021-03-24T22:04:36.804 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:36 smithi125 conmon[24927]: osd.2 (osd.2) 111 : cluster [DBG] 2021-03-24T22:04:36.804 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:36 smithi125 conmon[24927]: 7.8 scrub ok 2021-03-24T22:04:36.804 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:36 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T22:04:36.804 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:36 smithi125 conmon[24927]: -24T22:04:36.224063 2021-03-24T22:04:36.805 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:36 smithi125 conmon[24927]: +0000 osd.0 (osd.0) 138 : cluster [DBG] 2021-03-24T22:04:36.805 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:36 smithi125 conmon[24927]: 7.f deep-scrub ok 2021-03-24T22:04:36.805 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:36 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T22:04:36.805 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:36 smithi125 conmon[24927]: T22:04:36.405095 2021-03-24T22:04:36.805 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:36 smithi125 conmon[24927]: +0000 mon.a (mon.0) 1851 2021-03-24T22:04:36.806 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:36 smithi125 conmon[24927]: : cluster [DBG] osdmap e706: 8 total, 7 up, 8 in 2021-03-24T22:04:36.806 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T22:04:36.820 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T22:04:36.835 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:36.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:36.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:36.842 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T22:04:36.863 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T22:04:36.864 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-24T22:04:36.864 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-24T22:04:36.864 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-24T22:04:36.865 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-24T22:04:36.865 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-24T22:04:36.865 INFO:teuthology.orchestra.run.smithi125.stderr:Error ENXIO: problem getting command descriptions from osd.5 2021-03-24T22:04:36.865 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-24T22:04:36.866 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-24T22:04:36.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:36 smithi142 conmon[20125]: cluster 2021-03-24T22:04:34.958342+0000 mgr.y (mgr.14144) 627 : cluster 2021-03-24T22:04:36.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:36 smithi142 conmon[20125]: [DBG] pgmap v1219: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 107 active+clean; 635 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.5 KiB/s rd, 2.6 MiB/s wr, 11 op/s; 152/1875 objects degraded (8.107%); 40/1875 objects misplaced (2.133%) 2021-03-24T22:04:36.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:36 smithi142 conmon[20125]: cluster 2021-03-24T22:04:35.824877+0000 osd.2 (osd.2) 111 : cluster [DBG] 7.8 scrub ok 2021-03-24T22:04:36.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:36 smithi142 conmon[20125]: cluster 2021-03-24T22:04:36. 2021-03-24T22:04:36.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:36 smithi142 conmon[20125]: 224063+0000 osd.0 (osd.0) 138 : cluster [DBG] 7.f deep-scrub ok 2021-03-24T22:04:36.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:36 smithi142 conmon[20125]: cluster 2021-03-24T22: 2021-03-24T22:04:36.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:36 smithi142 conmon[20125]: 04:36.405095+0000 mon.a (mon.0) 1851 : cluster [DBG] osdmap e706: 8 total, 7 up, 8 in 2021-03-24T22:04:36.942 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:36.998 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:37.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:37.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:04:37.104 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:37.166 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:37.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:37.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:37.273 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:37.326 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:37.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:37.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:37.415 INFO:tasks.rados.rados.0.smithi125.stdout:2669: done (2 left) 2021-03-24T22:04:37.415 INFO:tasks.rados.rados.0.smithi125.stdout:2670: done (1 left) 2021-03-24T22:04:37.416 INFO:tasks.rados.rados.0.smithi125.stdout:2671: done (0 left) 2021-03-24T22:04:37.416 INFO:tasks.rados.rados.0.smithi125.stdout:2672: write oid 49 current snap is 311 2021-03-24T22:04:37.416 INFO:tasks.rados.rados.0.smithi125.stdout:2672: seq_num 927 ranges {780089=634536,1879396=620991,3240526=339563} 2021-03-24T22:04:37.423 INFO:tasks.rados.rados.0.smithi125.stdout:2672: writing smithi12556470-49 from 780089 to 1414625 tid 1 2021-03-24T22:04:37.430 INFO:tasks.rados.rados.0.smithi125.stdout:2672: writing smithi12556470-49 from 1879396 to 2500387 tid 2 2021-03-24T22:04:37.433 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:37.437 INFO:tasks.rados.rados.0.smithi125.stdout:2672: writing smithi12556470-49 from 3240526 to 3580089 tid 3 2021-03-24T22:04:37.437 INFO:tasks.rados.rados.0.smithi125.stdout:2673: write (excl) oid 34 current snap is 311 2021-03-24T22:04:37.438 INFO:tasks.rados.rados.0.smithi125.stdout:2673: seq_num 928 ranges {796896=426138,1654231=764649,2984302=480075} 2021-03-24T22:04:37.445 INFO:tasks.rados.rados.0.smithi125.stdout:2673: writing smithi12556470-34 from 796896 to 1223034 tid 1 2021-03-24T22:04:37.453 INFO:tasks.rados.rados.0.smithi125.stdout:2673: writing smithi12556470-34 from 1654231 to 2418880 tid 2 2021-03-24T22:04:37.461 INFO:tasks.rados.rados.0.smithi125.stdout:2673: writing smithi12556470-34 from 2984302 to 3464377 tid 3 2021-03-24T22:04:37.461 INFO:tasks.rados.rados.0.smithi125.stdout:2674: write oid 47 current snap is 311 2021-03-24T22:04:37.462 INFO:tasks.rados.rados.0.smithi125.stdout:2674: seq_num 929 ranges {413703=501387,1596360=624660,2895372=691347} 2021-03-24T22:04:37.466 INFO:tasks.rados.rados.0.smithi125.stdout:2674: writing smithi12556470-47 from 413703 to 915090 tid 1 2021-03-24T22:04:37.475 INFO:tasks.rados.rados.0.smithi125.stdout:2674: writing smithi12556470-47 from 1596360 to 2221020 tid 2 2021-03-24T22:04:37.481 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:37.484 INFO:tasks.rados.rados.0.smithi125.stdout:2674: writing smithi12556470-47 from 2895372 to 3586719 tid 3 2021-03-24T22:04:37.485 INFO:tasks.rados.rados.0.smithi125.stdout:2675: snap_remove snap 254 2021-03-24T22:04:37.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:37.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:37.587 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:37.639 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:37.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:37.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:37.744 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:37.758 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:37 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:04:37.759 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:37 smithi142 conmon[20125]: -24T22:04:36.595976+0000 osd.4 (osd.4) 119 : cluster [DBG] 2021-03-24T22:04:37.759 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:37 smithi142 conmon[20125]: purged_snaps scrub starts 2021-03-24T22:04:37.759 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:37 smithi142 conmon[20125]: cluster 2021-03-24T22: 2021-03-24T22:04:37.759 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:37 smithi142 conmon[20125]: 04:36.596921+0000 osd.4 (osd.4) 120 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:04:37.760 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:37 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:04:37.760 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:37 smithi142 conmon[20125]: 22:04:36.619525+0000 osd.1 ( 2021-03-24T22:04:37.760 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:37 smithi142 conmon[20125]: osd.1) 73 : cluster [DBG] 3.25 scrub ok 2021-03-24T22:04:37.760 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:37 smithi142 conmon[20125]: 2021-03-24T22:04:37.760 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:37 smithi142 conmon[20125]: audit 2021-03-24T22:04:37.761 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:37 smithi142 conmon[20125]: 2021-03-24T22:04 2021-03-24T22:04:37.761 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:37 smithi142 conmon[20125]: :36.704014+0000 mon.a (mon.0) 1852 : audit [DBG] from='client.? v1:172.21.15.125:0/46143693' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:04:37.761 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:37 smithi142 conmon[20125]: cluster 2021-03-24T22:04:37.761 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:37 smithi142 conmon[20125]: 2021-03-24 2021-03-24T22:04:37.763 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:37 smithi142 conmon[20125]: T22:04:37.120842+0000 2021-03-24T22:04:37.763 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:37 smithi142 conmon[20125]: osd.0 (osd.0) 139 : cluster [DBG] 2021-03-24T22:04:37.763 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:37 smithi142 conmon[20125]: 6.0 scrub ok 2021-03-24T22:04:37.763 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:37 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:04:37.764 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:37 smithi142 conmon[20125]: T22:04:37.274294+0000 2021-03-24T22:04:37.764 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:37 smithi142 conmon[20125]: osd.7 (osd.7) 116 : cluster [DBG] 2021-03-24T22:04:37.764 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:37 smithi142 conmon[20125]: 3.13 scrub ok 2021-03-24T22:04:37.764 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:37 smithi142 conmon[20125]: cluster 2021-03-24T22:04:37.765 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:37 smithi142 conmon[20125]: 2021-03-24T22:04 2021-03-24T22:04:37.765 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:37 smithi142 conmon[20125]: :37.278276+0000 osd.3 (osd.3) 2021-03-24T22:04:37.765 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:37 smithi142 conmon[20125]: 143 : cluster [DBG] 4.14 scrub ok 2021-03-24T22:04:37.765 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:37 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:04:37.766 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:37 smithi142 conmon[20125]: T22:04:37.406607+0000 mon.a (mon 2021-03-24T22:04:37.766 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:37 smithi142 conmon[20125]: .0) 1853 : cluster [DBG] osdmap e707: 8 total, 7 up, 8 in 2021-03-24T22:04:37.801 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:37.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:37.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:04:37.907 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:37.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[19936]: cluster 2021-03-24T22:04:36.595976+0000 osd.4 (osd.4 2021-03-24T22:04:37.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[19936]: ) 119 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:04:37.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[19936]: cluster 2021-03-24T22:04:36.596921+0000 osd.4 (osd.4) 120 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:04:37.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[19936]: cluster 2021-03-24T22:04:36.619525+0000 osd.1 (osd.1 2021-03-24T22:04:37.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[19936]: ) 73 : cluster [DBG] 3.25 scrub ok 2021-03-24T22:04:37.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[19936]: audit 2021-03-24T22:04:36.704014+0000 mon.a (mon.0) 1852 : audit [DBG] from='client.? v1:172.21.15.125:0/46143693' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:04:37.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[19936]: 2021-03-24T22:04:37.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[19936]: cluster 2021-03-24T22:04:37.120842+0000 osd.0 (osd.0 2021-03-24T22:04:37.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[19936]: ) 139 : cluster [DBG] 6.0 scrub ok 2021-03-24T22:04:37.915 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[19936]: cluster 2021-03-24T22:04: 2021-03-24T22:04:37.915 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[19936]: 37.274294+0000 osd.7 (osd.7) 116 : cluster [DBG] 3.13 scrub ok 2021-03-24T22:04:37.915 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[19936]: cluster 2021-03-24T22:04:37.915 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[19936]: 2021-03-24T22:04:37.278276+0000 osd.3 (osd.3) 143 : cluster [DBG] 4.14 scrub ok 2021-03-24T22:04:37.916 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[19936]: cluster 2021-03-24T22:04:37.406607+0000 2021-03-24T22:04:37.916 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[19936]: mon.a (mon.0) 1853 : cluster [DBG] osdmap e707: 8 total, 7 up, 8 in 2021-03-24T22:04:37.917 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: cluster 2021-03-24T22:04:37.917 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: 2021-03-24T22:04:37.917 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: 2021 2021-03-24T22:04:37.917 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: - 2021-03-24T22:04:37.918 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: 03 2021-03-24T22:04:37.918 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: - 2021-03-24T22:04:37.918 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: 24 2021-03-24T22:04:37.918 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: T 2021-03-24T22:04:37.919 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: 22 2021-03-24T22:04:37.919 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: : 2021-03-24T22:04:37.919 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: 04 2021-03-24T22:04:37.919 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: : 2021-03-24T22:04:37.919 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: 36 2021-03-24T22:04:37.920 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: . 2021-03-24T22:04:37.920 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: 595976 2021-03-24T22:04:37.920 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: +0000 2021-03-24T22:04:37.920 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: 2021-03-24T22:04:37.921 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: osd.4 2021-03-24T22:04:37.921 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: ( 2021-03-24T22:04:37.921 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: osd 2021-03-24T22:04:37.921 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: . 2021-03-24T22:04:37.921 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: 4 2021-03-24T22:04:37.922 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: ) 2021-03-24T22:04:37.922 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: 119 2021-03-24T22:04:37.922 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: : 2021-03-24T22:04:37.922 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: cluster 2021-03-24T22:04:37.923 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: 2021-03-24T22:04:37.923 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: [DBG] 2021-03-24T22:04:37.923 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: 2021-03-24T22:04:37.923 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: purged_snaps scrub starts 2021-03-24T22:04:37.924 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: 2021-03-24T22:04:37.924 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: cluster 2021-03-24T22:04:37.924 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: 2021-03-24T22:04:37.924 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: 2021 2021-03-24T22:04:37.924 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: - 2021-03-24T22:04:37.925 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: 03 2021-03-24T22:04:37.925 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: - 2021-03-24T22:04:37.925 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: 24 2021-03-24T22:04:37.925 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: T 2021-03-24T22:04:37.925 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: 22 2021-03-24T22:04:37.926 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: : 2021-03-24T22:04:37.926 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: 04 2021-03-24T22:04:37.926 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: : 2021-03-24T22:04:37.926 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: 36 2021-03-24T22:04:37.927 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: . 2021-03-24T22:04:37.927 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: 596921 2021-03-24T22:04:37.927 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: +0000 2021-03-24T22:04:37.927 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: 2021-03-24T22:04:37.927 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: osd.4 2021-03-24T22:04:37.928 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: ( 2021-03-24T22:04:37.928 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: osd 2021-03-24T22:04:37.928 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: . 2021-03-24T22:04:37.928 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: 4 2021-03-24T22:04:37.929 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: ) 2021-03-24T22:04:37.929 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: 120 2021-03-24T22:04:37.929 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: : 2021-03-24T22:04:37.929 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: cluster 2021-03-24T22:04:37.929 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: 2021-03-24T22:04:37.930 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: [DBG] 2021-03-24T22:04:37.930 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: 2021-03-24T22:04:37.930 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: purged_snaps scrub ok 2021-03-24T22:04:37.930 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: 2021-03-24T22:04:37.931 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: cluster 2021-03-24T22:04:37.931 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: 2021-03-24T22:04:37.931 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: 2021 2021-03-24T22:04:37.931 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: - 2021-03-24T22:04:37.932 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: 03 2021-03-24T22:04:37.932 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: - 2021-03-24T22:04:37.932 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: 24 2021-03-24T22:04:37.932 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: T 2021-03-24T22:04:37.932 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: 22 2021-03-24T22:04:37.933 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: : 2021-03-24T22:04:37.933 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: 04 2021-03-24T22:04:37.933 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: : 2021-03-24T22:04:37.933 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: 36 2021-03-24T22:04:37.934 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: . 2021-03-24T22:04:37.934 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: 619525 2021-03-24T22:04:37.934 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: +0000 2021-03-24T22:04:37.934 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: 2021-03-24T22:04:37.935 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: osd.1 2021-03-24T22:04:37.935 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: ( 2021-03-24T22:04:37.935 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: osd 2021-03-24T22:04:37.935 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: . 2021-03-24T22:04:37.935 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: 1 2021-03-24T22:04:37.936 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: ) 2021-03-24T22:04:37.936 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: 73 : cluster [DBG] 3.25 scrub ok 2021-03-24T22:04:37.936 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: audit 2021-03-24T22:04:37.936 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: 2021-03-24T22:04 2021-03-24T22:04:37.937 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: :36.704014+0000 mon.a ( 2021-03-24T22:04:37.937 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: mon.0) 1852 : audit 2021-03-24T22:04:37.937 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: [DBG] from='client.? v1:172.21.15.125:0/46143693' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:04:37.937 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: cluster 2021-03-24T22:04:37.937 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: 2021-03-24T22:04 2021-03-24T22:04:37.938 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: :37.120842+0000 osd.0 ( 2021-03-24T22:04:37.938 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: osd.0) 139 : cluster 2021-03-24T22:04:37.938 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: [DBG] 6.0 scrub ok 2021-03-24T22:04:37.938 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: cluster 2021-03-24T22:04:37.939 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: 2021-03-24T22:04 2021-03-24T22:04:37.939 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: :37.274294+0000 osd.7 2021-03-24T22:04:37.939 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: (osd.7) 116 : cluster 2021-03-24T22:04:37.939 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: [DBG] 3.13 scrub ok 2021-03-24T22:04:37.940 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: cluster 2021-03-24T22:04:37.940 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: 2021-03-24T22: 2021-03-24T22:04:37.940 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: 04:37.278276+0000 osd.3 2021-03-24T22:04:37.940 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: (osd.3) 143 : 2021-03-24T22:04:37.940 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: cluster [DBG] 4.14 scrub ok 2021-03-24T22:04:37.941 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: cluster 2021-03-24T22:04:37.941 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: 2021-03-24T22: 2021-03-24T22:04:37.941 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: 04:37.406607+0000 mon.a 2021-03-24T22:04:37.941 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: (mon.0) 1853 : cluster 2021-03-24T22:04:37.942 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:37 smithi125 conmon[24927]: [DBG] osdmap e707: 8 total, 7 up, 8 in 2021-03-24T22:04:37.966 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:37.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:37.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:38.073 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:38.129 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:38.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:38.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:38.236 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:38.288 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:38.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:38.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:38.301 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd set noscrub 2021-03-24T22:04:38.395 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:38.418 INFO:tasks.rados.rados.0.smithi125.stdout:2675: done (3 left) 2021-03-24T22:04:38.418 INFO:tasks.rados.rados.0.smithi125.stdout:2676: read oid 5 snap -1 2021-03-24T22:04:38.418 INFO:tasks.rados.rados.0.smithi125.stdout:2676: expect (ObjNum 921 snap 308 seq_num 921) 2021-03-24T22:04:38.419 INFO:tasks.rados.rados.0.smithi125.stdout:2677: read oid 36 snap -1 2021-03-24T22:04:38.419 INFO:tasks.rados.rados.0.smithi125.stdout:2677: expect (ObjNum 919 snap 307 seq_num 919) 2021-03-24T22:04:38.419 INFO:tasks.rados.rados.0.smithi125.stdout:2678: copy_from oid 45 from oid 38 current snap is 311 2021-03-24T22:04:38.419 INFO:tasks.rados.rados.0.smithi125.stdout:2679: read oid 30 snap -1 2021-03-24T22:04:38.420 INFO:tasks.rados.rados.0.smithi125.stdout:2679: expect (ObjNum 875 snap 283 seq_num 875) 2021-03-24T22:04:38.420 INFO:tasks.rados.rados.0.smithi125.stdout:2680: delete oid 1 current snap is 311 2021-03-24T22:04:38.420 INFO:tasks.rados.rados.0.smithi125.stdout:2672: finishing write tid 1 to smithi12556470-49 2021-03-24T22:04:38.420 INFO:tasks.rados.rados.0.smithi125.stdout:2672: finishing write tid 2 to smithi12556470-49 2021-03-24T22:04:38.421 INFO:tasks.rados.rados.0.smithi125.stdout:2672: finishing write tid 3 to smithi12556470-49 2021-03-24T22:04:38.421 INFO:tasks.rados.rados.0.smithi125.stdout:2672: finishing write tid 4 to smithi12556470-49 2021-03-24T22:04:38.421 INFO:tasks.rados.rados.0.smithi125.stdout:2672: finishing write tid 5 to smithi12556470-49 2021-03-24T22:04:38.421 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 49 v 947 (ObjNum 927 snap 311 seq_num 927) dirty exists 2021-03-24T22:04:38.421 INFO:tasks.rados.rados.0.smithi125.stdout:2672: left oid 49 (ObjNum 927 snap 311 seq_num 927) 2021-03-24T22:04:38.422 INFO:tasks.rados.rados.0.smithi125.stdout:2673: finishing write tid 1 to smithi12556470-34 2021-03-24T22:04:38.422 INFO:tasks.rados.rados.0.smithi125.stdout:2673: finishing write tid 2 to smithi12556470-34 2021-03-24T22:04:38.422 INFO:tasks.rados.rados.0.smithi125.stdout:2673: finishing write tid 3 to smithi12556470-34 2021-03-24T22:04:38.422 INFO:tasks.rados.rados.0.smithi125.stdout:2673: finishing write tid 4 to smithi12556470-34 2021-03-24T22:04:38.423 INFO:tasks.rados.rados.0.smithi125.stdout:2673: finishing write tid 5 to smithi12556470-34 2021-03-24T22:04:38.423 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 34 v 1419 (ObjNum 928 snap 311 seq_num 928) dirty exists 2021-03-24T22:04:38.423 INFO:tasks.rados.rados.0.smithi125.stdout:2673: left oid 34 (ObjNum 928 snap 311 seq_num 928) 2021-03-24T22:04:38.423 INFO:tasks.rados.rados.0.smithi125.stdout:2674: finishing write tid 1 to smithi12556470-47 2021-03-24T22:04:38.424 INFO:tasks.rados.rados.0.smithi125.stdout:2674: finishing write tid 2 to smithi12556470-47 2021-03-24T22:04:38.424 INFO:tasks.rados.rados.0.smithi125.stdout:2674: finishing write tid 3 to smithi12556470-47 2021-03-24T22:04:38.424 INFO:tasks.rados.rados.0.smithi125.stdout:2674: finishing write tid 4 to smithi12556470-47 2021-03-24T22:04:38.424 INFO:tasks.rados.rados.0.smithi125.stdout:2674: finishing write tid 5 to smithi12556470-47 2021-03-24T22:04:38.424 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 47 v 365 (ObjNum 929 snap 311 seq_num 929) dirty exists 2021-03-24T22:04:38.425 INFO:tasks.rados.rados.0.smithi125.stdout:2674: left oid 47 (ObjNum 929 snap 311 seq_num 929) 2021-03-24T22:04:38.429 INFO:tasks.rados.rados.0.smithi125.stdout:2679: expect (ObjNum 875 snap 283 seq_num 875) 2021-03-24T22:04:38.469 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:38.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:38.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:38.524 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-03-24T22:04:38.525 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-03-24T22:04:38.573 INFO:tasks.rados.rados.0.smithi125.stdout:2672: done (7 left) 2021-03-24T22:04:38.574 INFO:tasks.rados.rados.0.smithi125.stdout:2673: done (6 left) 2021-03-24T22:04:38.574 INFO:tasks.rados.rados.0.smithi125.stdout:2674: done (5 left) 2021-03-24T22:04:38.574 INFO:tasks.rados.rados.0.smithi125.stdout:2679: done (4 left) 2021-03-24T22:04:38.574 INFO:tasks.rados.rados.0.smithi125.stdout:2680: done (3 left) 2021-03-24T22:04:38.575 INFO:tasks.rados.rados.0.smithi125.stdout:2681: rollback oid 29 current snap is 311 2021-03-24T22:04:38.575 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 29 to 296 2021-03-24T22:04:38.575 INFO:tasks.rados.rados.0.smithi125.stdout:2676: expect (ObjNum 921 snap 308 seq_num 921) 2021-03-24T22:04:38.579 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:38.632 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:38.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:38.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:38.672 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:04:38.738 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:38.746 INFO:tasks.rados.rados.0.smithi125.stdout:2677: expect (ObjNum 919 snap 307 seq_num 919) 2021-03-24T22:04:38.791 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:38.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:38.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:38.854 INFO:tasks.rados.rados.0.smithi125.stdout:2678: finishing copy_from to smithi12556470-45 2021-03-24T22:04:38.854 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 45 v 731 (ObjNum 733 snap 235 seq_num 733) dirty exists 2021-03-24T22:04:38.854 INFO:tasks.rados.rados.0.smithi125.stdout:2678: finishing copy_from racing read to smithi12556470-45 2021-03-24T22:04:38.855 INFO:tasks.rados.rados.0.smithi125.stdout:2681: finishing rollback tid 0 to smithi12556470-29 2021-03-24T22:04:38.855 INFO:tasks.rados.rados.0.smithi125.stdout:2681: finishing rollback tid 1 to smithi12556470-29 2021-03-24T22:04:38.855 INFO:tasks.rados.rados.0.smithi125.stdout:2681: finishing rollback tid 2 to smithi12556470-29 2021-03-24T22:04:38.855 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 29 v 457 (ObjNum 872 snap 283 seq_num 872) dirty exists 2021-03-24T22:04:38.856 INFO:tasks.rados.rados.0.smithi125.stdout:2676: done (3 left) 2021-03-24T22:04:38.856 INFO:tasks.rados.rados.0.smithi125.stdout:2677: done (2 left) 2021-03-24T22:04:38.856 INFO:tasks.rados.rados.0.smithi125.stdout:2678: done (1 left) 2021-03-24T22:04:38.856 INFO:tasks.rados.rados.0.smithi125.stdout:2681: done (0 left) 2021-03-24T22:04:38.856 INFO:tasks.rados.rados.0.smithi125.stdout:2682: read oid 2 snap -1 2021-03-24T22:04:38.857 INFO:tasks.rados.rados.0.smithi125.stdout:2682: expect (ObjNum 917 snap 305 seq_num 917) 2021-03-24T22:04:38.857 INFO:tasks.rados.rados.0.smithi125.stdout:2683: rollback oid 42 current snap is 311 2021-03-24T22:04:38.857 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 42 to 269 2021-03-24T22:04:38.857 INFO:tasks.rados.rados.0.smithi125.stdout:2684: read oid 47 snap 308 2021-03-24T22:04:38.858 INFO:tasks.rados.rados.0.smithi125.stdout:2684: expect (ObjNum 466 snap 131 seq_num 466) 2021-03-24T22:04:38.858 INFO:tasks.rados.rados.0.smithi125.stdout:2685: rollback oid 17 current snap is 311 2021-03-24T22:04:38.858 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 17 to 297 2021-03-24T22:04:38.858 INFO:tasks.rados.rados.0.smithi125.stdout:2686: copy_from oid 30 from oid 31 current snap is 311 2021-03-24T22:04:38.858 INFO:tasks.rados.rados.0.smithi125.stdout:2687: read oid 4 snap 295 2021-03-24T22:04:38.859 INFO:tasks.rados.rados.0.smithi125.stdout:2687: expect (ObjNum 865 snap 281 seq_num 865) 2021-03-24T22:04:38.859 INFO:tasks.rados.rados.0.smithi125.stdout:2688: delete oid 25 current snap is 311 2021-03-24T22:04:38.860 INFO:tasks.rados.rados.0.smithi125.stdout:2686: finishing copy_from to smithi12556470-30 2021-03-24T22:04:38.860 INFO:tasks.rados.rados.0.smithi125.stdout:2686: got expected ENOENT (src dne) 2021-03-24T22:04:38.860 INFO:tasks.rados.rados.0.smithi125.stdout:2686: finishing copy_from racing read to smithi12556470-30 2021-03-24T22:04:38.861 INFO:tasks.rados.rados.0.smithi125.stdout:2683: finishing rollback tid 0 to smithi12556470-42 2021-03-24T22:04:38.861 INFO:tasks.rados.rados.0.smithi125.stdout:2687: expect (ObjNum 865 snap 281 seq_num 865) 2021-03-24T22:04:38.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:38 smithi142 conmon[20125]: cluster 2021-03-24T22:04: 2021-03-24T22:04:38.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:38 smithi142 conmon[20125]: 36.959160+0000 mgr.y (mgr.14144) 628 : cluster [DBG] pgmap v1222: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 107 active+clean; 638 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.2 MiB/s wr, 10 op/s; 152/1875 objects degraded (8.107%); 40/1875 objects misplaced (2.133%) 2021-03-24T22:04:38.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:38 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:04:38.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:38 smithi142 conmon[20125]: 24T22:04:37.590128+0000 osd.1 (osd.1) 74 : cluster [DBG] 7.1d scrub ok 2021-03-24T22:04:38.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:38 smithi142 conmon[20125]: cluster 2021-03-24T22:04:38.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:38 smithi142 conmon[20125]: 2021- 2021-03-24T22:04:38.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:38 smithi142 conmon[20125]: 03-24T22:04:37.814756+0000 osd.2 (osd.2 2021-03-24T22:04:38.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:38 smithi142 conmon[20125]: ) 112 : cluster [DBG] 2.1 scrub ok 2021-03-24T22:04:38.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:38 smithi142 conmon[20125]: cluster 2021-03-24T22:04:38.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:38 smithi142 conmon[20125]: 2021-03-24T22:04:38.228347+0000 2021-03-24T22:04:38.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:38 smithi142 conmon[20125]: osd.7 (osd.7) 117 : cluster [DBG] 5.10 deep-scrub ok 2021-03-24T22:04:38.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:38 smithi142 conmon[20125]: 2021-03-24T22:04:38.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:38 smithi142 conmon[20125]: cluster 2021-03-24T22:04:38. 2021-03-24T22:04:38.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:38 smithi142 conmon[20125]: 409170+0000 mon.a (mon.0) 1854 : cluster 2021-03-24T22:04:38.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:38 smithi142 conmon[20125]: [DBG] osdmap e708: 8 total, 7 up, 8 in 2021-03-24T22:04:38.892 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:38 smithi125 conmon[19936]: cluster 2021-03-24T22:04:36. 2021-03-24T22:04:38.893 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:38 smithi125 conmon[19936]: 959160+0000 mgr.y (mgr.14144) 628 : cluster [DBG] pgmap v1222: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 107 active+clean; 638 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.2 MiB/s wr, 10 op/s; 152/1875 objects degraded (8.107%); 40/1875 objects misplaced (2.133%) 2021-03-24T22:04:38.893 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:38 smithi125 conmon[19936]: cluster 2021-03-24T22:04:37. 2021-03-24T22:04:38.893 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:38 smithi125 conmon[19936]: 590128+0000 osd.1 (osd.1) 74 : cluster [DBG] 7.1d scrub ok 2021-03-24T22:04:38.893 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:38 smithi125 conmon[19936]: cluster 2021-03-24T22:04:37. 2021-03-24T22:04:38.894 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:38 smithi125 conmon[19936]: 814756+0000 osd.2 (osd.2) 112 : cluster 2021-03-24T22:04:38.894 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:38 smithi125 conmon[19936]: [DBG] 2.1 scrub ok 2021-03-24T22:04:38.894 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:38 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:04:38.894 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:38 smithi125 conmon[19936]: 04:38.228347+0000 osd.7 (osd 2021-03-24T22:04:38.895 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:38 smithi125 conmon[19936]: .7) 117 : cluster [DBG] 5.10 deep-scrub ok 2021-03-24T22:04:38.895 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:38 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:04:38.895 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:38 smithi125 conmon[19936]: -03-24T22:04:38 2021-03-24T22:04:38.895 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:38 smithi125 conmon[19936]: .409170+0000 mon.a (mon.0) 2021-03-24T22:04:38.896 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:38 smithi125 conmon[19936]: 1854 : cluster [DBG] osdmap e708: 8 total, 7 up, 8 in 2021-03-24T22:04:38.896 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:38 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:04:38.896 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:38 smithi125 conmon[24927]: 04:36.959160+0000 mgr.y (mgr.14144) 628 : cluster [DBG] pgmap v1222: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 107 active+clean; 638 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.2 MiB/s wr, 10 op/s; 152/1875 objects degraded (8.107%); 40/1875 objects misplaced (2.133%) 2021-03-24T22:04:38.896 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:38 smithi125 conmon[24927]: cluster 2021-03-24T22:04:38.897 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:38 smithi125 conmon[24927]: 2021-03-24T22:04:37 2021-03-24T22:04:38.897 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:38 smithi125 conmon[24927]: .590128+0000 osd.1 (osd.1) 2021-03-24T22:04:38.897 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:38 smithi125 conmon[24927]: 74 : cluster [DBG] 7.1d scrub ok 2021-03-24T22:04:38.897 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:38 smithi125 conmon[24927]: cluster 2021-03-24T22:04 2021-03-24T22:04:38.897 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:38 smithi125 conmon[24927]: :37.814756+0000 osd.2 (osd.2) 112 : 2021-03-24T22:04:38.898 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:38 smithi125 conmon[24927]: cluster [DBG] 2.1 scrub ok 2021-03-24T22:04:38.898 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:38 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T22:04:38.898 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:38 smithi125 conmon[24927]: -24T22:04:38.228347+0000 2021-03-24T22:04:38.898 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:38 smithi125 conmon[24927]: osd.7 (osd.7) 117 : cluster 2021-03-24T22:04:38.899 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:38 smithi125 conmon[24927]: [DBG] 5.10 deep-scrub ok 2021-03-24T22:04:38.899 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:38 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:04:38.899 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:38 smithi125 conmon[24927]: 24T22:04:38.409170+0000 2021-03-24T22:04:38.899 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:38 smithi125 conmon[24927]: mon.a (mon.0) 1854 : cluster [DBG] 2021-03-24T22:04:38.899 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:38 smithi125 conmon[24927]: osdmap e708: 8 total, 7 up, 8 in 2021-03-24T22:04:38.900 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:38.950 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:38.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:38.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:38.998 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2021-03-24T22:04:38.998 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T22:04:39.004 INFO:tasks.rados.rados.0.smithi125.stdout:2683: finishing rollback tid 1 to smithi12556470-42 2021-03-24T22:04:39.004 INFO:tasks.rados.rados.0.smithi125.stdout:2683: finishing rollback tid 2 to smithi12556470-42 2021-03-24T22:04:39.004 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 42 v 506 (ObjNum 736 snap 235 seq_num 736) dirty exists 2021-03-24T22:04:39.004 INFO:tasks.rados.rados.0.smithi125.stdout:2682: expect (ObjNum 917 snap 305 seq_num 917) 2021-03-24T22:04:39.054 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:39.113 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:39.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:39.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:04:39.196 INFO:tasks.rados.rados.0.smithi125.stdout:2685: finishing rollback tid 0 to smithi12556470-17 2021-03-24T22:04:39.196 INFO:tasks.rados.rados.0.smithi125.stdout:2685: finishing rollback tid 1 to smithi12556470-17 2021-03-24T22:04:39.196 INFO:tasks.rados.rados.0.smithi125.stdout:2685: finishing rollback tid 2 to smithi12556470-17 2021-03-24T22:04:39.197 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 17 v 1028 (ObjNum 744 snap 240 seq_num 744) dirty exists 2021-03-24T22:04:39.197 INFO:tasks.rados.rados.0.smithi125.stdout:2684: expect (ObjNum 466 snap 131 seq_num 466) 2021-03-24T22:04:39.218 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:39.278 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:39.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:39.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:04:39.319 INFO:tasks.rados.rados.0.smithi125.stdout:2682: done (6 left) 2021-03-24T22:04:39.320 INFO:tasks.rados.rados.0.smithi125.stdout:2683: done (5 left) 2021-03-24T22:04:39.320 INFO:tasks.rados.rados.0.smithi125.stdout:2684: done (4 left) 2021-03-24T22:04:39.321 INFO:tasks.rados.rados.0.smithi125.stdout:2685: done (3 left) 2021-03-24T22:04:39.321 INFO:tasks.rados.rados.0.smithi125.stdout:2686: done (2 left) 2021-03-24T22:04:39.322 INFO:tasks.rados.rados.0.smithi125.stdout:2687: done (1 left) 2021-03-24T22:04:39.322 INFO:tasks.rados.rados.0.smithi125.stdout:2688: done (0 left) 2021-03-24T22:04:39.322 INFO:tasks.rados.rados.0.smithi125.stdout:2689: write (excl) oid 20 current snap is 311 2021-03-24T22:04:39.323 INFO:tasks.rados.rados.0.smithi125.stdout:2689: seq_num 932 ranges {464124=727134,1771806=771987,2977641=641516} 2021-03-24T22:04:39.335 INFO:tasks.rados.rados.0.smithi125.stdout:2689: writing smithi12556470-20 from 464124 to 1191258 tid 1 2021-03-24T22:04:39.342 INFO:tasks.rados.rados.0.smithi125.stdout:2689: writing smithi12556470-20 from 1771806 to 2543793 tid 2 2021-03-24T22:04:39.349 INFO:tasks.rados.rados.0.smithi125.stdout:2689: writing smithi12556470-20 from 2977641 to 3619157 tid 3 2021-03-24T22:04:39.349 INFO:tasks.rados.rados.0.smithi125.stdout:2690: read oid 36 snap -1 2021-03-24T22:04:39.349 INFO:tasks.rados.rados.0.smithi125.stdout:2690: expect (ObjNum 919 snap 307 seq_num 919) 2021-03-24T22:04:39.349 INFO:tasks.rados.rados.0.smithi125.stdout:2691: write (excl) oid 3 current snap is 311 2021-03-24T22:04:39.350 INFO:tasks.rados.rados.0.smithi125.stdout:2689: finishing write tid 1 to smithi12556470-20 2021-03-24T22:04:39.350 INFO:tasks.rados.rados.0.smithi125.stdout:2691: seq_num 933 ranges {480931=402383,1313935=515645,2488711=452788,3443230=237701} 2021-03-24T22:04:39.354 INFO:tasks.rados.rados.0.smithi125.stdout:2691: writing smithi12556470-3 from 480931 to 883314 tid 1 2021-03-24T22:04:39.359 INFO:tasks.rados.rados.0.smithi125.stdout:2691: writing smithi12556470-3 from 1313935 to 1829580 tid 2 2021-03-24T22:04:39.365 INFO:tasks.rados.rados.0.smithi125.stdout:2691: writing smithi12556470-3 from 2488711 to 2941499 tid 3 2021-03-24T22:04:39.370 INFO:tasks.rados.rados.0.smithi125.stdout:2691: writing smithi12556470-3 from 3443230 to 3680931 tid 4 2021-03-24T22:04:39.370 INFO:tasks.rados.rados.0.smithi125.stdout:2692: copy_from oid 14 from oid 7 current snap is 311 2021-03-24T22:04:39.370 INFO:tasks.rados.rados.0.smithi125.stdout:2689: finishing write tid 2 to smithi12556470-20 2021-03-24T22:04:39.371 INFO:tasks.rados.rados.0.smithi125.stdout:2689: finishing write tid 3 to smithi12556470-20 2021-03-24T22:04:39.371 INFO:tasks.rados.rados.0.smithi125.stdout:2689: finishing write tid 4 to smithi12556470-20 2021-03-24T22:04:39.371 INFO:tasks.rados.rados.0.smithi125.stdout:2689: finishing write tid 5 to smithi12556470-20 2021-03-24T22:04:39.371 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 20 v 812 (ObjNum 932 snap 311 seq_num 932) dirty exists 2021-03-24T22:04:39.371 INFO:tasks.rados.rados.0.smithi125.stdout:2689: left oid 20 (ObjNum 932 snap 311 seq_num 932) 2021-03-24T22:04:39.372 INFO:tasks.rados.rados.0.smithi125.stdout:2689: done (3 left) 2021-03-24T22:04:39.372 INFO:tasks.rados.rados.0.smithi125.stdout:2693: rollback oid 27 current snap is 311 2021-03-24T22:04:39.372 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 27 to 282 2021-03-24T22:04:39.372 INFO:tasks.rados.rados.0.smithi125.stdout:2691: finishing write tid 1 to smithi12556470-3 2021-03-24T22:04:39.373 INFO:tasks.rados.rados.0.smithi125.stdout:2690: expect (ObjNum 919 snap 307 seq_num 919) 2021-03-24T22:04:39.384 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:39.438 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:39.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:39.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:04:39.490 INFO:tasks.rados.rados.0.smithi125.stdout:2691: finishing write tid 2 to smithi12556470-3 2021-03-24T22:04:39.490 INFO:tasks.rados.rados.0.smithi125.stdout:2691: finishing write tid 3 to smithi12556470-3 2021-03-24T22:04:39.490 INFO:tasks.rados.rados.0.smithi125.stdout:2691: finishing write tid 4 to smithi12556470-3 2021-03-24T22:04:39.491 INFO:tasks.rados.rados.0.smithi125.stdout:2691: finishing write tid 5 to smithi12556470-3 2021-03-24T22:04:39.491 INFO:tasks.rados.rados.0.smithi125.stdout:2691: finishing write tid 6 to smithi12556470-3 2021-03-24T22:04:39.491 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 3 v 318 (ObjNum 933 snap 311 seq_num 933) dirty exists 2021-03-24T22:04:39.491 INFO:tasks.rados.rados.0.smithi125.stdout:2691: left oid 3 (ObjNum 933 snap 311 seq_num 933) 2021-03-24T22:04:39.492 INFO:tasks.rados.rados.0.smithi125.stdout:2693: finishing rollback tid 0 to smithi12556470-27 2021-03-24T22:04:39.492 INFO:tasks.rados.rados.0.smithi125.stdout:2690: done (3 left) 2021-03-24T22:04:39.492 INFO:tasks.rados.rados.0.smithi125.stdout:2691: done (2 left) 2021-03-24T22:04:39.492 INFO:tasks.rados.rados.0.smithi125.stdout:2694: snap_create 2021-03-24T22:04:39.492 INFO:tasks.rados.rados.0.smithi125.stdout:2693: finishing rollback tid 1 to smithi12556470-27 2021-03-24T22:04:39.493 INFO:tasks.rados.rados.0.smithi125.stdout:2693: finishing rollback tid 2 to smithi12556470-27 2021-03-24T22:04:39.493 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 27 v 1422 (ObjNum 855 snap 278 seq_num 855) dirty exists 2021-03-24T22:04:39.493 INFO:tasks.rados.rados.0.smithi125.stdout:2692: finishing copy_from to smithi12556470-14 2021-03-24T22:04:39.493 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 14 v 887 (ObjNum 727 snap 233 seq_num 727) dirty exists 2021-03-24T22:04:39.494 INFO:tasks.rados.rados.0.smithi125.stdout:2692: finishing copy_from racing read to smithi12556470-14 2021-03-24T22:04:39.544 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:39.598 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:39.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:39.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:04:39.630 INFO:tasks.rados.rados.0.smithi125.stdout:2692: done (2 left) 2021-03-24T22:04:39.631 INFO:tasks.rados.rados.0.smithi125.stdout:2693: done (1 left) 2021-03-24T22:04:39.631 INFO:tasks.rados.rados.0.smithi125.stdout:2694: done (0 left) 2021-03-24T22:04:39.631 INFO:tasks.rados.rados.0.smithi125.stdout:2695: rollback oid 12 current snap is 312 2021-03-24T22:04:39.631 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 12 to 295 2021-03-24T22:04:39.632 INFO:tasks.rados.rados.0.smithi125.stdout:2696: copy_from oid 1 from oid 32 current snap is 312 2021-03-24T22:04:39.632 INFO:tasks.rados.rados.0.smithi125.stdout:2697: read oid 19 snap -1 2021-03-24T22:04:39.632 INFO:tasks.rados.rados.0.smithi125.stdout:2697: expect (ObjNum 736 snap 235 seq_num 736) 2021-03-24T22:04:39.632 INFO:tasks.rados.rados.0.smithi125.stdout:2698: rollback oid 21 current snap is 312 2021-03-24T22:04:39.633 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 21 to 238 2021-03-24T22:04:39.633 INFO:tasks.rados.rados.0.smithi125.stdout:2699: delete oid 34 current snap is 312 2021-03-24T22:04:39.640 INFO:tasks.rados.rados.0.smithi125.stdout:2696: finishing copy_from to smithi12556470-1 2021-03-24T22:04:39.640 INFO:tasks.rados.rados.0.smithi125.stdout:2696: got expected ENOENT (src dne) 2021-03-24T22:04:39.641 INFO:tasks.rados.rados.0.smithi125.stdout:2696: finishing copy_from racing read to smithi12556470-1 2021-03-24T22:04:39.641 INFO:tasks.rados.rados.0.smithi125.stdout:2698: finishing rollback tid 0 to smithi12556470-21 2021-03-24T22:04:39.641 INFO:tasks.rados.rados.0.smithi125.stdout:2698: finishing rollback tid 1 to smithi12556470-21 2021-03-24T22:04:39.642 INFO:tasks.rados.rados.0.smithi125.stdout:2698: finishing rollback tid 2 to smithi12556470-21 2021-03-24T22:04:39.642 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 21 v 1414 (ObjNum 590 snap 177 seq_num 590) dirty exists 2021-03-24T22:04:39.645 INFO:tasks.rados.rados.0.smithi125.stdout:2697: expect (ObjNum 736 snap 235 seq_num 736) 2021-03-24T22:04:39.704 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:39.754 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:39.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:39.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:39.793 INFO:tasks.rados.rados.0.smithi125.stdout:2696: done (4 left) 2021-03-24T22:04:39.793 INFO:tasks.rados.rados.0.smithi125.stdout:2697: done (3 left) 2021-03-24T22:04:39.793 INFO:tasks.rados.rados.0.smithi125.stdout:2698: done (2 left) 2021-03-24T22:04:39.793 INFO:tasks.rados.rados.0.smithi125.stdout:2699: done (1 left) 2021-03-24T22:04:39.794 INFO:tasks.rados.rados.0.smithi125.stdout:2700: rollback oid 20 current snap is 312 2021-03-24T22:04:39.794 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 20 to 301 2021-03-24T22:04:39.794 INFO:tasks.rados.rados.0.smithi125.stdout:2695: finishing rollback tid 1 to smithi12556470-12 2021-03-24T22:04:39.794 INFO:tasks.rados.rados.0.smithi125.stdout:2695: finishing rollback tid 2 to smithi12556470-12 2021-03-24T22:04:39.795 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 12 v 634 (ObjNum 752 snap 241 seq_num 752) dirty exists 2021-03-24T22:04:39.795 INFO:tasks.rados.rados.0.smithi125.stdout:2695: done (1 left) 2021-03-24T22:04:39.795 INFO:tasks.rados.rados.0.smithi125.stdout:2701: delete oid 44 current snap is 312 2021-03-24T22:04:39.796 INFO:tasks.rados.rados.0.smithi125.stdout:2700: finishing rollback tid 0 to smithi12556470-20 2021-03-24T22:04:39.796 INFO:tasks.rados.rados.0.smithi125.stdout:2701: done (1 left) 2021-03-24T22:04:39.796 INFO:tasks.rados.rados.0.smithi125.stdout:2702: write (excl) oid 43 current snap is 312 2021-03-24T22:04:39.797 INFO:tasks.rados.rados.0.smithi125.stdout:2700: finishing rollback tid 1 to smithi12556470-20 2021-03-24T22:04:39.797 INFO:tasks.rados.rados.0.smithi125.stdout:2702: seq_num 936 ranges {531352=744483,2005734=779325,3487333=244019} 2021-03-24T22:04:39.804 INFO:tasks.rados.rados.0.smithi125.stdout:2702: writing smithi12556470-43 from 531352 to 1275835 tid 1 2021-03-24T22:04:39.813 INFO:tasks.rados.rados.0.smithi125.stdout:2702: writing smithi12556470-43 from 2005734 to 2785059 tid 2 2021-03-24T22:04:39.818 INFO:tasks.rados.rados.0.smithi125.stdout:2702: writing smithi12556470-43 from 3487333 to 3731352 tid 3 2021-03-24T22:04:39.818 INFO:tasks.rados.rados.0.smithi125.stdout:2703: write (excl) oid 44 current snap is 312 2021-03-24T22:04:39.819 INFO:tasks.rados.rados.0.smithi125.stdout:2700: finishing rollback tid 2 to smithi12556470-20 2021-03-24T22:04:39.819 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 20 v 815 (ObjNum 873 snap 283 seq_num 873) dirty exists 2021-03-24T22:04:39.819 INFO:tasks.rados.rados.0.smithi125.stdout:2703: seq_num 937 ranges {548159=419732,1664216=522983,2714756=614229} 2021-03-24T22:04:39.824 INFO:tasks.rados.rados.0.smithi125.stdout:2703: writing smithi12556470-44 from 548159 to 967891 tid 1 2021-03-24T22:04:39.830 INFO:tasks.rados.rados.0.smithi125.stdout:2703: writing smithi12556470-44 from 1664216 to 2187199 tid 2 2021-03-24T22:04:39.837 INFO:tasks.rados.rados.0.smithi125.stdout:2703: writing smithi12556470-44 from 2714756 to 3328985 tid 3 2021-03-24T22:04:39.838 INFO:tasks.rados.rados.0.smithi125.stdout:2700: done (2 left) 2021-03-24T22:04:39.838 INFO:tasks.rados.rados.0.smithi125.stdout:2704: snap_remove snap 308 2021-03-24T22:04:39.861 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:39.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:39 smithi142 conmon[20125]: cluster 2021-03-24T22 2021-03-24T22:04:39.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:39 smithi142 conmon[20125]: :04:39.179098+0000 osd.0 (osd.0) 140 : cluster [DBG] 3.1 scrub ok 2021-03-24T22:04:39.874 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:39 smithi125 conmon[19936]: cluster 2021-03-24T22 2021-03-24T22:04:39.874 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:39 smithi125 conmon[19936]: :04:39.179098+0000 osd.0 (osd.0) 140 : cluster [DBG] 3.1 scrub ok 2021-03-24T22:04:39.875 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:39 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:04:39.875 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:39 smithi125 conmon[24927]: 24T22:04:39.179098+0000 osd.0 (osd.0) 140 : cluster [DBG] 3.1 scrub ok 2021-03-24T22:04:39.875 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:39 smithi125 conmon[24927]: 2021-03-24T22:04:39.906 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:39.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:39.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:40.013 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:40.067 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:40.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:40.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:04:40.105 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.6 dump_ops_in_flight 2021-03-24T22:04:40.173 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:40.223 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:40.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:40.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:40.331 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:40.378 INFO:tasks.rados.rados.0.smithi125.stdout:2704: done (2 left) 2021-03-24T22:04:40.379 INFO:tasks.rados.rados.0.smithi125.stdout:2705: copy_from oid 12 from oid 37 current snap is 312 2021-03-24T22:04:40.379 INFO:tasks.rados.rados.0.smithi125.stdout:2702: finishing write tid 1 to smithi12556470-43 2021-03-24T22:04:40.379 INFO:tasks.rados.rados.0.smithi125.stdout:2702: finishing write tid 2 to smithi12556470-43 2021-03-24T22:04:40.379 INFO:tasks.rados.rados.0.smithi125.stdout:2702: finishing write tid 3 to smithi12556470-43 2021-03-24T22:04:40.379 INFO:tasks.rados.rados.0.smithi125.stdout:2702: finishing write tid 4 to smithi12556470-43 2021-03-24T22:04:40.380 INFO:tasks.rados.rados.0.smithi125.stdout:2702: finishing write tid 5 to smithi12556470-43 2021-03-24T22:04:40.380 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 43 v 1637 (ObjNum 936 snap 312 seq_num 936) dirty exists 2021-03-24T22:04:40.380 INFO:tasks.rados.rados.0.smithi125.stdout:2702: left oid 43 (ObjNum 936 snap 312 seq_num 936) 2021-03-24T22:04:40.380 INFO:tasks.rados.rados.0.smithi125.stdout:2703: finishing write tid 1 to smithi12556470-44 2021-03-24T22:04:40.381 INFO:tasks.rados.rados.0.smithi125.stdout:2703: finishing write tid 2 to smithi12556470-44 2021-03-24T22:04:40.381 INFO:tasks.rados.rados.0.smithi125.stdout:2703: finishing write tid 3 to smithi12556470-44 2021-03-24T22:04:40.381 INFO:tasks.rados.rados.0.smithi125.stdout:2703: finishing write tid 4 to smithi12556470-44 2021-03-24T22:04:40.381 INFO:tasks.rados.rados.0.smithi125.stdout:2703: finishing write tid 5 to smithi12556470-44 2021-03-24T22:04:40.381 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 44 v 1419 (ObjNum 937 snap 312 seq_num 937) dirty exists 2021-03-24T22:04:40.382 INFO:tasks.rados.rados.0.smithi125.stdout:2703: left oid 44 (ObjNum 937 snap 312 seq_num 937) 2021-03-24T22:04:40.383 INFO:tasks.rados.rados.0.smithi125.stdout:2702: done (2 left) 2021-03-24T22:04:40.383 INFO:tasks.rados.rados.0.smithi125.stdout:2703: done (1 left) 2021-03-24T22:04:40.383 INFO:tasks.rados.rados.0.smithi125.stdout:2706: copy_from oid 43 from oid 42 current snap is 312 2021-03-24T22:04:40.383 INFO:tasks.rados.rados.0.smithi125.stdout:2707: snap_remove snap 264 2021-03-24T22:04:40.392 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:40.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:40.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:40.426 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T22:04:40.501 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:40.526 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-24T22:04:40.554 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:40.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:40.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:40.665 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:40.709 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:40.712 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:40 smithi125 conmon[24927]: cluster 2021-03-24T22:04:38 2021-03-24T22:04:40.712 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:40 smithi125 conmon[24927]: .959764+0000 mgr.y (mgr.14144) 629 : cluster [DBG] pgmap v1225: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 107 active+clean; 638 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 760 KiB/s wr, 3 op/s; 152/1875 objects degraded (8.107%); 40/1875 objects misplaced (2.133%) 2021-03-24T22:04:40.712 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:40 smithi125 conmon[24927]: cluster 2021-03-24T22:04: 2021-03-24T22:04:40.712 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:40 smithi125 conmon[24927]: 39.625673+0000 mon.a (mon.0) 1855 : cluster 2021-03-24T22:04:40.713 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:40 smithi125 conmon[24927]: [DBG] osdmap e709: 8 total, 7 up, 8 in 2021-03-24T22:04:40.713 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:40 smithi125 conmon[24927]: cluster 2021-03-24T22:04:40.713 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:40 smithi125 conmon[24927]: 2021-03-24T22:04:40.360313+0000 mon.a (mon. 2021-03-24T22:04:40.713 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:40 smithi125 conmon[24927]: 0) 1856 : cluster [WRN] Health check update: Degraded data redundancy: 152/1875 objects degraded (8.107%), 19 pgs degraded (PG_DEGRADED) 2021-03-24T22:04:40.713 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:40 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T22:04:40.714 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:40 smithi125 conmon[24927]: -24T22:04:40.372660+0000 mon.a (mon 2021-03-24T22:04:40.714 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:40 smithi125 conmon[24927]: .0) 1857 : cluster [DBG] osdmap e710: 8 total, 7 up, 8 in 2021-03-24T22:04:40.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:40.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:40.716 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:40 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T22:04:40.716 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:40 smithi125 conmon[19936]: 22:04:38.959764+0000 mgr.y (mgr.14144) 629 : cluster 2021-03-24T22:04:40.716 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:40 smithi125 conmon[19936]: [DBG] pgmap v1225: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 107 active+clean; 638 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 760 KiB/s wr, 3 op/s; 152/1875 objects degraded (8.107%); 40/1875 objects misplaced (2.133%) 2021-03-24T22:04:40.717 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:40 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:04:40.717 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:40 smithi125 conmon[19936]: 03-24T22:04:39.625673 2021-03-24T22:04:40.717 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:40 smithi125 conmon[19936]: +0000 mon.a (mon. 2021-03-24T22:04:40.717 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:40 smithi125 conmon[19936]: 0) 1855 : cluster 2021-03-24T22:04:40.718 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:40 smithi125 conmon[19936]: [DBG] osdmap e709: 8 total, 7 up, 8 in 2021-03-24T22:04:40.718 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:40 smithi125 conmon[19936]: cluster 2021-03-24T22:04:40.718 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:40 smithi125 conmon[19936]: 2021-03-24T 2021-03-24T22:04:40.718 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:40 smithi125 conmon[19936]: 22:04:40 2021-03-24T22:04:40.718 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:40 smithi125 conmon[19936]: .360313+0000 mon.a 2021-03-24T22:04:40.719 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:40 smithi125 conmon[19936]: (mon.0 2021-03-24T22:04:40.719 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:40 smithi125 conmon[19936]: ) 1856 : cluster 2021-03-24T22:04:40.719 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:40 smithi125 conmon[19936]: [WRN] Health check update: Degraded data redundancy: 152/1875 objects degraded (8.107%), 19 pgs degraded (PG_DEGRADED) 2021-03-24T22:04:40.719 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:40 smithi125 conmon[19936]: cluster 2021-03-24T22:04:40.720 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:40 smithi125 conmon[19936]: 2021- 2021-03-24T22:04:40.720 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:40 smithi125 conmon[19936]: 03-24T22:04:40 2021-03-24T22:04:40.720 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:40 smithi125 conmon[19936]: .372660+0000 mon.a ( 2021-03-24T22:04:40.720 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:40 smithi125 conmon[19936]: mon.0) 1857 2021-03-24T22:04:40.720 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:40 smithi125 conmon[19936]: : cluster [DBG] 2021-03-24T22:04:40.721 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:40 smithi125 conmon[19936]: osdmap e710: 8 total, 7 up, 8 in 2021-03-24T22:04:40.806 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:04:40.817 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:40.872 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:40.875 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:40 smithi142 conmon[20125]: cluster 2021-03-24T22:04:38.959764+0000 mgr.y (mgr.14144) 629 : 2021-03-24T22:04:40.876 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:40 smithi142 conmon[20125]: cluster [DBG] pgmap v1225: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 107 active+clean; 638 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 760 KiB/s wr, 3 op/s; 152/1875 objects degraded (8.107%); 40/1875 objects misplaced (2.133%) 2021-03-24T22:04:40.876 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:40 smithi142 conmon[20125]: cluster 2021-03-24T22:04:39.625673+0000 mon.a (mon 2021-03-24T22:04:40.876 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:40 smithi142 conmon[20125]: .0) 1855 : cluster [DBG] osdmap e709: 8 total, 7 up, 8 in 2021-03-24T22:04:40.876 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:40 smithi142 conmon[20125]: cluster 2021-03-24T22:04:40.877 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:40 smithi142 conmon[20125]: 2021-03-24T22:04:40.360313+0000 2021-03-24T22:04:40.877 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:40 smithi142 conmon[20125]: mon.a (mon.0) 1856 : cluster [WRN] Health check update: Degraded data redundancy: 152/1875 objects degraded (8.107%), 19 pgs degraded (PG_DEGRADED) 2021-03-24T22:04:40.877 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:40 smithi142 conmon[20125]: 2021-03-24T22:04:40.877 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:40 smithi142 conmon[20125]: cluster 2021-03-24T22:04:40.878 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:40 smithi142 conmon[20125]: 2021-03-24T22:04:40.372660+0000 2021-03-24T22:04:40.878 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:40 smithi142 conmon[20125]: mon.a (mon.0) 1857 : cluster [DBG] osdmap e710: 8 total, 7 up, 8 in 2021-03-24T22:04:40.878 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:40 smithi142 conmon[20125]: 2021-03-24T22:04:40.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:40.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:04:40.981 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:41.033 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:41.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:41.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:41.139 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:41.195 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:41.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:41.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:41.301 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:41.364 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:41.366 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 1] out_osds: [] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 6, 7] 2021-03-24T22:04:41.367 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-24T22:04:41.367 INFO:tasks.thrashosds.thrasher:inject_pause on 3 2021-03-24T22:04:41.368 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2021-03-24T22:04:41.368 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2021-03-24T22:04:41.368 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.3 config set filestore_inject_stall 3 2021-03-24T22:04:41.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:41.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:41.375 INFO:tasks.rados.rados.0.smithi125.stdout:2707: done (2 left) 2021-03-24T22:04:41.375 INFO:tasks.rados.rados.0.smithi125.stdout:2708: delete oid 15 current snap is 312 2021-03-24T22:04:41.375 INFO:tasks.rados.rados.0.smithi125.stdout:2705: finishing copy_from to smithi12556470-12 2021-03-24T22:04:41.375 INFO:tasks.rados.rados.0.smithi125.stdout:2705: got expected ENOENT (src dne) 2021-03-24T22:04:41.376 INFO:tasks.rados.rados.0.smithi125.stdout:2705: finishing copy_from racing read to smithi12556470-12 2021-03-24T22:04:41.376 INFO:tasks.rados.rados.0.smithi125.stdout:2706: finishing copy_from to smithi12556470-43 2021-03-24T22:04:41.376 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 43 v 506 (ObjNum 736 snap 235 seq_num 736) dirty exists 2021-03-24T22:04:41.376 INFO:tasks.rados.rados.0.smithi125.stdout:2706: finishing copy_from racing read to smithi12556470-43 2021-03-24T22:04:41.386 INFO:tasks.rados.rados.0.smithi125.stdout:2705: done (2 left) 2021-03-24T22:04:41.387 INFO:tasks.rados.rados.0.smithi125.stdout:2706: done (1 left) 2021-03-24T22:04:41.387 INFO:tasks.rados.rados.0.smithi125.stdout:2708: done (0 left) 2021-03-24T22:04:41.387 INFO:tasks.rados.rados.0.smithi125.stdout:2709: snap_create 2021-03-24T22:04:41.394 INFO:teuthology.orchestra.run.smithi125.stderr:noscrub is set 2021-03-24T22:04:41.472 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:41.526 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:41.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:41.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:04:41.630 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:41.684 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:41.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:41.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:41.789 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:41.801 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:41 smithi125 conmon[24927]: cluster 2021-03-24T22:04:40.677547+0000 osd.6 (osd.6) 66 : cluster [DBG] 5.11 scrub ok 2021-03-24T22:04:41.802 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:41 smithi125 conmon[24927]: audit 2021-03-24T22:04:40.774704+0000 mon.a (mon.0) 1858 : audit [INF] from='client.? v1:172.21.15.125:0/107489337' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T22:04:41.802 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:41 smithi125 conmon[24927]: cluster 2021-03-24T22:04:41.171312+0000 osd.7 (osd.7) 118 : cluster [DBG] 7.15 scrub ok 2021-03-24T22:04:41.802 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:41 smithi125 conmon[24927]: cluster 2021-03-24T22:04:41.216138+0000 osd.0 (osd.0) 141 : cluster [DBG] 7.1b scrub ok 2021-03-24T22:04:41.802 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:41 smithi125 conmon[24927]: cluster 2021-03-24T22:04:41.259090+0000 osd.7 (osd.7) 119 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:04:41.803 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:41 smithi125 conmon[24927]: cluster 2021-03-24T22:04:41.262782+0000 osd.7 (osd.7) 120 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:04:41.803 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:41 smithi125 conmon[24927]: cluster 2021-03-24T22:04:41.365797+0000 mon.a (mon.0) 1859 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:04:41.803 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:41 smithi125 conmon[24927]: audit 2021-03-24T22:04:41.372390+0000 mon.a (mon.0) 1860 : audit [INF] from='client.? v1:172.21.15.125:0/107489337' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T22:04:41.803 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:41 smithi125 conmon[24927]: cluster 2021-03-24T22:04:41.372462+0000 mon.a (mon.0) 1861 : cluster [DBG] osdmap e711: 8 total, 7 up, 8 in 2021-03-24T22:04:41.804 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:41 smithi125 conmon[19936]: cluster 2021-03-24T22:04:40.677547+0000 osd.6 (osd.6) 66 : cluster [DBG] 5.11 scrub ok 2021-03-24T22:04:41.804 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:41 smithi125 conmon[19936]: audit 2021-03-24T22:04:40.774704+0000 mon.a (mon.0) 1858 : audit [INF] from='client.? v1:172.21.15.125:0/107489337' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T22:04:41.804 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:41 smithi125 conmon[19936]: cluster 2021-03-24T22:04:41.171312+0000 osd.7 (osd.7) 118 : cluster [DBG] 7.15 scrub ok 2021-03-24T22:04:41.804 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:41 smithi125 conmon[19936]: cluster 2021-03-24T22:04:41.216138+0000 osd.0 (osd.0) 141 : cluster [DBG] 7.1b scrub ok 2021-03-24T22:04:41.805 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:41 smithi125 conmon[19936]: cluster 2021-03-24T22:04:41.259090+0000 osd.7 (osd.7) 119 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:04:41.805 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:41 smithi125 conmon[19936]: cluster 2021-03-24T22:04:41.262782+0000 osd.7 (osd.7) 120 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:04:41.805 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:41 smithi125 conmon[19936]: cluster 2021-03-24T22:04:41.365797+0000 mon.a (mon.0) 1859 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:04:41.805 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:41 smithi125 conmon[19936]: audit 2021-03-24T22:04:41.372390+0000 mon.a (mon.0) 1860 : audit [INF] from='client.? v1:172.21.15.125:0/107489337' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T22:04:41.806 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:41 smithi125 conmon[19936]: cluster 2021-03-24T22:04:41.372462+0000 mon.a (mon.0) 1861 : cluster [DBG] osdmap e711: 8 total, 7 up, 8 in 2021-03-24T22:04:41.857 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:41.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:41.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:41.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:41 smithi142 conmon[20125]: cluster 2021-03-24T22 2021-03-24T22:04:41.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:41 smithi142 conmon[20125]: :04:40.677547+0000 osd.6 (osd.6) 66 : cluster [DBG] 5.11 scrub ok 2021-03-24T22:04:41.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:41 smithi142 conmon[20125]: audit 2021-03-24T22:04:41.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:41 smithi142 conmon[20125]: 2021-03-24T22:04: 2021-03-24T22:04:41.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:41 smithi142 conmon[20125]: 40.774704+0000 mon.a (mon.0 2021-03-24T22:04:41.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:41 smithi142 conmon[20125]: ) 1858 : audit [INF] from='client.? v1:172.21.15.125:0/107489337' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T22:04:41.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:41 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:04:41.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:41 smithi142 conmon[20125]: -24T22:04:41.171312+0000 osd.7 2021-03-24T22:04:41.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:41 smithi142 conmon[20125]: (osd.7) 118 : cluster 2021-03-24T22:04:41.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:41 smithi142 conmon[20125]: [DBG] 7.15 scrub ok 2021-03-24T22:04:41.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:41 smithi142 conmon[20125]: cluster 2021-03-24T22: 2021-03-24T22:04:41.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:41 smithi142 conmon[20125]: 04:41.216138+0000 osd.0 (osd. 2021-03-24T22:04:41.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:41 smithi142 conmon[20125]: 0) 141 : cluster [DBG] 7.1b scrub ok 2021-03-24T22:04:41.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:41 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:04:41.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:41 smithi142 conmon[20125]: 03-24T22:04:41. 2021-03-24T22:04:41.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:41 smithi142 conmon[20125]: 259090+0000 osd.7 (osd. 2021-03-24T22:04:41.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:41 smithi142 conmon[20125]: 7) 119 : cluster [DBG] 2021-03-24T22:04:41.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:41 smithi142 conmon[20125]: purged_snaps scrub starts 2021-03-24T22:04:41.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:41 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:04:41.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:41 smithi142 conmon[20125]: -03-24T22:04: 2021-03-24T22:04:41.872 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:41 smithi142 conmon[20125]: 41.262782+0000 osd.7 ( 2021-03-24T22:04:41.872 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:41 smithi142 conmon[20125]: osd.7) 120 : cluster 2021-03-24T22:04:41.872 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:41 smithi142 conmon[20125]: [DBG] purged_snaps scrub ok 2021-03-24T22:04:41.872 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:41 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:04:41.873 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:41 smithi142 conmon[20125]: -24T22:04:41.365797+0000 2021-03-24T22:04:41.873 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:41 smithi142 conmon[20125]: mon.a (mon.0) 1859 : cluster [WRN] 2021-03-24T22:04:41.873 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:41 smithi142 conmon[20125]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:04:41.873 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:41 smithi142 conmon[20125]: audit 2021- 2021-03-24T22:04:41.874 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:41 smithi142 conmon[20125]: 03-24T22:04:41.372390+0000 mon.a (mon.0) 1860 : audit [INF] 2021-03-24T22:04:41.874 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:41 smithi142 conmon[20125]: from='client.? v1:172.21.15.125:0/107489337' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T22:04:41.874 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:41 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:04:41.874 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:41 smithi142 conmon[20125]: T22:04:41.372462+0000 mon.a 2021-03-24T22:04:41.874 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:41 smithi142 conmon[20125]: (mon.0) 1861 : cluster [DBG] 2021-03-24T22:04:41.875 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:41 smithi142 conmon[20125]: osdmap e711: 8 total, 7 up, 8 in 2021-03-24T22:04:41.961 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:42.022 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:42.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:42.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:42.130 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:42.191 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:42.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:42.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:42.297 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:42.351 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:42.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:42.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:42.383 INFO:tasks.rados.rados.0.smithi125.stdout:2709: done (0 left) 2021-03-24T22:04:42.384 INFO:tasks.rados.rados.0.smithi125.stdout:2710: rollback oid 6 current snap is 313 2021-03-24T22:04:42.384 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 6 to 269 2021-03-24T22:04:42.384 INFO:tasks.rados.rados.0.smithi125.stdout:2711: snap_create 2021-03-24T22:04:42.392 INFO:tasks.rados.rados.0.smithi125.stdout:2710: finishing rollback tid 0 to smithi12556470-6 2021-03-24T22:04:42.392 INFO:tasks.rados.rados.0.smithi125.stdout:2710: finishing rollback tid 1 to smithi12556470-6 2021-03-24T22:04:42.393 INFO:tasks.rados.rados.0.smithi125.stdout:2710: finishing rollback tid 2 to smithi12556470-6 2021-03-24T22:04:42.393 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 6 v 642 (ObjNum 801 snap 259 seq_num 801) dirty exists 2021-03-24T22:04:42.458 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:42.499 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:42.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:42.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:04:42.515 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2021-03-24T22:04:42.516 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T22:04:42.606 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:42.658 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:42.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:42.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:04:42.763 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:42.776 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:42 smithi142 conmon[20125]: cluster 2021-03-24T22:04:40.960624+0000 mgr.y (mgr.14144) 630 : cluster [DBG] pgmap v1228: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 107 active+clean; 650 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 152/1893 objects degraded (8.030%); 40/1893 objects misplaced (2.113%) 2021-03-24T22:04:42.777 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:42 smithi142 conmon[20125]: cluster 2021-03-24T22:04:42.378021+0000 mon.a (mon.0) 1862 : cluster [DBG] osdmap e712: 8 total, 7 up, 8 in 2021-03-24T22:04:42.818 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:42.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:42.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:42.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:42 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:04:42.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:42 smithi125 conmon[19936]: T22:04:40.960624+0000 mgr.y (mgr.14144) 630 : cluster [DBG] pgmap v1228: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 107 active+clean; 650 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 152/1893 objects degraded (8.030%); 40/1893 objects misplaced (2.113%) 2021-03-24T22:04:42.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:42 smithi125 conmon[19936]: cluster 2021-03-24T22:04:42 2021-03-24T22:04:42.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:42 smithi125 conmon[19936]: .378021+0000 mon.a (mon.0) 1862 : 2021-03-24T22:04:42.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:42 smithi125 conmon[19936]: cluster [DBG] osdmap e712: 8 total, 7 up, 8 in 2021-03-24T22:04:42.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:42 smithi125 conmon[24927]: cluster 2021-03-24T22 2021-03-24T22:04:42.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:42 smithi125 conmon[24927]: :04:40.960624+0000 mgr.y (mgr.14144) 630 : cluster [DBG] pgmap v1228: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 107 active+clean; 650 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 152/1893 objects degraded (8.030%); 40/1893 objects misplaced (2.113%) 2021-03-24T22:04:42.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:42 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T22:04:42.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:42 smithi125 conmon[24927]: T22:04:42.378021+0000 mon.a ( 2021-03-24T22:04:42.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:42 smithi125 conmon[24927]: mon.0) 1862 : cluster [DBG] osdmap e712: 8 total, 7 up, 8 in 2021-03-24T22:04:42.924 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:42.983 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:42.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:42.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:43.090 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:43.145 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:43.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:43.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:43.250 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:43.309 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:43.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:43.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:04:43.379 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:04:43.395 INFO:tasks.rados.rados.0.smithi125.stdout:2710: done (1 left) 2021-03-24T22:04:43.396 INFO:tasks.rados.rados.0.smithi125.stdout:2711: done (0 left) 2021-03-24T22:04:43.396 INFO:tasks.rados.rados.0.smithi125.stdout:2712: snap_remove snap 309 2021-03-24T22:04:43.417 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:43.517 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:43.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:43.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:04:43.612 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.6 dump_blocked_ops 2021-03-24T22:04:43.623 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:43.675 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:43.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:43.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:04:43.781 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:43.836 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:43.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:43.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:04:43.907 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T22:04:43.942 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:43.975 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T22:04:43.987 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:43.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:43.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:44.000 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T22:04:44.019 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T22:04:44.040 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T22:04:44.063 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T22:04:44.088 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T22:04:44.091 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:44.120 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T22:04:44.121 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-24T22:04:44.122 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-24T22:04:44.122 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-24T22:04:44.123 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-24T22:04:44.123 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-24T22:04:44.123 INFO:teuthology.orchestra.run.smithi125.stderr:Error ENXIO: problem getting command descriptions from osd.5 2021-03-24T22:04:44.123 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-24T22:04:44.124 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-24T22:04:44.144 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:44.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:44.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:04:44.251 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:44.305 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:44.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:44.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:44.413 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:44.414 INFO:tasks.rados.rados.0.smithi125.stdout:2712: done (0 left) 2021-03-24T22:04:44.415 INFO:tasks.rados.rados.0.smithi125.stdout:2713: read oid 7 snap -1 2021-03-24T22:04:44.415 INFO:tasks.rados.rados.0.smithi125.stdout:2713: expect (ObjNum 727 snap 233 seq_num 727) 2021-03-24T22:04:44.415 INFO:tasks.rados.rados.0.smithi125.stdout:2714: rollback oid 29 current snap is 314 2021-03-24T22:04:44.416 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 29 to 238 2021-03-24T22:04:44.416 INFO:tasks.rados.rados.0.smithi125.stdout:2715: write oid 40 current snap is 314 2021-03-24T22:04:44.416 INFO:tasks.rados.rados.0.smithi125.stdout:2715: seq_num 939 ranges {581773=570230,1664827=526652,2653249=636773} 2021-03-24T22:04:44.416 INFO:tasks.rados.rados.0.smithi125.stdout:2715: writing smithi12556470-40 from 581773 to 1152003 tid 1 2021-03-24T22:04:44.426 INFO:tasks.rados.rados.0.smithi125.stdout:2715: writing smithi12556470-40 from 1664827 to 2191479 tid 2 2021-03-24T22:04:44.434 INFO:tasks.rados.rados.0.smithi125.stdout:2715: writing smithi12556470-40 from 2653249 to 3290022 tid 3 2021-03-24T22:04:44.434 INFO:tasks.rados.rados.0.smithi125.stdout:2716: write (excl) oid 33 current snap is 314 2021-03-24T22:04:44.434 INFO:tasks.rados.rados.0.smithi125.stdout:2716: seq_num 940 ranges {598580=645479,1723309=786663,3080672=448045} 2021-03-24T22:04:44.442 INFO:tasks.rados.rados.0.smithi125.stdout:2716: writing smithi12556470-33 from 598580 to 1244059 tid 1 2021-03-24T22:04:44.450 INFO:tasks.rados.rados.0.smithi125.stdout:2716: writing smithi12556470-33 from 1723309 to 2509972 tid 2 2021-03-24T22:04:44.459 INFO:tasks.rados.rados.0.smithi125.stdout:2716: writing smithi12556470-33 from 3080672 to 3528717 tid 3 2021-03-24T22:04:44.460 INFO:tasks.rados.rados.0.smithi125.stdout:2717: copy_from oid 42 from oid 43 current snap is 314 2021-03-24T22:04:44.460 INFO:tasks.rados.rados.0.smithi125.stdout:2718: read oid 50 snap -1 2021-03-24T22:04:44.460 INFO:tasks.rados.rados.0.smithi125.stdout:2718: expect (ObjNum 852 snap 278 seq_num 852) 2021-03-24T22:04:44.460 INFO:tasks.rados.rados.0.smithi125.stdout:2719: write (excl) oid 8 current snap is 314 2021-03-24T22:04:44.461 INFO:tasks.rados.rados.0.smithi125.stdout:2714: finishing rollback tid 0 to smithi12556470-29 2021-03-24T22:04:44.461 INFO:tasks.rados.rados.0.smithi125.stdout:2714: finishing rollback tid 1 to smithi12556470-29 2021-03-24T22:04:44.461 INFO:tasks.rados.rados.0.smithi125.stdout:2714: finishing rollback tid 2 to smithi12556470-29 2021-03-24T22:04:44.461 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 29 v 465 (ObjNum 691 snap 218 seq_num 691) dirty exists 2021-03-24T22:04:44.462 INFO:tasks.rados.rados.0.smithi125.stdout:2713: expect (ObjNum 727 snap 233 seq_num 727) 2021-03-24T22:04:44.469 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:44.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:44.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:04:44.573 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:44.581 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:44 smithi125 conmon[19936]: cluster 2021-03-24T22:04:42 2021-03-24T22:04:44.581 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:44 smithi125 conmon[19936]: .961226+0000 mgr.y (mgr.14144) 631 : cluster [DBG] pgmap v1231: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 107 active+clean; 661 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 152/1902 objects degraded (7.992%); 40/1902 objects misplaced (2.103%) 2021-03-24T22:04:44.582 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:44 smithi125 conmon[19936]: cluster 2021-03-24T22:04:44.582 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:44 smithi125 conmon[19936]: 2021-03-24T22:04:43.381181+0000 mon.a (mon.0) 1863 : cluster 2021-03-24T22:04:44.582 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:44 smithi125 conmon[19936]: [DBG] osdmap e713: 8 total, 7 up, 8 in 2021-03-24T22:04:44.583 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:44 smithi125 conmon[19936]: audit 2021-03-24T22: 2021-03-24T22:04:44.583 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:44 smithi125 conmon[19936]: 04:43.949980+0000 mon.c (mon.2) 2021-03-24T22:04:44.583 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:44 smithi125 conmon[19936]: 104 : audit [DBG] from='client.? v1:172.21.15.125:0/1913484922' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:04:44.583 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:44 smithi125 conmon[24927]: cluster 2021-03-24T22:04:42. 2021-03-24T22:04:44.584 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:44 smithi125 conmon[24927]: 961226+0000 mgr.y (mgr.14144) 631 : cluster [DBG] pgmap v1231: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 107 active+clean; 661 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 152/1902 objects degraded (7.992%); 40/1902 objects misplaced (2.103%) 2021-03-24T22:04:44.584 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:44 smithi125 conmon[24927]: cluster 2021-03-24T22:04:44.584 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:44 smithi125 conmon[24927]: 2021-03-24T22:04:43.381181+0000 mon.a (mon.0) 1863 : cluster [DBG] 2021-03-24T22:04:44.584 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:44 smithi125 conmon[24927]: osdmap e713: 8 total, 7 up, 8 in 2021-03-24T22:04:44.585 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:44 smithi125 conmon[24927]: audit 2021- 2021-03-24T22:04:44.585 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:44 smithi125 conmon[24927]: 03-24T22:04:43.949980+0000 mon.c (mon.2) 104 : audit [DBG] 2021-03-24T22:04:44.585 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:44 smithi125 conmon[24927]: from='client.? v1:172.21.15.125:0/1913484922' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:04:44.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:44 smithi142 conmon[20125]: cluster 2021-03-24T22: 2021-03-24T22:04:44.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:44 smithi142 conmon[20125]: 04:42.961226+0000 mgr.y (mgr.14144) 631 : cluster [DBG] pgmap v1231: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 107 active+clean; 661 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 152/1902 objects degraded (7.992%); 40/1902 objects misplaced (2.103%) 2021-03-24T22:04:44.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:44 smithi142 conmon[20125]: cluster 2021-03-24T22:04:44.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:44 smithi142 conmon[20125]: 2021-03-24T22:04: 2021-03-24T22:04:44.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:44 smithi142 conmon[20125]: 43.381181+0000 mon.a (mon. 2021-03-24T22:04:44.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:44 smithi142 conmon[20125]: 0) 1863 : cluster [DBG] osdmap e713: 8 total, 7 up, 8 in 2021-03-24T22:04:44.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:44 smithi142 conmon[20125]: audit 2021-03-24T22:04:44.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:44 smithi142 conmon[20125]: 2021-03 2021-03-24T22:04:44.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:44 smithi142 conmon[20125]: -24T22:04:43.949980 2021-03-24T22:04:44.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:44 smithi142 conmon[20125]: +0000 mon.c (mon.2) 104 2021-03-24T22:04:44.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:44 smithi142 conmon[20125]: : audit [DBG] from='client.? v1:172.21.15.125:0/1913484922' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:04:44.619 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:44.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:44.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:44.666 INFO:tasks.rados.rados.0.smithi125.stdout:2715: finishing write tid 1 to smithi12556470-40 2021-03-24T22:04:44.667 INFO:tasks.rados.rados.0.smithi125.stdout:2715: finishing write tid 2 to smithi12556470-40 2021-03-24T22:04:44.667 INFO:tasks.rados.rados.0.smithi125.stdout:2715: finishing write tid 3 to smithi12556470-40 2021-03-24T22:04:44.668 INFO:tasks.rados.rados.0.smithi125.stdout:2715: finishing write tid 4 to smithi12556470-40 2021-03-24T22:04:44.668 INFO:tasks.rados.rados.0.smithi125.stdout:2715: finishing write tid 5 to smithi12556470-40 2021-03-24T22:04:44.669 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 40 v 1653 (ObjNum 939 snap 314 seq_num 939) dirty exists 2021-03-24T22:04:44.669 INFO:tasks.rados.rados.0.smithi125.stdout:2715: left oid 40 (ObjNum 939 snap 314 seq_num 939) 2021-03-24T22:04:44.670 INFO:tasks.rados.rados.0.smithi125.stdout:2719: seq_num 941 ranges {615387=720728,1781791=530321,3108095=707292} 2021-03-24T22:04:44.681 INFO:tasks.rados.rados.0.smithi125.stdout:2719: writing smithi12556470-8 from 615387 to 1336115 tid 1 2021-03-24T22:04:44.686 INFO:tasks.rados.rados.0.smithi125.stdout:2719: writing smithi12556470-8 from 1781791 to 2312112 tid 2 2021-03-24T22:04:44.696 INFO:tasks.rados.rados.0.smithi125.stdout:2719: writing smithi12556470-8 from 3108095 to 3815387 tid 3 2021-03-24T22:04:44.696 INFO:tasks.rados.rados.0.smithi125.stdout:2713: done (6 left) 2021-03-24T22:04:44.697 INFO:tasks.rados.rados.0.smithi125.stdout:2714: done (5 left) 2021-03-24T22:04:44.697 INFO:tasks.rados.rados.0.smithi125.stdout:2715: done (4 left) 2021-03-24T22:04:44.697 INFO:tasks.rados.rados.0.smithi125.stdout:2720: read oid 38 snap 310 2021-03-24T22:04:44.698 INFO:tasks.rados.rados.0.smithi125.stdout:2720: expect (ObjNum 733 snap 235 seq_num 733) 2021-03-24T22:04:44.698 INFO:tasks.rados.rados.0.smithi125.stdout:2721: read oid 32 snap -1 2021-03-24T22:04:44.699 INFO:tasks.rados.rados.0.smithi125.stdout:2721: expect deleted 2021-03-24T22:04:44.699 INFO:tasks.rados.rados.0.smithi125.stdout:2722: delete oid 46 current snap is 314 2021-03-24T22:04:44.699 INFO:tasks.rados.rados.0.smithi125.stdout:2718: expect (ObjNum 852 snap 278 seq_num 852) 2021-03-24T22:04:44.725 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:44.786 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:44.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:44.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:44.804 INFO:tasks.rados.rados.0.smithi125.stdout:2716: finishing write tid 1 to smithi12556470-33 2021-03-24T22:04:44.804 INFO:tasks.rados.rados.0.smithi125.stdout:2716: finishing write tid 2 to smithi12556470-33 2021-03-24T22:04:44.805 INFO:tasks.rados.rados.0.smithi125.stdout:2716: finishing write tid 3 to smithi12556470-33 2021-03-24T22:04:44.805 INFO:tasks.rados.rados.0.smithi125.stdout:2716: finishing write tid 4 to smithi12556470-33 2021-03-24T22:04:44.805 INFO:tasks.rados.rados.0.smithi125.stdout:2716: finishing write tid 5 to smithi12556470-33 2021-03-24T22:04:44.806 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 33 v 1044 (ObjNum 940 snap 314 seq_num 940) dirty exists 2021-03-24T22:04:44.806 INFO:tasks.rados.rados.0.smithi125.stdout:2716: left oid 33 (ObjNum 940 snap 314 seq_num 940) 2021-03-24T22:04:44.807 INFO:tasks.rados.rados.0.smithi125.stdout:2717: finishing copy_from to smithi12556470-42 2021-03-24T22:04:44.807 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 42 v 506 (ObjNum 736 snap 235 seq_num 736) dirty exists 2021-03-24T22:04:44.808 INFO:tasks.rados.rados.0.smithi125.stdout:2717: finishing copy_from racing read to smithi12556470-42 2021-03-24T22:04:44.808 INFO:tasks.rados.rados.0.smithi125.stdout:2716: done (6 left) 2021-03-24T22:04:44.808 INFO:tasks.rados.rados.0.smithi125.stdout:2717: done (5 left) 2021-03-24T22:04:44.809 INFO:tasks.rados.rados.0.smithi125.stdout:2718: done (4 left) 2021-03-24T22:04:44.809 INFO:tasks.rados.rados.0.smithi125.stdout:2722: done (3 left) 2021-03-24T22:04:44.810 INFO:tasks.rados.rados.0.smithi125.stdout:2723: copy_from oid 10 from oid 27 current snap is 314 2021-03-24T22:04:44.810 INFO:tasks.rados.rados.0.smithi125.stdout:2719: finishing write tid 1 to smithi12556470-8 2021-03-24T22:04:44.811 INFO:tasks.rados.rados.0.smithi125.stdout:2719: finishing write tid 2 to smithi12556470-8 2021-03-24T22:04:44.811 INFO:tasks.rados.rados.0.smithi125.stdout:2719: finishing write tid 3 to smithi12556470-8 2021-03-24T22:04:44.811 INFO:tasks.rados.rados.0.smithi125.stdout:2719: finishing write tid 4 to smithi12556470-8 2021-03-24T22:04:44.812 INFO:tasks.rados.rados.0.smithi125.stdout:2719: finishing write tid 5 to smithi12556470-8 2021-03-24T22:04:44.812 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 8 v 559 (ObjNum 941 snap 314 seq_num 941) dirty exists 2021-03-24T22:04:44.813 INFO:tasks.rados.rados.0.smithi125.stdout:2719: left oid 8 (ObjNum 941 snap 314 seq_num 941) 2021-03-24T22:04:44.813 INFO:tasks.rados.rados.0.smithi125.stdout:2720: expect (ObjNum 733 snap 235 seq_num 733) 2021-03-24T22:04:44.879 INFO:tasks.rados.rados.0.smithi125.stdout:2723: finishing copy_from to smithi12556470-10 2021-03-24T22:04:44.880 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 10 v 1422 (ObjNum 855 snap 278 seq_num 855) dirty exists 2021-03-24T22:04:44.880 INFO:tasks.rados.rados.0.smithi125.stdout:2723: finishing copy_from racing read to smithi12556470-10 2021-03-24T22:04:44.880 INFO:tasks.rados.rados.0.smithi125.stdout:2719: done (3 left) 2021-03-24T22:04:44.881 INFO:tasks.rados.rados.0.smithi125.stdout:2720: done (2 left) 2021-03-24T22:04:44.881 INFO:tasks.rados.rados.0.smithi125.stdout:2721: done (1 left) 2021-03-24T22:04:44.881 INFO:tasks.rados.rados.0.smithi125.stdout:2723: done (0 left) 2021-03-24T22:04:44.881 INFO:tasks.rados.rados.0.smithi125.stdout:2724: delete oid 27 current snap is 314 2021-03-24T22:04:44.883 INFO:tasks.rados.rados.0.smithi125.stdout:2724: done (0 left) 2021-03-24T22:04:44.883 INFO:tasks.rados.rados.0.smithi125.stdout:2725: write (excl) oid 36 current snap is 314 2021-03-24T22:04:44.884 INFO:tasks.rados.rados.0.smithi125.stdout:2725: seq_num 944 ranges {665808=662828,1957237=677648,3074011=609486} 2021-03-24T22:04:44.892 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:44.898 INFO:tasks.rados.rados.0.smithi125.stdout:2725: writing smithi12556470-36 from 665808 to 1328636 tid 1 2021-03-24T22:04:44.906 INFO:tasks.rados.rados.0.smithi125.stdout:2725: writing smithi12556470-36 from 1957237 to 2634885 tid 2 2021-03-24T22:04:44.913 INFO:tasks.rados.rados.0.smithi125.stdout:2725: writing smithi12556470-36 from 3074011 to 3683497 tid 3 2021-03-24T22:04:44.914 INFO:tasks.rados.rados.0.smithi125.stdout:2726: delete oid 30 current snap is 314 2021-03-24T22:04:44.914 INFO:tasks.rados.rados.0.smithi125.stdout:2725: finishing write tid 1 to smithi12556470-36 2021-03-24T22:04:44.914 INFO:tasks.rados.rados.0.smithi125.stdout:2725: finishing write tid 2 to smithi12556470-36 2021-03-24T22:04:44.916 INFO:tasks.rados.rados.0.smithi125.stdout:2726: done (1 left) 2021-03-24T22:04:44.917 INFO:tasks.rados.rados.0.smithi125.stdout:2727: read oid 39 snap -1 2021-03-24T22:04:44.917 INFO:tasks.rados.rados.0.smithi125.stdout:2727: expect deleted 2021-03-24T22:04:44.917 INFO:tasks.rados.rados.0.smithi125.stdout:2728: write oid 37 current snap is 314 2021-03-24T22:04:44.917 INFO:tasks.rados.rados.0.smithi125.stdout:2728: seq_num 946 ranges {699422=413326,1557848=681317,3012504=748383} 2021-03-24T22:04:44.928 INFO:tasks.rados.rados.0.smithi125.stdout:2728: writing smithi12556470-37 from 699422 to 1112748 tid 1 2021-03-24T22:04:44.936 INFO:tasks.rados.rados.0.smithi125.stdout:2728: writing smithi12556470-37 from 1557848 to 2239165 tid 2 2021-03-24T22:04:44.943 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:44.945 INFO:tasks.rados.rados.0.smithi125.stdout:2728: writing smithi12556470-37 from 3012504 to 3760887 tid 3 2021-03-24T22:04:44.945 INFO:tasks.rados.rados.0.smithi125.stdout:2729: write (excl) oid 6 current snap is 314 2021-03-24T22:04:44.946 INFO:tasks.rados.rados.0.smithi125.stdout:2729: seq_num 947 ranges {716229=488575,1616330=541328,2756280=559655} 2021-03-24T22:04:44.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:44.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:44.952 INFO:tasks.rados.rados.0.smithi125.stdout:2729: writing smithi12556470-6 from 716229 to 1204804 tid 1 2021-03-24T22:04:44.958 INFO:tasks.rados.rados.0.smithi125.stdout:2729: writing smithi12556470-6 from 1616330 to 2157658 tid 2 2021-03-24T22:04:44.966 INFO:tasks.rados.rados.0.smithi125.stdout:2729: writing smithi12556470-6 from 2756280 to 3315935 tid 3 2021-03-24T22:04:44.966 INFO:tasks.rados.rados.0.smithi125.stdout:2730: read oid 7 snap 301 2021-03-24T22:04:44.966 INFO:tasks.rados.rados.0.smithi125.stdout:2730: expect (ObjNum 727 snap 233 seq_num 727) 2021-03-24T22:04:44.966 INFO:tasks.rados.rados.0.smithi125.stdout:2731: rollback oid 20 current snap is 314 2021-03-24T22:04:44.967 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 20 to 252 2021-03-24T22:04:44.967 INFO:tasks.rados.rados.0.smithi125.stdout:2725: finishing write tid 3 to smithi12556470-36 2021-03-24T22:04:44.967 INFO:tasks.rados.rados.0.smithi125.stdout:2725: finishing write tid 4 to smithi12556470-36 2021-03-24T22:04:44.968 INFO:tasks.rados.rados.0.smithi125.stdout:2725: finishing write tid 5 to smithi12556470-36 2021-03-24T22:04:44.968 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 36 v 1040 (ObjNum 944 snap 314 seq_num 944) dirty exists 2021-03-24T22:04:44.968 INFO:tasks.rados.rados.0.smithi125.stdout:2725: left oid 36 (ObjNum 944 snap 314 seq_num 944) 2021-03-24T22:04:44.968 INFO:tasks.rados.rados.0.smithi125.stdout:2728: finishing write tid 1 to smithi12556470-37 2021-03-24T22:04:44.969 INFO:tasks.rados.rados.0.smithi125.stdout:2728: finishing write tid 2 to smithi12556470-37 2021-03-24T22:04:44.969 INFO:tasks.rados.rados.0.smithi125.stdout:2725: done (5 left) 2021-03-24T22:04:44.969 INFO:tasks.rados.rados.0.smithi125.stdout:2727: done (4 left) 2021-03-24T22:04:44.970 INFO:tasks.rados.rados.0.smithi125.stdout:2732: rollback oid 49 current snap is 314 2021-03-24T22:04:44.970 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 49 to 307 2021-03-24T22:04:44.970 INFO:tasks.rados.rados.0.smithi125.stdout:2728: finishing write tid 3 to smithi12556470-37 2021-03-24T22:04:44.970 INFO:tasks.rados.rados.0.smithi125.stdout:2728: finishing write tid 4 to smithi12556470-37 2021-03-24T22:04:44.970 INFO:tasks.rados.rados.0.smithi125.stdout:2728: finishing write tid 5 to smithi12556470-37 2021-03-24T22:04:44.971 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 37 v 736 (ObjNum 946 snap 314 seq_num 946) dirty exists 2021-03-24T22:04:44.971 INFO:tasks.rados.rados.0.smithi125.stdout:2728: left oid 37 (ObjNum 946 snap 314 seq_num 946) 2021-03-24T22:04:44.971 INFO:tasks.rados.rados.0.smithi125.stdout:2729: finishing write tid 1 to smithi12556470-6 2021-03-24T22:04:44.971 INFO:tasks.rados.rados.0.smithi125.stdout:2729: finishing write tid 2 to smithi12556470-6 2021-03-24T22:04:44.972 INFO:tasks.rados.rados.0.smithi125.stdout:2728: done (4 left) 2021-03-24T22:04:44.972 INFO:tasks.rados.rados.0.smithi125.stdout:2733: write (excl) oid 29 current snap is 314 2021-03-24T22:04:44.972 INFO:tasks.rados.rados.0.smithi125.stdout:2733: seq_num 948 ranges {733036=563824,2074812=684986,3183703=749333} 2021-03-24T22:04:44.972 INFO:tasks.rados.rados.0.smithi125.stdout:2733: writing smithi12556470-29 from 733036 to 1296860 tid 1 2021-03-24T22:04:44.981 INFO:tasks.rados.rados.0.smithi125.stdout:2733: writing smithi12556470-29 from 2074812 to 2759798 tid 2 2021-03-24T22:04:44.987 INFO:tasks.rados.rados.0.smithi125.stdout:2733: writing smithi12556470-29 from 3183703 to 3933036 tid 3 2021-03-24T22:04:44.988 INFO:tasks.rados.rados.0.smithi125.stdout:2734: write (excl) oid 25 current snap is 314 2021-03-24T22:04:44.988 INFO:tasks.rados.rados.0.smithi125.stdout:2734: seq_num 949 ranges {749843=639073,2133294=544997,3211126=582199} 2021-03-24T22:04:44.995 INFO:tasks.rados.rados.0.smithi125.stdout:2734: writing smithi12556470-25 from 749843 to 1388916 tid 1 2021-03-24T22:04:45.003 INFO:tasks.rados.rados.0.smithi125.stdout:2734: writing smithi12556470-25 from 2133294 to 2678291 tid 2 2021-03-24T22:04:45.010 INFO:tasks.rados.rados.0.smithi125.stdout:2734: writing smithi12556470-25 from 3211126 to 3793325 tid 3 2021-03-24T22:04:45.010 INFO:tasks.rados.rados.0.smithi125.stdout:2735: delete oid 37 current snap is 314 2021-03-24T22:04:45.011 INFO:tasks.rados.rados.0.smithi125.stdout:2731: finishing rollback tid 0 to smithi12556470-20 2021-03-24T22:04:45.011 INFO:tasks.rados.rados.0.smithi125.stdout:2729: finishing write tid 3 to smithi12556470-6 2021-03-24T22:04:45.012 INFO:tasks.rados.rados.0.smithi125.stdout:2731: finishing rollback tid 1 to smithi12556470-20 2021-03-24T22:04:45.012 INFO:tasks.rados.rados.0.smithi125.stdout:2731: finishing rollback tid 2 to smithi12556470-20 2021-03-24T22:04:45.012 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 20 v 818 (ObjNum 733 snap 235 seq_num 733) dirty exists 2021-03-24T22:04:45.012 INFO:tasks.rados.rados.0.smithi125.stdout:2729: finishing write tid 4 to smithi12556470-6 2021-03-24T22:04:45.013 INFO:tasks.rados.rados.0.smithi125.stdout:2729: finishing write tid 5 to smithi12556470-6 2021-03-24T22:04:45.013 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 6 v 649 (ObjNum 947 snap 314 seq_num 947) dirty exists 2021-03-24T22:04:45.013 INFO:tasks.rados.rados.0.smithi125.stdout:2729: left oid 6 (ObjNum 947 snap 314 seq_num 947) 2021-03-24T22:04:45.013 INFO:tasks.rados.rados.0.smithi125.stdout:2732: finishing rollback tid 0 to smithi12556470-49 2021-03-24T22:04:45.014 INFO:tasks.rados.rados.0.smithi125.stdout:2732: finishing rollback tid 1 to smithi12556470-49 2021-03-24T22:04:45.014 INFO:tasks.rados.rados.0.smithi125.stdout:2732: finishing rollback tid 2 to smithi12556470-49 2021-03-24T22:04:45.017 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 49 v 971 (ObjNum 802 snap 262 seq_num 802) dirty exists 2021-03-24T22:04:45.017 INFO:tasks.rados.rados.0.smithi125.stdout:2730: expect (ObjNum 727 snap 233 seq_num 727) 2021-03-24T22:04:45.049 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:45.100 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:45.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:45.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:45.196 INFO:tasks.rados.rados.0.smithi125.stdout:2733: finishing write tid 1 to smithi12556470-29 2021-03-24T22:04:45.197 INFO:tasks.rados.rados.0.smithi125.stdout:2733: finishing write tid 2 to smithi12556470-29 2021-03-24T22:04:45.197 INFO:tasks.rados.rados.0.smithi125.stdout:2729: done (6 left) 2021-03-24T22:04:45.198 INFO:tasks.rados.rados.0.smithi125.stdout:2730: done (5 left) 2021-03-24T22:04:45.198 INFO:tasks.rados.rados.0.smithi125.stdout:2731: done (4 left) 2021-03-24T22:04:45.198 INFO:tasks.rados.rados.0.smithi125.stdout:2732: done (3 left) 2021-03-24T22:04:45.199 INFO:tasks.rados.rados.0.smithi125.stdout:2735: done (2 left) 2021-03-24T22:04:45.199 INFO:tasks.rados.rados.0.smithi125.stdout:2736: read oid 40 snap -1 2021-03-24T22:04:45.200 INFO:tasks.rados.rados.0.smithi125.stdout:2736: expect (ObjNum 939 snap 314 seq_num 939) 2021-03-24T22:04:45.201 INFO:tasks.rados.rados.0.smithi125.stdout:2737: delete oid 3 current snap is 314 2021-03-24T22:04:45.201 INFO:tasks.rados.rados.0.smithi125.stdout:2733: finishing write tid 3 to smithi12556470-29 2021-03-24T22:04:45.201 INFO:tasks.rados.rados.0.smithi125.stdout:2733: finishing write tid 4 to smithi12556470-29 2021-03-24T22:04:45.202 INFO:tasks.rados.rados.0.smithi125.stdout:2733: finishing write tid 5 to smithi12556470-29 2021-03-24T22:04:45.202 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 29 v 471 (ObjNum 948 snap 314 seq_num 948) dirty exists 2021-03-24T22:04:45.203 INFO:tasks.rados.rados.0.smithi125.stdout:2733: left oid 29 (ObjNum 948 snap 314 seq_num 948) 2021-03-24T22:04:45.203 INFO:tasks.rados.rados.0.smithi125.stdout:2734: finishing write tid 1 to smithi12556470-25 2021-03-24T22:04:45.203 INFO:tasks.rados.rados.0.smithi125.stdout:2734: finishing write tid 2 to smithi12556470-25 2021-03-24T22:04:45.204 INFO:tasks.rados.rados.0.smithi125.stdout:2734: finishing write tid 3 to smithi12556470-25 2021-03-24T22:04:45.204 INFO:tasks.rados.rados.0.smithi125.stdout:2734: finishing write tid 4 to smithi12556470-25 2021-03-24T22:04:45.205 INFO:tasks.rados.rados.0.smithi125.stdout:2734: finishing write tid 5 to smithi12556470-25 2021-03-24T22:04:45.205 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 25 v 822 (ObjNum 949 snap 314 seq_num 949) dirty exists 2021-03-24T22:04:45.206 INFO:tasks.rados.rados.0.smithi125.stdout:2734: left oid 25 (ObjNum 949 snap 314 seq_num 949) 2021-03-24T22:04:45.206 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:45.208 INFO:tasks.rados.rados.0.smithi125.stdout:2733: done (3 left) 2021-03-24T22:04:45.208 INFO:tasks.rados.rados.0.smithi125.stdout:2734: done (2 left) 2021-03-24T22:04:45.208 INFO:tasks.rados.rados.0.smithi125.stdout:2737: done (1 left) 2021-03-24T22:04:45.209 INFO:tasks.rados.rados.0.smithi125.stdout:2738: copy_from oid 21 from oid 22 current snap is 314 2021-03-24T22:04:45.209 INFO:tasks.rados.rados.0.smithi125.stdout:2739: snap_create 2021-03-24T22:04:45.209 INFO:tasks.rados.rados.0.smithi125.stdout:2738: finishing copy_from to smithi12556470-21 2021-03-24T22:04:45.209 INFO:tasks.rados.rados.0.smithi125.stdout:2738: got expected ENOENT (src dne) 2021-03-24T22:04:45.209 INFO:tasks.rados.rados.0.smithi125.stdout:2738: finishing copy_from racing read to smithi12556470-21 2021-03-24T22:04:45.210 INFO:tasks.rados.rados.0.smithi125.stdout:2736: expect (ObjNum 939 snap 314 seq_num 939) 2021-03-24T22:04:45.272 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:45.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:45.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:45.378 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:45.404 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-24T22:04:45.435 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:45.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:45.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:04:45.496 INFO:tasks.rados.rados.0.smithi125.stdout:2736: done (2 left) 2021-03-24T22:04:45.497 INFO:tasks.rados.rados.0.smithi125.stdout:2738: done (1 left) 2021-03-24T22:04:45.497 INFO:tasks.rados.rados.0.smithi125.stdout:2739: done (0 left) 2021-03-24T22:04:45.497 INFO:tasks.rados.rados.0.smithi125.stdout:2740: read oid 9 snap -1 2021-03-24T22:04:45.497 INFO:tasks.rados.rados.0.smithi125.stdout:2740: expect deleted 2021-03-24T22:04:45.498 INFO:tasks.rados.rados.0.smithi125.stdout:2741: snap_remove snap 252 2021-03-24T22:04:45.541 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:45.556 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:45 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:04:45.556 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:45 smithi125 conmon[19936]: 04:44.390133+0000 mon.a (mon.0) 1864 : cluster [DBG] osdmap e714: 8 total, 7 up, 8 in 2021-03-24T22:04:45.556 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:45 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T22:04:45.556 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:45 smithi125 conmon[19936]: -24T22:04:45.364562+0000 mon.a ( 2021-03-24T22:04:45.557 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:45 smithi125 conmon[19936]: mon.0) 1865 : cluster [WRN] 2021-03-24T22:04:45.557 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:45 smithi125 conmon[19936]: Health check update: Degraded data redundancy: 152/1902 objects degraded (7.992%), 19 pgs degraded (PG_DEGRADED) 2021-03-24T22:04:45.557 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:45 smithi125 conmon[24927]: cluster 2021-03-24T22:04:44. 2021-03-24T22:04:45.557 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:45 smithi125 conmon[24927]: 390133+0000 mon.a (mon.0) 1864 : cluster [DBG] osdmap e714: 8 total, 7 up, 8 in 2021-03-24T22:04:45.558 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:45 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T22:04:45.558 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:45 smithi125 conmon[24927]: T22:04:45.364562+0000 mon.a (mon.0) 1865 : cluster [WRN] Health check update: Degraded data redundancy: 152/1902 objects degraded (7.992%), 19 pgs degraded (PG_DEGRADED) 2021-03-24T22:04:45.587 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:45.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:45.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:45.611 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:45 smithi142 conmon[20125]: cluster 2021-03-24T22:04:44.390133+0000 mon.a (mon.0) 1864 : cluster [DBG] 2021-03-24T22:04:45.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:45 smithi142 conmon[20125]: osdmap e714: 8 total, 7 up, 8 in 2021-03-24T22:04:45.612 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:45 smithi142 conmon[20125]: cluster 2021-03-24T22:04:45.364562+0000 mon.a (mon.0) 1865 : cluster [WRN] Health check update: Degraded data redundancy: 152/1902 objects degraded (7.992%), 19 pgs degraded (PG_DEGRADED) 2021-03-24T22:04:45.692 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:45.743 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:45.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:45.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:45.850 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:45.908 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:45.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:45.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:46.015 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:46.072 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:46.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:46.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:04:46.178 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:46.202 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2021-03-24T22:04:46.203 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2021-03-24T22:04:46.203 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2021-03-24T22:04:46.203 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2021-03-24T22:04:46.204 INFO:teuthology.orchestra.run.smithi142.stdout:} 2021-03-24T22:04:46.231 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:46.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:46.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:46.284 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:04:46.289 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-03-24T22:04:46.290 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-03-24T22:04:46.338 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:46.389 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:46.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:46.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:46.415 INFO:tasks.rados.rados.0.smithi125.stdout:2741: done (1 left) 2021-03-24T22:04:46.416 INFO:tasks.rados.rados.0.smithi125.stdout:2742: delete oid 34 current snap is 315 2021-03-24T22:04:46.438 INFO:tasks.rados.rados.0.smithi125.stdout:2740: done (1 left) 2021-03-24T22:04:46.438 INFO:tasks.rados.rados.0.smithi125.stdout:2742: done (0 left) 2021-03-24T22:04:46.438 INFO:tasks.rados.rados.0.smithi125.stdout:2743: copy_from oid 19 from oid 35 current snap is 315 2021-03-24T22:04:46.439 INFO:tasks.rados.rados.0.smithi125.stdout:2744: rollback oid 32 current snap is 315 2021-03-24T22:04:46.439 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 32 to 238 2021-03-24T22:04:46.439 INFO:tasks.rados.rados.0.smithi125.stdout:2745: rollback oid 11 current snap is 315 2021-03-24T22:04:46.439 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 11 to 288 2021-03-24T22:04:46.439 INFO:tasks.rados.rados.0.smithi125.stdout:2746: snap_remove snap 271 2021-03-24T22:04:46.480 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:46 smithi125 conmon[19936]: cluster 2021-03-24T22:04:44 2021-03-24T22:04:46.480 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:46 smithi125 conmon[19936]: .961914+0000 mgr.y (mgr.14144) 632 : cluster [DBG] pgmap v1234: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 107 active+clean; 673 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 14 MiB/s rd, 2.8 MiB/s wr, 23 op/s; 154/1914 objects degraded (8.046%); 40/1914 objects misplaced (2.090%) 2021-03-24T22:04:46.480 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:46 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:04:46.481 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:46 smithi125 conmon[19936]: 03-24T22:04:45.395009+0000 mon.a (mon.0) 1866 : cluster [DBG] osdmap e715: 8 total, 7 up, 8 in 2021-03-24T22:04:46.481 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:46 smithi125 conmon[19936]: 2021-03-24T22:04:46.482 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:46 smithi125 conmon[24927]: cluster 2021-03-24T22:04:44.961914+0000 mgr.y (mgr.14144) 632 : cluster [DBG] pgmap v1234: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 107 active+clean; 673 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 14 MiB/s rd, 2.8 MiB/s wr, 23 op/s; 154/1914 objects degraded (8.046%); 40/1914 objects misplaced (2.090%) 2021-03-24T22:04:46.482 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:46 smithi125 conmon[24927]: cluster 2021-03-24T22:04:45.395009+0000 mon.a (mon.0) 1866 : cluster [DBG] osdmap e715: 8 total, 7 up, 8 in 2021-03-24T22:04:46.494 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:46.557 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:46.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:46.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:46.666 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:46.715 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:46.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:46.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:46.797 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-24T22:04:46.798 INFO:teuthology.orchestra.run.smithi125.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2021-03-24T22:04:46.798 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-24T22:04:46.809 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:46 smithi142 conmon[20125]: cluster 2021-03-24T22:04:44.961914 2021-03-24T22:04:46.810 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:46 smithi142 conmon[20125]: +0000 mgr.y (mgr.14144) 632 : cluster [DBG] pgmap v1234: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 107 active+clean; 673 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 14 MiB/s rd, 2.8 MiB/s wr, 23 op/s; 154/1914 objects degraded (8.046%); 40/1914 objects misplaced (2.090%) 2021-03-24T22:04:46.810 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:46 smithi142 conmon[20125]: cluster 2021-03-24T22:04:46.811 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:46 smithi142 conmon[20125]: 2021-03-24T 2021-03-24T22:04:46.811 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:46 smithi142 conmon[20125]: 22:04:45.395009+0000 mon.a 2021-03-24T22:04:46.811 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:46 smithi142 conmon[20125]: (mon.0) 1866 : cluster 2021-03-24T22:04:46.811 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:46 smithi142 conmon[20125]: [DBG] osdmap e715: 8 total, 7 up, 8 in 2021-03-24T22:04:46.822 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:46.869 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:46.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:46.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:46.974 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:47.030 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:47.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:47.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:04:47.083 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.6 dump_historic_ops 2021-03-24T22:04:47.136 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:47.191 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:47.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:47.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:47.296 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:47.348 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:47.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:47.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:47.376 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T22:04:47.426 INFO:tasks.rados.rados.0.smithi125.stdout:2746: done (3 left) 2021-03-24T22:04:47.427 INFO:tasks.rados.rados.0.smithi125.stdout:2747: write (excl) oid 24 current snap is 315 2021-03-24T22:04:47.427 INFO:tasks.rados.rados.0.smithi125.stdout:2745: finishing rollback tid 1 to smithi12556470-11 2021-03-24T22:04:47.427 INFO:tasks.rados.rados.0.smithi125.stdout:2747: seq_num 953 ranges {417071=656422,1567222=449849} 2021-03-24T22:04:47.435 INFO:tasks.rados.rados.0.smithi125.stdout:2747: writing smithi12556470-24 from 417071 to 1073493 tid 1 2021-03-24T22:04:47.443 INFO:tasks.rados.rados.0.smithi125.stdout:2747: writing smithi12556470-24 from 1567222 to 2017071 tid 2 2021-03-24T22:04:47.443 INFO:tasks.rados.rados.0.smithi125.stdout:2748: read oid 39 snap -1 2021-03-24T22:04:47.443 INFO:tasks.rados.rados.0.smithi125.stdout:2748: expect deleted 2021-03-24T22:04:47.444 INFO:tasks.rados.rados.0.smithi125.stdout:2749: snap_remove snap 249 2021-03-24T22:04:47.453 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:47.461 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:47 smithi125 conmon[19936]: cluster 2021-03-24T22:04:46 2021-03-24T22:04:47.462 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:47 smithi125 conmon[19936]: .411250+0000 mon.a (mon.0) 1867 : cluster [DBG] osdmap e716: 8 total, 7 up, 8 in 2021-03-24T22:04:47.462 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:47 smithi125 conmon[24927]: cluster 2021-03-24T22:04:46. 2021-03-24T22:04:47.463 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:47 smithi125 conmon[24927]: 411250+0000 mon.a (mon.0) 1867 : cluster [DBG] osdmap e716: 8 total, 7 up, 8 in 2021-03-24T22:04:47.537 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:47.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:47.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:47.643 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:47.695 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:47.777 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:47 smithi142 conmon[20125]: cluster 2021-03-24T22:04:46.411250+0000 mon.a (mon.0) 1867 : cluster [DBG] osdmap e716: 8 total, 7 up, 8 in 2021-03-24T22:04:47.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:47.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:04:47.882 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:47.939 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:47.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:47.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:48.045 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:48.099 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:48.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:48.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:48.205 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:48.261 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:48.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:48.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:48.291 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-24T22:04:48.367 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:48.416 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:48.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:48.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:48.433 INFO:tasks.rados.rados.0.smithi125.stdout:2749: done (5 left) 2021-03-24T22:04:48.433 INFO:tasks.rados.rados.0.smithi125.stdout:2750: rollback oid 6 current snap is 315 2021-03-24T22:04:48.434 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 6 to 311 2021-03-24T22:04:48.434 INFO:tasks.rados.rados.0.smithi125.stdout:2751: snap_create 2021-03-24T22:04:48.434 INFO:tasks.rados.rados.0.smithi125.stdout:2745: finishing rollback tid 2 to smithi12556470-11 2021-03-24T22:04:48.434 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 11 v 556 (ObjNum 809 snap 265 seq_num 809) dirty exists 2021-03-24T22:04:48.435 INFO:tasks.rados.rados.0.smithi125.stdout:2743: finishing copy_from to smithi12556470-19 2021-03-24T22:04:48.435 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 19 v 231 (ObjNum 908 snap 300 seq_num 908) dirty exists 2021-03-24T22:04:48.435 INFO:tasks.rados.rados.0.smithi125.stdout:2743: finishing copy_from racing read to smithi12556470-19 2021-03-24T22:04:48.435 INFO:tasks.rados.rados.0.smithi125.stdout:2744: finishing rollback tid 1 to smithi12556470-32 2021-03-24T22:04:48.436 INFO:tasks.rados.rados.0.smithi125.stdout:2744: finishing rollback tid 2 to smithi12556470-32 2021-03-24T22:04:48.436 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 32 v 1655 (ObjNum 672 snap 209 seq_num 672) dirty exists 2021-03-24T22:04:48.436 INFO:tasks.rados.rados.0.smithi125.stdout:2747: finishing write tid 1 to smithi12556470-24 2021-03-24T22:04:48.436 INFO:tasks.rados.rados.0.smithi125.stdout:2747: finishing write tid 2 to smithi12556470-24 2021-03-24T22:04:48.437 INFO:tasks.rados.rados.0.smithi125.stdout:2747: finishing write tid 3 to smithi12556470-24 2021-03-24T22:04:48.437 INFO:tasks.rados.rados.0.smithi125.stdout:2747: finishing write tid 4 to smithi12556470-24 2021-03-24T22:04:48.437 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 24 v 1425 (ObjNum 953 snap 315 seq_num 953) dirty exists 2021-03-24T22:04:48.437 INFO:tasks.rados.rados.0.smithi125.stdout:2747: left oid 24 (ObjNum 953 snap 315 seq_num 953) 2021-03-24T22:04:48.460 INFO:tasks.rados.rados.0.smithi125.stdout:2750: finishing rollback tid 0 to smithi12556470-6 2021-03-24T22:04:48.460 INFO:tasks.rados.rados.0.smithi125.stdout:2750: finishing rollback tid 1 to smithi12556470-6 2021-03-24T22:04:48.461 INFO:tasks.rados.rados.0.smithi125.stdout:2750: finishing rollback tid 2 to smithi12556470-6 2021-03-24T22:04:48.461 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 6 v 659 (ObjNum 906 snap 300 seq_num 906) dirty exists 2021-03-24T22:04:48.522 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:48.573 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:48.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:48.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:04:48.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:48 smithi125 conmon[19936]: cluster 2021-03-24T22:04:46.962846+0000 2021-03-24T22:04:48.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:48 smithi125 conmon[19936]: mgr.y (mgr.14144) 633 : cluster [DBG] pgmap v1237: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 107 active+clean; 673 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 9.5 MiB/s rd, 2.5 MiB/s wr, 19 op/s; 154/1920 objects degraded (8.021%); 40/1920 objects misplaced (2.083%) 2021-03-24T22:04:48.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:48 smithi125 conmon[19936]: cluster 2021-03-24T22:04:47.422680+0000 mon.a (mon.0) 1868 : cluster [DBG] osdmap e717: 8 total, 7 up, 8 in 2021-03-24T22:04:48.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:48 smithi125 conmon[24927]: cluster 2021-03-24T22:04:48.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:48 smithi125 conmon[24927]: 2021-03-24T22:04:48.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:48 smithi125 conmon[24927]: 2021 2021-03-24T22:04:48.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:48 smithi125 conmon[24927]: - 2021-03-24T22:04:48.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:48 smithi125 conmon[24927]: 03 2021-03-24T22:04:48.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:48 smithi125 conmon[24927]: - 2021-03-24T22:04:48.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:48 smithi125 conmon[24927]: 24 2021-03-24T22:04:48.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:48 smithi125 conmon[24927]: T22:04:46. 2021-03-24T22:04:48.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:48 smithi125 conmon[24927]: 962846+0000 mgr.y (mgr.14144) 633 : cluster [DBG] pgmap v1237: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 107 active+clean; 673 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 9.5 MiB/s rd, 2.5 MiB/s wr, 19 op/s; 154/1920 objects degraded (8.021%); 40/1920 objects misplaced (2.083%) 2021-03-24T22:04:48.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:48 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:04:48.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:48 smithi125 conmon[24927]: -03-24T22:04:47.422680+0000 mon.a (mon. 2021-03-24T22:04:48.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:48 smithi125 conmon[24927]: 0) 1868 : cluster [DBG] osdmap e717: 8 total, 7 up, 8 in 2021-03-24T22:04:48.679 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:48.728 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:48 smithi142 conmon[20125]: cluster 2021-03-24T22:04:48.728 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:48 smithi142 conmon[20125]: 2021-03-24T22:04:48.728 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:48 smithi142 conmon[20125]: 2021 2021-03-24T22:04:48.729 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:48 smithi142 conmon[20125]: - 2021-03-24T22:04:48.729 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:48 smithi142 conmon[20125]: 03 2021-03-24T22:04:48.730 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:48 smithi142 conmon[20125]: - 2021-03-24T22:04:48.730 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:48 smithi142 conmon[20125]: 24 2021-03-24T22:04:48.730 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:48 smithi142 conmon[20125]: T 2021-03-24T22:04:48.730 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:48 smithi142 conmon[20125]: 22 2021-03-24T22:04:48.730 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:48 smithi142 conmon[20125]: : 2021-03-24T22:04:48.731 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:48 smithi142 conmon[20125]: 04 2021-03-24T22:04:48.731 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:48 smithi142 conmon[20125]: : 2021-03-24T22:04:48.731 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:48 smithi142 conmon[20125]: 46 2021-03-24T22:04:48.731 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:48 smithi142 conmon[20125]: . 2021-03-24T22:04:48.732 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:48 smithi142 conmon[20125]: 962846 2021-03-24T22:04:48.732 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:48 smithi142 conmon[20125]: +0000 mgr.y (mgr.14144) 633 : cluster [DBG] pgmap v1237: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 107 active+clean; 673 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 9.5 MiB/s rd, 2.5 MiB/s wr, 19 op/s; 154/1920 objects degraded (8.021%); 40/1920 objects misplaced (2.083%) 2021-03-24T22:04:48.732 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:48 smithi142 conmon[20125]: cluster 2021-03-24T22:04:47.422680+0000 mon.a (mon.0) 1868 : cluster [DBG] osdmap e717: 8 total, 7 up, 8 in 2021-03-24T22:04:48.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:48.734 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:48.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:04:48.836 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:48.886 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:48.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:48.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:48.927 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:04:48.992 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:49.032 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:49.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:49.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:04:49.137 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:49.192 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:49.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:49.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:49.288 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2021-03-24T22:04:49.288 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2021-03-24T22:04:49.288 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2021-03-24T22:04:49.289 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2021-03-24T22:04:49.289 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.289 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:5168 7.c 7.3b9cea0c (undecoded) ondisk+write+known_if_redirected e532)", 2021-03-24T22:04:49.289 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:01:42.355376+0000", 2021-03-24T22:04:49.290 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 186.92738590900001, 2021-03-24T22:04:49.290 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.016998772, 2021-03-24T22:04:49.290 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:04:49.290 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "reached pg", 2021-03-24T22:04:49.291 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:04:49.291 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:04:49.291 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:04:49.291 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 5168 2021-03-24T22:04:49.291 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.292 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:04:49.292 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.292 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:04:49.293 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:42.355376+0000", 2021-03-24T22:04:49.293 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:04:49.293 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.293 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.294 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:04:49.294 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:42.355376+0000", 2021-03-24T22:04:49.294 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999967 2021-03-24T22:04:49.294 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.295 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.295 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:04:49.295 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:42.355372+0000", 2021-03-24T22:04:49.295 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7163999999999999e-05 2021-03-24T22:04:49.296 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.296 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.296 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:04:49.296 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:42.355399+0000", 2021-03-24T22:04:49.297 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0228e-05 2021-03-24T22:04:49.297 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.297 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.297 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:04:49.297 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:42.355420+0000", 2021-03-24T22:04:49.298 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2044e-05 2021-03-24T22:04:49.298 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.298 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.298 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:04:49.298 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:42.355432+0000", 2021-03-24T22:04:49.299 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029462675000000001 2021-03-24T22:04:49.299 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.299 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.299 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:04:49.299 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:42.384894+0000", 2021-03-24T22:04:49.300 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2527000000000001e-05 2021-03-24T22:04:49.300 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.300 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.301 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for peered", 2021-03-24T22:04:49.301 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:42.384907+0000", 2021-03-24T22:04:49.301 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.98745425200000003 2021-03-24T22:04:49.301 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.301 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.302 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:04:49.302 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:43.372361+0000", 2021-03-24T22:04:49.302 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3349e-05 2021-03-24T22:04:49.302 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.303 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.303 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:04:49.303 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:43.372374+0000", 2021-03-24T22:04:49.303 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.016998772 2021-03-24T22:04:49.303 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:04:49.304 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:04:49.304 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:04:49.304 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.304 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.304 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:5619 7.19 7:992cb487:::smithi12556470-5:head [copy-from ver 374] snapc 1d8=[1d8,1d7,1d5,1d4,1d1,1cd,1cc,1c4,1bf,1bd,1bb,1b6,1b3,1ae,1ab,1a6,19e,19c,199,197,194,193,18b,178,172,16e,159,157,139] ondisk+write+known_if_redirected e589)", 2021-03-24T22:04:49.305 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:02:39.378299+0000", 2021-03-24T22:04:49.305 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 129.904462736, 2021-03-24T22:04:49.305 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028013256, 2021-03-24T22:04:49.305 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:04:49.306 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:04:49.306 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:04:49.306 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:04:49.306 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:04:49.306 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 5619 2021-03-24T22:04:49.307 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.307 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:04:49.307 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.307 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:04:49.308 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:39.378299+0000", 2021-03-24T22:04:49.308 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:04:49.308 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.308 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.309 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:04:49.309 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:39.378299+0000", 2021-03-24T22:04:49.309 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999948 2021-03-24T22:04:49.309 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.310 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.310 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:04:49.310 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:39.378294+0000", 2021-03-24T22:04:49.310 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0367999999999999e-05 2021-03-24T22:04:49.311 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.311 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.311 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:04:49.311 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:39.378314+0000", 2021-03-24T22:04:49.311 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8337e-05 2021-03-24T22:04:49.312 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.312 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.312 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:04:49.313 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:39.378342+0000", 2021-03-24T22:04:49.313 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.394e-05 2021-03-24T22:04:49.313 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.313 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.313 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:04:49.314 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:39.378356+0000", 2021-03-24T22:04:49.314 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000124299 2021-03-24T22:04:49.314 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.314 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.315 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:04:49.315 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:39.378480+0000", 2021-03-24T22:04:49.315 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0701e-05 2021-03-24T22:04:49.315 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.316 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.316 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op must wait for map", 2021-03-24T22:04:49.316 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:39.378491+0000", 2021-03-24T22:04:49.317 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.002734729 2021-03-24T22:04:49.317 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.317 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.318 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:04:49.318 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:39.381226+0000", 2021-03-24T22:04:49.318 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016190599999999999 2021-03-24T22:04:49.318 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.319 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.319 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:04:49.319 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:39.381388+0000", 2021-03-24T22:04:49.319 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0072900480000000004 2021-03-24T22:04:49.320 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.320 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.320 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 2,7", 2021-03-24T22:04:49.320 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:39.388678+0000", 2021-03-24T22:04:49.321 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.010422179 2021-03-24T22:04:49.321 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.321 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.321 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:04:49.322 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:39.399100+0000", 2021-03-24T22:04:49.322 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0065367130000000004 2021-03-24T22:04:49.322 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.322 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.323 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T22:04:49.323 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:39.405637+0000", 2021-03-24T22:04:49.323 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000348513 2021-03-24T22:04:49.323 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.324 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.324 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:04:49.324 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:39.405985+0000", 2021-03-24T22:04:49.324 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012643600000000001 2021-03-24T22:04:49.325 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.325 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.325 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:04:49.326 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:39.406112+0000", 2021-03-24T22:04:49.326 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00020035499999999999 2021-03-24T22:04:49.326 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.326 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.327 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:04:49.327 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:39.406312+0000", 2021-03-24T22:04:49.327 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028013256 2021-03-24T22:04:49.327 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:04:49.327 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:04:49.328 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:04:49.328 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.328 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.328 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:5620 7.19 7:992cb487:::smithi12556470-5:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e589)", 2021-03-24T22:04:49.329 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:02:39.378379+0000", 2021-03-24T22:04:49.329 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 129.90438281799999, 2021-03-24T22:04:49.329 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028362508000000002, 2021-03-24T22:04:49.329 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:04:49.330 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:04:49.330 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:04:49.330 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:04:49.330 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:04:49.331 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 5620 2021-03-24T22:04:49.331 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.331 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:04:49.331 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.331 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:04:49.332 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:39.378379+0000", 2021-03-24T22:04:49.332 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:04:49.332 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.332 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.332 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:04:49.333 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:39.378379+0000", 2021-03-24T22:04:49.333 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2021-03-24T22:04:49.333 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.333 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.334 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:04:49.334 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:39.378378+0000", 2021-03-24T22:04:49.334 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.69e-06 2021-03-24T22:04:49.334 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.335 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.335 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:04:49.335 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:39.378382+0000", 2021-03-24T22:04:49.335 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.7649999999999999e-06 2021-03-24T22:04:49.336 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.336 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.336 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:04:49.336 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:39.378391+0000", 2021-03-24T22:04:49.336 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5979999999999996e-06 2021-03-24T22:04:49.337 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.337 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.337 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:04:49.337 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:39.378397+0000", 2021-03-24T22:04:49.338 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016525900000000001 2021-03-24T22:04:49.338 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.338 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.338 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:04:49.338 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:39.378562+0000", 2021-03-24T22:04:49.339 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.5690000000000005e-06 2021-03-24T22:04:49.339 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.339 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.339 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting_for_map not empty", 2021-03-24T22:04:49.340 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:39.378571+0000", 2021-03-24T22:04:49.340 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0030041289999999999 2021-03-24T22:04:49.340 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.340 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.341 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:04:49.341 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:39.381575+0000", 2021-03-24T22:04:49.341 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0303000000000002e-05 2021-03-24T22:04:49.341 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.342 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.342 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for blocked object", 2021-03-24T22:04:49.342 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:39.381635+0000", 2021-03-24T22:04:49.342 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.013566168999999999 2021-03-24T22:04:49.343 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.343 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.343 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:04:49.343 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:39.395201+0000", 2021-03-24T22:04:49.344 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012091000000000001 2021-03-24T22:04:49.344 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.344 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.344 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for rw locks", 2021-03-24T22:04:49.344 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:39.395322+0000", 2021-03-24T22:04:49.345 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011102195 2021-03-24T22:04:49.345 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.345 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.345 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:04:49.346 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:39.406424+0000", 2021-03-24T22:04:49.346 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014873999999999999 2021-03-24T22:04:49.346 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.346 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.347 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:04:49.347 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:39.406573+0000", 2021-03-24T22:04:49.347 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016822300000000001 2021-03-24T22:04:49.347 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.348 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.348 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:04:49.348 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:39.406741+0000", 2021-03-24T22:04:49.348 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028362508000000002 2021-03-24T22:04:49.348 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:04:49.349 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:04:49.349 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:04:49.349 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.350 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.350 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:5642 7.23 7:c49de0b4:::smithi12556470-13:head [copy-from ver 788] snapc 1d9=[1d9,1d8,1d7,1d5,1d4,1d1,1cd,1cc,1c4,1bf,1bd,1bb,1b3,1ae,1ab,1a6,19e,19c,199,197,194,193,18b,178,172,16e,159,157,139] ondisk+write+known_if_redirected e591)", 2021-03-24T22:04:49.350 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:02:41.329992+0000", 2021-03-24T22:04:49.350 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 127.952769531, 2021-03-24T22:04:49.351 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029427207, 2021-03-24T22:04:49.351 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:04:49.351 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:04:49.352 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:04:49.352 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:04:49.352 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:04:49.352 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 5642 2021-03-24T22:04:49.353 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.353 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:04:49.353 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.354 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:04:49.354 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:41.329992+0000", 2021-03-24T22:04:49.354 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:04:49.355 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.355 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.355 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:04:49.355 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:41.329992+0000", 2021-03-24T22:04:49.355 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999967 2021-03-24T22:04:49.356 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.356 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.356 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:04:49.356 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:41.329989+0000", 2021-03-24T22:04:49.357 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2493999999999999e-05 2021-03-24T22:04:49.357 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.357 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.357 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:04:49.358 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:41.330001+0000", 2021-03-24T22:04:49.358 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0505000000000001e-05 2021-03-24T22:04:49.358 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.359 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.359 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:04:49.359 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:41.330022+0000", 2021-03-24T22:04:49.359 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9949999999999994e-06 2021-03-24T22:04:49.359 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.360 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.360 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:04:49.360 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:41.330031+0000", 2021-03-24T22:04:49.360 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018925300000000001 2021-03-24T22:04:49.360 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.361 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.361 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:04:49.361 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:41.330220+0000", 2021-03-24T22:04:49.361 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018625200000000001 2021-03-24T22:04:49.362 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.362 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.362 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:04:49.362 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:41.330406+0000", 2021-03-24T22:04:49.363 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0088721319999999996 2021-03-24T22:04:49.363 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.363 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.363 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1,3", 2021-03-24T22:04:49.363 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:41.339278+0000", 2021-03-24T22:04:49.364 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0056579320000000001 2021-03-24T22:04:49.364 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.364 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.364 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T22:04:49.365 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:41.344936+0000", 2021-03-24T22:04:49.365 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0092010600000000005 2021-03-24T22:04:49.365 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.365 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.365 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:04:49.366 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:41.354137+0000", 2021-03-24T22:04:49.366 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0051462000000000001 2021-03-24T22:04:49.366 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.366 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.367 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:04:49.367 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:41.359284+0000", 2021-03-24T22:04:49.367 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.4196000000000007e-05 2021-03-24T22:04:49.368 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.369 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.369 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:04:49.370 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:41.359348+0000", 2021-03-24T22:04:49.370 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1333000000000003e-05 2021-03-24T22:04:49.370 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.370 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.371 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:04:49.371 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:41.359419+0000", 2021-03-24T22:04:49.371 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029427207 2021-03-24T22:04:49.371 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:04:49.372 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:04:49.372 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:04:49.372 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.373 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.373 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:5643 7.23 7:c49de0b4:::smithi12556470-13:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e591)", 2021-03-24T22:04:49.373 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:02:41.330051+0000", 2021-03-24T22:04:49.374 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 127.952710724, 2021-03-24T22:04:49.374 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029624616999999999, 2021-03-24T22:04:49.374 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:04:49.375 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:04:49.375 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:04:49.375 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:04:49.375 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:04:49.375 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 5643 2021-03-24T22:04:49.376 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.376 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:04:49.376 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.377 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:04:49.377 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:41.330051+0000", 2021-03-24T22:04:49.377 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:04:49.377 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.377 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.378 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:04:49.378 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:41.330051+0000", 2021-03-24T22:04:49.379 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2021-03-24T22:04:49.379 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.379 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.379 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:04:49.380 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:41.330050+0000", 2021-03-24T22:04:49.380 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7579999999999998e-06 2021-03-24T22:04:49.380 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.380 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.380 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:04:49.381 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:41.330055+0000", 2021-03-24T22:04:49.381 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.1040000000000007e-06 2021-03-24T22:04:49.381 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.381 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.382 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:04:49.382 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:41.330063+0000", 2021-03-24T22:04:49.382 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.891e-05 2021-03-24T22:04:49.383 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.383 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.383 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:04:49.383 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:41.330082+0000", 2021-03-24T22:04:49.384 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00052406399999999998 2021-03-24T22:04:49.384 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.385 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.385 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:04:49.385 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:41.330606+0000", 2021-03-24T22:04:49.386 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7522000000000003e-05 2021-03-24T22:04:49.386 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.386 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.386 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for blocked object", 2021-03-24T22:04:49.387 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:41.330664+0000", 2021-03-24T22:04:49.387 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.013761851 2021-03-24T22:04:49.387 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.387 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.387 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:04:49.388 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:41.344426+0000", 2021-03-24T22:04:49.388 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1167999999999994e-05 2021-03-24T22:04:49.388 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.389 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.389 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for rw locks", 2021-03-24T22:04:49.389 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:41.344497+0000", 2021-03-24T22:04:49.389 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.014980991000000001 2021-03-24T22:04:49.390 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.390 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.390 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:04:49.390 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:41.359478+0000", 2021-03-24T22:04:49.391 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.8147999999999996e-05 2021-03-24T22:04:49.391 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.391 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.391 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:04:49.391 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:41.359576+0000", 2021-03-24T22:04:49.392 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.9320000000000005e-05 2021-03-24T22:04:49.392 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.392 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.393 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:04:49.393 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:02:41.359675+0000", 2021-03-24T22:04:49.393 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029624616999999999 2021-03-24T22:04:49.393 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:04:49.394 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:04:49.394 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:04:49.394 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.394 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.395 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:6121 7.c 7:32974111:::smithi12556470-2:head [copy-from ver 827] snapc 202=[202,201,1ff,1fd,1f8,1f3,1f0,1ef,1ee,1ec,1e6,1e1,1df,1de,1dc,1d8,1d4,1cd,1bf,1bd,19c,199,172,16e] ondisk+write+known_if_redirected e633)", 2021-03-24T22:04:49.395 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:03:23.467594+0000", 2021-03-24T22:04:49.395 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 85.815167582000001, 2021-03-24T22:04:49.396 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031992291999999999, 2021-03-24T22:04:49.396 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:04:49.396 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:04:49.396 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:04:49.397 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:04:49.397 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:04:49.397 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 6121 2021-03-24T22:04:49.397 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.397 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:04:49.398 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.398 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:04:49.398 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.467594+0000", 2021-03-24T22:04:49.398 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:04:49.399 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.399 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.400 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:04:49.400 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.467594+0000", 2021-03-24T22:04:49.400 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2021-03-24T22:04:49.400 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.400 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.401 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:04:49.401 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.467593+0000", 2021-03-24T22:04:49.401 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8450000000000002e-06 2021-03-24T22:04:49.401 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.402 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.402 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:04:49.402 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.467598+0000", 2021-03-24T22:04:49.402 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.2989999999999997e-06 2021-03-24T22:04:49.402 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.403 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.403 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:04:49.403 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.467606+0000", 2021-03-24T22:04:49.403 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1449999999999996e-06 2021-03-24T22:04:49.403 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.404 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.404 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:04:49.404 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.467613+0000", 2021-03-24T22:04:49.404 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00037132700000000002 2021-03-24T22:04:49.405 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.405 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.406 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:04:49.406 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.467984+0000", 2021-03-24T22:04:49.407 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00019477700000000001 2021-03-24T22:04:49.408 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.408 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.408 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:04:49.408 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.468179+0000", 2021-03-24T22:04:49.409 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.012803760000000001 2021-03-24T22:04:49.409 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.409 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.409 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3,7", 2021-03-24T22:04:49.410 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.480982+0000", 2021-03-24T22:04:49.410 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011532875 2021-03-24T22:04:49.410 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.410 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.410 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:04:49.411 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.492515+0000", 2021-03-24T22:04:49.411 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00058181400000000005 2021-03-24T22:04:49.411 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.411 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.412 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T22:04:49.412 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.493097+0000", 2021-03-24T22:04:49.412 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0063084619999999999 2021-03-24T22:04:49.412 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.412 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.413 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:04:49.413 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.499406+0000", 2021-03-24T22:04:49.413 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6482000000000004e-05 2021-03-24T22:04:49.413 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.414 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.414 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:04:49.414 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.499482+0000", 2021-03-24T22:04:49.414 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000104156 2021-03-24T22:04:49.414 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.415 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.415 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:04:49.415 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.499586+0000", 2021-03-24T22:04:49.415 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031992291999999999 2021-03-24T22:04:49.416 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:04:49.416 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:04:49.416 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:04:49.416 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.416 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.417 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:6122 7.c 7:32974111:::smithi12556470-2:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e633)", 2021-03-24T22:04:49.417 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:03:23.467622+0000", 2021-03-24T22:04:49.417 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 85.815140013000004, 2021-03-24T22:04:49.418 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032215336999999997, 2021-03-24T22:04:49.418 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:04:49.418 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:04:49.419 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:04:49.419 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:04:49.419 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:04:49.419 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 6122 2021-03-24T22:04:49.419 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.420 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:04:49.420 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.420 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:04:49.420 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.467622+0000", 2021-03-24T22:04:49.421 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:04:49.421 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.421 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.421 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:04:49.421 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.467622+0000", 2021-03-24T22:04:49.422 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2021-03-24T22:04:49.422 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.422 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.422 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:04:49.423 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.467621+0000", 2021-03-24T22:04:49.423 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.3989999999999997e-06 2021-03-24T22:04:49.423 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.423 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.424 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:04:49.424 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.467625+0000", 2021-03-24T22:04:49.424 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1046000000000001e-05 2021-03-24T22:04:49.425 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.425 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.425 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:04:49.425 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.467636+0000", 2021-03-24T22:04:49.426 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.5799999999999997e-06 2021-03-24T22:04:49.426 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.426 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.426 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:04:49.426 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.467643+0000", 2021-03-24T22:04:49.427 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00070741200000000004 2021-03-24T22:04:49.427 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.427 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.427 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:04:49.427 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.468350+0000", 2021-03-24T22:04:49.428 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.2981999999999999e-05 2021-03-24T22:04:49.428 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.429 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.429 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for blocked object", 2021-03-24T22:04:49.429 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.468433+0000", 2021-03-24T22:04:49.429 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.020434994000000001 2021-03-24T22:04:49.429 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.430 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.430 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:04:49.430 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.488868+0000", 2021-03-24T22:04:49.430 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012197399999999999 2021-03-24T22:04:49.431 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.431 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.431 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for rw locks", 2021-03-24T22:04:49.431 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.488990+0000", 2021-03-24T22:04:49.432 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.010652708 2021-03-24T22:04:49.432 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.432 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.432 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:04:49.432 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.499643+0000", 2021-03-24T22:04:49.433 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010662000000000001 2021-03-24T22:04:49.433 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.433 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.433 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:04:49.434 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.499749+0000", 2021-03-24T22:04:49.434 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.7607000000000003e-05 2021-03-24T22:04:49.434 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.434 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.434 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:04:49.435 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.499837+0000", 2021-03-24T22:04:49.435 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032215336999999997 2021-03-24T22:04:49.435 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:04:49.435 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:04:49.436 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:04:49.436 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.436 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.436 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:6222 7.d 7:b2218182:::smithi12556470-21:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e642)", 2021-03-24T22:04:49.436 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:03:32.409316+0000", 2021-03-24T22:04:49.437 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 76.873445184999994, 2021-03-24T22:04:49.437 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.982999293, 2021-03-24T22:04:49.437 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:04:49.437 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "reached pg", 2021-03-24T22:04:49.438 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:04:49.438 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:04:49.438 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:04:49.440 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 6222 2021-03-24T22:04:49.440 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.441 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:04:49.441 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.441 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:04:49.441 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:32.409316+0000", 2021-03-24T22:04:49.442 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:04:49.442 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.442 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.442 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:04:49.442 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:32.409316+0000", 2021-03-24T22:04:49.443 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2021-03-24T22:04:49.443 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.443 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.443 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:04:49.444 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:32.409315+0000", 2021-03-24T22:04:49.444 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4319999999999998e-06 2021-03-24T22:04:49.444 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.444 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.444 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:04:49.445 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:32.409321+0000", 2021-03-24T22:04:49.445 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.5619999999999998e-06 2021-03-24T22:04:49.445 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.445 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.446 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:04:49.446 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:32.409328+0000", 2021-03-24T22:04:49.446 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.2499999999999997e-06 2021-03-24T22:04:49.446 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.446 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.447 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:04:49.447 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:32.409334+0000", 2021-03-24T22:04:49.447 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7930000000000003e-05 2021-03-24T22:04:49.447 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.448 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.448 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:04:49.448 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:32.409392+0000", 2021-03-24T22:04:49.448 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0210999999999999e-05 2021-03-24T22:04:49.448 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.449 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.449 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for peered", 2021-03-24T22:04:49.449 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:32.409402+0000", 2021-03-24T22:04:49.449 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.98275195999999998 2021-03-24T22:04:49.450 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.450 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.451 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:04:49.451 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:33.392154+0000", 2021-03-24T22:04:49.451 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016194899999999999 2021-03-24T22:04:49.451 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.451 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.452 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:04:49.452 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:33.392316+0000", 2021-03-24T22:04:49.452 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.982999293 2021-03-24T22:04:49.452 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:04:49.453 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:04:49.453 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:04:49.453 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.453 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.453 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:6223 7.d 7:b2218182:::smithi12556470-21:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e642)", 2021-03-24T22:04:49.454 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:03:32.409350+0000", 2021-03-24T22:04:49.454 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 76.873412000000002, 2021-03-24T22:04:49.454 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.983107921, 2021-03-24T22:04:49.454 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:04:49.455 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "reached pg", 2021-03-24T22:04:49.455 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:04:49.455 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:04:49.455 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:04:49.456 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 6223 2021-03-24T22:04:49.456 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.456 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:04:49.456 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.457 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:04:49.457 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:32.409350+0000", 2021-03-24T22:04:49.457 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:04:49.457 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.457 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.458 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:04:49.458 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:32.409350+0000", 2021-03-24T22:04:49.458 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2021-03-24T22:04:49.458 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.459 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.459 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:04:49.459 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:32.409349+0000", 2021-03-24T22:04:49.459 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2540000000000003e-06 2021-03-24T22:04:49.459 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.460 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.460 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:04:49.460 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:32.409353+0000", 2021-03-24T22:04:49.460 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7969999999999997e-06 2021-03-24T22:04:49.461 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.461 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.461 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:04:49.461 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:32.409359+0000", 2021-03-24T22:04:49.461 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8630000000000001e-06 2021-03-24T22:04:49.462 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.462 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.462 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:04:49.462 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:32.409363+0000", 2021-03-24T22:04:49.463 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1345000000000002e-05 2021-03-24T22:04:49.463 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.463 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.463 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:04:49.463 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:32.409434+0000", 2021-03-24T22:04:49.464 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0365e-05 2021-03-24T22:04:49.464 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.464 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.464 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for peered", 2021-03-24T22:04:49.464 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:32.409444+0000", 2021-03-24T22:04:49.465 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.98290503500000004 2021-03-24T22:04:49.465 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.465 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.465 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:04:49.466 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:33.392349+0000", 2021-03-24T22:04:49.466 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000108089 2021-03-24T22:04:49.466 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.466 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.466 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:04:49.467 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:33.392457+0000", 2021-03-24T22:04:49.467 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.983107921 2021-03-24T22:04:49.467 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:04:49.467 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:04:49.468 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:04:49.468 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.468 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.468 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:6224 7.d 7:b2218182:::smithi12556470-21:head [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected e642)", 2021-03-24T22:04:49.469 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:03:32.409380+0000", 2021-03-24T22:04:49.469 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 76.873381335000005, 2021-03-24T22:04:49.469 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.98319098800000004, 2021-03-24T22:04:49.469 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:04:49.469 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "reached pg", 2021-03-24T22:04:49.470 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:04:49.470 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:04:49.470 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:04:49.470 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 6224 2021-03-24T22:04:49.471 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.471 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:04:49.471 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.471 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:04:49.472 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:32.409380+0000", 2021-03-24T22:04:49.472 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:04:49.472 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.472 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.472 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:04:49.473 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:32.409380+0000", 2021-03-24T22:04:49.473 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2021-03-24T22:04:49.473 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.473 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.474 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:04:49.474 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:32.409380+0000", 2021-03-24T22:04:49.474 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9380000000000001e-06 2021-03-24T22:04:49.474 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.475 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.475 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:04:49.475 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:32.409383+0000", 2021-03-24T22:04:49.475 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.9140000000000001e-06 2021-03-24T22:04:49.475 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.476 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.476 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:04:49.476 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:32.409387+0000", 2021-03-24T22:04:49.476 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0900000000000001e-06 2021-03-24T22:04:49.477 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.477 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.477 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:04:49.477 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:32.409390+0000", 2021-03-24T22:04:49.478 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9024999999999996e-05 2021-03-24T22:04:49.478 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.478 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.478 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:04:49.479 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:32.409479+0000", 2021-03-24T22:04:49.479 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.1699999999999997e-06 2021-03-24T22:04:49.479 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.480 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.480 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for peered", 2021-03-24T22:04:49.480 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:32.409487+0000", 2021-03-24T22:04:49.480 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.98298822799999996 2021-03-24T22:04:49.480 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.481 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.481 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:04:49.481 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:33.392475+0000", 2021-03-24T22:04:49.482 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6088000000000006e-05 2021-03-24T22:04:49.482 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.482 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.483 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:04:49.483 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:33.392571+0000", 2021-03-24T22:04:49.483 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.98319098800000004 2021-03-24T22:04:49.483 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:04:49.483 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:04:49.484 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:04:49.484 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.484 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.484 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:6374 7.c 7:32974111:::smithi12556470-2:head [copy-from ver 430] snapc 219=[219,218,217,214,211,210,20e,20d,209,204,1fd,1f8,1f3,1f0,1ef,1e1,1de,1dc,1d8,1d4,1cd,1bf,1bd,199,172,16e] ondisk+write+known_if_redirected e655)", 2021-03-24T22:04:49.485 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:03:45.363192+0000", 2021-03-24T22:04:49.485 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 63.919569938000002, 2021-03-24T22:04:49.485 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.101716506, 2021-03-24T22:04:49.485 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:04:49.486 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:04:49.486 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:04:49.486 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:04:49.486 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:04:49.487 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 6374 2021-03-24T22:04:49.487 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.487 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:04:49.487 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.487 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:04:49.488 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.363192+0000", 2021-03-24T22:04:49.488 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:04:49.488 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.488 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.489 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:04:49.489 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.363192+0000", 2021-03-24T22:04:49.489 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2021-03-24T22:04:49.489 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.489 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.490 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:04:49.490 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.363190+0000", 2021-03-24T22:04:49.490 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8980000000000002e-06 2021-03-24T22:04:49.490 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.491 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.491 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:04:49.491 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.363199+0000", 2021-03-24T22:04:49.491 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.9709999999999999e-06 2021-03-24T22:04:49.491 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.492 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.492 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:04:49.492 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.363209+0000", 2021-03-24T22:04:49.492 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9600000000000006e-06 2021-03-24T22:04:49.493 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.493 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.493 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:04:49.493 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.363218+0000", 2021-03-24T22:04:49.494 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1470999999999994e-05 2021-03-24T22:04:49.494 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.494 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.494 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:04:49.495 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.363289+0000", 2021-03-24T22:04:49.495 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000128021 2021-03-24T22:04:49.495 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.495 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.495 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:04:49.496 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.363417+0000", 2021-03-24T22:04:49.496 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.090174975 2021-03-24T22:04:49.496 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.496 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.497 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3,7", 2021-03-24T22:04:49.497 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.453592+0000", 2021-03-24T22:04:49.497 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0083544299999999995 2021-03-24T22:04:49.497 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.497 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.498 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:04:49.498 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.461947+0000", 2021-03-24T22:04:49.498 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000299795 2021-03-24T22:04:49.498 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.499 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.499 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T22:04:49.499 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.462246+0000", 2021-03-24T22:04:49.500 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0025305150000000001 2021-03-24T22:04:49.500 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.500 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.500 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:04:49.500 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.464777+0000", 2021-03-24T22:04:49.501 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.7652999999999998e-05 2021-03-24T22:04:49.501 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.501 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.501 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:04:49.502 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.464844+0000", 2021-03-24T22:04:49.502 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3607000000000003e-05 2021-03-24T22:04:49.502 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.502 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.502 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:04:49.503 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.464908+0000", 2021-03-24T22:04:49.503 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.101716506 2021-03-24T22:04:49.503 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:04:49.503 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:04:49.504 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:04:49.504 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.504 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.504 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:6375 7.c 7:32974111:::smithi12556470-2:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e655)", 2021-03-24T22:04:49.504 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:03:45.363568+0000", 2021-03-24T22:04:49.505 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 63.919193644000003, 2021-03-24T22:04:49.505 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.101765479, 2021-03-24T22:04:49.505 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:04:49.505 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:04:49.506 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:04:49.506 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:04:49.506 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:04:49.506 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 6375 2021-03-24T22:04:49.506 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.507 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:04:49.507 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.507 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:04:49.507 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.363568+0000", 2021-03-24T22:04:49.508 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:04:49.508 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.508 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.508 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:04:49.508 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.363568+0000", 2021-03-24T22:04:49.509 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2021-03-24T22:04:49.509 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.509 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.509 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:04:49.510 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.363567+0000", 2021-03-24T22:04:49.510 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0079999999999999e-06 2021-03-24T22:04:49.510 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.510 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.511 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:04:49.511 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.363573+0000", 2021-03-24T22:04:49.511 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9039999999999998e-06 2021-03-24T22:04:49.511 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.511 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.512 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:04:49.512 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.363582+0000", 2021-03-24T22:04:49.512 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1290000000000001e-06 2021-03-24T22:04:49.512 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.513 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.513 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:04:49.513 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.363588+0000", 2021-03-24T22:04:49.513 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9103000000000006e-05 2021-03-24T22:04:49.513 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.514 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.514 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:04:49.514 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.363657+0000", 2021-03-24T22:04:49.514 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8739000000000001e-05 2021-03-24T22:04:49.515 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.515 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.515 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for blocked object", 2021-03-24T22:04:49.515 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.363716+0000", 2021-03-24T22:04:49.515 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0951070979999999 2021-03-24T22:04:49.516 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.516 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.516 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:04:49.516 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.458823+0000", 2021-03-24T22:04:49.517 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00020338800000000001 2021-03-24T22:04:49.517 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.517 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.517 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for rw locks", 2021-03-24T22:04:49.517 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.459026+0000", 2021-03-24T22:04:49.518 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0059478939999999996 2021-03-24T22:04:49.518 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.518 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.518 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:04:49.519 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.464974+0000", 2021-03-24T22:04:49.519 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016329100000000001 2021-03-24T22:04:49.519 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.519 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.519 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:04:49.520 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.465137+0000", 2021-03-24T22:04:49.520 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000196121 2021-03-24T22:04:49.520 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.520 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.520 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:04:49.521 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.465333+0000", 2021-03-24T22:04:49.521 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.101765479 2021-03-24T22:04:49.521 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:04:49.521 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:04:49.522 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:04:49.522 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.522 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.522 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:6381 7.19 7:9c4f4ea5:::smithi12556470-41:head [write 521248~563025] snapc 219=[219,218,217,214,211,210,20e,20d,209,204,1fd,1f8,1f3,1f0,1ef,1e1,1de,1dc,1d8,1d4,1cd,1bf,1bd,199,172,16e] ondisk+write+known_if_redirected e655)", 2021-03-24T22:04:49.523 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:03:45.373171+0000", 2021-03-24T22:04:49.523 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 63.909590145000003, 2021-03-24T22:04:49.523 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038785187999999998, 2021-03-24T22:04:49.523 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:04:49.523 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:04:49.524 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:04:49.524 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:04:49.525 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:04:49.525 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 6381 2021-03-24T22:04:49.526 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.526 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:04:49.527 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.527 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:04:49.528 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.373171+0000", 2021-03-24T22:04:49.528 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:04:49.528 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.529 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.529 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:04:49.530 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.373171+0000", 2021-03-24T22:04:49.530 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999957 2021-03-24T22:04:49.531 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.531 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.532 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:04:49.532 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.373167+0000", 2021-03-24T22:04:49.532 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00070376200000000003 2021-03-24T22:04:49.533 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.533 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.534 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:04:49.534 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.373871+0000", 2021-03-24T22:04:49.535 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9023999999999999e-05 2021-03-24T22:04:49.535 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.536 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.536 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:04:49.536 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.373890+0000", 2021-03-24T22:04:49.537 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.4950000000000002e-06 2021-03-24T22:04:49.537 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.538 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.538 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:04:49.539 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.373897+0000", 2021-03-24T22:04:49.539 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2100000000000005e-05 2021-03-24T22:04:49.540 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.540 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.540 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:04:49.540 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.373960+0000", 2021-03-24T22:04:49.541 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000102253 2021-03-24T22:04:49.541 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.541 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.541 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:04:49.542 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.374062+0000", 2021-03-24T22:04:49.542 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000207677 2021-03-24T22:04:49.542 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.542 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.542 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 2,7", 2021-03-24T22:04:49.543 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.374269+0000", 2021-03-24T22:04:49.543 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0017298019999999999 2021-03-24T22:04:49.543 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.543 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.544 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T22:04:49.544 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.375999+0000", 2021-03-24T22:04:49.544 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00028531999999999999 2021-03-24T22:04:49.544 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.544 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.545 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:04:49.545 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.376285+0000", 2021-03-24T22:04:49.545 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.035515931000000001 2021-03-24T22:04:49.545 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.546 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.546 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:04:49.546 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.411801+0000", 2021-03-24T22:04:49.546 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1810000000000006e-05 2021-03-24T22:04:49.546 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.547 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.547 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:04:49.547 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.411862+0000", 2021-03-24T22:04:49.547 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4266999999999997e-05 2021-03-24T22:04:49.548 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.548 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.548 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:04:49.548 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.411957+0000", 2021-03-24T22:04:49.548 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038785187999999998 2021-03-24T22:04:49.549 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:04:49.549 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:04:49.549 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:04:49.549 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.550 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.550 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:6382 7.19 7:9c4f4ea5:::smithi12556470-41:head [write 1809501~596300] snapc 219=[219,218,217,214,211,210,20e,20d,209,204,1fd,1f8,1f3,1f0,1ef,1e1,1de,1dc,1d8,1d4,1cd,1bf,1bd,199,172,16e] ondisk+write+known_if_redirected e655)", 2021-03-24T22:04:49.550 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:03:45.380415+0000", 2021-03-24T22:04:49.550 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 63.902346102999999, 2021-03-24T22:04:49.550 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031748200999999997, 2021-03-24T22:04:49.551 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:04:49.551 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:04:49.551 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:04:49.551 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:04:49.552 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:04:49.552 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 6382 2021-03-24T22:04:49.552 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.552 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:04:49.552 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.553 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:04:49.553 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.380415+0000", 2021-03-24T22:04:49.553 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:04:49.553 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.553 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.554 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:04:49.554 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.380415+0000", 2021-03-24T22:04:49.554 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2021-03-24T22:04:49.554 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.555 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.555 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:04:49.555 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.380414+0000", 2021-03-24T22:04:49.555 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00054294300000000005 2021-03-24T22:04:49.555 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.556 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.556 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:04:49.556 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.380957+0000", 2021-03-24T22:04:49.556 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3733e-05 2021-03-24T22:04:49.557 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.557 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.557 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:04:49.557 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.380970+0000", 2021-03-24T22:04:49.557 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1659999999999997e-06 2021-03-24T22:04:49.558 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.558 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.558 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:04:49.558 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.380978+0000", 2021-03-24T22:04:49.559 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6363000000000001e-05 2021-03-24T22:04:49.559 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.559 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.559 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:04:49.559 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.381054+0000", 2021-03-24T22:04:49.560 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010126799999999999 2021-03-24T22:04:49.560 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.560 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.560 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:04:49.561 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.381155+0000", 2021-03-24T22:04:49.561 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015574499999999999 2021-03-24T22:04:49.561 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.561 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.561 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 2,7", 2021-03-24T22:04:49.562 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.381311+0000", 2021-03-24T22:04:49.562 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0018026559999999999 2021-03-24T22:04:49.562 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.562 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.562 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T22:04:49.563 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.383114+0000", 2021-03-24T22:04:49.563 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00053586900000000002 2021-03-24T22:04:49.563 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.563 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.564 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:04:49.564 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.383650+0000", 2021-03-24T22:04:49.564 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028389675999999999 2021-03-24T22:04:49.564 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.565 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.565 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:04:49.565 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.412039+0000", 2021-03-24T22:04:49.565 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2992999999999998e-05 2021-03-24T22:04:49.565 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.566 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.566 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:04:49.566 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.412102+0000", 2021-03-24T22:04:49.566 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1424e-05 2021-03-24T22:04:49.566 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.567 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.567 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:04:49.567 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.412164+0000", 2021-03-24T22:04:49.567 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031748200999999997 2021-03-24T22:04:49.568 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:04:49.568 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:04:49.568 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:04:49.568 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.568 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.569 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:6383 7.19 7:9c4f4ea5:::smithi12556470-41:head [setxattr _header (58) in=7b,truncate 2521248] snapc 219=[219,218,217,214,211,210,20e,20d,209,204,1fd,1f8,1f3,1f0,1ef,1e1,1de,1dc,1d8,1d4,1cd,1bf,1bd,199,172,16e] ondisk+write+known_if_redirected e655)", 2021-03-24T22:04:49.569 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:03:45.380992+0000", 2021-03-24T22:04:49.569 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 63.901769170999998, 2021-03-24T22:04:49.569 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031333936999999999, 2021-03-24T22:04:49.570 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:04:49.570 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:04:49.570 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:04:49.570 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:04:49.570 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:04:49.571 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 6383 2021-03-24T22:04:49.571 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.571 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:04:49.571 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.572 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:04:49.572 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.380992+0000", 2021-03-24T22:04:49.572 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:04:49.572 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.572 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.573 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:04:49.573 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.380992+0000", 2021-03-24T22:04:49.573 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2021-03-24T22:04:49.573 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.573 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.574 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:04:49.574 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.380991+0000", 2021-03-24T22:04:49.574 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6360000000000006e-06 2021-03-24T22:04:49.574 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.575 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.575 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:04:49.575 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.381001+0000", 2021-03-24T22:04:49.575 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.268e-06 2021-03-24T22:04:49.575 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.576 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.576 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:04:49.576 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.381007+0000", 2021-03-24T22:04:49.576 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.7940000000000002e-06 2021-03-24T22:04:49.577 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.577 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.577 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:04:49.577 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.381012+0000", 2021-03-24T22:04:49.577 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001391268 2021-03-24T22:04:49.578 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.578 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.578 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:04:49.578 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.382403+0000", 2021-03-24T22:04:49.578 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011455600000000001 2021-03-24T22:04:49.579 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.579 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.579 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:04:49.579 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.382518+0000", 2021-03-24T22:04:49.580 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014872299999999999 2021-03-24T22:04:49.580 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.580 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.580 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 2,7", 2021-03-24T22:04:49.580 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.382666+0000", 2021-03-24T22:04:49.581 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00060959899999999995 2021-03-24T22:04:49.581 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.581 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.581 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T22:04:49.582 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.383276+0000", 2021-03-24T22:04:49.582 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00048841800000000001 2021-03-24T22:04:49.582 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.582 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.582 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:04:49.583 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.383764+0000", 2021-03-24T22:04:49.583 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028443677000000001 2021-03-24T22:04:49.583 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.583 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.583 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:04:49.584 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.412208+0000", 2021-03-24T22:04:49.584 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.3863000000000003e-05 2021-03-24T22:04:49.584 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.585 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.586 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:04:49.586 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.412252+0000", 2021-03-24T22:04:49.587 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.4312000000000003e-05 2021-03-24T22:04:49.587 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.588 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.588 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:04:49.588 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.412326+0000", 2021-03-24T22:04:49.589 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031333936999999999 2021-03-24T22:04:49.589 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:04:49.590 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:04:49.590 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:04:49.591 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.591 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.591 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:6384 7.19 7:9c4f4ea5:::smithi12556470-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e655)", 2021-03-24T22:04:49.592 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:03:45.381047+0000", 2021-03-24T22:04:49.592 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 63.901714501000001, 2021-03-24T22:04:49.593 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031537978000000001, 2021-03-24T22:04:49.593 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:04:49.593 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:04:49.594 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:04:49.595 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:04:49.595 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:04:49.595 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 6384 2021-03-24T22:04:49.596 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.596 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:04:49.597 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.597 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:04:49.598 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.381047+0000", 2021-03-24T22:04:49.598 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:04:49.598 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.599 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.600 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:04:49.600 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.381047+0000", 2021-03-24T22:04:49.600 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2021-03-24T22:04:49.601 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.601 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.602 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:04:49.602 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.381046+0000", 2021-03-24T22:04:49.603 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.633e-06 2021-03-24T22:04:49.603 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.603 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.604 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:04:49.605 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.381049+0000", 2021-03-24T22:04:49.606 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9010000000000002e-06 2021-03-24T22:04:49.607 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.607 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.607 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:04:49.608 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.381054+0000", 2021-03-24T22:04:49.608 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8729999999999998e-06 2021-03-24T22:04:49.608 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.608 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.609 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:04:49.609 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.381059+0000", 2021-03-24T22:04:49.609 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0020873979999999999 2021-03-24T22:04:49.609 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.610 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.610 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:04:49.610 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.383146+0000", 2021-03-24T22:04:49.610 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8003000000000004e-05 2021-03-24T22:04:49.610 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.611 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.611 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for rw locks", 2021-03-24T22:04:49.611 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.383234+0000", 2021-03-24T22:04:49.611 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029153312000000001 2021-03-24T22:04:49.612 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.612 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.612 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:04:49.612 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.412388+0000", 2021-03-24T22:04:49.612 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2711000000000001e-05 2021-03-24T22:04:49.613 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.613 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.613 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:04:49.613 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.412480+0000", 2021-03-24T22:04:49.614 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000104707 2021-03-24T22:04:49.614 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.614 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.615 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:04:49.615 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.412585+0000", 2021-03-24T22:04:49.615 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031537978000000001 2021-03-24T22:04:49.616 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:04:49.616 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:04:49.616 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:04:49.616 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.617 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.617 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:6433 7.19 7:992cb487:::smithi12556470-5:head [copy-from ver 722] snapc 21e=[21e,21d,21b,219,218,217,214,211,210,20e,20d,209,204,1f8,1f3,1f0,1ef,1e1,1de,1dc,1d8,1d4,1cd,1bf,199,172,16e] ondisk+write+known_if_redirected e660)", 2021-03-24T22:04:49.617 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:03:50.352034+0000", 2021-03-24T22:04:49.618 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 58.930727681999997, 2021-03-24T22:04:49.618 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028079139999999999, 2021-03-24T22:04:49.618 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:04:49.618 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:04:49.619 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:04:49.619 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:04:49.619 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:04:49.619 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 6433 2021-03-24T22:04:49.620 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.620 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:04:49.620 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.620 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:04:49.620 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:50.352034+0000", 2021-03-24T22:04:49.621 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:04:49.621 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.621 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.621 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:04:49.622 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:50.352034+0000", 2021-03-24T22:04:49.622 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2021-03-24T22:04:49.622 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.622 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.622 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:04:49.623 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:50.352032+0000", 2021-03-24T22:04:49.623 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.9539999999999999e-06 2021-03-24T22:04:49.623 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.623 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.624 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:04:49.624 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:50.352042+0000", 2021-03-24T22:04:49.624 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7957000000000001e-05 2021-03-24T22:04:49.624 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.624 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.625 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:04:49.625 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:50.352059+0000", 2021-03-24T22:04:49.625 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2164999999999999e-05 2021-03-24T22:04:49.626 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.626 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.626 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:04:49.626 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:50.352082+0000", 2021-03-24T22:04:49.627 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2281000000000006e-05 2021-03-24T22:04:49.627 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.627 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.627 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:04:49.627 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:50.352174+0000", 2021-03-24T22:04:49.628 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0843999999999999e-05 2021-03-24T22:04:49.628 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.628 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.629 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op must wait for map", 2021-03-24T22:04:49.629 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:50.352195+0000", 2021-03-24T22:04:49.629 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0067887379999999999 2021-03-24T22:04:49.629 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.630 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.630 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:04:49.630 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:50.358984+0000", 2021-03-24T22:04:49.630 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000644741 2021-03-24T22:04:49.630 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.631 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.631 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:04:49.631 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:50.359628+0000", 2021-03-24T22:04:49.631 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0097069270000000006 2021-03-24T22:04:49.632 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.632 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.632 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 2,7", 2021-03-24T22:04:49.632 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:50.369335+0000", 2021-03-24T22:04:49.632 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0073280189999999999 2021-03-24T22:04:49.633 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.633 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.633 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:04:49.633 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:50.376663+0000", 2021-03-24T22:04:49.634 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0016268229999999999 2021-03-24T22:04:49.634 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.634 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.634 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:04:49.635 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:50.378290+0000", 2021-03-24T22:04:49.635 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001720823 2021-03-24T22:04:49.635 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.635 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.636 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T22:04:49.636 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:50.380011+0000", 2021-03-24T22:04:49.636 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.6569000000000001e-05 2021-03-24T22:04:49.636 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.637 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.637 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:04:49.637 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:50.380057+0000", 2021-03-24T22:04:49.638 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5569000000000003e-05 2021-03-24T22:04:49.638 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.639 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.639 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:04:49.639 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:50.380113+0000", 2021-03-24T22:04:49.640 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028079139999999999 2021-03-24T22:04:49.640 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:04:49.641 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:04:49.641 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:04:49.642 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.642 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.642 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:6434 7.19 7:992cb487:::smithi12556470-5:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e660)", 2021-03-24T22:04:49.643 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:03:50.352116+0000", 2021-03-24T22:04:49.643 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 58.930645435000002, 2021-03-24T22:04:49.644 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028689512, 2021-03-24T22:04:49.644 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:04:49.645 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:04:49.645 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:04:49.645 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:04:49.645 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:04:49.646 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 6434 2021-03-24T22:04:49.646 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.646 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:04:49.646 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.647 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:04:49.647 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:50.352116+0000", 2021-03-24T22:04:49.647 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:04:49.647 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.647 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.648 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:04:49.648 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:50.352116+0000", 2021-03-24T22:04:49.648 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2021-03-24T22:04:49.648 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.649 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.649 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:04:49.649 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:50.352115+0000", 2021-03-24T22:04:49.649 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000109635 2021-03-24T22:04:49.650 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.650 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.650 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:04:49.650 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:50.352225+0000", 2021-03-24T22:04:49.650 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6699999999999997e-06 2021-03-24T22:04:49.651 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.651 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.651 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:04:49.651 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:50.352232+0000", 2021-03-24T22:04:49.652 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9370000000000003e-06 2021-03-24T22:04:49.652 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.652 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.652 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:04:49.652 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:50.352237+0000", 2021-03-24T22:04:49.653 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0076719939999999997 2021-03-24T22:04:49.653 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.653 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.653 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:04:49.653 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:50.359909+0000", 2021-03-24T22:04:49.654 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0176999999999997e-05 2021-03-24T22:04:49.654 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.654 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.654 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for blocked object", 2021-03-24T22:04:49.655 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:50.359959+0000", 2021-03-24T22:04:49.655 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.016531852 2021-03-24T22:04:49.655 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.655 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.655 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:04:49.656 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:50.376491+0000", 2021-03-24T22:04:49.656 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000116177 2021-03-24T22:04:49.656 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.656 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.657 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for rw locks", 2021-03-24T22:04:49.657 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:50.376607+0000", 2021-03-24T22:04:49.657 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.003544527 2021-03-24T22:04:49.657 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.657 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.658 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:04:49.658 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:50.380151+0000", 2021-03-24T22:04:49.658 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011042199999999999 2021-03-24T22:04:49.658 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.658 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.659 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:04:49.659 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:50.380262+0000", 2021-03-24T22:04:49.659 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00054389800000000002 2021-03-24T22:04:49.659 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.660 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.660 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:04:49.660 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:50.380806+0000", 2021-03-24T22:04:49.660 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028689512 2021-03-24T22:04:49.660 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:04:49.661 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:04:49.661 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:04:49.661 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.661 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.661 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:6732 7.c 7:305739dc:::smithi12556470-36:head [write 645633~716191] snapc 246=[246,244,243,242,240,23f,23d,23b,236,235,234,231,22d,22c,226,225,21f,21d,21b,217,210,209,204,1f8,1ef,1e1,1de,1dc,1d8,1bf,199,172,16e] ondisk+write+known_if_redirected e700)", 2021-03-24T22:04:49.662 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:04:30.387337+0000", 2021-03-24T22:04:49.662 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 18.895424066, 2021-03-24T22:04:49.662 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.03521146, 2021-03-24T22:04:49.662 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:04:49.663 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:04:49.663 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:04:49.663 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:04:49.663 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:04:49.663 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 6732 2021-03-24T22:04:49.664 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.664 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:04:49.664 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.664 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:04:49.665 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:30.387337+0000", 2021-03-24T22:04:49.665 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:04:49.665 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.665 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.665 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:04:49.666 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:30.387337+0000", 2021-03-24T22:04:49.666 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999933 2021-03-24T22:04:49.666 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.666 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.666 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:04:49.667 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:30.387331+0000", 2021-03-24T22:04:49.667 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011328670000000001 2021-03-24T22:04:49.667 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.667 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.668 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:04:49.668 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:30.388464+0000", 2021-03-24T22:04:49.668 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9536e-05 2021-03-24T22:04:49.668 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.668 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.669 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:04:49.669 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:30.388493+0000", 2021-03-24T22:04:49.669 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2812e-05 2021-03-24T22:04:49.669 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.669 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.670 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:04:49.670 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:30.388506+0000", 2021-03-24T22:04:49.670 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6872000000000004e-05 2021-03-24T22:04:49.670 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.671 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.671 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:04:49.671 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:30.388603+0000", 2021-03-24T22:04:49.671 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018648100000000001 2021-03-24T22:04:49.671 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.672 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.672 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:04:49.672 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:30.388789+0000", 2021-03-24T22:04:49.672 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00032753099999999999 2021-03-24T22:04:49.673 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.673 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.673 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3,7", 2021-03-24T22:04:49.673 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:30.389117+0000", 2021-03-24T22:04:49.673 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0047535099999999999 2021-03-24T22:04:49.674 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.674 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.674 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T22:04:49.674 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:30.393870+0000", 2021-03-24T22:04:49.675 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011734843999999999 2021-03-24T22:04:49.675 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.675 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.675 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:04:49.675 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:30.405605+0000", 2021-03-24T22:04:49.676 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.016723930000000001 2021-03-24T22:04:49.676 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.676 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.676 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:04:49.676 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:30.422329+0000", 2021-03-24T22:04:49.677 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000112357 2021-03-24T22:04:49.677 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.677 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.677 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:04:49.678 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:30.422442+0000", 2021-03-24T22:04:49.678 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000107398 2021-03-24T22:04:49.678 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.678 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.678 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:04:49.679 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:30.422549+0000", 2021-03-24T22:04:49.679 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.03521146 2021-03-24T22:04:49.679 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:04:49.679 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:04:49.679 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:04:49.680 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.680 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.680 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:6733 7.c 7:305739dc:::smithi12556470-36:head [write 1927893~606315] snapc 246=[246,244,243,242,240,23f,23d,23b,236,235,234,231,22d,22c,226,225,21f,21d,21b,217,210,209,204,1f8,1ef,1e1,1de,1dc,1d8,1bf,199,172,16e] ondisk+write+known_if_redirected e700)", 2021-03-24T22:04:49.680 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:04:30.395133+0000", 2021-03-24T22:04:49.681 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 18.887628643999999, 2021-03-24T22:04:49.681 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027654464, 2021-03-24T22:04:49.681 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:04:49.681 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:04:49.681 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:04:49.682 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:04:49.682 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:04:49.682 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 6733 2021-03-24T22:04:49.682 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.683 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:04:49.683 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.683 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:04:49.683 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:30.395133+0000", 2021-03-24T22:04:49.683 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:04:49.684 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.684 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.684 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:04:49.684 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:30.395133+0000", 2021-03-24T22:04:49.684 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999952 2021-03-24T22:04:49.685 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.685 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.685 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:04:49.685 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:30.395128+0000", 2021-03-24T22:04:49.686 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00058171299999999998 2021-03-24T22:04:49.686 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.686 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.686 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:04:49.686 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:30.395710+0000", 2021-03-24T22:04:49.687 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5449999999999999e-05 2021-03-24T22:04:49.687 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.687 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.687 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:04:49.688 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:30.395735+0000", 2021-03-24T22:04:49.688 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1114000000000001e-05 2021-03-24T22:04:49.688 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.688 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.688 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:04:49.689 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:30.395747+0000", 2021-03-24T22:04:49.689 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000101451 2021-03-24T22:04:49.689 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.689 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.689 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:04:49.690 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:30.395848+0000", 2021-03-24T22:04:49.690 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011859200000000001 2021-03-24T22:04:49.690 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.690 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.691 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:04:49.691 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:30.395967+0000", 2021-03-24T22:04:49.691 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00019426400000000001 2021-03-24T22:04:49.691 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.691 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.692 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3,7", 2021-03-24T22:04:49.692 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:30.396161+0000", 2021-03-24T22:04:49.692 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0017338799999999999 2021-03-24T22:04:49.692 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.692 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.693 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T22:04:49.693 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:30.397895+0000", 2021-03-24T22:04:49.693 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0092778889999999992 2021-03-24T22:04:49.693 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.694 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.694 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:04:49.694 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:30.407173+0000", 2021-03-24T22:04:49.694 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.015458965 2021-03-24T22:04:49.694 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.695 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.695 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:04:49.695 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:30.422632+0000", 2021-03-24T22:04:49.695 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4376000000000007e-05 2021-03-24T22:04:49.696 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.696 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.696 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:04:49.696 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:30.422716+0000", 2021-03-24T22:04:49.696 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1388000000000002e-05 2021-03-24T22:04:49.697 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:04:49.697 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:04:49.697 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:04:49.697 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:30.422787+0000", 2021-03-24T22:04:49.697 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027654464 2021-03-24T22:04:49.698 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:04:49.698 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:04:49.698 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:04:49.698 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:04:49.699 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:04:49.699 INFO:teuthology.orchestra.run.smithi142.stdout:} 2021-03-24T22:04:49.699 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:49.701 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:49 smithi142 conmon[20125]: cluster 2021-03-24T22:04:48. 2021-03-24T22:04:49.701 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:49 smithi142 conmon[20125]: 427009+0000 mon.a (mon.0) 1869 : cluster [DBG] osdmap e718: 8 total, 7 up, 8 in 2021-03-24T22:04:49.701 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:49 smithi142 conmon[20125]: cluster 2021-03-24T22:04:49. 2021-03-24T22:04:49.701 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:49 smithi142 conmon[20125]: 428440+0000 mon.a (mon.0) 1870 : cluster [DBG] osdmap e719: 8 total, 7 up, 8 in 2021-03-24T22:04:49.702 INFO:tasks.rados.rados.0.smithi125.stdout:2743: done (6 left) 2021-03-24T22:04:49.703 INFO:tasks.rados.rados.0.smithi125.stdout:2744: done (5 left) 2021-03-24T22:04:49.703 INFO:tasks.rados.rados.0.smithi125.stdout:2745: done (4 left) 2021-03-24T22:04:49.703 INFO:tasks.rados.rados.0.smithi125.stdout:2747: done (3 left) 2021-03-24T22:04:49.703 INFO:tasks.rados.rados.0.smithi125.stdout:2748: done (2 left) 2021-03-24T22:04:49.703 INFO:tasks.rados.rados.0.smithi125.stdout:2750: done (1 left) 2021-03-24T22:04:49.704 INFO:tasks.rados.rados.0.smithi125.stdout:2751: done (0 left) 2021-03-24T22:04:49.704 INFO:tasks.rados.rados.0.smithi125.stdout:2752: delete oid 40 current snap is 316 2021-03-24T22:04:49.704 INFO:tasks.rados.rados.0.smithi125.stdout:2752: done (0 left) 2021-03-24T22:04:49.704 INFO:tasks.rados.rados.0.smithi125.stdout:2753: read oid 22 snap 297 2021-03-24T22:04:49.705 INFO:tasks.rados.rados.0.smithi125.stdout:2753: expect deleted 2021-03-24T22:04:49.705 INFO:tasks.rados.rados.0.smithi125.stdout:2754: read oid 18 snap 305 2021-03-24T22:04:49.705 INFO:tasks.rados.rados.0.smithi125.stdout:2754: expect (ObjNum 861 snap 279 seq_num 861) 2021-03-24T22:04:49.705 INFO:tasks.rados.rados.0.smithi125.stdout:2755: snap_remove snap 192 2021-03-24T22:04:49.706 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[19936]: cluster 2021-03-24T22:04:49.706 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[19936]: 2021-03-24T22:04:49.706 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[19936]: 2021 2021-03-24T22:04:49.707 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[19936]: - 2021-03-24T22:04:49.707 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[19936]: 03-24T22:04:48.427009+0000 mon.a (mon.0) 1869 : cluster [DBG] osdmap e718: 8 total, 7 up, 8 in 2021-03-24T22:04:49.707 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[19936]: cluster 2021-03-24T22:04:49.428440+0000 mon.a (mon.0) 1870 : cluster [DBG] osdmap e719: 8 total, 7 up, 8 in 2021-03-24T22:04:49.708 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: cluster 2021-03-24T22:04:49.708 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: 2021-03-24T22:04:49.708 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: 2021 2021-03-24T22:04:49.708 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: - 2021-03-24T22:04:49.709 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: 03 2021-03-24T22:04:49.709 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: - 2021-03-24T22:04:49.709 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: 24 2021-03-24T22:04:49.709 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: T 2021-03-24T22:04:49.709 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: 22 2021-03-24T22:04:49.710 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: : 2021-03-24T22:04:49.710 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: 04 2021-03-24T22:04:49.710 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: : 2021-03-24T22:04:49.710 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: 48 2021-03-24T22:04:49.711 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: . 2021-03-24T22:04:49.711 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: 427009 2021-03-24T22:04:49.711 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: +0000 2021-03-24T22:04:49.711 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: 2021-03-24T22:04:49.711 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: mon.a 2021-03-24T22:04:49.712 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: ( 2021-03-24T22:04:49.712 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: mon 2021-03-24T22:04:49.712 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: . 2021-03-24T22:04:49.712 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: 0 2021-03-24T22:04:49.712 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: ) 2021-03-24T22:04:49.713 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: 1869 2021-03-24T22:04:49.713 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: : 2021-03-24T22:04:49.713 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: cluster 2021-03-24T22:04:49.713 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: 2021-03-24T22:04:49.714 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: [DBG] 2021-03-24T22:04:49.714 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: 2021-03-24T22:04:49.714 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: osdmap e718: 8 total, 7 up, 8 in 2021-03-24T22:04:49.714 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: 2021-03-24T22:04:49.714 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: cluster 2021-03-24T22:04:49.715 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: 2021-03-24T22:04:49.715 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: 2021 2021-03-24T22:04:49.715 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: - 2021-03-24T22:04:49.715 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: 03 2021-03-24T22:04:49.716 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: - 2021-03-24T22:04:49.716 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: 24 2021-03-24T22:04:49.716 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: T 2021-03-24T22:04:49.716 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: 22 2021-03-24T22:04:49.716 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: : 2021-03-24T22:04:49.717 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: 04 2021-03-24T22:04:49.717 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: : 2021-03-24T22:04:49.717 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: 49 2021-03-24T22:04:49.717 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: . 2021-03-24T22:04:49.717 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: 428440 2021-03-24T22:04:49.718 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: +0000 2021-03-24T22:04:49.718 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: 2021-03-24T22:04:49.718 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: mon.a 2021-03-24T22:04:49.718 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: ( 2021-03-24T22:04:49.719 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: mon 2021-03-24T22:04:49.719 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: . 2021-03-24T22:04:49.719 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:49 smithi125 conmon[24927]: 0) 1870 : cluster [DBG] osdmap e719: 8 total, 7 up, 8 in 2021-03-24T22:04:49.774 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:49.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:49.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:49.880 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:49.937 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:49.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:49.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:50.042 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:50.098 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:50.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:50.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:04:50.205 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:50.263 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:50.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:50.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:50.340 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.7 dump_ops_in_flight 2021-03-24T22:04:50.371 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:50.387 INFO:tasks.rados.rados.0.smithi125.stdout:2755: done (2 left) 2021-03-24T22:04:50.388 INFO:tasks.rados.rados.0.smithi125.stdout:2756: snap_remove snap 295 2021-03-24T22:04:50.389 INFO:teuthology.orchestra.run.smithi125.stderr:nodeep-scrub is set 2021-03-24T22:04:50.465 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:50.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:50.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:50.575 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:50.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:50 smithi142 conmon[20125]: cluster 2021-03-24T22:04:48.963541+0000 mgr.y (mgr.14144) 634 2021-03-24T22:04:50.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:50 smithi142 conmon[20125]: : cluster [DBG] pgmap v1240: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 107 active+clean; 673 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 6.7 MiB/s rd, 476 KiB/s wr, 9 op/s; 154/1920 objects degraded (8.021%); 40/1920 objects misplaced (2.083%) 2021-03-24T22:04:50.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:50 smithi142 conmon[20125]: audit 2021-03-24T22:04:49.705722+0000 mon.a (mon.0) 1871 : audit [INF] from='client.? v1:172.21.15.125:0/1903120710' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:04:50.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:50 smithi142 conmon[20125]: cluster 2021-03-24T22:04:50.365998+0000 2021-03-24T22:04:50.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:50 smithi142 conmon[20125]: mon.a (mon.0) 1872 : cluster [WRN] Health check update: Degraded data redundancy: 154/1920 objects degraded (8.021%), 19 pgs degraded (PG_DEGRADED) 2021-03-24T22:04:50.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:50 smithi142 conmon[20125]: cluster 2021-03-24T22:04:50.367373+0000 mon.a (mon.0) 1873 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:04:50.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:50 smithi142 conmon[20125]: audit 2021-03-24T22:04:50.376729+0000 mon.a (mon.0) 1874 : audit [INF] from='client.? v1:172.21.15.125:0/1903120710' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T22:04:50.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:50 smithi142 conmon[20125]: cluster 2021-03-24T22:04:50.376877+0000 mon.a (mon 2021-03-24T22:04:50.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:50 smithi142 conmon[20125]: .0) 1875 : cluster [DBG] osdmap e720: 8 total, 7 up, 8 in 2021-03-24T22:04:50.620 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:50.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:50.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:50.662 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T22:04:50.726 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:50.739 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:50 smithi125 conmon[24927]: cluster 2021-03-24T22:04:48. 2021-03-24T22:04:50.740 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:50 smithi125 conmon[24927]: 963541+0000 mgr.y (mgr.14144) 634 : cluster [DBG] pgmap v1240: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 107 active+clean; 673 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 6.7 MiB/s rd, 476 KiB/s wr, 9 op/s; 154/1920 objects degraded (8.021%); 40/1920 objects misplaced (2.083%) 2021-03-24T22:04:50.740 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:50 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T22:04:50.740 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:50 smithi125 conmon[24927]: 24T22:04:49.705722+0000 mon.a (mon.0) 1871 : audit 2021-03-24T22:04:50.741 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:50 smithi125 conmon[24927]: [INF] from='client.? v1:172.21.15.125:0/1903120710' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:04:50.741 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:50 smithi125 conmon[24927]: cluster 2021-03-24T22:04:50.741 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:50 smithi125 conmon[24927]: 2021-03-24T22:04:50.365998+0000 mon.a 2021-03-24T22:04:50.741 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:50 smithi125 conmon[24927]: (mon.0) 1872 : cluster [WRN] Health check update: Degraded data redundancy: 154/1920 objects degraded (8.021%), 19 pgs degraded (PG_DEGRADED) 2021-03-24T22:04:50.741 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:50 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:04:50.742 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:50 smithi125 conmon[24927]: 24T22:04:50.367373+0000 mon.a 2021-03-24T22:04:50.742 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:50 smithi125 conmon[24927]: (mon.0) 1873 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:04:50.742 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:50 smithi125 conmon[24927]: audit 2021-03 2021-03-24T22:04:50.742 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:50 smithi125 conmon[24927]: -24T22:04:50.376729+0000 2021-03-24T22:04:50.743 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:50 smithi125 conmon[24927]: mon.a (mon.0) 1874 : audit [INF] from='client.? v1:172.21.15.125:0/1903120710' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T22:04:50.743 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:50 smithi125 conmon[24927]: 2021-03-24T22:04:50.743 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:50 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:04:50.743 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:50 smithi125 conmon[24927]: 03-24T22:04:50.376877+0000 mon.a (mon 2021-03-24T22:04:50.743 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:50 smithi125 conmon[24927]: .0) 1875 : cluster [DBG] osdmap e720: 8 total, 7 up, 8 in 2021-03-24T22:04:50.744 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:50 smithi125 conmon[19936]: cluster 2021-03-24T22:04:48.963541+0000 mgr.y 2021-03-24T22:04:50.744 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:50 smithi125 conmon[19936]: (mgr.14144) 634 : cluster [DBG] pgmap v1240: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 107 active+clean; 673 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 6.7 MiB/s rd, 476 KiB/s wr, 9 op/s; 154/1920 objects degraded (8.021%); 40/1920 objects misplaced (2.083%) 2021-03-24T22:04:50.744 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:50 smithi125 conmon[19936]: audit 2021-03-24T22:04:49.705722+0000 mon.a (mon.0) 1871 : audit [INF] 2021-03-24T22:04:50.746 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:50 smithi125 conmon[19936]: from='client.? v1:172.21.15.125:0/1903120710' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:04:50.746 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:50 smithi125 conmon[19936]: cluster 2021-03-24T22:04: 2021-03-24T22:04:50.747 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:50 smithi125 conmon[19936]: 50.365998+0000 mon.a (mon.0) 1872 : cluster [WRN] Health check update: Degraded data redundancy: 154/1920 objects degraded (8.021%), 19 pgs degraded (PG_DEGRADED) 2021-03-24T22:04:50.747 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:50 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T22:04:50.747 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:50 smithi125 conmon[19936]: -24T22:04:50.367373+0000 mon.a (mon 2021-03-24T22:04:50.747 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:50 smithi125 conmon[19936]: .0) 1873 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:04:50.748 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:50 smithi125 conmon[19936]: audit 2021-03-24 2021-03-24T22:04:50.748 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:50 smithi125 conmon[19936]: T22:04:50.376729+0000 mon.a (mon. 2021-03-24T22:04:50.748 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:50 smithi125 conmon[19936]: 0) 1874 : audit [INF] from='client.? v1:172.21.15.125:0/1903120710' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T22:04:50.748 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:50 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T22:04:50.749 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:50 smithi125 conmon[19936]: 22:04:50.376877+0000 mon.a (mon.0 2021-03-24T22:04:50.749 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:50 smithi125 conmon[19936]: ) 1875 : cluster [DBG] osdmap e720: 8 total, 7 up, 8 in 2021-03-24T22:04:50.789 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:50.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:50.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:04:50.894 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:50.954 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:50.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:50.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:51.062 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:51.119 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:51.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:51.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:51.226 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:51.285 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:51.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:51.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:51.391 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:51.394 INFO:tasks.rados.rados.0.smithi125.stdout:2756: done (2 left) 2021-03-24T22:04:51.395 INFO:tasks.rados.rados.0.smithi125.stdout:2757: delete oid 45 current snap is 316 2021-03-24T22:04:51.395 INFO:tasks.rados.rados.0.smithi125.stdout:2754: expect (ObjNum 861 snap 279 seq_num 861) 2021-03-24T22:04:51.487 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:51.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:51.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:04:51.597 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:51.617 INFO:tasks.rados.rados.0.smithi125.stdout:2753: done (2 left) 2021-03-24T22:04:51.618 INFO:tasks.rados.rados.0.smithi125.stdout:2754: done (1 left) 2021-03-24T22:04:51.618 INFO:tasks.rados.rados.0.smithi125.stdout:2757: done (0 left) 2021-03-24T22:04:51.618 INFO:tasks.rados.rados.0.smithi125.stdout:2758: read oid 28 snap -1 2021-03-24T22:04:51.618 INFO:tasks.rados.rados.0.smithi125.stdout:2758: expect (ObjNum 718 snap 229 seq_num 718) 2021-03-24T22:04:51.619 INFO:tasks.rados.rados.0.smithi125.stdout:2759: write (excl) oid 23 current snap is 316 2021-03-24T22:04:51.619 INFO:tasks.rados.rados.0.smithi125.stdout:2759: seq_num 956 ranges {467492=482169,1626315=441177} 2021-03-24T22:04:51.632 INFO:tasks.rados.rados.0.smithi125.stdout:2759: writing smithi12556470-23 from 467492 to 949661 tid 1 2021-03-24T22:04:51.638 INFO:tasks.rados.rados.0.smithi125.stdout:2759: writing smithi12556470-23 from 1626315 to 2067492 tid 2 2021-03-24T22:04:51.638 INFO:tasks.rados.rados.0.smithi125.stdout:2760: snap_remove snap 282 2021-03-24T22:04:51.647 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:51.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:51.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:04:51.752 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:51.813 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:51.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:51.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:51.919 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:51.979 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:51.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:51.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:52.043 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T22:04:52.066 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T22:04:52.085 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:52.089 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T22:04:52.112 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T22:04:52.138 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T22:04:52.140 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:52.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:52.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:52.164 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T22:04:52.187 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T22:04:52.188 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-24T22:04:52.188 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-24T22:04:52.189 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-24T22:04:52.189 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-24T22:04:52.189 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-24T22:04:52.189 INFO:teuthology.orchestra.run.smithi125.stderr:Error ENXIO: problem getting command descriptions from osd.5 2021-03-24T22:04:52.189 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-24T22:04:52.190 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-24T22:04:52.249 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:52.310 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:52.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:52.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:52.415 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:52.472 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:52.477 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:52 smithi125 conmon[19936]: cluster 2021-03-24T22:04 2021-03-24T22:04:52.478 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:52 smithi125 conmon[19936]: :50.964459+0000 mgr.y (mgr.14144) 635 : cluster [DBG] pgmap v1243: 168 pgs: 1 peering, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 107 active+clean; 700 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 4.9 MiB/s rd, 5.2 MiB/s wr, 23 op/s; 160/1938 objects degraded (8.256%); 20/1938 objects misplaced (1.032%); 395 B/s, 4 objects/s recovering 2021-03-24T22:04:52.478 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:52 smithi125 conmon[19936]: cluster 2021-03-24T22:04:52.480 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:52 smithi125 conmon[19936]: 2021-03-24T22:04:51.376675+0000 mon.a (mon. 2021-03-24T22:04:52.480 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:52 smithi125 conmon[19936]: 0) 1876 : cluster [DBG] osdmap e721: 8 total, 7 up, 8 in 2021-03-24T22:04:52.481 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:52 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:04:52.481 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:52 smithi125 conmon[19936]: 24T22:04:51. 2021-03-24T22:04:52.481 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:52 smithi125 conmon[19936]: 491950+0000 mon.a ( 2021-03-24T22:04:52.481 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:52 smithi125 conmon[19936]: mon.0) 1877 : cluster [INF] 2021-03-24T22:04:52.482 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:52 smithi125 conmon[19936]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2021-03-24T22:04:52.482 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:52 smithi125 conmon[19936]: audit 2021-03-24T22:04: 2021-03-24T22:04:52.482 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:52 smithi125 conmon[19936]: 52.014790+0000 mon.a (mon.0) 1878 : audit [DBG] 2021-03-24T22:04:52.483 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:52 smithi125 conmon[19936]: from='client.? v1:172.21.15.125:0/4202212003' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:04:52.484 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:52 smithi125 conmon[24927]: cluster 2021-03-24T22:04: 2021-03-24T22:04:52.484 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:52 smithi125 conmon[24927]: 50.964459+0000 mgr.y (mgr.14144) 635 : cluster [DBG] pgmap v1243: 168 pgs: 1 peering, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 107 active+clean; 700 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 4.9 MiB/s rd, 5.2 MiB/s wr, 23 op/s; 160/1938 objects degraded (8.256%); 20/1938 objects misplaced (1.032%); 395 B/s, 4 objects/s recovering 2021-03-24T22:04:52.485 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:52 smithi125 conmon[24927]: cluster 2021-03-24T22:04:52.486 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:52 smithi125 conmon[24927]: 2021-03-24T22:04: 2021-03-24T22:04:52.486 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:52 smithi125 conmon[24927]: 51.376675+0000 mon.a ( 2021-03-24T22:04:52.486 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:52 smithi125 conmon[24927]: mon.0) 1876 : cluster [DBG] osdmap e721: 8 total, 7 up, 8 in 2021-03-24T22:04:52.487 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:52 smithi125 conmon[24927]: cluster 2021-03-24T22:04:52.487 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:52 smithi125 conmon[24927]: 2021-03-24T22:04: 2021-03-24T22:04:52.487 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:52 smithi125 conmon[24927]: 51.491950+0000 mon.a (mon.0) 1877 2021-03-24T22:04:52.488 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:52 smithi125 conmon[24927]: : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2021-03-24T22:04:52.488 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:52 smithi125 conmon[24927]: audit 2021-03-24T22:04:52.488 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:52 smithi125 conmon[24927]: 2021-03-24T22 2021-03-24T22:04:52.489 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:52 smithi125 conmon[24927]: :04:52.014790+0000 mon.a 2021-03-24T22:04:52.489 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:52 smithi125 conmon[24927]: (mon.0) 1878 : audit [DBG] from='client.? v1:172.21.15.125:0/4202212003' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:04:52.489 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:52 smithi125 conmon[24927]: 2021-03-24T22:04:52.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:52.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:52.509 INFO:tasks.rados.rados.0.smithi125.stdout:2760: done (2 left) 2021-03-24T22:04:52.509 INFO:tasks.rados.rados.0.smithi125.stdout:2761: write oid 38 current snap is 316 2021-03-24T22:04:52.510 INFO:tasks.rados.rados.0.smithi125.stdout:2761: seq_num 957 ranges {484299=557418,1684797=399502} 2021-03-24T22:04:52.516 INFO:tasks.rados.rados.0.smithi125.stdout:2761: writing smithi12556470-38 from 484299 to 1041717 tid 1 2021-03-24T22:04:52.523 INFO:tasks.rados.rados.0.smithi125.stdout:2761: writing smithi12556470-38 from 1684797 to 2084299 tid 2 2021-03-24T22:04:52.523 INFO:tasks.rados.rados.0.smithi125.stdout:2762: snap_create 2021-03-24T22:04:52.530 INFO:tasks.rados.rados.0.smithi125.stdout:2758: expect (ObjNum 718 snap 229 seq_num 718) 2021-03-24T22:04:52.551 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:52 smithi142 conmon[20125]: cluster 2021-03-24T22:04:50 2021-03-24T22:04:52.552 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:52 smithi142 conmon[20125]: .964459+0000 mgr.y (mgr.14144) 635 : cluster [DBG] pgmap v1243: 168 pgs: 1 peering, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 107 active+clean; 700 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 4.9 MiB/s rd, 5.2 MiB/s wr, 23 op/s; 160/1938 objects degraded (8.256%); 20/1938 objects misplaced (1.032%); 395 B/s, 4 objects/s recovering 2021-03-24T22:04:52.552 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:52 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:04:52.552 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:52 smithi142 conmon[20125]: 22:04:51.376675+0000 mon.a (mon.0) 1876 : cluster [DBG] osdmap e721: 8 total, 7 up, 8 in 2021-03-24T22:04:52.552 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:52 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:04:52.553 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:52 smithi142 conmon[20125]: 03-24T22:04:51.491950+0000 mon.a (mon.0 2021-03-24T22:04:52.553 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:52 smithi142 conmon[20125]: ) 1877 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2021-03-24T22:04:52.553 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:52 smithi142 conmon[20125]: audit 2021-03-24T 2021-03-24T22:04:52.553 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:52 smithi142 conmon[20125]: 22:04:52.014790+0000 mon.a (mon 2021-03-24T22:04:52.554 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:52 smithi142 conmon[20125]: .0) 1878 : audit [DBG] from='client.? v1:172.21.15.125:0/4202212003' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:04:52.592 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:52.617 INFO:tasks.rados.rados.0.smithi125.stdout:2759: finishing write tid 1 to smithi12556470-23 2021-03-24T22:04:52.618 INFO:tasks.rados.rados.0.smithi125.stdout:2759: finishing write tid 2 to smithi12556470-23 2021-03-24T22:04:52.618 INFO:tasks.rados.rados.0.smithi125.stdout:2759: finishing write tid 3 to smithi12556470-23 2021-03-24T22:04:52.618 INFO:tasks.rados.rados.0.smithi125.stdout:2759: finishing write tid 4 to smithi12556470-23 2021-03-24T22:04:52.619 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 23 v 485 (ObjNum 956 snap 316 seq_num 956) dirty exists 2021-03-24T22:04:52.619 INFO:tasks.rados.rados.0.smithi125.stdout:2759: left oid 23 (ObjNum 956 snap 316 seq_num 956) 2021-03-24T22:04:52.619 INFO:tasks.rados.rados.0.smithi125.stdout:2761: finishing write tid 1 to smithi12556470-38 2021-03-24T22:04:52.619 INFO:tasks.rados.rados.0.smithi125.stdout:2761: finishing write tid 2 to smithi12556470-38 2021-03-24T22:04:52.619 INFO:tasks.rados.rados.0.smithi125.stdout:2761: finishing write tid 3 to smithi12556470-38 2021-03-24T22:04:52.620 INFO:tasks.rados.rados.0.smithi125.stdout:2761: finishing write tid 4 to smithi12556470-38 2021-03-24T22:04:52.620 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 38 v 734 (ObjNum 957 snap 316 seq_num 957) dirty exists 2021-03-24T22:04:52.620 INFO:tasks.rados.rados.0.smithi125.stdout:2761: left oid 38 (ObjNum 957 snap 316 seq_num 957) 2021-03-24T22:04:52.634 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:52.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:52.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:52.739 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:52.788 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:52.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:52.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:04:52.857 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2021-03-24T22:04:52.858 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T22:04:52.897 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:52.962 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:52.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:52.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:04:53.070 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:53.131 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:53.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:53.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:53.237 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:53.298 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:53.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:53.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:04:53.389 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-03-24T22:04:53.390 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-03-24T22:04:53.403 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:53.463 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:53.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:53.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:04:53.520 INFO:tasks.rados.rados.0.smithi125.stdout:2758: done (3 left) 2021-03-24T22:04:53.521 INFO:tasks.rados.rados.0.smithi125.stdout:2759: done (2 left) 2021-03-24T22:04:53.521 INFO:tasks.rados.rados.0.smithi125.stdout:2761: done (1 left) 2021-03-24T22:04:53.521 INFO:tasks.rados.rados.0.smithi125.stdout:2762: done (0 left) 2021-03-24T22:04:53.521 INFO:tasks.rados.rados.0.smithi125.stdout:2763: read oid 20 snap -1 2021-03-24T22:04:53.522 INFO:tasks.rados.rados.0.smithi125.stdout:2763: expect (ObjNum 733 snap 235 seq_num 733) 2021-03-24T22:04:53.522 INFO:tasks.rados.rados.0.smithi125.stdout:2764: write oid 18 current snap is 317 2021-03-24T22:04:53.522 INFO:tasks.rados.rados.0.smithi125.stdout:2764: seq_num 958 ranges {501106=749020,1743279=357827} 2021-03-24T22:04:53.523 INFO:tasks.rados.rados.0.smithi125.stdout:2764: writing smithi12556470-18 from 501106 to 1250126 tid 1 2021-03-24T22:04:53.528 INFO:tasks.rados.rados.0.smithi125.stdout:2764: writing smithi12556470-18 from 1743279 to 2101106 tid 2 2021-03-24T22:04:53.529 INFO:tasks.rados.rados.0.smithi125.stdout:2765: write oid 38 current snap is 317 2021-03-24T22:04:53.529 INFO:tasks.rados.rados.0.smithi125.stdout:2763: expect (ObjNum 733 snap 235 seq_num 733) 2021-03-24T22:04:53.569 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:53.615 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:53.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:53.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:53.624 INFO:tasks.rados.rados.0.smithi125.stdout:2764: finishing write tid 1 to smithi12556470-18 2021-03-24T22:04:53.624 INFO:tasks.rados.rados.0.smithi125.stdout:2764: finishing write tid 2 to smithi12556470-18 2021-03-24T22:04:53.624 INFO:tasks.rados.rados.0.smithi125.stdout:2765: seq_num 959 ranges {517913=424269,1401761=446989} 2021-03-24T22:04:53.629 INFO:tasks.rados.rados.0.smithi125.stdout:2765: writing smithi12556470-38 from 517913 to 942182 tid 1 2021-03-24T22:04:53.636 INFO:tasks.rados.rados.0.smithi125.stdout:2765: writing smithi12556470-38 from 1401761 to 1848750 tid 2 2021-03-24T22:04:53.636 INFO:tasks.rados.rados.0.smithi125.stdout:2763: done (2 left) 2021-03-24T22:04:53.636 INFO:tasks.rados.rados.0.smithi125.stdout:2766: delete oid 20 current snap is 317 2021-03-24T22:04:53.637 INFO:tasks.rados.rados.0.smithi125.stdout:2764: finishing write tid 3 to smithi12556470-18 2021-03-24T22:04:53.637 INFO:tasks.rados.rados.0.smithi125.stdout:2764: finishing write tid 4 to smithi12556470-18 2021-03-24T22:04:53.637 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 18 v 705 (ObjNum 958 snap 317 seq_num 958) dirty exists 2021-03-24T22:04:53.637 INFO:tasks.rados.rados.0.smithi125.stdout:2764: left oid 18 (ObjNum 958 snap 317 seq_num 958) 2021-03-24T22:04:53.638 INFO:tasks.rados.rados.0.smithi125.stdout:2765: finishing write tid 1 to smithi12556470-38 2021-03-24T22:04:53.640 INFO:tasks.rados.rados.0.smithi125.stdout:2765: finishing write tid 2 to smithi12556470-38 2021-03-24T22:04:53.640 INFO:tasks.rados.rados.0.smithi125.stdout:2765: finishing write tid 3 to smithi12556470-38 2021-03-24T22:04:53.641 INFO:tasks.rados.rados.0.smithi125.stdout:2765: finishing write tid 4 to smithi12556470-38 2021-03-24T22:04:53.641 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 38 v 737 (ObjNum 959 snap 317 seq_num 959) dirty exists 2021-03-24T22:04:53.641 INFO:tasks.rados.rados.0.smithi125.stdout:2765: left oid 38 (ObjNum 959 snap 317 seq_num 959) 2021-03-24T22:04:53.651 INFO:tasks.rados.rados.0.smithi125.stdout:2764: done (2 left) 2021-03-24T22:04:53.651 INFO:tasks.rados.rados.0.smithi125.stdout:2765: done (1 left) 2021-03-24T22:04:53.651 INFO:tasks.rados.rados.0.smithi125.stdout:2766: done (0 left) 2021-03-24T22:04:53.651 INFO:tasks.rados.rados.0.smithi125.stdout:2767: write (excl) oid 28 current snap is 317 2021-03-24T22:04:53.652 INFO:tasks.rados.rados.0.smithi125.stdout:2767: seq_num 961 ranges {551527=574767,1918725=232802} 2021-03-24T22:04:53.658 INFO:tasks.rados.rados.0.smithi125.stdout:2767: writing smithi12556470-28 from 551527 to 1126294 tid 1 2021-03-24T22:04:53.666 INFO:tasks.rados.rados.0.smithi125.stdout:2767: writing smithi12556470-28 from 1918725 to 2151527 tid 2 2021-03-24T22:04:53.666 INFO:tasks.rados.rados.0.smithi125.stdout:2768: write (excl) oid 41 current snap is 317 2021-03-24T22:04:53.666 INFO:tasks.rados.rados.0.smithi125.stdout:2767: finishing write tid 1 to smithi12556470-28 2021-03-24T22:04:53.667 INFO:tasks.rados.rados.0.smithi125.stdout:2768: seq_num 962 ranges {568334=650016,1860854=307480} 2021-03-24T22:04:53.672 INFO:tasks.rados.rados.0.smithi125.stdout:2768: writing smithi12556470-41 from 568334 to 1218350 tid 1 2021-03-24T22:04:53.676 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd unset noscrub 2021-03-24T22:04:53.679 INFO:tasks.rados.rados.0.smithi125.stdout:2768: writing smithi12556470-41 from 1860854 to 2168334 tid 2 2021-03-24T22:04:53.679 INFO:tasks.rados.rados.0.smithi125.stdout:2769: copy_from oid 8 from oid 34 current snap is 317 2021-03-24T22:04:53.680 INFO:tasks.rados.rados.0.smithi125.stdout:2767: finishing write tid 2 to smithi12556470-28 2021-03-24T22:04:53.680 INFO:tasks.rados.rados.0.smithi125.stdout:2767: finishing write tid 3 to smithi12556470-28 2021-03-24T22:04:53.680 INFO:tasks.rados.rados.0.smithi125.stdout:2767: finishing write tid 4 to smithi12556470-28 2021-03-24T22:04:53.680 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 28 v 1198 (ObjNum 961 snap 317 seq_num 961) dirty exists 2021-03-24T22:04:53.681 INFO:tasks.rados.rados.0.smithi125.stdout:2767: left oid 28 (ObjNum 961 snap 317 seq_num 961) 2021-03-24T22:04:53.681 INFO:tasks.rados.rados.0.smithi125.stdout:2767: done (2 left) 2021-03-24T22:04:53.681 INFO:tasks.rados.rados.0.smithi125.stdout:2770: write oid 20 current snap is 317 2021-03-24T22:04:53.681 INFO:tasks.rados.rados.0.smithi125.stdout:2770: seq_num 963 ranges {585141=725265,1919336=265805} 2021-03-24T22:04:53.687 INFO:tasks.rados.rados.0.smithi125.stdout:2770: writing smithi12556470-20 from 585141 to 1310406 tid 1 2021-03-24T22:04:53.692 INFO:tasks.rados.rados.0.smithi125.stdout:2770: writing smithi12556470-20 from 1919336 to 2185141 tid 2 2021-03-24T22:04:53.693 INFO:tasks.rados.rados.0.smithi125.stdout:2771: read oid 21 snap -1 2021-03-24T22:04:53.693 INFO:tasks.rados.rados.0.smithi125.stdout:2771: expect (ObjNum 590 snap 177 seq_num 590) 2021-03-24T22:04:53.693 INFO:tasks.rados.rados.0.smithi125.stdout:2772: snap_create 2021-03-24T22:04:53.693 INFO:tasks.rados.rados.0.smithi125.stdout:2769: finishing copy_from to smithi12556470-8 2021-03-24T22:04:53.693 INFO:tasks.rados.rados.0.smithi125.stdout:2769: got expected ENOENT (src dne) 2021-03-24T22:04:53.694 INFO:tasks.rados.rados.0.smithi125.stdout:2769: finishing copy_from racing read to smithi12556470-8 2021-03-24T22:04:53.694 INFO:tasks.rados.rados.0.smithi125.stdout:2768: finishing write tid 1 to smithi12556470-41 2021-03-24T22:04:53.695 INFO:tasks.rados.rados.0.smithi125.stdout:2768: finishing write tid 2 to smithi12556470-41 2021-03-24T22:04:53.695 INFO:tasks.rados.rados.0.smithi125.stdout:2768: finishing write tid 3 to smithi12556470-41 2021-03-24T22:04:53.695 INFO:tasks.rados.rados.0.smithi125.stdout:2768: finishing write tid 4 to smithi12556470-41 2021-03-24T22:04:53.695 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 41 v 1031 (ObjNum 962 snap 317 seq_num 962) dirty exists 2021-03-24T22:04:53.695 INFO:tasks.rados.rados.0.smithi125.stdout:2768: left oid 41 (ObjNum 962 snap 317 seq_num 962) 2021-03-24T22:04:53.696 INFO:tasks.rados.rados.0.smithi125.stdout:2770: finishing write tid 1 to smithi12556470-20 2021-03-24T22:04:53.696 INFO:tasks.rados.rados.0.smithi125.stdout:2770: finishing write tid 2 to smithi12556470-20 2021-03-24T22:04:53.697 INFO:tasks.rados.rados.0.smithi125.stdout:2770: finishing write tid 3 to smithi12556470-20 2021-03-24T22:04:53.697 INFO:tasks.rados.rados.0.smithi125.stdout:2770: finishing write tid 4 to smithi12556470-20 2021-03-24T22:04:53.697 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 20 v 826 (ObjNum 963 snap 317 seq_num 963) dirty exists 2021-03-24T22:04:53.697 INFO:tasks.rados.rados.0.smithi125.stdout:2770: left oid 20 (ObjNum 963 snap 317 seq_num 963) 2021-03-24T22:04:53.707 INFO:tasks.rados.rados.0.smithi125.stdout:2771: expect (ObjNum 590 snap 177 seq_num 590) 2021-03-24T22:04:53.720 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:53.776 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:53.779 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:53 smithi125 conmon[19936]: cluster 2021-03-24T22:04:52.503572+0000 mon.a (mon.0) 1879 : 2021-03-24T22:04:53.780 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:53 smithi125 conmon[19936]: cluster [DBG] osdmap e722: 8 total, 7 up, 8 in 2021-03-24T22:04:53.780 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:53 smithi125 conmon[24927]: cluster 2021-03-24T22:04:52.503572+0000 mon.a (mon.0) 1879 : cluster [DBG] osdmap e722: 8 total, 7 up, 8 in 2021-03-24T22:04:53.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:53.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:53.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:53 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:04:53.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:53 smithi142 conmon[20125]: 22:04:52.503572+0000 mon.a (mon.0) 1879 : cluster [DBG] osdmap e722: 8 total, 7 up, 8 in 2021-03-24T22:04:53.882 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:53.887 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 1] out_osds: [] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 6, 7] 2021-03-24T22:04:53.887 INFO:tasks.thrashosds.thrasher:Reviving osd 5 2021-03-24T22:04:53.887 INFO:tasks.cephadm.osd.5:Restarting osd.5 (starting--it wasn't running)... 2021-03-24T22:04:53.888 DEBUG:teuthology.orchestra.run.smithi142:osd.5> sudo journalctl -f -n 0 -u ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c@osd.5.service 2021-03-24T22:04:53.890 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl start ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c@osd.5 2021-03-24T22:04:53.905 INFO:journalctl@ceph.osd.5.smithi142.stdout:-- Logs begin at Wed 2021-03-03 23:51:07 UTC. -- 2021-03-24T22:04:53.925 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:53.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:53.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:53.956 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.7 dump_blocked_ops 2021-03-24T22:04:53.984 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:04:54.031 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:54.088 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:54.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:54.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:04:54.194 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:54.255 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:54.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:54.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:54.273 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T22:04:54.334 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 22:04:53 smithi142 systemd[1]: Starting Ceph osd.5 for 3cfa339a-8cea-11eb-90b2-001a4aab830c... 2021-03-24T22:04:54.360 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:54.413 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:54.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:54.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:04:54.518 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:54.526 INFO:tasks.rados.rados.0.smithi125.stdout:2768: done (4 left) 2021-03-24T22:04:54.527 INFO:tasks.rados.rados.0.smithi125.stdout:2769: done (3 left) 2021-03-24T22:04:54.527 INFO:tasks.rados.rados.0.smithi125.stdout:2770: done (2 left) 2021-03-24T22:04:54.527 INFO:tasks.rados.rados.0.smithi125.stdout:2771: done (1 left) 2021-03-24T22:04:54.528 INFO:tasks.rados.rados.0.smithi125.stdout:2772: done (0 left) 2021-03-24T22:04:54.528 INFO:tasks.rados.rados.0.smithi125.stdout:2773: write (excl) oid 42 current snap is 318 2021-03-24T22:04:54.528 INFO:tasks.rados.rados.0.smithi125.stdout:2773: seq_num 964 ranges {601948=400514,1577818=624130} 2021-03-24T22:04:54.540 INFO:tasks.rados.rados.0.smithi125.stdout:2773: writing smithi12556470-42 from 601948 to 1002462 tid 1 2021-03-24T22:04:54.542 INFO:tasks.rados.rados.0.smithi125.stdout:2773: writing smithi12556470-42 from 1577818 to 2201948 tid 2 2021-03-24T22:04:54.542 INFO:tasks.rados.rados.0.smithi125.stdout:2774: rollback oid 24 current snap is 318 2021-03-24T22:04:54.542 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 24 to 299 2021-03-24T22:04:54.542 INFO:tasks.rados.rados.0.smithi125.stdout:2775: write oid 11 current snap is 318 2021-03-24T22:04:54.543 INFO:tasks.rados.rados.0.smithi125.stdout:2775: seq_num 965 ranges {618755=475763,1636300=457996} 2021-03-24T22:04:54.548 INFO:tasks.rados.rados.0.smithi125.stdout:2775: writing smithi12556470-11 from 618755 to 1094518 tid 1 2021-03-24T22:04:54.555 INFO:tasks.rados.rados.0.smithi125.stdout:2775: writing smithi12556470-11 from 1636300 to 2094296 tid 2 2021-03-24T22:04:54.556 INFO:tasks.rados.rados.0.smithi125.stdout:2776: delete oid 12 current snap is 318 2021-03-24T22:04:54.556 INFO:tasks.rados.rados.0.smithi125.stdout:2773: finishing write tid 1 to smithi12556470-42 2021-03-24T22:04:54.556 INFO:tasks.rados.rados.0.smithi125.stdout:2774: finishing rollback tid 0 to smithi12556470-24 2021-03-24T22:04:54.557 INFO:tasks.rados.rados.0.smithi125.stdout:2774: finishing rollback tid 1 to smithi12556470-24 2021-03-24T22:04:54.557 INFO:tasks.rados.rados.0.smithi125.stdout:2774: finishing rollback tid 2 to smithi12556470-24 2021-03-24T22:04:54.557 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 24 v 1428 (ObjNum 851 snap 278 seq_num 851) dirty exists 2021-03-24T22:04:54.557 INFO:tasks.rados.rados.0.smithi125.stdout:2773: finishing write tid 2 to smithi12556470-42 2021-03-24T22:04:54.558 INFO:tasks.rados.rados.0.smithi125.stdout:2773: finishing write tid 3 to smithi12556470-42 2021-03-24T22:04:54.560 INFO:tasks.rados.rados.0.smithi125.stdout:2773: finishing write tid 4 to smithi12556470-42 2021-03-24T22:04:54.561 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 42 v 509 (ObjNum 964 snap 318 seq_num 964) dirty exists 2021-03-24T22:04:54.561 INFO:tasks.rados.rados.0.smithi125.stdout:2773: left oid 42 (ObjNum 964 snap 318 seq_num 964) 2021-03-24T22:04:54.561 INFO:tasks.rados.rados.0.smithi125.stdout:2773: done (3 left) 2021-03-24T22:04:54.561 INFO:tasks.rados.rados.0.smithi125.stdout:2774: done (2 left) 2021-03-24T22:04:54.562 INFO:tasks.rados.rados.0.smithi125.stdout:2776: done (1 left) 2021-03-24T22:04:54.562 INFO:tasks.rados.rados.0.smithi125.stdout:2777: copy_from oid 32 from oid 37 current snap is 318 2021-03-24T22:04:54.562 INFO:tasks.rados.rados.0.smithi125.stdout:2778: read oid 10 snap -1 2021-03-24T22:04:54.562 INFO:tasks.rados.rados.0.smithi125.stdout:2778: expect (ObjNum 855 snap 278 seq_num 855) 2021-03-24T22:04:54.563 INFO:tasks.rados.rados.0.smithi125.stdout:2779: delete oid 30 current snap is 318 2021-03-24T22:04:54.566 INFO:tasks.rados.rados.0.smithi125.stdout:2777: finishing copy_from to smithi12556470-32 2021-03-24T22:04:54.567 INFO:tasks.rados.rados.0.smithi125.stdout:2777: got expected ENOENT (src dne) 2021-03-24T22:04:54.568 INFO:tasks.rados.rados.0.smithi125.stdout:2775: finishing write tid 1 to smithi12556470-11 2021-03-24T22:04:54.570 INFO:tasks.rados.rados.0.smithi125.stdout:2775: finishing write tid 2 to smithi12556470-11 2021-03-24T22:04:54.570 INFO:tasks.rados.rados.0.smithi125.stdout:2775: finishing write tid 3 to smithi12556470-11 2021-03-24T22:04:54.571 INFO:tasks.rados.rados.0.smithi125.stdout:2775: finishing write tid 4 to smithi12556470-11 2021-03-24T22:04:54.571 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 11 v 559 (ObjNum 965 snap 318 seq_num 965) dirty exists 2021-03-24T22:04:54.571 INFO:tasks.rados.rados.0.smithi125.stdout:2775: left oid 11 (ObjNum 965 snap 318 seq_num 965) 2021-03-24T22:04:54.572 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:54.574 INFO:tasks.rados.rados.0.smithi125.stdout:2778: expect (ObjNum 855 snap 278 seq_num 855) 2021-03-24T22:04:54.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:54.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:04:54.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:54 smithi142 conmon[20125]: cluster 2021-03-24T22:04:52.965406+0000 mgr.y (mgr.14144) 636 : cluster 2021-03-24T22:04:54.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:54 smithi142 conmon[20125]: [DBG] pgmap v1246: 168 pgs: 1 peering, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 107 active+clean; 693 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 7.3 MiB/s rd, 5.2 MiB/s wr, 25 op/s; 160/1923 objects degraded (8.320%); 20/1923 objects misplaced (1.040%); 395 B/s, 4 objects/s recovering 2021-03-24T22:04:54.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:54 smithi142 conmon[20125]: cluster 2021-03-24T22:04:53.510318+0000 mon.a (mon.0) 2021-03-24T22:04:54.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:54 smithi142 conmon[20125]: 1880 : cluster [DBG] osdmap e723: 8 total, 7 up, 8 in 2021-03-24T22:04:54.618 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 22:04:54 smithi142 podman[90036]: 2021-03-24 22:04:54.335099058 +0000 UTC m=+0.261755304 container create 5dca950bfcb6dfa932d50d4a1d8d7f91610120a9d37b0ecf319f7c06c44596be (image=quay.ceph.io/ceph-ci/ceph@sha256:e9db06906b8d0a24f3ef6835ac5aa4b8eea2950a2560bad78f83d4f101eca0db, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5-activate, maintainer=Dimitri Savineau , GIT_BRANCH=HEAD, org.label-schema.build-date=20201204, GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06, RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d, org.label-schema.name=CentOS Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.18.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2021-03-24T22:04:54.618 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 22:04:54 smithi142 podman[90036]: 2021-03-24 22:04:54.501781738 +0000 UTC m=+0.428438027 container init 5dca950bfcb6dfa932d50d4a1d8d7f91610120a9d37b0ecf319f7c06c44596be (image=quay.ceph.io/ceph-ci/ceph@sha256:e9db06906b8d0a24f3ef6835ac5aa4b8eea2950a2560bad78f83d4f101eca0db, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5-activate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.18.0, org.label-schema.build-date=20201204, GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d, org.label-schema.name=CentOS Base Image, maintainer=Dimitri Savineau , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2021-03-24T22:04:54.618 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 22:04:54 smithi142 podman[90036]: 2021-03-24 22:04:54.535122649 +0000 UTC m=+0.461778889 container start 5dca950bfcb6dfa932d50d4a1d8d7f91610120a9d37b0ecf319f7c06c44596be (image=quay.ceph.io/ceph-ci/ceph@sha256:e9db06906b8d0a24f3ef6835ac5aa4b8eea2950a2560bad78f83d4f101eca0db, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5-activate, io.buildah.version=1.18.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, maintainer=Dimitri Savineau , RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d, GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20201204, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2021-03-24T22:04:54.618 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 22:04:54 smithi142 podman[90036]: 2021-03-24 22:04:54.535376572 +0000 UTC m=+0.462032815 container attach 5dca950bfcb6dfa932d50d4a1d8d7f91610120a9d37b0ecf319f7c06c44596be (image=quay.ceph.io/ceph-ci/ceph@sha256:e9db06906b8d0a24f3ef6835ac5aa4b8eea2950a2560bad78f83d4f101eca0db, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5-activate, RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d, org.label-schema.build-date=20201204, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.18.0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, maintainer=Dimitri Savineau , org.label-schema.name=CentOS Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06, org.label-schema.license=GPLv2) 2021-03-24T22:04:54.677 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:54.726 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:54.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:54.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:54.747 INFO:tasks.rados.rados.0.smithi125.stdout:2777: finishing copy_from racing read to smithi12556470-32 2021-03-24T22:04:54.747 INFO:tasks.rados.rados.0.smithi125.stdout:2775: done (3 left) 2021-03-24T22:04:54.747 INFO:tasks.rados.rados.0.smithi125.stdout:2777: done (2 left) 2021-03-24T22:04:54.748 INFO:tasks.rados.rados.0.smithi125.stdout:2778: done (1 left) 2021-03-24T22:04:54.748 INFO:tasks.rados.rados.0.smithi125.stdout:2779: done (0 left) 2021-03-24T22:04:54.748 INFO:tasks.rados.rados.0.smithi125.stdout:2780: rollback oid 41 current snap is 318 2021-03-24T22:04:54.748 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 41 to 316 2021-03-24T22:04:54.749 INFO:tasks.rados.rados.0.smithi125.stdout:2781: read oid 31 snap -1 2021-03-24T22:04:54.749 INFO:tasks.rados.rados.0.smithi125.stdout:2781: expect deleted 2021-03-24T22:04:54.749 INFO:tasks.rados.rados.0.smithi125.stdout:2782: delete oid 23 current snap is 318 2021-03-24T22:04:54.750 INFO:tasks.rados.rados.0.smithi125.stdout:2781: done (2 left) 2021-03-24T22:04:54.750 INFO:tasks.rados.rados.0.smithi125.stdout:2782: done (1 left) 2021-03-24T22:04:54.750 INFO:tasks.rados.rados.0.smithi125.stdout:2783: write oid 2 current snap is 318 2021-03-24T22:04:54.751 INFO:tasks.rados.rados.0.smithi125.stdout:2783: seq_num 969 ranges {685983=493112,1870228=415755} 2021-03-24T22:04:54.763 INFO:tasks.rados.rados.0.smithi125.stdout:2783: writing smithi12556470-2 from 685983 to 1179095 tid 1 2021-03-24T22:04:54.770 INFO:tasks.rados.rados.0.smithi125.stdout:2783: writing smithi12556470-2 from 1870228 to 2285983 tid 2 2021-03-24T22:04:54.770 INFO:tasks.rados.rados.0.smithi125.stdout:2784: write (excl) oid 10 current snap is 318 2021-03-24T22:04:54.770 INFO:tasks.rados.rados.0.smithi125.stdout:2780: finishing rollback tid 0 to smithi12556470-41 2021-03-24T22:04:54.771 INFO:tasks.rados.rados.0.smithi125.stdout:2784: seq_num 970 ranges {702790=568361,1812357=490433} 2021-03-24T22:04:54.776 INFO:tasks.rados.rados.0.smithi125.stdout:2784: writing smithi12556470-10 from 702790 to 1271151 tid 1 2021-03-24T22:04:54.782 INFO:tasks.rados.rados.0.smithi125.stdout:2784: writing smithi12556470-10 from 1812357 to 2302790 tid 2 2021-03-24T22:04:54.783 INFO:tasks.rados.rados.0.smithi125.stdout:2785: delete oid 23 current snap is 318 2021-03-24T22:04:54.783 INFO:tasks.rados.rados.0.smithi125.stdout:2780: finishing rollback tid 1 to smithi12556470-41 2021-03-24T22:04:54.783 INFO:tasks.rados.rados.0.smithi125.stdout:2780: finishing rollback tid 2 to smithi12556470-41 2021-03-24T22:04:54.783 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 41 v 1034 (ObjNum 916 snap 304 seq_num 916) dirty exists 2021-03-24T22:04:54.784 INFO:tasks.rados.rados.0.smithi125.stdout:2783: finishing write tid 1 to smithi12556470-2 2021-03-24T22:04:54.784 INFO:tasks.rados.rados.0.smithi125.stdout:2783: finishing write tid 2 to smithi12556470-2 2021-03-24T22:04:54.784 INFO:tasks.rados.rados.0.smithi125.stdout:2783: finishing write tid 3 to smithi12556470-2 2021-03-24T22:04:54.784 INFO:tasks.rados.rados.0.smithi125.stdout:2783: finishing write tid 4 to smithi12556470-2 2021-03-24T22:04:54.785 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 2 v 1078 (ObjNum 969 snap 318 seq_num 969) dirty exists 2021-03-24T22:04:54.785 INFO:tasks.rados.rados.0.smithi125.stdout:2783: left oid 2 (ObjNum 969 snap 318 seq_num 969) 2021-03-24T22:04:54.785 INFO:tasks.rados.rados.0.smithi125.stdout:2784: finishing write tid 1 to smithi12556470-10 2021-03-24T22:04:54.786 INFO:tasks.rados.rados.0.smithi125.stdout:2780: done (3 left) 2021-03-24T22:04:54.786 INFO:tasks.rados.rados.0.smithi125.stdout:2783: done (2 left) 2021-03-24T22:04:54.786 INFO:tasks.rados.rados.0.smithi125.stdout:2785: done (1 left) 2021-03-24T22:04:54.787 INFO:tasks.rados.rados.0.smithi125.stdout:2786: write (excl) oid 35 current snap is 318 2021-03-24T22:04:54.787 INFO:tasks.rados.rados.0.smithi125.stdout:2786: seq_num 972 ranges {736404=718859,1929321=407083} 2021-03-24T22:04:54.792 INFO:tasks.rados.rados.0.smithi125.stdout:2786: writing smithi12556470-35 from 736404 to 1455263 tid 1 2021-03-24T22:04:54.797 INFO:tasks.rados.rados.0.smithi125.stdout:2786: writing smithi12556470-35 from 1929321 to 2336404 tid 2 2021-03-24T22:04:54.798 INFO:tasks.rados.rados.0.smithi125.stdout:2787: write (excl) oid 38 current snap is 318 2021-03-24T22:04:54.798 INFO:tasks.rados.rados.0.smithi125.stdout:2784: finishing write tid 2 to smithi12556470-10 2021-03-24T22:04:54.798 INFO:tasks.rados.rados.0.smithi125.stdout:2787: seq_num 973 ranges {753211=794108,1987803=365408} 2021-03-24T22:04:54.806 INFO:tasks.rados.rados.0.smithi125.stdout:2787: writing smithi12556470-38 from 753211 to 1547319 tid 1 2021-03-24T22:04:54.811 INFO:tasks.rados.rados.0.smithi125.stdout:2787: writing smithi12556470-38 from 1987803 to 2353211 tid 2 2021-03-24T22:04:54.812 INFO:tasks.rados.rados.0.smithi125.stdout:2788: write (excl) oid 17 current snap is 318 2021-03-24T22:04:54.812 INFO:tasks.rados.rados.0.smithi125.stdout:2784: finishing write tid 3 to smithi12556470-10 2021-03-24T22:04:54.812 INFO:tasks.rados.rados.0.smithi125.stdout:2784: finishing write tid 4 to smithi12556470-10 2021-03-24T22:04:54.812 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 10 v 1431 (ObjNum 970 snap 318 seq_num 970) dirty exists 2021-03-24T22:04:54.812 INFO:tasks.rados.rados.0.smithi125.stdout:2784: left oid 10 (ObjNum 970 snap 318 seq_num 970) 2021-03-24T22:04:54.813 INFO:tasks.rados.rados.0.smithi125.stdout:2786: finishing write tid 1 to smithi12556470-35 2021-03-24T22:04:54.813 INFO:tasks.rados.rados.0.smithi125.stdout:2788: seq_num 974 ranges {770018=585710,2046285=323733} 2021-03-24T22:04:54.818 INFO:tasks.rados.rados.0.smithi125.stdout:2788: writing smithi12556470-17 from 770018 to 1355728 tid 1 2021-03-24T22:04:54.825 INFO:tasks.rados.rados.0.smithi125.stdout:2788: writing smithi12556470-17 from 2046285 to 2370018 tid 2 2021-03-24T22:04:54.826 INFO:tasks.rados.rados.0.smithi125.stdout:2784: done (3 left) 2021-03-24T22:04:54.826 INFO:tasks.rados.rados.0.smithi125.stdout:2789: read oid 31 snap -1 2021-03-24T22:04:54.826 INFO:tasks.rados.rados.0.smithi125.stdout:2789: expect deleted 2021-03-24T22:04:54.826 INFO:tasks.rados.rados.0.smithi125.stdout:2790: snap_create 2021-03-24T22:04:54.826 INFO:tasks.rados.rados.0.smithi125.stdout:2786: finishing write tid 2 to smithi12556470-35 2021-03-24T22:04:54.827 INFO:tasks.rados.rados.0.smithi125.stdout:2786: finishing write tid 3 to smithi12556470-35 2021-03-24T22:04:54.827 INFO:tasks.rados.rados.0.smithi125.stdout:2786: finishing write tid 4 to smithi12556470-35 2021-03-24T22:04:54.827 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 35 v 234 (ObjNum 972 snap 318 seq_num 972) dirty exists 2021-03-24T22:04:54.827 INFO:tasks.rados.rados.0.smithi125.stdout:2786: left oid 35 (ObjNum 972 snap 318 seq_num 972) 2021-03-24T22:04:54.828 INFO:tasks.rados.rados.0.smithi125.stdout:2787: finishing write tid 1 to smithi12556470-38 2021-03-24T22:04:54.828 INFO:tasks.rados.rados.0.smithi125.stdout:2787: finishing write tid 2 to smithi12556470-38 2021-03-24T22:04:54.828 INFO:tasks.rados.rados.0.smithi125.stdout:2787: finishing write tid 3 to smithi12556470-38 2021-03-24T22:04:54.828 INFO:tasks.rados.rados.0.smithi125.stdout:2787: finishing write tid 4 to smithi12556470-38 2021-03-24T22:04:54.828 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 38 v 740 (ObjNum 973 snap 318 seq_num 973) dirty exists 2021-03-24T22:04:54.829 INFO:tasks.rados.rados.0.smithi125.stdout:2787: left oid 38 (ObjNum 973 snap 318 seq_num 973) 2021-03-24T22:04:54.829 INFO:tasks.rados.rados.0.smithi125.stdout:2788: finishing write tid 1 to smithi12556470-17 2021-03-24T22:04:54.830 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:54.832 INFO:tasks.rados.rados.0.smithi125.stdout:2788: finishing write tid 2 to smithi12556470-17 2021-03-24T22:04:54.833 INFO:tasks.rados.rados.0.smithi125.stdout:2788: finishing write tid 3 to smithi12556470-17 2021-03-24T22:04:54.834 INFO:tasks.rados.rados.0.smithi125.stdout:2788: finishing write tid 4 to smithi12556470-17 2021-03-24T22:04:54.834 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 17 v 1037 (ObjNum 974 snap 318 seq_num 974) dirty exists 2021-03-24T22:04:54.834 INFO:tasks.rados.rados.0.smithi125.stdout:2788: left oid 17 (ObjNum 974 snap 318 seq_num 974) 2021-03-24T22:04:54.883 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:54.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:54.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:04:54.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:54 smithi125 conmon[19936]: cluster 2021-03-24T22:04:52.965406+0000 mgr.y ( 2021-03-24T22:04:54.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:54 smithi125 conmon[19936]: mgr.14144) 636 : cluster [DBG] pgmap v1246: 168 pgs: 1 peering, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 107 active+clean; 693 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 7.3 MiB/s rd, 5.2 MiB/s wr, 25 op/s; 160/1923 objects degraded (8.320%); 20/1923 objects misplaced (1.040%); 395 B/s, 4 objects/s recovering 2021-03-24T22:04:54.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:54 smithi125 conmon[19936]: cluster 2021-03-24T22:04: 2021-03-24T22:04:54.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:54 smithi125 conmon[19936]: 53.510318+0000 mon.a (mon.0) 2021-03-24T22:04:54.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:54 smithi125 conmon[19936]: 1880 : cluster [DBG] osdmap e723: 8 total, 7 up, 8 in 2021-03-24T22:04:54.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:54 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T22:04:54.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:54 smithi125 conmon[24927]: 22:04:52.965406+0000 mgr.y (mgr.14144) 636 : cluster [DBG] pgmap v1246: 168 pgs: 1 peering, 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 107 active+clean; 693 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 7.3 MiB/s rd, 5.2 MiB/s wr, 25 op/s; 160/1923 objects degraded (8.320%); 20/1923 objects misplaced (1.040%); 395 B/s, 4 objects/s recovering 2021-03-24T22:04:54.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:54 smithi125 conmon[24927]: cluster 2021-03-24T22 2021-03-24T22:04:54.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:54 smithi125 conmon[24927]: :04:53.510318+0000 mon.a (mon.0) 1880 : 2021-03-24T22:04:54.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:54 smithi125 conmon[24927]: cluster [DBG] osdmap e723: 8 total, 7 up, 8 in 2021-03-24T22:04:54.989 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:55.048 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:55.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:55.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:55.116 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 22:04:54 smithi142 bash[90036]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2021-03-24T22:04:55.117 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 22:04:54 smithi142 bash[90036]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-5 --no-mon-config 2021-03-24T22:04:55.117 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 22:04:54 smithi142 bash[90036]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5/block 2021-03-24T22:04:55.117 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 22:04:54 smithi142 bash[90036]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-5/block 2021-03-24T22:04:55.117 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 22:04:54 smithi142 bash[90036]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2021-03-24T22:04:55.118 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 22:04:54 smithi142 bash[90036]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2021-03-24T22:04:55.118 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 22:04:54 smithi142 bash[90036]: --> ceph-volume lvm activate successful for osd ID: 5 2021-03-24T22:04:55.155 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:55.213 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:55.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:55.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:55.319 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:55.377 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:55.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:55.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:55.393 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-24T22:04:55.490 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:55.504 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 22:04:55 smithi142 podman[90036]: 2021-03-24 22:04:55.272902234 +0000 UTC m=+1.199558564 container died 5dca950bfcb6dfa932d50d4a1d8d7f91610120a9d37b0ecf319f7c06c44596be (image=quay.ceph.io/ceph-ci/ceph@sha256:e9db06906b8d0a24f3ef6835ac5aa4b8eea2950a2560bad78f83d4f101eca0db, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5-activate) 2021-03-24T22:04:55.538 INFO:tasks.rados.rados.0.smithi125.stdout:2786: done (4 left) 2021-03-24T22:04:55.539 INFO:tasks.rados.rados.0.smithi125.stdout:2787: done (3 left) 2021-03-24T22:04:55.539 INFO:tasks.rados.rados.0.smithi125.stdout:2788: done (2 left) 2021-03-24T22:04:55.539 INFO:tasks.rados.rados.0.smithi125.stdout:2789: done (1 left) 2021-03-24T22:04:55.540 INFO:tasks.rados.rados.0.smithi125.stdout:2790: done (0 left) 2021-03-24T22:04:55.540 INFO:tasks.rados.rados.0.smithi125.stdout:2791: rollback oid 24 current snap is 319 2021-03-24T22:04:55.540 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 24 to 296 2021-03-24T22:04:55.540 INFO:tasks.rados.rados.0.smithi125.stdout:2792: read oid 50 snap -1 2021-03-24T22:04:55.540 INFO:tasks.rados.rados.0.smithi125.stdout:2792: expect (ObjNum 852 snap 278 seq_num 852) 2021-03-24T22:04:55.541 INFO:tasks.rados.rados.0.smithi125.stdout:2793: copy_from oid 48 from oid 4 current snap is 319 2021-03-24T22:04:55.541 INFO:tasks.rados.rados.0.smithi125.stdout:2794: rollback oid 37 current snap is 319 2021-03-24T22:04:55.541 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 37 to 303 2021-03-24T22:04:55.541 INFO:tasks.rados.rados.0.smithi125.stdout:2795: delete oid 3 current snap is 319 2021-03-24T22:04:55.547 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:55.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:55.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:04:55.553 INFO:tasks.rados.rados.0.smithi125.stdout:2791: finishing rollback tid 0 to smithi12556470-24 2021-03-24T22:04:55.553 INFO:tasks.rados.rados.0.smithi125.stdout:2794: finishing rollback tid 1 to smithi12556470-37 2021-03-24T22:04:55.553 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 37 v 737 (ObjNum 1116408000 snap 21916 seq_num 3154118864) dirty dne 2021-03-24T22:04:55.559 INFO:tasks.rados.rados.0.smithi125.stdout:2792: expect (ObjNum 852 snap 278 seq_num 852) 2021-03-24T22:04:55.652 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:55.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[19936]: cluster 2021-03-24T22:04:55.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[19936]: 2021-03-24T22:04:55.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[19936]: 2021 2021-03-24T22:04:55.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[19936]: - 2021-03-24T22:04:55.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[19936]: 03 2021-03-24T22:04:55.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[19936]: - 2021-03-24T22:04:55.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[19936]: 24T22:04:54.519927+0000 mon.a (mon. 2021-03-24T22:04:55.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[19936]: 0) 1881 : cluster [DBG] osdmap e724: 8 total, 7 up, 8 in 2021-03-24T22:04:55.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[19936]: cluster 2021-03-24T22:04:55.368960+0000 2021-03-24T22:04:55.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[19936]: mon.a (mon.0) 1882 : cluster [WRN] Health check update: Degraded data redundancy: 160/1923 objects degraded (8.320%), 19 pgs degraded (PG_DEGRADED) 2021-03-24T22:04:55.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[19936]: 2021-03-24T22:04:55.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[24927]: cluster 2021-03-24T22:04:55.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[24927]: 2021-03-24T22:04:55.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[24927]: 2021 2021-03-24T22:04:55.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[24927]: - 2021-03-24T22:04:55.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[24927]: 03 2021-03-24T22:04:55.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[24927]: - 2021-03-24T22:04:55.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[24927]: 24 2021-03-24T22:04:55.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[24927]: T 2021-03-24T22:04:55.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[24927]: 22 2021-03-24T22:04:55.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[24927]: : 2021-03-24T22:04:55.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[24927]: 04 2021-03-24T22:04:55.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[24927]: : 2021-03-24T22:04:55.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[24927]: 54 2021-03-24T22:04:55.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[24927]: . 2021-03-24T22:04:55.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[24927]: 519927 2021-03-24T22:04:55.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[24927]: +0000 2021-03-24T22:04:55.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[24927]: 2021-03-24T22:04:55.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[24927]: mon.a 2021-03-24T22:04:55.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[24927]: ( 2021-03-24T22:04:55.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[24927]: mon 2021-03-24T22:04:55.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[24927]: . 2021-03-24T22:04:55.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[24927]: 0 2021-03-24T22:04:55.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[24927]: ) 2021-03-24T22:04:55.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[24927]: 1881 2021-03-24T22:04:55.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[24927]: : 2021-03-24T22:04:55.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[24927]: cluster 2021-03-24T22:04:55.672 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[24927]: 2021-03-24T22:04:55.672 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[24927]: [DBG] 2021-03-24T22:04:55.672 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[24927]: 2021-03-24T22:04:55.673 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[24927]: osdmap e724: 8 total, 7 up, 8 in 2021-03-24T22:04:55.673 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[24927]: 2021-03-24T22:04:55.673 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[24927]: cluster 2021-03-24T22:04:55.673 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[24927]: 2021-03-24T22:04:55.674 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[24927]: 2021 2021-03-24T22:04:55.674 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[24927]: - 2021-03-24T22:04:55.674 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[24927]: 03 2021-03-24T22:04:55.674 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[24927]: - 2021-03-24T22:04:55.674 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[24927]: 24 2021-03-24T22:04:55.675 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[24927]: T 2021-03-24T22:04:55.675 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[24927]: 22 2021-03-24T22:04:55.675 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[24927]: : 2021-03-24T22:04:55.675 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[24927]: 04 2021-03-24T22:04:55.675 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[24927]: : 2021-03-24T22:04:55.676 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[24927]: 55 2021-03-24T22:04:55.676 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[24927]: . 2021-03-24T22:04:55.676 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[24927]: 368960 2021-03-24T22:04:55.676 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[24927]: +0000 2021-03-24T22:04:55.676 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[24927]: 2021-03-24T22:04:55.677 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[24927]: mon.a 2021-03-24T22:04:55.677 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[24927]: (mon.0) 1882 2021-03-24T22:04:55.677 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:55 smithi125 conmon[24927]: : cluster [WRN] Health check update: Degraded data redundancy: 160/1923 objects degraded (8.320%), 19 pgs degraded (PG_DEGRADED) 2021-03-24T22:04:55.704 INFO:tasks.rados.rados.0.smithi125.stdout:2793: finishing copy_from to smithi12556470-48 2021-03-24T22:04:55.704 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 48 v 552 (ObjNum 865 snap 281 seq_num 865) dirty exists 2021-03-24T22:04:55.705 INFO:tasks.rados.rados.0.smithi125.stdout:2793: finishing copy_from racing read to smithi12556470-48 2021-03-24T22:04:55.705 INFO:tasks.rados.rados.0.smithi125.stdout:2791: finishing rollback tid 1 to smithi12556470-24 2021-03-24T22:04:55.705 INFO:tasks.rados.rados.0.smithi125.stdout:2791: finishing rollback tid 2 to smithi12556470-24 2021-03-24T22:04:55.705 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 24 v 1434 (ObjNum 851 snap 278 seq_num 851) dirty exists 2021-03-24T22:04:55.706 INFO:tasks.rados.rados.0.smithi125.stdout:2791: done (4 left) 2021-03-24T22:04:55.706 INFO:tasks.rados.rados.0.smithi125.stdout:2792: done (3 left) 2021-03-24T22:04:55.706 INFO:tasks.rados.rados.0.smithi125.stdout:2793: done (2 left) 2021-03-24T22:04:55.706 INFO:tasks.rados.rados.0.smithi125.stdout:2794: done (1 left) 2021-03-24T22:04:55.707 INFO:tasks.rados.rados.0.smithi125.stdout:2795: done (0 left) 2021-03-24T22:04:55.707 INFO:tasks.rados.rados.0.smithi125.stdout:2796: rollback oid 39 current snap is 319 2021-03-24T22:04:55.707 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 39 to 288 2021-03-24T22:04:55.707 INFO:tasks.rados.rados.0.smithi125.stdout:2797: delete oid 30 current snap is 319 2021-03-24T22:04:55.708 INFO:tasks.rados.rados.0.smithi125.stdout:2797: done (1 left) 2021-03-24T22:04:55.708 INFO:tasks.rados.rados.0.smithi125.stdout:2798: delete oid 32 current snap is 319 2021-03-24T22:04:55.708 INFO:tasks.rados.rados.0.smithi125.stdout:2796: finishing rollback tid 1 to smithi12556470-39 2021-03-24T22:04:55.709 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 39 v 1034 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-03-24T22:04:55.710 INFO:tasks.rados.rados.0.smithi125.stdout:2796: done (1 left) 2021-03-24T22:04:55.710 INFO:tasks.rados.rados.0.smithi125.stdout:2798: done (0 left) 2021-03-24T22:04:55.711 INFO:tasks.rados.rados.0.smithi125.stdout:2799: write (excl) oid 37 current snap is 319 2021-03-24T22:04:55.711 INFO:tasks.rados.rados.0.smithi125.stdout:2799: seq_num 978 ranges {437246=486706,1363860=623668} 2021-03-24T22:04:55.716 INFO:tasks.rados.rados.0.smithi125.stdout:2799: writing smithi12556470-37 from 437246 to 923952 tid 1 2021-03-24T22:04:55.723 INFO:tasks.rados.rados.0.smithi125.stdout:2799: writing smithi12556470-37 from 1363860 to 1987528 tid 2 2021-03-24T22:04:55.723 INFO:tasks.rados.rados.0.smithi125.stdout:2800: write (excl) oid 4 current snap is 319 2021-03-24T22:04:55.723 INFO:tasks.rados.rados.0.smithi125.stdout:2799: finishing write tid 1 to smithi12556470-37 2021-03-24T22:04:55.724 INFO:tasks.rados.rados.0.smithi125.stdout:2800: seq_num 979 ranges {454053=561955,1422342=483679,2368404=85649} 2021-03-24T22:04:55.724 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:55.728 INFO:tasks.rados.rados.0.smithi125.stdout:2800: writing smithi12556470-4 from 454053 to 1016008 tid 1 2021-03-24T22:04:55.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:55.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:04:55.734 INFO:tasks.rados.rados.0.smithi125.stdout:2800: writing smithi12556470-4 from 1422342 to 1906021 tid 2 2021-03-24T22:04:55.737 INFO:tasks.rados.rados.0.smithi125.stdout:2800: writing smithi12556470-4 from 2368404 to 2454053 tid 3 2021-03-24T22:04:55.737 INFO:tasks.rados.rados.0.smithi125.stdout:2801: snap_create 2021-03-24T22:04:55.738 INFO:tasks.rados.rados.0.smithi125.stdout:2799: finishing write tid 2 to smithi12556470-37 2021-03-24T22:04:55.738 INFO:tasks.rados.rados.0.smithi125.stdout:2799: finishing write tid 3 to smithi12556470-37 2021-03-24T22:04:55.738 INFO:tasks.rados.rados.0.smithi125.stdout:2799: finishing write tid 4 to smithi12556470-37 2021-03-24T22:04:55.738 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 37 v 740 (ObjNum 978 snap 319 seq_num 978) dirty exists 2021-03-24T22:04:55.739 INFO:tasks.rados.rados.0.smithi125.stdout:2799: left oid 37 (ObjNum 978 snap 319 seq_num 978) 2021-03-24T22:04:55.739 INFO:tasks.rados.rados.0.smithi125.stdout:2800: finishing write tid 1 to smithi12556470-4 2021-03-24T22:04:55.739 INFO:tasks.rados.rados.0.smithi125.stdout:2800: finishing write tid 2 to smithi12556470-4 2021-03-24T22:04:55.740 INFO:tasks.rados.rados.0.smithi125.stdout:2800: finishing write tid 3 to smithi12556470-4 2021-03-24T22:04:55.740 INFO:tasks.rados.rados.0.smithi125.stdout:2800: finishing write tid 4 to smithi12556470-4 2021-03-24T22:04:55.741 INFO:tasks.rados.rados.0.smithi125.stdout:2800: finishing write tid 5 to smithi12556470-4 2021-03-24T22:04:55.741 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 4 v 622 (ObjNum 979 snap 319 seq_num 979) dirty exists 2021-03-24T22:04:55.741 INFO:tasks.rados.rados.0.smithi125.stdout:2800: left oid 4 (ObjNum 979 snap 319 seq_num 979) 2021-03-24T22:04:55.802 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:04:55.831 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:55.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:55 smithi142 conmon[20125]: cluster 2021-03-24T22:04:55.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:55 smithi142 conmon[20125]: 2021-03-24T22:04:55.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:55 smithi142 conmon[20125]: 2021 2021-03-24T22:04:55.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:55 smithi142 conmon[20125]: -03-24T22: 2021-03-24T22:04:55.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:55 smithi142 conmon[20125]: 04:54.519927+0000 mon.a (mon.0) 1881 : cluster [DBG] osdmap e724: 8 total, 7 up, 8 in 2021-03-24T22:04:55.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:55 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:04:55.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:55 smithi142 conmon[20125]: 03-24T22:04:55.368960+0000 mon.a (mon. 2021-03-24T22:04:55.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:55 smithi142 conmon[20125]: 0) 1882 : cluster [WRN] Health check update: Degraded data redundancy: 160/1923 objects degraded (8.320%), 19 pgs degraded (PG_DEGRADED) 2021-03-24T22:04:55.876 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:55.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:55.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:55.984 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:56.045 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:56.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:56.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:04:56.152 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:56.198 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:56.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:56.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:56.303 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:56.357 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:56.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:56.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:56.462 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:56.518 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:56.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:56.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:56.553 INFO:tasks.rados.rados.0.smithi125.stdout:2799: done (2 left) 2021-03-24T22:04:56.554 INFO:tasks.rados.rados.0.smithi125.stdout:2800: done (1 left) 2021-03-24T22:04:56.554 INFO:tasks.rados.rados.0.smithi125.stdout:2801: done (0 left) 2021-03-24T22:04:56.554 INFO:tasks.rados.rados.0.smithi125.stdout:2802: delete oid 30 current snap is 320 2021-03-24T22:04:56.558 INFO:teuthology.orchestra.run.smithi125.stderr:noscrub is unset 2021-03-24T22:04:56.560 INFO:tasks.rados.rados.0.smithi125.stdout:2802: done (0 left) 2021-03-24T22:04:56.561 INFO:tasks.rados.rados.0.smithi125.stdout:2803: copy_from oid 50 from oid 48 current snap is 320 2021-03-24T22:04:56.561 INFO:tasks.rados.rados.0.smithi125.stdout:2804: snap_create 2021-03-24T22:04:56.596 INFO:tasks.rados.rados.0.smithi125.stdout:2803: finishing copy_from to smithi12556470-50 2021-03-24T22:04:56.597 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 50 v 552 (ObjNum 865 snap 281 seq_num 865) dirty exists 2021-03-24T22:04:56.597 INFO:tasks.rados.rados.0.smithi125.stdout:2803: finishing copy_from racing read to smithi12556470-50 2021-03-24T22:04:56.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:56 smithi142 conmon[20125]: cluster 2021-03-24T22 2021-03-24T22:04:56.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:56 smithi142 conmon[20125]: :04:54.966722+0000 mgr.y (mgr.14144) 637 : cluster [DBG] pgmap v1249: 168 pgs: 1 active+clean+remapped, 1 peering, 40 active+undersized, 19 active+undersized+degraded, 107 active+clean; 683 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 6.5 MiB/s rd, 665 KiB/s wr, 12 op/s; 161/1917 objects degraded (8.399%); 20/1917 objects misplaced (1.043%) 2021-03-24T22:04:56.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:56 smithi142 conmon[20125]: cluster 2021-03-24T22:04:56.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:56 smithi142 conmon[20125]: 2021-03-24T22:04:55.528141+0000 2021-03-24T22:04:56.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:56 smithi142 conmon[20125]: mon.a (mon.0) 1883 : cluster 2021-03-24T22:04:56.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:56 smithi142 conmon[20125]: [DBG] osdmap e725: 8 total, 7 up, 8 in 2021-03-24T22:04:56.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:56 smithi142 conmon[20125]: audit 2021- 2021-03-24T22:04:56.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:56 smithi142 conmon[20125]: 03-24T22:04:56 2021-03-24T22:04:56.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:56 smithi142 conmon[20125]: .304670+0000 mon.c (mon.2 2021-03-24T22:04:56.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:56 smithi142 conmon[20125]: ) 105 : audit [INF] from='client.? v1:172.21.15.125:0/2737697106' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T22:04:56.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:56 smithi142 conmon[20125]: audit 2021 2021-03-24T22:04:56.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:56 smithi142 conmon[20125]: -03-24T22:04: 2021-03-24T22:04:56.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:56 smithi142 conmon[20125]: 56.305496+0000 mon.a (mon 2021-03-24T22:04:56.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:56 smithi142 conmon[20125]: .0) 1884 : audit [INF] 2021-03-24T22:04:56.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:56 smithi142 conmon[20125]: from='client.35711 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T22:04:56.625 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:56.680 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:56.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:56.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:56.785 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:56.798 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:56 smithi125 conmon[24927]: cluster 2021-03-24T22:04:56.798 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:56 smithi125 conmon[24927]: 2021-03-24T22:04 2021-03-24T22:04:56.799 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:56 smithi125 conmon[24927]: :54.966722+0000 mgr.y (mgr.14144) 637 : cluster [DBG] pgmap v1249: 168 pgs: 1 active+clean+remapped, 1 peering, 40 active+undersized, 19 active+undersized+degraded, 107 active+clean; 683 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 6.5 MiB/s rd, 665 KiB/s wr, 12 op/s; 161/1917 objects degraded (8.399%); 20/1917 objects misplaced (1.043%) 2021-03-24T22:04:56.800 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:56 smithi125 conmon[24927]: cluster 2021-03-24T22:04: 2021-03-24T22:04:56.800 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:56 smithi125 conmon[24927]: 55.528141+0000 mon.a (mon.0) 1883 2021-03-24T22:04:56.800 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:56 smithi125 conmon[24927]: : cluster [DBG] osdmap e725: 8 total, 7 up, 8 in 2021-03-24T22:04:56.800 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:56 smithi125 conmon[24927]: audit 2021-03-24 2021-03-24T22:04:56.800 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:56 smithi125 conmon[24927]: T22:04:56.304670+0000 mon.c ( 2021-03-24T22:04:56.801 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:56 smithi125 conmon[24927]: mon.2) 105 : audit [INF] from='client.? v1:172.21.15.125:0/2737697106' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T22:04:56.801 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:56 smithi125 conmon[24927]: audit 2021-03 2021-03-24T22:04:56.801 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:56 smithi125 conmon[24927]: -24T22:04:56.305496+0000 2021-03-24T22:04:56.801 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:56 smithi125 conmon[24927]: mon.a (mon.0) 1884 : audit [INF] 2021-03-24T22:04:56.802 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:56 smithi125 conmon[24927]: from='client.35711 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T22:04:56.802 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:56 smithi125 conmon[19936]: cluster 2021-03-24T22:04:54.966722+0000 mgr.y (mgr.14144 2021-03-24T22:04:56.803 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:56 smithi125 conmon[19936]: ) 637 : cluster [DBG] pgmap v1249: 168 pgs: 1 active+clean+remapped, 1 peering, 40 active+undersized, 19 active+undersized+degraded, 107 active+clean; 683 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 6.5 MiB/s rd, 665 KiB/s wr, 12 op/s; 161/1917 objects degraded (8.399%); 20/1917 objects misplaced (1.043%) 2021-03-24T22:04:56.804 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:56 smithi125 conmon[19936]: cluster 2021-03-24T22:04:55.528141+0000 mon.a (mon.0) 1883 : cluster [DBG] 2021-03-24T22:04:56.804 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:56 smithi125 conmon[19936]: osdmap e725: 8 total, 7 up, 8 in 2021-03-24T22:04:56.805 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:56 smithi125 conmon[19936]: audit 2021-03-24T22:04:56.304670+0000 mon.c (mon.2) 105 : audit [INF] from='client.? v1:172.21.15.125:0/2737697106' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T22:04:56.805 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:56 smithi125 conmon[19936]: audit 2021-03-24T 2021-03-24T22:04:56.806 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:56 smithi125 conmon[19936]: 22:04:56.305496+0000 2021-03-24T22:04:56.806 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:56 smithi125 conmon[19936]: mon.a (mon.0) 1884 : audit 2021-03-24T22:04:56.806 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:56 smithi125 conmon[19936]: [INF] from='client.35711 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T22:04:56.834 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:56.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:56.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:56.939 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:56.994 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:56.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:57.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:57.101 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:57.162 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:57.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:57.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:57.234 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 22:04:57 smithi142 podman[90258]: 2021-03-24 22:04:57.15129944 +0000 UTC m=+2.146165459 container remove 5dca950bfcb6dfa932d50d4a1d8d7f91610120a9d37b0ecf319f7c06c44596be (image=quay.ceph.io/ceph-ci/ceph@sha256:e9db06906b8d0a24f3ef6835ac5aa4b8eea2950a2560bad78f83d4f101eca0db, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5-activate, ceph=True, org.label-schema.schema-version=1.0, maintainer=Dimitri Savineau , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Base Image, GIT_BRANCH=HEAD, RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.18.0, org.label-schema.build-date=20201204, GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06) 2021-03-24T22:04:57.268 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:57.315 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:04:57.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:57.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:04:57.329 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2021-03-24T22:04:57.329 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T22:04:57.422 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:57.470 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:04:57.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:57.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:04:57.570 INFO:tasks.rados.rados.0.smithi125.stdout:2803: done (1 left) 2021-03-24T22:04:57.570 INFO:tasks.rados.rados.0.smithi125.stdout:2804: done (0 left) 2021-03-24T22:04:57.570 INFO:tasks.rados.rados.0.smithi125.stdout:2805: snap_remove snap 300 2021-03-24T22:04:57.574 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:57.620 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:57.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:57.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:57.726 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:57.775 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:57.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:57.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:57.866 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 22:04:57 smithi142 podman[90539]: 2021-03-24 22:04:57.752363611 +0000 UTC m=+0.376226274 container create 5672232733c58057b68bfeaf58a3c61c1a03c031ef72e4ee36883f3729cbdc76 (image=quay.ceph.io/ceph-ci/ceph@sha256:e9db06906b8d0a24f3ef6835ac5aa4b8eea2950a2560bad78f83d4f101eca0db, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5, GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.18.0, RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d, org.label-schema.schema-version=1.0, org.label-schema.build-date=20201204, maintainer=Dimitri Savineau , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2021-03-24T22:04:57.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:57 smithi142 conmon[20125]: cluster 2021-03-24T22:04: 2021-03-24T22:04:57.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:57 smithi142 conmon[20125]: 56.524656+0000 mon.a (mon.0) 1885 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:04:57.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:57 smithi142 conmon[20125]: audit 2021-03-24T22:04:56 2021-03-24T22:04:57.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:57 smithi142 conmon[20125]: .544770+0000 mon.a (mon.0) 1886 : audit [INF] from='client.35711 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T22:04:57.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:57 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:04:57.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:57 smithi142 conmon[20125]: 24T22:04:56.544871+0000 mon.a (mon. 2021-03-24T22:04:57.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:57 smithi142 conmon[20125]: 0) 1887 : cluster [DBG] osdmap e726: 8 total, 7 up, 8 in 2021-03-24T22:04:57.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:57 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:04:57.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:57 smithi142 conmon[20125]: -03-24T22:04:56.575712+0000 osd.1 ( 2021-03-24T22:04:57.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:57 smithi142 conmon[20125]: osd.1) 75 : cluster [DBG] 6.1 scrub ok 2021-03-24T22:04:57.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:57 smithi142 conmon[20125]: cluster 2021-03-24T22:04:57.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:57 smithi142 conmon[20125]: 2021-03-24 2021-03-24T22:04:57.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:57 smithi142 conmon[20125]: T22:04:56.628996+0000 osd.2 (osd. 2021-03-24T22:04:57.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:57 smithi142 conmon[20125]: 2) 113 : cluster [DBG] 7.20 scrub ok 2021-03-24T22:04:57.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:57 smithi142 conmon[20125]: cluster 2021-03-24T22:04:57.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:57 smithi142 conmon[20125]: 2021-03-24T22:04:57.253398 2021-03-24T22:04:57.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:57 smithi142 conmon[20125]: +0000 osd.3 (osd.3) 144 : cluster [DBG] 2021-03-24T22:04:57.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:57 smithi142 conmon[20125]: 7.3 scrub ok 2021-03-24T22:04:57.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:57 smithi142 conmon[20125]: cluster 2021-03-24T22 2021-03-24T22:04:57.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:57 smithi142 conmon[20125]: :04:57.279508+0000 osd.0 (osd.0) 2021-03-24T22:04:57.872 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:57 smithi142 conmon[20125]: 142 : cluster [DBG] 3.12 scrub ok 2021-03-24T22:04:57.881 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:57.894 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:57 smithi125 conmon[19936]: cluster 2021-03-24T22:04:57.894 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:57 smithi125 conmon[19936]: 2021-03-24T22 2021-03-24T22:04:57.895 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:57 smithi125 conmon[19936]: :04:56.524656+0000 mon.a (mon.0) 1885 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:04:57.895 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:57 smithi125 conmon[19936]: audit 2021- 2021-03-24T22:04:57.895 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:57 smithi125 conmon[19936]: 03-24T22:04:56.544770+0000 2021-03-24T22:04:57.896 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:57 smithi125 conmon[19936]: mon.a (mon.0) 1886 : 2021-03-24T22:04:57.896 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:57 smithi125 conmon[19936]: audit [INF] from='client.35711 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T22:04:57.896 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:57 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:04:57.896 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:57 smithi125 conmon[19936]: 03-24T22:04:56.544871+0000 mon.a (mon.0) 1887 : cluster 2021-03-24T22:04:57.896 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:57 smithi125 conmon[19936]: [DBG] osdmap e726: 8 total, 7 up, 8 in 2021-03-24T22:04:57.897 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:57 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T22:04:57.897 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:57 smithi125 conmon[19936]: 22:04:56.575712+0000 osd.1 (osd.1) 75 : cluster [DBG] 6.1 scrub ok 2021-03-24T22:04:57.897 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:57 smithi125 conmon[19936]: cluster 2021-03-24T22:04:57.897 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:57 smithi125 conmon[19936]: 2021-03-24T22:04:56.628996+0000 osd.2 (osd.2) 113 : cluster [DBG] 7.20 scrub ok 2021-03-24T22:04:57.897 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:57 smithi125 conmon[19936]: cluster 2021-03-24T22:04:57.253398+0000 osd.3 (osd.3 2021-03-24T22:04:57.898 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:57 smithi125 conmon[19936]: ) 144 : cluster [DBG] 7.3 scrub ok 2021-03-24T22:04:57.898 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:57 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:04:57.898 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:57 smithi125 conmon[19936]: 24T22:04:57.279508+0000 osd.0 (osd.0) 142 : cluster [DBG] 3.12 scrub ok 2021-03-24T22:04:57.899 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:57 smithi125 conmon[24927]: cluster 2021-03-24T22:04:56 2021-03-24T22:04:57.899 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:57 smithi125 conmon[24927]: .524656+0000 mon.a (mon.0) 1885 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:04:57.900 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:57 smithi125 conmon[24927]: audit 2021-03-24T 2021-03-24T22:04:57.900 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:57 smithi125 conmon[24927]: 22:04:56.544770+0000 mon.a ( 2021-03-24T22:04:57.900 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:57 smithi125 conmon[24927]: mon.0) 1886 : audit [INF] from='client.35711 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T22:04:57.900 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:57 smithi125 conmon[24927]: 2021-03-24T22:04:57.901 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:57 smithi125 conmon[24927]: cluster 2021-03-24T22:04:57.901 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:57 smithi125 conmon[24927]: 2021-03-24T22:04:56.544871 2021-03-24T22:04:57.901 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:57 smithi125 conmon[24927]: +0000 mon.a (mon.0) 1887 : cluster 2021-03-24T22:04:57.902 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:57 smithi125 conmon[24927]: [DBG] osdmap e726: 8 total, 7 up, 8 in 2021-03-24T22:04:57.902 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:57 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T22:04:57.902 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:57 smithi125 conmon[24927]: -24T22:04:56.575712 2021-03-24T22:04:57.902 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:57 smithi125 conmon[24927]: +0000 osd.1 (osd.1) 75 : 2021-03-24T22:04:57.903 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:57 smithi125 conmon[24927]: cluster [DBG] 6.1 scrub ok 2021-03-24T22:04:57.903 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:57 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T22:04:57.903 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:57 smithi125 conmon[24927]: 22:04:56.628996+0000 osd.2 (osd 2021-03-24T22:04:57.904 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:57 smithi125 conmon[24927]: .2) 113 : cluster [DBG] 7.20 scrub ok 2021-03-24T22:04:57.904 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:57 smithi125 conmon[24927]: 2021-03-24T22:04:57.904 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:57 smithi125 conmon[24927]: cluster 2021-03-24T22:04:57.905 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:57 smithi125 conmon[24927]: 2021-03-24T22:04:57.253398 2021-03-24T22:04:57.905 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:57 smithi125 conmon[24927]: +0000 osd.3 (osd.3) 144 : cluster 2021-03-24T22:04:57.905 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:57 smithi125 conmon[24927]: [DBG] 7.3 scrub ok 2021-03-24T22:04:57.906 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:57 smithi125 conmon[24927]: cluster 2021-03-24T22:04:57.906 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:57 smithi125 conmon[24927]: 2021-03-24T22:04:57.279508 2021-03-24T22:04:57.906 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:57 smithi125 conmon[24927]: +0000 osd.0 (osd.0) 142 : cluster 2021-03-24T22:04:57.907 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:57 smithi125 conmon[24927]: [DBG] 3.12 scrub ok 2021-03-24T22:04:57.943 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:57.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:57.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:58.048 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:58.108 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:58.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:58.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:58.215 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:58.280 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:58.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:58.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:58.385 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:58.450 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:58.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:58.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:04:58.556 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:58.567 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:58 smithi125 conmon[19936]: cluster 2021-03-24T22:04:56.968060+0000 2021-03-24T22:04:58.571 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:58 smithi125 conmon[19936]: mgr.y (mgr.14144) 638 : cluster [DBG] pgmap v1252: 168 pgs: 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 108 active+clean; 690 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 8.9 MiB/s rd, 2.3 MiB/s wr, 23 op/s; 165/1941 objects degraded (8.501%); 20/1941 objects misplaced (1.030%) 2021-03-24T22:04:58.572 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:58 smithi125 conmon[19936]: cluster 2021-03-24T22:04:57. 2021-03-24T22:04:58.572 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:58 smithi125 conmon[19936]: 567189+0000 mon.a (mon.0) 1888 : cluster [DBG] osdmap e727: 8 total, 7 up, 8 in 2021-03-24T22:04:58.573 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:58 smithi125 conmon[19936]: cluster 2021-03-24T22:04 2021-03-24T22:04:58.573 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:58 smithi125 conmon[19936]: :58.198515+0000 osd.7 (osd.7) 121 : 2021-03-24T22:04:58.573 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:58 smithi125 conmon[19936]: cluster [DBG] 7.5 scrub ok 2021-03-24T22:04:58.592 INFO:tasks.rados.rados.0.smithi125.stdout:2805: done (0 left) 2021-03-24T22:04:58.592 INFO:tasks.rados.rados.0.smithi125.stdout:2806: read oid 9 snap 304 2021-03-24T22:04:58.592 INFO:tasks.rados.rados.0.smithi125.stdout:2806: expect deleted 2021-03-24T22:04:58.593 INFO:tasks.rados.rados.0.smithi125.stdout:2807: delete oid 28 current snap is 321 2021-03-24T22:04:58.607 INFO:tasks.rados.rados.0.smithi125.stdout:2807: done (1 left) 2021-03-24T22:04:58.608 INFO:tasks.rados.rados.0.smithi125.stdout:2808: delete oid 42 current snap is 321 2021-03-24T22:04:58.613 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.7 dump_historic_ops 2021-03-24T22:04:58.626 INFO:tasks.rados.rados.0.smithi125.stdout:2806: done (1 left) 2021-03-24T22:04:58.627 INFO:tasks.rados.rados.0.smithi125.stdout:2808: done (0 left) 2021-03-24T22:04:58.627 INFO:tasks.rados.rados.0.smithi125.stdout:2809: snap_remove snap 269 2021-03-24T22:04:58.654 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:04:58.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:58.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:04:58.698 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.5 dump_ops_in_flight 2021-03-24T22:04:58.766 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:58.817 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:04:58.820 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:58 smithi125 conmon[24927]: cluster 2021-03-24T22:04:58.820 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:58 smithi125 conmon[24927]: 2021-03-24T22:04:58.820 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:58 smithi125 conmon[24927]: 2021 2021-03-24T22:04:58.821 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:58 smithi125 conmon[24927]: - 2021-03-24T22:04:58.821 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:58 smithi125 conmon[24927]: 03 2021-03-24T22:04:58.821 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:58 smithi125 conmon[24927]: - 2021-03-24T22:04:58.822 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:58 smithi125 conmon[24927]: 24 2021-03-24T22:04:58.822 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:58 smithi125 conmon[24927]: T 2021-03-24T22:04:58.822 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:58 smithi125 conmon[24927]: 22 2021-03-24T22:04:58.822 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:58 smithi125 conmon[24927]: : 2021-03-24T22:04:58.822 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:58 smithi125 conmon[24927]: 04 2021-03-24T22:04:58.823 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:58 smithi125 conmon[24927]: : 2021-03-24T22:04:58.823 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:58 smithi125 conmon[24927]: 56 2021-03-24T22:04:58.823 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:58 smithi125 conmon[24927]: . 2021-03-24T22:04:58.823 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:58 smithi125 conmon[24927]: 968060 2021-03-24T22:04:58.824 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:58 smithi125 conmon[24927]: +0000 2021-03-24T22:04:58.824 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:58 smithi125 conmon[24927]: 2021-03-24T22:04:58.824 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:58 smithi125 conmon[24927]: mgr.y 2021-03-24T22:04:58.824 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:58 smithi125 conmon[24927]: ( 2021-03-24T22:04:58.824 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:58 smithi125 conmon[24927]: mgr 2021-03-24T22:04:58.825 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:58 smithi125 conmon[24927]: . 2021-03-24T22:04:58.825 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:58 smithi125 conmon[24927]: 14144 2021-03-24T22:04:58.825 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:58 smithi125 conmon[24927]: ) 2021-03-24T22:04:58.825 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:58 smithi125 conmon[24927]: 638 2021-03-24T22:04:58.826 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:58 smithi125 conmon[24927]: : 2021-03-24T22:04:58.826 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:58 smithi125 conmon[24927]: cluster 2021-03-24T22:04:58.826 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:58 smithi125 conmon[24927]: [DBG] pgmap v1252: 168 pgs: 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 108 active+clean; 690 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 8.9 MiB/s rd, 2.3 MiB/s wr, 23 op/s; 165/1941 objects degraded (8.501%); 20/1941 objects misplaced (1.030%) 2021-03-24T22:04:58.826 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:58 smithi125 conmon[24927]: cluster 2021-03-24T22:04:57.567189+0000 mon.a (mon.0) 1888 : cluster [DBG] osdmap e727: 8 total, 7 up, 8 in 2021-03-24T22:04:58.827 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:58 smithi125 conmon[24927]: cluster 2021-03-24T22:04:58.198515+0000 osd.7 (osd.7) 121 : cluster [DBG] 7.5 scrub ok 2021-03-24T22:04:58.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:58.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:04:58.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:58 smithi142 conmon[20125]: cluster 2021-03-24T22:04:56.968060+0000 mgr.y (mgr 2021-03-24T22:04:58.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:58 smithi142 conmon[20125]: .14144) 638 : cluster [DBG] pgmap v1252: 168 pgs: 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 108 active+clean; 690 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 8.9 MiB/s rd, 2.3 MiB/s wr, 23 op/s; 165/1941 objects degraded (8.501%); 20/1941 objects misplaced (1.030%) 2021-03-24T22:04:58.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:58 smithi142 conmon[20125]: cluster 2021-03-24T22:04:57.567189+0000 mon.a (mon.0) 1888 : cluster [DBG] osdmap e727: 8 total, 7 up, 8 in 2021-03-24T22:04:58.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:58 smithi142 conmon[20125]: cluster 2021-03-24T22:04:58.198515+0000 osd.7 (osd.7) 121 2021-03-24T22:04:58.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:58 smithi142 conmon[20125]: : cluster [DBG] 7.5 scrub ok 2021-03-24T22:04:58.869 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 22:04:58 smithi142 podman[90539]: 2021-03-24 22:04:58.651051339 +0000 UTC m=+1.274914038 container init 5672232733c58057b68bfeaf58a3c61c1a03c031ef72e4ee36883f3729cbdc76 (image=quay.ceph.io/ceph-ci/ceph@sha256:e9db06906b8d0a24f3ef6835ac5aa4b8eea2950a2560bad78f83d4f101eca0db, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5, maintainer=Dimitri Savineau , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d, org.label-schema.license=GPLv2, GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06, GIT_CLEAN=True, org.label-schema.build-date=20201204, ceph=True, org.label-schema.name=CentOS Base Image, io.buildah.version=1.18.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2021-03-24T22:04:58.870 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 22:04:58 smithi142 podman[90539]: 2021-03-24 22:04:58.69273703 +0000 UTC m=+1.316599720 container start 5672232733c58057b68bfeaf58a3c61c1a03c031ef72e4ee36883f3729cbdc76 (image=quay.ceph.io/ceph-ci/ceph@sha256:e9db06906b8d0a24f3ef6835ac5aa4b8eea2950a2560bad78f83d4f101eca0db, name=ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5, CEPH_POINT_RELEASE=, io.buildah.version=1.18.0, org.label-schema.name=CentOS Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=50c62733fe9962908c16f14743ac0d9988ccdf06, org.label-schema.build-date=20201204, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-sage2-testing-2021-03-24-1043-pacific-d71e96d, maintainer=Dimitri Savineau , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2) 2021-03-24T22:04:58.870 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 22:04:58 smithi142 bash[90539]: 5672232733c58057b68bfeaf58a3c61c1a03c031ef72e4ee36883f3729cbdc76 2021-03-24T22:04:58.871 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 22:04:58 smithi142 systemd[1]: Started Ceph osd.5 for 3cfa339a-8cea-11eb-90b2-001a4aab830c. 2021-03-24T22:04:58.930 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:58.933 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T22:04:58.974 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:58.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:58.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:59.005 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T22:04:59.080 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:59.138 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:59.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:59.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:59.243 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:59.292 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:59.293 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 22:04:58 smithi142 conmon[90637]: debug 2021-03-24T22:04:58.987+0000 7ff07f500080 -1 Falling back to public interface 2021-03-24T22:04:59.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:59.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:59.313 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T22:04:59.333 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T22:04:59.362 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T22:04:59.382 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T22:04:59.397 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:59.405 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T22:04:59.434 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T22:04:59.446 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:04:59.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:59.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:04:59.462 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T22:04:59.463 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-24T22:04:59.463 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-24T22:04:59.463 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-24T22:04:59.464 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-24T22:04:59.465 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-24T22:04:59.465 INFO:teuthology.orchestra.run.smithi125.stderr:Error ENXIO: problem getting command descriptions from osd.5 2021-03-24T22:04:59.465 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-24T22:04:59.465 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-24T22:04:59.552 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:59.589 INFO:tasks.rados.rados.0.smithi125.stdout:2809: done (0 left) 2021-03-24T22:04:59.589 INFO:tasks.rados.rados.0.smithi125.stdout:2810: snap_create 2021-03-24T22:04:59.632 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:59.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:59.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:04:59.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:59 smithi125 conmon[19936]: cluster 2021-03-24T22:04:58.577140 2021-03-24T22:04:59.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:59 smithi125 conmon[19936]: +0000 mon.a (mon.0) 1889 : cluster [DBG] osdmap e728: 8 total, 7 up, 8 in 2021-03-24T22:04:59.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:59 smithi125 conmon[19936]: cluster 2021-03-24T22:04:58.618367+0000 osd.1 (osd 2021-03-24T22:04:59.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:59 smithi125 conmon[19936]: .1) 76 : cluster [DBG] 4.4 scrub ok 2021-03-24T22:04:59.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:59 smithi125 conmon[19936]: audit 2021-03-24T22:04:59.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:59 smithi125 conmon[19936]: 2021-03-24T22:04:59.290489+0000 mon.a (mon. 2021-03-24T22:04:59.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:04:59 smithi125 conmon[19936]: 0) 1890 : audit [DBG] from='client.? v1:172.21.15.125:0/696413329' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:04:59.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:59 smithi125 conmon[24927]: cluster 2021-03-24T22:04:59.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:59 smithi125 conmon[24927]: 2021-03-24T22:04:59.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:59 smithi125 conmon[24927]: 2021 2021-03-24T22:04:59.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:59 smithi125 conmon[24927]: - 2021-03-24T22:04:59.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:59 smithi125 conmon[24927]: 03 2021-03-24T22:04:59.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:59 smithi125 conmon[24927]: -24T22:04:58.577140+0000 mon.a (mon 2021-03-24T22:04:59.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:59 smithi125 conmon[24927]: .0) 1889 : cluster [DBG] osdmap e728: 8 total, 7 up, 8 in 2021-03-24T22:04:59.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:59 smithi125 conmon[24927]: cluster 2021-03-24T22:04:58.618367+0000 osd.1 (osd. 2021-03-24T22:04:59.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:59 smithi125 conmon[24927]: 1) 76 : cluster [DBG] 4.4 scrub ok 2021-03-24T22:04:59.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:59 smithi125 conmon[24927]: audit 2021-03-24T22:04 2021-03-24T22:04:59.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:04:59 smithi125 conmon[24927]: :59.290489+0000 mon.a (mon.0) 1890 : audit [DBG] from='client.? v1:172.21.15.125:0/696413329' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:04:59.680 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:59 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:04:59.681 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:59 smithi142 conmon[20125]: 24T22:04:58.577140+0000 mon.a (mon.0) 1889 : cluster [DBG] osdmap e728: 8 total, 7 up, 8 in 2021-03-24T22:04:59.681 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:59 smithi142 conmon[20125]: cluster 2021-03-24T22:04:59.681 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:59 smithi142 conmon[20125]: 2021-03-24T22:04 2021-03-24T22:04:59.681 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:59 smithi142 conmon[20125]: :58.618367+0000 osd.1 ( 2021-03-24T22:04:59.682 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:59 smithi142 conmon[20125]: osd.1) 76 : cluster [DBG] 2021-03-24T22:04:59.682 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:59 smithi142 conmon[20125]: 4.4 scrub ok 2021-03-24T22:04:59.682 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:59 smithi142 conmon[20125]: audit 2021-03 2021-03-24T22:04:59.682 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:59 smithi142 conmon[20125]: -24T22:04:59.290489 2021-03-24T22:04:59.683 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:59 smithi142 conmon[20125]: +0000 mon.a (mon.0) 1890 : audit 2021-03-24T22:04:59.683 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:04:59 smithi142 conmon[20125]: [DBG] from='client.? v1:172.21.15.125:0/696413329' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:04:59.741 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:59.790 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:04:59.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:59.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:04:59.895 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:59.949 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:04:59.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:04:59.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:05:00.055 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:00.108 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:00.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:00.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:05:00.215 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:00.273 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:00.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:00.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:05:00.381 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:00.386 INFO:tasks.rados.rados.0.smithi125.stdout:2810: done (0 left) 2021-03-24T22:05:00.386 INFO:tasks.rados.rados.0.smithi125.stdout:2811: read oid 28 snap -1 2021-03-24T22:05:00.387 INFO:tasks.rados.rados.0.smithi125.stdout:2811: expect deleted 2021-03-24T22:05:00.387 INFO:tasks.rados.rados.0.smithi125.stdout:2812: write oid 21 current snap is 322 2021-03-24T22:05:00.388 INFO:tasks.rados.rados.0.smithi125.stdout:2812: seq_num 983 ranges {521281=579304,1656270=774664} 2021-03-24T22:05:00.394 INFO:tasks.rados.rados.0.smithi125.stdout:2812: writing smithi12556470-21 from 521281 to 1100585 tid 1 2021-03-24T22:05:00.404 INFO:tasks.rados.rados.0.smithi125.stdout:2812: writing smithi12556470-21 from 1656270 to 2430934 tid 2 2021-03-24T22:05:00.405 INFO:tasks.rados.rados.0.smithi125.stdout:2811: done (1 left) 2021-03-24T22:05:00.406 INFO:tasks.rados.rados.0.smithi125.stdout:2813: snap_remove snap 216 2021-03-24T22:05:00.439 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:00.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:00.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:05:00.549 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:00.605 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:00.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:00.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:05:00.710 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:00.717 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-03-24T22:05:00.718 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-03-24T22:05:00.766 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:00.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:00.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:05:00.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:00 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:05:00.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:00 smithi142 conmon[20125]: 24T22:04:58.969185+0000 mgr.y (mgr.14144) 639 : cluster [DBG] pgmap v1255: 168 pgs: 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 108 active+clean; 690 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 4.7 MiB/s rd, 1.6 MiB/s wr, 12 op/s; 165/1941 objects degraded (8.501%); 20/1941 objects misplaced (1.030%) 2021-03-24T22:05:00.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:00 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:05:00.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:00 smithi142 conmon[20125]: T22:04:59.471384+0000 2021-03-24T22:05:00.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:00 smithi142 conmon[20125]: osd.2 (osd.2) 114 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:05:00.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:00 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:05:00.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:00 smithi142 conmon[20125]: T22:04:59.473696+0000 osd.2 (osd.2) 2021-03-24T22:05:00.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:00 smithi142 conmon[20125]: 115 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:05:00.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:00 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:05:00.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:00 smithi142 conmon[20125]: 03-24T22:04 2021-03-24T22:05:00.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:00 smithi142 conmon[20125]: :59.585767+0000 mon.a (mon.0 2021-03-24T22:05:00.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:00 smithi142 conmon[20125]: ) 1891 : cluster [DBG] osdmap e729: 8 total, 7 up, 8 in 2021-03-24T22:05:00.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:00 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:05:00.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:00 smithi142 conmon[20125]: 03-24T22:05:00.252313+0000 osd.7 (osd 2021-03-24T22:05:00.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:00 smithi142 conmon[20125]: .7) 122 : cluster [DBG] 7.22 scrub ok 2021-03-24T22:05:00.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:00 smithi142 conmon[20125]: cluster 2021-03-24T22 2021-03-24T22:05:00.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:00 smithi142 conmon[20125]: :05:00.371583+0000 mon.a (mon.0) 1892 : cluster 2021-03-24T22:05:00.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:00 smithi142 conmon[20125]: [WRN] Health check update: Degraded data redundancy: 165/1941 objects degraded (8.501%), 19 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2021-03-24T22:05:00.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:00 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:05:00.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:00 smithi142 conmon[20125]: -03-24T22:05:00.380442+0000 mon.a (mon.0) 2021-03-24T22:05:00.872 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:00 smithi142 conmon[20125]: 1893 : cluster [DBG] osdmap e730: 8 total, 7 up, 8 in 2021-03-24T22:05:00.873 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:00.912 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:00 smithi125 conmon[24927]: cluster 2021-03-24T22:04:58.969185 2021-03-24T22:05:00.912 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:00 smithi125 conmon[24927]: +0000 mgr.y (mgr.14144) 639 : cluster [DBG] pgmap v1255: 168 pgs: 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 108 active+clean; 690 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 4.7 MiB/s rd, 1.6 MiB/s wr, 12 op/s; 165/1941 objects degraded (8.501%); 20/1941 objects misplaced (1.030%) 2021-03-24T22:05:00.912 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:00 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:05:00.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:00 smithi125 conmon[24927]: 03-24T22:04:59.471384+0000 osd.2 (osd.2 2021-03-24T22:05:00.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:00 smithi125 conmon[24927]: ) 114 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:05:00.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:00 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:05:00.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:00 smithi125 conmon[24927]: 03-24T22:04:59.473696+0000 osd.2 (osd. 2021-03-24T22:05:00.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:00 smithi125 conmon[24927]: 2) 115 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:05:00.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:00 smithi125 conmon[24927]: cluster 2021-03-24T22:04:59 2021-03-24T22:05:00.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:00 smithi125 conmon[24927]: .585767+0000 mon.a (mon.0) 1891 : cluster [DBG] osdmap e729: 8 total, 7 up, 8 in 2021-03-24T22:05:00.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:00 smithi125 conmon[24927]: cluster 2021-03-24T22:05:00. 2021-03-24T22:05:00.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:00 smithi125 conmon[24927]: 252313+0000 osd.7 (osd.7) 122 : cluster [DBG] 7.22 scrub ok 2021-03-24T22:05:00.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:00 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T22:05:00.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:00 smithi125 conmon[24927]: 22:05:00.371583+0000 mon.a (mon 2021-03-24T22:05:00.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:00 smithi125 conmon[24927]: .0) 1892 : cluster [WRN] Health check update: Degraded data redundancy: 165/1941 objects degraded (8.501%), 19 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2021-03-24T22:05:00.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:00 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:05:00.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:00 smithi125 conmon[24927]: -03-24T22:05:00. 2021-03-24T22:05:00.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:00 smithi125 conmon[24927]: 380442+0000 mon.a (mon.0) 1893 : 2021-03-24T22:05:00.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:00 smithi125 conmon[24927]: cluster [DBG] osdmap e730: 8 total, 7 up, 8 in 2021-03-24T22:05:00.916 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:00 smithi125 conmon[19936]: cluster 2021-03-24T22:04: 2021-03-24T22:05:00.917 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:00 smithi125 conmon[19936]: 58.969185+0000 mgr.y (mgr.14144) 639 : cluster [DBG] pgmap v1255: 168 pgs: 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 108 active+clean; 690 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 4.7 MiB/s rd, 1.6 MiB/s wr, 12 op/s; 165/1941 objects degraded (8.501%); 20/1941 objects misplaced (1.030%) 2021-03-24T22:05:00.917 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:00 smithi125 conmon[19936]: cluster 2021-03-24T22:04:59.471384+0000 osd.2 (osd 2021-03-24T22:05:00.917 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:00 smithi125 conmon[19936]: .2) 114 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:05:00.917 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:00 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:05:00.918 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:00 smithi125 conmon[19936]: -03-24T22:04:59.473696+0000 osd.2 (osd.2) 115 : cluster [DBG] 2021-03-24T22:05:00.918 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:00 smithi125 conmon[19936]: purged_snaps scrub ok 2021-03-24T22:05:00.918 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:00 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:05:00.918 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:00 smithi125 conmon[19936]: 03-24T22:04:59.585767+0000 mon.a (mon.0) 1891 2021-03-24T22:05:00.918 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:00 smithi125 conmon[19936]: : cluster [DBG] osdmap e729: 8 total, 7 up, 8 in 2021-03-24T22:05:00.919 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:00 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T22:05:00.919 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:00 smithi125 conmon[19936]: -24T22:05:00.252313+0000 osd.7 ( 2021-03-24T22:05:00.919 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:00 smithi125 conmon[19936]: osd.7) 122 : cluster [DBG] 2021-03-24T22:05:00.919 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:00 smithi125 conmon[19936]: 7.22 scrub ok 2021-03-24T22:05:00.919 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:00 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:05:00.920 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:00 smithi125 conmon[19936]: 24T22:05:00.371583+0000 2021-03-24T22:05:00.920 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:00 smithi125 conmon[19936]: mon.a (mon.0) 1892 2021-03-24T22:05:00.920 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:00 smithi125 conmon[19936]: : cluster [WRN] Health check update: Degraded data redundancy: 165/1941 objects degraded (8.501%), 19 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2021-03-24T22:05:00.920 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:00 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:05:00.921 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:00 smithi125 conmon[19936]: -03-24T22:05:00 2021-03-24T22:05:00.921 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:00 smithi125 conmon[19936]: .380442+0000 mon.a (mon.0) 2021-03-24T22:05:00.921 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:00 smithi125 conmon[19936]: 1893 : cluster [DBG] osdmap e730: 8 total, 7 up, 8 in 2021-03-24T22:05:00.921 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-24T22:05:00.928 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:00.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:00.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:05:01.035 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:01.090 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:01.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:01.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:05:01.197 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:01.220 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:05:01.256 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:01.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:01.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:05:01.360 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:01.409 INFO:tasks.rados.rados.0.smithi125.stdout:2813: done (1 left) 2021-03-24T22:05:01.409 INFO:tasks.rados.rados.0.smithi125.stdout:2814: snap_create 2021-03-24T22:05:01.409 INFO:tasks.rados.rados.0.smithi125.stdout:2812: finishing write tid 1 to smithi12556470-21 2021-03-24T22:05:01.409 INFO:tasks.rados.rados.0.smithi125.stdout:2812: finishing write tid 2 to smithi12556470-21 2021-03-24T22:05:01.410 INFO:tasks.rados.rados.0.smithi125.stdout:2812: finishing write tid 3 to smithi12556470-21 2021-03-24T22:05:01.410 INFO:tasks.rados.rados.0.smithi125.stdout:2812: finishing write tid 4 to smithi12556470-21 2021-03-24T22:05:01.410 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 21 v 1422 (ObjNum 983 snap 322 seq_num 983) dirty exists 2021-03-24T22:05:01.410 INFO:tasks.rados.rados.0.smithi125.stdout:2812: left oid 21 (ObjNum 983 snap 322 seq_num 983) 2021-03-24T22:05:01.481 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:01.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:01.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:05:01.589 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:01.629 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:01 smithi125 conmon[24927]: cluster 2021-03-24T22:05:00 2021-03-24T22:05:01.630 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:01 smithi125 conmon[24927]: .634530+0000 osd.1 (osd.1) 77 : cluster [DBG] 4.9 scrub ok 2021-03-24T22:05:01.630 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:01 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:05:01.630 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:01 smithi125 conmon[24927]: 24T22:05:01.161121+0000 osd.3 (osd.3) 145 : cluster [DBG] 3.18 scrub ok 2021-03-24T22:05:01.630 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:01 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:05:01.631 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:01 smithi125 conmon[24927]: -03-24T22:05:01.394134+0000 mon.a 2021-03-24T22:05:01.631 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:01 smithi125 conmon[24927]: (mon.0) 1894 : cluster [DBG] osdmap e731: 8 total, 7 up, 8 in 2021-03-24T22:05:01.631 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:01 smithi125 conmon[19936]: cluster 2021-03-24T22:05: 2021-03-24T22:05:01.632 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:01 smithi125 conmon[19936]: 00.634530+0000 osd.1 (osd.1) 77 : cluster [DBG] 4.9 scrub ok 2021-03-24T22:05:01.632 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:01 smithi125 conmon[19936]: cluster 2021-03-24T22:05 2021-03-24T22:05:01.632 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:01 smithi125 conmon[19936]: :01.161121+0000 osd.3 (osd.3) 145 : cluster 2021-03-24T22:05:01.632 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:01 smithi125 conmon[19936]: [DBG] 3.18 scrub ok 2021-03-24T22:05:01.633 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:01 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T22:05:01.633 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:01 smithi125 conmon[19936]: 22:05:01.394134+0000 mon.a 2021-03-24T22:05:01.633 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:01 smithi125 conmon[19936]: (mon.0) 1894 : cluster [DBG] 2021-03-24T22:05:01.633 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:01 smithi125 conmon[19936]: osdmap e731: 8 total, 7 up, 8 in 2021-03-24T22:05:01.648 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:01.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:01.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:05:01.752 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:01.808 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:01.811 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:01 smithi142 conmon[20125]: cluster 2021-03-24T22:05:00.634530+0000 osd.1 (osd.1) 2021-03-24T22:05:01.812 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:01 smithi142 conmon[20125]: 77 : cluster [DBG] 4.9 scrub ok 2021-03-24T22:05:01.812 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:01 smithi142 conmon[20125]: cluster 2021-03-24T22:05:01.161121+0000 osd.3 (osd.3) 145 : cluster [DBG] 3.18 scrub ok 2021-03-24T22:05:01.812 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:01 smithi142 conmon[20125]: cluster 2021-03-24T22:05:01.394134+0000 mon.a (mon.0) 1894 : 2021-03-24T22:05:01.812 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:01 smithi142 conmon[20125]: cluster [DBG] osdmap e731: 8 total, 7 up, 8 in 2021-03-24T22:05:01.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:01.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:05:01.916 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:01.965 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:01.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:01.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:05:02.070 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:02.120 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:02.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:02.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:05:02.196 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2021-03-24T22:05:02.197 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2021-03-24T22:05:02.197 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2021-03-24T22:05:02.198 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2021-03-24T22:05:02.198 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.198 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:2099 7.17 7:ec6a359b:::smithi12556470-33:head [copy-from ver 0] snapc 95=[95,93,92,8d,8b,82] ondisk+write+known_if_redirected e255)", 2021-03-24T22:05:02.198 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:57:04.197620+0000", 2021-03-24T22:05:02.199 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 477.99632677, 2021-03-24T22:05:02.199 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.95919054100000001, 2021-03-24T22:05:02.199 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:05:02.199 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:05:02.200 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:05:02.200 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:05:02.200 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:05:02.200 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 2099 2021-03-24T22:05:02.201 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.201 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:05:02.201 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.201 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:05:02.202 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.197620+0000", 2021-03-24T22:05:02.202 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:05:02.202 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.203 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.203 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:05:02.203 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.197620+0000", 2021-03-24T22:05:02.203 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999962 2021-03-24T22:05:02.204 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.204 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.204 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:05:02.204 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.197616+0000", 2021-03-24T22:05:02.205 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7652999999999999e-05 2021-03-24T22:05:02.205 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.205 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.206 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:05:02.206 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.197634+0000", 2021-03-24T22:05:02.206 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6417000000000001e-05 2021-03-24T22:05:02.206 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.207 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.207 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:05:02.207 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.197660+0000", 2021-03-24T22:05:02.207 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9066e-05 2021-03-24T22:05:02.208 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.208 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.208 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:05:02.209 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.197680+0000", 2021-03-24T22:05:02.209 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00017438800000000001 2021-03-24T22:05:02.209 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.209 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.210 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:05:02.210 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.197854+0000", 2021-03-24T22:05:02.210 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3726e-05 2021-03-24T22:05:02.210 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.211 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.211 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op must wait for map", 2021-03-24T22:05:02.211 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.197878+0000", 2021-03-24T22:05:02.212 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0099756870000000004 2021-03-24T22:05:02.212 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.212 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.212 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:05:02.213 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.207853+0000", 2021-03-24T22:05:02.213 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3523000000000007e-05 2021-03-24T22:05:02.213 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.213 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.214 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:05:02.214 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.207937+0000", 2021-03-24T22:05:02.214 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.948873784 2021-03-24T22:05:02.214 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.215 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.215 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:05:02.215 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:05.156811+0000", 2021-03-24T22:05:02.216 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.95919054100000001 2021-03-24T22:05:02.216 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:05:02.218 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:05:02.218 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:05:02.218 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.219 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.219 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:2100 7.17 7:ec6a359b:::smithi12556470-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e255)", 2021-03-24T22:05:02.219 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:57:04.198076+0000", 2021-03-24T22:05:02.220 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 477.99587039599999, 2021-03-24T22:05:02.220 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.95908168400000005, 2021-03-24T22:05:02.220 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:05:02.220 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:05:02.221 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:05:02.221 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:05:02.221 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:05:02.221 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 2100 2021-03-24T22:05:02.222 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.222 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:05:02.222 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.222 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:05:02.223 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.198076+0000", 2021-03-24T22:05:02.223 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:05:02.223 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.223 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.224 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:05:02.224 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.198076+0000", 2021-03-24T22:05:02.224 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999971 2021-03-24T22:05:02.224 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.225 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.225 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:05:02.225 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.198074+0000", 2021-03-24T22:05:02.226 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3433e-05 2021-03-24T22:05:02.226 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.226 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.226 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:05:02.227 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.198087+0000", 2021-03-24T22:05:02.227 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.262e-05 2021-03-24T22:05:02.227 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.228 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.228 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:05:02.228 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.198110+0000", 2021-03-24T22:05:02.228 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6405999999999998e-05 2021-03-24T22:05:02.229 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.229 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.229 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:05:02.230 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.198126+0000", 2021-03-24T22:05:02.230 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000109786 2021-03-24T22:05:02.230 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.231 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.231 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:05:02.231 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.198236+0000", 2021-03-24T22:05:02.231 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0522e-05 2021-03-24T22:05:02.232 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.232 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.232 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting_for_map not empty", 2021-03-24T22:05:02.233 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.198246+0000", 2021-03-24T22:05:02.233 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0098265899999999996 2021-03-24T22:05:02.233 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.233 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.234 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:05:02.234 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.208073+0000", 2021-03-24T22:05:02.234 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7707999999999999e-05 2021-03-24T22:05:02.234 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.235 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.235 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for blocked object", 2021-03-24T22:05:02.235 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:04.208111+0000", 2021-03-24T22:05:02.236 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.94884332199999999 2021-03-24T22:05:02.236 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.236 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.237 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:05:02.237 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:05.156954+0000", 2021-03-24T22:05:02.237 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.8762000000000001e-05 2021-03-24T22:05:02.237 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.238 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.238 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:05:02.238 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:05.157053+0000", 2021-03-24T22:05:02.239 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000105321 2021-03-24T22:05:02.239 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.239 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.239 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:05:02.240 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:57:05.157158+0000", 2021-03-24T22:05:02.240 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.95908168400000005 2021-03-24T22:05:02.240 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:05:02.240 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:05:02.241 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:05:02.241 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.241 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.241 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:3112 7.17 7:ec6a359b:::smithi12556470-33:df [sparse-read 0~3008486,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e340)", 2021-03-24T22:05:02.242 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:58:30.211016+0000", 2021-03-24T22:05:02.242 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 391.98293096899999, 2021-03-24T22:05:02.242 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.99935758200000002, 2021-03-24T22:05:02.243 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:05:02.243 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:05:02.243 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:05:02.243 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:05:02.244 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:05:02.244 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 3112 2021-03-24T22:05:02.244 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.244 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:05:02.245 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.245 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:05:02.245 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211016+0000", 2021-03-24T22:05:02.246 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:05:02.246 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.246 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.246 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:05:02.247 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211016+0000", 2021-03-24T22:05:02.247 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2021-03-24T22:05:02.247 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.247 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.248 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:05:02.248 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211013+0000", 2021-03-24T22:05:02.248 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0642000000000001e-05 2021-03-24T22:05:02.248 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.249 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.249 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:05:02.249 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211024+0000", 2021-03-24T22:05:02.250 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2675e-05 2021-03-24T22:05:02.250 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.250 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.251 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:05:02.251 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211037+0000", 2021-03-24T22:05:02.251 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6210000000000004e-06 2021-03-24T22:05:02.251 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.252 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.252 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:05:02.252 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211044+0000", 2021-03-24T22:05:02.252 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.4183999999999994e-05 2021-03-24T22:05:02.253 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.254 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.254 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:05:02.254 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211109+0000", 2021-03-24T22:05:02.254 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1910000000000006e-06 2021-03-24T22:05:02.255 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.255 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.255 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for peered", 2021-03-24T22:05:02.256 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211118+0000", 2021-03-24T22:05:02.256 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.99877172599999997 2021-03-24T22:05:02.256 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.256 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.257 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:05:02.257 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:31.209889+0000", 2021-03-24T22:05:02.257 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000169588 2021-03-24T22:05:02.257 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.258 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.258 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:05:02.258 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:31.210059+0000", 2021-03-24T22:05:02.258 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00031441100000000001 2021-03-24T22:05:02.259 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.259 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.259 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:05:02.259 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:31.210373+0000", 2021-03-24T22:05:02.260 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.99935758200000002 2021-03-24T22:05:02.260 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:05:02.260 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:05:02.260 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:05:02.261 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.261 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.261 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:3113 7.17 7:ec6a359b:::smithi12556470-33:df [sparse-read 0~3008486] snapc 0=[] ondisk+read+known_if_redirected e340)", 2021-03-24T22:05:02.261 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:58:30.211055+0000", 2021-03-24T22:05:02.262 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 391.98289198600003, 2021-03-24T22:05:02.270 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.99976562300000005, 2021-03-24T22:05:02.270 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:05:02.271 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:05:02.271 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:05:02.271 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:05:02.271 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:05:02.271 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 3113 2021-03-24T22:05:02.272 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.272 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:05:02.272 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.272 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:05:02.273 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211055+0000", 2021-03-24T22:05:02.273 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:05:02.273 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.274 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.274 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:05:02.274 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211055+0000", 2021-03-24T22:05:02.275 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2021-03-24T22:05:02.275 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.275 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.276 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:05:02.276 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211054+0000", 2021-03-24T22:05:02.276 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8150000000000002e-06 2021-03-24T22:05:02.277 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.277 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.278 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:05:02.278 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211057+0000", 2021-03-24T22:05:02.278 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.7319999999999993e-06 2021-03-24T22:05:02.279 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.279 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.279 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:05:02.280 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211067+0000", 2021-03-24T22:05:02.280 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1489999999999998e-06 2021-03-24T22:05:02.281 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.281 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.282 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:05:02.282 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211070+0000", 2021-03-24T22:05:02.283 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.1780000000000006e-05 2021-03-24T22:05:02.283 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.283 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.283 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:05:02.284 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211152+0000", 2021-03-24T22:05:02.284 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.8709999999999999e-06 2021-03-24T22:05:02.285 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.285 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.285 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for peered", 2021-03-24T22:05:02.285 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211159+0000", 2021-03-24T22:05:02.285 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.999350081 2021-03-24T22:05:02.286 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.286 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.286 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:05:02.286 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:31.210509+0000", 2021-03-24T22:05:02.287 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.0376000000000003e-05 2021-03-24T22:05:02.287 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.287 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.287 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:05:02.288 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:31.210599+0000", 2021-03-24T22:05:02.288 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00022137900000000001 2021-03-24T22:05:02.288 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.289 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.289 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:05:02.289 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:31.210820+0000", 2021-03-24T22:05:02.290 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.99976562300000005 2021-03-24T22:05:02.290 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:05:02.290 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:05:02.291 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:05:02.291 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.291 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.291 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:3114 7.17 7:ec6a359b:::smithi12556470-33:df [sparse-read 0~3008486] snapc 0=[] ondisk+read+known_if_redirected e340)", 2021-03-24T22:05:02.292 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T21:58:30.211092+0000", 2021-03-24T22:05:02.292 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 391.98285507899999, 2021-03-24T22:05:02.292 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.000019572, 2021-03-24T22:05:02.292 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:05:02.293 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:05:02.293 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:05:02.293 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:05:02.293 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:05:02.294 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 3114 2021-03-24T22:05:02.294 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.294 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:05:02.294 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.295 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:05:02.295 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211092+0000", 2021-03-24T22:05:02.296 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:05:02.296 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.296 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.297 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:05:02.297 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211092+0000", 2021-03-24T22:05:02.298 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2021-03-24T22:05:02.298 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.299 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.299 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:05:02.300 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211091+0000", 2021-03-24T22:05:02.300 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2789999999999999e-06 2021-03-24T22:05:02.300 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.301 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.301 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:05:02.302 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211095+0000", 2021-03-24T22:05:02.302 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3040000000000001e-06 2021-03-24T22:05:02.302 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.303 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.303 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:05:02.303 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211101+0000", 2021-03-24T22:05:02.303 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4950000000000002e-06 2021-03-24T22:05:02.304 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.304 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.304 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:05:02.305 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211105+0000", 2021-03-24T22:05:02.305 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.8471000000000002e-05 2021-03-24T22:05:02.306 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.306 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.306 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:05:02.307 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211174+0000", 2021-03-24T22:05:02.308 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.9419999999999997e-06 2021-03-24T22:05:02.308 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.308 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.309 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for peered", 2021-03-24T22:05:02.309 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:30.211178+0000", 2021-03-24T22:05:02.309 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.99967128199999999 2021-03-24T22:05:02.310 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.310 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.310 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:05:02.311 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:31.210849+0000", 2021-03-24T22:05:02.311 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011547 2021-03-24T22:05:02.311 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.311 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.312 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:05:02.312 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:31.210965+0000", 2021-03-24T22:05:02.313 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014683099999999999 2021-03-24T22:05:02.313 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.314 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.314 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:05:02.314 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T21:58:31.211111+0000", 2021-03-24T22:05:02.314 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.000019572 2021-03-24T22:05:02.315 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:05:02.315 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:05:02.315 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:05:02.315 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.316 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.316 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.24388.0:544 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e490)", 2021-03-24T22:05:02.316 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:01:00.454826+0000", 2021-03-24T22:05:02.316 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 241.73912058900001, 2021-03-24T22:05:02.316 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.86119727800000001, 2021-03-24T22:05:02.317 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:05:02.317 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:05:02.317 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:05:02.318 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.24388", 2021-03-24T22:05:02.318 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.142:0/626803041", 2021-03-24T22:05:02.319 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 544 2021-03-24T22:05:02.319 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.319 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:05:02.319 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.320 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:05:02.320 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:00.454826+0000", 2021-03-24T22:05:02.320 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:05:02.320 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.321 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.321 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:05:02.321 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:00.454826+0000", 2021-03-24T22:05:02.322 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999957 2021-03-24T22:05:02.322 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.323 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.323 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:05:02.324 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:00.454822+0000", 2021-03-24T22:05:02.324 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8848999999999999e-05 2021-03-24T22:05:02.324 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.325 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.325 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:05:02.326 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:00.454841+0000", 2021-03-24T22:05:02.326 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8226000000000003e-05 2021-03-24T22:05:02.327 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.327 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.327 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:05:02.328 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:00.454879+0000", 2021-03-24T22:05:02.328 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5817000000000001e-05 2021-03-24T22:05:02.329 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.329 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.330 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:05:02.330 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:00.454895+0000", 2021-03-24T22:05:02.330 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015252899999999999 2021-03-24T22:05:02.331 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.331 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.331 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:05:02.332 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:00.455047+0000", 2021-03-24T22:05:02.332 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1634000000000002e-05 2021-03-24T22:05:02.332 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.333 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.333 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for peered", 2021-03-24T22:05:02.334 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:00.455079+0000", 2021-03-24T22:05:02.334 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.85495980199999999 2021-03-24T22:05:02.334 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.334 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.335 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:05:02.335 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:01.310039+0000", 2021-03-24T22:05:02.335 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013964399999999999 2021-03-24T22:05:02.335 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.335 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.336 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for missing object", 2021-03-24T22:05:02.336 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:01.310178+0000", 2021-03-24T22:05:02.336 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.005680488 2021-03-24T22:05:02.336 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.337 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.337 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:05:02.337 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:01.315859+0000", 2021-03-24T22:05:02.337 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3120000000000006e-05 2021-03-24T22:05:02.337 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.338 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.338 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:05:02.338 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:01.315922+0000", 2021-03-24T22:05:02.338 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000101475 2021-03-24T22:05:02.339 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.339 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.339 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:05:02.339 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:01.316024+0000", 2021-03-24T22:05:02.340 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.86119727800000001 2021-03-24T22:05:02.340 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:05:02.340 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:05:02.340 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:05:02.341 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.341 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.341 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(7.19 508/491 [PushOp(7:9b48f357:::smithi12556470-17:114, version: 455'474, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:9b48f357:::smithi12556470-17:114@455'474, size: 3142975, copy_subset: [0~3142975], clone_subset: {}, snapset: 166=[]:{114=[104,fa],11b=[11b],127=[127],165=[165,160,15e,159,157,153],166=[166]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-03-24T22:05:02.341 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:01:19.123177+0000", 2021-03-24T22:05:02.342 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 223.070769407, 2021-03-24T22:05:02.342 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.088544329000000005, 2021-03-24T22:05:02.342 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:05:02.343 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:05:02.343 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:05:02.344 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.344 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:05:02.345 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.123177+0000", 2021-03-24T22:05:02.345 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:05:02.345 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.346 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.346 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:05:02.347 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.123177+0000", 2021-03-24T22:05:02.347 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999733 2021-03-24T22:05:02.348 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.348 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.348 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:05:02.349 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.123150+0000", 2021-03-24T22:05:02.349 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00178196 2021-03-24T22:05:02.350 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.350 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.351 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:05:02.351 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.124932+0000", 2021-03-24T22:05:02.351 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4739e-05 2021-03-24T22:05:02.351 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.351 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.352 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:05:02.352 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.124977+0000", 2021-03-24T22:05:02.352 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3717e-05 2021-03-24T22:05:02.352 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.353 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.353 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:05:02.353 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.125001+0000", 2021-03-24T22:05:02.353 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.086317912999999996 2021-03-24T22:05:02.354 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.354 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.354 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:05:02.354 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.211319+0000", 2021-03-24T22:05:02.355 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8689999999999993e-06 2021-03-24T22:05:02.355 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.355 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.355 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:05:02.355 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.211328+0000", 2021-03-24T22:05:02.356 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00039407100000000002 2021-03-24T22:05:02.356 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.356 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.356 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:05:02.356 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.211722+0000", 2021-03-24T22:05:02.357 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.088544329000000005 2021-03-24T22:05:02.357 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:05:02.357 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:05:02.357 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:05:02.358 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.358 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.358 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(7.19 509/491 [PushOp(7:9b48f357:::smithi12556470-17:11b, version: 455'475, data_included: [521149~630541,1701900~418035], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:9b48f357:::smithi12556470-17:11b@455'475, size: 2521149, copy_subset: [521149~630541,1701900~628502], clone_subset: {7:9b48f357:::smithi12556470-17:114=[0~521149,1151690~550210,2330402~190747]}, snapset: 166=[]:{114=[104,fa],11b=[11b],127=[127],165=[165,160,15e,159,157,153],166=[166]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2119935, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2021-03-24T22:05:02.358 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:01:19.472229+0000", 2021-03-24T22:05:02.358 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 222.721717641, 2021-03-24T22:05:02.359 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.089665769000000006, 2021-03-24T22:05:02.359 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:05:02.359 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:05:02.360 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:05:02.360 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.360 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:05:02.360 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.472229+0000", 2021-03-24T22:05:02.361 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:05:02.361 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.361 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.361 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:05:02.361 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.472229+0000", 2021-03-24T22:05:02.362 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2021-03-24T22:05:02.362 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.362 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.362 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:05:02.363 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.472228+0000", 2021-03-24T22:05:02.363 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00095283100000000001 2021-03-24T22:05:02.363 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.363 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.363 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:05:02.364 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.473181+0000", 2021-03-24T22:05:02.364 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4737999999999999e-05 2021-03-24T22:05:02.364 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.364 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.365 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:05:02.365 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.473205+0000", 2021-03-24T22:05:02.365 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3817e-05 2021-03-24T22:05:02.365 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.365 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.366 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:05:02.366 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.473219+0000", 2021-03-24T22:05:02.367 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.088154109999999994 2021-03-24T22:05:02.367 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.367 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.368 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:05:02.368 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.561373+0000", 2021-03-24T22:05:02.369 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.578e-06 2021-03-24T22:05:02.369 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.369 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.370 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:05:02.370 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.561378+0000", 2021-03-24T22:05:02.371 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00051709599999999998 2021-03-24T22:05:02.371 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.371 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.371 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:05:02.372 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.561895+0000", 2021-03-24T22:05:02.372 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.089665769000000006 2021-03-24T22:05:02.372 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:05:02.373 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:05:02.373 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:05:02.373 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.373 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.374 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(7.19 509/491 [PushOp(7:9b48f357:::smithi12556470-17:11b, version: 455'475, data_included: [2119935~210467], data_size: 210467, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:9b48f357:::smithi12556470-17:11b@455'475, size: 2521149, copy_subset: [521149~630541,1701900~628502], clone_subset: {7:9b48f357:::smithi12556470-17:114=[0~521149,1151690~550210,2330402~190747]}, snapset: 166=[]:{114=[104,fa],11b=[11b],127=[127],165=[165,160,15e,159,157,153],166=[166]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2330402, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2119935, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-03-24T22:05:02.374 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:01:19.563380+0000", 2021-03-24T22:05:02.374 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 222.63056711999999, 2021-03-24T22:05:02.374 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.088066119999999998, 2021-03-24T22:05:02.375 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:05:02.375 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:05:02.375 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:05:02.375 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.376 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:05:02.376 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.563380+0000", 2021-03-24T22:05:02.376 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:05:02.376 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.377 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.377 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:05:02.377 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.563380+0000", 2021-03-24T22:05:02.377 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2021-03-24T22:05:02.378 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.378 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.378 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:05:02.379 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.563379+0000", 2021-03-24T22:05:02.379 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00020590499999999999 2021-03-24T22:05:02.379 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.380 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.380 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:05:02.380 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.563585+0000", 2021-03-24T22:05:02.381 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1863999999999999e-05 2021-03-24T22:05:02.381 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.382 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.382 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:05:02.382 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.563607+0000", 2021-03-24T22:05:02.382 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8245000000000001e-05 2021-03-24T22:05:02.383 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.383 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.384 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:05:02.385 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.563635+0000", 2021-03-24T22:05:02.385 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.084418923000000007 2021-03-24T22:05:02.385 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.385 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.386 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:05:02.386 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.648054+0000", 2021-03-24T22:05:02.387 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1840000000000001e-06 2021-03-24T22:05:02.387 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.388 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.388 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:05:02.389 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.648058+0000", 2021-03-24T22:05:02.389 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0033879330000000001 2021-03-24T22:05:02.390 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.390 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.390 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:05:02.390 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.651446+0000", 2021-03-24T22:05:02.391 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.088066119999999998 2021-03-24T22:05:02.391 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:05:02.392 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:05:02.392 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:05:02.392 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.392 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.393 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(7.19 509/491 [PushOp(7:9b48f357:::smithi12556470-17:127, version: 455'477, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:9b48f357:::smithi12556470-17:127@455'477, size: 2874096, copy_subset: [0~2874096], clone_subset: {7:9b48f357:::smithi12556470-17:11b=[]}, snapset: 166=[]:{114=[104,fa],11b=[11b],127=[127],165=[165,160,15e,159,157,153],166=[166]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-03-24T22:05:02.393 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:01:19.826600+0000", 2021-03-24T22:05:02.393 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 222.36734709500001, 2021-03-24T22:05:02.393 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.088799362000000007, 2021-03-24T22:05:02.394 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:05:02.394 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:05:02.395 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:05:02.395 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.395 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:05:02.396 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.826600+0000", 2021-03-24T22:05:02.396 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:05:02.397 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.397 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.398 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:05:02.398 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.826600+0000", 2021-03-24T22:05:02.399 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2021-03-24T22:05:02.399 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.399 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.400 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:05:02.400 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.826599+0000", 2021-03-24T22:05:02.400 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00097982800000000008 2021-03-24T22:05:02.400 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.401 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.401 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:05:02.401 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.827578+0000", 2021-03-24T22:05:02.401 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.084e-05 2021-03-24T22:05:02.402 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.402 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.402 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:05:02.402 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.827599+0000", 2021-03-24T22:05:02.403 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2388000000000001e-05 2021-03-24T22:05:02.403 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.403 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.403 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:05:02.403 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.827612+0000", 2021-03-24T22:05:02.404 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.087205894000000006 2021-03-24T22:05:02.404 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.404 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.404 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:05:02.405 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.914818+0000", 2021-03-24T22:05:02.405 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0579999999999997e-06 2021-03-24T22:05:02.405 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.405 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.405 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:05:02.409 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.914823+0000", 2021-03-24T22:05:02.410 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00057645600000000004 2021-03-24T22:05:02.410 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.411 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.411 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:05:02.411 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:19.915399+0000", 2021-03-24T22:05:02.412 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.088799362000000007 2021-03-24T22:05:02.412 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:05:02.413 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:05:02.413 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:05:02.414 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.414 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.415 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(7.19 510/491 [PushOp(7:9c4f4ea5:::smithi12556470-41:119, version: 455'479, data_included: [2097152~722659,3066106~325917], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:9c4f4ea5:::smithi12556470-41:119@455'479, size: 3731220, copy_subset: [0~3731220], clone_subset: {}, snapset: 149=[]:{119=[104,fa],149=[139,136,129,127,11b]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3392023, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-03-24T22:05:02.415 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:01:20.359492+0000", 2021-03-24T22:05:02.415 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 221.834454507, 2021-03-24T22:05:02.416 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.087993191999999998, 2021-03-24T22:05:02.416 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:05:02.417 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:05:02.417 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:05:02.418 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.418 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:05:02.419 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:20.359492+0000", 2021-03-24T22:05:02.419 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:05:02.420 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.420 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.420 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:05:02.421 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:20.359492+0000", 2021-03-24T22:05:02.421 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999971 2021-03-24T22:05:02.422 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.422 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.423 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:05:02.423 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:20.359489+0000", 2021-03-24T22:05:02.424 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001299632 2021-03-24T22:05:02.424 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.424 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.425 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:05:02.425 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:20.360789+0000", 2021-03-24T22:05:02.426 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.3445000000000002e-05 2021-03-24T22:05:02.426 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.427 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.427 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:05:02.427 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:20.360832+0000", 2021-03-24T22:05:02.428 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2379999999999999e-05 2021-03-24T22:05:02.428 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.428 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.429 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:05:02.429 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:20.360855+0000", 2021-03-24T22:05:02.429 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.086302106000000003 2021-03-24T22:05:02.429 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.430 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.430 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:05:02.430 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:20.447157+0000", 2021-03-24T22:05:02.430 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1219999999999997e-06 2021-03-24T22:05:02.431 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.431 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.431 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:05:02.432 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:20.447161+0000", 2021-03-24T22:05:02.433 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00032459299999999997 2021-03-24T22:05:02.433 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.433 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.433 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:05:02.434 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:20.447486+0000", 2021-03-24T22:05:02.434 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.087993191999999998 2021-03-24T22:05:02.434 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:05:02.435 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:05:02.435 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:05:02.435 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.436 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.436 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(7.19 510/491 [PushOp(7:9b48f357:::smithi12556470-17:166, version: 469'504, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:9b48f357:::smithi12556470-17:166@469'504, size: 2353079, copy_subset: [0~2353079], clone_subset: {7:9b48f357:::smithi12556470-17:127=[]}, snapset: 166=[]:{114=[104,fa],11b=[11b],127=[127],165=[165,160,15e,159,157,153],166=[166]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-03-24T22:05:02.436 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:01:20.884182+0000", 2021-03-24T22:05:02.437 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 221.30976468599999, 2021-03-24T22:05:02.437 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.088397383999999996, 2021-03-24T22:05:02.437 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:05:02.437 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:05:02.438 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:05:02.438 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.438 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:05:02.438 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:20.884182+0000", 2021-03-24T22:05:02.439 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:05:02.439 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.439 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.440 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:05:02.440 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:20.884182+0000", 2021-03-24T22:05:02.440 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999971 2021-03-24T22:05:02.440 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.440 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.441 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:05:02.441 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:20.884179+0000", 2021-03-24T22:05:02.441 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001025679 2021-03-24T22:05:02.442 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.442 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.443 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:05:02.443 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:20.885205+0000", 2021-03-24T22:05:02.443 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.7153000000000001e-05 2021-03-24T22:05:02.443 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.444 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.444 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:05:02.444 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:20.885252+0000", 2021-03-24T22:05:02.444 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4223000000000002e-05 2021-03-24T22:05:02.445 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.445 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.445 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:05:02.445 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:20.885276+0000", 2021-03-24T22:05:02.445 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.086821926999999993 2021-03-24T22:05:02.446 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.446 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.446 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:05:02.449 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:20.972098+0000", 2021-03-24T22:05:02.449 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0560000000000004e-06 2021-03-24T22:05:02.450 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.450 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.450 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:05:02.450 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:20.972103+0000", 2021-03-24T22:05:02.451 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00047635599999999999 2021-03-24T22:05:02.451 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.452 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.452 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:05:02.453 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:20.972580+0000", 2021-03-24T22:05:02.453 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.088397383999999996 2021-03-24T22:05:02.453 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:05:02.453 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:05:02.454 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:05:02.454 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.454 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.455 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(7.19 511/491 [PushOp(7:9b48f357:::smithi12556470-17:165, version: 481'520, data_included: [2119071~248646,2770646~473204], data_size: 721850, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:9b48f357:::smithi12556470-17:165@481'520, size: 3243850, copy_subset: [0~3243850], clone_subset: {7:9b48f357:::smithi12556470-17:127=[],7:9b48f357:::smithi12556470-17:166=[]}, snapset: 166=[]:{114=[104,fa],11b=[11b],127=[127],165=[165,160,15e,159,157,153],166=[166]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3243850, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2119071, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-03-24T22:05:02.455 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:01:21.495555+0000", 2021-03-24T22:05:02.455 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 220.69839166, 2021-03-24T22:05:02.455 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.089930140000000006, 2021-03-24T22:05:02.456 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:05:02.456 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:05:02.456 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:05:02.457 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.457 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:05:02.457 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:21.495555+0000", 2021-03-24T22:05:02.458 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:05:02.458 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.458 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.459 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:05:02.459 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:21.495555+0000", 2021-03-24T22:05:02.459 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2021-03-24T22:05:02.459 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.460 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.460 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:05:02.460 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:21.495554+0000", 2021-03-24T22:05:02.461 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00060347000000000005 2021-03-24T22:05:02.461 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.461 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.461 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:05:02.462 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:21.496158+0000", 2021-03-24T22:05:02.462 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1359e-05 2021-03-24T22:05:02.462 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.463 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.463 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:05:02.463 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:21.496179+0000", 2021-03-24T22:05:02.463 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9062999999999999e-05 2021-03-24T22:05:02.464 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.464 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.464 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:05:02.465 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:21.496208+0000", 2021-03-24T22:05:02.465 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.085999172999999998 2021-03-24T22:05:02.465 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.466 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.466 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:05:02.466 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:21.582207+0000", 2021-03-24T22:05:02.466 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.7330000000000003e-06 2021-03-24T22:05:02.467 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.467 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.467 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:05:02.468 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:21.582212+0000", 2021-03-24T22:05:02.468 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0032732009999999999 2021-03-24T22:05:02.468 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.468 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.469 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:05:02.469 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:21.585485+0000", 2021-03-24T22:05:02.469 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.089930140000000006 2021-03-24T22:05:02.470 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:05:02.470 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:05:02.470 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:05:02.470 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.471 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.471 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(7.19 512/491 [PushOp(7:992cb487:::smithi12556470-5:head, version: 489'536, data_included: [2896863~61301], data_size: 61301, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:992cb487:::smithi12556470-5:head@489'536, size: 2958164, copy_subset: [558164~552353,1803789~454563,2453954~504210], clone_subset: {7:992cb487:::smithi12556470-5:177=[0~558164,1110517~693272,2258352~195602]}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2958164, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2896863, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-03-24T22:05:02.472 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:01:22.483192+0000", 2021-03-24T22:05:02.472 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 219.710754831, 2021-03-24T22:05:02.472 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.088451157000000002, 2021-03-24T22:05:02.472 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:05:02.473 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:05:02.473 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:05:02.473 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.474 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:05:02.474 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:22.483192+0000", 2021-03-24T22:05:02.474 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:05:02.474 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.475 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.475 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:05:02.475 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:22.483192+0000", 2021-03-24T22:05:02.476 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2021-03-24T22:05:02.476 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.476 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.476 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:05:02.477 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:22.483190+0000", 2021-03-24T22:05:02.477 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9178000000000007e-05 2021-03-24T22:05:02.477 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.478 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.478 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:05:02.478 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:22.483269+0000", 2021-03-24T22:05:02.478 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4663000000000003e-05 2021-03-24T22:05:02.479 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.479 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.479 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:05:02.480 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:22.483314+0000", 2021-03-24T22:05:02.480 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2889000000000001e-05 2021-03-24T22:05:02.480 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.480 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.481 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:05:02.481 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:22.483337+0000", 2021-03-24T22:05:02.481 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.084227195000000005 2021-03-24T22:05:02.481 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.482 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.482 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:05:02.482 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:22.567564+0000", 2021-03-24T22:05:02.483 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.9276999999999997e-05 2021-03-24T22:05:02.483 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.483 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.483 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:05:02.484 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:22.567603+0000", 2021-03-24T22:05:02.484 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0040401880000000001 2021-03-24T22:05:02.484 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.484 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.484 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:05:02.485 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:22.571643+0000", 2021-03-24T22:05:02.485 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.088451157000000002 2021-03-24T22:05:02.485 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:05:02.485 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:05:02.487 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:05:02.487 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.488 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.488 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.24388.0:691 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected e640)", 2021-03-24T22:05:02.488 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:03:31.505734+0000", 2021-03-24T22:05:02.489 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 90.688212753000002, 2021-03-24T22:05:02.489 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.88189851900000005, 2021-03-24T22:05:02.489 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:05:02.489 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:05:02.490 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:05:02.490 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.24388", 2021-03-24T22:05:02.490 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.142:0/626803041", 2021-03-24T22:05:02.491 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 691 2021-03-24T22:05:02.491 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.491 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:05:02.491 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.492 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:05:02.492 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:31.505734+0000", 2021-03-24T22:05:02.492 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:05:02.492 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.493 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.493 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:05:02.493 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:31.505734+0000", 2021-03-24T22:05:02.494 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999952 2021-03-24T22:05:02.494 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.494 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.495 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:05:02.495 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:31.505730+0000", 2021-03-24T22:05:02.495 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4457e-05 2021-03-24T22:05:02.496 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.496 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.496 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:05:02.496 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:31.505754+0000", 2021-03-24T22:05:02.497 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2378999999999993e-05 2021-03-24T22:05:02.497 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.497 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.497 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:05:02.498 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:31.505816+0000", 2021-03-24T22:05:02.498 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3139e-05 2021-03-24T22:05:02.498 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.499 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.499 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:05:02.500 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:31.505840+0000", 2021-03-24T22:05:02.500 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00020648300000000001 2021-03-24T22:05:02.500 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.500 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.501 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:05:02.501 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:31.506046+0000", 2021-03-24T22:05:02.501 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7819999999999999e-05 2021-03-24T22:05:02.502 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.502 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.502 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for peered", 2021-03-24T22:05:02.503 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:31.506064+0000", 2021-03-24T22:05:02.503 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.88126212599999998 2021-03-24T22:05:02.503 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.504 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.504 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:05:02.504 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:32.387326+0000", 2021-03-24T22:05:02.505 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012952100000000001 2021-03-24T22:05:02.505 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.505 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.505 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:05:02.505 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:32.387455+0000", 2021-03-24T22:05:02.506 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00017714099999999999 2021-03-24T22:05:02.506 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.506 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.506 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:05:02.507 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:32.387633+0000", 2021-03-24T22:05:02.507 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.88189851900000005 2021-03-24T22:05:02.507 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:05:02.507 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:05:02.508 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:05:02.508 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.508 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.509 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:6388 7.22 7:454f21f6:::smithi12556470-8:head [write 538055~638274] snapc 219=[219,218,217,214,211,210,20e,20d,209,204,1f8,1f3,1f0,1ef,1e1,1de,1dc,1d8,1d4,1cd,1bf,1bd,199,172,16e] ondisk+write+known_if_redirected e656)", 2021-03-24T22:05:02.509 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:03:46.362651+0000", 2021-03-24T22:05:02.509 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 75.831295404000002, 2021-03-24T22:05:02.509 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.10354008100000001, 2021-03-24T22:05:02.510 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:05:02.510 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:05:02.510 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:05:02.511 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:05:02.511 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:05:02.511 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 6388 2021-03-24T22:05:02.511 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.511 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:05:02.512 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.512 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:05:02.512 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.362651+0000", 2021-03-24T22:05:02.512 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:05:02.513 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.513 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.513 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:05:02.513 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.362651+0000", 2021-03-24T22:05:02.513 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2021-03-24T22:05:02.514 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.514 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.514 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:05:02.514 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.362650+0000", 2021-03-24T22:05:02.515 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001114767 2021-03-24T22:05:02.515 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.515 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.515 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:05:02.515 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.363764+0000", 2021-03-24T22:05:02.516 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6348e-05 2021-03-24T22:05:02.516 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.516 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.516 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:05:02.517 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.363781+0000", 2021-03-24T22:05:02.517 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2790000000000001e-06 2021-03-24T22:05:02.517 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.518 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.518 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:05:02.519 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.363788+0000", 2021-03-24T22:05:02.519 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3058000000000005e-05 2021-03-24T22:05:02.520 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.520 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.520 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:05:02.521 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.363851+0000", 2021-03-24T22:05:02.521 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3801000000000001e-05 2021-03-24T22:05:02.522 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.522 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.523 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for peered", 2021-03-24T22:05:02.523 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.363865+0000", 2021-03-24T22:05:02.523 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.094604193000000003 2021-03-24T22:05:02.524 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.524 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.525 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:05:02.525 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.458469+0000", 2021-03-24T22:05:02.526 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00027249000000000002 2021-03-24T22:05:02.526 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.526 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.526 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:05:02.527 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.458742+0000", 2021-03-24T22:05:02.527 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00032368300000000001 2021-03-24T22:05:02.527 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.528 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.528 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1,4", 2021-03-24T22:05:02.528 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.459065+0000", 2021-03-24T22:05:02.529 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0049690070000000001 2021-03-24T22:05:02.529 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.530 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.530 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T22:05:02.530 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.464034+0000", 2021-03-24T22:05:02.531 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00033590699999999998 2021-03-24T22:05:02.531 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.532 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.532 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:05:02.533 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.464370+0000", 2021-03-24T22:05:02.533 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001266487 2021-03-24T22:05:02.534 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.534 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.534 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:05:02.535 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.465637+0000", 2021-03-24T22:05:02.535 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000119158 2021-03-24T22:05:02.536 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.536 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.537 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:05:02.537 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.465756+0000", 2021-03-24T22:05:02.537 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00043566699999999999 2021-03-24T22:05:02.538 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.538 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.539 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:05:02.539 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.466192+0000", 2021-03-24T22:05:02.540 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.10354008100000001 2021-03-24T22:05:02.540 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:05:02.540 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:05:02.540 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:05:02.541 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.541 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.541 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:6389 7.22 7:454f21f6:::smithi12556470-8:head [write 1867983~670072] snapc 219=[219,218,217,214,211,210,20e,20d,209,204,1f8,1f3,1f0,1ef,1e1,1de,1dc,1d8,1d4,1cd,1bf,1bd,199,172,16e] ondisk+write+known_if_redirected e656)", 2021-03-24T22:05:02.541 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:03:46.370016+0000", 2021-03-24T22:05:02.542 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 75.823931057999999, 2021-03-24T22:05:02.542 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.097041275999999996, 2021-03-24T22:05:02.542 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:05:02.542 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:05:02.542 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:05:02.543 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:05:02.543 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:05:02.543 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 6389 2021-03-24T22:05:02.543 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.544 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:05:02.544 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.544 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:05:02.544 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.370016+0000", 2021-03-24T22:05:02.544 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:05:02.545 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.545 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.545 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:05:02.545 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.370016+0000", 2021-03-24T22:05:02.546 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2021-03-24T22:05:02.546 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.546 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.546 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:05:02.547 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.370014+0000", 2021-03-24T22:05:02.547 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00060345400000000001 2021-03-24T22:05:02.547 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.547 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.547 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:05:02.548 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.370618+0000", 2021-03-24T22:05:02.548 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3202e-05 2021-03-24T22:05:02.548 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.548 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.549 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:05:02.549 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.370631+0000", 2021-03-24T22:05:02.549 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.446e-06 2021-03-24T22:05:02.549 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.549 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.550 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:05:02.550 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.370636+0000", 2021-03-24T22:05:02.550 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7750000000000001e-05 2021-03-24T22:05:02.550 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.551 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.551 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:05:02.551 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.370694+0000", 2021-03-24T22:05:02.551 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2575000000000001e-05 2021-03-24T22:05:02.551 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.552 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.552 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for peered", 2021-03-24T22:05:02.552 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.370707+0000", 2021-03-24T22:05:02.552 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.089851150000000005 2021-03-24T22:05:02.553 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.553 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.553 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:05:02.553 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.460558+0000", 2021-03-24T22:05:02.553 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00017766400000000001 2021-03-24T22:05:02.554 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.554 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.554 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:05:02.554 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.460735+0000", 2021-03-24T22:05:02.555 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00027698799999999998 2021-03-24T22:05:02.555 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.555 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.555 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1,4", 2021-03-24T22:05:02.555 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.461012+0000", 2021-03-24T22:05:02.556 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0030440089999999999 2021-03-24T22:05:02.556 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.556 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.556 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T22:05:02.556 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.464056+0000", 2021-03-24T22:05:02.557 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00040113399999999998 2021-03-24T22:05:02.557 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.557 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.557 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:05:02.558 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.464457+0000", 2021-03-24T22:05:02.558 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001969681 2021-03-24T22:05:02.558 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.558 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.558 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:05:02.559 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.466427+0000", 2021-03-24T22:05:02.559 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00017614200000000001 2021-03-24T22:05:02.559 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.559 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.559 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:05:02.560 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.466603+0000", 2021-03-24T22:05:02.560 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00045378900000000002 2021-03-24T22:05:02.560 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.560 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.561 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:05:02.561 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.467057+0000", 2021-03-24T22:05:02.561 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.097041275999999996 2021-03-24T22:05:02.561 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:05:02.561 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:05:02.562 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:05:02.562 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.562 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.562 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:6390 7.22 7:454f21f6:::smithi12556470-8:head [setxattr _header (57) in=7b,truncate 2538055] snapc 219=[219,218,217,214,211,210,20e,20d,209,204,1f8,1f3,1f0,1ef,1e1,1de,1dc,1d8,1d4,1cd,1bf,1bd,199,172,16e] ondisk+write+known_if_redirected e656)", 2021-03-24T22:05:02.563 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:03:46.370646+0000", 2021-03-24T22:05:02.563 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 75.823300693999997, 2021-03-24T22:05:02.563 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.096840674000000002, 2021-03-24T22:05:02.563 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:05:02.563 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:05:02.564 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:05:02.564 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:05:02.564 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:05:02.564 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 6390 2021-03-24T22:05:02.564 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.565 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:05:02.565 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.565 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:05:02.565 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.370646+0000", 2021-03-24T22:05:02.565 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:05:02.566 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.566 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.566 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:05:02.566 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.370646+0000", 2021-03-24T22:05:02.567 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2021-03-24T22:05:02.567 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.567 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.567 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:05:02.567 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.370645+0000", 2021-03-24T22:05:02.568 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8290000000000001e-06 2021-03-24T22:05:02.568 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.568 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.568 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:05:02.569 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.370649+0000", 2021-03-24T22:05:02.569 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6899999999999992e-06 2021-03-24T22:05:02.569 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.569 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.569 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:05:02.570 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.370657+0000", 2021-03-24T22:05:02.570 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6709999999999999e-06 2021-03-24T22:05:02.570 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.570 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.571 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:05:02.571 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.370660+0000", 2021-03-24T22:05:02.571 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3026000000000001e-05 2021-03-24T22:05:02.571 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.571 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.572 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:05:02.572 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.370743+0000", 2021-03-24T22:05:02.572 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0320000000000003e-06 2021-03-24T22:05:02.572 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.572 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.573 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for peered", 2021-03-24T22:05:02.573 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.370751+0000", 2021-03-24T22:05:02.573 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.091521704999999995 2021-03-24T22:05:02.573 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.573 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.574 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:05:02.574 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.462272+0000", 2021-03-24T22:05:02.574 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000145391 2021-03-24T22:05:02.574 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.575 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.575 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:05:02.575 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.462418+0000", 2021-03-24T22:05:02.575 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000244035 2021-03-24T22:05:02.576 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.576 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.576 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1,4", 2021-03-24T22:05:02.576 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.462662+0000", 2021-03-24T22:05:02.576 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0014480490000000001 2021-03-24T22:05:02.577 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.577 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.577 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T22:05:02.577 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.464110+0000", 2021-03-24T22:05:02.577 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00053739499999999997 2021-03-24T22:05:02.578 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.578 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.578 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:05:02.578 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.464647+0000", 2021-03-24T22:05:02.579 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.002517334 2021-03-24T22:05:02.579 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.579 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.579 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:05:02.579 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.467164+0000", 2021-03-24T22:05:02.580 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2069000000000002e-05 2021-03-24T22:05:02.580 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.580 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.580 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:05:02.580 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.467257+0000", 2021-03-24T22:05:02.581 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00023028999999999999 2021-03-24T22:05:02.581 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.581 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.581 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:05:02.582 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.467487+0000", 2021-03-24T22:05:02.582 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.096840674000000002 2021-03-24T22:05:02.582 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:05:02.582 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:05:02.582 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:05:02.583 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.583 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.583 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:6391 7.22 7:454f21f6:::smithi12556470-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e656)", 2021-03-24T22:05:02.583 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:03:46.370682+0000", 2021-03-24T22:05:02.584 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 75.823264488000007, 2021-03-24T22:05:02.584 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.097224654999999993, 2021-03-24T22:05:02.584 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:05:02.584 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:05:02.584 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:05:02.585 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:05:02.585 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:05:02.585 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 6391 2021-03-24T22:05:02.585 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.585 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:05:02.586 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.586 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:05:02.586 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.370682+0000", 2021-03-24T22:05:02.586 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:05:02.587 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.587 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.587 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:05:02.587 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.370682+0000", 2021-03-24T22:05:02.588 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2021-03-24T22:05:02.588 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.588 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.588 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:05:02.589 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.370682+0000", 2021-03-24T22:05:02.589 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0190000000000001e-06 2021-03-24T22:05:02.589 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.589 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.589 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:05:02.590 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.370684+0000", 2021-03-24T22:05:02.590 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.96e-06 2021-03-24T22:05:02.590 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.590 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.590 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:05:02.591 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.370692+0000", 2021-03-24T22:05:02.591 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8030000000000001e-06 2021-03-24T22:05:02.591 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.591 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.592 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:05:02.592 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.370695+0000", 2021-03-24T22:05:02.592 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6218999999999996e-05 2021-03-24T22:05:02.592 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.592 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.593 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:05:02.593 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.370771+0000", 2021-03-24T22:05:02.593 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.7910000000000004e-06 2021-03-24T22:05:02.593 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.593 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.594 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for peered", 2021-03-24T22:05:02.594 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.370776+0000", 2021-03-24T22:05:02.594 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.092529859000000006 2021-03-24T22:05:02.602 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.602 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.602 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:05:02.602 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.463306+0000", 2021-03-24T22:05:02.603 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00020765099999999999 2021-03-24T22:05:02.604 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.604 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.604 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for rw locks", 2021-03-24T22:05:02.605 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.463513+0000", 2021-03-24T22:05:02.605 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0040383470000000003 2021-03-24T22:05:02.605 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.605 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.606 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:05:02.606 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.467552+0000", 2021-03-24T22:05:02.606 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014007399999999999 2021-03-24T22:05:02.606 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.607 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.607 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:05:02.607 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.467692+0000", 2021-03-24T22:05:02.607 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000215341 2021-03-24T22:05:02.608 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.608 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.608 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:05:02.608 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.467907+0000", 2021-03-24T22:05:02.609 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.097224654999999993 2021-03-24T22:05:02.609 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:05:02.609 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:05:02.609 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:05:02.610 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.610 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.610 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:6917 7.17 7:ec6a359b:::smithi12556470-33:head [write 598580~645479,stat] snapc 253=[253,252,24f,24d,247,246,244,243,242,240,23f,23d,23b,236,235,234,231,22d,22c,225,21f,21d,210,209,204,1ef,1de,1dc,1d8,1bf,199,172,16e] ondisk+write+known_if_redirected e714)", 2021-03-24T22:05:02.611 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:04:44.443421+0000", 2021-03-24T22:05:02.611 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 17.750525719999999, 2021-03-24T22:05:02.611 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.036895271, 2021-03-24T22:05:02.612 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:05:02.612 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:05:02.612 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:05:02.612 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:05:02.612 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:05:02.613 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 6917 2021-03-24T22:05:02.613 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.613 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:05:02.613 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.613 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:05:02.614 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:44.443421+0000", 2021-03-24T22:05:02.614 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:05:02.614 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.614 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.615 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:05:02.615 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:44.443421+0000", 2021-03-24T22:05:02.615 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.99999 2021-03-24T22:05:02.615 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.615 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.616 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:05:02.616 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:44.443411+0000", 2021-03-24T22:05:02.616 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00079949999999999997 2021-03-24T22:05:02.616 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.617 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.617 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:05:02.617 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:44.444211+0000", 2021-03-24T22:05:02.617 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5933999999999998e-05 2021-03-24T22:05:02.617 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.618 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.618 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:05:02.618 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:44.444247+0000", 2021-03-24T22:05:02.618 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3492000000000001e-05 2021-03-24T22:05:02.618 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.619 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.619 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:05:02.619 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:44.444260+0000", 2021-03-24T22:05:02.619 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.7167000000000002e-05 2021-03-24T22:05:02.620 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.620 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.620 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:05:02.620 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:44.444337+0000", 2021-03-24T22:05:02.620 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015719900000000001 2021-03-24T22:05:02.621 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.621 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.621 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:05:02.621 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:44.444494+0000", 2021-03-24T22:05:02.622 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00033164300000000001 2021-03-24T22:05:02.622 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.622 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.622 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 2", 2021-03-24T22:05:02.622 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:44.444826+0000", 2021-03-24T22:05:02.623 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0025392119999999999 2021-03-24T22:05:02.623 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.623 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.623 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T22:05:02.624 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:44.447365+0000", 2021-03-24T22:05:02.624 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.03258291 2021-03-24T22:05:02.624 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.624 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.625 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:05:02.625 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:44.479948+0000", 2021-03-24T22:05:02.625 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8844999999999994e-05 2021-03-24T22:05:02.625 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.626 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.626 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:05:02.626 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:44.480037+0000", 2021-03-24T22:05:02.626 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00027940099999999998 2021-03-24T22:05:02.626 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:05:02.627 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:05:02.627 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:05:02.627 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:44.480316+0000", 2021-03-24T22:05:02.627 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.036895271 2021-03-24T22:05:02.628 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:05:02.628 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:05:02.628 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:05:02.628 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:05:02.628 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:05:02.629 INFO:teuthology.orchestra.run.smithi142.stdout:} 2021-03-24T22:05:02.629 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:02.630 INFO:tasks.rados.rados.0.smithi125.stdout:2812: done (1 left) 2021-03-24T22:05:02.631 INFO:tasks.rados.rados.0.smithi125.stdout:2814: done (0 left) 2021-03-24T22:05:02.631 INFO:tasks.rados.rados.0.smithi125.stdout:2815: snap_create 2021-03-24T22:05:02.631 INFO:journalctl@ceph.osd.5.smithi142.stdout:Mar 24 22:05:02 smithi142 conmon[90637]: debug 2021-03-24T22:05:02.195+0000 7ff07f500080 -1 osd.5 662 log_to_monitors {default=true} 2021-03-24T22:05:02.632 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2021-03-24T22:05:02.632 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2021-03-24T22:05:02.633 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2021-03-24T22:05:02.633 INFO:teuthology.orchestra.run.smithi142.stdout:} 2021-03-24T22:05:02.680 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:02.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:02.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:05:02.719 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-24T22:05:02.785 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:02.830 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:02.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:02.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:05:02.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:02 smithi142 conmon[20125]: cluster 2021-03-24T22:05:00.970492+0000 mgr.y (mgr.14144) 640 : 2021-03-24T22:05:02.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:02 smithi142 conmon[20125]: cluster [DBG] pgmap v1258: 168 pgs: 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 108 active+clean; 700 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 865 KiB/s wr, 10 op/s; 171/1980 objects degraded (8.636%); 20/1980 objects misplaced (1.010%) 2021-03-24T22:05:02.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:02 smithi142 conmon[20125]: cluster 2021-03-24T22:05:01.661034+0000 osd.6 (osd 2021-03-24T22:05:02.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:02 smithi142 conmon[20125]: .6) 67 : cluster [DBG] 7.23 scrub ok 2021-03-24T22:05:02.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:02 smithi142 conmon[20125]: audit 2021-03-24T22: 2021-03-24T22:05:02.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:02 smithi142 conmon[20125]: 05:02.200471+0000 mon.a (mon.0 2021-03-24T22:05:02.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:02 smithi142 conmon[20125]: ) 1895 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-03-24T22:05:02.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:02 smithi142 conmon[20125]: audit 2021-03- 2021-03-24T22:05:02.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:02 smithi142 conmon[20125]: 24T22:05:02.202646+0000 mon.b 2021-03-24T22:05:02.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:02 smithi142 conmon[20125]: (mon.1) 40 : audit [INF] from='osd.5 v1:172.21.15.142:6804/1323664769' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-03-24T22:05:02.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:02 smithi142 conmon[20125]: 2021-03-24T22:05:02.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:02 smithi142 conmon[20125]: cluster 2021-03-24T22:05:02.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:02 smithi142 conmon[20125]: 2021-03-24T22:05:02.207446+0000 osd.3 2021-03-24T22:05:02.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:02 smithi142 conmon[20125]: (osd.3) 146 : cluster [DBG] 3.b scrub ok 2021-03-24T22:05:02.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:02 smithi142 conmon[20125]: 2021-03-24T22:05:02.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:02 smithi142 conmon[20125]: audit 2021-03-24T22:05:02.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:02 smithi142 conmon[20125]: 2021-03-24T22:05:02.398149+0000 mon.a 2021-03-24T22:05:02.872 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:02 smithi142 conmon[20125]: (mon.0) 1896 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2021-03-24T22:05:02.872 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:02 smithi142 conmon[20125]: 2021-03-24T22:05:02.872 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:02 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:05:02.872 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:02 smithi142 conmon[20125]: -03-24T22:05:02.398358+0000 mon.a ( 2021-03-24T22:05:02.873 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:02 smithi142 conmon[20125]: mon.0) 1897 : cluster [DBG] osdmap e732: 8 total, 7 up, 8 in 2021-03-24T22:05:02.873 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:02 smithi142 conmon[20125]: audit 2021 2021-03-24T22:05:02.873 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:02 smithi142 conmon[20125]: -03-24T22:05:02.402465 2021-03-24T22:05:02.874 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:02 smithi142 conmon[20125]: +0000 mon.a (mon.0) 1898 : audit 2021-03-24T22:05:02.874 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:02 smithi142 conmon[20125]: [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2021-03-24T22:05:02.874 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:02 smithi142 conmon[20125]: audit 2021-03-24T22:05:02.875 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:02 smithi142 conmon[20125]: 2021 2021-03-24T22:05:02.875 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:02 smithi142 conmon[20125]: -03-24T22:05:02.404703 2021-03-24T22:05:02.875 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:02 smithi142 conmon[20125]: +0000 mon.b (mon.1) 41 : audit 2021-03-24T22:05:02.876 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:02 smithi142 conmon[20125]: [INF] from='osd.5 v1:172.21.15.142:6804/1323664769' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2021-03-24T22:05:02.912 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[24927]: cluster 2021-03-24T22:05:00 2021-03-24T22:05:02.912 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[24927]: .970492+0000 mgr.y (mgr.14144) 640 : cluster [DBG] pgmap v1258: 168 pgs: 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 108 active+clean; 700 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 865 KiB/s wr, 10 op/s; 171/1980 objects degraded (8.636%); 20/1980 objects misplaced (1.010%) 2021-03-24T22:05:02.912 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T22:05:02.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[24927]: -24T22:05:01.661034+0000 osd.6 (osd.6) 67 : cluster [DBG] 7.23 scrub ok 2021-03-24T22:05:02.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[24927]: audit 2021-03-24T22: 2021-03-24T22:05:02.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[24927]: 05:02.200471+0000 mon.a (mon.0) 1895 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-03-24T22:05:02.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[24927]: 2021-03-24T22:05:02.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[24927]: audit 2021-03-24T22:05:02.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[24927]: 2021-03-24T22:05:02.202646+0000 mon.b (mon. 2021-03-24T22:05:02.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[24927]: 1) 40 : audit [INF] from='osd.5 v1:172.21.15.142:6804/1323664769' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-03-24T22:05:02.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[24927]: cluster 2021-03-24T22:05:02. 2021-03-24T22:05:02.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[24927]: 207446+0000 osd.3 (osd.3) 146 : cluster [DBG] 3.b scrub ok 2021-03-24T22:05:02.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[24927]: audit 2021-03-24T22:05:02.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[24927]: 2021-03-24T22 2021-03-24T22:05:02.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[24927]: :05:02.398149+0000 mon.a (mon.0) 1896 : 2021-03-24T22:05:02.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[24927]: audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2021-03-24T22:05:02.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:05:02.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[24927]: 05:02.398358+0000 mon.a (mon.0) 1897 : 2021-03-24T22:05:02.917 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[24927]: cluster [DBG] osdmap e732: 8 total, 7 up, 8 in 2021-03-24T22:05:02.917 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[24927]: audit 2021-03-24 2021-03-24T22:05:02.917 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[24927]: T22:05:02.402465+0000 mon.a (mon 2021-03-24T22:05:02.917 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[24927]: .0) 1898 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2021-03-24T22:05:02.918 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[24927]: audit 2021- 2021-03-24T22:05:02.918 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[24927]: 03-24T22:05:02.404703+0000 2021-03-24T22:05:02.918 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[24927]: mon.b (mon.1) 41 : audit [INF] from='osd.5 v1:172.21.15.142:6804/1323664769' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2021-03-24T22:05:02.919 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T22:05:02.919 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: -24T22:05:00.970492+0000 mgr.y (mgr.14144) 640 : cluster 2021-03-24T22:05:02.919 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: [DBG] pgmap v1258: 168 pgs: 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 108 active+clean; 700 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 865 KiB/s wr, 10 op/s; 171/1980 objects degraded (8.636%); 20/1980 objects misplaced (1.010%) 2021-03-24T22:05:02.920 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: cluster 2021-03-24T22:05:02.920 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: 2021-03-24T22 2021-03-24T22:05:02.920 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: :05:01. 2021-03-24T22:05:02.920 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: 661034+0000 osd.6 (osd. 2021-03-24T22:05:02.921 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: 6) 67 : cluster [DBG] 7.23 scrub ok 2021-03-24T22:05:02.921 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: 2021-03-24T22:05:02.921 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: audit 2021-03-24T22:05:02.921 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: 2021- 2021-03-24T22:05:02.922 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: 03-24T 2021-03-24T22:05:02.922 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: 22:05: 2021-03-24T22:05:02.922 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: 02.200471+0000 2021-03-24T22:05:02.922 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: mon.a (mon. 2021-03-24T22:05:02.922 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: 0) 1895 : audit [INF] 2021-03-24T22:05:02.923 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-03-24T22:05:02.923 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: audit 2021-03-24T22:05:02.923 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: 2021-03-24T 2021-03-24T22:05:02.923 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: 22:05:02 2021-03-24T22:05:02.924 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: .202646+0000 mon.b ( 2021-03-24T22:05:02.924 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: mon.1) 40 : 2021-03-24T22:05:02.924 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: audit [INF] from='osd.5 v1:172.21.15.142:6804/1323664769' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-03-24T22:05:02.925 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: 2021-03-24T22:05:02.925 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: cluster 2021-03-24T22:05:02.925 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: 2021-03- 2021-03-24T22:05:02.925 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: 24T22: 2021-03-24T22:05:02.926 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: 05:02.207446 2021-03-24T22:05:02.926 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: +0000 osd.3 (osd 2021-03-24T22:05:02.926 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: .3) 146 : cluster 2021-03-24T22:05:02.926 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: [DBG] 3.b scrub ok 2021-03-24T22:05:02.927 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: audit 2021-03-24T22:05:02.927 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: 2021-03-24T22 2021-03-24T22:05:02.927 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: :05:02.398149+0000 2021-03-24T22:05:02.928 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: mon.a (mon.0) 2021-03-24T22:05:02.928 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: 1896 : audit [INF] 2021-03-24T22:05:02.928 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2021-03-24T22:05:02.929 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:05:02.929 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: -03-24 2021-03-24T22:05:02.929 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: T22: 2021-03-24T22:05:02.930 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: 05:02. 2021-03-24T22:05:02.930 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: 398358+0000 2021-03-24T22:05:02.930 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: mon.a (mon. 2021-03-24T22:05:02.930 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: 0) 1897 : 2021-03-24T22:05:02.931 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: cluster [DBG] 2021-03-24T22:05:02.931 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: osdmap e732: 8 total, 7 up, 8 in 2021-03-24T22:05:02.931 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: audit 2021-03-24T22:05:02.931 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: 2021-03-24T 2021-03-24T22:05:02.931 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: 22:05: 2021-03-24T22:05:02.932 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: 02.402465 2021-03-24T22:05:02.932 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: +0000 mon.a ( 2021-03-24T22:05:02.932 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: mon.0) 2021-03-24T22:05:02.932 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: 1898 : audit [INF] 2021-03-24T22:05:02.933 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2021-03-24T22:05:02.933 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: audit 2021-03-24T22:05:02.933 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: 2021-03-24T22: 2021-03-24T22:05:02.933 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: 05:02.404703+0000 mon.b 2021-03-24T22:05:02.933 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: (mon.1) 41 : audit 2021-03-24T22:05:02.934 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:02 smithi125 conmon[19936]: [INF] from='osd.5 v1:172.21.15.142:6804/1323664769' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2021-03-24T22:05:02.939 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:02.991 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:02.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:02.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:05:03.018 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:05:03.099 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:03.157 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:03.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:03.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:05:03.263 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:03.308 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:03.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:03.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:05:03.413 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:03.428 INFO:tasks.rados.rados.0.smithi125.stdout:2815: done (0 left) 2021-03-24T22:05:03.428 INFO:tasks.rados.rados.0.smithi125.stdout:2816: copy_from oid 38 from oid 23 current snap is 324 2021-03-24T22:05:03.429 INFO:tasks.rados.rados.0.smithi125.stdout:2817: read oid 15 snap -1 2021-03-24T22:05:03.429 INFO:tasks.rados.rados.0.smithi125.stdout:2817: expect deleted 2021-03-24T22:05:03.429 INFO:tasks.rados.rados.0.smithi125.stdout:2818: read oid 44 snap -1 2021-03-24T22:05:03.429 INFO:tasks.rados.rados.0.smithi125.stdout:2818: expect (ObjNum 937 snap 312 seq_num 937) 2021-03-24T22:05:03.430 INFO:tasks.rados.rados.0.smithi125.stdout:2819: read oid 7 snap -1 2021-03-24T22:05:03.430 INFO:tasks.rados.rados.0.smithi125.stdout:2819: expect (ObjNum 727 snap 233 seq_num 727) 2021-03-24T22:05:03.430 INFO:tasks.rados.rados.0.smithi125.stdout:2820: read oid 27 snap -1 2021-03-24T22:05:03.430 INFO:tasks.rados.rados.0.smithi125.stdout:2820: expect deleted 2021-03-24T22:05:03.431 INFO:tasks.rados.rados.0.smithi125.stdout:2821: rollback oid 33 current snap is 324 2021-03-24T22:05:03.431 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 33 to 251 2021-03-24T22:05:03.431 INFO:tasks.rados.rados.0.smithi125.stdout:2822: snap_create 2021-03-24T22:05:03.435 INFO:tasks.rados.rados.0.smithi125.stdout:2819: expect (ObjNum 727 snap 233 seq_num 727) 2021-03-24T22:05:03.470 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:03.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:03.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:05:03.575 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:03.627 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:03.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:03.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:05:03.634 INFO:tasks.rados.rados.0.smithi125.stdout:2818: expect (ObjNum 937 snap 312 seq_num 937) 2021-03-24T22:05:03.731 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:03.746 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:03 smithi142 conmon[20125]: cluster 2021-03-24T22:05:02 2021-03-24T22:05:03.746 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:03 smithi142 conmon[20125]: .664953+0000 osd.6 (osd.6) 68 : cluster [DBG] 3.a scrub ok 2021-03-24T22:05:03.746 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:03 smithi142 conmon[20125]: cluster 2021-03-24T22:05:03.168150 2021-03-24T22:05:03.746 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:03 smithi142 conmon[20125]: +0000 osd.3 (osd.3) 147 : cluster [DBG] 7.2 scrub ok 2021-03-24T22:05:03.747 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:03 smithi142 conmon[20125]: cluster 2021-03-24T22:05:03.392255 2021-03-24T22:05:03.747 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:03 smithi142 conmon[20125]: +0000 mon.a (mon.0) 1899 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2021-03-24T22:05:03.747 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:03 smithi142 conmon[20125]: 2021-03-24T22:05:03.747 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:03 smithi142 conmon[20125]: cluster 2021-03-24T22:05:03 2021-03-24T22:05:03.748 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:03 smithi142 conmon[20125]: .404371+0000 mon.a (mon.0) 1900 2021-03-24T22:05:03.748 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:03 smithi142 conmon[20125]: : cluster [INF] osd.5 v1:172.21.15.142:6804/1323664769 boot 2021-03-24T22:05:03.748 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:03 smithi142 conmon[20125]: cluster 2021-03-24T22:05:03.748 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:03 smithi142 conmon[20125]: 2021-03-24T22:05:03.404456+0000 mon.a (mon 2021-03-24T22:05:03.748 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:03 smithi142 conmon[20125]: .0) 1901 : cluster [DBG] osdmap e733: 8 total, 8 up, 8 in 2021-03-24T22:05:03.749 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:03 smithi142 conmon[20125]: audit 2021-03-24T22 2021-03-24T22:05:03.749 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:03 smithi142 conmon[20125]: :05:03.416360+0000 mon.a (mon.0) 1902 : audit 2021-03-24T22:05:03.749 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:03 smithi142 conmon[20125]: [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-03-24T22:05:03.787 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:03.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:03.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:05:03.799 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[24927]: cluster 2021-03-24T22:05:02.664953+0000 2021-03-24T22:05:03.799 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[24927]: osd.6 (osd.6) 68 : cluster [DBG] 3.a scrub ok 2021-03-24T22:05:03.799 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[24927]: cluster 2021-03-24T22:05: 2021-03-24T22:05:03.799 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[24927]: 03.168150+0000 osd.3 (osd.3) 147 : cluster [DBG] 7.2 scrub ok 2021-03-24T22:05:03.800 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[24927]: cluster 2021-03-24T22 2021-03-24T22:05:03.800 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[24927]: :05:03.392255+0000 mon.a (mon.0) 1899 : cluster [INF] 2021-03-24T22:05:03.800 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[24927]: Health check cleared: OSD_DOWN (was: 1 osds down) 2021-03-24T22:05:03.800 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:05:03.801 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[24927]: -03-24T22:05:03.404371+0000 mon.a (mon.0) 1900 : 2021-03-24T22:05:03.801 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[24927]: cluster [INF] osd.5 v1:172.21.15.142:6804/1323664769 boot 2021-03-24T22:05:03.801 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[24927]: cluster 2021-03-24T22:05:03.404456 2021-03-24T22:05:03.801 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[24927]: +0000 mon.a (mon.0) 1901 : cluster [DBG] osdmap e733: 8 total, 8 up, 8 in 2021-03-24T22:05:03.801 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[24927]: 2021-03-24T22:05:03.802 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[24927]: audit 2021-03 2021-03-24T22:05:03.802 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[24927]: -24T22:05:03.416360+0000 mon.a (mon.0) 1902 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-03-24T22:05:03.802 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[24927]: 2021-03-24T22:05:03.803 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T22:05:03.803 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[19936]: -24T22:05:02.664953+0000 osd.6 (osd.6) 68 : cluster 2021-03-24T22:05:03.803 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[19936]: [DBG] 3.a scrub ok 2021-03-24T22:05:03.803 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[19936]: cluster 2021-03-24T22:05:03.804 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[19936]: 2021-03-24T22: 2021-03-24T22:05:03.804 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[19936]: 05:03. 2021-03-24T22:05:03.804 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[19936]: 168150+0000 osd.3 (osd 2021-03-24T22:05:03.804 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[19936]: .3) 147 : cluster 2021-03-24T22:05:03.804 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[19936]: [DBG] 7.2 scrub ok 2021-03-24T22:05:03.805 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[19936]: cluster 2021-03-24T22:05:03.805 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[19936]: 2021-03-24T 2021-03-24T22:05:03.805 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[19936]: 22:05:03. 2021-03-24T22:05:03.806 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[19936]: 392255+0000 mon.a ( 2021-03-24T22:05:03.806 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[19936]: mon.0) 1899 : 2021-03-24T22:05:03.806 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[19936]: cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2021-03-24T22:05:03.806 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[19936]: cluster 2021-03-24T22:05:03.806 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[19936]: 2021-03-24T 2021-03-24T22:05:03.807 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[19936]: 22:05:03 2021-03-24T22:05:03.808 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[19936]: .404371+0000 2021-03-24T22:05:03.809 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[19936]: mon.a (mon.0 2021-03-24T22:05:03.809 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[19936]: ) 1900 : cluster 2021-03-24T22:05:03.809 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[19936]: [INF] osd.5 v1:172.21.15.142:6804/1323664769 boot 2021-03-24T22:05:03.809 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[19936]: cluster 2021-03-24T22:05:03.810 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[19936]: 2021-03-24T 2021-03-24T22:05:03.810 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[19936]: 22:05:03 2021-03-24T22:05:03.810 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[19936]: .404456+0000 2021-03-24T22:05:03.810 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[19936]: mon.a (mon.0 2021-03-24T22:05:03.810 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[19936]: ) 1901 : cluster 2021-03-24T22:05:03.811 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[19936]: [DBG] osdmap e733: 8 total, 8 up, 8 in 2021-03-24T22:05:03.811 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[19936]: audit 2021-03-24T22:05:03.811 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[19936]: 2021-03-24 2021-03-24T22:05:03.811 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[19936]: T22:05 2021-03-24T22:05:03.812 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[19936]: :03.416360 2021-03-24T22:05:03.812 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[19936]: +0000 mon.a (mon 2021-03-24T22:05:03.812 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[19936]: .0) 1902 : 2021-03-24T22:05:03.812 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[19936]: audit [DBG] 2021-03-24T22:05:03.812 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:03 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-03-24T22:05:03.893 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:03.956 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:03.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:03.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:05:04.047 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.0 dump_ops_in_flight 2021-03-24T22:05:04.063 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:04.078 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd reweight 0 1 2021-03-24T22:05:04.143 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:04.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:04.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:05:04.249 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:04.306 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:04.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:04.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:05:04.410 INFO:tasks.rados.rados.0.smithi125.stdout:2817: done (6 left) 2021-03-24T22:05:04.410 INFO:tasks.rados.rados.0.smithi125.stdout:2818: done (5 left) 2021-03-24T22:05:04.411 INFO:tasks.rados.rados.0.smithi125.stdout:2819: done (4 left) 2021-03-24T22:05:04.411 INFO:tasks.rados.rados.0.smithi125.stdout:2820: done (3 left) 2021-03-24T22:05:04.411 INFO:tasks.rados.rados.0.smithi125.stdout:2822: done (2 left) 2021-03-24T22:05:04.411 INFO:tasks.rados.rados.0.smithi125.stdout:2823: snap_create 2021-03-24T22:05:04.414 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:04.426 INFO:teuthology.orchestra.run.smithi125.stderr:nodeep-scrub is unset 2021-03-24T22:05:04.476 INFO:tasks.rados.rados.0.smithi125.stdout:2816: finishing copy_from to smithi12556470-38 2021-03-24T22:05:04.477 INFO:tasks.rados.rados.0.smithi125.stdout:2816: got expected ENOENT (src dne) 2021-03-24T22:05:04.477 INFO:tasks.rados.rados.0.smithi125.stdout:2816: finishing copy_from racing read to smithi12556470-38 2021-03-24T22:05:04.477 INFO:tasks.rados.rados.0.smithi125.stdout:2821: finishing rollback tid 0 to smithi12556470-33 2021-03-24T22:05:04.477 INFO:tasks.rados.rados.0.smithi125.stdout:2821: finishing rollback tid 1 to smithi12556470-33 2021-03-24T22:05:04.477 INFO:tasks.rados.rados.0.smithi125.stdout:2821: finishing rollback tid 2 to smithi12556470-33 2021-03-24T22:05:04.478 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 33 v 1437 (ObjNum 764 snap 249 seq_num 764) dirty exists 2021-03-24T22:05:04.478 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:04.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:04.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:05:04.585 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:04.636 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:04.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:04.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:05:04.744 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:04.753 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:04 smithi125 conmon[24927]: cluster 2021-03-24T22:05:02 2021-03-24T22:05:04.754 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:04 smithi125 conmon[24927]: .971530+0000 mgr.y (mgr.14144) 641 : cluster [DBG] pgmap v1261: 168 pgs: 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 108 active+clean; 698 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.2 MiB/s wr, 11 op/s; 171/1977 objects degraded (8.649%); 20/1977 objects misplaced (1.012%) 2021-03-24T22:05:04.754 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:04 smithi125 conmon[24927]: audit 2021-03-24T22:05 2021-03-24T22:05:04.755 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:04 smithi125 conmon[24927]: :03.589837+0000 mon.a (mon.0) 2021-03-24T22:05:04.755 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:04 smithi125 conmon[24927]: 1903 : audit [INF] from='client.? v1:172.21.15.125:0/2783440468' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:05:04.755 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:04 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T22:05:04.755 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:04 smithi125 conmon[24927]: -24T22:05:04.320663+0000 2021-03-24T22:05:04.755 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:04 smithi125 conmon[24927]: osd.0 (osd.0) 143 : cluster [DBG] 3.3 scrub ok 2021-03-24T22:05:04.756 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:04 smithi125 conmon[24927]: 2021-03-24T22:05:04.756 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:04 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T22:05:04.756 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:04 smithi125 conmon[24927]: 22:05:04.396690+0000 mon.a (mon 2021-03-24T22:05:04.756 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:04 smithi125 conmon[24927]: .0) 1904 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T22:05:04.757 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:04 smithi125 conmon[24927]: audit 2021 2021-03-24T22:05:04.757 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:04 smithi125 conmon[24927]: -03-24T22:05:04.406853+0000 2021-03-24T22:05:04.757 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:04 smithi125 conmon[24927]: mon.a (mon.0) 1905 : audit [INF] 2021-03-24T22:05:04.757 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:04 smithi125 conmon[24927]: from='client.? v1:172.21.15.125:0/2783440468' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T22:05:04.758 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:04 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T22:05:04.758 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:04 smithi125 conmon[24927]: 22:05:04.406919+0000 mon.a (mon 2021-03-24T22:05:04.758 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:04 smithi125 conmon[24927]: .0) 1906 : cluster [DBG] osdmap e734: 8 total, 8 up, 8 in 2021-03-24T22:05:04.758 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:04 smithi125 conmon[19936]: cluster 2021-03-24T22:05:02.971530 2021-03-24T22:05:04.759 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:04 smithi125 conmon[19936]: +0000 mgr.y (mgr.14144) 641 : cluster [DBG] pgmap v1261: 168 pgs: 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 108 active+clean; 698 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.2 MiB/s wr, 11 op/s; 171/1977 objects degraded (8.649%); 20/1977 objects misplaced (1.012%) 2021-03-24T22:05:04.759 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:04 smithi125 conmon[19936]: audit 2021-03-24T22:05:03.589837+0000 mon.a (mon.0) 1903 : audit 2021-03-24T22:05:04.759 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:04 smithi125 conmon[19936]: [INF] from='client.? v1:172.21.15.125:0/2783440468' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:05:04.759 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:04 smithi125 conmon[19936]: cluster 2021-03-24T22:05:04.320663 2021-03-24T22:05:04.760 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:04 smithi125 conmon[19936]: +0000 osd.0 (osd.0) 143 : cluster [DBG] 3.3 scrub ok 2021-03-24T22:05:04.760 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:04 smithi125 conmon[19936]: cluster 2021-03-24T22 2021-03-24T22:05:04.760 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:04 smithi125 conmon[19936]: :05:04.396690+0000 mon.a (mon.0) 2021-03-24T22:05:04.760 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:04 smithi125 conmon[19936]: 1904 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T22:05:04.760 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:04 smithi125 conmon[19936]: audit 2021-03-24T22 2021-03-24T22:05:04.761 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:04 smithi125 conmon[19936]: :05:04.406853+0000 mon.a (mon.0 2021-03-24T22:05:04.761 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:04 smithi125 conmon[19936]: ) 1905 : audit [INF] from='client.? v1:172.21.15.125:0/2783440468' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T22:05:04.761 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:04 smithi125 conmon[19936]: cluster 2021-03-24T22:05 2021-03-24T22:05:04.761 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:04 smithi125 conmon[19936]: :04.406919+0000 mon.a (mon.0) 1906 2021-03-24T22:05:04.762 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:04 smithi125 conmon[19936]: : cluster [DBG] osdmap e734: 8 total, 8 up, 8 in 2021-03-24T22:05:04.788 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:04.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:04.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:05:04.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:04 smithi142 conmon[20125]: cluster 2021-03-24T22:05:02.971530+0000 mgr.y (mgr.14144) 641 : cluster 2021-03-24T22:05:04.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:04 smithi142 conmon[20125]: [DBG] pgmap v1261: 168 pgs: 1 active+clean+remapped, 40 active+undersized, 19 active+undersized+degraded, 108 active+clean; 698 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.2 MiB/s wr, 11 op/s; 171/1977 objects degraded (8.649%); 20/1977 objects misplaced (1.012%) 2021-03-24T22:05:04.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:04 smithi142 conmon[20125]: audit 2021-03-24T22:05:03.589837+0000 mon.a (mon.0) 1903 : 2021-03-24T22:05:04.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:04 smithi142 conmon[20125]: audit [INF] from='client.? v1:172.21.15.125:0/2783440468' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:05:04.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:04 smithi142 conmon[20125]: cluster 2021-03-24T22:05:04.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:04 smithi142 conmon[20125]: 2021-03-24T22:05:04.320663+0000 2021-03-24T22:05:04.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:04 smithi142 conmon[20125]: osd.0 (osd.0) 143 : cluster 2021-03-24T22:05:04.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:04 smithi142 conmon[20125]: [DBG] 3.3 scrub ok 2021-03-24T22:05:04.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:04 smithi142 conmon[20125]: cluster 2021-03-24T22 2021-03-24T22:05:04.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:04 smithi142 conmon[20125]: :05:04.396690+0000 mon.a (mon.0 2021-03-24T22:05:04.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:04 smithi142 conmon[20125]: ) 1904 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T22:05:04.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:04 smithi142 conmon[20125]: audit 2021-03-24T 2021-03-24T22:05:04.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:04 smithi142 conmon[20125]: 22:05:04.406853+0000 mon.a (mon 2021-03-24T22:05:04.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:04 smithi142 conmon[20125]: .0) 1905 : audit [INF] from='client.? v1:172.21.15.125:0/2783440468' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T22:05:04.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:04 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:05:04.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:04 smithi142 conmon[20125]: 03-24T22:05:04 2021-03-24T22:05:04.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:04 smithi142 conmon[20125]: .406919+0000 mon.a (mon.0) 1906 : cluster 2021-03-24T22:05:04.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:04 smithi142 conmon[20125]: [DBG] osdmap e734: 8 total, 8 up, 8 in 2021-03-24T22:05:04.893 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:04.952 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:04.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:04.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:05:05.058 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:05.120 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:05.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:05.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:05:05.225 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:05.279 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:05.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:05.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:05:05.386 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:05.415 INFO:tasks.rados.rados.0.smithi125.stdout:2816: done (2 left) 2021-03-24T22:05:05.415 INFO:tasks.rados.rados.0.smithi125.stdout:2821: done (1 left) 2021-03-24T22:05:05.416 INFO:tasks.rados.rados.0.smithi125.stdout:2823: done (0 left) 2021-03-24T22:05:05.416 INFO:tasks.rados.rados.0.smithi125.stdout:2824: write (excl) oid 48 current snap is 326 2021-03-24T22:05:05.416 INFO:tasks.rados.rados.0.smithi125.stdout:2824: seq_num 984 ranges {538088=654553,1714752=634675} 2021-03-24T22:05:05.424 INFO:tasks.rados.rados.0.smithi125.stdout:2824: writing smithi12556470-48 from 538088 to 1192641 tid 1 2021-03-24T22:05:05.431 INFO:tasks.rados.rados.0.smithi125.stdout:2824: writing smithi12556470-48 from 1714752 to 2349427 tid 2 2021-03-24T22:05:05.431 INFO:tasks.rados.rados.0.smithi125.stdout:2825: snap_remove snap 312 2021-03-24T22:05:05.443 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:05.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:05.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:05:05.549 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:05.606 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:05.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:05.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:05:05.713 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:05.775 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:05.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:05.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:05:05.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:05 smithi142 conmon[20125]: cluster 2021-03-24T22:05:03.201691 2021-03-24T22:05:05.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:05 smithi142 conmon[20125]: +0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:05:05.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:05 smithi142 conmon[20125]: cluster 2021-03-24T22:05:03.206535+0000 osd.5 (osd.5) 2 : cluster 2021-03-24T22:05:05.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:05 smithi142 conmon[20125]: [DBG] purged_snaps scrub found 3 strays 2021-03-24T22:05:05.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:05 smithi142 conmon[20125]: audit 2021-03-24T22: 2021-03-24T22:05:05.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:05 smithi142 conmon[20125]: 05:04.836376+0000 mon.a (mon.0) 1907 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T22:05:05.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:05 smithi142 conmon[20125]: 2021-03-24T22:05:05.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:05 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:05:05.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:05 smithi142 conmon[20125]: 22:05:05.374339+0000 mon.a (mon.0) 1908 : cluster [WRN] 2021-03-24T22:05:05.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:05 smithi142 conmon[20125]: Health check update: Degraded data redundancy: 171/1977 objects degraded (8.649%), 19 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2021-03-24T22:05:05.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:05 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:05:05.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:05 smithi142 conmon[20125]: -03-24T22:05:05.410590+0000 mon.a (mon.0) 1909 2021-03-24T22:05:05.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:05 smithi142 conmon[20125]: : cluster [DBG] osdmap e735: 8 total, 8 up, 8 in 2021-03-24T22:05:05.881 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:05:05.882 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:05.883 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:05:05.940 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:05.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:05.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:05:06.046 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:06.105 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:06.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:06.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:05:06.155 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:05 smithi125 conmon[19936]: cluster 2021-03-24T22 2021-03-24T22:05:06.155 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:05 smithi125 conmon[19936]: :05:03.201691+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:05:06.155 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:05 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T22:05:06.155 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:05 smithi125 conmon[19936]: 22:05:03.206535+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub found 3 strays 2021-03-24T22:05:06.156 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:05 smithi125 conmon[19936]: 2021-03-24T22:05:06.156 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:05 smithi125 conmon[19936]: audit 2021-03-24T22:05: 2021-03-24T22:05:06.156 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:05 smithi125 conmon[19936]: 04.836376+0000 mon.a (mon.0) 1907 : audit 2021-03-24T22:05:06.156 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:05 smithi125 conmon[19936]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T22:05:06.157 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:05 smithi125 conmon[19936]: cluster 2021-03-24T22:05: 2021-03-24T22:05:06.157 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:05 smithi125 conmon[19936]: 05.374339+0000 mon.a (mon.0) 1908 : cluster 2021-03-24T22:05:06.157 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:05 smithi125 conmon[19936]: [WRN] Health check update: Degraded data redundancy: 171/1977 objects degraded (8.649%), 19 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2021-03-24T22:05:06.157 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:05 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:05:06.158 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:05 smithi125 conmon[19936]: 05:05.410590+0000 mon.a (mon.0) 1909 2021-03-24T22:05:06.158 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:05 smithi125 conmon[19936]: : cluster [DBG] osdmap e735: 8 total, 8 up, 8 in 2021-03-24T22:05:06.158 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:05 smithi125 conmon[24927]: cluster 2021-03-24T22:05:03. 2021-03-24T22:05:06.159 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:05 smithi125 conmon[24927]: 201691+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:05:06.159 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:05 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:05:06.159 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:05 smithi125 conmon[24927]: 03-24T22:05:03.206535+0000 osd.5 (osd.5) 2 : 2021-03-24T22:05:06.159 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:05 smithi125 conmon[24927]: cluster [DBG] purged_snaps scrub found 3 strays 2021-03-24T22:05:06.160 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:05 smithi125 conmon[24927]: audit 2021-03-24T22: 2021-03-24T22:05:06.160 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:05 smithi125 conmon[24927]: 05:04.836376+0000 mon.a (mon.0) 1907 2021-03-24T22:05:06.160 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:05 smithi125 conmon[24927]: : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T22:05:06.160 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:05 smithi125 conmon[24927]: cluster 2021-03-24T22:05: 2021-03-24T22:05:06.160 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:05 smithi125 conmon[24927]: 05.374339+0000 mon.a (mon.0) 1908 : cluster 2021-03-24T22:05:06.161 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:05 smithi125 conmon[24927]: [WRN] Health check update: Degraded data redundancy: 171/1977 objects degraded (8.649%), 19 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2021-03-24T22:05:06.161 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:05 smithi125 conmon[24927]: cluster 2021-03-24T22:05:05. 2021-03-24T22:05:06.161 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:05 smithi125 conmon[24927]: 410590+0000 mon.a (mon.0) 1909 : cluster [DBG] 2021-03-24T22:05:06.161 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:05 smithi125 conmon[24927]: osdmap e735: 8 total, 8 up, 8 in 2021-03-24T22:05:06.211 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:06.271 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:06.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:06.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:05:06.377 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:06.416 INFO:tasks.rados.rados.0.smithi125.stdout:2825: done (1 left) 2021-03-24T22:05:06.416 INFO:tasks.rados.rados.0.smithi125.stdout:2826: snap_remove snap 324 2021-03-24T22:05:06.465 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:06.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:06.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:05:06.570 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:06.578 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T22:05:06.596 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T22:05:06.620 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T22:05:06.631 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:06.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:06.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:05:06.646 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T22:05:06.668 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T22:05:06.691 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T22:05:06.694 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:06 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:05:06.695 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:06 smithi125 conmon[24927]: 05:04.972835+0000 mgr.y (mgr.14144) 642 : cluster [DBG] pgmap v1264: 168 pgs: 1 remapped+peering, 19 peering, 30 active+undersized, 10 active+undersized+degraded, 108 active+clean; 677 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 65/1944 objects degraded (3.344%) 2021-03-24T22:05:06.695 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:06 smithi125 conmon[24927]: cluster 2021-03-24T22:05: 2021-03-24T22:05:06.695 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:06 smithi125 conmon[24927]: 05.665321+0000 osd.1 (osd.1) 78 : cluster 2021-03-24T22:05:06.696 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:06 smithi125 conmon[24927]: [DBG] 3.1c scrub ok 2021-03-24T22:05:06.696 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:06 smithi125 conmon[24927]: cluster 2021-03-24T22 2021-03-24T22:05:06.696 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:06 smithi125 conmon[24927]: :05:06.161390+0000 osd.5 (osd.5 2021-03-24T22:05:06.697 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:06 smithi125 conmon[24927]: ) 3 : cluster [DBG] 5.15 scrub ok 2021-03-24T22:05:06.697 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:06 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T22:05:06.698 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:06 smithi125 conmon[24927]: -24T22:05:06.410618+0000 mon.a ( 2021-03-24T22:05:06.698 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:06 smithi125 conmon[24927]: mon.0) 1910 : cluster [DBG] osdmap e736: 8 total, 8 up, 8 in 2021-03-24T22:05:06.698 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:06 smithi125 conmon[24927]: audit 2021- 2021-03-24T22:05:06.699 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:06 smithi125 conmon[24927]: 03-24T22:05:06.550507+0000 mon.c 2021-03-24T22:05:06.699 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:06 smithi125 conmon[24927]: (mon.2) 106 : audit [DBG] from='client.? v1:172.21.15.125:0/335327748' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:05:06.700 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:06 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T22:05:06.700 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:06 smithi125 conmon[19936]: 22:05:04.972835+0000 mgr.y (mgr.14144) 642 : cluster [DBG] pgmap v1264: 168 pgs: 1 remapped+peering, 19 peering, 30 active+undersized, 10 active+undersized+degraded, 108 active+clean; 677 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 65/1944 objects degraded (3.344%) 2021-03-24T22:05:06.701 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:06 smithi125 conmon[19936]: 2021-03-24T22:05:06.701 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:06 smithi125 conmon[19936]: cluster 2021-03-24T22:05:06.702 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:06 smithi125 conmon[19936]: 2021-03-24T22: 2021-03-24T22:05:06.702 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:06 smithi125 conmon[19936]: 05:05.665321+0000 2021-03-24T22:05:06.703 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:06 smithi125 conmon[19936]: osd.1 (osd.1) 78 2021-03-24T22:05:06.703 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:06 smithi125 conmon[19936]: : cluster [DBG] 3.1c scrub ok 2021-03-24T22:05:06.703 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:06 smithi125 conmon[19936]: cluster 2021-03-24T22:05:06.704 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:06 smithi125 conmon[19936]: 2021-03-24T22 2021-03-24T22:05:06.704 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:06 smithi125 conmon[19936]: :05:06 2021-03-24T22:05:06.705 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:06 smithi125 conmon[19936]: .161390+0000 2021-03-24T22:05:06.705 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:06 smithi125 conmon[19936]: osd.5 (osd. 2021-03-24T22:05:06.705 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:06 smithi125 conmon[19936]: 5) 3 : cluster 2021-03-24T22:05:06.706 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:06 smithi125 conmon[19936]: [DBG] 5.15 scrub ok 2021-03-24T22:05:06.706 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:06 smithi125 conmon[19936]: cluster 2021-03-24T22:05:06.707 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:06 smithi125 conmon[19936]: 2021-03-24T 2021-03-24T22:05:06.707 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:06 smithi125 conmon[19936]: 22:05:06 2021-03-24T22:05:06.708 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:06 smithi125 conmon[19936]: .410618+0000 mon.a 2021-03-24T22:05:06.708 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:06 smithi125 conmon[19936]: (mon. 2021-03-24T22:05:06.709 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:06 smithi125 conmon[19936]: 0) 1910 : 2021-03-24T22:05:06.709 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:06 smithi125 conmon[19936]: cluster [DBG] 2021-03-24T22:05:06.709 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:06 smithi125 conmon[19936]: osdmap e736: 8 total, 8 up, 8 in 2021-03-24T22:05:06.710 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:06 smithi125 conmon[19936]: audit 2021-03-24T22:05:06.710 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:06 smithi125 conmon[19936]: 2021-03-24T 2021-03-24T22:05:06.711 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:06 smithi125 conmon[19936]: 22:05:06 2021-03-24T22:05:06.711 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:06 smithi125 conmon[19936]: .550507+0000 2021-03-24T22:05:06.712 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:06 smithi125 conmon[19936]: mon.c (mon 2021-03-24T22:05:06.712 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:06 smithi125 conmon[19936]: .2) 2021-03-24T22:05:06.712 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:06 smithi125 conmon[19936]: 106 : audit 2021-03-24T22:05:06.713 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:06 smithi125 conmon[19936]: [DBG] 2021-03-24T22:05:06.713 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:06 smithi125 conmon[19936]: from='client.? v1:172.21.15.125:0/335327748' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:05:06.717 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T22:05:06.737 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:06.741 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T22:05:06.744 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-24T22:05:06.744 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-24T22:05:06.744 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-24T22:05:06.745 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-24T22:05:06.745 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-24T22:05:06.745 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-24T22:05:06.745 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-24T22:05:06.746 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-24T22:05:06.789 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:06.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:06.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:05:06.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:06 smithi142 conmon[20125]: cluster 2021-03-24T22:05:04.972835+0000 mgr.y (mgr.14144) 642 : cluster [DBG] pgmap v1264: 168 pgs: 1 remapped+peering, 19 peering, 30 active+undersized, 10 active+undersized+degraded, 108 active+clean; 677 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 65/1944 objects degraded (3.344%) 2021-03-24T22:05:06.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:06 smithi142 conmon[20125]: cluster 2021-03-24T22:05:05.665321+0000 osd.1 (osd.1) 78 : cluster [DBG] 3.1c scrub ok 2021-03-24T22:05:06.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:06 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:05:06.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:06 smithi142 conmon[20125]: 22:05:06.161390+0000 osd.5 (osd.5) 3 : cluster [DBG] 2021-03-24T22:05:06.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:06 smithi142 conmon[20125]: 5.15 scrub ok 2021-03-24T22:05:06.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:06 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:05:06.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:06 smithi142 conmon[20125]: 24T22:05:06.410618+0000 mon.a (mon.0 2021-03-24T22:05:06.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:06 smithi142 conmon[20125]: ) 1910 : cluster [DBG] osdmap e736: 8 total, 8 up, 8 in 2021-03-24T22:05:06.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:06 smithi142 conmon[20125]: audit 2021 2021-03-24T22:05:06.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:06 smithi142 conmon[20125]: -03-24T22:05:06.550507 2021-03-24T22:05:06.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:06 smithi142 conmon[20125]: +0000 mon.c (mon.2) 106 : audit 2021-03-24T22:05:06.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:06 smithi142 conmon[20125]: [DBG] from='client.? v1:172.21.15.125:0/335327748' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:05:06.894 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:06.953 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:06.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:06.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:05:07.060 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:07.118 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:07.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:07.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:05:07.224 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:07.279 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:07.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:07.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:05:07.386 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:07.418 INFO:tasks.rados.rados.0.smithi125.stdout:2826: done (1 left) 2021-03-24T22:05:07.419 INFO:tasks.rados.rados.0.smithi125.stdout:2827: snap_remove snap 195 2021-03-24T22:05:07.440 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:07.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:07.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:05:07.545 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:07.606 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:07.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:07.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:05:07.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:07 smithi125 conmon[24927]: cluster 2021-03-24T22:05:07.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:07 smithi125 conmon[24927]: 2021-03-24T22:05:06.622366+0000 osd.1 (osd.1) 79 : cluster [DBG] 4.2 scrub ok 2021-03-24T22:05:07.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:07 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:05:07.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:07 smithi125 conmon[24927]: 05:06.638218+0000 osd.6 (osd.6) 69 : cluster 2021-03-24T22:05:07.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:07 smithi125 conmon[24927]: [DBG] 5.2 scrub ok 2021-03-24T22:05:07.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:07 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T22:05:07.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:07 smithi125 conmon[24927]: 22:05:06.701581+0000 osd.2 (osd.2) 2021-03-24T22:05:07.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:07 smithi125 conmon[24927]: 116 : cluster [DBG] 6.17 scrub ok 2021-03-24T22:05:07.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:07 smithi125 conmon[24927]: cluster 2021-03-24T22:05:07.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:07 smithi125 conmon[24927]: 2021-03-24T22:05:07.414691 2021-03-24T22:05:07.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:07 smithi125 conmon[24927]: +0000 mon.a (mon.0) 1911 : cluster [DBG] osdmap e737: 8 total, 8 up, 8 in 2021-03-24T22:05:07.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:07 smithi125 conmon[24927]: 2021-03-24T22:05:07.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:07 smithi125 conmon[19936]: cluster 2021-03-24T22:05:06.622366+0000 osd.1 (osd.1) 79 : cluster 2021-03-24T22:05:07.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:07 smithi125 conmon[19936]: [DBG] 4.2 scrub ok 2021-03-24T22:05:07.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:07 smithi125 conmon[19936]: cluster 2021-03-24T22:05:06.638218+0000 osd.6 (osd.6) 69 : cluster [DBG] 5.2 scrub ok 2021-03-24T22:05:07.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:07 smithi125 conmon[19936]: cluster 2021-03-24T22:05:06.701581+0000 osd.2 (osd 2021-03-24T22:05:07.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:07 smithi125 conmon[19936]: .2) 116 : cluster [DBG] 6.17 scrub ok 2021-03-24T22:05:07.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:07 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:05:07.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:07 smithi125 conmon[19936]: -03-24T22:05:07.414691+0000 mon.a (mon.0) 1911 : cluster [DBG] 2021-03-24T22:05:07.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:07 smithi125 conmon[19936]: osdmap e737: 8 total, 8 up, 8 in 2021-03-24T22:05:07.713 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:07.767 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:07.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:07.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:05:07.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:07 smithi142 conmon[20125]: cluster 2021-03-24T22:05:06.622366+0000 osd.1 (osd.1) 2021-03-24T22:05:07.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:07 smithi142 conmon[20125]: 79 : cluster [DBG] 4.2 scrub ok 2021-03-24T22:05:07.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:07 smithi142 conmon[20125]: cluster 2021-03-24T22:05:06.638218+0000 osd.6 (osd.6) 69 : cluster [DBG] 5.2 scrub ok 2021-03-24T22:05:07.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:07 smithi142 conmon[20125]: cluster 2021-03-24T22:05: 2021-03-24T22:05:07.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:07 smithi142 conmon[20125]: 06.701581+0000 osd.2 (osd.2) 116 : cluster [DBG] 2021-03-24T22:05:07.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:07 smithi142 conmon[20125]: 6.17 scrub ok 2021-03-24T22:05:07.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:07 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:05:07.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:07 smithi142 conmon[20125]: -24T22:05:07.414691+0000 mon.a (mon.0 2021-03-24T22:05:07.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:07 smithi142 conmon[20125]: ) 1911 : cluster [DBG] osdmap e737: 8 total, 8 up, 8 in 2021-03-24T22:05:07.874 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:07.929 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:07.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:07.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:05:08.036 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:08.095 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:08.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:08.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:05:08.203 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:08.214 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd set noscrub 2021-03-24T22:05:08.260 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:08.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:08.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:05:08.365 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:08.417 INFO:tasks.rados.rados.0.smithi125.stdout:2827: done (1 left) 2021-03-24T22:05:08.418 INFO:tasks.rados.rados.0.smithi125.stdout:2828: read oid 13 snap -1 2021-03-24T22:05:08.419 INFO:tasks.rados.rados.0.smithi125.stdout:2828: expect (ObjNum 925 snap 308 seq_num 925) 2021-03-24T22:05:08.419 INFO:tasks.rados.rados.0.smithi125.stdout:2829: snap_create 2021-03-24T22:05:08.420 INFO:tasks.rados.rados.0.smithi125.stdout:2824: finishing write tid 1 to smithi12556470-48 2021-03-24T22:05:08.420 INFO:tasks.rados.rados.0.smithi125.stdout:2824: finishing write tid 2 to smithi12556470-48 2021-03-24T22:05:08.420 INFO:tasks.rados.rados.0.smithi125.stdout:2824: finishing write tid 3 to smithi12556470-48 2021-03-24T22:05:08.421 INFO:tasks.rados.rados.0.smithi125.stdout:2824: finishing write tid 4 to smithi12556470-48 2021-03-24T22:05:08.421 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 48 v 1017 (ObjNum 984 snap 326 seq_num 984) dirty exists 2021-03-24T22:05:08.422 INFO:tasks.rados.rados.0.smithi125.stdout:2824: left oid 48 (ObjNum 984 snap 326 seq_num 984) 2021-03-24T22:05:08.423 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:08.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:08.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:05:08.433 INFO:tasks.rados.rados.0.smithi125.stdout:2828: expect (ObjNum 925 snap 308 seq_num 925) 2021-03-24T22:05:08.528 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:08.589 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:08.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:08.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:05:08.696 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:08.754 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:08.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:08.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:05:08.862 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:08.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:08 smithi142 conmon[20125]: cluster 2021-03-24T22:05:06.973945+0000 mgr.y (mgr.14144) 643 : cluster 2021-03-24T22:05:08.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:08 smithi142 conmon[20125]: [DBG] pgmap v1267: 168 pgs: 1 remapped+peering, 19 peering, 12 active+undersized, 3 active+undersized+degraded, 133 active+clean; 661 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 22/1917 objects degraded (1.148%) 2021-03-24T22:05:08.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:08 smithi142 conmon[20125]: cluster 2021-03-24T22:05:07.629366+0000 2021-03-24T22:05:08.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:08 smithi142 conmon[20125]: osd.1 (osd.1) 80 : cluster [DBG] 5.12 scrub ok 2021-03-24T22:05:08.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:08 smithi142 conmon[20125]: 2021-03-24T22:05:08.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:08 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:05:08.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:08 smithi142 conmon[20125]: -03-24T22:05:08 2021-03-24T22:05:08.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:08 smithi142 conmon[20125]: .414040+0000 mon.a (mon.0 2021-03-24T22:05:08.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:08 smithi142 conmon[20125]: ) 1912 : cluster [DBG] osdmap e738: 8 total, 8 up, 8 in 2021-03-24T22:05:08.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:08 smithi142 conmon[20125]: 2021-03-24T22:05:08.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:08 smithi125 conmon[19936]: cluster 2021-03-24T22:05:06 2021-03-24T22:05:08.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:08 smithi125 conmon[19936]: .973945+0000 mgr.y (mgr.14144) 643 : cluster [DBG] pgmap v1267: 168 pgs: 1 remapped+peering, 19 peering, 12 active+undersized, 3 active+undersized+degraded, 133 active+clean; 661 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 22/1917 objects degraded (1.148%) 2021-03-24T22:05:08.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:08 smithi125 conmon[19936]: cluster 2021-03-24T22:05:07.629366+0000 osd.1 (osd.1) 80 : 2021-03-24T22:05:08.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:08 smithi125 conmon[19936]: cluster [DBG] 5.12 scrub ok 2021-03-24T22:05:08.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:08 smithi125 conmon[19936]: cluster 2021-03-24T22 2021-03-24T22:05:08.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:08 smithi125 conmon[19936]: :05:08.414040+0000 mon.a (mon.0) 1912 : cluster [DBG] 2021-03-24T22:05:08.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:08 smithi125 conmon[19936]: osdmap e738: 8 total, 8 up, 8 in 2021-03-24T22:05:08.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:08 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:05:08.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:08 smithi125 conmon[24927]: 05:06.973945+0000 mgr.y (mgr.14144) 643 : cluster [DBG] pgmap v1267: 168 pgs: 1 remapped+peering, 19 peering, 12 active+undersized, 3 active+undersized+degraded, 133 active+clean; 661 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 22/1917 objects degraded (1.148%) 2021-03-24T22:05:08.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:08 smithi125 conmon[24927]: cluster 2021-03-24T22 2021-03-24T22:05:08.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:08 smithi125 conmon[24927]: :05:07.629366+0000 osd.1 (osd.1) 2021-03-24T22:05:08.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:08 smithi125 conmon[24927]: 80 : cluster [DBG] 5.12 scrub ok 2021-03-24T22:05:08.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:08 smithi125 conmon[24927]: cluster 2021-03-24T22:05:08.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:08 smithi125 conmon[24927]: 2021-03-24 2021-03-24T22:05:08.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:08 smithi125 conmon[24927]: T22:05:08.414040+0000 mon.a (mon 2021-03-24T22:05:08.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:08 smithi125 conmon[24927]: .0) 1912 : cluster [DBG] osdmap e738: 8 total, 8 up, 8 in 2021-03-24T22:05:08.916 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:08.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:08.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:05:09.022 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:09.071 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:09.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:09.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:05:09.177 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:09.234 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:09.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:09.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:05:09.342 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:09.398 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:09.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:09.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:05:09.426 INFO:tasks.rados.rados.0.smithi125.stdout:2824: done (2 left) 2021-03-24T22:05:09.426 INFO:tasks.rados.rados.0.smithi125.stdout:2828: done (1 left) 2021-03-24T22:05:09.426 INFO:tasks.rados.rados.0.smithi125.stdout:2829: done (0 left) 2021-03-24T22:05:09.427 INFO:tasks.rados.rados.0.smithi125.stdout:2830: read oid 39 snap 326 2021-03-24T22:05:09.427 INFO:tasks.rados.rados.0.smithi125.stdout:2830: expect deleted 2021-03-24T22:05:09.427 INFO:tasks.rados.rados.0.smithi125.stdout:2831: read oid 47 snap -1 2021-03-24T22:05:09.427 INFO:tasks.rados.rados.0.smithi125.stdout:2831: expect (ObjNum 929 snap 311 seq_num 929) 2021-03-24T22:05:09.427 INFO:tasks.rados.rados.0.smithi125.stdout:2832: copy_from oid 50 from oid 8 current snap is 327 2021-03-24T22:05:09.428 INFO:tasks.rados.rados.0.smithi125.stdout:2833: write oid 32 current snap is 327 2021-03-24T22:05:09.428 INFO:tasks.rados.rados.0.smithi125.stdout:2833: seq_num 985 ranges {554895=729802,1773234=778333} 2021-03-24T22:05:09.437 INFO:tasks.rados.rados.0.smithi125.stdout:2833: writing smithi12556470-32 from 554895 to 1284697 tid 1 2021-03-24T22:05:09.443 INFO:tasks.rados.rados.0.smithi125.stdout:2833: writing smithi12556470-32 from 1773234 to 2551567 tid 2 2021-03-24T22:05:09.444 INFO:tasks.rados.rados.0.smithi125.stdout:2834: read oid 38 snap -1 2021-03-24T22:05:09.444 INFO:tasks.rados.rados.0.smithi125.stdout:2834: expect (ObjNum 973 snap 318 seq_num 973) 2021-03-24T22:05:09.444 INFO:tasks.rados.rados.0.smithi125.stdout:2835: delete oid 17 current snap is 327 2021-03-24T22:05:09.445 INFO:tasks.rados.rados.0.smithi125.stdout:2831: expect (ObjNum 929 snap 311 seq_num 929) 2021-03-24T22:05:09.506 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:09.558 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:09.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:09.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:05:09.614 INFO:tasks.rados.rados.0.smithi125.stdout:2834: expect (ObjNum 973 snap 318 seq_num 973) 2021-03-24T22:05:09.664 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:09.727 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:09.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:09.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:05:09.748 INFO:tasks.rados.rados.0.smithi125.stdout:2833: finishing write tid 1 to smithi12556470-32 2021-03-24T22:05:09.749 INFO:tasks.rados.rados.0.smithi125.stdout:2833: finishing write tid 2 to smithi12556470-32 2021-03-24T22:05:09.749 INFO:tasks.rados.rados.0.smithi125.stdout:2833: finishing write tid 3 to smithi12556470-32 2021-03-24T22:05:09.749 INFO:tasks.rados.rados.0.smithi125.stdout:2833: finishing write tid 4 to smithi12556470-32 2021-03-24T22:05:09.749 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 32 v 1744 (ObjNum 985 snap 327 seq_num 985) dirty exists 2021-03-24T22:05:09.750 INFO:tasks.rados.rados.0.smithi125.stdout:2833: left oid 32 (ObjNum 985 snap 327 seq_num 985) 2021-03-24T22:05:09.750 INFO:tasks.rados.rados.0.smithi125.stdout:2832: finishing copy_from to smithi12556470-50 2021-03-24T22:05:09.750 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 50 v 559 (ObjNum 941 snap 314 seq_num 941) dirty exists 2021-03-24T22:05:09.750 INFO:tasks.rados.rados.0.smithi125.stdout:2832: finishing copy_from racing read to smithi12556470-50 2021-03-24T22:05:09.750 INFO:tasks.rados.rados.0.smithi125.stdout:2830: done (5 left) 2021-03-24T22:05:09.751 INFO:tasks.rados.rados.0.smithi125.stdout:2831: done (4 left) 2021-03-24T22:05:09.751 INFO:tasks.rados.rados.0.smithi125.stdout:2832: done (3 left) 2021-03-24T22:05:09.751 INFO:tasks.rados.rados.0.smithi125.stdout:2833: done (2 left) 2021-03-24T22:05:09.751 INFO:tasks.rados.rados.0.smithi125.stdout:2834: done (1 left) 2021-03-24T22:05:09.752 INFO:tasks.rados.rados.0.smithi125.stdout:2835: done (0 left) 2021-03-24T22:05:09.752 INFO:tasks.rados.rados.0.smithi125.stdout:2836: copy_from oid 31 from oid 6 current snap is 327 2021-03-24T22:05:09.752 INFO:tasks.rados.rados.0.smithi125.stdout:2837: snap_create 2021-03-24T22:05:09.766 INFO:tasks.rados.rados.0.smithi125.stdout:2836: finishing copy_from to smithi12556470-31 2021-03-24T22:05:09.766 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 31 v 659 (ObjNum 906 snap 300 seq_num 906) dirty exists 2021-03-24T22:05:09.766 INFO:tasks.rados.rados.0.smithi125.stdout:2836: finishing copy_from racing read to smithi12556470-31 2021-03-24T22:05:09.774 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:05:09.775 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[19936]: T22:05:08.570222+0000 osd.6 (osd.6) 70 : cluster [DBG] 6.1e scrub ok 2021-03-24T22:05:09.775 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[19936]: cluster 2021-03-24T22:05:09.775 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[19936]: 2021-03-24T 2021-03-24T22:05:09.776 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[19936]: 22:05: 2021-03-24T22:05:09.776 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[19936]: 08.670231+0000 osd.2 2021-03-24T22:05:09.776 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[19936]: (osd.2) 2021-03-24T22:05:09.777 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[19936]: 117 : cluster [DBG] 5.b deep-scrub ok 2021-03-24T22:05:09.777 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[19936]: 2021-03-24T22:05:09.777 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[19936]: cluster 2021-03-24T22:05:09.777 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[19936]: 2021-03-24 2021-03-24T22:05:09.778 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[19936]: T22:05 2021-03-24T22:05:09.778 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[19936]: :09.177537 2021-03-24T22:05:09.778 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[19936]: +0000 osd.5 ( 2021-03-24T22:05:09.778 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[19936]: osd.5) 4 2021-03-24T22:05:09.778 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[19936]: : cluster [DBG] 2021-03-24T22:05:09.779 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[19936]: 4.15 scrub ok 2021-03-24T22:05:09.779 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[19936]: cluster 2021-03-24T22:05:09.779 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[19936]: 2021-03-24 2021-03-24T22:05:09.779 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[19936]: T22:05 2021-03-24T22:05:09.779 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[19936]: :09.243581 2021-03-24T22:05:09.780 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[19936]: +0000 osd.3 ( 2021-03-24T22:05:09.780 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[19936]: osd.3) 2021-03-24T22:05:09.780 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[19936]: 148 : cluster 2021-03-24T22:05:09.780 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[19936]: [DBG] 6.13 scrub ok 2021-03-24T22:05:09.781 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[19936]: cluster 2021-03-24T22:05:09.781 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[19936]: 2021-03- 2021-03-24T22:05:09.781 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[19936]: 24T22: 2021-03-24T22:05:09.781 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[19936]: 05:09. 2021-03-24T22:05:09.781 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[19936]: 312259+0000 osd.0 2021-03-24T22:05:09.782 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[19936]: (osd.0 2021-03-24T22:05:09.782 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[19936]: ) 144 : cluster 2021-03-24T22:05:09.782 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[19936]: [DBG] 5.e scrub ok 2021-03-24T22:05:09.782 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[19936]: cluster 2021-03-24T22:05:09.782 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[19936]: 2021-03-24T 2021-03-24T22:05:09.783 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[19936]: 22:05:09.421850 2021-03-24T22:05:09.783 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[19936]: +0000 mon.a (mon. 2021-03-24T22:05:09.783 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[19936]: 0) 1913 : cluster [DBG] 2021-03-24T22:05:09.783 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[19936]: osdmap e739: 8 total, 8 up, 8 in 2021-03-24T22:05:09.784 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:05:09.784 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[24927]: 24T22:05:08.570222+0000 osd.6 (osd.6) 70 : cluster 2021-03-24T22:05:09.784 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[24927]: [DBG] 6.1e scrub ok 2021-03-24T22:05:09.784 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[24927]: cluster 2021-03-24T22:05:09.785 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[24927]: 2021-03-24 2021-03-24T22:05:09.785 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[24927]: T22:05: 2021-03-24T22:05:09.785 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[24927]: 08.670231+0000 2021-03-24T22:05:09.785 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[24927]: osd.2 (osd.2) 2021-03-24T22:05:09.785 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[24927]: 117 : cluster [DBG] 2021-03-24T22:05:09.786 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[24927]: 5.b deep-scrub ok 2021-03-24T22:05:09.786 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[24927]: cluster 2021-03-24T22:05:09.786 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[24927]: 2021-03- 2021-03-24T22:05:09.786 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[24927]: 24T22:05 2021-03-24T22:05:09.787 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[24927]: :09.177537+0000 2021-03-24T22:05:09.787 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[24927]: osd.5 (osd.5) 2021-03-24T22:05:09.787 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[24927]: 4 : cluster [DBG] 2021-03-24T22:05:09.787 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[24927]: 4.15 scrub ok 2021-03-24T22:05:09.787 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[24927]: cluster 2021-03-24T22:05:09.788 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[24927]: 2021-03- 2021-03-24T22:05:09.788 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[24927]: 24T22:05 2021-03-24T22:05:09.788 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[24927]: :09.243581+0000 2021-03-24T22:05:09.788 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[24927]: osd.3 (osd.3) 2021-03-24T22:05:09.788 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[24927]: 148 : cluster [DBG] 2021-03-24T22:05:09.789 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[24927]: 6.13 scrub ok 2021-03-24T22:05:09.789 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[24927]: cluster 2021-03-24T22:05:09.789 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[24927]: 2021-03- 2021-03-24T22:05:09.789 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[24927]: 24T22:05 2021-03-24T22:05:09.789 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[24927]: :09.312259+0000 2021-03-24T22:05:09.790 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[24927]: osd.0 (osd.0) 2021-03-24T22:05:09.790 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[24927]: 144 : cluster [DBG] 2021-03-24T22:05:09.790 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[24927]: 5.e scrub ok 2021-03-24T22:05:09.790 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[24927]: 2021-03-24T22:05:09.791 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[24927]: cluster 2021-03-24T22:05:09.791 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[24927]: 2021-03-24 2021-03-24T22:05:09.791 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[24927]: T22:05:09. 2021-03-24T22:05:09.791 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[24927]: 421850+0000 mon.a (mon.0 2021-03-24T22:05:09.791 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[24927]: ) 1913 : cluster [DBG] 2021-03-24T22:05:09.792 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:09 smithi125 conmon[24927]: osdmap e739: 8 total, 8 up, 8 in 2021-03-24T22:05:09.834 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:09.848 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:09 smithi142 conmon[20125]: cluster 2021-03-24T22:05: 2021-03-24T22:05:09.849 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:09 smithi142 conmon[20125]: 08.570222+0000 osd.6 (osd.6) 70 : cluster [DBG] 6.1e scrub ok 2021-03-24T22:05:09.849 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:09 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:05:09.849 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:09 smithi142 conmon[20125]: 24T22:05:08.670231+0000 osd.2 (osd.2) 117 : cluster 2021-03-24T22:05:09.849 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:09 smithi142 conmon[20125]: [DBG] 5.b deep-scrub ok 2021-03-24T22:05:09.850 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:09 smithi142 conmon[20125]: cluster 2021-03-24T22:05 2021-03-24T22:05:09.850 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:09 smithi142 conmon[20125]: :09.177537+0000 osd.5 (osd.5) 4 2021-03-24T22:05:09.850 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:09 smithi142 conmon[20125]: : cluster [DBG] 4.15 scrub ok 2021-03-24T22:05:09.850 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:09 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:05:09.851 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:09 smithi142 conmon[20125]: T22:05:09.243581+0000 osd.3 ( 2021-03-24T22:05:09.851 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:09 smithi142 conmon[20125]: osd.3) 148 : cluster [DBG] 6.13 scrub ok 2021-03-24T22:05:09.851 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:09 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:05:09.851 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:09 smithi142 conmon[20125]: -03-24T22:05:09.312259 2021-03-24T22:05:09.852 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:09 smithi142 conmon[20125]: +0000 osd.0 (osd.0) 144 : cluster 2021-03-24T22:05:09.852 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:09 smithi142 conmon[20125]: [DBG] 5.e scrub ok 2021-03-24T22:05:09.852 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:09 smithi142 conmon[20125]: cluster 2021-03-24T22: 2021-03-24T22:05:09.852 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:09 smithi142 conmon[20125]: 05:09.421850+0000 mon.a (mon.0 2021-03-24T22:05:09.852 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:09 smithi142 conmon[20125]: ) 1913 : cluster [DBG] osdmap e739: 8 total, 8 up, 8 in 2021-03-24T22:05:09.894 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:09.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:09.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:05:10.002 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:10.018 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:05:10.063 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:10.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:10.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:05:10.171 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:10.231 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:10.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:10.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:05:10.337 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:10.386 INFO:tasks.rados.rados.0.smithi125.stdout:2836: done (1 left) 2021-03-24T22:05:10.387 INFO:tasks.rados.rados.0.smithi125.stdout:2837: done (0 left) 2021-03-24T22:05:10.387 INFO:tasks.rados.rados.0.smithi125.stdout:2838: read oid 35 snap -1 2021-03-24T22:05:10.387 INFO:tasks.rados.rados.0.smithi125.stdout:2838: expect (ObjNum 972 snap 318 seq_num 972) 2021-03-24T22:05:10.388 INFO:tasks.rados.rados.0.smithi125.stdout:2839: delete oid 44 current snap is 328 2021-03-24T22:05:10.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:10.411 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:10.411 INFO:tasks.rados.rados.0.smithi125.stdout:2838: expect (ObjNum 972 snap 318 seq_num 972) 2021-03-24T22:05:10.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:05:10.513 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:10.516 INFO:tasks.rados.rados.0.smithi125.stdout:2838: done (1 left) 2021-03-24T22:05:10.516 INFO:tasks.rados.rados.0.smithi125.stdout:2839: done (0 left) 2021-03-24T22:05:10.516 INFO:tasks.rados.rados.0.smithi125.stdout:2840: write oid 13 current snap is 328 2021-03-24T22:05:10.517 INFO:tasks.rados.rados.0.smithi125.stdout:2840: seq_num 988 ranges {605316=555549,1832327=642013} 2021-03-24T22:05:10.526 INFO:tasks.rados.rados.0.smithi125.stdout:2840: writing smithi12556470-13 from 605316 to 1160865 tid 1 2021-03-24T22:05:10.535 INFO:tasks.rados.rados.0.smithi125.stdout:2840: writing smithi12556470-13 from 1832327 to 2474340 tid 2 2021-03-24T22:05:10.536 INFO:tasks.rados.rados.0.smithi125.stdout:2841: snap_remove snap 291 2021-03-24T22:05:10.559 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:10.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:10.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:05:10.668 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:10.712 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:10.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:10.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:05:10.818 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:10.864 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:10 smithi125 conmon[24927]: cluster 2021-03-24T22:05 2021-03-24T22:05:10.864 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:10 smithi125 conmon[24927]: :08.974754+0000 mgr.y (mgr.14144) 644 : cluster [DBG] pgmap v1270: 168 pgs: 15 active+clean+wait, 19 peering, 134 active+clean; 661 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2021-03-24T22:05:10.864 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:10 smithi125 conmon[24927]: cluster 2021-03-24T22:05:10.865 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:10 smithi125 conmon[24927]: 2021-03-24T22:05 2021-03-24T22:05:10.865 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:10 smithi125 conmon[24927]: :09.628246+0000 mon.a (mon 2021-03-24T22:05:10.865 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:10 smithi125 conmon[24927]: .0) 1914 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 22/1917 objects degraded (1.148%), 3 pgs degraded, 15 pgs undersized) 2021-03-24T22:05:10.865 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:10 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:05:10.865 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:10 smithi125 conmon[24927]: -03-24T22:05:09 2021-03-24T22:05:10.866 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:10 smithi125 conmon[24927]: .628317+0000 mon.a (mon. 2021-03-24T22:05:10.866 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:10 smithi125 conmon[24927]: 0) 1915 : cluster [INF] Cluster is now healthy 2021-03-24T22:05:10.866 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:10 smithi125 conmon[24927]: audit 2021-03-24T22:05:10.866 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:10 smithi125 conmon[24927]: 2021-03-24T22:05 2021-03-24T22:05:10.867 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:10 smithi125 conmon[24927]: :09.849542+0000 2021-03-24T22:05:10.867 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:10 smithi125 conmon[24927]: mon.a (mon.0) 1916 : audit 2021-03-24T22:05:10.867 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:10 smithi125 conmon[24927]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:05:10.867 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:10 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:05:10.867 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:10 smithi125 conmon[24927]: 03-24T22:05:10 2021-03-24T22:05:10.868 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:10 smithi125 conmon[24927]: .383070+0000 mon.a (mon.0) 1917 2021-03-24T22:05:10.868 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:10 smithi125 conmon[24927]: : cluster [DBG] osdmap e740: 8 total, 8 up, 8 in 2021-03-24T22:05:10.868 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:10 smithi125 conmon[19936]: cluster 2021-03-24T22:05 2021-03-24T22:05:10.869 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:10 smithi125 conmon[19936]: :08.974754+0000 mgr.y (mgr.14144) 644 : cluster [DBG] pgmap v1270: 168 pgs: 15 active+clean+wait, 19 peering, 134 active+clean; 661 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2021-03-24T22:05:10.869 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:10 smithi125 conmon[19936]: cluster 2021-03-24T22:05:10.869 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:10 smithi125 conmon[19936]: 2021-03-24T22:05:09. 2021-03-24T22:05:10.869 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:10 smithi125 conmon[19936]: 628246+0000 mon.a (mon.0) 1914 : 2021-03-24T22:05:10.870 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:10 smithi125 conmon[19936]: cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 22/1917 objects degraded (1.148%), 3 pgs degraded, 15 pgs undersized) 2021-03-24T22:05:10.870 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:10 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:05:10.870 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:10 smithi125 conmon[19936]: 03-24T22:05:09.628317 2021-03-24T22:05:10.870 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:10 smithi125 conmon[19936]: +0000 mon.a (mon.0) 2021-03-24T22:05:10.870 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:10 smithi125 conmon[19936]: 1915 : cluster [INF] Cluster is now healthy 2021-03-24T22:05:10.871 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:10 smithi125 conmon[19936]: 2021-03-24T22:05:10.871 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:10 smithi125 conmon[19936]: audit 2021- 2021-03-24T22:05:10.871 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:10 smithi125 conmon[19936]: 03-24T22:05:09. 2021-03-24T22:05:10.871 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:10 smithi125 conmon[19936]: 849542+0000 mon.a (mon. 2021-03-24T22:05:10.871 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:10 smithi125 conmon[19936]: 0) 1916 : audit [INF] 2021-03-24T22:05:10.872 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:10 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:05:10.872 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:10 smithi125 conmon[19936]: cluster 2021-03-24T22:05:10.872 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:10 smithi125 conmon[19936]: 2021-03-24T22:05:10 2021-03-24T22:05:10.872 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:10 smithi125 conmon[19936]: .383070+0000 mon.a (mon.0 2021-03-24T22:05:10.873 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:10 smithi125 conmon[19936]: ) 1917 : cluster [DBG] osdmap e740: 8 total, 8 up, 8 in 2021-03-24T22:05:10.873 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:10 smithi125 conmon[19936]: 2021-03-24T22:05:10.873 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:10.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:10.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:05:10.979 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:11.036 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:11.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:11.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:05:11.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:10 smithi142 conmon[20125]: cluster 2021-03-24T22:05: 2021-03-24T22:05:11.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:10 smithi142 conmon[20125]: 08.974754+0000 mgr.y (mgr.14144) 644 : cluster [DBG] pgmap v1270: 168 pgs: 15 active+clean+wait, 19 peering, 134 active+clean; 661 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2021-03-24T22:05:11.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:10 smithi142 conmon[20125]: cluster 2021-03-24T22: 2021-03-24T22:05:11.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:10 smithi142 conmon[20125]: 05:09.628246+0000 mon.a (mon.0) 1914 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 22/1917 objects degraded (1.148%), 3 pgs degraded, 15 pgs undersized) 2021-03-24T22:05:11.119 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:10 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:05:11.119 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:10 smithi142 conmon[20125]: 22:05:09.628317+0000 mon.a (mon.0) 1915 : 2021-03-24T22:05:11.119 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:10 smithi142 conmon[20125]: cluster [INF] Cluster is now healthy 2021-03-24T22:05:11.119 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:10 smithi142 conmon[20125]: audit 2021-03-24T22: 2021-03-24T22:05:11.120 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:10 smithi142 conmon[20125]: 05:09.849542+0000 mon.a (mon.0 2021-03-24T22:05:11.120 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:10 smithi142 conmon[20125]: ) 1916 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:05:11.120 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:10 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:05:11.121 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:10 smithi142 conmon[20125]: T22:05:10.383070+0000 mon.a ( 2021-03-24T22:05:11.121 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:10 smithi142 conmon[20125]: mon.0) 1917 : cluster [DBG] osdmap e740: 8 total, 8 up, 8 in 2021-03-24T22:05:11.142 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:11.200 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:11.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:11.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:05:11.307 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:11.363 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:11.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:11.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:05:11.406 INFO:tasks.rados.rados.0.smithi125.stdout:2841: done (1 left) 2021-03-24T22:05:11.406 INFO:tasks.rados.rados.0.smithi125.stdout:2842: read oid 7 snap -1 2021-03-24T22:05:11.407 INFO:tasks.rados.rados.0.smithi125.stdout:2842: expect (ObjNum 727 snap 233 seq_num 727) 2021-03-24T22:05:11.407 INFO:tasks.rados.rados.0.smithi125.stdout:2843: rollback oid 33 current snap is 328 2021-03-24T22:05:11.407 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 33 to 293 2021-03-24T22:05:11.407 INFO:tasks.rados.rados.0.smithi125.stdout:2844: copy_from oid 11 from oid 12 current snap is 328 2021-03-24T22:05:11.407 INFO:tasks.rados.rados.0.smithi125.stdout:2845: copy_from oid 26 from oid 37 current snap is 328 2021-03-24T22:05:11.408 INFO:tasks.rados.rados.0.smithi125.stdout:2846: snap_remove snap 306 2021-03-24T22:05:11.418 INFO:teuthology.orchestra.run.smithi125.stderr:reweighted osd.0 to 1 (10000) 2021-03-24T22:05:11.469 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:11.524 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:11.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:11.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:05:11.632 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:11.695 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:11.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:11.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:05:11.794 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-24T22:05:11.801 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:11.808 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:11 smithi125 conmon[24927]: cluster 2021-03-24T22:05:11.808 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:11 smithi125 conmon[24927]: 2021-03-24T22:05: 2021-03-24T22:05:11.808 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:11 smithi125 conmon[24927]: 10.501484+0000 osd.1 (osd.1) 81 : cluster [DBG] 2.2 scrub ok 2021-03-24T22:05:11.809 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:11 smithi125 conmon[24927]: audit 2021-03-24T22:05:11.809 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:11 smithi125 conmon[24927]: 2021-03-24T22:05:10.691742+0000 2021-03-24T22:05:11.809 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:11 smithi125 conmon[24927]: mon.c (mon.2) 107 : audit [INF] from='client.? v1:172.21.15.125:0/1612394588' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2021-03-24T22:05:11.809 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:11 smithi125 conmon[24927]: 2021-03-24T22:05:11.810 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:11 smithi125 conmon[24927]: audit 2021-03-24T22:05 2021-03-24T22:05:11.810 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:11 smithi125 conmon[24927]: :10.692247+0000 mon.a (mon.0) 1918 2021-03-24T22:05:11.810 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:11 smithi125 conmon[24927]: : audit [INF] from='client.35741 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2021-03-24T22:05:11.810 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:11 smithi125 conmon[24927]: cluster 2021-03-24T22:05:11.810 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:11 smithi125 conmon[24927]: 2021-03-24T22:05:10.975678+0000 2021-03-24T22:05:11.811 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:11 smithi125 conmon[24927]: mgr.y (mgr.14144) 645 : cluster [DBG] pgmap v1273: 168 pgs: 15 active+clean+wait, 153 active+clean; 558 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 315 KiB/s wr, 2 op/s; 13 MiB/s, 12 objects/s recovering 2021-03-24T22:05:11.811 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:11 smithi125 conmon[24927]: 2021-03-24T22:05:11.811 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:11 smithi125 conmon[24927]: audit 2021-03-24T22:05:11.811 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:11 smithi125 conmon[24927]: 2021-03-24T22:05:11.400209 2021-03-24T22:05:11.812 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:11 smithi125 conmon[24927]: +0000 mon.a (mon.0) 1919 : audit 2021-03-24T22:05:11.812 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:11 smithi125 conmon[24927]: [INF] from='client.35741 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2021-03-24T22:05:11.812 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:11 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T22:05:11.812 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:11 smithi125 conmon[24927]: 22:05:11.400533+0000 mon.a (mon.0 2021-03-24T22:05:11.812 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:11 smithi125 conmon[24927]: ) 1920 : cluster [DBG] osdmap e741: 8 total, 8 up, 8 in 2021-03-24T22:05:11.813 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:11 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:05:11.813 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:11 smithi125 conmon[19936]: T22:05:10.501484+0000 osd.1 (osd.1) 81 : cluster [DBG] 2.2 scrub ok 2021-03-24T22:05:11.813 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:11 smithi125 conmon[19936]: audit 2021- 2021-03-24T22:05:11.814 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:11 smithi125 conmon[19936]: 03-24T22:05:10.691742 2021-03-24T22:05:11.814 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:11 smithi125 conmon[19936]: +0000 mon.c (mon.2) 107 2021-03-24T22:05:11.814 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:11 smithi125 conmon[19936]: : audit [INF] from='client.? v1:172.21.15.125:0/1612394588' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2021-03-24T22:05:11.814 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:11 smithi125 conmon[19936]: audit 2021 2021-03-24T22:05:11.815 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:11 smithi125 conmon[19936]: -03-24T22:05:10. 2021-03-24T22:05:11.815 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:11 smithi125 conmon[19936]: 692247+0000 mon.a (mon.0) 1918 : audit 2021-03-24T22:05:11.815 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:11 smithi125 conmon[19936]: [INF] from='client.35741 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2021-03-24T22:05:11.815 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:11 smithi125 conmon[19936]: cluster 2021-03-24T22:05:10.975678+0000 mgr.y (mgr. 2021-03-24T22:05:11.815 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:11 smithi125 conmon[19936]: 14144) 645 : cluster [DBG] pgmap v1273: 168 pgs: 15 active+clean+wait, 153 active+clean; 558 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 315 KiB/s wr, 2 op/s; 13 MiB/s, 12 objects/s recovering 2021-03-24T22:05:11.816 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:11 smithi125 conmon[19936]: audit 2021-03-24T22:05:11.400209+0000 mon.a 2021-03-24T22:05:11.816 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:11 smithi125 conmon[19936]: (mon.0) 1919 : audit [INF] from='client.35741 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2021-03-24T22:05:11.816 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:11 smithi125 conmon[19936]: cluster 2021-03-24T22:05:11.400533+0000 mon.a (mon 2021-03-24T22:05:11.817 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:11 smithi125 conmon[19936]: .0) 1920 : cluster [DBG] osdmap e741: 8 total, 8 up, 8 in 2021-03-24T22:05:11.859 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:11.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:11.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:05:11.965 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:11.980 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:11 smithi142 conmon[20125]: cluster 2021-03-24T22:05: 2021-03-24T22:05:11.980 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:11 smithi142 conmon[20125]: 10.501484+0000 osd.1 (osd.1) 81 : cluster [DBG] 2.2 scrub ok 2021-03-24T22:05:11.981 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:11 smithi142 conmon[20125]: audit 2021-03-24T 2021-03-24T22:05:11.981 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:11 smithi142 conmon[20125]: 22:05:10.691742+0000 mon.c (mon.2) 107 : audit [INF] 2021-03-24T22:05:11.981 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:11 smithi142 conmon[20125]: from='client.? v1:172.21.15.125:0/1612394588' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2021-03-24T22:05:11.981 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:11 smithi142 conmon[20125]: audit 2021 2021-03-24T22:05:11.982 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:11 smithi142 conmon[20125]: -03-24T22:05:10.692247+0000 mon.a (mon 2021-03-24T22:05:11.982 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:11 smithi142 conmon[20125]: .0) 1918 : audit [INF] from='client.35741 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2021-03-24T22:05:11.982 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:11 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:05:11.984 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:11 smithi142 conmon[20125]: 22:05:10.975678+0000 mgr.y (mgr. 2021-03-24T22:05:11.985 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:11 smithi142 conmon[20125]: 14144) 645 : cluster [DBG] pgmap v1273: 168 pgs: 15 active+clean+wait, 153 active+clean; 558 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 315 KiB/s wr, 2 op/s; 13 MiB/s, 12 objects/s recovering 2021-03-24T22:05:11.985 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:11 smithi142 conmon[20125]: audit 2021-03-24T22 2021-03-24T22:05:11.986 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:11 smithi142 conmon[20125]: :05:11.400209+0000 mon.a (mon. 2021-03-24T22:05:11.986 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:11 smithi142 conmon[20125]: 0) 1919 : audit [INF] from='client.35741 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2021-03-24T22:05:11.987 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:11 smithi142 conmon[20125]: cluster 2021-03-24T22 2021-03-24T22:05:11.987 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:11 smithi142 conmon[20125]: :05:11.400533+0000 mon.a (mon. 2021-03-24T22:05:11.988 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:11 smithi142 conmon[20125]: 0) 1920 : cluster [DBG] osdmap e741: 8 total, 8 up, 8 in 2021-03-24T22:05:12.021 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:12.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:12.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:05:12.128 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:12.186 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:12.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:12.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:05:12.292 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:12.351 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:12.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:12.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:05:12.407 INFO:tasks.rados.rados.0.smithi125.stdout:2846: done (5 left) 2021-03-24T22:05:12.408 INFO:tasks.rados.rados.0.smithi125.stdout:2847: copy_from oid 21 from oid 24 current snap is 328 2021-03-24T22:05:12.408 INFO:tasks.rados.rados.0.smithi125.stdout:2840: finishing write tid 1 to smithi12556470-13 2021-03-24T22:05:12.408 INFO:tasks.rados.rados.0.smithi125.stdout:2840: finishing write tid 2 to smithi12556470-13 2021-03-24T22:05:12.408 INFO:tasks.rados.rados.0.smithi125.stdout:2840: finishing write tid 3 to smithi12556470-13 2021-03-24T22:05:12.409 INFO:tasks.rados.rados.0.smithi125.stdout:2840: finishing write tid 4 to smithi12556470-13 2021-03-24T22:05:12.409 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 13 v 803 (ObjNum 988 snap 328 seq_num 988) dirty exists 2021-03-24T22:05:12.409 INFO:tasks.rados.rados.0.smithi125.stdout:2840: left oid 13 (ObjNum 988 snap 328 seq_num 988) 2021-03-24T22:05:12.409 INFO:tasks.rados.rados.0.smithi125.stdout:2844: finishing copy_from to smithi12556470-11 2021-03-24T22:05:12.410 INFO:tasks.rados.rados.0.smithi125.stdout:2844: got expected ENOENT (src dne) 2021-03-24T22:05:12.410 INFO:tasks.rados.rados.0.smithi125.stdout:2842: expect (ObjNum 727 snap 233 seq_num 727) 2021-03-24T22:05:12.459 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:12.509 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:12.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:12.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:05:12.600 INFO:tasks.rados.rados.0.smithi125.stdout:2844: finishing copy_from racing read to smithi12556470-11 2021-03-24T22:05:12.601 INFO:tasks.rados.rados.0.smithi125.stdout:2843: finishing rollback tid 0 to smithi12556470-33 2021-03-24T22:05:12.601 INFO:tasks.rados.rados.0.smithi125.stdout:2845: finishing copy_from to smithi12556470-26 2021-03-24T22:05:12.601 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 26 v 740 (ObjNum 978 snap 319 seq_num 978) dirty exists 2021-03-24T22:05:12.602 INFO:tasks.rados.rados.0.smithi125.stdout:2845: finishing copy_from racing read to smithi12556470-26 2021-03-24T22:05:12.602 INFO:tasks.rados.rados.0.smithi125.stdout:2843: finishing rollback tid 1 to smithi12556470-33 2021-03-24T22:05:12.603 INFO:tasks.rados.rados.0.smithi125.stdout:2843: finishing rollback tid 2 to smithi12556470-33 2021-03-24T22:05:12.603 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 33 v 1440 (ObjNum 764 snap 249 seq_num 764) dirty exists 2021-03-24T22:05:12.603 INFO:tasks.rados.rados.0.smithi125.stdout:2840: done (5 left) 2021-03-24T22:05:12.604 INFO:tasks.rados.rados.0.smithi125.stdout:2842: done (4 left) 2021-03-24T22:05:12.604 INFO:tasks.rados.rados.0.smithi125.stdout:2843: done (3 left) 2021-03-24T22:05:12.605 INFO:tasks.rados.rados.0.smithi125.stdout:2844: done (2 left) 2021-03-24T22:05:12.606 INFO:tasks.rados.rados.0.smithi125.stdout:2845: done (1 left) 2021-03-24T22:05:12.606 INFO:tasks.rados.rados.0.smithi125.stdout:2848: read oid 22 snap -1 2021-03-24T22:05:12.607 INFO:tasks.rados.rados.0.smithi125.stdout:2848: expect deleted 2021-03-24T22:05:12.607 INFO:tasks.rados.rados.0.smithi125.stdout:2849: read oid 10 snap 304 2021-03-24T22:05:12.607 INFO:tasks.rados.rados.0.smithi125.stdout:2849: expect (ObjNum 907 snap 300 seq_num 907) 2021-03-24T22:05:12.608 INFO:tasks.rados.rados.0.smithi125.stdout:2847: finishing copy_from to smithi12556470-21 2021-03-24T22:05:12.608 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 21 v 1434 (ObjNum 851 snap 278 seq_num 851) dirty exists 2021-03-24T22:05:12.609 INFO:tasks.rados.rados.0.smithi125.stdout:2847: finishing copy_from racing read to smithi12556470-21 2021-03-24T22:05:12.609 INFO:tasks.rados.rados.0.smithi125.stdout:2847: done (2 left) 2021-03-24T22:05:12.610 INFO:tasks.rados.rados.0.smithi125.stdout:2850: rollback oid 40 current snap is 328 2021-03-24T22:05:12.610 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 40 to 311 2021-03-24T22:05:12.610 INFO:tasks.rados.rados.0.smithi125.stdout:2851: delete oid 5 current snap is 328 2021-03-24T22:05:12.612 INFO:tasks.rados.rados.0.smithi125.stdout:2848: done (3 left) 2021-03-24T22:05:12.612 INFO:tasks.rados.rados.0.smithi125.stdout:2851: done (2 left) 2021-03-24T22:05:12.613 INFO:tasks.rados.rados.0.smithi125.stdout:2852: rollback oid 24 current snap is 328 2021-03-24T22:05:12.613 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 24 to 303 2021-03-24T22:05:12.614 INFO:tasks.rados.rados.0.smithi125.stdout:2853: snap_remove snap 320 2021-03-24T22:05:12.616 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:12.653 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-24T22:05:12.653 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T22:05:12.669 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:12.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:12.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:05:12.780 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:12.833 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:12.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:12.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:05:12.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:12 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:05:12.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:12 smithi125 conmon[19936]: T22:05:11.459208+0000 osd.1 (osd.1) 82 : cluster [DBG] 5.19 scrub ok 2021-03-24T22:05:12.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:12 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:05:12.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:12 smithi125 conmon[19936]: -03-24T22:05:12. 2021-03-24T22:05:12.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:12 smithi125 conmon[19936]: 146052+0000 osd.5 (osd.5) 5 2021-03-24T22:05:12.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:12 smithi125 conmon[19936]: : cluster [DBG] 5.7 scrub ok 2021-03-24T22:05:12.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:12 smithi125 conmon[19936]: cluster 2021-03-24T22:05:12.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:12 smithi125 conmon[19936]: 2021-03-24T22:05: 2021-03-24T22:05:12.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:12 smithi125 conmon[19936]: 12.403559+0000 mon.a (mon 2021-03-24T22:05:12.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:12 smithi125 conmon[19936]: .0) 1921 : cluster [DBG] osdmap e742: 8 total, 8 up, 8 in 2021-03-24T22:05:12.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:12 smithi125 conmon[24927]: cluster 2021-03-24T22 2021-03-24T22:05:12.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:12 smithi125 conmon[24927]: :05:11.459208+0000 osd.1 (osd.1) 82 : cluster [DBG] 5.19 scrub ok 2021-03-24T22:05:12.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:12 smithi125 conmon[24927]: cluster 2021-03-24T22:05:12.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:12 smithi125 conmon[24927]: 2021-03 2021-03-24T22:05:12.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:12 smithi125 conmon[24927]: -24T22:05:12.146052+0000 osd.5 2021-03-24T22:05:12.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:12 smithi125 conmon[24927]: (osd.5) 5 : cluster 2021-03-24T22:05:12.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:12 smithi125 conmon[24927]: [DBG] 5.7 scrub ok 2021-03-24T22:05:12.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:12 smithi125 conmon[24927]: cluster 2021-03-24T22:05:12.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:12 smithi125 conmon[24927]: 2021-03-24T22:05 2021-03-24T22:05:12.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:12 smithi125 conmon[24927]: :12.403559+0000 mon.a (mon 2021-03-24T22:05:12.917 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:12 smithi125 conmon[24927]: .0) 1921 : cluster [DBG] 2021-03-24T22:05:12.917 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:12 smithi125 conmon[24927]: osdmap e742: 8 total, 8 up, 8 in 2021-03-24T22:05:12.941 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:12.955 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:12 smithi142 conmon[20125]: cluster 2021-03-24T22:05:11.459208+0000 osd.1 (osd.1) 82 : cluster [DBG] 5.19 scrub ok 2021-03-24T22:05:12.956 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:12 smithi142 conmon[20125]: cluster 2021-03-24T22:05:12.146052+0000 osd.5 (osd.5) 5 : cluster [DBG] 5.7 scrub ok 2021-03-24T22:05:12.956 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:12 smithi142 conmon[20125]: cluster 2021-03-24T22:05:12.403559+0000 mon.a (mon.0) 1921 : cluster [DBG] osdmap e742: 8 total, 8 up, 8 in 2021-03-24T22:05:12.997 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:13.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:13.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:05:13.103 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:13.160 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:13.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:13.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:05:13.267 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:13.325 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:13.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:13.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:05:13.414 INFO:tasks.rados.rados.0.smithi125.stdout:2853: done (3 left) 2021-03-24T22:05:13.414 INFO:tasks.rados.rados.0.smithi125.stdout:2854: delete oid 8 current snap is 328 2021-03-24T22:05:13.415 INFO:tasks.rados.rados.0.smithi125.stdout:2850: finishing rollback tid 1 to smithi12556470-40 2021-03-24T22:05:13.415 INFO:tasks.rados.rados.0.smithi125.stdout:2850: finishing rollback tid 2 to smithi12556470-40 2021-03-24T22:05:13.415 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 40 v 1755 (ObjNum 906 snap 300 seq_num 906) dirty exists 2021-03-24T22:05:13.415 INFO:tasks.rados.rados.0.smithi125.stdout:2849: expect (ObjNum 907 snap 300 seq_num 907) 2021-03-24T22:05:13.432 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:13.478 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:13.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:13.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:05:13.528 INFO:tasks.rados.rados.0.smithi125.stdout:2852: finishing rollback tid 0 to smithi12556470-24 2021-03-24T22:05:13.529 INFO:tasks.rados.rados.0.smithi125.stdout:2852: finishing rollback tid 1 to smithi12556470-24 2021-03-24T22:05:13.529 INFO:tasks.rados.rados.0.smithi125.stdout:2852: finishing rollback tid 2 to smithi12556470-24 2021-03-24T22:05:13.529 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 24 v 1443 (ObjNum 851 snap 278 seq_num 851) dirty exists 2021-03-24T22:05:13.529 INFO:tasks.rados.rados.0.smithi125.stdout:2849: done (3 left) 2021-03-24T22:05:13.530 INFO:tasks.rados.rados.0.smithi125.stdout:2850: done (2 left) 2021-03-24T22:05:13.530 INFO:tasks.rados.rados.0.smithi125.stdout:2852: done (1 left) 2021-03-24T22:05:13.530 INFO:tasks.rados.rados.0.smithi125.stdout:2854: done (0 left) 2021-03-24T22:05:13.530 INFO:tasks.rados.rados.0.smithi125.stdout:2855: write oid 11 current snap is 328 2021-03-24T22:05:13.530 INFO:tasks.rados.rados.0.smithi125.stdout:2855: seq_num 991 ranges {655737=497649,1607773=789340} 2021-03-24T22:05:13.544 INFO:tasks.rados.rados.0.smithi125.stdout:2855: writing smithi12556470-11 from 655737 to 1153386 tid 1 2021-03-24T22:05:13.551 INFO:tasks.rados.rados.0.smithi125.stdout:2855: writing smithi12556470-11 from 1607773 to 2397113 tid 2 2021-03-24T22:05:13.552 INFO:tasks.rados.rados.0.smithi125.stdout:2856: rollback oid 16 current snap is 328 2021-03-24T22:05:13.552 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 16 to 311 2021-03-24T22:05:13.552 INFO:tasks.rados.rados.0.smithi125.stdout:2855: finishing write tid 1 to smithi12556470-11 2021-03-24T22:05:13.553 INFO:tasks.rados.rados.0.smithi125.stdout:2857: copy_from oid 1 from oid 33 current snap is 328 2021-03-24T22:05:13.553 INFO:tasks.rados.rados.0.smithi125.stdout:2858: snap_remove snap 284 2021-03-24T22:05:13.584 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:13.634 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:13.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:13.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:05:13.741 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:13.754 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd reweight 2 1 2021-03-24T22:05:13.798 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:13.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:13.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:05:13.905 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:13.962 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:13.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:13.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:05:14.069 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:14.122 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.0 dump_blocked_ops 2021-03-24T22:05:14.123 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:14.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:14.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:05:14.157 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:05:14.230 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:14.285 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:14.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:14.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:05:14.304 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:05:14.392 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:14.417 INFO:tasks.rados.rados.0.smithi125.stdout:2858: done (3 left) 2021-03-24T22:05:14.418 INFO:tasks.rados.rados.0.smithi125.stdout:2859: write (excl) oid 41 current snap is 328 2021-03-24T22:05:14.418 INFO:tasks.rados.rados.0.smithi125.stdout:2859: seq_num 992 ranges {672544=572898,1666255=532998} 2021-03-24T22:05:14.424 INFO:tasks.rados.rados.0.smithi125.stdout:2859: writing smithi12556470-41 from 672544 to 1245442 tid 1 2021-03-24T22:05:14.432 INFO:tasks.rados.rados.0.smithi125.stdout:2859: writing smithi12556470-41 from 1666255 to 2199253 tid 2 2021-03-24T22:05:14.432 INFO:tasks.rados.rados.0.smithi125.stdout:2855: finishing write tid 2 to smithi12556470-11 2021-03-24T22:05:14.432 INFO:tasks.rados.rados.0.smithi125.stdout:2855: finishing write tid 3 to smithi12556470-11 2021-03-24T22:05:14.432 INFO:tasks.rados.rados.0.smithi125.stdout:2855: finishing write tid 4 to smithi12556470-11 2021-03-24T22:05:14.433 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 11 v 562 (ObjNum 991 snap 328 seq_num 991) dirty exists 2021-03-24T22:05:14.433 INFO:tasks.rados.rados.0.smithi125.stdout:2855: left oid 11 (ObjNum 991 snap 328 seq_num 991) 2021-03-24T22:05:14.433 INFO:tasks.rados.rados.0.smithi125.stdout:2856: finishing rollback tid 0 to smithi12556470-16 2021-03-24T22:05:14.433 INFO:tasks.rados.rados.0.smithi125.stdout:2856: finishing rollback tid 1 to smithi12556470-16 2021-03-24T22:05:14.434 INFO:tasks.rados.rados.0.smithi125.stdout:2856: finishing rollback tid 2 to smithi12556470-16 2021-03-24T22:05:14.434 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 16 v 831 (ObjNum 875 snap 283 seq_num 875) dirty exists 2021-03-24T22:05:14.434 INFO:tasks.rados.rados.0.smithi125.stdout:2857: finishing copy_from to smithi12556470-1 2021-03-24T22:05:14.434 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 1 v 1440 (ObjNum 764 snap 249 seq_num 764) dirty exists 2021-03-24T22:05:14.435 INFO:tasks.rados.rados.0.smithi125.stdout:2857: finishing copy_from racing read to smithi12556470-1 2021-03-24T22:05:14.435 INFO:tasks.rados.rados.0.smithi125.stdout:2855: done (3 left) 2021-03-24T22:05:14.435 INFO:tasks.rados.rados.0.smithi125.stdout:2856: done (2 left) 2021-03-24T22:05:14.435 INFO:tasks.rados.rados.0.smithi125.stdout:2857: done (1 left) 2021-03-24T22:05:14.435 INFO:tasks.rados.rados.0.smithi125.stdout:2860: read oid 38 snap -1 2021-03-24T22:05:14.436 INFO:tasks.rados.rados.0.smithi125.stdout:2860: expect (ObjNum 973 snap 318 seq_num 973) 2021-03-24T22:05:14.436 INFO:tasks.rados.rados.0.smithi125.stdout:2861: read oid 44 snap -1 2021-03-24T22:05:14.436 INFO:tasks.rados.rados.0.smithi125.stdout:2861: expect deleted 2021-03-24T22:05:14.436 INFO:tasks.rados.rados.0.smithi125.stdout:2862: snap_create 2021-03-24T22:05:14.437 INFO:tasks.rados.rados.0.smithi125.stdout:2859: finishing write tid 1 to smithi12556470-41 2021-03-24T22:05:14.440 INFO:tasks.rados.rados.0.smithi125.stdout:2859: finishing write tid 2 to smithi12556470-41 2021-03-24T22:05:14.440 INFO:tasks.rados.rados.0.smithi125.stdout:2859: finishing write tid 3 to smithi12556470-41 2021-03-24T22:05:14.441 INFO:tasks.rados.rados.0.smithi125.stdout:2859: finishing write tid 4 to smithi12556470-41 2021-03-24T22:05:14.441 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 41 v 1050 (ObjNum 992 snap 328 seq_num 992) dirty exists 2021-03-24T22:05:14.442 INFO:tasks.rados.rados.0.smithi125.stdout:2859: left oid 41 (ObjNum 992 snap 328 seq_num 992) 2021-03-24T22:05:14.442 INFO:tasks.rados.rados.0.smithi125.stdout:2860: expect (ObjNum 973 snap 318 seq_num 973) 2021-03-24T22:05:14.452 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:14.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:14.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:05:14.544 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:14 smithi125 conmon[19936]: cluster 2021-03-24T22:05:12.976747+0000 mgr.y (mgr.14144) 646 : cluster [DBG] pgmap v1276: 168 pgs: 15 active+clean+wait, 153 active+clean; 539 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.2 MiB/s rd, 607 KiB/s wr, 9 op/s; 13 MiB/s, 8 objects/s recovering 2021-03-24T22:05:14.544 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:14 smithi125 conmon[19936]: cluster 2021-03-24T22:05:13.182569+0000 osd.7 (osd.7) 123 : cluster [DBG] 7.17 scrub ok 2021-03-24T22:05:14.544 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:14 smithi125 conmon[19936]: cluster 2021-03-24T22:05:13 2021-03-24T22:05:14.544 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:14 smithi125 conmon[19936]: .207186+0000 osd.3 (osd.3) 149 : cluster [DBG] 5.0 scrub ok 2021-03-24T22:05:14.545 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:14 smithi125 conmon[19936]: cluster 2021-03-24T22:05:13.406260+0000 2021-03-24T22:05:14.545 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:14 smithi125 conmon[19936]: mon.a (mon.0) 1922 : cluster [DBG] osdmap e743: 8 total, 8 up, 8 in 2021-03-24T22:05:14.545 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:14 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:05:14.545 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:14 smithi125 conmon[19936]: 03-24T22:05:13.446714+0000 osd.1 ( 2021-03-24T22:05:14.546 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:14 smithi125 conmon[19936]: osd.1) 83 : cluster [DBG] 4.7 scrub ok 2021-03-24T22:05:14.546 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:14 smithi125 conmon[24927]: cluster 2021-03-24T22:05:12.976747+0000 2021-03-24T22:05:14.546 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:14 smithi125 conmon[24927]: mgr.y (mgr.14144) 646 : cluster [DBG] pgmap v1276: 168 pgs: 15 active+clean+wait, 153 active+clean; 539 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.2 MiB/s rd, 607 KiB/s wr, 9 op/s; 13 MiB/s, 8 objects/s recovering 2021-03-24T22:05:14.546 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:14 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:05:14.547 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:14 smithi125 conmon[24927]: -03-24T22:05:13.182569+0000 osd.7 (osd.7) 123 : cluster [DBG] 7.17 scrub ok 2021-03-24T22:05:14.547 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:14 smithi125 conmon[24927]: cluster 2021-03-24T22:05:13.207186+0000 osd.3 (osd.3) 149 : cluster [DBG] 5.0 scrub ok 2021-03-24T22:05:14.547 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:14 smithi125 conmon[24927]: cluster 2021-03-24T22:05:13.406260+0000 mon.a (mon.0) 1922 : cluster [DBG] osdmap e743: 8 total, 8 up, 8 in 2021-03-24T22:05:14.547 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:14 smithi125 conmon[24927]: cluster 2021-03-24T22:05:13.446714+0000 osd.1 (osd.1) 83 : cluster [DBG] 4.7 scrub ok 2021-03-24T22:05:14.548 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:05:14.557 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:14.617 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:14.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:14.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:05:14.721 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:14.771 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:14.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:14.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:05:14.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:14 smithi142 conmon[20125]: cluster 2021-03-24T22:05:12.976747+0000 2021-03-24T22:05:14.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:14 smithi142 conmon[20125]: mgr.y (mgr.14144) 646 : cluster [DBG] pgmap v1276: 168 pgs: 15 active+clean+wait, 153 active+clean; 539 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.2 MiB/s rd, 607 KiB/s wr, 9 op/s; 13 MiB/s, 8 objects/s recovering 2021-03-24T22:05:14.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:14 smithi142 conmon[20125]: cluster 2021-03-24T22:05:13.182569+0000 2021-03-24T22:05:14.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:14 smithi142 conmon[20125]: osd.7 (osd.7) 123 : cluster [DBG] 7.17 scrub ok 2021-03-24T22:05:14.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:14 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:05:14.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:14 smithi142 conmon[20125]: 03-24T22:05:13.207186+0000 osd.3 2021-03-24T22:05:14.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:14 smithi142 conmon[20125]: (osd.3) 149 : cluster [DBG] 5.0 scrub ok 2021-03-24T22:05:14.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:14 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:05:14.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:14 smithi142 conmon[20125]: -24T22:05:13.406260+0000 mon.a ( 2021-03-24T22:05:14.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:14 smithi142 conmon[20125]: mon.0) 1922 : cluster [DBG] osdmap e743: 8 total, 8 up, 8 in 2021-03-24T22:05:14.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:14 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:05:14.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:14 smithi142 conmon[20125]: -03-24T22:05:13.446714+0000 2021-03-24T22:05:14.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:14 smithi142 conmon[20125]: osd.1 (osd.1) 83 : cluster [DBG] 2021-03-24T22:05:14.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:14 smithi142 conmon[20125]: 4.7 scrub ok 2021-03-24T22:05:14.878 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:14.927 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:14.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:14.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:05:15.035 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:15.088 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:15.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:15.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:05:15.194 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:15.250 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:15.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:15.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:05:15.356 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:15.399 INFO:tasks.rados.rados.0.smithi125.stdout:2859: done (3 left) 2021-03-24T22:05:15.400 INFO:tasks.rados.rados.0.smithi125.stdout:2860: done (2 left) 2021-03-24T22:05:15.400 INFO:tasks.rados.rados.0.smithi125.stdout:2861: done (1 left) 2021-03-24T22:05:15.400 INFO:tasks.rados.rados.0.smithi125.stdout:2862: done (0 left) 2021-03-24T22:05:15.400 INFO:tasks.rados.rados.0.smithi125.stdout:2863: write (excl) oid 2 current snap is 329 2021-03-24T22:05:15.401 INFO:tasks.rados.rados.0.smithi125.stdout:2863: seq_num 993 ranges {689351=648147,2124737=564614} 2021-03-24T22:05:15.401 INFO:teuthology.orchestra.run.smithi125.stderr:noscrub is set 2021-03-24T22:05:15.408 INFO:tasks.rados.rados.0.smithi125.stdout:2863: writing smithi12556470-2 from 689351 to 1337498 tid 1 2021-03-24T22:05:15.417 INFO:tasks.rados.rados.0.smithi125.stdout:2863: writing smithi12556470-2 from 2124737 to 2689351 tid 2 2021-03-24T22:05:15.417 INFO:tasks.rados.rados.0.smithi125.stdout:2864: rollback oid 9 current snap is 329 2021-03-24T22:05:15.418 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 9 to 304 2021-03-24T22:05:15.418 INFO:tasks.rados.rados.0.smithi125.stdout:2863: finishing write tid 1 to smithi12556470-2 2021-03-24T22:05:15.418 INFO:tasks.rados.rados.0.smithi125.stdout:2865: rollback oid 17 current snap is 329 2021-03-24T22:05:15.419 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 17 to 326 2021-03-24T22:05:15.419 INFO:tasks.rados.rados.0.smithi125.stdout:2866: snap_remove snap 251 2021-03-24T22:05:15.419 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:15.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:15.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:05:15.530 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:15.543 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[24927]: cluster 2021-03-24T22:05:15.543 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[24927]: 2021-03-24T22:05:15.543 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[24927]: 2021-03-24T22:05:14. 2021-03-24T22:05:15.544 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[24927]: 123424+0000 osd.7 (osd.7) 124 : cluster [DBG] 5.4 scrub ok 2021-03-24T22:05:15.544 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[24927]: cluster 2021-03-24T22 2021-03-24T22:05:15.544 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[24927]: :05:14.247906+0000 osd.3 (osd.3) 150 : cluster 2021-03-24T22:05:15.545 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[24927]: [DBG] 6.b scrub ok 2021-03-24T22:05:15.545 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[24927]: cluster 2021-03-24T22:05:14. 2021-03-24T22:05:15.545 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[24927]: 412836+0000 mon.a (mon.0) 1923 : cluster [DBG] 2021-03-24T22:05:15.546 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[24927]: osdmap e744: 8 total, 8 up, 8 in 2021-03-24T22:05:15.546 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:05:15.546 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[24927]: -03-24T22:05:14.424328+0000 osd.6 (osd.6) 71 2021-03-24T22:05:15.547 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[24927]: : cluster [DBG] 5.f scrub ok 2021-03-24T22:05:15.547 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[24927]: cluster 2021-03-24T22:05:14 2021-03-24T22:05:15.547 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[24927]: .489810+0000 osd.0 (osd.0) 145 : cluster [DBG] 2021-03-24T22:05:15.547 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[24927]: purged_snaps scrub starts 2021-03-24T22:05:15.547 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[24927]: cluster 2021-03-24T22:05:14 2021-03-24T22:05:15.548 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[24927]: .491206+0000 osd.0 (osd.0) 146 2021-03-24T22:05:15.548 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[24927]: : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:05:15.548 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[24927]: audit 2021 2021-03-24T22:05:15.549 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[24927]: -03-24T22:05:14 2021-03-24T22:05:15.549 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[24927]: .544373+0000 mon.a (mon.0) 1924 2021-03-24T22:05:15.549 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[24927]: : audit [INF] from='client.? v1:172.21.15.125:0/1238408257' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T22:05:15.550 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T22:05:15.550 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[24927]: 24T22:05:14.764408+0000 2021-03-24T22:05:15.550 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[24927]: mon.a (mon.0) 1925 : audit [DBG] 2021-03-24T22:05:15.550 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[24927]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T22:05:15.551 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[24927]: audit 2021-03-24T 2021-03-24T22:05:15.551 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[24927]: 22:05:15.097926+0000 mon.a 2021-03-24T22:05:15.551 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[24927]: (mon.0) 1926 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:05:15.551 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[24927]: 2021-03-24T22:05:15.552 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:05:15.553 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[24927]: 05:15.154476+0000 osd.5 (osd.5) 6 2021-03-24T22:05:15.554 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[24927]: : cluster [DBG] 6.19 deep-scrub ok 2021-03-24T22:05:15.554 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[24927]: cluster 2021-03-24T22 2021-03-24T22:05:15.554 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[24927]: :05:15.381403+0000 mon.a (mon.0) 1927 2021-03-24T22:05:15.554 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[24927]: : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:05:15.555 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[24927]: audit 2021-03-24T22:05:15 2021-03-24T22:05:15.555 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[24927]: .390098+0000 mon.a (mon.0) 1928 : audit [INF] 2021-03-24T22:05:15.555 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[24927]: from='client.? v1:172.21.15.125:0/1238408257' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T22:05:15.556 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[24927]: cluster 2021-03-24T22:05 2021-03-24T22:05:15.556 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[24927]: :15.390213+0000 mon.a (mon.0 2021-03-24T22:05:15.557 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[24927]: ) 1929 : cluster [DBG] osdmap e745: 8 total, 8 up, 8 in 2021-03-24T22:05:15.558 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[19936]: cluster 2021-03-24T22:05:14.123424+0000 osd.7 (osd. 2021-03-24T22:05:15.558 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[19936]: 7) 124 : cluster [DBG] 5.4 scrub ok 2021-03-24T22:05:15.559 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[19936]: cluster 2021-03-24T22:05:14.247906+0000 osd.3 (osd.3) 150 : cluster [DBG] 6.b scrub ok 2021-03-24T22:05:15.559 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[19936]: cluster 2021-03-24T22:05 2021-03-24T22:05:15.560 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[19936]: :14.412836+0000 mon.a (mon.0) 1923 : cluster [DBG] osdmap e744: 8 total, 8 up, 8 in 2021-03-24T22:05:15.560 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T22:05:15.561 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[19936]: 22:05:14.424328+0000 osd.6 (osd.6) 71 : cluster [DBG] 5.f scrub ok 2021-03-24T22:05:15.561 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T22:05:15.561 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[19936]: 22:05:14.489810+0000 osd.0 (osd.0) 145 : 2021-03-24T22:05:15.562 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[19936]: cluster [DBG] purged_snaps scrub starts 2021-03-24T22:05:15.563 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[19936]: cluster 2021-03-24T22:05:15.563 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[19936]: 2021-03-24T22:05:14.491206+0000 osd.0 (osd.0) 146 : cluster 2021-03-24T22:05:15.564 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[19936]: [DBG] purged_snaps scrub ok 2021-03-24T22:05:15.564 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[19936]: audit 2021-03-24T22:05:15.565 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[19936]: 2021-03-24T22:05:14.544373+0000 mon.a (mon.0) 1924 2021-03-24T22:05:15.565 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[19936]: : audit [INF] from='client.? v1:172.21.15.125:0/1238408257' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T22:05:15.566 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[19936]: audit 2021-03-24T22:05:15.566 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[19936]: 2021-03-24T22:05:14.764408+0000 mon.a (mon.0) 1925 : 2021-03-24T22:05:15.567 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[19936]: audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T22:05:15.567 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[19936]: audit 2021-03-24T22:05:15.567 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[19936]: 2021-03-24T22:05:15.097926+0000 mon.a (mon.0) 1926 : 2021-03-24T22:05:15.568 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[19936]: audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:05:15.568 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[19936]: cluster 2021-03-24T22:05:15.569 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[19936]: 2021-03-24T22:05:15.154476+0000 osd.5 (osd.5) 6 : 2021-03-24T22:05:15.569 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[19936]: cluster [DBG] 6.19 deep-scrub ok 2021-03-24T22:05:15.570 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[19936]: cluster 2021-03-24T22:05:15.381403+0000 2021-03-24T22:05:15.570 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[19936]: mon.a (mon.0) 1927 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:05:15.571 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[19936]: audit 2021-03-24T22: 2021-03-24T22:05:15.571 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[19936]: 05:15.390098+0000 mon.a (mon.0) 1928 : audit 2021-03-24T22:05:15.572 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[19936]: [INF] from='client.? v1:172.21.15.125:0/1238408257' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T22:05:15.572 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:05:15.573 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[19936]: -03-24T22:05:15.390213+0000 mon.a (mon.0) 1929 : cluster 2021-03-24T22:05:15.574 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:15 smithi125 conmon[19936]: [DBG] osdmap e745: 8 total, 8 up, 8 in 2021-03-24T22:05:15.585 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:15.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:15.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:05:15.692 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:15.744 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:15.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:15.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:05:15.851 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:15.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:15 smithi142 conmon[20125]: cluster 2021-03-24T22:05:14.123424+0000 osd.7 ( 2021-03-24T22:05:15.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:15 smithi142 conmon[20125]: osd.7) 124 : cluster [DBG] 5.4 scrub ok 2021-03-24T22:05:15.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:15 smithi142 conmon[20125]: cluster 2021-03-24T22:05:14.247906+0000 osd.3 (osd.3) 150 : cluster [DBG] 6.b scrub ok 2021-03-24T22:05:15.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:15 smithi142 conmon[20125]: cluster 2021-03-24T22:05:14.412836+0000 mon.a (mon.0) 1923 : cluster [DBG] osdmap e744: 8 total, 8 up, 8 in 2021-03-24T22:05:15.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:15 smithi142 conmon[20125]: cluster 2021-03-24T22: 2021-03-24T22:05:15.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:15 smithi142 conmon[20125]: 05:14.424328+0000 osd.6 (osd.6) 71 : cluster [DBG] 2021-03-24T22:05:15.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:15 smithi142 conmon[20125]: 5.f scrub ok 2021-03-24T22:05:15.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:15 smithi142 conmon[20125]: cluster 2021-03-24T22:05:15.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:15 smithi142 conmon[20125]: 2021-03-24T22:05:14.489810+0000 osd.0 (osd.0) 145 : cluster 2021-03-24T22:05:15.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:15 smithi142 conmon[20125]: [DBG] purged_snaps scrub starts 2021-03-24T22:05:15.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:15 smithi142 conmon[20125]: cluster 2021-03-24T22:05:15.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:15 smithi142 conmon[20125]: 2021-03-24T22:05:14.491206+0000 osd.0 (osd.0) 146 : cluster 2021-03-24T22:05:15.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:15 smithi142 conmon[20125]: [DBG] purged_snaps scrub ok 2021-03-24T22:05:15.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:15 smithi142 conmon[20125]: audit 2021-03-24T22:05:14.544373+0000 mon.a ( 2021-03-24T22:05:15.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:15 smithi142 conmon[20125]: mon.0) 1924 : audit [INF] from='client.? v1:172.21.15.125:0/1238408257' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T22:05:15.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:15 smithi142 conmon[20125]: audit 2021-03-24T22:05: 2021-03-24T22:05:15.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:15 smithi142 conmon[20125]: 14.764408+0000 mon.a (mon.0) 1925 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T22:05:15.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:15 smithi142 conmon[20125]: 2021-03-24T22:05:15.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:15 smithi142 conmon[20125]: audit 2021-03 2021-03-24T22:05:15.872 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:15 smithi142 conmon[20125]: -24T22:05:15.097926+0000 mon.a (mon.0) 1926 : audit [INF] 2021-03-24T22:05:15.872 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:15 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:05:15.872 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:15 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:05:15.872 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:15 smithi142 conmon[20125]: -24T22:05:15.154476+0000 osd.5 (osd.5) 6 : cluster [DBG] 6.19 deep-scrub ok 2021-03-24T22:05:15.873 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:15 smithi142 conmon[20125]: 2021-03-24T22:05:15.873 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:15 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:05:15.873 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:15 smithi142 conmon[20125]: 03-24T22:05:15.381403+0000 mon.a (mon.0) 1927 : cluster [WRN] 2021-03-24T22:05:15.873 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:15 smithi142 conmon[20125]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:05:15.873 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:15 smithi142 conmon[20125]: audit 2021-03-24T22:05:15.874 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:15 smithi142 conmon[20125]: 2021-03-24T22:05:15.390098+0000 mon.a (mon.0) 1928 : audit [INF] 2021-03-24T22:05:15.874 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:15 smithi142 conmon[20125]: from='client.? v1:172.21.15.125:0/1238408257' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T22:05:15.874 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:15 smithi142 conmon[20125]: cluster 2021-03-24T22:05:15.874 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:15 smithi142 conmon[20125]: 2021-03-24T22:05:15.390213+0000 mon.a (mon.0) 1929 : cluster [DBG] 2021-03-24T22:05:15.875 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:15 smithi142 conmon[20125]: osdmap e745: 8 total, 8 up, 8 in 2021-03-24T22:05:15.909 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:15.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:15.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:05:16.016 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:16.073 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:16.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:16.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:05:16.180 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:16.236 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:16.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:16.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:05:16.343 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:16.398 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:16.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:16.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:05:16.434 INFO:tasks.rados.rados.0.smithi125.stdout:2866: done (3 left) 2021-03-24T22:05:16.434 INFO:tasks.rados.rados.0.smithi125.stdout:2867: write oid 37 current snap is 329 2021-03-24T22:05:16.434 INFO:tasks.rados.rados.0.smithi125.stdout:2867: seq_num 994 ranges {706158=723396,2183219=522939} 2021-03-24T22:05:16.441 INFO:tasks.rados.rados.0.smithi125.stdout:2867: writing smithi12556470-37 from 706158 to 1429554 tid 1 2021-03-24T22:05:16.450 INFO:tasks.rados.rados.0.smithi125.stdout:2867: writing smithi12556470-37 from 2183219 to 2706158 tid 2 2021-03-24T22:05:16.450 INFO:tasks.rados.rados.0.smithi125.stdout:2868: delete oid 4 current snap is 329 2021-03-24T22:05:16.451 INFO:tasks.rados.rados.0.smithi125.stdout:2865: finishing rollback tid 1 to smithi12556470-17 2021-03-24T22:05:16.451 INFO:tasks.rados.rados.0.smithi125.stdout:2865: finishing rollback tid 2 to smithi12556470-17 2021-03-24T22:05:16.451 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 17 v 1058 (ObjNum 974 snap 318 seq_num 974) dirty exists 2021-03-24T22:05:16.451 INFO:tasks.rados.rados.0.smithi125.stdout:2864: finishing rollback tid 1 to smithi12556470-9 2021-03-24T22:05:16.451 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 9 v 1583 (ObjNum 1117543520 snap 21916 seq_num 0) dirty dne 2021-03-24T22:05:16.452 INFO:tasks.rados.rados.0.smithi125.stdout:2863: finishing write tid 2 to smithi12556470-2 2021-03-24T22:05:16.452 INFO:tasks.rados.rados.0.smithi125.stdout:2863: finishing write tid 3 to smithi12556470-2 2021-03-24T22:05:16.452 INFO:tasks.rados.rados.0.smithi125.stdout:2863: finishing write tid 4 to smithi12556470-2 2021-03-24T22:05:16.452 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 2 v 1114 (ObjNum 993 snap 329 seq_num 993) dirty exists 2021-03-24T22:05:16.453 INFO:tasks.rados.rados.0.smithi125.stdout:2863: left oid 2 (ObjNum 993 snap 329 seq_num 993) 2021-03-24T22:05:16.453 INFO:tasks.rados.rados.0.smithi125.stdout:2867: finishing write tid 1 to smithi12556470-37 2021-03-24T22:05:16.455 INFO:tasks.rados.rados.0.smithi125.stdout:2867: finishing write tid 2 to smithi12556470-37 2021-03-24T22:05:16.459 INFO:tasks.rados.rados.0.smithi125.stdout:2867: finishing write tid 3 to smithi12556470-37 2021-03-24T22:05:16.460 INFO:tasks.rados.rados.0.smithi125.stdout:2863: done (4 left) 2021-03-24T22:05:16.460 INFO:tasks.rados.rados.0.smithi125.stdout:2864: done (3 left) 2021-03-24T22:05:16.461 INFO:tasks.rados.rados.0.smithi125.stdout:2865: done (2 left) 2021-03-24T22:05:16.461 INFO:tasks.rados.rados.0.smithi125.stdout:2868: done (1 left) 2021-03-24T22:05:16.461 INFO:tasks.rados.rados.0.smithi125.stdout:2869: read oid 50 snap -1 2021-03-24T22:05:16.462 INFO:tasks.rados.rados.0.smithi125.stdout:2869: expect (ObjNum 941 snap 314 seq_num 941) 2021-03-24T22:05:16.462 INFO:tasks.rados.rados.0.smithi125.stdout:2870: write oid 7 current snap is 329 2021-03-24T22:05:16.463 INFO:tasks.rados.rados.0.smithi125.stdout:2870: seq_num 996 ranges {739772=590247,1900183=540336} 2021-03-24T22:05:16.465 INFO:tasks.rados.rados.0.smithi125.stdout:2870: writing smithi12556470-7 from 739772 to 1330019 tid 1 2021-03-24T22:05:16.473 INFO:tasks.rados.rados.0.smithi125.stdout:2870: writing smithi12556470-7 from 1900183 to 2440519 tid 2 2021-03-24T22:05:16.474 INFO:tasks.rados.rados.0.smithi125.stdout:2871: delete oid 43 current snap is 329 2021-03-24T22:05:16.476 INFO:tasks.rados.rados.0.smithi125.stdout:2867: finishing write tid 4 to smithi12556470-37 2021-03-24T22:05:16.476 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 37 v 743 (ObjNum 994 snap 329 seq_num 994) dirty exists 2021-03-24T22:05:16.476 INFO:tasks.rados.rados.0.smithi125.stdout:2867: left oid 37 (ObjNum 994 snap 329 seq_num 994) 2021-03-24T22:05:16.477 INFO:tasks.rados.rados.0.smithi125.stdout:2869: expect (ObjNum 941 snap 314 seq_num 941) 2021-03-24T22:05:16.503 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[19936]: cluster 2021-03-24T22:05:14.977541+0000 2021-03-24T22:05:16.503 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[19936]: mgr.y (mgr.14144) 647 : cluster [DBG] pgmap v1279: 168 pgs: 15 active+clean+wait, 153 active+clean; 562 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 3.9 MiB/s wr, 20 op/s 2021-03-24T22:05:16.504 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[19936]: cluster 2021-03-24T22:05:15.379474+0000 osd.6 (osd.6) 72 : cluster [DBG] 6.a scrub ok 2021-03-24T22:05:16.504 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[19936]: audit 2021- 2021-03-24T22:05:16.504 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[19936]: 03-24T22:05:15.446538+0000 mon.a (mon.0) 1930 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T22:05:16.504 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:05:16.505 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[19936]: -03-24T22:05:16.425390+0000 mon.a (mon.0) 1931 : cluster [DBG] 2021-03-24T22:05:16.505 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[19936]: osdmap e746: 8 total, 8 up, 8 in 2021-03-24T22:05:16.505 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:16.506 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: cluster 2021-03-24T22:05:16.507 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: 2021-03-24T22:05:16.507 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: 2021 2021-03-24T22:05:16.507 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: - 2021-03-24T22:05:16.507 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: 03 2021-03-24T22:05:16.508 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: - 2021-03-24T22:05:16.508 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: 24 2021-03-24T22:05:16.508 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: T 2021-03-24T22:05:16.508 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: 22 2021-03-24T22:05:16.509 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: : 2021-03-24T22:05:16.509 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: 05 2021-03-24T22:05:16.509 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: : 2021-03-24T22:05:16.509 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: 14 2021-03-24T22:05:16.509 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: . 2021-03-24T22:05:16.510 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: 977541 2021-03-24T22:05:16.510 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: +0000 2021-03-24T22:05:16.510 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: 2021-03-24T22:05:16.510 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: mgr.y 2021-03-24T22:05:16.510 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: ( 2021-03-24T22:05:16.511 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: mgr 2021-03-24T22:05:16.511 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: . 2021-03-24T22:05:16.511 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: 14144 2021-03-24T22:05:16.511 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: ) 2021-03-24T22:05:16.512 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: 647 2021-03-24T22:05:16.512 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: : 2021-03-24T22:05:16.512 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: cluster 2021-03-24T22:05:16.512 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: 2021-03-24T22:05:16.512 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: [DBG] 2021-03-24T22:05:16.513 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: 2021-03-24T22:05:16.513 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: pgmap v1279: 168 pgs: 15 active+clean+wait, 153 active+clean; 562 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 3.9 MiB/s wr, 20 op/s 2021-03-24T22:05:16.513 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: 2021-03-24T22:05:16.513 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: cluster 2021-03-24T22:05:16.513 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: 2021-03-24T22:05:16.514 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: 2021 2021-03-24T22:05:16.514 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: - 2021-03-24T22:05:16.514 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: 03 2021-03-24T22:05:16.515 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: - 2021-03-24T22:05:16.515 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: 24 2021-03-24T22:05:16.515 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: T 2021-03-24T22:05:16.515 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: 22 2021-03-24T22:05:16.564 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:16.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:16.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:05:16.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:16 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:05:16.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:16 smithi142 conmon[20125]: -24T22:05:14.977541+0000 mgr.y (mgr.14144) 647 : cluster [DBG] 2021-03-24T22:05:16.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:16 smithi142 conmon[20125]: pgmap v1279: 168 pgs: 15 active+clean+wait, 153 active+clean; 562 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 3.9 MiB/s wr, 20 op/s 2021-03-24T22:05:16.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:16 smithi142 conmon[20125]: cluster 2021-03-24T22:05:16.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:16 smithi142 conmon[20125]: 2021-03-24T22:05 2021-03-24T22:05:16.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:16 smithi142 conmon[20125]: :15.379474+0000 2021-03-24T22:05:16.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:16 smithi142 conmon[20125]: osd.6 (osd. 2021-03-24T22:05:16.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:16 smithi142 conmon[20125]: 6) 72 : cluster 2021-03-24T22:05:16.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:16 smithi142 conmon[20125]: [DBG] 6.a scrub ok 2021-03-24T22:05:16.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:16 smithi142 conmon[20125]: audit 2021- 2021-03-24T22:05:16.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:16 smithi142 conmon[20125]: 03-24T22:05:15 2021-03-24T22:05:16.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:16 smithi142 conmon[20125]: .446538+0000 mon.a ( 2021-03-24T22:05:16.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:16 smithi142 conmon[20125]: mon.0) 1930 : 2021-03-24T22:05:16.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:16 smithi142 conmon[20125]: audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T22:05:16.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:16 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:05:16.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:16 smithi142 conmon[20125]: 03-24T22:05:16 2021-03-24T22:05:16.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:16 smithi142 conmon[20125]: .425390+0000 mon.a ( 2021-03-24T22:05:16.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:16 smithi142 conmon[20125]: mon.0) 1931 : 2021-03-24T22:05:16.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:16 smithi142 conmon[20125]: cluster [DBG] osdmap e746: 8 total, 8 up, 8 in 2021-03-24T22:05:16.670 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:16.707 INFO:tasks.rados.rados.0.smithi125.stdout:2870: finishing write tid 1 to smithi12556470-7 2021-03-24T22:05:16.708 INFO:tasks.rados.rados.0.smithi125.stdout:2870: finishing write tid 2 to smithi12556470-7 2021-03-24T22:05:16.708 INFO:tasks.rados.rados.0.smithi125.stdout:2867: done (3 left) 2021-03-24T22:05:16.709 INFO:tasks.rados.rados.0.smithi125.stdout:2869: done (2 left) 2021-03-24T22:05:16.714 INFO:tasks.rados.rados.0.smithi125.stdout:2871: done (1 left) 2021-03-24T22:05:16.716 INFO:tasks.rados.rados.0.smithi125.stdout:2872: read oid 23 snap -1 2021-03-24T22:05:16.716 INFO:tasks.rados.rados.0.smithi125.stdout:2872: expect deleted 2021-03-24T22:05:16.716 INFO:tasks.rados.rados.0.smithi125.stdout:2873: copy_from oid 46 from oid 16 current snap is 329 2021-03-24T22:05:16.716 INFO:tasks.rados.rados.0.smithi125.stdout:2870: finishing write tid 3 to smithi12556470-7 2021-03-24T22:05:16.717 INFO:tasks.rados.rados.0.smithi125.stdout:2870: finishing write tid 4 to smithi12556470-7 2021-03-24T22:05:16.717 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 7 v 1126 (ObjNum 996 snap 329 seq_num 996) dirty exists 2021-03-24T22:05:16.717 INFO:tasks.rados.rados.0.smithi125.stdout:2870: left oid 7 (ObjNum 996 snap 329 seq_num 996) 2021-03-24T22:05:16.717 INFO:tasks.rados.rados.0.smithi125.stdout:2870: done (2 left) 2021-03-24T22:05:16.717 INFO:tasks.rados.rados.0.smithi125.stdout:2874: read oid 3 snap -1 2021-03-24T22:05:16.718 INFO:tasks.rados.rados.0.smithi125.stdout:2874: expect deleted 2021-03-24T22:05:16.718 INFO:tasks.rados.rados.0.smithi125.stdout:2875: rollback oid 10 current snap is 329 2021-03-24T22:05:16.718 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 10 to 293 2021-03-24T22:05:16.718 INFO:tasks.rados.rados.0.smithi125.stdout:2876: delete oid 34 current snap is 329 2021-03-24T22:05:16.720 INFO:tasks.rados.rados.0.smithi125.stdout:2872: done (4 left) 2021-03-24T22:05:16.720 INFO:tasks.rados.rados.0.smithi125.stdout:2874: done (3 left) 2021-03-24T22:05:16.720 INFO:tasks.rados.rados.0.smithi125.stdout:2876: done (2 left) 2021-03-24T22:05:16.721 INFO:tasks.rados.rados.0.smithi125.stdout:2877: rollback oid 33 current snap is 329 2021-03-24T22:05:16.721 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 33 to 314 2021-03-24T22:05:16.721 INFO:tasks.rados.rados.0.smithi125.stdout:2878: delete oid 32 current snap is 329 2021-03-24T22:05:16.721 INFO:tasks.rados.rados.0.smithi125.stdout:2878: done (3 left) 2021-03-24T22:05:16.721 INFO:tasks.rados.rados.0.smithi125.stdout:2879: snap_create 2021-03-24T22:05:16.722 INFO:tasks.rados.rados.0.smithi125.stdout:2875: finishing rollback tid 0 to smithi12556470-10 2021-03-24T22:05:16.722 INFO:tasks.rados.rados.0.smithi125.stdout:2875: finishing rollback tid 1 to smithi12556470-10 2021-03-24T22:05:16.722 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 10 v 1445 (ObjNum 1117435056 snap 21916 seq_num 1116329856) dirty dne 2021-03-24T22:05:16.722 INFO:tasks.rados.rados.0.smithi125.stdout:2877: finishing rollback tid 0 to smithi12556470-33 2021-03-24T22:05:16.723 INFO:tasks.rados.rados.0.smithi125.stdout:2877: finishing rollback tid 1 to smithi12556470-33 2021-03-24T22:05:16.724 INFO:tasks.rados.rados.0.smithi125.stdout:2877: finishing rollback tid 2 to smithi12556470-33 2021-03-24T22:05:16.724 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 33 v 1448 (ObjNum 940 snap 314 seq_num 940) dirty exists 2021-03-24T22:05:16.724 INFO:tasks.rados.rados.0.smithi125.stdout:2873: finishing copy_from to smithi12556470-46 2021-03-24T22:05:16.725 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 46 v 831 (ObjNum 875 snap 283 seq_num 875) dirty exists 2021-03-24T22:05:16.725 INFO:tasks.rados.rados.0.smithi125.stdout:2873: finishing copy_from racing read to smithi12556470-46 2021-03-24T22:05:16.726 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:16.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:16.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:05:16.830 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:16.846 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: : 2021-03-24T22:05:16.846 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: 05 2021-03-24T22:05:16.846 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: : 2021-03-24T22:05:16.846 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: 15 2021-03-24T22:05:16.847 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: . 2021-03-24T22:05:16.847 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: 379474 2021-03-24T22:05:16.847 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: +0000 2021-03-24T22:05:16.847 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: 2021-03-24T22:05:16.848 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: osd.6 2021-03-24T22:05:16.848 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: ( 2021-03-24T22:05:16.848 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: osd 2021-03-24T22:05:16.848 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: . 2021-03-24T22:05:16.849 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: 6 2021-03-24T22:05:16.849 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: ) 2021-03-24T22:05:16.849 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: 72 2021-03-24T22:05:16.849 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: : 2021-03-24T22:05:16.849 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: cluster 2021-03-24T22:05:16.850 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: 2021-03-24T22:05:16.850 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: [DBG] 2021-03-24T22:05:16.850 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: 2021-03-24T22:05:16.850 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: 6.a scrub ok 2021-03-24T22:05:16.851 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: 2021-03-24T22:05:16.851 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: audit 2021-03-24T22:05:16.851 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: 2021-03-24T22:05:16.851 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: 2021 2021-03-24T22:05:16.852 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: - 2021-03-24T22:05:16.852 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: 03 2021-03-24T22:05:16.852 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: - 2021-03-24T22:05:16.852 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: 24 2021-03-24T22:05:16.853 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: T 2021-03-24T22:05:16.853 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: 22 2021-03-24T22:05:16.853 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: : 2021-03-24T22:05:16.853 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: 05 2021-03-24T22:05:16.853 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: : 2021-03-24T22:05:16.854 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: 15 2021-03-24T22:05:16.854 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: . 2021-03-24T22:05:16.854 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: 446538 2021-03-24T22:05:16.854 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: +0000 2021-03-24T22:05:16.855 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: 2021-03-24T22:05:16.855 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: mon.a 2021-03-24T22:05:16.855 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: ( 2021-03-24T22:05:16.855 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: mon 2021-03-24T22:05:16.856 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: . 2021-03-24T22:05:16.856 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: 0 2021-03-24T22:05:16.856 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: ) 2021-03-24T22:05:16.856 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: 1930 2021-03-24T22:05:16.856 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: : 2021-03-24T22:05:16.857 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: audit 2021-03-24T22:05:16.857 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: 2021-03-24T22:05:16.857 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: [INF] 2021-03-24T22:05:16.857 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: 2021-03-24T22:05:16.857 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T22:05:16.858 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: 2021-03-24T22:05:16.858 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: cluster 2021-03-24T22:05:16.858 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: 2021-03-24T22:05:16.859 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: 2021 2021-03-24T22:05:16.861 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: - 2021-03-24T22:05:16.861 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: 03 2021-03-24T22:05:16.862 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: - 2021-03-24T22:05:16.862 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: 24 2021-03-24T22:05:16.863 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: T 2021-03-24T22:05:16.863 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: 22:05:16.425390 2021-03-24T22:05:16.863 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:16 smithi125 conmon[24927]: +0000 mon.a (mon.0) 1931 : cluster [DBG] osdmap e746: 8 total, 8 up, 8 in 2021-03-24T22:05:16.883 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:16.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:16.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:05:16.989 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:17.041 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:17.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:17.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:05:17.149 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:17.207 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:17.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:17.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:05:17.313 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:17.371 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:17.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:17.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:05:17.450 INFO:tasks.rados.rados.0.smithi125.stdout:2873: done (3 left) 2021-03-24T22:05:17.451 INFO:tasks.rados.rados.0.smithi125.stdout:2875: done (2 left) 2021-03-24T22:05:17.451 INFO:tasks.rados.rados.0.smithi125.stdout:2877: done (1 left) 2021-03-24T22:05:17.451 INFO:tasks.rados.rados.0.smithi125.stdout:2879: done (0 left) 2021-03-24T22:05:17.452 INFO:tasks.rados.rados.0.smithi125.stdout:2880: delete oid 28 current snap is 330 2021-03-24T22:05:17.455 INFO:tasks.rados.rados.0.smithi125.stdout:2880: done (0 left) 2021-03-24T22:05:17.455 INFO:tasks.rados.rados.0.smithi125.stdout:2881: snap_create 2021-03-24T22:05:17.477 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:17.535 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:17.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:17.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:05:17.641 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:17.701 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:17.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:17.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:05:17.811 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:17.871 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:17.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:17.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:05:17.985 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:18.039 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:18.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:18.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:05:18.145 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:18.206 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:18.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:18.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:05:18.313 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:18.367 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:18.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:18.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:05:18.466 INFO:tasks.rados.rados.0.smithi125.stdout:2881: done (0 left) 2021-03-24T22:05:18.467 INFO:tasks.rados.rados.0.smithi125.stdout:2882: rollback oid 15 current snap is 331 2021-03-24T22:05:18.467 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 15 to 319 2021-03-24T22:05:18.468 INFO:tasks.rados.rados.0.smithi125.stdout:2883: write (excl) oid 12 current snap is 331 2021-03-24T22:05:18.468 INFO:tasks.rados.rados.0.smithi125.stdout:2883: seq_num 1001 ranges {423807=566492,1676240=407685,2739004=84803} 2021-03-24T22:05:18.469 INFO:tasks.rados.rados.0.smithi125.stdout:2883: writing smithi12556470-12 from 423807 to 990299 tid 1 2021-03-24T22:05:18.474 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:18.475 INFO:tasks.rados.rados.0.smithi125.stdout:2883: writing smithi12556470-12 from 1676240 to 2083925 tid 2 2021-03-24T22:05:18.482 INFO:tasks.rados.rados.0.smithi125.stdout:2883: writing smithi12556470-12 from 2739004 to 2823807 tid 3 2021-03-24T22:05:18.483 INFO:tasks.rados.rados.0.smithi125.stdout:2884: write oid 45 current snap is 331 2021-03-24T22:05:18.483 INFO:tasks.rados.rados.0.smithi125.stdout:2882: finishing rollback tid 1 to smithi12556470-15 2021-03-24T22:05:18.483 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 15 v 1025 (ObjNum 1118634736 snap 21916 seq_num 0) dirty dne 2021-03-24T22:05:18.483 INFO:tasks.rados.rados.0.smithi125.stdout:2884: seq_num 1002 ranges {440614=641741,1734722=551343} 2021-03-24T22:05:18.485 INFO:tasks.rados.rados.0.smithi125.stdout:2884: writing smithi12556470-45 from 440614 to 1082355 tid 1 2021-03-24T22:05:18.493 INFO:tasks.rados.rados.0.smithi125.stdout:2884: writing smithi12556470-45 from 1734722 to 2286065 tid 2 2021-03-24T22:05:18.494 INFO:tasks.rados.rados.0.smithi125.stdout:2882: done (2 left) 2021-03-24T22:05:18.494 INFO:tasks.rados.rados.0.smithi125.stdout:2885: read oid 26 snap -1 2021-03-24T22:05:18.494 INFO:tasks.rados.rados.0.smithi125.stdout:2885: expect (ObjNum 978 snap 319 seq_num 978) 2021-03-24T22:05:18.494 INFO:tasks.rados.rados.0.smithi125.stdout:2886: copy_from oid 5 from oid 7 current snap is 331 2021-03-24T22:05:18.495 INFO:tasks.rados.rados.0.smithi125.stdout:2887: delete oid 34 current snap is 331 2021-03-24T22:05:18.497 INFO:tasks.rados.rados.0.smithi125.stdout:2887: done (4 left) 2021-03-24T22:05:18.497 INFO:tasks.rados.rados.0.smithi125.stdout:2888: write oid 4 current snap is 331 2021-03-24T22:05:18.497 INFO:tasks.rados.rados.0.smithi125.stdout:2888: seq_num 1004 ranges {474228=508592,1451686=555012,2704920=169308} 2021-03-24T22:05:18.504 INFO:tasks.rados.rados.0.smithi125.stdout:2888: writing smithi12556470-4 from 474228 to 982820 tid 1 2021-03-24T22:05:18.510 INFO:tasks.rados.rados.0.smithi125.stdout:2888: writing smithi12556470-4 from 1451686 to 2006698 tid 2 2021-03-24T22:05:18.516 INFO:tasks.rados.rados.0.smithi125.stdout:2888: writing smithi12556470-4 from 2704920 to 2874228 tid 3 2021-03-24T22:05:18.517 INFO:tasks.rados.rados.0.smithi125.stdout:2889: read oid 48 snap 301 2021-03-24T22:05:18.517 INFO:tasks.rados.rados.0.smithi125.stdout:2889: expect (ObjNum 900 snap 298 seq_num 900) 2021-03-24T22:05:18.517 INFO:tasks.rados.rados.0.smithi125.stdout:2890: read oid 1 snap 293 2021-03-24T22:05:18.517 INFO:tasks.rados.rados.0.smithi125.stdout:2890: expect (ObjNum 808 snap 265 seq_num 808) 2021-03-24T22:05:18.518 INFO:tasks.rados.rados.0.smithi125.stdout:2891: snap_create 2021-03-24T22:05:18.518 INFO:tasks.rados.rados.0.smithi125.stdout:2884: finishing write tid 1 to smithi12556470-45 2021-03-24T22:05:18.518 INFO:tasks.rados.rados.0.smithi125.stdout:2884: finishing write tid 2 to smithi12556470-45 2021-03-24T22:05:18.518 INFO:tasks.rados.rados.0.smithi125.stdout:2884: finishing write tid 3 to smithi12556470-45 2021-03-24T22:05:18.519 INFO:tasks.rados.rados.0.smithi125.stdout:2884: finishing write tid 4 to smithi12556470-45 2021-03-24T22:05:18.519 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 45 v 735 (ObjNum 1002 snap 331 seq_num 1002) dirty exists 2021-03-24T22:05:18.519 INFO:tasks.rados.rados.0.smithi125.stdout:2884: left oid 45 (ObjNum 1002 snap 331 seq_num 1002) 2021-03-24T22:05:18.519 INFO:tasks.rados.rados.0.smithi125.stdout:2885: expect (ObjNum 978 snap 319 seq_num 978) 2021-03-24T22:05:18.526 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:18.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:18.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:05:18.600 INFO:tasks.rados.rados.0.smithi125.stdout:2883: finishing write tid 1 to smithi12556470-12 2021-03-24T22:05:18.600 INFO:tasks.rados.rados.0.smithi125.stdout:2883: finishing write tid 2 to smithi12556470-12 2021-03-24T22:05:18.601 INFO:tasks.rados.rados.0.smithi125.stdout:2888: finishing write tid 1 to smithi12556470-4 2021-03-24T22:05:18.601 INFO:tasks.rados.rados.0.smithi125.stdout:2886: finishing copy_from to smithi12556470-5 2021-03-24T22:05:18.601 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 5 v 1126 (ObjNum 996 snap 329 seq_num 996) dirty exists 2021-03-24T22:05:18.601 INFO:tasks.rados.rados.0.smithi125.stdout:2886: finishing copy_from racing read to smithi12556470-5 2021-03-24T22:05:18.602 INFO:tasks.rados.rados.0.smithi125.stdout:2883: finishing write tid 3 to smithi12556470-12 2021-03-24T22:05:18.602 INFO:tasks.rados.rados.0.smithi125.stdout:2883: finishing write tid 4 to smithi12556470-12 2021-03-24T22:05:18.602 INFO:tasks.rados.rados.0.smithi125.stdout:2883: finishing write tid 5 to smithi12556470-12 2021-03-24T22:05:18.603 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 12 v 697 (ObjNum 1001 snap 331 seq_num 1001) dirty exists 2021-03-24T22:05:18.603 INFO:tasks.rados.rados.0.smithi125.stdout:2883: left oid 12 (ObjNum 1001 snap 331 seq_num 1001) 2021-03-24T22:05:18.603 INFO:tasks.rados.rados.0.smithi125.stdout:2888: finishing write tid 2 to smithi12556470-4 2021-03-24T22:05:18.603 INFO:tasks.rados.rados.0.smithi125.stdout:2889: expect (ObjNum 900 snap 298 seq_num 900) 2021-03-24T22:05:18.633 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:18.652 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:18 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:05:18.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:18 smithi125 conmon[19936]: 05:16.978454+0000 mgr.y (mgr.14144) 648 : cluster [DBG] pgmap v1282: 168 pgs: 15 active+clean+wait, 153 active+clean; 569 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 4.7 MiB/s wr, 22 op/s 2021-03-24T22:05:18.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:18 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:05:18.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:18 smithi125 conmon[19936]: 03-24T22:05:17.444838+0000 mon.a (mon.0) 1932 : cluster 2021-03-24T22:05:18.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:18 smithi125 conmon[19936]: [DBG] osdmap e747: 8 total, 8 up, 8 in 2021-03-24T22:05:18.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:18 smithi125 conmon[24927]: cluster 2021-03-24T22 2021-03-24T22:05:18.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:18 smithi125 conmon[24927]: :05:16.978454+0000 mgr.y (mgr.14144) 648 : cluster [DBG] pgmap v1282: 168 pgs: 15 active+clean+wait, 153 active+clean; 569 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 4.7 MiB/s wr, 22 op/s 2021-03-24T22:05:18.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:18 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T22:05:18.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:18 smithi125 conmon[24927]: T22:05:17.444838+0000 mon.a (mon 2021-03-24T22:05:18.714 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:18 smithi125 conmon[24927]: .0) 1932 : cluster [DBG] osdmap e747: 8 total, 8 up, 8 in 2021-03-24T22:05:18.715 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:18.716 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:18 smithi142 conmon[20125]: cluster 2021-03-24T22:05:16 2021-03-24T22:05:18.716 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:18 smithi142 conmon[20125]: .978454+0000 mgr.y (mgr.14144) 648 : cluster [DBG] pgmap v1282: 168 pgs: 15 active+clean+wait, 153 active+clean; 569 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 4.7 MiB/s wr, 22 op/s 2021-03-24T22:05:18.717 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:18 smithi142 conmon[20125]: cluster 2021-03-24T22:05 2021-03-24T22:05:18.717 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:18 smithi142 conmon[20125]: :17.444838+0000 mon.a (mon.0) 1932 : cluster [DBG] osdmap e747: 8 total, 8 up, 8 in 2021-03-24T22:05:18.718 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T22:05:18.718 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T22:05:18.718 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T22:05:18.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:18.719 INFO:tasks.rados.rados.0.smithi125.stdout:2888: finishing write tid 3 to smithi12556470-4 2021-03-24T22:05:18.719 INFO:tasks.rados.rados.0.smithi125.stdout:2888: finishing write tid 4 to smithi12556470-4 2021-03-24T22:05:18.719 INFO:tasks.rados.rados.0.smithi125.stdout:2888: finishing write tid 5 to smithi12556470-4 2021-03-24T22:05:18.720 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 4 v 672 (ObjNum 1004 snap 331 seq_num 1004) dirty exists 2021-03-24T22:05:18.720 INFO:tasks.rados.rados.0.smithi125.stdout:2888: left oid 4 (ObjNum 1004 snap 331 seq_num 1004) 2021-03-24T22:05:18.720 INFO:tasks.rados.rados.0.smithi125.stdout:2890: expect (ObjNum 808 snap 265 seq_num 808) 2021-03-24T22:05:18.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:05:18.729 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T22:05:18.806 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T22:05:18.807 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T22:05:18.808 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T22:05:18.822 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:18.832 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T22:05:18.835 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-24T22:05:18.835 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-24T22:05:18.835 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-24T22:05:18.836 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-24T22:05:18.836 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-24T22:05:18.837 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-24T22:05:18.837 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-24T22:05:18.837 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-24T22:05:18.874 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:18.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:18.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:05:18.979 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:19.038 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:19.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:19.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:05:19.145 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:19.208 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:19.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:19.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:05:19.314 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:19.373 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:19.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:19.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:05:19.472 INFO:tasks.rados.rados.0.smithi125.stdout:2883: done (7 left) 2021-03-24T22:05:19.473 INFO:tasks.rados.rados.0.smithi125.stdout:2884: done (6 left) 2021-03-24T22:05:19.473 INFO:tasks.rados.rados.0.smithi125.stdout:2885: done (5 left) 2021-03-24T22:05:19.473 INFO:tasks.rados.rados.0.smithi125.stdout:2886: done (4 left) 2021-03-24T22:05:19.474 INFO:tasks.rados.rados.0.smithi125.stdout:2888: done (3 left) 2021-03-24T22:05:19.474 INFO:tasks.rados.rados.0.smithi125.stdout:2889: done (2 left) 2021-03-24T22:05:19.474 INFO:tasks.rados.rados.0.smithi125.stdout:2890: done (1 left) 2021-03-24T22:05:19.474 INFO:tasks.rados.rados.0.smithi125.stdout:2891: done (0 left) 2021-03-24T22:05:19.474 INFO:tasks.rados.rados.0.smithi125.stdout:2892: read oid 21 snap -1 2021-03-24T22:05:19.475 INFO:tasks.rados.rados.0.smithi125.stdout:2892: expect (ObjNum 851 snap 278 seq_num 851) 2021-03-24T22:05:19.475 INFO:tasks.rados.rados.0.smithi125.stdout:2893: snap_create 2021-03-24T22:05:19.477 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:19.488 INFO:tasks.rados.rados.0.smithi125.stdout:2892: expect (ObjNum 851 snap 278 seq_num 851) 2021-03-24T22:05:19.527 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:19.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:19.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:05:19.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:19 smithi142 conmon[20125]: cluster 2021-03-24T22: 2021-03-24T22:05:19.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:19 smithi142 conmon[20125]: 05:18.455827+0000 mon.a (mon.0) 1933 : cluster [DBG] osdmap e748: 8 total, 8 up, 8 in 2021-03-24T22:05:19.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:19 smithi142 conmon[20125]: audit 2021-03-24T22:05:18 2021-03-24T22:05:19.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:19 smithi142 conmon[20125]: .635756+0000 mon.c (mon.2) 108 : audit [DBG] from='client.? v1:172.21.15.125:0/4167008101' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:05:19.632 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:19.680 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:19.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:19.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:05:19.787 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:19.849 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:19.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:19.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:05:19.866 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:19 smithi125 conmon[19936]: cluster 2021-03-24T22:05:18. 2021-03-24T22:05:19.867 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:19 smithi125 conmon[19936]: 455827+0000 mon.a (mon.0) 1933 : cluster [DBG] osdmap e748: 8 total, 8 up, 8 in 2021-03-24T22:05:19.867 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:19 smithi125 conmon[19936]: audit 2021-03-24T22:05:18.635756+0000 mon.c (mon.2) 108 : audit [DBG] from='client.? v1:172.21.15.125:0/4167008101' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:05:19.867 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:19 smithi125 conmon[19936]: 2021-03-24T22:05:19.868 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:19 smithi125 conmon[24927]: cluster 2021-03-24T22:05:19.868 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:19 smithi125 conmon[24927]: 2021-03-24T22:05:19.868 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:19 smithi125 conmon[24927]: 2021 2021-03-24T22:05:19.868 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:19 smithi125 conmon[24927]: - 2021-03-24T22:05:19.869 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:19 smithi125 conmon[24927]: 03 2021-03-24T22:05:19.869 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:19 smithi125 conmon[24927]: - 2021-03-24T22:05:19.869 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:19 smithi125 conmon[24927]: 24 2021-03-24T22:05:19.869 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:19 smithi125 conmon[24927]: T 2021-03-24T22:05:19.870 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:19 smithi125 conmon[24927]: 22 2021-03-24T22:05:19.870 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:19 smithi125 conmon[24927]: : 2021-03-24T22:05:19.870 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:19 smithi125 conmon[24927]: 05 2021-03-24T22:05:19.870 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:19 smithi125 conmon[24927]: :18.455827+0000 mon.a (mon.0) 1933 : cluster [DBG] osdmap e748: 8 total, 8 up, 8 in 2021-03-24T22:05:19.870 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:19 smithi125 conmon[24927]: 2021-03-24T22:05:19.871 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:19 smithi125 conmon[24927]: audit 2021-03-24T22:05:18.635756+0000 mon.c (mon.2) 108 : audit [DBG] from='client.? v1:172.21.15.125:0/4167008101' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:05:19.956 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:20.017 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:20.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:20.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:05:20.113 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-24T22:05:20.123 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:20.187 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:20.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:20.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:05:20.298 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:20.356 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:20.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:20.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:05:20.397 INFO:tasks.rados.rados.0.smithi125.stdout:2892: done (1 left) 2021-03-24T22:05:20.397 INFO:tasks.rados.rados.0.smithi125.stdout:2893: done (0 left) 2021-03-24T22:05:20.398 INFO:tasks.rados.rados.0.smithi125.stdout:2894: delete oid 4 current snap is 333 2021-03-24T22:05:20.408 INFO:tasks.rados.rados.0.smithi125.stdout:2894: done (0 left) 2021-03-24T22:05:20.409 INFO:tasks.rados.rados.0.smithi125.stdout:2895: read oid 45 snap -1 2021-03-24T22:05:20.409 INFO:tasks.rados.rados.0.smithi125.stdout:2895: expect (ObjNum 1002 snap 331 seq_num 1002) 2021-03-24T22:05:20.409 INFO:tasks.rados.rados.0.smithi125.stdout:2896: write oid 50 current snap is 333 2021-03-24T22:05:20.409 INFO:tasks.rados.rados.0.smithi125.stdout:2896: seq_num 1006 ranges {507842=659090,1568650=558681,2759766=148076} 2021-03-24T22:05:20.410 INFO:teuthology.orchestra.run.smithi125.stderr:reweighted osd.2 to 1 (10000) 2021-03-24T22:05:20.415 INFO:tasks.rados.rados.0.smithi125.stdout:2896: writing smithi12556470-50 from 507842 to 1166932 tid 1 2021-03-24T22:05:20.421 INFO:tasks.rados.rados.0.smithi125.stdout:2896: writing smithi12556470-50 from 1568650 to 2127331 tid 2 2021-03-24T22:05:20.425 INFO:tasks.rados.rados.0.smithi125.stdout:2896: writing smithi12556470-50 from 2759766 to 2907842 tid 3 2021-03-24T22:05:20.426 INFO:tasks.rados.rados.0.smithi125.stdout:2895: expect (ObjNum 1002 snap 331 seq_num 1002) 2021-03-24T22:05:20.462 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:20.516 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:20.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:20.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:05:20.533 INFO:tasks.rados.rados.0.smithi125.stdout:2896: finishing write tid 1 to smithi12556470-50 2021-03-24T22:05:20.534 INFO:tasks.rados.rados.0.smithi125.stdout:2896: finishing write tid 2 to smithi12556470-50 2021-03-24T22:05:20.534 INFO:tasks.rados.rados.0.smithi125.stdout:2896: finishing write tid 3 to smithi12556470-50 2021-03-24T22:05:20.534 INFO:tasks.rados.rados.0.smithi125.stdout:2896: finishing write tid 4 to smithi12556470-50 2021-03-24T22:05:20.535 INFO:tasks.rados.rados.0.smithi125.stdout:2896: finishing write tid 5 to smithi12556470-50 2021-03-24T22:05:20.535 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 50 v 883 (ObjNum 1006 snap 333 seq_num 1006) dirty exists 2021-03-24T22:05:20.535 INFO:tasks.rados.rados.0.smithi125.stdout:2896: left oid 50 (ObjNum 1006 snap 333 seq_num 1006) 2021-03-24T22:05:20.536 INFO:tasks.rados.rados.0.smithi125.stdout:2895: done (1 left) 2021-03-24T22:05:20.536 INFO:tasks.rados.rados.0.smithi125.stdout:2896: done (0 left) 2021-03-24T22:05:20.536 INFO:tasks.rados.rados.0.smithi125.stdout:2897: read oid 41 snap -1 2021-03-24T22:05:20.536 INFO:tasks.rados.rados.0.smithi125.stdout:2897: expect (ObjNum 992 snap 328 seq_num 992) 2021-03-24T22:05:20.537 INFO:tasks.rados.rados.0.smithi125.stdout:2898: snap_remove snap 332 2021-03-24T22:05:20.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:20 smithi142 conmon[20125]: cluster 2021-03-24T22:05:18.810234+0000 osd.3 (osd 2021-03-24T22:05:20.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:20 smithi142 conmon[20125]: .3) 151 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:05:20.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:20 smithi142 conmon[20125]: cluster 2021-03-24T22:05:18.814469+0000 osd.3 (osd.3) 152 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:05:20.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:20 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:05:20.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:20 smithi142 conmon[20125]: -24T22:05:18.979210+0000 mgr.y (mgr.14144) 649 : cluster [DBG] pgmap v1285: 168 pgs: 168 active+clean; 569 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 5.5 MiB/s rd, 1.1 MiB/s wr, 9 op/s 2021-03-24T22:05:20.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:20 smithi142 conmon[20125]: cluster 2021-03-24T22:05: 2021-03-24T22:05:20.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:20 smithi142 conmon[20125]: 19.465155+0000 mon.a (mon.0) 1934 : cluster [DBG] 2021-03-24T22:05:20.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:20 smithi142 conmon[20125]: osdmap e749: 8 total, 8 up, 8 in 2021-03-24T22:05:20.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:20 smithi142 conmon[20125]: audit 2021-03 2021-03-24T22:05:20.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:20 smithi142 conmon[20125]: -24T22:05:20.247277+0000 mon.c (mon 2021-03-24T22:05:20.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:20 smithi142 conmon[20125]: .2) 109 : audit [INF] from='client.? v1:172.21.15.125:0/392945891' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2021-03-24T22:05:20.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:20 smithi142 conmon[20125]: audit 2021 2021-03-24T22:05:20.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:20 smithi142 conmon[20125]: -03-24T22:05:20.247740+0000 mon.a ( 2021-03-24T22:05:20.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:20 smithi142 conmon[20125]: mon.0) 1935 : audit [INF] from='client.35759 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2021-03-24T22:05:20.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:20 smithi142 conmon[20125]: audit 2021-03-24T22:05:20.625 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:20 smithi142 conmon[20125]: 2021-03-24T22:05:20.389877+0000 mon.a 2021-03-24T22:05:20.625 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:20 smithi142 conmon[20125]: (mon.0) 1936 : audit [INF] from='client.35759 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2021-03-24T22:05:20.626 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:20 smithi142 conmon[20125]: cluster 2021-03-24T22:05:20.626 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:20 smithi142 conmon[20125]: 2021-03-24T22:05:20.389969+0000 mon.a 2021-03-24T22:05:20.627 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:20 smithi142 conmon[20125]: (mon.0) 1937 : cluster [DBG] osdmap e750: 8 total, 8 up, 8 in 2021-03-24T22:05:20.628 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:20.688 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:20.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:20.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:05:20.795 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:20.859 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:20.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:20.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:05:20.877 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:20 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T22:05:20.878 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:20 smithi125 conmon[19936]: 22:05:18.810234+0000 osd.3 (osd.3) 151 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:05:20.878 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:20 smithi125 conmon[19936]: cluster 2021-03-24T22:05:18 2021-03-24T22:05:20.878 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:20 smithi125 conmon[19936]: .814469+0000 osd.3 (osd.3) 152 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:05:20.878 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:20 smithi125 conmon[19936]: cluster 2021-03-24T22:05:18 2021-03-24T22:05:20.879 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:20 smithi125 conmon[19936]: .979210+0000 mgr.y (mgr.14144) 649 : 2021-03-24T22:05:20.879 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:20 smithi125 conmon[19936]: cluster [DBG] pgmap v1285: 168 pgs: 168 active+clean; 569 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 5.5 MiB/s rd, 1.1 MiB/s wr, 9 op/s 2021-03-24T22:05:20.879 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:20 smithi125 conmon[19936]: cluster 2021-03-24T22:05 2021-03-24T22:05:20.879 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:20 smithi125 conmon[19936]: :19.465155+0000 mon.a (mon.0) 1934 2021-03-24T22:05:20.879 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:20 smithi125 conmon[19936]: : cluster [DBG] osdmap e749: 8 total, 8 up, 8 in 2021-03-24T22:05:20.880 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:20 smithi125 conmon[19936]: audit 2021-03-24T22:05:20 2021-03-24T22:05:20.880 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:20 smithi125 conmon[19936]: .247277+0000 mon.c (mon.2) 109 : audit 2021-03-24T22:05:20.880 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:20 smithi125 conmon[19936]: [INF] from='client.? v1:172.21.15.125:0/392945891' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2021-03-24T22:05:20.880 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:20 smithi125 conmon[19936]: audit 2021-03-24T22:05 2021-03-24T22:05:20.881 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:20 smithi125 conmon[19936]: :20.247740+0000 mon.a (mon.0) 1935 2021-03-24T22:05:20.881 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:20 smithi125 conmon[19936]: : audit [INF] from='client.35759 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2021-03-24T22:05:20.881 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:20 smithi125 conmon[19936]: audit 2021- 2021-03-24T22:05:20.881 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:20 smithi125 conmon[19936]: 03-24T22:05:20.389877+0000 2021-03-24T22:05:20.881 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:20 smithi125 conmon[19936]: mon.a (mon.0) 1936 : audit [INF] 2021-03-24T22:05:20.882 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:20 smithi125 conmon[19936]: from='client.35759 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2021-03-24T22:05:20.882 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:20 smithi125 conmon[19936]: cluster 2021-03-24T22:05 2021-03-24T22:05:20.882 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:20 smithi125 conmon[19936]: :20.389969+0000 mon.a (mon.0) 1937 2021-03-24T22:05:20.882 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:20 smithi125 conmon[19936]: : cluster [DBG] osdmap e750: 8 total, 8 up, 8 in 2021-03-24T22:05:20.883 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:20 smithi125 conmon[24927]: cluster 2021-03-24T22:05:20.883 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:20 smithi125 conmon[24927]: 2021-03-24T22:05:18.810234 2021-03-24T22:05:20.883 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:20 smithi125 conmon[24927]: +0000 osd.3 (osd.3) 151 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:05:20.884 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:20 smithi125 conmon[24927]: cluster 2021-03-24T22:05:18.814469+0000 2021-03-24T22:05:20.884 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:20 smithi125 conmon[24927]: osd.3 (osd.3) 152 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:05:20.884 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:20 smithi125 conmon[24927]: cluster 2021-03-24T22:05:18.979210+0000 mgr.y ( 2021-03-24T22:05:20.884 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:20 smithi125 conmon[24927]: mgr.14144) 649 : cluster [DBG] pgmap v1285: 168 pgs: 168 active+clean; 569 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 5.5 MiB/s rd, 1.1 MiB/s wr, 9 op/s 2021-03-24T22:05:20.885 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:20 smithi125 conmon[24927]: cluster 2021-03-24T22:05:19.465155 2021-03-24T22:05:20.885 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:20 smithi125 conmon[24927]: +0000 mon.a (mon.0) 1934 : cluster [DBG] osdmap e749: 8 total, 8 up, 8 in 2021-03-24T22:05:20.885 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:20 smithi125 conmon[24927]: audit 2021-03-24T22:05: 2021-03-24T22:05:20.886 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:20 smithi125 conmon[24927]: 20.247277+0000 mon.c (mon.2) 109 : audit [INF] from='client.? v1:172.21.15.125:0/392945891' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2021-03-24T22:05:20.886 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:20 smithi125 conmon[24927]: audit 2021- 2021-03-24T22:05:20.886 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:20 smithi125 conmon[24927]: 03-24T22:05:20.247740+0000 mon.a (mon.0) 1935 : audit [INF] from='client.35759 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2021-03-24T22:05:20.886 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:20 smithi125 conmon[24927]: audit 2021-03-24 2021-03-24T22:05:20.886 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:20 smithi125 conmon[24927]: T22:05:20.389877+0000 mon.a (mon.0) 1936 : audit [INF] from='client.35759 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2021-03-24T22:05:20.887 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:20 smithi125 conmon[24927]: cluster 2021-03-24T22:05: 2021-03-24T22:05:20.887 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:20 smithi125 conmon[24927]: 20.389969+0000 mon.a (mon.0) 1937 : cluster [DBG] osdmap e750: 8 total, 8 up, 8 in 2021-03-24T22:05:20.963 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:20.986 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:05:21.020 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:21.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:21.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:05:21.125 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:21.175 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:21.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:21.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:05:21.281 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:21.337 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:21.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:21.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:05:21.445 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:21.455 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-24T22:05:21.455 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-24T22:05:21.456 INFO:teuthology.orchestra.run.smithi125.stdout: "complaint_time": 30, 2021-03-24T22:05:21.456 INFO:teuthology.orchestra.run.smithi125.stdout: "num_blocked_ops": 0 2021-03-24T22:05:21.456 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-24T22:05:21.498 INFO:tasks.rados.rados.0.smithi125.stdout:2898: done (1 left) 2021-03-24T22:05:21.499 INFO:tasks.rados.rados.0.smithi125.stdout:2899: snap_create 2021-03-24T22:05:21.499 INFO:tasks.rados.rados.0.smithi125.stdout:2897: expect (ObjNum 992 snap 328 seq_num 992) 2021-03-24T22:05:21.502 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:21.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:21.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:05:21.607 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:21.656 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:21.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:21.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:05:21.763 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:21.822 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:21.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:21.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:05:21.928 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:21.990 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:21.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:21.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:05:22.096 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:22.152 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:22.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:22.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:05:22.259 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:22.317 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:22.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:22.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:05:22.425 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:22.486 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:22.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:22.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:05:22.497 INFO:tasks.rados.rados.0.smithi125.stdout:2897: done (1 left) 2021-03-24T22:05:22.498 INFO:tasks.rados.rados.0.smithi125.stdout:2899: done (0 left) 2021-03-24T22:05:22.498 INFO:tasks.rados.rados.0.smithi125.stdout:2900: rollback oid 28 current snap is 334 2021-03-24T22:05:22.499 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 28 to 325 2021-03-24T22:05:22.499 INFO:tasks.rados.rados.0.smithi125.stdout:2901: snap_create 2021-03-24T22:05:22.502 INFO:tasks.rados.rados.0.smithi125.stdout:2900: finishing rollback tid 1 to smithi12556470-28 2021-03-24T22:05:22.502 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 28 v 1199 (ObjNum 1117626448 snap 21916 seq_num 1116443088) dirty dne 2021-03-24T22:05:22.592 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:22.652 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:22.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:22.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:05:22.763 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:22.777 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:22 smithi142 conmon[20125]: cluster 2021-03-24T22:05: 2021-03-24T22:05:22.777 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:22 smithi142 conmon[20125]: 20.980070+0000 mgr.y (mgr.14144) 650 : cluster [DBG] pgmap v1288: 168 pgs: 168 active+clean; 554 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 1.8 MiB/s wr, 17 op/s 2021-03-24T22:05:22.778 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:22 smithi142 conmon[20125]: cluster 2021-03-24T22:05:21 2021-03-24T22:05:22.778 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:22 smithi142 conmon[20125]: .491777+0000 mon.a (mon.0) 1938 : cluster [DBG] osdmap e751: 8 total, 8 up, 8 in 2021-03-24T22:05:22.823 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:22.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:22.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:05:22.908 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-24T22:05:22.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:22 smithi125 conmon[19936]: cluster 2021-03-24T22:05:20 2021-03-24T22:05:22.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:22 smithi125 conmon[19936]: .980070+0000 mgr.y (mgr.14144) 650 : cluster [DBG] pgmap v1288: 168 pgs: 168 active+clean; 554 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 1.8 MiB/s wr, 17 op/s 2021-03-24T22:05:22.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:22 smithi125 conmon[19936]: cluster 2021-03-24T22:05 2021-03-24T22:05:22.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:22 smithi125 conmon[19936]: :21.491777+0000 mon.a (mon.0) 1938 : cluster 2021-03-24T22:05:22.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:22 smithi125 conmon[19936]: [DBG] osdmap e751: 8 total, 8 up, 8 in 2021-03-24T22:05:22.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:22 smithi125 conmon[24927]: cluster 2021-03-24T22:05:20.980070+0000 2021-03-24T22:05:22.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:22 smithi125 conmon[24927]: mgr.y (mgr.14144) 650 : cluster [DBG] pgmap v1288: 168 pgs: 168 active+clean; 554 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 1.8 MiB/s wr, 17 op/s 2021-03-24T22:05:22.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:22 smithi125 conmon[24927]: cluster 2021-03-24T22:05:21.491777+0000 2021-03-24T22:05:22.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:22 smithi125 conmon[24927]: mon.a (mon.0) 1938 : cluster [DBG] osdmap e751: 8 total, 8 up, 8 in 2021-03-24T22:05:22.932 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:22.993 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:22.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:22.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:05:23.100 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:23.153 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:23.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:23.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:05:23.259 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:23.321 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:23.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:23.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:05:23.427 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:23.464 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd reweight 3 1 2021-03-24T22:05:23.476 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:23.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:23.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:05:23.490 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.0 dump_historic_ops 2021-03-24T22:05:23.499 INFO:tasks.rados.rados.0.smithi125.stdout:2900: done (1 left) 2021-03-24T22:05:23.499 INFO:tasks.rados.rados.0.smithi125.stdout:2901: done (0 left) 2021-03-24T22:05:23.500 INFO:tasks.rados.rados.0.smithi125.stdout:2902: delete oid 34 current snap is 335 2021-03-24T22:05:23.508 INFO:tasks.rados.rados.0.smithi125.stdout:2902: done (0 left) 2021-03-24T22:05:23.509 INFO:tasks.rados.rados.0.smithi125.stdout:2903: read oid 10 snap 329 2021-03-24T22:05:23.509 INFO:tasks.rados.rados.0.smithi125.stdout:2903: expect deleted 2021-03-24T22:05:23.509 INFO:tasks.rados.rados.0.smithi125.stdout:2904: snap_remove snap 325 2021-03-24T22:05:23.512 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:23 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:05:23.513 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:23 smithi125 conmon[24927]: 05:22.492087+0000 mon.a (mon.0) 1939 : cluster [DBG] osdmap e752: 8 total, 8 up, 8 in 2021-03-24T22:05:23.513 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:23 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:05:23.513 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:23 smithi125 conmon[24927]: 24T22:05:22.683174+0000 osd.6 ( 2021-03-24T22:05:23.513 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:23 smithi125 conmon[24927]: osd.6) 73 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:05:23.514 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:23 smithi125 conmon[24927]: cluster 2021-03-24T22:05:23.514 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:23 smithi125 conmon[24927]: 2021-03-24T22:05:22.687022+0000 2021-03-24T22:05:23.514 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:23 smithi125 conmon[24927]: osd.6 (osd.6) 74 : cluster [DBG] 2021-03-24T22:05:23.514 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:23 smithi125 conmon[24927]: purged_snaps scrub ok 2021-03-24T22:05:23.515 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:23 smithi125 conmon[24927]: cluster 2021-03-24T22:05: 2021-03-24T22:05:23.515 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:23 smithi125 conmon[24927]: 23.494719+0000 mon.a (mon.0) 1940 : 2021-03-24T22:05:23.515 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:23 smithi125 conmon[19936]: cluster 2021-03-24T22:05:22.492087+0000 mon.a 2021-03-24T22:05:23.516 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:23 smithi125 conmon[19936]: (mon.0) 1939 : cluster [DBG] osdmap e752: 8 total, 8 up, 8 in 2021-03-24T22:05:23.516 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:23 smithi125 conmon[19936]: cluster 2021-03-24T22:05:22.683174+0000 osd.6 (osd.6) 73 2021-03-24T22:05:23.516 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:23 smithi125 conmon[19936]: : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:05:23.516 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:23 smithi125 conmon[19936]: cluster 2021-03-24T22:05:23.517 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:23 smithi125 conmon[19936]: 2021-03-24T22:05:22 2021-03-24T22:05:23.517 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:23 smithi125 conmon[19936]: .687022+0000 osd.6 (osd.6) 74 : cluster 2021-03-24T22:05:23.517 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:23 smithi125 conmon[19936]: [DBG] purged_snaps scrub ok 2021-03-24T22:05:23.517 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:23 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:05:23.518 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:23 smithi125 conmon[19936]: 03-24T22:05:23.494719+0000 mon.a 2021-03-24T22:05:23.518 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:23 smithi125 conmon[19936]: (mon.0) 1940 : cluster [DBG] osdmap e753: 8 total, 8 up, 8 in 2021-03-24T22:05:23.584 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:23.612 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:05:23.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:23 smithi142 conmon[20125]: cluster 2021-03-24T22:05:23.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:23 smithi142 conmon[20125]: 2021-03-24T22:05:22 2021-03-24T22:05:23.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:23 smithi142 conmon[20125]: .492087+0000 mon.a (mon.0) 1939 : cluster [DBG] osdmap e752: 8 total, 8 up, 8 in 2021-03-24T22:05:23.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:23 smithi142 conmon[20125]: cluster 2021-03-24T22:05:22.683174+0000 osd.6 (osd. 2021-03-24T22:05:23.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:23 smithi142 conmon[20125]: 6) 73 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:05:23.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:23 smithi142 conmon[20125]: cluster 2021-03-24T22:05: 2021-03-24T22:05:23.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:23 smithi142 conmon[20125]: 22.687022+0000 osd.6 (osd.6) 74 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:05:23.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:23 smithi142 conmon[20125]: cluster 2021-03-24T22:05:23.494719+0000 mon.a (mon. 2021-03-24T22:05:23.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:23 smithi142 conmon[20125]: 0) 1940 : cluster [DBG] osdmap e753: 8 total, 8 up, 8 in 2021-03-24T22:05:23.642 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:23.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:23.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:05:23.752 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:23.764 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:23 smithi125 conmon[24927]: cluster [DBG] osdmap e753: 8 total, 8 up, 8 in 2021-03-24T22:05:23.800 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:23.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:23.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:05:23.906 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:23.960 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:23.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:23.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:05:24.066 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:24.123 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:24.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:24.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:05:24.230 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:24.291 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:24.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:24.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:05:24.396 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:24.451 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:24.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:24.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:05:24.530 INFO:tasks.rados.rados.0.smithi125.stdout:2904: done (1 left) 2021-03-24T22:05:24.531 INFO:tasks.rados.rados.0.smithi125.stdout:2905: write oid 19 current snap is 335 2021-03-24T22:05:24.531 INFO:tasks.rados.rados.0.smithi125.stdout:2905: seq_num 1008 ranges {541456=409588,1569261=562350,2814612=126844} 2021-03-24T22:05:24.531 INFO:tasks.rados.rados.0.smithi125.stdout:2905: writing smithi12556470-19 from 541456 to 951044 tid 1 2021-03-24T22:05:24.535 INFO:tasks.rados.rados.0.smithi125.stdout:2905: writing smithi12556470-19 from 1569261 to 2131611 tid 2 2021-03-24T22:05:24.541 INFO:tasks.rados.rados.0.smithi125.stdout:2905: writing smithi12556470-19 from 2814612 to 2941456 tid 3 2021-03-24T22:05:24.541 INFO:tasks.rados.rados.0.smithi125.stdout:2905: finishing write tid 1 to smithi12556470-19 2021-03-24T22:05:24.541 INFO:tasks.rados.rados.0.smithi125.stdout:2905: finishing write tid 2 to smithi12556470-19 2021-03-24T22:05:24.541 INFO:tasks.rados.rados.0.smithi125.stdout:2903: done (1 left) 2021-03-24T22:05:24.542 INFO:tasks.rados.rados.0.smithi125.stdout:2906: copy_from oid 1 from oid 25 current snap is 335 2021-03-24T22:05:24.542 INFO:tasks.rados.rados.0.smithi125.stdout:2907: write (excl) oid 31 current snap is 335 2021-03-24T22:05:24.542 INFO:tasks.rados.rados.0.smithi125.stdout:2907: seq_num 1009 ranges {558263=484837,1627743=706008} 2021-03-24T22:05:24.547 INFO:tasks.rados.rados.0.smithi125.stdout:2907: writing smithi12556470-31 from 558263 to 1043100 tid 1 2021-03-24T22:05:24.555 INFO:tasks.rados.rados.0.smithi125.stdout:2907: writing smithi12556470-31 from 1627743 to 2333751 tid 2 2021-03-24T22:05:24.556 INFO:tasks.rados.rados.0.smithi125.stdout:2908: delete oid 27 current snap is 335 2021-03-24T22:05:24.556 INFO:tasks.rados.rados.0.smithi125.stdout:2905: finishing write tid 3 to smithi12556470-19 2021-03-24T22:05:24.556 INFO:tasks.rados.rados.0.smithi125.stdout:2905: finishing write tid 4 to smithi12556470-19 2021-03-24T22:05:24.556 INFO:tasks.rados.rados.0.smithi125.stdout:2905: finishing write tid 5 to smithi12556470-19 2021-03-24T22:05:24.557 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 19 v 683 (ObjNum 1008 snap 335 seq_num 1008) dirty exists 2021-03-24T22:05:24.557 INFO:tasks.rados.rados.0.smithi125.stdout:2905: left oid 19 (ObjNum 1008 snap 335 seq_num 1008) 2021-03-24T22:05:24.557 INFO:tasks.rados.rados.0.smithi125.stdout:2907: finishing write tid 1 to smithi12556470-31 2021-03-24T22:05:24.557 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:24.558 INFO:tasks.rados.rados.0.smithi125.stdout:2905: done (3 left) 2021-03-24T22:05:24.559 INFO:tasks.rados.rados.0.smithi125.stdout:2908: done (2 left) 2021-03-24T22:05:24.559 INFO:tasks.rados.rados.0.smithi125.stdout:2909: snap_create 2021-03-24T22:05:24.561 INFO:tasks.rados.rados.0.smithi125.stdout:2907: finishing write tid 2 to smithi12556470-31 2021-03-24T22:05:24.563 INFO:tasks.rados.rados.0.smithi125.stdout:2907: finishing write tid 3 to smithi12556470-31 2021-03-24T22:05:24.565 INFO:tasks.rados.rados.0.smithi125.stdout:2906: finishing copy_from to smithi12556470-1 2021-03-24T22:05:24.565 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 1 v 822 (ObjNum 949 snap 314 seq_num 949) dirty exists 2021-03-24T22:05:24.566 INFO:tasks.rados.rados.0.smithi125.stdout:2906: finishing copy_from racing read to smithi12556470-1 2021-03-24T22:05:24.567 INFO:tasks.rados.rados.0.smithi125.stdout:2907: finishing write tid 4 to smithi12556470-31 2021-03-24T22:05:24.567 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 31 v 1784 (ObjNum 1009 snap 335 seq_num 1009) dirty exists 2021-03-24T22:05:24.567 INFO:tasks.rados.rados.0.smithi125.stdout:2907: left oid 31 (ObjNum 1009 snap 335 seq_num 1009) 2021-03-24T22:05:24.603 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:24.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:24.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:05:24.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:24 smithi125 conmon[24927]: cluster 2021-03-24T22:05:22 2021-03-24T22:05:24.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:24 smithi125 conmon[24927]: .980846+0000 mgr.y (mgr.14144) 651 : cluster [DBG] pgmap v1291: 168 pgs: 168 active+clean; 552 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 2.7 MiB/s wr, 22 op/s 2021-03-24T22:05:24.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:24 smithi125 conmon[19936]: cluster 2021-03-24T22:05:22.980846+0000 mgr.y 2021-03-24T22:05:24.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:24 smithi125 conmon[19936]: (mgr.14144) 651 : cluster [DBG] pgmap v1291: 168 pgs: 168 active+clean; 552 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 2.7 MiB/s wr, 22 op/s 2021-03-24T22:05:24.708 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:24.770 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:24.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:24.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:05:24.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:24 smithi142 conmon[20125]: cluster 2021-03-24T22:05:22.980846+0000 mgr.y (mgr. 2021-03-24T22:05:24.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:24 smithi142 conmon[20125]: 14144) 651 : cluster [DBG] pgmap v1291: 168 pgs: 168 active+clean; 552 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 2.7 MiB/s wr, 22 op/s 2021-03-24T22:05:24.876 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:24.942 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:24.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:24.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:05:24.965 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:05:25.047 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:25.100 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:05:25.101 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:25.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:25.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:05:25.206 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:25.261 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:25.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:25.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:05:25.367 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:25.405 INFO:tasks.rados.rados.0.smithi125.stdout:2906: done (2 left) 2021-03-24T22:05:25.405 INFO:tasks.rados.rados.0.smithi125.stdout:2907: done (1 left) 2021-03-24T22:05:25.405 INFO:tasks.rados.rados.0.smithi125.stdout:2909: done (0 left) 2021-03-24T22:05:25.406 INFO:tasks.rados.rados.0.smithi125.stdout:2910: delete oid 13 current snap is 336 2021-03-24T22:05:25.412 INFO:tasks.rados.rados.0.smithi125.stdout:2910: done (0 left) 2021-03-24T22:05:25.412 INFO:tasks.rados.rados.0.smithi125.stdout:2911: read oid 48 snap -1 2021-03-24T22:05:25.413 INFO:tasks.rados.rados.0.smithi125.stdout:2911: expect (ObjNum 984 snap 326 seq_num 984) 2021-03-24T22:05:25.413 INFO:tasks.rados.rados.0.smithi125.stdout:2912: snap_remove snap 302 2021-03-24T22:05:25.432 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:25.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:25.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:05:25.536 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:25.589 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:25.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:25.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:05:25.694 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:25.753 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:25.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:25.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:05:25.860 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:25.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:25 smithi142 conmon[20125]: cluster 2021-03-24T22:05: 2021-03-24T22:05:25.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:25 smithi142 conmon[20125]: 24.515861+0000 mon.a (mon.0) 1941 : cluster [DBG] osdmap e754: 8 total, 8 up, 8 in 2021-03-24T22:05:25.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:25 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:05:25.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:25 smithi142 conmon[20125]: T22:05:25.395629+0000 mon.a (mon.0) 1942 : cluster 2021-03-24T22:05:25.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:25 smithi142 conmon[20125]: [DBG] osdmap e755: 8 total, 8 up, 8 in 2021-03-24T22:05:25.874 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:25 smithi125 conmon[19936]: cluster 2021-03-24T22:05:24.515861+0000 mon.a (mon.0) 1941 2021-03-24T22:05:25.875 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:25 smithi125 conmon[19936]: : cluster [DBG] osdmap e754: 8 total, 8 up, 8 in 2021-03-24T22:05:25.875 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:25 smithi125 conmon[19936]: cluster 2021-03-24T22:05:25.395629+0000 mon.a (mon.0) 1942 : cluster [DBG] osdmap e755: 8 total, 8 up, 8 in 2021-03-24T22:05:25.875 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:25 smithi125 conmon[24927]: cluster 2021-03-24T22:05 2021-03-24T22:05:25.875 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:25 smithi125 conmon[24927]: :24.515861+0000 mon.a (mon.0) 1941 : cluster [DBG] osdmap e754: 8 total, 8 up, 8 in 2021-03-24T22:05:25.876 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:25 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T22:05:25.876 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:25 smithi125 conmon[24927]: 22:05:25.395629+0000 mon.a (mon.0) 2021-03-24T22:05:25.876 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:25 smithi125 conmon[24927]: 1942 : cluster [DBG] osdmap e755: 8 total, 8 up, 8 in 2021-03-24T22:05:25.916 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:25.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:25.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:05:26.021 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:26.078 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:26.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:26.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:05:26.185 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:26.246 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:26.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:26.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:05:26.354 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:26.418 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:26.422 INFO:tasks.rados.rados.0.smithi125.stdout:2912: done (1 left) 2021-03-24T22:05:26.422 INFO:tasks.rados.rados.0.smithi125.stdout:2913: write (excl) oid 5 current snap is 336 2021-03-24T22:05:26.423 INFO:tasks.rados.rados.0.smithi125.stdout:2913: seq_num 1012 ranges {608684=426937,1803189=569688,2924304=84380} 2021-03-24T22:05:26.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:26.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:05:26.428 INFO:tasks.rados.rados.0.smithi125.stdout:2913: writing smithi12556470-5 from 608684 to 1035621 tid 1 2021-03-24T22:05:26.445 INFO:tasks.rados.rados.0.smithi125.stdout:2913: writing smithi12556470-5 from 1803189 to 2372877 tid 2 2021-03-24T22:05:26.449 INFO:tasks.rados.rados.0.smithi125.stdout:2913: writing smithi12556470-5 from 2924304 to 3008684 tid 3 2021-03-24T22:05:26.449 INFO:tasks.rados.rados.0.smithi125.stdout:2911: expect (ObjNum 984 snap 326 seq_num 984) 2021-03-24T22:05:26.527 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:26.584 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:26.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:26.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:05:26.625 INFO:tasks.rados.rados.0.smithi125.stdout:2913: finishing write tid 1 to smithi12556470-5 2021-03-24T22:05:26.626 INFO:tasks.rados.rados.0.smithi125.stdout:2913: finishing write tid 2 to smithi12556470-5 2021-03-24T22:05:26.626 INFO:tasks.rados.rados.0.smithi125.stdout:2913: finishing write tid 3 to smithi12556470-5 2021-03-24T22:05:26.626 INFO:tasks.rados.rados.0.smithi125.stdout:2913: finishing write tid 4 to smithi12556470-5 2021-03-24T22:05:26.627 INFO:tasks.rados.rados.0.smithi125.stdout:2913: finishing write tid 5 to smithi12556470-5 2021-03-24T22:05:26.627 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 5 v 1130 (ObjNum 1012 snap 336 seq_num 1012) dirty exists 2021-03-24T22:05:26.627 INFO:tasks.rados.rados.0.smithi125.stdout:2913: left oid 5 (ObjNum 1012 snap 336 seq_num 1012) 2021-03-24T22:05:26.627 INFO:tasks.rados.rados.0.smithi125.stdout:2911: done (1 left) 2021-03-24T22:05:26.628 INFO:tasks.rados.rados.0.smithi125.stdout:2913: done (0 left) 2021-03-24T22:05:26.628 INFO:tasks.rados.rados.0.smithi125.stdout:2914: write oid 20 current snap is 336 2021-03-24T22:05:26.628 INFO:tasks.rados.rados.0.smithi125.stdout:2914: seq_num 1013 ranges {625491=502186,1861671=713346} 2021-03-24T22:05:26.635 INFO:tasks.rados.rados.0.smithi125.stdout:2914: writing smithi12556470-20 from 625491 to 1127677 tid 1 2021-03-24T22:05:26.647 INFO:tasks.rados.rados.0.smithi125.stdout:2914: writing smithi12556470-20 from 1861671 to 2575017 tid 2 2021-03-24T22:05:26.647 INFO:tasks.rados.rados.0.smithi125.stdout:2915: read oid 38 snap 334 2021-03-24T22:05:26.647 INFO:tasks.rados.rados.0.smithi125.stdout:2915: expect (ObjNum 973 snap 318 seq_num 973) 2021-03-24T22:05:26.648 INFO:tasks.rados.rados.0.smithi125.stdout:2916: read oid 47 snap -1 2021-03-24T22:05:26.648 INFO:tasks.rados.rados.0.smithi125.stdout:2916: expect (ObjNum 929 snap 311 seq_num 929) 2021-03-24T22:05:26.648 INFO:tasks.rados.rados.0.smithi125.stdout:2917: write (excl) oid 45 current snap is 336 2021-03-24T22:05:26.648 INFO:tasks.rados.rados.0.smithi125.stdout:2914: finishing write tid 1 to smithi12556470-20 2021-03-24T22:05:26.649 INFO:tasks.rados.rados.0.smithi125.stdout:2917: seq_num 1014 ranges {642298=577435,1920153=573357,2979150=63148} 2021-03-24T22:05:26.654 INFO:tasks.rados.rados.0.smithi125.stdout:2917: writing smithi12556470-45 from 642298 to 1219733 tid 1 2021-03-24T22:05:26.661 INFO:tasks.rados.rados.0.smithi125.stdout:2917: writing smithi12556470-45 from 1920153 to 2493510 tid 2 2021-03-24T22:05:26.666 INFO:tasks.rados.rados.0.smithi125.stdout:2917: writing smithi12556470-45 from 2979150 to 3042298 tid 3 2021-03-24T22:05:26.666 INFO:tasks.rados.rados.0.smithi125.stdout:2918: read oid 31 snap -1 2021-03-24T22:05:26.666 INFO:tasks.rados.rados.0.smithi125.stdout:2918: expect (ObjNum 1009 snap 335 seq_num 1009) 2021-03-24T22:05:26.667 INFO:tasks.rados.rados.0.smithi125.stdout:2919: read oid 11 snap -1 2021-03-24T22:05:26.667 INFO:tasks.rados.rados.0.smithi125.stdout:2919: expect (ObjNum 991 snap 328 seq_num 991) 2021-03-24T22:05:26.667 INFO:tasks.rados.rados.0.smithi125.stdout:2920: snap_remove snap 328 2021-03-24T22:05:26.691 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:26.743 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:26.746 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:26 smithi125 conmon[19936]: cluster 2021-03-24T22:05:24.981359+0000 mgr.y (mgr.14144) 652 : cluster [DBG] pgmap v1294: 168 pgs: 168 active+clean; 553 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.4 MiB/s rd, 1.2 MiB/s wr, 8 op/s 2021-03-24T22:05:26.747 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:26 smithi125 conmon[19936]: cluster 2021-03-24T22:05:26.416528+0000 mon.a (mon.0) 1943 : cluster [DBG] osdmap e756: 8 total, 8 up, 8 in 2021-03-24T22:05:26.747 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:26 smithi125 conmon[24927]: cluster 2021-03-24T22:05:24.981359+0000 mgr.y (mgr.14144) 652 : cluster [DBG] pgmap v1294: 168 pgs: 168 active+clean; 553 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.4 MiB/s rd, 1.2 MiB/s wr, 8 op/s 2021-03-24T22:05:26.747 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:26 smithi125 conmon[24927]: cluster 2021-03-24T22:05:26.416528+0000 mon.a (mon.0) 1943 : cluster [DBG] osdmap e756: 8 total, 8 up, 8 in 2021-03-24T22:05:26.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:26.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:05:26.850 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:26.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:26 smithi142 conmon[20125]: cluster 2021-03-24T22:05 2021-03-24T22:05:26.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:26 smithi142 conmon[20125]: :24.981359+0000 mgr.y (mgr.14144) 652 : cluster [DBG] pgmap v1294: 168 pgs: 168 active+clean; 553 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.4 MiB/s rd, 1.2 MiB/s wr, 8 op/s 2021-03-24T22:05:26.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:26 smithi142 conmon[20125]: cluster 2021-03-24T22 2021-03-24T22:05:26.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:26 smithi142 conmon[20125]: :05:26.416528+0000 mon.a (mon.0) 1943 : cluster [DBG] 2021-03-24T22:05:26.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:26 smithi142 conmon[20125]: osdmap e756: 8 total, 8 up, 8 in 2021-03-24T22:05:26.908 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:26.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:26.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:05:27.015 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:27.078 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:27.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:27.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:05:27.185 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:27.248 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:27.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:27.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:05:27.353 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:27.415 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:27.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:27.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:05:27.522 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:27.553 INFO:tasks.rados.rados.0.smithi125.stdout:2920: done (6 left) 2021-03-24T22:05:27.553 INFO:tasks.rados.rados.0.smithi125.stdout:2921: rollback oid 36 current snap is 336 2021-03-24T22:05:27.553 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 36 to 327 2021-03-24T22:05:27.553 INFO:tasks.rados.rados.0.smithi125.stdout:2922: snap_remove snap 238 2021-03-24T22:05:27.583 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:27.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:27.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:05:27.688 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:27.741 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:27.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:27.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:05:27.848 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:27.903 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:27.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:27.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:05:28.010 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:28.057 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:28.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:28.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:05:28.165 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:28.221 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:28.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:28.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:05:28.327 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:28.345 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T22:05:28.365 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T22:05:28.383 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:28.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:28.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:05:28.390 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T22:05:28.411 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T22:05:28.436 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T22:05:28.467 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T22:05:28.490 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:28.491 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T22:05:28.526 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T22:05:28.527 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-24T22:05:28.527 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-24T22:05:28.527 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-24T22:05:28.528 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-24T22:05:28.528 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-24T22:05:28.528 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-24T22:05:28.528 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-24T22:05:28.528 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-24T22:05:28.550 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:28.556 INFO:tasks.rados.rados.0.smithi125.stdout:2922: done (7 left) 2021-03-24T22:05:28.556 INFO:tasks.rados.rados.0.smithi125.stdout:2923: copy_from oid 34 from oid 17 current snap is 336 2021-03-24T22:05:28.556 INFO:tasks.rados.rados.0.smithi125.stdout:2924: rollback oid 8 current snap is 336 2021-03-24T22:05:28.557 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 8 to 329 2021-03-24T22:05:28.557 INFO:tasks.rados.rados.0.smithi125.stdout:2925: snap_remove snap 311 2021-03-24T22:05:28.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:28.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:05:28.570 INFO:teuthology.orchestra.run.smithi125.stderr:nodeep-scrub is set 2021-03-24T22:05:28.661 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:28.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:28 smithi125 conmon[19936]: cluster 2021-03-24T22:05:26.982299+0000 mgr.y (mgr.14144 2021-03-24T22:05:28.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:28 smithi125 conmon[19936]: ) 653 : cluster [DBG] pgmap v1297: 168 pgs: 168 active+clean; 553 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 877 KiB/s rd, 334 KiB/s wr, 5 op/s 2021-03-24T22:05:28.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:28 smithi125 conmon[19936]: cluster 2021-03-24T22:05:27.549488+0000 mon.a (mon.0) 1944 : cluster [DBG] osdmap e757: 8 total, 8 up, 8 in 2021-03-24T22:05:28.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:28 smithi125 conmon[19936]: audit 2021-03-24T22:05:28.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:28 smithi125 conmon[19936]: 2021-03-24T22:05:28.317300+0000 mon.a (mon.0) 1945 : audit [DBG] from='client.? v1:172.21.15.125:0/2623106903' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:05:28.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:28 smithi125 conmon[19936]: audit 2021-03-24T22:05: 2021-03-24T22:05:28.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:28 smithi125 conmon[19936]: 28.493341+0000 mon.a (mon.0) 1946 : audit 2021-03-24T22:05:28.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:28 smithi125 conmon[19936]: [INF] from='client.? v1:172.21.15.125:0/4141522229' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:05:28.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:28 smithi125 conmon[24927]: cluster 2021-03-24T22:05:26.982299+0000 mgr.y (mgr.14144) 2021-03-24T22:05:28.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:28 smithi125 conmon[24927]: 653 : cluster [DBG] pgmap v1297: 168 pgs: 168 active+clean; 553 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 877 KiB/s rd, 334 KiB/s wr, 5 op/s 2021-03-24T22:05:28.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:28 smithi125 conmon[24927]: cluster 2021-03-24T22:05:27.549488+0000 mon.a (mon.0) 1944 : cluster 2021-03-24T22:05:28.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:28 smithi125 conmon[24927]: [DBG] osdmap e757: 8 total, 8 up, 8 in 2021-03-24T22:05:28.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:28 smithi125 conmon[24927]: audit 2021-03-24T22:05 2021-03-24T22:05:28.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:28 smithi125 conmon[24927]: :28.317300+0000 mon.a (mon.0) 1945 : audit [DBG] from='client.? v1:172.21.15.125:0/2623106903' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:05:28.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:28 smithi125 conmon[24927]: audit 2021-03-24T 2021-03-24T22:05:28.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:28 smithi125 conmon[24927]: 22:05:28.493341+0000 mon.a (mon.0) 1946 2021-03-24T22:05:28.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:28 smithi125 conmon[24927]: : audit [INF] from='client.? v1:172.21.15.125:0/4141522229' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:05:28.723 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:28.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:28.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:05:28.828 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:28.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:28 smithi142 conmon[20125]: cluster 2021-03-24T22:05:26.982299+0000 mgr.y ( 2021-03-24T22:05:28.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:28 smithi142 conmon[20125]: mgr.14144) 653 : cluster [DBG] pgmap v1297: 168 pgs: 168 active+clean; 553 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 877 KiB/s rd, 334 KiB/s wr, 5 op/s 2021-03-24T22:05:28.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:28 smithi142 conmon[20125]: cluster 2021-03-24T22:05:27.549488+0000 mon.a (mon.0) 1944 : cluster [DBG] osdmap e757: 8 total, 8 up, 8 in 2021-03-24T22:05:28.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:28 smithi142 conmon[20125]: audit 2021-03-24T22:05:28.317300+0000 mon.a (mon.0) 1945 : audit [DBG] from='client.? v1:172.21.15.125:0/2623106903' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:05:28.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:28 smithi142 conmon[20125]: audit 2021-03-24T22:05:28 2021-03-24T22:05:28.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:28 smithi142 conmon[20125]: .493341+0000 mon.a (mon.0) 1946 : audit [INF] from='client.? v1:172.21.15.125:0/4141522229' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:05:28.887 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:28.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:28.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:05:28.993 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:29.052 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:29.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:29.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:05:29.157 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:29.218 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:29.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:29.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:05:29.324 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:29.381 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:29.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:29.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:05:29.487 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:29.544 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:29.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:29.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:05:29.557 INFO:tasks.rados.rados.0.smithi125.stdout:2925: done (9 left) 2021-03-24T22:05:29.558 INFO:tasks.rados.rados.0.smithi125.stdout:2926: snap_remove snap 313 2021-03-24T22:05:29.650 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:29.714 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:29.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:29.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:05:29.822 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:29.834 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:29 smithi125 conmon[19936]: cluster 2021-03-24T22:05:28.543730+0000 mon.a (mon.0) 2021-03-24T22:05:29.834 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:29 smithi125 conmon[19936]: 1947 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:05:29.835 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:29 smithi125 conmon[19936]: audit 2021-03-24T22:05:28.551527+0000 mon.a (mon.0) 1948 : audit [INF] from='client.? v1:172.21.15.125:0/4141522229' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T22:05:29.835 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:29 smithi125 conmon[19936]: cluster 2021-03-24T22:05:28.551670+0000 mon.a (mon.0) 1949 : cluster [DBG] 2021-03-24T22:05:29.835 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:29 smithi125 conmon[19936]: osdmap e758: 8 total, 8 up, 8 in 2021-03-24T22:05:29.835 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:29 smithi125 conmon[19936]: cluster 2021-03-24T22:05:29. 2021-03-24T22:05:29.835 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:29 smithi125 conmon[19936]: 554761+0000 mon.a (mon.0) 1950 : cluster [DBG] osdmap e759: 8 total, 8 up, 8 in 2021-03-24T22:05:29.836 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:29 smithi125 conmon[19936]: 2021-03-24T22:05:29.836 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:29 smithi125 conmon[24927]: cluster 2021-03-24T22:05:29.836 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:29 smithi125 conmon[24927]: 2021-03-24T22:05 2021-03-24T22:05:29.836 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:29 smithi125 conmon[24927]: :28.543730+0000 mon.a (mon.0) 1947 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:05:29.837 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:29 smithi125 conmon[24927]: audit 2021-03-24T22:05:28.551527 2021-03-24T22:05:29.837 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:29 smithi125 conmon[24927]: +0000 mon.a (mon.0) 1948 : audit [INF] from='client.? v1:172.21.15.125:0/4141522229' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T22:05:29.837 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:29 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T22:05:29.837 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:29 smithi125 conmon[24927]: T22:05:28.551670+0000 mon.a (mon.0) 2021-03-24T22:05:29.838 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:29 smithi125 conmon[24927]: 1949 : cluster [DBG] osdmap e758: 8 total, 8 up, 8 in 2021-03-24T22:05:29.838 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:29 smithi125 conmon[24927]: cluster 2021-03-24T22 2021-03-24T22:05:29.838 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:29 smithi125 conmon[24927]: :05:29.554761+0000 mon.a (mon.0) 1950 2021-03-24T22:05:29.838 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:29 smithi125 conmon[24927]: : cluster [DBG] osdmap e759: 8 total, 8 up, 8 in 2021-03-24T22:05:29.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:29 smithi142 conmon[20125]: cluster 2021-03-24T22:05:29.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:29 smithi142 conmon[20125]: 2021-03-24T22:05:29.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:29 smithi142 conmon[20125]: 2021 2021-03-24T22:05:29.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:29 smithi142 conmon[20125]: - 2021-03-24T22:05:29.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:29 smithi142 conmon[20125]: 03 2021-03-24T22:05:29.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:29 smithi142 conmon[20125]: - 2021-03-24T22:05:29.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:29 smithi142 conmon[20125]: 24 2021-03-24T22:05:29.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:29 smithi142 conmon[20125]: T 2021-03-24T22:05:29.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:29 smithi142 conmon[20125]: 22 2021-03-24T22:05:29.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:29 smithi142 conmon[20125]: :05:28.543730+0000 mon.a 2021-03-24T22:05:29.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:29 smithi142 conmon[20125]: (mon.0) 1947 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:05:29.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:29 smithi142 conmon[20125]: audit 2021 2021-03-24T22:05:29.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:29 smithi142 conmon[20125]: -03-24T22:05:28. 2021-03-24T22:05:29.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:29 smithi142 conmon[20125]: 551527+0000 mon.a (mon.0 2021-03-24T22:05:29.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:29 smithi142 conmon[20125]: ) 1948 : audit [INF] from='client.? v1:172.21.15.125:0/4141522229' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T22:05:29.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:29 smithi142 conmon[20125]: 2021-03-24T22:05:29.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:29 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:05:29.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:29 smithi142 conmon[20125]: -24T22:05:28.551670 2021-03-24T22:05:29.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:29 smithi142 conmon[20125]: +0000 mon.a (mon.0) 1949 : 2021-03-24T22:05:29.872 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:29 smithi142 conmon[20125]: cluster [DBG] osdmap e758: 8 total, 8 up, 8 in 2021-03-24T22:05:29.872 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:29 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:05:29.872 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:29 smithi142 conmon[20125]: T22:05:29.554761+0000 2021-03-24T22:05:29.873 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:29 smithi142 conmon[20125]: mon.a (mon.0) 1950 : 2021-03-24T22:05:29.873 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:29 smithi142 conmon[20125]: cluster [DBG] osdmap e759: 8 total, 8 up, 8 in 2021-03-24T22:05:29.875 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:29.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:29.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:05:29.981 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:30.036 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:30.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:30.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:05:30.143 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:30.199 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:30.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:30.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:05:30.305 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:30.363 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:30.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:30.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:05:30.411 INFO:tasks.rados.rados.0.smithi125.stdout:2926: done (9 left) 2021-03-24T22:05:30.412 INFO:tasks.rados.rados.0.smithi125.stdout:2927: rollback oid 26 current snap is 336 2021-03-24T22:05:30.412 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 26 to 260 2021-03-24T22:05:30.412 INFO:tasks.rados.rados.0.smithi125.stdout:2928: snap_remove snap 310 2021-03-24T22:05:30.470 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:30.516 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:30.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:30.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:05:30.620 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:30.674 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:30.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:30.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:05:30.780 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:30.839 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:30.843 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:30 smithi142 conmon[20125]: cluster 2021-03-24T22:05:28. 2021-03-24T22:05:30.844 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:30 smithi142 conmon[20125]: 983114+0000 mgr.y (mgr.14144) 654 : cluster [DBG] pgmap v1300: 168 pgs: 168 active+clean; 535 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 291 KiB/s wr, 5 op/s 2021-03-24T22:05:30.844 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:30 smithi142 conmon[20125]: cluster 2021-03-24T22:05:30 2021-03-24T22:05:30.845 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:30 smithi142 conmon[20125]: .403165+0000 mon.a (mon.0) 1951 : cluster [DBG] 2021-03-24T22:05:30.845 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:30 smithi142 conmon[20125]: osdmap e760: 8 total, 8 up, 8 in 2021-03-24T22:05:30.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:30.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:05:30.912 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:30 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:05:30.912 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:30 smithi125 conmon[24927]: 05:28.983114+0000 mgr.y (mgr.14144) 654 : cluster [DBG] pgmap v1300: 168 pgs: 168 active+clean; 535 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 291 KiB/s wr, 5 op/s 2021-03-24T22:05:30.912 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:30 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T22:05:30.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:30 smithi125 conmon[24927]: T22:05:30.403165+0000 mon.a (mon. 2021-03-24T22:05:30.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:30 smithi125 conmon[24927]: 0) 1951 : cluster [DBG] osdmap e760: 8 total, 8 up, 8 in 2021-03-24T22:05:30.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:30 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:05:30.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:30 smithi125 conmon[19936]: 05:28.983114+0000 mgr.y (mgr.14144) 654 : cluster [DBG] pgmap v1300: 168 pgs: 168 active+clean; 535 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 291 KiB/s wr, 5 op/s 2021-03-24T22:05:30.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:30 smithi125 conmon[19936]: cluster 2021-03-24T22:05:30.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:30 smithi125 conmon[19936]: 2021-03-24T22:05:30.403165+0000 mon.a (mon.0) 1951 : cluster [DBG] osdmap e760: 8 total, 8 up, 8 in 2021-03-24T22:05:30.949 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:31.002 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:31.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:31.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:05:31.108 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:31.163 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:31.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:31.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:05:31.270 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:31.319 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:31.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:31.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:05:31.405 INFO:tasks.rados.rados.0.smithi125.stdout:2928: done (10 left) 2021-03-24T22:05:31.405 INFO:tasks.rados.rados.0.smithi125.stdout:2929: rollback oid 16 current snap is 336 2021-03-24T22:05:31.405 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 16 to 317 2021-03-24T22:05:31.406 INFO:tasks.rados.rados.0.smithi125.stdout:2915: expect (ObjNum 973 snap 318 seq_num 973) 2021-03-24T22:05:31.424 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:31.476 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:31.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:31.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:05:31.554 INFO:tasks.rados.rados.0.smithi125.stdout:2914: finishing write tid 2 to smithi12556470-20 2021-03-24T22:05:31.555 INFO:tasks.rados.rados.0.smithi125.stdout:2914: finishing write tid 3 to smithi12556470-20 2021-03-24T22:05:31.555 INFO:tasks.rados.rados.0.smithi125.stdout:2914: finishing write tid 4 to smithi12556470-20 2021-03-24T22:05:31.555 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 20 v 895 (ObjNum 1013 snap 336 seq_num 1013) dirty exists 2021-03-24T22:05:31.555 INFO:tasks.rados.rados.0.smithi125.stdout:2914: left oid 20 (ObjNum 1013 snap 336 seq_num 1013) 2021-03-24T22:05:31.555 INFO:tasks.rados.rados.0.smithi125.stdout:2916: expect (ObjNum 929 snap 311 seq_num 929) 2021-03-24T22:05:31.583 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:31.643 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:31.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:31.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:05:31.731 INFO:tasks.rados.rados.0.smithi125.stdout:2917: finishing write tid 1 to smithi12556470-45 2021-03-24T22:05:31.731 INFO:tasks.rados.rados.0.smithi125.stdout:2917: finishing write tid 2 to smithi12556470-45 2021-03-24T22:05:31.731 INFO:tasks.rados.rados.0.smithi125.stdout:2917: finishing write tid 3 to smithi12556470-45 2021-03-24T22:05:31.731 INFO:tasks.rados.rados.0.smithi125.stdout:2917: finishing write tid 4 to smithi12556470-45 2021-03-24T22:05:31.732 INFO:tasks.rados.rados.0.smithi125.stdout:2917: finishing write tid 5 to smithi12556470-45 2021-03-24T22:05:31.732 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 45 v 739 (ObjNum 1014 snap 336 seq_num 1014) dirty exists 2021-03-24T22:05:31.732 INFO:tasks.rados.rados.0.smithi125.stdout:2917: left oid 45 (ObjNum 1014 snap 336 seq_num 1014) 2021-03-24T22:05:31.732 INFO:tasks.rados.rados.0.smithi125.stdout:2918: expect (ObjNum 1009 snap 335 seq_num 1009) 2021-03-24T22:05:31.750 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:31.807 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:31.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:31.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:05:31.888 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-24T22:05:31.888 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T22:05:31.914 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:31.968 INFO:tasks.rados.rados.0.smithi125.stdout:2919: expect (ObjNum 991 snap 328 seq_num 991) 2021-03-24T22:05:31.971 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:31.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:31.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:05:32.078 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:32.132 INFO:tasks.rados.rados.0.smithi125.stdout:2921: finishing rollback tid 0 to smithi12556470-36 2021-03-24T22:05:32.132 INFO:tasks.rados.rados.0.smithi125.stdout:2921: finishing rollback tid 1 to smithi12556470-36 2021-03-24T22:05:32.133 INFO:tasks.rados.rados.0.smithi125.stdout:2921: finishing rollback tid 2 to smithi12556470-36 2021-03-24T22:05:32.133 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 36 v 1141 (ObjNum 944 snap 314 seq_num 944) dirty exists 2021-03-24T22:05:32.133 INFO:tasks.rados.rados.0.smithi125.stdout:2914: done (10 left) 2021-03-24T22:05:32.134 INFO:tasks.rados.rados.0.smithi125.stdout:2915: done (9 left) 2021-03-24T22:05:32.134 INFO:tasks.rados.rados.0.smithi125.stdout:2916: done (8 left) 2021-03-24T22:05:32.134 INFO:tasks.rados.rados.0.smithi125.stdout:2917: done (7 left) 2021-03-24T22:05:32.134 INFO:tasks.rados.rados.0.smithi125.stdout:2918: done (6 left) 2021-03-24T22:05:32.135 INFO:tasks.rados.rados.0.smithi125.stdout:2919: done (5 left) 2021-03-24T22:05:32.135 INFO:tasks.rados.rados.0.smithi125.stdout:2921: done (4 left) 2021-03-24T22:05:32.135 INFO:tasks.rados.rados.0.smithi125.stdout:2930: snap_remove snap 333 2021-03-24T22:05:32.139 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:32.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:32.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:05:32.246 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:32.305 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:32.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:32.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:05:32.412 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:32.472 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:32.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:32.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:05:32.579 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:32.590 INFO:tasks.rados.rados.0.smithi125.stdout:2930: done (4 left) 2021-03-24T22:05:32.591 INFO:tasks.rados.rados.0.smithi125.stdout:2931: rollback oid 41 current snap is 336 2021-03-24T22:05:32.591 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 41 to 327 2021-03-24T22:05:32.591 INFO:tasks.rados.rados.0.smithi125.stdout:2932: snap_create 2021-03-24T22:05:32.592 INFO:tasks.rados.rados.0.smithi125.stdout:2924: finishing rollback tid 1 to smithi12556470-8 2021-03-24T22:05:32.592 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 8 v 560 (ObjNum 1116494592 snap 21916 seq_num 1117603392) dirty dne 2021-03-24T22:05:32.592 INFO:tasks.rados.rados.0.smithi125.stdout:2923: finishing copy_from to smithi12556470-34 2021-03-24T22:05:32.592 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 34 v 1058 (ObjNum 974 snap 318 seq_num 974) dirty exists 2021-03-24T22:05:32.592 INFO:tasks.rados.rados.0.smithi125.stdout:2923: finishing copy_from racing read to smithi12556470-34 2021-03-24T22:05:32.593 INFO:tasks.rados.rados.0.smithi125.stdout:2927: finishing rollback tid 0 to smithi12556470-26 2021-03-24T22:05:32.593 INFO:tasks.rados.rados.0.smithi125.stdout:2927: finishing rollback tid 1 to smithi12556470-26 2021-03-24T22:05:32.593 INFO:tasks.rados.rados.0.smithi125.stdout:2927: finishing rollback tid 2 to smithi12556470-26 2021-03-24T22:05:32.593 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 26 v 837 (ObjNum 794 snap 257 seq_num 794) dirty exists 2021-03-24T22:05:32.594 INFO:tasks.rados.rados.0.smithi125.stdout:2929: finishing rollback tid 0 to smithi12556470-16 2021-03-24T22:05:32.594 INFO:tasks.rados.rados.0.smithi125.stdout:2929: finishing rollback tid 1 to smithi12556470-16 2021-03-24T22:05:32.594 INFO:tasks.rados.rados.0.smithi125.stdout:2929: finishing rollback tid 2 to smithi12556470-16 2021-03-24T22:05:32.594 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 16 v 834 (ObjNum 875 snap 283 seq_num 875) dirty exists 2021-03-24T22:05:32.596 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:32 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:05:32.596 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:32 smithi125 conmon[19936]: 24T22:05:30.983848+0000 mgr.y (mgr.14144) 655 : cluster [DBG] pgmap v1303: 168 pgs: 168 active+clean; 518 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 2.6 MiB/s wr, 16 op/s 2021-03-24T22:05:32.596 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:32 smithi125 conmon[19936]: cluster 2021-03-24T22:05:31. 2021-03-24T22:05:32.596 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:32 smithi125 conmon[19936]: 401590+0000 mon.a (mon.0) 1952 : cluster [DBG] 2021-03-24T22:05:32.596 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:32 smithi125 conmon[19936]: osdmap e761: 8 total, 8 up, 8 in 2021-03-24T22:05:32.597 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:32 smithi125 conmon[19936]: audit 2021-03-24 2021-03-24T22:05:32.597 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:32 smithi125 conmon[19936]: T22:05:31.960198+0000 mon.a (mon.0) 2021-03-24T22:05:32.597 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:32 smithi125 conmon[19936]: 1953 : audit [INF] from='client.? v1:172.21.15.125:0/1374229308' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2021-03-24T22:05:32.598 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:32 smithi125 conmon[24927]: cluster 2021-03-24T22 2021-03-24T22:05:32.598 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:32 smithi125 conmon[24927]: :05:30.983848+0000 mgr.y (mgr.14144) 655 : cluster [DBG] pgmap v1303: 168 pgs: 168 active+clean; 518 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 2.6 MiB/s wr, 16 op/s 2021-03-24T22:05:32.598 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:32 smithi125 conmon[24927]: cluster 2021-03-24T22:05:32.598 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:32 smithi125 conmon[24927]: 2021-03-24T22:05:31.401590 2021-03-24T22:05:32.599 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:32 smithi125 conmon[24927]: +0000 mon.a (mon.0) 1952 : cluster [DBG] 2021-03-24T22:05:32.599 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:32 smithi125 conmon[24927]: osdmap e761: 8 total, 8 up, 8 in 2021-03-24T22:05:32.599 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:32 smithi125 conmon[24927]: audit 2021-03-24T22:05: 2021-03-24T22:05:32.599 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:32 smithi125 conmon[24927]: 31.960198+0000 mon.a (mon.0) 1953 : 2021-03-24T22:05:32.599 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:32 smithi125 conmon[24927]: audit [INF] from='client.? v1:172.21.15.125:0/1374229308' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2021-03-24T22:05:32.602 INFO:teuthology.orchestra.run.smithi125.stderr:reweighted osd.3 to 1 (10000) 2021-03-24T22:05:32.605 INFO:tasks.rados.rados.0.smithi125.stdout:2931: finishing rollback tid 0 to smithi12556470-41 2021-03-24T22:05:32.610 INFO:tasks.rados.rados.0.smithi125.stdout:2931: finishing rollback tid 1 to smithi12556470-41 2021-03-24T22:05:32.611 INFO:tasks.rados.rados.0.smithi125.stdout:2931: finishing rollback tid 2 to smithi12556470-41 2021-03-24T22:05:32.611 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 41 v 1133 (ObjNum 916 snap 304 seq_num 916) dirty exists 2021-03-24T22:05:32.631 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:32.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:32.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:05:32.735 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:32.790 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:32.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:32.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:05:32.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:32 smithi142 conmon[20125]: cluster 2021-03-24T22:05:30 2021-03-24T22:05:32.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:32 smithi142 conmon[20125]: .983848+0000 mgr.y (mgr.14144) 655 : cluster [DBG] pgmap v1303: 168 pgs: 168 active+clean; 518 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 2.6 MiB/s wr, 16 op/s 2021-03-24T22:05:32.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:32 smithi142 conmon[20125]: cluster 2021-03-24T22:05: 2021-03-24T22:05:32.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:32 smithi142 conmon[20125]: 31.401590+0000 mon.a (mon.0) 1952 : cluster [DBG] osdmap e761: 8 total, 8 up, 8 in 2021-03-24T22:05:32.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:32 smithi142 conmon[20125]: audit 2021-03-24T22:05:31.960198+0000 2021-03-24T22:05:32.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:32 smithi142 conmon[20125]: mon.a (mon.0) 1953 : audit [INF] from='client.? v1:172.21.15.125:0/1374229308' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2021-03-24T22:05:32.898 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:32.954 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:32.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:32.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:05:33.065 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:33.076 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.1 dump_ops_in_flight 2021-03-24T22:05:33.124 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:33.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:33.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:05:33.230 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:33.279 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:33.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:33.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:05:33.384 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:33.445 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:33.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:33.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:05:33.472 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd unset noscrub 2021-03-24T22:05:33.501 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-24T22:05:33.551 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:33.598 INFO:tasks.rados.rados.0.smithi125.stdout:2923: done (5 left) 2021-03-24T22:05:33.599 INFO:tasks.rados.rados.0.smithi125.stdout:2924: done (4 left) 2021-03-24T22:05:33.599 INFO:tasks.rados.rados.0.smithi125.stdout:2927: done (3 left) 2021-03-24T22:05:33.599 INFO:tasks.rados.rados.0.smithi125.stdout:2929: done (2 left) 2021-03-24T22:05:33.599 INFO:tasks.rados.rados.0.smithi125.stdout:2931: done (1 left) 2021-03-24T22:05:33.600 INFO:tasks.rados.rados.0.smithi125.stdout:2932: done (0 left) 2021-03-24T22:05:33.600 INFO:tasks.rados.rados.0.smithi125.stdout:2933: read oid 7 snap -1 2021-03-24T22:05:33.600 INFO:tasks.rados.rados.0.smithi125.stdout:2933: expect (ObjNum 996 snap 329 seq_num 996) 2021-03-24T22:05:33.600 INFO:tasks.rados.rados.0.smithi125.stdout:2934: rollback oid 36 current snap is 337 2021-03-24T22:05:33.600 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 36 to 288 2021-03-24T22:05:33.601 INFO:tasks.rados.rados.0.smithi125.stdout:2935: snap_remove snap 303 2021-03-24T22:05:33.616 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:33.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:33.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:05:33.723 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:33.736 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:33 smithi142 conmon[20125]: audit 2021-03-24T22:05:32.587143+0000 mon.a (mon.0) 1954 : 2021-03-24T22:05:33.737 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:33 smithi142 conmon[20125]: audit [INF] from='client.? v1:172.21.15.125:0/1374229308' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2021-03-24T22:05:33.737 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:33 smithi142 conmon[20125]: cluster 2021-03-24T22:05:32.587412+0000 mon.a (mon.0) 1955 2021-03-24T22:05:33.737 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:33 smithi142 conmon[20125]: : cluster [DBG] osdmap e762: 8 total, 8 up, 8 in 2021-03-24T22:05:33.778 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:33.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:33.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:05:33.885 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:33.895 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:33 smithi125 conmon[19936]: audit 2021-03-24T22:05:32.587143+0000 2021-03-24T22:05:33.896 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:33 smithi125 conmon[19936]: mon.a (mon.0) 1954 : audit [INF] from='client.? v1:172.21.15.125:0/1374229308' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2021-03-24T22:05:33.896 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:33 smithi125 conmon[19936]: cluster 2021-03-24T22:05:32.587412+0000 mon.a (mon.0) 1955 2021-03-24T22:05:33.896 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:33 smithi125 conmon[19936]: : cluster [DBG] osdmap e762: 8 total, 8 up, 8 in 2021-03-24T22:05:33.897 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:33 smithi125 conmon[24927]: audit 2021-03-24T22:05:32.587143+0000 mon.a (mon. 2021-03-24T22:05:33.897 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:33 smithi125 conmon[24927]: 0) 1954 : audit [INF] from='client.? v1:172.21.15.125:0/1374229308' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2021-03-24T22:05:33.897 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:33 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:05:33.897 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:33 smithi125 conmon[24927]: 24T22:05:32 2021-03-24T22:05:33.898 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:33 smithi125 conmon[24927]: .587412+0000 mon.a ( 2021-03-24T22:05:33.898 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:33 smithi125 conmon[24927]: mon.0) 1955 : 2021-03-24T22:05:33.898 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:33 smithi125 conmon[24927]: cluster [DBG] osdmap e762: 8 total, 8 up, 8 in 2021-03-24T22:05:33.937 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:33.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:33.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:05:33.965 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd reweight 4 1 2021-03-24T22:05:33.997 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:05:33.999 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:05:34.042 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:34.051 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:05:34.100 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:34.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:34.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:05:34.207 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:34.259 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:34.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:34.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:05:34.364 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:34.426 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:34.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:34.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:05:34.532 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:34.584 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:05:34.598 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:34.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:34.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:05:34.614 INFO:tasks.rados.rados.0.smithi125.stdout:2935: done (2 left) 2021-03-24T22:05:34.614 INFO:tasks.rados.rados.0.smithi125.stdout:2936: write oid 28 current snap is 337 2021-03-24T22:05:34.614 INFO:tasks.rados.rados.0.smithi125.stdout:2936: seq_num 1015 ranges {659105=652684,1862282=717015} 2021-03-24T22:05:34.623 INFO:tasks.rados.rados.0.smithi125.stdout:2936: writing smithi12556470-28 from 659105 to 1311789 tid 1 2021-03-24T22:05:34.634 INFO:tasks.rados.rados.0.smithi125.stdout:2936: writing smithi12556470-28 from 1862282 to 2579297 tid 2 2021-03-24T22:05:34.634 INFO:tasks.rados.rados.0.smithi125.stdout:2937: write oid 43 current snap is 337 2021-03-24T22:05:34.634 INFO:tasks.rados.rados.0.smithi125.stdout:2937: seq_num 1016 ranges {675912=727933,1920764=460673,2801290=274622} 2021-03-24T22:05:34.644 INFO:tasks.rados.rados.0.smithi125.stdout:2937: writing smithi12556470-43 from 675912 to 1403845 tid 1 2021-03-24T22:05:34.650 INFO:tasks.rados.rados.0.smithi125.stdout:2937: writing smithi12556470-43 from 1920764 to 2381437 tid 2 2021-03-24T22:05:34.654 INFO:tasks.rados.rados.0.smithi125.stdout:2937: writing smithi12556470-43 from 2801290 to 3075912 tid 3 2021-03-24T22:05:34.654 INFO:tasks.rados.rados.0.smithi125.stdout:2933: expect (ObjNum 996 snap 329 seq_num 996) 2021-03-24T22:05:34.711 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:34.725 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:34 smithi142 conmon[20125]: cluster 2021-03-24T22:05:32.984591+0000 mgr.y (mgr.14144) 656 2021-03-24T22:05:34.725 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:34 smithi142 conmon[20125]: : cluster [DBG] pgmap v1306: 168 pgs: 168 active+clean; 502 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 2.6 MiB/s wr, 17 op/s 2021-03-24T22:05:34.725 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:34 smithi142 conmon[20125]: cluster 2021-03-24T22:05:33.594885+0000 mon.a (mon.0) 1956 : cluster [DBG] osdmap e763: 8 total, 8 up, 8 in 2021-03-24T22:05:34.773 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:34.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:34.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:05:34.830 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:34 smithi125 conmon[19936]: cluster 2021-03-24T22:05:32.984591+0000 mgr.y (mgr.14144 2021-03-24T22:05:34.830 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:34 smithi125 conmon[19936]: ) 656 : cluster [DBG] pgmap v1306: 168 pgs: 168 active+clean; 502 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 2.6 MiB/s wr, 17 op/s 2021-03-24T22:05:34.831 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:34 smithi125 conmon[19936]: cluster 2021-03-24T22:05:33.594885+0000 mon.a (mon.0) 1956 : cluster [DBG] osdmap e763: 8 total, 8 up, 8 in 2021-03-24T22:05:34.831 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:34 smithi125 conmon[24927]: cluster 2021-03-24T22:05:34.831 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:34 smithi125 conmon[24927]: 2021-03-24T22:05:34.832 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:34 smithi125 conmon[24927]: 2021 2021-03-24T22:05:34.832 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:34 smithi125 conmon[24927]: - 2021-03-24T22:05:34.832 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:34 smithi125 conmon[24927]: 03 2021-03-24T22:05:34.832 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:34 smithi125 conmon[24927]: - 2021-03-24T22:05:34.833 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:34 smithi125 conmon[24927]: 24 2021-03-24T22:05:34.833 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:34 smithi125 conmon[24927]: T22:05:32.984591+0000 2021-03-24T22:05:34.833 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:34 smithi125 conmon[24927]: mgr.y (mgr.14144) 656 : cluster [DBG] pgmap v1306: 168 pgs: 168 active+clean; 502 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 2.6 MiB/s wr, 17 op/s 2021-03-24T22:05:34.833 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:34 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:05:34.834 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:34 smithi125 conmon[24927]: 03-24T22:05:33.594885+0000 mon.a ( 2021-03-24T22:05:34.834 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:34 smithi125 conmon[24927]: mon.0) 1956 : cluster [DBG] osdmap e763: 8 total, 8 up, 8 in 2021-03-24T22:05:34.835 INFO:tasks.rados.rados.0.smithi125.stdout:2934: finishing rollback tid 0 to smithi12556470-36 2021-03-24T22:05:34.835 INFO:tasks.rados.rados.0.smithi125.stdout:2934: finishing rollback tid 1 to smithi12556470-36 2021-03-24T22:05:34.836 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 36 v 1176 (ObjNum 1 snap 32741 seq_num 1846801056) dirty dne 2021-03-24T22:05:34.836 INFO:tasks.rados.rados.0.smithi125.stdout:2936: finishing write tid 1 to smithi12556470-28 2021-03-24T22:05:34.836 INFO:tasks.rados.rados.0.smithi125.stdout:2933: done (3 left) 2021-03-24T22:05:34.836 INFO:tasks.rados.rados.0.smithi125.stdout:2934: done (2 left) 2021-03-24T22:05:34.836 INFO:tasks.rados.rados.0.smithi125.stdout:2938: rollback oid 27 current snap is 337 2021-03-24T22:05:34.837 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 27 to 322 2021-03-24T22:05:34.837 INFO:tasks.rados.rados.0.smithi125.stdout:2936: finishing write tid 2 to smithi12556470-28 2021-03-24T22:05:34.837 INFO:tasks.rados.rados.0.smithi125.stdout:2936: finishing write tid 3 to smithi12556470-28 2021-03-24T22:05:34.838 INFO:tasks.rados.rados.0.smithi125.stdout:2936: finishing write tid 4 to smithi12556470-28 2021-03-24T22:05:34.838 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 28 v 1202 (ObjNum 1015 snap 337 seq_num 1015) dirty exists 2021-03-24T22:05:34.838 INFO:tasks.rados.rados.0.smithi125.stdout:2936: left oid 28 (ObjNum 1015 snap 337 seq_num 1015) 2021-03-24T22:05:34.839 INFO:tasks.rados.rados.0.smithi125.stdout:2937: finishing write tid 1 to smithi12556470-43 2021-03-24T22:05:34.839 INFO:tasks.rados.rados.0.smithi125.stdout:2937: finishing write tid 2 to smithi12556470-43 2021-03-24T22:05:34.839 INFO:tasks.rados.rados.0.smithi125.stdout:2937: finishing write tid 3 to smithi12556470-43 2021-03-24T22:05:34.839 INFO:tasks.rados.rados.0.smithi125.stdout:2937: finishing write tid 4 to smithi12556470-43 2021-03-24T22:05:34.839 INFO:tasks.rados.rados.0.smithi125.stdout:2937: finishing write tid 5 to smithi12556470-43 2021-03-24T22:05:34.840 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 43 v 1820 (ObjNum 1016 snap 337 seq_num 1016) dirty exists 2021-03-24T22:05:34.840 INFO:tasks.rados.rados.0.smithi125.stdout:2937: left oid 43 (ObjNum 1016 snap 337 seq_num 1016) 2021-03-24T22:05:34.840 INFO:tasks.rados.rados.0.smithi125.stdout:2936: done (2 left) 2021-03-24T22:05:34.840 INFO:tasks.rados.rados.0.smithi125.stdout:2937: done (1 left) 2021-03-24T22:05:34.841 INFO:tasks.rados.rados.0.smithi125.stdout:2939: snap_remove snap 323 2021-03-24T22:05:34.881 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:34.935 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:34.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:34.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:05:35.042 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:35.100 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:35.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:35.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:05:35.205 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:35.263 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:35.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:35.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:05:35.370 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:35.414 INFO:tasks.rados.rados.0.smithi125.stdout:2939: done (1 left) 2021-03-24T22:05:35.415 INFO:tasks.rados.rados.0.smithi125.stdout:2940: write (excl) oid 3 current snap is 337 2021-03-24T22:05:35.415 INFO:tasks.rados.rados.0.smithi125.stdout:2940: seq_num 1017 ranges {692719=403182,1579246=720684,2828713=264006} 2021-03-24T22:05:35.421 INFO:tasks.rados.rados.0.smithi125.stdout:2940: writing smithi12556470-3 from 692719 to 1095901 tid 1 2021-03-24T22:05:35.428 INFO:tasks.rados.rados.0.smithi125.stdout:2940: writing smithi12556470-3 from 1579246 to 2299930 tid 2 2021-03-24T22:05:35.432 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:35.434 INFO:tasks.rados.rados.0.smithi125.stdout:2940: writing smithi12556470-3 from 2828713 to 3092719 tid 3 2021-03-24T22:05:35.435 INFO:tasks.rados.rados.0.smithi125.stdout:2941: write oid 32 current snap is 337 2021-03-24T22:05:35.435 INFO:tasks.rados.rados.0.smithi125.stdout:2938: finishing rollback tid 1 to smithi12556470-27 2021-03-24T22:05:35.435 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 27 v 1424 (ObjNum 1118636544 snap 21916 seq_num 0) dirty dne 2021-03-24T22:05:35.435 INFO:tasks.rados.rados.0.smithi125.stdout:2941: seq_num 1018 ranges {709526=478431,1637728=464342,2856136=253390} 2021-03-24T22:05:35.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:35.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:05:35.442 INFO:tasks.rados.rados.0.smithi125.stdout:2941: writing smithi12556470-32 from 709526 to 1187957 tid 1 2021-03-24T22:05:35.448 INFO:tasks.rados.rados.0.smithi125.stdout:2941: writing smithi12556470-32 from 1637728 to 2102070 tid 2 2021-03-24T22:05:35.455 INFO:tasks.rados.rados.0.smithi125.stdout:2941: writing smithi12556470-32 from 2856136 to 3109526 tid 3 2021-03-24T22:05:35.455 INFO:tasks.rados.rados.0.smithi125.stdout:2938: done (2 left) 2021-03-24T22:05:35.455 INFO:tasks.rados.rados.0.smithi125.stdout:2942: write (excl) oid 8 current snap is 337 2021-03-24T22:05:35.456 INFO:tasks.rados.rados.0.smithi125.stdout:2940: finishing write tid 1 to smithi12556470-3 2021-03-24T22:05:35.456 INFO:tasks.rados.rados.0.smithi125.stdout:2942: seq_num 1019 ranges {726333=670033,2096210=724353} 2021-03-24T22:05:35.462 INFO:tasks.rados.rados.0.smithi125.stdout:2942: writing smithi12556470-8 from 726333 to 1396366 tid 1 2021-03-24T22:05:35.470 INFO:tasks.rados.rados.0.smithi125.stdout:2942: writing smithi12556470-8 from 2096210 to 2820563 tid 2 2021-03-24T22:05:35.471 INFO:tasks.rados.rados.0.smithi125.stdout:2943: write (excl) oid 46 current snap is 337 2021-03-24T22:05:35.471 INFO:tasks.rados.rados.0.smithi125.stdout:2940: finishing write tid 2 to smithi12556470-3 2021-03-24T22:05:35.471 INFO:tasks.rados.rados.0.smithi125.stdout:2943: seq_num 1020 ranges {743140=745282,2154692=468011} 2021-03-24T22:05:35.479 INFO:tasks.rados.rados.0.smithi125.stdout:2943: writing smithi12556470-46 from 743140 to 1488422 tid 1 2021-03-24T22:05:35.485 INFO:tasks.rados.rados.0.smithi125.stdout:2943: writing smithi12556470-46 from 2154692 to 2622703 tid 2 2021-03-24T22:05:35.486 INFO:tasks.rados.rados.0.smithi125.stdout:2944: write oid 43 current snap is 337 2021-03-24T22:05:35.486 INFO:tasks.rados.rados.0.smithi125.stdout:2944: seq_num 1021 ranges {759947=420531,1813174=728022} 2021-03-24T22:05:35.492 INFO:tasks.rados.rados.0.smithi125.stdout:2944: writing smithi12556470-43 from 759947 to 1180478 tid 1 2021-03-24T22:05:35.500 INFO:tasks.rados.rados.0.smithi125.stdout:2944: writing smithi12556470-43 from 1813174 to 2541196 tid 2 2021-03-24T22:05:35.500 INFO:tasks.rados.rados.0.smithi125.stdout:2945: read oid 34 snap -1 2021-03-24T22:05:35.501 INFO:tasks.rados.rados.0.smithi125.stdout:2945: expect (ObjNum 974 snap 318 seq_num 974) 2021-03-24T22:05:35.501 INFO:tasks.rados.rados.0.smithi125.stdout:2946: write (excl) oid 31 current snap is 337 2021-03-24T22:05:35.501 INFO:tasks.rados.rados.0.smithi125.stdout:2940: finishing write tid 3 to smithi12556470-3 2021-03-24T22:05:35.501 INFO:tasks.rados.rados.0.smithi125.stdout:2946: seq_num 1022 ranges {776754=495780,1871656=471680,2965828=210926} 2021-03-24T22:05:35.507 INFO:tasks.rados.rados.0.smithi125.stdout:2946: writing smithi12556470-31 from 776754 to 1272534 tid 1 2021-03-24T22:05:35.514 INFO:tasks.rados.rados.0.smithi125.stdout:2946: writing smithi12556470-31 from 1871656 to 2343336 tid 2 2021-03-24T22:05:35.519 INFO:tasks.rados.rados.0.smithi125.stdout:2946: writing smithi12556470-31 from 2965828 to 3176754 tid 3 2021-03-24T22:05:35.519 INFO:tasks.rados.rados.0.smithi125.stdout:2947: delete oid 23 current snap is 337 2021-03-24T22:05:35.520 INFO:tasks.rados.rados.0.smithi125.stdout:2940: finishing write tid 4 to smithi12556470-3 2021-03-24T22:05:35.520 INFO:tasks.rados.rados.0.smithi125.stdout:2940: finishing write tid 5 to smithi12556470-3 2021-03-24T22:05:35.520 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 3 v 323 (ObjNum 1017 snap 337 seq_num 1017) dirty exists 2021-03-24T22:05:35.520 INFO:tasks.rados.rados.0.smithi125.stdout:2940: left oid 3 (ObjNum 1017 snap 337 seq_num 1017) 2021-03-24T22:05:35.521 INFO:tasks.rados.rados.0.smithi125.stdout:2941: finishing write tid 1 to smithi12556470-32 2021-03-24T22:05:35.521 INFO:tasks.rados.rados.0.smithi125.stdout:2942: finishing write tid 1 to smithi12556470-8 2021-03-24T22:05:35.521 INFO:tasks.rados.rados.0.smithi125.stdout:2943: finishing write tid 1 to smithi12556470-46 2021-03-24T22:05:35.521 INFO:tasks.rados.rados.0.smithi125.stdout:2945: expect (ObjNum 974 snap 318 seq_num 974) 2021-03-24T22:05:35.542 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:35.588 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:35.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:35.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:05:35.681 INFO:tasks.rados.rados.0.smithi125.stdout:2941: finishing write tid 2 to smithi12556470-32 2021-03-24T22:05:35.682 INFO:tasks.rados.rados.0.smithi125.stdout:2941: finishing write tid 3 to smithi12556470-32 2021-03-24T22:05:35.682 INFO:tasks.rados.rados.0.smithi125.stdout:2941: finishing write tid 4 to smithi12556470-32 2021-03-24T22:05:35.682 INFO:tasks.rados.rados.0.smithi125.stdout:2941: finishing write tid 5 to smithi12556470-32 2021-03-24T22:05:35.683 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 32 v 1826 (ObjNum 1018 snap 337 seq_num 1018) dirty exists 2021-03-24T22:05:35.683 INFO:tasks.rados.rados.0.smithi125.stdout:2941: left oid 32 (ObjNum 1018 snap 337 seq_num 1018) 2021-03-24T22:05:35.683 INFO:tasks.rados.rados.0.smithi125.stdout:2944: finishing write tid 1 to smithi12556470-43 2021-03-24T22:05:35.683 INFO:tasks.rados.rados.0.smithi125.stdout:2943: finishing write tid 2 to smithi12556470-46 2021-03-24T22:05:35.683 INFO:tasks.rados.rados.0.smithi125.stdout:2942: finishing write tid 2 to smithi12556470-8 2021-03-24T22:05:35.684 INFO:tasks.rados.rados.0.smithi125.stdout:2942: finishing write tid 3 to smithi12556470-8 2021-03-24T22:05:35.684 INFO:tasks.rados.rados.0.smithi125.stdout:2943: finishing write tid 3 to smithi12556470-46 2021-03-24T22:05:35.684 INFO:tasks.rados.rados.0.smithi125.stdout:2942: finishing write tid 4 to smithi12556470-8 2021-03-24T22:05:35.684 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 8 v 563 (ObjNum 1019 snap 337 seq_num 1019) dirty exists 2021-03-24T22:05:35.684 INFO:tasks.rados.rados.0.smithi125.stdout:2942: left oid 8 (ObjNum 1019 snap 337 seq_num 1019) 2021-03-24T22:05:35.685 INFO:tasks.rados.rados.0.smithi125.stdout:2943: finishing write tid 4 to smithi12556470-46 2021-03-24T22:05:35.685 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 46 v 834 (ObjNum 1020 snap 337 seq_num 1020) dirty exists 2021-03-24T22:05:35.685 INFO:tasks.rados.rados.0.smithi125.stdout:2943: left oid 46 (ObjNum 1020 snap 337 seq_num 1020) 2021-03-24T22:05:35.685 INFO:tasks.rados.rados.0.smithi125.stdout:2944: finishing write tid 2 to smithi12556470-43 2021-03-24T22:05:35.686 INFO:tasks.rados.rados.0.smithi125.stdout:2944: finishing write tid 3 to smithi12556470-43 2021-03-24T22:05:35.686 INFO:tasks.rados.rados.0.smithi125.stdout:2940: done (7 left) 2021-03-24T22:05:35.686 INFO:tasks.rados.rados.0.smithi125.stdout:2941: done (6 left) 2021-03-24T22:05:35.687 INFO:tasks.rados.rados.0.smithi125.stdout:2942: done (5 left) 2021-03-24T22:05:35.687 INFO:tasks.rados.rados.0.smithi125.stdout:2943: done (4 left) 2021-03-24T22:05:35.687 INFO:tasks.rados.rados.0.smithi125.stdout:2945: done (3 left) 2021-03-24T22:05:35.687 INFO:tasks.rados.rados.0.smithi125.stdout:2947: done (2 left) 2021-03-24T22:05:35.688 INFO:tasks.rados.rados.0.smithi125.stdout:2948: copy_from oid 28 from oid 6 current snap is 337 2021-03-24T22:05:35.688 INFO:tasks.rados.rados.0.smithi125.stdout:2944: finishing write tid 4 to smithi12556470-43 2021-03-24T22:05:35.688 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 43 v 1829 (ObjNum 1021 snap 337 seq_num 1021) dirty exists 2021-03-24T22:05:35.688 INFO:tasks.rados.rados.0.smithi125.stdout:2944: left oid 43 (ObjNum 1021 snap 337 seq_num 1021) 2021-03-24T22:05:35.689 INFO:tasks.rados.rados.0.smithi125.stdout:2946: finishing write tid 1 to smithi12556470-31 2021-03-24T22:05:35.689 INFO:tasks.rados.rados.0.smithi125.stdout:2946: finishing write tid 2 to smithi12556470-31 2021-03-24T22:05:35.689 INFO:tasks.rados.rados.0.smithi125.stdout:2946: finishing write tid 3 to smithi12556470-31 2021-03-24T22:05:35.689 INFO:tasks.rados.rados.0.smithi125.stdout:2946: finishing write tid 4 to smithi12556470-31 2021-03-24T22:05:35.689 INFO:tasks.rados.rados.0.smithi125.stdout:2946: finishing write tid 5 to smithi12556470-31 2021-03-24T22:05:35.690 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 31 v 1834 (ObjNum 1022 snap 337 seq_num 1022) dirty exists 2021-03-24T22:05:35.690 INFO:tasks.rados.rados.0.smithi125.stdout:2946: left oid 31 (ObjNum 1022 snap 337 seq_num 1022) 2021-03-24T22:05:35.690 INFO:tasks.rados.rados.0.smithi125.stdout:2944: done (2 left) 2021-03-24T22:05:35.690 INFO:tasks.rados.rados.0.smithi125.stdout:2946: done (1 left) 2021-03-24T22:05:35.691 INFO:tasks.rados.rados.0.smithi125.stdout:2949: write (excl) oid 47 current snap is 337 2021-03-24T22:05:35.691 INFO:tasks.rados.rados.0.smithi125.stdout:2949: seq_num 1024 ranges {410368=646278,1472267=475349,2504321=405600} 2021-03-24T22:05:35.692 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:35.694 INFO:tasks.rados.rados.0.smithi125.stdout:2949: writing smithi12556470-47 from 410368 to 1056646 tid 1 2021-03-24T22:05:35.701 INFO:tasks.rados.rados.0.smithi125.stdout:2949: writing smithi12556470-47 from 1472267 to 1947616 tid 2 2021-03-24T22:05:35.707 INFO:tasks.rados.rados.0.smithi125.stdout:2949: writing smithi12556470-47 from 2504321 to 2909921 tid 3 2021-03-24T22:05:35.707 INFO:tasks.rados.rados.0.smithi125.stdout:2950: read oid 42 snap 322 2021-03-24T22:05:35.707 INFO:tasks.rados.rados.0.smithi125.stdout:2950: expect deleted 2021-03-24T22:05:35.707 INFO:tasks.rados.rados.0.smithi125.stdout:2951: snap_remove snap 315 2021-03-24T22:05:35.708 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:35 smithi142 conmon[20125]: cluster 2021-03-24T22:05:34.596722+0000 mon.a (mon.0) 1957 : cluster 2021-03-24T22:05:35.708 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:35 smithi142 conmon[20125]: [DBG] osdmap e764: 8 total, 8 up, 8 in 2021-03-24T22:05:35.708 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:35 smithi142 conmon[20125]: cluster 2021-03-24T22:05:35.406335+0000 mon.a (mon.0) 1958 : cluster [DBG] osdmap e765: 8 total, 8 up, 8 in 2021-03-24T22:05:35.709 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:35 smithi142 conmon[20125]: 2021-03-24T22:05:35.746 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:35.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:35.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:05:35.849 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:35 smithi125 conmon[19936]: cluster 2021-03-24T22:05:34.596722 2021-03-24T22:05:35.849 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:35 smithi125 conmon[19936]: +0000 mon.a (mon.0) 1957 : cluster [DBG] osdmap e764: 8 total, 8 up, 8 in 2021-03-24T22:05:35.850 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:35 smithi125 conmon[19936]: cluster 2021-03-24T22:05:35.406335+0000 mon.a ( 2021-03-24T22:05:35.850 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:35 smithi125 conmon[19936]: mon.0) 1958 : cluster [DBG] osdmap e765: 8 total, 8 up, 8 in 2021-03-24T22:05:35.850 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:35 smithi125 conmon[24927]: cluster 2021-03-24T22:05: 2021-03-24T22:05:35.851 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:35 smithi125 conmon[24927]: 34.596722+0000 mon.a (mon.0) 1957 : cluster [DBG] osdmap e764: 8 total, 8 up, 8 in 2021-03-24T22:05:35.851 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:35 smithi125 conmon[24927]: cluster 2021-03-24T22:05:35.851 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:35 smithi125 conmon[24927]: 2021-03-24T22:05:35.406335+0000 2021-03-24T22:05:35.851 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:35 smithi125 conmon[24927]: mon.a (mon.0) 1958 : cluster [DBG] osdmap e765: 8 total, 8 up, 8 in 2021-03-24T22:05:35.852 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:35 smithi125 conmon[24927]: 2021-03-24T22:05:35.852 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:35.913 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:35.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:35.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:05:36.018 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:36.079 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:36.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:36.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:05:36.184 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:36.232 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:36.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:36.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:05:36.340 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:36.396 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:36.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:36.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:05:36.431 INFO:tasks.rados.rados.0.smithi125.stdout:2951: done (3 left) 2021-03-24T22:05:36.431 INFO:tasks.rados.rados.0.smithi125.stdout:2952: rollback oid 30 current snap is 337 2021-03-24T22:05:36.432 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 30 to 275 2021-03-24T22:05:36.432 INFO:tasks.rados.rados.0.smithi125.stdout:2948: finishing copy_from to smithi12556470-28 2021-03-24T22:05:36.432 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 28 v 659 (ObjNum 906 snap 300 seq_num 906) dirty exists 2021-03-24T22:05:36.432 INFO:tasks.rados.rados.0.smithi125.stdout:2948: finishing copy_from racing read to smithi12556470-28 2021-03-24T22:05:36.433 INFO:tasks.rados.rados.0.smithi125.stdout:2949: finishing write tid 1 to smithi12556470-47 2021-03-24T22:05:36.433 INFO:tasks.rados.rados.0.smithi125.stdout:2949: finishing write tid 2 to smithi12556470-47 2021-03-24T22:05:36.433 INFO:tasks.rados.rados.0.smithi125.stdout:2949: finishing write tid 3 to smithi12556470-47 2021-03-24T22:05:36.433 INFO:tasks.rados.rados.0.smithi125.stdout:2949: finishing write tid 4 to smithi12556470-47 2021-03-24T22:05:36.434 INFO:tasks.rados.rados.0.smithi125.stdout:2949: finishing write tid 5 to smithi12556470-47 2021-03-24T22:05:36.434 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 47 v 369 (ObjNum 1024 snap 337 seq_num 1024) dirty exists 2021-03-24T22:05:36.434 INFO:tasks.rados.rados.0.smithi125.stdout:2949: left oid 47 (ObjNum 1024 snap 337 seq_num 1024) 2021-03-24T22:05:36.434 INFO:tasks.rados.rados.0.smithi125.stdout:2948: done (3 left) 2021-03-24T22:05:36.435 INFO:tasks.rados.rados.0.smithi125.stdout:2949: done (2 left) 2021-03-24T22:05:36.435 INFO:tasks.rados.rados.0.smithi125.stdout:2950: done (1 left) 2021-03-24T22:05:36.435 INFO:tasks.rados.rados.0.smithi125.stdout:2953: write (excl) oid 5 current snap is 337 2021-03-24T22:05:36.435 INFO:tasks.rados.rados.0.smithi125.stdout:2953: seq_num 1025 ranges {427175=721527,1930749=735360} 2021-03-24T22:05:36.439 INFO:tasks.rados.rados.0.smithi125.stdout:2953: writing smithi12556470-5 from 427175 to 1148702 tid 1 2021-03-24T22:05:36.449 INFO:tasks.rados.rados.0.smithi125.stdout:2953: writing smithi12556470-5 from 1930749 to 2666109 tid 2 2021-03-24T22:05:36.450 INFO:tasks.rados.rados.0.smithi125.stdout:2954: rollback oid 46 current snap is 337 2021-03-24T22:05:36.451 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 46 to 293 2021-03-24T22:05:36.451 INFO:tasks.rados.rados.0.smithi125.stdout:2952: finishing rollback tid 1 to smithi12556470-30 2021-03-24T22:05:36.451 INFO:tasks.rados.rados.0.smithi125.stdout:2952: finishing rollback tid 2 to smithi12556470-30 2021-03-24T22:05:36.451 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 30 v 831 (ObjNum 764 snap 249 seq_num 764) dirty exists 2021-03-24T22:05:36.452 INFO:tasks.rados.rados.0.smithi125.stdout:2952: done (2 left) 2021-03-24T22:05:36.452 INFO:tasks.rados.rados.0.smithi125.stdout:2955: snap_create 2021-03-24T22:05:36.453 INFO:tasks.rados.rados.0.smithi125.stdout:2953: finishing write tid 1 to smithi12556470-5 2021-03-24T22:05:36.456 INFO:tasks.rados.rados.0.smithi125.stdout:2954: finishing rollback tid 0 to smithi12556470-46 2021-03-24T22:05:36.457 INFO:tasks.rados.rados.0.smithi125.stdout:2954: finishing rollback tid 1 to smithi12556470-46 2021-03-24T22:05:36.457 INFO:tasks.rados.rados.0.smithi125.stdout:2954: finishing rollback tid 2 to smithi12556470-46 2021-03-24T22:05:36.457 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 46 v 837 (ObjNum 497 snap 145 seq_num 497) dirty exists 2021-03-24T22:05:36.458 INFO:tasks.rados.rados.0.smithi125.stdout:2953: finishing write tid 2 to smithi12556470-5 2021-03-24T22:05:36.458 INFO:tasks.rados.rados.0.smithi125.stdout:2953: finishing write tid 3 to smithi12556470-5 2021-03-24T22:05:36.459 INFO:tasks.rados.rados.0.smithi125.stdout:2953: finishing write tid 4 to smithi12556470-5 2021-03-24T22:05:36.459 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 5 v 1136 (ObjNum 1025 snap 337 seq_num 1025) dirty exists 2021-03-24T22:05:36.460 INFO:tasks.rados.rados.0.smithi125.stdout:2953: left oid 5 (ObjNum 1025 snap 337 seq_num 1025) 2021-03-24T22:05:36.503 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:36.551 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:36.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:36.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:05:36.625 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:36 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:05:36.625 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:36 smithi125 conmon[24927]: 05:34.985317+0000 mgr.y (mgr.14144) 657 : cluster [DBG] pgmap v1309: 168 pgs: 168 active+clean; 484 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 264 KiB/s wr, 6 op/s 2021-03-24T22:05:36.626 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:36 smithi125 conmon[24927]: cluster 2021-03-24T22:05:35. 2021-03-24T22:05:36.626 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:36 smithi125 conmon[24927]: 470077+0000 osd.1 (osd.1) 84 : cluster 2021-03-24T22:05:36.626 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:36 smithi125 conmon[24927]: [DBG] purged_snaps scrub starts 2021-03-24T22:05:36.626 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:36 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T22:05:36.627 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:36 smithi125 conmon[24927]: -24T22:05:35. 2021-03-24T22:05:36.627 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:36 smithi125 conmon[24927]: 470912+0000 osd.1 (osd.1) 85 : cluster 2021-03-24T22:05:36.627 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:36 smithi125 conmon[24927]: [DBG] purged_snaps scrub ok 2021-03-24T22:05:36.627 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:36 smithi125 conmon[19936]: cluster 2021-03-24T22:05:34.985317+0000 2021-03-24T22:05:36.628 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:36 smithi125 conmon[19936]: mgr.y (mgr.14144) 657 : cluster [DBG] pgmap v1309: 168 pgs: 168 active+clean; 484 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 264 KiB/s wr, 6 op/s 2021-03-24T22:05:36.628 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:36 smithi125 conmon[19936]: cluster 2021-03-24T22:05:35.470077+0000 osd.1 (osd 2021-03-24T22:05:36.628 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:36 smithi125 conmon[19936]: .1) 84 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:05:36.628 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:36 smithi125 conmon[19936]: cluster 2021-03-24T22 2021-03-24T22:05:36.629 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:36 smithi125 conmon[19936]: :05:35.470912+0000 osd.1 (osd.1) 85 : 2021-03-24T22:05:36.629 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:36 smithi125 conmon[19936]: cluster [DBG] purged_snaps scrub ok 2021-03-24T22:05:36.629 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:36 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:05:36.629 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:36 smithi125 conmon[19936]: T22:05:36.422969+0000 mon.a (mon. 2021-03-24T22:05:36.629 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:36 smithi125 conmon[19936]: 0) 1959 : cluster [DBG] osdmap e766: 8 total, 8 up, 8 in 2021-03-24T22:05:36.658 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:36.712 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:36.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:36.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:05:36.820 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:36.877 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:36.881 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:36 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T22:05:36.882 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:36 smithi125 conmon[24927]: 22:05:36.422969+0000 mon.a (mon.0) 2021-03-24T22:05:36.882 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:36 smithi125 conmon[24927]: 1959 : cluster [DBG] osdmap e766: 8 total, 8 up, 8 in 2021-03-24T22:05:36.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:36.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:05:36.985 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:37.039 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:37.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:37.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:05:37.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:36 smithi142 conmon[20125]: cluster 2021-03-24T22:05:34.985317+0000 mgr.y (mgr.14144) 657 : 2021-03-24T22:05:37.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:36 smithi142 conmon[20125]: cluster [DBG] pgmap v1309: 168 pgs: 168 active+clean; 484 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 264 KiB/s wr, 6 op/s 2021-03-24T22:05:37.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:36 smithi142 conmon[20125]: cluster 2021-03-24T22:05:35.470077+0000 osd.1 (osd 2021-03-24T22:05:37.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:36 smithi142 conmon[20125]: .1) 84 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:05:37.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:36 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:05:37.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:36 smithi142 conmon[20125]: 22:05:35.470912+0000 osd.1 (osd 2021-03-24T22:05:37.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:36 smithi142 conmon[20125]: .1) 85 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:05:37.119 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:36 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:05:37.119 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:36 smithi142 conmon[20125]: -03-24T22:05:36 2021-03-24T22:05:37.120 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:36 smithi142 conmon[20125]: .422969+0000 mon.a (mon.0) 1959 : 2021-03-24T22:05:37.120 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:36 smithi142 conmon[20125]: cluster [DBG] osdmap e766: 8 total, 8 up, 8 in 2021-03-24T22:05:37.146 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:37.198 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:37.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:37.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:05:37.305 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:37.368 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:37.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:37.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:05:37.428 INFO:tasks.rados.rados.0.smithi125.stdout:2953: done (2 left) 2021-03-24T22:05:37.428 INFO:tasks.rados.rados.0.smithi125.stdout:2954: done (1 left) 2021-03-24T22:05:37.429 INFO:tasks.rados.rados.0.smithi125.stdout:2955: done (0 left) 2021-03-24T22:05:37.429 INFO:tasks.rados.rados.0.smithi125.stdout:2956: read oid 3 snap -1 2021-03-24T22:05:37.429 INFO:tasks.rados.rados.0.smithi125.stdout:2956: expect (ObjNum 1017 snap 337 seq_num 1017) 2021-03-24T22:05:37.429 INFO:tasks.rados.rados.0.smithi125.stdout:2957: delete oid 29 current snap is 338 2021-03-24T22:05:37.439 INFO:tasks.rados.rados.0.smithi125.stdout:2957: done (1 left) 2021-03-24T22:05:37.439 INFO:tasks.rados.rados.0.smithi125.stdout:2958: rollback oid 42 current snap is 338 2021-03-24T22:05:37.440 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 42 to 317 2021-03-24T22:05:37.440 INFO:tasks.rados.rados.0.smithi125.stdout:2959: snap_create 2021-03-24T22:05:37.440 INFO:tasks.rados.rados.0.smithi125.stdout:2958: finishing rollback tid 1 to smithi12556470-42 2021-03-24T22:05:37.441 INFO:tasks.rados.rados.0.smithi125.stdout:2958: finishing rollback tid 2 to smithi12556470-42 2021-03-24T22:05:37.441 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 42 v 512 (ObjNum 736 snap 235 seq_num 736) dirty exists 2021-03-24T22:05:37.442 INFO:tasks.rados.rados.0.smithi125.stdout:2956: expect (ObjNum 1017 snap 337 seq_num 1017) 2021-03-24T22:05:37.474 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:37.529 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:37.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:37.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:05:37.636 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:37.688 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:37.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:37.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:05:37.795 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:37.850 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:37.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:37.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:05:37.956 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:38.016 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:38.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:38.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:05:38.121 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:38.183 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:38.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:38.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:05:38.290 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:38.354 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:38.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:38.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:05:38.434 INFO:tasks.rados.rados.0.smithi125.stdout:2956: done (2 left) 2021-03-24T22:05:38.435 INFO:tasks.rados.rados.0.smithi125.stdout:2958: done (1 left) 2021-03-24T22:05:38.435 INFO:tasks.rados.rados.0.smithi125.stdout:2959: done (0 left) 2021-03-24T22:05:38.436 INFO:tasks.rados.rados.0.smithi125.stdout:2960: read oid 30 snap -1 2021-03-24T22:05:38.436 INFO:tasks.rados.rados.0.smithi125.stdout:2960: expect (ObjNum 764 snap 249 seq_num 764) 2021-03-24T22:05:38.436 INFO:tasks.rados.rados.0.smithi125.stdout:2961: snap_create 2021-03-24T22:05:38.439 INFO:tasks.rados.rados.0.smithi125.stdout:2960: expect (ObjNum 764 snap 249 seq_num 764) 2021-03-24T22:05:38.461 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:38.512 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:38.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:38.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:05:38.618 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:38.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:38 smithi125 conmon[19936]: cluster 2021-03-24T22:05:36.986168+0000 mgr.y (mgr.14144) 658 : cluster [DBG] pgmap v1312: 168 pgs: 168 active+clean; 488 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.2 MiB/s rd, 339 KiB/s wr, 7 op/s 2021-03-24T22:05:38.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:38 smithi125 conmon[19936]: cluster 2021-03-24T22:05:37.424253+0000 mon.a (mon. 2021-03-24T22:05:38.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:38 smithi125 conmon[19936]: 0) 1960 : cluster [DBG] osdmap e767: 8 total, 8 up, 8 in 2021-03-24T22:05:38.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:38 smithi125 conmon[19936]: cluster 2021-03-24T22:05:37.613271+0000 osd.4 (osd.4) 121 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:05:38.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:38 smithi125 conmon[19936]: cluster 2021-03-24T22:05:37.615553+0000 osd.4 (osd.4) 122 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:05:38.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:38 smithi125 conmon[24927]: cluster 2021-03-24T22:05:36.986168+0000 mgr.y (mgr. 2021-03-24T22:05:38.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:38 smithi125 conmon[24927]: 14144) 658 : cluster [DBG] pgmap v1312: 168 pgs: 168 active+clean; 488 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.2 MiB/s rd, 339 KiB/s wr, 7 op/s 2021-03-24T22:05:38.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:38 smithi125 conmon[24927]: cluster 2021-03-24T22:05:37.424253+0000 mon.a (mon.0) 1960 : cluster [DBG] osdmap e767: 8 total, 8 up, 8 in 2021-03-24T22:05:38.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:38 smithi125 conmon[24927]: cluster 2021-03-24T22:05:38.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:38 smithi125 conmon[24927]: 2021-03-24T22:05:37.613271+0000 osd.4 (osd.4) 121 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:05:38.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:38 smithi125 conmon[24927]: cluster 2021-03-24T22:05:38.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:38 smithi125 conmon[24927]: 2021-03-24T22:05:37.615553+0000 osd.4 (osd.4) 122 : cluster 2021-03-24T22:05:38.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:38 smithi125 conmon[24927]: [DBG] purged_snaps scrub ok 2021-03-24T22:05:38.673 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:38.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:38.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:05:38.778 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:38.833 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:38.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:38.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:05:38.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:38 smithi142 conmon[20125]: cluster 2021-03-24T22:05:36.986168+0000 2021-03-24T22:05:38.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:38 smithi142 conmon[20125]: mgr.y (mgr.14144) 658 : cluster [DBG] pgmap v1312: 168 pgs: 168 active+clean; 488 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.2 MiB/s rd, 339 KiB/s wr, 7 op/s 2021-03-24T22:05:38.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:38 smithi142 conmon[20125]: cluster 2021-03-24T22:05:37.424253+0000 mon.a (mon 2021-03-24T22:05:38.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:38 smithi142 conmon[20125]: .0) 1960 : cluster [DBG] osdmap e767: 8 total, 8 up, 8 in 2021-03-24T22:05:38.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:38 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:05:38.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:38 smithi142 conmon[20125]: -03-24T22:05 2021-03-24T22:05:38.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:38 smithi142 conmon[20125]: :37.613271+0000 osd.4 (osd. 2021-03-24T22:05:38.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:38 smithi142 conmon[20125]: 4) 121 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:05:38.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:38 smithi142 conmon[20125]: cluster 2021-03-24T22:05:38.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:38 smithi142 conmon[20125]: 2021-03-24T22: 2021-03-24T22:05:38.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:38 smithi142 conmon[20125]: 05:37.615553+0000 osd.4 (osd 2021-03-24T22:05:38.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:38 smithi142 conmon[20125]: .4) 122 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:05:38.937 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:38.993 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:38.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:38.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:05:39.100 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:39.153 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:39.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:39.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:05:39.258 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:39.319 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:39.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:39.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:05:39.425 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:39.435 INFO:tasks.rados.rados.0.smithi125.stdout:2960: done (1 left) 2021-03-24T22:05:39.435 INFO:tasks.rados.rados.0.smithi125.stdout:2961: done (0 left) 2021-03-24T22:05:39.436 INFO:tasks.rados.rados.0.smithi125.stdout:2962: snap_remove snap 338 2021-03-24T22:05:39.485 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:39.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:39.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:05:39.575 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:39 smithi125 conmon[19936]: cluster 2021-03-24T22:05:38.427009+0000 mon.a (mon 2021-03-24T22:05:39.575 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:39 smithi125 conmon[19936]: .0) 1961 : cluster [DBG] osdmap e768: 8 total, 8 up, 8 in 2021-03-24T22:05:39.576 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:39 smithi125 conmon[19936]: cluster 2021-03-24T22:05:39.430486+0000 mon.a (mon.0) 1962 : cluster [DBG] osdmap e769: 8 total, 8 up, 8 in 2021-03-24T22:05:39.576 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:39 smithi125 conmon[24927]: cluster 2021-03-24T22:05:39.577 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:39 smithi125 conmon[24927]: 2021-03-24T22:05:39.577 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:39 smithi125 conmon[24927]: 2021 2021-03-24T22:05:39.578 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:39 smithi125 conmon[24927]: - 2021-03-24T22:05:39.578 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:39 smithi125 conmon[24927]: 03 2021-03-24T22:05:39.578 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:39 smithi125 conmon[24927]: - 2021-03-24T22:05:39.579 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:39 smithi125 conmon[24927]: 24 2021-03-24T22:05:39.579 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:39 smithi125 conmon[24927]: T 2021-03-24T22:05:39.580 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:39 smithi125 conmon[24927]: 22 2021-03-24T22:05:39.580 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:39 smithi125 conmon[24927]: : 2021-03-24T22:05:39.581 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:39 smithi125 conmon[24927]: 05 2021-03-24T22:05:39.581 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:39 smithi125 conmon[24927]: : 2021-03-24T22:05:39.581 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:39 smithi125 conmon[24927]: 38 2021-03-24T22:05:39.582 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:39 smithi125 conmon[24927]: . 2021-03-24T22:05:39.582 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:39 smithi125 conmon[24927]: 427009 2021-03-24T22:05:39.583 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:39 smithi125 conmon[24927]: +0000 2021-03-24T22:05:39.583 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:39 smithi125 conmon[24927]: 2021-03-24T22:05:39.584 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:39 smithi125 conmon[24927]: mon.a 2021-03-24T22:05:39.584 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:39 smithi125 conmon[24927]: ( 2021-03-24T22:05:39.585 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:39 smithi125 conmon[24927]: mon 2021-03-24T22:05:39.585 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:39 smithi125 conmon[24927]: . 2021-03-24T22:05:39.586 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:39 smithi125 conmon[24927]: 0 2021-03-24T22:05:39.586 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:39 smithi125 conmon[24927]: ) 2021-03-24T22:05:39.586 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:39 smithi125 conmon[24927]: 1961 : cluster [DBG] osdmap e768: 8 total, 8 up, 8 in 2021-03-24T22:05:39.587 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:39 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:05:39.587 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:39 smithi125 conmon[24927]: -03-24T22:05:39.430486+0000 mon.a (mon.0) 1962 : 2021-03-24T22:05:39.588 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:39 smithi125 conmon[24927]: cluster [DBG] osdmap e769: 8 total, 8 up, 8 in 2021-03-24T22:05:39.592 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:39.653 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:39.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:39.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:05:39.758 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:39.813 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:39.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:39.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:05:39.861 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-24T22:05:39.862 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T22:05:39.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:39 smithi142 conmon[20125]: cluster 2021-03-24T22:05:38.427009+0000 2021-03-24T22:05:39.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:39 smithi142 conmon[20125]: mon.a (mon.0) 1961 : cluster [DBG] osdmap e768: 8 total, 8 up, 8 in 2021-03-24T22:05:39.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:39 smithi142 conmon[20125]: cluster 2021-03-24T22:05:39.430486+0000 mon.a (mon. 2021-03-24T22:05:39.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:39 smithi142 conmon[20125]: 0) 1962 : cluster [DBG] osdmap e769: 8 total, 8 up, 8 in 2021-03-24T22:05:39.920 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:39.975 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:39.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:39.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:05:40.081 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:40.142 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:40.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:40.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:05:40.248 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:40.309 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:40.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:40.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:05:40.411 INFO:tasks.rados.rados.0.smithi125.stdout:2962: done (0 left) 2021-03-24T22:05:40.412 INFO:tasks.rados.rados.0.smithi125.stdout:2963: snap_create 2021-03-24T22:05:40.415 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:40.463 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:40.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:40.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:05:40.568 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:40.575 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:40 smithi125 conmon[19936]: cluster 2021-03-24T22:05:38. 2021-03-24T22:05:40.613 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:40.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:40 smithi142 conmon[20125]: cluster 2021-03-24T22:05:38 2021-03-24T22:05:40.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:40 smithi142 conmon[20125]: .986751+0000 mgr.y (mgr.14144) 659 : cluster [DBG] pgmap v1315: 168 pgs: 168 active+clean; 503 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 15 op/s 2021-03-24T22:05:40.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:40 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:05:40.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:40 smithi142 conmon[20125]: -24T22:05:40.406788+0000 mon.a 2021-03-24T22:05:40.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:40 smithi142 conmon[20125]: (mon.0) 1963 : 2021-03-24T22:05:40.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:40 smithi142 conmon[20125]: cluster [DBG] osdmap e770: 8 total, 8 up, 8 in 2021-03-24T22:05:40.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:40.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:05:40.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:40 smithi125 conmon[19936]: 986751+0000 mgr.y (mgr.14144) 659 : cluster [DBG] pgmap v1315: 168 pgs: 168 active+clean; 503 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 15 op/s 2021-03-24T22:05:40.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:40 smithi125 conmon[19936]: cluster 2021-03-24T22:05:40.406788+0000 2021-03-24T22:05:40.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:40 smithi125 conmon[19936]: mon.a (mon.0) 1963 : cluster [DBG] osdmap e770: 8 total, 8 up, 8 in 2021-03-24T22:05:40.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:40 smithi125 conmon[24927]: cluster 2021-03-24T22:05:38 2021-03-24T22:05:40.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:40 smithi125 conmon[24927]: .986751+0000 mgr.y (mgr.14144) 659 : cluster [DBG] pgmap v1315: 168 pgs: 168 active+clean; 503 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 15 op/s 2021-03-24T22:05:40.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:40 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:05:40.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:40 smithi125 conmon[24927]: -03-24T22:05:40.406788+0000 mon.a (mon.0) 2021-03-24T22:05:40.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:40 smithi125 conmon[24927]: 1963 : cluster [DBG] osdmap e770: 8 total, 8 up, 8 in 2021-03-24T22:05:40.723 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:40.778 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:40.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:40.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:05:40.886 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:40.948 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:40.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:40.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:05:41.055 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:41.111 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:41.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:41.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:05:41.218 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:41.272 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:41.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:41.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:05:41.379 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:41.418 INFO:tasks.rados.rados.0.smithi125.stdout:2963: done (0 left) 2021-03-24T22:05:41.418 INFO:tasks.rados.rados.0.smithi125.stdout:2964: delete oid 8 current snap is 341 2021-03-24T22:05:41.421 INFO:teuthology.orchestra.run.smithi125.stderr:reweighted osd.4 to 1 (10000) 2021-03-24T22:05:41.427 INFO:tasks.rados.rados.0.smithi125.stdout:2964: done (0 left) 2021-03-24T22:05:41.427 INFO:tasks.rados.rados.0.smithi125.stdout:2965: snap_create 2021-03-24T22:05:41.440 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:41.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:41.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:05:41.516 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.1 dump_blocked_ops 2021-03-24T22:05:41.546 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:41.597 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:41.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:41.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:05:41.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:41 smithi142 conmon[20125]: audit 2021-03-24T22:05:40.570135 2021-03-24T22:05:41.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:41 smithi142 conmon[20125]: +0000 mon.c (mon.2) 110 : audit [INF] from='client.? v1:172.21.15.125:0/1545589325' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2021-03-24T22:05:41.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:41 smithi142 conmon[20125]: audit 2021-03-24T22:05:40.570626 2021-03-24T22:05:41.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:41 smithi142 conmon[20125]: +0000 mon.a (mon.0) 1964 : audit [INF] from='client.35783 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2021-03-24T22:05:41.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:41 smithi142 conmon[20125]: audit 2021-03-24T22:05:41 2021-03-24T22:05:41.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:41 smithi142 conmon[20125]: .307422+0000 mon.c (mon.2) 111 : audit 2021-03-24T22:05:41.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:41 smithi142 conmon[20125]: [DBG] from='client.? v1:172.21.15.125:0/3710140404' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:05:41.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:41 smithi142 conmon[20125]: audit 2021-03-24T22: 2021-03-24T22:05:41.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:41 smithi142 conmon[20125]: 05:41.410812+0000 mon.a (mon.0) 1965 : 2021-03-24T22:05:41.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:41 smithi142 conmon[20125]: audit [INF] from='client.35783 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2021-03-24T22:05:41.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:41 smithi142 conmon[20125]: cluster 2021-03-24T22: 2021-03-24T22:05:41.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:41 smithi142 conmon[20125]: 05:41.410942+0000 mon.a (mon.0) 1966 2021-03-24T22:05:41.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:41 smithi142 conmon[20125]: : cluster [DBG] osdmap e771: 8 total, 8 up, 8 in 2021-03-24T22:05:41.702 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:41 smithi125 conmon[19936]: audit 2021-03-24T22:05:40.570135+0000 mon.c (mon.2) 110 : audit [INF] from='client.? v1:172.21.15.125:0/1545589325' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2021-03-24T22:05:41.702 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:41 smithi125 conmon[19936]: audit 2021-03-24T22:05:40.570626+0000 mon.a (mon.0) 1964 : audit [INF] from='client.35783 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2021-03-24T22:05:41.702 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:41 smithi125 conmon[19936]: audit 2021-03-24T22:05:41.307422+0000 mon.c (mon.2) 111 : audit [DBG] from='client.? v1:172.21.15.125:0/3710140404' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:05:41.703 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:41 smithi125 conmon[19936]: audit 2021-03-24T22:05:41.410812+0000 mon.a (mon.0) 1965 : audit [INF] from='client.35783 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2021-03-24T22:05:41.703 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:41 smithi125 conmon[19936]: cluster 2021-03-24T22:05:41.410942+0000 mon.a (mon.0) 1966 : cluster [DBG] osdmap e771: 8 total, 8 up, 8 in 2021-03-24T22:05:41.703 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:41 smithi125 conmon[24927]: audit 2021-03-24T22:05 2021-03-24T22:05:41.704 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:41 smithi125 conmon[24927]: :40.570135+0000 mon.c (mon.2) 110 : audit [INF] from='client.? v1:172.21.15.125:0/1545589325' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2021-03-24T22:05:41.704 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:41 smithi125 conmon[24927]: audit 2021-03-24T 2021-03-24T22:05:41.704 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:41 smithi125 conmon[24927]: 22:05:40.570626+0000 mon.a 2021-03-24T22:05:41.704 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:41 smithi125 conmon[24927]: (mon.0) 1964 : audit [INF] 2021-03-24T22:05:41.704 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:41 smithi125 conmon[24927]: from='client.35783 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2021-03-24T22:05:41.705 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:41 smithi125 conmon[24927]: audit 2021-03-24 2021-03-24T22:05:41.705 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:41 smithi125 conmon[24927]: T22:05:41.307422+0000 2021-03-24T22:05:41.705 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:41 smithi125 conmon[24927]: mon.c (mon.2) 111 : audit 2021-03-24T22:05:41.705 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:41 smithi125 conmon[24927]: [DBG] from='client.? v1:172.21.15.125:0/3710140404' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:05:41.706 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:41 smithi125 conmon[24927]: audit 2021-03 2021-03-24T22:05:41.706 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:41 smithi125 conmon[24927]: -24T22:05:41.410812+0000 2021-03-24T22:05:41.706 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:41 smithi125 conmon[24927]: mon.a (mon.0) 1965 : 2021-03-24T22:05:41.706 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:41 smithi125 conmon[24927]: audit [INF] from='client.35783 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2021-03-24T22:05:41.706 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:41 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:05:41.707 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:41 smithi125 conmon[24927]: -03-24T22:05:41 2021-03-24T22:05:41.707 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:41 smithi125 conmon[24927]: .410942+0000 mon.a (mon.0) 1966 : 2021-03-24T22:05:41.707 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:41 smithi125 conmon[24927]: cluster [DBG] osdmap e771: 8 total, 8 up, 8 in 2021-03-24T22:05:41.708 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:41.764 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:41.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:41.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:05:41.869 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:41.919 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:41.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:41.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:05:42.026 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:42.042 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T22:05:42.043 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T22:05:42.043 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T22:05:42.043 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T22:05:42.043 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T22:05:42.043 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T22:05:42.044 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T22:05:42.044 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T22:05:42.044 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-24T22:05:42.045 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-24T22:05:42.045 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-24T22:05:42.045 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-24T22:05:42.045 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-24T22:05:42.046 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-24T22:05:42.046 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-24T22:05:42.046 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-24T22:05:42.082 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:42.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:42.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:05:42.187 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:42.245 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:42.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:42.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:05:42.352 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:42.413 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:42.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:42.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:05:42.419 INFO:tasks.rados.rados.0.smithi125.stdout:2965: done (0 left) 2021-03-24T22:05:42.419 INFO:tasks.rados.rados.0.smithi125.stdout:2966: read oid 9 snap -1 2021-03-24T22:05:42.420 INFO:tasks.rados.rados.0.smithi125.stdout:2966: expect deleted 2021-03-24T22:05:42.420 INFO:tasks.rados.rados.0.smithi125.stdout:2967: read oid 27 snap -1 2021-03-24T22:05:42.420 INFO:tasks.rados.rados.0.smithi125.stdout:2967: expect deleted 2021-03-24T22:05:42.420 INFO:tasks.rados.rados.0.smithi125.stdout:2968: copy_from oid 7 from oid 20 current snap is 342 2021-03-24T22:05:42.421 INFO:tasks.rados.rados.0.smithi125.stdout:2969: write (excl) oid 4 current snap is 342 2021-03-24T22:05:42.421 INFO:tasks.rados.rados.0.smithi125.stdout:2969: seq_num 1028 ranges {477596=663627,1706195=482687,2614013=567041} 2021-03-24T22:05:42.422 INFO:teuthology.orchestra.run.smithi125.stderr:noscrub is unset 2021-03-24T22:05:42.425 INFO:tasks.rados.rados.0.smithi125.stdout:2969: writing smithi12556470-4 from 477596 to 1141223 tid 1 2021-03-24T22:05:42.431 INFO:tasks.rados.rados.0.smithi125.stdout:2969: writing smithi12556470-4 from 1706195 to 2188882 tid 2 2021-03-24T22:05:42.442 INFO:tasks.rados.rados.0.smithi125.stdout:2969: writing smithi12556470-4 from 2614013 to 3181054 tid 3 2021-03-24T22:05:42.442 INFO:tasks.rados.rados.0.smithi125.stdout:2970: rollback oid 24 current snap is 342 2021-03-24T22:05:42.443 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 24 to 329 2021-03-24T22:05:42.443 INFO:tasks.rados.rados.0.smithi125.stdout:2969: finishing write tid 1 to smithi12556470-4 2021-03-24T22:05:42.443 INFO:tasks.rados.rados.0.smithi125.stdout:2969: finishing write tid 2 to smithi12556470-4 2021-03-24T22:05:42.443 INFO:tasks.rados.rados.0.smithi125.stdout:2966: done (4 left) 2021-03-24T22:05:42.444 INFO:tasks.rados.rados.0.smithi125.stdout:2967: done (3 left) 2021-03-24T22:05:42.444 INFO:tasks.rados.rados.0.smithi125.stdout:2971: read oid 21 snap -1 2021-03-24T22:05:42.444 INFO:tasks.rados.rados.0.smithi125.stdout:2971: expect (ObjNum 851 snap 278 seq_num 851) 2021-03-24T22:05:42.444 INFO:tasks.rados.rados.0.smithi125.stdout:2972: write oid 30 current snap is 342 2021-03-24T22:05:42.444 INFO:tasks.rados.rados.0.smithi125.stdout:2972: seq_num 1029 ranges {494403=738876,1764677=626345,3041436=252967} 2021-03-24T22:05:42.449 INFO:tasks.rados.rados.0.smithi125.stdout:2972: writing smithi12556470-30 from 494403 to 1233279 tid 1 2021-03-24T22:05:42.452 INFO:tasks.rados.rados.0.smithi125.stdout:2972: writing smithi12556470-30 from 1764677 to 2391022 tid 2 2021-03-24T22:05:42.457 INFO:tasks.rados.rados.0.smithi125.stdout:2972: writing smithi12556470-30 from 3041436 to 3294403 tid 3 2021-03-24T22:05:42.458 INFO:tasks.rados.rados.0.smithi125.stdout:2973: delete oid 33 current snap is 342 2021-03-24T22:05:42.462 INFO:tasks.rados.rados.0.smithi125.stdout:2973: done (5 left) 2021-03-24T22:05:42.462 INFO:tasks.rados.rados.0.smithi125.stdout:2974: snap_remove snap 301 2021-03-24T22:05:42.519 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:42.573 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:42.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:42.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:05:42.678 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:42.692 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:42 smithi125 conmon[19936]: cluster 2021-03-24T22:05:40.987429+0000 mgr.y ( 2021-03-24T22:05:42.692 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:42 smithi125 conmon[19936]: mgr.14144) 660 : cluster [DBG] pgmap v1318: 168 pgs: 168 active+clean; 510 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 5.5 MiB/s rd, 3.3 MiB/s wr, 19 op/s 2021-03-24T22:05:42.693 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:42 smithi125 conmon[19936]: audit 2021-03-24T22:05:41.859639+0000 mon.c (mon.2) 112 : audit [INF] from='client.? v1:172.21.15.125:0/2705060145' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T22:05:42.693 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:42 smithi125 conmon[19936]: audit 2021-03-24T22:05:41.860255+0000 mon.a (mon.0) 1967 2021-03-24T22:05:42.693 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:42 smithi125 conmon[19936]: : audit [INF] from='client.35795 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T22:05:42.693 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:42 smithi125 conmon[19936]: cluster 2021-03-24T22:05:42.694 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:42 smithi125 conmon[19936]: 2021-03-24T22:05:42.401648+0000 mon.a 2021-03-24T22:05:42.694 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:42 smithi125 conmon[19936]: (mon.0) 1968 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:05:42.694 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:42 smithi125 conmon[19936]: audit 2021 2021-03-24T22:05:42.694 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:42 smithi125 conmon[19936]: -03-24T22:05:42. 2021-03-24T22:05:42.694 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:42 smithi125 conmon[19936]: 410164+0000 mon.a (mon.0) 1969 : audit 2021-03-24T22:05:42.695 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:42 smithi125 conmon[19936]: [INF] from='client.35795 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T22:05:42.695 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:42 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T22:05:42.695 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:42 smithi125 conmon[19936]: -24T22:05:42.410339+0000 mon.a (mon. 2021-03-24T22:05:42.695 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:42 smithi125 conmon[19936]: 0) 1970 : cluster [DBG] osdmap e772: 8 total, 8 up, 8 in 2021-03-24T22:05:42.696 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:42 smithi125 conmon[24927]: cluster 2021-03-24T22:05:42.696 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:42 smithi125 conmon[24927]: 2021-03 2021-03-24T22:05:42.696 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:42 smithi125 conmon[24927]: -24T22:05:40.987429+0000 mgr.y (mgr.14144) 660 : cluster [DBG] pgmap v1318: 168 pgs: 168 active+clean; 510 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 5.5 MiB/s rd, 3.3 MiB/s wr, 19 op/s 2021-03-24T22:05:42.697 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:42 smithi125 conmon[24927]: audit 2021- 2021-03-24T22:05:42.697 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:42 smithi125 conmon[24927]: 03-24T22:05:41.859639+0000 mon.c (mon.2) 2021-03-24T22:05:42.697 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:42 smithi125 conmon[24927]: 112 : audit [INF] from='client.? v1:172.21.15.125:0/2705060145' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T22:05:42.697 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:42 smithi125 conmon[24927]: audit 2021 2021-03-24T22:05:42.698 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:42 smithi125 conmon[24927]: -03-24T22:05:41.860255+0000 mon.a ( 2021-03-24T22:05:42.698 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:42 smithi125 conmon[24927]: mon.0) 1967 : audit [INF] from='client.35795 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T22:05:42.698 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:42 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:05:42.698 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:42 smithi125 conmon[24927]: -03-24T22:05:42.401648+0000 mon.a ( 2021-03-24T22:05:42.699 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:42 smithi125 conmon[24927]: mon.0) 1968 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:05:42.699 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:42 smithi125 conmon[24927]: audit 2021- 2021-03-24T22:05:42.699 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:42 smithi125 conmon[24927]: 03-24T22:05:42.410164+0000 mon.a ( 2021-03-24T22:05:42.699 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:42 smithi125 conmon[24927]: mon.0) 1969 : audit [INF] from='client.35795 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T22:05:42.700 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:42 smithi125 conmon[24927]: cluster 2021-03-24T22:05:42.700 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:42 smithi125 conmon[24927]: 2021-03 2021-03-24T22:05:42.700 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:42 smithi125 conmon[24927]: -24T22:05:42.410339+0000 2021-03-24T22:05:42.700 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:42 smithi125 conmon[24927]: mon.a (mon.0) 1970 2021-03-24T22:05:42.701 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:42 smithi125 conmon[24927]: : cluster [DBG] osdmap e772: 8 total, 8 up, 8 in 2021-03-24T22:05:42.735 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:42.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:42.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:05:42.839 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:42.854 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:42 smithi142 conmon[20125]: cluster 2021-03-24T22:05:40.987429 2021-03-24T22:05:42.855 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:42 smithi142 conmon[20125]: +0000 mgr.y (mgr.14144) 660 : cluster [DBG] pgmap v1318: 168 pgs: 168 active+clean; 510 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 5.5 MiB/s rd, 3.3 MiB/s wr, 19 op/s 2021-03-24T22:05:42.855 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:42 smithi142 conmon[20125]: audit 2021-03-24T22:05:41.859639+0000 mon.c (mon.2) 112 : 2021-03-24T22:05:42.855 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:42 smithi142 conmon[20125]: audit [INF] from='client.? v1:172.21.15.125:0/2705060145' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T22:05:42.856 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:42 smithi142 conmon[20125]: audit 2021-03-24T22:05:42.856 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:42 smithi142 conmon[20125]: 2021 2021-03-24T22:05:42.856 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:42 smithi142 conmon[20125]: -03-24T22 2021-03-24T22:05:42.856 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:42 smithi142 conmon[20125]: :05:41.860255+0000 mon.a (mon. 2021-03-24T22:05:42.856 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:42 smithi142 conmon[20125]: 0) 1967 : audit [INF] 2021-03-24T22:05:42.857 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:42 smithi142 conmon[20125]: from='client.35795 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T22:05:42.857 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:42 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:05:42.857 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:42 smithi142 conmon[20125]: -03-24T22:05 2021-03-24T22:05:42.857 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:42 smithi142 conmon[20125]: :42.401648+0000 mon.a 2021-03-24T22:05:42.858 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:42 smithi142 conmon[20125]: (mon.0) 1968 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:05:42.858 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:42 smithi142 conmon[20125]: 2021-03-24T22:05:42.858 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:42 smithi142 conmon[20125]: audit 2021 2021-03-24T22:05:42.858 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:42 smithi142 conmon[20125]: -03-24T22:05:42.410164+0000 mon.a (mon.0) 1969 2021-03-24T22:05:42.858 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:42 smithi142 conmon[20125]: : audit [INF] from='client.35795 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T22:05:42.859 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:42 smithi142 conmon[20125]: cluster 2021-03-24T22:05:42.410339 2021-03-24T22:05:42.859 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:42 smithi142 conmon[20125]: +0000 mon.a (mon.0) 1970 : cluster [DBG] 2021-03-24T22:05:42.859 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:42 smithi142 conmon[20125]: osdmap e772: 8 total, 8 up, 8 in 2021-03-24T22:05:42.897 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:42.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:42.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:05:43.004 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:43.061 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:43.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:43.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:05:43.168 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:43.230 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:43.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:43.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:05:43.336 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:43.391 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:43.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:43.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:05:43.476 INFO:tasks.rados.rados.0.smithi125.stdout:2974: done (5 left) 2021-03-24T22:05:43.477 INFO:tasks.rados.rados.0.smithi125.stdout:2975: read oid 39 snap 296 2021-03-24T22:05:43.477 INFO:tasks.rados.rados.0.smithi125.stdout:2975: expect (ObjNum 856 snap 278 seq_num 856) 2021-03-24T22:05:43.477 INFO:tasks.rados.rados.0.smithi125.stdout:2976: rollback oid 16 current snap is 342 2021-03-24T22:05:43.477 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 16 to 340 2021-03-24T22:05:43.479 INFO:tasks.rados.rados.0.smithi125.stdout:2977: rollback oid 1 current snap is 342 2021-03-24T22:05:43.479 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 1 to 304 2021-03-24T22:05:43.479 INFO:tasks.rados.rados.0.smithi125.stdout:2978: write oid 35 current snap is 342 2021-03-24T22:05:43.479 INFO:tasks.rados.rados.0.smithi125.stdout:2978: seq_num 1031 ranges {528017=489374,1481641=630014,2696282=400857} 2021-03-24T22:05:43.483 INFO:tasks.rados.rados.0.smithi125.stdout:2978: writing smithi12556470-35 from 528017 to 1017391 tid 1 2021-03-24T22:05:43.491 INFO:tasks.rados.rados.0.smithi125.stdout:2978: writing smithi12556470-35 from 1481641 to 2111655 tid 2 2021-03-24T22:05:43.498 INFO:tasks.rados.rados.0.smithi125.stdout:2978: writing smithi12556470-35 from 2696282 to 3097139 tid 3 2021-03-24T22:05:43.498 INFO:tasks.rados.rados.0.smithi125.stdout:2979: copy_from oid 10 from oid 2 current snap is 342 2021-03-24T22:05:43.499 INFO:tasks.rados.rados.0.smithi125.stdout:2970: finishing rollback tid 0 to smithi12556470-24 2021-03-24T22:05:43.499 INFO:tasks.rados.rados.0.smithi125.stdout:2970: finishing rollback tid 1 to smithi12556470-24 2021-03-24T22:05:43.499 INFO:tasks.rados.rados.0.smithi125.stdout:2970: finishing rollback tid 2 to smithi12556470-24 2021-03-24T22:05:43.499 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 24 v 1451 (ObjNum 851 snap 278 seq_num 851) dirty exists 2021-03-24T22:05:43.499 INFO:tasks.rados.rados.0.smithi125.stdout:2969: finishing write tid 3 to smithi12556470-4 2021-03-24T22:05:43.500 INFO:tasks.rados.rados.0.smithi125.stdout:2969: finishing write tid 4 to smithi12556470-4 2021-03-24T22:05:43.500 INFO:tasks.rados.rados.0.smithi125.stdout:2969: finishing write tid 5 to smithi12556470-4 2021-03-24T22:05:43.500 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 4 v 723 (ObjNum 1028 snap 342 seq_num 1028) dirty exists 2021-03-24T22:05:43.500 INFO:tasks.rados.rados.0.smithi125.stdout:2969: left oid 4 (ObjNum 1028 snap 342 seq_num 1028) 2021-03-24T22:05:43.501 INFO:tasks.rados.rados.0.smithi125.stdout:2971: expect (ObjNum 851 snap 278 seq_num 851) 2021-03-24T22:05:43.501 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:43.548 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:43.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:43.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:05:43.652 INFO:tasks.rados.rados.0.smithi125.stdout:2968: finishing copy_from to smithi12556470-7 2021-03-24T22:05:43.653 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 7 v 895 (ObjNum 1013 snap 336 seq_num 1013) dirty exists 2021-03-24T22:05:43.653 INFO:tasks.rados.rados.0.smithi125.stdout:2968: finishing copy_from racing read to smithi12556470-7 2021-03-24T22:05:43.653 INFO:tasks.rados.rados.0.smithi125.stdout:2972: finishing write tid 1 to smithi12556470-30 2021-03-24T22:05:43.653 INFO:tasks.rados.rados.0.smithi125.stdout:2972: finishing write tid 2 to smithi12556470-30 2021-03-24T22:05:43.654 INFO:tasks.rados.rados.0.smithi125.stdout:2972: finishing write tid 3 to smithi12556470-30 2021-03-24T22:05:43.654 INFO:tasks.rados.rados.0.smithi125.stdout:2972: finishing write tid 4 to smithi12556470-30 2021-03-24T22:05:43.654 INFO:tasks.rados.rados.0.smithi125.stdout:2972: finishing write tid 5 to smithi12556470-30 2021-03-24T22:05:43.654 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 30 v 835 (ObjNum 1029 snap 342 seq_num 1029) dirty exists 2021-03-24T22:05:43.654 INFO:tasks.rados.rados.0.smithi125.stdout:2972: left oid 30 (ObjNum 1029 snap 342 seq_num 1029) 2021-03-24T22:05:43.655 INFO:tasks.rados.rados.0.smithi125.stdout:2976: finishing rollback tid 0 to smithi12556470-16 2021-03-24T22:05:43.655 INFO:tasks.rados.rados.0.smithi125.stdout:2975: expect (ObjNum 856 snap 278 seq_num 856) 2021-03-24T22:05:43.655 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:43.708 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:43.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:43.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:05:43.732 INFO:tasks.rados.rados.0.smithi125.stdout:2977: finishing rollback tid 0 to smithi12556470-1 2021-03-24T22:05:43.733 INFO:tasks.rados.rados.0.smithi125.stdout:2976: finishing rollback tid 1 to smithi12556470-16 2021-03-24T22:05:43.733 INFO:tasks.rados.rados.0.smithi125.stdout:2976: finishing rollback tid 2 to smithi12556470-16 2021-03-24T22:05:43.733 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 16 v 837 (ObjNum 875 snap 283 seq_num 875) dirty exists 2021-03-24T22:05:43.733 INFO:tasks.rados.rados.0.smithi125.stdout:2978: finishing write tid 1 to smithi12556470-35 2021-03-24T22:05:43.734 INFO:tasks.rados.rados.0.smithi125.stdout:2977: finishing rollback tid 1 to smithi12556470-1 2021-03-24T22:05:43.734 INFO:tasks.rados.rados.0.smithi125.stdout:2978: finishing write tid 2 to smithi12556470-35 2021-03-24T22:05:43.734 INFO:tasks.rados.rados.0.smithi125.stdout:2977: finishing rollback tid 2 to smithi12556470-1 2021-03-24T22:05:43.734 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 1 v 1443 (ObjNum 808 snap 265 seq_num 808) dirty exists 2021-03-24T22:05:43.734 INFO:tasks.rados.rados.0.smithi125.stdout:2978: finishing write tid 3 to smithi12556470-35 2021-03-24T22:05:43.735 INFO:tasks.rados.rados.0.smithi125.stdout:2978: finishing write tid 4 to smithi12556470-35 2021-03-24T22:05:43.735 INFO:tasks.rados.rados.0.smithi125.stdout:2978: finishing write tid 5 to smithi12556470-35 2021-03-24T22:05:43.735 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 35 v 238 (ObjNum 1031 snap 342 seq_num 1031) dirty exists 2021-03-24T22:05:43.735 INFO:tasks.rados.rados.0.smithi125.stdout:2978: left oid 35 (ObjNum 1031 snap 342 seq_num 1031) 2021-03-24T22:05:43.735 INFO:tasks.rados.rados.0.smithi125.stdout:2979: finishing copy_from to smithi12556470-10 2021-03-24T22:05:43.736 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 10 v 1114 (ObjNum 993 snap 329 seq_num 993) dirty exists 2021-03-24T22:05:43.736 INFO:tasks.rados.rados.0.smithi125.stdout:2979: finishing copy_from racing read to smithi12556470-10 2021-03-24T22:05:43.736 INFO:tasks.rados.rados.0.smithi125.stdout:2968: done (9 left) 2021-03-24T22:05:43.736 INFO:tasks.rados.rados.0.smithi125.stdout:2969: done (8 left) 2021-03-24T22:05:43.737 INFO:tasks.rados.rados.0.smithi125.stdout:2970: done (7 left) 2021-03-24T22:05:43.737 INFO:tasks.rados.rados.0.smithi125.stdout:2971: done (6 left) 2021-03-24T22:05:43.737 INFO:tasks.rados.rados.0.smithi125.stdout:2972: done (5 left) 2021-03-24T22:05:43.737 INFO:tasks.rados.rados.0.smithi125.stdout:2975: done (4 left) 2021-03-24T22:05:43.737 INFO:tasks.rados.rados.0.smithi125.stdout:2976: done (3 left) 2021-03-24T22:05:43.738 INFO:tasks.rados.rados.0.smithi125.stdout:2977: done (2 left) 2021-03-24T22:05:43.738 INFO:tasks.rados.rados.0.smithi125.stdout:2978: done (1 left) 2021-03-24T22:05:43.738 INFO:tasks.rados.rados.0.smithi125.stdout:2979: done (0 left) 2021-03-24T22:05:43.738 INFO:tasks.rados.rados.0.smithi125.stdout:2980: read oid 35 snap 307 2021-03-24T22:05:43.738 INFO:tasks.rados.rados.0.smithi125.stdout:2980: expect (ObjNum 908 snap 300 seq_num 908) 2021-03-24T22:05:43.739 INFO:tasks.rados.rados.0.smithi125.stdout:2981: delete oid 2 current snap is 342 2021-03-24T22:05:43.739 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:43 smithi125 conmon[24927]: cluster 2021-03-24T22:05 2021-03-24T22:05:43.740 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:43 smithi125 conmon[24927]: :41.527555+0000 osd.7 (osd.7) 125 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:05:43.740 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:43 smithi125 conmon[24927]: cluster 2021-03-24T22:05:43.740 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:43 smithi125 conmon[24927]: 2021-03-24T22:05:41.532499+0000 osd.7 (osd 2021-03-24T22:05:43.740 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:43 smithi125 conmon[24927]: .7) 126 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:05:43.741 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:43 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T22:05:43.741 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:43 smithi125 conmon[24927]: -24T22:05:43.068113+0000 2021-03-24T22:05:43.741 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:43 smithi125 conmon[24927]: osd.2 (osd.2) 118 : cluster [DBG] 2021-03-24T22:05:43.741 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:43 smithi125 conmon[24927]: 6.1c scrub ok 2021-03-24T22:05:43.742 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:43 smithi125 conmon[19936]: cluster 2021-03-24T22:05:41. 2021-03-24T22:05:43.742 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:43 smithi125 conmon[19936]: 527555+0000 osd.7 (osd.7) 125 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:05:43.742 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:43 smithi125 conmon[19936]: cluster 2021-03-24T22:05:41 2021-03-24T22:05:43.742 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:43 smithi125 conmon[19936]: .532499+0000 osd.7 (osd.7) 126 : cluster [DBG] 2021-03-24T22:05:43.743 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:43 smithi125 conmon[19936]: purged_snaps scrub ok 2021-03-24T22:05:43.743 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:43 smithi125 conmon[19936]: cluster 2021-03-24T22:05:43.743 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:43 smithi125 conmon[19936]: 2021-03-24T22:05: 2021-03-24T22:05:43.743 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:43 smithi125 conmon[19936]: 43.068113+0000 osd.2 (osd. 2021-03-24T22:05:43.743 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:43 smithi125 conmon[19936]: 2) 118 : cluster [DBG] 6.1c scrub ok 2021-03-24T22:05:43.744 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:43 smithi125 conmon[19936]: 2021-03-24T22:05:43.744 INFO:tasks.rados.rados.0.smithi125.stdout:2981: done (1 left) 2021-03-24T22:05:43.744 INFO:tasks.rados.rados.0.smithi125.stdout:2982: delete oid 27 current snap is 342 2021-03-24T22:05:43.745 INFO:tasks.rados.rados.0.smithi125.stdout:2980: expect (ObjNum 908 snap 300 seq_num 908) 2021-03-24T22:05:43.815 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:43.829 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:43 smithi142 conmon[20125]: cluster 2021-03-24T22:05 2021-03-24T22:05:43.829 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:43 smithi142 conmon[20125]: :41.527555+0000 osd.7 (osd.7) 125 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:05:43.829 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:43 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:05:43.829 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:43 smithi142 conmon[20125]: 22:05:41.532499+0000 osd.7 (osd.7) 126 : cluster 2021-03-24T22:05:43.830 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:43 smithi142 conmon[20125]: [DBG] purged_snaps scrub ok 2021-03-24T22:05:43.830 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:43 smithi142 conmon[20125]: cluster 2021-03-24T22: 2021-03-24T22:05:43.830 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:43 smithi142 conmon[20125]: 05:43.068113+0000 osd.2 (osd.2) 118 2021-03-24T22:05:43.830 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:43 smithi142 conmon[20125]: : cluster [DBG] 6.1c scrub ok 2021-03-24T22:05:43.855 INFO:tasks.rados.rados.0.smithi125.stdout:2980: done (1 left) 2021-03-24T22:05:43.856 INFO:tasks.rados.rados.0.smithi125.stdout:2982: done (0 left) 2021-03-24T22:05:43.856 INFO:tasks.rados.rados.0.smithi125.stdout:2983: rollback oid 28 current snap is 342 2021-03-24T22:05:43.856 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 28 to 340 2021-03-24T22:05:43.856 INFO:tasks.rados.rados.0.smithi125.stdout:2984: delete oid 25 current snap is 342 2021-03-24T22:05:43.860 INFO:tasks.rados.rados.0.smithi125.stdout:2984: done (1 left) 2021-03-24T22:05:43.861 INFO:tasks.rados.rados.0.smithi125.stdout:2985: copy_from oid 35 from oid 49 current snap is 342 2021-03-24T22:05:43.861 INFO:tasks.rados.rados.0.smithi125.stdout:2986: write (excl) oid 22 current snap is 342 2021-03-24T22:05:43.861 INFO:tasks.rados.rados.0.smithi125.stdout:2986: seq_num 1035 ranges {595245=506723,1715569=637352,2805974=562298} 2021-03-24T22:05:43.867 INFO:tasks.rados.rados.0.smithi125.stdout:2986: writing smithi12556470-22 from 595245 to 1101968 tid 1 2021-03-24T22:05:43.874 INFO:tasks.rados.rados.0.smithi125.stdout:2986: writing smithi12556470-22 from 1715569 to 2352921 tid 2 2021-03-24T22:05:43.880 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:43.881 INFO:tasks.rados.rados.0.smithi125.stdout:2986: writing smithi12556470-22 from 2805974 to 3368272 tid 3 2021-03-24T22:05:43.881 INFO:tasks.rados.rados.0.smithi125.stdout:2987: read oid 43 snap -1 2021-03-24T22:05:43.882 INFO:tasks.rados.rados.0.smithi125.stdout:2987: expect (ObjNum 1021 snap 337 seq_num 1021) 2021-03-24T22:05:43.882 INFO:tasks.rados.rados.0.smithi125.stdout:2988: snap_remove snap 319 2021-03-24T22:05:43.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:43.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:05:43.987 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:44.048 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:44.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:44.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:05:44.155 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:44.212 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:44.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:44.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:05:44.319 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:44.373 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:44.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:44.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:05:44.464 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd reweight 5 1 2021-03-24T22:05:44.478 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:44.486 INFO:tasks.rados.rados.0.smithi125.stdout:2988: done (4 left) 2021-03-24T22:05:44.486 INFO:tasks.rados.rados.0.smithi125.stdout:2989: write (excl) oid 46 current snap is 342 2021-03-24T22:05:44.486 INFO:tasks.rados.rados.0.smithi125.stdout:2983: finishing rollback tid 0 to smithi12556470-28 2021-03-24T22:05:44.486 INFO:tasks.rados.rados.0.smithi125.stdout:2989: seq_num 1036 ranges {612052=581972,1657698=497363,2717044=695008} 2021-03-24T22:05:44.494 INFO:tasks.rados.rados.0.smithi125.stdout:2989: writing smithi12556470-46 from 612052 to 1194024 tid 1 2021-03-24T22:05:44.501 INFO:tasks.rados.rados.0.smithi125.stdout:2989: writing smithi12556470-46 from 1657698 to 2155061 tid 2 2021-03-24T22:05:44.509 INFO:tasks.rados.rados.0.smithi125.stdout:2989: writing smithi12556470-46 from 2717044 to 3412052 tid 3 2021-03-24T22:05:44.510 INFO:tasks.rados.rados.0.smithi125.stdout:2990: delete oid 5 current snap is 342 2021-03-24T22:05:44.510 INFO:tasks.rados.rados.0.smithi125.stdout:2983: finishing rollback tid 1 to smithi12556470-28 2021-03-24T22:05:44.510 INFO:tasks.rados.rados.0.smithi125.stdout:2983: finishing rollback tid 2 to smithi12556470-28 2021-03-24T22:05:44.510 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 28 v 1205 (ObjNum 906 snap 300 seq_num 906) dirty exists 2021-03-24T22:05:44.511 INFO:tasks.rados.rados.0.smithi125.stdout:2986: finishing write tid 1 to smithi12556470-22 2021-03-24T22:05:44.511 INFO:tasks.rados.rados.0.smithi125.stdout:2986: finishing write tid 2 to smithi12556470-22 2021-03-24T22:05:44.511 INFO:tasks.rados.rados.0.smithi125.stdout:2985: finishing copy_from to smithi12556470-35 2021-03-24T22:05:44.511 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 35 v 971 (ObjNum 802 snap 262 seq_num 802) dirty exists 2021-03-24T22:05:44.512 INFO:tasks.rados.rados.0.smithi125.stdout:2985: finishing copy_from racing read to smithi12556470-35 2021-03-24T22:05:44.512 INFO:tasks.rados.rados.0.smithi125.stdout:2986: finishing write tid 3 to smithi12556470-22 2021-03-24T22:05:44.512 INFO:tasks.rados.rados.0.smithi125.stdout:2986: finishing write tid 4 to smithi12556470-22 2021-03-24T22:05:44.512 INFO:tasks.rados.rados.0.smithi125.stdout:2986: finishing write tid 5 to smithi12556470-22 2021-03-24T22:05:44.512 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 22 v 743 (ObjNum 1035 snap 342 seq_num 1035) dirty exists 2021-03-24T22:05:44.513 INFO:tasks.rados.rados.0.smithi125.stdout:2986: left oid 22 (ObjNum 1035 snap 342 seq_num 1035) 2021-03-24T22:05:44.513 INFO:tasks.rados.rados.0.smithi125.stdout:2987: expect (ObjNum 1021 snap 337 seq_num 1021) 2021-03-24T22:05:44.538 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:44.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:44.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:05:44.642 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:44.671 INFO:tasks.rados.rados.0.smithi125.stdout:2983: done (5 left) 2021-03-24T22:05:44.672 INFO:tasks.rados.rados.0.smithi125.stdout:2985: done (4 left) 2021-03-24T22:05:44.672 INFO:tasks.rados.rados.0.smithi125.stdout:2986: done (3 left) 2021-03-24T22:05:44.672 INFO:tasks.rados.rados.0.smithi125.stdout:2987: done (2 left) 2021-03-24T22:05:44.672 INFO:tasks.rados.rados.0.smithi125.stdout:2990: done (1 left) 2021-03-24T22:05:44.672 INFO:tasks.rados.rados.0.smithi125.stdout:2991: snap_create 2021-03-24T22:05:44.673 INFO:tasks.rados.rados.0.smithi125.stdout:2989: finishing write tid 1 to smithi12556470-46 2021-03-24T22:05:44.673 INFO:tasks.rados.rados.0.smithi125.stdout:2989: finishing write tid 2 to smithi12556470-46 2021-03-24T22:05:44.673 INFO:tasks.rados.rados.0.smithi125.stdout:2989: finishing write tid 3 to smithi12556470-46 2021-03-24T22:05:44.673 INFO:tasks.rados.rados.0.smithi125.stdout:2989: finishing write tid 4 to smithi12556470-46 2021-03-24T22:05:44.674 INFO:tasks.rados.rados.0.smithi125.stdout:2989: finishing write tid 5 to smithi12556470-46 2021-03-24T22:05:44.674 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 46 v 841 (ObjNum 1036 snap 342 seq_num 1036) dirty exists 2021-03-24T22:05:44.674 INFO:tasks.rados.rados.0.smithi125.stdout:2989: left oid 46 (ObjNum 1036 snap 342 seq_num 1036) 2021-03-24T22:05:44.689 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:44 smithi125 conmon[19936]: cluster 2021-03-24T22:05:42.988333+0000 mgr.y (mgr.14144) 661 : cluster [DBG] pgmap v1321: 168 pgs: 168 active+clean; 512 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 8.8 MiB/s rd, 1.0 MiB/s wr, 12 op/s 2021-03-24T22:05:44.689 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:44 smithi125 conmon[19936]: cluster 2021-03-24T22:05:43.282667+0000 osd.5 (osd.5) 7 : cluster [DBG] 3.16 scrub ok 2021-03-24T22:05:44.690 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:44 smithi125 conmon[19936]: cluster 2021-03-24T22:05:43.472039+0000 mon.a (mon.0) 1971 2021-03-24T22:05:44.690 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:44 smithi125 conmon[19936]: : cluster [DBG] osdmap e773: 8 total, 8 up, 8 in 2021-03-24T22:05:44.690 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:44 smithi125 conmon[24927]: cluster 2021-03-24T22:05: 2021-03-24T22:05:44.691 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:44 smithi125 conmon[24927]: 42.988333+0000 mgr.y (mgr.14144) 661 : cluster [DBG] pgmap v1321: 168 pgs: 168 active+clean; 512 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 8.8 MiB/s rd, 1.0 MiB/s wr, 12 op/s 2021-03-24T22:05:44.691 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:44 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:05:44.691 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:44 smithi125 conmon[24927]: 05:43.282667+0000 osd.5 (osd.5) 7 : cluster [DBG] 3.16 scrub ok 2021-03-24T22:05:44.691 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:44 smithi125 conmon[24927]: cluster 2021-03-24T22:05:44.691 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:44 smithi125 conmon[24927]: 2021-03-24T22:05:43.472039+0000 2021-03-24T22:05:44.692 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:44 smithi125 conmon[24927]: mon.a (mon.0) 1971 : cluster [DBG] osdmap e773: 8 total, 8 up, 8 in 2021-03-24T22:05:44.696 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:44.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:44.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:05:44.716 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:05:44.788 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:05:44.802 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:44.816 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:44 smithi142 conmon[20125]: cluster 2021-03-24T22:05:42. 2021-03-24T22:05:44.816 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:44 smithi142 conmon[20125]: 988333+0000 mgr.y (mgr.14144) 661 : cluster [DBG] pgmap v1321: 168 pgs: 168 active+clean; 512 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 8.8 MiB/s rd, 1.0 MiB/s wr, 12 op/s 2021-03-24T22:05:44.817 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:44 smithi142 conmon[20125]: cluster 2021-03-24T22:05:43.282667+0000 2021-03-24T22:05:44.817 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:44 smithi142 conmon[20125]: osd.5 (osd.5) 7 : cluster [DBG] 3.16 scrub ok 2021-03-24T22:05:44.817 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:44 smithi142 conmon[20125]: cluster 2021-03-24T22: 2021-03-24T22:05:44.817 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:44 smithi142 conmon[20125]: 05:43.472039+0000 mon.a (mon.0) 1971 2021-03-24T22:05:44.818 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:44 smithi142 conmon[20125]: : cluster [DBG] osdmap e773: 8 total, 8 up, 8 in 2021-03-24T22:05:44.865 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:44.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:44.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:05:44.972 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:45.028 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:45.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:45.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:05:45.132 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:45.194 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:45.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:45.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:05:45.300 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:45.363 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:45.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:45.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:05:45.414 INFO:tasks.rados.rados.0.smithi125.stdout:2989: done (1 left) 2021-03-24T22:05:45.415 INFO:tasks.rados.rados.0.smithi125.stdout:2991: done (0 left) 2021-03-24T22:05:45.415 INFO:tasks.rados.rados.0.smithi125.stdout:2992: read oid 46 snap -1 2021-03-24T22:05:45.415 INFO:tasks.rados.rados.0.smithi125.stdout:2992: expect (ObjNum 1036 snap 342 seq_num 1036) 2021-03-24T22:05:45.415 INFO:tasks.rados.rados.0.smithi125.stdout:2993: delete oid 3 current snap is 343 2021-03-24T22:05:45.422 INFO:tasks.rados.rados.0.smithi125.stdout:2993: done (1 left) 2021-03-24T22:05:45.422 INFO:tasks.rados.rados.0.smithi125.stdout:2994: copy_from oid 50 from oid 20 current snap is 343 2021-03-24T22:05:45.422 INFO:tasks.rados.rados.0.smithi125.stdout:2995: read oid 3 snap -1 2021-03-24T22:05:45.422 INFO:tasks.rados.rados.0.smithi125.stdout:2995: expect deleted 2021-03-24T22:05:45.423 INFO:tasks.rados.rados.0.smithi125.stdout:2996: read oid 14 snap 339 2021-03-24T22:05:45.423 INFO:tasks.rados.rados.0.smithi125.stdout:2996: expect (ObjNum 727 snap 233 seq_num 727) 2021-03-24T22:05:45.423 INFO:tasks.rados.rados.0.smithi125.stdout:2997: rollback oid 16 current snap is 343 2021-03-24T22:05:45.423 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 16 to 260 2021-03-24T22:05:45.424 INFO:tasks.rados.rados.0.smithi125.stdout:2998: delete oid 37 current snap is 343 2021-03-24T22:05:45.434 INFO:tasks.rados.rados.0.smithi125.stdout:2992: expect (ObjNum 1036 snap 342 seq_num 1036) 2021-03-24T22:05:45.469 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:45.523 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:45.526 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:45 smithi125 conmon[19936]: cluster 2021-03-24T22:05:44.265267+0000 osd.5 ( 2021-03-24T22:05:45.526 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:45 smithi125 conmon[19936]: osd.5) 8 : cluster [DBG] 4.1 scrub ok 2021-03-24T22:05:45.526 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:45 smithi125 conmon[19936]: cluster 2021-03-24T22:05:44.482548+0000 mon.a (mon.0) 1972 : cluster [DBG] osdmap e774: 8 total, 8 up, 8 in 2021-03-24T22:05:45.527 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:45 smithi125 conmon[19936]: cluster 2021-03-24T22:05:44.537805+0000 osd.1 (osd.1) 86 : cluster [DBG] 6.4 scrub ok 2021-03-24T22:05:45.527 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:45 smithi125 conmon[19936]: cluster 2021-03-24T22:05:45.410478+0000 mon.a (mon.0) 1973 : cluster [DBG] 2021-03-24T22:05:45.527 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:45 smithi125 conmon[19936]: osdmap e775: 8 total, 8 up, 8 in 2021-03-24T22:05:45.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:45.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:05:45.529 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:45 smithi125 conmon[24927]: cluster 2021-03-24T22:05 2021-03-24T22:05:45.529 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:45 smithi125 conmon[24927]: :44.265267+0000 osd.5 (osd.5) 8 : cluster [DBG] 4.1 scrub ok 2021-03-24T22:05:45.529 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:45 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:05:45.530 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:45 smithi125 conmon[24927]: -03-24T22:05:44.482548+0000 mon.a 2021-03-24T22:05:45.530 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:45 smithi125 conmon[24927]: (mon.0) 1972 : cluster [DBG] osdmap e774: 8 total, 8 up, 8 in 2021-03-24T22:05:45.530 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:45 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:05:45.530 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:45 smithi125 conmon[24927]: 03-24T22:05:44.537805+0000 osd.1 (osd.1 2021-03-24T22:05:45.531 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:45 smithi125 conmon[24927]: ) 86 : cluster [DBG] 6.4 scrub ok 2021-03-24T22:05:45.531 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:45 smithi125 conmon[24927]: cluster 2021-03-24T22:05:45.531 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:45 smithi125 conmon[24927]: 2021-03 2021-03-24T22:05:45.531 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:45 smithi125 conmon[24927]: -24T22:05:45.410478+0000 mon.a ( 2021-03-24T22:05:45.531 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:45 smithi125 conmon[24927]: mon.0) 1973 : cluster [DBG] osdmap e775: 8 total, 8 up, 8 in 2021-03-24T22:05:45.625 INFO:tasks.rados.rados.0.smithi125.stdout:2996: expect (ObjNum 727 snap 233 seq_num 727) 2021-03-24T22:05:45.630 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:45.682 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:45.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:45.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:05:45.789 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:45.808 INFO:tasks.rados.rados.0.smithi125.stdout:2997: finishing rollback tid 0 to smithi12556470-16 2021-03-24T22:05:45.808 INFO:tasks.rados.rados.0.smithi125.stdout:2997: finishing rollback tid 1 to smithi12556470-16 2021-03-24T22:05:45.808 INFO:tasks.rados.rados.0.smithi125.stdout:2997: finishing rollback tid 2 to smithi12556470-16 2021-03-24T22:05:45.808 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 16 v 840 (ObjNum 605 snap 184 seq_num 605) dirty exists 2021-03-24T22:05:45.809 INFO:tasks.rados.rados.0.smithi125.stdout:2994: finishing copy_from to smithi12556470-50 2021-03-24T22:05:45.809 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 50 v 895 (ObjNum 1013 snap 336 seq_num 1013) dirty exists 2021-03-24T22:05:45.809 INFO:tasks.rados.rados.0.smithi125.stdout:2994: finishing copy_from racing read to smithi12556470-50 2021-03-24T22:05:45.809 INFO:tasks.rados.rados.0.smithi125.stdout:2992: done (5 left) 2021-03-24T22:05:45.810 INFO:tasks.rados.rados.0.smithi125.stdout:2994: done (4 left) 2021-03-24T22:05:45.810 INFO:tasks.rados.rados.0.smithi125.stdout:2995: done (3 left) 2021-03-24T22:05:45.810 INFO:tasks.rados.rados.0.smithi125.stdout:2996: done (2 left) 2021-03-24T22:05:45.810 INFO:tasks.rados.rados.0.smithi125.stdout:2997: done (1 left) 2021-03-24T22:05:45.810 INFO:tasks.rados.rados.0.smithi125.stdout:2998: done (0 left) 2021-03-24T22:05:45.811 INFO:tasks.rados.rados.0.smithi125.stdout:2999: snap_remove snap 297 2021-03-24T22:05:45.845 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:45.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:45.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:05:45.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:45 smithi142 conmon[20125]: cluster 2021-03-24T22:05:44. 2021-03-24T22:05:45.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:45 smithi142 conmon[20125]: 265267+0000 osd.5 (osd.5) 8 : cluster [DBG] 4.1 scrub ok 2021-03-24T22:05:45.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:45 smithi142 conmon[20125]: cluster 2021-03-24T22:05:44.482548+0000 mon.a (mon.0) 1972 2021-03-24T22:05:45.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:45 smithi142 conmon[20125]: : cluster [DBG] osdmap e774: 8 total, 8 up, 8 in 2021-03-24T22:05:45.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:45 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:05:45.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:45 smithi142 conmon[20125]: -24T22:05:44.537805+0000 osd.1 (osd.1) 2021-03-24T22:05:45.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:45 smithi142 conmon[20125]: 86 : cluster [DBG] 6.4 scrub ok 2021-03-24T22:05:45.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:45 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:05:45.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:45 smithi142 conmon[20125]: -24T22:05:45.410478+0000 mon.a ( 2021-03-24T22:05:45.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:45 smithi142 conmon[20125]: mon.0) 1973 : cluster [DBG] osdmap e775: 8 total, 8 up, 8 in 2021-03-24T22:05:45.951 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:46.016 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:46.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:46.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:05:46.122 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:46.181 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:46.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:46.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:05:46.288 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:46.344 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:46.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:46.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:05:46.449 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:46.506 INFO:tasks.rados.rados.0.smithi125.stdout:2999: done (0 left) 2021-03-24T22:05:46.507 INFO:tasks.rados.rados.0.smithi125.stdout:3000: delete oid 9 current snap is 343 2021-03-24T22:05:46.508 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:46.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:46.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:05:46.517 INFO:tasks.rados.rados.0.smithi125.stdout:3000: done (0 left) 2021-03-24T22:05:46.517 INFO:tasks.rados.rados.0.smithi125.stdout:3001: snap_remove snap 342 2021-03-24T22:05:46.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:46 smithi142 conmon[20125]: cluster 2021-03-24T22:05:44.988971+0000 mgr.y (mgr 2021-03-24T22:05:46.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:46 smithi142 conmon[20125]: .14144) 662 : cluster [DBG] pgmap v1324: 168 pgs: 168 active+clean; 525 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.0 MiB/s wr, 19 op/s 2021-03-24T22:05:46.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:46 smithi142 conmon[20125]: cluster 2021-03-24T22:05:45.014986+0000 osd.3 (osd.3) 153 : cluster [DBG] 4.11 scrub ok 2021-03-24T22:05:46.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:46 smithi142 conmon[20125]: cluster 2021-03-24T22:05:46.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:46 smithi142 conmon[20125]: 2021-03-24T22:05:45.107880+0000 osd.7 (osd.7) 127 : cluster [DBG] 6.12 scrub ok 2021-03-24T22:05:46.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:46 smithi142 conmon[20125]: cluster 2021-03-24T22:05:46.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:46 smithi142 conmon[20125]: 2021-03-24T22:05:45.490281+0000 osd.1 (osd.1) 87 : 2021-03-24T22:05:46.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:46 smithi142 conmon[20125]: cluster [DBG] 6.1d scrub ok 2021-03-24T22:05:46.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:46 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:05:46.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:46 smithi142 conmon[20125]: 03-24T22:05:46.070343+0000 osd.2 (osd.2) 119 : cluster [DBG] 7.e scrub ok 2021-03-24T22:05:46.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:46 smithi142 conmon[20125]: 2021-03-24T22:05:46.620 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:46.669 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:46.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:46.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:05:46.680 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-24T22:05:46.712 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-24T22:05:46.739 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:46 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T22:05:46.739 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:46 smithi125 conmon[24927]: 22:05:44.988971+0000 mgr.y (mgr.14144) 662 : cluster [DBG] pgmap v1324: 168 pgs: 168 active+clean; 525 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.0 MiB/s wr, 19 op/s 2021-03-24T22:05:46.740 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:46 smithi125 conmon[24927]: cluster 2021-03-24T22:05:45. 2021-03-24T22:05:46.740 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:46 smithi125 conmon[24927]: 014986+0000 osd.3 (osd.3) 153 : cluster 2021-03-24T22:05:46.740 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:46 smithi125 conmon[24927]: [DBG] 4.11 scrub ok 2021-03-24T22:05:46.740 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:46 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:05:46.740 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:46 smithi125 conmon[24927]: 03-24T22:05:45.107880 2021-03-24T22:05:46.741 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:46 smithi125 conmon[24927]: +0000 osd.7 (osd.7) 127 : cluster 2021-03-24T22:05:46.741 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:46 smithi125 conmon[24927]: [DBG] 6.12 scrub ok 2021-03-24T22:05:46.741 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:46 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:05:46.741 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:46 smithi125 conmon[24927]: 03-24T22:05:45.490281 2021-03-24T22:05:46.742 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:46 smithi125 conmon[24927]: +0000 osd.1 (osd.1) 87 : cluster 2021-03-24T22:05:46.742 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:46 smithi125 conmon[24927]: [DBG] 6.1d scrub ok 2021-03-24T22:05:46.742 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:46 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T22:05:46.742 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:46 smithi125 conmon[24927]: -24T22:05:46.070343+0000 2021-03-24T22:05:46.742 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:46 smithi125 conmon[24927]: osd.2 (osd.2) 119 : cluster [DBG] 2021-03-24T22:05:46.743 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:46 smithi125 conmon[24927]: 7.e scrub ok 2021-03-24T22:05:46.743 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:46 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:05:46.743 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:46 smithi125 conmon[19936]: T22:05:44.988971+0000 mgr.y (mgr.14144) 662 : cluster [DBG] pgmap v1324: 168 pgs: 168 active+clean; 525 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.0 MiB/s wr, 19 op/s 2021-03-24T22:05:46.743 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:46 smithi125 conmon[19936]: cluster 2021-03-24T22:05:46.744 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:46 smithi125 conmon[19936]: 2021-03 2021-03-24T22:05:46.744 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:46 smithi125 conmon[19936]: -24T22:05:45.014986+0000 osd.3 ( 2021-03-24T22:05:46.744 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:46 smithi125 conmon[19936]: osd.3) 153 : cluster [DBG] 4.11 scrub ok 2021-03-24T22:05:46.744 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:46 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T22:05:46.745 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:46 smithi125 conmon[19936]: 22:05:45.107880+0000 osd.7 (osd. 2021-03-24T22:05:46.745 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:46 smithi125 conmon[19936]: 7) 127 : cluster [DBG] 6.12 scrub ok 2021-03-24T22:05:46.745 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:46 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:05:46.745 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:46 smithi125 conmon[19936]: 03-24T22:05:45.490281 2021-03-24T22:05:46.745 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:46 smithi125 conmon[19936]: +0000 osd.1 (osd.1) 87 : 2021-03-24T22:05:46.746 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:46 smithi125 conmon[19936]: cluster [DBG] 6.1d scrub ok 2021-03-24T22:05:46.746 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:46 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:05:46.746 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:46 smithi125 conmon[19936]: T22:05:46.070343+0000 2021-03-24T22:05:46.746 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:46 smithi125 conmon[19936]: osd.2 (osd.2) 119 : cluster [DBG] 2021-03-24T22:05:46.747 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:46 smithi125 conmon[19936]: 7.e scrub ok 2021-03-24T22:05:46.775 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:46.837 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:46.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:46.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:05:46.943 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:47.005 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:47.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:47.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:05:47.111 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:47.168 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:47.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:47.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:05:47.275 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:47.332 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:47.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:47.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:05:47.439 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:47.490 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:47.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:47.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:05:47.515 INFO:tasks.rados.rados.0.smithi125.stdout:3001: done (0 left) 2021-03-24T22:05:47.515 INFO:tasks.rados.rados.0.smithi125.stdout:3002: rollback oid 34 current snap is 343 2021-03-24T22:05:47.515 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 34 to 335 2021-03-24T22:05:47.516 INFO:tasks.rados.rados.0.smithi125.stdout:3003: write (excl) oid 32 current snap is 343 2021-03-24T22:05:47.516 INFO:tasks.rados.rados.0.smithi125.stdout:3003: seq_num 1041 ranges {696087=558217,1833755=648359,3137806=358281} 2021-03-24T22:05:47.524 INFO:tasks.rados.rados.0.smithi125.stdout:3003: writing smithi12556470-32 from 696087 to 1254304 tid 1 2021-03-24T22:05:47.532 INFO:tasks.rados.rados.0.smithi125.stdout:3003: writing smithi12556470-32 from 1833755 to 2482114 tid 2 2021-03-24T22:05:47.538 INFO:tasks.rados.rados.0.smithi125.stdout:3003: writing smithi12556470-32 from 3137806 to 3496087 tid 3 2021-03-24T22:05:47.539 INFO:tasks.rados.rados.0.smithi125.stdout:3004: write oid 14 current snap is 343 2021-03-24T22:05:47.539 INFO:tasks.rados.rados.0.smithi125.stdout:3002: finishing rollback tid 0 to smithi12556470-34 2021-03-24T22:05:47.539 INFO:tasks.rados.rados.0.smithi125.stdout:3004: seq_num 1042 ranges {712894=749819,2008590=792017,3281582=231312} 2021-03-24T22:05:47.546 INFO:tasks.rados.rados.0.smithi125.stdout:3004: writing smithi12556470-14 from 712894 to 1462713 tid 1 2021-03-24T22:05:47.554 INFO:tasks.rados.rados.0.smithi125.stdout:3004: writing smithi12556470-14 from 2008590 to 2800607 tid 2 2021-03-24T22:05:47.559 INFO:tasks.rados.rados.0.smithi125.stdout:3004: writing smithi12556470-14 from 3281582 to 3512894 tid 3 2021-03-24T22:05:47.559 INFO:tasks.rados.rados.0.smithi125.stdout:3002: finishing rollback tid 1 to smithi12556470-34 2021-03-24T22:05:47.559 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 34 v 1426 (ObjNum 1117669152 snap 21916 seq_num 1114771472) dirty dne 2021-03-24T22:05:47.559 INFO:tasks.rados.rados.0.smithi125.stdout:3003: finishing write tid 1 to smithi12556470-32 2021-03-24T22:05:47.560 INFO:tasks.rados.rados.0.smithi125.stdout:3003: finishing write tid 2 to smithi12556470-32 2021-03-24T22:05:47.560 INFO:tasks.rados.rados.0.smithi125.stdout:3003: finishing write tid 3 to smithi12556470-32 2021-03-24T22:05:47.560 INFO:tasks.rados.rados.0.smithi125.stdout:3003: finishing write tid 4 to smithi12556470-32 2021-03-24T22:05:47.560 INFO:tasks.rados.rados.0.smithi125.stdout:3003: finishing write tid 5 to smithi12556470-32 2021-03-24T22:05:47.560 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 32 v 1856 (ObjNum 1041 snap 343 seq_num 1041) dirty exists 2021-03-24T22:05:47.561 INFO:tasks.rados.rados.0.smithi125.stdout:3003: left oid 32 (ObjNum 1041 snap 343 seq_num 1041) 2021-03-24T22:05:47.561 INFO:tasks.rados.rados.0.smithi125.stdout:3004: finishing write tid 1 to smithi12556470-14 2021-03-24T22:05:47.561 INFO:tasks.rados.rados.0.smithi125.stdout:3002: done (2 left) 2021-03-24T22:05:47.561 INFO:tasks.rados.rados.0.smithi125.stdout:3003: done (1 left) 2021-03-24T22:05:47.562 INFO:tasks.rados.rados.0.smithi125.stdout:3005: copy_from oid 7 from oid 44 current snap is 343 2021-03-24T22:05:47.562 INFO:tasks.rados.rados.0.smithi125.stdout:3006: snap_create 2021-03-24T22:05:47.564 INFO:tasks.rados.rados.0.smithi125.stdout:3004: finishing write tid 2 to smithi12556470-14 2021-03-24T22:05:47.564 INFO:tasks.rados.rados.0.smithi125.stdout:3004: finishing write tid 3 to smithi12556470-14 2021-03-24T22:05:47.565 INFO:tasks.rados.rados.0.smithi125.stdout:3004: finishing write tid 4 to smithi12556470-14 2021-03-24T22:05:47.566 INFO:tasks.rados.rados.0.smithi125.stdout:3004: finishing write tid 5 to smithi12556470-14 2021-03-24T22:05:47.566 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 14 v 1861 (ObjNum 1042 snap 343 seq_num 1042) dirty exists 2021-03-24T22:05:47.566 INFO:tasks.rados.rados.0.smithi125.stdout:3004: left oid 14 (ObjNum 1042 snap 343 seq_num 1042) 2021-03-24T22:05:47.566 INFO:tasks.rados.rados.0.smithi125.stdout:3005: finishing copy_from to smithi12556470-7 2021-03-24T22:05:47.567 INFO:tasks.rados.rados.0.smithi125.stdout:3005: got expected ENOENT (src dne) 2021-03-24T22:05:47.567 INFO:tasks.rados.rados.0.smithi125.stdout:3005: finishing copy_from racing read to smithi12556470-7 2021-03-24T22:05:47.584 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:05:47.597 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:47.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:47 smithi142 conmon[20125]: cluster 2021-03-24T22:05:47.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:47 smithi142 conmon[20125]: 2021-03-24T22: 2021-03-24T22:05:47.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:47 smithi142 conmon[20125]: 05:46.303525+0000 osd.5 (osd.5) 9 : cluster [DBG] 6.1a scrub ok 2021-03-24T22:05:47.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:47 smithi142 conmon[20125]: cluster 2021-03-24T22:05:46.499134+0000 mon.a 2021-03-24T22:05:47.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:47 smithi142 conmon[20125]: (mon.0) 1974 : cluster [DBG] osdmap e776: 8 total, 8 up, 8 in 2021-03-24T22:05:47.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:47 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:05:47.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:47 smithi142 conmon[20125]: -03-24T22:05:47.047518 2021-03-24T22:05:47.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:47 smithi142 conmon[20125]: +0000 osd.2 (osd.2) 120 : cluster 2021-03-24T22:05:47.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:47 smithi142 conmon[20125]: [DBG] 5.d scrub ok 2021-03-24T22:05:47.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:47 smithi142 conmon[20125]: cluster 2021-03-24T22:05: 2021-03-24T22:05:47.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:47 smithi142 conmon[20125]: 47.360785+0000 osd.6 (osd.6) 75 2021-03-24T22:05:47.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:47 smithi142 conmon[20125]: : cluster [DBG] 3.10 scrub ok 2021-03-24T22:05:47.647 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:47.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:47.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:05:47.685 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:05:47.717 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:47 smithi125 conmon[19936]: cluster 2021-03-24T22:05:46. 2021-03-24T22:05:47.717 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:47 smithi125 conmon[19936]: 303525+0000 osd.5 (osd.5) 9 : cluster [DBG] 6.1a scrub ok 2021-03-24T22:05:47.718 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:47 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:05:47.718 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:47 smithi125 conmon[19936]: 24T22:05:46.499134+0000 mon.a (mon.0 2021-03-24T22:05:47.718 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:47 smithi125 conmon[19936]: ) 1974 : cluster [DBG] osdmap e776: 8 total, 8 up, 8 in 2021-03-24T22:05:47.718 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:47 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:05:47.719 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:47 smithi125 conmon[19936]: -03-24T22:05:47.047518 2021-03-24T22:05:47.719 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:47 smithi125 conmon[19936]: +0000 osd.2 (osd.2) 2021-03-24T22:05:47.719 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:47 smithi125 conmon[19936]: 120 : cluster [DBG] 5.d scrub ok 2021-03-24T22:05:47.719 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:47 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T22:05:47.720 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:47 smithi125 conmon[19936]: 22:05:47.360785+0000 osd.6 (osd.6) 2021-03-24T22:05:47.720 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:47 smithi125 conmon[19936]: 75 : cluster [DBG] 3.10 scrub ok 2021-03-24T22:05:47.720 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:47 smithi125 conmon[24927]: cluster 2021-03-24T22 2021-03-24T22:05:47.721 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:47 smithi125 conmon[24927]: :05:46.303525+0000 osd.5 (osd.5) 9 : cluster [DBG] 6.1a scrub ok 2021-03-24T22:05:47.721 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:47 smithi125 conmon[24927]: cluster 2021-03-24T22:05:47.722 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:47 smithi125 conmon[24927]: 2021-03-24T22:05:46. 2021-03-24T22:05:47.722 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:47 smithi125 conmon[24927]: 499134+0000 mon.a (mon.0) 1974 2021-03-24T22:05:47.722 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:47 smithi125 conmon[24927]: : cluster [DBG] osdmap e776: 8 total, 8 up, 8 in 2021-03-24T22:05:47.723 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:47 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:05:47.723 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:47 smithi125 conmon[24927]: -03-24T22:05:47.047518 2021-03-24T22:05:47.723 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:47 smithi125 conmon[24927]: +0000 osd.2 (osd.2) 120 : cluster 2021-03-24T22:05:47.723 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:47 smithi125 conmon[24927]: [DBG] 5.d scrub ok 2021-03-24T22:05:47.723 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:47 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:05:47.724 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:47 smithi125 conmon[24927]: 24T22:05:47.360785+0000 osd.6 2021-03-24T22:05:47.724 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:47 smithi125 conmon[24927]: (osd.6) 75 : cluster [DBG] 2021-03-24T22:05:47.724 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:47 smithi125 conmon[24927]: 3.10 scrub ok 2021-03-24T22:05:47.752 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:47.806 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:47.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:47.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:05:47.906 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-24T22:05:47.907 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-24T22:05:47.907 INFO:teuthology.orchestra.run.smithi125.stdout: "complaint_time": 30, 2021-03-24T22:05:47.907 INFO:teuthology.orchestra.run.smithi125.stdout: "num_blocked_ops": 0 2021-03-24T22:05:47.908 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-24T22:05:47.911 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:47.957 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:47.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:47.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:05:48.065 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:48.119 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:48.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:48.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:05:48.226 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:48.284 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:48.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:48.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:05:48.390 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:48.446 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:48.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:48.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:05:48.517 INFO:tasks.rados.rados.0.smithi125.stdout:3004: done (2 left) 2021-03-24T22:05:48.517 INFO:tasks.rados.rados.0.smithi125.stdout:3005: done (1 left) 2021-03-24T22:05:48.517 INFO:tasks.rados.rados.0.smithi125.stdout:3006: done (0 left) 2021-03-24T22:05:48.517 INFO:tasks.rados.rados.0.smithi125.stdout:3007: copy_from oid 4 from oid 20 current snap is 344 2021-03-24T22:05:48.518 INFO:tasks.rados.rados.0.smithi125.stdout:3008: snap_remove snap 288 2021-03-24T22:05:48.551 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:48.596 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:48.600 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[24927]: cluster 2021-03-24T22:05:48.601 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[24927]: 2021-03-24T22:05:48.601 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[24927]: 2021-03-24T22:05:46.989549+0000 mgr.y (mgr.14144) 663 : cluster [DBG] pgmap v1327: 168 pgs: 168 active+clean; 533 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 3.8 MiB/s wr, 22 op/s 2021-03-24T22:05:48.601 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[24927]: cluster 2021-03-24T22:05:47.006815+0000 osd.3 (osd.3) 154 : cluster [DBG] 4.10 scrub ok 2021-03-24T22:05:48.601 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[24927]: cluster 2021-03-24T22:05:47.327195+0000 osd.5 (osd.5) 10 : cluster [DBG] 6.d scrub ok 2021-03-24T22:05:48.602 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[24927]: cluster 2021-03-24T22:05:47.459857+0000 osd.0 (osd.0) 147 : cluster [DBG] 5.3 scrub ok 2021-03-24T22:05:48.602 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[24927]: cluster 2021-03-24T22:05:47.508382+0000 mon.a (mon.0) 1975 : cluster [DBG] osdmap e777: 8 total, 8 up, 8 in 2021-03-24T22:05:48.602 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[24927]: cluster 2021-03-24T22:05:48.075818+0000 osd.2 (osd.2) 121 : cluster [DBG] 3.1d scrub ok 2021-03-24T22:05:48.602 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[24927]: cluster 2021-03-24T22:05:48.406563+0000 osd.6 (osd.6 2021-03-24T22:05:48.603 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[24927]: ) 76 : cluster [DBG] 4.13 scrub ok 2021-03-24T22:05:48.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:48.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:05:48.605 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[19936]: cluster 2021-03-24T22:05:48.605 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[19936]: 2021-03-24T22:05:48.605 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[19936]: 2021 2021-03-24T22:05:48.605 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[19936]: - 2021-03-24T22:05:48.606 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[19936]: 03 2021-03-24T22:05:48.606 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[19936]: - 2021-03-24T22:05:48.606 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[19936]: 24 2021-03-24T22:05:48.606 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[19936]: T 2021-03-24T22:05:48.606 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[19936]: 22 2021-03-24T22:05:48.607 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[19936]: : 2021-03-24T22:05:48.607 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[19936]: 05 2021-03-24T22:05:48.607 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[19936]: : 2021-03-24T22:05:48.607 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[19936]: 46 2021-03-24T22:05:48.608 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[19936]: . 2021-03-24T22:05:48.608 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[19936]: 989549 2021-03-24T22:05:48.608 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[19936]: +0000 2021-03-24T22:05:48.608 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[19936]: 2021-03-24T22:05:48.608 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[19936]: mgr.y 2021-03-24T22:05:48.609 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[19936]: ( 2021-03-24T22:05:48.609 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[19936]: mgr 2021-03-24T22:05:48.609 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[19936]: . 2021-03-24T22:05:48.609 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[19936]: 14144 2021-03-24T22:05:48.610 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[19936]: ) 2021-03-24T22:05:48.610 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[19936]: 663 2021-03-24T22:05:48.610 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[19936]: : 2021-03-24T22:05:48.610 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[19936]: cluster 2021-03-24T22:05:48.610 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[19936]: 2021-03-24T22:05:48.611 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[19936]: [DBG] 2021-03-24T22:05:48.611 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[19936]: 2021-03-24T22:05:48.611 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[19936]: pgmap v1327: 168 pgs: 168 active+clean; 533 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 3.8 MiB/s wr, 22 op/s 2021-03-24T22:05:48.611 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[19936]: 2021-03-24T22:05:48.612 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T22:05:48.612 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[19936]: 22:05:47.006815+0000 osd.3 (osd.3) 154 : 2021-03-24T22:05:48.612 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[19936]: cluster [DBG] 4.10 scrub ok 2021-03-24T22:05:48.612 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[19936]: cluster 2021-03-24T22:05:47 2021-03-24T22:05:48.612 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[19936]: .327195+0000 osd.5 (osd.5) 10 : cluster 2021-03-24T22:05:48.613 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[19936]: [DBG] 6.d scrub ok 2021-03-24T22:05:48.613 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[19936]: cluster 2021-03-24T22:05:48.613 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[19936]: 2021-03-24T22:05:47.459857+0000 osd.0 (osd. 2021-03-24T22:05:48.613 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[19936]: 0) 147 : cluster [DBG] 5.3 scrub ok 2021-03-24T22:05:48.613 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:05:48.614 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[19936]: 05:47.508382+0000 mon.a (mon.0) 1975 : 2021-03-24T22:05:48.614 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[19936]: cluster [DBG] osdmap e777: 8 total, 8 up, 8 in 2021-03-24T22:05:48.614 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[19936]: cluster 2021-03-24T22:05: 2021-03-24T22:05:48.614 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[19936]: 48.075818+0000 osd.2 (osd.2) 121 : cluster 2021-03-24T22:05:48.615 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[19936]: [DBG] 3.1d scrub ok 2021-03-24T22:05:48.615 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[19936]: cluster 2021-03-24T22:05:48.406563 2021-03-24T22:05:48.615 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[19936]: +0000 osd.6 (osd.6) 76 : cluster [DBG] 4.13 scrub ok 2021-03-24T22:05:48.615 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:48 smithi125 conmon[19936]: 2021-03-24T22:05:48.705 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:48.757 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:48.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:48.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:05:48.863 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:48.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:48 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:05:48.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:48 smithi142 conmon[20125]: 03-24T22:05:46.989549+0000 mgr.y (mgr.14144) 663 : cluster [DBG] pgmap v1327: 168 pgs: 168 active+clean; 533 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 3.8 MiB/s wr, 22 op/s 2021-03-24T22:05:48.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:48 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:05:48.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:48 smithi142 conmon[20125]: -03-24T22:05:47.006815+0000 2021-03-24T22:05:48.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:48 smithi142 conmon[20125]: osd.3 (osd.3) 154 : cluster [DBG] 2021-03-24T22:05:48.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:48 smithi142 conmon[20125]: 4.10 scrub ok 2021-03-24T22:05:48.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:48 smithi142 conmon[20125]: cluster 2021-03-24T22:05:48.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:48 smithi142 conmon[20125]: 2021-03-24T22:05:47.327195+0000 2021-03-24T22:05:48.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:48 smithi142 conmon[20125]: osd.5 (osd.5) 10 : cluster [DBG] 6.d scrub ok 2021-03-24T22:05:48.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:48 smithi142 conmon[20125]: 2021-03-24T22:05:48.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:48 smithi142 conmon[20125]: cluster 2021-03-24T22:05:48.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:48 smithi142 conmon[20125]: 2021-03-24T22:05:47.459857+0000 osd.0 ( 2021-03-24T22:05:48.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:48 smithi142 conmon[20125]: osd.0) 147 : cluster [DBG] 5.3 scrub ok 2021-03-24T22:05:48.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:48 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:05:48.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:48 smithi142 conmon[20125]: -03-24T22:05:47.508382 2021-03-24T22:05:48.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:48 smithi142 conmon[20125]: +0000 mon.a (mon.0) 1975 : cluster [DBG] 2021-03-24T22:05:48.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:48 smithi142 conmon[20125]: osdmap e777: 8 total, 8 up, 8 in 2021-03-24T22:05:48.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:48 smithi142 conmon[20125]: cluster 2021-03-24T22:05: 2021-03-24T22:05:48.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:48 smithi142 conmon[20125]: 48.075818+0000 osd.2 (osd.2) 121 : 2021-03-24T22:05:48.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:48 smithi142 conmon[20125]: cluster [DBG] 3.1d scrub ok 2021-03-24T22:05:48.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:48 smithi142 conmon[20125]: cluster 2021-03-24T22: 2021-03-24T22:05:48.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:48 smithi142 conmon[20125]: 05:48.406563+0000 osd.6 (osd.6) 2021-03-24T22:05:48.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:48 smithi142 conmon[20125]: 76 : cluster [DBG] 4.13 scrub ok 2021-03-24T22:05:48.923 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:48.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:48.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:05:49.029 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:49.087 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:49.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:49.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:05:49.192 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:49.248 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:49.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:49.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:05:49.355 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:49.414 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:49.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:49.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:05:49.520 INFO:tasks.rados.rados.0.smithi125.stdout:3008: done (1 left) 2021-03-24T22:05:49.520 INFO:tasks.rados.rados.0.smithi125.stdout:3009: snap_create 2021-03-24T22:05:49.520 INFO:tasks.rados.rados.0.smithi125.stdout:3007: finishing copy_from to smithi12556470-4 2021-03-24T22:05:49.521 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 4 v 895 (ObjNum 1013 snap 336 seq_num 1013) dirty exists 2021-03-24T22:05:49.521 INFO:tasks.rados.rados.0.smithi125.stdout:3007: finishing copy_from racing read to smithi12556470-4 2021-03-24T22:05:49.536 INFO:teuthology.orchestra.run.smithi125.stderr:reweighted osd.5 to 1 (10000) 2021-03-24T22:05:49.552 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:49.595 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:49.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:49.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:05:49.701 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:49.764 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:49.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:49.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:05:49.849 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[19936]: cluster 2021-03-24T22:05:48.437753+0000 osd.0 (osd. 2021-03-24T22:05:49.850 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[19936]: 0) 148 : cluster [DBG] 3.1f scrub ok 2021-03-24T22:05:49.850 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[19936]: cluster 2021-03-24T22:05:48.511188+0000 mon.a (mon.0) 1976 : cluster [DBG] osdmap e778: 8 total, 8 up, 8 in 2021-03-24T22:05:49.850 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[19936]: audit 2021-03-24T22:05:48.667127+0000 mon.c (mon.2) 113 2021-03-24T22:05:49.851 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[19936]: : audit [INF] from='client.? v1:172.21.15.125:0/4212235963' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2021-03-24T22:05:49.851 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[19936]: audit 2021-03-24T22:05 2021-03-24T22:05:49.851 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[19936]: :48.667525+0000 mon.a (mon.0) 1977 : audit 2021-03-24T22:05:49.851 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[19936]: [INF] from='client.35801 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2021-03-24T22:05:49.852 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[19936]: cluster 2021-03-24T22:05 2021-03-24T22:05:49.852 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[19936]: :49.398333+0000 osd.6 (osd.6) 77 : 2021-03-24T22:05:49.852 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[19936]: cluster [DBG] 5.18 scrub ok 2021-03-24T22:05:49.852 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[19936]: audit 2021-03-24T22 2021-03-24T22:05:49.852 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[19936]: :05:49.517028+0000 mon.a (mon.0) 2021-03-24T22:05:49.853 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[19936]: 1978 : audit [INF] from='client.35801 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2021-03-24T22:05:49.853 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:05:49.853 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[19936]: T22:05:49.517118+0000 mon.a (mon. 2021-03-24T22:05:49.853 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[19936]: 0) 1979 : cluster [DBG] osdmap e779: 8 total, 8 up, 8 in 2021-03-24T22:05:49.854 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[24927]: cluster 2021-03-24T22:05:49.854 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[24927]: 2021-03-24T22:05:49.854 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[24927]: 2021 2021-03-24T22:05:49.854 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[24927]: - 2021-03-24T22:05:49.855 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[24927]: 03-24T22:05:48.437753+0000 osd.0 ( 2021-03-24T22:05:49.855 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[24927]: osd.0) 148 : cluster [DBG] 3.1f scrub ok 2021-03-24T22:05:49.855 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[24927]: cluster 2021-03-24T22:05:49.855 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[24927]: 2021-03-24T22: 2021-03-24T22:05:49.856 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[24927]: 05:48.511188 2021-03-24T22:05:49.856 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[24927]: +0000 mon.a (mon.0) 1976 : cluster [DBG] osdmap e778: 8 total, 8 up, 8 in 2021-03-24T22:05:49.856 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[24927]: audit 2021-03-24T22:05:49.856 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[24927]: 2021-03-24T22:05:49.856 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[24927]: 2021 2021-03-24T22:05:49.857 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[24927]: -03-24T22:05:48 2021-03-24T22:05:49.857 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[24927]: .667127+0000 mon.c (mon.2) 113 : audit [INF] from='client.? v1:172.21.15.125:0/4212235963' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2021-03-24T22:05:49.857 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[24927]: 2021-03-24T22:05:49.857 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T22:05:49.858 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[24927]: 24T22:05:48.667525+0000 mon.a (mon.0) 1977 : 2021-03-24T22:05:49.858 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[24927]: audit [INF] from='client.35801 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2021-03-24T22:05:49.858 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:05:49.858 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[24927]: 24T22:05:49.398333+0000 osd.6 (osd.6) 77 : cluster 2021-03-24T22:05:49.859 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[24927]: [DBG] 5.18 scrub ok 2021-03-24T22:05:49.859 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[24927]: audit 2021-03 2021-03-24T22:05:49.859 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[24927]: -24T22:05:49 2021-03-24T22:05:49.859 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[24927]: . 2021-03-24T22:05:49.859 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[24927]: 517028+0000 mon.a ( 2021-03-24T22:05:49.860 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[24927]: mon.0) 1978 : audit [INF] from='client.35801 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2021-03-24T22:05:49.860 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[24927]: cluster 2021-03-24T22:05:49.860 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[24927]: 2021-03-24T22:05:49.860 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[24927]: 2021 2021-03-24T22:05:49.860 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[24927]: - 2021-03-24T22:05:49.861 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[24927]: 03 2021-03-24T22:05:49.861 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[24927]: - 2021-03-24T22:05:49.861 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[24927]: 24 2021-03-24T22:05:49.861 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[24927]: T 2021-03-24T22:05:49.862 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[24927]: 22 2021-03-24T22:05:49.862 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[24927]: : 2021-03-24T22:05:49.862 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[24927]: 05 2021-03-24T22:05:49.862 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[24927]: : 2021-03-24T22:05:49.862 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[24927]: 49 2021-03-24T22:05:49.863 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[24927]: . 2021-03-24T22:05:49.863 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[24927]: 517118 2021-03-24T22:05:49.863 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[24927]: +0000 2021-03-24T22:05:49.863 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[24927]: mon.a (mon.0) 1979 : cluster [DBG] 2021-03-24T22:05:49.864 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:49 smithi125 conmon[24927]: osdmap e779: 8 total, 8 up, 8 in 2021-03-24T22:05:49.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:49 smithi142 conmon[20125]: cluster 2021-03-24T22:05:48.437753+0000 osd.0 (osd.0) 148 : cluster 2021-03-24T22:05:49.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:49 smithi142 conmon[20125]: [DBG] 3.1f scrub ok 2021-03-24T22:05:49.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:49 smithi142 conmon[20125]: cluster 2021-03-24T22:05:48.511188+0000 mon.a (mon.0) 1976 : cluster [DBG] osdmap e778: 8 total, 8 up, 8 in 2021-03-24T22:05:49.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:49 smithi142 conmon[20125]: audit 2021 2021-03-24T22:05:49.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:49 smithi142 conmon[20125]: -03-24T22:05:48.667127+0000 mon.c (mon.2) 113 : audit [INF] from='client.? v1:172.21.15.125:0/4212235963' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2021-03-24T22:05:49.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:49 smithi142 conmon[20125]: audit 2021-03 2021-03-24T22:05:49.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:49 smithi142 conmon[20125]: -24T22:05:48.667525+0000 mon.a (mon.0) 1977 : audit [INF] from='client.35801 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2021-03-24T22:05:49.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:49 smithi142 conmon[20125]: cluster 2021-03-24T22:05:49.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:49 smithi142 conmon[20125]: 2021-03-24T22:05:49.398333+0000 osd.6 (osd.6) 77 : cluster [DBG] 5.18 scrub ok 2021-03-24T22:05:49.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:49 smithi142 conmon[20125]: audit 2021- 2021-03-24T22:05:49.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:49 smithi142 conmon[20125]: 03-24T22:05:49.517028+0000 mon.a (mon.0) 1978 : audit [INF] from='client.35801 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2021-03-24T22:05:49.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:49 smithi142 conmon[20125]: cluster 2021-03-24T22:05:49.517118+0000 2021-03-24T22:05:49.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:49 smithi142 conmon[20125]: mon.a (mon.0) 1979 : cluster [DBG] osdmap e779: 8 total, 8 up, 8 in 2021-03-24T22:05:49.871 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:49.925 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:49.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:49.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:05:50.032 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:50.092 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:50.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:50.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:05:50.198 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:50.256 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:50.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:50.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:05:50.361 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:50.413 INFO:tasks.rados.rados.0.smithi125.stdout:3007: done (1 left) 2021-03-24T22:05:50.414 INFO:tasks.rados.rados.0.smithi125.stdout:3009: done (0 left) 2021-03-24T22:05:50.414 INFO:tasks.rados.rados.0.smithi125.stdout:3010: read oid 3 snap 339 2021-03-24T22:05:50.414 INFO:tasks.rados.rados.0.smithi125.stdout:3010: expect (ObjNum 1017 snap 337 seq_num 1017) 2021-03-24T22:05:50.414 INFO:tasks.rados.rados.0.smithi125.stdout:3011: rollback oid 6 current snap is 345 2021-03-24T22:05:50.415 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 6 to 343 2021-03-24T22:05:50.415 INFO:tasks.rados.rados.0.smithi125.stdout:3012: write (excl) oid 11 current snap is 345 2021-03-24T22:05:50.415 INFO:tasks.rados.rados.0.smithi125.stdout:3012: seq_num 1043 ranges {729701=425068,1667072=652028,2909005=485180} 2021-03-24T22:05:50.424 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:50.427 INFO:tasks.rados.rados.0.smithi125.stdout:3012: writing smithi12556470-11 from 729701 to 1154769 tid 1 2021-03-24T22:05:50.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:50.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:05:50.438 INFO:tasks.rados.rados.0.smithi125.stdout:3012: writing smithi12556470-11 from 1667072 to 2319100 tid 2 2021-03-24T22:05:50.446 INFO:tasks.rados.rados.0.smithi125.stdout:3012: writing smithi12556470-11 from 2909005 to 3394185 tid 3 2021-03-24T22:05:50.446 INFO:tasks.rados.rados.0.smithi125.stdout:3010: expect (ObjNum 1017 snap 337 seq_num 1017) 2021-03-24T22:05:50.533 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:50.557 INFO:tasks.rados.rados.0.smithi125.stdout:3011: finishing rollback tid 0 to smithi12556470-6 2021-03-24T22:05:50.557 INFO:tasks.rados.rados.0.smithi125.stdout:3011: finishing rollback tid 1 to smithi12556470-6 2021-03-24T22:05:50.558 INFO:tasks.rados.rados.0.smithi125.stdout:3011: finishing rollback tid 2 to smithi12556470-6 2021-03-24T22:05:50.558 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 6 v 723 (ObjNum 906 snap 300 seq_num 906) dirty exists 2021-03-24T22:05:50.558 INFO:tasks.rados.rados.0.smithi125.stdout:3012: finishing write tid 1 to smithi12556470-11 2021-03-24T22:05:50.558 INFO:tasks.rados.rados.0.smithi125.stdout:3012: finishing write tid 2 to smithi12556470-11 2021-03-24T22:05:50.558 INFO:tasks.rados.rados.0.smithi125.stdout:3012: finishing write tid 3 to smithi12556470-11 2021-03-24T22:05:50.559 INFO:tasks.rados.rados.0.smithi125.stdout:3012: finishing write tid 4 to smithi12556470-11 2021-03-24T22:05:50.559 INFO:tasks.rados.rados.0.smithi125.stdout:3010: done (2 left) 2021-03-24T22:05:50.559 INFO:tasks.rados.rados.0.smithi125.stdout:3011: done (1 left) 2021-03-24T22:05:50.559 INFO:tasks.rados.rados.0.smithi125.stdout:3013: snap_create 2021-03-24T22:05:50.560 INFO:tasks.rados.rados.0.smithi125.stdout:3012: finishing write tid 5 to smithi12556470-11 2021-03-24T22:05:50.560 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 11 v 566 (ObjNum 1043 snap 345 seq_num 1043) dirty exists 2021-03-24T22:05:50.560 INFO:tasks.rados.rados.0.smithi125.stdout:3012: left oid 11 (ObjNum 1043 snap 345 seq_num 1043) 2021-03-24T22:05:50.576 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:50.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:50.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:05:50.681 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:50.729 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:50 smithi125 conmon[19936]: cluster 2021-03-24T22:05 2021-03-24T22:05:50.729 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:50 smithi125 conmon[19936]: :48.990338+0000 mgr.y (mgr.14144) 664 : cluster [DBG] pgmap v1330: 168 pgs: 168 active+clean; 539 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 7.4 MiB/s rd, 2.3 MiB/s wr, 15 op/s 2021-03-24T22:05:50.730 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:50 smithi125 conmon[19936]: cluster 2021-03-24T22:05:50.730 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:50 smithi125 conmon[19936]: 2021-03-24T22:05:49.381549+0000 2021-03-24T22:05:50.730 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:50 smithi125 conmon[19936]: osd.5 (osd.5) 11 : cluster [DBG] 6.7 scrub ok 2021-03-24T22:05:50.730 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:50 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:05:50.730 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:50 smithi125 conmon[19936]: -03-24T22:05:49.560275+0000 osd.1 2021-03-24T22:05:50.731 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:50 smithi125 conmon[19936]: (osd.1) 88 : cluster [DBG] 3.1a scrub ok 2021-03-24T22:05:50.731 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:50 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:05:50.731 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:50 smithi125 conmon[19936]: -03-24T22:05:50.105607+0000 osd.7 ( 2021-03-24T22:05:50.731 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:50 smithi125 conmon[19936]: osd.7) 128 : cluster [DBG] 6.5 scrub ok 2021-03-24T22:05:50.732 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:50 smithi125 conmon[19936]: cluster 2021-03-24T22:05:50.732 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:50 smithi125 conmon[19936]: 2021-03-24T22:05:50.410191+0000 2021-03-24T22:05:50.732 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:50 smithi125 conmon[19936]: mon.a (mon.0) 1980 : cluster [DBG] 2021-03-24T22:05:50.732 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:50 smithi125 conmon[19936]: osdmap e780: 8 total, 8 up, 8 in 2021-03-24T22:05:50.733 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:50 smithi125 conmon[24927]: cluster 2021-03-24T22:05: 2021-03-24T22:05:50.733 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:50 smithi125 conmon[24927]: 48.990338+0000 mgr.y (mgr.14144) 664 : cluster [DBG] pgmap v1330: 168 pgs: 168 active+clean; 539 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 7.4 MiB/s rd, 2.3 MiB/s wr, 15 op/s 2021-03-24T22:05:50.733 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:50 smithi125 conmon[24927]: cluster 2021-03-24T22:05 2021-03-24T22:05:50.733 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:50 smithi125 conmon[24927]: :49.381549+0000 osd.5 (osd.5 2021-03-24T22:05:50.733 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:50 smithi125 conmon[24927]: ) 11 : cluster [DBG] 6.7 scrub ok 2021-03-24T22:05:50.734 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:50 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T22:05:50.734 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:50 smithi125 conmon[24927]: T22:05:49.560275+0000 osd.1 ( 2021-03-24T22:05:50.734 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:50 smithi125 conmon[24927]: osd.1) 88 : cluster [DBG] 3.1a scrub ok 2021-03-24T22:05:50.734 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:50 smithi125 conmon[24927]: 2021-03-24T22:05:50.734 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:50 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:05:50.735 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:50 smithi125 conmon[24927]: 03-24T22:05:50.105607+0000 osd.7 (osd 2021-03-24T22:05:50.735 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:50 smithi125 conmon[24927]: .7) 128 : cluster [DBG] 6.5 scrub ok 2021-03-24T22:05:50.735 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:50 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T22:05:50.735 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:50 smithi125 conmon[24927]: T22:05:50.410191+0000 mon.a 2021-03-24T22:05:50.736 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:50 smithi125 conmon[24927]: (mon.0) 1980 : cluster [DBG] 2021-03-24T22:05:50.736 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:50 smithi125 conmon[24927]: osdmap e780: 8 total, 8 up, 8 in 2021-03-24T22:05:50.740 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:50.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:50.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:05:50.761 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.1 dump_historic_ops 2021-03-24T22:05:50.847 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:50.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:50 smithi142 conmon[20125]: cluster 2021-03-24T22:05: 2021-03-24T22:05:50.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:50 smithi142 conmon[20125]: 48.990338+0000 mgr.y (mgr.14144) 664 : cluster [DBG] pgmap v1330: 168 pgs: 168 active+clean; 539 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 7.4 MiB/s rd, 2.3 MiB/s wr, 15 op/s 2021-03-24T22:05:50.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:50 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:05:50.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:50 smithi142 conmon[20125]: 22:05:49.381549+0000 osd.5 (osd.5) 11 : cluster [DBG] 6.7 scrub ok 2021-03-24T22:05:50.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:50 smithi142 conmon[20125]: 2021-03-24T22:05:50.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:50 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:05:50.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:50 smithi142 conmon[20125]: 03-24T22:05:49.560275+0000 osd.1 (osd. 2021-03-24T22:05:50.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:50 smithi142 conmon[20125]: 1) 88 : cluster [DBG] 3.1a scrub ok 2021-03-24T22:05:50.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:50 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:05:50.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:50 smithi142 conmon[20125]: 22:05:50.105607+0000 osd.7 (osd 2021-03-24T22:05:50.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:50 smithi142 conmon[20125]: .7) 128 : cluster [DBG] 6.5 scrub ok 2021-03-24T22:05:50.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:50 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:05:50.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:50 smithi142 conmon[20125]: T22:05:50.410191+0000 mon.a ( 2021-03-24T22:05:50.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:50 smithi142 conmon[20125]: mon.0) 1980 : cluster [DBG] osdmap e780: 8 total, 8 up, 8 in 2021-03-24T22:05:50.898 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:50.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:50.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:05:51.003 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:51.053 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:51.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:51.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:05:51.159 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:51.218 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:51.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:51.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:05:51.324 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:51.383 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:51.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:51.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:05:51.491 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:51.551 INFO:tasks.rados.rados.0.smithi125.stdout:3012: done (1 left) 2021-03-24T22:05:51.552 INFO:tasks.rados.rados.0.smithi125.stdout:3013: done (0 left) 2021-03-24T22:05:51.552 INFO:tasks.rados.rados.0.smithi125.stdout:3014: delete oid 44 current snap is 346 2021-03-24T22:05:51.554 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:51.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:51.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:05:51.559 INFO:tasks.rados.rados.0.smithi125.stdout:3014: done (0 left) 2021-03-24T22:05:51.559 INFO:tasks.rados.rados.0.smithi125.stdout:3015: delete oid 27 current snap is 346 2021-03-24T22:05:51.561 INFO:tasks.rados.rados.0.smithi125.stdout:3015: done (0 left) 2021-03-24T22:05:51.562 INFO:tasks.rados.rados.0.smithi125.stdout:3016: copy_from oid 33 from oid 41 current snap is 346 2021-03-24T22:05:51.562 INFO:tasks.rados.rados.0.smithi125.stdout:3017: snap_remove snap 321 2021-03-24T22:05:51.636 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:51 smithi125 conmon[24927]: cluster 2021-03-24T22:05:51.637 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:51 smithi125 conmon[24927]: 2021-03-24T22:05:51.659 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:51.722 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:51.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:51.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:05:51.828 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:51.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:51 smithi142 conmon[20125]: cluster 2021-03-24T22 2021-03-24T22:05:51.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:51 smithi142 conmon[20125]: :05:51.038072+0000 osd.2 (osd.2) 122 : cluster [DBG] 5.6 scrub ok 2021-03-24T22:05:51.878 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:51.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:51.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:05:51.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:51 smithi125 conmon[19936]: cluster 2021-03-24T22:05:51.038072+0000 osd.2 (osd.2) 122 : cluster [DBG] 5.6 scrub ok 2021-03-24T22:05:51.912 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:51 smithi125 conmon[24927]: 2021 2021-03-24T22:05:51.912 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:51 smithi125 conmon[24927]: - 2021-03-24T22:05:51.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:51 smithi125 conmon[24927]: 03 2021-03-24T22:05:51.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:51 smithi125 conmon[24927]: - 2021-03-24T22:05:51.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:51 smithi125 conmon[24927]: 24T22:05:51. 2021-03-24T22:05:51.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:51 smithi125 conmon[24927]: 038072+0000 osd.2 (osd.2) 122 : cluster [DBG] 5.6 scrub ok 2021-03-24T22:05:51.983 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:52.040 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:52.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:52.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:05:52.146 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:52.206 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:52.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:52.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:05:52.313 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:52.367 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:52.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:52.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:05:52.473 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:52.537 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:52.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:52.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:05:52.555 INFO:tasks.rados.rados.0.smithi125.stdout:3017: done (1 left) 2021-03-24T22:05:52.555 INFO:tasks.rados.rados.0.smithi125.stdout:3018: read oid 39 snap -1 2021-03-24T22:05:52.555 INFO:tasks.rados.rados.0.smithi125.stdout:3018: expect deleted 2021-03-24T22:05:52.556 INFO:tasks.rados.rados.0.smithi125.stdout:3019: snap_remove snap 318 2021-03-24T22:05:52.643 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:52.700 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:52.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:52.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:05:52.754 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:52 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T22:05:52.755 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:52 smithi125 conmon[19936]: 22:05:50.990905+0000 mgr.y (mgr.14144) 665 : cluster [DBG] pgmap v1333: 168 pgs: 168 active+clean; 545 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 2.7 MiB/s wr, 11 op/s 2021-03-24T22:05:52.755 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:52 smithi125 conmon[19936]: cluster 2021-03-24T22:05:52.755 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:52 smithi125 conmon[19936]: 2021-03-24T 2021-03-24T22:05:52.756 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:52 smithi125 conmon[19936]: 22:05:51 2021-03-24T22:05:52.756 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:52 smithi125 conmon[19936]: .326619+0000 2021-03-24T22:05:52.756 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:52 smithi125 conmon[19936]: osd.5 (osd.5) 2021-03-24T22:05:52.756 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:52 smithi125 conmon[19936]: 12 : cluster [DBG] 3.d scrub ok 2021-03-24T22:05:52.756 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:52 smithi125 conmon[19936]: 2021-03-24T22:05:52.757 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:52 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:05:52.757 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:52 smithi125 conmon[19936]: -03-24T22 2021-03-24T22:05:52.757 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:52 smithi125 conmon[19936]: :05:51. 2021-03-24T22:05:52.757 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:52 smithi125 conmon[19936]: 442465+0000 osd.0 ( 2021-03-24T22:05:52.758 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:52 smithi125 conmon[19936]: osd.0) 149 2021-03-24T22:05:52.758 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:52 smithi125 conmon[19936]: : cluster [DBG] 2021-03-24T22:05:52.758 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:52 smithi125 conmon[19936]: 3.1b scrub ok 2021-03-24T22:05:52.758 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:52 smithi125 conmon[19936]: cluster 2021-03-24T22:05:52.758 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:52 smithi125 conmon[19936]: 2021- 2021-03-24T22:05:52.759 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:52 smithi125 conmon[19936]: 03-24T22: 2021-03-24T22:05:52.759 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:52 smithi125 conmon[19936]: 05:51.542314 2021-03-24T22:05:52.759 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:52 smithi125 conmon[19936]: +0000 mon.a ( 2021-03-24T22:05:52.759 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:52 smithi125 conmon[19936]: mon.0) 1981 2021-03-24T22:05:52.760 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:52 smithi125 conmon[19936]: : cluster [DBG] 2021-03-24T22:05:52.760 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:52 smithi125 conmon[19936]: osdmap e781: 8 total, 8 up, 8 in 2021-03-24T22:05:52.760 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:52 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:05:52.760 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:52 smithi125 conmon[19936]: -03-24T22 2021-03-24T22:05:52.760 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:52 smithi125 conmon[19936]: :05: 2021-03-24T22:05:52.761 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:52 smithi125 conmon[19936]: 52.398719+0000 2021-03-24T22:05:52.761 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:52 smithi125 conmon[19936]: osd.0 (osd.0 2021-03-24T22:05:52.761 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:52 smithi125 conmon[19936]: ) 150 : cluster 2021-03-24T22:05:52.761 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:52 smithi125 conmon[19936]: [DBG] 4.3 scrub ok 2021-03-24T22:05:52.762 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:52 smithi125 conmon[24927]: cluster 2021-03-24T22:05:50.990905 2021-03-24T22:05:52.762 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:52 smithi125 conmon[24927]: +0000 mgr.y (mgr.14144) 665 : cluster [DBG] pgmap v1333: 168 pgs: 168 active+clean; 545 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 2.7 MiB/s wr, 11 op/s 2021-03-24T22:05:52.762 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:52 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T22:05:52.763 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:52 smithi125 conmon[24927]: -24T22:05:51.326619+0000 osd.5 (osd.5) 12 : cluster [DBG] 3.d scrub ok 2021-03-24T22:05:52.763 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:52 smithi125 conmon[24927]: cluster 2021-03-24T22:05:51.442465 2021-03-24T22:05:52.763 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:52 smithi125 conmon[24927]: +0000 osd.0 (osd.0) 149 : cluster [DBG] 3.1b scrub ok 2021-03-24T22:05:52.763 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:52 smithi125 conmon[24927]: cluster 2021-03-24T22:05: 2021-03-24T22:05:52.764 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:52 smithi125 conmon[24927]: 51.542314+0000 mon.a (mon.0) 1981 : cluster [DBG] osdmap e781: 8 total, 8 up, 8 in 2021-03-24T22:05:52.764 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:52 smithi125 conmon[24927]: cluster 2021-03-24T22:05 2021-03-24T22:05:52.764 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:52 smithi125 conmon[24927]: :52.398719+0000 osd.0 (osd.0) 150 : cluster [DBG] 4.3 scrub ok 2021-03-24T22:05:52.770 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:05:52.796 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd reweight 6 1 2021-03-24T22:05:52.804 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:52.858 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:52.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:52.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:05:52.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:52 smithi142 conmon[20125]: cluster 2021-03-24T22:05 2021-03-24T22:05:52.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:52 smithi142 conmon[20125]: :50.990905+0000 mgr.y (mgr.14144) 665 : cluster [DBG] pgmap v1333: 168 pgs: 168 active+clean; 545 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 2.7 MiB/s wr, 11 op/s 2021-03-24T22:05:52.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:52 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:05:52.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:52 smithi142 conmon[20125]: 22:05:51.326619+0000 osd.5 (osd.5) 12 : cluster [DBG] 2021-03-24T22:05:52.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:52 smithi142 conmon[20125]: 3.d scrub ok 2021-03-24T22:05:52.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:52 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:05:52.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:52 smithi142 conmon[20125]: 03-24T22:05:51.442465+0000 osd.0 (osd 2021-03-24T22:05:52.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:52 smithi142 conmon[20125]: .0) 149 : cluster [DBG] 3.1b scrub ok 2021-03-24T22:05:52.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:52 smithi142 conmon[20125]: 2021-03-24T22:05:52.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:52 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:05:52.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:52 smithi142 conmon[20125]: -03-24T22:05:51.542314+0000 mon.a (mon. 2021-03-24T22:05:52.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:52 smithi142 conmon[20125]: 0) 1981 : cluster [DBG] osdmap e781: 8 total, 8 up, 8 in 2021-03-24T22:05:52.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:52 smithi142 conmon[20125]: cluster 2021-03-24T22:05:52.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:52 smithi142 conmon[20125]: 2021- 2021-03-24T22:05:52.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:52 smithi142 conmon[20125]: 03-24T22:05:52 2021-03-24T22:05:52.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:52 smithi142 conmon[20125]: .398719+0000 osd.0 (osd. 2021-03-24T22:05:52.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:52 smithi142 conmon[20125]: 0) 150 : cluster [DBG] 2021-03-24T22:05:52.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:52 smithi142 conmon[20125]: 4.3 scrub ok 2021-03-24T22:05:52.964 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:53.018 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:53.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:53.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:05:53.126 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:53.171 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:53.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:53.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:05:53.276 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:53.334 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:53.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:53.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:05:53.438 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:53.486 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:53.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:53.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:05:53.574 INFO:tasks.rados.rados.0.smithi125.stdout:3019: done (2 left) 2021-03-24T22:05:53.575 INFO:tasks.rados.rados.0.smithi125.stdout:3020: write oid 46 current snap is 346 2021-03-24T22:05:53.575 INFO:tasks.rados.rados.0.smithi125.stdout:3020: seq_num 1046 ranges {780122=650815,2126165=799355} 2021-03-24T22:05:53.576 INFO:teuthology.orchestra.run.smithi125.stderr:nodeep-scrub is unset 2021-03-24T22:05:53.578 INFO:tasks.rados.rados.0.smithi125.stdout:3020: writing smithi12556470-46 from 780122 to 1430937 tid 1 2021-03-24T22:05:53.588 INFO:tasks.rados.rados.0.smithi125.stdout:3020: writing smithi12556470-46 from 2126165 to 2925520 tid 2 2021-03-24T22:05:53.588 INFO:tasks.rados.rados.0.smithi125.stdout:3021: write oid 20 current snap is 346 2021-03-24T22:05:53.588 INFO:tasks.rados.rados.0.smithi125.stdout:3016: finishing copy_from to smithi12556470-33 2021-03-24T22:05:53.589 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 33 v 1133 (ObjNum 916 snap 304 seq_num 916) dirty exists 2021-03-24T22:05:53.589 INFO:tasks.rados.rados.0.smithi125.stdout:3016: finishing copy_from racing read to smithi12556470-33 2021-03-24T22:05:53.590 INFO:tasks.rados.rados.0.smithi125.stdout:3021: seq_num 1047 ranges {796929=726064,2184647=659366,3302344=294585} 2021-03-24T22:05:53.592 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:53.595 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:53 smithi125 conmon[19936]: cluster 2021-03-24T22:05:52.372022 2021-03-24T22:05:53.595 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:53 smithi125 conmon[19936]: +0000 osd.5 (osd.5) 13 : cluster [DBG] 5.1b scrub ok 2021-03-24T22:05:53.596 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:53 smithi125 conmon[19936]: cluster 2021-03-24T22:05:52.552636+0000 mon.a (mon.0) 1982 : cluster [DBG] 2021-03-24T22:05:53.596 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:53 smithi125 conmon[19936]: osdmap e782: 8 total, 8 up, 8 in 2021-03-24T22:05:53.596 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:53 smithi125 conmon[19936]: cluster 2021-03-24T22:05:53.597 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:53 smithi125 conmon[19936]: 2021-03-24T22:05:52.854925+0000 osd.3 ( 2021-03-24T22:05:53.597 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:53 smithi125 conmon[19936]: osd.3) 155 : cluster [DBG] 6.11 scrub ok 2021-03-24T22:05:53.598 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:53 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:05:53.598 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:53 smithi125 conmon[19936]: 24T22:05:53.144220+0000 osd.7 2021-03-24T22:05:53.599 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:53 smithi125 conmon[19936]: (osd.7) 129 : cluster [DBG] 2021-03-24T22:05:53.599 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:53 smithi125 conmon[19936]: 3.15 scrub ok 2021-03-24T22:05:53.599 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:53 smithi125 conmon[19936]: audit 2021-03-24T22:05:53.600 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:53 smithi125 conmon[19936]: 2021-03-24T22:05:53.366896+0000 2021-03-24T22:05:53.600 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:53 smithi125 conmon[19936]: mon.c (mon.2) 114 : audit [INF] from='client.? v1:172.21.15.125:0/3190559375' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:05:53.601 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:53 smithi125 conmon[19936]: audit 2021-03-24T22:05:53.601 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:53 smithi125 conmon[19936]: 2021-03-24T22:05:53.367691 2021-03-24T22:05:53.601 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:53 smithi125 conmon[19936]: +0000 mon.a (mon.0) 1983 : audit [INF] from='client.35807 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:05:53.602 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:53 smithi125 conmon[19936]: audit 2021 2021-03-24T22:05:53.602 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:53 smithi125 conmon[19936]: -03-24T22:05:53. 2021-03-24T22:05:53.603 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:53 smithi125 conmon[19936]: 548201+0000 mon.a (mon.0) 1984 : audit 2021-03-24T22:05:53.603 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:53 smithi125 conmon[19936]: [DBG] from='client.? v1:172.21.15.125:0/759929309' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:05:53.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:53 smithi125 conmon[24927]: cluster 2021-03-24T22:05: 2021-03-24T22:05:53.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:53 smithi125 conmon[24927]: 52.372022+0000 osd.5 (osd.5) 13 : cluster [DBG] 5.1b scrub ok 2021-03-24T22:05:53.604 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:53 smithi125 conmon[24927]: cluster 2021-03-24T22:05:52.552636+0000 2021-03-24T22:05:53.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:53 smithi125 conmon[24927]: mon.a (mon.0) 1982 : cluster [DBG] osdmap e782: 8 total, 8 up, 8 in 2021-03-24T22:05:53.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:53 smithi125 conmon[24927]: cluster 2021-03-24T22:05:53.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:53 smithi125 conmon[24927]: 2021-03-24T22:05:52.854925+0000 2021-03-24T22:05:53.605 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:53 smithi125 conmon[24927]: osd.3 (osd.3) 155 : cluster [DBG] 6.11 scrub ok 2021-03-24T22:05:53.606 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:53 smithi125 conmon[24927]: 2021-03-24T22:05:53.606 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:53 smithi125 conmon[24927]: cluster 2021-03-24T22:05:53.606 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:53 smithi125 conmon[24927]: 2021-03-24T22:05:53.144220 2021-03-24T22:05:53.606 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:53 smithi125 conmon[24927]: +0000 osd.7 (osd.7) 129 : cluster [DBG] 2021-03-24T22:05:53.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:53 smithi125 conmon[24927]: 3.15 scrub ok 2021-03-24T22:05:53.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:53 smithi125 conmon[24927]: audit 2021-03-24 2021-03-24T22:05:53.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:53 smithi125 conmon[24927]: T22:05:53.366896+0000 mon.c (mon 2021-03-24T22:05:53.607 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:53 smithi125 conmon[24927]: .2) 114 : audit [INF] from='client.? v1:172.21.15.125:0/3190559375' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:05:53.608 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:53 smithi125 conmon[24927]: audit 2021- 2021-03-24T22:05:53.608 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:53 smithi125 conmon[24927]: 03-24T22:05:53.367691+0000 mon.a ( 2021-03-24T22:05:53.608 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:53 smithi125 conmon[24927]: mon.0) 1983 : audit [INF] from='client.35807 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:05:53.608 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:53 smithi125 conmon[24927]: audit 2021- 2021-03-24T22:05:53.608 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:53 smithi125 conmon[24927]: 03-24T22:05:53.548201+0000 mon.a 2021-03-24T22:05:53.609 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:53 smithi125 conmon[24927]: (mon.0) 1984 : audit [DBG] from='client.? v1:172.21.15.125:0/759929309' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:05:53.609 INFO:tasks.rados.rados.0.smithi125.stdout:3021: writing smithi12556470-20 from 796929 to 1522993 tid 1 2021-03-24T22:05:53.610 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T22:05:53.610 INFO:tasks.rados.rados.0.smithi125.stdout:3021: writing smithi12556470-20 from 2184647 to 2844013 tid 2 2021-03-24T22:05:53.613 INFO:tasks.rados.rados.0.smithi125.stdout:3021: writing smithi12556470-20 from 3302344 to 3596929 tid 3 2021-03-24T22:05:53.614 INFO:tasks.rados.rados.0.smithi125.stdout:3016: done (3 left) 2021-03-24T22:05:53.614 INFO:tasks.rados.rados.0.smithi125.stdout:3018: done (2 left) 2021-03-24T22:05:53.614 INFO:tasks.rados.rados.0.smithi125.stdout:3022: read oid 27 snap -1 2021-03-24T22:05:53.614 INFO:tasks.rados.rados.0.smithi125.stdout:3022: expect deleted 2021-03-24T22:05:53.615 INFO:tasks.rados.rados.0.smithi125.stdout:3023: read oid 11 snap -1 2021-03-24T22:05:53.615 INFO:tasks.rados.rados.0.smithi125.stdout:3023: expect (ObjNum 1043 snap 345 seq_num 1043) 2021-03-24T22:05:53.615 INFO:tasks.rados.rados.0.smithi125.stdout:3020: finishing write tid 1 to smithi12556470-46 2021-03-24T22:05:53.615 INFO:tasks.rados.rados.0.smithi125.stdout:3024: read oid 1 snap 331 2021-03-24T22:05:53.616 INFO:tasks.rados.rados.0.smithi125.stdout:3024: expect (ObjNum 764 snap 249 seq_num 764) 2021-03-24T22:05:53.616 INFO:tasks.rados.rados.0.smithi125.stdout:3025: read oid 28 snap -1 2021-03-24T22:05:53.616 INFO:tasks.rados.rados.0.smithi125.stdout:3025: expect (ObjNum 906 snap 300 seq_num 906) 2021-03-24T22:05:53.616 INFO:tasks.rados.rados.0.smithi125.stdout:3026: copy_from oid 2 from oid 48 current snap is 346 2021-03-24T22:05:53.616 INFO:tasks.rados.rados.0.smithi125.stdout:3020: finishing write tid 2 to smithi12556470-46 2021-03-24T22:05:53.617 INFO:tasks.rados.rados.0.smithi125.stdout:3027: snap_create 2021-03-24T22:05:53.617 INFO:tasks.rados.rados.0.smithi125.stdout:3020: finishing write tid 3 to smithi12556470-46 2021-03-24T22:05:53.617 INFO:tasks.rados.rados.0.smithi125.stdout:3020: finishing write tid 4 to smithi12556470-46 2021-03-24T22:05:53.617 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 46 v 845 (ObjNum 1046 snap 346 seq_num 1046) dirty exists 2021-03-24T22:05:53.618 INFO:tasks.rados.rados.0.smithi125.stdout:3020: left oid 46 (ObjNum 1046 snap 346 seq_num 1046) 2021-03-24T22:05:53.618 INFO:tasks.rados.rados.0.smithi125.stdout:3021: finishing write tid 1 to smithi12556470-20 2021-03-24T22:05:53.618 INFO:tasks.rados.rados.0.smithi125.stdout:3021: finishing write tid 2 to smithi12556470-20 2021-03-24T22:05:53.619 INFO:tasks.rados.rados.0.smithi125.stdout:3021: finishing write tid 3 to smithi12556470-20 2021-03-24T22:05:53.619 INFO:tasks.rados.rados.0.smithi125.stdout:3021: finishing write tid 4 to smithi12556470-20 2021-03-24T22:05:53.620 INFO:tasks.rados.rados.0.smithi125.stdout:3021: finishing write tid 5 to smithi12556470-20 2021-03-24T22:05:53.620 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 20 v 974 (ObjNum 1047 snap 346 seq_num 1047) dirty exists 2021-03-24T22:05:53.621 INFO:tasks.rados.rados.0.smithi125.stdout:3021: left oid 20 (ObjNum 1047 snap 346 seq_num 1047) 2021-03-24T22:05:53.624 INFO:tasks.rados.rados.0.smithi125.stdout:3025: expect (ObjNum 906 snap 300 seq_num 906) 2021-03-24T22:05:53.627 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T22:05:53.644 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:53.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:53.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:05:53.655 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T22:05:53.683 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T22:05:53.707 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T22:05:53.729 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T22:05:53.749 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:53.758 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T22:05:53.782 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T22:05:53.784 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-24T22:05:53.785 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-24T22:05:53.785 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-24T22:05:53.785 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-24T22:05:53.785 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-24T22:05:53.786 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-24T22:05:53.786 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-24T22:05:53.786 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-24T22:05:53.800 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:53.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:53.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:05:53.854 INFO:tasks.rados.rados.0.smithi125.stdout:3023: expect (ObjNum 1043 snap 345 seq_num 1043) 2021-03-24T22:05:53.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:53 smithi142 conmon[20125]: cluster 2021-03-24T22:05 2021-03-24T22:05:53.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:53 smithi142 conmon[20125]: :52.372022+0000 osd.5 (osd.5) 13 : cluster [DBG] 5.1b scrub ok 2021-03-24T22:05:53.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:53 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:05:53.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:53 smithi142 conmon[20125]: 22:05:52.552636+0000 mon.a (mon.0) 1982 : cluster [DBG] osdmap e782: 8 total, 8 up, 8 in 2021-03-24T22:05:53.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:53 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:05:53.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:53 smithi142 conmon[20125]: 22:05:52.854925+0000 osd.3 (osd 2021-03-24T22:05:53.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:53 smithi142 conmon[20125]: .3) 155 : cluster [DBG] 6.11 scrub ok 2021-03-24T22:05:53.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:53 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:05:53.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:53 smithi142 conmon[20125]: T22:05:53.144220+0000 osd.7 (osd. 2021-03-24T22:05:53.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:53 smithi142 conmon[20125]: 7) 129 : cluster [DBG] 3.15 scrub ok 2021-03-24T22:05:53.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:53 smithi142 conmon[20125]: audit 2021-03- 2021-03-24T22:05:53.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:53 smithi142 conmon[20125]: 24T22:05:53.366896+0000 mon.c (mon.2) 114 : audit 2021-03-24T22:05:53.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:53 smithi142 conmon[20125]: [INF] from='client.? v1:172.21.15.125:0/3190559375' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:05:53.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:53 smithi142 conmon[20125]: audit 2021 2021-03-24T22:05:53.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:53 smithi142 conmon[20125]: -03-24T22:05:53.367691+0000 mon.a (mon.0) 2021-03-24T22:05:53.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:53 smithi142 conmon[20125]: 1983 : audit [INF] from='client.35807 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:05:53.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:53 smithi142 conmon[20125]: audit 2021-03-24T22 2021-03-24T22:05:53.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:53 smithi142 conmon[20125]: :05:53.548201+0000 mon.a (mon. 2021-03-24T22:05:53.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:53 smithi142 conmon[20125]: 0) 1984 : audit [DBG] from='client.? v1:172.21.15.125:0/759929309' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:05:53.906 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:53.963 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:53.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:53.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:05:54.067 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:54.117 INFO:tasks.rados.rados.0.smithi125.stdout:3026: finishing copy_from to smithi12556470-2 2021-03-24T22:05:54.118 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 2 v 1017 (ObjNum 984 snap 326 seq_num 984) dirty exists 2021-03-24T22:05:54.118 INFO:tasks.rados.rados.0.smithi125.stdout:3026: finishing copy_from racing read to smithi12556470-2 2021-03-24T22:05:54.118 INFO:tasks.rados.rados.0.smithi125.stdout:3024: expect (ObjNum 764 snap 249 seq_num 764) 2021-03-24T22:05:54.126 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:54.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:54.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:05:54.232 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:54.282 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:54.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:54.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:05:54.388 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:54.446 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:54.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:54.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:05:54.553 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:54.573 INFO:tasks.rados.rados.0.smithi125.stdout:3020: done (7 left) 2021-03-24T22:05:54.573 INFO:tasks.rados.rados.0.smithi125.stdout:3021: done (6 left) 2021-03-24T22:05:54.573 INFO:tasks.rados.rados.0.smithi125.stdout:3022: done (5 left) 2021-03-24T22:05:54.574 INFO:tasks.rados.rados.0.smithi125.stdout:3023: done (4 left) 2021-03-24T22:05:54.574 INFO:tasks.rados.rados.0.smithi125.stdout:3024: done (3 left) 2021-03-24T22:05:54.574 INFO:tasks.rados.rados.0.smithi125.stdout:3025: done (2 left) 2021-03-24T22:05:54.574 INFO:tasks.rados.rados.0.smithi125.stdout:3026: done (1 left) 2021-03-24T22:05:54.575 INFO:tasks.rados.rados.0.smithi125.stdout:3027: done (0 left) 2021-03-24T22:05:54.575 INFO:tasks.rados.rados.0.smithi125.stdout:3028: rollback oid 28 current snap is 347 2021-03-24T22:05:54.575 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 28 to 317 2021-03-24T22:05:54.575 INFO:tasks.rados.rados.0.smithi125.stdout:3029: snap_remove snap 340 2021-03-24T22:05:54.609 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:54.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:54.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:05:54.715 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:54.768 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:54.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:54.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:05:54.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:54 smithi142 conmon[20125]: cluster 2021-03-24T22:05 2021-03-24T22:05:54.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:54 smithi142 conmon[20125]: :52.991482+0000 mgr.y (mgr.14144) 666 : cluster [DBG] pgmap v1336: 168 pgs: 168 active+clean; 540 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.4 MiB/s wr, 4 op/s 2021-03-24T22:05:54.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:54 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:05:54.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:54 smithi142 conmon[20125]: -03-24T22:05:53.363145+0000 osd.5 2021-03-24T22:05:54.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:54 smithi142 conmon[20125]: (osd.5) 14 : cluster [DBG] 2021-03-24T22:05:54.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:54 smithi142 conmon[20125]: 7.0 scrub ok 2021-03-24T22:05:54.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:54 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:05:54.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:54 smithi142 conmon[20125]: 24T22:05:53.550941+0000 2021-03-24T22:05:54.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:54 smithi142 conmon[20125]: mon.a (mon.0) 1985 : cluster [INF] 2021-03-24T22:05:54.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:54 smithi142 conmon[20125]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T22:05:54.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:54 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:05:54.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:54 smithi142 conmon[20125]: 22:05:53.551011+0000 mon.a 2021-03-24T22:05:54.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:54 smithi142 conmon[20125]: (mon.0) 1986 : cluster [INF] Cluster is now healthy 2021-03-24T22:05:54.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:54 smithi142 conmon[20125]: 2021-03-24T22:05:54.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:54 smithi142 conmon[20125]: audit 2021-03-24T22:05:54.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:54 smithi142 conmon[20125]: 2021-03-24T22:05:53. 2021-03-24T22:05:54.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:54 smithi142 conmon[20125]: 563819+0000 mon.a (mon.0) 1987 : audit 2021-03-24T22:05:54.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:54 smithi142 conmon[20125]: [INF] from='client.35807 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T22:05:54.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:54 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:05:54.872 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:54 smithi142 conmon[20125]: 22:05:53.563928+0000 mon.a ( 2021-03-24T22:05:54.872 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:54 smithi142 conmon[20125]: mon.0) 1988 : cluster [DBG] osdmap e783: 8 total, 8 up, 8 in 2021-03-24T22:05:54.872 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:54 smithi142 conmon[20125]: cluster 2021-03-24T22:05:54.872 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:54 smithi142 conmon[20125]: 2021-03-24T22:05 2021-03-24T22:05:54.872 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:54 smithi142 conmon[20125]: :53.982222+0000 osd.2 (osd.2) 2021-03-24T22:05:54.873 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:54 smithi142 conmon[20125]: 123 : cluster [DBG] 3.21 scrub ok 2021-03-24T22:05:54.873 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:54 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:05:54.873 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:54 smithi142 conmon[20125]: -24T22:05:54.205099 2021-03-24T22:05:54.873 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:54 smithi142 conmon[20125]: +0000 osd.6 (osd.6) 78 : cluster 2021-03-24T22:05:54.874 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:54 smithi142 conmon[20125]: [DBG] 4.5 deep-scrub ok 2021-03-24T22:05:54.875 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:54.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:54 smithi125 conmon[19936]: cluster 2021-03-24T22:05:52 2021-03-24T22:05:54.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:54 smithi125 conmon[19936]: .991482+0000 mgr.y (mgr.14144) 666 : cluster [DBG] pgmap v1336: 168 pgs: 168 active+clean; 540 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.4 MiB/s wr, 4 op/s 2021-03-24T22:05:54.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:54 smithi125 conmon[19936]: cluster 2021-03-24T22 2021-03-24T22:05:54.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:54 smithi125 conmon[19936]: :05:53.363145+0000 osd.5 (osd.5) 14 : cluster [DBG] 7.0 scrub ok 2021-03-24T22:05:54.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:54 smithi125 conmon[19936]: cluster 2021-03-24T22:05:54.915 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:54 smithi125 conmon[19936]: 2021-03-24T22:05:53.550941+0000 2021-03-24T22:05:54.915 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:54 smithi125 conmon[19936]: mon.a (mon.0) 1985 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T22:05:54.915 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:54 smithi125 conmon[19936]: cluster 2021-03-24T22:05:54.916 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:54 smithi125 conmon[19936]: 2021-03-24T22:05:53.551011+0000 2021-03-24T22:05:54.916 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:54 smithi125 conmon[19936]: mon.a (mon.0) 1986 : cluster [INF] Cluster is now healthy 2021-03-24T22:05:54.916 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:54 smithi125 conmon[19936]: 2021-03-24T22:05:54.917 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:54 smithi125 conmon[19936]: audit 2021-03-24T22: 2021-03-24T22:05:54.917 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:54 smithi125 conmon[19936]: 05:53.563819+0000 mon.a (mon.0) 2021-03-24T22:05:54.917 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:54 smithi125 conmon[19936]: 1987 : audit [INF] from='client.35807 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T22:05:54.917 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:54 smithi125 conmon[19936]: cluster 2021-03-24T22:05:54.918 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:54 smithi125 conmon[19936]: 2021-03-24T22:05 2021-03-24T22:05:54.918 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:54 smithi125 conmon[19936]: :53.563928+0000 mon.a (mon.0) 1988 2021-03-24T22:05:54.918 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:54 smithi125 conmon[19936]: : cluster [DBG] osdmap e783: 8 total, 8 up, 8 in 2021-03-24T22:05:54.919 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:54 smithi125 conmon[19936]: cluster 2021-03-24T22:05:54.919 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:54 smithi125 conmon[19936]: 2021-03-24T22:05 2021-03-24T22:05:54.919 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:54 smithi125 conmon[19936]: :53.982222+0000 osd.2 (osd.2) 123 : 2021-03-24T22:05:54.919 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:54 smithi125 conmon[19936]: cluster [DBG] 3.21 scrub ok 2021-03-24T22:05:54.919 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:54 smithi125 conmon[19936]: cluster 2021-03-24T22:05:54.920 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:54 smithi125 conmon[19936]: 2021-03-24T22:05:54.205099 2021-03-24T22:05:54.920 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:54 smithi125 conmon[19936]: +0000 osd.6 (osd.6) 78 : cluster [DBG] 2021-03-24T22:05:54.920 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:54 smithi125 conmon[19936]: 4.5 deep-scrub ok 2021-03-24T22:05:54.921 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:54 smithi125 conmon[24927]: cluster 2021-03-24T22 2021-03-24T22:05:54.921 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:54 smithi125 conmon[24927]: :05:52.991482+0000 mgr.y (mgr.14144) 666 : cluster [DBG] pgmap v1336: 168 pgs: 168 active+clean; 540 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.4 MiB/s wr, 4 op/s 2021-03-24T22:05:54.921 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:54 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T22:05:54.922 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:54 smithi125 conmon[24927]: T22:05:53.363145+0000 osd.5 (osd.5) 14 : cluster [DBG] 7.0 scrub ok 2021-03-24T22:05:54.922 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:54 smithi125 conmon[24927]: cluster 2021-03-24T22:05:54.923 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:54 smithi125 conmon[24927]: 2021-03-24T22: 2021-03-24T22:05:54.923 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:54 smithi125 conmon[24927]: 05:53.550941+0000 mon.a (mon.0 2021-03-24T22:05:54.923 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:54 smithi125 conmon[24927]: ) 1985 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T22:05:54.923 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:54 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:05:54.924 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:54 smithi125 conmon[24927]: 03-24T22:05: 2021-03-24T22:05:54.924 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:54 smithi125 conmon[24927]: 53.551011+0000 mon.a (mon.0) 1986 : 2021-03-24T22:05:54.924 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:54 smithi125 conmon[24927]: cluster [INF] Cluster is now healthy 2021-03-24T22:05:54.924 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:54 smithi125 conmon[24927]: audit 2021-03-24T22:05:54.924 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:54 smithi125 conmon[24927]: 2021-03-24T22:05:53.563819+0000 mon.a 2021-03-24T22:05:54.925 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:54 smithi125 conmon[24927]: (mon.0) 1987 : audit [INF] from='client.35807 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T22:05:54.925 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:54 smithi125 conmon[24927]: cluster 2021-03-24T22:05:54.925 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:54 smithi125 conmon[24927]: 2021-03-24T22:05:53.563928+0000 mon.a (mon.0) 1988 : 2021-03-24T22:05:54.926 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:54 smithi125 conmon[24927]: cluster [DBG] osdmap e783: 8 total, 8 up, 8 in 2021-03-24T22:05:54.926 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:54 smithi125 conmon[24927]: cluster 2021-03-24T22:05:54.926 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:54 smithi125 conmon[24927]: 2021-03-24T22:05:53.982222+0000 osd.2 (osd.2) 123 : cluster 2021-03-24T22:05:54.926 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:54 smithi125 conmon[24927]: [DBG] 3.21 scrub ok 2021-03-24T22:05:54.927 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:54 smithi125 conmon[24927]: cluster 2021-03-24T22:05:54.927 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:54 smithi125 conmon[24927]: 2021-03-24T22:05:54.205099+0000 osd.6 (osd. 2021-03-24T22:05:54.927 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:54 smithi125 conmon[24927]: 6) 78 : cluster [DBG] 4.5 deep-scrub ok 2021-03-24T22:05:54.930 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:54.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:54.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:05:55.036 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:55.094 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:55.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:55.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:05:55.200 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:55.257 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:55.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:55.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:05:55.364 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:55.418 INFO:tasks.rados.rados.0.smithi125.stdout:3029: done (1 left) 2021-03-24T22:05:55.419 INFO:tasks.rados.rados.0.smithi125.stdout:3030: write oid 24 current snap is 347 2021-03-24T22:05:55.419 INFO:tasks.rados.rados.0.smithi125.stdout:3030: seq_num 1048 ranges {413736=401313,1326776=403024,2413414=457893,3438716=175020} 2021-03-24T22:05:55.428 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:55.433 INFO:tasks.rados.rados.0.smithi125.stdout:3030: writing smithi12556470-24 from 413736 to 815049 tid 1 2021-03-24T22:05:55.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:55.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:05:55.440 INFO:tasks.rados.rados.0.smithi125.stdout:3030: writing smithi12556470-24 from 1326776 to 1729800 tid 2 2021-03-24T22:05:55.447 INFO:tasks.rados.rados.0.smithi125.stdout:3030: writing smithi12556470-24 from 2413414 to 2871307 tid 3 2021-03-24T22:05:55.451 INFO:tasks.rados.rados.0.smithi125.stdout:3030: writing smithi12556470-24 from 3438716 to 3613736 tid 4 2021-03-24T22:05:55.452 INFO:tasks.rados.rados.0.smithi125.stdout:3031: snap_remove snap 296 2021-03-24T22:05:55.536 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:55.592 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:55.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:55.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:05:55.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:05:55.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[19936]: 05:54.393907+0000 osd.5 (osd.5) 15 : cluster 2021-03-24T22:05:55.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[19936]: [DBG] 3.28 scrub ok 2021-03-24T22:05:55.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[19936]: cluster 2021-03-24T22:05:55.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[19936]: 2021-03-24T22:05: 2021-03-24T22:05:55.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[19936]: 54.570218+0000 mon.a 2021-03-24T22:05:55.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[19936]: (mon.0) 2021-03-24T22:05:55.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[19936]: 1989 : cluster [DBG] 2021-03-24T22:05:55.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[19936]: osdmap e784: 8 total, 8 up, 8 in 2021-03-24T22:05:55.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[19936]: cluster 2021-03-24T22:05:55.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[19936]: 2021-03-24T22:05: 2021-03-24T22:05:55.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[19936]: 55.032064+0000 osd.2 (osd 2021-03-24T22:05:55.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[19936]: .2) 124 : cluster [DBG] 2021-03-24T22:05:55.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[19936]: 6.8 scrub ok 2021-03-24T22:05:55.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[19936]: cluster 2021-03-24T22:05:55.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[19936]: 2021-03-24T22:05 2021-03-24T22:05:55.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[19936]: :55.373260+0000 osd.0 ( 2021-03-24T22:05:55.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[19936]: osd.0) 151 : cluster 2021-03-24T22:05:55.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[19936]: [DBG] 7.13 scrub ok 2021-03-24T22:05:55.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[19936]: cluster 2021-03-24T22:05:55.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[19936]: 2021-03-24T22 2021-03-24T22:05:55.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[19936]: :05:55.413489 2021-03-24T22:05:55.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[19936]: +0000 mon.a (mon.0) 1990 2021-03-24T22:05:55.668 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[19936]: : cluster [DBG] osdmap e785: 8 total, 8 up, 8 in 2021-03-24T22:05:55.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:05:55.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[24927]: 03-24T22:05:54.393907+0000 2021-03-24T22:05:55.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[24927]: osd.5 (osd.5) 2021-03-24T22:05:55.669 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[24927]: 15 : cluster [DBG] 2021-03-24T22:05:55.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[24927]: 3.28 scrub ok 2021-03-24T22:05:55.670 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[24927]: cluster 2021-03-24T22:05:55.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[24927]: 2021-03-24T22 2021-03-24T22:05:55.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[24927]: :05:54.570218+0000 2021-03-24T22:05:55.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[24927]: mon.a (mon.0) 1989 : 2021-03-24T22:05:55.672 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[24927]: cluster [DBG] osdmap e784: 8 total, 8 up, 8 in 2021-03-24T22:05:55.672 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[24927]: 2021-03-24T22:05:55.673 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T22:05:55.673 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[24927]: -24T22:05 2021-03-24T22:05:55.673 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[24927]: :55.032064+0000 2021-03-24T22:05:55.674 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[24927]: osd.2 (osd.2 2021-03-24T22:05:55.674 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[24927]: ) 124 : cluster [DBG] 2021-03-24T22:05:55.674 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[24927]: 6.8 scrub ok 2021-03-24T22:05:55.675 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[24927]: cluster 2021-03-24T22:05:55.675 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[24927]: 2021-03-24T 2021-03-24T22:05:55.675 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[24927]: 22:05:55 2021-03-24T22:05:55.675 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[24927]: .373260+0000 osd.0 ( 2021-03-24T22:05:55.676 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[24927]: osd.0) 151 2021-03-24T22:05:55.676 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[24927]: : cluster [DBG] 7.13 scrub ok 2021-03-24T22:05:55.676 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[24927]: 2021-03-24T22:05:55.676 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[24927]: cluster 2021-03-24T22:05:55.676 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[24927]: 2021-03-24T22 2021-03-24T22:05:55.677 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[24927]: :05:55.413489+0000 2021-03-24T22:05:55.677 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[24927]: mon.a (mon.0) 1990 : cluster 2021-03-24T22:05:55.677 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:55 smithi125 conmon[24927]: [DBG] osdmap e785: 8 total, 8 up, 8 in 2021-03-24T22:05:55.698 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:55.758 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:55.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:55.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:05:55.865 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:55.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:55 smithi142 conmon[20125]: cluster 2021-03-24T22:05:54.393907+0000 osd.5 (osd.5) 15 : cluster [DBG] 2021-03-24T22:05:55.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:55 smithi142 conmon[20125]: 3.28 scrub ok 2021-03-24T22:05:55.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:55 smithi142 conmon[20125]: cluster 2021-03-24T22:05:54.570218+0000 mon.a (mon.0) 1989 : cluster [DBG] osdmap e784: 8 total, 8 up, 8 in 2021-03-24T22:05:55.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:55 smithi142 conmon[20125]: cluster 2021-03-24T22:05:55.032064+0000 osd.2 (osd.2) 2021-03-24T22:05:55.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:55 smithi142 conmon[20125]: 124 : cluster [DBG] 6.8 scrub ok 2021-03-24T22:05:55.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:55 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:05:55.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:55 smithi142 conmon[20125]: 22:05:55.373260+0000 osd.0 (osd.0 2021-03-24T22:05:55.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:55 smithi142 conmon[20125]: ) 151 : cluster [DBG] 7.13 scrub ok 2021-03-24T22:05:55.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:55 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:05:55.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:55 smithi142 conmon[20125]: 03-24T22:05:55.413489+0000 2021-03-24T22:05:55.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:55 smithi142 conmon[20125]: mon.a (mon.0) 1990 : cluster [DBG] osdmap e785: 8 total, 8 up, 8 in 2021-03-24T22:05:55.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:55 smithi142 conmon[20125]: 2021-03-24T22:05:55.925 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:55.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:55.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:05:56.032 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:56.090 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:56.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:56.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:05:56.197 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:56.260 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:56.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:56.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:05:56.367 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:56.421 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:05:56.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:56.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:05:56.444 INFO:tasks.rados.rados.0.smithi125.stdout:3031: done (2 left) 2021-03-24T22:05:56.444 INFO:tasks.rados.rados.0.smithi125.stdout:3032: read oid 35 snap -1 2021-03-24T22:05:56.444 INFO:tasks.rados.rados.0.smithi125.stdout:3032: expect (ObjNum 802 snap 262 seq_num 802) 2021-03-24T22:05:56.445 INFO:tasks.rados.rados.0.smithi125.stdout:3033: snap_remove snap 326 2021-03-24T22:05:56.527 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:56.589 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:56.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:56.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:05:56.695 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:56.763 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:56.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:56.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:05:56.813 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:56 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T22:05:56.813 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:56 smithi125 conmon[24927]: 22:05:54.992163+0000 mgr.y (mgr.14144) 667 : cluster [DBG] pgmap v1339: 168 pgs: 168 active+clean; 542 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 1.9 MiB/s wr, 15 op/s 2021-03-24T22:05:56.813 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:56 smithi125 conmon[24927]: cluster 2021-03-24T22:05:56.814 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:56 smithi125 conmon[24927]: 2021-03-24T22:05:55.541919+0000 osd.1 (osd.1) 89 : cluster [DBG] 2021-03-24T22:05:56.814 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:56 smithi125 conmon[24927]: 7.1e scrub ok 2021-03-24T22:05:56.814 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:56 smithi125 conmon[24927]: cluster 2021-03-24T22:05:56. 2021-03-24T22:05:56.814 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:56 smithi125 conmon[24927]: 072435+0000 osd.2 (osd.2) 125 : cluster [DBG] 3.9 deep-scrub ok 2021-03-24T22:05:56.815 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:56 smithi125 conmon[24927]: 2021-03-24T22:05:56.815 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:56 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T22:05:56.815 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:56 smithi125 conmon[24927]: T22:05:56.358935+0000 osd.0 (osd.0) 152 : cluster [DBG] 2021-03-24T22:05:56.815 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:56 smithi125 conmon[24927]: 7.6 scrub ok 2021-03-24T22:05:56.815 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:56 smithi125 conmon[24927]: cluster 2021-03-24T22:05:56.816 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:56 smithi125 conmon[24927]: 2021-03-24T22:05:56.438174+0000 mon.a (mon.0) 1991 2021-03-24T22:05:56.816 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:56 smithi125 conmon[24927]: : cluster [DBG] osdmap e786: 8 total, 8 up, 8 in 2021-03-24T22:05:56.816 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:56 smithi125 conmon[19936]: cluster 2021-03-24T22:05:54.992163+0000 2021-03-24T22:05:56.816 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:56 smithi125 conmon[19936]: mgr.y (mgr.14144) 667 : cluster [DBG] pgmap v1339: 168 pgs: 168 active+clean; 542 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 1.9 MiB/s wr, 15 op/s 2021-03-24T22:05:56.817 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:56 smithi125 conmon[19936]: cluster 2021-03-24T22:05:55.541919+0000 osd.1 (osd.1) 89 : cluster [DBG] 7.1e scrub ok 2021-03-24T22:05:56.817 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:56 smithi125 conmon[19936]: cluster 2021-03-24T22:05 2021-03-24T22:05:56.817 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:56 smithi125 conmon[19936]: :56.072435+0000 osd.2 (osd.2) 125 : cluster [DBG] 2021-03-24T22:05:56.817 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:56 smithi125 conmon[19936]: 3.9 deep-scrub ok 2021-03-24T22:05:56.818 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:56 smithi125 conmon[19936]: cluster 2021-03-24T22:05 2021-03-24T22:05:56.818 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:56 smithi125 conmon[19936]: :56.358935+0000 osd.0 (osd.0) 152 : cluster 2021-03-24T22:05:56.818 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:56 smithi125 conmon[19936]: [DBG] 7.6 scrub ok 2021-03-24T22:05:56.818 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:56 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:05:56.819 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:56 smithi125 conmon[19936]: 05:56.438174+0000 mon.a (mon.0) 1991 : cluster 2021-03-24T22:05:56.819 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:56 smithi125 conmon[19936]: [DBG] osdmap e786: 8 total, 8 up, 8 in 2021-03-24T22:05:56.824 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:05:56.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:56 smithi142 conmon[20125]: cluster 2021-03-24T22:05:54 2021-03-24T22:05:56.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:56 smithi142 conmon[20125]: .992163+0000 mgr.y (mgr.14144) 667 : cluster [DBG] pgmap v1339: 168 pgs: 168 active+clean; 542 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 1.9 MiB/s wr, 15 op/s 2021-03-24T22:05:56.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:56 smithi142 conmon[20125]: cluster 2021-03-24T22:05:55.541919 2021-03-24T22:05:56.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:56 smithi142 conmon[20125]: +0000 osd.1 (osd.1) 89 : cluster [DBG] 7.1e scrub ok 2021-03-24T22:05:56.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:56 smithi142 conmon[20125]: cluster 2021-03-24T22 2021-03-24T22:05:56.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:56 smithi142 conmon[20125]: :05:56.072435+0000 osd.2 (osd.2) 2021-03-24T22:05:56.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:56 smithi142 conmon[20125]: 125 : cluster [DBG] 3.9 deep-scrub ok 2021-03-24T22:05:56.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:56 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:05:56.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:56 smithi142 conmon[20125]: 24T22:05:56.358935+0000 osd.0 (osd 2021-03-24T22:05:56.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:56 smithi142 conmon[20125]: .0) 152 : cluster [DBG] 7.6 scrub ok 2021-03-24T22:05:56.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:56 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:05:56.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:56 smithi142 conmon[20125]: 03-24T22:05:56.438174+0000 mon.a ( 2021-03-24T22:05:56.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:56 smithi142 conmon[20125]: mon.0) 1991 : cluster [DBG] osdmap e786: 8 total, 8 up, 8 in 2021-03-24T22:05:56.870 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:56.920 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:56.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:56.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:05:57.025 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:57.092 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:57.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:57.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:05:57.197 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:57.258 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:57.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:57.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:05:57.363 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:57.415 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:57.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:57.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:05:57.450 INFO:tasks.rados.rados.0.smithi125.stdout:3033: done (3 left) 2021-03-24T22:05:57.451 INFO:tasks.rados.rados.0.smithi125.stdout:3034: snap_create 2021-03-24T22:05:57.451 INFO:tasks.rados.rados.0.smithi125.stdout:3028: finishing rollback tid 0 to smithi12556470-28 2021-03-24T22:05:57.451 INFO:tasks.rados.rados.0.smithi125.stdout:3028: finishing rollback tid 1 to smithi12556470-28 2021-03-24T22:05:57.451 INFO:tasks.rados.rados.0.smithi125.stdout:3028: finishing rollback tid 2 to smithi12556470-28 2021-03-24T22:05:57.452 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 28 v 1208 (ObjNum 961 snap 317 seq_num 961) dirty exists 2021-03-24T22:05:57.452 INFO:tasks.rados.rados.0.smithi125.stdout:3030: finishing write tid 1 to smithi12556470-24 2021-03-24T22:05:57.452 INFO:tasks.rados.rados.0.smithi125.stdout:3030: finishing write tid 2 to smithi12556470-24 2021-03-24T22:05:57.452 INFO:tasks.rados.rados.0.smithi125.stdout:3030: finishing write tid 3 to smithi12556470-24 2021-03-24T22:05:57.452 INFO:tasks.rados.rados.0.smithi125.stdout:3030: finishing write tid 4 to smithi12556470-24 2021-03-24T22:05:57.453 INFO:tasks.rados.rados.0.smithi125.stdout:3030: finishing write tid 5 to smithi12556470-24 2021-03-24T22:05:57.453 INFO:tasks.rados.rados.0.smithi125.stdout:3030: finishing write tid 6 to smithi12556470-24 2021-03-24T22:05:57.453 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 24 v 1457 (ObjNum 1048 snap 347 seq_num 1048) dirty exists 2021-03-24T22:05:57.453 INFO:tasks.rados.rados.0.smithi125.stdout:3030: left oid 24 (ObjNum 1048 snap 347 seq_num 1048) 2021-03-24T22:05:57.454 INFO:tasks.rados.rados.0.smithi125.stdout:3032: expect (ObjNum 802 snap 262 seq_num 802) 2021-03-24T22:05:57.521 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:57.575 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:57.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:57.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:05:57.680 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:57.728 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:57.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:57.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:05:57.833 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:57.888 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:05:57.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:57.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:05:57.995 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:58.050 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:05:58.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:58.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:05:58.157 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:58.222 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:58.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:58.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:05:58.329 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:58.381 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:58.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:58.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:05:58.450 INFO:tasks.rados.rados.0.smithi125.stdout:3028: done (3 left) 2021-03-24T22:05:58.451 INFO:tasks.rados.rados.0.smithi125.stdout:3030: done (2 left) 2021-03-24T22:05:58.451 INFO:tasks.rados.rados.0.smithi125.stdout:3032: done (1 left) 2021-03-24T22:05:58.451 INFO:tasks.rados.rados.0.smithi125.stdout:3034: done (0 left) 2021-03-24T22:05:58.451 INFO:tasks.rados.rados.0.smithi125.stdout:3035: write oid 36 current snap is 348 2021-03-24T22:05:58.452 INFO:tasks.rados.rados.0.smithi125.stdout:3035: seq_num 1049 ranges {430543=476562,1385258=663035,2840837=669165} 2021-03-24T22:05:58.457 INFO:tasks.rados.rados.0.smithi125.stdout:3035: writing smithi12556470-36 from 430543 to 907105 tid 1 2021-03-24T22:05:58.464 INFO:tasks.rados.rados.0.smithi125.stdout:3035: writing smithi12556470-36 from 1385258 to 2048293 tid 2 2021-03-24T22:05:58.473 INFO:tasks.rados.rados.0.smithi125.stdout:3035: writing smithi12556470-36 from 2840837 to 3510002 tid 3 2021-03-24T22:05:58.473 INFO:tasks.rados.rados.0.smithi125.stdout:3036: rollback oid 14 current snap is 348 2021-03-24T22:05:58.474 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 14 to 336 2021-03-24T22:05:58.474 INFO:tasks.rados.rados.0.smithi125.stdout:3035: finishing write tid 1 to smithi12556470-36 2021-03-24T22:05:58.474 INFO:tasks.rados.rados.0.smithi125.stdout:3035: finishing write tid 2 to smithi12556470-36 2021-03-24T22:05:58.475 INFO:tasks.rados.rados.0.smithi125.stdout:3037: copy_from oid 42 from oid 39 current snap is 348 2021-03-24T22:05:58.475 INFO:tasks.rados.rados.0.smithi125.stdout:3038: copy_from oid 17 from oid 4 current snap is 348 2021-03-24T22:05:58.475 INFO:tasks.rados.rados.0.smithi125.stdout:3039: snap_create 2021-03-24T22:05:58.480 INFO:tasks.rados.rados.0.smithi125.stdout:3037: finishing copy_from to smithi12556470-42 2021-03-24T22:05:58.481 INFO:tasks.rados.rados.0.smithi125.stdout:3037: got expected ENOENT (src dne) 2021-03-24T22:05:58.482 INFO:tasks.rados.rados.0.smithi125.stdout:3037: finishing copy_from racing read to smithi12556470-42 2021-03-24T22:05:58.482 INFO:tasks.rados.rados.0.smithi125.stdout:3035: finishing write tid 3 to smithi12556470-36 2021-03-24T22:05:58.482 INFO:tasks.rados.rados.0.smithi125.stdout:3035: finishing write tid 4 to smithi12556470-36 2021-03-24T22:05:58.483 INFO:tasks.rados.rados.0.smithi125.stdout:3035: finishing write tid 5 to smithi12556470-36 2021-03-24T22:05:58.483 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 36 v 1253 (ObjNum 1049 snap 348 seq_num 1049) dirty exists 2021-03-24T22:05:58.483 INFO:tasks.rados.rados.0.smithi125.stdout:3035: left oid 36 (ObjNum 1049 snap 348 seq_num 1049) 2021-03-24T22:05:58.489 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:58.492 INFO:tasks.rados.rados.0.smithi125.stdout:3036: finishing rollback tid 0 to smithi12556470-14 2021-03-24T22:05:58.493 INFO:tasks.rados.rados.0.smithi125.stdout:3036: finishing rollback tid 1 to smithi12556470-14 2021-03-24T22:05:58.495 INFO:tasks.rados.rados.0.smithi125.stdout:3038: finishing copy_from to smithi12556470-17 2021-03-24T22:05:58.495 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 17 v 895 (ObjNum 1013 snap 336 seq_num 1013) dirty exists 2021-03-24T22:05:58.496 INFO:tasks.rados.rados.0.smithi125.stdout:3038: finishing copy_from racing read to smithi12556470-17 2021-03-24T22:05:58.496 INFO:tasks.rados.rados.0.smithi125.stdout:3036: finishing rollback tid 2 to smithi12556470-14 2021-03-24T22:05:58.496 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 14 v 1893 (ObjNum 727 snap 233 seq_num 727) dirty exists 2021-03-24T22:05:58.545 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:58.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:58.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:05:58.629 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:58 smithi125 conmon[19936]: cluster 2021-03-24T22:05:56.993044+0000 mgr.y (mgr.14144 2021-03-24T22:05:58.629 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:58 smithi125 conmon[19936]: ) 668 : cluster [DBG] pgmap v1342: 168 pgs: 168 active+clean; 544 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 2.5 MiB/s wr, 14 op/s 2021-03-24T22:05:58.629 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:58 smithi125 conmon[19936]: cluster 2021-03-24T22:05:57.159934+0000 osd.7 (osd.7) 130 : cluster [DBG] 6.15 scrub ok 2021-03-24T22:05:58.630 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:58 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:05:58.630 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:58 smithi125 conmon[19936]: 03-24T22:05:57.440124+0000 mon.a (mon.0) 1992 : cluster [DBG] osdmap e787: 8 total, 8 up, 8 in 2021-03-24T22:05:58.630 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:58 smithi125 conmon[24927]: cluster 2021-03-24T22:05:58.630 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:58 smithi125 conmon[24927]: 2021 2021-03-24T22:05:58.631 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:58 smithi125 conmon[24927]: -03-24T 2021-03-24T22:05:58.631 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:58 smithi125 conmon[24927]: 22:05:56 2021-03-24T22:05:58.631 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:58 smithi125 conmon[24927]: .993044+0000 mgr.y 2021-03-24T22:05:58.631 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:58 smithi125 conmon[24927]: (mgr.14144) 668 2021-03-24T22:05:58.632 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:58 smithi125 conmon[24927]: : cluster [DBG] 2021-03-24T22:05:58.632 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:58 smithi125 conmon[24927]: pgmap v1342: 168 pgs: 168 active+clean; 544 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 2.5 MiB/s wr, 14 op/s 2021-03-24T22:05:58.632 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:58 smithi125 conmon[24927]: cluster 2021-03-24T22:05:58.632 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:58 smithi125 conmon[24927]: 2021-03-24T22:05 2021-03-24T22:05:58.632 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:58 smithi125 conmon[24927]: :57.159934+0000 2021-03-24T22:05:58.633 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:58 smithi125 conmon[24927]: osd.7 (osd. 2021-03-24T22:05:58.633 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:58 smithi125 conmon[24927]: 7) 130 : cluster 2021-03-24T22:05:58.633 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:58 smithi125 conmon[24927]: [DBG] 6.15 scrub ok 2021-03-24T22:05:58.634 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:58 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:05:58.634 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:58 smithi125 conmon[24927]: -03-24T22: 2021-03-24T22:05:58.634 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:58 smithi125 conmon[24927]: 05:57.440124+0000 2021-03-24T22:05:58.634 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:58 smithi125 conmon[24927]: mon.a (mon. 2021-03-24T22:05:58.635 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:58 smithi125 conmon[24927]: 0) 1992 : cluster [DBG] osdmap e787: 8 total, 8 up, 8 in 2021-03-24T22:05:58.635 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:58 smithi125 conmon[24927]: 2021-03-24T22:05:58.651 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:58.705 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:58.709 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:58 smithi142 conmon[20125]: cluster 2021-03-24T22:05:56.993044+0000 mgr.y (mgr.14144) 668 : 2021-03-24T22:05:58.710 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:58 smithi142 conmon[20125]: cluster [DBG] pgmap v1342: 168 pgs: 168 active+clean; 544 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 2.5 MiB/s wr, 14 op/s 2021-03-24T22:05:58.710 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:58 smithi142 conmon[20125]: cluster 2021-03-24T22:05:57.159934+0000 osd.7 (osd.7) 130 : cluster 2021-03-24T22:05:58.710 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:58 smithi142 conmon[20125]: [DBG] 6.15 scrub ok 2021-03-24T22:05:58.710 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:58 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:05:58.711 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:58 smithi142 conmon[20125]: 03-24T22:05:57.440124+0000 mon.a ( 2021-03-24T22:05:58.711 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:58 smithi142 conmon[20125]: mon.0) 1992 : cluster [DBG] osdmap e787: 8 total, 8 up, 8 in 2021-03-24T22:05:58.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:58.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:05:58.790 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-24T22:05:58.791 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T22:05:58.814 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:58.846 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd set noscrub 2021-03-24T22:05:58.871 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:58.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:58.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:05:58.886 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-24T22:05:58.977 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:59.040 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:59.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:59.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:05:59.150 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:59.205 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:05:59.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:59.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:05:59.312 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:59.373 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:05:59.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:59.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:05:59.466 INFO:tasks.rados.rados.0.smithi125.stdout:3035: done (4 left) 2021-03-24T22:05:59.467 INFO:tasks.rados.rados.0.smithi125.stdout:3036: done (3 left) 2021-03-24T22:05:59.467 INFO:tasks.rados.rados.0.smithi125.stdout:3037: done (2 left) 2021-03-24T22:05:59.467 INFO:tasks.rados.rados.0.smithi125.stdout:3038: done (1 left) 2021-03-24T22:05:59.467 INFO:tasks.rados.rados.0.smithi125.stdout:3039: done (0 left) 2021-03-24T22:05:59.468 INFO:tasks.rados.rados.0.smithi125.stdout:3040: read oid 34 snap -1 2021-03-24T22:05:59.468 INFO:tasks.rados.rados.0.smithi125.stdout:3040: expect deleted 2021-03-24T22:05:59.470 INFO:tasks.rados.rados.0.smithi125.stdout:3041: write oid 40 current snap is 349 2021-03-24T22:05:59.470 INFO:tasks.rados.rados.0.smithi125.stdout:3041: seq_num 1050 ranges {447350=668164,1560093=406693,2584613=480437} 2021-03-24T22:05:59.474 INFO:tasks.rados.rados.0.smithi125.stdout:3041: writing smithi12556470-40 from 447350 to 1115514 tid 1 2021-03-24T22:05:59.479 INFO:tasks.rados.rados.0.smithi125.stdout:3041: writing smithi12556470-40 from 1560093 to 1966786 tid 2 2021-03-24T22:05:59.480 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:59.486 INFO:tasks.rados.rados.0.smithi125.stdout:3041: writing smithi12556470-40 from 2584613 to 3065050 tid 3 2021-03-24T22:05:59.486 INFO:tasks.rados.rados.0.smithi125.stdout:3042: write oid 44 current snap is 349 2021-03-24T22:05:59.486 INFO:tasks.rados.rados.0.smithi125.stdout:3041: finishing write tid 1 to smithi12556470-40 2021-03-24T22:05:59.486 INFO:tasks.rados.rados.0.smithi125.stdout:3041: finishing write tid 2 to smithi12556470-40 2021-03-24T22:05:59.487 INFO:tasks.rados.rados.0.smithi125.stdout:3042: seq_num 1051 ranges {464157=743413,1618575=666704,3012036=408062} 2021-03-24T22:05:59.492 INFO:tasks.rados.rados.0.smithi125.stdout:3042: writing smithi12556470-44 from 464157 to 1207570 tid 1 2021-03-24T22:05:59.498 INFO:tasks.rados.rados.0.smithi125.stdout:3042: writing smithi12556470-44 from 1618575 to 2285279 tid 2 2021-03-24T22:05:59.506 INFO:tasks.rados.rados.0.smithi125.stdout:3042: writing smithi12556470-44 from 3012036 to 3420098 tid 3 2021-03-24T22:05:59.506 INFO:tasks.rados.rados.0.smithi125.stdout:3040: done (2 left) 2021-03-24T22:05:59.507 INFO:tasks.rados.rados.0.smithi125.stdout:3043: read oid 34 snap -1 2021-03-24T22:05:59.507 INFO:tasks.rados.rados.0.smithi125.stdout:3043: expect deleted 2021-03-24T22:05:59.507 INFO:tasks.rados.rados.0.smithi125.stdout:3044: snap_remove snap 260 2021-03-24T22:05:59.534 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:05:59.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:59.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:05:59.639 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:59.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:59 smithi125 conmon[24927]: cluster 2021-03-24T22:05:57.864589+0000 osd.3 (osd.3) 156 : cluster [DBG] 5.13 scrub ok 2021-03-24T22:05:59.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:59 smithi125 conmon[24927]: cluster 2021-03-24T22:05:58.410200+0000 osd.0 (osd.0) 153 : cluster [DBG] 3.17 scrub ok 2021-03-24T22:05:59.652 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:59 smithi125 conmon[24927]: cluster 2021-03-24T22:05:58.443450+0000 mon.a (mon.0) 1993 : cluster [DBG] osdmap e788: 8 total, 8 up, 8 in 2021-03-24T22:05:59.653 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:05:59 smithi125 conmon[24927]: cluster 2021-03-24T22:05:59.132863+0000 osd.6 (osd.6) 79 : cluster [DBG] 7.d scrub ok 2021-03-24T22:05:59.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:59 smithi125 conmon[19936]: cluster 2021-03-24T22:05:59.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:59 smithi125 conmon[19936]: 2021-03-24 2021-03-24T22:05:59.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:59 smithi125 conmon[19936]: T22:05:57.864589+0000 osd.3 (osd.3) 156 : cluster [DBG] 5.13 scrub ok 2021-03-24T22:05:59.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:59 smithi125 conmon[19936]: cluster 2021-03-24T22:05:59.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:59 smithi125 conmon[19936]: 2021-03-24T22 2021-03-24T22:05:59.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:59 smithi125 conmon[19936]: :05:58.410200 2021-03-24T22:05:59.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:59 smithi125 conmon[19936]: +0000 osd.0 (osd. 2021-03-24T22:05:59.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:59 smithi125 conmon[19936]: 0) 153 : cluster [DBG] 3.17 scrub ok 2021-03-24T22:05:59.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:59 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T22:05:59.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:59 smithi125 conmon[19936]: -24T22:05:58.443450+0000 mon.a (mon.0) 1993 : cluster [DBG] 2021-03-24T22:05:59.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:59 smithi125 conmon[19936]: osdmap e788: 8 total, 8 up, 8 in 2021-03-24T22:05:59.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:59 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:05:59.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:59 smithi125 conmon[19936]: 03-24T22:05:59.132863+0000 osd.6 (osd.6) 79 : cluster 2021-03-24T22:05:59.656 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:05:59 smithi125 conmon[19936]: [DBG] 7.d scrub ok 2021-03-24T22:05:59.700 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:05:59.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:59.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:05:59.804 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:59.820 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:59 smithi142 conmon[20125]: cluster 2021-03-24T22:05:57. 2021-03-24T22:05:59.820 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:59 smithi142 conmon[20125]: 864589+0000 osd.3 (osd.3) 156 : cluster [DBG] 5.13 scrub ok 2021-03-24T22:05:59.821 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:59 smithi142 conmon[20125]: cluster 2021-03-24T22:05:58.410200+0000 osd.0 (osd 2021-03-24T22:05:59.821 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:59 smithi142 conmon[20125]: .0) 153 : cluster [DBG] 3.17 scrub ok 2021-03-24T22:05:59.821 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:59 smithi142 conmon[20125]: cluster 2021-03-24T22: 2021-03-24T22:05:59.821 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:59 smithi142 conmon[20125]: 05:58.443450+0000 mon.a (mon.0) 2021-03-24T22:05:59.821 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:59 smithi142 conmon[20125]: 1993 : cluster [DBG] osdmap e788: 8 total, 8 up, 8 in 2021-03-24T22:05:59.822 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:59 smithi142 conmon[20125]: cluster 2021-03-24T22:05 2021-03-24T22:05:59.822 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:59 smithi142 conmon[20125]: :59.132863+0000 osd.6 (osd.6) 79 : cluster [DBG] 2021-03-24T22:05:59.822 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:05:59 smithi142 conmon[20125]: 7.d scrub ok 2021-03-24T22:05:59.860 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:05:59.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:05:59.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:05:59.967 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:00.027 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:00.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:00.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:06:00.134 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:00.179 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:06:00.195 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:00.196 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.2 dump_ops_in_flight 2021-03-24T22:06:00.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:00.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:06:00.222 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:06:00.301 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:00.353 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:00.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:00.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:06:00.419 INFO:tasks.rados.rados.0.smithi125.stdout:3044: done (3 left) 2021-03-24T22:06:00.420 INFO:tasks.rados.rados.0.smithi125.stdout:3045: write oid 20 current snap is 349 2021-03-24T22:06:00.420 INFO:tasks.rados.rados.0.smithi125.stdout:3041: finishing write tid 3 to smithi12556470-40 2021-03-24T22:06:00.420 INFO:tasks.rados.rados.0.smithi125.stdout:3041: finishing write tid 4 to smithi12556470-40 2021-03-24T22:06:00.420 INFO:tasks.rados.rados.0.smithi125.stdout:3041: finishing write tid 5 to smithi12556470-40 2021-03-24T22:06:00.421 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 40 v 1898 (ObjNum 1050 snap 349 seq_num 1050) dirty exists 2021-03-24T22:06:00.421 INFO:tasks.rados.rados.0.smithi125.stdout:3041: left oid 40 (ObjNum 1050 snap 349 seq_num 1050) 2021-03-24T22:06:00.421 INFO:tasks.rados.rados.0.smithi125.stdout:3042: finishing write tid 1 to smithi12556470-44 2021-03-24T22:06:00.421 INFO:tasks.rados.rados.0.smithi125.stdout:3042: finishing write tid 2 to smithi12556470-44 2021-03-24T22:06:00.421 INFO:tasks.rados.rados.0.smithi125.stdout:3042: finishing write tid 3 to smithi12556470-44 2021-03-24T22:06:00.422 INFO:tasks.rados.rados.0.smithi125.stdout:3042: finishing write tid 4 to smithi12556470-44 2021-03-24T22:06:00.422 INFO:tasks.rados.rados.0.smithi125.stdout:3042: finishing write tid 5 to smithi12556470-44 2021-03-24T22:06:00.422 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 44 v 1438 (ObjNum 1051 snap 349 seq_num 1051) dirty exists 2021-03-24T22:06:00.422 INFO:tasks.rados.rados.0.smithi125.stdout:3042: left oid 44 (ObjNum 1051 snap 349 seq_num 1051) 2021-03-24T22:06:00.423 INFO:tasks.rados.rados.0.smithi125.stdout:3045: seq_num 1052 ranges {480964=418662,1560704=410362,2523106=619334} 2021-03-24T22:06:00.432 INFO:tasks.rados.rados.0.smithi125.stdout:3045: writing smithi12556470-20 from 480964 to 899626 tid 1 2021-03-24T22:06:00.433 INFO:tasks.rados.rados.0.smithi125.stdout:3045: writing smithi12556470-20 from 1560704 to 1971066 tid 2 2021-03-24T22:06:00.441 INFO:tasks.rados.rados.0.smithi125.stdout:3045: writing smithi12556470-20 from 2523106 to 3142440 tid 3 2021-03-24T22:06:00.442 INFO:teuthology.orchestra.run.smithi125.stderr:reweighted osd.6 to 1 (10000) 2021-03-24T22:06:00.444 INFO:tasks.rados.rados.0.smithi125.stdout:3041: done (3 left) 2021-03-24T22:06:00.445 INFO:tasks.rados.rados.0.smithi125.stdout:3042: done (2 left) 2021-03-24T22:06:00.445 INFO:tasks.rados.rados.0.smithi125.stdout:3043: done (1 left) 2021-03-24T22:06:00.445 INFO:tasks.rados.rados.0.smithi125.stdout:3046: read oid 14 snap 316 2021-03-24T22:06:00.446 INFO:tasks.rados.rados.0.smithi125.stdout:3046: expect (ObjNum 727 snap 233 seq_num 727) 2021-03-24T22:06:00.446 INFO:tasks.rados.rados.0.smithi125.stdout:3047: read oid 30 snap -1 2021-03-24T22:06:00.446 INFO:tasks.rados.rados.0.smithi125.stdout:3047: expect (ObjNum 1029 snap 342 seq_num 1029) 2021-03-24T22:06:00.446 INFO:tasks.rados.rados.0.smithi125.stdout:3048: read oid 38 snap 346 2021-03-24T22:06:00.446 INFO:tasks.rados.rados.0.smithi125.stdout:3048: expect (ObjNum 973 snap 318 seq_num 973) 2021-03-24T22:06:00.447 INFO:tasks.rados.rados.0.smithi125.stdout:3049: read oid 17 snap -1 2021-03-24T22:06:00.447 INFO:tasks.rados.rados.0.smithi125.stdout:3049: expect (ObjNum 1013 snap 336 seq_num 1013) 2021-03-24T22:06:00.447 INFO:tasks.rados.rados.0.smithi125.stdout:3050: snap_remove snap 307 2021-03-24T22:06:00.460 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:00.519 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:00.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:00.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:06:00.626 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:00.685 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:00.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:00.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:06:00.783 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[19936]: cluster 2021-03-24T22:06:00.783 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[19936]: 2021-03-24T22:06:00.783 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[19936]: 2021 2021-03-24T22:06:00.783 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[19936]: - 2021-03-24T22:06:00.784 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[19936]: 03 2021-03-24T22:06:00.784 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[19936]: - 2021-03-24T22:06:00.784 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[19936]: 24 2021-03-24T22:06:00.784 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[19936]: T 2021-03-24T22:06:00.785 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[19936]: 22 2021-03-24T22:06:00.785 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[19936]: :05:58.994053+0000 mgr.y (mgr.14144) 669 : cluster [DBG] pgmap v1345: 168 pgs: 168 active+clean; 542 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail 2021-03-24T22:06:00.785 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[19936]: cluster 2021-03-24T22:05:59.171792+0000 osd.7 (osd.7) 131 : cluster [DBG] 1.0 scrub ok 2021-03-24T22:06:00.785 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[19936]: cluster 2021-03-24T22:05:59.395647+0000 osd.0 (osd.0) 154 : cluster [DBG] 6.10 scrub ok 2021-03-24T22:06:00.785 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[19936]: cluster 2021-03-24T22:05:59.463452+0000 mon.a (mon.0) 1994 : cluster [DBG] osdmap e789: 8 total, 8 up, 8 in 2021-03-24T22:06:00.786 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[19936]: cluster 2021-03-24T22:05:59.526937+0000 osd.1 (osd.1) 90 : cluster [DBG] 7.b scrub ok 2021-03-24T22:06:00.786 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[19936]: cluster 2021-03-24T22:06:00.786 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[19936]: 2021 2021-03-24T22:06:00.786 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[19936]: - 2021-03-24T22:06:00.787 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[19936]: 03 2021-03-24T22:06:00.787 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[19936]: - 2021-03-24T22:06:00.787 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[19936]: 24 2021-03-24T22:06:00.787 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[19936]: T22:06:00.101067+0000 osd.6 (osd.6) 80 : cluster [DBG] 5.1f scrub ok 2021-03-24T22:06:00.788 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[19936]: audit 2021-03-24T22:06:00.788 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[19936]: 2021 2021-03-24T22:06:00.788 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[19936]: - 2021-03-24T22:06:00.788 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[19936]: 03 2021-03-24T22:06:00.789 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[19936]: - 2021-03-24T22:06:00.789 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[19936]: 24 2021-03-24T22:06:00.789 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[19936]: T 2021-03-24T22:06:00.789 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[19936]: 22:06:00.286389+0000 mon.c (mon.2) 115 : audit [INF] from='client.? v1:172.21.15.125:0/4186628508' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-03-24T22:06:00.790 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[19936]: audit 2021-03-24T22:06:00.286904+0000 mon.a (mon.0) 1995 : audit [INF] from='client.35819 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-03-24T22:06:00.790 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[19936]: audit 2021-03-24T22:06:00.414273+0000 mon.a (mon.0) 1996 : audit [INF] from='client.35819 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2021-03-24T22:06:00.790 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[19936]: cluster 2021-03-24T22:06:00.414379+0000 mon.a (mon.0) 1997 : cluster [DBG] osdmap e790: 8 total, 8 up, 8 in 2021-03-24T22:06:00.791 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: cluster 2021-03-24T22:06:00.791 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: 2021-03-24T22:06:00.791 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: 2021 2021-03-24T22:06:00.792 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: - 2021-03-24T22:06:00.792 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: 03 2021-03-24T22:06:00.792 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: - 2021-03-24T22:06:00.792 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: 24 2021-03-24T22:06:00.793 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: T 2021-03-24T22:06:00.793 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: 22 2021-03-24T22:06:00.793 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: : 2021-03-24T22:06:00.793 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: 05 2021-03-24T22:06:00.794 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: : 2021-03-24T22:06:00.794 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: 58 2021-03-24T22:06:00.794 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: . 2021-03-24T22:06:00.795 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: 994053 2021-03-24T22:06:00.795 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: +0000 2021-03-24T22:06:00.795 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: 2021-03-24T22:06:00.795 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: mgr.y 2021-03-24T22:06:00.796 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: ( 2021-03-24T22:06:00.796 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: mgr 2021-03-24T22:06:00.796 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: . 2021-03-24T22:06:00.796 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: 14144 2021-03-24T22:06:00.796 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: ) 2021-03-24T22:06:00.797 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: 669 2021-03-24T22:06:00.797 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: : 2021-03-24T22:06:00.797 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: cluster 2021-03-24T22:06:00.797 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: 2021-03-24T22:06:00.798 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: [DBG] 2021-03-24T22:06:00.798 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: 2021-03-24T22:06:00.798 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: pgmap v1345: 168 pgs: 168 active+clean; 542 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail 2021-03-24T22:06:00.798 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: 2021-03-24T22:06:00.799 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: cluster 2021-03-24T22:06:00.799 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: 2021-03-24T22:06:00.799 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: 2021 2021-03-24T22:06:00.799 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: - 2021-03-24T22:06:00.800 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: 03 2021-03-24T22:06:00.800 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: - 2021-03-24T22:06:00.800 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: 24 2021-03-24T22:06:00.800 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: T 2021-03-24T22:06:00.800 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: 22 2021-03-24T22:06:00.801 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: : 2021-03-24T22:06:00.801 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: 05 2021-03-24T22:06:00.801 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: : 2021-03-24T22:06:00.801 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: 59 2021-03-24T22:06:00.802 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: .171792+0000 osd.7 (osd.7) 131 : 2021-03-24T22:06:00.802 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: cluster [DBG] 1.0 scrub ok 2021-03-24T22:06:00.802 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:06:00.802 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: 05:59.395647+0000 osd.0 (osd.0) 154 : cluster [DBG] 6.10 scrub ok 2021-03-24T22:06:00.803 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:06:00.803 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: 24T22:05:59.463452+0000 mon.a (mon 2021-03-24T22:06:00.803 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: .0) 1994 : cluster [DBG] osdmap e789: 8 total, 8 up, 8 in 2021-03-24T22:06:00.803 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: cluster 2021-03-24T22:06:00.804 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: 2021-03-24T22:05:59.526937+0000 2021-03-24T22:06:00.804 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: osd.1 (osd.1) 90 : cluster [DBG] 2021-03-24T22:06:00.804 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: 7.b scrub ok 2021-03-24T22:06:00.804 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: cluster 2021-03-24T22 2021-03-24T22:06:00.804 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: :06:00.101067+0000 osd.6 (osd 2021-03-24T22:06:00.805 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: .6) 80 : cluster [DBG] 5.1f scrub ok 2021-03-24T22:06:00.805 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: audit 2021 2021-03-24T22:06:00.805 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: -03-24T22:06:00.286389+0000 2021-03-24T22:06:00.806 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: mon.c (mon.2) 115 : audit [INF] 2021-03-24T22:06:00.806 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: from='client.? v1:172.21.15.125:0/4186628508' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-03-24T22:06:00.807 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: audit 2021-03-24 2021-03-24T22:06:00.807 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: T22:06:00.286904+0000 mon.a ( 2021-03-24T22:06:00.808 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: mon.0) 1995 : audit [INF] from='client.35819 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-03-24T22:06:00.808 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: audit 2021 2021-03-24T22:06:00.809 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: -03-24T22:06:00.414273 2021-03-24T22:06:00.809 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: +0000 mon.a (mon.0) 1996 : audit 2021-03-24T22:06:00.809 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: [INF] from='client.35819 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2021-03-24T22:06:00.810 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: cluster 2021-03-24T22 2021-03-24T22:06:00.810 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: :06:00.414379+0000 mon.a (mon. 2021-03-24T22:06:00.811 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:00 smithi125 conmon[24927]: 0) 1997 : cluster [DBG] osdmap e790: 8 total, 8 up, 8 in 2021-03-24T22:06:00.811 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:00.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:00 smithi142 conmon[20125]: cluster 2021-03-24T22:05:58.994053+0000 mgr.y ( 2021-03-24T22:06:00.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:00 smithi142 conmon[20125]: mgr.14144) 669 : cluster [DBG] pgmap v1345: 168 pgs: 168 active+clean; 542 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail 2021-03-24T22:06:00.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:00 smithi142 conmon[20125]: cluster 2021-03-24T22:05:59.171792+0000 osd.7 (osd.7) 131 : cluster [DBG] 1.0 scrub ok 2021-03-24T22:06:00.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:00 smithi142 conmon[20125]: cluster 2021-03-24T22:05:59.395647+0000 osd.0 ( 2021-03-24T22:06:00.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:00 smithi142 conmon[20125]: osd.0) 154 : cluster [DBG] 6.10 scrub ok 2021-03-24T22:06:00.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:00 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:06:00.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:00 smithi142 conmon[20125]: -03-24T22:05: 2021-03-24T22:06:00.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:00 smithi142 conmon[20125]: 59.463452+0000 mon.a (mon.0) 1994 2021-03-24T22:06:00.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:00 smithi142 conmon[20125]: : cluster [DBG] osdmap e789: 8 total, 8 up, 8 in 2021-03-24T22:06:00.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:00 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:06:00.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:00 smithi142 conmon[20125]: -24T22:05:59.526937+0000 2021-03-24T22:06:00.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:00 smithi142 conmon[20125]: osd.1 (osd.1) 90 : cluster [DBG] 2021-03-24T22:06:00.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:00 smithi142 conmon[20125]: 7.b scrub ok 2021-03-24T22:06:00.872 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:00 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:06:00.872 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:00 smithi142 conmon[20125]: 22:06:00.101067+0000 osd.6 ( 2021-03-24T22:06:00.872 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:00 smithi142 conmon[20125]: osd.6) 80 : cluster [DBG] 5.1f scrub ok 2021-03-24T22:06:00.872 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:00 smithi142 conmon[20125]: audit 2021-03-24T22:06:00.872 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:00 smithi142 conmon[20125]: 2021-03-24T22: 2021-03-24T22:06:00.873 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:00 smithi142 conmon[20125]: 06:00.286389+0000 mon.c (mon.2 2021-03-24T22:06:00.873 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:00 smithi142 conmon[20125]: ) 115 : audit [INF] from='client.? v1:172.21.15.125:0/4186628508' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-03-24T22:06:00.873 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:00 smithi142 conmon[20125]: audit 2021 2021-03-24T22:06:00.873 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:00 smithi142 conmon[20125]: -03-24T22:06: 2021-03-24T22:06:00.874 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:00 smithi142 conmon[20125]: 00.286904+0000 mon.a (mon.0) 1995 2021-03-24T22:06:00.874 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:00 smithi142 conmon[20125]: : audit [INF] from='client.35819 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-03-24T22:06:00.874 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:00 smithi142 conmon[20125]: audit 2021-03 2021-03-24T22:06:00.874 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:00 smithi142 conmon[20125]: -24T22:06:00.414273 2021-03-24T22:06:00.874 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:00 smithi142 conmon[20125]: +0000 mon.a (mon.0) 1996 : audit 2021-03-24T22:06:00.875 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:00 smithi142 conmon[20125]: [INF] from='client.35819 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2021-03-24T22:06:00.875 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:00 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:06:00.875 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:00 smithi142 conmon[20125]: 22:06:00.414379+0000 mon.a ( 2021-03-24T22:06:00.875 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:00 smithi142 conmon[20125]: mon.0) 1997 : cluster [DBG] osdmap e790: 8 total, 8 up, 8 in 2021-03-24T22:06:00.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:00.877 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:00.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:06:00.978 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:01.035 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:01.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:01.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:06:01.142 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:01.204 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:01.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:01.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:06:01.309 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:01.364 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:01.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:01.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:06:01.442 INFO:tasks.rados.rados.0.smithi125.stdout:3046: expect (ObjNum 727 snap 233 seq_num 727) 2021-03-24T22:06:01.470 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:01.521 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:01.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:01.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:01.625 INFO:tasks.rados.rados.0.smithi125.stdout:3048: expect (ObjNum 973 snap 318 seq_num 973) 2021-03-24T22:06:01.627 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:01.686 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:01.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:01.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:06:01.756 INFO:tasks.rados.rados.0.smithi125.stdout:3047: expect (ObjNum 1029 snap 342 seq_num 1029) 2021-03-24T22:06:01.793 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:01.806 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:01 smithi125 conmon[24927]: cluster 2021-03-24T22:06:00 2021-03-24T22:06:01.806 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:01 smithi125 conmon[24927]: .436494+0000 osd.0 (osd.0) 155 : cluster [DBG] 3.6 scrub ok 2021-03-24T22:06:01.806 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:01 smithi125 conmon[24927]: cluster 2021-03-24T22 2021-03-24T22:06:01.807 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:01 smithi125 conmon[24927]: :06:00.459317+0000 osd.2 (osd. 2021-03-24T22:06:01.807 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:01 smithi125 conmon[24927]: 2) 126 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:06:01.807 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:01 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:06:01.807 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:01 smithi125 conmon[24927]: 24T22:06:00.463864+0000 osd.2 (osd.2) 127 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:06:01.808 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:01 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:06:01.808 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:01 smithi125 conmon[24927]: -03-24T22:06:01. 2021-03-24T22:06:01.808 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:01 smithi125 conmon[24927]: 013557+0000 osd.2 (osd.2) 128 2021-03-24T22:06:01.808 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:01 smithi125 conmon[24927]: : cluster [DBG] 5.a scrub ok 2021-03-24T22:06:01.808 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:01 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:06:01.809 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:01 smithi125 conmon[24927]: -03-24T22:06:01.427818+0000 mon.a ( 2021-03-24T22:06:01.809 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:01 smithi125 conmon[24927]: mon.0) 1998 : cluster [DBG] osdmap e791: 8 total, 8 up, 8 in 2021-03-24T22:06:01.809 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:01 smithi125 conmon[19936]: cluster 2021-03-24T22:06:00.436494 2021-03-24T22:06:01.810 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:01 smithi125 conmon[19936]: +0000 osd.0 (osd.0) 155 : cluster [DBG] 3.6 scrub ok 2021-03-24T22:06:01.810 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:01 smithi125 conmon[19936]: cluster 2021-03-24T22:06:00.459317+0000 osd.2 (osd. 2021-03-24T22:06:01.810 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:01 smithi125 conmon[19936]: 2) 126 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:06:01.810 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:01 smithi125 conmon[19936]: cluster 2021-03-24T22:06:01.811 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:01 smithi125 conmon[19936]: 2021-03-24T22 2021-03-24T22:06:01.811 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:01 smithi125 conmon[19936]: :06:00.463864+0000 osd.2 (osd.2) 127 2021-03-24T22:06:01.811 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:01 smithi125 conmon[19936]: : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:06:01.812 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:01 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T22:06:01.812 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:01 smithi125 conmon[19936]: 22:06:01.013557+0000 osd.2 (osd.2) 128 : cluster 2021-03-24T22:06:01.812 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:01 smithi125 conmon[19936]: [DBG] 5.a scrub ok 2021-03-24T22:06:01.812 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:01 smithi125 conmon[19936]: cluster 2021-03-24T22:06:01.813 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:01 smithi125 conmon[19936]: 2021-03-24T22:06:01.427818+0000 mon.a (mon.0) 1998 : cluster [DBG] 2021-03-24T22:06:01.813 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:01 smithi125 conmon[19936]: osdmap e791: 8 total, 8 up, 8 in 2021-03-24T22:06:01.849 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:01.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:01.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:06:01.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:01 smithi142 conmon[20125]: cluster 2021-03-24T22:06:00.436494+0000 osd.0 (osd. 2021-03-24T22:06:01.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:01 smithi142 conmon[20125]: 0) 155 : cluster [DBG] 3.6 scrub ok 2021-03-24T22:06:01.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:01 smithi142 conmon[20125]: cluster 2021-03-24T22:06:00.459317+0000 osd.2 (osd.2) 126 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:06:01.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:01 smithi142 conmon[20125]: cluster 2021-03-24T22: 2021-03-24T22:06:01.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:01 smithi142 conmon[20125]: 06:00.463864+0000 osd.2 (osd.2) 127 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:06:01.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:01 smithi142 conmon[20125]: cluster 2021-03-24T22: 2021-03-24T22:06:01.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:01 smithi142 conmon[20125]: 06:01.013557+0000 osd.2 (osd.2) 2021-03-24T22:06:01.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:01 smithi142 conmon[20125]: 128 : cluster [DBG] 5.a scrub ok 2021-03-24T22:06:01.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:01 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:06:01.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:01 smithi142 conmon[20125]: 22:06:01.427818+0000 mon.a ( 2021-03-24T22:06:01.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:01 smithi142 conmon[20125]: mon.0) 1998 : cluster [DBG] 2021-03-24T22:06:01.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:01 smithi142 conmon[20125]: osdmap e791: 8 total, 8 up, 8 in 2021-03-24T22:06:01.958 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:01.986 INFO:tasks.rados.rados.0.smithi125.stdout:3045: finishing write tid 1 to smithi12556470-20 2021-03-24T22:06:01.987 INFO:tasks.rados.rados.0.smithi125.stdout:3045: finishing write tid 2 to smithi12556470-20 2021-03-24T22:06:01.987 INFO:tasks.rados.rados.0.smithi125.stdout:3046: done (5 left) 2021-03-24T22:06:01.987 INFO:tasks.rados.rados.0.smithi125.stdout:3047: done (4 left) 2021-03-24T22:06:01.987 INFO:tasks.rados.rados.0.smithi125.stdout:3048: done (3 left) 2021-03-24T22:06:01.987 INFO:tasks.rados.rados.0.smithi125.stdout:3050: done (2 left) 2021-03-24T22:06:01.988 INFO:tasks.rados.rados.0.smithi125.stdout:3051: read oid 50 snap -1 2021-03-24T22:06:01.988 INFO:tasks.rados.rados.0.smithi125.stdout:3051: expect (ObjNum 1013 snap 336 seq_num 1013) 2021-03-24T22:06:01.988 INFO:tasks.rados.rados.0.smithi125.stdout:3052: copy_from oid 13 from oid 24 current snap is 349 2021-03-24T22:06:01.988 INFO:tasks.rados.rados.0.smithi125.stdout:3045: finishing write tid 3 to smithi12556470-20 2021-03-24T22:06:01.989 INFO:tasks.rados.rados.0.smithi125.stdout:3049: expect (ObjNum 1013 snap 336 seq_num 1013) 2021-03-24T22:06:02.019 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:02.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:02.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:06:02.125 INFO:tasks.rados.rados.0.smithi125.stdout:3045: finishing write tid 4 to smithi12556470-20 2021-03-24T22:06:02.126 INFO:tasks.rados.rados.0.smithi125.stdout:3045: finishing write tid 5 to smithi12556470-20 2021-03-24T22:06:02.126 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 20 v 997 (ObjNum 1052 snap 349 seq_num 1052) dirty exists 2021-03-24T22:06:02.126 INFO:tasks.rados.rados.0.smithi125.stdout:3045: left oid 20 (ObjNum 1052 snap 349 seq_num 1052) 2021-03-24T22:06:02.126 INFO:tasks.rados.rados.0.smithi125.stdout:3051: expect (ObjNum 1013 snap 336 seq_num 1013) 2021-03-24T22:06:02.128 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:02.188 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:02.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:02.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:06:02.259 INFO:tasks.rados.rados.0.smithi125.stdout:3052: finishing copy_from to smithi12556470-13 2021-03-24T22:06:02.259 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 13 v 1457 (ObjNum 1048 snap 347 seq_num 1048) dirty exists 2021-03-24T22:06:02.260 INFO:tasks.rados.rados.0.smithi125.stdout:3052: finishing copy_from racing read to smithi12556470-13 2021-03-24T22:06:02.260 INFO:tasks.rados.rados.0.smithi125.stdout:3045: done (3 left) 2021-03-24T22:06:02.260 INFO:tasks.rados.rados.0.smithi125.stdout:3049: done (2 left) 2021-03-24T22:06:02.260 INFO:tasks.rados.rados.0.smithi125.stdout:3051: done (1 left) 2021-03-24T22:06:02.261 INFO:tasks.rados.rados.0.smithi125.stdout:3052: done (0 left) 2021-03-24T22:06:02.261 INFO:tasks.rados.rados.0.smithi125.stdout:3053: copy_from oid 48 from oid 16 current snap is 349 2021-03-24T22:06:02.261 INFO:tasks.rados.rados.0.smithi125.stdout:3054: write oid 37 current snap is 349 2021-03-24T22:06:02.261 INFO:tasks.rados.rados.0.smithi125.stdout:3054: seq_num 1053 ranges {497771=493911,1619186=554020,2950529=430606} 2021-03-24T22:06:02.275 INFO:tasks.rados.rados.0.smithi125.stdout:3054: writing smithi12556470-37 from 497771 to 991682 tid 1 2021-03-24T22:06:02.282 INFO:tasks.rados.rados.0.smithi125.stdout:3054: writing smithi12556470-37 from 1619186 to 2173206 tid 2 2021-03-24T22:06:02.289 INFO:tasks.rados.rados.0.smithi125.stdout:3054: writing smithi12556470-37 from 2950529 to 3381135 tid 3 2021-03-24T22:06:02.290 INFO:tasks.rados.rados.0.smithi125.stdout:3055: snap_remove snap 290 2021-03-24T22:06:02.292 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:02.349 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:02.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:02.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:02.456 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:02.480 INFO:tasks.rados.rados.0.smithi125.stdout:3055: done (2 left) 2021-03-24T22:06:02.480 INFO:tasks.rados.rados.0.smithi125.stdout:3056: read oid 19 snap -1 2021-03-24T22:06:02.481 INFO:tasks.rados.rados.0.smithi125.stdout:3056: expect (ObjNum 1008 snap 335 seq_num 1008) 2021-03-24T22:06:02.481 INFO:tasks.rados.rados.0.smithi125.stdout:3057: rollback oid 8 current snap is 349 2021-03-24T22:06:02.481 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 8 to 314 2021-03-24T22:06:02.481 INFO:tasks.rados.rados.0.smithi125.stdout:3058: snap_create 2021-03-24T22:06:02.482 INFO:tasks.rados.rados.0.smithi125.stdout:3053: finishing copy_from to smithi12556470-48 2021-03-24T22:06:02.482 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 48 v 840 (ObjNum 605 snap 184 seq_num 605) dirty exists 2021-03-24T22:06:02.483 INFO:tasks.rados.rados.0.smithi125.stdout:3053: finishing copy_from racing read to smithi12556470-48 2021-03-24T22:06:02.483 INFO:tasks.rados.rados.0.smithi125.stdout:3054: finishing write tid 1 to smithi12556470-37 2021-03-24T22:06:02.483 INFO:tasks.rados.rados.0.smithi125.stdout:3054: finishing write tid 2 to smithi12556470-37 2021-03-24T22:06:02.483 INFO:tasks.rados.rados.0.smithi125.stdout:3054: finishing write tid 3 to smithi12556470-37 2021-03-24T22:06:02.484 INFO:tasks.rados.rados.0.smithi125.stdout:3054: finishing write tid 4 to smithi12556470-37 2021-03-24T22:06:02.484 INFO:tasks.rados.rados.0.smithi125.stdout:3054: finishing write tid 5 to smithi12556470-37 2021-03-24T22:06:02.484 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 37 v 849 (ObjNum 1053 snap 349 seq_num 1053) dirty exists 2021-03-24T22:06:02.484 INFO:tasks.rados.rados.0.smithi125.stdout:3054: left oid 37 (ObjNum 1053 snap 349 seq_num 1053) 2021-03-24T22:06:02.488 INFO:tasks.rados.rados.0.smithi125.stdout:3057: finishing rollback tid 1 to smithi12556470-8 2021-03-24T22:06:02.491 INFO:tasks.rados.rados.0.smithi125.stdout:3057: finishing rollback tid 2 to smithi12556470-8 2021-03-24T22:06:02.492 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 8 v 566 (ObjNum 941 snap 314 seq_num 941) dirty exists 2021-03-24T22:06:02.505 INFO:tasks.rados.rados.0.smithi125.stdout:3056: expect (ObjNum 1008 snap 335 seq_num 1008) 2021-03-24T22:06:02.514 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:02.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:02.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:02.621 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:02.678 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:02.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:02.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:06:02.784 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:02.795 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:02 smithi125 conmon[24927]: cluster 2021-03-24T22:06:00.994663+0000 mgr.y (mgr.14144) 2021-03-24T22:06:02.795 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:02 smithi125 conmon[24927]: 670 : cluster [DBG] pgmap v1348: 168 pgs: 168 active+clean; 536 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail 2021-03-24T22:06:02.795 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:02 smithi125 conmon[24927]: cluster 2021-03-24T22:06:01.442561+0000 osd.5 (osd.5) 16 : cluster [DBG] 3.5 scrub ok 2021-03-24T22:06:02.795 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:02 smithi125 conmon[24927]: cluster 2021-03-24T22:06:01.977303+0000 osd.2 (osd.2) 129 : cluster [DBG] 6.f scrub ok 2021-03-24T22:06:02.798 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:02 smithi125 conmon[19936]: cluster 2021-03-24T22:06:02.798 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:02 smithi125 conmon[19936]: 2021-03-24T22:06:02.798 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:02 smithi125 conmon[19936]: 2021 2021-03-24T22:06:02.799 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:02 smithi125 conmon[19936]: - 2021-03-24T22:06:02.799 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:02 smithi125 conmon[19936]: 03 2021-03-24T22:06:02.799 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:02 smithi125 conmon[19936]: - 2021-03-24T22:06:02.799 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:02 smithi125 conmon[19936]: 24 2021-03-24T22:06:02.800 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:02 smithi125 conmon[19936]: T 2021-03-24T22:06:02.800 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:02 smithi125 conmon[19936]: 22 2021-03-24T22:06:02.800 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:02 smithi125 conmon[19936]: : 2021-03-24T22:06:02.800 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:02 smithi125 conmon[19936]: 06 2021-03-24T22:06:02.800 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:02 smithi125 conmon[19936]: : 2021-03-24T22:06:02.801 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:02 smithi125 conmon[19936]: 00 2021-03-24T22:06:02.801 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:02 smithi125 conmon[19936]: .994663+0000 mgr.y (mgr.14144) 670 : cluster 2021-03-24T22:06:02.801 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:02 smithi125 conmon[19936]: [DBG] pgmap v1348: 168 pgs: 168 active+clean; 536 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail 2021-03-24T22:06:02.802 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:02 smithi125 conmon[19936]: cluster 2021-03-24T22:06:01.442561+0000 2021-03-24T22:06:02.802 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:02 smithi125 conmon[19936]: osd.5 (osd.5) 16 : cluster [DBG] 3.5 scrub ok 2021-03-24T22:06:02.802 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:02 smithi125 conmon[19936]: cluster 2021-03-24T22:06:01. 2021-03-24T22:06:02.802 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:02 smithi125 conmon[19936]: 977303+0000 osd.2 (osd.2) 129 : cluster [DBG] 6.f scrub ok 2021-03-24T22:06:02.802 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:02 smithi125 conmon[19936]: 2021-03-24T22:06:02.833 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:02.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:02.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:06:02.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:02 smithi142 conmon[20125]: cluster 2021-03-24T22:06:00.994663+0000 2021-03-24T22:06:02.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:02 smithi142 conmon[20125]: mgr.y (mgr.14144) 670 : cluster [DBG] pgmap v1348: 168 pgs: 168 active+clean; 536 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail 2021-03-24T22:06:02.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:02 smithi142 conmon[20125]: cluster 2021-03-24T22:06:01.442561+0000 osd.5 (osd.5) 16 : cluster [DBG] 3.5 scrub ok 2021-03-24T22:06:02.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:02 smithi142 conmon[20125]: cluster 2021-03-24T22:06:01 2021-03-24T22:06:02.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:02 smithi142 conmon[20125]: .977303+0000 osd.2 (osd.2) 129 : cluster [DBG] 6.f scrub ok 2021-03-24T22:06:02.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:02 smithi142 conmon[20125]: 2021-03-24T22:06:02.937 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:02.994 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:02.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:03.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:03.086 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:06:03.101 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:03.157 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:03.158 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd reweight 7 1 2021-03-24T22:06:03.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:03.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:06:03.263 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:03.312 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:03.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:03.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:06:03.419 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:03.479 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:03.481 INFO:tasks.rados.rados.0.smithi125.stdout:3053: done (4 left) 2021-03-24T22:06:03.481 INFO:tasks.rados.rados.0.smithi125.stdout:3054: done (3 left) 2021-03-24T22:06:03.481 INFO:tasks.rados.rados.0.smithi125.stdout:3056: done (2 left) 2021-03-24T22:06:03.482 INFO:tasks.rados.rados.0.smithi125.stdout:3057: done (1 left) 2021-03-24T22:06:03.482 INFO:tasks.rados.rados.0.smithi125.stdout:3058: done (0 left) 2021-03-24T22:06:03.482 INFO:tasks.rados.rados.0.smithi125.stdout:3059: snap_create 2021-03-24T22:06:03.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:03.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:03.585 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:03.642 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:03.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:03.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:06:03.749 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:03.764 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:03 smithi142 conmon[20125]: cluster 2021-03-24T22:06:02.409360+0000 osd.5 (osd.5) 17 : cluster [DBG] 5.16 deep-scrub ok 2021-03-24T22:06:03.764 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:03 smithi142 conmon[20125]: 2021-03-24T22:06:03.764 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:03 smithi142 conmon[20125]: cluster 2021-03-24T22:06:02.474961+0000 mon.a (mon 2021-03-24T22:06:03.764 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:03 smithi142 conmon[20125]: .0) 1999 : cluster [DBG] osdmap e792: 8 total, 8 up, 8 in 2021-03-24T22:06:03.765 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:03 smithi142 conmon[20125]: cluster 2021-03-24T22:06:02 2021-03-24T22:06:03.765 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:03 smithi142 conmon[20125]: .476560+0000 osd.0 (osd.0) 156 : cluster 2021-03-24T22:06:03.765 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:03 smithi142 conmon[20125]: [DBG] 6.18 scrub ok 2021-03-24T22:06:03.765 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:03 smithi142 conmon[20125]: cluster 2021-03-24T22:06:03.766 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:03 smithi142 conmon[20125]: 2021-03-24T22:06:02.509397+0000 osd.1 ( 2021-03-24T22:06:03.766 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:03 smithi142 conmon[20125]: osd.1) 91 : cluster [DBG] 3.4 scrub ok 2021-03-24T22:06:03.766 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:03 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:06:03.766 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:03 smithi142 conmon[20125]: 24T22:06:03.024079+0000 osd.2 ( 2021-03-24T22:06:03.766 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:03 smithi142 conmon[20125]: osd.2) 130 : cluster [DBG] 7.8 scrub ok 2021-03-24T22:06:03.767 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:03 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:06:03.767 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:03 smithi142 conmon[20125]: -24T22:06:03.477601+0000 mon.a 2021-03-24T22:06:03.767 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:03 smithi142 conmon[20125]: (mon.0) 2000 : cluster [DBG] osdmap e793: 8 total, 8 up, 8 in 2021-03-24T22:06:03.805 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:03.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:03.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:06:03.911 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:03.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:03 smithi125 conmon[19936]: cluster 2021-03-24T22:06:02.409360+0000 osd.5 (osd.5) 17 : cluster [DBG] 5.16 deep-scrub ok 2021-03-24T22:06:03.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:03 smithi125 conmon[19936]: cluster 2021-03-24T22:06:02.474961+0000 mon.a (mon.0) 1999 : cluster [DBG] osdmap e792: 8 total, 8 up, 8 in 2021-03-24T22:06:03.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:03 smithi125 conmon[19936]: cluster 2021-03-24T22:06:02.476560+0000 osd.0 (osd.0 2021-03-24T22:06:03.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:03 smithi125 conmon[19936]: ) 156 : cluster [DBG] 6.18 scrub ok 2021-03-24T22:06:03.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:03 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:06:03.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:03 smithi125 conmon[19936]: 03-24T22:06:02.509397+0000 osd.1 (osd.1 2021-03-24T22:06:03.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:03 smithi125 conmon[19936]: ) 91 : cluster [DBG] 3.4 scrub ok 2021-03-24T22:06:03.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:03 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T22:06:03.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:03 smithi125 conmon[19936]: -24T22:06:03.024079+0000 2021-03-24T22:06:03.915 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:03 smithi125 conmon[19936]: osd.2 (osd.2) 130 : cluster [DBG] 2021-03-24T22:06:03.915 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:03 smithi125 conmon[19936]: 7.8 scrub ok 2021-03-24T22:06:03.915 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:03 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T22:06:03.915 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:03 smithi125 conmon[19936]: 22:06:03.477601+0000 mon.a (mon 2021-03-24T22:06:03.915 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:03 smithi125 conmon[19936]: .0) 2000 : cluster [DBG] osdmap e793: 8 total, 8 up, 8 in 2021-03-24T22:06:03.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:03 smithi125 conmon[24927]: cluster 2021-03-24T22:06:03.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:03 smithi125 conmon[24927]: 2021-03-24 2021-03-24T22:06:03.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:03 smithi125 conmon[24927]: T22:06:02.409360+0000 osd.5 (osd.5) 17 : cluster [DBG] 5.16 deep-scrub ok 2021-03-24T22:06:03.917 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:03 smithi125 conmon[24927]: cluster 2021-03-24T22:06:02 2021-03-24T22:06:03.917 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:03 smithi125 conmon[24927]: .474961+0000 mon.a (mon.0) 1999 : cluster [DBG] osdmap e792: 8 total, 8 up, 8 in 2021-03-24T22:06:03.917 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:03 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:06:03.917 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:03 smithi125 conmon[24927]: 24T22:06:02.476560+0000 osd.0 (osd.0) 156 : cluster [DBG] 2021-03-24T22:06:03.918 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:03 smithi125 conmon[24927]: 6.18 scrub ok 2021-03-24T22:06:03.918 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:03 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:06:03.919 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:03 smithi125 conmon[24927]: -03-24T22 2021-03-24T22:06:03.919 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:03 smithi125 conmon[24927]: :06:02.509397+0000 osd.1 (osd.1) 91 : 2021-03-24T22:06:03.919 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:03 smithi125 conmon[24927]: cluster [DBG] 3.4 scrub ok 2021-03-24T22:06:03.919 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:03 smithi125 conmon[24927]: cluster 2021-03-24T22:06:03.024079+0000 osd.2 ( 2021-03-24T22:06:03.919 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:03 smithi125 conmon[24927]: osd.2) 130 : cluster [DBG] 7.8 scrub ok 2021-03-24T22:06:03.920 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:03 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:06:03.920 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:03 smithi125 conmon[24927]: -03-24T22:06:03.477601+0000 mon.a (mon.0) 2000 : cluster [DBG] osdmap e793: 8 total, 8 up, 8 in 2021-03-24T22:06:03.975 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:03.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:03.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:06:04.080 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:04.142 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:04.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:04.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:06:04.248 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:04.302 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:04.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:04.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:06:04.407 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:04.464 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:04.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:04.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:06:04.485 INFO:tasks.rados.rados.0.smithi125.stdout:3059: done (0 left) 2021-03-24T22:06:04.485 INFO:tasks.rados.rados.0.smithi125.stdout:3060: delete oid 6 current snap is 351 2021-03-24T22:06:04.498 INFO:tasks.rados.rados.0.smithi125.stdout:3060: done (0 left) 2021-03-24T22:06:04.498 INFO:tasks.rados.rados.0.smithi125.stdout:3061: write oid 19 current snap is 351 2021-03-24T22:06:04.499 INFO:tasks.rados.rados.0.smithi125.stdout:3061: seq_num 1055 ranges {531385=644409,1736150=557689,3005375=569503} 2021-03-24T22:06:04.504 INFO:tasks.rados.rados.0.smithi125.stdout:3061: writing smithi12556470-19 from 531385 to 1175794 tid 1 2021-03-24T22:06:04.511 INFO:tasks.rados.rados.0.smithi125.stdout:3061: writing smithi12556470-19 from 1736150 to 2293839 tid 2 2021-03-24T22:06:04.519 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:04 smithi125 conmon[19936]: cluster 2021-03-24T22:06:02.995426+0000 mgr.y (mgr 2021-03-24T22:06:04.519 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:04 smithi125 conmon[19936]: .14144) 671 : cluster [DBG] pgmap v1351: 168 pgs: 168 active+clean; 527 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail 2021-03-24T22:06:04.520 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:04 smithi125 conmon[19936]: cluster 2021-03-24T22:06:04.053520+0000 osd.2 (osd.2) 131 : cluster [DBG] 5.8 scrub ok 2021-03-24T22:06:04.520 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:04 smithi125 conmon[19936]: cluster 2021-03-24T22:06:04 2021-03-24T22:06:04.520 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:04 smithi125 conmon[19936]: .142848+0000 osd.6 (osd.6) 81 : cluster [DBG] 3.20 scrub ok 2021-03-24T22:06:04.520 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:04 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:06:04.520 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:04 smithi125 conmon[19936]: T22:06:04.179690+0000 osd.5 (osd.5) 2021-03-24T22:06:04.521 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:04 smithi125 conmon[19936]: 18 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:06:04.521 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:04 smithi125 conmon[19936]: cluster 2021-03-24T22:06: 2021-03-24T22:06:04.521 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:04 smithi125 conmon[19936]: 04.183829+0000 osd.5 (osd.5) 19 : cluster [DBG] purged_snaps scrub found 32 strays 2021-03-24T22:06:04.521 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:04 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T22:06:04.522 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:04 smithi125 conmon[19936]: -24T22:06:04.478997+0000 mon.a (mon 2021-03-24T22:06:04.522 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:04 smithi125 conmon[19936]: .0) 2001 : cluster [DBG] osdmap e794: 8 total, 8 up, 8 in 2021-03-24T22:06:04.523 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:04 smithi125 conmon[24927]: cluster 2021-03-24T22:06:04.523 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:04 smithi125 conmon[24927]: 2021-03-24T22:06: 2021-03-24T22:06:04.523 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:04 smithi125 conmon[24927]: 02.995426+0000 mgr.y (mgr.14144) 671 : cluster [DBG] pgmap v1351: 168 pgs: 168 active+clean; 527 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail 2021-03-24T22:06:04.523 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:04 smithi125 conmon[24927]: cluster 2021-03-24T22:06:04.524 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:04 smithi125 conmon[24927]: 2021 2021-03-24T22:06:04.524 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:04 smithi125 conmon[24927]: -03-24T22:06 2021-03-24T22:06:04.524 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:04 smithi125 conmon[24927]: :04.053520+0000 osd.2 (osd.2) 131 : cluster [DBG] 5.8 scrub ok 2021-03-24T22:06:04.525 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:04 smithi125 conmon[24927]: cluster 2021-03-24T22:06:04.525 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:04 smithi125 conmon[24927]: 2021-03-24T22:06:04 2021-03-24T22:06:04.525 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:04 smithi125 conmon[24927]: .142848+0000 osd.6 (osd.6) 81 : cluster 2021-03-24T22:06:04.525 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:04 smithi125 conmon[24927]: [DBG] 3.20 scrub ok 2021-03-24T22:06:04.525 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:04 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:06:04.526 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:04 smithi125 conmon[24927]: 03-24T22:06:04.179690+0000 osd.5 ( 2021-03-24T22:06:04.526 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:04 smithi125 conmon[24927]: osd.5) 18 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:06:04.526 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:04 smithi125 conmon[24927]: cluster 2021-03-24T22:06:04.526 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:04 smithi125 conmon[24927]: 2021 2021-03-24T22:06:04.526 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:04 smithi125 conmon[24927]: -03-24T22:06:04.183829 2021-03-24T22:06:04.527 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:04 smithi125 conmon[24927]: +0000 osd.5 (osd.5) 19 : cluster [DBG] purged_snaps scrub found 32 strays 2021-03-24T22:06:04.527 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:04 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T22:06:04.527 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:04 smithi125 conmon[24927]: -24T22:06:04.478997+0000 mon.a ( 2021-03-24T22:06:04.528 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:04 smithi125 conmon[24927]: mon.0) 2001 : cluster [DBG] osdmap e794: 8 total, 8 up, 8 in 2021-03-24T22:06:04.529 INFO:tasks.rados.rados.0.smithi125.stdout:3061: writing smithi12556470-19 from 3005375 to 3574878 tid 3 2021-03-24T22:06:04.529 INFO:tasks.rados.rados.0.smithi125.stdout:3062: write (excl) oid 26 current snap is 351 2021-03-24T22:06:04.529 INFO:tasks.rados.rados.0.smithi125.stdout:3062: seq_num 1056 ranges {548192=719658,1678279=417700,2516445=780775} 2021-03-24T22:06:04.529 INFO:tasks.rados.rados.0.smithi125.stdout:3062: writing smithi12556470-26 from 548192 to 1267850 tid 1 2021-03-24T22:06:04.531 INFO:tasks.rados.rados.0.smithi125.stdout:3062: writing smithi12556470-26 from 1678279 to 2095979 tid 2 2021-03-24T22:06:04.540 INFO:tasks.rados.rados.0.smithi125.stdout:3062: writing smithi12556470-26 from 2516445 to 3297220 tid 3 2021-03-24T22:06:04.540 INFO:tasks.rados.rados.0.smithi125.stdout:3063: write (excl) oid 38 current snap is 351 2021-03-24T22:06:04.540 INFO:tasks.rados.rados.0.smithi125.stdout:3061: finishing write tid 1 to smithi12556470-19 2021-03-24T22:06:04.541 INFO:tasks.rados.rados.0.smithi125.stdout:3063: seq_num 1057 ranges {564999=511260,1853114=561358,3060221=592047} 2021-03-24T22:06:04.545 INFO:tasks.rados.rados.0.smithi125.stdout:3063: writing smithi12556470-38 from 564999 to 1076259 tid 1 2021-03-24T22:06:04.553 INFO:tasks.rados.rados.0.smithi125.stdout:3063: writing smithi12556470-38 from 1853114 to 2414472 tid 2 2021-03-24T22:06:04.560 INFO:tasks.rados.rados.0.smithi125.stdout:3063: writing smithi12556470-38 from 3060221 to 3652268 tid 3 2021-03-24T22:06:04.561 INFO:tasks.rados.rados.0.smithi125.stdout:3064: copy_from oid 40 from oid 30 current snap is 351 2021-03-24T22:06:04.561 INFO:tasks.rados.rados.0.smithi125.stdout:3061: finishing write tid 2 to smithi12556470-19 2021-03-24T22:06:04.561 INFO:tasks.rados.rados.0.smithi125.stdout:3061: finishing write tid 3 to smithi12556470-19 2021-03-24T22:06:04.562 INFO:tasks.rados.rados.0.smithi125.stdout:3061: finishing write tid 4 to smithi12556470-19 2021-03-24T22:06:04.562 INFO:tasks.rados.rados.0.smithi125.stdout:3061: finishing write tid 5 to smithi12556470-19 2021-03-24T22:06:04.562 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 19 v 899 (ObjNum 1055 snap 351 seq_num 1055) dirty exists 2021-03-24T22:06:04.562 INFO:tasks.rados.rados.0.smithi125.stdout:3061: left oid 19 (ObjNum 1055 snap 351 seq_num 1055) 2021-03-24T22:06:04.562 INFO:tasks.rados.rados.0.smithi125.stdout:3062: finishing write tid 1 to smithi12556470-26 2021-03-24T22:06:04.563 INFO:tasks.rados.rados.0.smithi125.stdout:3062: finishing write tid 2 to smithi12556470-26 2021-03-24T22:06:04.563 INFO:tasks.rados.rados.0.smithi125.stdout:3061: done (3 left) 2021-03-24T22:06:04.563 INFO:tasks.rados.rados.0.smithi125.stdout:3065: write oid 31 current snap is 351 2021-03-24T22:06:04.563 INFO:tasks.rados.rados.0.smithi125.stdout:3065: seq_num 1058 ranges {581806=586509,1911596=421369,3087644=403319} 2021-03-24T22:06:04.568 INFO:tasks.rados.rados.0.smithi125.stdout:3065: writing smithi12556470-31 from 581806 to 1168315 tid 1 2021-03-24T22:06:04.571 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:04.574 INFO:tasks.rados.rados.0.smithi125.stdout:3065: writing smithi12556470-31 from 1911596 to 2332965 tid 2 2021-03-24T22:06:04.583 INFO:tasks.rados.rados.0.smithi125.stdout:3065: writing smithi12556470-31 from 3087644 to 3490963 tid 3 2021-03-24T22:06:04.583 INFO:tasks.rados.rados.0.smithi125.stdout:3066: read oid 45 snap -1 2021-03-24T22:06:04.583 INFO:tasks.rados.rados.0.smithi125.stdout:3066: expect (ObjNum 1014 snap 336 seq_num 1014) 2021-03-24T22:06:04.584 INFO:tasks.rados.rados.0.smithi125.stdout:3067: delete oid 29 current snap is 351 2021-03-24T22:06:04.584 INFO:tasks.rados.rados.0.smithi125.stdout:3062: finishing write tid 3 to smithi12556470-26 2021-03-24T22:06:04.584 INFO:tasks.rados.rados.0.smithi125.stdout:3062: finishing write tid 4 to smithi12556470-26 2021-03-24T22:06:04.584 INFO:tasks.rados.rados.0.smithi125.stdout:3062: finishing write tid 5 to smithi12556470-26 2021-03-24T22:06:04.584 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 26 v 841 (ObjNum 1056 snap 351 seq_num 1056) dirty exists 2021-03-24T22:06:04.585 INFO:tasks.rados.rados.0.smithi125.stdout:3062: left oid 26 (ObjNum 1056 snap 351 seq_num 1056) 2021-03-24T22:06:04.585 INFO:tasks.rados.rados.0.smithi125.stdout:3063: finishing write tid 1 to smithi12556470-38 2021-03-24T22:06:04.585 INFO:tasks.rados.rados.0.smithi125.stdout:3063: finishing write tid 2 to smithi12556470-38 2021-03-24T22:06:04.585 INFO:tasks.rados.rados.0.smithi125.stdout:3063: finishing write tid 3 to smithi12556470-38 2021-03-24T22:06:04.586 INFO:tasks.rados.rados.0.smithi125.stdout:3063: finishing write tid 4 to smithi12556470-38 2021-03-24T22:06:04.586 INFO:tasks.rados.rados.0.smithi125.stdout:3063: finishing write tid 5 to smithi12556470-38 2021-03-24T22:06:04.586 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 38 v 744 (ObjNum 1057 snap 351 seq_num 1057) dirty exists 2021-03-24T22:06:04.586 INFO:tasks.rados.rados.0.smithi125.stdout:3063: left oid 38 (ObjNum 1057 snap 351 seq_num 1057) 2021-03-24T22:06:04.588 INFO:tasks.rados.rados.0.smithi125.stdout:3062: done (5 left) 2021-03-24T22:06:04.589 INFO:tasks.rados.rados.0.smithi125.stdout:3063: done (4 left) 2021-03-24T22:06:04.589 INFO:tasks.rados.rados.0.smithi125.stdout:3067: done (3 left) 2021-03-24T22:06:04.589 INFO:tasks.rados.rados.0.smithi125.stdout:3068: copy_from oid 37 from oid 41 current snap is 351 2021-03-24T22:06:04.590 INFO:tasks.rados.rados.0.smithi125.stdout:3069: copy_from oid 22 from oid 23 current snap is 351 2021-03-24T22:06:04.590 INFO:tasks.rados.rados.0.smithi125.stdout:3070: snap_create 2021-03-24T22:06:04.593 INFO:tasks.rados.rados.0.smithi125.stdout:3066: expect (ObjNum 1014 snap 336 seq_num 1014) 2021-03-24T22:06:04.625 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:04.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:04.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:06:04.733 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:04.756 INFO:tasks.rados.rados.0.smithi125.stdout:3069: finishing copy_from to smithi12556470-22 2021-03-24T22:06:04.756 INFO:tasks.rados.rados.0.smithi125.stdout:3069: got expected ENOENT (src dne) 2021-03-24T22:06:04.756 INFO:tasks.rados.rados.0.smithi125.stdout:3069: finishing copy_from racing read to smithi12556470-22 2021-03-24T22:06:04.757 INFO:tasks.rados.rados.0.smithi125.stdout:3064: finishing copy_from to smithi12556470-40 2021-03-24T22:06:04.757 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 40 v 835 (ObjNum 1029 snap 342 seq_num 1029) dirty exists 2021-03-24T22:06:04.757 INFO:tasks.rados.rados.0.smithi125.stdout:3064: finishing copy_from racing read to smithi12556470-40 2021-03-24T22:06:04.757 INFO:tasks.rados.rados.0.smithi125.stdout:3065: finishing write tid 1 to smithi12556470-31 2021-03-24T22:06:04.758 INFO:tasks.rados.rados.0.smithi125.stdout:3065: finishing write tid 2 to smithi12556470-31 2021-03-24T22:06:04.758 INFO:tasks.rados.rados.0.smithi125.stdout:3065: finishing write tid 3 to smithi12556470-31 2021-03-24T22:06:04.758 INFO:tasks.rados.rados.0.smithi125.stdout:3065: finishing write tid 4 to smithi12556470-31 2021-03-24T22:06:04.758 INFO:tasks.rados.rados.0.smithi125.stdout:3065: finishing write tid 5 to smithi12556470-31 2021-03-24T22:06:04.759 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 31 v 1923 (ObjNum 1058 snap 351 seq_num 1058) dirty exists 2021-03-24T22:06:04.759 INFO:tasks.rados.rados.0.smithi125.stdout:3065: left oid 31 (ObjNum 1058 snap 351 seq_num 1058) 2021-03-24T22:06:04.759 INFO:tasks.rados.rados.0.smithi125.stdout:3068: finishing copy_from to smithi12556470-37 2021-03-24T22:06:04.759 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 37 v 1133 (ObjNum 916 snap 304 seq_num 916) dirty exists 2021-03-24T22:06:04.759 INFO:tasks.rados.rados.0.smithi125.stdout:3068: finishing copy_from racing read to smithi12556470-37 2021-03-24T22:06:04.760 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:06:04.782 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:04.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:04.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:06:04.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:04 smithi142 conmon[20125]: cluster 2021-03-24T22:06: 2021-03-24T22:06:04.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:04 smithi142 conmon[20125]: 02.995426+0000 mgr.y (mgr.14144) 671 : cluster [DBG] pgmap v1351: 168 pgs: 168 active+clean; 527 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail 2021-03-24T22:06:04.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:04 smithi142 conmon[20125]: cluster 2021-03-24T22:06:04.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:04 smithi142 conmon[20125]: 2021-03-24T22:06:04.053520 2021-03-24T22:06:04.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:04 smithi142 conmon[20125]: +0000 osd.2 (osd.2) 131 : cluster 2021-03-24T22:06:04.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:04 smithi142 conmon[20125]: [DBG] 5.8 scrub ok 2021-03-24T22:06:04.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:04 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:06:04.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:04 smithi142 conmon[20125]: -03-24T22:06:04. 2021-03-24T22:06:04.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:04 smithi142 conmon[20125]: 142848+0000 osd.6 (osd.6 2021-03-24T22:06:04.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:04 smithi142 conmon[20125]: ) 81 : cluster [DBG] 3.20 scrub ok 2021-03-24T22:06:04.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:04 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:06:04.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:04 smithi142 conmon[20125]: -03-24T22:06:04 2021-03-24T22:06:04.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:04 smithi142 conmon[20125]: .179690+0000 osd.5 (osd. 2021-03-24T22:06:04.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:04 smithi142 conmon[20125]: 5) 18 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:06:04.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:04 smithi142 conmon[20125]: 2021-03-24T22:06:04.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:04 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:06:04.872 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:04 smithi142 conmon[20125]: -24T22:06:04. 2021-03-24T22:06:04.872 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:04 smithi142 conmon[20125]: 183829+0000 osd.5 (osd.5) 2021-03-24T22:06:04.872 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:04 smithi142 conmon[20125]: 19 : cluster [DBG] purged_snaps scrub found 32 strays 2021-03-24T22:06:04.872 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:04 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:06:04.874 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:04 smithi142 conmon[20125]: -03-24T22:06:04 2021-03-24T22:06:04.875 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:04 smithi142 conmon[20125]: .478997+0000 mon.a (mon.0 2021-03-24T22:06:04.876 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:04 smithi142 conmon[20125]: ) 2001 : cluster [DBG] osdmap e794: 8 total, 8 up, 8 in 2021-03-24T22:06:04.888 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:04.935 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:04.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:04.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:06:04.950 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T22:06:04.971 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T22:06:04.992 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T22:06:05.019 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T22:06:05.041 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:05.057 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T22:06:05.075 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T22:06:05.095 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:05.099 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T22:06:05.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:05.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:06:05.123 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T22:06:05.126 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-24T22:06:05.126 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-24T22:06:05.127 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-24T22:06:05.127 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-24T22:06:05.128 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-24T22:06:05.128 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-24T22:06:05.128 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-24T22:06:05.129 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-24T22:06:05.203 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:05.260 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:05.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:05.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:06:05.366 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:05.426 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:05.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:05.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:05.434 INFO:tasks.rados.rados.0.smithi125.stdout:3064: done (5 left) 2021-03-24T22:06:05.434 INFO:tasks.rados.rados.0.smithi125.stdout:3065: done (4 left) 2021-03-24T22:06:05.435 INFO:tasks.rados.rados.0.smithi125.stdout:3066: done (3 left) 2021-03-24T22:06:05.435 INFO:tasks.rados.rados.0.smithi125.stdout:3068: done (2 left) 2021-03-24T22:06:05.435 INFO:tasks.rados.rados.0.smithi125.stdout:3069: done (1 left) 2021-03-24T22:06:05.435 INFO:tasks.rados.rados.0.smithi125.stdout:3070: done (0 left) 2021-03-24T22:06:05.435 INFO:tasks.rados.rados.0.smithi125.stdout:3071: copy_from oid 3 from oid 28 current snap is 352 2021-03-24T22:06:05.436 INFO:tasks.rados.rados.0.smithi125.stdout:3072: write (excl) oid 18 current snap is 352 2021-03-24T22:06:05.436 INFO:tasks.rados.rados.0.smithi125.stdout:3072: seq_num 1060 ranges {615420=737007,1912207=425038,3026137=542216} 2021-03-24T22:06:05.436 INFO:tasks.rados.rados.0.smithi125.stdout:3072: writing smithi12556470-18 from 615420 to 1352427 tid 1 2021-03-24T22:06:05.441 INFO:tasks.rados.rados.0.smithi125.stdout:3072: writing smithi12556470-18 from 1912207 to 2337245 tid 2 2021-03-24T22:06:05.450 INFO:tasks.rados.rados.0.smithi125.stdout:3072: writing smithi12556470-18 from 3026137 to 3568353 tid 3 2021-03-24T22:06:05.450 INFO:tasks.rados.rados.0.smithi125.stdout:3073: read oid 39 snap -1 2021-03-24T22:06:05.451 INFO:tasks.rados.rados.0.smithi125.stdout:3073: expect deleted 2021-03-24T22:06:05.451 INFO:tasks.rados.rados.0.smithi125.stdout:3074: copy_from oid 48 from oid 33 current snap is 352 2021-03-24T22:06:05.451 INFO:tasks.rados.rados.0.smithi125.stdout:3072: finishing write tid 1 to smithi12556470-18 2021-03-24T22:06:05.451 INFO:tasks.rados.rados.0.smithi125.stdout:3075: write oid 41 current snap is 352 2021-03-24T22:06:05.452 INFO:tasks.rados.rados.0.smithi125.stdout:3075: seq_num 1061 ranges {632227=412256,1570689=568696,2653560=753488} 2021-03-24T22:06:05.456 INFO:tasks.rados.rados.0.smithi125.stdout:3075: writing smithi12556470-41 from 632227 to 1044483 tid 1 2021-03-24T22:06:05.463 INFO:tasks.rados.rados.0.smithi125.stdout:3075: writing smithi12556470-41 from 1570689 to 2139385 tid 2 2021-03-24T22:06:05.472 INFO:tasks.rados.rados.0.smithi125.stdout:3075: writing smithi12556470-41 from 2653560 to 3407048 tid 3 2021-03-24T22:06:05.472 INFO:tasks.rados.rados.0.smithi125.stdout:3076: snap_remove snap 299 2021-03-24T22:06:05.532 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:05.541 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[19936]: cluster 2021-03-24T22:06:05.542 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[19936]: 2021-03-24T22:06:03.906109+0000 osd.3 (osd 2021-03-24T22:06:05.542 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[19936]: .3) 157 : cluster [DBG] 6.c scrub ok 2021-03-24T22:06:05.542 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[19936]: cluster 2021-03-24T22:06 2021-03-24T22:06:05.542 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[19936]: :04.178039+0000 osd.7 (osd.7) 132 : cluster [DBG] 3.f scrub ok 2021-03-24T22:06:05.543 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[19936]: audit 2021-03- 2021-03-24T22:06:05.543 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[19936]: 24T22:06:04.843436+0000 mon.a (mon.0) 2002 : audit 2021-03-24T22:06:05.543 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[19936]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T22:06:05.543 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[19936]: audit 2021-03-24T22:06:05.544 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[19936]: 2021-03-24T22:06:04.925429+0000 mon.c (mon.2) 2021-03-24T22:06:05.544 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[19936]: 116 : audit [DBG] from='client.? v1:172.21.15.125:0/2871836947' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:06:05.544 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[19936]: audit 2021-03-24T22:06: 2021-03-24T22:06:05.544 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[19936]: 05.338970+0000 mon.a (mon.0) 2021-03-24T22:06:05.544 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[19936]: 2003 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 2]}]: dispatch 2021-03-24T22:06:05.545 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[19936]: audit 2021-03-24T22:06:05.339501+0000 2021-03-24T22:06:05.545 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[19936]: mon.a (mon.0) 2004 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2021-03-24T22:06:05.545 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[19936]: audit 2021 2021-03-24T22:06:05.545 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[19936]: -03-24T22:06:05. 2021-03-24T22:06:05.546 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[19936]: 420155+0000 mon.a (mon.0) 2005 2021-03-24T22:06:05.546 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[19936]: : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 2]}]': finished 2021-03-24T22:06:05.546 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[19936]: audit 2021-03-24T22:06 2021-03-24T22:06:05.546 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[19936]: :05.420304+0000 mon.a (mon.0) 2006 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2021-03-24T22:06:05.546 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[19936]: 2021-03-24T22:06:05.547 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:06:05.547 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[19936]: -03-24T22:06:05.420412+0000 2021-03-24T22:06:05.547 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[19936]: mon.a (mon.0) 2007 : cluster [DBG] 2021-03-24T22:06:05.547 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[19936]: osdmap e795: 8 total, 8 up, 8 in 2021-03-24T22:06:05.587 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:05.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:05.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:05.694 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:05.752 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:05.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:05.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:06:05.859 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:05.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:05 smithi142 conmon[20125]: cluster 2021-03-24T22:06:03 2021-03-24T22:06:05.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:05 smithi142 conmon[20125]: .906109+0000 osd.3 (osd.3) 157 : cluster [DBG] 6.c scrub ok 2021-03-24T22:06:05.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:05 smithi142 conmon[20125]: cluster 2021-03-24T22:06:04.178039 2021-03-24T22:06:05.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:05 smithi142 conmon[20125]: +0000 osd.7 (osd.7) 132 : cluster [DBG] 3.f scrub ok 2021-03-24T22:06:05.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:05 smithi142 conmon[20125]: audit 2021-03-24T22:06:04.843436+0000 2021-03-24T22:06:05.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:05 smithi142 conmon[20125]: mon.a (mon.0) 2002 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T22:06:05.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:05 smithi142 conmon[20125]: audit 2021-03-24T22: 2021-03-24T22:06:05.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:05 smithi142 conmon[20125]: 06:04.925429+0000 mon.c (mon.2) 116 : audit 2021-03-24T22:06:05.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:05 smithi142 conmon[20125]: [DBG] from='client.? v1:172.21.15.125:0/2871836947' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:06:05.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:05 smithi142 conmon[20125]: audit 2021-03-24T22:06:05.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:05 smithi142 conmon[20125]: 2021-03-24T22:06:05.338970+0000 mon.a (mon.0) 2003 : audit 2021-03-24T22:06:05.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:05 smithi142 conmon[20125]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 2]}]: dispatch 2021-03-24T22:06:05.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:05 smithi142 conmon[20125]: audit 2021-03-24T22:06:05.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:05 smithi142 conmon[20125]: 2021-03-24T22:06:05.339501+0000 mon.a (mon.0) 2004 : audit 2021-03-24T22:06:05.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:05 smithi142 conmon[20125]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2021-03-24T22:06:05.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:05 smithi142 conmon[20125]: audit 2021-03-24T22:06:05.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:05 smithi142 conmon[20125]: 2021-03-24T22:06:05.420155+0000 mon.a (mon.0) 2005 : audit 2021-03-24T22:06:05.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:05 smithi142 conmon[20125]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 2]}]': finished 2021-03-24T22:06:05.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:05 smithi142 conmon[20125]: audit 2021-03-24T22:06:05.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:05 smithi142 conmon[20125]: 2021-03-24T22:06:05.420304+0000 mon.a (mon.0) 2006 : audit [INF] 2021-03-24T22:06:05.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:05 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2021-03-24T22:06:05.872 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:05 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:06:05.872 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:05 smithi142 conmon[20125]: 24T22:06:05.420412+0000 mon.a (mon.0) 2007 : cluster [DBG] 2021-03-24T22:06:05.872 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:05 smithi142 conmon[20125]: osdmap e795: 8 total, 8 up, 8 in 2021-03-24T22:06:05.873 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:06:05.873 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[24927]: 24T22:06:03.906109+0000 osd.3 (osd 2021-03-24T22:06:05.874 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[24927]: .3) 157 : cluster 2021-03-24T22:06:05.874 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[24927]: [DBG] 6.c scrub ok 2021-03-24T22:06:05.874 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[24927]: cluster 2021-03-24T22:06:05.874 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[24927]: 2021-03-24T 2021-03-24T22:06:05.875 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[24927]: 22:06:04 2021-03-24T22:06:05.875 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[24927]: .178039+0000 osd.7 (osd 2021-03-24T22:06:05.875 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[24927]: .7) 132 : 2021-03-24T22:06:05.875 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[24927]: cluster [DBG] 3.f scrub ok 2021-03-24T22:06:05.875 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[24927]: 2021-03-24T22:06:05.876 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[24927]: audit 2021-03-24T22:06:05.876 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[24927]: 2021-03- 2021-03-24T22:06:05.876 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[24927]: 24T22:06:04.843436+0000 mon.a (mon. 2021-03-24T22:06:05.876 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[24927]: 0) 2002 : audit 2021-03-24T22:06:05.877 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[24927]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T22:06:05.877 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[24927]: audit 2021-03-24T22:06:05.877 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[24927]: 2021-03-24 2021-03-24T22:06:05.877 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[24927]: T22:06:04 2021-03-24T22:06:05.877 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[24927]: .925429+0000 mon.c (mon. 2021-03-24T22:06:05.878 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[24927]: 2) 116 : audit 2021-03-24T22:06:05.878 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[24927]: [DBG] from='client.? v1:172.21.15.125:0/2871836947' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:06:05.878 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[24927]: audit 2021-03-24T22:06:05.878 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[24927]: 2021-03-24T22 2021-03-24T22:06:05.879 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[24927]: :06:05.338970+0000 2021-03-24T22:06:05.879 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[24927]: mon.a (mon.0) 2003 : 2021-03-24T22:06:05.879 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[24927]: audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 2]}]: dispatch 2021-03-24T22:06:05.879 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[24927]: audit 2021-03-24T22:06:05.879 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[24927]: 2021-03-24 2021-03-24T22:06:05.880 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[24927]: T22:06:05. 2021-03-24T22:06:05.880 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[24927]: 339501+0000 mon.a (mon.0) 2021-03-24T22:06:05.880 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[24927]: 2004 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2021-03-24T22:06:05.881 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[24927]: audit 2021-03-24T22:06:05.881 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[24927]: 2021-03-24T 2021-03-24T22:06:05.881 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[24927]: 22:06:05 2021-03-24T22:06:05.881 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[24927]: .420155+0000 mon.a (mon.0 2021-03-24T22:06:05.881 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[24927]: ) 2005 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 2]}]': finished 2021-03-24T22:06:05.882 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[24927]: audit 2021-03-24T22:06:05.882 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[24927]: 2021-03 2021-03-24T22:06:05.882 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[24927]: -24T22:06:05.420304+0000 2021-03-24T22:06:05.882 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[24927]: mon.a (mon. 2021-03-24T22:06:05.883 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[24927]: 0) 2006 : audit [INF] 2021-03-24T22:06:05.883 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[24927]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2021-03-24T22:06:05.883 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[24927]: cluster 2021-03-24T22:06:05.883 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[24927]: 2021-03-24T 2021-03-24T22:06:05.883 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[24927]: 22:06:05. 2021-03-24T22:06:05.884 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[24927]: 420412+0000 mon.a (mon.0) 2007 : cluster [DBG] 2021-03-24T22:06:05.884 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:05 smithi125 conmon[24927]: osdmap e795: 8 total, 8 up, 8 in 2021-03-24T22:06:05.914 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:05.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:05.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:06:06.021 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:06.082 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:06.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:06.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:06:06.190 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:06.246 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:06.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:06.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:06:06.353 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:06.408 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:06.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:06.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:06:06.443 INFO:tasks.rados.rados.0.smithi125.stdout:3076: done (5 left) 2021-03-24T22:06:06.444 INFO:tasks.rados.rados.0.smithi125.stdout:3077: snap_create 2021-03-24T22:06:06.444 INFO:tasks.rados.rados.0.smithi125.stdout:3072: finishing write tid 2 to smithi12556470-18 2021-03-24T22:06:06.444 INFO:tasks.rados.rados.0.smithi125.stdout:3071: finishing copy_from to smithi12556470-3 2021-03-24T22:06:06.445 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 3 v 1208 (ObjNum 961 snap 317 seq_num 961) dirty exists 2021-03-24T22:06:06.445 INFO:tasks.rados.rados.0.smithi125.stdout:3071: finishing copy_from racing read to smithi12556470-3 2021-03-24T22:06:06.445 INFO:tasks.rados.rados.0.smithi125.stdout:3072: finishing write tid 3 to smithi12556470-18 2021-03-24T22:06:06.445 INFO:tasks.rados.rados.0.smithi125.stdout:3072: finishing write tid 4 to smithi12556470-18 2021-03-24T22:06:06.445 INFO:tasks.rados.rados.0.smithi125.stdout:3072: finishing write tid 5 to smithi12556470-18 2021-03-24T22:06:06.446 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 18 v 709 (ObjNum 1060 snap 352 seq_num 1060) dirty exists 2021-03-24T22:06:06.446 INFO:tasks.rados.rados.0.smithi125.stdout:3072: left oid 18 (ObjNum 1060 snap 352 seq_num 1060) 2021-03-24T22:06:06.446 INFO:tasks.rados.rados.0.smithi125.stdout:3075: finishing write tid 1 to smithi12556470-41 2021-03-24T22:06:06.446 INFO:tasks.rados.rados.0.smithi125.stdout:3075: finishing write tid 2 to smithi12556470-41 2021-03-24T22:06:06.448 INFO:tasks.rados.rados.0.smithi125.stdout:3075: finishing write tid 3 to smithi12556470-41 2021-03-24T22:06:06.449 INFO:tasks.rados.rados.0.smithi125.stdout:3075: finishing write tid 4 to smithi12556470-41 2021-03-24T22:06:06.449 INFO:tasks.rados.rados.0.smithi125.stdout:3075: finishing write tid 5 to smithi12556470-41 2021-03-24T22:06:06.449 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 41 v 1205 (ObjNum 1061 snap 352 seq_num 1061) dirty exists 2021-03-24T22:06:06.449 INFO:tasks.rados.rados.0.smithi125.stdout:3075: left oid 41 (ObjNum 1061 snap 352 seq_num 1061) 2021-03-24T22:06:06.450 INFO:tasks.rados.rados.0.smithi125.stdout:3074: finishing copy_from to smithi12556470-48 2021-03-24T22:06:06.450 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 48 v 1133 (ObjNum 916 snap 304 seq_num 916) dirty exists 2021-03-24T22:06:06.450 INFO:tasks.rados.rados.0.smithi125.stdout:3074: finishing copy_from racing read to smithi12556470-48 2021-03-24T22:06:06.515 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:06.571 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:06 smithi125 conmon[19936]: cluster 2021-03-24T22:06:04 2021-03-24T22:06:06.571 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:06 smithi125 conmon[19936]: .996083+0000 mgr.y (mgr.14144) 672 : cluster [DBG] pgmap v1354: 168 pgs: 168 active+clean; 496 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 3.2 MiB/s wr, 22 op/s 2021-03-24T22:06:06.572 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:06 smithi125 conmon[19936]: cluster 2021-03-24T22:06:06.572 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:06 smithi125 conmon[19936]: 2021-03-24T22:06:05.470004+0000 osd.0 (osd.0) 157 2021-03-24T22:06:06.572 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:06 smithi125 conmon[19936]: : cluster [DBG] 7.9 deep-scrub ok 2021-03-24T22:06:06.572 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:06 smithi125 conmon[19936]: cluster 2021-03-24T22:06:06.573 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:06 smithi125 conmon[19936]: 2021-03-24T22: 2021-03-24T22:06:06.573 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:06 smithi125 conmon[19936]: 06:06.437296+0000 mon.a (mon 2021-03-24T22:06:06.573 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:06 smithi125 conmon[19936]: .0) 2008 : cluster [DBG] osdmap e796: 8 total, 8 up, 8 in 2021-03-24T22:06:06.573 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:06 smithi125 conmon[24927]: cluster 2021-03-24T22:06:04.996083+0000 mgr.y (mgr.14144) 672 : cluster 2021-03-24T22:06:06.574 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:06 smithi125 conmon[24927]: [DBG] pgmap v1354: 168 pgs: 168 active+clean; 496 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 3.2 MiB/s wr, 22 op/s 2021-03-24T22:06:06.574 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:06 smithi125 conmon[24927]: cluster 2021-03-24T22:06:05.470004+0000 osd.0 (osd.0) 157 : cluster [DBG] 7.9 deep-scrub ok 2021-03-24T22:06:06.574 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:06 smithi125 conmon[24927]: cluster 2021-03-24T22:06:06.574 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:06 smithi125 conmon[24927]: 2021-03-24T22:06:06.437296+0000 mon.a (mon.0) 2008 : cluster [DBG] osdmap e796: 8 total, 8 up, 8 in 2021-03-24T22:06:06.575 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:06.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:06.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:06.681 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:06.742 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:06.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:06.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:06:06.848 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:06.862 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:06 smithi142 conmon[20125]: cluster 2021-03-24T22:06:04.996083+0000 mgr.y (mgr.14144 2021-03-24T22:06:06.863 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:06 smithi142 conmon[20125]: ) 672 : cluster [DBG] pgmap v1354: 168 pgs: 168 active+clean; 496 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 3.2 MiB/s wr, 22 op/s 2021-03-24T22:06:06.863 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:06 smithi142 conmon[20125]: cluster 2021-03-24T22:06 2021-03-24T22:06:06.863 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:06 smithi142 conmon[20125]: :05.470004+0000 osd.0 (osd.0) 157 : cluster 2021-03-24T22:06:06.863 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:06 smithi142 conmon[20125]: [DBG] 7.9 deep-scrub ok 2021-03-24T22:06:06.864 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:06 smithi142 conmon[20125]: cluster 2021-03-24T22:06:06.864 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:06 smithi142 conmon[20125]: 2021-03-24T22:06:06 2021-03-24T22:06:06.864 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:06 smithi142 conmon[20125]: .437296+0000 mon.a (mon.0) 2008 2021-03-24T22:06:06.864 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:06 smithi142 conmon[20125]: : cluster [DBG] osdmap e796: 8 total, 8 up, 8 in 2021-03-24T22:06:06.904 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:06.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:06.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:06:07.012 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:07.066 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:07.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:07.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:06:07.173 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:07.232 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:07.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:07.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:06:07.339 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:07.397 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:07.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:07.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:06:07.443 INFO:tasks.rados.rados.0.smithi125.stdout:3071: done (5 left) 2021-03-24T22:06:07.444 INFO:tasks.rados.rados.0.smithi125.stdout:3072: done (4 left) 2021-03-24T22:06:07.444 INFO:tasks.rados.rados.0.smithi125.stdout:3073: done (3 left) 2021-03-24T22:06:07.444 INFO:tasks.rados.rados.0.smithi125.stdout:3074: done (2 left) 2021-03-24T22:06:07.444 INFO:tasks.rados.rados.0.smithi125.stdout:3075: done (1 left) 2021-03-24T22:06:07.445 INFO:tasks.rados.rados.0.smithi125.stdout:3077: done (0 left) 2021-03-24T22:06:07.445 INFO:tasks.rados.rados.0.smithi125.stdout:3078: read oid 32 snap 322 2021-03-24T22:06:07.445 INFO:tasks.rados.rados.0.smithi125.stdout:3078: expect deleted 2021-03-24T22:06:07.445 INFO:tasks.rados.rados.0.smithi125.stdout:3079: read oid 27 snap -1 2021-03-24T22:06:07.445 INFO:tasks.rados.rados.0.smithi125.stdout:3079: expect deleted 2021-03-24T22:06:07.446 INFO:tasks.rados.rados.0.smithi125.stdout:3080: read oid 31 snap -1 2021-03-24T22:06:07.446 INFO:tasks.rados.rados.0.smithi125.stdout:3080: expect (ObjNum 1058 snap 351 seq_num 1058) 2021-03-24T22:06:07.446 INFO:tasks.rados.rados.0.smithi125.stdout:3081: delete oid 18 current snap is 353 2021-03-24T22:06:07.450 INFO:tasks.rados.rados.0.smithi125.stdout:3078: done (3 left) 2021-03-24T22:06:07.450 INFO:tasks.rados.rados.0.smithi125.stdout:3081: done (2 left) 2021-03-24T22:06:07.450 INFO:tasks.rados.rados.0.smithi125.stdout:3082: snap_remove snap 322 2021-03-24T22:06:07.503 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:07.565 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:07.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:07.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:06:07.672 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:07.734 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:07.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:07.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:06:07.841 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:07.899 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:07.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:07.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:06:08.009 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:08.071 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:08.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:08.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:06:08.177 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:08.231 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:08.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:08.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:06:08.338 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:08.403 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:08.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:08.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:06:08.439 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:08 smithi125 conmon[24927]: cluster 2021-03-24T22:06:06 2021-03-24T22:06:08.440 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:08 smithi125 conmon[24927]: .879623+0000 osd.3 ( 2021-03-24T22:06:08.440 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:08 smithi125 conmon[24927]: osd.3) 158 : cluster [DBG] 3.7 scrub ok 2021-03-24T22:06:08.440 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:08 smithi125 conmon[24927]: cluster 2021-03-24T22:06:06.996945+0000 mgr.y (mgr.14144) 673 : cluster [DBG] pgmap v1357: 168 pgs: 168 active+clean; 498 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 4.2 MiB/s wr, 23 op/s 2021-03-24T22:06:08.440 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:08 smithi125 conmon[24927]: cluster 2021-03-24T22:06:07.382706+0000 osd.5 (osd.5) 20 : cluster [DBG] 4.8 scrub ok 2021-03-24T22:06:08.441 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:08 smithi125 conmon[24927]: cluster 2021-03-24T22:06:07.437114+0000 mon.a (mon.0) 2009 : cluster [DBG] 2021-03-24T22:06:08.441 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:08 smithi125 conmon[19936]: cluster 2021-03-24T22:06:06.879623+0000 osd.3 (osd.3) 158 : cluster [DBG] 3.7 scrub ok 2021-03-24T22:06:08.441 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:08 smithi125 conmon[19936]: cluster 2021-03-24T22:06:08.441 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:08 smithi125 conmon[19936]: 2021-03-24T22:06:06.996945+0000 mgr.y (mgr.14144) 673 : cluster [DBG] pgmap v1357: 168 pgs: 168 active+clean; 498 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 4.2 MiB/s wr, 23 op/s 2021-03-24T22:06:08.442 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:08 smithi125 conmon[19936]: cluster 2021-03-24T22:06:07.382706+0000 osd.5 (osd.5) 20 : cluster 2021-03-24T22:06:08.442 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:08 smithi125 conmon[19936]: [DBG] 4.8 scrub ok 2021-03-24T22:06:08.442 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:08 smithi125 conmon[19936]: cluster 2021-03-24T22:06: 2021-03-24T22:06:08.442 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:08 smithi125 conmon[19936]: 07.437114+0000 mon.a (mon.0) 2009 : cluster 2021-03-24T22:06:08.443 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:08 smithi125 conmon[19936]: [DBG] osdmap e797: 8 total, 8 up, 8 in 2021-03-24T22:06:08.443 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:08 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:06:08.443 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:08 smithi125 conmon[19936]: 03-24T22:06:07.439833+0000 osd.0 ( 2021-03-24T22:06:08.443 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:08 smithi125 conmon[19936]: osd.0) 158 : cluster [DBG] 4.7 starting backfill to osd.2 from (0'0,0'0] MAX to 544'32 2021-03-24T22:06:08.444 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:08 smithi125 conmon[19936]: audit 2021-03-24T22:06:07.763274+0000 mon.a 2021-03-24T22:06:08.444 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:08 smithi125 conmon[19936]: (mon.0) 2010 : audit [INF] 2021-03-24T22:06:08.444 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:08 smithi125 conmon[19936]: from='client.? v1:172.21.15.125:0/4270879672' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T22:06:08.454 INFO:tasks.rados.rados.0.smithi125.stdout:3082: done (2 left) 2021-03-24T22:06:08.454 INFO:tasks.rados.rados.0.smithi125.stdout:3083: read oid 24 snap -1 2021-03-24T22:06:08.455 INFO:tasks.rados.rados.0.smithi125.stdout:3083: expect (ObjNum 1048 snap 347 seq_num 1048) 2021-03-24T22:06:08.455 INFO:tasks.rados.rados.0.smithi125.stdout:3084: write (excl) oid 47 current snap is 353 2021-03-24T22:06:08.455 INFO:tasks.rados.rados.0.smithi125.stdout:3084: seq_num 1063 ranges {665841=562754,1687653=572365,2708406=776032} 2021-03-24T22:06:08.459 INFO:teuthology.orchestra.run.smithi125.stderr:noscrub is set 2021-03-24T22:06:08.459 INFO:tasks.rados.rados.0.smithi125.stdout:3084: writing smithi12556470-47 from 665841 to 1228595 tid 1 2021-03-24T22:06:08.468 INFO:tasks.rados.rados.0.smithi125.stdout:3084: writing smithi12556470-47 from 1687653 to 2260018 tid 2 2021-03-24T22:06:08.476 INFO:tasks.rados.rados.0.smithi125.stdout:3084: writing smithi12556470-47 from 2708406 to 3484438 tid 3 2021-03-24T22:06:08.483 INFO:tasks.rados.rados.0.smithi125.stdout:3080: expect (ObjNum 1058 snap 351 seq_num 1058) 2021-03-24T22:06:08.510 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:08.573 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:08.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:08.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:08.679 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:08.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:08 smithi142 conmon[20125]: cluster 2021-03-24T22:06:06. 2021-03-24T22:06:08.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:08 smithi142 conmon[20125]: 879623+0000 osd.3 (osd.3) 158 : cluster [DBG] 3.7 scrub ok 2021-03-24T22:06:08.694 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:08 smithi142 conmon[20125]: cluster 2021-03-24T22:06:06.996945+0000 mgr.y ( 2021-03-24T22:06:08.695 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:08 smithi142 conmon[20125]: mgr.14144) 673 : cluster [DBG] pgmap v1357: 168 pgs: 168 active+clean; 498 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 4.2 MiB/s wr, 23 op/s 2021-03-24T22:06:08.695 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:08 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:06:08.696 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:08 smithi142 conmon[20125]: -24T22:06:07.382706+0000 osd.5 (osd.5 2021-03-24T22:06:08.697 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:08 smithi142 conmon[20125]: ) 20 : cluster [DBG] 4.8 scrub ok 2021-03-24T22:06:08.697 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:08 smithi142 conmon[20125]: cluster 2021-03-24T22:06:07.437114 2021-03-24T22:06:08.697 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:08 smithi142 conmon[20125]: +0000 mon.a (mon.0) 2009 : cluster [DBG] osdmap e797: 8 total, 8 up, 8 in 2021-03-24T22:06:08.698 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:08 smithi142 conmon[20125]: cluster 2021-03-24T22:06:07.439833+0000 osd.0 (osd.0 2021-03-24T22:06:08.698 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:08 smithi142 conmon[20125]: ) 158 : cluster [DBG] 4.7 starting backfill to osd.2 from (0'0,0'0] MAX to 544'32 2021-03-24T22:06:08.698 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:08 smithi142 conmon[20125]: audit 2021-03-24T22:06:08.698 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:08 smithi142 conmon[20125]: 2021-03-24T22:06:07.763274+0000 mon.a (mon.0) 2010 : audit [INF] from='client.? v1:172.21.15.125:0/4270879672' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T22:06:08.704 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:08 smithi125 conmon[24927]: osdmap e797: 8 total, 8 up, 8 in 2021-03-24T22:06:08.705 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:08 smithi125 conmon[24927]: cluster 2021-03-24T22:06:07.439833+0000 2021-03-24T22:06:08.705 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:08 smithi125 conmon[24927]: osd.0 (osd.0) 158 : cluster [DBG] 4.7 starting backfill to osd.2 from (0'0,0'0] MAX to 544'32 2021-03-24T22:06:08.705 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:08 smithi125 conmon[24927]: audit 2021-03-24T22:06:08.705 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:08 smithi125 conmon[24927]: 2021-03-24T22:06:07.763274+0000 2021-03-24T22:06:08.705 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:08 smithi125 conmon[24927]: mon.a (mon.0) 2010 : audit [INF] from='client.? v1:172.21.15.125:0/4270879672' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T22:06:08.743 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:08.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:08.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:06:08.848 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:08.852 INFO:tasks.rados.rados.0.smithi125.stdout:3083: expect (ObjNum 1048 snap 347 seq_num 1048) 2021-03-24T22:06:08.902 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:08.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:08.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:06:09.009 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:09.059 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:09.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:09.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:06:09.082 INFO:tasks.rados.rados.0.smithi125.stdout:3084: finishing write tid 1 to smithi12556470-47 2021-03-24T22:06:09.083 INFO:tasks.rados.rados.0.smithi125.stdout:3084: finishing write tid 2 to smithi12556470-47 2021-03-24T22:06:09.083 INFO:tasks.rados.rados.0.smithi125.stdout:3084: finishing write tid 3 to smithi12556470-47 2021-03-24T22:06:09.083 INFO:tasks.rados.rados.0.smithi125.stdout:3084: finishing write tid 4 to smithi12556470-47 2021-03-24T22:06:09.084 INFO:tasks.rados.rados.0.smithi125.stdout:3084: finishing write tid 5 to smithi12556470-47 2021-03-24T22:06:09.084 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 47 v 395 (ObjNum 1063 snap 353 seq_num 1063) dirty exists 2021-03-24T22:06:09.084 INFO:tasks.rados.rados.0.smithi125.stdout:3084: left oid 47 (ObjNum 1063 snap 353 seq_num 1063) 2021-03-24T22:06:09.084 INFO:tasks.rados.rados.0.smithi125.stdout:3079: done (3 left) 2021-03-24T22:06:09.084 INFO:tasks.rados.rados.0.smithi125.stdout:3080: done (2 left) 2021-03-24T22:06:09.085 INFO:tasks.rados.rados.0.smithi125.stdout:3083: done (1 left) 2021-03-24T22:06:09.085 INFO:tasks.rados.rados.0.smithi125.stdout:3084: done (0 left) 2021-03-24T22:06:09.085 INFO:tasks.rados.rados.0.smithi125.stdout:3085: snap_remove snap 293 2021-03-24T22:06:09.165 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:09.221 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:09.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:09.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:09.329 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:09.389 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:09.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:09.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:09.461 INFO:tasks.rados.rados.0.smithi125.stdout:3085: done (0 left) 2021-03-24T22:06:09.461 INFO:tasks.rados.rados.0.smithi125.stdout:3086: copy_from oid 20 from oid 1 current snap is 353 2021-03-24T22:06:09.461 INFO:tasks.rados.rados.0.smithi125.stdout:3087: snap_remove snap 305 2021-03-24T22:06:09.494 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:09.537 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:09.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:09.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:06:09.644 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:09.700 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:09.701 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-24T22:06:09.703 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 24 22:06:09 smithi142 conmon[33921]: debug 2021-03-24T22:06:09.703 INFO:journalctl@ceph.osd.7.smithi142.stdout:Mar 24 22:06:09 smithi142 conmon[33921]: 2021-03-24T22:06:09.477+0000 7fc7b2416700 -1 osd.7 pg_epoch: 799 pg[7.17( v 799'1316 (0'0,799'1316] local-lis/les=733/734 n=19 ec=198/99 lis/c=733/733 les/c/f=734/734/0 sis=733) [7,5,2] r=0 lpr=733 luod=798'1314 crt=799'1316 lcod 798'1312 mlcod 798'1312 active+clean+snaptrim trimq=[231~1] ps=[1ef~1,22c~1,23b~1,247~1,266~1]] removing snap head 2021-03-24T22:06:09.704 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:09 smithi142 conmon[20125]: cluster 2021-03-24T22:06 2021-03-24T22:06:09.704 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:09 smithi142 conmon[20125]: :07.907622+0000 osd.3 (osd.3) 159 : cluster [DBG] 6.1f deep-scrub ok 2021-03-24T22:06:09.705 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:09 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:06:09.705 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:09 smithi142 conmon[20125]: 03-24T22:06:08.071691+0000 osd.7 ( 2021-03-24T22:06:09.705 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:09 smithi142 conmon[20125]: osd.7) 133 : cluster [DBG] 2021-03-24T22:06:09.705 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:09 smithi142 conmon[20125]: 3.14 scrub ok 2021-03-24T22:06:09.706 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:09 smithi142 conmon[20125]: cluster 2021-03-24T22:06: 2021-03-24T22:06:09.706 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:09 smithi142 conmon[20125]: 08.433346+0000 mon.a (mon.0) 2011 : 2021-03-24T22:06:09.706 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:09 smithi142 conmon[20125]: cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:06:09.706 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:09 smithi142 conmon[20125]: audit 2021 2021-03-24T22:06:09.706 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:09 smithi142 conmon[20125]: -03-24T22:06:08. 2021-03-24T22:06:09.707 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:09 smithi142 conmon[20125]: 445884+0000 mon.a (mon.0) 2012 : audit 2021-03-24T22:06:09.707 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:09 smithi142 conmon[20125]: [INF] from='client.? v1:172.21.15.125:0/4270879672' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T22:06:09.707 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:09 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:06:09.707 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:09 smithi142 conmon[20125]: -24T22:06:08.445972 2021-03-24T22:06:09.708 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:09 smithi142 conmon[20125]: +0000 mon.a (mon.0) 2013 : cluster 2021-03-24T22:06:09.708 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:09 smithi142 conmon[20125]: [DBG] osdmap e798: 8 total, 8 up, 8 in 2021-03-24T22:06:09.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:09.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:06:09.712 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:09 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:06:09.712 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:09 smithi125 conmon[24927]: 06:07.907622+0000 osd.3 (osd.3) 159 : cluster [DBG] 6.1f deep-scrub ok 2021-03-24T22:06:09.712 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:09 smithi125 conmon[24927]: cluster 2021-03-24T22:06 2021-03-24T22:06:09.712 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:09 smithi125 conmon[24927]: :08.071691+0000 osd.7 (osd.7) 133 : 2021-03-24T22:06:09.713 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:09 smithi125 conmon[24927]: cluster [DBG] 3.14 scrub ok 2021-03-24T22:06:09.713 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:09 smithi125 conmon[24927]: cluster 2021-03-24T22:06:09.713 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:09 smithi125 conmon[24927]: 2021-03-24T22:06:08.433346+0000 2021-03-24T22:06:09.713 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:09 smithi125 conmon[24927]: mon.a (mon.0) 2011 : cluster [WRN] 2021-03-24T22:06:09.713 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:09 smithi125 conmon[24927]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:06:09.714 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:09 smithi125 conmon[24927]: audit 2021-03 2021-03-24T22:06:09.714 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:09 smithi125 conmon[24927]: -24T22:06:08.445884+0000 mon.a (mon 2021-03-24T22:06:09.714 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:09 smithi125 conmon[24927]: .0) 2012 : audit [INF] from='client.? v1:172.21.15.125:0/4270879672' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T22:06:09.714 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:09 smithi125 conmon[24927]: cluster 2021-03-24T22:06:09.715 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:09 smithi125 conmon[24927]: 2021-03-24T22:06:08.445972+0000 2021-03-24T22:06:09.715 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:09 smithi125 conmon[24927]: mon.a (mon.0) 2013 : cluster [DBG] osdmap e798: 8 total, 8 up, 8 in 2021-03-24T22:06:09.715 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:09 smithi125 conmon[24927]: 2021-03-24T22:06:09.715 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:09 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:06:09.716 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:09 smithi125 conmon[19936]: T22:06:07.907622+0000 osd.3 (osd.3) 159 : cluster [DBG] 6.1f deep-scrub ok 2021-03-24T22:06:09.716 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:09 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:06:09.716 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:09 smithi125 conmon[19936]: 24T22:06:08.071691+0000 osd.7 (osd.7) 133 : cluster 2021-03-24T22:06:09.716 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:09 smithi125 conmon[19936]: [DBG] 3.14 scrub ok 2021-03-24T22:06:09.717 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:09 smithi125 conmon[19936]: cluster 2021-03-24T22:06: 2021-03-24T22:06:09.717 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:09 smithi125 conmon[19936]: 08.433346+0000 mon.a (mon.0) 2011 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:06:09.717 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:09 smithi125 conmon[19936]: 2021-03-24T22:06:09.717 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:09 smithi125 conmon[19936]: audit 2021-03-24T22: 2021-03-24T22:06:09.718 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:09 smithi125 conmon[19936]: 06:08.445884+0000 mon.a (mon.0) 2012 : audit [INF] from='client.? v1:172.21.15.125:0/4270879672' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T22:06:09.718 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:09 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:06:09.718 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:09 smithi125 conmon[19936]: 06:08.445972+0000 mon.a (mon.0 2021-03-24T22:06:09.718 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:09 smithi125 conmon[19936]: ) 2013 : cluster [DBG] osdmap e798: 8 total, 8 up, 8 in 2021-03-24T22:06:09.810 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:09.866 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:09.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:09.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:09.925 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-24T22:06:09.925 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-24T22:06:09.926 INFO:teuthology.orchestra.run.smithi125.stdout: "num_ops": 0 2021-03-24T22:06:09.926 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-24T22:06:09.971 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:10.034 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:10.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:10.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:06:10.141 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:10.200 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:10.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:10.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:06:10.306 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:10.362 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:10.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:10.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:06:10.430 INFO:tasks.rados.rados.0.smithi125.stdout:3087: done (1 left) 2021-03-24T22:06:10.430 INFO:tasks.rados.rados.0.smithi125.stdout:3088: delete oid 30 current snap is 353 2021-03-24T22:06:10.431 INFO:tasks.rados.rados.0.smithi125.stdout:3086: finishing copy_from to smithi12556470-20 2021-03-24T22:06:10.431 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 20 v 1443 (ObjNum 808 snap 265 seq_num 808) dirty exists 2021-03-24T22:06:10.431 INFO:tasks.rados.rados.0.smithi125.stdout:3086: finishing copy_from racing read to smithi12556470-20 2021-03-24T22:06:10.463 INFO:tasks.rados.rados.0.smithi125.stdout:3086: done (1 left) 2021-03-24T22:06:10.463 INFO:tasks.rados.rados.0.smithi125.stdout:3088: done (0 left) 2021-03-24T22:06:10.463 INFO:tasks.rados.rados.0.smithi125.stdout:3089: snap_create 2021-03-24T22:06:10.470 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:10.515 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:10.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:10.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:06:10.620 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:10.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:10 smithi125 conmon[24927]: cluster 2021-03-24T22:06:10.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:10 smithi125 conmon[24927]: 2021-03-24T22:06: 2021-03-24T22:06:10.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:10 smithi125 conmon[24927]: 08.997582+0000 mgr.y (mgr.14144) 674 : cluster [DBG] pgmap v1360: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 507 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 6.8 MiB/s rd, 5.0 MiB/s wr, 14 op/s; 11/1662 objects misplaced (0.662%) 2021-03-24T22:06:10.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:10 smithi125 conmon[24927]: cluster 2021-03-24T22:06:10.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:10 smithi125 conmon[24927]: 2021-03-24T22:06:09.444725+0000 mon.a (mon.0) 2014 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:06:10.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:10 smithi125 conmon[24927]: cluster 2021-03-24T22:06:09.456235 2021-03-24T22:06:10.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:10 smithi125 conmon[24927]: +0000 mon.a (mon.0) 2015 : cluster [DBG] osdmap e799: 8 total, 8 up, 8 in 2021-03-24T22:06:10.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:10 smithi125 conmon[24927]: 2021-03-24T22:06:10.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:10 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:06:10.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:10 smithi125 conmon[24927]: 03-24T22:06:10.425633+0000 2021-03-24T22:06:10.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:10 smithi125 conmon[24927]: mon.a (mon.0) 2016 : cluster [DBG] 2021-03-24T22:06:10.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:10 smithi125 conmon[24927]: osdmap e800: 8 total, 8 up, 8 in 2021-03-24T22:06:10.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:10 smithi125 conmon[19936]: cluster 2021-03-24T22:06:08.997582+0000 mgr.y ( 2021-03-24T22:06:10.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:10 smithi125 conmon[19936]: mgr.14144) 674 : cluster [DBG] pgmap v1360: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 507 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 6.8 MiB/s rd, 5.0 MiB/s wr, 14 op/s; 11/1662 objects misplaced (0.662%) 2021-03-24T22:06:10.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:10 smithi125 conmon[19936]: cluster 2021-03-24T22:06:09.444725+0000 mon.a (mon.0) 2014 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:06:10.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:10 smithi125 conmon[19936]: cluster 2021-03-24T22:06:09.456235+0000 mon.a (mon. 2021-03-24T22:06:10.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:10 smithi125 conmon[19936]: 0) 2015 : cluster [DBG] osdmap e799: 8 total, 8 up, 8 in 2021-03-24T22:06:10.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:10 smithi125 conmon[19936]: cluster 2021-03-24T22:06 2021-03-24T22:06:10.667 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:10 smithi125 conmon[19936]: :10.425633+0000 mon.a (mon.0) 2016 : cluster [DBG] osdmap e800: 8 total, 8 up, 8 in 2021-03-24T22:06:10.677 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:10.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:10.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:10.782 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:10.796 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:10 smithi142 conmon[20125]: cluster 2021-03-24T22:06:10.796 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:10 smithi142 conmon[20125]: 2021-03-24T22:06:10.797 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:10 smithi142 conmon[20125]: 2021 2021-03-24T22:06:10.797 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:10 smithi142 conmon[20125]: - 2021-03-24T22:06:10.797 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:10 smithi142 conmon[20125]: 03 2021-03-24T22:06:10.798 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:10 smithi142 conmon[20125]: - 2021-03-24T22:06:10.798 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:10 smithi142 conmon[20125]: 24 2021-03-24T22:06:10.799 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:10 smithi142 conmon[20125]: T 2021-03-24T22:06:10.799 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:10 smithi142 conmon[20125]: 22 2021-03-24T22:06:10.799 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:10 smithi142 conmon[20125]: : 2021-03-24T22:06:10.799 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:10 smithi142 conmon[20125]: 06 2021-03-24T22:06:10.799 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:10 smithi142 conmon[20125]: : 2021-03-24T22:06:10.800 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:10 smithi142 conmon[20125]: 08 2021-03-24T22:06:10.800 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:10 smithi142 conmon[20125]: . 2021-03-24T22:06:10.800 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:10 smithi142 conmon[20125]: 997582 2021-03-24T22:06:10.800 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:10 smithi142 conmon[20125]: +0000 2021-03-24T22:06:10.801 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:10 smithi142 conmon[20125]: 2021-03-24T22:06:10.801 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:10 smithi142 conmon[20125]: mgr.y 2021-03-24T22:06:10.801 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:10 smithi142 conmon[20125]: ( 2021-03-24T22:06:10.801 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:10 smithi142 conmon[20125]: mgr 2021-03-24T22:06:10.801 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:10 smithi142 conmon[20125]: . 2021-03-24T22:06:10.802 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:10 smithi142 conmon[20125]: 14144 2021-03-24T22:06:10.802 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:10 smithi142 conmon[20125]: ) 674 : cluster [DBG] pgmap v1360: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 507 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 6.8 MiB/s rd, 5.0 MiB/s wr, 14 op/s; 11/1662 objects misplaced (0.662%) 2021-03-24T22:06:10.802 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:10 smithi142 conmon[20125]: cluster 2021-03-24T22 2021-03-24T22:06:10.802 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:10 smithi142 conmon[20125]: :06:09.444725+0000 mon.a (mon.0) 2014 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2021-03-24T22:06:10.803 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:10 smithi142 conmon[20125]: 2021-03-24T22:06:10.803 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:10 smithi142 conmon[20125]: cluster 2021-03-24T22:06:09 2021-03-24T22:06:10.803 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:10 smithi142 conmon[20125]: .456235+0000 mon.a (mon.0) 2015 : cluster [DBG] 2021-03-24T22:06:10.803 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:10 smithi142 conmon[20125]: osdmap e799: 8 total, 8 up, 8 in 2021-03-24T22:06:10.804 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:10 smithi142 conmon[20125]: cluster 2021-03-24T22:06:10 2021-03-24T22:06:10.804 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:10 smithi142 conmon[20125]: .425633+0000 mon.a (mon.0) 2016 : cluster [DBG] 2021-03-24T22:06:10.804 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:10 smithi142 conmon[20125]: osdmap e800: 8 total, 8 up, 8 in 2021-03-24T22:06:10.844 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:10.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:10.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:06:10.951 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:11.011 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:11.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:11.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:06:11.118 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:11.176 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:11.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:11.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:06:11.281 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:11.338 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:11.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:11.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:06:11.444 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:11.464 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:06:11.473 INFO:tasks.rados.rados.0.smithi125.stdout:3089: done (0 left) 2021-03-24T22:06:11.473 INFO:tasks.rados.rados.0.smithi125.stdout:3090: read oid 42 snap -1 2021-03-24T22:06:11.474 INFO:tasks.rados.rados.0.smithi125.stdout:3090: expect (ObjNum 736 snap 235 seq_num 736) 2021-03-24T22:06:11.474 INFO:tasks.rados.rados.0.smithi125.stdout:3091: read oid 10 snap -1 2021-03-24T22:06:11.475 INFO:tasks.rados.rados.0.smithi125.stdout:3091: expect (ObjNum 993 snap 329 seq_num 993) 2021-03-24T22:06:11.475 INFO:tasks.rados.rados.0.smithi125.stdout:3092: write oid 19 current snap is 354 2021-03-24T22:06:11.476 INFO:tasks.rados.rados.0.smithi125.stdout:3092: seq_num 1065 ranges {699455=429605,1804617=576034,3163252=514929} 2021-03-24T22:06:11.479 INFO:tasks.rados.rados.0.smithi125.stdout:3092: writing smithi12556470-19 from 699455 to 1129060 tid 1 2021-03-24T22:06:11.481 INFO:teuthology.orchestra.run.smithi125.stderr:reweighted osd.7 to 1 (10000) 2021-03-24T22:06:11.489 INFO:tasks.rados.rados.0.smithi125.stdout:3092: writing smithi12556470-19 from 1804617 to 2380651 tid 2 2021-03-24T22:06:11.496 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.2 dump_blocked_ops 2021-03-24T22:06:11.497 INFO:tasks.rados.rados.0.smithi125.stdout:3092: writing smithi12556470-19 from 3163252 to 3678181 tid 3 2021-03-24T22:06:11.497 INFO:tasks.rados.rados.0.smithi125.stdout:3093: rollback oid 1 current snap is 354 2021-03-24T22:06:11.498 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 1 to 331 2021-03-24T22:06:11.498 INFO:tasks.rados.rados.0.smithi125.stdout:3090: expect (ObjNum 736 snap 235 seq_num 736) 2021-03-24T22:06:11.505 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:11.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:11.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:06:11.610 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:11.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:11 smithi142 conmon[20125]: audit 2021-03-24T22:06 2021-03-24T22:06:11.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:11 smithi142 conmon[20125]: :10.594093+0000 mon.a (mon.0) 2017 : audit [INF] from='client.? v1:172.21.15.125:0/1657532070' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2021-03-24T22:06:11.623 INFO:tasks.rados.rados.0.smithi125.stdout:3091: expect (ObjNum 993 snap 329 seq_num 993) 2021-03-24T22:06:11.669 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:11.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:11.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:11.775 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:11.788 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:11 smithi125 conmon[19936]: audit 2021-03-24T22:06:11.788 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:11 smithi125 conmon[19936]: 2021-03-24T22:06:10.594093+0000 2021-03-24T22:06:11.789 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:11 smithi125 conmon[19936]: mon.a (mon.0) 2017 : audit [INF] from='client.? v1:172.21.15.125:0/1657532070' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2021-03-24T22:06:11.789 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:11 smithi125 conmon[24927]: audit 2021-03-24T22:06:10.594093+0000 mon.a (mon.0) 2017 : audit [INF] from='client.? v1:172.21.15.125:0/1657532070' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2021-03-24T22:06:11.817 INFO:tasks.rados.rados.0.smithi125.stdout:3092: finishing write tid 1 to smithi12556470-19 2021-03-24T22:06:11.818 INFO:tasks.rados.rados.0.smithi125.stdout:3092: finishing write tid 2 to smithi12556470-19 2021-03-24T22:06:11.818 INFO:tasks.rados.rados.0.smithi125.stdout:3090: done (3 left) 2021-03-24T22:06:11.819 INFO:tasks.rados.rados.0.smithi125.stdout:3091: done (2 left) 2021-03-24T22:06:11.819 INFO:tasks.rados.rados.0.smithi125.stdout:3094: read oid 43 snap -1 2021-03-24T22:06:11.819 INFO:tasks.rados.rados.0.smithi125.stdout:3094: expect (ObjNum 1021 snap 337 seq_num 1021) 2021-03-24T22:06:11.819 INFO:tasks.rados.rados.0.smithi125.stdout:3095: read oid 26 snap -1 2021-03-24T22:06:11.819 INFO:tasks.rados.rados.0.smithi125.stdout:3095: expect (ObjNum 1056 snap 351 seq_num 1056) 2021-03-24T22:06:11.820 INFO:tasks.rados.rados.0.smithi125.stdout:3096: snap_create 2021-03-24T22:06:11.820 INFO:tasks.rados.rados.0.smithi125.stdout:3093: finishing rollback tid 0 to smithi12556470-1 2021-03-24T22:06:11.821 INFO:tasks.rados.rados.0.smithi125.stdout:3092: finishing write tid 3 to smithi12556470-19 2021-03-24T22:06:11.821 INFO:tasks.rados.rados.0.smithi125.stdout:3093: finishing rollback tid 1 to smithi12556470-1 2021-03-24T22:06:11.821 INFO:tasks.rados.rados.0.smithi125.stdout:3092: finishing write tid 4 to smithi12556470-19 2021-03-24T22:06:11.822 INFO:tasks.rados.rados.0.smithi125.stdout:3092: finishing write tid 5 to smithi12556470-19 2021-03-24T22:06:11.822 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 19 v 903 (ObjNum 1065 snap 354 seq_num 1065) dirty exists 2021-03-24T22:06:11.822 INFO:tasks.rados.rados.0.smithi125.stdout:3092: left oid 19 (ObjNum 1065 snap 354 seq_num 1065) 2021-03-24T22:06:11.822 INFO:tasks.rados.rados.0.smithi125.stdout:3093: finishing rollback tid 2 to smithi12556470-1 2021-03-24T22:06:11.823 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 1 v 1446 (ObjNum 764 snap 249 seq_num 764) dirty exists 2021-03-24T22:06:11.823 INFO:tasks.rados.rados.0.smithi125.stdout:3094: expect (ObjNum 1021 snap 337 seq_num 1021) 2021-03-24T22:06:11.826 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:11.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:11.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:06:11.932 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:11.973 INFO:tasks.rados.rados.0.smithi125.stdout:3095: expect (ObjNum 1056 snap 351 seq_num 1056) 2021-03-24T22:06:11.986 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:11.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:11.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:06:12.092 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:12.144 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:12.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:12.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:06:12.217 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-24T22:06:12.250 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:12.311 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:12.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:12.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:12.418 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:12.468 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:12.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:12.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:06:12.479 INFO:tasks.rados.rados.0.smithi125.stdout:3092: done (4 left) 2021-03-24T22:06:12.479 INFO:tasks.rados.rados.0.smithi125.stdout:3093: done (3 left) 2021-03-24T22:06:12.479 INFO:tasks.rados.rados.0.smithi125.stdout:3094: done (2 left) 2021-03-24T22:06:12.479 INFO:tasks.rados.rados.0.smithi125.stdout:3095: done (1 left) 2021-03-24T22:06:12.480 INFO:tasks.rados.rados.0.smithi125.stdout:3096: done (0 left) 2021-03-24T22:06:12.480 INFO:tasks.rados.rados.0.smithi125.stdout:3097: snap_create 2021-03-24T22:06:12.577 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:12.591 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:12 smithi142 conmon[20125]: cluster 2021-03-24T22:06:10.998280+0000 mgr.y (mgr.14144) 675 : cluster 2021-03-24T22:06:12.591 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:12 smithi142 conmon[20125]: [DBG] pgmap v1363: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 503 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 6.5 MiB/s rd, 4.1 MiB/s wr, 13 op/s; 11/1653 objects misplaced (0.665%) 2021-03-24T22:06:12.592 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:12 smithi142 conmon[20125]: audit 2021-03-24T22:06:11.468283+0000 2021-03-24T22:06:12.592 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:12 smithi142 conmon[20125]: mon.a (mon.0) 2018 : audit [INF] from='client.? v1:172.21.15.125:0/1657532070' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2021-03-24T22:06:12.592 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:12 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:06:12.592 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:12 smithi142 conmon[20125]: 24T22:06:11.468352+0000 2021-03-24T22:06:12.593 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:12 smithi142 conmon[20125]: mon.a (mon.0) 2019 : cluster [DBG] 2021-03-24T22:06:12.593 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:12 smithi142 conmon[20125]: osdmap e801: 8 total, 8 up, 8 in 2021-03-24T22:06:12.632 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:12.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:12.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:12.657 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd reweight 1 1 2021-03-24T22:06:12.687 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:06:12.741 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:12.743 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:12 smithi125 conmon[19936]: cluster 2021-03-24T22:06:10.998280+0000 mgr.y (mgr.14144) 675 : cluster [DBG] pgmap v1363: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 503 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 6.5 MiB/s rd, 4.1 MiB/s wr, 13 op/s; 11/1653 objects misplaced (0.665%) 2021-03-24T22:06:12.744 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:12 smithi125 conmon[19936]: audit 2021-03-24T22:06:11.468283+0000 mon.a (mon.0) 2018 : audit [INF] from='client.? v1:172.21.15.125:0/1657532070' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2021-03-24T22:06:12.744 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:12 smithi125 conmon[19936]: cluster 2021-03-24T22:06:12.744 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:12 smithi125 conmon[19936]: 2021-03-24T22:06:11.468352+0000 mon.a (mon.0) 2019 : cluster [DBG] osdmap e801: 8 total, 8 up, 8 in 2021-03-24T22:06:12.744 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:12 smithi125 conmon[24927]: cluster 2021-03-24T22:06:10.998280+0000 mgr.y (mgr.14144) 675 : cluster [DBG] pgmap v1363: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 503 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 6.5 MiB/s rd, 4.1 MiB/s wr, 13 op/s; 11/1653 objects misplaced (0.665%) 2021-03-24T22:06:12.745 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:12 smithi125 conmon[24927]: audit 2021-03-24T22:06:11.468283+0000 mon.a (mon.0) 2018 : audit [INF] from='client.? v1:172.21.15.125:0/1657532070' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2021-03-24T22:06:12.745 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:12 smithi125 conmon[24927]: cluster 2021-03-24T22:06:11.468352+0000 mon.a (mon.0) 2019 : cluster [DBG] osdmap e801: 8 total, 8 up, 8 in 2021-03-24T22:06:12.803 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:12.805 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:06:12.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:12.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:06:12.908 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:12.971 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:12.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:12.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:06:13.078 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:13.141 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:13.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:13.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:06:13.247 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:13.307 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:13.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:13.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:13.413 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:13.466 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:13.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:13.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:06:13.487 INFO:tasks.rados.rados.0.smithi125.stdout:3097: done (0 left) 2021-03-24T22:06:13.487 INFO:tasks.rados.rados.0.smithi125.stdout:3098: write (excl) oid 34 current snap is 356 2021-03-24T22:06:13.487 INFO:tasks.rados.rados.0.smithi125.stdout:3098: seq_num 1066 ranges {716262=504854,1863099=719692,3074322=726201} 2021-03-24T22:06:13.493 INFO:tasks.rados.rados.0.smithi125.stdout:3098: writing smithi12556470-34 from 716262 to 1221116 tid 1 2021-03-24T22:06:13.501 INFO:tasks.rados.rados.0.smithi125.stdout:3098: writing smithi12556470-34 from 1863099 to 2582791 tid 2 2021-03-24T22:06:13.508 INFO:tasks.rados.rados.0.smithi125.stdout:3098: writing smithi12556470-34 from 3074322 to 3800523 tid 3 2021-03-24T22:06:13.509 INFO:tasks.rados.rados.0.smithi125.stdout:3099: snap_create 2021-03-24T22:06:13.509 INFO:tasks.rados.rados.0.smithi125.stdout:3098: finishing write tid 1 to smithi12556470-34 2021-03-24T22:06:13.509 INFO:tasks.rados.rados.0.smithi125.stdout:3098: finishing write tid 2 to smithi12556470-34 2021-03-24T22:06:13.513 INFO:tasks.rados.rados.0.smithi125.stdout:3098: finishing write tid 3 to smithi12556470-34 2021-03-24T22:06:13.513 INFO:tasks.rados.rados.0.smithi125.stdout:3098: finishing write tid 4 to smithi12556470-34 2021-03-24T22:06:13.514 INFO:tasks.rados.rados.0.smithi125.stdout:3098: finishing write tid 5 to smithi12556470-34 2021-03-24T22:06:13.514 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 34 v 1430 (ObjNum 1066 snap 356 seq_num 1066) dirty exists 2021-03-24T22:06:13.514 INFO:tasks.rados.rados.0.smithi125.stdout:3098: left oid 34 (ObjNum 1066 snap 356 seq_num 1066) 2021-03-24T22:06:13.572 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:13.586 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:13 smithi142 conmon[20125]: cluster 2021-03-24T22:06:12. 2021-03-24T22:06:13.586 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:13 smithi142 conmon[20125]: 474605+0000 mon.a (mon.0) 2020 : cluster [DBG] osdmap e802: 8 total, 8 up, 8 in 2021-03-24T22:06:13.627 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:13.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:13.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:06:13.733 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:13.747 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:13 smithi125 conmon[24927]: cluster 2021-03-24T22:06:12. 2021-03-24T22:06:13.747 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:13 smithi125 conmon[24927]: 474605+0000 mon.a (mon.0) 2020 : cluster [DBG] osdmap e802: 8 total, 8 up, 8 in 2021-03-24T22:06:13.748 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:13 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T22:06:13.748 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:13 smithi125 conmon[19936]: -24T22:06:12.474605+0000 mon.a (mon.0) 2021-03-24T22:06:13.748 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:13 smithi125 conmon[19936]: 2020 : cluster [DBG] osdmap e802: 8 total, 8 up, 8 in 2021-03-24T22:06:13.787 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:13.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:13.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:06:13.893 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:13.951 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:13.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:13.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:06:14.014 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:06:14.056 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:14.111 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:14.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:14.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:06:14.219 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:14.275 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:14.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:14.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:06:14.382 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:14.443 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:14.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:14.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:06:14.496 INFO:tasks.rados.rados.0.smithi125.stdout:3098: done (1 left) 2021-03-24T22:06:14.496 INFO:tasks.rados.rados.0.smithi125.stdout:3099: done (0 left) 2021-03-24T22:06:14.497 INFO:tasks.rados.rados.0.smithi125.stdout:3100: read oid 21 snap 351 2021-03-24T22:06:14.497 INFO:tasks.rados.rados.0.smithi125.stdout:3100: expect (ObjNum 851 snap 278 seq_num 851) 2021-03-24T22:06:14.497 INFO:tasks.rados.rados.0.smithi125.stdout:3101: write oid 14 current snap is 357 2021-03-24T22:06:14.498 INFO:tasks.rados.rados.0.smithi125.stdout:3101: seq_num 1067 ranges {733069=580103,1921581=579703,3218098=537473} 2021-03-24T22:06:14.503 INFO:tasks.rados.rados.0.smithi125.stdout:3101: writing smithi12556470-14 from 733069 to 1313172 tid 1 2021-03-24T22:06:14.511 INFO:tasks.rados.rados.0.smithi125.stdout:3101: writing smithi12556470-14 from 1921581 to 2501284 tid 2 2021-03-24T22:06:14.520 INFO:tasks.rados.rados.0.smithi125.stdout:3101: writing smithi12556470-14 from 3218098 to 3755571 tid 3 2021-03-24T22:06:14.521 INFO:tasks.rados.rados.0.smithi125.stdout:3102: rollback oid 9 current snap is 357 2021-03-24T22:06:14.521 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 9 to 317 2021-03-24T22:06:14.521 INFO:tasks.rados.rados.0.smithi125.stdout:3100: expect (ObjNum 851 snap 278 seq_num 851) 2021-03-24T22:06:14.549 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:14.608 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:14.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:14.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:06:14.645 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:14 smithi125 conmon[19936]: cluster 2021-03-24T22 2021-03-24T22:06:14.646 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:14 smithi125 conmon[19936]: :06:12.999122+0000 mgr.y (mgr.14144) 676 : cluster [DBG] pgmap v1366: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 495 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4.0 MiB/s rd, 1.7 MiB/s wr, 7 op/s; 11/1641 objects misplaced (0.670%) 2021-03-24T22:06:14.646 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:14 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:06:14.646 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:14 smithi125 conmon[19936]: -03-24T22:06:13 2021-03-24T22:06:14.646 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:14 smithi125 conmon[19936]: .475187+0000 mon.a (mon.0 2021-03-24T22:06:14.646 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:14 smithi125 conmon[19936]: ) 2021 : cluster [DBG] osdmap e803: 8 total, 8 up, 8 in 2021-03-24T22:06:14.647 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:14 smithi125 conmon[24927]: cluster 2021-03-24T22:06 2021-03-24T22:06:14.647 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:14 smithi125 conmon[24927]: :12.999122+0000 mgr.y (mgr.14144) 676 : cluster [DBG] pgmap v1366: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 495 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4.0 MiB/s rd, 1.7 MiB/s wr, 7 op/s; 11/1641 objects misplaced (0.670%) 2021-03-24T22:06:14.647 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:14 smithi125 conmon[24927]: cluster 2021-03-24T22 2021-03-24T22:06:14.648 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:14 smithi125 conmon[24927]: :06:13.475187+0000 mon.a (mon. 2021-03-24T22:06:14.648 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:14 smithi125 conmon[24927]: 0) 2021 : cluster [DBG] osdmap e803: 8 total, 8 up, 8 in 2021-03-24T22:06:14.675 INFO:tasks.rados.rados.0.smithi125.stdout:3101: finishing write tid 1 to smithi12556470-14 2021-03-24T22:06:14.675 INFO:tasks.rados.rados.0.smithi125.stdout:3101: finishing write tid 2 to smithi12556470-14 2021-03-24T22:06:14.675 INFO:tasks.rados.rados.0.smithi125.stdout:3101: finishing write tid 3 to smithi12556470-14 2021-03-24T22:06:14.676 INFO:tasks.rados.rados.0.smithi125.stdout:3101: finishing write tid 4 to smithi12556470-14 2021-03-24T22:06:14.676 INFO:tasks.rados.rados.0.smithi125.stdout:3101: finishing write tid 5 to smithi12556470-14 2021-03-24T22:06:14.676 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 14 v 1942 (ObjNum 1067 snap 357 seq_num 1067) dirty exists 2021-03-24T22:06:14.676 INFO:tasks.rados.rados.0.smithi125.stdout:3101: left oid 14 (ObjNum 1067 snap 357 seq_num 1067) 2021-03-24T22:06:14.677 INFO:tasks.rados.rados.0.smithi125.stdout:3102: finishing rollback tid 1 to smithi12556470-9 2021-03-24T22:06:14.677 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 9 v 1583 (ObjNum 1 snap 32741 seq_num 1116453712) dirty dne 2021-03-24T22:06:14.677 INFO:tasks.rados.rados.0.smithi125.stdout:3100: done (2 left) 2021-03-24T22:06:14.677 INFO:tasks.rados.rados.0.smithi125.stdout:3101: done (1 left) 2021-03-24T22:06:14.677 INFO:tasks.rados.rados.0.smithi125.stdout:3102: done (0 left) 2021-03-24T22:06:14.678 INFO:tasks.rados.rados.0.smithi125.stdout:3103: rollback oid 1 current snap is 357 2021-03-24T22:06:14.678 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 1 to 344 2021-03-24T22:06:14.678 INFO:tasks.rados.rados.0.smithi125.stdout:3104: write oid 32 current snap is 357 2021-03-24T22:06:14.678 INFO:tasks.rados.rados.0.smithi125.stdout:3104: seq_num 1068 ranges {749876=655352,1863710=723361,3012815=748745} 2021-03-24T22:06:14.690 INFO:tasks.rados.rados.0.smithi125.stdout:3104: writing smithi12556470-32 from 749876 to 1405228 tid 1 2021-03-24T22:06:14.697 INFO:tasks.rados.rados.0.smithi125.stdout:3104: writing smithi12556470-32 from 1863710 to 2587071 tid 2 2021-03-24T22:06:14.705 INFO:tasks.rados.rados.0.smithi125.stdout:3104: writing smithi12556470-32 from 3012815 to 3761560 tid 3 2021-03-24T22:06:14.706 INFO:tasks.rados.rados.0.smithi125.stdout:3105: rollback oid 8 current snap is 357 2021-03-24T22:06:14.706 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 8 to 331 2021-03-24T22:06:14.706 INFO:tasks.rados.rados.0.smithi125.stdout:3103: finishing rollback tid 0 to smithi12556470-1 2021-03-24T22:06:14.707 INFO:tasks.rados.rados.0.smithi125.stdout:3103: finishing rollback tid 1 to smithi12556470-1 2021-03-24T22:06:14.707 INFO:tasks.rados.rados.0.smithi125.stdout:3103: finishing rollback tid 2 to smithi12556470-1 2021-03-24T22:06:14.707 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 1 v 1449 (ObjNum 808 snap 265 seq_num 808) dirty exists 2021-03-24T22:06:14.707 INFO:tasks.rados.rados.0.smithi125.stdout:3104: finishing write tid 1 to smithi12556470-32 2021-03-24T22:06:14.707 INFO:tasks.rados.rados.0.smithi125.stdout:3104: finishing write tid 2 to smithi12556470-32 2021-03-24T22:06:14.708 INFO:tasks.rados.rados.0.smithi125.stdout:3103: done (2 left) 2021-03-24T22:06:14.708 INFO:tasks.rados.rados.0.smithi125.stdout:3106: read oid 23 snap -1 2021-03-24T22:06:14.708 INFO:tasks.rados.rados.0.smithi125.stdout:3106: expect deleted 2021-03-24T22:06:14.708 INFO:tasks.rados.rados.0.smithi125.stdout:3107: snap_create 2021-03-24T22:06:14.711 INFO:tasks.rados.rados.0.smithi125.stdout:3104: finishing write tid 3 to smithi12556470-32 2021-03-24T22:06:14.712 INFO:tasks.rados.rados.0.smithi125.stdout:3105: finishing rollback tid 0 to smithi12556470-8 2021-03-24T22:06:14.712 INFO:tasks.rados.rados.0.smithi125.stdout:3104: finishing write tid 4 to smithi12556470-32 2021-03-24T22:06:14.712 INFO:tasks.rados.rados.0.smithi125.stdout:3104: finishing write tid 5 to smithi12556470-32 2021-03-24T22:06:14.712 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 32 v 1948 (ObjNum 1068 snap 357 seq_num 1068) dirty exists 2021-03-24T22:06:14.713 INFO:tasks.rados.rados.0.smithi125.stdout:3104: left oid 32 (ObjNum 1068 snap 357 seq_num 1068) 2021-03-24T22:06:14.713 INFO:tasks.rados.rados.0.smithi125.stdout:3105: finishing rollback tid 1 to smithi12556470-8 2021-03-24T22:06:14.713 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 8 v 568 (ObjNum 288 snap 0 seq_num 49) dirty dne 2021-03-24T22:06:14.713 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:14.757 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:14.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:14.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:06:14.863 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:14.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:14 smithi142 conmon[20125]: cluster 2021-03-24T22:06 2021-03-24T22:06:14.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:14 smithi142 conmon[20125]: :12.999122+0000 mgr.y (mgr.14144) 676 : cluster [DBG] pgmap v1366: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 495 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4.0 MiB/s rd, 1.7 MiB/s wr, 7 op/s; 11/1641 objects misplaced (0.670%) 2021-03-24T22:06:14.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:14 smithi142 conmon[20125]: cluster 2021-03-24T22:06:13.475187+0000 2021-03-24T22:06:14.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:14 smithi142 conmon[20125]: mon.a (mon.0) 2021 : cluster [DBG] osdmap e803: 8 total, 8 up, 8 in 2021-03-24T22:06:14.926 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:14.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:14.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:06:15.031 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:15.090 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:15.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:15.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:06:15.195 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:15.252 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:15.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:15.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:15.358 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:15.413 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:15.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:15.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:06:15.429 INFO:tasks.rados.rados.0.smithi125.stdout:3104: done (3 left) 2021-03-24T22:06:15.430 INFO:tasks.rados.rados.0.smithi125.stdout:3105: done (2 left) 2021-03-24T22:06:15.430 INFO:tasks.rados.rados.0.smithi125.stdout:3106: done (1 left) 2021-03-24T22:06:15.430 INFO:tasks.rados.rados.0.smithi125.stdout:3107: done (0 left) 2021-03-24T22:06:15.431 INFO:tasks.rados.rados.0.smithi125.stdout:3108: rollback oid 5 current snap is 358 2021-03-24T22:06:15.431 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 5 to 349 2021-03-24T22:06:15.431 INFO:tasks.rados.rados.0.smithi125.stdout:3109: write (excl) oid 11 current snap is 358 2021-03-24T22:06:15.431 INFO:tasks.rados.rados.0.smithi125.stdout:3109: seq_num 1069 ranges {766683=730601,1922192=583372,3156591=676370} 2021-03-24T22:06:15.439 INFO:tasks.rados.rados.0.smithi125.stdout:3109: writing smithi12556470-11 from 766683 to 1497284 tid 1 2021-03-24T22:06:15.446 INFO:tasks.rados.rados.0.smithi125.stdout:3109: writing smithi12556470-11 from 1922192 to 2505564 tid 2 2021-03-24T22:06:15.455 INFO:tasks.rados.rados.0.smithi125.stdout:3109: writing smithi12556470-11 from 3156591 to 3832961 tid 3 2021-03-24T22:06:15.455 INFO:tasks.rados.rados.0.smithi125.stdout:3110: rollback oid 40 current snap is 358 2021-03-24T22:06:15.455 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 40 to 334 2021-03-24T22:06:15.455 INFO:tasks.rados.rados.0.smithi125.stdout:3108: finishing rollback tid 1 to smithi12556470-5 2021-03-24T22:06:15.456 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 5 v 1140 (ObjNum 1116504848 snap 21916 seq_num 1116457648) dirty dne 2021-03-24T22:06:15.456 INFO:tasks.rados.rados.0.smithi125.stdout:3109: finishing write tid 1 to smithi12556470-11 2021-03-24T22:06:15.456 INFO:tasks.rados.rados.0.smithi125.stdout:3109: finishing write tid 2 to smithi12556470-11 2021-03-24T22:06:15.456 INFO:tasks.rados.rados.0.smithi125.stdout:3108: done (2 left) 2021-03-24T22:06:15.457 INFO:tasks.rados.rados.0.smithi125.stdout:3111: write oid 25 current snap is 358 2021-03-24T22:06:15.457 INFO:tasks.rados.rados.0.smithi125.stdout:3111: seq_num 1070 ranges {783490=405850,1980674=727030,3184014=487642} 2021-03-24T22:06:15.461 INFO:tasks.rados.rados.0.smithi125.stdout:3111: writing smithi12556470-25 from 783490 to 1189340 tid 1 2021-03-24T22:06:15.470 INFO:tasks.rados.rados.0.smithi125.stdout:3111: writing smithi12556470-25 from 1980674 to 2707704 tid 2 2021-03-24T22:06:15.477 INFO:tasks.rados.rados.0.smithi125.stdout:3111: writing smithi12556470-25 from 3184014 to 3671656 tid 3 2021-03-24T22:06:15.477 INFO:tasks.rados.rados.0.smithi125.stdout:3112: snap_create 2021-03-24T22:06:15.477 INFO:tasks.rados.rados.0.smithi125.stdout:3109: finishing write tid 3 to smithi12556470-11 2021-03-24T22:06:15.478 INFO:tasks.rados.rados.0.smithi125.stdout:3109: finishing write tid 4 to smithi12556470-11 2021-03-24T22:06:15.478 INFO:tasks.rados.rados.0.smithi125.stdout:3109: finishing write tid 5 to smithi12556470-11 2021-03-24T22:06:15.478 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 11 v 570 (ObjNum 1069 snap 358 seq_num 1069) dirty exists 2021-03-24T22:06:15.478 INFO:tasks.rados.rados.0.smithi125.stdout:3109: left oid 11 (ObjNum 1069 snap 358 seq_num 1069) 2021-03-24T22:06:15.478 INFO:tasks.rados.rados.0.smithi125.stdout:3110: finishing rollback tid 0 to smithi12556470-40 2021-03-24T22:06:15.479 INFO:tasks.rados.rados.0.smithi125.stdout:3110: finishing rollback tid 1 to smithi12556470-40 2021-03-24T22:06:15.479 INFO:tasks.rados.rados.0.smithi125.stdout:3110: finishing rollback tid 2 to smithi12556470-40 2021-03-24T22:06:15.479 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 40 v 1952 (ObjNum 906 snap 300 seq_num 906) dirty exists 2021-03-24T22:06:15.479 INFO:tasks.rados.rados.0.smithi125.stdout:3111: finishing write tid 1 to smithi12556470-25 2021-03-24T22:06:15.480 INFO:tasks.rados.rados.0.smithi125.stdout:3111: finishing write tid 2 to smithi12556470-25 2021-03-24T22:06:15.480 INFO:tasks.rados.rados.0.smithi125.stdout:3111: finishing write tid 3 to smithi12556470-25 2021-03-24T22:06:15.481 INFO:tasks.rados.rados.0.smithi125.stdout:3111: finishing write tid 4 to smithi12556470-25 2021-03-24T22:06:15.481 INFO:tasks.rados.rados.0.smithi125.stdout:3111: finishing write tid 5 to smithi12556470-25 2021-03-24T22:06:15.481 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 25 v 1447 (ObjNum 1070 snap 358 seq_num 1070) dirty exists 2021-03-24T22:06:15.481 INFO:tasks.rados.rados.0.smithi125.stdout:3111: left oid 25 (ObjNum 1070 snap 358 seq_num 1070) 2021-03-24T22:06:15.495 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:15 smithi125 conmon[24927]: cluster 2021-03-24T22:06:14. 2021-03-24T22:06:15.496 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:15 smithi125 conmon[19936]: cluster 2021-03-24T22:06:14. 2021-03-24T22:06:15.496 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:15 smithi125 conmon[19936]: 491421+0000 mon.a (mon.0) 2022 : cluster [DBG] osdmap e804: 8 total, 8 up, 8 in 2021-03-24T22:06:15.496 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:15 smithi125 conmon[19936]: audit 2021-03-24 2021-03-24T22:06:15.497 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:15 smithi125 conmon[19936]: T22:06:15.104560+0000 mon.a (mon.0) 2023 2021-03-24T22:06:15.497 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:15 smithi125 conmon[19936]: : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T22:06:15.497 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:15 smithi125 conmon[19936]: cluster 2021-03-24T22:06:15.497 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:15 smithi125 conmon[19936]: 2021-03-24T22 2021-03-24T22:06:15.497 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:15 smithi125 conmon[19936]: :06:15.423756+0000 mon.a (mon 2021-03-24T22:06:15.498 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:15 smithi125 conmon[19936]: .0) 2024 : cluster [DBG] osdmap e805: 8 total, 8 up, 8 in 2021-03-24T22:06:15.498 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:15 smithi125 conmon[19936]: 2021-03-24T22:06:15.498 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:15 smithi125 conmon[19936]: audit 2021-03-24T22:06:15.498 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:15 smithi125 conmon[19936]: 2021-03-24T22 2021-03-24T22:06:15.499 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:15 smithi125 conmon[19936]: :06:15.466168+0000 mon.a ( 2021-03-24T22:06:15.499 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:15 smithi125 conmon[19936]: mon.0) 2025 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:06:15.499 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:15 smithi125 conmon[19936]: 2021-03-24T22:06:15.499 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:15 smithi125 conmon[19936]: audit 2021-03-24T 2021-03-24T22:06:15.499 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:15 smithi125 conmon[19936]: 22:06:15.469889+0000 mon.a ( 2021-03-24T22:06:15.500 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:15 smithi125 conmon[19936]: mon.0) 2026 : audit [INF] 2021-03-24T22:06:15.500 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:15 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T22:06:15.517 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:15.569 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:15.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:15.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:06:15.676 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:15.734 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:15.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:15.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:06:15.840 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:15.854 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:15 smithi142 conmon[20125]: cluster 2021-03-24T22:06:14 2021-03-24T22:06:15.854 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:15 smithi142 conmon[20125]: .491421+0000 mon.a (mon.0) 2022 : cluster [DBG] osdmap e804: 8 total, 8 up, 8 in 2021-03-24T22:06:15.855 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:15 smithi142 conmon[20125]: audit 2021-03-24T22:06:15.104560+0000 2021-03-24T22:06:15.855 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:15 smithi142 conmon[20125]: mon.a (mon.0) 2023 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T22:06:15.855 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:15 smithi142 conmon[20125]: cluster 2021-03-24T22:06:15 2021-03-24T22:06:15.855 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:15 smithi142 conmon[20125]: .423756+0000 mon.a (mon.0) 2024 : cluster [DBG] 2021-03-24T22:06:15.855 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:15 smithi142 conmon[20125]: osdmap e805: 8 total, 8 up, 8 in 2021-03-24T22:06:15.856 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:15 smithi142 conmon[20125]: audit 2021-03-24T22:06:15.856 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:15 smithi142 conmon[20125]: 2021-03-24T22:06:15.466168+0000 mon.a ( 2021-03-24T22:06:15.856 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:15 smithi142 conmon[20125]: mon.0) 2025 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:06:15.856 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:15 smithi142 conmon[20125]: audit 2021-03-24T22:06:15 2021-03-24T22:06:15.857 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:15 smithi142 conmon[20125]: .469889+0000 mon.a (mon.0) 2026 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T22:06:15.896 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:15.900 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:15 smithi125 conmon[24927]: 491421+0000 mon.a (mon.0) 2022 : cluster [DBG] osdmap e804: 8 total, 8 up, 8 in 2021-03-24T22:06:15.901 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:15 smithi125 conmon[24927]: audit 2021-03-24T22:06:15.104560+0000 mon.a (mon. 2021-03-24T22:06:15.901 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:15 smithi125 conmon[24927]: 0) 2023 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T22:06:15.902 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:15 smithi125 conmon[24927]: cluster 2021-03-24T22:06:15.423756+0000 mon.a (mon.0) 2021-03-24T22:06:15.902 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:15 smithi125 conmon[24927]: 2024 : cluster [DBG] osdmap e805: 8 total, 8 up, 8 in 2021-03-24T22:06:15.902 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:15 smithi125 conmon[24927]: audit 2021-03-24T22:06:15.466168+0000 mon.a (mon 2021-03-24T22:06:15.903 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:15 smithi125 conmon[24927]: .0) 2025 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:06:15.903 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:15 smithi125 conmon[24927]: audit 2021-03-24 2021-03-24T22:06:15.903 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:15 smithi125 conmon[24927]: T22:06:15.469889+0000 mon.a (mon.0) 2026 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T22:06:15.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:15.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:06:16.007 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:16.065 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:16.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:16.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:16.171 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:16.231 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:16.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:16.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:06:16.338 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:16.398 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:16.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:16.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:16.473 INFO:tasks.rados.rados.0.smithi125.stdout:3109: done (3 left) 2021-03-24T22:06:16.473 INFO:tasks.rados.rados.0.smithi125.stdout:3110: done (2 left) 2021-03-24T22:06:16.473 INFO:tasks.rados.rados.0.smithi125.stdout:3111: done (1 left) 2021-03-24T22:06:16.474 INFO:tasks.rados.rados.0.smithi125.stdout:3112: done (0 left) 2021-03-24T22:06:16.474 INFO:tasks.rados.rados.0.smithi125.stdout:3113: snap_remove snap 330 2021-03-24T22:06:16.505 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:16.558 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:16.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:16.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:06:16.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:16 smithi142 conmon[20125]: cluster 2021-03-24T22:06:14.797748+0000 osd.0 2021-03-24T22:06:16.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:16 smithi142 conmon[20125]: (osd.0) 159 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:06:16.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:16 smithi142 conmon[20125]: cluster 2021-03-24T22:06:14.800687+0000 osd.0 2021-03-24T22:06:16.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:16 smithi142 conmon[20125]: (osd.0) 160 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:06:16.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:16 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:06:16.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:16 smithi142 conmon[20125]: 22:06:14.999931+0000 mgr.y (mgr 2021-03-24T22:06:16.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:16 smithi142 conmon[20125]: .14144) 677 : cluster [DBG] pgmap v1369: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 482 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 1.7 MiB/s wr, 14 op/s; 11/1614 objects misplaced (0.682%) 2021-03-24T22:06:16.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:16 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:06:16.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:16 smithi142 conmon[20125]: 03-24T22:06:16.470370+0000 2021-03-24T22:06:16.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:16 smithi142 conmon[20125]: mon.a (mon.0) 2027 : cluster 2021-03-24T22:06:16.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:16 smithi142 conmon[20125]: [DBG] osdmap e806: 8 total, 8 up, 8 in 2021-03-24T22:06:16.666 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:16.724 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:16.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:16.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:06:16.820 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:16 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:06:16.821 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:16 smithi125 conmon[19936]: 06:14.797748+0000 osd.0 (osd.0) 159 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:06:16.821 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:16 smithi125 conmon[19936]: cluster 2021-03-24T22:06:14.800687+0000 osd.0 ( 2021-03-24T22:06:16.822 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:16 smithi125 conmon[19936]: osd.0) 160 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:06:16.822 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:16 smithi125 conmon[19936]: cluster 2021-03-24T22:06: 2021-03-24T22:06:16.822 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:16 smithi125 conmon[19936]: 14.999931+0000 mgr.y (mgr.14144) 677 : 2021-03-24T22:06:16.823 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:16 smithi125 conmon[19936]: cluster [DBG] pgmap v1369: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 482 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 1.7 MiB/s wr, 14 op/s; 11/1614 objects misplaced (0.682%) 2021-03-24T22:06:16.823 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:16 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T22:06:16.824 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:16 smithi125 conmon[19936]: -24T22:06:16.470370+0000 mon.a ( 2021-03-24T22:06:16.824 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:16 smithi125 conmon[19936]: mon.0) 2027 : cluster [DBG] osdmap e806: 8 total, 8 up, 8 in 2021-03-24T22:06:16.825 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:16 smithi125 conmon[24927]: cluster 2021-03-24T22:06 2021-03-24T22:06:16.825 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:16 smithi125 conmon[24927]: :14.797748+0000 osd.0 (osd.0) 159 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:06:16.826 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:16 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T22:06:16.826 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:16 smithi125 conmon[24927]: -24T22:06:14 2021-03-24T22:06:16.826 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:16 smithi125 conmon[24927]: .800687+0000 osd.0 (osd. 2021-03-24T22:06:16.827 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:16 smithi125 conmon[24927]: 0) 160 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:06:16.827 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:16 smithi125 conmon[24927]: 2021-03-24T22:06:16.827 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:16 smithi125 conmon[24927]: cluster 2021-03-24T22:06: 2021-03-24T22:06:16.828 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:16 smithi125 conmon[24927]: 14.999931+0000 mgr.y (mgr.14144) 677 : cluster [DBG] 2021-03-24T22:06:16.828 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:16 smithi125 conmon[24927]: pgmap v1369: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 482 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 1.7 MiB/s wr, 14 op/s; 11/1614 objects misplaced (0.682%) 2021-03-24T22:06:16.828 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:16 smithi125 conmon[24927]: cluster 2021-03-24T22:06:16.828 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:16 smithi125 conmon[24927]: 2021-03-24 2021-03-24T22:06:16.829 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:16 smithi125 conmon[24927]: T22:06:16.470370+0000 mon.a (mon.0) 2027 2021-03-24T22:06:16.829 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:16 smithi125 conmon[24927]: : cluster [DBG] osdmap e806: 8 total, 8 up, 8 in 2021-03-24T22:06:16.831 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:16.890 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:16.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:16.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:06:16.998 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:17.066 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:17.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:17.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:06:17.092 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T22:06:17.093 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T22:06:17.093 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T22:06:17.093 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T22:06:17.108 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T22:06:17.132 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T22:06:17.157 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T22:06:17.170 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:17.186 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T22:06:17.191 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-24T22:06:17.191 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-24T22:06:17.192 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-24T22:06:17.192 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-24T22:06:17.192 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-24T22:06:17.193 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-24T22:06:17.193 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-24T22:06:17.194 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-24T22:06:17.222 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:17.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:17.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:06:17.328 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:17.389 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:17.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:17.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:17.482 INFO:tasks.rados.rados.0.smithi125.stdout:3113: done (0 left) 2021-03-24T22:06:17.483 INFO:tasks.rados.rados.0.smithi125.stdout:3114: write (excl) oid 35 current snap is 359 2021-03-24T22:06:17.483 INFO:tasks.rados.rados.0.smithi125.stdout:3114: seq_num 1071 ranges {400297=481099,1639156=361141} 2021-03-24T22:06:17.485 INFO:tasks.rados.rados.0.smithi125.stdout:3114: writing smithi12556470-35 from 400297 to 881396 tid 1 2021-03-24T22:06:17.493 INFO:tasks.rados.rados.0.smithi125.stdout:3114: writing smithi12556470-35 from 1639156 to 2000297 tid 2 2021-03-24T22:06:17.494 INFO:tasks.rados.rados.0.smithi125.stdout:3115: rollback oid 9 current snap is 359 2021-03-24T22:06:17.494 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 9 to 349 2021-03-24T22:06:17.494 INFO:tasks.rados.rados.0.smithi125.stdout:3116: snap_remove snap 304 2021-03-24T22:06:17.497 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:17.540 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:17.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:17.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:06:17.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:17 smithi142 conmon[20125]: audit 2021-03-24T22:06:16.973335+0000 mon.a (mon.0) 2028 : audit [DBG] from='client.? v1:172.21.15.125:0/4137111897' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:06:17.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:17 smithi142 conmon[20125]: cluster 2021-03-24T22:06:17.472956+0000 mon.a (mon.0) 2029 : cluster [DBG] osdmap e807: 8 total, 8 up, 8 in 2021-03-24T22:06:17.645 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:17.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:17 smithi125 conmon[19936]: audit 2021-03-24T22:06:16.973335+0000 mon.a ( 2021-03-24T22:06:17.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:17 smithi125 conmon[19936]: mon.0) 2028 : audit [DBG] from='client.? v1:172.21.15.125:0/4137111897' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:06:17.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:17 smithi125 conmon[19936]: cluster 2021-03-24T22:06:17.472956+0000 mon.a (mon.0 2021-03-24T22:06:17.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:17 smithi125 conmon[19936]: ) 2029 : cluster [DBG] osdmap e807: 8 total, 8 up, 8 in 2021-03-24T22:06:17.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:17 smithi125 conmon[24927]: audit 2021-03-24T22:06:16 2021-03-24T22:06:17.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:17 smithi125 conmon[24927]: .973335+0000 mon.a (mon.0) 2028 : audit [DBG] from='client.? v1:172.21.15.125:0/4137111897' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:06:17.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:17 smithi125 conmon[24927]: cluster 2021-03-24T22:06:17.472956+0000 2021-03-24T22:06:17.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:17 smithi125 conmon[24927]: mon.a (mon.0) 2029 : cluster [DBG] osdmap e807: 8 total, 8 up, 8 in 2021-03-24T22:06:17.701 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:17.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:17.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:17.808 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:17.869 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:17.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:17.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:06:17.974 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:18.030 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:18.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:18.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:06:18.136 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:18.191 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:18.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:18.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:06:18.297 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:18.353 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:18.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:18.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:06:18.460 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:18.486 INFO:tasks.rados.rados.0.smithi125.stdout:3116: done (2 left) 2021-03-24T22:06:18.487 INFO:tasks.rados.rados.0.smithi125.stdout:3117: copy_from oid 4 from oid 2 current snap is 359 2021-03-24T22:06:18.487 INFO:tasks.rados.rados.0.smithi125.stdout:3114: finishing write tid 1 to smithi12556470-35 2021-03-24T22:06:18.487 INFO:tasks.rados.rados.0.smithi125.stdout:3114: finishing write tid 2 to smithi12556470-35 2021-03-24T22:06:18.488 INFO:tasks.rados.rados.0.smithi125.stdout:3114: finishing write tid 3 to smithi12556470-35 2021-03-24T22:06:18.488 INFO:tasks.rados.rados.0.smithi125.stdout:3114: finishing write tid 4 to smithi12556470-35 2021-03-24T22:06:18.488 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 35 v 974 (ObjNum 1071 snap 359 seq_num 1071) dirty exists 2021-03-24T22:06:18.488 INFO:tasks.rados.rados.0.smithi125.stdout:3114: left oid 35 (ObjNum 1071 snap 359 seq_num 1071) 2021-03-24T22:06:18.489 INFO:tasks.rados.rados.0.smithi125.stdout:3115: finishing rollback tid 1 to smithi12556470-9 2021-03-24T22:06:18.489 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 9 v 1583 (ObjNum 1117425536 snap 21916 seq_num 1117675056) dirty dne 2021-03-24T22:06:18.489 INFO:tasks.rados.rados.0.smithi125.stdout:3114: done (2 left) 2021-03-24T22:06:18.489 INFO:tasks.rados.rados.0.smithi125.stdout:3115: done (1 left) 2021-03-24T22:06:18.489 INFO:tasks.rados.rados.0.smithi125.stdout:3118: write (excl) oid 16 current snap is 359 2021-03-24T22:06:18.490 INFO:tasks.rados.rados.0.smithi125.stdout:3118: seq_num 1072 ranges {417104=672701,1813991=203113} 2021-03-24T22:06:18.491 INFO:tasks.rados.rados.0.smithi125.stdout:3118: writing smithi12556470-16 from 417104 to 1089805 tid 1 2021-03-24T22:06:18.497 INFO:tasks.rados.rados.0.smithi125.stdout:3118: writing smithi12556470-16 from 1813991 to 2017104 tid 2 2021-03-24T22:06:18.498 INFO:tasks.rados.rados.0.smithi125.stdout:3119: delete oid 46 current snap is 359 2021-03-24T22:06:18.513 INFO:tasks.rados.rados.0.smithi125.stdout:3119: done (2 left) 2021-03-24T22:06:18.514 INFO:tasks.rados.rados.0.smithi125.stdout:3120: snap_create 2021-03-24T22:06:18.520 INFO:tasks.rados.rados.0.smithi125.stdout:3118: finishing write tid 1 to smithi12556470-16 2021-03-24T22:06:18.522 INFO:tasks.rados.rados.0.smithi125.stdout:3118: finishing write tid 2 to smithi12556470-16 2021-03-24T22:06:18.523 INFO:tasks.rados.rados.0.smithi125.stdout:3118: finishing write tid 3 to smithi12556470-16 2021-03-24T22:06:18.523 INFO:tasks.rados.rados.0.smithi125.stdout:3118: finishing write tid 4 to smithi12556470-16 2021-03-24T22:06:18.523 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 16 v 843 (ObjNum 1072 snap 359 seq_num 1072) dirty exists 2021-03-24T22:06:18.524 INFO:tasks.rados.rados.0.smithi125.stdout:3118: left oid 16 (ObjNum 1072 snap 359 seq_num 1072) 2021-03-24T22:06:18.526 INFO:tasks.rados.rados.0.smithi125.stdout:3117: finishing copy_from to smithi12556470-4 2021-03-24T22:06:18.526 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 4 v 1017 (ObjNum 984 snap 326 seq_num 984) dirty exists 2021-03-24T22:06:18.527 INFO:tasks.rados.rados.0.smithi125.stdout:3117: finishing copy_from racing read to smithi12556470-4 2021-03-24T22:06:18.529 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:18.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:18.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:18.549 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:18 smithi125 conmon[19936]: cluster 2021-03-24T22:06:17.000493+0000 mgr.y 2021-03-24T22:06:18.549 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:18 smithi125 conmon[19936]: (mgr.14144) 678 : cluster [DBG] pgmap v1372: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 482 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 1.7 MiB/s wr, 16 op/s; 11/1614 objects misplaced (0.682%) 2021-03-24T22:06:18.549 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:18 smithi125 conmon[19936]: cluster 2021-03-24T22:06:18.477260+0000 mon.a (mon.0) 2030 : cluster [DBG] osdmap e808: 8 total, 8 up, 8 in 2021-03-24T22:06:18.550 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:18 smithi125 conmon[24927]: cluster 2021-03-24T22:06:17.000493+0000 mgr.y 2021-03-24T22:06:18.550 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:18 smithi125 conmon[24927]: (mgr.14144) 678 : cluster [DBG] pgmap v1372: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 482 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 1.7 MiB/s wr, 16 op/s; 11/1614 objects misplaced (0.682%) 2021-03-24T22:06:18.550 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:18 smithi125 conmon[24927]: cluster 2021-03-24T22:06:18.477260+0000 mon.a (mon.0) 2030 : cluster [DBG] osdmap e808: 8 total, 8 up, 8 in 2021-03-24T22:06:18.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:18 smithi142 conmon[20125]: cluster 2021-03-24T22:06:17.000493+0000 mgr.y 2021-03-24T22:06:18.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:18 smithi142 conmon[20125]: (mgr.14144) 678 : cluster [DBG] pgmap v1372: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 482 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 1.7 MiB/s wr, 16 op/s; 11/1614 objects misplaced (0.682%) 2021-03-24T22:06:18.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:18 smithi142 conmon[20125]: cluster 2021-03-24T22:06:18.477260+0000 mon.a (mon.0) 2030 : cluster [DBG] osdmap e808: 8 total, 8 up, 8 in 2021-03-24T22:06:18.618 INFO:journalctl@ceph.osd.6.smithi142.stdout:Mar 24 22:06:18 smithi142 conmon[30596]: debug 2021-03-24T22:06:18.501+0000 7feff9829700 -1 osd.6 pg_epoch: 808 pg[7.c( v 807'1303 (0'0,807'1303] local-lis/les=555/556 n=15 ec=99/99 lis/c=555/555 les/c/f=556/556/0 sis=555) [6,3,7] r=0 lpr=555 crt=807'1303 lcod 807'1301 mlcod 807'1301 active+clean+snaptrim trimq=[243~1] ps=[231~1,244~1,276~1]] removing snap head 2021-03-24T22:06:18.633 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:18.686 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:18.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:18.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:06:18.776 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-24T22:06:18.776 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T22:06:18.790 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:18.838 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:18.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:18.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:06:18.945 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:19.004 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:19.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:19.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:06:19.109 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:19.171 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:19.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:19.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:06:19.277 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:19.333 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:19.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:19.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:06:19.437 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:19.486 INFO:tasks.rados.rados.0.smithi125.stdout:3117: done (2 left) 2021-03-24T22:06:19.487 INFO:tasks.rados.rados.0.smithi125.stdout:3118: done (1 left) 2021-03-24T22:06:19.487 INFO:tasks.rados.rados.0.smithi125.stdout:3120: done (0 left) 2021-03-24T22:06:19.487 INFO:tasks.rados.rados.0.smithi125.stdout:3121: snap_create 2021-03-24T22:06:19.495 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:19.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:19.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:19.600 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:19.656 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:19.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:19.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:06:19.763 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:19.822 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:19.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:19.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:19.929 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:19.982 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:19.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:19.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:20.089 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:20.143 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:20.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:20.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:06:20.298 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:20.354 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:20.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:20.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:20.438 INFO:tasks.rados.rados.0.smithi125.stdout:3121: done (0 left) 2021-03-24T22:06:20.438 INFO:tasks.rados.rados.0.smithi125.stdout:3122: write oid 13 current snap is 361 2021-03-24T22:06:20.439 INFO:tasks.rados.rados.0.smithi125.stdout:3122: seq_num 1074 ranges {450718=423199,1414602=636116} 2021-03-24T22:06:20.443 INFO:tasks.rados.rados.0.smithi125.stdout:3122: writing smithi12556470-13 from 450718 to 873917 tid 1 2021-03-24T22:06:20.450 INFO:tasks.rados.rados.0.smithi125.stdout:3122: writing smithi12556470-13 from 1414602 to 2050718 tid 2 2021-03-24T22:06:20.451 INFO:tasks.rados.rados.0.smithi125.stdout:3123: write (excl) oid 20 current snap is 361 2021-03-24T22:06:20.451 INFO:tasks.rados.rados.0.smithi125.stdout:3123: seq_num 1075 ranges {467525=498448,1473084=594379} 2021-03-24T22:06:20.457 INFO:tasks.rados.rados.0.smithi125.stdout:3123: writing smithi12556470-20 from 467525 to 965973 tid 1 2021-03-24T22:06:20.461 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:20.464 INFO:tasks.rados.rados.0.smithi125.stdout:3123: writing smithi12556470-20 from 1473084 to 2067463 tid 2 2021-03-24T22:06:20.464 INFO:tasks.rados.rados.0.smithi125.stdout:3124: copy_from oid 47 from oid 44 current snap is 361 2021-03-24T22:06:20.465 INFO:tasks.rados.rados.0.smithi125.stdout:3122: finishing write tid 1 to smithi12556470-13 2021-03-24T22:06:20.465 INFO:tasks.rados.rados.0.smithi125.stdout:3122: finishing write tid 2 to smithi12556470-13 2021-03-24T22:06:20.465 INFO:tasks.rados.rados.0.smithi125.stdout:3122: finishing write tid 3 to smithi12556470-13 2021-03-24T22:06:20.465 INFO:tasks.rados.rados.0.smithi125.stdout:3122: finishing write tid 4 to smithi12556470-13 2021-03-24T22:06:20.466 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 13 v 1460 (ObjNum 1074 snap 361 seq_num 1074) dirty exists 2021-03-24T22:06:20.466 INFO:tasks.rados.rados.0.smithi125.stdout:3122: left oid 13 (ObjNum 1074 snap 361 seq_num 1074) 2021-03-24T22:06:20.466 INFO:tasks.rados.rados.0.smithi125.stdout:3123: finishing write tid 1 to smithi12556470-20 2021-03-24T22:06:20.466 INFO:tasks.rados.rados.0.smithi125.stdout:3122: done (2 left) 2021-03-24T22:06:20.467 INFO:tasks.rados.rados.0.smithi125.stdout:3125: delete oid 42 current snap is 361 2021-03-24T22:06:20.472 INFO:tasks.rados.rados.0.smithi125.stdout:3125: done (2 left) 2021-03-24T22:06:20.472 INFO:tasks.rados.rados.0.smithi125.stdout:3126: write oid 27 current snap is 361 2021-03-24T22:06:20.473 INFO:tasks.rados.rados.0.smithi125.stdout:3126: seq_num 1077 ranges {501139=648946,1873695=227444} 2021-03-24T22:06:20.481 INFO:tasks.rados.rados.0.smithi125.stdout:3126: writing smithi12556470-27 from 501139 to 1150085 tid 1 2021-03-24T22:06:20.486 INFO:tasks.rados.rados.0.smithi125.stdout:3126: writing smithi12556470-27 from 1873695 to 2101139 tid 2 2021-03-24T22:06:20.488 INFO:tasks.rados.rados.0.smithi125.stdout:3127: copy_from oid 46 from oid 38 current snap is 361 2021-03-24T22:06:20.488 INFO:tasks.rados.rados.0.smithi125.stdout:3123: finishing write tid 2 to smithi12556470-20 2021-03-24T22:06:20.488 INFO:tasks.rados.rados.0.smithi125.stdout:3123: finishing write tid 3 to smithi12556470-20 2021-03-24T22:06:20.488 INFO:tasks.rados.rados.0.smithi125.stdout:3123: finishing write tid 4 to smithi12556470-20 2021-03-24T22:06:20.489 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 20 v 1450 (ObjNum 1075 snap 361 seq_num 1075) dirty exists 2021-03-24T22:06:20.489 INFO:tasks.rados.rados.0.smithi125.stdout:3123: left oid 20 (ObjNum 1075 snap 361 seq_num 1075) 2021-03-24T22:06:20.489 INFO:tasks.rados.rados.0.smithi125.stdout:3123: done (3 left) 2021-03-24T22:06:20.490 INFO:tasks.rados.rados.0.smithi125.stdout:3128: read oid 32 snap -1 2021-03-24T22:06:20.490 INFO:tasks.rados.rados.0.smithi125.stdout:3128: expect (ObjNum 1068 snap 357 seq_num 1068) 2021-03-24T22:06:20.491 INFO:tasks.rados.rados.0.smithi125.stdout:3129: write oid 41 current snap is 361 2021-03-24T22:06:20.491 INFO:tasks.rados.rados.0.smithi125.stdout:3129: seq_num 1078 ranges {517946=724195,1932177=185769} 2021-03-24T22:06:20.496 INFO:tasks.rados.rados.0.smithi125.stdout:3129: writing smithi12556470-41 from 517946 to 1242141 tid 1 2021-03-24T22:06:20.502 INFO:tasks.rados.rados.0.smithi125.stdout:3129: writing smithi12556470-41 from 1932177 to 2117946 tid 2 2021-03-24T22:06:20.502 INFO:tasks.rados.rados.0.smithi125.stdout:3130: copy_from oid 26 from oid 24 current snap is 361 2021-03-24T22:06:20.502 INFO:tasks.rados.rados.0.smithi125.stdout:3131: write (excl) oid 4 current snap is 361 2021-03-24T22:06:20.503 INFO:tasks.rados.rados.0.smithi125.stdout:3131: seq_num 1079 ranges {534753=799444,1990659=144094} 2021-03-24T22:06:20.510 INFO:tasks.rados.rados.0.smithi125.stdout:3131: writing smithi12556470-4 from 534753 to 1334197 tid 1 2021-03-24T22:06:20.511 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:20.515 INFO:tasks.rados.rados.0.smithi125.stdout:3131: writing smithi12556470-4 from 1990659 to 2134753 tid 2 2021-03-24T22:06:20.516 INFO:tasks.rados.rados.0.smithi125.stdout:3132: snap_create 2021-03-24T22:06:20.516 INFO:tasks.rados.rados.0.smithi125.stdout:3128: expect (ObjNum 1068 snap 357 seq_num 1068) 2021-03-24T22:06:20.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:20.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:06:20.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:20 smithi142 conmon[20125]: cluster 2021-03-24T22:06:19.001506+0000 mgr.y (mgr. 2021-03-24T22:06:20.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:20 smithi142 conmon[20125]: 14144) 679 : cluster [DBG] pgmap v1375: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 481 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 934 KiB/s wr, 5 op/s; 11/1611 objects misplaced (0.683%) 2021-03-24T22:06:20.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:20 smithi142 conmon[20125]: cluster 2021-03-24T22:06:19.072228+0000 osd.3 (osd.3) 160 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:06:20.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:20 smithi142 conmon[20125]: cluster 2021-03-24T22:06:19.077936+0000 osd.3 (osd.3) 161 : cluster [DBG] 2021-03-24T22:06:20.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:20 smithi142 conmon[20125]: purged_snaps scrub ok 2021-03-24T22:06:20.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:20 smithi142 conmon[20125]: cluster 2021-03-24T22:06:19.480738+0000 mon.a (mon.0) 2031 : cluster 2021-03-24T22:06:20.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:20 smithi142 conmon[20125]: [DBG] osdmap e809: 8 total, 8 up, 8 in 2021-03-24T22:06:20.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:20 smithi142 conmon[20125]: cluster 2021-03-24T22:06:20.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:20 smithi142 conmon[20125]: 2021-03-24T22:06:20.434525+0000 mon.a (mon.0) 2032 : cluster 2021-03-24T22:06:20.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:20 smithi142 conmon[20125]: [DBG] osdmap e810: 8 total, 8 up, 8 in 2021-03-24T22:06:20.620 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:20.672 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:20.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:20.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:06:20.721 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[19936]: cluster 2021-03-24T22:06:20.722 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[19936]: 2021-03-24T22:06 2021-03-24T22:06:20.722 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[19936]: :19.001506+0000 mgr.y (mgr.14144) 679 : cluster [DBG] pgmap v1375: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 481 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 934 KiB/s wr, 5 op/s; 11/1611 objects misplaced (0.683%) 2021-03-24T22:06:20.722 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:06:20.723 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[19936]: -03-24T22:06:19.072228+0000 osd.3 ( 2021-03-24T22:06:20.723 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[19936]: osd.3) 160 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:06:20.723 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T22:06:20.723 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[19936]: -24T22:06:19.077936+0000 osd.3 (osd.3) 2021-03-24T22:06:20.723 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[19936]: 161 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:06:20.724 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T22:06:20.724 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[19936]: 22:06:19.480738+0000 mon.a (mon.0) 2021-03-24T22:06:20.724 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[19936]: 2031 : cluster [DBG] osdmap e809: 8 total, 8 up, 8 in 2021-03-24T22:06:20.724 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T22:06:20.725 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[19936]: 22:06:20.434525+0000 mon.a (mon.0) 2021-03-24T22:06:20.725 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[19936]: 2032 : cluster [DBG] osdmap e810: 8 total, 8 up, 8 in 2021-03-24T22:06:20.725 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[24927]: cluster 2021-03-24T22:06:20.725 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[24927]: 2021-03-24T22:06:20.726 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[24927]: 2021 2021-03-24T22:06:20.726 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[24927]: - 2021-03-24T22:06:20.726 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[24927]: 03 2021-03-24T22:06:20.726 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[24927]: - 2021-03-24T22:06:20.727 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[24927]: 24 2021-03-24T22:06:20.727 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[24927]: T 2021-03-24T22:06:20.727 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[24927]: 22 2021-03-24T22:06:20.727 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[24927]: : 2021-03-24T22:06:20.728 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[24927]: 06 2021-03-24T22:06:20.728 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[24927]: : 2021-03-24T22:06:20.728 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[24927]: 19 2021-03-24T22:06:20.729 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[24927]: . 2021-03-24T22:06:20.730 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[24927]: 001506 2021-03-24T22:06:20.730 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[24927]: +0000 2021-03-24T22:06:20.730 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[24927]: 2021-03-24T22:06:20.730 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[24927]: mgr.y 2021-03-24T22:06:20.731 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[24927]: ( 2021-03-24T22:06:20.731 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[24927]: mgr 2021-03-24T22:06:20.731 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[24927]: . 2021-03-24T22:06:20.731 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[24927]: 14144 2021-03-24T22:06:20.732 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[24927]: ) 2021-03-24T22:06:20.732 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[24927]: 679 2021-03-24T22:06:20.732 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[24927]: : 2021-03-24T22:06:20.732 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[24927]: cluster 2021-03-24T22:06:20.732 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[24927]: 2021-03-24T22:06:20.733 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[24927]: [DBG] 2021-03-24T22:06:20.733 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[24927]: 2021-03-24T22:06:20.733 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[24927]: pgmap v1375: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 481 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 934 KiB/s wr, 5 op/s; 11/1611 objects misplaced (0.683%) 2021-03-24T22:06:20.733 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[24927]: 2021-03-24T22:06:20.733 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[24927]: cluster 2021-03-24T22:06:20.734 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[24927]: 2021-03-24T22:06:20.734 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[24927]: 2021 2021-03-24T22:06:20.734 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[24927]: -03-24T22 2021-03-24T22:06:20.735 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[24927]: :06:19.072228+0000 osd.3 (osd.3) 160 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:06:20.735 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T22:06:20.736 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[24927]: -24T22:06:19.077936+0000 osd.3 (osd 2021-03-24T22:06:20.736 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[24927]: .3) 161 : cluster [DBG] 2021-03-24T22:06:20.736 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[24927]: purged_snaps scrub ok 2021-03-24T22:06:20.736 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[24927]: cluster 2021-03-24T22:06 2021-03-24T22:06:20.737 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[24927]: :19.480738+0000 mon.a (mon.0) 2031 : 2021-03-24T22:06:20.737 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[24927]: cluster [DBG] osdmap e809: 8 total, 8 up, 8 in 2021-03-24T22:06:20.737 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[24927]: cluster 2021-03-24T22:06:20 2021-03-24T22:06:20.737 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[24927]: .434525+0000 mon.a (mon.0) 2032 : cluster [DBG] 2021-03-24T22:06:20.737 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:20 smithi125 conmon[24927]: osdmap e810: 8 total, 8 up, 8 in 2021-03-24T22:06:20.765 INFO:tasks.rados.rados.0.smithi125.stdout:3126: finishing write tid 1 to smithi12556470-27 2021-03-24T22:06:20.765 INFO:tasks.rados.rados.0.smithi125.stdout:3126: finishing write tid 2 to smithi12556470-27 2021-03-24T22:06:20.765 INFO:tasks.rados.rados.0.smithi125.stdout:3124: finishing copy_from to smithi12556470-47 2021-03-24T22:06:20.765 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 47 v 1438 (ObjNum 1051 snap 349 seq_num 1051) dirty exists 2021-03-24T22:06:20.766 INFO:tasks.rados.rados.0.smithi125.stdout:3126: finishing write tid 3 to smithi12556470-27 2021-03-24T22:06:20.766 INFO:tasks.rados.rados.0.smithi125.stdout:3126: finishing write tid 4 to smithi12556470-27 2021-03-24T22:06:20.766 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 27 v 1433 (ObjNum 1077 snap 361 seq_num 1077) dirty exists 2021-03-24T22:06:20.766 INFO:tasks.rados.rados.0.smithi125.stdout:3126: left oid 27 (ObjNum 1077 snap 361 seq_num 1077) 2021-03-24T22:06:20.767 INFO:tasks.rados.rados.0.smithi125.stdout:3124: finishing copy_from racing read to smithi12556470-47 2021-03-24T22:06:20.767 INFO:tasks.rados.rados.0.smithi125.stdout:3129: finishing write tid 1 to smithi12556470-41 2021-03-24T22:06:20.767 INFO:tasks.rados.rados.0.smithi125.stdout:3127: finishing copy_from to smithi12556470-46 2021-03-24T22:06:20.767 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 46 v 744 (ObjNum 1057 snap 351 seq_num 1057) dirty exists 2021-03-24T22:06:20.767 INFO:tasks.rados.rados.0.smithi125.stdout:3129: finishing write tid 2 to smithi12556470-41 2021-03-24T22:06:20.768 INFO:tasks.rados.rados.0.smithi125.stdout:3127: finishing copy_from racing read to smithi12556470-46 2021-03-24T22:06:20.768 INFO:tasks.rados.rados.0.smithi125.stdout:3129: finishing write tid 3 to smithi12556470-41 2021-03-24T22:06:20.768 INFO:tasks.rados.rados.0.smithi125.stdout:3129: finishing write tid 4 to smithi12556470-41 2021-03-24T22:06:20.768 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 41 v 1244 (ObjNum 1078 snap 361 seq_num 1078) dirty exists 2021-03-24T22:06:20.769 INFO:tasks.rados.rados.0.smithi125.stdout:3129: left oid 41 (ObjNum 1078 snap 361 seq_num 1078) 2021-03-24T22:06:20.769 INFO:tasks.rados.rados.0.smithi125.stdout:3131: finishing write tid 1 to smithi12556470-4 2021-03-24T22:06:20.769 INFO:tasks.rados.rados.0.smithi125.stdout:3130: finishing copy_from to smithi12556470-26 2021-03-24T22:06:20.769 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 26 v 1457 (ObjNum 1048 snap 347 seq_num 1048) dirty exists 2021-03-24T22:06:20.769 INFO:tasks.rados.rados.0.smithi125.stdout:3130: finishing copy_from racing read to smithi12556470-26 2021-03-24T22:06:20.770 INFO:tasks.rados.rados.0.smithi125.stdout:3131: finishing write tid 2 to smithi12556470-4 2021-03-24T22:06:20.770 INFO:tasks.rados.rados.0.smithi125.stdout:3131: finishing write tid 3 to smithi12556470-4 2021-03-24T22:06:20.770 INFO:tasks.rados.rados.0.smithi125.stdout:3131: finishing write tid 4 to smithi12556470-4 2021-03-24T22:06:20.771 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 4 v 1020 (ObjNum 1079 snap 361 seq_num 1079) dirty exists 2021-03-24T22:06:20.771 INFO:tasks.rados.rados.0.smithi125.stdout:3131: left oid 4 (ObjNum 1079 snap 361 seq_num 1079) 2021-03-24T22:06:20.777 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:20.823 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.2 dump_historic_ops 2021-03-24T22:06:20.835 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:20.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:20.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:06:20.943 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:21.002 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:21.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:21.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:06:21.107 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:21.126 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:06:21.166 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:21.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:21.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:06:21.273 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:21.333 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:21.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:21.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:06:21.439 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:21.499 INFO:tasks.rados.rados.0.smithi125.stdout:3124: done (7 left) 2021-03-24T22:06:21.499 INFO:tasks.rados.rados.0.smithi125.stdout:3126: done (6 left) 2021-03-24T22:06:21.499 INFO:tasks.rados.rados.0.smithi125.stdout:3127: done (5 left) 2021-03-24T22:06:21.500 INFO:tasks.rados.rados.0.smithi125.stdout:3128: done (4 left) 2021-03-24T22:06:21.500 INFO:tasks.rados.rados.0.smithi125.stdout:3129: done (3 left) 2021-03-24T22:06:21.500 INFO:tasks.rados.rados.0.smithi125.stdout:3130: done (2 left) 2021-03-24T22:06:21.500 INFO:tasks.rados.rados.0.smithi125.stdout:3131: done (1 left) 2021-03-24T22:06:21.500 INFO:tasks.rados.rados.0.smithi125.stdout:3132: done (0 left) 2021-03-24T22:06:21.501 INFO:tasks.rados.rados.0.smithi125.stdout:3133: read oid 25 snap 335 2021-03-24T22:06:21.501 INFO:tasks.rados.rados.0.smithi125.stdout:3133: expect (ObjNum 949 snap 314 seq_num 949) 2021-03-24T22:06:21.501 INFO:tasks.rados.rados.0.smithi125.stdout:3134: delete oid 15 current snap is 362 2021-03-24T22:06:21.503 INFO:teuthology.orchestra.run.smithi125.stderr:reweighted osd.1 to 1 (10000) 2021-03-24T22:06:21.503 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:21.505 INFO:teuthology.orchestra.run.smithi125.stderr:nodeep-scrub is set 2021-03-24T22:06:21.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:21.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:06:21.511 INFO:tasks.rados.rados.0.smithi125.stdout:3133: expect (ObjNum 949 snap 314 seq_num 949) 2021-03-24T22:06:21.566 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:21 smithi125 conmon[24927]: audit 2021-03-24T22:06:21.260575+0000 mon.a ( 2021-03-24T22:06:21.567 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:21 smithi125 conmon[24927]: mon.0) 2033 : audit [INF] from='client.? v1:172.21.15.125:0/2743030335' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2021-03-24T22:06:21.567 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:21 smithi125 conmon[24927]: audit 2021-03-24T22:06:21.344909+0000 mon.a (mon.0) 2034 : audit [INF] from='client.? v1:172.21.15.125:0/1820086005' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:06:21.567 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:21 smithi125 conmon[19936]: audit 2021-03-24T22:06:21.568 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:21 smithi125 conmon[19936]: 2021-03-24T22:06:21.568 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:21 smithi125 conmon[19936]: 2021 2021-03-24T22:06:21.568 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:21 smithi125 conmon[19936]: -03-24T22:06:21.260575+0000 mon.a (mon.0) 2033 : audit [INF] from='client.? v1:172.21.15.125:0/2743030335' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2021-03-24T22:06:21.568 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:21 smithi125 conmon[19936]: audit 2021-03-24T22:06:21.344909+0000 mon.a (mon.0) 2034 : audit [INF] from='client.? v1:172.21.15.125:0/1820086005' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:06:21.612 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:21.662 INFO:tasks.rados.rados.0.smithi125.stdout:3133: done (1 left) 2021-03-24T22:06:21.662 INFO:tasks.rados.rados.0.smithi125.stdout:3134: done (0 left) 2021-03-24T22:06:21.663 INFO:tasks.rados.rados.0.smithi125.stdout:3135: rollback oid 46 current snap is 362 2021-03-24T22:06:21.663 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 46 to 351 2021-03-24T22:06:21.663 INFO:tasks.rados.rados.0.smithi125.stdout:3136: copy_from oid 36 from oid 10 current snap is 362 2021-03-24T22:06:21.663 INFO:tasks.rados.rados.0.smithi125.stdout:3137: snap_create 2021-03-24T22:06:21.669 INFO:tasks.rados.rados.0.smithi125.stdout:3135: finishing rollback tid 0 to smithi12556470-46 2021-03-24T22:06:21.670 INFO:tasks.rados.rados.0.smithi125.stdout:3135: finishing rollback tid 1 to smithi12556470-46 2021-03-24T22:06:21.670 INFO:tasks.rados.rados.0.smithi125.stdout:3135: finishing rollback tid 2 to smithi12556470-46 2021-03-24T22:06:21.670 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 46 v 1137 (ObjNum 1046 snap 346 seq_num 1046) dirty exists 2021-03-24T22:06:21.676 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:21.679 INFO:tasks.rados.rados.0.smithi125.stdout:3136: finishing copy_from to smithi12556470-36 2021-03-24T22:06:21.679 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 36 v 1114 (ObjNum 993 snap 329 seq_num 993) dirty exists 2021-03-24T22:06:21.680 INFO:tasks.rados.rados.0.smithi125.stdout:3136: finishing copy_from racing read to smithi12556470-36 2021-03-24T22:06:21.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:21.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:21.782 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:21.795 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:21 smithi142 conmon[20125]: audit 2021-03-24T22: 2021-03-24T22:06:21.795 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:21 smithi142 conmon[20125]: 06:21.260575+0000 mon.a (mon.0) 2033 : audit [INF] from='client.? v1:172.21.15.125:0/2743030335' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2021-03-24T22:06:21.795 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:21 smithi142 conmon[20125]: audit 2021-03-24T22:06:21.344909+0000 2021-03-24T22:06:21.795 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:21 smithi142 conmon[20125]: mon.a (mon.0) 2034 : audit [INF] from='client.? v1:172.21.15.125:0/1820086005' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:06:21.836 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:21.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:21.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:06:21.942 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:22.002 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:22.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:22.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:06:22.109 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:22.166 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:22.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:22.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:06:22.273 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:22.332 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:22.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:22.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:06:22.441 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:22.503 INFO:tasks.rados.rados.0.smithi125.stdout:3135: done (2 left) 2021-03-24T22:06:22.503 INFO:tasks.rados.rados.0.smithi125.stdout:3136: done (1 left) 2021-03-24T22:06:22.503 INFO:tasks.rados.rados.0.smithi125.stdout:3137: done (0 left) 2021-03-24T22:06:22.504 INFO:tasks.rados.rados.0.smithi125.stdout:3138: read oid 1 snap -1 2021-03-24T22:06:22.504 INFO:tasks.rados.rados.0.smithi125.stdout:3138: expect (ObjNum 808 snap 265 seq_num 808) 2021-03-24T22:06:22.504 INFO:tasks.rados.rados.0.smithi125.stdout:3139: read oid 3 snap -1 2021-03-24T22:06:22.504 INFO:tasks.rados.rados.0.smithi125.stdout:3139: expect (ObjNum 961 snap 317 seq_num 961) 2021-03-24T22:06:22.505 INFO:tasks.rados.rados.0.smithi125.stdout:3140: copy_from oid 39 from oid 44 current snap is 363 2021-03-24T22:06:22.505 INFO:tasks.rados.rados.0.smithi125.stdout:3141: rollback oid 10 current snap is 363 2021-03-24T22:06:22.505 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 10 to 334 2021-03-24T22:06:22.506 INFO:tasks.rados.rados.0.smithi125.stdout:3142: read oid 48 snap 316 2021-03-24T22:06:22.506 INFO:tasks.rados.rados.0.smithi125.stdout:3142: expect (ObjNum 774 snap 251 seq_num 774) 2021-03-24T22:06:22.507 INFO:tasks.rados.rados.0.smithi125.stdout:3143: snap_remove snap 349 2021-03-24T22:06:22.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:22.510 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:22.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:06:22.612 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:22.625 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:22 smithi125 conmon[19936]: cluster 2021-03-24T22:06:21.002241+0000 mgr.y (mgr. 2021-03-24T22:06:22.625 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:22 smithi125 conmon[19936]: 14144) 680 : cluster [DBG] pgmap v1378: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 469 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.3 MiB/s wr, 15 op/s; 11/1590 objects misplaced (0.692%) 2021-03-24T22:06:22.625 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:22 smithi125 conmon[19936]: cluster 2021-03-24T22:06:21.476747+0000 mon.a (mon.0) 2035 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:06:22.626 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:22 smithi125 conmon[19936]: audit 2021-03-24T22:06:21.494010+0000 mon.a (mon 2021-03-24T22:06:22.626 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:22 smithi125 conmon[19936]: .0) 2036 : audit [INF] from='client.? v1:172.21.15.125:0/2743030335' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2021-03-24T22:06:22.626 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:22 smithi125 conmon[19936]: audit 2021-03 2021-03-24T22:06:22.626 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:22 smithi125 conmon[19936]: -24T22:06:21.494091+0000 mon.a (mon.0) 2037 : audit [INF] from='client.? v1:172.21.15.125:0/1820086005' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T22:06:22.627 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:22 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:06:22.627 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:22 smithi125 conmon[19936]: T22:06:21.494163+0000 mon.a (mon.0) 2038 : cluster [DBG] osdmap e811: 8 total, 8 up, 8 in 2021-03-24T22:06:22.627 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:22 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:06:22.627 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:22 smithi125 conmon[24927]: 06:21.002241+0000 mgr.y (mgr.14144) 680 : cluster [DBG] pgmap v1378: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 469 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.3 MiB/s wr, 15 op/s; 11/1590 objects misplaced (0.692%) 2021-03-24T22:06:22.628 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:22 smithi125 conmon[24927]: cluster 2021-03-24T22:06:21.476747+0000 2021-03-24T22:06:22.628 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:22 smithi125 conmon[24927]: mon.a (mon.0) 2035 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:06:22.628 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:22 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T22:06:22.628 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:22 smithi125 conmon[24927]: 24T22:06:21.494010+0000 mon.a (mon.0) 2036 2021-03-24T22:06:22.629 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:22 smithi125 conmon[24927]: : audit [INF] from='client.? v1:172.21.15.125:0/2743030335' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2021-03-24T22:06:22.629 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:22 smithi125 conmon[24927]: audit 2021-03-24T22:06:22.629 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:22 smithi125 conmon[24927]: 2021-03-24T22:06:21.494091+0000 mon.a (mon.0) 2037 : audit 2021-03-24T22:06:22.629 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:22 smithi125 conmon[24927]: [INF] from='client.? v1:172.21.15.125:0/1820086005' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T22:06:22.629 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:22 smithi125 conmon[24927]: cluster 2021-03-24T22:06: 2021-03-24T22:06:22.630 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:22 smithi125 conmon[24927]: 21.494163+0000 mon.a (mon.0) 2038 : cluster [DBG] osdmap e811: 8 total, 8 up, 8 in 2021-03-24T22:06:22.675 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:22.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:22.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:06:22.741 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-24T22:06:22.783 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:22.797 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:22 smithi142 conmon[20125]: cluster 2021-03-24T22:06 2021-03-24T22:06:22.798 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:22 smithi142 conmon[20125]: :21.002241+0000 mgr.y (mgr.14144) 680 : cluster [DBG] pgmap v1378: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 469 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.3 MiB/s wr, 15 op/s; 11/1590 objects misplaced (0.692%) 2021-03-24T22:06:22.798 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:22 smithi142 conmon[20125]: cluster 2021-03-24T22:06: 2021-03-24T22:06:22.798 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:22 smithi142 conmon[20125]: 21.476747+0000 mon.a (mon.0) 2035 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:06:22.799 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:22 smithi142 conmon[20125]: audit 2021-03-24 2021-03-24T22:06:22.799 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:22 smithi142 conmon[20125]: T22:06:21.494010+0000 mon.a (mon.0) 2036 : audit 2021-03-24T22:06:22.799 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:22 smithi142 conmon[20125]: [INF] from='client.? v1:172.21.15.125:0/2743030335' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2021-03-24T22:06:22.799 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:22 smithi142 conmon[20125]: audit 2021-03-24T22 2021-03-24T22:06:22.800 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:22 smithi142 conmon[20125]: :06:21.494091+0000 mon.a (mon.0) 2037 2021-03-24T22:06:22.800 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:22 smithi142 conmon[20125]: : audit [INF] from='client.? v1:172.21.15.125:0/1820086005' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T22:06:22.800 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:22 smithi142 conmon[20125]: cluster 2021-03-24T22:06:22.800 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:22 smithi142 conmon[20125]: 2021-03-24T22:06:21.494163+0000 mon.a (mon.0) 2038 : cluster 2021-03-24T22:06:22.801 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:22 smithi142 conmon[20125]: [DBG] osdmap e811: 8 total, 8 up, 8 in 2021-03-24T22:06:22.837 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:22.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:22.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:06:22.943 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:22.993 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:22.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:22.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:06:23.097 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:23.151 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:23.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:23.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:06:23.257 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:23.319 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:23.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:23.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:06:23.424 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:23.484 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:23.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:23.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:06:23.511 INFO:tasks.rados.rados.0.smithi125.stdout:3143: done (5 left) 2021-03-24T22:06:23.511 INFO:tasks.rados.rados.0.smithi125.stdout:3144: delete oid 31 current snap is 363 2021-03-24T22:06:23.512 INFO:tasks.rados.rados.0.smithi125.stdout:3139: expect (ObjNum 961 snap 317 seq_num 961) 2021-03-24T22:06:23.591 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:23.621 INFO:tasks.rados.rados.0.smithi125.stdout:3138: expect (ObjNum 808 snap 265 seq_num 808) 2021-03-24T22:06:23.649 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:23.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:23.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:06:23.753 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:23.767 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:23 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:06:23.768 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:23 smithi142 conmon[20125]: T22:06:22.498737+0000 mon.a (mon.0) 2039 : cluster [DBG] osdmap e812: 8 total, 8 up, 8 in 2021-03-24T22:06:23.768 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:23 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:06:23.768 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:23 smithi142 conmon[20125]: -03-24T22:06: 2021-03-24T22:06:23.769 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:23 smithi142 conmon[20125]: 23.418525+0000 osd.6 ( 2021-03-24T22:06:23.769 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:23 smithi142 conmon[20125]: osd.6) 82 : cluster 2021-03-24T22:06:23.769 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:23 smithi142 conmon[20125]: [DBG] purged_snaps scrub starts 2021-03-24T22:06:23.769 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:23 smithi142 conmon[20125]: cluster 2021-03-24T22:06:23.769 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:23 smithi142 conmon[20125]: 2021-03-24T22 2021-03-24T22:06:23.770 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:23 smithi142 conmon[20125]: :06:23.420832+0000 2021-03-24T22:06:23.771 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:23 smithi142 conmon[20125]: osd.6 (osd.6) 83 2021-03-24T22:06:23.771 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:23 smithi142 conmon[20125]: : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:06:23.777 INFO:tasks.rados.rados.0.smithi125.stdout:3141: finishing rollback tid 0 to smithi12556470-10 2021-03-24T22:06:23.777 INFO:tasks.rados.rados.0.smithi125.stdout:3141: finishing rollback tid 1 to smithi12556470-10 2021-03-24T22:06:23.778 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 10 v 1459 (ObjNum 1117615696 snap 21916 seq_num 1116474416) dirty dne 2021-03-24T22:06:23.778 INFO:tasks.rados.rados.0.smithi125.stdout:3142: expect (ObjNum 774 snap 251 seq_num 774) 2021-03-24T22:06:23.812 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:23.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:23.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:23.875 INFO:tasks.rados.rados.0.smithi125.stdout:3140: finishing copy_from to smithi12556470-39 2021-03-24T22:06:23.875 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 39 v 1438 (ObjNum 1051 snap 349 seq_num 1051) dirty exists 2021-03-24T22:06:23.876 INFO:tasks.rados.rados.0.smithi125.stdout:3140: finishing copy_from racing read to smithi12556470-39 2021-03-24T22:06:23.876 INFO:tasks.rados.rados.0.smithi125.stdout:3138: done (5 left) 2021-03-24T22:06:23.876 INFO:tasks.rados.rados.0.smithi125.stdout:3139: done (4 left) 2021-03-24T22:06:23.876 INFO:tasks.rados.rados.0.smithi125.stdout:3140: done (3 left) 2021-03-24T22:06:23.877 INFO:tasks.rados.rados.0.smithi125.stdout:3141: done (2 left) 2021-03-24T22:06:23.877 INFO:tasks.rados.rados.0.smithi125.stdout:3142: done (1 left) 2021-03-24T22:06:23.877 INFO:tasks.rados.rados.0.smithi125.stdout:3144: done (0 left) 2021-03-24T22:06:23.877 INFO:tasks.rados.rados.0.smithi125.stdout:3145: read oid 3 snap 336 2021-03-24T22:06:23.878 INFO:tasks.rados.rados.0.smithi125.stdout:3145: expect deleted 2021-03-24T22:06:23.878 INFO:tasks.rados.rados.0.smithi125.stdout:3146: copy_from oid 24 from oid 2 current snap is 363 2021-03-24T22:06:23.878 INFO:tasks.rados.rados.0.smithi125.stdout:3147: rollback oid 21 current snap is 363 2021-03-24T22:06:23.878 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 21 to 360 2021-03-24T22:06:23.879 INFO:tasks.rados.rados.0.smithi125.stdout:3148: copy_from oid 18 from oid 38 current snap is 363 2021-03-24T22:06:23.879 INFO:tasks.rados.rados.0.smithi125.stdout:3149: rollback oid 29 current snap is 363 2021-03-24T22:06:23.879 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 29 to 344 2021-03-24T22:06:23.879 INFO:tasks.rados.rados.0.smithi125.stdout:3150: rollback oid 25 current snap is 363 2021-03-24T22:06:23.879 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 25 to 353 2021-03-24T22:06:23.880 INFO:tasks.rados.rados.0.smithi125.stdout:3151: delete oid 40 current snap is 363 2021-03-24T22:06:23.880 INFO:tasks.rados.rados.0.smithi125.stdout:3149: finishing rollback tid 1 to smithi12556470-29 2021-03-24T22:06:23.881 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 29 v 515 (ObjNum 288 snap 0 seq_num 49) dirty dne 2021-03-24T22:06:23.882 INFO:tasks.rados.rados.0.smithi125.stdout:3150: finishing rollback tid 0 to smithi12556470-25 2021-03-24T22:06:23.882 INFO:tasks.rados.rados.0.smithi125.stdout:3147: finishing rollback tid 0 to smithi12556470-21 2021-03-24T22:06:23.882 INFO:tasks.rados.rados.0.smithi125.stdout:3145: done (6 left) 2021-03-24T22:06:23.882 INFO:tasks.rados.rados.0.smithi125.stdout:3149: done (5 left) 2021-03-24T22:06:23.883 INFO:tasks.rados.rados.0.smithi125.stdout:3151: done (4 left) 2021-03-24T22:06:23.883 INFO:tasks.rados.rados.0.smithi125.stdout:3152: copy_from oid 28 from oid 23 current snap is 363 2021-03-24T22:06:23.883 INFO:tasks.rados.rados.0.smithi125.stdout:3150: finishing rollback tid 1 to smithi12556470-25 2021-03-24T22:06:23.883 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 25 v 1452 (ObjNum 1117562832 snap 21916 seq_num 3308612768) dirty dne 2021-03-24T22:06:23.883 INFO:tasks.rados.rados.0.smithi125.stdout:3150: done (4 left) 2021-03-24T22:06:23.884 INFO:tasks.rados.rados.0.smithi125.stdout:3153: snap_create 2021-03-24T22:06:23.885 INFO:tasks.rados.rados.0.smithi125.stdout:3152: finishing copy_from to smithi12556470-28 2021-03-24T22:06:23.885 INFO:tasks.rados.rados.0.smithi125.stdout:3152: got expected ENOENT (src dne) 2021-03-24T22:06:23.885 INFO:tasks.rados.rados.0.smithi125.stdout:3152: finishing copy_from racing read to smithi12556470-28 2021-03-24T22:06:23.885 INFO:tasks.rados.rados.0.smithi125.stdout:3147: finishing rollback tid 1 to smithi12556470-21 2021-03-24T22:06:23.886 INFO:tasks.rados.rados.0.smithi125.stdout:3147: finishing rollback tid 2 to smithi12556470-21 2021-03-24T22:06:23.886 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 21 v 1441 (ObjNum 851 snap 278 seq_num 851) dirty exists 2021-03-24T22:06:23.904 INFO:tasks.rados.rados.0.smithi125.stdout:3146: finishing copy_from to smithi12556470-24 2021-03-24T22:06:23.904 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 24 v 1017 (ObjNum 984 snap 326 seq_num 984) dirty exists 2021-03-24T22:06:23.904 INFO:tasks.rados.rados.0.smithi125.stdout:3148: finishing copy_from to smithi12556470-18 2021-03-24T22:06:23.905 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 18 v 744 (ObjNum 1057 snap 351 seq_num 1057) dirty exists 2021-03-24T22:06:23.905 INFO:tasks.rados.rados.0.smithi125.stdout:3146: finishing copy_from racing read to smithi12556470-24 2021-03-24T22:06:23.905 INFO:tasks.rados.rados.0.smithi125.stdout:3148: finishing copy_from racing read to smithi12556470-18 2021-03-24T22:06:23.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:23 smithi125 conmon[19936]: cluster 2021-03-24T22:06:22 2021-03-24T22:06:23.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:23 smithi125 conmon[19936]: .498737+0000 mon.a (mon.0) 2039 : cluster [DBG] osdmap e812: 8 total, 8 up, 8 in 2021-03-24T22:06:23.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:23 smithi125 conmon[19936]: cluster 2021-03-24T22:06:23.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:23 smithi125 conmon[19936]: 2021-03-24 2021-03-24T22:06:23.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:23 smithi125 conmon[19936]: T22:06:23.418525+0000 osd.6 (osd.6) 82 : cluster 2021-03-24T22:06:23.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:23 smithi125 conmon[19936]: [DBG] purged_snaps scrub starts 2021-03-24T22:06:23.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:23 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:06:23.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:23 smithi125 conmon[19936]: -03-24T22:06:23.420832+0000 osd.6 (osd 2021-03-24T22:06:23.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:23 smithi125 conmon[19936]: .6) 83 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:06:23.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:23 smithi125 conmon[24927]: cluster 2021-03-24T22:06:22 2021-03-24T22:06:23.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:23 smithi125 conmon[24927]: .498737+0000 mon.a (mon.0) 2039 : cluster [DBG] osdmap e812: 8 total, 8 up, 8 in 2021-03-24T22:06:23.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:23 smithi125 conmon[24927]: cluster 2021-03-24T22:06:23.418525+0000 2021-03-24T22:06:23.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:23 smithi125 conmon[24927]: osd.6 (osd.6) 82 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:06:23.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:23 smithi125 conmon[24927]: cluster 2021-03-24T22:06 2021-03-24T22:06:23.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:23 smithi125 conmon[24927]: :23.420832+0000 osd.6 (osd.6) 83 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:06:23.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:23 smithi125 conmon[24927]: 2021-03-24T22:06:23.919 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:23.974 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:23.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:23.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:06:24.080 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:24.144 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:24.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:24.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:06:24.250 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:24.309 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:24.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:24.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:06:24.415 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:24.471 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:24.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:24.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:06:24.520 INFO:tasks.rados.rados.0.smithi125.stdout:3146: done (4 left) 2021-03-24T22:06:24.521 INFO:tasks.rados.rados.0.smithi125.stdout:3147: done (3 left) 2021-03-24T22:06:24.521 INFO:tasks.rados.rados.0.smithi125.stdout:3148: done (2 left) 2021-03-24T22:06:24.521 INFO:tasks.rados.rados.0.smithi125.stdout:3152: done (1 left) 2021-03-24T22:06:24.522 INFO:tasks.rados.rados.0.smithi125.stdout:3153: done (0 left) 2021-03-24T22:06:24.522 INFO:tasks.rados.rados.0.smithi125.stdout:3154: copy_from oid 36 from oid 27 current snap is 364 2021-03-24T22:06:24.523 INFO:tasks.rados.rados.0.smithi125.stdout:3155: read oid 7 snap -1 2021-03-24T22:06:24.523 INFO:tasks.rados.rados.0.smithi125.stdout:3155: expect (ObjNum 1013 snap 336 seq_num 1013) 2021-03-24T22:06:24.523 INFO:tasks.rados.rados.0.smithi125.stdout:3156: copy_from oid 32 from oid 41 current snap is 364 2021-03-24T22:06:24.524 INFO:tasks.rados.rados.0.smithi125.stdout:3157: snap_remove snap 362 2021-03-24T22:06:24.577 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:24.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:24 smithi142 conmon[20125]: cluster 2021-03-24T22:06:23. 2021-03-24T22:06:24.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:24 smithi142 conmon[20125]: 002742+0000 mgr.y (mgr.14144) 681 : cluster [DBG] pgmap v1381: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 469 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 2.3 MiB/s wr, 11 op/s; 11/1590 objects misplaced (0.692%) 2021-03-24T22:06:24.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:24 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:06:24.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:24 smithi142 conmon[20125]: 24T22:06:23.508205+0000 mon.a (mon.0) 2040 : cluster [DBG] osdmap e813: 8 total, 8 up, 8 in 2021-03-24T22:06:24.618 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:06:24.628 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2021-03-24T22:06:24.629 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph pg dump --format=json 2021-03-24T22:06:24.634 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:24.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:24.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:06:24.740 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:24.752 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:24 smithi125 conmon[19936]: cluster 2021-03-24T22:06:23.002742+0000 mgr.y (mgr.14144) 681 : 2021-03-24T22:06:24.753 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:24 smithi125 conmon[19936]: cluster [DBG] pgmap v1381: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 469 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 2.3 MiB/s wr, 11 op/s; 11/1590 objects misplaced (0.692%) 2021-03-24T22:06:24.753 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:24 smithi125 conmon[19936]: cluster 2021-03-24T22:06:23.508205+0000 mon.a (mon.0) 2040 : cluster [DBG] osdmap e813: 8 total, 8 up, 8 in 2021-03-24T22:06:24.753 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:24 smithi125 conmon[24927]: cluster 2021-03-24T22:06:23 2021-03-24T22:06:24.753 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:24 smithi125 conmon[24927]: .002742+0000 mgr.y (mgr.14144) 681 : cluster [DBG] pgmap v1381: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 469 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 2.3 MiB/s wr, 11 op/s; 11/1590 objects misplaced (0.692%) 2021-03-24T22:06:24.754 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:24 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T22:06:24.754 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:24 smithi125 conmon[24927]: -24T22:06:23. 2021-03-24T22:06:24.754 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:24 smithi125 conmon[24927]: 508205+0000 mon.a (mon 2021-03-24T22:06:24.754 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:24 smithi125 conmon[24927]: .0) 2040 : cluster 2021-03-24T22:06:24.755 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:24 smithi125 conmon[24927]: [DBG] osdmap e813: 8 total, 8 up, 8 in 2021-03-24T22:06:24.792 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:24.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:24.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:06:24.898 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:24.949 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:24.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:24.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:06:25.062 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:25.097 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-24T22:06:25.097 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T22:06:25.120 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:25.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:25.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:06:25.227 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:25.281 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:25.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:25.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:06:25.388 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:25.437 INFO:tasks.rados.rados.0.smithi125.stdout:3157: done (3 left) 2021-03-24T22:06:25.437 INFO:tasks.rados.rados.0.smithi125.stdout:3158: rollback oid 11 current snap is 364 2021-03-24T22:06:25.437 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 11 to 358 2021-03-24T22:06:25.438 INFO:tasks.rados.rados.0.smithi125.stdout:3159: snap_remove snap 359 2021-03-24T22:06:25.448 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:25.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:25.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:06:25.557 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:25.613 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:25.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:25.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:06:25.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:25 smithi125 conmon[19936]: cluster 2021-03-24T22:06: 2021-03-24T22:06:25.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:25 smithi125 conmon[19936]: 24.514290+0000 mon.a (mon.0) 2041 : cluster [DBG] osdmap e814: 8 total, 8 up, 8 in 2021-03-24T22:06:25.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:25 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:06:25.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:25 smithi125 conmon[19936]: T22:06:25.432209+0000 mon.a (mon.0) 2042 : cluster [DBG] osdmap e815: 8 total, 8 up, 8 in 2021-03-24T22:06:25.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:25 smithi125 conmon[24927]: cluster 2021-03-24T22:06:24 2021-03-24T22:06:25.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:25 smithi125 conmon[24927]: .514290+0000 mon.a (mon.0) 2041 : cluster [DBG] osdmap e814: 8 total, 8 up, 8 in 2021-03-24T22:06:25.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:25 smithi125 conmon[24927]: cluster 2021-03-24T22:06:25.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:25 smithi125 conmon[24927]: 2021-03-24T22:06:25.432209+0000 mon.a (mon.0) 2042 : cluster 2021-03-24T22:06:25.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:25 smithi125 conmon[24927]: [DBG] osdmap e815: 8 total, 8 up, 8 in 2021-03-24T22:06:25.720 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:25.778 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:25.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:25.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:06:25.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:25 smithi142 conmon[20125]: cluster 2021-03-24T22:06 2021-03-24T22:06:25.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:25 smithi142 conmon[20125]: :24.514290+0000 mon.a (mon.0) 2041 : cluster [DBG] osdmap e814: 8 total, 8 up, 8 in 2021-03-24T22:06:25.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:25 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:06:25.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:25 smithi142 conmon[20125]: T22:06:25.432209+0000 mon.a (mon.0) 2042 : cluster [DBG] osdmap e815: 8 total, 8 up, 8 in 2021-03-24T22:06:25.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:25 smithi142 conmon[20125]: 2021-03-24T22:06:25.886 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:25.947 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:25.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:25.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:06:26.054 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:26.110 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:26.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:26.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:06:26.218 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:26.273 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:26.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:26.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:06:26.380 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:26.443 INFO:tasks.rados.rados.0.smithi125.stdout:3159: done (4 left) 2021-03-24T22:06:26.443 INFO:tasks.rados.rados.0.smithi125.stdout:3160: snap_create 2021-03-24T22:06:26.443 INFO:tasks.rados.rados.0.smithi125.stdout:3155: expect (ObjNum 1013 snap 336 seq_num 1013) 2021-03-24T22:06:26.444 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:26.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:26.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:06:26.527 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:26 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:06:26.527 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:26 smithi125 conmon[24927]: 24T22:06:25.003284+0000 mgr.y (mgr.14144) 2021-03-24T22:06:26.527 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:26 smithi125 conmon[24927]: 682 : cluster [DBG] pgmap v1384: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 492 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 4.8 MiB/s wr, 18 op/s; 11/1626 objects misplaced (0.677%) 2021-03-24T22:06:26.528 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:26 smithi125 conmon[24927]: cluster 2021-03-24T22:06:26.528 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:26 smithi125 conmon[24927]: 2021-03-24T22 2021-03-24T22:06:26.528 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:26 smithi125 conmon[24927]: :06:26.433518 2021-03-24T22:06:26.552 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:26.577 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd unset noscrub 2021-03-24T22:06:26.585 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:26 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:06:26.586 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:26 smithi125 conmon[19936]: 06:25.003284+0000 mgr.y (mgr.14144) 682 : cluster [DBG] pgmap v1384: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 492 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 4.8 MiB/s wr, 18 op/s; 11/1626 objects misplaced (0.677%) 2021-03-24T22:06:26.586 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:26 smithi125 conmon[19936]: cluster 2021-03-24T22:06:26.433518+0000 2021-03-24T22:06:26.587 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:26 smithi125 conmon[19936]: mon.a (mon.0) 2043 : cluster [DBG] osdmap e816: 8 total, 8 up, 8 in 2021-03-24T22:06:26.587 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:26 smithi125 conmon[24927]: +0000 mon.a (mon. 2021-03-24T22:06:26.587 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:26 smithi125 conmon[24927]: 0) 2043 : cluster 2021-03-24T22:06:26.588 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:26 smithi125 conmon[24927]: [DBG] osdmap e816: 8 total, 8 up, 8 in 2021-03-24T22:06:26.588 INFO:tasks.rados.rados.0.smithi125.stdout:3156: finishing copy_from to smithi12556470-32 2021-03-24T22:06:26.588 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 32 v 1244 (ObjNum 1078 snap 361 seq_num 1078) dirty exists 2021-03-24T22:06:26.589 INFO:tasks.rados.rados.0.smithi125.stdout:3156: finishing copy_from racing read to smithi12556470-32 2021-03-24T22:06:26.589 INFO:tasks.rados.rados.0.smithi125.stdout:3154: finishing copy_from to smithi12556470-36 2021-03-24T22:06:26.589 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 36 v 1433 (ObjNum 1077 snap 361 seq_num 1077) dirty exists 2021-03-24T22:06:26.589 INFO:tasks.rados.rados.0.smithi125.stdout:3154: finishing copy_from racing read to smithi12556470-36 2021-03-24T22:06:26.590 INFO:tasks.rados.rados.0.smithi125.stdout:3158: finishing rollback tid 0 to smithi12556470-11 2021-03-24T22:06:26.590 INFO:tasks.rados.rados.0.smithi125.stdout:3158: finishing rollback tid 1 to smithi12556470-11 2021-03-24T22:06:26.590 INFO:tasks.rados.rados.0.smithi125.stdout:3158: finishing rollback tid 2 to smithi12556470-11 2021-03-24T22:06:26.590 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 11 v 573 (ObjNum 1069 snap 358 seq_num 1069) dirty exists 2021-03-24T22:06:26.596 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:26.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:26.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:06:26.700 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:26.714 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:26 smithi142 conmon[20125]: cluster 2021-03-24T22:06: 2021-03-24T22:06:26.714 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:26 smithi142 conmon[20125]: 25.003284+0000 mgr.y (mgr.14144) 682 : cluster [DBG] pgmap v1384: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 492 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 4.8 MiB/s wr, 18 op/s; 11/1626 objects misplaced (0.677%) 2021-03-24T22:06:26.714 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:26 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:06:26.715 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:26 smithi142 conmon[20125]: T22:06:26.433518+0000 mon.a (mon.0) 2043 : cluster [DBG] osdmap e816: 8 total, 8 up, 8 in 2021-03-24T22:06:26.756 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:26.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:26.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:06:26.863 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:26.915 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:26.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:26.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:06:27.022 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:27.076 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:27.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:27.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:06:27.183 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:27.238 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:27.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:27.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:27.345 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:27.353 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:06:27.378 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.3 dump_ops_in_flight 2021-03-24T22:06:27.407 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:27.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:27.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:06:27.439 INFO:tasks.rados.rados.0.smithi125.stdout:3154: done (4 left) 2021-03-24T22:06:27.439 INFO:tasks.rados.rados.0.smithi125.stdout:3155: done (3 left) 2021-03-24T22:06:27.440 INFO:tasks.rados.rados.0.smithi125.stdout:3156: done (2 left) 2021-03-24T22:06:27.440 INFO:tasks.rados.rados.0.smithi125.stdout:3158: done (1 left) 2021-03-24T22:06:27.440 INFO:tasks.rados.rados.0.smithi125.stdout:3160: done (0 left) 2021-03-24T22:06:27.441 INFO:tasks.rados.rados.0.smithi125.stdout:3161: snap_create 2021-03-24T22:06:27.451 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:06:27.512 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:27.570 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:27.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:27.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:27.676 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:27.727 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:27.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:27.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:06:27.832 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:27.883 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:27.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:27.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:06:27.989 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:28.042 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:28.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:28.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:06:28.149 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:28.206 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:28.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:28.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:06:28.312 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:28.370 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:28.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:28.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:06:28.445 INFO:tasks.rados.rados.0.smithi125.stdout:3161: done (0 left) 2021-03-24T22:06:28.445 INFO:tasks.rados.rados.0.smithi125.stdout:3162: snap_remove snap 357 2021-03-24T22:06:28.476 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:28.527 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:28.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:28.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:06:28.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:28 smithi142 conmon[20125]: cluster 2021-03-24T22:06:27.004146+0000 mgr.y ( 2021-03-24T22:06:28.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:28 smithi142 conmon[20125]: mgr.14144) 683 : cluster [DBG] pgmap v1387: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 501 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 6.8 MiB/s wr, 24 op/s; 11/1635 objects misplaced (0.673%) 2021-03-24T22:06:28.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:28 smithi142 conmon[20125]: cluster 2021-03-24T22:06:27.431983+0000 mon.a (mon.0) 2044 : cluster [DBG] osdmap e817: 8 total, 8 up, 8 in 2021-03-24T22:06:28.635 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:28.696 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:28.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:28.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:06:28.709 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:28 smithi125 conmon[19936]: cluster 2021-03-24T22:06:27.004146+0000 mgr.y (mgr.14144) 683 : cluster 2021-03-24T22:06:28.709 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:28 smithi125 conmon[19936]: [DBG] pgmap v1387: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 501 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 6.8 MiB/s wr, 24 op/s; 11/1635 objects misplaced (0.673%) 2021-03-24T22:06:28.710 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:28 smithi125 conmon[19936]: cluster 2021-03-24T22:06:27.431983+0000 mon.a (mon.0) 2044 : cluster [DBG] osdmap e817: 8 total, 8 up, 8 in 2021-03-24T22:06:28.710 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:28 smithi125 conmon[24927]: cluster 2021-03-24T22:06:27.004146+0000 mgr.y 2021-03-24T22:06:28.710 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:28 smithi125 conmon[24927]: (mgr.14144) 683 : cluster [DBG] pgmap v1387: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 501 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 6.8 MiB/s wr, 24 op/s; 11/1635 objects misplaced (0.673%) 2021-03-24T22:06:28.711 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:28 smithi125 conmon[24927]: cluster 2021-03-24T22:06:27.431983 2021-03-24T22:06:28.711 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:28 smithi125 conmon[24927]: +0000 mon.a (mon.0) 2044 : cluster [DBG] osdmap e817: 8 total, 8 up, 8 in 2021-03-24T22:06:28.802 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:28.862 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:28.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:28.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:06:28.967 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:28.977 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:06:29.026 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:29.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:29.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:06:29.133 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:29.181 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T22:06:29.181 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T22:06:29.181 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T22:06:29.182 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:29.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:29.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:06:29.192 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T22:06:29.218 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T22:06:29.241 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T22:06:29.270 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T22:06:29.287 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:29.301 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T22:06:29.303 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-24T22:06:29.303 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-24T22:06:29.303 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-24T22:06:29.304 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-24T22:06:29.304 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-24T22:06:29.304 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-24T22:06:29.304 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-24T22:06:29.304 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-24T22:06:29.328 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:29.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:29.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:06:29.434 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:29.442 INFO:tasks.rados.rados.0.smithi125.stdout:3162: done (0 left) 2021-03-24T22:06:29.442 INFO:tasks.rados.rados.0.smithi125.stdout:3163: read oid 2 snap -1 2021-03-24T22:06:29.442 INFO:tasks.rados.rados.0.smithi125.stdout:3163: expect (ObjNum 984 snap 326 seq_num 984) 2021-03-24T22:06:29.443 INFO:tasks.rados.rados.0.smithi125.stdout:3164: rollback oid 46 current snap is 366 2021-03-24T22:06:29.443 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 46 to 354 2021-03-24T22:06:29.443 INFO:tasks.rados.rados.0.smithi125.stdout:3165: delete oid 15 current snap is 366 2021-03-24T22:06:29.469 INFO:tasks.rados.rados.0.smithi125.stdout:3165: done (2 left) 2021-03-24T22:06:29.469 INFO:tasks.rados.rados.0.smithi125.stdout:3166: read oid 35 snap -1 2021-03-24T22:06:29.470 INFO:tasks.rados.rados.0.smithi125.stdout:3166: expect (ObjNum 1071 snap 359 seq_num 1071) 2021-03-24T22:06:29.470 INFO:tasks.rados.rados.0.smithi125.stdout:3167: snap_remove snap 317 2021-03-24T22:06:29.491 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:29.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:29.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:06:29.595 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:29.609 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:29 smithi125 conmon[19936]: cluster 2021-03-24T22:06:28.438619+0000 mon.a (mon.0) 2045 2021-03-24T22:06:29.610 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:29 smithi125 conmon[19936]: : cluster [DBG] osdmap e818: 8 total, 8 up, 8 in 2021-03-24T22:06:29.610 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:29 smithi125 conmon[19936]: audit 2021-03-24T22:06:29.094900+0000 mon.c (mon.2) 117 : audit [DBG] from='client.? v1:172.21.15.125:0/117482075' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:06:29.610 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:29 smithi125 conmon[19936]: cluster 2021-03-24T22:06:29.438610+0000 mon.a (mon.0) 2046 : cluster [DBG] osdmap e819: 8 total, 8 up, 8 in 2021-03-24T22:06:29.611 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:29 smithi125 conmon[24927]: cluster 2021-03-24T22:06:28.438619+0000 2021-03-24T22:06:29.611 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:29 smithi125 conmon[24927]: mon.a (mon.0) 2045 : cluster [DBG] osdmap e818: 8 total, 8 up, 8 in 2021-03-24T22:06:29.611 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:29 smithi125 conmon[24927]: audit 2021-03-24T22:06:29 2021-03-24T22:06:29.611 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:29 smithi125 conmon[24927]: .094900+0000 mon.c (mon.2) 117 : audit [DBG] from='client.? v1:172.21.15.125:0/117482075' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:06:29.612 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:29 smithi125 conmon[24927]: cluster 2021-03-24T22:06:29 2021-03-24T22:06:29.612 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:29 smithi125 conmon[24927]: .438610+0000 mon.a (mon.0) 2046 : cluster [DBG] 2021-03-24T22:06:29.612 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:29 smithi125 conmon[24927]: osdmap e819: 8 total, 8 up, 8 in 2021-03-24T22:06:29.653 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:29.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:29.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:06:29.758 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:29.772 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:29 smithi142 conmon[20125]: cluster 2021-03-24T22:06:28.438619+0000 mon.a ( 2021-03-24T22:06:29.772 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:29 smithi142 conmon[20125]: mon.0) 2045 : cluster [DBG] osdmap e818: 8 total, 8 up, 8 in 2021-03-24T22:06:29.772 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:29 smithi142 conmon[20125]: audit 2021-03-24T22:06:29.094900+0000 mon.c (mon.2) 117 : audit [DBG] from='client.? v1:172.21.15.125:0/117482075' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:06:29.772 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:29 smithi142 conmon[20125]: cluster 2021-03-24T22:06:29.438610+0000 mon.a (mon.0) 2046 : cluster [DBG] 2021-03-24T22:06:29.773 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:29 smithi142 conmon[20125]: osdmap e819: 8 total, 8 up, 8 in 2021-03-24T22:06:29.813 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:29.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:29.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:06:29.920 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:29.979 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:29.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:29.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:06:30.086 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:30.141 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:30.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:30.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:06:30.247 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:30.304 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:30.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:30.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:06:30.411 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:30.446 INFO:tasks.rados.rados.0.smithi125.stdout:3167: done (3 left) 2021-03-24T22:06:30.446 INFO:tasks.rados.rados.0.smithi125.stdout:3168: snap_remove snap 341 2021-03-24T22:06:30.489 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:30.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:30.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:06:30.596 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:30.610 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:30 smithi125 conmon[19936]: cluster 2021-03-24T22:06:29.005029+0000 mgr.y (mgr.14144) 684 : cluster [DBG] pgmap v1390: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 503 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 5.5 MiB/s rd, 2.5 MiB/s wr, 6 op/s; 11/1638 objects misplaced (0.672%) 2021-03-24T22:06:30.610 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:30 smithi125 conmon[19936]: cluster 2021-03-24T22:06:30.436053+0000 mon.a (mon.0) 2047 : cluster [DBG] osdmap e820: 8 total, 8 up, 8 in 2021-03-24T22:06:30.611 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:30 smithi125 conmon[24927]: cluster 2021-03-24T22:06:29.005029+0000 mgr.y (mgr.14144) 684 : cluster [DBG] pgmap v1390: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 503 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 5.5 MiB/s rd, 2.5 MiB/s wr, 6 op/s; 11/1638 objects misplaced (0.672%) 2021-03-24T22:06:30.611 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:30 smithi125 conmon[24927]: cluster 2021-03-24T22:06:30.436053+0000 mon.a (mon.0) 2047 : cluster [DBG] osdmap e820: 8 total, 8 up, 8 in 2021-03-24T22:06:30.616 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:30 smithi142 conmon[20125]: cluster 2021-03-24T22:06:29.005029+0000 mgr.y (mgr.14144 2021-03-24T22:06:30.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:30 smithi142 conmon[20125]: ) 684 : cluster [DBG] pgmap v1390: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 503 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 5.5 MiB/s rd, 2.5 MiB/s wr, 6 op/s; 11/1638 objects misplaced (0.672%) 2021-03-24T22:06:30.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:30 smithi142 conmon[20125]: cluster 2021-03-24T22:06:30.436053+0000 mon.a (mon.0) 2047 : cluster [DBG] osdmap e820: 8 total, 8 up, 8 in 2021-03-24T22:06:30.658 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:30.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:30.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:06:30.766 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:30.830 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:30.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:30.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:06:30.934 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:30.989 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:30.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:30.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:06:31.095 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:31.151 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:31.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:31.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:06:31.256 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:31.313 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:31.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:31.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:06:31.421 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:31.440 INFO:tasks.rados.rados.0.smithi125.stdout:3168: done (3 left) 2021-03-24T22:06:31.440 INFO:tasks.rados.rados.0.smithi125.stdout:3169: write (excl) oid 26 current snap is 366 2021-03-24T22:06:31.441 INFO:tasks.rados.rados.0.smithi125.stdout:3169: seq_num 1084 ranges {618788=492042,1883069=335719} 2021-03-24T22:06:31.447 INFO:tasks.rados.rados.0.smithi125.stdout:3169: writing smithi12556470-26 from 618788 to 1110830 tid 1 2021-03-24T22:06:31.454 INFO:tasks.rados.rados.0.smithi125.stdout:3169: writing smithi12556470-26 from 1883069 to 2218788 tid 2 2021-03-24T22:06:31.454 INFO:tasks.rados.rados.0.smithi125.stdout:3170: delete oid 40 current snap is 366 2021-03-24T22:06:31.455 INFO:tasks.rados.rados.0.smithi125.stdout:3163: expect (ObjNum 984 snap 326 seq_num 984) 2021-03-24T22:06:31.485 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:31.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:31.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:06:31.590 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:31.625 INFO:tasks.rados.rados.0.smithi125.stdout:3163: done (4 left) 2021-03-24T22:06:31.625 INFO:tasks.rados.rados.0.smithi125.stdout:3170: done (3 left) 2021-03-24T22:06:31.626 INFO:tasks.rados.rados.0.smithi125.stdout:3171: read oid 9 snap -1 2021-03-24T22:06:31.626 INFO:tasks.rados.rados.0.smithi125.stdout:3171: expect deleted 2021-03-24T22:06:31.626 INFO:tasks.rados.rados.0.smithi125.stdout:3172: write (excl) oid 15 current snap is 366 2021-03-24T22:06:31.626 INFO:tasks.rados.rados.0.smithi125.stdout:3164: finishing rollback tid 0 to smithi12556470-46 2021-03-24T22:06:31.627 INFO:tasks.rados.rados.0.smithi125.stdout:3164: finishing rollback tid 1 to smithi12556470-46 2021-03-24T22:06:31.627 INFO:tasks.rados.rados.0.smithi125.stdout:3164: finishing rollback tid 2 to smithi12556470-46 2021-03-24T22:06:31.627 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 46 v 1140 (ObjNum 1046 snap 346 seq_num 1046) dirty exists 2021-03-24T22:06:31.627 INFO:tasks.rados.rados.0.smithi125.stdout:3172: seq_num 1086 ranges {652402=642540,1883680=368722} 2021-03-24T22:06:31.632 INFO:tasks.rados.rados.0.smithi125.stdout:3172: writing smithi12556470-15 from 652402 to 1294942 tid 1 2021-03-24T22:06:31.638 INFO:tasks.rados.rados.0.smithi125.stdout:3172: writing smithi12556470-15 from 1883680 to 2252402 tid 2 2021-03-24T22:06:31.638 INFO:tasks.rados.rados.0.smithi125.stdout:3164: done (4 left) 2021-03-24T22:06:31.638 INFO:tasks.rados.rados.0.smithi125.stdout:3173: read oid 11 snap -1 2021-03-24T22:06:31.638 INFO:tasks.rados.rados.0.smithi125.stdout:3173: expect (ObjNum 1069 snap 358 seq_num 1069) 2021-03-24T22:06:31.639 INFO:tasks.rados.rados.0.smithi125.stdout:3174: copy_from oid 20 from oid 1 current snap is 366 2021-03-24T22:06:31.639 INFO:tasks.rados.rados.0.smithi125.stdout:3166: expect (ObjNum 1071 snap 359 seq_num 1071) 2021-03-24T22:06:31.651 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:31.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:31.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:31.758 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:31.812 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:31.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:31.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:06:31.823 INFO:tasks.rados.rados.0.smithi125.stdout:3169: finishing write tid 1 to smithi12556470-26 2021-03-24T22:06:31.823 INFO:tasks.rados.rados.0.smithi125.stdout:3169: finishing write tid 2 to smithi12556470-26 2021-03-24T22:06:31.823 INFO:tasks.rados.rados.0.smithi125.stdout:3166: done (5 left) 2021-03-24T22:06:31.824 INFO:tasks.rados.rados.0.smithi125.stdout:3175: delete oid 24 current snap is 366 2021-03-24T22:06:31.824 INFO:tasks.rados.rados.0.smithi125.stdout:3169: finishing write tid 3 to smithi12556470-26 2021-03-24T22:06:31.824 INFO:tasks.rados.rados.0.smithi125.stdout:3169: finishing write tid 4 to smithi12556470-26 2021-03-24T22:06:31.824 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 26 v 1460 (ObjNum 1084 snap 366 seq_num 1084) dirty exists 2021-03-24T22:06:31.825 INFO:tasks.rados.rados.0.smithi125.stdout:3169: left oid 26 (ObjNum 1084 snap 366 seq_num 1084) 2021-03-24T22:06:31.826 INFO:tasks.rados.rados.0.smithi125.stdout:3172: finishing write tid 1 to smithi12556470-15 2021-03-24T22:06:31.826 INFO:tasks.rados.rados.0.smithi125.stdout:3172: finishing write tid 2 to smithi12556470-15 2021-03-24T22:06:31.826 INFO:tasks.rados.rados.0.smithi125.stdout:3172: finishing write tid 3 to smithi12556470-15 2021-03-24T22:06:31.826 INFO:tasks.rados.rados.0.smithi125.stdout:3172: finishing write tid 4 to smithi12556470-15 2021-03-24T22:06:31.827 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 15 v 1436 (ObjNum 1086 snap 366 seq_num 1086) dirty exists 2021-03-24T22:06:31.827 INFO:tasks.rados.rados.0.smithi125.stdout:3172: left oid 15 (ObjNum 1086 snap 366 seq_num 1086) 2021-03-24T22:06:31.827 INFO:tasks.rados.rados.0.smithi125.stdout:3173: expect (ObjNum 1069 snap 358 seq_num 1069) 2021-03-24T22:06:31.917 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:31.975 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:31.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:31.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:06:32.080 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:32.116 INFO:tasks.rados.rados.0.smithi125.stdout:3174: finishing copy_from to smithi12556470-20 2021-03-24T22:06:32.116 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 20 v 1449 (ObjNum 808 snap 265 seq_num 808) dirty exists 2021-03-24T22:06:32.117 INFO:tasks.rados.rados.0.smithi125.stdout:3174: finishing copy_from racing read to smithi12556470-20 2021-03-24T22:06:32.117 INFO:tasks.rados.rados.0.smithi125.stdout:3169: done (5 left) 2021-03-24T22:06:32.117 INFO:tasks.rados.rados.0.smithi125.stdout:3171: done (4 left) 2021-03-24T22:06:32.117 INFO:tasks.rados.rados.0.smithi125.stdout:3172: done (3 left) 2021-03-24T22:06:32.118 INFO:tasks.rados.rados.0.smithi125.stdout:3173: done (2 left) 2021-03-24T22:06:32.118 INFO:tasks.rados.rados.0.smithi125.stdout:3174: done (1 left) 2021-03-24T22:06:32.118 INFO:tasks.rados.rados.0.smithi125.stdout:3175: done (0 left) 2021-03-24T22:06:32.118 INFO:tasks.rados.rados.0.smithi125.stdout:3176: read oid 22 snap -1 2021-03-24T22:06:32.118 INFO:tasks.rados.rados.0.smithi125.stdout:3176: expect (ObjNum 1035 snap 342 seq_num 1035) 2021-03-24T22:06:32.119 INFO:tasks.rados.rados.0.smithi125.stdout:3177: copy_from oid 19 from oid 18 current snap is 366 2021-03-24T22:06:32.119 INFO:tasks.rados.rados.0.smithi125.stdout:3178: write oid 46 current snap is 366 2021-03-24T22:06:32.119 INFO:tasks.rados.rados.0.smithi125.stdout:3178: seq_num 1088 ranges {686016=509391,1716997=569019} 2021-03-24T22:06:32.122 INFO:tasks.rados.rados.0.smithi125.stdout:3178: writing smithi12556470-46 from 686016 to 1195407 tid 1 2021-03-24T22:06:32.131 INFO:tasks.rados.rados.0.smithi125.stdout:3178: writing smithi12556470-46 from 1716997 to 2286016 tid 2 2021-03-24T22:06:32.131 INFO:tasks.rados.rados.0.smithi125.stdout:3179: rollback oid 15 current snap is 366 2021-03-24T22:06:32.131 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 15 to 351 2021-03-24T22:06:32.132 INFO:tasks.rados.rados.0.smithi125.stdout:3176: expect (ObjNum 1035 snap 342 seq_num 1035) 2021-03-24T22:06:32.141 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:32.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:32.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:32.245 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:32.247 INFO:tasks.rados.rados.0.smithi125.stdout:3178: finishing write tid 1 to smithi12556470-46 2021-03-24T22:06:32.248 INFO:tasks.rados.rados.0.smithi125.stdout:3178: finishing write tid 2 to smithi12556470-46 2021-03-24T22:06:32.248 INFO:tasks.rados.rados.0.smithi125.stdout:3176: done (3 left) 2021-03-24T22:06:32.248 INFO:tasks.rados.rados.0.smithi125.stdout:3180: rollback oid 4 current snap is 366 2021-03-24T22:06:32.249 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 4 to 356 2021-03-24T22:06:32.249 INFO:tasks.rados.rados.0.smithi125.stdout:3178: finishing write tid 3 to smithi12556470-46 2021-03-24T22:06:32.249 INFO:tasks.rados.rados.0.smithi125.stdout:3179: finishing rollback tid 0 to smithi12556470-15 2021-03-24T22:06:32.249 INFO:tasks.rados.rados.0.smithi125.stdout:3178: finishing write tid 4 to smithi12556470-46 2021-03-24T22:06:32.249 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 46 v 1143 (ObjNum 1088 snap 366 seq_num 1088) dirty exists 2021-03-24T22:06:32.250 INFO:tasks.rados.rados.0.smithi125.stdout:3178: left oid 46 (ObjNum 1088 snap 366 seq_num 1088) 2021-03-24T22:06:32.250 INFO:tasks.rados.rados.0.smithi125.stdout:3179: finishing rollback tid 1 to smithi12556470-15 2021-03-24T22:06:32.250 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 15 v 1438 (ObjNum 0 snap 0 seq_num 1114771472) dirty dne 2021-03-24T22:06:32.250 INFO:tasks.rados.rados.0.smithi125.stdout:3177: finishing copy_from to smithi12556470-19 2021-03-24T22:06:32.251 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 19 v 744 (ObjNum 1057 snap 351 seq_num 1057) dirty exists 2021-03-24T22:06:32.251 INFO:tasks.rados.rados.0.smithi125.stdout:3177: finishing copy_from racing read to smithi12556470-19 2021-03-24T22:06:32.251 INFO:tasks.rados.rados.0.smithi125.stdout:3177: done (3 left) 2021-03-24T22:06:32.251 INFO:tasks.rados.rados.0.smithi125.stdout:3178: done (2 left) 2021-03-24T22:06:32.251 INFO:tasks.rados.rados.0.smithi125.stdout:3179: done (1 left) 2021-03-24T22:06:32.252 INFO:tasks.rados.rados.0.smithi125.stdout:3181: read oid 44 snap -1 2021-03-24T22:06:32.252 INFO:tasks.rados.rados.0.smithi125.stdout:3181: expect (ObjNum 1051 snap 349 seq_num 1051) 2021-03-24T22:06:32.252 INFO:tasks.rados.rados.0.smithi125.stdout:3182: snap_create 2021-03-24T22:06:32.253 INFO:tasks.rados.rados.0.smithi125.stdout:3180: finishing rollback tid 0 to smithi12556470-4 2021-03-24T22:06:32.254 INFO:tasks.rados.rados.0.smithi125.stdout:3180: finishing rollback tid 1 to smithi12556470-4 2021-03-24T22:06:32.255 INFO:tasks.rados.rados.0.smithi125.stdout:3180: finishing rollback tid 2 to smithi12556470-4 2021-03-24T22:06:32.255 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 4 v 1023 (ObjNum 1013 snap 336 seq_num 1013) dirty exists 2021-03-24T22:06:32.261 INFO:tasks.rados.rados.0.smithi125.stdout:3181: expect (ObjNum 1051 snap 349 seq_num 1051) 2021-03-24T22:06:32.303 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:32.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:32.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:06:32.410 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:32.472 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:32.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:32.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:06:32.487 INFO:tasks.rados.rados.0.smithi125.stdout:3180: done (2 left) 2021-03-24T22:06:32.487 INFO:tasks.rados.rados.0.smithi125.stdout:3181: done (1 left) 2021-03-24T22:06:32.487 INFO:tasks.rados.rados.0.smithi125.stdout:3182: done (0 left) 2021-03-24T22:06:32.488 INFO:tasks.rados.rados.0.smithi125.stdout:3183: delete oid 15 current snap is 367 2021-03-24T22:06:32.488 INFO:tasks.rados.rados.0.smithi125.stdout:3183: done (0 left) 2021-03-24T22:06:32.488 INFO:tasks.rados.rados.0.smithi125.stdout:3184: snap_create 2021-03-24T22:06:32.587 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:32.599 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:32 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T22:06:32.599 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:32 smithi125 conmon[19936]: -24T22:06:31.005888+0000 mgr.y (mgr.14144) 685 : cluster [DBG] pgmap v1393: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 502 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 517 KiB/s wr, 3 op/s; 11/1638 objects misplaced (0.672%) 2021-03-24T22:06:32.599 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:32 smithi125 conmon[19936]: cluster 2021-03-24T22:06:32.600 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:32 smithi125 conmon[19936]: 2021-03-24T 2021-03-24T22:06:32.600 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:32 smithi125 conmon[19936]: 22:06:31.435361+0000 mon.a (mon.0) 2048 : cluster [DBG] 2021-03-24T22:06:32.600 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:32 smithi125 conmon[19936]: osdmap e821: 8 total, 8 up, 8 in 2021-03-24T22:06:32.601 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:32 smithi125 conmon[24927]: cluster 2021-03-24T22:06:31.005888+0000 mgr.y (mgr.14144) 685 : cluster [DBG] pgmap v1393: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 502 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 517 KiB/s wr, 3 op/s; 11/1638 objects misplaced (0.672%) 2021-03-24T22:06:32.601 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:32 smithi125 conmon[24927]: cluster 2021-03-24T22:06:31.435361+0000 mon.a (mon.0) 2048 : cluster [DBG] osdmap e821: 8 total, 8 up, 8 in 2021-03-24T22:06:32.632 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:32.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:32.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:06:32.738 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:32.739 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-24T22:06:32.744 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2021-03-24T22:06:32.831 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:32.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:32.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:06:32.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:32 smithi142 conmon[20125]: cluster 2021-03-24T22:06:32.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:32 smithi142 conmon[20125]: 2021- 2021-03-24T22:06:32.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:32 smithi142 conmon[20125]: 03-24T22:06:31.005888+0000 mgr.y (mgr.14144) 685 : cluster [DBG] pgmap v1393: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 502 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 517 KiB/s wr, 3 op/s; 11/1638 objects misplaced (0.672%) 2021-03-24T22:06:32.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:32 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:06:32.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:32 smithi142 conmon[20125]: 03-24T22:06:31.435361+0000 mon.a (mon.0) 2048 : cluster 2021-03-24T22:06:32.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:32 smithi142 conmon[20125]: [DBG] osdmap e821: 8 total, 8 up, 8 in 2021-03-24T22:06:32.937 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:32.999 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:33.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:33.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:06:33.103 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:33.156 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:33.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:33.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:06:33.265 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:33.327 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:33.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:33.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:06:33.435 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:33.490 INFO:tasks.rados.rados.0.smithi125.stdout:3184: done (0 left) 2021-03-24T22:06:33.490 INFO:tasks.rados.rados.0.smithi125.stdout:3185: rollback oid 42 current snap is 368 2021-03-24T22:06:33.491 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 42 to 365 2021-03-24T22:06:33.491 INFO:tasks.rados.rados.0.smithi125.stdout:3186: rollback oid 4 current snap is 368 2021-03-24T22:06:33.491 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 4 to 364 2021-03-24T22:06:33.491 INFO:tasks.rados.rados.0.smithi125.stdout:3187: snap_create 2021-03-24T22:06:33.497 INFO:teuthology.orchestra.run.smithi125.stderr:noscrub is unset 2021-03-24T22:06:33.498 INFO:tasks.rados.rados.0.smithi125.stdout:3185: finishing rollback tid 1 to smithi12556470-42 2021-03-24T22:06:33.499 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 42 v 513 (ObjNum 1953066355 snap 842099048 seq_num 875967797) dirty dne 2021-03-24T22:06:33.500 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:33.503 INFO:tasks.rados.rados.0.smithi125.stdout:3186: finishing rollback tid 0 to smithi12556470-4 2021-03-24T22:06:33.503 INFO:tasks.rados.rados.0.smithi125.stdout:3186: finishing rollback tid 1 to smithi12556470-4 2021-03-24T22:06:33.503 INFO:tasks.rados.rados.0.smithi125.stdout:3186: finishing rollback tid 2 to smithi12556470-4 2021-03-24T22:06:33.503 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 4 v 1026 (ObjNum 1079 snap 361 seq_num 1079) dirty exists 2021-03-24T22:06:33.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:33.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:06:33.607 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:33.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:33 smithi142 conmon[20125]: cluster 2021-03-24T22:06:32. 2021-03-24T22:06:33.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:33 smithi142 conmon[20125]: 474445+0000 mon.a (mon.0) 2049 : cluster [DBG] osdmap e822: 8 total, 8 up, 8 in 2021-03-24T22:06:33.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:33 smithi142 conmon[20125]: audit 2021-03-24T22:06:32.955840 2021-03-24T22:06:33.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:33 smithi142 conmon[20125]: +0000 mon.c (mon.2) 118 : audit [INF] from='client.? v1:172.21.15.125:0/1116926486' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T22:06:33.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:33 smithi142 conmon[20125]: audit 2021-03-24T22:06:32.956398+0000 mon.a (mon 2021-03-24T22:06:33.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:33 smithi142 conmon[20125]: .0) 2050 : audit [INF] from='client.35870 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T22:06:33.664 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:33.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:33.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:06:33.771 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:33.797 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:33 smithi125 conmon[19936]: cluster 2021-03-24T22:06:32.474445+0000 2021-03-24T22:06:33.798 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:33 smithi125 conmon[19936]: mon.a (mon.0) 2049 : cluster [DBG] osdmap e822: 8 total, 8 up, 8 in 2021-03-24T22:06:33.798 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:33 smithi125 conmon[19936]: audit 2021-03-24T22:06:32.955840+0000 mon.c (mon.2) 118 : audit 2021-03-24T22:06:33.798 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:33 smithi125 conmon[19936]: [INF] from='client.? v1:172.21.15.125:0/1116926486' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T22:06:33.798 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:33 smithi125 conmon[19936]: audit 2021- 2021-03-24T22:06:33.799 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:33 smithi125 conmon[19936]: 03-24T22:06:32.956398+0000 mon.a (mon.0) 2050 : audit [INF] 2021-03-24T22:06:33.799 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:33 smithi125 conmon[19936]: from='client.35870 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T22:06:33.799 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:33 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T22:06:33.800 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:33 smithi125 conmon[24927]: T22:06:32.474445+0000 mon.a (mon.0) 2049 : cluster [DBG] osdmap e822: 8 total, 8 up, 8 in 2021-03-24T22:06:33.800 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:33 smithi125 conmon[24927]: audit 2021 2021-03-24T22:06:33.800 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:33 smithi125 conmon[24927]: -03-24T22:06:32.955840+0000 2021-03-24T22:06:33.800 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:33 smithi125 conmon[24927]: mon.c (mon.2) 118 : audit [INF] 2021-03-24T22:06:33.801 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:33 smithi125 conmon[24927]: from='client.? v1:172.21.15.125:0/1116926486' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T22:06:33.801 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:33 smithi125 conmon[24927]: audit 2021-03-24T22: 2021-03-24T22:06:33.801 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:33 smithi125 conmon[24927]: 06:32.956398+0000 mon.a (mon.0) 2021-03-24T22:06:33.801 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:33 smithi125 conmon[24927]: 2050 : audit [INF] from='client.35870 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T22:06:33.832 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:33.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:33.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:06:33.937 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:33.984 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:33.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:33.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:06:34.031 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-24T22:06:34.031 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-24T22:06:34.031 INFO:teuthology.orchestra.run.smithi125.stdout: "num_ops": 0 2021-03-24T22:06:34.032 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-24T22:06:34.090 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:34.108 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":1395,"stamp":"2021-03-24T22:06:32.470207+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":525642257,"num_objects":545,"num_object_clones":179,"num_object_copies":1635,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11,"num_objects_unfound":0,"num_objects_dirty":545,"num_whiteouts":12,"num_read":4955,"num_read_kb":3269553,"num_write":3158,"num_write_kb":892631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3474,"num_bytes_recovered":1262218788,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20847,"ondisk_log_size":20847,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":506,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1577272,"kb_used_data":1214200,"kb_used_omap":164,"kb_used_meta":362843,"kb_avail":748383944,"statfs":{"total":767960285184,"available":766345158656,"internally_reserved":0,"allocated":1243340800,"data_stored":1646474819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":168054,"internal_metadata":371552138},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":792307,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":7446,"num_write":3,"num_write_kb":2069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.037946"},"pg_stats":[{"pgid":"3.29","version":"0'0","reported_seq":"468","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.444554+0000","last_change":"2021-03-24T22:03:45.428201+0000","last_active":"2021-03-24T22:06:28.444554+0000","last_peered":"2021-03-24T22:06:28.444554+0000","last_clean":"2021-03-24T22:06:28.444554+0000","last_became_active":"2021-03-24T22:03:45.427894+0000","last_became_peered":"2021-03-24T22:03:45.427894+0000","last_unstale":"2021-03-24T22:06:28.444554+0000","last_undegraded":"2021-03-24T22:06:28.444554+0000","last_fullsized":"2021-03-24T22:06:28.444554+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:55.592795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:55.592795+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":"343","reported_epoch":"820","state":"active+clean","last_fresh":"2021-03-24T22:06:30.443663+0000","last_change":"2021-03-24T22:05:54.394091+0000","last_active":"2021-03-24T22:06:30.443663+0000","last_peered":"2021-03-24T22:06:30.443663+0000","last_clean":"2021-03-24T22:06:30.443663+0000","last_became_active":"2021-03-24T22:05:04.450159+0000","last_became_peered":"2021-03-24T22:05:04.450159+0000","last_unstale":"2021-03-24T22:06:30.443663+0000","last_undegraded":"2021-03-24T22:06:30.443663+0000","last_fullsized":"2021-03-24T22:06:30.443663+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:54.393923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:05:54.393923+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"89'2","reported_seq":"488","reported_epoch":"816","state":"active+clean","last_fresh":"2021-03-24T22:06:26.453624+0000","last_change":"2021-03-24T22:04:28.159451+0000","last_active":"2021-03-24T22:06:26.453624+0000","last_peered":"2021-03-24T22:06:26.453624+0000","last_clean":"2021-03-24T22:06:26.453624+0000","last_became_active":"2021-03-24T21:59:14.374418+0000","last_became_peered":"2021-03-24T21:59:14.374418+0000","last_unstale":"2021-03-24T22:06:26.453624+0000","last_undegraded":"2021-03-24T22:06:26.453624+0000","last_fullsized":"2021-03-24T22:06:26.453624+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":6,"last_scrub":"89'2","last_scrub_stamp":"2021-03-24T22:04:28.159278+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2021-03-24T21:57:58.491830+0000","last_clean_scrub_stamp":"2021-03-24T22:04:28.159278+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":"830","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.443391+0000","last_change":"2021-03-24T22:05:04.443648+0000","last_active":"2021-03-24T22:06:28.443391+0000","last_peered":"2021-03-24T22:06:28.443391+0000","last_clean":"2021-03-24T22:06:28.443391+0000","last_became_active":"2021-03-24T22:05:04.442258+0000","last_became_peered":"2021-03-24T22:05:04.442258+0000","last_unstale":"2021-03-24T22:06:28.443391+0000","last_undegraded":"2021-03-24T22:06:28.443391+0000","last_fullsized":"2021-03-24T22:06:28.443391+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:43.239983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:03:43.239983+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":"276","reported_epoch":"820","state":"active+clean","last_fresh":"2021-03-24T22:06:30.433759+0000","last_change":"2021-03-24T22:05:53.982341+0000","last_active":"2021-03-24T22:06:30.433759+0000","last_peered":"2021-03-24T22:06:30.433759+0000","last_clean":"2021-03-24T22:06:30.433759+0000","last_became_active":"2021-03-24T22:05:04.448985+0000","last_became_peered":"2021-03-24T22:05:04.448985+0000","last_unstale":"2021-03-24T22:06:30.433759+0000","last_undegraded":"2021-03-24T22:06:30.433759+0000","last_fullsized":"2021-03-24T22:06:30.433759+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:53.982228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:05:53.982228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"61'1","reported_seq":"315","reported_epoch":"819","state":"active+clean","last_fresh":"2021-03-24T22:06:29.447131+0000","last_change":"2021-03-24T22:06:04.143008+0000","last_active":"2021-03-24T22:06:29.447131+0000","last_peered":"2021-03-24T22:06:29.447131+0000","last_clean":"2021-03-24T22:06:29.447131+0000","last_became_active":"2021-03-24T22:05:04.441568+0000","last_became_peered":"2021-03-24T22:05:04.441568+0000","last_unstale":"2021-03-24T22:06:29.447131+0000","last_undegraded":"2021-03-24T22:06:29.447131+0000","last_fullsized":"2021-03-24T22:06:29.447131+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":6,"last_scrub":"61'1","last_scrub_stamp":"2021-03-24T22:06:04.142856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:06:04.142856+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":"797","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.444614+0000","last_change":"2021-03-24T22:02:57.502941+0000","last_active":"2021-03-24T22:06:28.444614+0000","last_peered":"2021-03-24T22:06:28.444614+0000","last_clean":"2021-03-24T22:06:28.444614+0000","last_became_active":"2021-03-24T21:59:14.370506+0000","last_became_peered":"2021-03-24T21:59:14.370506+0000","last_unstale":"2021-03-24T22:06:28.444614+0000","last_undegraded":"2021-03-24T22:06:28.444614+0000","last_fullsized":"2021-03-24T22:06:28.444614+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:57.502803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:02:57.502803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.23","version":"820'506","reported_seq":"2936","reported_epoch":"820","state":"active+clean","last_fresh":"2021-03-24T22:06:30.466428+0000","last_change":"2021-03-24T22:06:30.466823+0000","last_active":"2021-03-24T22:06:30.466428+0000","last_peered":"2021-03-24T22:06:30.466428+0000","last_clean":"2021-03-24T22:06:30.466428+0000","last_became_active":"2021-03-24T22:03:49.378704+0000","last_became_peered":"2021-03-24T22:03:49.378704+0000","last_unstale":"2021-03-24T22:06:30.466428+0000","last_undegraded":"2021-03-24T22:06:30.466428+0000","last_fullsized":"2021-03-24T22:06:30.466428+0000","mapping_epoch":658,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":659,"parent":"0.0","parent_split_bits":6,"last_scrub":"731'441","last_scrub_stamp":"2021-03-24T22:05:01.661041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:05:01.661041+0000","log_size":506,"ondisk_log_size":506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12829074,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":12,"num_read_kb":10122,"num_write":10,"num_write_kb":5736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"25e","length":"1"},{"start":"2af","length":"1"},{"start":"2b3","length":"1"},{"start":"2b6","length":"1"}]},{"pgid":"3.26","version":"0'0","reported_seq":"305","reported_epoch":"819","state":"active+clean","last_fresh":"2021-03-24T22:06:29.447834+0000","last_change":"2021-03-24T22:04:21.723861+0000","last_active":"2021-03-24T22:06:29.447834+0000","last_peered":"2021-03-24T22:06:29.447834+0000","last_clean":"2021-03-24T22:06:29.447834+0000","last_became_active":"2021-03-24T22:01:57.311128+0000","last_became_peered":"2021-03-24T22:01:57.311128+0000","last_unstale":"2021-03-24T22:06:29.447834+0000","last_undegraded":"2021-03-24T22:06:29.447834+0000","last_fullsized":"2021-03-24T22:06:29.447834+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:04:21.723761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:04:21.723761+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"813'387","reported_seq":"2047","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.006151+0000","last_change":"2021-03-24T22:06:26.451454+0000","last_active":"2021-03-24T22:06:28.006151+0000","last_peered":"2021-03-24T22:06:28.006151+0000","last_clean":"2021-03-24T22:06:28.006151+0000","last_became_active":"2021-03-24T22:03:51.364119+0000","last_became_peered":"2021-03-24T22:03:51.364119+0000","last_unstale":"2021-03-24T22:06:28.006151+0000","last_undegraded":"2021-03-24T22:06:28.006151+0000","last_fullsized":"2021-03-24T22:06:28.006151+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":6,"last_scrub":"729'308","last_scrub_stamp":"2021-03-24T22:05:00.252320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:05:00.252320+0000","log_size":387,"ondisk_log_size":387,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13388469,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":4,"num_read_kb":1,"num_write":9,"num_write_kb":1363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2a3","length":"1"},{"start":"2b3","length":"1"},{"start":"2b6","length":"1"}]},{"pgid":"4.c","version":"544'37","reported_seq":"546","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.445126+0000","last_change":"2021-03-24T22:03:03.453859+0000","last_active":"2021-03-24T22:06:28.445126+0000","last_peered":"2021-03-24T22:06:28.445126+0000","last_clean":"2021-03-24T22:06:28.445126+0000","last_became_active":"2021-03-24T22:01:50.301325+0000","last_became_peered":"2021-03-24T22:01:50.301325+0000","last_unstale":"2021-03-24T22:06:28.445126+0000","last_undegraded":"2021-03-24T22:06:28.445126+0000","last_fullsized":"2021-03-24T22:06:28.445126+0000","mapping_epoch":539,"log_start":"69'15","ondisk_log_start":"69'15","created":62,"last_epoch_clean":540,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'37","last_scrub_stamp":"2021-03-24T22:03:03.453713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:03:03.453713+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":15,"num_object_clones":0,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":81,"num_read_kb":61,"num_write":51,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":1266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"785","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.444831+0000","last_change":"2021-03-24T22:05:02.207625+0000","last_active":"2021-03-24T22:06:28.444831+0000","last_peered":"2021-03-24T22:06:28.444831+0000","last_clean":"2021-03-24T22:06:28.444831+0000","last_became_active":"2021-03-24T21:51:45.375830+0000","last_became_peered":"2021-03-24T21:51:45.375830+0000","last_unstale":"2021-03-24T22:06:28.444831+0000","last_undegraded":"2021-03-24T22:06:28.444831+0000","last_fullsized":"2021-03-24T22:06:28.444831+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:02.207454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:05:02.207454+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"734'14","reported_seq":"1061","reported_epoch":"820","state":"active+clean","last_fresh":"2021-03-24T22:06:30.437807+0000","last_change":"2021-03-24T22:05:47.047703+0000","last_active":"2021-03-24T22:06:30.437807+0000","last_peered":"2021-03-24T22:06:30.437807+0000","last_clean":"2021-03-24T22:06:30.437807+0000","last_became_active":"2021-03-24T22:05:04.430698+0000","last_became_peered":"2021-03-24T22:05:04.430698+0000","last_unstale":"2021-03-24T22:06:30.437807+0000","last_undegraded":"2021-03-24T22:06:30.437807+0000","last_fullsized":"2021-03-24T22:06:30.437807+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"734'14","last_scrub_stamp":"2021-03-24T22:05:47.047527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:05:47.047527+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"179","reported_epoch":"818","state":"active+clean","last_fresh":"2021-03-24T22:06:28.438318+0000","last_change":"2021-03-24T22:04:22.588952+0000","last_active":"2021-03-24T22:06:28.438318+0000","last_peered":"2021-03-24T22:06:28.438318+0000","last_clean":"2021-03-24T22:06:28.438318+0000","last_became_active":"2021-03-24T22:03:45.430245+0000","last_became_peered":"2021-03-24T22:03:45.430245+0000","last_unstale":"2021-03-24T22:06:28.438318+0000","last_undegraded":"2021-03-24T22:06:28.438318+0000","last_fullsized":"2021-03-24T22:06:28.438318+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:04:22.588884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:04:22.588884+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.f","version":"813'725","reported_seq":"3154","reported_epoch":"816","state":"active+clean","last_fresh":"2021-03-24T22:06:26.457017+0000","last_change":"2021-03-24T22:06:26.457017+0000","last_active":"2021-03-24T22:06:26.457017+0000","last_peered":"2021-03-24T22:06:26.457017+0000","last_clean":"2021-03-24T22:06:26.457017+0000","last_became_active":"2021-03-24T22:02:10.313297+0000","last_became_peered":"2021-03-24T22:02:10.313297+0000","last_unstale":"2021-03-24T22:06:26.457017+0000","last_undegraded":"2021-03-24T22:06:26.457017+0000","last_fullsized":"2021-03-24T22:06:26.457017+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"705'573","last_scrub_stamp":"2021-03-24T22:04:36.224078+0000","last_deep_scrub":"705'573","last_deep_scrub_stamp":"2021-03-24T22:04:36.224078+0000","last_clean_scrub_stamp":"2021-03-24T22:04:36.224078+0000","log_size":725,"ondisk_log_size":725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29660336,"num_objects":11,"num_object_clones":9,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":136,"num_read_kb":142456,"num_write":95,"num_write_kb":37438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":46545574,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2a3","length":"1"},{"start":"2b3","length":"1"},{"start":"2b6","length":"1"}]},{"pgid":"4.d","version":"544'41","reported_seq":"529","reported_epoch":"820","state":"active+clean","last_fresh":"2021-03-24T22:06:30.437485+0000","last_change":"2021-03-24T22:04:29.904628+0000","last_active":"2021-03-24T22:06:30.437485+0000","last_peered":"2021-03-24T22:06:30.437485+0000","last_clean":"2021-03-24T22:06:30.437485+0000","last_became_active":"2021-03-24T22:03:45.430156+0000","last_became_peered":"2021-03-24T22:03:45.430156+0000","last_unstale":"2021-03-24T22:06:30.437485+0000","last_undegraded":"2021-03-24T22:06:30.437485+0000","last_fullsized":"2021-03-24T22:06:30.437485+0000","mapping_epoch":654,"log_start":"69'21","ondisk_log_start":"69'21","created":62,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'41","last_scrub_stamp":"2021-03-24T22:04:29.904532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:04:29.904532+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1578,"num_objects":21,"num_object_clones":0,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":106,"num_read_kb":71,"num_write":60,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"319","reported_epoch":"819","state":"active+clean","last_fresh":"2021-03-24T22:06:29.447338+0000","last_change":"2021-03-24T22:05:02.665356+0000","last_active":"2021-03-24T22:06:29.447338+0000","last_peered":"2021-03-24T22:06:29.447338+0000","last_clean":"2021-03-24T22:06:29.447338+0000","last_became_active":"2021-03-24T22:03:45.419665+0000","last_became_peered":"2021-03-24T22:03:45.419665+0000","last_unstale":"2021-03-24T22:06:29.447338+0000","last_undegraded":"2021-03-24T22:06:29.447338+0000","last_fullsized":"2021-03-24T22:06:29.447338+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:02.664970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:55:01.169461+0000","last_clean_scrub_stamp":"2021-03-24T22:05:02.664970+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"179","reported_epoch":"818","state":"active+clean","last_fresh":"2021-03-24T22:06:28.439624+0000","last_change":"2021-03-24T22:03:56.955470+0000","last_active":"2021-03-24T22:06:28.439624+0000","last_peered":"2021-03-24T22:06:28.439624+0000","last_clean":"2021-03-24T22:06:28.439624+0000","last_became_active":"2021-03-24T22:03:45.430779+0000","last_became_peered":"2021-03-24T22:03:45.430779+0000","last_unstale":"2021-03-24T22:06:28.439624+0000","last_undegraded":"2021-03-24T22:06:28.439624+0000","last_fullsized":"2021-03-24T22:06:28.439624+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:56.955183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:03:56.955183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"266","reported_epoch":"820","state":"active+clean","last_fresh":"2021-03-24T22:06:30.437905+0000","last_change":"2021-03-24T22:06:01.977397+0000","last_active":"2021-03-24T22:06:30.437905+0000","last_peered":"2021-03-24T22:06:30.437905+0000","last_clean":"2021-03-24T22:06:30.437905+0000","last_became_active":"2021-03-24T22:02:22.482380+0000","last_became_peered":"2021-03-24T22:02:22.482380+0000","last_unstale":"2021-03-24T22:06:30.437905+0000","last_undegraded":"2021-03-24T22:06:30.437905+0000","last_fullsized":"2021-03-24T22:06:30.437905+0000","mapping_epoch":571,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:01.977309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:06:01.977309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"821'313","reported_seq":"2597","reported_epoch":"821","state":"active+clean","last_fresh":"2021-03-24T22:06:31.459991+0000","last_change":"2021-03-24T22:06:31.461762+0000","last_active":"2021-03-24T22:06:31.459991+0000","last_peered":"2021-03-24T22:06:31.459991+0000","last_clean":"2021-03-24T22:06:31.459991+0000","last_became_active":"2021-03-24T22:05:04.448963+0000","last_became_peered":"2021-03-24T22:05:04.448963+0000","last_unstale":"2021-03-24T22:06:31.459991+0000","last_undegraded":"2021-03-24T22:06:31.459991+0000","last_fullsized":"2021-03-24T22:06:31.459991+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"773'278","last_scrub_stamp":"2021-03-24T22:05:46.070371+0000","last_deep_scrub":"546'150","last_deep_scrub_stamp":"2021-03-24T22:01:56.662245+0000","last_clean_scrub_stamp":"2021-03-24T22:05:46.070371+0000","log_size":313,"ondisk_log_size":313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7395841,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":93813,"num_write":56,"num_write_kb":14865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":8580883,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"25e","length":"1"},{"start":"28e","length":"1"},{"start":"2af","length":"1"},{"start":"2b3","length":"1"},{"start":"2b6","length":"1"}]},{"pgid":"4.e","version":"544'45","reported_seq":"683","reported_epoch":"819","state":"active+clean","last_fresh":"2021-03-24T22:06:29.447607+0000","last_change":"2021-03-24T22:03:56.694071+0000","last_active":"2021-03-24T22:06:29.447607+0000","last_peered":"2021-03-24T22:06:29.447607+0000","last_clean":"2021-03-24T22:06:29.447607+0000","last_became_active":"2021-03-24T22:03:53.368885+0000","last_became_peered":"2021-03-24T22:03:53.368885+0000","last_unstale":"2021-03-24T22:06:29.447607+0000","last_undegraded":"2021-03-24T22:06:29.447607+0000","last_fullsized":"2021-03-24T22:06:29.447607+0000","mapping_epoch":662,"log_start":"89'27","ondisk_log_start":"89'27","created":62,"last_epoch_clean":663,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'45","last_scrub_stamp":"2021-03-24T22:03:56.693900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:03:56.693900+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1672,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":92,"num_read_kb":60,"num_write":50,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":5016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"807","reported_epoch":"820","state":"active+clean","last_fresh":"2021-03-24T22:06:30.435601+0000","last_change":"2021-03-24T22:05:56.072585+0000","last_active":"2021-03-24T22:06:30.435601+0000","last_peered":"2021-03-24T22:06:30.435601+0000","last_clean":"2021-03-24T22:06:30.435601+0000","last_became_active":"2021-03-24T22:01:57.302297+0000","last_became_peered":"2021-03-24T22:01:57.302297+0000","last_unstale":"2021-03-24T22:06:30.435601+0000","last_undegraded":"2021-03-24T22:06:30.435601+0000","last_fullsized":"2021-03-24T22:06:30.435601+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:56.072443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T22:05:56.072443+0000","last_clean_scrub_stamp":"2021-03-24T22:05:56.072443+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"311","reported_epoch":"819","state":"active+clean","last_fresh":"2021-03-24T22:06:29.448346+0000","last_change":"2021-03-24T22:05:14.424426+0000","last_active":"2021-03-24T22:06:29.448346+0000","last_peered":"2021-03-24T22:06:29.448346+0000","last_clean":"2021-03-24T22:06:29.448346+0000","last_became_active":"2021-03-24T22:05:04.428211+0000","last_became_peered":"2021-03-24T22:05:04.428211+0000","last_unstale":"2021-03-24T22:06:29.448346+0000","last_undegraded":"2021-03-24T22:06:29.448346+0000","last_fullsized":"2021-03-24T22:06:29.448346+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:14.424333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:05:14.424333+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"851","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.444957+0000","last_change":"2021-03-24T22:06:03.906295+0000","last_active":"2021-03-24T22:06:28.444957+0000","last_peered":"2021-03-24T22:06:28.444957+0000","last_clean":"2021-03-24T22:06:28.444957+0000","last_became_active":"2021-03-24T22:05:04.447129+0000","last_became_peered":"2021-03-24T22:05:04.447129+0000","last_unstale":"2021-03-24T22:06:28.444957+0000","last_undegraded":"2021-03-24T22:06:28.444957+0000","last_fullsized":"2021-03-24T22:06:28.444957+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:03.906118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:06:03.906118+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"820'843","reported_seq":"3350","reported_epoch":"820","state":"active+clean","last_fresh":"2021-03-24T22:06:30.473641+0000","last_change":"2021-03-24T22:06:30.474438+0000","last_active":"2021-03-24T22:06:30.473641+0000","last_peered":"2021-03-24T22:06:30.473641+0000","last_clean":"2021-03-24T22:06:30.473641+0000","last_became_active":"2021-03-24T22:03:53.379184+0000","last_became_peered":"2021-03-24T22:03:53.379184+0000","last_unstale":"2021-03-24T22:06:30.473641+0000","last_undegraded":"2021-03-24T22:06:30.473641+0000","last_fullsized":"2021-03-24T22:06:30.473641+0000","mapping_epoch":662,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":663,"parent":"0.0","parent_split_bits":0,"last_scrub":"787'795","last_scrub_stamp":"2021-03-24T22:05:59.132881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:05:59.132881+0000","log_size":843,"ondisk_log_size":843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17355123,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":132,"num_read_kb":168311,"num_write":60,"num_write_kb":26017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":179,"num_bytes_recovered":70071133,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"25e","length":"1"},{"start":"2af","length":"1"},{"start":"2b3","length":"1"},{"start":"2b6","length":"1"}]},{"pgid":"4.f","version":"544'46","reported_seq":"1008","reported_epoch":"818","state":"active+clean","last_fresh":"2021-03-24T22:06:28.436855+0000","last_change":"2021-03-24T22:04:30.667918+0000","last_active":"2021-03-24T22:06:28.436855+0000","last_peered":"2021-03-24T22:06:28.436855+0000","last_clean":"2021-03-24T22:06:28.436855+0000","last_became_active":"2021-03-24T22:04:18.462489+0000","last_became_peered":"2021-03-24T22:04:18.462489+0000","last_unstale":"2021-03-24T22:06:28.436855+0000","last_undegraded":"2021-03-24T22:06:28.436855+0000","last_fullsized":"2021-03-24T22:06:28.436855+0000","mapping_epoch":687,"log_start":"274'26","ondisk_log_start":"274'26","created":62,"last_epoch_clean":688,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'46","last_scrub_stamp":"2021-03-24T22:04:30.667850+0000","last_deep_scrub":"274'26","last_deep_scrub_stamp":"2021-03-24T21:57:24.499090+0000","last_clean_scrub_stamp":"2021-03-24T22:04:30.667850+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1472,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":97,"num_read_kb":65,"num_write":56,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":8410,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":"862","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.442937+0000","last_change":"2021-03-24T22:03:45.428200+0000","last_active":"2021-03-24T22:06:28.442937+0000","last_peered":"2021-03-24T22:06:28.442937+0000","last_clean":"2021-03-24T22:06:28.442937+0000","last_became_active":"2021-03-24T22:03:45.427907+0000","last_became_peered":"2021-03-24T22:03:45.427907+0000","last_unstale":"2021-03-24T22:06:28.442937+0000","last_undegraded":"2021-03-24T22:06:28.442937+0000","last_fullsized":"2021-03-24T22:06:28.442937+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:39.439170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:39.439170+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"734'7","reported_seq":"448","reported_epoch":"816","state":"active+clean","last_fresh":"2021-03-24T22:06:26.457636+0000","last_change":"2021-03-24T22:05:09.312441+0000","last_active":"2021-03-24T22:06:26.457636+0000","last_peered":"2021-03-24T22:06:26.457636+0000","last_clean":"2021-03-24T22:06:26.457636+0000","last_became_active":"2021-03-24T22:05:04.447711+0000","last_became_peered":"2021-03-24T22:05:04.447711+0000","last_unstale":"2021-03-24T22:06:26.457636+0000","last_undegraded":"2021-03-24T22:06:26.457636+0000","last_fullsized":"2021-03-24T22:06:26.457636+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"734'7","last_scrub_stamp":"2021-03-24T22:05:09.312269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:05:09.312269+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"652","reported_epoch":"820","state":"active+clean","last_fresh":"2021-03-24T22:06:30.445251+0000","last_change":"2021-03-24T22:05:47.327420+0000","last_active":"2021-03-24T22:06:30.445251+0000","last_peered":"2021-03-24T22:06:30.445251+0000","last_clean":"2021-03-24T22:06:30.445251+0000","last_became_active":"2021-03-24T22:05:04.451010+0000","last_became_peered":"2021-03-24T22:05:04.451010+0000","last_unstale":"2021-03-24T22:06:30.445251+0000","last_undegraded":"2021-03-24T22:06:30.445251+0000","last_fullsized":"2021-03-24T22:06:30.445251+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:47.327213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:05:47.327213+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"820'1329","reported_seq":"4019","reported_epoch":"820","state":"active+clean","last_fresh":"2021-03-24T22:06:30.459725+0000","last_change":"2021-03-24T22:06:30.460051+0000","last_active":"2021-03-24T22:06:30.459725+0000","last_peered":"2021-03-24T22:06:30.459725+0000","last_clean":"2021-03-24T22:06:30.459725+0000","last_became_active":"2021-03-24T22:02:06.309737+0000","last_became_peered":"2021-03-24T22:02:06.309737+0000","last_unstale":"2021-03-24T22:06:30.459725+0000","last_undegraded":"2021-03-24T22:06:30.459725+0000","last_fullsized":"2021-03-24T22:06:30.459725+0000","mapping_epoch":555,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":556,"parent":"0.0","parent_split_bits":0,"last_scrub":"659'936","last_scrub_stamp":"2021-03-24T22:03:53.662470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:03:53.662470+0000","log_size":1329,"ondisk_log_size":1329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34478423,"num_objects":14,"num_object_clones":11,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":250,"num_read_kb":267614,"num_write":151,"num_write_kb":63494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":296,"num_bytes_recovered":189413266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"25e","length":"1"},{"start":"2af","length":"1"},{"start":"2b3","length":"1"},{"start":"2b6","length":"1"}]},{"pgid":"5.1","version":"0'0","reported_seq":"754","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.443787+0000","last_change":"2021-03-24T22:03:00.472146+0000","last_active":"2021-03-24T22:06:28.443787+0000","last_peered":"2021-03-24T22:06:28.443787+0000","last_clean":"2021-03-24T22:06:28.443787+0000","last_became_active":"2021-03-24T22:01:57.305430+0000","last_became_peered":"2021-03-24T22:01:57.305430+0000","last_unstale":"2021-03-24T22:06:28.443787+0000","last_undegraded":"2021-03-24T22:06:28.443787+0000","last_fullsized":"2021-03-24T22:06:28.443787+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:00.471949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:25.220436+0000","last_clean_scrub_stamp":"2021-03-24T22:03:00.471949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"544'34","reported_seq":"987","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.442885+0000","last_change":"2021-03-24T22:04:30.205418+0000","last_active":"2021-03-24T22:06:28.442885+0000","last_peered":"2021-03-24T22:06:28.442885+0000","last_clean":"2021-03-24T22:06:28.442885+0000","last_became_active":"2021-03-24T22:01:57.309916+0000","last_became_peered":"2021-03-24T22:01:57.309916+0000","last_unstale":"2021-03-24T22:06:28.442885+0000","last_undegraded":"2021-03-24T22:06:28.442885+0000","last_fullsized":"2021-03-24T22:06:28.442885+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'34","last_scrub_stamp":"2021-03-24T22:04:30.205247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:04:30.205247+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":64,"num_read_kb":44,"num_write":36,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":3384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"836","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.444398+0000","last_change":"2021-03-24T22:06:06.879783+0000","last_active":"2021-03-24T22:06:28.444398+0000","last_peered":"2021-03-24T22:06:28.444398+0000","last_clean":"2021-03-24T22:06:28.444398+0000","last_became_active":"2021-03-24T22:01:57.314345+0000","last_became_peered":"2021-03-24T22:01:57.314345+0000","last_unstale":"2021-03-24T22:06:28.444398+0000","last_undegraded":"2021-03-24T22:06:28.444398+0000","last_fullsized":"2021-03-24T22:06:28.444398+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:06.879633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:06:06.879633+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"353","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.443708+0000","last_change":"2021-03-24T22:03:45.222962+0000","last_active":"2021-03-24T22:06:28.443708+0000","last_peered":"2021-03-24T22:06:28.443708+0000","last_clean":"2021-03-24T22:06:28.443708+0000","last_became_active":"2021-03-24T22:00:49.305612+0000","last_became_peered":"2021-03-24T22:00:49.305612+0000","last_unstale":"2021-03-24T22:06:28.443708+0000","last_undegraded":"2021-03-24T22:06:28.443708+0000","last_fullsized":"2021-03-24T22:06:28.443708+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:45.222743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:03:45.222743+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"810'856","reported_seq":"3063","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.445413+0000","last_change":"2021-03-24T22:06:26.445170+0000","last_active":"2021-03-24T22:06:28.445413+0000","last_peered":"2021-03-24T22:06:28.445413+0000","last_clean":"2021-03-24T22:06:28.445413+0000","last_became_active":"2021-03-24T21:57:26.169302+0000","last_became_peered":"2021-03-24T21:57:26.169302+0000","last_unstale":"2021-03-24T22:06:28.445413+0000","last_undegraded":"2021-03-24T22:06:28.445413+0000","last_fullsized":"2021-03-24T22:06:28.445413+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'770","last_scrub_stamp":"2021-03-24T22:04:57.253408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:04:57.253408+0000","log_size":856,"ondisk_log_size":856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13329040,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":71,"num_read_kb":53829,"num_write":79,"num_write_kb":34116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2a3","length":"1"},{"start":"2b3","length":"1"},{"start":"2b6","length":"1"}]},{"pgid":"5.0","version":"658'6","reported_seq":"1010","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.442819+0000","last_change":"2021-03-24T22:05:13.207391+0000","last_active":"2021-03-24T22:06:28.442819+0000","last_peered":"2021-03-24T22:06:28.442819+0000","last_clean":"2021-03-24T22:06:28.442819+0000","last_became_active":"2021-03-24T22:03:45.432883+0000","last_became_peered":"2021-03-24T22:03:45.432883+0000","last_unstale":"2021-03-24T22:06:28.442819+0000","last_undegraded":"2021-03-24T22:06:28.442819+0000","last_fullsized":"2021-03-24T22:06:28.442819+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"658'6","last_scrub_stamp":"2021-03-24T22:05:13.207196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:05:13.207196+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"544'34","reported_seq":"372","reported_epoch":"820","state":"active+clean","last_fresh":"2021-03-24T22:06:30.444141+0000","last_change":"2021-03-24T22:05:44.265489+0000","last_active":"2021-03-24T22:06:30.444141+0000","last_peered":"2021-03-24T22:06:30.444141+0000","last_clean":"2021-03-24T22:06:30.444141+0000","last_became_active":"2021-03-24T22:05:04.433178+0000","last_became_peered":"2021-03-24T22:05:04.433178+0000","last_unstale":"2021-03-24T22:06:30.444141+0000","last_undegraded":"2021-03-24T22:06:30.444141+0000","last_fullsized":"2021-03-24T22:06:30.444141+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'34","last_scrub_stamp":"2021-03-24T22:05:44.265284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:05:44.265284+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":58,"num_read_kb":39,"num_write":34,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":940,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"851","reported_epoch":"816","state":"active+clean","last_fresh":"2021-03-24T22:06:26.457423+0000","last_change":"2021-03-24T22:06:06.448287+0000","last_active":"2021-03-24T22:06:26.457423+0000","last_peered":"2021-03-24T22:06:26.457423+0000","last_clean":"2021-03-24T22:06:26.457423+0000","last_became_active":"2021-03-24T22:06:06.447793+0000","last_became_peered":"2021-03-24T22:06:06.447793+0000","last_unstale":"2021-03-24T22:06:26.457423+0000","last_undegraded":"2021-03-24T22:06:26.457423+0000","last_fullsized":"2021-03-24T22:06:26.457423+0000","mapping_epoch":795,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":796,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:00.436502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:06:00.436502+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"283","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.006839+0000","last_change":"2021-03-24T22:04:22.339077+0000","last_active":"2021-03-24T22:06:28.006839+0000","last_peered":"2021-03-24T22:06:28.006839+0000","last_clean":"2021-03-24T22:06:28.006839+0000","last_became_active":"2021-03-24T22:01:57.316610+0000","last_became_peered":"2021-03-24T22:01:57.316610+0000","last_unstale":"2021-03-24T22:06:28.006839+0000","last_undegraded":"2021-03-24T22:06:28.006839+0000","last_fullsized":"2021-03-24T22:06:28.006839+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:04:22.338910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:04:22.338910+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"272'64","reported_seq":"2300","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.443459+0000","last_change":"2021-03-24T22:06:26.436251+0000","last_active":"2021-03-24T22:06:28.443459+0000","last_peered":"2021-03-24T22:06:28.443459+0000","last_clean":"2021-03-24T22:06:28.443459+0000","last_became_active":"2021-03-24T22:03:45.428845+0000","last_became_peered":"2021-03-24T22:03:45.428845+0000","last_unstale":"2021-03-24T22:06:28.443459+0000","last_undegraded":"2021-03-24T22:06:28.443459+0000","last_fullsized":"2021-03-24T22:06:28.443459+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'64","last_scrub_stamp":"2021-03-24T22:05:03.168158+0000","last_deep_scrub":"272'64","last_deep_scrub_stamp":"2021-03-24T21:59:08.406188+0000","last_clean_scrub_stamp":"2021-03-24T22:05:03.168158+0000","log_size":64,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2a3","length":"1"},{"start":"2b3","length":"1"},{"start":"2b6","length":"1"}]},{"pgid":"5.3","version":"734'9","reported_seq":"1014","reported_epoch":"816","state":"active+clean","last_fresh":"2021-03-24T22:06:26.457261+0000","last_change":"2021-03-24T22:05:47.459972+0000","last_active":"2021-03-24T22:06:26.457261+0000","last_peered":"2021-03-24T22:06:26.457261+0000","last_clean":"2021-03-24T22:06:26.457261+0000","last_became_active":"2021-03-24T22:05:04.447618+0000","last_became_peered":"2021-03-24T22:05:04.447618+0000","last_unstale":"2021-03-24T22:06:26.457261+0000","last_undegraded":"2021-03-24T22:06:26.457261+0000","last_fullsized":"2021-03-24T22:06:26.457261+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"734'9","last_scrub_stamp":"2021-03-24T22:05:47.459863+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-24T21:53:26.584678+0000","last_clean_scrub_stamp":"2021-03-24T22:05:47.459863+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"544'26","reported_seq":"218","reported_epoch":"818","state":"active+clean","last_fresh":"2021-03-24T22:06:28.437024+0000","last_change":"2021-03-24T22:05:06.622482+0000","last_active":"2021-03-24T22:06:28.437024+0000","last_peered":"2021-03-24T22:06:28.437024+0000","last_clean":"2021-03-24T22:06:28.437024+0000","last_became_active":"2021-03-24T22:05:04.449351+0000","last_became_peered":"2021-03-24T22:05:04.449351+0000","last_unstale":"2021-03-24T22:06:28.437024+0000","last_undegraded":"2021-03-24T22:06:28.437024+0000","last_fullsized":"2021-03-24T22:06:28.437024+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'26","last_scrub_stamp":"2021-03-24T22:05:06.622374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:05:06.622374+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":46,"num_read_kb":29,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"615","reported_epoch":"820","state":"active+clean","last_fresh":"2021-03-24T22:06:30.445654+0000","last_change":"2021-03-24T22:06:01.442658+0000","last_active":"2021-03-24T22:06:30.445654+0000","last_peered":"2021-03-24T22:06:30.445654+0000","last_clean":"2021-03-24T22:06:30.445654+0000","last_became_active":"2021-03-24T22:05:04.449973+0000","last_became_peered":"2021-03-24T22:05:04.449973+0000","last_unstale":"2021-03-24T22:06:30.445654+0000","last_undegraded":"2021-03-24T22:06:30.445654+0000","last_fullsized":"2021-03-24T22:06:30.445654+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:01.442569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:06:01.442569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"775","reported_epoch":"816","state":"active+clean","last_fresh":"2021-03-24T22:06:26.454422+0000","last_change":"2021-03-24T22:04:37.121136+0000","last_active":"2021-03-24T22:06:26.454422+0000","last_peered":"2021-03-24T22:06:26.454422+0000","last_clean":"2021-03-24T22:06:26.454422+0000","last_became_active":"2021-03-24T21:51:51.759627+0000","last_became_peered":"2021-03-24T21:51:51.759627+0000","last_unstale":"2021-03-24T22:06:26.454422+0000","last_undegraded":"2021-03-24T22:06:26.454422+0000","last_fullsized":"2021-03-24T22:06:26.454422+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:04:37.120859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:04:37.120859+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"816'559","reported_seq":"2887","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.443476+0000","last_change":"2021-03-24T22:06:26.443672+0000","last_active":"2021-03-24T22:06:28.443476+0000","last_peered":"2021-03-24T22:06:28.443476+0000","last_clean":"2021-03-24T22:06:28.443476+0000","last_became_active":"2021-03-24T22:02:30.322922+0000","last_became_peered":"2021-03-24T22:02:30.322922+0000","last_unstale":"2021-03-24T22:06:28.443476+0000","last_undegraded":"2021-03-24T22:06:28.443476+0000","last_fullsized":"2021-03-24T22:06:28.443476+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"695'455","last_scrub_stamp":"2021-03-24T22:04:28.210940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:04:28.210940+0000","log_size":559,"ondisk_log_size":559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13630574,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":73,"num_read_kb":81889,"num_write":52,"num_write_kb":15980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":24969090,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2a3","length":"1"},{"start":"2b3","length":"1"},{"start":"2b6","length":"1"}]},{"pgid":"5.2","version":"0'0","reported_seq":"311","reported_epoch":"819","state":"active+clean","last_fresh":"2021-03-24T22:06:29.447171+0000","last_change":"2021-03-24T22:05:06.638365+0000","last_active":"2021-03-24T22:06:29.447171+0000","last_peered":"2021-03-24T22:06:29.447171+0000","last_clean":"2021-03-24T22:06:29.447171+0000","last_became_active":"2021-03-24T22:05:04.444370+0000","last_became_peered":"2021-03-24T22:05:04.444370+0000","last_unstale":"2021-03-24T22:06:29.447171+0000","last_undegraded":"2021-03-24T22:06:29.447171+0000","last_fullsized":"2021-03-24T22:06:29.447171+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:06.638226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:05:06.638226+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"544'31","reported_seq":"995","reported_epoch":"816","state":"active+clean","last_fresh":"2021-03-24T22:06:26.457552+0000","last_change":"2021-03-24T22:05:52.398909+0000","last_active":"2021-03-24T22:06:26.457552+0000","last_peered":"2021-03-24T22:06:26.457552+0000","last_clean":"2021-03-24T22:06:26.457552+0000","last_became_active":"2021-03-24T22:05:04.450428+0000","last_became_peered":"2021-03-24T22:05:04.450428+0000","last_unstale":"2021-03-24T22:06:26.457552+0000","last_undegraded":"2021-03-24T22:06:26.457552+0000","last_fullsized":"2021-03-24T22:06:26.457552+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'31","last_scrub_stamp":"2021-03-24T22:05:52.398731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:05:52.398731+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":60,"num_read_kb":42,"num_write":34,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":2568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"194","reported_epoch":"818","state":"active+clean","last_fresh":"2021-03-24T22:06:28.439527+0000","last_change":"2021-03-24T22:06:02.509466+0000","last_active":"2021-03-24T22:06:28.439527+0000","last_peered":"2021-03-24T22:06:28.439527+0000","last_clean":"2021-03-24T22:06:28.439527+0000","last_became_active":"2021-03-24T22:05:04.453406+0000","last_became_peered":"2021-03-24T22:05:04.453406+0000","last_unstale":"2021-03-24T22:06:28.439527+0000","last_undegraded":"2021-03-24T22:06:28.439527+0000","last_fullsized":"2021-03-24T22:06:28.439527+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:02.509401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:06:02.509401+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"179","reported_epoch":"818","state":"active+clean","last_fresh":"2021-03-24T22:06:28.436157+0000","last_change":"2021-03-24T22:04:56.575839+0000","last_active":"2021-03-24T22:06:28.436157+0000","last_peered":"2021-03-24T22:06:28.436157+0000","last_clean":"2021-03-24T22:06:28.436157+0000","last_became_active":"2021-03-24T22:03:45.426809+0000","last_became_peered":"2021-03-24T22:03:45.426809+0000","last_unstale":"2021-03-24T22:06:28.436157+0000","last_undegraded":"2021-03-24T22:06:28.436157+0000","last_fullsized":"2021-03-24T22:06:28.436157+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:04:56.575723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:04:56.575723+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"175'22","reported_seq":"2114","reported_epoch":"821","state":"active+clean","last_fresh":"2021-03-24T22:06:31.449562+0000","last_change":"2021-03-24T22:06:31.450587+0000","last_active":"2021-03-24T22:06:31.449562+0000","last_peered":"2021-03-24T22:06:31.449562+0000","last_clean":"2021-03-24T22:06:31.449562+0000","last_became_active":"2021-03-24T22:05:04.443450+0000","last_became_peered":"2021-03-24T22:05:04.443450+0000","last_unstale":"2021-03-24T22:06:31.449562+0000","last_undegraded":"2021-03-24T22:06:31.449562+0000","last_fullsized":"2021-03-24T22:06:31.449562+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'22","last_scrub_stamp":"2021-03-24T22:05:53.363157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:05:53.363157+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"25e","length":"1"},{"start":"28e","length":"1"},{"start":"2af","length":"1"},{"start":"2b3","length":"1"},{"start":"2b6","length":"1"}]},{"pgid":"5.7","version":"0'0","reported_seq":"655","reported_epoch":"820","state":"active+clean","last_fresh":"2021-03-24T22:06:30.444883+0000","last_change":"2021-03-24T22:05:14.400723+0000","last_active":"2021-03-24T22:06:30.444883+0000","last_peered":"2021-03-24T22:06:30.444883+0000","last_clean":"2021-03-24T22:06:30.444883+0000","last_became_active":"2021-03-24T22:05:04.439506+0000","last_became_peered":"2021-03-24T22:05:04.439506+0000","last_unstale":"2021-03-24T22:06:30.444883+0000","last_undegraded":"2021-03-24T22:06:30.444883+0000","last_fullsized":"2021-03-24T22:06:30.444883+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:12.146058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:05:12.146058+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"544'34","reported_seq":"1315","reported_epoch":"816","state":"active+clean","last_fresh":"2021-03-24T22:06:26.456769+0000","last_change":"2021-03-24T22:03:54.403711+0000","last_active":"2021-03-24T22:06:26.456769+0000","last_peered":"2021-03-24T22:06:26.456769+0000","last_clean":"2021-03-24T22:06:26.456769+0000","last_became_active":"2021-03-24T22:03:54.401857+0000","last_became_peered":"2021-03-24T22:03:54.401857+0000","last_unstale":"2021-03-24T22:06:26.456769+0000","last_undegraded":"2021-03-24T22:06:26.456769+0000","last_fullsized":"2021-03-24T22:06:26.456769+0000","mapping_epoch":663,"log_start":"69'16","ondisk_log_start":"69'16","created":62,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'34","last_scrub_stamp":"2021-03-24T22:02:59.519401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:02:59.519401+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1836,"num_objects":21,"num_object_clones":0,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":95,"num_read_kb":61,"num_write":51,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":157,"num_bytes_recovered":13482,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"66'5","reported_seq":"584","reported_epoch":"818","state":"active+clean","last_fresh":"2021-03-24T22:06:28.506585+0000","last_change":"2021-03-24T22:04:30.329098+0000","last_active":"2021-03-24T22:06:28.506585+0000","last_peered":"2021-03-24T22:06:28.506585+0000","last_clean":"2021-03-24T22:06:28.506585+0000","last_became_active":"2021-03-24T22:03:45.433742+0000","last_became_peered":"2021-03-24T22:03:45.433742+0000","last_unstale":"2021-03-24T22:06:28.506585+0000","last_undegraded":"2021-03-24T22:06:28.506585+0000","last_fullsized":"2021-03-24T22:06:28.506585+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2021-03-24T22:04:30.328893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:39.286101+0000","last_clean_scrub_stamp":"2021-03-24T22:04:30.328893+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":343,"num_read_kb":338,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"795","reported_epoch":"816","state":"active+clean","last_fresh":"2021-03-24T22:06:26.455036+0000","last_change":"2021-03-24T22:04:39.179364+0000","last_active":"2021-03-24T22:06:26.455036+0000","last_peered":"2021-03-24T22:06:26.455036+0000","last_clean":"2021-03-24T22:06:26.455036+0000","last_became_active":"2021-03-24T21:59:12.281282+0000","last_became_peered":"2021-03-24T21:59:12.281282+0000","last_unstale":"2021-03-24T22:06:26.455036+0000","last_undegraded":"2021-03-24T22:06:26.455036+0000","last_fullsized":"2021-03-24T22:06:26.455036+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:04:39.179121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:04:39.179121+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"179","reported_epoch":"818","state":"active+clean","last_fresh":"2021-03-24T22:06:28.439740+0000","last_change":"2021-03-24T22:05:44.537883+0000","last_active":"2021-03-24T22:06:28.439740+0000","last_peered":"2021-03-24T22:06:28.439740+0000","last_clean":"2021-03-24T22:06:28.439740+0000","last_became_active":"2021-03-24T22:03:45.424562+0000","last_became_peered":"2021-03-24T22:03:45.424562+0000","last_unstale":"2021-03-24T22:06:28.439740+0000","last_undegraded":"2021-03-24T22:06:28.439740+0000","last_fullsized":"2021-03-24T22:06:28.439740+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:44.537809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:05:44.537809+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"195'69","reported_seq":"855","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.007240+0000","last_change":"2021-03-24T22:06:26.453402+0000","last_active":"2021-03-24T22:06:28.007240+0000","last_peered":"2021-03-24T22:06:28.007240+0000","last_clean":"2021-03-24T22:06:28.007240+0000","last_became_active":"2021-03-24T22:01:57.317669+0000","last_became_peered":"2021-03-24T22:01:57.317669+0000","last_unstale":"2021-03-24T22:06:28.007240+0000","last_undegraded":"2021-03-24T22:06:28.007240+0000","last_fullsized":"2021-03-24T22:06:28.007240+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'69","last_scrub_stamp":"2021-03-24T22:04:58.198525+0000","last_deep_scrub":"195'69","last_deep_scrub_stamp":"2021-03-24T21:56:49.886058+0000","last_clean_scrub_stamp":"2021-03-24T22:04:58.198525+0000","log_size":69,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2a3","length":"1"},{"start":"2b3","length":"1"},{"start":"2b6","length":"1"}]},{"pgid":"5.4","version":"0'0","reported_seq":"305","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.007285+0000","last_change":"2021-03-24T22:05:14.123612+0000","last_active":"2021-03-24T22:06:28.007285+0000","last_peered":"2021-03-24T22:06:28.007285+0000","last_clean":"2021-03-24T22:06:28.007285+0000","last_became_active":"2021-03-24T22:05:04.451670+0000","last_became_peered":"2021-03-24T22:05:04.451670+0000","last_unstale":"2021-03-24T22:06:28.007285+0000","last_undegraded":"2021-03-24T22:06:28.007285+0000","last_fullsized":"2021-03-24T22:06:28.007285+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:14.123439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:05:14.123439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"783'36","reported_seq":"377","reported_epoch":"819","state":"active+clean","last_fresh":"2021-03-24T22:06:29.453482+0000","last_change":"2021-03-24T22:05:54.205230+0000","last_active":"2021-03-24T22:06:29.453482+0000","last_peered":"2021-03-24T22:06:29.453482+0000","last_clean":"2021-03-24T22:06:29.453482+0000","last_became_active":"2021-03-24T22:01:57.311205+0000","last_became_peered":"2021-03-24T22:01:57.311205+0000","last_unstale":"2021-03-24T22:06:29.453482+0000","last_undegraded":"2021-03-24T22:06:29.453482+0000","last_fullsized":"2021-03-24T22:06:29.453482+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"783'36","last_scrub_stamp":"2021-03-24T22:05:54.205108+0000","last_deep_scrub":"783'36","last_deep_scrub_stamp":"2021-03-24T22:05:54.205108+0000","last_clean_scrub_stamp":"2021-03-24T22:05:54.205108+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":27,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":1398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"824","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.443548+0000","last_change":"2021-03-24T22:05:04.443282+0000","last_active":"2021-03-24T22:06:28.443548+0000","last_peered":"2021-03-24T22:06:28.443548+0000","last_clean":"2021-03-24T22:06:28.443548+0000","last_became_active":"2021-03-24T22:05:04.441717+0000","last_became_peered":"2021-03-24T22:05:04.441717+0000","last_unstale":"2021-03-24T22:06:28.443548+0000","last_undegraded":"2021-03-24T22:06:28.443548+0000","last_fullsized":"2021-03-24T22:06:28.443548+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:33.533840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:33.533840+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"288","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.006351+0000","last_change":"2021-03-24T22:05:59.171885+0000","last_active":"2021-03-24T22:06:28.006351+0000","last_peered":"2021-03-24T22:06:28.006351+0000","last_clean":"2021-03-24T22:06:28.006351+0000","last_became_active":"2021-03-24T22:01:57.315729+0000","last_became_peered":"2021-03-24T22:01:57.315729+0000","last_unstale":"2021-03-24T22:06:28.006351+0000","last_undegraded":"2021-03-24T22:06:28.006351+0000","last_fullsized":"2021-03-24T22:06:28.006351+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:59.171797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:49:51.529051+0000","last_clean_scrub_stamp":"2021-03-24T22:05:59.171797+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"607","reported_epoch":"820","state":"active+clean","last_fresh":"2021-03-24T22:06:30.445156+0000","last_change":"2021-03-24T22:05:49.381846+0000","last_active":"2021-03-24T22:06:30.445156+0000","last_peered":"2021-03-24T22:06:30.445156+0000","last_clean":"2021-03-24T22:06:30.445156+0000","last_became_active":"2021-03-24T22:05:04.435232+0000","last_became_peered":"2021-03-24T22:05:04.435232+0000","last_unstale":"2021-03-24T22:06:30.445156+0000","last_undegraded":"2021-03-24T22:06:30.445156+0000","last_fullsized":"2021-03-24T22:06:30.445156+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:49.381569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:05:49.381569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"816'429","reported_seq":"2961","reported_epoch":"816","state":"active+clean","last_fresh":"2021-03-24T22:06:26.462992+0000","last_change":"2021-03-24T22:06:26.462992+0000","last_active":"2021-03-24T22:06:26.462992+0000","last_peered":"2021-03-24T22:06:26.462992+0000","last_clean":"2021-03-24T22:06:26.462992+0000","last_became_active":"2021-03-24T22:02:31.331916+0000","last_became_peered":"2021-03-24T22:02:31.331916+0000","last_unstale":"2021-03-24T22:06:26.462992+0000","last_undegraded":"2021-03-24T22:06:26.462992+0000","last_fullsized":"2021-03-24T22:06:26.462992+0000","mapping_epoch":580,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":581,"parent":"0.0","parent_split_bits":0,"last_scrub":"783'396","last_scrub_stamp":"2021-03-24T22:05:56.358947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:05:56.358947+0000","log_size":429,"ondisk_log_size":429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18272400,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":16742,"num_write":6,"num_write_kb":5404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2a3","length":"1"},{"start":"2b3","length":"1"},{"start":"2b6","length":"1"}]},{"pgid":"5.6","version":"0'0","reported_seq":"281","reported_epoch":"820","state":"active+clean","last_fresh":"2021-03-24T22:06:30.437946+0000","last_change":"2021-03-24T22:05:51.038309+0000","last_active":"2021-03-24T22:06:30.437946+0000","last_peered":"2021-03-24T22:06:30.437946+0000","last_clean":"2021-03-24T22:06:30.437946+0000","last_became_active":"2021-03-24T22:05:04.423152+0000","last_became_peered":"2021-03-24T22:05:04.423152+0000","last_unstale":"2021-03-24T22:06:30.437946+0000","last_undegraded":"2021-03-24T22:06:30.437946+0000","last_fullsized":"2021-03-24T22:06:30.437946+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:51.038089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:05:51.038089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"544'32","reported_seq":"544","reported_epoch":"816","state":"active+remapped+backfill_toofull","last_fresh":"2021-03-24T22:06:26.457557+0000","last_change":"2021-03-24T22:06:07.453465+0000","last_active":"2021-03-24T22:06:26.457557+0000","last_peered":"2021-03-24T22:06:26.457557+0000","last_clean":"2021-03-24T22:00:59.408047+0000","last_became_active":"2021-03-24T22:06:07.450002+0000","last_became_peered":"2021-03-24T22:06:07.450002+0000","last_unstale":"2021-03-24T22:06:26.457557+0000","last_undegraded":"2021-03-24T22:06:26.457557+0000","last_fullsized":"2021-03-24T22:06:26.457557+0000","mapping_epoch":796,"log_start":"89'22","ondisk_log_start":"89'22","created":62,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'32","last_scrub_stamp":"2021-03-24T22:05:13.446721+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2021-03-24T21:53:19.717244+0000","last_clean_scrub_stamp":"2021-03-24T22:05:13.446721+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":802,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":42,"num_write":35,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1118,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[0,5,1],"avail_no_missing":["0","1","5"],"object_location_counts":[{"shards":"0,1,5","objects":11}],"blocked_by":[],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"801","reported_epoch":"820","state":"active+clean","last_fresh":"2021-03-24T22:06:30.435700+0000","last_change":"2021-03-24T22:04:37.814851+0000","last_active":"2021-03-24T22:06:30.435700+0000","last_peered":"2021-03-24T22:06:30.435700+0000","last_clean":"2021-03-24T22:06:30.435700+0000","last_became_active":"2021-03-24T21:51:40.351000+0000","last_became_peered":"2021-03-24T21:51:40.351000+0000","last_unstale":"2021-03-24T22:06:30.435700+0000","last_undegraded":"2021-03-24T22:06:30.435700+0000","last_fullsized":"2021-03-24T22:06:30.435700+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:04:37.814761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:39.286101+0000","last_clean_scrub_stamp":"2021-03-24T22:04:37.814761+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"61'1","reported_seq":"179","reported_epoch":"818","state":"active+clean","last_fresh":"2021-03-24T22:06:28.436986+0000","last_change":"2021-03-24T22:04:34.667264+0000","last_active":"2021-03-24T22:06:28.436986+0000","last_peered":"2021-03-24T22:06:28.436986+0000","last_clean":"2021-03-24T22:06:28.436986+0000","last_became_active":"2021-03-24T22:03:45.426685+0000","last_became_peered":"2021-03-24T22:03:45.426685+0000","last_unstale":"2021-03-24T22:06:28.436986+0000","last_undegraded":"2021-03-24T22:06:28.436986+0000","last_fullsized":"2021-03-24T22:06:28.436986+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2021-03-24T22:04:34.667176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:04:34.667176+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"288","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.007536+0000","last_change":"2021-03-24T22:05:50.105714+0000","last_active":"2021-03-24T22:06:28.007536+0000","last_peered":"2021-03-24T22:06:28.007536+0000","last_clean":"2021-03-24T22:06:28.007536+0000","last_became_active":"2021-03-24T22:01:57.318275+0000","last_became_peered":"2021-03-24T22:01:57.318275+0000","last_unstale":"2021-03-24T22:06:28.007536+0000","last_undegraded":"2021-03-24T22:06:28.007536+0000","last_fullsized":"2021-03-24T22:06:28.007536+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:50.105616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:05:50.105616+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"808'485","reported_seq":"2926","reported_epoch":"816","state":"active+clean","last_fresh":"2021-03-24T22:06:26.458281+0000","last_change":"2021-03-24T22:06:26.458281+0000","last_active":"2021-03-24T22:06:26.458281+0000","last_peered":"2021-03-24T22:06:26.458281+0000","last_clean":"2021-03-24T22:06:26.458281+0000","last_became_active":"2021-03-24T22:03:50.369486+0000","last_became_peered":"2021-03-24T22:03:50.369486+0000","last_unstale":"2021-03-24T22:06:26.458281+0000","last_undegraded":"2021-03-24T22:06:26.458281+0000","last_fullsized":"2021-03-24T22:06:26.458281+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"616'351","last_scrub_stamp":"2021-03-24T22:03:06.473123+0000","last_deep_scrub":"616'351","last_deep_scrub_stamp":"2021-03-24T22:03:06.473123+0000","last_clean_scrub_stamp":"2021-03-24T22:03:06.473123+0000","log_size":485,"ondisk_log_size":485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10321090,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":96,"num_read_kb":102337,"num_write":61,"num_write_kb":23603,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":35984561,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2a3","length":"1"},{"start":"2b3","length":"1"},{"start":"2b6","length":"1"}]},{"pgid":"5.5","version":"0'0","reported_seq":"831","reported_epoch":"816","state":"active+clean","last_fresh":"2021-03-24T22:06:26.457436+0000","last_change":"2021-03-24T22:04:27.142762+0000","last_active":"2021-03-24T22:06:26.457436+0000","last_peered":"2021-03-24T22:06:26.457436+0000","last_clean":"2021-03-24T22:06:26.457436+0000","last_became_active":"2021-03-24T22:03:45.430121+0000","last_became_peered":"2021-03-24T22:03:45.430121+0000","last_unstale":"2021-03-24T22:06:26.457436+0000","last_undegraded":"2021-03-24T22:06:26.457436+0000","last_fullsized":"2021-03-24T22:06:26.457436+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:04:27.142661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:04:27.142661+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"544'37","reported_seq":"207","reported_epoch":"818","state":"active+clean","last_fresh":"2021-03-24T22:06:28.439430+0000","last_change":"2021-03-24T22:04:58.618464+0000","last_active":"2021-03-24T22:06:28.439430+0000","last_peered":"2021-03-24T22:06:28.439430+0000","last_clean":"2021-03-24T22:06:28.439430+0000","last_became_active":"2021-03-24T22:03:45.426862+0000","last_became_peered":"2021-03-24T22:03:45.426862+0000","last_unstale":"2021-03-24T22:06:28.439430+0000","last_undegraded":"2021-03-24T22:06:28.439430+0000","last_fullsized":"2021-03-24T22:06:28.439430+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'37","last_scrub_stamp":"2021-03-24T22:04:58.618374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:04:58.618374+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":70,"num_read_kb":47,"num_write":40,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":5724,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"89'2","reported_seq":"419","reported_epoch":"816","state":"active+clean","last_fresh":"2021-03-24T22:06:26.457768+0000","last_change":"2021-03-24T22:05:04.320737+0000","last_active":"2021-03-24T22:06:26.457768+0000","last_peered":"2021-03-24T22:06:26.457768+0000","last_clean":"2021-03-24T22:06:26.457768+0000","last_became_active":"2021-03-24T22:01:43.393718+0000","last_became_peered":"2021-03-24T22:01:43.393718+0000","last_unstale":"2021-03-24T22:06:26.457768+0000","last_undegraded":"2021-03-24T22:06:26.457768+0000","last_fullsized":"2021-03-24T22:06:26.457768+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2021-03-24T22:05:04.320667+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2021-03-24T21:57:58.491830+0000","last_clean_scrub_stamp":"2021-03-24T22:05:04.320667+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"194","reported_epoch":"818","state":"active+clean","last_fresh":"2021-03-24T22:06:28.436484+0000","last_change":"2021-03-24T22:05:10.501601+0000","last_active":"2021-03-24T22:06:28.436484+0000","last_peered":"2021-03-24T22:06:28.436484+0000","last_clean":"2021-03-24T22:06:28.436484+0000","last_became_active":"2021-03-24T22:05:04.450107+0000","last_became_peered":"2021-03-24T22:05:04.450107+0000","last_unstale":"2021-03-24T22:06:28.436484+0000","last_undegraded":"2021-03-24T22:06:28.436484+0000","last_fullsized":"2021-03-24T22:06:28.436484+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:10.501490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:06.800417+0000","last_clean_scrub_stamp":"2021-03-24T22:05:10.501490+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"830","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.444171+0000","last_change":"2021-03-24T22:02:55.490903+0000","last_active":"2021-03-24T22:06:28.444171+0000","last_peered":"2021-03-24T22:06:28.444171+0000","last_clean":"2021-03-24T22:06:28.444171+0000","last_became_active":"2021-03-24T22:01:57.314372+0000","last_became_peered":"2021-03-24T22:01:57.314372+0000","last_unstale":"2021-03-24T22:06:28.444171+0000","last_undegraded":"2021-03-24T22:06:28.444171+0000","last_fullsized":"2021-03-24T22:06:28.444171+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:55.490744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:52:59.706511+0000","last_clean_scrub_stamp":"2021-03-24T22:02:55.490744+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"820'620","reported_seq":"3005","reported_epoch":"820","state":"active+clean","last_fresh":"2021-03-24T22:06:30.466500+0000","last_change":"2021-03-24T22:06:30.466918+0000","last_active":"2021-03-24T22:06:30.466500+0000","last_peered":"2021-03-24T22:06:30.466500+0000","last_clean":"2021-03-24T22:06:30.466500+0000","last_became_active":"2021-03-24T22:01:47.305050+0000","last_became_peered":"2021-03-24T22:01:47.305050+0000","last_unstale":"2021-03-24T22:06:30.466500+0000","last_undegraded":"2021-03-24T22:06:30.466500+0000","last_fullsized":"2021-03-24T22:06:30.466500+0000","mapping_epoch":536,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":537,"parent":"0.0","parent_split_bits":0,"last_scrub":"699'500","last_scrub_stamp":"2021-03-24T22:04:31.592343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:04:31.592343+0000","log_size":620,"ondisk_log_size":620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7767212,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":7099,"num_write":67,"num_write_kb":23311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":55849492,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"25e","length":"1"},{"start":"2af","length":"1"},{"start":"2b3","length":"1"},{"start":"2b6","length":"1"}]},{"pgid":"4.b","version":"544'17","reported_seq":"937","reported_epoch":"816","state":"active+clean","last_fresh":"2021-03-24T22:06:26.457371+0000","last_change":"2021-03-24T22:04:34.106367+0000","last_active":"2021-03-24T22:06:26.457371+0000","last_peered":"2021-03-24T22:06:26.457371+0000","last_clean":"2021-03-24T22:06:26.457371+0000","last_became_active":"2021-03-24T22:03:46.443398+0000","last_became_peered":"2021-03-24T22:03:46.443398+0000","last_unstale":"2021-03-24T22:06:26.457371+0000","last_undegraded":"2021-03-24T22:06:26.457371+0000","last_fullsized":"2021-03-24T22:06:26.457371+0000","mapping_epoch":655,"log_start":"91'9","ondisk_log_start":"91'9","created":62,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'17","last_scrub_stamp":"2021-03-24T22:04:34.106163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:04:34.106163+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1158,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":58,"num_read_kb":36,"num_write":30,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2316,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"820","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.445039+0000","last_change":"2021-03-24T22:05:04.453096+0000","last_active":"2021-03-24T22:06:28.445039+0000","last_peered":"2021-03-24T22:06:28.445039+0000","last_clean":"2021-03-24T22:06:28.445039+0000","last_became_active":"2021-03-24T22:05:04.446424+0000","last_became_peered":"2021-03-24T22:05:04.446424+0000","last_unstale":"2021-03-24T22:06:28.445039+0000","last_undegraded":"2021-03-24T22:06:28.445039+0000","last_fullsized":"2021-03-24T22:06:28.445039+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:01:55.431129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T22:01:55.431129+0000","last_clean_scrub_stamp":"2021-03-24T22:01:55.431129+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"793","reported_epoch":"820","state":"active+clean","last_fresh":"2021-03-24T22:06:30.436048+0000","last_change":"2021-03-24T22:06:01.013669+0000","last_active":"2021-03-24T22:06:30.436048+0000","last_peered":"2021-03-24T22:06:30.436048+0000","last_clean":"2021-03-24T22:06:30.436048+0000","last_became_active":"2021-03-24T21:51:49.411921+0000","last_became_peered":"2021-03-24T21:51:49.411921+0000","last_unstale":"2021-03-24T22:06:30.436048+0000","last_undegraded":"2021-03-24T22:06:30.436048+0000","last_fullsized":"2021-03-24T22:06:30.436048+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:01.013563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:06:01.013563+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"838","reported_epoch":"816","state":"active+clean","last_fresh":"2021-03-24T22:06:26.454399+0000","last_change":"2021-03-24T22:04:31.107367+0000","last_active":"2021-03-24T22:06:26.454399+0000","last_peered":"2021-03-24T22:06:26.454399+0000","last_clean":"2021-03-24T22:06:26.454399+0000","last_became_active":"2021-03-24T22:02:22.323876+0000","last_became_peered":"2021-03-24T22:02:22.323876+0000","last_unstale":"2021-03-24T22:06:26.454399+0000","last_undegraded":"2021-03-24T22:06:26.454399+0000","last_fullsized":"2021-03-24T22:06:26.454399+0000","mapping_epoch":571,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:04:31.107254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:35.636299+0000","last_clean_scrub_stamp":"2021-03-24T22:04:31.107254+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"821'333","reported_seq":"2744","reported_epoch":"821","state":"active+clean","last_fresh":"2021-03-24T22:06:31.452021+0000","last_change":"2021-03-24T22:06:31.458489+0000","last_active":"2021-03-24T22:06:31.452021+0000","last_peered":"2021-03-24T22:06:31.452021+0000","last_clean":"2021-03-24T22:06:31.452021+0000","last_became_active":"2021-03-24T22:02:01.325483+0000","last_became_peered":"2021-03-24T22:02:01.325483+0000","last_unstale":"2021-03-24T22:06:31.452021+0000","last_undegraded":"2021-03-24T22:06:31.452021+0000","last_fullsized":"2021-03-24T22:06:31.452021+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"792'330","last_scrub_stamp":"2021-03-24T22:06:03.024091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:06:03.024091+0000","log_size":333,"ondisk_log_size":333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5244246,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":87,"num_read_kb":99622,"num_write":63,"num_write_kb":17849,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":21446378,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"25e","length":"1"},{"start":"28e","length":"1"},{"start":"2af","length":"1"},{"start":"2b3","length":"1"},{"start":"2b6","length":"1"}]},{"pgid":"4.a","version":"544'45","reported_seq":"786","reported_epoch":"819","state":"active+clean","last_fresh":"2021-03-24T22:06:29.447051+0000","last_change":"2021-03-24T22:04:26.640197+0000","last_active":"2021-03-24T22:06:29.447051+0000","last_peered":"2021-03-24T22:06:29.447051+0000","last_clean":"2021-03-24T22:06:29.447051+0000","last_became_active":"2021-03-24T22:04:26.561945+0000","last_became_peered":"2021-03-24T22:04:26.561945+0000","last_unstale":"2021-03-24T22:06:29.447051+0000","last_undegraded":"2021-03-24T22:06:29.447051+0000","last_fullsized":"2021-03-24T22:06:29.447051+0000","mapping_epoch":695,"log_start":"69'23","ondisk_log_start":"69'23","created":62,"last_epoch_clean":696,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'45","last_scrub_stamp":"2021-03-24T22:04:26.640132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:04:26.640132+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1466,"num_objects":21,"num_object_clones":0,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":104,"num_read_kb":71,"num_write":60,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":7174,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":"443","reported_epoch":"820","state":"active+clean","last_fresh":"2021-03-24T22:06:30.445355+0000","last_change":"2021-03-24T22:05:51.326820+0000","last_active":"2021-03-24T22:06:30.445355+0000","last_peered":"2021-03-24T22:06:30.445355+0000","last_clean":"2021-03-24T22:06:30.445355+0000","last_became_active":"2021-03-24T22:05:04.439833+0000","last_became_peered":"2021-03-24T22:05:04.439833+0000","last_unstale":"2021-03-24T22:06:30.445355+0000","last_undegraded":"2021-03-24T22:06:30.445355+0000","last_fullsized":"2021-03-24T22:06:30.445355+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:51.326637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:05:51.326637+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"276","reported_epoch":"820","state":"active+clean","last_fresh":"2021-03-24T22:06:30.434003+0000","last_change":"2021-03-24T22:05:08.670410+0000","last_active":"2021-03-24T22:06:30.434003+0000","last_peered":"2021-03-24T22:06:30.434003+0000","last_clean":"2021-03-24T22:06:30.434003+0000","last_became_active":"2021-03-24T22:05:04.426559+0000","last_became_peered":"2021-03-24T22:05:04.426559+0000","last_unstale":"2021-03-24T22:06:30.434003+0000","last_undegraded":"2021-03-24T22:06:30.434003+0000","last_fullsized":"2021-03-24T22:06:30.434003+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:08.670240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T22:05:08.670240+0000","last_clean_scrub_stamp":"2021-03-24T22:05:08.670240+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"586","reported_epoch":"820","state":"active+clean","last_fresh":"2021-03-24T22:06:30.433600+0000","last_change":"2021-03-24T22:05:55.032235+0000","last_active":"2021-03-24T22:06:30.433600+0000","last_peered":"2021-03-24T22:06:30.433600+0000","last_clean":"2021-03-24T22:06:30.433600+0000","last_became_active":"2021-03-24T22:01:57.311325+0000","last_became_peered":"2021-03-24T22:01:57.311325+0000","last_unstale":"2021-03-24T22:06:30.433600+0000","last_undegraded":"2021-03-24T22:06:30.433600+0000","last_fullsized":"2021-03-24T22:06:30.433600+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:55.032073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:05:55.032073+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"808'546","reported_seq":"2966","reported_epoch":"816","state":"active+clean","last_fresh":"2021-03-24T22:06:26.454781+0000","last_change":"2021-03-24T22:06:26.454781+0000","last_active":"2021-03-24T22:06:26.454781+0000","last_peered":"2021-03-24T22:06:26.454781+0000","last_clean":"2021-03-24T22:06:26.454781+0000","last_became_active":"2021-03-24T22:03:49.375663+0000","last_became_peered":"2021-03-24T22:03:49.375663+0000","last_unstale":"2021-03-24T22:06:26.454781+0000","last_undegraded":"2021-03-24T22:06:26.454781+0000","last_fullsized":"2021-03-24T22:06:26.454781+0000","mapping_epoch":658,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":659,"parent":"0.0","parent_split_bits":0,"last_scrub":"795'538","last_scrub_stamp":"2021-03-24T22:06:05.470009+0000","last_deep_scrub":"795'538","last_deep_scrub_stamp":"2021-03-24T22:06:05.470009+0000","last_clean_scrub_stamp":"2021-03-24T22:06:05.470009+0000","log_size":546,"ondisk_log_size":546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7915421,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":71,"num_read_kb":81142,"num_write":64,"num_write_kb":18343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":32161412,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2a3","length":"1"},{"start":"2b3","length":"1"},{"start":"2b6","length":"1"}]},{"pgid":"5.8","version":"0'0","reported_seq":"846","reported_epoch":"820","state":"active+clean","last_fresh":"2021-03-24T22:06:30.432168+0000","last_change":"2021-03-24T22:06:04.053700+0000","last_active":"2021-03-24T22:06:30.432168+0000","last_peered":"2021-03-24T22:06:30.432168+0000","last_clean":"2021-03-24T22:06:30.432168+0000","last_became_active":"2021-03-24T22:03:45.432819+0000","last_became_peered":"2021-03-24T22:03:45.432819+0000","last_unstale":"2021-03-24T22:06:30.432168+0000","last_undegraded":"2021-03-24T22:06:30.432168+0000","last_fullsized":"2021-03-24T22:06:30.432168+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:04.053530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:06:04.053530+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"544'38","reported_seq":"525","reported_epoch":"818","state":"active+clean","last_fresh":"2021-03-24T22:06:28.435980+0000","last_change":"2021-03-24T22:05:00.634636+0000","last_active":"2021-03-24T22:06:28.435980+0000","last_peered":"2021-03-24T22:06:28.435980+0000","last_clean":"2021-03-24T22:06:28.435980+0000","last_became_active":"2021-03-24T22:04:50.402497+0000","last_became_peered":"2021-03-24T22:04:50.402497+0000","last_unstale":"2021-03-24T22:06:28.435980+0000","last_undegraded":"2021-03-24T22:06:28.435980+0000","last_fullsized":"2021-03-24T22:06:28.435980+0000","mapping_epoch":719,"log_start":"69'20","ondisk_log_start":"69'20","created":62,"last_epoch_clean":720,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'38","last_scrub_stamp":"2021-03-24T22:05:00.634536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:05:00.634536+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1582,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":100,"num_read_kb":66,"num_write":56,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":4746,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":"301","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.008003+0000","last_change":"2021-03-24T22:04:33.248733+0000","last_active":"2021-03-24T22:06:28.008003+0000","last_peered":"2021-03-24T22:06:28.008003+0000","last_clean":"2021-03-24T22:06:28.008003+0000","last_became_active":"2021-03-24T22:03:45.437360+0000","last_became_peered":"2021-03-24T22:03:45.437360+0000","last_unstale":"2021-03-24T22:06:28.008003+0000","last_undegraded":"2021-03-24T22:06:28.008003+0000","last_fullsized":"2021-03-24T22:06:28.008003+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:04:33.248542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:04:33.248542+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"852","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.445287+0000","last_change":"2021-03-24T22:05:14.248048+0000","last_active":"2021-03-24T22:06:28.445287+0000","last_peered":"2021-03-24T22:06:28.445287+0000","last_clean":"2021-03-24T22:06:28.445287+0000","last_became_active":"2021-03-24T22:03:45.436096+0000","last_became_peered":"2021-03-24T22:03:45.436096+0000","last_unstale":"2021-03-24T22:06:28.445287+0000","last_undegraded":"2021-03-24T22:06:28.445287+0000","last_fullsized":"2021-03-24T22:06:28.445287+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:14.247913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:22.724840+0000","last_clean_scrub_stamp":"2021-03-24T22:05:14.247913+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"813'390","reported_seq":"2622","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.443652+0000","last_change":"2021-03-24T22:06:26.444095+0000","last_active":"2021-03-24T22:06:28.443652+0000","last_peered":"2021-03-24T22:06:28.443652+0000","last_clean":"2021-03-24T22:06:28.443652+0000","last_became_active":"2021-03-24T22:05:04.442753+0000","last_became_peered":"2021-03-24T22:05:04.442753+0000","last_unstale":"2021-03-24T22:06:28.443652+0000","last_undegraded":"2021-03-24T22:06:28.443652+0000","last_fullsized":"2021-03-24T22:06:28.443652+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"624'277","last_scrub_stamp":"2021-03-24T22:03:14.428074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:03:14.428074+0000","log_size":390,"ondisk_log_size":390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7500172,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":42,"num_read_kb":48474,"num_write":15,"num_write_kb":7826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1181773,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2a3","length":"1"},{"start":"2b3","length":"1"},{"start":"2b6","length":"1"}]},{"pgid":"5.9","version":"548'11","reported_seq":"341","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.006850+0000","last_change":"2021-03-24T22:04:28.355884+0000","last_active":"2021-03-24T22:06:28.006850+0000","last_peered":"2021-03-24T22:06:28.006850+0000","last_clean":"2021-03-24T22:06:28.006850+0000","last_became_active":"2021-03-24T22:01:57.313563+0000","last_became_peered":"2021-03-24T22:01:57.313563+0000","last_unstale":"2021-03-24T22:06:28.006850+0000","last_undegraded":"2021-03-24T22:06:28.006850+0000","last_fullsized":"2021-03-24T22:06:28.006850+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"548'11","last_scrub_stamp":"2021-03-24T22:04:28.355700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:04:28.355700+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"544'46","reported_seq":"802","reported_epoch":"820","state":"active+clean","last_fresh":"2021-03-24T22:06:30.443708+0000","last_change":"2021-03-24T22:06:07.382916+0000","last_active":"2021-03-24T22:06:30.443708+0000","last_peered":"2021-03-24T22:06:30.443708+0000","last_clean":"2021-03-24T22:06:30.443708+0000","last_became_active":"2021-03-24T22:05:05.422281+0000","last_became_peered":"2021-03-24T22:05:05.422281+0000","last_unstale":"2021-03-24T22:06:30.443708+0000","last_undegraded":"2021-03-24T22:06:30.443708+0000","last_fullsized":"2021-03-24T22:06:30.443708+0000","mapping_epoch":734,"log_start":"97'28","ondisk_log_start":"97'28","created":62,"last_epoch_clean":735,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'46","last_scrub_stamp":"2021-03-24T22:06:07.382720+0000","last_deep_scrub":"97'28","last_deep_scrub_stamp":"2021-03-24T21:54:19.157265+0000","last_clean_scrub_stamp":"2021-03-24T22:06:07.382720+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1616,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":96,"num_read_kb":63,"num_write":53,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":5308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"89'4","reported_seq":"294","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.008102+0000","last_change":"2021-03-24T22:06:04.178234+0000","last_active":"2021-03-24T22:06:28.008102+0000","last_peered":"2021-03-24T22:06:28.008102+0000","last_clean":"2021-03-24T22:06:28.008102+0000","last_became_active":"2021-03-24T22:01:57.316325+0000","last_became_peered":"2021-03-24T22:01:57.316325+0000","last_unstale":"2021-03-24T22:06:28.008102+0000","last_undegraded":"2021-03-24T22:06:28.008102+0000","last_fullsized":"2021-03-24T22:06:28.008102+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'4","last_scrub_stamp":"2021-03-24T22:06:04.178049+0000","last_deep_scrub":"89'4","last_deep_scrub_stamp":"2021-03-24T21:53:19.926637+0000","last_clean_scrub_stamp":"2021-03-24T22:06:04.178049+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"316","reported_epoch":"819","state":"active+clean","last_fresh":"2021-03-24T22:06:29.447517+0000","last_change":"2021-03-24T22:05:15.379559+0000","last_active":"2021-03-24T22:06:29.447517+0000","last_peered":"2021-03-24T22:06:29.447517+0000","last_clean":"2021-03-24T22:06:29.447517+0000","last_became_active":"2021-03-24T22:05:04.444421+0000","last_became_peered":"2021-03-24T22:05:04.444421+0000","last_unstale":"2021-03-24T22:06:29.447517+0000","last_undegraded":"2021-03-24T22:06:29.447517+0000","last_fullsized":"2021-03-24T22:06:29.447517+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:15.379478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:05:15.379478+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"272'275","reported_seq":"2105","reported_epoch":"819","state":"active+clean","last_fresh":"2021-03-24T22:06:29.438904+0000","last_change":"2021-03-24T22:06:29.439450+0000","last_active":"2021-03-24T22:06:29.438904+0000","last_peered":"2021-03-24T22:06:29.438904+0000","last_clean":"2021-03-24T22:06:29.438904+0000","last_became_active":"2021-03-24T22:05:04.425270+0000","last_became_peered":"2021-03-24T22:05:04.425270+0000","last_unstale":"2021-03-24T22:06:29.438904+0000","last_undegraded":"2021-03-24T22:06:29.438904+0000","last_fullsized":"2021-03-24T22:06:29.438904+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'275","last_scrub_stamp":"2021-03-24T22:05:59.526944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:05:59.526944+0000","log_size":275,"ondisk_log_size":275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2a3","length":"1"},{"start":"2af","length":"1"},{"start":"2b3","length":"1"},{"start":"2b6","length":"1"}]},{"pgid":"3.10","version":"0'0","reported_seq":"318","reported_epoch":"819","state":"active+clean","last_fresh":"2021-03-24T22:06:29.447127+0000","last_change":"2021-03-24T22:05:47.360975+0000","last_active":"2021-03-24T22:06:29.447127+0000","last_peered":"2021-03-24T22:06:29.447127+0000","last_clean":"2021-03-24T22:06:29.447127+0000","last_became_active":"2021-03-24T22:05:04.440682+0000","last_became_peered":"2021-03-24T22:05:04.440682+0000","last_unstale":"2021-03-24T22:06:29.447127+0000","last_undegraded":"2021-03-24T22:06:29.447127+0000","last_fullsized":"2021-03-24T22:06:29.447127+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:47.360795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:05:47.360795+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"671","reported_epoch":"820","state":"active+clean","last_fresh":"2021-03-24T22:06:30.445040+0000","last_change":"2021-03-24T22:06:02.409659+0000","last_active":"2021-03-24T22:06:30.445040+0000","last_peered":"2021-03-24T22:06:30.445040+0000","last_clean":"2021-03-24T22:06:30.445040+0000","last_became_active":"2021-03-24T22:05:04.437651+0000","last_became_peered":"2021-03-24T22:05:04.437651+0000","last_unstale":"2021-03-24T22:06:30.445040+0000","last_undegraded":"2021-03-24T22:06:30.445040+0000","last_fullsized":"2021-03-24T22:06:30.445040+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:02.409388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T22:06:02.409388+0000","last_clean_scrub_stamp":"2021-03-24T22:06:02.409388+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"288","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.007359+0000","last_change":"2021-03-24T22:05:57.160116+0000","last_active":"2021-03-24T22:06:28.007359+0000","last_peered":"2021-03-24T22:06:28.007359+0000","last_clean":"2021-03-24T22:06:28.007359+0000","last_became_active":"2021-03-24T22:01:57.312376+0000","last_became_peered":"2021-03-24T22:01:57.312376+0000","last_unstale":"2021-03-24T22:06:28.007359+0000","last_undegraded":"2021-03-24T22:06:28.007359+0000","last_fullsized":"2021-03-24T22:06:28.007359+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:57.159943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:05:57.159943+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.14","version":"816'1064","reported_seq":"3341","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.444744+0000","last_change":"2021-03-24T22:06:26.447791+0000","last_active":"2021-03-24T22:06:28.444744+0000","last_peered":"2021-03-24T22:06:28.444744+0000","last_clean":"2021-03-24T22:06:28.444744+0000","last_became_active":"2021-03-24T22:05:04.453660+0000","last_became_peered":"2021-03-24T22:05:04.453660+0000","last_unstale":"2021-03-24T22:06:28.444744+0000","last_undegraded":"2021-03-24T22:06:28.444744+0000","last_fullsized":"2021-03-24T22:06:28.444744+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":735,"parent":"0.0","parent_split_bits":5,"last_scrub":"651'772","last_scrub_stamp":"2021-03-24T22:03:42.250616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:03:42.250616+0000","log_size":1064,"ondisk_log_size":1064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36590799,"num_objects":16,"num_object_clones":13,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":1,"num_read":101,"num_read_kb":113196,"num_write":65,"num_write_kb":28283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":14235619,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2a3","length":"1"},{"start":"2b3","length":"1"},{"start":"2b6","length":"1"}]},{"pgid":"3.11","version":"0'0","reported_seq":"310","reported_epoch":"819","state":"active+clean","last_fresh":"2021-03-24T22:06:29.447219+0000","last_change":"2021-03-24T22:04:32.551783+0000","last_active":"2021-03-24T22:06:29.447219+0000","last_peered":"2021-03-24T22:06:29.447219+0000","last_clean":"2021-03-24T22:06:29.447219+0000","last_became_active":"2021-03-24T22:01:57.304423+0000","last_became_peered":"2021-03-24T22:01:57.304423+0000","last_unstale":"2021-03-24T22:06:29.447219+0000","last_undegraded":"2021-03-24T22:06:29.447219+0000","last_fullsized":"2021-03-24T22:06:29.447219+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:04:32.551682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:04:32.551682+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"848","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.444846+0000","last_change":"2021-03-24T22:03:45.435984+0000","last_active":"2021-03-24T22:06:28.444846+0000","last_peered":"2021-03-24T22:06:28.444846+0000","last_clean":"2021-03-24T22:06:28.444846+0000","last_became_active":"2021-03-24T22:03:45.434516+0000","last_became_peered":"2021-03-24T22:03:45.434516+0000","last_unstale":"2021-03-24T22:06:28.444846+0000","last_undegraded":"2021-03-24T22:06:28.444846+0000","last_fullsized":"2021-03-24T22:06:28.444846+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:56.523309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:55.493713+0000","last_clean_scrub_stamp":"2021-03-24T22:02:56.523309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"822","reported_epoch":"820","state":"active+clean","last_fresh":"2021-03-24T22:06:30.434182+0000","last_change":"2021-03-24T22:04:33.896133+0000","last_active":"2021-03-24T22:06:30.434182+0000","last_peered":"2021-03-24T22:06:30.434182+0000","last_clean":"2021-03-24T22:06:30.434182+0000","last_became_active":"2021-03-24T22:01:57.312185+0000","last_became_peered":"2021-03-24T22:01:57.312185+0000","last_unstale":"2021-03-24T22:06:30.434182+0000","last_undegraded":"2021-03-24T22:06:30.434182+0000","last_fullsized":"2021-03-24T22:06:30.434182+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:04:33.895981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:04:33.895981+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"816'409","reported_seq":"2918","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.007588+0000","last_change":"2021-03-24T22:06:26.456257+0000","last_active":"2021-03-24T22:06:28.007588+0000","last_peered":"2021-03-24T22:06:28.007588+0000","last_clean":"2021-03-24T22:06:28.007588+0000","last_became_active":"2021-03-24T22:03:51.363965+0000","last_became_peered":"2021-03-24T22:03:51.363965+0000","last_unstale":"2021-03-24T22:06:28.007588+0000","last_undegraded":"2021-03-24T22:06:28.007588+0000","last_fullsized":"2021-03-24T22:06:28.007588+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":5,"last_scrub":"710'317","last_scrub_stamp":"2021-03-24T22:04:41.171323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:04:41.171323+0000","log_size":409,"ondisk_log_size":409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14082423,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":83,"num_read_kb":115822,"num_write":62,"num_write_kb":23983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":85409227,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2a3","length":"1"},{"start":"2b3","length":"1"},{"start":"2b6","length":"1"}]},{"pgid":"4.15","version":"544'19","reported_seq":"739","reported_epoch":"820","state":"active+clean","last_fresh":"2021-03-24T22:06:30.445498+0000","last_change":"2021-03-24T22:05:14.370392+0000","last_active":"2021-03-24T22:06:30.445498+0000","last_peered":"2021-03-24T22:06:30.445498+0000","last_clean":"2021-03-24T22:06:30.445498+0000","last_became_active":"2021-03-24T22:05:04.433341+0000","last_became_peered":"2021-03-24T22:05:04.433341+0000","last_unstale":"2021-03-24T22:06:30.445498+0000","last_undegraded":"2021-03-24T22:06:30.445498+0000","last_fullsized":"2021-03-24T22:06:30.445498+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'19","last_scrub_stamp":"2021-03-24T22:05:09.177568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:05:09.177568+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":36,"num_read_kb":22,"num_write":20,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2202,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":"822","reported_epoch":"816","state":"active+clean","last_fresh":"2021-03-24T22:06:26.455207+0000","last_change":"2021-03-24T22:04:57.279696+0000","last_active":"2021-03-24T22:06:26.455207+0000","last_peered":"2021-03-24T22:06:26.455207+0000","last_clean":"2021-03-24T22:06:26.455207+0000","last_became_active":"2021-03-24T22:01:57.315967+0000","last_became_peered":"2021-03-24T22:01:57.315967+0000","last_unstale":"2021-03-24T22:06:26.455207+0000","last_undegraded":"2021-03-24T22:06:26.455207+0000","last_fullsized":"2021-03-24T22:06:26.455207+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:04:57.279517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:05.473348+0000","last_clean_scrub_stamp":"2021-03-24T22:04:57.279517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"573'12","reported_seq":"1047","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.445236+0000","last_change":"2021-03-24T22:03:02.456113+0000","last_active":"2021-03-24T22:06:28.445236+0000","last_peered":"2021-03-24T22:06:28.445236+0000","last_clean":"2021-03-24T22:06:28.445236+0000","last_became_active":"2021-03-24T22:02:22.326377+0000","last_became_peered":"2021-03-24T22:02:22.326377+0000","last_unstale":"2021-03-24T22:06:28.445236+0000","last_undegraded":"2021-03-24T22:06:28.445236+0000","last_fullsized":"2021-03-24T22:06:28.445236+0000","mapping_epoch":571,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":0,"last_scrub":"573'12","last_scrub_stamp":"2021-03-24T22:03:02.455946+0000","last_deep_scrub":"86'3","last_deep_scrub_stamp":"2021-03-24T21:52:56.678111+0000","last_clean_scrub_stamp":"2021-03-24T22:03:02.455946+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"772","reported_epoch":"820","state":"active+clean","last_fresh":"2021-03-24T22:06:30.437788+0000","last_change":"2021-03-24T22:05:06.701715+0000","last_active":"2021-03-24T22:06:30.437788+0000","last_peered":"2021-03-24T22:06:30.437788+0000","last_clean":"2021-03-24T22:06:30.437788+0000","last_became_active":"2021-03-24T22:05:04.429425+0000","last_became_peered":"2021-03-24T22:05:04.429425+0000","last_unstale":"2021-03-24T22:06:30.437788+0000","last_undegraded":"2021-03-24T22:06:30.437788+0000","last_fullsized":"2021-03-24T22:06:30.437788+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:06.701589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:05:06.701589+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.16","version":"813'551","reported_seq":"2983","reported_epoch":"816","state":"active+clean","last_fresh":"2021-03-24T22:06:26.458338+0000","last_change":"2021-03-24T22:06:26.458338+0000","last_active":"2021-03-24T22:06:26.458338+0000","last_peered":"2021-03-24T22:06:26.458338+0000","last_clean":"2021-03-24T22:06:26.458338+0000","last_became_active":"2021-03-24T22:02:08.322570+0000","last_became_peered":"2021-03-24T22:02:08.322570+0000","last_unstale":"2021-03-24T22:06:26.458338+0000","last_undegraded":"2021-03-24T22:06:26.458338+0000","last_fullsized":"2021-03-24T22:06:26.458338+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"606'362","last_scrub_stamp":"2021-03-24T22:02:57.478372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:02:57.478372+0000","log_size":551,"ondisk_log_size":551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3933036,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":104,"num_read_kb":117115,"num_write":81,"num_write_kb":19657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":30734442,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2a3","length":"1"},{"start":"2b3","length":"1"},{"start":"2b6","length":"1"}]},{"pgid":"4.14","version":"544'24","reported_seq":"1062","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.445303+0000","last_change":"2021-03-24T22:04:37.278384+0000","last_active":"2021-03-24T22:06:28.445303+0000","last_peered":"2021-03-24T22:06:28.445303+0000","last_clean":"2021-03-24T22:06:28.445303+0000","last_became_active":"2021-03-24T22:03:45.433725+0000","last_became_peered":"2021-03-24T22:03:45.433725+0000","last_unstale":"2021-03-24T22:06:28.445303+0000","last_undegraded":"2021-03-24T22:06:28.445303+0000","last_fullsized":"2021-03-24T22:06:28.445303+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'24","last_scrub_stamp":"2021-03-24T22:04:37.278283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:04:37.278283+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":42,"num_read_kb":29,"num_write":24,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":4992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":"283","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.007057+0000","last_change":"2021-03-24T22:04:37.274416+0000","last_active":"2021-03-24T22:06:28.007057+0000","last_peered":"2021-03-24T22:06:28.007057+0000","last_clean":"2021-03-24T22:06:28.007057+0000","last_became_active":"2021-03-24T22:01:57.318685+0000","last_became_peered":"2021-03-24T22:01:57.318685+0000","last_unstale":"2021-03-24T22:06:28.007057+0000","last_undegraded":"2021-03-24T22:06:28.007057+0000","last_fullsized":"2021-03-24T22:06:28.007057+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:04:37.274302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:04:37.274302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"743'12","reported_seq":"819","reported_epoch":"820","state":"active+clean","last_fresh":"2021-03-24T22:06:30.444891+0000","last_change":"2021-03-24T22:05:14.419465+0000","last_active":"2021-03-24T22:06:30.444891+0000","last_peered":"2021-03-24T22:06:30.444891+0000","last_clean":"2021-03-24T22:06:30.444891+0000","last_became_active":"2021-03-24T22:05:04.451042+0000","last_became_peered":"2021-03-24T22:05:04.451042+0000","last_unstale":"2021-03-24T22:06:30.444891+0000","last_undegraded":"2021-03-24T22:06:30.444891+0000","last_fullsized":"2021-03-24T22:06:30.444891+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"664'11","last_scrub_stamp":"2021-03-24T22:05:06.161410+0000","last_deep_scrub":"86'3","last_deep_scrub_stamp":"2021-03-24T21:53:08.815163+0000","last_clean_scrub_stamp":"2021-03-24T22:05:06.161410+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"820","reported_epoch":"816","state":"active+clean","last_fresh":"2021-03-24T22:06:26.457323+0000","last_change":"2021-03-24T22:03:57.076812+0000","last_active":"2021-03-24T22:06:26.457323+0000","last_peered":"2021-03-24T22:06:26.457323+0000","last_clean":"2021-03-24T22:06:26.457323+0000","last_became_active":"2021-03-24T22:01:57.316729+0000","last_became_peered":"2021-03-24T22:01:57.316729+0000","last_unstale":"2021-03-24T22:06:26.457323+0000","last_undegraded":"2021-03-24T22:06:26.457323+0000","last_fullsized":"2021-03-24T22:06:26.457323+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:57.076558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:03:57.076558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.17","version":"816'1352","reported_seq":"3937","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.008035+0000","last_change":"2021-03-24T22:06:26.457048+0000","last_active":"2021-03-24T22:06:28.008035+0000","last_peered":"2021-03-24T22:06:28.008035+0000","last_clean":"2021-03-24T22:06:28.008035+0000","last_became_active":"2021-03-24T22:05:04.460028+0000","last_became_peered":"2021-03-24T22:05:04.460028+0000","last_unstale":"2021-03-24T22:06:28.008035+0000","last_undegraded":"2021-03-24T22:06:28.008035+0000","last_fullsized":"2021-03-24T22:06:28.008035+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":5,"last_scrub":"742'1137","last_scrub_stamp":"2021-03-24T22:05:13.182580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:05:13.182580+0000","log_size":1352,"ondisk_log_size":1352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38986348,"num_objects":18,"num_object_clones":14,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":1,"num_read":345,"num_read_kb":367385,"num_write":247,"num_write_kb":90305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":257,"num_bytes_recovered":169206653,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2a3","length":"1"},{"start":"2b3","length":"1"},{"start":"2b6","length":"1"}]},{"pgid":"4.13","version":"584'32","reported_seq":"400","reported_epoch":"819","state":"active+clean","last_fresh":"2021-03-24T22:06:29.447597+0000","last_change":"2021-03-24T22:05:48.406758+0000","last_active":"2021-03-24T22:06:29.447597+0000","last_peered":"2021-03-24T22:06:29.447597+0000","last_clean":"2021-03-24T22:06:29.447597+0000","last_became_active":"2021-03-24T22:03:45.426065+0000","last_became_peered":"2021-03-24T22:03:45.426065+0000","last_unstale":"2021-03-24T22:06:29.447597+0000","last_undegraded":"2021-03-24T22:06:29.447597+0000","last_fullsized":"2021-03-24T22:06:29.447597+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"584'32","last_scrub_stamp":"2021-03-24T22:05:48.406579+0000","last_deep_scrub":"584'32","last_deep_scrub_stamp":"2021-03-24T22:02:34.813628+0000","last_clean_scrub_stamp":"2021-03-24T22:05:48.406579+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":29,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":3680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":"290","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.007523+0000","last_change":"2021-03-24T22:06:08.071800+0000","last_active":"2021-03-24T22:06:28.007523+0000","last_peered":"2021-03-24T22:06:28.007523+0000","last_clean":"2021-03-24T22:06:28.007523+0000","last_became_active":"2021-03-24T22:01:57.307018+0000","last_became_peered":"2021-03-24T22:01:57.307018+0000","last_unstale":"2021-03-24T22:06:28.007523+0000","last_undegraded":"2021-03-24T22:06:28.007523+0000","last_fullsized":"2021-03-24T22:06:28.007523+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:08.071696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:06:08.071696+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"194","reported_epoch":"818","state":"active+clean","last_fresh":"2021-03-24T22:06:28.436766+0000","last_change":"2021-03-24T22:05:07.629502+0000","last_active":"2021-03-24T22:06:28.436766+0000","last_peered":"2021-03-24T22:06:28.436766+0000","last_clean":"2021-03-24T22:06:28.436766+0000","last_became_active":"2021-03-24T22:05:04.451084+0000","last_became_peered":"2021-03-24T22:05:04.451084+0000","last_unstale":"2021-03-24T22:06:28.436766+0000","last_undegraded":"2021-03-24T22:06:28.436766+0000","last_fullsized":"2021-03-24T22:06:28.436766+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:07.629376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:57:31.118050+0000","last_clean_scrub_stamp":"2021-03-24T22:05:07.629376+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"804","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.444327+0000","last_change":"2021-03-24T22:05:52.855070+0000","last_active":"2021-03-24T22:06:28.444327+0000","last_peered":"2021-03-24T22:06:28.444327+0000","last_clean":"2021-03-24T22:06:28.444327+0000","last_became_active":"2021-03-24T22:05:04.435491+0000","last_became_peered":"2021-03-24T22:05:04.435491+0000","last_unstale":"2021-03-24T22:06:28.444327+0000","last_undegraded":"2021-03-24T22:06:28.444327+0000","last_fullsized":"2021-03-24T22:06:28.444327+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:52.854930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:23.707914+0000","last_clean_scrub_stamp":"2021-03-24T22:05:52.854930+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.10","version":"813'236","reported_seq":"1056","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.443898+0000","last_change":"2021-03-24T22:06:26.433305+0000","last_active":"2021-03-24T22:06:28.443898+0000","last_peered":"2021-03-24T22:06:28.443898+0000","last_clean":"2021-03-24T22:06:28.443898+0000","last_became_active":"2021-03-24T22:05:04.438010+0000","last_became_peered":"2021-03-24T22:05:04.438010+0000","last_unstale":"2021-03-24T22:06:28.443898+0000","last_undegraded":"2021-03-24T22:06:28.443898+0000","last_fullsized":"2021-03-24T22:06:28.443898+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":5,"last_scrub":"579'81","last_scrub_stamp":"2021-03-24T22:02:30.763305+0000","last_deep_scrub":"277'24","last_deep_scrub_stamp":"2021-03-24T21:57:28.119420+0000","last_clean_scrub_stamp":"2021-03-24T22:02:30.763305+0000","log_size":236,"ondisk_log_size":236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11918454,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":106,"num_read_kb":101733,"num_write":61,"num_write_kb":21017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":32974145,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2a3","length":"1"},{"start":"2b3","length":"1"},{"start":"2b6","length":"1"}]},{"pgid":"4.12","version":"544'21","reported_seq":"201","reported_epoch":"818","state":"active+clean","last_fresh":"2021-03-24T22:06:28.437955+0000","last_change":"2021-03-24T22:04:33.627668+0000","last_active":"2021-03-24T22:06:28.437955+0000","last_peered":"2021-03-24T22:06:28.437955+0000","last_clean":"2021-03-24T22:06:28.437955+0000","last_became_active":"2021-03-24T22:03:45.427113+0000","last_became_peered":"2021-03-24T22:03:45.427113+0000","last_unstale":"2021-03-24T22:06:28.437955+0000","last_undegraded":"2021-03-24T22:06:28.437955+0000","last_fullsized":"2021-03-24T22:06:28.437955+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'21","last_scrub_stamp":"2021-03-24T22:04:33.627568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:04:33.627568+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":43,"num_read_kb":25,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":3150,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":"288","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.007773+0000","last_change":"2021-03-24T22:05:53.144403+0000","last_active":"2021-03-24T22:06:28.007773+0000","last_peered":"2021-03-24T22:06:28.007773+0000","last_clean":"2021-03-24T22:06:28.007773+0000","last_became_active":"2021-03-24T22:01:57.318513+0000","last_became_peered":"2021-03-24T22:01:57.318513+0000","last_unstale":"2021-03-24T22:06:28.007773+0000","last_undegraded":"2021-03-24T22:06:28.007773+0000","last_fullsized":"2021-03-24T22:06:28.007773+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:53.144236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T22:03:09.417465+0000","last_clean_scrub_stamp":"2021-03-24T22:05:53.144236+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"827","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.445027+0000","last_change":"2021-03-24T22:05:57.864744+0000","last_active":"2021-03-24T22:06:28.445027+0000","last_peered":"2021-03-24T22:06:28.445027+0000","last_clean":"2021-03-24T22:06:28.445027+0000","last_became_active":"2021-03-24T22:03:45.435094+0000","last_became_peered":"2021-03-24T22:03:45.435094+0000","last_unstale":"2021-03-24T22:06:28.445027+0000","last_undegraded":"2021-03-24T22:06:28.445027+0000","last_fullsized":"2021-03-24T22:06:28.445027+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:57.864596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:05:57.864596+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"846","reported_epoch":"816","state":"active+clean","last_fresh":"2021-03-24T22:06:26.455844+0000","last_change":"2021-03-24T22:05:59.395794+0000","last_active":"2021-03-24T22:06:26.455844+0000","last_peered":"2021-03-24T22:06:26.455844+0000","last_clean":"2021-03-24T22:06:26.455844+0000","last_became_active":"2021-03-24T22:05:04.447958+0000","last_became_peered":"2021-03-24T22:05:04.447958+0000","last_unstale":"2021-03-24T22:06:26.455844+0000","last_undegraded":"2021-03-24T22:06:26.455844+0000","last_fullsized":"2021-03-24T22:06:26.455844+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:59.395655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:05:59.395655+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.11","version":"813'1180","reported_seq":"3720","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.007012+0000","last_change":"2021-03-24T22:06:26.456689+0000","last_active":"2021-03-24T22:06:28.007012+0000","last_peered":"2021-03-24T22:06:28.007012+0000","last_clean":"2021-03-24T22:06:28.007012+0000","last_became_active":"2021-03-24T22:02:04.322694+0000","last_became_peered":"2021-03-24T22:02:04.322694+0000","last_unstale":"2021-03-24T22:06:28.007012+0000","last_undegraded":"2021-03-24T22:06:28.007012+0000","last_fullsized":"2021-03-24T22:06:28.007012+0000","mapping_epoch":553,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":554,"parent":"0.0","parent_split_bits":5,"last_scrub":"703'940","last_scrub_stamp":"2021-03-24T22:04:35.296170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:04:35.296170+0000","log_size":1180,"ondisk_log_size":1180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36454689,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":226,"num_read_kb":256954,"num_write":161,"num_write_kb":70451,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":325,"num_bytes_recovered":148467814,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2a3","length":"1"},{"start":"2b3","length":"1"},{"start":"2b6","length":"1"}]},{"pgid":"4.11","version":"544'30","reported_seq":"1002","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.443690+0000","last_change":"2021-03-24T22:05:45.015199+0000","last_active":"2021-03-24T22:06:28.443690+0000","last_peered":"2021-03-24T22:06:28.443690+0000","last_clean":"2021-03-24T22:06:28.443690+0000","last_became_active":"2021-03-24T22:01:57.310370+0000","last_became_peered":"2021-03-24T22:01:57.310370+0000","last_unstale":"2021-03-24T22:06:28.443690+0000","last_undegraded":"2021-03-24T22:06:28.443690+0000","last_fullsized":"2021-03-24T22:06:28.443690+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'30","last_scrub_stamp":"2021-03-24T22:05:45.015001+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2021-03-24T21:52:54.597505+0000","last_clean_scrub_stamp":"2021-03-24T22:05:45.015001+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":41,"num_read_kb":28,"num_write":23,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":3480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":"672","reported_epoch":"820","state":"active+clean","last_fresh":"2021-03-24T22:06:30.445148+0000","last_change":"2021-03-24T22:05:43.282870+0000","last_active":"2021-03-24T22:06:30.445148+0000","last_peered":"2021-03-24T22:06:30.445148+0000","last_clean":"2021-03-24T22:06:30.445148+0000","last_became_active":"2021-03-24T22:05:04.439770+0000","last_became_peered":"2021-03-24T22:05:04.439770+0000","last_unstale":"2021-03-24T22:06:30.445148+0000","last_undegraded":"2021-03-24T22:06:30.445148+0000","last_fullsized":"2021-03-24T22:06:30.445148+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:43.282684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:05:43.282684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"283","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.006087+0000","last_change":"2021-03-24T22:04:38.228550+0000","last_active":"2021-03-24T22:06:28.006087+0000","last_peered":"2021-03-24T22:06:28.006087+0000","last_clean":"2021-03-24T22:06:28.006087+0000","last_became_active":"2021-03-24T22:01:57.307995+0000","last_became_peered":"2021-03-24T22:01:57.307995+0000","last_unstale":"2021-03-24T22:06:28.006087+0000","last_undegraded":"2021-03-24T22:06:28.006087+0000","last_fullsized":"2021-03-24T22:06:28.006087+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:04:38.228363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T22:04:38.228363+0000","last_clean_scrub_stamp":"2021-03-24T22:04:38.228363+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"792","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.445405+0000","last_change":"2021-03-24T22:05:09.243747+0000","last_active":"2021-03-24T22:06:28.445405+0000","last_peered":"2021-03-24T22:06:28.445405+0000","last_clean":"2021-03-24T22:06:28.445405+0000","last_became_active":"2021-03-24T22:01:43.389017+0000","last_became_peered":"2021-03-24T22:01:43.389017+0000","last_unstale":"2021-03-24T22:06:28.445405+0000","last_undegraded":"2021-03-24T22:06:28.445405+0000","last_fullsized":"2021-03-24T22:06:28.445405+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:09.243590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:47.540793+0000","last_clean_scrub_stamp":"2021-03-24T22:05:09.243590+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.12","version":"183'26","reported_seq":"903","reported_epoch":"820","state":"active+clean","last_fresh":"2021-03-24T22:06:30.436426+0000","last_change":"2021-03-24T22:06:30.437301+0000","last_active":"2021-03-24T22:06:30.436426+0000","last_peered":"2021-03-24T22:06:30.436426+0000","last_clean":"2021-03-24T22:06:30.436426+0000","last_became_active":"2021-03-24T22:01:43.392614+0000","last_became_peered":"2021-03-24T22:01:43.392614+0000","last_unstale":"2021-03-24T22:06:30.436426+0000","last_undegraded":"2021-03-24T22:06:30.436426+0000","last_fullsized":"2021-03-24T22:06:30.436426+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":5,"last_scrub":"183'26","last_scrub_stamp":"2021-03-24T22:04:25.596444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:04:25.596444+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"25e","length":"1"},{"start":"2af","length":"1"},{"start":"2b3","length":"1"},{"start":"2b6","length":"1"}]},{"pgid":"4.10","version":"544'16","reported_seq":"1076","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.442987+0000","last_change":"2021-03-24T22:05:47.006934+0000","last_active":"2021-03-24T22:06:28.442987+0000","last_peered":"2021-03-24T22:06:28.442987+0000","last_clean":"2021-03-24T22:06:28.442987+0000","last_became_active":"2021-03-24T22:03:45.428746+0000","last_became_peered":"2021-03-24T22:03:45.428746+0000","last_unstale":"2021-03-24T22:06:28.442987+0000","last_undegraded":"2021-03-24T22:06:28.442987+0000","last_fullsized":"2021-03-24T22:06:28.442987+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'16","last_scrub_stamp":"2021-03-24T22:05:47.006825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:05:47.006825+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":36,"num_read_kb":18,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":10016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":"816","reported_epoch":"816","state":"active+clean","last_fresh":"2021-03-24T22:06:26.457434+0000","last_change":"2021-03-24T22:05:58.410296+0000","last_active":"2021-03-24T22:06:26.457434+0000","last_peered":"2021-03-24T22:06:26.457434+0000","last_clean":"2021-03-24T22:06:26.457434+0000","last_became_active":"2021-03-24T22:05:04.458069+0000","last_became_peered":"2021-03-24T22:05:04.458069+0000","last_unstale":"2021-03-24T22:06:26.457434+0000","last_undegraded":"2021-03-24T22:06:26.457434+0000","last_fullsized":"2021-03-24T22:06:26.457434+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:58.410205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:05:58.410205+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"312","reported_epoch":"819","state":"active+clean","last_fresh":"2021-03-24T22:06:29.446819+0000","last_change":"2021-03-24T22:04:40.677638+0000","last_active":"2021-03-24T22:06:29.446819+0000","last_peered":"2021-03-24T22:06:29.446819+0000","last_clean":"2021-03-24T22:06:29.446819+0000","last_became_active":"2021-03-24T22:01:57.304385+0000","last_became_peered":"2021-03-24T22:01:57.304385+0000","last_unstale":"2021-03-24T22:06:29.446819+0000","last_undegraded":"2021-03-24T22:06:29.446819+0000","last_fullsized":"2021-03-24T22:06:29.446819+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:04:40.677553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:04:40.677553+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"290","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.006427+0000","last_change":"2021-03-24T22:05:45.108061+0000","last_active":"2021-03-24T22:06:28.006427+0000","last_peered":"2021-03-24T22:06:28.006427+0000","last_clean":"2021-03-24T22:06:28.006427+0000","last_became_active":"2021-03-24T22:01:57.315718+0000","last_became_peered":"2021-03-24T22:01:57.315718+0000","last_unstale":"2021-03-24T22:06:28.006427+0000","last_undegraded":"2021-03-24T22:06:28.006427+0000","last_fullsized":"2021-03-24T22:06:28.006427+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:45.107889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:57:59.979772+0000","last_clean_scrub_stamp":"2021-03-24T22:05:45.107889+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.13","version":"813'761","reported_seq":"3460","reported_epoch":"816","state":"active+clean","last_fresh":"2021-03-24T22:06:26.458953+0000","last_change":"2021-03-24T22:06:26.458953+0000","last_active":"2021-03-24T22:06:26.458953+0000","last_peered":"2021-03-24T22:06:26.458953+0000","last_clean":"2021-03-24T22:06:26.458953+0000","last_became_active":"2021-03-24T22:02:28.318669+0000","last_became_peered":"2021-03-24T22:02:28.318669+0000","last_unstale":"2021-03-24T22:06:26.458953+0000","last_undegraded":"2021-03-24T22:06:26.458953+0000","last_fullsized":"2021-03-24T22:06:26.458953+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":5,"last_scrub":"783'727","last_scrub_stamp":"2021-03-24T22:05:55.373277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:05:55.373277+0000","log_size":761,"ondisk_log_size":761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16444675,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2a3","length":"1"},{"start":"2b3","length":"1"},{"start":"2b6","length":"1"}]},{"pgid":"3.18","version":"0'0","reported_seq":"816","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.443088+0000","last_change":"2021-03-24T22:05:01.161274+0000","last_active":"2021-03-24T22:06:28.443088+0000","last_peered":"2021-03-24T22:06:28.443088+0000","last_clean":"2021-03-24T22:06:28.443088+0000","last_became_active":"2021-03-24T22:03:45.428772+0000","last_became_peered":"2021-03-24T22:03:45.428772+0000","last_unstale":"2021-03-24T22:06:28.443088+0000","last_undegraded":"2021-03-24T22:06:28.443088+0000","last_fullsized":"2021-03-24T22:06:28.443088+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:01.161130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:05:01.161130+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"840","reported_epoch":"816","state":"active+clean","last_fresh":"2021-03-24T22:06:26.457718+0000","last_change":"2021-03-24T22:04:35.127105+0000","last_active":"2021-03-24T22:06:26.457718+0000","last_peered":"2021-03-24T22:06:26.457718+0000","last_clean":"2021-03-24T22:06:26.457718+0000","last_became_active":"2021-03-24T22:02:22.328021+0000","last_became_peered":"2021-03-24T22:02:22.328021+0000","last_unstale":"2021-03-24T22:06:26.457718+0000","last_undegraded":"2021-03-24T22:06:26.457718+0000","last_fullsized":"2021-03-24T22:06:26.457718+0000","mapping_epoch":571,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:04:35.126877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:59:17.040623+0000","last_clean_scrub_stamp":"2021-03-24T22:04:35.126877+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"194","reported_epoch":"818","state":"active+clean","last_fresh":"2021-03-24T22:06:28.442801+0000","last_change":"2021-03-24T22:05:45.490399+0000","last_active":"2021-03-24T22:06:28.442801+0000","last_peered":"2021-03-24T22:06:28.442801+0000","last_clean":"2021-03-24T22:06:28.442801+0000","last_became_active":"2021-03-24T22:05:04.432365+0000","last_became_peered":"2021-03-24T22:05:04.432365+0000","last_unstale":"2021-03-24T22:06:28.442801+0000","last_undegraded":"2021-03-24T22:06:28.442801+0000","last_fullsized":"2021-03-24T22:06:28.442801+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:45.490290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:05:45.490290+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1c","version":"273'245","reported_seq":"2554","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.444943+0000","last_change":"2021-03-24T22:06:26.444814+0000","last_active":"2021-03-24T22:06:28.444943+0000","last_peered":"2021-03-24T22:06:28.444943+0000","last_clean":"2021-03-24T22:06:28.444943+0000","last_became_active":"2021-03-24T22:05:04.445911+0000","last_became_peered":"2021-03-24T22:05:04.445911+0000","last_unstale":"2021-03-24T22:06:28.444943+0000","last_undegraded":"2021-03-24T22:06:28.444943+0000","last_fullsized":"2021-03-24T22:06:28.444943+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":245,"ondisk_log_size":245,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8581786,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":70,"num_read_kb":67597,"num_write":52,"num_write_kb":18055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2a3","length":"1"},{"start":"2b3","length":"1"},{"start":"2b6","length":"1"}]},{"pgid":"3.19","version":"0'0","reported_seq":"179","reported_epoch":"818","state":"active+clean","last_fresh":"2021-03-24T22:06:28.435699+0000","last_change":"2021-03-24T22:04:25.611828+0000","last_active":"2021-03-24T22:06:28.435699+0000","last_peered":"2021-03-24T22:06:28.435699+0000","last_clean":"2021-03-24T22:06:28.435699+0000","last_became_active":"2021-03-24T22:03:45.430602+0000","last_became_peered":"2021-03-24T22:03:45.430602+0000","last_unstale":"2021-03-24T22:06:28.435699+0000","last_undegraded":"2021-03-24T22:06:28.435699+0000","last_fullsized":"2021-03-24T22:06:28.435699+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:04:25.611742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:57:26.417613+0000","last_clean_scrub_stamp":"2021-03-24T22:04:25.611742+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"613'8","reported_seq":"376","reported_epoch":"819","state":"active+clean","last_fresh":"2021-03-24T22:06:29.448633+0000","last_change":"2021-03-24T22:06:00.101350+0000","last_active":"2021-03-24T22:06:29.448633+0000","last_peered":"2021-03-24T22:06:29.448633+0000","last_clean":"2021-03-24T22:06:29.448633+0000","last_became_active":"2021-03-24T22:03:03.353426+0000","last_became_peered":"2021-03-24T22:03:03.353426+0000","last_unstale":"2021-03-24T22:06:29.448633+0000","last_undegraded":"2021-03-24T22:06:29.448633+0000","last_fullsized":"2021-03-24T22:06:29.448633+0000","mapping_epoch":612,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":613,"parent":"0.0","parent_split_bits":0,"last_scrub":"613'8","last_scrub_stamp":"2021-03-24T22:06:00.101091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:06:00.101091+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"599","reported_epoch":"820","state":"active+clean","last_fresh":"2021-03-24T22:06:30.433900+0000","last_change":"2021-03-24T22:05:43.068387+0000","last_active":"2021-03-24T22:06:30.433900+0000","last_peered":"2021-03-24T22:06:30.433900+0000","last_clean":"2021-03-24T22:06:30.433900+0000","last_became_active":"2021-03-24T22:05:04.423989+0000","last_became_peered":"2021-03-24T22:05:04.423989+0000","last_unstale":"2021-03-24T22:06:30.433900+0000","last_undegraded":"2021-03-24T22:06:30.433900+0000","last_fullsized":"2021-03-24T22:06:30.433900+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:43.068130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:05:43.068130+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1d","version":"808'731","reported_seq":"3342","reported_epoch":"819","state":"active+clean","last_fresh":"2021-03-24T22:06:29.444443+0000","last_change":"2021-03-24T22:06:29.445342+0000","last_active":"2021-03-24T22:06:29.444443+0000","last_peered":"2021-03-24T22:06:29.444443+0000","last_clean":"2021-03-24T22:06:29.444443+0000","last_became_active":"2021-03-24T22:03:53.384369+0000","last_became_peered":"2021-03-24T22:03:53.384369+0000","last_unstale":"2021-03-24T22:06:29.444443+0000","last_undegraded":"2021-03-24T22:06:29.444443+0000","last_fullsized":"2021-03-24T22:06:29.444443+0000","mapping_epoch":662,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":663,"parent":"0.0","parent_split_bits":5,"last_scrub":"695'623","last_scrub_stamp":"2021-03-24T22:04:37.590137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:04:37.590137+0000","log_size":731,"ondisk_log_size":731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13252624,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":21335,"num_write":16,"num_write_kb":6286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2a3","length":"1"},{"start":"2af","length":"1"},{"start":"2b3","length":"1"},{"start":"2b6","length":"1"}]},{"pgid":"3.1a","version":"0'0","reported_seq":"179","reported_epoch":"818","state":"active+clean","last_fresh":"2021-03-24T22:06:28.437394+0000","last_change":"2021-03-24T22:05:49.560351+0000","last_active":"2021-03-24T22:06:28.437394+0000","last_peered":"2021-03-24T22:06:28.437394+0000","last_clean":"2021-03-24T22:06:28.437394+0000","last_became_active":"2021-03-24T22:03:45.431828+0000","last_became_peered":"2021-03-24T22:03:45.431828+0000","last_unstale":"2021-03-24T22:06:28.437394+0000","last_undegraded":"2021-03-24T22:06:28.437394+0000","last_fullsized":"2021-03-24T22:06:28.437394+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:49.560279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:52:54.212459+0000","last_clean_scrub_stamp":"2021-03-24T22:05:49.560279+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"353","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.445067+0000","last_change":"2021-03-24T22:02:17.672555+0000","last_active":"2021-03-24T22:06:28.445067+0000","last_peered":"2021-03-24T22:06:28.445067+0000","last_clean":"2021-03-24T22:06:28.445067+0000","last_became_active":"2021-03-24T22:00:49.305569+0000","last_became_peered":"2021-03-24T22:00:49.305569+0000","last_unstale":"2021-03-24T22:06:28.445067+0000","last_undegraded":"2021-03-24T22:06:28.445067+0000","last_fullsized":"2021-03-24T22:06:28.445067+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:17.672325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:02:17.672325+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"840","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.444316+0000","last_change":"2021-03-24T22:06:07.907835+0000","last_active":"2021-03-24T22:06:28.444316+0000","last_peered":"2021-03-24T22:06:28.444316+0000","last_clean":"2021-03-24T22:06:28.444316+0000","last_became_active":"2021-03-24T22:05:04.423569+0000","last_became_peered":"2021-03-24T22:05:04.423569+0000","last_unstale":"2021-03-24T22:06:28.444316+0000","last_undegraded":"2021-03-24T22:06:28.444316+0000","last_fullsized":"2021-03-24T22:06:28.444316+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:07.907636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T22:06:07.907636+0000","last_clean_scrub_stamp":"2021-03-24T22:06:07.907636+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1e","version":"265'45","reported_seq":"527","reported_epoch":"819","state":"active+clean","last_fresh":"2021-03-24T22:06:29.439104+0000","last_change":"2021-03-24T22:06:29.440370+0000","last_active":"2021-03-24T22:06:29.439104+0000","last_peered":"2021-03-24T22:06:29.439104+0000","last_clean":"2021-03-24T22:06:29.439104+0000","last_became_active":"2021-03-24T22:05:04.432331+0000","last_became_peered":"2021-03-24T22:05:04.432331+0000","last_unstale":"2021-03-24T22:06:29.439104+0000","last_undegraded":"2021-03-24T22:06:29.439104+0000","last_fullsized":"2021-03-24T22:06:29.439104+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":5,"last_scrub":"265'45","last_scrub_stamp":"2021-03-24T22:05:55.541936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:05:55.541936+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2a3","length":"1"},{"start":"2af","length":"1"},{"start":"2b3","length":"1"},{"start":"2b6","length":"1"}]},{"pgid":"3.1b","version":"0'0","reported_seq":"840","reported_epoch":"816","state":"active+clean","last_fresh":"2021-03-24T22:06:26.457659+0000","last_change":"2021-03-24T22:05:51.442642+0000","last_active":"2021-03-24T22:06:26.457659+0000","last_peered":"2021-03-24T22:06:26.457659+0000","last_clean":"2021-03-24T22:06:26.457659+0000","last_became_active":"2021-03-24T22:03:32.380554+0000","last_became_peered":"2021-03-24T22:03:32.380554+0000","last_unstale":"2021-03-24T22:06:26.457659+0000","last_undegraded":"2021-03-24T22:06:26.457659+0000","last_fullsized":"2021-03-24T22:06:26.457659+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:51.442473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:05:51.442473+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"179","reported_epoch":"818","state":"active+clean","last_fresh":"2021-03-24T22:06:28.442536+0000","last_change":"2021-03-24T22:04:27.649967+0000","last_active":"2021-03-24T22:06:28.442536+0000","last_peered":"2021-03-24T22:06:28.442536+0000","last_clean":"2021-03-24T22:06:28.442536+0000","last_became_active":"2021-03-24T22:03:45.433015+0000","last_became_peered":"2021-03-24T22:03:45.433015+0000","last_unstale":"2021-03-24T22:06:28.442536+0000","last_undegraded":"2021-03-24T22:06:28.442536+0000","last_fullsized":"2021-03-24T22:06:28.442536+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:04:27.649863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:04:27.649863+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"311","reported_epoch":"819","state":"active+clean","last_fresh":"2021-03-24T22:06:29.447737+0000","last_change":"2021-03-24T22:05:08.570430+0000","last_active":"2021-03-24T22:06:29.447737+0000","last_peered":"2021-03-24T22:06:29.447737+0000","last_clean":"2021-03-24T22:06:29.447737+0000","last_became_active":"2021-03-24T22:05:04.431178+0000","last_became_peered":"2021-03-24T22:05:04.431178+0000","last_unstale":"2021-03-24T22:06:29.447737+0000","last_undegraded":"2021-03-24T22:06:29.447737+0000","last_fullsized":"2021-03-24T22:06:29.447737+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:08.570238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:05:08.570238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1f","version":"272'99","reported_seq":"898","reported_epoch":"820","state":"active+clean","last_fresh":"2021-03-24T22:06:30.462803+0000","last_change":"2021-03-24T22:06:30.463312+0000","last_active":"2021-03-24T22:06:30.462803+0000","last_peered":"2021-03-24T22:06:30.462803+0000","last_clean":"2021-03-24T22:06:30.462803+0000","last_became_active":"2021-03-24T22:01:43.385324+0000","last_became_peered":"2021-03-24T22:01:43.385324+0000","last_unstale":"2021-03-24T22:06:30.462803+0000","last_undegraded":"2021-03-24T22:06:30.462803+0000","last_fullsized":"2021-03-24T22:06:30.462803+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'99","last_scrub_stamp":"2021-03-24T22:03:57.739288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:03:57.739288+0000","log_size":99,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"25e","length":"1"},{"start":"2af","length":"1"},{"start":"2b3","length":"1"},{"start":"2b6","length":"1"}]},{"pgid":"3.1c","version":"0'0","reported_seq":"194","reported_epoch":"818","state":"active+clean","last_fresh":"2021-03-24T22:06:28.439322+0000","last_change":"2021-03-24T22:05:05.665557+0000","last_active":"2021-03-24T22:06:28.439322+0000","last_peered":"2021-03-24T22:06:28.439322+0000","last_clean":"2021-03-24T22:06:28.439322+0000","last_became_active":"2021-03-24T22:05:04.453666+0000","last_became_peered":"2021-03-24T22:05:04.453666+0000","last_unstale":"2021-03-24T22:06:28.439322+0000","last_undegraded":"2021-03-24T22:06:28.439322+0000","last_fullsized":"2021-03-24T22:06:28.439322+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:05.665337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:05:05.665337+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"301","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.006472+0000","last_change":"2021-03-24T22:04:27.335659+0000","last_active":"2021-03-24T22:06:28.006472+0000","last_peered":"2021-03-24T22:06:28.006472+0000","last_clean":"2021-03-24T22:06:28.006472+0000","last_became_active":"2021-03-24T22:03:45.434093+0000","last_became_peered":"2021-03-24T22:03:45.434093+0000","last_unstale":"2021-03-24T22:06:28.006472+0000","last_undegraded":"2021-03-24T22:06:28.006472+0000","last_fullsized":"2021-03-24T22:06:28.006472+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:04:27.335482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:56:51.977306+0000","last_clean_scrub_stamp":"2021-03-24T22:04:27.335482+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"645","reported_epoch":"820","state":"active+clean","last_fresh":"2021-03-24T22:06:30.443741+0000","last_change":"2021-03-24T22:05:15.154680+0000","last_active":"2021-03-24T22:06:30.443741+0000","last_peered":"2021-03-24T22:06:30.443741+0000","last_clean":"2021-03-24T22:06:30.443741+0000","last_became_active":"2021-03-24T22:05:04.437068+0000","last_became_peered":"2021-03-24T22:05:04.437068+0000","last_unstale":"2021-03-24T22:06:30.443741+0000","last_undegraded":"2021-03-24T22:06:30.443741+0000","last_fullsized":"2021-03-24T22:06:30.443741+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:15.154493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T22:05:15.154493+0000","last_clean_scrub_stamp":"2021-03-24T22:05:15.154493+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.18","version":"816'289","reported_seq":"2305","reported_epoch":"816","state":"active+clean","last_fresh":"2021-03-24T22:06:26.457333+0000","last_change":"2021-03-24T22:06:26.457333+0000","last_active":"2021-03-24T22:06:26.457333+0000","last_peered":"2021-03-24T22:06:26.457333+0000","last_clean":"2021-03-24T22:06:26.457333+0000","last_became_active":"2021-03-24T22:02:00.304128+0000","last_became_peered":"2021-03-24T22:02:00.304128+0000","last_unstale":"2021-03-24T22:06:26.457333+0000","last_undegraded":"2021-03-24T22:06:26.457333+0000","last_fullsized":"2021-03-24T22:06:26.457333+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":5,"last_scrub":"624'193","last_scrub_stamp":"2021-03-24T22:03:43.233875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:03:43.233875+0000","log_size":289,"ondisk_log_size":289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7109856,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":54,"num_read_kb":47612,"num_write":36,"num_write_kb":12535,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":32240200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2a3","length":"1"},{"start":"2b3","length":"1"},{"start":"2b6","length":"1"}]},{"pgid":"3.1d","version":"0'0","reported_seq":"867","reported_epoch":"820","state":"active+clean","last_fresh":"2021-03-24T22:06:30.437368+0000","last_change":"2021-03-24T22:05:48.075900+0000","last_active":"2021-03-24T22:06:30.437368+0000","last_peered":"2021-03-24T22:06:30.437368+0000","last_clean":"2021-03-24T22:06:30.437368+0000","last_became_active":"2021-03-24T22:05:04.430681+0000","last_became_peered":"2021-03-24T22:05:04.430681+0000","last_unstale":"2021-03-24T22:06:30.437368+0000","last_undegraded":"2021-03-24T22:06:30.437368+0000","last_fullsized":"2021-03-24T22:06:30.437368+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:48.075822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:05:48.075822+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"644","reported_epoch":"820","state":"active+clean","last_fresh":"2021-03-24T22:06:30.444595+0000","last_change":"2021-03-24T22:05:52.372115+0000","last_active":"2021-03-24T22:06:30.444595+0000","last_peered":"2021-03-24T22:06:30.444595+0000","last_clean":"2021-03-24T22:06:30.444595+0000","last_became_active":"2021-03-24T22:05:04.432965+0000","last_became_peered":"2021-03-24T22:05:04.432965+0000","last_unstale":"2021-03-24T22:06:30.444595+0000","last_undegraded":"2021-03-24T22:06:30.444595+0000","last_fullsized":"2021-03-24T22:06:30.444595+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:52.372027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:05:52.372027+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"852","reported_epoch":"816","state":"active+clean","last_fresh":"2021-03-24T22:06:26.454675+0000","last_change":"2021-03-24T22:06:02.476829+0000","last_active":"2021-03-24T22:06:26.454675+0000","last_peered":"2021-03-24T22:06:26.454675+0000","last_clean":"2021-03-24T22:06:26.454675+0000","last_became_active":"2021-03-24T22:03:45.431424+0000","last_became_peered":"2021-03-24T22:03:45.431424+0000","last_unstale":"2021-03-24T22:06:26.454675+0000","last_undegraded":"2021-03-24T22:06:26.454675+0000","last_fullsized":"2021-03-24T22:06:26.454675+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:02.476582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:06:02.476582+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.19","version":"820'1256","reported_seq":"3839","reported_epoch":"820","state":"active+clean","last_fresh":"2021-03-24T22:06:30.462216+0000","last_change":"2021-03-24T22:06:30.462698+0000","last_active":"2021-03-24T22:06:30.462216+0000","last_peered":"2021-03-24T22:06:30.462216+0000","last_clean":"2021-03-24T22:06:30.462216+0000","last_became_active":"2021-03-24T22:02:07.304440+0000","last_became_peered":"2021-03-24T22:02:07.304440+0000","last_unstale":"2021-03-24T22:06:30.462216+0000","last_undegraded":"2021-03-24T22:06:30.462216+0000","last_fullsized":"2021-03-24T22:06:30.462216+0000","mapping_epoch":556,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":557,"parent":"0.0","parent_split_bits":5,"last_scrub":"664'844","last_scrub_stamp":"2021-03-24T22:03:55.671094+0000","last_deep_scrub":"455'480","last_deep_scrub_stamp":"2021-03-24T22:00:26.193730+0000","last_clean_scrub_stamp":"2021-03-24T22:03:55.671094+0000","log_size":1256,"ondisk_log_size":1256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31937817,"num_objects":16,"num_object_clones":13,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":1,"num_read":251,"num_read_kb":216920,"num_write":231,"num_write_kb":94479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":265,"num_bytes_recovered":154749512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"25e","length":"1"},{"start":"2af","length":"1"},{"start":"2b3","length":"1"},{"start":"2b6","length":"1"}]},{"pgid":"3.1e","version":"0'0","reported_seq":"840","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.444296+0000","last_change":"2021-03-24T22:04:27.240608+0000","last_active":"2021-03-24T22:06:28.444296+0000","last_peered":"2021-03-24T22:06:28.444296+0000","last_clean":"2021-03-24T22:06:28.444296+0000","last_became_active":"2021-03-24T22:01:43.381824+0000","last_became_peered":"2021-03-24T22:01:43.381824+0000","last_unstale":"2021-03-24T22:06:28.444296+0000","last_undegraded":"2021-03-24T22:06:28.444296+0000","last_fullsized":"2021-03-24T22:06:28.444296+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:04:27.240496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:04:27.240496+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"319","reported_epoch":"819","state":"active+clean","last_fresh":"2021-03-24T22:06:29.447399+0000","last_change":"2021-03-24T22:05:49.398497+0000","last_active":"2021-03-24T22:06:29.447399+0000","last_peered":"2021-03-24T22:06:29.447399+0000","last_clean":"2021-03-24T22:06:29.447399+0000","last_became_active":"2021-03-24T22:03:45.416917+0000","last_became_peered":"2021-03-24T22:03:45.416917+0000","last_unstale":"2021-03-24T22:06:29.447399+0000","last_undegraded":"2021-03-24T22:06:29.447399+0000","last_fullsized":"2021-03-24T22:06:29.447399+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:49.398341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:05:49.398341+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"848","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.445186+0000","last_change":"2021-03-24T22:03:08.392175+0000","last_active":"2021-03-24T22:06:28.445186+0000","last_peered":"2021-03-24T22:06:28.445186+0000","last_clean":"2021-03-24T22:06:28.445186+0000","last_became_active":"2021-03-24T22:01:57.310369+0000","last_became_peered":"2021-03-24T22:01:57.310369+0000","last_unstale":"2021-03-24T22:06:28.445186+0000","last_undegraded":"2021-03-24T22:06:28.445186+0000","last_fullsized":"2021-03-24T22:06:28.445186+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:08.392060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:03:08.392060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"820'426","reported_seq":"2802","reported_epoch":"820","state":"active+clean","last_fresh":"2021-03-24T22:06:30.461989+0000","last_change":"2021-03-24T22:06:30.462503+0000","last_active":"2021-03-24T22:06:30.461989+0000","last_peered":"2021-03-24T22:06:30.461989+0000","last_clean":"2021-03-24T22:06:30.461989+0000","last_became_active":"2021-03-24T22:05:04.454694+0000","last_became_peered":"2021-03-24T22:05:04.454694+0000","last_unstale":"2021-03-24T22:06:30.461989+0000","last_undegraded":"2021-03-24T22:06:30.461989+0000","last_fullsized":"2021-03-24T22:06:30.461989+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":426,"ondisk_log_size":426,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10762694,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":118,"num_read_kb":134350,"num_write":97,"num_write_kb":36283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":39653620,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"25e","length":"1"},{"start":"2af","length":"1"},{"start":"2b3","length":"1"},{"start":"2b6","length":"1"}]},{"pgid":"3.1f","version":"0'0","reported_seq":"839","reported_epoch":"816","state":"active+clean","last_fresh":"2021-03-24T22:06:26.457332+0000","last_change":"2021-03-24T22:05:48.437882+0000","last_active":"2021-03-24T22:06:26.457332+0000","last_peered":"2021-03-24T22:06:26.457332+0000","last_clean":"2021-03-24T22:06:26.457332+0000","last_became_active":"2021-03-24T22:05:04.457596+0000","last_became_peered":"2021-03-24T22:05:04.457596+0000","last_unstale":"2021-03-24T22:06:26.457332+0000","last_undegraded":"2021-03-24T22:06:26.457332+0000","last_fullsized":"2021-03-24T22:06:26.457332+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:48.437760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:05:48.437760+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"194","reported_epoch":"818","state":"active+clean","last_fresh":"2021-03-24T22:06:28.436315+0000","last_change":"2021-03-24T22:05:11.459315+0000","last_active":"2021-03-24T22:06:28.436315+0000","last_peered":"2021-03-24T22:06:28.436315+0000","last_clean":"2021-03-24T22:06:28.436315+0000","last_became_active":"2021-03-24T22:05:04.447664+0000","last_became_peered":"2021-03-24T22:05:04.447664+0000","last_unstale":"2021-03-24T22:06:28.436315+0000","last_undegraded":"2021-03-24T22:06:28.436315+0000","last_fullsized":"2021-03-24T22:06:28.436315+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:11.459219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:14.710709+0000","last_clean_scrub_stamp":"2021-03-24T22:05:11.459219+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"747","reported_epoch":"820","state":"active+clean","last_fresh":"2021-03-24T22:06:30.444713+0000","last_change":"2021-03-24T22:05:46.303724+0000","last_active":"2021-03-24T22:06:30.444713+0000","last_peered":"2021-03-24T22:06:30.444713+0000","last_clean":"2021-03-24T22:06:30.444713+0000","last_became_active":"2021-03-24T22:05:04.443147+0000","last_became_peered":"2021-03-24T22:05:04.443147+0000","last_unstale":"2021-03-24T22:06:30.444713+0000","last_undegraded":"2021-03-24T22:06:30.444713+0000","last_fullsized":"2021-03-24T22:06:30.444713+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:46.303541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:05:46.303541+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1b","version":"816'1987","reported_seq":"4287","reported_epoch":"816","state":"active+clean","last_fresh":"2021-03-24T22:06:26.467826+0000","last_change":"2021-03-24T22:06:26.467826+0000","last_active":"2021-03-24T22:06:26.467826+0000","last_peered":"2021-03-24T22:06:26.467826+0000","last_clean":"2021-03-24T22:06:26.467826+0000","last_became_active":"2021-03-24T21:57:33.176110+0000","last_became_peered":"2021-03-24T21:57:33.176110+0000","last_unstale":"2021-03-24T22:06:26.467826+0000","last_undegraded":"2021-03-24T22:06:26.467826+0000","last_fullsized":"2021-03-24T22:06:26.467826+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":5,"last_scrub":"710'1638","last_scrub_stamp":"2021-03-24T22:04:41.216144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:04:41.216144+0000","log_size":1987,"ondisk_log_size":1987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":66646937,"num_objects":29,"num_object_clones":23,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":3,"num_read":309,"num_read_kb":316188,"num_write":250,"num_write_kb":109158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2a3","length":"1"},{"start":"2b3","length":"1"},{"start":"2b6","length":"1"}]},{"pgid":"3.24","version":"0'0","reported_seq":"446","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.444469+0000","last_change":"2021-03-24T22:03:41.269971+0000","last_active":"2021-03-24T22:06:28.444469+0000","last_peered":"2021-03-24T22:06:28.444469+0000","last_clean":"2021-03-24T22:06:28.444469+0000","last_became_active":"2021-03-24T21:59:14.369606+0000","last_became_peered":"2021-03-24T21:59:14.369606+0000","last_unstale":"2021-03-24T22:06:28.444469+0000","last_undegraded":"2021-03-24T22:06:28.444469+0000","last_fullsized":"2021-03-24T22:06:28.444469+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:41.269900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:03:41.269900+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"175'22","reported_seq":"1961","reported_epoch":"821","state":"active+clean","last_fresh":"2021-03-24T22:06:31.445873+0000","last_change":"2021-03-24T22:06:31.447921+0000","last_active":"2021-03-24T22:06:31.445873+0000","last_peered":"2021-03-24T22:06:31.445873+0000","last_clean":"2021-03-24T22:06:31.445873+0000","last_became_active":"2021-03-24T22:03:45.432211+0000","last_became_peered":"2021-03-24T22:03:45.432211+0000","last_unstale":"2021-03-24T22:06:31.445873+0000","last_undegraded":"2021-03-24T22:06:31.445873+0000","last_fullsized":"2021-03-24T22:06:31.445873+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":6,"last_scrub":"175'22","last_scrub_stamp":"2021-03-24T22:04:56.629000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:04:56.629000+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"25e","length":"1"},{"start":"28e","length":"1"},{"start":"2af","length":"1"},{"start":"2b3","length":"1"},{"start":"2b6","length":"1"}]},{"pgid":"3.25","version":"0'0","reported_seq":"179","reported_epoch":"818","state":"active+clean","last_fresh":"2021-03-24T22:06:28.443061+0000","last_change":"2021-03-24T22:04:36.619633+0000","last_active":"2021-03-24T22:06:28.443061+0000","last_peered":"2021-03-24T22:06:28.443061+0000","last_clean":"2021-03-24T22:06:28.443061+0000","last_became_active":"2021-03-24T22:03:45.426841+0000","last_became_peered":"2021-03-24T22:03:45.426841+0000","last_unstale":"2021-03-24T22:06:28.443061+0000","last_undegraded":"2021-03-24T22:06:28.443061+0000","last_fullsized":"2021-03-24T22:06:28.443061+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:04:36.619532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:04:36.619532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.21","version":"816'809","reported_seq":"3407","reported_epoch":"817","state":"active+clean","last_fresh":"2021-03-24T22:06:28.443955+0000","last_change":"2021-03-24T22:06:26.448006+0000","last_active":"2021-03-24T22:06:28.443955+0000","last_peered":"2021-03-24T22:06:28.443955+0000","last_clean":"2021-03-24T22:06:28.443955+0000","last_became_active":"2021-03-24T22:05:04.434209+0000","last_became_peered":"2021-03-24T22:05:04.434209+0000","last_unstale":"2021-03-24T22:06:28.443955+0000","last_undegraded":"2021-03-24T22:06:28.443955+0000","last_fullsized":"2021-03-24T22:06:28.443955+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":809,"ondisk_log_size":809,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29827462,"num_objects":13,"num_object_clones":12,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":225,"num_read_kb":218576,"num_write":176,"num_write_kb":66631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":68248426,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2a3","length":"1"},{"start":"2b3","length":"1"},{"start":"2b6","length":"1"}]}],"pool_stats":[{"poolid":7,"num_pg":36,"stat_sum":{"num_bytes":525617021,"num_objects":229,"num_object_clones":179,"num_object_copies":687,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":229,"num_whiteouts":12,"num_read":3136,"num_read_kb":3268234,"num_write":2324,"num_write_kb":892468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2259,"num_bytes_recovered":1262123220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1186807808,"data_stored":1613223849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20239,"ondisk_log_size":20239,"up":108,"acting":108,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":343,"num_read_kb":338,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":3,"num_pg":42,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10235,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":126,"acting":126,"num_store_stats":8},{"poolid":4,"num_pg":22,"stat_sum":{"num_bytes":23524,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1475,"num_read_kb":980,"num_write":828,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1183,"num_bytes_recovered":93828,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966080,"data_stored":70572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":514,"ondisk_log_size":514,"up":66,"acting":66,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201461,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":270760,"kb_used_data":201312,"kb_used_omap":0,"kb_used_meta":69440,"kb_avail":93474392,"statfs":{"total":95995035648,"available":95717777408,"internally_reserved":0,"allocated":206143488,"data_stored":279785118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":71106560},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 22:06:17 2021","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":2.5150000000000001,"15min":1.4510000000000001},"min":{"1min":0.57699999999999996,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.7410000000000001,"5min":150.68000000000001,"15min":150.68000000000001},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.97099999999999997,"5min":2.5699999999999998,"15min":1.5429999999999999},"min":{"1min":0.52300000000000002,"5min":0.36299999999999999,"15min":0.27000000000000002},"max":{"1min":2.2639999999999998,"5min":153.91,"15min":153.91},"last":1.4279999999999999}]},{"osd":1,"last update":"Wed Mar 24 22:06:17 2021","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":0.83999999999999997,"15min":0.88500000000000001},"min":{"1min":0.59599999999999997,"5min":0.318,"15min":0.308},"max":{"1min":1.8180000000000001,"5min":1.8180000000000001,"15min":5.7750000000000004},"last":0.80500000000000005},{"interface":"front","average":{"1min":1.0089999999999999,"5min":0.878,"15min":0.94999999999999996},"min":{"1min":0.61699999999999999,"5min":0.316,"15min":0.316},"max":{"1min":1.623,"5min":1.623,"15min":6.1849999999999996},"last":0.93799999999999994}]},{"osd":2,"last update":"Wed Mar 24 22:06:17 2021","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.83999999999999997,"15min":0.98099999999999998},"min":{"1min":0.69299999999999995,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.982,"5min":1.982,"15min":12.135999999999999},"last":0.84899999999999998},{"interface":"front","average":{"1min":1.0449999999999999,"5min":0.92500000000000004,"15min":1.0149999999999999},"min":{"1min":0.56399999999999995,"5min":0.45900000000000002,"15min":0.42999999999999999},"max":{"1min":1.881,"5min":1.881,"15min":12.178000000000001},"last":1.1919999999999999}]},{"osd":3,"last update":"Wed Mar 24 22:06:17 2021","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":0.998,"15min":1.0640000000000001},"min":{"1min":0.81699999999999995,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":2.04,"5min":6.1449999999999996,"15min":12.529},"last":1.23},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.0329999999999999,"15min":1.081},"min":{"1min":0.66700000000000004,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.9159999999999999,"5min":3.0510000000000002,"15min":12.666},"last":1.2749999999999999}]},{"osd":4,"last update":"Wed Mar 24 22:06:17 2021","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":0.96199999999999997,"15min":0.98799999999999999},"min":{"1min":0.80600000000000005,"5min":0.34799999999999998,"15min":0.33500000000000002},"max":{"1min":1.714,"5min":1.714,"15min":6.2089999999999996},"last":1.5469999999999999},{"interface":"front","average":{"1min":1.1160000000000001,"5min":1.012,"15min":1.0720000000000001},"min":{"1min":0.89400000000000002,"5min":0.42399999999999999,"15min":0.41399999999999998},"max":{"1min":2.1360000000000001,"5min":2.1360000000000001,"15min":12.244},"last":1.038}]},{"osd":5,"last update":"Wed Mar 24 22:06:07 2021","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":1.0740000000000001},{"interface":"front","average":{"1min":1.149,"5min":1.149,"15min":1.149},"min":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":1.444,"5min":1.444,"15min":1.444},"last":1.121}]},{"osd":6,"last update":"Wed Mar 24 22:06:17 2021","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.101,"15min":1.137},"min":{"1min":0.72299999999999998,"5min":0.51800000000000002,"15min":0.40899999999999997},"max":{"1min":1.7370000000000001,"5min":1.7370000000000001,"15min":6.1159999999999997},"last":1.6850000000000001},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.111,"15min":1.165},"min":{"1min":0.80400000000000005,"5min":0.502,"15min":0.434},"max":{"1min":2.0830000000000002,"5min":2.0830000000000002,"15min":12.287000000000001},"last":1.5700000000000001}]}]},{"osd":6,"up_from":47,"seq":201863463098,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":186208,"kb_used_data":146840,"kb_used_omap":0,"kb_used_meta":39360,"kb_avail":93558944,"statfs":{"total":95995035648,"available":95804358656,"internally_reserved":0,"allocated":150364160,"data_stored":204587077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40304640},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 22:05:43 2021","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":1.054,"15min":1.008},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.30399999999999999},"max":{"1min":1.5629999999999999,"5min":19.257999999999999,"15min":19.257999999999999},"last":1.044},{"interface":"front","average":{"1min":0.81699999999999995,"5min":1.077,"15min":0.98999999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":1.673,"5min":16.795000000000002,"15min":16.795000000000002},"last":0.63}]},{"osd":1,"last update":"Wed Mar 24 22:05:43 2021","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":1.1619999999999999,"15min":0.999},"min":{"1min":0.39000000000000001,"5min":0.38400000000000001,"15min":0.34799999999999998},"max":{"1min":1.0629999999999999,"5min":21.995999999999999,"15min":21.995999999999999},"last":0.91600000000000004},{"interface":"front","average":{"1min":0.91200000000000003,"5min":1.2609999999999999,"15min":1.0920000000000001},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.3220000000000001,"5min":26.762,"15min":26.762},"last":1.1379999999999999}]},{"osd":2,"last update":"Wed Mar 24 22:05:43 2021","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":1.117,"15min":1.0129999999999999},"min":{"1min":0.51000000000000001,"5min":0.42399999999999999,"15min":0.35299999999999998},"max":{"1min":2.1419999999999999,"5min":13.044,"15min":13.044},"last":1.3109999999999999},{"interface":"front","average":{"1min":0.93500000000000005,"5min":1.3560000000000001,"15min":1.0960000000000001},"min":{"1min":0.435,"5min":0.41199999999999998,"15min":0.33500000000000002},"max":{"1min":1.732,"5min":31.375,"15min":31.375},"last":0.84399999999999997}]},{"osd":3,"last update":"Wed Mar 24 22:05:43 2021","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":1.2909999999999999,"15min":1.0860000000000001},"min":{"1min":0.501,"5min":0.47899999999999998,"15min":0.38400000000000001},"max":{"1min":1.5740000000000001,"5min":22.07,"15min":22.07},"last":0.97199999999999998},{"interface":"front","average":{"1min":1.05,"5min":1.157,"15min":1.109},"min":{"1min":0.51900000000000002,"5min":0.48599999999999999,"15min":0.41299999999999998},"max":{"1min":1.974,"5min":11.583,"15min":11.583},"last":0.93700000000000006}]},{"osd":4,"last update":"Wed Mar 24 22:05:43 2021","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":1.0860000000000001,"15min":1.028},"min":{"1min":0.48799999999999999,"5min":0.44700000000000001,"15min":0.27200000000000002},"max":{"1min":1.3839999999999999,"5min":8.4350000000000005,"15min":8.4350000000000005},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.98999999999999999,"5min":1.2729999999999999,"15min":1.1319999999999999},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.42199999999999999},"max":{"1min":1.3109999999999999,"5min":22.033000000000001,"15min":22.033000000000001},"last":1.0249999999999999}]},{"osd":5,"last update":"Wed Mar 24 22:06:04 2021","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.087,"15min":1.087},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.976,"5min":1.976,"15min":1.976},"last":1.089},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.98999999999999999}]},{"osd":7,"last update":"Wed Mar 24 22:06:07 2021","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.1719999999999999,"15min":1.1899999999999999},"min":{"1min":0.57299999999999995,"5min":0.46999999999999997,"15min":0.30499999999999999},"max":{"1min":1.403,"5min":8.3599999999999994,"15min":9.5909999999999993},"last":1.036},{"interface":"front","average":{"1min":1.135,"5min":1.159,"15min":1.1779999999999999},"min":{"1min":0.59099999999999997,"5min":0.51300000000000001,"15min":0.442},"max":{"1min":1.353,"5min":8.7959999999999994,"15min":8.7959999999999994},"last":1.2110000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575055,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":127928,"kb_used_data":82352,"kb_used_omap":0,"kb_used_meta":45568,"kb_avail":93617224,"statfs":{"total":95995035648,"available":95864037376,"internally_reserved":0,"allocated":84328448,"data_stored":106315418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":46661632},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 22:06:03 2021","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.82499999999999996,"15min":0.83299999999999996},"min":{"1min":0.32700000000000001,"5min":0.27700000000000002,"15min":0.20899999999999999},"max":{"1min":2.0110000000000001,"5min":11.189,"15min":21.085000000000001},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.92400000000000004,"15min":0.80700000000000005},"min":{"1min":0.35799999999999998,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.7430000000000001,"5min":8.5180000000000007,"15min":8.5180000000000007},"last":0.72399999999999998}]},{"osd":2,"last update":"Wed Mar 24 22:06:11 2021","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.77600000000000002,"15min":0.70399999999999996},"min":{"1min":0.38600000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.792,"5min":9.9870000000000001,"15min":9.9870000000000001},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.97899999999999998,"15min":0.85099999999999998},"min":{"1min":0.40400000000000003,"5min":0.29599999999999999,"15min":0.27200000000000002},"max":{"1min":1.415,"5min":9.2059999999999995,"15min":9.2059999999999995},"last":0.86099999999999999}]},{"osd":3,"last update":"Wed Mar 24 22:05:45 2021","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":1.0269999999999999,"15min":0.95499999999999996},"min":{"1min":0.38800000000000001,"5min":0.37,"15min":0.29299999999999998},"max":{"1min":0.92900000000000005,"5min":8.5990000000000002,"15min":17.032},"last":1.1200000000000001},{"interface":"front","average":{"1min":0.82199999999999995,"5min":1.103,"15min":1.0169999999999999},"min":{"1min":0.49299999999999999,"5min":0.33800000000000002,"15min":0.316},"max":{"1min":2.1349999999999998,"5min":15.775,"15min":16.908999999999999},"last":0.93100000000000005}]},{"osd":4,"last update":"Wed Mar 24 22:06:11 2021","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":1.0800000000000001,"15min":0.98899999999999999},"min":{"1min":0.63,"5min":0.49099999999999999,"15min":0.34399999999999997},"max":{"1min":1.2969999999999999,"5min":9.6150000000000002,"15min":9.6150000000000002},"last":1.1830000000000001},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.075,"15min":1.0629999999999999},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.35599999999999998},"max":{"1min":1.5209999999999999,"5min":3.9359999999999999,"15min":3.9359999999999999},"last":1.0149999999999999}]},{"osd":5,"last update":"Wed Mar 24 22:06:07 2021","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.6379999999999999,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":1.0700000000000001},{"interface":"front","average":{"1min":1.1719999999999999,"5min":1.1719999999999999,"15min":1.1719999999999999},"min":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"max":{"1min":2.0750000000000002,"5min":2.0750000000000002,"15min":2.0750000000000002},"last":1.1439999999999999}]},{"osd":6,"last update":"Wed Mar 24 22:05:45 2021","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.2649999999999999,"15min":1.194},"min":{"1min":0.63400000000000001,"5min":0.49399999999999999,"15min":0.48799999999999999},"max":{"1min":2.0459999999999998,"5min":11.254,"15min":11.254},"last":1.0389999999999999},{"interface":"front","average":{"1min":1.052,"5min":1.1850000000000001,"15min":1.131},"min":{"1min":0.70099999999999996,"5min":0.51900000000000002,"15min":0.38100000000000001},"max":{"1min":1.5329999999999999,"5min":9.7300000000000004,"15min":9.7300000000000004},"last":1.323}]},{"osd":7,"last update":"Wed Mar 24 22:06:11 2021","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.159,"15min":1.177},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.38300000000000001},"max":{"1min":1.706,"5min":3.8439999999999999,"15min":4.4699999999999998},"last":1.234},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.2909999999999999,"15min":1.2250000000000001},"min":{"1min":0.70799999999999996,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":2.105,"5min":11.289,"15min":11.289},"last":1.254}]}]},{"osd":0,"up_from":8,"seq":34359738578,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":200744,"kb_used_data":152416,"kb_used_omap":0,"kb_used_meta":48320,"kb_avail":93544408,"statfs":{"total":95995035648,"available":95789473792,"internally_reserved":0,"allocated":156073984,"data_stored":210427250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":49479680},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Mar 24 22:06:06 2021","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.754,"15min":0.70199999999999996},"min":{"1min":0.253,"5min":0.253,"15min":0.249},"max":{"1min":1.0429999999999999,"5min":5.0419999999999998,"15min":5.0419999999999998},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.71299999999999997,"15min":0.76000000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.0660000000000001,"5min":5.4320000000000004,"15min":8.8989999999999991},"last":0.84999999999999998}]},{"osd":2,"last update":"Wed Mar 24 22:06:25 2021","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.68300000000000005,"15min":0.77700000000000002},"min":{"1min":0.47799999999999998,"5min":0.308,"15min":0.28999999999999998},"max":{"1min":0.88700000000000001,"5min":3.8980000000000001,"15min":15.949999999999999},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.753,"5min":0.70099999999999996,"15min":0.77400000000000002},"min":{"1min":0.40400000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.0960000000000001,"5min":1.3580000000000001,"15min":4.766},"last":1.018}]},{"osd":3,"last update":"Wed Mar 24 22:05:41 2021","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":1.121,"15min":0.95599999999999996},"min":{"1min":0.40899999999999997,"5min":0.33800000000000002,"15min":0.28999999999999998},"max":{"1min":1.1659999999999999,"5min":24.75,"15min":24.75},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.89400000000000002,"5min":1.117,"15min":1.008},"min":{"1min":0.40300000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":1.4890000000000001,"5min":24.832999999999998,"15min":24.832999999999998},"last":0.77300000000000002}]},{"osd":4,"last update":"Wed Mar 24 22:06:06 2021","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":0.95599999999999996,"15min":1.0369999999999999},"min":{"1min":0.45500000000000002,"5min":0.373,"15min":0.373},"max":{"1min":1.3360000000000001,"5min":1.4790000000000001,"15min":5.0330000000000004},"last":0.97199999999999998},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.111,"15min":1.0920000000000001},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.44900000000000001},"max":{"1min":1.663,"5min":12.366,"15min":12.366},"last":0.79800000000000004}]},{"osd":5,"last update":"Wed Mar 24 22:06:06 2021","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.522,"5min":1.522,"15min":1.522},"last":0.876},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.746,"5min":1.746,"15min":1.746},"last":0.61599999999999999}]},{"osd":6,"last update":"Wed Mar 24 22:05:41 2021","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.0089999999999999,"15min":1.1379999999999999},"min":{"1min":0.56000000000000005,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.542,"5min":3.9529999999999998,"15min":15.983000000000001},"last":0.81499999999999995},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.137,"15min":1.157},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.53400000000000003},"max":{"1min":1.9970000000000001,"5min":5.2699999999999996,"15min":5.2699999999999996},"last":1.095}]},{"osd":7,"last update":"Wed Mar 24 22:06:16 2021","interfaces":[{"interface":"back","average":{"1min":1.2170000000000001,"5min":1.127,"15min":1.1639999999999999},"min":{"1min":0.67600000000000005,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.8100000000000001,"5min":4.9470000000000001,"15min":6.1319999999999997},"last":0.995},{"interface":"front","average":{"1min":1.254,"5min":1.077,"15min":1.1850000000000001},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":2.6890000000000001,"5min":3.984,"15min":16.007000000000001},"last":0.93500000000000005}]}]},{"osd":2,"up_from":18,"seq":77309411532,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":219860,"kb_used_data":171532,"kb_used_omap":0,"kb_used_meta":48320,"kb_avail":93525292,"statfs":{"total":95995035648,"available":95769899008,"internally_reserved":0,"allocated":175648768,"data_stored":237023305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":49479680},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 22:06:18 2021","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.80900000000000005,"15min":0.76000000000000001},"min":{"1min":0.32400000000000001,"5min":0.254,"15min":0.254},"max":{"1min":1.052,"5min":7.468,"15min":10.054},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.78100000000000003,"15min":0.78600000000000003},"min":{"1min":0.46100000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":2.302,"5min":7.8979999999999997,"15min":12.099},"last":0.91200000000000003}]},{"osd":1,"last update":"Wed Mar 24 22:06:18 2021","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.88700000000000001,"15min":0.84399999999999997},"min":{"1min":0.41199999999999998,"5min":0.32600000000000001,"15min":0.28100000000000003},"max":{"1min":1.0269999999999999,"5min":12.343999999999999,"15min":17.954999999999998},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.86099999999999999,"15min":0.79900000000000004},"min":{"1min":0.36399999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":2.1190000000000002,"5min":7.4989999999999997,"15min":7.4989999999999997},"last":0.82599999999999996}]},{"osd":3,"last update":"Wed Mar 24 22:05:45 2021","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.97099999999999997,"15min":0.97399999999999998},"min":{"1min":0.44900000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.181,"5min":12.308,"15min":25.009},"last":0.95299999999999996},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.97499999999999998,"15min":0.92300000000000004},"min":{"1min":0.50600000000000001,"5min":0.34499999999999997,"15min":0.33900000000000002},"max":{"1min":1.5489999999999999,"5min":7.5199999999999996,"15min":7.5199999999999996},"last":1.022}]},{"osd":4,"last update":"Wed Mar 24 22:06:07 2021","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":1.1140000000000001,"15min":1.1339999999999999},"min":{"1min":0.68000000000000005,"5min":0.45300000000000001,"15min":0.44400000000000001},"max":{"1min":2.6589999999999998,"5min":12.467000000000001,"15min":24.914000000000001},"last":0.79100000000000004},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.0229999999999999,"15min":1.034},"min":{"1min":0.65200000000000002,"5min":0.47199999999999998,"15min":0.41299999999999998},"max":{"1min":1.4419999999999999,"5min":2.8290000000000002,"15min":2.8290000000000002},"last":1.131}]},{"osd":5,"last update":"Wed Mar 24 22:06:07 2021","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.139,"15min":1.139},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.639,"5min":1.639,"15min":1.639},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":2.6869999999999998,"5min":2.6869999999999998,"15min":2.6869999999999998},"last":1.3280000000000001}]},{"osd":6,"last update":"Wed Mar 24 22:05:45 2021","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.0700000000000001,"15min":1.1060000000000001},"min":{"1min":0.69799999999999995,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.7110000000000001,"5min":2.7719999999999998,"15min":2.7719999999999998},"last":1.3580000000000001},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.1020000000000001,"15min":1.121},"min":{"1min":0.84999999999999998,"5min":0.52500000000000002,"15min":0.47899999999999998},"max":{"1min":1.452,"5min":4.3369999999999997,"15min":6.6520000000000001},"last":1.3939999999999999}]},{"osd":7,"last update":"Wed Mar 24 22:06:09 2021","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.167,"15min":1.1839999999999999},"min":{"1min":0.73099999999999998,"5min":0.51800000000000002,"15min":0.442},"max":{"1min":2.6269999999999998,"5min":4.3630000000000004,"15min":6.7720000000000002},"last":1.4430000000000001},{"interface":"front","average":{"1min":1.202,"5min":1.258,"15min":1.29},"min":{"1min":0.73999999999999999,"5min":0.505,"15min":0.41999999999999998},"max":{"1min":2.73,"5min":12.492000000000001,"15min":24.969999999999999},"last":1.1020000000000001}]}]},{"osd":3,"up_from":26,"seq":111669149895,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303184,"kb_used_data":243592,"kb_used_omap":0,"kb_used_meta":59584,"kb_avail":93441968,"statfs":{"total":95995035648,"available":95684575232,"internally_reserved":0,"allocated":249438208,"data_stored":328004335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":61014016},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 22:05:40 2021","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":1.056,"15min":0.89500000000000002},"min":{"1min":0.29699999999999999,"5min":0.27500000000000002,"15min":0.23899999999999999},"max":{"1min":0.84799999999999998,"5min":27.548999999999999,"15min":27.548999999999999},"last":1.0129999999999999},{"interface":"front","average":{"1min":0.70499999999999996,"5min":1.0149999999999999,"15min":0.90400000000000003},"min":{"1min":0.39600000000000002,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.097,"5min":26.593,"15min":26.593},"last":0.85999999999999999}]},{"osd":1,"last update":"Wed Mar 24 22:05:40 2021","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.76000000000000001,"15min":0.78500000000000003},"min":{"1min":0.378,"5min":0.314,"15min":0.26700000000000002},"max":{"1min":1.1879999999999999,"5min":5.2610000000000001,"15min":14.324},"last":0.96099999999999997},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.78700000000000003,"15min":0.79000000000000004},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28000000000000003},"max":{"1min":1.0620000000000001,"5min":4.5540000000000003,"15min":12.631},"last":0.98599999999999999}]},{"osd":2,"last update":"Wed Mar 24 22:05:40 2021","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.85399999999999998,"15min":0.84299999999999997},"min":{"1min":0.5,"5min":0.34699999999999998,"15min":0.316},"max":{"1min":1.0600000000000001,"5min":7.9450000000000003,"15min":12.510999999999999},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.81599999999999995,"15min":0.84299999999999997},"min":{"1min":0.47999999999999998,"5min":0.41799999999999998,"15min":0.34899999999999998},"max":{"1min":1.177,"5min":4.5640000000000001,"15min":11.317},"last":0.64800000000000002}]},{"osd":4,"last update":"Wed Mar 24 22:06:02 2021","interfaces":[{"interface":"back","average":{"1min":1.204,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.89500000000000002,"5min":0.622,"15min":0.42999999999999999},"max":{"1min":2.5880000000000001,"5min":2.5880000000000001,"15min":2.601},"last":0.80700000000000005},{"interface":"front","average":{"1min":1.54,"5min":1.1060000000000001,"15min":1.0640000000000001},"min":{"1min":0.81799999999999995,"5min":0.47899999999999998,"15min":0.32900000000000001},"max":{"1min":4.0700000000000003,"5min":4.9569999999999999,"15min":14.291},"last":1.4430000000000001}]},{"osd":5,"last update":"Wed Mar 24 22:06:08 2021","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"min":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":2.7400000000000002,"5min":2.7400000000000002,"15min":2.7400000000000002},"last":1.0640000000000001},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"min":{"1min":0.879,"5min":0.879,"15min":0.879},"max":{"1min":2.6949999999999998,"5min":2.6949999999999998,"15min":2.6949999999999998},"last":0.93700000000000006}]},{"osd":6,"last update":"Wed Mar 24 22:05:40 2021","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.0669999999999999,"15min":1.103},"min":{"1min":0.745,"5min":0.59799999999999998,"15min":0.40000000000000002},"max":{"1min":1.5189999999999999,"5min":4.9950000000000001,"15min":14.255000000000001},"last":1.4910000000000001},{"interface":"front","average":{"1min":1.073,"5min":1.036,"15min":1.1080000000000001},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.39700000000000002},"max":{"1min":2.5579999999999998,"5min":2.5579999999999998,"15min":12.666},"last":0.89900000000000002}]},{"osd":7,"last update":"Wed Mar 24 22:06:08 2021","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.089,"15min":1.1180000000000001},"min":{"1min":0.81499999999999995,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":2.2189999999999999,"5min":2.2189999999999999,"15min":2.633},"last":1.141},{"interface":"front","average":{"1min":1.5960000000000001,"5min":1.216,"15min":1.23},"min":{"1min":0.98999999999999999,"5min":0.60399999999999998,"15min":0.44700000000000001},"max":{"1min":4.1159999999999997,"5min":5.0190000000000001,"15min":14.727},"last":1.3420000000000001}]}]},{"osd":4,"up_from":32,"seq":137438953665,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":106504,"kb_used_data":67200,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":93638648,"statfs":{"total":95995035648,"available":95885975552,"internally_reserved":0,"allocated":68812800,"data_stored":91109171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 22:06:09 2021","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":0.999,"15min":1.034},"min":{"1min":0.60999999999999999,"5min":0.437,"15min":0.223},"max":{"1min":1.373,"5min":5.9050000000000002,"15min":41.472000000000001},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.95099999999999996,"15min":1.1499999999999999},"min":{"1min":0.60899999999999999,"5min":0.41999999999999998,"15min":0.34000000000000002},"max":{"1min":1.3100000000000001,"5min":5.3719999999999999,"15min":44.185000000000002},"last":0.89100000000000001}]},{"osd":1,"last update":"Wed Mar 24 22:06:09 2021","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":1.0189999999999999,"15min":0.94999999999999996},"min":{"1min":0.57999999999999996,"5min":0.42699999999999999,"15min":0.38100000000000001},"max":{"1min":1.3460000000000001,"5min":7.5060000000000002,"15min":7.5060000000000002},"last":0.748},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.149,"15min":0.99199999999999999},"min":{"1min":0.70699999999999996,"5min":0.48399999999999999,"15min":0.36399999999999999},"max":{"1min":1.528,"5min":11.983000000000001,"15min":11.983000000000001},"last":0.81000000000000005}]},{"osd":2,"last update":"Wed Mar 24 22:06:09 2021","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.87,"15min":0.90500000000000003},"min":{"1min":0.66300000000000003,"5min":0.46300000000000002,"15min":0.35899999999999999},"max":{"1min":1.2929999999999999,"5min":3.21,"15min":4.4379999999999997},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.995,"5min":1.0009999999999999,"15min":0.98899999999999999},"min":{"1min":0.626,"5min":0.439,"15min":0.40500000000000003},"max":{"1min":1.3640000000000001,"5min":4.3970000000000002,"15min":5.7999999999999998},"last":0.93999999999999995}]},{"osd":3,"last update":"Wed Mar 24 22:06:09 2021","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":1.085,"15min":1.038},"min":{"1min":0.64700000000000002,"5min":0.501,"15min":0.439},"max":{"1min":1.2869999999999999,"5min":3.9809999999999999,"15min":4.8049999999999997},"last":0.86599999999999999},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.0209999999999999,"15min":1.01},"min":{"1min":0.84999999999999998,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.3779999999999999,"5min":2.4700000000000002,"15min":4.1189999999999998},"last":1.0620000000000001}]},{"osd":5,"last update":"Wed Mar 24 22:06:03 2021","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.22,"5min":1.22,"15min":1.22},"last":1.1100000000000001},{"interface":"front","average":{"1min":1.202,"5min":1.202,"15min":1.202},"min":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":1.9550000000000001,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":1.024}]},{"osd":6,"last update":"Wed Mar 24 22:05:31 2021","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.2110000000000001,"15min":1.0880000000000001},"min":{"1min":0.59299999999999997,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":2.2309999999999999,"5min":7.7679999999999998,"15min":7.7679999999999998},"last":0.99199999999999999},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.331,"15min":1.149},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.41099999999999998},"max":{"1min":2.7389999999999999,"5min":14.122999999999999,"15min":14.122999999999999},"last":0.96499999999999997}]},{"osd":7,"last update":"Wed Mar 24 22:06:09 2021","interfaces":[{"interface":"back","average":{"1min":1.3089999999999999,"5min":1.196,"15min":1.147},"min":{"1min":0.87,"5min":0.46200000000000002,"15min":0.44400000000000001},"max":{"1min":1.8700000000000001,"5min":2.399,"15min":3.3479999999999999},"last":1.0049999999999999},{"interface":"front","average":{"1min":1.23,"5min":1.1160000000000001,"15min":1.1319999999999999},"min":{"1min":0.97299999999999998,"5min":0.49399999999999999,"15min":0.49299999999999999},"max":{"1min":1.7090000000000001,"5min":3.1680000000000001,"15min":3.1680000000000001},"last":1.196}]}]},{"osd":5,"up_from":733,"seq":3148211027987,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":162084,"kb_used_data":148956,"kb_used_omap":164,"kb_used_meta":12955,"kb_avail":93583068,"statfs":{"total":95995035648,"available":95829061632,"internally_reserved":0,"allocated":152530944,"data_stored":189223145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":168054,"internal_metadata":13266826},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 22:06:03 2021","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.6699999999999999,"5min":1.6699999999999999,"15min":1.6699999999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.4339999999999999,"5min":1.4339999999999999,"15min":1.4339999999999999},"last":0.75800000000000001}]},{"osd":1,"last update":"Wed Mar 24 22:06:03 2021","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.425,"5min":1.425,"15min":1.425},"last":0.96999999999999997},{"interface":"front","average":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":2.0939999999999999,"5min":2.0939999999999999,"15min":2.0939999999999999},"last":0.86299999999999999}]},{"osd":2,"last update":"Wed Mar 24 22:06:03 2021","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.778,"5min":1.778,"15min":1.778},"last":1.091},{"interface":"front","average":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":1.056}]},{"osd":3,"last update":"Wed Mar 24 22:06:03 2021","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.669,"5min":1.669,"15min":1.669},"last":0.94299999999999995},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"min":{"1min":0.755,"5min":0.755,"15min":0.755},"max":{"1min":1.863,"5min":1.863,"15min":1.863},"last":1.2050000000000001}]},{"osd":4,"last update":"Wed Mar 24 22:06:03 2021","interfaces":[{"interface":"back","average":{"1min":1.093,"5min":1.093,"15min":1.093},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":2.5219999999999998,"5min":2.5219999999999998,"15min":2.5219999999999998},"last":1.4219999999999999},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":2.2450000000000001,"5min":2.2450000000000001,"15min":2.2450000000000001},"last":1.024}]},{"osd":6,"last update":"Wed Mar 24 22:06:03 2021","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":1.131,"15min":1.131},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":2.0550000000000002,"5min":2.0550000000000002,"15min":2.0550000000000002},"last":1.284},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.74,"5min":1.74,"15min":1.74},"last":1.786}]},{"osd":7,"last update":"Wed Mar 24 22:06:03 2021","interfaces":[{"interface":"back","average":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":3.0430000000000001,"5min":3.0430000000000001,"15min":3.0430000000000001},"last":1.137},{"interface":"front","average":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":2.698,"5min":2.698,"15min":2.698},"last":1.5880000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10235,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":5284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":475136,"data_stored":17122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":5418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":241664,"data_stored":8622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":307200,"data_stored":11122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":5674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":10356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":6974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":149229568,"data_stored":206472990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77152256,"data_stored":102336469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":168787968,"data_stored":233049270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":242511872,"data_stored":324041398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":61820928,"data_stored":87150335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":145645568,"data_stored":185240842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":142409728,"data_stored":199109197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":199249920,"data_stored":275823348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-03-24T22:06:34.120 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph pg dump --format=json 2021-03-24T22:06:34.121 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-24T22:06:34.149 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:34.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:34.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:34.255 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:34.309 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:34.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:34.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:06:34.417 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:34.482 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:34.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:34.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:06:34.490 INFO:tasks.rados.rados.0.smithi125.stdout:3185: done (2 left) 2021-03-24T22:06:34.491 INFO:tasks.rados.rados.0.smithi125.stdout:3186: done (1 left) 2021-03-24T22:06:34.491 INFO:tasks.rados.rados.0.smithi125.stdout:3187: done (0 left) 2021-03-24T22:06:34.491 INFO:tasks.rados.rados.0.smithi125.stdout:3188: rollback oid 9 current snap is 369 2021-03-24T22:06:34.491 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 9 to 344 2021-03-24T22:06:34.491 INFO:tasks.rados.rados.0.smithi125.stdout:3189: delete oid 23 current snap is 369 2021-03-24T22:06:34.496 INFO:tasks.rados.rados.0.smithi125.stdout:3189: done (1 left) 2021-03-24T22:06:34.496 INFO:tasks.rados.rados.0.smithi125.stdout:3190: read oid 1 snap -1 2021-03-24T22:06:34.496 INFO:tasks.rados.rados.0.smithi125.stdout:3190: expect (ObjNum 808 snap 265 seq_num 808) 2021-03-24T22:06:34.496 INFO:tasks.rados.rados.0.smithi125.stdout:3191: read oid 33 snap -1 2021-03-24T22:06:34.497 INFO:tasks.rados.rados.0.smithi125.stdout:3191: expect (ObjNum 916 snap 304 seq_num 916) 2021-03-24T22:06:34.497 INFO:tasks.rados.rados.0.smithi125.stdout:3192: delete oid 5 current snap is 369 2021-03-24T22:06:34.498 INFO:tasks.rados.rados.0.smithi125.stdout:3188: finishing rollback tid 1 to smithi12556470-9 2021-03-24T22:06:34.499 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 9 v 1583 (ObjNum 1112276096 snap 21916 seq_num 4000000) dirty dne 2021-03-24T22:06:34.505 INFO:tasks.rados.rados.0.smithi125.stdout:3188: done (3 left) 2021-03-24T22:06:34.506 INFO:tasks.rados.rados.0.smithi125.stdout:3192: done (2 left) 2021-03-24T22:06:34.506 INFO:tasks.rados.rados.0.smithi125.stdout:3193: rollback oid 27 current snap is 369 2021-03-24T22:06:34.506 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 27 to 347 2021-03-24T22:06:34.506 INFO:tasks.rados.rados.0.smithi125.stdout:3194: snap_remove snap 355 2021-03-24T22:06:34.588 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:34.646 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:34.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:34.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:06:34.751 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:34.763 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:34 smithi125 conmon[24927]: audit 2021-03-24T22:06:32.704101+0000 mgr.y (mgr.14144) 686 : audit [DBG] from='client.35864 v1:172.21.15.125:0/3963636701' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T22:06:34.763 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:34 smithi125 conmon[24927]: cluster 2021-03-24T22:06:33.006511+0000 mgr.y (mgr.14144) 687 : cluster [DBG] pgmap v1396: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 501 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 0 B/s wr, 3 op/s; 11/1635 objects misplaced (0.673%) 2021-03-24T22:06:34.764 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:34 smithi125 conmon[24927]: cluster 2021-03-24T22:06:33.470590+0000 mon.a (mon.0) 2051 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:06:34.764 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:34 smithi125 conmon[24927]: audit 2021-03-24T22:06:33.485428+0000 mon.a (mon.0) 2052 : audit [INF] from='client.35870 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T22:06:34.764 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:34 smithi125 conmon[24927]: cluster 2021-03-24T22:06:33.485508+0000 mon.a (mon.0) 2053 : cluster [DBG] osdmap e823: 8 total, 8 up, 8 in 2021-03-24T22:06:34.764 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:34 smithi125 conmon[24927]: cluster 2021-03-24T22:06:33.822397+0000 osd.1 (osd.1) 92 : cluster [DBG] 5.c scrub ok 2021-03-24T22:06:34.765 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:34 smithi125 conmon[19936]: audit 2021-03-24T22:06:32 2021-03-24T22:06:34.765 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:34 smithi125 conmon[19936]: .704101+0000 mgr.y (mgr.14144) 686 : audit [DBG] from='client.35864 v1:172.21.15.125:0/3963636701' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T22:06:34.765 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:34 smithi125 conmon[19936]: cluster 2021-03-24T22:06 2021-03-24T22:06:34.765 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:34 smithi125 conmon[19936]: :33.006511+0000 mgr.y (mgr.14144) 687 : cluster [DBG] pgmap v1396: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 501 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 0 B/s wr, 3 op/s; 11/1635 objects misplaced (0.673%) 2021-03-24T22:06:34.766 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:34 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T22:06:34.766 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:34 smithi125 conmon[19936]: -24T22:06:33.470590+0000 mon.a (mon 2021-03-24T22:06:34.766 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:34 smithi125 conmon[19936]: .0) 2051 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:06:34.766 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:34 smithi125 conmon[19936]: audit 2021-03-24T 2021-03-24T22:06:34.766 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:34 smithi125 conmon[19936]: 22:06:33.485428+0000 mon.a (mon.0 2021-03-24T22:06:34.767 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:34 smithi125 conmon[19936]: ) 2052 : audit [INF] from='client.35870 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T22:06:34.767 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:34 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:06:34.767 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:34 smithi125 conmon[19936]: 06:33.485508+0000 mon.a (mon.0) 2053 : 2021-03-24T22:06:34.767 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:34 smithi125 conmon[19936]: cluster [DBG] osdmap e823: 8 total, 8 up, 8 in 2021-03-24T22:06:34.767 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:34 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:06:34.768 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:34 smithi125 conmon[19936]: T22:06:33.822397+0000 osd.1 (osd. 2021-03-24T22:06:34.768 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:34 smithi125 conmon[19936]: 1) 92 : cluster [DBG] 5.c scrub ok 2021-03-24T22:06:34.802 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:34.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:34.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:06:34.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:34 smithi142 conmon[20125]: audit 2021-03-24T22:06:32.704101+0000 mgr.y (mgr.14144) 686 : 2021-03-24T22:06:34.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:34 smithi142 conmon[20125]: audit [DBG] from='client.35864 v1:172.21.15.125:0/3963636701' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T22:06:34.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:34 smithi142 conmon[20125]: cluster 2021-03-24T22:06:33.006511+0000 mgr.y (mgr.14144 2021-03-24T22:06:34.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:34 smithi142 conmon[20125]: ) 687 : cluster [DBG] pgmap v1396: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 501 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 0 B/s wr, 3 op/s; 11/1635 objects misplaced (0.673%) 2021-03-24T22:06:34.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:34 smithi142 conmon[20125]: cluster 2021-03-24T22:06:34.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:34 smithi142 conmon[20125]: 2021-03-24T22:06:33.470590+0000 mon.a (mon 2021-03-24T22:06:34.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:34 smithi142 conmon[20125]: .0) 2051 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:06:34.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:34 smithi142 conmon[20125]: audit 2021-03- 2021-03-24T22:06:34.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:34 smithi142 conmon[20125]: 24T22:06:33.485428+0000 mon.a 2021-03-24T22:06:34.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:34 smithi142 conmon[20125]: (mon.0) 2052 : audit [INF] 2021-03-24T22:06:34.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:34 smithi142 conmon[20125]: from='client.35870 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T22:06:34.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:34 smithi142 conmon[20125]: cluster 2021-03-24T22: 2021-03-24T22:06:34.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:34 smithi142 conmon[20125]: 06:33.485508+0000 mon.a (mon.0 2021-03-24T22:06:34.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:34 smithi142 conmon[20125]: ) 2053 : cluster [DBG] osdmap e823: 8 total, 8 up, 8 in 2021-03-24T22:06:34.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:34 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:06:34.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:34 smithi142 conmon[20125]: 24T22:06:33.822397+0000 osd.1 2021-03-24T22:06:34.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:34 smithi142 conmon[20125]: (osd.1) 92 : cluster [DBG] 5.c scrub ok 2021-03-24T22:06:34.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:34 smithi142 conmon[20125]: 2021-03-24T22:06:34.909 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:34.964 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:34.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:34.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:06:35.071 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:35.131 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:35.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:35.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:06:35.238 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:35.299 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:35.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:35.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:06:35.406 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:35.445 INFO:tasks.rados.rados.0.smithi125.stdout:3194: done (3 left) 2021-03-24T22:06:35.445 INFO:tasks.rados.rados.0.smithi125.stdout:3195: read oid 2 snap 314 2021-03-24T22:06:35.445 INFO:tasks.rados.rados.0.smithi125.stdout:3195: expect (ObjNum 917 snap 305 seq_num 917) 2021-03-24T22:06:35.446 INFO:tasks.rados.rados.0.smithi125.stdout:3196: copy_from oid 36 from oid 21 current snap is 369 2021-03-24T22:06:35.446 INFO:tasks.rados.rados.0.smithi125.stdout:3197: read oid 34 snap 346 2021-03-24T22:06:35.446 INFO:tasks.rados.rados.0.smithi125.stdout:3197: expect deleted 2021-03-24T22:06:35.446 INFO:tasks.rados.rados.0.smithi125.stdout:3198: snap_create 2021-03-24T22:06:35.454 INFO:tasks.rados.rados.0.smithi125.stdout:3193: finishing rollback tid 0 to smithi12556470-27 2021-03-24T22:06:35.455 INFO:tasks.rados.rados.0.smithi125.stdout:3193: finishing rollback tid 1 to smithi12556470-27 2021-03-24T22:06:35.455 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 27 v 1435 (ObjNum 1117423312 snap 21916 seq_num 1114771472) dirty dne 2021-03-24T22:06:35.455 INFO:tasks.rados.rados.0.smithi125.stdout:3190: expect (ObjNum 808 snap 265 seq_num 808) 2021-03-24T22:06:35.475 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:35.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:35.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:06:35.581 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:35.618 INFO:tasks.rados.rados.0.smithi125.stdout:3191: expect (ObjNum 916 snap 304 seq_num 916) 2021-03-24T22:06:35.632 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:35.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:35.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:06:35.736 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:35.796 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:35.801 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:35 smithi142 conmon[20125]: cluster 2021-03-24T22:06:34.008419+0000 osd.7 ( 2021-03-24T22:06:35.801 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:35 smithi142 conmon[20125]: osd.7) 134 : cluster [DBG] 5.1a scrub ok 2021-03-24T22:06:35.801 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:35 smithi142 conmon[20125]: cluster 2021-03-24T22:06:34.167230+0000 osd.2 (osd.2) 132 : cluster [DBG] 4.d scrub ok 2021-03-24T22:06:35.802 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:35 smithi142 conmon[20125]: cluster 2021-03-24T22:06:35.802 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:35 smithi142 conmon[20125]: 2021-03-24T22:06:34.336420+0000 osd.5 (osd.5) 21 : cluster [DBG] 6.19 scrub ok 2021-03-24T22:06:35.802 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:35 smithi142 conmon[20125]: cluster 2021-03-24T22: 2021-03-24T22:06:35.802 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:35 smithi142 conmon[20125]: 06:34.487482+0000 mon.a (mon.0) 2054 : cluster [DBG] osdmap e824: 8 total, 8 up, 8 in 2021-03-24T22:06:35.803 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:35 smithi142 conmon[20125]: cluster 2021-03-24T22:06:34 2021-03-24T22:06:35.803 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:35 smithi142 conmon[20125]: .796153+0000 osd.1 (osd.1) 93 : cluster [DBG] 6.e scrub ok 2021-03-24T22:06:35.803 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:35 smithi142 conmon[20125]: cluster 2021-03-24T22:06: 2021-03-24T22:06:35.803 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:35 smithi142 conmon[20125]: 35.291277+0000 osd.6 (osd.6) 84 : cluster [DBG] 7.1a scrub ok 2021-03-24T22:06:35.803 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:35 smithi142 conmon[20125]: cluster 2021-03-24T22:06:35.804 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:35 smithi142 conmon[20125]: 2021-03-24T22:06:35.435307+0000 mon.a (mon.0) 2055 : cluster [DBG] osdmap e825: 8 total, 8 up, 8 in 2021-03-24T22:06:35.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:35.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:06:35.810 INFO:tasks.rados.rados.0.smithi125.stdout:3195: expect (ObjNum 917 snap 305 seq_num 917) 2021-03-24T22:06:35.906 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:35.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:35 smithi125 conmon[19936]: cluster 2021-03-24T22:06:34. 2021-03-24T22:06:35.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:35 smithi125 conmon[19936]: 008419+0000 osd.7 (osd.7) 134 : cluster [DBG] 5.1a scrub ok 2021-03-24T22:06:35.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:35 smithi125 conmon[19936]: cluster 2021-03-24T22:06:34.167230+0000 osd.2 (osd 2021-03-24T22:06:35.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:35 smithi125 conmon[19936]: .2) 132 : cluster [DBG] 4.d scrub ok 2021-03-24T22:06:35.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:35 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:06:35.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:35 smithi125 conmon[19936]: 06:34.336420+0000 osd.5 (osd.5) 21 : cluster [DBG] 2021-03-24T22:06:35.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:35 smithi125 conmon[19936]: 6.19 scrub ok 2021-03-24T22:06:35.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:35 smithi125 conmon[19936]: cluster 2021-03-24T22:06:35.915 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:35 smithi125 conmon[19936]: 2021-03-24T22: 2021-03-24T22:06:35.915 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:35 smithi125 conmon[19936]: 06:34.487482+0000 mon.a (mon.0) 2054 : cluster 2021-03-24T22:06:35.915 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:35 smithi125 conmon[19936]: [DBG] osdmap e824: 8 total, 8 up, 8 in 2021-03-24T22:06:35.916 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:35 smithi125 conmon[19936]: cluster 2021-03-24T22:06:35.916 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:35 smithi125 conmon[19936]: 2021-03-24T22:06:34.796153 2021-03-24T22:06:35.916 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:35 smithi125 conmon[19936]: +0000 osd.1 (osd.1) 93 : cluster [DBG] 6.e scrub ok 2021-03-24T22:06:35.916 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:35 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:06:35.916 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:35 smithi125 conmon[19936]: 03-24T22:06:35.291277+0000 2021-03-24T22:06:35.917 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:35 smithi125 conmon[19936]: osd.6 (osd.6) 84 : cluster [DBG] 7.1a scrub ok 2021-03-24T22:06:35.917 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:35 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:06:35.917 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:35 smithi125 conmon[19936]: T22:06:35.435307+0000 mon.a (mon.0) 2021-03-24T22:06:35.917 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:35 smithi125 conmon[19936]: 2055 : cluster [DBG] osdmap e825: 8 total, 8 up, 8 in 2021-03-24T22:06:35.918 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:35 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T22:06:35.918 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:35 smithi125 conmon[24927]: 22:06:34.008419+0000 osd.7 (osd.7) 134 : cluster [DBG] 5.1a scrub ok 2021-03-24T22:06:35.918 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:35 smithi125 conmon[24927]: 2021-03-24T22:06:35.919 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:35 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:06:35.919 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:35 smithi125 conmon[24927]: -03-24T22:06:34.167230 2021-03-24T22:06:35.919 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:35 smithi125 conmon[24927]: +0000 osd.2 (osd.2) 132 : 2021-03-24T22:06:35.919 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:35 smithi125 conmon[24927]: cluster [DBG] 4.d scrub ok 2021-03-24T22:06:35.919 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:35 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:06:35.920 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:35 smithi125 conmon[24927]: -03-24T22:06:34. 2021-03-24T22:06:35.920 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:35 smithi125 conmon[24927]: 336420+0000 osd.5 (osd.5) 2021-03-24T22:06:35.920 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:35 smithi125 conmon[24927]: 21 : cluster [DBG] 6.19 scrub ok 2021-03-24T22:06:35.920 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:35 smithi125 conmon[24927]: cluster 2021-03-24T22:06:35.920 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:35 smithi125 conmon[24927]: 2021-03-24T22:06: 2021-03-24T22:06:35.921 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:35 smithi125 conmon[24927]: 34.487482+0000 mon.a (mon.0) 2054 : cluster 2021-03-24T22:06:35.921 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:35 smithi125 conmon[24927]: [DBG] osdmap e824: 8 total, 8 up, 8 in 2021-03-24T22:06:35.921 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:35 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T22:06:35.921 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:35 smithi125 conmon[24927]: T22:06:34.796153+0000 osd.1 2021-03-24T22:06:35.922 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:35 smithi125 conmon[24927]: (osd.1) 93 : cluster [DBG] 2021-03-24T22:06:35.922 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:35 smithi125 conmon[24927]: 6.e scrub ok 2021-03-24T22:06:35.922 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:35 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T22:06:35.922 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:35 smithi125 conmon[24927]: T22:06:35.291277 2021-03-24T22:06:35.922 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:35 smithi125 conmon[24927]: +0000 osd.6 (osd.6) 84 : cluster [DBG] 2021-03-24T22:06:35.923 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:35 smithi125 conmon[24927]: 7.1a scrub ok 2021-03-24T22:06:35.923 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:35 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:06:35.923 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:35 smithi125 conmon[24927]: -03-24T22:06:35.435307+0000 2021-03-24T22:06:35.923 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:35 smithi125 conmon[24927]: mon.a (mon.0) 2055 : cluster 2021-03-24T22:06:35.923 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:35 smithi125 conmon[24927]: [DBG] osdmap e825: 8 total, 8 up, 8 in 2021-03-24T22:06:35.961 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:35.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:35.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:06:35.995 INFO:tasks.rados.rados.0.smithi125.stdout:3196: finishing copy_from to smithi12556470-36 2021-03-24T22:06:35.996 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 36 v 1441 (ObjNum 851 snap 278 seq_num 851) dirty exists 2021-03-24T22:06:35.996 INFO:tasks.rados.rados.0.smithi125.stdout:3196: finishing copy_from racing read to smithi12556470-36 2021-03-24T22:06:36.069 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:36.122 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:36.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:36.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:06:36.229 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:36.282 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:36.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:36.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:06:36.388 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:36.402 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:06:36.424 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.3 dump_blocked_ops 2021-03-24T22:06:36.448 INFO:tasks.rados.rados.0.smithi125.stdout:3190: done (6 left) 2021-03-24T22:06:36.448 INFO:tasks.rados.rados.0.smithi125.stdout:3191: done (5 left) 2021-03-24T22:06:36.448 INFO:tasks.rados.rados.0.smithi125.stdout:3193: done (4 left) 2021-03-24T22:06:36.448 INFO:tasks.rados.rados.0.smithi125.stdout:3195: done (3 left) 2021-03-24T22:06:36.449 INFO:tasks.rados.rados.0.smithi125.stdout:3196: done (2 left) 2021-03-24T22:06:36.449 INFO:tasks.rados.rados.0.smithi125.stdout:3197: done (1 left) 2021-03-24T22:06:36.449 INFO:tasks.rados.rados.0.smithi125.stdout:3198: done (0 left) 2021-03-24T22:06:36.449 INFO:tasks.rados.rados.0.smithi125.stdout:3199: snap_remove snap 335 2021-03-24T22:06:36.450 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:36.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:36.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:06:36.476 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:06:36.558 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:36.604 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:36.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:36.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:36.658 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:36 smithi125 conmon[24927]: cluster 2021-03-24T22:06:35 2021-03-24T22:06:36.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:36 smithi125 conmon[24927]: .007298+0000 mgr.y (mgr.14144) 688 : cluster [DBG] pgmap v1399: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 515 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 7.0 MiB/s rd, 2.2 MiB/s wr, 11 op/s; 11/1653 objects misplaced (0.665%) 2021-03-24T22:06:36.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:36 smithi125 conmon[24927]: cluster 2021-03-24T22:06:35. 2021-03-24T22:06:36.659 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:36 smithi125 conmon[24927]: 800656+0000 osd.1 (osd.1) 94 : cluster [DBG] 4.f scrub ok 2021-03-24T22:06:36.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:36 smithi125 conmon[24927]: 2021-03-24T22:06:36.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:36 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:06:36.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:36 smithi125 conmon[24927]: 03-24T22:06:36.049660+0000 osd.7 (osd.7) 135 : cluster 2021-03-24T22:06:36.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:36 smithi125 conmon[24927]: [DBG] 6.3 scrub ok 2021-03-24T22:06:36.660 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:36 smithi125 conmon[24927]: cluster 2021-03-24T22:06:36.444805 2021-03-24T22:06:36.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:36 smithi125 conmon[24927]: +0000 mon.a (mon.0) 2056 : cluster [DBG] osdmap e826: 8 total, 8 up, 8 in 2021-03-24T22:06:36.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:36 smithi125 conmon[24927]: 2021-03-24T22:06:36.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:36 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:06:36.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:36 smithi125 conmon[19936]: 24T22:06:35.007298+0000 mgr.y (mgr.14144) 688 : cluster [DBG] 2021-03-24T22:06:36.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:36 smithi125 conmon[19936]: pgmap v1399: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 515 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 7.0 MiB/s rd, 2.2 MiB/s wr, 11 op/s; 11/1653 objects misplaced (0.665%) 2021-03-24T22:06:36.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:36 smithi125 conmon[19936]: cluster 2021-03-24T22:06:36.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:36 smithi125 conmon[19936]: 2021-03-24T22 2021-03-24T22:06:36.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:36 smithi125 conmon[19936]: :06:35.800656 2021-03-24T22:06:36.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:36 smithi125 conmon[19936]: +0000 osd.1 (osd 2021-03-24T22:06:36.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:36 smithi125 conmon[19936]: .1) 94 : cluster 2021-03-24T22:06:36.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:36 smithi125 conmon[19936]: [DBG] 4.f scrub ok 2021-03-24T22:06:36.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:36 smithi125 conmon[19936]: cluster 2021-03-24T22:06:36.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:36 smithi125 conmon[19936]: 2021-03-24T 2021-03-24T22:06:36.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:36 smithi125 conmon[19936]: 22:06:36. 2021-03-24T22:06:36.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:36 smithi125 conmon[19936]: 049660+0000 osd.7 (osd 2021-03-24T22:06:36.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:36 smithi125 conmon[19936]: .7) 135 : cluster 2021-03-24T22:06:36.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:36 smithi125 conmon[19936]: [DBG] 6.3 scrub ok 2021-03-24T22:06:36.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:36 smithi125 conmon[19936]: cluster 2021-03-24T22:06:36.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:36 smithi125 conmon[19936]: 2021-03-24T22 2021-03-24T22:06:36.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:36 smithi125 conmon[19936]: :06:36.444805 2021-03-24T22:06:36.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:36 smithi125 conmon[19936]: +0000 mon.a (mon. 2021-03-24T22:06:36.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:36 smithi125 conmon[19936]: 0) 2056 : cluster 2021-03-24T22:06:36.666 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:36 smithi125 conmon[19936]: [DBG] osdmap e826: 8 total, 8 up, 8 in 2021-03-24T22:06:36.713 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:36.769 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:36.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:36.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:06:36.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:36 smithi142 conmon[20125]: cluster 2021-03-24T22:06:35.007298+0000 mgr.y (mgr.14144) 2021-03-24T22:06:36.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:36 smithi142 conmon[20125]: 688 : cluster [DBG] pgmap v1399: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 515 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 7.0 MiB/s rd, 2.2 MiB/s wr, 11 op/s; 11/1653 objects misplaced (0.665%) 2021-03-24T22:06:36.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:36 smithi142 conmon[20125]: cluster 2021-03-24T22:06:35.800656+0000 osd.1 (osd.1) 94 : cluster [DBG] 4.f scrub ok 2021-03-24T22:06:36.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:36 smithi142 conmon[20125]: cluster 2021-03-24T22:06:36. 2021-03-24T22:06:36.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:36 smithi142 conmon[20125]: 049660+0000 osd.7 (osd.7) 135 : cluster [DBG] 6.3 scrub ok 2021-03-24T22:06:36.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:36 smithi142 conmon[20125]: cluster 2021-03-24T22 2021-03-24T22:06:36.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:36 smithi142 conmon[20125]: :06:36.444805+0000 mon.a (mon.0) 2056 : cluster [DBG] osdmap e826: 8 total, 8 up, 8 in 2021-03-24T22:06:36.876 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:36.935 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:36.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:36.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:37.030 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:06:37.041 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:37.102 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:37.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:37.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:37.209 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:37.263 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:37.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:37.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:06:37.371 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:37.431 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:37.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:37.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:06:37.448 INFO:tasks.rados.rados.0.smithi125.stdout:3199: done (0 left) 2021-03-24T22:06:37.449 INFO:tasks.rados.rados.0.smithi125.stdout:3200: snap_remove snap 369 2021-03-24T22:06:37.537 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:37.594 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:37.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:37.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:06:37.700 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:37.754 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:37.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:37.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:06:37.760 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[24927]: cluster 2021-03-24T22:06:37.760 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[24927]: 2021-03 2021-03-24T22:06:37.760 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[24927]: -24T22:06:36.314946+0000 osd.5 (osd.5) 22 : cluster [DBG] 2021-03-24T22:06:37.760 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[24927]: 4.15 scrub ok 2021-03-24T22:06:37.760 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:06:37.761 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[24927]: 03-24T22:06:36.436890+0000 osd.1 2021-03-24T22:06:37.761 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[24927]: (osd.1) 95 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:06:37.761 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:06:37.761 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[24927]: -03-24T22:06:36.438575+0000 osd.1 (osd 2021-03-24T22:06:37.762 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[24927]: .1) 96 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:06:37.762 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:06:37.762 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[24927]: -03-24T22:06:36.792533+0000 osd.1 2021-03-24T22:06:37.762 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[24927]: (osd.1) 97 : cluster [DBG] 3.25 scrub ok 2021-03-24T22:06:37.763 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[24927]: cluster 2021-03-24T22:06:37.763 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[24927]: 2021-03-24T22:06:37.049836+0000 2021-03-24T22:06:37.763 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[24927]: osd.7 (osd.7) 136 : cluster [DBG] 3.e scrub ok 2021-03-24T22:06:37.763 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[24927]: 2021-03-24T22:06:37.763 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[24927]: cluster 2021-03-24T22:06:37. 2021-03-24T22:06:37.764 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[24927]: 118203+0000 osd.2 (osd.2) 133 : cluster [DBG] 2021-03-24T22:06:37.764 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[24927]: 2.1 scrub ok 2021-03-24T22:06:37.764 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:06:37.764 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[24927]: -03-24T22:06:37.445928+0000 mon.a (mon 2021-03-24T22:06:37.764 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[24927]: .0) 2057 : 2021-03-24T22:06:37.765 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[24927]: cluster [DBG] osdmap e827: 8 total, 8 up, 8 in 2021-03-24T22:06:37.765 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T22:06:37.765 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[19936]: -24T22:06:36.314946+0000 osd.5 (osd.5) 22 : cluster [DBG] 2021-03-24T22:06:37.766 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[19936]: 4.15 scrub ok 2021-03-24T22:06:37.766 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:06:37.766 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[19936]: 03-24T22:06:36.436890+0000 2021-03-24T22:06:37.766 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[19936]: osd.1 (osd.1) 95 : cluster 2021-03-24T22:06:37.767 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[19936]: [DBG] purged_snaps scrub starts 2021-03-24T22:06:37.767 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:06:37.767 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[19936]: 03-24T22:06:36.438575 2021-03-24T22:06:37.767 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[19936]: +0000 osd.1 (osd.1) 96 2021-03-24T22:06:37.767 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[19936]: : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:06:37.768 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:06:37.768 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[19936]: -03-24T22:06:36.792533 2021-03-24T22:06:37.768 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[19936]: +0000 osd.1 (osd.1) 97 2021-03-24T22:06:37.769 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[19936]: : cluster [DBG] 3.25 scrub ok 2021-03-24T22:06:37.769 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:06:37.769 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[19936]: 03-24T22:06:37.049836 2021-03-24T22:06:37.769 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[19936]: +0000 osd.7 (osd.7) 2021-03-24T22:06:37.770 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[19936]: 136 : cluster [DBG] 2021-03-24T22:06:37.770 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[19936]: 3.e scrub ok 2021-03-24T22:06:37.770 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[19936]: cluster 2021-03-24T22:06:37.770 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[19936]: 2021-03-24T22: 2021-03-24T22:06:37.771 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[19936]: 06:37.118203+0000 osd.2 2021-03-24T22:06:37.771 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[19936]: (osd.2) 133 : cluster 2021-03-24T22:06:37.771 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[19936]: [DBG] 2.1 scrub ok 2021-03-24T22:06:37.771 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[19936]: cluster 2021-03-24T22:06:37.771 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[19936]: 2021-03 2021-03-24T22:06:37.772 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[19936]: -24T22:06: 2021-03-24T22:06:37.772 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[19936]: 37.445928+0000 2021-03-24T22:06:37.772 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[19936]: mon.a (mon.0) 2021-03-24T22:06:37.772 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[19936]: 2057 : cluster [DBG] 2021-03-24T22:06:37.773 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:37 smithi125 conmon[19936]: osdmap e827: 8 total, 8 up, 8 in 2021-03-24T22:06:37.861 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:37.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:37 smithi142 conmon[20125]: cluster 2021-03-24T22:06:36.314946+0000 osd.5 (osd.5) 22 : cluster [DBG] 4.15 scrub ok 2021-03-24T22:06:37.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:37 smithi142 conmon[20125]: cluster 2021-03-24T22:06:36.436890+0000 osd.1 (osd.1) 95 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:06:37.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:37 smithi142 conmon[20125]: cluster 2021-03-24T22:06:36.438575+0000 osd.1 (osd.1) 96 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:06:37.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:37 smithi142 conmon[20125]: cluster 2021-03-24T22:06:36.792533+0000 osd.1 (osd.1) 97 : cluster [DBG] 3.25 scrub ok 2021-03-24T22:06:37.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:37 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:06:37.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:37 smithi142 conmon[20125]: -24T22:06:37.049836+0000 osd.7 (osd.7) 136 : cluster [DBG] 3.e scrub ok 2021-03-24T22:06:37.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:37 smithi142 conmon[20125]: cluster 2021-03-24T22:06:37.118203+0000 osd.2 (osd.2) 133 : cluster [DBG] 2.1 scrub ok 2021-03-24T22:06:37.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:37 smithi142 conmon[20125]: cluster 2021-03-24T22:06:37.445928+0000 mon.a (mon.0) 2057 : cluster [DBG] osdmap e827: 8 total, 8 up, 8 in 2021-03-24T22:06:37.919 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:37.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:37.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:06:38.026 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:38.082 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:38.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:38.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:06:38.190 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:38.206 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-24T22:06:38.252 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:38.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:38.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:06:38.359 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:38.414 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:38.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:38.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:06:38.452 INFO:tasks.rados.rados.0.smithi125.stdout:3200: done (0 left) 2021-03-24T22:06:38.452 INFO:tasks.rados.rados.0.smithi125.stdout:3201: rollback oid 13 current snap is 370 2021-03-24T22:06:38.453 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 13 to 331 2021-03-24T22:06:38.453 INFO:tasks.rados.rados.0.smithi125.stdout:3202: copy_from oid 40 from oid 12 current snap is 370 2021-03-24T22:06:38.453 INFO:tasks.rados.rados.0.smithi125.stdout:3203: snap_remove snap 345 2021-03-24T22:06:38.519 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:38.570 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:38.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:38.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:06:38.675 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:38.726 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:38.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:38.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:06:38.832 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:38.846 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:38 smithi142 conmon[20125]: cluster 2021-03-24T22:06:37.008094+0000 mgr.y (mgr.14144) 689 : cluster 2021-03-24T22:06:38.846 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:38 smithi142 conmon[20125]: [DBG] pgmap v1402: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 520 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 11/1656 objects misplaced (0.664%) 2021-03-24T22:06:38.847 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:38 smithi142 conmon[20125]: cluster 2021-03-24T22:06:38.254751+0000 osd.6 (osd.6) 85 : cluster [DBG] 4.e scrub ok 2021-03-24T22:06:38.847 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:38 smithi142 conmon[20125]: cluster 2021-03-24T22:06:38.447464+0000 mon.a (mon.0) 2058 : cluster [DBG] osdmap e828: 8 total, 8 up, 8 in 2021-03-24T22:06:38.889 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:38.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:38.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:06:38.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:38 smithi125 conmon[19936]: cluster 2021-03-24T22:06:37.008094+0000 mgr.y 2021-03-24T22:06:38.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:38 smithi125 conmon[19936]: (mgr.14144) 689 : cluster [DBG] pgmap v1402: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 520 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 11/1656 objects misplaced (0.664%) 2021-03-24T22:06:38.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:38 smithi125 conmon[19936]: cluster 2021-03-24T22 2021-03-24T22:06:38.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:38 smithi125 conmon[19936]: :06:38.254751+0000 osd.6 (osd.6) 85 : cluster [DBG] 4.e scrub ok 2021-03-24T22:06:38.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:38 smithi125 conmon[19936]: 2021-03-24T22:06:38.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:38 smithi125 conmon[19936]: cluster 2021-03-24T22:06 2021-03-24T22:06:38.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:38 smithi125 conmon[19936]: :38.447464+0000 mon.a (mon.0) 2058 : cluster 2021-03-24T22:06:38.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:38 smithi125 conmon[19936]: [DBG] osdmap e828: 8 total, 8 up, 8 in 2021-03-24T22:06:38.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:38 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:06:38.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:38 smithi125 conmon[24927]: 24T22:06:37.008094+0000 mgr.y 2021-03-24T22:06:38.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:38 smithi125 conmon[24927]: (mgr.14144) 2021-03-24T22:06:38.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:38 smithi125 conmon[24927]: 689 : cluster [DBG] 2021-03-24T22:06:38.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:38 smithi125 conmon[24927]: pgmap v1402: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 520 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 11/1656 objects misplaced (0.664%) 2021-03-24T22:06:38.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:38 smithi125 conmon[24927]: cluster 2021-03-24T22:06:38.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:38 smithi125 conmon[24927]: 2021-03-24T22 2021-03-24T22:06:38.918 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:38 smithi125 conmon[24927]: :06:38.254751 2021-03-24T22:06:38.918 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:38 smithi125 conmon[24927]: +0000 osd.6 (osd. 2021-03-24T22:06:38.919 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:38 smithi125 conmon[24927]: 6) 85 : cluster 2021-03-24T22:06:38.919 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:38 smithi125 conmon[24927]: [DBG] 4.e scrub ok 2021-03-24T22:06:38.919 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:38 smithi125 conmon[24927]: cluster 2021-03-24T22:06:38.919 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:38 smithi125 conmon[24927]: 2021-03-24T22 2021-03-24T22:06:38.919 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:38 smithi125 conmon[24927]: :06:38.447464 2021-03-24T22:06:38.920 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:38 smithi125 conmon[24927]: +0000 mon.a (mon. 2021-03-24T22:06:38.920 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:38 smithi125 conmon[24927]: 0) 2058 : cluster 2021-03-24T22:06:38.920 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:38 smithi125 conmon[24927]: [DBG] osdmap e828: 8 total, 8 up, 8 in 2021-03-24T22:06:38.996 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:39.057 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:39.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:39.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:06:39.161 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:39.223 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:39.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:39.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:06:39.330 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:39.386 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:39.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:39.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:06:39.459 INFO:tasks.rados.rados.0.smithi125.stdout:3203: done (2 left) 2021-03-24T22:06:39.459 INFO:tasks.rados.rados.0.smithi125.stdout:3204: snap_remove snap 353 2021-03-24T22:06:39.490 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:39.539 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:39.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:39.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:06:39.646 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:39.652 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:39 smithi125 conmon[19936]: cluster 2021-03-24T22:06:38.590650+0000 osd.4 (osd 2021-03-24T22:06:39.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:39 smithi125 conmon[19936]: .4) 123 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:06:39.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:39 smithi125 conmon[19936]: cluster 2021-03-24T22:06:38.592782+0000 2021-03-24T22:06:39.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:39 smithi125 conmon[19936]: osd.4 (osd.4) 124 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:06:39.653 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:39 smithi125 conmon[19936]: cluster 2021-03-24T22:06:39.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:39 smithi125 conmon[19936]: 2021-03-24T22:06:38.755178+0000 osd.1 (osd.1) 2021-03-24T22:06:39.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:39 smithi125 conmon[19936]: 98 : cluster [DBG] 5.1d scrub ok 2021-03-24T22:06:39.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:39 smithi125 conmon[19936]: cluster 2021-03-24T22:06:39. 2021-03-24T22:06:39.654 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:39 smithi125 conmon[19936]: 453203+0000 mon.a (mon.0) 2059 : cluster [DBG] 2021-03-24T22:06:39.655 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:39 smithi125 conmon[19936]: osdmap e829: 8 total, 8 up, 8 in 2021-03-24T22:06:39.655 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:39 smithi125 conmon[24927]: cluster 2021-03-24T22:06:38.590650 2021-03-24T22:06:39.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:39 smithi125 conmon[24927]: +0000 osd.4 (osd.4) 123 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:06:39.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:39 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:06:39.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:39 smithi125 conmon[24927]: 24T22:06:38.592782+0000 osd.4 (osd.4) 124 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:06:39.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:39 smithi125 conmon[24927]: cluster 2021-03-24T22:06:39.656 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:39 smithi125 conmon[24927]: 2021-03-24T22:06:38.755178+0000 osd.1 ( 2021-03-24T22:06:39.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:39 smithi125 conmon[24927]: osd.1) 98 : cluster [DBG] 5.1d scrub ok 2021-03-24T22:06:39.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:39 smithi125 conmon[24927]: cluster 2021-03-24T22:06:39.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:39 smithi125 conmon[24927]: 2021-03-24T22:06:39.453203+0000 mon.a 2021-03-24T22:06:39.657 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:39 smithi125 conmon[24927]: (mon.0) 2059 : cluster [DBG] osdmap e829: 8 total, 8 up, 8 in 2021-03-24T22:06:39.696 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:39.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:39.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:06:39.801 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:39.851 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:39.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:39.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:06:39.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:39 smithi142 conmon[20125]: cluster 2021-03-24T22:06: 2021-03-24T22:06:39.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:39 smithi142 conmon[20125]: 38.590650+0000 osd.4 (osd.4) 123 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:06:39.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:39 smithi142 conmon[20125]: cluster 2021-03-24T22:06:38.592782 2021-03-24T22:06:39.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:39 smithi142 conmon[20125]: +0000 osd.4 (osd.4) 124 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:06:39.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:39 smithi142 conmon[20125]: cluster 2021-03-24T22:06:38. 2021-03-24T22:06:39.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:39 smithi142 conmon[20125]: 755178+0000 osd.1 (osd.1) 98 2021-03-24T22:06:39.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:39 smithi142 conmon[20125]: : cluster [DBG] 5.1d scrub ok 2021-03-24T22:06:39.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:39 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:06:39.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:39 smithi142 conmon[20125]: -24T22:06:39.453203+0000 mon.a (mon 2021-03-24T22:06:39.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:39 smithi142 conmon[20125]: .0) 2059 : cluster [DBG] osdmap e829: 8 total, 8 up, 8 in 2021-03-24T22:06:39.957 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:40.019 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:40.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:40.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:06:40.124 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:40.181 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:40.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:40.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:40.287 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:40.344 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:40.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:40.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:06:40.451 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:40.453 INFO:tasks.rados.rados.0.smithi125.stdout:3204: done (2 left) 2021-03-24T22:06:40.454 INFO:tasks.rados.rados.0.smithi125.stdout:3205: snap_remove snap 314 2021-03-24T22:06:40.498 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:40.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:40.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:06:40.604 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:40.701 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:40 smithi125 conmon[24927]: cluster 2021-03-24T22:06:39 2021-03-24T22:06:40.701 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:40 smithi125 conmon[24927]: .009025+0000 mgr.y (mgr.14144) 690 : cluster [DBG] pgmap v1405: 168 pgs: 1 active+clean+remapped, 167 active+clean; 522 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2021-03-24T22:06:40.701 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:40 smithi125 conmon[24927]: cluster 2021-03-24T22:06:39.560682+0000 mon.a (mon. 2021-03-24T22:06:40.702 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:40 smithi125 conmon[24927]: 0) 2060 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2021-03-24T22:06:40.702 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:40 smithi125 conmon[24927]: cluster 2021-03-24T22:06:39.566790+0000 osd.0 (osd 2021-03-24T22:06:40.702 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:40 smithi125 conmon[24927]: .0) 161 : cluster [DBG] 7.16 scrub ok 2021-03-24T22:06:40.702 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:40 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T22:06:40.702 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:40 smithi125 conmon[24927]: T22:06:40.439564+0000 mon.a (mon.0) 2061 : cluster [DBG] osdmap e830: 8 total, 8 up, 8 in 2021-03-24T22:06:40.703 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:40 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:06:40.703 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:40 smithi125 conmon[19936]: 06:39.009025+0000 mgr.y (mgr.14144) 690 : cluster [DBG] pgmap v1405: 168 pgs: 1 active+clean+remapped, 167 active+clean; 522 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2021-03-24T22:06:40.703 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:40 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:06:40.704 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:40 smithi125 conmon[19936]: T22:06:39 2021-03-24T22:06:40.704 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:40 smithi125 conmon[19936]: .560682+0000 mon.a 2021-03-24T22:06:40.704 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:40 smithi125 conmon[19936]: (mon.0) 2060 2021-03-24T22:06:40.704 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:40 smithi125 conmon[19936]: : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2021-03-24T22:06:40.705 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:40 smithi125 conmon[19936]: cluster 2021-03-24T22:06:40.705 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:40 smithi125 conmon[19936]: 2021-03-24T22 2021-03-24T22:06:40.705 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:40 smithi125 conmon[19936]: :06:39.566790 2021-03-24T22:06:40.705 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:40 smithi125 conmon[19936]: +0000 osd.0 (osd.0 2021-03-24T22:06:40.705 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:40 smithi125 conmon[19936]: ) 161 : cluster [DBG] 2021-03-24T22:06:40.706 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:40 smithi125 conmon[19936]: 7.16 scrub ok 2021-03-24T22:06:40.706 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:40 smithi125 conmon[19936]: cluster 2021-03-24T22:06:40.706 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:40 smithi125 conmon[19936]: 2021-03 2021-03-24T22:06:40.706 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:40 smithi125 conmon[19936]: -24T22:06: 2021-03-24T22:06:40.707 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:40 smithi125 conmon[19936]: 40.439564+0000 mon.a (mon.0) 2021-03-24T22:06:40.707 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:40 smithi125 conmon[19936]: 2061 : cluster [DBG] osdmap e830: 8 total, 8 up, 8 in 2021-03-24T22:06:40.707 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:40 smithi125 conmon[19936]: 2021-03-24T22:06:40.720 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:06:40.762 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:40.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:40.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:06:40.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:40 smithi142 conmon[20125]: cluster 2021-03-24T22:06:39.009025+0000 mgr.y (mgr.14144) 690 : cluster [DBG] 2021-03-24T22:06:40.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:40 smithi142 conmon[20125]: pgmap v1405: 168 pgs: 1 active+clean+remapped, 167 active+clean; 522 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2021-03-24T22:06:40.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:40 smithi142 conmon[20125]: cluster 2021-03-24T22:06:39.560682+0000 mon.a (mon.0) 2060 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2021-03-24T22:06:40.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:40 smithi142 conmon[20125]: cluster 2021-03-24T22: 2021-03-24T22:06:40.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:40 smithi142 conmon[20125]: 06:39.566790+0000 osd.0 ( 2021-03-24T22:06:40.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:40 smithi142 conmon[20125]: osd.0) 161 : cluster [DBG] 2021-03-24T22:06:40.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:40 smithi142 conmon[20125]: 7.16 scrub ok 2021-03-24T22:06:40.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:40 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:06:40.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:40 smithi142 conmon[20125]: 22:06:40.439564+0000 mon.a (mon. 2021-03-24T22:06:40.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:40 smithi142 conmon[20125]: 0) 2061 : cluster [DBG] osdmap e830: 8 total, 8 up, 8 in 2021-03-24T22:06:40.869 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:40.921 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:40.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:40.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:06:41.027 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:41.086 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:41.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:41.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:41.192 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:41.202 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T22:06:41.224 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T22:06:41.237 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:41.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:41.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:06:41.244 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T22:06:41.268 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T22:06:41.290 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T22:06:41.320 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T22:06:41.340 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T22:06:41.342 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:41.386 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T22:06:41.386 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-24T22:06:41.386 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-24T22:06:41.387 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-24T22:06:41.387 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-24T22:06:41.387 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-24T22:06:41.387 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-24T22:06:41.388 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-24T22:06:41.388 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-24T22:06:41.393 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:41.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:41.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:06:41.462 INFO:tasks.rados.rados.0.smithi125.stdout:3205: done (2 left) 2021-03-24T22:06:41.463 INFO:tasks.rados.rados.0.smithi125.stdout:3206: write oid 50 current snap is 370 2021-03-24T22:06:41.463 INFO:tasks.rados.rados.0.smithi125.stdout:3206: seq_num 1092 ranges {753244=410387,1834572=518672} 2021-03-24T22:06:41.468 INFO:tasks.rados.rados.0.smithi125.stdout:3206: writing smithi12556470-50 from 753244 to 1163631 tid 1 2021-03-24T22:06:41.481 INFO:tasks.rados.rados.0.smithi125.stdout:3206: writing smithi12556470-50 from 1834572 to 2353244 tid 2 2021-03-24T22:06:41.482 INFO:tasks.rados.rados.0.smithi125.stdout:3201: finishing rollback tid 0 to smithi12556470-13 2021-03-24T22:06:41.483 INFO:tasks.rados.rados.0.smithi125.stdout:3201: finishing rollback tid 1 to smithi12556470-13 2021-03-24T22:06:41.483 INFO:tasks.rados.rados.0.smithi125.stdout:3201: finishing rollback tid 2 to smithi12556470-13 2021-03-24T22:06:41.483 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 13 v 1463 (ObjNum 988 snap 328 seq_num 988) dirty exists 2021-03-24T22:06:41.483 INFO:tasks.rados.rados.0.smithi125.stdout:3202: finishing copy_from to smithi12556470-40 2021-03-24T22:06:41.484 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 40 v 697 (ObjNum 1001 snap 331 seq_num 1001) dirty exists 2021-03-24T22:06:41.484 INFO:tasks.rados.rados.0.smithi125.stdout:3202: finishing copy_from racing read to smithi12556470-40 2021-03-24T22:06:41.484 INFO:tasks.rados.rados.0.smithi125.stdout:3201: done (2 left) 2021-03-24T22:06:41.484 INFO:tasks.rados.rados.0.smithi125.stdout:3202: done (1 left) 2021-03-24T22:06:41.484 INFO:tasks.rados.rados.0.smithi125.stdout:3207: write (excl) oid 27 current snap is 370 2021-03-24T22:06:41.485 INFO:tasks.rados.rados.0.smithi125.stdout:3207: seq_num 1093 ranges {770051=485636,1776701=511047} 2021-03-24T22:06:41.490 INFO:tasks.rados.rados.0.smithi125.stdout:3207: writing smithi12556470-27 from 770051 to 1255687 tid 1 2021-03-24T22:06:41.497 INFO:tasks.rados.rados.0.smithi125.stdout:3207: writing smithi12556470-27 from 1776701 to 2287748 tid 2 2021-03-24T22:06:41.497 INFO:tasks.rados.rados.0.smithi125.stdout:3208: read oid 14 snap -1 2021-03-24T22:06:41.498 INFO:tasks.rados.rados.0.smithi125.stdout:3208: expect (ObjNum 1067 snap 357 seq_num 1067) 2021-03-24T22:06:41.498 INFO:tasks.rados.rados.0.smithi125.stdout:3209: write oid 9 current snap is 370 2021-03-24T22:06:41.498 INFO:tasks.rados.rados.0.smithi125.stdout:3206: finishing write tid 1 to smithi12556470-50 2021-03-24T22:06:41.498 INFO:tasks.rados.rados.0.smithi125.stdout:3206: finishing write tid 2 to smithi12556470-50 2021-03-24T22:06:41.498 INFO:tasks.rados.rados.0.smithi125.stdout:3206: finishing write tid 3 to smithi12556470-50 2021-03-24T22:06:41.499 INFO:tasks.rados.rados.0.smithi125.stdout:3206: finishing write tid 4 to smithi12556470-50 2021-03-24T22:06:41.499 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 50 v 1455 (ObjNum 1092 snap 370 seq_num 1092) dirty exists 2021-03-24T22:06:41.499 INFO:tasks.rados.rados.0.smithi125.stdout:3206: left oid 50 (ObjNum 1092 snap 370 seq_num 1092) 2021-03-24T22:06:41.500 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:41.501 INFO:tasks.rados.rados.0.smithi125.stdout:3209: seq_num 1094 ranges {786858=560885,1835183=551675} 2021-03-24T22:06:41.514 INFO:tasks.rados.rados.0.smithi125.stdout:3209: writing smithi12556470-9 from 786858 to 1347743 tid 1 2021-03-24T22:06:41.521 INFO:tasks.rados.rados.0.smithi125.stdout:3209: writing smithi12556470-9 from 1835183 to 2386858 tid 2 2021-03-24T22:06:41.521 INFO:tasks.rados.rados.0.smithi125.stdout:3206: done (3 left) 2021-03-24T22:06:41.522 INFO:tasks.rados.rados.0.smithi125.stdout:3210: write (excl) oid 23 current snap is 370 2021-03-24T22:06:41.522 INFO:tasks.rados.rados.0.smithi125.stdout:3207: finishing write tid 1 to smithi12556470-27 2021-03-24T22:06:41.522 INFO:tasks.rados.rados.0.smithi125.stdout:3210: seq_num 1095 ranges {403665=752487,1610018=514716} 2021-03-24T22:06:41.527 INFO:tasks.rados.rados.0.smithi125.stdout:3210: writing smithi12556470-23 from 403665 to 1156152 tid 1 2021-03-24T22:06:41.533 INFO:tasks.rados.rados.0.smithi125.stdout:3210: writing smithi12556470-23 from 1610018 to 2124734 tid 2 2021-03-24T22:06:41.533 INFO:tasks.rados.rados.0.smithi125.stdout:3211: snap_create 2021-03-24T22:06:41.534 INFO:tasks.rados.rados.0.smithi125.stdout:3207: finishing write tid 2 to smithi12556470-27 2021-03-24T22:06:41.534 INFO:tasks.rados.rados.0.smithi125.stdout:3207: finishing write tid 3 to smithi12556470-27 2021-03-24T22:06:41.534 INFO:tasks.rados.rados.0.smithi125.stdout:3207: finishing write tid 4 to smithi12556470-27 2021-03-24T22:06:41.534 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 27 v 1438 (ObjNum 1093 snap 370 seq_num 1093) dirty exists 2021-03-24T22:06:41.535 INFO:tasks.rados.rados.0.smithi125.stdout:3207: left oid 27 (ObjNum 1093 snap 370 seq_num 1093) 2021-03-24T22:06:41.535 INFO:tasks.rados.rados.0.smithi125.stdout:3208: expect (ObjNum 1067 snap 357 seq_num 1067) 2021-03-24T22:06:41.550 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:41.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:41.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:41.656 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:41.704 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:41.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:41.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:06:41.797 INFO:tasks.rados.rados.0.smithi125.stdout:3209: finishing write tid 1 to smithi12556470-9 2021-03-24T22:06:41.797 INFO:tasks.rados.rados.0.smithi125.stdout:3209: finishing write tid 2 to smithi12556470-9 2021-03-24T22:06:41.798 INFO:tasks.rados.rados.0.smithi125.stdout:3209: finishing write tid 3 to smithi12556470-9 2021-03-24T22:06:41.798 INFO:tasks.rados.rados.0.smithi125.stdout:3209: finishing write tid 4 to smithi12556470-9 2021-03-24T22:06:41.799 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 9 v 2049 (ObjNum 1094 snap 370 seq_num 1094) dirty exists 2021-03-24T22:06:41.799 INFO:tasks.rados.rados.0.smithi125.stdout:3209: left oid 9 (ObjNum 1094 snap 370 seq_num 1094) 2021-03-24T22:06:41.799 INFO:tasks.rados.rados.0.smithi125.stdout:3210: finishing write tid 1 to smithi12556470-23 2021-03-24T22:06:41.800 INFO:tasks.rados.rados.0.smithi125.stdout:3210: finishing write tid 2 to smithi12556470-23 2021-03-24T22:06:41.800 INFO:tasks.rados.rados.0.smithi125.stdout:3210: finishing write tid 3 to smithi12556470-23 2021-03-24T22:06:41.801 INFO:tasks.rados.rados.0.smithi125.stdout:3210: finishing write tid 4 to smithi12556470-23 2021-03-24T22:06:41.802 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 23 v 554 (ObjNum 1095 snap 370 seq_num 1095) dirty exists 2021-03-24T22:06:41.802 INFO:tasks.rados.rados.0.smithi125.stdout:3210: left oid 23 (ObjNum 1095 snap 370 seq_num 1095) 2021-03-24T22:06:41.803 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:41 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T22:06:41.803 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:41 smithi125 conmon[24927]: -24T22:06:40.570948+0000 osd.0 (osd. 2021-03-24T22:06:41.803 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:41 smithi125 conmon[24927]: 0) 162 : cluster [DBG] 2021-03-24T22:06:41.803 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:41 smithi125 conmon[24927]: 4.6 scrub ok 2021-03-24T22:06:41.804 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:41 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:06:41.804 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:41 smithi125 conmon[24927]: -03-24T22:06:41.076871 2021-03-24T22:06:41.804 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:41 smithi125 conmon[24927]: +0000 osd.3 (osd.3) 162 : 2021-03-24T22:06:41.804 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:41 smithi125 conmon[24927]: cluster [DBG] 3.c scrub ok 2021-03-24T22:06:41.805 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:41 smithi125 conmon[24927]: audit 2021-03-24T22:06:41.805 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:41 smithi125 conmon[24927]: 2021-03 2021-03-24T22:06:41.805 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:41 smithi125 conmon[24927]: -24T22:06:41. 2021-03-24T22:06:41.805 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:41 smithi125 conmon[24927]: 176499+0000 mon.a (mon. 2021-03-24T22:06:41.806 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:41 smithi125 conmon[24927]: 0) 2062 : audit [DBG] 2021-03-24T22:06:41.807 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:41 smithi125 conmon[24927]: from='client.? v1:172.21.15.125:0/2054610750' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:06:41.807 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:41 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:06:41.807 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:41 smithi125 conmon[24927]: -03-24T22:06:41 2021-03-24T22:06:41.807 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:41 smithi125 conmon[24927]: .250459+0000 osd.6 (osd 2021-03-24T22:06:41.807 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:41 smithi125 conmon[24927]: .6) 86 : cluster [DBG] 2021-03-24T22:06:41.808 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:41 smithi125 conmon[24927]: 7.c scrub ok 2021-03-24T22:06:41.808 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:41 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:06:41.808 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:41 smithi125 conmon[24927]: 03-24T22:06:41. 2021-03-24T22:06:41.808 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:41 smithi125 conmon[24927]: 456685+0000 mon.a (mon. 2021-03-24T22:06:41.809 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:41 smithi125 conmon[24927]: 0) 2063 : cluster [DBG] 2021-03-24T22:06:41.809 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:41 smithi125 conmon[24927]: osdmap e831: 8 total, 8 up, 8 in 2021-03-24T22:06:41.810 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:41 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T22:06:41.810 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:41 smithi125 conmon[19936]: -24T22:06:40.570948+0000 osd.0 (osd.0) 162 : cluster 2021-03-24T22:06:41.811 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:41 smithi125 conmon[19936]: [DBG] 4.6 scrub ok 2021-03-24T22:06:41.811 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:41 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:06:41.811 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:41 smithi125 conmon[19936]: -03-24T22:06 2021-03-24T22:06:41.811 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:41 smithi125 conmon[19936]: :41.076871+0000 osd.3 2021-03-24T22:06:41.812 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:41 smithi125 conmon[19936]: (osd.3) 162 : cluster 2021-03-24T22:06:41.812 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:41 smithi125 conmon[19936]: [DBG] 3.c scrub ok 2021-03-24T22:06:41.812 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:41 smithi125 conmon[19936]: audit 2021- 2021-03-24T22:06:41.813 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:41 smithi125 conmon[19936]: 03-24T22:06:41.176499+0000 2021-03-24T22:06:41.813 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:41 smithi125 conmon[19936]: mon.a (mon.0) 2062 : audit [DBG] from='client.? v1:172.21.15.125:0/2054610750' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:06:41.813 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:41 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:06:41.813 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:41 smithi125 conmon[19936]: -03-24T22:06: 2021-03-24T22:06:41.814 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:41 smithi125 conmon[19936]: 41.250459+0000 osd.6 (osd. 2021-03-24T22:06:41.814 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:41 smithi125 conmon[19936]: 6) 86 : cluster [DBG] 7.c scrub ok 2021-03-24T22:06:41.814 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:41 smithi125 conmon[19936]: 2021-03-24T22:06:41.815 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:41 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:06:41.815 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:41 smithi125 conmon[19936]: 03-24T22:06: 2021-03-24T22:06:41.815 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:41 smithi125 conmon[19936]: 41.456685+0000 mon.a (mon.0 2021-03-24T22:06:41.816 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:41 smithi125 conmon[19936]: ) 2063 : cluster [DBG] osdmap e831: 8 total, 8 up, 8 in 2021-03-24T22:06:41.816 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:41.830 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:41 smithi142 conmon[20125]: cluster 2021-03-24T22:06:40.570948+0000 osd.0 (osd.0) 162 : 2021-03-24T22:06:41.831 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:41 smithi142 conmon[20125]: cluster [DBG] 4.6 scrub ok 2021-03-24T22:06:41.831 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:41 smithi142 conmon[20125]: cluster 2021-03-24T22:06:41.076871+0000 osd.3 (osd 2021-03-24T22:06:41.832 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:41 smithi142 conmon[20125]: .3) 162 : cluster [DBG] 3.c scrub ok 2021-03-24T22:06:41.832 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:41 smithi142 conmon[20125]: audit 2021-03-24T22:06:41.832 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:41 smithi142 conmon[20125]: 2021-03-24T22:06:41 2021-03-24T22:06:41.833 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:41 smithi142 conmon[20125]: .176499+0000 mon.a (mon.0) 2062 : 2021-03-24T22:06:41.833 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:41 smithi142 conmon[20125]: audit [DBG] from='client.? v1:172.21.15.125:0/2054610750' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:06:41.834 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:41 smithi142 conmon[20125]: cluster 2021-03-24T22 2021-03-24T22:06:41.834 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:41 smithi142 conmon[20125]: :06:41.250459+0000 osd.6 (osd. 2021-03-24T22:06:41.834 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:41 smithi142 conmon[20125]: 6) 86 : cluster [DBG] 7.c scrub ok 2021-03-24T22:06:41.835 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:41 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:06:41.835 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:41 smithi142 conmon[20125]: 03-24T22:06:41.456685+0000 2021-03-24T22:06:41.835 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:41 smithi142 conmon[20125]: mon.a (mon.0) 2063 : cluster 2021-03-24T22:06:41.836 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:41 smithi142 conmon[20125]: [DBG] osdmap e831: 8 total, 8 up, 8 in 2021-03-24T22:06:41.874 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:41.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:41.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:06:41.940 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-24T22:06:41.940 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T22:06:41.979 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:42.036 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:42.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:42.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:06:42.144 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:42.208 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:42.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:42.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:06:42.313 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:42.372 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:42.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:42.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:42.468 INFO:tasks.rados.rados.0.smithi125.stdout:3207: done (4 left) 2021-03-24T22:06:42.468 INFO:tasks.rados.rados.0.smithi125.stdout:3208: done (3 left) 2021-03-24T22:06:42.469 INFO:tasks.rados.rados.0.smithi125.stdout:3209: done (2 left) 2021-03-24T22:06:42.469 INFO:tasks.rados.rados.0.smithi125.stdout:3210: done (1 left) 2021-03-24T22:06:42.469 INFO:tasks.rados.rados.0.smithi125.stdout:3211: done (0 left) 2021-03-24T22:06:42.469 INFO:tasks.rados.rados.0.smithi125.stdout:3212: read oid 1 snap -1 2021-03-24T22:06:42.470 INFO:tasks.rados.rados.0.smithi125.stdout:3212: expect (ObjNum 808 snap 265 seq_num 808) 2021-03-24T22:06:42.470 INFO:tasks.rados.rados.0.smithi125.stdout:3213: read oid 40 snap -1 2021-03-24T22:06:42.470 INFO:tasks.rados.rados.0.smithi125.stdout:3213: expect (ObjNum 1001 snap 331 seq_num 1001) 2021-03-24T22:06:42.470 INFO:tasks.rados.rados.0.smithi125.stdout:3214: copy_from oid 18 from oid 47 current snap is 371 2021-03-24T22:06:42.470 INFO:tasks.rados.rados.0.smithi125.stdout:3215: read oid 27 snap -1 2021-03-24T22:06:42.471 INFO:tasks.rados.rados.0.smithi125.stdout:3215: expect (ObjNum 1093 snap 370 seq_num 1093) 2021-03-24T22:06:42.471 INFO:tasks.rados.rados.0.smithi125.stdout:3216: copy_from oid 3 from oid 25 current snap is 371 2021-03-24T22:06:42.471 INFO:tasks.rados.rados.0.smithi125.stdout:3217: write (excl) oid 5 current snap is 371 2021-03-24T22:06:42.471 INFO:tasks.rados.rados.0.smithi125.stdout:3217: seq_num 1096 ranges {420472=427736,1268500=658374,2347953=72519} 2021-03-24T22:06:42.474 INFO:tasks.rados.rados.0.smithi125.stdout:3217: writing smithi12556470-5 from 420472 to 848208 tid 1 2021-03-24T22:06:42.478 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:42.481 INFO:tasks.rados.rados.0.smithi125.stdout:3217: writing smithi12556470-5 from 1268500 to 1926874 tid 2 2021-03-24T22:06:42.484 INFO:tasks.rados.rados.0.smithi125.stdout:3217: writing smithi12556470-5 from 2347953 to 2420472 tid 3 2021-03-24T22:06:42.484 INFO:tasks.rados.rados.0.smithi125.stdout:3218: write (excl) oid 15 current snap is 371 2021-03-24T22:06:42.484 INFO:tasks.rados.rados.0.smithi125.stdout:3218: seq_num 1097 ranges {437279=502985,1610629=518385} 2021-03-24T22:06:42.491 INFO:tasks.rados.rados.0.smithi125.stdout:3218: writing smithi12556470-15 from 437279 to 940264 tid 1 2021-03-24T22:06:42.497 INFO:tasks.rados.rados.0.smithi125.stdout:3218: writing smithi12556470-15 from 1610629 to 2129014 tid 2 2021-03-24T22:06:42.497 INFO:tasks.rados.rados.0.smithi125.stdout:3219: copy_from oid 14 from oid 2 current snap is 371 2021-03-24T22:06:42.498 INFO:tasks.rados.rados.0.smithi125.stdout:3215: expect (ObjNum 1093 snap 370 seq_num 1093) 2021-03-24T22:06:42.522 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:42.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:42.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:42.583 INFO:tasks.rados.rados.0.smithi125.stdout:3216: finishing copy_from to smithi12556470-3 2021-03-24T22:06:42.584 INFO:tasks.rados.rados.0.smithi125.stdout:3216: got expected ENOENT (src dne) 2021-03-24T22:06:42.585 INFO:tasks.rados.rados.0.smithi125.stdout:3216: finishing copy_from racing read to smithi12556470-3 2021-03-24T22:06:42.585 INFO:tasks.rados.rados.0.smithi125.stdout:3213: expect (ObjNum 1001 snap 331 seq_num 1001) 2021-03-24T22:06:42.628 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:42.678 INFO:tasks.rados.rados.0.smithi125.stdout:3212: expect (ObjNum 808 snap 265 seq_num 808) 2021-03-24T22:06:42.679 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:42.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:42.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:06:42.785 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:42.799 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:42 smithi142 conmon[20125]: cluster 2021-03-24T22:06:41.009948+0000 mgr.y (mgr.14144) 691 2021-03-24T22:06:42.799 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:42 smithi142 conmon[20125]: : cluster [DBG] pgmap v1408: 168 pgs: 1 active+clean+remapped, 167 active+clean; 525 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 8.9 MiB/s rd, 689 KiB/s wr, 7 op/s; 200 B/s, 2 objects/s recovering 2021-03-24T22:06:42.799 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:42 smithi142 conmon[20125]: cluster 2021-03-24T22:06:41.330499+0000 osd.5 2021-03-24T22:06:42.799 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:42 smithi142 conmon[20125]: (osd.5) 23 : cluster [DBG] 5.7 scrub ok 2021-03-24T22:06:42.800 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:42 smithi142 conmon[20125]: cluster 2021-03-24T22: 2021-03-24T22:06:42.800 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:42 smithi142 conmon[20125]: 06:41.587458+0000 osd.0 (osd.0 2021-03-24T22:06:42.800 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:42 smithi142 conmon[20125]: ) 163 : cluster [DBG] 7.4 scrub ok 2021-03-24T22:06:42.800 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:42 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:06:42.801 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:42 smithi142 conmon[20125]: T22:06:41.617247+0000 osd.7 ( 2021-03-24T22:06:42.801 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:42 smithi142 conmon[20125]: osd.7) 137 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:06:42.801 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:42 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:06:42.801 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:42 smithi142 conmon[20125]: 03-24T22:06:41.621780+0000 2021-03-24T22:06:42.801 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:42 smithi142 conmon[20125]: osd.7 (osd.7) 138 : cluster [DBG] 2021-03-24T22:06:42.802 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:42 smithi142 conmon[20125]: purged_snaps scrub ok 2021-03-24T22:06:42.802 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:42 smithi142 conmon[20125]: cluster 2021-03-24T22:06:42.802 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:42 smithi142 conmon[20125]: 2021-03-24T22:06:41. 2021-03-24T22:06:42.802 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:42 smithi142 conmon[20125]: 782234+0000 osd.1 (osd.1) 99 : cluster 2021-03-24T22:06:42.803 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:42 smithi142 conmon[20125]: [DBG] 4.12 scrub ok 2021-03-24T22:06:42.803 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:42 smithi142 conmon[20125]: cluster 2021-03-24T22:06:42.803 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:42 smithi142 conmon[20125]: 2021-03-24T22:06:42 2021-03-24T22:06:42.803 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:42 smithi142 conmon[20125]: .462958+0000 mon.a (mon.0) 2064 : 2021-03-24T22:06:42.803 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:42 smithi142 conmon[20125]: cluster [DBG] osdmap e832: 8 total, 8 up, 8 in 2021-03-24T22:06:42.832 INFO:tasks.rados.rados.0.smithi125.stdout:3217: finishing write tid 1 to smithi12556470-5 2021-03-24T22:06:42.832 INFO:tasks.rados.rados.0.smithi125.stdout:3217: finishing write tid 2 to smithi12556470-5 2021-03-24T22:06:42.832 INFO:tasks.rados.rados.0.smithi125.stdout:3214: finishing copy_from to smithi12556470-18 2021-03-24T22:06:42.833 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 18 v 1438 (ObjNum 1051 snap 349 seq_num 1051) dirty exists 2021-03-24T22:06:42.833 INFO:tasks.rados.rados.0.smithi125.stdout:3217: finishing write tid 3 to smithi12556470-5 2021-03-24T22:06:42.833 INFO:tasks.rados.rados.0.smithi125.stdout:3217: finishing write tid 4 to smithi12556470-5 2021-03-24T22:06:42.833 INFO:tasks.rados.rados.0.smithi125.stdout:3214: finishing copy_from racing read to smithi12556470-18 2021-03-24T22:06:42.833 INFO:tasks.rados.rados.0.smithi125.stdout:3218: finishing write tid 1 to smithi12556470-15 2021-03-24T22:06:42.834 INFO:tasks.rados.rados.0.smithi125.stdout:3217: finishing write tid 5 to smithi12556470-5 2021-03-24T22:06:42.834 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 5 v 1287 (ObjNum 1096 snap 371 seq_num 1096) dirty exists 2021-03-24T22:06:42.834 INFO:tasks.rados.rados.0.smithi125.stdout:3217: left oid 5 (ObjNum 1096 snap 371 seq_num 1096) 2021-03-24T22:06:42.834 INFO:tasks.rados.rados.0.smithi125.stdout:3218: finishing write tid 2 to smithi12556470-15 2021-03-24T22:06:42.835 INFO:tasks.rados.rados.0.smithi125.stdout:3218: finishing write tid 3 to smithi12556470-15 2021-03-24T22:06:42.835 INFO:tasks.rados.rados.0.smithi125.stdout:3218: finishing write tid 4 to smithi12556470-15 2021-03-24T22:06:42.835 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 15 v 1444 (ObjNum 1097 snap 371 seq_num 1097) dirty exists 2021-03-24T22:06:42.835 INFO:tasks.rados.rados.0.smithi125.stdout:3218: left oid 15 (ObjNum 1097 snap 371 seq_num 1097) 2021-03-24T22:06:42.835 INFO:tasks.rados.rados.0.smithi125.stdout:3219: finishing copy_from to smithi12556470-14 2021-03-24T22:06:42.836 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 14 v 1017 (ObjNum 984 snap 326 seq_num 984) dirty exists 2021-03-24T22:06:42.836 INFO:tasks.rados.rados.0.smithi125.stdout:3219: finishing copy_from racing read to smithi12556470-14 2021-03-24T22:06:42.836 INFO:tasks.rados.rados.0.smithi125.stdout:3212: done (7 left) 2021-03-24T22:06:42.836 INFO:tasks.rados.rados.0.smithi125.stdout:3213: done (6 left) 2021-03-24T22:06:42.837 INFO:tasks.rados.rados.0.smithi125.stdout:3214: done (5 left) 2021-03-24T22:06:42.837 INFO:tasks.rados.rados.0.smithi125.stdout:3215: done (4 left) 2021-03-24T22:06:42.837 INFO:tasks.rados.rados.0.smithi125.stdout:3216: done (3 left) 2021-03-24T22:06:42.837 INFO:tasks.rados.rados.0.smithi125.stdout:3217: done (2 left) 2021-03-24T22:06:42.837 INFO:tasks.rados.rados.0.smithi125.stdout:3218: done (1 left) 2021-03-24T22:06:42.838 INFO:tasks.rados.rados.0.smithi125.stdout:3219: done (0 left) 2021-03-24T22:06:42.838 INFO:tasks.rados.rados.0.smithi125.stdout:3220: read oid 24 snap -1 2021-03-24T22:06:42.838 INFO:tasks.rados.rados.0.smithi125.stdout:3220: expect deleted 2021-03-24T22:06:42.838 INFO:tasks.rados.rados.0.smithi125.stdout:3221: snap_create 2021-03-24T22:06:42.840 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:42.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:42.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:06:42.912 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:42 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T22:06:42.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:42 smithi125 conmon[24927]: -24T22:06:41.009948+0000 mgr.y (mgr.14144) 691 : cluster [DBG] pgmap v1408: 168 pgs: 1 active+clean+remapped, 167 active+clean; 525 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 8.9 MiB/s rd, 689 KiB/s wr, 7 op/s; 200 B/s, 2 objects/s recovering 2021-03-24T22:06:42.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:42 smithi125 conmon[24927]: cluster 2021-03-24T22:06:41. 2021-03-24T22:06:42.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:42 smithi125 conmon[24927]: 330499+0000 osd.5 (osd.5) 23 : cluster [DBG] 5.7 scrub ok 2021-03-24T22:06:42.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:42 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:06:42.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:42 smithi125 conmon[24927]: -03-24T22:06:41.587458+0000 osd.0 (osd.0) 163 2021-03-24T22:06:42.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:42 smithi125 conmon[24927]: : cluster [DBG] 7.4 scrub ok 2021-03-24T22:06:42.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:42 smithi125 conmon[24927]: cluster 2021-03-24T22:06:41.617247+0000 osd.7 2021-03-24T22:06:42.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:42 smithi125 conmon[24927]: (osd.7) 137 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:06:42.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:42 smithi125 conmon[24927]: cluster 2021-03-24T22:06:41.621780+0000 2021-03-24T22:06:42.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:42 smithi125 conmon[24927]: osd.7 (osd.7) 138 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:06:42.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:42 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:06:42.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:42 smithi125 conmon[24927]: -03-24T22:06:41.782234+0000 osd.1 2021-03-24T22:06:42.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:42 smithi125 conmon[24927]: (osd.1) 99 : cluster [DBG] 4.12 scrub ok 2021-03-24T22:06:42.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:42 smithi125 conmon[24927]: cluster 2021-03-24T22:06:42.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:42 smithi125 conmon[24927]: 2021-03-24T22:06:42.462958+0000 2021-03-24T22:06:42.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:42 smithi125 conmon[24927]: mon.a (mon.0) 2064 : cluster [DBG] osdmap e832: 8 total, 8 up, 8 in 2021-03-24T22:06:42.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:42 smithi125 conmon[24927]: 2021-03-24T22:06:42.917 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:42 smithi125 conmon[19936]: cluster 2021-03-24T22:06 2021-03-24T22:06:42.917 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:42 smithi125 conmon[19936]: :41.009948+0000 mgr.y (mgr.14144) 691 : cluster [DBG] pgmap v1408: 168 pgs: 1 active+clean+remapped, 167 active+clean; 525 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 8.9 MiB/s rd, 689 KiB/s wr, 7 op/s; 200 B/s, 2 objects/s recovering 2021-03-24T22:06:42.917 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:42 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:06:42.918 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:42 smithi125 conmon[19936]: -03-24T22:06:41.330499+0000 osd.5 (osd.5) 23 : cluster [DBG] 2021-03-24T22:06:42.918 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:42 smithi125 conmon[19936]: 5.7 scrub ok 2021-03-24T22:06:42.918 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:42 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:06:42.919 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:42 smithi125 conmon[19936]: 06:41.587458+0000 osd.0 (osd.0) 163 2021-03-24T22:06:42.919 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:42 smithi125 conmon[19936]: : cluster [DBG] 7.4 scrub ok 2021-03-24T22:06:42.919 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:42 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:06:42.919 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:42 smithi125 conmon[19936]: 06:41.617247+0000 osd.7 (osd.7) 2021-03-24T22:06:42.919 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:42 smithi125 conmon[19936]: 137 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:06:42.920 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:42 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:06:42.920 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:42 smithi125 conmon[19936]: 06:41.621780+0000 osd.7 (osd.7) 138 2021-03-24T22:06:42.920 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:42 smithi125 conmon[19936]: : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:06:42.920 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:42 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:06:42.921 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:42 smithi125 conmon[19936]: 06:41.782234+0000 osd.1 (osd.1) 99 2021-03-24T22:06:42.921 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:42 smithi125 conmon[19936]: : cluster [DBG] 4.12 scrub ok 2021-03-24T22:06:42.921 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:42 smithi125 conmon[19936]: cluster 2021-03-24T22:06 2021-03-24T22:06:42.921 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:42 smithi125 conmon[19936]: :42.462958+0000 mon.a (mon.0) 2064 2021-03-24T22:06:42.921 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:42 smithi125 conmon[19936]: : cluster [DBG] osdmap e832: 8 total, 8 up, 8 in 2021-03-24T22:06:42.946 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:43.003 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:43.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:43.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:06:43.110 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:43.173 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:43.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:43.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:06:43.278 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:43.337 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:43.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:43.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:06:43.444 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:43.499 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:43.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:43.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:43.602 INFO:tasks.rados.rados.0.smithi125.stdout:3220: done (1 left) 2021-03-24T22:06:43.603 INFO:tasks.rados.rados.0.smithi125.stdout:3221: done (0 left) 2021-03-24T22:06:43.603 INFO:tasks.rados.rados.0.smithi125.stdout:3222: snap_create 2021-03-24T22:06:43.606 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:43.660 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:43.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:43.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:43.767 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:43.828 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:43.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:43.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:06:43.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:43 smithi142 conmon[20125]: cluster 2021-03-24T22:06:42.199892+0000 osd.2 (osd.2) 134 : cluster [DBG] 2021-03-24T22:06:43.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:43 smithi142 conmon[20125]: 6.14 scrub ok 2021-03-24T22:06:43.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:43 smithi142 conmon[20125]: cluster 2021-03-24T22:06:43.019312+0000 osd.7 (osd.7) 139 : cluster [DBG] 5.9 scrub ok 2021-03-24T22:06:43.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:43 smithi142 conmon[20125]: cluster 2021-03-24T22:06:43.544378+0000 osd.0 (osd.0) 164 : cluster [DBG] 7.18 scrub ok 2021-03-24T22:06:43.933 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:43.951 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:43 smithi125 conmon[24927]: cluster 2021-03-24T22:06:42.199892+0000 osd.2 (osd.2) 134 : 2021-03-24T22:06:43.951 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:43 smithi125 conmon[24927]: cluster [DBG] 6.14 scrub ok 2021-03-24T22:06:43.952 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:43 smithi125 conmon[24927]: cluster 2021-03-24T22:06:43.019312+0000 osd.7 (osd.7) 139 : cluster [DBG] 5.9 scrub ok 2021-03-24T22:06:43.952 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:43 smithi125 conmon[24927]: cluster 2021-03-24T22:06:43.544378+0000 osd.0 (osd.0) 164 2021-03-24T22:06:43.952 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:43 smithi125 conmon[24927]: : cluster [DBG] 7.18 scrub ok 2021-03-24T22:06:43.952 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:43 smithi125 conmon[19936]: cluster 2021-03-24T22:06:42.199892+0000 osd.2 (osd.2) 2021-03-24T22:06:43.953 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:43 smithi125 conmon[19936]: 134 : cluster [DBG] 6.14 scrub ok 2021-03-24T22:06:43.953 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:43 smithi125 conmon[19936]: cluster 2021-03-24T22:06:43.019312+0000 osd.7 (osd.7) 139 : cluster [DBG] 5.9 scrub ok 2021-03-24T22:06:43.953 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:43 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:06:43.953 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:43 smithi125 conmon[19936]: 06:43.544378+0000 osd.0 (osd.0) 164 : cluster [DBG] 2021-03-24T22:06:43.954 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:43 smithi125 conmon[19936]: 7.18 scrub ok 2021-03-24T22:06:43.989 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:43.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:43.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:06:44.096 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:44.157 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:44.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:44.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:06:44.266 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:44.326 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:44.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:44.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:06:44.432 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:44.493 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:44.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:44.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:06:44.600 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:44.609 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[19936]: cluster 2021-03-24T22:06:44.609 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[19936]: 2021- 2021-03-24T22:06:44.609 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[19936]: 03-24T22:06:43.010711+0000 mgr.y (mgr.14144) 692 : cluster [DBG] pgmap v1411: 168 pgs: 168 active+clean; 524 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 8.9 MiB/s rd, 0 B/s wr, 6 op/s 2021-03-24T22:06:44.610 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:06:44.610 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[19936]: -03-24T22:06:43.172314+0000 osd.2 (osd.2) 135 2021-03-24T22:06:44.610 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[19936]: : cluster [DBG] 7.20 scrub ok 2021-03-24T22:06:44.610 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:06:44.610 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[19936]: 24T22:06:43.305175+0000 osd.5 (osd.5 2021-03-24T22:06:44.611 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[19936]: ) 24 : cluster [DBG] 5.15 scrub ok 2021-03-24T22:06:44.611 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:06:44.618 INFO:tasks.rados.rados.0.smithi125.stdout:3222: done (0 left) 2021-03-24T22:06:44.618 INFO:tasks.rados.rados.0.smithi125.stdout:3223: write oid 32 current snap is 373 2021-03-24T22:06:44.619 INFO:tasks.rados.rados.0.smithi125.stdout:3223: seq_num 1098 ranges {454086=578234,1669111=662043} 2021-03-24T22:06:44.620 INFO:tasks.rados.rados.0.smithi125.stdout:3223: writing smithi12556470-32 from 454086 to 1032320 tid 1 2021-03-24T22:06:44.629 INFO:tasks.rados.rados.0.smithi125.stdout:3223: writing smithi12556470-32 from 1669111 to 2331154 tid 2 2021-03-24T22:06:44.629 INFO:tasks.rados.rados.0.smithi125.stdout:3223: finishing write tid 1 to smithi12556470-32 2021-03-24T22:06:44.629 INFO:tasks.rados.rados.0.smithi125.stdout:3224: read oid 28 snap -1 2021-03-24T22:06:44.629 INFO:tasks.rados.rados.0.smithi125.stdout:3224: expect (ObjNum 961 snap 317 seq_num 961) 2021-03-24T22:06:44.630 INFO:tasks.rados.rados.0.smithi125.stdout:3225: snap_create 2021-03-24T22:06:44.635 INFO:tasks.rados.rados.0.smithi125.stdout:3224: expect (ObjNum 961 snap 317 seq_num 961) 2021-03-24T22:06:44.650 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:44.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:44.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:44.740 INFO:tasks.rados.rados.0.smithi125.stdout:3223: finishing write tid 2 to smithi12556470-32 2021-03-24T22:06:44.740 INFO:tasks.rados.rados.0.smithi125.stdout:3223: finishing write tid 3 to smithi12556470-32 2021-03-24T22:06:44.740 INFO:tasks.rados.rados.0.smithi125.stdout:3223: finishing write tid 4 to smithi12556470-32 2021-03-24T22:06:44.741 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 32 v 2055 (ObjNum 1098 snap 373 seq_num 1098) dirty exists 2021-03-24T22:06:44.741 INFO:tasks.rados.rados.0.smithi125.stdout:3223: left oid 32 (ObjNum 1098 snap 373 seq_num 1098) 2021-03-24T22:06:44.757 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:44.815 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:44.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:44.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:06:44.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:44 smithi142 conmon[20125]: cluster 2021-03-24T22:06:43.010711 2021-03-24T22:06:44.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:44 smithi142 conmon[20125]: +0000 mgr.y (mgr.14144) 692 : cluster [DBG] pgmap v1411: 168 pgs: 168 active+clean; 524 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 8.9 MiB/s rd, 0 B/s wr, 6 op/s 2021-03-24T22:06:44.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:44 smithi142 conmon[20125]: cluster 2021-03-24T22:06:43 2021-03-24T22:06:44.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:44 smithi142 conmon[20125]: .172314+0000 osd.2 (osd.2) 135 : cluster [DBG] 7.20 scrub ok 2021-03-24T22:06:44.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:44 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:06:44.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:44 smithi142 conmon[20125]: 22:06:43.305175+0000 osd.5 (osd.5) 2021-03-24T22:06:44.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:44 smithi142 conmon[20125]: 24 : cluster [DBG] 5.15 scrub ok 2021-03-24T22:06:44.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:44 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:06:44.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:44 smithi142 conmon[20125]: -24T22:06:43.597600+0000 mon.a (mon 2021-03-24T22:06:44.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:44 smithi142 conmon[20125]: .0) 2065 : cluster [DBG] osdmap e833: 8 total, 8 up, 8 in 2021-03-24T22:06:44.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:44 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:06:44.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:44 smithi142 conmon[20125]: 03-24T22:06:44.075299+0000 osd.3 ( 2021-03-24T22:06:44.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:44 smithi142 conmon[20125]: osd.3) 163 : cluster [DBG] 3.29 scrub ok 2021-03-24T22:06:44.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:44 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:06:44.870 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:44 smithi142 conmon[20125]: 03-24T22:06:44.506303+0000 osd.0 ( 2021-03-24T22:06:44.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:44 smithi142 conmon[20125]: osd.0) 165 : cluster [DBG] 6.16 scrub ok 2021-03-24T22:06:44.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:44 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:06:44.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:44 smithi142 conmon[20125]: -03-24T22:06:44.600372+0000 mon.a 2021-03-24T22:06:44.871 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:44 smithi142 conmon[20125]: (mon.0) 2066 : cluster [DBG] osdmap e834: 8 total, 8 up, 8 in 2021-03-24T22:06:44.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[19936]: 24T22:06:43.597600+0000 mon.a (mon.0 2021-03-24T22:06:44.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[19936]: ) 2065 : cluster [DBG] osdmap e833: 8 total, 8 up, 8 in 2021-03-24T22:06:44.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:06:44.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[19936]: 03-24 2021-03-24T22:06:44.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[19936]: T22:06:44.075299+0000 osd.3 (osd.3) 2021-03-24T22:06:44.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[19936]: 163 : cluster [DBG] 3.29 scrub ok 2021-03-24T22:06:44.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[19936]: cluster 2021-03-24T22 2021-03-24T22:06:44.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[19936]: :06:44.506303+0000 osd.0 (osd.0) 165 : 2021-03-24T22:06:44.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[19936]: cluster [DBG] 6.16 scrub ok 2021-03-24T22:06:44.915 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[19936]: cluster 2021-03-24T22:06 2021-03-24T22:06:44.915 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[19936]: :44.600372+0000 mon.a (mon.0) 2066 : cluster 2021-03-24T22:06:44.915 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[19936]: [DBG] osdmap e834: 8 total, 8 up, 8 in 2021-03-24T22:06:44.917 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[24927]: cluster 2021-03-24T22:06:44.917 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[24927]: 2021-03-24T22:06:44.917 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[24927]: 2021 2021-03-24T22:06:44.917 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[24927]: - 2021-03-24T22:06:44.918 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[24927]: 03 2021-03-24T22:06:44.918 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[24927]: - 2021-03-24T22:06:44.918 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[24927]: 24 2021-03-24T22:06:44.918 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[24927]: T 2021-03-24T22:06:44.919 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[24927]: 22 2021-03-24T22:06:44.919 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[24927]: : 2021-03-24T22:06:44.919 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[24927]: 06 2021-03-24T22:06:44.919 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[24927]: : 2021-03-24T22:06:44.919 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[24927]: 43.010711+0000 mgr.y (mgr.14144) 692 : cluster 2021-03-24T22:06:44.920 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[24927]: [DBG] pgmap v1411: 168 pgs: 168 active+clean; 524 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 8.9 MiB/s rd, 0 B/s wr, 6 op/s 2021-03-24T22:06:44.920 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:06:44.920 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[24927]: -03-24T22:06:43.172314+0000 osd.2 (osd.2) 135 : cluster [DBG] 7.20 scrub ok 2021-03-24T22:06:44.921 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[24927]: cluster 2021-03-24T22:06:44.921 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[24927]: 2021-03-24T22:06:43.305175+0000 osd.5 (osd.5 2021-03-24T22:06:44.921 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[24927]: ) 24 : cluster [DBG] 5.15 scrub ok 2021-03-24T22:06:44.921 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:06:44.921 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[24927]: -03-24T22:06:43 2021-03-24T22:06:44.922 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[24927]: .597600+0000 mon.a (mon.0) 2021-03-24T22:06:44.922 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[24927]: 2065 : cluster [DBG] osdmap e833: 8 total, 8 up, 8 in 2021-03-24T22:06:44.922 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[24927]: cluster 2021-03-24T22:06:44.922 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[24927]: 2021-03-24T22:06:44.075299+0000 osd.3 (osd. 2021-03-24T22:06:44.923 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[24927]: 3) 163 : cluster [DBG] 3.29 scrub ok 2021-03-24T22:06:44.923 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T22:06:44.923 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[24927]: 22:06:44.506303+0000 osd.0 2021-03-24T22:06:44.923 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[24927]: (osd.0) 165 : cluster [DBG] 2021-03-24T22:06:44.923 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[24927]: 6.16 scrub ok 2021-03-24T22:06:44.924 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:06:44.924 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[24927]: -03-24T22:06:44.600372 2021-03-24T22:06:44.924 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:44 smithi125 conmon[24927]: +0000 mon.a (mon.0) 2066 : cluster [DBG] osdmap e834: 8 total, 8 up, 8 in 2021-03-24T22:06:44.924 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:44.964 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-24T22:06:44.968 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2021-03-24T22:06:44.973 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:44.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:44.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:06:45.079 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:45.104 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.3 dump_historic_ops 2021-03-24T22:06:45.132 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:45.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:45.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:06:45.238 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:45.289 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:45.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:45.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:06:45.395 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:45.450 INFO:tasks.rados.rados.0.smithi125.stdout:3223: done (2 left) 2021-03-24T22:06:45.450 INFO:tasks.rados.rados.0.smithi125.stdout:3224: done (1 left) 2021-03-24T22:06:45.450 INFO:tasks.rados.rados.0.smithi125.stdout:3225: done (0 left) 2021-03-24T22:06:45.451 INFO:tasks.rados.rados.0.smithi125.stdout:3226: copy_from oid 44 from oid 48 current snap is 374 2021-03-24T22:06:45.451 INFO:tasks.rados.rados.0.smithi125.stdout:3227: copy_from oid 19 from oid 13 current snap is 374 2021-03-24T22:06:45.451 INFO:tasks.rados.rados.0.smithi125.stdout:3228: read oid 32 snap -1 2021-03-24T22:06:45.452 INFO:tasks.rados.rados.0.smithi125.stdout:3228: expect (ObjNum 1098 snap 373 seq_num 1098) 2021-03-24T22:06:45.452 INFO:tasks.rados.rados.0.smithi125.stdout:3229: snap_create 2021-03-24T22:06:45.455 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:45.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:45.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:06:45.462 INFO:tasks.rados.rados.0.smithi125.stdout:3228: expect (ObjNum 1098 snap 373 seq_num 1098) 2021-03-24T22:06:45.562 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:45.614 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:45.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:45.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:06:45.646 INFO:tasks.rados.rados.0.smithi125.stdout:3226: finishing copy_from to smithi12556470-44 2021-03-24T22:06:45.646 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 44 v 1133 (ObjNum 916 snap 304 seq_num 916) dirty exists 2021-03-24T22:06:45.647 INFO:tasks.rados.rados.0.smithi125.stdout:3226: finishing copy_from racing read to smithi12556470-44 2021-03-24T22:06:45.647 INFO:tasks.rados.rados.0.smithi125.stdout:3227: finishing copy_from to smithi12556470-19 2021-03-24T22:06:45.647 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 19 v 1463 (ObjNum 988 snap 328 seq_num 988) dirty exists 2021-03-24T22:06:45.647 INFO:tasks.rados.rados.0.smithi125.stdout:3227: finishing copy_from racing read to smithi12556470-19 2021-03-24T22:06:45.719 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:45.775 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:45.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:45.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:45.882 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:45.895 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:45 smithi142 conmon[20125]: cluster 2021-03-24T22:06:44.161989+0000 osd.2 (osd.2) 136 : cluster [DBG] 5.b scrub ok 2021-03-24T22:06:45.895 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:45 smithi142 conmon[20125]: cluster 2021-03-24T22:06:45.441273+0000 mon.a (mon.0) 2067 : cluster [DBG] osdmap e835: 8 total, 8 up, 8 in 2021-03-24T22:06:45.919 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:45 smithi125 conmon[19936]: cluster 2021-03-24T22:06:44.161989+0000 osd.2 (osd.2) 136 : cluster [DBG] 5.b scrub ok 2021-03-24T22:06:45.919 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:45 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:06:45.920 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:45 smithi125 conmon[19936]: T22:06:45.441273+0000 mon.a (mon.0) 2067 : cluster [DBG] osdmap e835: 8 total, 8 up, 8 in 2021-03-24T22:06:45.920 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:45 smithi125 conmon[24927]: cluster 2021-03-24T22:06:44. 2021-03-24T22:06:45.920 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:45 smithi125 conmon[24927]: 161989+0000 osd.2 (osd.2) 136 : cluster [DBG] 5.b scrub ok 2021-03-24T22:06:45.920 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:45 smithi125 conmon[24927]: cluster 2021-03-24T22:06:45.441273 2021-03-24T22:06:45.921 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:45 smithi125 conmon[24927]: +0000 mon.a (mon.0) 2067 : cluster [DBG] osdmap e835: 8 total, 8 up, 8 in 2021-03-24T22:06:45.938 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:45.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:45.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:06:46.044 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:46.100 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:46.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:46.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:06:46.206 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:46.265 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:46.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:46.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:06:46.371 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:46.429 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:46.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:46.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:06:46.463 INFO:tasks.rados.rados.0.smithi125.stdout:3226: done (3 left) 2021-03-24T22:06:46.464 INFO:tasks.rados.rados.0.smithi125.stdout:3227: done (2 left) 2021-03-24T22:06:46.464 INFO:tasks.rados.rados.0.smithi125.stdout:3228: done (1 left) 2021-03-24T22:06:46.464 INFO:tasks.rados.rados.0.smithi125.stdout:3229: done (0 left) 2021-03-24T22:06:46.464 INFO:tasks.rados.rados.0.smithi125.stdout:3230: snap_remove snap 334 2021-03-24T22:06:46.467 INFO:teuthology.orchestra.run.smithi125.stderr:nodeep-scrub is unset 2021-03-24T22:06:46.537 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:46.559 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:06:46.587 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":1413,"stamp":"2021-03-24T22:06:44.594611+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":546125623,"num_objects":550,"num_object_clones":184,"num_object_copies":1650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":550,"num_whiteouts":10,"num_read":4964,"num_read_kb":3289710,"num_write":3107,"num_write_kb":878068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3404,"num_bytes_recovered":1222565970,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21290,"ondisk_log_size":21290,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":506,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1597188,"kb_used_data":1245636,"kb_used_omap":164,"kb_used_meta":351323,"kb_avail":748364028,"statfs":{"total":767960285184,"available":766324764672,"internally_reserved":0,"allocated":1275531264,"data_stored":1688090050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":168054,"internal_metadata":359755658},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1437931,"num_objects":-3,"num_object_clones":-3,"num_object_copies":-9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":-3,"num_read":46,"num_read_kb":51561,"num_write":19,"num_write_kb":6687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.160721"},"pg_stats":[{"pgid":"3.29","version":"0'0","reported_seq":"483","reported_epoch":"832","state":"active+clean","last_fresh":"2021-03-24T22:06:42.469927+0000","last_change":"2021-03-24T22:03:45.428201+0000","last_active":"2021-03-24T22:06:42.469927+0000","last_peered":"2021-03-24T22:06:42.469927+0000","last_clean":"2021-03-24T22:06:42.469927+0000","last_became_active":"2021-03-24T22:03:45.427894+0000","last_became_peered":"2021-03-24T22:03:45.427894+0000","last_unstale":"2021-03-24T22:06:42.469927+0000","last_undegraded":"2021-03-24T22:06:42.469927+0000","last_fullsized":"2021-03-24T22:06:42.469927+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:57:55.592795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:57:55.592795+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":"352","reported_epoch":"830","state":"active+clean","last_fresh":"2021-03-24T22:06:40.457703+0000","last_change":"2021-03-24T22:05:54.394091+0000","last_active":"2021-03-24T22:06:40.457703+0000","last_peered":"2021-03-24T22:06:40.457703+0000","last_clean":"2021-03-24T22:06:40.457703+0000","last_became_active":"2021-03-24T22:05:04.450159+0000","last_became_peered":"2021-03-24T22:05:04.450159+0000","last_unstale":"2021-03-24T22:06:40.457703+0000","last_undegraded":"2021-03-24T22:06:40.457703+0000","last_fullsized":"2021-03-24T22:06:40.457703+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:54.393923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:05:54.393923+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"89'2","reported_seq":"503","reported_epoch":"831","state":"active+clean","last_fresh":"2021-03-24T22:06:41.465330+0000","last_change":"2021-03-24T22:04:28.159451+0000","last_active":"2021-03-24T22:06:41.465330+0000","last_peered":"2021-03-24T22:06:41.465330+0000","last_clean":"2021-03-24T22:06:41.465330+0000","last_became_active":"2021-03-24T21:59:14.374418+0000","last_became_peered":"2021-03-24T21:59:14.374418+0000","last_unstale":"2021-03-24T22:06:41.465330+0000","last_undegraded":"2021-03-24T22:06:41.465330+0000","last_fullsized":"2021-03-24T22:06:41.465330+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":6,"last_scrub":"89'2","last_scrub_stamp":"2021-03-24T22:04:28.159278+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2021-03-24T21:57:58.491830+0000","last_clean_scrub_stamp":"2021-03-24T22:04:28.159278+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":"845","reported_epoch":"832","state":"active+clean","last_fresh":"2021-03-24T22:06:42.469290+0000","last_change":"2021-03-24T22:05:04.443648+0000","last_active":"2021-03-24T22:06:42.469290+0000","last_peered":"2021-03-24T22:06:42.469290+0000","last_clean":"2021-03-24T22:06:42.469290+0000","last_became_active":"2021-03-24T22:05:04.442258+0000","last_became_peered":"2021-03-24T22:05:04.442258+0000","last_unstale":"2021-03-24T22:06:42.469290+0000","last_undegraded":"2021-03-24T22:06:42.469290+0000","last_fullsized":"2021-03-24T22:06:42.469290+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:43.239983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:03:43.239983+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":"286","reported_epoch":"830","state":"active+clean","last_fresh":"2021-03-24T22:06:40.450343+0000","last_change":"2021-03-24T22:05:53.982341+0000","last_active":"2021-03-24T22:06:40.450343+0000","last_peered":"2021-03-24T22:06:40.450343+0000","last_clean":"2021-03-24T22:06:40.450343+0000","last_became_active":"2021-03-24T22:05:04.448985+0000","last_became_peered":"2021-03-24T22:05:04.448985+0000","last_unstale":"2021-03-24T22:06:40.450343+0000","last_undegraded":"2021-03-24T22:06:40.450343+0000","last_fullsized":"2021-03-24T22:06:40.450343+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:53.982228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:05:53.982228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"61'1","reported_seq":"325","reported_epoch":"829","state":"active+clean","last_fresh":"2021-03-24T22:06:39.460522+0000","last_change":"2021-03-24T22:06:04.143008+0000","last_active":"2021-03-24T22:06:39.460522+0000","last_peered":"2021-03-24T22:06:39.460522+0000","last_clean":"2021-03-24T22:06:39.460522+0000","last_became_active":"2021-03-24T22:05:04.441568+0000","last_became_peered":"2021-03-24T22:05:04.441568+0000","last_unstale":"2021-03-24T22:06:39.460522+0000","last_undegraded":"2021-03-24T22:06:39.460522+0000","last_fullsized":"2021-03-24T22:06:39.460522+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":6,"last_scrub":"61'1","last_scrub_stamp":"2021-03-24T22:06:04.142856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:06:04.142856+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":"812","reported_epoch":"832","state":"active+clean","last_fresh":"2021-03-24T22:06:42.475975+0000","last_change":"2021-03-24T22:02:57.502941+0000","last_active":"2021-03-24T22:06:42.475975+0000","last_peered":"2021-03-24T22:06:42.475975+0000","last_clean":"2021-03-24T22:06:42.475975+0000","last_became_active":"2021-03-24T21:59:14.370506+0000","last_became_peered":"2021-03-24T21:59:14.370506+0000","last_unstale":"2021-03-24T22:06:42.475975+0000","last_undegraded":"2021-03-24T22:06:42.475975+0000","last_fullsized":"2021-03-24T22:06:42.475975+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:57.502803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:02:57.502803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.23","version":"830'516","reported_seq":"2981","reported_epoch":"830","state":"active+clean","last_fresh":"2021-03-24T22:06:40.473614+0000","last_change":"2021-03-24T22:06:40.473898+0000","last_active":"2021-03-24T22:06:40.473614+0000","last_peered":"2021-03-24T22:06:40.473614+0000","last_clean":"2021-03-24T22:06:40.473614+0000","last_became_active":"2021-03-24T22:03:49.378704+0000","last_became_peered":"2021-03-24T22:03:49.378704+0000","last_unstale":"2021-03-24T22:06:40.473614+0000","last_undegraded":"2021-03-24T22:06:40.473614+0000","last_fullsized":"2021-03-24T22:06:40.473614+0000","mapping_epoch":658,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":659,"parent":"0.0","parent_split_bits":6,"last_scrub":"731'441","last_scrub_stamp":"2021-03-24T22:05:01.661041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:05:01.661041+0000","log_size":516,"ondisk_log_size":516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15434390,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":10122,"num_write":12,"num_write_kb":5736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"27d","length":"1"},{"start":"297","length":"1"},{"start":"2ab","length":"1"},{"start":"2ad","length":"1"},{"start":"2c4","length":"1"}]},{"pgid":"3.26","version":"0'0","reported_seq":"315","reported_epoch":"829","state":"active+clean","last_fresh":"2021-03-24T22:06:39.462083+0000","last_change":"2021-03-24T22:04:21.723861+0000","last_active":"2021-03-24T22:06:39.462083+0000","last_peered":"2021-03-24T22:06:39.462083+0000","last_clean":"2021-03-24T22:06:39.462083+0000","last_became_active":"2021-03-24T22:01:57.311128+0000","last_became_peered":"2021-03-24T22:01:57.311128+0000","last_unstale":"2021-03-24T22:06:39.462083+0000","last_undegraded":"2021-03-24T22:06:39.462083+0000","last_fullsized":"2021-03-24T22:06:39.462083+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:04:21.723761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:04:21.723761+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"831'395","reported_seq":"2108","reported_epoch":"831","state":"active+clean","last_fresh":"2021-03-24T22:06:41.474601+0000","last_change":"2021-03-24T22:06:41.474601+0000","last_active":"2021-03-24T22:06:41.474601+0000","last_peered":"2021-03-24T22:06:41.474601+0000","last_clean":"2021-03-24T22:06:41.474601+0000","last_became_active":"2021-03-24T22:03:51.364119+0000","last_became_peered":"2021-03-24T22:03:51.364119+0000","last_unstale":"2021-03-24T22:06:41.474601+0000","last_undegraded":"2021-03-24T22:06:41.474601+0000","last_fullsized":"2021-03-24T22:06:41.474601+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":6,"last_scrub":"729'308","last_scrub_stamp":"2021-03-24T22:05:00.252320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:05:00.252320+0000","log_size":395,"ondisk_log_size":395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13388469,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":4,"num_read_kb":1,"num_write":9,"num_write_kb":1363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"255","length":"1"},{"start":"27d","length":"1"},{"start":"297","length":"1"},{"start":"2ab","length":"1"},{"start":"2ad","length":"1"},{"start":"2c4","length":"1"}]},{"pgid":"4.c","version":"544'37","reported_seq":"561","reported_epoch":"832","state":"active+clean","last_fresh":"2021-03-24T22:06:42.473974+0000","last_change":"2021-03-24T22:03:03.453859+0000","last_active":"2021-03-24T22:06:42.473974+0000","last_peered":"2021-03-24T22:06:42.473974+0000","last_clean":"2021-03-24T22:06:42.473974+0000","last_became_active":"2021-03-24T22:01:50.301325+0000","last_became_peered":"2021-03-24T22:01:50.301325+0000","last_unstale":"2021-03-24T22:06:42.473974+0000","last_undegraded":"2021-03-24T22:06:42.473974+0000","last_fullsized":"2021-03-24T22:06:42.473974+0000","mapping_epoch":539,"log_start":"69'15","ondisk_log_start":"69'15","created":62,"last_epoch_clean":540,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'37","last_scrub_stamp":"2021-03-24T22:03:03.453713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:03:03.453713+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":15,"num_object_clones":0,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":81,"num_read_kb":61,"num_write":51,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":1266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"800","reported_epoch":"832","state":"active+clean","last_fresh":"2021-03-24T22:06:42.470085+0000","last_change":"2021-03-24T22:05:02.207625+0000","last_active":"2021-03-24T22:06:42.470085+0000","last_peered":"2021-03-24T22:06:42.470085+0000","last_clean":"2021-03-24T22:06:42.470085+0000","last_became_active":"2021-03-24T21:51:45.375830+0000","last_became_peered":"2021-03-24T21:51:45.375830+0000","last_unstale":"2021-03-24T22:06:42.470085+0000","last_undegraded":"2021-03-24T22:06:42.470085+0000","last_fullsized":"2021-03-24T22:06:42.470085+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:02.207454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:05:02.207454+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"734'14","reported_seq":"1073","reported_epoch":"830","state":"active+clean","last_fresh":"2021-03-24T22:06:40.452044+0000","last_change":"2021-03-24T22:05:47.047703+0000","last_active":"2021-03-24T22:06:40.452044+0000","last_peered":"2021-03-24T22:06:40.452044+0000","last_clean":"2021-03-24T22:06:40.452044+0000","last_became_active":"2021-03-24T22:05:04.430698+0000","last_became_peered":"2021-03-24T22:05:04.430698+0000","last_unstale":"2021-03-24T22:06:40.452044+0000","last_undegraded":"2021-03-24T22:06:40.452044+0000","last_fullsized":"2021-03-24T22:06:40.452044+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"734'14","last_scrub_stamp":"2021-03-24T22:05:47.047527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:05:47.047527+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"194","reported_epoch":"828","state":"active+clean","last_fresh":"2021-03-24T22:06:38.452461+0000","last_change":"2021-03-24T22:06:34.796244+0000","last_active":"2021-03-24T22:06:38.452461+0000","last_peered":"2021-03-24T22:06:38.452461+0000","last_clean":"2021-03-24T22:06:38.452461+0000","last_became_active":"2021-03-24T22:03:45.430245+0000","last_became_peered":"2021-03-24T22:03:45.430245+0000","last_unstale":"2021-03-24T22:06:38.452461+0000","last_undegraded":"2021-03-24T22:06:38.452461+0000","last_fullsized":"2021-03-24T22:06:38.452461+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:34.796158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:06:34.796158+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.f","version":"830'748","reported_seq":"3229","reported_epoch":"831","state":"active+clean","last_fresh":"2021-03-24T22:06:41.467907+0000","last_change":"2021-03-24T22:06:41.467907+0000","last_active":"2021-03-24T22:06:41.467907+0000","last_peered":"2021-03-24T22:06:41.467907+0000","last_clean":"2021-03-24T22:06:41.467907+0000","last_became_active":"2021-03-24T22:02:10.313297+0000","last_became_peered":"2021-03-24T22:02:10.313297+0000","last_unstale":"2021-03-24T22:06:41.467907+0000","last_undegraded":"2021-03-24T22:06:41.467907+0000","last_fullsized":"2021-03-24T22:06:41.467907+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"705'573","last_scrub_stamp":"2021-03-24T22:04:36.224078+0000","last_deep_scrub":"705'573","last_deep_scrub_stamp":"2021-03-24T22:04:36.224078+0000","last_clean_scrub_stamp":"2021-03-24T22:04:36.224078+0000","log_size":748,"ondisk_log_size":748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31946352,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":137,"num_read_kb":142457,"num_write":101,"num_write_kb":38492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":46545574,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"255","length":"1"},{"start":"27d","length":"1"},{"start":"297","length":"1"},{"start":"2ab","length":"1"},{"start":"2ad","length":"1"},{"start":"2c4","length":"1"}]},{"pgid":"4.d","version":"544'41","reported_seq":"544","reported_epoch":"830","state":"active+clean","last_fresh":"2021-03-24T22:06:40.459260+0000","last_change":"2021-03-24T22:06:34.167443+0000","last_active":"2021-03-24T22:06:40.459260+0000","last_peered":"2021-03-24T22:06:40.459260+0000","last_clean":"2021-03-24T22:06:40.459260+0000","last_became_active":"2021-03-24T22:03:45.430156+0000","last_became_peered":"2021-03-24T22:03:45.430156+0000","last_unstale":"2021-03-24T22:06:40.459260+0000","last_undegraded":"2021-03-24T22:06:40.459260+0000","last_fullsized":"2021-03-24T22:06:40.459260+0000","mapping_epoch":654,"log_start":"69'21","ondisk_log_start":"69'21","created":62,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'41","last_scrub_stamp":"2021-03-24T22:06:34.167247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:06:34.167247+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1578,"num_objects":21,"num_object_clones":0,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":106,"num_read_kb":71,"num_write":60,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"329","reported_epoch":"829","state":"active+clean","last_fresh":"2021-03-24T22:06:39.461062+0000","last_change":"2021-03-24T22:05:02.665356+0000","last_active":"2021-03-24T22:06:39.461062+0000","last_peered":"2021-03-24T22:06:39.461062+0000","last_clean":"2021-03-24T22:06:39.461062+0000","last_became_active":"2021-03-24T22:03:45.419665+0000","last_became_peered":"2021-03-24T22:03:45.419665+0000","last_unstale":"2021-03-24T22:06:39.461062+0000","last_undegraded":"2021-03-24T22:06:39.461062+0000","last_fullsized":"2021-03-24T22:06:39.461062+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:02.664970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:55:01.169461+0000","last_clean_scrub_stamp":"2021-03-24T22:05:02.664970+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"194","reported_epoch":"828","state":"active+clean","last_fresh":"2021-03-24T22:06:38.450510+0000","last_change":"2021-03-24T22:06:33.822549+0000","last_active":"2021-03-24T22:06:38.450510+0000","last_peered":"2021-03-24T22:06:38.450510+0000","last_clean":"2021-03-24T22:06:38.450510+0000","last_became_active":"2021-03-24T22:03:45.430779+0000","last_became_peered":"2021-03-24T22:03:45.430779+0000","last_unstale":"2021-03-24T22:06:38.450510+0000","last_undegraded":"2021-03-24T22:06:38.450510+0000","last_fullsized":"2021-03-24T22:06:38.450510+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:33.822406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:06:33.822406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"276","reported_epoch":"830","state":"active+clean","last_fresh":"2021-03-24T22:06:40.459407+0000","last_change":"2021-03-24T22:06:01.977397+0000","last_active":"2021-03-24T22:06:40.459407+0000","last_peered":"2021-03-24T22:06:40.459407+0000","last_clean":"2021-03-24T22:06:40.459407+0000","last_became_active":"2021-03-24T22:02:22.482380+0000","last_became_peered":"2021-03-24T22:02:22.482380+0000","last_unstale":"2021-03-24T22:06:40.459407+0000","last_undegraded":"2021-03-24T22:06:40.459407+0000","last_fullsized":"2021-03-24T22:06:40.459407+0000","mapping_epoch":571,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:01.977309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:06:01.977309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"831'319","reported_seq":"2639","reported_epoch":"831","state":"active+clean","last_fresh":"2021-03-24T22:06:41.492911+0000","last_change":"2021-03-24T22:06:41.495282+0000","last_active":"2021-03-24T22:06:41.492911+0000","last_peered":"2021-03-24T22:06:41.492911+0000","last_clean":"2021-03-24T22:06:41.492911+0000","last_became_active":"2021-03-24T22:05:04.448963+0000","last_became_peered":"2021-03-24T22:05:04.448963+0000","last_unstale":"2021-03-24T22:06:41.492911+0000","last_undegraded":"2021-03-24T22:06:41.492911+0000","last_fullsized":"2021-03-24T22:06:41.492911+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"773'278","last_scrub_stamp":"2021-03-24T22:05:46.070371+0000","last_deep_scrub":"546'150","last_deep_scrub_stamp":"2021-03-24T22:01:56.662245+0000","last_clean_scrub_stamp":"2021-03-24T22:05:46.070371+0000","log_size":319,"ondisk_log_size":319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6203689,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":106,"num_read_kb":93813,"num_write":56,"num_write_kb":14865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":8580883,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"255","length":"1"},{"start":"27d","length":"1"},{"start":"297","length":"1"},{"start":"2ab","length":"1"},{"start":"2ad","length":"1"},{"start":"2c4","length":"1"}]},{"pgid":"4.e","version":"544'45","reported_seq":"698","reported_epoch":"829","state":"active+clean","last_fresh":"2021-03-24T22:06:39.462490+0000","last_change":"2021-03-24T22:06:38.254933+0000","last_active":"2021-03-24T22:06:39.462490+0000","last_peered":"2021-03-24T22:06:39.462490+0000","last_clean":"2021-03-24T22:06:39.462490+0000","last_became_active":"2021-03-24T22:03:53.368885+0000","last_became_peered":"2021-03-24T22:03:53.368885+0000","last_unstale":"2021-03-24T22:06:39.462490+0000","last_undegraded":"2021-03-24T22:06:39.462490+0000","last_fullsized":"2021-03-24T22:06:39.462490+0000","mapping_epoch":662,"log_start":"89'27","ondisk_log_start":"89'27","created":62,"last_epoch_clean":663,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'45","last_scrub_stamp":"2021-03-24T22:06:38.254767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:06:38.254767+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1672,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":92,"num_read_kb":60,"num_write":50,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":5016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"817","reported_epoch":"830","state":"active+clean","last_fresh":"2021-03-24T22:06:40.450443+0000","last_change":"2021-03-24T22:05:56.072585+0000","last_active":"2021-03-24T22:06:40.450443+0000","last_peered":"2021-03-24T22:06:40.450443+0000","last_clean":"2021-03-24T22:06:40.450443+0000","last_became_active":"2021-03-24T22:01:57.302297+0000","last_became_peered":"2021-03-24T22:01:57.302297+0000","last_unstale":"2021-03-24T22:06:40.450443+0000","last_undegraded":"2021-03-24T22:06:40.450443+0000","last_fullsized":"2021-03-24T22:06:40.450443+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:56.072443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T22:05:56.072443+0000","last_clean_scrub_stamp":"2021-03-24T22:05:56.072443+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"321","reported_epoch":"829","state":"active+clean","last_fresh":"2021-03-24T22:06:39.462532+0000","last_change":"2021-03-24T22:05:14.424426+0000","last_active":"2021-03-24T22:06:39.462532+0000","last_peered":"2021-03-24T22:06:39.462532+0000","last_clean":"2021-03-24T22:06:39.462532+0000","last_became_active":"2021-03-24T22:05:04.428211+0000","last_became_peered":"2021-03-24T22:05:04.428211+0000","last_unstale":"2021-03-24T22:06:39.462532+0000","last_undegraded":"2021-03-24T22:06:39.462532+0000","last_fullsized":"2021-03-24T22:06:39.462532+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:14.424333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:05:14.424333+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"866","reported_epoch":"832","state":"active+clean","last_fresh":"2021-03-24T22:06:42.473579+0000","last_change":"2021-03-24T22:06:03.906295+0000","last_active":"2021-03-24T22:06:42.473579+0000","last_peered":"2021-03-24T22:06:42.473579+0000","last_clean":"2021-03-24T22:06:42.473579+0000","last_became_active":"2021-03-24T22:05:04.447129+0000","last_became_peered":"2021-03-24T22:05:04.447129+0000","last_unstale":"2021-03-24T22:06:42.473579+0000","last_undegraded":"2021-03-24T22:06:42.473579+0000","last_fullsized":"2021-03-24T22:06:42.473579+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:03.906118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:06:03.906118+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"830'853","reported_seq":"3400","reported_epoch":"830","state":"active+clean","last_fresh":"2021-03-24T22:06:40.475637+0000","last_change":"2021-03-24T22:06:40.476170+0000","last_active":"2021-03-24T22:06:40.475637+0000","last_peered":"2021-03-24T22:06:40.475637+0000","last_clean":"2021-03-24T22:06:40.475637+0000","last_became_active":"2021-03-24T22:03:53.379184+0000","last_became_peered":"2021-03-24T22:03:53.379184+0000","last_unstale":"2021-03-24T22:06:40.475637+0000","last_undegraded":"2021-03-24T22:06:40.475637+0000","last_fullsized":"2021-03-24T22:06:40.475637+0000","mapping_epoch":662,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":663,"parent":"0.0","parent_split_bits":0,"last_scrub":"787'795","last_scrub_stamp":"2021-03-24T22:05:59.132881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:05:59.132881+0000","log_size":853,"ondisk_log_size":853,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17355123,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":140,"num_read_kb":179025,"num_write":60,"num_write_kb":26017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":179,"num_bytes_recovered":70071133,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"27d","length":"1"},{"start":"297","length":"1"},{"start":"2ab","length":"1"},{"start":"2ad","length":"1"},{"start":"2c4","length":"1"}]},{"pgid":"4.f","version":"544'46","reported_seq":"1023","reported_epoch":"828","state":"active+clean","last_fresh":"2021-03-24T22:06:38.450478+0000","last_change":"2021-03-24T22:06:35.800840+0000","last_active":"2021-03-24T22:06:38.450478+0000","last_peered":"2021-03-24T22:06:38.450478+0000","last_clean":"2021-03-24T22:06:38.450478+0000","last_became_active":"2021-03-24T22:04:18.462489+0000","last_became_peered":"2021-03-24T22:04:18.462489+0000","last_unstale":"2021-03-24T22:06:38.450478+0000","last_undegraded":"2021-03-24T22:06:38.450478+0000","last_fullsized":"2021-03-24T22:06:38.450478+0000","mapping_epoch":687,"log_start":"274'26","ondisk_log_start":"274'26","created":62,"last_epoch_clean":688,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'46","last_scrub_stamp":"2021-03-24T22:06:35.800671+0000","last_deep_scrub":"274'26","last_deep_scrub_stamp":"2021-03-24T21:57:24.499090+0000","last_clean_scrub_stamp":"2021-03-24T22:06:35.800671+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1472,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":97,"num_read_kb":65,"num_write":56,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":8410,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":"877","reported_epoch":"832","state":"active+clean","last_fresh":"2021-03-24T22:06:42.468458+0000","last_change":"2021-03-24T22:03:45.428200+0000","last_active":"2021-03-24T22:06:42.468458+0000","last_peered":"2021-03-24T22:06:42.468458+0000","last_clean":"2021-03-24T22:06:42.468458+0000","last_became_active":"2021-03-24T22:03:45.427907+0000","last_became_peered":"2021-03-24T22:03:45.427907+0000","last_unstale":"2021-03-24T22:06:42.468458+0000","last_undegraded":"2021-03-24T22:06:42.468458+0000","last_fullsized":"2021-03-24T22:06:42.468458+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:56:39.439170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:56:39.439170+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"734'7","reported_seq":"466","reported_epoch":"831","state":"active+clean","last_fresh":"2021-03-24T22:06:41.458747+0000","last_change":"2021-03-24T22:05:09.312441+0000","last_active":"2021-03-24T22:06:41.458747+0000","last_peered":"2021-03-24T22:06:41.458747+0000","last_clean":"2021-03-24T22:06:41.458747+0000","last_became_active":"2021-03-24T22:05:04.447711+0000","last_became_peered":"2021-03-24T22:05:04.447711+0000","last_unstale":"2021-03-24T22:06:41.458747+0000","last_undegraded":"2021-03-24T22:06:41.458747+0000","last_fullsized":"2021-03-24T22:06:41.458747+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"734'7","last_scrub_stamp":"2021-03-24T22:05:09.312269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:05:09.312269+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"661","reported_epoch":"830","state":"active+clean","last_fresh":"2021-03-24T22:06:40.458232+0000","last_change":"2021-03-24T22:05:47.327420+0000","last_active":"2021-03-24T22:06:40.458232+0000","last_peered":"2021-03-24T22:06:40.458232+0000","last_clean":"2021-03-24T22:06:40.458232+0000","last_became_active":"2021-03-24T22:05:04.451010+0000","last_became_peered":"2021-03-24T22:05:04.451010+0000","last_unstale":"2021-03-24T22:06:40.458232+0000","last_undegraded":"2021-03-24T22:06:40.458232+0000","last_fullsized":"2021-03-24T22:06:40.458232+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:47.327213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:05:47.327213+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"830'1351","reported_seq":"4079","reported_epoch":"830","state":"active+clean","last_fresh":"2021-03-24T22:06:40.464947+0000","last_change":"2021-03-24T22:06:40.465623+0000","last_active":"2021-03-24T22:06:40.464947+0000","last_peered":"2021-03-24T22:06:40.464947+0000","last_clean":"2021-03-24T22:06:40.464947+0000","last_became_active":"2021-03-24T22:02:06.309737+0000","last_became_peered":"2021-03-24T22:02:06.309737+0000","last_unstale":"2021-03-24T22:06:40.464947+0000","last_undegraded":"2021-03-24T22:06:40.464947+0000","last_fullsized":"2021-03-24T22:06:40.464947+0000","mapping_epoch":555,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":556,"parent":"0.0","parent_split_bits":0,"last_scrub":"659'936","last_scrub_stamp":"2021-03-24T22:03:53.662470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:03:53.662470+0000","log_size":1351,"ondisk_log_size":1351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36781180,"num_objects":15,"num_object_clones":12,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":261,"num_read_kb":279911,"num_write":158,"num_write_kb":66732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":296,"num_bytes_recovered":189413266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"27d","length":"1"},{"start":"297","length":"1"},{"start":"2ab","length":"1"},{"start":"2ad","length":"1"},{"start":"2c4","length":"1"}]},{"pgid":"5.1","version":"0'0","reported_seq":"769","reported_epoch":"832","state":"active+clean","last_fresh":"2021-03-24T22:06:42.469540+0000","last_change":"2021-03-24T22:03:00.472146+0000","last_active":"2021-03-24T22:06:42.469540+0000","last_peered":"2021-03-24T22:06:42.469540+0000","last_clean":"2021-03-24T22:06:42.469540+0000","last_became_active":"2021-03-24T22:01:57.305430+0000","last_became_peered":"2021-03-24T22:01:57.305430+0000","last_unstale":"2021-03-24T22:06:42.469540+0000","last_undegraded":"2021-03-24T22:06:42.469540+0000","last_fullsized":"2021-03-24T22:06:42.469540+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:00.471949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:25.220436+0000","last_clean_scrub_stamp":"2021-03-24T22:03:00.471949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"544'34","reported_seq":"1002","reported_epoch":"832","state":"active+clean","last_fresh":"2021-03-24T22:06:42.468371+0000","last_change":"2021-03-24T22:04:30.205418+0000","last_active":"2021-03-24T22:06:42.468371+0000","last_peered":"2021-03-24T22:06:42.468371+0000","last_clean":"2021-03-24T22:06:42.468371+0000","last_became_active":"2021-03-24T22:01:57.309916+0000","last_became_peered":"2021-03-24T22:01:57.309916+0000","last_unstale":"2021-03-24T22:06:42.468371+0000","last_undegraded":"2021-03-24T22:06:42.468371+0000","last_fullsized":"2021-03-24T22:06:42.468371+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'34","last_scrub_stamp":"2021-03-24T22:04:30.205247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:04:30.205247+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":64,"num_read_kb":44,"num_write":36,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":3384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"851","reported_epoch":"832","state":"active+clean","last_fresh":"2021-03-24T22:06:42.475865+0000","last_change":"2021-03-24T22:06:06.879783+0000","last_active":"2021-03-24T22:06:42.475865+0000","last_peered":"2021-03-24T22:06:42.475865+0000","last_clean":"2021-03-24T22:06:42.475865+0000","last_became_active":"2021-03-24T22:01:57.314345+0000","last_became_peered":"2021-03-24T22:01:57.314345+0000","last_unstale":"2021-03-24T22:06:42.475865+0000","last_undegraded":"2021-03-24T22:06:42.475865+0000","last_fullsized":"2021-03-24T22:06:42.475865+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:06.879633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:06:06.879633+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"368","reported_epoch":"832","state":"active+clean","last_fresh":"2021-03-24T22:06:42.469714+0000","last_change":"2021-03-24T22:03:45.222962+0000","last_active":"2021-03-24T22:06:42.469714+0000","last_peered":"2021-03-24T22:06:42.469714+0000","last_clean":"2021-03-24T22:06:42.469714+0000","last_became_active":"2021-03-24T22:00:49.305612+0000","last_became_peered":"2021-03-24T22:00:49.305612+0000","last_unstale":"2021-03-24T22:06:42.469714+0000","last_undegraded":"2021-03-24T22:06:42.469714+0000","last_fullsized":"2021-03-24T22:06:42.469714+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:45.222743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:03:45.222743+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"831'864","reported_seq":"3128","reported_epoch":"832","state":"active+clean","last_fresh":"2021-03-24T22:06:42.477610+0000","last_change":"2021-03-24T22:06:41.486582+0000","last_active":"2021-03-24T22:06:42.477610+0000","last_peered":"2021-03-24T22:06:42.477610+0000","last_clean":"2021-03-24T22:06:42.477610+0000","last_became_active":"2021-03-24T21:57:26.169302+0000","last_became_peered":"2021-03-24T21:57:26.169302+0000","last_unstale":"2021-03-24T22:06:42.477610+0000","last_undegraded":"2021-03-24T22:06:42.477610+0000","last_fullsized":"2021-03-24T22:06:42.477610+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'770","last_scrub_stamp":"2021-03-24T22:04:57.253408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:04:57.253408+0000","log_size":864,"ondisk_log_size":864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15699091,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":82,"num_read_kb":64068,"num_write":85,"num_write_kb":35091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"255","length":"1"},{"start":"27d","length":"1"},{"start":"297","length":"1"},{"start":"2ab","length":"1"},{"start":"2ad","length":"1"},{"start":"2c4","length":"1"}]},{"pgid":"5.0","version":"658'6","reported_seq":"1028","reported_epoch":"832","state":"active+clean","last_fresh":"2021-03-24T22:06:42.468277+0000","last_change":"2021-03-24T22:05:13.207391+0000","last_active":"2021-03-24T22:06:42.468277+0000","last_peered":"2021-03-24T22:06:42.468277+0000","last_clean":"2021-03-24T22:06:42.468277+0000","last_became_active":"2021-03-24T22:03:45.432883+0000","last_became_peered":"2021-03-24T22:03:45.432883+0000","last_unstale":"2021-03-24T22:06:42.468277+0000","last_undegraded":"2021-03-24T22:06:42.468277+0000","last_fullsized":"2021-03-24T22:06:42.468277+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"658'6","last_scrub_stamp":"2021-03-24T22:05:13.207196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:05:13.207196+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"544'34","reported_seq":"381","reported_epoch":"830","state":"active+clean","last_fresh":"2021-03-24T22:06:40.459551+0000","last_change":"2021-03-24T22:05:44.265489+0000","last_active":"2021-03-24T22:06:40.459551+0000","last_peered":"2021-03-24T22:06:40.459551+0000","last_clean":"2021-03-24T22:06:40.459551+0000","last_became_active":"2021-03-24T22:05:04.433178+0000","last_became_peered":"2021-03-24T22:05:04.433178+0000","last_unstale":"2021-03-24T22:06:40.459551+0000","last_undegraded":"2021-03-24T22:06:40.459551+0000","last_fullsized":"2021-03-24T22:06:40.459551+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'34","last_scrub_stamp":"2021-03-24T22:05:44.265284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:05:44.265284+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":58,"num_read_kb":39,"num_write":34,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":940,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"866","reported_epoch":"831","state":"active+clean","last_fresh":"2021-03-24T22:06:41.458328+0000","last_change":"2021-03-24T22:06:06.448287+0000","last_active":"2021-03-24T22:06:41.458328+0000","last_peered":"2021-03-24T22:06:41.458328+0000","last_clean":"2021-03-24T22:06:41.458328+0000","last_became_active":"2021-03-24T22:06:06.447793+0000","last_became_peered":"2021-03-24T22:06:06.447793+0000","last_unstale":"2021-03-24T22:06:41.458328+0000","last_undegraded":"2021-03-24T22:06:41.458328+0000","last_fullsized":"2021-03-24T22:06:41.458328+0000","mapping_epoch":795,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":796,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:00.436502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:06:00.436502+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"302","reported_epoch":"831","state":"active+clean","last_fresh":"2021-03-24T22:06:41.467993+0000","last_change":"2021-03-24T22:06:36.049879+0000","last_active":"2021-03-24T22:06:41.467993+0000","last_peered":"2021-03-24T22:06:41.467993+0000","last_clean":"2021-03-24T22:06:41.467993+0000","last_became_active":"2021-03-24T22:01:57.316610+0000","last_became_peered":"2021-03-24T22:01:57.316610+0000","last_unstale":"2021-03-24T22:06:41.467993+0000","last_undegraded":"2021-03-24T22:06:41.467993+0000","last_fullsized":"2021-03-24T22:06:41.467993+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:36.049669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:06:36.049669+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"272'64","reported_seq":"2354","reported_epoch":"832","state":"active+clean","last_fresh":"2021-03-24T22:06:42.469417+0000","last_change":"2021-03-24T22:06:41.484672+0000","last_active":"2021-03-24T22:06:42.469417+0000","last_peered":"2021-03-24T22:06:42.469417+0000","last_clean":"2021-03-24T22:06:42.469417+0000","last_became_active":"2021-03-24T22:03:45.428845+0000","last_became_peered":"2021-03-24T22:03:45.428845+0000","last_unstale":"2021-03-24T22:06:42.469417+0000","last_undegraded":"2021-03-24T22:06:42.469417+0000","last_fullsized":"2021-03-24T22:06:42.469417+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'64","last_scrub_stamp":"2021-03-24T22:05:03.168158+0000","last_deep_scrub":"272'64","last_deep_scrub_stamp":"2021-03-24T21:59:08.406188+0000","last_clean_scrub_stamp":"2021-03-24T22:05:03.168158+0000","log_size":64,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"255","length":"1"},{"start":"27d","length":"1"},{"start":"297","length":"1"},{"start":"2ab","length":"1"},{"start":"2ad","length":"1"},{"start":"2c4","length":"1"}]},{"pgid":"5.3","version":"734'9","reported_seq":"1032","reported_epoch":"831","state":"active+clean","last_fresh":"2021-03-24T22:06:41.464811+0000","last_change":"2021-03-24T22:05:47.459972+0000","last_active":"2021-03-24T22:06:41.464811+0000","last_peered":"2021-03-24T22:06:41.464811+0000","last_clean":"2021-03-24T22:06:41.464811+0000","last_became_active":"2021-03-24T22:05:04.447618+0000","last_became_peered":"2021-03-24T22:05:04.447618+0000","last_unstale":"2021-03-24T22:06:41.464811+0000","last_undegraded":"2021-03-24T22:06:41.464811+0000","last_fullsized":"2021-03-24T22:06:41.464811+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"734'9","last_scrub_stamp":"2021-03-24T22:05:47.459863+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-24T21:53:26.584678+0000","last_clean_scrub_stamp":"2021-03-24T22:05:47.459863+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"544'26","reported_seq":"228","reported_epoch":"828","state":"active+clean","last_fresh":"2021-03-24T22:06:38.451725+0000","last_change":"2021-03-24T22:05:06.622482+0000","last_active":"2021-03-24T22:06:38.451725+0000","last_peered":"2021-03-24T22:06:38.451725+0000","last_clean":"2021-03-24T22:06:38.451725+0000","last_became_active":"2021-03-24T22:05:04.449351+0000","last_became_peered":"2021-03-24T22:05:04.449351+0000","last_unstale":"2021-03-24T22:06:38.451725+0000","last_undegraded":"2021-03-24T22:06:38.451725+0000","last_fullsized":"2021-03-24T22:06:38.451725+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'26","last_scrub_stamp":"2021-03-24T22:05:06.622374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:05:06.622374+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":46,"num_read_kb":29,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"624","reported_epoch":"830","state":"active+clean","last_fresh":"2021-03-24T22:06:40.457807+0000","last_change":"2021-03-24T22:06:01.442658+0000","last_active":"2021-03-24T22:06:40.457807+0000","last_peered":"2021-03-24T22:06:40.457807+0000","last_clean":"2021-03-24T22:06:40.457807+0000","last_became_active":"2021-03-24T22:05:04.449973+0000","last_became_peered":"2021-03-24T22:05:04.449973+0000","last_unstale":"2021-03-24T22:06:40.457807+0000","last_undegraded":"2021-03-24T22:06:40.457807+0000","last_fullsized":"2021-03-24T22:06:40.457807+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:01.442569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:06:01.442569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"790","reported_epoch":"831","state":"active+clean","last_fresh":"2021-03-24T22:06:41.457821+0000","last_change":"2021-03-24T22:04:37.121136+0000","last_active":"2021-03-24T22:06:41.457821+0000","last_peered":"2021-03-24T22:06:41.457821+0000","last_clean":"2021-03-24T22:06:41.457821+0000","last_became_active":"2021-03-24T21:51:51.759627+0000","last_became_peered":"2021-03-24T21:51:51.759627+0000","last_unstale":"2021-03-24T22:06:41.457821+0000","last_undegraded":"2021-03-24T22:06:41.457821+0000","last_fullsized":"2021-03-24T22:06:41.457821+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:04:37.120859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:04:37.120859+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"830'573","reported_seq":"2958","reported_epoch":"832","state":"active+clean","last_fresh":"2021-03-24T22:06:42.468167+0000","last_change":"2021-03-24T22:06:41.490010+0000","last_active":"2021-03-24T22:06:42.468167+0000","last_peered":"2021-03-24T22:06:42.468167+0000","last_clean":"2021-03-24T22:06:42.468167+0000","last_became_active":"2021-03-24T22:02:30.322922+0000","last_became_peered":"2021-03-24T22:02:30.322922+0000","last_unstale":"2021-03-24T22:06:42.468167+0000","last_undegraded":"2021-03-24T22:06:42.468167+0000","last_fullsized":"2021-03-24T22:06:42.468167+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"695'455","last_scrub_stamp":"2021-03-24T22:04:28.210940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:04:28.210940+0000","log_size":573,"ondisk_log_size":573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13630574,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":82,"num_read_kb":100739,"num_write":52,"num_write_kb":15980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":24969090,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"255","length":"1"},{"start":"27d","length":"1"},{"start":"297","length":"1"},{"start":"2ab","length":"1"},{"start":"2ad","length":"1"},{"start":"2c4","length":"1"}]},{"pgid":"5.2","version":"0'0","reported_seq":"321","reported_epoch":"829","state":"active+clean","last_fresh":"2021-03-24T22:06:39.460761+0000","last_change":"2021-03-24T22:05:06.638365+0000","last_active":"2021-03-24T22:06:39.460761+0000","last_peered":"2021-03-24T22:06:39.460761+0000","last_clean":"2021-03-24T22:06:39.460761+0000","last_became_active":"2021-03-24T22:05:04.444370+0000","last_became_peered":"2021-03-24T22:05:04.444370+0000","last_unstale":"2021-03-24T22:06:39.460761+0000","last_undegraded":"2021-03-24T22:06:39.460761+0000","last_fullsized":"2021-03-24T22:06:39.460761+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:06.638226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:05:06.638226+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"544'31","reported_seq":"1010","reported_epoch":"831","state":"active+clean","last_fresh":"2021-03-24T22:06:41.464940+0000","last_change":"2021-03-24T22:05:52.398909+0000","last_active":"2021-03-24T22:06:41.464940+0000","last_peered":"2021-03-24T22:06:41.464940+0000","last_clean":"2021-03-24T22:06:41.464940+0000","last_became_active":"2021-03-24T22:05:04.450428+0000","last_became_peered":"2021-03-24T22:05:04.450428+0000","last_unstale":"2021-03-24T22:06:41.464940+0000","last_undegraded":"2021-03-24T22:06:41.464940+0000","last_fullsized":"2021-03-24T22:06:41.464940+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'31","last_scrub_stamp":"2021-03-24T22:05:52.398731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:05:52.398731+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":60,"num_read_kb":42,"num_write":34,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":2568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"204","reported_epoch":"828","state":"active+clean","last_fresh":"2021-03-24T22:06:38.450239+0000","last_change":"2021-03-24T22:06:02.509466+0000","last_active":"2021-03-24T22:06:38.450239+0000","last_peered":"2021-03-24T22:06:38.450239+0000","last_clean":"2021-03-24T22:06:38.450239+0000","last_became_active":"2021-03-24T22:05:04.453406+0000","last_became_peered":"2021-03-24T22:05:04.453406+0000","last_unstale":"2021-03-24T22:06:38.450239+0000","last_undegraded":"2021-03-24T22:06:38.450239+0000","last_fullsized":"2021-03-24T22:06:38.450239+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:02.509401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:06:02.509401+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"189","reported_epoch":"828","state":"active+clean","last_fresh":"2021-03-24T22:06:38.450633+0000","last_change":"2021-03-24T22:04:56.575839+0000","last_active":"2021-03-24T22:06:38.450633+0000","last_peered":"2021-03-24T22:06:38.450633+0000","last_clean":"2021-03-24T22:06:38.450633+0000","last_became_active":"2021-03-24T22:03:45.426809+0000","last_became_peered":"2021-03-24T22:03:45.426809+0000","last_unstale":"2021-03-24T22:06:38.450633+0000","last_undegraded":"2021-03-24T22:06:38.450633+0000","last_fullsized":"2021-03-24T22:06:38.450633+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:04:56.575723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:04:56.575723+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"175'22","reported_seq":"2149","reported_epoch":"831","state":"active+clean","last_fresh":"2021-03-24T22:06:41.477727+0000","last_change":"2021-03-24T22:06:41.478268+0000","last_active":"2021-03-24T22:06:41.477727+0000","last_peered":"2021-03-24T22:06:41.477727+0000","last_clean":"2021-03-24T22:06:41.477727+0000","last_became_active":"2021-03-24T22:05:04.443450+0000","last_became_peered":"2021-03-24T22:05:04.443450+0000","last_unstale":"2021-03-24T22:06:41.477727+0000","last_undegraded":"2021-03-24T22:06:41.477727+0000","last_fullsized":"2021-03-24T22:06:41.477727+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'22","last_scrub_stamp":"2021-03-24T22:05:53.363157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:05:53.363157+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"255","length":"1"},{"start":"27d","length":"1"},{"start":"297","length":"1"},{"start":"2ab","length":"1"},{"start":"2ad","length":"1"},{"start":"2c4","length":"1"}]},{"pgid":"5.7","version":"0'0","reported_seq":"669","reported_epoch":"830","state":"active+clean","last_fresh":"2021-03-24T22:06:41.330786+0000","last_change":"2021-03-24T22:06:41.330687+0000","last_active":"2021-03-24T22:06:41.330786+0000","last_peered":"2021-03-24T22:06:41.330786+0000","last_clean":"2021-03-24T22:06:41.330786+0000","last_became_active":"2021-03-24T22:05:04.439506+0000","last_became_peered":"2021-03-24T22:05:04.439506+0000","last_unstale":"2021-03-24T22:06:41.330786+0000","last_undegraded":"2021-03-24T22:06:41.330786+0000","last_fullsized":"2021-03-24T22:06:41.330786+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:41.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:06:41.330509+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"544'34","reported_seq":"1335","reported_epoch":"831","state":"active+clean","last_fresh":"2021-03-24T22:06:41.458232+0000","last_change":"2021-03-24T22:06:40.571163+0000","last_active":"2021-03-24T22:06:41.458232+0000","last_peered":"2021-03-24T22:06:41.458232+0000","last_clean":"2021-03-24T22:06:41.458232+0000","last_became_active":"2021-03-24T22:03:54.401857+0000","last_became_peered":"2021-03-24T22:03:54.401857+0000","last_unstale":"2021-03-24T22:06:41.458232+0000","last_undegraded":"2021-03-24T22:06:41.458232+0000","last_fullsized":"2021-03-24T22:06:41.458232+0000","mapping_epoch":663,"log_start":"69'16","ondisk_log_start":"69'16","created":62,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'34","last_scrub_stamp":"2021-03-24T22:06:40.570957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:06:40.570957+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1836,"num_objects":21,"num_object_clones":0,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":95,"num_read_kb":61,"num_write":51,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":157,"num_bytes_recovered":13482,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"66'5","reported_seq":"613","reported_epoch":"832","state":"active+clean","last_fresh":"2021-03-24T22:06:42.765423+0000","last_change":"2021-03-24T22:04:30.329098+0000","last_active":"2021-03-24T22:06:42.765423+0000","last_peered":"2021-03-24T22:06:42.765423+0000","last_clean":"2021-03-24T22:06:42.765423+0000","last_became_active":"2021-03-24T22:03:45.433742+0000","last_became_peered":"2021-03-24T22:03:45.433742+0000","last_unstale":"2021-03-24T22:06:42.765423+0000","last_undegraded":"2021-03-24T22:06:42.765423+0000","last_fullsized":"2021-03-24T22:06:42.765423+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2021-03-24T22:04:30.328893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:39.286101+0000","last_clean_scrub_stamp":"2021-03-24T22:04:30.328893+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":358,"num_read_kb":353,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"810","reported_epoch":"831","state":"active+clean","last_fresh":"2021-03-24T22:06:41.459239+0000","last_change":"2021-03-24T22:04:39.179364+0000","last_active":"2021-03-24T22:06:41.459239+0000","last_peered":"2021-03-24T22:06:41.459239+0000","last_clean":"2021-03-24T22:06:41.459239+0000","last_became_active":"2021-03-24T21:59:12.281282+0000","last_became_peered":"2021-03-24T21:59:12.281282+0000","last_unstale":"2021-03-24T22:06:41.459239+0000","last_undegraded":"2021-03-24T22:06:41.459239+0000","last_fullsized":"2021-03-24T22:06:41.459239+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:04:39.179121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:04:39.179121+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"189","reported_epoch":"828","state":"active+clean","last_fresh":"2021-03-24T22:06:38.450461+0000","last_change":"2021-03-24T22:05:44.537883+0000","last_active":"2021-03-24T22:06:38.450461+0000","last_peered":"2021-03-24T22:06:38.450461+0000","last_clean":"2021-03-24T22:06:38.450461+0000","last_became_active":"2021-03-24T22:03:45.424562+0000","last_became_peered":"2021-03-24T22:03:45.424562+0000","last_unstale":"2021-03-24T22:06:38.450461+0000","last_undegraded":"2021-03-24T22:06:38.450461+0000","last_fullsized":"2021-03-24T22:06:38.450461+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:44.537809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:05:44.537809+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"195'69","reported_seq":"908","reported_epoch":"831","state":"active+clean","last_fresh":"2021-03-24T22:06:41.475133+0000","last_change":"2021-03-24T22:06:41.475133+0000","last_active":"2021-03-24T22:06:41.475133+0000","last_peered":"2021-03-24T22:06:41.475133+0000","last_clean":"2021-03-24T22:06:41.475133+0000","last_became_active":"2021-03-24T22:01:57.317669+0000","last_became_peered":"2021-03-24T22:01:57.317669+0000","last_unstale":"2021-03-24T22:06:41.475133+0000","last_undegraded":"2021-03-24T22:06:41.475133+0000","last_fullsized":"2021-03-24T22:06:41.475133+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'69","last_scrub_stamp":"2021-03-24T22:04:58.198525+0000","last_deep_scrub":"195'69","last_deep_scrub_stamp":"2021-03-24T21:56:49.886058+0000","last_clean_scrub_stamp":"2021-03-24T22:04:58.198525+0000","log_size":69,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"255","length":"1"},{"start":"27d","length":"1"},{"start":"297","length":"1"},{"start":"2ab","length":"1"},{"start":"2ad","length":"1"},{"start":"2c4","length":"1"}]},{"pgid":"5.4","version":"0'0","reported_seq":"319","reported_epoch":"831","state":"active+clean","last_fresh":"2021-03-24T22:06:41.468727+0000","last_change":"2021-03-24T22:05:14.123612+0000","last_active":"2021-03-24T22:06:41.468727+0000","last_peered":"2021-03-24T22:06:41.468727+0000","last_clean":"2021-03-24T22:06:41.468727+0000","last_became_active":"2021-03-24T22:05:04.451670+0000","last_became_peered":"2021-03-24T22:05:04.451670+0000","last_unstale":"2021-03-24T22:06:41.468727+0000","last_undegraded":"2021-03-24T22:06:41.468727+0000","last_fullsized":"2021-03-24T22:06:41.468727+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:14.123439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:05:14.123439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"783'36","reported_seq":"387","reported_epoch":"829","state":"active+clean","last_fresh":"2021-03-24T22:06:39.462008+0000","last_change":"2021-03-24T22:05:54.205230+0000","last_active":"2021-03-24T22:06:39.462008+0000","last_peered":"2021-03-24T22:06:39.462008+0000","last_clean":"2021-03-24T22:06:39.462008+0000","last_became_active":"2021-03-24T22:01:57.311205+0000","last_became_peered":"2021-03-24T22:01:57.311205+0000","last_unstale":"2021-03-24T22:06:39.462008+0000","last_undegraded":"2021-03-24T22:06:39.462008+0000","last_fullsized":"2021-03-24T22:06:39.462008+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"783'36","last_scrub_stamp":"2021-03-24T22:05:54.205108+0000","last_deep_scrub":"783'36","last_deep_scrub_stamp":"2021-03-24T22:05:54.205108+0000","last_clean_scrub_stamp":"2021-03-24T22:05:54.205108+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":27,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":1398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"839","reported_epoch":"832","state":"active+clean","last_fresh":"2021-03-24T22:06:42.469535+0000","last_change":"2021-03-24T22:05:04.443282+0000","last_active":"2021-03-24T22:06:42.469535+0000","last_peered":"2021-03-24T22:06:42.469535+0000","last_clean":"2021-03-24T22:06:42.469535+0000","last_became_active":"2021-03-24T22:05:04.441717+0000","last_became_peered":"2021-03-24T22:05:04.441717+0000","last_unstale":"2021-03-24T22:06:42.469535+0000","last_undegraded":"2021-03-24T22:06:42.469535+0000","last_fullsized":"2021-03-24T22:06:42.469535+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:33.533840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:33.533840+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"302","reported_epoch":"831","state":"active+clean","last_fresh":"2021-03-24T22:06:41.467103+0000","last_change":"2021-03-24T22:05:59.171885+0000","last_active":"2021-03-24T22:06:41.467103+0000","last_peered":"2021-03-24T22:06:41.467103+0000","last_clean":"2021-03-24T22:06:41.467103+0000","last_became_active":"2021-03-24T22:01:57.315729+0000","last_became_peered":"2021-03-24T22:01:57.315729+0000","last_unstale":"2021-03-24T22:06:41.467103+0000","last_undegraded":"2021-03-24T22:06:41.467103+0000","last_fullsized":"2021-03-24T22:06:41.467103+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:59.171797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:49:51.529051+0000","last_clean_scrub_stamp":"2021-03-24T22:05:59.171797+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"616","reported_epoch":"830","state":"active+clean","last_fresh":"2021-03-24T22:06:40.461132+0000","last_change":"2021-03-24T22:05:49.381846+0000","last_active":"2021-03-24T22:06:40.461132+0000","last_peered":"2021-03-24T22:06:40.461132+0000","last_clean":"2021-03-24T22:06:40.461132+0000","last_became_active":"2021-03-24T22:05:04.435232+0000","last_became_peered":"2021-03-24T22:05:04.435232+0000","last_unstale":"2021-03-24T22:06:40.461132+0000","last_undegraded":"2021-03-24T22:06:40.461132+0000","last_fullsized":"2021-03-24T22:06:40.461132+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:49.381569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:05:49.381569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"831'449","reported_seq":"3036","reported_epoch":"831","state":"active+clean","last_fresh":"2021-03-24T22:06:41.474264+0000","last_change":"2021-03-24T22:06:41.474264+0000","last_active":"2021-03-24T22:06:41.474264+0000","last_peered":"2021-03-24T22:06:41.474264+0000","last_clean":"2021-03-24T22:06:41.474264+0000","last_became_active":"2021-03-24T22:02:31.331916+0000","last_became_peered":"2021-03-24T22:02:31.331916+0000","last_unstale":"2021-03-24T22:06:41.474264+0000","last_undegraded":"2021-03-24T22:06:41.474264+0000","last_fullsized":"2021-03-24T22:06:41.474264+0000","mapping_epoch":580,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":581,"parent":"0.0","parent_split_bits":0,"last_scrub":"783'396","last_scrub_stamp":"2021-03-24T22:05:56.358947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:05:56.358947+0000","log_size":449,"ondisk_log_size":449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19100161,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":14,"num_read_kb":16743,"num_write":10,"num_write_kb":6213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"255","length":"1"},{"start":"27d","length":"1"},{"start":"297","length":"1"},{"start":"2ab","length":"1"},{"start":"2ad","length":"1"},{"start":"2c4","length":"1"}]},{"pgid":"5.6","version":"0'0","reported_seq":"291","reported_epoch":"830","state":"active+clean","last_fresh":"2021-03-24T22:06:40.459111+0000","last_change":"2021-03-24T22:05:51.038309+0000","last_active":"2021-03-24T22:06:40.459111+0000","last_peered":"2021-03-24T22:06:40.459111+0000","last_clean":"2021-03-24T22:06:40.459111+0000","last_became_active":"2021-03-24T22:05:04.423152+0000","last_became_peered":"2021-03-24T22:05:04.423152+0000","last_unstale":"2021-03-24T22:06:40.459111+0000","last_undegraded":"2021-03-24T22:06:40.459111+0000","last_fullsized":"2021-03-24T22:06:40.459111+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:51.038089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:05:51.038089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"544'32","reported_seq":"592","reported_epoch":"830","state":"active+clean","last_fresh":"2021-03-24T22:06:40.470125+0000","last_change":"2021-03-24T22:06:40.463408+0000","last_active":"2021-03-24T22:06:40.470125+0000","last_peered":"2021-03-24T22:06:40.470125+0000","last_clean":"2021-03-24T22:06:40.470125+0000","last_became_active":"2021-03-24T22:06:40.463022+0000","last_became_peered":"2021-03-24T22:06:40.463022+0000","last_unstale":"2021-03-24T22:06:40.470125+0000","last_undegraded":"2021-03-24T22:06:40.470125+0000","last_fullsized":"2021-03-24T22:06:40.470125+0000","mapping_epoch":829,"log_start":"89'22","ondisk_log_start":"89'22","created":62,"last_epoch_clean":830,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'32","last_scrub_stamp":"2021-03-24T22:05:13.446721+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2021-03-24T21:53:19.717244+0000","last_clean_scrub_stamp":"2021-03-24T22:05:13.446721+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":802,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":42,"num_write":35,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"816","reported_epoch":"830","state":"active+clean","last_fresh":"2021-03-24T22:06:40.450519+0000","last_change":"2021-03-24T22:06:37.118363+0000","last_active":"2021-03-24T22:06:40.450519+0000","last_peered":"2021-03-24T22:06:40.450519+0000","last_clean":"2021-03-24T22:06:40.450519+0000","last_became_active":"2021-03-24T21:51:40.351000+0000","last_became_peered":"2021-03-24T21:51:40.351000+0000","last_unstale":"2021-03-24T22:06:40.450519+0000","last_undegraded":"2021-03-24T22:06:40.450519+0000","last_fullsized":"2021-03-24T22:06:40.450519+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:37.118211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:39.286101+0000","last_clean_scrub_stamp":"2021-03-24T22:06:37.118211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"61'1","reported_seq":"189","reported_epoch":"828","state":"active+clean","last_fresh":"2021-03-24T22:06:38.451102+0000","last_change":"2021-03-24T22:04:34.667264+0000","last_active":"2021-03-24T22:06:38.451102+0000","last_peered":"2021-03-24T22:06:38.451102+0000","last_clean":"2021-03-24T22:06:38.451102+0000","last_became_active":"2021-03-24T22:03:45.426685+0000","last_became_peered":"2021-03-24T22:03:45.426685+0000","last_unstale":"2021-03-24T22:06:38.451102+0000","last_undegraded":"2021-03-24T22:06:38.451102+0000","last_fullsized":"2021-03-24T22:06:38.451102+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2021-03-24T22:04:34.667176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:04:34.667176+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"302","reported_epoch":"831","state":"active+clean","last_fresh":"2021-03-24T22:06:41.470015+0000","last_change":"2021-03-24T22:05:50.105714+0000","last_active":"2021-03-24T22:06:41.470015+0000","last_peered":"2021-03-24T22:06:41.470015+0000","last_clean":"2021-03-24T22:06:41.470015+0000","last_became_active":"2021-03-24T22:01:57.318275+0000","last_became_peered":"2021-03-24T22:01:57.318275+0000","last_unstale":"2021-03-24T22:06:41.470015+0000","last_undegraded":"2021-03-24T22:06:41.470015+0000","last_fullsized":"2021-03-24T22:06:41.470015+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:50.105616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:05:50.105616+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"831'493","reported_seq":"2993","reported_epoch":"831","state":"active+clean","last_fresh":"2021-03-24T22:06:41.587628+0000","last_change":"2021-03-24T22:06:41.587582+0000","last_active":"2021-03-24T22:06:41.587628+0000","last_peered":"2021-03-24T22:06:41.587628+0000","last_clean":"2021-03-24T22:06:41.587628+0000","last_became_active":"2021-03-24T22:03:50.369486+0000","last_became_peered":"2021-03-24T22:03:50.369486+0000","last_unstale":"2021-03-24T22:06:41.587628+0000","last_undegraded":"2021-03-24T22:06:41.587628+0000","last_fullsized":"2021-03-24T22:06:41.587628+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"831'493","last_scrub_stamp":"2021-03-24T22:06:41.587466+0000","last_deep_scrub":"616'351","last_deep_scrub_stamp":"2021-03-24T22:03:06.473123+0000","last_clean_scrub_stamp":"2021-03-24T22:06:41.587466+0000","log_size":493,"ondisk_log_size":493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9513521,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":96,"num_read_kb":102337,"num_write":61,"num_write_kb":23603,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":35984561,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"255","length":"1"},{"start":"27d","length":"1"},{"start":"297","length":"1"},{"start":"2ab","length":"1"},{"start":"2ad","length":"1"},{"start":"2c4","length":"1"}]},{"pgid":"5.5","version":"0'0","reported_seq":"846","reported_epoch":"831","state":"active+clean","last_fresh":"2021-03-24T22:06:41.457971+0000","last_change":"2021-03-24T22:04:27.142762+0000","last_active":"2021-03-24T22:06:41.457971+0000","last_peered":"2021-03-24T22:06:41.457971+0000","last_clean":"2021-03-24T22:06:41.457971+0000","last_became_active":"2021-03-24T22:03:45.430121+0000","last_became_peered":"2021-03-24T22:03:45.430121+0000","last_unstale":"2021-03-24T22:06:41.457971+0000","last_undegraded":"2021-03-24T22:06:41.457971+0000","last_fullsized":"2021-03-24T22:06:41.457971+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:04:27.142661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:04:27.142661+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"544'37","reported_seq":"217","reported_epoch":"828","state":"active+clean","last_fresh":"2021-03-24T22:06:38.450144+0000","last_change":"2021-03-24T22:04:58.618464+0000","last_active":"2021-03-24T22:06:38.450144+0000","last_peered":"2021-03-24T22:06:38.450144+0000","last_clean":"2021-03-24T22:06:38.450144+0000","last_became_active":"2021-03-24T22:03:45.426862+0000","last_became_peered":"2021-03-24T22:03:45.426862+0000","last_unstale":"2021-03-24T22:06:38.450144+0000","last_undegraded":"2021-03-24T22:06:38.450144+0000","last_fullsized":"2021-03-24T22:06:38.450144+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'37","last_scrub_stamp":"2021-03-24T22:04:58.618374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:04:58.618374+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":70,"num_read_kb":47,"num_write":40,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":5724,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"89'2","reported_seq":"434","reported_epoch":"831","state":"active+clean","last_fresh":"2021-03-24T22:06:41.465167+0000","last_change":"2021-03-24T22:05:04.320737+0000","last_active":"2021-03-24T22:06:41.465167+0000","last_peered":"2021-03-24T22:06:41.465167+0000","last_clean":"2021-03-24T22:06:41.465167+0000","last_became_active":"2021-03-24T22:01:43.393718+0000","last_became_peered":"2021-03-24T22:01:43.393718+0000","last_unstale":"2021-03-24T22:06:41.465167+0000","last_undegraded":"2021-03-24T22:06:41.465167+0000","last_fullsized":"2021-03-24T22:06:41.465167+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2021-03-24T22:05:04.320667+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2021-03-24T21:57:58.491830+0000","last_clean_scrub_stamp":"2021-03-24T22:05:04.320667+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"204","reported_epoch":"828","state":"active+clean","last_fresh":"2021-03-24T22:06:38.451455+0000","last_change":"2021-03-24T22:05:10.501601+0000","last_active":"2021-03-24T22:06:38.451455+0000","last_peered":"2021-03-24T22:06:38.451455+0000","last_clean":"2021-03-24T22:06:38.451455+0000","last_became_active":"2021-03-24T22:05:04.450107+0000","last_became_peered":"2021-03-24T22:05:04.450107+0000","last_unstale":"2021-03-24T22:06:38.451455+0000","last_undegraded":"2021-03-24T22:06:38.451455+0000","last_fullsized":"2021-03-24T22:06:38.451455+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:10.501490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:06.800417+0000","last_clean_scrub_stamp":"2021-03-24T22:05:10.501490+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"845","reported_epoch":"832","state":"active+clean","last_fresh":"2021-03-24T22:06:42.474009+0000","last_change":"2021-03-24T22:02:55.490903+0000","last_active":"2021-03-24T22:06:42.474009+0000","last_peered":"2021-03-24T22:06:42.474009+0000","last_clean":"2021-03-24T22:06:42.474009+0000","last_became_active":"2021-03-24T22:01:57.314372+0000","last_became_peered":"2021-03-24T22:01:57.314372+0000","last_unstale":"2021-03-24T22:06:42.474009+0000","last_undegraded":"2021-03-24T22:06:42.474009+0000","last_fullsized":"2021-03-24T22:06:42.474009+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:55.490744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:52:59.706511+0000","last_clean_scrub_stamp":"2021-03-24T22:02:55.490744+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"830'630","reported_seq":"3051","reported_epoch":"830","state":"active+clean","last_fresh":"2021-03-24T22:06:40.477469+0000","last_change":"2021-03-24T22:06:40.477940+0000","last_active":"2021-03-24T22:06:40.477469+0000","last_peered":"2021-03-24T22:06:40.477469+0000","last_clean":"2021-03-24T22:06:40.477469+0000","last_became_active":"2021-03-24T22:01:47.305050+0000","last_became_peered":"2021-03-24T22:01:47.305050+0000","last_unstale":"2021-03-24T22:06:40.477469+0000","last_undegraded":"2021-03-24T22:06:40.477469+0000","last_fullsized":"2021-03-24T22:06:40.477469+0000","mapping_epoch":536,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":537,"parent":"0.0","parent_split_bits":0,"last_scrub":"699'500","last_scrub_stamp":"2021-03-24T22:04:31.592343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:04:31.592343+0000","log_size":630,"ondisk_log_size":630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7767212,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":7099,"num_write":67,"num_write_kb":23311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":55849492,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"27d","length":"1"},{"start":"297","length":"1"},{"start":"2ab","length":"1"},{"start":"2ad","length":"1"},{"start":"2c4","length":"1"}]},{"pgid":"4.b","version":"544'17","reported_seq":"952","reported_epoch":"831","state":"active+clean","last_fresh":"2021-03-24T22:06:41.465275+0000","last_change":"2021-03-24T22:04:34.106367+0000","last_active":"2021-03-24T22:06:41.465275+0000","last_peered":"2021-03-24T22:06:41.465275+0000","last_clean":"2021-03-24T22:06:41.465275+0000","last_became_active":"2021-03-24T22:03:46.443398+0000","last_became_peered":"2021-03-24T22:03:46.443398+0000","last_unstale":"2021-03-24T22:06:41.465275+0000","last_undegraded":"2021-03-24T22:06:41.465275+0000","last_fullsized":"2021-03-24T22:06:41.465275+0000","mapping_epoch":655,"log_start":"91'9","ondisk_log_start":"91'9","created":62,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'17","last_scrub_stamp":"2021-03-24T22:04:34.106163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:04:34.106163+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1158,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":58,"num_read_kb":36,"num_write":30,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2316,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"840","reported_epoch":"832","state":"active+clean","last_fresh":"2021-03-24T22:06:42.473681+0000","last_change":"2021-03-24T22:06:41.077009+0000","last_active":"2021-03-24T22:06:42.473681+0000","last_peered":"2021-03-24T22:06:42.473681+0000","last_clean":"2021-03-24T22:06:42.473681+0000","last_became_active":"2021-03-24T22:05:04.446424+0000","last_became_peered":"2021-03-24T22:05:04.446424+0000","last_unstale":"2021-03-24T22:06:42.473681+0000","last_undegraded":"2021-03-24T22:06:42.473681+0000","last_fullsized":"2021-03-24T22:06:42.473681+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:41.076877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T22:01:55.431129+0000","last_clean_scrub_stamp":"2021-03-24T22:06:41.076877+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"803","reported_epoch":"830","state":"active+clean","last_fresh":"2021-03-24T22:06:40.449815+0000","last_change":"2021-03-24T22:06:01.013669+0000","last_active":"2021-03-24T22:06:40.449815+0000","last_peered":"2021-03-24T22:06:40.449815+0000","last_clean":"2021-03-24T22:06:40.449815+0000","last_became_active":"2021-03-24T21:51:49.411921+0000","last_became_peered":"2021-03-24T21:51:49.411921+0000","last_unstale":"2021-03-24T22:06:40.449815+0000","last_undegraded":"2021-03-24T22:06:40.449815+0000","last_fullsized":"2021-03-24T22:06:40.449815+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:01.013563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:06:01.013563+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"853","reported_epoch":"831","state":"active+clean","last_fresh":"2021-03-24T22:06:41.459020+0000","last_change":"2021-03-24T22:04:31.107367+0000","last_active":"2021-03-24T22:06:41.459020+0000","last_peered":"2021-03-24T22:06:41.459020+0000","last_clean":"2021-03-24T22:06:41.459020+0000","last_became_active":"2021-03-24T22:02:22.323876+0000","last_became_peered":"2021-03-24T22:02:22.323876+0000","last_unstale":"2021-03-24T22:06:41.459020+0000","last_undegraded":"2021-03-24T22:06:41.459020+0000","last_fullsized":"2021-03-24T22:06:41.459020+0000","mapping_epoch":571,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:04:31.107254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:35.636299+0000","last_clean_scrub_stamp":"2021-03-24T22:04:31.107254+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"821'333","reported_seq":"2780","reported_epoch":"831","state":"active+clean","last_fresh":"2021-03-24T22:06:41.478712+0000","last_change":"2021-03-24T22:06:41.482105+0000","last_active":"2021-03-24T22:06:41.478712+0000","last_peered":"2021-03-24T22:06:41.478712+0000","last_clean":"2021-03-24T22:06:41.478712+0000","last_became_active":"2021-03-24T22:02:01.325483+0000","last_became_peered":"2021-03-24T22:02:01.325483+0000","last_unstale":"2021-03-24T22:06:41.478712+0000","last_undegraded":"2021-03-24T22:06:41.478712+0000","last_fullsized":"2021-03-24T22:06:41.478712+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"792'330","last_scrub_stamp":"2021-03-24T22:06:03.024091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:06:03.024091+0000","log_size":333,"ondisk_log_size":333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5244246,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":87,"num_read_kb":99622,"num_write":63,"num_write_kb":17849,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":21446378,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"255","length":"1"},{"start":"27d","length":"1"},{"start":"297","length":"1"},{"start":"2ab","length":"1"},{"start":"2ad","length":"1"},{"start":"2c4","length":"1"}]},{"pgid":"4.a","version":"544'45","reported_seq":"796","reported_epoch":"829","state":"active+clean","last_fresh":"2021-03-24T22:06:39.460809+0000","last_change":"2021-03-24T22:04:26.640197+0000","last_active":"2021-03-24T22:06:39.460809+0000","last_peered":"2021-03-24T22:06:39.460809+0000","last_clean":"2021-03-24T22:06:39.460809+0000","last_became_active":"2021-03-24T22:04:26.561945+0000","last_became_peered":"2021-03-24T22:04:26.561945+0000","last_unstale":"2021-03-24T22:06:39.460809+0000","last_undegraded":"2021-03-24T22:06:39.460809+0000","last_fullsized":"2021-03-24T22:06:39.460809+0000","mapping_epoch":695,"log_start":"69'23","ondisk_log_start":"69'23","created":62,"last_epoch_clean":696,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'45","last_scrub_stamp":"2021-03-24T22:04:26.640132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:04:26.640132+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1466,"num_objects":21,"num_object_clones":0,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":104,"num_read_kb":71,"num_write":60,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":7174,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":"452","reported_epoch":"830","state":"active+clean","last_fresh":"2021-03-24T22:06:40.460584+0000","last_change":"2021-03-24T22:05:51.326820+0000","last_active":"2021-03-24T22:06:40.460584+0000","last_peered":"2021-03-24T22:06:40.460584+0000","last_clean":"2021-03-24T22:06:40.460584+0000","last_became_active":"2021-03-24T22:05:04.439833+0000","last_became_peered":"2021-03-24T22:05:04.439833+0000","last_unstale":"2021-03-24T22:06:40.460584+0000","last_undegraded":"2021-03-24T22:06:40.460584+0000","last_fullsized":"2021-03-24T22:06:40.460584+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:51.326637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:05:51.326637+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"286","reported_epoch":"830","state":"active+clean","last_fresh":"2021-03-24T22:06:40.458309+0000","last_change":"2021-03-24T22:05:08.670410+0000","last_active":"2021-03-24T22:06:40.458309+0000","last_peered":"2021-03-24T22:06:40.458309+0000","last_clean":"2021-03-24T22:06:40.458309+0000","last_became_active":"2021-03-24T22:05:04.426559+0000","last_became_peered":"2021-03-24T22:05:04.426559+0000","last_unstale":"2021-03-24T22:06:40.458309+0000","last_undegraded":"2021-03-24T22:06:40.458309+0000","last_fullsized":"2021-03-24T22:06:40.458309+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:08.670240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T22:05:08.670240+0000","last_clean_scrub_stamp":"2021-03-24T22:05:08.670240+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"596","reported_epoch":"830","state":"active+clean","last_fresh":"2021-03-24T22:06:40.448582+0000","last_change":"2021-03-24T22:05:55.032235+0000","last_active":"2021-03-24T22:06:40.448582+0000","last_peered":"2021-03-24T22:06:40.448582+0000","last_clean":"2021-03-24T22:06:40.448582+0000","last_became_active":"2021-03-24T22:01:57.311325+0000","last_became_peered":"2021-03-24T22:01:57.311325+0000","last_unstale":"2021-03-24T22:06:40.448582+0000","last_undegraded":"2021-03-24T22:06:40.448582+0000","last_fullsized":"2021-03-24T22:06:40.448582+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:55.032073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:05:55.032073+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"831'554","reported_seq":"3029","reported_epoch":"831","state":"active+clean","last_fresh":"2021-03-24T22:06:41.535213+0000","last_change":"2021-03-24T22:06:41.486938+0000","last_active":"2021-03-24T22:06:41.535213+0000","last_peered":"2021-03-24T22:06:41.535213+0000","last_clean":"2021-03-24T22:06:41.535213+0000","last_became_active":"2021-03-24T22:03:49.375663+0000","last_became_peered":"2021-03-24T22:03:49.375663+0000","last_unstale":"2021-03-24T22:06:41.535213+0000","last_undegraded":"2021-03-24T22:06:41.535213+0000","last_fullsized":"2021-03-24T22:06:41.535213+0000","mapping_epoch":658,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":659,"parent":"0.0","parent_split_bits":0,"last_scrub":"795'538","last_scrub_stamp":"2021-03-24T22:06:05.470009+0000","last_deep_scrub":"795'538","last_deep_scrub_stamp":"2021-03-24T22:06:05.470009+0000","last_clean_scrub_stamp":"2021-03-24T22:06:05.470009+0000","log_size":554,"ondisk_log_size":554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7899950,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":73,"num_read_kb":81143,"num_write":68,"num_write_kb":19581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":32161412,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"255","length":"1"},{"start":"27d","length":"1"},{"start":"297","length":"1"},{"start":"2ab","length":"1"},{"start":"2ad","length":"1"},{"start":"2c4","length":"1"}]},{"pgid":"5.8","version":"0'0","reported_seq":"856","reported_epoch":"830","state":"active+clean","last_fresh":"2021-03-24T22:06:40.448324+0000","last_change":"2021-03-24T22:06:04.053700+0000","last_active":"2021-03-24T22:06:40.448324+0000","last_peered":"2021-03-24T22:06:40.448324+0000","last_clean":"2021-03-24T22:06:40.448324+0000","last_became_active":"2021-03-24T22:03:45.432819+0000","last_became_peered":"2021-03-24T22:03:45.432819+0000","last_unstale":"2021-03-24T22:06:40.448324+0000","last_undegraded":"2021-03-24T22:06:40.448324+0000","last_fullsized":"2021-03-24T22:06:40.448324+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:04.053530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:06:04.053530+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"544'38","reported_seq":"535","reported_epoch":"828","state":"active+clean","last_fresh":"2021-03-24T22:06:38.450481+0000","last_change":"2021-03-24T22:05:00.634636+0000","last_active":"2021-03-24T22:06:38.450481+0000","last_peered":"2021-03-24T22:06:38.450481+0000","last_clean":"2021-03-24T22:06:38.450481+0000","last_became_active":"2021-03-24T22:04:50.402497+0000","last_became_peered":"2021-03-24T22:04:50.402497+0000","last_unstale":"2021-03-24T22:06:38.450481+0000","last_undegraded":"2021-03-24T22:06:38.450481+0000","last_fullsized":"2021-03-24T22:06:38.450481+0000","mapping_epoch":719,"log_start":"69'20","ondisk_log_start":"69'20","created":62,"last_epoch_clean":720,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'38","last_scrub_stamp":"2021-03-24T22:05:00.634536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:05:00.634536+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1582,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":100,"num_read_kb":66,"num_write":56,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":4746,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":"320","reported_epoch":"831","state":"active+clean","last_fresh":"2021-03-24T22:06:41.470621+0000","last_change":"2021-03-24T22:06:37.050021+0000","last_active":"2021-03-24T22:06:41.470621+0000","last_peered":"2021-03-24T22:06:41.470621+0000","last_clean":"2021-03-24T22:06:41.470621+0000","last_became_active":"2021-03-24T22:03:45.437360+0000","last_became_peered":"2021-03-24T22:03:45.437360+0000","last_unstale":"2021-03-24T22:06:41.470621+0000","last_undegraded":"2021-03-24T22:06:41.470621+0000","last_fullsized":"2021-03-24T22:06:41.470621+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:37.049845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:06:37.049845+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"867","reported_epoch":"832","state":"active+clean","last_fresh":"2021-03-24T22:06:42.470225+0000","last_change":"2021-03-24T22:05:14.248048+0000","last_active":"2021-03-24T22:06:42.470225+0000","last_peered":"2021-03-24T22:06:42.470225+0000","last_clean":"2021-03-24T22:06:42.470225+0000","last_became_active":"2021-03-24T22:03:45.436096+0000","last_became_peered":"2021-03-24T22:03:45.436096+0000","last_unstale":"2021-03-24T22:06:42.470225+0000","last_undegraded":"2021-03-24T22:06:42.470225+0000","last_fullsized":"2021-03-24T22:06:42.470225+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:14.247913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:22.724840+0000","last_clean_scrub_stamp":"2021-03-24T22:05:14.247913+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"829'394","reported_seq":"2680","reported_epoch":"832","state":"active+clean","last_fresh":"2021-03-24T22:06:42.469647+0000","last_change":"2021-03-24T22:06:41.486184+0000","last_active":"2021-03-24T22:06:42.469647+0000","last_peered":"2021-03-24T22:06:42.469647+0000","last_clean":"2021-03-24T22:06:42.469647+0000","last_became_active":"2021-03-24T22:05:04.442753+0000","last_became_peered":"2021-03-24T22:05:04.442753+0000","last_unstale":"2021-03-24T22:06:42.469647+0000","last_undegraded":"2021-03-24T22:06:42.469647+0000","last_fullsized":"2021-03-24T22:06:42.469647+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"624'277","last_scrub_stamp":"2021-03-24T22:03:14.428074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:03:14.428074+0000","log_size":394,"ondisk_log_size":394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7500172,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":42,"num_read_kb":48474,"num_write":15,"num_write_kb":7826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1181773,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"255","length":"1"},{"start":"27d","length":"1"},{"start":"297","length":"1"},{"start":"2ab","length":"1"},{"start":"2ad","length":"1"},{"start":"2c4","length":"1"}]},{"pgid":"5.9","version":"548'11","reported_seq":"363","reported_epoch":"832","state":"active+clean","last_fresh":"2021-03-24T22:06:43.019415+0000","last_change":"2021-03-24T22:06:43.019415+0000","last_active":"2021-03-24T22:06:43.019415+0000","last_peered":"2021-03-24T22:06:43.019415+0000","last_clean":"2021-03-24T22:06:43.019415+0000","last_became_active":"2021-03-24T22:01:57.313563+0000","last_became_peered":"2021-03-24T22:01:57.313563+0000","last_unstale":"2021-03-24T22:06:43.019415+0000","last_undegraded":"2021-03-24T22:06:43.019415+0000","last_fullsized":"2021-03-24T22:06:43.019415+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"548'11","last_scrub_stamp":"2021-03-24T22:06:43.019318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:06:43.019318+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"544'46","reported_seq":"811","reported_epoch":"830","state":"active+clean","last_fresh":"2021-03-24T22:06:40.458119+0000","last_change":"2021-03-24T22:06:07.382916+0000","last_active":"2021-03-24T22:06:40.458119+0000","last_peered":"2021-03-24T22:06:40.458119+0000","last_clean":"2021-03-24T22:06:40.458119+0000","last_became_active":"2021-03-24T22:05:05.422281+0000","last_became_peered":"2021-03-24T22:05:05.422281+0000","last_unstale":"2021-03-24T22:06:40.458119+0000","last_undegraded":"2021-03-24T22:06:40.458119+0000","last_fullsized":"2021-03-24T22:06:40.458119+0000","mapping_epoch":734,"log_start":"97'28","ondisk_log_start":"97'28","created":62,"last_epoch_clean":735,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'46","last_scrub_stamp":"2021-03-24T22:06:07.382720+0000","last_deep_scrub":"97'28","last_deep_scrub_stamp":"2021-03-24T21:54:19.157265+0000","last_clean_scrub_stamp":"2021-03-24T22:06:07.382720+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1616,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":96,"num_read_kb":63,"num_write":53,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":5308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"89'4","reported_seq":"308","reported_epoch":"831","state":"active+clean","last_fresh":"2021-03-24T22:06:41.470241+0000","last_change":"2021-03-24T22:06:04.178234+0000","last_active":"2021-03-24T22:06:41.470241+0000","last_peered":"2021-03-24T22:06:41.470241+0000","last_clean":"2021-03-24T22:06:41.470241+0000","last_became_active":"2021-03-24T22:01:57.316325+0000","last_became_peered":"2021-03-24T22:01:57.316325+0000","last_unstale":"2021-03-24T22:06:41.470241+0000","last_undegraded":"2021-03-24T22:06:41.470241+0000","last_fullsized":"2021-03-24T22:06:41.470241+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'4","last_scrub_stamp":"2021-03-24T22:06:04.178049+0000","last_deep_scrub":"89'4","last_deep_scrub_stamp":"2021-03-24T21:53:19.926637+0000","last_clean_scrub_stamp":"2021-03-24T22:06:04.178049+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"326","reported_epoch":"829","state":"active+clean","last_fresh":"2021-03-24T22:06:39.461300+0000","last_change":"2021-03-24T22:05:15.379559+0000","last_active":"2021-03-24T22:06:39.461300+0000","last_peered":"2021-03-24T22:06:39.461300+0000","last_clean":"2021-03-24T22:06:39.461300+0000","last_became_active":"2021-03-24T22:05:04.444421+0000","last_became_peered":"2021-03-24T22:05:04.444421+0000","last_unstale":"2021-03-24T22:06:39.461300+0000","last_undegraded":"2021-03-24T22:06:39.461300+0000","last_fullsized":"2021-03-24T22:06:39.461300+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:15.379478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:05:15.379478+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"272'275","reported_seq":"2141","reported_epoch":"829","state":"active+clean","last_fresh":"2021-03-24T22:06:39.461483+0000","last_change":"2021-03-24T22:06:39.461839+0000","last_active":"2021-03-24T22:06:39.461483+0000","last_peered":"2021-03-24T22:06:39.461483+0000","last_clean":"2021-03-24T22:06:39.461483+0000","last_became_active":"2021-03-24T22:05:04.425270+0000","last_became_peered":"2021-03-24T22:05:04.425270+0000","last_unstale":"2021-03-24T22:06:39.461483+0000","last_undegraded":"2021-03-24T22:06:39.461483+0000","last_fullsized":"2021-03-24T22:06:39.461483+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'275","last_scrub_stamp":"2021-03-24T22:05:59.526944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:05:59.526944+0000","log_size":275,"ondisk_log_size":275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"25e","length":"1"},{"start":"27d","length":"1"},{"start":"28e","length":"1"},{"start":"297","length":"1"},{"start":"2ad","length":"1"},{"start":"2af","length":"1"},{"start":"2c4","length":"1"}]},{"pgid":"3.10","version":"0'0","reported_seq":"328","reported_epoch":"829","state":"active+clean","last_fresh":"2021-03-24T22:06:39.460740+0000","last_change":"2021-03-24T22:05:47.360975+0000","last_active":"2021-03-24T22:06:39.460740+0000","last_peered":"2021-03-24T22:06:39.460740+0000","last_clean":"2021-03-24T22:06:39.460740+0000","last_became_active":"2021-03-24T22:05:04.440682+0000","last_became_peered":"2021-03-24T22:05:04.440682+0000","last_unstale":"2021-03-24T22:06:39.460740+0000","last_undegraded":"2021-03-24T22:06:39.460740+0000","last_fullsized":"2021-03-24T22:06:39.460740+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:47.360795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:05:47.360795+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"680","reported_epoch":"830","state":"active+clean","last_fresh":"2021-03-24T22:06:40.458445+0000","last_change":"2021-03-24T22:06:02.409659+0000","last_active":"2021-03-24T22:06:40.458445+0000","last_peered":"2021-03-24T22:06:40.458445+0000","last_clean":"2021-03-24T22:06:40.458445+0000","last_became_active":"2021-03-24T22:05:04.437651+0000","last_became_peered":"2021-03-24T22:05:04.437651+0000","last_unstale":"2021-03-24T22:06:40.458445+0000","last_undegraded":"2021-03-24T22:06:40.458445+0000","last_fullsized":"2021-03-24T22:06:40.458445+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:02.409388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T22:06:02.409388+0000","last_clean_scrub_stamp":"2021-03-24T22:06:02.409388+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"302","reported_epoch":"831","state":"active+clean","last_fresh":"2021-03-24T22:06:41.469906+0000","last_change":"2021-03-24T22:05:57.160116+0000","last_active":"2021-03-24T22:06:41.469906+0000","last_peered":"2021-03-24T22:06:41.469906+0000","last_clean":"2021-03-24T22:06:41.469906+0000","last_became_active":"2021-03-24T22:01:57.312376+0000","last_became_peered":"2021-03-24T22:01:57.312376+0000","last_unstale":"2021-03-24T22:06:41.469906+0000","last_undegraded":"2021-03-24T22:06:41.469906+0000","last_fullsized":"2021-03-24T22:06:41.469906+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:57.159943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:05:57.159943+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.14","version":"831'1102","reported_seq":"3429","reported_epoch":"832","state":"active+clean","last_fresh":"2021-03-24T22:06:42.478633+0000","last_change":"2021-03-24T22:06:41.495860+0000","last_active":"2021-03-24T22:06:42.478633+0000","last_peered":"2021-03-24T22:06:42.478633+0000","last_clean":"2021-03-24T22:06:42.478633+0000","last_became_active":"2021-03-24T22:05:04.453660+0000","last_became_peered":"2021-03-24T22:05:04.453660+0000","last_unstale":"2021-03-24T22:06:42.478633+0000","last_undegraded":"2021-03-24T22:06:42.478633+0000","last_fullsized":"2021-03-24T22:06:42.478633+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":735,"parent":"0.0","parent_split_bits":5,"last_scrub":"651'772","last_scrub_stamp":"2021-03-24T22:03:42.250616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:03:42.250616+0000","log_size":1102,"ondisk_log_size":1102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":41099914,"num_objects":18,"num_object_clones":15,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":1,"num_read":103,"num_read_kb":113197,"num_write":70,"num_write_kb":32688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":14235619,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"255","length":"1"},{"start":"27d","length":"1"},{"start":"297","length":"1"},{"start":"2ab","length":"1"},{"start":"2ad","length":"1"},{"start":"2c4","length":"1"}]},{"pgid":"3.11","version":"0'0","reported_seq":"320","reported_epoch":"829","state":"active+clean","last_fresh":"2021-03-24T22:06:39.467865+0000","last_change":"2021-03-24T22:04:32.551783+0000","last_active":"2021-03-24T22:06:39.467865+0000","last_peered":"2021-03-24T22:06:39.467865+0000","last_clean":"2021-03-24T22:06:39.467865+0000","last_became_active":"2021-03-24T22:01:57.304423+0000","last_became_peered":"2021-03-24T22:01:57.304423+0000","last_unstale":"2021-03-24T22:06:39.467865+0000","last_undegraded":"2021-03-24T22:06:39.467865+0000","last_fullsized":"2021-03-24T22:06:39.467865+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:04:32.551682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:04:32.551682+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"863","reported_epoch":"832","state":"active+clean","last_fresh":"2021-03-24T22:06:42.476151+0000","last_change":"2021-03-24T22:03:45.435984+0000","last_active":"2021-03-24T22:06:42.476151+0000","last_peered":"2021-03-24T22:06:42.476151+0000","last_clean":"2021-03-24T22:06:42.476151+0000","last_became_active":"2021-03-24T22:03:45.434516+0000","last_became_peered":"2021-03-24T22:03:45.434516+0000","last_unstale":"2021-03-24T22:06:42.476151+0000","last_undegraded":"2021-03-24T22:06:42.476151+0000","last_fullsized":"2021-03-24T22:06:42.476151+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:56.523309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:55.493713+0000","last_clean_scrub_stamp":"2021-03-24T22:02:56.523309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"832","reported_epoch":"830","state":"active+clean","last_fresh":"2021-03-24T22:06:40.458570+0000","last_change":"2021-03-24T22:04:33.896133+0000","last_active":"2021-03-24T22:06:40.458570+0000","last_peered":"2021-03-24T22:06:40.458570+0000","last_clean":"2021-03-24T22:06:40.458570+0000","last_became_active":"2021-03-24T22:01:57.312185+0000","last_became_peered":"2021-03-24T22:01:57.312185+0000","last_unstale":"2021-03-24T22:06:40.458570+0000","last_undegraded":"2021-03-24T22:06:40.458570+0000","last_fullsized":"2021-03-24T22:06:40.458570+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:04:33.895981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:04:33.895981+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"831'425","reported_seq":"2988","reported_epoch":"832","state":"active+clean","last_fresh":"2021-03-24T22:06:42.465165+0000","last_change":"2021-03-24T22:06:41.495214+0000","last_active":"2021-03-24T22:06:42.465165+0000","last_peered":"2021-03-24T22:06:42.465165+0000","last_clean":"2021-03-24T22:06:42.465165+0000","last_became_active":"2021-03-24T22:03:51.363965+0000","last_became_peered":"2021-03-24T22:03:51.363965+0000","last_unstale":"2021-03-24T22:06:42.465165+0000","last_undegraded":"2021-03-24T22:06:42.465165+0000","last_fullsized":"2021-03-24T22:06:42.465165+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":5,"last_scrub":"710'317","last_scrub_stamp":"2021-03-24T22:04:41.171323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:04:41.171323+0000","log_size":425,"ondisk_log_size":425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14082423,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":83,"num_read_kb":115822,"num_write":62,"num_write_kb":23983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":85409227,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"255","length":"1"},{"start":"27d","length":"1"},{"start":"297","length":"1"},{"start":"2ab","length":"1"},{"start":"2ad","length":"1"},{"start":"2c4","length":"1"}]},{"pgid":"4.15","version":"544'19","reported_seq":"753","reported_epoch":"830","state":"active+clean","last_fresh":"2021-03-24T22:06:40.460675+0000","last_change":"2021-03-24T22:06:36.315165+0000","last_active":"2021-03-24T22:06:40.460675+0000","last_peered":"2021-03-24T22:06:40.460675+0000","last_clean":"2021-03-24T22:06:40.460675+0000","last_became_active":"2021-03-24T22:05:04.433341+0000","last_became_peered":"2021-03-24T22:05:04.433341+0000","last_unstale":"2021-03-24T22:06:40.460675+0000","last_undegraded":"2021-03-24T22:06:40.460675+0000","last_fullsized":"2021-03-24T22:06:40.460675+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'19","last_scrub_stamp":"2021-03-24T22:06:36.314962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:06:36.314962+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":36,"num_read_kb":22,"num_write":20,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2202,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":"837","reported_epoch":"831","state":"active+clean","last_fresh":"2021-03-24T22:06:41.457687+0000","last_change":"2021-03-24T22:04:57.279696+0000","last_active":"2021-03-24T22:06:41.457687+0000","last_peered":"2021-03-24T22:06:41.457687+0000","last_clean":"2021-03-24T22:06:41.457687+0000","last_became_active":"2021-03-24T22:01:57.315967+0000","last_became_peered":"2021-03-24T22:01:57.315967+0000","last_unstale":"2021-03-24T22:06:41.457687+0000","last_undegraded":"2021-03-24T22:06:41.457687+0000","last_fullsized":"2021-03-24T22:06:41.457687+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:04:57.279517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:05.473348+0000","last_clean_scrub_stamp":"2021-03-24T22:04:57.279517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"573'12","reported_seq":"1065","reported_epoch":"832","state":"active+clean","last_fresh":"2021-03-24T22:06:42.475778+0000","last_change":"2021-03-24T22:03:02.456113+0000","last_active":"2021-03-24T22:06:42.475778+0000","last_peered":"2021-03-24T22:06:42.475778+0000","last_clean":"2021-03-24T22:06:42.475778+0000","last_became_active":"2021-03-24T22:02:22.326377+0000","last_became_peered":"2021-03-24T22:02:22.326377+0000","last_unstale":"2021-03-24T22:06:42.475778+0000","last_undegraded":"2021-03-24T22:06:42.475778+0000","last_fullsized":"2021-03-24T22:06:42.475778+0000","mapping_epoch":571,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":0,"last_scrub":"573'12","last_scrub_stamp":"2021-03-24T22:03:02.455946+0000","last_deep_scrub":"86'3","last_deep_scrub_stamp":"2021-03-24T21:52:56.678111+0000","last_clean_scrub_stamp":"2021-03-24T22:03:02.455946+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"782","reported_epoch":"830","state":"active+clean","last_fresh":"2021-03-24T22:06:40.457208+0000","last_change":"2021-03-24T22:05:06.701715+0000","last_active":"2021-03-24T22:06:40.457208+0000","last_peered":"2021-03-24T22:06:40.457208+0000","last_clean":"2021-03-24T22:06:40.457208+0000","last_became_active":"2021-03-24T22:05:04.429425+0000","last_became_peered":"2021-03-24T22:05:04.429425+0000","last_unstale":"2021-03-24T22:06:40.457208+0000","last_undegraded":"2021-03-24T22:06:40.457208+0000","last_fullsized":"2021-03-24T22:06:40.457208+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:06.701589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:05:06.701589+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.16","version":"831'557","reported_seq":"3048","reported_epoch":"831","state":"active+clean","last_fresh":"2021-03-24T22:06:41.465863+0000","last_change":"2021-03-24T22:06:41.465863+0000","last_active":"2021-03-24T22:06:41.465863+0000","last_peered":"2021-03-24T22:06:41.465863+0000","last_clean":"2021-03-24T22:06:41.465863+0000","last_became_active":"2021-03-24T22:02:08.322570+0000","last_became_peered":"2021-03-24T22:02:08.322570+0000","last_unstale":"2021-03-24T22:06:41.465863+0000","last_undegraded":"2021-03-24T22:06:41.465863+0000","last_fullsized":"2021-03-24T22:06:41.465863+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"827'555","last_scrub_stamp":"2021-03-24T22:06:39.566799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:06:39.566799+0000","log_size":557,"ondisk_log_size":557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3933036,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":104,"num_read_kb":117115,"num_write":81,"num_write_kb":19657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":30734442,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"255","length":"1"},{"start":"27d","length":"1"},{"start":"297","length":"1"},{"start":"2ab","length":"1"},{"start":"2ad","length":"1"},{"start":"2c4","length":"1"}]},{"pgid":"4.14","version":"544'24","reported_seq":"1077","reported_epoch":"832","state":"active+clean","last_fresh":"2021-03-24T22:06:42.475902+0000","last_change":"2021-03-24T22:04:37.278384+0000","last_active":"2021-03-24T22:06:42.475902+0000","last_peered":"2021-03-24T22:06:42.475902+0000","last_clean":"2021-03-24T22:06:42.475902+0000","last_became_active":"2021-03-24T22:03:45.433725+0000","last_became_peered":"2021-03-24T22:03:45.433725+0000","last_unstale":"2021-03-24T22:06:42.475902+0000","last_undegraded":"2021-03-24T22:06:42.475902+0000","last_fullsized":"2021-03-24T22:06:42.475902+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'24","last_scrub_stamp":"2021-03-24T22:04:37.278283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:04:37.278283+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":42,"num_read_kb":29,"num_write":24,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":4992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":"297","reported_epoch":"831","state":"active+clean","last_fresh":"2021-03-24T22:06:41.469249+0000","last_change":"2021-03-24T22:04:37.274416+0000","last_active":"2021-03-24T22:06:41.469249+0000","last_peered":"2021-03-24T22:06:41.469249+0000","last_clean":"2021-03-24T22:06:41.469249+0000","last_became_active":"2021-03-24T22:01:57.318685+0000","last_became_peered":"2021-03-24T22:01:57.318685+0000","last_unstale":"2021-03-24T22:06:41.469249+0000","last_undegraded":"2021-03-24T22:06:41.469249+0000","last_fullsized":"2021-03-24T22:06:41.469249+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:04:37.274302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:04:37.274302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"743'12","reported_seq":"830","reported_epoch":"830","state":"active+clean","last_fresh":"2021-03-24T22:06:40.458109+0000","last_change":"2021-03-24T22:05:14.419465+0000","last_active":"2021-03-24T22:06:40.458109+0000","last_peered":"2021-03-24T22:06:40.458109+0000","last_clean":"2021-03-24T22:06:40.458109+0000","last_became_active":"2021-03-24T22:05:04.451042+0000","last_became_peered":"2021-03-24T22:05:04.451042+0000","last_unstale":"2021-03-24T22:06:40.458109+0000","last_undegraded":"2021-03-24T22:06:40.458109+0000","last_fullsized":"2021-03-24T22:06:40.458109+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"664'11","last_scrub_stamp":"2021-03-24T22:05:06.161410+0000","last_deep_scrub":"86'3","last_deep_scrub_stamp":"2021-03-24T21:53:08.815163+0000","last_clean_scrub_stamp":"2021-03-24T22:05:06.161410+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"835","reported_epoch":"831","state":"active+clean","last_fresh":"2021-03-24T22:06:41.458103+0000","last_change":"2021-03-24T22:03:57.076812+0000","last_active":"2021-03-24T22:06:41.458103+0000","last_peered":"2021-03-24T22:06:41.458103+0000","last_clean":"2021-03-24T22:06:41.458103+0000","last_became_active":"2021-03-24T22:01:57.316729+0000","last_became_peered":"2021-03-24T22:01:57.316729+0000","last_unstale":"2021-03-24T22:06:41.458103+0000","last_undegraded":"2021-03-24T22:06:41.458103+0000","last_fullsized":"2021-03-24T22:06:41.458103+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:57.076558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:03:57.076558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.17","version":"831'1390","reported_seq":"4022","reported_epoch":"831","state":"active+clean","last_fresh":"2021-03-24T22:06:41.498716+0000","last_change":"2021-03-24T22:06:41.498716+0000","last_active":"2021-03-24T22:06:41.498716+0000","last_peered":"2021-03-24T22:06:41.498716+0000","last_clean":"2021-03-24T22:06:41.498716+0000","last_became_active":"2021-03-24T22:05:04.460028+0000","last_became_peered":"2021-03-24T22:05:04.460028+0000","last_unstale":"2021-03-24T22:06:41.498716+0000","last_undegraded":"2021-03-24T22:06:41.498716+0000","last_fullsized":"2021-03-24T22:06:41.498716+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":5,"last_scrub":"742'1137","last_scrub_stamp":"2021-03-24T22:05:13.182580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:05:13.182580+0000","log_size":1390,"ondisk_log_size":1390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37594391,"num_objects":18,"num_object_clones":14,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":2,"num_read":355,"num_read_kb":387285,"num_write":248,"num_write_kb":90305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":257,"num_bytes_recovered":169206653,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"255","length":"1"},{"start":"27d","length":"1"},{"start":"297","length":"1"},{"start":"2ab","length":"1"},{"start":"2ad","length":"1"},{"start":"2c4","length":"1"}]},{"pgid":"4.13","version":"584'32","reported_seq":"410","reported_epoch":"829","state":"active+clean","last_fresh":"2021-03-24T22:06:39.464390+0000","last_change":"2021-03-24T22:05:48.406758+0000","last_active":"2021-03-24T22:06:39.464390+0000","last_peered":"2021-03-24T22:06:39.464390+0000","last_clean":"2021-03-24T22:06:39.464390+0000","last_became_active":"2021-03-24T22:03:45.426065+0000","last_became_peered":"2021-03-24T22:03:45.426065+0000","last_unstale":"2021-03-24T22:06:39.464390+0000","last_undegraded":"2021-03-24T22:06:39.464390+0000","last_fullsized":"2021-03-24T22:06:39.464390+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"584'32","last_scrub_stamp":"2021-03-24T22:05:48.406579+0000","last_deep_scrub":"584'32","last_deep_scrub_stamp":"2021-03-24T22:02:34.813628+0000","last_clean_scrub_stamp":"2021-03-24T22:05:48.406579+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":29,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":3680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":"304","reported_epoch":"831","state":"active+clean","last_fresh":"2021-03-24T22:06:41.470623+0000","last_change":"2021-03-24T22:06:08.071800+0000","last_active":"2021-03-24T22:06:41.470623+0000","last_peered":"2021-03-24T22:06:41.470623+0000","last_clean":"2021-03-24T22:06:41.470623+0000","last_became_active":"2021-03-24T22:01:57.307018+0000","last_became_peered":"2021-03-24T22:01:57.307018+0000","last_unstale":"2021-03-24T22:06:41.470623+0000","last_undegraded":"2021-03-24T22:06:41.470623+0000","last_fullsized":"2021-03-24T22:06:41.470623+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:08.071696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:06:08.071696+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"204","reported_epoch":"828","state":"active+clean","last_fresh":"2021-03-24T22:06:38.451583+0000","last_change":"2021-03-24T22:05:07.629502+0000","last_active":"2021-03-24T22:06:38.451583+0000","last_peered":"2021-03-24T22:06:38.451583+0000","last_clean":"2021-03-24T22:06:38.451583+0000","last_became_active":"2021-03-24T22:05:04.451084+0000","last_became_peered":"2021-03-24T22:05:04.451084+0000","last_unstale":"2021-03-24T22:06:38.451583+0000","last_undegraded":"2021-03-24T22:06:38.451583+0000","last_fullsized":"2021-03-24T22:06:38.451583+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:07.629376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:57:31.118050+0000","last_clean_scrub_stamp":"2021-03-24T22:05:07.629376+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"819","reported_epoch":"832","state":"active+clean","last_fresh":"2021-03-24T22:06:42.469822+0000","last_change":"2021-03-24T22:05:52.855070+0000","last_active":"2021-03-24T22:06:42.469822+0000","last_peered":"2021-03-24T22:06:42.469822+0000","last_clean":"2021-03-24T22:06:42.469822+0000","last_became_active":"2021-03-24T22:05:04.435491+0000","last_became_peered":"2021-03-24T22:05:04.435491+0000","last_unstale":"2021-03-24T22:06:42.469822+0000","last_undegraded":"2021-03-24T22:06:42.469822+0000","last_fullsized":"2021-03-24T22:06:42.469822+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:52.854930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:23.707914+0000","last_clean_scrub_stamp":"2021-03-24T22:05:52.854930+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.10","version":"831'252","reported_seq":"1129","reported_epoch":"832","state":"active+clean","last_fresh":"2021-03-24T22:06:42.468163+0000","last_change":"2021-03-24T22:06:41.484292+0000","last_active":"2021-03-24T22:06:42.468163+0000","last_peered":"2021-03-24T22:06:42.468163+0000","last_clean":"2021-03-24T22:06:42.468163+0000","last_became_active":"2021-03-24T22:05:04.438010+0000","last_became_peered":"2021-03-24T22:05:04.438010+0000","last_unstale":"2021-03-24T22:06:42.468163+0000","last_undegraded":"2021-03-24T22:06:42.468163+0000","last_fullsized":"2021-03-24T22:06:42.468163+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":5,"last_scrub":"579'81","last_scrub_stamp":"2021-03-24T22:02:30.763305+0000","last_deep_scrub":"277'24","last_deep_scrub_stamp":"2021-03-24T21:57:28.119420+0000","last_clean_scrub_stamp":"2021-03-24T22:02:30.763305+0000","log_size":252,"ondisk_log_size":252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11918454,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":115,"num_read_kb":111507,"num_write":61,"num_write_kb":21017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":32974145,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"255","length":"1"},{"start":"27d","length":"1"},{"start":"297","length":"1"},{"start":"2ab","length":"1"},{"start":"2ad","length":"1"},{"start":"2c4","length":"1"}]},{"pgid":"4.12","version":"544'21","reported_seq":"211","reported_epoch":"828","state":"active+clean","last_fresh":"2021-03-24T22:06:38.452140+0000","last_change":"2021-03-24T22:04:33.627668+0000","last_active":"2021-03-24T22:06:38.452140+0000","last_peered":"2021-03-24T22:06:38.452140+0000","last_clean":"2021-03-24T22:06:38.452140+0000","last_became_active":"2021-03-24T22:03:45.427113+0000","last_became_peered":"2021-03-24T22:03:45.427113+0000","last_unstale":"2021-03-24T22:06:38.452140+0000","last_undegraded":"2021-03-24T22:06:38.452140+0000","last_fullsized":"2021-03-24T22:06:38.452140+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'21","last_scrub_stamp":"2021-03-24T22:04:33.627568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:04:33.627568+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":43,"num_read_kb":25,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":3150,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":"302","reported_epoch":"831","state":"active+clean","last_fresh":"2021-03-24T22:06:41.470792+0000","last_change":"2021-03-24T22:05:53.144403+0000","last_active":"2021-03-24T22:06:41.470792+0000","last_peered":"2021-03-24T22:06:41.470792+0000","last_clean":"2021-03-24T22:06:41.470792+0000","last_became_active":"2021-03-24T22:01:57.318513+0000","last_became_peered":"2021-03-24T22:01:57.318513+0000","last_unstale":"2021-03-24T22:06:41.470792+0000","last_undegraded":"2021-03-24T22:06:41.470792+0000","last_fullsized":"2021-03-24T22:06:41.470792+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:53.144236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T22:03:09.417465+0000","last_clean_scrub_stamp":"2021-03-24T22:05:53.144236+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"842","reported_epoch":"832","state":"active+clean","last_fresh":"2021-03-24T22:06:42.470129+0000","last_change":"2021-03-24T22:05:57.864744+0000","last_active":"2021-03-24T22:06:42.470129+0000","last_peered":"2021-03-24T22:06:42.470129+0000","last_clean":"2021-03-24T22:06:42.470129+0000","last_became_active":"2021-03-24T22:03:45.435094+0000","last_became_peered":"2021-03-24T22:03:45.435094+0000","last_unstale":"2021-03-24T22:06:42.470129+0000","last_undegraded":"2021-03-24T22:06:42.470129+0000","last_fullsized":"2021-03-24T22:06:42.470129+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:57.864596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:05:57.864596+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"861","reported_epoch":"831","state":"active+clean","last_fresh":"2021-03-24T22:06:41.457708+0000","last_change":"2021-03-24T22:05:59.395794+0000","last_active":"2021-03-24T22:06:41.457708+0000","last_peered":"2021-03-24T22:06:41.457708+0000","last_clean":"2021-03-24T22:06:41.457708+0000","last_became_active":"2021-03-24T22:05:04.447958+0000","last_became_peered":"2021-03-24T22:05:04.447958+0000","last_unstale":"2021-03-24T22:06:41.457708+0000","last_undegraded":"2021-03-24T22:06:41.457708+0000","last_fullsized":"2021-03-24T22:06:41.457708+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:59.395655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:05:59.395655+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.11","version":"831'1200","reported_seq":"3797","reported_epoch":"832","state":"active+clean","last_fresh":"2021-03-24T22:06:42.470376+0000","last_change":"2021-03-24T22:06:41.476585+0000","last_active":"2021-03-24T22:06:42.470376+0000","last_peered":"2021-03-24T22:06:42.470376+0000","last_clean":"2021-03-24T22:06:42.470376+0000","last_became_active":"2021-03-24T22:02:04.322694+0000","last_became_peered":"2021-03-24T22:02:04.322694+0000","last_unstale":"2021-03-24T22:06:42.470376+0000","last_undegraded":"2021-03-24T22:06:42.470376+0000","last_fullsized":"2021-03-24T22:06:42.470376+0000","mapping_epoch":553,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":554,"parent":"0.0","parent_split_bits":5,"last_scrub":"703'940","last_scrub_stamp":"2021-03-24T22:04:35.296170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:04:35.296170+0000","log_size":1200,"ondisk_log_size":1200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36454689,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":243,"num_read_kb":285360,"num_write":161,"num_write_kb":70451,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":325,"num_bytes_recovered":148467814,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"255","length":"1"},{"start":"27d","length":"1"},{"start":"297","length":"1"},{"start":"2ab","length":"1"},{"start":"2ad","length":"1"},{"start":"2c4","length":"1"}]},{"pgid":"4.11","version":"544'30","reported_seq":"1017","reported_epoch":"832","state":"active+clean","last_fresh":"2021-03-24T22:06:42.469456+0000","last_change":"2021-03-24T22:05:45.015199+0000","last_active":"2021-03-24T22:06:42.469456+0000","last_peered":"2021-03-24T22:06:42.469456+0000","last_clean":"2021-03-24T22:06:42.469456+0000","last_became_active":"2021-03-24T22:01:57.310370+0000","last_became_peered":"2021-03-24T22:01:57.310370+0000","last_unstale":"2021-03-24T22:06:42.469456+0000","last_undegraded":"2021-03-24T22:06:42.469456+0000","last_fullsized":"2021-03-24T22:06:42.469456+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'30","last_scrub_stamp":"2021-03-24T22:05:45.015001+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2021-03-24T21:52:54.597505+0000","last_clean_scrub_stamp":"2021-03-24T22:05:45.015001+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":41,"num_read_kb":28,"num_write":23,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":3480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":"681","reported_epoch":"830","state":"active+clean","last_fresh":"2021-03-24T22:06:40.458355+0000","last_change":"2021-03-24T22:05:43.282870+0000","last_active":"2021-03-24T22:06:40.458355+0000","last_peered":"2021-03-24T22:06:40.458355+0000","last_clean":"2021-03-24T22:06:40.458355+0000","last_became_active":"2021-03-24T22:05:04.439770+0000","last_became_peered":"2021-03-24T22:05:04.439770+0000","last_unstale":"2021-03-24T22:06:40.458355+0000","last_undegraded":"2021-03-24T22:06:40.458355+0000","last_fullsized":"2021-03-24T22:06:40.458355+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:43.282684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:05:43.282684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"297","reported_epoch":"831","state":"active+clean","last_fresh":"2021-03-24T22:06:41.466929+0000","last_change":"2021-03-24T22:04:38.228550+0000","last_active":"2021-03-24T22:06:41.466929+0000","last_peered":"2021-03-24T22:06:41.466929+0000","last_clean":"2021-03-24T22:06:41.466929+0000","last_became_active":"2021-03-24T22:01:57.307995+0000","last_became_peered":"2021-03-24T22:01:57.307995+0000","last_unstale":"2021-03-24T22:06:41.466929+0000","last_undegraded":"2021-03-24T22:06:41.466929+0000","last_fullsized":"2021-03-24T22:06:41.466929+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:04:38.228363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T22:04:38.228363+0000","last_clean_scrub_stamp":"2021-03-24T22:04:38.228363+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"807","reported_epoch":"832","state":"active+clean","last_fresh":"2021-03-24T22:06:42.476644+0000","last_change":"2021-03-24T22:05:09.243747+0000","last_active":"2021-03-24T22:06:42.476644+0000","last_peered":"2021-03-24T22:06:42.476644+0000","last_clean":"2021-03-24T22:06:42.476644+0000","last_became_active":"2021-03-24T22:01:43.389017+0000","last_became_peered":"2021-03-24T22:01:43.389017+0000","last_unstale":"2021-03-24T22:06:42.476644+0000","last_undegraded":"2021-03-24T22:06:42.476644+0000","last_fullsized":"2021-03-24T22:06:42.476644+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:09.243590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:47.540793+0000","last_clean_scrub_stamp":"2021-03-24T22:05:09.243590+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.12","version":"183'26","reported_seq":"939","reported_epoch":"830","state":"active+clean","last_fresh":"2021-03-24T22:06:40.464398+0000","last_change":"2021-03-24T22:06:40.464883+0000","last_active":"2021-03-24T22:06:40.464398+0000","last_peered":"2021-03-24T22:06:40.464398+0000","last_clean":"2021-03-24T22:06:40.464398+0000","last_became_active":"2021-03-24T22:01:43.392614+0000","last_became_peered":"2021-03-24T22:01:43.392614+0000","last_unstale":"2021-03-24T22:06:40.464398+0000","last_undegraded":"2021-03-24T22:06:40.464398+0000","last_fullsized":"2021-03-24T22:06:40.464398+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":5,"last_scrub":"183'26","last_scrub_stamp":"2021-03-24T22:04:25.596444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:04:25.596444+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"27d","length":"1"},{"start":"297","length":"1"},{"start":"2ab","length":"1"},{"start":"2ad","length":"1"},{"start":"2c4","length":"1"}]},{"pgid":"4.10","version":"544'16","reported_seq":"1091","reported_epoch":"832","state":"active+clean","last_fresh":"2021-03-24T22:06:42.468573+0000","last_change":"2021-03-24T22:05:47.006934+0000","last_active":"2021-03-24T22:06:42.468573+0000","last_peered":"2021-03-24T22:06:42.468573+0000","last_clean":"2021-03-24T22:06:42.468573+0000","last_became_active":"2021-03-24T22:03:45.428746+0000","last_became_peered":"2021-03-24T22:03:45.428746+0000","last_unstale":"2021-03-24T22:06:42.468573+0000","last_undegraded":"2021-03-24T22:06:42.468573+0000","last_fullsized":"2021-03-24T22:06:42.468573+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'16","last_scrub_stamp":"2021-03-24T22:05:47.006825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:05:47.006825+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":36,"num_read_kb":18,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":10016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":"831","reported_epoch":"831","state":"active+clean","last_fresh":"2021-03-24T22:06:41.458730+0000","last_change":"2021-03-24T22:05:58.410296+0000","last_active":"2021-03-24T22:06:41.458730+0000","last_peered":"2021-03-24T22:06:41.458730+0000","last_clean":"2021-03-24T22:06:41.458730+0000","last_became_active":"2021-03-24T22:05:04.458069+0000","last_became_peered":"2021-03-24T22:05:04.458069+0000","last_unstale":"2021-03-24T22:06:41.458730+0000","last_undegraded":"2021-03-24T22:06:41.458730+0000","last_fullsized":"2021-03-24T22:06:41.458730+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:58.410205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:05:58.410205+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"322","reported_epoch":"829","state":"active+clean","last_fresh":"2021-03-24T22:06:39.467420+0000","last_change":"2021-03-24T22:04:40.677638+0000","last_active":"2021-03-24T22:06:39.467420+0000","last_peered":"2021-03-24T22:06:39.467420+0000","last_clean":"2021-03-24T22:06:39.467420+0000","last_became_active":"2021-03-24T22:01:57.304385+0000","last_became_peered":"2021-03-24T22:01:57.304385+0000","last_unstale":"2021-03-24T22:06:39.467420+0000","last_undegraded":"2021-03-24T22:06:39.467420+0000","last_fullsized":"2021-03-24T22:06:39.467420+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:04:40.677553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:04:40.677553+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"304","reported_epoch":"831","state":"active+clean","last_fresh":"2021-03-24T22:06:41.466880+0000","last_change":"2021-03-24T22:05:45.108061+0000","last_active":"2021-03-24T22:06:41.466880+0000","last_peered":"2021-03-24T22:06:41.466880+0000","last_clean":"2021-03-24T22:06:41.466880+0000","last_became_active":"2021-03-24T22:01:57.315718+0000","last_became_peered":"2021-03-24T22:01:57.315718+0000","last_unstale":"2021-03-24T22:06:41.466880+0000","last_undegraded":"2021-03-24T22:06:41.466880+0000","last_fullsized":"2021-03-24T22:06:41.466880+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:45.107889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:57:59.979772+0000","last_clean_scrub_stamp":"2021-03-24T22:05:45.107889+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.13","version":"831'775","reported_seq":"3527","reported_epoch":"831","state":"active+clean","last_fresh":"2021-03-24T22:06:41.503053+0000","last_change":"2021-03-24T22:06:41.503053+0000","last_active":"2021-03-24T22:06:41.503053+0000","last_peered":"2021-03-24T22:06:41.503053+0000","last_clean":"2021-03-24T22:06:41.503053+0000","last_became_active":"2021-03-24T22:02:28.318669+0000","last_became_peered":"2021-03-24T22:02:28.318669+0000","last_unstale":"2021-03-24T22:06:41.503053+0000","last_undegraded":"2021-03-24T22:06:41.503053+0000","last_fullsized":"2021-03-24T22:06:41.503053+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":5,"last_scrub":"783'727","last_scrub_stamp":"2021-03-24T22:05:55.373277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:05:55.373277+0000","log_size":775,"ondisk_log_size":775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12928677,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"255","length":"1"},{"start":"27d","length":"1"},{"start":"297","length":"1"},{"start":"2ab","length":"1"},{"start":"2ad","length":"1"},{"start":"2c4","length":"1"}]},{"pgid":"3.18","version":"0'0","reported_seq":"831","reported_epoch":"832","state":"active+clean","last_fresh":"2021-03-24T22:06:42.468664+0000","last_change":"2021-03-24T22:05:01.161274+0000","last_active":"2021-03-24T22:06:42.468664+0000","last_peered":"2021-03-24T22:06:42.468664+0000","last_clean":"2021-03-24T22:06:42.468664+0000","last_became_active":"2021-03-24T22:03:45.428772+0000","last_became_peered":"2021-03-24T22:03:45.428772+0000","last_unstale":"2021-03-24T22:06:42.468664+0000","last_undegraded":"2021-03-24T22:06:42.468664+0000","last_fullsized":"2021-03-24T22:06:42.468664+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:01.161130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:05:01.161130+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"855","reported_epoch":"831","state":"active+clean","last_fresh":"2021-03-24T22:06:41.458844+0000","last_change":"2021-03-24T22:04:35.127105+0000","last_active":"2021-03-24T22:06:41.458844+0000","last_peered":"2021-03-24T22:06:41.458844+0000","last_clean":"2021-03-24T22:06:41.458844+0000","last_became_active":"2021-03-24T22:02:22.328021+0000","last_became_peered":"2021-03-24T22:02:22.328021+0000","last_unstale":"2021-03-24T22:06:41.458844+0000","last_undegraded":"2021-03-24T22:06:41.458844+0000","last_fullsized":"2021-03-24T22:06:41.458844+0000","mapping_epoch":571,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:04:35.126877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:59:17.040623+0000","last_clean_scrub_stamp":"2021-03-24T22:04:35.126877+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"204","reported_epoch":"828","state":"active+clean","last_fresh":"2021-03-24T22:06:38.452870+0000","last_change":"2021-03-24T22:05:45.490399+0000","last_active":"2021-03-24T22:06:38.452870+0000","last_peered":"2021-03-24T22:06:38.452870+0000","last_clean":"2021-03-24T22:06:38.452870+0000","last_became_active":"2021-03-24T22:05:04.432365+0000","last_became_peered":"2021-03-24T22:05:04.432365+0000","last_unstale":"2021-03-24T22:06:38.452870+0000","last_undegraded":"2021-03-24T22:06:38.452870+0000","last_fullsized":"2021-03-24T22:06:38.452870+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:45.490290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:05:45.490290+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1c","version":"273'245","reported_seq":"2608","reported_epoch":"832","state":"active+clean","last_fresh":"2021-03-24T22:06:42.473450+0000","last_change":"2021-03-24T22:06:41.489697+0000","last_active":"2021-03-24T22:06:42.473450+0000","last_peered":"2021-03-24T22:06:42.473450+0000","last_clean":"2021-03-24T22:06:42.473450+0000","last_became_active":"2021-03-24T22:05:04.445911+0000","last_became_peered":"2021-03-24T22:05:04.445911+0000","last_unstale":"2021-03-24T22:06:42.473450+0000","last_undegraded":"2021-03-24T22:06:42.473450+0000","last_fullsized":"2021-03-24T22:06:42.473450+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":245,"ondisk_log_size":245,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8581786,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":70,"num_read_kb":67597,"num_write":52,"num_write_kb":18055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"255","length":"1"},{"start":"27d","length":"1"},{"start":"297","length":"1"},{"start":"2ab","length":"1"},{"start":"2ad","length":"1"},{"start":"2c4","length":"1"}]},{"pgid":"3.19","version":"0'0","reported_seq":"189","reported_epoch":"828","state":"active+clean","last_fresh":"2021-03-24T22:06:38.450321+0000","last_change":"2021-03-24T22:04:25.611828+0000","last_active":"2021-03-24T22:06:38.450321+0000","last_peered":"2021-03-24T22:06:38.450321+0000","last_clean":"2021-03-24T22:06:38.450321+0000","last_became_active":"2021-03-24T22:03:45.430602+0000","last_became_peered":"2021-03-24T22:03:45.430602+0000","last_unstale":"2021-03-24T22:06:38.450321+0000","last_undegraded":"2021-03-24T22:06:38.450321+0000","last_fullsized":"2021-03-24T22:06:38.450321+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:04:25.611742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:57:26.417613+0000","last_clean_scrub_stamp":"2021-03-24T22:04:25.611742+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"613'8","reported_seq":"388","reported_epoch":"829","state":"active+clean","last_fresh":"2021-03-24T22:06:39.462817+0000","last_change":"2021-03-24T22:06:00.101350+0000","last_active":"2021-03-24T22:06:39.462817+0000","last_peered":"2021-03-24T22:06:39.462817+0000","last_clean":"2021-03-24T22:06:39.462817+0000","last_became_active":"2021-03-24T22:03:03.353426+0000","last_became_peered":"2021-03-24T22:03:03.353426+0000","last_unstale":"2021-03-24T22:06:39.462817+0000","last_undegraded":"2021-03-24T22:06:39.462817+0000","last_fullsized":"2021-03-24T22:06:39.462817+0000","mapping_epoch":612,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":613,"parent":"0.0","parent_split_bits":0,"last_scrub":"613'8","last_scrub_stamp":"2021-03-24T22:06:00.101091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:06:00.101091+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"609","reported_epoch":"830","state":"active+clean","last_fresh":"2021-03-24T22:06:40.458356+0000","last_change":"2021-03-24T22:05:43.068387+0000","last_active":"2021-03-24T22:06:40.458356+0000","last_peered":"2021-03-24T22:06:40.458356+0000","last_clean":"2021-03-24T22:06:40.458356+0000","last_became_active":"2021-03-24T22:05:04.423989+0000","last_became_peered":"2021-03-24T22:05:04.423989+0000","last_unstale":"2021-03-24T22:06:40.458356+0000","last_undegraded":"2021-03-24T22:06:40.458356+0000","last_fullsized":"2021-03-24T22:06:40.458356+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:43.068130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:05:43.068130+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1d","version":"827'733","reported_seq":"3383","reported_epoch":"829","state":"active+clean","last_fresh":"2021-03-24T22:06:39.461595+0000","last_change":"2021-03-24T22:06:39.462053+0000","last_active":"2021-03-24T22:06:39.461595+0000","last_peered":"2021-03-24T22:06:39.461595+0000","last_clean":"2021-03-24T22:06:39.461595+0000","last_became_active":"2021-03-24T22:03:53.384369+0000","last_became_peered":"2021-03-24T22:03:53.384369+0000","last_unstale":"2021-03-24T22:06:39.461595+0000","last_undegraded":"2021-03-24T22:06:39.461595+0000","last_fullsized":"2021-03-24T22:06:39.461595+0000","mapping_epoch":662,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":663,"parent":"0.0","parent_split_bits":5,"last_scrub":"695'623","last_scrub_stamp":"2021-03-24T22:04:37.590137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:04:37.590137+0000","log_size":733,"ondisk_log_size":733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13252624,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":36270,"num_write":16,"num_write_kb":6286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"25e","length":"1"},{"start":"27d","length":"1"},{"start":"28e","length":"1"},{"start":"297","length":"1"},{"start":"2ad","length":"1"},{"start":"2af","length":"1"},{"start":"2c4","length":"1"}]},{"pgid":"3.1a","version":"0'0","reported_seq":"189","reported_epoch":"828","state":"active+clean","last_fresh":"2021-03-24T22:06:38.451855+0000","last_change":"2021-03-24T22:05:49.560351+0000","last_active":"2021-03-24T22:06:38.451855+0000","last_peered":"2021-03-24T22:06:38.451855+0000","last_clean":"2021-03-24T22:06:38.451855+0000","last_became_active":"2021-03-24T22:03:45.431828+0000","last_became_peered":"2021-03-24T22:03:45.431828+0000","last_unstale":"2021-03-24T22:06:38.451855+0000","last_undegraded":"2021-03-24T22:06:38.451855+0000","last_fullsized":"2021-03-24T22:06:38.451855+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:49.560279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:52:54.212459+0000","last_clean_scrub_stamp":"2021-03-24T22:05:49.560279+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"368","reported_epoch":"832","state":"active+clean","last_fresh":"2021-03-24T22:06:42.473822+0000","last_change":"2021-03-24T22:02:17.672555+0000","last_active":"2021-03-24T22:06:42.473822+0000","last_peered":"2021-03-24T22:06:42.473822+0000","last_clean":"2021-03-24T22:06:42.473822+0000","last_became_active":"2021-03-24T22:00:49.305569+0000","last_became_peered":"2021-03-24T22:00:49.305569+0000","last_unstale":"2021-03-24T22:06:42.473822+0000","last_undegraded":"2021-03-24T22:06:42.473822+0000","last_fullsized":"2021-03-24T22:06:42.473822+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:17.672325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:02:17.672325+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"855","reported_epoch":"832","state":"active+clean","last_fresh":"2021-03-24T22:06:42.475698+0000","last_change":"2021-03-24T22:06:07.907835+0000","last_active":"2021-03-24T22:06:42.475698+0000","last_peered":"2021-03-24T22:06:42.475698+0000","last_clean":"2021-03-24T22:06:42.475698+0000","last_became_active":"2021-03-24T22:05:04.423569+0000","last_became_peered":"2021-03-24T22:05:04.423569+0000","last_unstale":"2021-03-24T22:06:42.475698+0000","last_undegraded":"2021-03-24T22:06:42.475698+0000","last_fullsized":"2021-03-24T22:06:42.475698+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:07.907636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T22:06:07.907636+0000","last_clean_scrub_stamp":"2021-03-24T22:06:07.907636+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1e","version":"265'45","reported_seq":"563","reported_epoch":"829","state":"active+clean","last_fresh":"2021-03-24T22:06:39.459711+0000","last_change":"2021-03-24T22:06:39.460816+0000","last_active":"2021-03-24T22:06:39.459711+0000","last_peered":"2021-03-24T22:06:39.459711+0000","last_clean":"2021-03-24T22:06:39.459711+0000","last_became_active":"2021-03-24T22:05:04.432331+0000","last_became_peered":"2021-03-24T22:05:04.432331+0000","last_unstale":"2021-03-24T22:06:39.459711+0000","last_undegraded":"2021-03-24T22:06:39.459711+0000","last_fullsized":"2021-03-24T22:06:39.459711+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":5,"last_scrub":"265'45","last_scrub_stamp":"2021-03-24T22:05:55.541936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:05:55.541936+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"25e","length":"1"},{"start":"27d","length":"1"},{"start":"28e","length":"1"},{"start":"297","length":"1"},{"start":"2ad","length":"1"},{"start":"2af","length":"1"},{"start":"2c4","length":"1"}]},{"pgid":"3.1b","version":"0'0","reported_seq":"855","reported_epoch":"831","state":"active+clean","last_fresh":"2021-03-24T22:06:41.465058+0000","last_change":"2021-03-24T22:05:51.442642+0000","last_active":"2021-03-24T22:06:41.465058+0000","last_peered":"2021-03-24T22:06:41.465058+0000","last_clean":"2021-03-24T22:06:41.465058+0000","last_became_active":"2021-03-24T22:03:32.380554+0000","last_became_peered":"2021-03-24T22:03:32.380554+0000","last_unstale":"2021-03-24T22:06:41.465058+0000","last_undegraded":"2021-03-24T22:06:41.465058+0000","last_fullsized":"2021-03-24T22:06:41.465058+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:51.442473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:05:51.442473+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"194","reported_epoch":"828","state":"active+clean","last_fresh":"2021-03-24T22:06:38.755321+0000","last_change":"2021-03-24T22:06:38.755278+0000","last_active":"2021-03-24T22:06:38.755321+0000","last_peered":"2021-03-24T22:06:38.755321+0000","last_clean":"2021-03-24T22:06:38.755321+0000","last_became_active":"2021-03-24T22:03:45.433015+0000","last_became_peered":"2021-03-24T22:03:45.433015+0000","last_unstale":"2021-03-24T22:06:38.755321+0000","last_undegraded":"2021-03-24T22:06:38.755321+0000","last_fullsized":"2021-03-24T22:06:38.755321+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:38.755184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:06:38.755184+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"321","reported_epoch":"829","state":"active+clean","last_fresh":"2021-03-24T22:06:39.461809+0000","last_change":"2021-03-24T22:05:08.570430+0000","last_active":"2021-03-24T22:06:39.461809+0000","last_peered":"2021-03-24T22:06:39.461809+0000","last_clean":"2021-03-24T22:06:39.461809+0000","last_became_active":"2021-03-24T22:05:04.431178+0000","last_became_peered":"2021-03-24T22:05:04.431178+0000","last_unstale":"2021-03-24T22:06:39.461809+0000","last_undegraded":"2021-03-24T22:06:39.461809+0000","last_fullsized":"2021-03-24T22:06:39.461809+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:08.570238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:05:08.570238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1f","version":"272'99","reported_seq":"934","reported_epoch":"830","state":"active+clean","last_fresh":"2021-03-24T22:06:40.466001+0000","last_change":"2021-03-24T22:06:40.466376+0000","last_active":"2021-03-24T22:06:40.466001+0000","last_peered":"2021-03-24T22:06:40.466001+0000","last_clean":"2021-03-24T22:06:40.466001+0000","last_became_active":"2021-03-24T22:01:43.385324+0000","last_became_peered":"2021-03-24T22:01:43.385324+0000","last_unstale":"2021-03-24T22:06:40.466001+0000","last_undegraded":"2021-03-24T22:06:40.466001+0000","last_fullsized":"2021-03-24T22:06:40.466001+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'99","last_scrub_stamp":"2021-03-24T22:03:57.739288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:03:57.739288+0000","log_size":99,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"27d","length":"1"},{"start":"297","length":"1"},{"start":"2ab","length":"1"},{"start":"2ad","length":"1"},{"start":"2c4","length":"1"}]},{"pgid":"3.1c","version":"0'0","reported_seq":"204","reported_epoch":"828","state":"active+clean","last_fresh":"2021-03-24T22:06:38.450368+0000","last_change":"2021-03-24T22:05:05.665557+0000","last_active":"2021-03-24T22:06:38.450368+0000","last_peered":"2021-03-24T22:06:38.450368+0000","last_clean":"2021-03-24T22:06:38.450368+0000","last_became_active":"2021-03-24T22:05:04.453666+0000","last_became_peered":"2021-03-24T22:05:04.453666+0000","last_unstale":"2021-03-24T22:06:38.450368+0000","last_undegraded":"2021-03-24T22:06:38.450368+0000","last_fullsized":"2021-03-24T22:06:38.450368+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:05.665337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:05:05.665337+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"320","reported_epoch":"831","state":"active+clean","last_fresh":"2021-03-24T22:06:41.467315+0000","last_change":"2021-03-24T22:06:34.008599+0000","last_active":"2021-03-24T22:06:41.467315+0000","last_peered":"2021-03-24T22:06:41.467315+0000","last_clean":"2021-03-24T22:06:41.467315+0000","last_became_active":"2021-03-24T22:03:45.434093+0000","last_became_peered":"2021-03-24T22:03:45.434093+0000","last_unstale":"2021-03-24T22:06:41.467315+0000","last_undegraded":"2021-03-24T22:06:41.467315+0000","last_fullsized":"2021-03-24T22:06:41.467315+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:34.008428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:56:51.977306+0000","last_clean_scrub_stamp":"2021-03-24T22:06:34.008428+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"659","reported_epoch":"830","state":"active+clean","last_fresh":"2021-03-24T22:06:40.457347+0000","last_change":"2021-03-24T22:06:34.336607+0000","last_active":"2021-03-24T22:06:40.457347+0000","last_peered":"2021-03-24T22:06:40.457347+0000","last_clean":"2021-03-24T22:06:40.457347+0000","last_became_active":"2021-03-24T22:05:04.437068+0000","last_became_peered":"2021-03-24T22:05:04.437068+0000","last_unstale":"2021-03-24T22:06:40.457347+0000","last_undegraded":"2021-03-24T22:06:40.457347+0000","last_fullsized":"2021-03-24T22:06:40.457347+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:34.336429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T22:05:15.154493+0000","last_clean_scrub_stamp":"2021-03-24T22:06:34.336429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.18","version":"831'304","reported_seq":"2379","reported_epoch":"832","state":"active+clean","last_fresh":"2021-03-24T22:06:43.544642+0000","last_change":"2021-03-24T22:06:43.544642+0000","last_active":"2021-03-24T22:06:43.544642+0000","last_peered":"2021-03-24T22:06:43.544642+0000","last_clean":"2021-03-24T22:06:43.544642+0000","last_became_active":"2021-03-24T22:02:00.304128+0000","last_became_peered":"2021-03-24T22:02:00.304128+0000","last_unstale":"2021-03-24T22:06:43.544642+0000","last_undegraded":"2021-03-24T22:06:43.544642+0000","last_fullsized":"2021-03-24T22:06:43.544642+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":5,"last_scrub":"831'304","last_scrub_stamp":"2021-03-24T22:06:43.544396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:06:43.544396+0000","log_size":304,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7109856,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":54,"num_read_kb":47612,"num_write":36,"num_write_kb":12535,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":32240200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"255","length":"1"},{"start":"27d","length":"1"},{"start":"297","length":"1"},{"start":"2ab","length":"1"},{"start":"2ad","length":"1"},{"start":"2c4","length":"1"}]},{"pgid":"3.1d","version":"0'0","reported_seq":"877","reported_epoch":"830","state":"active+clean","last_fresh":"2021-03-24T22:06:40.458525+0000","last_change":"2021-03-24T22:05:48.075900+0000","last_active":"2021-03-24T22:06:40.458525+0000","last_peered":"2021-03-24T22:06:40.458525+0000","last_clean":"2021-03-24T22:06:40.458525+0000","last_became_active":"2021-03-24T22:05:04.430681+0000","last_became_peered":"2021-03-24T22:05:04.430681+0000","last_unstale":"2021-03-24T22:06:40.458525+0000","last_undegraded":"2021-03-24T22:06:40.458525+0000","last_fullsized":"2021-03-24T22:06:40.458525+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:48.075822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:05:48.075822+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"653","reported_epoch":"830","state":"active+clean","last_fresh":"2021-03-24T22:06:40.460548+0000","last_change":"2021-03-24T22:05:52.372115+0000","last_active":"2021-03-24T22:06:40.460548+0000","last_peered":"2021-03-24T22:06:40.460548+0000","last_clean":"2021-03-24T22:06:40.460548+0000","last_became_active":"2021-03-24T22:05:04.432965+0000","last_became_peered":"2021-03-24T22:05:04.432965+0000","last_unstale":"2021-03-24T22:06:40.460548+0000","last_undegraded":"2021-03-24T22:06:40.460548+0000","last_fullsized":"2021-03-24T22:06:40.460548+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:52.372027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:05:52.372027+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"867","reported_epoch":"831","state":"active+clean","last_fresh":"2021-03-24T22:06:41.458276+0000","last_change":"2021-03-24T22:06:02.476829+0000","last_active":"2021-03-24T22:06:41.458276+0000","last_peered":"2021-03-24T22:06:41.458276+0000","last_clean":"2021-03-24T22:06:41.458276+0000","last_became_active":"2021-03-24T22:03:45.431424+0000","last_became_peered":"2021-03-24T22:03:45.431424+0000","last_unstale":"2021-03-24T22:06:41.458276+0000","last_undegraded":"2021-03-24T22:06:41.458276+0000","last_fullsized":"2021-03-24T22:06:41.458276+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:02.476582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:06:02.476582+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.19","version":"830'1277","reported_seq":"3893","reported_epoch":"830","state":"active+clean","last_fresh":"2021-03-24T22:06:40.463760+0000","last_change":"2021-03-24T22:06:40.464307+0000","last_active":"2021-03-24T22:06:40.463760+0000","last_peered":"2021-03-24T22:06:40.463760+0000","last_clean":"2021-03-24T22:06:40.463760+0000","last_became_active":"2021-03-24T22:02:07.304440+0000","last_became_peered":"2021-03-24T22:02:07.304440+0000","last_unstale":"2021-03-24T22:06:40.463760+0000","last_undegraded":"2021-03-24T22:06:40.463760+0000","last_fullsized":"2021-03-24T22:06:40.463760+0000","mapping_epoch":556,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":557,"parent":"0.0","parent_split_bits":5,"last_scrub":"664'844","last_scrub_stamp":"2021-03-24T22:03:55.671094+0000","last_deep_scrub":"455'480","last_deep_scrub_stamp":"2021-03-24T22:00:26.193730+0000","last_clean_scrub_stamp":"2021-03-24T22:03:55.671094+0000","log_size":1277,"ondisk_log_size":1277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31937817,"num_objects":16,"num_object_clones":13,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":1,"num_read":251,"num_read_kb":216920,"num_write":231,"num_write_kb":94479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":265,"num_bytes_recovered":154749512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"27d","length":"1"},{"start":"297","length":"1"},{"start":"2ab","length":"1"},{"start":"2ad","length":"1"},{"start":"2c4","length":"1"}]},{"pgid":"3.1e","version":"0'0","reported_seq":"855","reported_epoch":"832","state":"active+clean","last_fresh":"2021-03-24T22:06:42.473706+0000","last_change":"2021-03-24T22:04:27.240608+0000","last_active":"2021-03-24T22:06:42.473706+0000","last_peered":"2021-03-24T22:06:42.473706+0000","last_clean":"2021-03-24T22:06:42.473706+0000","last_became_active":"2021-03-24T22:01:43.381824+0000","last_became_peered":"2021-03-24T22:01:43.381824+0000","last_unstale":"2021-03-24T22:06:42.473706+0000","last_undegraded":"2021-03-24T22:06:42.473706+0000","last_fullsized":"2021-03-24T22:06:42.473706+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:04:27.240496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:04:27.240496+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"329","reported_epoch":"829","state":"active+clean","last_fresh":"2021-03-24T22:06:39.462260+0000","last_change":"2021-03-24T22:05:49.398497+0000","last_active":"2021-03-24T22:06:39.462260+0000","last_peered":"2021-03-24T22:06:39.462260+0000","last_clean":"2021-03-24T22:06:39.462260+0000","last_became_active":"2021-03-24T22:03:45.416917+0000","last_became_peered":"2021-03-24T22:03:45.416917+0000","last_unstale":"2021-03-24T22:06:39.462260+0000","last_undegraded":"2021-03-24T22:06:39.462260+0000","last_fullsized":"2021-03-24T22:06:39.462260+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:49.398341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:05:49.398341+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"863","reported_epoch":"832","state":"active+clean","last_fresh":"2021-03-24T22:06:42.470179+0000","last_change":"2021-03-24T22:03:08.392175+0000","last_active":"2021-03-24T22:06:42.470179+0000","last_peered":"2021-03-24T22:06:42.470179+0000","last_clean":"2021-03-24T22:06:42.470179+0000","last_became_active":"2021-03-24T22:01:57.310369+0000","last_became_peered":"2021-03-24T22:01:57.310369+0000","last_unstale":"2021-03-24T22:06:42.470179+0000","last_undegraded":"2021-03-24T22:06:42.470179+0000","last_fullsized":"2021-03-24T22:06:42.470179+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:08.392060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:03:08.392060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"829'432","reported_seq":"2850","reported_epoch":"830","state":"active+clean","last_fresh":"2021-03-24T22:06:40.465293+0000","last_change":"2021-03-24T22:06:40.465706+0000","last_active":"2021-03-24T22:06:40.465293+0000","last_peered":"2021-03-24T22:06:40.465293+0000","last_clean":"2021-03-24T22:06:40.465293+0000","last_became_active":"2021-03-24T22:05:04.454694+0000","last_became_peered":"2021-03-24T22:05:04.454694+0000","last_unstale":"2021-03-24T22:06:40.465293+0000","last_undegraded":"2021-03-24T22:06:40.465293+0000","last_fullsized":"2021-03-24T22:06:40.465293+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":5,"last_scrub":"821'428","last_scrub_stamp":"2021-03-24T22:06:35.291286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:06:35.291286+0000","log_size":432,"ondisk_log_size":432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12745920,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"27d","length":"1"},{"start":"297","length":"1"},{"start":"2ab","length":"1"},{"start":"2ad","length":"1"},{"start":"2c4","length":"1"}]},{"pgid":"3.1f","version":"0'0","reported_seq":"854","reported_epoch":"831","state":"active+clean","last_fresh":"2021-03-24T22:06:41.458509+0000","last_change":"2021-03-24T22:05:48.437882+0000","last_active":"2021-03-24T22:06:41.458509+0000","last_peered":"2021-03-24T22:06:41.458509+0000","last_clean":"2021-03-24T22:06:41.458509+0000","last_became_active":"2021-03-24T22:05:04.457596+0000","last_became_peered":"2021-03-24T22:05:04.457596+0000","last_unstale":"2021-03-24T22:06:41.458509+0000","last_undegraded":"2021-03-24T22:06:41.458509+0000","last_fullsized":"2021-03-24T22:06:41.458509+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:48.437760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:05:48.437760+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"204","reported_epoch":"828","state":"active+clean","last_fresh":"2021-03-24T22:06:38.450768+0000","last_change":"2021-03-24T22:05:11.459315+0000","last_active":"2021-03-24T22:06:38.450768+0000","last_peered":"2021-03-24T22:06:38.450768+0000","last_clean":"2021-03-24T22:06:38.450768+0000","last_became_active":"2021-03-24T22:05:04.447664+0000","last_became_peered":"2021-03-24T22:05:04.447664+0000","last_unstale":"2021-03-24T22:06:38.450768+0000","last_undegraded":"2021-03-24T22:06:38.450768+0000","last_fullsized":"2021-03-24T22:06:38.450768+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:11.459219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:14.710709+0000","last_clean_scrub_stamp":"2021-03-24T22:05:11.459219+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"756","reported_epoch":"830","state":"active+clean","last_fresh":"2021-03-24T22:06:40.457358+0000","last_change":"2021-03-24T22:05:46.303724+0000","last_active":"2021-03-24T22:06:40.457358+0000","last_peered":"2021-03-24T22:06:40.457358+0000","last_clean":"2021-03-24T22:06:40.457358+0000","last_became_active":"2021-03-24T22:05:04.443147+0000","last_became_peered":"2021-03-24T22:05:04.443147+0000","last_unstale":"2021-03-24T22:06:40.457358+0000","last_undegraded":"2021-03-24T22:06:40.457358+0000","last_fullsized":"2021-03-24T22:06:40.457358+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:46.303541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:05:46.303541+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1b","version":"832'2051","reported_seq":"4401","reported_epoch":"832","state":"active+clean","last_fresh":"2021-03-24T22:06:42.517764+0000","last_change":"2021-03-24T22:06:41.525056+0000","last_active":"2021-03-24T22:06:42.517764+0000","last_peered":"2021-03-24T22:06:42.517764+0000","last_clean":"2021-03-24T22:06:42.517764+0000","last_became_active":"2021-03-24T21:57:33.176110+0000","last_became_peered":"2021-03-24T21:57:33.176110+0000","last_unstale":"2021-03-24T22:06:42.517764+0000","last_undegraded":"2021-03-24T22:06:42.517764+0000","last_fullsized":"2021-03-24T22:06:42.517764+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":5,"last_scrub":"710'1638","last_scrub_stamp":"2021-03-24T22:04:41.216144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:04:41.216144+0000","log_size":2051,"ondisk_log_size":2051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":70613917,"num_objects":29,"num_object_clones":23,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":1,"num_read":329,"num_read_kb":345561,"num_write":256,"num_write_kb":115482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"255","length":"1"},{"start":"27d","length":"1"},{"start":"297","length":"1"},{"start":"2ab","length":"1"},{"start":"2ad","length":"1"},{"start":"2c4","length":"1"}]},{"pgid":"3.24","version":"0'0","reported_seq":"461","reported_epoch":"832","state":"active+clean","last_fresh":"2021-03-24T22:06:42.473142+0000","last_change":"2021-03-24T22:03:41.269971+0000","last_active":"2021-03-24T22:06:42.473142+0000","last_peered":"2021-03-24T22:06:42.473142+0000","last_clean":"2021-03-24T22:06:42.473142+0000","last_became_active":"2021-03-24T21:59:14.369606+0000","last_became_peered":"2021-03-24T21:59:14.369606+0000","last_unstale":"2021-03-24T22:06:42.473142+0000","last_undegraded":"2021-03-24T22:06:42.473142+0000","last_fullsized":"2021-03-24T22:06:42.473142+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:41.269900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:03:41.269900+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"175'22","reported_seq":"1997","reported_epoch":"831","state":"active+clean","last_fresh":"2021-03-24T22:06:41.478296+0000","last_change":"2021-03-24T22:06:41.481930+0000","last_active":"2021-03-24T22:06:41.478296+0000","last_peered":"2021-03-24T22:06:41.478296+0000","last_clean":"2021-03-24T22:06:41.478296+0000","last_became_active":"2021-03-24T22:03:45.432211+0000","last_became_peered":"2021-03-24T22:03:45.432211+0000","last_unstale":"2021-03-24T22:06:41.478296+0000","last_undegraded":"2021-03-24T22:06:41.478296+0000","last_fullsized":"2021-03-24T22:06:41.478296+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":6,"last_scrub":"175'22","last_scrub_stamp":"2021-03-24T22:04:56.629000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:04:56.629000+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"255","length":"1"},{"start":"27d","length":"1"},{"start":"297","length":"1"},{"start":"2ab","length":"1"},{"start":"2ad","length":"1"},{"start":"2c4","length":"1"}]},{"pgid":"3.25","version":"0'0","reported_seq":"194","reported_epoch":"828","state":"active+clean","last_fresh":"2021-03-24T22:06:38.452748+0000","last_change":"2021-03-24T22:06:36.792829+0000","last_active":"2021-03-24T22:06:38.452748+0000","last_peered":"2021-03-24T22:06:38.452748+0000","last_clean":"2021-03-24T22:06:38.452748+0000","last_became_active":"2021-03-24T22:03:45.426841+0000","last_became_peered":"2021-03-24T22:03:45.426841+0000","last_unstale":"2021-03-24T22:06:38.452748+0000","last_undegraded":"2021-03-24T22:06:38.452748+0000","last_fullsized":"2021-03-24T22:06:38.452748+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:36.792559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:06:36.792559+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.21","version":"831'845","reported_seq":"3490","reported_epoch":"832","state":"active+clean","last_fresh":"2021-03-24T22:06:42.469621+0000","last_change":"2021-03-24T22:06:41.484874+0000","last_active":"2021-03-24T22:06:42.469621+0000","last_peered":"2021-03-24T22:06:42.469621+0000","last_clean":"2021-03-24T22:06:42.469621+0000","last_became_active":"2021-03-24T22:05:04.434209+0000","last_became_peered":"2021-03-24T22:05:04.434209+0000","last_unstale":"2021-03-24T22:06:42.469621+0000","last_undegraded":"2021-03-24T22:06:42.469621+0000","last_fullsized":"2021-03-24T22:06:42.469621+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":845,"ondisk_log_size":845,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36382753,"num_objects":15,"num_object_clones":14,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":226,"num_read_kb":218576,"num_write":181,"num_write_kb":70308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":68248426,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"255","length":"1"},{"start":"27d","length":"1"},{"start":"297","length":"1"},{"start":"2ab","length":"1"},{"start":"2ad","length":"1"},{"start":"2c4","length":"1"}]}],"pool_stats":[{"poolid":7,"num_pg":36,"stat_sum":{"num_bytes":546100387,"num_objects":234,"num_object_clones":184,"num_object_copies":702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":10,"num_read":3130,"num_read_kb":3288376,"num_write":2273,"num_write_kb":877905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2178,"num_bytes_recovered":1222469600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1216774144,"data_stored":1651690480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20682,"ondisk_log_size":20682,"up":108,"acting":108,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":358,"num_read_kb":353,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":3,"num_pg":42,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10235,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":126,"acting":126,"num_store_stats":8},{"poolid":4,"num_pg":22,"stat_sum":{"num_bytes":23524,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1475,"num_read_kb":980,"num_write":828,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1194,"num_bytes_recovered":94630,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1990656,"data_stored":71374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":514,"ondisk_log_size":514,"up":66,"acting":66,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201464,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":259672,"kb_used_data":205712,"kb_used_omap":0,"kb_used_meta":53952,"kb_avail":93485480,"statfs":{"total":95995035648,"available":95729131520,"internally_reserved":0,"allocated":210649088,"data_stored":289951102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":55246848},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 22:06:17 2021","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":2.5150000000000001,"15min":1.4510000000000001},"min":{"1min":0.57699999999999996,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.7410000000000001,"5min":150.68000000000001,"15min":150.68000000000001},"last":0.96099999999999997},{"interface":"front","average":{"1min":0.97099999999999997,"5min":2.5699999999999998,"15min":1.5429999999999999},"min":{"1min":0.52300000000000002,"5min":0.36299999999999999,"15min":0.27000000000000002},"max":{"1min":2.2639999999999998,"5min":153.91,"15min":153.91},"last":1.085}]},{"osd":1,"last update":"Wed Mar 24 22:06:17 2021","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":0.83999999999999997,"15min":0.88500000000000001},"min":{"1min":0.59599999999999997,"5min":0.318,"15min":0.308},"max":{"1min":1.8180000000000001,"5min":1.8180000000000001,"15min":5.7750000000000004},"last":1.254},{"interface":"front","average":{"1min":1.0089999999999999,"5min":0.878,"15min":0.94999999999999996},"min":{"1min":0.61699999999999999,"5min":0.316,"15min":0.316},"max":{"1min":1.623,"5min":1.623,"15min":6.1849999999999996},"last":0.89100000000000001}]},{"osd":2,"last update":"Wed Mar 24 22:06:17 2021","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.83999999999999997,"15min":0.98099999999999998},"min":{"1min":0.69299999999999995,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.982,"5min":1.982,"15min":12.135999999999999},"last":1.0249999999999999},{"interface":"front","average":{"1min":1.0449999999999999,"5min":0.92500000000000004,"15min":1.0149999999999999},"min":{"1min":0.56399999999999995,"5min":0.45900000000000002,"15min":0.42999999999999999},"max":{"1min":1.881,"5min":1.881,"15min":12.178000000000001},"last":1.3660000000000001}]},{"osd":3,"last update":"Wed Mar 24 22:06:17 2021","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":0.998,"15min":1.0640000000000001},"min":{"1min":0.81699999999999995,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":2.04,"5min":6.1449999999999996,"15min":12.529},"last":1.1990000000000001},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.0329999999999999,"15min":1.081},"min":{"1min":0.66700000000000004,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.9159999999999999,"5min":3.0510000000000002,"15min":12.666},"last":1.0529999999999999}]},{"osd":4,"last update":"Wed Mar 24 22:06:17 2021","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":0.96199999999999997,"15min":0.98799999999999999},"min":{"1min":0.80600000000000005,"5min":0.34799999999999998,"15min":0.33500000000000002},"max":{"1min":1.714,"5min":1.714,"15min":6.2089999999999996},"last":1.1779999999999999},{"interface":"front","average":{"1min":1.1160000000000001,"5min":1.012,"15min":1.0720000000000001},"min":{"1min":0.89400000000000002,"5min":0.42399999999999999,"15min":0.41399999999999998},"max":{"1min":2.1360000000000001,"5min":2.1360000000000001,"15min":12.244},"last":1.4410000000000001}]},{"osd":5,"last update":"Wed Mar 24 22:06:07 2021","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":1.145},{"interface":"front","average":{"1min":1.149,"5min":1.149,"15min":1.149},"min":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":1.444,"5min":1.444,"15min":1.444},"last":1.399}]},{"osd":6,"last update":"Wed Mar 24 22:06:17 2021","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.101,"15min":1.137},"min":{"1min":0.72299999999999998,"5min":0.51800000000000002,"15min":0.40899999999999997},"max":{"1min":1.7370000000000001,"5min":1.7370000000000001,"15min":6.1159999999999997},"last":1.2949999999999999},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.111,"15min":1.165},"min":{"1min":0.80400000000000005,"5min":0.502,"15min":0.434},"max":{"1min":2.0830000000000002,"5min":2.0830000000000002,"15min":12.287000000000001},"last":1.3340000000000001}]}]},{"osd":6,"up_from":47,"seq":201863463100,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":188084,"kb_used_data":148268,"kb_used_omap":0,"kb_used_meta":39808,"kb_avail":93557068,"statfs":{"total":95995035648,"available":95802437632,"internally_reserved":0,"allocated":151826432,"data_stored":206793490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40763392},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 22:05:43 2021","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":1.054,"15min":1.008},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.30399999999999999},"max":{"1min":1.5629999999999999,"5min":19.257999999999999,"15min":19.257999999999999},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.81699999999999995,"5min":1.077,"15min":0.98999999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":1.673,"5min":16.795000000000002,"15min":16.795000000000002},"last":1.008}]},{"osd":1,"last update":"Wed Mar 24 22:05:43 2021","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":1.1619999999999999,"15min":0.999},"min":{"1min":0.39000000000000001,"5min":0.38400000000000001,"15min":0.34799999999999998},"max":{"1min":1.0629999999999999,"5min":21.995999999999999,"15min":21.995999999999999},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.91200000000000003,"5min":1.2609999999999999,"15min":1.0920000000000001},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.3220000000000001,"5min":26.762,"15min":26.762},"last":1.1319999999999999}]},{"osd":2,"last update":"Wed Mar 24 22:05:43 2021","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":1.117,"15min":1.0129999999999999},"min":{"1min":0.51000000000000001,"5min":0.42399999999999999,"15min":0.35299999999999998},"max":{"1min":2.1419999999999999,"5min":13.044,"15min":13.044},"last":1.2569999999999999},{"interface":"front","average":{"1min":0.93500000000000005,"5min":1.3560000000000001,"15min":1.0960000000000001},"min":{"1min":0.435,"5min":0.41199999999999998,"15min":0.33500000000000002},"max":{"1min":1.732,"5min":31.375,"15min":31.375},"last":0.93500000000000005}]},{"osd":3,"last update":"Wed Mar 24 22:05:43 2021","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":1.2909999999999999,"15min":1.0860000000000001},"min":{"1min":0.501,"5min":0.47899999999999998,"15min":0.38400000000000001},"max":{"1min":1.5740000000000001,"5min":22.07,"15min":22.07},"last":0.98199999999999998},{"interface":"front","average":{"1min":1.05,"5min":1.157,"15min":1.109},"min":{"1min":0.51900000000000002,"5min":0.48599999999999999,"15min":0.41299999999999998},"max":{"1min":1.974,"5min":11.583,"15min":11.583},"last":1.05}]},{"osd":4,"last update":"Wed Mar 24 22:05:43 2021","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":1.0860000000000001,"15min":1.028},"min":{"1min":0.48799999999999999,"5min":0.44700000000000001,"15min":0.27200000000000002},"max":{"1min":1.3839999999999999,"5min":8.4350000000000005,"15min":8.4350000000000005},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.98999999999999999,"5min":1.2729999999999999,"15min":1.1319999999999999},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.42199999999999999},"max":{"1min":1.3109999999999999,"5min":22.033000000000001,"15min":22.033000000000001},"last":1.292}]},{"osd":5,"last update":"Wed Mar 24 22:06:04 2021","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.087,"15min":1.087},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.976,"5min":1.976,"15min":1.976},"last":1.0820000000000001},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":1.1140000000000001}]},{"osd":7,"last update":"Wed Mar 24 22:06:07 2021","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.1719999999999999,"15min":1.1899999999999999},"min":{"1min":0.57299999999999995,"5min":0.46999999999999997,"15min":0.30499999999999999},"max":{"1min":1.403,"5min":8.3599999999999994,"15min":9.5909999999999993},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.135,"5min":1.159,"15min":1.1779999999999999},"min":{"1min":0.59099999999999997,"5min":0.51300000000000001,"15min":0.442},"max":{"1min":1.353,"5min":8.7959999999999994,"15min":8.7959999999999994},"last":1.325}]}]},{"osd":1,"up_from":13,"seq":55834575057,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":111984,"kb_used_data":82472,"kb_used_omap":0,"kb_used_meta":29504,"kb_avail":93633168,"statfs":{"total":95995035648,"available":95880364032,"internally_reserved":0,"allocated":84451328,"data_stored":105572402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":30212096},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 22:06:03 2021","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.82499999999999996,"15min":0.83299999999999996},"min":{"1min":0.32700000000000001,"5min":0.27700000000000002,"15min":0.20899999999999999},"max":{"1min":2.0110000000000001,"5min":11.189,"15min":21.085000000000001},"last":1.169},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.92400000000000004,"15min":0.80700000000000005},"min":{"1min":0.35799999999999998,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.7430000000000001,"5min":8.5180000000000007,"15min":8.5180000000000007},"last":0.79400000000000004}]},{"osd":2,"last update":"Wed Mar 24 22:06:11 2021","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.77600000000000002,"15min":0.70399999999999996},"min":{"1min":0.38600000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.792,"5min":9.9870000000000001,"15min":9.9870000000000001},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.97899999999999998,"15min":0.85099999999999998},"min":{"1min":0.40400000000000003,"5min":0.29599999999999999,"15min":0.27200000000000002},"max":{"1min":1.415,"5min":9.2059999999999995,"15min":9.2059999999999995},"last":0.82799999999999996}]},{"osd":3,"last update":"Wed Mar 24 22:05:45 2021","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":1.0269999999999999,"15min":0.95499999999999996},"min":{"1min":0.38800000000000001,"5min":0.37,"15min":0.29299999999999998},"max":{"1min":0.92900000000000005,"5min":8.5990000000000002,"15min":17.032},"last":0.755},{"interface":"front","average":{"1min":0.82199999999999995,"5min":1.103,"15min":1.0169999999999999},"min":{"1min":0.49299999999999999,"5min":0.33800000000000002,"15min":0.316},"max":{"1min":2.1349999999999998,"5min":15.775,"15min":16.908999999999999},"last":0.96899999999999997}]},{"osd":4,"last update":"Wed Mar 24 22:06:11 2021","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":1.0800000000000001,"15min":0.98899999999999999},"min":{"1min":0.63,"5min":0.49099999999999999,"15min":0.34399999999999997},"max":{"1min":1.2969999999999999,"5min":9.6150000000000002,"15min":9.6150000000000002},"last":1.0189999999999999},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.075,"15min":1.0629999999999999},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.35599999999999998},"max":{"1min":1.5209999999999999,"5min":3.9359999999999999,"15min":3.9359999999999999},"last":1.03}]},{"osd":5,"last update":"Wed Mar 24 22:06:07 2021","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.6379999999999999,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":0.92100000000000004},{"interface":"front","average":{"1min":1.1719999999999999,"5min":1.1719999999999999,"15min":1.1719999999999999},"min":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"max":{"1min":2.0750000000000002,"5min":2.0750000000000002,"15min":2.0750000000000002},"last":1.347}]},{"osd":6,"last update":"Wed Mar 24 22:05:45 2021","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.2649999999999999,"15min":1.194},"min":{"1min":0.63400000000000001,"5min":0.49399999999999999,"15min":0.48799999999999999},"max":{"1min":2.0459999999999998,"5min":11.254,"15min":11.254},"last":1.2789999999999999},{"interface":"front","average":{"1min":1.052,"5min":1.1850000000000001,"15min":1.131},"min":{"1min":0.70099999999999996,"5min":0.51900000000000002,"15min":0.38100000000000001},"max":{"1min":1.5329999999999999,"5min":9.7300000000000004,"15min":9.7300000000000004},"last":0.78300000000000003}]},{"osd":7,"last update":"Wed Mar 24 22:06:11 2021","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.159,"15min":1.177},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.38300000000000001},"max":{"1min":1.706,"5min":3.8439999999999999,"15min":4.4699999999999998},"last":0.877},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.2909999999999999,"15min":1.2250000000000001},"min":{"1min":0.70799999999999996,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":2.105,"5min":11.289,"15min":11.289},"last":1.3120000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738581,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":205856,"kb_used_data":152664,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":93539296,"statfs":{"total":95995035648,"available":95784239104,"internally_reserved":0,"allocated":156327936,"data_stored":207172663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Mar 24 22:06:06 2021","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.754,"15min":0.70199999999999996},"min":{"1min":0.253,"5min":0.253,"15min":0.249},"max":{"1min":1.0429999999999999,"5min":5.0419999999999998,"15min":5.0419999999999998},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.71299999999999997,"15min":0.76000000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.0660000000000001,"5min":5.4320000000000004,"15min":8.8989999999999991},"last":0.60399999999999998}]},{"osd":2,"last update":"Wed Mar 24 22:06:25 2021","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.68300000000000005,"15min":0.77700000000000002},"min":{"1min":0.47799999999999998,"5min":0.308,"15min":0.28999999999999998},"max":{"1min":0.88700000000000001,"5min":3.8980000000000001,"15min":15.949999999999999},"last":0.438},{"interface":"front","average":{"1min":0.753,"5min":0.70099999999999996,"15min":0.77400000000000002},"min":{"1min":0.40400000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.0960000000000001,"5min":1.3580000000000001,"15min":4.766},"last":0.5}]},{"osd":3,"last update":"Wed Mar 24 22:05:41 2021","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":1.121,"15min":0.95599999999999996},"min":{"1min":0.40899999999999997,"5min":0.33800000000000002,"15min":0.28999999999999998},"max":{"1min":1.1659999999999999,"5min":24.75,"15min":24.75},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.89400000000000002,"5min":1.117,"15min":1.008},"min":{"1min":0.40300000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":1.4890000000000001,"5min":24.832999999999998,"15min":24.832999999999998},"last":0.56200000000000006}]},{"osd":4,"last update":"Wed Mar 24 22:06:06 2021","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":0.95599999999999996,"15min":1.0369999999999999},"min":{"1min":0.45500000000000002,"5min":0.373,"15min":0.373},"max":{"1min":1.3360000000000001,"5min":1.4790000000000001,"15min":5.0330000000000004},"last":0.63700000000000001},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.111,"15min":1.0920000000000001},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.44900000000000001},"max":{"1min":1.663,"5min":12.366,"15min":12.366},"last":1.04}]},{"osd":5,"last update":"Wed Mar 24 22:06:06 2021","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.522,"5min":1.522,"15min":1.522},"last":0.78200000000000003},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.746,"5min":1.746,"15min":1.746},"last":1.175}]},{"osd":6,"last update":"Wed Mar 24 22:05:41 2021","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.0089999999999999,"15min":1.1379999999999999},"min":{"1min":0.56000000000000005,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.542,"5min":3.9529999999999998,"15min":15.983000000000001},"last":0.73199999999999998},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.137,"15min":1.157},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.53400000000000003},"max":{"1min":1.9970000000000001,"5min":5.2699999999999996,"15min":5.2699999999999996},"last":0.85199999999999998}]},{"osd":7,"last update":"Wed Mar 24 22:06:16 2021","interfaces":[{"interface":"back","average":{"1min":1.2170000000000001,"5min":1.127,"15min":1.1639999999999999},"min":{"1min":0.67600000000000005,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.8100000000000001,"5min":4.9470000000000001,"15min":6.1319999999999997},"last":1.129},{"interface":"front","average":{"1min":1.254,"5min":1.077,"15min":1.1850000000000001},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":2.6890000000000001,"5min":3.984,"15min":16.007000000000001},"last":0.83799999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411534,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":230872,"kb_used_data":177936,"kb_used_omap":0,"kb_used_meta":52928,"kb_avail":93514280,"statfs":{"total":95995035648,"available":95758622720,"internally_reserved":0,"allocated":182206464,"data_stored":243499431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":54198272},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 22:06:18 2021","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.80900000000000005,"15min":0.76000000000000001},"min":{"1min":0.32400000000000001,"5min":0.254,"15min":0.254},"max":{"1min":1.052,"5min":7.468,"15min":10.054},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.78100000000000003,"15min":0.78600000000000003},"min":{"1min":0.46100000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":2.302,"5min":7.8979999999999997,"15min":12.099},"last":0.66400000000000003}]},{"osd":1,"last update":"Wed Mar 24 22:06:18 2021","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.88700000000000001,"15min":0.84399999999999997},"min":{"1min":0.41199999999999998,"5min":0.32600000000000001,"15min":0.28100000000000003},"max":{"1min":1.0269999999999999,"5min":12.343999999999999,"15min":17.954999999999998},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.86099999999999999,"15min":0.79900000000000004},"min":{"1min":0.36399999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":2.1190000000000002,"5min":7.4989999999999997,"15min":7.4989999999999997},"last":0.79500000000000004}]},{"osd":3,"last update":"Wed Mar 24 22:05:45 2021","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.97099999999999997,"15min":0.97399999999999998},"min":{"1min":0.44900000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.181,"5min":12.308,"15min":25.009},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.97499999999999998,"15min":0.92300000000000004},"min":{"1min":0.50600000000000001,"5min":0.34499999999999997,"15min":0.33900000000000002},"max":{"1min":1.5489999999999999,"5min":7.5199999999999996,"15min":7.5199999999999996},"last":1.2569999999999999}]},{"osd":4,"last update":"Wed Mar 24 22:06:07 2021","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":1.1140000000000001,"15min":1.1339999999999999},"min":{"1min":0.68000000000000005,"5min":0.45300000000000001,"15min":0.44400000000000001},"max":{"1min":2.6589999999999998,"5min":12.467000000000001,"15min":24.914000000000001},"last":0.91800000000000004},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.0229999999999999,"15min":1.034},"min":{"1min":0.65200000000000002,"5min":0.47199999999999998,"15min":0.41299999999999998},"max":{"1min":1.4419999999999999,"5min":2.8290000000000002,"15min":2.8290000000000002},"last":1.0349999999999999}]},{"osd":5,"last update":"Wed Mar 24 22:06:07 2021","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.139,"15min":1.139},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.639,"5min":1.639,"15min":1.639},"last":1.075},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":2.6869999999999998,"5min":2.6869999999999998,"15min":2.6869999999999998},"last":0.86899999999999999}]},{"osd":6,"last update":"Wed Mar 24 22:05:45 2021","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.0700000000000001,"15min":1.1060000000000001},"min":{"1min":0.69799999999999995,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.7110000000000001,"5min":2.7719999999999998,"15min":2.7719999999999998},"last":1.129},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.1020000000000001,"15min":1.121},"min":{"1min":0.84999999999999998,"5min":0.52500000000000002,"15min":0.47899999999999998},"max":{"1min":1.452,"5min":4.3369999999999997,"15min":6.6520000000000001},"last":1.1060000000000001}]},{"osd":7,"last update":"Wed Mar 24 22:06:09 2021","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.167,"15min":1.1839999999999999},"min":{"1min":0.73099999999999998,"5min":0.51800000000000002,"15min":0.442},"max":{"1min":2.6269999999999998,"5min":4.3630000000000004,"15min":6.7720000000000002},"last":1.0509999999999999},{"interface":"front","average":{"1min":1.202,"5min":1.258,"15min":1.29},"min":{"1min":0.73999999999999999,"5min":0.505,"15min":0.41999999999999998},"max":{"1min":2.73,"5min":12.492000000000001,"15min":24.969999999999999},"last":0.84499999999999997}]}]},{"osd":3,"up_from":26,"seq":111669149898,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":325056,"kb_used_data":260280,"kb_used_omap":0,"kb_used_meta":64768,"kb_avail":93420096,"statfs":{"total":95995035648,"available":95662178304,"internally_reserved":0,"allocated":266526720,"data_stored":353940079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":66322432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 22:06:43 2021","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.81899999999999995,"15min":0.91500000000000004},"min":{"1min":0.51200000000000001,"5min":0.29699999999999999,"15min":0.23899999999999999},"max":{"1min":3.2959999999999998,"5min":13.829000000000001,"15min":27.548999999999999},"last":0.53400000000000003},{"interface":"front","average":{"1min":1.165,"5min":0.83299999999999996,"15min":0.93400000000000005},"min":{"1min":0.495,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":6.8310000000000004,"5min":6.8310000000000004,"15min":26.593},"last":0.56599999999999995}]},{"osd":1,"last update":"Wed Mar 24 22:06:43 2021","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":0.81000000000000005,"15min":0.80300000000000005},"min":{"1min":0.46200000000000002,"5min":0.314,"15min":0.26700000000000002},"max":{"1min":3.802,"5min":5.2610000000000001,"15min":14.324},"last":0.70299999999999996},{"interface":"front","average":{"1min":1.081,"5min":0.86299999999999999,"15min":0.81200000000000006},"min":{"1min":0.51000000000000001,"5min":0.28899999999999998,"15min":0.28000000000000003},"max":{"1min":5.2910000000000004,"5min":5.2910000000000004,"15min":12.631},"last":0.84099999999999997}]},{"osd":2,"last update":"Wed Mar 24 22:06:43 2021","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.81399999999999995,"15min":0.85299999999999998},"min":{"1min":0.45900000000000002,"5min":0.34699999999999998,"15min":0.316},"max":{"1min":2.827,"5min":5.0439999999999996,"15min":12.510999999999999},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.84899999999999998,"15min":0.85599999999999998},"min":{"1min":0.50900000000000001,"5min":0.42799999999999999,"15min":0.34899999999999998},"max":{"1min":2.7170000000000001,"5min":4.5640000000000001,"15min":11.317},"last":0.67300000000000004}]},{"osd":4,"last update":"Wed Mar 24 22:06:02 2021","interfaces":[{"interface":"back","average":{"1min":1.204,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.89500000000000002,"5min":0.622,"15min":0.42999999999999999},"max":{"1min":2.5880000000000001,"5min":2.5880000000000001,"15min":2.601},"last":0.73099999999999998},{"interface":"front","average":{"1min":1.54,"5min":1.1060000000000001,"15min":1.0640000000000001},"min":{"1min":0.81799999999999995,"5min":0.47899999999999998,"15min":0.32900000000000001},"max":{"1min":4.0700000000000003,"5min":4.9569999999999999,"15min":14.291},"last":0.94699999999999995}]},{"osd":5,"last update":"Wed Mar 24 22:06:08 2021","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"min":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":2.7400000000000002,"5min":2.7400000000000002,"15min":2.7400000000000002},"last":0.97399999999999998},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"min":{"1min":0.879,"5min":0.879,"15min":0.879},"max":{"1min":2.6949999999999998,"5min":2.6949999999999998,"15min":2.6949999999999998},"last":1.022}]},{"osd":6,"last update":"Wed Mar 24 22:06:43 2021","interfaces":[{"interface":"back","average":{"1min":1.45,"5min":1.135,"15min":1.1240000000000001},"min":{"1min":0.874,"5min":0.59799999999999998,"15min":0.40000000000000002},"max":{"1min":4.0369999999999999,"5min":4.9950000000000001,"15min":14.255000000000001},"last":1.004},{"interface":"front","average":{"1min":1.286,"5min":1.0720000000000001,"15min":1.117},"min":{"1min":0.78100000000000003,"5min":0.54800000000000004,"15min":0.39700000000000002},"max":{"1min":3.5859999999999999,"5min":3.5859999999999999,"15min":12.666},"last":1.2729999999999999}]},{"osd":7,"last update":"Wed Mar 24 22:06:08 2021","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.089,"15min":1.1180000000000001},"min":{"1min":0.81499999999999995,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":2.2189999999999999,"5min":2.2189999999999999,"15min":2.633},"last":1.077},{"interface":"front","average":{"1min":1.5960000000000001,"5min":1.216,"15min":1.23},"min":{"1min":0.98999999999999999,"5min":0.60399999999999998,"15min":0.44700000000000001},"max":{"1min":4.1159999999999997,"5min":5.0190000000000001,"15min":14.727},"last":1.0840000000000001}]}]},{"osd":4,"up_from":32,"seq":137438953668,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":115152,"kb_used_data":71304,"kb_used_omap":0,"kb_used_meta":43840,"kb_avail":93630000,"statfs":{"total":95995035648,"available":95877120000,"internally_reserved":0,"allocated":73015296,"data_stored":92386272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":44892160},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 22:06:09 2021","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":0.999,"15min":1.034},"min":{"1min":0.60999999999999999,"5min":0.437,"15min":0.223},"max":{"1min":1.373,"5min":5.9050000000000002,"15min":41.472000000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.95099999999999996,"15min":1.1499999999999999},"min":{"1min":0.60899999999999999,"5min":0.41999999999999998,"15min":0.34000000000000002},"max":{"1min":1.3100000000000001,"5min":5.3719999999999999,"15min":44.185000000000002},"last":0.52700000000000002}]},{"osd":1,"last update":"Wed Mar 24 22:06:09 2021","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":1.0189999999999999,"15min":0.94999999999999996},"min":{"1min":0.57999999999999996,"5min":0.42699999999999999,"15min":0.38100000000000001},"max":{"1min":1.3460000000000001,"5min":7.5060000000000002,"15min":7.5060000000000002},"last":0.79000000000000004},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.149,"15min":0.99199999999999999},"min":{"1min":0.70699999999999996,"5min":0.48399999999999999,"15min":0.36399999999999999},"max":{"1min":1.528,"5min":11.983000000000001,"15min":11.983000000000001},"last":0.76100000000000001}]},{"osd":2,"last update":"Wed Mar 24 22:06:09 2021","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.87,"15min":0.90500000000000003},"min":{"1min":0.66300000000000003,"5min":0.46300000000000002,"15min":0.35899999999999999},"max":{"1min":1.2929999999999999,"5min":3.21,"15min":4.4379999999999997},"last":0.62},{"interface":"front","average":{"1min":0.995,"5min":1.0009999999999999,"15min":0.98899999999999999},"min":{"1min":0.626,"5min":0.439,"15min":0.40500000000000003},"max":{"1min":1.3640000000000001,"5min":4.3970000000000002,"15min":5.7999999999999998},"last":1.0029999999999999}]},{"osd":3,"last update":"Wed Mar 24 22:06:09 2021","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":1.085,"15min":1.038},"min":{"1min":0.64700000000000002,"5min":0.501,"15min":0.439},"max":{"1min":1.2869999999999999,"5min":3.9809999999999999,"15min":4.8049999999999997},"last":1.0740000000000001},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.0209999999999999,"15min":1.01},"min":{"1min":0.84999999999999998,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.3779999999999999,"5min":2.4700000000000002,"15min":4.1189999999999998},"last":0.55800000000000005}]},{"osd":5,"last update":"Wed Mar 24 22:06:03 2021","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.22,"5min":1.22,"15min":1.22},"last":0.878},{"interface":"front","average":{"1min":1.202,"5min":1.202,"15min":1.202},"min":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":1.9550000000000001,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.70999999999999996}]},{"osd":6,"last update":"Wed Mar 24 22:06:34 2021","interfaces":[{"interface":"back","average":{"1min":1.101,"5min":1.1919999999999999,"15min":1.0900000000000001},"min":{"1min":0.73299999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.7250000000000001,"5min":7.7679999999999998,"15min":7.7679999999999998},"last":0.91000000000000003},{"interface":"front","average":{"1min":1.246,"5min":1.3620000000000001,"15min":1.161},"min":{"1min":0.78000000000000003,"5min":0.53300000000000003,"15min":0.41099999999999998},"max":{"1min":2.923,"5min":14.122999999999999,"15min":14.122999999999999},"last":0.94899999999999995}]},{"osd":7,"last update":"Wed Mar 24 22:06:09 2021","interfaces":[{"interface":"back","average":{"1min":1.3089999999999999,"5min":1.196,"15min":1.147},"min":{"1min":0.87,"5min":0.46200000000000002,"15min":0.44400000000000001},"max":{"1min":1.8700000000000001,"5min":2.399,"15min":3.3479999999999999},"last":1.0309999999999999},{"interface":"front","average":{"1min":1.23,"5min":1.1160000000000001,"15min":1.1319999999999999},"min":{"1min":0.97299999999999998,"5min":0.49399999999999999,"15min":0.49299999999999999},"max":{"1min":1.7090000000000001,"5min":3.1680000000000001,"15min":3.1680000000000001},"last":0.81999999999999995}]}]},{"osd":5,"up_from":733,"seq":3148211027989,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":160512,"kb_used_data":147000,"kb_used_omap":164,"kb_used_meta":13339,"kb_avail":93584640,"statfs":{"total":95995035648,"available":95830671360,"internally_reserved":0,"allocated":150528000,"data_stored":188774611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":168054,"internal_metadata":13660042},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 22:06:03 2021","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.6699999999999999,"5min":1.6699999999999999,"15min":1.6699999999999999},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.4339999999999999,"5min":1.4339999999999999,"15min":1.4339999999999999},"last":0.60799999999999998}]},{"osd":1,"last update":"Wed Mar 24 22:06:03 2021","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.425,"5min":1.425,"15min":1.425},"last":1.139},{"interface":"front","average":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":2.0939999999999999,"5min":2.0939999999999999,"15min":2.0939999999999999},"last":0.95599999999999996}]},{"osd":2,"last update":"Wed Mar 24 22:06:03 2021","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.778,"5min":1.778,"15min":1.778},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.81499999999999995}]},{"osd":3,"last update":"Wed Mar 24 22:06:03 2021","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.669,"5min":1.669,"15min":1.669},"last":1.101},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"min":{"1min":0.755,"5min":0.755,"15min":0.755},"max":{"1min":1.863,"5min":1.863,"15min":1.863},"last":1.1910000000000001}]},{"osd":4,"last update":"Wed Mar 24 22:06:03 2021","interfaces":[{"interface":"back","average":{"1min":1.093,"5min":1.093,"15min":1.093},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":2.5219999999999998,"5min":2.5219999999999998,"15min":2.5219999999999998},"last":0.998},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":2.2450000000000001,"5min":2.2450000000000001,"15min":2.2450000000000001},"last":1.1639999999999999}]},{"osd":6,"last update":"Wed Mar 24 22:06:03 2021","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":1.131,"15min":1.131},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":2.0550000000000002,"5min":2.0550000000000002,"15min":2.0550000000000002},"last":1.054},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.74,"5min":1.74,"15min":1.74},"last":1.234}]},{"osd":7,"last update":"Wed Mar 24 22:06:03 2021","interfaces":[{"interface":"back","average":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":3.0430000000000001,"5min":3.0430000000000001,"15min":3.0430000000000001},"last":1.3129999999999999},{"interface":"front","average":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":2.698,"5min":2.698,"15min":2.698},"last":1.2889999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10235,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":5284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":475136,"data_stored":17122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":176128,"data_stored":6220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":241664,"data_stored":8622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":307200,"data_stored":11122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":5674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":10356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":6974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":149299200,"data_stored":203111351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77152256,"data_stored":101522047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":173236224,"data_stored":235500303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":259428352,"data_stored":349877141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":65839104,"data_stored":88320384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":143519744,"data_stored":184720864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":144715776,"data_stored":202749059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":203583488,"data_stored":285889331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-03-24T22:06:46.599 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2021-03-24T22:06:46.599 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-24T22:06:46.600 INFO:tasks.thrashosds.thrasher:fixing pg num pool iscsi 2021-03-24T22:06:46.601 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph pg dump --format=json 2021-03-24T22:06:46.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:46.602 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:46.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:06:46.704 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:46.718 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:46 smithi142 conmon[20125]: audit 2021-03-24T22:06:44.935340+0000 mgr.y (mgr.14144) 693 : 2021-03-24T22:06:46.719 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:46 smithi142 conmon[20125]: audit [DBG] from='client.15966 v1:172.21.15.125:0/1984436339' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T22:06:46.719 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:46 smithi142 conmon[20125]: cluster 2021-03-24T22:06:45.011398+0000 mgr.y (mgr.14144) 694 : cluster 2021-03-24T22:06:46.719 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:46 smithi142 conmon[20125]: [DBG] pgmap v1414: 168 pgs: 168 active+clean; 521 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 1.6 MiB/s wr, 15 op/s 2021-03-24T22:06:46.720 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:46 smithi142 conmon[20125]: audit 2021-03- 2021-03-24T22:06:46.720 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:46 smithi142 conmon[20125]: 24T22:06:45.686125+0000 mon.a (mon.0) 2021-03-24T22:06:46.720 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:46 smithi142 conmon[20125]: 2068 : audit [INF] from='client.? v1:172.21.15.125:0/2826679618' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:06:46.720 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:46 smithi142 conmon[20125]: cluster 2021-03-24T22 2021-03-24T22:06:46.721 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:46 smithi142 conmon[20125]: :06:46.160687+0000 osd.7 (osd. 2021-03-24T22:06:46.721 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:46 smithi142 conmon[20125]: 7) 140 : cluster [DBG] 7.11 scrub ok 2021-03-24T22:06:46.721 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:46 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:06:46.721 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:46 smithi142 conmon[20125]: 24T22:06:46.444145+0000 mon.a 2021-03-24T22:06:46.722 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:46 smithi142 conmon[20125]: (mon.0) 2069 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T22:06:46.722 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:46 smithi142 conmon[20125]: 2021-03-24T22:06:46.722 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:46 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:06:46.722 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:46 smithi142 conmon[20125]: -03-24T22:06:46.444215+0000 mon.a ( 2021-03-24T22:06:46.723 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:46 smithi142 conmon[20125]: mon.0) 2070 : cluster [INF] Cluster is now healthy 2021-03-24T22:06:46.723 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:46 smithi142 conmon[20125]: audit 2021- 2021-03-24T22:06:46.723 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:46 smithi142 conmon[20125]: 03-24T22:06:46.457628+0000 2021-03-24T22:06:46.723 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:46 smithi142 conmon[20125]: mon.a (mon.0) 2071 : audit [INF] 2021-03-24T22:06:46.724 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:46 smithi142 conmon[20125]: from='client.? v1:172.21.15.125:0/2826679618' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T22:06:46.724 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:46 smithi142 conmon[20125]: cluster 2021-03-24T22:06:46.724 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:46 smithi142 conmon[20125]: 2021-03-24T22:06:46.457744+0000 2021-03-24T22:06:46.724 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:46 smithi142 conmon[20125]: mon.a (mon.0) 2072 : cluster [DBG] 2021-03-24T22:06:46.724 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:46 smithi142 conmon[20125]: osdmap e836: 8 total, 8 up, 8 in 2021-03-24T22:06:46.725 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:46 smithi142 conmon[20125]: cluster 2021-03-24T22:06:46.725 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:46 smithi142 conmon[20125]: 2021-03-24T22:06:46.478713+0000 osd.0 2021-03-24T22:06:46.725 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:46 smithi142 conmon[20125]: (osd.0) 166 : cluster [DBG] 3.23 scrub ok 2021-03-24T22:06:46.725 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:46 smithi142 conmon[20125]: 2021-03-24T22:06:46.763 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:46.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:46.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:46.811 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-24T22:06:46.836 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:46 smithi125 conmon[19936]: audit 2021-03-24T22:06:44.935340+0000 mgr.y (mgr. 2021-03-24T22:06:46.836 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:46 smithi125 conmon[19936]: 14144) 693 : audit [DBG] from='client.15966 v1:172.21.15.125:0/1984436339' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T22:06:46.837 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:46 smithi125 conmon[19936]: cluster 2021-03-24T22:06:45.011398+0000 mgr.y (mgr.14144) 694 : cluster [DBG] pgmap v1414: 168 pgs: 168 active+clean; 521 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 1.6 MiB/s wr, 15 op/s 2021-03-24T22:06:46.837 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:46 smithi125 conmon[19936]: audit 2021-03-24T22: 2021-03-24T22:06:46.837 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:46 smithi125 conmon[19936]: 06:45.686125+0000 mon.a (mon.0) 2068 : audit [INF] from='client.? v1:172.21.15.125:0/2826679618' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:06:46.837 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:46 smithi125 conmon[19936]: cluster 2021-03-24T22:06:46.160687+0000 osd.7 (osd.7) 140 : cluster 2021-03-24T22:06:46.838 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:46 smithi125 conmon[19936]: [DBG] 7.11 scrub ok 2021-03-24T22:06:46.838 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:46 smithi125 conmon[19936]: cluster 2021-03-24T22:06: 2021-03-24T22:06:46.838 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:46 smithi125 conmon[19936]: 46.444145+0000 mon.a (mon.0) 2069 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T22:06:46.838 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:46 smithi125 conmon[19936]: cluster 2021-03-24T22:06:46.444215+0000 2021-03-24T22:06:46.838 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:46 smithi125 conmon[19936]: mon.a (mon.0) 2070 : cluster [INF] Cluster is now healthy 2021-03-24T22:06:46.839 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:46 smithi125 conmon[19936]: audit 2021-03-24T22:06:46.457628 2021-03-24T22:06:46.839 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:46 smithi125 conmon[19936]: +0000 mon.a (mon.0) 2071 : audit [INF] from='client.? v1:172.21.15.125:0/2826679618' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T22:06:46.839 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:46 smithi125 conmon[19936]: cluster 2021-03-24T22:06:46. 2021-03-24T22:06:46.839 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:46 smithi125 conmon[19936]: 457744+0000 mon.a (mon.0) 2072 : cluster [DBG] osdmap e836: 8 total, 8 up, 8 in 2021-03-24T22:06:46.840 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:46 smithi125 conmon[19936]: cluster 2021-03-24T22:06: 2021-03-24T22:06:46.840 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:46 smithi125 conmon[19936]: 46.478713+0000 osd.0 (osd.0) 166 : cluster [DBG] 3.23 scrub ok 2021-03-24T22:06:46.840 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:46 smithi125 conmon[24927]: audit 2021-03-24T22:06:44.935340 2021-03-24T22:06:46.841 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:46 smithi125 conmon[24927]: +0000 mgr.y (mgr.14144) 693 : audit [DBG] from='client.15966 v1:172.21.15.125:0/1984436339' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T22:06:46.841 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:46 smithi125 conmon[24927]: cluster 2021-03-24T22:06 2021-03-24T22:06:46.841 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:46 smithi125 conmon[24927]: :45.011398+0000 mgr.y (mgr.14144) 694 2021-03-24T22:06:46.841 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:46 smithi125 conmon[24927]: : cluster [DBG] pgmap v1414: 168 pgs: 168 active+clean; 521 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 1.6 MiB/s wr, 15 op/s 2021-03-24T22:06:46.842 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:46 smithi125 conmon[24927]: audit 2021 2021-03-24T22:06:46.842 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:46 smithi125 conmon[24927]: -03-24T22:06:45.686125+0000 2021-03-24T22:06:46.842 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:46 smithi125 conmon[24927]: mon.a (mon.0) 2068 : audit [INF] from='client.? v1:172.21.15.125:0/2826679618' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:06:46.842 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:46 smithi125 conmon[24927]: cluster 2021-03-24T22:06:46.160687+0000 osd.7 (osd.7) 2021-03-24T22:06:46.843 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:46 smithi125 conmon[24927]: 140 : cluster [DBG] 7.11 scrub ok 2021-03-24T22:06:46.843 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:46 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:06:46.843 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:46 smithi125 conmon[24927]: 24T22:06:46.444145 2021-03-24T22:06:46.843 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:46 smithi125 conmon[24927]: +0000 mon.a (mon.0) 2069 : cluster 2021-03-24T22:06:46.843 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:46 smithi125 conmon[24927]: [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-24T22:06:46.844 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:46 smithi125 conmon[24927]: cluster 2021-03-24T22:06:46.844 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:46 smithi125 conmon[24927]: 2021-03-24T22 2021-03-24T22:06:46.844 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:46 smithi125 conmon[24927]: :06:46.444215+0000 mon.a (mon.0 2021-03-24T22:06:46.844 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:46 smithi125 conmon[24927]: ) 2070 : cluster [INF] Cluster is now healthy 2021-03-24T22:06:46.844 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:46 smithi125 conmon[24927]: audit 2021-03-24T22:06:46.845 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:46 smithi125 conmon[24927]: 2021-03-24T22:06:46 2021-03-24T22:06:46.845 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:46 smithi125 conmon[24927]: .457628+0000 mon.a (mon.0) 2071 2021-03-24T22:06:46.845 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:46 smithi125 conmon[24927]: : audit [INF] from='client.? v1:172.21.15.125:0/2826679618' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-24T22:06:46.845 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:46 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:06:46.846 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:46 smithi125 conmon[24927]: -03-24T22:06:46.457744+0000 2021-03-24T22:06:46.846 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:46 smithi125 conmon[24927]: mon.a (mon.0) 2072 : cluster [DBG] osdmap e836: 8 total, 8 up, 8 in 2021-03-24T22:06:46.846 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:46 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:06:46.846 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:46 smithi125 conmon[24927]: 06:46.478713+0000 osd.0 (osd.0) 166 : 2021-03-24T22:06:46.846 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:46 smithi125 conmon[24927]: cluster [DBG] 3.23 scrub ok 2021-03-24T22:06:46.869 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:46.927 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:46.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:46.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:06:47.033 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:47.080 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:47.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:47.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:06:47.185 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:47.242 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:47.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:47.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:06:47.349 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:47.410 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:47.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:47.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:06:47.461 INFO:tasks.rados.rados.0.smithi125.stdout:3230: done (0 left) 2021-03-24T22:06:47.462 INFO:tasks.rados.rados.0.smithi125.stdout:3231: snap_remove snap 352 2021-03-24T22:06:47.517 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:47.574 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:47.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:47.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:06:47.679 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:06:47.679 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:47.709 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:06:47.732 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:47.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:47.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:06:47.839 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:47.887 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:47 smithi125 conmon[24927]: cluster 2021-03-24T22:06:46.851604+0000 osd.1 (osd 2021-03-24T22:06:47.887 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:47 smithi125 conmon[24927]: .1) 100 : cluster [DBG] 3.19 scrub ok 2021-03-24T22:06:47.887 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:47 smithi125 conmon[24927]: cluster 2021-03-24T22:06:47.083567+0000 osd.3 (osd 2021-03-24T22:06:47.888 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:47 smithi125 conmon[24927]: .3) 164 : cluster [DBG] 3.8 scrub ok 2021-03-24T22:06:47.888 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:47 smithi125 conmon[24927]: cluster 2021-03-24T22 2021-03-24T22:06:47.888 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:47 smithi125 conmon[24927]: :06:47.449140+0000 osd.0 (osd.0) 167 : cluster [DBG] 2021-03-24T22:06:47.888 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:47 smithi125 conmon[24927]: 5.5 scrub ok 2021-03-24T22:06:47.889 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:47 smithi125 conmon[24927]: cluster 2021-03-24T22:06:47 2021-03-24T22:06:47.889 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:47 smithi125 conmon[24927]: .454131+0000 mon.a (mon.0) 2073 : 2021-03-24T22:06:47.889 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:47 smithi125 conmon[24927]: cluster [DBG] osdmap e837: 8 total, 8 up, 8 in 2021-03-24T22:06:47.889 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:47 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T22:06:47.890 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:47 smithi125 conmon[19936]: 22:06:46.851604+0000 osd.1 (osd.1) 100 : cluster [DBG] 3.19 scrub ok 2021-03-24T22:06:47.890 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:47 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:06:47.890 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:47 smithi125 conmon[19936]: 06:47.083567+0000 osd.3 (osd.3) 164 : cluster [DBG] 3.8 scrub ok 2021-03-24T22:06:47.890 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:47 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:06:47.891 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:47 smithi125 conmon[19936]: 24T22:06:47.449140+0000 osd.0 (osd.0) 167 : cluster [DBG] 5.5 scrub ok 2021-03-24T22:06:47.891 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:47 smithi125 conmon[19936]: cluster 2021-03-24T22:06 2021-03-24T22:06:47.891 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:47 smithi125 conmon[19936]: :47.454131+0000 mon.a (mon.0) 2073 : cluster [DBG] osdmap e837: 8 total, 8 up, 8 in 2021-03-24T22:06:47.895 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:47.899 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:47 smithi142 conmon[20125]: cluster 2021-03-24T22:06:46.851604+0000 osd.1 (osd.1) 100 : 2021-03-24T22:06:47.899 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:47 smithi142 conmon[20125]: cluster [DBG] 3.19 scrub ok 2021-03-24T22:06:47.899 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:47 smithi142 conmon[20125]: cluster 2021-03-24T22:06:47.083567+0000 2021-03-24T22:06:47.900 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:47 smithi142 conmon[20125]: osd.3 (osd.3) 164 : cluster [DBG] 3.8 scrub ok 2021-03-24T22:06:47.900 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:47 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:06:47.900 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:47 smithi142 conmon[20125]: -03-24T22:06:47 2021-03-24T22:06:47.900 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:47 smithi142 conmon[20125]: .449140+0000 osd.0 (osd.0 2021-03-24T22:06:47.901 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:47 smithi142 conmon[20125]: ) 167 : cluster [DBG] 5.5 scrub ok 2021-03-24T22:06:47.901 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:47 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:06:47.901 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:47 smithi142 conmon[20125]: -03-24T22:06:47 2021-03-24T22:06:47.901 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:47 smithi142 conmon[20125]: .454131+0000 mon.a (mon. 2021-03-24T22:06:47.901 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:47 smithi142 conmon[20125]: 0) 2073 : cluster [DBG] osdmap e837: 8 total, 8 up, 8 in 2021-03-24T22:06:47.902 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:47 smithi142 conmon[20125]: 2021-03-24T22:06:47.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:47.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:06:48.005 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:48.062 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:48.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:48.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:06:48.173 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:48.228 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:48.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:48.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:06:48.334 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:48.388 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:48.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:48.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:06:48.464 INFO:tasks.rados.rados.0.smithi125.stdout:3231: done (0 left) 2021-03-24T22:06:48.465 INFO:tasks.rados.rados.0.smithi125.stdout:3232: delete oid 3 current snap is 375 2021-03-24T22:06:48.491 INFO:tasks.rados.rados.0.smithi125.stdout:3232: done (0 left) 2021-03-24T22:06:48.491 INFO:tasks.rados.rados.0.smithi125.stdout:3233: read oid 10 snap -1 2021-03-24T22:06:48.492 INFO:tasks.rados.rados.0.smithi125.stdout:3233: expect deleted 2021-03-24T22:06:48.492 INFO:tasks.rados.rados.0.smithi125.stdout:3234: read oid 43 snap -1 2021-03-24T22:06:48.492 INFO:tasks.rados.rados.0.smithi125.stdout:3234: expect (ObjNum 1021 snap 337 seq_num 1021) 2021-03-24T22:06:48.493 INFO:tasks.rados.rados.0.smithi125.stdout:3235: write (excl) oid 19 current snap is 375 2021-03-24T22:06:48.493 INFO:tasks.rados.rados.0.smithi125.stdout:3235: seq_num 1100 ranges {487700=728732,1786075=665712} 2021-03-24T22:06:48.494 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:48.496 INFO:tasks.rados.rados.0.smithi125.stdout:3235: writing smithi12556470-19 from 487700 to 1216432 tid 1 2021-03-24T22:06:48.502 INFO:tasks.rados.rados.0.smithi125.stdout:3235: writing smithi12556470-19 from 1786075 to 2451787 tid 2 2021-03-24T22:06:48.502 INFO:tasks.rados.rados.0.smithi125.stdout:3236: write oid 31 current snap is 375 2021-03-24T22:06:48.503 INFO:tasks.rados.rados.0.smithi125.stdout:3236: seq_num 1101 ranges {504507=403981,1328204=409370,2252362=252145} 2021-03-24T22:06:48.507 INFO:tasks.rados.rados.0.smithi125.stdout:3236: writing smithi12556470-31 from 504507 to 908488 tid 1 2021-03-24T22:06:48.513 INFO:tasks.rados.rados.0.smithi125.stdout:3236: writing smithi12556470-31 from 1328204 to 1737574 tid 2 2021-03-24T22:06:48.517 INFO:tasks.rados.rados.0.smithi125.stdout:3236: writing smithi12556470-31 from 2252362 to 2504507 tid 3 2021-03-24T22:06:48.518 INFO:tasks.rados.rados.0.smithi125.stdout:3235: finishing write tid 1 to smithi12556470-19 2021-03-24T22:06:48.518 INFO:tasks.rados.rados.0.smithi125.stdout:3234: expect (ObjNum 1021 snap 337 seq_num 1021) 2021-03-24T22:06:48.557 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:48.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:48.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:06:48.663 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:48.694 INFO:tasks.rados.rados.0.smithi125.stdout:3235: finishing write tid 2 to smithi12556470-19 2021-03-24T22:06:48.695 INFO:tasks.rados.rados.0.smithi125.stdout:3235: finishing write tid 3 to smithi12556470-19 2021-03-24T22:06:48.695 INFO:tasks.rados.rados.0.smithi125.stdout:3235: finishing write tid 4 to smithi12556470-19 2021-03-24T22:06:48.695 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 19 v 1466 (ObjNum 1100 snap 375 seq_num 1100) dirty exists 2021-03-24T22:06:48.695 INFO:tasks.rados.rados.0.smithi125.stdout:3235: left oid 19 (ObjNum 1100 snap 375 seq_num 1100) 2021-03-24T22:06:48.696 INFO:tasks.rados.rados.0.smithi125.stdout:3233: done (3 left) 2021-03-24T22:06:48.696 INFO:tasks.rados.rados.0.smithi125.stdout:3234: done (2 left) 2021-03-24T22:06:48.696 INFO:tasks.rados.rados.0.smithi125.stdout:3235: done (1 left) 2021-03-24T22:06:48.696 INFO:tasks.rados.rados.0.smithi125.stdout:3237: rollback oid 9 current snap is 375 2021-03-24T22:06:48.697 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 9 to 365 2021-03-24T22:06:48.697 INFO:tasks.rados.rados.0.smithi125.stdout:3236: finishing write tid 1 to smithi12556470-31 2021-03-24T22:06:48.697 INFO:tasks.rados.rados.0.smithi125.stdout:3236: finishing write tid 2 to smithi12556470-31 2021-03-24T22:06:48.697 INFO:tasks.rados.rados.0.smithi125.stdout:3236: finishing write tid 3 to smithi12556470-31 2021-03-24T22:06:48.697 INFO:tasks.rados.rados.0.smithi125.stdout:3236: finishing write tid 4 to smithi12556470-31 2021-03-24T22:06:48.698 INFO:tasks.rados.rados.0.smithi125.stdout:3236: finishing write tid 5 to smithi12556470-31 2021-03-24T22:06:48.698 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 31 v 2073 (ObjNum 1101 snap 375 seq_num 1101) dirty exists 2021-03-24T22:06:48.698 INFO:tasks.rados.rados.0.smithi125.stdout:3236: left oid 31 (ObjNum 1101 snap 375 seq_num 1101) 2021-03-24T22:06:48.698 INFO:tasks.rados.rados.0.smithi125.stdout:3236: done (1 left) 2021-03-24T22:06:48.699 INFO:tasks.rados.rados.0.smithi125.stdout:3238: snap_remove snap 371 2021-03-24T22:06:48.727 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:48.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:48.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:06:48.741 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:48 smithi125 conmon[24927]: cluster 2021-03-24T22:06 2021-03-24T22:06:48.742 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:48 smithi125 conmon[24927]: :47.012231+0000 mgr.y (mgr.14144) 695 : cluster [DBG] pgmap v1417: 168 pgs: 168 active+clean; 530 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 3.8 MiB/s wr, 20 op/s 2021-03-24T22:06:48.742 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:48 smithi125 conmon[24927]: cluster 2021-03-24T22:06:48.742 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:48 smithi125 conmon[24927]: 2021-03-24T22:06:48.063852+0000 osd.3 (osd 2021-03-24T22:06:48.743 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:48 smithi125 conmon[24927]: .3) 165 : cluster [DBG] 5.1c scrub ok 2021-03-24T22:06:48.743 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:48 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T22:06:48.743 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:48 smithi125 conmon[24927]: 22:06:48.128924+0000 osd.7 (osd 2021-03-24T22:06:48.743 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:48 smithi125 conmon[24927]: .7) 141 : cluster [DBG] 2.0 deep-scrub ok 2021-03-24T22:06:48.744 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:48 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:06:48.744 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:48 smithi125 conmon[24927]: 03-24T22:06:48.457892+0000 2021-03-24T22:06:48.744 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:48 smithi125 conmon[24927]: mon.a (mon.0) 2074 : cluster [DBG] 2021-03-24T22:06:48.744 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:48 smithi125 conmon[24927]: osdmap e838: 8 total, 8 up, 8 in 2021-03-24T22:06:48.745 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:48 smithi125 conmon[19936]: cluster 2021-03-24T22:06: 2021-03-24T22:06:48.745 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:48 smithi125 conmon[19936]: 47.012231+0000 mgr.y (mgr.14144) 695 : cluster [DBG] pgmap v1417: 168 pgs: 168 active+clean; 530 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 3.8 MiB/s wr, 20 op/s 2021-03-24T22:06:48.745 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:48 smithi125 conmon[19936]: cluster 2021-03-24T22:06:48.745 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:48 smithi125 conmon[19936]: 2021-03-24T22:06:48.063852+0000 osd.3 (osd.3) 2021-03-24T22:06:48.745 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:48 smithi125 conmon[19936]: 165 : cluster [DBG] 5.1c scrub ok 2021-03-24T22:06:48.746 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:48 smithi125 conmon[19936]: cluster 2021-03-24T22:06:48.746 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:48 smithi125 conmon[19936]: 2021-03-24T22: 2021-03-24T22:06:48.746 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:48 smithi125 conmon[19936]: 06:48.128924+0000 osd.7 (osd.7) 2021-03-24T22:06:48.746 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:48 smithi125 conmon[19936]: 141 : cluster [DBG] 2.0 deep-scrub ok 2021-03-24T22:06:48.747 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:48 smithi125 conmon[19936]: cluster 2021-03-24T22:06:48.747 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:48 smithi125 conmon[19936]: 2021-03-24T22: 2021-03-24T22:06:48.747 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:48 smithi125 conmon[19936]: 06:48.457892+0000 mon.a (mon.0) 2021-03-24T22:06:48.747 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:48 smithi125 conmon[19936]: 2074 : cluster [DBG] osdmap e838: 8 total, 8 up, 8 in 2021-03-24T22:06:48.834 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:48.892 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:48.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:48.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:06:48.998 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:49.011 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:48 smithi142 conmon[20125]: cluster 2021-03-24T22:06:47.012231+0000 mgr.y (mgr.14144) 695 : cluster 2021-03-24T22:06:49.012 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:48 smithi142 conmon[20125]: [DBG] pgmap v1417: 168 pgs: 168 active+clean; 530 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 3.8 MiB/s wr, 20 op/s 2021-03-24T22:06:49.012 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:48 smithi142 conmon[20125]: cluster 2021-03-24T22:06:48.063852+0000 osd.3 (osd.3) 165 : cluster 2021-03-24T22:06:49.012 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:48 smithi142 conmon[20125]: [DBG] 5.1c scrub ok 2021-03-24T22:06:49.013 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:48 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:06:49.013 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:48 smithi142 conmon[20125]: 03-24T22:06:48.128924+0000 osd.7 (osd. 2021-03-24T22:06:49.013 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:48 smithi142 conmon[20125]: 7) 141 : cluster [DBG] 2.0 deep-scrub ok 2021-03-24T22:06:49.013 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:48 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:06:49.014 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:48 smithi142 conmon[20125]: 24T22:06:48.457892+0000 mon.a ( 2021-03-24T22:06:49.014 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:48 smithi142 conmon[20125]: mon.0) 2074 : cluster [DBG] osdmap e838: 8 total, 8 up, 8 in 2021-03-24T22:06:49.055 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:49.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:49.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:06:49.161 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:49.218 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:49.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:49.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:49.325 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:49.374 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:49.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:49.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:06:49.480 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:49.536 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:49.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:49.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:06:49.640 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:49.658 INFO:tasks.rados.rados.0.smithi125.stdout:3238: done (1 left) 2021-03-24T22:06:49.658 INFO:tasks.rados.rados.0.smithi125.stdout:3239: snap_create 2021-03-24T22:06:49.658 INFO:tasks.rados.rados.0.smithi125.stdout:3237: finishing rollback tid 0 to smithi12556470-9 2021-03-24T22:06:49.659 INFO:tasks.rados.rados.0.smithi125.stdout:3237: finishing rollback tid 1 to smithi12556470-9 2021-03-24T22:06:49.659 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 9 v 2076 (ObjNum 1953066355 snap 842099048 seq_num 875967797) dirty dne 2021-03-24T22:06:49.661 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:49 smithi125 conmon[24927]: cluster 2021-03-24 2021-03-24T22:06:49.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:49 smithi125 conmon[24927]: T22:06:48.211978+0000 osd.2 (osd.2 2021-03-24T22:06:49.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:49 smithi125 conmon[24927]: ) 137 : cluster [DBG] 6.17 scrub ok 2021-03-24T22:06:49.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:49 smithi125 conmon[24927]: cluster 2021-03-24T22:06:49.080831 2021-03-24T22:06:49.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:49 smithi125 conmon[24927]: +0000 osd.7 (osd.7) 142 : cluster [DBG] 3.13 scrub ok 2021-03-24T22:06:49.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:49 smithi125 conmon[24927]: cluster 2021-03-24T22:06 2021-03-24T22:06:49.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:49 smithi125 conmon[24927]: :49.184832+0000 osd.6 (osd.6) 87 : cluster [DBG] 2021-03-24T22:06:49.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:49 smithi125 conmon[24927]: 7.19 scrub ok 2021-03-24T22:06:49.694 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:49.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:49.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:06:49.698 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd set noscrub 2021-03-24T22:06:49.799 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:49.859 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:49.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:49.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:49.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:49 smithi125 conmon[19936]: cluster 2021-03-24T22:06:49.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:49 smithi125 conmon[19936]: 2021-03-24T22:06:49.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:49 smithi125 conmon[19936]: 2021-03-24T22:06 2021-03-24T22:06:49.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:49 smithi125 conmon[19936]: :48.211978+0000 osd.2 (osd.2) 137 : cluster [DBG] 6.17 scrub ok 2021-03-24T22:06:49.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:49 smithi125 conmon[19936]: cluster 2021-03-24T22:06:49.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:49 smithi125 conmon[19936]: 2021-03-24T22:06:49.080831+0000 2021-03-24T22:06:49.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:49 smithi125 conmon[19936]: osd.7 (osd.7) 142 : cluster [DBG] 2021-03-24T22:06:49.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:49 smithi125 conmon[19936]: 3.13 scrub ok 2021-03-24T22:06:49.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:49 smithi125 conmon[19936]: cluster 2021-03-24T22:06:49.184832 2021-03-24T22:06:49.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:49 smithi125 conmon[19936]: +0000 osd.6 (osd.6) 87 : cluster [DBG] 2021-03-24T22:06:49.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:49 smithi125 conmon[19936]: 7.19 scrub ok 2021-03-24T22:06:49.965 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:49.979 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:49 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:06:49.979 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:49 smithi142 conmon[20125]: 22:06:48.211978+0000 osd.2 (osd.2) 137 : cluster [DBG] 6.17 scrub ok 2021-03-24T22:06:49.980 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:49 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:06:49.980 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:49 smithi142 conmon[20125]: T22:06:49.080831 2021-03-24T22:06:49.980 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:49 smithi142 conmon[20125]: +0000 osd.7 (osd.7) 2021-03-24T22:06:49.980 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:49 smithi142 conmon[20125]: 142 : cluster [DBG] 3.13 scrub ok 2021-03-24T22:06:49.981 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:49 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:06:49.981 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:49 smithi142 conmon[20125]: 03-24T22:06:49. 2021-03-24T22:06:49.981 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:49 smithi142 conmon[20125]: 184832+0000 osd.6 (osd.6 2021-03-24T22:06:49.981 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:49 smithi142 conmon[20125]: ) 87 : cluster [DBG] 7.19 scrub ok 2021-03-24T22:06:50.019 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:50.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:50.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:06:50.125 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:50.182 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:50.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:50.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:06:50.289 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:50.346 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:50.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:50.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:06:50.449 INFO:tasks.rados.rados.0.smithi125.stdout:3237: done (1 left) 2021-03-24T22:06:50.449 INFO:tasks.rados.rados.0.smithi125.stdout:3239: done (0 left) 2021-03-24T22:06:50.450 INFO:tasks.rados.rados.0.smithi125.stdout:3240: snap_remove snap 354 2021-03-24T22:06:50.451 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:50.504 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:50.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:50.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:06:50.609 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:50.668 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:50.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:50.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:06:50.773 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:50.841 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:50.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:50.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:06:50.912 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:50 smithi125 conmon[24927]: cluster 2021-03-24T22:06:48 2021-03-24T22:06:50.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:50 smithi125 conmon[24927]: .910925+0000 osd.1 (osd.1) 101 : cluster [DBG] 7.1d scrub ok 2021-03-24T22:06:50.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:50 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:06:50.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:50 smithi125 conmon[24927]: -03-24T22:06:49.013203+0000 mgr.y (mgr.14144) 696 : cluster [DBG] pgmap v1420: 168 pgs: 168 active+clean; 534 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 8.2 MiB/s rd, 2.8 MiB/s wr, 14 op/s 2021-03-24T22:06:50.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:50 smithi125 conmon[24927]: cluster 2021-03-24T22:06:49 2021-03-24T22:06:50.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:50 smithi125 conmon[24927]: .246022+0000 osd.2 (osd.2) 138 : cluster [DBG] 4.7 scrub ok 2021-03-24T22:06:50.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:50 smithi125 conmon[24927]: 2021-03-24T22:06:50.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:50 smithi125 conmon[24927]: cluster 2021-03 2021-03-24T22:06:50.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:50 smithi125 conmon[24927]: -24T22:06:49.326503+0000 osd.5 (osd.5) 25 : cluster [DBG] 4.1 scrub ok 2021-03-24T22:06:50.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:50 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:06:50.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:50 smithi125 conmon[24927]: 06:49.653783+0000 mon.a (mon.0) 2075 : cluster [DBG] 2021-03-24T22:06:50.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:50 smithi125 conmon[24927]: osdmap e839: 8 total, 8 up, 8 in 2021-03-24T22:06:50.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:50 smithi125 conmon[24927]: cluster 2021-03-24T22:06 2021-03-24T22:06:50.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:50 smithi125 conmon[24927]: :50.445378+0000 mon.a (mon.0) 2076 : cluster [DBG] osdmap e840: 8 total, 8 up, 8 in 2021-03-24T22:06:50.916 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:50 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:06:50.916 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:50 smithi125 conmon[19936]: T22:06:48.910925+0000 osd.1 (osd.1) 101 : cluster [DBG] 7.1d scrub ok 2021-03-24T22:06:50.916 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:50 smithi125 conmon[19936]: cluster 2021-03-24T22:06:49.013203+0000 mgr.y (mgr.14144) 696 : 2021-03-24T22:06:50.916 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:50 smithi125 conmon[19936]: cluster [DBG] pgmap v1420: 168 pgs: 168 active+clean; 534 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 8.2 MiB/s rd, 2.8 MiB/s wr, 14 op/s 2021-03-24T22:06:50.917 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:50 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:06:50.917 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:50 smithi125 conmon[19936]: 03-24T22:06:49.246022+0000 osd.2 (osd.2) 138 : cluster [DBG] 4.7 scrub ok 2021-03-24T22:06:50.917 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:50 smithi125 conmon[19936]: cluster 2021-03-24T22:06: 2021-03-24T22:06:50.917 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:50 smithi125 conmon[19936]: 49.326503+0000 osd.5 (osd.5) 25 : cluster [DBG] 2021-03-24T22:06:50.917 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:50 smithi125 conmon[19936]: 4.1 scrub ok 2021-03-24T22:06:50.918 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:50 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:06:50.918 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:50 smithi125 conmon[19936]: T22:06:49.653783+0000 mon.a (mon.0) 2075 : cluster [DBG] osdmap e839: 8 total, 8 up, 8 in 2021-03-24T22:06:50.918 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:50 smithi125 conmon[19936]: 2021-03-24T22:06:50.918 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:50 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T22:06:50.918 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:50 smithi125 conmon[19936]: 22:06:50.445378+0000 mon.a (mon.0) 2076 : cluster [DBG] osdmap e840: 8 total, 8 up, 8 in 2021-03-24T22:06:50.919 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:50 smithi125 conmon[19936]: 2021-03-24T22:06:50.947 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:51.008 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:51.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:51.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:06:51.114 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:51.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:50 smithi142 conmon[20125]: cluster 2021-03-24T22:06:48.910925 2021-03-24T22:06:51.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:50 smithi142 conmon[20125]: +0000 osd.1 (osd.1) 101 : cluster [DBG] 7.1d scrub ok 2021-03-24T22:06:51.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:50 smithi142 conmon[20125]: cluster 2021-03-24T22:06:51.119 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:50 smithi142 conmon[20125]: 2021-03- 2021-03-24T22:06:51.119 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:50 smithi142 conmon[20125]: 24T22: 2021-03-24T22:06:51.119 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:50 smithi142 conmon[20125]: 06:49. 2021-03-24T22:06:51.119 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:50 smithi142 conmon[20125]: 013203+0000 mgr.y ( 2021-03-24T22:06:51.120 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:50 smithi142 conmon[20125]: mgr.14144) 2021-03-24T22:06:51.120 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:50 smithi142 conmon[20125]: 696 : cluster [DBG] 2021-03-24T22:06:51.120 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:50 smithi142 conmon[20125]: pgmap v1420: 168 pgs: 168 active+clean; 534 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 8.2 MiB/s rd, 2.8 MiB/s wr, 14 op/s 2021-03-24T22:06:51.120 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:50 smithi142 conmon[20125]: cluster 2021-03-24T22:06:51.120 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:50 smithi142 conmon[20125]: 2021-03 2021-03-24T22:06:51.121 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:50 smithi142 conmon[20125]: -24T22 2021-03-24T22:06:51.121 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:50 smithi142 conmon[20125]: :06:49 2021-03-24T22:06:51.121 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:50 smithi142 conmon[20125]: .246022+0000 2021-03-24T22:06:51.121 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:50 smithi142 conmon[20125]: osd.2 (osd.2 2021-03-24T22:06:51.121 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:50 smithi142 conmon[20125]: ) 138 : cluster 2021-03-24T22:06:51.122 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:50 smithi142 conmon[20125]: [DBG] 4.7 scrub ok 2021-03-24T22:06:51.122 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:50 smithi142 conmon[20125]: cluster 2021-03-24T22:06:51.122 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:50 smithi142 conmon[20125]: 2021-03-24T 2021-03-24T22:06:51.122 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:50 smithi142 conmon[20125]: 22:06:49.326503 2021-03-24T22:06:51.123 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:50 smithi142 conmon[20125]: +0000 osd.5 (osd.5 2021-03-24T22:06:51.123 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:50 smithi142 conmon[20125]: ) 25 : cluster 2021-03-24T22:06:51.123 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:50 smithi142 conmon[20125]: [DBG] 4.1 scrub ok 2021-03-24T22:06:51.123 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:50 smithi142 conmon[20125]: cluster 2021-03-24T22:06:51.123 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:50 smithi142 conmon[20125]: 2021-03 2021-03-24T22:06:51.124 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:50 smithi142 conmon[20125]: -24T22 2021-03-24T22:06:51.124 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:50 smithi142 conmon[20125]: :06:49 2021-03-24T22:06:51.124 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:50 smithi142 conmon[20125]: .653783+0000 mon.a 2021-03-24T22:06:51.124 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:50 smithi142 conmon[20125]: (mon.0 2021-03-24T22:06:51.125 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:50 smithi142 conmon[20125]: ) 2075 : 2021-03-24T22:06:51.125 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:50 smithi142 conmon[20125]: cluster 2021-03-24T22:06:51.125 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:50 smithi142 conmon[20125]: [DBG] osdmap e839: 8 total, 8 up, 8 in 2021-03-24T22:06:51.125 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:50 smithi142 conmon[20125]: cluster 2021-03-24T22:06:51.125 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:50 smithi142 conmon[20125]: 2021-03 2021-03-24T22:06:51.126 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:50 smithi142 conmon[20125]: -24T22:06:50 2021-03-24T22:06:51.126 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:50 smithi142 conmon[20125]: .445378+0000 2021-03-24T22:06:51.126 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:50 smithi142 conmon[20125]: mon.a (mon 2021-03-24T22:06:51.126 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:50 smithi142 conmon[20125]: .0) 2021-03-24T22:06:51.127 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:50 smithi142 conmon[20125]: 2076 : cluster 2021-03-24T22:06:51.127 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:50 smithi142 conmon[20125]: [DBG] osdmap e840: 8 total, 8 up, 8 in 2021-03-24T22:06:51.168 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:51.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:51.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:51.273 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:51.334 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:51.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:51.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:06:51.441 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:51.451 INFO:tasks.rados.rados.0.smithi125.stdout:3240: done (0 left) 2021-03-24T22:06:51.452 INFO:tasks.rados.rados.0.smithi125.stdout:3241: delete oid 33 current snap is 376 2021-03-24T22:06:51.468 INFO:tasks.rados.rados.0.smithi125.stdout:3241: done (0 left) 2021-03-24T22:06:51.469 INFO:tasks.rados.rados.0.smithi125.stdout:3242: snap_remove snap 346 2021-03-24T22:06:51.497 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:51.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:51.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:06:51.559 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:06:51.602 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:51.664 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:51.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:51.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:06:51.771 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:51.829 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:51.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:51.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:51.936 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:51.998 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:52.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:52.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:06:52.103 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:52.155 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:52.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:52.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:06:52.236 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T22:06:52.260 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:52.262 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T22:06:52.282 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T22:06:52.303 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-24T22:06:52.306 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2021-03-24T22:06:52.307 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T22:06:52.323 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:52.327 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T22:06:52.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:52.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:06:52.353 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T22:06:52.384 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T22:06:52.407 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T22:06:52.408 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-24T22:06:52.409 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-24T22:06:52.409 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-24T22:06:52.409 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-24T22:06:52.409 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-24T22:06:52.410 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-24T22:06:52.410 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-24T22:06:52.410 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-24T22:06:52.430 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:52.452 INFO:tasks.rados.rados.0.smithi125.stdout:3242: done (0 left) 2021-03-24T22:06:52.453 INFO:tasks.rados.rados.0.smithi125.stdout:3243: read oid 9 snap 336 2021-03-24T22:06:52.453 INFO:tasks.rados.rados.0.smithi125.stdout:3243: expect deleted 2021-03-24T22:06:52.453 INFO:tasks.rados.rados.0.smithi125.stdout:3244: delete oid 21 current snap is 376 2021-03-24T22:06:52.470 INFO:tasks.rados.rados.0.smithi125.stdout:3243: done (1 left) 2021-03-24T22:06:52.470 INFO:tasks.rados.rados.0.smithi125.stdout:3244: done (0 left) 2021-03-24T22:06:52.470 INFO:tasks.rados.rados.0.smithi125.stdout:3245: rollback oid 40 current snap is 376 2021-03-24T22:06:52.471 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 40 to 347 2021-03-24T22:06:52.471 INFO:tasks.rados.rados.0.smithi125.stdout:3246: read oid 12 snap -1 2021-03-24T22:06:52.471 INFO:tasks.rados.rados.0.smithi125.stdout:3246: expect (ObjNum 1001 snap 331 seq_num 1001) 2021-03-24T22:06:52.471 INFO:tasks.rados.rados.0.smithi125.stdout:3247: read oid 45 snap -1 2021-03-24T22:06:52.472 INFO:tasks.rados.rados.0.smithi125.stdout:3247: expect (ObjNum 1014 snap 336 seq_num 1014) 2021-03-24T22:06:52.472 INFO:tasks.rados.rados.0.smithi125.stdout:3248: read oid 18 snap -1 2021-03-24T22:06:52.472 INFO:tasks.rados.rados.0.smithi125.stdout:3248: expect (ObjNum 1051 snap 349 seq_num 1051) 2021-03-24T22:06:52.472 INFO:tasks.rados.rados.0.smithi125.stdout:3249: read oid 13 snap 336 2021-03-24T22:06:52.472 INFO:tasks.rados.rados.0.smithi125.stdout:3249: expect deleted 2021-03-24T22:06:52.473 INFO:tasks.rados.rados.0.smithi125.stdout:3250: read oid 31 snap 370 2021-03-24T22:06:52.473 INFO:tasks.rados.rados.0.smithi125.stdout:3250: expect deleted 2021-03-24T22:06:52.473 INFO:tasks.rados.rados.0.smithi125.stdout:3251: read oid 36 snap 361 2021-03-24T22:06:52.473 INFO:tasks.rados.rados.0.smithi125.stdout:3251: expect (ObjNum 1049 snap 348 seq_num 1049) 2021-03-24T22:06:52.474 INFO:tasks.rados.rados.0.smithi125.stdout:3252: snap_create 2021-03-24T22:06:52.475 INFO:tasks.rados.rados.0.smithi125.stdout:3247: expect (ObjNum 1014 snap 336 seq_num 1014) 2021-03-24T22:06:52.495 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:52.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:52.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:06:52.573 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:52 smithi125 conmon[24927]: cluster 2021-03-24T22:06:51.014167+0000 2021-03-24T22:06:52.573 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:52 smithi125 conmon[24927]: mgr.y (mgr.14144) 697 : cluster [DBG] pgmap v1423: 168 pgs: 168 active+clean; 535 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 8.2 MiB/s rd, 1.5 MiB/s wr, 12 op/s 2021-03-24T22:06:52.573 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:52 smithi125 conmon[24927]: cluster 2021-03-24T22:06:51.129929+0000 osd.3 2021-03-24T22:06:52.574 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:52 smithi125 conmon[24927]: (osd.3) 166 : cluster [DBG] 7.10 deep-scrub ok 2021-03-24T22:06:52.574 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:52 smithi125 conmon[24927]: cluster 2021-03-24T22:06:51 2021-03-24T22:06:52.574 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:52 smithi125 conmon[24927]: .157656+0000 osd.6 (osd.6) 88 : cluster 2021-03-24T22:06:52.574 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:52 smithi125 conmon[24927]: [DBG] 7.1f scrub ok 2021-03-24T22:06:52.575 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:52 smithi125 conmon[24927]: cluster 2021-03-24T22:06 2021-03-24T22:06:52.575 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:52 smithi125 conmon[24927]: :51.447620+0000 mon.a (mon.0) 2077 2021-03-24T22:06:52.575 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:52 smithi125 conmon[24927]: : cluster [DBG] osdmap e841: 8 total, 8 up, 8 in 2021-03-24T22:06:52.575 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:52 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T22:06:52.575 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:52 smithi125 conmon[24927]: 24T22:06:52.207853+0000 mon.c ( 2021-03-24T22:06:52.576 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:52 smithi125 conmon[24927]: mon.2) 119 : audit [DBG] from='client.? v1:172.21.15.125:0/1561773375' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:06:52.577 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:52 smithi125 conmon[19936]: cluster 2021-03-24T22:06:52.577 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:52 smithi125 conmon[19936]: 2021-03-24T22:06:52.577 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:52 smithi125 conmon[19936]: 2021 2021-03-24T22:06:52.578 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:52 smithi125 conmon[19936]: - 2021-03-24T22:06:52.578 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:52 smithi125 conmon[19936]: 03 2021-03-24T22:06:52.578 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:52 smithi125 conmon[19936]: - 2021-03-24T22:06:52.578 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:52 smithi125 conmon[19936]: 24 2021-03-24T22:06:52.579 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:52 smithi125 conmon[19936]: T 2021-03-24T22:06:52.579 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:52 smithi125 conmon[19936]: 22:06:51.014167+0000 mgr.y (mgr.14144) 697 : cluster 2021-03-24T22:06:52.579 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:52 smithi125 conmon[19936]: [DBG] pgmap v1423: 168 pgs: 168 active+clean; 535 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 8.2 MiB/s rd, 1.5 MiB/s wr, 12 op/s 2021-03-24T22:06:52.579 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:52 smithi125 conmon[19936]: cluster 2021-03-24T22:06:52.580 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:52 smithi125 conmon[19936]: 2021-03-24T22:06:51.129929+0000 osd.3 (osd.3) 166 : cluster [DBG] 2021-03-24T22:06:52.580 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:52 smithi125 conmon[19936]: 7.10 deep-scrub ok 2021-03-24T22:06:52.580 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:52 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:06:52.580 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:52 smithi125 conmon[19936]: -03-24T22:06:51.157656+0000 osd.6 (osd.6 2021-03-24T22:06:52.581 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:52 smithi125 conmon[19936]: ) 88 : cluster [DBG] 7.1f scrub ok 2021-03-24T22:06:52.581 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:52 smithi125 conmon[19936]: cluster 2021-03 2021-03-24T22:06:52.581 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:52 smithi125 conmon[19936]: -24T22:06:51.447620+0000 mon.a 2021-03-24T22:06:52.581 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:52 smithi125 conmon[19936]: (mon.0) 2077 : cluster [DBG] osdmap e841: 8 total, 8 up, 8 in 2021-03-24T22:06:52.581 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:52 smithi125 conmon[19936]: audit 2021-03-24T22:06:52.582 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:52 smithi125 conmon[19936]: 2021-03-24T22:06:52.207853+0000 mon.c (mon 2021-03-24T22:06:52.582 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:52 smithi125 conmon[19936]: .2) 119 : audit [DBG] from='client.? v1:172.21.15.125:0/1561773375' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:06:52.604 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:52.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:52 smithi142 conmon[20125]: cluster 2021-03-24T22:06:52.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:52 smithi142 conmon[20125]: 2021-03-24T22:06:52.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:52 smithi142 conmon[20125]: 2021 2021-03-24T22:06:52.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:52 smithi142 conmon[20125]: - 2021-03-24T22:06:52.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:52 smithi142 conmon[20125]: 03 2021-03-24T22:06:52.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:52 smithi142 conmon[20125]: - 2021-03-24T22:06:52.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:52 smithi142 conmon[20125]: 24 2021-03-24T22:06:52.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:52 smithi142 conmon[20125]: T 2021-03-24T22:06:52.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:52 smithi142 conmon[20125]: 22 2021-03-24T22:06:52.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:52 smithi142 conmon[20125]: :06:51.014167+0000 mgr.y (mgr 2021-03-24T22:06:52.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:52 smithi142 conmon[20125]: .14144) 697 : cluster [DBG] pgmap v1423: 168 pgs: 168 active+clean; 535 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 8.2 MiB/s rd, 1.5 MiB/s wr, 12 op/s 2021-03-24T22:06:52.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:52 smithi142 conmon[20125]: cluster 2021-03-24T22:06:52.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:52 smithi142 conmon[20125]: 2021-03-24T22:06:52.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:52 smithi142 conmon[20125]: 2021 2021-03-24T22:06:52.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:52 smithi142 conmon[20125]: -03-24T22:06: 2021-03-24T22:06:52.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:52 smithi142 conmon[20125]: 51.129929+0000 osd.3 (osd.3) 166 : cluster [DBG] 7.10 deep-scrub ok 2021-03-24T22:06:52.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:52 smithi142 conmon[20125]: cluster 2021-03-24T22:06:51.157656+0000 osd.6 (osd.6) 88 2021-03-24T22:06:52.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:52 smithi142 conmon[20125]: : cluster [DBG] 7.1f scrub ok 2021-03-24T22:06:52.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:52 smithi142 conmon[20125]: cluster 2021-03-24T22:06:51. 2021-03-24T22:06:52.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:52 smithi142 conmon[20125]: 447620+0000 mon.a (mon.0) 2077 : cluster [DBG] osdmap e841: 8 total, 8 up, 8 in 2021-03-24T22:06:52.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:52 smithi142 conmon[20125]: 2021-03-24T22:06:52.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:52 smithi142 conmon[20125]: audit 2021-03-24T22:06:52.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:52 smithi142 conmon[20125]: 2021-03-24T22:06:52.207853+0000 mon.c (mon. 2021-03-24T22:06:52.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:52 smithi142 conmon[20125]: 2) 119 : audit [DBG] from='client.? v1:172.21.15.125:0/1561773375' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:06:52.637 INFO:tasks.rados.rados.0.smithi125.stdout:3246: expect (ObjNum 1001 snap 331 seq_num 1001) 2021-03-24T22:06:52.667 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:52.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:52.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:06:52.735 INFO:tasks.rados.rados.0.smithi125.stdout:3245: finishing rollback tid 0 to smithi12556470-40 2021-03-24T22:06:52.735 INFO:tasks.rados.rados.0.smithi125.stdout:3245: finishing rollback tid 1 to smithi12556470-40 2021-03-24T22:06:52.735 INFO:tasks.rados.rados.0.smithi125.stdout:3245: finishing rollback tid 2 to smithi12556470-40 2021-03-24T22:06:52.736 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 40 v 2092 (ObjNum 906 snap 300 seq_num 906) dirty exists 2021-03-24T22:06:52.736 INFO:tasks.rados.rados.0.smithi125.stdout:3248: expect (ObjNum 1051 snap 349 seq_num 1051) 2021-03-24T22:06:52.771 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:52.828 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:52.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:52.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:52.851 INFO:tasks.rados.rados.0.smithi125.stdout:3251: expect (ObjNum 1049 snap 348 seq_num 1049) 2021-03-24T22:06:52.938 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:52.994 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:52.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:52.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:06:53.099 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:53.156 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:53.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:53.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:06:53.264 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:53.320 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:53.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:53.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:06:53.425 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:53.475 INFO:tasks.rados.rados.0.smithi125.stdout:3245: done (7 left) 2021-03-24T22:06:53.476 INFO:tasks.rados.rados.0.smithi125.stdout:3246: done (6 left) 2021-03-24T22:06:53.476 INFO:tasks.rados.rados.0.smithi125.stdout:3247: done (5 left) 2021-03-24T22:06:53.476 INFO:tasks.rados.rados.0.smithi125.stdout:3248: done (4 left) 2021-03-24T22:06:53.477 INFO:tasks.rados.rados.0.smithi125.stdout:3249: done (3 left) 2021-03-24T22:06:53.477 INFO:tasks.rados.rados.0.smithi125.stdout:3250: done (2 left) 2021-03-24T22:06:53.477 INFO:tasks.rados.rados.0.smithi125.stdout:3251: done (1 left) 2021-03-24T22:06:53.477 INFO:tasks.rados.rados.0.smithi125.stdout:3252: done (0 left) 2021-03-24T22:06:53.477 INFO:tasks.rados.rados.0.smithi125.stdout:3253: read oid 31 snap 344 2021-03-24T22:06:53.478 INFO:tasks.rados.rados.0.smithi125.stdout:3253: expect (ObjNum 1022 snap 337 seq_num 1022) 2021-03-24T22:06:53.478 INFO:tasks.rados.rados.0.smithi125.stdout:3254: write (excl) oid 42 current snap is 377 2021-03-24T22:06:53.478 INFO:tasks.rados.rados.0.smithi125.stdout:3254: seq_num 1104 ranges {554928=746081,2020003=534925} 2021-03-24T22:06:53.484 INFO:tasks.rados.rados.0.smithi125.stdout:3254: writing smithi12556470-42 from 554928 to 1301009 tid 1 2021-03-24T22:06:53.487 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:53.491 INFO:tasks.rados.rados.0.smithi125.stdout:3254: writing smithi12556470-42 from 2020003 to 2554928 tid 2 2021-03-24T22:06:53.491 INFO:tasks.rados.rados.0.smithi125.stdout:3255: read oid 1 snap 339 2021-03-24T22:06:53.491 INFO:tasks.rados.rados.0.smithi125.stdout:3255: expect (ObjNum 949 snap 314 seq_num 949) 2021-03-24T22:06:53.491 INFO:tasks.rados.rados.0.smithi125.stdout:3256: write (excl) oid 21 current snap is 377 2021-03-24T22:06:53.492 INFO:tasks.rados.rados.0.smithi125.stdout:3256: seq_num 1105 ranges {571735=421330,1562132=416708} 2021-03-24T22:06:53.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:53.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:06:53.496 INFO:tasks.rados.rados.0.smithi125.stdout:3256: writing smithi12556470-21 from 571735 to 993065 tid 1 2021-03-24T22:06:53.503 INFO:tasks.rados.rados.0.smithi125.stdout:3256: writing smithi12556470-21 from 1562132 to 1978840 tid 2 2021-03-24T22:06:53.503 INFO:tasks.rados.rados.0.smithi125.stdout:3257: snap_create 2021-03-24T22:06:53.503 INFO:tasks.rados.rados.0.smithi125.stdout:3253: expect (ObjNum 1022 snap 337 seq_num 1022) 2021-03-24T22:06:53.595 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:53.608 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:53 smithi142 conmon[20125]: audit 2021-03-24T22:06:52.282283+0000 mgr.y (mgr.14144) 698 : audit 2021-03-24T22:06:53.609 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:53 smithi142 conmon[20125]: [DBG] from='client.15978 v1:172.21.15.125:0/168194873' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T22:06:53.609 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:53 smithi142 conmon[20125]: cluster 2021-03-24T22:06:52.448416+0000 mon.a (mon.0) 2078 : cluster [DBG] osdmap e842: 8 total, 8 up, 8 in 2021-03-24T22:06:53.658 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:53.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:53.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:53.765 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:53.777 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:53 smithi125 conmon[24927]: audit 2021-03-24T22:06:52.282283+0000 mgr.y (mgr.14144) 2021-03-24T22:06:53.778 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:53 smithi125 conmon[24927]: 698 : audit [DBG] from='client.15978 v1:172.21.15.125:0/168194873' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T22:06:53.778 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:53 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:06:53.778 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:53 smithi125 conmon[24927]: 24T22:06:52.448416+0000 mon.a (mon.0) 2021-03-24T22:06:53.779 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:53 smithi125 conmon[24927]: 2078 : cluster [DBG] osdmap e842: 8 total, 8 up, 8 in 2021-03-24T22:06:53.779 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:53 smithi125 conmon[19936]: audit 2021-03-24 2021-03-24T22:06:53.779 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:53 smithi125 conmon[19936]: T22:06:52.282283+0000 mgr.y (mgr.14144) 698 : audit [DBG] from='client.15978 v1:172.21.15.125:0/168194873' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-24T22:06:53.780 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:53 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:06:53.780 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:53 smithi125 conmon[19936]: -03-24T22:06:52. 2021-03-24T22:06:53.780 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:53 smithi125 conmon[19936]: 448416+0000 mon.a (mon.0 2021-03-24T22:06:53.780 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:53 smithi125 conmon[19936]: ) 2078 : cluster [DBG] osdmap e842: 8 total, 8 up, 8 in 2021-03-24T22:06:53.806 INFO:tasks.rados.rados.0.smithi125.stdout:3254: finishing write tid 1 to smithi12556470-42 2021-03-24T22:06:53.807 INFO:tasks.rados.rados.0.smithi125.stdout:3254: finishing write tid 2 to smithi12556470-42 2021-03-24T22:06:53.807 INFO:tasks.rados.rados.0.smithi125.stdout:3254: finishing write tid 3 to smithi12556470-42 2021-03-24T22:06:53.807 INFO:tasks.rados.rados.0.smithi125.stdout:3254: finishing write tid 4 to smithi12556470-42 2021-03-24T22:06:53.807 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 42 v 516 (ObjNum 1104 snap 377 seq_num 1104) dirty exists 2021-03-24T22:06:53.808 INFO:tasks.rados.rados.0.smithi125.stdout:3254: left oid 42 (ObjNum 1104 snap 377 seq_num 1104) 2021-03-24T22:06:53.808 INFO:tasks.rados.rados.0.smithi125.stdout:3256: finishing write tid 1 to smithi12556470-21 2021-03-24T22:06:53.808 INFO:tasks.rados.rados.0.smithi125.stdout:3256: finishing write tid 2 to smithi12556470-21 2021-03-24T22:06:53.808 INFO:tasks.rados.rados.0.smithi125.stdout:3256: finishing write tid 3 to smithi12556470-21 2021-03-24T22:06:53.809 INFO:tasks.rados.rados.0.smithi125.stdout:3256: finishing write tid 4 to smithi12556470-21 2021-03-24T22:06:53.809 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 21 v 1445 (ObjNum 1105 snap 377 seq_num 1105) dirty exists 2021-03-24T22:06:53.809 INFO:tasks.rados.rados.0.smithi125.stdout:3256: left oid 21 (ObjNum 1105 snap 377 seq_num 1105) 2021-03-24T22:06:53.809 INFO:tasks.rados.rados.0.smithi125.stdout:3255: expect (ObjNum 949 snap 314 seq_num 949) 2021-03-24T22:06:53.821 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:53.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:53.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:06:53.925 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:53.972 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:53.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:53.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:06:54.077 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:54.131 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:54.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:54.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:06:54.181 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-24T22:06:54.181 INFO:teuthology.orchestra.run.smithi125.stdout: "size": 20, 2021-03-24T22:06:54.182 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 600, 2021-03-24T22:06:54.182 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [ 2021-03-24T22:06:54.182 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.182 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:2394 7.23 7:c49de0b4:::smithi12556470-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e275)", 2021-03-24T22:06:54.183 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:57:25.176373+0000", 2021-03-24T22:06:54.183 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 568.59096771600002, 2021-03-24T22:06:54.183 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.025101692, 2021-03-24T22:06:54.183 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:06:54.183 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T22:06:54.184 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:06:54.184 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:06:54.184 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:06:54.184 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 2394 2021-03-24T22:06:54.185 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.185 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:06:54.185 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.185 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:06:54.185 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176373+0000", 2021-03-24T22:06:54.186 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:06:54.186 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.186 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.186 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:06:54.187 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176373+0000", 2021-03-24T22:06:54.187 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T22:06:54.187 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.187 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.187 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:06:54.188 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176371+0000", 2021-03-24T22:06:54.188 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.08e-06 2021-03-24T22:06:54.188 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.188 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.189 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:06:54.189 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176378+0000", 2021-03-24T22:06:54.189 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1309e-05 2021-03-24T22:06:54.189 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.189 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.190 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:06:54.190 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176389+0000", 2021-03-24T22:06:54.190 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.8714e-05 2021-03-24T22:06:54.190 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.191 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.191 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:06:54.191 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176418+0000", 2021-03-24T22:06:54.191 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6766000000000004e-05 2021-03-24T22:06:54.191 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.192 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.192 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.192 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176485+0000", 2021-03-24T22:06:54.192 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.1589999999999999e-06 2021-03-24T22:06:54.192 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.193 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.193 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:06:54.193 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:25.176494+0000", 2021-03-24T22:06:54.193 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0071017390000001 2021-03-24T22:06:54.194 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.194 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.194 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.194 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.183596+0000", 2021-03-24T22:06:54.195 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6001999999999996e-05 2021-03-24T22:06:54.195 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.195 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.195 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:06:54.195 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.183662+0000", 2021-03-24T22:06:54.196 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.017585719 2021-03-24T22:06:54.196 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.196 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.196 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.197 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.201247+0000", 2021-03-24T22:06:54.197 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000106111 2021-03-24T22:06:54.197 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.197 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.197 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:06:54.198 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.201354+0000", 2021-03-24T22:06:54.198 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000120763 2021-03-24T22:06:54.198 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.198 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.199 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:06:54.199 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:57:26.201474+0000", 2021-03-24T22:06:54.199 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.025101692 2021-03-24T22:06:54.199 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:06:54.199 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:06:54.200 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:06:54.200 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.200 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.200 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:3950 7.c 7:32974111:::smithi12556470-2:head [write 675780~495523,stat] snapc 136=[136,133,132,130,12d,12c,12b,129,128,127,126,120,11e,11b,11a,119,110,10f,10c,106,105,104,fa,f8,e7] ondisk+write+known_if_redirected e417)", 2021-03-24T22:06:54.201 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:47.297626+0000", 2021-03-24T22:06:54.201 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 426.46971422500002, 2021-03-24T22:06:54.201 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0613052380000001, 2021-03-24T22:06:54.201 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:06:54.201 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:06:54.202 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:06:54.202 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:06:54.202 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:06:54.202 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3950 2021-03-24T22:06:54.203 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.203 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:06:54.203 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.203 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:06:54.203 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.297626+0000", 2021-03-24T22:06:54.204 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:06:54.204 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.204 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.204 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:06:54.205 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.297626+0000", 2021-03-24T22:06:54.205 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999976 2021-03-24T22:06:54.205 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.205 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.205 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:06:54.206 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.297624+0000", 2021-03-24T22:06:54.206 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00032374599999999997 2021-03-24T22:06:54.206 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.206 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.207 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:06:54.207 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.297948+0000", 2021-03-24T22:06:54.207 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1829e-05 2021-03-24T22:06:54.207 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.207 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.208 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:06:54.208 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.297959+0000", 2021-03-24T22:06:54.208 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.9770000000000003e-06 2021-03-24T22:06:54.208 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.209 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.209 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:06:54.209 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.297966+0000", 2021-03-24T22:06:54.209 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001568081 2021-03-24T22:06:54.209 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.210 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.210 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.210 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299534+0000", 2021-03-24T22:06:54.210 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.7069999999999998e-06 2021-03-24T22:06:54.211 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.211 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.211 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:06:54.211 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299543+0000", 2021-03-24T22:06:54.211 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.02708646 2021-03-24T22:06:54.212 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.212 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.212 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.212 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.326630+0000", 2021-03-24T22:06:54.212 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000180722 2021-03-24T22:06:54.213 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.213 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.213 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:06:54.213 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.326810+0000", 2021-03-24T22:06:54.214 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00025251599999999999 2021-03-24T22:06:54.214 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.214 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.214 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,2", 2021-03-24T22:06:54.215 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.327063+0000", 2021-03-24T22:06:54.215 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0041170629999999998 2021-03-24T22:06:54.215 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.215 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.215 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:06:54.216 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.331180+0000", 2021-03-24T22:06:54.216 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.024822364999999999 2021-03-24T22:06:54.216 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.216 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.217 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:06:54.217 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.356002+0000", 2021-03-24T22:06:54.217 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000346561 2021-03-24T22:06:54.217 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.217 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.218 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:06:54.218 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.356349+0000", 2021-03-24T22:06:54.218 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00048905499999999996 2021-03-24T22:06:54.218 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.218 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.219 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:06:54.219 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.356838+0000", 2021-03-24T22:06:54.219 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0020934259999999998 2021-03-24T22:06:54.219 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.220 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.220 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:06:54.220 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.358931+0000", 2021-03-24T22:06:54.220 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0613052380000001 2021-03-24T22:06:54.220 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:06:54.221 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:06:54.221 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:06:54.221 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.221 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.222 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:3951 7.c 7:32974111:::smithi12556470-2:head [write 1682747~714511] snapc 136=[136,133,132,130,12d,12c,12b,129,128,127,126,120,11e,11b,11a,119,110,10f,10c,106,105,104,fa,f8,e7] ondisk+write+known_if_redirected e417)", 2021-03-24T22:06:54.222 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:47.298949+0000", 2021-03-24T22:06:54.222 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 426.468391285, 2021-03-24T22:06:54.222 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0693593290000001, 2021-03-24T22:06:54.222 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:06:54.223 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:06:54.223 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:06:54.223 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:06:54.223 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:06:54.224 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3951 2021-03-24T22:06:54.224 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.224 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:06:54.224 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.225 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:06:54.225 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.298949+0000", 2021-03-24T22:06:54.225 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:06:54.225 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.226 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.226 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:06:54.226 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.298949+0000", 2021-03-24T22:06:54.226 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T22:06:54.226 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.227 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.227 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:06:54.227 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.298948+0000", 2021-03-24T22:06:54.227 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000412256 2021-03-24T22:06:54.228 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.228 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.228 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:06:54.228 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299360+0000", 2021-03-24T22:06:54.229 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1399999999999999e-05 2021-03-24T22:06:54.229 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.229 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.229 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:06:54.229 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299371+0000", 2021-03-24T22:06:54.230 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.3679999999999996e-06 2021-03-24T22:06:54.230 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.230 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.230 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:06:54.231 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299379+0000", 2021-03-24T22:06:54.231 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00048704199999999998 2021-03-24T22:06:54.231 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.231 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.231 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.232 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299866+0000", 2021-03-24T22:06:54.232 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.277e-06 2021-03-24T22:06:54.232 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.232 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.233 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:06:54.233 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299873+0000", 2021-03-24T22:06:54.233 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0281777940000001 2021-03-24T22:06:54.233 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.233 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.234 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.234 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.328051+0000", 2021-03-24T22:06:54.234 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000151185 2021-03-24T22:06:54.234 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.235 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.235 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:06:54.235 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.328202+0000", 2021-03-24T22:06:54.235 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.030788554999999999 2021-03-24T22:06:54.236 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.236 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.236 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.236 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.358991+0000", 2021-03-24T22:06:54.237 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012337099999999999 2021-03-24T22:06:54.237 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.237 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.237 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:06:54.237 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.359114+0000", 2021-03-24T22:06:54.238 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00022504400000000001 2021-03-24T22:06:54.238 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.238 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.238 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,2", 2021-03-24T22:06:54.239 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.359339+0000", 2021-03-24T22:06:54.239 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0044580849999999997 2021-03-24T22:06:54.239 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.239 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.239 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:06:54.240 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.363797+0000", 2021-03-24T22:06:54.240 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0031527479999999999 2021-03-24T22:06:54.240 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.240 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.240 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:06:54.241 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.366950+0000", 2021-03-24T22:06:54.241 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001068685 2021-03-24T22:06:54.241 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.241 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.242 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:06:54.242 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.368018+0000", 2021-03-24T22:06:54.242 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.4264999999999995e-05 2021-03-24T22:06:54.242 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.242 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.243 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:06:54.243 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.368093+0000", 2021-03-24T22:06:54.243 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00021562 2021-03-24T22:06:54.243 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.244 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.244 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:06:54.244 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.368308+0000", 2021-03-24T22:06:54.244 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0693593290000001 2021-03-24T22:06:54.244 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:06:54.245 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:06:54.245 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:06:54.245 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.245 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.245 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:3952 7.c 7:32974111:::smithi12556470-2:head [write 3012769~63011] snapc 136=[136,133,132,130,12d,12c,12b,129,128,127,126,120,11e,11b,11a,119,110,10f,10c,106,105,104,fa,f8,e7] ondisk+write+known_if_redirected e417)", 2021-03-24T22:06:54.246 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:47.299385+0000", 2021-03-24T22:06:54.246 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 426.46795493899998, 2021-03-24T22:06:54.246 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.070508668, 2021-03-24T22:06:54.246 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:06:54.247 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:06:54.247 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:06:54.247 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:06:54.247 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:06:54.247 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3952 2021-03-24T22:06:54.248 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.248 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:06:54.248 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.248 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:06:54.249 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299385+0000", 2021-03-24T22:06:54.249 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:06:54.249 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.249 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.249 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:06:54.250 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299385+0000", 2021-03-24T22:06:54.250 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T22:06:54.250 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.250 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.251 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:06:54.251 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299384+0000", 2021-03-24T22:06:54.251 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.6712000000000001e-05 2021-03-24T22:06:54.251 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.251 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.252 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:06:54.252 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299411+0000", 2021-03-24T22:06:54.252 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6420000000000001e-06 2021-03-24T22:06:54.252 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.253 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.253 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:06:54.253 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299418+0000", 2021-03-24T22:06:54.253 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6809999999999997e-06 2021-03-24T22:06:54.253 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.254 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.254 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:06:54.254 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299422+0000", 2021-03-24T22:06:54.254 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000472101 2021-03-24T22:06:54.255 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.255 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.255 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.255 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299895+0000", 2021-03-24T22:06:54.255 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.0889999999999996e-06 2021-03-24T22:06:54.256 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.256 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.256 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:06:54.256 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299901+0000", 2021-03-24T22:06:54.257 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0283333750000001 2021-03-24T22:06:54.257 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.257 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.257 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.257 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.328234+0000", 2021-03-24T22:06:54.258 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000104591 2021-03-24T22:06:54.258 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.258 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.258 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:06:54.258 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.328339+0000", 2021-03-24T22:06:54.259 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.033718655 2021-03-24T22:06:54.259 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.259 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.259 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.260 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.362057+0000", 2021-03-24T22:06:54.260 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00014753499999999999 2021-03-24T22:06:54.260 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.260 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.260 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:06:54.261 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.362205+0000", 2021-03-24T22:06:54.261 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000219259 2021-03-24T22:06:54.261 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.261 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.262 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,2", 2021-03-24T22:06:54.262 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.362424+0000", 2021-03-24T22:06:54.262 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001410356 2021-03-24T22:06:54.262 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.262 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.263 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:06:54.263 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.363834+0000", 2021-03-24T22:06:54.263 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.003124342 2021-03-24T22:06:54.263 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.264 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.264 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:06:54.264 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.366959+0000", 2021-03-24T22:06:54.264 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0014798490000000001 2021-03-24T22:06:54.265 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.265 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.265 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:06:54.265 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.368439+0000", 2021-03-24T22:06:54.265 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001231179 2021-03-24T22:06:54.266 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.266 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.266 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:06:54.266 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.369670+0000", 2021-03-24T22:06:54.267 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00022420900000000001 2021-03-24T22:06:54.267 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.267 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.267 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:06:54.268 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.369894+0000", 2021-03-24T22:06:54.268 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.070508668 2021-03-24T22:06:54.268 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:06:54.268 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:06:54.268 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:06:54.269 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.269 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.269 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:3953 7.c 7:32974111:::smithi12556470-2:head [setxattr _header (57) in=7b,truncate 3075780] snapc 136=[136,133,132,130,12d,12c,12b,129,128,127,126,120,11e,11b,11a,119,110,10f,10c,106,105,104,fa,f8,e7] ondisk+write+known_if_redirected e417)", 2021-03-24T22:06:54.269 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:47.299427+0000", 2021-03-24T22:06:54.270 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 426.46791317100002, 2021-03-24T22:06:54.270 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0708905799999999, 2021-03-24T22:06:54.270 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:06:54.270 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:06:54.270 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:06:54.271 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:06:54.271 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:06:54.271 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3953 2021-03-24T22:06:54.271 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.272 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:06:54.272 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.272 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:06:54.272 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299427+0000", 2021-03-24T22:06:54.272 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:06:54.273 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.273 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.273 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:06:54.273 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299427+0000", 2021-03-24T22:06:54.274 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999995 2021-03-24T22:06:54.274 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.274 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.274 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:06:54.274 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299427+0000", 2021-03-24T22:06:54.275 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.4259999999999999e-06 2021-03-24T22:06:54.275 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.275 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.275 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:06:54.276 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299430+0000", 2021-03-24T22:06:54.276 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1189999999999996e-06 2021-03-24T22:06:54.276 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.276 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.276 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:06:54.277 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299435+0000", 2021-03-24T22:06:54.277 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9140000000000001e-06 2021-03-24T22:06:54.277 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.277 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.278 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:06:54.278 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299439+0000", 2021-03-24T22:06:54.278 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00048008799999999999 2021-03-24T22:06:54.278 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.278 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.279 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.279 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299919+0000", 2021-03-24T22:06:54.279 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.835e-06 2021-03-24T22:06:54.279 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.280 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.280 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:06:54.280 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299925+0000", 2021-03-24T22:06:54.280 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.028441138 2021-03-24T22:06:54.280 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.281 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.281 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.281 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.328366+0000", 2021-03-24T22:06:54.281 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010274500000000001 2021-03-24T22:06:54.281 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.282 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.282 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:06:54.282 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.328469+0000", 2021-03-24T22:06:54.282 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.034522276999999997 2021-03-24T22:06:54.283 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.283 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.283 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.283 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.362991+0000", 2021-03-24T22:06:54.284 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000114469 2021-03-24T22:06:54.284 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.284 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.284 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:06:54.284 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.363106+0000", 2021-03-24T22:06:54.285 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00021205599999999999 2021-03-24T22:06:54.285 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.285 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.285 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,2", 2021-03-24T22:06:54.286 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.363318+0000", 2021-03-24T22:06:54.286 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00096840900000000004 2021-03-24T22:06:54.286 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.286 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.286 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:06:54.287 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.364286+0000", 2021-03-24T22:06:54.287 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002681143 2021-03-24T22:06:54.287 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.287 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.288 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:06:54.288 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.366967+0000", 2021-03-24T22:06:54.288 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002998559 2021-03-24T22:06:54.288 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.288 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.289 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:06:54.289 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.369966+0000", 2021-03-24T22:06:54.289 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012207299999999999 2021-03-24T22:06:54.289 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.289 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.290 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:06:54.290 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.370088+0000", 2021-03-24T22:06:54.290 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00022991299999999999 2021-03-24T22:06:54.290 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.291 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.291 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:06:54.291 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.370318+0000", 2021-03-24T22:06:54.291 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0708905799999999 2021-03-24T22:06:54.291 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:06:54.292 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:06:54.292 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:06:54.292 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.292 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.293 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:3954 7.c 7:32974111:::smithi12556470-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e417)", 2021-03-24T22:06:54.293 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T21:59:47.299443+0000", 2021-03-24T22:06:54.293 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 426.46789748800001, 2021-03-24T22:06:54.293 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.071832683, 2021-03-24T22:06:54.293 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:06:54.294 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T22:06:54.294 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:06:54.294 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:06:54.294 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:06:54.295 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3954 2021-03-24T22:06:54.295 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.295 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:06:54.295 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.295 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:06:54.296 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299443+0000", 2021-03-24T22:06:54.296 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:06:54.296 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.296 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.296 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:06:54.297 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299443+0000", 2021-03-24T22:06:54.297 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999995 2021-03-24T22:06:54.297 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.297 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.298 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:06:54.298 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299442+0000", 2021-03-24T22:06:54.298 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.2239999999999998e-06 2021-03-24T22:06:54.298 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.298 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.299 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:06:54.299 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299445+0000", 2021-03-24T22:06:54.299 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.3809999999999998e-06 2021-03-24T22:06:54.299 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.300 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.300 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:06:54.300 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299450+0000", 2021-03-24T22:06:54.300 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9160000000000003e-06 2021-03-24T22:06:54.300 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.301 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.301 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:06:54.301 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299454+0000", 2021-03-24T22:06:54.301 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00049169700000000003 2021-03-24T22:06:54.301 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.302 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.302 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.302 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299946+0000", 2021-03-24T22:06:54.302 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.003e-06 2021-03-24T22:06:54.303 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.303 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.303 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:06:54.303 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:47.299952+0000", 2021-03-24T22:06:54.303 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0285410699999999 2021-03-24T22:06:54.304 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.304 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.304 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.304 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.328493+0000", 2021-03-24T22:06:54.305 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0015757410000000001 2021-03-24T22:06:54.305 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.305 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.305 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:06:54.305 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.330068+0000", 2021-03-24T22:06:54.306 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.033581221000000001 2021-03-24T22:06:54.306 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.306 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.306 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.307 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.363650+0000", 2021-03-24T22:06:54.307 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010419700000000001 2021-03-24T22:06:54.307 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.307 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.308 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:06:54.308 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.363754+0000", 2021-03-24T22:06:54.308 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0070060599999999997 2021-03-24T22:06:54.308 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.308 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.309 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.309 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.370760+0000", 2021-03-24T22:06:54.309 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00020776099999999999 2021-03-24T22:06:54.309 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.309 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.310 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:06:54.310 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.370968+0000", 2021-03-24T22:06:54.311 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00030779100000000002 2021-03-24T22:06:54.311 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.311 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.311 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:06:54.312 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T21:59:48.371275+0000", 2021-03-24T22:06:54.312 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.071832683 2021-03-24T22:06:54.312 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:06:54.312 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:06:54.312 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:06:54.313 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.313 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.313 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:4147 7.21 7:86662589:::smithi12556470-19:head [write 712762~401056,stat] snapc 143=[143,140,13d,13b,139,137,136,12d,12b,129,128,127,126,120,11e,11b,11a,119,110,10f,10c,105,104,fa,f8,e7] ondisk+write+known_if_redirected e432)", 2021-03-24T22:06:54.313 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:00:02.248651+0000", 2021-03-24T22:06:54.314 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 411.51868917600001, 2021-03-24T22:06:54.314 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0717329640000002, 2021-03-24T22:06:54.314 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:06:54.314 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:06:54.315 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:06:54.315 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:06:54.315 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:06:54.315 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4147 2021-03-24T22:06:54.315 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.316 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:06:54.316 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.316 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:06:54.316 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.248651+0000", 2021-03-24T22:06:54.317 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:06:54.317 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.317 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.317 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:06:54.317 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.248651+0000", 2021-03-24T22:06:54.318 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999967 2021-03-24T22:06:54.318 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.318 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.318 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:06:54.318 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.248648+0000", 2021-03-24T22:06:54.319 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000104109 2021-03-24T22:06:54.319 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.319 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.319 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:06:54.320 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.248752+0000", 2021-03-24T22:06:54.320 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002002804 2021-03-24T22:06:54.320 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.320 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.320 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:06:54.321 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.250755+0000", 2021-03-24T22:06:54.321 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5437000000000001e-05 2021-03-24T22:06:54.321 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.321 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.322 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:06:54.322 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.250770+0000", 2021-03-24T22:06:54.322 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0014452029999999999 2021-03-24T22:06:54.322 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.322 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.323 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.323 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.252215+0000", 2021-03-24T22:06:54.323 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.8291000000000002e-05 2021-03-24T22:06:54.323 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.324 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.324 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:06:54.324 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.252234+0000", 2021-03-24T22:06:54.324 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99172324700000003 2021-03-24T22:06:54.324 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.325 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.325 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.325 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:03.243957+0000", 2021-03-24T22:06:54.325 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.9359999999999994e-06 2021-03-24T22:06:54.325 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.326 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.326 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:06:54.326 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:03.243967+0000", 2021-03-24T22:06:54.326 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0558876720000001 2021-03-24T22:06:54.327 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.327 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.327 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.327 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.299854+0000", 2021-03-24T22:06:54.327 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00045708100000000002 2021-03-24T22:06:54.328 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.328 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.328 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for degraded object", 2021-03-24T22:06:54.328 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.300311+0000", 2021-03-24T22:06:54.329 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0088695200000000005 2021-03-24T22:06:54.329 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.329 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.329 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.329 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.309181+0000", 2021-03-24T22:06:54.330 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000101866 2021-03-24T22:06:54.330 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.330 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.330 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:06:54.331 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.309283+0000", 2021-03-24T22:06:54.331 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00025108500000000001 2021-03-24T22:06:54.331 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.331 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.331 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,5", 2021-03-24T22:06:54.332 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.309534+0000", 2021-03-24T22:06:54.332 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001499727 2021-03-24T22:06:54.332 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.332 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.333 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:06:54.333 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.311034+0000", 2021-03-24T22:06:54.333 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001462262 2021-03-24T22:06:54.333 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.333 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.334 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:06:54.334 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.312496+0000", 2021-03-24T22:06:54.334 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0076004449999999999 2021-03-24T22:06:54.334 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.335 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.335 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:06:54.335 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.320096+0000", 2021-03-24T22:06:54.335 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.8819999999999993e-05 2021-03-24T22:06:54.335 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.336 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.336 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:06:54.336 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.320185+0000", 2021-03-24T22:06:54.336 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00019890500000000001 2021-03-24T22:06:54.336 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.337 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.337 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:06:54.337 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.320384+0000", 2021-03-24T22:06:54.337 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0717329640000002 2021-03-24T22:06:54.338 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:06:54.338 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:06:54.338 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:06:54.338 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.338 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.339 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:4148 7.21 7:86662589:::smithi12556470-19:head [write 1537867~645855] snapc 143=[143,140,13d,13b,139,137,136,12d,12b,129,128,127,126,120,11e,11b,11a,119,110,10f,10c,105,104,fa,f8,e7] ondisk+write+known_if_redirected e432)", 2021-03-24T22:06:54.339 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:00:02.257275+0000", 2021-03-24T22:06:54.339 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 411.51006485099998, 2021-03-24T22:06:54.339 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.067389844, 2021-03-24T22:06:54.340 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:06:54.340 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:06:54.340 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:06:54.340 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:06:54.340 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:06:54.341 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4148 2021-03-24T22:06:54.341 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.341 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:06:54.341 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.342 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:06:54.342 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.257275+0000", 2021-03-24T22:06:54.342 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:06:54.342 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.342 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.343 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:06:54.343 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.257275+0000", 2021-03-24T22:06:54.343 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999962 2021-03-24T22:06:54.344 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.344 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.344 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:06:54.344 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.257271+0000", 2021-03-24T22:06:54.345 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00017391499999999999 2021-03-24T22:06:54.345 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.345 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.345 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:06:54.346 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.257445+0000", 2021-03-24T22:06:54.346 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0049961399999999996 2021-03-24T22:06:54.346 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.346 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.347 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:06:54.347 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.262441+0000", 2021-03-24T22:06:54.347 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2265e-05 2021-03-24T22:06:54.347 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.347 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.348 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:06:54.348 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.262454+0000", 2021-03-24T22:06:54.348 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.013052934 2021-03-24T22:06:54.348 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.349 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.349 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.349 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.275507+0000", 2021-03-24T22:06:54.349 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.5969999999999998e-06 2021-03-24T22:06:54.350 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.350 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.350 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:06:54.350 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.275516+0000", 2021-03-24T22:06:54.351 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.96847310900000005 2021-03-24T22:06:54.351 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.351 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.351 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.352 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:03.243989+0000", 2021-03-24T22:06:54.352 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.626e-06 2021-03-24T22:06:54.352 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.352 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.353 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:06:54.353 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:03.243995+0000", 2021-03-24T22:06:54.353 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0563536250000001 2021-03-24T22:06:54.353 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.353 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.354 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.354 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.300349+0000", 2021-03-24T22:06:54.354 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.6805000000000007e-05 2021-03-24T22:06:54.354 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.355 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.355 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for degraded object", 2021-03-24T22:06:54.355 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.300435+0000", 2021-03-24T22:06:54.355 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.009989965 2021-03-24T22:06:54.356 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.356 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.356 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.356 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.310425+0000", 2021-03-24T22:06:54.356 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000105184 2021-03-24T22:06:54.357 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.357 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.358 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:06:54.359 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.310531+0000", 2021-03-24T22:06:54.359 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0099567490000000008 2021-03-24T22:06:54.359 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.359 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.360 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.360 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.320487+0000", 2021-03-24T22:06:54.360 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00014939599999999999 2021-03-24T22:06:54.360 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.361 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.361 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:06:54.361 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.320637+0000", 2021-03-24T22:06:54.361 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00031880400000000001 2021-03-24T22:06:54.361 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.362 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.362 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,5", 2021-03-24T22:06:54.362 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.320956+0000", 2021-03-24T22:06:54.362 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.003448596 2021-03-24T22:06:54.363 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.363 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.363 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:06:54.363 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.324404+0000", 2021-03-24T22:06:54.364 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.4194e-05 2021-03-24T22:06:54.364 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.364 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.364 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:06:54.365 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.324418+0000", 2021-03-24T22:06:54.365 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.0278000000000002e-05 2021-03-24T22:06:54.365 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.365 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.365 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:06:54.366 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.324469+0000", 2021-03-24T22:06:54.366 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.4826999999999994e-05 2021-03-24T22:06:54.366 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.366 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.367 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:06:54.367 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.324543+0000", 2021-03-24T22:06:54.367 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000121864 2021-03-24T22:06:54.367 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.368 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.368 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:06:54.368 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.324665+0000", 2021-03-24T22:06:54.368 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.067389844 2021-03-24T22:06:54.369 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:06:54.369 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:06:54.369 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:06:54.370 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.370 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.370 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:4149 7.21 7:86662589:::smithi12556470-19:head [write 2860355~652407] snapc 143=[143,140,13d,13b,139,137,136,12d,12b,129,128,127,126,120,11e,11b,11a,119,110,10f,10c,105,104,fa,f8,e7] ondisk+write+known_if_redirected e432)", 2021-03-24T22:06:54.371 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:00:02.264982+0000", 2021-03-24T22:06:54.371 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 411.50235817100003, 2021-03-24T22:06:54.371 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.060287465, 2021-03-24T22:06:54.371 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:06:54.372 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:06:54.372 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:06:54.372 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:06:54.372 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:06:54.373 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4149 2021-03-24T22:06:54.373 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.373 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:06:54.373 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.374 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:06:54.374 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.264982+0000", 2021-03-24T22:06:54.374 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:06:54.374 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.375 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.375 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:06:54.375 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.264982+0000", 2021-03-24T22:06:54.375 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999881 2021-03-24T22:06:54.375 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.376 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.376 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:06:54.376 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.264970+0000", 2021-03-24T22:06:54.376 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0010432900000000001 2021-03-24T22:06:54.377 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.377 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.377 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:06:54.377 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.266013+0000", 2021-03-24T22:06:54.378 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0014047409999999999 2021-03-24T22:06:54.378 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.378 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.378 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:06:54.379 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.267418+0000", 2021-03-24T22:06:54.379 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2789000000000001e-05 2021-03-24T22:06:54.379 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.379 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.380 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:06:54.380 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.267431+0000", 2021-03-24T22:06:54.380 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0081054939999999995 2021-03-24T22:06:54.380 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.381 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.381 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.382 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.275536+0000", 2021-03-24T22:06:54.383 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.7870000000000004e-06 2021-03-24T22:06:54.383 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.383 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.383 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:06:54.384 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.275543+0000", 2021-03-24T22:06:54.384 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.96846978399999994 2021-03-24T22:06:54.384 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.384 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.384 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.385 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:03.244013+0000", 2021-03-24T22:06:54.385 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6439999999999997e-06 2021-03-24T22:06:54.385 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.385 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.385 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:06:54.386 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:03.244019+0000", 2021-03-24T22:06:54.386 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.056445708 2021-03-24T22:06:54.386 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.386 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.387 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.387 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.300464+0000", 2021-03-24T22:06:54.387 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001583262 2021-03-24T22:06:54.387 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.388 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.388 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for degraded object", 2021-03-24T22:06:54.388 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.302048+0000", 2021-03-24T22:06:54.388 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0085309080000000002 2021-03-24T22:06:54.389 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.389 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.389 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.389 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.310579+0000", 2021-03-24T22:06:54.390 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.8817e-05 2021-03-24T22:06:54.390 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.390 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.390 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:06:54.390 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.310667+0000", 2021-03-24T22:06:54.391 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011484482000000001 2021-03-24T22:06:54.391 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.391 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.391 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.392 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.322152+0000", 2021-03-24T22:06:54.392 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.6207999999999999e-05 2021-03-24T22:06:54.392 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.392 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.393 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:06:54.393 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.322228+0000", 2021-03-24T22:06:54.393 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00014250999999999999 2021-03-24T22:06:54.393 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.393 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.394 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,5", 2021-03-24T22:06:54.394 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.322371+0000", 2021-03-24T22:06:54.394 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0020567459999999999 2021-03-24T22:06:54.394 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.395 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.395 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:06:54.395 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.324427+0000", 2021-03-24T22:06:54.395 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00031726900000000001 2021-03-24T22:06:54.395 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.396 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.396 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:06:54.396 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.324745+0000", 2021-03-24T22:06:54.397 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00031702800000000001 2021-03-24T22:06:54.397 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.397 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.398 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:06:54.398 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.325062+0000", 2021-03-24T22:06:54.398 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.4227999999999997e-05 2021-03-24T22:06:54.398 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.399 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.399 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:06:54.399 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.325106+0000", 2021-03-24T22:06:54.399 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00016376499999999999 2021-03-24T22:06:54.400 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.400 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.400 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:06:54.400 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.325270+0000", 2021-03-24T22:06:54.400 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.060287465 2021-03-24T22:06:54.401 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:06:54.401 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:06:54.401 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:06:54.401 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.401 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.402 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:4150 7.21 7:86662589:::smithi12556470-19:head [setxattr _header (58) in=7b,truncate 3512762] snapc 143=[143,140,13d,13b,139,137,136,12d,12b,129,128,127,126,120,11e,11b,11a,119,110,10f,10c,105,104,fa,f8,e7] ondisk+write+known_if_redirected e432)", 2021-03-24T22:06:54.402 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:00:02.267443+0000", 2021-03-24T22:06:54.402 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 411.49989779700002, 2021-03-24T22:06:54.402 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.058203818, 2021-03-24T22:06:54.403 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:06:54.403 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:06:54.403 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:06:54.403 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:06:54.403 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:06:54.404 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4150 2021-03-24T22:06:54.404 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.404 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:06:54.404 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.404 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:06:54.405 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.267443+0000", 2021-03-24T22:06:54.405 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:06:54.405 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.405 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.406 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:06:54.406 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.267443+0000", 2021-03-24T22:06:54.406 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T22:06:54.406 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.407 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.407 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:06:54.407 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.267441+0000", 2021-03-24T22:06:54.407 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.1050000000000001e-06 2021-03-24T22:06:54.407 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.408 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.408 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:06:54.408 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.267450+0000", 2021-03-24T22:06:54.408 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.1170000000000004e-06 2021-03-24T22:06:54.409 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.409 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.409 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:06:54.409 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.267458+0000", 2021-03-24T22:06:54.409 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6759999999999998e-06 2021-03-24T22:06:54.410 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.410 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.410 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:06:54.410 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.267463+0000", 2021-03-24T22:06:54.411 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0081025090000000008 2021-03-24T22:06:54.411 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.411 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.411 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.411 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.275565+0000", 2021-03-24T22:06:54.412 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4189999999999999e-06 2021-03-24T22:06:54.412 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.412 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.412 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:06:54.412 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.275572+0000", 2021-03-24T22:06:54.413 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.96846758399999999 2021-03-24T22:06:54.413 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.413 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.413 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.413 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:03.244039+0000", 2021-03-24T22:06:54.414 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5249999999999996e-06 2021-03-24T22:06:54.414 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.414 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.414 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:06:54.415 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:03.244045+0000", 2021-03-24T22:06:54.415 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.058038864 2021-03-24T22:06:54.415 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.415 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.416 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.416 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.302084+0000", 2021-03-24T22:06:54.416 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.0760000000000005e-05 2021-03-24T22:06:54.416 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.416 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.417 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for degraded object", 2021-03-24T22:06:54.417 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.302174+0000", 2021-03-24T22:06:54.417 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0085442020000000007 2021-03-24T22:06:54.417 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.417 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.418 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.418 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.310718+0000", 2021-03-24T22:06:54.418 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.1786999999999994e-05 2021-03-24T22:06:54.418 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.419 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.419 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:06:54.419 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.310810+0000", 2021-03-24T22:06:54.419 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012559539 2021-03-24T22:06:54.420 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.420 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.420 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.420 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.323370+0000", 2021-03-24T22:06:54.420 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000127611 2021-03-24T22:06:54.421 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.421 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.421 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:06:54.421 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.323497+0000", 2021-03-24T22:06:54.421 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000301575 2021-03-24T22:06:54.422 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.422 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.422 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,5", 2021-03-24T22:06:54.422 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.323799+0000", 2021-03-24T22:06:54.423 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00088734600000000001 2021-03-24T22:06:54.423 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.423 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.424 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:06:54.424 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.324686+0000", 2021-03-24T22:06:54.424 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00066636400000000002 2021-03-24T22:06:54.424 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.424 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.425 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:06:54.425 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.325353+0000", 2021-03-24T22:06:54.425 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.1169999999999997e-05 2021-03-24T22:06:54.425 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.426 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.426 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:06:54.426 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.325434+0000", 2021-03-24T22:06:54.426 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.8564000000000002e-05 2021-03-24T22:06:54.426 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.427 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.427 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:06:54.427 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.325512+0000", 2021-03-24T22:06:54.427 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00013387499999999999 2021-03-24T22:06:54.428 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.428 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.428 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:06:54.428 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.325646+0000", 2021-03-24T22:06:54.428 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.058203818 2021-03-24T22:06:54.429 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:06:54.429 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:06:54.429 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:06:54.429 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.429 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.430 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:4151 7.21 7:86662589:::smithi12556470-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e432)", 2021-03-24T22:06:54.430 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:00:02.267468+0000", 2021-03-24T22:06:54.430 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 411.499872713, 2021-03-24T22:06:54.430 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0584516220000002, 2021-03-24T22:06:54.431 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:06:54.431 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T22:06:54.431 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:06:54.431 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:06:54.431 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:06:54.432 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4151 2021-03-24T22:06:54.432 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.432 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:06:54.432 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.433 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:06:54.433 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.267468+0000", 2021-03-24T22:06:54.433 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:06:54.433 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.434 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.434 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:06:54.434 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.267468+0000", 2021-03-24T22:06:54.434 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999995 2021-03-24T22:06:54.434 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.435 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.435 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:06:54.435 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.267467+0000", 2021-03-24T22:06:54.435 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.3579999999999999e-06 2021-03-24T22:06:54.435 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.436 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.436 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:06:54.436 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.267470+0000", 2021-03-24T22:06:54.436 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.3289999999999999e-06 2021-03-24T22:06:54.437 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.437 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.437 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:06:54.437 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.267477+0000", 2021-03-24T22:06:54.438 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3869999999999997e-06 2021-03-24T22:06:54.438 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.438 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.438 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:06:54.438 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.267481+0000", 2021-03-24T22:06:54.439 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0081161340000000005 2021-03-24T22:06:54.439 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.439 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.439 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.440 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.275597+0000", 2021-03-24T22:06:54.440 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.925e-06 2021-03-24T22:06:54.440 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.441 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.441 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:06:54.441 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:02.275603+0000", 2021-03-24T22:06:54.441 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.96846318600000003 2021-03-24T22:06:54.441 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.442 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.442 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.442 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:03.244066+0000", 2021-03-24T22:06:54.442 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.4770000000000002e-06 2021-03-24T22:06:54.443 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.443 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.443 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:06:54.443 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:03.244072+0000", 2021-03-24T22:06:54.443 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.058135101 2021-03-24T22:06:54.444 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.444 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.444 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.444 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.302207+0000", 2021-03-24T22:06:54.445 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.2978000000000005e-05 2021-03-24T22:06:54.445 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.445 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.445 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for degraded object", 2021-03-24T22:06:54.445 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.302280+0000", 2021-03-24T22:06:54.446 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0085603620000000002 2021-03-24T22:06:54.446 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.446 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.446 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.446 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.310840+0000", 2021-03-24T22:06:54.447 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.5315999999999994e-05 2021-03-24T22:06:54.447 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.447 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.447 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:06:54.448 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.310926+0000", 2021-03-24T22:06:54.448 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.013304694000000001 2021-03-24T22:06:54.448 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.448 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.449 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.449 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.324230+0000", 2021-03-24T22:06:54.449 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00011130599999999999 2021-03-24T22:06:54.449 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.450 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.450 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:06:54.450 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.324342+0000", 2021-03-24T22:06:54.450 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0013441519999999999 2021-03-24T22:06:54.450 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.451 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.451 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.451 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.325686+0000", 2021-03-24T22:06:54.451 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5726999999999997e-05 2021-03-24T22:06:54.451 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.452 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.452 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:06:54.452 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.325741+0000", 2021-03-24T22:06:54.452 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00017782400000000001 2021-03-24T22:06:54.453 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.453 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.453 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:06:54.453 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:00:04.325919+0000", 2021-03-24T22:06:54.453 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0584516220000002 2021-03-24T22:06:54.454 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:06:54.454 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:06:54.454 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:06:54.454 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.454 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.455 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:4763 7.c 7:31cb714a:::smithi12556470-7:head [write 625392~569702] snapc 17d=[17d,17c,179,178,177,176,172,16e,16d,168,166,165,160,15e,159,157,153,139,136,129,127,11b,104,fa] ondisk+write+known_if_redirected e491)", 2021-03-24T22:06:54.455 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:01:01.275026+0000", 2021-03-24T22:06:54.455 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 352.49231454599999, 2021-03-24T22:06:54.455 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.06165336, 2021-03-24T22:06:54.455 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:06:54.456 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:06:54.456 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:06:54.456 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:06:54.456 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:06:54.457 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4763 2021-03-24T22:06:54.457 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.457 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:06:54.457 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.458 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:06:54.458 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.275026+0000", 2021-03-24T22:06:54.458 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:06:54.458 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.459 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.459 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:06:54.459 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.275026+0000", 2021-03-24T22:06:54.459 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999976 2021-03-24T22:06:54.459 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.460 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.460 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:06:54.460 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.275023+0000", 2021-03-24T22:06:54.460 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00032953799999999999 2021-03-24T22:06:54.461 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.461 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.461 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:06:54.461 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.275353+0000", 2021-03-24T22:06:54.461 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.3589999999999999e-05 2021-03-24T22:06:54.462 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.462 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.462 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:06:54.462 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.275377+0000", 2021-03-24T22:06:54.463 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.4150000000000005e-06 2021-03-24T22:06:54.463 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.463 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.463 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:06:54.463 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.275386+0000", 2021-03-24T22:06:54.464 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012973314999999999 2021-03-24T22:06:54.464 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.464 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.464 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.464 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.288359+0000", 2021-03-24T22:06:54.465 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.8028000000000006e-05 2021-03-24T22:06:54.465 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.465 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.466 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:06:54.466 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.288437+0000", 2021-03-24T22:06:54.466 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.039910114 2021-03-24T22:06:54.466 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.466 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.467 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.467 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.328347+0000", 2021-03-24T22:06:54.467 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00022543800000000001 2021-03-24T22:06:54.467 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.467 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.468 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:06:54.468 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.328573+0000", 2021-03-24T22:06:54.468 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00039166100000000002 2021-03-24T22:06:54.468 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.469 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.469 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,7", 2021-03-24T22:06:54.469 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.328965+0000", 2021-03-24T22:06:54.469 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0071784850000000001 2021-03-24T22:06:54.469 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.470 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.470 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:06:54.470 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.336143+0000", 2021-03-24T22:06:54.470 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00024411499999999999 2021-03-24T22:06:54.470 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.471 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.471 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:06:54.471 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.336387+0000", 2021-03-24T22:06:54.471 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.4493999999999995e-05 2021-03-24T22:06:54.472 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.472 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.472 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:06:54.472 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.336462+0000", 2021-03-24T22:06:54.472 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.0748999999999999e-05 2021-03-24T22:06:54.473 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.473 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.473 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:06:54.473 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.336502+0000", 2021-03-24T22:06:54.474 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00017668299999999999 2021-03-24T22:06:54.474 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.474 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.474 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:06:54.475 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.336679+0000", 2021-03-24T22:06:54.475 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.06165336 2021-03-24T22:06:54.475 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:06:54.475 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:06:54.475 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:06:54.476 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.476 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.476 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:4764 7.c 7:31cb714a:::smithi12556470-7:head [write 1637717~745548] snapc 17d=[17d,17c,179,178,177,176,172,16e,16d,168,166,165,160,15e,159,157,153,139,136,129,127,11b,104,fa] ondisk+write+known_if_redirected e491)", 2021-03-24T22:06:54.476 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:01:01.285992+0000", 2021-03-24T22:06:54.477 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 352.48134815499998, 2021-03-24T22:06:54.477 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0510351490000001, 2021-03-24T22:06:54.477 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:06:54.477 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:06:54.478 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:06:54.478 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:06:54.478 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:06:54.478 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4764 2021-03-24T22:06:54.478 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.479 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:06:54.479 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.479 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:06:54.479 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.285992+0000", 2021-03-24T22:06:54.480 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:06:54.480 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.480 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.480 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:06:54.481 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.285992+0000", 2021-03-24T22:06:54.481 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999981 2021-03-24T22:06:54.481 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.481 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.482 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:06:54.482 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.285990+0000", 2021-03-24T22:06:54.482 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00046463299999999999 2021-03-24T22:06:54.483 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.484 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.484 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:06:54.485 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286455+0000", 2021-03-24T22:06:54.485 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.464e-05 2021-03-24T22:06:54.485 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.486 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.486 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:06:54.486 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286469+0000", 2021-03-24T22:06:54.486 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.6230000000000003e-06 2021-03-24T22:06:54.487 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.487 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.488 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:06:54.488 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286478+0000", 2021-03-24T22:06:54.488 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002016393 2021-03-24T22:06:54.489 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.489 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.489 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.490 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.288494+0000", 2021-03-24T22:06:54.490 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.5879000000000001e-05 2021-03-24T22:06:54.490 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.490 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.491 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:06:54.491 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.288520+0000", 2021-03-24T22:06:54.491 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.04205784 2021-03-24T22:06:54.491 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.492 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.492 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.492 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.330578+0000", 2021-03-24T22:06:54.492 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000176718 2021-03-24T22:06:54.492 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.493 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.494 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:06:54.494 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.330755+0000", 2021-03-24T22:06:54.495 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00047453099999999999 2021-03-24T22:06:54.495 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.495 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.495 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,7", 2021-03-24T22:06:54.496 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.331229+0000", 2021-03-24T22:06:54.496 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.005170661 2021-03-24T22:06:54.496 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.496 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.497 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:06:54.497 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.336400+0000", 2021-03-24T22:06:54.497 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00033368500000000001 2021-03-24T22:06:54.497 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.498 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.498 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:06:54.498 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.336734+0000", 2021-03-24T22:06:54.498 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.5241999999999996e-05 2021-03-24T22:06:54.499 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.499 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.499 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:06:54.499 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.336809+0000", 2021-03-24T22:06:54.500 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.8261999999999999e-05 2021-03-24T22:06:54.500 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.500 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.500 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:06:54.500 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.336867+0000", 2021-03-24T22:06:54.501 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000160037 2021-03-24T22:06:54.501 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.502 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.502 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:06:54.503 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.337027+0000", 2021-03-24T22:06:54.503 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0510351490000001 2021-03-24T22:06:54.503 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:06:54.503 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:06:54.504 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:06:54.504 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.504 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.504 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:4765 7.c 7:31cb714a:::smithi12556470-7:head [write 2848542~176850] snapc 17d=[17d,17c,179,178,177,176,172,16e,16d,168,166,165,160,15e,159,157,153,139,136,129,127,11b,104,fa] ondisk+write+known_if_redirected e491)", 2021-03-24T22:06:54.505 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:01:01.286489+0000", 2021-03-24T22:06:54.505 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 352.48085128399998, 2021-03-24T22:06:54.505 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0507179900000001, 2021-03-24T22:06:54.505 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:06:54.505 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:06:54.506 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:06:54.506 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:06:54.506 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:06:54.506 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4765 2021-03-24T22:06:54.507 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.507 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:06:54.507 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.507 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:06:54.507 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286489+0000", 2021-03-24T22:06:54.508 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:06:54.508 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.508 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.508 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:06:54.509 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286489+0000", 2021-03-24T22:06:54.509 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999986 2021-03-24T22:06:54.509 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.509 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.510 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:06:54.510 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286488+0000", 2021-03-24T22:06:54.510 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000106881 2021-03-24T22:06:54.511 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.511 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.512 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:06:54.512 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286595+0000", 2021-03-24T22:06:54.512 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.9570000000000002e-06 2021-03-24T22:06:54.512 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.513 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.513 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:06:54.513 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286603+0000", 2021-03-24T22:06:54.513 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.7579999999999998e-06 2021-03-24T22:06:54.514 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.514 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.514 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:06:54.514 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286608+0000", 2021-03-24T22:06:54.515 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00353412 2021-03-24T22:06:54.516 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.516 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.516 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.516 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.290143+0000", 2021-03-24T22:06:54.517 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.6600000000000001e-06 2021-03-24T22:06:54.517 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.517 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.517 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:06:54.518 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.290151+0000", 2021-03-24T22:06:54.518 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.04322256 2021-03-24T22:06:54.518 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.518 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.519 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.519 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.333374+0000", 2021-03-24T22:06:54.519 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000160939 2021-03-24T22:06:54.520 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.520 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.520 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:06:54.520 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.333535+0000", 2021-03-24T22:06:54.520 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00051926299999999995 2021-03-24T22:06:54.522 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.522 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.522 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,7", 2021-03-24T22:06:54.523 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.334054+0000", 2021-03-24T22:06:54.523 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002355233 2021-03-24T22:06:54.523 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.523 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.523 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:06:54.524 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.336409+0000", 2021-03-24T22:06:54.524 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00036032499999999999 2021-03-24T22:06:54.524 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.524 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.525 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:06:54.525 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.336769+0000", 2021-03-24T22:06:54.525 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00030155200000000002 2021-03-24T22:06:54.525 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.525 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.526 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:06:54.526 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.337071+0000", 2021-03-24T22:06:54.526 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.5614000000000002e-05 2021-03-24T22:06:54.526 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.526 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.527 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:06:54.527 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.337107+0000", 2021-03-24T22:06:54.527 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000100364 2021-03-24T22:06:54.528 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.528 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.528 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:06:54.528 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.337207+0000", 2021-03-24T22:06:54.528 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0507179900000001 2021-03-24T22:06:54.529 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:06:54.529 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:06:54.529 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:06:54.529 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.531 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.531 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:4766 7.c 7:31cb714a:::smithi12556470-7:head [setxattr _header (57) in=7b,truncate 3025392] snapc 17d=[17d,17c,179,178,177,176,172,16e,16d,168,166,165,160,15e,159,157,153,139,136,129,127,11b,104,fa] ondisk+write+known_if_redirected e491)", 2021-03-24T22:06:54.531 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:01:01.286615+0000", 2021-03-24T22:06:54.532 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 352.48072514299997, 2021-03-24T22:06:54.532 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0508728430000001, 2021-03-24T22:06:54.532 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:06:54.532 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:06:54.532 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:06:54.533 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:06:54.533 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:06:54.533 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4766 2021-03-24T22:06:54.533 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.534 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:06:54.534 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.534 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:06:54.534 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286615+0000", 2021-03-24T22:06:54.535 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:06:54.535 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.535 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.535 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:06:54.535 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286615+0000", 2021-03-24T22:06:54.536 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.999999 2021-03-24T22:06:54.536 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.536 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.536 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:06:54.537 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286614+0000", 2021-03-24T22:06:54.537 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6700000000000002e-06 2021-03-24T22:06:54.537 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.537 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.537 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:06:54.538 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286619+0000", 2021-03-24T22:06:54.538 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.3249999999999998e-06 2021-03-24T22:06:54.538 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.538 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.538 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:06:54.539 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286624+0000", 2021-03-24T22:06:54.540 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6290000000000001e-06 2021-03-24T22:06:54.540 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.540 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.540 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:06:54.540 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286629+0000", 2021-03-24T22:06:54.541 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0035533230000000002 2021-03-24T22:06:54.541 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.541 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.541 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.541 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.290182+0000", 2021-03-24T22:06:54.542 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.5209999999999999e-06 2021-03-24T22:06:54.542 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.542 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.542 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:06:54.543 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.290189+0000", 2021-03-24T22:06:54.543 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.044822132 2021-03-24T22:06:54.543 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.543 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.543 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.544 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.335011+0000", 2021-03-24T22:06:54.544 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012871200000000001 2021-03-24T22:06:54.544 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.544 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.544 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:06:54.545 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.335140+0000", 2021-03-24T22:06:54.545 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000225832 2021-03-24T22:06:54.545 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.545 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.546 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,7", 2021-03-24T22:06:54.546 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.335365+0000", 2021-03-24T22:06:54.546 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0010523500000000001 2021-03-24T22:06:54.546 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.546 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.547 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:06:54.547 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.336418+0000", 2021-03-24T22:06:54.547 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00084202099999999998 2021-03-24T22:06:54.547 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.548 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.548 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:06:54.548 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.337260+0000", 2021-03-24T22:06:54.548 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.0683999999999999e-05 2021-03-24T22:06:54.549 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.549 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.549 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:06:54.549 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.337300+0000", 2021-03-24T22:06:54.550 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.4521999999999999e-05 2021-03-24T22:06:54.550 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.550 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.550 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:06:54.550 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.337335+0000", 2021-03-24T22:06:54.551 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00015305900000000001 2021-03-24T22:06:54.551 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.551 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.551 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:06:54.552 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.337488+0000", 2021-03-24T22:06:54.552 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0508728430000001 2021-03-24T22:06:54.552 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:06:54.552 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:06:54.552 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:06:54.553 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.553 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.553 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:4767 7.c 7:31cb714a:::smithi12556470-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e491)", 2021-03-24T22:06:54.553 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:01:01.286635+0000", 2021-03-24T22:06:54.553 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 352.48070564900002, 2021-03-24T22:06:54.554 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.051186642, 2021-03-24T22:06:54.554 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:06:54.554 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T22:06:54.554 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:06:54.555 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:06:54.555 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:06:54.555 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4767 2021-03-24T22:06:54.555 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.555 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:06:54.556 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.556 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:06:54.556 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286635+0000", 2021-03-24T22:06:54.556 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:06:54.556 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.557 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.557 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:06:54.557 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286635+0000", 2021-03-24T22:06:54.557 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999995 2021-03-24T22:06:54.558 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.558 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.558 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:06:54.558 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286634+0000", 2021-03-24T22:06:54.558 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.633e-06 2021-03-24T22:06:54.559 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.559 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.559 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:06:54.559 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286637+0000", 2021-03-24T22:06:54.559 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.7699999999999998e-06 2021-03-24T22:06:54.560 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.560 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.560 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:06:54.560 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286643+0000", 2021-03-24T22:06:54.560 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6349999999999997e-06 2021-03-24T22:06:54.561 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.561 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.562 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:06:54.562 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286647+0000", 2021-03-24T22:06:54.562 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0037662949999999998 2021-03-24T22:06:54.563 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.563 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.563 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.563 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.290414+0000", 2021-03-24T22:06:54.564 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.4619999999999996e-06 2021-03-24T22:06:54.564 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.564 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.564 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:06:54.564 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.290422+0000", 2021-03-24T22:06:54.565 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0454484660000001 2021-03-24T22:06:54.565 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.565 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.565 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.565 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.335870+0000", 2021-03-24T22:06:54.566 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010779400000000001 2021-03-24T22:06:54.566 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.566 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.566 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-24T22:06:54.567 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.335978+0000", 2021-03-24T22:06:54.567 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001556396 2021-03-24T22:06:54.567 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.567 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.567 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.568 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.337535+0000", 2021-03-24T22:06:54.568 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010119999999999999 2021-03-24T22:06:54.568 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.568 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.569 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:06:54.569 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.337636+0000", 2021-03-24T22:06:54.569 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00018547400000000001 2021-03-24T22:06:54.569 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.569 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.570 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:06:54.570 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.337821+0000", 2021-03-24T22:06:54.570 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.051186642 2021-03-24T22:06:54.570 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:06:54.570 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:06:54.571 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:06:54.571 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.571 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.571 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:4768 7.21 7:86662589:::smithi12556470-19:head [stat,delete] snapc 17d=[17d,17c,179,178,177,176,172,16e,16d,168,166,165,160,15e,159,157,153,139,136,129,127,11b,104,fa] ondisk+write+known_if_redirected e491)", 2021-03-24T22:06:54.572 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:01:01.286651+0000", 2021-03-24T22:06:54.572 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 352.48068921499998, 2021-03-24T22:06:54.572 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0516308409999999, 2021-03-24T22:06:54.572 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:06:54.572 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:06:54.573 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:06:54.573 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:06:54.573 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:06:54.573 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4768 2021-03-24T22:06:54.574 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.574 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:06:54.574 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.574 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:06:54.574 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286651+0000", 2021-03-24T22:06:54.575 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:06:54.575 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.575 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.575 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:06:54.575 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286651+0000", 2021-03-24T22:06:54.576 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999995 2021-03-24T22:06:54.576 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.576 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.576 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:06:54.577 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286651+0000", 2021-03-24T22:06:54.577 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.4959999999999999e-06 2021-03-24T22:06:54.577 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.577 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.577 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:06:54.578 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286653+0000", 2021-03-24T22:06:54.578 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.7709999999999997e-06 2021-03-24T22:06:54.578 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.578 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.578 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:06:54.579 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286658+0000", 2021-03-24T22:06:54.579 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.66e-06 2021-03-24T22:06:54.579 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.579 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.580 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:06:54.580 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.286664+0000", 2021-03-24T22:06:54.580 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0010190959999999999 2021-03-24T22:06:54.581 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.581 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.581 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.581 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.287683+0000", 2021-03-24T22:06:54.581 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.145e-05 2021-03-24T22:06:54.582 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.582 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.582 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:06:54.582 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:01.287694+0000", 2021-03-24T22:06:54.582 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.048107345 2021-03-24T22:06:54.583 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.583 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.583 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.583 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.335801+0000", 2021-03-24T22:06:54.584 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00025443900000000002 2021-03-24T22:06:54.584 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.584 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.584 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:06:54.584 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.336056+0000", 2021-03-24T22:06:54.585 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00044708600000000002 2021-03-24T22:06:54.585 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.585 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.585 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 5,7", 2021-03-24T22:06:54.585 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.336503+0000", 2021-03-24T22:06:54.586 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0013468359999999999 2021-03-24T22:06:54.586 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.586 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.586 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:06:54.587 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.337850+0000", 2021-03-24T22:06:54.587 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.1340999999999993e-05 2021-03-24T22:06:54.587 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.587 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.587 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:06:54.588 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.337921+0000", 2021-03-24T22:06:54.588 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.0593999999999999e-05 2021-03-24T22:06:54.588 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.588 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.588 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-24T22:06:54.589 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.337952+0000", 2021-03-24T22:06:54.589 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00011551 2021-03-24T22:06:54.589 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.589 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.590 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-24T22:06:54.590 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.338067+0000", 2021-03-24T22:06:54.590 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00021471399999999999 2021-03-24T22:06:54.590 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.590 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.591 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:06:54.591 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:01:02.338282+0000", 2021-03-24T22:06:54.591 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0516308409999999 2021-03-24T22:06:54.591 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:06:54.591 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:06:54.592 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:06:54.592 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.592 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.592 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:6371 7.10 7:0ece3580:::smithi12556470-35:head [read 0~2487634,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e655)", 2021-03-24T22:06:54.593 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:03:45.401411+0000", 2021-03-24T22:06:54.593 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 188.36592916000001, 2021-03-24T22:06:54.593 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.056283334, 2021-03-24T22:06:54.593 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:06:54.593 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T22:06:54.594 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:06:54.594 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:06:54.594 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:06:54.594 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 6371 2021-03-24T22:06:54.594 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.595 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:06:54.595 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.595 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:06:54.595 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:45.401411+0000", 2021-03-24T22:06:54.596 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:06:54.596 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.596 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.596 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:06:54.596 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:45.401411+0000", 2021-03-24T22:06:54.597 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999971 2021-03-24T22:06:54.597 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.597 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.597 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:06:54.597 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:45.401409+0000", 2021-03-24T22:06:54.598 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.7960000000000007e-06 2021-03-24T22:06:54.598 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.598 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.598 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:06:54.599 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:45.401418+0000", 2021-03-24T22:06:54.599 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1674999999999999e-05 2021-03-24T22:06:54.599 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.599 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.599 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:06:54.600 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:45.401430+0000", 2021-03-24T22:06:54.600 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.7100000000000002e-06 2021-03-24T22:06:54.600 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.600 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.601 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:06:54.601 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:45.401440+0000", 2021-03-24T22:06:54.601 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0015687570000000001 2021-03-24T22:06:54.601 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.601 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.602 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.602 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:45.403008+0000", 2021-03-24T22:06:54.602 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0859000000000001e-05 2021-03-24T22:06:54.602 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.602 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.603 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:06:54.603 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:45.403019+0000", 2021-03-24T22:06:54.603 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0516452110000001 2021-03-24T22:06:54.603 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.604 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.604 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.604 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:46.454665+0000", 2021-03-24T22:06:54.604 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00014620900000000001 2021-03-24T22:06:54.604 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.605 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.605 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:06:54.605 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:46.454811+0000", 2021-03-24T22:06:54.605 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0028837509999999999 2021-03-24T22:06:54.605 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.606 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.606 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:06:54.606 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:46.457694+0000", 2021-03-24T22:06:54.606 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.056283334 2021-03-24T22:06:54.606 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:06:54.607 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:06:54.607 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:06:54.607 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.607 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.608 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:6372 7.10 7:0ece3580:::smithi12556470-35:head [sparse-read 0~2065878] snapc 0=[] ondisk+read+known_if_redirected e655)", 2021-03-24T22:06:54.608 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:03:45.401450+0000", 2021-03-24T22:06:54.608 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 188.36589037600001, 2021-03-24T22:06:54.608 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0567293149999999, 2021-03-24T22:06:54.608 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:06:54.609 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T22:06:54.609 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:06:54.609 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:06:54.609 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:06:54.609 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 6372 2021-03-24T22:06:54.610 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.610 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:06:54.610 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.610 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:06:54.611 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:45.401450+0000", 2021-03-24T22:06:54.611 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:06:54.611 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.611 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.611 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:06:54.612 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:45.401450+0000", 2021-03-24T22:06:54.612 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999995 2021-03-24T22:06:54.612 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.612 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.612 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:06:54.613 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:45.401449+0000", 2021-03-24T22:06:54.613 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.281e-06 2021-03-24T22:06:54.613 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.613 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.613 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:06:54.614 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:45.401453+0000", 2021-03-24T22:06:54.614 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.3589999999999999e-06 2021-03-24T22:06:54.614 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.614 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.615 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:06:54.615 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:45.401458+0000", 2021-03-24T22:06:54.615 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.4800000000000003e-06 2021-03-24T22:06:54.615 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.615 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.616 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:06:54.616 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:45.401462+0000", 2021-03-24T22:06:54.616 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0027488669999999999 2021-03-24T22:06:54.616 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.616 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.617 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.617 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:45.404211+0000", 2021-03-24T22:06:54.617 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0886e-05 2021-03-24T22:06:54.617 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.617 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.618 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:06:54.618 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:45.404222+0000", 2021-03-24T22:06:54.618 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.053514386 2021-03-24T22:06:54.618 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.619 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.619 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.619 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:46.457737+0000", 2021-03-24T22:06:54.619 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00014444600000000001 2021-03-24T22:06:54.619 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.620 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.620 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:06:54.620 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:46.457881+0000", 2021-03-24T22:06:54.620 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00029818999999999999 2021-03-24T22:06:54.621 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.621 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.621 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:06:54.621 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:46.458179+0000", 2021-03-24T22:06:54.621 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0567293149999999 2021-03-24T22:06:54.622 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:06:54.622 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:06:54.622 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:06:54.622 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.622 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.623 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14760.0:6373 7.10 7:0ece3580:::smithi12556470-35:head [read 0~2487634,checksum] snapc 0=[] ondisk+read+known_if_redirected e655)", 2021-03-24T22:06:54.623 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-24T22:03:45.401467+0000", 2021-03-24T22:06:54.623 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 188.36587353799999, 2021-03-24T22:06:54.623 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.057669988, 2021-03-24T22:06:54.624 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-24T22:06:54.624 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-24T22:06:54.624 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-24T22:06:54.624 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14760", 2021-03-24T22:06:54.624 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:06:54.625 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 6373 2021-03-24T22:06:54.625 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.625 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-24T22:06:54.625 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.625 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-24T22:06:54.626 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:45.401467+0000", 2021-03-24T22:06:54.626 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-24T22:06:54.626 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.626 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.627 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-24T22:06:54.627 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:45.401467+0000", 2021-03-24T22:06:54.627 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4294967295.9999995 2021-03-24T22:06:54.627 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.627 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.628 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-24T22:06:54.628 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:45.401466+0000", 2021-03-24T22:06:54.628 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.6430000000000001e-06 2021-03-24T22:06:54.628 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.628 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.629 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-24T22:06:54.629 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:45.401470+0000", 2021-03-24T22:06:54.629 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2900000000000002e-06 2021-03-24T22:06:54.629 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.630 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.630 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-24T22:06:54.630 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:45.401475+0000", 2021-03-24T22:06:54.630 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.0239999999999999e-06 2021-03-24T22:06:54.630 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.631 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.631 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-24T22:06:54.631 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:45.401479+0000", 2021-03-24T22:06:54.631 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0027671940000000002 2021-03-24T22:06:54.631 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.632 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.632 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.632 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:45.404246+0000", 2021-03-24T22:06:54.632 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.8539999999999999e-06 2021-03-24T22:06:54.633 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.633 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.633 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-24T22:06:54.633 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:45.404253+0000", 2021-03-24T22:06:54.633 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.053960521 2021-03-24T22:06:54.634 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.634 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.634 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-24T22:06:54.634 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:46.458214+0000", 2021-03-24T22:06:54.634 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00015035600000000001 2021-03-24T22:06:54.635 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.635 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.636 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-24T22:06:54.636 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:46.458364+0000", 2021-03-24T22:06:54.636 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000772571 2021-03-24T22:06:54.636 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-24T22:06:54.637 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-24T22:06:54.637 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-24T22:06:54.637 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-24T22:03:46.459137+0000", 2021-03-24T22:06:54.637 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.057669988 2021-03-24T22:06:54.637 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:06:54.638 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:06:54.638 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:06:54.638 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-24T22:06:54.638 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-24T22:06:54.639 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-24T22:06:54.639 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:54.641 INFO:tasks.rados.rados.0.smithi125.stdout:3253: done (4 left) 2021-03-24T22:06:54.641 INFO:tasks.rados.rados.0.smithi125.stdout:3254: done (3 left) 2021-03-24T22:06:54.641 INFO:tasks.rados.rados.0.smithi125.stdout:3255: done (2 left) 2021-03-24T22:06:54.642 INFO:tasks.rados.rados.0.smithi125.stdout:3256: done (1 left) 2021-03-24T22:06:54.642 INFO:tasks.rados.rados.0.smithi125.stdout:3257: done (0 left) 2021-03-24T22:06:54.642 INFO:tasks.rados.rados.0.smithi125.stdout:3258: write (excl) oid 47 current snap is 378 2021-03-24T22:06:54.642 INFO:tasks.rados.rados.0.smithi125.stdout:3258: seq_num 1106 ranges {588542=496579,1620614=676719} 2021-03-24T22:06:54.643 INFO:tasks.rados.rados.0.smithi125.stdout:3258: writing smithi12556470-47 from 588542 to 1085121 tid 1 2021-03-24T22:06:54.643 INFO:tasks.rados.rados.0.smithi125.stdout:3258: writing smithi12556470-47 from 1620614 to 2297333 tid 2 2021-03-24T22:06:54.643 INFO:tasks.rados.rados.0.smithi125.stdout:3259: read oid 31 snap -1 2021-03-24T22:06:54.643 INFO:tasks.rados.rados.0.smithi125.stdout:3259: expect (ObjNum 1101 snap 375 seq_num 1101) 2021-03-24T22:06:54.643 INFO:tasks.rados.rados.0.smithi125.stdout:3260: snap_create 2021-03-24T22:06:54.644 INFO:tasks.rados.rados.0.smithi125.stdout:3258: finishing write tid 1 to smithi12556470-47 2021-03-24T22:06:54.644 INFO:tasks.rados.rados.0.smithi125.stdout:3258: finishing write tid 2 to smithi12556470-47 2021-03-24T22:06:54.644 INFO:tasks.rados.rados.0.smithi125.stdout:3258: finishing write tid 3 to smithi12556470-47 2021-03-24T22:06:54.644 INFO:tasks.rados.rados.0.smithi125.stdout:3258: finishing write tid 4 to smithi12556470-47 2021-03-24T22:06:54.644 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 47 v 1441 (ObjNum 1106 snap 378 seq_num 1106) dirty exists 2021-03-24T22:06:54.645 INFO:tasks.rados.rados.0.smithi125.stdout:3258: left oid 47 (ObjNum 1106 snap 378 seq_num 1106) 2021-03-24T22:06:54.645 INFO:tasks.rados.rados.0.smithi125.stdout:3259: expect (ObjNum 1101 snap 375 seq_num 1101) 2021-03-24T22:06:54.712 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:54.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:54.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:06:54.817 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:54.833 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:54 smithi142 conmon[20125]: cluster 2021-03-24T22:06: 2021-03-24T22:06:54.833 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:54 smithi142 conmon[20125]: 53.014926+0000 mgr.y (mgr.14144) 699 : cluster [DBG] pgmap v1426: 168 pgs: 168 active+clean; 535 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 977 KiB/s wr, 3 op/s 2021-03-24T22:06:54.833 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:54 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:06:54.833 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:54 smithi142 conmon[20125]: T22:06:53.471101+0000 mon.a (mon.0) 2079 : cluster [DBG] osdmap e843: 8 total, 8 up, 8 in 2021-03-24T22:06:54.876 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:54.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:54.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:06:54.912 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:54 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:06:54.912 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:54 smithi125 conmon[24927]: 06:53.014926+0000 mgr.y (mgr.14144) 699 : cluster [DBG] pgmap v1426: 168 pgs: 168 active+clean; 535 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 977 KiB/s wr, 3 op/s 2021-03-24T22:06:54.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:54 smithi125 conmon[24927]: cluster 2021-03-24T22:06:53.471101 2021-03-24T22:06:54.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:54 smithi125 conmon[24927]: +0000 mon.a (mon.0) 2079 : cluster [DBG] osdmap e843: 8 total, 8 up, 8 in 2021-03-24T22:06:54.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:54 smithi125 conmon[19936]: cluster 2021-03-24T22:06:53. 2021-03-24T22:06:54.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:54 smithi125 conmon[19936]: 014926+0000 mgr.y (mgr.14144) 699 : cluster [DBG] pgmap v1426: 168 pgs: 168 active+clean; 535 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 977 KiB/s wr, 3 op/s 2021-03-24T22:06:54.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:54 smithi125 conmon[19936]: cluster 2021-03-24T 2021-03-24T22:06:54.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:54 smithi125 conmon[19936]: 22:06:53.471101+0000 mon.a (mon.0) 2079 : cluster [DBG] 2021-03-24T22:06:54.915 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:54 smithi125 conmon[19936]: osdmap e843: 8 total, 8 up, 8 in 2021-03-24T22:06:54.982 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:55.042 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:55.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:55.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:06:55.148 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:55.206 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:55.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:55.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:06:55.313 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:55.363 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:55.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:55.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:06:55.455 INFO:tasks.rados.rados.0.smithi125.stdout:3258: done (2 left) 2021-03-24T22:06:55.456 INFO:tasks.rados.rados.0.smithi125.stdout:3259: done (1 left) 2021-03-24T22:06:55.456 INFO:tasks.rados.rados.0.smithi125.stdout:3260: done (0 left) 2021-03-24T22:06:55.456 INFO:tasks.rados.rados.0.smithi125.stdout:3261: read oid 44 snap -1 2021-03-24T22:06:55.456 INFO:tasks.rados.rados.0.smithi125.stdout:3261: expect (ObjNum 916 snap 304 seq_num 916) 2021-03-24T22:06:55.457 INFO:tasks.rados.rados.0.smithi125.stdout:3262: delete oid 23 current snap is 379 2021-03-24T22:06:55.462 INFO:tasks.rados.rados.0.smithi125.stdout:3262: done (1 left) 2021-03-24T22:06:55.462 INFO:tasks.rados.rados.0.smithi125.stdout:3263: snap_create 2021-03-24T22:06:55.470 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:55.474 INFO:tasks.rados.rados.0.smithi125.stdout:3261: expect (ObjNum 916 snap 304 seq_num 916) 2021-03-24T22:06:55.487 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":1424,"stamp":"2021-03-24T22:06:51.442704+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":560950468,"num_objects":555,"num_object_clones":188,"num_object_copies":1665,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":555,"num_whiteouts":9,"num_read":5007,"num_read_kb":3323210,"num_write":3134,"num_write_kb":893256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3404,"num_bytes_recovered":1222565970,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21457,"ondisk_log_size":21457,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":505,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1626168,"kb_used_data":1273016,"kb_used_omap":164,"kb_used_meta":352923,"kb_avail":748335048,"statfs":{"total":767960285184,"available":766295089152,"internally_reserved":0,"allocated":1303568384,"data_stored":1716328093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":168054,"internal_metadata":361394058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5227256,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":36,"num_read_kb":33498,"num_write":16,"num_write_kb":6162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.992817"},"pg_stats":[{"pgid":"3.29","version":"0'0","reported_seq":"493","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.456641+0000","last_change":"2021-03-24T22:06:44.075464+0000","last_active":"2021-03-24T22:06:47.456641+0000","last_peered":"2021-03-24T22:06:47.456641+0000","last_clean":"2021-03-24T22:06:47.456641+0000","last_became_active":"2021-03-24T22:03:45.427894+0000","last_became_peered":"2021-03-24T22:03:45.427894+0000","last_unstale":"2021-03-24T22:06:47.456641+0000","last_undegraded":"2021-03-24T22:06:47.456641+0000","last_fullsized":"2021-03-24T22:06:47.456641+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:44.075308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:06:44.075308+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":"357","reported_epoch":"835","state":"active+clean","last_fresh":"2021-03-24T22:06:45.443468+0000","last_change":"2021-03-24T22:05:54.394091+0000","last_active":"2021-03-24T22:06:45.443468+0000","last_peered":"2021-03-24T22:06:45.443468+0000","last_clean":"2021-03-24T22:06:45.443468+0000","last_became_active":"2021-03-24T22:05:04.450159+0000","last_became_peered":"2021-03-24T22:05:04.450159+0000","last_unstale":"2021-03-24T22:06:45.443468+0000","last_undegraded":"2021-03-24T22:06:45.443468+0000","last_fullsized":"2021-03-24T22:06:45.443468+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:54.393923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:05:54.393923+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"89'2","reported_seq":"514","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.465475+0000","last_change":"2021-03-24T22:06:46.478893+0000","last_active":"2021-03-24T22:06:47.465475+0000","last_peered":"2021-03-24T22:06:47.465475+0000","last_clean":"2021-03-24T22:06:47.465475+0000","last_became_active":"2021-03-24T21:59:14.374418+0000","last_became_peered":"2021-03-24T21:59:14.374418+0000","last_unstale":"2021-03-24T22:06:47.465475+0000","last_undegraded":"2021-03-24T22:06:47.465475+0000","last_fullsized":"2021-03-24T22:06:47.465475+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":6,"last_scrub":"89'2","last_scrub_stamp":"2021-03-24T22:06:46.478724+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2021-03-24T21:57:58.491830+0000","last_clean_scrub_stamp":"2021-03-24T22:06:46.478724+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":"850","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.453420+0000","last_change":"2021-03-24T22:05:04.443648+0000","last_active":"2021-03-24T22:06:47.453420+0000","last_peered":"2021-03-24T22:06:47.453420+0000","last_clean":"2021-03-24T22:06:47.453420+0000","last_became_active":"2021-03-24T22:05:04.442258+0000","last_became_peered":"2021-03-24T22:05:04.442258+0000","last_unstale":"2021-03-24T22:06:47.453420+0000","last_undegraded":"2021-03-24T22:06:47.453420+0000","last_fullsized":"2021-03-24T22:06:47.453420+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:43.239983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:03:43.239983+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":"291","reported_epoch":"835","state":"active+clean","last_fresh":"2021-03-24T22:06:45.536227+0000","last_change":"2021-03-24T22:05:53.982341+0000","last_active":"2021-03-24T22:06:45.536227+0000","last_peered":"2021-03-24T22:06:45.536227+0000","last_clean":"2021-03-24T22:06:45.536227+0000","last_became_active":"2021-03-24T22:05:04.448985+0000","last_became_peered":"2021-03-24T22:05:04.448985+0000","last_unstale":"2021-03-24T22:06:45.536227+0000","last_undegraded":"2021-03-24T22:06:45.536227+0000","last_fullsized":"2021-03-24T22:06:45.536227+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:53.982228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:05:53.982228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"61'1","reported_seq":"334","reported_epoch":"839","state":"active+clean","last_fresh":"2021-03-24T22:06:49.730628+0000","last_change":"2021-03-24T22:06:04.143008+0000","last_active":"2021-03-24T22:06:49.730628+0000","last_peered":"2021-03-24T22:06:49.730628+0000","last_clean":"2021-03-24T22:06:49.730628+0000","last_became_active":"2021-03-24T22:05:04.441568+0000","last_became_peered":"2021-03-24T22:05:04.441568+0000","last_unstale":"2021-03-24T22:06:49.730628+0000","last_undegraded":"2021-03-24T22:06:49.730628+0000","last_fullsized":"2021-03-24T22:06:49.730628+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":6,"last_scrub":"61'1","last_scrub_stamp":"2021-03-24T22:06:04.142856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:06:04.142856+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":"817","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.455928+0000","last_change":"2021-03-24T22:02:57.502941+0000","last_active":"2021-03-24T22:06:47.455928+0000","last_peered":"2021-03-24T22:06:47.455928+0000","last_clean":"2021-03-24T22:06:47.455928+0000","last_became_active":"2021-03-24T21:59:14.370506+0000","last_became_peered":"2021-03-24T21:59:14.370506+0000","last_unstale":"2021-03-24T22:06:47.455928+0000","last_undegraded":"2021-03-24T22:06:47.455928+0000","last_fullsized":"2021-03-24T22:06:47.455928+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:57.502803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:02:57.502803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.23","version":"838'522","reported_seq":"3015","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-24T22:06:50.703218+0000","last_change":"2021-03-24T22:06:49.735401+0000","last_active":"2021-03-24T22:06:50.703218+0000","last_peered":"2021-03-24T22:06:50.703218+0000","last_clean":"2021-03-24T22:06:50.703218+0000","last_became_active":"2021-03-24T22:03:49.378704+0000","last_became_peered":"2021-03-24T22:03:49.378704+0000","last_unstale":"2021-03-24T22:06:50.703218+0000","last_undegraded":"2021-03-24T22:06:50.703218+0000","last_fullsized":"2021-03-24T22:06:50.703218+0000","mapping_epoch":658,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":659,"parent":"0.0","parent_split_bits":6,"last_scrub":"731'441","last_scrub_stamp":"2021-03-24T22:05:01.661041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:05:01.661041+0000","log_size":522,"ondisk_log_size":522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15434390,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":10122,"num_write":12,"num_write_kb":5736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"27b","length":"1"},{"start":"2a7","length":"1"},{"start":"2cb","length":"1"}]},{"pgid":"3.26","version":"0'0","reported_seq":"324","reported_epoch":"839","state":"active+clean","last_fresh":"2021-03-24T22:06:49.731763+0000","last_change":"2021-03-24T22:04:21.723861+0000","last_active":"2021-03-24T22:06:49.731763+0000","last_peered":"2021-03-24T22:06:49.731763+0000","last_clean":"2021-03-24T22:06:49.731763+0000","last_became_active":"2021-03-24T22:01:57.311128+0000","last_became_peered":"2021-03-24T22:01:57.311128+0000","last_unstale":"2021-03-24T22:06:49.731763+0000","last_undegraded":"2021-03-24T22:06:49.731763+0000","last_fullsized":"2021-03-24T22:06:49.731763+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:04:21.723761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:04:21.723761+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"838'397","reported_seq":"2125","reported_epoch":"838","state":"active+clean","last_fresh":"2021-03-24T22:06:48.486049+0000","last_change":"2021-03-24T22:06:48.486346+0000","last_active":"2021-03-24T22:06:48.486049+0000","last_peered":"2021-03-24T22:06:48.486049+0000","last_clean":"2021-03-24T22:06:48.486049+0000","last_became_active":"2021-03-24T22:03:51.364119+0000","last_became_peered":"2021-03-24T22:03:51.364119+0000","last_unstale":"2021-03-24T22:06:48.486049+0000","last_undegraded":"2021-03-24T22:06:48.486049+0000","last_fullsized":"2021-03-24T22:06:48.486049+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":6,"last_scrub":"729'308","last_scrub_stamp":"2021-03-24T22:05:00.252320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:05:00.252320+0000","log_size":397,"ondisk_log_size":397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13388469,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":4,"num_read_kb":1,"num_write":9,"num_write_kb":1363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"255","length":"1"},{"start":"27b","length":"1"},{"start":"297","length":"1"},{"start":"2a7","length":"1"},{"start":"2ab","length":"1"}]},{"pgid":"4.c","version":"544'37","reported_seq":"566","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.456420+0000","last_change":"2021-03-24T22:03:03.453859+0000","last_active":"2021-03-24T22:06:47.456420+0000","last_peered":"2021-03-24T22:06:47.456420+0000","last_clean":"2021-03-24T22:06:47.456420+0000","last_became_active":"2021-03-24T22:01:50.301325+0000","last_became_peered":"2021-03-24T22:01:50.301325+0000","last_unstale":"2021-03-24T22:06:47.456420+0000","last_undegraded":"2021-03-24T22:06:47.456420+0000","last_fullsized":"2021-03-24T22:06:47.456420+0000","mapping_epoch":539,"log_start":"69'15","ondisk_log_start":"69'15","created":62,"last_epoch_clean":540,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'37","last_scrub_stamp":"2021-03-24T22:03:03.453713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:03:03.453713+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":15,"num_object_clones":0,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":81,"num_read_kb":61,"num_write":51,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":1266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"805","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.454918+0000","last_change":"2021-03-24T22:05:02.207625+0000","last_active":"2021-03-24T22:06:47.454918+0000","last_peered":"2021-03-24T22:06:47.454918+0000","last_clean":"2021-03-24T22:06:47.454918+0000","last_became_active":"2021-03-24T21:51:45.375830+0000","last_became_peered":"2021-03-24T21:51:45.375830+0000","last_unstale":"2021-03-24T22:06:47.454918+0000","last_undegraded":"2021-03-24T22:06:47.454918+0000","last_fullsized":"2021-03-24T22:06:47.454918+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:02.207454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:05:02.207454+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"734'14","reported_seq":"1079","reported_epoch":"835","state":"active+clean","last_fresh":"2021-03-24T22:06:45.539828+0000","last_change":"2021-03-24T22:05:47.047703+0000","last_active":"2021-03-24T22:06:45.539828+0000","last_peered":"2021-03-24T22:06:45.539828+0000","last_clean":"2021-03-24T22:06:45.539828+0000","last_became_active":"2021-03-24T22:05:04.430698+0000","last_became_peered":"2021-03-24T22:05:04.430698+0000","last_unstale":"2021-03-24T22:06:45.539828+0000","last_undegraded":"2021-03-24T22:06:45.539828+0000","last_fullsized":"2021-03-24T22:06:45.539828+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"734'14","last_scrub_stamp":"2021-03-24T22:05:47.047527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:05:47.047527+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"204","reported_epoch":"838","state":"active+clean","last_fresh":"2021-03-24T22:06:48.459418+0000","last_change":"2021-03-24T22:06:34.796244+0000","last_active":"2021-03-24T22:06:48.459418+0000","last_peered":"2021-03-24T22:06:48.459418+0000","last_clean":"2021-03-24T22:06:48.459418+0000","last_became_active":"2021-03-24T22:03:45.430245+0000","last_became_peered":"2021-03-24T22:03:45.430245+0000","last_unstale":"2021-03-24T22:06:48.459418+0000","last_undegraded":"2021-03-24T22:06:48.459418+0000","last_fullsized":"2021-03-24T22:06:48.459418+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:34.796158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:06:34.796158+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.f","version":"838'754","reported_seq":"3249","reported_epoch":"838","state":"active+clean","last_fresh":"2021-03-24T22:06:48.480872+0000","last_change":"2021-03-24T22:06:48.481481+0000","last_active":"2021-03-24T22:06:48.480872+0000","last_peered":"2021-03-24T22:06:48.480872+0000","last_clean":"2021-03-24T22:06:48.480872+0000","last_became_active":"2021-03-24T22:02:10.313297+0000","last_became_peered":"2021-03-24T22:02:10.313297+0000","last_unstale":"2021-03-24T22:06:48.480872+0000","last_undegraded":"2021-03-24T22:06:48.480872+0000","last_fullsized":"2021-03-24T22:06:48.480872+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"705'573","last_scrub_stamp":"2021-03-24T22:04:36.224078+0000","last_deep_scrub":"705'573","last_deep_scrub_stamp":"2021-03-24T22:04:36.224078+0000","last_clean_scrub_stamp":"2021-03-24T22:04:36.224078+0000","log_size":754,"ondisk_log_size":754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31946352,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":137,"num_read_kb":142457,"num_write":101,"num_write_kb":38492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":46545574,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"255","length":"1"},{"start":"27b","length":"1"},{"start":"297","length":"1"},{"start":"2a7","length":"1"},{"start":"2ab","length":"1"}]},{"pgid":"4.d","version":"544'41","reported_seq":"549","reported_epoch":"835","state":"active+clean","last_fresh":"2021-03-24T22:06:45.540224+0000","last_change":"2021-03-24T22:06:34.167443+0000","last_active":"2021-03-24T22:06:45.540224+0000","last_peered":"2021-03-24T22:06:45.540224+0000","last_clean":"2021-03-24T22:06:45.540224+0000","last_became_active":"2021-03-24T22:03:45.430156+0000","last_became_peered":"2021-03-24T22:03:45.430156+0000","last_unstale":"2021-03-24T22:06:45.540224+0000","last_undegraded":"2021-03-24T22:06:45.540224+0000","last_fullsized":"2021-03-24T22:06:45.540224+0000","mapping_epoch":654,"log_start":"69'21","ondisk_log_start":"69'21","created":62,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'41","last_scrub_stamp":"2021-03-24T22:06:34.167247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:06:34.167247+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1578,"num_objects":21,"num_object_clones":0,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":106,"num_read_kb":71,"num_write":60,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"338","reported_epoch":"839","state":"active+clean","last_fresh":"2021-03-24T22:06:49.731174+0000","last_change":"2021-03-24T22:05:02.665356+0000","last_active":"2021-03-24T22:06:49.731174+0000","last_peered":"2021-03-24T22:06:49.731174+0000","last_clean":"2021-03-24T22:06:49.731174+0000","last_became_active":"2021-03-24T22:03:45.419665+0000","last_became_peered":"2021-03-24T22:03:45.419665+0000","last_unstale":"2021-03-24T22:06:49.731174+0000","last_undegraded":"2021-03-24T22:06:49.731174+0000","last_fullsized":"2021-03-24T22:06:49.731174+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:02.664970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:55:01.169461+0000","last_clean_scrub_stamp":"2021-03-24T22:05:02.664970+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"204","reported_epoch":"838","state":"active+clean","last_fresh":"2021-03-24T22:06:48.459148+0000","last_change":"2021-03-24T22:06:33.822549+0000","last_active":"2021-03-24T22:06:48.459148+0000","last_peered":"2021-03-24T22:06:48.459148+0000","last_clean":"2021-03-24T22:06:48.459148+0000","last_became_active":"2021-03-24T22:03:45.430779+0000","last_became_peered":"2021-03-24T22:03:45.430779+0000","last_unstale":"2021-03-24T22:06:48.459148+0000","last_undegraded":"2021-03-24T22:06:48.459148+0000","last_fullsized":"2021-03-24T22:06:48.459148+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:33.822406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:06:33.822406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"281","reported_epoch":"835","state":"active+clean","last_fresh":"2021-03-24T22:06:45.541201+0000","last_change":"2021-03-24T22:06:01.977397+0000","last_active":"2021-03-24T22:06:45.541201+0000","last_peered":"2021-03-24T22:06:45.541201+0000","last_clean":"2021-03-24T22:06:45.541201+0000","last_became_active":"2021-03-24T22:02:22.482380+0000","last_became_peered":"2021-03-24T22:02:22.482380+0000","last_unstale":"2021-03-24T22:06:45.541201+0000","last_undegraded":"2021-03-24T22:06:45.541201+0000","last_fullsized":"2021-03-24T22:06:45.541201+0000","mapping_epoch":571,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:01.977309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:06:01.977309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"831'319","reported_seq":"2645","reported_epoch":"835","state":"active+clean","last_fresh":"2021-03-24T22:06:45.540217+0000","last_change":"2021-03-24T22:06:41.495282+0000","last_active":"2021-03-24T22:06:45.540217+0000","last_peered":"2021-03-24T22:06:45.540217+0000","last_clean":"2021-03-24T22:06:45.540217+0000","last_became_active":"2021-03-24T22:05:04.448963+0000","last_became_peered":"2021-03-24T22:05:04.448963+0000","last_unstale":"2021-03-24T22:06:45.540217+0000","last_undegraded":"2021-03-24T22:06:45.540217+0000","last_fullsized":"2021-03-24T22:06:45.540217+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"773'278","last_scrub_stamp":"2021-03-24T22:05:46.070371+0000","last_deep_scrub":"546'150","last_deep_scrub_stamp":"2021-03-24T22:01:56.662245+0000","last_clean_scrub_stamp":"2021-03-24T22:05:46.070371+0000","log_size":319,"ondisk_log_size":319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6203689,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":106,"num_read_kb":93813,"num_write":56,"num_write_kb":14865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":8580883,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"255","length":"1"},{"start":"297","length":"1"},{"start":"2ab","length":"1"}]},{"pgid":"4.e","version":"544'45","reported_seq":"707","reported_epoch":"839","state":"active+clean","last_fresh":"2021-03-24T22:06:49.731390+0000","last_change":"2021-03-24T22:06:38.254933+0000","last_active":"2021-03-24T22:06:49.731390+0000","last_peered":"2021-03-24T22:06:49.731390+0000","last_clean":"2021-03-24T22:06:49.731390+0000","last_became_active":"2021-03-24T22:03:53.368885+0000","last_became_peered":"2021-03-24T22:03:53.368885+0000","last_unstale":"2021-03-24T22:06:49.731390+0000","last_undegraded":"2021-03-24T22:06:49.731390+0000","last_fullsized":"2021-03-24T22:06:49.731390+0000","mapping_epoch":662,"log_start":"89'27","ondisk_log_start":"89'27","created":62,"last_epoch_clean":663,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'45","last_scrub_stamp":"2021-03-24T22:06:38.254767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:06:38.254767+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1672,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":92,"num_read_kb":60,"num_write":50,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":5016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"822","reported_epoch":"835","state":"active+clean","last_fresh":"2021-03-24T22:06:45.536329+0000","last_change":"2021-03-24T22:05:56.072585+0000","last_active":"2021-03-24T22:06:45.536329+0000","last_peered":"2021-03-24T22:06:45.536329+0000","last_clean":"2021-03-24T22:06:45.536329+0000","last_became_active":"2021-03-24T22:01:57.302297+0000","last_became_peered":"2021-03-24T22:01:57.302297+0000","last_unstale":"2021-03-24T22:06:45.536329+0000","last_undegraded":"2021-03-24T22:06:45.536329+0000","last_fullsized":"2021-03-24T22:06:45.536329+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:56.072443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T22:05:56.072443+0000","last_clean_scrub_stamp":"2021-03-24T22:05:56.072443+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"330","reported_epoch":"839","state":"active+clean","last_fresh":"2021-03-24T22:06:49.732052+0000","last_change":"2021-03-24T22:05:14.424426+0000","last_active":"2021-03-24T22:06:49.732052+0000","last_peered":"2021-03-24T22:06:49.732052+0000","last_clean":"2021-03-24T22:06:49.732052+0000","last_became_active":"2021-03-24T22:05:04.428211+0000","last_became_peered":"2021-03-24T22:05:04.428211+0000","last_unstale":"2021-03-24T22:06:49.732052+0000","last_undegraded":"2021-03-24T22:06:49.732052+0000","last_fullsized":"2021-03-24T22:06:49.732052+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:14.424333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:05:14.424333+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"871","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.456084+0000","last_change":"2021-03-24T22:06:03.906295+0000","last_active":"2021-03-24T22:06:47.456084+0000","last_peered":"2021-03-24T22:06:47.456084+0000","last_clean":"2021-03-24T22:06:47.456084+0000","last_became_active":"2021-03-24T22:05:04.447129+0000","last_became_peered":"2021-03-24T22:05:04.447129+0000","last_unstale":"2021-03-24T22:06:47.456084+0000","last_undegraded":"2021-03-24T22:06:47.456084+0000","last_fullsized":"2021-03-24T22:06:47.456084+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:03.906118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:06:03.906118+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"839'867","reported_seq":"3439","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-24T22:06:50.703923+0000","last_change":"2021-03-24T22:06:49.740234+0000","last_active":"2021-03-24T22:06:50.703923+0000","last_peered":"2021-03-24T22:06:50.703923+0000","last_clean":"2021-03-24T22:06:50.703923+0000","last_became_active":"2021-03-24T22:03:53.379184+0000","last_became_peered":"2021-03-24T22:03:53.379184+0000","last_unstale":"2021-03-24T22:06:50.703923+0000","last_undegraded":"2021-03-24T22:06:50.703923+0000","last_fullsized":"2021-03-24T22:06:50.703923+0000","mapping_epoch":662,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":663,"parent":"0.0","parent_split_bits":0,"last_scrub":"787'795","last_scrub_stamp":"2021-03-24T22:05:59.132881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:05:59.132881+0000","log_size":867,"ondisk_log_size":867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20750335,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":141,"num_read_kb":179025,"num_write":61,"num_write_kb":29333,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":179,"num_bytes_recovered":70071133,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"27b","length":"1"},{"start":"2a7","length":"1"},{"start":"2cb","length":"1"}]},{"pgid":"4.f","version":"544'46","reported_seq":"1033","reported_epoch":"838","state":"active+clean","last_fresh":"2021-03-24T22:06:48.457652+0000","last_change":"2021-03-24T22:06:35.800840+0000","last_active":"2021-03-24T22:06:48.457652+0000","last_peered":"2021-03-24T22:06:48.457652+0000","last_clean":"2021-03-24T22:06:48.457652+0000","last_became_active":"2021-03-24T22:04:18.462489+0000","last_became_peered":"2021-03-24T22:04:18.462489+0000","last_unstale":"2021-03-24T22:06:48.457652+0000","last_undegraded":"2021-03-24T22:06:48.457652+0000","last_fullsized":"2021-03-24T22:06:48.457652+0000","mapping_epoch":687,"log_start":"274'26","ondisk_log_start":"274'26","created":62,"last_epoch_clean":688,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'46","last_scrub_stamp":"2021-03-24T22:06:35.800671+0000","last_deep_scrub":"274'26","last_deep_scrub_stamp":"2021-03-24T21:57:24.499090+0000","last_clean_scrub_stamp":"2021-03-24T22:06:35.800671+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1472,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":97,"num_read_kb":65,"num_write":56,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":8410,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":"887","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.455392+0000","last_change":"2021-03-24T22:06:47.083675+0000","last_active":"2021-03-24T22:06:47.455392+0000","last_peered":"2021-03-24T22:06:47.455392+0000","last_clean":"2021-03-24T22:06:47.455392+0000","last_became_active":"2021-03-24T22:03:45.427907+0000","last_became_peered":"2021-03-24T22:03:45.427907+0000","last_unstale":"2021-03-24T22:06:47.455392+0000","last_undegraded":"2021-03-24T22:06:47.455392+0000","last_fullsized":"2021-03-24T22:06:47.455392+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:47.083576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:06:47.083576+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"734'7","reported_seq":"473","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.464351+0000","last_change":"2021-03-24T22:05:09.312441+0000","last_active":"2021-03-24T22:06:47.464351+0000","last_peered":"2021-03-24T22:06:47.464351+0000","last_clean":"2021-03-24T22:06:47.464351+0000","last_became_active":"2021-03-24T22:05:04.447711+0000","last_became_peered":"2021-03-24T22:05:04.447711+0000","last_unstale":"2021-03-24T22:06:47.464351+0000","last_undegraded":"2021-03-24T22:06:47.464351+0000","last_fullsized":"2021-03-24T22:06:47.464351+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"734'7","last_scrub_stamp":"2021-03-24T22:05:09.312269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:05:09.312269+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"666","reported_epoch":"835","state":"active+clean","last_fresh":"2021-03-24T22:06:45.446091+0000","last_change":"2021-03-24T22:05:47.327420+0000","last_active":"2021-03-24T22:06:45.446091+0000","last_peered":"2021-03-24T22:06:45.446091+0000","last_clean":"2021-03-24T22:06:45.446091+0000","last_became_active":"2021-03-24T22:05:04.451010+0000","last_became_peered":"2021-03-24T22:05:04.451010+0000","last_unstale":"2021-03-24T22:06:45.446091+0000","last_undegraded":"2021-03-24T22:06:45.446091+0000","last_fullsized":"2021-03-24T22:06:45.446091+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:47.327213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:05:47.327213+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"838'1368","reported_seq":"4128","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-24T22:06:50.702944+0000","last_change":"2021-03-24T22:06:49.734467+0000","last_active":"2021-03-24T22:06:50.702944+0000","last_peered":"2021-03-24T22:06:50.702944+0000","last_clean":"2021-03-24T22:06:50.702944+0000","last_became_active":"2021-03-24T22:02:06.309737+0000","last_became_peered":"2021-03-24T22:02:06.309737+0000","last_unstale":"2021-03-24T22:06:50.702944+0000","last_undegraded":"2021-03-24T22:06:50.702944+0000","last_fullsized":"2021-03-24T22:06:50.702944+0000","mapping_epoch":555,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":556,"parent":"0.0","parent_split_bits":0,"last_scrub":"830'1351","last_scrub_stamp":"2021-03-24T22:06:41.250471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:06:41.250471+0000","log_size":1368,"ondisk_log_size":1368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39218459,"num_objects":16,"num_object_clones":12,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":263,"num_read_kb":279912,"num_write":162,"num_write_kb":67731,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":296,"num_bytes_recovered":189413266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"27b","length":"1"},{"start":"2a7","length":"1"},{"start":"2cb","length":"1"}]},{"pgid":"5.1","version":"0'0","reported_seq":"774","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.453874+0000","last_change":"2021-03-24T22:03:00.472146+0000","last_active":"2021-03-24T22:06:47.453874+0000","last_peered":"2021-03-24T22:06:47.453874+0000","last_clean":"2021-03-24T22:06:47.453874+0000","last_became_active":"2021-03-24T22:01:57.305430+0000","last_became_peered":"2021-03-24T22:01:57.305430+0000","last_unstale":"2021-03-24T22:06:47.453874+0000","last_undegraded":"2021-03-24T22:06:47.453874+0000","last_fullsized":"2021-03-24T22:06:47.453874+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:00.471949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:25.220436+0000","last_clean_scrub_stamp":"2021-03-24T22:03:00.471949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"544'34","reported_seq":"1007","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.453110+0000","last_change":"2021-03-24T22:04:30.205418+0000","last_active":"2021-03-24T22:06:47.453110+0000","last_peered":"2021-03-24T22:06:47.453110+0000","last_clean":"2021-03-24T22:06:47.453110+0000","last_became_active":"2021-03-24T22:01:57.309916+0000","last_became_peered":"2021-03-24T22:01:57.309916+0000","last_unstale":"2021-03-24T22:06:47.453110+0000","last_undegraded":"2021-03-24T22:06:47.453110+0000","last_fullsized":"2021-03-24T22:06:47.453110+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'34","last_scrub_stamp":"2021-03-24T22:04:30.205247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:04:30.205247+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":64,"num_read_kb":44,"num_write":36,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":3384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"856","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.455507+0000","last_change":"2021-03-24T22:06:06.879783+0000","last_active":"2021-03-24T22:06:47.455507+0000","last_peered":"2021-03-24T22:06:47.455507+0000","last_clean":"2021-03-24T22:06:47.455507+0000","last_became_active":"2021-03-24T22:01:57.314345+0000","last_became_peered":"2021-03-24T22:01:57.314345+0000","last_unstale":"2021-03-24T22:06:47.455507+0000","last_undegraded":"2021-03-24T22:06:47.455507+0000","last_fullsized":"2021-03-24T22:06:47.455507+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:06.879633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:06:06.879633+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"373","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.453918+0000","last_change":"2021-03-24T22:03:45.222962+0000","last_active":"2021-03-24T22:06:47.453918+0000","last_peered":"2021-03-24T22:06:47.453918+0000","last_clean":"2021-03-24T22:06:47.453918+0000","last_became_active":"2021-03-24T22:00:49.305612+0000","last_became_peered":"2021-03-24T22:00:49.305612+0000","last_unstale":"2021-03-24T22:06:47.453918+0000","last_undegraded":"2021-03-24T22:06:47.453918+0000","last_fullsized":"2021-03-24T22:06:47.453918+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:45.222743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:03:45.222743+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"831'864","reported_seq":"3142","reported_epoch":"838","state":"active+clean","last_fresh":"2021-03-24T22:06:48.493371+0000","last_change":"2021-03-24T22:06:48.493698+0000","last_active":"2021-03-24T22:06:48.493371+0000","last_peered":"2021-03-24T22:06:48.493371+0000","last_clean":"2021-03-24T22:06:48.493371+0000","last_became_active":"2021-03-24T21:57:26.169302+0000","last_became_peered":"2021-03-24T21:57:26.169302+0000","last_unstale":"2021-03-24T22:06:48.493371+0000","last_undegraded":"2021-03-24T22:06:48.493371+0000","last_fullsized":"2021-03-24T22:06:48.493371+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'770","last_scrub_stamp":"2021-03-24T22:04:57.253408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:04:57.253408+0000","log_size":864,"ondisk_log_size":864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15699091,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":82,"num_read_kb":64068,"num_write":85,"num_write_kb":35091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"255","length":"1"},{"start":"27b","length":"1"},{"start":"297","length":"1"},{"start":"2a7","length":"1"},{"start":"2ab","length":"1"}]},{"pgid":"5.0","version":"658'6","reported_seq":"1034","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.452913+0000","last_change":"2021-03-24T22:05:13.207391+0000","last_active":"2021-03-24T22:06:47.452913+0000","last_peered":"2021-03-24T22:06:47.452913+0000","last_clean":"2021-03-24T22:06:47.452913+0000","last_became_active":"2021-03-24T22:03:45.432883+0000","last_became_peered":"2021-03-24T22:03:45.432883+0000","last_unstale":"2021-03-24T22:06:47.452913+0000","last_undegraded":"2021-03-24T22:06:47.452913+0000","last_fullsized":"2021-03-24T22:06:47.452913+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"658'6","last_scrub_stamp":"2021-03-24T22:05:13.207196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:05:13.207196+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"544'34","reported_seq":"386","reported_epoch":"835","state":"active+clean","last_fresh":"2021-03-24T22:06:45.444309+0000","last_change":"2021-03-24T22:05:44.265489+0000","last_active":"2021-03-24T22:06:45.444309+0000","last_peered":"2021-03-24T22:06:45.444309+0000","last_clean":"2021-03-24T22:06:45.444309+0000","last_became_active":"2021-03-24T22:05:04.433178+0000","last_became_peered":"2021-03-24T22:05:04.433178+0000","last_unstale":"2021-03-24T22:06:45.444309+0000","last_undegraded":"2021-03-24T22:06:45.444309+0000","last_fullsized":"2021-03-24T22:06:45.444309+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'34","last_scrub_stamp":"2021-03-24T22:05:44.265284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:05:44.265284+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":58,"num_read_kb":39,"num_write":34,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":940,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"872","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.463869+0000","last_change":"2021-03-24T22:06:06.448287+0000","last_active":"2021-03-24T22:06:47.463869+0000","last_peered":"2021-03-24T22:06:47.463869+0000","last_clean":"2021-03-24T22:06:47.463869+0000","last_became_active":"2021-03-24T22:06:06.447793+0000","last_became_peered":"2021-03-24T22:06:06.447793+0000","last_unstale":"2021-03-24T22:06:47.463869+0000","last_undegraded":"2021-03-24T22:06:47.463869+0000","last_fullsized":"2021-03-24T22:06:47.463869+0000","mapping_epoch":795,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":796,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:00.436502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:06:00.436502+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"308","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.471486+0000","last_change":"2021-03-24T22:06:36.049879+0000","last_active":"2021-03-24T22:06:47.471486+0000","last_peered":"2021-03-24T22:06:47.471486+0000","last_clean":"2021-03-24T22:06:47.471486+0000","last_became_active":"2021-03-24T22:01:57.316610+0000","last_became_peered":"2021-03-24T22:01:57.316610+0000","last_unstale":"2021-03-24T22:06:47.471486+0000","last_undegraded":"2021-03-24T22:06:47.471486+0000","last_fullsized":"2021-03-24T22:06:47.471486+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:36.049669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:06:36.049669+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"272'64","reported_seq":"2368","reported_epoch":"838","state":"active+clean","last_fresh":"2021-03-24T22:06:48.473753+0000","last_change":"2021-03-24T22:06:48.475761+0000","last_active":"2021-03-24T22:06:48.473753+0000","last_peered":"2021-03-24T22:06:48.473753+0000","last_clean":"2021-03-24T22:06:48.473753+0000","last_became_active":"2021-03-24T22:03:45.428845+0000","last_became_peered":"2021-03-24T22:03:45.428845+0000","last_unstale":"2021-03-24T22:06:48.473753+0000","last_undegraded":"2021-03-24T22:06:48.473753+0000","last_fullsized":"2021-03-24T22:06:48.473753+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'64","last_scrub_stamp":"2021-03-24T22:05:03.168158+0000","last_deep_scrub":"272'64","last_deep_scrub_stamp":"2021-03-24T21:59:08.406188+0000","last_clean_scrub_stamp":"2021-03-24T22:05:03.168158+0000","log_size":64,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"255","length":"1"},{"start":"27b","length":"1"},{"start":"297","length":"1"},{"start":"2a7","length":"1"},{"start":"2ab","length":"1"}]},{"pgid":"5.3","version":"734'9","reported_seq":"1039","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.462686+0000","last_change":"2021-03-24T22:05:47.459972+0000","last_active":"2021-03-24T22:06:47.462686+0000","last_peered":"2021-03-24T22:06:47.462686+0000","last_clean":"2021-03-24T22:06:47.462686+0000","last_became_active":"2021-03-24T22:05:04.447618+0000","last_became_peered":"2021-03-24T22:05:04.447618+0000","last_unstale":"2021-03-24T22:06:47.462686+0000","last_undegraded":"2021-03-24T22:06:47.462686+0000","last_fullsized":"2021-03-24T22:06:47.462686+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"734'9","last_scrub_stamp":"2021-03-24T22:05:47.459863+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-24T21:53:26.584678+0000","last_clean_scrub_stamp":"2021-03-24T22:05:47.459863+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"544'26","reported_seq":"238","reported_epoch":"838","state":"active+clean","last_fresh":"2021-03-24T22:06:48.457883+0000","last_change":"2021-03-24T22:05:06.622482+0000","last_active":"2021-03-24T22:06:48.457883+0000","last_peered":"2021-03-24T22:06:48.457883+0000","last_clean":"2021-03-24T22:06:48.457883+0000","last_became_active":"2021-03-24T22:05:04.449351+0000","last_became_peered":"2021-03-24T22:05:04.449351+0000","last_unstale":"2021-03-24T22:06:48.457883+0000","last_undegraded":"2021-03-24T22:06:48.457883+0000","last_fullsized":"2021-03-24T22:06:48.457883+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'26","last_scrub_stamp":"2021-03-24T22:05:06.622374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:05:06.622374+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":46,"num_read_kb":29,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"629","reported_epoch":"835","state":"active+clean","last_fresh":"2021-03-24T22:06:45.446454+0000","last_change":"2021-03-24T22:06:01.442658+0000","last_active":"2021-03-24T22:06:45.446454+0000","last_peered":"2021-03-24T22:06:45.446454+0000","last_clean":"2021-03-24T22:06:45.446454+0000","last_became_active":"2021-03-24T22:05:04.449973+0000","last_became_peered":"2021-03-24T22:05:04.449973+0000","last_unstale":"2021-03-24T22:06:45.446454+0000","last_undegraded":"2021-03-24T22:06:45.446454+0000","last_fullsized":"2021-03-24T22:06:45.446454+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:01.442569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:06:01.442569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"796","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.462376+0000","last_change":"2021-03-24T22:04:37.121136+0000","last_active":"2021-03-24T22:06:47.462376+0000","last_peered":"2021-03-24T22:06:47.462376+0000","last_clean":"2021-03-24T22:06:47.462376+0000","last_became_active":"2021-03-24T21:51:51.759627+0000","last_became_peered":"2021-03-24T21:51:51.759627+0000","last_unstale":"2021-03-24T22:06:47.462376+0000","last_undegraded":"2021-03-24T22:06:47.462376+0000","last_fullsized":"2021-03-24T22:06:47.462376+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:04:37.120859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:04:37.120859+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"838'577","reported_seq":"2976","reported_epoch":"838","state":"active+clean","last_fresh":"2021-03-24T22:06:48.493035+0000","last_change":"2021-03-24T22:06:48.493297+0000","last_active":"2021-03-24T22:06:48.493035+0000","last_peered":"2021-03-24T22:06:48.493035+0000","last_clean":"2021-03-24T22:06:48.493035+0000","last_became_active":"2021-03-24T22:02:30.322922+0000","last_became_peered":"2021-03-24T22:02:30.322922+0000","last_unstale":"2021-03-24T22:06:48.493035+0000","last_undegraded":"2021-03-24T22:06:48.493035+0000","last_fullsized":"2021-03-24T22:06:48.493035+0000","mapping_epoch":579,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":580,"parent":"0.0","parent_split_bits":0,"last_scrub":"695'455","last_scrub_stamp":"2021-03-24T22:04:28.210940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:04:28.210940+0000","log_size":577,"ondisk_log_size":577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13630574,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":82,"num_read_kb":100739,"num_write":52,"num_write_kb":15980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":24969090,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"255","length":"1"},{"start":"27b","length":"1"},{"start":"297","length":"1"},{"start":"2a7","length":"1"},{"start":"2ab","length":"1"}]},{"pgid":"5.2","version":"0'0","reported_seq":"330","reported_epoch":"839","state":"active+clean","last_fresh":"2021-03-24T22:06:49.730947+0000","last_change":"2021-03-24T22:05:06.638365+0000","last_active":"2021-03-24T22:06:49.730947+0000","last_peered":"2021-03-24T22:06:49.730947+0000","last_clean":"2021-03-24T22:06:49.730947+0000","last_became_active":"2021-03-24T22:05:04.444370+0000","last_became_peered":"2021-03-24T22:05:04.444370+0000","last_unstale":"2021-03-24T22:06:49.730947+0000","last_undegraded":"2021-03-24T22:06:49.730947+0000","last_fullsized":"2021-03-24T22:06:49.730947+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:06.638226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:05:06.638226+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"544'31","reported_seq":"1016","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.462706+0000","last_change":"2021-03-24T22:05:52.398909+0000","last_active":"2021-03-24T22:06:47.462706+0000","last_peered":"2021-03-24T22:06:47.462706+0000","last_clean":"2021-03-24T22:06:47.462706+0000","last_became_active":"2021-03-24T22:05:04.450428+0000","last_became_peered":"2021-03-24T22:05:04.450428+0000","last_unstale":"2021-03-24T22:06:47.462706+0000","last_undegraded":"2021-03-24T22:06:47.462706+0000","last_fullsized":"2021-03-24T22:06:47.462706+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'31","last_scrub_stamp":"2021-03-24T22:05:52.398731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:05:52.398731+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":60,"num_read_kb":42,"num_write":34,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":2568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"214","reported_epoch":"838","state":"active+clean","last_fresh":"2021-03-24T22:06:48.456687+0000","last_change":"2021-03-24T22:06:02.509466+0000","last_active":"2021-03-24T22:06:48.456687+0000","last_peered":"2021-03-24T22:06:48.456687+0000","last_clean":"2021-03-24T22:06:48.456687+0000","last_became_active":"2021-03-24T22:05:04.453406+0000","last_became_peered":"2021-03-24T22:05:04.453406+0000","last_unstale":"2021-03-24T22:06:48.456687+0000","last_undegraded":"2021-03-24T22:06:48.456687+0000","last_fullsized":"2021-03-24T22:06:48.456687+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:02.509401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:06:02.509401+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"199","reported_epoch":"838","state":"active+clean","last_fresh":"2021-03-24T22:06:48.458649+0000","last_change":"2021-03-24T22:04:56.575839+0000","last_active":"2021-03-24T22:06:48.458649+0000","last_peered":"2021-03-24T22:06:48.458649+0000","last_clean":"2021-03-24T22:06:48.458649+0000","last_became_active":"2021-03-24T22:03:45.426809+0000","last_became_peered":"2021-03-24T22:03:45.426809+0000","last_unstale":"2021-03-24T22:06:48.458649+0000","last_undegraded":"2021-03-24T22:06:48.458649+0000","last_fullsized":"2021-03-24T22:06:48.458649+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:04:56.575723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:04:56.575723+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"175'22","reported_seq":"2155","reported_epoch":"835","state":"active+clean","last_fresh":"2021-03-24T22:06:45.444590+0000","last_change":"2021-03-24T22:06:41.478268+0000","last_active":"2021-03-24T22:06:45.444590+0000","last_peered":"2021-03-24T22:06:45.444590+0000","last_clean":"2021-03-24T22:06:45.444590+0000","last_became_active":"2021-03-24T22:05:04.443450+0000","last_became_peered":"2021-03-24T22:05:04.443450+0000","last_unstale":"2021-03-24T22:06:45.444590+0000","last_undegraded":"2021-03-24T22:06:45.444590+0000","last_fullsized":"2021-03-24T22:06:45.444590+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'22","last_scrub_stamp":"2021-03-24T22:05:53.363157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:05:53.363157+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"255","length":"1"},{"start":"297","length":"1"},{"start":"2ab","length":"1"}]},{"pgid":"5.7","version":"0'0","reported_seq":"674","reported_epoch":"835","state":"active+clean","last_fresh":"2021-03-24T22:06:45.447229+0000","last_change":"2021-03-24T22:06:41.330687+0000","last_active":"2021-03-24T22:06:45.447229+0000","last_peered":"2021-03-24T22:06:45.447229+0000","last_clean":"2021-03-24T22:06:45.447229+0000","last_became_active":"2021-03-24T22:05:04.439506+0000","last_became_peered":"2021-03-24T22:05:04.439506+0000","last_unstale":"2021-03-24T22:06:45.447229+0000","last_undegraded":"2021-03-24T22:06:45.447229+0000","last_fullsized":"2021-03-24T22:06:45.447229+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:41.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:06:41.330509+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"544'34","reported_seq":"1341","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.463771+0000","last_change":"2021-03-24T22:06:40.571163+0000","last_active":"2021-03-24T22:06:47.463771+0000","last_peered":"2021-03-24T22:06:47.463771+0000","last_clean":"2021-03-24T22:06:47.463771+0000","last_became_active":"2021-03-24T22:03:54.401857+0000","last_became_peered":"2021-03-24T22:03:54.401857+0000","last_unstale":"2021-03-24T22:06:47.463771+0000","last_undegraded":"2021-03-24T22:06:47.463771+0000","last_fullsized":"2021-03-24T22:06:47.463771+0000","mapping_epoch":663,"log_start":"69'16","ondisk_log_start":"69'16","created":62,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'34","last_scrub_stamp":"2021-03-24T22:06:40.570957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:06:40.570957+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1836,"num_objects":21,"num_object_clones":0,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":95,"num_read_kb":61,"num_write":51,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":157,"num_bytes_recovered":13482,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"837'6","reported_seq":"630","reported_epoch":"838","state":"active+clean","last_fresh":"2021-03-24T22:06:48.467493+0000","last_change":"2021-03-24T22:06:48.129118+0000","last_active":"2021-03-24T22:06:48.467493+0000","last_peered":"2021-03-24T22:06:48.467493+0000","last_clean":"2021-03-24T22:06:48.467493+0000","last_became_active":"2021-03-24T22:03:45.433742+0000","last_became_peered":"2021-03-24T22:03:45.433742+0000","last_unstale":"2021-03-24T22:06:48.467493+0000","last_undegraded":"2021-03-24T22:06:48.467493+0000","last_fullsized":"2021-03-24T22:06:48.467493+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"837'6","last_scrub_stamp":"2021-03-24T22:06:48.128942+0000","last_deep_scrub":"837'6","last_deep_scrub_stamp":"2021-03-24T22:06:48.128942+0000","last_clean_scrub_stamp":"2021-03-24T22:06:48.128942+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":363,"num_read_kb":358,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"816","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.462439+0000","last_change":"2021-03-24T22:04:39.179364+0000","last_active":"2021-03-24T22:06:47.462439+0000","last_peered":"2021-03-24T22:06:47.462439+0000","last_clean":"2021-03-24T22:06:47.462439+0000","last_became_active":"2021-03-24T21:59:12.281282+0000","last_became_peered":"2021-03-24T21:59:12.281282+0000","last_unstale":"2021-03-24T22:06:47.462439+0000","last_undegraded":"2021-03-24T22:06:47.462439+0000","last_fullsized":"2021-03-24T22:06:47.462439+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:04:39.179121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:04:39.179121+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"199","reported_epoch":"838","state":"active+clean","last_fresh":"2021-03-24T22:06:48.459241+0000","last_change":"2021-03-24T22:05:44.537883+0000","last_active":"2021-03-24T22:06:48.459241+0000","last_peered":"2021-03-24T22:06:48.459241+0000","last_clean":"2021-03-24T22:06:48.459241+0000","last_became_active":"2021-03-24T22:03:45.424562+0000","last_became_peered":"2021-03-24T22:03:45.424562+0000","last_unstale":"2021-03-24T22:06:48.459241+0000","last_undegraded":"2021-03-24T22:06:48.459241+0000","last_fullsized":"2021-03-24T22:06:48.459241+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:44.537809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:05:44.537809+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"195'69","reported_seq":"923","reported_epoch":"838","state":"active+clean","last_fresh":"2021-03-24T22:06:48.486445+0000","last_change":"2021-03-24T22:06:48.486914+0000","last_active":"2021-03-24T22:06:48.486445+0000","last_peered":"2021-03-24T22:06:48.486445+0000","last_clean":"2021-03-24T22:06:48.486445+0000","last_became_active":"2021-03-24T22:01:57.317669+0000","last_became_peered":"2021-03-24T22:01:57.317669+0000","last_unstale":"2021-03-24T22:06:48.486445+0000","last_undegraded":"2021-03-24T22:06:48.486445+0000","last_fullsized":"2021-03-24T22:06:48.486445+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'69","last_scrub_stamp":"2021-03-24T22:04:58.198525+0000","last_deep_scrub":"195'69","last_deep_scrub_stamp":"2021-03-24T21:56:49.886058+0000","last_clean_scrub_stamp":"2021-03-24T22:04:58.198525+0000","log_size":69,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"255","length":"1"},{"start":"27b","length":"1"},{"start":"297","length":"1"},{"start":"2a7","length":"1"},{"start":"2ab","length":"1"}]},{"pgid":"5.4","version":"0'0","reported_seq":"325","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.472813+0000","last_change":"2021-03-24T22:05:14.123612+0000","last_active":"2021-03-24T22:06:47.472813+0000","last_peered":"2021-03-24T22:06:47.472813+0000","last_clean":"2021-03-24T22:06:47.472813+0000","last_became_active":"2021-03-24T22:05:04.451670+0000","last_became_peered":"2021-03-24T22:05:04.451670+0000","last_unstale":"2021-03-24T22:06:47.472813+0000","last_undegraded":"2021-03-24T22:06:47.472813+0000","last_fullsized":"2021-03-24T22:06:47.472813+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:14.123439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:05:14.123439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"783'36","reported_seq":"396","reported_epoch":"839","state":"active+clean","last_fresh":"2021-03-24T22:06:49.732011+0000","last_change":"2021-03-24T22:05:54.205230+0000","last_active":"2021-03-24T22:06:49.732011+0000","last_peered":"2021-03-24T22:06:49.732011+0000","last_clean":"2021-03-24T22:06:49.732011+0000","last_became_active":"2021-03-24T22:01:57.311205+0000","last_became_peered":"2021-03-24T22:01:57.311205+0000","last_unstale":"2021-03-24T22:06:49.732011+0000","last_undegraded":"2021-03-24T22:06:49.732011+0000","last_fullsized":"2021-03-24T22:06:49.732011+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"783'36","last_scrub_stamp":"2021-03-24T22:05:54.205108+0000","last_deep_scrub":"783'36","last_deep_scrub_stamp":"2021-03-24T22:05:54.205108+0000","last_clean_scrub_stamp":"2021-03-24T22:05:54.205108+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":27,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":1398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"844","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.453609+0000","last_change":"2021-03-24T22:05:04.443282+0000","last_active":"2021-03-24T22:06:47.453609+0000","last_peered":"2021-03-24T22:06:47.453609+0000","last_clean":"2021-03-24T22:06:47.453609+0000","last_became_active":"2021-03-24T22:05:04.441717+0000","last_became_peered":"2021-03-24T22:05:04.441717+0000","last_unstale":"2021-03-24T22:06:47.453609+0000","last_undegraded":"2021-03-24T22:06:47.453609+0000","last_fullsized":"2021-03-24T22:06:47.453609+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:55:33.533840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T21:55:33.533840+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"308","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.471881+0000","last_change":"2021-03-24T22:05:59.171885+0000","last_active":"2021-03-24T22:06:47.471881+0000","last_peered":"2021-03-24T22:06:47.471881+0000","last_clean":"2021-03-24T22:06:47.471881+0000","last_became_active":"2021-03-24T22:01:57.315729+0000","last_became_peered":"2021-03-24T22:01:57.315729+0000","last_unstale":"2021-03-24T22:06:47.471881+0000","last_undegraded":"2021-03-24T22:06:47.471881+0000","last_fullsized":"2021-03-24T22:06:47.471881+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:59.171797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:49:51.529051+0000","last_clean_scrub_stamp":"2021-03-24T22:05:59.171797+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"621","reported_epoch":"835","state":"active+clean","last_fresh":"2021-03-24T22:06:45.447743+0000","last_change":"2021-03-24T22:05:49.381846+0000","last_active":"2021-03-24T22:06:45.447743+0000","last_peered":"2021-03-24T22:06:45.447743+0000","last_clean":"2021-03-24T22:06:45.447743+0000","last_became_active":"2021-03-24T22:05:04.435232+0000","last_became_peered":"2021-03-24T22:05:04.435232+0000","last_unstale":"2021-03-24T22:06:45.447743+0000","last_undegraded":"2021-03-24T22:06:45.447743+0000","last_fullsized":"2021-03-24T22:06:45.447743+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:49.381569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:05:49.381569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"838'453","reported_seq":"3055","reported_epoch":"838","state":"active+clean","last_fresh":"2021-03-24T22:06:48.485031+0000","last_change":"2021-03-24T22:06:48.485573+0000","last_active":"2021-03-24T22:06:48.485031+0000","last_peered":"2021-03-24T22:06:48.485031+0000","last_clean":"2021-03-24T22:06:48.485031+0000","last_became_active":"2021-03-24T22:02:31.331916+0000","last_became_peered":"2021-03-24T22:02:31.331916+0000","last_unstale":"2021-03-24T22:06:48.485031+0000","last_undegraded":"2021-03-24T22:06:48.485031+0000","last_fullsized":"2021-03-24T22:06:48.485031+0000","mapping_epoch":580,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":581,"parent":"0.0","parent_split_bits":0,"last_scrub":"783'396","last_scrub_stamp":"2021-03-24T22:05:56.358947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:05:56.358947+0000","log_size":453,"ondisk_log_size":453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19100161,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":14,"num_read_kb":16743,"num_write":10,"num_write_kb":6213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"255","length":"1"},{"start":"27b","length":"1"},{"start":"297","length":"1"},{"start":"2a7","length":"1"},{"start":"2ab","length":"1"}]},{"pgid":"5.6","version":"0'0","reported_seq":"296","reported_epoch":"835","state":"active+clean","last_fresh":"2021-03-24T22:06:45.541664+0000","last_change":"2021-03-24T22:05:51.038309+0000","last_active":"2021-03-24T22:06:45.541664+0000","last_peered":"2021-03-24T22:06:45.541664+0000","last_clean":"2021-03-24T22:06:45.541664+0000","last_became_active":"2021-03-24T22:05:04.423152+0000","last_became_peered":"2021-03-24T22:05:04.423152+0000","last_unstale":"2021-03-24T22:06:45.541664+0000","last_undegraded":"2021-03-24T22:06:45.541664+0000","last_fullsized":"2021-03-24T22:06:45.541664+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:51.038089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:05:51.038089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"544'32","reported_seq":"597","reported_epoch":"835","state":"active+clean","last_fresh":"2021-03-24T22:06:45.540746+0000","last_change":"2021-03-24T22:06:40.463408+0000","last_active":"2021-03-24T22:06:45.540746+0000","last_peered":"2021-03-24T22:06:45.540746+0000","last_clean":"2021-03-24T22:06:45.540746+0000","last_became_active":"2021-03-24T22:06:40.463022+0000","last_became_peered":"2021-03-24T22:06:40.463022+0000","last_unstale":"2021-03-24T22:06:45.540746+0000","last_undegraded":"2021-03-24T22:06:45.540746+0000","last_fullsized":"2021-03-24T22:06:45.540746+0000","mapping_epoch":829,"log_start":"89'22","ondisk_log_start":"89'22","created":62,"last_epoch_clean":830,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'32","last_scrub_stamp":"2021-03-24T22:05:13.446721+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2021-03-24T21:53:19.717244+0000","last_clean_scrub_stamp":"2021-03-24T22:05:13.446721+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":802,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":42,"num_write":35,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"821","reported_epoch":"835","state":"active+clean","last_fresh":"2021-03-24T22:06:45.536353+0000","last_change":"2021-03-24T22:06:37.118363+0000","last_active":"2021-03-24T22:06:45.536353+0000","last_peered":"2021-03-24T22:06:45.536353+0000","last_clean":"2021-03-24T22:06:45.536353+0000","last_became_active":"2021-03-24T21:51:40.351000+0000","last_became_peered":"2021-03-24T21:51:40.351000+0000","last_unstale":"2021-03-24T22:06:45.536353+0000","last_undegraded":"2021-03-24T22:06:45.536353+0000","last_fullsized":"2021-03-24T22:06:45.536353+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:37.118211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:39.286101+0000","last_clean_scrub_stamp":"2021-03-24T22:06:37.118211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"61'1","reported_seq":"199","reported_epoch":"838","state":"active+clean","last_fresh":"2021-03-24T22:06:48.457869+0000","last_change":"2021-03-24T22:04:34.667264+0000","last_active":"2021-03-24T22:06:48.457869+0000","last_peered":"2021-03-24T22:06:48.457869+0000","last_clean":"2021-03-24T22:06:48.457869+0000","last_became_active":"2021-03-24T22:03:45.426685+0000","last_became_peered":"2021-03-24T22:03:45.426685+0000","last_unstale":"2021-03-24T22:06:48.457869+0000","last_undegraded":"2021-03-24T22:06:48.457869+0000","last_fullsized":"2021-03-24T22:06:48.457869+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2021-03-24T22:04:34.667176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:04:34.667176+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"308","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.472218+0000","last_change":"2021-03-24T22:05:50.105714+0000","last_active":"2021-03-24T22:06:47.472218+0000","last_peered":"2021-03-24T22:06:47.472218+0000","last_clean":"2021-03-24T22:06:47.472218+0000","last_became_active":"2021-03-24T22:01:57.318275+0000","last_became_peered":"2021-03-24T22:01:57.318275+0000","last_unstale":"2021-03-24T22:06:47.472218+0000","last_undegraded":"2021-03-24T22:06:47.472218+0000","last_fullsized":"2021-03-24T22:06:47.472218+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:50.105616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:05:50.105616+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"831'493","reported_seq":"3011","reported_epoch":"838","state":"active+clean","last_fresh":"2021-03-24T22:06:48.461072+0000","last_change":"2021-03-24T22:06:48.461786+0000","last_active":"2021-03-24T22:06:48.461072+0000","last_peered":"2021-03-24T22:06:48.461072+0000","last_clean":"2021-03-24T22:06:48.461072+0000","last_became_active":"2021-03-24T22:03:50.369486+0000","last_became_peered":"2021-03-24T22:03:50.369486+0000","last_unstale":"2021-03-24T22:06:48.461072+0000","last_undegraded":"2021-03-24T22:06:48.461072+0000","last_fullsized":"2021-03-24T22:06:48.461072+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"831'493","last_scrub_stamp":"2021-03-24T22:06:41.587466+0000","last_deep_scrub":"616'351","last_deep_scrub_stamp":"2021-03-24T22:03:06.473123+0000","last_clean_scrub_stamp":"2021-03-24T22:06:41.587466+0000","log_size":493,"ondisk_log_size":493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9513521,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":106,"num_read_kb":114953,"num_write":61,"num_write_kb":23603,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":35984561,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"255","length":"1"},{"start":"27b","length":"1"},{"start":"297","length":"1"},{"start":"2a7","length":"1"},{"start":"2ab","length":"1"}]},{"pgid":"5.5","version":"0'0","reported_seq":"857","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.463255+0000","last_change":"2021-03-24T22:06:47.449275+0000","last_active":"2021-03-24T22:06:47.463255+0000","last_peered":"2021-03-24T22:06:47.463255+0000","last_clean":"2021-03-24T22:06:47.463255+0000","last_became_active":"2021-03-24T22:03:45.430121+0000","last_became_peered":"2021-03-24T22:03:45.430121+0000","last_unstale":"2021-03-24T22:06:47.463255+0000","last_undegraded":"2021-03-24T22:06:47.463255+0000","last_fullsized":"2021-03-24T22:06:47.463255+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:47.449147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:06:47.449147+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"544'37","reported_seq":"227","reported_epoch":"838","state":"active+clean","last_fresh":"2021-03-24T22:06:48.459043+0000","last_change":"2021-03-24T22:04:58.618464+0000","last_active":"2021-03-24T22:06:48.459043+0000","last_peered":"2021-03-24T22:06:48.459043+0000","last_clean":"2021-03-24T22:06:48.459043+0000","last_became_active":"2021-03-24T22:03:45.426862+0000","last_became_peered":"2021-03-24T22:03:45.426862+0000","last_unstale":"2021-03-24T22:06:48.459043+0000","last_undegraded":"2021-03-24T22:06:48.459043+0000","last_fullsized":"2021-03-24T22:06:48.459043+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'37","last_scrub_stamp":"2021-03-24T22:04:58.618374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:04:58.618374+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":70,"num_read_kb":47,"num_write":40,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":5724,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"89'2","reported_seq":"440","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.462836+0000","last_change":"2021-03-24T22:05:04.320737+0000","last_active":"2021-03-24T22:06:47.462836+0000","last_peered":"2021-03-24T22:06:47.462836+0000","last_clean":"2021-03-24T22:06:47.462836+0000","last_became_active":"2021-03-24T22:01:43.393718+0000","last_became_peered":"2021-03-24T22:01:43.393718+0000","last_unstale":"2021-03-24T22:06:47.462836+0000","last_undegraded":"2021-03-24T22:06:47.462836+0000","last_fullsized":"2021-03-24T22:06:47.462836+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2021-03-24T22:05:04.320667+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2021-03-24T21:57:58.491830+0000","last_clean_scrub_stamp":"2021-03-24T22:05:04.320667+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"214","reported_epoch":"838","state":"active+clean","last_fresh":"2021-03-24T22:06:48.457781+0000","last_change":"2021-03-24T22:05:10.501601+0000","last_active":"2021-03-24T22:06:48.457781+0000","last_peered":"2021-03-24T22:06:48.457781+0000","last_clean":"2021-03-24T22:06:48.457781+0000","last_became_active":"2021-03-24T22:05:04.450107+0000","last_became_peered":"2021-03-24T22:05:04.450107+0000","last_unstale":"2021-03-24T22:06:48.457781+0000","last_undegraded":"2021-03-24T22:06:48.457781+0000","last_fullsized":"2021-03-24T22:06:48.457781+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:10.501490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:06.800417+0000","last_clean_scrub_stamp":"2021-03-24T22:05:10.501490+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"850","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.455773+0000","last_change":"2021-03-24T22:02:55.490903+0000","last_active":"2021-03-24T22:06:47.455773+0000","last_peered":"2021-03-24T22:06:47.455773+0000","last_clean":"2021-03-24T22:06:47.455773+0000","last_became_active":"2021-03-24T22:01:57.314372+0000","last_became_peered":"2021-03-24T22:01:57.314372+0000","last_unstale":"2021-03-24T22:06:47.455773+0000","last_undegraded":"2021-03-24T22:06:47.455773+0000","last_fullsized":"2021-03-24T22:06:47.455773+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:55.490744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:52:59.706511+0000","last_clean_scrub_stamp":"2021-03-24T22:02:55.490744+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"838'636","reported_seq":"3084","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-24T22:06:50.702865+0000","last_change":"2021-03-24T22:06:49.734580+0000","last_active":"2021-03-24T22:06:50.702865+0000","last_peered":"2021-03-24T22:06:50.702865+0000","last_clean":"2021-03-24T22:06:50.702865+0000","last_became_active":"2021-03-24T22:01:47.305050+0000","last_became_peered":"2021-03-24T22:01:47.305050+0000","last_unstale":"2021-03-24T22:06:50.702865+0000","last_undegraded":"2021-03-24T22:06:50.702865+0000","last_fullsized":"2021-03-24T22:06:50.702865+0000","mapping_epoch":536,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":537,"parent":"0.0","parent_split_bits":0,"last_scrub":"699'500","last_scrub_stamp":"2021-03-24T22:04:31.592343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:04:31.592343+0000","log_size":636,"ondisk_log_size":636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7767212,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":7099,"num_write":67,"num_write_kb":23311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":55849492,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"27b","length":"1"},{"start":"2a7","length":"1"},{"start":"2cb","length":"1"}]},{"pgid":"4.b","version":"544'17","reported_seq":"958","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.462855+0000","last_change":"2021-03-24T22:04:34.106367+0000","last_active":"2021-03-24T22:06:47.462855+0000","last_peered":"2021-03-24T22:06:47.462855+0000","last_clean":"2021-03-24T22:06:47.462855+0000","last_became_active":"2021-03-24T22:03:46.443398+0000","last_became_peered":"2021-03-24T22:03:46.443398+0000","last_unstale":"2021-03-24T22:06:47.462855+0000","last_undegraded":"2021-03-24T22:06:47.462855+0000","last_fullsized":"2021-03-24T22:06:47.462855+0000","mapping_epoch":655,"log_start":"91'9","ondisk_log_start":"91'9","created":62,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'17","last_scrub_stamp":"2021-03-24T22:04:34.106163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:04:34.106163+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1158,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":58,"num_read_kb":36,"num_write":30,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2316,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"845","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.456197+0000","last_change":"2021-03-24T22:06:41.077009+0000","last_active":"2021-03-24T22:06:47.456197+0000","last_peered":"2021-03-24T22:06:47.456197+0000","last_clean":"2021-03-24T22:06:47.456197+0000","last_became_active":"2021-03-24T22:05:04.446424+0000","last_became_peered":"2021-03-24T22:05:04.446424+0000","last_unstale":"2021-03-24T22:06:47.456197+0000","last_undegraded":"2021-03-24T22:06:47.456197+0000","last_fullsized":"2021-03-24T22:06:47.456197+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:41.076877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T22:01:55.431129+0000","last_clean_scrub_stamp":"2021-03-24T22:06:41.076877+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"808","reported_epoch":"835","state":"active+clean","last_fresh":"2021-03-24T22:06:45.541094+0000","last_change":"2021-03-24T22:06:01.013669+0000","last_active":"2021-03-24T22:06:45.541094+0000","last_peered":"2021-03-24T22:06:45.541094+0000","last_clean":"2021-03-24T22:06:45.541094+0000","last_became_active":"2021-03-24T21:51:49.411921+0000","last_became_peered":"2021-03-24T21:51:49.411921+0000","last_unstale":"2021-03-24T22:06:45.541094+0000","last_undegraded":"2021-03-24T22:06:45.541094+0000","last_fullsized":"2021-03-24T22:06:45.541094+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:01.013563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:06:01.013563+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"859","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.462272+0000","last_change":"2021-03-24T22:04:31.107367+0000","last_active":"2021-03-24T22:06:47.462272+0000","last_peered":"2021-03-24T22:06:47.462272+0000","last_clean":"2021-03-24T22:06:47.462272+0000","last_became_active":"2021-03-24T22:02:22.323876+0000","last_became_peered":"2021-03-24T22:02:22.323876+0000","last_unstale":"2021-03-24T22:06:47.462272+0000","last_undegraded":"2021-03-24T22:06:47.462272+0000","last_fullsized":"2021-03-24T22:06:47.462272+0000","mapping_epoch":571,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:04:31.107254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:35.636299+0000","last_clean_scrub_stamp":"2021-03-24T22:04:31.107254+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"821'333","reported_seq":"2787","reported_epoch":"835","state":"active+clean","last_fresh":"2021-03-24T22:06:45.539850+0000","last_change":"2021-03-24T22:06:41.482105+0000","last_active":"2021-03-24T22:06:45.539850+0000","last_peered":"2021-03-24T22:06:45.539850+0000","last_clean":"2021-03-24T22:06:45.539850+0000","last_became_active":"2021-03-24T22:02:01.325483+0000","last_became_peered":"2021-03-24T22:02:01.325483+0000","last_unstale":"2021-03-24T22:06:45.539850+0000","last_undegraded":"2021-03-24T22:06:45.539850+0000","last_fullsized":"2021-03-24T22:06:45.539850+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"792'330","last_scrub_stamp":"2021-03-24T22:06:03.024091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:06:03.024091+0000","log_size":333,"ondisk_log_size":333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5244246,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":88,"num_read_kb":99622,"num_write":63,"num_write_kb":17849,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":21446378,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"255","length":"1"},{"start":"297","length":"1"},{"start":"2ab","length":"1"}]},{"pgid":"4.a","version":"544'45","reported_seq":"805","reported_epoch":"839","state":"active+clean","last_fresh":"2021-03-24T22:06:49.730840+0000","last_change":"2021-03-24T22:04:26.640197+0000","last_active":"2021-03-24T22:06:49.730840+0000","last_peered":"2021-03-24T22:06:49.730840+0000","last_clean":"2021-03-24T22:06:49.730840+0000","last_became_active":"2021-03-24T22:04:26.561945+0000","last_became_peered":"2021-03-24T22:04:26.561945+0000","last_unstale":"2021-03-24T22:06:49.730840+0000","last_undegraded":"2021-03-24T22:06:49.730840+0000","last_fullsized":"2021-03-24T22:06:49.730840+0000","mapping_epoch":695,"log_start":"69'23","ondisk_log_start":"69'23","created":62,"last_epoch_clean":696,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'45","last_scrub_stamp":"2021-03-24T22:04:26.640132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:04:26.640132+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1466,"num_objects":21,"num_object_clones":0,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":104,"num_read_kb":71,"num_write":60,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":7174,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":"457","reported_epoch":"835","state":"active+clean","last_fresh":"2021-03-24T22:06:45.446215+0000","last_change":"2021-03-24T22:05:51.326820+0000","last_active":"2021-03-24T22:06:45.446215+0000","last_peered":"2021-03-24T22:06:45.446215+0000","last_clean":"2021-03-24T22:06:45.446215+0000","last_became_active":"2021-03-24T22:05:04.439833+0000","last_became_peered":"2021-03-24T22:05:04.439833+0000","last_unstale":"2021-03-24T22:06:45.446215+0000","last_undegraded":"2021-03-24T22:06:45.446215+0000","last_fullsized":"2021-03-24T22:06:45.446215+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:51.326637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:05:51.326637+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"296","reported_epoch":"835","state":"active+clean","last_fresh":"2021-03-24T22:06:45.541986+0000","last_change":"2021-03-24T22:06:44.162065+0000","last_active":"2021-03-24T22:06:45.541986+0000","last_peered":"2021-03-24T22:06:45.541986+0000","last_clean":"2021-03-24T22:06:45.541986+0000","last_became_active":"2021-03-24T22:05:04.426559+0000","last_became_peered":"2021-03-24T22:05:04.426559+0000","last_unstale":"2021-03-24T22:06:45.541986+0000","last_undegraded":"2021-03-24T22:06:45.541986+0000","last_fullsized":"2021-03-24T22:06:45.541986+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:44.161993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T22:05:08.670240+0000","last_clean_scrub_stamp":"2021-03-24T22:06:44.161993+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"601","reported_epoch":"835","state":"active+clean","last_fresh":"2021-03-24T22:06:45.539556+0000","last_change":"2021-03-24T22:05:55.032235+0000","last_active":"2021-03-24T22:06:45.539556+0000","last_peered":"2021-03-24T22:06:45.539556+0000","last_clean":"2021-03-24T22:06:45.539556+0000","last_became_active":"2021-03-24T22:01:57.311325+0000","last_became_peered":"2021-03-24T22:01:57.311325+0000","last_unstale":"2021-03-24T22:06:45.539556+0000","last_undegraded":"2021-03-24T22:06:45.539556+0000","last_fullsized":"2021-03-24T22:06:45.539556+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:55.032073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:05:55.032073+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"831'554","reported_seq":"3044","reported_epoch":"838","state":"active+clean","last_fresh":"2021-03-24T22:06:48.460305+0000","last_change":"2021-03-24T22:06:48.460930+0000","last_active":"2021-03-24T22:06:48.460305+0000","last_peered":"2021-03-24T22:06:48.460305+0000","last_clean":"2021-03-24T22:06:48.460305+0000","last_became_active":"2021-03-24T22:03:49.375663+0000","last_became_peered":"2021-03-24T22:03:49.375663+0000","last_unstale":"2021-03-24T22:06:48.460305+0000","last_undegraded":"2021-03-24T22:06:48.460305+0000","last_fullsized":"2021-03-24T22:06:48.460305+0000","mapping_epoch":658,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":659,"parent":"0.0","parent_split_bits":0,"last_scrub":"795'538","last_scrub_stamp":"2021-03-24T22:06:05.470009+0000","last_deep_scrub":"795'538","last_deep_scrub_stamp":"2021-03-24T22:06:05.470009+0000","last_clean_scrub_stamp":"2021-03-24T22:06:05.470009+0000","log_size":554,"ondisk_log_size":554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7899950,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":73,"num_read_kb":81143,"num_write":68,"num_write_kb":19581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":32161412,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"255","length":"1"},{"start":"27b","length":"1"},{"start":"297","length":"1"},{"start":"2a7","length":"1"},{"start":"2ab","length":"1"}]},{"pgid":"5.8","version":"0'0","reported_seq":"861","reported_epoch":"835","state":"active+clean","last_fresh":"2021-03-24T22:06:45.539262+0000","last_change":"2021-03-24T22:06:04.053700+0000","last_active":"2021-03-24T22:06:45.539262+0000","last_peered":"2021-03-24T22:06:45.539262+0000","last_clean":"2021-03-24T22:06:45.539262+0000","last_became_active":"2021-03-24T22:03:45.432819+0000","last_became_peered":"2021-03-24T22:03:45.432819+0000","last_unstale":"2021-03-24T22:06:45.539262+0000","last_undegraded":"2021-03-24T22:06:45.539262+0000","last_fullsized":"2021-03-24T22:06:45.539262+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:04.053530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:06:04.053530+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"544'38","reported_seq":"545","reported_epoch":"838","state":"active+clean","last_fresh":"2021-03-24T22:06:48.458850+0000","last_change":"2021-03-24T22:05:00.634636+0000","last_active":"2021-03-24T22:06:48.458850+0000","last_peered":"2021-03-24T22:06:48.458850+0000","last_clean":"2021-03-24T22:06:48.458850+0000","last_became_active":"2021-03-24T22:04:50.402497+0000","last_became_peered":"2021-03-24T22:04:50.402497+0000","last_unstale":"2021-03-24T22:06:48.458850+0000","last_undegraded":"2021-03-24T22:06:48.458850+0000","last_fullsized":"2021-03-24T22:06:48.458850+0000","mapping_epoch":719,"log_start":"69'20","ondisk_log_start":"69'20","created":62,"last_epoch_clean":720,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'38","last_scrub_stamp":"2021-03-24T22:05:00.634536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:05:00.634536+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1582,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":100,"num_read_kb":66,"num_write":56,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":4746,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":"326","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.474786+0000","last_change":"2021-03-24T22:06:37.050021+0000","last_active":"2021-03-24T22:06:47.474786+0000","last_peered":"2021-03-24T22:06:47.474786+0000","last_clean":"2021-03-24T22:06:47.474786+0000","last_became_active":"2021-03-24T22:03:45.437360+0000","last_became_peered":"2021-03-24T22:03:45.437360+0000","last_unstale":"2021-03-24T22:06:47.474786+0000","last_undegraded":"2021-03-24T22:06:47.474786+0000","last_fullsized":"2021-03-24T22:06:47.474786+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:37.049845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:06:37.049845+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"872","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.455358+0000","last_change":"2021-03-24T22:05:14.248048+0000","last_active":"2021-03-24T22:06:47.455358+0000","last_peered":"2021-03-24T22:06:47.455358+0000","last_clean":"2021-03-24T22:06:47.455358+0000","last_became_active":"2021-03-24T22:03:45.436096+0000","last_became_peered":"2021-03-24T22:03:45.436096+0000","last_unstale":"2021-03-24T22:06:47.455358+0000","last_undegraded":"2021-03-24T22:06:47.455358+0000","last_fullsized":"2021-03-24T22:06:47.455358+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:14.247913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:22.724840+0000","last_clean_scrub_stamp":"2021-03-24T22:05:14.247913+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"838'396","reported_seq":"2696","reported_epoch":"838","state":"active+clean","last_fresh":"2021-03-24T22:06:48.490088+0000","last_change":"2021-03-24T22:06:48.490438+0000","last_active":"2021-03-24T22:06:48.490088+0000","last_peered":"2021-03-24T22:06:48.490088+0000","last_clean":"2021-03-24T22:06:48.490088+0000","last_became_active":"2021-03-24T22:05:04.442753+0000","last_became_peered":"2021-03-24T22:05:04.442753+0000","last_unstale":"2021-03-24T22:06:48.490088+0000","last_undegraded":"2021-03-24T22:06:48.490088+0000","last_fullsized":"2021-03-24T22:06:48.490088+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"624'277","last_scrub_stamp":"2021-03-24T22:03:14.428074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:03:14.428074+0000","log_size":396,"ondisk_log_size":396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7500172,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":42,"num_read_kb":48474,"num_write":15,"num_write_kb":7826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1181773,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"255","length":"1"},{"start":"27b","length":"1"},{"start":"297","length":"1"},{"start":"2a7","length":"1"},{"start":"2ab","length":"1"}]},{"pgid":"5.9","version":"548'11","reported_seq":"370","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.475677+0000","last_change":"2021-03-24T22:06:43.019415+0000","last_active":"2021-03-24T22:06:47.475677+0000","last_peered":"2021-03-24T22:06:47.475677+0000","last_clean":"2021-03-24T22:06:47.475677+0000","last_became_active":"2021-03-24T22:01:57.313563+0000","last_became_peered":"2021-03-24T22:01:57.313563+0000","last_unstale":"2021-03-24T22:06:47.475677+0000","last_undegraded":"2021-03-24T22:06:47.475677+0000","last_fullsized":"2021-03-24T22:06:47.475677+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"548'11","last_scrub_stamp":"2021-03-24T22:06:43.019318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:06:43.019318+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"544'46","reported_seq":"816","reported_epoch":"835","state":"active+clean","last_fresh":"2021-03-24T22:06:45.443857+0000","last_change":"2021-03-24T22:06:07.382916+0000","last_active":"2021-03-24T22:06:45.443857+0000","last_peered":"2021-03-24T22:06:45.443857+0000","last_clean":"2021-03-24T22:06:45.443857+0000","last_became_active":"2021-03-24T22:05:05.422281+0000","last_became_peered":"2021-03-24T22:05:05.422281+0000","last_unstale":"2021-03-24T22:06:45.443857+0000","last_undegraded":"2021-03-24T22:06:45.443857+0000","last_fullsized":"2021-03-24T22:06:45.443857+0000","mapping_epoch":734,"log_start":"97'28","ondisk_log_start":"97'28","created":62,"last_epoch_clean":735,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'46","last_scrub_stamp":"2021-03-24T22:06:07.382720+0000","last_deep_scrub":"97'28","last_deep_scrub_stamp":"2021-03-24T21:54:19.157265+0000","last_clean_scrub_stamp":"2021-03-24T22:06:07.382720+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1616,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":96,"num_read_kb":63,"num_write":53,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":5308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"89'4","reported_seq":"314","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.473434+0000","last_change":"2021-03-24T22:06:04.178234+0000","last_active":"2021-03-24T22:06:47.473434+0000","last_peered":"2021-03-24T22:06:47.473434+0000","last_clean":"2021-03-24T22:06:47.473434+0000","last_became_active":"2021-03-24T22:01:57.316325+0000","last_became_peered":"2021-03-24T22:01:57.316325+0000","last_unstale":"2021-03-24T22:06:47.473434+0000","last_undegraded":"2021-03-24T22:06:47.473434+0000","last_fullsized":"2021-03-24T22:06:47.473434+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'4","last_scrub_stamp":"2021-03-24T22:06:04.178049+0000","last_deep_scrub":"89'4","last_deep_scrub_stamp":"2021-03-24T21:53:19.926637+0000","last_clean_scrub_stamp":"2021-03-24T22:06:04.178049+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"335","reported_epoch":"839","state":"active+clean","last_fresh":"2021-03-24T22:06:49.731389+0000","last_change":"2021-03-24T22:05:15.379559+0000","last_active":"2021-03-24T22:06:49.731389+0000","last_peered":"2021-03-24T22:06:49.731389+0000","last_clean":"2021-03-24T22:06:49.731389+0000","last_became_active":"2021-03-24T22:05:04.444421+0000","last_became_peered":"2021-03-24T22:05:04.444421+0000","last_unstale":"2021-03-24T22:06:49.731389+0000","last_undegraded":"2021-03-24T22:06:49.731389+0000","last_fullsized":"2021-03-24T22:06:49.731389+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:15.379478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:05:15.379478+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"272'275","reported_seq":"2173","reported_epoch":"839","state":"active+clean","last_fresh":"2021-03-24T22:06:49.659907+0000","last_change":"2021-03-24T22:06:49.660212+0000","last_active":"2021-03-24T22:06:49.659907+0000","last_peered":"2021-03-24T22:06:49.659907+0000","last_clean":"2021-03-24T22:06:49.659907+0000","last_became_active":"2021-03-24T22:05:04.425270+0000","last_became_peered":"2021-03-24T22:05:04.425270+0000","last_unstale":"2021-03-24T22:06:49.659907+0000","last_undegraded":"2021-03-24T22:06:49.659907+0000","last_fullsized":"2021-03-24T22:06:49.659907+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'275","last_scrub_stamp":"2021-03-24T22:05:59.526944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:05:59.526944+0000","log_size":275,"ondisk_log_size":275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"255","length":"1"},{"start":"27b","length":"1"},{"start":"297","length":"1"},{"start":"2a7","length":"1"},{"start":"2ab","length":"1"},{"start":"2cb","length":"1"}]},{"pgid":"3.10","version":"0'0","reported_seq":"337","reported_epoch":"839","state":"active+clean","last_fresh":"2021-03-24T22:06:49.730778+0000","last_change":"2021-03-24T22:05:47.360975+0000","last_active":"2021-03-24T22:06:49.730778+0000","last_peered":"2021-03-24T22:06:49.730778+0000","last_clean":"2021-03-24T22:06:49.730778+0000","last_became_active":"2021-03-24T22:05:04.440682+0000","last_became_peered":"2021-03-24T22:05:04.440682+0000","last_unstale":"2021-03-24T22:06:49.730778+0000","last_undegraded":"2021-03-24T22:06:49.730778+0000","last_fullsized":"2021-03-24T22:06:49.730778+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:47.360795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:05:47.360795+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"685","reported_epoch":"835","state":"active+clean","last_fresh":"2021-03-24T22:06:45.446756+0000","last_change":"2021-03-24T22:06:02.409659+0000","last_active":"2021-03-24T22:06:45.446756+0000","last_peered":"2021-03-24T22:06:45.446756+0000","last_clean":"2021-03-24T22:06:45.446756+0000","last_became_active":"2021-03-24T22:05:04.437651+0000","last_became_peered":"2021-03-24T22:05:04.437651+0000","last_unstale":"2021-03-24T22:06:45.446756+0000","last_undegraded":"2021-03-24T22:06:45.446756+0000","last_fullsized":"2021-03-24T22:06:45.446756+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:02.409388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T22:06:02.409388+0000","last_clean_scrub_stamp":"2021-03-24T22:06:02.409388+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"308","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.472142+0000","last_change":"2021-03-24T22:05:57.160116+0000","last_active":"2021-03-24T22:06:47.472142+0000","last_peered":"2021-03-24T22:06:47.472142+0000","last_clean":"2021-03-24T22:06:47.472142+0000","last_became_active":"2021-03-24T22:01:57.312376+0000","last_became_peered":"2021-03-24T22:01:57.312376+0000","last_unstale":"2021-03-24T22:06:47.472142+0000","last_undegraded":"2021-03-24T22:06:47.472142+0000","last_fullsized":"2021-03-24T22:06:47.472142+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:57.159943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:05:57.159943+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.14","version":"838'1112","reported_seq":"3451","reported_epoch":"838","state":"active+clean","last_fresh":"2021-03-24T22:06:48.498456+0000","last_change":"2021-03-24T22:06:48.498728+0000","last_active":"2021-03-24T22:06:48.498456+0000","last_peered":"2021-03-24T22:06:48.498456+0000","last_clean":"2021-03-24T22:06:48.498456+0000","last_became_active":"2021-03-24T22:05:04.453660+0000","last_became_peered":"2021-03-24T22:05:04.453660+0000","last_unstale":"2021-03-24T22:06:48.498456+0000","last_undegraded":"2021-03-24T22:06:48.498456+0000","last_fullsized":"2021-03-24T22:06:48.498456+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":735,"parent":"0.0","parent_split_bits":5,"last_scrub":"651'772","last_scrub_stamp":"2021-03-24T22:03:42.250616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:03:42.250616+0000","log_size":1112,"ondisk_log_size":1112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":41099914,"num_objects":18,"num_object_clones":15,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":1,"num_read":103,"num_read_kb":113197,"num_write":70,"num_write_kb":32688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":14235619,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"255","length":"1"},{"start":"27b","length":"1"},{"start":"297","length":"1"},{"start":"2a7","length":"1"},{"start":"2ab","length":"1"}]},{"pgid":"3.11","version":"0'0","reported_seq":"329","reported_epoch":"839","state":"active+clean","last_fresh":"2021-03-24T22:06:49.732138+0000","last_change":"2021-03-24T22:04:32.551783+0000","last_active":"2021-03-24T22:06:49.732138+0000","last_peered":"2021-03-24T22:06:49.732138+0000","last_clean":"2021-03-24T22:06:49.732138+0000","last_became_active":"2021-03-24T22:01:57.304423+0000","last_became_peered":"2021-03-24T22:01:57.304423+0000","last_unstale":"2021-03-24T22:06:49.732138+0000","last_undegraded":"2021-03-24T22:06:49.732138+0000","last_fullsized":"2021-03-24T22:06:49.732138+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:04:32.551682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:04:32.551682+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"868","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.455852+0000","last_change":"2021-03-24T22:03:45.435984+0000","last_active":"2021-03-24T22:06:47.455852+0000","last_peered":"2021-03-24T22:06:47.455852+0000","last_clean":"2021-03-24T22:06:47.455852+0000","last_became_active":"2021-03-24T22:03:45.434516+0000","last_became_peered":"2021-03-24T22:03:45.434516+0000","last_unstale":"2021-03-24T22:06:47.455852+0000","last_undegraded":"2021-03-24T22:06:47.455852+0000","last_fullsized":"2021-03-24T22:06:47.455852+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:02:56.523309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:55.493713+0000","last_clean_scrub_stamp":"2021-03-24T22:02:56.523309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"842","reported_epoch":"835","state":"active+clean","last_fresh":"2021-03-24T22:06:45.537258+0000","last_change":"2021-03-24T22:06:42.200017+0000","last_active":"2021-03-24T22:06:45.537258+0000","last_peered":"2021-03-24T22:06:45.537258+0000","last_clean":"2021-03-24T22:06:45.537258+0000","last_became_active":"2021-03-24T22:01:57.312185+0000","last_became_peered":"2021-03-24T22:01:57.312185+0000","last_unstale":"2021-03-24T22:06:45.537258+0000","last_undegraded":"2021-03-24T22:06:45.537258+0000","last_fullsized":"2021-03-24T22:06:45.537258+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:42.199899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:06:42.199899+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"838'429","reported_seq":"3007","reported_epoch":"838","state":"active+clean","last_fresh":"2021-03-24T22:06:48.495320+0000","last_change":"2021-03-24T22:06:48.495978+0000","last_active":"2021-03-24T22:06:48.495320+0000","last_peered":"2021-03-24T22:06:48.495320+0000","last_clean":"2021-03-24T22:06:48.495320+0000","last_became_active":"2021-03-24T22:03:51.363965+0000","last_became_peered":"2021-03-24T22:03:51.363965+0000","last_unstale":"2021-03-24T22:06:48.495320+0000","last_undegraded":"2021-03-24T22:06:48.495320+0000","last_fullsized":"2021-03-24T22:06:48.495320+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":5,"last_scrub":"710'317","last_scrub_stamp":"2021-03-24T22:04:41.171323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:04:41.171323+0000","log_size":429,"ondisk_log_size":429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14082423,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":83,"num_read_kb":115822,"num_write":62,"num_write_kb":23983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":85409227,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"255","length":"1"},{"start":"27b","length":"1"},{"start":"297","length":"1"},{"start":"2a7","length":"1"},{"start":"2ab","length":"1"}]},{"pgid":"4.15","version":"544'19","reported_seq":"758","reported_epoch":"835","state":"active+clean","last_fresh":"2021-03-24T22:06:45.446372+0000","last_change":"2021-03-24T22:06:36.315165+0000","last_active":"2021-03-24T22:06:45.446372+0000","last_peered":"2021-03-24T22:06:45.446372+0000","last_clean":"2021-03-24T22:06:45.446372+0000","last_became_active":"2021-03-24T22:05:04.433341+0000","last_became_peered":"2021-03-24T22:05:04.433341+0000","last_unstale":"2021-03-24T22:06:45.446372+0000","last_undegraded":"2021-03-24T22:06:45.446372+0000","last_fullsized":"2021-03-24T22:06:45.446372+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'19","last_scrub_stamp":"2021-03-24T22:06:36.314962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:06:36.314962+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":36,"num_read_kb":22,"num_write":20,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2202,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":"843","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.462718+0000","last_change":"2021-03-24T22:04:57.279696+0000","last_active":"2021-03-24T22:06:47.462718+0000","last_peered":"2021-03-24T22:06:47.462718+0000","last_clean":"2021-03-24T22:06:47.462718+0000","last_became_active":"2021-03-24T22:01:57.315967+0000","last_became_peered":"2021-03-24T22:01:57.315967+0000","last_unstale":"2021-03-24T22:06:47.462718+0000","last_undegraded":"2021-03-24T22:06:47.462718+0000","last_fullsized":"2021-03-24T22:06:47.462718+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:04:57.279517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:05.473348+0000","last_clean_scrub_stamp":"2021-03-24T22:04:57.279517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"573'12","reported_seq":"1071","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.458123+0000","last_change":"2021-03-24T22:03:02.456113+0000","last_active":"2021-03-24T22:06:47.458123+0000","last_peered":"2021-03-24T22:06:47.458123+0000","last_clean":"2021-03-24T22:06:47.458123+0000","last_became_active":"2021-03-24T22:02:22.326377+0000","last_became_peered":"2021-03-24T22:02:22.326377+0000","last_unstale":"2021-03-24T22:06:47.458123+0000","last_undegraded":"2021-03-24T22:06:47.458123+0000","last_fullsized":"2021-03-24T22:06:47.458123+0000","mapping_epoch":571,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":0,"last_scrub":"573'12","last_scrub_stamp":"2021-03-24T22:03:02.455946+0000","last_deep_scrub":"86'3","last_deep_scrub_stamp":"2021-03-24T21:52:56.678111+0000","last_clean_scrub_stamp":"2021-03-24T22:03:02.455946+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"787","reported_epoch":"835","state":"active+clean","last_fresh":"2021-03-24T22:06:45.541069+0000","last_change":"2021-03-24T22:05:06.701715+0000","last_active":"2021-03-24T22:06:45.541069+0000","last_peered":"2021-03-24T22:06:45.541069+0000","last_clean":"2021-03-24T22:06:45.541069+0000","last_became_active":"2021-03-24T22:05:04.429425+0000","last_became_peered":"2021-03-24T22:05:04.429425+0000","last_unstale":"2021-03-24T22:06:45.541069+0000","last_undegraded":"2021-03-24T22:06:45.541069+0000","last_fullsized":"2021-03-24T22:06:45.541069+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:06.701589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:05:06.701589+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.16","version":"837'559","reported_seq":"3065","reported_epoch":"838","state":"active+clean","last_fresh":"2021-03-24T22:06:48.481554+0000","last_change":"2021-03-24T22:06:48.482137+0000","last_active":"2021-03-24T22:06:48.481554+0000","last_peered":"2021-03-24T22:06:48.481554+0000","last_clean":"2021-03-24T22:06:48.481554+0000","last_became_active":"2021-03-24T22:02:08.322570+0000","last_became_peered":"2021-03-24T22:02:08.322570+0000","last_unstale":"2021-03-24T22:06:48.481554+0000","last_undegraded":"2021-03-24T22:06:48.481554+0000","last_fullsized":"2021-03-24T22:06:48.481554+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":5,"last_scrub":"827'555","last_scrub_stamp":"2021-03-24T22:06:39.566799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:06:39.566799+0000","log_size":559,"ondisk_log_size":559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3933036,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":104,"num_read_kb":117115,"num_write":81,"num_write_kb":19657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":30734442,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"255","length":"1"},{"start":"27b","length":"1"},{"start":"297","length":"1"},{"start":"2a7","length":"1"},{"start":"2ab","length":"1"}]},{"pgid":"4.14","version":"544'24","reported_seq":"1082","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.457826+0000","last_change":"2021-03-24T22:04:37.278384+0000","last_active":"2021-03-24T22:06:47.457826+0000","last_peered":"2021-03-24T22:06:47.457826+0000","last_clean":"2021-03-24T22:06:47.457826+0000","last_became_active":"2021-03-24T22:03:45.433725+0000","last_became_peered":"2021-03-24T22:03:45.433725+0000","last_unstale":"2021-03-24T22:06:47.457826+0000","last_undegraded":"2021-03-24T22:06:47.457826+0000","last_fullsized":"2021-03-24T22:06:47.457826+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'24","last_scrub_stamp":"2021-03-24T22:04:37.278283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:04:37.278283+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":42,"num_read_kb":29,"num_write":24,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":4992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":"308","reported_epoch":"838","state":"active+clean","last_fresh":"2021-03-24T22:06:49.081020+0000","last_change":"2021-03-24T22:06:49.081020+0000","last_active":"2021-03-24T22:06:49.081020+0000","last_peered":"2021-03-24T22:06:49.081020+0000","last_clean":"2021-03-24T22:06:49.081020+0000","last_became_active":"2021-03-24T22:01:57.318685+0000","last_became_peered":"2021-03-24T22:01:57.318685+0000","last_unstale":"2021-03-24T22:06:49.081020+0000","last_undegraded":"2021-03-24T22:06:49.081020+0000","last_fullsized":"2021-03-24T22:06:49.081020+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:49.080845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:06:49.080845+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"743'12","reported_seq":"841","reported_epoch":"835","state":"active+clean","last_fresh":"2021-03-24T22:06:45.445849+0000","last_change":"2021-03-24T22:06:43.305379+0000","last_active":"2021-03-24T22:06:45.445849+0000","last_peered":"2021-03-24T22:06:45.445849+0000","last_clean":"2021-03-24T22:06:45.445849+0000","last_became_active":"2021-03-24T22:05:04.451042+0000","last_became_peered":"2021-03-24T22:05:04.451042+0000","last_unstale":"2021-03-24T22:06:45.445849+0000","last_undegraded":"2021-03-24T22:06:45.445849+0000","last_fullsized":"2021-03-24T22:06:45.445849+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"743'12","last_scrub_stamp":"2021-03-24T22:06:43.305191+0000","last_deep_scrub":"86'3","last_deep_scrub_stamp":"2021-03-24T21:53:08.815163+0000","last_clean_scrub_stamp":"2021-03-24T22:06:43.305191+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"846","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.467053+0000","last_change":"2021-03-24T22:06:44.506424+0000","last_active":"2021-03-24T22:06:47.467053+0000","last_peered":"2021-03-24T22:06:47.467053+0000","last_clean":"2021-03-24T22:06:47.467053+0000","last_became_active":"2021-03-24T22:01:57.316729+0000","last_became_peered":"2021-03-24T22:01:57.316729+0000","last_unstale":"2021-03-24T22:06:47.467053+0000","last_undegraded":"2021-03-24T22:06:47.467053+0000","last_fullsized":"2021-03-24T22:06:47.467053+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:44.506317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:06:44.506317+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.17","version":"838'1398","reported_seq":"4043","reported_epoch":"838","state":"active+clean","last_fresh":"2021-03-24T22:06:48.493202+0000","last_change":"2021-03-24T22:06:48.493693+0000","last_active":"2021-03-24T22:06:48.493202+0000","last_peered":"2021-03-24T22:06:48.493202+0000","last_clean":"2021-03-24T22:06:48.493202+0000","last_became_active":"2021-03-24T22:05:04.460028+0000","last_became_peered":"2021-03-24T22:05:04.460028+0000","last_unstale":"2021-03-24T22:06:48.493202+0000","last_undegraded":"2021-03-24T22:06:48.493202+0000","last_fullsized":"2021-03-24T22:06:48.493202+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":5,"last_scrub":"742'1137","last_scrub_stamp":"2021-03-24T22:05:13.182580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:05:13.182580+0000","log_size":1398,"ondisk_log_size":1398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37594391,"num_objects":18,"num_object_clones":14,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":2,"num_read":355,"num_read_kb":387285,"num_write":248,"num_write_kb":90305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":257,"num_bytes_recovered":169206653,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"255","length":"1"},{"start":"27b","length":"1"},{"start":"297","length":"1"},{"start":"2a7","length":"1"},{"start":"2ab","length":"1"}]},{"pgid":"4.13","version":"584'32","reported_seq":"419","reported_epoch":"839","state":"active+clean","last_fresh":"2021-03-24T22:06:49.732873+0000","last_change":"2021-03-24T22:05:48.406758+0000","last_active":"2021-03-24T22:06:49.732873+0000","last_peered":"2021-03-24T22:06:49.732873+0000","last_clean":"2021-03-24T22:06:49.732873+0000","last_became_active":"2021-03-24T22:03:45.426065+0000","last_became_peered":"2021-03-24T22:03:45.426065+0000","last_unstale":"2021-03-24T22:06:49.732873+0000","last_undegraded":"2021-03-24T22:06:49.732873+0000","last_fullsized":"2021-03-24T22:06:49.732873+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"584'32","last_scrub_stamp":"2021-03-24T22:05:48.406579+0000","last_deep_scrub":"584'32","last_deep_scrub_stamp":"2021-03-24T22:02:34.813628+0000","last_clean_scrub_stamp":"2021-03-24T22:05:48.406579+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":29,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":3680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":"310","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.472875+0000","last_change":"2021-03-24T22:06:08.071800+0000","last_active":"2021-03-24T22:06:47.472875+0000","last_peered":"2021-03-24T22:06:47.472875+0000","last_clean":"2021-03-24T22:06:47.472875+0000","last_became_active":"2021-03-24T22:01:57.307018+0000","last_became_peered":"2021-03-24T22:01:57.307018+0000","last_unstale":"2021-03-24T22:06:47.472875+0000","last_undegraded":"2021-03-24T22:06:47.472875+0000","last_fullsized":"2021-03-24T22:06:47.472875+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:08.071696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:06:08.071696+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"214","reported_epoch":"838","state":"active+clean","last_fresh":"2021-03-24T22:06:48.457823+0000","last_change":"2021-03-24T22:05:07.629502+0000","last_active":"2021-03-24T22:06:48.457823+0000","last_peered":"2021-03-24T22:06:48.457823+0000","last_clean":"2021-03-24T22:06:48.457823+0000","last_became_active":"2021-03-24T22:05:04.451084+0000","last_became_peered":"2021-03-24T22:05:04.451084+0000","last_unstale":"2021-03-24T22:06:48.457823+0000","last_undegraded":"2021-03-24T22:06:48.457823+0000","last_fullsized":"2021-03-24T22:06:48.457823+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:07.629376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:57:31.118050+0000","last_clean_scrub_stamp":"2021-03-24T22:05:07.629376+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"824","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.454866+0000","last_change":"2021-03-24T22:05:52.855070+0000","last_active":"2021-03-24T22:06:47.454866+0000","last_peered":"2021-03-24T22:06:47.454866+0000","last_clean":"2021-03-24T22:06:47.454866+0000","last_became_active":"2021-03-24T22:05:04.435491+0000","last_became_peered":"2021-03-24T22:05:04.435491+0000","last_unstale":"2021-03-24T22:06:47.454866+0000","last_undegraded":"2021-03-24T22:06:47.454866+0000","last_fullsized":"2021-03-24T22:06:47.454866+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:52.854930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:23.707914+0000","last_clean_scrub_stamp":"2021-03-24T22:05:52.854930+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.10","version":"838'256","reported_seq":"1147","reported_epoch":"838","state":"active+clean","last_fresh":"2021-03-24T22:06:48.490855+0000","last_change":"2021-03-24T22:06:48.491129+0000","last_active":"2021-03-24T22:06:48.490855+0000","last_peered":"2021-03-24T22:06:48.490855+0000","last_clean":"2021-03-24T22:06:48.490855+0000","last_became_active":"2021-03-24T22:05:04.438010+0000","last_became_peered":"2021-03-24T22:05:04.438010+0000","last_unstale":"2021-03-24T22:06:48.490855+0000","last_undegraded":"2021-03-24T22:06:48.490855+0000","last_fullsized":"2021-03-24T22:06:48.490855+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":5,"last_scrub":"579'81","last_scrub_stamp":"2021-03-24T22:02:30.763305+0000","last_deep_scrub":"277'24","last_deep_scrub_stamp":"2021-03-24T21:57:28.119420+0000","last_clean_scrub_stamp":"2021-03-24T22:02:30.763305+0000","log_size":256,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11918454,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":115,"num_read_kb":111507,"num_write":61,"num_write_kb":21017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":32974145,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"255","length":"1"},{"start":"27b","length":"1"},{"start":"297","length":"1"},{"start":"2a7","length":"1"},{"start":"2ab","length":"1"}]},{"pgid":"4.12","version":"544'21","reported_seq":"226","reported_epoch":"838","state":"active+clean","last_fresh":"2021-03-24T22:06:48.458045+0000","last_change":"2021-03-24T22:06:41.782352+0000","last_active":"2021-03-24T22:06:48.458045+0000","last_peered":"2021-03-24T22:06:48.458045+0000","last_clean":"2021-03-24T22:06:48.458045+0000","last_became_active":"2021-03-24T22:03:45.427113+0000","last_became_peered":"2021-03-24T22:03:45.427113+0000","last_unstale":"2021-03-24T22:06:48.458045+0000","last_undegraded":"2021-03-24T22:06:48.458045+0000","last_fullsized":"2021-03-24T22:06:48.458045+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'21","last_scrub_stamp":"2021-03-24T22:06:41.782244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:06:41.782244+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":43,"num_read_kb":25,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":3150,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":"308","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.472592+0000","last_change":"2021-03-24T22:05:53.144403+0000","last_active":"2021-03-24T22:06:47.472592+0000","last_peered":"2021-03-24T22:06:47.472592+0000","last_clean":"2021-03-24T22:06:47.472592+0000","last_became_active":"2021-03-24T22:01:57.318513+0000","last_became_peered":"2021-03-24T22:01:57.318513+0000","last_unstale":"2021-03-24T22:06:47.472592+0000","last_undegraded":"2021-03-24T22:06:47.472592+0000","last_fullsized":"2021-03-24T22:06:47.472592+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:53.144236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T22:03:09.417465+0000","last_clean_scrub_stamp":"2021-03-24T22:05:53.144236+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"847","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.455080+0000","last_change":"2021-03-24T22:05:57.864744+0000","last_active":"2021-03-24T22:06:47.455080+0000","last_peered":"2021-03-24T22:06:47.455080+0000","last_clean":"2021-03-24T22:06:47.455080+0000","last_became_active":"2021-03-24T22:03:45.435094+0000","last_became_peered":"2021-03-24T22:03:45.435094+0000","last_unstale":"2021-03-24T22:06:47.455080+0000","last_undegraded":"2021-03-24T22:06:47.455080+0000","last_fullsized":"2021-03-24T22:06:47.455080+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:57.864596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:05:57.864596+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"867","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.462269+0000","last_change":"2021-03-24T22:05:59.395794+0000","last_active":"2021-03-24T22:06:47.462269+0000","last_peered":"2021-03-24T22:06:47.462269+0000","last_clean":"2021-03-24T22:06:47.462269+0000","last_became_active":"2021-03-24T22:05:04.447958+0000","last_became_peered":"2021-03-24T22:05:04.447958+0000","last_unstale":"2021-03-24T22:06:47.462269+0000","last_undegraded":"2021-03-24T22:06:47.462269+0000","last_fullsized":"2021-03-24T22:06:47.462269+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:59.395655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:05:59.395655+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.11","version":"838'1206","reported_seq":"3823","reported_epoch":"838","state":"active+clean","last_fresh":"2021-03-24T22:06:48.481772+0000","last_change":"2021-03-24T22:06:48.482212+0000","last_active":"2021-03-24T22:06:48.481772+0000","last_peered":"2021-03-24T22:06:48.481772+0000","last_clean":"2021-03-24T22:06:48.481772+0000","last_became_active":"2021-03-24T22:02:04.322694+0000","last_became_peered":"2021-03-24T22:02:04.322694+0000","last_unstale":"2021-03-24T22:06:48.481772+0000","last_undegraded":"2021-03-24T22:06:48.481772+0000","last_fullsized":"2021-03-24T22:06:48.481772+0000","mapping_epoch":553,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":554,"parent":"0.0","parent_split_bits":5,"last_scrub":"831'1200","last_scrub_stamp":"2021-03-24T22:06:46.160704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:06:46.160704+0000","log_size":1206,"ondisk_log_size":1206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36454689,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":243,"num_read_kb":285360,"num_write":161,"num_write_kb":70451,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":325,"num_bytes_recovered":148467814,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"255","length":"1"},{"start":"27b","length":"1"},{"start":"297","length":"1"},{"start":"2a7","length":"1"},{"start":"2ab","length":"1"}]},{"pgid":"4.11","version":"544'30","reported_seq":"1022","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.453684+0000","last_change":"2021-03-24T22:05:45.015199+0000","last_active":"2021-03-24T22:06:47.453684+0000","last_peered":"2021-03-24T22:06:47.453684+0000","last_clean":"2021-03-24T22:06:47.453684+0000","last_became_active":"2021-03-24T22:01:57.310370+0000","last_became_peered":"2021-03-24T22:01:57.310370+0000","last_unstale":"2021-03-24T22:06:47.453684+0000","last_undegraded":"2021-03-24T22:06:47.453684+0000","last_fullsized":"2021-03-24T22:06:47.453684+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'30","last_scrub_stamp":"2021-03-24T22:05:45.015001+0000","last_deep_scrub":"86'22","last_deep_scrub_stamp":"2021-03-24T21:52:54.597505+0000","last_clean_scrub_stamp":"2021-03-24T22:05:45.015001+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":41,"num_read_kb":28,"num_write":23,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":3480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":"686","reported_epoch":"835","state":"active+clean","last_fresh":"2021-03-24T22:06:45.446757+0000","last_change":"2021-03-24T22:05:43.282870+0000","last_active":"2021-03-24T22:06:45.446757+0000","last_peered":"2021-03-24T22:06:45.446757+0000","last_clean":"2021-03-24T22:06:45.446757+0000","last_became_active":"2021-03-24T22:05:04.439770+0000","last_became_peered":"2021-03-24T22:05:04.439770+0000","last_unstale":"2021-03-24T22:06:45.446757+0000","last_undegraded":"2021-03-24T22:06:45.446757+0000","last_fullsized":"2021-03-24T22:06:45.446757+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:43.282684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:05:43.282684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"303","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.471685+0000","last_change":"2021-03-24T22:04:38.228550+0000","last_active":"2021-03-24T22:06:47.471685+0000","last_peered":"2021-03-24T22:06:47.471685+0000","last_clean":"2021-03-24T22:06:47.471685+0000","last_became_active":"2021-03-24T22:01:57.307995+0000","last_became_peered":"2021-03-24T22:01:57.307995+0000","last_unstale":"2021-03-24T22:06:47.471685+0000","last_undegraded":"2021-03-24T22:06:47.471685+0000","last_fullsized":"2021-03-24T22:06:47.471685+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:04:38.228363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T22:04:38.228363+0000","last_clean_scrub_stamp":"2021-03-24T22:04:38.228363+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"812","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.455774+0000","last_change":"2021-03-24T22:05:09.243747+0000","last_active":"2021-03-24T22:06:47.455774+0000","last_peered":"2021-03-24T22:06:47.455774+0000","last_clean":"2021-03-24T22:06:47.455774+0000","last_became_active":"2021-03-24T22:01:43.389017+0000","last_became_peered":"2021-03-24T22:01:43.389017+0000","last_unstale":"2021-03-24T22:06:47.455774+0000","last_undegraded":"2021-03-24T22:06:47.455774+0000","last_fullsized":"2021-03-24T22:06:47.455774+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:09.243590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:53:47.540793+0000","last_clean_scrub_stamp":"2021-03-24T22:05:09.243590+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.12","version":"183'26","reported_seq":"966","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-24T22:06:50.701455+0000","last_change":"2021-03-24T22:06:49.732445+0000","last_active":"2021-03-24T22:06:50.701455+0000","last_peered":"2021-03-24T22:06:50.701455+0000","last_clean":"2021-03-24T22:06:50.701455+0000","last_became_active":"2021-03-24T22:01:43.392614+0000","last_became_peered":"2021-03-24T22:01:43.392614+0000","last_unstale":"2021-03-24T22:06:50.701455+0000","last_undegraded":"2021-03-24T22:06:50.701455+0000","last_fullsized":"2021-03-24T22:06:50.701455+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":5,"last_scrub":"183'26","last_scrub_stamp":"2021-03-24T22:04:25.596444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:04:25.596444+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"27b","length":"1"},{"start":"2a7","length":"1"},{"start":"2cb","length":"1"}]},{"pgid":"4.10","version":"544'16","reported_seq":"1096","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.453370+0000","last_change":"2021-03-24T22:05:47.006934+0000","last_active":"2021-03-24T22:06:47.453370+0000","last_peered":"2021-03-24T22:06:47.453370+0000","last_clean":"2021-03-24T22:06:47.453370+0000","last_became_active":"2021-03-24T22:03:45.428746+0000","last_became_peered":"2021-03-24T22:03:45.428746+0000","last_unstale":"2021-03-24T22:06:47.453370+0000","last_undegraded":"2021-03-24T22:06:47.453370+0000","last_fullsized":"2021-03-24T22:06:47.453370+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":656,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'16","last_scrub_stamp":"2021-03-24T22:05:47.006825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:46.359205+0000","last_clean_scrub_stamp":"2021-03-24T22:05:47.006825+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":36,"num_read_kb":18,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":10016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":"837","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.463772+0000","last_change":"2021-03-24T22:05:58.410296+0000","last_active":"2021-03-24T22:06:47.463772+0000","last_peered":"2021-03-24T22:06:47.463772+0000","last_clean":"2021-03-24T22:06:47.463772+0000","last_became_active":"2021-03-24T22:05:04.458069+0000","last_became_peered":"2021-03-24T22:05:04.458069+0000","last_unstale":"2021-03-24T22:06:47.463772+0000","last_undegraded":"2021-03-24T22:06:47.463772+0000","last_fullsized":"2021-03-24T22:06:47.463772+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:58.410205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:05:58.410205+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"331","reported_epoch":"839","state":"active+clean","last_fresh":"2021-03-24T22:06:49.730946+0000","last_change":"2021-03-24T22:04:40.677638+0000","last_active":"2021-03-24T22:06:49.730946+0000","last_peered":"2021-03-24T22:06:49.730946+0000","last_clean":"2021-03-24T22:06:49.730946+0000","last_became_active":"2021-03-24T22:01:57.304385+0000","last_became_peered":"2021-03-24T22:01:57.304385+0000","last_unstale":"2021-03-24T22:06:49.730946+0000","last_undegraded":"2021-03-24T22:06:49.730946+0000","last_fullsized":"2021-03-24T22:06:49.730946+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:04:40.677553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:04:40.677553+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"310","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.470730+0000","last_change":"2021-03-24T22:05:45.108061+0000","last_active":"2021-03-24T22:06:47.470730+0000","last_peered":"2021-03-24T22:06:47.470730+0000","last_clean":"2021-03-24T22:06:47.470730+0000","last_became_active":"2021-03-24T22:01:57.315718+0000","last_became_peered":"2021-03-24T22:01:57.315718+0000","last_unstale":"2021-03-24T22:06:47.470730+0000","last_undegraded":"2021-03-24T22:06:47.470730+0000","last_fullsized":"2021-03-24T22:06:47.470730+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:45.107889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:57:59.979772+0000","last_clean_scrub_stamp":"2021-03-24T22:05:45.107889+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.13","version":"837'777","reported_seq":"3544","reported_epoch":"838","state":"active+clean","last_fresh":"2021-03-24T22:06:48.474853+0000","last_change":"2021-03-24T22:06:48.475878+0000","last_active":"2021-03-24T22:06:48.474853+0000","last_peered":"2021-03-24T22:06:48.474853+0000","last_clean":"2021-03-24T22:06:48.474853+0000","last_became_active":"2021-03-24T22:02:28.318669+0000","last_became_peered":"2021-03-24T22:02:28.318669+0000","last_unstale":"2021-03-24T22:06:48.474853+0000","last_undegraded":"2021-03-24T22:06:48.474853+0000","last_fullsized":"2021-03-24T22:06:48.474853+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":5,"last_scrub":"783'727","last_scrub_stamp":"2021-03-24T22:05:55.373277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:05:55.373277+0000","log_size":777,"ondisk_log_size":777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12928677,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"255","length":"1"},{"start":"27b","length":"1"},{"start":"297","length":"1"},{"start":"2a7","length":"1"},{"start":"2ab","length":"1"}]},{"pgid":"3.18","version":"0'0","reported_seq":"836","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.453583+0000","last_change":"2021-03-24T22:05:01.161274+0000","last_active":"2021-03-24T22:06:47.453583+0000","last_peered":"2021-03-24T22:06:47.453583+0000","last_clean":"2021-03-24T22:06:47.453583+0000","last_became_active":"2021-03-24T22:03:45.428772+0000","last_became_peered":"2021-03-24T22:03:45.428772+0000","last_unstale":"2021-03-24T22:06:47.453583+0000","last_undegraded":"2021-03-24T22:06:47.453583+0000","last_fullsized":"2021-03-24T22:06:47.453583+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:01.161130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:05:01.161130+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"861","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.464440+0000","last_change":"2021-03-24T22:04:35.127105+0000","last_active":"2021-03-24T22:06:47.464440+0000","last_peered":"2021-03-24T22:06:47.464440+0000","last_clean":"2021-03-24T22:06:47.464440+0000","last_became_active":"2021-03-24T22:02:22.328021+0000","last_became_peered":"2021-03-24T22:02:22.328021+0000","last_unstale":"2021-03-24T22:06:47.464440+0000","last_undegraded":"2021-03-24T22:06:47.464440+0000","last_fullsized":"2021-03-24T22:06:47.464440+0000","mapping_epoch":571,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:04:35.126877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:59:17.040623+0000","last_clean_scrub_stamp":"2021-03-24T22:04:35.126877+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"214","reported_epoch":"838","state":"active+clean","last_fresh":"2021-03-24T22:06:48.461163+0000","last_change":"2021-03-24T22:05:45.490399+0000","last_active":"2021-03-24T22:06:48.461163+0000","last_peered":"2021-03-24T22:06:48.461163+0000","last_clean":"2021-03-24T22:06:48.461163+0000","last_became_active":"2021-03-24T22:05:04.432365+0000","last_became_peered":"2021-03-24T22:05:04.432365+0000","last_unstale":"2021-03-24T22:06:48.461163+0000","last_undegraded":"2021-03-24T22:06:48.461163+0000","last_fullsized":"2021-03-24T22:06:48.461163+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:45.490290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:05:45.490290+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1c","version":"273'245","reported_seq":"2622","reported_epoch":"838","state":"active+clean","last_fresh":"2021-03-24T22:06:48.469557+0000","last_change":"2021-03-24T22:06:48.474434+0000","last_active":"2021-03-24T22:06:48.469557+0000","last_peered":"2021-03-24T22:06:48.469557+0000","last_clean":"2021-03-24T22:06:48.469557+0000","last_became_active":"2021-03-24T22:05:04.445911+0000","last_became_peered":"2021-03-24T22:05:04.445911+0000","last_unstale":"2021-03-24T22:06:48.469557+0000","last_undegraded":"2021-03-24T22:06:48.469557+0000","last_fullsized":"2021-03-24T22:06:48.469557+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":245,"ondisk_log_size":245,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8581786,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":70,"num_read_kb":67597,"num_write":52,"num_write_kb":18055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"255","length":"1"},{"start":"27b","length":"1"},{"start":"297","length":"1"},{"start":"2a7","length":"1"},{"start":"2ab","length":"1"}]},{"pgid":"3.19","version":"0'0","reported_seq":"204","reported_epoch":"838","state":"active+clean","last_fresh":"2021-03-24T22:06:48.457250+0000","last_change":"2021-03-24T22:06:46.851738+0000","last_active":"2021-03-24T22:06:48.457250+0000","last_peered":"2021-03-24T22:06:48.457250+0000","last_clean":"2021-03-24T22:06:48.457250+0000","last_became_active":"2021-03-24T22:03:45.430602+0000","last_became_peered":"2021-03-24T22:03:45.430602+0000","last_unstale":"2021-03-24T22:06:48.457250+0000","last_undegraded":"2021-03-24T22:06:48.457250+0000","last_fullsized":"2021-03-24T22:06:48.457250+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:46.851611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:57:26.417613+0000","last_clean_scrub_stamp":"2021-03-24T22:06:46.851611+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"613'8","reported_seq":"399","reported_epoch":"839","state":"active+clean","last_fresh":"2021-03-24T22:06:49.732627+0000","last_change":"2021-03-24T22:06:00.101350+0000","last_active":"2021-03-24T22:06:49.732627+0000","last_peered":"2021-03-24T22:06:49.732627+0000","last_clean":"2021-03-24T22:06:49.732627+0000","last_became_active":"2021-03-24T22:03:03.353426+0000","last_became_peered":"2021-03-24T22:03:03.353426+0000","last_unstale":"2021-03-24T22:06:49.732627+0000","last_undegraded":"2021-03-24T22:06:49.732627+0000","last_fullsized":"2021-03-24T22:06:49.732627+0000","mapping_epoch":612,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":613,"parent":"0.0","parent_split_bits":0,"last_scrub":"613'8","last_scrub_stamp":"2021-03-24T22:06:00.101091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:06:00.101091+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"614","reported_epoch":"835","state":"active+clean","last_fresh":"2021-03-24T22:06:45.536881+0000","last_change":"2021-03-24T22:05:43.068387+0000","last_active":"2021-03-24T22:06:45.536881+0000","last_peered":"2021-03-24T22:06:45.536881+0000","last_clean":"2021-03-24T22:06:45.536881+0000","last_became_active":"2021-03-24T22:05:04.423989+0000","last_became_peered":"2021-03-24T22:05:04.423989+0000","last_unstale":"2021-03-24T22:06:45.536881+0000","last_undegraded":"2021-03-24T22:06:45.536881+0000","last_fullsized":"2021-03-24T22:06:45.536881+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:43.068130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:05:43.068130+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1d","version":"837'737","reported_seq":"3424","reported_epoch":"839","state":"active+clean","last_fresh":"2021-03-24T22:06:49.661822+0000","last_change":"2021-03-24T22:06:49.662174+0000","last_active":"2021-03-24T22:06:49.661822+0000","last_peered":"2021-03-24T22:06:49.661822+0000","last_clean":"2021-03-24T22:06:49.661822+0000","last_became_active":"2021-03-24T22:03:53.384369+0000","last_became_peered":"2021-03-24T22:03:53.384369+0000","last_unstale":"2021-03-24T22:06:49.661822+0000","last_undegraded":"2021-03-24T22:06:49.661822+0000","last_fullsized":"2021-03-24T22:06:49.661822+0000","mapping_epoch":662,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":663,"parent":"0.0","parent_split_bits":5,"last_scrub":"837'737","last_scrub_stamp":"2021-03-24T22:06:48.910932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:06:48.910932+0000","log_size":737,"ondisk_log_size":737,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10933093,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":36270,"num_write":16,"num_write_kb":6286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"255","length":"1"},{"start":"27b","length":"1"},{"start":"297","length":"1"},{"start":"2a7","length":"1"},{"start":"2ab","length":"1"},{"start":"2cb","length":"1"}]},{"pgid":"3.1a","version":"0'0","reported_seq":"199","reported_epoch":"838","state":"active+clean","last_fresh":"2021-03-24T22:06:48.457928+0000","last_change":"2021-03-24T22:05:49.560351+0000","last_active":"2021-03-24T22:06:48.457928+0000","last_peered":"2021-03-24T22:06:48.457928+0000","last_clean":"2021-03-24T22:06:48.457928+0000","last_became_active":"2021-03-24T22:03:45.431828+0000","last_became_peered":"2021-03-24T22:03:45.431828+0000","last_unstale":"2021-03-24T22:06:48.457928+0000","last_undegraded":"2021-03-24T22:06:48.457928+0000","last_fullsized":"2021-03-24T22:06:48.457928+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:49.560279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:52:54.212459+0000","last_clean_scrub_stamp":"2021-03-24T22:05:49.560279+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"378","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:48.064062+0000","last_change":"2021-03-24T22:06:48.063995+0000","last_active":"2021-03-24T22:06:48.064062+0000","last_peered":"2021-03-24T22:06:48.064062+0000","last_clean":"2021-03-24T22:06:48.064062+0000","last_became_active":"2021-03-24T22:00:49.305569+0000","last_became_peered":"2021-03-24T22:00:49.305569+0000","last_unstale":"2021-03-24T22:06:48.064062+0000","last_undegraded":"2021-03-24T22:06:48.064062+0000","last_fullsized":"2021-03-24T22:06:48.064062+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:48.063858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:06:48.063858+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"860","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.455411+0000","last_change":"2021-03-24T22:06:07.907835+0000","last_active":"2021-03-24T22:06:47.455411+0000","last_peered":"2021-03-24T22:06:47.455411+0000","last_clean":"2021-03-24T22:06:47.455411+0000","last_became_active":"2021-03-24T22:05:04.423569+0000","last_became_peered":"2021-03-24T22:05:04.423569+0000","last_unstale":"2021-03-24T22:06:47.455411+0000","last_undegraded":"2021-03-24T22:06:47.455411+0000","last_fullsized":"2021-03-24T22:06:47.455411+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:07.907636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T22:06:07.907636+0000","last_clean_scrub_stamp":"2021-03-24T22:06:07.907636+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1e","version":"265'45","reported_seq":"595","reported_epoch":"839","state":"active+clean","last_fresh":"2021-03-24T22:06:49.658753+0000","last_change":"2021-03-24T22:06:49.659055+0000","last_active":"2021-03-24T22:06:49.658753+0000","last_peered":"2021-03-24T22:06:49.658753+0000","last_clean":"2021-03-24T22:06:49.658753+0000","last_became_active":"2021-03-24T22:05:04.432331+0000","last_became_peered":"2021-03-24T22:05:04.432331+0000","last_unstale":"2021-03-24T22:06:49.658753+0000","last_undegraded":"2021-03-24T22:06:49.658753+0000","last_fullsized":"2021-03-24T22:06:49.658753+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":5,"last_scrub":"265'45","last_scrub_stamp":"2021-03-24T22:05:55.541936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:05:55.541936+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"255","length":"1"},{"start":"27b","length":"1"},{"start":"297","length":"1"},{"start":"2a7","length":"1"},{"start":"2ab","length":"1"},{"start":"2cb","length":"1"}]},{"pgid":"3.1b","version":"0'0","reported_seq":"861","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.462746+0000","last_change":"2021-03-24T22:05:51.442642+0000","last_active":"2021-03-24T22:06:47.462746+0000","last_peered":"2021-03-24T22:06:47.462746+0000","last_clean":"2021-03-24T22:06:47.462746+0000","last_became_active":"2021-03-24T22:03:32.380554+0000","last_became_peered":"2021-03-24T22:03:32.380554+0000","last_unstale":"2021-03-24T22:06:47.462746+0000","last_undegraded":"2021-03-24T22:06:47.462746+0000","last_fullsized":"2021-03-24T22:06:47.462746+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:51.442473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:05:51.442473+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"204","reported_epoch":"838","state":"active+clean","last_fresh":"2021-03-24T22:06:48.460984+0000","last_change":"2021-03-24T22:06:38.755278+0000","last_active":"2021-03-24T22:06:48.460984+0000","last_peered":"2021-03-24T22:06:48.460984+0000","last_clean":"2021-03-24T22:06:48.460984+0000","last_became_active":"2021-03-24T22:03:45.433015+0000","last_became_peered":"2021-03-24T22:03:45.433015+0000","last_unstale":"2021-03-24T22:06:48.460984+0000","last_undegraded":"2021-03-24T22:06:48.460984+0000","last_fullsized":"2021-03-24T22:06:48.460984+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:38.755184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:06:38.755184+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"330","reported_epoch":"839","state":"active+clean","last_fresh":"2021-03-24T22:06:49.731637+0000","last_change":"2021-03-24T22:05:08.570430+0000","last_active":"2021-03-24T22:06:49.731637+0000","last_peered":"2021-03-24T22:06:49.731637+0000","last_clean":"2021-03-24T22:06:49.731637+0000","last_became_active":"2021-03-24T22:05:04.431178+0000","last_became_peered":"2021-03-24T22:05:04.431178+0000","last_unstale":"2021-03-24T22:06:49.731637+0000","last_undegraded":"2021-03-24T22:06:49.731637+0000","last_fullsized":"2021-03-24T22:06:49.731637+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:08.570238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:05:08.570238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1f","version":"272'99","reported_seq":"961","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-24T22:06:50.702094+0000","last_change":"2021-03-24T22:06:49.733563+0000","last_active":"2021-03-24T22:06:50.702094+0000","last_peered":"2021-03-24T22:06:50.702094+0000","last_clean":"2021-03-24T22:06:50.702094+0000","last_became_active":"2021-03-24T22:01:43.385324+0000","last_became_peered":"2021-03-24T22:01:43.385324+0000","last_unstale":"2021-03-24T22:06:50.702094+0000","last_undegraded":"2021-03-24T22:06:50.702094+0000","last_fullsized":"2021-03-24T22:06:50.702094+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":5,"last_scrub":"272'99","last_scrub_stamp":"2021-03-24T22:03:57.739288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:03:57.739288+0000","log_size":99,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"27b","length":"1"},{"start":"2a7","length":"1"},{"start":"2cb","length":"1"}]},{"pgid":"3.1c","version":"0'0","reported_seq":"214","reported_epoch":"838","state":"active+clean","last_fresh":"2021-03-24T22:06:48.456976+0000","last_change":"2021-03-24T22:05:05.665557+0000","last_active":"2021-03-24T22:06:48.456976+0000","last_peered":"2021-03-24T22:06:48.456976+0000","last_clean":"2021-03-24T22:06:48.456976+0000","last_became_active":"2021-03-24T22:05:04.453666+0000","last_became_peered":"2021-03-24T22:05:04.453666+0000","last_unstale":"2021-03-24T22:06:48.456976+0000","last_undegraded":"2021-03-24T22:06:48.456976+0000","last_fullsized":"2021-03-24T22:06:48.456976+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:05.665337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:05:05.665337+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"326","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.470892+0000","last_change":"2021-03-24T22:06:34.008599+0000","last_active":"2021-03-24T22:06:47.470892+0000","last_peered":"2021-03-24T22:06:47.470892+0000","last_clean":"2021-03-24T22:06:47.470892+0000","last_became_active":"2021-03-24T22:03:45.434093+0000","last_became_peered":"2021-03-24T22:03:45.434093+0000","last_unstale":"2021-03-24T22:06:47.470892+0000","last_undegraded":"2021-03-24T22:06:47.470892+0000","last_fullsized":"2021-03-24T22:06:47.470892+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:34.008428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:56:51.977306+0000","last_clean_scrub_stamp":"2021-03-24T22:06:34.008428+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"664","reported_epoch":"835","state":"active+clean","last_fresh":"2021-03-24T22:06:45.444065+0000","last_change":"2021-03-24T22:06:34.336607+0000","last_active":"2021-03-24T22:06:45.444065+0000","last_peered":"2021-03-24T22:06:45.444065+0000","last_clean":"2021-03-24T22:06:45.444065+0000","last_became_active":"2021-03-24T22:05:04.437068+0000","last_became_peered":"2021-03-24T22:05:04.437068+0000","last_unstale":"2021-03-24T22:06:45.444065+0000","last_undegraded":"2021-03-24T22:06:45.444065+0000","last_fullsized":"2021-03-24T22:06:45.444065+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:34.336429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T22:05:15.154493+0000","last_clean_scrub_stamp":"2021-03-24T22:06:34.336429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.18","version":"838'306","reported_seq":"2396","reported_epoch":"838","state":"active+clean","last_fresh":"2021-03-24T22:06:48.473948+0000","last_change":"2021-03-24T22:06:48.474565+0000","last_active":"2021-03-24T22:06:48.473948+0000","last_peered":"2021-03-24T22:06:48.473948+0000","last_clean":"2021-03-24T22:06:48.473948+0000","last_became_active":"2021-03-24T22:02:00.304128+0000","last_became_peered":"2021-03-24T22:02:00.304128+0000","last_unstale":"2021-03-24T22:06:48.473948+0000","last_undegraded":"2021-03-24T22:06:48.473948+0000","last_fullsized":"2021-03-24T22:06:48.473948+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":5,"last_scrub":"831'304","last_scrub_stamp":"2021-03-24T22:06:43.544396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:06:43.544396+0000","log_size":306,"ondisk_log_size":306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7109856,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":54,"num_read_kb":47612,"num_write":36,"num_write_kb":12535,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":32240200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"255","length":"1"},{"start":"27b","length":"1"},{"start":"297","length":"1"},{"start":"2a7","length":"1"},{"start":"2ab","length":"1"}]},{"pgid":"3.1d","version":"0'0","reported_seq":"882","reported_epoch":"835","state":"active+clean","last_fresh":"2021-03-24T22:06:45.538010+0000","last_change":"2021-03-24T22:05:48.075900+0000","last_active":"2021-03-24T22:06:45.538010+0000","last_peered":"2021-03-24T22:06:45.538010+0000","last_clean":"2021-03-24T22:06:45.538010+0000","last_became_active":"2021-03-24T22:05:04.430681+0000","last_became_peered":"2021-03-24T22:05:04.430681+0000","last_unstale":"2021-03-24T22:06:45.538010+0000","last_undegraded":"2021-03-24T22:06:45.538010+0000","last_fullsized":"2021-03-24T22:06:45.538010+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:48.075822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:05:48.075822+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"658","reported_epoch":"835","state":"active+clean","last_fresh":"2021-03-24T22:06:45.445587+0000","last_change":"2021-03-24T22:05:52.372115+0000","last_active":"2021-03-24T22:06:45.445587+0000","last_peered":"2021-03-24T22:06:45.445587+0000","last_clean":"2021-03-24T22:06:45.445587+0000","last_became_active":"2021-03-24T22:05:04.432965+0000","last_became_peered":"2021-03-24T22:05:04.432965+0000","last_unstale":"2021-03-24T22:06:45.445587+0000","last_undegraded":"2021-03-24T22:06:45.445587+0000","last_fullsized":"2021-03-24T22:06:45.445587+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:52.372027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:05:52.372027+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"873","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.463334+0000","last_change":"2021-03-24T22:06:02.476829+0000","last_active":"2021-03-24T22:06:47.463334+0000","last_peered":"2021-03-24T22:06:47.463334+0000","last_clean":"2021-03-24T22:06:47.463334+0000","last_became_active":"2021-03-24T22:03:45.431424+0000","last_became_peered":"2021-03-24T22:03:45.431424+0000","last_unstale":"2021-03-24T22:06:47.463334+0000","last_undegraded":"2021-03-24T22:06:47.463334+0000","last_fullsized":"2021-03-24T22:06:47.463334+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:02.476582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:06:02.476582+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.19","version":"838'1295","reported_seq":"3942","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-24T22:06:50.703482+0000","last_change":"2021-03-24T22:06:49.732654+0000","last_active":"2021-03-24T22:06:50.703482+0000","last_peered":"2021-03-24T22:06:50.703482+0000","last_clean":"2021-03-24T22:06:50.703482+0000","last_became_active":"2021-03-24T22:02:07.304440+0000","last_became_peered":"2021-03-24T22:02:07.304440+0000","last_unstale":"2021-03-24T22:06:50.703482+0000","last_undegraded":"2021-03-24T22:06:50.703482+0000","last_fullsized":"2021-03-24T22:06:50.703482+0000","mapping_epoch":556,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":557,"parent":"0.0","parent_split_bits":5,"last_scrub":"838'1295","last_scrub_stamp":"2021-03-24T22:06:49.184837+0000","last_deep_scrub":"455'480","last_deep_scrub_stamp":"2021-03-24T22:00:26.193730+0000","last_clean_scrub_stamp":"2021-03-24T22:06:49.184837+0000","log_size":1295,"ondisk_log_size":1295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34358289,"num_objects":16,"num_object_clones":13,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":253,"num_read_kb":216921,"num_write":236,"num_write_kb":95611,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":265,"num_bytes_recovered":154749512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"27b","length":"1"},{"start":"2a7","length":"1"},{"start":"2cb","length":"1"}]},{"pgid":"3.1e","version":"0'0","reported_seq":"860","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.455293+0000","last_change":"2021-03-24T22:04:27.240608+0000","last_active":"2021-03-24T22:06:47.455293+0000","last_peered":"2021-03-24T22:06:47.455293+0000","last_clean":"2021-03-24T22:06:47.455293+0000","last_became_active":"2021-03-24T22:01:43.381824+0000","last_became_peered":"2021-03-24T22:01:43.381824+0000","last_unstale":"2021-03-24T22:06:47.455293+0000","last_undegraded":"2021-03-24T22:06:47.455293+0000","last_fullsized":"2021-03-24T22:06:47.455293+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:04:27.240496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:04:27.240496+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"338","reported_epoch":"839","state":"active+clean","last_fresh":"2021-03-24T22:06:49.730953+0000","last_change":"2021-03-24T22:05:49.398497+0000","last_active":"2021-03-24T22:06:49.730953+0000","last_peered":"2021-03-24T22:06:49.730953+0000","last_clean":"2021-03-24T22:06:49.730953+0000","last_became_active":"2021-03-24T22:03:45.416917+0000","last_became_peered":"2021-03-24T22:03:45.416917+0000","last_unstale":"2021-03-24T22:06:49.730953+0000","last_undegraded":"2021-03-24T22:06:49.730953+0000","last_fullsized":"2021-03-24T22:06:49.730953+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:49.398341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:48.384232+0000","last_clean_scrub_stamp":"2021-03-24T22:05:49.398341+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"868","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.455226+0000","last_change":"2021-03-24T22:03:08.392175+0000","last_active":"2021-03-24T22:06:47.455226+0000","last_peered":"2021-03-24T22:06:47.455226+0000","last_clean":"2021-03-24T22:06:47.455226+0000","last_became_active":"2021-03-24T22:01:57.310369+0000","last_became_peered":"2021-03-24T22:01:57.310369+0000","last_unstale":"2021-03-24T22:06:47.455226+0000","last_undegraded":"2021-03-24T22:06:47.455226+0000","last_fullsized":"2021-03-24T22:06:47.455226+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:08.392060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:03:08.392060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"838'440","reported_seq":"2885","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-24T22:06:50.702217+0000","last_change":"2021-03-24T22:06:49.736404+0000","last_active":"2021-03-24T22:06:50.702217+0000","last_peered":"2021-03-24T22:06:50.702217+0000","last_clean":"2021-03-24T22:06:50.702217+0000","last_became_active":"2021-03-24T22:05:04.454694+0000","last_became_peered":"2021-03-24T22:05:04.454694+0000","last_unstale":"2021-03-24T22:06:50.702217+0000","last_undegraded":"2021-03-24T22:06:50.702217+0000","last_fullsized":"2021-03-24T22:06:50.702217+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":5,"last_scrub":"821'428","last_scrub_stamp":"2021-03-24T22:06:35.291286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:06:35.291286+0000","log_size":440,"ondisk_log_size":440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13769857,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":3579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"27b","length":"1"},{"start":"2a7","length":"1"},{"start":"2cb","length":"1"}]},{"pgid":"3.1f","version":"0'0","reported_seq":"860","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.463967+0000","last_change":"2021-03-24T22:05:48.437882+0000","last_active":"2021-03-24T22:06:47.463967+0000","last_peered":"2021-03-24T22:06:47.463967+0000","last_clean":"2021-03-24T22:06:47.463967+0000","last_became_active":"2021-03-24T22:05:04.457596+0000","last_became_peered":"2021-03-24T22:05:04.457596+0000","last_unstale":"2021-03-24T22:06:47.463967+0000","last_undegraded":"2021-03-24T22:06:47.463967+0000","last_fullsized":"2021-03-24T22:06:47.463967+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:48.437760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:05:48.437760+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"214","reported_epoch":"838","state":"active+clean","last_fresh":"2021-03-24T22:06:48.458776+0000","last_change":"2021-03-24T22:05:11.459315+0000","last_active":"2021-03-24T22:06:48.458776+0000","last_peered":"2021-03-24T22:06:48.458776+0000","last_clean":"2021-03-24T22:06:48.458776+0000","last_became_active":"2021-03-24T22:05:04.447664+0000","last_became_peered":"2021-03-24T22:05:04.447664+0000","last_unstale":"2021-03-24T22:06:48.458776+0000","last_undegraded":"2021-03-24T22:06:48.458776+0000","last_fullsized":"2021-03-24T22:06:48.458776+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:11.459219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:14.710709+0000","last_clean_scrub_stamp":"2021-03-24T22:05:11.459219+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"761","reported_epoch":"835","state":"active+clean","last_fresh":"2021-03-24T22:06:45.445886+0000","last_change":"2021-03-24T22:05:46.303724+0000","last_active":"2021-03-24T22:06:45.445886+0000","last_peered":"2021-03-24T22:06:45.445886+0000","last_clean":"2021-03-24T22:06:45.445886+0000","last_became_active":"2021-03-24T22:05:04.443147+0000","last_became_peered":"2021-03-24T22:05:04.443147+0000","last_unstale":"2021-03-24T22:06:45.445886+0000","last_undegraded":"2021-03-24T22:06:45.445886+0000","last_fullsized":"2021-03-24T22:06:45.445886+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:05:46.303541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:50.731137+0000","last_clean_scrub_stamp":"2021-03-24T22:05:46.303541+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1b","version":"838'2076","reported_seq":"4444","reported_epoch":"838","state":"active+clean","last_fresh":"2021-03-24T22:06:48.697724+0000","last_change":"2021-03-24T22:06:48.498973+0000","last_active":"2021-03-24T22:06:48.697724+0000","last_peered":"2021-03-24T22:06:48.697724+0000","last_clean":"2021-03-24T22:06:48.697724+0000","last_became_active":"2021-03-24T21:57:33.176110+0000","last_became_peered":"2021-03-24T21:57:33.176110+0000","last_unstale":"2021-03-24T22:06:48.697724+0000","last_undegraded":"2021-03-24T22:06:48.697724+0000","last_fullsized":"2021-03-24T22:06:48.697724+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":5,"last_scrub":"710'1638","last_scrub_stamp":"2021-03-24T22:04:41.216144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:04:41.216144+0000","log_size":2076,"ondisk_log_size":2076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":74481633,"num_objects":31,"num_object_clones":25,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":1,"num_read":347,"num_read_kb":366437,"num_write":267,"num_write_kb":117736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"255","length":"1"},{"start":"27b","length":"1"},{"start":"297","length":"1"},{"start":"2a7","length":"1"},{"start":"2ab","length":"1"}]},{"pgid":"3.24","version":"0'0","reported_seq":"466","reported_epoch":"837","state":"active+clean","last_fresh":"2021-03-24T22:06:47.455030+0000","last_change":"2021-03-24T22:03:41.269971+0000","last_active":"2021-03-24T22:06:47.455030+0000","last_peered":"2021-03-24T22:06:47.455030+0000","last_clean":"2021-03-24T22:06:47.455030+0000","last_became_active":"2021-03-24T21:59:14.369606+0000","last_became_peered":"2021-03-24T21:59:14.369606+0000","last_unstale":"2021-03-24T22:06:47.455030+0000","last_undegraded":"2021-03-24T22:06:47.455030+0000","last_fullsized":"2021-03-24T22:06:47.455030+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:03:41.269900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:03:41.269900+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"175'22","reported_seq":"2008","reported_epoch":"835","state":"active+clean","last_fresh":"2021-03-24T22:06:45.539055+0000","last_change":"2021-03-24T22:06:43.172439+0000","last_active":"2021-03-24T22:06:45.539055+0000","last_peered":"2021-03-24T22:06:45.539055+0000","last_clean":"2021-03-24T22:06:45.539055+0000","last_became_active":"2021-03-24T22:03:45.432211+0000","last_became_peered":"2021-03-24T22:03:45.432211+0000","last_unstale":"2021-03-24T22:06:45.539055+0000","last_undegraded":"2021-03-24T22:06:45.539055+0000","last_fullsized":"2021-03-24T22:06:45.539055+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":6,"last_scrub":"175'22","last_scrub_stamp":"2021-03-24T22:06:43.172323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T22:06:43.172323+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"255","length":"1"},{"start":"297","length":"1"},{"start":"2ab","length":"1"}]},{"pgid":"3.25","version":"0'0","reported_seq":"204","reported_epoch":"838","state":"active+clean","last_fresh":"2021-03-24T22:06:48.460901+0000","last_change":"2021-03-24T22:06:36.792829+0000","last_active":"2021-03-24T22:06:48.460901+0000","last_peered":"2021-03-24T22:06:48.460901+0000","last_clean":"2021-03-24T22:06:48.460901+0000","last_became_active":"2021-03-24T22:03:45.426841+0000","last_became_peered":"2021-03-24T22:03:45.426841+0000","last_unstale":"2021-03-24T22:06:48.460901+0000","last_undegraded":"2021-03-24T22:06:48.460901+0000","last_fullsized":"2021-03-24T22:06:48.460901+0000","mapping_epoch":654,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":655,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T22:06:36.792559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:51:44.337765+0000","last_clean_scrub_stamp":"2021-03-24T22:06:36.792559+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.21","version":"838'857","reported_seq":"3515","reported_epoch":"838","state":"active+clean","last_fresh":"2021-03-24T22:06:48.504594+0000","last_change":"2021-03-24T22:06:48.498233+0000","last_active":"2021-03-24T22:06:48.504594+0000","last_peered":"2021-03-24T22:06:48.504594+0000","last_clean":"2021-03-24T22:06:48.504594+0000","last_became_active":"2021-03-24T22:05:04.434209+0000","last_became_peered":"2021-03-24T22:05:04.434209+0000","last_unstale":"2021-03-24T22:06:48.504594+0000","last_undegraded":"2021-03-24T22:06:48.504594+0000","last_fullsized":"2021-03-24T22:06:48.504594+0000","mapping_epoch":733,"log_start":"0'0","ondisk_log_start":"0'0","created":275,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T21:54:21.134778+0000","last_clean_scrub_stamp":"2021-03-24T21:54:21.134778+0000","log_size":857,"ondisk_log_size":857,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40382513,"num_objects":17,"num_object_clones":16,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":229,"num_read_kb":218577,"num_write":186,"num_write_kb":74216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":68248426,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"255","length":"1"},{"start":"27b","length":"1"},{"start":"297","length":"1"},{"start":"2a7","length":"1"},{"start":"2ab","length":"1"}]}],"pool_stats":[{"poolid":7,"num_pg":36,"stat_sum":{"num_bytes":560925232,"num_objects":239,"num_object_clones":188,"num_object_copies":717,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":239,"num_whiteouts":9,"num_read":3168,"num_read_kb":3321871,"num_write":2300,"num_write_kb":893093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2178,"num_bytes_recovered":1222469600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1249722368,"data_stored":1688139161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20848,"ondisk_log_size":20848,"up":108,"acting":108,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":363,"num_read_kb":358,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":3,"num_pg":42,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10235,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":126,"acting":126,"num_store_stats":8},{"poolid":4,"num_pg":22,"stat_sum":{"num_bytes":23524,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1475,"num_read_kb":980,"num_write":828,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1194,"num_bytes_recovered":94630,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966080,"data_stored":70572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":514,"ondisk_log_size":514,"up":66,"acting":66,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201465,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":263912,"kb_used_data":209696,"kb_used_omap":0,"kb_used_meta":54208,"kb_avail":93481240,"statfs":{"total":95995035648,"available":95724789760,"internally_reserved":0,"allocated":214728704,"data_stored":295086602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":55508992},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 22:06:17 2021","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":2.5150000000000001,"15min":1.4510000000000001},"min":{"1min":0.57699999999999996,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.7410000000000001,"5min":150.68000000000001,"15min":150.68000000000001},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.97099999999999997,"5min":2.5699999999999998,"15min":1.5429999999999999},"min":{"1min":0.52300000000000002,"5min":0.36299999999999999,"15min":0.27000000000000002},"max":{"1min":2.2639999999999998,"5min":153.91,"15min":153.91},"last":1.155}]},{"osd":1,"last update":"Wed Mar 24 22:06:17 2021","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":0.83999999999999997,"15min":0.88500000000000001},"min":{"1min":0.59599999999999997,"5min":0.318,"15min":0.308},"max":{"1min":1.8180000000000001,"5min":1.8180000000000001,"15min":5.7750000000000004},"last":1.0249999999999999},{"interface":"front","average":{"1min":1.0089999999999999,"5min":0.878,"15min":0.94999999999999996},"min":{"1min":0.61699999999999999,"5min":0.316,"15min":0.316},"max":{"1min":1.623,"5min":1.623,"15min":6.1849999999999996},"last":2.23}]},{"osd":2,"last update":"Wed Mar 24 22:06:17 2021","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.83999999999999997,"15min":0.98099999999999998},"min":{"1min":0.69299999999999995,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.982,"5min":1.982,"15min":12.135999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":1.0449999999999999,"5min":0.92500000000000004,"15min":1.0149999999999999},"min":{"1min":0.56399999999999995,"5min":0.45900000000000002,"15min":0.42999999999999999},"max":{"1min":1.881,"5min":1.881,"15min":12.178000000000001},"last":1.1240000000000001}]},{"osd":3,"last update":"Wed Mar 24 22:06:17 2021","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":0.998,"15min":1.0640000000000001},"min":{"1min":0.81699999999999995,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":2.04,"5min":6.1449999999999996,"15min":12.529},"last":0.90500000000000003},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.0329999999999999,"15min":1.081},"min":{"1min":0.66700000000000004,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.9159999999999999,"5min":3.0510000000000002,"15min":12.666},"last":1.2929999999999999}]},{"osd":4,"last update":"Wed Mar 24 22:06:17 2021","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":0.96199999999999997,"15min":0.98799999999999999},"min":{"1min":0.80600000000000005,"5min":0.34799999999999998,"15min":0.33500000000000002},"max":{"1min":1.714,"5min":1.714,"15min":6.2089999999999996},"last":1.0569999999999999},{"interface":"front","average":{"1min":1.1160000000000001,"5min":1.012,"15min":1.0720000000000001},"min":{"1min":0.89400000000000002,"5min":0.42399999999999999,"15min":0.41399999999999998},"max":{"1min":2.1360000000000001,"5min":2.1360000000000001,"15min":12.244},"last":1.53}]},{"osd":5,"last update":"Wed Mar 24 22:06:07 2021","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":1.212},{"interface":"front","average":{"1min":1.149,"5min":1.149,"15min":1.149},"min":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":1.444,"5min":1.444,"15min":1.444},"last":1.238}]},{"osd":6,"last update":"Wed Mar 24 22:06:17 2021","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.101,"15min":1.137},"min":{"1min":0.72299999999999998,"5min":0.51800000000000002,"15min":0.40899999999999997},"max":{"1min":1.7370000000000001,"5min":1.7370000000000001,"15min":6.1159999999999997},"last":1.3680000000000001},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.111,"15min":1.165},"min":{"1min":0.80400000000000005,"5min":0.502,"15min":0.434},"max":{"1min":2.0830000000000002,"5min":2.0830000000000002,"15min":12.287000000000001},"last":1.4019999999999999}]}]},{"osd":6,"up_from":47,"seq":201863463102,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":193684,"kb_used_data":153484,"kb_used_omap":0,"kb_used_meta":40192,"kb_avail":93551468,"statfs":{"total":95995035648,"available":95796703232,"internally_reserved":0,"allocated":157167616,"data_stored":210904944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41156608},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 22:06:45 2021","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":1.0669999999999999,"15min":1.002},"min":{"1min":0.41799999999999998,"5min":0.32900000000000001,"15min":0.30399999999999999},"max":{"1min":1.2370000000000001,"5min":19.257999999999999,"15min":19.257999999999999},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.88400000000000001,"5min":1.0720000000000001,"15min":0.98799999999999999},"min":{"1min":0.38100000000000001,"5min":0.313,"15min":0.313},"max":{"1min":1.488,"5min":16.795000000000002,"15min":16.795000000000002},"last":0.59999999999999998}]},{"osd":1,"last update":"Wed Mar 24 22:06:45 2021","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":1.0940000000000001,"15min":0.98999999999999999},"min":{"1min":0.58099999999999996,"5min":0.38400000000000001,"15min":0.34799999999999998},"max":{"1min":1.1319999999999999,"5min":21.995999999999999,"15min":21.995999999999999},"last":0.749},{"interface":"front","average":{"1min":0.90800000000000003,"5min":1.2689999999999999,"15min":1.091},"min":{"1min":0.47999999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.2909999999999999,"5min":26.762,"15min":26.762},"last":0.85899999999999999}]},{"osd":2,"last update":"Wed Mar 24 22:06:45 2021","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":1.1180000000000001,"15min":1.0109999999999999},"min":{"1min":0.57599999999999996,"5min":0.42399999999999999,"15min":0.35299999999999998},"max":{"1min":1.393,"5min":13.044,"15min":13.044},"last":1.0580000000000001},{"interface":"front","average":{"1min":0.93400000000000005,"5min":1.3240000000000001,"15min":1.093},"min":{"1min":0.47099999999999997,"5min":0.41199999999999998,"15min":0.33500000000000002},"max":{"1min":1.5740000000000001,"5min":31.375,"15min":31.375},"last":0.88500000000000001}]},{"osd":3,"last update":"Wed Mar 24 22:06:45 2021","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":1.2989999999999999,"15min":1.0820000000000001},"min":{"1min":0.71599999999999997,"5min":0.47899999999999998,"15min":0.38400000000000001},"max":{"1min":1.0600000000000001,"5min":22.07,"15min":22.07},"last":1.079},{"interface":"front","average":{"1min":1.006,"5min":1.131,"15min":1.109},"min":{"1min":0.66400000000000003,"5min":0.48599999999999999,"15min":0.41299999999999998},"max":{"1min":1.4039999999999999,"5min":11.583,"15min":11.583},"last":0.99299999999999999}]},{"osd":4,"last update":"Wed Mar 24 22:06:45 2021","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":1.0720000000000001,"15min":1.0309999999999999},"min":{"1min":0.63400000000000001,"5min":0.44700000000000001,"15min":0.27200000000000002},"max":{"1min":1.4379999999999999,"5min":8.4350000000000005,"15min":8.4350000000000005},"last":0.80400000000000005},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.264,"15min":1.139},"min":{"1min":0.69999999999999996,"5min":0.46700000000000003,"15min":0.42199999999999999},"max":{"1min":1.427,"5min":22.033000000000001,"15min":22.033000000000001},"last":0.92500000000000004}]},{"osd":5,"last update":"Wed Mar 24 22:06:04 2021","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.087,"15min":1.087},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.976,"5min":1.976,"15min":1.976},"last":1.167},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":1.0229999999999999}]},{"osd":7,"last update":"Wed Mar 24 22:06:07 2021","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.1719999999999999,"15min":1.1899999999999999},"min":{"1min":0.57299999999999995,"5min":0.46999999999999997,"15min":0.30499999999999999},"max":{"1min":1.403,"5min":8.3599999999999994,"15min":9.5909999999999993},"last":1.115},{"interface":"front","average":{"1min":1.135,"5min":1.159,"15min":1.1779999999999999},"min":{"1min":0.59099999999999997,"5min":0.51300000000000001,"15min":0.442},"max":{"1min":1.353,"5min":8.7959999999999994,"15min":8.7959999999999994},"last":1.1379999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575059,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":112812,"kb_used_data":82980,"kb_used_omap":0,"kb_used_meta":29824,"kb_avail":93632340,"statfs":{"total":95995035648,"available":95879516160,"internally_reserved":0,"allocated":84971520,"data_stored":105379095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":30539776},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 22:06:03 2021","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.82499999999999996,"15min":0.83299999999999996},"min":{"1min":0.32700000000000001,"5min":0.27700000000000002,"15min":0.20899999999999999},"max":{"1min":2.0110000000000001,"5min":11.189,"15min":21.085000000000001},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.92400000000000004,"15min":0.80700000000000005},"min":{"1min":0.35799999999999998,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.7430000000000001,"5min":8.5180000000000007,"15min":8.5180000000000007},"last":0.80800000000000005}]},{"osd":2,"last update":"Wed Mar 24 22:06:11 2021","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.77600000000000002,"15min":0.70399999999999996},"min":{"1min":0.38600000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.792,"5min":9.9870000000000001,"15min":9.9870000000000001},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.97899999999999998,"15min":0.85099999999999998},"min":{"1min":0.40400000000000003,"5min":0.29599999999999999,"15min":0.27200000000000002},"max":{"1min":1.415,"5min":9.2059999999999995,"15min":9.2059999999999995},"last":0.95999999999999996}]},{"osd":3,"last update":"Wed Mar 24 22:06:46 2021","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":0.96599999999999997,"15min":0.91500000000000004},"min":{"1min":0.56699999999999995,"5min":0.37,"15min":0.29299999999999998},"max":{"1min":1.7410000000000001,"5min":5.468,"15min":8.5990000000000002},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.069,"15min":0.98599999999999999},"min":{"1min":0.5,"5min":0.33800000000000002,"15min":0.316},"max":{"1min":2.0099999999999998,"5min":15.775,"15min":15.775},"last":0.998}]},{"osd":4,"last update":"Wed Mar 24 22:06:11 2021","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":1.0800000000000001,"15min":0.98899999999999999},"min":{"1min":0.63,"5min":0.49099999999999999,"15min":0.34399999999999997},"max":{"1min":1.2969999999999999,"5min":9.6150000000000002,"15min":9.6150000000000002},"last":1.0620000000000001},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.075,"15min":1.0629999999999999},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.35599999999999998},"max":{"1min":1.5209999999999999,"5min":3.9359999999999999,"15min":3.9359999999999999},"last":1.087}]},{"osd":5,"last update":"Wed Mar 24 22:06:07 2021","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.6379999999999999,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":1.26},{"interface":"front","average":{"1min":1.1719999999999999,"5min":1.1719999999999999,"15min":1.1719999999999999},"min":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"max":{"1min":2.0750000000000002,"5min":2.0750000000000002,"15min":2.0750000000000002},"last":1.236}]},{"osd":6,"last update":"Wed Mar 24 22:06:46 2021","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.2869999999999999,"15min":1.1970000000000001},"min":{"1min":0.878,"5min":0.49399999999999999,"15min":0.48799999999999999},"max":{"1min":1.6279999999999999,"5min":11.254,"15min":11.254},"last":1.1080000000000001},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.218,"15min":1.1279999999999999},"min":{"1min":0.70499999999999996,"5min":0.55600000000000005,"15min":0.38100000000000001},"max":{"1min":1.6020000000000001,"5min":9.7300000000000004,"15min":9.7300000000000004},"last":1.2030000000000001}]},{"osd":7,"last update":"Wed Mar 24 22:06:11 2021","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.159,"15min":1.177},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.38300000000000001},"max":{"1min":1.706,"5min":3.8439999999999999,"15min":4.4699999999999998},"last":1.367},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.2909999999999999,"15min":1.2250000000000001},"min":{"1min":0.70799999999999996,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":2.105,"5min":11.289,"15min":11.289},"last":1.3480000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738582,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":210584,"kb_used_data":157264,"kb_used_omap":0,"kb_used_meta":53312,"kb_avail":93534568,"statfs":{"total":95995035648,"available":95779397632,"internally_reserved":0,"allocated":161038336,"data_stored":212934407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":54591488},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Mar 24 22:06:06 2021","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.754,"15min":0.70199999999999996},"min":{"1min":0.253,"5min":0.253,"15min":0.249},"max":{"1min":1.0429999999999999,"5min":5.0419999999999998,"15min":5.0419999999999998},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.71299999999999997,"15min":0.76000000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.0660000000000001,"5min":5.4320000000000004,"15min":8.8989999999999991},"last":0.34499999999999997}]},{"osd":2,"last update":"Wed Mar 24 22:06:25 2021","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.68300000000000005,"15min":0.77700000000000002},"min":{"1min":0.47799999999999998,"5min":0.308,"15min":0.28999999999999998},"max":{"1min":0.88700000000000001,"5min":3.8980000000000001,"15min":15.949999999999999},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.753,"5min":0.70099999999999996,"15min":0.77400000000000002},"min":{"1min":0.40400000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.0960000000000001,"5min":1.3580000000000001,"15min":4.766},"last":0.67100000000000004}]},{"osd":3,"last update":"Wed Mar 24 22:06:44 2021","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":1.123,"15min":0.95499999999999996},"min":{"1min":0.48799999999999999,"5min":0.33800000000000002,"15min":0.28999999999999998},"max":{"1min":1.528,"5min":24.75,"15min":24.75},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.89700000000000002,"5min":1.1439999999999999,"15min":1.006},"min":{"1min":0.55500000000000005,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":1.5529999999999999,"5min":24.832999999999998,"15min":24.832999999999998},"last":0.50600000000000001}]},{"osd":4,"last update":"Wed Mar 24 22:06:06 2021","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":0.95599999999999996,"15min":1.0369999999999999},"min":{"1min":0.45500000000000002,"5min":0.373,"15min":0.373},"max":{"1min":1.3360000000000001,"5min":1.4790000000000001,"15min":5.0330000000000004},"last":1.0209999999999999},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.111,"15min":1.0920000000000001},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.44900000000000001},"max":{"1min":1.663,"5min":12.366,"15min":12.366},"last":1.038}]},{"osd":5,"last update":"Wed Mar 24 22:06:06 2021","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.522,"5min":1.522,"15min":1.522},"last":0.73799999999999999},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.746,"5min":1.746,"15min":1.746},"last":0.86699999999999999}]},{"osd":6,"last update":"Wed Mar 24 22:06:44 2021","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.032,"15min":1.1399999999999999},"min":{"1min":0.73199999999999998,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.696,"5min":3.9529999999999998,"15min":15.983000000000001},"last":0.80900000000000005},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.1499999999999999,"15min":1.1499999999999999},"min":{"1min":0.70199999999999996,"5min":0.55200000000000005,"15min":0.53400000000000003},"max":{"1min":1.6379999999999999,"5min":5.2699999999999996,"15min":5.2699999999999996},"last":0.93400000000000005}]},{"osd":7,"last update":"Wed Mar 24 22:06:16 2021","interfaces":[{"interface":"back","average":{"1min":1.2170000000000001,"5min":1.127,"15min":1.1639999999999999},"min":{"1min":0.67600000000000005,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.8100000000000001,"5min":4.9470000000000001,"15min":6.1319999999999997},"last":0.82299999999999995},{"interface":"front","average":{"1min":1.254,"5min":1.077,"15min":1.1850000000000001},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":2.6890000000000001,"5min":3.984,"15min":16.007000000000001},"last":0.76200000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411535,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":234056,"kb_used_data":180928,"kb_used_omap":0,"kb_used_meta":53120,"kb_avail":93511096,"statfs":{"total":95995035648,"available":95755362304,"internally_reserved":0,"allocated":185270272,"data_stored":245609980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":54394880},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 22:06:18 2021","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.80900000000000005,"15min":0.76000000000000001},"min":{"1min":0.32400000000000001,"5min":0.254,"15min":0.254},"max":{"1min":1.052,"5min":7.468,"15min":10.054},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.78100000000000003,"15min":0.78600000000000003},"min":{"1min":0.46100000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":2.302,"5min":7.8979999999999997,"15min":12.099},"last":0.34300000000000003}]},{"osd":1,"last update":"Wed Mar 24 22:06:18 2021","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.88700000000000001,"15min":0.84399999999999997},"min":{"1min":0.41199999999999998,"5min":0.32600000000000001,"15min":0.28100000000000003},"max":{"1min":1.0269999999999999,"5min":12.343999999999999,"15min":17.954999999999998},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.86099999999999999,"15min":0.79900000000000004},"min":{"1min":0.36399999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":2.1190000000000002,"5min":7.4989999999999997,"15min":7.4989999999999997},"last":0.48399999999999999}]},{"osd":3,"last update":"Wed Mar 24 22:06:45 2021","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.97899999999999998,"15min":0.98199999999999998},"min":{"1min":0.52300000000000002,"5min":0.374,"15min":0.33400000000000002},"max":{"1min":2.8559999999999999,"5min":12.308,"15min":25.009},"last":0.52300000000000002},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.016,"15min":0.93600000000000005},"min":{"1min":0.57499999999999996,"5min":0.34499999999999997,"15min":0.33900000000000002},"max":{"1min":2.871,"5min":7.5199999999999996,"15min":7.5199999999999996},"last":0.57499999999999996}]},{"osd":4,"last update":"Wed Mar 24 22:06:07 2021","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":1.1140000000000001,"15min":1.1339999999999999},"min":{"1min":0.68000000000000005,"5min":0.45300000000000001,"15min":0.44400000000000001},"max":{"1min":2.6589999999999998,"5min":12.467000000000001,"15min":24.914000000000001},"last":0.62},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.0229999999999999,"15min":1.034},"min":{"1min":0.65200000000000002,"5min":0.47199999999999998,"15min":0.41299999999999998},"max":{"1min":1.4419999999999999,"5min":2.8290000000000002,"15min":2.8290000000000002},"last":0.80900000000000005}]},{"osd":5,"last update":"Wed Mar 24 22:06:07 2021","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.139,"15min":1.139},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.639,"5min":1.639,"15min":1.639},"last":0.66600000000000004},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":2.6869999999999998,"5min":2.6869999999999998,"15min":2.6869999999999998},"last":0.89000000000000001}]},{"osd":6,"last update":"Wed Mar 24 22:06:45 2021","interfaces":[{"interface":"back","average":{"1min":1.2250000000000001,"5min":1.1060000000000001,"15min":1.103},"min":{"1min":0.74299999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.649,"5min":2.702,"15min":2.7719999999999998},"last":0.74299999999999999},{"interface":"front","average":{"1min":1.319,"5min":1.1639999999999999,"15min":1.131},"min":{"1min":0.76600000000000001,"5min":0.52500000000000002,"15min":0.47899999999999998},"max":{"1min":2.7309999999999999,"5min":4.3369999999999997,"15min":6.6520000000000001},"last":0.84499999999999997}]},{"osd":7,"last update":"Wed Mar 24 22:06:09 2021","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.167,"15min":1.1839999999999999},"min":{"1min":0.73099999999999998,"5min":0.51800000000000002,"15min":0.442},"max":{"1min":2.6269999999999998,"5min":4.3630000000000004,"15min":6.7720000000000002},"last":0.70199999999999996},{"interface":"front","average":{"1min":1.202,"5min":1.258,"15min":1.29},"min":{"1min":0.73999999999999999,"5min":0.505,"15min":0.41999999999999998},"max":{"1min":2.73,"5min":12.492000000000001,"15min":24.969999999999999},"last":0.78200000000000003}]}]},{"osd":3,"up_from":26,"seq":111669149899,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":329080,"kb_used_data":264112,"kb_used_omap":0,"kb_used_meta":64960,"kb_avail":93416072,"statfs":{"total":95995035648,"available":95658057728,"internally_reserved":0,"allocated":270450688,"data_stored":358911927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":66519040},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 22:06:43 2021","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.81899999999999995,"15min":0.91500000000000004},"min":{"1min":0.51200000000000001,"5min":0.29699999999999999,"15min":0.23899999999999999},"max":{"1min":3.2959999999999998,"5min":13.829000000000001,"15min":27.548999999999999},"last":0.53400000000000003},{"interface":"front","average":{"1min":1.165,"5min":0.83299999999999996,"15min":0.93400000000000005},"min":{"1min":0.495,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":6.8310000000000004,"5min":6.8310000000000004,"15min":26.593},"last":0.56599999999999995}]},{"osd":1,"last update":"Wed Mar 24 22:06:43 2021","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":0.81000000000000005,"15min":0.80300000000000005},"min":{"1min":0.46200000000000002,"5min":0.314,"15min":0.26700000000000002},"max":{"1min":3.802,"5min":5.2610000000000001,"15min":14.324},"last":0.70299999999999996},{"interface":"front","average":{"1min":1.081,"5min":0.86299999999999999,"15min":0.81200000000000006},"min":{"1min":0.51000000000000001,"5min":0.28899999999999998,"15min":0.28000000000000003},"max":{"1min":5.2910000000000004,"5min":5.2910000000000004,"15min":12.631},"last":0.84099999999999997}]},{"osd":2,"last update":"Wed Mar 24 22:06:43 2021","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.81399999999999995,"15min":0.85299999999999998},"min":{"1min":0.45900000000000002,"5min":0.34699999999999998,"15min":0.316},"max":{"1min":2.827,"5min":5.0439999999999996,"15min":12.510999999999999},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.84899999999999998,"15min":0.85599999999999998},"min":{"1min":0.50900000000000001,"5min":0.42799999999999999,"15min":0.34899999999999998},"max":{"1min":2.7170000000000001,"5min":4.5640000000000001,"15min":11.317},"last":0.67300000000000004}]},{"osd":4,"last update":"Wed Mar 24 22:06:02 2021","interfaces":[{"interface":"back","average":{"1min":1.204,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.89500000000000002,"5min":0.622,"15min":0.42999999999999999},"max":{"1min":2.5880000000000001,"5min":2.5880000000000001,"15min":2.601},"last":0.73099999999999998},{"interface":"front","average":{"1min":1.54,"5min":1.1060000000000001,"15min":1.0640000000000001},"min":{"1min":0.81799999999999995,"5min":0.47899999999999998,"15min":0.32900000000000001},"max":{"1min":4.0700000000000003,"5min":4.9569999999999999,"15min":14.291},"last":0.94699999999999995}]},{"osd":5,"last update":"Wed Mar 24 22:06:08 2021","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"min":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":2.7400000000000002,"5min":2.7400000000000002,"15min":2.7400000000000002},"last":0.97399999999999998},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"min":{"1min":0.879,"5min":0.879,"15min":0.879},"max":{"1min":2.6949999999999998,"5min":2.6949999999999998,"15min":2.6949999999999998},"last":1.022}]},{"osd":6,"last update":"Wed Mar 24 22:06:43 2021","interfaces":[{"interface":"back","average":{"1min":1.45,"5min":1.135,"15min":1.1240000000000001},"min":{"1min":0.874,"5min":0.59799999999999998,"15min":0.40000000000000002},"max":{"1min":4.0369999999999999,"5min":4.9950000000000001,"15min":14.255000000000001},"last":1.004},{"interface":"front","average":{"1min":1.286,"5min":1.0720000000000001,"15min":1.117},"min":{"1min":0.78100000000000003,"5min":0.54800000000000004,"15min":0.39700000000000002},"max":{"1min":3.5859999999999999,"5min":3.5859999999999999,"15min":12.666},"last":1.2729999999999999}]},{"osd":7,"last update":"Wed Mar 24 22:06:08 2021","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.089,"15min":1.1180000000000001},"min":{"1min":0.81499999999999995,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":2.2189999999999999,"5min":2.2189999999999999,"15min":2.633},"last":1.077},{"interface":"front","average":{"1min":1.5960000000000001,"5min":1.216,"15min":1.23},"min":{"1min":0.98999999999999999,"5min":0.60399999999999998,"15min":0.44700000000000001},"max":{"1min":4.1159999999999997,"5min":5.0190000000000001,"15min":14.727},"last":1.0840000000000001}]}]},{"osd":4,"up_from":32,"seq":137438953669,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":115276,"kb_used_data":71364,"kb_used_omap":0,"kb_used_meta":43904,"kb_avail":93629876,"statfs":{"total":95995035648,"available":95876993024,"internally_reserved":0,"allocated":73076736,"data_stored":92421650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":44957696},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 22:06:09 2021","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":0.999,"15min":1.034},"min":{"1min":0.60999999999999999,"5min":0.437,"15min":0.223},"max":{"1min":1.373,"5min":5.9050000000000002,"15min":41.472000000000001},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.95099999999999996,"15min":1.1499999999999999},"min":{"1min":0.60899999999999999,"5min":0.41999999999999998,"15min":0.34000000000000002},"max":{"1min":1.3100000000000001,"5min":5.3719999999999999,"15min":44.185000000000002},"last":0.76200000000000001}]},{"osd":1,"last update":"Wed Mar 24 22:06:09 2021","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":1.0189999999999999,"15min":0.94999999999999996},"min":{"1min":0.57999999999999996,"5min":0.42699999999999999,"15min":0.38100000000000001},"max":{"1min":1.3460000000000001,"5min":7.5060000000000002,"15min":7.5060000000000002},"last":0.81999999999999995},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.149,"15min":0.99199999999999999},"min":{"1min":0.70699999999999996,"5min":0.48399999999999999,"15min":0.36399999999999999},"max":{"1min":1.528,"5min":11.983000000000001,"15min":11.983000000000001},"last":0.72199999999999998}]},{"osd":2,"last update":"Wed Mar 24 22:06:09 2021","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.87,"15min":0.90500000000000003},"min":{"1min":0.66300000000000003,"5min":0.46300000000000002,"15min":0.35899999999999999},"max":{"1min":1.2929999999999999,"5min":3.21,"15min":4.4379999999999997},"last":1.1040000000000001},{"interface":"front","average":{"1min":0.995,"5min":1.0009999999999999,"15min":0.98899999999999999},"min":{"1min":0.626,"5min":0.439,"15min":0.40500000000000003},"max":{"1min":1.3640000000000001,"5min":4.3970000000000002,"15min":5.7999999999999998},"last":0.66200000000000003}]},{"osd":3,"last update":"Wed Mar 24 22:06:09 2021","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":1.085,"15min":1.038},"min":{"1min":0.64700000000000002,"5min":0.501,"15min":0.439},"max":{"1min":1.2869999999999999,"5min":3.9809999999999999,"15min":4.8049999999999997},"last":0.91600000000000004},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.0209999999999999,"15min":1.01},"min":{"1min":0.84999999999999998,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.3779999999999999,"5min":2.4700000000000002,"15min":4.1189999999999998},"last":0.94499999999999995}]},{"osd":5,"last update":"Wed Mar 24 22:06:03 2021","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.22,"5min":1.22,"15min":1.22},"last":0.98299999999999998},{"interface":"front","average":{"1min":1.202,"5min":1.202,"15min":1.202},"min":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":1.9550000000000001,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":1.1919999999999999}]},{"osd":6,"last update":"Wed Mar 24 22:06:34 2021","interfaces":[{"interface":"back","average":{"1min":1.101,"5min":1.1919999999999999,"15min":1.0900000000000001},"min":{"1min":0.73299999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.7250000000000001,"5min":7.7679999999999998,"15min":7.7679999999999998},"last":1.0229999999999999},{"interface":"front","average":{"1min":1.246,"5min":1.3620000000000001,"15min":1.161},"min":{"1min":0.78000000000000003,"5min":0.53300000000000003,"15min":0.41099999999999998},"max":{"1min":2.923,"5min":14.122999999999999,"15min":14.122999999999999},"last":1.1379999999999999}]},{"osd":7,"last update":"Wed Mar 24 22:06:09 2021","interfaces":[{"interface":"back","average":{"1min":1.3089999999999999,"5min":1.196,"15min":1.147},"min":{"1min":0.87,"5min":0.46200000000000002,"15min":0.44400000000000001},"max":{"1min":1.8700000000000001,"5min":2.399,"15min":3.3479999999999999},"last":1.2150000000000001},{"interface":"front","average":{"1min":1.23,"5min":1.1160000000000001,"15min":1.1319999999999999},"min":{"1min":0.97299999999999998,"5min":0.49399999999999999,"15min":0.49299999999999999},"max":{"1min":1.7090000000000001,"5min":3.1680000000000001,"15min":3.1680000000000001},"last":1.258}]}]},{"osd":5,"up_from":733,"seq":3148211027990,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":166764,"kb_used_data":153188,"kb_used_omap":164,"kb_used_meta":13403,"kb_avail":93578388,"statfs":{"total":95995035648,"available":95824269312,"internally_reserved":0,"allocated":156864512,"data_stored":195079488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":168054,"internal_metadata":13725578},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Mar 24 22:06:03 2021","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.6699999999999999,"5min":1.6699999999999999,"15min":1.6699999999999999},"last":1.2490000000000001},{"interface":"front","average":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.4339999999999999,"5min":1.4339999999999999,"15min":1.4339999999999999},"last":0.621}]},{"osd":1,"last update":"Wed Mar 24 22:06:03 2021","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.425,"5min":1.425,"15min":1.425},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":2.0939999999999999,"5min":2.0939999999999999,"15min":2.0939999999999999},"last":0.89700000000000002}]},{"osd":2,"last update":"Wed Mar 24 22:06:03 2021","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.778,"5min":1.778,"15min":1.778},"last":1.111},{"interface":"front","average":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.71899999999999997}]},{"osd":3,"last update":"Wed Mar 24 22:06:03 2021","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.669,"5min":1.669,"15min":1.669},"last":1.1719999999999999},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"min":{"1min":0.755,"5min":0.755,"15min":0.755},"max":{"1min":1.863,"5min":1.863,"15min":1.863},"last":1.4159999999999999}]},{"osd":4,"last update":"Wed Mar 24 22:06:03 2021","interfaces":[{"interface":"back","average":{"1min":1.093,"5min":1.093,"15min":1.093},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":2.5219999999999998,"5min":2.5219999999999998,"15min":2.5219999999999998},"last":1.0169999999999999},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":2.2450000000000001,"5min":2.2450000000000001,"15min":2.2450000000000001},"last":1.22}]},{"osd":6,"last update":"Wed Mar 24 22:06:03 2021","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":1.131,"15min":1.131},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":2.0550000000000002,"5min":2.0550000000000002,"15min":2.0550000000000002},"last":1.343},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.74,"5min":1.74,"15min":1.74},"last":0.877}]},{"osd":7,"last update":"Wed Mar 24 22:06:03 2021","interfaces":[{"interface":"back","average":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":3.0430000000000001,"5min":3.0430000000000001,"15min":3.0430000000000001},"last":1.056},{"interface":"front","average":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":2.698,"5min":2.698,"15min":2.698},"last":1.601}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10235,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":5284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":450560,"data_stored":16320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":176128,"data_stored":6220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":241664,"data_stored":8622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":307200,"data_stored":11122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":5674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":10356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":6974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":155025408,"data_stored":209903213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77574144,"data_stored":101258464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":178200576,"data_stored":241528174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":265764864,"data_stored":358366807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":65839104,"data_stored":88320384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":149794816,"data_stored":190990337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":149934080,"data_stored":206789435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":207589376,"data_stored":290982347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-03-24T22:06:55.503 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd pool set iscsi pgp_num 3 2021-03-24T22:06:55.520 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:55.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:55.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:06:55.580 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:55 smithi125 conmon[24927]: cluster 2021-03-24T22:06: 2021-03-24T22:06:55.581 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:55 smithi125 conmon[24927]: 54.022616+0000 osd.3 (osd.3) 167 : cluster [DBG] 3.27 scrub ok 2021-03-24T22:06:55.581 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:55 smithi125 conmon[24927]: cluster 2021-03-24T22:06:55.581 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:55 smithi125 conmon[24927]: 2021-03- 2021-03-24T22:06:55.582 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:55 smithi125 conmon[24927]: 24T22:06:54.484171+0000 2021-03-24T22:06:55.582 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:55 smithi125 conmon[24927]: mon.a (mon.0) 2080 : cluster [DBG] 2021-03-24T22:06:55.582 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:55 smithi125 conmon[24927]: osdmap e844: 8 total, 8 up, 8 in 2021-03-24T22:06:55.582 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:55 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T22:06:55.582 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:55 smithi125 conmon[24927]: 22:06:54.490328+0000 osd.0 ( 2021-03-24T22:06:55.583 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:55 smithi125 conmon[24927]: osd.0) 168 : cluster [DBG] 6.9 scrub ok 2021-03-24T22:06:55.583 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:55 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:06:55.583 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:55 smithi125 conmon[24927]: -03-24T22:06:55. 2021-03-24T22:06:55.583 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:55 smithi125 conmon[24927]: 447028+0000 mon.a (mon.0) 2081 : 2021-03-24T22:06:55.584 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:55 smithi125 conmon[24927]: cluster [DBG] osdmap e845: 8 total, 8 up, 8 in 2021-03-24T22:06:55.584 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:55 smithi125 conmon[19936]: cluster 2021-03-24T22:06:54.022616+0000 2021-03-24T22:06:55.584 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:55 smithi125 conmon[19936]: osd.3 (osd.3) 167 : cluster [DBG] 3.27 scrub ok 2021-03-24T22:06:55.584 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:55 smithi125 conmon[19936]: cluster 2021-03-24T22:06:54.484171+0000 mon.a (mon.0) 2080 : cluster [DBG] osdmap e844: 8 total, 8 up, 8 in 2021-03-24T22:06:55.585 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:55 smithi125 conmon[19936]: cluster 2021-03-24T22:06:55.585 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:55 smithi125 conmon[19936]: 2021-03-24T22:06 2021-03-24T22:06:55.585 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:55 smithi125 conmon[19936]: :54.490328+0000 osd.0 (osd 2021-03-24T22:06:55.585 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:55 smithi125 conmon[19936]: .0) 168 : cluster [DBG] 2021-03-24T22:06:55.586 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:55 smithi125 conmon[19936]: 6.9 scrub ok 2021-03-24T22:06:55.586 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:55 smithi125 conmon[19936]: cluster 2021 2021-03-24T22:06:55.586 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:55 smithi125 conmon[19936]: -03-24T22:06:55 2021-03-24T22:06:55.586 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:55 smithi125 conmon[19936]: .447028+0000 mon.a (mon. 2021-03-24T22:06:55.587 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:55 smithi125 conmon[19936]: 0) 2081 : cluster [DBG] 2021-03-24T22:06:55.587 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:55 smithi125 conmon[19936]: osdmap e845: 8 total, 8 up, 8 in 2021-03-24T22:06:55.628 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:55.684 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:55.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:55.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:55.790 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:55.805 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:55 smithi142 conmon[20125]: cluster 2021-03-24T22:06:54.022616+0000 osd.3 (osd.3) 167 : 2021-03-24T22:06:55.805 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:55 smithi142 conmon[20125]: cluster [DBG] 3.27 scrub ok 2021-03-24T22:06:55.805 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:55 smithi142 conmon[20125]: cluster 2021-03-24T22:06:54.484171+0000 mon.a (mon.0) 2021-03-24T22:06:55.806 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:55 smithi142 conmon[20125]: 2080 : cluster [DBG] osdmap e844: 8 total, 8 up, 8 in 2021-03-24T22:06:55.806 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:55 smithi142 conmon[20125]: cluster 2021-03-24T22:06:55.806 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:55 smithi142 conmon[20125]: 2021-03-24T22:06:54.490328+0000 osd.0 (osd 2021-03-24T22:06:55.807 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:55 smithi142 conmon[20125]: .0) 168 : cluster [DBG] 6.9 scrub ok 2021-03-24T22:06:55.807 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:55 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:06:55.807 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:55 smithi142 conmon[20125]: 24T22:06:55.447028+0000 mon.a ( 2021-03-24T22:06:55.807 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:55 smithi142 conmon[20125]: mon.0) 2081 : cluster [DBG] osdmap e845: 8 total, 8 up, 8 in 2021-03-24T22:06:55.848 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:55.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:55.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:06:55.954 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:56.134 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:56.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:56.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:06:56.240 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:56.296 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:56.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:56.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:06:56.403 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:56.464 INFO:tasks.rados.rados.0.smithi125.stdout:3261: done (1 left) 2021-03-24T22:06:56.464 INFO:tasks.rados.rados.0.smithi125.stdout:3263: done (0 left) 2021-03-24T22:06:56.465 INFO:tasks.rados.rados.0.smithi125.stdout:3264: delete oid 28 current snap is 380 2021-03-24T22:06:56.469 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:56.472 INFO:tasks.rados.rados.0.smithi125.stdout:3264: done (0 left) 2021-03-24T22:06:56.473 INFO:tasks.rados.rados.0.smithi125.stdout:3265: read oid 6 snap -1 2021-03-24T22:06:56.473 INFO:tasks.rados.rados.0.smithi125.stdout:3265: expect deleted 2021-03-24T22:06:56.473 INFO:tasks.rados.rados.0.smithi125.stdout:3266: copy_from oid 7 from oid 40 current snap is 380 2021-03-24T22:06:56.473 INFO:tasks.rados.rados.0.smithi125.stdout:3267: write (excl) oid 3 current snap is 380 2021-03-24T22:06:56.474 INFO:tasks.rados.rados.0.smithi125.stdout:3267: seq_num 1109 ranges {638963=722326,2079707=424046} 2021-03-24T22:06:56.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:56.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:06:56.480 INFO:tasks.rados.rados.0.smithi125.stdout:3267: writing smithi12556470-3 from 638963 to 1361289 tid 1 2021-03-24T22:06:56.489 INFO:tasks.rados.rados.0.smithi125.stdout:3267: writing smithi12556470-3 from 2079707 to 2503753 tid 2 2021-03-24T22:06:56.490 INFO:tasks.rados.rados.0.smithi125.stdout:3268: rollback oid 8 current snap is 380 2021-03-24T22:06:56.490 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 8 to 364 2021-03-24T22:06:56.491 INFO:tasks.rados.rados.0.smithi125.stdout:3265: done (3 left) 2021-03-24T22:06:56.491 INFO:tasks.rados.rados.0.smithi125.stdout:3269: write oid 20 current snap is 380 2021-03-24T22:06:56.491 INFO:tasks.rados.rados.0.smithi125.stdout:3267: finishing write tid 1 to smithi12556470-3 2021-03-24T22:06:56.492 INFO:tasks.rados.rados.0.smithi125.stdout:3269: seq_num 1110 ranges {655770=513928,1854542=684057} 2021-03-24T22:06:56.497 INFO:tasks.rados.rados.0.smithi125.stdout:3269: writing smithi12556470-20 from 655770 to 1169698 tid 1 2021-03-24T22:06:56.505 INFO:tasks.rados.rados.0.smithi125.stdout:3269: writing smithi12556470-20 from 1854542 to 2538599 tid 2 2021-03-24T22:06:56.505 INFO:tasks.rados.rados.0.smithi125.stdout:3270: snap_create 2021-03-24T22:06:56.505 INFO:tasks.rados.rados.0.smithi125.stdout:3268: finishing rollback tid 1 to smithi12556470-8 2021-03-24T22:06:56.506 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 8 v 568 (ObjNum 1117634496 snap 21916 seq_num 3308612800) dirty dne 2021-03-24T22:06:56.506 INFO:tasks.rados.rados.0.smithi125.stdout:3267: finishing write tid 2 to smithi12556470-3 2021-03-24T22:06:56.506 INFO:tasks.rados.rados.0.smithi125.stdout:3267: finishing write tid 3 to smithi12556470-3 2021-03-24T22:06:56.506 INFO:tasks.rados.rados.0.smithi125.stdout:3267: finishing write tid 4 to smithi12556470-3 2021-03-24T22:06:56.507 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 3 v 1212 (ObjNum 1109 snap 380 seq_num 1109) dirty exists 2021-03-24T22:06:56.507 INFO:tasks.rados.rados.0.smithi125.stdout:3267: left oid 3 (ObjNum 1109 snap 380 seq_num 1109) 2021-03-24T22:06:56.507 INFO:tasks.rados.rados.0.smithi125.stdout:3266: finishing copy_from to smithi12556470-7 2021-03-24T22:06:56.507 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 7 v 2092 (ObjNum 906 snap 300 seq_num 906) dirty exists 2021-03-24T22:06:56.507 INFO:tasks.rados.rados.0.smithi125.stdout:3266: finishing copy_from racing read to smithi12556470-7 2021-03-24T22:06:56.508 INFO:tasks.rados.rados.0.smithi125.stdout:3269: finishing write tid 1 to smithi12556470-20 2021-03-24T22:06:56.509 INFO:tasks.rados.rados.0.smithi125.stdout:3269: finishing write tid 2 to smithi12556470-20 2021-03-24T22:06:56.509 INFO:tasks.rados.rados.0.smithi125.stdout:3269: finishing write tid 3 to smithi12556470-20 2021-03-24T22:06:56.509 INFO:tasks.rados.rados.0.smithi125.stdout:3269: finishing write tid 4 to smithi12556470-20 2021-03-24T22:06:56.509 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 20 v 1458 (ObjNum 1110 snap 380 seq_num 1110) dirty exists 2021-03-24T22:06:56.510 INFO:tasks.rados.rados.0.smithi125.stdout:3269: left oid 20 (ObjNum 1110 snap 380 seq_num 1110) 2021-03-24T22:06:56.577 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:56.637 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:06:56.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:56.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:06:56.643 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:06:56.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:56 smithi125 conmon[24927]: cluster 2021-03-24T22:06 2021-03-24T22:06:56.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:56 smithi125 conmon[24927]: :55.015577+0000 mgr.y (mgr.14144) 700 : cluster [DBG] pgmap v1429: 168 pgs: 168 active+clean; 540 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.4 MiB/s rd, 313 KiB/s wr, 10 op/s 2021-03-24T22:06:56.671 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:56 smithi125 conmon[24927]: cluster 2021-03-24T22:06:56.461264+0000 mon.a (mon.0) 2082 : cluster [DBG] osdmap e846: 8 total, 8 up, 8 in 2021-03-24T22:06:56.672 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:56 smithi125 conmon[19936]: cluster 2021-03-24T22:06:55.015577+0000 mgr.y (mgr.14144 2021-03-24T22:06:56.673 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:56 smithi125 conmon[19936]: ) 700 : cluster [DBG] pgmap v1429: 168 pgs: 168 active+clean; 540 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.4 MiB/s rd, 313 KiB/s wr, 10 op/s 2021-03-24T22:06:56.673 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:56 smithi125 conmon[19936]: cluster 2021-03-24T22:06:56.461264+0000 mon.a (mon.0) 2082 : cluster [DBG] osdmap e846: 8 total, 8 up, 8 in 2021-03-24T22:06:56.675 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.4 dump_ops_in_flight 2021-03-24T22:06:56.743 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:56.790 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:06:56.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:56.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:06:56.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:56 smithi142 conmon[20125]: cluster 2021-03-24T22:06:55.015577 2021-03-24T22:06:56.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:56 smithi142 conmon[20125]: +0000 mgr.y (mgr.14144) 700 : cluster [DBG] pgmap v1429: 168 pgs: 168 active+clean; 540 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.4 MiB/s rd, 313 KiB/s wr, 10 op/s 2021-03-24T22:06:56.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:56 smithi142 conmon[20125]: cluster 2021-03-24T22:06:56.461264+0000 mon.a (mon.0 2021-03-24T22:06:56.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:56 smithi142 conmon[20125]: ) 2082 : cluster [DBG] osdmap e846: 8 total, 8 up, 8 in 2021-03-24T22:06:56.895 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:56.953 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:56.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:56.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:06:56.977 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T22:06:57.060 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:57.105 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:57.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:57.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:06:57.212 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:57.264 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:57.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:57.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:06:57.370 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:57.430 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:57.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:57.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:06:57.506 INFO:tasks.rados.rados.0.smithi125.stdout:3266: done (4 left) 2021-03-24T22:06:57.507 INFO:tasks.rados.rados.0.smithi125.stdout:3267: done (3 left) 2021-03-24T22:06:57.507 INFO:tasks.rados.rados.0.smithi125.stdout:3268: done (2 left) 2021-03-24T22:06:57.507 INFO:tasks.rados.rados.0.smithi125.stdout:3269: done (1 left) 2021-03-24T22:06:57.508 INFO:tasks.rados.rados.0.smithi125.stdout:3270: done (0 left) 2021-03-24T22:06:57.508 INFO:tasks.rados.rados.0.smithi125.stdout:3271: read oid 15 snap 366 2021-03-24T22:06:57.508 INFO:tasks.rados.rados.0.smithi125.stdout:3271: expect deleted 2021-03-24T22:06:57.508 INFO:tasks.rados.rados.0.smithi125.stdout:3272: write oid 4 current snap is 381 2021-03-24T22:06:57.508 INFO:tasks.rados.rados.0.smithi125.stdout:3272: seq_num 1111 ranges {672577=589177,1913024=427715} 2021-03-24T22:06:57.511 INFO:tasks.rados.rados.0.smithi125.stdout:3272: writing smithi12556470-4 from 672577 to 1261754 tid 1 2021-03-24T22:06:57.514 INFO:teuthology.orchestra.run.smithi125.stderr:noscrub is set 2021-03-24T22:06:57.518 INFO:tasks.rados.rados.0.smithi125.stdout:3272: writing smithi12556470-4 from 1913024 to 2340739 tid 2 2021-03-24T22:06:57.519 INFO:tasks.rados.rados.0.smithi125.stdout:3273: rollback oid 33 current snap is 381 2021-03-24T22:06:57.519 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 33 to 363 2021-03-24T22:06:57.519 INFO:tasks.rados.rados.0.smithi125.stdout:3272: finishing write tid 1 to smithi12556470-4 2021-03-24T22:06:57.519 INFO:tasks.rados.rados.0.smithi125.stdout:3271: done (2 left) 2021-03-24T22:06:57.520 INFO:tasks.rados.rados.0.smithi125.stdout:3274: read oid 11 snap -1 2021-03-24T22:06:57.520 INFO:tasks.rados.rados.0.smithi125.stdout:3274: expect (ObjNum 1069 snap 358 seq_num 1069) 2021-03-24T22:06:57.520 INFO:tasks.rados.rados.0.smithi125.stdout:3275: copy_from oid 44 from oid 50 current snap is 381 2021-03-24T22:06:57.520 INFO:tasks.rados.rados.0.smithi125.stdout:3276: write (excl) oid 45 current snap is 381 2021-03-24T22:06:57.521 INFO:tasks.rados.rados.0.smithi125.stdout:3276: seq_num 1112 ranges {689384=664426,1971506=687726} 2021-03-24T22:06:57.526 INFO:tasks.rados.rados.0.smithi125.stdout:3276: writing smithi12556470-45 from 689384 to 1353810 tid 1 2021-03-24T22:06:57.534 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:57.535 INFO:tasks.rados.rados.0.smithi125.stdout:3276: writing smithi12556470-45 from 1971506 to 2659232 tid 2 2021-03-24T22:06:57.536 INFO:tasks.rados.rados.0.smithi125.stdout:3277: write oid 3 current snap is 381 2021-03-24T22:06:57.536 INFO:tasks.rados.rados.0.smithi125.stdout:3273: finishing rollback tid 1 to smithi12556470-33 2021-03-24T22:06:57.536 INFO:tasks.rados.rados.0.smithi125.stdout:3277: seq_num 1113 ranges {706191=739675,1913635=431384} 2021-03-24T22:06:57.544 INFO:tasks.rados.rados.0.smithi125.stdout:3277: writing smithi12556470-3 from 706191 to 1445866 tid 1 2021-03-24T22:06:57.550 INFO:tasks.rados.rados.0.smithi125.stdout:3277: writing smithi12556470-3 from 1913635 to 2345019 tid 2 2021-03-24T22:06:57.550 INFO:tasks.rados.rados.0.smithi125.stdout:3278: read oid 37 snap 365 2021-03-24T22:06:57.550 INFO:tasks.rados.rados.0.smithi125.stdout:3278: expect (ObjNum 916 snap 304 seq_num 916) 2021-03-24T22:06:57.551 INFO:tasks.rados.rados.0.smithi125.stdout:3279: write oid 48 current snap is 381 2021-03-24T22:06:57.551 INFO:tasks.rados.rados.0.smithi125.stdout:3279: seq_num 1114 ranges {722998=414924,1572117=575042} 2021-03-24T22:06:57.556 INFO:tasks.rados.rados.0.smithi125.stdout:3279: writing smithi12556470-48 from 722998 to 1137922 tid 1 2021-03-24T22:06:57.562 INFO:tasks.rados.rados.0.smithi125.stdout:3279: writing smithi12556470-48 from 1572117 to 2147159 tid 2 2021-03-24T22:06:57.563 INFO:tasks.rados.rados.0.smithi125.stdout:3280: copy_from oid 30 from oid 14 current snap is 381 2021-03-24T22:06:57.563 INFO:tasks.rados.rados.0.smithi125.stdout:3273: finishing rollback tid 2 to smithi12556470-33 2021-03-24T22:06:57.563 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 33 v 1463 (ObjNum 916 snap 304 seq_num 916) dirty exists 2021-03-24T22:06:57.564 INFO:tasks.rados.rados.0.smithi125.stdout:3274: expect (ObjNum 1069 snap 358 seq_num 1069) 2021-03-24T22:06:57.579 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:57.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:57.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:06:57.684 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-24T22:06:57.686 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:57.743 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:57.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:57.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:06:57.850 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:57.864 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:57 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:06:57.864 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:57 smithi125 conmon[19936]: 06:56.543668+0000 osd.0 (osd.0) 169 : cluster [DBG] 7.f scrub ok 2021-03-24T22:06:57.864 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:57 smithi125 conmon[19936]: audit 2021-03-24T 2021-03-24T22:06:57.865 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:57 smithi125 conmon[19936]: 22:06:57.209654+0000 mon.c (mon.2) 120 : audit 2021-03-24T22:06:57.865 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:57 smithi125 conmon[19936]: [INF] from='client.? v1:172.21.15.125:0/669366984' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T22:06:57.865 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:57 smithi125 conmon[19936]: audit 2021-03-24T22:06:57.210132+0000 mon.a 2021-03-24T22:06:57.865 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:57 smithi125 conmon[19936]: (mon.0) 2083 : audit [INF] from='client.35903 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T22:06:57.866 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:57 smithi125 conmon[24927]: cluster 2021-03-24T22:06:57.866 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:57 smithi125 conmon[24927]: 2021-03-24T22:06 2021-03-24T22:06:57.867 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:57 smithi125 conmon[24927]: :56.543668+0000 osd.0 (osd.0) 169 : cluster [DBG] 7.f scrub ok 2021-03-24T22:06:57.867 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:57 smithi125 conmon[24927]: audit 2021-03-24T22:06:57.209654+0000 mon.c ( 2021-03-24T22:06:57.867 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:57 smithi125 conmon[24927]: mon.2) 120 : audit [INF] from='client.? v1:172.21.15.125:0/669366984' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T22:06:57.867 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:57 smithi125 conmon[24927]: audit 2021-03-24T22:06:57.210132+0000 mon.a 2021-03-24T22:06:57.868 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:57 smithi125 conmon[24927]: (mon.0) 2083 : audit [INF] from='client.35903 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T22:06:57.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:57 smithi142 conmon[20125]: cluster 2021-03-24T22:06:56.543668+0000 osd.0 (osd.0) 169 : cluster [DBG] 7.f scrub ok 2021-03-24T22:06:57.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:57 smithi142 conmon[20125]: audit 2021-03-24T22:06:57.209654+0000 mon.c ( 2021-03-24T22:06:57.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:57 smithi142 conmon[20125]: mon.2) 120 : audit [INF] from='client.? v1:172.21.15.125:0/669366984' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T22:06:57.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:57 smithi142 conmon[20125]: audit 2021-03-24T22:06:57.210132+0000 mon.a (mon.0) 2083 : audit [INF] from='client.35903 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-24T22:06:57.877 INFO:tasks.rados.rados.0.smithi125.stdout:3272: finishing write tid 2 to smithi12556470-4 2021-03-24T22:06:57.878 INFO:tasks.rados.rados.0.smithi125.stdout:3272: finishing write tid 3 to smithi12556470-4 2021-03-24T22:06:57.878 INFO:tasks.rados.rados.0.smithi125.stdout:3272: finishing write tid 4 to smithi12556470-4 2021-03-24T22:06:57.878 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 4 v 1469 (ObjNum 1111 snap 381 seq_num 1111) dirty exists 2021-03-24T22:06:57.878 INFO:tasks.rados.rados.0.smithi125.stdout:3272: left oid 4 (ObjNum 1111 snap 381 seq_num 1111) 2021-03-24T22:06:57.879 INFO:tasks.rados.rados.0.smithi125.stdout:3276: finishing write tid 1 to smithi12556470-45 2021-03-24T22:06:57.879 INFO:tasks.rados.rados.0.smithi125.stdout:3275: finishing copy_from to smithi12556470-44 2021-03-24T22:06:57.879 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 44 v 1455 (ObjNum 1092 snap 370 seq_num 1092) dirty exists 2021-03-24T22:06:57.879 INFO:tasks.rados.rados.0.smithi125.stdout:3275: finishing copy_from racing read to smithi12556470-44 2021-03-24T22:06:57.879 INFO:tasks.rados.rados.0.smithi125.stdout:3276: finishing write tid 2 to smithi12556470-45 2021-03-24T22:06:57.880 INFO:tasks.rados.rados.0.smithi125.stdout:3276: finishing write tid 3 to smithi12556470-45 2021-03-24T22:06:57.880 INFO:tasks.rados.rados.0.smithi125.stdout:3276: finishing write tid 4 to smithi12556470-45 2021-03-24T22:06:57.880 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 45 v 746 (ObjNum 1112 snap 381 seq_num 1112) dirty exists 2021-03-24T22:06:57.880 INFO:tasks.rados.rados.0.smithi125.stdout:3276: left oid 45 (ObjNum 1112 snap 381 seq_num 1112) 2021-03-24T22:06:57.881 INFO:tasks.rados.rados.0.smithi125.stdout:3277: finishing write tid 1 to smithi12556470-3 2021-03-24T22:06:57.881 INFO:tasks.rados.rados.0.smithi125.stdout:3277: finishing write tid 2 to smithi12556470-3 2021-03-24T22:06:57.881 INFO:tasks.rados.rados.0.smithi125.stdout:3277: finishing write tid 3 to smithi12556470-3 2021-03-24T22:06:57.881 INFO:tasks.rados.rados.0.smithi125.stdout:3277: finishing write tid 4 to smithi12556470-3 2021-03-24T22:06:57.881 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 3 v 1215 (ObjNum 1113 snap 381 seq_num 1113) dirty exists 2021-03-24T22:06:57.882 INFO:tasks.rados.rados.0.smithi125.stdout:3277: left oid 3 (ObjNum 1113 snap 381 seq_num 1113) 2021-03-24T22:06:57.882 INFO:tasks.rados.rados.0.smithi125.stdout:3278: expect (ObjNum 916 snap 304 seq_num 916) 2021-03-24T22:06:57.904 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:57.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:57.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:06:58.008 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:58.067 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:58.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:58.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:06:58.072 INFO:tasks.rados.rados.0.smithi125.stdout:3279: finishing write tid 1 to smithi12556470-48 2021-03-24T22:06:58.072 INFO:tasks.rados.rados.0.smithi125.stdout:3279: finishing write tid 2 to smithi12556470-48 2021-03-24T22:06:58.072 INFO:tasks.rados.rados.0.smithi125.stdout:3279: finishing write tid 3 to smithi12556470-48 2021-03-24T22:06:58.073 INFO:tasks.rados.rados.0.smithi125.stdout:3279: finishing write tid 4 to smithi12556470-48 2021-03-24T22:06:58.073 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 48 v 1452 (ObjNum 1114 snap 381 seq_num 1114) dirty exists 2021-03-24T22:06:58.073 INFO:tasks.rados.rados.0.smithi125.stdout:3279: left oid 48 (ObjNum 1114 snap 381 seq_num 1114) 2021-03-24T22:06:58.073 INFO:tasks.rados.rados.0.smithi125.stdout:3280: finishing copy_from to smithi12556470-30 2021-03-24T22:06:58.073 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 30 v 1017 (ObjNum 984 snap 326 seq_num 984) dirty exists 2021-03-24T22:06:58.074 INFO:tasks.rados.rados.0.smithi125.stdout:3280: finishing copy_from racing read to smithi12556470-30 2021-03-24T22:06:58.074 INFO:tasks.rados.rados.0.smithi125.stdout:3272: done (8 left) 2021-03-24T22:06:58.074 INFO:tasks.rados.rados.0.smithi125.stdout:3273: done (7 left) 2021-03-24T22:06:58.074 INFO:tasks.rados.rados.0.smithi125.stdout:3274: done (6 left) 2021-03-24T22:06:58.074 INFO:tasks.rados.rados.0.smithi125.stdout:3275: done (5 left) 2021-03-24T22:06:58.075 INFO:tasks.rados.rados.0.smithi125.stdout:3276: done (4 left) 2021-03-24T22:06:58.075 INFO:tasks.rados.rados.0.smithi125.stdout:3277: done (3 left) 2021-03-24T22:06:58.075 INFO:tasks.rados.rados.0.smithi125.stdout:3278: done (2 left) 2021-03-24T22:06:58.075 INFO:tasks.rados.rados.0.smithi125.stdout:3279: done (1 left) 2021-03-24T22:06:58.076 INFO:tasks.rados.rados.0.smithi125.stdout:3280: done (0 left) 2021-03-24T22:06:58.076 INFO:tasks.rados.rados.0.smithi125.stdout:3281: read oid 30 snap -1 2021-03-24T22:06:58.076 INFO:tasks.rados.rados.0.smithi125.stdout:3281: expect (ObjNum 984 snap 326 seq_num 984) 2021-03-24T22:06:58.076 INFO:tasks.rados.rados.0.smithi125.stdout:3282: copy_from oid 36 from oid 3 current snap is 381 2021-03-24T22:06:58.076 INFO:tasks.rados.rados.0.smithi125.stdout:3283: snap_create 2021-03-24T22:06:58.078 INFO:tasks.rados.rados.0.smithi125.stdout:3281: expect (ObjNum 984 snap 326 seq_num 984) 2021-03-24T22:06:58.172 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:58.232 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:58.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:58.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:06:58.251 INFO:tasks.rados.rados.0.smithi125.stdout:3282: finishing copy_from to smithi12556470-36 2021-03-24T22:06:58.252 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 36 v 1215 (ObjNum 1113 snap 381 seq_num 1113) dirty exists 2021-03-24T22:06:58.253 INFO:tasks.rados.rados.0.smithi125.stdout:3282: finishing copy_from racing read to smithi12556470-36 2021-03-24T22:06:58.339 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:58.394 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:58.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:58.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:06:58.500 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:58.515 INFO:tasks.rados.rados.0.smithi125.stdout:3281: done (2 left) 2021-03-24T22:06:58.515 INFO:tasks.rados.rados.0.smithi125.stdout:3282: done (1 left) 2021-03-24T22:06:58.516 INFO:tasks.rados.rados.0.smithi125.stdout:3283: done (0 left) 2021-03-24T22:06:58.516 INFO:tasks.rados.rados.0.smithi125.stdout:3284: delete oid 39 current snap is 382 2021-03-24T22:06:58.526 INFO:tasks.rados.rados.0.smithi125.stdout:3284: done (0 left) 2021-03-24T22:06:58.527 INFO:tasks.rados.rados.0.smithi125.stdout:3285: read oid 2 snap -1 2021-03-24T22:06:58.527 INFO:tasks.rados.rados.0.smithi125.stdout:3285: expect (ObjNum 984 snap 326 seq_num 984) 2021-03-24T22:06:58.527 INFO:tasks.rados.rados.0.smithi125.stdout:3286: write (excl) oid 14 current snap is 382 2021-03-24T22:06:58.528 INFO:tasks.rados.rados.0.smithi125.stdout:3286: seq_num 1116 ranges {756612=565422,2089081=578711} 2021-03-24T22:06:58.537 INFO:tasks.rados.rados.0.smithi125.stdout:3286: writing smithi12556470-14 from 756612 to 1322034 tid 1 2021-03-24T22:06:58.548 INFO:tasks.rados.rados.0.smithi125.stdout:3286: writing smithi12556470-14 from 2089081 to 2667792 tid 2 2021-03-24T22:06:58.549 INFO:tasks.rados.rados.0.smithi125.stdout:3287: delete oid 36 current snap is 382 2021-03-24T22:06:58.549 INFO:tasks.rados.rados.0.smithi125.stdout:3285: expect (ObjNum 984 snap 326 seq_num 984) 2021-03-24T22:06:58.552 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:58.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:58.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:58.658 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:58.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:58 smithi125 conmon[19936]: cluster 2021-03-24T22:06 2021-03-24T22:06:58.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:58 smithi125 conmon[19936]: :57.016341+0000 mgr.y (mgr.14144) 701 : cluster [DBG] pgmap v1432: 168 pgs: 168 active+clean; 545 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 22 MiB/s rd, 798 KiB/s wr, 21 op/s 2021-03-24T22:06:58.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:58 smithi125 conmon[19936]: cluster 2021-03-24T22:06:57.236658+0000 2021-03-24T22:06:58.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:58 smithi125 conmon[19936]: osd.5 (osd.5) 26 : cluster [DBG] 6.d scrub ok 2021-03-24T22:06:58.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:58 smithi125 conmon[19936]: 2021-03-24T22:06:58.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:58 smithi125 conmon[19936]: cluster 2021-03-24T22:06:58.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:58 smithi125 conmon[19936]: 2021-03-24T22:06:57.484562+0000 mon.a (mon.0) 2021-03-24T22:06:58.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:58 smithi125 conmon[19936]: 2084 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:06:58.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:58 smithi125 conmon[19936]: audit 2021-03-24T22:06:57.498574 2021-03-24T22:06:58.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:58 smithi125 conmon[19936]: +0000 mon.a (mon.0) 2085 : audit [INF] from='client.35903 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T22:06:58.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:58 smithi125 conmon[19936]: cluster 2021-03-24T22:06:57 2021-03-24T22:06:58.664 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:58 smithi125 conmon[19936]: .498643+0000 mon.a (mon.0) 2086 : cluster [DBG] 2021-03-24T22:06:58.665 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:58 smithi125 conmon[19936]: osdmap e847: 8 total, 8 up, 8 in 2021-03-24T22:06:58.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:58 smithi125 conmon[24927]: cluster 2021-03-24T22 2021-03-24T22:06:58.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:58 smithi125 conmon[24927]: :06:57.016341+0000 mgr.y (mgr.14144) 701 : cluster [DBG] pgmap v1432: 168 pgs: 168 active+clean; 545 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 22 MiB/s rd, 798 KiB/s wr, 21 op/s 2021-03-24T22:06:58.665 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:58 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:06:58.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:58 smithi125 conmon[24927]: 03-24T22:06:57.236658+0000 2021-03-24T22:06:58.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:58 smithi125 conmon[24927]: osd.5 (osd.5) 26 : cluster [DBG] 6.d scrub ok 2021-03-24T22:06:58.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:58 smithi125 conmon[24927]: 2021-03-24T22:06:58.666 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:58 smithi125 conmon[24927]: cluster 2021-03-24T22:06: 2021-03-24T22:06:58.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:58 smithi125 conmon[24927]: 57.484562+0000 mon.a (mon.0) 2084 : 2021-03-24T22:06:58.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:58 smithi125 conmon[24927]: cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:06:58.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:58 smithi125 conmon[24927]: audit 2021-03-24 2021-03-24T22:06:58.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:58 smithi125 conmon[24927]: T22:06:57.498574+0000 mon.a (mon. 2021-03-24T22:06:58.667 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:58 smithi125 conmon[24927]: 0) 2085 : audit [INF] from='client.35903 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T22:06:58.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:58 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:06:58.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:58 smithi125 conmon[24927]: 03-24T22:06:57.498643+0000 2021-03-24T22:06:58.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:58 smithi125 conmon[24927]: mon.a (mon.0) 2086 : cluster [DBG] osdmap e847: 8 total, 8 up, 8 in 2021-03-24T22:06:58.668 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:58 smithi125 conmon[24927]: 2021-03-24T22:06:58.716 INFO:tasks.rados.rados.0.smithi125.stdout:3286: finishing write tid 1 to smithi12556470-14 2021-03-24T22:06:58.716 INFO:tasks.rados.rados.0.smithi125.stdout:3285: done (2 left) 2021-03-24T22:06:58.716 INFO:tasks.rados.rados.0.smithi125.stdout:3287: done (1 left) 2021-03-24T22:06:58.717 INFO:tasks.rados.rados.0.smithi125.stdout:3288: read oid 8 snap 374 2021-03-24T22:06:58.717 INFO:tasks.rados.rados.0.smithi125.stdout:3288: expect deleted 2021-03-24T22:06:58.717 INFO:tasks.rados.rados.0.smithi125.stdout:3289: snap_create 2021-03-24T22:06:58.717 INFO:tasks.rados.rados.0.smithi125.stdout:3286: finishing write tid 2 to smithi12556470-14 2021-03-24T22:06:58.717 INFO:tasks.rados.rados.0.smithi125.stdout:3286: finishing write tid 3 to smithi12556470-14 2021-03-24T22:06:58.718 INFO:tasks.rados.rados.0.smithi125.stdout:3286: finishing write tid 4 to smithi12556470-14 2021-03-24T22:06:58.718 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 14 v 2104 (ObjNum 1116 snap 382 seq_num 1116) dirty exists 2021-03-24T22:06:58.718 INFO:tasks.rados.rados.0.smithi125.stdout:3286: left oid 14 (ObjNum 1116 snap 382 seq_num 1116) 2021-03-24T22:06:58.719 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:06:58.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:58.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:06:58.823 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:58.837 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:58 smithi142 conmon[20125]: cluster 2021-03-24T22:06:57 2021-03-24T22:06:58.837 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:58 smithi142 conmon[20125]: .016341+0000 mgr.y (mgr.14144) 701 : cluster [DBG] pgmap v1432: 168 pgs: 168 active+clean; 545 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 22 MiB/s rd, 798 KiB/s wr, 21 op/s 2021-03-24T22:06:58.838 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:58 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:06:58.838 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:58 smithi142 conmon[20125]: T22:06:57.236658+0000 osd.5 (osd.5) 26 : cluster [DBG] 6.d scrub ok 2021-03-24T22:06:58.838 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:58 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:06:58.838 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:58 smithi142 conmon[20125]: 22:06:57.484562+0000 mon.a (mon.0) 2084 : cluster 2021-03-24T22:06:58.839 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:58 smithi142 conmon[20125]: [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:06:58.839 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:58 smithi142 conmon[20125]: audit 2021 2021-03-24T22:06:58.839 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:58 smithi142 conmon[20125]: -03-24T22:06:57.498574+0000 mon.a (mon.0) 2085 : audit 2021-03-24T22:06:58.840 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:58 smithi142 conmon[20125]: [INF] from='client.35903 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-24T22:06:58.840 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:58 smithi142 conmon[20125]: cluster 2021-03-24T22:06:57.498643+0000 mon.a ( 2021-03-24T22:06:58.840 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:58 smithi142 conmon[20125]: mon.0) 2086 : cluster [DBG] osdmap e847: 8 total, 8 up, 8 in 2021-03-24T22:06:58.876 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:06:58.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:58.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:06:58.983 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:59.042 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:59.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:59.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:06:59.145 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:59.201 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:06:59.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:59.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:06:59.308 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:59.373 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:06:59.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:59.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:06:59.479 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:59.481 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:06:59.519 INFO:tasks.rados.rados.0.smithi125.stdout:3286: done (2 left) 2021-03-24T22:06:59.520 INFO:tasks.rados.rados.0.smithi125.stdout:3288: done (1 left) 2021-03-24T22:06:59.520 INFO:tasks.rados.rados.0.smithi125.stdout:3289: done (0 left) 2021-03-24T22:06:59.521 INFO:tasks.rados.rados.0.smithi125.stdout:3290: copy_from oid 37 from oid 20 current snap is 383 2021-03-24T22:06:59.521 INFO:tasks.rados.rados.0.smithi125.stdout:3291: write (excl) oid 38 current snap is 383 2021-03-24T22:06:59.521 INFO:tasks.rados.rados.0.smithi125.stdout:3291: seq_num 1118 ranges {790226=432273,1806045=582380} 2021-03-24T22:06:59.534 INFO:tasks.rados.rados.0.smithi125.stdout:3291: writing smithi12556470-38 from 790226 to 1222499 tid 1 2021-03-24T22:06:59.541 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:06:59.542 INFO:tasks.rados.rados.0.smithi125.stdout:3291: writing smithi12556470-38 from 1806045 to 2388425 tid 2 2021-03-24T22:06:59.542 INFO:tasks.rados.rados.0.smithi125.stdout:3292: snap_remove snap 343 2021-03-24T22:06:59.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:59.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:06:59.572 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:59 smithi142 conmon[20125]: cluster 2021-03-24T22:06:58.510249+0000 mon.a (mon.0) 2087 : 2021-03-24T22:06:59.573 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:06:59 smithi142 conmon[20125]: cluster [DBG] osdmap e848: 8 total, 8 up, 8 in 2021-03-24T22:06:59.647 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:59.654 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2021-03-24T22:06:59.655 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2021-03-24T22:06:59.655 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2021-03-24T22:06:59.655 INFO:teuthology.orchestra.run.smithi142.stdout:} 2021-03-24T22:06:59.661 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:06:59 smithi125 conmon[19936]: cluster 2021-03-24T22:06:58.510249+0000 mon.a (mon.0) 2087 : cluster [DBG] osdmap e848: 8 total, 8 up, 8 in 2021-03-24T22:06:59.662 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:06:59 smithi125 conmon[24927]: cluster 2021-03-24T22:06:58.510249+0000 mon.a (mon.0) 2087 : cluster [DBG] osdmap e848: 8 total, 8 up, 8 in 2021-03-24T22:06:59.704 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:59.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:59.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:06:59.810 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:59.879 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:06:59.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:06:59.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:06:59.988 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:07:00.039 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:07:00.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:00.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:07:00.146 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:07:00.201 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:07:00.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:00.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:07:00.309 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:07:00.364 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:07:00.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:00.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:07:00.458 INFO:tasks.rados.rados.0.smithi125.stdout:3292: done (2 left) 2021-03-24T22:07:00.459 INFO:tasks.rados.rados.0.smithi125.stdout:3293: delete oid 6 current snap is 383 2021-03-24T22:07:00.459 INFO:tasks.rados.rados.0.smithi125.stdout:3291: finishing write tid 1 to smithi12556470-38 2021-03-24T22:07:00.459 INFO:tasks.rados.rados.0.smithi125.stdout:3290: finishing copy_from to smithi12556470-37 2021-03-24T22:07:00.459 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 37 v 1458 (ObjNum 1110 snap 380 seq_num 1110) dirty exists 2021-03-24T22:07:00.460 INFO:tasks.rados.rados.0.smithi125.stdout:3290: finishing copy_from racing read to smithi12556470-37 2021-03-24T22:07:00.460 INFO:tasks.rados.rados.0.smithi125.stdout:3291: finishing write tid 2 to smithi12556470-38 2021-03-24T22:07:00.460 INFO:tasks.rados.rados.0.smithi125.stdout:3291: finishing write tid 3 to smithi12556470-38 2021-03-24T22:07:00.460 INFO:tasks.rados.rados.0.smithi125.stdout:3291: finishing write tid 4 to smithi12556470-38 2021-03-24T22:07:00.461 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 38 v 747 (ObjNum 1118 snap 383 seq_num 1118) dirty exists 2021-03-24T22:07:00.461 INFO:tasks.rados.rados.0.smithi125.stdout:3291: left oid 38 (ObjNum 1118 snap 383 seq_num 1118) 2021-03-24T22:07:00.461 INFO:teuthology.orchestra.run.smithi125.stderr:set pool 2 pgp_num to 3 2021-03-24T22:07:00.471 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:07:00.478 INFO:tasks.rados.rados.0.smithi125.stdout:3290: done (2 left) 2021-03-24T22:07:00.479 INFO:tasks.rados.rados.0.smithi125.stdout:3291: done (1 left) 2021-03-24T22:07:00.479 INFO:tasks.rados.rados.0.smithi125.stdout:3293: done (0 left) 2021-03-24T22:07:00.479 INFO:tasks.rados.rados.0.smithi125.stdout:3294: read oid 38 snap -1 2021-03-24T22:07:00.479 INFO:tasks.rados.rados.0.smithi125.stdout:3294: expect (ObjNum 1118 snap 383 seq_num 1118) 2021-03-24T22:07:00.479 INFO:tasks.rados.rados.0.smithi125.stdout:3295: write oid 39 current snap is 383 2021-03-24T22:07:00.480 INFO:tasks.rados.rados.0.smithi125.stdout:3295: seq_num 1120 ranges {423840=582771,1523009=586049,2657046=166794} 2021-03-24T22:07:00.480 INFO:tasks.rados.rados.0.smithi125.stdout:3295: writing smithi12556470-39 from 423840 to 1006611 tid 1 2021-03-24T22:07:00.487 INFO:tasks.rados.rados.0.smithi125.stdout:3295: writing smithi12556470-39 from 1523009 to 2109058 tid 2 2021-03-24T22:07:00.493 INFO:tasks.rados.rados.0.smithi125.stdout:3295: writing smithi12556470-39 from 2657046 to 2823840 tid 3 2021-03-24T22:07:00.493 INFO:tasks.rados.rados.0.smithi125.stdout:3296: write oid 35 current snap is 383 2021-03-24T22:07:00.493 INFO:tasks.rados.rados.0.smithi125.stdout:3295: finishing write tid 1 to smithi12556470-39 2021-03-24T22:07:00.494 INFO:tasks.rados.rados.0.smithi125.stdout:3295: finishing write tid 2 to smithi12556470-39 2021-03-24T22:07:00.494 INFO:tasks.rados.rados.0.smithi125.stdout:3296: seq_num 1121 ranges {440647=658020,1581491=446060,2800822=39825} 2021-03-24T22:07:00.500 INFO:tasks.rados.rados.0.smithi125.stdout:3296: writing smithi12556470-35 from 440647 to 1098667 tid 1 2021-03-24T22:07:00.505 INFO:tasks.rados.rados.0.smithi125.stdout:3296: writing smithi12556470-35 from 1581491 to 2027551 tid 2 2021-03-24T22:07:00.510 INFO:tasks.rados.rados.0.smithi125.stdout:3296: writing smithi12556470-35 from 2800822 to 2840647 tid 3 2021-03-24T22:07:00.510 INFO:tasks.rados.rados.0.smithi125.stdout:3297: write oid 3 current snap is 383 2021-03-24T22:07:00.510 INFO:tasks.rados.rados.0.smithi125.stdout:3294: expect (ObjNum 1118 snap 383 seq_num 1118) 2021-03-24T22:07:00.516 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:07:00.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:00.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:07:00.621 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:07:00.676 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:07:00.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:00.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:07:00.720 INFO:tasks.rados.rados.0.smithi125.stdout:3295: finishing write tid 3 to smithi12556470-39 2021-03-24T22:07:00.720 INFO:tasks.rados.rados.0.smithi125.stdout:3295: finishing write tid 4 to smithi12556470-39 2021-03-24T22:07:00.720 INFO:tasks.rados.rados.0.smithi125.stdout:3295: finishing write tid 5 to smithi12556470-39 2021-03-24T22:07:00.721 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 39 v 1468 (ObjNum 1120 snap 383 seq_num 1120) dirty exists 2021-03-24T22:07:00.721 INFO:tasks.rados.rados.0.smithi125.stdout:3295: left oid 39 (ObjNum 1120 snap 383 seq_num 1120) 2021-03-24T22:07:00.721 INFO:tasks.rados.rados.0.smithi125.stdout:3296: finishing write tid 1 to smithi12556470-35 2021-03-24T22:07:00.721 INFO:tasks.rados.rados.0.smithi125.stdout:3296: finishing write tid 2 to smithi12556470-35 2021-03-24T22:07:00.722 INFO:tasks.rados.rados.0.smithi125.stdout:3296: finishing write tid 3 to smithi12556470-35 2021-03-24T22:07:00.722 INFO:tasks.rados.rados.0.smithi125.stdout:3296: finishing write tid 4 to smithi12556470-35 2021-03-24T22:07:00.722 INFO:tasks.rados.rados.0.smithi125.stdout:3296: finishing write tid 5 to smithi12556470-35 2021-03-24T22:07:00.722 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 35 v 978 (ObjNum 1121 snap 383 seq_num 1121) dirty exists 2021-03-24T22:07:00.722 INFO:tasks.rados.rados.0.smithi125.stdout:3296: left oid 35 (ObjNum 1121 snap 383 seq_num 1121) 2021-03-24T22:07:00.723 INFO:tasks.rados.rados.0.smithi125.stdout:3297: seq_num 1122 ranges {457454=733269,1923620=589718} 2021-03-24T22:07:00.731 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:00 smithi142 conmon[20125]: cluster 2021-03-24T22:06:59.017233+0000 mgr.y ( 2021-03-24T22:07:00.732 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:00 smithi142 conmon[20125]: mgr.14144) 702 : cluster [DBG] pgmap v1435: 168 pgs: 168 active+clean; 547 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 764 KiB/s wr, 17 op/s 2021-03-24T22:07:00.732 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:00 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:07:00.732 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:00 smithi142 conmon[20125]: 22:06:59.516394+0000 mon.a (mon.0) 2088 : cluster [DBG] 2021-03-24T22:07:00.732 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:00 smithi142 conmon[20125]: osdmap e849: 8 total, 8 up, 8 in 2021-03-24T22:07:00.733 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:00 smithi142 conmon[20125]: audit 2021-03 2021-03-24T22:07:00.733 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:00 smithi142 conmon[20125]: -24T22:07:00.182636+0000 mon.c (mon. 2021-03-24T22:07:00.733 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:00 smithi142 conmon[20125]: 2) 121 : audit [INF] from='client.? v1:172.21.15.125:0/2041382552' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"}]: dispatch 2021-03-24T22:07:00.733 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:00 smithi142 conmon[20125]: audit 2021-03-24T22:07:00.183236+0000 2021-03-24T22:07:00.734 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:00 smithi142 conmon[20125]: mon.a (mon.0) 2089 : audit [INF] from='client.35909 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"}]: dispatch 2021-03-24T22:07:00.734 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:00 smithi142 conmon[20125]: 2021-03-24T22:07:00.734 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:00 smithi142 conmon[20125]: audit 2021-03-24T22:07:00.734 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:00 smithi142 conmon[20125]: 2021-03-24T22:07:00.450538+0000 mon.a 2021-03-24T22:07:00.735 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:00 smithi142 conmon[20125]: (mon.0) 2090 : audit [INF] from='client.35909 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"}]': finished 2021-03-24T22:07:00.735 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:00 smithi142 conmon[20125]: 2021-03-24T22:07:00.735 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:00 smithi142 conmon[20125]: cluster 2021-03-24T22:07:00.735 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:00 smithi142 conmon[20125]: 2021-03-24T22:07:00.450716+0000 2021-03-24T22:07:00.735 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:00 smithi142 conmon[20125]: mon.a (mon.0) 2091 : cluster [DBG] 2021-03-24T22:07:00.736 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:00 smithi142 conmon[20125]: osdmap e850: 8 total, 8 up, 8 in 2021-03-24T22:07:00.736 INFO:tasks.rados.rados.0.smithi125.stdout:3297: writing smithi12556470-3 from 457454 to 1190723 tid 1 2021-03-24T22:07:00.744 INFO:tasks.rados.rados.0.smithi125.stdout:3297: writing smithi12556470-3 from 1923620 to 2513338 tid 2 2021-03-24T22:07:00.744 INFO:tasks.rados.rados.0.smithi125.stdout:3294: done (3 left) 2021-03-24T22:07:00.744 INFO:tasks.rados.rados.0.smithi125.stdout:3295: done (2 left) 2021-03-24T22:07:00.745 INFO:tasks.rados.rados.0.smithi125.stdout:3296: done (1 left) 2021-03-24T22:07:00.745 INFO:tasks.rados.rados.0.smithi125.stdout:3298: delete oid 1 current snap is 383 2021-03-24T22:07:00.745 INFO:tasks.rados.rados.0.smithi125.stdout:3297: finishing write tid 1 to smithi12556470-3 2021-03-24T22:07:00.749 INFO:tasks.rados.rados.0.smithi125.stdout:3298: done (1 left) 2021-03-24T22:07:00.749 INFO:tasks.rados.rados.0.smithi125.stdout:3299: snap_create 2021-03-24T22:07:00.749 INFO:tasks.rados.rados.0.smithi125.stdout:3297: finishing write tid 2 to smithi12556470-3 2021-03-24T22:07:00.750 INFO:tasks.rados.rados.0.smithi125.stdout:3297: finishing write tid 3 to smithi12556470-3 2021-03-24T22:07:00.750 INFO:tasks.rados.rados.0.smithi125.stdout:3297: finishing write tid 4 to smithi12556470-3 2021-03-24T22:07:00.751 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 3 v 1218 (ObjNum 1122 snap 383 seq_num 1122) dirty exists 2021-03-24T22:07:00.751 INFO:tasks.rados.rados.0.smithi125.stdout:3297: left oid 3 (ObjNum 1122 snap 383 seq_num 1122) 2021-03-24T22:07:00.763 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.4 dump_blocked_ops 2021-03-24T22:07:00.781 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:07:00.794 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:07:00.794 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[19936]: 24T22:06:59.017233+0000 mgr.y (mgr.14144) 2021-03-24T22:07:00.795 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[19936]: 702 : cluster [DBG] pgmap v1435: 168 pgs: 168 active+clean; 547 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 764 KiB/s wr, 17 op/s 2021-03-24T22:07:00.795 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[19936]: 2021-03-24T22:07:00.795 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[19936]: cluster 2021-03-24T22:07:00.795 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[19936]: 2021-03-24T22:07:00.796 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[19936]: 2021-03-24T 2021-03-24T22:07:00.796 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[19936]: 22:06:59 2021-03-24T22:07:00.796 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[19936]: .516394+0000 mon.a 2021-03-24T22:07:00.797 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[19936]: (mon.0) 2021-03-24T22:07:00.797 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[19936]: 2088 : cluster [DBG] 2021-03-24T22:07:00.797 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[19936]: osdmap e849: 8 total, 8 up, 8 in 2021-03-24T22:07:00.797 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[19936]: audit 2021-03-24T22 2021-03-24T22:07:00.798 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[19936]: :07:00.182636+0000 mon.c (mon.2) 121 : audit [INF] from='client.? v1:172.21.15.125:0/2041382552' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"}]: dispatch 2021-03-24T22:07:00.798 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[19936]: audit 2021-03-24 2021-03-24T22:07:00.798 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[19936]: T22:07:00.183236+0000 mon.a (mon. 2021-03-24T22:07:00.798 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[19936]: 0) 2089 : audit [INF] from='client.35909 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"}]: dispatch 2021-03-24T22:07:00.799 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[19936]: audit 2021-03- 2021-03-24T22:07:00.799 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[19936]: 24T22:07:00.450538+0000 mon.a ( 2021-03-24T22:07:00.799 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[19936]: mon.0) 2090 : audit [INF] from='client.35909 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"}]': finished 2021-03-24T22:07:00.799 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:07:00.800 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[19936]: 24T22:07:00.450716+0000 mon.a (mon 2021-03-24T22:07:00.800 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[19936]: .0) 2091 : cluster [DBG] osdmap e850: 8 total, 8 up, 8 in 2021-03-24T22:07:00.800 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:07:00.801 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[24927]: 03-24T22:06:59.017233+0000 2021-03-24T22:07:00.801 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[24927]: mgr.y (mgr.14144 2021-03-24T22:07:00.801 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[24927]: ) 702 : cluster 2021-03-24T22:07:00.801 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[24927]: [DBG] pgmap v1435: 168 pgs: 168 active+clean; 547 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 764 KiB/s wr, 17 op/s 2021-03-24T22:07:00.801 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[24927]: cluster 2021 2021-03-24T22:07:00.802 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[24927]: -03-24T22: 2021-03-24T22:07:00.802 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[24927]: 06:59.516394+0000 2021-03-24T22:07:00.802 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[24927]: mon.a (mon.0) 2021-03-24T22:07:00.803 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[24927]: 2088 : cluster [DBG] 2021-03-24T22:07:00.803 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[24927]: osdmap e849: 8 total, 8 up, 8 in 2021-03-24T22:07:00.803 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[24927]: audit 2021-03-24T22:07:00.803 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[24927]: 2021-03-24 2021-03-24T22:07:00.804 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[24927]: T22:07: 2021-03-24T22:07:00.804 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[24927]: 00.182636+0000 mon.c ( 2021-03-24T22:07:00.805 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[24927]: mon.2) 121 : audit 2021-03-24T22:07:00.805 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[24927]: [INF] from='client.? v1:172.21.15.125:0/2041382552' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"}]: dispatch 2021-03-24T22:07:00.805 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[24927]: audit 2021 2021-03-24T22:07:00.805 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[24927]: -03-24T22: 2021-03-24T22:07:00.805 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[24927]: 07:00.183236+0000 2021-03-24T22:07:00.806 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[24927]: mon.a (mon. 2021-03-24T22:07:00.806 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[24927]: 0) 2089 : audit 2021-03-24T22:07:00.806 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[24927]: [INF] from='client.35909 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"}]: dispatch 2021-03-24T22:07:00.807 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[24927]: audit 2021 2021-03-24T22:07:00.807 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[24927]: -03-24T22:07 2021-03-24T22:07:00.807 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[24927]: :00.450538+0000 mon.a 2021-03-24T22:07:00.807 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[24927]: (mon.0) 2090 : audit 2021-03-24T22:07:00.808 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[24927]: [INF] from='client.35909 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"}]': finished 2021-03-24T22:07:00.808 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[24927]: cluster 2021-03-24T22:07:00.808 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[24927]: 2021-03-24T22 2021-03-24T22:07:00.808 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[24927]: :07:00.450716+0000 2021-03-24T22:07:00.809 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[24927]: mon.a (mon.0) 2091 : 2021-03-24T22:07:00.809 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:00 smithi125 conmon[24927]: cluster [DBG] osdmap e850: 8 total, 8 up, 8 in 2021-03-24T22:07:00.831 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:07:00.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:00.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:07:00.938 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:07:00.981 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:07:00.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:00.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:07:01.065 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T22:07:01.086 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:07:01.142 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:07:01.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:01.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:07:01.249 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:07:01.308 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:07:01.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:01.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:07:01.415 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:07:01.447 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-24T22:07:01.469 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:07:01.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:01.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:07:01.566 INFO:tasks.rados.rados.0.smithi125.stdout:3297: done (1 left) 2021-03-24T22:07:01.566 INFO:tasks.rados.rados.0.smithi125.stdout:3299: done (0 left) 2021-03-24T22:07:01.567 INFO:tasks.rados.rados.0.smithi125.stdout:3300: write (excl) oid 33 current snap is 384 2021-03-24T22:07:01.567 INFO:tasks.rados.rados.0.smithi125.stdout:3300: seq_num 1124 ranges {491068=483767,1640584=593387,2650385=240683} 2021-03-24T22:07:01.571 INFO:tasks.rados.rados.0.smithi125.stdout:3300: writing smithi12556470-33 from 491068 to 974835 tid 1 2021-03-24T22:07:01.575 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:07:01.580 INFO:tasks.rados.rados.0.smithi125.stdout:3300: writing smithi12556470-33 from 1640584 to 2233971 tid 2 2021-03-24T22:07:01.583 INFO:tasks.rados.rados.0.smithi125.stdout:3300: writing smithi12556470-33 from 2650385 to 2891068 tid 3 2021-03-24T22:07:01.584 INFO:tasks.rados.rados.0.smithi125.stdout:3301: read oid 41 snap -1 2021-03-24T22:07:01.584 INFO:tasks.rados.rados.0.smithi125.stdout:3301: expect (ObjNum 1078 snap 361 seq_num 1078) 2021-03-24T22:07:01.584 INFO:tasks.rados.rados.0.smithi125.stdout:3302: delete oid 45 current snap is 384 2021-03-24T22:07:01.587 INFO:tasks.rados.rados.0.smithi125.stdout:3302: done (2 left) 2021-03-24T22:07:01.588 INFO:tasks.rados.rados.0.smithi125.stdout:3303: read oid 42 snap 380 2021-03-24T22:07:01.588 INFO:tasks.rados.rados.0.smithi125.stdout:3303: expect (ObjNum 1104 snap 377 seq_num 1104) 2021-03-24T22:07:01.588 INFO:tasks.rados.rados.0.smithi125.stdout:3304: snap_remove snap 344 2021-03-24T22:07:01.634 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:07:01.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:01.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:07:01.738 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:07:01.788 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:07:01.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:01.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:07:01.894 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:07:01.960 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:07:01.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:01.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:07:02.065 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:07:02.127 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:07:02.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:02.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:07:02.233 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:07:02.294 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:07:02.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:02.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:07:02.348 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:07:02.400 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:07:02.459 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:07:02.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:02.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:07:02.565 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:07:02.574 INFO:tasks.rados.rados.0.smithi125.stdout:3304: done (3 left) 2021-03-24T22:07:02.575 INFO:tasks.rados.rados.0.smithi125.stdout:3305: copy_from oid 30 from oid 44 current snap is 384 2021-03-24T22:07:02.578 INFO:tasks.rados.rados.0.smithi125.stdout:3303: expect (ObjNum 1104 snap 377 seq_num 1104) 2021-03-24T22:07:02.626 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:07:02.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:02.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:07:02.682 INFO:tasks.rados.rados.0.smithi125.stdout:3300: finishing write tid 1 to smithi12556470-33 2021-03-24T22:07:02.682 INFO:tasks.rados.rados.0.smithi125.stdout:3301: expect (ObjNum 1078 snap 361 seq_num 1078) 2021-03-24T22:07:02.722 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:02 smithi142 conmon[20125]: cluster 2021-03-24T22:07:02.723 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:02 smithi142 conmon[20125]: 2021-03-24T22:07:02.723 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:02 smithi142 conmon[20125]: 2021 2021-03-24T22:07:02.723 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:02 smithi142 conmon[20125]: - 2021-03-24T22:07:02.723 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:02 smithi142 conmon[20125]: 03 2021-03-24T22:07:02.724 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:02 smithi142 conmon[20125]: - 2021-03-24T22:07:02.724 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:02 smithi142 conmon[20125]: 24 2021-03-24T22:07:02.724 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:02 smithi142 conmon[20125]: T 2021-03-24T22:07:02.724 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:02 smithi142 conmon[20125]: 22 2021-03-24T22:07:02.725 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:02 smithi142 conmon[20125]: : 2021-03-24T22:07:02.725 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:02 smithi142 conmon[20125]: 07 2021-03-24T22:07:02.725 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:02 smithi142 conmon[20125]: : 2021-03-24T22:07:02.725 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:02 smithi142 conmon[20125]: 01.018145+0000 mgr.y (mgr.14144) 703 : cluster 2021-03-24T22:07:02.725 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:02 smithi142 conmon[20125]: [DBG] pgmap v1438: 168 pgs: 168 active+clean; 566 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 4.2 MiB/s wr, 23 op/s 2021-03-24T22:07:02.726 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:02 smithi142 conmon[20125]: cluster 2021-03-24 2021-03-24T22:07:02.726 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:02 smithi142 conmon[20125]: T22:07:01.421910+0000 osd.2 (osd.2) 139 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:07:02.726 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:02 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:07:02.726 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:02 smithi142 conmon[20125]: -24T22:07:01.427251+0000 osd.2 2021-03-24T22:07:02.727 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:02 smithi142 conmon[20125]: (osd.2) 140 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:07:02.727 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:02 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:07:02.727 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:02 smithi142 conmon[20125]: 03-24T22:07:01.562538+0000 2021-03-24T22:07:02.727 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:02 smithi142 conmon[20125]: mon.a (mon.0) 2092 : cluster [DBG] osdmap e851: 8 total, 8 up, 8 in 2021-03-24T22:07:02.727 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:02 smithi142 conmon[20125]: 2021-03-24T22:07:02.731 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:07:02.791 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:07:02.792 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:02 smithi125 conmon[24927]: cluster 2021-03-24T22:07:01.018145+0000 mgr.y ( 2021-03-24T22:07:02.793 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:02 smithi125 conmon[24927]: mgr.14144) 703 : cluster [DBG] pgmap v1438: 168 pgs: 168 active+clean; 566 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 4.2 MiB/s wr, 23 op/s 2021-03-24T22:07:02.793 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:02 smithi125 conmon[24927]: cluster 2021-03-24T22:07:01.421910+0000 osd.2 (osd.2) 139 : 2021-03-24T22:07:02.793 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:02 smithi125 conmon[24927]: cluster [DBG] purged_snaps scrub starts 2021-03-24T22:07:02.793 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:02 smithi125 conmon[24927]: cluster 2021-03-24T22:07:01.427251+0000 osd.2 (osd.2) 140 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:07:02.794 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:02 smithi125 conmon[24927]: cluster 2021-03-24T22:07:01.562538+0000 mon.a (mon.0) 2092 : cluster [DBG] osdmap e851: 8 total, 8 up, 8 in 2021-03-24T22:07:02.794 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:02 smithi125 conmon[19936]: cluster 2021-03-24T22:07:01.018145+0000 2021-03-24T22:07:02.794 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:02 smithi125 conmon[19936]: mgr.y (mgr.14144) 703 : cluster [DBG] pgmap v1438: 168 pgs: 168 active+clean; 566 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 4.2 MiB/s wr, 23 op/s 2021-03-24T22:07:02.795 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:02 smithi125 conmon[19936]: cluster 2021-03-24T22:07:01.421910+0000 osd.2 (osd.2) 139 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:07:02.795 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:02 smithi125 conmon[19936]: cluster 2021-03-24T22:07:01.427251+0000 2021-03-24T22:07:02.795 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:02 smithi125 conmon[19936]: osd.2 (osd.2) 140 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:07:02.795 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:02 smithi125 conmon[19936]: 2021-03-24T22:07:02.795 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:02 smithi125 conmon[19936]: cluster 2021-03-24T22:07:02.796 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:02 smithi125 conmon[19936]: 2021-03-24T22:07:01.562538+0000 mon.a 2021-03-24T22:07:02.796 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:02 smithi125 conmon[19936]: (mon.0) 2092 : cluster [DBG] 2021-03-24T22:07:02.796 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:02 smithi125 conmon[19936]: osdmap e851: 8 total, 8 up, 8 in 2021-03-24T22:07:02.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:02.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:07:02.805 INFO:tasks.rados.rados.0.smithi125.stdout:3300: finishing write tid 2 to smithi12556470-33 2021-03-24T22:07:02.806 INFO:tasks.rados.rados.0.smithi125.stdout:3300: finishing write tid 3 to smithi12556470-33 2021-03-24T22:07:02.806 INFO:tasks.rados.rados.0.smithi125.stdout:3300: finishing write tid 4 to smithi12556470-33 2021-03-24T22:07:02.806 INFO:tasks.rados.rados.0.smithi125.stdout:3300: finishing write tid 5 to smithi12556470-33 2021-03-24T22:07:02.806 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 33 v 1472 (ObjNum 1124 snap 384 seq_num 1124) dirty exists 2021-03-24T22:07:02.807 INFO:tasks.rados.rados.0.smithi125.stdout:3300: left oid 33 (ObjNum 1124 snap 384 seq_num 1124) 2021-03-24T22:07:02.807 INFO:tasks.rados.rados.0.smithi125.stdout:3305: finishing copy_from to smithi12556470-30 2021-03-24T22:07:02.807 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 30 v 1455 (ObjNum 1092 snap 370 seq_num 1092) dirty exists 2021-03-24T22:07:02.807 INFO:tasks.rados.rados.0.smithi125.stdout:3305: finishing copy_from racing read to smithi12556470-30 2021-03-24T22:07:02.807 INFO:tasks.rados.rados.0.smithi125.stdout:3300: done (3 left) 2021-03-24T22:07:02.808 INFO:tasks.rados.rados.0.smithi125.stdout:3301: done (2 left) 2021-03-24T22:07:02.808 INFO:tasks.rados.rados.0.smithi125.stdout:3303: done (1 left) 2021-03-24T22:07:02.808 INFO:tasks.rados.rados.0.smithi125.stdout:3305: done (0 left) 2021-03-24T22:07:02.808 INFO:tasks.rados.rados.0.smithi125.stdout:3306: write (excl) oid 17 current snap is 384 2021-03-24T22:07:02.809 INFO:tasks.rados.rados.0.smithi125.stdout:3306: seq_num 1126 ranges {524682=750618,1757548=597056} 2021-03-24T22:07:02.814 INFO:tasks.rados.rados.0.smithi125.stdout:3306: writing smithi12556470-17 from 524682 to 1275300 tid 1 2021-03-24T22:07:02.823 INFO:tasks.rados.rados.0.smithi125.stdout:3306: writing smithi12556470-17 from 1757548 to 2354604 tid 2 2021-03-24T22:07:02.823 INFO:tasks.rados.rados.0.smithi125.stdout:3307: rollback oid 20 current snap is 384 2021-03-24T22:07:02.823 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 20 to 373 2021-03-24T22:07:02.824 INFO:tasks.rados.rados.0.smithi125.stdout:3306: finishing write tid 1 to smithi12556470-17 2021-03-24T22:07:02.825 INFO:tasks.rados.rados.0.smithi125.stdout:3308: write (excl) oid 37 current snap is 384 2021-03-24T22:07:02.825 INFO:tasks.rados.rados.0.smithi125.stdout:3308: seq_num 1127 ranges {541489=425867,1416030=740714,2732654=208835} 2021-03-24T22:07:02.827 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T22:07:02.829 INFO:tasks.rados.rados.0.smithi125.stdout:3308: writing smithi12556470-37 from 541489 to 967356 tid 1 2021-03-24T22:07:02.837 INFO:tasks.rados.rados.0.smithi125.stdout:3308: writing smithi12556470-37 from 1416030 to 2156744 tid 2 2021-03-24T22:07:02.843 INFO:tasks.rados.rados.0.smithi125.stdout:3308: writing smithi12556470-37 from 2732654 to 2941489 tid 3 2021-03-24T22:07:02.844 INFO:tasks.rados.rados.0.smithi125.stdout:3309: rollback oid 18 current snap is 384 2021-03-24T22:07:02.844 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 18 to 360 2021-03-24T22:07:02.844 INFO:tasks.rados.rados.0.smithi125.stdout:3306: finishing write tid 2 to smithi12556470-17 2021-03-24T22:07:02.844 INFO:tasks.rados.rados.0.smithi125.stdout:3307: finishing rollback tid 0 to smithi12556470-20 2021-03-24T22:07:02.845 INFO:tasks.rados.rados.0.smithi125.stdout:3307: finishing rollback tid 1 to smithi12556470-20 2021-03-24T22:07:02.846 INFO:tasks.rados.rados.0.smithi125.stdout:3307: finishing rollback tid 2 to smithi12556470-20 2021-03-24T22:07:02.846 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 20 v 1461 (ObjNum 808 snap 265 seq_num 808) dirty exists 2021-03-24T22:07:02.846 INFO:tasks.rados.rados.0.smithi125.stdout:3308: finishing write tid 1 to smithi12556470-37 2021-03-24T22:07:02.846 INFO:tasks.rados.rados.0.smithi125.stdout:3306: finishing write tid 3 to smithi12556470-17 2021-03-24T22:07:02.847 INFO:tasks.rados.rados.0.smithi125.stdout:3306: finishing write tid 4 to smithi12556470-17 2021-03-24T22:07:02.847 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 17 v 1313 (ObjNum 1126 snap 384 seq_num 1126) dirty exists 2021-03-24T22:07:02.848 INFO:tasks.rados.rados.0.smithi125.stdout:3306: left oid 17 (ObjNum 1126 snap 384 seq_num 1126) 2021-03-24T22:07:02.848 INFO:tasks.rados.rados.0.smithi125.stdout:3306: done (3 left) 2021-03-24T22:07:02.848 INFO:tasks.rados.rados.0.smithi125.stdout:3307: done (2 left) 2021-03-24T22:07:02.848 INFO:tasks.rados.rados.0.smithi125.stdout:3310: rollback oid 9 current snap is 384 2021-03-24T22:07:02.849 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 9 to 382 2021-03-24T22:07:02.849 INFO:tasks.rados.rados.0.smithi125.stdout:3311: rollback oid 27 current snap is 384 2021-03-24T22:07:02.849 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 27 to 351 2021-03-24T22:07:02.849 INFO:tasks.rados.rados.0.smithi125.stdout:3312: write (excl) oid 50 current snap is 384 2021-03-24T22:07:02.849 INFO:tasks.rados.rados.0.smithi125.stdout:3312: seq_num 1128 ranges {558296=501116,1474512=600725,2760077=198219} 2021-03-24T22:07:02.851 INFO:tasks.rados.rados.0.smithi125.stdout:3312: writing smithi12556470-50 from 558296 to 1059412 tid 1 2021-03-24T22:07:02.857 INFO:tasks.rados.rados.0.smithi125.stdout:3312: writing smithi12556470-50 from 1474512 to 2075237 tid 2 2021-03-24T22:07:02.858 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T22:07:02.863 INFO:tasks.rados.rados.0.smithi125.stdout:3312: writing smithi12556470-50 from 2760077 to 2958296 tid 3 2021-03-24T22:07:02.863 INFO:tasks.rados.rados.0.smithi125.stdout:3313: delete oid 43 current snap is 384 2021-03-24T22:07:02.864 INFO:tasks.rados.rados.0.smithi125.stdout:3308: finishing write tid 2 to smithi12556470-37 2021-03-24T22:07:02.864 INFO:tasks.rados.rados.0.smithi125.stdout:3309: finishing rollback tid 0 to smithi12556470-18 2021-03-24T22:07:02.864 INFO:tasks.rados.rados.0.smithi125.stdout:3309: finishing rollback tid 1 to smithi12556470-18 2021-03-24T22:07:02.864 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 18 v 1440 (ObjNum 1117783712 snap 21916 seq_num 0) dirty dne 2021-03-24T22:07:02.865 INFO:tasks.rados.rados.0.smithi125.stdout:3311: finishing rollback tid 0 to smithi12556470-27 2021-03-24T22:07:02.865 INFO:tasks.rados.rados.0.smithi125.stdout:3308: finishing write tid 3 to smithi12556470-37 2021-03-24T22:07:02.865 INFO:tasks.rados.rados.0.smithi125.stdout:3310: finishing rollback tid 1 to smithi12556470-9 2021-03-24T22:07:02.865 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 9 v 2076 (ObjNum 0 snap 21916 seq_num 1117465072) dirty dne 2021-03-24T22:07:02.866 INFO:tasks.rados.rados.0.smithi125.stdout:3308: finishing write tid 4 to smithi12556470-37 2021-03-24T22:07:02.866 INFO:tasks.rados.rados.0.smithi125.stdout:3311: finishing rollback tid 1 to smithi12556470-27 2021-03-24T22:07:02.866 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 27 v 1440 (ObjNum 1117691536 snap 21916 seq_num 1117602112) dirty dne 2021-03-24T22:07:02.866 INFO:tasks.rados.rados.0.smithi125.stdout:3308: finishing write tid 5 to smithi12556470-37 2021-03-24T22:07:02.866 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 37 v 1462 (ObjNum 1127 snap 384 seq_num 1127) dirty exists 2021-03-24T22:07:02.867 INFO:tasks.rados.rados.0.smithi125.stdout:3308: left oid 37 (ObjNum 1127 snap 384 seq_num 1127) 2021-03-24T22:07:02.867 INFO:tasks.rados.rados.0.smithi125.stdout:3312: finishing write tid 1 to smithi12556470-50 2021-03-24T22:07:02.867 INFO:tasks.rados.rados.0.smithi125.stdout:3312: finishing write tid 2 to smithi12556470-50 2021-03-24T22:07:02.868 INFO:tasks.rados.rados.0.smithi125.stdout:3312: finishing write tid 3 to smithi12556470-50 2021-03-24T22:07:02.868 INFO:tasks.rados.rados.0.smithi125.stdout:3308: done (5 left) 2021-03-24T22:07:02.868 INFO:tasks.rados.rados.0.smithi125.stdout:3309: done (4 left) 2021-03-24T22:07:02.869 INFO:tasks.rados.rados.0.smithi125.stdout:3310: done (3 left) 2021-03-24T22:07:02.869 INFO:tasks.rados.rados.0.smithi125.stdout:3311: done (2 left) 2021-03-24T22:07:02.869 INFO:tasks.rados.rados.0.smithi125.stdout:3313: done (1 left) 2021-03-24T22:07:02.869 INFO:tasks.rados.rados.0.smithi125.stdout:3314: snap_remove snap 366 2021-03-24T22:07:02.878 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T22:07:02.896 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T22:07:02.899 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:07:02.934 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T22:07:02.948 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:07:02.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:02.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:07:02.957 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T22:07:02.982 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T22:07:03.004 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T22:07:03.006 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-24T22:07:03.007 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-24T22:07:03.007 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-24T22:07:03.007 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-24T22:07:03.007 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-24T22:07:03.008 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-24T22:07:03.008 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-24T22:07:03.008 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-24T22:07:03.053 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:07:03.105 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:07:03.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:03.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:07:03.211 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:07:03.269 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:07:03.274 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2021-03-24T22:07:03.274 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T22:07:03.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:03.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:07:03.377 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:07:03.431 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:07:03.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:03.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:07:03.538 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:07:03.592 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:03 smithi125 conmon[19936]: cluster 2021-03-24T22:07:02.569089+0000 mon.a (mon.0) 2021-03-24T22:07:03.593 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:03 smithi125 conmon[19936]: 2093 : cluster [DBG] osdmap e852: 8 total, 8 up, 8 in 2021-03-24T22:07:03.593 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:03 smithi125 conmon[19936]: audit 2021-03-24T22:07:02.800032+0000 mon.c (mon.2) 122 : audit [DBG] from='client.? v1:172.21.15.125:0/1902478526' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:07:03.601 INFO:tasks.rados.rados.0.smithi125.stdout:3314: done (1 left) 2021-03-24T22:07:03.601 INFO:tasks.rados.rados.0.smithi125.stdout:3315: read oid 16 snap -1 2021-03-24T22:07:03.602 INFO:tasks.rados.rados.0.smithi125.stdout:3315: expect (ObjNum 1072 snap 359 seq_num 1072) 2021-03-24T22:07:03.602 INFO:tasks.rados.rados.0.smithi125.stdout:3316: read oid 12 snap -1 2021-03-24T22:07:03.602 INFO:tasks.rados.rados.0.smithi125.stdout:3316: expect (ObjNum 1001 snap 331 seq_num 1001) 2021-03-24T22:07:03.602 INFO:tasks.rados.rados.0.smithi125.stdout:3317: copy_from oid 21 from oid 45 current snap is 384 2021-03-24T22:07:03.603 INFO:tasks.rados.rados.0.smithi125.stdout:3318: snap_remove snap 368 2021-03-24T22:07:03.603 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:07:03.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:03.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:07:03.714 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:07:03.775 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:07:03.779 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:03 smithi142 conmon[20125]: cluster 2021-03-24T22:07:02. 2021-03-24T22:07:03.779 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:03 smithi142 conmon[20125]: 569089+0000 mon.a (mon.0) 2093 : cluster [DBG] osdmap e852: 8 total, 8 up, 8 in 2021-03-24T22:07:03.780 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:03 smithi142 conmon[20125]: audit 2021-03-24T22: 2021-03-24T22:07:03.780 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:03 smithi142 conmon[20125]: 07:02.800032+0000 mon.c (mon.2) 122 : audit 2021-03-24T22:07:03.780 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:03 smithi142 conmon[20125]: [DBG] from='client.? v1:172.21.15.125:0/1902478526' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:07:03.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:03.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:07:03.883 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:07:03.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:03 smithi125 conmon[24927]: cluster 2021-03-24T22:07:02.569089+0000 2021-03-24T22:07:03.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:03 smithi125 conmon[24927]: mon.a (mon.0) 2093 : cluster [DBG] osdmap e852: 8 total, 8 up, 8 in 2021-03-24T22:07:03.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:03 smithi125 conmon[24927]: audit 2021-03-24T22:07:02.800032+0000 mon.c (mon.2) 122 : audit [DBG] from='client.? v1:172.21.15.125:0/1902478526' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:07:03.937 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:07:03.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:03.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:07:04.043 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:07:04.096 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:07:04.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:04.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:07:04.204 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:07:04.258 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:07:04.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:04.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:07:04.366 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:07:04.404 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.4 dump_historic_ops 2021-03-24T22:07:04.421 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:07:04.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:04.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:07:04.528 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:07:04.587 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:07:04.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:04.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:07:04.608 INFO:tasks.rados.rados.0.smithi125.stdout:3318: done (4 left) 2021-03-24T22:07:04.608 INFO:tasks.rados.rados.0.smithi125.stdout:3319: read oid 36 snap -1 2021-03-24T22:07:04.609 INFO:tasks.rados.rados.0.smithi125.stdout:3319: expect deleted 2021-03-24T22:07:04.609 INFO:tasks.rados.rados.0.smithi125.stdout:3320: rollback oid 13 current snap is 384 2021-03-24T22:07:04.609 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 13 to 370 2021-03-24T22:07:04.609 INFO:tasks.rados.rados.0.smithi125.stdout:3321: delete oid 33 current snap is 384 2021-03-24T22:07:04.610 INFO:tasks.rados.rados.0.smithi125.stdout:3312: finishing write tid 4 to smithi12556470-50 2021-03-24T22:07:04.610 INFO:tasks.rados.rados.0.smithi125.stdout:3312: finishing write tid 5 to smithi12556470-50 2021-03-24T22:07:04.610 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 50 v 1465 (ObjNum 1128 snap 384 seq_num 1128) dirty exists 2021-03-24T22:07:04.610 INFO:tasks.rados.rados.0.smithi125.stdout:3312: left oid 50 (ObjNum 1128 snap 384 seq_num 1128) 2021-03-24T22:07:04.610 INFO:tasks.rados.rados.0.smithi125.stdout:3315: expect (ObjNum 1072 snap 359 seq_num 1072) 2021-03-24T22:07:04.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:04 smithi125 conmon[19936]: cluster 2021-03-24T22:07:03.018867+0000 2021-03-24T22:07:04.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:04 smithi125 conmon[19936]: mgr.y (mgr.14144) 704 : cluster [DBG] pgmap v1441: 168 pgs: 168 active+clean; 570 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 4.8 MiB/s wr, 23 op/s 2021-03-24T22:07:04.662 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:04 smithi125 conmon[19936]: cluster 2021-03-24T22:07:03.592717+0000 mon.a ( 2021-03-24T22:07:04.663 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:04 smithi125 conmon[19936]: mon.0) 2094 : cluster [DBG] osdmap e853: 8 total, 8 up, 8 in 2021-03-24T22:07:04.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:04 smithi125 conmon[24927]: cluster 2021-03-24T22:07:03 2021-03-24T22:07:04.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:04 smithi125 conmon[24927]: .018867+0000 mgr.y (mgr.14144) 704 : cluster [DBG] pgmap v1441: 168 pgs: 168 active+clean; 570 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 4.8 MiB/s wr, 23 op/s 2021-03-24T22:07:04.663 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:04 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:07:04.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:04 smithi125 conmon[24927]: 07:03.592717+0000 mon.a 2021-03-24T22:07:04.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:04 smithi125 conmon[24927]: (mon.0) 2094 : cluster 2021-03-24T22:07:04.664 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:04 smithi125 conmon[24927]: [DBG] osdmap e853: 8 total, 8 up, 8 in 2021-03-24T22:07:04.695 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:07:04.710 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T22:07:04.735 INFO:tasks.rados.rados.0.smithi125.stdout:3317: finishing copy_from to smithi12556470-21 2021-03-24T22:07:04.736 INFO:tasks.rados.rados.0.smithi125.stdout:3317: got expected ENOENT (src dne) 2021-03-24T22:07:04.736 INFO:tasks.rados.rados.0.smithi125.stdout:3317: finishing copy_from racing read to smithi12556470-21 2021-03-24T22:07:04.736 INFO:tasks.rados.rados.0.smithi125.stdout:3312: done (6 left) 2021-03-24T22:07:04.736 INFO:tasks.rados.rados.0.smithi125.stdout:3315: done (5 left) 2021-03-24T22:07:04.737 INFO:tasks.rados.rados.0.smithi125.stdout:3317: done (4 left) 2021-03-24T22:07:04.737 INFO:tasks.rados.rados.0.smithi125.stdout:3321: done (3 left) 2021-03-24T22:07:04.737 INFO:tasks.rados.rados.0.smithi125.stdout:3322: write oid 7 current snap is 384 2021-03-24T22:07:04.737 INFO:tasks.rados.rados.0.smithi125.stdout:3316: expect (ObjNum 1001 snap 331 seq_num 1001) 2021-03-24T22:07:04.746 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:07:04.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:04.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:07:04.834 INFO:tasks.rados.rados.0.smithi125.stdout:3322: seq_num 1131 ranges {608717=726863,1933605=748052} 2021-03-24T22:07:04.842 INFO:tasks.rados.rados.0.smithi125.stdout:3322: writing smithi12556470-7 from 608717 to 1335580 tid 1 2021-03-24T22:07:04.851 INFO:tasks.rados.rados.0.smithi125.stdout:3322: writing smithi12556470-7 from 1933605 to 2681657 tid 2 2021-03-24T22:07:04.851 INFO:tasks.rados.rados.0.smithi125.stdout:3316: done (3 left) 2021-03-24T22:07:04.852 INFO:tasks.rados.rados.0.smithi125.stdout:3323: delete oid 22 current snap is 384 2021-03-24T22:07:04.852 INFO:tasks.rados.rados.0.smithi125.stdout:3320: finishing rollback tid 0 to smithi12556470-13 2021-03-24T22:07:04.852 INFO:tasks.rados.rados.0.smithi125.stdout:3320: finishing rollback tid 1 to smithi12556470-13 2021-03-24T22:07:04.852 INFO:tasks.rados.rados.0.smithi125.stdout:3320: finishing rollback tid 2 to smithi12556470-13 2021-03-24T22:07:04.852 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 13 v 1466 (ObjNum 988 snap 328 seq_num 988) dirty exists 2021-03-24T22:07:04.853 INFO:tasks.rados.rados.0.smithi125.stdout:3322: finishing write tid 1 to smithi12556470-7 2021-03-24T22:07:04.853 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:07:04.854 INFO:tasks.rados.rados.0.smithi125.stdout:3319: done (3 left) 2021-03-24T22:07:04.855 INFO:tasks.rados.rados.0.smithi125.stdout:3320: done (2 left) 2021-03-24T22:07:04.855 INFO:tasks.rados.rados.0.smithi125.stdout:3323: done (1 left) 2021-03-24T22:07:04.855 INFO:tasks.rados.rados.0.smithi125.stdout:3324: write (excl) oid 47 current snap is 384 2021-03-24T22:07:04.855 INFO:tasks.rados.rados.0.smithi125.stdout:3324: seq_num 1133 ranges {642331=593714,1766922=751721} 2021-03-24T22:07:04.861 INFO:tasks.rados.rados.0.smithi125.stdout:3324: writing smithi12556470-47 from 642331 to 1236045 tid 1 2021-03-24T22:07:04.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:04 smithi142 conmon[20125]: cluster 2021-03-24T22:07:03.018867+0000 mgr.y (mgr.14144) 704 : cluster [DBG] pgmap v1441: 168 pgs: 168 active+clean; 570 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 4.8 MiB/s wr, 23 op/s 2021-03-24T22:07:04.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:04 smithi142 conmon[20125]: cluster 2021-03-24T22:07:03.592717+0000 mon.a (mon.0) 2094 : cluster [DBG] osdmap e853: 8 total, 8 up, 8 in 2021-03-24T22:07:04.869 INFO:tasks.rados.rados.0.smithi125.stdout:3324: writing smithi12556470-47 from 1766922 to 2518643 tid 2 2021-03-24T22:07:04.869 INFO:tasks.rados.rados.0.smithi125.stdout:3325: snap_create 2021-03-24T22:07:04.870 INFO:tasks.rados.rados.0.smithi125.stdout:3322: finishing write tid 2 to smithi12556470-7 2021-03-24T22:07:04.870 INFO:tasks.rados.rados.0.smithi125.stdout:3322: finishing write tid 3 to smithi12556470-7 2021-03-24T22:07:04.870 INFO:tasks.rados.rados.0.smithi125.stdout:3322: finishing write tid 4 to smithi12556470-7 2021-03-24T22:07:04.870 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 7 v 2096 (ObjNum 1131 snap 384 seq_num 1131) dirty exists 2021-03-24T22:07:04.870 INFO:tasks.rados.rados.0.smithi125.stdout:3322: left oid 7 (ObjNum 1131 snap 384 seq_num 1131) 2021-03-24T22:07:04.871 INFO:tasks.rados.rados.0.smithi125.stdout:3324: finishing write tid 1 to smithi12556470-47 2021-03-24T22:07:04.874 INFO:tasks.rados.rados.0.smithi125.stdout:3324: finishing write tid 2 to smithi12556470-47 2021-03-24T22:07:04.875 INFO:tasks.rados.rados.0.smithi125.stdout:3324: finishing write tid 3 to smithi12556470-47 2021-03-24T22:07:04.875 INFO:tasks.rados.rados.0.smithi125.stdout:3324: finishing write tid 4 to smithi12556470-47 2021-03-24T22:07:04.875 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 47 v 1444 (ObjNum 1133 snap 384 seq_num 1133) dirty exists 2021-03-24T22:07:04.876 INFO:tasks.rados.rados.0.smithi125.stdout:3324: left oid 47 (ObjNum 1133 snap 384 seq_num 1133) 2021-03-24T22:07:04.904 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:07:04.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:04.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:07:05.009 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:07:05.062 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:07:05.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:05.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:07:05.171 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:07:05.235 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:07:05.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:05.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:07:05.340 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:07:05.394 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:07:05.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:05.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:07:05.500 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:07:05.550 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:07:05.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:05.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:07:05.616 INFO:tasks.rados.rados.0.smithi125.stdout:3322: done (2 left) 2021-03-24T22:07:05.617 INFO:tasks.rados.rados.0.smithi125.stdout:3324: done (1 left) 2021-03-24T22:07:05.617 INFO:tasks.rados.rados.0.smithi125.stdout:3325: done (0 left) 2021-03-24T22:07:05.617 INFO:tasks.rados.rados.0.smithi125.stdout:3326: delete oid 3 current snap is 385 2021-03-24T22:07:05.619 INFO:tasks.rados.rados.0.smithi125.stdout:3326: done (0 left) 2021-03-24T22:07:05.619 INFO:tasks.rados.rados.0.smithi125.stdout:3327: write (excl) oid 23 current snap is 385 2021-03-24T22:07:05.619 INFO:tasks.rados.rados.0.smithi125.stdout:3327: seq_num 1135 ranges {675945=744212,2167533=755390} 2021-03-24T22:07:05.627 INFO:tasks.rados.rados.0.smithi125.stdout:3327: writing smithi12556470-23 from 675945 to 1420157 tid 1 2021-03-24T22:07:05.636 INFO:tasks.rados.rados.0.smithi125.stdout:3327: writing smithi12556470-23 from 2167533 to 2922923 tid 2 2021-03-24T22:07:05.636 INFO:tasks.rados.rados.0.smithi125.stdout:3328: snap_remove snap 327 2021-03-24T22:07:05.657 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:07:05.710 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:07:05.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:05.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:07:05.816 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:07:05.830 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:05 smithi125 conmon[24927]: cluster 2021-03-24T22:07:04.600668+0000 mon.a 2021-03-24T22:07:05.831 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:05 smithi125 conmon[24927]: (mon.0) 2095 : cluster [DBG] osdmap e854: 8 total, 8 up, 8 in 2021-03-24T22:07:05.831 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:05 smithi125 conmon[24927]: audit 2021-03-24T22:07:04.853739+0000 mon.a ( 2021-03-24T22:07:05.831 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:05 smithi125 conmon[24927]: mon.0) 2096 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T22:07:05.832 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:05 smithi125 conmon[24927]: cluster 2021-03-24T22:07:05.159143+0000 2021-03-24T22:07:05.832 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:05 smithi125 conmon[24927]: osd.5 (osd.5) 27 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:07:05.832 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:05 smithi125 conmon[24927]: 2021-03-24T22:07:05.832 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:05 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:07:05.832 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:05 smithi125 conmon[24927]: 03-24T22:07:05.162173+0000 osd.5 (osd.5 2021-03-24T22:07:05.833 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:05 smithi125 conmon[24927]: ) 28 : cluster [DBG] purged_snaps scrub found 34 strays 2021-03-24T22:07:05.833 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:05 smithi125 conmon[24927]: 2021-03-24T22:07:05.833 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:05 smithi125 conmon[19936]: cluster 2021-03-24 2021-03-24T22:07:05.834 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:05 smithi125 conmon[19936]: T22:07:04.600668+0000 mon.a (mon.0) 2095 : cluster [DBG] osdmap e854: 8 total, 8 up, 8 in 2021-03-24T22:07:05.834 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:05 smithi125 conmon[19936]: audit 2021-03-24T22:07:05.834 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:05 smithi125 conmon[19936]: 2021-03-24T 2021-03-24T22:07:05.834 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:05 smithi125 conmon[19936]: 22:07:04. 2021-03-24T22:07:05.834 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:05 smithi125 conmon[19936]: 853739+0000 mon.a (mon 2021-03-24T22:07:05.835 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:05 smithi125 conmon[19936]: .0) 2096 : audit 2021-03-24T22:07:05.835 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:05 smithi125 conmon[19936]: [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T22:07:05.835 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:05 smithi125 conmon[19936]: cluster 2021-03-24T22:07:05.835 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:05 smithi125 conmon[19936]: 2021-03 2021-03-24T22:07:05.835 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:05 smithi125 conmon[19936]: -24T22:07:05 2021-03-24T22:07:05.836 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:05 smithi125 conmon[19936]: .159143+0000 osd.5 ( 2021-03-24T22:07:05.836 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:05 smithi125 conmon[19936]: osd.5) 27 : 2021-03-24T22:07:05.836 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:05 smithi125 conmon[19936]: cluster [DBG] purged_snaps scrub starts 2021-03-24T22:07:05.836 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:05 smithi125 conmon[19936]: cluster 2021-03-24T22 2021-03-24T22:07:05.837 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:05 smithi125 conmon[19936]: :07:05.162173+0000 osd.5 2021-03-24T22:07:05.837 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:05 smithi125 conmon[19936]: (osd.5) 28 : cluster 2021-03-24T22:07:05.837 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:05 smithi125 conmon[19936]: [DBG] purged_snaps scrub found 34 strays 2021-03-24T22:07:05.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:05 smithi142 conmon[20125]: cluster 2021-03-24T22:07:04 2021-03-24T22:07:05.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:05 smithi142 conmon[20125]: .600668+0000 mon.a (mon.0) 2095 : cluster [DBG] osdmap e854: 8 total, 8 up, 8 in 2021-03-24T22:07:05.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:05 smithi142 conmon[20125]: audit 2021-03-24T22:07:04.853739+0000 2021-03-24T22:07:05.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:05 smithi142 conmon[20125]: mon.a (mon.0) 2096 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-24T22:07:05.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:05 smithi142 conmon[20125]: cluster 2021-03-24T22:07: 2021-03-24T22:07:05.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:05 smithi142 conmon[20125]: 05.159143+0000 osd.5 (osd.5) 27 : cluster 2021-03-24T22:07:05.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:05 smithi142 conmon[20125]: [DBG] purged_snaps scrub starts 2021-03-24T22:07:05.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:05 smithi142 conmon[20125]: cluster 2021-03-24T 2021-03-24T22:07:05.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:05 smithi142 conmon[20125]: 22:07:05.162173+0000 osd.5 (osd.5 2021-03-24T22:07:05.869 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:05 smithi142 conmon[20125]: ) 28 : cluster [DBG] purged_snaps scrub found 34 strays 2021-03-24T22:07:05.871 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:07:05.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:05.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:07:05.978 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:07:06.033 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:07:06.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:06.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:07:06.139 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:07:06.197 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:07:06.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:06.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:07:06.302 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:07:06.351 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:07:06.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:06.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:07:06.457 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:07:06.512 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:07:06.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:06.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:07:06.619 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:07:06.624 INFO:tasks.rados.rados.0.smithi125.stdout:3328: done (1 left) 2021-03-24T22:07:06.625 INFO:tasks.rados.rados.0.smithi125.stdout:3329: read oid 18 snap -1 2021-03-24T22:07:06.625 INFO:tasks.rados.rados.0.smithi125.stdout:3329: expect deleted 2021-03-24T22:07:06.625 INFO:tasks.rados.rados.0.smithi125.stdout:3330: read oid 43 snap -1 2021-03-24T22:07:06.625 INFO:tasks.rados.rados.0.smithi125.stdout:3330: expect deleted 2021-03-24T22:07:06.626 INFO:tasks.rados.rados.0.smithi125.stdout:3331: copy_from oid 36 from oid 8 current snap is 385 2021-03-24T22:07:06.626 INFO:tasks.rados.rados.0.smithi125.stdout:3332: write (excl) oid 24 current snap is 385 2021-03-24T22:07:06.626 INFO:tasks.rados.rados.0.smithi125.stdout:3332: seq_num 1136 ranges {692752=419461,1826015=615401,2979461=113291} 2021-03-24T22:07:06.630 INFO:tasks.rados.rados.0.smithi125.stdout:3332: writing smithi12556470-24 from 692752 to 1112213 tid 1 2021-03-24T22:07:06.638 INFO:tasks.rados.rados.0.smithi125.stdout:3332: writing smithi12556470-24 from 1826015 to 2441416 tid 2 2021-03-24T22:07:06.643 INFO:tasks.rados.rados.0.smithi125.stdout:3332: writing smithi12556470-24 from 2979461 to 3092752 tid 3 2021-03-24T22:07:06.643 INFO:tasks.rados.rados.0.smithi125.stdout:3327: finishing write tid 1 to smithi12556470-23 2021-03-24T22:07:06.644 INFO:tasks.rados.rados.0.smithi125.stdout:3327: finishing write tid 2 to smithi12556470-23 2021-03-24T22:07:06.644 INFO:tasks.rados.rados.0.smithi125.stdout:3327: finishing write tid 3 to smithi12556470-23 2021-03-24T22:07:06.644 INFO:tasks.rados.rados.0.smithi125.stdout:3327: finishing write tid 4 to smithi12556470-23 2021-03-24T22:07:06.644 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 23 v 559 (ObjNum 1135 snap 385 seq_num 1135) dirty exists 2021-03-24T22:07:06.644 INFO:tasks.rados.rados.0.smithi125.stdout:3327: left oid 23 (ObjNum 1135 snap 385 seq_num 1135) 2021-03-24T22:07:06.645 INFO:tasks.rados.rados.0.smithi125.stdout:3327: done (4 left) 2021-03-24T22:07:06.645 INFO:tasks.rados.rados.0.smithi125.stdout:3329: done (3 left) 2021-03-24T22:07:06.646 INFO:tasks.rados.rados.0.smithi125.stdout:3330: done (2 left) 2021-03-24T22:07:06.646 INFO:tasks.rados.rados.0.smithi125.stdout:3333: snap_create 2021-03-24T22:07:06.646 INFO:tasks.rados.rados.0.smithi125.stdout:3332: finishing write tid 1 to smithi12556470-24 2021-03-24T22:07:06.647 INFO:teuthology.orchestra.run.smithi125.stderr:nodeep-scrub is set 2021-03-24T22:07:06.647 INFO:tasks.rados.rados.0.smithi125.stdout:3331: finishing copy_from to smithi12556470-36 2021-03-24T22:07:06.648 INFO:tasks.rados.rados.0.smithi125.stdout:3331: got expected ENOENT (src dne) 2021-03-24T22:07:06.649 INFO:tasks.rados.rados.0.smithi125.stdout:3331: finishing copy_from racing read to smithi12556470-36 2021-03-24T22:07:06.651 INFO:tasks.rados.rados.0.smithi125.stdout:3332: finishing write tid 2 to smithi12556470-24 2021-03-24T22:07:06.652 INFO:tasks.rados.rados.0.smithi125.stdout:3332: finishing write tid 3 to smithi12556470-24 2021-03-24T22:07:06.655 INFO:tasks.rados.rados.0.smithi125.stdout:3332: finishing write tid 4 to smithi12556470-24 2021-03-24T22:07:06.655 INFO:tasks.rados.rados.0.smithi125.stdout:3332: finishing write tid 5 to smithi12556470-24 2021-03-24T22:07:06.655 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 24 v 1477 (ObjNum 1136 snap 385 seq_num 1136) dirty exists 2021-03-24T22:07:06.655 INFO:tasks.rados.rados.0.smithi125.stdout:3332: left oid 24 (ObjNum 1136 snap 385 seq_num 1136) 2021-03-24T22:07:06.672 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:07:06.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:06.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:07:06.739 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:06 smithi125 conmon[19936]: cluster 2021-03-24T22:07:05.019466+0000 mgr.y (mgr.14144) 705 : cluster [DBG] 2021-03-24T22:07:06.739 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:06 smithi125 conmon[19936]: pgmap v1444: 168 pgs: 168 active+clean; 581 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.0 MiB/s rd, 3.6 MiB/s wr, 22 op/s 2021-03-24T22:07:06.740 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:06 smithi125 conmon[19936]: cluster 2021-03-24T22:07:05.609737+0000 mon.a (mon.0) 2097 : cluster [DBG] osdmap e855: 8 total, 8 up, 8 in 2021-03-24T22:07:06.740 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:06 smithi125 conmon[19936]: audit 2021-03-24T22:07:05.694117+0000 2021-03-24T22:07:06.740 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:06 smithi125 conmon[19936]: mon.a (mon.0) 2098 : audit [INF] from='client.? v1:172.21.15.125:0/928500354' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:07:06.740 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:06 smithi125 conmon[24927]: cluster 2021-03-24T22:07:05.019466+0000 2021-03-24T22:07:06.741 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:06 smithi125 conmon[24927]: mgr.y (mgr.14144) 705 : cluster [DBG] pgmap v1444: 168 pgs: 168 active+clean; 581 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.0 MiB/s rd, 3.6 MiB/s wr, 22 op/s 2021-03-24T22:07:06.741 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:06 smithi125 conmon[24927]: cluster 2021-03-24T22:07:05.609737+0000 mon.a (mon. 2021-03-24T22:07:06.741 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:06 smithi125 conmon[24927]: 0) 2097 : cluster [DBG] osdmap e855: 8 total, 8 up, 8 in 2021-03-24T22:07:06.741 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:06 smithi125 conmon[24927]: audit 2021-03-24T22:07:05 2021-03-24T22:07:06.742 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:06 smithi125 conmon[24927]: .694117+0000 mon.a (mon.0) 2098 : audit 2021-03-24T22:07:06.742 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:06 smithi125 conmon[24927]: [INF] from='client.? v1:172.21.15.125:0/928500354' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:07:06.777 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:07:06.791 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2021-03-24T22:07:06.791 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T22:07:06.824 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:07:06.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:06.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:07:06.866 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:06 smithi142 conmon[20125]: cluster 2021-03-24T22:07:05.019466+0000 mgr.y (mgr.14144) 705 : cluster [DBG] pgmap v1444: 168 pgs: 168 active+clean; 581 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.0 MiB/s rd, 3.6 MiB/s wr, 22 op/s 2021-03-24T22:07:06.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:06 smithi142 conmon[20125]: cluster 2021-03-24T22:07:05.609737+0000 mon.a (mon.0) 2097 : cluster [DBG] osdmap e855: 8 total, 8 up, 8 in 2021-03-24T22:07:06.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:06 smithi142 conmon[20125]: audit 2021-03-24T22:07:05.694117+0000 mon.a (mon.0) 2098 : audit [INF] from='client.? v1:172.21.15.125:0/928500354' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-24T22:07:06.929 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:07:06.992 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:07:06.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:06.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:07:07.072 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-24T22:07:07.097 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:07:07.156 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:07:07.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:07.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:07:07.261 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:07:07.320 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:07:07.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:07.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:07:07.352 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 1] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2021-03-24T22:07:07.353 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-24T22:07:07.353 INFO:tasks.thrashosds.thrasher:inject_pause on 2 2021-03-24T22:07:07.354 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2021-03-24T22:07:07.354 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2021-03-24T22:07:07.354 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.2 config set filestore_inject_stall 3 2021-03-24T22:07:07.427 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:07:07.484 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:07:07.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:07.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:07:07.591 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:07:07.639 INFO:tasks.rados.rados.0.smithi125.stdout:3331: done (2 left) 2021-03-24T22:07:07.640 INFO:tasks.rados.rados.0.smithi125.stdout:3332: done (1 left) 2021-03-24T22:07:07.640 INFO:tasks.rados.rados.0.smithi125.stdout:3333: done (0 left) 2021-03-24T22:07:07.640 INFO:tasks.rados.rados.0.smithi125.stdout:3334: copy_from oid 44 from oid 25 current snap is 386 2021-03-24T22:07:07.641 INFO:tasks.rados.rados.0.smithi125.stdout:3335: delete oid 33 current snap is 386 2021-03-24T22:07:07.646 INFO:tasks.rados.rados.0.smithi125.stdout:3334: finishing copy_from to smithi12556470-44 2021-03-24T22:07:07.646 INFO:tasks.rados.rados.0.smithi125.stdout:3334: got expected ENOENT (src dne) 2021-03-24T22:07:07.646 INFO:tasks.rados.rados.0.smithi125.stdout:3334: finishing copy_from racing read to smithi12556470-44 2021-03-24T22:07:07.647 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:07:07.649 INFO:tasks.rados.rados.0.smithi125.stdout:3334: done (1 left) 2021-03-24T22:07:07.649 INFO:tasks.rados.rados.0.smithi125.stdout:3335: done (0 left) 2021-03-24T22:07:07.650 INFO:tasks.rados.rados.0.smithi125.stdout:3336: read oid 28 snap 383 2021-03-24T22:07:07.650 INFO:tasks.rados.rados.0.smithi125.stdout:3336: expect deleted 2021-03-24T22:07:07.650 INFO:tasks.rados.rados.0.smithi125.stdout:3337: rollback oid 45 current snap is 386 2021-03-24T22:07:07.650 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 45 to 356 2021-03-24T22:07:07.651 INFO:tasks.rados.rados.0.smithi125.stdout:3338: delete oid 7 current snap is 386 2021-03-24T22:07:07.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:07.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:07:07.655 INFO:tasks.rados.rados.0.smithi125.stdout:3337: finishing rollback tid 1 to smithi12556470-45 2021-03-24T22:07:07.656 INFO:tasks.rados.rados.0.smithi125.stdout:3337: finishing rollback tid 2 to smithi12556470-45 2021-03-24T22:07:07.656 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 45 v 755 (ObjNum 1014 snap 336 seq_num 1014) dirty exists 2021-03-24T22:07:07.657 INFO:tasks.rados.rados.0.smithi125.stdout:3336: done (2 left) 2021-03-24T22:07:07.657 INFO:tasks.rados.rados.0.smithi125.stdout:3337: done (1 left) 2021-03-24T22:07:07.657 INFO:tasks.rados.rados.0.smithi125.stdout:3338: done (0 left) 2021-03-24T22:07:07.657 INFO:tasks.rados.rados.0.smithi125.stdout:3339: snap_create 2021-03-24T22:07:07.754 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:07:07.811 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:07:07.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:07.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:07:07.892 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:07 smithi142 conmon[20125]: cluster 2021-03-24T22:07:06.605709+0000 mon.a (mon.0) 2099 2021-03-24T22:07:07.893 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:07 smithi142 conmon[20125]: : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:07:07.893 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:07 smithi142 conmon[20125]: audit 2021-03-24T22:07:06 2021-03-24T22:07:07.893 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:07 smithi142 conmon[20125]: .615286+0000 mon.a (mon.0) 2100 : audit 2021-03-24T22:07:07.894 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:07 smithi142 conmon[20125]: [INF] from='client.? v1:172.21.15.125:0/928500354' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T22:07:07.894 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:07 smithi142 conmon[20125]: cluster 2021 2021-03-24T22:07:07.895 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:07 smithi142 conmon[20125]: -03-24T22:07:06. 2021-03-24T22:07:07.895 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:07 smithi142 conmon[20125]: 615484+0000 mon.a (mon.0) 2021-03-24T22:07:07.896 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:07 smithi142 conmon[20125]: 2101 : cluster [DBG] osdmap e856: 8 total, 8 up, 8 in 2021-03-24T22:07:07.897 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.6 dump_ops_in_flight 2021-03-24T22:07:07.900 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:07 smithi125 conmon[19936]: cluster 2021-03-24T22:07:06.605709+0000 mon.a (mon 2021-03-24T22:07:07.901 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:07 smithi125 conmon[19936]: .0) 2099 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:07:07.901 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:07 smithi125 conmon[19936]: audit 2021-03-24T22:07:06.615286+0000 mon.a (mon.0) 2100 : audit [INF] from='client.? v1:172.21.15.125:0/928500354' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T22:07:07.902 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:07 smithi125 conmon[19936]: cluster 2021-03-24T22:07:07.902 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:07 smithi125 conmon[19936]: 2021-03-24T22:07:06.615484+0000 mon.a (mon.0) 2101 : cluster [DBG] osdmap e856: 8 total, 8 up, 8 in 2021-03-24T22:07:07.903 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:07 smithi125 conmon[24927]: cluster 2021-03-24T22:07:06.605709+0000 mon.a (mon. 2021-03-24T22:07:07.903 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:07 smithi125 conmon[24927]: 0) 2099 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:07:07.904 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:07 smithi125 conmon[24927]: audit 2021-03-24T22:07:06.615286+0000 mon.a (mon 2021-03-24T22:07:07.904 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:07 smithi125 conmon[24927]: .0) 2100 : audit [INF] from='client.? v1:172.21.15.125:0/928500354' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-24T22:07:07.904 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:07 smithi125 conmon[24927]: cluster 2021-03-24T22:07:07.905 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:07 smithi125 conmon[24927]: 2021-03-24T22:07:06.615484+0000 mon.a (mon.0) 2101 2021-03-24T22:07:07.905 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:07 smithi125 conmon[24927]: : cluster [DBG] osdmap e856: 8 total, 8 up, 8 in 2021-03-24T22:07:07.911 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:07:07.916 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:07:07.966 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:07:07.968 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:07:07.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:07.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:07:08.077 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:07:08.123 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:07:08.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:08.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:07:08.201 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T22:07:08.228 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:07:08.273 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:07:08.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:08.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:07:08.379 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:07:08.432 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:07:08.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:08.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:07:08.539 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:07:08.597 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:07:08.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:08.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:07:08.649 INFO:tasks.rados.rados.0.smithi125.stdout:3339: done (0 left) 2021-03-24T22:07:08.649 INFO:tasks.rados.rados.0.smithi125.stdout:3340: snap_create 2021-03-24T22:07:08.703 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:07:08.716 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:08 smithi125 conmon[19936]: cluster 2021-03-24T22:07: 2021-03-24T22:07:08.716 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:08 smithi125 conmon[19936]: 07.020165+0000 mgr.y (mgr.14144) 706 : cluster [DBG] pgmap v1447: 168 pgs: 168 active+clean; 584 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.4 MiB/s wr, 25 op/s 2021-03-24T22:07:08.716 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:08 smithi125 conmon[19936]: cluster 2021-03-24T22:07:07 2021-03-24T22:07:08.716 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:08 smithi125 conmon[19936]: .635534+0000 mon.a (mon.0) 2102 : cluster [DBG] osdmap e857: 8 total, 8 up, 8 in 2021-03-24T22:07:08.717 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:08 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:07:08.717 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:08 smithi125 conmon[24927]: 24T22:07:07.020165+0000 mgr.y (mgr.14144) 706 : cluster 2021-03-24T22:07:08.717 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:08 smithi125 conmon[24927]: [DBG] pgmap v1447: 168 pgs: 168 active+clean; 584 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.4 MiB/s wr, 25 op/s 2021-03-24T22:07:08.718 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:08 smithi125 conmon[24927]: cluster 2021-03-24T22:07:08.718 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:08 smithi125 conmon[24927]: 2021-03-24T22:07:07 2021-03-24T22:07:08.718 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:08 smithi125 conmon[24927]: .635534+0000 mon.a (mon. 2021-03-24T22:07:08.718 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:08 smithi125 conmon[24927]: 0) 2102 : cluster [DBG] osdmap e857: 8 total, 8 up, 8 in 2021-03-24T22:07:08.718 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:08 smithi125 conmon[24927]: 2021-03-24T22:07:08.760 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:07:08.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:08.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:07:08.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:08 smithi142 conmon[20125]: cluster 2021-03-24T22:07:07.020165 2021-03-24T22:07:08.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:08 smithi142 conmon[20125]: +0000 mgr.y (mgr.14144) 706 : cluster [DBG] pgmap v1447: 168 pgs: 168 active+clean; 584 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.4 MiB/s wr, 25 op/s 2021-03-24T22:07:08.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:08 smithi142 conmon[20125]: cluster 2021-03-24T22:07:07.635534+0000 mon.a 2021-03-24T22:07:08.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:08 smithi142 conmon[20125]: (mon.0) 2102 : cluster [DBG] osdmap e857: 8 total, 8 up, 8 in 2021-03-24T22:07:08.868 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:07:08.931 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:07:08.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:08.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:07:09.036 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:07:09.099 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:07:09.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:09.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:07:09.205 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:07:09.263 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:07:09.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:09.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:07:09.369 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:07:09.425 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:07:09.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:09.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:07:09.532 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:07:09.594 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:07:09.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:09.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:07:09.671 INFO:tasks.rados.rados.0.smithi125.stdout:3340: done (0 left) 2021-03-24T22:07:09.671 INFO:tasks.rados.rados.0.smithi125.stdout:3341: write oid 42 current snap is 388 2021-03-24T22:07:09.671 INFO:tasks.rados.rados.0.smithi125.stdout:3341: seq_num 1139 ranges {743173=645208,1885108=762728} 2021-03-24T22:07:09.674 INFO:tasks.rados.rados.0.smithi125.stdout:3341: writing smithi12556470-42 from 743173 to 1388381 tid 1 2021-03-24T22:07:09.683 INFO:tasks.rados.rados.0.smithi125.stdout:3341: writing smithi12556470-42 from 1885108 to 2647836 tid 2 2021-03-24T22:07:09.683 INFO:tasks.rados.rados.0.smithi125.stdout:3342: write (excl) oid 18 current snap is 388 2021-03-24T22:07:09.683 INFO:tasks.rados.rados.0.smithi125.stdout:3342: seq_num 1140 ranges {759980=720457,1943590=506386,2856447=303533} 2021-03-24T22:07:09.691 INFO:tasks.rados.rados.0.smithi125.stdout:3342: writing smithi12556470-18 from 759980 to 1480437 tid 1 2021-03-24T22:07:09.697 INFO:tasks.rados.rados.0.smithi125.stdout:3342: writing smithi12556470-18 from 1943590 to 2449976 tid 2 2021-03-24T22:07:09.700 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:07:09.702 INFO:tasks.rados.rados.0.smithi125.stdout:3342: writing smithi12556470-18 from 2856447 to 3159980 tid 3 2021-03-24T22:07:09.703 INFO:tasks.rados.rados.0.smithi125.stdout:3343: write (excl) oid 48 current snap is 388 2021-03-24T22:07:09.703 INFO:tasks.rados.rados.0.smithi125.stdout:3341: finishing write tid 1 to smithi12556470-42 2021-03-24T22:07:09.703 INFO:tasks.rados.rados.0.smithi125.stdout:3343: seq_num 1141 ranges {776787=512059,1718425=766397,3000223=176564} 2021-03-24T22:07:09.711 INFO:tasks.rados.rados.0.smithi125.stdout:3343: writing smithi12556470-48 from 776787 to 1288846 tid 1 2021-03-24T22:07:09.720 INFO:tasks.rados.rados.0.smithi125.stdout:3343: writing smithi12556470-48 from 1718425 to 2484822 tid 2 2021-03-24T22:07:09.725 INFO:tasks.rados.rados.0.smithi125.stdout:3343: writing smithi12556470-48 from 3000223 to 3176787 tid 3 2021-03-24T22:07:09.725 INFO:tasks.rados.rados.0.smithi125.stdout:3344: write (excl) oid 14 current snap is 388 2021-03-24T22:07:09.725 INFO:tasks.rados.rados.0.smithi125.stdout:3341: finishing write tid 2 to smithi12556470-42 2021-03-24T22:07:09.726 INFO:tasks.rados.rados.0.smithi125.stdout:3344: seq_num 1142 ranges {793594=587308,2060554=510055} 2021-03-24T22:07:09.733 INFO:tasks.rados.rados.0.smithi125.stdout:3344: writing smithi12556470-14 from 793594 to 1380902 tid 1 2021-03-24T22:07:09.741 INFO:tasks.rados.rados.0.smithi125.stdout:3344: writing smithi12556470-14 from 2060554 to 2570609 tid 2 2021-03-24T22:07:09.741 INFO:tasks.rados.rados.0.smithi125.stdout:3345: read oid 30 snap -1 2021-03-24T22:07:09.741 INFO:tasks.rados.rados.0.smithi125.stdout:3345: expect (ObjNum 1092 snap 370 seq_num 1092) 2021-03-24T22:07:09.742 INFO:tasks.rados.rados.0.smithi125.stdout:3346: snap_create 2021-03-24T22:07:09.742 INFO:tasks.rados.rados.0.smithi125.stdout:3341: finishing write tid 3 to smithi12556470-42 2021-03-24T22:07:09.742 INFO:tasks.rados.rados.0.smithi125.stdout:3341: finishing write tid 4 to smithi12556470-42 2021-03-24T22:07:09.742 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 42 v 519 (ObjNum 1139 snap 388 seq_num 1139) dirty exists 2021-03-24T22:07:09.743 INFO:tasks.rados.rados.0.smithi125.stdout:3341: left oid 42 (ObjNum 1139 snap 388 seq_num 1139) 2021-03-24T22:07:09.743 INFO:tasks.rados.rados.0.smithi125.stdout:3342: finishing write tid 1 to smithi12556470-18 2021-03-24T22:07:09.743 INFO:tasks.rados.rados.0.smithi125.stdout:3342: finishing write tid 2 to smithi12556470-18 2021-03-24T22:07:09.743 INFO:tasks.rados.rados.0.smithi125.stdout:3342: finishing write tid 3 to smithi12556470-18 2021-03-24T22:07:09.743 INFO:tasks.rados.rados.0.smithi125.stdout:3342: finishing write tid 4 to smithi12556470-18 2021-03-24T22:07:09.744 INFO:tasks.rados.rados.0.smithi125.stdout:3342: finishing write tid 5 to smithi12556470-18 2021-03-24T22:07:09.744 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 18 v 1444 (ObjNum 1140 snap 388 seq_num 1140) dirty exists 2021-03-24T22:07:09.744 INFO:tasks.rados.rados.0.smithi125.stdout:3342: left oid 18 (ObjNum 1140 snap 388 seq_num 1140) 2021-03-24T22:07:09.744 INFO:tasks.rados.rados.0.smithi125.stdout:3343: finishing write tid 1 to smithi12556470-48 2021-03-24T22:07:09.745 INFO:tasks.rados.rados.0.smithi125.stdout:3343: finishing write tid 2 to smithi12556470-48 2021-03-24T22:07:09.745 INFO:tasks.rados.rados.0.smithi125.stdout:3343: finishing write tid 3 to smithi12556470-48 2021-03-24T22:07:09.745 INFO:tasks.rados.rados.0.smithi125.stdout:3343: finishing write tid 4 to smithi12556470-48 2021-03-24T22:07:09.745 INFO:tasks.rados.rados.0.smithi125.stdout:3343: finishing write tid 5 to smithi12556470-48 2021-03-24T22:07:09.746 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 48 v 1457 (ObjNum 1141 snap 388 seq_num 1141) dirty exists 2021-03-24T22:07:09.746 INFO:tasks.rados.rados.0.smithi125.stdout:3343: left oid 48 (ObjNum 1141 snap 388 seq_num 1141) 2021-03-24T22:07:09.746 INFO:tasks.rados.rados.0.smithi125.stdout:3344: finishing write tid 1 to smithi12556470-14 2021-03-24T22:07:09.747 INFO:tasks.rados.rados.0.smithi125.stdout:3345: expect (ObjNum 1092 snap 370 seq_num 1092) 2021-03-24T22:07:09.748 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:07:09.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:09.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:07:09.853 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:07:09.858 INFO:tasks.rados.rados.0.smithi125.stdout:3344: finishing write tid 2 to smithi12556470-14 2021-03-24T22:07:09.859 INFO:tasks.rados.rados.0.smithi125.stdout:3344: finishing write tid 3 to smithi12556470-14 2021-03-24T22:07:09.859 INFO:tasks.rados.rados.0.smithi125.stdout:3344: finishing write tid 4 to smithi12556470-14 2021-03-24T22:07:09.859 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 14 v 2147 (ObjNum 1142 snap 388 seq_num 1142) dirty exists 2021-03-24T22:07:09.859 INFO:tasks.rados.rados.0.smithi125.stdout:3344: left oid 14 (ObjNum 1142 snap 388 seq_num 1142) 2021-03-24T22:07:09.865 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:09 smithi142 conmon[20125]: cluster 2021-03-24T22:07:08.645856+0000 mon.a (mon.0) 2103 : cluster [DBG] osdmap e858: 8 total, 8 up, 8 in 2021-03-24T22:07:09.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:09 smithi125 conmon[19936]: cluster 2021-03-24T22:07:09.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:09 smithi125 conmon[19936]: 2021-03-24 2021-03-24T22:07:09.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:09 smithi125 conmon[19936]: T22:07:08.645856+0000 mon.a (mon.0) 2103 : cluster [DBG] osdmap e858: 8 total, 8 up, 8 in 2021-03-24T22:07:09.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:09 smithi125 conmon[24927]: cluster 2021-03-24T22:07:08.645856+0000 mon.a (mon.0) 2103 : cluster [DBG] osdmap e858: 8 total, 8 up, 8 in 2021-03-24T22:07:09.913 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:07:09.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:09.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:07:09.934 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd unset noscrub 2021-03-24T22:07:10.020 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:07:10.079 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:07:10.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:10.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:07:10.183 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:07:10.241 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:07:10.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:10.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:07:10.346 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:07:10.362 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2021-03-24T22:07:10.362 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-24T22:07:10.391 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:07:10.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:10.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:07:10.461 INFO:tasks.rados.rados.0.smithi125.stdout:3341: done (5 left) 2021-03-24T22:07:10.461 INFO:tasks.rados.rados.0.smithi125.stdout:3342: done (4 left) 2021-03-24T22:07:10.461 INFO:tasks.rados.rados.0.smithi125.stdout:3343: done (3 left) 2021-03-24T22:07:10.461 INFO:tasks.rados.rados.0.smithi125.stdout:3344: done (2 left) 2021-03-24T22:07:10.462 INFO:tasks.rados.rados.0.smithi125.stdout:3345: done (1 left) 2021-03-24T22:07:10.462 INFO:tasks.rados.rados.0.smithi125.stdout:3346: done (0 left) 2021-03-24T22:07:10.462 INFO:tasks.rados.rados.0.smithi125.stdout:3347: rollback oid 31 current snap is 389 2021-03-24T22:07:10.462 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 31 to 380 2021-03-24T22:07:10.463 INFO:tasks.rados.rados.0.smithi125.stdout:3348: copy_from oid 47 from oid 2 current snap is 389 2021-03-24T22:07:10.463 INFO:tasks.rados.rados.0.smithi125.stdout:3349: snap_create 2021-03-24T22:07:10.479 INFO:tasks.rados.rados.0.smithi125.stdout:3347: finishing rollback tid 0 to smithi12556470-31 2021-03-24T22:07:10.480 INFO:tasks.rados.rados.0.smithi125.stdout:3347: finishing rollback tid 1 to smithi12556470-31 2021-03-24T22:07:10.480 INFO:tasks.rados.rados.0.smithi125.stdout:3347: finishing rollback tid 2 to smithi12556470-31 2021-03-24T22:07:10.480 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 31 v 2151 (ObjNum 1101 snap 375 seq_num 1101) dirty exists 2021-03-24T22:07:10.486 INFO:tasks.rados.rados.0.smithi125.stdout:3348: finishing copy_from to smithi12556470-47 2021-03-24T22:07:10.486 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 47 v 1017 (ObjNum 984 snap 326 seq_num 984) dirty exists 2021-03-24T22:07:10.487 INFO:tasks.rados.rados.0.smithi125.stdout:3348: finishing copy_from racing read to smithi12556470-47 2021-03-24T22:07:10.497 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:07:10.541 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:07:10.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:10.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:07:10.648 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:07:10.699 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:07:10.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:10.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:07:10.805 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:07:10.867 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:07:10.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:10.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:07:10.912 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:10 smithi125 conmon[24927]: cluster 2021-03-24T22:07:09 2021-03-24T22:07:10.912 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:10 smithi125 conmon[24927]: .021094+0000 mgr.y (mgr.14144) 707 : cluster [DBG] pgmap v1450: 168 pgs: 168 active+clean; 583 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 4.8 MiB/s rd, 1.0 MiB/s wr, 10 op/s 2021-03-24T22:07:10.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:10 smithi125 conmon[24927]: cluster 2021-03- 2021-03-24T22:07:10.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:10 smithi125 conmon[24927]: 24T22:07:09.662080+0000 mon.a (mon.0) 2104 : cluster [DBG] osdmap e859: 8 total, 8 up, 8 in 2021-03-24T22:07:10.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:10 smithi125 conmon[24927]: cluster 2021-03-24T22:07:10 2021-03-24T22:07:10.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:10 smithi125 conmon[24927]: .454914+0000 mon.a (mon.0) 2105 : cluster [DBG] 2021-03-24T22:07:10.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:10 smithi125 conmon[24927]: osdmap e860: 8 total, 8 up, 8 in 2021-03-24T22:07:10.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:10 smithi125 conmon[19936]: cluster 2021-03-24T22:07 2021-03-24T22:07:10.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:10 smithi125 conmon[19936]: :09.021094+0000 mgr.y (mgr.14144) 707 : cluster [DBG] pgmap v1450: 168 pgs: 168 active+clean; 583 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 4.8 MiB/s rd, 1.0 MiB/s wr, 10 op/s 2021-03-24T22:07:10.914 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:10 smithi125 conmon[19936]: cluster 2021-03-24T22:07:10.915 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:10 smithi125 conmon[19936]: 2021-03-24T22:07:09.662080+0000 mon.a (mon.0) 2104 : cluster [DBG] 2021-03-24T22:07:10.915 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:10 smithi125 conmon[19936]: osdmap e859: 8 total, 8 up, 8 in 2021-03-24T22:07:10.915 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:10 smithi125 conmon[19936]: cluster 2021-03-24T22 2021-03-24T22:07:10.915 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:10 smithi125 conmon[19936]: :07:10.454914+0000 mon.a (mon.0) 2021-03-24T22:07:10.915 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:10 smithi125 conmon[19936]: 2105 : cluster [DBG] osdmap e860: 8 total, 8 up, 8 in 2021-03-24T22:07:10.973 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:07:11.027 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:07:11.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:11.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:07:11.085 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:07:11.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:10 smithi142 conmon[20125]: cluster 2021-03-24T22:07:09.021094+0000 mgr.y (mgr.14144) 707 2021-03-24T22:07:11.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:10 smithi142 conmon[20125]: : cluster [DBG] pgmap v1450: 168 pgs: 168 active+clean; 583 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 4.8 MiB/s rd, 1.0 MiB/s wr, 10 op/s 2021-03-24T22:07:11.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:10 smithi142 conmon[20125]: cluster 2021-03-24T22:07:09.662080+0000 mon.a (mon.0) 2104 : cluster [DBG] osdmap e859: 8 total, 8 up, 8 in 2021-03-24T22:07:11.117 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:10 smithi142 conmon[20125]: cluster 2021-03-24T22:07:10.454914+0000 mon.a (mon.0) 2105 : 2021-03-24T22:07:11.118 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:10 smithi142 conmon[20125]: cluster [DBG] osdmap e860: 8 total, 8 up, 8 in 2021-03-24T22:07:11.135 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:07:11.189 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:07:11.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:11.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:07:11.266 INFO:teuthology.orchestra.run.smithi125.stdout:osd.0: {} 2021-03-24T22:07:11.289 INFO:teuthology.orchestra.run.smithi125.stdout:osd.1: {} 2021-03-24T22:07:11.294 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:07:11.314 INFO:teuthology.orchestra.run.smithi125.stdout:osd.2: {} 2021-03-24T22:07:11.336 INFO:teuthology.orchestra.run.smithi125.stdout:osd.3: {} 2021-03-24T22:07:11.343 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:07:11.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:11.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:07:11.360 INFO:teuthology.orchestra.run.smithi125.stdout:osd.4: {} 2021-03-24T22:07:11.382 INFO:teuthology.orchestra.run.smithi125.stdout:osd.5: {} 2021-03-24T22:07:11.411 INFO:teuthology.orchestra.run.smithi125.stdout:osd.6: {} 2021-03-24T22:07:11.436 INFO:teuthology.orchestra.run.smithi125.stdout:osd.7: {} 2021-03-24T22:07:11.438 INFO:teuthology.orchestra.run.smithi125.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-24T22:07:11.438 INFO:teuthology.orchestra.run.smithi125.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-24T22:07:11.438 INFO:teuthology.orchestra.run.smithi125.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-24T22:07:11.439 INFO:teuthology.orchestra.run.smithi125.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-24T22:07:11.439 INFO:teuthology.orchestra.run.smithi125.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-24T22:07:11.439 INFO:teuthology.orchestra.run.smithi125.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-24T22:07:11.439 INFO:teuthology.orchestra.run.smithi125.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-24T22:07:11.440 INFO:teuthology.orchestra.run.smithi125.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-24T22:07:11.448 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:07:11.464 INFO:tasks.rados.rados.0.smithi125.stdout:3347: done (2 left) 2021-03-24T22:07:11.464 INFO:tasks.rados.rados.0.smithi125.stdout:3348: done (1 left) 2021-03-24T22:07:11.465 INFO:tasks.rados.rados.0.smithi125.stdout:3349: done (0 left) 2021-03-24T22:07:11.465 INFO:tasks.rados.rados.0.smithi125.stdout:3350: snap_create 2021-03-24T22:07:11.504 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:07:11.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:11.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:07:11.531 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.6 dump_blocked_ops 2021-03-24T22:07:11.613 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:07:11.658 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:07:11.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:11.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:07:11.686 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-24T22:07:11.686 INFO:teuthology.orchestra.run.smithi125.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2021-03-24T22:07:11.686 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-24T22:07:11.761 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:07:11.812 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:07:11.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:11.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:07:11.819 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T22:07:11.899 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:11 smithi125 conmon[19936]: cluster 2021-03-24T22:07:11. 2021-03-24T22:07:11.900 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:11 smithi125 conmon[19936]: 021817+0000 mgr.y (mgr.14144) 708 : cluster [DBG] pgmap v1453: 168 pgs: 168 active+clean; 582 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.3 MiB/s wr, 9 op/s 2021-03-24T22:07:11.900 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:11 smithi125 conmon[19936]: audit 2021-03-24T22:07:11.238121+0000 mon.a ( 2021-03-24T22:07:11.900 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:11 smithi125 conmon[19936]: mon.0) 2106 : audit [DBG] from='client.? v1:172.21.15.125:0/925240816' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:07:11.900 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:11 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:07:11.901 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:11 smithi125 conmon[19936]: 07:11.458511+0000 mon.a (mon.0) 2107 : cluster [DBG] osdmap e861: 8 total, 8 up, 8 in 2021-03-24T22:07:11.901 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:11 smithi125 conmon[24927]: cluster 2021-03-24T22:07: 2021-03-24T22:07:11.901 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:11 smithi125 conmon[24927]: 11.021817+0000 mgr.y (mgr.14144) 708 : cluster [DBG] pgmap v1453: 168 pgs: 168 active+clean; 582 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.3 MiB/s wr, 9 op/s 2021-03-24T22:07:11.901 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:11 smithi125 conmon[24927]: audit 2021-03-24T22:07: 2021-03-24T22:07:11.902 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:11 smithi125 conmon[24927]: 11.238121+0000 mon.a (mon.0) 2106 : audit 2021-03-24T22:07:11.902 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:11 smithi125 conmon[24927]: [DBG] from='client.? v1:172.21.15.125:0/925240816' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:07:11.902 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:11 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T22:07:11.902 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:11 smithi125 conmon[24927]: 22:07:11.458511+0000 mon.a (mon.0 2021-03-24T22:07:11.903 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:11 smithi125 conmon[24927]: ) 2107 : cluster [DBG] osdmap e861: 8 total, 8 up, 8 in 2021-03-24T22:07:11.917 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:07:11.973 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:07:11.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:11.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:07:12.082 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:07:12.087 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:11 smithi142 conmon[20125]: cluster 2021-03-24T22:07:11.021817+0000 mgr.y (mgr.14144) 708 : 2021-03-24T22:07:12.087 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:11 smithi142 conmon[20125]: cluster [DBG] pgmap v1453: 168 pgs: 168 active+clean; 582 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.3 MiB/s wr, 9 op/s 2021-03-24T22:07:12.087 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:11 smithi142 conmon[20125]: audit 2021-03-24T22:07:11.238121+0000 mon.a (mon. 2021-03-24T22:07:12.088 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:11 smithi142 conmon[20125]: 0) 2106 : audit [DBG] from='client.? v1:172.21.15.125:0/925240816' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-24T22:07:12.088 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:11 smithi142 conmon[20125]: cluster 2021-03-24T22:07:12.088 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:11 smithi142 conmon[20125]: 2021-03-24T22:07:11.458511+0000 2021-03-24T22:07:12.088 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:11 smithi142 conmon[20125]: mon.a (mon.0) 2107 : cluster [DBG] osdmap e861: 8 total, 8 up, 8 in 2021-03-24T22:07:12.089 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:11 smithi142 conmon[20125]: 2021-03-24T22:07:12.133 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:07:12.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:12.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:07:12.240 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:07:12.300 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:07:12.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:12.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:07:12.407 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:07:12.463 INFO:tasks.rados.rados.0.smithi125.stdout:3350: done (0 left) 2021-03-24T22:07:12.464 INFO:tasks.rados.rados.0.smithi125.stdout:3351: read oid 21 snap 376 2021-03-24T22:07:12.464 INFO:tasks.rados.rados.0.smithi125.stdout:3351: expect deleted 2021-03-24T22:07:12.464 INFO:tasks.rados.rados.0.smithi125.stdout:3352: snap_remove snap 365 2021-03-24T22:07:12.470 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:07:12.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:12.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:07:12.574 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:07:12.627 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:07:12.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:12.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:07:12.734 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:07:12.795 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:07:12.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:12.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:07:12.901 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:07:12.956 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:07:12.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:12.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:07:13.062 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:07:13.120 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:07:13.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:13.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:07:13.226 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:07:13.293 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:07:13.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:13.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:07:13.399 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:07:13.462 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:07:13.463 INFO:tasks.rados.rados.0.smithi125.stdout:3352: done (1 left) 2021-03-24T22:07:13.464 INFO:tasks.rados.rados.0.smithi125.stdout:3353: snap_remove snap 379 2021-03-24T22:07:13.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:13.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:07:13.528 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:13 smithi142 conmon[20125]: cluster 2021-03-24T22:07:12.459426+0000 mon.a (mon.0) 2108 : cluster [DBG] osdmap e862: 8 total, 8 up, 8 in 2021-03-24T22:07:13.571 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:07:13.584 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:13 smithi125 conmon[19936]: cluster 2021-03-24T22:07:13.585 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:13 smithi125 conmon[19936]: 2021-03-24T22:07:13.585 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:13 smithi125 conmon[19936]: 2021 2021-03-24T22:07:13.585 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:13 smithi125 conmon[19936]: -03-24T22:07:12.459426 2021-03-24T22:07:13.585 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:13 smithi125 conmon[19936]: +0000 mon.a (mon.0) 2108 : cluster [DBG] osdmap e862: 8 total, 8 up, 8 in 2021-03-24T22:07:13.586 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:13 smithi125 conmon[24927]: cluster 2021-03-24T22:07:13.586 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:13 smithi125 conmon[24927]: 2021-03-24T22:07:13.586 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:13 smithi125 conmon[24927]: 2021-03-24T22:07:12.459426 2021-03-24T22:07:13.587 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:13 smithi125 conmon[24927]: +0000 mon.a (mon.0) 2108 : cluster [DBG] osdmap e862: 8 total, 8 up, 8 in 2021-03-24T22:07:13.626 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3 2021-03-24T22:07:13.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:13.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.3' 2021-03-24T22:07:13.732 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:07:13.782 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:07:13.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:13.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:07:13.866 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2021-03-24T22:07:13.866 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2021-03-24T22:07:13.866 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2021-03-24T22:07:13.867 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2021-03-24T22:07:13.867 INFO:teuthology.orchestra.run.smithi142.stdout:} 2021-03-24T22:07:13.889 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:07:13.946 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:07:13.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:13.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:07:14.050 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:07:14.106 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:07:14.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:14.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:07:14.212 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:07:14.269 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:07:14.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:14.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:07:14.376 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:07:14.431 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:07:14.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:14.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:07:14.469 INFO:tasks.rados.rados.0.smithi125.stdout:3353: done (1 left) 2021-03-24T22:07:14.469 INFO:tasks.rados.rados.0.smithi125.stdout:3354: snap_create 2021-03-24T22:07:14.540 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:07:14.560 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:14 smithi125 conmon[19936]: cluster 2021-03-24T22:07:13.022485+0000 mgr.y (mgr.14144) 709 : cluster [DBG] pgmap v1456: 168 pgs: 168 active+clean; 582 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 983 KiB/s wr, 7 op/s 2021-03-24T22:07:14.560 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:14 smithi125 conmon[19936]: 2021-03-24T22:07:14.561 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:14 smithi125 conmon[19936]: cluster 2021-03-24T22:07:13.460154+0000 mon.a (mon.0) 2109 : cluster [DBG] osdmap e863: 8 total, 8 up, 8 in 2021-03-24T22:07:14.561 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:14 smithi125 conmon[24927]: cluster 2021-03-24T22:07:14.561 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:14 smithi125 conmon[24927]: 2021-03-24T22:07:14.562 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:14 smithi125 conmon[24927]: 2021-03-24T22: 2021-03-24T22:07:14.562 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:14 smithi125 conmon[24927]: 07:13.022485+0000 mgr.y (mgr.14144) 709 : cluster [DBG] pgmap v1456: 168 pgs: 168 active+clean; 582 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 983 KiB/s wr, 7 op/s 2021-03-24T22:07:14.562 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:14 smithi125 conmon[24927]: cluster 2021-03-24T22:07:14.562 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:14 smithi125 conmon[24927]: 2021-03-24T22:07:13.460154+0000 mon.a (mon 2021-03-24T22:07:14.562 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:14 smithi125 conmon[24927]: .0) 2109 : cluster [DBG] osdmap e863: 8 total, 8 up, 8 in 2021-03-24T22:07:14.602 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:07:14.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:14.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:07:14.708 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:07:14.763 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:07:14.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:14.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:07:14.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:14 smithi142 conmon[20125]: cluster 2021-03-24T22 2021-03-24T22:07:14.867 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:14 smithi142 conmon[20125]: :07:13.022485+0000 mgr.y (mgr.14144) 709 : cluster [DBG] pgmap v1456: 168 pgs: 168 active+clean; 582 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 983 KiB/s wr, 7 op/s 2021-03-24T22:07:14.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:14 smithi142 conmon[20125]: cluster 2021-03-24T22 2021-03-24T22:07:14.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:14 smithi142 conmon[20125]: :07:13.460154+0000 2021-03-24T22:07:14.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:14 smithi142 conmon[20125]: mon.a (mon.0) 2109 2021-03-24T22:07:14.868 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:14 smithi142 conmon[20125]: : cluster [DBG] osdmap e863: 8 total, 8 up, 8 in 2021-03-24T22:07:14.869 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:07:14.925 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:07:14.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:14.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:07:14.960 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.6 dump_historic_ops 2021-03-24T22:07:15.033 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:07:15.088 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:07:15.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:15.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:07:15.192 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:07:15.246 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:07:15.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:15.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:07:15.264 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.b/config 2021-03-24T22:07:15.352 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:07:15.408 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:07:15.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:15.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:07:15.465 INFO:tasks.rados.rados.0.smithi125.stdout:3351: done (1 left) 2021-03-24T22:07:15.466 INFO:tasks.rados.rados.0.smithi125.stdout:3354: done (0 left) 2021-03-24T22:07:15.466 INFO:tasks.rados.rados.0.smithi125.stdout:3355: read oid 39 snap -1 2021-03-24T22:07:15.466 INFO:tasks.rados.rados.0.smithi125.stdout:3355: expect (ObjNum 1120 snap 383 seq_num 1120) 2021-03-24T22:07:15.466 INFO:tasks.rados.rados.0.smithi125.stdout:3356: copy_from oid 42 from oid 4 current snap is 392 2021-03-24T22:07:15.467 INFO:tasks.rados.rados.0.smithi125.stdout:3357: write (excl) oid 16 current snap is 392 2021-03-24T22:07:15.467 INFO:tasks.rados.rados.0.smithi125.stdout:3357: seq_num 1143 ranges {410401=662557,1719036=770066,2938716=271685} 2021-03-24T22:07:15.468 INFO:teuthology.orchestra.run.smithi125.stderr:noscrub is unset 2021-03-24T22:07:15.472 INFO:tasks.rados.rados.0.smithi125.stdout:3357: writing smithi12556470-16 from 410401 to 1072958 tid 1 2021-03-24T22:07:15.483 INFO:tasks.rados.rados.0.smithi125.stdout:3357: writing smithi12556470-16 from 1719036 to 2489102 tid 2 2021-03-24T22:07:15.488 INFO:tasks.rados.rados.0.smithi125.stdout:3357: writing smithi12556470-16 from 2938716 to 3210401 tid 3 2021-03-24T22:07:15.489 INFO:tasks.rados.rados.0.smithi125.stdout:3358: snap_create 2021-03-24T22:07:15.489 INFO:tasks.rados.rados.0.smithi125.stdout:3355: expect (ObjNum 1120 snap 383 seq_num 1120) 2021-03-24T22:07:15.499 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:15 smithi125 conmon[19936]: cluster 2021-03-24T22:07:14.463350+0000 mon.a (mon. 2021-03-24T22:07:15.499 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:15 smithi125 conmon[19936]: 0) 2110 : cluster [DBG] osdmap e864: 8 total, 8 up, 8 in 2021-03-24T22:07:15.500 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:15 smithi125 conmon[19936]: audit 2021-03-24T22:07:15.356298+0000 mon.a (mon.0) 2111 : audit 2021-03-24T22:07:15.500 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:15 smithi125 conmon[19936]: [INF] from='client.? v1:172.21.15.125:0/1089452897' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T22:07:15.500 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:15 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:07:15.500 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:15 smithi125 conmon[19936]: 07:15.449028+0000 mon.a (mon.0) 2112 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:07:15.501 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:15 smithi125 conmon[19936]: audit 2021-03-24T22:07:15. 2021-03-24T22:07:15.501 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:15 smithi125 conmon[19936]: 458022+0000 mon.a (mon.0) 2113 : audit [INF] from='client.? v1:172.21.15.125:0/1089452897' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T22:07:15.501 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:15 smithi125 conmon[19936]: 2021-03-24T22:07:15.501 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:15 smithi125 conmon[19936]: cluster 2021-03-24T22:07:15 2021-03-24T22:07:15.501 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:15 smithi125 conmon[19936]: .458212+0000 mon.a (mon.0) 2114 : cluster [DBG] osdmap e865: 8 total, 8 up, 8 in 2021-03-24T22:07:15.502 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:15 smithi125 conmon[24927]: cluster 2021-03-24T22:07:14.463350 2021-03-24T22:07:15.502 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:15 smithi125 conmon[24927]: +0000 mon.a (mon.0) 2110 : cluster [DBG] osdmap e864: 8 total, 8 up, 8 in 2021-03-24T22:07:15.503 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:15 smithi125 conmon[24927]: audit 2021-03-24T 2021-03-24T22:07:15.503 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:15 smithi125 conmon[24927]: 22:07:15.356298+0000 mon.a (mon.0) 2111 : audit 2021-03-24T22:07:15.503 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:15 smithi125 conmon[24927]: [INF] from='client.? v1:172.21.15.125:0/1089452897' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T22:07:15.503 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:15 smithi125 conmon[24927]: cluster 2021-03-24T22:07 2021-03-24T22:07:15.503 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:15 smithi125 conmon[24927]: :15.449028+0000 mon.a (mon.0 2021-03-24T22:07:15.504 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:15 smithi125 conmon[24927]: ) 2112 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:07:15.504 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:15 smithi125 conmon[24927]: audit 2021-03-24T22:07:15.504 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:15 smithi125 conmon[24927]: 2021-03-24T22:07:15.458022 2021-03-24T22:07:15.504 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:15 smithi125 conmon[24927]: +0000 mon.a (mon.0) 2113 : audit [INF] 2021-03-24T22:07:15.505 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:15 smithi125 conmon[24927]: from='client.? v1:172.21.15.125:0/1089452897' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T22:07:15.505 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:15 smithi125 conmon[24927]: cluster 2021-03-24T22 2021-03-24T22:07:15.505 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:15 smithi125 conmon[24927]: :07:15.458212+0000 mon.a (mon 2021-03-24T22:07:15.505 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:15 smithi125 conmon[24927]: .0) 2114 : cluster [DBG] osdmap e865: 8 total, 8 up, 8 in 2021-03-24T22:07:15.515 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:07:15.562 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:07:15.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:15.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:07:15.598 INFO:tasks.rados.rados.0.smithi125.stdout:3357: finishing write tid 1 to smithi12556470-16 2021-03-24T22:07:15.599 INFO:tasks.rados.rados.0.smithi125.stdout:3357: finishing write tid 2 to smithi12556470-16 2021-03-24T22:07:15.599 INFO:tasks.rados.rados.0.smithi125.stdout:3357: finishing write tid 3 to smithi12556470-16 2021-03-24T22:07:15.599 INFO:tasks.rados.rados.0.smithi125.stdout:3357: finishing write tid 4 to smithi12556470-16 2021-03-24T22:07:15.599 INFO:tasks.rados.rados.0.smithi125.stdout:3357: finishing write tid 5 to smithi12556470-16 2021-03-24T22:07:15.599 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 16 v 847 (ObjNum 1143 snap 392 seq_num 1143) dirty exists 2021-03-24T22:07:15.600 INFO:tasks.rados.rados.0.smithi125.stdout:3357: left oid 16 (ObjNum 1143 snap 392 seq_num 1143) 2021-03-24T22:07:15.600 INFO:tasks.rados.rados.0.smithi125.stdout:3356: finishing copy_from to smithi12556470-42 2021-03-24T22:07:15.600 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 42 v 1469 (ObjNum 1111 snap 381 seq_num 1111) dirty exists 2021-03-24T22:07:15.600 INFO:tasks.rados.rados.0.smithi125.stdout:3356: finishing copy_from racing read to smithi12556470-42 2021-03-24T22:07:15.639 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:15 smithi142 conmon[20125]: cluster 2021-03-24T22:07:14.463350+0000 mon.a ( 2021-03-24T22:07:15.640 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:15 smithi142 conmon[20125]: mon.0) 2110 : cluster [DBG] osdmap e864: 8 total, 8 up, 8 in 2021-03-24T22:07:15.640 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:15 smithi142 conmon[20125]: audit 2021-03-24T22:07:15.356298+0000 mon.a (mon.0) 2111 : audit [INF] from='client.? v1:172.21.15.125:0/1089452897' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-24T22:07:15.640 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:15 smithi142 conmon[20125]: cluster 2021-03-24T22:07:15.449028+0000 mon.a (mon.0) 2112 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-24T22:07:15.640 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:15 smithi142 conmon[20125]: 2021-03-24T22:07:15.641 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:15 smithi142 conmon[20125]: audit 2021-03-24 2021-03-24T22:07:15.641 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:15 smithi142 conmon[20125]: T22:07:15.458022+0000 mon.a (mon.0) 2113 : audit [INF] from='client.? v1:172.21.15.125:0/1089452897' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-24T22:07:15.641 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:15 smithi142 conmon[20125]: cluster 2021-03-24T22:07:15 2021-03-24T22:07:15.641 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:15 smithi142 conmon[20125]: .458212+0000 mon.a (mon.0) 2114 : cluster [DBG] osdmap e865: 8 total, 8 up, 8 in 2021-03-24T22:07:15.669 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:07:15.722 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6 2021-03-24T22:07:15.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:15.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.6' 2021-03-24T22:07:15.827 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:07:15.874 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:07:15.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:15.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:07:15.979 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:07:16.040 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:07:16.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:16.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:07:16.147 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:07:16.203 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:07:16.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:16.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:07:16.309 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:07:16.371 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:07:16.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:16.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:07:16.477 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:07:16.478 INFO:tasks.rados.rados.0.smithi125.stdout:3355: done (3 left) 2021-03-24T22:07:16.479 INFO:tasks.rados.rados.0.smithi125.stdout:3356: done (2 left) 2021-03-24T22:07:16.479 INFO:tasks.rados.rados.0.smithi125.stdout:3357: done (1 left) 2021-03-24T22:07:16.479 INFO:tasks.rados.rados.0.smithi125.stdout:3358: done (0 left) 2021-03-24T22:07:16.479 INFO:tasks.rados.rados.0.smithi125.stdout:3359: snap_remove snap 384 2021-03-24T22:07:16.523 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0 2021-03-24T22:07:16.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:16.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.0' 2021-03-24T22:07:16.617 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:16 smithi142 conmon[20125]: cluster 2021-03-24T22:07:14.811547+0000 2021-03-24T22:07:16.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:16 smithi142 conmon[20125]: osd.0 (osd.0) 170 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:07:16.618 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:16 smithi142 conmon[20125]: cluster 2021-03-24T22:07:14.815829+0000 osd.0 (osd.0) 171 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:07:16.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:16 smithi142 conmon[20125]: cluster 2021-03-24T22:07:15.023438+0000 mgr.y (mgr. 2021-03-24T22:07:16.619 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:16 smithi142 conmon[20125]: 14144) 710 : cluster [DBG] pgmap v1459: 168 pgs: 168 active+clean; 590 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.5 MiB/s wr, 8 op/s 2021-03-24T22:07:16.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:16 smithi142 conmon[20125]: audit 2021-03-24T22:07:15.491572+0000 2021-03-24T22:07:16.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:16 smithi142 conmon[20125]: mon.a (mon.0) 2115 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T22:07:16.620 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:16 smithi142 conmon[20125]: audit 2021-03-24T22:07: 2021-03-24T22:07:16.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:16 smithi142 conmon[20125]: 15.492233+0000 mon.a (mon.0) 2116 : audit [INF] 2021-03-24T22:07:16.621 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:16 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T22:07:16.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:16 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:07:16.622 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:16 smithi142 conmon[20125]: 24T22:07:15.506779+0000 osd.0 (osd.0) 172 : cluster [DBG] 2021-03-24T22:07:16.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:16 smithi142 conmon[20125]: 3.1 scrub ok 2021-03-24T22:07:16.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:16 smithi142 conmon[20125]: audit 2021-03 2021-03-24T22:07:16.623 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:16 smithi142 conmon[20125]: -24T22:07:15.831640+0000 mon.a (mon.0) 2117 : audit [INF] 2021-03-24T22:07:16.624 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:16 smithi142 conmon[20125]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:07:16.625 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:16 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:07:16.625 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:16 smithi142 conmon[20125]: -24T22:07:16.160942+0000 osd.6 (osd.6) 89 : cluster 2021-03-24T22:07:16.626 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:16 smithi142 conmon[20125]: [DBG] 7.12 scrub ok 2021-03-24T22:07:16.626 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:16 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:07:16.627 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:16 smithi142 conmon[20125]: 03-24T22:07:16.473725+0000 mon.a (mon.0) 2118 : cluster 2021-03-24T22:07:16.627 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:16 smithi142 conmon[20125]: [DBG] osdmap e866: 8 total, 8 up, 8 in 2021-03-24T22:07:16.628 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-24T22:07:16.630 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:07:16.688 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:07:16.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:16.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:07:16.793 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:07:16.854 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5 2021-03-24T22:07:16.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:16.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.5' 2021-03-24T22:07:16.912 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:16 smithi125 conmon[24927]: cluster 2021-03-24T22:07:14.811547 2021-03-24T22:07:16.912 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:16 smithi125 conmon[24927]: +0000 osd.0 (osd.0) 170 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:07:16.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:16 smithi125 conmon[24927]: cluster 2021-03-24T22:07:16.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:16 smithi125 conmon[24927]: 2021-03-24T22:07:14.815829+0000 osd.0 (osd.0) 2021-03-24T22:07:16.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:16 smithi125 conmon[24927]: 171 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:07:16.913 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:16 smithi125 conmon[24927]: cluster 2021- 2021-03-24T22:07:16.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:16 smithi125 conmon[24927]: 03-24T22:07:15.023438+0000 2021-03-24T22:07:16.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:16 smithi125 conmon[24927]: mgr.y (mgr.14144) 710 : cluster [DBG] 2021-03-24T22:07:16.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:16 smithi125 conmon[24927]: pgmap v1459: 168 pgs: 168 active+clean; 590 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.5 MiB/s wr, 8 op/s 2021-03-24T22:07:16.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:16 smithi125 conmon[24927]: audit 2021-03- 2021-03-24T22:07:16.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:16 smithi125 conmon[24927]: 24T22:07:15.491572+0000 mon.a 2021-03-24T22:07:16.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:16 smithi125 conmon[24927]: (mon.0) 2115 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T22:07:16.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:16 smithi125 conmon[24927]: 2021-03-24T22:07:16.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:16 smithi125 conmon[24927]: audit 2021-03-24T22 2021-03-24T22:07:16.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:16 smithi125 conmon[24927]: :07:15.492233+0000 mon.a (mon.0) 2021-03-24T22:07:16.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:16 smithi125 conmon[24927]: 2116 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T22:07:16.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:16 smithi125 conmon[24927]: cluster 2021-03-24T22:07:15 2021-03-24T22:07:16.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:16 smithi125 conmon[24927]: .506779+0000 osd.0 (osd.0) 172 : cluster [DBG] 3.1 scrub ok 2021-03-24T22:07:16.917 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:16 smithi125 conmon[24927]: 2021-03-24T22:07:16.917 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:16 smithi125 conmon[24927]: audit 2021-03-24T22:07:16.917 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:16 smithi125 conmon[24927]: 2021-03-24T22:07:15.831640+0000 mon.a ( 2021-03-24T22:07:16.917 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:16 smithi125 conmon[24927]: mon.0) 2117 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:07:16.917 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:16 smithi125 conmon[24927]: cluster 2021-03-24T22:07: 2021-03-24T22:07:16.918 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:16 smithi125 conmon[24927]: 16.160942+0000 osd.6 (osd.6) 89 : cluster [DBG] 7.12 scrub ok 2021-03-24T22:07:16.918 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:16 smithi125 conmon[24927]: cluster 2021-03-24T22: 2021-03-24T22:07:16.918 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:16 smithi125 conmon[24927]: 07:16.473725+0000 mon.a (mon.0) 2118 : cluster [DBG] osdmap e866: 8 total, 8 up, 8 in 2021-03-24T22:07:16.919 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:16 smithi125 conmon[19936]: cluster 2021-03-24T22:07:14.811547+0000 osd.0 (osd.0) 170 : cluster [DBG] purged_snaps scrub starts 2021-03-24T22:07:16.919 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:16 smithi125 conmon[19936]: cluster 2021-03-24T22:07:14.815829+0000 osd.0 (osd.0) 171 : cluster [DBG] purged_snaps scrub ok 2021-03-24T22:07:16.919 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:16 smithi125 conmon[19936]: cluster 2021-03-24T22:07:15.023438+0000 mgr.y (mgr.14144) 710 : cluster [DBG] pgmap v1459: 168 pgs: 168 active+clean; 590 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.5 MiB/s wr, 8 op/s 2021-03-24T22:07:16.919 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:16 smithi125 conmon[19936]: audit 2021-03-24T22:07:16.919 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:16 smithi125 conmon[19936]: 2021-03-24T22:07:15.491572+0000 mon.a (mon.0) 2115 : audit [DBG] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-24T22:07:16.920 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:16 smithi125 conmon[19936]: audit 2021-03-24T22:07 2021-03-24T22:07:16.920 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:16 smithi125 conmon[19936]: :15.492233+0000 mon.a (mon.0) 2116 : audit [INF] 2021-03-24T22:07:16.920 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:16 smithi125 conmon[19936]: from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-24T22:07:16.921 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:16 smithi125 conmon[19936]: cluster 2021-03-24T22: 2021-03-24T22:07:16.921 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:16 smithi125 conmon[19936]: 07:15.506779+0000 osd.0 (osd 2021-03-24T22:07:16.921 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:16 smithi125 conmon[19936]: .0) 172 : cluster [DBG] 3.1 scrub ok 2021-03-24T22:07:16.922 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:16 smithi125 conmon[19936]: audit 2021-03 2021-03-24T22:07:16.922 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:16 smithi125 conmon[19936]: -24T22:07:15.831640+0000 mon.a (mon 2021-03-24T22:07:16.923 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:16 smithi125 conmon[19936]: .0) 2117 : audit [INF] from='mgr.14144 v1:172.21.15.125:0/3330039452' entity='mgr.y' 2021-03-24T22:07:16.923 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:16 smithi125 conmon[19936]: cluster 2021-03- 2021-03-24T22:07:16.924 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:16 smithi125 conmon[19936]: 24T22:07:16.160942+0000 osd.6 (osd.6) 89 : cluster [DBG] 2021-03-24T22:07:16.924 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:16 smithi125 conmon[19936]: 7.12 scrub ok 2021-03-24T22:07:16.924 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:16 smithi125 conmon[19936]: cluster 2021-03-24T22:07:16.473725+0000 mon.a 2021-03-24T22:07:16.925 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:16 smithi125 conmon[19936]: (mon.0) 2118 : cluster [DBG] osdmap e866: 8 total, 8 up, 8 in 2021-03-24T22:07:16.944 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/3cfa339a-8cea-11eb-90b2-001a4aab830c/mon.a/config 2021-03-24T22:07:16.960 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:07:17.015 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:07:17.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:17.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:07:17.121 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:07:17.184 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:07:17.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:17.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:07:17.290 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:07:17.349 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:07:17.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:17.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:07:17.393 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2021-03-24T22:07:17.394 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2021-03-24T22:07:17.394 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2021-03-24T22:07:17.394 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2021-03-24T22:07:17.395 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.395 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:5168 7.c 7.3b9cea0c (undecoded) ondisk+write+known_if_redirected e532)", 2021-03-24T22:07:17.395 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:01:42.355376+0000", 2021-03-24T22:07:17.395 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 335.03384108199998, 2021-03-24T22:07:17.395 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.016998772, 2021-03-24T22:07:17.396 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:07:17.396 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "reached pg", 2021-03-24T22:07:17.396 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:07:17.397 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:07:17.397 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:07:17.397 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 5168 2021-03-24T22:07:17.397 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.397 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:07:17.398 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.398 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:07:17.398 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:42.355376+0000", 2021-03-24T22:07:17.398 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:07:17.399 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.399 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.399 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:07:17.399 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:42.355376+0000", 2021-03-24T22:07:17.399 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999967 2021-03-24T22:07:17.400 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.400 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.400 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:07:17.401 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:42.355372+0000", 2021-03-24T22:07:17.401 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7163999999999999e-05 2021-03-24T22:07:17.401 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.401 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.402 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:07:17.402 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:42.355399+0000", 2021-03-24T22:07:17.402 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0228e-05 2021-03-24T22:07:17.402 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.403 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.403 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:07:17.403 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:42.355420+0000", 2021-03-24T22:07:17.403 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2044e-05 2021-03-24T22:07:17.404 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.404 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.404 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:07:17.404 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:42.355432+0000", 2021-03-24T22:07:17.405 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029462675000000001 2021-03-24T22:07:17.405 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.405 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.405 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:07:17.405 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:42.384894+0000", 2021-03-24T22:07:17.406 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2527000000000001e-05 2021-03-24T22:07:17.406 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.406 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.406 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for peered", 2021-03-24T22:07:17.406 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:42.384907+0000", 2021-03-24T22:07:17.408 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.98745425200000003 2021-03-24T22:07:17.408 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.408 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.408 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:07:17.409 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:43.372361+0000", 2021-03-24T22:07:17.409 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3349e-05 2021-03-24T22:07:17.409 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.410 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.410 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:07:17.410 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:01:43.372374+0000", 2021-03-24T22:07:17.410 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.016998772 2021-03-24T22:07:17.411 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:07:17.411 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:07:17.411 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:07:17.411 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.411 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.412 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:6121 7.c 7:32974111:::smithi12556470-2:head [copy-from ver 827] snapc 202=[202,201,1ff,1fd,1f8,1f3,1f0,1ef,1ee,1ec,1e6,1e1,1df,1de,1dc,1d8,1d4,1cd,1bf,1bd,19c,199,172,16e] ondisk+write+known_if_redirected e633)", 2021-03-24T22:07:17.412 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:03:23.467594+0000", 2021-03-24T22:07:17.412 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 233.92162275499999, 2021-03-24T22:07:17.413 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031992291999999999, 2021-03-24T22:07:17.413 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:07:17.413 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:07:17.413 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:07:17.413 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:07:17.414 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:07:17.414 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 6121 2021-03-24T22:07:17.414 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.414 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:07:17.415 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.415 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:07:17.415 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.467594+0000", 2021-03-24T22:07:17.415 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:07:17.416 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.416 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.417 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:07:17.417 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.467594+0000", 2021-03-24T22:07:17.417 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2021-03-24T22:07:17.417 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.418 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.418 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:07:17.418 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.467593+0000", 2021-03-24T22:07:17.419 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8450000000000002e-06 2021-03-24T22:07:17.419 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.419 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.419 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:07:17.419 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.467598+0000", 2021-03-24T22:07:17.420 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.2989999999999997e-06 2021-03-24T22:07:17.420 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.420 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.420 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:07:17.421 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.467606+0000", 2021-03-24T22:07:17.421 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1449999999999996e-06 2021-03-24T22:07:17.421 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.421 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.421 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:07:17.422 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.467613+0000", 2021-03-24T22:07:17.422 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00037132700000000002 2021-03-24T22:07:17.422 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.422 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.422 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:07:17.423 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.467984+0000", 2021-03-24T22:07:17.423 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00019477700000000001 2021-03-24T22:07:17.423 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.423 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.424 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:07:17.424 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.468179+0000", 2021-03-24T22:07:17.424 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.012803760000000001 2021-03-24T22:07:17.424 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.424 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.425 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3,7", 2021-03-24T22:07:17.425 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.480982+0000", 2021-03-24T22:07:17.425 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011532875 2021-03-24T22:07:17.425 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.425 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.426 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:07:17.426 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.492515+0000", 2021-03-24T22:07:17.426 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00058181400000000005 2021-03-24T22:07:17.426 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.427 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.427 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T22:07:17.427 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.493097+0000", 2021-03-24T22:07:17.427 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0063084619999999999 2021-03-24T22:07:17.427 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.428 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.428 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:07:17.428 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.499406+0000", 2021-03-24T22:07:17.429 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6482000000000004e-05 2021-03-24T22:07:17.429 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.429 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.429 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:07:17.429 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.499482+0000", 2021-03-24T22:07:17.430 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000104156 2021-03-24T22:07:17.430 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.430 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.430 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:07:17.431 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.499586+0000", 2021-03-24T22:07:17.431 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031992291999999999 2021-03-24T22:07:17.431 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:07:17.431 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:07:17.431 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:07:17.432 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.432 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.432 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:6122 7.c 7:32974111:::smithi12556470-2:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e633)", 2021-03-24T22:07:17.432 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:03:23.467622+0000", 2021-03-24T22:07:17.432 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 233.92159518599999, 2021-03-24T22:07:17.433 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032215336999999997, 2021-03-24T22:07:17.433 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:07:17.433 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:07:17.433 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:07:17.434 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:07:17.434 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:07:17.434 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 6122 2021-03-24T22:07:17.434 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.434 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:07:17.435 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.435 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:07:17.435 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.467622+0000", 2021-03-24T22:07:17.435 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:07:17.436 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.436 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.436 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:07:17.436 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.467622+0000", 2021-03-24T22:07:17.436 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2021-03-24T22:07:17.437 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.437 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.437 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:07:17.437 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.467621+0000", 2021-03-24T22:07:17.437 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.3989999999999997e-06 2021-03-24T22:07:17.438 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.438 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.438 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:07:17.438 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.467625+0000", 2021-03-24T22:07:17.439 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1046000000000001e-05 2021-03-24T22:07:17.439 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.439 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.439 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:07:17.439 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.467636+0000", 2021-03-24T22:07:17.440 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.5799999999999997e-06 2021-03-24T22:07:17.440 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.440 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.440 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:07:17.441 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.467643+0000", 2021-03-24T22:07:17.441 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00070741200000000004 2021-03-24T22:07:17.441 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.441 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.441 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:07:17.442 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.468350+0000", 2021-03-24T22:07:17.442 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.2981999999999999e-05 2021-03-24T22:07:17.442 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.442 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.442 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for blocked object", 2021-03-24T22:07:17.443 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.468433+0000", 2021-03-24T22:07:17.443 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.020434994000000001 2021-03-24T22:07:17.443 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.443 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.444 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:07:17.444 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.488868+0000", 2021-03-24T22:07:17.444 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012197399999999999 2021-03-24T22:07:17.444 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.444 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.445 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for rw locks", 2021-03-24T22:07:17.445 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.488990+0000", 2021-03-24T22:07:17.445 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.010652708 2021-03-24T22:07:17.445 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.445 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.446 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:07:17.446 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.499643+0000", 2021-03-24T22:07:17.446 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010662000000000001 2021-03-24T22:07:17.446 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.447 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.447 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:07:17.447 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.499749+0000", 2021-03-24T22:07:17.447 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.7607000000000003e-05 2021-03-24T22:07:17.447 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.448 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.448 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:07:17.448 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:23.499837+0000", 2021-03-24T22:07:17.448 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032215336999999997 2021-03-24T22:07:17.448 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:07:17.449 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:07:17.449 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:07:17.449 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.449 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.450 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:6222 7.d 7:b2218182:::smithi12556470-21:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e642)", 2021-03-24T22:07:17.450 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:03:32.409316+0000", 2021-03-24T22:07:17.450 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 224.97990035800001, 2021-03-24T22:07:17.450 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.982999293, 2021-03-24T22:07:17.450 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:07:17.451 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "reached pg", 2021-03-24T22:07:17.451 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:07:17.451 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:07:17.451 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:07:17.451 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 6222 2021-03-24T22:07:17.452 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.452 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:07:17.452 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.452 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:07:17.453 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:32.409316+0000", 2021-03-24T22:07:17.453 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:07:17.453 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.453 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.453 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:07:17.454 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:32.409316+0000", 2021-03-24T22:07:17.454 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2021-03-24T22:07:17.454 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.454 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.455 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:07:17.455 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:32.409315+0000", 2021-03-24T22:07:17.455 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4319999999999998e-06 2021-03-24T22:07:17.455 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.455 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.456 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:07:17.456 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:32.409321+0000", 2021-03-24T22:07:17.457 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.5619999999999998e-06 2021-03-24T22:07:17.458 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.458 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.458 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:07:17.458 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:32.409328+0000", 2021-03-24T22:07:17.459 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.2499999999999997e-06 2021-03-24T22:07:17.459 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.459 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.459 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:07:17.460 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:32.409334+0000", 2021-03-24T22:07:17.460 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7930000000000003e-05 2021-03-24T22:07:17.460 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.460 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.460 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:07:17.461 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:32.409392+0000", 2021-03-24T22:07:17.461 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0210999999999999e-05 2021-03-24T22:07:17.461 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.462 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.462 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for peered", 2021-03-24T22:07:17.462 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:32.409402+0000", 2021-03-24T22:07:17.462 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.98275195999999998 2021-03-24T22:07:17.462 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.463 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.463 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:07:17.463 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:33.392154+0000", 2021-03-24T22:07:17.463 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016194899999999999 2021-03-24T22:07:17.464 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.464 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.464 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:07:17.464 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:33.392316+0000", 2021-03-24T22:07:17.464 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.982999293 2021-03-24T22:07:17.465 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:07:17.465 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:07:17.465 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:07:17.465 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.465 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.466 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:6223 7.d 7:b2218182:::smithi12556470-21:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e642)", 2021-03-24T22:07:17.466 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:03:32.409350+0000", 2021-03-24T22:07:17.466 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 224.979867173, 2021-03-24T22:07:17.466 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.983107921, 2021-03-24T22:07:17.467 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:07:17.467 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "reached pg", 2021-03-24T22:07:17.467 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:07:17.467 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:07:17.467 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:07:17.468 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 6223 2021-03-24T22:07:17.468 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.468 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:07:17.468 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.469 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:07:17.469 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:32.409350+0000", 2021-03-24T22:07:17.469 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:07:17.469 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.470 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.470 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:07:17.470 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:32.409350+0000", 2021-03-24T22:07:17.470 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2021-03-24T22:07:17.471 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.471 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.471 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:07:17.471 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:32.409349+0000", 2021-03-24T22:07:17.471 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2540000000000003e-06 2021-03-24T22:07:17.472 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.472 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.472 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:07:17.472 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:32.409353+0000", 2021-03-24T22:07:17.473 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7969999999999997e-06 2021-03-24T22:07:17.473 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.473 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.473 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:07:17.473 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:32.409359+0000", 2021-03-24T22:07:17.474 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8630000000000001e-06 2021-03-24T22:07:17.474 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.474 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.474 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:07:17.474 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:32.409363+0000", 2021-03-24T22:07:17.475 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1345000000000002e-05 2021-03-24T22:07:17.475 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.475 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.475 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:07:17.476 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:32.409434+0000", 2021-03-24T22:07:17.476 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0365e-05 2021-03-24T22:07:17.476 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.476 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.476 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for peered", 2021-03-24T22:07:17.477 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:32.409444+0000", 2021-03-24T22:07:17.477 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.98290503500000004 2021-03-24T22:07:17.478 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.478 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.478 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:07:17.478 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:33.392349+0000", 2021-03-24T22:07:17.479 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000108089 2021-03-24T22:07:17.479 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.479 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.479 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:07:17.479 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:33.392457+0000", 2021-03-24T22:07:17.480 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.983107921 2021-03-24T22:07:17.480 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:07:17.480 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:07:17.480 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:07:17.481 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.481 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.481 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:6224 7.d 7:b2218182:::smithi12556470-21:head [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected e642)", 2021-03-24T22:07:17.481 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:03:32.409380+0000", 2021-03-24T22:07:17.481 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 224.97983650800001, 2021-03-24T22:07:17.482 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.98319098800000004, 2021-03-24T22:07:17.482 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:07:17.482 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "reached pg", 2021-03-24T22:07:17.482 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:07:17.482 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:07:17.483 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:07:17.483 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 6224 2021-03-24T22:07:17.483 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.483 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:07:17.484 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.484 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:07:17.484 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:32.409380+0000", 2021-03-24T22:07:17.484 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:07:17.484 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.485 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.485 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:07:17.485 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:32.409380+0000", 2021-03-24T22:07:17.485 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2021-03-24T22:07:17.486 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.486 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.486 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:07:17.486 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:32.409380+0000", 2021-03-24T22:07:17.486 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9380000000000001e-06 2021-03-24T22:07:17.487 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.487 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.487 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:07:17.487 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:32.409383+0000", 2021-03-24T22:07:17.487 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.9140000000000001e-06 2021-03-24T22:07:17.488 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.488 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.488 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:07:17.488 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:32.409387+0000", 2021-03-24T22:07:17.488 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0900000000000001e-06 2021-03-24T22:07:17.489 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.489 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.489 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:07:17.490 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:32.409390+0000", 2021-03-24T22:07:17.490 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9024999999999996e-05 2021-03-24T22:07:17.490 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.490 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.491 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:07:17.491 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:32.409479+0000", 2021-03-24T22:07:17.491 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.1699999999999997e-06 2021-03-24T22:07:17.491 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.491 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.492 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for peered", 2021-03-24T22:07:17.494 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:32.409487+0000", 2021-03-24T22:07:17.494 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.98298822799999996 2021-03-24T22:07:17.495 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.495 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.495 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:07:17.496 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:33.392475+0000", 2021-03-24T22:07:17.497 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6088000000000006e-05 2021-03-24T22:07:17.498 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.498 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.499 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:07:17.499 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:33.392571+0000", 2021-03-24T22:07:17.500 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.98319098800000004 2021-03-24T22:07:17.500 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:07:17.501 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:07:17.501 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:07:17.501 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.502 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.502 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:6374 7.c 7:32974111:::smithi12556470-2:head [copy-from ver 430] snapc 219=[219,218,217,214,211,210,20e,20d,209,204,1fd,1f8,1f3,1f0,1ef,1e1,1de,1dc,1d8,1d4,1cd,1bf,1bd,199,172,16e] ondisk+write+known_if_redirected e655)", 2021-03-24T22:07:17.503 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:03:45.363192+0000", 2021-03-24T22:07:17.504 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 212.026025111, 2021-03-24T22:07:17.505 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.101716506, 2021-03-24T22:07:17.505 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:07:17.506 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:07:17.506 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:07:17.507 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:07:17.507 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:07:17.507 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 6374 2021-03-24T22:07:17.508 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.508 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:07:17.509 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.509 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:07:17.510 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.363192+0000", 2021-03-24T22:07:17.510 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:07:17.511 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.515 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.515 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:07:17.516 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.363192+0000", 2021-03-24T22:07:17.517 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2021-03-24T22:07:17.517 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.518 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.518 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:07:17.518 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.363190+0000", 2021-03-24T22:07:17.519 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8980000000000002e-06 2021-03-24T22:07:17.519 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.519 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.519 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:07:17.519 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.363199+0000", 2021-03-24T22:07:17.520 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.9709999999999999e-06 2021-03-24T22:07:17.520 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.520 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.520 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:07:17.521 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.363209+0000", 2021-03-24T22:07:17.521 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9600000000000006e-06 2021-03-24T22:07:17.521 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.521 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.521 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:07:17.522 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.363218+0000", 2021-03-24T22:07:17.522 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1470999999999994e-05 2021-03-24T22:07:17.522 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.522 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.523 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:07:17.523 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.363289+0000", 2021-03-24T22:07:17.523 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000128021 2021-03-24T22:07:17.523 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.524 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.524 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:07:17.524 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.363417+0000", 2021-03-24T22:07:17.524 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.090174975 2021-03-24T22:07:17.524 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.525 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.525 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3,7", 2021-03-24T22:07:17.525 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.453592+0000", 2021-03-24T22:07:17.525 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0083544299999999995 2021-03-24T22:07:17.526 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.526 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.526 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:07:17.526 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.461947+0000", 2021-03-24T22:07:17.527 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000299795 2021-03-24T22:07:17.527 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.528 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.528 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T22:07:17.528 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.462246+0000", 2021-03-24T22:07:17.529 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0025305150000000001 2021-03-24T22:07:17.529 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.530 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.530 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:07:17.531 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.464777+0000", 2021-03-24T22:07:17.531 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.7652999999999998e-05 2021-03-24T22:07:17.531 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.531 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.532 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:07:17.532 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.464844+0000", 2021-03-24T22:07:17.532 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3607000000000003e-05 2021-03-24T22:07:17.532 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.533 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.533 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:07:17.533 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.464908+0000", 2021-03-24T22:07:17.533 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.101716506 2021-03-24T22:07:17.533 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:07:17.534 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:07:17.534 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:07:17.534 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.534 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.535 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:6375 7.c 7:32974111:::smithi12556470-2:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e655)", 2021-03-24T22:07:17.535 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:03:45.363568+0000", 2021-03-24T22:07:17.535 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 212.02564881699999, 2021-03-24T22:07:17.535 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.101765479, 2021-03-24T22:07:17.535 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:07:17.536 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:07:17.536 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:07:17.536 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:07:17.536 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:07:17.537 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 6375 2021-03-24T22:07:17.537 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.537 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:07:17.537 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.537 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:07:17.538 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.363568+0000", 2021-03-24T22:07:17.538 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:07:17.538 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.538 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.539 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:07:17.539 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.363568+0000", 2021-03-24T22:07:17.539 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2021-03-24T22:07:17.539 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.539 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.540 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:07:17.540 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.363567+0000", 2021-03-24T22:07:17.540 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0079999999999999e-06 2021-03-24T22:07:17.540 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.540 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.541 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:07:17.541 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.363573+0000", 2021-03-24T22:07:17.541 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9039999999999998e-06 2021-03-24T22:07:17.541 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.542 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.542 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:07:17.542 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.363582+0000", 2021-03-24T22:07:17.542 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1290000000000001e-06 2021-03-24T22:07:17.542 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.543 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.543 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:07:17.543 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.363588+0000", 2021-03-24T22:07:17.543 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9103000000000006e-05 2021-03-24T22:07:17.544 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.544 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.544 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:07:17.544 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.363657+0000", 2021-03-24T22:07:17.544 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8739000000000001e-05 2021-03-24T22:07:17.545 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.545 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.545 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for blocked object", 2021-03-24T22:07:17.545 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.363716+0000", 2021-03-24T22:07:17.545 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0951070979999999 2021-03-24T22:07:17.546 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.546 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.546 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:07:17.546 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.458823+0000", 2021-03-24T22:07:17.547 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00020338800000000001 2021-03-24T22:07:17.547 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.547 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.547 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for rw locks", 2021-03-24T22:07:17.547 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.459026+0000", 2021-03-24T22:07:17.548 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0059478939999999996 2021-03-24T22:07:17.548 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.548 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.548 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:07:17.549 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.464974+0000", 2021-03-24T22:07:17.549 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016329100000000001 2021-03-24T22:07:17.549 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.549 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.549 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:07:17.550 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.465137+0000", 2021-03-24T22:07:17.550 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000196121 2021-03-24T22:07:17.550 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.551 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.551 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:07:17.552 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:46.465333+0000", 2021-03-24T22:07:17.552 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.101765479 2021-03-24T22:07:17.553 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:07:17.553 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:07:17.553 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:07:17.553 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.554 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.554 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:6381 7.19 7:9c4f4ea5:::smithi12556470-41:head [write 521248~563025] snapc 219=[219,218,217,214,211,210,20e,20d,209,204,1fd,1f8,1f3,1f0,1ef,1e1,1de,1dc,1d8,1d4,1cd,1bf,1bd,199,172,16e] ondisk+write+known_if_redirected e655)", 2021-03-24T22:07:17.554 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:03:45.373171+0000", 2021-03-24T22:07:17.555 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 212.01604531800001, 2021-03-24T22:07:17.555 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038785187999999998, 2021-03-24T22:07:17.555 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:07:17.555 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:07:17.555 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:07:17.556 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:07:17.556 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:07:17.556 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 6381 2021-03-24T22:07:17.556 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.556 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:07:17.557 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.557 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:07:17.557 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.373171+0000", 2021-03-24T22:07:17.557 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:07:17.558 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.558 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.558 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:07:17.558 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.373171+0000", 2021-03-24T22:07:17.558 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999957 2021-03-24T22:07:17.559 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.559 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.559 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:07:17.559 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.373167+0000", 2021-03-24T22:07:17.560 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00070376200000000003 2021-03-24T22:07:17.560 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.560 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.560 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:07:17.560 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.373871+0000", 2021-03-24T22:07:17.561 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9023999999999999e-05 2021-03-24T22:07:17.561 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.561 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.562 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:07:17.562 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.373890+0000", 2021-03-24T22:07:17.562 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.4950000000000002e-06 2021-03-24T22:07:17.562 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.562 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.563 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:07:17.563 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.373897+0000", 2021-03-24T22:07:17.563 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2100000000000005e-05 2021-03-24T22:07:17.563 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.564 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.564 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:07:17.564 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.373960+0000", 2021-03-24T22:07:17.564 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000102253 2021-03-24T22:07:17.564 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.565 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.565 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:07:17.565 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.374062+0000", 2021-03-24T22:07:17.565 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000207677 2021-03-24T22:07:17.566 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.566 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.566 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 2,7", 2021-03-24T22:07:17.566 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.374269+0000", 2021-03-24T22:07:17.566 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0017298019999999999 2021-03-24T22:07:17.567 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.567 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.567 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T22:07:17.567 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.375999+0000", 2021-03-24T22:07:17.567 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00028531999999999999 2021-03-24T22:07:17.568 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.568 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.568 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:07:17.568 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.376285+0000", 2021-03-24T22:07:17.569 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.035515931000000001 2021-03-24T22:07:17.569 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.569 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.569 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:07:17.569 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.411801+0000", 2021-03-24T22:07:17.570 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1810000000000006e-05 2021-03-24T22:07:17.570 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.570 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.570 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:07:17.571 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.411862+0000", 2021-03-24T22:07:17.571 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4266999999999997e-05 2021-03-24T22:07:17.571 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.571 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.571 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:07:17.572 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.411957+0000", 2021-03-24T22:07:17.572 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038785187999999998 2021-03-24T22:07:17.572 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:07:17.572 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:07:17.573 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:07:17.573 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.573 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.573 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:6382 7.19 7:9c4f4ea5:::smithi12556470-41:head [write 1809501~596300] snapc 219=[219,218,217,214,211,210,20e,20d,209,204,1fd,1f8,1f3,1f0,1ef,1e1,1de,1dc,1d8,1d4,1cd,1bf,1bd,199,172,16e] ondisk+write+known_if_redirected e655)", 2021-03-24T22:07:17.573 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:03:45.380415+0000", 2021-03-24T22:07:17.574 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 212.00880127600001, 2021-03-24T22:07:17.574 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031748200999999997, 2021-03-24T22:07:17.574 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:07:17.574 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:07:17.574 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:07:17.575 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:07:17.575 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:07:17.575 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 6382 2021-03-24T22:07:17.575 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.576 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:07:17.576 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.576 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:07:17.576 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.380415+0000", 2021-03-24T22:07:17.576 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:07:17.577 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.577 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.577 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:07:17.577 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.380415+0000", 2021-03-24T22:07:17.577 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2021-03-24T22:07:17.578 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.578 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.578 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:07:17.578 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.380414+0000", 2021-03-24T22:07:17.579 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00054294300000000005 2021-03-24T22:07:17.579 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.579 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.579 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:07:17.579 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.380957+0000", 2021-03-24T22:07:17.580 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3733e-05 2021-03-24T22:07:17.580 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.580 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.580 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:07:17.581 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.380970+0000", 2021-03-24T22:07:17.581 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1659999999999997e-06 2021-03-24T22:07:17.581 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.581 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.581 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:07:17.582 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.380978+0000", 2021-03-24T22:07:17.582 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6363000000000001e-05 2021-03-24T22:07:17.582 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.582 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.583 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:07:17.583 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.381054+0000", 2021-03-24T22:07:17.583 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010126799999999999 2021-03-24T22:07:17.583 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.583 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.584 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:07:17.584 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.381155+0000", 2021-03-24T22:07:17.584 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015574499999999999 2021-03-24T22:07:17.584 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.584 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.585 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 2,7", 2021-03-24T22:07:17.585 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.381311+0000", 2021-03-24T22:07:17.585 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0018026559999999999 2021-03-24T22:07:17.585 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.586 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.586 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T22:07:17.586 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.383114+0000", 2021-03-24T22:07:17.586 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00053586900000000002 2021-03-24T22:07:17.586 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.587 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.587 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:07:17.587 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.383650+0000", 2021-03-24T22:07:17.587 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028389675999999999 2021-03-24T22:07:17.587 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.588 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.588 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:07:17.588 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.412039+0000", 2021-03-24T22:07:17.588 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2992999999999998e-05 2021-03-24T22:07:17.589 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.589 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.589 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:07:17.589 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.412102+0000", 2021-03-24T22:07:17.589 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1424e-05 2021-03-24T22:07:17.590 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.590 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.590 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:07:17.590 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:03:45.412164+0000", 2021-03-24T22:07:17.591 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031748200999999997 2021-03-24T22:07:17.591 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:07:17.591 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:07:17.591 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:07:17.591 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.592 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.592 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:6732 7.c 7:305739dc:::smithi12556470-36:head [write 645633~716191] snapc 246=[246,244,243,242,240,23f,23d,23b,236,235,234,231,22d,22c,226,225,21f,21d,21b,217,210,209,204,1f8,1ef,1e1,1de,1dc,1d8,1bf,199,172,16e] ondisk+write+known_if_redirected e700)", 2021-03-24T22:07:17.592 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:04:30.387337+0000", 2021-03-24T22:07:17.592 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 167.001879239, 2021-03-24T22:07:17.592 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.03521146, 2021-03-24T22:07:17.593 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:07:17.593 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:07:17.593 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:07:17.593 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:07:17.594 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:07:17.594 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 6732 2021-03-24T22:07:17.594 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.594 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:07:17.594 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.595 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:07:17.595 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:30.387337+0000", 2021-03-24T22:07:17.595 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:07:17.595 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.596 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.596 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:07:17.596 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:30.387337+0000", 2021-03-24T22:07:17.596 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999933 2021-03-24T22:07:17.596 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.597 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.597 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:07:17.597 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:30.387331+0000", 2021-03-24T22:07:17.597 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011328670000000001 2021-03-24T22:07:17.597 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.598 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.598 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:07:17.598 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:30.388464+0000", 2021-03-24T22:07:17.598 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9536e-05 2021-03-24T22:07:17.599 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.599 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.599 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:07:17.599 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:30.388493+0000", 2021-03-24T22:07:17.599 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2812e-05 2021-03-24T22:07:17.600 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.600 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.600 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:07:17.600 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:30.388506+0000", 2021-03-24T22:07:17.600 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6872000000000004e-05 2021-03-24T22:07:17.601 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.601 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.601 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:07:17.601 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:30.388603+0000", 2021-03-24T22:07:17.601 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018648100000000001 2021-03-24T22:07:17.602 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.602 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.602 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:07:17.602 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:30.388789+0000", 2021-03-24T22:07:17.603 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00032753099999999999 2021-03-24T22:07:17.603 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.603 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.603 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3,7", 2021-03-24T22:07:17.603 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:30.389117+0000", 2021-03-24T22:07:17.604 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0047535099999999999 2021-03-24T22:07:17.604 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.604 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.604 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T22:07:17.605 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:30.393870+0000", 2021-03-24T22:07:17.605 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011734843999999999 2021-03-24T22:07:17.605 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.605 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.605 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:07:17.606 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:30.405605+0000", 2021-03-24T22:07:17.606 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.016723930000000001 2021-03-24T22:07:17.606 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.606 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.607 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:07:17.607 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:30.422329+0000", 2021-03-24T22:07:17.607 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000112357 2021-03-24T22:07:17.607 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.607 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.608 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:07:17.608 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:30.422442+0000", 2021-03-24T22:07:17.608 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000107398 2021-03-24T22:07:17.608 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.608 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.609 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:07:17.609 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:04:30.422549+0000", 2021-03-24T22:07:17.609 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.03521146 2021-03-24T22:07:17.609 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:07:17.609 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:07:17.610 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:07:17.610 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.610 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.610 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:7789 7.19 7:9c4f4ea5:::smithi12556470-41:head [write 1570689~568696] snapc 2a5=[2a5,2a4,2a3,29f,29e,29a,297,296,294,28e,28b,289,285,27d,27b,277,276,275,26e,266,25e,25d,255,244,243,23b,231,210] ondisk+write+known_if_redirected e795)", 2021-03-24T22:07:17.611 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:06:05.464982+0000", 2021-03-24T22:07:17.611 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 71.924234229000007, 2021-03-24T22:07:17.611 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.059123416999999998, 2021-03-24T22:07:17.611 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:07:17.611 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:07:17.612 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:07:17.612 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:07:17.612 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:07:17.612 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 7789 2021-03-24T22:07:17.613 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.613 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:07:17.613 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.613 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:07:17.613 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.464982+0000", 2021-03-24T22:07:17.614 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:07:17.614 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.614 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.614 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:07:17.615 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.464982+0000", 2021-03-24T22:07:17.615 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999971 2021-03-24T22:07:17.615 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.615 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.615 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:07:17.616 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.464980+0000", 2021-03-24T22:07:17.616 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00226654 2021-03-24T22:07:17.616 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.616 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.616 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:07:17.617 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.467246+0000", 2021-03-24T22:07:17.617 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9604000000000001e-05 2021-03-24T22:07:17.617 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.617 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.618 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:07:17.618 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.467266+0000", 2021-03-24T22:07:17.618 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1582e-05 2021-03-24T22:07:17.618 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.618 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.619 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:07:17.619 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.467278+0000", 2021-03-24T22:07:17.619 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7515000000000001e-05 2021-03-24T22:07:17.619 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.619 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.620 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:07:17.620 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.467335+0000", 2021-03-24T22:07:17.620 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011945 2021-03-24T22:07:17.620 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.621 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.621 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:07:17.621 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.467455+0000", 2021-03-24T22:07:17.621 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00019271499999999999 2021-03-24T22:07:17.621 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.622 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.622 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 2,7", 2021-03-24T22:07:17.622 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.467647+0000", 2021-03-24T22:07:17.622 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0046442810000000001 2021-03-24T22:07:17.623 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.623 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.623 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T22:07:17.623 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.472292+0000", 2021-03-24T22:07:17.623 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000426864 2021-03-24T22:07:17.624 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.624 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.624 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:07:17.624 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.472718+0000", 2021-03-24T22:07:17.624 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.051239369 2021-03-24T22:07:17.625 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.625 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.625 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:07:17.625 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.523958+0000", 2021-03-24T22:07:17.626 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6876000000000001e-05 2021-03-24T22:07:17.626 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.626 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.626 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:07:17.626 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.524035+0000", 2021-03-24T22:07:17.627 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1252000000000002e-05 2021-03-24T22:07:17.627 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.627 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.627 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:07:17.627 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.524106+0000", 2021-03-24T22:07:17.628 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.059123416999999998 2021-03-24T22:07:17.628 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:07:17.628 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:07:17.628 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:07:17.629 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.629 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.629 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:7790 7.19 7:9c4f4ea5:::smithi12556470-41:head [write 2653560~753488] snapc 2a5=[2a5,2a4,2a3,29f,29e,29a,297,296,294,28e,28b,289,285,27d,27b,277,276,275,26e,266,25e,25d,255,244,243,23b,231,210] ondisk+write+known_if_redirected e795)", 2021-03-24T22:07:17.629 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:06:05.472956+0000", 2021-03-24T22:07:17.629 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 71.916260324000007, 2021-03-24T22:07:17.630 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.051974415000000003, 2021-03-24T22:07:17.630 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:07:17.630 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:07:17.630 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:07:17.631 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:07:17.631 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:07:17.631 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 7790 2021-03-24T22:07:17.631 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.631 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:07:17.632 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.632 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:07:17.632 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.472956+0000", 2021-03-24T22:07:17.632 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:07:17.633 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.633 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.633 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:07:17.633 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.472956+0000", 2021-03-24T22:07:17.633 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2021-03-24T22:07:17.634 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.634 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.634 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:07:17.634 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.472955+0000", 2021-03-24T22:07:17.634 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00078441000000000003 2021-03-24T22:07:17.635 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.635 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.635 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:07:17.635 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.473739+0000", 2021-03-24T22:07:17.636 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7912999999999999e-05 2021-03-24T22:07:17.636 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.636 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.636 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:07:17.636 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.473757+0000", 2021-03-24T22:07:17.637 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.7540000000000001e-06 2021-03-24T22:07:17.637 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.637 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.637 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:07:17.637 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.473767+0000", 2021-03-24T22:07:17.638 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.0536000000000002e-05 2021-03-24T22:07:17.638 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.638 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.638 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:07:17.639 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.473857+0000", 2021-03-24T22:07:17.639 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000129321 2021-03-24T22:07:17.639 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.639 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.639 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:07:17.640 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.473986+0000", 2021-03-24T22:07:17.640 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000227199 2021-03-24T22:07:17.640 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.640 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.641 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 2,7", 2021-03-24T22:07:17.641 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.474214+0000", 2021-03-24T22:07:17.641 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023819412000000002 2021-03-24T22:07:17.641 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.641 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.642 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T22:07:17.642 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.498033+0000", 2021-03-24T22:07:17.642 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00051381799999999998 2021-03-24T22:07:17.642 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.642 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.643 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:07:17.643 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.498547+0000", 2021-03-24T22:07:17.643 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026167927000000001 2021-03-24T22:07:17.643 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.644 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.644 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:07:17.644 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.524715+0000", 2021-03-24T22:07:17.644 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011929099999999999 2021-03-24T22:07:17.644 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.645 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.645 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:07:17.645 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.524834+0000", 2021-03-24T22:07:17.645 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6706000000000005e-05 2021-03-24T22:07:17.645 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.646 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.646 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:07:17.646 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.524931+0000", 2021-03-24T22:07:17.646 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.051974415000000003 2021-03-24T22:07:17.647 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:07:17.647 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:07:17.647 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:07:17.647 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.647 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.648 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:7791 7.19 7:9c4f4ea5:::smithi12556470-41:head [setxattr _header (58) in=7b,truncate 3832227] snapc 2a5=[2a5,2a4,2a3,29f,29e,29a,297,296,294,28e,28b,289,285,27d,27b,277,276,275,26e,266,25e,25d,255,244,243,23b,231,210] ondisk+write+known_if_redirected e795)", 2021-03-24T22:07:17.648 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:06:05.473813+0000", 2021-03-24T22:07:17.648 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 71.915403276000006, 2021-03-24T22:07:17.648 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.051353812999999998, 2021-03-24T22:07:17.649 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:07:17.649 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:07:17.649 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:07:17.649 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:07:17.649 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:07:17.650 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 7791 2021-03-24T22:07:17.650 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.650 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:07:17.650 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.650 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:07:17.651 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.473813+0000", 2021-03-24T22:07:17.651 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:07:17.651 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.651 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.652 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:07:17.652 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.473813+0000", 2021-03-24T22:07:17.652 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2021-03-24T22:07:17.652 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.652 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.653 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:07:17.653 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.473812+0000", 2021-03-24T22:07:17.653 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.8199999999999999e-06 2021-03-24T22:07:17.653 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.653 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.654 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:07:17.654 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.473819+0000", 2021-03-24T22:07:17.654 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.8539999999999992e-06 2021-03-24T22:07:17.654 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.655 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.655 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:07:17.655 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.473829+0000", 2021-03-24T22:07:17.655 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.0559999999999999e-06 2021-03-24T22:07:17.655 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.656 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.656 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:07:17.656 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.473836+0000", 2021-03-24T22:07:17.656 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001741118 2021-03-24T22:07:17.657 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.657 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.657 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:07:17.657 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.475577+0000", 2021-03-24T22:07:17.657 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00017867999999999999 2021-03-24T22:07:17.658 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.658 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.658 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:07:17.658 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.475755+0000", 2021-03-24T22:07:17.658 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00027003699999999999 2021-03-24T22:07:17.659 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.659 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.659 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 2,7", 2021-03-24T22:07:17.659 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.476025+0000", 2021-03-24T22:07:17.660 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022019692 2021-03-24T22:07:17.660 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.660 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.660 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T22:07:17.660 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.498045+0000", 2021-03-24T22:07:17.661 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00055304300000000002 2021-03-24T22:07:17.661 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.661 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.661 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:07:17.661 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.498598+0000", 2021-03-24T22:07:17.662 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026413981 2021-03-24T22:07:17.662 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.662 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.662 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:07:17.663 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.525012+0000", 2021-03-24T22:07:17.663 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2785000000000002e-05 2021-03-24T22:07:17.663 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.663 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.663 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:07:17.664 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.525085+0000", 2021-03-24T22:07:17.664 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.2311000000000005e-05 2021-03-24T22:07:17.664 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.664 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.664 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:07:17.665 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.525167+0000", 2021-03-24T22:07:17.665 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.051353812999999998 2021-03-24T22:07:17.665 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:07:17.665 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:07:17.666 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:07:17.666 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.666 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.666 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:7792 7.19 7:9c4f4ea5:::smithi12556470-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e795)", 2021-03-24T22:07:17.666 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:06:05.473844+0000", 2021-03-24T22:07:17.671 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 71.915373131999999, 2021-03-24T22:07:17.671 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.051685135, 2021-03-24T22:07:17.671 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:07:17.671 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:07:17.671 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:07:17.672 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:07:17.672 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:07:17.672 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 7792 2021-03-24T22:07:17.672 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.673 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:07:17.673 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.673 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:07:17.673 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.473844+0000", 2021-03-24T22:07:17.673 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:07:17.674 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.674 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.674 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:07:17.674 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.473844+0000", 2021-03-24T22:07:17.675 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2021-03-24T22:07:17.675 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.675 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.675 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:07:17.675 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.473843+0000", 2021-03-24T22:07:17.676 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.5730000000000001e-06 2021-03-24T22:07:17.676 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.676 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.676 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:07:17.677 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.473847+0000", 2021-03-24T22:07:17.677 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4319999999999999e-06 2021-03-24T22:07:17.677 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.677 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.677 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:07:17.678 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.473856+0000", 2021-03-24T22:07:17.678 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.789e-06 2021-03-24T22:07:17.678 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.678 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.678 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:07:17.679 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.473862+0000", 2021-03-24T22:07:17.679 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0028242010000000001 2021-03-24T22:07:17.679 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.679 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.680 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:07:17.680 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.476686+0000", 2021-03-24T22:07:17.680 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000103123 2021-03-24T22:07:17.680 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.680 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.681 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for rw locks", 2021-03-24T22:07:17.681 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.476789+0000", 2021-03-24T22:07:17.681 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.048442702999999997 2021-03-24T22:07:17.681 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.681 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.682 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:07:17.682 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.525232+0000", 2021-03-24T22:07:17.682 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013574299999999999 2021-03-24T22:07:17.682 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.683 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.683 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:07:17.683 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.525367+0000", 2021-03-24T22:07:17.683 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016143200000000001 2021-03-24T22:07:17.683 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.684 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.684 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:07:17.684 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:05.525529+0000", 2021-03-24T22:07:17.684 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.051685135 2021-03-24T22:07:17.685 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:07:17.685 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:07:17.685 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:07:17.685 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.685 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.686 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:8060 7.c 7:305739dc:::smithi12556470-36:head [copy-from ver 1441] snapc 2c2=[2c2,2c1,2c0,2bc,2bb,2b8,2b5,2b4,2b0,2ae,2ac,2ab,2a7,2a5,2a4,29f,29e,29a,297,296,294,28b,289,285,27d,27b,277,275,26e,25d,255,210] ondisk+write+known_if_redirected e825)", 2021-03-24T22:07:17.686 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:06:35.446013+0000", 2021-03-24T22:07:17.686 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 41.943203351000001, 2021-03-24T22:07:17.686 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.040258978000000001, 2021-03-24T22:07:17.687 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:07:17.687 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-24T22:07:17.687 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:07:17.687 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:07:17.687 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:07:17.688 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 8060 2021-03-24T22:07:17.688 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.688 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:07:17.688 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.688 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:07:17.689 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:35.446013+0000", 2021-03-24T22:07:17.689 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:07:17.689 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.689 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.690 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:07:17.690 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:35.446013+0000", 2021-03-24T22:07:17.690 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2021-03-24T22:07:17.690 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.690 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.691 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:07:17.691 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:35.446013+0000", 2021-03-24T22:07:17.691 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5650000000000001e-06 2021-03-24T22:07:17.691 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.692 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.692 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:07:17.692 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:35.446018+0000", 2021-03-24T22:07:17.692 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0232e-05 2021-03-24T22:07:17.692 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.693 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.693 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:07:17.693 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:35.446029+0000", 2021-03-24T22:07:17.693 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.3040000000000004e-06 2021-03-24T22:07:17.693 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.694 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.694 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:07:17.694 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:35.446034+0000", 2021-03-24T22:07:17.694 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013084199999999999 2021-03-24T22:07:17.695 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.695 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.695 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:07:17.695 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:35.446165+0000", 2021-03-24T22:07:17.695 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0599999999999998e-06 2021-03-24T22:07:17.696 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.696 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.696 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting_for_map not empty", 2021-03-24T22:07:17.696 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:35.446170+0000", 2021-03-24T22:07:17.696 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.010286045000000001 2021-03-24T22:07:17.697 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.697 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.697 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:07:17.697 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:35.456456+0000", 2021-03-24T22:07:17.698 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.7979999999999994e-05 2021-03-24T22:07:17.698 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.698 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.698 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:07:17.698 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:35.456554+0000", 2021-03-24T22:07:17.699 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0083839899999999992 2021-03-24T22:07:17.699 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.699 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.699 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3,7", 2021-03-24T22:07:17.700 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:35.464938+0000", 2021-03-24T22:07:17.700 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0091201939999999999 2021-03-24T22:07:17.700 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.700 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.700 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2021-03-24T22:07:17.701 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:35.474058+0000", 2021-03-24T22:07:17.701 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00053956000000000002 2021-03-24T22:07:17.701 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.701 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.701 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:07:17.702 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:35.474598+0000", 2021-03-24T22:07:17.702 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011545389 2021-03-24T22:07:17.702 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.702 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.703 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2021-03-24T22:07:17.703 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:35.486143+0000", 2021-03-24T22:07:17.703 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1906e-05 2021-03-24T22:07:17.703 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.703 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.704 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2021-03-24T22:07:17.704 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:35.486205+0000", 2021-03-24T22:07:17.704 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.7512e-05 2021-03-24T22:07:17.704 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.704 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.705 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:07:17.705 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:35.486272+0000", 2021-03-24T22:07:17.705 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.040258978000000001 2021-03-24T22:07:17.705 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:07:17.706 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:07:17.706 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:07:17.706 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.706 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.706 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:8061 7.c 7:305739dc:::smithi12556470-36:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e825)", 2021-03-24T22:07:17.707 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:06:35.446095+0000", 2021-03-24T22:07:17.707 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 41.943121714999997, 2021-03-24T22:07:17.707 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.040434441000000002, 2021-03-24T22:07:17.707 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:07:17.707 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:07:17.708 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:07:17.708 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:07:17.708 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:07:17.708 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 8061 2021-03-24T22:07:17.709 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.709 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:07:17.709 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.709 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:07:17.709 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:35.446095+0000", 2021-03-24T22:07:17.710 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:07:17.710 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.710 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.710 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:07:17.710 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:35.446095+0000", 2021-03-24T22:07:17.711 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2021-03-24T22:07:17.711 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.711 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.711 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:07:17.712 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:35.446094+0000", 2021-03-24T22:07:17.712 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.536e-06 2021-03-24T22:07:17.712 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.712 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.712 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:07:17.713 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:35.446099+0000", 2021-03-24T22:07:17.713 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3899999999999999e-06 2021-03-24T22:07:17.713 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.713 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.713 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:07:17.714 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:35.446108+0000", 2021-03-24T22:07:17.714 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.4080000000000003e-06 2021-03-24T22:07:17.714 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.714 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.715 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:07:17.715 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:35.446115+0000", 2021-03-24T22:07:17.715 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.2384000000000003e-05 2021-03-24T22:07:17.715 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.716 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.716 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:07:17.716 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:35.446197+0000", 2021-03-24T22:07:17.716 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1570000000000003e-06 2021-03-24T22:07:17.716 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.717 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.717 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting_for_map not empty", 2021-03-24T22:07:17.717 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:35.446204+0000", 2021-03-24T22:07:17.717 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.014252233 2021-03-24T22:07:17.717 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.718 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.718 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:07:17.718 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:35.460457+0000", 2021-03-24T22:07:17.718 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1527999999999997e-05 2021-03-24T22:07:17.719 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.719 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.719 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for blocked object", 2021-03-24T22:07:17.719 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:35.460528+0000", 2021-03-24T22:07:17.719 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0090685650000000007 2021-03-24T22:07:17.720 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.720 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.720 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:07:17.720 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:35.469597+0000", 2021-03-24T22:07:17.720 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1063000000000005e-05 2021-03-24T22:07:17.721 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.721 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.721 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for rw locks", 2021-03-24T22:07:17.721 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:35.469688+0000", 2021-03-24T22:07:17.722 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.016634431000000002 2021-03-24T22:07:17.722 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.722 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.722 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:07:17.722 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:35.486322+0000", 2021-03-24T22:07:17.723 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010351100000000001 2021-03-24T22:07:17.723 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.723 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.723 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:07:17.724 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:35.486426+0000", 2021-03-24T22:07:17.724 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000103658 2021-03-24T22:07:17.724 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.724 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.724 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:07:17.725 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:06:35.486529+0000", 2021-03-24T22:07:17.725 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.040434441000000002 2021-03-24T22:07:17.725 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:07:17.725 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:07:17.725 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:07:17.726 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.726 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.726 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:8318 7.19 7:9c4f4ea5:::smithi12556470-41:head [read 0~2117946,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e851)", 2021-03-24T22:07:17.726 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:07:01.584810+0000", 2021-03-24T22:07:17.727 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 15.804406885000001, 2021-03-24T22:07:17.727 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.039421379999999999, 2021-03-24T22:07:17.727 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:07:17.727 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:07:17.727 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:07:17.728 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:07:17.728 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:07:17.728 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 8318 2021-03-24T22:07:17.728 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.728 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:07:17.729 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.729 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:07:17.729 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:07:01.584810+0000", 2021-03-24T22:07:17.729 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:07:17.730 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.730 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.730 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:07:17.730 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:07:01.584810+0000", 2021-03-24T22:07:17.730 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999943 2021-03-24T22:07:17.731 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.731 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.731 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:07:17.731 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:07:01.584804+0000", 2021-03-24T22:07:17.731 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3207999999999999e-05 2021-03-24T22:07:17.732 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.732 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.732 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:07:17.732 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:07:01.584827+0000", 2021-03-24T22:07:17.733 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0253999999999998e-05 2021-03-24T22:07:17.733 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.733 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.733 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:07:17.733 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:07:01.584868+0000", 2021-03-24T22:07:17.734 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6319000000000001e-05 2021-03-24T22:07:17.734 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.734 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.734 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:07:17.734 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:07:01.584884+0000", 2021-03-24T22:07:17.735 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015603599999999999 2021-03-24T22:07:17.735 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.735 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.735 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:07:17.736 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:07:01.585040+0000", 2021-03-24T22:07:17.736 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7856999999999999e-05 2021-03-24T22:07:17.736 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.736 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.736 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op must wait for map", 2021-03-24T22:07:17.737 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:07:01.585058+0000", 2021-03-24T22:07:17.737 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.036086820999999998 2021-03-24T22:07:17.737 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.737 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.737 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:07:17.738 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:07:01.621145+0000", 2021-03-24T22:07:17.738 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00025149900000000003 2021-03-24T22:07:17.738 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.738 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.739 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:07:17.739 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:07:01.621396+0000", 2021-03-24T22:07:17.739 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0028351259999999999 2021-03-24T22:07:17.739 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.739 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.740 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:07:17.740 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:07:01.624231+0000", 2021-03-24T22:07:17.740 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.039421379999999999 2021-03-24T22:07:17.740 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:07:17.740 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:07:17.741 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:07:17.741 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.741 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.741 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:8319 7.19 7:9c4f4ea5:::smithi12556470-41:head [read 0~2117946,checksum] snapc 0=[] ondisk+read+known_if_redirected e851)", 2021-03-24T22:07:17.742 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:07:01.584916+0000", 2021-03-24T22:07:17.742 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 15.804301098, 2021-03-24T22:07:17.742 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.041172035000000003, 2021-03-24T22:07:17.742 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:07:17.742 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:07:17.743 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:07:17.743 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:07:17.743 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:07:17.743 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 8319 2021-03-24T22:07:17.744 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.744 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:07:17.744 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.744 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:07:17.744 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:07:01.584916+0000", 2021-03-24T22:07:17.745 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:07:17.745 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.745 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.745 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:07:17.745 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:07:01.584916+0000", 2021-03-24T22:07:17.746 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2021-03-24T22:07:17.746 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.746 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.746 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:07:17.747 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:07:01.584914+0000", 2021-03-24T22:07:17.747 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9030000000000004e-06 2021-03-24T22:07:17.747 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.747 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.747 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:07:17.748 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:07:01.584922+0000", 2021-03-24T22:07:17.748 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.8630000000000003e-06 2021-03-24T22:07:17.748 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.748 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.748 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:07:17.749 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:07:01.584932+0000", 2021-03-24T22:07:17.749 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.0369999999999997e-06 2021-03-24T22:07:17.749 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.749 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.750 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:07:17.750 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:07:01.584939+0000", 2021-03-24T22:07:17.750 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00026045799999999998 2021-03-24T22:07:17.750 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.750 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.751 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:07:17.751 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:07:01.585200+0000", 2021-03-24T22:07:17.751 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5211000000000001e-05 2021-03-24T22:07:17.751 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.751 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.752 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting_for_map not empty", 2021-03-24T22:07:17.752 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:07:01.585215+0000", 2021-03-24T22:07:17.752 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.039105147 2021-03-24T22:07:17.752 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.753 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.753 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:07:17.753 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:07:01.624320+0000", 2021-03-24T22:07:17.753 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00022859000000000001 2021-03-24T22:07:17.753 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.754 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.754 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:07:17.754 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:07:01.624548+0000", 2021-03-24T22:07:17.754 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001539161 2021-03-24T22:07:17.755 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.755 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.755 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:07:17.755 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:07:01.626088+0000", 2021-03-24T22:07:17.755 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.041172035000000003 2021-03-24T22:07:17.756 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:07:17.756 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:07:17.756 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:07:17.756 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.756 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.757 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.14760.0:8320 7.19 7:9c4f4ea5:::smithi12556470-41:head [sparse-read 0~1478258] snapc 0=[] ondisk+read+known_if_redirected e851)", 2021-03-24T22:07:17.757 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2021-03-24T22:07:01.585011+0000", 2021-03-24T22:07:17.757 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 15.804205594000001, 2021-03-24T22:07:17.757 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.041329713999999997, 2021-03-24T22:07:17.758 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2021-03-24T22:07:17.758 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2021-03-24T22:07:17.758 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2021-03-24T22:07:17.758 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.14760", 2021-03-24T22:07:17.758 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "v1:172.21.15.125:0/2843332919", 2021-03-24T22:07:17.759 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 8320 2021-03-24T22:07:17.759 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.759 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2021-03-24T22:07:17.759 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.760 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2021-03-24T22:07:17.760 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:07:01.585011+0000", 2021-03-24T22:07:17.760 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2021-03-24T22:07:17.760 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.760 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.761 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2021-03-24T22:07:17.761 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:07:01.585011+0000", 2021-03-24T22:07:17.761 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2021-03-24T22:07:17.761 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.761 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.762 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2021-03-24T22:07:17.762 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:07:01.585010+0000", 2021-03-24T22:07:17.762 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0420000000000002e-06 2021-03-24T22:07:17.762 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.763 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.763 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2021-03-24T22:07:17.763 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:07:01.585015+0000", 2021-03-24T22:07:17.763 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.1519999999999992e-06 2021-03-24T22:07:17.763 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.764 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.764 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2021-03-24T22:07:17.764 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:07:01.585023+0000", 2021-03-24T22:07:17.764 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2650000000000002e-06 2021-03-24T22:07:17.764 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.765 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.765 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2021-03-24T22:07:17.765 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:07:01.585030+0000", 2021-03-24T22:07:17.765 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00041299499999999999 2021-03-24T22:07:17.766 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.766 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.766 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:07:17.766 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:07:01.585443+0000", 2021-03-24T22:07:17.766 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4642e-05 2021-03-24T22:07:17.767 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.767 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.767 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting_for_map not empty", 2021-03-24T22:07:17.767 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:07:01.585457+0000", 2021-03-24T22:07:17.767 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.040669782000000002 2021-03-24T22:07:17.768 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.768 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.768 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2021-03-24T22:07:17.768 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:07:01.626127+0000", 2021-03-24T22:07:17.769 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.7151000000000006e-05 2021-03-24T22:07:17.769 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.769 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.769 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2021-03-24T22:07:17.769 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:07:01.626214+0000", 2021-03-24T22:07:17.770 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012674499999999999 2021-03-24T22:07:17.770 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2021-03-24T22:07:17.770 INFO:teuthology.orchestra.run.smithi142.stdout: { 2021-03-24T22:07:17.770 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2021-03-24T22:07:17.771 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2021-03-24T22:07:01.626341+0000", 2021-03-24T22:07:17.771 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.041329713999999997 2021-03-24T22:07:17.771 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:07:17.771 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:07:17.771 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:07:17.772 INFO:teuthology.orchestra.run.smithi142.stdout: } 2021-03-24T22:07:17.772 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2021-03-24T22:07:17.772 INFO:teuthology.orchestra.run.smithi142.stdout:} 2021-03-24T22:07:17.772 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:07:17.774 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:17 smithi142 conmon[20125]: cluster 2021-03-24T22:07:15.784139+0000 osd.3 ( 2021-03-24T22:07:17.774 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:17 smithi142 conmon[20125]: osd.3) 168 : cluster [DBG] 5.17 scrub ok 2021-03-24T22:07:17.775 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:17 smithi142 conmon[20125]: cluster 2021-03-24T22:07:16.235052+0000 osd.2 (osd.2) 2021-03-24T22:07:17.775 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:17 smithi142 conmon[20125]: 141 : cluster [DBG] 6.1c scrub ok 2021-03-24T22:07:17.775 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:17 smithi142 conmon[20125]: cluster 2021-03-24T22:07:17.775 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:17 smithi142 conmon[20125]: 2021-03-24T22:07:17.212789+0000 osd.6 (osd.6) 90 : cluster [DBG] 4.a scrub ok 2021-03-24T22:07:17.775 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:17 smithi142 conmon[20125]: cluster 2021- 2021-03-24T22:07:17.776 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:17 smithi142 conmon[20125]: 03-24T22:07:17.477770+0000 mon.a (mon.0 2021-03-24T22:07:17.776 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:17 smithi142 conmon[20125]: ) 2119 : cluster [DBG] osdmap e867: 8 total, 8 up, 8 in 2021-03-24T22:07:17.777 INFO:tasks.rados.rados.0.smithi125.stdout:3359: done (0 left) 2021-03-24T22:07:17.777 INFO:tasks.rados.rados.0.smithi125.stdout:3360: delete oid 12 current snap is 393 2021-03-24T22:07:17.777 INFO:tasks.rados.rados.0.smithi125.stdout:3360: done (0 left) 2021-03-24T22:07:17.777 INFO:tasks.rados.rados.0.smithi125.stdout:3361: write (excl) oid 40 current snap is 393 2021-03-24T22:07:17.778 INFO:tasks.rados.rados.0.smithi125.stdout:3361: seq_num 1145 ranges {444015=413055,1436000=773735,2709915=534100} 2021-03-24T22:07:17.778 INFO:tasks.rados.rados.0.smithi125.stdout:3361: writing smithi12556470-40 from 444015 to 857070 tid 1 2021-03-24T22:07:17.778 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:17 smithi125 conmon[19936]: cluster 2021-03-24T22:07:15.784139+0000 osd.3 (osd.3) 168 : 2021-03-24T22:07:17.778 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:17 smithi125 conmon[19936]: cluster [DBG] 5.17 scrub ok 2021-03-24T22:07:17.779 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:17 smithi125 conmon[19936]: cluster 2021-03-24T22:07:16.235052+0000 osd.2 (osd.2) 141 : cluster [DBG] 6.1c scrub ok 2021-03-24T22:07:17.779 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:17 smithi125 conmon[19936]: cluster 2021-03-24T22 2021-03-24T22:07:17.779 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:17 smithi125 conmon[19936]: :07:17.212789+0000 2021-03-24T22:07:17.781 INFO:tasks.rados.rados.0.smithi125.stdout:3361: writing smithi12556470-40 from 1436000 to 2209735 tid 2 2021-03-24T22:07:17.782 INFO:tasks.rados.rados.0.smithi125.stdout:3361: writing smithi12556470-40 from 2709915 to 3244015 tid 3 2021-03-24T22:07:17.782 INFO:tasks.rados.rados.0.smithi125.stdout:3362: read oid 42 snap 348 2021-03-24T22:07:17.782 INFO:tasks.rados.rados.0.smithi125.stdout:3362: expect (ObjNum 736 snap 235 seq_num 736) 2021-03-24T22:07:17.782 INFO:tasks.rados.rados.0.smithi125.stdout:3363: read oid 35 snap -1 2021-03-24T22:07:17.783 INFO:tasks.rados.rados.0.smithi125.stdout:3363: expect (ObjNum 1121 snap 383 seq_num 1121) 2021-03-24T22:07:17.783 INFO:tasks.rados.rados.0.smithi125.stdout:3364: copy_from oid 31 from oid 16 current snap is 393 2021-03-24T22:07:17.783 INFO:tasks.rados.rados.0.smithi125.stdout:3361: finishing write tid 1 to smithi12556470-40 2021-03-24T22:07:17.783 INFO:tasks.rados.rados.0.smithi125.stdout:3365: rollback oid 10 current snap is 393 2021-03-24T22:07:17.783 INFO:tasks.rados.rados.0.smithi125.stdout:rollback oid 10 to 390 2021-03-24T22:07:17.784 INFO:tasks.rados.rados.0.smithi125.stdout:3361: finishing write tid 2 to smithi12556470-40 2021-03-24T22:07:17.784 INFO:tasks.rados.rados.0.smithi125.stdout:3366: copy_from oid 37 from oid 9 current snap is 393 2021-03-24T22:07:17.784 INFO:tasks.rados.rados.0.smithi125.stdout:3367: copy_from oid 3 from oid 1 current snap is 393 2021-03-24T22:07:17.784 INFO:tasks.rados.rados.0.smithi125.stdout:3368: snap_create 2021-03-24T22:07:17.785 INFO:tasks.rados.rados.0.smithi125.stdout:3367: finishing copy_from to smithi12556470-3 2021-03-24T22:07:17.785 INFO:tasks.rados.rados.0.smithi125.stdout:3367: got expected ENOENT (src dne) 2021-03-24T22:07:17.785 INFO:tasks.rados.rados.0.smithi125.stdout:3367: finishing copy_from racing read to smithi12556470-3 2021-03-24T22:07:17.785 INFO:tasks.rados.rados.0.smithi125.stdout:3365: finishing rollback tid 1 to smithi12556470-10 2021-03-24T22:07:17.785 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 10 v 1459 (ObjNum 1117725456 snap 21916 seq_num 3308612576) dirty dne 2021-03-24T22:07:17.786 INFO:tasks.rados.rados.0.smithi125.stdout:3362: expect (ObjNum 736 snap 235 seq_num 736) 2021-03-24T22:07:17.786 INFO:tasks.rados.rados.0.smithi125.stdout:3361: finishing write tid 3 to smithi12556470-40 2021-03-24T22:07:17.786 INFO:tasks.rados.rados.0.smithi125.stdout:3366: finishing copy_from to smithi12556470-37 2021-03-24T22:07:17.786 INFO:tasks.rados.rados.0.smithi125.stdout:3366: got expected ENOENT (src dne) 2021-03-24T22:07:17.787 INFO:tasks.rados.rados.0.smithi125.stdout:3361: finishing write tid 4 to smithi12556470-40 2021-03-24T22:07:17.787 INFO:tasks.rados.rados.0.smithi125.stdout:3366: finishing copy_from racing read to smithi12556470-37 2021-03-24T22:07:17.787 INFO:tasks.rados.rados.0.smithi125.stdout:3361: finishing write tid 5 to smithi12556470-40 2021-03-24T22:07:17.787 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 40 v 2172 (ObjNum 1145 snap 393 seq_num 1145) dirty exists 2021-03-24T22:07:17.787 INFO:tasks.rados.rados.0.smithi125.stdout:3361: left oid 40 (ObjNum 1145 snap 393 seq_num 1145) 2021-03-24T22:07:17.788 INFO:tasks.rados.rados.0.smithi125.stdout:3363: expect (ObjNum 1121 snap 383 seq_num 1121) 2021-03-24T22:07:17.837 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:07:17.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:17.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:07:17.853 INFO:tasks.rados.rados.0.smithi125.stdout:3364: finishing copy_from to smithi12556470-31 2021-03-24T22:07:17.853 INFO:tasks.rados.rados.0.smithi125.stdout:update_object_version oid 31 v 847 (ObjNum 1143 snap 392 seq_num 1143) dirty exists 2021-03-24T22:07:17.853 INFO:tasks.rados.rados.0.smithi125.stdout:3364: finishing copy_from racing read to smithi12556470-31 2021-03-24T22:07:17.912 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:17 smithi125 conmon[19936]: osd.6 (osd.6) 90 : cluster 2021-03-24T22:07:17.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:17 smithi125 conmon[19936]: [DBG] 4.a scrub ok 2021-03-24T22:07:17.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:17 smithi125 conmon[19936]: cluster 2021- 2021-03-24T22:07:17.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:17 smithi125 conmon[19936]: 03-24T22:07:17.477770+0000 2021-03-24T22:07:17.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:17 smithi125 conmon[19936]: mon.a (mon.0) 2119 : cluster [DBG] osdmap e867: 8 total, 8 up, 8 in 2021-03-24T22:07:17.913 INFO:journalctl@ceph.mon.a.smithi125.stdout:Mar 24 22:07:17 smithi125 conmon[19936]: 2021-03-24T22:07:17.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:17 smithi125 conmon[24927]: cluster 2021-03-24T22:07:15.784139+0000 2021-03-24T22:07:17.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:17 smithi125 conmon[24927]: osd.3 (osd.3) 168 : cluster [DBG] 5.17 scrub ok 2021-03-24T22:07:17.914 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:17 smithi125 conmon[24927]: cluster 2021-03-24T22:07:16.235052+0000 osd.2 (osd 2021-03-24T22:07:17.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:17 smithi125 conmon[24927]: .2) 141 : cluster [DBG] 6.1c scrub ok 2021-03-24T22:07:17.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:17 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T22:07:17.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:17 smithi125 conmon[24927]: 22:07:17.212789+0000 osd.6 (osd.6) 2021-03-24T22:07:17.915 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:17 smithi125 conmon[24927]: 90 : cluster [DBG] 4.a scrub ok 2021-03-24T22:07:17.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:17 smithi125 conmon[24927]: cluster 2021-03-24T 2021-03-24T22:07:17.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:17 smithi125 conmon[24927]: 22:07:17.477770+0000 mon.a (mon.0 2021-03-24T22:07:17.916 INFO:journalctl@ceph.mon.c.smithi125.stdout:Mar 24 22:07:17 smithi125 conmon[24927]: ) 2119 : cluster [DBG] osdmap e867: 8 total, 8 up, 8 in 2021-03-24T22:07:17.944 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:07:18.004 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4 2021-03-24T22:07:18.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:18.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.4' 2021-03-24T22:07:18.109 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:07:18.172 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2 2021-03-24T22:07:18.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:18.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.2' 2021-03-24T22:07:18.279 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:07:18.338 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:07:18.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:18.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:07:18.445 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:07:18.508 INFO:teuthology.orchestra.run.smithi125.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1 2021-03-24T22:07:18.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:18.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.1' 2021-03-24T22:07:18.521 INFO:tasks.rados.rados.0.smithi125.stdout:3361: done (7 left) 2021-03-24T22:07:18.521 INFO:tasks.rados.rados.0.smithi125.stdout:3362: done (6 left) 2021-03-24T22:07:18.521 INFO:tasks.rados.rados.0.smithi125.stdout:3363: done (5 left) 2021-03-24T22:07:18.521 INFO:tasks.rados.rados.0.smithi125.stdout:3364: done (4 left) 2021-03-24T22:07:18.522 INFO:tasks.rados.rados.0.smithi125.stdout:3365: done (3 left) 2021-03-24T22:07:18.522 INFO:tasks.rados.rados.0.smithi125.stdout:3366: done (2 left) 2021-03-24T22:07:18.522 INFO:tasks.rados.rados.0.smithi125.stdout:3367: done (1 left) 2021-03-24T22:07:18.522 INFO:tasks.rados.rados.0.smithi125.stdout:3368: done (0 left) 2021-03-24T22:07:18.523 INFO:tasks.rados.rados.0.smithi125.stdout:3369: snap_create 2021-03-24T22:07:18.528 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph daemon osd.7 dump_ops_in_flight 2021-03-24T22:07:18.616 DEBUG:teuthology.orchestra.run.smithi142:> sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:07:18.669 INFO:teuthology.orchestra.run.smithi142.stderr:Error response from daemon: Cannot kill container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7: No such container: ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7 2021-03-24T22:07:18.673 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:18 smithi142 conmon[20125]: cluster 2021-03-24T22:07:16.825316+0000 2021-03-24T22:07:18.674 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:18 smithi142 conmon[20125]: osd.3 (osd.3) 169 : cluster [DBG] 5.14 scrub ok 2021-03-24T22:07:18.674 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:18 smithi142 conmon[20125]: cluster 2021-03-24T22:07 2021-03-24T22:07:18.674 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:18 smithi142 conmon[20125]: :16.926214+0000 osd.7 (osd.7) 143 : cluster [DBG] 5.10 scrub ok 2021-03-24T22:07:18.675 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:18 smithi142 conmon[20125]: cluster 2021-03- 2021-03-24T22:07:18.675 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:18 smithi142 conmon[20125]: 24T22:07:17.024227+0000 mgr.y (mgr 2021-03-24T22:07:18.675 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:18 smithi142 conmon[20125]: .14144) 711 : cluster [DBG] pgmap v1462: 168 pgs: 168 active+clean; 592 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 10 op/s 2021-03-24T22:07:18.675 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:18 smithi142 conmon[20125]: cluster 2021-03 2021-03-24T22:07:18.675 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:18 smithi142 conmon[20125]: -24T22:07:18.161699+0000 2021-03-24T22:07:18.676 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:18 smithi142 conmon[20125]: osd.6 (osd.6) 91 : cluster [DBG] 2021-03-24T22:07:18.676 INFO:journalctl@ceph.mon.b.smithi142.stdout:Mar 24 22:07:18 smithi142 conmon[20125]: 3.26 scrub ok 2021-03-24T22:07:18.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T22:07:18.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi142 with status 1: 'sudo docker kill -s 1 ceph-3cfa339a-8cea-11eb-90b2-001a4aab830c-osd.7' 2021-03-24T22:07:18.684 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:d71e96d216190f28f757f409532464ab0eca4ea9 shell --fsid 3cfa339a-8cea-11eb-90b2-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-24T22:07:18.762 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2021-03-24T22:07:18.764 DEBUG:teuthology.task.console_log:Killing console logger for smithi125 2021-03-24T22:07:18.765 DEBUG:teuthology.task.console_log:Killing console logger for smithi142 2021-03-24T22:07:18.765 DEBUG:teuthology.exit:Finished running handlers